| *** thrash is now known as thrash|g0ne | 00:02 | |
| *** tbonds has joined #tripleo | 00:07 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 00:10 |
| *** tosky has quit IRC | 00:18 | |
| *** tobias_fiberdata has quit IRC | 00:36 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: [WIP] Quickstart transition: fallback to old method on stable branches https://review.openstack.org/445461 | 00:42 |
| *** deadnull has quit IRC | 00:48 | |
| openstackgerrit | Ian Wienand proposed openstack/instack master: Call dib-run-parts from dib https://review.openstack.org/446796 | 00:55 |
| *** limao has joined #tripleo | 01:01 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 01:10 |
| *** sshnaidm is now known as sshnaidm|off | 01:19 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Run dib-run-parts out of /tmp https://review.openstack.org/446769 | 01:23 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Don't provide dib-run-parts https://review.openstack.org/446285 | 01:23 |
| EmilienM | bkero: any good news? | 01:37 |
| EmilienM | panda|zZ: ^ | 01:38 |
| *** dmacpher-afk is now known as dmacpher | 01:41 | |
| *** funzo has joined #tripleo | 01:41 | |
| *** funzo has quit IRC | 01:45 | |
| *** Rodolof1 has quit IRC | 01:53 | |
| bkero | EmilienM: Sadly no. The tests failed. | 01:59 |
| EmilienM | bkero: what is the plan b? revert? | 01:59 |
| bkero | EmilienM: http://logs.openstack.org/50/446750/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/4bc7b91/ | 01:59 |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 02:10 |
| *** liupe__ has joined #tripleo | 02:24 | |
| *** liupe__ has quit IRC | 02:25 | |
| *** tzumainn has quit IRC | 02:25 | |
| *** fragatin_ has joined #tripleo | 02:34 | |
| *** george_goh|gone is now known as george_goh | 02:34 | |
| *** yamahata has quit IRC | 02:35 | |
| openstackgerrit | xhzhf proposed openstack/tripleo-incubator master: H803 hacking have been deprecated https://review.openstack.org/413938 | 02:35 |
| *** fzdarsky_ has joined #tripleo | 02:36 | |
| *** fragatina has quit IRC | 02:37 | |
| *** oaken_chris has quit IRC | 02:38 | |
| *** fragatin_ has quit IRC | 02:39 | |
| *** fzdarsky|afk has quit IRC | 02:40 | |
| *** funzo has joined #tripleo | 02:42 | |
| *** funzo has quit IRC | 02:46 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 03:10 |
| *** psahoo has joined #tripleo | 03:15 | |
| *** rwsu has joined #tripleo | 03:23 | |
| *** chlong has quit IRC | 03:25 | |
| *** gbarros has quit IRC | 03:28 | |
| *** funzo has joined #tripleo | 03:42 | |
| *** funzo has quit IRC | 03:47 | |
| *** udesale has joined #tripleo | 04:00 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 04:10 |
| *** yamahata has joined #tripleo | 04:13 | |
| *** owalsh_ has joined #tripleo | 04:20 | |
| *** skramaja has joined #tripleo | 04:20 | |
| *** owalsh has quit IRC | 04:23 | |
| *** fragatina has joined #tripleo | 04:31 | |
| *** fragatina has quit IRC | 04:32 | |
| *** fragatina has joined #tripleo | 04:33 | |
| *** funzo has joined #tripleo | 04:43 | |
| *** funzo has quit IRC | 04:47 | |
| *** ratailor has joined #tripleo | 04:57 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 05:10 |
| *** nyechiel has joined #tripleo | 05:21 | |
| *** fragatina has quit IRC | 05:40 | |
| *** yprokule has joined #tripleo | 05:58 | |
| *** ealcaniz has joined #tripleo | 06:07 | |
| *** masco has joined #tripleo | 06:10 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 06:10 |
| openstackgerrit | Ian Wienand proposed openstack/instack master: Call dib-run-parts from dib https://review.openstack.org/446796 | 06:12 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Run dib-run-parts out of /tmp https://review.openstack.org/446769 | 06:19 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Install subunit-filters as required by subunit2junitxml https://review.openstack.org/446856 | 06:22 |
| *** saneax-_-|AFK is now known as saneax | 06:27 | |
| *** Vijayendra has joined #tripleo | 06:30 | |
| *** abehl has joined #tripleo | 06:32 | |
| *** lmiccini has joined #tripleo | 06:33 | |
| *** jaganathan has joined #tripleo | 06:41 | |
| *** dparkes has joined #tripleo | 06:41 | |
| *** pgadiya has joined #tripleo | 06:42 | |
| *** funzo has joined #tripleo | 06:45 | |
| *** pgadiya has left #tripleo | 06:45 | |
| *** yolanda has joined #tripleo | 06:45 | |
| *** funzo has quit IRC | 06:49 | |
| bkero | EmilienM, trown|outtypewww: I found it. http://logs.openstack.org/50/446750/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/4bc7b91/logs/undercloud/home/jenkins/toci_vxlan_networking.sh.log.txt.gz | 06:58 |
| *** dparkes has quit IRC | 06:59 | |
| *** djain has joined #tripleo | 07:01 | |
| *** dparkes has joined #tripleo | 07:02 | |
| bkero | tripleo.sh --multinode-setup calls --repo-setup and that isn't including the STABLE_RELEASE environment variable. | 07:04 |
| bkero | (also yay for wget --recursive -e robots=off http://logs/ && grep | 07:04 |
| bkero | ) | 07:04 |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: oooq quickstart: Add STABLE_RELEASE env variable to subnodes https://review.openstack.org/446746 | 07:05 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-heat-templates stable/ocata: TEST CHANGE, IGNORE https://review.openstack.org/446865 | 07:07 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: [TEST JOB] test tempest-12.2.0 for Mitaka https://review.openstack.org/446868 | 07:08 |
| bkero | http://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/scripts/tripleo.sh#n1303 | 07:09 |
| bkero | Ideally we can use a different vxlan setup script directly | 07:09 |
| bkero | I'm guessing that altering that script will break other jobs though. | 07:09 |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 07:10 |
| bkero | commented on the launchpad bug | 07:15 |
| *** jaosorior has joined #tripleo | 07:16 | |
| bkero | Someone tell me if I sound crazy? :) | 07:16 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Switch stackrc and undercloud.py to use Keystone v3 https://review.openstack.org/446752 | 07:18 |
| jaosorior | bkero: alright, pass the sound file | 07:19 |
| bkero | jaosorior: mic is broken on my laptop :< | 07:23 |
| bkero | I'll offer this instead: http://staff.osuosl.org/~bkero/The%20Trial%20in%20Concert.mp3 | 07:23 |
| jaosorior | bkero: well, that was pretty good | 07:24 |
| jaosorior | bkero: not very crazy sounding though | 07:24 |
| jaosorior | s/was/is/ | 07:25 |
| bkero | http://ocremix.org/remix/OCR00800 | 07:25 |
| * jaosorior still listening to the song | 07:25 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Refactor: block-device filesystem creation, mount and fstab https://review.openstack.org/444586 | 07:25 |
| *** cylopez has joined #tripleo | 07:28 | |
| *** ccamacho has joined #tripleo | 07:29 | |
| *** tesseract has joined #tripleo | 07:34 | |
| openstackgerrit | Ian Wienand proposed openstack/instack master: Call dib-run-parts from dib https://review.openstack.org/446796 | 07:36 |
| *** abehl has quit IRC | 07:42 | |
| *** pcaruana has joined #tripleo | 07:43 | |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: Collect more info about docker https://review.openstack.org/441050 | 07:45 |
| *** funzo has joined #tripleo | 07:46 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Switch stackrc and undercloud.py to use Keystone v3 https://review.openstack.org/446752 | 07:50 |
| *** funzo has quit IRC | 07:50 | |
| jaosorior | jistr: hey dude, could you check this out https://review.openstack.org/#/c/444876/ ? | 07:52 |
| *** tobias_fiberdata has joined #tripleo | 07:53 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: add caching the GetParametersAction https://review.openstack.org/444220 | 07:55 |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #tripleo | 08:00 | |
| jaosorior | mandre: I need some help debugging the keystone container, let me know if you have some time | 08:00 |
| *** limao has quit IRC | 08:00 | |
| mandre | jaosorior: I do now | 08:01 |
| *** limao has joined #tripleo | 08:01 | |
| jaosorior | mandre: so I tried to move the keystone fernet binding to the container to use a directory instead of having to specify the files individually | 08:02 |
| jaosorior | mandre: this is the commit https://review.openstack.org/#/c/446473/ | 08:02 |
| jaosorior | mandre: it depends on this one that enables the volumes to skip empty ones https://review.openstack.org/#/c/446472/2 | 08:02 |
| jaosorior | mandre: but anyway, in this case it isn't empty (since it's the default) | 08:02 |
| mandre | jaosorior: so far so good | 08:03 |
| *** karthiks has quit IRC | 08:03 | |
| *** pmannidi has quit IRC | 08:04 | |
| *** jlinkes has joined #tripleo | 08:04 | |
| jaosorior | mandre: so, right now, it's failing because of this error (I think): /etc/keystone/fernet-keys does not exist | 08:04 |
| jaosorior | mandre: however, I find no logs from at what step that happened and such http://logs.openstack.org/73/446473/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/ec5a4dd/logs/oooq/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 08:04 |
| jaosorior | well, i know it happened in ControllerContainersDeployment_Step3 | 08:04 |
| jaosorior | but not the exact action it was doing | 08:04 |
| *** jpena|off is now known as jpena | 08:05 | |
| mandre | jaosorior: looks like your bind mount is not working as expected, http://logs.openstack.org/73/446473/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/ec5a4dd/logs/oooq/overcloud-controller-0/var/log/extra/docker/containers/keystone/docker_info.log.txt.gz | 08:08 |
| mandre | jaosorior: it's not mounting /etc/keystone/fernet-keys inside your keystone container | 08:08 |
| *** fzdarsky_ is now known as fzdarsky | 08:10 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 08:10 |
| *** gfidente has joined #tripleo | 08:11 | |
| jaosorior | mandre: what runs first? the bits in docker_config or the bits in docker_puppet_tasks? | 08:11 |
| mandre | jaosorior: normally the docker_puppet_tasks | 08:14 |
| jaosorior | mandre: right, so the deal is that puppet (or I guess it would run in the docker_puppet_tasks) would usually create this directory :/ | 08:14 |
| *** paramite has joined #tripleo | 08:15 | |
| *** chem has quit IRC | 08:16 | |
| *** chem has joined #tripleo | 08:16 | |
| mandre | jaosorior: it looks to me that keystone_fernet_tokens was set to false | 08:17 |
| *** zoli|gone is now known as zoli|wfh | 08:17 | |
| *** jpena is now known as jpena|off | 08:18 | |
| *** florianf has joined #tripleo | 08:18 | |
| mandre | jaosorior: that would explain why it didn't bind mount the fernet-keys directory, and why puppet didn't generate the keys in the first place | 08:18 |
| mandre | jaosorior: this is the content of /etc/keystone that was generated by puppet, http://logs.openstack.org/73/446473/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-containers-oooq-nv/ec5a4dd/logs/oooq/overcloud-controller-0/var/log/extra/docker/containers/keystone/etc/keystone/ | 08:19 |
| *** karthiks has joined #tripleo | 08:19 | |
| *** paramite has quit IRC | 08:21 | |
| *** apetrich has quit IRC | 08:21 | |
| mandre | jaosorior: fwiw the docker keystone service defaults to uuid, https://review.openstack.org/#/c/446473/2/docker/services/keystone.yaml@36 | 08:21 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: docker/keystone: Bind mount entire fernet keys repository https://review.openstack.org/446473 | 08:23 |
| jaosorior | mandre: the default in puppet/services/keystone is fernet :/ | 08:23 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: create history for the overcloud deploy command https://review.openstack.org/446617 | 08:24 |
| *** jpena|off is now known as jpena | 08:25 | |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: oooq quickstart: Add STABLE_RELEASE env variable to subnodes https://review.openstack.org/446746 | 08:26 |
| *** suuuper has joined #tripleo | 08:30 | |
| *** abehl has joined #tripleo | 08:30 | |
| mandre | jaosorior: could it be that the default value from the docker service is overriding the one from puppet? | 08:32 |
| *** apetrich has joined #tripleo | 08:33 | |
| openstackgerrit | Luca Lorenzetto proposed openstack/tripleo-heat-templates master: Pick dynamically the first node for stack validation https://review.openstack.org/446887 | 08:35 |
| *** skramaja has quit IRC | 08:37 | |
| *** amoralej|off is now known as amoralej | 08:38 | |
| mandre | jaosorior: the docker services are evaluated before the puppet ones, so it's the one providing the default value for KeystoneTokenProvider | 08:39 |
| *** karthiks has quit IRC | 08:39 | |
| openstackgerrit | Ian Wienand proposed openstack/instack master: Call dib-run-parts from dib https://review.openstack.org/446796 | 08:39 |
| jaosorior | mandre: well, there is no parameter_defaults that sets that value | 08:39 |
| jaosorior | mandre: so it seems to me that the fact that they have different values makes them evaluate differently on each stack | 08:40 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: create history for the overcloud deploy command https://review.openstack.org/446617 | 08:40 |
| *** udesale has quit IRC | 08:41 | |
| *** udesale has joined #tripleo | 08:41 | |
| jaosorior | marios: could you check this one too https://review.openstack.org/#/c/444885/ ? :D\ | 08:43 |
| *** funzo has joined #tripleo | 08:46 | |
| openstackgerrit | Luca Lorenzetto proposed openstack/tripleo-heat-templates stable/newton: Moved some variable definitions to global scope https://review.openstack.org/446893 | 08:46 |
| *** djain has quit IRC | 08:48 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Switch stackrc and undercloud.py to use Keystone v3 https://review.openstack.org/446752 | 08:50 |
| *** funzo has quit IRC | 08:51 | |
| *** karthiks has joined #tripleo | 08:51 | |
| marios | jaosorior: sure | 08:52 |
| *** shardy has joined #tripleo | 09:01 | |
| jaosorior | marios: thanks dude, answered | 09:04 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Remove certificate request bits from service profiles https://review.openstack.org/444891 | 09:05 |
| jaosorior | mandre: could you check this out https://review.openstack.org/#/c/446593/ | 09:06 |
| jaosorior | mandre: also, do you have a running containerized overcloud somewhere? | 09:07 |
| jaosorior | mandre: if you do, could you check if mod_ssl is available? | 09:07 |
| marios | thanks jaosorior | 09:08 |
| *** bogdando has joined #tripleo | 09:09 | |
| mandre | jaosorior: in which container do you want me to look specifically | 09:10 |
| *** dtantsur|sick is now known as dtantsur | 09:10 | |
| jaosorior | mandre: lets try keystone | 09:10 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 09:10 |
| jaosorior | mandre: well, keystone and heat :P | 09:10 |
| *** chem has quit IRC | 09:13 | |
| mandre | jaosorior: nope, none of them have mod_ssl | 09:13 |
| *** saibarspeis has joined #tripleo | 09:13 | |
| mandre | jaosorior: they should be added to the kolla images IMO | 09:13 |
| *** skramaja has joined #tripleo | 09:14 | |
| jaosorior | mandre: wasn't Dan talking about moving away from kolla? or was it just their utils? | 09:15 |
| shardy | jaosorior: I thought it was just the configuration interfaces, not the images | 09:16 |
| jaosorior | ok | 09:17 |
| mandre | jaosorior: yep, Dan referred to how we pass config to the containers, not moving away from the kolla images | 09:18 |
| jaosorior | mandre: ok, thanks. I'll attempt to do a commit to add those to kolla then | 09:19 |
| mandre | jaosorior: see an example of removing the kolla_config here, https://review.openstack.org/#/c/445620/ | 09:20 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/446907 | 09:20 |
| *** mcornea has joined #tripleo | 09:21 | |
| dtantsur | folks, could you check two easy reviews that unblock using ironic as a stand-alone role? https://review.openstack.org/#/c/446037/ and https://review.openstack.org/#/c/446074/ | 09:23 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/446910 | 09:23 |
| *** karimb has joined #tripleo | 09:24 | |
| *** limao has quit IRC | 09:25 | |
| *** zoli|wfh is now known as zoli|brb | 09:27 | |
| shardy | dtantsur: lgtm, +2 | 09:27 |
| jaosorior | mandre: added you to the commit in kolla | 09:27 |
| dtantsur | thnx | 09:29 |
| *** chem has joined #tripleo | 09:34 | |
| *** athomas has joined #tripleo | 09:38 | |
| *** karthiks is now known as karthiks_afk | 09:42 | |
| *** zoli|brb is now known as zoli|wfh | 09:43 | |
| saneax | hi folks, pike would have puppet 4 (planned) or still pupept 3 | 09:44 |
| *** ckyriakidou has joined #tripleo | 09:46 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 09:47 |
| *** funzo has joined #tripleo | 09:47 | |
| shardy | saneax: Hi, my trunk/pike dev environment has 4.6.2 | 09:49 |
| saneax | shardy, thanks | 09:49 |
| saneax | so just in case if we are writing modules for pike, it should be on puppet 4 | 09:50 |
| *** funzo has quit IRC | 09:52 | |
| *** gchamoul|afk is now known as gchamoul | 09:56 | |
| *** gchamoul is now known as gchamoul|afk | 09:56 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Add NodeCreateBatchSize parameter https://review.openstack.org/446927 | 09:56 |
| shardy | rook, jkilpatr: ^^ FYI ref the discussion about needing 30 node batches due to Ironic/TFTP | 09:57 |
| dtantsur | shardy, this is all good. we need to talk, however, about possibility of multi-node undercloud with ironic services split away.. | 10:02 |
| gfidente | ping jistr | 10:02 |
| jistr | gfidente: oh right :) | 10:02 |
| *** salmankhan has joined #tripleo | 10:03 | |
| *** rcernin has joined #tripleo | 10:04 | |
| *** tosky has joined #tripleo | 10:05 | |
| *** gchamoul|afk is now known as gchamoul | 10:06 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 10:10 |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: WIP oooq quickstart: Add STABLE_RELEASE env variable to subnodes https://review.openstack.org/446746 | 10:10 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 10:11 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-heat-templates stable/ocata: TEST CHANGE, IGNORE https://review.openstack.org/446865 | 10:11 |
| *** nyechiel has quit IRC | 10:17 | |
| *** aufi has joined #tripleo | 10:20 | |
| *** akrivoka has joined #tripleo | 10:24 | |
| apetrich | Morning | 10:26 |
| *** deadnull has joined #tripleo | 10:30 | |
| *** deadnull has quit IRC | 10:30 | |
| sshnaidm|off | bkero, we have function subnodes_scp_deploy_env which set up env for tripleo.sh, it's used by panda|zZ here: https://github.com/openstack-infra/tripleo-ci/blob/7fdbe42018867634fb7c465da82f14d93c081b57/toci_gate_test-oooq.sh#L205-L205 | 10:33 |
| sshnaidm|off | bkero, but it didn't work in case of toci_vlan_... script, because you have different $tripleo_root location | 10:34 |
| sshnaidm|off | bkero, it's better to align it across scripts or use this function before you run multinode-setup | 10:35 |
| shardy | dtantsur: Hmm, so can you explain the reasons for wanting to split the ironic services? | 10:35 |
| dtantsur | shardy, I was told by an experienced operator that you can't too much with scaling TFTP, except for moving it to a separate service. | 10:35 |
| shardy | dtantsur: and, if we had to do that, could it be a special "overcloud" stack that runs the ironic services as a custom role? | 10:35 |
| dtantsur | shardy, I don't think I understand your suggestion fully. Maybe? | 10:36 |
| shardy | dtantsur: so we'd have to run multiple boxes just to work around a tftp bottleneck? | 10:36 |
| shardy | that sounds sub-optimal, since it doesn't sounds like the hardware was the bottleneck | 10:36 |
| shardy | dtantsur: I'm suggesting that we don't invent a new way to deploy ironic on a separate node, but instead deploy multiple overcloud stacks, one running just ironic, and one running the actual overcloud | 10:37 |
| dtantsur | another option is to get rid of TFTP :) this is vendor-specific, certain vendors do not have virtual media, and nor do they have iPXE built-in | 10:37 |
| *** saibarspeis has quit IRC | 10:37 | |
| dtantsur | mmmmm | 10:37 |
| dtantsur | shardy, may have interesting network implications, but should work | 10:37 |
| dtantsur | but does it imply essentially getting seed back? | 10:38 |
| dtantsur | with undercloud = seed, and this ironic overcloud = new undercloud | 10:38 |
| dtantsur | ? | 10:38 |
| shardy | dtantsur: honestly, I'd much prefer we worked out a way to solve the tftp issue, as I assume this means one node isn't enough as we'll just move the tftp bottleneck to a different box? | 10:38 |
| *** florianf has quit IRC | 10:38 | |
| shardy | dtantsur: yeah, something like that | 10:39 |
| shardy | ugh | 10:39 |
| dtantsur | kinda | 10:39 |
| *** salmankhan has quit IRC | 10:39 | |
| dtantsur | I mean, TFTP is bad :) it's hard to get solved. what we do is work around. one work around is to use virtual media (vendor-specific), the second one is to throw in more TFTP box(es), the third one is layered deployment | 10:39 |
| dtantsur | jkilpatr actually suggested a more sophisticated layered approach IIRC | 10:40 |
| dtantsur | with undercloud deploying more ironic nodes, then all them together deploying overcloud, then additional ironic nodes torn down and used for scaling up the overcloud a bit more | 10:40 |
| dtantsur | not quite possible with current tripleo, but interesting | 10:40 |
| shardy | dtantsur: so, if the hardware/nics aren't saturated, what exactly is the bottleneck? | 10:41 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix repositories setup https://review.openstack.org/446949 | 10:42 |
| dtantsur | shardy, what I learn it that an HTTP service saturates a TFTP service quite easily. the symptoms are apparently the same. I don't know more details | 10:42 |
| shardy | dtantsur: can we run more tftp services without adding more physical hardware? | 10:42 |
| shardy | e.g on multiple ports, in containers, or ... | 10:42 |
| dtantsur | 1. no, 2. OMG NO | 10:42 |
| shardy | to reach the point where the hardware actually is the bottleneck? | 10:43 |
| shardy | Ok so we just throw more hardware at the problem and have it 10% utilized? | 10:43 |
| dtantsur | well, it's not me who invented TFTP, don't shoot the messenger ;) | 10:43 |
| *** salmankhan has joined #tripleo | 10:44 | |
| shardy | dtantsur: sure, I'm just throwing ideas around as IMO adding hardware when the current hardware isn't saturated is just wrong | 10:44 |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-heat-templates stable/ocata: DONT REVIEW: test ocata bug https://review.openstack.org/446951 | 10:45 |
| dtantsur | I don't like it either, at all | 10:45 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 10:45 |
| dtantsur | it may be something like kernel pipelining issue | 10:45 |
| shardy | and the traditional solution to that is more threads/processes/nics/etc | 10:45 |
| dtantsur | not really | 10:45 |
| bkero | sshnaidm|off: Yes, I'm trying to rewrite some of the toci-vxlan-networking extras role to be less fancy | 10:46 |
| dtantsur | if active TCP communication makes UDP package drop rate too high, more threads on user-space side won't help much | 10:46 |
| *** rcernin has quit IRC | 10:46 | |
| bkero | sshnaidm|off: I fear that the devstack-gate vxlan scripts will use their incorrect repos to install openvswitch and neutron, and isntalling packages later might resultin a bad state | 10:48 |
| *** funzo has joined #tripleo | 10:48 | |
| *** paramite has joined #tripleo | 10:48 | |
| *** udesale has quit IRC | 10:48 | |
| sshnaidm|off | bkero, but it works in old multinode jobs | 10:49 |
| *** florianf has joined #tripleo | 10:49 | |
| sshnaidm|off | bkero, I submitted a workaround until we fix it properly, take a look: https://review.openstack.org/#/c/446949/ | 10:50 |
| bkero | sshnaidm|off: Yes it does. However, old multinode jobs didn't have ansible in the middle making environment variables ambiguous. | 10:50 |
| bkero | sshnaidm|off: correct, I read it. I'm not sure it will work | 10:50 |
| bkero | We'll see | 10:50 |
| sshnaidm|off | it's tested here: https://review.openstack.org/446951 | 10:51 |
| adarazs | bkero, sshnaidm|off: thanks for jumping on this. let me know if it looks like you fixed it and there's something to review and submit. | 10:52 |
| *** funzo has quit IRC | 10:53 | |
| * bkero been on this for the last 5 hours, trying various fixes. >_> I might not be in much tomorrow. | 10:53 | |
| *** jkilpatr has quit IRC | 10:53 | |
| adarazs | bkero: understood, just dump your brain state when you're planning to leave and then I'll pick it up :) | 10:53 |
| *** yamahata has quit IRC | 10:54 | |
| *** athomas has quit IRC | 10:54 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Add role specific information to the service template https://review.openstack.org/437956 | 10:54 |
| bkero | adarazs: ok | 10:54 |
| bkero | adarazs: It's pretty well documented in the launchpad bug already | 10:54 |
| adarazs | I saw that. really neat, thanks! I just saw you made that 3 hours ago, so I figured there might be something more. | 10:55 |
| *** jaosorior has quit IRC | 10:55 | |
| *** jaosorior has joined #tripleo | 10:56 | |
| jaosorior | bandini: hey dude, could you check this out https://review.openstack.org/#/c/444885/? | 10:58 |
| bandini | jaosorior: on it | 10:58 |
| jaosorior | bandini: and this is you have some time https://review.openstack.org/#/c/444876/ | 10:59 |
| bkero | adarazs: The method I attempted didn't work. I suspect it's because the environment variable isn't being passed correctly to the undercloud-setup role, although I'm not sure why. I set up a small example ansible playbook test: http://paste.openstack.org/show/603107/" | 10:59 |
| *** suuuper has quit IRC | 11:00 | |
| bkero | Exported environment variables are correctly sourced in, although local ones are not. I need to find out which STABLE_RELEASE is | 11:00 |
| EmilienM | hello | 11:00 |
| jaosorior | EmilienM: sup dude | 11:01 |
| adarazs | EmilienM: hi | 11:01 |
| bkero | EmilienM: Hey there, good morning | 11:01 |
| adarazs | EmilienM: saw your email about the stable stuff, bkero is still on it o_o and if he leaves I'll take over. | 11:01 |
| adarazs | bkero: yeah, that's what I'd expect, yeah. if you don't export it, it doesn't propagate to the subshells | 11:02 |
| EmilienM | go sleep bkero! | 11:02 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Switch stackrc and undercloud.py to use Keystone v3 https://review.openstack.org/446752 | 11:03 |
| bkero | EmilienM: plenty of time to sleep later :) | 11:03 |
| *** karimb has quit IRC | 11:04 | |
| adarazs | heh, karimb took your advice :) | 11:05 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 11:05 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Bind redis-sentinel to its network https://review.openstack.org/446956 | 11:06 |
| bandini | jaosorior: ack, will look later | 11:06 |
| *** ansmith has joined #tripleo | 11:06 | |
| *** karimb has joined #tripleo | 11:06 | |
| jaosorior | bandini: thanks dude | 11:06 |
| bkero | adarazs: pretty sure it's exported though, must be something else | 11:07 |
| openstackgerrit | Martin André proposed openstack-infra/tripleo-ci master: Add undercloud-containers job to cistatus https://review.openstack.org/446957 | 11:07 |
| *** salmankhan has quit IRC | 11:08 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 11:10 |
| *** salmankhan has joined #tripleo | 11:10 | |
| *** athomas has joined #tripleo | 11:17 | |
| EmilienM | bkero, adarazs: so where are we now | 11:17 |
| bkero | EmilienM: We've determined the problem. sshnaidm|off has a review with a possible solution, and so do I. We're both waiting on CI to tell us if we've got it. | 11:18 |
| bkero | EmilienM: I posted my diagnosis on the launchpad bug. sshnaidm|off's solution is to re-run the repo-setup role to correct the repos after --multinode-setup installs the incorrect ones | 11:18 |
| bkero | My solution is to make --multinode-setup install the correct ones (ideally it wouldn't install them at all) | 11:19 |
| EmilienM | mhh I see ok | 11:20 |
| EmilienM | bkero: thanks for staying so late... | 11:20 |
| bkero | EmilienM: No problem. I accidentally had too much caffeine earlier in the day anyway. | 11:21 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 11:22 |
| * adarazs is on detox from caffeine, I think I overdid it in the last few weeks. | 11:22 | |
| bkero | Having a good espresso machine in the house is a dangerous thing. | 11:22 |
| bkero | My coffee:beer ratio is out of equilibrium | 11:23 |
| adarazs | heh :) | 11:23 |
| * adarazs goes and grabs some spicy indian food to counter the crave for coffee. | 11:24 | |
| *** adarazs is now known as adarazs_lunch | 11:24 | |
| EmilienM | bkero: I have a preference for what you're doing | 11:25 |
| EmilienM | iiuc what sshnaidm|off does is writting multiple times the repo (another time) | 11:25 |
| EmilienM | it sounds overkill | 11:25 |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: WIP oooq quickstart: Add STABLE_RELEASE env variable to subnodes https://review.openstack.org/446746 | 11:25 |
| *** jkilpatr has joined #tripleo | 11:26 | |
| bkero | I think it is, but it would get the gates working | 11:26 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Explicitly configure credentials used by ironic to access other services https://review.openstack.org/446037 | 11:27 |
| *** nyechiel has joined #tripleo | 11:27 | |
| EmilienM | yeah, let's see both | 11:27 |
| jaosorior | sshnaidm|off, bkero is there a thing such as the postci logs for the oooq jobs? | 11:28 |
| bkero | jaosorior: yes | 11:28 |
| jaosorior | they don't seem to print the full failures list | 11:28 |
| jaosorior | so debugging stuff is not as easy as it used to | 11:28 |
| bkero | It is there, although I can't tell you how complete it is | 11:28 |
| bkero | Likely more things will be in logs/undercloud/home/jenkins/*.log depending on where in the job it failed | 11:29 |
| jaosorior | ah | 11:29 |
| EmilienM | jaosorior: there is no postci file like before | 11:29 |
| jaosorior | bkero: that's what I'm talking about, it doesn't show the full failures list from the stack | 11:29 |
| EmilienM | jaosorior: you need to know where in oooq it failed (which task) and look at the right log in logs/undercloud/home/jenkins/*.log | 11:29 |
| EmilienM | but I think they're working on documenting where are useful logs on the front page | 11:30 |
| panda|zZ | jaosorior: EmilienM actually, there is | 11:30 |
| jaosorior | panda|zZ, bkero: where does the get_host_info end up? | 11:30 |
| jaosorior | that's the one that prints the full stack failures list | 11:30 |
| panda|zZ | jaosorior: in /var/log/extra | 11:30 |
| jaosorior | wow, what the hell | 11:31 |
| jaosorior | this error is pretty strange | 11:31 |
| EmilienM | panda|zZ: http://logs.openstack.org/77/445977/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/1b5f3ce/logs/undercloud/var/log/extra/ | 11:31 |
| EmilienM | where? | 11:31 |
| panda|zZ | jaosorior: the full stack failure is in /var/log/postci, but it will be moved in undercloud:/home/jenkins/failed_deployment | 11:31 |
| jaosorior | http://logs.openstack.org/91/444891/10/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/8934a8a/logs/undercloud/home/jenkins/overcloud_deploy.log.txt.gz | 11:31 |
| EmilienM | ah ok | 11:32 |
| EmilienM | http://logs.openstack.org/77/445977/2/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/1b5f3ce/logs/undercloud/var/log/postci.txt.gz | 11:32 |
| EmilienM | jaosorior: ^ | 11:33 |
| EmilienM | jaosorior: I took another patch but path should be the same | 11:33 |
| jaosorior | EmilienM: thanks; it seems it was an issue downloading packages X_x | 11:34 |
| EmilienM | bkero: why did you depends-on https://review.openstack.org/#/c/446934/4 also? | 11:35 |
| *** ratailor has quit IRC | 11:35 | |
| EmilienM | bkero: I'm confused why you're mixing the fix with repo and vxlan thing | 11:35 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: [ironic] expose default boot_option in configuration and change it to local https://review.openstack.org/446961 | 11:36 |
| *** salmankhan1 has joined #tripleo | 11:36 | |
| *** athomas has quit IRC | 11:36 | |
| *** salmankhan has quit IRC | 11:37 | |
| *** salmankhan1 is now known as salmankhan | 11:37 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: DO NOT MERGE: testing ironic default_boot_option https://review.openstack.org/446962 | 11:37 |
| bkero | EmilienM: the base tripleo-ci review just added some extra verbosity, which I wanted for insurance until I had the exact cause of the problem nailed down | 11:39 |
| EmilienM | bkero: I see | 11:39 |
| bkero | The review you mentioned is where I'm attempting to fix the extras role | 11:39 |
| dtantsur | folks (especially EmilienM I guess), any quick idea what to do about http://logs.openstack.org/03/351803/3/check/gate-tripleo-ci-centos-7-undercloud/ca254f7/logs/var/log/undercloud_install.txt.gz#_2017-03-08_16_58_00_000 ? | 11:45 |
| dtantsur | I'm trying to resurrect the change to switch ironic to wsgi... | 11:45 |
| jaosorior | dtantsur: gotta change the dependenci in puppet-ironic | 11:46 |
| *** saibarspeis has joined #tripleo | 11:47 | |
| jaosorior | dtantsur: something similar was done for heat and other services IIRC | 11:47 |
| dtantsur | link handy? | 11:47 |
| jaosorior | looking for it | 11:47 |
| EmilienM | dtantsur: looking | 11:48 |
| EmilienM | ah that | 11:48 |
| *** funzo has joined #tripleo | 11:49 | |
| EmilienM | yeah, there must be some orchestration in instack-undercloud that make it fail | 11:49 |
| jaosorior | EmilienM: is it in instack-undercloud? | 11:49 |
| EmilienM | probably but not 100% sure | 11:50 |
| EmilienM | gimme 5 min | 11:50 |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: WIP oooq quickstart: Add STABLE_RELEASE env variable to subnodes https://review.openstack.org/446746 | 11:51 |
| bkero | EmilienM: sshnaidm|off's review had successful tests: http://logs.openstack.org/51/446951/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/cad1033/ | 11:52 |
| *** suuuper has joined #tripleo | 11:52 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 11:53 |
| *** funzo has quit IRC | 11:53 | |
| *** abishop has joined #tripleo | 11:53 | |
| EmilienM | bkero: wow indeed, let me check the verisn | 11:54 |
| EmilienM | dtantsur: back in a min | 11:54 |
| *** jayg|g0n3 is now known as jayg | 11:54 | |
| EmilienM | puppet-tripleo-6.3.1-0.20170314022623.cf77c11.el7.centos.noarch | 11:54 |
| EmilienM | yeah we're good | 11:54 |
| EmilienM | I've +2 https://review.openstack.org/#/c/446949/ - let's wait for results but I suggest we go for this one until we clean things up | 11:55 |
| bkero | Sounds good to me | 11:55 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Explicitly configure credentials used by ironic to access other services https://review.openstack.org/446074 | 11:55 |
| bkero | I'll keep my patch stack to help me clean up the tripleo-quickstart-extras role that's misbehaving | 11:55 |
| jaosorior | dtantsur: what's the patch? | 11:55 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Keep existing data for containerized MongoDB https://review.openstack.org/442469 | 11:56 |
| dtantsur | jaosorior, https://review.openstack.org/351803 | 11:56 |
| EmilienM | Keystone_endpoint<||> -> Service['ironic-api'] | 11:56 |
| EmilienM | that's it | 11:56 |
| bkero | adarazs_lunch: I'd recommend we start logging deploy.env from toci | 11:56 |
| EmilienM | Keystone_endpoint depends on httpd, so we can't start ironic-api AFTER | 11:57 |
| EmilienM | dtantsur, jaosorior ^ | 11:57 |
| jaosorior | EmilienM: that is correct, but that's not a problem with this specific patch. that's an issue in puppet-ironic | 11:58 |
| *** zoli|wfh is now known as zoli|lunch | 11:58 | |
| *** b00tcat has joined #tripleo | 11:58 | |
| EmilienM | jaosorior: in manifests/keystone/auth.pp indeed | 11:58 |
| jaosorior | yep | 11:59 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Switch stackrc and undercloud.py to use Keystone v3 https://review.openstack.org/446752 | 11:59 |
| EmilienM | jaosorior: see https://review.openstack.org/#/c/351803/3/elements/puppet-stack-config/puppet-stack-config.pp | 11:59 |
| dtantsur | EmilienM, jaosorior, I see, thanks for investigation! should it just be dropped or is it more complex? | 11:59 |
| EmilienM | jaosorior: wow, thanks for your help on keystone v3 stuff | 12:00 |
| EmilienM | now I want to hug you again | 12:00 |
| * dtantsur tries removing | 12:00 | |
| jaosorior | dtantsur: just drop it, should be fine | 12:00 |
| jaosorior | EmilienM: no biggie | 12:01 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Revert "Revert "Deploy and Upgrade Ironic to run in mod_wsgi"" https://review.openstack.org/351803 | 12:01 |
| *** jaosorior is now known as jaosorior_brb | 12:01 | |
| *** salmankhan has quit IRC | 12:02 | |
| *** rhallisey has joined #tripleo | 12:03 | |
| *** rhallisey has quit IRC | 12:04 | |
| *** shardy is now known as shardy_lunch | 12:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-heat-templates master: Updated from global requirements https://review.openstack.org/440124 | 12:04 |
| *** ccamacho is now known as ccamacho|lunch | 12:04 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/ocata: panko: Do db_sync in api manifest https://review.openstack.org/444338 | 12:06 |
| EmilienM | this should bring back scenario001 on stable/ocata ^ with depends-on sshnaidm's patch on tripleo-ci | 12:06 |
| *** salmankhan has joined #tripleo | 12:07 | |
| bkero | EmilienM: !! fixed it properly ... almost. http://logs.openstack.org/65/446865/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/8cf644d/logs/undercloud/home/jenkins/toci_vxlan_networking.sh.log.txt.gz | 12:08 |
| bkero | It's installing the correct repos now, I think it wasn't picking up deploy.env due to a paths issue | 12:09 |
| sshnaidm|off | jaosorior_brb, now it's in undercloud/var/log/postci.txt http://logs.openstack.org/51/445951/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/f0461d6/logs/undercloud/var/log/postci.txt.gz | 12:09 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 12:10 |
| sshnaidm|off | bkero, I think we need to move code from tripleo.sh to your toci_network.. script | 12:11 |
| sshnaidm|off | bkero, it would be proper solution | 12:11 |
| bkero | sshnaidm|off: I agree. It was just a bit too hairy to do the for a first pass. | 12:12 |
| *** dprince has joined #tripleo | 12:12 | |
| EmilienM | sshnaidm|off: thanks for https://review.openstack.org/#/c/446949 and working on friday.. we're going to merge it | 12:13 |
| sshnaidm|off | bkero, I think my workaround works, so we can adopt it for now while preparing a good persistent solution | 12:13 |
| sshnaidm|off | EmilienM, np | 12:13 |
| EmilienM | sshnaidm|off: yes, good plan | 12:14 |
| EmilienM | just waiting for trown to be online and we'll approve it | 12:14 |
| *** morazi has joined #tripleo | 12:16 | |
| *** bkopilov has quit IRC | 12:16 | |
| bkero | meeting in 2h30m, going to catch a little sleep | 12:17 |
| *** jaganathan has quit IRC | 12:17 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 12:17 |
| openstackgerrit | Dmitry Tantsur proposed openstack/instack-undercloud master: Explicitly configure credentials used by ironic to access inspector and service catalog https://review.openstack.org/446981 | 12:18 |
| *** adarazs_lunch is now known as adarazs | 12:19 | |
| *** bfournie has quit IRC | 12:21 | |
| *** athomas has joined #tripleo | 12:23 | |
| *** saneax is now known as saneax-_-|AFK | 12:27 | |
| *** lblanchard has joined #tripleo | 12:31 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 12:34 |
| *** trown|outtypewww is now known as trown | 12:39 | |
| *** rlandy has joined #tripleo | 12:39 | |
| openstackgerrit | Merged openstack/instack-undercloud master: Request HAProxy certificate using certmonger_user manifest https://review.openstack.org/444885 | 12:40 |
| trown | thanks bkero sshnaidm|off you guys rock | 12:40 |
| openstackgerrit | Merged openstack/tripleo-ui stable/ocata: Imported Translations from Zanata https://review.openstack.org/446910 | 12:40 |
| *** dbecker has joined #tripleo | 12:47 | |
| *** limao has joined #tripleo | 12:48 | |
| *** thrash|g0ne is now known as thrash | 12:49 | |
| *** funzo_ has joined #tripleo | 12:49 | |
| *** bfournie has joined #tripleo | 12:52 | |
| *** funzo_ has quit IRC | 12:54 | |
| *** psahoo has quit IRC | 12:59 | |
| *** rhallisey has joined #tripleo | 13:01 | |
| *** zoli|lunch is now known as zoli|wfh | 13:07 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Parameterize OVB stack flavors https://review.openstack.org/445163 | 13:08 |
| *** masco has quit IRC | 13:08 | |
| *** karimb has quit IRC | 13:09 | |
| *** Goneri has joined #tripleo | 13:10 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 13:10 |
| *** dbecker has quit IRC | 13:10 | |
| *** nyechiel has quit IRC | 13:10 | |
| *** jpena is now known as jpena|lunch | 13:12 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Update ci-script to enable TripleO gate dependencies https://review.openstack.org/431764 | 13:12 |
| *** ccamacho|lunch is now known as ccamacho | 13:13 | |
| *** fultonj has joined #tripleo | 13:14 | |
| *** mat128 has quit IRC | 13:14 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Add option to pass a config file with full path to collect-logs.sh https://review.openstack.org/437711 | 13:15 |
| *** [1]cdearborn has joined #tripleo | 13:16 | |
| *** jcoufal has joined #tripleo | 13:17 | |
| *** karimb has joined #tripleo | 13:18 | |
| *** bkopilov has joined #tripleo | 13:18 | |
| *** tzumainn has joined #tripleo | 13:19 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add default image for Ocata deploys https://review.openstack.org/444942 | 13:20 |
| *** oaken_chris has joined #tripleo | 13:20 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 13:22 |
| *** rbrady-afk is now known as rbrady | 13:22 | |
| EmilienM | pradk: it sounds like https://review.openstack.org/#/c/444338/ will fail in CI: gate-tripleo-ci-centos-7-scenario001-multinode-upgrades failed :/ | 13:23 |
| *** salmankhan has quit IRC | 13:23 | |
| pradk | oh it does? | 13:24 |
| EmilienM | http://logs.openstack.org/38/444338/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-upgrades/ef8b063/logs/postci.txt.gz#_2017-03-17_13_17_44_000 | 13:24 |
| pradk | isnt this the change we discussed at the ptg with bandini | 13:24 |
| EmilienM | bandini: in my memory, you had some issues with upgrades | 13:24 |
| EmilienM | bandini: have you seen this log before? It's a newton deployment before the upgrade to ocata | 13:24 |
| *** limao_ has joined #tripleo | 13:27 | |
| pradk | EmilienM, i had a deps issue with https://review.openstack.org/#/c/430447/ | 13:27 |
| pradk | EmilienM, thats was because ocata is newer than pike | 13:28 |
| pradk | :( | 13:28 |
| *** ratailor has joined #tripleo | 13:28 | |
| pradk | may be its hitting the same | 13:28 |
| *** ealcaniz has quit IRC | 13:28 | |
| pradk | i did the sem ver trick upstream for ceilo to bump pike | 13:28 |
| *** limao has quit IRC | 13:29 | |
| * bandini looks | 13:29 | |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-docs master: Add warning about undercloud.conf configuration https://review.openstack.org/446997 | 13:30 |
| bandini | EmilienM: looking, sec | 13:30 |
| *** chlong has joined #tripleo | 13:32 | |
| EmilienM | pradk: no your stuff is on master | 13:32 |
| EmilienM | pradk: mine is on stable/ocata | 13:32 |
| pradk | ah ok | 13:33 |
| EmilienM | it sounds like a transient issue on my case | 13:33 |
| EmilienM | because scenario002 is upraded successfuly | 13:33 |
| pradk | iirc bandini had similkar issue where he was getting update to pike at some point | 13:33 |
| pradk | thought it was ocata patch | 13:34 |
| EmilienM | bandini: are you aware about some race conditions where /usr/bin/clustercheck and mysql fails at deployment in newton? | 13:34 |
| *** bkopilov has quit IRC | 13:34 | |
| EmilienM | because if yes, we're hitting it in the upgrade jobs in stable/ocata | 13:34 |
| bandini | so I had ocata/stable upgrade issues here https://bugs.launchpad.net/tripleo/+bug/1673030 but a fix was committed for that | 13:36 |
| openstack | Launchpad bug 1673030 in tripleo "ocata/stable upgrade jobs are broken (httpd restart failures)" [Critical,Fix committed] - Assigned to Michele Baldessari (michele) | 13:36 |
| EmilienM | ok not related | 13:36 |
| bandini | let me see your failure more in detail | 13:36 |
| EmilienM | mhh looking at logstash it looks like a valid issue on newton | 13:37 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Add option to pass a config file with full path to collect-logs.sh https://review.openstack.org/437711 | 13:37 |
| EmilienM | we frequently hit that in gate-tripleo-ci-centos-7-ovb-ha-newton job | 13:37 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Allow libvirt starting to fail for setup https://review.openstack.org/447000 | 13:38 |
| *** Rodolof has joined #tripleo | 13:38 | |
| EmilienM | if someone wants to see the query: http://logstash.openstack.org/#dashboard/file/logstash.json?query=build_name%3A%20*tripleo-ci*%20AND%20build_status%3A%20FAILURE%20AND%20message%3A%20%5C%22Error%3A%20resources.AllNodesDeploySteps.resources.ControllerDeployment_Step2.resources%5B0%5D%5C%22 | 13:39 |
| *** Rodolof has left #tripleo | 13:39 | |
| *** weshay_pto is now known as weshay_sick | 13:40 | |
| EmilienM | it started for real on 9th march | 13:40 |
| *** pkovar has joined #tripleo | 13:42 | |
| *** ratailor has quit IRC | 13:43 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 13:44 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add option to build delorean changes and include the repo https://review.openstack.org/434004 | 13:44 |
| *** nyechiel has joined #tripleo | 13:45 | |
| bandini | EmilienM: can we confirm this happens only on newton? (just to focus my research) | 13:46 |
| EmilienM | bandini: logstash query says it happens in all branches but I need to make sure it's the same error, give me 2 min | 13:48 |
| *** bkopilov has joined #tripleo | 13:48 | |
| *** gbarros has joined #tripleo | 13:48 | |
| bandini | ack | 13:48 |
| EmilienM | bandini: yeah we see it here: https://review.openstack.org/#/c/441988/ | 13:49 |
| EmilienM | it's in pike too | 13:49 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/puppet-tripleo master: Remove certificate request bits from service profiles https://review.openstack.org/444891 | 13:49 |
| bandini | ack | 13:49 |
| EmilienM | http://logs.openstack.org/88/441988/2/check-tripleo/gate-tripleo-ci-centos-7-ovb-ha/2a2aec8/logs/postci.txt#_2017-03-13_10_34_08_000 | 13:49 |
| EmilienM | jaosorior_brb: quick heads-up on our keystonev3 work. We'll need to do the same thing for tripleoclient and remove overcloud.rc3 to have a single overcloudrc that use versionless endpoint; wdyt? | 13:50 |
| *** jaosorior_brb is now known as jaosorior | 13:50 | |
| *** funzo_ has joined #tripleo | 13:50 | |
| jaosorior | EmilienM: that would be good. | 13:50 |
| openstackgerrit | John Trowbridge proposed openstack/puppet-tripleo master: DNM testing scenario gates https://review.openstack.org/445977 | 13:52 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Switch stackrc and undercloud.py to use Keystone v3 https://review.openstack.org/446752 | 13:52 |
| *** ratailor has joined #tripleo | 13:53 | |
| *** nyechiel has quit IRC | 13:54 | |
| *** funzo_ has quit IRC | 13:55 | |
| *** shardy_lunch is now known as shardy | 13:56 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/ocata: panko: Do db_sync in api manifest https://review.openstack.org/444338 | 13:56 |
| EmilienM | pradk: trying to save time, pushed a commit message change to kick off jobs again and hope it works this time | 13:56 |
| EmilienM | pradk: I put the 2 failures in comment but both are transient I think | 13:56 |
| *** adarazs is now known as adarazs_afk | 13:57 | |
| bandini | I think I know what the issue is | 13:57 |
| bandini | let me have a coffe and think about it a bit more | 13:57 |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-docs master: Document scale introspection issues https://review.openstack.org/447008 | 13:58 |
| sai | ping tripleo, seeing An unexpected error prevented the server from fulfilling your request. (HTTP 500) (Request-ID: req-94055d70-775e-4662-9348-f8f4cb41ba51) as soon as fire the command for scaling up stack..no idea where i should be looking | 13:59 |
| sai | advice appreciated | 13:59 |
| * EmilienM sends all coffee he can to bandini | 13:59 | |
| shardy | sai: I'd look in the heat-engine log for a backtrace, or a clue as to which service 500'd | 14:00 |
| sai | shardy: ack, thanks | 14:04 |
| *** salmankhan has joined #tripleo | 14:04 | |
| sai | restarted nova and neutron on uc..seems to be going further | 14:05 |
| bandini | EmilienM: lol | 14:07 |
| bandini | EmilienM, pradk: I am going to file a bug this is a race in how we do a couple of things | 14:08 |
| EmilienM | bandini: take your time to document it | 14:08 |
| *** jpena|lunch is now known as jpena | 14:08 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: [overcloud-deploy] Fix hosts file generation https://review.openstack.org/438625 | 14:09 |
| *** brault_ is now known as brault|away | 14:09 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 14:10 |
| d0ugal | EmilienM: Hey | 14:10 |
| EmilienM | d0ugal: o/ | 14:10 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Re-Add bigswitch agent support https://review.openstack.org/446029 | 14:10 |
| d0ugal | EmilienM: I want to standardise the message we send over Zaqar from workflows. Where/how do you think I should do that? | 14:10 |
| d0ugal | EmilienM: They are mostly the same at the moment, but there is some variance - getting them all on the same page should make it easier for consumers. | 14:11 |
| *** ccamacho is now known as ccamacho|afk | 14:11 | |
| d0ugal | EmilienM: I was thinking of writing a spec, but as there are no real work items, I wasn't sure it made sense. | 14:11 |
| EmilienM | d0ugal: you want a common format of send_message in a task in a workflow? | 14:14 |
| d0ugal | EmilienM: essentially, yeah | 14:14 |
| d0ugal | EmilienM: and I want to establish a pattern for sending multiple messages - so consumers know when a message is the last vs just status updates. | 14:14 |
| jaosorior | mandre: hey dude, if you have a chance, can you help me out checking what happened with this commit? https://review.openstack.org/#/c/446911/ | 14:15 |
| EmilienM | d0ugal: it would be cool to write a common pattern stored in Mistral and being able to re-use it in the workflow tasks | 14:15 |
| *** brault|away is now known as brault | 14:16 | |
| mandre | jaosorior: the ubuntu-src gate is broken because of https://bugs.launchpad.net/kolla/+bug/1673776, reason for the -1 | 14:16 |
| openstack | Launchpad bug 1673776 in kolla "Bifrost build fails on Ubuntu from source" [Undecided,New] - Assigned to Mohammed Naser (mnaser) | 14:16 |
| mandre | jaosorior: let me see what's going on with the other failing jobs | 14:17 |
| sai | shardy: i cancelled update of overcloud stack(ctrl+c) and now when i try to update it fails as stack is in update in progress, anyway to workaround this?? | 14:19 |
| shardy | sai: ctrl-c doesn't cancel the update, it just stops showing the events | 14:20 |
| *** dparkes has quit IRC | 14:21 | |
| sai | shardy: got it, thanks | 14:21 |
| shardy | sai: there is an openstack stack cancel update command you can use to force cancel the update | 14:22 |
| shardy | also restarting heat-engine on the undercloud will do it | 14:22 |
| *** prateek has quit IRC | 14:22 | |
| shardy | but I'd suggest only doing that if you're sure things are broken, it's better to wait for the update to complete/fail | 14:22 |
| sai | shardy: heat-stack-cancel-update that put things in a real bad state last time i tried (ROLLBACK_FAILED) | 14:22 |
| *** florianf has quit IRC | 14:23 | |
| shardy | sai: Ok, sounds like we need a heat option that just forces the current operation to stop instead of attempting to roll back | 14:23 |
| *** chlong has quit IRC | 14:23 | |
| shardy | restarting heat-engine does do that, as a last resort | 14:24 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 14:24 |
| sai | shardy: yes, it would be good to give the user a way to do this via a heat command than restarting, just looks like a better way | 14:24 |
| shardy | sai: yeah I agree | 14:24 |
| sai | shardy: i will write up an rfe | 14:24 |
| shardy | sai: +1, sounds good, thanks | 14:25 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Fix repositories setup https://review.openstack.org/446949 | 14:26 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: docker: Add metadata_settings to optional parameters for yaml validate https://review.openstack.org/446593 | 14:26 |
| *** panda|zZ has quit IRC | 14:28 | |
| sai | shardy: https://bugs.launchpad.net/heat/+bug/1673794 | 14:28 |
| openstack | Launchpad bug 1673794 in heat "[RFE]: Give user a way to cancel stack update through a heat command" [Undecided,New] | 14:28 |
| EmilienM | d0ugal: does it make sense what I said? | 14:29 |
| d0ugal | EmilienM: Yeah, I think so - but I think it would be a bit specific to add it to Mistral itself. | 14:29 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Containerize panko api service https://review.openstack.org/439138 | 14:30 |
| d0ugal | EmilienM: maybe I am over thinking it, but I thought it might be good to agree and document the zaqar message structure we use somewhere | 14:30 |
| EmilienM | d0ugal: maybe a plugin to specify some patterns that we can re-use and use this plugin in tripleo-common workflows | 14:30 |
| EmilienM | d0ugal: where we would define the common pattern for zaqar meessages | 14:30 |
| d0ugal | EmilienM: yeah, that is a cool idea. | 14:30 |
| EmilienM | therve: fyi ^ | 14:30 |
| d0ugal | mistral-zaqar-workflows or something | 14:30 |
| d0ugal | rbrady, toure: fyi too ^ | 14:31 |
| d0ugal | EmilienM: thanks, I think that probably is a better direction. | 14:32 |
| EmilienM | cool | 14:32 |
| *** panda has joined #tripleo | 14:34 | |
| *** amoralej is now known as amoralej|lunch | 14:34 | |
| therve | EmilienM, There are some format that appears from time to time :) | 14:36 |
| * therve tries to remember last time we discussed it | 14:36 | |
| *** limao_ has quit IRC | 14:36 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: [gate-quickstart] Fix some wild typos https://review.openstack.org/445431 | 14:36 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Update accessing overcloud documentation https://review.openstack.org/438965 | 14:37 |
| EmilienM | therve: fyi https://etherpad.openstack.org/p/tripleo-etcd-transition - it might interest you | 14:37 |
| *** pkovar has quit IRC | 14:37 | |
| *** florianf has joined #tripleo | 14:38 | |
| *** gbarros has quit IRC | 14:38 | |
| therve | d0ugal, EmilienM http://lists.openstack.org/pipermail/openstack-dev/2016-May/095575.html | 14:38 |
| therve | There is some message from me in the thread | 14:39 |
| EmilienM | oh nice | 14:39 |
| EmilienM | even from d0ugal \o/ | 14:39 |
| therve | EmilienM, Have you read about confd? | 14:39 |
| d0ugal | haha, wow, I forgot about that thread | 14:39 |
| *** pkovar has joined #tripleo | 14:40 | |
| d0ugal | so it seems jtomasek|afk is about 9 months ahead of me :-) | 14:40 |
| EmilienM | therve: in the thread you mean? | 14:40 |
| therve | EmilienM, Yeah, but also in general. I see that you mention "other services (Apache, etc) still need config files." | 14:41 |
| pradk | dprince, mandre, can we get this in https://review.openstack.org/#/c/439138/ | 14:41 |
| EmilienM | therve: it seems like we could use a combination of confd (for services like apache, etc) and etcd (for openstack and all other services from confd)? | 14:42 |
| EmilienM | is that correct? | 14:42 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-quickstart-extras master: Fix the regexp extracting the number of controllers https://review.openstack.org/447017 | 14:42 |
| therve | EmilienM, Well. Or use confd for everything :) | 14:42 |
| *** Vijayendra has quit IRC | 14:42 | |
| EmilienM | therve: confd can use etcd for scalable storage backend, right? | 14:42 |
| therve | EmilienM, Yes | 14:43 |
| EmilienM | because afik, confd doesn't handle storage at scale | 14:43 |
| therve | I don't think confd handle storage at all | 14:43 |
| EmilienM | right | 14:43 |
| EmilienM | therve: can you add this thought on https://etherpad.openstack.org/p/tripleo-etcd-transition maybe? it would be a good addition, specially where we mention apache, etc | 14:43 |
| *** tosky has quit IRC | 14:43 | |
| EmilienM | therve: I added something L41, feel free to add more | 14:44 |
| *** tosky has joined #tripleo | 14:44 | |
| openstackgerrit | Merged openstack/python-tripleoclient master: Handle multiple messages from workflow executions https://review.openstack.org/406178 | 14:46 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Make sure PrePuppet runs before any Deployment_Step https://review.openstack.org/447020 | 14:46 |
| bandini | EmilienM, pradk: ^ | 14:47 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 14:47 |
| EmilienM | oh wow | 14:47 |
| therve | EmilienM, I added something l16 worth of note... | 14:47 |
| EmilienM | bandini: this one sounds critical and backportable until newton | 14:47 |
| EmilienM | therve: merci | 14:48 |
| bandini | EmilienM: yep. we broke it after mitaka it seems | 14:48 |
| *** adarazs_afk is now known as adarazs | 14:48 | |
| *** trown is now known as trown|brb | 14:49 | |
| EmilienM | therve: etcd2 would work with eventlet? | 14:49 |
| *** gbarros has joined #tripleo | 14:49 | |
| therve | EmilienM, Yeah | 14:49 |
| therve | EmilienM, Even with WSGI, plenty of stuff uses eventlet. nova conductor/compute, heat engine, etc | 14:50 |
| EmilienM | therve: mhh, interesting. Do you know if there is WIP to support eventlet in etcd3 ? | 14:50 |
| therve | EmilienM, It's an issue with grpc. I haven't found a real answer for now. | 14:50 |
| EmilienM | bandini: really nice catch on this one | 14:51 |
| *** funzo_ has joined #tripleo | 14:51 | |
| EmilienM | therve: i'm not sure how bad would it be to use etcv2 for now and migrate one day | 14:51 |
| EmilienM | I've seen migration paths but I never tried it | 14:51 |
| therve | EmilienM, Yeah I don't know. | 14:51 |
| *** trown|brb is now known as trown | 14:52 | |
| EmilienM | we need to make sure that etc2 fits with our needs in term of features and security | 14:52 |
| bandini | EmilienM: jistr clued me in for the most part :) | 14:52 |
| dprince | pradk: yes, I think it looks good. Will review one more time | 14:52 |
| therve | EmilienM, It kinds of suck to be stucked with the legacy version before even introducing the feature | 14:53 |
| EmilienM | therve: if you have time, it might be worth commenting on [openstack-dev] [oslo][devstack][all] ZooKeeper vs etcd for Tooz/DLM | 14:53 |
| EmilienM | therve: and mention about this limitation... | 14:53 |
| EmilienM | therve: etcd3 is considered as stable? | 14:53 |
| therve | EmilienM, Yeah I chatted with jd and jay already | 14:53 |
| EmilienM | right, 3.0.0 is stable since June 30, 2016 | 14:54 |
| *** dprince has quit IRC | 14:54 | |
| dmsimard | EmilienM: can we merge this ? https://review.openstack.org/#/c/433206/ | 14:54 |
| dmsimard | It's the deprecation notice you pointed out the other day | 14:54 |
| EmilienM | dmsimard: thx! | 14:55 |
| *** funzo_ has quit IRC | 14:55 | |
| *** panda_ has joined #tripleo | 14:57 | |
| EmilienM | therve: I'm wondering if we could have an abstraction (but it starts to be complex to have abstractions of abstractions) | 14:58 |
| *** marst has quit IRC | 14:58 | |
| jd_ | EmilienM: etcdv2 is partially there for a couple of years and nobody ever used it so | 14:58 |
| therve | EmilienM, Yeah I don't think it's worth it. If we need one, let's use confd. | 14:59 |
| *** panda is now known as Guest88783 | 14:59 | |
| *** panda_ is now known as panda | 14:59 | |
| *** trown is now known as trown|mtg | 15:00 | |
| EmilienM | therve, jd_ : so an option would be to write a confd backend for oslo.config and configure confd to use etcdv3 backend? | 15:00 |
| therve | EmilienM, I think you need a confd template per service | 15:01 |
| *** Guest88783 has quit IRC | 15:01 | |
| remix_tj | to override a resource defined in overcloud-resource-registry-puppet.yaml (in my case OS::TripleO::Services::SaharaApi and OS::TripleO::Services::SaharaEngine) is enough to create an environment file containing a resource_registry with a different definition? | 15:01 |
| sai | shardy: zaneb on deleting the overcloud stack i routinely see one or two neutron ports left behind on the undercloud and it looks likt that is because they aren't bound to a host when the delete is attempted..is this expected behaviour? | 15:02 |
| EmilienM | therve: I'm not familiar enough with confd but I'll read and try it this weekend. | 15:02 |
| zaneb | sai: I wouldn't say expected, no | 15:02 |
| shardy | remix_tj: yes generally it is | 15:03 |
| therve | EmilienM, Yeah it's relatively straightforward. You have a config file like this: https://github.com/kelseyhightower/confd/blob/master/docs/template-resources.md#example which is "static". And then a template which tells confd where to put key/values | 15:03 |
| *** marst has joined #tripleo | 15:04 | |
| zaneb | sai: but Nova's interaction with Neutron is an ongoing tyre fire, so not _that_ surprising either | 15:04 |
| remix_tj | shardy: so any definition in environment files overrides what defined in the yaml file? | 15:04 |
| therve | EmilienM, Like https://github.com/kelseyhightower/confd/blob/master/docs/templates.md#example-usage | 15:04 |
| sai | zaneb: got it | 15:05 |
| *** marst has quit IRC | 15:05 | |
| shardy | remix_tj: yes, the overcloud-resource-registry.yaml is included by default, and any -e one.yaml -e two.yaml additional files update the default environment | 15:05 |
| dmsimard | therve, EmilienM: kind of like jinja :P | 15:05 |
| therve | dmsimard, Yeah... | 15:05 |
| shardy | remix_tj: the last -e option wins in case of conflicting definitions in one/two.yaml | 15:05 |
| remix_tj | shardy: good, thanks | 15:05 |
| remix_tj | perfect | 15:05 |
| * jd_ does not confd either | 15:05 | |
| *** saibarspeis has quit IRC | 15:06 | |
| EmilienM | therve: templates are bad, but that's already how we configure Apache, HAproxy with Puppet | 15:06 |
| EmilienM | as long as we don't do templates with OpenStack services, i'm ok | 15:06 |
| * therve shrugs | 15:07 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673585 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1673585 in tripleo "quickstart: ocata/newton overclouds are deployed with pike repos" [Critical,Triaged] | 15:10 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 15:11 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 15:12 |
| *** dprince has joined #tripleo | 15:12 | |
| *** amoralej|lunch is now known as amoralej | 15:13 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/tripleo-ci master: Do not merge: Test python-heat-agents package build https://review.openstack.org/439178 | 15:16 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/tripleo-ci master: Do not merge: Test python-heat-agents package build https://review.openstack.org/439178 | 15:16 |
| openstackgerrit | David Moreau Simard proposed openstack/tripleo-heat-templates master: Do not merge: Test python-heat-agents package build https://review.openstack.org/439179 | 15:17 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Create the neutron-sanity-check validations https://review.openstack.org/381118 | 15:17 |
| openstackgerrit | Jiri Stransky proposed openstack/tripleo-heat-templates master: Keep existing data for containerized ironic-conductor https://review.openstack.org/443173 | 15:18 |
| *** pkovar has quit IRC | 15:19 | |
| *** pkovar has joined #tripleo | 15:19 | |
| EmilienM | I'm removing the alert | 15:21 |
| jkilpatr | is there an OVB irc channel? I can't make heads or tails of why ovb setup insists on making it's own public network but at the same time requires an existing one or it exists with an error message | 15:22 |
| *** yprokule has quit IRC | 15:22 | |
| EmilienM | jkilpatr: there is no channel for OVB. You might want to use this one | 15:23 |
| jkilpatr | EmilienM, cool thanks. Trying to get OVB deployed in my baremetal CI so that I can automate baremetal cloud -> ovb cloud -> larger scale tests, but I can' | 15:24 |
| jkilpatr | can't seem to figure out the networks. | 15:24 |
| jistr | gfidente, dprince: could either of you please review the diff since last patchset here? (7 lines :) ) https://review.openstack.org/#/c/443112/3..4 | 15:26 |
| *** ipsecguy_ is now known as ipsecguy | 15:27 | |
| *** dsariel has joined #tripleo | 15:28 | |
| EmilienM | if tht cores can review https://review.openstack.org/#/c/447020 before the weekend please, we'll also backport it into ocata & newton asap | 15:28 |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Fix ansible-lint.sh script for roles and playbooks https://review.openstack.org/431456 | 15:29 |
| *** dciabrin is now known as dciabrin|away | 15:29 | |
| openstackgerrit | Merged openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/446907 | 15:29 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Fix setfiles command https://review.openstack.org/447033 | 15:30 |
| bkero | Wow, tripleo.sh REALLY does not want to be sourced. | 15:32 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add option not to skip building images https://review.openstack.org/443085 | 15:32 |
| panda | bkero: what did it say ? | 15:33 |
| *** trown|mtg is now known as trown | 15:34 | |
| *** jcoufal has quit IRC | 15:34 | |
| bkero | panda: http://logs.openstack.org/65/446865/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/e723423/logs/undercloud/home/jenkins/toci_vxlan_networking.sh.log.txt.gz | 15:34 |
| bkero | panda: git clone it and try sourcing it. :) It'll just close your terminal emulator | 15:34 |
| bnemec | jkilpatr: There are sort of two levels of networking in OVB - the host networks and the overcloud networks. | 15:35 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 15:36 |
| bnemec | jkilpatr: There are two host networks: private and external | 15:36 |
| bnemec | The overcloud networks are the TripleO networks, provision, public, internal-api, etc. | 15:36 |
| jkilpatr | yup, but when I run the ovb create command it makes networks public_net_name-ovb_id for public and provision | 15:37 |
| jkilpatr | but not private | 15:37 |
| jkilpatr | which has me confused | 15:37 |
| bnemec | jkilpatr: Yeah, private is supposed to be configured by the host cloud operator. | 15:38 |
| jkilpatr | especially since if I don't have a network already existing as public_net_name heat stack fails to create, even though it's making a different public net anyways. | 15:38 |
| bnemec | jkilpatr: Although I believe there is a way to have OVB create that and hook it up to the external network. Let me find it. | 15:38 |
| jkilpatr | bnemec, I don't really care if I make it or if ovb makes it, I'm just confused as to how it seems to want both. | 15:38 |
| *** gchamoul is now known as gchamoul|afk | 15:38 | |
| weshay_sick | rlandy, do you want to chat about the security issues re: rdo-cloud? | 15:39 |
| rlandy | weshay_sick: sure | 15:41 |
| *** mcornea has quit IRC | 15:41 | |
| bnemec | jkilpatr: You can use https://github.com/cybertron/openstack-virtual-baremetal/blob/master/templates/env.yaml.example#L37 to have OVB create the private network. | 15:41 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: Add timestamps to quickstart logs https://review.openstack.org/446075 | 15:41 |
| * adarazs off | 15:41 | |
| jkilpatr | bnemec, no problem on the private network front, I'm having trouble with the public net, like I said if I don't make a network named 'public' before starting the stack create it fails, but the stack itself makes a network named public-1 then seems to try and attach a port from public to public-1 or somthing equally stupid. | 15:42 |
| bnemec | jkilpatr: Let me see if I can pull a useful networking diagram from my current deployment. That might help explain the difference. | 15:42 |
| *** ratailor has quit IRC | 15:48 | |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: create history for the overcloud deploy command https://review.openstack.org/446617 | 15:49 |
| bnemec | jkilpatr: Here's the full networking diagram: https://fedorapeople.org/~bnemec/ovb-net-iso.png | 15:51 |
| *** funzo_ has joined #tripleo | 15:52 | |
| bnemec | All of the -test networks are non-routable OVB networks. | 15:52 |
| bnemec | external is the network where floating ips live. Note that it is _not_ used by the virtual overcloud deployment. It's just to provide access to the undercloud. | 15:53 |
| jaosorior | EmilienM: so, regarding the instack-undercloud patch | 15:53 |
| jaosorior | EmilienM: we need either promotion to happen, or to get the latest mistralclient | 15:53 |
| jaosorior | EmilienM: I had fixed the keystone issue there, but we still have the version of mistralclient without the fix. | 15:53 |
| EmilienM | nice | 15:54 |
| EmilienM | it can wait a promotion | 15:54 |
| EmilienM | AFIK the client needs a bump in rdoinfo | 15:54 |
| bnemec | jkilpatr: BTW, how are you deploying? I would strongly suggest http://openstack-virtual-baremetal.readthedocs.io/en/latest/deploy/quintupleo.html if you're not already using it. | 15:54 |
| EmilienM | which is automatic when the release is done and pushed in openstack/requirements | 15:54 |
| EmilienM | jaosorior: I'm on PTO from now, let's catchup later | 15:54 |
| jkilpatr | bnemec, that's exactly what I'm using could I see your invocation? | 15:54 |
| jaosorior | EmilienM: sure man, enjoy | 15:54 |
| EmilienM | jaosorior: but i'll check later how to bump mistralclient | 15:55 |
| bnemec | jkilpatr: This was deployed with: bin/deploy.py --quintupleo --name test --id test --poll | 15:55 |
| EmilienM | jaosorior: we need 1. a release in mistralclient with the fix 2. a push in openstack/requirements 3. the (automatic) patch in rdoinfo that bumps it and 4. a promotion | 15:55 |
| bnemec | Let me grab the env file I used. | 15:55 |
| jaosorior | EmilienM: it's no hurry man, no worries. Enjoy your PTO and have a good one :D | 15:55 |
| EmilienM | thanks | 15:55 |
| bnemec | jkilpatr: http://paste.openstack.org/show/603157/ | 15:56 |
| jkilpatr | bnemec, does it load a config file by default when yo udo that I have --env pointing to a config file? | 15:56 |
| *** funzo_ has quit IRC | 15:57 | |
| bnemec | jkilpatr: Yeah, it defaults to "env.yaml" in the current directory. | 15:57 |
| jkilpatr | bnemec, so it expects the private and public net to be ready when you run that but it will create provision and hte others/ | 15:57 |
| *** dsariel has quit IRC | 15:57 | |
| *** dciabrin|away is now known as dciabrin | 15:57 | |
| bnemec | jkilpatr: private and external should exist. public and provision are created by the templates. | 15:57 |
| jkilpatr | bnemec, whats the difference between external and public? | 15:58 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 15:58 |
| bnemec | jkilpatr: external is the actual host cloud external network. public is a fake "public" network used by the virtual overcloud. | 15:59 |
| jkilpatr | ok so I was getting those mixed up | 15:59 |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK https://review.openstack.org/430277 | 16:00 |
| jkilpatr | bnemec, thanks a lot, I'm going to try and apply this | 16:00 |
| bnemec | Yeah, this is the downside of a cloud within a cloud. There's some overlap in the names. | 16:00 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: [ironic] expose default boot_option in configuration and change it to local https://review.openstack.org/446961 | 16:00 |
| bnemec | public_net and provision_net should probably have been named overcloud_public_net and overcloud_provision_net, but for historical reasons they weren't. | 16:00 |
| bnemec | And it's hard to rename Heat template parameters right now. | 16:00 |
| bnemec | At least without breaking existing users. | 16:01 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: DO NOT MERGE: testing ironic default_boot_option https://review.openstack.org/446962 | 16:01 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add dracut-regenerate elements https://review.openstack.org/447047 | 16:04 |
| *** thrash is now known as thrash|biab | 16:04 | |
| mwhahaha | so with the new oooq jobs, where did the deployment command logs go | 16:04 |
| bkero | mwhahaha: look in logs/undercloud/home/jenkins/ | 16:05 |
| mwhahaha | do we not capture the failures anymore? | 16:05 |
| mwhahaha | as much as i hated trying to read postci.log at least it had that :/ | 16:06 |
| *** abehl has quit IRC | 16:06 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 16:06 |
| panda | mwhahaha: the deployment is in /home/jenkins/overcloud_deploy.sh.txt.gz, deployment log is in /home/jenkins/overcloud_deploy.log.txt.gz, failures are in /var/log/postci.txt.gz, but a patch is in place to add /home/jenkins/failed_deployment.log.txt.gz | 16:06 |
| mwhahaha | i don't suppose we could have an html page generated witht hese links to things | 16:07 |
| panda | sorry, not add, activate, it was broken and needed to be fixed. | 16:07 |
| *** pkovar1 has joined #tripleo | 16:07 | |
| mwhahaha | cause i don't really have time to hunt around these directories to find anything | 16:07 |
| *** paramite has quit IRC | 16:07 | |
| mwhahaha | failed_deployment.log.txt.gz doesn't exist btw | 16:08 |
| panda | mwhahaha: is in the queue, but will not probably happen before next week | 16:08 |
| mwhahaha | http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a6432ed/logs/undercloud/home/jenkins/ | 16:08 |
| *** psahoo has joined #tripleo | 16:08 | |
| panda | mwhahaha: yes, this patch https://review.openstack.org/446042 needs to land first | 16:08 |
| bkero | mwhahaha: There is an ARA report in there as well. That is a generated HTML page. | 16:09 |
| *** pkovar has quit IRC | 16:09 | |
| mwhahaha | ara didn't show any failures | 16:09 |
| *** numans has quit IRC | 16:09 | |
| mwhahaha | no i want the logs (i don't actually care about ara at teh moment) | 16:09 |
| panda | mwhahaha: for now we're keeping http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a6432ed/logs/undercloud/var/log/postci.txt.gz | 16:13 |
| mwhahaha | yea postci used to be under logs/ | 16:13 |
| mwhahaha | which is why i said having an html page with the new links would be beneficial | 16:13 |
| mwhahaha | since they are now burried 3 levels deep | 16:13 |
| *** numans has joined #tripleo | 16:13 | |
| *** rbrady is now known as rbrady-food | 16:13 | |
| panda | mwhahaha: yes, we are working on a README + TOC before starting phase2 | 16:14 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 16:14 |
| mwhahaha | might want to jsut send a note to the ML at the minimum about the new file locations | 16:14 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 16:15 |
| panda | good idea, this is not the first time someone asked for it, and it will not be the last | 16:15 |
| mwhahaha | bandini: https://bugs.launchpad.net/tripleo/+bug/1673030 i just commented but i'm still seeing it | 16:17 |
| openstack | Launchpad bug 1673030 in tripleo "ocata/stable upgrade jobs are broken (httpd restart failures)" [Critical,Fix committed] - Assigned to Michele Baldessari (michele) | 16:17 |
| bandini | mwhahaha: oh crap. let me take a look | 16:18 |
| bandini | mwhahaha: do you have a link for a broken job? | 16:18 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Modify PreNetwork host config inline with role-specific parameters https://review.openstack.org/447051 | 16:19 |
| mwhahaha | bandini: http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a6432ed/logs/undercloud/var/log/postci.txt.gz | 16:19 |
| *** jcoufal has joined #tripleo | 16:19 | |
| bandini | looking | 16:19 |
| mwhahaha | bandini: change 446011 to fix instack-undercloud | 16:19 |
| *** ckyriakidou has quit IRC | 16:20 | |
| mandre | jaosorior: you may find https://review.openstack.org/#/c/445690/ interesting | 16:21 |
| *** Vijayendra has joined #tripleo | 16:21 | |
| jaosorior | mandre: I actually do. That's useful! | 16:21 |
| bandini | mwhahaha: I see. so the bug I was seeing (and the review that fixed it) was only related to upgrades. I had never seen it undercloud previously | 16:21 |
| bandini | but iirc we had some bugs open about pulling from wrong repos, lemme see | 16:22 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Modify PreNetwork host config inline with role-specific parameters https://review.openstack.org/447051 | 16:22 |
| mwhahaha | bandini: it's not on the undercloud | 16:23 |
| mwhahaha | bandini: it was from the heat deploy, it looks like heat-api was running in daemon mode and http can't bind to 8000 | 16:23 |
| *** bnemec is now known as beekneemech | 16:23 | |
| bandini | mwhahaha: sorry yeah I meant not an upgrade | 16:23 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Switch stackrc and undercloud.py to use Keystone v3 https://review.openstack.org/446752 | 16:23 |
| mwhahaha | bandini: ah yea, i wonder if we missed a THT backport or something when we enabeld heat via httpd | 16:24 |
| mwhahaha | jaosorior: -^ | 16:24 |
| bandini | mwhahaha: but heat via httpd is a pike-only thing so we should not see this at all on stable/ocata things | 16:24 |
| bandini | mwhahaha: that is why I think we're pulling in some too new puppet-tripleo maybe? | 16:24 |
| bandini | that was the case for the upgrade issue at least | 16:25 |
| mwhahaha | bandini: possibly,i thought that to be fixed | 16:25 |
| weshay_sick | panda, mwhahaha that is my bad I was supposed to have that done yesterday | 16:25 |
| weshay_sick | re: the readme | 16:25 |
| bandini | mwhahaha: so did I tbh | 16:25 |
| *** egafford has joined #tripleo | 16:26 | |
| *** thrash|biab is now known as thrash | 16:26 | |
| bandini | mwhahaha: I thought https://review.openstack.org/#/c/446949/ fixed it | 16:26 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: create history for the overcloud deploy command https://review.openstack.org/446617 | 16:26 |
| mwhahaha | i wonder if it's the image | 16:26 |
| mwhahaha | i wonder if it's not pulling the stable/ocata overcloud image | 16:26 |
| * mwhahaha shrugs | 16:26 | |
| mwhahaha | you people and your newfangled ci frameworks | 16:27 |
| bandini | yeah, I have been trying for the whole week to land two stable/ocata patches and still have not managed | 16:27 |
| panda | we had problems with the stable branches. They should have beens solved now, or are very close to be solved | 16:28 |
| mwhahaha | panda: well we're still having problems | 16:28 |
| mwhahaha | panda: where's teh log to see what images were used | 16:29 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: create history for the overcloud deploy command https://review.openstack.org/446617 | 16:30 |
| panda | this should have fixed the stable branches https://review.openstack.org/446949 | 16:30 |
| panda | mwhahaha: a sec | 16:30 |
| mwhahaha | panda: repos != overcloud image | 16:30 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Change image links on ci.centos to use rdo_trunk path https://review.openstack.org/447056 | 16:33 |
| mwhahaha | anyway i've got to run, i switch Bug 1673030 back to confirmed | 16:33 |
| openstack | bug 1673030 in tripleo "ocata/stable jobs are broken (httpd restart failures)" [Critical,Confirmed] https://launchpad.net/bugs/1673030 - Assigned to Michele Baldessari (michele) | 16:33 |
| mwhahaha | i'll check on it when i get back | 16:33 |
| bandini | aye | 16:33 |
| panda | add the alert tag | 16:34 |
| bandini | done | 16:35 |
| panda | bandini: mh, actually ... if you're seeing it on upgrades, maybe it's not oooq fault, that is not yet transitioned\ | 16:36 |
| bandini | panda: no I fixed the upgrade part. mwhahaha saw it on normal jobs as of 1h ago | 16:38 |
| bandini | that's why he changed the description | 16:38 |
| bandini | see last comment | 16:38 |
| panda | mwhahaha: bandini we are not logging the url of the image, but the only place where it's set is in the release files, and we're sure that ocata release config is called | 16:39 |
| panda | http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a6432ed/console.html#_2017-03-17_14_10_08_410070 | 16:39 |
| panda | http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a6432ed/console.html#_2017-03-17_14_10_08_410070 | 16:39 |
| panda | the release file is /home/jenkins/workspace/gate-tripleo-ci-centos-7-nonha-multinode-oooq/.quickstart/config/release/tripleo-ci/ocata.yml | 16:40 |
| *** trown is now known as trown|lunch | 16:40 | |
| panda | I have to run, I'll check out later | 16:40 |
| panda | trown|lunch: also ^ | 16:40 |
| *** panda is now known as panda|bbl | 16:41 | |
| trozet | EmilienM, mwhahaha: can you take a look at https://review.openstack.org/#/c/425383/ ? I added unit testing and release notes | 16:41 |
| *** abehl has joined #tripleo | 16:47 | |
| *** funzo_ has joined #tripleo | 16:52 | |
| *** jaosorior has quit IRC | 16:55 | |
| *** skramaja has quit IRC | 16:57 | |
| *** funzo_ has quit IRC | 16:57 | |
| *** egafford has quit IRC | 17:03 | |
| *** jlinkes has quit IRC | 17:03 | |
| *** athomas has quit IRC | 17:07 | |
| *** salmankhan has quit IRC | 17:09 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Add dracut-regenerate elements https://review.openstack.org/447047 | 17:09 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673030 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1673030 in tripleo "ocata/stable jobs are broken (httpd restart failures)" [Critical,Confirmed] - Assigned to Michele Baldessari (michele) | 17:10 |
| *** ckyriakidou has joined #tripleo | 17:13 | |
| *** dtantsur is now known as dtantsur|afk | 17:13 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Add lookup plugin for tripleo heat templates https://review.openstack.org/441164 | 17:16 |
| *** pcaruana has quit IRC | 17:18 | |
| *** egafford has joined #tripleo | 17:20 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: WIP: Add lvm management to diskimage-builder https://review.openstack.org/444403 | 17:20 |
| *** psahoo has quit IRC | 17:22 | |
| *** zoli|wfh is now known as zoli|gone | 17:22 | |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: create history for the overcloud deploy command https://review.openstack.org/446617 | 17:23 |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Always execute autorelabel https://review.openstack.org/447076 | 17:24 |
| marios | /win 36 | 17:24 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Port the network environment validation https://review.openstack.org/341586 | 17:25 |
| *** ckyriakidou has quit IRC | 17:26 | |
| *** rbrady-food is now known as rbrady | 17:27 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/instack-undercloud master: Add option to configure ceilometer with mongodb backend https://review.openstack.org/446707 | 17:28 |
| *** cylopez has quit IRC | 17:30 | |
| *** aufi has quit IRC | 17:32 | |
| *** jpena is now known as jpena|off | 17:34 | |
| *** lmiccini has quit IRC | 17:35 | |
| *** yamahata has joined #tripleo | 17:44 | |
| *** trown|lunch is now known as trown | 17:44 | |
| *** suuuper has quit IRC | 17:45 | |
| *** fragatina has joined #tripleo | 17:46 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/tripleo-ci master: Do not merge: Test python-heat-agents package build https://review.openstack.org/439178 | 17:47 |
| *** bogdando has quit IRC | 17:48 | |
| *** salmankhan has joined #tripleo | 17:49 | |
| *** toure is now known as toure|biab | 17:51 | |
| *** Vijayendra has quit IRC | 17:53 | |
| *** funzo_ has joined #tripleo | 17:53 | |
| *** akrivoka has quit IRC | 17:54 | |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-docs master: Document scale introspection issues https://review.openstack.org/447008 | 17:55 |
| *** thrash is now known as thrash|brb | 17:57 | |
| *** funzo_ has quit IRC | 17:58 | |
| openstackgerrit | Justin Kilpatrick proposed openstack/tripleo-docs master: Add warning about undercloud.conf configuration https://review.openstack.org/446997 | 18:04 |
| *** abehl has quit IRC | 18:06 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673030 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1673030 in tripleo "ocata/stable jobs are broken (httpd restart failures)" [Critical,Confirmed] - Assigned to Michele Baldessari (michele) | 18:10 |
| *** suuuper has joined #tripleo | 18:10 | |
| *** egafford has quit IRC | 18:12 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Use stevedore for module config of block device https://review.openstack.org/447090 | 18:16 |
| beagles | dsneddon: just a heads up that those jobs were probably failing because of the bug mentioned ^^ | 18:16 |
| beagles | well, not that one .. (race conditon) 1673030 | 18:16 |
| dsneddon | beagles, OK, I'll halt on my rechecks until that gets worked out | 18:17 |
| *** pkovar1 has quit IRC | 18:19 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Update ci-script to enable TripleO gate dependencies https://review.openstack.org/431764 | 18:21 |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Add default image for Ocata deploys https://review.openstack.org/444942 | 18:21 |
| *** tbonds has quit IRC | 18:22 | |
| rook | shardy: around? i have a stack update always timing out on : | 18:24 |
| *** egafford has joined #tripleo | 18:24 | |
| *** tbonds has joined #tripleo | 18:24 | |
| rook | shardy: https://gist.github.com/jtaleric/d87a7ded971fe1e907193457d5a5da7f | 18:24 |
| rook | everything on the node itself looks fine | 18:24 |
| rook | ok | 18:27 |
| rook | nm | 18:27 |
| rook | I have never seen this before | 18:27 |
| rook | https://gist.github.com/jtaleric/270ce10ee79ab36c58355288a8a2fbd9 | 18:27 |
| rook | shardy ^ | 18:27 |
| *** pblaho has quit IRC | 18:29 | |
| *** liverpooler has quit IRC | 18:31 | |
| *** liverpooler has joined #tripleo | 18:31 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/tripleo-ci master: Do not merge: Test python-heat-agents package build https://review.openstack.org/439178 | 18:32 |
| *** gfidente has quit IRC | 18:37 | |
| *** rhallisey has quit IRC | 18:37 | |
| *** rhallisey has joined #tripleo | 18:38 | |
| *** salmankhan has quit IRC | 18:39 | |
| *** trown is now known as trown|brb | 18:43 | |
| *** alop has joined #tripleo | 18:48 | |
| *** tesseract has quit IRC | 18:50 | |
| *** funzo has joined #tripleo | 18:54 | |
| *** thrash|brb is now known as thrash | 18:56 | |
| *** tosky has quit IRC | 18:58 | |
| *** funzo has quit IRC | 18:59 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Use stevedore for module config of block device https://review.openstack.org/447090 | 18:59 |
| openstackgerrit | John Trowbridge proposed openstack/tripleo-quickstart-extras master: Collect the last 500 lines from the journal in text form https://review.openstack.org/447099 | 19:01 |
| *** toure|biab is now known as toure | 19:04 | |
| *** suuuper has quit IRC | 19:05 | |
| *** amoralej is now known as amoralej|off | 19:09 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 19:09 |
| *** cwolferh has quit IRC | 19:09 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673030 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1673030 in tripleo "ocata/stable jobs are broken (httpd restart failures)" [Critical,Confirmed] - Assigned to Michele Baldessari (michele) | 19:10 |
| Ng | would any tripleoclient reviewers have a moment to look at https://review.openstack.org/#/c/442497/ ? :) | 19:10 |
| rook | any os-collect-config gurus around? | 19:11 |
| rook | I have a single node in a stack update that is failing, and it seems to be related to os-refresh-config ? https://gist.github.com/jtaleric/270ce10ee79ab36c58355288a8a2fbd9#file-gistfile2-txt | 19:11 |
| *** florianf has quit IRC | 19:12 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Keep existing data for containerized RabbitMQ https://review.openstack.org/442557 | 19:12 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Keep existing data for containerized Swift https://review.openstack.org/443112 | 19:12 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: docker/keystone: add metadata_settings to output https://review.openstack.org/446483 | 19:12 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: DO NOT MERGE: Test for OVB yum update hang https://review.openstack.org/446026 | 19:19 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-puppet-elements master: Fix permissions of element files https://review.openstack.org/447105 | 19:25 |
| *** b00tcat has quit IRC | 19:27 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: DO NOT MERGE - Intermediate step waiting merge https://review.openstack.org/447106 | 19:28 |
| Ng | rook: do you get the same traceback if you run the first os-apply-config command in that 06-rhel-registration script? ( os-apply-config --key rh_registration.activation_key --type raw --key-default '' ) | 19:32 |
| Ng | rook: (I would expect so, just want to narrow down the scope here) | 19:32 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: WIP: Testing vxlan networking change https://review.openstack.org/446934 | 19:33 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement verify profiles as a custom action https://review.openstack.org/429684 | 19:33 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check ironic boot configuration as a custom action https://review.openstack.org/428225 | 19:33 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check flavors as a custom action https://review.openstack.org/428178 | 19:33 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check node counts as a custom action https://review.openstack.org/429757 | 19:33 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Add workflow wrapping all predeployment validations https://review.openstack.org/445914 | 19:33 |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Implement check boot images as a custom action https://review.openstack.org/428177 | 19:33 |
| openstackgerrit | Ben Nemec proposed openstack/instack master: Allow instack to work with pre-v2 versions of dib https://review.openstack.org/447116 | 19:43 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud stable/mitaka: Nothing to see here https://review.openstack.org/447117 | 19:43 |
| *** shardy has quit IRC | 19:44 | |
| openstackgerrit | yolanda.robla proposed openstack/diskimage-builder master: Always execute autorelabel https://review.openstack.org/447076 | 19:46 |
| beagles | dsneddon: I think I found a bug in the NM_CONTROLLED patch for os-net-config | 19:47 |
| beagles | dsneddon: nm_controlled is set to json.get('nm_controlled') where I think it should have been set to the strutils.bool_from_string(str(json.get('nm_controlled', False))) | 19:48 |
| beagles | dsneddon: the result is that if the value appears in the json at all it is treated as true | 19:48 |
| *** rbrady is now known as rbrady-afk | 19:48 | |
| beagles | I'll write a patch up | 19:49 |
| openstackgerrit | Ben Nemec proposed openstack/instack master: Allow instack to work with pre-v2 versions of dib https://review.openstack.org/447116 | 19:52 |
| openstackgerrit | Ben Nemec proposed openstack/instack-undercloud stable/mitaka: Nothing to see here https://review.openstack.org/447117 | 19:52 |
| beekneemech | Oh no, I don't need to run unit tests. I'm sure my code is fine. | 19:52 |
| beekneemech | Someday I'll learn. :-) | 19:53 |
| bkero | EmilienM, panda|bbl, sshnaidm|off: I think I have a patch that solves the repo problem properly. | 19:54 |
| *** trown|brb is now known as trown | 19:54 | |
| *** funzo has joined #tripleo | 19:55 | |
| beekneemech | Oh shoot, we don't have mitaka multinode jobs. | 19:55 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add option to build delorean changes and include the repo https://review.openstack.org/434004 | 19:55 |
| beekneemech | That means I get to wait 4 hours for results. :-/ | 19:56 |
| *** d0ugal has quit IRC | 19:56 | |
| beagles | dsneddon: a clarification - if you specify it as a *string* it doesn't allow setting as false | 19:59 |
| *** funzo has quit IRC | 19:59 | |
| dsneddon | beagles, Oh, interesting. Is that because it does an automatic type conversion? | 19:59 |
| beagles | dsneddon: I guess it's if you specify "nm_controlled": false it knows it wants a bool | 20:00 |
| beagles | dsneddon: but "nm_controlled": "false" it doesn't it just sets a value and where we do "if foo.nm_controlled" if it has a value it gets treated as true | 20:00 |
| beagles | dsneddon: I'm not sure what bools from templates get written as for networkconfigs now... so maybe it isn't an issue at all? | 20:01 |
| beagles | dsneddon: well.. it's an issue of consistency with other bool things but it isn't *as serious* of an issue | 20:02 |
| * beagles shrugs | 20:02 | |
| bkero | EmilienM: yay works. http://logs.openstack.org/65/446865/2/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/6c0ad81/logs/undercloud/home/jenkins/toci_vxlan_networking.sh.log.txt.gz | 20:03 |
| dsneddon | beagles, I think this might only be an issue with the example. | 20:04 |
| dsneddon | beagles, If we were doing this via Heat, we'd be passing a YAML, and the problem doesn't appear there, right?> | 20:05 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart master: Modify call to quickstart with options for repo include https://review.openstack.org/425447 | 20:05 |
| beagles | dsneddon: possibly - I'm testing against hand modified config.json at the moment on osp10 just to validate the package | 20:06 |
| dsneddon | beagles, Oh, wait, nevermind. The issue is in the json.get, you're right. | 20:07 |
| dsneddon | beagles, It should be the same as when we read in defroute, which is the way you showed above. | 20:08 |
| beagles | dsneddon: ack | 20:08 |
| rook | Ng: yes | 20:09 |
| rook | https://gist.github.com/jtaleric/c35f97d3bda97e10c9305296ad8cbfbd | 20:09 |
| rook | Ng this is a live cloud, so I am concerned how to get around this. | 20:09 |
| rook | I am trying to fix the stack update, so I can further scale it up. | 20:09 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: Add option to build delorean changes and include the repo https://review.openstack.org/434004 | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673030 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1673030 in tripleo "ocata/stable jobs are broken (httpd restart failures)" [Critical,Confirmed] - Assigned to Michele Baldessari (michele) | 20:10 |
| *** Goneri has quit IRC | 20:13 | |
| *** fzdarsky has quit IRC | 20:13 | |
| *** d0ugal has joined #tripleo | 20:13 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct https://review.openstack.org/446934 | 20:16 |
| openstackgerrit | Corey O'Brien proposed openstack/diskimage-builder master: Don't unmount missing paths If the path is missing, unmount_dir fails and skips the EXTRA_DETACH and EXTRA_UNMOUNT steps https://review.openstack.org/446621 | 20:16 |
| Ng | rook: take a look at /var/lib/os-collect-config/os_config_files.json - that's the default for the opts.os_config_files it's trying to load/parse | 20:17 |
| Ng | rook: sorry, I need to disappear now, but my guess is that file is either going to be empty or broken somehow | 20:18 |
| *** snecklifter_ has quit IRC | 20:18 | |
| *** salmankhan has joined #tripleo | 20:19 | |
| *** salmankhan has quit IRC | 20:23 | |
| openstackgerrit | Andreas Florath proposed openstack/diskimage-builder master: Use stevedore for module config of block device https://review.openstack.org/447090 | 20:25 |
| *** yolanda has quit IRC | 20:26 | |
| *** yolanda has joined #tripleo | 20:26 | |
| rook | HG0O9BVCO9a\_ | 20:28 |
| rook | ?oxz( | 20:28 |
| dtrainor | do we have a doc that describes the logic behind os-net-config ? under what conditions data is used or referenced etc | 20:28 |
| dtrainor | next stop, reading the source, but after today i'm looking to digest something a little more friendly | 20:28 |
| rook | sorry 2 year old | 20:28 |
| dsneddon | We do demonstrate most of the usage in the etc/os-net-config/samples directory in the source tree | 20:29 |
| rook | Ng that file is blank | 20:29 |
| dtrainor | er, os-collect-config is what i'm after, i think | 20:30 |
| dtrainor | ran in to a problem with nic ordering on a host and hope to learn something from that | 20:30 |
| rook | Ng could I reuse one from a different node and update it to match what should be for the specific compute node. | 20:30 |
| openstackgerrit | Brent Eagles proposed openstack/os-net-config master: Handle bools-as-strings nm_controlled values https://review.openstack.org/447126 | 20:30 |
| *** snecklifter_ has joined #tripleo | 20:30 | |
| openstackgerrit | Merged openstack/os-net-config stable/ocata: Allow setting NM_CONTROLLED=yes in os-net-config ifcfg files https://review.openstack.org/445060 | 20:31 |
| *** bfournie has quit IRC | 20:32 | |
| mwhahaha | any of you oooq folks around? looks like on the multinode jobs the repo-setup on the subnode isn't getting the stable branch info | 20:34 |
| mwhahaha | https://review.openstack.org/#/c/446121/ solved the upgrade but this is just normal stable/* branches | 20:34 |
| mwhahaha | not sure how to add that into oooq for the role | 20:35 |
| *** rbowen has quit IRC | 20:37 | |
| *** panda|bbl is now known as panda | 20:37 | |
| *** jayg is now known as jayg|g0n3 | 20:39 | |
| *** jcoufal has quit IRC | 20:39 | |
| panda | mwhahaha: interesting | 20:40 |
| mwhahaha | panda: you've come back :D | 20:40 |
| sai | dsneddon: quick question should the undercloud be handing out ips for dhcp/introspection only to interface with macs in isntackenv.json...i see its handing out a bunch of ips to already deployed nodes when introspecting some other nodes in the same env | 20:40 |
| panda | mwhahaha: I have to wait until the daemon goes to sleep | 20:40 |
| mwhahaha | panda: i know this pain | 20:40 |
| panda | mwhahaha: this is a fix for non oooq jobs right ? | 20:40 |
| mwhahaha | panda: http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/a6432ed/logs/subnode-2/syslog.txt.gz#_Mar_17_14_10_51 | 20:40 |
| mwhahaha | panda: no it's oooq | 20:41 |
| dsneddon | sai, when you start introspection, the DHCP server for introspection will hand out IPs from the introspection range to any MAC which isn't already registered. | 20:41 |
| panda | mwhahaha: ok, I think it's the same issue sagi worked around, and bkero is working to close completely | 20:41 |
| dsneddon | sai, Once you have introspected a host, that MAC should be added to the firewall preventing it from getting an introspection IP again. | 20:42 |
| bkero | panda: https://review.openstack.org/#/c/446934/ should fix it | 20:42 |
| mwhahaha | panda: also over on tripleo-puppet-elements it's getting some bizaro repo for the undercloud, i'm seeing vs 10.0.0 of the packages instead of 11.0.0 | 20:42 |
| dsneddon | sai, The question is why are your already-deployed nodes getting introspection IPs through DHCP? | 20:42 |
| dsneddon | sai, Is it possible you have more than one NIC attached to the provisioning network? | 20:42 |
| sai | dsneddon: this is turning out to be problematic (already deployed nodes have some unused nics that are set to dhcp by default os-net-config doesnt touch them), so when u have a cloud already up and try to introspect some other nodes the already deployed nodes are picking up the introspection ips | 20:43 |
| mwhahaha | panda: https://review.openstack.org/#/c/447105/ undercloud fails because of missing puppet modules if you look int he logs the packages are not pike versions | 20:43 |
| dsneddon | sai, The workaround is to add those NICs to the NIC config template with use_dhcp: false. | 20:43 |
| sai | dsneddon: they have fours nics and i sue only em2..the other three nics have dhcp by default | 20:43 |
| sai | dsneddon: yeh just realized, do we have a cleanner solution than workaround? or is this just how it is supposed to be working | 20:43 |
| sai | rook: ^ | 20:44 |
| dsneddon | sai, yes, the expectation is that you will configure custom NIC templates which disable those NICs. The other possibility is to put all NICs into an active/backup bond, or an OVS balance-slb bond. | 20:44 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct https://review.openstack.org/446934 | 20:44 |
| sai | dsneddon: got it, maybe we should document this if it is not already | 20:44 |
| dsneddon | sai, Either way will work, but in active/backup only one NIC will be active at one time and will fail over. In balance-slb mode, OVS will only use one NIC per VLAN at any time, but if you have multiple VLANs it will balance them across the links. | 20:44 |
| dsneddon | sai, I believe this is well documented. We recommend either only using a single provisioning NIC, or using bonds for multiple NICs. | 20:45 |
| dsneddon | sai, Actually, I take it back. I think it's only covered in product documentation. That would be a good addition to tripleo-docs. | 20:45 |
| sai | dsneddon: i use only one nic for provisioning, problem is there are other unused nics that have bootproto set to dhcp (because i havent used these nics in the nic-configs0 | 20:46 |
| sai | single-nic-vlans is the config i use inspite of having multiple nics on the machine, that is the problem | 20:46 |
| *** dprince has quit IRC | 20:46 | |
| sai | and i see this only when some nodes are already deployed and i try to introspect other nodes (introispection doesnt work because already deployed nodes eat up the ips undercloud provides) | 20:47 |
| panda | bkero: does https://trunk.rdoproject.org/centos7/current/ contain packages for master ? | 20:47 |
| mwhahaha | panda: it does | 20:48 |
| sai | dsneddon: i was expecting more of a "if there are nics u r not going to use in the OC, make sure to include them in the nic-templates and set dhcp: false" i nthe documentation | 20:48 |
| bkero | panda: that repo isn't linked from trunk.rdoproject.org | 20:48 |
| bkero | I don't know what it does | 20:48 |
| mwhahaha | it's current current (the bleeding edite) i believe | 20:49 |
| panda | mwhahaha: where do you see the mismatching package version ? | 20:49 |
| mwhahaha | panda: are we using current somewhere? (we shouldn't be) | 20:50 |
| dsneddon | sai Agreed, I'm putting it on my todo list for the next revision I do to tripleo-docs. First I have to make sure a fix to the last update I made gets merged. | 20:50 |
| mwhahaha | panda: we should beusing current-passed-ci in ci i thought | 20:50 |
| panda | mwhahaha: master/tripleo-ci is using it for some packages, or we won't be able to gate | 20:50 |
| sai | dsneddon: i will put an rfe in, so you don't forget :) | 20:50 |
| dsneddon | sai, Perfect, because it's always good to have a Closes-Bug: in a review. | 20:51 |
| panda | mwhahaha: https://github.com/openstack/tripleo-quickstart/blob/master/config/release/tripleo-ci/master.yml#L28 | 20:51 |
| sai | dsneddon: haha so here? https://launchpad.net/tripleo | 20:51 |
| beekneemech | current is used for tripleo packages because we gate on those and should always know that they work. | 20:51 |
| dsneddon | sai, well, since https://bugs.launchpad.net/tripleo-docs/ doesn't exist, then I would say https://bugs.launchpad.net/tripleo/ | 20:52 |
| mwhahaha | panda: ah yea that's right, yea those are fine | 20:52 |
| mwhahaha | panda: but those should be skipped for s table | 20:52 |
| panda | mwhahaha: they are, the change yopu pasted me was for master https://review.openstack.org/#/c/447105/ | 20:53 |
| mwhahaha | panda: yea let me dig into that one seperately as that might be something else | 20:54 |
| mwhahaha | panda: i'd rather fix the stable/ocata problem https://review.openstack.org/#/c/446011/ | 20:54 |
| *** funzo has joined #tripleo | 20:55 | |
| sai | dsneddon: https://bugs.launchpad.net/tripleo/+bug/1673882 | 20:56 |
| openstack | Launchpad bug 1673882 in tripleo "[TripleO-Docs] The nic-config related documentation should mention that unused itnerfaces in overcloud should be set to dhcp_false" [Undecided,New] | 20:56 |
| panda | mwhahaha: I think it timed out cloning the script that create the vxlan for multinode, bkero can you confim ? http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/3af9223/logs/undercloud/home/jenkins/toci_vxlan_networking_multinode.sh.log.txt.gz | 20:58 |
| panda | script is killed by signal 15 | 20:59 |
| panda | maybe it took too much to clone the repo | 20:59 |
| mwhahaha | panda: no the failure was for deploy failure because of httpd | 20:59 |
| mwhahaha | or maybe i have too many windows up | 21:00 |
| panda | mwhahaha: on this change ? https://review.openstack.org/#/c/446011/ | 21:00 |
| * mwhahaha closes everything down and starts again | 21:00 | |
| bkero | panda: Hard to say | 21:00 |
| panda | mwhahaha: I see the oooq job failing when it tries to create the vxlan magic for multinode | 21:00 |
| *** funzo has quit IRC | 21:00 | |
| *** lblanchard has quit IRC | 21:01 | |
| mwhahaha | panda: yea last failure was teh vxlan http://logs.openstack.org/11/446011/1/check/gate-tripleo-ci-centos-7-nonha-multinode-oooq/3af9223/console.html#_2017-03-17_16_33_39_247364 | 21:02 |
| panda | bkero: tripleo.sh under quickstart is clearly giving signs of rejection, meybse we should bring up and prioritize the work to replace it .. | 21:02 |
| bkero | panda: it = the vxlan networking things? | 21:03 |
| bkero | I have a patch incoming for it | 21:03 |
| panda | mwhahaha: ok, that should be transient | 21:03 |
| panda | bkero: in general, to bootstrapnode and to create vxlan | 21:03 |
| mwhahaha | panda: yea rechecking, now i'll get back to digging into the other problem | 21:03 |
| bkero | panda: The bootstrapping is fine, and I refactored the creation of vxlan | 21:03 |
| panda | mwhahaha: ok, I'll be here until I collapse to the ground. | 21:04 |
| *** thrash is now known as thrash|wknd | 21:06 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-puppet-elements master: Fix permissions of element files https://review.openstack.org/447105 | 21:06 |
| *** abishop has quit IRC | 21:06 | |
| rook | sai is that any different than what I recommended? | 21:07 |
| *** jkilpatr has quit IRC | 21:07 | |
| panda | bkero: ok, thanks, I guess I won't be satisfied until we replace them, and I'll unnecessarily blame the transplant for a while. | 21:07 |
| *** trown is now known as trown|outtypewww | 21:08 | |
| openstackgerrit | Ben Kero proposed openstack-infra/tripleo-ci master: Revert second repo-setup run for overcloud nodes https://review.openstack.org/447142 | 21:09 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673030 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1673030 in tripleo "ocata/stable jobs are broken (httpd restart failures)" [Critical,Confirmed] - Assigned to Michele Baldessari (michele) | 21:10 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Re-Add bigswitch agent support https://review.openstack.org/446029 | 21:10 |
| *** rlandy has quit IRC | 21:11 | |
| openstackgerrit | Ben Kero proposed openstack/tripleo-quickstart-extras master: Refactor the toci-vxlan-networking script to be more correct https://review.openstack.org/446934 | 21:11 |
| bkero | panda: There's the revert, and there's the refactor | 21:12 |
| bkero | That combination should provide a proper fix | 21:12 |
| *** snecklifter_ has quit IRC | 21:18 | |
| bkero | Just waiting on testing now. | 21:20 |
| *** ipsecguy has quit IRC | 21:21 | |
| *** ipsecguy has joined #tripleo | 21:22 | |
| *** pradk has quit IRC | 21:26 | |
| *** snecklifter_ has joined #tripleo | 21:30 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Enables OpenDaylight Clustering in HA deployments https://review.openstack.org/425383 | 21:46 |
| *** fultonj has quit IRC | 21:47 | |
| *** karimb has quit IRC | 21:51 | |
| *** oaken_chris has quit IRC | 21:51 | |
| *** funzo has joined #tripleo | 21:56 | |
| *** funzo has quit IRC | 22:01 | |
| *** salmankhan has joined #tripleo | 22:06 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1673030 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1673030 in tripleo "ocata/stable jobs are broken (httpd restart failures)" [Critical,Confirmed] - Assigned to Michele Baldessari (michele) | 22:10 |
| *** yamahata has quit IRC | 22:10 | |
| *** cwolferh has joined #tripleo | 22:11 | |
| *** egafford has quit IRC | 22:13 | |
| bkero | panda: So the patches work. | 22:16 |
| openstackgerrit | Ben Kero proposed openstack/tripleo-heat-templates stable/ocata: TEST CHANGE, IGNORE https://review.openstack.org/446865 | 22:17 |
| panda | bkero: ok, great. I've looked and the changes, and they are very tripleo CI specific. So iether we move that piece of role in tripleo-ci, or we generalize it. | 22:18 |
| *** saneax-_-|AFK is now known as saneax | 22:19 | |
| bkero | panda: Where would you suggest putting it? | 22:19 |
| panda | bkero: I'd say we can create a smaller role from that task, create a toci-quickstart/roles dir and copy it there, then copy the role over extras dir before launching quickstart. It's a bit like what we do with playbooks in toci right now | 22:22 |
| panda | until we have time to create a specific set of roles to be installed as requirements to quickstart, instead of copying everything | 22:22 |
| panda | bkero: makes sense ? would it work ? | 22:22 |
| panda | bkero: this is my opinion, of course, meybe we decide to merge it as it is and generalize when we work on the libvirt reproducer | 22:23 |
| bkero | panda: That makes sense. That's going to be kind of nasty to do that to all the subnodes | 22:24 |
| bkero | but thankfully in this case we won't need to | 22:24 |
| bkero | panda: Even in the old case we're still cloning tripleo-ci and running the script from scripts/tripleo.sh | 22:26 |
| bkero | So this is actually making it more portable. But you're right, it's going to break libvirt-based deployments, but only because tripleo-ci isn't being cloned. | 22:26 |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud stable/ocata: Use purged for firewalld https://review.openstack.org/443679 | 22:27 |
| bkero | libvirt-based deployments aren't using multinode or vxlan networking right now though, so that's a non-issue | 22:28 |
| panda | yes, so I wouldn't be suprised if we decide to merge it as it is. I'll comment what I think on the change. | 22:28 |
| bkero | Sounds good to me. Please excuse me if I don't fix your issues right away, I'm going to wait for further testing to finish, which should be another 1 hour to 1.5 hours. | 22:29 |
| panda | bkero: absolutely, this was more a concern for the future than a immediate request. | 22:30 |
| bkero | Sure thing | 22:30 |
| *** alop has quit IRC | 22:31 | |
| bkero | and I'm open to alternate implementation suggestiosn -- but this one seems by far the more stable option, even if it is some copypasta from tripleo.sh's multinode_setup() | 22:31 |
| *** snecklifter has quit IRC | 22:35 | |
| EmilienM | bkero, panda: is there anything I need to review? | 22:37 |
| EmilienM | back from half day pto | 22:38 |
| bkero | EmilienM: https://review.openstack.org/#/c/446934/ https://review.openstack.org/#/c/447142/ | 22:38 |
| EmilienM | ack | 22:38 |
| bkero | EmilienM: I'm waiting for the scenario jobs to be done running against 446865 to test them | 22:38 |
| bkero | But they pass non-scenario jobs and passed scenario jobs in previous testing | 22:38 |
| EmilienM | ok both patches are not ready for review, CI results are not seen | 22:39 |
| EmilienM | I'll look at them over the weekend | 22:39 |
| bkero | EmilienM: CI results are and will be in 446865 | 22:39 |
| bkero | Oh, I never let those finish...so they're not there. Bleh. Sorry. :< | 22:40 |
| EmilienM | ah | 22:40 |
| bkero | To speed up development time I shortened the playbook to end after vxlan networking completed, which only took around 15 minutes. So I never waited the extra hour for the other jobs to complete. Now that I think it's complete I re-enabled the whole playbook. | 22:41 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: TEST: DONT RECHECK or REVIEW: periodic jobs https://review.openstack.org/359215 | 22:43 |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud stable/newton: Align stars to fix CI https://review.openstack.org/447161 | 22:44 |
| openstackgerrit | Alex Schultz proposed openstack/instack-undercloud stable/newton: Use purged for firewalld https://review.openstack.org/443680 | 22:44 |
| *** funzo has joined #tripleo | 22:57 | |
| *** jbadiapa has quit IRC | 23:01 | |
| *** liverpooler has quit IRC | 23:02 | |
| *** funzo has quit IRC | 23:02 | |
| bkero | EmilienM, panda: Well, the tests are failing right where they're supposed to due to the panko-db-sync problem. End of this file: http://logs.openstack.org/65/446865/3/check/gate-tripleo-ci-centos-7-scenario001-multinode-oooq/3ebddfa/logs/undercloud/var/log/postci.txt.gz | 23:27 |
| panda | panko again ? :( | 23:30 |
| bkero | Was it ever fixed? | 23:39 |
| openstackgerrit | Merged openstack/instack-undercloud stable/ocata: Align stars to fix CI https://review.openstack.org/446011 | 23:42 |
| *** salmankhan has quit IRC | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
