| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 00:10 |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| *** salmankhan has joined #tripleo | 00:12 | |
| *** bfournie has joined #tripleo | 00:27 | |
| *** salmankhan has quit IRC | 00:48 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 01:10 |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| *** daidv has joined #tripleo | 01:13 | |
| *** cshastri has joined #tripleo | 01:23 | |
| *** numans has quit IRC | 01:35 | |
| *** numans has joined #tripleo | 01:37 | |
| *** dobson has quit IRC | 01:45 | |
| *** dobson has joined #tripleo | 01:48 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 02:10 |
| *** artom_ has quit IRC | 02:29 | |
| *** artom_ has joined #tripleo | 02:29 | |
| *** artom_ has quit IRC | 02:30 | |
| *** artom_ has joined #tripleo | 02:30 | |
| *** agopi has joined #tripleo | 03:04 | |
| *** bkopilov has quit IRC | 03:10 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 03:10 |
| *** anilvenkata has joined #tripleo | 03:20 | |
| *** links has joined #tripleo | 03:34 | |
| *** psahoo has joined #tripleo | 03:52 | |
| openstackgerrit | zenghui.shi proposed openstack/tripleo-heat-templates master: Add PTP composable service https://review.openstack.org/491317 | 04:01 |
| *** ramishra has joined #tripleo | 04:08 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 04:10 |
| *** bkopilov has joined #tripleo | 04:31 | |
| *** ykarel has joined #tripleo | 04:42 | |
| *** udesale has joined #tripleo | 04:43 | |
| *** shreshtha has joined #tripleo | 04:50 | |
| openstackgerrit | Pranali Deore proposed openstack/tripleo-heat-templates master: Add support of netapp configuration for glance in THT https://review.openstack.org/530680 | 05:01 |
| *** pdeore has joined #tripleo | 05:06 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 05:10 |
| *** karthiks has joined #tripleo | 05:24 | |
| *** dpawar has joined #tripleo | 05:25 | |
| *** psachin has joined #tripleo | 05:32 | |
| *** skramaja has joined #tripleo | 05:45 | |
| *** agopi has quit IRC | 05:45 | |
| *** agopi has joined #tripleo | 05:45 | |
| *** masco has joined #tripleo | 05:54 | |
| *** pgadiya has joined #tripleo | 05:56 | |
| *** agopi has quit IRC | 06:00 | |
| *** jaganathan has joined #tripleo | 06:05 | |
| *** jbadiapa has joined #tripleo | 06:07 | |
| *** skramaja has quit IRC | 06:09 | |
| *** skramaja_ has joined #tripleo | 06:09 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 06:10 |
| *** skramaja_ has quit IRC | 06:13 | |
| *** skramaja_ has joined #tripleo | 06:14 | |
| *** akane has joined #tripleo | 06:16 | |
| *** akane_ has joined #tripleo | 06:16 | |
| *** jbadiapa has quit IRC | 06:26 | |
| *** yprokule has joined #tripleo | 06:28 | |
| *** ratailor has joined #tripleo | 06:30 | |
| *** jbadiapa has joined #tripleo | 06:31 | |
| *** ameeda has joined #tripleo | 06:56 | |
| *** ameeda has joined #tripleo | 06:57 | |
| *** dsariel has quit IRC | 07:05 | |
| *** jaosorior has joined #tripleo | 07:07 | |
| jaosorior | Happy new year folks! | 07:08 |
| jaosorior | :D | 07:08 |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 07:10 |
| *** pcaruana has joined #tripleo | 07:24 | |
| *** pcaruana has quit IRC | 07:25 | |
| *** pcaruana has joined #tripleo | 07:26 | |
| *** pcaruana has quit IRC | 07:29 | |
| *** pcaruana has joined #tripleo | 07:29 | |
| *** agurenko has joined #tripleo | 07:32 | |
| *** pgadiya has quit IRC | 07:44 | |
| *** tosky has joined #tripleo | 07:48 | |
| *** matbu has joined #tripleo | 07:52 | |
| *** jaosorior has quit IRC | 07:58 | |
| *** nyechiel has joined #tripleo | 08:00 | |
| *** jaosorior has joined #tripleo | 08:00 | |
| *** Guest87 has joined #tripleo | 08:04 | |
| *** Guest87 has quit IRC | 08:07 | |
| *** Guest87 has joined #tripleo | 08:08 | |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 08:10 |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| *** jtomasek has joined #tripleo | 08:10 | |
| *** jtomasek has quit IRC | 08:12 | |
| *** jtomasek has joined #tripleo | 08:12 | |
| *** mcornea has joined #tripleo | 08:15 | |
| *** sri_ has quit IRC | 08:17 | |
| *** tesseract has joined #tripleo | 08:20 | |
| *** sri_ has joined #tripleo | 08:27 | |
| *** ykarel has quit IRC | 08:31 | |
| *** akane_ has left #tripleo | 08:31 | |
| *** ykarel has joined #tripleo | 08:32 | |
| *** akane has left #tripleo | 08:32 | |
| *** akane has joined #tripleo | 08:33 | |
| *** oidgar has joined #tripleo | 08:49 | |
| *** moshele has joined #tripleo | 08:50 | |
| *** tvignaud has joined #tripleo | 08:50 | |
| *** mdnadeem has joined #tripleo | 08:51 | |
| *** cylopez has joined #tripleo | 08:52 | |
| *** ccamacho has joined #tripleo | 08:55 | |
| *** jpich has joined #tripleo | 08:58 | |
| *** leseb has joined #tripleo | 09:04 | |
| *** paramite has joined #tripleo | 09:09 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 09:10 |
| *** yolanda has quit IRC | 09:13 | |
| openstackgerrit | Sergii Golovatiuk proposed openstack/python-tripleoclient master: Add Openstack Undercloud backup https://review.openstack.org/466213 | 09:14 |
| *** yolanda has joined #tripleo | 09:16 | |
| *** cylopez has left #tripleo | 09:17 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack/tripleo-docs master: Do not use -y for package install https://review.openstack.org/530711 | 09:19 |
| *** paramite has quit IRC | 09:20 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix unittest in tempest sendmail tool https://review.openstack.org/530713 | 09:22 |
| *** zshi has quit IRC | 09:24 | |
| *** bogdando has joined #tripleo | 09:28 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common master: Add UC backup Mistral workflow definition and actions https://review.openstack.org/517610 | 09:28 |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add Openstack Undercloud backup https://review.openstack.org/466213 | 09:30 |
| *** shardy has joined #tripleo | 09:32 | |
| *** lucas-afk is now known as lucasagomes | 09:33 | |
| *** dbecker has joined #tripleo | 09:34 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart master: Cleaning up featureset files from tripleo-ci related variables https://review.openstack.org/512225 | 09:35 |
| *** zshi has joined #tripleo | 09:38 | |
| *** agurenko has quit IRC | 09:44 | |
| *** agurenko has joined #tripleo | 09:48 | |
| *** dpawar has quit IRC | 09:51 | |
| *** gfidente has joined #tripleo | 09:52 | |
| *** gfidente has quit IRC | 09:52 | |
| *** gfidente has joined #tripleo | 09:52 | |
| *** derekh has joined #tripleo | 09:55 | |
| *** dsariel has joined #tripleo | 10:00 | |
| *** salmankhan has joined #tripleo | 10:03 | |
| *** stendulker has joined #tripleo | 10:07 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Add new SELinux validation check https://review.openstack.org/511211 | 10:07 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Add new nova-event-callback validation https://review.openstack.org/513333 | 10:08 |
| *** nyechiel has quit IRC | 10:09 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 10:10 |
| *** nyechiel has joined #tripleo | 10:11 | |
| *** zshi has quit IRC | 10:12 | |
| *** zshi has joined #tripleo | 10:13 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common master: Add UC backup Mistral workflow definition and actions https://review.openstack.org/517610 | 10:14 |
| *** stendulker has left #tripleo | 10:14 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add Openstack Undercloud backup https://review.openstack.org/466213 | 10:20 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-common master: Add UC backup Mistral workflow definition and actions https://review.openstack.org/517610 | 10:23 |
| *** pgadiya has joined #tripleo | 10:26 | |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add Openstack Undercloud backup https://review.openstack.org/466213 | 10:27 |
| *** ramishra has quit IRC | 10:35 | |
| *** akrivoka has joined #tripleo | 10:37 | |
| *** ramishra has joined #tripleo | 10:48 | |
| *** rfolco has joined #tripleo | 10:54 | |
| *** udesale has quit IRC | 10:56 | |
| *** rfolco has quit IRC | 10:58 | |
| mhenkel | good morning and Happy New Year! | 11:00 |
| *** pgadiya has quit IRC | 11:01 | |
| mhenkel | I have a question on step 5 for container based deployments: I created my container based service, it is started in step 5 (and is active on the host) but the overcloud creation is stuck with overcloud-AllNodesDeploySteps-kyai6l456kko-ContrailControllerDeployment_Step5-raaxsxdgdpc6 | 11:01 |
| mhenkel | status is CREATE_IN_PROGRESS | 11:02 |
| mhenkel | I logged into the node and couldn't spot any obvious messages | 11:02 |
| *** matbu has quit IRC | 11:02 | |
| shardy | mhenkel: Hi, happy new year! Is ansible running on the node (try ps ax | grep heat or ps ax | grep ansible to see if it's running via the heat hook) | 11:03 |
| shardy | mhenkel: also check /var/log/messages to see what the last command run via the heat hook was, and if it completed and signalled back to heat OK | 11:04 |
| *** matbu has joined #tripleo | 11:04 | |
| mhenkel | 21635 ? Rl 3:04 /usr/bin/python2 /usr/bin/ansible-playbook -i localhost, --module-path /usr/share/ansible-modules /var/lib/heat-config/heat-config-ansible/4291dc61-c4b7-4b0c-8ae8-e39495dc9828_playbook.yaml --extra-vars @/var/lib/heat-config/heat-config-ansible/4291dc61-c4b7-4b0c-8ae8-e39495dc9828_variables.json | 11:04 |
| mhenkel | 22593 ? S 0:00 /usr/bin/python2 /usr/bin/ansible-playbook -i localhost, --module-path /usr/share/ansible-modules /var/lib/heat-config/heat-config-ansible/4291dc61-c4b7-4b0c-8ae8-e39495dc9828_playbook.yaml --extra-vars @/var/lib/heat-config/heat-config-ansible/4291dc61-c4b7-4b0c-8ae8-e39495dc9828_variables.json | 11:04 |
| shardy | mhenkel: Ok, so you can kill that if you're sure it's stuck, then re-run with -vvv looking for clues | 11:05 |
| *** numans has quit IRC | 11:06 | |
| *** cylopez has joined #tripleo | 11:08 | |
| *** numans has joined #tripleo | 11:09 | |
| *** hjensas has quit IRC | 11:09 | |
| mhenkel | is that the stuck process: Jan 2 05:27:43 localhost os-collect-config: [2018-01-02 10:27:43,791] (heat-config) [DEBUG] Running /usr/libexec/heat-config/hooks/ansible < /var/lib/heat-config/deployed/4291dc61-c4b7-4b0c-8ae8-e39495dc9828.json ? | 11:09 |
| *** hjensas has joined #tripleo | 11:09 | |
| *** hjensas has quit IRC | 11:09 | |
| *** hjensas has joined #tripleo | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 11:10 |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| shardy | mhenkel: yes it's the same, os-collect-config runs heat-config, which runs ansible-playbook | 11:10 |
| *** psahoo has quit IRC | 11:14 | |
| *** pgadiya has joined #tripleo | 11:15 | |
| *** dpawar has joined #tripleo | 11:16 | |
| shardy | mhenkel: FYI it's also possible to completely disable running ansible via heat, and just run it from the undercloud: | 11:18 |
| shardy | https://github.com/openstack/tripleo-heat-templates/blob/master/environments/config-download-environment.yaml | 11:18 |
| shardy | mhenkel: you pass that environment file, then use "openstack overcloud config download" to download the playbooks etc | 11:18 |
| shardy | mhenkel: sometimes that is easier than debugging on the node, but it's essentially doing the same deploy steps | 11:18 |
| *** rfolco has joined #tripleo | 11:19 | |
| *** ratailor has quit IRC | 11:21 | |
| *** bkopilov has quit IRC | 11:23 | |
| mhenkel | shardy: I tried that but it runs all playbooks when I only want to debug a single one | 11:23 |
| mhenkel | I also tried to limit it to one but it didn't work | 11:24 |
| shardy | mhenkel: you can limit it with tags, see deploy_steps_playbook.yml, each set of task includes is tagged | 11:24 |
| mhenkel | ok, will give it a try | 11:25 |
| shardy | mhenkel: you can also cut/paste to create a temporary playbook to run the specific thing you need to develop/test | 11:25 |
| *** shreshtha has quit IRC | 11:26 | |
| *** masco has quit IRC | 11:29 | |
| mhenkel | shardy: running this manually on the node succeeds: | 11:30 |
| mhenkel | "/usr/bin/python2 /usr/bin/ansible-playbook -i localhost -vvv --module-path /usr/share/ansible-modules /var/lib/heat-config/heat-config-ansible/4291dc61-c4b7-4b0c-8ae8-e39495dc9828_playbook.yaml --extra-vars @/var/lib/heat-config/heat-config-ansible/4291dc61-c4b7-4b0c-8ae8-e39495dc9828_variables.json" | 11:30 |
| mhenkel | localhost : ok=9 changed=1 unreachable=0 failed=0 | 11:30 |
| shardy | mhenkel: Hmm, strange, I'd probably redeploy with config-download-environment, then try running the entire deploy sequence via ansible-playbook on the undercloud (unless you can figure out where it was stuck from the logs that is) | 11:32 |
| mhenkel | doing that right now | 11:33 |
| *** jaganathan has quit IRC | 11:33 | |
| *** pdeore_ has joined #tripleo | 11:35 | |
| *** pdeore has quit IRC | 11:35 | |
| *** etingof has joined #tripleo | 11:36 | |
| mhenkel | simple rerun doesn't work because of a lot of permission deny errors on the openstack node: | 11:37 |
| mhenkel | fatal: [192.168.24.13]: FAILED! => {"msg": "Failed to get information on remote file (/var/log/nova/readme.txt): Permission denied"} | 11:37 |
| *** nyechiel has quit IRC | 11:37 | |
| *** psahoo has joined #tripleo | 11:38 | |
| shardy | mhenkel: Hmm, we should have become: true where needed, but try with ansible-playbook -b | 11:38 |
| shardy | mhenkel: is thsi pike or trunk? | 11:38 |
| mhenkel | trunk | 11:38 |
| *** nyechiel has joined #tripleo | 11:39 | |
| mhenkel | -b works | 11:39 |
| shardy | mhenkel: ack, Ok we're missing some become: true then I guess - we don't see it run via heat as it's all run as root | 11:39 |
| *** mdnadeem has quit IRC | 11:40 | |
| *** dpawar has quit IRC | 11:42 | |
| openstackgerrit | Steven Hardy proposed openstack/python-tripleoclient master: WIP convert undercloud deploy to use TripleoInventory class https://review.openstack.org/529652 | 11:44 |
| openstackgerrit | Rafael Folco proposed openstack/tripleo-quickstart-extras master: [tempestmail] Custom email for different matchers https://review.openstack.org/529590 | 11:53 |
| *** pdeore_ has quit IRC | 11:54 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart-extras master: Add option to run job automatically in reproducing https://review.openstack.org/530401 | 11:57 |
| mhenkel | shardy: running the full deploy sequence from the undercloud worked. I will move the container creation from step 5 to 4 and see if that makes any difference | 12:00 |
| *** ykarel has quit IRC | 12:00 | |
| *** pkovar has joined #tripleo | 12:04 | |
| *** mdnadeem has joined #tripleo | 12:04 | |
| *** jkilpatr has joined #tripleo | 12:07 | |
| openstackgerrit | Rafael Folco proposed openstack/tripleo-quickstart-extras master: [tempestmail] Custom email for different matchers https://review.openstack.org/529590 | 12:07 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 12:10 |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-heat-templates stable/ocata: Check for yum lock befor all yum* operations. https://review.openstack.org/530725 | 12:10 |
| *** cshastri has quit IRC | 12:12 | |
| *** links has quit IRC | 12:13 | |
| openstackgerrit | Rafael Folco proposed openstack/tripleo-quickstart-extras master: [tempestmail] Add instruction to run unit tests https://review.openstack.org/530504 | 12:13 |
| *** cylopez has quit IRC | 12:16 | |
| *** cylopez has joined #tripleo | 12:19 | |
| *** Guest87 has quit IRC | 12:21 | |
| *** moshele has quit IRC | 12:22 | |
| *** jwb has joined #tripleo | 12:25 | |
| *** jjoyce|holiday is now known as jjoyce | 12:29 | |
| *** pkovar has quit IRC | 12:30 | |
| *** lucasagomes is now known as lucas-hugry | 12:30 | |
| *** gfidente has quit IRC | 12:30 | |
| *** ykarel has joined #tripleo | 12:30 | |
| *** bfournie has quit IRC | 12:32 | |
| *** bfournie has joined #tripleo | 12:32 | |
| *** moshele has joined #tripleo | 12:33 | |
| *** jaganathan has joined #tripleo | 12:33 | |
| *** moshele has quit IRC | 12:36 | |
| *** moshele has joined #tripleo | 12:37 | |
| *** anilvenkata has quit IRC | 12:39 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-heat-templates master: Add tls roles for undercloud https://review.openstack.org/517079 | 12:39 |
| *** jlabarre has joined #tripleo | 12:45 | |
| *** atoth has joined #tripleo | 12:48 | |
| *** akane has quit IRC | 12:52 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz data https://review.openstack.org/525603 | 12:52 |
| *** aputtur_ has joined #tripleo | 12:53 | |
| *** jaganathan has quit IRC | 12:57 | |
| *** jcoufal has joined #tripleo | 12:59 | |
| *** liverpooler has joined #tripleo | 13:00 | |
| *** pradk has joined #tripleo | 13:02 | |
| jaosorior | pradk: sup dude! how's it going? | 13:03 |
| jaosorior | happy new year :D | 13:03 |
| *** moshele has quit IRC | 13:03 | |
| oidgar | hi all, is somebody here knows if there is a flow diagram (or something similar) for tripleo overlcoud deployment? (aka t-h-t overcloud.j2 -> roles-data.yaml and so on) | 13:04 |
| *** dougbtv_ is now known as dougbtv | 13:05 | |
| *** moshele has joined #tripleo | 13:05 | |
| *** rlandy has joined #tripleo | 13:07 | |
| *** mburned has quit IRC | 13:09 | |
| *** mburned has joined #tripleo | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 13:10 |
| jaosorior | oidgar: hey! by the way, got any docs on the octavia TLS bits that I could read? | 13:12 |
| *** tzumainn has joined #tripleo | 13:13 | |
| oidgar | jaosorior: hey! checking | 13:13 |
| *** abishop has joined #tripleo | 13:13 | |
| *** rlandy is now known as rlandy|ruck | 13:14 | |
| *** pblaho has joined #tripleo | 13:14 | |
| oidgar | jaosorior: unfortunately Octavia does not have any documentation related to TLS, but I have a mail with some details about it, forwarding it to you now | 13:15 |
| oidgar | jaosorior: If you have specific questions, I might have some answers for that | 13:16 |
| jaosorior | oidgar: wasn't it the case that octavia was using barbican to store the certs' private keys? | 13:17 |
| pradk | jaosorior, hey, you too! | 13:17 |
| jaosorior | pradk: did you guys get around the TLS issue for the containerized undercloud? | 13:18 |
| mhenkel | shardy: same thing. now it gets stuck in step 4. This is the last heat command executed: | 13:19 |
| mhenkel | Jan 2 07:42:16 localhost os-collect-config: [2018-01-02 12:42:16,130] (heat-config) [DEBUG] Running /usr/libexec/heat-config/hooks/ansible < /var/lib/heat-config/deployed/6aae18c9-f641-4d0c-90d0-dfb0e716cd75.json | 13:19 |
| pradk | jaosorior, i was on paternity leave all of dec.. so not sure how far they got, lets check with dprince when he is back | 13:19 |
| mhenkel | container is up and running | 13:19 |
| jaosorior | pradk: oh, hey congrats!! | 13:19 |
| oidgar | jaosorior: no, we are talking about the management traffic certificate between octavia services and the amphora | 13:19 |
| pradk | jaosorior, thx! | 13:19 |
| oidgar | jaosorior: as I know, octavia uses barbican to store TLS termination certificates for its members | 13:20 |
| mhenkel | shardy: this is a list of ansible related processes running: https://pastebin.com/6XNJ7Cc4 | 13:20 |
| *** skramaja_ has quit IRC | 13:22 | |
| *** pblaho has quit IRC | 13:22 | |
| *** gfidente has joined #tripleo | 13:23 | |
| *** gfidente has quit IRC | 13:23 | |
| *** gfidente has joined #tripleo | 13:23 | |
| *** bfournie has joined #tripleo | 13:23 | |
| *** pgadiya has quit IRC | 13:23 | |
| *** ramishra has quit IRC | 13:26 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/python-tripleoclient master: WIP: Handle user-provided TLS certificate/key for the undercloud https://review.openstack.org/530737 | 13:29 |
| jaosorior | pradk: started working on this part ^^ need to test it out, and write tests too | 13:29 |
| pradk | cool | 13:30 |
| *** Guest87 has joined #tripleo | 13:31 | |
| *** pkovar has joined #tripleo | 13:36 | |
| *** pblaho has joined #tripleo | 13:36 | |
| *** udesale has joined #tripleo | 13:36 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/python-tripleoclient master: WIP: Handle user-provided TLS certificate/key for the undercloud https://review.openstack.org/530737 | 13:36 |
| *** lucas-hugry is now known as lucasagomes | 13:38 | |
| *** moshele has quit IRC | 13:40 | |
| *** moshele has joined #tripleo | 13:41 | |
| *** pgadiya has joined #tripleo | 13:47 | |
| *** rhallisey has joined #tripleo | 13:49 | |
| *** jmelvin has joined #tripleo | 13:50 | |
| *** trown|outtypewww is now known as trown|rover | 13:51 | |
| mfedosin | hello! I have a problem with the quickstart... | 13:51 |
| mfedosin | actually it fails here http://paste.openstack.org/show/636123/ | 13:51 |
| mfedosin | the task is called "modify-image : Run virt-customize on the provided image" | 13:52 |
| mfedosin | from the log I understood that there is something wrong with the gnocci package | 13:52 |
| mfedosin | http://paste.openstack.org/show/636121/ | 13:53 |
| mfedosin | do you know what it could be? | 13:53 |
| trown|rover | mfedosin: that does indeed look like some issue with the gnocci packaging, what release is that? | 13:57 |
| *** liverpooler has quit IRC | 13:58 | |
| mwhahaha | i think they dropped the sqlalchemy package | 13:58 |
| * mwhahaha goes and looks for the review | 13:58 | |
| mwhahaha | so something in rdo is out of sync | 13:58 |
| mwhahaha | https://review.rdoproject.org/r/#/c/10449/ | 13:59 |
| ykarel | looks like alfredo's comment was valid one | 13:59 |
| mwhahaha | indeed | 14:00 |
| ykarel | iit should be < 4.1.3 | 14:00 |
| trown|rover | yep | 14:00 |
| ykarel | mrunge, ^^ | 14:00 |
| mwhahaha | also, happy tuesday or something | 14:01 |
| trown|rover | weee | 14:01 |
| *** ecerquei has joined #tripleo | 14:01 | |
| dpeacock | Happy New Year all :-) | 14:01 |
| mrunge | ykarel, ? | 14:01 |
| ykarel | mrunge, gnocchi sqlalchemy issue: https://review.rdoproject.org/r/#/c/10449/5 | 14:01 |
| oidgar | do we have a meeting now? | 14:02 |
| * jrist wonders same | 14:02 | |
| mwhahaha | no | 14:02 |
| mwhahaha | i see you guys don't read your email :D | 14:02 |
| mrunge | ykarel, the question is, why is it installed and tried to be updated | 14:02 |
| mwhahaha | we canceled last weeks and this week's meeting due to the holidays | 14:02 |
| ykarel | mrunge, mrunge need to check, but looks it's already in the image, | 14:02 |
| ykarel | and repo_setup updates it | 14:03 |
| trown|rover | mrunge: ya images would have it | 14:03 |
| mrunge | ykarel, and yes, it would have been easier, if we could bump release | 14:03 |
| oidgar | mwhahaha: missed this mail somehow :) | 14:03 |
| jrist | mwhahaha: :) just got up | 14:03 |
| ykarel | mrunge, yes < 4.1.3 should fix it | 14:03 |
| trown|rover | mrunge: in general, we do care very much about trunk, all of our gates are using trunk | 14:04 |
| *** nyechiel_ has joined #tripleo | 14:04 | |
| *** nyechiel has quit IRC | 14:05 | |
| *** zaneb has joined #tripleo | 14:05 | |
| mrunge | ykarel, https://review.rdoproject.org/r/#/c/11110/3 | 14:05 |
| mfedosin | trown|rover: I use master-tripleo-ci | 14:05 |
| *** eck`gone is now known as eck` | 14:06 | |
| ykarel | mrunge, yeah it's being handled in the patch | 14:07 |
| trown|rover | ykarel: mrunge, I think that patch needs to be < 4.1.3 also though | 14:07 |
| trown|rover | instead of 4.1.<date> | 14:07 |
| trown|rover | err 4.1.2-0.<date> | 14:08 |
| mrunge | trown|rover, ykarel please comment on that patch | 14:08 |
| trown|rover | mrunge: will do | 14:08 |
| mrunge | that may be helpful | 14:08 |
| mrunge | although gnocchi was removed from rdo | 14:08 |
| mrunge | iirc | 14:09 |
| mhenkel | shardy: I found the 'offender': when the container is not yet running, paunch gets stuck: | 14:09 |
| mhenkel | "/usr/bin/python2 /usr/bin/paunch --debug apply --file /var/lib/tripleo-config/hashed-docker-container-startup-config-step_4.json --config-id tripleo_step4 --managed-by tripleo-ContrailController" | 14:09 |
| mrunge | no idea, when it get updated or pulled from its new location | 14:09 |
| mhenkel | it works if the container is up | 14:09 |
| *** raildo has joined #tripleo | 14:10 | |
| mhenkel | and here we go: "detach": false, | 14:10 |
| *** Goneri has joined #tripleo | 14:10 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 14:10 |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 14:11 |
| *** ooolpbot has quit IRC | 14:11 | |
| ykarel | mrunge, gnochhi is still build in rdo: https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L1692-L1715 | 14:11 |
| *** pblaho has quit IRC | 14:11 | |
| *** pblaho has joined #tripleo | 14:11 | |
| openstackgerrit | Or Idgar proposed openstack/tripleo-heat-templates master: WIP Filling certificate hieradata for Octavia when TLS everywhere is enabled https://review.openstack.org/530641 | 14:11 |
| mrunge | ykarel, I couldn't find it anymore here https://trunk.rdoproject.org/centos7-master/report.html | 14:12 |
| ykarel | mrunge, https://trunk.rdoproject.org/api-centos-master-uc/api/report.html?package=openstack-gnocchi | 14:12 |
| ykarel | yes last build was on 2017-12-09 14:27:57 | 14:13 |
| mrunge | ykarel, no, the build was on dec 22, the commit on dec 09 | 14:13 |
| mrunge | why is there that lag? | 14:13 |
| mwhahaha | promotions | 14:14 |
| mrunge | I see | 14:14 |
| mwhahaha | also we're probably not syncing commits from the new repo | 14:14 |
| mwhahaha | since you moved out of the openstack world, it isnt' getting notified on new comments | 14:14 |
| mwhahaha | should ask #rdo | 14:14 |
| ykarel | mrunge, because it's pinned: https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L1697 | 14:15 |
| *** anilvenkata has joined #tripleo | 14:15 | |
| mrunge | aha, that makes sense then, ykarel | 14:15 |
| mrunge | there were contributions *today*, but no builds | 14:15 |
| jtomasek | o/ | 14:16 |
| ykarel | mrunge, yes, new commits are only build if the project is not pinned | 14:17 |
| mwhahaha | there haven't been any new contributions to the stable/4.1 branch since dec 9 | 14:17 |
| mwhahaha | so we'd need to unpin queens | 14:17 |
| *** pkovar has quit IRC | 14:17 | |
| shardy | mhenkel: aha, good to hear you found the issue, perhaps we should add a validation to paunch to catch that and fail | 14:19 |
| *** pkovar has joined #tripleo | 14:19 | |
| mhenkel | yes, a check would be nice, on the other side... it was quite obvious. I don't know why I had that line in my service definition in the first instance... I copied it from some example | 14:20 |
| *** numans has quit IRC | 14:22 | |
| ykarel | mwhahaha, ack queens should be unpinned to catch failures from new commits if any | 14:22 |
| mwhahaha | i'm sure it'll totally be broken | 14:22 |
| mwhahaha | but yea | 14:22 |
| *** numans has joined #tripleo | 14:23 | |
| *** jobewan has joined #tripleo | 14:24 | |
| *** pgadiya has quit IRC | 14:24 | |
| arxcruz | o/ | 14:27 |
| *** moshele has quit IRC | 14:29 | |
| *** lblanchard has joined #tripleo | 14:32 | |
| *** moshele has joined #tripleo | 14:33 | |
| *** psachin has quit IRC | 14:37 | |
| *** bkopilov has joined #tripleo | 14:39 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add incoming storage driver param https://review.openstack.org/524248 | 14:42 |
| *** trozet has joined #tripleo | 14:45 | |
| *** pall is now known as pabelanger | 14:45 | |
| *** pchavva has joined #tripleo | 14:52 | |
| *** jmelvin has quit IRC | 14:52 | |
| openstackgerrit | Joe Talerico proposed openstack/tripleo-common master: Lowering the number of ansible forks to 25 https://review.openstack.org/530752 | 14:55 |
| *** salmankhan has quit IRC | 14:56 | |
| *** nyechiel_ has quit IRC | 14:56 | |
| *** dsariel has quit IRC | 14:57 | |
| openstackgerrit | Pradeep Kilambi proposed openstack/puppet-tripleo master: Add incoming storage driver param https://review.openstack.org/524240 | 14:58 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 15:00 | |
| *** jobewan has quit IRC | 15:02 | |
| alee | EmilienM, mwhahaha - happy new year! | 15:03 |
| alee | EmilienM, mwhahaha https://review.openstack.org/#/c/527515/ when you get a chance please. | 15:04 |
| mwhahaha | alee: done | 15:04 |
| alee | mwhahaha, thanks :) | 15:04 |
| EmilienM | hello | 15:05 |
| openstackgerrit | Pradeep Kilambi proposed openstack/tripleo-heat-templates master: Add S3 storage driver as an option and its config https://review.openstack.org/521093 | 15:05 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 15:10 |
| ykarel | EmilienM, hi | 15:11 |
| ykarel | EmilienM, are you aware of: Could not autoload puppet/provider/archive/curl: Could not find parent provider ruby of curl. | 15:12 |
| *** moshele has quit IRC | 15:13 | |
| ykarel | encountered in pike phase 1 poi jobs: https://trello.com/c/NAIs0IKr/330-poi-all-scenarios-1st-puppet-run-failing-with-error-could-not-autoload-puppet-provider-archive-curl-could-not-find-parent-provid | 15:13 |
| *** moshele has joined #tripleo | 15:15 | |
| EmilienM | ykarel: I'm not | 15:18 |
| *** mathiasb_ is now known as mathiasb | 15:18 | |
| ykarel | EmilienM, Ok, who can assist in that | 15:18 |
| *** cshastri has joined #tripleo | 15:20 | |
| *** jobewan has joined #tripleo | 15:20 | |
| EmilienM | ykarel: what's the puppet error? have you read logs? | 15:20 |
| ykarel | EmilienM, i debugged it, and after creating /etc/puppet/environments directory it got fixed | 15:21 |
| EmilienM | I know what it is | 15:22 |
| EmilienM | Could not autoload puppet/type/archive | 15:22 |
| ykarel | yes | 15:22 |
| EmilienM | it's a depedency issue on the rpm | 15:22 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-common master: Poll heat events and send it to messaging queue https://review.openstack.org/529567 | 15:22 |
| EmilienM | you need puppet-archives | 15:22 |
| ykarel | EmilienM, puppet-archive is installed | 15:22 |
| EmilienM | the good version? | 15:23 |
| ykarel | https://centos.logs.rdoproject.org/weirdo-pike-promote-puppet-openstack-scenario001/105/weirdo-project/logs/rpm-qa.txt.gz | 15:23 |
| *** dsariel has joined #tripleo | 15:23 | |
| ykarel | puppet-archive-2.2.0-0.20171221130716.cf9570c.el7.centos.noarch | 15:23 |
| *** cshastri has quit IRC | 15:26 | |
| EmilienM | ykarel: the version of puppet-rabbitmq is maybe too old, let me check | 15:26 |
| EmilienM | yeah, see rdoinfo | 15:27 |
| EmilienM | ykarel: https://github.com/redhat-openstack/rdoinfo/blob/master/rdo.yml#L809-L810 I think | 15:27 |
| EmilienM | ykarel: we should deploy 0ec48b31d4ac2b0dda44ace71abc9b8b6f5f4c66 | 15:27 |
| *** jmelvin has joined #tripleo | 15:28 | |
| ykarel | EmilienM, but packages are installed from trunk repo, so 0ec48b31d4ac2b0dda44ace71abc9b8b6f5f4c66 should be installed | 15:30 |
| arxcruz | EmilienM: http://logs.openstack.org/03/525603/8/check-tripleo/tripleo-ci-centos-7-ovb-ha-oooq-ipv6/df6ff62/logs/stackviz/#/stdin | 15:31 |
| arxcruz | ;) stackviz working | 15:31 |
| ykarel | EmilienM, puppet-rabbitmq-8.0.1-0.20171213201209.0ec48b3.el7.centos.noarch is installed | 15:31 |
| *** ecerquei has quit IRC | 15:32 | |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz data https://review.openstack.org/525603 | 15:33 |
| *** alee is now known as alee_afk | 15:33 | |
| *** psahoo has quit IRC | 15:34 | |
| EmilienM | arxcruz: good news! | 15:35 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Update containers when the overcloud is containerized https://review.openstack.org/529399 | 15:35 |
| mwhahaha | slagle: are we actually using the local collector for os-collect-config somewhere? like for deployed server or anything? | 15:38 |
| dalvarez | EmilienM, ping | 15:40 |
| *** cylopez has quit IRC | 15:40 | |
| EmilienM | dalvarez: what's up | 15:41 |
| slagle | mwhahaha: yes | 15:41 |
| dalvarez | EmilienM, heya! regarding https://bugs.launchpad.net/tripleo/+bug/1740698 i understood that this wasn't a voting job as per: https://review.rdoproject.org/r/#/c/10795/ | 15:41 |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 15:41 |
| *** pkovar has quit IRC | 15:41 | |
| mwhahaha | slagle: ok thought we might | 15:41 |
| EmilienM | dalvarez: it was not in the gate, it was in promotion jobs, where OVN is part of it (scenario007, fs030) | 15:42 |
| dalvarez | EmilienM, from the rdo project patch i understood that it was non voting for promotion "[...] Note this patch is not adding this job as voting for promotion" | 15:42 |
| dalvarez | EmilienM, but i was checking in ci-config repo and i couldn't find it either | 15:43 |
| EmilienM | dalvarez: scenario007 is part of promotion pipeline | 15:43 |
| dalvarez | im not familiar with this though, trying to understand it because i thought that scenario007 wouldn't still block promotion as i understood from the commit message | 15:44 |
| dalvarez | EmilienM, ah ok that explains it all, how could i check that it's part of promotion pipeline and voting? | 15:44 |
| *** pkovar has joined #tripleo | 15:44 | |
| EmilienM | dalvarez: if you revert https://review.openstack.org/#/c/530593/ , scenario007 should fail, and you would see the error | 15:45 |
| EmilienM | dalvarez: since we had a promotion, so latest networking-ovn | 15:45 |
| *** agurenko has quit IRC | 15:45 | |
| *** bnemec-xmas is now known as bnemec | 15:46 | |
| openstackgerrit | Alex Schultz proposed openstack/os-collect-config master: Lower local collector log level https://review.openstack.org/530759 | 15:46 |
| dalvarez | EmilienM, got it then the last port of the commit message in https://review.rdoproject.org/r/#/c/10795/ is misleading to me since i would've said that it won't affect the RDO promotion | 15:46 |
| *** liverpooler has joined #tripleo | 15:46 | |
| dalvarez | s/port/part | 15:47 |
| *** nyechiel_ has joined #tripleo | 15:52 | |
| *** marrusl has joined #tripleo | 15:53 | |
| *** oidgar has quit IRC | 16:00 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 16:01 | |
| *** ecerquei has joined #tripleo | 16:05 | |
| *** jobewan has quit IRC | 16:06 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 16:10 |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| *** jobewan has joined #tripleo | 16:10 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs032: activate config-download https://review.openstack.org/528761 | 16:11 |
| *** ecerquei_ has joined #tripleo | 16:12 | |
| shardy | Hmm anyone know why cistatus.tripleo.org and http://tripleo.org/cistatus.html don't contain data for the last few days? | 16:12 |
| shardy | I'm trying to figure out if tripleo-ci-centos-7-containers-multinode is broken everywhere, but it's hard to get an overview without those pages updating | 16:13 |
| mwhahaha | zuul has a dashboard now | 16:13 |
| mwhahaha | which makes that easier to see | 16:13 |
| mwhahaha | http://zuulv3.openstack.org/builds.html | 16:14 |
| mwhahaha | it's broken | 16:14 |
| *** chlong has joined #tripleo | 16:15 | |
| mwhahaha | no passes since 2018-01-01T02:25:29 | 16:15 |
| *** ecerquei has quit IRC | 16:15 | |
| *** oidgar has joined #tripleo | 16:15 | |
| shardy | mwhahaha: ack Ok I'm looking into it - we landed some tripleo-ansible-inventory changes which may be the cause | 16:15 |
| shardy | although those landed earlier than that ... | 16:15 |
| * shardy looks at logs | 16:15 | |
| mwhahaha | http://logs.openstack.org/90/529590/4/check/tripleo-ci-centos-7-containers-multinode/0132355/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-01-02_13_33_36 | 16:16 |
| mwhahaha | yea the AnsibleGEnerateInventoryAction | 16:16 |
| mwhahaha | seems bad | 16:16 |
| mwhahaha | execve() arg 3 contains a non-string value | 16:17 |
| shardy | mwhahaha: yeah the changes to the inventory generation seem to have broken the action | 16:17 |
| *** alee_afk is now known as alee | 16:18 | |
| EmilienM | trown|rover: I'm trying to enable config-download on fs032 (3nodes-multinode) but it failes because multinode_args overrides all parameters from config/general_config/featureset-multinode-common.yml - Any good reason to keep using multinode_args ? IMHO we should try to use proper parameters | 16:19 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: Clean trailing whitespace in HciCeph roles https://review.openstack.org/530764 | 16:19 |
| openstackgerrit | Merged openstack/tripleo-docs master: RHEL: Updates for 7.4 https://review.openstack.org/530386 | 16:21 |
| trown|rover | EmilienM: what do you mean by "proper" | 16:21 |
| *** rlandy|ruck is now known as rlandy|ruck|mtg | 16:21 | |
| EmilienM | trown|rover: something that doesn't override multinode-common.yaml | 16:21 |
| weshay | EmilienM, happy new year sir | 16:22 |
| EmilienM | hey happy new year too! | 16:22 |
| trown|rover | EmilienM: I am confused ... what about multinode_args is overriding that common config (which is only things for all multinode jobs btw)? | 16:25 |
| EmilienM | trown|rover: "deploy_multinode" boolean for example | 16:26 |
| EmilienM | trown|rover: default is false, so for this job it became false when we override the multinode_args | 16:26 |
| trown|rover | EmilienM: I am still lost... https://github.com/openstack/tripleo-quickstart-extras/blob/9449198547dcdd4cf15d6a7a7e8671becb055d2f/roles/overcloud-deploy/tasks/pre-deploy.yml is where we do stuff with multinode_args | 16:27 |
| EmilienM | trown|rover: yes but now we miss https://github.com/openstack/tripleo-quickstart-extras/blob/9449198547dcdd4cf15d6a7a7e8671becb055d2f/roles/overcloud-deploy/tasks/pre-deploy.yml#L210 | 16:27 |
| EmilienM | because deploy_multinode is false | 16:28 |
| EmilienM | I have a fix incoming | 16:28 |
| ccamacho | EmilienM weshay do you have something quick to deploy OOOQ with containers and HA? I had before .quickstart/config/general_config/containers_minimal.yml but it does not exists anymore | 16:28 |
| weshay | hrm.. that should still exist | 16:28 |
| * weshay looks | 16:28 | |
| ccamacho | wuuppps | 16:29 |
| weshay | ccamacho, pull the latest tqe source your virtenv and run python setup.py install | 16:29 |
| trown|rover | EmilienM: i dont think there is anything going on with overrides... just that https://github.com/openstack/tripleo-quickstart/blob/cbeb7b84f064d5b40ea9332d2eff6dd3b8847a8e/config/general_config/featureset032.yml#L24 is not using the multinode common config | 16:30 |
| ccamacho | oki so its in quickstart extras thanks weshay | 16:30 |
| *** salmankhan has joined #tripleo | 16:31 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs032: switch to extra_args https://review.openstack.org/530767 | 16:31 |
| EmilienM | trown|rover: ^ let me know if it's clear enough | 16:31 |
| trown|rover | EmilienM: ya I understand your issue now... just not your description of it :P | 16:32 |
| *** dsariel has quit IRC | 16:32 | |
| trown|rover | EmilienM: that may work... not sure though, the order of those arguments is really tricky | 16:32 |
| EmilienM | trown|rover: sorry for my confusion, hard to explain :) | 16:32 |
| EmilienM | trown|rover: yeah I'll make sure the CI job pass well | 16:33 |
| trown|rover | EmilienM: the issue is that featureset032 is an oddball and doesnt use the multinode common config | 16:33 |
| EmilienM | trown|rover: why? | 16:33 |
| EmilienM | it makes things more complex to re-use | 16:33 |
| *** dsariel has joined #tripleo | 16:34 | |
| trown|rover | ya... i remember having issues with getting the arguments ordered right... which btw your patch will still not have deploy_multinode=true | 16:34 |
| EmilienM | trown|rover: why not? | 16:34 |
| trown|rover | EmilienM: because nothing is setting that... that is what I was trying to explain | 16:35 |
| EmilienM | trown|rover: yes, it does, in config/general_config/featureset-multinode-common.yml | 16:35 |
| EmilienM | trown|rover: which is loaded by tripleo-ci scripts | 16:36 |
| trown|rover | EmilienM: nothing about using multinode_args overrides multinode-common | 16:36 |
| EmilienM | but it was overridden by multinode_args I guess | 16:36 |
| trown|rover | EmilienM: so if it wasnt there when using featureset032 before... it wont be there with your patch... looking for where exactly featureset032 is excluded | 16:36 |
| EmilienM | trown|rover: we'll see soon, since https://review.openstack.org/#/c/526138/ landed | 16:37 |
| EmilienM | we can use that to tell variables now | 16:37 |
| EmilienM | weshay++ on that | 16:37 |
| *** dsariel has quit IRC | 16:38 | |
| *** oidgar has quit IRC | 16:38 | |
| trown|rover | EmilienM: ah ok... so it is using the common config... multinode_args set in that file was just overriding multinode_args via set_fact because config file is higher precedence... so I think just adding the "--overcloud-ssh-user {{ undercloud_user }}" to featureset032 would be better | 16:41 |
| *** pcaruana has quit IRC | 16:41 | |
| trown|rover | EmilienM: extra_args are really meant for users to be able to have a place to override | 16:41 |
| EmilienM | trown|rover: the problem is that we have some logic that runs when deploy_multinode is set to true, that we need for 3nodes jobs | 16:42 |
| EmilienM | so adding this argument will only solve our immediate need but not sure we won't miss any more tasks | 16:42 |
| trown|rover | EmilienM: but deploy_multinode is set to true | 16:42 |
| *** moshele has quit IRC | 16:43 | |
| *** ChanServ sets mode: +o mwhahaha | 16:43 | |
| EmilienM | so why the overcloud-ssh-user isn't set? | 16:43 |
| trown|rover | nothing about multinode_args overrides the setting of deploy_multinode | 16:43 |
| *** mwhahaha changes topic to "CI status: RED (containers-multinode is broken), see alerts and read ML. | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 16:43 | |
| trown|rover | because precedence of set_fact | 16:43 |
| EmilienM | trown|rover: see http://logs.openstack.org/61/528761/6/check/tripleo-ci-centos-7-3nodes-multinode/3006eb7/job-output.txt.gz#_2017-12-20_01_18_23_052029 | 16:45 |
| trown|rover | EmilienM: it doesnt merge multinode_args from config with those set when deploy_multinode is true... it is just if those are set in config, they do not get overridden | 16:45 |
| EmilienM | featureset-multinode-common.yml is loaded | 16:45 |
| trown|rover | EmilienM: right | 16:46 |
| mwhahaha | shardy: fyi i created a bug for the containers-multinode https://bugs.launchpad.net/tripleo/+bug/1740891 | 16:47 |
| openstack | Launchpad bug 1740891 in tripleo "deploy fails with execve() arg 3 contains a non-string value" [Critical,Triaged] | 16:47 |
| EmilienM | trown|rover: ok I see now | 16:47 |
| EmilienM | trown|rover: I'll just add --overcloud-ssh-user then | 16:47 |
| *** yprokule has quit IRC | 16:47 | |
| EmilienM | trown|rover: it's just weird that we have to maintain the list of arguments for this job | 16:48 |
| EmilienM | trown|rover: if you see what I mean | 16:48 |
| shardy | mwhahaha: ack thanks, I'll take it - unable to reproduce locally atm but will keep digging | 16:48 |
| trown|rover | EmilienM: ya I think that should solve the issue... | 16:48 |
| trown|rover | EmilienM: ya... that job uses different args then other multinode though | 16:48 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Add missing --overcloud-ssh-user parameter https://review.openstack.org/530767 | 16:50 |
| EmilienM | trown|rover: fair enough | 16:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs032: activate config-download https://review.openstack.org/528761 | 16:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: DNM - ssh-keyscan second subnode https://review.openstack.org/528802 | 16:50 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs032: activate config-download https://review.openstack.org/528761 | 16:51 |
| *** oidgar has joined #tripleo | 16:52 | |
| shardy | Hrm actually mistral run-action tripleo.ansible-generate-inventory does reproduce it, tripleo-ansible-inventory works fine manually tho | 16:53 |
| mwhahaha | shardy: i bet we're passing a number rather than a string because heat | 16:53 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Update the doc link https://review.openstack.org/518297 | 16:54 |
| shardy | mwhahaha: quite possibly, it looks like arg3 is the environment dict tho? | 16:54 |
| shardy | it doesn't look like oslo processutils changed recently tho, nor eventlet | 16:54 |
| mwhahaha | *command | 16:54 |
| mwhahaha | though it could be the environment | 16:54 |
| shardy | https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/ansible.py#L550 | 16:55 |
| *** udesale has quit IRC | 16:55 | |
| shardy | I'll dig more now I've got a reproducer | 16:55 |
| mwhahaha | shardy: what's inventory_path | 16:55 |
| mwhahaha | oh nm | 16:55 |
| * mwhahaha bothers to read 4 lines up | 16:56 | |
| shardy | mwhahaha: yeah we create it from work_dir, args are params='{u'ansible_ssh_user': u'tripleo-admin', u'work_dir': u'/var/lib/mistral/5ebb0ccf-67b4-4666-addf-f31f0c523366'} | 16:57 |
| shardy | doesn't seem to matter tho, it fails with empty params | 16:57 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/pike: Allow to increase docker daemon verbosity https://review.openstack.org/530773 | 16:57 |
| mwhahaha | shardy: where does extra_env_variables come from | 16:58 |
| *** dpawar has joined #tripleo | 16:58 | |
| *** dpawar has quit IRC | 16:58 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common master: Add UC backup Mistral workflow definition and actions https://review.openstack.org/517610 | 16:58 |
| *** ManoX has joined #tripleo | 16:59 | |
| *** dpawar has joined #tripleo | 16:59 | |
| shardy | mwhahaha: hmm, not sure, think that's only for running the playbook tho, not AnsibleGenerateInventoryAction? | 17:00 |
| *** nyechiel_ has quit IRC | 17:00 | |
| ManoX | hello, I am attempting to deploy an overcloud on an existing server following the steps here http://tripleo.org/install/advanced_deployment/deployed_server.html . Looks like the command `overcloud deploy` just hangs for a long time (couple of hours) at this stage `[overcloud.ControllerDeployedServer.0.UpdateConfig]: CREATE_COMPLETE state changed` | 17:03 |
| ManoX | . Is there a known issue around this? | 17:03 |
| *** chkumar246 has joined #tripleo | 17:06 | |
| mwhahaha | shardy: it's dying trying to execute the playbook | 17:06 |
| shardy | mwhahaha: Hmm, where do you see that? The executor logs show mistral.actions.action_factory.AnsibleGenerateInventoryAction is failing running the subprocess to run tripleo-ansible-inventory | 17:07 |
| mwhahaha | shardy: nm i was looking at the other env_varables in the same action file | 17:07 |
| * shardy may be missing something | 17:07 | |
| *** rlandy|ruck|mtg is now known as rlandy|ruck | 17:07 | |
| *** ecerquei_ has quit IRC | 17:08 | |
| shardy | initially I thought it was something I broke in the inventory script, but it seems to be something else | 17:08 |
| *** rwsu has joined #tripleo | 17:08 | |
| *** ecerquei has joined #tripleo | 17:08 | |
| *** ecerquei has quit IRC | 17:08 | |
| mwhahaha | shardy: i'm not really sure where the execve is coming from either | 17:08 |
| openstackgerrit | Carlos Camacho proposed openstack/python-tripleoclient master: Add Openstack Undercloud backup https://review.openstack.org/466213 | 17:08 |
| shardy | mwhahaha: yeah the actual error is kind of obscured by the python subprocess error path | 17:09 |
| shardy | attempting to trace back from the raise atm | 17:09 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740891 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 17:10 |
| openstack | Launchpad bug 1740891 in tripleo "deploy fails with execve() arg 3 contains a non-string value" [Critical,Triaged] - Assigned to Steven Hardy (shardy) | 17:10 |
| *** mcornea has quit IRC | 17:10 | |
| mwhahaha | shardy: the only place i found a reference to execve was in eventlet/green/http/server.py | 17:11 |
| mwhahaha | and arg 3 is still env | 17:11 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: WIP - deploy-overcloud/config-download: up to 4 nodes support https://review.openstack.org/530775 | 17:12 |
| mwhahaha | shardy: could it be that one of the env_variables from the context is bad? | 17:12 |
| mwhahaha | https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/ansible.py#L537-L542 | 17:12 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: WIP - deploy-overcloud/config-download: up to 4 nodes support https://review.openstack.org/530775 | 17:13 |
| shardy | mwhahaha: yeah it looks like the issue is some values are None, adding or '' to all of them in the action gets to a new error | 17:13 |
| *** ecerquei has joined #tripleo | 17:14 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Fix puppet config volume for iscsid in containers https://review.openstack.org/524187 | 17:14 |
| shardy | mwhahaha: now the ansible inventory code is a library, we should probably just make the action use it, but it'd be good to figure out why this broke anyway | 17:15 |
| shardy | 'OS_USERNAME': None, 'OS_PROJECT_NAME': None | 17:15 |
| shardy | that seems to be what breaks it | 17:15 |
| mwhahaha | so we've lost the credentials somewhere | 17:16 |
| mwhahaha | this might just be a mistral regression since this seems to have failed around the time of the last promotion | 17:16 |
| *** rmascena has joined #tripleo | 17:16 | |
| shardy | yeah seems like it | 17:16 |
| shardy | d0ugal, thrash: ^^ | 17:16 |
| *** raildo has quit IRC | 17:18 | |
| *** d0ugal has quit IRC | 17:19 | |
| *** ccamacho1 has joined #tripleo | 17:20 | |
| *** ccamacho has quit IRC | 17:20 | |
| mwhahaha | https://github.com/openstack/mistral/compare/f1e2385...33babdf | 17:20 |
| mwhahaha | from previous known good package to first failure | 17:20 |
| *** d0ugal has joined #tripleo | 17:20 | |
| mwhahaha | which happened from the 20171229 package | 17:20 |
| *** oidgar has quit IRC | 17:20 | |
| mwhahaha | https://github.com/openstack/mistral/commit/dd4a4bd440ba3fc14f0da782b99d64e7b338a1e7 perhaps? | 17:22 |
| *** karthiks has quit IRC | 17:22 | |
| *** jpich has quit IRC | 17:23 | |
| shardy | mwhahaha: yeah I think that missed the project and user name | 17:25 |
| shardy | adding them and re-trying | 17:25 |
| shardy | I guess we're missing these jobs from the promotion pipeline :( | 17:25 |
| *** lucasagomes is now known as lucas-afk | 17:25 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-common master: Add UC backup Mistral workflow definition and actions https://review.openstack.org/517610 | 17:26 |
| *** pkovar has quit IRC | 17:28 | |
| shardy | https://github.com/openstack/mistral-lib/blob/master/mistral_lib/actions/context.py#L30 | 17:28 |
| shardy | Ok so by deprecated, they mean set to None ;) | 17:28 |
| *** ccamacho1 has quit IRC | 17:29 | |
| *** fragatina has joined #tripleo | 17:29 | |
| mwhahaha | which totally screws with execve? :D | 17:29 |
| mwhahaha | perhaps setting to ''? | 17:29 |
| weshay | shardy, containers-multinode? | 17:29 |
| mwhahaha | weshay: yea | 17:30 |
| shardy | That doesn't help because we need the data, not None | 17:30 |
| shardy | weshay: yeah I think I found the issue, testing a fix now | 17:30 |
| shardy | tl;dr mistral broke us | 17:30 |
| mwhahaha | bad mistral, bad. | 17:30 |
| shardy | which is better than me breaking us I guess, which is what I initially suspected ;) | 17:30 |
| *** fragatin_ has joined #tripleo | 17:31 | |
| *** bogdando has quit IRC | 17:32 | |
| *** ykarel has quit IRC | 17:33 | |
| *** fragatina has quit IRC | 17:34 | |
| *** Guest87 has quit IRC | 17:35 | |
| *** salmankhan has quit IRC | 17:36 | |
| *** gkadam_ has joined #tripleo | 17:36 | |
| *** salmankhan has joined #tripleo | 17:38 | |
| weshay | shardy, mwhahaha rlandy|ruck trown|rover fyi https://review.rdoproject.org/r/#/c/11138/ | 17:41 |
| openstackgerrit | Mike Fedosin proposed openstack/tripleo-heat-templates master: Check openshift-ansible directory exists https://review.openstack.org/530484 | 17:41 |
| shardy | weshay: nice, thanks! :) | 17:42 |
| *** mdnadeem has quit IRC | 17:42 | |
| openstackgerrit | Juan Badia Payno proposed openstack/puppet-tripleo master: Fluentd: Transform path by service https://review.openstack.org/530780 | 17:44 |
| *** trown|rover is now known as trown|lunch | 17:48 | |
| *** catintheroof has joined #tripleo | 17:51 | |
| *** ecerquei is now known as ecerquei|mtg | 17:56 | |
| *** derekh has quit IRC | 17:59 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common master: Update AnsibleGenerateInventoryAction to match new mistral interfaces https://review.openstack.org/530783 | 18:00 |
| shardy | mwhahaha, jaosorior, weshay: ^^ | 18:00 |
| shardy | that fixes it locally (with the related mistral patch), lets see what CI thinks | 18:01 |
| mwhahaha | k | 18:01 |
| shardy | I'll also look at porting the action to the TripleOInventory class as a follow-up | 18:01 |
| *** dsneddon has joined #tripleo | 18:04 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 18:10 |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740891 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1740891 in tripleo "project_name and user_name no longer set in mistral action context" [Critical,In progress] - Assigned to Steven Hardy (shardy) | 18:10 |
| *** oidgar has joined #tripleo | 18:23 | |
| *** agopi has joined #tripleo | 18:26 | |
| *** fragatin_ has quit IRC | 18:31 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-common master: Update AnsibleGenerateInventoryAction to match new mistral interfaces https://review.openstack.org/530783 | 18:38 |
| *** jkilpatr has quit IRC | 18:46 | |
| *** dpawar has quit IRC | 18:46 | |
| *** dpawar has joined #tripleo | 18:47 | |
| *** trown|lunch is now known as trown | 18:49 | |
| *** trown is now known as trown|rover | 18:49 | |
| *** dpawar has quit IRC | 18:51 | |
| *** tesseract has quit IRC | 18:59 | |
| *** jkilpatr has joined #tripleo | 19:00 | |
| *** rwsu has quit IRC | 19:09 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 19:10 |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740891 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1740891 in tripleo "project_name and user_name no longer set in mistral action context" [Critical,In progress] - Assigned to Steven Hardy (shardy) | 19:10 |
| *** rwsu has joined #tripleo | 19:13 | |
| *** lebauce_ has joined #tripleo | 19:14 | |
| *** lebauce has quit IRC | 19:14 | |
| *** gfidente has quit IRC | 19:15 | |
| *** ecerquei|mtg has quit IRC | 19:25 | |
| *** dbecker has quit IRC | 19:25 | |
| *** ecerquei has joined #tripleo | 19:25 | |
| *** anilvenkata has quit IRC | 19:26 | |
| jkilpatr | is anyone on this BZ? https://bugs.launchpad.net/tripleo/+bug/1649350 | 19:28 |
| openstack | Launchpad bug 1649350 in tripleo "Introspection fails with 'mistral engine error' on Ocata if an 'ironic node-update' is done before introspection runs " [High,In progress] - Assigned to Adriano Petrich (apetrich) | 19:28 |
| jkilpatr | it's disrupting my Pike testing as well | 19:28 |
| jkilpatr | I would really rather not remove installation essential features from my CI jobs because they never work... that's not a great sign. | 19:29 |
| *** moshele has joined #tripleo | 19:34 | |
| *** salmankhan has quit IRC | 19:42 | |
| EmilienM | slagle, trown|rover: please review https://review.openstack.org/#/c/530767/ (I did recheck for an unrelated issue) | 19:45 |
| openstackgerrit | Luke Short proposed openstack/tripleo-quickstart master: Support the Fedora operating system https://review.openstack.org/530793 | 19:46 |
| *** jmelvin has quit IRC | 19:49 | |
| *** jmelvin has joined #tripleo | 19:54 | |
| *** moshele has quit IRC | 19:54 | |
| *** salmankhan has joined #tripleo | 19:55 | |
| weshay | apetrich, ping re: https://bugs.launchpad.net/tripleo/+bug/1649350 | 20:00 |
| openstack | Launchpad bug 1649350 in tripleo "Introspection fails with 'mistral engine error' on Ocata if an 'ironic node-update' is done before introspection runs " [High,In progress] - Assigned to Adriano Petrich (apetrich) | 20:00 |
| *** d0ugal has quit IRC | 20:00 | |
| apetrich | weshay, today is a holiday in Scotland. I | 20:01 |
| apetrich | weshay, I'm back tomorrow morning to look at it | 20:01 |
| weshay | apetrich, aye.. sorry to disturb. I'll catch you tomorrow | 20:01 |
| weshay | :) | 20:01 |
| apetrich | no worries | 20:01 |
| apetrich | weshay, at least it won't be a boring morning of catching up with emails :) | 20:02 |
| *** shardy has quit IRC | 20:04 | |
| *** noslzzp has joined #tripleo | 20:06 | |
| *** bkopilov has quit IRC | 20:06 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740698 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740891 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1740698 in tripleo "OVN: tempest fails on floating-ip" [Critical,Triaged] - Assigned to Numan Siddique (numansiddique) | 20:10 |
| openstack | Launchpad bug 1740891 in tripleo "project_name and user_name no longer set in mistral action context" [Critical,In progress] - Assigned to Steven Hardy (shardy) | 20:10 |
| *** dbecker has joined #tripleo | 20:13 | |
| *** Goneri has quit IRC | 20:14 | |
| *** lblanchard has quit IRC | 20:16 | |
| slagle | EmilienM: i don't think you should be maintaining multinode_args in 2 places | 20:16 |
| slagle | EmilienM: it's what led to the bug in the first place | 20:16 |
| EmilienM | slagle: this is exactly what I said to trown | 20:17 |
| slagle | EmilienM: the 3nodes changes are additive to multinode_args | 20:17 |
| EmilienM | slagle: see my first commit https://review.openstack.org/#/c/530767/1/config/general_config/featureset032.yml | 20:17 |
| *** lblanchard has joined #tripleo | 20:18 | |
| slagle | EmilienM: yea, i guess that's more what I thought the fix would be. or a multinode_extra_args. (i don't know what else extra_args is used for) | 20:18 |
| *** d0ugal has joined #tripleo | 20:19 | |
| slagle | EmilienM: i'm only guessing at the intent, but it seems the set_fact in pre_deploy.yml was meant to be universal to all multinode jobs | 20:20 |
| slagle | EmilienM: and setting it in the featureset config causing that task to be ignored is unintentional | 20:20 |
| *** lblanchard has quit IRC | 20:21 | |
| EmilienM | slagle: true | 20:21 |
| EmilienM | trown|rover: what are your thoughts? Since you wrote that featureset | 20:22 |
| *** rfolco has quit IRC | 20:22 | |
| *** dbecker has quit IRC | 20:24 | |
| *** fragatina has joined #tripleo | 20:24 | |
| *** jkilpatr has quit IRC | 20:27 | |
| *** chlong has quit IRC | 20:30 | |
| *** yamahata has quit IRC | 20:35 | |
| *** jkilpatr has joined #tripleo | 20:41 | |
| *** chlong has joined #tripleo | 20:43 | |
| *** aputtur_ has quit IRC | 20:45 | |
| *** catintheroof has quit IRC | 20:46 | |
| *** catintheroof has joined #tripleo | 20:46 | |
| *** nyechiel_ has joined #tripleo | 20:48 | |
| *** catintheroof has quit IRC | 20:51 | |
| *** salmankhan has quit IRC | 20:56 | |
| *** fragatina has quit IRC | 21:00 | |
| *** jkilpatr has quit IRC | 21:04 | |
| *** salmankhan has joined #tripleo | 21:05 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Don't run check-tripleo OVB jobs frm RH1 anymore https://review.openstack.org/526481 | 21:06 |
| weshay | EmilienM, oh reh reh really | 21:09 |
| EmilienM | weshay: yeah | 21:09 |
| *** liverpooler has quit IRC | 21:09 | |
| weshay | nice | 21:09 |
| weshay | EmilienM, all my status pages are down atm | 21:09 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740891 | 21:10 |
| openstack | Launchpad bug 1740891 in tripleo "project_name and user_name no longer set in mistral action context" [Critical,In progress] - Assigned to Steven Hardy (shardy) | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| weshay | EmilienM, we should probably get sf to vote -2 on the jobs in sf first | 21:10 |
| weshay | ya? | 21:10 |
| weshay | or propose that | 21:10 |
| *** pchavva has quit IRC | 21:10 | |
| EmilienM | weshay: -2? | 21:11 |
| weshay | ya.. we discussed it at the last summit | 21:11 |
| weshay | getting ovb to be sudo gating in check | 21:11 |
| EmilienM | weshay: you mean verify vote power? | 21:11 |
| weshay | and upgrade jobs | 21:11 |
| weshay | ya | 21:11 |
| EmilienM | gotcha | 21:12 |
| EmilienM | yes we should get that | 21:12 |
| *** pradk has quit IRC | 21:16 | |
| *** agopi has quit IRC | 21:16 | |
| *** Guest87 has joined #tripleo | 21:21 | |
| *** Goneri has joined #tripleo | 21:22 | |
| *** Guest87 has quit IRC | 21:22 | |
| *** nyechiel_ has quit IRC | 21:25 | |
| *** fragatina has joined #tripleo | 21:31 | |
| *** fragatina has quit IRC | 21:32 | |
| *** fragatina has joined #tripleo | 21:32 | |
| *** jkilpatr has joined #tripleo | 21:33 | |
| *** oidgar has quit IRC | 21:38 | |
| *** jcoufal has quit IRC | 21:42 | |
| *** rcernin has joined #tripleo | 21:45 | |
| *** ecerquei has quit IRC | 21:52 | |
| *** trown|rover is now known as trown|outtypewww | 21:59 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Use skopeo for tag discover https://review.openstack.org/528945 | 22:08 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Move more prepare logic into kolla_builder https://review.openstack.org/526579 | 22:08 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Use push_destination as the registry host in env file https://review.openstack.org/528616 | 22:08 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Discover every tag on prepare https://review.openstack.org/529215 | 22:08 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Prepare action: extra arguments https://review.openstack.org/526580 | 22:08 |
| *** chlong has quit IRC | 22:09 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740891 | 22:10 |
| openstack | Launchpad bug 1740891 in tripleo "project_name and user_name no longer set in mistral action context" [Critical,In progress] - Assigned to Steven Hardy (shardy) | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| *** abishop has quit IRC | 22:25 | |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Move more prepare logic into kolla_builder https://review.openstack.org/526576 | 22:32 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Add --tag-from-label to prepare command https://review.openstack.org/529445 | 22:32 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Remove "Add subnodes to known hosts" task https://review.openstack.org/530808 | 22:33 |
| *** rmascena has quit IRC | 22:34 | |
| *** jobewan has quit IRC | 22:37 | |
| EmilienM | weshay: hey | 22:37 |
| EmilienM | weshay: I'm debugging slagle's patches to enable config-download on OVB jobs | 22:38 |
| EmilienM | weshay: and I found out that we don't have overcloud logs on https://logs.rdoproject.org/31/525331/20/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z593ef7fb57bc427ab175b7e677bf7743/ | 22:38 |
| weshay | EmilienM /me looks | 22:38 |
| weshay | EmilienM, qq is the error here w/ openstack-gnochi https://ci.centos.org/artifacts/rdo/jenkins-tripleo-quickstart-extras-gate-master-tripleo-ci-delorean-full-minimal_pacemaker-4554/172.19.2.113/home/stack/_home_stack_repo_setup.sh.log.gz resolved by https://review.rdoproject.org/r/#/c/11110/ | 22:39 |
| weshay | rlandy|ruck, chatting here now | 22:39 |
| EmilienM | weshay: ok | 22:39 |
| EmilienM | weshay: merging | 22:39 |
| EmilienM | weshay: if broken I'll revert again :P | 22:39 |
| EmilienM | slagle: have you seen that? https://logs.rdoproject.org/31/525331/20/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z593ef7fb57bc427ab175b7e677bf7743/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2018-01-02_22_23_09 | 22:40 |
| rlandy|ruck | weshay: thanks | 22:40 |
| EmilienM | slagle: this is the CI job from https://review.openstack.org/#/c/525331/ that runs in RDO CI, so without timeout | 22:40 |
| *** pmannidi has joined #tripleo | 22:41 | |
| openstackgerrit | Dan Sneddon proposed openstack/tripleo-heat-templates master: Render NIC config templates with jinja2 https://review.openstack.org/523638 | 22:41 |
| weshay | EmilienM, slagle ya.. we should have logs there especially for ovb | 22:41 |
| openstackgerrit | Merged openstack/instack stable/pike: Update .gitreview for stable/pike https://review.openstack.org/488636 | 22:43 |
| *** salmankhan has quit IRC | 22:44 | |
| *** salmankhan has joined #tripleo | 22:44 | |
| *** fragatina has quit IRC | 22:44 | |
| *** catintheroof has joined #tripleo | 22:46 | |
| weshay | EmilienM, slagle in this case.. for some reason the overcloud nodes where unreachable | 22:48 |
| weshay | EmilienM, slagle https://logs.rdoproject.org/31/525331/20/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z593ef7fb57bc427ab175b7e677bf7743/quickstart_collect_logs.txt.gz | 22:48 |
| EmilienM | weshay: weird, overcloud was deployed fine | 22:48 |
| weshay | EmilienM, look for Tuesday 02 January 2018 22:24:23 +0000 | 22:48 |
| EmilienM | well | 22:48 |
| EmilienM | not fine, finished, but it started | 22:49 |
| weshay | EmilienM, https://logs.rdoproject.org/31/525331/20/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z593ef7fb57bc427ab175b7e677bf7743/undercloud/home/jenkins/overcloud_deploy.log.txt.gz#_2018-01-02_22_23_09 | 22:50 |
| weshay | the deploy failed | 22:50 |
| EmilienM | right I'm debugging it now | 22:50 |
| weshay | Failed to run action [action_ex_id=cd13a4ae-abc0-4ae3-b0d4-0643764e0593, action_cls='<class 'mistral.actions.action_factory.AnsibleGenerateInventoryAction'>', attributes='{}', params='{u'ansible_ssh_user': u'tripleo-admin', u'work_dir': u'/var/lib/mistral/10bd82d1-c5b8-484c-982b-cc5412067dc4'}'] | 22:50 |
| weshay | 2018-01-02 22:23:09 | execve() arg 3 contains a non-string value | 22:50 |
| weshay | k | 22:50 |
| EmilienM | https://logs.rdoproject.org/31/525331/20/openstack-check/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master/Z593ef7fb57bc427ab175b7e677bf7743/undercloud/var/log/mistral/executor.log.txt.gz#_2018-01-02_22_23_08_496 | 22:52 |
| *** salmankhan has quit IRC | 22:53 | |
| slagle | EmilienM: that's just the issue as to why CI has been red all day | 22:57 |
| slagle | EmilienM: not related to config-download + ovb | 22:58 |
| slagle | we promoted a bad mistral | 22:58 |
| EmilienM | damn | 22:59 |
| EmilienM | slagle: https://bugs.launchpad.net/tripleo/+bug/1740891 ? | 22:59 |
| openstack | Launchpad bug 1740891 in tripleo "project_name and user_name no longer set in mistral action context" [Critical,In progress] - Assigned to Steven Hardy (shardy) | 22:59 |
| slagle | EmilienM: yes | 22:59 |
| EmilienM | damn ok | 22:59 |
| slagle | we should probably unpromote to be honest | 22:59 |
| openstackgerrit | Juan Badia Payno proposed openstack/puppet-tripleo master: Fluentd: Transform path by service https://review.openstack.org/530780 | 23:00 |
| *** eck` is now known as eck`gone | 23:00 | |
| *** jmelvin has quit IRC | 23:00 | |
| slagle | considering our gate is blocked until shardy's fixes merge, *and* we promote again | 23:00 |
| EmilienM | damn | 23:00 |
| slagle | but i don't know if unpromoting is possible | 23:00 |
| EmilienM | not easily I guess | 23:00 |
| EmilienM | how could we promote with the bug? | 23:01 |
| slagle | we should really only promote with the same set of CI jobs that also vote | 23:01 |
| slagle | b/c that i'd guess ^ | 23:01 |
| slagle | i guess containers-multinode is not a promotion job! | 23:01 |
| EmilienM | thrash: who can land https://review.openstack.org/#/c/530781/ ? | 23:01 |
| EmilienM | slagle: which is what we do | 23:01 |
| EmilienM | our promotion jobs are almost the sames | 23:01 |
| EmilienM | but yeah there is some diff | 23:01 |
| slagle | EmilienM: well then now i'm asking you :). how did we promote then? | 23:01 |
| EmilienM | and right, this job isn't. | 23:02 |
| slagle | OK! | 23:02 |
| slagle | so not the same :) | 23:02 |
| EmilienM | right.. i'm promoting a change | 23:02 |
| slagle | ok. we agree :) | 23:02 |
| EmilienM | weshay fixed it already | 23:03 |
| EmilienM | weshay: thx man | 23:04 |
| EmilienM | and trown|outtypewww as well | 23:04 |
| weshay | slagle, ya.. multinode-containers unfortunately was not in the promote critieria | 23:04 |
| weshay | that was a miss | 23:04 |
| weshay | sorry about that | 23:04 |
| weshay | limited resources, choices were made that I no longer have the context for | 23:06 |
| slagle | it's all good | 23:08 |
| *** jobewan has joined #tripleo | 23:08 | |
| EmilienM | weshay: another way is to carry the patch in the mistral-distgit | 23:08 |
| EmilienM | it's ugly but would unblock us | 23:08 |
| slagle | i was just raising the idea of them being the same...i'm not sure of the context either | 23:08 |
| EmilienM | another option is to disable voting on this job | 23:08 |
| EmilienM | another option is to disable config-download on this job | 23:08 |
| slagle | EmilienM: No to both | 23:08 |
| EmilienM | the last option is to wait | 23:08 |
| EmilienM | I vote for #1 | 23:08 |
| slagle | we can't keep moving forward by disabling voting. that doesnt fix anything | 23:09 |
| EmilienM | yeah I know | 23:09 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1740891 | 23:10 |
| openstack | Launchpad bug 1740891 in tripleo "project_name and user_name no longer set in mistral action context" [Critical,In progress] - Assigned to Steven Hardy (shardy) | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| *** gkadam has joined #tripleo | 23:16 | |
| *** gkadam_ has quit IRC | 23:16 | |
| *** jtomasek has quit IRC | 23:17 | |
| *** yamahata has joined #tripleo | 23:18 | |
| EmilienM | slagle, weshay: I think we have a chicken&egg problem for https://review.rdoproject.org/r/#/c/11140/ | 23:19 |
| EmilienM | because to land https://review.openstack.org/#/c/530783/ we need the patch to be merged and promoted | 23:19 |
| EmilienM | but to promote we'll probably need https://review.openstack.org/#/c/530783/ first | 23:19 |
| EmilienM | unless we remove fs010 from promotion pipeline, promote, and then we can land https://review.openstack.org/#/c/530783/ | 23:21 |
| EmilienM | but fs010 will never pass the promotion jobs without both iiuc | 23:21 |
| EmilienM | unless we deploy mistral from current | 23:26 |
| EmilienM | temporarly | 23:26 |
| EmilienM | nevermind ^ we need to build new containers anyway | 23:28 |
| *** bfournie has quit IRC | 23:31 | |
| EmilienM | tbh, I'm not sure what's best | 23:31 |
| EmilienM | what I know is, it's a mess | 23:31 |
| mwhahaha | so we used to exclude mistral to deploy current | 23:31 |
| mwhahaha | but that doesn't work with containers | 23:32 |
| mwhahaha | thanks containers | 23:32 |
| EmilienM | I think what we should do: | 23:32 |
| EmilienM | 1) Merge https://review.rdoproject.org/r/#/c/11140/ | 23:32 |
| EmilienM | 2) Disable fs010 in promotion pipeline (revert trown + weshay patches) | 23:32 |
| EmilienM | 3) Get promotion | 23:32 |
| EmilienM | 4) Land https://review.openstack.org/#/c/530781/ | 23:32 |
| EmilienM | err | 23:32 |
| EmilienM | 4) https://review.openstack.org/#/c/530783 | 23:32 |
| EmilienM | 5) Revert https://review.rdoproject.org/r/#/c/11140/ and re-activate fs010 in promotion | 23:33 |
| mwhahaha | are there currently any other promotion blockers? | 23:33 |
| EmilienM | yeah good question | 23:33 |
| EmilienM | we just merged a gnocchi stuff, I hope it won't cause issues | 23:33 |
| EmilienM | weshay: any other blockers so far? | 23:33 |
| weshay | rlandy|ruck, should have all the blockers in a list /me gets what I know of | 23:34 |
| weshay | EmilienM, in master | 23:35 |
| weshay | fs10 and fs16 are failing | 23:35 |
| * weshay looking at why | 23:35 | |
| mwhahaha | 10's failing because of this problem | 23:35 |
| EmilienM | fs10, normal | 23:35 |
| EmilienM | fs16 not sure | 23:35 |
| EmilienM | logs? | 23:35 |
| weshay | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset016-master/e52d4cf/ | 23:36 |
| EmilienM | which is scenario001-multinode-containers btw | 23:36 |
| EmilienM | ok looking | 23:36 |
| EmilienM | 2018-01-02 23:08:15 | 2018-01-02 23:08:09Z [overcloud]: CREATE_FAILED Timed out | 23:36 |
| weshay | DeployedServerBootstrapConfig: Stack CREATE cancelled | 23:36 |
| mwhahaha | that reminds me of our swift/heat convergance issue | 23:37 |
| *** jobewan has quit IRC | 23:37 | |
| EmilienM | let's hope it was random | 23:38 |
| * mwhahaha does not look forwards to tracking that one down again | 23:38 | |
| EmilienM | weshay, mwhahaha : you ok with the plan? | 23:38 |
| weshay | I see a couple of neutron errors in that job | 23:38 |
| EmilienM | if not, propose something else | 23:38 |
| weshay | EmilienM, let's just remove fs10 from the promotion criteria | 23:38 |
| weshay | the job can keep running | 23:38 |
| EmilienM | if yes, please revert your patches | 23:38 |
| EmilienM | weshay: oh, let me do the revert, you can land theml | 23:39 |
| mwhahaha | do we know if we'll get eyes on the mistral stuff tomorrow? | 23:39 |
| weshay | EmilienM, https://review.rdoproject.org/r/#/c/11139/ | 23:39 |
| *** tosky has quit IRC | 23:39 | |
| EmilienM | mwhahaha: who knows who is here this week | 23:41 |
| weshay | rlandy|ruck, https://review.openstack.org/#/c/472607/168/roles/overcloud-deploy/GATE-CHECK doesn't need a rebase :) | 23:44 |
| weshay | afaict | 23:44 |
| rlandy|ruck | right | 23:45 |
| rlandy|ruck | weshay: wrt 004 - this run of the same test was successful http://logs.openstack.org/75/530775/2/check/tripleo-ci-centos-7-scenario004-multinode-oooq-container/03b993d/ - so yeah - setup issue | 23:47 |
| mwhahaha | what about https://bugs.launchpad.net/tripleo/+bug/1740934 | 23:54 |
| openstack | Launchpad bug 1740934 in tripleo "Tracker Bug: Tempest fails with packaging error - python-oslo-db-tests" [Critical,Triaged] | 23:54 |
| mwhahaha | looks like a version number mismatch | 23:54 |
| mwhahaha | where it's getting a 4.25.1-0 vs 4.25.1-1 | 23:55 |
| rlandy|ruck | EmilienM: can we close https://bugs.launchpad.net/tripleo/+bug/1739626 out - looks fixed | 23:57 |
| openstack | Launchpad bug 1739626 in tripleo "featureset008 is failing in Pike jobs - exec-setfacl-manila-manila" [Critical,In progress] - Assigned to Emilien Macchi (emilienm) | 23:57 |
| mwhahaha | yes that should have been fixed | 23:57 |
| mwhahaha | https://review.openstack.org/#/c/529406/ was merged | 23:57 |
| weshay | toure, you around? https://review.openstack.org/#/c/530781/ | 23:59 |
| weshay | mwhahaha, so there is nobody around to merge https://review.openstack.org/#/c/530781/ | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!