| *** wolverineav has quit IRC | 00:03 | |
| *** wolverineav has joined #tripleo | 00:04 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-docs master: WIP document workflow driven container prepare https://review.openstack.org/553104 | 00:07 |
|---|---|---|
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 00:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 00:10 |
| *** sanjayu has joined #tripleo | 00:10 | |
| *** itlinux has joined #tripleo | 00:12 | |
| *** wolverineav has quit IRC | 00:18 | |
| *** wolverineav has joined #tripleo | 00:21 | |
| *** khyr0n has quit IRC | 00:28 | |
| *** wolverineav has quit IRC | 00:29 | |
| *** wolverineav has joined #tripleo | 00:30 | |
| *** jobcespedes has quit IRC | 00:32 | |
| *** wolverineav has quit IRC | 00:40 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Move build_service_filter to kolla_builder from tripleoclient https://review.openstack.org/554738 | 00:41 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: WIP Perform multiple container image prepares and merge result https://review.openstack.org/554739 | 00:41 |
| openstackgerrit | Ian Wienand proposed openstack/tripleo-common master: Ensure output of shlex is quoted https://review.openstack.org/554684 | 00:52 |
| *** wolverineav has joined #tripleo | 00:53 | |
| *** wolverin_ has joined #tripleo | 00:55 | |
| *** wolverineav has quit IRC | 00:59 | |
| *** mcornea has joined #tripleo | 01:05 | |
| *** mcornea has quit IRC | 01:06 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 01:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 01:10 |
| *** wolverin_ has quit IRC | 01:13 | |
| *** jobcespedes has joined #tripleo | 01:13 | |
| *** wolverineav has joined #tripleo | 01:13 | |
| *** wolverineav has quit IRC | 01:17 | |
| *** jobcespedes has quit IRC | 01:18 | |
| *** dmacpher has joined #tripleo | 01:27 | |
| *** bfournie has joined #tripleo | 01:35 | |
| *** moshele has joined #tripleo | 01:39 | |
| *** fragatina has quit IRC | 01:39 | |
| *** fragatina has joined #tripleo | 01:41 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/tripleo-ci master: Install tripleo-common from source https://review.openstack.org/554705 | 01:41 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: [DNM] testing 554684 https://review.openstack.org/554685 | 01:42 |
| *** fragatina has quit IRC | 01:46 | |
| *** jobcespedes has joined #tripleo | 01:48 | |
| *** ebarrera has quit IRC | 01:50 | |
| *** cshastri has joined #tripleo | 02:00 | |
| *** jobcespedes has quit IRC | 02:00 | |
| *** agopi has joined #tripleo | 02:03 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 02:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 02:10 |
| *** atoth has quit IRC | 02:20 | |
| *** myoung|afk is now known as myoung | 02:23 | |
| *** myoung is now known as myoung|afk | 02:27 | |
| *** psachin has joined #tripleo | 02:39 | |
| *** wolverineav has joined #tripleo | 02:56 | |
| *** fragatina has joined #tripleo | 03:00 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 03:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 03:10 |
| *** rlandy|bbl is now known as rlandy | 03:15 | |
| *** wolverineav has quit IRC | 03:16 | |
| *** wolverineav has joined #tripleo | 03:17 | |
| *** wolverineav has quit IRC | 03:21 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix default partition type https://review.openstack.org/554771 | 03:22 |
| *** jaganathan has quit IRC | 03:27 | |
| *** jaganathan has joined #tripleo | 03:27 | |
| *** ramishra has joined #tripleo | 03:30 | |
| *** ykarel has joined #tripleo | 03:36 | |
| *** psahoo has joined #tripleo | 03:44 | |
| *** shreshtha has joined #tripleo | 03:48 | |
| *** dpawar has joined #tripleo | 03:52 | |
| *** skramaja has joined #tripleo | 03:53 | |
| *** skramaja_ has joined #tripleo | 03:58 | |
| *** tzumainn has quit IRC | 03:59 | |
| *** skramaja has quit IRC | 03:59 | |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart master: [DNM] Enable network isolation for Queens+ releases in FS020 https://review.openstack.org/554528 | 04:00 |
| *** links has joined #tripleo | 04:00 | |
| *** udesale has joined #tripleo | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 04:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 04:10 |
| *** skramaja_ is now known as skramaja | 04:11 | |
| openstackgerrit | Merged openstack/tripleo-upgrade master: New major upgrade workflow implementation. https://review.openstack.org/548336 | 04:16 |
| *** pdeore has joined #tripleo | 04:20 | |
| *** radeks has joined #tripleo | 04:29 | |
| *** radeks has quit IRC | 04:30 | |
| *** radeks has joined #tripleo | 04:30 | |
| *** pgadiya has joined #tripleo | 04:35 | |
| *** rlandy has quit IRC | 04:39 | |
| *** ratailor has joined #tripleo | 05:02 | |
| *** ratailor_ has joined #tripleo | 05:04 | |
| *** ratailor has quit IRC | 05:07 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 05:10 |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 05:10 |
| *** dpawar has quit IRC | 05:31 | |
| *** mdnadeem has joined #tripleo | 05:32 | |
| *** dpawar has joined #tripleo | 05:32 | |
| *** fragatina has quit IRC | 05:35 | |
| *** fragatina has joined #tripleo | 05:35 | |
| *** moshele has quit IRC | 05:39 | |
| *** dsariel has joined #tripleo | 05:45 | |
| *** akane_ has joined #tripleo | 05:58 | |
| *** assassin has joined #tripleo | 06:00 | |
| *** masco has joined #tripleo | 06:01 | |
| *** assassin has quit IRC | 06:04 | |
| *** karthiks has quit IRC | 06:06 | |
| *** agurenko has joined #tripleo | 06:06 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 06:10 |
| *** jcoufal has joined #tripleo | 06:12 | |
| *** yprokule has joined #tripleo | 06:15 | |
| *** ihrachys has quit IRC | 06:15 | |
| *** jfrancoa has joined #tripleo | 06:18 | |
| *** jfrancoa has quit IRC | 06:19 | |
| *** jfrancoa has joined #tripleo | 06:19 | |
| *** jcoufal_ has joined #tripleo | 06:20 | |
| *** udesale has quit IRC | 06:21 | |
| *** jcoufal has quit IRC | 06:21 | |
| *** udesale has joined #tripleo | 06:21 | |
| *** ratailor_ has quit IRC | 06:22 | |
| *** karthiks has joined #tripleo | 06:23 | |
| *** ratailor has joined #tripleo | 06:24 | |
| *** marios has joined #tripleo | 06:27 | |
| *** jcoufal has joined #tripleo | 06:27 | |
| *** yprokule has quit IRC | 06:28 | |
| *** yprokule_ has joined #tripleo | 06:28 | |
| *** yprokule_ is now known as yprokule | 06:29 | |
| Tengu | hello there | 06:29 |
| *** radeks has quit IRC | 06:30 | |
| *** jcoufal_ has quit IRC | 06:30 | |
| *** moshele has joined #tripleo | 06:31 | |
| *** dpawar has quit IRC | 06:31 | |
| *** dbecker has quit IRC | 06:31 | |
| *** radeks has joined #tripleo | 06:32 | |
| *** dpawar has joined #tripleo | 06:32 | |
| *** ratailor_ has joined #tripleo | 06:32 | |
| *** waleedm has joined #tripleo | 06:33 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui stable/queens: Imported Translations from Zanata https://review.openstack.org/554806 | 06:34 |
| *** ratailor has quit IRC | 06:35 | |
| *** karthiks has quit IRC | 06:35 | |
| *** StevenK has quit IRC | 06:35 | |
| *** sdake has quit IRC | 06:35 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/554808 | 06:36 |
| *** jbadiapa has joined #tripleo | 06:36 | |
| *** StevenK has joined #tripleo | 06:36 | |
| *** ratailor_ has quit IRC | 06:37 | |
| *** sdake has joined #tripleo | 06:37 | |
| *** sdake has joined #tripleo | 06:37 | |
| *** karthiks has joined #tripleo | 06:39 | |
| *** hjensas has quit IRC | 06:40 | |
| *** paramite_ has quit IRC | 06:41 | |
| *** ratailor has joined #tripleo | 06:41 | |
| *** dbecker has joined #tripleo | 06:46 | |
| *** agopi has quit IRC | 06:49 | |
| *** gkadam has joined #tripleo | 06:50 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: WIP: Create import-role role. https://review.openstack.org/553492 | 06:53 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: WIP: Add undercloud upgrades playbook. https://review.openstack.org/553827 | 06:53 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: WIP: Run playbooks with custom args https://review.openstack.org/553474 | 07:02 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Use tripleo-upgrade role in undercloud upgrades job. https://review.openstack.org/548974 | 07:02 |
| *** jaosorior has quit IRC | 07:05 | |
| *** oscar has joined #tripleo | 07:05 | |
| *** aufi has joined #tripleo | 07:06 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 07:10 |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 07:10 |
| *** akrivoka has joined #tripleo | 07:11 | |
| openstackgerrit | waleed mousa proposed openstack/os-net-config master: Adding ethtool command after binding dpdk drivers in Mellanox nics https://review.openstack.org/554048 | 07:11 |
| *** cylopez has joined #tripleo | 07:12 | |
| *** cylopez has quit IRC | 07:14 | |
| *** cylopez has joined #tripleo | 07:15 | |
| *** cylopez has left #tripleo | 07:15 | |
| *** rcernin has quit IRC | 07:21 | |
| *** pmannidi has quit IRC | 07:22 | |
| *** guits__ has joined #tripleo | 07:25 | |
| *** quiquell has joined #tripleo | 07:27 | |
| *** holser__ has joined #tripleo | 07:32 | |
| *** dpawar has quit IRC | 07:34 | |
| *** nyechiel_ has joined #tripleo | 07:35 | |
| openstackgerrit | Harald Jensås proposed openstack/python-tripleoclient master: Fix Genconfig - no HOME in environment https://review.openstack.org/554678 | 07:36 |
| *** moshele has quit IRC | 07:36 | |
| *** hjensas has joined #tripleo | 07:38 | |
| *** hjensas has quit IRC | 07:38 | |
| *** hjensas has joined #tripleo | 07:38 | |
| *** dmacpher has quit IRC | 07:42 | |
| *** jaosorior has joined #tripleo | 07:44 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-common master: Install python2-networking-baremetal in neutron-server https://review.openstack.org/545452 | 07:46 |
| *** moshele has joined #tripleo | 07:46 | |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud master: Use the new dnsmasq PXE filter in ironic-inspector https://review.openstack.org/523944 | 07:47 |
| *** dpawar has joined #tripleo | 07:48 | |
| *** moshele has quit IRC | 07:50 | |
| *** moshele has joined #tripleo | 07:51 | |
| *** yamahata has joined #tripleo | 07:55 | |
| openstackgerrit | Adriano Petrich proposed openstack/tripleo-common master: Move password generation to deployment phase https://review.openstack.org/542143 | 08:02 |
| *** ebarrera has joined #tripleo | 08:05 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Add multinode-overcloud-update playbook to run list https://review.openstack.org/547058 | 08:06 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Fix update of pacemaker container images during major upgrade https://review.openstack.org/547476 | 08:09 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 08:10 |
| *** ebarrera has quit IRC | 08:12 | |
| *** ebarrera has joined #tripleo | 08:13 | |
| *** florianf has joined #tripleo | 08:13 | |
| *** dparkes has joined #tripleo | 08:13 | |
| *** nyechiel_ has quit IRC | 08:14 | |
| *** nyechiel_ has joined #tripleo | 08:24 | |
| *** bogdando has joined #tripleo | 08:25 | |
| *** tesseract has joined #tripleo | 08:31 | |
| *** ffiore has joined #tripleo | 08:32 | |
| *** ratailor has quit IRC | 08:35 | |
| *** ratailor has joined #tripleo | 08:35 | |
| *** amoralej|off is now known as amoralej | 08:36 | |
| *** matbu has quit IRC | 08:37 | |
| *** ccamacho has joined #tripleo | 08:38 | |
| *** matbu has joined #tripleo | 08:39 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-common master: Add ironic-neutron-agent container https://review.openstack.org/547321 | 08:39 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/queens: WIP: fix fluentd upgrade tasks during ffu. https://review.openstack.org/554831 | 08:42 |
| *** jpena|off is now known as jpena | 08:43 | |
| *** tosky has joined #tripleo | 08:43 | |
| *** matbu has quit IRC | 08:47 | |
| *** chem|eod is now known as chem | 08:48 | |
| *** matbu has joined #tripleo | 08:48 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: WIP Upgrade data on disk on mariadb major upgrade https://review.openstack.org/546666 | 08:49 |
| *** tesseract has quit IRC | 08:51 | |
| *** tesseract has joined #tripleo | 08:52 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: New major upgrade workflow implementation. https://review.openstack.org/552082 | 08:53 |
| *** tesseract has quit IRC | 08:54 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: DNM: Testing P->Q from queens branch. https://review.openstack.org/552080 | 08:55 |
| *** skramaja has quit IRC | 08:56 | |
| *** tesseract has joined #tripleo | 08:57 | |
| *** lucas-afk is now known as lucasagomes | 08:59 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: Ensure ansible-pacemaker is present on the undercloud. https://review.openstack.org/554236 | 08:59 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: DNM: Add debug and don't fail on validation for ffu. https://review.openstack.org/552063 | 08:59 |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-upgrade master: Add the ability to limit the hosts where to apply FFU https://review.openstack.org/554833 | 09:00 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: DNM: Testing FFU from queens branch. https://review.openstack.org/552064 | 09:01 |
| *** jpich has joined #tripleo | 09:02 | |
| *** arxcruz|off is now known as arxcruz | 09:04 | |
| bandini | marios: can you come at me on https://review.openstack.org/#/c/554306/, bro? | 09:04 |
| bandini | (fairly simple cherry-pick) | 09:04 |
| *** marios has quit IRC | 09:05 | |
| *** marios has joined #tripleo | 09:05 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/queens: Fix newton compat mode for deployed server. https://review.openstack.org/552980 | 09:08 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/ocata: Extract local CA if it expired https://review.openstack.org/554423 | 09:09 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/queens: WIP: fix fluentd upgrade tasks during ffu. https://review.openstack.org/554831 | 09:09 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 09:10 |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: DNM: Testing FFU from queens branch. https://review.openstack.org/552064 | 09:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 09:10 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: Refactor setting default CA https://review.openstack.org/554835 | 09:11 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: Read bytes for default CA https://review.openstack.org/554836 | 09:11 |
| *** ukalifon has joined #tripleo | 09:14 | |
| bandini | ccamacho: thank you, sir :) | 09:16 |
| bandini | marios: unping, carlos did the magic! | 09:16 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/paunch master: Allow to limit cgroup cpu shares https://review.openstack.org/554539 | 09:19 |
| ccamacho | hey bandini np | 09:23 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/newton: Disallow TLS v1.0 from HAProxy https://review.openstack.org/554422 | 09:24 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Ensure ansible-pacemaker is present on the undercloud. https://review.openstack.org/554262 | 09:24 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Standardise Workflow messaging and optionally persist messages https://review.openstack.org/425060 | 09:25 |
| *** jaganathan has quit IRC | 09:26 | |
| bandini | jaosorior: ENOTIME on https://review.openstack.org/551286 or other reasons? | 09:28 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/paunch master: Allow to limit cgroup cpu shares https://review.openstack.org/554539 | 09:28 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Standardise Workflow messaging and optionally persist messages https://review.openstack.org/425060 | 09:28 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Allow for passing boot-time vars/args to OC nodes https://review.openstack.org/552967 | 09:29 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Add an openshift-cns service https://review.openstack.org/543933 | 09:29 |
| jaosorior | bandini: your first guess | 09:30 |
| bandini | jaosorior: ah ok :) | 09:31 |
| jaosorior | :( | 09:31 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/instack-undercloud master: Enable TLS by default https://review.openstack.org/552382 | 09:33 |
| *** akane_ has quit IRC | 09:34 | |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-heat-templates master: Make HA containers log to /var/log/containers after upgrade https://review.openstack.org/553424 | 09:34 |
| *** derekh has joined #tripleo | 09:34 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Use the standard messaging in the baremetal workbook https://review.openstack.org/552460 | 09:36 |
| marios | bandini: sorry missed it | 09:37 |
| bandini | marios: you're still my favourite sadopanda | 09:37 |
| * marios waves arms in the air | 09:38 | |
| bandini | lol | 09:38 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Use the standard messaging in the fernet-key-rotate workbook https://review.openstack.org/554552 | 09:38 |
| *** suuuper has joined #tripleo | 09:38 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Use the standard messaging in the networks workbook https://review.openstack.org/554593 | 09:39 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Use the standard messaging in the plan_management workbook https://review.openstack.org/554595 | 09:40 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Use the standard messaging in the package_update workbook https://review.openstack.org/554594 | 09:40 |
| Tengu | hmm. what would happen if I drop the content of /var/lib/heat-config/deployed/ directory prior trying my upgrade thing? any idea? marios maybe? :) | 09:41 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Use the standard messaging in the derive_parameters workbook https://review.openstack.org/554548 | 09:41 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Remove the lacp bond mode constraints https://review.openstack.org/554846 | 09:43 |
| marios | Tengu: o/ why are you wanting to do that :). well if i recall you're doing a ocata to pike? at a guess it would no longer skip all those deployments that were already applied ;) are you having problems getting some particular software deployment (from the tht) to run | 09:44 |
| Tengu | marios: pike BM to pike container :D | 09:44 |
| Tengu | the infamous migration that nobody wants to know about ;) | 09:45 |
| marios | Tengu: right you're doing pike to pike (noop cos no repo switch) | 09:45 |
| openstackgerrit | Oliver Walsh proposed openstack/instack-undercloud master: Set undercloud nova notification_format to 'unversioned' https://review.openstack.org/554847 | 09:45 |
| marios | Tengu: ;) you're on the bleeding edge man | 09:45 |
| Tengu | marios: yep. I'm hitting a nasty situation with pacemaker in fact: https://bugs.launchpad.net/tripleo/+bug/1756876 | 09:45 |
| openstack | Launchpad bug 1756876 in tripleo "Upgrade "pike bm -> pike container": pacemaker issue" [Low,Triaged] | 09:45 |
| Tengu | marios: pacemaker got shut down on the controllers, and after that, puppet wants a quorum for the cluster before going forward. Of course, this won't work. like, at all. | 09:46 |
| Tengu | so I'm trying to find a way to kind of… well… force some re-deploy and such. | 09:46 |
| marios | Tengu: so during the upgrade steps the cluster goes down then up again, i.e. before running puppet/config | 09:46 |
| Tengu | marios: ah. it apparently doesn't goes up again… | 09:47 |
| marios | Tengu: let me check the bug and add some pointers but i see bandini has already checked in which is good to see | 09:47 |
| Tengu | marios: :) thanks for your time. I've re-armed the lab so that I'm ready for more :) | 09:47 |
| Tengu | ah, Michele is bandini - good to know :) | 09:48 |
| openstackgerrit | Dougal Matthews proposed openstack/instack-undercloud master: Use the default queue when calling create_deployment_plan https://review.openstack.org/554630 | 09:51 |
| *** akane_ has joined #tripleo | 09:51 | |
| *** panda|off is now known as panda | 09:52 | |
| *** gfidente has joined #tripleo | 09:54 | |
| *** gfidente has quit IRC | 09:54 | |
| *** gfidente has joined #tripleo | 09:54 | |
| marios | Tengu: not sure if that helps, but really we need more info to be able to help (i.e. something went wrong on one of those upgrade tasks, i'll bet so that the cluster start is failing) | 09:56 |
| Tengu | marios: I'll comment up as soon as I'm ready for another run :) | 09:57 |
| oscar | Hi, trying to deploy a pike overcloud but keep failing at overcloud.AllNodesDeploySteps.ControllerDeployment_Step3.0: "Error: /Stage[main]/Nova::Cell_v2::Discover_hosts/Exec[nova-cell_v2-discover_hosts]: Failed to call refresh: nova-manage cell_v2 discover_hosts returned 1 instead of one of [0]". Does anyone know what could be causing that? | 09:57 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: WIP: Add undercloud upgrades playbook. https://review.openstack.org/553827 | 09:57 |
| Tengu | marios: hmm so basically I should ensure pacemaker is up'n'running after step 4 of the upgrade_task related to controllers, right? I'll check that, as well as the other things you mention. | 09:58 |
| marios | Tengu: ack, wrt names/nicks... i totally spent 5 minutes thinking "Cédric" is 'one of tengus colleagues that must be helping out with this' | 09:58 |
| Tengu | XD | 09:59 |
| marios | sorry i tend to remember and call people by their irc nicks in real life, ask bandini | 09:59 |
| *** salmankhan has joined #tripleo | 09:59 | |
| chem | jistr: hi, featureset037 is the one you're using for testing the update right ? | 09:59 |
| Tengu | marios: np, I tend to do the same | 09:59 |
| oscar | and if I try to run nova-manage cell_v2 discover_hosts manually I get a mysql error: Unknown column 'cn.host' in 'field list' | 09:59 |
| jistr | chem: yup | 09:59 |
| Tengu | marios: and ppl knowing my IRC nick usually call me "tengu". | 09:59 |
| chem | jistr: It was about to get deleted from rdo-cloud I think https://review.rdoproject.org/r/#/c/12160/9/jobs/tripleo-upstream.yml (currently reviewing it) | 10:00 |
| Tengu | ah. at last I found how to make ansible wait for a server to come back to life after a reboot \o/ | 10:01 |
| jistr | chem: oh... thanks for bringing this up. We shouldn't remove it, at least from master. cc myoung|afk | 10:01 |
| chem | jistr: yeah, I'll minus -1 when I'm done | 10:01 |
| jistr | chem, myoung|afk : but wait Matt's patch actually doesn't remove it from master. I think it's ok to remove it from the older branches, at least for now | 10:02 |
| jistr | we can re-add if necessary, no need to waste resources for now | 10:02 |
| chem | jistr: heu ... it's removing it from master as far as I can tell | 10:03 |
| chem | jistr: http://paste.openstack.org/show/707327/ ? | 10:04 |
| chem | jistr: will remove master job no ? | 10:04 |
| *** egallen has joined #tripleo | 10:06 | |
| *** skramaja has joined #tripleo | 10:07 | |
| jistr | chem: actually i have no idea how this stuff works :D In the other file, i see the job still under `tripleo-upgrades-check-branchless` list, so i assumed it doesn't get removed, but yea maybe it does get removed if we remove it from the other file? I don't know what's the difference between those files. | 10:07 |
| chem | jistr: on has to match the other kindof stuff, anyway myoung|afk will know | 10:08 |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: [Experimental] Unit Test Mistral Workflows https://review.openstack.org/553389 | 10:09 |
| jistr | chem: so one could be job definitions and the other a list of triggers? that'd make sense, just wondering why all the job definitions are under project section with `name: tripleo-quickstart`, but maybe that's ok | 10:09 |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 10:10 |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 10:10 |
| *** salmankhan1 has joined #tripleo | 10:11 | |
| *** salmankhan has quit IRC | 10:11 | |
| *** salmankhan1 is now known as salmankhan | 10:11 | |
| chem | jistr: yeah reviewing 15k lines of yaml in one file is always a thing of beauty | 10:12 |
| chem | jistr: I miss autogen | 10:12 |
| jistr | chem: yea :D btw looking at that file with job lists, we should probably add a section for tripleo-upgrade repo and add the tripleo-upgrades-check-branchless. I may just post it on top of myoung|afk patch and let folks review. | 10:13 |
| *** assassin has joined #tripleo | 10:13 | |
| chem | jistr: well not sure what this "tripleo-upgrades-check-branchless" means ... tripleo-upgarde is "branchfull" | 10:13 |
| chem | jistr: this is just plain confusing | 10:14 |
| chem | IMHO | 10:14 |
| jistr | chem: yea for me too, hopefully not so much for CI folks, we need to sync up with them | 10:14 |
| *** ktibi has joined #tripleo | 10:15 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Include new CLI changes for overcloud update. https://review.openstack.org/550517 | 10:16 |
| *** dpawar has quit IRC | 10:17 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Optionally run config download at the end of stack create/update https://review.openstack.org/554224 | 10:18 |
| chem | jistr: oki the branchless is for them to be able to run the tripleo-upgrade check under oooq projects (which are branchless) | 10:18 |
| ccamacho | hey mwhahaha sorry to bug you, quick question. Im testing a few puppet-nova patches and im doing it patching tht and puppet-nova on deployment but it takes too much time for testing simple stuff.. Do you know if there is an easy way of running puppet-apply configuring the parameters together with the manifest to test ? | 10:19 |
| ccamacho | Im trying to do it but im not able to run it correctly | 10:19 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add cpu shares cgroup limits for neutron ovs agent https://review.openstack.org/554863 | 10:20 |
| jistr | chem: ok so i guess we shouldn't run the -branchless on tripleo-upgrade then? (which is branchful :) ) so i'll not post that patch | 10:25 |
| *** marios has quit IRC | 10:25 | |
| *** marios has joined #tripleo | 10:25 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Add cpu shares cgroup limits for neutron ovs agent https://review.openstack.org/554863 | 10:25 |
| *** akane__ has joined #tripleo | 10:26 | |
| *** skramaja has quit IRC | 10:26 | |
| Tengu | marios: small question: bandini talked about the order - if I push all the docker-related things to the last positions of my "-e" arguments, it should be OK, on that part, right? | 10:26 |
| *** social has joined #tripleo | 10:28 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Limit neutron_ovs_agent CPU to 15% in CI scenarios https://review.openstack.org/554869 | 10:29 |
| *** akane_ has quit IRC | 10:29 | |
| *** salmankhan has quit IRC | 10:30 | |
| *** dpawar has joined #tripleo | 10:33 | |
| *** salmankhan has joined #tripleo | 10:34 | |
| *** egallen has quit IRC | 10:36 | |
| *** egallen has joined #tripleo | 10:37 | |
| *** fzdarsky has joined #tripleo | 10:41 | |
| Tengu | o_O wow. just get another error. Never saw that one… | 10:45 |
| *** egallen has quit IRC | 10:45 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-upgrade master: Add the ability to limit the hosts where to apply FFU https://review.openstack.org/554833 | 10:46 |
| Tengu | -.- ok. a VM seems to have crashed. of course. | 10:46 |
| openstackgerrit | mathieu bultel proposed openstack/tripleo-heat-templates master: Do not create NetworkVlanID is the value is not defined https://review.openstack.org/554872 | 10:46 |
| *** ffiore_ has joined #tripleo | 10:47 | |
| *** ffiore has quit IRC | 10:47 | |
| *** zoli is now known as zoli|lunch | 10:49 | |
| *** dtantsur|afk is now known as dtantsur | 10:49 | |
| *** aputtur__ has quit IRC | 10:52 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-heat-templates master: Fix queries for already installed packages https://review.openstack.org/554499 | 10:52 |
| *** kmy has quit IRC | 10:52 | |
| *** yamahata has quit IRC | 10:52 | |
| *** kmy has joined #tripleo | 10:53 | |
| *** paramite_ has joined #tripleo | 10:53 | |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Fix overcloud services connectivity validation https://review.openstack.org/553832 | 11:02 |
| *** udesale_ has joined #tripleo | 11:04 | |
| *** numans is now known as numans_afk | 11:05 | |
| *** udesale has quit IRC | 11:07 | |
| *** sshnaidm|sick is now known as sshnaidm | 11:08 | |
| myoung|afk | jistr, chem, re the patch to remove old upgreade jobs, this is leftover work/debt from our sprint in feb when we were putting a toe in the water for upgrade jobs (https://trello.com/c/3UFgRWtk/565-remove-old-upgrade-jobs-in-sf). Nothing's sacred, that patch was proactive cleanup, not to be merged until we had the final upgrade jobs in place. If it's not necessary any more please patch over it or we can abandon. I'm not sure ( | 11:09 |
| myoung|afk | personally) what current state of upgrade job(s) are... | 11:09 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Fix overcloud services connectivity validation https://review.openstack.org/553832 | 11:09 |
| myoung|afk | jistr, chem, please advise :) I'll be online (for real) in another 90... | 11:09 |
| * myoung|afk wanders off to make morning coffee | 11:09 | |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 11:10 |
| *** gyankum has joined #tripleo | 11:10 | |
| *** dpawar has quit IRC | 11:10 | |
| chem | myoung|afk: jistr and I think most of it is valid, if you don't mind then, I'll upload a newest version of it that reflect the current status | 11:11 |
| jpich | honza: A few people mentioned "need to update stuff so the new endpoints work in containerised undercloud as well" about that instack patch, were you working on it? I'm going to look into it today if not, if that's fine by you | 11:12 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Implement MasqueradeNetworks services https://review.openstack.org/553427 | 11:13 |
| *** udesale_ has quit IRC | 11:13 | |
| myoung|afk | chem: don't mind at all! one big team...feel free :) | 11:15 |
| *** egallen has joined #tripleo | 11:20 | |
| *** cschwede has joined #tripleo | 11:23 | |
| honza | jpich: i believe it's done --- but i shall double check | 11:25 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common stable/queens: Remove the noop deploystep for upgrade converge step https://review.openstack.org/554880 | 11:25 |
| jpich | honza: Oh, cool! If you have a link to the review(s) I'd love to see how that's done | 11:25 |
| *** kopecmartin has joined #tripleo | 11:26 | |
| *** egallen has quit IRC | 11:27 | |
| *** adarazs is now known as adarazs_lunch | 11:27 | |
| openstackgerrit | Sagi Shnaidman proposed openstack/tripleo-quickstart master: Ignore empty values for dlrn hashes https://review.openstack.org/554882 | 11:27 |
| *** cshastri has quit IRC | 11:27 | |
| honza | jpich: https://review.openstack.org/#/c/515490/27..30 | 11:27 |
| sshnaidm | jfrancoa, ^^ | 11:28 |
| *** bfournie has quit IRC | 11:28 | |
| jpich | honza: Thanks! | 11:28 |
| honza | jpich: ps 27 is where i started it, and then other people fixed a few things | 11:28 |
| jfrancoa | sshnaidm: wow! that was fast :-D | 11:28 |
| *** bfournie has joined #tripleo | 11:28 | |
| jpich | honza: team work | 11:28 |
| sshnaidm | jfrancoa, beside of that you need to pass empty vars in you playbook to dlrn_hash_path and dlrn_hash_path_newest to zeroize them | 11:29 |
| *** numans_afk is now known as numans | 11:29 | |
| jfrancoa | sshnaidm: ok, thanks a lot. I'll give it a try! | 11:29 |
| *** skramaja has joined #tripleo | 11:30 | |
| *** bfournie has quit IRC | 11:33 | |
| *** dmacpher has joined #tripleo | 11:34 | |
| *** abishop has joined #tripleo | 11:35 | |
| chem | myoung|afk: jistr ack, will do then | 11:36 |
| *** pdeore_ has joined #tripleo | 11:38 | |
| *** pdeore has quit IRC | 11:38 | |
| *** jpena is now known as jpena|off | 11:39 | |
| *** jpena|off is now known as jpena | 11:40 | |
| openstackgerrit | Harald Jensås proposed openstack/python-tripleoclient master: Contanerized Undercloud - Routed Spine-Leaf https://review.openstack.org/543455 | 11:41 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Use ironic::inspector::dnsmasq_ip_subnets https://review.openstack.org/543582 | 11:41 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Use IronicInspectorSubnets in undercloud.yaml https://review.openstack.org/547325 | 11:41 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add static routes for routed ctlplane https://review.openstack.org/545109 | 11:41 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add ctlplane networking for routed networks https://review.openstack.org/547326 | 11:41 |
| honza | jpich: /me dead => "Honza & co" :) | 11:41 |
| jpich | :-) | 11:42 |
| openstackgerrit | John Fulton proposed openstack/tripleo-heat-templates master: DO NOT MEREGE: testing scenario001 without CephClient https://review.openstack.org/554884 | 11:43 |
| bogdando | it seems folks that cgroups do not applied for https://review.openstack.org/#/c/554869/1 :o | 11:45 |
| bogdando | I have a local libvirt env | 11:45 |
| bogdando | if someone wants to look into | 11:45 |
| bogdando | then, we could submit z bz or something | 11:45 |
| bogdando | owalsh: ^^ perchance?.. | 11:46 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Remove unnecessary parameters from featureset047. https://review.openstack.org/553850 | 11:47 |
| bogdando | jaosorior: ^^ jfyi :) security squad might be interested as well | 11:47 |
| jaosorior | bogdando: reading the docs https://docs.docker.com/config/containers/resource_constraints/#cpu it says "This is only enforced when CPU cycles are constrained. When plenty of CPU cycles are available, all containers use as much CPU as they need. In that way, this is a soft limit. --cpu-shares does not prevent containers from being scheduled in swarm mode. It prioritizes container CPU resources for the | 11:48 |
| jaosorior | available CPU cycles. It does not guarantee or reserve any specific CPU access." | 11:48 |
| bogdando | um | 11:48 |
| bogdando | so we need to bundle this param with something more restrictive | 11:48 |
| jaosorior | bogdando: we just need to figure out what | 11:48 |
| bogdando | jaosorior: well done, thanks! let's figure out something then | 11:48 |
| * jaosorior reading docs | 11:48 | |
| bogdando | yeah | 11:48 |
| owalsh | bogdando, jaosorior: is that a problem? Why don't we want to use the cycles if they are available? | 11:49 |
| bogdando | owalsh: well, not sure I have really something available on my env :D | 11:50 |
| bogdando | LA >3.5 for 2 vcpu | 11:50 |
| owalsh | hmm, doesn't necessarily mean CPU is flat out, I/O bound maybe? | 11:51 |
| bogdando | interesting... | 11:51 |
| mwhahaha | ccamacho: no there really isn't especially with containers. You could go try and rerun the docker-puppet.py bits manually on an already deployed system after you patch the modules. | 11:51 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix newton compat mode for deployed server. https://review.openstack.org/552923 | 11:52 |
| openstackgerrit | Merged openstack/instack-undercloud stable/queens: Mariadb online upgrade after yum update https://review.openstack.org/554306 | 11:52 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/queens: New major upgrade workflow implementation. https://review.openstack.org/552082 | 11:52 |
| bogdando | owalsh: https://pastebin.com/D11CedHu my env info | 11:53 |
| bogdando | wa 0, so that's not IO | 11:53 |
| owalsh | bogdando: only using half the available CPU time | 11:54 |
| bogdando | though, you're probably right, and 1.8 is ~2 which shows like my 2 vCPUs are not super busy | 11:54 |
| owalsh | 100% for a process == 1 full CPU | 11:55 |
| bogdando | I wonder if we should keep that as is, or add cpu counts for container as well | 11:55 |
| bogdando | I feel like neutron is mining bitcoins on my env :< | 11:56 |
| owalsh | would guess that neutron-openvswitch is in a polling loop so 100% CPU is to be expected | 11:56 |
| * owalsh is just guessing though, networking guys might have some input | 11:57 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/paunch master: Allow configuring security options https://review.openstack.org/554542 | 11:57 |
| *** dsariel has quit IRC | 11:58 | |
| *** jlabarre has quit IRC | 11:58 | |
| hjensas | bogdando: We should get the neutron fix for the ovs issue soon. | 12:00 |
| bogdando | hjensas: ack, though we can still have some handy experience from that case :) | 12:01 |
| bogdando | for future | 12:01 |
| hjensas | bogdando: sure. :) | 12:01 |
| *** bfournie has joined #tripleo | 12:01 | |
| *** adarazs_lunch is now known as adarazs | 12:01 | |
| *** ansmith has joined #tripleo | 12:03 | |
| *** aputtur__ has joined #tripleo | 12:03 | |
| *** atoth has joined #tripleo | 12:04 | |
| *** rfolco has joined #tripleo | 12:05 | |
| *** dprince has joined #tripleo | 12:06 | |
| *** pchavva has joined #tripleo | 12:07 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 12:10 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade master: We need to be root to install package. https://review.openstack.org/554887 | 12:10 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade master: FFU: We need to be root to install ansible-pacemaker package. https://review.openstack.org/554887 | 12:10 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: Ensure ansible-pacemaker is present on the undercloud. https://review.openstack.org/554236 | 12:11 |
| *** zoli|lunch is now known as zoli | 12:11 | |
| *** akane__ has quit IRC | 12:11 | |
| *** raildo has joined #tripleo | 12:12 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: Ensure ansible-pacemaker is present on the undercloud. https://review.openstack.org/554236 | 12:12 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: DNM: Add debug and don't fail on validation for ffu. https://review.openstack.org/552063 | 12:12 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: DNM: Testing FFU from queens branch. https://review.openstack.org/552064 | 12:12 |
| *** pkovar has joined #tripleo | 12:13 | |
| *** akane has joined #tripleo | 12:13 | |
| openstackgerrit | Merged openstack/puppet-tripleo master: Fixes incorrect ownership of ODL TLS cert/key https://review.openstack.org/554537 | 12:13 |
| *** dsariel has joined #tripleo | 12:13 | |
| *** raildo has quit IRC | 12:14 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: Ensure ansible-pacemaker is present on the undercloud. https://review.openstack.org/554236 | 12:14 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: DNM: Add debug and don't fail on validation for ffu. https://review.openstack.org/552063 | 12:14 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-upgrade stable/queens: DNM: Testing FFU from queens branch. https://review.openstack.org/552064 | 12:14 |
| openstackgerrit | Florian Fuchs proposed openstack/tripleo-validations master: Fix MySQL Open Files Limit validation https://review.openstack.org/554888 | 12:15 |
| *** lucasagomes is now known as lucas-hungry | 12:16 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates stable/queens: Fix newton compat mode for deployed server. https://review.openstack.org/552980 | 12:17 |
| sshnaidm | jfrancoa, is it related to what you are doing now? https://bugs.launchpad.net/tripleo/+bug/1735792 | 12:18 |
| openstack | Launchpad bug 1735792 in tripleo "tripleo-ci-centos-7-undercloud-upgrades is running tripleo.sh vs oooq" [High,Triaged] - Assigned to wes hayutin (weshayutin) | 12:18 |
| jfrancoa | sshnaidm: exactly, all these patches are to move away that job from running tripleo.sh into tripleo-quickstart + tripleo-upgrade | 12:19 |
| sshnaidm | jfrancoa, ok, will update it then.. | 12:19 |
| jfrancoa | sshnaidm: I didn't know there was a bug, so I'll add the LP into all the related commits | 12:19 |
| jfrancoa | and assign it to me, please | 12:20 |
| sshnaidm | jfrancoa, cool, thanks | 12:20 |
| jfrancoa | sshnaidm: np | 12:20 |
| *** jpena is now known as jpena|lunch | 12:20 | |
| *** dsariel has quit IRC | 12:21 | |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Use tripleo-upgrade role in undercloud upgrades job. https://review.openstack.org/548974 | 12:23 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-quickstart-extras master: Collect installed cron jobs https://review.openstack.org/554889 | 12:24 |
| *** raildo has joined #tripleo | 12:24 | |
| openstackgerrit | Dougal Matthews proposed openstack/tripleo-common master: Verify the Swift container exists with a small utility workflow https://review.openstack.org/528213 | 12:27 |
| *** pradk has quit IRC | 12:29 | |
| *** dprince has quit IRC | 12:29 | |
| *** artom has joined #tripleo | 12:29 | |
| *** pkovar has quit IRC | 12:29 | |
| *** panda is now known as panda|lunch | 12:30 | |
| *** pdeore_ has quit IRC | 12:30 | |
| *** masco has quit IRC | 12:30 | |
| *** artom has quit IRC | 12:31 | |
| *** psahoo has quit IRC | 12:32 | |
| Tengu | marios: I think I have some news: apparently something goes wrong on 2 of the controllers, and pacemaker show them as "offline". After a quick check, the *network* seems the culprit: both nodes have an issue, apparently they are unable to ping their default gateway (at least) | 12:32 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Add validation-errors-nonfatal and debug into updates job. https://review.openstack.org/554891 | 12:32 |
| Tengu | and they are apparently unable to ping the public IPs associated to the nodes… duh. | 12:33 |
| openstackgerrit | Jose Luis Franco proposed openstack-infra/tripleo-ci master: Add multinode-overcloud-update playbook to run list https://review.openstack.org/547058 | 12:33 |
| *** trown|outtypewww is now known as trown|ruck | 12:34 | |
| *** artom has joined #tripleo | 12:34 | |
| dciabrin_ | morning o/ is anybody hitting https://bugs.launchpad.net/tripleo/+bug/1755485 atm ? it seems job tripleo-ci-centos-7-3nodes-multinode is broken in the gate | 12:35 |
| openstack | Launchpad bug 1755485 in tripleo "Barbican tempest test failing to ssh to cirros image" [Critical,Triaged] | 12:35 |
| *** rlandy has joined #tripleo | 12:35 | |
| *** lblanchard has joined #tripleo | 12:38 | |
| *** lblanchard has quit IRC | 12:39 | |
| trown|ruck | dciabrin_: the gate queue does not look out of sorts... It is possible that is just a low failure rate intermittent issue | 12:39 |
| dciabrin_ | trown|ruck, ack thx I'll see if it reproduces | 12:40 |
| *** masco has joined #tripleo | 12:42 | |
| *** lblanchard has joined #tripleo | 12:42 | |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud master: Fix help string for subnets option https://review.openstack.org/554895 | 12:42 |
| *** pgadiya has quit IRC | 12:44 | |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud master: Fix help string for subnets option https://review.openstack.org/554895 | 12:44 |
| *** panda|lunch is now known as panda | 12:45 | |
| Tengu | marios: apparently… I can go further now. | 12:47 |
| Tengu | … failed. so. what's the next error now :D | 12:48 |
| *** aputtur has joined #tripleo | 12:49 | |
| *** florianf_ has joined #tripleo | 12:51 | |
| Tengu | duh… stack broken :( | 12:52 |
| Tengu | this might explain. ERROR: The specified reference "ControllerDeployment_Step1" (in WorkflowTasks_Step2) is incorrect. | 12:52 |
| *** pdeore has joined #tripleo | 12:52 | |
| *** eck`gone is now known as eck` | 12:53 | |
| openstackgerrit | yolanda.robla proposed openstack/tripleo-heat-templates master: Fix queries for already installed packages https://review.openstack.org/554499 | 12:53 |
| *** florianf has quit IRC | 12:53 | |
| *** pdeore has quit IRC | 12:53 | |
| Tengu | darn! ceph!! X( | 12:54 |
| *** tcw has joined #tripleo | 12:56 | |
| Tengu | gfidente: are you here? I have a "small" question regarding ceph. | 12:57 |
| gfidente | Tengu HEY | 12:57 |
| Tengu | gfidente: \o/ great :) | 12:57 |
| gfidente | Tengu I saw the error, I guess you're trying to update a stack in UPDATE_FAILED state right? | 12:58 |
| Tengu | gfidente: nope | 12:58 |
| Tengu | gfidente: we had a small/quick talk at the PTG/Dublin - I'm trying to move from a baremetal pike to container pike - and in that move, from puppet-ceph to ansible-ceph/containers | 12:58 |
| gfidente | yep I remember that | 12:58 |
| Tengu | gfidente: apparently, there are some checks/references that make mistral fail badly, even in advanced steps. | 12:59 |
| openstackgerrit | Harald Jensås proposed openstack/instack-undercloud master: Fix next_hop for metadata service host route on local_subnet https://review.openstack.org/554908 | 12:59 |
| Tengu | gfidente: even if I have a "watch rm -f cephstorage_extraconfig.json" on the ceph-storage nodes, apparently it wasn't enough and mistral got a hint there was a puppet stuff: Workflow 'tripleo.storage.v1.ceph-install' [RUNNING -> ERROR, msg=Ceph deployment stopped, puppet-ceph hieradata found. Convert it into ceph-ansible variables. [u'ceph::profile::params::osds']] | 12:59 |
| jaosorior | lhinds: yo | 13:00 |
| lhinds | jaosorior: o/ | 13:00 |
| jaosorior | lhinds: Lets wait a bit for more folks to show up | 13:00 |
| gfidente | Tengu ah yeah we do that on purpose | 13:00 |
| lhinds | sure! | 13:00 |
| jaosorior | d0ugal: around? | 13:00 |
| *** mcornea has joined #tripleo | 13:00 | |
| Tengu | gfidente: do you have any idea if what I want to do is possible, and if so how? or should I just drop ceph-storage and re-deploy them with containers from scratch? | 13:00 |
| gfidente | Tengu you have to convert the old disks mapping from ceph::profile::params::osds to the cepha-ansible for | 13:00 |
| gfidente | *format | 13:00 |
| owalsh | lhinds, jaosorior: o/ | 13:00 |
| Tengu | gfidente: well, I think I did it. | 13:01 |
| gfidente | and *remove* ceph::profile::params::osds from the env files | 13:01 |
| *** egallen has joined #tripleo | 13:01 | |
| gfidente | we wanted to make sure people did the conversion | 13:01 |
| gfidente | becuse if they dont, ceph-ansibe might just lose all the data | 13:01 |
| openstackgerrit | Brent Eagles proposed openstack/puppet-tripleo master: Adding wrapper scripts for neutron agent subprocesses https://review.openstack.org/550224 | 13:01 |
| gfidente | so you have to convert it and remove the old one | 13:01 |
| Tengu | gfidente: no env file specify that, and I drop the hiera file in addition. | 13:01 |
| openstackgerrit | Brent Eagles proposed openstack/tripleo-heat-templates master: Generate and mount wrappers for neutron agent processes https://review.openstack.org/550823 | 13:01 |
| raildo | o/ | 13:02 |
| d0ugal | jaosorior: yup! | 13:02 |
| jaosorior | #startmeeting TripleO Security Squad | 13:02 |
| openstack | Meeting started Wed Mar 21 13:02:20 2018 UTC and is due to finish in 60 minutes. The chair is jaosorior. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:02 |
| *** openstack changes topic to " (Meeting topic: TripleO Security Squad)" | 13:02 | |
| openstack | The meeting name has been set to 'tripleo_security_squad' | 13:02 |
| gfidente | Tengu can I see your cmdline? | 13:02 |
| *** adarazs is now known as adarazs_afk | 13:02 | |
| jaosorior | Hey! d0ugal, lhinds, owalsh | 13:02 |
| Tengu | gfidente: 2s | 13:02 |
| Tengu | gfidente: basically I did that: http://paste.openstack.org/show/707512/ | 13:03 |
| jaosorior | So, today should be a shorter meeting than last time :D | 13:03 |
| d0ugal | :) | 13:03 |
| jaosorior | should I wait a bit more for other folks? or should we start already? | 13:03 |
| lhinds | I think we can kick off with d0ugal here now | 13:03 |
| jaosorior | Alright! | 13:03 |
| lhinds | mistral is the first topic | 13:03 |
| Tengu | jaosorior: oh, meeting? here? | 13:03 |
| jaosorior | #topic Mistral Secret Storage | 13:03 |
| *** openstack changes topic to "Mistral Secret Storage (Meeting topic: TripleO Security Squad)" | 13:03 | |
| d0ugal | apetrich, thrash, rbrady, toure ^ we are going to chat about mistral and secrets if you want to join. | 13:03 |
| Tengu | gfidente: do you take part in the meeting? | 13:03 |
| thrash | d0ugal: ack | 13:04 |
| jaosorior | Tengu: yes. It's the weekly Security Squad meeting | 13:04 |
| apetrich | oh dear | 13:04 |
| gfidente | Tengu security squad? | 13:04 |
| Tengu | jaosorior: oh. I'll go DM with gfidente then :) | 13:04 |
| rbrady | d0ugal: ack | 13:04 |
| *** cdearborn has joined #tripleo | 13:04 | |
| jaosorior | So, we've been talking a while about needing secret storage for mistral | 13:04 |
| jaosorior | This is due to the fact that we store a LOT of sensitive information there | 13:04 |
| jaosorior | the overcloud private keys and passwords namely | 13:05 |
| openstackgerrit | Tim Rozet proposed openstack/puppet-tripleo stable/queens: Fixes incorrect ownership of ODL TLS cert/key https://review.openstack.org/554909 | 13:05 |
| *** dprince has joined #tripleo | 13:05 | |
| jaosorior | Being TripleO an active user of mistral, I would like it to "beta" or take into use any solution that we have in mind | 13:05 |
| jaosorior | Also, having talked to thrash in the PTG, I also volunteer to help out on the coding side of mistral if more hands are needed. | 13:06 |
| jaosorior | But I would like to talk and understand what are the main challenges on this side | 13:06 |
| d0ugal | so, first I think we need to clarify exactly what is stored and why. | 13:07 |
| jaosorior | sure | 13:07 |
| d0ugal | Mistral has a database that is mostly in-flight only. We store all the heat parameters etc. while the workflow is being executed | 13:07 |
| d0ugal | They are then stored for 48 hours afterwards | 13:07 |
| d0ugal | Mistral does log lots of information, and parameters may be logged at times - but I think this has been reduced (or possibly stopped) | 13:08 |
| thrash | I think the more sensitive stuff is stored in a mistral environment, is it not? | 13:08 |
| d0ugal | thrash: no, it is stored in Swift now | 13:08 |
| thrash | d0ugal: ack | 13:08 |
| apetrich | d0ugal, parameters are logged in debug only now | 13:08 |
| apetrich | as with most sensitive info AFAIK | 13:09 |
| d0ugal | The only information stored in mistral long term is two different "environments" - blobs of json basically | 13:09 |
| d0ugal | These are the ssh keys for overcloud nodes, iirc | 13:09 |
| *** dpawar has joined #tripleo | 13:09 | |
| d0ugal | and .. | 13:09 |
| *** pkovar has joined #tripleo | 13:09 | |
| *** egallen has quit IRC | 13:09 | |
| jaosorior | d0ugal: which environments? | 13:10 |
| d0ugal | undercloud_ceilometer_snmpd_password and undercloud_db_password | 13:10 |
| d0ugal | tripleo.undercloud-config and "ssh_keys" | 13:10 |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 13:10 |
| d0ugal | They can be viewed with... | 13:10 |
| jaosorior | d0ugal: why do we specifically store those passwords in mistral and not swift? | 13:10 |
| d0ugal | $ mistral environment-get tripleo.undercloud-config | 13:10 |
| d0ugal | $ mistral environment-get ssh_keys | 13:10 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: Include connectivity check prepare scripts during FFU https://review.openstack.org/554914 | 13:11 |
| d0ugal | jaosorior: good question. Mostly for legacy reasoning I think. They could be moved to swift | 13:11 |
| owalsh | ssh_keys is the heat-admin key? | 13:11 |
| d0ugal | owalsh: I believe so, but I am not sure. | 13:11 |
| jaosorior | d0ugal: would be great if we would keep all the passwords in one place. So we can secure that one place at some point. | 13:11 |
| Tengu | (use gopass + gpg :D) | 13:11 |
| d0ugal | jaosorior: The tripleo.undercloud-config environment is related to the undercloud itself, rather than a plan - I think that is why it is in mistral. | 13:11 |
| d0ugal | jaosorior: +1 | 13:12 |
| *** udesale has joined #tripleo | 13:12 | |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-quickstart-extras master: Collect installed cron jobs https://review.openstack.org/554889 | 13:12 |
| d0ugal | I think the ssh_keys environment was added out of simplicity, we didn't have a better plan at the time. | 13:12 |
| jaosorior | thrash, apetrich does anybody know what ssh_keys actually is? is it the keys for heat-admin? | 13:12 |
| trozet | can another core help out with reviewing https://review.openstack.org/#/c/553788/1 please? | 13:13 |
| *** dpawar has quit IRC | 13:13 | |
| dtantsur | folks.. I know it may sound provocative, but is it possible to add configuration steps to a service template that are NOT written in puppet | 13:13 |
| dtantsur | ? | 13:13 |
| d0ugal | jaosorior: I can find out. | 13:13 |
| thrash | jaosorior: I think so. Would need to double check. | 13:13 |
| dtantsur | I don't really want to spend half of cycle doing a trivial thing like 'call a command, get its result' | 13:13 |
| d0ugal | or shadower and mandre would know if they are around | 13:13 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add ctlplane networking for routed networks https://review.openstack.org/547326 | 13:13 |
| jaosorior | either way, there's a private key there, which would be considered sensitive info. So we need to secure it somehow | 13:14 |
| d0ugal | jaosorior: +1 | 13:14 |
| hjensas | derekh: ^^ Can you have a look at the python script there? Make sure I don't mess up the ipv6 stuff again? | 13:14 |
| dtantsur | EmilienM: hey, maybe you know (re my question above) | 13:14 |
| jaosorior | d0ugal, thrash: One option would be to move all that to swift. And rely on swift encryption (which we don't have right now, but we could enable) | 13:14 |
| apetrich | jaosorior, during ping test (and I think tempest as well but not 100% sure) the keys to the created servers are stored in an env in mistral | 13:14 |
| thrash | jaosorior: +1000 | 13:15 |
| d0ugal | jaosorior: I didn't know swift had that option, sounds like a good (and easy?) starting point. | 13:15 |
| jaosorior | thrash, d0ugal, apetrich: Would you guys be able to dedicate some time to move those to swift? | 13:15 |
| thrash | jaosorior: somebody can, yes. :) | 13:15 |
| jaosorior | d0ugal, to be able to do that, we probably need barbican in the undercloud, but that's something alee and me can work on. | 13:16 |
| d0ugal | jaosorior: we are going to do some planning soon, so we could open a bug for this and consider it then | 13:16 |
| jaosorior | d0ugal, apetrich, thrash: So, having moved those environments to be stored in swift. Would that be the last bits of sensitive info stored in mistral? | 13:16 |
| owalsh | if it's only used the the pingtest/tempest key do we care? | 13:16 |
| thrash | jaosorior: I think from a tripleo perspective, that's a good bet. | 13:17 |
| apetrich | owalsh, not only those keys unfortunately | 13:17 |
| owalsh | apetrich: ack | 13:17 |
| jaosorior | owalsh: it sure depends on the user that pingtest/tempest uses. If it's heat-admin it's problematic, since it's able to do sudo su. | 13:17 |
| d0ugal | jaosorior: do you could storing for 48 hours as storing? :) | 13:18 |
| *** myoung|afk is now known as myoung | 13:18 | |
| owalsh | jaosorior: runs as stack AFAIK | 13:18 |
| d0ugal | jaosorior: we also probably need to do some checking of the logs and/or protection there against future leaks | 13:18 |
| jaosorior | d0ugal: I need to double check on that one. lhinds what do you think? | 13:18 |
| jaosorior | d0ugal: definitely | 13:18 |
| lhinds | jaosorior: just reading.. | 13:18 |
| lhinds | I guess time could be configurable for now (if that's what you were refering to) | 13:19 |
| *** masco has quit IRC | 13:19 | |
| lhinds | or log integrity? | 13:20 |
| jaosorior | Log integrity is something we should cover, so we should report any issues as mistral bugs and get those fixed. | 13:20 |
| jaosorior | lhinds: but currently mistral stores the heat environments (which might contain sensitive info) for a limited time (48 hours) | 13:20 |
| jaosorior | lhinds: is this something we can live with, or should we also avoid this? | 13:21 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add ctlplane networking for routed networks https://review.openstack.org/547326 | 13:21 |
| d0ugal | FWIW, fixing this in Mistral will likely be very hard. | 13:21 |
| *** jmelvin has joined #tripleo | 13:21 | |
| alee | o/ | 13:21 |
| lhinds | so it would be difficult to encrypt the heat envs? | 13:21 |
| lhinds | (stored in mistal) | 13:22 |
| d0ugal | lhinds: I think so, mistral internally duplicates them in a few places to optimize db lookup | 13:22 |
| lhinds | d0ugal: ack | 13:22 |
| jaosorior | d0ugal: I thought the generated heat environments were all stored in swift. | 13:23 |
| lhinds | so i think as far as time periods, any time window is a potential exploit window (although shorted better of course) | 13:23 |
| d0ugal | jaosorior: they are - but while the workflow is running and for 48 hours after they are also in Mistral | 13:23 |
| jaosorior | d0ugal: is it possible to disable that? | 13:23 |
| d0ugal | jaosorior: yes, they could be deleted when the workflow finishes, but it is extremely useful for debugging etc. | 13:24 |
| d0ugal | We actually increased the time, the default is 1 hour irrc | 13:24 |
| d0ugal | iirc* | 13:24 |
| *** ratailor has quit IRC | 13:24 | |
| jaosorior | d0ugal: how is it useful for debugging? | 13:24 |
| d0ugal | jaosorior: when the execution is stored you can inspect it and find out exactly what happened, what inputs and outputs happened at every point in the workflow | 13:25 |
| d0ugal | jaosorior: you can even restart workflows in the middle etc. | 13:25 |
| lhinds | has there been any BP / LP for encrypting heat envs stored in mistral (so it's on the radar so to speak). I could take a look at the code, can't promise anything as new to mistral | 13:25 |
| *** jpena|lunch is now known as jpena | 13:25 | |
| d0ugal | it is a bit like having the interactive debugger you have in most programming languages (but via a rest api :)) | 13:25 |
| jaosorior | d0ugal: What about making that attribute configurable? In the hardening docs we could then tell folks to lower that time, or disable it entirely. | 13:25 |
| lhinds | but with a key in barbican, it should be doable. | 13:26 |
| d0ugal | jaosorior: it is configured by instack-undercloud, can users change those puppet settings? | 13:26 |
| jaosorior | should be possible | 13:26 |
| jaosorior | depending on how it's configured | 13:26 |
| *** amoralej is now known as amoralej|lunch | 13:27 | |
| jaosorior | Need to double-check if the instack-undercloud hieradata takes precedence or the hieradata overrides do. but it should be doable. | 13:27 |
| d0ugal | lhinds: there was a blueprint for mistral for securing secrets. I think both rbrady and thrash had a look at doing it. So they know more about that than me. | 13:27 |
| jaosorior | #action For now, we will document how to lower the time mistral stores heat environments and add it to the hardening guide. | 13:28 |
| d0ugal | jaosorior: FYI, here is the setting: https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L671 | 13:28 |
| jaosorior | #link https://github.com/openstack/instack-undercloud/blob/master/elements/puppet-stack-config/puppet-stack-config.yaml.template#L671 | 13:28 |
| lhinds | d0ugal / rbrady / thrash if you manage to dig it out (the BP) please paste if for me. | 13:28 |
| d0ugal | lhinds: looking for it. | 13:28 |
| lhinds | thanks d0ugal | 13:29 |
| d0ugal | lhinds: https://blueprints.launchpad.net/mistral/+spec/mistral-secure-sensitive-data | 13:29 |
| lhinds | so configurable as first port of call, and then ideal future functionaility to encrypt | 13:29 |
| d0ugal | See the spec linked at the top and there was a patch, but I think that got stuck. | 13:29 |
| *** lucas-hungry is now known as lucasagomes | 13:29 | |
| lhinds | so there is a fair whack of code there, any reason for the abandon by Brad? | 13:30 |
| d0ugal | jaosorior: should I open a bug for the mistral environments? | 13:31 |
| jaosorior | d0ugal: that would be great | 13:31 |
| d0ugal | k, on it | 13:31 |
| *** chlong has quit IRC | 13:31 | |
| alee | d0ugal, I'm having trouble finding the actual spec .. | 13:32 |
| d0ugal | alee: https://specs.openstack.org/openstack/mistral-specs/specs/pike/approved/secure-sensitive-data.html | 13:32 |
| lhinds | alee: spec has gone missing, but some code here: | 13:32 |
| lhinds | https://review.openstack.org/#/c/459747/ | 13:32 |
| jaosorior | #link https://specs.openstack.org/openstack/mistral-specs/specs/pike/approved/secure-sensitive-data.html | 13:32 |
| alee | ah cool thanks | 13:32 |
| d0ugal | I think the spec was moved because it missed the openstack release | 13:32 |
| *** pkovar has quit IRC | 13:32 | |
| d0ugal | Which is a bad idea it seems :) | 13:32 |
| lhinds | k, found the spec: | 13:33 |
| jaosorior | Alright, but at least for the short term we have a plan | 13:33 |
| lhinds | #link https://github.com/openstack/mistral-specs/blob/master/specs/pike/approved/secure-sensitive-data.rst | 13:33 |
| jaosorior | * Move all sensitive data to swift (to have it all in one place) | 13:34 |
| lhinds | ok, brad is thrash, got it now | 13:34 |
| jaosorior | * Document how to reduce time mistral stores heat environments) | 13:34 |
| thrash | lhinds: :D | 13:34 |
| d0ugal | #link https://bugs.launchpad.net/tripleo/+bug/1757430 | 13:34 |
| openstack | Launchpad bug 1757430 in tripleo "The ssh_keys and tripleo.undercloud-config Mistral environments should be move to swift" [High,Confirmed] | 13:34 |
| *** pkovar has joined #tripleo | 13:34 | |
| jaosorior | and then we can focus on securing swift instead, which already can encrypt with barbican. | 13:34 |
| *** jlabarre has joined #tripleo | 13:34 | |
| jaosorior | d0ugal: awesome | 13:35 |
| jaosorior | thanks | 13:35 |
| *** pkovar has quit IRC | 13:36 | |
| d0ugal | np | 13:36 |
| jaosorior | Anything else someone wants to bring up about this topic? | 13:36 |
| lhinds | nothing from me this week | 13:37 |
| jaosorior | ok | 13:37 |
| openstackgerrit | Harald Jensås proposed openstack/python-tripleoclient master: Fix Genconfig - no HOME in environment https://review.openstack.org/554678 | 13:37 |
| *** adarazs_afk is now known as adarazs | 13:37 | |
| jaosorior | Thanks d0ugal, thrash and apetrich for joining | 13:37 |
| jaosorior | #topic Work progress udpate | 13:37 |
| *** openstack changes topic to "Work progress udpate (Meeting topic: TripleO Security Squad)" | 13:37 | |
| d0ugal | jaosorior: np, thanks for the input! | 13:38 |
| jaosorior | Just a heads up for folks in the squad, there are a bunch of reviews for different items in the etherpad https://etherpad.openstack.org/p/tripleo-security-squad (Maybe we need to come up with an easier way to track those) | 13:38 |
| jaosorior | so reviews are appreciated | 13:38 |
| jaosorior | Right now, most of the work that I've been doing has been on enabling TLS by default (which hopefully almost merges for the undercloud https://review.openstack.org/#/c/552382/ ) | 13:39 |
| *** pkovar has joined #tripleo | 13:39 | |
| jaosorior | I'm also working on enabling it by default in the overcloud, so if someone is intersted in joining that work or testing, let me know. | 13:39 |
| jaosorior | that's all on my side. | 13:40 |
| alee | jaosorior, I'll probably ping you about joining that work later today or tomorrow | 13:40 |
| jaosorior | alee: awesome | 13:40 |
| jaosorior | #topic Any other business | 13:41 |
| *** openstack changes topic to "Any other business (Meeting topic: TripleO Security Squad)" | 13:41 | |
| jaosorior | Anything else someone wants to bring up to the squad? | 13:41 |
| alee | jaosorior, I think we wanted to do a quick meeting to identify secrets to be secured/ passwords etc. | 13:41 |
| alee | jaosorior, did we want to schedule that? | 13:41 |
| jaosorior | alee: that would be good. | 13:41 |
| jaosorior | alee: Any day/time preference? | 13:42 |
| alee | jaosorior, how about tommorow? | 13:42 |
| jaosorior | works for me | 13:42 |
| openstackgerrit | Martin André proposed openstack/tripleo-common master: Pass connection info via ansible config file https://review.openstack.org/554526 | 13:42 |
| alee | morning my time -- say 10 am EST? | 13:42 |
| *** dtrainor has quit IRC | 13:42 | |
| jaosorior | alee: that works for me. 2pm UTC | 13:43 |
| jaosorior | lhinds: does that work for you? | 13:43 |
| lhinds | jaosorior: thats fine for me | 13:43 |
| lhinds | I have a work shop thing, but might be able to leave a little early | 13:44 |
| lhinds | (it's remote) | 13:44 |
| jaosorior | lhinds, alee: I'll poke you tomorrow then before the time. | 13:45 |
| jaosorior | Anybody else is welcome to join | 13:45 |
| jaosorior | Anything else someone would like to bring up? | 13:46 |
| jaosorior | Alright | 13:47 |
| jaosorior | thanks everyone for joining! | 13:47 |
| jaosorior | #endmeeting | 13:47 |
| *** openstack changes topic to "Welcome to Rocky. CI status - Promotions: Yellow; check/gate: Green; RDO CI jobs: Green | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 13:47 | |
| openstack | Meeting ended Wed Mar 21 13:47:17 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:47 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-03-21-13.02.html | 13:47 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-03-21-13.02.txt | 13:47 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tripleo_security_squad/2018/tripleo_security_squad.2018-03-21-13.02.log.html | 13:47 |
| *** psachin has quit IRC | 13:48 | |
| *** dtrainor has joined #tripleo | 13:48 | |
| *** jfrancoa has quit IRC | 13:50 | |
| *** ihrachys has joined #tripleo | 13:51 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Implement MasqueradeNetworks services https://review.openstack.org/553427 | 13:53 |
| openstackgerrit | Merged openstack/tripleo-ui stable/queens: Imported Translations from Zanata https://review.openstack.org/554806 | 13:53 |
| openstackgerrit | Merged openstack/tripleo-ui master: Imported Translations from Zanata https://review.openstack.org/554808 | 13:54 |
| *** dtrainor has quit IRC | 13:58 | |
| *** ktibi has quit IRC | 13:59 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: WIP: TLS by default for the overcloud https://review.openstack.org/554926 | 14:00 |
| *** cdearborn_ has joined #tripleo | 14:01 | |
| EmilienM | dtantsur: hello, have you found an answer to your question? | 14:02 |
| EmilienM | dtantsur: otherwise I can help | 14:02 |
| dtantsur | EmilienM: hi, no, I haven't. I'm wondering if puppet is still our only option to do things install-time | 14:04 |
| EmilienM | dtantsur: can you tell me exactly what you want to do? | 14:04 |
| dtantsur | EmilienM: I need to run an 'openstack' command, parse its output and based on it run another command | 14:05 |
| dtantsur | which is like 5-10 lines everywhere expect for puppet, where it requires you to have a PhD | 14:05 |
| dtantsur | :) | 14:05 |
| EmilienM | lol | 14:05 |
| mwhahaha | that's not a puppet problem | 14:05 |
| dtantsur | so, I ended up with https://review.openstack.org/554885 but it makes my eyes bleed | 14:05 |
| *** hjensas has quit IRC | 14:06 | |
| dtantsur | mwhahaha: well, it's a problem only in puppet, so yes, it IS a puppet problem | 14:06 |
| mwhahaha | or you could integrate it in a command | 14:06 |
| mwhahaha | so that no one has to do 5-10 lines | 14:06 |
| mwhahaha | so no, it's not a puppet problem | 14:06 |
| dtantsur | yes, it's mine problem, because I have to use puppet >_< | 14:06 |
| mwhahaha | why not do it in python and expose it ina single command | 14:06 |
| dtantsur | s/mine/my/ | 14:07 |
| mwhahaha | right so this is a common issue with openstack in that we provide a bunch of things that require an operator to wire info together | 14:07 |
| mwhahaha | and know what they need to do | 14:07 |
| dtantsur | we cannot really patch openstackclient for any pattern that can some up | 14:07 |
| mwhahaha | this is a recurring pattern which is awful | 14:08 |
| *** csmart has quit IRC | 14:08 | |
| * mwhahaha points to octavia | 14:08 | |
| dtantsur | as a side note: I have no clue why temporary URLs even need configuring.. a question for swift folks, I guess | 14:08 |
| dtantsur | anyway, I have to do something. I've been stuck with this for months.. | 14:09 |
| *** csmart has joined #tripleo | 14:10 | |
| *** salmankhan has quit IRC | 14:10 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 14:10 |
| *** dsariel has joined #tripleo | 14:11 | |
| mwhahaha | dtantsur: so what exactly is the issue with what you have in puppet other than providers are painful? | 14:11 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-quickstart master: Fix image_cache_expire_days https://review.openstack.org/554627 | 14:12 |
| dtantsur | mwhahaha: my problem is that some trivial things are very non-trivial (and yes.. providers are painful) | 14:12 |
| dtantsur | anyway, I'm open to any practical ideas to solve my problem | 14:12 |
| mwhahaha | dtantsur: the point of puppet is to allow us to do it idempotently which is generally not considered in any other methods | 14:12 |
| mwhahaha | so it's not that it's a puppet problem, it's a deficiency in other toolings | 14:12 |
| mwhahaha | ie shell has no idempotent concept | 14:13 |
| dtantsur | well, as idempotently as you implement it, which is not any different from other toolings | 14:13 |
| mwhahaha | ansible less so | 14:13 |
| * mwhahaha shrugs | 14:13 | |
| dtantsur | well, providers are only idempotent if you make them idempotent. just like you bash scripts, ansible playbooks, etc | 14:13 |
| dtantsur | anyway | 14:13 |
| mwhahaha | well we're working on it | 14:13 |
| mwhahaha | but we keep having to fix things for other people cause no one else is helping | 14:13 |
| dtantsur | as I said, I'm open to whatever you suggest on doing it, including reviewing my patch and telling me how wrong I am ;) | 14:14 |
| *** skramaja has quit IRC | 14:14 | |
| mwhahaha | dtantsur: would be useful to understand at a higher level what you're actually trying to do | 14:14 |
| mwhahaha | dtantsur: the blueprint indicates swift/glance but i'm not sure why this temp url stuff can't be implemented elsewhere | 14:14 |
| dtantsur | mwhahaha: automate bullet points 2 and 3 of http://tripleo.org/install/advanced_deployment/ansible_deploy_interface.html#enabling-temporary-urls | 14:15 |
| *** salmankhan has joined #tripleo | 14:15 | |
| *** itlinux has quit IRC | 14:16 | |
| *** itlinux has joined #tripleo | 14:17 | |
| *** cdearborn has quit IRC | 14:17 | |
| *** psahoo has joined #tripleo | 14:17 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: ironic/undercloud: align configuration with instack-undercloud https://review.openstack.org/550638 | 14:17 |
| *** gkadam_ has joined #tripleo | 14:18 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: undercloud: workaround for masquerading network in CI/OVB https://review.openstack.org/553620 | 14:18 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: roles: rename overcloud-prep-containers to prep-containers https://review.openstack.org/543014 | 14:18 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: prep-containers: include containerized undercloud bits https://review.openstack.org/543024 | 14:18 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: undercloud: add missing TLS environments when preparing containers https://review.openstack.org/545444 | 14:18 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: Remove adjust-interface-mtus script https://review.openstack.org/546216 | 14:18 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart-extras master: DO NOT REVIEW - Workarounds for containerized undercloud https://review.openstack.org/545628 | 14:18 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: undercloud: remove IronicInspectorCollectors in environment https://review.openstack.org/554302 | 14:18 |
| mwhahaha | dtantsur: i think the correct provider would be on the swift account | 14:18 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Undercloud: inspection_runbench, inspection_extras https://review.openstack.org/546626 | 14:18 |
| *** myoung is now known as myoung|rover | 14:19 | |
| dtantsur | mwhahaha: I *think* I'm doing something like that, but puppet providers make me cry | 14:19 |
| mwhahaha | dtantsur: so i think you're problem is that no one really touches puppet-swift as we have these types of concepts for properties in other providers already (like glance) | 14:19 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Move API cors config to their services https://review.openstack.org/554386 | 14:19 |
| mwhahaha | dtantsur: kinda yea but you're basically running into the fact no one has bothered to keep up for the last 4 years | 14:19 |
| *** salmankhan has quit IRC | 14:20 | |
| dtantsur | le sigh | 14:20 |
| *** salmankhan has joined #tripleo | 14:20 | |
| *** myoung|rover is now known as myoung|rover|mtg | 14:20 | |
| *** gkadam has quit IRC | 14:20 | |
| dtantsur | I'm cargo-culting stuff from nova, so maybe it'll be fine | 14:21 |
| mwhahaha | it won't be | 14:21 |
| dtantsur | \o/ | 14:21 |
| *** jaganathan has joined #tripleo | 14:21 | |
| mwhahaha | nova is probably the worst example :D | 14:21 |
| dtantsur | \o/ \o/ | 14:21 |
| * dtantsur jumps out of the window | 14:21 | |
| mwhahaha | dtantsur: you could ask the storage dfg to make it configurable :D | 14:21 |
| *** itlinux has quit IRC | 14:22 | |
| EmilienM | bogdando: thx for updating the patch, we'll see how that works | 14:22 |
| dtantsur | do we have somebody here understanding swift? | 14:22 |
| * mwhahaha points to cschwede | 14:22 | |
| mwhahaha | so how do we handle this on the undercloud | 14:23 |
| mwhahaha | or wait maybe we are doing it when we run the commands that rely on this | 14:23 |
| *** ktibi has joined #tripleo | 14:23 | |
| dtantsur | we're not doing this in the undercloud yet | 14:23 |
| jistr | matbu, chem: please when you have some time take a look at the pre_upgrade_rolling_tasks review https://review.openstack.org/#/c/552073 | 14:23 |
| dtantsur | but given that we create networks in bash.... | 14:24 |
| mwhahaha | dtantsur: we leverage temp urls | 14:24 |
| mwhahaha | in other ways | 14:24 |
| *** cshastri has joined #tripleo | 14:24 | |
| dtantsur | mmm, interesting | 14:24 |
| mwhahaha | let me see how we do this | 14:24 |
| cschwede | what's the issue with swift? | 14:24 |
| gfidente | dtantsur cargo-culting from nova | 14:24 |
| gfidente | dtantsur even you references are too much for me | 14:24 |
| mwhahaha | cschwede: dtantsur is unhappily trying to manage temp urls for an account | 14:24 |
| dtantsur | cschwede: yeah, I essentially wonder why I even have to create a temporary URL key myself.. | 14:25 |
| *** derekh has quit IRC | 14:25 | |
| mwhahaha | dtantsur: so we get away with it by handling it in the action that interacts with swift, https://github.com/openstack/tripleo-common/blob/master/scripts/upload-swift-artifacts#L135 | 14:25 |
| dtantsur | I understand why I may want to set it to something, but why not have a sane default? | 14:25 |
| matbu | jistr: ack | 14:25 |
| mwhahaha | dtantsur: so why can't the operation in ironic do it rather than it be preconfigured | 14:25 |
| dtantsur | mwhahaha: ah, I remember that. so yes, we do it in bash | 14:25 |
| *** ykarel is now known as ykarel|away | 14:25 | |
| *** jfrancoa has joined #tripleo | 14:25 | |
| mwhahaha | dtantsur: right but it's done at the usage point, so is there a reason it can't be done in the action calling swift | 14:26 |
| dtantsur | mwhahaha: because it will be racy, if I understand it right. imagine several conductors do it simultaneously | 14:26 |
| mwhahaha | dtantsur: or are you seting the key in a config | 14:26 |
| dtantsur | mwhahaha: we used to set it in the config, I've fixed it already | 14:26 |
| cschwede | dtantsur: because you actually might not want a key at all? if there is no key, the feature is not working, which is indeed sth some users want | 14:26 |
| cschwede | dtantsur: so depending on whom you ask, there are different "sane" defaults :) | 14:26 |
| dtantsur | cschwede: a weird way to disable a feature, if you ask me.. | 14:26 |
| openstackgerrit | mathieu bultel proposed openstack/python-tripleoclient master: WIP -- do not inherit converge from deploycommand class https://review.openstack.org/554934 | 14:26 |
| mwhahaha | dtantsur: are you new to openstack? :D | 14:27 |
| *** derekh has joined #tripleo | 14:27 | |
| dtantsur | ask chandankumar :D | 14:27 |
| mwhahaha | consistency is not our forte | 14:27 |
| *** bfournie has quit IRC | 14:27 | |
| cschwede | dtantsur: it's really up to the user, the operator typically enables it cluster-wide, and the user can decide if it is needed on an account or per-container basis | 14:27 |
| mwhahaha | neither is enabling/disabling features | 14:27 |
| *** nyechiel_ has quit IRC | 14:27 | |
| dtantsur | cschwede: well, $ openstack object store account set --temporary-urls-enabled would work so much better for me.. | 14:28 |
| dtantsur | I think the issue is merging two actions into one: enabling temporary URLs and setting the key | 14:28 |
| dtantsur | the former can be run idempotent, the latter, generally speaking, not | 14:28 |
| *** bfournie has joined #tripleo | 14:29 | |
| cschwede | but the amount of requests is the same? either i create a random key and set it, or it is set by default and i need to read it from the metadata? | 14:29 |
| dtantsur | cschwede: what happens if two ironic conductors generate a random key and try to set it? | 14:29 |
| dtantsur | at the same time? | 14:29 |
| mwhahaha | dtantsur: so if you just want to ensure it's set, it's a bootstrap exec command during the deployment | 14:30 |
| *** ykarel|away has quit IRC | 14:30 | |
| mwhahaha | dtantsur: it doesn't have to be puppet necessarily | 14:30 |
| cschwede | dtantsur: last one "wins" (assuming that the same time does not exist, there will be some microseconds between the requests) | 14:30 |
| dtantsur | mwhahaha: well, that's the question I started with: can I bypass puppet? :) | 14:30 |
| mwhahaha | dtantsur: but it comes down to preping swift correctly, but is it done on the udnercloud/overcloud | 14:30 |
| dtantsur | cschwede: right, so one conductor will end up with an invalid key, right? | 14:31 |
| cschwede | right | 14:31 |
| mwhahaha | dtantsur: well i wanted to know what you were actually trying to do :D | 14:31 |
| dtantsur | heh | 14:31 |
| dtantsur | mwhahaha: okay, so the "bootstrap exec command". what is it? do you have an example? | 14:31 |
| cschwede | so talking about swift on the undercloud, there is already a key set during deployment? that could be used? | 14:31 |
| mwhahaha | you focused on how much puppet sucks rather than explaing what you were actually trying to do | 14:31 |
| dtantsur | cschwede: it's on a different account | 14:31 |
| cschwede | ah, got it | 14:31 |
| mwhahaha | ie configure swift from a single host during the deployment | 14:31 |
| * dtantsur puts aside his opinion on puppet | 14:32 | |
| cschwede | dtantsur: there is already an action that creates the key in mistral, can't that be reused for the other account? | 14:32 |
| dtantsur | cschwede: we just discovered that its done in bash | 14:32 |
| cschwede | ie before puppet et al are running? | 14:32 |
| dtantsur | wait | 14:32 |
| dtantsur | how can we do anything with swift before swift is installed by puppet? | 14:33 |
| mwhahaha | dtantsur: are you trying to configure this on the overcloud or udndercloud | 14:33 |
| cschwede | dtantsur: oh sorry, i misunderstood. i thought it was after UC install | 14:33 |
| mwhahaha | dtantsur: i think that changes the conversation | 14:33 |
| dtantsur | mwhahaha: both. we can start with either, if it's easier | 14:33 |
| mwhahaha | well the solution may be different | 14:33 |
| mwhahaha | so it matters | 14:33 |
| mwhahaha | if you will need to do it on both, then we need a deployment solution | 14:34 |
| mwhahaha | if you need to do it on the undercloud only then we already have mistral actions to do i think | 14:34 |
| mwhahaha | anyway sec | 14:34 |
| dtantsur | at least on the undercloud. ideally, both. | 14:34 |
| mwhahaha | also containerized undercloud is probably the targeted solution right? | 14:34 |
| *** aputtur has quit IRC | 14:35 | |
| mwhahaha | or are you going to need to backport this | 14:35 |
| mwhahaha | if so then it has to be puppet | 14:35 |
| dtantsur | mwhahaha: no backports | 14:35 |
| mwhahaha | k | 14:35 |
| cschwede | dtantsur: so this one exists since Newton: https://github.com/openstack/tripleo-common/blob/master/tripleo_common/actions/swifthelper.py | 14:36 |
| mwhahaha | dtantsur: example of bootstrap_host_exec https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-api.yaml#L223-L237 | 14:36 |
| cschwede | dtantsur: which is used to set the tempurl key on the UC | 14:36 |
| dtantsur | cschwede: thanks | 14:36 |
| dtantsur | mwhahaha: nice. can it used overcloudrc credentials though? | 14:37 |
| mwhahaha | dtantsur: so it runs during the deployment on the host. not sure what's available in terms of creds | 14:37 |
| mwhahaha | dtantsur: so if it requires creds that's usually a post configuration of some sort | 14:38 |
| dtantsur | well, kind of, yes | 14:39 |
| mwhahaha | dtantsur: swift does something like... https://github.com/openstack/tripleo-heat-templates/blob/fefecf633ab42a9bf2e4fc95a5927db6e9a17153/docker/services/swift-proxy.yaml#L102-L126 | 14:39 |
| mwhahaha | to pull creds out of the config | 14:39 |
| mwhahaha | so if it has access to the creds you can craft a magical (terrible) shell script | 14:39 |
| dtantsur | aha! | 14:39 |
| *** egallen has joined #tripleo | 14:40 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: [FFU] Hook to allow user to pass a custom script for repo switching. https://review.openstack.org/539503 | 14:40 |
| *** amoralej|lunch is now known as amoralej | 14:40 | |
| dtantsur | mwhahaha: thanks, this is probably what I need. And it will prevent you from hearing more complaints about puppet, at least in the near future ;) | 14:42 |
| * dtantsur cannot guarantee absence of complaints about kolla though | 14:42 | |
| mwhahaha | :D | 14:42 |
| mwhahaha | trown|ruck:, myoung|rover|mtg: so do we have a bug for all the 3node tempest failures? seems to be ssh connection issues, is that an overlap of an existing bug? | 14:48 |
| mwhahaha | trozet: myoung|rover|mtg: example http://logs.openstack.org/29/550029/3/check/tripleo-ci-centos-7-3nodes-multinode/d413483/job-output.txt.gz#_2018-03-21_08_56_05_800194 | 14:48 |
| owalsh | mwhahaha: are my horrible hacky docker_config scripts reproducing? | 14:49 |
| mwhahaha | owalsh: you know it | 14:49 |
| trozet | mwhahaha: you mean trown^^^? | 14:49 |
| mwhahaha | i do | 14:49 |
| mwhahaha | trozet: unless you want to fi xit | 14:49 |
| trown|ruck | trozet: you take it | 14:49 |
| trown|ruck | trozet: you got this | 14:49 |
| trown|ruck | :) | 14:49 |
| trozet | trown|ruck: i dont think im qualified :) | 14:50 |
| mwhahaha | no one is qualified, we're all winging it | 14:50 |
| openstackgerrit | Andy Smith proposed openstack/tripleo-heat-templates master: Support separate oslo.messaging services for RPC and Notification https://review.openstack.org/507963 | 14:50 |
| * trown|ruck confirms this | 14:50 | |
| trown|ruck | mwhahaha: seems possibly related to https://bugs.launchpad.net/tripleo/+bug/1755485 ... but maybe not | 14:51 |
| openstack | Launchpad bug 1755485 in tripleo "Barbican tempest test failing to ssh to cirros image" [Critical,Triaged] | 14:51 |
| trown|ruck | mwhahaha: might not be just barbican that fails to ssh to cirros | 14:51 |
| mwhahaha | trown|ruck: it's possible | 14:51 |
| *** gyankum has quit IRC | 14:51 | |
| mwhahaha | it might be an ovs issue or something, would point to a multinode issue | 14:52 |
| trozet | hey guys what generates/controls logs in /var/log/containers? | 14:52 |
| mwhahaha | trozet: it's how the containers mount their logs i think | 14:52 |
| mwhahaha | i think we map /var/log/containers/<container>/ as /var/log | 14:53 |
| trozet | mwhahaha: whats the difference then between that and docker logs | 14:53 |
| trozet | mwhahaha: do services just output same logs in stdout and the file? | 14:53 |
| mwhahaha | trozet: container/ logs are usually service output logs | 14:53 |
| mwhahaha | which docker/ folder are you talking about? | 14:53 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-ui master: eslint: use as-needed for arrow-body-style https://review.openstack.org/546707 | 14:53 |
| trozet | mwhahaha: when you do docker logs <container> vs /var/log/containers/<service>/ | 14:54 |
| mwhahaha | docker logs <container> is stdout i think | 14:54 |
| owalsh | trozet: I think jaosorior added something to t-h-t to control whether we log to /var/log/containers/<serivce> or the docker logs (stdout/err) | 14:54 |
| owalsh | default is /var/log/containers/<service> | 14:55 |
| *** ykarel|away has joined #tripleo | 14:55 | |
| *** ykarel|away is now known as ykarel | 14:55 | |
| trozet | owalsh: yeah so for opendaylight, i only see docker logs work, theres nothing in /var/log/containers/opendaylight | 14:55 |
| trozet | owalsh: but i see other services there, so trying to figure out what is missing | 14:55 |
| *** jaganathan has quit IRC | 14:56 | |
| dtantsur | can someone please remind me how start_order works: do smaller values get executed first? | 14:56 |
| jaosorior | trozet: ultimately (when kubernetes comes) it would be better to mvoe to docker logs <service> | 14:56 |
| owalsh | dtantsur: smaller first, default is 0 IIRC. NB it's host scope | 14:57 |
| dtantsur | thnx | 14:57 |
| *** akane has quit IRC | 14:57 | |
| mwhahaha | trozet: do you have a docker/services/logging/files/opendaylight.yaml? | 14:57 |
| openstackgerrit | Merged openstack/instack-undercloud master: Enable TLS by default https://review.openstack.org/552382 | 14:57 |
| trozet | jaosorior: i see directories there for every service, some services have no logs in their directory though | 14:58 |
| mwhahaha | trozet: see https://github.com/openstack/tripleo-heat-templates/tree/107b610923ba5d39f90c3a6a63bf2d3642e1b35d/docker/services/logging/files | 14:58 |
| trozet | jaosorior: but there is no directory for ODL | 14:58 |
| jaosorior | trozet: I didn't do the patches for ODL. So Id on't really know why that was done. But ultimately if you can access them via docker logs <odl container name>, it's in the right direction :D | 14:58 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-quickstart-extras master: Collect installed cron jobs https://review.openstack.org/554889 | 14:58 |
| jaosorior | trown|ruck: could you check this out https://review.openstack.org/#/c/552781/ ? | 14:59 |
| *** yamahata has joined #tripleo | 14:59 | |
| trozet | jaosorior: we changed ODL to not output logs to a file anymore and only stdout so that docker logs works | 14:59 |
| *** agopi has joined #tripleo | 15:00 | |
| trown|ruck | jaosorior: moved it to top of my list | 15:00 |
| *** nyechiel_ has joined #tripleo | 15:00 | |
| trozet | jaosorior: so is it acceptable to not use this logging/files stuff in THT and just use docker logs? | 15:00 |
| jaosorior | trozet: well, I think it is. That's ultimately where we wanna go. | 15:01 |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/tripleo-common master: WIP: TLS by default for the overcloud https://review.openstack.org/554926 | 15:01 |
| rbrady | Workflows squad status meeting: https://etherpad.openstack.org/p/tripleo-workflows-squad-status | 15:01 |
| trozet | jaosorior: ok ty | 15:01 |
| rbrady | ^^ rbrady,d0ugal,apetrich,thrash,toure,jtomasek | 15:01 |
| *** dtrainor has joined #tripleo | 15:02 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-common master: Add and fix healthcheck scripts for Octavia services https://review.openstack.org/554946 | 15:02 |
| d0ugal | rbrady: omw | 15:02 |
| *** nyechiel_ has quit IRC | 15:03 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: Add support to ironic "direct" deploy interface https://review.openstack.org/529342 | 15:05 |
| *** mdnadeem has quit IRC | 15:06 | |
| *** cshastri has quit IRC | 15:08 | |
| *** agurenko has quit IRC | 15:08 | |
| *** moshele has quit IRC | 15:09 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 15:10 |
| *** wojdec has joined #tripleo | 15:11 | |
| *** thrash is now known as thrash|biab | 15:13 | |
| *** gkadam_ has quit IRC | 15:15 | |
| *** abishop has quit IRC | 15:19 | |
| *** etingof has quit IRC | 15:19 | |
| *** ukalifon has quit IRC | 15:20 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DNM - test containerized undercloud upgrades https://review.openstack.org/553633 | 15:26 |
| *** oscar has quit IRC | 15:27 | |
| *** egallen has quit IRC | 15:28 | |
| *** egallen has joined #tripleo | 15:30 | |
| *** kbyrne has quit IRC | 15:30 | |
| *** egallen has quit IRC | 15:30 | |
| *** agopi is now known as agopi|lunch | 15:31 | |
| openstackgerrit | Merged openstack/tripleo-quickstart-extras master: Ensure gated packages are installed during upgrade. https://review.openstack.org/549414 | 15:31 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: FFU: We need to be root to install ansible-pacemaker package. https://review.openstack.org/554887 | 15:31 |
| *** chem has quit IRC | 15:32 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-common master: [WIP] Activate another set of healthchecks https://review.openstack.org/550508 | 15:34 |
| *** psahoo has quit IRC | 15:34 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-common master: Add and fix healthcheck scripts for Octavia services https://review.openstack.org/554946 | 15:35 |
| *** liverpooler has joined #tripleo | 15:35 | |
| *** kbyrne has joined #tripleo | 15:35 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Remove unnecessary parameters from featureset047. https://review.openstack.org/553850 | 15:37 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Remove yum update from repo_cmd_after. https://review.openstack.org/554951 | 15:37 |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates stable/queens: [FFU] Hook to allow user to pass a custom script for repo switching. https://review.openstack.org/554953 | 15:39 |
| *** florianf_ has quit IRC | 15:39 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: WIP: Add undercloud upgrades playbook. https://review.openstack.org/553827 | 15:40 |
| *** florianf has joined #tripleo | 15:42 | |
| *** thrash|biab is now known as thrash | 15:43 | |
| thrash | mwhahaha: would you say this validation no longer serves a purpose? https://github.com/openstack/tripleo-common/blob/master/workbooks/validations.yaml#L274-L337 | 15:44 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-common master: [WIP] Activate another set of healthchecks https://review.openstack.org/550508 | 15:44 |
| mwhahaha | thrash: has it been moved to tripleo-validations? | 15:45 |
| thrash | mwhahaha: Was starting that work... But I don't even feel like it is necessary? | 15:46 |
| mwhahaha | thrash: i'm unsure, it's basically checking that the ironic stuff is properly loaded before it gets kicked off. I kinda think that's important | 15:46 |
| thrash | mwhahaha: Ack. I think I made the mistake of checking against an ovb env. :) | 15:47 |
| thrash | mwhahaha: I'll continue with what I was doing then. :D | 15:47 |
| mwhahaha | yes carry on :D | 15:47 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-docs master: WIP: Add FFU docs https://review.openstack.org/549892 | 15:48 |
| *** suuuper has quit IRC | 15:51 | |
| *** suuuper has joined #tripleo | 15:52 | |
| *** suuuper has quit IRC | 15:52 | |
| *** suuuper has joined #tripleo | 15:52 | |
| *** khrystoph has quit IRC | 15:53 | |
| openstackgerrit | Attila Darazs proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/472607 | 15:56 |
| *** paramite_ has quit IRC | 15:57 | |
| openstackgerrit | Merged openstack/tripleo-ui master: eslint: use as-needed for arrow-body-style https://review.openstack.org/546707 | 15:58 |
| *** yamahata has quit IRC | 16:00 | |
| *** khrystoph has joined #tripleo | 16:02 | |
| *** dparkes has quit IRC | 16:03 | |
| *** jlabarre has quit IRC | 16:07 | |
| *** itlinux has joined #tripleo | 16:07 | |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: WIP Fix up fence_compute parameters https://review.openstack.org/554975 | 16:08 |
| itlinux | hello all and good morning from Cali rainy day today! | 16:08 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Add purge to Nova cleanup cron. https://review.openstack.org/551966 | 16:09 |
| *** khrystoph has quit IRC | 16:10 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757474 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 16:10 |
| *** etingof has joined #tripleo | 16:10 | |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1757474 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset027-master fails with undefined groups.overcloud" [Critical,Triaged] | 16:10 |
| *** aufi has quit IRC | 16:11 | |
| *** itlinux has quit IRC | 16:15 | |
| *** waleedm has quit IRC | 16:16 | |
| *** yolanda_ has joined #tripleo | 16:19 | |
| *** yolanda has quit IRC | 16:19 | |
| alee | weshay, arxcruz looks like the mtu patch did not fix the barbican CIX issue | 16:21 |
| arxcruz | alee: :( | 16:21 |
| *** khrystoph has joined #tripleo | 16:21 | |
| arxcruz | alee: so, let's try to reproduce it again | 16:21 |
| *** derekh has quit IRC | 16:21 | |
| arxcruz | ans see if we can reach the root cause | 16:21 |
| *** derekh has joined #tripleo | 16:22 | |
| alee | arxcruz, any other ideas? yeah -- I'm going to run the reproducer script | 16:22 |
| arxcruz | alee: no ideas, once you have the env, let me know, i can digg a little bit | 16:22 |
| *** wolverineav has joined #tripleo | 16:22 | |
| alee | beagles, maybe you could take a look? https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/6e4ffba/undercloud/home/jenkins/tempest/tempest.html.gz | 16:22 |
| *** dsariel has quit IRC | 16:23 | |
| alee | same problem as before -- looks like we get an instance, attach a floating ip and a volume , and try to ssh to the instance and that fails. | 16:23 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Create vhost_socket_dir with proper permissions https://review.openstack.org/553788 | 16:23 |
| ccamacho | hey owalsh o/ its faster here :) so, the thing is that we can play with ruby on the puppet manifest, but at the end it should be translated to something like | 16:23 |
| ccamacho | * * * * * nova-manage db purge --before <here a date> and that date will be fixed.. maybe we will need another parameter like --expire <number> | 16:23 |
| alee | Failed to establish authenticated ssh connection | 16:23 |
| ccamacho | and the number can be translated to the current date - n days | 16:24 |
| *** kopecmartin has quit IRC | 16:24 | |
| alee | beagles, can't see any errors in the nova or neutron logs | 16:24 |
| *** khrystoph has quit IRC | 16:26 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade master: Include new CLI changes for overcloud update. https://review.openstack.org/550517 | 16:27 |
| *** trown|ruck has quit IRC | 16:28 | |
| *** suuuper has quit IRC | 16:28 | |
| *** dpawar has joined #tripleo | 16:28 | |
| alee | arxcruz, actually -- whats the password to connect supposed to be? | 16:29 |
| alee | 502 1326 ERROR barbican_tempest_plugin.tests.scenario.manager User: cirros, Password: None | 16:29 |
| *** yprokule has quit IRC | 16:30 | |
| arxcruz | alee: it's supposed to use ssh keys, not password, nevertherless the cirros password is cubswin:) | 16:31 |
| *** egallen has joined #tripleo | 16:32 | |
| alee | arxcruz, yeah - thats prob fine then .. | 16:32 |
| *** abishop has joined #tripleo | 16:33 | |
| *** ramishra has quit IRC | 16:37 | |
| *** hjensas has joined #tripleo | 16:38 | |
| *** karthiks has quit IRC | 16:38 | |
| *** pkovar has quit IRC | 16:40 | |
| *** thrash is now known as thrash|biab | 16:40 | |
| mwhahaha | ccamacho: you could just add a bash date generation command in the cron entry | 16:41 |
| *** dmacpher has quit IRC | 16:41 | |
| *** agopi|lunch has quit IRC | 16:41 | |
| ccamacho | mwhahaha \o/ yeah! | 16:41 |
| ccamacho | thanks! | 16:41 |
| *** agopi|lunch has joined #tripleo | 16:42 | |
| openstackgerrit | Jiri Stransky proposed openstack/python-tripleoclient stable/pike: Get message from websocket instead from zaqarclient directly https://review.openstack.org/554986 | 16:42 |
| mwhahaha | ccamacho: date +%Y-%m-%d -d "-7 days" | 16:42 |
| mwhahaha | seems to work | 16:42 |
| *** egallen has quit IRC | 16:42 | |
| openstackgerrit | Jiri Stransky proposed openstack/python-tripleoclient stable/pike: Get message from websocket instead from zaqarclient directly https://review.openstack.org/554986 | 16:45 |
| trozet | mwhahaha: do i need to recheck this (3rd party ci failure) or can you set workflow? https://review.openstack.org/#/c/554909/ | 16:47 |
| *** myoung|rover|mtg is now known as myoung | 16:48 | |
| mwhahaha | myoung|rover|mtg, weshay: rdo cloud die again? | 16:48 |
| beagles | alee, I wonder if there something wrong happening with the metadata | 16:48 |
| mwhahaha | beagles: are you looking into the ssh timeout thing? We're also seeing it ont he 3 node jobs | 16:48 |
| * myoung flips a coin and peers at mwhahaha | 16:48 | |
| arxcruz | lol | 16:48 |
| mwhahaha | myoung: dat reliability | 16:48 |
| beagles | mwhahaha, do we know if the VFms are actually coming up? | 16:49 |
| beagles | VMs | 16:49 |
| mwhahaha | beagles: not sure i hadn't looked, just noticed that we seem to be hitting something very similar to the barbican tempest problem in the 3 node jobs | 16:49 |
| beagles | I guess we wouldn't be getting authentication errors if they wren't | 16:49 |
| beagles | oic | 16:49 |
| beagles | mwhahaha, sorry I thought you meant generally :) so it is barbican specific | 16:50 |
| alee | beagles, yeah - I noticed that the metadata was not being retrieved .. | 16:50 |
| mwhahaha | well i wasn't sure if it was or not | 16:50 |
| *** itlinux has joined #tripleo | 16:50 | |
| beagles | mwhahaha, k | 16:50 |
| mwhahaha | trozet: i +A'd it. no need for the 3rd party on those | 16:50 |
| trozet | mwhahaha: ty | 16:51 |
| mwhahaha | beagles: http://logs.openstack.org/13/554213/1/check/tripleo-ci-centos-7-3nodes-multinode/85d6763/job-output.txt.gz#_2018-03-21_12_47_13_935784 example, i'll go poke at the nova logs | 16:51 |
| alee | mwhahaha, beagles sorry I'm confused -- did you say you were seeing this in other scenarios too - or just in the barbican test case? | 16:51 |
| mwhahaha | alee: i'm seeing an ssh timed out in 3 node jobs a bunch today | 16:52 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates master: Default environment/services/* to docker https://review.openstack.org/550060 | 16:52 |
| mwhahaha | alee: so if that's what you're seeing in the tempest results, then possibly | 16:52 |
| *** myoung is now known as myoung|food | 16:53 | |
| *** quiquell has quit IRC | 16:54 | |
| weshay | mwhahaha, not sure | 16:54 |
| mwhahaha | weshay: https://review.rdoproject.org/jenkins/job/gate-tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-queens/221/console wasn't looking promissing (failure from 30 mins ago) | 16:55 |
| mwhahaha | ResourceInError: resources.baremetal_server: Went to status ERROR due to "Message: No valid host was found. There are not enough hosts available., Code: 500" | 16:55 |
| *** karthiks has joined #tripleo | 16:55 | |
| weshay | k | 16:55 |
| weshay | mwhahaha, I'll check the tenant | 16:56 |
| mwhahaha | alee, beagles is this what you were talking about for failed metadata: http://logs.openstack.org/13/554213/1/check/tripleo-ci-centos-7-3nodes-multinode/85d6763/logs/subnode-3/var/log/containers/nova/nova-compute.log.txt.gz#_2018-03-21_12_45_21_480 | 16:57 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-docs master: WIP: Add docs for Q upgrade workflow https://review.openstack.org/535859 | 16:58 |
| mwhahaha | I see: failed to get http://169.254.169.254/2009-04-04/user-data | 16:58 |
| beagles | mwhahaha, yeah - I'm wondering if the ssh key isn't getting configured because metadata isn't available | 17:00 |
| *** etingof has quit IRC | 17:00 | |
| gfidente | therve was looking with fultonj into https://review.openstack.org/#/c/551920/ | 17:00 |
| gfidente | d0ugal ^^ | 17:00 |
| * mwhahaha checks against a successful job | 17:00 | |
| gfidente | I see you wrote there it can interrupt regular workflows | 17:00 |
| alee | right | 17:00 |
| gfidente | was trying to understand why that is? | 17:00 |
| mwhahaha | hmm i don't see the same output on success | 17:01 |
| mwhahaha | probably cause we don't call get console output | 17:02 |
| mwhahaha | so we don't bother logging it | 17:02 |
| mwhahaha | brilliant | 17:02 |
| *** udesale has quit IRC | 17:03 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart master: duplicate fs20 for libvirt https://review.openstack.org/554991 | 17:03 |
| *** hjensas has quit IRC | 17:05 | |
| mwhahaha | beagles: i'm seeing requests in the api-metadata log | 17:05 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Add purge to Nova cleanup cron. https://review.openstack.org/551966 | 17:06 |
| beagles | mwhahaha, mm well that's something | 17:06 |
| mwhahaha | the last time this happened the undercloud was serving the metadata up | 17:06 |
| * mwhahaha checks it's not that again | 17:06 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-common master: Force ANSIBLE_LOAD_CALLBACK_PLUGINS to False for collect_nodes_uuid https://review.openstack.org/552636 | 17:06 |
| mwhahaha | beagles: for example in the job i'm looking at, http://logs.openstack.org/13/554213/1/check/tripleo-ci-centos-7-3nodes-multinode/85d6763/logs/subnode-3/var/log/containers/nova/nova-api-metadata.log.txt.gz#_2018-03-21_12_40_21_489 | 17:07 |
| *** salmankhan has quit IRC | 17:08 | |
| *** marios has quit IRC | 17:08 | |
| *** marios has joined #tripleo | 17:08 | |
| *** trown has joined #tripleo | 17:09 | |
| beagles | mwhahaha, ack | 17:10 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757474 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1757474 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset027-master fails with undefined groups.overcloud" [Critical,Triaged] | 17:10 |
| *** panda is now known as panda|off | 17:10 | |
| *** trown is now known as trown|lunch | 17:11 | |
| EmilienM | bogdando: I probably missed something but I don't see tripleo-undercloud-passwords.yaml generated anymore | 17:11 |
| openstackgerrit | Carlos Camacho proposed openstack/tripleo-heat-templates master: Add purge to Nova cleanup cron. https://review.openstack.org/551966 | 17:11 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/puppet-tripleo stable/queens: Replace perl with awk https://review.openstack.org/554599 | 17:12 |
| weshay | mwhahaha, when you have moment of clarity and peace, I would like to destroy that by running that bug about network-isolation by you | 17:12 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/puppet-tripleo stable/pike: Replace perl with awk https://review.openstack.org/554993 | 17:12 |
| mwhahaha | weshay: pfft clarity is overrated | 17:12 |
| *** hjensas has joined #tripleo | 17:12 | |
| *** hjensas has quit IRC | 17:12 | |
| *** hjensas has joined #tripleo | 17:12 | |
| mwhahaha | weshay: whatcha got | 17:12 |
| EmilienM | bogdando: I think that's because of https://review.openstack.org/#/c/542875 | 17:12 |
| *** salmankhan has joined #tripleo | 17:12 | |
| EmilienM | bogdando: it broke the undercloud upgrades to be containerized | 17:13 |
| weshay | mwhahaha, so we've been poking at this by turning net-iso on/off https://bugs.launchpad.net/tripleo/+bug/1757111 | 17:13 |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 17:13 |
| weshay | mwhahaha, yatin has some interesting comments to read through on https://review.openstack.org/#/c/554528/ | 17:13 |
| bogdando | EmilienM: ._. | 17:13 |
| *** dsneddon has quit IRC | 17:13 | |
| EmilienM | bogdando: I think there are things not backward compatible in that patch | 17:14 |
| mwhahaha | weshay: i would like to understand why we're now making net-iso required (it wasn't previously) so something's changed and probably not for the better | 17:14 |
| weshay | mwhahaha, we're not trying to make it required | 17:14 |
| EmilienM | bogdando: but tripleo-undercloud-passwords.yaml is no more handled on ~ directory | 17:14 |
| bogdando | EmilienM: let's revert then | 17:15 |
| weshay | mwhahaha, what we noticed is that w/ net-iso everything works.. everything being a full tempest run.. w/o net-iso we see networking issues that cause a lot of tempest failures.. around 50 | 17:15 |
| bogdando | I'm not also sure what the comment in https://review.openstack.org/#/c/542875/47/tripleoclient/constants.py means | 17:15 |
| weshay | mwhahaha, so in the interest of keeping non-net-iso deployments working in queens, I'm bringing this to your attention | 17:15 |
| weshay | queens/master | 17:15 |
| mwhahaha | weshay: what do networking folks say? | 17:16 |
| weshay | what ever happened, happend recently.. | 17:16 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Allow for passing boot-time vars/args to OC nodes https://review.openstack.org/552967 | 17:16 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Add an openshift-cns service https://review.openstack.org/543933 | 17:16 |
| weshay | I'll go get them fair point | 17:16 |
| EmilienM | bogdando: the file is now on .undercloud-heat-installer/tripleo-undercloud-passwords.yaml | 17:16 |
| mwhahaha | weshay: cause it seems to be a regression somewhere, i'm wondering if it's the same probelm we're seeing with the ssh stuff | 17:16 |
| bogdando | EmilienM: would a small symlink patch restored the backwards compat then? | 17:16 |
| EmilienM | bogdando: why did you put files into .undercloud-heat-installer directory? | 17:17 |
| EmilienM | and not HOME ? | 17:17 |
| bogdando | EmilienM: it comes from the comments | 17:17 |
| bogdando | and proposals... | 17:17 |
| bogdando | and my imagination :D | 17:17 |
| bogdando | wrt the implementation | 17:18 |
| *** zoli is now known as zoli|gone | 17:18 | |
| *** zoli|gone is now known as zoli | 17:18 | |
| EmilienM | bogdando: well, the tripleo-undercloud-passwords.yaml generated is no longer based on existing ~/undercloud-passwords.conf | 17:18 |
| EmilienM | and it breaks upgrades | 17:18 |
| *** marios has quit IRC | 17:19 | |
| mwhahaha | weshay: did you know that the dhcp client stuff was udpated recently in centos (wonder if related) | 17:19 |
| owalsh | ccamacho: yea, +1, was just about to suggest what mwhahaha already had | 17:20 |
| *** chem has joined #tripleo | 17:20 | |
| weshay | I did not know that | 17:20 |
| mwhahaha | weshay: in comparing 77 vs 78 for queens | 17:20 |
| mwhahaha | the dhcp client stuff changed | 17:20 |
| *** dsneddon has joined #tripleo | 17:20 | |
| mwhahaha | weshay: https://www.diffchecker.com/KCbL3JUz | 17:21 |
| EmilienM | bogdando: what do we do? | 17:24 |
| *** hjensas has quit IRC | 17:25 | |
| mwhahaha | weshay: though that's supposed to just be branding, i guess the next step is to go through the few components and see if there is anything that's changed in those | 17:25 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Make running workflows more robust https://review.openstack.org/549751 | 17:25 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Make Validation actions use startWorkflow https://review.openstack.org/550086 | 17:25 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Make Nodes workflow actions use startWorkflow https://review.openstack.org/550232 | 17:25 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Refactor RolesActions to use startWorkflow https://review.openstack.org/552532 | 17:25 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Refactor LoggerActions to use startWorkflow https://review.openstack.org/552545 | 17:25 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Refactor PlansActions to use startWorkflow https://review.openstack.org/552638 | 17:25 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Ignore empty values for dlrn hashes https://review.openstack.org/554882 | 17:26 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart master: Remove unnecessary parameters from featureset047. https://review.openstack.org/553850 | 17:26 |
| d0ugal | gfidente: what's up? | 17:26 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-quickstart-extras master: WIP: Add undercloud upgrades playbook. https://review.openstack.org/553827 | 17:26 |
| *** bogdando has quit IRC | 17:26 | |
| *** ykarel is now known as ykarel|afk | 17:26 | |
| gfidente | d0ugal https://review.openstack.org/#/c/552452/ why is it affecting regular workflows too? | 17:27 |
| mwhahaha | gfidente: it needed to be on another queue | 17:27 |
| mwhahaha | gfidente: otherwise i think it's injecting a failure | 17:27 |
| d0ugal | gfidente: Yeah, I think it just meant the messages were confusing. | 17:28 |
| gfidente | mwhahaha yeah but my point is, it seems to be affecting workflows which don't use the zaqar queue | 17:28 |
| d0ugal | agreed | 17:28 |
| d0ugal | but really, clients should filter by execution id in the workflow messages :) | 17:28 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Add space above Edit Configuration button https://review.openstack.org/554501 | 17:28 |
| d0ugal | that is what tripleoclient does | 17:28 |
| *** rbowen has quit IRC | 17:28 | |
| *** agopi|lunch is now known as agopi| | 17:28 | |
| *** agopi| is now known as agopi | 17:28 | |
| gfidente | d0ugal wait, I am saying that the ceph-ansible workflow, which does not use any zaqar queue | 17:29 |
| *** rbowen has joined #tripleo | 17:29 | |
| weshay | mwhahaha, yatin pointed out https://review.openstack.org/#/c/548554/ /me checking the rpms in the working job | 17:29 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/pike: DNM: Test undercloud upgrades P->Q. https://review.openstack.org/554996 | 17:29 |
| gfidente | was affected by the issue caused by the cron trigger on a completely different workflow | 17:29 |
| d0ugal | gfidente: indeed, it shouldn't impact that workflow | 17:29 |
| mwhahaha | weshay: that wason't merged on queens until 7 days ago https://review.openstack.org/#/c/550965 | 17:29 |
| mwhahaha | weshay: not likely the cause of the master issues | 17:29 |
| d0ugal | gfidente: are you telling me it was affected? | 17:30 |
| d0ugal | gfidente: or are you asking me if it was | 17:30 |
| *** florianf has quit IRC | 17:30 | |
| gfidente | d0ugal I think it was | 17:30 |
| ccamacho | owalsh thanks :) | 17:30 |
| d0ugal | gfidente: if it was affected, can you give me more details? in what way? | 17:30 |
| *** holser__ has quit IRC | 17:30 | |
| d0ugal | gfidente: I gotta run in a minute, but I'd like to look into it. because it 100% shouldn't have been affected :) | 17:30 |
| gfidente | d0ugal ack, we might be able to collect logs | 17:31 |
| mwhahaha | it's likely that it broke the calling workflow and not actually the ceph one | 17:31 |
| *** lucasagomes is now known as lucas-afk | 17:31 | |
| mwhahaha | so it breaks the deployment one | 17:31 |
| gfidente | mwhahaha yeah which is probably overcloud_deploy | 17:31 |
| mwhahaha | right | 17:31 |
| gfidente | but in the engine log I saw the req- for ceph-install fail | 17:31 |
| gfidente | timing out after execution | 17:31 |
| gfidente | even though ansible-playbook returned 0 | 17:31 |
| gfidente | I'll see if I can collect good logs | 17:32 |
| *** jpich has quit IRC | 17:33 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Change default output-dir to be $HOME https://review.openstack.org/554997 | 17:34 |
| *** moshele has joined #tripleo | 17:34 | |
| mwhahaha | weshay: so i'm not seeing anything that sticks out in that diff for queens between 77 and 78 which makes me thing it's probably more rdo cloud than anything | 17:35 |
| weshay | mwhahaha, I can recreate the issue outside of rdo-cloud | 17:35 |
| mwhahaha | weshay: that would also help explain why net-iso vs non-net-iso solves it | 17:35 |
| weshay | mwhahaha, via libvirt | 17:35 |
| mwhahaha | orly | 17:36 |
| mwhahaha | weshay: i wonder if i this is fallout from the docker iptables stuff | 17:36 |
| weshay | mwhahaha, I have two libvirt deployments for fs20 one w/ net-iso one w/o and the deployment w/o fails | 17:36 |
| mwhahaha | weshay: if so i would like to punch peoples | 17:36 |
| *** dpawar has quit IRC | 17:37 | |
| *** NobodyCam has quit IRC | 17:37 | |
| *** Tyrantelf_ has quit IRC | 17:37 | |
| *** Hazelesque has quit IRC | 17:38 | |
| *** Hazelesque has joined #tripleo | 17:38 | |
| *** alee_ has joined #tripleo | 17:38 | |
| *** v1k0d3n has quit IRC | 17:38 | |
| *** Tyrantelf has joined #tripleo | 17:39 | |
| *** alee has quit IRC | 17:39 | |
| *** andreaf has quit IRC | 17:39 | |
| *** NobodyCam has joined #tripleo | 17:39 | |
| *** andreaf_ has joined #tripleo | 17:39 | |
| *** v1k0d3n has joined #tripleo | 17:40 | |
| *** moshele has quit IRC | 17:41 | |
| mwhahaha | weshay: so do the neutron folks have anything to say why the port binding is failing, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/4cf9425/overcloud-novacompute-bar-0/var/log/containers/nova/nova-compute.log.txt.gz#_2018-03-21_07_03_53_855 | 17:41 |
| *** andreaf_ is now known as andreaf | 17:41 | |
| d0ugal | gfidente, mwhahaha - it shouldn't break the calling workflow either - workflows don't read from zaqar (well, other than the UI logging one) | 17:41 |
| d0ugal | gfidente: I'll inspect logs tomorrow :) | 17:41 |
| d0ugal | and I need to read the bug, I never fully understood the change | 17:42 |
| mwhahaha | it was a side effect of i think how we changes the reading of teh queue | 17:42 |
| mwhahaha | or something where it gets an error it just pukes | 17:42 |
| gfidente | d0ugal which hopes had I to understand it then | 17:42 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Change default output-dir to be $HOME https://review.openstack.org/554997 | 17:42 |
| d0ugal | mwhahaha: yeah, but that still doesn't make sense to me :) | 17:43 |
| gfidente | I support ehe pukes idea though | 17:43 |
| d0ugal | anyway, I really gotta run - guests arrived at my house | 17:43 |
| gfidente | tell them | 17:43 |
| gfidente | about it | 17:43 |
| mwhahaha | so it may not be the workflow that dies, but the client thinks it fails and then nukes things | 17:43 |
| mwhahaha | gfidente: he might like the person, i wouldn't subject anyone i know to our problems | 17:43 |
| *** myoung|food is now known as myoung | 17:45 | |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DNM - test containerized undercloud upgrades https://review.openstack.org/553633 | 17:46 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-upgrade master: DNM - containerized undercloud upgrade https://review.openstack.org/553629 | 17:47 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-upgrade master: DNM - containerized undercloud upgrade https://review.openstack.org/553629 | 17:47 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: DNM - test containerized undercloud upgrades https://review.openstack.org/553633 | 17:48 |
| *** thrash|biab is now known as thrash | 17:49 | |
| mwhahaha | weshay: honestly it looks like openvswitch problems | 17:49 |
| weshay | ya | 17:49 |
| mwhahaha | weshay: i traced it from nova, to neutron and to the metadata agent and there's some errors in ovs-vswitchd | 17:50 |
| *** ffiore_ has quit IRC | 17:53 | |
| weshay | i see this on the compute node | 17:53 |
| weshay | Mar 21 06:36:44 overcloud-novacompute-bar-0 ovs-vsctl[18050]: ovs|00001|db_ctl_base|ERR|unix:/var/run/openvswitch/db.sock: database connection failed (No such file or directory) | 17:53 |
| EmilienM | dprince, mwhahaha : when you have time please take a look at https://review.openstack.org/#/c/550608/5/doc/source/install/containers_deployment/3rd_party.rst | 17:53 |
| *** pickle has quit IRC | 17:53 | |
| *** pickle has joined #tripleo | 17:53 | |
| weshay | hrm... but that is containerized now | 17:54 |
| mwhahaha | weshay: openvswitch is not containerized | 17:55 |
| mwhahaha | never has been | 17:55 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Add environment to enable Designate https://review.openstack.org/555006 | 17:56 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Deploy Designate in scenario003 https://review.openstack.org/555007 | 17:56 |
| weshay | oh sorry.. was looking at the agent.. not the service | 17:56 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Begin adding environments with all params for a service https://review.openstack.org/475924 | 17:56 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Add ability to generate an environment index https://review.openstack.org/491925 | 17:56 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: WIP: Add service config env with all Designate settings https://review.openstack.org/555008 | 17:56 |
| beagles | weshay, mwhahaha, is OVS not running on the. server? | 17:56 |
| *** haleyb has quit IRC | 17:56 | |
| *** ebarrera has quit IRC | 17:56 | |
| mwhahaha | beagles: which server | 17:56 |
| beagles | compute | 17:56 |
| beagles | just wondering from weshay's comment | 17:57 |
| weshay | openvswitch-2.8.2-1.el7.x86_64 | 17:57 |
| weshay | openvswitch-ovn-central-2.8.2-1.el7.x86_64 | 17:57 |
| weshay | openvswitch-ovn-common-2.8.2-1.el7.x86_64 | 17:57 |
| weshay | openvswitch-ovn-host-2.8.2-1.el7.x86_64 | 17:57 |
| dprince | EmilienM: ack | 17:57 |
| weshay | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/4cf9425/overcloud-novacompute-bar-0/var/log/extra/rpm-list.txt.gz | 17:57 |
| mwhahaha | beagles: i don't see logs for it | 17:57 |
| mwhahaha | so maybe | 17:57 |
| beagles | mmm... | 17:57 |
| mwhahaha | it's not | 17:58 |
| beagles | I predict things will not be happy if there is no OVS running on the compute ;) | 17:58 |
| * mwhahaha wishes we had some sort of basic service validation before we did anything else | 17:59 | |
| * beagles nods | 17:59 | |
| weshay | don't see it running https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/4cf9425/overcloud-novacompute-bar-0/var/log/extra/pstree.txt.gz | 17:59 |
| mwhahaha | yea i don't see it in the sysctl service list from host_info either | 17:59 |
| mwhahaha | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/4cf9425/overcloud-novacompute-bar-0/var/log/host_info.txt.gz | 18:00 |
| *** derekh has quit IRC | 18:00 | |
| mwhahaha | where did it go | 18:00 |
| Tengu | hello guys :) | 18:01 |
| weshay | best data I have atm.. is to compare w/ pike | 18:02 |
| weshay | https://logs.rdoproject.org/openstack-periodic-24hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/5c746b0/overcloud-novacompute-bar-0/var/log/extra/pstree.txt.gz | 18:02 |
| weshay | and it's there | 18:02 |
| weshay | https://logs.rdoproject.org/openstack-periodic-24hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/5c746b0/overcloud-novacompute-bar-0/var/log/openvswitch/ | 18:02 |
| beagles | ah whew | 18:02 |
| beagles | obw | 18:02 |
| beagles | you are showing pike | 18:03 |
| dprince | EmilienM: nice job on the new docs btw. My -1 is really for minor things, just so you notice it | 18:03 |
| Tengu | oh, hello beagles :). do you have a few minutes for a network issue? | 18:03 |
| weshay | beagles, ya.. that was my only reference to something working atm | 18:03 |
| EmilienM | dprince: ok good, I'll look | 18:03 |
| beagles | weshay, ah oka | 18:03 |
| beagles | looks like ovs isn't being started | 18:03 |
| beagles | Tengu, depends :) | 18:03 |
| Tengu | beagles: doing an *upgrade* on pike (pike BM -> pike container - upgrade, not update), network seems broken on the three controllers I have. I heard you might know about it. | 18:04 |
| Tengu | beagles: if not, it's not a big issue, since apparently restarting the "network" service on the three controllers seems to correct the situation. | 18:05 |
| beagles | Tengu, interesting | 18:05 |
| *** pblaho has quit IRC | 18:05 | |
| Tengu | beagles: I know this usage isn't supported and it's kind of weird, but… ;) | 18:05 |
| Tengu | beagles: I just respawned my lab in order to re-run the upgrade process - if you're really interested, I can come back tomorrow with logs. | 18:06 |
| beagles | Tengu, not sure exactly what would cause that. The thing I'm looking at mainly has to do with process lifetimes and containers | 18:06 |
| beagles | Tengu, I don't think restarting networking would effect it | 18:06 |
| *** quiquell has joined #tripleo | 18:06 | |
| beagles | Tengu, it's worth reporting/cataloging ! | 18:07 |
| Tengu | beagles: hmm ok. well, symptoms: controllers can't ping their default route anymore, nor do DNS resolutions. fun part, management network still work - I can ssh from the undercloud. | 18:07 |
| Tengu | beagles: so I'll let the script run and crash, and report back tomorrow the logs. What kind of logs would you need? /var/log/messages I guess, and… ? | 18:08 |
| Tengu | neutron maybe? | 18:08 |
| beagles | Tengu, yeah, sounds about right | 18:09 |
| Tengu | ok. so stay tuned :). | 18:09 |
| Tengu | preparing the lab and fire. | 18:09 |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-heat-templates master: Containerize Neutron LBaaS service plugin https://review.openstack.org/555011 | 18:09 |
| Tengu | (it's soooo good to have a lab that can be respawned at will…) | 18:09 |
| mwhahaha | weshay: i thought ovs was enabled on the image | 18:09 |
| EmilienM | mwhahaha: do you hav ea package diff from before (working) to now? | 18:09 |
| mwhahaha | EmilienM: no cause someone broke logging | 18:10 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757474 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1757474 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset027-master fails with undefined groups.overcloud" [Critical,Triaged] | 18:10 |
| EmilienM | let me see RDO history | 18:10 |
| mwhahaha | (of course they did) | 18:10 |
| weshay | mwhahaha, ya.. logging here is killing this | 18:10 |
| mwhahaha | so at the moment it looks like ovs isn't running on the compute node | 18:10 |
| mwhahaha | i thought that was enabled by default from the image | 18:10 |
| mwhahaha | so i'm trying to track down where that's handled | 18:10 |
| EmilienM | so we updated in QUeens: https://review.rdoproject.org/r/#/c/12580/ | 18:11 |
| EmilienM | but looking for pike now | 18:11 |
| *** quiquell has quit IRC | 18:11 | |
| mwhahaha | in pike we see https://logs.rdoproject.org/openstack-periodic-24hr/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-pike/5c746b0/overcloud-novacompute-bar-0/var/log/journal.txt.gz#_Mar_21_05_10_49 | 18:11 |
| mwhahaha | but iun queens it's missing, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset020-queens/4cf9425/overcloud-novacompute-bar-0/var/log/journal.txt.gz#_Mar_21_06_30_30 | 18:12 |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-quickstart-extras master: Add support to mixed upgrade for overcloud-prep-container role. https://review.openstack.org/540473 | 18:13 |
| EmilienM | why do we have openvswitch-2.8.2-1.el7.x86_64 ? | 18:13 |
| *** jpena is now known as jpena|off | 18:14 | |
| EmilienM | we should have openvswitch-2.7.3-1.1fc27.el7 | 18:14 |
| EmilienM | if I read rdoinfo | 18:14 |
| mwhahaha | we bumped to 2.8 a while ago | 18:14 |
| EmilienM | ah nevermind, I read queens logs | 18:14 |
| EmilienM | the issue is only on pike right? | 18:14 |
| weshay | ya.. in queens.. we don't see Mar 21 05:10:49 localhost.localdomain ovs-ctl[743]: Creating empty database /etc/openvswitch/conf.db [ OK ] | 18:15 |
| mwhahaha | EmilienM: No queens + master | 18:15 |
| EmilienM | ahh | 18:15 |
| *** dtantsur is now known as dtantsur|afk | 18:15 | |
| mwhahaha | network templates regression? | 18:16 |
| mwhahaha | does os-net-config enable ovs? | 18:16 |
| *** atoth has quit IRC | 18:16 | |
| beagles | it might as a side effect | 18:17 |
| weshay | mwhahaha, I have a box w/ this | 18:18 |
| beagles | ifup-ovssystem or something | 18:18 |
| weshay | if you want to jump on | 18:18 |
| mwhahaha | weshay: sure | 18:18 |
| beagles | ifup-ovs | 18:18 |
| *** nyechiel_ has joined #tripleo | 18:19 | |
| *** trown|lunch is now known as trown | 18:19 | |
| mwhahaha | weshay: so that one has the service running | 18:20 |
| mwhahaha | oh wait it failed | 18:20 |
| *** EmilienM is now known as mimi | 18:21 | |
| *** mimi is now known as EmilienM | 18:21 | |
| *** gfidente is now known as gfidente|afk | 18:22 | |
| trown | lol @mimi | 18:22 |
| trown | that is my Mom's grandma name | 18:22 |
| weshay | same | 18:22 |
| weshay | trown, we're live debugging the fs20 issue | 18:22 |
| weshay | if you want to jump on | 18:22 |
| trown | sure | 18:22 |
| mwhahaha | wat the hell | 18:23 |
| mwhahaha | i wonder if this is related to the permissions changes we did for ODL | 18:23 |
| * mwhahaha looks at trozet | 18:24 | |
| mwhahaha | or was it sriov | 18:24 |
| mwhahaha | sec let me go look for the patch | 18:24 |
| trozet | mwhahaha: what's up? | 18:24 |
| weshay | trozet, openvswitch is not starting on the compute nodes | 18:25 |
| mwhahaha | trozet: did we change something around vswitch for ODL or was that nfv | 18:25 |
| *** jfrancoa has quit IRC | 18:25 | |
| * beagles vaguely recalls something nfv related | 18:25 | |
| trozet | weshay, mwhahaha: there was a bug where neutron certs were not created on compute nodes so neutron-openvswitch-agent wouldnt start, we fixed that though | 18:25 |
| trozet | weshay: openvswitch wont start or neutron-ovs agent? | 18:26 |
| weshay | openvswitch | 18:26 |
| trozet | weshay: TLS deployment or no? | 18:26 |
| weshay | ah good question | 18:27 |
| mwhahaha | Mar 21 18:22:03 overcloud-novacompute-bar-0 ovsdb-server[129745]: ovs|00005|ovsdb_jsonrpc_server|ERR|punix:/var/run/openvswitch/db.sock: listen failed: Is a directory | 18:27 |
| mwhahaha | wonder if the socket is a folder which is causing the problems | 18:27 |
| trown | hmm like somethiing mkdiring var/run/openvswitch/db.sock ? | 18:28 |
| weshay | this is not tls afaict | 18:28 |
| mwhahaha | docker will create it as a folder if it doesn't exist | 18:28 |
| mwhahaha | i vaguely remember something abotu this | 18:28 |
| trozet | mwhahaha: i thought OVS is not containerized? | 18:28 |
| mwhahaha | it's not | 18:28 |
| mwhahaha | but it might be getting hit by something | 18:29 |
| *** ffiore has joined #tripleo | 18:29 | |
| trozet | mwhahaha: i dont see why anything would make db.sock in tripleo | 18:29 |
| ykarel|afk | mwhahaha, can you check my comment on the patch: i think it's relevant | 18:30 |
| ykarel|afk | https://review.openstack.org/#/c/554528/ | 18:30 |
| trozet | weshay: if you can give me login to the setup I don tmind taking a look | 18:30 |
| *** haleyb has joined #tripleo | 18:30 | |
| ykarel|afk | mwhahaha, running puppet to start ovs from container is causing it i think, few days back before containerizing neutron-ovs agent it started ovs | 18:31 |
| mwhahaha | ykarel|afk: yea but ovs i think used to be started outside of puppet | 18:32 |
| ykarel|afk | mwhahaha, from the log i only found that it's either started by os-net-config or ovs-agent | 18:32 |
| *** hjensas has joined #tripleo | 18:32 | |
| * mwhahaha isn't sure which is supposed to be starting it | 18:33 | |
| weshay | what did you do to start it? | 18:33 |
| weshay | just start? | 18:33 |
| mwhahaha | restarted it a few times | 18:34 |
| ykarel|afk | mwhahaha, https://github.com/openstack/puppet-neutron/blob/master/manifests/agents/ml2/ovs.pp#L211 | 18:34 |
| ykarel|afk | puppet start it manage_vswitch is true, which is true by default | 18:34 |
| ykarel|afk | after containerizing it's not working as specified in the https://review.rdoproject.org/paste/show/87/ | 18:35 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: DNM: use rhos-release role pin_puddle option https://review.openstack.org/555018 | 18:35 |
| mwhahaha | ykarel|afk: hmm ok so we were inheriting previously and we need to account for the thing that needs to be starting ovs | 18:35 |
| ykarel|afk | mwhahaha, yes | 18:36 |
| mwhahaha | k i'll poke at it a bit more after my meeting | 18:36 |
| trozet | mwhahaha: ovs will actually be started before this | 18:36 |
| trozet | mwhahaha: os-net-config | 18:37 |
| mwhahaha | well it should be | 18:37 |
| mwhahaha | but isn't | 18:37 |
| mwhahaha | so somewhere we lost that startup | 18:37 |
| mwhahaha | or something | 18:37 |
| * beagles guesses that it is running on the controller due to the presence of an OVS based interface/bridge (.e.g. br-ex) | 18:37 | |
| beagles | ifup'ing that would start it up I think | 18:37 |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Allow custom sequence of playbooks https://review.openstack.org/546501 | 18:38 |
| dsneddon | mwhahaha, beagles: Yeah, os-net-config writes out the ifcfg files that define the OVS bridges, then running ifup on the bridge starts OVS. | 18:38 |
| mwhahaha | so if it doesn't start until if up and a container started before it's up'ed the db.sock is a directory | 18:38 |
| *** khyr0n has joined #tripleo | 18:38 | |
| mwhahaha | preventing it from starting | 18:38 |
| openstackgerrit | Merged openstack/tripleo-upgrade stable/pike: Remove ceph osd hieradata during upgrade https://review.openstack.org/553572 | 18:38 |
| mwhahaha | which may be the problem | 18:38 |
| mwhahaha | it needs to be started before all the containers | 18:39 |
| dsneddon | mwhahaha, You might have to put something in firstboot to start OVS, then, because os-net-config runs pretty early in the process | 18:39 |
| ykarel|afk | mwhahaha, but on compute without network isolation there is nothing created on /etc/os-net-config/config.json | 18:39 |
| mwhahaha | it' snot relying on vswitch | 18:39 |
| mwhahaha | IIUC | 18:40 |
| dsneddon | mwhahaha, ykarel|afk: Without network isolation, the compute nodes use net-config-noop.yaml, which leads to an empty config.json | 18:40 |
| ykarel|afk | dsneddon, yes what should be done in this case to start ovs? | 18:40 |
| dsneddon | ykarel|afk, As I was suggesting, I think a firstboot script to start OVS would work. | 18:40 |
| ykarel|afk | Ok | 18:41 |
| ykarel|afk | or there is something called host-prep-task, won't that work | 18:41 |
| ykarel|afk | i don't know much about it | 18:41 |
| dsneddon | ykarel|afk, It might, I don't know anything about that either | 18:42 |
| trozet | dsneddon: why wouldnt you just systemctl enable openvswitch on the disk? | 18:42 |
| dsneddon | ykarel|afk, Here is how to write a firstboot script: https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/extra_config.html | 18:42 |
| dsneddon | trozet, Yeah, if you want to modify the image that's another easy fix | 18:42 |
| trozet | dsneddon: since the dataplane relies on openvswitch its safe to always have it enabled | 18:42 |
| dsneddon | trozet, Yeah, agreed | 18:43 |
| ykarel|afk | trozet, but so everywhere ovs is used, no other option like linuxbridge, etc | 18:43 |
| dsneddon | trozet, We already have logic to restart OVS with DPDK, which is the one case where os-net-config actually runs "systemctl restart openvswitch" | 18:43 |
| openstackgerrit | Brent Eagles proposed openstack/puppet-tripleo master: Adding wrapper scripts for neutron agent subprocesses https://review.openstack.org/550224 | 18:43 |
| trozet | dsneddon: if the os-net-config is empty, then maybe OVS is getting started by https://github.com/openstack/puppet-vswitch/blob/master/manifests/ovs.pp#L94 | 18:44 |
| ykarel|afk | trozet, but from containers ^^ is not working | 18:44 |
| trozet | ykarel|afk: i dont understand what you mean by containers | 18:44 |
| trozet | ykarel|afk: oh because tis the ML2? | 18:45 |
| ykarel|afk | trozet, https://review.rdoproject.org/paste/show/87/ | 18:45 |
| trozet | ykarel|afk: yeah this is why i said a while ago this needs to be removed from ML2 | 18:45 |
| trozet | ykarel|afk: and tried to make openvswitch its own service in tripleo | 18:45 |
| dsneddon | trozet, My assumptions about OVS getting started because of the ifup script could be incorrect. Maybe it was started to begin with? | 18:45 |
| ykarel|afk | trozet, ack | 18:46 |
| trozet | ykarel|afk: so in the container this is skipped becuase it isnt in puppet tags | 18:46 |
| *** rbowen has quit IRC | 18:46 | |
| ykarel|afk | trozet, systemctl don't work in chroot | 18:46 |
| ykarel|afk | ()[root@overcloud-novacompute-bar-0 /]# systemctl status openvswitch | 18:47 |
| ykarel|afk | Running in chroot, ignoring request. | 18:47 |
| trozet | ykarel|afk: yeah but i dont think it should even attempt it, right? only the right puppet tags will get executed | 18:47 |
| trozet | ykarel|afk: so then what tried to bring up OVS :) ? | 18:47 |
| ykarel|afk | trozet, haven't checked that yet | 18:47 |
| dsneddon | trozet, I know that I'm going to have to modify os-net-config to somehow restart the OVS container (which won't be restarted with "systemctl restart openvswitch"), so I'm curious about how to do that with an OVS container. | 18:47 |
| Tengu | beagles: just started the upgrade script. Will open an issue on launchpad once I get some information. | 18:48 |
| beagles | Tengu, ack thx | 18:48 |
| trozet | dsneddon: can you just use docker python api and restart it? | 18:49 |
| dsneddon | trozet, I suppose that would work. | 18:50 |
| *** aputtur__ has quit IRC | 18:50 | |
| *** aputtur has joined #tripleo | 18:50 | |
| dsneddon | trozet, That seems like it would depend on knowing the name of a specific container, but I'm not sure there is anything more standardized than that. | 18:50 |
| dsneddon | ykarel|afk, I think trozet was right, the openvswitch service gets enabled here: https://github.com/openstack/puppet-vswitch/blob/master/manifests/ovs.pp#L85 | 18:51 |
| ykarel|afk | dsneddon, and where this puppet module called up? | 18:51 |
| ykarel|afk | and when | 18:51 |
| trozet | dsneddon: i dont know why ovsdb-server is listed there i think that service is started when OVS is started | 18:51 |
| mwhahaha | it is started by openvswitch | 18:52 |
| mwhahaha | as a dependency service | 18:52 |
| *** raildo has quit IRC | 18:52 | |
| trozet | mwhahaha: yeah so thats kind of weird | 18:52 |
| *** fragatina has quit IRC | 18:53 | |
| *** nyechiel_ has quit IRC | 18:53 | |
| itlinux | hello guys, I want to build a bond with a specific nic.. using the mac address what's the option I should look at? Thanks | 18:55 |
| itlinux | so I can add that to my template. | 18:56 |
| *** jlabarre has joined #tripleo | 18:57 | |
| *** ebarrera has joined #tripleo | 18:58 | |
| itlinux | since I could not find the answer in the https://docs.openstack.org/tripleo-docs/latest/install/advanced_deployment/network_isolation.html | 18:58 |
| dsneddon | itlinux, There is a way to do that, using mapping.yaml. I'm trying to find the documentation. | 18:58 |
| itlinux | thanks dsneddon: | 18:58 |
| trozet | ykarel|afk: called from neutron-ovs-agent.yaml which as you mentioned is a container | 18:59 |
| ykarel|afk | and that's not working | 19:00 |
| trozet | ykarel|afk: with puppet_tags: neutron_config,neutron_agent_ovs,neutron_plugin_ml2 | 19:00 |
| trozet | ykarel|afk: so that doesnt start ovs or try to | 19:01 |
| trozet | ykarel|afk: ah but you know what | 19:01 |
| mwhahaha | ykarel|afk, trozet, beagles: so i think we need a hostprep task for the neutron-vos-agent to ensure ovs is started https://github.com/openstack/tripleo-heat-templates/blob/a175c9e6aaf5d35f653fd14f05cf04ba069ea710/docker/services/neutron-ovs-agent.yaml | 19:01 |
| trozet | ykarel|afk: nvm was going to say the neutron-ovs-agent systemd service depends on openvswitch, but that doesnt matter cause no systemd in the container | 19:01 |
| mwhahaha | because we were inheriting the ovs service getting managed via the agent | 19:01 |
| mwhahaha | and not explicitly doing it anywhere | 19:02 |
| mwhahaha | which worked when it wasn't containerized | 19:02 |
| trozet | mwhahaha: so i was just going to ask how you didnt hit this before...is it because this is the first attempt at containerizing the ovs agent? | 19:02 |
| dsneddon | itlinux, Can you read this? https://access.redhat.com/solutions/2940021 | 19:02 |
| mwhahaha | cause i think it's specifically this causing problems: https://github.com/openstack/tripleo-heat-templates/blob/a175c9e6aaf5d35f653fd14f05cf04ba069ea710/docker/services/neutron-ovs-agent.yaml#L138 | 19:02 |
| dsneddon | itlinux, If you don't have an account, I can paste the contents to paste.openstack.org | 19:02 |
| openstackgerrit | Dmitry Tantsur proposed openstack/tripleo-heat-templates master: Add support to ironic "direct" deploy interface https://review.openstack.org/529342 | 19:03 |
| mwhahaha | i think the mounting of /var/run/openvswitch/db.sock before the service is causing problems | 19:03 |
| mwhahaha | trozet: evidently we weren't actually properly ensuring all the services were containerized | 19:03 |
| mwhahaha | until recently | 19:03 |
| * mwhahaha looks around | 19:03 | |
| beagles | mmm.. no that's not quite right | 19:03 |
| ykarel|afk | mwhahaha, i tried without it as well, and got Running in chroot, ignoring request. | 19:03 |
| ykarel|afk | i mean without mounting of /var/run/openvswitch/db.sock | 19:04 |
| beagles | the ovs agent has been running in a container for quite some time (all of the octavia work was done this way) | 19:04 |
| mwhahaha | beagles: it might not have been in fs020 | 19:04 |
| beagles | mwhahaha, ah I see | 19:04 |
| trozet | mwhahaha: oh now i see the problem, you guys mount db.sock int eh docker service | 19:04 |
| trozet | mwhahaha: now it is clear :) | 19:04 |
| mwhahaha | right | 19:05 |
| mwhahaha | so when docker comes along and mounts it it's a directory | 19:05 |
| mwhahaha | which is the error i saw from the service | 19:05 |
| *** brault has joined #tripleo | 19:05 | |
| mwhahaha | so we need to ensure ovs is running before any of the docker bits | 19:05 |
| trozet | mwhahaha: yep thats it | 19:05 |
| itlinux | thanks dsneddon: I do not have an account.. I did when I was at Red Hat :) | 19:05 |
| ykarel|afk | mwhahaha, default containerization done 13 days ago: https://review.openstack.org/#/c/548554/ and it started failing after it | 19:05 |
| mwhahaha | or change the mount | 19:05 |
| mwhahaha | ykarel|afk: right so we must have 'fixed' something when we switched it | 19:05 |
| ykarel|afk | mwhahaha, yes | 19:06 |
| trozet | mwhahaha: i think the *right* way is to use an openvswitch service to control openvswitch | 19:06 |
| mwhahaha | that was previously still inheriting a baremetal thing | 19:06 |
| *** salmankhan has quit IRC | 19:06 | |
| dsneddon | itlinux, Here is the file that actually does the work: https://github.com/openstack/tripleo-heat-templates/blob/master/firstboot/os-net-config-mappings.yaml | 19:06 |
| mwhahaha | trozet: well yes, but we don't have an openvswitch service officially | 19:06 |
| * mwhahaha shrugs | 19:06 | |
| trozet | mwhahaha: https://github.com/openstack/tripleo-heat-templates/blob/a175c9e6aaf5d35f653fd14f05cf04ba069ea710/puppet/services/openvswitch.yaml | 19:06 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: DNM: use rhos-release role pin_puddle option https://review.openstack.org/555018 | 19:06 |
| mwhahaha | is that missing? | 19:06 |
| itlinux | ok.. | 19:06 |
| trozet | mwhahaha: that service should call puppet-vswitch to start ovs and configure it | 19:06 |
| mwhahaha | from fs020 | 19:06 |
| beagles | mwhahaha, that was a dpdk only kind of deal | 19:07 |
| itlinux | so I should use the -e ...templates/ option to get this right | 19:07 |
| trozet | mwhahaha: no its just the way it is done now that service is inherited by neutron-ovs-agent, and does not actually control OVS | 19:07 |
| trozet | mwhahaha: but it should in the future | 19:07 |
| *** brault_ has quit IRC | 19:07 | |
| mwhahaha | yea probably | 19:07 |
| trozet | mwhahaha: especially if in the future you may want to containerze ovs, it needs to be its own legit service | 19:07 |
| mwhahaha | that's the missing config bits | 19:08 |
| dsneddon | itlinux, The script will loop through all nodes (node1, node2, etc.), and when it finds a matching MAC, it will lay down the mapping for that node (whatever node number it is, doesn't matter). | 19:08 |
| mwhahaha | since it was always just inheirieting | 19:08 |
| *** shreshtha has quit IRC | 19:08 | |
| mwhahaha | anyway i much lunch, i shall continue investigating later | 19:08 |
| dsneddon | itlinux, I'll paste the instructions for you | 19:08 |
| itlinux | thanks dsneddon: | 19:09 |
| trozet | mwhahaha: https://bugs.launchpad.net/tripleo/+bug/1656096 | 19:09 |
| openstack | Launchpad bug 1656096 in tripleo "[RFE] Split Open Vswitch its own service" [Wishlist,In progress] - Assigned to Tim Rozet (trozet) | 19:09 |
| trozet | ykarel|afk:^ | 19:09 |
| dsneddon | itlinux, http://paste.openstack.org/show/707901/ | 19:09 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757474 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1757474 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset027-master fails with undefined groups.overcloud" [Critical,Triaged] | 19:10 |
| ykarel|afk | trozet, ack | 19:11 |
| itlinux | dsneddon: this file does not exist /usr/share/openstack-tripleo-heat-templates/firstboot/os-net-config-mappings.yaml | 19:11 |
| *** jcoufal has quit IRC | 19:12 | |
| dsneddon | itlinux, Interesting. What version of TripleO are you using? Does /usr/share/openstack-tripleo-heat-templates exist? | 19:13 |
| itlinux | pike | 19:13 |
| itlinux | one sec.. | 19:14 |
| itlinux | I was on the controller dam a$$ | 19:14 |
| dsneddon | itlinux, It's pretty easy. Here's an example of what you add to network-environment.yaml to make it work: http://paste.openstack.org/show/707913/ | 19:18 |
| itlinux | thanks | 19:19 |
| itlinux | so I could just add my option nic to my config like this.. | 19:19 |
| dsneddon | itlinux, In that example, only the last line in resource_registry and the NetConfigDataLookup: block are what you add to your own file | 19:20 |
| *** myoung is now known as myoung|biab | 19:22 | |
| *** ykarel|afk is now known as ykarel|away | 19:22 | |
| *** dprince has quit IRC | 19:23 | |
| *** jaosorior has quit IRC | 19:23 | |
| *** dprince has joined #tripleo | 19:23 | |
| *** ykarel|away has quit IRC | 19:28 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Add pre_upgrade_rolling_tasks https://review.openstack.org/552073 | 19:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Update service readme files https://review.openstack.org/553321 | 19:34 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fixes ODL container failing to start due to missing etc config https://review.openstack.org/553079 | 19:34 |
| openstackgerrit | Merged openstack/tripleo-validations master: Fix MySQL Open Files Limit validation https://review.openstack.org/554888 | 19:34 |
| itlinux | dsneddon: so node1 I assume that is like compute1 and etc.. | 19:34 |
| itlinux | how do I specify the diff between compute / controller? | 19:34 |
| *** liverpooler has quit IRC | 19:34 | |
| *** radeks_ has joined #tripleo | 19:36 | |
| itlinux | dsneddon: I guess this looks correct? http://paste.openstack.org/show/707932/ | 19:36 |
| dsneddon | itlinux, It really doesn't matter. The script will loop through each node and as soon as it finds a matching MAC address it knows it has found the right node and will write that node's mapping to disk. | 19:37 |
| *** tesseract has quit IRC | 19:38 | |
| dsneddon | itlinux, What you have there looks fine, but you also need the reference to os-net-config-mappings.yaml in the resource_registry: section of that same file | 19:38 |
| *** radeks has quit IRC | 19:38 | |
| *** pickle is now known as dhill_ | 19:39 | |
| itlinux | ok so I just put all nodes in seq and then the role is assigned by the other file I have http://paste.openstack.org/show/707936/ and I could add them here. instead.. | 19:39 |
| *** radeks_ has quit IRC | 19:42 | |
| dsneddon | itlinux, It really doesn't matter what sequence the nodes are in NetConfigDataLookup:, the script will loop through on each node, and find the matching MAC address. | 19:42 |
| *** radeks_ has joined #tripleo | 19:43 | |
| *** ffiore has quit IRC | 19:43 | |
| dsneddon | itlinux, It doesn't rely on the order that you put nodes into NetConfigDataLookup, only that a MAC address matches for each node. | 19:43 |
| itlinux | ok here is my new script then :) http://paste.openstack.org/show/707938/ | 19:43 |
| itlinux | if you can give it a blessing :) dsneddon: | 19:43 |
| *** dprince has quit IRC | 19:43 | |
| dsneddon | itlinux, Not quite. This line goes under "resource_registry:" OS::TripleO::NodeUserData: /usr/share/openstack-tripleo-heat-templates/firstboot/os-net-config-mappings.yaml | 19:43 |
| dsneddon | itlinux, And you have too much indentation in the NetConfigDataLookup block | 19:44 |
| *** jaosorior has joined #tripleo | 19:44 | |
| itlinux | ok fixing now.. | 19:44 |
| *** sri_ has quit IRC | 19:46 | |
| alee_ | mwhahaha, beagles -- just getting back to this -- any progress> | 19:46 |
| alee_ | ? | 19:46 |
| itlinux | http://paste.openstack.org/show/707944/ dsneddon: | 19:47 |
| dsneddon | itlinux, That looks right. If you want to be super-picky, you can add one whitespace in front of the OS::TripleO... to line it up with the rest. | 19:48 |
| itlinux | thanks will do now | 19:48 |
| *** athomas has quit IRC | 19:48 | |
| itlinux | final version http://paste.openstack.org/show/707946/ | 19:49 |
| itlinux | thanks again dsneddon: much appreciated! | 19:50 |
| *** rfolco is now known as rfolco|ruck | 19:51 | |
| openstackgerrit | Tim Rozet proposed openstack/tripleo-heat-templates stable/queens: Fixes ODL container failing to start due to missing etc config https://review.openstack.org/555035 | 19:55 |
| *** moshele has joined #tripleo | 19:55 | |
| *** eck` is now known as eck`gone | 19:57 | |
| mwhahaha | alee_: so we left it at it seems like the openvswitch service is not properly started on the system | 20:03 |
| mwhahaha | alee_: so there's probably a few ways to tackle that | 20:03 |
| mwhahaha | i'm going to look at something to see if it's that we just need to stop mounting /var/run/openvswitch/db.sock and mount the dir instead | 20:03 |
| *** chem has quit IRC | 20:04 | |
| mwhahaha | there's another problem in that we appear to be doing vsctl commands and it wasn't running and we didn't fail | 20:05 |
| *** chem has joined #tripleo | 20:05 | |
| *** chem has quit IRC | 20:06 | |
| *** moshele has quit IRC | 20:06 | |
| weshay | mwhahaha++ | 20:06 |
| alee_ | mwhahaha, how do we know that the openvspwitch service was not properly started? | 20:06 |
| mwhahaha | alee_: it's not running and there are no logs | 20:06 |
| mwhahaha | alee_: from weshay's reproducer we see that it's failing to launch because /var/run/openvswitch/db.sock is a folder evidently | 20:06 |
| mwhahaha | Mar 21 18:22:03 overcloud-novacompute-bar-0 ovsdb-server[129745]: ovs|00005|ovsdb_jsonrpc_server|ERR|punix:/var/run/openvswitch/db.sock: listen failed: Is a directory | 20:07 |
| *** chem has joined #tripleo | 20:07 | |
| *** radeks_ has quit IRC | 20:07 | |
| mwhahaha | weshay: i'm off that box now | 20:07 |
| * mwhahaha goes to fiddle with code | 20:07 | |
| weshay | k.. thanks | 20:08 |
| alee_ | mwhahaha, I wonder if thats the same case in https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/6e4ffba/undercloud/home/jenkins/tempest/tempest.html.gz | 20:08 |
| alee_ | mwhahaha, which is the case in the trello board | 20:08 |
| *** akrivoka has quit IRC | 20:09 | |
| alee_ | mwhahaha, I see openvswitch logs there I think ,, | 20:09 |
| Tengu | beagles: stoll there? apparently OVS has some issues, its log if full of 2018-03-21T19:30:10.742Z|00359|rconn|WARN|br-ex<->tcp:127.0.0.1:6633: connection failed (Connection refused) lines (with other br-FOO) | 20:09 |
| Tengu | *sitll | 20:09 |
| mwhahaha | alee_: oh right so that's different, i was looking at the fs020 error | 20:09 |
| mwhahaha | alee_: let me check that one | 20:09 |
| Tengu | … darn. should go to bed, can't type anymore. | 20:09 |
| mwhahaha | too many failures | 20:09 |
| alee_ | mwhahaha, ah - and I thought you'd solved my isssue :) | 20:10 |
| mwhahaha | alee_: so that one is different | 20:10 |
| mwhahaha | alee_: but that one looks like my 3node failures | 20:10 |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757474 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1757474 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset027-master fails with undefined groups.overcloud" [Critical,Triaged] | 20:10 |
| mwhahaha | alee_: if you look in the ovs-vswitchd you see errors about no such device | 20:10 |
| mwhahaha | alee_: so we might need an ovs expert on that one | 20:10 |
| mwhahaha | alee_: https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset017-queens/6e4ffba/subnode-2/var/log/openvswitch/ovs-vswitchd.log.txt.gz | 20:10 |
| mwhahaha | alee_: 2018-03-21T06:49:47.454Z|00056|netdev_linux|WARN|tap13c9ce98-76: removing policing failed: No such device | 20:10 |
| mwhahaha | stuff like that | 20:10 |
| Tengu | hmmmmm. miht be linked to the LBaaS… its log is also full of issues. duh. | 20:10 |
| Tengu | anyway. will open the issue tomorrow, it's late here now. | 20:11 |
| *** radeks_ has joined #tripleo | 20:11 | |
| alee_ | mwhahaha, ok - who do we need to look at that? | 20:12 |
| mwhahaha | the mystical networking folks | 20:12 |
| * mwhahaha waves hands | 20:12 | |
| * mwhahaha has no idea | 20:12 | |
| alee_ | beagles, ? | 20:13 |
| alee_ | mwhahaha, ok - I see that on ykarel's reproducer machine too | 20:14 |
| * bnemec would argue that none of the ci categories is green right now | 20:15 | |
| ianw | hi everyone, https://review.openstack.org/554705 seems to be failing in the gate ... it's a bit of an issue because it blocks dib | 20:15 |
| bnemec | In case anyone else is wondering if they should recheck | 20:15 |
| bnemec | ianw: It's a known problem. I think mwhahaha is looking into it. | 20:16 |
| *** mwhahaha changes topic to "Welcome to Rocky. CI status - Promotions: RED; check/gate: RED; RDO CI jobs: Questionable | http://tripleo.org/ | https://docs.openstack.org/tripleo-docs/latest/" | 20:16 | |
| beagles | alee_, that's pretty wild | 20:17 |
| beagles | alee_, but... | 20:17 |
| ianw | ok, thanks. i might have to drop the tripleo test from dib for a while as we need a new release | 20:17 |
| beagles | alee_, it reeks of network namespace issues | 20:18 |
| bnemec | mwhahaha: Thanks | 20:19 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Remove tripleo jobs https://review.openstack.org/555037 | 20:19 |
| mwhahaha | weshay: in the promotion jobs, where are the images build from? is there a cached set i could pull? | 20:23 |
| weshay | mwhahaha, https://images.rdoproject.org/master/rdo_trunk/ | 20:24 |
| weshay | mwhahaha, tripleo-ci-testing would be the very latest that didn't pass | 20:24 |
| mwhahaha | k | 20:24 |
| mwhahaha | i'm so sick of looking into images | 20:25 |
| mwhahaha | next time it's an image problem i'm leaving | 20:25 |
| * mwhahaha shakes fists at libguestfs | 20:25 | |
| slagle | what color is Questionable | 20:26 |
| slagle | let's use Chartreuse | 20:26 |
| mwhahaha | works for me | 20:28 |
| bnemec | Chartreuse is pretty close to green though. | 20:30 |
| bnemec | Not that most people probably know that. :-) | 20:30 |
| * bnemec has chartreuse fishing lures | 20:30 | |
| mwhahaha | soooo | 20:31 |
| mwhahaha | we no longer automagically enable openvswitch | 20:31 |
| mwhahaha | as of queens | 20:31 |
| mwhahaha | it's missing from the overcloud image | 20:31 |
| mwhahaha | in pike, /etc/systemd/system/multi-user.target.wants/openvswitch.service is defined | 20:32 |
| mwhahaha | not so much for master/queens | 20:32 |
| mwhahaha | i have no idea how any of this has worked for the last few months | 20:32 |
| * mwhahaha gives up | 20:32 | |
| hjensas | slagle: Purple is the color most associated with ambiguity. Like other colors made by combining two primary colors, it is seen as __uncertain__ and equivocal. ("Eva Heller, Psychologie de la couleur: effets et symboliques) | 20:33 |
| *** jaosorior_ has joined #tripleo | 20:33 | |
| * bnemec looks at http://tripleo.org/ | 20:34 | |
| bnemec | Yep, color scheme checks out. | 20:34 |
| hjensas | lol | 20:34 |
| *** eck`gone is now known as eck` | 20:36 | |
| *** aputtur has quit IRC | 20:37 | |
| *** jaosorior has quit IRC | 20:37 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Add HostsEntry to undercloud's /etc/hosts https://review.openstack.org/555041 | 20:37 |
| weshay | mwhahaha, hrm.. so now we a check on DIB? | 20:41 |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: Use rhos-release role pin puddle option https://review.openstack.org/555018 | 20:41 |
| mwhahaha | so anyone want to take a guess as to how openvswitch was previously enabled by default on the overcloud-full.qcow2 | 20:41 |
| mwhahaha | weshay: I am here: ¯\_(ツ)_/¯ | 20:41 |
| weshay | weee | 20:42 |
| * mwhahaha assumes packaging | 20:42 | |
| * mwhahaha has no idea | 20:42 | |
| mwhahaha | cause pike had 2.7.3 | 20:42 |
| mwhahaha | and queens has 2.8.2 | 20:42 |
| weshay | rc.fit | 20:42 |
| mwhahaha | but i didn't spot anything in the spec | 20:42 |
| *** amoralej is now known as amoralej|off | 20:43 | |
| mwhahaha | ah ha | 20:45 |
| mwhahaha | https://github.com/openstack/tripleo-image-elements/blob/master/elements/openvswitch/install.d/74-openvswitch | 20:45 |
| mwhahaha | which hasn't changed in 2 years | 20:45 |
| * mwhahaha blames bnemec | 20:46 | |
| mwhahaha | he who last touches, it supports it for life | 20:46 |
| *** myoung|biab is now known as myoung | 20:47 | |
| bnemec | We were still using that?! | 20:47 |
| mwhahaha | i have no idea | 20:48 |
| mwhahaha | but that's the only reference i can find to enabling openvswitch :D | 20:48 |
| mwhahaha | though the service name is probably wrong now | 20:48 |
| mwhahaha | need to go find the build logs | 20:48 |
| *** radeks_ has quit IRC | 20:48 | |
| bnemec | I mean, it wouldn't be the first time something I thought we had removed was still in use. | 20:48 |
| mwhahaha | weshay: do we keep the build logs for the images stuff around somewhere? | 20:49 |
| weshay | getting | 20:49 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Add support for writing hostnames to inventory https://review.openstack.org/555049 | 20:49 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Begin adding environments with all params for a service https://review.openstack.org/475924 | 20:49 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: Add ability to generate an environment index https://review.openstack.org/491925 | 20:49 |
| openstackgerrit | Ben Nemec proposed openstack/tripleo-heat-templates master: WIP: Add service config env with all Designate settings https://review.openstack.org/555008 | 20:49 |
| weshay | mwhahaha, https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/e4728dd/undercloud/home/jenkins/overcloud_image_build.log.txt.gz | 20:49 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Add support for writing hostnames to inventory https://review.openstack.org/555049 | 20:51 |
| mwhahaha | bnemec: so yea we were still using it | 20:52 |
| mwhahaha | https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-pike-upload/c54e735/undercloud/home/jenkins/overcloud_image_build.log.txt.gz#_2018-02-12_15_07_15 | 20:52 |
| mwhahaha | exists in pike | 20:52 |
| mwhahaha | does not in queens | 20:52 |
| *** khyr0n has quit IRC | 20:52 | |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Use container_images_file for all image prepare https://review.openstack.org/554676 | 20:54 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Do container image prepare during undercloud deploy https://review.openstack.org/546024 | 20:54 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Use the build_service_filter from kolla_builder https://review.openstack.org/555051 | 20:54 |
| weshay | trown, fyi ^ | 20:55 |
| openstackgerrit | James Slagle proposed openstack/tripleo-validations master: Add --use-hostnames to tripleo-ansible-inventory https://review.openstack.org/555052 | 20:55 |
| mwhahaha | we lost a bunch of stuff | 20:56 |
| mwhahaha | pike https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-pike-upload/c54e735/undercloud/home/jenkins/overcloud_image_build.log.txt.gz#_2018-02-12_15_07_16 | 20:56 |
| mwhahaha | vs queens https://logs.rdoproject.org/openstack-periodic/periodic-tripleo-ci-centos-7-ovb-1ctlr_1comp-featureset002-queens-upload/e4728dd/undercloud/home/jenkins/overcloud_image_build.log.txt.gz#_2018-03-21_06_18_25 | 20:56 |
| trown | interesting | 20:57 |
| bnemec | mwhahaha: Oh, I bet it's: https://github.com/openstack/tripleo-image-elements/blob/98b9c6a5145bbcb46a6d22265f63e219199599ba/elements/os-net-config/element-deps | 20:57 |
| bnemec | We removed that recently, right? | 20:57 |
| bnemec | openvswitch was previously getting pulled in as a dep of os-net-config. | 20:58 |
| mwhahaha | we did? | 20:58 |
| mwhahaha | we probably did | 20:58 |
| mwhahaha | so we need to add it back in or something | 20:58 |
| bnemec | Thought so. Let me look. | 20:58 |
| bnemec | mwhahaha: https://github.com/openstack/tripleo-common/commit/bce76efbcdf39383cae627c46634f2ca1b9aaf6b#diff-28d0b8f7801642ca03031c884319bc5a | 20:59 |
| bnemec | No idea how any of this has worked since then though. | 20:59 |
| mwhahaha | magic | 20:59 |
| mwhahaha | cause we weren't properly containerizing things | 20:59 |
| mwhahaha | so it was getting started else where | 20:59 |
| mwhahaha | good i didn't have anything to do with that change so i can blame everyone else | 20:59 |
| * mwhahaha points fingers | 20:59 | |
| bnemec | lol | 21:00 |
| weshay | it's his favorite thing | 21:00 |
| * bnemec 's favorite thing is "I told you so" | 21:00 | |
| bnemec | Not sure it applies in this case though. | 21:00 |
| weshay | https://goo.gl/images/v4d7re | 21:00 |
| mwhahaha | well i guess it's time to add openvswitch back in | 21:01 |
| mwhahaha | we should have explicitly had that defined anyway | 21:01 |
| mwhahaha | i'm more concerned how this worked at all downstream | 21:02 |
| *** trown is now known as trown|outtypewww | 21:02 | |
| mwhahaha | but that's a whole other issue | 21:02 |
| *** fragatina has joined #tripleo | 21:06 | |
| mwhahaha | heh if we were using the overcloud-realtime-compute images we wouldn't have these problems | 21:07 |
| weshay | mwhahaha, it worked due to net-iso covering it up | 21:07 |
| weshay | not sure if there is a mix of jobs w/ and w/o net-iso | 21:08 |
| mwhahaha | well i'm surprised that the lack of openvswitch starting didn't show up somewhere else | 21:08 |
| weshay | mwhahaha, also not sure if lon's selinux patch is a valid | 21:08 |
| weshay | mwhahaha, meh.. who needs networking | 21:08 |
| mwhahaha | devstack is a single node, it's good enough for prod right | 21:08 |
| *** moshele has joined #tripleo | 21:09 | |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 21:10 |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,Triaged] | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757474 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1757474 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset027-master fails with undefined groups.overcloud" [Critical,Triaged] | 21:10 |
| *** gfidente|afk has quit IRC | 21:10 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Add openvswitch element back in https://review.openstack.org/555056 | 21:10 |
| mwhahaha | weshay, bnemec -^ | 21:10 |
| jaosorior_ | weshay: multinode jobs don't use netiso afaik | 21:11 |
| *** pchavva has quit IRC | 21:11 | |
| mwhahaha | jaosorior_: yea cause it's only a single node | 21:11 |
| weshay | jaosorior_, yup.. either do a couple of the ovb jobs | 21:11 |
| mwhahaha | so it's geting started elsewhere | 21:11 |
| mwhahaha | it's just the compute nodes where this manifests itself | 21:11 |
| jaosorior_ | mwhahaha: ovb doesn't use netiso? wtf | 21:11 |
| weshay | mwhahaha, backporting to queens as well? | 21:11 |
| mwhahaha | weshay: yea will be | 21:11 |
| mwhahaha | once we verify it work sin master | 21:11 |
| weshay | sinner | 21:12 |
| mwhahaha | pretty much | 21:12 |
| *** bnemec is now known as sin-master | 21:12 | |
| sin-master | And it's not even Friday yet! | 21:12 |
| weshay | rfolco|ruck, look for master fs20 first.. no queens backport yet | 21:12 |
| *** sin-master is now known as bnemec | 21:12 | |
| rfolco|ruck | weshay, ack | 21:12 |
| mwhahaha | ok so now we need someone from networking to fix the barbican/3node thing in ovs | 21:13 |
| * mwhahaha moves on to that issue next so we can downgrade to rainbow from qustionable | 21:13 | |
| weshay | lolz | 21:14 |
| mwhahaha | alright now where did i put those logs for that issue | 21:15 |
| *** raildo has joined #tripleo | 21:15 | |
| openstackgerrit | Liz Blanchard proposed openstack/tripleo-ui master: Remove h1 page header on deployment plan page https://review.openstack.org/555060 | 21:16 |
| weshay | mwhahaha, fwiw.. it might be easier / faster to test the fix on queens than master | 21:16 |
| weshay | as we'll surely hit some other bs on master | 21:16 |
| mwhahaha | well if we can actually land the stupid thing | 21:17 |
| * mwhahaha sighs | 21:17 | |
| weshay | lol | 21:17 |
| mwhahaha | the 3node thing blocks the thing that'll block that from landing | 21:17 |
| mwhahaha | since the image building job is dorked | 21:17 |
| mwhahaha | which is what ianw was talking about | 21:18 |
| mwhahaha | so we need to figure out why 3node is flakey | 21:18 |
| weshay | ya.. saw that | 21:18 |
| weshay | hrm | 21:19 |
| *** chem has quit IRC | 21:20 | |
| EmilienM | sorry for spam | 21:21 |
| EmilienM | but I hav eno choice to rebase | 21:21 |
| *** chem has joined #tripleo | 21:21 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: undercloud: workaround for masquerading network in CI/OVB https://review.openstack.org/553620 | 21:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: undercloud: workaround for masquerading network in CI/OVB https://review.openstack.org/553620 | 21:21 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: roles: rename overcloud-prep-containers to prep-containers https://review.openstack.org/543014 | 21:22 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: prep-containers: include containerized undercloud bits https://review.openstack.org/543024 | 21:22 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: undercloud: add missing TLS environments when preparing containers https://review.openstack.org/545444 | 21:22 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: Remove adjust-interface-mtus script https://review.openstack.org/546216 | 21:22 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart-extras master: DO NOT REVIEW - Workarounds for containerized undercloud https://review.openstack.org/545628 | 21:22 |
| * mwhahaha blames EmilienM | 21:23 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: Run ovb-ha (fs001) with a containerized undercloud https://review.openstack.org/542556 | 21:23 |
| EmilienM | again? | 21:23 |
| mwhahaha | you were on that os-net-config patch review, i can blame you :D | 21:24 |
| EmilienM | right | 21:25 |
| EmilienM | it was my ghost | 21:25 |
| *** bfournie has quit IRC | 21:25 | |
| mwhahaha | the ghost of reviews past | 21:25 |
| *** ansmith has quit IRC | 21:26 | |
| * bnemec both +2'd and WIP'd it | 21:27 | |
| bnemec | So...neutral? | 21:27 |
| *** ktibi has quit IRC | 21:27 | |
| bnemec | I believe I also explicitly said it could merge when it wouldn't break CI. :-P | 21:27 |
| EmilienM | why our CI jobs didn't fail? | 21:28 |
| mwhahaha | they did, eventually | 21:28 |
| mwhahaha | house of very thick cards | 21:29 |
| weshay | castle of cards? | 21:31 |
| mwhahaha | with a moat! | 21:31 |
| weshay | in fact we do have a moat | 21:32 |
| weshay | EmilienM, you still in europe? | 21:32 |
| EmilienM | no I'm in canadaland | 21:33 |
| openstackgerrit | Harald Jensås proposed openstack/python-tripleoclient master: Contanerized Undercloud - Routed Spine-Leaf https://review.openstack.org/543455 | 21:40 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add ctlplane networking for routed networks https://review.openstack.org/547326 | 21:41 |
| *** tcw has quit IRC | 21:42 | |
| EmilienM | omg this is awesome https://beagle-hound.readthedocs.io/en/latest/ | 21:44 |
| *** agopi is now known as agopi|dinner | 21:46 | |
| *** rbrady is now known as rbrady-afk | 21:46 | |
| mwhahaha | rfolco|ruck, weshay: so has no one opened a bug for the flakey 3node yet? | 21:47 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Manage upgrades to a containerized undercloud https://review.openstack.org/549624 | 21:47 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Manage upgrades to a containerized undercloud https://review.openstack.org/549624 | 21:48 |
| weshay | mwhahaha, I'm talking to rfolco|ruck about it right now | 21:48 |
| weshay | we'll have one up shortly | 21:48 |
| mwhahaha | k i think it also needs some eyes from the networking dfg on it | 21:48 |
| * mwhahaha doesn't see anything glaring | 21:48 | |
| mwhahaha | other than some ovs-vswitchd warnings about No such device | 21:49 |
| *** moshele has quit IRC | 21:50 | |
| *** itlinux has quit IRC | 21:51 | |
| weshay | mwhahaha, I see timeouts in some of the neutron agents.. where do you see the no such device | 21:52 |
| EmilienM | in the journal | 21:52 |
| EmilienM | no? | 21:53 |
| * weshay looks | 21:53 | |
| mwhahaha | no | 21:53 |
| EmilienM | have we kidnapped ihrachys yet? | 21:53 |
| mwhahaha | http://logs.openstack.org/13/554213/1/check/tripleo-ci-centos-7-3nodes-multinode/85d6763/logs/subnode-3/var/log/openvswitch/ovs-vswitchd.log.txt.gz | 21:53 |
| mwhahaha | 2018-03-21T12:35:35.875Z|00077|bridge|INFO|bridge br-int: added interface qg-08994f8f-0d on port 3 | 21:53 |
| mwhahaha | 2018-03-21T12:35:36.525Z|00078|bridge|INFO|bridge br-int: added interface tapce4be856-e6 on port 4 | 21:53 |
| mwhahaha | 2018-03-21T12:35:37.452Z|00079|netdev_linux|INFO|ioctl(SIOCGIFHWADDR) on qg-08994f8f-0d device failed: No such device | 21:53 |
| *** agopi|dinner has quit IRC | 21:53 | |
| EmilienM | I think "No such device" is garbage | 21:54 |
| ihrachys | wat. scrolling up | 21:54 |
| ihrachys | yeah no such device happens all the time | 21:54 |
| weshay | lots of neutron errors in http://logs.openstack.org/22/531322/13/gate/tripleo-ci-centos-7-3nodes-multinode/fb044de/logs/subnode-3/var/log/extra/errors.txt.gz | 21:54 |
| weshay | time outs though | 21:54 |
| * weshay checks concurrency.. again | 21:54 | |
| mwhahaha | there is usally one on start | 21:55 |
| mwhahaha | since we lost service containment with the docker switch | 21:55 |
| mwhahaha | rabbit may or maynot be started | 21:55 |
| ihrachys | right. or host is overloaded so it will eventually back off timeout and hopefully manage to get reply | 21:55 |
| *** pcaruana has quit IRC | 21:55 | |
| weshay | heh.. just the one test there. so it's not overloaded | 21:57 |
| *** rfolco|ruck is now known as rfolco|off | 21:57 | |
| mwhahaha | yea that's a service startup error | 21:57 |
| ihrachys | what's the definition of 'flakey' | 21:57 |
| mwhahaha | ihrachys: 25% failure | 21:57 |
| mwhahaha | so not 100% | 21:57 |
| mwhahaha | but enough to be blocking | 21:58 |
| ihrachys | ok | 21:58 |
| mwhahaha | ihrachys: so according to http://cistatus.tripleo.org/ we failed 38% today | 21:59 |
| mwhahaha | tripleo-ci-centos-7-3nodes-multinode | 21:59 |
| mwhahaha | on the gate, http://cistatus.tripleo.org:8000/ we've failed 18% | 21:59 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Use container_images_file for all image prepare https://review.openstack.org/554676 | 21:59 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Use the build_service_filter from kolla_builder https://review.openstack.org/555051 | 21:59 |
| openstackgerrit | Steve Baker proposed openstack/python-tripleoclient master: Do container image prepare during undercloud deploy https://review.openstack.org/546024 | 21:59 |
| mwhahaha | the last 3 have been the tempest fails | 22:00 |
| *** jmelvin has quit IRC | 22:00 | |
| ihrachys | "tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.100 via SSH timed out." | 22:02 |
| ihrachys | I love those | 22:02 |
| ihrachys | so informative /s | 22:02 |
| ihrachys | probably half of issues I ever look at start with this | 22:02 |
| mwhahaha | the last time this was the stupid thing where the metadata was leaking to the undercloud | 22:02 |
| ihrachys | well this is catch-all thingy | 22:03 |
| ihrachys | "SOMETHING HAPPENED" | 22:03 |
| * mwhahaha is familiar with this poor error messaging concept | 22:03 | |
| ihrachys | so fip created, it gets to ACTIVE, but ssh times out. classic. | 22:04 |
| mwhahaha | magic black hole | 22:06 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Mount openvswitch dir rather than socket https://review.openstack.org/555077 | 22:06 |
| *** cdearborn_ has quit IRC | 22:08 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade stable/queens: Set osd_scenario and journals during ceph params conversion https://review.openstack.org/555079 | 22:08 |
| * mwhahaha wanders off | 22:08 | |
| weshay | ihrachys, fyi https://bugs.launchpad.net/tripleo/+bug/1757556 | 22:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 22:10 |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 22:10 |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757474 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1757474 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset027-master fails with undefined groups.overcloud" [Critical,Triaged] | 22:10 |
| *** raildo has quit IRC | 22:15 | |
| ihrachys | mwhahaha, weshay the test claims ping worked | 22:16 |
| ihrachys | but ssh timed out | 22:16 |
| ihrachys | so probably security groups? | 22:17 |
| weshay | as in they were not open? | 22:18 |
| weshay | ihrachys, so.. I'll bring up an env.. so we can poke at it | 22:18 |
| weshay | easier that way | 22:18 |
| weshay | fyi ihrachys http://logs.openstack.org/22/531322/13/gate/tripleo-ci-centos-7-3nodes-multinode/fb044de/logs/reproducer-quickstart.sh | 22:19 |
| *** yamahata has joined #tripleo | 22:23 | |
| ihrachys | weshay, as in maybe port is closed, which may be either test doesn't configure it (since it passes I don't think it's the issue) or ovs agent fails to configure it | 22:24 |
| *** rcernin has joined #tripleo | 22:25 | |
| ihrachys | the test creates those rules at ~16:17:32,713 so that side is ok | 22:25 |
| *** liverpooler has joined #tripleo | 22:26 | |
| mwhahaha | It works sometimes | 22:26 |
| mwhahaha | So it seems like a race somewhere | 22:26 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Revert "Prepare t-h-t for undercloud in a work dir" https://review.openstack.org/555085 | 22:28 |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Revert "Prepare t-h-t for undercloud in a work dir" https://review.openstack.org/555085 | 22:29 |
| *** threestrands has joined #tripleo | 22:29 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Manage upgrades to a containerized undercloud https://review.openstack.org/549624 | 22:29 |
| *** threestrands has quit IRC | 22:30 | |
| *** threestrands has joined #tripleo | 22:30 | |
| *** d0ugal has quit IRC | 22:34 | |
| *** ccamacho has quit IRC | 22:35 | |
| openstackgerrit | Marius Cornea proposed openstack/tripleo-upgrade master: Include connectivity check prepare scripts during FFU https://review.openstack.org/554914 | 22:36 |
| *** ccamacho has joined #tripleo | 22:36 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Manage upgrades to a containerized undercloud https://review.openstack.org/549624 | 22:37 |
| *** d0ugal has joined #tripleo | 22:37 | |
| *** thrash is now known as thrash|g0ne | 22:39 | |
| *** mcornea has quit IRC | 22:57 | |
| *** raildo has joined #tripleo | 23:06 | |
| *** Goneri has quit IRC | 23:06 | |
| *** dparkes has joined #tripleo | 23:07 | |
| *** jtomasek has quit IRC | 23:08 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757111 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757174 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757474 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1757556 | 23:10 |
| openstack | Launchpad bug 1757111 in tripleo " fs020(both queens/master) tempest tests failing while booting an instance" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1757174 in tripleo "tripleo-buildimage-overcloud-full-centos-7 failing with diskimage_builder.element_dependencies.MissingElementException: Element 'size=4096'' not found" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1757474 in tripleo "periodic-tripleo-ci-centos-7-multinode-1ctlr-featureset027-master fails with undefined groups.overcloud" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1757556 in tripleo "timeouts in neutron are causing ssh failures in tempest test instances" [Critical,Triaged] | 23:10 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Fix default partition type https://review.openstack.org/554771 | 23:11 |
| *** itlinux has joined #tripleo | 23:20 | |
| *** tosky has quit IRC | 23:28 | |
| *** dparkes has quit IRC | 23:31 | |
| openstackgerrit | Emilien Macchi proposed openstack/python-tripleoclient master: Manage upgrades to a containerized undercloud https://review.openstack.org/549624 | 23:44 |
| openstackgerrit | James Slagle proposed openstack/tripleo-heat-templates master: Add HostsEntry to undercloud's /etc/hosts https://review.openstack.org/555041 | 23:48 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Add support for writing hostnames to inventory https://review.openstack.org/555049 | 23:48 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Retry previously failed deployments https://review.openstack.org/554276 | 23:57 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!