| *** colonwq has quit IRC | 00:01 | |
| *** colonwq has joined #tripleo | 00:02 | |
| *** colonwq has quit IRC | 00:09 | |
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
|---|---|---|
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 00:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 00:10 |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 00:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 00:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 00:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 00:10 |
| *** slaweq has joined #tripleo | 00:11 | |
| *** colonwq has joined #tripleo | 00:11 | |
| *** slaweq has quit IRC | 00:16 | |
| *** medberry has joined #tripleo | 00:20 | |
| *** medberry has quit IRC | 00:20 | |
| *** medberry has joined #tripleo | 00:20 | |
| *** dmacpher has quit IRC | 00:30 | |
| *** tvignaud has quit IRC | 00:32 | |
| *** lblanchard has joined #tripleo | 00:40 | |
| *** tvignaud has joined #tripleo | 00:47 | |
| openstackgerrit | Merged openstack/tripleo-common stable/queens: Add missing service for DockerCinderConfigImage https://review.openstack.org/586715 | 00:52 |
| *** Petersingh has joined #tripleo | 00:54 | |
| *** colonwq has quit IRC | 01:04 | |
| *** colonwq has joined #tripleo | 01:06 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 01:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 01:10 |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 01:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 01:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 01:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 01:10 |
| *** mrsoul` has joined #tripleo | 01:10 | |
| *** mrsoul has quit IRC | 01:13 | |
| *** colonwq has quit IRC | 01:13 | |
| *** colonwq has joined #tripleo | 01:13 | |
| *** ayoung has quit IRC | 01:27 | |
| *** dmacpher has joined #tripleo | 01:33 | |
| *** jtcressy has quit IRC | 01:35 | |
| *** lblanchard has quit IRC | 01:43 | |
| *** jtcressy has joined #tripleo | 01:46 | |
| *** Petersingh is now known as Petersingh|afk | 02:02 | |
| *** pliu_ has joined #tripleo | 02:05 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 02:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 02:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 02:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 02:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 02:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 02:10 |
| *** slaweq has joined #tripleo | 02:11 | |
| *** slaweq has quit IRC | 02:15 | |
| *** Petersingh|afk has quit IRC | 02:23 | |
| *** Petersingh has joined #tripleo | 02:23 | |
| *** jtcressy has quit IRC | 02:24 | |
| *** jtcressy has joined #tripleo | 02:24 | |
| *** bfournie has joined #tripleo | 02:25 | |
| *** eck` is now known as eck`gone | 02:36 | |
| *** morazi has quit IRC | 02:57 | |
| *** psachin has joined #tripleo | 02:57 | |
| *** ramishra has joined #tripleo | 03:07 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 03:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 03:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 03:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 03:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 03:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 03:10 |
| *** slaweq has joined #tripleo | 03:11 | |
| *** slaweq has quit IRC | 03:15 | |
| *** skramaja has joined #tripleo | 03:23 | |
| *** gkadam has joined #tripleo | 03:30 | |
| *** yolanda_ has joined #tripleo | 03:38 | |
| *** links has joined #tripleo | 03:39 | |
| *** udesale has joined #tripleo | 03:40 | |
| *** yolanda has quit IRC | 03:41 | |
| *** colonwq has quit IRC | 03:54 | |
| *** colonwq has joined #tripleo | 03:55 | |
| *** psahoo has joined #tripleo | 03:58 | |
| *** mschuppert has joined #tripleo | 04:05 | |
| *** itlinux has quit IRC | 04:06 | |
| *** colonwq has quit IRC | 04:09 | |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 04:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 04:10 |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 04:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 04:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 04:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 04:10 |
| *** links has quit IRC | 04:15 | |
| *** medberry has quit IRC | 04:16 | |
| *** karthiks has joined #tripleo | 04:18 | |
| *** dmacpher has quit IRC | 04:30 | |
| *** chkumar|trekk is now known as chandankumar | 04:30 | |
| *** jaganathan has joined #tripleo | 04:30 | |
| *** links has joined #tripleo | 04:33 | |
| *** jtcressy has quit IRC | 04:35 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Move LVM cleanup phase into cleanup https://review.openstack.org/579102 | 04:49 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Only detach device if all partitions have been cleaned https://review.openstack.org/576876 | 04:49 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a PartitionTableNode to graph https://review.openstack.org/579059 | 04:49 |
| *** shreshtha has joined #tripleo | 04:56 | |
| *** agurenko has joined #tripleo | 05:07 | |
| *** dmacpher has joined #tripleo | 05:07 | |
| *** Petersingh_ has joined #tripleo | 05:10 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 05:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 05:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 05:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 05:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 05:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 05:10 |
| *** slaweq has joined #tripleo | 05:11 | |
| *** Petersingh has quit IRC | 05:12 | |
| *** cshastri has joined #tripleo | 05:15 | |
| *** pdeore has joined #tripleo | 05:15 | |
| *** slaweq has quit IRC | 05:15 | |
| *** ykarel| has joined #tripleo | 05:20 | |
| *** pliu_ has quit IRC | 05:22 | |
| *** dmacpher has quit IRC | 05:25 | |
| *** bkopilov has joined #tripleo | 05:25 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New function: run_ansible_playbook https://review.openstack.org/586538 | 05:28 |
| *** nenad has joined #tripleo | 05:31 | |
| *** dparkes has joined #tripleo | 05:32 | |
| *** nenad has quit IRC | 05:33 | |
| *** ykarel| is now known as ykarel | 05:35 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Enable support for running refstack tests in TQE https://review.openstack.org/570719 | 05:43 |
| *** Petersingh_ is now known as Petersingh|afk | 05:44 | |
| *** holser_ has joined #tripleo | 05:50 | |
| *** jaosorior has joined #tripleo | 05:51 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Add fs055 to run refstack tests https://review.openstack.org/570884 | 05:56 |
| openstackgerrit | Chandan Kumar proposed openstack-infra/tripleo-ci master: Add FS055 job as experimental to run refstack tests https://review.openstack.org/570892 | 05:57 |
| *** yprokule has joined #tripleo | 05:58 | |
| *** Petersingh|afk is now known as Petersingh | 06:04 | |
| *** ksambor has joined #tripleo | 06:05 | |
| *** waleedm has joined #tripleo | 06:08 | |
| waleedm | Hi guys, I still have problem with introspection for baremetal nodes, as following: | 06:08 |
| waleedm | Failed to prepare node def465cf-cf64-4ff4-8f90-af2b3b1dc6fc for cleaning: Failed to create neutron ports for any PXE enabled port on node | 06:08 |
| waleedm | Could anyone help with that, please ? | 06:08 |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New function: run_ansible_playbook https://review.openstack.org/586538 | 06:08 |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 06:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 06:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 06:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 06:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 06:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 06:10 |
| *** slaweq has joined #tripleo | 06:11 | |
| *** ksambor has quit IRC | 06:14 | |
| *** ratailor has joined #tripleo | 06:14 | |
| *** slaweq has quit IRC | 06:16 | |
| *** lvdombrkr has joined #tripleo | 06:18 | |
| *** jtomasek has joined #tripleo | 06:18 | |
| *** quiquell has joined #tripleo | 06:19 | |
| *** jfrancoa has joined #tripleo | 06:19 | |
| *** brault has quit IRC | 06:19 | |
| *** colonwq has joined #tripleo | 06:20 | |
| *** lvdombrkr89 has joined #tripleo | 06:21 | |
| *** lvdombrkr has quit IRC | 06:23 | |
| *** colonwq has quit IRC | 06:25 | |
| *** colonwq has joined #tripleo | 06:26 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Don't quote names with sgdisk https://review.openstack.org/578265 | 06:26 |
| *** ksambor has joined #tripleo | 06:26 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Only detach device if all partitions have been cleaned https://review.openstack.org/576876 | 06:27 |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add a PartitionTableNode to graph https://review.openstack.org/579059 | 06:27 |
| *** dmacpher has joined #tripleo | 06:27 | |
| *** jtcressy has joined #tripleo | 06:30 | |
| *** melwitt has quit IRC | 06:31 | |
| *** jtcressy has quit IRC | 06:31 | |
| *** Petersingh_ has joined #tripleo | 06:31 | |
| *** sdake has quit IRC | 06:31 | |
| *** melwitt has joined #tripleo | 06:32 | |
| *** melwitt is now known as Guest9714 | 06:32 | |
| *** sdake has joined #tripleo | 06:32 | |
| *** sdake has quit IRC | 06:32 | |
| *** sdake has joined #tripleo | 06:32 | |
| *** colonwq has quit IRC | 06:33 | |
| *** Petersingh has quit IRC | 06:33 | |
| *** agopi has quit IRC | 06:35 | |
| *** nyechiel has joined #tripleo | 06:35 | |
| *** threestrands has joined #tripleo | 06:38 | |
| *** threestrands has quit IRC | 06:38 | |
| *** threestrands has joined #tripleo | 06:38 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: [WIP] HTTP test during update/upgrade/ffwd. https://review.openstack.org/586030 | 06:39 |
| *** khyr0n has quit IRC | 06:39 | |
| *** jfrancoa has quit IRC | 06:43 | |
| *** jfrancoa has joined #tripleo | 06:43 | |
| *** ccamacho has joined #tripleo | 06:44 | |
| *** holser_ has quit IRC | 06:44 | |
| *** Haresh has joined #tripleo | 06:45 | |
| *** janki has joined #tripleo | 06:47 | |
| *** dmacpher has quit IRC | 06:50 | |
| *** jidar has quit IRC | 06:52 | |
| *** jidar has joined #tripleo | 06:53 | |
| *** slaweq has joined #tripleo | 06:56 | |
| *** pcaruana has joined #tripleo | 06:56 | |
| *** zoli is now known as zoli|wfh | 06:57 | |
| *** zoli|wfh is now known as zoli | 06:57 | |
| *** brault has joined #tripleo | 06:59 | |
| *** ramishra has quit IRC | 07:00 | |
| *** ramishra has joined #tripleo | 07:01 | |
| *** tesseract has joined #tripleo | 07:04 | |
| cschwede | Hello! Can someone please review patch 581990 ? Already got one +2, needs another one/approval if ok. Thanks! | 07:08 |
| *** amoralej|off is now known as amoralej | 07:08 | |
| cschwede | https://review.openstack.org/#/c/581990/ | 07:08 |
| *** rcernin has quit IRC | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 07:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 07:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 07:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 07:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 07:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 07:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,New] - Assigned to Rafael Folco (rafaelfolco) | 07:10 |
| *** pcaruana has quit IRC | 07:14 | |
| *** pcaruana has joined #tripleo | 07:18 | |
| *** Petersingh_ is now known as Petersingh|lunch | 07:18 | |
| openstackgerrit | Ricardo Noriega proposed openstack/tripleo-heat-templates master: Sharing BGPVPN Heat plugin volume https://review.openstack.org/585320 | 07:19 |
| *** yolanda_ is now known as yolanda | 07:21 | |
| *** jpich has joined #tripleo | 07:21 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Update the config for FS021 https://review.openstack.org/583202 | 07:22 |
| *** florianf has joined #tripleo | 07:23 | |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: [DNM] Add fs050 job to test from-release-upgrades without role changes https://review.openstack.org/576129 | 07:29 |
| *** threestrands has quit IRC | 07:32 | |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-heat-templates stable/queens: DNM: Test Gnocchi Upgrades P->Q. https://review.openstack.org/565231 | 07:33 |
| *** jtomasek_ has joined #tripleo | 07:34 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: Workload flavor customization. https://review.openstack.org/586961 | 07:34 |
| *** jtomasek has quit IRC | 07:34 | |
| *** avivgt has joined #tripleo | 07:37 | |
| *** ykarel is now known as ykarel|lunch | 07:38 | |
| *** paramite has joined #tripleo | 07:42 | |
| *** paramite has quit IRC | 07:42 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New preflight check for the undercloud: disk space https://review.openstack.org/586541 | 07:43 |
| *** peereb has joined #tripleo | 07:43 | |
| *** Petersingh|lunch has quit IRC | 07:46 | |
| *** Petersingh has joined #tripleo | 07:46 | |
| *** amoralej_ has joined #tripleo | 07:47 | |
| *** jpena has joined #tripleo | 07:47 | |
| quiquell | mrunge: Are you there ? | 07:50 |
| mrunge | quiquell: depends :P | 07:51 |
| quiquell | mrunge: Found and issue with kolla and collectd and I see you as commiter in github | 07:51 |
| mrunge | yes? | 07:51 |
| quiquell | mrunge: https://bugs.launchpad.net/tripleo/+bug/1784307 | 07:52 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,New] - Assigned to Rafael Folco (rafaelfolco) | 07:52 |
| quiquell | Looks like it's restarting | 07:52 |
| quiquell | mrunge: Don't know where to look | 07:52 |
| mrunge | quiquell: without knowing what causes the restart, I can not help you | 07:52 |
| quiquell | mrunge: What logs can I check ? | 07:53 |
| mrunge | quiquell: first step would be to provide a log, or a config | 07:53 |
| quiquell | mrunge: docker_journald doesn't help | 07:53 |
| quiquell | mrunge: this is the failing log http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-30_02_16_20 | 07:54 |
| mrunge | quiquell: your link ppoints to pacemaker | 07:54 |
| mrunge | err, the link in the bug points to pacemaker | 07:54 |
| quiquell | mrunge: 192.168.24.1:8787/tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" 7 minutes ago Restarting (1) 16 seconds ago collectd" | 07:54 |
| quiquell | It's not pacemaker, look some lines under it | 07:54 |
| quiquell | the timstamp have few lines in it | 07:55 |
| quiquell | mrunge: Or it could be pacemaker updating collectd ? I have no idea about it | 07:55 |
| mrunge | nope | 07:55 |
| mrunge | but otherwise, I don't have my crystal ball with me, sorry | 07:56 |
| quiquell | mrunge: Yep sorry, I have started to debug it, that's all I have for now | 07:56 |
| mrunge | quiquell: isn't there a /var/lib/container/collectd log? | 07:56 |
| quiquell | mrunge: Wait... I was looking for it in the undercloud, but this is overcloud | 07:57 |
| mrunge | right, collectd is not installed at the undercloud | 07:57 |
| * quiquell looking at oc now | 07:57 | |
| quiquell | mrunge: this is it http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/containers/collectd/ | 07:58 |
| quiquell | mrunge: this is a passing one http://logs.openstack.org/13/586213/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/0e53c4b/logs/subnode-2/var/log/containers/collectd/collectd.log.txt.gz | 07:59 |
| quiquell | They look the sme | 08:01 |
| quiquell | same | 08:01 |
| quiquell | mrunge: I think the issue is here http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/extra/docker/containers/collectd/stdout.log.txt.gz | 08:02 |
| mrunge | quiquell: so, from the logs it looks like someone is trying to load the collectd-rrdtool plugin | 08:02 |
| quiquell | mrunge: In the passing job the error is the same, so it's not related | 08:02 |
| quiquell | mrunge: If you check here http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/extra/docker/containers/collectd/stdout.log.txt.gz | 08:03 |
| *** bogdando has joined #tripleo | 08:03 | |
| quiquell | mrunge: We have a "Error: Reading the config file failed!" | 08:03 |
| waleedm | Hi guys, I still have problem with introspection for baremetal nodes, as following: | 08:04 |
| waleedm | Failed to prepare node def465cf-cf64-4ff4-8f90-af2b3b1dc6fc for cleaning: Failed to create neutron ports for any PXE enabled port on node | 08:04 |
| waleedm | Could anyone help with that, please ? | 08:04 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add host routes to subnets https://review.openstack.org/580235 | 08:04 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: Add per-network routes to NIC templates https://review.openstack.org/580236 | 08:04 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-heat-templates master: host_routes using get_attr (Composable Networks) https://review.openstack.org/580596 | 08:04 |
| *** hjensas has joined #tripleo | 08:04 | |
| *** hjensas has quit IRC | 08:04 | |
| *** hjensas has joined #tripleo | 08:04 | |
| mrunge | quiquell: hard to guess what happened there | 08:05 |
| *** ukalifon has joined #tripleo | 08:05 | |
| mrunge | quiquell: the last change happened upstream 13 days ago | 08:05 |
| mrunge | quiquell: did the job fail for the last 13 days? | 08:05 |
| quiquell | mrunge: Nope, it's new like 2 days or so... maybe some interaction, "Error: Reading the config file failed!" happend in the passing one too :-( | 08:06 |
| mrunge | quiquell: btw. the rrdtool issue needs to get fixed, otherwise it may hide some other issues | 08:06 |
| quiquell | mrunge: Nah forget, I will dig deeper and go back to you with specifics | 08:06 |
| quiquell | mrunge: ack, thanks man | 08:07 |
| quiquell | mrunge: and the Error: Reading the config file failed! | 08:07 |
| quiquell | ? | 08:07 |
| mrunge | quiquell: isn't the config prepared by ooo? | 08:07 |
| mrunge | especially, we are not proposing to use rrdtool | 08:07 |
| *** derekh has joined #tripleo | 08:08 | |
| quiquell | mrunge: this is the config http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/extra/docker/containers/collectd/ | 08:09 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 08:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 08:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 08:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 08:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 08:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,New] - Assigned to Rafael Folco (rafaelfolco) | 08:10 |
| mrunge | quiquell: where is the collectd.conf file? | 08:11 |
| mrunge | quiquell: I don't see that file, and that would explain the issue | 08:11 |
| quiquell | mrunge: That should be inside the collectd container ? | 08:12 |
| mrunge | quiquell: probably | 08:12 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: Add BootParams service to ovb-ha https://review.openstack.org/566241 | 08:13 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Enabling CI for OVS-DPDK deployment https://review.openstack.org/586969 | 08:13 |
| quiquell | mrunge: so inception... | 08:13 |
| *** mdnadeem has joined #tripleo | 08:14 | |
| mrunge | quiquell: the linked job fails in both: validation and overcloud jobs | 08:14 |
| quiquell | mrunge: maybe now the issues we have see before, broke the thing. | 08:15 |
| mrunge | quiquell: can not read config file makes me think, there was an error either generating or providing the config file | 08:16 |
| mrunge | both are in ooo. good luck | 08:17 |
| quiquell | jrist: Do you know where do we generate the config for collectd kolla container ? | 08:17 |
| jbadiapa | mrunge, quiquell --> http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/config-data/collectd/etc/collectd.conf.txt.gz | 08:17 |
| jbadiapa | mrunge, quiquell, I think this is the file that you were looking for | 08:18 |
| quiquell | jbadiapa: Thanks so much | 08:18 |
| quiquell | mrunge: This is the file used by the container ? | 08:18 |
| mrunge | quiquell: I have no idea | 08:18 |
| mrunge | quiquell: but I see an issue at first glance | 08:19 |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-quickstart master: WIP: Enable OVS-DPDK in CI on featureset034 https://review.openstack.org/537886 | 08:19 |
| mrunge | this fqdnlookup true may be a reason for not starting | 08:19 |
| quiquell | mrunge: In the passing job is true too | 08:19 |
| bandini | jaosorior: https://review.openstack.org/#/c/586862/ https://review.openstack.org/#/c/586863/ (pike-only tls fixes around redis, if you have a min) | 08:19 |
| *** mhenkel_ has joined #tripleo | 08:20 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-quickstart master: WIP: Enable OVS-DPDK in CI on featureset034 https://review.openstack.org/537886 | 08:20 |
| quiquell | mrunge: Weird passing job, has collectd restarting too... maybe the check was not in place | 08:20 |
| mrunge | quiquell: there should be a log file in the container, or where-ever that is written to: "/var/log/collectd/collectd.log" | 08:20 |
| jbadiapa | mrunge, http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/containers/collectd/collectd.log.txt.gz | 08:21 |
| jaosorior | bandini: did you have a chance to test that? | 08:21 |
| quiquell | mrunge: This is it http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/containers/collectd/collectd.log.txt.gz | 08:21 |
| *** ykarel|lunch is now known as ykarel | 08:22 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: HTTP test during update/upgrade/ffwd. https://review.openstack.org/586030 | 08:22 |
| bandini | jaosorior: dciabrin tested it on a single node deploy, just do the review and I will +W when the three-node deploy completes successfully? | 08:23 |
| mrunge | quiquell: as said before, you should fix the rrdtool stuff | 08:23 |
| *** skramaja_ has joined #tripleo | 08:24 | |
| mrunge | quiquell: if we want rrdtool included in the container, it should be added via kolla | 08:24 |
| quiquell | mrunge: You mean in the dockerfile ? | 08:24 |
| mrunge | quiquell: yes | 08:24 |
| quiquell | mrunge: Add to the list of packages here https://github.com/openstack/kolla/blob/master/docker/collectd/Dockerfile.j2 ? | 08:25 |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New preflight check for the undercloud: disk space https://review.openstack.org/586541 | 08:25 |
| *** skramaja is now known as Guest77205 | 08:25 | |
| *** skramaja_ is now known as skramaja | 08:25 | |
| quiquell | mrunge: so ooo is trying to cinfigure collectd with rrdtool, but the plugin is not in the kolla container, that's it ? | 08:26 |
| mrunge | quiquell: that is one issue we see here. but that won't make the container restart | 08:27 |
| mrunge | quiquell: the plugin will just get ignored | 08:27 |
| quiquell | mrunge: We have also this intel_rdt: Error initializing PQoS library! | 08:28 |
| mrunge | right, there seems to be a missing lib | 08:28 |
| quiquell | but looks like it continues loading plugins | 08:28 |
| mrunge | right. it does | 08:28 |
| quiquell | So it's not the stopper | 08:28 |
| *** etingof has quit IRC | 08:28 | |
| mrunge | and note, that plugin is included since january | 08:28 |
| quiquell | mrunge: This is the last line before restart "[2018-07-30 02:08:58] plugin_load: plugin "write_prometheus" successfully loaded." | 08:29 |
| *** holser_ has joined #tripleo | 08:29 | |
| mrunge | quiquell: I can not see, where this is coming from, especially why it is getting loaded | 08:30 |
| mrunge | there is no loadplugin directive for this | 08:30 |
| mrunge | in configs | 08:30 |
| mrunge | http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/config-data/collectd/etc/collectd.d/ | 08:30 |
| *** etingof has joined #tripleo | 08:30 | |
| quiquell | mrunge: you have read my mind | 08:30 |
| *** d0ugal has joined #tripleo | 08:31 | |
| *** d0ugal has quit IRC | 08:31 | |
| *** d0ugal has joined #tripleo | 08:31 | |
| mrunge | quiquell: and again, the change for adding write_prometheus has been included in kolla for some time now | 08:31 |
| *** mhenkel_ has quit IRC | 08:32 | |
| quiquell | mrunge: Can be a timing issue, in the passing job, have restarted 3 times, at fail job has pass 4 times. | 08:32 |
| quiquell | mrunge: So maybe now it need another spin to get up and runnig | 08:33 |
| *** mhenkel_ has joined #tripleo | 08:33 | |
| *** mhenkel_ has quit IRC | 08:33 | |
| quiquell | mrunge: the health check is like from 10 days ago... maybe it's not really ready yet | 08:35 |
| quiquell | Have to be it | 08:35 |
| quiquell | https://github.com/openstack/tripleo-heat-templates/commit/bd1d5d72caf25010e373f1ad2ed6ebc5aee96914 | 08:36 |
| * mrunge clicks | 08:36 | |
| quiquell | owalsh: Good morning | 08:36 |
| quiquell | owalsh: You there ? | 08:37 |
| quiquell | mrunge: or maybe we have being always broken, and not checking it :-( | 08:37 |
| *** pblaho has joined #tripleo | 08:38 | |
| *** colonwq has joined #tripleo | 08:38 | |
| quiquell | mrunge: merge of the health check is from 27 | 08:38 |
| quiquell | mrunge: More or less when it has start to fail :-/ | 08:38 |
| quiquell | bogdando: Do you know everything about this https://github.com/openstack/tripleo-heat-templates/commit/0dd0b623798599b4ae0b3ff6f9ce4249c00c14df ? | 08:39 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/queens: Copy-in libvirt certs via kolla extended/start https://review.openstack.org/586977 | 08:40 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/queens: Copy-in redis certs via kolla extended/start https://review.openstack.org/586978 | 08:40 |
| *** skramaja_ has joined #tripleo | 08:40 | |
| quiquell | mrunge: The only thing I can do is try to make health check happy | 08:40 |
| *** skramaja has quit IRC | 08:41 | |
| *** skramaja_ is now known as skramaja | 08:41 | |
| bogdando | quiquell: I know that it reimplements the reverted thing | 08:42 |
| quiquell | bogdando: Looks like we screw the collectd kolla container config, and now we are detecting it | 08:43 |
| bogdando | quiquell: got logs? | 08:44 |
| *** skramaja_ has joined #tripleo | 08:46 | |
| *** skramaja is now known as Guest22393 | 08:46 | |
| *** skramaja_ is now known as skramaja | 08:46 | |
| quiquell | bogdando: The bug https://bugs.launchpad.net/tripleo/+bug/1784307 | 08:46 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,New] - Assigned to Gabriele Cerami (gcerami) | 08:46 |
| *** Guest22393 has quit IRC | 08:46 | |
| bogdando | thanks, looking, quiquell | 08:46 |
| *** gfidente has joined #tripleo | 08:46 | |
| *** gfidente has quit IRC | 08:46 | |
| *** gfidente has joined #tripleo | 08:46 | |
| bogdando | isn't what puppe tapply failed with "Warning: Unknown variable: 'ensure'. at /etc/puppet/modules/cinder/manifests/backup.pp:83:18" ? | 08:47 |
| bogdando | that* | 08:47 |
| quiquell | bogdando: Tha's a warning or really a failure ? | 08:48 |
| quiquell | bogdando: What really stops overcloud deploy is the health check of collectd | 08:48 |
| quiquell | bogdando: This could be caused by the warning ? | 08:48 |
| bogdando | nope, but that warning points to another 100% bug in puppet as well | 08:49 |
| bogdando | in puppet-tripleo | 08:49 |
| quiquell | bogdando: Damn it... | 08:49 |
| bogdando | bandini: hi! ^^ jfyi :) | 08:49 |
| quiquell | bogdando: btw, any idea on the collectd thing ? | 08:49 |
| bogdando | still looking | 08:49 |
| quiquell | bogdando: Ok ok | 08:50 |
| bogdando | got repro'ed env perchance? | 08:50 |
| quiquell | bogdando: I am starting it right now | 08:50 |
| quiquell | bogdando: Let's see if it works RDO cloud was acting weird last week | 08:50 |
| quiquell | bogdando: What I see is a "Error: Reading the config file failed!" at http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/extra/docker/containers/collectd/stdout.log.txt.gz | 08:51 |
| quiquell | bogdando: This is the config file it's failing ? http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/extra/docker/containers/collectd/config.json.txt.gz | 08:51 |
| bogdando | quiquell: so that commit https://github.com/openstack/tripleo-heat-templates/commit/0dd0b623798599b4ae0b3ff6f9ce4249c00c14df just did its job well, making the deployment fail early for the valid case | 08:52 |
| quiquell | Or are the generatd ones ? | 08:52 |
| quiquell | bogdando: Yep... | 08:52 |
| bogdando | and the root cause is bad collectd config | 08:52 |
| quiquell | bogdando: Have look at passing jobs and they where failing too, but not detected | 08:52 |
| quiquell | bogdando: This is my current line of thinking | 08:52 |
| quiquell | bogdando: Who knows how wrong I can be :-/ | 08:52 |
| quiquell | bogdando: The generated config is here http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/config-data/collectd/etc/ | 08:53 |
| bogdando | yeah, that tells me nothing :( | 08:54 |
| bogdando | may be it has some more verbose tool to check it | 08:54 |
| bogdando | live | 08:54 |
| bogdando | I have not idea | 08:55 |
| quiquell | bogdando: collectd.logs http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/containers/collectd/collectd.log.txt.gz | 08:55 |
| quiquell | bogdando: A lot of erros are there | 08:55 |
| quiquell | mrunge: Also point out that we should fix them | 08:55 |
| bogdando | yeah, I see. FWIW, the deployment failure looks right to me cuz of that thing | 08:55 |
| * bandini reads back | 08:56 | |
| *** mdnadeem_ has joined #tripleo | 08:56 | |
| bogdando | I mean it is nice we now have deployments not ignoring when running such broken containers | 08:56 |
| quiquell | bogdando: So we have "plugin_load: Could not find plugin "rrdtool" in /usr/lib64/collectd" | 08:56 |
| quiquell | bogdando: and "plugin_load: Could not find plugin "rrdtool" in /usr/lib64/collectd" | 08:56 |
| quiquell | bogdando: We can try to fix both, and see if it works | 08:57 |
| bogdando | looks like some kolla issues | 08:57 |
| bogdando | or tripleo specific overrides for collectd | 08:57 |
| quiquell | bogdando: I mean plugin_load: Could not find plugin "rrdtool" in /usr/lib64/collectd | 08:57 |
| *** mdnadeem has quit IRC | 08:57 | |
| quiquell | damn... the PQoS library | 08:57 |
| quiquell | thing | 08:57 |
| bogdando | yeah, it should be installed from some rpm | 08:57 |
| quiquell | Fuck... copy paste | 08:57 |
| bogdando | so kolla, or tripleo overrides for it | 08:58 |
| quiquell | I think we have some logs of the package installation of the images... | 08:58 |
| quiquell | at docker_journald | 08:58 |
| quiquell | https://github.com/openstack/kolla/blob/master/docker/collectd/Dockerfile.j2 is missing rrd | 09:01 |
| quiquell | Maybe we have to add it there | 09:01 |
| *** shardy has joined #tripleo | 09:04 | |
| *** salmankhan has joined #tripleo | 09:04 | |
| *** sshnaidm has quit IRC | 09:07 | |
| *** sshnaidm has joined #tripleo | 09:09 | |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 09:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 09:10 |
| owalsh | quiquell: hi | 09:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 09:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 09:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 09:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 09:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,New] - Assigned to Gabriele Cerami (gcerami) | 09:10 |
| *** sshnaidm is now known as sshnaidm|afk | 09:12 | |
| *** jfrancoa has quit IRC | 09:13 | |
| *** jfrancoa has joined #tripleo | 09:15 | |
| *** quiquell is now known as quiquell|mtg | 09:15 | |
| mrunge | quiquell|mtg: bogdando: collectd usually continues to work, even if a plugin fails to load | 09:19 |
| mrunge | quiquell|mtg: bogdando: I can only see tripleo using collectd-rrdtool, I would suggest to add it there then | 09:19 |
| mrunge | and I'll have a look at the pqos lib thingy | 09:20 |
| *** rpioso is now known as rpioso|afk | 09:22 | |
| *** psachin` has joined #tripleo | 09:22 | |
| *** psachin has quit IRC | 09:22 | |
| *** paramite has joined #tripleo | 09:24 | |
| *** pdeore has quit IRC | 09:26 | |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-quickstart master: Enable containerized undercloud in scenario000-upgrades. https://review.openstack.org/583515 | 09:32 |
| *** dtantsur|afk is now known as dtantsur | 09:35 | |
| *** quiquell|mtg is now known as quiquell | 09:37 | |
| quiquell | owalsh: Looks like after adding the healthcheck it discover that we have issues at collectd | 09:38 |
| quiquell | mrunge: Have read something about kernel modules and pqos lib | 09:38 |
| owalsh | quiquell: ack | 09:38 |
| quiquell | owalsh: https://bugs.launchpad.net/tripleo/+bug/1784307 | 09:38 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,New] - Assigned to Gabriele Cerami (gcerami) | 09:38 |
| mrunge | quiquell: first thing I'd like to check is, if intel-cmt-cat is really installed in the container | 09:39 |
| mrunge | quiquell: it should be | 09:39 |
| quiquell | mrunge: We have a docker_journld, maybe the info is there | 09:39 |
| mrunge | quiquell: next issue I see is, collectd-rdt is tried to get loaded; however there is no load directive for that | 09:40 |
| mrunge | same as for write_prometheus | 09:40 |
| quiquell | mrunge: Maybe it has a by default stuff constructed inside it ? | 09:41 |
| mrunge | quiquell: it would help to add AutoLoadPligin false to collectd.conf file | 09:41 |
| quiquell | mrunge: Don't know how many layers I have to look up to know where to do that :-) | 09:42 |
| mrunge | quiquell: me neither :-/ | 09:42 |
| mrunge | the reproducer script fails for me, since openstack stack is not provided or installe | 09:43 |
| mrunge | any idea what rpm provides "stack" ? | 09:43 |
| quiquell | mrunge: Maybe this also help you http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/undercloud/home/zuul/docker_journalctl.log.txt.gz | 09:43 |
| bogdando | jfrancoa: hi! https://review.openstack.org/#/c/465047/ let's see if it is ready now | 09:43 |
| bogdando | fingers crossed | 09:43 |
| quiquell | mrunge: You are missing heat | 09:43 |
| bogdando | updated https://review.openstack.org/#/c/583515/ as well, jfrancoa | 09:44 |
| quiquell | mrunge: python-heatclient | 09:44 |
| quiquell | mrunge: pip install --user python-heatclient | 09:44 |
| jfrancoa | bogdando: I'll start praying everything I remember (it's not too much :-D) | 09:44 |
| bogdando | :D | 09:44 |
| *** ratailor has quit IRC | 09:44 | |
| *** colonwq has quit IRC | 09:44 | |
| mrunge | quiquell: thank you, but no, I consider pip as harmful | 09:44 |
| Tengu | hmm. nice Monday with CI playing tricks :D | 09:45 |
| *** colonwq has joined #tripleo | 09:45 | |
| quiquell | mrunge: then yum python2-heatclient | 09:46 |
| mrunge | quiquell: yes, already added it to my playbook, thank you :) | 09:46 |
| quiquell | Tengu: We are on it... but we cannot land the patches that fix it, | 09:46 |
| openstackgerrit | Merged openstack/instack-undercloud master: Restart rsyslog after installing Swift https://review.openstack.org/581990 | 09:46 |
| quiquell | Tengu: https://bugs.launchpad.net/tripleo/+bug/1784307 | 09:47 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,New] - Assigned to Gabriele Cerami (gcerami) | 09:47 |
| Tengu | quiquell: damn. CI-ception that is? :D | 09:47 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: QDR for metrics collection purposes https://review.openstack.org/572312 | 09:47 |
| quiquell | Tengu: yes man... it's like the perfect storm... you don't want to know | 09:47 |
| *** sshnaidm|afk has quit IRC | 09:47 | |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-common master: Make MetricsQdr service use qdrouterd image https://review.openstack.org/578749 | 09:47 |
| Tengu | quiquell: hm, so collectd config error? | 09:49 |
| quiquell | Tengu: Feels like, but looks like the plugin stuff is not the main issue, as collectd continues... | 09:50 |
| mrunge | adding AutoLoadPlugin false to the collectd.conf should help | 09:50 |
| quiquell | mrunge: Yep, trying to find where, bogdando, Tengu ^? | 09:50 |
| Tengu | http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/containers/collectd/collectd.log.txt.gz nice... | 09:50 |
| quiquell | Tengu: collectd is super unhappy right now | 09:51 |
| Tengu | of course it is. apparently there are issues in its configuration. How would you be if you had issues in your vision analysis brain part? :D | 09:52 |
| Tengu | basically there are two missing things in collectd configuration, according to its logs: Could not find plugin "rrdtool" in /usr/lib64/collectd (missing rrdtool things for stats recording) and intel_rdt: Error initializing PQoS library! | 09:53 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: Fix centos opstools repo deactivation https://review.openstack.org/577146 | 09:53 |
| quiquell | mrunge: Looks like overcloud deploy has pass in the last recheck I did, so this is not deterministic... :-/ | 09:55 |
| mrunge | quiquell: so, apparently autoloadplugin is not supported by puppet-collectd right now | 09:55 |
| Tengu | well, if rrdtool is to be loaded - it has to be installed as well. No the case apparently. Thus error. http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/config-data/collectd/etc/collectd.d/10-rrdtool.conf.txt.gz | 09:56 |
| Tengu | question is: why loading rrd if stats are sent to gnocchi? | 09:56 |
| mrunge | Tengu: that is nothing new | 09:56 |
| mrunge | Tengu: iirc, someone wanted to collect metrics during tripleo runs | 09:57 |
| mrunge | however, collectd will continue to run, even if a plugin is not being loaded | 09:58 |
| mrunge | or fails to load | 09:58 |
| mrunge | and the change was back in March(?) | 09:58 |
| openstackgerrit | Martin Mágr proposed openstack/tripleo-heat-templates master: Add reflection of RpcPort to health checks https://review.openstack.org/583629 | 09:58 |
| quiquell | mrunge: Does it make sense that it's not deterministic ? would it depend on the type of underlying maching running it ? | 09:58 |
| mrunge | quiquell: nope, it does not | 09:59 |
| quiquell | mrunge: I have a running job from the latest and it's passing, let if finish and check | 09:59 |
| quiquell | mrunge: but the last run of the noop change we have fails | 09:59 |
| mrunge | quiquell: I assume there is a timing issue somewhere else | 09:59 |
| quiquell | mrunge: So maybe it depends in something else that is installing at the same time ? | 10:00 |
| mrunge | quiquell: the collectd container should not depend on anything external | 10:00 |
| *** ratailor has joined #tripleo | 10:00 | |
| mrunge | if it does, that's an issue | 10:00 |
| owalsh | quiquell: which job is passing? | 10:01 |
| quiquell | mrunge: Job timed out... we are so unlucky ... | 10:01 |
| *** ksambor is now known as ksambor|lunch | 10:01 | |
| mrunge | fail to deploy? welcome to my world | 10:01 |
| quiquell | owalsh: http://logs.openstack.org/99/586499/1/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/5c707c2/ | 10:01 |
| quiquell | mrunge: overcloud deploy is good here ^ | 10:01 |
| quiquell | but it timed out :-( | 10:02 |
| quiquell | So we don't have logs | 10:02 |
| bogdando | https://www.youtube.com/watch?v=Ev-Ru1QpTqU | 10:02 |
| quiquell | also is very inception, we need the fix to have the logs so no logs in the fix | 10:02 |
| quiquell | bogdando: https://review.openstack.org/#/c/585528/ is not going to merge until we make this land https://review.openstack.org/#/c/586499/ | 10:04 |
| bogdando | ._. | 10:05 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container https://review.openstack.org/584368 | 10:06 |
| *** bkopilov has quit IRC | 10:07 | |
| *** slaweq has quit IRC | 10:07 | |
| *** colonwq has quit IRC | 10:08 | |
| *** colonwq has joined #tripleo | 10:09 | |
| *** slaweq has joined #tripleo | 10:09 | |
| *** brault has quit IRC | 10:10 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 10:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 10:10 |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 10:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 10:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 10:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 10:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,New] - Assigned to Gabriele Cerami (gcerami) | 10:10 |
| *** Petersingh has quit IRC | 10:12 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Remove single quote from docker inspect command in tempest run https://review.openstack.org/586213 | 10:13 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container https://review.openstack.org/584368 | 10:19 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Remove single quote from docker inspect command in tempest run https://review.openstack.org/586213 | 10:19 |
| *** colonwq has quit IRC | 10:22 | |
| *** colonwq has joined #tripleo | 10:22 | |
| quiquell | mrunge: At the end "intel_rdt: Error initializing PQoS library" can be fatal ? | 10:22 |
| mrunge | quiquell: nope | 10:23 |
| quiquell | mrunge: So none of the fails at the logs justify the restart ? | 10:23 |
| mrunge | quiquell: not really, no | 10:23 |
| quiquell | mrunge: Could be that it need more time or retries to be up and running ? | 10:23 |
| mrunge | quiquell: the first thing would be the FQDNLookup thing. I'd set it to false | 10:24 |
| mrunge | but you'd see something in logs | 10:24 |
| quiquell | mrunge: There is no logs pointing to that | 10:24 |
| mrunge | yupp | 10:24 |
| mrunge | quiquell: the think, which makes me really concerned is, you see all the plugins getting loaded, but there is no load directive for them | 10:26 |
| quiquell | mrunge: So maybe we are looking at the wrong config | 10:26 |
| mrunge | yes? | 10:26 |
| quiquell | mrunge: Or it's loading a default want as a fallback from error at config | 10:26 |
| jaosorior | bandini: /exit | 10:26 |
| *** jaosorior has quit IRC | 10:26 | |
| quiquell | mrunge: Don't know man I am a little CI man | 10:27 |
| quiquell | mrunge: Comparing with the latest working one | 10:27 |
| chandankumar | bogdando: Hello | 10:27 |
| mrunge | quiquell: the last successful tripleo deployment for me was in November last year | 10:27 |
| mrunge | afterwards, all attempts failed | 10:27 |
| mrunge | and there were many | 10:27 |
| quiquell | mrunge: this works http://logs.openstack.org/13/586213/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/f8352ab/logs/ | 10:27 |
| quiquell | I mean it passes | 10:28 |
| chandankumar | bogdando: https://review.openstack.org/#/c/584368/3/roles/validate-tempest/defaults/main.yml@27 | 10:28 |
| bogdando | chandankumar: hi | 10:28 |
| chandankumar | bogdando: so will i directly use local_docker_registery_host there? | 10:28 |
| bogdando | I'm not sure, may be the way to go is setting docker_registry_host to local_docker_registery_host ?.. | 10:29 |
| bogdando | in featuresets or so | 10:29 |
| mrunge | quiquell: what catches my eye here is, collectd is restarted as well, but after 13 mins, not after 6 mins | 10:29 |
| quiquell | mrunge: It's doing the health check http://logs.openstack.org/13/586213/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/f8352ab/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-07-30_07_36_21 | 10:29 |
| chandankumar | bogdando: tempest_container depends on containerized UC, when there is a containerized UC there will be local_registery | 10:30 |
| quiquell | mrunge: In the working one Restarting (1) 6 minutes ago | 10:30 |
| quiquell | mrunge: So maybe collectd takes some restarts to work | 10:31 |
| chandankumar | in FS we enable undercloud_enable_tempest to true then tempest container will work | 10:31 |
| mrunge | quiquell: not really. | 10:31 |
| *** leanderthal has joined #tripleo | 10:31 | |
| mrunge | quiquell: any idea how the health of collectd container is determined? | 10:32 |
| bogdando | chandankumar: ack, then you can rely on containerized_undercloud: true | 10:33 |
| quiquell | mrunge: This is the mechanism https://github.com/openstack/tripleo-heat-templates/commit/bd1d5d72caf25010e373f1ad2ed6ebc5aee96914 | 10:33 |
| quiquell | mrunge: Looks like it has zero tolerance, then why this last job is passing ? | 10:34 |
| quiquell | owalsh: Do have the health check any tolerance ? | 10:34 |
| Tengu | apparently the healtcheck as in tripleo-common/healthcheck is : collectdctl -s /var/run/collectd-socket listval | 10:34 |
| Tengu | https://github.com/openstack/tripleo-common/blob/master/healthcheck/collectd <- | 10:35 |
| quiquell | Tengu: But we are talking about container healthcheck https://github.com/openstack/tripleo-heat-templates/commit/bd1d5d72caf25010e373f1ad2ed6ebc5aee96914 | 10:35 |
| owalsh | quiquell: the task that checks the health waits five minutes | 10:35 |
| quiquell | Tengu: something new from this friday | 10:35 |
| owalsh | if any are restarting/unhealthy | 10:35 |
| *** ksambor|lunch is now known as ksambor | 10:35 | |
| owalsh | quiquell: looks like we can sometimes miss a restarting container though | 10:35 |
| quiquell | owalsh: I have a passing job with a restart on the logs | 10:36 |
| Tengu | quiquell: well, the container healthcheck should be check by the code I just linked. | 10:36 |
| quiquell | owalsh: That's expected ? | 10:36 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container https://review.openstack.org/584368 | 10:36 |
| owalsh | quiquell: no, that's what I mean, it should fail if there is a restarting container | 10:36 |
| quiquell | Tengu: Ahh ok, so the container reports back the output from the script | 10:36 |
| Tengu | i.e. http://logs.openstack.org/45/560445/99/check/tripleo-ci-centos-7-scenario001-multinode-oooq-container/bf6c481/logs/subnode-2/var/log/extra/docker/containers/collectd/docker_info.log.txt.gz | 10:36 |
| Tengu | quiquell: you can check in the log the state -^ | 10:37 |
| Tengu | "Status": "unhealthy", | 10:37 |
| quiquell | owalsh: This one is passing http://logs.openstack.org/13/586213/1/gate/tripleo-ci-centos-7-scenario001-multinode-oooq-container/f8352ab/logs/subnode-2/var/log/extra/docker/docker_allinfo.log.txt.gz | 10:37 |
| Tengu | damn ;). | 10:37 |
| quiquell | owalsh: with a restart on collectd | 10:37 |
| owalsh | quiquell: https://review.openstack.org/#/c/584119/10/common/deploy-steps-tasks.yaml | 10:37 |
| owalsh | quiquell: implies docker doesn't report is as restarting | 10:37 |
| *** colonwq has quit IRC | 10:38 | |
| quiquell | Tengu, owalsh: so we have a unhealthy collectd but the job is passing | 10:39 |
| quiquell | So we have two problems false positives at healtcheck and collectd failing, how happy I am | 10:39 |
| Tengu | quiquell: basically, most of the containers have a healthcheck embedded, and they report their status via the exic code, and docker takes actions like restarting it if it's unhealthy. the new check you linked just checks the output/status | 10:39 |
| mrunge | quiquell: Tengu the collectd health check from triplo-common should fail, unless the corresponding plugin providing the socket is loaded | 10:40 |
| Tengu | mrunge: +1 | 10:40 |
| mrunge | quiquell: Tengu which should not be loaded | 10:40 |
| quiquell | Tengu: Tengo it not only outpus it it also make the playbook fail | 10:40 |
| owalsh | quiquell: could be that the container doesn't fail quickly, and docker reports it as running if it stays up for long enough | 10:40 |
| openstackgerrit | Arx Cruz proposed openstack-infra/tripleo-ci master: Fix stackviz data https://review.openstack.org/582468 | 10:40 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix stackviz installation https://review.openstack.org/580361 | 10:40 |
| Tengu | quiquell: of course, as "unhealthy" isn't an acceptable state :). | 10:40 |
| quiquell | owalsh: Logs could be newer than the check ? | 10:41 |
| mrunge | Tengu: quiquell question is, why so many collectd plugins are loaded, especially when the config states otherwise | 10:41 |
| owalsh | quiquell: and healthchecks do no report unhealthy until 3x healthchecks fail | 10:41 |
| Tengu | mrunge: what config? note: I think puppet-collectd loads a couple of plugins by default. | 10:41 |
| quiquell | owalsh: Do the healthcheck finish until all the state is beyond running ? | 10:42 |
| mrunge | Tengu: for example collectd-rdt is installed, but there is no config for that | 10:42 |
| mrunge | but you see an error about it (pqos library...) | 10:42 |
| mrunge | or the prometheus plugin is loaded | 10:42 |
| mrunge | or the ping plugin | 10:42 |
| quiquell | Tengu couls also be that health check fails but collectd works after too much retries ? | 10:42 |
| Tengu | hmm yep. wait. checking the config itself. | 10:43 |
| quiquell | owalsh: Was for you the question sorry | 10:43 |
| Tengu | quiquell: well, if the healtcheck fails, docker engine should restart the container | 10:43 |
| mrunge | either there is a autopluginload true hidden | 10:43 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Remove single quote from docker inspect command in tempest run https://review.openstack.org/586213 | 10:43 |
| Tengu | had that kind of issue with some ironic-pxe-tftp container in fact. | 10:43 |
| mrunge | or we are looking at the wrong config | 10:43 |
| owalsh | Tengu: no, docker doesn't restart unhealthy containers | 10:43 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Remove single quote from docker inspect command in tempest run https://review.openstack.org/586213 | 10:43 |
| quiquell | Tengo: Do you know if collectdctl -s /var/run/collectd-socket listval can take some time to pass ? | 10:44 |
| Tengu | owalsh: well, it restarting it, at least for the ironic-pxe-tftp :). | 10:44 |
| Tengu | quiquell: no idea, wait, checking in my undercloud. | 10:44 |
| quiquell | Tengu: I mean race condition | 10:44 |
| mrunge | quiquell: if you don't have collectd unixsock plugin installed, it will fail instantly | 10:44 |
| owalsh | Tengu: then container is exiting with non-zero | 10:44 |
| Tengu | damn..... not deployed with containers -.-'. | 10:44 |
| * mrunge heads to the kitchen to feed some hungry mouths | 10:45 | |
| Tengu | owalsh: there were mutliple issues, all of them caused by the fact xinetd was still launching the tftpd service on the host, preventing the container to listen on the same interface - so yeah, maybe it was due to the exit status. That was last week -.-'. | 10:45 |
| *** udesale has quit IRC | 10:47 | |
| quiquell | I am going to have it reproduced | 10:47 |
| owalsh | Tengu: yea, that would cause a restart loop I expect (if restart: always was set) | 10:48 |
| Tengu | owalsh: yep. funnily, there isn't such an option with podman, apparently :] | 10:48 |
| Tengu | but that's another story :D | 10:48 |
| *** zzzeek has quit IRC | 10:49 | |
| quiquell | owalsh, Tengu: This is the bug, https://bugs.launchpad.net/tripleo/+bug/1784307 | 10:49 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 10:49 |
| *** colonwq has joined #tripleo | 10:49 | |
| quiquell | In case you want to add stuff to get it noted | 10:49 |
| *** panda|rover|off is now known as panda|rover | 10:50 | |
| quiquell | panda|rover: Good morning | 10:50 |
| panda|rover | morning | 10:50 |
| panda|rover | but there's nothing good | 10:51 |
| quiquell | panda|rover: Agree... | 10:51 |
| Tengu | quiquell: added the healthcheck link + state of the collectd container. | 10:51 |
| quiquell | panda|rover: Looks like there is a new health check merged on 27th that prevent the patched we need to land | 10:51 |
| quiquell | https://bugs.launchpad.net/tripleo/+bug/1784307 | 10:51 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 10:51 |
| quiquell | panda|rover: Also the error is non deterministic :-/ | 10:52 |
| panda|rover | quiquell: yes, apparently many containers are unhealthyt now | 10:52 |
| quiquell | panda|rover: Perfect timing, to fix the gates :-(((( | 10:52 |
| Tengu | quiquell: a quick-fix might be to revert the health check path in order to get the "so badly needed patch" to merge, then re-apply the health-thingy and go further down? | 10:52 |
| quiquell | Tengu: don't know... panda|rover what do you say ? | 10:53 |
| panda|rover | at this point everything is a risk, we're merging blindly | 10:53 |
| panda|rover | but we need to unblock the situation | 10:53 |
| *** zzzeek has joined #tripleo | 10:53 | |
| quiquell | panda|rover: Agree | 10:53 |
| Tengu | pragamtic thinking: revert the blocking patch | 10:54 |
| quiquell | panda|rover: the tags need to be there | 10:54 |
| quiquell | Even openstack infra failed on friday... :-/ | 10:55 |
| panda|rover | the tags and playbook patch is essential yes, and it has proven to be the mostly tested, everything was passing at different tages | 10:55 |
| quiquell | panda|rover: We don't know if anything else will prevent reverting health-check from merge... now I am at panda mode | 10:56 |
| panda|rover | ... | 10:56 |
| *** ratailor has quit IRC | 10:56 | |
| *** rh-jelabarre has joined #tripleo | 10:56 | |
| quiquell | panda|rover: Do you want to sync or something ? | 10:57 |
| owalsh | quiquell: I wonder if a restarting container briefly goes to an exited status before docker restarts it... would explain why it sometimes passes | 10:58 |
| panda|rover | quiquell: your bug is a duplicate of https://bugs.launchpad.net/tripleo/+bug/1784072 ? | 10:59 |
| openstack | Launchpad bug 1784072 in tripleo "scenario001 fails due to unhealthy collectd container" [Critical,Invalid] | 10:59 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: fix typos in docs and .gitignore https://review.openstack.org/585487 | 10:59 |
| panda|rover | quiquell: or https://bugs.launchpad.net/tripleo/+bug/1784073 | 10:59 |
| openstack | Launchpad bug 1784073 in tripleo "scenario003 fails because many containers are unhealthy" [Critical,Invalid] | 10:59 |
| panda|rover | invalid ? | 10:59 |
| quiquell | panda|rover: Yep | 10:59 |
| panda|rover | hhhmm | 10:59 |
| panda|rover | why invalid | 10:59 |
| quiquell | panda|rover: Didn't find it | 11:00 |
| panda|rover | ah, beacause it's not consistent | 11:00 |
| quiquell | panda|rover: Because is non determiknistic | 11:00 |
| panda|rover | ok | 11:00 |
| quiquell | panda|rover: We can r-open alex one and mark as duplicate mine, copy the comments | 11:01 |
| panda|rover | do we have any idea of fix for the containers failures ? | 11:04 |
| quiquell | panda|rover: Looks like non of the errors at collectd.log is fatal | 11:04 |
| panda|rover | and are the containers effectively unhealthy or the health check is bugged ? | 11:04 |
| quiquell | panda|rover: Maybe is a race condition | 11:05 |
| *** sshnaidm|afk has joined #tripleo | 11:05 | |
| owalsh | panda|rover, quiquell: if the container is restarting then it's exiting with non-zero rc | 11:05 |
| *** jangutter has joined #tripleo | 11:05 | |
| *** gbarros has joined #tripleo | 11:06 | |
| panda|rover | is there any possibility anything passes throgh the gates now ? | 11:06 |
| quiquell | panda|rover: Look at noop, don't feel like it | 11:07 |
| quiquell | panda|rover: Only the scenario001 is the one failing at noops | 11:07 |
| owalsh | panda|rover, quiquell: but if the containers are running but still unhealthy after 5 minutes then something ain't right... either the healthchecks are broken or there are lots of races | 11:08 |
| panda|rover | ok so we need the project adminitrators to merge anything. Next thing we need is a clear list of things we should merge | 11:08 |
| quiquell | panda|rover: we don't want to try to revert health check ? | 11:09 |
| panda|rover | owalsh: yeah, move forward with this seems not an option, revert is the best course of action | 11:09 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 11:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 11:10 |
| panda|rover | quiquell: yes, so, my suggestion is to merge https://review.openstack.org/586499, at tleat most of the tests were succedeing before the containrs issue. With this we avoid other false positives. Then rever the containers check, | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 11:10 |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 11:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 11:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 11:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 11:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 11:10 |
| panda|rover | openstack: thanks, let's see what else is critical ... | 11:11 |
| panda|rover | what's rdo cloud status ? | 11:11 |
| quiquell | panda|rover: To merge that we need to revert the health check or to be lucky as hell | 11:11 |
| quiquell | panda|rover: Or can we force it ? | 11:11 |
| quiquell | panda|rover: RDO is good | 11:11 |
| quiquell | https://semaphore.cloud.upshift.engineering.redhat.com/ | 11:11 |
| panda|rover | quiquell: I think project administrator can +2 and bypass jobs verifications, but I'm not sure they want to do that | 11:12 |
| panda|rover | if theyt don't want to do that, yes, we can try with revert first, then tags and playbooks last | 11:12 |
| quiquell | panda|rover: Yep but we don't know if anything else will prevent the revert to merge... everything feels bad... Damn I need my PTO | 11:13 |
| panda|rover | quiquell: ok even my tenant is working properly, so rdo cloud should be at least partially solved | 11:13 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Enable support for running refstack tests in TQE https://review.openstack.org/570719 | 11:14 |
| quiquell | panda|rover: I am near to reproduce this, if it's of any help | 11:14 |
| Tengu | bogdando: answered to your comment on my run_ansible thingy. | 11:14 |
| panda|rover | quiquell: we are left with: tags and playbook + updates and false positives, containers health, the only thing I'm missing from the critical is the disable-noveau | 11:15 |
| panda|rover | let's see how critical it is. | 11:15 |
| quiquell | panda|rover: What's that | 11:15 |
| panda|rover | quiquell: https://bugs.launchpad.net/tripleo/+bug/1784078 | 11:16 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 11:16 |
| quiquell | panda|rover: noop is no longer failing at this I think | 11:16 |
| panda|rover | quiquell: it's critical for queens, is the container problem showing in all branches ? | 11:16 |
| panda|rover | or just mater ? | 11:16 |
| quiquell | panda|rover: Fix was merge | 11:16 |
| quiquell | panda|rover: http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&from=1532776612152&to=1532949412152&var-launchpad_tags=alert&var-promotion_names=current-tripleo&var-promotion_names=current-tripleo-rdo&var-promotion_names=current-tripleo-rdo-testing&var-releases=master&var-releases=queens&var-releases=pike&var-releases=ocata&panelId=205&fullscreen | 11:16 |
| panda|rover | mmhh, why is it not fixed then ? | 11:17 |
| quiquell | panda|rover: You mean the bug not closed ? | 11:18 |
| panda|rover | quiquell: yep, maybe launchpad does not consider fix taht are not in master | 11:18 |
| panda|rover | quiquell: anyway, one thing less to think about, so we are left with two blocking | 11:19 |
| quiquell | panda|rover: Could be, don't know if you can specify the branch in the bug | 11:19 |
| quiquell | updates + health check | 11:19 |
| quiquell | that's it ? | 11:19 |
| panda|rover | quiquell: ok, let's revert the container check, what's the change that needs to be reverted ? | 11:19 |
| panda|rover | quiquell: well, yeah updates + tags + playbook + upgrades | 11:19 |
| quiquell | panda|rover: is in the launchpad | 11:19 |
| quiquell | panda|rover: Also, I have see minor issues at emitter trying to get RDO .repo file | 11:20 |
| quiquell | panda|rover: It fails after 10 retries, but maybe just a network hiccup at RDO | 11:20 |
| quiquell | https://bugs.launchpad.net/tripleo/+bug/1784351 | 11:21 |
| openstack | Launchpad bug 1784351 in tripleo "Failed to retrieve repo file from https://trunk.rdoproject.org/centos7-master/current-tripleo/delorean.repo after 10 retries" [High,New] - Assigned to Gabriele Cerami (gcerami) | 11:21 |
| panda|rover | owalsh: you confirm that is the cause of the failures ? is it ok to have it reverted or you have other solutions to propose ? | 11:21 |
| openstackgerrit | Rafal Szmigiel proposed openstack/tripleo-common master: Changing the method Octavia's controller hostname is obtained from shell command to use Ansible's ansible_fqdn fact. https://review.openstack.org/587004 | 11:21 |
| quiquell | panda|rover: This is the commit | 11:22 |
| quiquell | https://github.com/openstack/tripleo-heat-templates/commit/0dd0b623798599b4ae0b3ff6f9ce4249c00c14df | 11:22 |
| openstackgerrit | Rafal Szmigiel proposed openstack/tripleo-common master: Changing the method Octavia's controller hostname is obtained from shell command to use Ansible's ansible_fqdn fact. https://review.openstack.org/587004 | 11:22 |
| quiquell | panda|rover, owalsh: I am going to do a revert on it ok ? | 11:22 |
| owalsh | panda|rover: well, cause of the failure is the unhealthy/restarting containers... but ok to revert until we sort this out | 11:22 |
| owalsh | the check is working as it should, although we appear to miss the restarting collectd container sometimes | 11:24 |
| *** bkopilov has joined #tripleo | 11:24 | |
| panda|rover | owalsh: ok, so the unhealthyness is real, your change is just showing it. I understand. Your patch should next be accompanied with the corresponding fixes to make scenario jobs pass. | 11:24 |
| panda|rover | owalsh: ok, we are in damage congtrol at the moment, so revert is the quickest way, probably not the best | 11:25 |
| quiquell | owalsh: But still the health check passes with some failures at collectd, so still there is something weird there | 11:25 |
| quiquell | owalsh: The health check is only for master ? | 11:25 |
| owalsh | quiquell: yea, not backported | 11:25 |
| *** zoli is now known as zoli|doctor | 11:26 | |
| quiquell | owalsh: Cool thanks | 11:26 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: Revert "Fix deploy health checks" https://review.openstack.org/587006 | 11:28 |
| quiquell | owalsh, panda|rover: ^ | 11:28 |
| *** jpena is now known as jpena|lunch | 11:29 | |
| quiquell | panda|rover: Going to wait for scenario001 to finish in jistr patch in case it passes the health check and get merged | 11:30 |
| quiquell | panda|rover: ok ? | 11:30 |
| quiquell | panda|rover: I mean to add the Depends-On there | 11:30 |
| panda|rover | quiquell: ok | 11:31 |
| *** jaosorior has joined #tripleo | 11:32 | |
| quiquell | panda|rover: scenarios are exercises at gates ? | 11:32 |
| quiquell | our playbooks + tags review does not | 11:32 |
| quiquell | but don't know at jistr | 11:32 |
| *** quiquell is now known as quiquell|lunch | 11:33 | |
| jistr | i think some scenarios are run in gate too | 11:33 |
| panda|rover | quiquell: we are testing scenario001 in that tripleo-ci patch | 11:33 |
| quiquell|lunch | panda|rover: Ok na I will wait to see if it passes... so we don't have to wait for the revert o be merged | 11:34 |
| *** rfolco|off is now known as rfolco|ruck | 11:36 | |
| panda|rover | jistr: yes, it's per project, so anything that uses *-minimal set of jobs checks at least scenario000, if it uses *-full it checks all the scenarios | 11:36 |
| *** morazi has joined #tripleo | 11:36 | |
| quiquell|lunch | tripleo-common has scenario001 in the gates... | 11:38 |
| quiquell|lunch | panda|rover: we have to be very luck if it passes | 11:38 |
| quiquell|lunch | twice | 11:38 |
| panda|rover | quiquell|lunch: well with the change reverted, it should pass | 11:39 |
| quiquell|lunch | panda|rover: Yep, talking about the actual running in the review, I am keeping it and not adding the Depends-On | 11:39 |
| panda|rover | quiquell|lunch: ah, ok | 11:40 |
| quiquell|lunch | panda|rover: To see if it save us some time, before revert is merged | 11:40 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes from toci_gate_test.sh https://review.openstack.org/587012 | 11:41 |
| *** ade_lee has joined #tripleo | 11:44 | |
| openstackgerrit | Cédric Jeanneret proposed openstack/python-tripleoclient master: New function: run_ansible_playbook https://review.openstack.org/586538 | 11:44 |
| *** wolverineav has joined #tripleo | 11:44 | |
| openstackgerrit | Merged openstack/os-net-config master: Fix numbered NIC mapping when using dotted VLAN notation https://review.openstack.org/586507 | 11:47 |
| *** ansmith has quit IRC | 11:48 | |
| *** shreshtha has quit IRC | 11:48 | |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Gate on scenario010 https://review.openstack.org/587015 | 11:48 |
| openstackgerrit | Rafal Szmigiel proposed openstack/tripleo-common master: Changing the method to obtain controller's FQDN. https://review.openstack.org/587004 | 11:49 |
| *** panda|rover is now known as panda|lunch | 11:51 | |
| openstackgerrit | Saravanan KR proposed openstack/tripleo-heat-templates master: WIP: Enabling CI for OVS-DPDK deployment https://review.openstack.org/586969 | 11:52 |
| *** ohochman has joined #tripleo | 11:52 | |
| openstackgerrit | Carlos Goncalves proposed openstack/tripleo-common master: Changing the method to obtain controller's FQDN https://review.openstack.org/587004 | 11:53 |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Fix parameter name used to create the Manila CephX keyring https://review.openstack.org/585562 | 11:54 |
| *** Haresh has quit IRC | 11:56 | |
| *** pchavva has joined #tripleo | 12:01 | |
| *** gbarros has quit IRC | 12:04 | |
| *** pradk has joined #tripleo | 12:04 | |
| sshnaidm|afk | if somebody has +2 on kolla, please review promotion blocker: https://review.openstack.org/#/c/586904/ | 12:05 |
| *** gbarros has joined #tripleo | 12:05 | |
| openstackgerrit | Giulio Fidente proposed openstack/tripleo-heat-templates master: Fix parameter name used to create the Manila CephX keyring https://review.openstack.org/585562 | 12:05 |
| *** pchavva has quit IRC | 12:06 | |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 12:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 12:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 12:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiří Stránský (jistr) | 12:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 12:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,Triaged] - Assigned to Gabriele Cerami (gcerami) | 12:10 |
| *** quiquell|lunch is now known as quiquell | 12:11 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix errors using multiple-nics templates w/o VLANs defined https://review.openstack.org/586721 | 12:14 |
| *** mdnadeem_ has quit IRC | 12:18 | |
| *** mdnadeem has joined #tripleo | 12:18 | |
| *** Haresh has joined #tripleo | 12:18 | |
| *** mdnadeem_ has joined #tripleo | 12:18 | |
| *** peereb has quit IRC | 12:19 | |
| openstackgerrit | Martin Schuppert proposed openstack/puppet-tripleo master: Move nova-metadata api to httpd wsgi https://review.openstack.org/582622 | 12:19 |
| *** ratailor has joined #tripleo | 12:19 | |
| *** gkadam has quit IRC | 12:19 | |
| *** dmacpher has joined #tripleo | 12:20 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Remove single quote from docker inspect command in tempest run https://review.openstack.org/586213 | 12:20 |
| openstackgerrit | Gabriele Cerami proposed openstack/tripleo-heat-templates master: Revert "Fix deploy health checks" https://review.openstack.org/587006 | 12:20 |
| *** tzumainn has joined #tripleo | 12:21 | |
| *** mdnadeem has quit IRC | 12:22 | |
| *** medberry has joined #tripleo | 12:23 | |
| *** medberry has quit IRC | 12:23 | |
| *** medberry has joined #tripleo | 12:23 | |
| EmilienM | panda|lunch, quiquell: re: https://review.openstack.org/#/c/587006/ | 12:27 |
| EmilienM | why did you we get a promotion of collectd container if it now fails in check/gate? | 12:27 |
| quiquell | EmilienM: Intermitent failure... | 12:27 |
| quiquell | EmilienM: the 27th the health check get merged without issues, now it's failing sometimes (legit though) | 12:28 |
| mwhahaha | EmilienM: I think the collectd failures are cloud specific | 12:28 |
| mwhahaha | i think we need to hold off the health check stuff until stein | 12:28 |
| EmilienM | yeah | 12:28 |
| *** thrash|g0ne is now known as thrash | 12:29 | |
| *** rlandy has joined #tripleo | 12:31 | |
| weshay | mwhahaha +1 | 12:32 |
| weshay | good idea, not stable enough | 12:32 |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** panda|lunch is now known as panda|rover | 12:32 | |
| *** jcoufal has joined #tripleo | 12:34 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-common master: Fix overwriting downloaded config files https://review.openstack.org/586499 | 12:34 |
| EmilienM | mwhahaha: https://bugs.launchpad.net/tripleo/+bug/1784068 is it something we should fix in rc1 you think? | 12:37 |
| openstack | Launchpad bug 1784068 in tripleo "undercloud role does not have timezone service" [Medium,Triaged] | 12:37 |
| mwhahaha | EmilienM: did we do TZ management in instack-undercloud? | 12:37 |
| mwhahaha | EmilienM: if not, i think it should get pushed | 12:38 |
| bogdando | EmilienM, mwhahaha, quiquell: intermittent failures point to faily containers, not the health check. I think it should be re-reverted in rocky | 12:39 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates stable/queens: Revert "Delete not-used services-docker files" https://review.openstack.org/586480 | 12:39 |
| bogdando | it helps to find legit failures | 12:39 |
| mwhahaha | bogdando: yea i get that but let's not block ci unless we figure those out | 12:40 |
| bogdando | sure | 12:40 |
| bogdando | let's please not post pone to stain, that's my point | 12:40 |
| chandankumar | bogdando: I am getting some error on that http://logs.openstack.org/68/584368/4/check/tripleo-ci-centos-7-undercloud-oooq/5a91499/job-output.txt.gz#_2018-07-30_11_29_02_344627 | 12:40 |
| quiquell | bogdando: We need this to fix gates, and also it's not exercising 001 scenario | 12:40 |
| bogdando | ack | 12:40 |
| chandankumar | bogdando: on the same patch https://review.openstack.org/#/c/584368/4/roles/validate-tempest/defaults/main.yml@27 | 12:40 |
| quiquell | bogdando: scenario001 have to be exercises at changes on the healthcheck parts, to ensure that we get it right in the re-revert | 12:40 |
| bogdando | okay , that works | 12:41 |
| *** gbarros has quit IRC | 12:41 | |
| bogdando | chandankumar: so LGTM then, that was me failed to read codesearch results | 12:41 |
| *** mjturek has joined #tripleo | 12:42 | |
| chandankumar | bogdando: How to fix that issue? | 12:42 |
| EmilienM | mwhahaha: we didn't configure timezone in "instack-undercloud", so +1 for stein now, and maybe backportable if needed. | 12:42 |
| bogdando | chandankumar: if undercloud_enable_tempest is defined and undercloud_enable_tempest|bool | 12:42 |
| mwhahaha | EmilienM: yea since we don't expose it via undercloud.conf we probably should just leave it up to the end user to configure it | 12:43 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-upgrade stable/queens: DNM: Test pike->queens upgrades. https://review.openstack.org/563616 | 12:43 |
| openstackgerrit | Bogdan Dobrelya proposed openstack/tripleo-heat-templates stable/queens: Revert "Move remnants of environments/services-docker" https://review.openstack.org/586484 | 12:43 |
| *** Haresh has quit IRC | 12:44 | |
| *** links has quit IRC | 12:44 | |
| *** zoli|doctor is now known as zoli | 12:45 | |
| *** zoli is now known as zoli|wfh | 12:46 | |
| *** zoli|wfh is now known as zoli | 12:46 | |
| *** mcornea has joined #tripleo | 12:46 | |
| *** agopi has joined #tripleo | 12:49 | |
| *** mugsie has quit IRC | 12:49 | |
| *** mugsie has joined #tripleo | 12:49 | |
| *** mugsie has quit IRC | 12:49 | |
| *** mugsie has joined #tripleo | 12:49 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: Exercise scenario001 with changes at common https://review.openstack.org/587051 | 12:50 |
| *** amoralej is now known as amoralej|lunch | 12:51 | |
| *** mjturek has quit IRC | 12:51 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container https://review.openstack.org/584368 | 12:53 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Remove single quote from docker inspect command in tempest run https://review.openstack.org/586213 | 12:53 |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: Exercise scenarios with changes at common https://review.openstack.org/587051 | 12:54 |
| *** edmondsw has joined #tripleo | 12:56 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-heat-templates master: [DNM] To test coverage of common https://review.openstack.org/587052 | 12:56 |
| *** links has joined #tripleo | 12:56 | |
| *** amoralej|lunch is now known as amoralej | 12:57 | |
| *** pradk has quit IRC | 12:58 | |
| quiquell | owalsh: To add more CI to changes on tht common/ https://review.openstack.org/#/c/587051/ | 12:58 |
| owalsh | quiquell: ack | 12:59 |
| quiquell | owalsh: Don't know if it's too much | 13:00 |
| owalsh | quiquell: don't think so, stuff in common dir affects pretty much everything | 13:01 |
| *** ksambor has quit IRC | 13:01 | |
| EmilienM | mwhahaha: https://review.openstack.org/#/c/587051 ^ I think that's what we said on Friday we should do, right? | 13:01 |
| mwhahaha | EmilienM: yea | 13:02 |
| quiquell | There are some issues at RDO trunk repo, downloading .repo files... | 13:05 |
| quiquell | Jiri's docker_config thing is failing there http://logs.openstack.org/99/586499/2/check/tripleo-ci-centos-7-undercloud-oooq/01e8bc6/job-output.txt.gz#_2018-07-30_12_45_20_884295 | 13:05 |
| *** ansmith has joined #tripleo | 13:06 | |
| *** edmondsw has quit IRC | 13:06 | |
| sshnaidm|afk | quiquell, we had a lot of such recently | 13:06 |
| *** rbrady has joined #tripleo | 13:06 | |
| *** rbrady has joined #tripleo | 13:06 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:06 | |
| mwhahaha | so it was proposed that we should actually be using the mirror for that fetch | 13:07 |
| sshnaidm | quiquell, most likely it's related to rdo cloud outage.. | 13:07 |
| mwhahaha | that might help | 13:07 |
| sshnaidm | yeah, pabelanger liked this idea | 13:07 |
| *** janki has quit IRC | 13:08 | |
| *** janki has joined #tripleo | 13:08 | |
| *** bfournie has quit IRC | 13:09 | |
| *** jroll has quit IRC | 13:10 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 13:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 13:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 13:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 13:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 13:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 13:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 13:10 |
| *** myoung has joined #tripleo | 13:10 | |
| *** agurenko has quit IRC | 13:10 | |
| *** jroll has joined #tripleo | 13:10 | |
| *** bfournie has joined #tripleo | 13:11 | |
| *** agurenko has joined #tripleo | 13:12 | |
| *** ksambor has joined #tripleo | 13:14 | |
| *** bnemec has joined #tripleo | 13:20 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-common master: Fix overwriting downloaded config files https://review.openstack.org/586499 | 13:20 |
| ccamacho | hey folks can you vote here, please? https://review.openstack.org/#/c/586184/ | 13:21 |
| *** toure|gone is now known as toure | 13:23 | |
| *** mcornea has quit IRC | 13:23 | |
| *** mcornea has joined #tripleo | 13:24 | |
| *** agopi is now known as agopi|brb | 13:25 | |
| openstackgerrit | Ryan Brady proposed openstack/tripleo-common master: Create get_flattened_parameters workflow https://review.openstack.org/558883 | 13:25 |
| quiquell | sshnaidm: ack | 13:26 |
| quiquell | sshnaidm: Maybe we need some retries at the Get DLRN hash, and also emit_releases | 13:26 |
| quiquell | sshnaidm: As a improve | 13:26 |
| sshnaidm | quiquell, doesn't emit script have retries? | 13:27 |
| *** eck`gone is now known as eck` | 13:27 | |
| quiquell | sshnaidm: yep 10, but maybe it's not enough, maybe more time between retries | 13:27 |
| *** rpioso|afk is now known as rpioso | 13:29 | |
| sshnaidm | quiquell, maybe, not sure it will help much. Worth to check using mirror for that, but it also can have problems, like not fast update after repo is promoted, for example | 13:30 |
| *** agopi|brb has quit IRC | 13:30 | |
| *** udesale has joined #tripleo | 13:30 | |
| quiquell | sshnaidm: Ok let's wait for stuff to settle down | 13:30 |
| EmilienM | sdoran: good morning, Tengu is working on https://review.openstack.org/#/c/586538 and we wanted to get your feedback on this patch. Do you think it's the good place for the function? I'm thinking about ansible_runner for the future, and I was thinking we could have had this function in tripleo-common, for the transition to ansible_runner in the future | 13:31 |
| *** psahoo has quit IRC | 13:31 | |
| * sdoran looking | 13:32 | |
| * Tengu runs and hides | 13:32 | |
| *** lblanchard has joined #tripleo | 13:32 | |
| *** artom has joined #tripleo | 13:33 | |
| *** marrusl has joined #tripleo | 13:34 | |
| *** udesale has quit IRC | 13:35 | |
| *** skramaja has quit IRC | 13:36 | |
| Tengu | sdoran: did the patch kill you with its ugliness? :| | 13:38 |
| *** Goneri has joined #tripleo | 13:38 | |
| bogdando | basically, it makes sense to me to move it to tripleo_common ansible.py actions | 13:40 |
| bogdando | client imports it for ansible.cfg generation | 13:41 |
| bfournie | Hi EmilienM, when you have a chance, this backport could use a look - https://review.openstack.org/#/c/583985/ | 13:41 |
| bogdando | so if we'd want import ansible runner in the commons, to not create cross-dependencies | 13:41 |
| *** mjturek has joined #tripleo | 13:42 | |
| Tengu | bogdando: fine for me. That said, the run_ansible_playbook has a dep on the run_command_and_log from tripleocli :D | 13:42 |
| * bogdando sigh | 13:42 | |
| EmilienM | bfournie: ack | 13:42 |
| Tengu | but that can be changed with some other calls, why not from oslo libs | 13:42 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Create a Timesync service declaration https://review.openstack.org/586679 | 13:43 |
| bogdando | jfrancoa: https://review.openstack.org/#/c/465047/ passed | 13:46 |
| openstackgerrit | John Eckersberg proposed openstack/tripleo-heat-templates master: Switch back to client-local queue masters https://review.openstack.org/587064 | 13:46 |
| bogdando | so did https://review.openstack.org/#/c/583515/ !! \w/ | 13:46 |
| bogdando | jfrancoa, sshnaidm, marios: ^^ | 13:46 |
| bfournie | EmilienM: thanks! | 13:47 |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: HTTP test during update/upgrade/ffwd. https://review.openstack.org/586030 | 13:51 |
| *** agopi has joined #tripleo | 13:54 | |
| ccamacho | please vote here https://doodle.com/poll/dzqyg93czqeitbcu | 13:54 |
| ccamacho | nooooo | 13:54 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/pike: Improve nova statedir ownership logic https://review.openstack.org/587066 | 13:54 |
| ccamacho | dont vote :P wrong chat... | 13:54 |
| *** ratailor has quit IRC | 13:55 | |
| * mwhahaha votes for the most inconvenient time | 13:55 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: Open ports 80 and 443 for WEB test. https://review.openstack.org/587072 | 14:05 |
| *** ukalifon has quit IRC | 14:06 | |
| *** psachin`` has joined #tripleo | 14:06 | |
| *** psachin` has quit IRC | 14:06 | |
| *** pradk has joined #tripleo | 14:08 | |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 14:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 14:10 |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 14:10 |
| sdoran | Tengu: EmilienM If the goal is a placeholder in tripleoclient to later insert Ansible Runner, I think this is ok. But we should avoid letting this grow to enumerating every Ansible command line option in favor of building a Runner input directory. | 14:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Quique Llorente (quiquell) | 14:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 14:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 14:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 14:10 |
| *** waleedm has quit IRC | 14:10 | |
| sdoran | Runner kinda punts on this by asking for all the command line options as a single string in `env/cmdline`. There's no validation of the flags currently, but that's something we could add to our function. | 14:11 |
| *** mcornea has quit IRC | 14:15 | |
| *** mcornea has joined #tripleo | 14:15 | |
| sdoran | I'm also curious about the goal of this abstraction. Is it just to provide a way to run Ansible via the `tripleo` command line? | 14:16 |
| sdoran | I think there's value in that, just curious if that's the intention. Ansible commands can get a bit long and complex, which is why Job Templates in Tower exist. | 14:17 |
| *** quiquell has quit IRC | 14:17 | |
| *** ksambor has quit IRC | 14:18 | |
| sdoran | We could provide a streamlined way of executing Ansible playbooks for a defined set of scenarios, essentially making it easy to run TripleO Ansible content. | 14:18 |
| *** Mantorok has quit IRC | 14:19 | |
| Tengu | sdoran: so we might keep it in the python-tripleoclient? | 14:19 |
| *** ksambor has joined #tripleo | 14:20 | |
| *** dmacpher has quit IRC | 14:21 | |
| Tengu | sdoran: maybe we can jump in a BJ session? Would be faster I guess. | 14:21 |
| EmilienM | sdoran: it's not a pure abstraction. It's really for the containerized undercloud now, where Ansible is currently called by a subprocess ansible-playbook - we try to create a function based on that | 14:21 |
| *** agurenko has quit IRC | 14:23 | |
| EmilienM | Tengu, sdoran: I probably wanted something that would be used by overcloud/undercloud/standalone at the same time. Maybe re-use what is in tripleo-common | 14:23 |
| EmilienM | and make it in one single place so when we transition to ansible-runner it's done once. | 14:24 |
| Tengu | I don't have anything against that. It's a fair point of view. | 14:24 |
| Tengu | if we need a way in a wider tripleo env to call a simple "ansible-playbook" outside of the tripleoclient, let's push the run_ansible_playbook in tripleo-common. | 14:25 |
| *** wolverineav has quit IRC | 14:26 | |
| Tengu | and I can replace the utils.run_command_and_log by some oslo_concurrency.processutils.execute() I guess. | 14:26 |
| Tengu | until we get ansible-runner :9. | 14:26 |
| *** wolverineav has joined #tripleo | 14:26 | |
| Tengu | sdoran, EmilienM -^ sounds like a plan? | 14:27 |
| pabelanger | keep in mind, there is license concerns about importing ansible (gpl) into openstack, (apache2) | 14:27 |
| pabelanger | and mostly why people use shell command | 14:27 |
| *** links has quit IRC | 14:27 | |
| Tengu | pabelanger: hm, it's not "importing" ansible, just calling its executable - for now at least. That's not in conflict I guess? | 14:28 |
| pabelanger | Tengu: yah, that's mostly how we do it in zuul today, subprocess to ansible-playbook | 14:28 |
| pabelanger | so, that should be fine | 14:28 |
| Tengu | pabelanger: regarding ansible-runner, nothing has been done for now, and we (at least I) didn't even checked how to integrate it | 14:28 |
| Tengu | pabelanger: good :). | 14:28 |
| EmilienM | Tengu: i'm fine with having the function in tripleoclient | 14:28 |
| Tengu | EmilienM: oh, ok. so nothing to change then? | 14:29 |
| pabelanger | apparently ansible-runner is apache2 | 14:29 |
| pabelanger | TIL | 14:29 |
| EmilienM | Tengu: I guess not, but I wanted sdoran's feedback on it | 14:29 |
| *** ramishra has quit IRC | 14:29 | |
| *** karthiks has quit IRC | 14:29 | |
| Tengu | EmilienM: fine for me :). So we can just wait for some more review and make it to Master then. | 14:29 |
| Tengu | pabelanger: apparently yep, Apache2, and it's not importing ansible either, just a kind of "glue" for the non-stable API interface of ansible. | 14:30 |
| *** ramishra has joined #tripleo | 14:30 | |
| sdoran | I would not be surprised if the Apache 2.0 license was used to make it easier to use in OpenStack. | 14:30 |
| Tengu | pabelanger: so we'll check on that later. There are a couple of "issues", one being: no package available for rhel/centos platforms :] | 14:30 |
| sdoran | Being able to import it as a Python library was done for OpenStack/at the request of someone in OpenStack land. :) | 14:31 |
| *** karthiks has joined #tripleo | 14:31 | |
| Tengu | btw, anyone cares to add a w+1 on that one? https://review.openstack.org/#/c/586518/ not sure about Gate status, but apparently things are going through. Eventually. | 14:31 |
| Tengu | sdoran: \o/ | 14:31 |
| Tengu | bogdando: you're also OK with the run_ansible_playbook staying in tripleoclient? | 14:32 |
| Tengu | oh, and, also - anyone having a better way to mock the subprocess.Popen in there? https://review.openstack.org/#/c/586538/6/tripleoclient/tests/test_utils.py@65 - I'm a bit stuck on that, and not really happy with the current """solution""" I found.... | 14:34 |
| *** udesale has joined #tripleo | 14:34 | |
| *** pblaho has quit IRC | 14:34 | |
| safchain | Hi, I'm facing an issue with the Skydive deployment as external deploy task, step 5, keystone doesn't seem to be available at this step. I get 401 http code. Something changed ? Keystone API was available at this step few weeks ago. | 14:36 |
| *** pblaho has joined #tripleo | 14:37 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: Add Browbeat env settings to rdocloud https://review.openstack.org/583576 | 14:37 |
| EmilienM | safchain: keystone should be available at step3 the last time I checked | 14:37 |
| EmilienM | let me see again | 14:37 |
| *** corvus has joined #tripleo | 14:38 | |
| EmilienM | yeah step3, all endpoints and admin role is created | 14:38 |
| *** dtantsur is now known as dtantsur|brb | 14:38 | |
| *** shreshtha has joined #tripleo | 14:38 | |
| *** dparkes has quit IRC | 14:40 | |
| *** ykarel is now known as ykarel|away | 14:41 | |
| bogdando | Tengu: yeah | 14:41 |
| safchain | EmilienM, ok but if I source the overcloudrc file I get 401 error, weird | 14:41 |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: nova: add parameters to configure nova::cron::archive_deleted_rows https://review.openstack.org/584998 | 14:44 |
| Tengu | bogdando: perfect then :). | 14:44 |
| Tengu | thanks EmilienM for the w :). | 14:44 |
| *** bogdando has quit IRC | 14:45 | |
| openstackgerrit | Ricardo Noriega proposed openstack/tripleo-heat-templates master: Sharing BGPVPN Heat plugin volume https://review.openstack.org/585320 | 14:46 |
| *** cdearborn has joined #tripleo | 14:46 | |
| jfrancoa | bogdando: that's great! however, I am checking the failing legacy-tripleo-ci-centos-7-multinode-1ctlr-featureset037-updates-master RDO job and it seems to be failing still as we don't have a Depends-On https://review.openstack.org/#/c/586499/3 . Shouldn't we include it in https://review.openstack.org/#/c/583515/ ? | 14:46 |
| *** slaweq has quit IRC | 14:48 | |
| *** stendulker has joined #tripleo | 14:50 | |
| *** ykarel|away has quit IRC | 14:51 | |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix ssl warnings on tempest containerized https://review.openstack.org/580384 | 14:56 |
| *** medberry has quit IRC | 14:57 | |
| openstackgerrit | Raoul Scarazzini proposed openstack/tripleo-ha-utils master: Apply/Delete latency during HA validation https://review.openstack.org/585707 | 14:58 |
| openstackgerrit | Arx Cruz proposed openstack/tripleo-quickstart-extras master: Fix ssl warnings on tempest containerized https://review.openstack.org/580384 | 14:58 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-heat-templates stable/queens: Upgrades: Refactor playbooks to set facts https://review.openstack.org/576520 | 14:58 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-heat-templates stable/queens: Upgrades: Refactor package removal to step3 https://review.openstack.org/576521 | 14:58 |
| *** slaweq has joined #tripleo | 15:01 | |
| *** janki has quit IRC | 15:03 | |
| *** pblaho has quit IRC | 15:05 | |
| *** ksambor has quit IRC | 15:10 | |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 15:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 15:10 |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 15:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiri Stransky (jistran) | 15:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 15:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 15:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 15:10 |
| *** leanderthal has quit IRC | 15:10 | |
| *** pcaruana has quit IRC | 15:16 | |
| mwhahaha | so many alerts | 15:18 |
| panda|rover | mwhahaha: IIUC https://bugs.launchpad.net/tripleo/+bug/1784015 is solved, not sure why the bug is not closed | 15:20 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 15:20 |
| mwhahaha | panda|rover: because i used Related-Bug instead of Closes-Bug | 15:20 |
| mwhahaha | panda|rover: actually cause https://review.openstack.org/#/c/586591/ never landed | 15:20 |
| panda|rover | mwhahaha: ok | 15:20 |
| mwhahaha | panda|rover: did we get a promotion? that's the only way that gets resolved i think | 15:21 |
| *** atoth has quit IRC | 15:21 | |
| panda|rover | mwhahaha: not yet, we need this https://review.openstack.org/587006 to merge firstr | 15:21 |
| *** Haresh has joined #tripleo | 15:21 | |
| *** atoth has joined #tripleo | 15:22 | |
| panda|rover | mwhahaha: also another kolla patch to fails in promotion | 15:22 |
| *** agurenko has joined #tripleo | 15:23 | |
| mwhahaha | k | 15:23 |
| *** agurenko has quit IRC | 15:23 | |
| *** rwsu has joined #tripleo | 15:25 | |
| *** janki has joined #tripleo | 15:27 | |
| *** avivgt has quit IRC | 15:28 | |
| *** jchhatbar has joined #tripleo | 15:29 | |
| *** janki has quit IRC | 15:30 | |
| *** jchhatbar has quit IRC | 15:30 | |
| *** jchhatbar has joined #tripleo | 15:31 | |
| *** rwsu has quit IRC | 15:34 | |
| *** lvdombrkr89 has quit IRC | 15:37 | |
| *** dtantsur|brb is now known as dtantsur | 15:37 | |
| *** yprokule has quit IRC | 15:37 | |
| *** myoung has quit IRC | 15:44 | |
| *** dparkes has joined #tripleo | 15:45 | |
| *** Haresh has quit IRC | 15:46 | |
| *** rwsu has joined #tripleo | 15:47 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/pike: Improve nova statedir ownership logic https://review.openstack.org/587066 | 15:48 |
| *** cshastri has quit IRC | 15:48 | |
| *** tesseract has quit IRC | 15:49 | |
| *** ramishra has quit IRC | 15:51 | |
| *** jpich has quit IRC | 15:52 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates stable/pike: Improve nova statedir ownership logic https://review.openstack.org/587066 | 15:55 |
| *** psachin`` has quit IRC | 15:56 | |
| *** zoli is now known as zoli|gone | 15:57 | |
| *** zoli|gone is now known as zoli | 15:58 | |
| *** Mantorok has joined #tripleo | 15:58 | |
| *** Mantorok has quit IRC | 16:00 | |
| *** khyr0n has joined #tripleo | 16:01 | |
| *** Mantorok has joined #tripleo | 16:01 | |
| *** jfrancoa has quit IRC | 16:05 | |
| *** pradk has quit IRC | 16:06 | |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 16:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783540 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 16:10 |
| openstack | Launchpad bug 1783540 in tripleo "RDO cloud is not in operational state" [Critical,Triaged] - Assigned to chandan kumar (chkumar246) | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 16:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiri Stransky (jistran) | 16:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 16:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 16:10 |
| *** itlinux has joined #tripleo | 16:10 | |
| *** dparkes has quit IRC | 16:11 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Mount my.cnf.d into the db_sync container for Barbican and Octavia. https://review.openstack.org/586621 | 16:11 |
| openstackgerrit | Merged openstack/tripleo-common stable/queens: ensure unique ironic node ID with UCS driver https://review.openstack.org/583985 | 16:11 |
| openstackgerrit | Merged openstack/tripleo-upgrade master: Workload flavor customization. https://review.openstack.org/586961 | 16:11 |
| *** itlinux has quit IRC | 16:12 | |
| *** itlinux has joined #tripleo | 16:12 | |
| openstackgerrit | jacky06 proposed openstack/puppet-tripleo master: Update the links to https https://review.openstack.org/587102 | 16:14 |
| *** myoung has joined #tripleo | 16:14 | |
| *** Guest9714 is now known as melwitt | 16:15 | |
| *** pradk has joined #tripleo | 16:16 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: split and move the script creation https://review.openstack.org/587103 | 16:16 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: split and move the script creation https://review.openstack.org/587103 | 16:18 |
| *** itlinux_ has joined #tripleo | 16:19 | |
| sri_ | dsneddon, Hi quick questio what are the chances of configuring wrong ovs bonds will bomb the entire networking in a data cente(nexus switches goes 100% cpu usage)r | 16:20 |
| EmilienM | can we reduce the # of alerts? are they all needed here? | 16:20 |
| *** itlinux has quit IRC | 16:20 | |
| itlinux_ | hello all, I am running Pike, however, when I tried to move glance into cinder.. the vms were unable to spin up.. as soon as I moved it back they worked.. any tips on what else should I look for since the changes to have glance in cinder look like two lines.. Thanks | 16:21 |
| *** rlandy is now known as rlandy|brb | 16:21 | |
| *** stendulker has quit IRC | 16:25 | |
| *** ccamacho has quit IRC | 16:26 | |
| sri_ | itlinux_, Hi because you're here, quick question what are the chances of configuring wrong ovs bonds will bomb the entire networking in a data centre(nexus switches goes 100% cpu usage) | 16:26 |
| itlinux_ | if the bonds are not configured right there is no link.. in my experience you will not be able the bond up | 16:26 |
| itlinux_ | so there should not be any data going through | 16:27 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: split and move the script creation https://review.openstack.org/587103 | 16:27 |
| *** dparkes has joined #tripleo | 16:28 | |
| *** mdnadeem_ has quit IRC | 16:31 | |
| sri_ | itlinux_, yes that makes sense, couple of days ago networking went unresponsive in one of our data center ,my network admin saying it's because of I've configure the wrong bonds, he's saying he don't know what happed but it has to be me :) | 16:31 |
| itlinux_ | ahh so he just blames you! | 16:32 |
| itlinux_ | and he does not know what cause the problem!! | 16:32 |
| itlinux_ | nice! | 16:32 |
| itlinux_ | if the bonds are down there is not networking data going through | 16:33 |
| itlinux_ | you could ask the neutron channel but I am pretty sure that's the case. | 16:33 |
| sri_ | itlinux_, in fact hes blaming TripleO, LOL, he wanted to do manual deployment | 16:35 |
| *** panda|rover is now known as panda|rover|off | 16:36 | |
| itlinux_ | tell him he is nuts! | 16:36 |
| sri_ | itlinux_, anyway thanks a lot for your inputs | 16:36 |
| itlinux_ | simply ask him how long it will take him to deploy a 20 nodes .. with HA etc.. if he can do it in less than 90 min.. then I will consider it. :) | 16:37 |
| itlinux_ | no worries sri_: | 16:37 |
| sri_ | itlinux_, i don't have balls to say that :P | 16:37 |
| itlinux_ | you should ask .. | 16:38 |
| itlinux_ | and see how long it will take.. to deploy in HA, with pacemaker, corosync, etc.. | 16:38 |
| itlinux_ | I have done this many times and it takes less than 90 min now... on 23 nodes.. 20computes and 3 controllers | 16:39 |
| *** jfrancoa has joined #tripleo | 16:41 | |
| sri_ | itlinux_, yes, when we deploying manually there is a 100% chance of making mistakes(syntax, spelling mistakes) | 16:42 |
| *** udesale has quit IRC | 16:43 | |
| *** yamahata has quit IRC | 16:45 | |
| *** agurenko has joined #tripleo | 16:45 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Fix python3 support in yaml-validate script. https://review.openstack.org/586518 | 16:47 |
| *** jfrancoa has quit IRC | 16:48 | |
| *** agurenko has quit IRC | 16:52 | |
| weshay | rfolco|ruck, https://review.openstack.org/#/c/587006/ | 16:52 |
| mwhahaha | EmilienM: http://logs.openstack.org/91/582991/1/check/puppet-openstack-unit-4.8-centos-7/8e3a86a/job-output.txt.gz#_2018-07-30_14_19_29_491655 that's a new one, did we drop redis from the puppetfile in puppet-tripleo/p-o-i? | 16:53 |
| weshay | rfolco|ruck, https://review.openstack.org/#/c/586499/ | 16:54 |
| EmilienM | mwhahaha: not afik but let me see | 16:54 |
| mwhahaha | we did | 16:55 |
| itlinux_ | sri_: that's only one part.. how about upgrades etc.. | 16:55 |
| * mwhahaha remembers tobasco was messing with redis for bionic | 16:55 | |
| openstackgerrit | jacky06 proposed openstack/puppet-pacemaker master: Add the missing somke testing https://review.openstack.org/587154 | 16:56 |
| openstackgerrit | jacky06 proposed openstack/puppet-tripleo master: Add the missing somke testing https://review.openstack.org/587155 | 16:56 |
| mwhahaha | EmilienM: https://review.openstack.org/#/c/566465/11/Puppetfile | 16:56 |
| EmilienM | mwhahaha: ok so we need to re-add it to puppet-tripleo | 16:57 |
| mwhahaha | yea | 16:58 |
| mwhahaha | let me file a bug/fix | 16:58 |
| sri_ | itlinux_, yes, can you recommend the home lab setup with (one switch and router with 3 servers with 4 nics) is that is the enough | 16:59 |
| dsneddon | sri_, As long as you are configuring the OVS bridge to have one member, the bond, and that bond has multiple member interfaces, it should be alright. If however you tried to attach two interfaces to the same bridge, without a bond in between, then you could create an L2 loop. | 16:59 |
| *** rlandy|brb is now known as rlandy | 16:59 | |
| dsneddon | sri_, The best way to avoid problems is to use LACP on both switch and host side (balance-tcp mode for OVS bonds). | 17:00 |
| dsneddon | sri_, But you shouldn't have a loop on the network just because you misconfigured an OVS bond. | 17:00 |
| *** derekh has quit IRC | 17:01 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Pull in redis for unit tests https://review.openstack.org/587162 | 17:02 |
| mwhahaha | EmilienM: -^ | 17:02 |
| *** gfidente has quit IRC | 17:03 | |
| sri_ | dsneddon, it all very clear to me now, thanks for the explanation :) | 17:04 |
| dsneddon | sri_, I'd be happy to validate your NIC config templates if you would like. I'd need both the NIC configs and the network-environment.yaml (Or wherever you are storing things like the BondInterfaceOvsOptions | 17:04 |
| EmilienM | mwhahaha: thx | 17:04 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Enable HAProxy mode http for Swift and Ceph RGW https://review.openstack.org/582991 | 17:05 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-tripleo master: Prevent triggering firewall actions while configuring HA services https://review.openstack.org/583648 | 17:05 |
| sri_ | dsneddon, http://paste.openstack.org/show/726861/ | 17:06 |
| *** jpena is now known as jpena|off | 17:10 | |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 17:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784307 | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784422 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 17:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiri Stransky (jistran) | 17:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 17:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 17:10 |
| openstack | Launchpad bug 1784307 in tripleo "tripleomaster/centos-binary-collectd:current-tripleo-updated-20180730001257 \"kolla_start\" Restarting" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 17:10 |
| openstack | Launchpad bug 1784422 in tripleo "puppet-tripleo unit tests broken due to missing redis" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 17:10 |
| itlinux_ | sri_: 4 nics total or each server? | 17:13 |
| itlinux_ | do your switch support vlans? | 17:14 |
| sri_ | itlinux_, yes 4 nics for server, and I need buy switch | 17:16 |
| itlinux_ | use two for bond and one to pxe | 17:16 |
| itlinux_ | that's the min you would need to get it going | 17:16 |
| itlinux_ | use nic1 , nic2 etc in the templates.. depending on the type of bonds you may have to use linux bridge with ovs.. | 17:17 |
| *** yamahata has joined #tripleo | 17:18 | |
| sri_ | itlinux_, Ok, I need to look into all the options available in bonding, thanks :) | 17:20 |
| *** salmankhan has quit IRC | 17:23 | |
| *** amoralej is now known as amoralej|off | 17:26 | |
| *** shardy has quit IRC | 17:28 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Revert "Fix deploy health checks" https://review.openstack.org/587006 | 17:29 |
| weshay | EmilienM, mwhahaha https://review.openstack.org/#/c/586499/ I think is ready for workflow | 17:32 |
| EmilienM | roger that | 17:32 |
| weshay | the ovb failures are unrelated | 17:32 |
| EmilienM | we already approved it | 17:32 |
| mwhahaha | weshay: do try and keep up :D | 17:33 |
| weshay | lolz | 17:33 |
| weshay | good to be back | 17:33 |
| mwhahaha | for some definition of good | 17:34 |
| weshay | right | 17:34 |
| weshay | dam guys | 17:37 |
| *** ohochman has quit IRC | 17:38 | |
| *** zbitter has joined #tripleo | 17:40 | |
| *** zaneb has quit IRC | 17:42 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: Update way to determine deployment status in plans list https://review.openstack.org/583219 | 17:42 |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: List deployment failures on deployment failure https://review.openstack.org/578712 | 17:42 |
| *** ykarel|away has joined #tripleo | 17:45 | |
| *** jaganathan has quit IRC | 17:50 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/tripleo-quickstart master: Fix/enable the Tempest tests for Manila https://review.openstack.org/509554 | 18:00 |
| openstackgerrit | Honza Pokorny proposed openstack/tripleo-quickstart master: Add a flag to create the reproducer script https://review.openstack.org/586843 | 18:08 |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 18:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 18:10 |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784422 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 18:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiri Stransky (jistran) | 18:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 18:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 18:10 |
| openstack | Launchpad bug 1784422 in tripleo "puppet-tripleo unit tests broken due to missing redis" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 18:10 |
| *** myoung is now known as myoung|lunch | 18:22 | |
| *** akrivoka has quit IRC | 18:23 | |
| *** zbitter is now known as zaneb | 18:26 | |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/pike: Remove INSECURE_REGISTRY from docker_registry.pp https://review.openstack.org/587190 | 18:30 |
| openstackgerrit | Emilien Macchi proposed openstack/puppet-tripleo stable/pike: Remove INSECURE_REGISTRY from docker_registry.pp https://review.openstack.org/587190 | 18:31 |
| mwhahaha | EmilienM: oh did you find the missing patch | 18:31 |
| *** dtantsur is now known as dtantsur|afk | 18:31 | |
| openstackgerrit | Jiri Tomasek proposed openstack/tripleo-ui master: List deployment failures on deployment failure https://review.openstack.org/578712 | 18:31 |
| EmilienM | mwhahaha: yes | 18:31 |
| EmilienM | mwhahaha: and good news, no need to patch instack-undercloud I think | 18:31 |
| mwhahaha | always a bonus | 18:32 |
| *** sri__ has joined #tripleo | 18:42 | |
| openstackgerrit | Brad P. Crochet proposed openstack/tripleo-common master: Use keystone group for loading auth params https://review.openstack.org/585904 | 18:44 |
| *** holser_ has quit IRC | 18:45 | |
| *** dparkes has quit IRC | 19:02 | |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart-extras master: Use local docker registry host for tempest container https://review.openstack.org/584368 | 19:05 |
| openstackgerrit | yatin proposed openstack/tripleo-quickstart-extras master: Remove single quote from docker inspect command in tempest run https://review.openstack.org/586213 | 19:06 |
| *** salmankhan has joined #tripleo | 19:07 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/queens: Fix gnocchi auth mode to basic https://review.openstack.org/584968 | 19:08 |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783866 | 19:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784422 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 19:10 |
| openstack | Launchpad bug 1783866 in tripleo "fs037 updates: Failed to update nodes - Controller" [Critical,In progress] - Assigned to Jiri Stransky (jistran) | 19:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 19:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1784422 in tripleo "puppet-tripleo unit tests broken due to missing redis" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 19:10 |
| *** sri__ has quit IRC | 19:11 | |
| *** salmankhan has quit IRC | 19:12 | |
| *** holser_ has joined #tripleo | 19:15 | |
| *** holser_ has quit IRC | 19:29 | |
| *** holser_ has joined #tripleo | 19:29 | |
| *** ykarel|away has quit IRC | 19:30 | |
| openstackgerrit | Merged openstack/tripleo-common master: Fix overwriting downloaded config files https://review.openstack.org/586499 | 19:30 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-pacemaker master: Pin net-telnet to 0.1.1 https://review.openstack.org/587202 | 19:33 |
| *** morazi has quit IRC | 19:34 | |
| *** myoung|lunch is now known as myoung | 19:34 | |
| openstackgerrit | Sean McGinnis proposed openstack/ansible-role-container-registry master: Fix package README errors https://review.openstack.org/587205 | 19:36 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-pacemaker master: Initial support for adding cluster nodes to an existing cluster https://review.openstack.org/585150 | 19:44 |
| *** holser_ has quit IRC | 19:46 | |
| *** jtcressy has joined #tripleo | 19:47 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: NUMA aware vswitches https://review.openstack.org/578742 | 19:54 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Pull in redis for unit tests https://review.openstack.org/587162 | 19:54 |
| mwhahaha | jillr: where's the cookiecutter for the ansible role bits? | 20:00 |
| mwhahaha | nm i found it | 20:01 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/ocata: Define default route for Management network https://review.openstack.org/579961 | 20:02 |
| jtcressy | mwhahaha or anyone available: It seems i'm still having some issues with my overcloud. I'm able to schedule instances and network them to the internet and everything, but I've noticed that creating block storage containers does NOT work, I can't load the "Load Balancers" section of horizon dashboard, and I can't list/create cluster templates for container infra (at least not through horizon so far). | 20:04 |
| jtcressy | object storage containers* | 20:05 |
| mwhahaha | jtcressy: glance problems still? | 20:05 |
| mwhahaha | i don't think the load balancer bits work in horizon | 20:06 |
| jtcressy | Nope, images are working fine. glance is working fine. | 20:06 |
| mwhahaha | what do you mean creating block storage containers? | 20:07 |
| jtcressy | I meant object storage containers | 20:07 |
| jtcressy | In horizon, Project>Object Store>Containers>Create. Fails every time. | 20:07 |
| mwhahaha | did you deploy swift? | 20:08 |
| * mwhahaha doesn't use horizon so not sure what it's doing | 20:08 | |
| jtcressy | Not specifically... but I assume object storage is redirected to RGW since i'm using that instead. Or maybe swift is just an API proxy to that ? | 20:08 |
| mwhahaha | i guess it depends on how it's setup, you can use ceph rgw. but that might be broken | 20:09 |
| * mwhahaha once again defers to fultonj around rgw | 20:09 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 20:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 20:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 20:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 20:10 |
| *** waleedm has joined #tripleo | 20:15 | |
| *** nyechiel has quit IRC | 20:18 | |
| *** jchhatbar has quit IRC | 20:18 | |
| jtcressy | mwhahaha fultonj: Trying to create containers via CLI gives this error output: "Requested Range Not Satisfiable (HTTP 416)" | 20:18 |
| *** ade_lee has quit IRC | 20:20 | |
| *** AgnosticDBA has joined #tripleo | 20:21 | |
| fultonj | jtcressy: "ceph -s" | 20:21 |
| fultonj | is the ceph cluster in HEALTH_OK? | 20:21 |
| *** ade_lee has joined #tripleo | 20:21 | |
| fultonj | you'd run that from a controller node | 20:22 |
| fultonj | as root | 20:22 |
| AgnosticDBA | hi, new to TripleO, installing on a cloud server using quickstart, getting "Unauthorized Connection to Keystone API could not be established" on the tripleo-ui. | 20:25 |
| *** waleedm has quit IRC | 20:25 | |
| *** waleedm has joined #tripleo | 20:25 | |
| *** medberry has joined #tripleo | 20:26 | |
| *** medberry has quit IRC | 20:26 | |
| *** medberry has joined #tripleo | 20:26 | |
| AgnosticDBA | should have said, I would appreciate any advice | 20:26 |
| *** medberry has quit IRC | 20:27 | |
| *** medberry has joined #tripleo | 20:27 | |
| *** medberry has quit IRC | 20:27 | |
| *** medberry has joined #tripleo | 20:27 | |
| *** ade_lee has quit IRC | 20:27 | |
| mwhahaha | AgnosticDBA: which version? | 20:29 |
| mwhahaha | AgnosticDBA: there's issues with the way quickstart is configuring the tunnel for the UI. It's recommended to use sshuttle to connect directly to the provisioning network and use the UI that way | 20:30 |
| openstackgerrit | wes hayutin proposed openstack/puppet-tripleo stable/newton: remove scenario005 from experimental https://review.openstack.org/587215 | 20:30 |
| openstackgerrit | wes hayutin proposed openstack/puppet-tripleo stable/ocata: remove scenario005 from experimental https://review.openstack.org/587216 | 20:32 |
| *** medberry has quit IRC | 20:32 | |
| openstackgerrit | wes hayutin proposed openstack/puppet-tripleo stable/pike: remove scenario005 from experimental https://review.openstack.org/587217 | 20:33 |
| openstackgerrit | Alex Schultz proposed openstack/ansible-role-tripleo-cookiecutter master: Switch README to RST and add lint https://review.openstack.org/587218 | 20:33 |
| mwhahaha | jillr, sdoran, EmilienM -^ some role cookie cutter related fixes | 20:33 |
| AgnosticDBA | latest via git clone https://github.com/openstack/tripleo-quickstart | 20:33 |
| *** dparkes has joined #tripleo | 20:34 | |
| weshay | pabelanger, my patches to fix that zuul config error ^ https://review.openstack.org/587215 https://review.openstack.org/587217 | 20:34 |
| mwhahaha | AgnosticDBA: yea so the best bet is to use sshuttle to the undercloud host to expose proper networks for auth. It's related to containerization and how it was faking the configuration previously | 20:34 |
| EmilienM | mwhahaha: +2 | 20:34 |
| openstackgerrit | Emilien Macchi proposed openstack-infra/tripleo-ci master: Prepare Docker Registry + Containers in pre-run https://review.openstack.org/580037 | 20:35 |
| weshay | ya.. that is cool | 20:35 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: ansible: replace yum module by package module when possible https://review.openstack.org/584480 | 20:35 |
| openstackgerrit | Alex Schultz proposed openstack/ansible-role-tripleo-cookiecutter master: Switch README to RST and add lint https://review.openstack.org/587218 | 20:35 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: fs050: enable TLS https://review.openstack.org/585333 | 20:35 |
| openstackgerrit | Alex Schultz proposed openstack/ansible-role-tripleo-cookiecutter master: Switch README to RST and add lint https://review.openstack.org/587218 | 20:35 |
| jtcressy | fultonj: "ceph-s" shows HEALTH_OK | 20:37 |
| jtcressy | "rgw: 3 daemons active" | 20:37 |
| *** rpioso is now known as rpioso|afk | 20:38 | |
| *** ade_lee has joined #tripleo | 20:38 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-pacemaker master: Pin net-telnet to 0.1.1 https://review.openstack.org/587202 | 20:39 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-pacemaker master: Initial support for adding cluster nodes to an existing cluster https://review.openstack.org/585150 | 20:39 |
| AgnosticDBA | @mwhahaha tried that without success, will try again | 20:39 |
| mwhahaha | AgnosticDBA: tried what | 20:39 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-pacemaker master: Remove the pcmk_is_remote fact https://review.openstack.org/583482 | 20:39 |
| *** cdearborn has quit IRC | 20:40 | |
| *** ade_lee has quit IRC | 20:40 | |
| *** ade_lee has joined #tripleo | 20:41 | |
| fultonj | jtcressy: do you see anything similar to https://tracker.ceph.com/issues/21497 in your logs? | 20:41 |
| jtcressy | Where can I find the logs for this specifically? | 20:41 |
| *** akhilaki has joined #tripleo | 20:42 | |
| fultonj | an rgw container should be running on a controller | 20:42 |
| fultonj | docker ps | grep rgw --> ID | 20:42 |
| fultonj | docker logs $ID | 20:42 |
| *** ansmith has quit IRC | 20:43 | |
| AgnosticDBA | @mwhahaha "sshuttle to the undercloud host" https://docs.openstack.org/tripleo-quickstart/latest/accessing-overcloud.html#sshuttle | 20:43 |
| fultonj | command you used to create the container? | 20:43 |
| mwhahaha | AgnosticDBA: yea that except it's probably 192.168.24.0/24. Then you should be able to login to the UI using teh 192.168.24.x address | 20:44 |
| mwhahaha | AgnosticDBA: if it's still giving you unable to auth, you should double check via the cli that you can list the endpoints | 20:44 |
| jtcressy | I didnt create the container manually so I don't know. I found an rgw container on one of the controllers and watched for errors but cant see any when I tried to create an object storage container again via the UI. | 20:44 |
| jtcressy | fultonj: ^ | 20:45 |
| *** medberry has joined #tripleo | 20:45 | |
| *** medberry has quit IRC | 20:45 | |
| *** medberry has joined #tripleo | 20:45 | |
| *** ade_lee has quit IRC | 20:45 | |
| fultonj | i'm not sure horizon will be configured to use RGW out of the box | 20:46 |
| *** holser_ has joined #tripleo | 20:47 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Add support for containerized networking-ansible ML2 plugin https://review.openstack.org/585194 | 20:48 |
| jtcressy | fultonj: I see this in the rgw logs on one of the controllers: "PUT /swift/v1/test HTTP/1.1" | 20:48 |
| jtcressy | (I was creating an obj storage container named "test") | 20:49 |
| fultonj | jtcressy: try to docker exec -ti $ID /bin/bash | 20:50 |
| fultonj | into the container and have a look around | 20:50 |
| fultonj | i have to go | 20:50 |
| jtcressy | I'm not sure what to look for | 20:51 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Allow deferred undercloud push_destination detection https://review.openstack.org/585589 | 20:54 |
| EmilienM | weshay: can we reduce the # of alerts? things are merging now no? | 20:57 |
| *** khyr0n_ has joined #tripleo | 20:57 | |
| *** khyr0n has quit IRC | 20:58 | |
| *** hamzy_ is now known as hamzy | 20:59 | |
| *** khyr0n_ has quit IRC | 20:59 | |
| *** khyr0n has joined #tripleo | 20:59 | |
| weshay | EmilienM, yes.. I've been going through them, only one patch merged | 21:02 |
| weshay | I think another is about to | 21:02 |
| weshay | EmilienM, https://review.openstack.org/585528 is about to | 21:03 |
| *** khyr0n has quit IRC | 21:07 | |
| itlinux_ | hello guys, I am not sure what could be the cause of this issue.. as a admin I can create the container as a normal user I cannot | 21:07 |
| *** agopi is now known as agopi|brb | 21:07 | |
| AgnosticDBA | @mwhahaha where is the sshuttle run, on the undercloud VM or virthost? | 21:08 |
| mwhahaha | AgnosticDBA: your system | 21:08 |
| *** medberry has quit IRC | 21:09 | |
| mwhahaha | AgnosticDBA: from your system -> undercloud | 21:09 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783399 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783857 | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784015 | 21:10 |
| openstack | Launchpad bug 1783399 in tripleo "containerized undercloud upgrade jobs isn't testing upgrades anymore" [Critical,Triaged] - Assigned to Rafael Folco (rafaelfolco) | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 21:10 |
| openstack | Launchpad bug 1783857 in tripleo "TripleO CI jobs false positives" [Critical,In progress] - Assigned to Gabriele Cerami (gcerami) | 21:10 |
| openstack | Launchpad bug 1784015 in tripleo "ovb image build broken due to diskimage_builder.element_dependencies.MissingElementException: Element 'disable-nouveau' not found" [Critical,In progress] - Assigned to Alex Schultz (alex-schultz) | 21:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 21:10 |
| openstackgerrit | James Slagle proposed openstack/tripleo-common master: Add override_ansible_cfg https://review.openstack.org/584087 | 21:11 |
| *** agopi|brb has quit IRC | 21:12 | |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: DNM - Test work for using zuulV3 native in RDO Cloud https://review.openstack.org/587228 | 21:12 |
| openstackgerrit | Ronelle Landy proposed openstack-infra/tripleo-ci master: DNM - Test work for using zuulV3 native in RDO Cloud https://review.openstack.org/587228 | 21:13 |
| AgnosticDBA | @mwhahaha some more info, my browser is on my laptop, TripleO installed on cloud server - on the browser I can reach the TripleO homepage, it's the login to keystone that fails | 21:14 |
| mwhahaha | AgnosticDBA: i'm aware, the issue is that the way the UI works, there's a config that uses specific IPs. So it assumes your browser is sitting on the same network | 21:15 |
| mwhahaha | AgnosticDBA: so you need to sshuttle to expose the expected network to your laptop so auth works | 21:15 |
| mwhahaha | AgnosticDBA: the UI is just javascript which is attempting to access the net configured in the undercloud. It's likely 192.168.24.x which is not fully available externally when using quickstart. | 21:15 |
| mwhahaha | AgnosticDBA: quickstart used to work around this, however with the latest version of tripleo this functionality no longer works | 21:16 |
| *** itlinux_ has quit IRC | 21:16 | |
| *** itlinux has joined #tripleo | 21:17 | |
| *** waleedm has quit IRC | 21:18 | |
| AgnosticDBA | @mwhahaha so the sshuttle needs to be from my laptop to 192.168.24.x via my cloud server IP address | 21:19 |
| mwhahaha | AgnosticDBA: yes that's likely the solution | 21:20 |
| mwhahaha | AgnosticDBA: if the cloud server ip == the undercloud | 21:20 |
| *** florianf has quit IRC | 21:21 | |
| *** lblanchard has quit IRC | 21:22 | |
| AgnosticDBA | @mwhahaha that's the problem, I need an extra hop, as the cloud server ip is baremetal where everything TripleO runs | 21:23 |
| mwhahaha | AgnosticDBA: you can use ssh proxy to address the extra hop | 21:23 |
| mwhahaha | AgnosticDBA: https://serverfault.com/questions/826585/chaining-sshuttle-commands-over-two-hops | 21:24 |
| * mwhahaha had this discussion with jrist on friday | 21:24 | |
| *** pradk has quit IRC | 21:24 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Exercise scenarios with changes at common https://review.openstack.org/587051 | 21:25 |
| *** dsneddon has quit IRC | 21:29 | |
| *** dsneddon has joined #tripleo | 21:31 | |
| jrist | haha | 21:33 |
| jrist | yup | 21:33 |
| jrist | I can maybe help too AgnosticDBA | 21:33 |
| *** rfolco|ruck is now known as rfolco|off | 21:34 | |
| *** ade_lee has joined #tripleo | 21:35 | |
| *** Goneri has quit IRC | 21:36 | |
| openstackgerrit | wes hayutin proposed openstack-infra/tripleo-ci master: Remove tripleo.sh --bootstrap-subnodes from toci_gate_test.sh https://review.openstack.org/587012 | 21:37 |
| itlinux | hello all, can someone give me some ideas on why my tenant cannot create a container within swift, and as a admin I can.. here is the logs http://paste.openstack.org/show/726886/ | 21:44 |
| itlinux | much appreciated. | 21:45 |
| AgnosticDBA | @jrist I'm following the page @mwhahaha shared, setting up the config file, do you have it setup already? | 21:45 |
| *** bnemec has quit IRC | 21:46 | |
| openstackgerrit | Alex Schultz proposed openstack/ansible-role-tripleo-cookiecutter master: Switch README to RST and add lint https://review.openstack.org/587218 | 21:49 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-heat-templates master: Add block to step_0 https://review.openstack.org/587235 | 21:50 |
| openstackgerrit | Alex Schultz proposed openstack/ansible-role-tripleo-cookiecutter master: Switch README to RST and add lint https://review.openstack.org/587218 | 21:50 |
| jrist | nope, I'm using a local tripleo-ui setup and the one on the undercloud is tunneled | 21:52 |
| openstackgerrit | Sergii Golovatiuk proposed openstack/tripleo-heat-templates master: Add block to step_0 Add block to step_6 for neutron-api.yaml Add block to step_1 for nova-compute.yaml https://review.openstack.org/587235 | 21:58 |
| *** itlinux has quit IRC | 21:59 | |
| *** myoung has quit IRC | 21:59 | |
| *** holser_ has quit IRC | 22:00 | |
| *** paramite has quit IRC | 22:01 | |
| *** rcernin has joined #tripleo | 22:02 | |
| *** tzumainn has quit IRC | 22:04 | |
| *** ansmith has joined #tripleo | 22:04 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 22:10 |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 22:10 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Action to perform container image prepare https://review.openstack.org/585913 | 22:16 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Add container_image_prepare_params to deployment workflow https://review.openstack.org/586418 | 22:16 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: Create script tripleo-container-image-prepare https://review.openstack.org/587239 | 22:16 |
| *** pmannidi has joined #tripleo | 22:17 | |
| *** threestrands has joined #tripleo | 22:18 | |
| *** threestrands has quit IRC | 22:18 | |
| *** threestrands has joined #tripleo | 22:18 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/tripleo-quickstart master: Fix/enable the Tempest tests for Manila https://review.openstack.org/509554 | 22:18 |
| openstackgerrit | Merged openstack/tripleo-common master: kolla overrides: Remove yum cache https://review.openstack.org/586305 | 22:23 |
| openstackgerrit | Merged openstack/puppet-tripleo master: Remove setting root user for ovn dbs pacemaker bundle https://review.openstack.org/582500 | 22:23 |
| *** wolverineav has quit IRC | 22:25 | |
| *** wolverineav has joined #tripleo | 22:26 | |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: [Ocata,Pike,Queens-Only] Fix Cinder's Netapp backend https://review.openstack.org/583734 | 22:30 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/queens: Remove notification_driver parameter from heat profile https://review.openstack.org/585129 | 22:30 |
| openstackgerrit | Merged openstack/puppet-tripleo stable/ocata: Remove notification_driver parameter from heat profile https://review.openstack.org/585620 | 22:30 |
| *** mcornea has quit IRC | 22:33 | |
| *** wolverineav has quit IRC | 22:33 | |
| openstackgerrit | Merged openstack-infra/tripleo-ci master: Fix TAG and playbooks logic for all upgrade type jobs https://review.openstack.org/585528 | 22:35 |
| *** mcornea has joined #tripleo | 22:37 | |
| *** jcoufal has quit IRC | 22:40 | |
| *** tcw has quit IRC | 22:42 | |
| *** tcw has joined #tripleo | 22:42 | |
| *** thrash is now known as thrash|g0ne | 22:43 | |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: make reproducer bash syntax more portable https://review.openstack.org/581012 | 22:45 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Break out image prepare into its own "service" https://review.openstack.org/581918 | 22:45 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: WIP Always enable prepare service for docker clouds https://review.openstack.org/581919 | 22:45 |
| jtcressy | fultonj, mwhahaha: So I tried to scale up my cluster, and during the stack update it runs through the ceph-ansible playbook. I verified that the ceph cluster is still HEALTH_OK and no data loss has occurred, but on one of my nodes I got this error during the ceph-ansible playbook: | 22:45 |
| jtcressy | Error: Partition(s) 1, 2 on /dev/sdc have been written, but we have been unable to inform the kernel of the change, probably because it/they are in use. As a result, the old partition(s) will remain in use. You should reboot now before making further changes. | 22:45 |
| jtcressy | fultonj, mwhahaha: It seems it's updating the partition info on one of the OSD's, but it has trouble updating the kernel's partition information. This seems to be a trivial error, however it stops the ceph-ansible playbook in its tracks, which in turn stops the heat stack update, and my entire deployment is halted. The only way I've gotten around this in the past was to completely wipe and start over, but that's unacceptable. | 22:47 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: Break out image prepare into its own "service" https://review.openstack.org/581918 | 22:52 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-heat-templates master: WIP Always enable prepare service for docker clouds https://review.openstack.org/581919 | 22:52 |
| *** mcornea has quit IRC | 23:02 | |
| *** mschuppert has quit IRC | 23:04 | |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1783762 | 23:10 |
| openstack | Launchpad bug 1783762 in tripleo "Containerized Ironic BM (ILO) provisioning issue: DriverLoadError: Driver, hardware type or interface ilo-pxe could not be loaded. Reason: [Errno 13] Permission denied: '/var/lib/ironic/httpboot/" [Critical,Triaged] | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1784078 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1784078 in tripleo "scenario jobs in stable/queens are broken with ansible file issues: could not locate file in lookup: Controller/step_config.pp" [Critical,Triaged] | 23:10 |
| *** sshnaidm is now known as sshnaidm|afk | 23:23 | |
| *** akhilaki has quit IRC | 23:36 | |
| *** AgnosticDBA has quit IRC | 23:43 | |
| openstackgerrit | Goutham Pacha Ravi proposed openstack/paunch master: Switch docs to openstackdocstheme https://review.openstack.org/587256 | 23:58 |
| *** etingof has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!