| openstackgerrit | RedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras https://review.openstack.org/560445 | 00:00 |
|---|---|---|
| *** ooolpbot has joined #tripleo | 00:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 00:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 00:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 00:10 |
| *** ooolpbot has quit IRC | 00:10 | |
| *** agopi|brb has joined #tripleo | 00:14 | |
| *** sanjayu__ has quit IRC | 00:31 | |
| *** phuongnh has joined #tripleo | 01:03 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Modify the zuul inventory to pass to the reproducer https://review.openstack.org/617401 | 01:03 |
| *** rlandy is now known as rlandy|bbl | 01:06 | |
| *** tzumainn has quit IRC | 01:09 | |
| *** ooolpbot has joined #tripleo | 01:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 01:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 01:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 01:10 |
| *** ooolpbot has quit IRC | 01:10 | |
| *** mjturek has quit IRC | 01:33 | |
| *** huynq has joined #tripleo | 01:43 | |
| *** huynq has left #tripleo | 01:44 | |
| *** thrash is now known as thrash|g0ne | 01:56 | |
| *** ooolpbot has joined #tripleo | 02:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 02:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 02:10 |
| *** ooolpbot has quit IRC | 02:10 | |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 02:10 |
| weshay | bah.. everything is getting f'd again | 02:21 |
| weshay | FYI https://bugs.launchpad.net/tripleo/+bug/1803024 | 02:32 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 02:32 |
| *** mrsoul has quit IRC | 02:41 | |
| *** mschuppert has quit IRC | 02:41 | |
| *** psachin has joined #tripleo | 03:00 | |
| *** ooolpbot has joined #tripleo | 03:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 03:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 03:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 03:10 |
| *** ooolpbot has quit IRC | 03:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 03:10 |
| *** apetrich has quit IRC | 03:15 | |
| *** eck` is now known as eck`gone | 03:28 | |
| *** jaganathan_ has joined #tripleo | 03:39 | |
| mwhahaha | EmilienM, weshay: Bug 1803024 is likely because of puppet 5.5.6. iptables on the undercloud is missing all of the rules | 03:41 |
| openstack | bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] https://launchpad.net/bugs/1803024 | 03:41 |
| *** mwhahaha changes topic to "CI Status: master is RED | https://docs.openstack.org/tripleo-docs/latest/" | 03:42 | |
| *** udesale has joined #tripleo | 03:43 | |
| *** pdeore has joined #tripleo | 03:50 | |
| *** ykarel has joined #tripleo | 03:53 | |
| *** redrobot has quit IRC | 03:58 | |
| *** ramishra has joined #tripleo | 04:02 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Convert dynamic lookups to use colon notation https://review.openstack.org/617441 | 04:07 |
| mwhahaha | Tengu: when you wake up can you check -^ as I think the dot notation is now completely broken in puppet5 | 04:08 |
| *** ooolpbot has joined #tripleo | 04:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 04:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 04:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 04:10 |
| *** ooolpbot has quit IRC | 04:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 04:10 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-heat-templates master: Convert dynamic lookups to use colon notation https://review.openstack.org/617441 | 04:21 |
| *** rlandy|bbl is now known as rlandy | 04:23 | |
| *** rlandy has quit IRC | 04:23 | |
| openstackgerrit | Steve Baker proposed openstack/ansible-role-tripleo-modify-image master: Set WORKDIR / in the dockerfiles https://review.openstack.org/617442 | 04:27 |
| openstackgerrit | Steve Baker proposed openstack/ansible-role-tripleo-modify-image master: Use a tempfile for the modified Dockerfile https://review.openstack.org/617443 | 04:27 |
| *** owalsh_ has joined #tripleo | 04:35 | |
| *** owalsh has quit IRC | 04:39 | |
| *** pdeore has quit IRC | 04:55 | |
| *** threestrands has joined #tripleo | 05:02 | |
| *** chandankumar has joined #tripleo | 05:09 | |
| *** chandankumar is now known as chkumar|ruck | 05:10 | |
| *** ooolpbot has joined #tripleo | 05:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 05:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 05:10 |
| *** ooolpbot has quit IRC | 05:10 | |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 05:10 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 05:10 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart-extras master: Make build-test-packages more resilient https://review.openstack.org/616937 | 05:13 |
| chkumar|ruck | ykarel, Hello | 05:13 |
| *** ykarel has quit IRC | 05:13 | |
| *** ykarel has joined #tripleo | 05:19 | |
| *** janki has joined #tripleo | 05:37 | |
| *** yprokule has joined #tripleo | 05:57 | |
| *** ykarel_ has joined #tripleo | 06:00 | |
| *** ykarel has quit IRC | 06:03 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: Cleanup irrelevant scripts from previous releases. https://review.openstack.org/615370 | 06:07 |
| *** ykarel_ is now known as ykarel | 06:08 | |
| *** ooolpbot has joined #tripleo | 06:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 06:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 06:10 |
| *** ooolpbot has quit IRC | 06:10 | |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 06:10 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 06:10 |
| *** udesale has quit IRC | 06:16 | |
| *** mburned_out is now known as mburned | 06:16 | |
| *** udesale has joined #tripleo | 06:16 | |
| *** shardy has joined #tripleo | 06:19 | |
| openstackgerrit | Janki Chhatbar proposed openstack/puppet-tripleo master: Pass variable from puppet-tripleo to puppet-neutron https://review.openstack.org/617265 | 06:19 |
| chkumar|ruck | due to puppet5 changes jobs are going to fail which are in gate queue | 06:19 |
| *** sanjayu__ has joined #tripleo | 06:24 | |
| *** radeks has joined #tripleo | 06:34 | |
| *** agurenko has joined #tripleo | 06:34 | |
| *** shardy has quit IRC | 06:37 | |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: WIP implement python based uploader https://review.openstack.org/616018 | 06:41 |
| openstackgerrit | Steve Baker proposed openstack/tripleo-common master: WIP make python uploader the default https://review.openstack.org/616019 | 06:41 |
| *** ykarel_ has joined #tripleo | 06:42 | |
| openstackgerrit | Yurii Prokulevych proposed openstack/tripleo-upgrade master: Drop remaining ssh-user from templates. https://review.openstack.org/617455 | 06:44 |
| *** ykarel has quit IRC | 06:45 | |
| *** ksambor has joined #tripleo | 06:46 | |
| *** quiquell|off is now known as quiquell | 06:50 | |
| quiquell | mwhahaha: ack | 06:50 |
| openstackgerrit | Quique Llorente proposed openstack/python-tripleoclient master: Use corrent ansible-playbook cmd for py3 https://review.openstack.org/616579 | 06:58 |
| openstackgerrit | zhouxinyong proposed openstack/diskimage-builder master: delete the duplicate words in package-outside-debootstrap-ac93e9ce991819f1.yaml https://review.openstack.org/617501 | 07:02 |
| *** skramaja has joined #tripleo | 07:05 | |
| openstackgerrit | zhouxinyong proposed openstack/diskimage-builder master: fix some errors for ill-syntax in README.rst https://review.openstack.org/617502 | 07:06 |
| *** mburned is now known as mburned_out | 07:09 | |
| *** ooolpbot has joined #tripleo | 07:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 07:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 07:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 07:10 |
| *** ooolpbot has quit IRC | 07:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 07:10 |
| *** jfrancoa has joined #tripleo | 07:11 | |
| Tengu | mwhahaha: hey :). will have a look. | 07:14 |
| Tengu | mwhahaha: oh, funky. I think I talked about that last year already in fact :D | 07:16 |
| chkumar|ruck | Tengu, we have reverted puppet-5 change in RDO https://review.rdoproject.org/r/#/c/17333/1 | 07:20 |
| Tengu | chkumar|ruck: well, of course ;). puppet5 has some new ABI we can't directly use. | 07:20 |
| Tengu | I didn't follow all the changes at my previous work, but we had to make "some" changes to our puppet receipts. | 07:21 |
| chkumar|ruck | Tengu, does this change https://review.openstack.org/#/c/617441/ will not break after the revert? | 07:21 |
| Tengu | chkumar|ruck: it «should not» I think. lemme check something, I'm pretty sure I did stuff while pushing my dynamic haproxy thingy. | 07:22 |
| chkumar|ruck | Tengu, sure, thanks :-) | 07:22 |
| Tengu | https://github.com/openstack/puppet-tripleo/blob/master/manifests/haproxy/service_endpoints.pp#L36-L40 | 07:24 |
| Tengu | soooo... well. hm. I think we might need to do something regarding the firewall. | 07:25 |
| *** ykarel_ has quit IRC | 07:25 | |
| Tengu | but at least for haproxy, I made it in a way to support both notation. | 07:25 |
| *** ykarel_ has joined #tripleo | 07:26 | |
| chkumar|ruck | Tengu, do we need some change there? | 07:26 |
| Tengu | https://github.com/openstack/puppet-tripleo/blob/master/manifests/firewall/service_rules.pp#L34-L36 | 07:26 |
| Tengu | nope. | 07:26 |
| Tengu | bam. | 07:26 |
| Tengu | supported. | 07:26 |
| Tengu | the code above is for the old dot notation. | 07:27 |
| chkumar|ruck | ok | 07:27 |
| Tengu | so I think we're safe. | 07:27 |
| Tengu | yeah. 11 months ago. | 07:27 |
| * Tengu foresees things | 07:27 | |
| Tengu | :) | 07:27 |
| openstackgerrit | Damien Ciabrini proposed openstack/tripleo-common master: DNR overcloud deployment with mariadb 10.3.10 and galera 3.23 https://review.openstack.org/617355 | 07:29 |
| *** mschuppert has joined #tripleo | 07:30 | |
| *** ykarel__ has joined #tripleo | 07:30 | |
| chkumar|ruck | marios, sshnaidm \o/ | 07:32 |
| chkumar|ruck | marios, sshnaidm we need this patch https://review.openstack.org/#/c/617441/2 to unblock gate | 07:32 |
| *** ykarel_ has quit IRC | 07:33 | |
| quiquell | chkumar|ruck: massive drone patch | 07:33 |
| chkumar|ruck | quiquell, yes :-) | 07:34 |
| quiquell | jfrancoa: to unlock gates https://review.openstack.org/#/c/617441 | 07:34 |
| quiquell | jistr: ^? | 07:34 |
| chkumar|ruck | quiquell, container upgrade job is failing with same overcloud deploy issue | 07:35 |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations stable/rocky: Fix the undercloud-heat-purge-deleted validation https://review.openstack.org/617511 | 07:36 |
| *** sanjayu__ has quit IRC | 07:40 | |
| *** saneax has joined #tripleo | 07:43 | |
| *** saneax has quit IRC | 07:44 | |
| *** gkadam has joined #tripleo | 07:46 | |
| jfrancoa | quiquell: reviewing | 07:47 |
| *** saneax has joined #tripleo | 07:47 | |
| quiquell | jfrancoa: to unblock gates | 07:49 |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-quickstart master: Set undercloud_cloud_domain/hostname in fs03,47,50 https://review.openstack.org/617514 | 07:50 |
| *** rodolof has joined #tripleo | 07:50 | |
| chkumar|ruck | ykarel__, ^^ | 07:50 |
| *** ykarel__ is now known as ykarel | 07:50 | |
| jfrancoa | quiquell: done | 07:51 |
| Tengu | cool :) | 07:52 |
| ykarel | chkumar|ruck, today morning i ran fs050 and it passed, not sure if that change is required | 07:53 |
| ykarel | chkumar|ruck, https://review.rdoproject.org/r/#/c/13943/ | 07:53 |
| chkumar|ruck | ykarel, can you check fs03 and fs047 also? | 07:54 |
| chkumar|ruck | or let me confirm that | 07:54 |
| quiquell | jfrancoa: thanks | 07:54 |
| ykarel | chkumar|ruck, so fs050 passed because it has containerized_undercloud: false | 07:56 |
| ykarel | and in fs003 u are setting that here: https://review.openstack.org/#/c/615766/3/config/general_config/featureset003.yml@5 | 07:59 |
| chkumar|ruck | ykarel, so keep the assumption this way if containerized_undercloud to false there it is not needed | 08:01 |
| chkumar|ruck | ykarel, otherwise it might needed? | 08:01 |
| *** slaweq has joined #tripleo | 08:01 | |
| ykarel | chkumar|ruck, so the issue started with: Smarter containerization defaults https://review.openstack.org/#/c/611601/, one workaround was to set containerized_undercloud=false where issue is seen and hostname setting is done in: https://review.openstack.org/#/c/615730/ | 08:03 |
| ykarel | so in short we are cleaning up issues created by https://review.openstack.org/#/c/611601/ | 08:04 |
| *** threestrands has quit IRC | 08:04 | |
| chkumar|ruck | ykarel, yes, | 08:06 |
| *** ooolpbot has joined #tripleo | 08:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 08:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 08:10 |
| *** ooolpbot has quit IRC | 08:10 | |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 08:10 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 08:10 |
| *** paramite has joined #tripleo | 08:17 | |
| openstackgerrit | Slawek Kaplonski proposed openstack/tripleo-heat-templates master: Remove external_network_bridge Neutron option https://review.openstack.org/604764 | 08:23 |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Add customized libvirt-guests unit file to properly shutdown instances https://review.openstack.org/617521 | 08:24 |
| *** quiquell has quit IRC | 08:32 | |
| *** quiquell has joined #tripleo | 08:33 | |
| *** amoralej|off is now known as amoralej | 08:36 | |
| *** tosky has joined #tripleo | 08:38 | |
| chkumar|ruck | FYI! puppet-5 is removed from rdo, Feel free to recheck your patches if you hits this bug https://bugs.launchpad.net/tripleo/+bug/1803024 | 08:49 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 08:49 |
| *** itlinux has joined #tripleo | 08:50 | |
| *** jpena|off is now known as jpena | 08:50 | |
| *** itlinux has quit IRC | 08:50 | |
| *** aufi has joined #tripleo | 08:53 | |
| *** aufi_ has joined #tripleo | 08:54 | |
| *** aufi_ has quit IRC | 08:56 | |
| *** aufi has quit IRC | 08:58 | |
| *** aufi has joined #tripleo | 08:58 | |
| *** shardy has joined #tripleo | 09:02 | |
| chkumar|ruck | slaweq, Hello | 09:02 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Use ipc: host with containers that use pcs https://review.openstack.org/617525 | 09:02 |
| chkumar|ruck | slaweq, Please have a look at this tempest log http://logs.openstack.org/03/616203/9/gate/tripleo-ci-centos-7-standalone/75f1be5/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-11-12_17_40_46 | 09:03 |
| slaweq | chkumar|ruck: hi, looking | 09:03 |
| chkumar|ruck | slaweq, from cirros image when it is trying to find the instance id it is failing | 09:03 |
| slaweq | chkumar|ruck: see here: http://logs.openstack.org/03/616203/9/gate/tripleo-ci-centos-7-standalone/75f1be5/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-11-12_17_40_46 - it don't even have fixed IP configured | 09:05 |
| *** ccamacho has joined #tripleo | 09:06 | |
| slaweq | chkumar|ruck: where I can find overcloud logs in this job? | 09:06 |
| chkumar|ruck | slaweq it is a standalone job | 09:06 |
| chkumar|ruck | slaweq, http://logs.openstack.org/03/616203/9/gate/tripleo-ci-centos-7-standalone/75f1be5/logs/undercloud/var/log/extra/docker/containers/ | 09:07 |
| chkumar|ruck | slaweq, here is all the container logs | 09:07 |
| *** janki has quit IRC | 09:07 | |
| slaweq | thx | 09:07 |
| chkumar|ruck | slaweq, on mutliple recheck it passes | 09:07 |
| *** ykarel is now known as ykarel|lunch | 09:10 | |
| chkumar|ruck | slaweq, https://bugs.launchpad.net/tripleo/+bug/1802971 here is the bug we are tracking it | 09:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 09:10 |
| *** ooolpbot has joined #tripleo | 09:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 09:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 09:10 |
| *** ooolpbot has quit IRC | 09:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 09:10 |
| *** jpich has joined #tripleo | 09:11 | |
| *** aufi_ has joined #tripleo | 09:15 | |
| *** aufi has quit IRC | 09:18 | |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-common master: Install ironic-staging-drivers in ironic-conductor https://review.openstack.org/617538 | 09:25 |
| chkumar|ruck | slaweq, I tried to reproduce locally but it passed there | 09:26 |
| *** derekh has joined #tripleo | 09:26 | |
| chkumar|ruck | slaweq, on multiple rechecks it passes | 09:26 |
| slaweq | chkumar|ruck: I think we have similar issue on neutron's gates from time to time | 09:26 |
| slaweq | in this case which You gave me, it was fine for first spawned instance and it failed for second one | 09:27 |
| slaweq | is it always like that? or sometimes it fails on spawning first vm? | 09:27 |
| chkumar|ruck | slaweq, it is always like that | 09:27 |
| *** shardy has quit IRC | 09:27 | |
| chkumar|ruck | slaweq, first vm spanning works fine but second one fails | 09:28 |
| *** ramishra has quit IRC | 09:28 | |
| chkumar|ruck | slaweq, Do we have a bug tracking the same issue on neutron side? | 09:32 |
| *** maufart__ has joined #tripleo | 09:32 | |
| slaweq | chkumar|ruck: I don't think so | 09:33 |
| chkumar|ruck | slaweq, anyworkaround, and debug info we can add on tripleo side to get more info? | 09:33 |
| chkumar|ruck | if it happens again so that we can troubleshoot it | 09:34 |
| slaweq | chkumar|ruck: I'm looking in logs of this job logs from dnsmasq for this network, do You know if it is somewhere? | 09:34 |
| *** ramishra has joined #tripleo | 09:34 | |
| chkumar|ruck | Tengu, ^^ | 09:34 |
| *** aufi_ has quit IRC | 09:35 | |
| Tengu | hmm ? | 09:36 |
| Tengu | ah, dnsmasq container started by the neutron_dhcp container? | 09:36 |
| slaweq | Tengu: yes | 09:37 |
| Tengu | I don't think there are logs for that one unfortunately. if any, it should be in /var/log/containers/neutron directoriy. | 09:37 |
| Tengu | but afaik there's only the log for the "main" dhcp container, the one starting the dnsmasq "sidecar" | 09:37 |
| Tengu | maybe beagles can have some more info about that | 09:38 |
| slaweq | :/ | 09:38 |
| slaweq | Tengu: yes, I will ask him later about it | 09:38 |
| slaweq | chkumar|ruck: I will also add neutron to affected project in this bug and will check if it's exactly same issue if we will spot it in Neutron | 09:38 |
| chkumar|ruck | slaweq, sure, till then I will work on adding a elastic-recheck query for the same | 09:39 |
| chkumar|ruck | slaweq, Tengu thanks :-) | 09:39 |
| Tengu | np :) | 09:39 |
| * Tengu goes back to his "fetch metrics" work | 09:39 | |
| *** akrivoka has joined #tripleo | 09:43 | |
| *** ykarel|lunch is now known as ykarel | 09:45 | |
| *** pcaruana has joined #tripleo | 09:56 | |
| *** jtomasek has joined #tripleo | 10:00 | |
| *** jtomasek has quit IRC | 10:01 | |
| *** ooolpbot has joined #tripleo | 10:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 10:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 10:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 10:10 |
| *** ooolpbot has quit IRC | 10:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 10:10 |
| chkumar|ruck | Tengu, Hello | 10:14 |
| chkumar|ruck | Tengu, I need some help here to debug the issue http://logs.rdoproject.org/33/614633/6/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/d39ceb6/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz it is failing with no valid host found | 10:14 |
| chkumar|ruck | Tengu, as per the error I got http://logs.rdoproject.org/33/614633/6/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/d39ceb6/logs/undercloud/var/log/containers/nova/nova-conductor.log.txt.gz | 10:15 |
| chkumar|ruck | Tengu, any other place I can look to find out the issue? | 10:15 |
| *** ramishra_ has joined #tripleo | 10:18 | |
| openstackgerrit | James Slagle proposed openstack/tripleo-common stable/rocky: Sync state if needed during retrieval https://review.openstack.org/617555 | 10:19 |
| *** ramishra has quit IRC | 10:20 | |
| *** hkominos_ has joined #tripleo | 10:24 | |
| hkominos_ | Hi all. I have a quick question regarding overriding ansible.cfg. Do I need to provide an ansible.cfg with all the required configs or will tripleo populate what I have not provided with what it needs and just attach the extra config that I have given ? | 10:25 |
| *** owalsh_ is now known as owalsh | 10:27 | |
| *** slaweq_ has joined #tripleo | 10:30 | |
| Tengu | chkumar|ruck: hmm wait. | 10:32 |
| Tengu | chkumar|ruck: there should be something with the Filters. | 10:34 |
| Tengu | but I don't remember what's the right string to search for, nor the right nova log for that -.-. | 10:35 |
| *** psachin has quit IRC | 10:36 | |
| *** slaweq_ has quit IRC | 10:39 | |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: DNM: testing shortwiring to skip playbook runs https://review.openstack.org/616994 | 10:40 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml https://review.openstack.org/616993 | 10:43 |
| *** shardy has joined #tripleo | 10:43 | |
| *** janki has joined #tripleo | 10:43 | |
| *** ccamacho has quit IRC | 10:47 | |
| *** psachin has joined #tripleo | 10:48 | |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-common stable/queens: [Queens only] Fix the default for docker registry. https://review.openstack.org/611329 | 10:54 |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-common stable/pike: [Pike only] Fix docker registry url. https://review.openstack.org/611596 | 10:57 |
| openstackgerrit | Sorin Sbarnea proposed openstack/python-tripleoclient master: Avoid printing b'' across logged output https://review.openstack.org/617363 | 10:58 |
| *** florianf|afk is now known as florianf|summit | 11:00 | |
| *** apetrich has joined #tripleo | 11:02 | |
| chkumar|ruck | slaweq, weshay added https://review.openstack.org/#/c/617579/ added elastic recheck for flakky tempest tests | 11:09 |
| *** ooolpbot has joined #tripleo | 11:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 11:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 11:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 11:10 |
| *** ooolpbot has quit IRC | 11:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 11:10 |
| *** mburned_out is now known as mburned | 11:11 | |
| *** morazi has joined #tripleo | 11:11 | |
| ramishra_ | chkumar|ruck: I updated 1802971 with some errors I noticed in the logs. Is that happening very often? looks to me like some kind of infra issue possibly | 11:13 |
| *** panda|rover|lch is now known as panda|rover | 11:13 | |
| *** chkumar|ruck has quit IRC | 11:14 | |
| *** chandankumar has joined #tripleo | 11:15 | |
| *** chandankumar is now known as chkumar|ruck | 11:15 | |
| openstackgerrit | Rico Lin proposed openstack/os-collect-config stable/pike: Add region support https://review.openstack.org/617593 | 11:19 |
| openstackgerrit | Rico Lin proposed openstack/os-collect-config stable/ocata: Add region support https://review.openstack.org/617594 | 11:19 |
| chkumar|ruck | slaweq, https://bugs.launchpad.net/tripleo/+bug/1802971/comments/5 | 11:19 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 11:19 |
| *** rodolof has quit IRC | 11:22 | |
| *** rodolof has joined #tripleo | 11:23 | |
| *** shardy has quit IRC | 11:24 | |
| *** apetrich has quit IRC | 11:26 | |
| *** bnemec has joined #tripleo | 11:26 | |
| *** phuongnh has quit IRC | 11:30 | |
| *** ykarel_ has joined #tripleo | 11:30 | |
| openstackgerrit | Gael Chamoulaud proposed openstack/tripleo-validations master: Fix mysql_open_files_limit validation https://review.openstack.org/617603 | 11:31 |
| *** ykarel has quit IRC | 11:33 | |
| *** bnemec has quit IRC | 11:35 | |
| *** dciabrin has quit IRC | 11:36 | |
| *** chkumar has joined #tripleo | 11:36 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/python-tripleoclient master: Add in-repo zuul jobs definitions and updates them https://review.openstack.org/617606 | 11:36 |
| *** chkumar|ruck has quit IRC | 11:37 | |
| *** chkumar is now known as chkumar|ruck | 11:37 | |
| *** ykarel_ is now known as ykarel | 11:45 | |
| ykarel | chkumar|ruck, check placement logs as well | 11:46 |
| ykarel | i mean to look for No valid host issue | 11:47 |
| *** raildo has joined #tripleo | 11:51 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Use ipc: host with containers that use pcs https://review.openstack.org/617525 | 11:53 |
| chkumar|ruck | ykarel, ack | 11:57 |
| *** morazi has quit IRC | 11:57 | |
| ykarel | chkumar|ruck, to me it looks like hjensas patch has caused some issue, is https://review.openstack.org/#/c/614540/10/extraconfig/post_deploy/undercloud_post.py@65 correct? MEMORY_MB=1?, i can be wrong if any ovb job passed after that. | 11:57 |
| ykarel | dtantsur|afk, owalsh can u check that ^^ | 11:57 |
| ykarel | chkumar|ruck, are u aware of any ovb job passing in master after that patch? | 11:59 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout https://review.openstack.org/617617 | 12:00 |
| chkumar|ruck | ykarel, currently almost all are broken, we need this one first https://review.openstack.org/#/c/616937/ to unblock | 12:02 |
| chkumar|ruck | due to overcloud deploy issue | 12:02 |
| ykarel | chkumar|ruck, but with that patch also ovb failing, right? | 12:03 |
| ykarel | in master | 12:03 |
| chkumar|ruck | ykarel, ovb is blocked on dlrn permission denied | 12:03 |
| ykarel | chkumar|ruck, i meant with Depends-On on the fix | 12:04 |
| ykarel | ovb are passing or not | 12:04 |
| *** ccamacho has joined #tripleo | 12:04 | |
| chkumar|ruck | ykarel, one min let me check that all | 12:04 |
| ykarel | last vexhost one passed on 8th: https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost&branch=master | 12:04 |
| ykarel | the patch i mentioned merged on 9th | 12:04 |
| ykarel | so good to check if any master ovb job passed(with dlrn permission issue fix) after 9th | 12:05 |
| ykarel | if not good to try MEMORY_MB=0 change | 12:05 |
| panda|rover | chkumar|ruck: ^ | 12:06 |
| chkumar|ruck | panda|rover, ykarel testing both patches together on vexhost here https://review.rdoproject.org/r/#/c/13943/ | 12:07 |
| openstackgerrit | Sorin Sbarnea proposed openstack/tripleo-quickstart master: Avoids use of ignore_errors while loading OS specific variables https://review.openstack.org/617118 | 12:08 |
| ykarel | chkumar|ruck, ack | 12:10 |
| *** ooolpbot has joined #tripleo | 12:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 12:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 12:10 |
| *** ooolpbot has quit IRC | 12:10 | |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 12:10 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 12:10 |
| chkumar|ruck | slaweq, I am trying to recreate the proper query | 12:10 |
| *** mburned is now known as mburned_out | 12:14 | |
| owalsh | ykarel: yea, pretty sure that should be 0 | 12:17 |
| janki | marios, hi | 12:17 |
| ykarel | owalsh, ack chkumar|ruck panda|rover ^^ | 12:17 |
| chkumar|ruck | ykarel, submitting a patvch | 12:18 |
| ykarel | chkumar|ruck, ack also check that patch if something else was also wrong apart from MEMORY_MB | 12:18 |
| chkumar|ruck | ykarel, owalsh http://git.openstack.org/cgit/openstack/tripleo-docs/tree/doc/source/install/advanced_deployment/baremetal_overcloud.rst#n606 | 12:21 |
| chkumar|ruck | only memory was wrong there | 12:21 |
| ykarel | yup i worked on similar issue so was aware | 12:22 |
| ykarel | chkumar|ruck, with wrong i meant anything else when transitioning from shell script to python | 12:22 |
| panda|rover | chkumar|ruck: jobs are going to fail again in the queue | 12:23 |
| ykarel | but ok to start with MEMORY_MB change | 12:23 |
| panda|rover | chkumar|ruck: time to abandon changes ... | 12:23 |
| chkumar|ruck | panda|rover, yes, alot of changes needed to be abandon in order to clear the gates | 12:24 |
| *** rh-jelabarre has joined #tripleo | 12:24 | |
| *** rh-jelabarre has quit IRC | 12:25 | |
| *** ccamacho has quit IRC | 12:26 | |
| *** rh-jelabarre has joined #tripleo | 12:26 | |
| *** ccamacho has joined #tripleo | 12:26 | |
| openstackgerrit | Chandan Kumar proposed openstack/tripleo-heat-templates master: Set flavor property resources:MEMORY_MB to 0 https://review.openstack.org/617621 | 12:26 |
| chkumar|ruck | panda|rover, ykarel ^^ | 12:27 |
| chkumar|ruck | panda|rover, can we start abandoning patches? | 12:28 |
| *** ccamacho has quit IRC | 12:28 | |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart master: Repurpose featureset 5 for new standalone-scenario001 job https://review.openstack.org/616871 | 12:29 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job https://review.openstack.org/616872 | 12:29 |
| ykarel | chkumar|ruck, ack | 12:29 |
| *** ccamacho has joined #tripleo | 12:29 | |
| panda|rover | do you know if they're ususally using a script ? | 12:29 |
| ykarel | chkumar|ruck, good to push a seperate patch to test ^^, keep current jobs running to confirm the issue | 12:29 |
| *** apetrich has joined #tripleo | 12:29 | |
| panda|rover | otherwise I'll have to write donw all the patches to abandon and restore them after. | 12:30 |
| panda|rover | I'll abandon only the ones that are currently in the gates | 12:30 |
| chkumar|ruck | panda|rover, no idea, | 12:30 |
| chkumar|ruck | panda|rover, https://review.openstack.org/616937 https://review.openstack.org/617441 | 12:30 |
| chkumar|ruck | doesnot needs to be abandon | 12:31 |
| chkumar|ruck | panda|rover, and this also https://review.openstack.org/#/c/617308/ | 12:31 |
| *** abishop has quit IRC | 12:32 | |
| *** bnemec has joined #tripleo | 12:33 | |
| chkumar|ruck | ykarel, where you test stuff apart from noop jobs in RDO? | 12:33 |
| *** ccamacho has quit IRC | 12:33 | |
| *** ccamacho has joined #tripleo | 12:33 | |
| ykarel | chkumar|ruck, u can push change to rdo-jobs repo or any other repo | 12:34 |
| chkumar|ruck | ykarel, ack | 12:35 |
| ykarel | chkumar|ruck, like https://review.rdoproject.org/r/#/c/17115/1/zuul.d/projects.yaml | 12:35 |
| *** maufart__ has quit IRC | 12:36 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Run tempest from package in non_containerized env https://review.openstack.org/615766 | 12:38 |
| *** ccamacho has quit IRC | 12:38 | |
| panda|rover | 1someone did not read the mail and is approving stuff .. | 12:39 |
| chkumar|ruck | panda|rover, please update the channel topic | 12:40 |
| *** jpena is now known as jpena|lunch | 12:40 | |
| panda|rover | chkumar|ruck: I'm not op | 12:41 |
| panda|rover | and I'm not even in the access list, and the topic is locked | 12:41 |
| chkumar|ruck | EmilienM, mwhahaha we need to update the channel topic in order to stop people doing workflow | 12:42 |
| EmilienM | ok | 12:42 |
| EmilienM | let me do it | 12:42 |
| *** EmilienM changes topic to "CI Status: Do not RECHECK or APPROVE any patch now. master is RED | https://docs.openstack.org/tripleo-docs/latest/" | 12:43 | |
| chkumar|ruck | EmilienM, thanks! | 12:43 |
| *** vpickard_ is now known as vpickard | 12:43 | |
| EmilienM | we also need to purge the gate right? | 12:43 |
| chkumar|ruck | EmilienM, yes, | 12:43 |
| panda|rover | EmilienM: do you have a script for that ? | 12:43 |
| EmilienM | no by hand | 12:43 |
| panda|rover | chkumar|ruck: why shouldn't I abandon 617308 ? | 12:44 |
| *** morazi has joined #tripleo | 12:44 | |
| EmilienM | do you want me to do it? | 12:45 |
| chkumar|ruck | panda|rover, https://review.openstack.org/#/c/617308/ needed to fix container build in promotion pipeline | 12:45 |
| ykarel | chkumar|ruck, btw what is blocking in gate? why abandoning patches? | 12:46 |
| ykarel | chkumar|ruck, we reverted puppet, still there any other issue? | 12:46 |
| panda|rover | EmilienM: it's time for me to try | 12:46 |
| chkumar|ruck | ykarel, there is a long zuul queue | 12:46 |
| EmilienM | chkumar|ruck: it's not a reason | 12:46 |
| panda|rover | chkumar|ruck: ok | 12:46 |
| chkumar|ruck | ykarel, https://review.openstack.org/617441 | 12:46 |
| EmilienM | the only reason to reset a gate is when we know there is a bug that prevent patches to land | 12:47 |
| EmilienM | do we have puppet4 back now? | 12:47 |
| ykarel | chkumar|ruck, yup EmilienM is correct | 12:47 |
| panda|rover | the reason is not the long queue, is that without 617441 every job is going to fail, IIUC | 12:47 |
| EmilienM | https://review.openstack.org/#/c/617441/ is for Puppet5 | 12:47 |
| ykarel | EmilienM, yes puppet 4 in place | 12:47 |
| EmilienM | and AFIK puppet4 is back | 12:47 |
| EmilienM | it's on #rdo | 12:47 |
| panda|rover | ok | 12:47 |
| chkumar|ruck | needed in order to fix overcloud deploy failing due tot t this https://bugs.launchpad.net/tripleo/+bug/1803024 | 12:47 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 12:47 |
| *** arxcruz is now known as arxcruz|summit\ | 12:47 | |
| EmilienM | https://trunk.rdoproject.org/centos7-master/deps/latest/noarch/puppet-4.8.2-1.el7.noarch.rpm | 12:47 |
| EmilienM | it's latest ^ | 12:47 |
| EmilienM | so gate is good | 12:47 |
| ykarel | yup correct | 12:47 |
| *** apetrich has quit IRC | 12:47 | |
| EmilienM | don't touch it unless there is another bug | 12:47 |
| *** arxcruz|summit\ is now known as arxcruz|summit | 12:48 | |
| panda|rover | ykarel: since when was that package replaced ? | 12:48 |
| chkumar|ruck | EmilienM, this one is needed https://review.openstack.org/617441 -> to fix this on https://bugs.launchpad.net/tripleo/+bug/1803024 | 12:48 |
| ykarel | panda|rover, around 4 hours back | 12:48 |
| panda|rover | chkumar|ruck: that is needed only for puppet5 | 12:48 |
| panda|rover | chkumar|ruck: but now we have puppet4 again | 12:49 |
| chkumar|ruck | panda|rover, it will work with puppet4 already confirmed with panda | 12:49 |
| EmilienM | chkumar|ruck: like I said, it's for Puppet5 | 12:49 |
| chkumar|ruck | panda|rover, it will work with puppet4 already confirmed with Tengu | 12:49 |
| weshay | chkumar|ruck, ykarel are the package check jobs working at all for master? | 12:49 |
| panda|rover | ykarel: mmh so all the jobs in queue should already be using that | 12:49 |
| *** thrash|g0ne is now known as thrash | 12:50 | |
| chkumar|ruck | EmilienM, ok | 12:51 |
| Tengu | EmilienM: I pushed a patch something like 11 months ago in order to support both "::" and old, deprecated "." notation in puppet-tripleo :) | 12:51 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/tripleo-ci master: Fix if condition in bash https://review.openstack.org/617629 | 12:51 |
| Tengu | EmilienM: so yeah, it should be transparent. I was using the "::" notation at my former job without any issue. | 12:51 |
| EmilienM | ok | 12:51 |
| Tengu | because dot is bad. | 12:52 |
| ykarel | weshay, u mean deps update? the patch that bumped puppet5? | 12:52 |
| sshnaidm | panda|rover, marios please something trivial and urgent: https://review.openstack.org/617629 | 12:52 |
| weshay | ykarel, http://logs.rdoproject.org/87/17187/5/check/legacy-rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset016/de9585f/ | 12:52 |
| Tengu | fun part: DOT certification for motorbike helmet is probably the worst in the world. | 12:52 |
| Tengu | we should ban dot | 12:52 |
| Tengu | :3 | 12:52 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: Use ipc: host with containers that use pcs https://review.openstack.org/617525 | 12:53 |
| weshay | ykarel, chkumar|ruck every success here, seems like a false positive https://review.rdoproject.org/zuul/builds?job_name=legacy-rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset016 | 12:53 |
| ykarel | weshay, yes there is some issue with that job with rdo-cloud provider. and for update part i think EmilienM haikel had some disscussion as seeing the comments on the patch | 12:53 |
| ykarel | weshay, those jobs runns only when there is change in master deps so jobs running for few minutes and showing SUCCESS are true | 12:54 |
| weshay | ykarel, regardless of the patch, that job .. can you show me a log where it's working correctly | 12:54 |
| ykarel | weshay, okk fetching, but iirc that job started failing after vexxhost change in tripleo-ci | 12:55 |
| ykarel | in rdo-cloud provider | 12:55 |
| EmilienM | damn puppet5 | 12:55 |
| ykarel | rdo-cloud-tripleo is fine | 12:55 |
| ykarel | amoralej, ^^ do we have fix in place ^^ | 12:55 |
| amoralej | yes | 12:55 |
| amoralej | it should | 12:55 |
| amoralej | let me know otherwise | 12:56 |
| ykarel | amoralej, u fixed provider? | 12:56 |
| ykarel | the job is still running with rdo-cloud provider and tripleo-ci don't have a change to handle that | 12:56 |
| weshay | ykarel, so imho that would be a reason to open a lp w/ a promotion blocker | 12:56 |
| fultonj | i have a oooq-extras patch where i try to use the collect-logs roles to collect a file my job produces, however it is not getting collected. does anyone see anything wrong with my patch? https://review.openstack.org/#/c/617368/2/roles/collect-logs/defaults/main.yml | 12:57 |
| weshay | ykarel, running w/o any jobs to check/gate package changes is not acceptable | 12:57 |
| fultonj | AFICT the file was created http://logs.openstack.org/88/615988/2/check/tripleo-ci-split-controlplane-standalone/118f43e/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-11-12_23_29_06 | 12:57 |
| *** ykarel_ has joined #tripleo | 12:59 | |
| panda|rover | fultonj: that list is for the file that we need to pass to logstash tfor it to index :) | 12:59 |
| panda|rover | fultonj: the list you're looking for is the first on the file | 12:59 |
| *** matbu has quit IRC | 13:00 | |
| *** derekh has quit IRC | 13:00 | |
| fultonj | panda|rover: indeed artcl_logstash_files != artcl_collect_list , i jumped to the bottom of the file too quickly to make my edit. thanks panda|rover | 13:00 |
| weshay | ykarel, can you please open a promotion-blocker bug, so we can track the issue w/ the lack of coverage atm | 13:01 |
| ykarel_ | weshay, link for long for success job: https://logs.rdoproject.org/66/17266/1/check/legacy-rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset016/dc7fc64 | 13:01 |
| ykarel_ | weshay, agree, let me check what's missed | 13:02 |
| *** ykarel has quit IRC | 13:02 | |
| EmilienM | things should merge | 13:02 |
| EmilienM | again don't touch the gate | 13:03 |
| weshay | ykarel_, ya.. this looks good https://logs.rdoproject.org/66/17266/1/check/legacy-rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset016/dc7fc64/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz | 13:03 |
| ykarel_ | weshay, that ran with rdo-cloud-tripleo | 13:03 |
| weshay | ykarel_, why isn't haikel in this channel | 13:04 |
| ykarel_ | with rdo-cloud provider there is some issue | 13:04 |
| ykarel_ | weshay, no idea | 13:04 |
| weshay | nhicher, ah ya.. we were looking at that | 13:04 |
| *** ykarel_ is now known as ykarel | 13:05 | |
| weshay | ykarel, ping me with a lp when you have one | 13:05 |
| ykarel | weshay, ack | 13:06 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Replace multinode-scenario001 with standalone-scenario001 https://review.openstack.org/617635 | 13:06 |
| openstackgerrit | Marios Andreou proposed openstack/ansible-role-tripleo-modify-image master: Replace multinode-scenario001 with standalone-scenario001 https://review.openstack.org/617636 | 13:06 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo master: Replace multinode-scenario001 with standalone-scenario001 https://review.openstack.org/617637 | 13:06 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common master: Replace multinode-scenario001 with standalone-scenario001 https://review.openstack.org/617638 | 13:06 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job https://review.openstack.org/616872 | 13:07 |
| *** morazi has quit IRC | 13:07 | |
| *** dhill_ has joined #tripleo | 13:09 | |
| *** ooolpbot has joined #tripleo | 13:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 13:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 13:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 13:10 |
| *** ooolpbot has quit IRC | 13:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 13:10 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Replace multinode-scenario001 with standalone-scenario001 https://review.openstack.org/617635 | 13:11 |
| *** abishop has joined #tripleo | 13:11 | |
| *** saneax has quit IRC | 13:15 | |
| openstackgerrit | John Fulton proposed openstack/tripleo-quickstart-extras master: WIP/DNM: Add split_control_plane_{controller,compute} booleans https://review.openstack.org/617368 | 13:17 |
| *** amoralej is now known as amoralej|lunch | 13:18 | |
| *** aufi has joined #tripleo | 13:19 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: WIP - Implement tripleo-systemd-wrapper role https://review.openstack.org/617377 | 13:21 |
| ykarel | weshay, amoralej|lunch https://bugs.launchpad.net/tripleo/+bug/1803133 | 13:21 |
| openstack | Launchpad bug 1803133 in tripleo "jobs running with "rdo-cloud" provider are broken, mainly jobs running on package update in RDO" [Undecided,New] | 13:21 |
| *** rlandy has joined #tripleo | 13:27 | |
| *** saneax has joined #tripleo | 13:29 | |
| *** bnemec has quit IRC | 13:29 | |
| *** tbonds has quit IRC | 13:30 | |
| weshay | ykarel, thanks | 13:31 |
| ykarel | weshay, should i push a fix or u will take care | 13:32 |
| ykarel | i have added possible cause/fix in the bug | 13:32 |
| weshay | ykarel, if you have a fix go 4 it | 13:32 |
| ykarel | weshay, ack sending in few | 13:32 |
| weshay | sshnaidm, https://nb01.openstack.org/images/ | 13:34 |
| *** rodolof has quit IRC | 13:35 | |
| *** rodolof has joined #tripleo | 13:36 | |
| *** apetrich has joined #tripleo | 13:38 | |
| *** jpena|lunch is now known as jpena | 13:40 | |
| panda|rover | the queue reset again | 13:41 |
| openstackgerrit | yatin proposed openstack-infra/tripleo-ci master: Also support rdo-cloud provider https://review.openstack.org/617653 | 13:42 |
| ykarel | weshay, ^^ | 13:42 |
| *** janki has quit IRC | 13:44 | |
| panda|rover | chkumar|ruck: http://logs.openstack.org/70/616270/1/gate/tripleo-ci-centos-7-containers-multinode/90327ea/job-output.txt.gz | 13:46 |
| *** vinaykns has joined #tripleo | 13:47 | |
| *** chem has quit IRC | 13:49 | |
| *** chem has joined #tripleo | 13:49 | |
| chkumar|ruck | weshay, https://review.rdoproject.org/zuul/stream/7d119e089ddf49e39739313f6d1c442e?logfile=console.log | 13:52 |
| *** lblanchard has joined #tripleo | 13:53 | |
| chkumar|ruck | weshay, https://review.rdoproject.org/r/#/c/17339 | 13:53 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Fix new node detection https://review.openstack.org/613637 | 13:53 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Do not include node scale up playbook in case of new masters https://review.openstack.org/613641 | 13:53 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Make openshift-master service idempotent https://review.openstack.org/605796 | 13:53 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Add hosts to expected ansible groups https://review.openstack.org/617659 | 13:53 |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Call etcd scaleup playbook when adding master nodes https://review.openstack.org/617660 | 13:53 |
| *** dciabrin has joined #tripleo | 13:53 | |
| *** mcornea has joined #tripleo | 13:54 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout https://review.openstack.org/617617 | 13:54 |
| *** hamzy has quit IRC | 13:55 | |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: Use ipc: host with containers that use pcs https://review.openstack.org/617525 | 13:57 |
| *** hberaud has quit IRC | 13:57 | |
| *** toure is now known as toure|biab | 14:00 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: WIP - Implement tripleo-systemd-wrapper role https://review.openstack.org/617377 | 14:02 |
| *** spsurya has joined #tripleo | 14:03 | |
| *** hgibson has joined #tripleo | 14:05 | |
| *** apetrich has quit IRC | 14:06 | |
| *** tbonds has joined #tripleo | 14:07 | |
| weshay | chkumar|ruck, panda|rover https://review.openstack.org/#/c/617441/ | 14:09 |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-quickstart master: [Configuration] Increasing the ssh timeout. https://review.openstack.org/617663 | 14:10 |
| *** ooolpbot has joined #tripleo | 14:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 14:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 14:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 14:10 |
| *** ooolpbot has quit IRC | 14:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 14:10 |
| weshay | chkumar|ruck, panda|rover https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-centos-7-master-containers-build | 14:10 |
| *** tzumainn has joined #tripleo | 14:10 | |
| chkumar|ruck | panda|rover, https://review.openstack.org/#/c/617308/ | 14:14 |
| *** udesale has quit IRC | 14:16 | |
| openstackgerrit | Steven Hardy proposed openstack/tripleo-heat-templates master: Revert "Revert "Convert *tasks from bootstrap_nodeid to short_bootstrap_node_name"" https://review.openstack.org/611800 | 14:17 |
| weshay | ykarel++ | 14:17 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: WIP / fs010: enable Podman on the overcloud https://review.openstack.org/612526 | 14:17 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-quickstart master: WIP / fs010: enable Podman on the overcloud https://review.openstack.org/612526 | 14:17 |
| *** chkumar|ruck has quit IRC | 14:17 | |
| *** toure|biab is now known as toure | 14:19 | |
| hjensas | ykarel: nice finding! Thank you! | 14:20 |
| *** dsneddon has joined #tripleo | 14:22 | |
| openstackgerrit | Quique Llorente proposed openstack/tripleo-quickstart-extras master: DNM To see python version from openstack command https://review.openstack.org/617666 | 14:27 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: To test standalone-28 https://review.openstack.org/615479 | 14:29 |
| openstackgerrit | Lukas Bezdicka proposed openstack/tripleo-heat-templates master: Fix Upgrade of horizon service https://review.openstack.org/617667 | 14:29 |
| *** agopi|brb is now known as agopi | 14:29 | |
| *** derekh has joined #tripleo | 14:32 | |
| *** udesale has joined #tripleo | 14:33 | |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-quickstart master: [Configuration] Remove mitaka file. https://review.openstack.org/617670 | 14:35 |
| *** amoralej|lunch is now known as amoralej | 14:37 | |
| *** mjturek has joined #tripleo | 14:37 | |
| *** ykarel has quit IRC | 14:38 | |
| *** ykarel has joined #tripleo | 14:39 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/python-tripleoclient master: Add missing py37 toxenv and corrected default envlist https://review.openstack.org/617606 | 14:39 |
| *** ykarel_ has joined #tripleo | 14:41 | |
| dciabrin | EmilienM, hey will you have time doing a bluejeans later with bandini to discuss pcmk2 in CI? | 14:43 |
| EmilienM | dciabrin: anytime, even now | 14:43 |
| *** ykarel has quit IRC | 14:44 | |
| dciabrin | EmilienM, ack url in PM | 14:44 |
| EmilienM | dciabrin: weshay might want to join or something | 14:45 |
| dciabrin | sure I'm inviting him as well | 14:45 |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-quickstart master: [Configuration] Remove mitaka file. https://review.openstack.org/617670 | 14:48 |
| *** morazi has joined #tripleo | 14:52 | |
| *** ykarel__ has joined #tripleo | 14:52 | |
| *** tbonds has quit IRC | 14:52 | |
| *** ykarel_ has quit IRC | 14:55 | |
| *** jistr is now known as jistr|mtg | 14:57 | |
| openstackgerrit | Marios Andreou proposed openstack/ansible-role-tripleo-modify-image master: Remove multinode-scenario001 from zuul.d/layout https://review.openstack.org/617636 | 14:58 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-tripleo master: Remove multinode-scenario001 from zuul.d/layout https://review.openstack.org/617637 | 14:58 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-common master: Remove multinode-scenario001 from zuul.d/layout https://review.openstack.org/617638 | 14:58 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-heat-templates master: Remove multinode-scenario001 from zuul.d/layout https://review.openstack.org/617635 | 14:58 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job https://review.openstack.org/616872 | 14:59 |
| *** Vorrtex has joined #tripleo | 15:01 | |
| *** skramaja has quit IRC | 15:01 | |
| *** jaganathan_ has quit IRC | 15:03 | |
| *** ykarel__ is now known as ykarel | 15:03 | |
| *** bnemec has joined #tripleo | 15:04 | |
| rfolco | ci community mtg starts now at https://bluejeans.com/4113567798 if anyone has anything to discuss | 15:04 |
| *** ykarel is now known as ykarel|away | 15:05 | |
| fultonj | rfolco: i'm joining now, thanks | 15:08 |
| weshay | dciabrin, bandini in ci office hours now | 15:08 |
| weshay | fyi.. | 15:08 |
| bandini | weshay: ack. we're both in an upgrade call atm | 15:09 |
| bandini | we'll likely get back to you either next week or something | 15:09 |
| panda|rover | cfontain: do you have a third camera too ? :) | 15:09 |
| *** ooolpbot has joined #tripleo | 15:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 15:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 15:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 15:10 |
| *** ooolpbot has quit IRC | 15:10 | |
| *** bnemec has quit IRC | 15:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 15:10 |
| *** matbu has joined #tripleo | 15:12 | |
| openstackgerrit | Martin Schuppert proposed openstack/tripleo-heat-templates master: Cleanup nova metadata port in nova api service https://review.openstack.org/617686 | 15:14 |
| *** cylopez has joined #tripleo | 15:16 | |
| *** saneax has quit IRC | 15:17 | |
| *** agurenko has quit IRC | 15:19 | |
| *** cfontain has quit IRC | 15:21 | |
| *** tbonds has joined #tripleo | 15:24 | |
| *** cfontain has joined #tripleo | 15:24 | |
| *** ccamacho has joined #tripleo | 15:24 | |
| cfontain | panda|rover: :-D Bluejeans has frozen my whole X server, but nothing a systemctl restart gdm can't fix. (And joining with my phone saved the day ;-) ) | 15:25 |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-quickstart master: [Configuration] Remove mitaka file. https://review.openstack.org/617670 | 15:26 |
| quiquell | mwhahaha: you there ? | 15:27 |
| mwhahaha | quiquell: sup | 15:27 |
| quiquell | mwhahaha: have a python3 poltergeist here | 15:27 |
| mwhahaha | oh noes | 15:27 |
| *** apetrich has joined #tripleo | 15:27 | |
| quiquell | mwhahaha: yep :-( | 15:27 |
| quiquell | http://logs.openstack.org/79/615479/31/check/tripleo-ci-fedora-28-standalone/338fc99/logs/undercloud/home/zuul/standalone_deploy.log.txt.gz#_2018-11-13_14_56_57 | 15:28 |
| *** matbu has quit IRC | 15:28 | |
| quiquell | mwhahaha: but python_cmd is still 2 :-( | 15:28 |
| *** matbu has joined #tripleo | 15:28 | |
| mwhahaha | quiquell: yea so i know why that is | 15:28 |
| quiquell | mwhahaha: why is that ? I am depending on the change at python-tripleoclient that fix that | 15:29 |
| mwhahaha | quiquell: so by default ansible's python interpreter is 2 even on fedora | 15:29 |
| mwhahaha | quiquell: so we need to specify/force it to python3 on fedora | 15:29 |
| mwhahaha | i was going to try and figure that out today | 15:29 |
| quiquell | mwhahaha: I am depending on this https://review.openstack.org/616579 | 15:29 |
| *** hberaud has joined #tripleo | 15:29 | |
| quiquell | mwhahaha: this force use ansible-playbook-3 | 15:29 |
| mwhahaha | quiquell: yea that doesn't change the default python interpreter that the stuff gets run i don't think | 15:29 |
| quiquell | mwhahaha: Ahhh the ansible_python_interpreter :-) | 15:30 |
| mwhahaha | yea | 15:30 |
| quiquell | mwhahaha: I forced it into the release file :-) | 15:30 |
| mwhahaha | we might get away with a quickstart hack for fedora, not sure as i haven't been able to fully think about it | 15:30 |
| quiquell | mwhahaha: Maybe we can change the standalone.sh.h2 template to pass it | 15:30 |
| mwhahaha | quiquell: it's likely that we'll want to set it in the global ansible.cfg or something for fedora | 15:31 |
| quiquell | mwhahaha: I know now I know why at least, will try to come with something | 15:31 |
| quiquell | mwhahaha: thanks | 15:31 |
| quiquell | mwhahaha: We can set it in the inventory :-/ | 15:33 |
| mwhahaha | well the problem is knowing when we're running on python3 | 15:34 |
| mwhahaha | so it's probably best to set externally if we know we're running on fedora | 15:34 |
| mwhahaha | that's kinda why i need to think about the best way to handle this | 15:35 |
| *** cylopez has quit IRC | 15:35 | |
| mwhahaha | as we used to have some backwards compatibility of undercloud issues. though not sure how that's going to work with this release | 15:35 |
| quiquell | mwhahaha: ok | 15:35 |
| *** apetrich has quit IRC | 15:35 | |
| *** psachin has quit IRC | 15:38 | |
| *** quiquell is now known as quiquell|off | 15:38 | |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Render playbooks minimal, move everything to roles https://review.openstack.org/607288 | 15:40 |
| openstackgerrit | Raildo Mascena proposed openstack/tripleo-heat-templates stable/pike: Mount the public TLS certificate for HAProxy on up(date|grade) on pacemaker https://review.openstack.org/617701 | 15:40 |
| *** rodolof has quit IRC | 15:42 | |
| *** hberaud has quit IRC | 15:43 | |
| *** salmankhan has joined #tripleo | 15:43 | |
| *** hberaud has joined #tripleo | 15:44 | |
| *** hberaud has quit IRC | 15:46 | |
| *** hberaud has joined #tripleo | 15:46 | |
| *** morazi has quit IRC | 15:49 | |
| *** ccamacho has quit IRC | 15:54 | |
| *** salmankhan has quit IRC | 15:57 | |
| *** salmankhan has joined #tripleo | 15:57 | |
| openstackgerrit | Quique Llorente proposed openstack/python-tripleoclient master: WIP: Force ansible_python_interpreter https://review.openstack.org/617716 | 15:57 |
| quiquell|off | mwhahaha: going to test this ^ | 15:57 |
| mwhahaha | k | 15:58 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: To test standalone-28 https://review.openstack.org/615479 | 15:58 |
| mwhahaha | quiquell|off: yea that should fix up the undercloud/standalone. doesn't necessarily work for the overcloud tho | 15:58 |
| *** jistr|mtg is now known as jistr | 15:59 | |
| quiquell|off | mwhahaha: That's why it's a WIP, also I don't know if using the python version for the openstack command is a good thing | 15:59 |
| quiquell|off | mwhahaha: We are suppose to have decide already | 15:59 |
| *** quiquell|off is now known as quiquell | 16:00 | |
| quiquell | mwhahaha: this is the ansible.cfg we use ? http://git.openstack.org/cgit/openstack/python-tripleoclient/tree/tripleoclient/v1/tripleo_deploy.py#n1145 | 16:00 |
| mwhahaha | quiquell: i think it's ok for the undercloud/standalone because we will have a single version of that. The issue is for the overcloud because it may be a 7 host | 16:00 |
| mwhahaha | quiquell: yea if it's not already provided | 16:01 |
| *** salmankhan has quit IRC | 16:01 | |
| quiquell | mwhahaha: Ok we can hack there too, ok nah let's just use the other to discover mor sh.. | 16:01 |
| mwhahaha | quiquell: http://git.openstack.org/cgit/openstack/python-tripleoclient/tree/tripleoclient/v1/tripleo_deploy.py#n1136 which is why i said maybe we hack in with quickstart | 16:01 |
| mwhahaha | but i think it's OK the way you've done it for the tripleo deploy command | 16:01 |
| mwhahaha | we'll just need to document that in a release note | 16:02 |
| quiquell | mwhahaha: At the end, feels like the correct thing is be explicit about it like in the release file or something | 16:02 |
| quiquell | mwhahaha: so we can have even diferent release file for differen convitaions of distro + python version | 16:03 |
| mwhahaha | quiquell: for CI that's a ok thing but for end users we should probably expose it via an option for overriding | 16:03 |
| quiquell | mwhahaha: yep | 16:03 |
| mwhahaha | defaulting to the python version that teh command ran as | 16:03 |
| quiquell | mwhahaha: ack, thanks again mate, more tomorrow | 16:03 |
| *** quiquell is now known as quiquell|off | 16:04 | |
| mwhahaha | quiquell|off: np, i'll try and throw up a patch for the configurability today | 16:04 |
| *** bnemec has joined #tripleo | 16:06 | |
| openstackgerrit | Natal Ngétal proposed openstack/tripleo-quickstart master: [Configuration] Remove mitaka file. https://review.openstack.org/617670 | 16:06 |
| *** yprokule has quit IRC | 16:06 | |
| Hobbestigrou | someone can review this patch please https://review.openstack.org/#/c/617670/ ? | 16:07 |
| Hobbestigrou | and this one also please https://review.openstack.org/#/c/616157/ ? | 16:07 |
| *** ooolpbot has joined #tripleo | 16:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 16:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 16:10 |
| *** ooolpbot has quit IRC | 16:10 | |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 16:10 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 16:10 |
| Hobbestigrou | and for the end this one please https://review.openstack.org/#/c/615906/ | 16:11 |
| *** spsurya has quit IRC | 16:12 | |
| chem | beagles: hey o/ | 16:16 |
| *** itlinux has joined #tripleo | 16:17 | |
| chem | beagles: so there is an issue with the ovs patch for the upgrade, and the standalone upgrade caught it :) | 16:17 |
| chem | beagles: -> https://bugs.launchpad.net/tripleo/+bug/1803154 | 16:17 |
| openstack | Launchpad bug 1803154 in tripleo "Upgrade fail during "remove old OpenvSwitch package"" [Critical,Triaged] | 16:17 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Render playbooks minimal, move everything to roles https://review.openstack.org/607288 | 16:18 |
| openstackgerrit | Thomas Herve proposed openstack/tripleo-common stable/rocky: Fix config-download timeout https://review.openstack.org/617727 | 16:18 |
| openstackgerrit | Jose Luis Franco proposed openstack/tripleo-common master: Fix ironic-pxe healthcheck. https://review.openstack.org/617728 | 16:20 |
| *** ksambor has quit IRC | 16:22 | |
| *** gkadam has quit IRC | 16:23 | |
| *** bnemec has quit IRC | 16:23 | |
| *** ykarel_ has joined #tripleo | 16:27 | |
| *** thrash is now known as thrash|biab | 16:27 | |
| openstackgerrit | Athlan-Guyot sofer proposed openstack/tripleo-heat-templates master: Fix ansible conditional for ovs upgrade. https://review.openstack.org/617732 | 16:28 |
| chem | beagles: did that ^ with a depends-on on the standalone-ugprade jobs, maybe that's all what is needed. | 16:30 |
| *** ykarel|away has quit IRC | 16:30 | |
| *** itlinux has quit IRC | 16:32 | |
| *** ramishra_ has quit IRC | 16:32 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/python-tripleoclient master: Avoid printing b'' across logged output https://review.openstack.org/617363 | 16:39 |
| *** irclogbot_3 has joined #tripleo | 16:41 | |
| openstackgerrit | Merged openstack/tripleo-quickstart master: Set undercloud_cloud_domain/hostname in fs023 https://review.openstack.org/617308 | 16:42 |
| *** aufi has quit IRC | 16:43 | |
| *** itlinux has joined #tripleo | 16:43 | |
| *** irclogbot_3 has quit IRC | 16:43 | |
| *** akrivoka has quit IRC | 16:47 | |
| *** akrivoka has joined #tripleo | 16:47 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates master: Python 3 compatibility: convert raw_input to input https://review.openstack.org/610365 | 16:49 |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: switch documentation job to new PTI https://review.openstack.org/615910 | 16:49 |
| dtrainor | Is there a debug setting for zaqar that's better than "debug=true"? Something that shows socket-level operations etc | 16:53 |
| *** jfrancoa has quit IRC | 16:56 | |
| *** d0ugal has quit IRC | 16:57 | |
| mwhahaha | dtrainor: probably not without custom logging rules or something | 16:58 |
| dtrainor | custom logging rules in zaqar? | 16:58 |
| mwhahaha | in pytyhon | 16:58 |
| mwhahaha | ie hacking it :D | 16:58 |
| *** paramite has quit IRC | 16:58 | |
| weshay | bfournie, jrist https://bugzilla.redhat.com/show_bug.cgi?id=1642466 | 16:58 |
| openstack | bugzilla.redhat.com bug 1642466 in python-virtualbmc "Drop vbmc from OSP-14" [High,Modified] - Assigned to rhos-maint | 16:58 |
| dtrainor | oh right | 16:58 |
| weshay | bfournie, any idea if the package is being removed from rdo? | 16:59 |
| dtrainor | I'm troubleshooting an issue where mod_proxy in Apache throws a 502 and passes that to the client, and UI gets mad about it. It's transient. I see nothing but clean shutdowns from Zaqar. The UI tries to use or reuse at WebSocket connection that's been closed but mod_proxy doesn't realize it yet. | 17:00 |
| dtrainor | The resulting error is: AH01084: pass request body failed to 192.168.24.3:9000 (192.168.24.3) | 17:00 |
| dtrainor | It seems that Zaqar closes the connection to the client after it sends a message | 17:01 |
| jrist | weshay: thx | 17:01 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Render playbooks minimal, move everything to roles https://review.openstack.org/607288 | 17:01 |
| mwhahaha | dtrainor: where do we front zaqar witha apche? is that the ssl proxy stuff? | 17:07 |
| EmilienM | weshay, panda|rover: can I change topic to green? | 17:07 |
| EmilienM | things are merging | 17:07 |
| dtrainor | yes, it is | 17:07 |
| panda|rover | mmmh tempest failed on a standalone job in gate | 17:07 |
| panda|rover | http://logs.openstack.org/59/615959/6/gate/tripleo-ci-centos-7-standalone/1e39a59/job-output.txt.gz | 17:08 |
| mwhahaha | dtrainor: do we use mod_proxy or mod_proxy_wstunnel:? | 17:08 |
| weshay | panda|rover, there's an alert on that | 17:08 |
| * weshay looks | 17:08 | |
| panda|rover | EmilienM: the queue is still clogged | 17:08 |
| mwhahaha | tempest is never flakey | 17:08 |
| panda|rover | we're at 20h | 17:08 |
| mwhahaha | must be tripleo | 17:08 |
| * mwhahaha shows himself out | 17:08 | |
| weshay | lolz | 17:08 |
| dtrainor | mwhahaha, mod_proxy_wstunnel when the scheme for the backend is ws:// or wss:// | 17:08 |
| dtrainor | *automagically | 17:08 |
| weshay | oh wait.. did you guys see a guy walk by? Kind of looks like a goat | 17:09 |
| EmilienM | yeah its tempest | 17:09 |
| EmilienM | http://logs.openstack.org/59/615959/6/gate/tripleo-ci-centos-7-standalone/1e39a59/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-11-13_14_57_56 | 17:09 |
| weshay | bad attitue, but very handsome | 17:09 |
| EmilienM | 2018-11-13 14:57:56 | tempest.lib.exceptions.SSHTimeout: Connection to the 192.168.24.103 via SSH timed out. | 17:09 |
| EmilienM | weshay: me? | 17:09 |
| mwhahaha | smashing good looks i must say | 17:09 |
| weshay | lolz | 17:09 |
| dtrainor | haha | 17:09 |
| weshay | of course the french guy thinks we're talking about him | 17:09 |
| *** ooolpbot has joined #tripleo | 17:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 17:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 17:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 17:10 |
| *** ooolpbot has quit IRC | 17:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 17:10 |
| mwhahaha | dtrainor: you'd probably have better luck turning on apache debug | 17:10 |
| dtrainor | i've been all over that. even using mod_dumpio (headache) i can see every input and output. The error through occurs outside of dumpio though: [proxy:error] [pid 523] (32)Broken pipe: [client 192.168.24.1:47986] AH01084: pass request body failed to 192.168.24.3:9000 (192.168.24.3) | 17:11 |
| dtrainor | then following up with: [proxy:debug] [pid 523] proxy_util.c(2218): AH00943: WS: has released connection for (192.168.24.3) | 17:12 |
| mwhahaha | dtrainor: connection timeout mismatch? | 17:12 |
| *** agopi is now known as agopi|food | 17:12 | |
| mwhahaha | or something to that eefect? | 17:12 |
| mwhahaha | many my typing is terrible today | 17:12 |
| dtrainor | i was hoping it was that easy but I cannot find a value that would represent a timeout or a ttl on the zaqar side | 17:12 |
| mwhahaha | effect | 17:12 |
| dtrainor | many mine is today, too | 17:12 |
| weshay | mwhahaha, EmilienM usefulness of a keystone only standalone as a reference? | 17:13 |
| weshay | imho worth doing it | 17:13 |
| bfournie | weshay: that's what I understood, I'll check with jjoyce | 17:13 |
| weshay | bfournie, k | 17:13 |
| dtrainor | mwhahaha, the only timeout-specific things I see mentioned about zaqar are of rpc parameters | 17:14 |
| mwhahaha | dtrainor: that's probably a question for people who actually touched zaqar (therve maybe?) | 17:14 |
| dtrainor | yes, he's been a great resource | 17:15 |
| openstackgerrit | Quique Llorente proposed openstack-infra/tripleo-ci master: DNM: To test standalone-28 https://review.openstack.org/615479 | 17:17 |
| bfournie | weshay: yeah its just removed from osp-14 | 17:17 |
| *** dsneddon has quit IRC | 17:18 | |
| mwhahaha | EmilienM: soooo python3 + ansible. Any thoughts on the best way to use the python3 interpreter for overcloud deployments? By default i think it uses python2 and we have to manually specify 3. | 17:18 |
| EmilienM | mwhahaha: create a stupid wrapper | 17:18 |
| mwhahaha | EmilienM: The problem is that for config download that's all in the tripleo-common inventory so i'm not sure the best way to specify it, got any thoughts on where we should keep that information | 17:18 |
| mwhahaha | EmilienM: for tripleo deploy it's ez cause we can add a cmd line param and it's just a single host. but for overcloud would we possibly have multiple version in a single overcloud? | 17:19 |
| *** jpich has quit IRC | 17:19 | |
| *** thrash|biab is now known as thrash | 17:20 | |
| mwhahaha | i don't really want to store it in heat somehow but i'm not sure we have other options | 17:20 |
| mwhahaha | :/ | 17:20 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart master: Add featureset060 for standalone-scenario001 job https://review.openstack.org/617754 | 17:20 |
| openstackgerrit | Marios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job https://review.openstack.org/616872 | 17:20 |
| EmilienM | mwhahaha: well | 17:21 |
| *** itlinux has quit IRC | 17:21 | |
| EmilienM | mwhahaha: like EnablePython3 in THT and do what needs to be done before running config-downlaod steps? | 17:21 |
| EmilienM | I don't see anything else now | 17:21 |
| panda|rover | gate is going to reset again. timeout in updates job | 17:22 |
| mwhahaha | meh | 17:22 |
| EmilienM | yeah just reminder: CI Status: Do not RECHECK or APPROVE any patch now. | 17:22 |
| EmilienM | mwhahaha: since it's for overcloud, I think our only interface is heat | 17:23 |
| EmilienM | if it was undercloud we could have put that in undercloud.conf | 17:23 |
| EmilienM | and hack in tripleoclient | 17:23 |
| therve | dtrainor: Can you show the apache config? | 17:23 |
| mwhahaha | for undercloud we control our destiny better | 17:23 |
| mwhahaha | EmilienM: so undercloud just uses whatever version the openstack client is, but the overcloud can have a mix | 17:23 |
| openstackgerrit | Marios Andreou proposed openstack/tripleo-quickstart master: Add featureset060 for standalone-scenario001 job https://review.openstack.org/616871 | 17:24 |
| mwhahaha | which is problemmattic | 17:24 |
| dtrainor | therve, http://paste.openstack.org/show/734774/ | 17:24 |
| dtrainor | i just added the keepalive, i'm trying with disablereuse shortly | 17:24 |
| dtrainor | at least that would clean up the proxy backend requests after use | 17:24 |
| *** jpich has joined #tripleo | 17:25 | |
| *** udesale has quit IRC | 17:25 | |
| therve | dtrainor: Do you manage to reliably reproduce? | 17:26 |
| dtrainor | nope, it's transient | 17:26 |
| dtrainor | which is what is making this particularly difficult to troubleshoot | 17:26 |
| dtrainor | half tempted to try it as a balancemember using the same backend.... at least then apache wouldn't burp with the request, but instead just effectively re-try the connection to the same backend | 17:28 |
| dtrainor | the only means of "probably" reproducing this is to do a lot of UI activity, usually that just comes down to a full refresh | 17:29 |
| dtrainor | i think it ultimately comes down to what mwhahaha suggested, that there's a timeout mismatch. i see no means of configuring client request or persistence timeouts in zaqar, and that would seem like a logical solution | 17:31 |
| therve | dtrainor: Does that happen when an operation is in progress? | 17:32 |
| therve | Or just random browsing? | 17:32 |
| dtrainor | I'm testing with disablereuse=on and the results so far "seem" promising because I'm not reproducing the 502 after all these refreshes where it would "probably" happen. I'll revert and test some more, and measure performance impact (so far, I can tell very little impact) | 17:32 |
| dtrainor | there's no correlation, it happens at random | 17:32 |
| dtrainor | i've tested while data is in transit or an operation is in progress, and when the connection is idle - seems to not make a difference in reproducing this | 17:34 |
| dtrainor | about 50 refreshes later I haven't been able to reproduce this after setting disablereuse=on, when I "probably should" have seen it by nw | 17:35 |
| dtrainor | this frown is turning upside down | 17:35 |
| *** ykarel_ is now known as ykarel|pto | 17:36 | |
| therve | dtrainor: You could try upgrading autobahn | 17:38 |
| therve | We use a 3 years old version for reasons | 17:38 |
| dtrainor | heh | 17:38 |
| dtrainor | this is downstream rocky btw | 17:39 |
| dtrainor | which ships with python-autobahn-0.10.9-1.gitcf10233.el7ost.noarch | 17:39 |
| *** jtomasek has joined #tripleo | 17:41 | |
| *** apetrich has joined #tripleo | 17:41 | |
| therve | Yeah that's awful :/ | 17:41 |
| therve | dtrainor: http://paste.openstack.org/show/734777/ maybe give you more debug info | 17:43 |
| therve | dtrainor: http://paste.openstack.org/show/734778/ too | 17:45 |
| dtrainor | appreciate it, thanks | 17:45 |
| *** Vorrtex has quit IRC | 17:46 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: WIP - Implement tripleo-systemd-wrapper role https://review.openstack.org/617377 | 17:46 |
| *** jpich has quit IRC | 17:46 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: WIP - Implement tripleo-systemd-wrapper role https://review.openstack.org/617377 | 17:47 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Switch Neutron DHCP dnsmasq to use SystemD wrapper https://review.openstack.org/617393 | 17:47 |
| *** jtomasek has quit IRC | 17:51 | |
| *** eck`gone is now known as eck` | 17:53 | |
| openstackgerrit | Martin André proposed openstack/tripleo-heat-templates master: Call etcd scaleup playbook when adding master nodes https://review.openstack.org/616584 | 17:54 |
| *** trown is now known as trown|lunch | 17:56 | |
| dtrainor | therve, i'm happy with disablereuse=on right now. there should be a note to update autobahn for reasons. doing some final performance tests and then i'll put up a patch | 17:57 |
| *** amoralej is now known as amoralej|of | 17:59 | |
| *** amoralej|of is now known as amoralej|off | 17:59 | |
| *** ykarel|pto has quit IRC | 17:59 | |
| *** derekh has quit IRC | 18:00 | |
| *** vinaykns has quit IRC | 18:03 | |
| dtrainor | thanks again for your help | 18:05 |
| *** jpena is now known as jpena|off | 18:07 | |
| *** slaweq_ has joined #tripleo | 18:09 | |
| *** toure is now known as toure|food | 18:10 | |
| *** ooolpbot has joined #tripleo | 18:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 18:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 18:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 18:10 |
| *** ooolpbot has quit IRC | 18:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 18:10 |
| *** slaweq has quit IRC | 18:10 | |
| *** Chaserjim has joined #tripleo | 18:10 | |
| *** apetrich has quit IRC | 18:11 | |
| *** apetrich has joined #tripleo | 18:11 | |
| *** panda|rover is now known as panda|off | 18:11 | |
| Chaserjim | Hello, Im curious if anyone else is encountering the same problem I am regarding scale-in and scale-out of overcloud nodes? https://bugs.launchpad.net/tripleo/+bug/1802969 | 18:13 |
| openstack | Launchpad bug 1802969 in tripleo "openstack overcloud node delete times out, even if the stack update operation finished. YAQL Expression errors in mistral/engine.log" [Undecided,New] | 18:13 |
| openstackgerrit | Gabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout https://review.openstack.org/617617 | 18:15 |
| openstackgerrit | Harald Jensås proposed openstack/tripleo-common master: Install ironic-staging-drivers in ironic-conductor https://review.openstack.org/617538 | 18:28 |
| mwhahaha | Chaserjim: there was a bug on that | 18:45 |
| mwhahaha | Chaserjim: fixed by https://review.openstack.org/#/c/609705/ | 18:46 |
| *** toure|food is now known as toure | 18:51 | |
| Chaserjim | @mwhahaha I have that patch set in my code base - im still encountering the problem listed in the bug | 18:53 |
| mwhahaha | Chaserjim: but is it in the workflow in mistral | 18:54 |
| mwhahaha | Chaserjim: if you pulled the code down but didn't reload the workbooks it doesn't take effect | 18:54 |
| *** agopi|food is now known as agopi | 19:09 | |
| *** ooolpbot has joined #tripleo | 19:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 19:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 19:10 |
| *** ooolpbot has quit IRC | 19:10 | |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 19:10 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 19:10 |
| *** akrivoka has quit IRC | 19:11 | |
| *** irclogbot_3 has joined #tripleo | 19:12 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Add advanced deployment options https://review.openstack.org/617793 | 19:15 |
| *** srini_ has joined #tripleo | 19:22 | |
| *** hkominos_ has quit IRC | 19:24 | |
| *** trown|lunch is now known as trown | 19:25 | |
| EmilienM | mwhahaha: good approach on https://review.openstack.org/#/c/617793/ | 19:30 |
| mwhahaha | the fun part is the tripleoclient code | 19:31 |
| mwhahaha | for some definition of fun that i am not yet familiar with | 19:31 |
| Tengu | at least we do have fun with tripleo and its close friends ;) | 19:32 |
| Tengu | life would be less interesting. | 19:32 |
| weshay | mwhahaha, oooh | 19:33 |
| weshay | mwhahaha, is that completely unrelated to what mistral would use for ansible? | 19:34 |
| mwhahaha | weshay: not completely | 19:34 |
| weshay | k k | 19:34 |
| * weshay reads the blueprint | 19:34 | |
| mwhahaha | weshay: but it is related to which version of python the ansible run by mistral would use | 19:34 |
| mwhahaha | so being able to specify that we should run in python3 on a python3 host | 19:35 |
| mwhahaha | or vice versa | 19:35 |
| mwhahaha | right now fedora28 the default invocation of anusble against it will use python2 | 19:35 |
| mwhahaha | so you have to pass in the python interpreter to use as python3 if you want python3 | 19:35 |
| weshay | mwhahaha, well + ur workaround | 19:35 |
| mwhahaha | so this allows the deployment to be able tos pecify | 19:35 |
| mwhahaha | so once i get the client code, we could control it from the deployment itself | 19:35 |
| weshay | good k | 19:36 |
| *** apetrich has quit IRC | 19:36 | |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-heat-templates master: upgrade: remove Docker containers now managed by Podman https://review.openstack.org/615209 | 19:44 |
| *** apetrich has joined #tripleo | 19:49 | |
| *** srini_ has quit IRC | 19:51 | |
| openstackgerrit | Merged openstack/tripleo-heat-templates stable/rocky: Sets ODL OVSDB inactivity probe timer https://review.openstack.org/615376 | 20:00 |
| *** dsneddon has joined #tripleo | 20:05 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-validations master: Add ansible_python_interpreter to the inventory script https://review.openstack.org/617821 | 20:06 |
| *** dsneddon has quit IRC | 20:10 | |
| *** irclogbot_3 has quit IRC | 20:10 | |
| *** ooolpbot has joined #tripleo | 20:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 20:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 20:10 |
| *** ooolpbot has quit IRC | 20:10 | |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 20:10 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 20:10 |
| *** irclogbot_3 has joined #tripleo | 20:14 | |
| *** radeks has quit IRC | 20:16 | |
| *** tbonds has quit IRC | 20:16 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Allow ansible python interpreter be configurable https://review.openstack.org/617716 | 20:32 |
| openstackgerrit | Merged openstack/tripleo-common stable/rocky: Delete old tarball from config container on download https://review.openstack.org/615713 | 20:36 |
| openstackgerrit | Michele Baldessari proposed openstack/puppet-tripleo master: WIP Allow the container backend to be configurable https://review.openstack.org/617832 | 20:37 |
| openstackgerrit | Michele Baldessari proposed openstack/tripleo-heat-templates master: WIP Pass the container backend explicitely https://review.openstack.org/617833 | 20:38 |
| *** hjensas has quit IRC | 20:40 | |
| *** ssbarnea has quit IRC | 20:42 | |
| *** ssbarnea|bkp2 has joined #tripleo | 20:43 | |
| *** tbonds has joined #tripleo | 20:46 | |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Allow ansible python interpreter be configurable https://review.openstack.org/617716 | 20:49 |
| *** ooolpbot has joined #tripleo | 21:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 21:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 21:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 21:10 |
| *** ooolpbot has quit IRC | 21:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 21:10 |
| *** raildo has quit IRC | 21:14 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Add advanced deployment options https://review.openstack.org/617793 | 21:21 |
| *** zaneb has joined #tripleo | 21:23 | |
| *** d0ugal has joined #tripleo | 21:23 | |
| *** d0ugal has quit IRC | 21:24 | |
| *** d0ugal has joined #tripleo | 21:24 | |
| *** lblanchard has quit IRC | 21:24 | |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove configs parameter from TripleoInventory https://review.openstack.org/617842 | 21:25 |
| *** Chaserjim has quit IRC | 21:32 | |
| *** zaneb has quit IRC | 21:43 | |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/puppet-tripleo master: Add a connector to Qpid router https://review.openstack.org/611979 | 21:46 |
| *** hjensas has joined #tripleo | 21:47 | |
| *** jtomasek has joined #tripleo | 21:49 | |
| *** jtomasek has quit IRC | 21:50 | |
| *** pcaruana has quit IRC | 21:56 | |
| openstackgerrit | Nagasai Vinaykumar Kapalavai proposed openstack/tripleo-heat-templates master: Update to the ceilometer publisher list https://review.openstack.org/600843 | 21:58 |
| *** vpickard is now known as vpickard_ | 22:00 | |
| *** abishop has quit IRC | 22:06 | |
| *** matbu has quit IRC | 22:09 | |
| *** lyarwood has quit IRC | 22:09 | |
| *** panda|off has quit IRC | 22:09 | |
| *** matbu has joined #tripleo | 22:09 | |
| *** etingof has quit IRC | 22:09 | |
| *** etingof has joined #tripleo | 22:10 | |
| *** ooolpbot has joined #tripleo | 22:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 22:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 22:10 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 22:10 |
| *** ooolpbot has quit IRC | 22:10 | |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 22:10 |
| EmilienM | weshay: ping | 22:10 |
| *** panda has joined #tripleo | 22:10 | |
| weshay | EmilienM, ahoy | 22:10 |
| *** dtrainor has quit IRC | 22:11 | |
| EmilienM | weshay: do we know what makes the gate timeouting? | 22:11 |
| EmilienM | is it tempest? | 22:11 |
| *** trown is now known as trown|outtypewww | 22:11 | |
| *** dtrainor has joined #tripleo | 22:11 | |
| EmilienM | I think we should purge the gate | 22:11 |
| EmilienM | nothing on master is landing | 22:11 |
| weshay | EmilienM, that is reseting the gate w/ the standalone job ya | 22:11 |
| EmilienM | is it https://bugs.launchpad.net/tripleo/+bug/1802971 ? | 22:11 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 22:11 |
| weshay | EmilienM, see the alert bug | 22:12 |
| weshay | EmilienM, kind of wondering if using ovn will fix it | 22:12 |
| weshay | russel's patch | 22:12 |
| weshay | but there is an open bug on neutron | 22:12 |
| EmilienM | why would it help | 22:12 |
| EmilienM | it's adding an environment file | 22:12 |
| EmilienM | that isn't use anywhere | 22:12 |
| EmilienM | ... | 22:12 |
| *** mcornea has quit IRC | 22:13 | |
| weshay | EmilienM, oh that's true the env option hasn't merged | 22:13 |
| EmilienM | there is no other clue?? | 22:13 |
| EmilienM | which env option | 22:14 |
| EmilienM | link plz | 22:14 |
| weshay | getting | 22:14 |
| weshay | https://review.openstack.org/#/c/617022/ | 22:14 |
| weshay | there is russel's | 22:14 |
| weshay | and | 22:14 |
| *** boazel has joined #tripleo | 22:14 | |
| EmilienM | ? | 22:15 |
| EmilienM | http://logs.openstack.org/03/616203/9/gate/tripleo-ci-centos-7-containers-multinode/099c730/job-output.txt.gz#_2018-11-13_21_08_31_970863 | 22:16 |
| EmilienM | there are some mirror issues | 22:16 |
| weshay | https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/standalone/templates/standalone.sh.j2#L9 | 22:16 |
| EmilienM | i'll report that on infra | 22:16 |
| *** rlandy is now known as rlandy|brb | 22:16 | |
| weshay | EmilienM, only a few of those | 22:16 |
| EmilienM | weshay: what about this lone? | 22:16 |
| *** florianf|summit is now known as florianf|afk | 22:16 | |
| weshay | I saw two maybe 3 | 22:16 |
| EmilienM | I don't understand your links and standalone thing | 22:17 |
| EmilienM | Russel's patch has nothing to do with the timeouts | 22:17 |
| weshay | EmilienM, so my question is would using ovn vs.. ovs possibly clear up the neutron issue desribed in https://bugs.launchpad.net/tripleo/+bug/1802971 | 22:17 |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 22:17 |
| *** boazel_ has quit IRC | 22:17 | |
| EmilienM | I'm not sure we want to do that now | 22:17 |
| weshay | EmilienM, It's a ssh timeout mind you.. not a job timeout.. you probably know that | 22:17 |
| EmilienM | weshay: let's go on bj | 22:17 |
| weshay | EmilienM, either do I.. I've seen other tempest | 22:17 |
| weshay | bah | 22:17 |
| weshay | k | 22:17 |
| *** leifmadsen has quit IRC | 22:19 | |
| *** vinaykns has joined #tripleo | 22:22 | |
| *** leifmadsen has joined #tripleo | 22:24 | |
| *** slaweq_ has quit IRC | 22:25 | |
| EmilienM | mwhahaha: we still have scenarios voting in gate for tripleo-ci | 22:25 |
| EmilienM | http://zuul.openstack.org/build/9b7580d409874724b0502ba98ca73537 | 22:25 |
| *** slaweq has joined #tripleo | 22:25 | |
| mwhahaha | it was non-voting | 22:26 |
| EmilienM | http://zuul.openstack.org/builds?result=FAILURE&pipeline=gate | 22:26 |
| mwhahaha | https://review.openstack.org/#/c/604706/ | 22:26 |
| EmilienM | I'm trying to look why gate resets | 22:26 |
| mwhahaha | timeouts on upgrades and containers-multinode is what i've seen | 22:27 |
| EmilienM | http://logs.openstack.org/01/611801/13/gate/tripleo-ci-centos-7-scenario000-multinode-oooq-container-updates/0a261bd/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-13_09_58_17 | 22:27 |
| EmilienM | so one job failed because RAX was down, | 22:27 |
| mwhahaha | EmilienM: so that tripleo-ci job had scenarios in the gate because it needs to be rebased | 22:28 |
| EmilienM | on stable: http://logs.openstack.org/76/615776/1/gate/tripleo-ci-centos-7-scenario002-multinode-oooq-container/e5c59ff/logs/undercloud/home/zuul/tempest.log.txt.gz#_2018-11-13_16_32_03 | 22:29 |
| EmilienM | issue with temepst Swift | 22:29 |
| *** vkapalav has joined #tripleo | 22:29 | |
| EmilienM | another infra issue: http://logs.openstack.org/01/611801/13/gate/tripleo-ci-centos-7-containers-multinode/e555a23/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-13_10_16_48 | 22:30 |
| EmilienM | http://mirror.regionone.limestone.openstack.org:8080 404 issue | 22:30 |
| *** vinaykns has quit IRC | 22:31 | |
| weshay | EmilienM, http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1 | 22:31 |
| weshay | EmilienM, http://dashboard-ci.tripleo.org/d/cEEjGFFmz/cockpit?orgId=1&panelId=61&fullscreen | 22:31 |
| *** toure is now known as toure|gone | 22:33 | |
| EmilienM | DEBUG: 17440 -- Error while pulling image: Get http://192.168.24.1:8787/v1/repositories/tripleomaster/centos-binary-neutron-server/images: dial tcp 192.168.24.1:8787: getsockopt: no route to host", | 22:33 |
| EmilienM | http://logs.openstack.org/06/604706/32/gate/tripleo-ci-centos-7-scenario004-multinode-oooq-container/4d25e70/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz#_2018-11-13_09_00_03 | 22:33 |
| EmilienM | this one isn't good too | 22:33 |
| mwhahaha | that was the puppet5 issue | 22:33 |
| EmilienM | happening on overcloud jobs on master | 22:33 |
| EmilienM | ah ok | 22:33 |
| mwhahaha | iptables rules | 22:34 |
| EmilienM | k sorry* | 22:34 |
| *** sshnaidm is now known as sshnaidm|pto | 22:34 | |
| EmilienM | mwhahaha: I'm purging the gate | 22:35 |
| EmilienM | and try to isolate the issues | 22:35 |
| mwhahaha | why | 22:35 |
| mwhahaha | there's no point | 22:35 |
| mwhahaha | it's just going to make things worse | 22:35 |
| EmilienM | but it doesn't make sense to make the gate busy | 22:35 |
| EmilienM | we are wasting resources | 22:35 |
| mwhahaha | check will continue to waste the resources | 22:35 |
| mwhahaha | we need to identify the bug before purging the gate | 22:36 |
| EmilienM | still less than having gate | 22:36 |
| weshay | two known issues | 22:36 |
| mwhahaha | otherwise it'll be impossible to identify in check | 22:36 |
| weshay | some pip bs.. | 22:36 |
| EmilienM | when is the last thing that landed on master | 22:36 |
| weshay | and the volumebootpattern in standalone | 22:36 |
| EmilienM | it was https://review.openstack.org/610365 5 hours ago | 22:37 |
| EmilienM | so in 5h we kept timeouting | 22:37 |
| *** artom has quit IRC | 22:37 | |
| mwhahaha | so what's the solution for volumeboot pattern? | 22:37 |
| mwhahaha | should we just ignore the test? | 22:37 |
| weshay | no | 22:38 |
| mwhahaha | why | 22:38 |
| EmilienM | yes | 22:38 |
| EmilienM | let's ignore | 22:38 |
| weshay | we should get some neutron guys over here | 22:38 |
| EmilienM | http://logs.openstack.org/03/616203/9/gate/tripleo-ci-centos-7-standalone/1d53ba4/logs/undercloud/home/zuul/tempest.log.txt.gz | 22:38 |
| EmilienM | these are logs from a job that just failed in gate ^ | 22:38 |
| EmilienM | let's ignore that test | 22:38 |
| weshay | IT"S NOT FAILING OFTEN ENOUGH TO SKIP | 22:38 |
| mwhahaha | what is the criteria | 22:38 |
| mwhahaha | also is this a new test? | 22:39 |
| weshay | mwhahaha, http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone&pipeline=gate | 22:39 |
| weshay | EmilienM, http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-standalone&pipeline=gate | 22:39 |
| EmilienM | ianw: ^ situation is messy | 22:39 |
| mwhahaha | weshay: i would say we should revert, https://review.openstack.org/#/c/615133/ | 22:40 |
| weshay | ianw, we need a better package gate | 22:40 |
| weshay | puppet 5 blew it up | 22:40 |
| mwhahaha | as it introduced this less stable test | 22:40 |
| mwhahaha | that we did not previously have | 22:41 |
| mwhahaha | so you could either ignore that one test, or we could revert it all | 22:41 |
| EmilienM | yes | 22:41 |
| EmilienM | let's purge the gate | 22:41 |
| mwhahaha | this is why i didn't really want all these tests on this job | 22:41 |
| EmilienM | and try a revert of this one | 22:41 |
| weshay | mwhahaha, I think it found a real fucking bug | 22:41 |
| weshay | the neutron guys are looking at it now | 22:41 |
| mwhahaha | ok that's nice and how long to get a fix+promotion? | 22:41 |
| weshay | mwhahaha, get on emiliens call | 22:41 |
| mwhahaha | for us if we find the solution, revert | 22:41 |
| weshay | mwhahaha, EmilienM we have a way to skip a tempest test when needed | 22:41 |
| mwhahaha | well it's needed but you said no | 22:42 |
| mwhahaha | w/o criteria | 22:42 |
| weshay | mwhahaha, https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/vars/tempest_skip_master.yml | 22:42 |
| mwhahaha | so IMHO, if a patch introduced a performance issue we revert until fix lands then you can revert revert | 22:42 |
| weshay | if we want to skip that test.. let's do it where it's tracked | 22:42 |
| weshay | AHHH | 22:42 |
| weshay | it was working | 22:43 |
| mwhahaha | until | 22:43 |
| mwhahaha | ...? | 22:43 |
| *** redrobot has joined #tripleo | 22:44 | |
| *** vkapalav has quit IRC | 22:45 | |
| EmilienM | ianw: do you have the ability to promote a patch in the gate? | 22:46 |
| *** rlandy|brb is now known as rlandy | 22:47 | |
| ianw | EmilienM: yep | 22:48 |
| openstackgerrit | wes hayutin proposed openstack/tripleo-quickstart-extras master: remove volumebootpattern from master https://review.openstack.org/617845 | 22:49 |
| EmilienM | ianw: https://review.openstack.org/#/c/617845/ please | 22:50 |
| EmilienM | ianw: if we can merge it directly it would be awesome otherwise just promote to the gate | 22:50 |
| EmilienM | thanks a lot | 22:50 |
| ianw | EmilienM: we can merge it directly ... generally just very shy about doing that because we can end up in a loop of needing more changes to force merge to fix the last force merge :) | 22:52 |
| ianw | fungi: ^ what do you think on this one? as it's just skipping a test right, looks fairly straight forward | 22:52 |
| EmilienM | ianw: in the meantime maybe we can move it to gate | 22:53 |
| ianw | EmilienM: yep, I'm comfortable with putting at top of gate, just bringing up windows .... | 22:54 |
| EmilienM | it would save us hours to just land it | 22:55 |
| EmilienM | but if it's risky well | 22:55 |
| openstackgerrit | Alex Schultz proposed openstack/python-tripleoclient master: Allow ansible python interpreter be configurable https://review.openstack.org/617716 | 22:57 |
| EmilienM | ianw: thx | 22:58 |
| ianw | EmilienM: 617845,1 at top of gate queue now ... | 22:58 |
| EmilienM | weshay: ^ | 22:58 |
| EmilienM | hopefully it lands and stabilize a bit | 22:58 |
| EmilienM | mwhahaha, weshay I'm sending an email | 22:59 |
| ianw | i mean, i can force merge, i guess if something does go wrong it really only holds up the triple-o gate, so it's not like i've broken everything :) i am around all day to help with any further fixes, but is it getting late for you to monitor the gate after we do it? | 22:59 |
| *** mburned_out is now known as mburned | 23:00 | |
| ianw | i just don't want to commit something that somehow breaks the gate even further and then we're even in a worse position and i don't know really how the guts of how to debug it | 23:00 |
| fungi | ianw: EmilienM: 617845 looks like it _should_ be safe enough | 23:01 |
| EmilienM | up to you folks I'm fine either way | 23:01 |
| mwhahaha | it's fine to let it ride in the gate | 23:01 |
| EmilienM | I actually want to see if we hit other problems in gate now | 23:01 |
| EmilienM | I'll be back here after dinner | 23:01 |
| ianw | ok, well ping at any point :) | 23:04 |
| openstackgerrit | Emilien Macchi proposed openstack/tripleo-common master: WIP - Implement tripleo-systemd-wrapper role https://review.openstack.org/617377 | 23:04 |
| EmilienM | ianw: ack, thx again | 23:05 |
| *** ooolpbot has joined #tripleo | 23:10 | |
| ooolpbot | URGENT TRIPLEO TASKS NEED ATTENTION | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1802971 | 23:10 |
| ooolpbot | https://bugs.launchpad.net/tripleo/+bug/1803024 | 23:10 |
| *** ooolpbot has quit IRC | 23:10 | |
| openstack | Launchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged] | 23:10 |
| openstack | Launchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged] | 23:10 |
| openstackgerrit | Alex Schultz proposed openstack/tripleo-common master: Remove configs parameter from TripleoInventory https://review.openstack.org/617842 | 23:11 |
| mwhahaha | heh failed in the gate | 23:21 |
| mwhahaha | EmilienM, weshay: screwed by new podman http://logs.openstack.org/45/617845/1/gate/tripleo-ci-centos-7-containers-multinode/bf2e095/logs/undercloud/home/zuul/install_packages.sh.log.txt.gz#_2018-11-13_23_14_47 | 23:21 |
| *** florianf|afk has quit IRC | 23:23 | |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Modify the zuul inventory to pass to the reproducer https://review.openstack.org/617401 | 23:25 |
| openstackgerrit | Ronelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Copy the zuul inventory to run the reproducer https://review.openstack.org/616637 | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!