Wednesday, 2018-11-14

openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates master: GATE CHECK for TripleO  https://review.openstack.org/60429800:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/rocky: GATE CHECK for TripleO  https://review.openstack.org/60429300:00
*** aedc has quit IRC00:02
*** ooolpbot has joined #tripleo00:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION00:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297100:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302400:10
*** ooolpbot has quit IRC00:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]00:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]00:10
*** tosky has quit IRC00:55
EmilienMmwhahaha: I tested it00:58
EmilienMlike twice00:58
EmilienMtoday00:58
EmilienM(well I tested it from source)00:58
mwhahahaEmilienM: it's conflicting with h docker package00:59
EmilienMok some rpm shanigans00:59
mwhahahaYes00:59
EmilienMfuck00:59
mwhahahaWe must gate the podman stuff00:59
EmilienMhow00:59
mwhahahaThis is terrible00:59
EmilienMI'm filling a launchpad01:01
mwhahahaWe need to figure out a gatable dependency method01:02
mwhahahaBetween this and ceph ansible we keep getting hit with stuff01:02
mwhahahaBe cause we have no visibility01:03
EmilienMhttps://bugs.launchpad.net/tripleo/+bug/180323201:03
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docer" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)01:03
mwhahahaDlrn deps too but that one isn't automatic01:03
EmilienMweshay: ^ https://bugs.launchpad.net/tripleo/+bug/1803232 fyi01:04
EmilienMI'm sending an email to the maintener and proposing to disable podman01:04
EmilienMianw: will need your help01:04
EmilienMianw: we need to force merge https://review.openstack.org/#/c/617845/01:05
*** mjturek has quit IRC01:05
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "(squash) disabling podman everywhere in gate""  https://review.openstack.org/61785201:05
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "(squash) disabling podman everywhere in gate""  https://review.openstack.org/61785201:05
EmilienMianw: and promote^ in the gate01:05
EmilienMianw: even force merge if you don't mind01:07
EmilienMmwhahaha: email sent, revert proposed01:07
*** phuongnh has joined #tripleo01:09
*** ooolpbot has joined #tripleo01:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297101:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]01:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302401:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323201:10
*** ooolpbot has quit IRC01:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]01:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)01:10
openstackgerritMerged openstack/tripleo-quickstart master: Revert "Revert "(squash) disabling podman everywhere in gate""  https://review.openstack.org/61785201:17
openstackgerritMerged openstack/tripleo-quickstart-extras master: remove volumebootpattern from master  https://review.openstack.org/61784501:18
EmilienM^01:20
EmilienMmwhahaha, weshay ^ podman disabled everywhere and the tempest test is now skipped01:20
mwhahahaK01:21
*** rh-jelabarre has quit IRC01:21
EmilienMI'm purging the gate01:21
EmilienMand rechecking some01:22
EmilienMit's all red now01:22
EmilienMwell it'll reset itself01:22
* EmilienM off01:22
EmilienMI also removed the alert on https://bugs.launchpad.net/tripleo/+bug/180323201:25
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)01:25
EmilienMsince we're not deploying podman anymore01:25
*** korean102 has joined #tripleo01:41
korean102Tengu: HI01:41
*** tbonds has quit IRC01:42
*** dhill_ has quit IRC01:53
*** chem has quit IRC01:56
*** chem has joined #tripleo01:57
*** mburned is now known as mburned_out02:02
*** mburned_out is now known as mburned02:02
*** ooolpbot has joined #tripleo02:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297102:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]02:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302402:10
*** ooolpbot has quit IRC02:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]02:10
openstackgerritMerged openstack/python-tripleoclient stable/rocky: Only return plans if available from the message  https://review.openstack.org/61735102:11
openstackgerritMerged openstack/tripleo-upgrade stable/rocky: Add container image prepare script to upgrade workflow.  https://review.openstack.org/61643702:15
openstackgerritMerged openstack/tripleo-validations master: Add python 3.6 unit test job  https://review.openstack.org/61674002:15
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785502:26
*** mrsoul has joined #tripleo02:37
*** mschuppert has quit IRC02:41
*** rlandy has quit IRC02:49
openstackgerritMerged openstack/tripleo-validations stable/rocky: Fix the undercloud-heat-purge-deleted validation  https://review.openstack.org/61751102:49
*** psachin has joined #tripleo02:50
openstackgerritMerged openstack/tripleo-validations stable/rocky: Use int builtin filter for testing purpose  https://review.openstack.org/61417903:01
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Define UpgradeRemoveUnusedPackages parameter in nova-ironic.yaml  https://review.openstack.org/61721503:01
*** Chaserjim has joined #tripleo03:03
*** Chaserjim has quit IRC03:05
*** Chaserjim has joined #tripleo03:05
*** Chaserjim has quit IRC03:06
*** ooolpbot has joined #tripleo03:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION03:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297103:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302403:10
*** ooolpbot has quit IRC03:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]03:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]03:10
*** tzumainn has quit IRC03:14
*** jaganathan has joined #tripleo03:17
*** Chaserjim has joined #tripleo03:18
*** redrobot has quit IRC03:27
mwhahahaquiquell|off: fyi, the fedora job is now getting to step3 with https://review.openstack.org/#/c/617716/ and the various depends-on. i'll have to look at what's failing in step3 tomorrow as the logs aren't very obvious03:39
*** ramishra_ has joined #tripleo03:40
*** thrash is now known as thrash|g0ne03:56
*** udesale has joined #tripleo04:05
*** Chaserjim has quit IRC04:07
*** ooolpbot has joined #tripleo04:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297104:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]04:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302404:10
*** ooolpbot has quit IRC04:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]04:10
openstackgerritMerged openstack/tripleo-common stable/rocky: Add health check script for Sahara API  https://review.openstack.org/61230304:16
openstackgerritMerged openstack/tripleo-common stable/rocky: Add health checks for Swift services  https://review.openstack.org/61389304:16
openstackgerritMerged openstack/tripleo-common stable/pike: Ensure that GetHostCpusListAction.run() returns a deterministic result  https://review.openstack.org/61590604:16
*** Chaserjim has joined #tripleo04:41
*** udesale has quit IRC04:50
*** Chaserjim has quit IRC04:58
*** Chaserjim has joined #tripleo04:59
*** udesale has joined #tripleo05:02
*** Chaserjim has quit IRC05:04
*** ooolpbot has joined #tripleo05:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297105:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]05:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302405:10
*** ooolpbot has quit IRC05:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]05:10
*** korean102 is now known as korean10105:11
openstackgerritNguyen Hai proposed openstack/tripleo-specs master: Add Train specs directory  https://review.openstack.org/61787605:23
*** chandankumar has joined #tripleo05:25
openstackgerritMerged openstack/tripleo-heat-templates stable/queens: Sets ODL OVSDB inactivity probe timer  https://review.openstack.org/61537705:28
openstackgerritNguyen Hai proposed openstack/tripleo-specs master: Add Train specs directory  https://review.openstack.org/61787605:34
*** Chaserjim has joined #tripleo05:35
openstackgerritMerged openstack/tripleo-common stable/rocky: Don't reset parameter cache in a loop  https://review.openstack.org/61711705:40
*** Chaserjim has quit IRC05:42
*** janki has joined #tripleo05:44
*** saneax has joined #tripleo05:57
*** chandankumar is now known as chkumar|ruck06:04
*** d0ugal has quit IRC06:04
*** ooolpbot has joined #tripleo06:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION06:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297106:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302406:10
*** ooolpbot has quit IRC06:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]06:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]06:10
*** Chaserjim has joined #tripleo06:11
*** d0ugal has joined #tripleo06:18
openstackgerritJanki Chhatbar proposed openstack/puppet-tripleo master: Pass variable from puppet-tripleo to puppet-neutron  https://review.openstack.org/61726506:19
openstackgerritJanki Chhatbar proposed openstack/puppet-tripleo stable/queens: Add support for ODL-OVS IPv6 deployment  https://review.openstack.org/61223806:22
*** Chaserjim has quit IRC06:26
*** Chaserjim has joined #tripleo06:28
*** rfolco has quit IRC06:29
*** apetrich has quit IRC06:31
*** radeks has joined #tripleo06:31
*** Chaserjim has quit IRC06:34
*** ratailor has joined #tripleo06:35
*** aufi has joined #tripleo06:38
*** radeks_ has joined #tripleo06:38
*** radeks has quit IRC06:38
*** ksambor has joined #tripleo06:40
*** hjensas has quit IRC06:51
*** agurenko has joined #tripleo07:03
*** mburned is now known as mburned_out07:07
*** ooolpbot has joined #tripleo07:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION07:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297107:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302407:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323207:10
*** ooolpbot has quit IRC07:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]07:10
*** Chaserjim has joined #tripleo07:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]07:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)07:10
*** skramaja has joined #tripleo07:11
*** paramite has joined #tripleo07:11
*** gkadam has joined #tripleo07:12
Tenguhumpf. so basically we can't install tripleoclient, as it has a hard deps on podman07:13
Tenguthat's kind of a Monday Bug, not a Wednesday.07:13
*** aufi has quit IRC07:13
chkumar|ruckTengu, yes07:14
chkumar|ruckTengu, fix in progress07:14
chkumar|ruckTengu, https://review.rdoproject.org/r/#/c/17347/07:15
Tenguchkumar|ruck: what fix? can't fix that, until a new podman package is issued :/07:15
chkumar|ruckTengu, testing here https://review.rdoproject.org/r/1734807:15
chkumar|ruckTengu, try with the older version07:15
chkumar|ruckTengu, http://cbs.centos.org/koji/buildinfo?buildID=24562 -> got built yesterday07:16
chkumar|ruckwhich broke our ci07:16
Tenguyeah, I saw that.07:16
Tenguah, ok, so you're pinning the version in the python-tripleoclient07:16
chkumar|ruckTengu, yes07:17
chkumar|ruckTengu, just install older version not newer version07:17
Tenguyeah, so have to pre-install podman in some way on my undercloud node - and that will explode anyway for the overcloud image update. yay07:17
chkumar|ruckTengu, so basically we need a workaround on overcloud image deploy side also not to update podman package?07:18
Tenguchkumar|ruck: I don't think so, I have added a step in my lab for overcloud image update.07:19
TenguI can disable it, but it will breack things anyway due to selinux packages not being up-to-date for my tests.07:19
Tengubut that's only on my env I think.07:19
Tengubut basically this can break the building of the overcloud images ;).07:20
*** mschuppert has joined #tripleo07:27
*** salmankhan has joined #tripleo07:29
* chkumar|ruck needs to wait till jpena|off amoralej|off comes online07:31
Tenguchkumar|ruck: ok, so, I suspect container team wants to replace docker - the current package includes all docker manpage with a redirect to podman doc.07:32
Tenguchkumar|ruck: I just fetched the new RPM, and opened it - there's not only the "docker-attach". actually they did a bad thing when building the package.07:33
*** salmankhan has quit IRC07:33
Tenguchkumar|ruck: added a comment on the LP.07:37
*** quiquell|off is now known as quiquell07:39
*** pcaruana has joined #tripleo07:42
*** agurenko has quit IRC07:44
*** hjensas has joined #tripleo07:48
*** salmankhan has joined #tripleo07:51
*** jfrancoa has joined #tripleo07:54
*** salmankhan has quit IRC07:55
*** Chaserjim has quit IRC07:57
*** aedc has joined #tripleo07:57
*** rodolof has joined #tripleo07:58
openstackgerritRedHat RDO CI proposed openstack/tripleo-heat-templates stable/queens: GATE CHECK for TripleO  https://review.openstack.org/56722408:00
openstackgerritRedHat RDO CI proposed openstack/tripleo-quickstart-extras master: GATE CHECK for quickstart-extras  https://review.openstack.org/56044508:00
*** rodolof has quit IRC08:05
*** rodolof has joined #tripleo08:05
*** phuongnh has quit IRC08:06
*** ratailor_ has joined #tripleo08:09
*** ooolpbot has joined #tripleo08:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297108:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]08:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302408:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323208:10
*** ooolpbot has quit IRC08:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]08:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)08:10
*** ratailor__ has joined #tripleo08:11
*** ratailor has quit IRC08:12
*** ratailor has joined #tripleo08:14
*** ratailor_ has quit IRC08:14
*** ratailor__ has quit IRC08:16
*** ratailor has quit IRC08:20
*** ratailor has joined #tripleo08:22
*** Chaserjim has joined #tripleo08:27
*** tosky has joined #tripleo08:42
*** iurygregory is now known as iurygregory|mtg08:42
*** pbandark has joined #tripleo08:43
*** pbandark has quit IRC08:49
korean101Tengu: HI08:49
korean101Tengu: do you use epyc servers? to deploy openstack?08:50
korean101Tengu: compute nodes?08:50
*** pbandark has joined #tripleo08:52
*** cylopez has joined #tripleo08:53
*** jpena|off is now known as jpena08:54
openstackgerritMartin Mágr proposed openstack/tripleo-heat-templates stable/rocky: Collectd documentation refactor  https://review.openstack.org/61790908:57
*** chem has quit IRC09:01
*** chem has joined #tripleo09:01
*** jpich has joined #tripleo09:02
openstackgerritMartin André proposed openstack/tripleo-heat-templates master: Call etcd scaleup playbook when adding master nodes  https://review.openstack.org/61658409:09
*** cylopez has left #tripleo09:10
*** ooolpbot has joined #tripleo09:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297109:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]09:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302409:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323209:10
*** ooolpbot has quit IRC09:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]09:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)09:10
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart master: Add featureset060 for standalone-scenario001 job  https://review.openstack.org/61687109:10
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Fixed logstash file name for tempest  https://review.openstack.org/61791309:11
openstackgerritChristian Schwede proposed openstack/tripleo-heat-templates master: Restrict number of Swift workers to 1 on the undercloud  https://review.openstack.org/61622109:13
*** aufi has joined #tripleo09:13
*** apetrich has joined #tripleo09:14
*** pdeore has joined #tripleo09:18
*** paramite has quit IRC09:36
*** amoralej|off is now known as amoralej09:36
*** lee2 has joined #tripleo09:38
*** lee2 is now known as lyarwood09:38
*** akrivoka has joined #tripleo09:39
*** derekh has joined #tripleo09:40
*** sanjayu_ has joined #tripleo09:40
*** cylopez has joined #tripleo09:40
*** saneax has quit IRC09:41
openstackgerritMichele Baldessari proposed openstack/puppet-tripleo master: WIP Allow the container backend to be configurable  https://review.openstack.org/61783209:43
*** jtomasek has joined #tripleo09:48
*** cylopez has quit IRC09:48
*** Chaserjim has quit IRC09:52
*** jtomasek has quit IRC09:55
*** blinkiz has joined #tripleo09:56
*** apetrich has quit IRC10:00
*** chem has quit IRC10:01
*** chem has joined #tripleo10:02
*** pbandark has quit IRC10:02
Tengukorean101: hello! I have one epyc node, but it's for dev purpose, and only for deploy testing.10:04
*** yprokule has joined #tripleo10:05
*** chem has quit IRC10:06
*** iurygregory|mtg is now known as iurygregory10:09
*** ooolpbot has joined #tripleo10:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297110:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]10:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302410:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323210:10
*** ooolpbot has quit IRC10:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]10:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)10:10
*** chem has joined #tripleo10:11
jfrancoaTengu: hey, one doubt. Do we still keep the /openstack/healthcheck file in the podman containers?10:12
openstackgerritNatal Ngétal proposed openstack-infra/tripleo-ci master: [Core] Remove mitaka.  https://review.openstack.org/61792110:13
openstackgerritNatal Ngétal proposed openstack-infra/tripleo-ci master: [Core] Remove mitaka.  https://review.openstack.org/61792110:14
Tengujfrancoa: well, we use the same images, so it's still present - the issue is, it's not used for now10:14
Tengujfrancoa: and I didn't see conditional mount for that one in case it's provided by the host10:15
jfrancoaTengu: ok, I'm thinking about substituting the former pre-upgrade tasks which check: https://github.com/openstack/tripleo-heat-templates/blob/master/docker/services/nova-api.yaml#L346-L350 by an execution of the healthcheck script "docker exec <cont_name> /openstack/healtcheck10:16
jfrancoaTengu: this way, we could verify the service health state even though podman doesn't provide health status, does it make sense?10:17
Tengujfrancoa: *container_cli exec  then :) but yeah, might be the right thing10:18
jfrancoaTengu: right! thanks for the reminder10:18
Tengu;)10:18
jfrancoaTengu: I'll do some first scratches and add you as reviewer too if you don't mind10:19
Tenguno problem10:19
jfrancoaTengu: thanks for the info10:19
*** salmankhan has joined #tripleo10:20
*** Chaserjim has joined #tripleo10:21
*** Chaserjim has quit IRC10:26
*** florianf has joined #tripleo10:26
*** Chaserjim has joined #tripleo10:26
*** ccamacho has joined #tripleo10:27
*** ccamacho has quit IRC10:27
*** ccamacho has joined #tripleo10:27
*** Chaserjim has quit IRC10:27
*** Chaserjim has joined #tripleo10:28
*** Chaserjim has quit IRC10:32
*** jaganathan has quit IRC10:34
*** akrivoka has quit IRC10:42
Hobbestigrousomeone know if gertty is still maintain ?10:51
*** akrivoka has joined #tripleo10:52
*** udesale has quit IRC10:59
chkumar|ruckTengu, hello10:59
Tenguchkumar|ruck: hey :)10:59
chkumar|ruckTengu, now install packages step passed, now we have new problem https://logs.rdoproject.org/openstack-periodic/git.openstack.org/openstack-infra/tripleo-ci/master/periodic-tripleo-centos-7-master-containers-build/06738e4/logs/undercloud/home/zuul/undercloud_install.log.txt.gz#_2018-11-14_10_08_4210:59
Tenguchkumar|ruck: you can drop the change to the python-tripleoclient package in fact.11:00
chkumar|ruckTengu, at this step http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/puppet/services/podman.yaml#n5911:00
Tenguit's working just fine with the new podman package.11:00
*** Chaserjim has joined #tripleo11:00
Tenguwell, of course, yum will try to install the latest while python-tripleclient does NOT want the latest :)11:01
*** cylopez has joined #tripleo11:01
Tengubut as said: there's a new package in https://buildlogs.centos.org/centos/7/virt/x86_64/container/ and this one solves the issue.11:01
chkumar|ruckamoralej, ^^11:02
Tenguadded a comment in the LP.11:03
chkumar|ruckTengu, reverting the changes11:04
Tengu\o/11:04
Tenguah, it was already integrated. yeah. of course. I have podman 0.10.2.11:05
*** rfolco has joined #tripleo11:06
amoralejchkumar|ruck, so the good one is podman-0.11.1-2.gita4adfe5.el7.x86_64.rpm ?11:07
amoralejwhich build fixes it?11:07
chkumar|ruckamoralej, yes11:07
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Fix the introspection interface look up  https://review.openstack.org/61792511:07
chkumar|ruckamoralej, as Tengu mentioned we are good to go with the new package11:08
amoralejbut in the job failing it's trying to install11:10
amoralejpodman-0.11.1-1.gita4adfe5.el7.x86_64.rpm, nopodman-0.11.1-2.gita4adfe5.el7.x86_64.rpm11:10
*** ooolpbot has joined #tripleo11:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION11:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297111:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302411:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323211:10
*** ooolpbot has quit IRC11:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]11:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]11:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)11:10
Tenguamoralej: it's because of the "latest" instruction in t-h-t for podman package. It conflicts with the pinning in tripleoclient11:10
chkumar|ruckamoralej, it is happening because of this http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/puppet/services/podman.yaml#n5911:11
*** pdeore has quit IRC11:23
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761711:24
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761711:26
*** jtomasek has joined #tripleo11:28
*** ratailor has quit IRC11:30
*** jtomasek has quit IRC11:32
*** cylopez has quit IRC11:34
*** chem has quit IRC11:37
*** salmankhan has quit IRC11:38
*** salmankhan has joined #tripleo11:39
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761711:39
*** hjensas has quit IRC11:39
*** salmankhan has quit IRC11:41
*** salmankhan has joined #tripleo11:41
*** Chaserjim has quit IRC11:41
*** chem has joined #tripleo11:44
*** salmankhan has quit IRC11:46
*** Chaserjim has joined #tripleo11:55
*** Chaserjim has quit IRC12:00
*** raildo has joined #tripleo12:06
*** jfrancoa has quit IRC12:08
*** ooolpbot has joined #tripleo12:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297112:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]12:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302412:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323212:10
*** ooolpbot has quit IRC12:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]12:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,Triaged] - Assigned to Emilien Macchi (emilienm)12:10
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761712:13
openstackgerritChandan Kumar proposed openstack/tripleo-quickstart-extras master: Fixed logstash file name for tempest  https://review.openstack.org/61791312:14
*** ansmith has quit IRC12:14
openstackgerritQuique Llorente proposed openstack-infra/tripleo-ci master: DNM: To test zuul-in-a-box  https://review.openstack.org/61793012:15
Tenguchkumar|ruck: amoralej so yeah, last confirmation if needed: compiling my own python-tripleoclient without the podman version pinning makes the deploy just working fine :). the "state: latest" works as expected without any issue.12:20
amoralejthanks Tengu12:21
*** bfournie has quit IRC12:23
*** jfrancoa has joined #tripleo12:24
*** Chaserjim has joined #tripleo12:29
*** jpena is now known as jpena|lunch12:31
pandahas anyone seens this recently ?12:32
panda2018-11-13 18:16:19 |     "panic: runtime error: invalid memory address or nil pointer dereference",12:32
panda2018-11-13 18:16:19 |     "[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0xc13197]",12:32
panda2018-11-13 18:16:19 |     "goroutine 1 [running]:",12:32
panda2018-11-13 18:16:19 |     "github.com/containers/libpod/vendor/github.com/containers/image/storage.(*storageImageDestination).Commit(0xc4200ee540, 0x13f0620, 0xc420034140, 0x0, 0x0)",12:32
Tengupanda: yeah, happens sometimes with podman pull12:32
Tengupanda: but apparently that doesn't break the image nor the deploy.12:32
Tenguas there are retries at some points.12:32
pandaTengu: it broke undercloud install in one job12:33
pandaor maybe I'm looking at the wrong error12:33
Tengupanda: I think this is the wrong error.12:33
pandaTengu: ok, maybe this is the fatal one12:33
panda2018-11-13 18:16:19 |     "2018-11-13 18:13:39,017 ERROR: 4514 -- Failed running docker-puppet.py for heat_api",12:33
panda2018-11-13 18:16:19 |     "2018-11-13 18:13:39,018 ERROR: 4514 -- error mounting image volumes: unable to find user root: no matching entries in passwd file",12:33
pandaok who removed root from passwd now ? :)12:33
Tenguno idea ^^. but that's the one you're looking for.12:34
*** vpickard_ is now known as vpickard12:34
pandamh failed yesterday on only for the undercloud job.12:36
Tengupanda: it happens from time to time.12:36
Tengupanda: seems to be random though. Had it once for redis12:36
Tengubut no real way to reproduce it :(12:36
Tengufor instance a deploy against master is workin fine on my lab....12:37
Tenguso yeah. seems random. I love that kind of issues -.-.12:37
pandaTengu: can we run CI on your lab ?12:37
TenguXD12:37
chkumar|ruckEmilienM, we need to traige this card https://trello.com/c/0UFdfUEQ/820-cixlp1803232tripleociproa-podman-0111-rpm-conflicts-with-docker12:37
EmilienMchkumar|ruck: I'm on it12:38
chkumar|ruckEmilienM, thanks :-)12:38
*** bnemec has joined #tripleo12:39
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699312:39
*** quiquell is now known as quiquell|lunch12:40
*** panda is now known as panda|rover12:42
*** pdeore has joined #tripleo12:43
*** rh-jelabarre has joined #tripleo12:44
EmilienMchkumar|ruck: do the gate still reset?12:44
EmilienMchkumar|ruck: we did that last night https://review.openstack.org/#/c/617845/12:44
chkumar|ruckEmilienM, yup, seen, gate is pretty much empty12:44
EmilienMchkumar|ruck: someone purged it?12:45
chkumar|ruckEmilienM, nope12:45
EmilienMinteresting12:45
chkumar|ruckEmilienM, podman issue failed all the gates12:45
chkumar|ruckand check jobs12:45
EmilienM"good"12:45
EmilienM...12:45
EmilienMchkumar|ruck: I'm working on a preventive action today12:45
EmilienMso it doesn't happen again12:46
chkumar|ruckEmilienM, you can sync with apevec or Emilienm12:46
EmilienMI can sync with myself? ok12:46
EmilienMthat's good news :D12:46
pdeoreslagle, owalsh, jistr, could you please review https://review.openstack.org/#/c/613195/ whenever you are free ?12:46
pdeorefultonj ^12:47
* jistr off for lunch now but will look when i'm back12:47
fultonjhi pdeore: sure, i'll review that today12:48
pdeorefultonj, jistr Thanks !! :)12:48
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Revert "Revert "Revert "(squash) disabling podman everywhere in gate"""  https://review.openstack.org/61793712:48
openstackgerritKarthik S proposed openstack/tripleo-heat-templates master: Add bootparams service for Controller roles  https://review.openstack.org/61793812:49
*** karthiks has joined #tripleo12:50
*** salmankhan has joined #tripleo12:52
*** abishop has joined #tripleo12:54
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761712:54
openstackgerritMerged openstack/tripleo-common stable/rocky: Add container images for openshift 3.11  https://review.openstack.org/61577612:56
*** zaneb has joined #tripleo12:56
*** bfournie has joined #tripleo12:56
*** salmankhan has quit IRC12:57
chkumar|ruckEmilienM, sorry with number80 I mean12:59
*** trown|outtypewww is now known as trown13:01
EmilienMchkumar|ruck: no prob :)13:05
openstackgerritMarios Andreou proposed openstack/tripleo-quickstart-extras master: Create reproducer_extra_vars.yml with vars used in run-v3.yaml  https://review.openstack.org/61699313:06
*** bnemec has quit IRC13:08
*** ooolpbot has joined #tripleo13:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297113:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]13:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180302413:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323213:10
*** ooolpbot has quit IRC13:10
openstackLaunchpad bug 1803024 in tripleo "pulling containers is failing the overcloud deployment " [Critical,Triaged]13:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)13:10
*** radeks_ has quit IRC13:12
*** Chaserjim has quit IRC13:14
*** rlandy has joined #tripleo13:14
*** jaosorior has joined #tripleo13:14
*** jaosorior has quit IRC13:18
*** dhill_ has joined #tripleo13:20
owalshpdeore: ack, will take a look today13:21
*** matbu has quit IRC13:23
*** matbu has joined #tripleo13:25
*** udesale has joined #tripleo13:27
*** zaneb has quit IRC13:28
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP Pass the container backend explicitely  https://review.openstack.org/61783313:28
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: WIP / fs010: enable Podman on the overcloud  https://review.openstack.org/61252613:29
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: WIP / fs010: enable Podman on the overcloud  https://review.openstack.org/61252613:29
*** radeks has joined #tripleo13:29
EmilienMbandini: rebased ^13:29
*** amoralej is now known as amoralej|lunch13:31
*** rh-jelabarre has quit IRC13:32
*** quiquell|lunch is now known as quiquell13:32
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761713:32
*** jfrancoa has quit IRC13:33
bandiniack awesome13:34
*** toure|gone is now known as toure13:34
*** rh-jelabarre has joined #tripleo13:35
*** rh-jelabarre has quit IRC13:36
*** chkumar|ruck has quit IRC13:37
*** jpena|lunch is now known as jpena13:37
*** lblanchard has joined #tripleo13:39
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: WIP / fs010: enable Podman on the overcloud  https://review.openstack.org/61252613:39
*** Chaserjim has joined #tripleo13:40
openstackgerritJavier Peña proposed openstack-infra/tripleo-ci master: Replace call to map-project-name script with rdopkg  https://review.openstack.org/55261213:40
openstackgerritJavier Peña proposed openstack/tripleo-quickstart-extras master: Build packages using DLRN in reverse order  https://review.openstack.org/60079213:41
*** rh-jelabarre has joined #tripleo13:41
*** jfrancoa has joined #tripleo13:47
*** ansmith has joined #tripleo13:48
*** ccamacho has quit IRC13:51
*** matbu has quit IRC13:51
*** openstackgerrit has quit IRC13:51
*** matbu has joined #tripleo13:53
*** ccamacho has joined #tripleo13:53
*** vpickard is now known as vpickard_13:53
*** Chaserjim has quit IRC13:55
*** mcornea has joined #tripleo13:59
*** vpickard_ is now known as vpickard14:01
*** akrivoka has quit IRC14:01
*** akrivoka has joined #tripleo14:02
*** devep has joined #tripleo14:03
*** openstackgerrit has joined #tripleo14:03
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job  https://review.openstack.org/61687214:03
*** bfournie has quit IRC14:03
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job  https://review.openstack.org/61687214:04
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761714:04
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job  https://review.openstack.org/61687214:07
*** akrivoka has quit IRC14:08
*** salmankhan has joined #tripleo14:08
*** akrivoka has joined #tripleo14:08
*** tzumainn has joined #tripleo14:10
*** ooolpbot has joined #tripleo14:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION14:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297114:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323214:10
*** ooolpbot has quit IRC14:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]14:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)14:10
*** redrobot has joined #tripleo14:14
devepMorning TripleO masters. question - does anyone know how to model ceph rack aware crash map with ssd and sata pools using tripleo?14:15
*** akrivoka has quit IRC14:16
*** morazi has joined #tripleo14:19
*** itlinux has joined #tripleo14:20
EmilienMFYI, no recheck/+w until we have some green on master here: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode14:22
weshaypanda|rover, Tengu so the issue here w/ podoman .11 https://logs.rdoproject.org/33/614633/6/openstack-check/tripleo-ci-centos-7-ovb-3ctlr_1comp-featureset001-master-vexxhost/697a0e0/logs/undercloud/home/zuul/install_built_repo.log.txt.gz#_2018-11-14_11_24_4414:22
weshayis resolved w/ the new packages14:22
weshayin https://buildlogs.centos.org/centos/7/virt/x86_64/container/14:22
weshay?14:22
EmilienMweshay: yes it's resolved14:22
EmilienMwe are waiting for the new tripleoclient to be built and pushed in repos14:22
EmilienMweshay: i'm working on preventive action now, so we gate every new podman rpm14:23
*** mfedosin has quit IRC14:23
Tenguweshay: as EmilienM said :)14:23
*** salmankhan has quit IRC14:23
*** salmankhan has joined #tripleo14:24
panda|roverI still haven't seen many jobs passing today.14:26
*** pdeore has quit IRC14:26
*** akrivoka has joined #tripleo14:26
openstackgerritMichele Baldessari proposed openstack/tripleo-heat-templates master: Remove unnecessary logs due to misconfiguration from pacemaker  https://review.openstack.org/61304314:27
weshaypanda|rover, do we need to sync up?14:28
*** amoralej|lunch is now known as amoralej14:30
panda|roverweshay: maybe after I talk with amoralej rdoinfo, I need seom informations14:30
*** cylopez has joined #tripleo14:31
weshaypanda|rover, k.. I'll do it np14:31
*** salmankhan has quit IRC14:33
*** cylopez has quit IRC14:35
*** bfournie has joined #tripleo14:41
*** agopi has quit IRC14:41
*** salmankhan has joined #tripleo14:44
*** saneax has joined #tripleo14:46
*** lblanchard has quit IRC14:47
*** sanjayu_ has quit IRC14:48
*** lblanchard has joined #tripleo14:48
*** Chaserjim has joined #tripleo14:51
*** cylopez has joined #tripleo14:53
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785514:57
*** gkadam has quit IRC14:58
*** mjturek has joined #tripleo14:58
*** agopi has joined #tripleo15:00
*** itlinux has quit IRC15:00
*** agopi_ has joined #tripleo15:01
*** itlinux has joined #tripleo15:02
*** agopi has quit IRC15:04
EmilienMstill lot of red on http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode&branch=master15:05
EmilienMstill waiting for the tripleoclient rpm15:06
* EmilienM checks why it's still red15:06
*** skramaja has quit IRC15:06
*** agopi_ is now known as agopi15:06
*** itlinux has quit IRC15:07
EmilienMhttps://review.rdoproject.org/r/#/c/17349/ hasn't been built yet :/15:07
*** chem has quit IRC15:07
*** apetrich has joined #tripleo15:09
*** ooolpbot has joined #tripleo15:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION15:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297115:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323215:10
*** ooolpbot has quit IRC15:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]15:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)15:10
panda|roverEmilienM: how much is standalone using ovs ?15:12
EmilienMhow much?15:12
EmilienM34%15:12
EmilienMsometimes 3815:12
EmilienMI don't understand that quesiton15:12
EmilienMstandalone is deploying OpenStack with neutron ml2 ovs driver15:13
EmilienMand there is a plan to migrate to OVN driven15:13
EmilienMdoes it answer the question?15:13
panda|roverEmilienM: how much testing are we getting out on ovs from a standalone deployment , compared to multinode >?15:14
panda|roverEmilienM: a bit more specific this one ? :)15:14
EmilienMah15:14
EmilienMthat's better :D15:14
EmilienMpanda|rover: good question. I think the same really.15:14
EmilienMas the overcloud is one node and standalone is one node too15:15
EmilienMso the functionality is really the same.15:15
EmilienMthe real multinode testing is when deploying on OVB15:15
EmilienMso tunnels are tested, etc15:15
panda|roverEmilienM: great answer thanks.15:16
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Allow ansible python interpreter be configurable  https://review.openstack.org/61771615:16
weshaypanda|rover, have you seen the thread on openstack-dev regarding moving all the jobs to ovn?15:17
panda|roverweshay: yes, but ovn still uses ovs15:19
EmilienMhttps://trunk.rdoproject.org/centos7-master/report.html15:19
EmilienMI'm not sure why the check/gate is stillr ed15:19
fultonji want to recheck a WIP job which is not at the gate to see the effect of changes in a depends-on patch; should i not do that?15:19
EmilienMwe are pulling an old tripleoclient it seems, not the very latest15:19
fultonjhttps://review.openstack.org/#/c/615988/15:19
EmilienMno recheckp lz15:19
EmilienMour CI is red15:20
*** kopecmartin has joined #tripleo15:20
*** hjensas has joined #tripleo15:20
fultonjEmilienM: new patches will also trigger CI right?15:20
EmilienMyes15:20
EmilienMand it'll fail15:20
EmilienMit's fine15:21
EmilienMjust no recheck plz15:21
fultonji won't recheck, just being consistent, i shouldn't proppose any changes either15:21
*** kopecmartin has quit IRC15:22
*** kopecmartin has joined #tripleo15:24
*** kopecmartin is now known as kopecmartin|off15:24
openstackgerritGael Chamoulaud proposed openstack/tripleo-validations master: Fix the introspection interface look up  https://review.openstack.org/61792515:25
*** salmankhan has quit IRC15:25
*** salmankhan has joined #tripleo15:26
*** salmankhan has quit IRC15:33
EmilienMamoralej: I think we need your help here15:34
EmilienMamoralej: our check pipeline is still red15:34
EmilienMhttp://logs.openstack.org/57/616157/5/check/tripleo-ci-centos-7-containers-multinode/7216a57/logs/undercloud/home/zuul/install_built_repo.log.txt.gz15:34
EmilienMeven though we reverted the patch in tripleoclient-distgit for podman pin15:34
EmilienMit's still trying to pull it15:34
*** jtomasek has joined #tripleo15:35
*** Chaserjim has quit IRC15:36
openstackgerritHarald Jensås proposed openstack/tripleo-heat-templates master: Add Storage network to IronicConductor role  https://review.openstack.org/61798715:39
EmilienMjpena ^ need your help plz15:40
jpenaEmilienM: the package was build in RDO Trunk at 11:57 (6 minutes before the log started), so maybe the repo the job used was an earlier one, I'll check15:43
EmilienMok15:44
EmilienMjpena: so we should be good now?15:44
*** rodolof has quit IRC15:44
jpenaEmilienM: that's it. The delorean-current.repo file used by the job was the previous commit, see http://logs.openstack.org/57/616157/5/check/tripleo-ci-centos-7-containers-multinode/7216a57/logs/undercloud/etc/yum.repos.d/delorean-current.repo.txt.gz15:44
jpenaEmilienM: yes, we should be good now15:44
EmilienMk15:46
EmilienMjpena: thx15:46
*** artom has joined #tripleo15:46
*** artom has quit IRC15:47
*** artom has joined #tripleo15:47
*** cylopez has quit IRC15:49
jankiEmilienM, hi15:50
EmilienMjanki: hey15:50
EmilienMjpena: you're right15:51
EmilienMI see some green now: http://zuul.openstack.org/builds?job_name=tripleo-ci-centos-7-containers-multinode&branch=master15:51
EmilienMI think we can recheck15:51
EmilienMmwhahaha, weshay : would you be ok to send https://review.openstack.org/#/c/617937/ to the gate now?15:52
EmilienMso we don't loose coverage15:52
mwhahahameh15:52
weshayEmilienM, +1 it will only impact our lives15:52
weshayso why not15:52
weshay:)15:52
jankiEmilienM, I am deploying OSP13 with ODL on IPv6 environment and am hitting this https://pastebin.com/xfcxEjDE which seems to be https://bugs.launchpad.net/tripleo/+bug/155324315:53
openstackLaunchpad bug 1553243 in tripleo "CI: Resource CREATE failed: Error: resources.ComputeAllNodesValidationDeployment.resources[0]" [Critical,Fix released] - Assigned to James Slagle (james-slagle)15:53
mwhahahai don't see a rush to get that merged, but we can +A it15:53
mwhahahaand let it flow through the gate15:53
EmilienMthanks15:53
*** salmankhan has joined #tripleo15:54
*** EmilienM changes topic to "CI Status: Freshly green - *slowly* recheck & approve patches please | https://docs.openstack.org/tripleo-docs/latest/"15:54
*** salmankhan has quit IRC15:55
*** salmankhan has joined #tripleo15:56
amoralejEmilienM, https://review.rdoproject.org/r/#/c/17351/ can be merged15:57
amoralejsee my comment15:57
jankiEmilienM, but when I ping that address from compute nodes manually, its successful15:58
EmilienMamoralej: what changes are needed in jobs and project config?15:59
EmilienMamoralej: can we do it before we merge this patch so we actually test it?15:59
*** jrist has quit IRC15:59
EmilienMamoralej: I hate merging things not tested15:59
*** salmankhan has quit IRC16:00
*** tosky has quit IRC16:00
*** tosky has joined #tripleo16:00
*** vpickard has quit IRC16:01
openstackgerritAlex Schultz proposed openstack/python-tripleoclient master: Mock configuration load in unit tests  https://review.openstack.org/61799216:02
*** Vorrtex has joined #tripleo16:03
weshayamoralej, 2018-11-14 15:08:15.287536 | primary |   "msg": "The task includes an option with an undefined variable. The error was: 'buildset_artifacts_url' is undefined\n\nThe error appears to have been in '/tmp/tmpqozl6w0s/ae16474001054451a0b2a1c4aad56d78/untrusted/project_0/review.rdoproject.org/rdo-jobs/playbooks/legacy/rdoinfo-tripleo-master-testing-centos-7-multinode-1ctlr-featureset016/run.yaml': line 89, column 7, but may\nbe16:04
weshayelsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n    - shell:\n      ^ here\n"16:04
*** cylopez has joined #tripleo16:04
amoralejweshay, yes16:04
*** vpickard has joined #tripleo16:04
amoralejthat's because that job does not have a parent job which creates a temporary repo16:04
amoralejthat is used to gate16:04
amoralejwhich is how that job is designed16:04
amoralejthat job was intented to use in rdoinfo only16:04
weshayamoralej, ok.. so let's write up the bug.. add it as a promotion-blocker and push on it16:05
amoralejnop16:05
amoralejsorry16:05
amoralejthat's new feature, not bug :)16:05
amoralejdistgits never had oooq deployment in distgit gates16:05
EmilienMjpena: what do you suggest then?16:05
amoralejthey have legacy-DLRN-build-tripleo job which tests that packages can be installed16:06
EmilienMif this job isn't meant to be run on distgit projects16:06
EmilienMjpena: let's take it on #rdo16:06
*** ccamacho has quit IRC16:07
*** irclogbot_3 has quit IRC16:07
*** aufi has quit IRC16:07
*** Chaserjim has joined #tripleo16:07
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785516:09
*** ooolpbot has joined #tripleo16:10
ooolpbotURGENT TRIPLEO TASKS NEED ATTENTION16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180297116:10
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]16:10
ooolpbothttps://bugs.launchpad.net/tripleo/+bug/180323216:10
*** ooolpbot has quit IRC16:10
openstackLaunchpad bug 1803232 in tripleo "podman 0.11.1 rpm conflicts with docker" [Critical,In progress] - Assigned to Emilien Macchi (emilienm)16:10
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785516:10
*** vkapalav has joined #tripleo16:10
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785516:11
*** irclogbot_3 has joined #tripleo16:11
EmilienMpanda|rover, weshay: I'm removing https://bugs.launchpad.net/tripleo/+bug/1802971 from alerts16:11
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]16:11
EmilienMthe workaround merged16:11
EmilienMand I added ci tech-debt  tags16:11
*** Chaserjim has quit IRC16:12
*** jrist has joined #tripleo16:12
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job  https://review.openstack.org/61687216:12
EmilienMjanki: i never tried odl16:12
EmilienMso I don't really know16:12
jankiEmilienM, I think the error is irrespective of ODL. Its more TripleO related as it is happening at computenodevalidation step16:13
jankiEmilienM, 2018-11-10 09:03:37Z [overcloud.ComputeAllNodesValidationDeployment]: CREATE_FAILED  Error: resources.ComputeAllNodesValidationDeployment.resources[1]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 116:14
janki Ping to 2620:52:0:13b8:5054:ff:fe3e:11 failed. Retrying...16:14
*** salmankhan has joined #tripleo16:14
EmilienMjanki: and the ping6 worked manually?16:15
jankiEmilienM, yes16:15
jankifrom both computes to this controller (2620:52:0:13b8:5054:ff:fe3e:11), it worked fine. setup is 3 controllers, 2 computes16:15
jankiEmilienM, this claims to have solved the issue https://review.openstack.org/#/c/289568/116:16
panda|roverEmilienM: ack16:18
jankiEmilienM, would adding a dependency on controllernodedeployment and computenodedeployment for computenodedeploymentvcalidation solve it?16:19
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job  https://review.openstack.org/61687216:19
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761716:21
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job  https://review.openstack.org/61687216:22
*** morazi has quit IRC16:22
*** ccamacho has joined #tripleo16:23
*** Chaserjim has joined #tripleo16:27
*** pcaruana has quit IRC16:29
*** Chaserjim has quit IRC16:29
*** Chaserjim has joined #tripleo16:30
jankiEmilienM, also this is not happening always. I am doing this stability tests for ODL where I deploy overcloud with ODL, test few use-cases, delete overcloud and redeploy in a loop. and got this error on 12th iteration16:30
*** Chaserjim has quit IRC16:34
*** ccamacho has quit IRC16:35
*** cylopez has quit IRC16:36
*** udesale has quit IRC16:36
openstackgerritJose Luis Franco proposed openstack/tripleo-heat-templates master: WIP: Add pre-upgrade validation step for containerized services.  https://review.openstack.org/58250216:36
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799716:38
openstackgerritMarios Andreou proposed openstack-infra/tripleo-ci master: Add new tripleo-ci-centos-7-standalone-scenario001 job  https://review.openstack.org/61687216:41
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799716:43
mwhahahaweshay: i thought we disable dthe volume boot pattern test, is that not properly ignored in the standalone? http://logs.openstack.org/37/616937/10/check/tripleo-ci-centos-7-standalone/86a54e7/logs/tempest.html.gz16:44
*** jfrancoa has quit IRC16:44
*** adobrodu has quit IRC16:46
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799716:48
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761716:49
weshaymwhahaha,  is ocata dead for us or not yet?16:50
mwhahahaweshay: kinda?16:50
weshayrfolco, lolz ^16:50
mwhahahaweshay: techinally ocata is the first extendend maintence release for openstack itself. I asked that we eol our stuff, but i don't think it was ever done16:50
* weshay looks at volboot16:50
*** ramishra_ has quit IRC16:51
mwhahahain terms of tripleo, we've always assumed ocata would EOL, but then the upstream had other ideas. I think we're (tripleo) all ok with no longer doing ocata patches. but not sure how that affects rdo16:51
*** zaneb has joined #tripleo16:52
weshayhrm.. mwhahaha in that log file the blacklist is empty16:53
* weshay checks other jobs16:53
*** zaneb has quit IRC16:57
*** salmankhan has quit IRC16:57
*** Chaserjim has joined #tripleo16:58
*** psachin has quit IRC16:58
*** Chaserjim has quit IRC17:00
*** Chaserjim has joined #tripleo17:01
*** ksambor has quit IRC17:01
*** salmankhan has joined #tripleo17:04
*** jtomasek has quit IRC17:04
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761717:05
*** Chaserjim has quit IRC17:05
*** jpich has quit IRC17:07
*** salmankhan has quit IRC17:08
*** agopi is now known as agopi|food17:10
*** pcaruana has joined #tripleo17:11
mwhahahaweshay: sooo i'm not seeing where tempest_skip_master.yml is actually used17:12
weshaysec17:12
* weshay looking again.. had to chat w/ some folks17:12
*** janki has quit IRC17:14
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: DNM: Remove newton and ocata jobs  https://review.openstack.org/61799917:14
*** devep has quit IRC17:15
weshayhrm.. http://logs.openstack.org/29/617629/1/gate/tripleo-ci-centos-7-containers-multinode/c13efa5/job-output.txt.gz#_2018-11-14_04_19_50_28010517:17
weshayhttp://logs.openstack.org/29/617629/1/gate/tripleo-ci-centos-7-containers-multinode/c13efa5/job-output.txt.gz#_2018-11-14_04_19_50_28010517:17
*** thrash|g0ne is now known as thrash17:17
weshaymwhahaha, hrm..http://codesearch.openstack.org/?q=check_tempest_bugs&i=nope&files=&repos=17:18
mwhahahai feel i must break out: ಠ_ಠ17:18
weshayone eye'd willie?17:18
weshaystill digging17:19
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: DNM: Remove newton and ocata jobs  https://review.openstack.org/61799917:19
mwhahahaweshay: so i found, https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/tasks/pre-tempest.yml#L2717:19
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799717:19
weshaymwhahaha, ah.. that is for tempest mailer.. sorry.. it also appears earlier17:20
mwhahahaso it loaded the vars17:21
mwhahahahttp://logs.openstack.org/37/616937/10/check/tripleo-ci-centos-7-standalone/86a54e7/job-output.txt.gz#_2018-11-14_14_05_34_48820717:21
weshayhttps://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/tasks/pre-tempest.yml#L3217:21
mwhahahathis stuff is way too complicated17:22
weshaymwhahaha, sush17:22
weshaynoting mwhahaha can't have things both ways.. :)17:22
mwhahahawe need to stop adding features17:23
mwhahahaso it's in the skip_file17:23
mwhahahabut that's not used?17:23
mwhahahahttp://logs.openstack.org/37/616937/10/check/tripleo-ci-centos-7-standalone/86a54e7/logs/undercloud/home/zuul/17:23
mwhahahaskip_file exists w/ it int here17:23
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761717:23
mwhahahawait it might not be in there17:23
weshaymwhahaha, sec.. I just decompressed the file17:23
*** florianf is now known as florianf|afk17:24
mwhahahano it's not17:24
mwhahahacause it's only the undercloud things17:24
*** Chaserjim has joined #tripleo17:24
weshaydang it chunk.io17:25
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart master: Move standalone job to podman instead of docker  https://review.openstack.org/61397517:26
weshayyup17:26
weshaymwhahaha, it's only undercloud things17:26
mwhahahaಠ_ಠ17:26
mwhahahai'm assuming it is because of tempest_undercloud17:27
* mwhahaha goes and tries to find the value of that17:28
*** chem has joined #tripleo17:28
mwhahahameh17:29
mwhahahai found it17:29
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: DNM: Remove newton and ocata jobs  https://review.openstack.org/61799917:29
weshayhttp://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/validate-tempest/templates/skip_file.j217:30
mwhahahaweshay: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset052.yml#L3217:30
weshay        "tempest_undercloud": true,17:30
mwhahahait's because tempest_undercloud in standalone. because i think tempest_overcloud does things not compatible with the standalone17:31
* weshay blames me17:31
weshayso does git17:31
weshayhrm..17:32
weshaymwhahaha, so the fix is of course more complicated then one would like17:32
mwhahahait's because of the stupid tempest deployer input file17:32
weshaywe need to point tempest at the single node17:32
weshayand not expect another node.. like the overcloud17:32
mwhahahatempest_overcloud assumes tempest deployer input file which we don't have17:32
weshaybut that same variable is also used for the skip17:32
weshayI think I can fix this in the jinja17:33
mwhahahaso without rewriting the validate-tempest role, i think we might consider tempest_standalone17:33
weshaymwhahaha, as a var?17:34
weshayya17:34
mwhahahai hate it though17:34
mwhahahaand the only reason we have to do this is because of that stupid deployer input file17:34
weshaymwhahaha, the skip list for the overcloud and standalone should be the same17:34
weshayreally? that's a whole can of worms17:35
weshayI sense the presence of luigi17:35
mwhahahaweshay: https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/validate-tempest/templates/configure-tempest.sh.j2#L138-L14117:36
mwhahahathat's the only reason we can't use tempest_overcloud17:36
mwhahahain fact we have a hack on the next two lines because of that17:36
* mwhahaha flips tables 17:37
weshaywill fix it here http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/validate-tempest/templates/skip_file.j217:37
weshaymwhahaha, please give me 2min17:38
*** akrivoka has quit IRC17:39
*** yprokule has quit IRC17:40
openstackgerritMerged openstack/tripleo-heat-templates stable/rocky: Update to OpenShift 3.11 release  https://review.openstack.org/61577817:42
openstackgerritMerged openstack/tripleo-heat-templates stable/rocky: Add network data for use with openshift deployments  https://review.openstack.org/61643817:44
*** panda|rover is now known as panda|rover|off17:47
*** Chaserjim has quit IRC17:47
weshayok.. looking again17:48
*** pbandark has joined #tripleo17:52
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761717:52
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: upgrade: remove Docker containers now managed by Podman  https://review.openstack.org/61520917:54
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: upgrade: remove Docker containers now managed by Podman  https://review.openstack.org/61520917:55
pbandarkhi17:58
pbandarkI am trying to understand the boot process of overcloud nodes. As per my understanding, "agent.kernel" & "agent.ramdisk" are used(fetched due to /httpboot/inspector.ipxe)17:59
pbandarkwhile introspecting the overcloud nodes. Further during the overcloud deployment(`overcloud deploy`), "bm-deploy-kernel" & "bm-deploy-ramdisk" images are used(fetched due to /httpboot/boot.ipxe)17:59
pbandarkto boot the nodes. But, then I am unable to understand how "overcloud-full" image is written to the overcloud nodes and when "overcloud-full-initrd", "overcloud-full-vmlinuz" come in to picture.17:59
pbandarkCan anyone help me to understand it better?18:00
openstackgerritRafael Folco proposed openstack-infra/tripleo-ci master: WIP: Remove newton and ocata jobs  https://review.openstack.org/61799918:00
*** agopi|food is now known as agopi18:00
mwhahahapbandark: ironic uses them to write out to the system. you're question is more around how does ironic lay down the image on hardware (i have no idea of the itnernals)18:00
mwhahahas/you're/your18:00
pbandarkmwhahaha, yeah its done by ironic but I am trying to understand how the operation is performed(for ex. using iscsi initiator/target etc.)18:03
pbandarkand when "overcloud-full-initrd", "overcloud-full-vmlinuz" come in to picture.18:03
*** chem has quit IRC18:03
mwhahahapbandark: i believe it's because the ironic bits don't reboot the system but rather just load the kernel, but that's really a question for the ironic irc channel18:04
*** derekh has quit IRC18:04
pbandarkok18:04
mwhahahaquiquell: i figured out what the fedora standalone is failing on in step3, https://bugs.launchpad.net/tripleo/+bug/1803411 not sure the fix18:06
openstackLaunchpad bug 1803411 in tripleo "nova_statedir_owner fails in the fedora28 standalone" [High,Triaged]18:06
openstackgerritGabriele Cerami proposed openstack-infra/tripleo-ci master: Collect logs: Handle errors and timeout  https://review.openstack.org/61761718:06
openstackgerritMerged openstack/tripleo-heat-templates stable/rocky: Remove unused networks from OpenShift roles  https://review.openstack.org/61643918:06
*** chandankumar has joined #tripleo18:07
chandankumarEmilienM, hello18:07
mwhahahachandankumar: i blocked it and EmilienM went out, he'll be back later18:08
chandankumarEmilienM, https://review.rdoproject.org/r/#/c/17351/ I have left a comment on this18:08
chandankumarmwhahaha, if new version of podman is available it will fail18:08
mwhahahayea thats why i -2'd that packaging change18:09
chandankumarmwhahaha, thanks !18:09
mwhahahai'll chat about it with EmilienM later18:09
*** devep has joined #tripleo18:09
chandankumarmwhahaha, I think we need to wait till we have zuul + pagure integration in place on sf side18:09
weshaychandankumar, hey.. you have a minute18:10
*** thrash is now known as thrash|biab18:11
*** psachin has joined #tripleo18:12
chandankumarweshay, yes18:13
weshaychandankumar, k.. the standalone in fs052 is set as tempest_undercloud because it's running against the single node18:14
weshaychandankumar, this presents as a bug w/ the skip list18:14
weshayor maybe I just put the skip for validatebootpattern in the wrong section18:15
weshaychandankumar, so /me wondering how we want to think of the tempest tests as they run against the standalone18:15
weshayI think logically they are equiv to overcloud tests18:15
weshaywdyt18:15
*** jpena is now known as jpena|off18:16
chandankumarweshay, yes, you are correct, it is the same as running the tempest tests as on overcloud18:16
weshaychandankumar, ok.. so in that case.. I think we need to update the logic here http://git.openstack.org/cgit/openstack/tripleo-quickstart-extras/tree/roles/validate-tempest/templates/skip_file.j218:17
chandankumarweshay, the reason for timing out the tests are not clear yet as per this bug https://bugs.launchpad.net/tripleo/+bug/180297118:18
openstackLaunchpad bug 1802971 in tripleo "tempest volume_boot_pattern and basic_ops running concurrently causing timeouts" [Critical,Triaged]18:18
chandankumarweshay, I might need to look into that part18:18
weshaychandankumar, yes.. but we added volume_boot_pattern to the skip18:19
weshayand it's not picked up18:19
*** trown is now known as trown|lunch18:19
weshaybecause standalone is defined as undercloud_tempest18:19
weshaychandankumar, is that part clear18:19
weshayI'm testing a patch now18:19
chandankumarweshay, I was encouraging to run standalone without skip list and keep the blacklisted tests in fs itself18:19
weshaychandankumar, we shouldn't have 3 different ways18:20
weshayman18:20
weshayjust make the skip logic work for standalone18:20
chandankumarweshay, I need to dig into that why it is not picking18:20
weshaychandankumar, there may be MANY singlenode jobs18:20
weshaychandankumar, so I disagree w/ that path18:20
weshayI can share my screen18:20
weshayif you want.. or we can talk about it tomorrow18:20
weshayI think we just need a jinja if, elseif18:21
chandankumarweshay, feel free to paste the details on etherpad, I can find details tomorrow18:21
chandankumarmay be come up with a working solution18:21
weshaychandankumar, k k18:21
weshaythanks for the sanity check18:21
weshaywill keep in the loop w/ this for sure18:21
chandankumarweshay, you might be correct, we have lots of standlone jobs, may be there is some problem with zuul v3 overrides (I am not sure), need to dig into that18:22
chandankumarweshay, https://review.rdoproject.org/r/#/c/17351/ apart from standalone I need your feedback on this18:22
*** psachin has quit IRC18:25
*** thrash|biab is now known as thrash18:37
*** jrist has quit IRC18:43
weshaymeh.. more code but it's clear18:51
*** chandankumar has quit IRC18:53
openstackgerritMerged openstack/tripleo-quickstart master: Revert "Revert "Revert "(squash) disabling podman everywhere in gate"""  https://review.openstack.org/61793718:54
*** jrist has joined #tripleo18:54
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: break out the tempest skip list generation by deployment  https://review.openstack.org/61800818:57
weshaymwhahaha, fyi ^18:58
*** jtomasek has joined #tripleo18:58
*** devep has quit IRC19:01
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Make build-test-packages more resilient  https://review.openstack.org/61693719:01
openstackgerritwes hayutin proposed openstack/tripleo-quickstart-extras master: Make build-test-packages more resilient  https://review.openstack.org/61693719:01
openstackgerritMichele Baldessari proposed openstack/puppet-pacemaker master: WIP Retry backup_cib()  https://review.openstack.org/61801119:26
*** trown|lunch is now known as trown19:28
openstackgerritRadoslaw Smigielski proposed openstack/paunch master: Default log location failes for non-root users  https://review.openstack.org/61801419:34
*** xek_ has joined #tripleo19:37
*** xek has quit IRC19:40
*** jtomasek has quit IRC19:42
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: upgrade: remove Docker containers now managed by Podman  https://review.openstack.org/61520919:48
*** jtomasek has joined #tripleo19:48
*** amoralej is now known as amoralej|off19:48
openstackgerritEmilien Macchi proposed openstack/tripleo-common master: WIP - Implement tripleo-systemd-wrapper role  https://review.openstack.org/61737719:52
*** pcaruana has quit IRC19:54
*** apetrich has quit IRC19:57
openstackgerritDan Trainor proposed openstack/puppet-tripleo master: Enable disablereuse option in Zaqar proxy backend  https://review.openstack.org/61801919:59
*** apetrich has joined #tripleo20:10
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: Do not deploy Docker if ContainerCli is set to podman  https://review.openstack.org/61802020:13
openstackgerritSorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Enable yamllint on modified files  https://review.openstack.org/61802120:21
*** artom has quit IRC20:22
*** vpickard is now known as vpickard_20:30
ssbarnea|bkp2mwhahaha: EmilienM - I have some serious concerns regarding the approach taken on https://review.openstack.org/#/c/616579/ -- i do not think it the right path.20:31
mwhahahassbarnea|bkp2: ok?20:31
ssbarnea|bkp2it breaks the product by relying on something that does not exist in a normal installation: ansible-playbook-2 ...!?20:32
mwhahahawell if it didn't exist, CI would fail20:32
mwhahahai think it exists in our packaging of ansible20:32
mwhahahassbarnea|bkp2: are you using rpm installed ansible?20:33
ssbarnea|bkp2mwhahaha: yes but there was a change to add a symlink but this will *not* work with an ansible installed from a wheel for example.20:33
mwhahahassbarnea|bkp2: we don't support that20:33
mwhahahawe only support it installed via packaging20:33
mwhahahaand do not support local installs of ansible20:33
ssbarnea|bkp2the fact that it works on CI ... on a modified system does not make it right.20:33
*** jtomasek has quit IRC20:33
mwhahahaci is not supposed to be a modified system20:33
mwhahahaif its a modified system our CI is not right20:33
weshayssbarnea|bkp2, the rpm has the link20:34
weshaythe ansible here is the one use to deploy the overcloud20:34
mwhahahayour use case of pip installed ansible is not supported in tripleo and never has been20:34
weshaynot the ansible one would use or invoke outside of tripleo/osp20:34
weshayright20:34
weshayalthough revisiting all the ansible versions used on branches should be a topic sometime down the road20:35
weshayunless that has been fixed20:35
mwhahahassbarnea|bkp2: TBH thats not my favorite patch as part of this python3 stuff and really it's a problem with the fact that fedora has both 2 and 3 installed which goes away if we have an OS that only has python320:36
mwhahahabut our packaging has both ansible-playbook-2 and ansible-playbook-3 depending on what is installed20:37
ssbarnea|bkp2mwhahaha: i still find it too likely to break. I would be ok with an approach like "command -v ansible-plauybook-2 || command -v ansible-playbook", but switching from -2 suffix seems too risky as we all know this did not exist.20:38
ssbarnea|bkp2if we have a fallback logic we should be much-much better.20:38
weshayhttps://github.com/rdo-packages?utf8=%E2%9C%93&q=ansible&type=&language= ?20:39
ssbarnea|bkp2also I am sure that at some point the -3 will be dropped.20:39
mwhahahanot likely20:39
mwhahahait's a fedora/rdo convention20:39
ssbarnea|bkp2mwhahaha: how much you want to bet on that? it may take some time but i doubt it will not happen. the version suffix was promoted my PSF too as a *temporary* measure. there is a PEP for that which states that at some point the recommendation will change to dropping it.... more exactly after py27 reaches EOL.20:41
mwhahahabecause we've been burned byt this in the past20:42
mwhahahaand -3 is a simlink tot he python version20:42
mwhahahaof ansible-playbook-3.620:42
mwhahahaor ansible-playbook-3.720:42
mwhahahalook, i get your point and i don't like it but in this case the packaging is what we support and this is the way rdo has decided to handle multiple versions20:43
ssbarnea|bkp2i will be perfectly ok if we would have a fallback: when "-[23]" does not exist to use the version without suffiox.20:43
mwhahahai'd rather that we not install the python2 version but we've still got dependencies so we need a path forward and this isn't bash and you're asking for something even more complicated than we already have20:43
ssbarnea|bkp2mwhahaha: yes, this is a valid point and we start testing py2 removal as soon we have a succesfull deployment on py3.20:44
mwhahahai've had a successful deploy on py320:45
mwhahahabut that came on ln -sf /usr/bin/python /usr/bin/python320:45
ssbarnea|bkp2mwhahaha: well... we didn't run a full tempest ;)20:45
* mwhahaha isn't in the mood and goes back to additional stupid python shim crap20:45
*** pbandark has quit IRC20:48
ssbarnea|bkp2yeah, i know we need to write the ansible version of my previous command, but that's not really an issue.  but this can wait until one of us will face the missing symlink issue.20:49
*** raildo has quit IRC20:51
ssbarnea|bkp2mwhahaha: can you have a look at https://review.openstack.org/#/c/617363/  ? -- it solves the annoying b'' output lines.20:53
*** agopi is now known as agopi|brb20:54
mwhahahak20:54
*** artom has joined #tripleo20:55
trozetmwhahaha: im confused by this: https://github.com/openstack/tripleo-repos/blob/master/tripleo_repos/main.py#L16120:57
trozetmwhahaha: there is queens current-tripleo, i just went to look for rocky curren-tripleo it isn't there20:57
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add python shim for docker config scripts  https://review.openstack.org/61802720:57
mwhahahatrozet: take that up with the RDO/TripleO CI folks20:58
mwhahahassbarnea|bkp2: -^ should get us a passing deploy on fedora20:58
trozetweshay:^^?20:58
*** agopi|brb has quit IRC20:58
openstackgerritSorin Sbarnea proposed openstack/tripleo-quickstart-extras master: Harden yaml linting on modified files  https://review.openstack.org/61802121:00
weshaytrozet, what's up21:02
trozetweshay: trying to use tripleo-repos with rocky and current-tripleo, but hte tripleo-repos gives me this error: https://github.com/openstack/tripleo-repos/blob/master/tripleo_repos/main.py#L16121:03
mwhahahatrozet: if current-tripleo exists on stable branches we can just patch in tripleo-repos. i think that bit of code might be in the original commit so not sure the history on it21:04
weshayhttps://trunk.rdoproject.org/centos7-rocky/21:05
mwhahahathat code was likely before we had current-tripleo or something21:05
trozetmwhahaha: ok we can remove that then?21:05
mwhahahayea it's from the initial commit 3 years ago21:06
mwhahahatrozet: patches welcome21:06
trozetmwhahaha: ok i'll delete that check then21:06
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add python shim for docker config scripts  https://review.openstack.org/61802721:12
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add python shim for docker config scripts  https://review.openstack.org/61802721:23
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Modify the zuul inventory to pass to the reproducer  https://review.openstack.org/61740121:29
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Rewrite nodepool-setup role to be used with v3 playbooks  https://review.openstack.org/61785521:30
openstackgerritRonelle Landy proposed openstack/tripleo-quickstart-extras master: WIP: Add a new role to clone repos and include zuul changes  https://review.openstack.org/61799721:30
*** radeks has quit IRC21:35
*** ansmith has quit IRC21:35
*** xek_ has quit IRC21:39
openstackgerritTim Rozet proposed openstack/tripleo-repos master: Removes checks for branch and current-tripleo/dev  https://review.openstack.org/61804521:43
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add python shim for docker config scripts  https://review.openstack.org/61802721:49
*** Chaserjim has joined #tripleo21:52
*** trown is now known as trown|outtypewww21:53
*** Chaserjim has quit IRC21:56
openstackgerritAlan Bishop proposed openstack/tripleo-heat-templates master: Configure cinder's access to the nova API  https://review.openstack.org/61666922:00
weshaymwhahaha, EmilienM https://review.openstack.org/#/c/618008/22:06
weshayfixes the issues w/ the skip list22:06
weshaylinks are in the review22:06
weshayrlandy, ^22:07
mwhahahaweshay: hack tastic :D22:08
weshaybully22:09
weshayat least it's clear code22:09
mwhahahanot really cause we shouldn't be really be relying on os_cloud :D but oh well22:10
mwhahahahonestly validate-tempest shouldn't even understand the concept of undercloud/overcloud22:10
mwhahahathat seems to be the recuring issue with the oooq roles22:10
weshaymwhahaha, I'll fix that.. but we would have needed two changes to do that22:10
weshaymwhahaha, there were issues if I recall when trying to configure tempest for the undercloud execution22:11
weshayanyway.. this is all getting redone22:11
weshayand you know this22:11
mwhahahayea it's cause of the deployer identifier stuff22:11
mwhahahaand the lack of nova compute22:11
mwhahahabut that's because we aren't properly exposing what's required to setup tempest. hopefully os_tempest resolves some of thi stuff22:12
*** Vorrtex has quit IRC22:12
weshayya.. just trying to get light on that atm22:12
*** abishop has quit IRC22:14
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Add docker bridge ip option to the undercloud.conf  https://review.openstack.org/61805422:20
*** Chaserjim has joined #tripleo22:24
openstackgerritmathieu bultel proposed openstack/python-tripleoclient master: Add docker bridge ip option to the undercloud.conf  https://review.openstack.org/61805422:28
*** rh-jelabarre has quit IRC22:40
jristgentlemen and ladies22:43
jristI have a weird question22:43
jristdo we still need containers-default-parameters.yaml ?22:43
jristmy understanding is no22:43
jristbut it's still in http://git.openstack.org/cgit/openstack/tripleo-common/tree/tripleo_common/constants.py22:44
matbujrist: why no ?22:45
jristthought the URLs were dynamically created?22:45
matbujrist: ha yep, im not aware of that dynamic creation but make sense :)22:46
jristI'll admit I only sort of know what I'm talking about22:46
mwhahahajrist: is it required? not necessarily22:46
jristanyway it is effecting deployments in UI22:46
mwhahahais it still a thing? yes meaning it can still be provided22:47
jristhttps://bugzilla.redhat.com/show_bug.cgi?id=164891822:47
openstackbugzilla.redhat.com bug 1648918 in openstack-tripleo-ui "The UI should not create containers-default-parameters.yaml any more" [Urgent,Assigned] - Assigned to jrist22:47
jristI think I just need to remove it from http://git.openstack.org/cgit/openstack/tripleo-heat-templates/tree/capabilities-map.yaml22:48
mwhahahajrist: so either is still acceptable, the new way is the ContatinerImagePrepare bit22:48
jristyeah22:48
jristthanks mwhahaha22:48
jristI think I just need to remove it from THT22:48
mwhahahatechincally i think if you don't provide one, we generate it22:49
jristsince UI kicks off ContainerImagePrepare22:49
jristexactly22:49
jristI think it might be even if you provide one22:49
jrist...22:49
jristyaml all the way down22:52
jristmwhahaha: are you familiar with that capabilities map?22:52
mwhahahanope22:52
* jrist git blame22:53
jristmwhahaha: previously pulled in environments/docker.yaml22:54
jristI don't suppose that's still a thing22:54
mwhahahait still exists but has no content anymore22:55
jristI see that22:55
jristI am just going to remove this containerized chunk of yaml22:55
mwhahahaweshay, quiquell, ssbarnea|bkp2, EmilienM: first successful fedora28 standaloen job: http://logs.openstack.org/27/618027/4/check/tripleo-ci-fedora-28-standalone/d8d6beb/23:03
EmilienMmwhahaha: let's call it a good day23:04
EmilienMmwhahaha: I don't see tempest :P23:04
* EmilienM hides23:05
mwhahahabut we need to land, https://review.openstack.org/#/c/616579/ https://review.openstack.org/#/c/616203/ https://review.openstack.org/#/c/617793/ https://review.openstack.org/#/c/618027/23:05
mwhahahaat the moment i'll just take a deployment pass for the win23:05
EmilienMnice23:06
mwhahahai have no idea why tempest didn't run23:06
mwhahahaoh job config23:07
mwhahahalet me try turning that on23:07
openstackgerritAlex Schultz proposed openstack-infra/tripleo-ci master: Enable tempest for fedora28 standalone  https://review.openstack.org/61805623:07
*** boazel_ has joined #tripleo23:15
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: WIP - Switch Neutron DHCP agent to use SystemD wrappers  https://review.openstack.org/61739323:15
*** boazel has quit IRC23:18
*** Chaserjim has quit IRC23:18
openstackgerritEmilien Macchi proposed openstack/tripleo-heat-templates master: upgrade: remove Docker containers now managed by Podman  https://review.openstack.org/61520923:18
openstackgerritMerged openstack/tripleo-heat-templates master: Set flavor property resources:MEMORY_MB to 0  https://review.openstack.org/61762123:20
openstackgerritMerged openstack/tripleo-heat-templates master: Convert dynamic lookups to use colon notation  https://review.openstack.org/61744123:20
mwhahahak we can switch to puppet 5 now23:21
EmilienMmwhahaha: have you tested it?23:23
EmilienMlast time I said so I failed23:23
EmilienMlet me try again23:23
mwhahahai'll test it tomorrow23:23
mwhahahawe can depends on my f28 patch it'll show it23:24
EmilienMmwhahaha: where would you update it?23:24
mwhahahatomorrow after https://review.openstack.org/#/c/616203/ lands i can rebase https://review.openstack.org/#/c/618027/ to get the THT changes23:25
mwhahahawhich will show the iptables rules if they work or not on the fedora job23:25
mwhahahaif you look at the fedora job results i posted earlier the iptables rules are missing23:25
mwhahahawhich doesn't affect standalone because it's all local23:25
EmilienMah right23:26
EmilienMand it's puppet5 already23:26
mwhahahayea23:26
mwhahahahttp://logs.openstack.org/27/618027/4/check/tripleo-ci-fedora-28-standalone/d8d6beb/logs/undercloud/var/log/extra/network.txt.gz23:26
EmilienMshould we roll directly to puppet6?23:26
mwhahahagod no23:26
EmilienMmaybe wait one cycle23:26
mwhahahayea maybe T23:26
EmilienMmwhahaha: train23:27
EmilienMthey picked a name23:27
EmilienMOpenStack Train23:27
mwhahahayea and it's stupid23:27
mwhahahai'll stick with T23:27
mwhahahathx23:27
EmilienMlol23:27
EmilienMloveit23:27
EmilienMConnectionError: HTTPSConnectionPool(host='registry-1.docker.io', port=443): Max retries exceeded with url: /v2/ (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x7faa5ba11650>: Failed to establish a new connection: [Errno -2] Name or service not known',))23:27
EmilienMI see a bunch of that in standalone jobs23:27
EmilienMit makes me wonder if we hit infra proxies in that job23:28
mwhahahathat seems to be a dns problem23:28
EmilienMhttp://logs.openstack.org/01/611801/13/check/tripleo-ci-centos-7-standalone/4cfbd48/logs/undercloud/etc/docker/daemon.json.txt.gz23:28
EmilienMno mirror configured23:28
EmilienM#oops23:28
mwhahahaman i am so tired of refixing the same thing over and over23:29
EmilienMhttp://logs.openstack.org/03/616203/9/check/tripleo-ci-centos-7-undercloud-containers/fd141cf/logs/undercloud/etc/docker/daemon.json.txt.gz23:29
EmilienMundercloud has it23:29
EmilienMlet me take this one23:29
mwhahahait's cause we don't specify in the standalone_parameters.yaml23:29
mwhahahaso we'll need to add that to the standalone role in oooq-extras23:29
EmilienMok let me fix that23:29
*** mcornea has quit IRC23:32
openstackgerritEmilien Macchi proposed openstack/tripleo-quickstart-extras master: standalone: configure DockerRegistryMirror when possible  https://review.openstack.org/61806223:33
EmilienMmwhahaha: ^NINJAJINJA23:33
EmilienMbut23:34
EmilienMthere is something else23:34
EmilienMI'm not sure standalone uses toci-quickstart/config/testenv/multinode.yml23:35
EmilienMwhich export undercloud_docker_registry_mirror23:35
mwhahahanow it does not23:36
mwhahahas/now/no23:36
mwhahahahttp://logs.openstack.org/01/611801/13/check/tripleo-ci-centos-7-standalone/4cfbd48/logs/playbook_executions.log23:36
EmilienMno we don't23:36
EmilienMhttp://logs.openstack.org/01/611801/13/check/tripleo-ci-centos-7-standalone/4cfbd48/logs/undercloud/var/log/extra/dump_variables_vars.json.txt.gz23:36
EmilienMdamn23:36
EmilienMlet me fix that too23:36
EmilienMwait23:37
EmilienMstandalone.yml -> multinode.yml23:37
EmilienMwtf23:38
EmilienMwe'll see23:39
mwhahahayea that's everytime i have to look into these23:39
mwhahahait's available on the env via NODEPOOL_DOCKER_REGISTRY_PROXY23:39
EmilienMI think https://review.openstack.org/618062 is enough23:39
EmilienMwe'll see23:39
mwhahahai don't23:39
mwhahahabut we'll see23:40
EmilienMmwhahaha: what's missing?23:40
mwhahahathe code that would actually set the var23:41
EmilienMmwhahaha: https://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci-quickstart/config/testenv/multinode.yml#n1023:41
EmilienMthat?23:41
EmilienMhttps://git.openstack.org/cgit/openstack-infra/tripleo-ci/tree/toci-quickstart/config/testenv/singlenode.yml23:41
EmilienMwhich is a symlink to multinode23:41
mwhahahahmm k23:42
mwhahahawe'll see23:42
openstackgerritAlex Schultz proposed openstack/tripleo-heat-templates master: Add python shim for docker config scripts  https://review.openstack.org/61802723:46
*** chem has joined #tripleo23:49
openstackgerritSorin Sbarnea proposed openstack-infra/tripleo-ci master: Avoid ignoring failure to collect missing folders  https://review.openstack.org/59407723:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!