Tuesday, 2020-04-14

*** openstack has joined #oooq09:53
*** ChanServ sets mode: +o openstack09:53
*** dpawlik has quit IRC09:55
*** ysandeep|lunch is now known as ysandeep09:55
*** dpawlik has joined #oooq09:56
marioschandankumar: recheck roullette :/10:00
marioslooks like fbo is out today :/ he sorted it last time (NODE_FAILURES)10:00
*** owalsh_afk is now known as owalsh10:20
*** ysandeep is now known as ysandeep|afk10:37
*** jmasud has quit IRC10:41
zbrarxcruz: marios I need review of artcl https://review.opendev.org/#/c/717723/ -- which also has a hackmd document attached to to its taiga ticket.10:45
zbrafaik the filter was the only thing preventing role use under zuul10:46
zbrarxcruz: I used a depends-on on https://review.opendev.org/#/c/702676/ which was created by you and IMHO it did work well. can you double check?10:46
arxcruzzbr:  sure, i already review your patch, sorry, it was holiday here in germany friday and monday :)10:47
arxcruzgetting back today10:47
zbrsame here10:47
zbri am really curious if it does behave correctly under zuul now, this would be the first thing we needed to check.10:47
marioszbr: ack will add to list10:48
*** chem has quit IRC10:50
*** chem has joined #oooq10:52
*** ysandeep|afk is now known as ysandeep|rover10:53
*** pojadhav is now known as pojadhav|brb11:08
*** apetrich has quit IRC11:09
*** apetrich has joined #oooq11:11
weshay|rucksoniya29, 1-111:31
*** udesale_ has joined #oooq11:34
*** apetrich has quit IRC11:34
*** jpena is now known as jpena|lunch11:36
weshay|ruckchandankumar, is soniya29 there?11:36
*** udesale has quit IRC11:36
weshay|ruckchandankumar, HA.. we're all at home nvrmind11:36
soniya29weshay|ruck, yeah11:38
*** ysandeep|rover is now known as ysandeep|coffee11:39
*** apetrich has joined #oooq11:41
*** ysandeep|coffee is now known as ysandeep11:51
*** ratailor has quit IRC11:52
ysandeepweshay|ruck, hey o/ good morning! Fyi.. we tested Revert of "only use tripleo-ansible in required-projects for train+" https://review.rdoproject.org/r/#/c/26418/ and its working. testproject passed - https://review.rdoproject.org/r/#/c/26415/ . So we think removing branch for stein(tripleo-ansible) was enough11:54
*** ysandeep is now known as ysandeep|rover11:56
marioschandankumar: arxcruz: kopecmartin: seen that one before ? 2020-04-14 06:00:56 | /var/lib/tempest/tempest_container.sh: line 47: /usr/bin/discover-tempest-config: No such file or directory @@ https://logserver.rdoproject.org/94/715394/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/9461a21/logs/undercloud/home/zuul/tempest.log.txt.gz12:01
mariosysandeep|rover: cc fyi ^^^ i didn't see it on the ruck|rover notes12:02
ysandeep|rovermarios, checking ^^12:02
weshay|ruckysandeep|rover, ah k12:03
*** pojadhav|brb is now known as pojadhav12:03
weshay|ruckysandeep|rover, where is the test?12:03
ysandeep|roverweshay|ruck, your test patch https://review.rdoproject.org/r/#/c/26415/ .. i just updated the commit message depends on12:04
ysandeep|rovermarios, not seen this before, I will open a bz for this.12:06
mariosysandeep|rover: k ... might be specific to fs39 ... don't worry about the bz/you sound busy i can file one if we need it thanks12:06
ysandeep|rovermarios, hmm.. fs39 looks in bad shape https://review.rdoproject.org/zuul/builds?job_name=tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset03912:08
mariosysandeep|rover: yeah i'm working there ... jus the job was green earlier so wondering if something changed in the tempest config is why i ping chandankumar arxcruz and kopecmartin12:09
mariosysandeep|rover: thanks for looking12:10
weshay|ruckcan folks help me and double check this page renders for them http://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline?orgId=112:10
arxcruzit seems python-tempestconf wasn't installed properly12:10
weshay|ruckfyi.. this is not working for migi atm12:10
ysandeep|rovermarios, okay, my pleasure :)12:10
mariosweshay|ruck: looks ok here /firefox... lemme try chrome12:10
mariosarxcruz: ah... arxcruz i'm missing something? or you mean that's what the error says/means12:11
kopecmartinmarios: i'm not aware of any changes in python-tempetsconf which could cause that error, it seems it's not installed correctly within the container12:11
marioskopecmartin: arxcruz: thanks for checking... could it be somethign there? https://review.opendev.org/#/c/715397/8/config/general_config/featureset039.yml  ... but as i said, i had green runs with that before... not sure what/where i'm missing12:11
kopecmartinthe tempest container is a black magic, we have a bug for that12:11
ysandeep|roverweshay|ruck, only seeing 2 jobs under baremetal.. i remember we used to have long list of jobs there..12:13
weshay|ruckysandeep|rover, where you looking?12:14
*** amoralej is now known as amoralej|lunch12:14
*** rlandy has joined #oooq12:14
ysandeep|roverweshay http://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline?orgId=1&fullscreen&panelId=42212:15
weshay|ruckysandeep|rover, change the date time http://dashboard-ci.tripleo.org/d/FE8Hf29Wz/component-pipeline?orgId=1&fullscreen&panelId=422&from=now-7d&to=now12:15
rlandymarios: hey!12:16
rlandypls tell me we have working nodes today12:16
kopecmartinmarios: tempest container is also doing 'rpm -qa | grep tempest' https://logserver.rdoproject.org/94/715394/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/9461a21/logs/undercloud/home/zuul/tempest_container.sh.txt.gz12:17
ysandeep|roverweshay|ruck, yeah- No data to show12:17
kopecmartinmarios: but when you check the log https://logserver.rdoproject.org/94/715394/4/openstack-check/tripleo-ci-centos-8-ovb-3ctlr_1comp_1supp-featureset039/9461a21/logs/undercloud/home/zuul/tempest.log.txt.gz12:17
kopecmartinmarios: no python-tempestconf is there .. it's a dependecy problem12:17
kopecmartinpython-tempestconf wasn't installed within the container12:17
mariosrlandy: no ... recheck roullette \o/12:18
mariosplace yo bets12:18
marioskopecmartin: thank you for pointers... will have a closer look in a bit noting those for now12:18
rlandymarios: I lost hands down last night12:18
chandankumarrlandy, morning12:19
rlandychandankumar: hi12:19
weshay|ruckchandankumar, so re: pcsd.. are we suspecting the images are missing files/rpms again?12:19
weshay|ruckhttps://trello.com/c/VX93jXg8/1391-cixlp1867602tripleociproa-overcloud-deploy-failed-due-overcloud-image-missing-some-folders-under-var12:19
weshay|ruckysandeep|rover, ^12:19
chandankumarrlandy, weshay|ruck panda marios https://hackmd.io/A8YPJTyTSju2vb1wFxAPNg?view12:19
chandankumarweshay|ruck, I think so, not looked at that patch today, will work now12:19
rlandymarios: yay!! at least we are getting some credit now - ttps://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/27/check/periodic-tripleo-build-containers-rhel-8-master/720cd85/logs/containers-successfully-built.log12:22
mariosrlandy: yes :D12:22
rlandybefore it was reporting no success12:22
mariosrlandy: ... fighting the healthcheck links at the moment like https://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/27/check/periodic-tripleo-build-containers-rhel-8-master/720cd85/logs/buildah-builds/kolla-builds/800cc1b7-06ac-4c9c-bf98-8f0f944757b1/docker/barbican-worker/barbican-worker-build.log chmod: cannot operate on dangling symlink '/openstack/healthcheck'12:23
rlandymarios: should we chat about that?12:23
rlandyI attempted a change yesterday12:23
mariosrlandy: sure lets have our call early? ... in 7 mins ok?12:23
rlandymarios: actually 37 mins12:24
rlandylet's do it12:24
mariosrlandy: no i mean 15:23 < rlandy> marios: should we chat about that?12:24
mariosrlandy: like i know we have it in 37 mins and that is fine by me i thought you wanted to talk earlier12:24
* marios runs for coffee either way 12:24
rlandypojadhav: ^^ going to meet with marios quickly- will ping you when meeting is done to continue tls12:25
pojadhavrlandy, sure12:25
weshay|ruckchandankumar, do you have any test project jobs that are running the rpm -V command yet12:31
weshay|ruckchandankumar, interested because master promotion is blocked on the issue atm12:31
rlandychandankumar: fs002 for master?12:33
chandankumarrlandy, yes12:33
rlandymarios: in gm when you are ready12:33
mariosrlandy: o/12:34
mariosrlandy: joining now12:34
chandankumarweshay|ruck, give me a min12:34
rlandychandankumar: will catch up with you after meetings12:34
weshay|ruckchandankumar, aye12:35
chandankumarweshay|ruck, https://review.rdoproject.org/r/2643012:37
chandankumarand weshay|ruck https://review.opendev.org/#/c/718946/12:37
chandankumarmarios, I will update the comments once I am done with experiments regarding image build12:38
*** jpena|lunch is now known as jpena12:40
weshay|ruckthanks12:42
* weshay|ruck watches12:42
marioshttps://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/24/check/periodic-tripleo-build-containers-rhel-8-master/9147bea/job-output.txt12:43
mariosrlandy: ^12:44
rlandymarios: http://git.app.eng.bos.redhat.com/git/openstack-tripleo-common.git/tree/healthcheck?h=rhos-17.0-trunk-patches12:44
chandankumarweshay|ruck, rlandy regarding downstream image server, we can reuse the PSI hosted internal log server, apevec will work on getting new namespace from IT pnt12:53
chandankumarrfolco, weshay|ruck pre sprint planning meeting gmeet link?13:00
rlandychandankumar: sounds good13:00
weshay|ruckchandankumar, rfolco https://meet.google.com/ywt-ctnk-skw?authuser=113:00
rlandypojadhav: ready ... https://meet.google.com/oiv-geho-mai13:01
pojadhavrlandy, yup13:01
*** amoralej|lunch is now known as amoralej13:03
*** Goneri has joined #oooq13:10
weshay|ruck"sprint coordination for ussuri release"13:16
weshay|ruckrfolco, chandankumar ^13:16
chandankumarweshay|ruck, https://hackmd.io/gfRO_KFRRAurN8dwTpnIIQ13:21
rlandypojadhav: https://github.com/openstack/tripleo-quickstart/blob/master/config/general_config/featureset039.yml13:21
chandankumarweshay|ruck, https://tree.taiga.io/project/tripleo-ci-board/us/160613:21
*** TrevorV has joined #oooq13:22
*** mjturek1 has joined #oooq13:24
chandankumarysandeep|rover, is something wrong with tripleo gate jobs?13:26
chandankumarweshay|ruck, rfolco https://review.rdoproject.org/r/#/c/26340/13:29
ysandeep|roverchandankumar, i don't see any failed job in gate queue - http://tripleo-cockpit.usersys.redhat.com/d/9DmvErfZz/cockpit?orgId=1&from=now%2Fd&to=now%2Fd but dashboard is not giving latest info... checking particularly13:32
*** jmasud has joined #oooq13:37
*** jmasud has quit IRC13:39
chandankumararxcruz, https://hackmd.io/A8YPJTyTSju2vb1wFxAPNg?view13:39
mariosrlandy: ... we can't patch ... that's the cost of our kolla directory hacks.. i.e. we can't patch it... https://code.engineering.redhat.com/gerrit/#/c/196858/28/playbooks/tripleo-buildcontainers/pre.yaml@10713:43
mariosrlandy: gonna try a direct copy for now so we can proceed13:43
rlandymarios: you can patch before our hacks13:43
mariosrlandy: yeah but then we nuke everything under docker/* so we can't do thtat13:45
mariosrlandy: it *would* apply before our hacks but then we are replacing all the contents of kolla/docker/13:45
rlandymarios: k - dircet copy - whatever is easiest13:46
*** ykarel is now known as ykarel|away13:50
*** derekh has quit IRC13:56
*** derekh has joined #oooq13:56
*** derekh has quit IRC13:59
*** derekh has joined #oooq14:00
rlandymarios: how's the node situation going? worth pinging rhos-ops or ok now?14:14
mariosrlandy: haven't seen it in a few runs now.. lets see14:15
*** pojadhav is now known as pojadhav|brb14:15
mariosrlandy: both me and chandankumar did ping rhos-ops earlier but looks like folks might be out... perhaps you have better luck in US timezone but anyway seems gone ... ish for now14:15
rlandypojadhav: hey - want to continue?14:16
rlandymarios: yeah - will try the USA-timezone people14:16
rlandypojadhav|brb: sorry - ready to continue when you get back - or if it's late for you, can leave email14:18
*** mjturek1 is now known as mjturek14:19
ysandeep|roverweshay|ruck, seeing random failures https://tinyurl.com/wygjx3n , checking for common point14:19
*** udesale_ has quit IRC14:20
weshay|ruckysandeep|rover, looking14:21
weshay|ruckysandeep|rover, hrm.. I wonder if I broke the cockpit in my latest change14:22
weshay|ruckbecause we're not seeing these http://dashboard-ci.tripleo.org/d/wb8HBhrWk/cockpit?orgId=1&fullscreen&panelId=6114:22
* weshay|ruck reverts14:22
weshay|ruckysandeep|rover, https://review.rdoproject.org/r/#/c/26437/14:23
weshay|ruckysandeep|rover, chandankumar was trying to help rami w/ heat monitoring14:23
weshay|ruckchandankumar, can we chat about overcloud images for a min..14:24
ysandeep|roverweshay|ruck, ack14:24
weshay|rucksince we're out of mtgs .. w/ ysandeep|rover14:24
weshay|ruckrlandy, ysandeep|rover don't let me forget.. next sprint I'd like ysandeep|rover to start learning our bm env14:24
ysandeep|roverweshay|ruck, will surely do, i would love to learn our bm envs and help rlandy manage them.14:25
ysandeep|roverweshay|ruck, failues in gate so far seems related to particular patches - https://review.opendev.org/#/c/717186/ and https://review.opendev.org/#/c/713439/ .. But you will now in sometime once more patches comes in gate queue14:26
weshay|ruckysandeep|rover, I saw a lot of mirror failures this morning14:27
pojadhav|brbrlandy, ready now :) can sync up14:27
weshay|ruckno action required14:27
*** pojadhav|brb is now known as pojadhav14:27
ysandeep|roverweshay|ruck, https://review.opendev.org/#/c/719542/ will help in future14:28
rlandyysandeep|rover: weshay|ruck: sure - I'm out wed, thurs - can pick up after that14:29
*** sshnaidm has joined #oooq14:29
ysandeep|roverrlandy, sure, thank you!14:29
weshay|ruckysandeep|rover, aye https://trunk.rdoproject.org/centos8-master/build-deps/latest/noarch/14:29
weshay|ruckI see mock :)14:29
*** sshnaidm has quit IRC14:30
weshay|ruckysandeep|rover, we need a testproject change that has a depends-on w/ an open openstack change14:31
weshay|ruckfor https://review.opendev.org/#/c/719542/314:31
weshay|ruckysandeep|rover, know what I mean?14:31
weshay|ruckysandeep|rover, do my comments make sense?14:32
rlandypojadhav: k - let's continue ... on google meet14:33
ysandeep|roverweshay|ruck, https://review.opendev.org/#/c/719553/14:33
pojadhavrlandy, yup14:33
ysandeep|roverchecking what failed on the test patch.14:35
ysandeep|rovertempest failues14:35
ysandeep|roverweshay|ruck, see you tomorrow.. if you gets some time please review my https://review.rdoproject.org/r/#/c/26418/14:39
ysandeep|rovertested here https://review.rdoproject.org/r/#/c/26415/14:39
pojadhavrlandy, https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/freeipa-setup/templates/deploy_freeipa.sh.j2#L84-L12414:40
weshay|ruckysandeep|rover, thanks14:41
weshay|ruckreviews please https://review.opendev.org/#/c/719542/14:44
chandankumarweshay|ruck, yes14:47
weshay|ruckchandankumar, 0/ let's chat for 2 min.. max14:47
weshay|ruckchandankumar, https://meet.google.com/jhv-dgzc-fpt?authuser=114:48
weshay|rucksorry14:48
*** ysandeep|rover is now known as ysandeep|away14:50
rlandyweshay|ruck: can you schedule me a few minutes this afternoon to touch about next sprint before my PTO?14:55
weshay|ruckrlandy, yup14:58
weshay|ruckrlandy, available now14:59
rlandyweshay|ruck: talking with pojadhav  - will ping in a few14:59
rlandyweshay|ruck: k- ready to meet when you are https://meet.google.com/ntz-jpua-heu15:08
rlandyweshay|ruck: https://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/31/check/periodic-tripleo-build-containers-rhel-8-master/3bcb5cf/logs/15:10
amoralejweshay|ruck, are you hitting issue https://b1aa75b1b80b47ce21cf-3a8bdf82174e44fa5c863ba9b47d4ff0.ssl.cf1.rackcdn.com/717186/5/gate/tripleo-ci-centos-8-standalone/b0faa3b/logs/undercloud/home/zuul/repo_setup.log in more places?15:15
amoralejykarel mentioned it to me but i can't reproduce it locally15:16
amoralejnot sure if it's specific to some job or ci environment15:16
*** pojadhav is now known as pojadhav|dinner15:23
*** bogdando has quit IRC15:47
mariosrlandy: getting ready to go in a little while.. just watching the latest run now https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/df7caba3d64d44659197c2f49a9fd261?logfile=console.log15:58
mariosrlandy: i did a core.dump() @ https://code.engineering.redhat.com/gerrit/#/c/196858/32/playbooks/tripleo-buildcontainers/pre.yaml  for where/what i did today to help you pickup if you have time today15:58
mariosrlandy: i will continue in my morning15:58
rlandymarios: reading back15:59
rlandycool - will pick up the work now16:00
mariosrlandy: ack let the current run finish please first :D16:00
* marios watching 16:00
mariosrlandy: you'll see in my notes/status i left on pre... the collectd thing didn't work/even direct copy caused an error (commented there with details) so i just -readded it to the upstream skip list16:01
mariosfor now (tm)16:02
rlandymarios: that's fine16:02
rlandymarios: if all we have to skip is one container - we'll be lucky16:03
rlandymarios: really more interested in seeing the healthcheck work16:03
mariosrlandy: yes i am waiting to check that now... just failed... i ssupect its the same but waiting for the logs to come in16:03
mariosrlandy: damn same16:07
rlandymarios: k - I'll try another approach16:07
marios* https://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/32/check/periodic-tripleo-build-containers-rhel-8-master/df7caba/logs/buildah-builds/kolla-builds/63512df3-762a-41e7-92f1-00ccd203a768/docker/octavia-api/octavia-api-build.log  * https://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/32/check/periodic-tripleo-build-containers-rhel-8-master/df7caba/logs/build.log  *16:08
marioshttps://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/32/check/periodic-tripleo-build-containers-rhel-8-master/df7caba/logs/containers-successfully-built.log16:08
mariosrlandy: k good luck then16:08
* marios hands off patches16:09
rlandythanks - I'll need it16:09
*** sanjayu__ has quit IRC16:10
rlandy2020-04-14 15:42:02.334774 | TASK [debug]16:11
rlandy2020-04-14 15:42:02.415946 | primary | ok:16:11
rlandy2020-04-14 15:42:02.416209 | primary | {16:11
rlandy2020-04-14 15:42:02.416249 | primary |   "<class 'dict'>": "VARIABLE IS NOT DEFINED!: "16:11
rlandy2020-04-14 15:42:02.416321 | primary | }16:11
mariosrlandy: yeah that was a debug but don't worry it isn't fatal16:12
mariosrlandy: can just remove that16:12
rlandymarios: that copy should have worked16:12
mariosrlandy: i think you need to add stdoutlines to the result to see it16:12
mariosrlandy: the copy did work at least it seems to execute the task OK16:12
rlandyyep16:12
rlandywonder if it's a permissions thing16:13
mariosrlandy: might be ... i initially tried sudo in shell but instead using become after digging a bit16:13
mariosrlandy: but 2020-04-14 15:41:59.130416 | TASK [Copy openstack-tripleo-common to /usr/share]16:13
marios2020-04-14 15:42:02.179542 | primary | ok: Runtime: 0:00:02.44468016:13
mariosso don't know16:13
rlandymarios: k - no - I'll try a few more tricks16:13
rlandynp16:14
*** jmasud has joined #oooq16:14
*** rlandy is now known as rlandy|brb16:14
*** marios is now known as marios|out16:20
*** marios|out has quit IRC16:25
*** pojadhav|dinner is now known as pojadhav16:26
weshay|ruckzbr, you know anything about download-logs.sh? in the root dir of all the upstream logs16:34
zbrweshay|ruck: not so much, but I seen it about two weeks ago.16:34
weshay|ruckzbr, does it work for you?16:35
zbrsend me a link on one and i will test it.16:35
*** rlandy|brb is now known as rlandy16:35
zbrnever used it myself.16:35
weshay|ruckzbr, lazy https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_1b5/719542/3/check/tripleo-ci-centos-8-standalone/1b55bee/download-logs.sh16:36
zbri got a 40416:38
zbrwhile running the file16:38
zbrwas this from a on old/recycled build?16:38
zbrhmm, that is weird, is from today.16:39
zbrlet me ask on intra16:40
*** pojadhav is now known as pojadhav|away16:40
amoralejweshay|ruck, chandankumar centos7 is not longer used for master, right?16:43
weshay|rucknope16:44
amoralejwe are doing final cleanuop16:44
amoralejcleanup16:44
amoralejso will definetively fail to build anything in centos716:44
amoralejand we've stopped dlrn builder centos7-master16:44
amoralejjust fyi16:44
weshay|ruckk. .thanks16:45
weshay|ruckamoralej, I saw16:45
weshay|ruck2020-04-14 10:55:16 |  Problem: cannot install the best update candidate for package openssl-libs-1:1.1.1c-2.el8.i68616:45
weshay|ruck2020-04-14 10:55:16 |   - openssl-libs-1:1.1.1c-2.el8_1.1.i686 has inferior architecture16:45
weshay|ruckearlier..16:45
weshay|ruckamoralej, was there a change related to that?16:45
amoralejno16:45
amoralejcentos was updated16:46
amoralejbut openssl-libs update was 4 days ago16:46
weshay|ruckk16:46
amoralejlooks like issue with mirror or something like that16:47
amoralejif you see it, let us know16:47
amoraleji tried to reproduce it with the same exact mirror16:47
amoralejbut i couldn't16:47
weshay|ruckk.. thanks for poking at it16:47
zbrweshay|ruck: answer from upstream: 404 is expected for aborted buildsets, it is planned to be fixed but is not very simple.16:52
weshay|ruckaborted?16:52
weshay|ruckwas that aborted?16:53
weshay|ruckzbr, it's not16:53
*** derekh has quit IRC16:54
zbrbuild*set*16:54
weshay|ruckhttp://zuul.openstack.org/status search 71954216:54
weshay|ruckzbr, that job is green passed done16:54
zbrsomething else may have triggered an abort of it, clark said is expected in that case.16:54
*** jmasud has quit IRC16:54
weshay|ruckzbr, how is that in anyway aborted?16:54
*** jmasud has joined #oooq16:55
zbrlets see what #openstack-infra has to comment on this....16:56
*** dtantsur is now known as dtantsur|afk16:57
*** jbadiapa has quit IRC17:02
zbrweshay|ruck: it will start to work when the entire builset will finish.17:03
weshay|ruckOH17:03
weshay|ruckzbr, k :)17:03
zbrbut while is still running it does not work. yep, known, planned to be fixed, some day.17:03
weshay|ruckmakes more sense17:03
weshay|ruckheh17:03
weshay|ruckk17:03
zbrbut i will  make a PR to put a clean warning instead of the confusing 40417:03
weshay|ruckzbr++17:05
*** jpena is now known as jpena|off17:05
weshay|ruckzbr, hey.. should I be able to run tox molecule locally w/ py3717:10
weshay|ruckfor ci-config rdo17:10
zbrweshay|ruck: if your py37 has libselinux on redhat platforms, yes.17:15
zbrweshay|ruck: short answer: yes, molecule is tested with py37 on ci too.17:16
weshay|ruckwhy is this dumb manager failing17:16
zbrin fact we may even have py38 testing, not sure.17:16
weshay|ruckzbr, have python3 virtenv17:16
weshay|ruckrunning tox molecule; tox molecule_delegated17:16
zbrtry it, if it fails is not due to molecule itself.17:16
weshay|ruckgetting molecule and molecule_delegated not found17:16
weshay|ruckalso added py37 in tox.ini.. didn't help... doing $something stupid17:16
weshay|ruckreport has 0 tests run17:17
weshay|ruckthis worked in the past.. for me, I *think*17:17
*** Goneri has quit IRC17:23
weshay|ruckzbr, am I crazy or stupid or both.. should just be #tox molecule no?17:27
zbr"tox -e molecule"17:27
weshay|ruckDOH.. I knew that .. FAK17:28
zbrwhich repo?17:28
weshay|ruckhttps://github.com/rdo-infra/ci-config17:28
zbrtox output can be bit confusing when you forget -e, because it will run default testenv with a posarg17:28
weshay|ruckya.. thank you.. I'm being stupid.. thanks for the correction17:29
zbrso "tox molecule" does run "tox -etestenv -- molecule" if I rememeber well, based on what the default env does, you would get something like that.17:29
zbrno problem, a pleasure when the fix is o simple17:29
*** amoralej is now known as amoralej|off17:30
weshay|ruck:)17:31
weshay|ruckit's working.. and NICE17:31
*** Goneri has joined #oooq17:52
*** dpawlik has quit IRC18:02
weshay|ruckrfolco, rlandy will need 5 min to review get_hash again.. have the tests passing now.. but could use serious review https://review.rdoproject.org/r/#/c/26346/18:11
rlandyweshay|ruck: cool - looking18:11
*** jmasud has quit IRC18:12
rlandyweshay|ruck: left comments on the use of release ... let me know what you think18:20
rlandythe rest looks good18:20
weshay|ruckk.. rfolco how about you?18:20
weshay|ruckah.. see +218:21
rfolcoweshay|ruck, gave my +2 already18:21
* weshay|ruck looks at comments18:21
rfolcok18:21
rlandyweshay|ruck: iow - release is always upstream and we add osp_release|default(release)18:22
rlandywe have some of that code somewhere18:22
weshay|ruckrlandy, was thinking we'd add a when: osp-release is defined and have two tasks18:22
weshay|ruckrlandy, I can add that in now18:22
weshay|rucksee what comes up from it18:22
rlandyweshay|ruck: don't worry about the current dlrn jobs downstream - I'll fix them to work with the final version of this change18:23
weshay|ruckrlandy, so osp_release can not be.. osp-18:26
weshay|ruckfor this to work18:26
weshay|ruckjust tried18:26
rlandyugh /o\18:26
rlandyweshay|ruck: ok -  if it works in the upstream it's ok18:27
rlandywe can rework it for downstream in a follow up18:27
rlandywe can make a combined var18:27
weshay|ruckretrying18:29
weshay|ruckrlandy, take it back18:29
weshay|ruckrhos- works18:29
rlandyosp-17 - ok?18:29
weshay|ruckrlandy, k. will add a working stub for osp18:29
weshay|ruckrelease: train18:30
weshay|ruckosp_release: rhos-1618:30
weshay|ruckyes that works18:30
rlandycool18:30
*** jmasud has joined #oooq18:30
weshay|ruckrlandy, k.. latest patch should work w/ internal18:32
rlandylooking18:32
weshay|ruckrlandy, and in jobs that don't use a release file.. we'll add osp_release to zuul18:32
weshay|ruckshould work both ways then18:32
rlandysure18:32
rlandyweshay|ruck: looks good - we need just one more step with osp_release18:34
weshay|ruckwhat's that?18:35
rlandyweshay|ruck: hmm18:37
rlandylet me rethink that18:37
rlandy dlrn_baseurl[release] }18:38
rlandydlrn_hash_baseurl: "{{ dlrnapi_protocol }}://{{ dlrnapi_host }}/{{ dlrn_baseurl[release] }}"18:38
weshay|ruckskip list review please https://review.opendev.org/#/c/720050/18:39
rlandydlrn_baseurl:18:39
rlandy  master: centos8-master18:39
rlandyweshay|ruck: ^^18:39
weshay|ruckrlandy, which line?18:39
weshay|ruckoh I see18:39
rlandyhttps://review.rdoproject.org/r/#/c/26346/9/ci-scripts/infra-setup/roles/get_hash/tasks/get_hash.yaml18:39
rlandy19 where the comment is18:39
rlandySTEP 6: RUN mkdir -p /openstack &&     ln -s /usr/share/openstack-tripleo-common/healthcheck/aodh-api /openstack/healthcheck &&     chmod a+rx /openstack/healthcheck18:40
rlandychmod: cannot operate on dangling symlink '/openstack/healthcheck'18:40
rlandywhy, oh why are you still failing??????18:40
rlandyweshay|ruck: +2 on https://review.opendev.org/#/c/720050 - seen that on and off for ages18:43
rlandyinfra issue?18:43
weshay|ruckupdated18:43
weshay|ruckrlandy, no.. I bet it's a tempest setting18:43
rlandyweshay|ruck: +2 on https://review.rdoproject.org/r/#/c/26346/ - let's try it18:46
weshay|ruckrlandy, rfolco thanks for the reviews18:46
rlandyweshay|ruck: question on this dangling symlink error ...18:46
weshay|ruckaye18:46
rlandySTEP 6: RUN mkdir -p /openstack &&     ln -s /usr/share/openstack-tripleo-common/healthcheck/aodh-api /openstack/healthcheck &&     chmod a+rx /openstack/healthcheck18:46
rlandychmod: cannot operate on dangling symlink '/openstack/healthcheck'18:46
rlandywe've been doing all sorts of things to copy  /usr/share/openstack-tripleo-common/healthcheck on the node18:47
rlandybut iiuc, this is looking for  /usr/share/openstack-tripleo-common/healthcheck on the container?18:47
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/35/check/periodic-tripleo-build-containers-rhel-8-master/4881853/logs/buildah-builds/kolla-builds/5f4800a4-1e77-4bb8-beca-a748fddc7636/docker/aodh-api/aodh-api-build.log18:47
weshay|ruckrlandy, is the root cause; downstream is openstack-tripleo-common and upstream is tripleo-commmon?18:48
rlandyweshay|ruck: I don;t think so ...18:49
rlandyln -s /usr/share/openstack-tripleo-common/healthcheck/aodh-api18:49
weshay|ruckrlandy, and you've tried cp -R /usr/share/openstack-tripleo-common to /usr/share/tripleo-common18:49
weshay|ruckso both are present?18:49
rlandy^^ where is that?18:49
weshay|ruckrlandy, let's chat18:49
rlandyon the node or on the container?18:49
rlandyok18:49
weshay|ruckcontainer I think18:49
weshay|ruckhttps://meet.google.com/mny-zqpz-cht?authuser=118:49
weshay|ruckrlandy, https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/800b384/logs/buildah-builds/kolla-builds/653d2f91-bd70-4c60-8254-470ad1babe54/docker/aodh/aodh-api/aodh-api-build.log18:59
weshay|ruckopenstack-tripleo-common-container-base19:07
weshay|ruckrlandy, ^19:07
weshay|ruckrlandy, https://logserver.rdoproject.org/openstack-periodic-master/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-centos-8-master-containers-build-push/800b384/logs/buildah-builds/kolla-builds/653d2f91-bd70-4c60-8254-470ad1babe54/docker/base/base-build.log19:08
*** jfrancoa has quit IRC19:16
*** jmasud has quit IRC19:42
*** Goneri has quit IRC19:57
*** Goneri has joined #oooq19:58
*** jmasud has joined #oooq20:16
weshay|ruckrlandy, ok.. that change worked for my job :)20:18
weshay|ruckhad to merge it though.. we're both running container builds right now :)  WINNER TAKES ALL20:18
rlandyweshay|ruck: which change?20:19
weshay|ruckget_hash20:19
rlandyah ok20:19
rlandypressure is on now :)20:19
weshay|ruckhttps://review.rdoproject.org/zuul/stream/5779d8b7fd914c6baf714f0e8b4e6538?logfile=console.log20:19
weshay|ruckmy build20:19
weshay|ruckyour build https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/stream/483e5bb94bac47ccb399c62d18e8a6af?logfile=console.log20:19
weshay|ruckRACE20:19
weshay|ruckI started building containers 4 min before you did20:20
weshay|ruckI HAVE THE LEAD!!20:20
rlandyno fair!!!20:20
weshay|rucklet's see which cloud is faster20:20
weshay|ruckrlandy, Emilien thinks including that rpm will solve the issue20:20
rlandylet's hope so20:20
rlandywe'll see20:20
rlandyweshay|ruck: tls question ...20:21
rlandyI am ignoring the free-ipa install role we had (as it's used in fs039 and I don;t want to mess with that)20:21
weshay|ruckya.. don't mess w/ it :)20:22
rlandyand adding a new task within the multinode ipa role that pretty much copies a lot of https://gitlab.cee.redhat.com/OSP-DFG-security/upshift-ansible/blob/master/roles/freeipa/tasks/main.yaml20:22
rlandyso we will have a divergence of the two for a bit20:22
rlandyuntil we prove out the new task20:22
weshay|ruckwhich lines are we duplicating20:22
rlandyone sec - will commit changes20:23
weshay|ruckwould prefer to see us add a role to tripleo--ipa20:23
rlandythere is no install role in https://opendev.org/x/tripleo-ipa/src/branch/master/tripleo_ipa/roles20:30
weshay|ruckrlandy, there can be though20:32
rlandyweshay|ruck: https://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/38/check/periodic-tripleo-build-containers-rhel-8-master/483e5bb/logs/build.log20:32
rlandy2020-04-14 16:28:14 | ============= SUMMARY ===============20:32
rlandy2020-04-14 16:28:14 | Expected................... 9120:32
rlandy2020-04-14 16:28:14 | Built...................... 7520:32
rlandy2020-04-14 16:28:14 | =====================================20:32
rlandy2020-04-14 16:28:14 |20:32
weshay|ruckah20:32
weshay|ruckbetter20:32
weshay|ruckmine built but did not push :(20:33
weshay|ruckhttps://logserver.rdoproject.org/72/26172/11/check/periodic-tripleo-centos-8-train-containers-build-push/5779d8b/job-output.txt20:33
weshay|ruck POST-RUN START: [trusted : review.rdoproject.org/config/playbooks/tripleo-ci-periodic-base/containers-push-podman-login-push.yaml@master]20:33
weshay|ruck2020-04-14 20:25:47.289128 |20:33
weshay|ruck2020-04-14 20:25:47.289255 | PLAY [primary]20:33
weshay|ruck2020-04-14 20:25:47.314488 |20:33
weshay|ruck2020-04-14 20:25:47.314613 | TASK [debug ansible_distribution]20:33
weshay|ruck2020-04-14 20:25:47.361515 | ansible_distribution = CentOS20:33
weshay|ruck2020-04-14 20:25:47.392670 |20:33
weshay|ruck2020-04-14 20:25:47.392752 | TASK [debug ansible_distribution_major_version]20:33
weshay|ruck2020-04-14 20:25:47.441976 | ansible_distribution_major_version = 820:33
weshay|ruck2020-04-14 20:25:47.464346 |20:33
weshay|ruck2020-04-14 20:25:47.464424 | TASK [debug zuul_success]20:33
weshay|ruck2020-04-14 20:25:47.507515 | zuul_success = True20:33
weshay|ruck2020-04-14 20:25:47.530664 |20:33
weshay|ruck2020-04-14 20:25:47.530757 | TASK [debug push_containers_podman]20:33
weshay|ruck2020-04-14 20:25:47.579736 | push_containers_podman = True20:33
weshay|ruck2020-04-14 20:25:47.602047 |20:33
weshay|ruck2020-04-14 20:25:47.602150 | TASK [Install podman]20:33
weshay|ruck2020-04-14 20:25:55.316560 | primary | changed20:33
weshay|ruck2020-04-14 20:25:55.368993 |20:33
weshay|ruck2020-04-14 20:25:55.369220 | TASK [Retrieve built images]20:33
rlandyweird20:33
weshay|ruck2020-04-14 20:25:56.449291 | primary | ok: Runtime: 0:00:00.00562720:33
weshay|ruckvs. TASK [Retrieve built images]20:33
weshay|ruck2020-04-14 08:43:26.625334 | primary | trunk.registry.rdoproject.org/tripleomaster/centos-binary-ironic-pxe:2a321efa8e5174dc4094eb3743f1b3a720:33
weshay|ruck2020-04-14 08:43:26.625389 | primary | trunk.registry.rdoproject.org/tripleomaster/centos-binary-octavia-api:2a321efa8e5174dc4094eb3743f1b3a720:33
weshay|ruckrlandy, so.. better to build the install role in their role.. so they can review it20:34
weshay|ruckhrm.. because this is blank https://logserver.rdoproject.org/72/26172/11/check/periodic-tripleo-centos-8-train-containers-build-push/5779d8b/logs/containers-successfully-built.log20:34
weshay|ruckbut they all worked20:35
weshay|ruckhttps://logserver.rdoproject.org/72/26172/11/check/periodic-tripleo-centos-8-train-containers-build-push/5779d8b/logs/buildah-builds/kolla-builds/09351314-7919-4045-8061-c5b1d955dc4c/docker/heat/heat-base/heat-base-build.log20:35
rlandyhttps://review.opendev.org/#/c/706373/48..49/roles/ipa-multinode/tasks/ipaserver-subnode-install.yml20:35
weshay|ruckso.. I'll look at the script20:35
rlandyweshay|ruck: ^^ ipa-instal diff20:35
weshay|ruckis there anything ci specific there?20:36
rlandyweshay|ruck: re: blank - that happened to us as well20:36
weshay|ruckrlandy, is it random?20:36
rlandylook at the criteria - we changed it ...20:36
rlandygetting20:36
weshay|ruckshould I fix or try again20:36
weshay|ruckI think it's the shell script.. j220:36
weshay|ruckrlandy, let's talk to ade about https://review.opendev.org/#/c/706373/48..49/roles/ipa-multinode/tasks/ipaserver-subnode-install.yml20:37
rlandyhttps://review.opendev.org/#/c/717895/16/roles/build-containers/templates/build.sh.j220:37
rlandyok20:37
rlandyit's looking for tripleo-ci_branch20:37
rlandymaybe stable>20:37
weshay|ruckah20:37
weshay|ruckk20:38
weshay|ruckrlandy, let's see if ade is cool w/ adding the install steps to the tripleo-ipa20:38
rlandyweshay|ruck: he said if we duplicate he would include them20:39
rlandyI just want to see if it works firdt20:39
rlandyfirst20:39
weshay|ruckrlandy, k20:39
rlandyyou know me, always expect the worst20:39
weshay|ruckrlandy, then wf -1 it.. and just use for testproject20:39
weshay|rucksudo {{ container_cli }} images | grep tripleo{% if osp_branch is defined %}{{ osp_branch }}{% else %}{{ ci_branch }}{% endif %}/{{ kolla_base }}-binary > containers-successfully-built.log20:40
weshay|ruckI mean.. why do we need to grep for the branch20:40
weshay|ruckin theory in ci the system is always clean20:40
rlandyweshay|ruck: it's the way the containers are named20:42
rlandybut either way20:42
rlandyNo match for argument: python3-networking-mlnx20:42
rlandyError: Unable to find a match: python3-networking-mlnx20:42
rlandy^^ neutron failure20:43
rlandymay be legit20:43
*** jmasud has quit IRC20:43
weshay|ruckmelonox20:43
rlandy Problem: package openstack-mistral-common-10.0.0-0.20200408162853.a7da00d.el8ost.noarch requires python3-mistral = 10.0.0-0.20200408162853.a7da00d.el8ost, but none of the providers can be installed20:44
rlandy  - conflicting requests20:44
rlandythe failures now are all dependencies20:44
rlandyweshay|ruck: ^^ try and fix that?20:44
weshay|ruckno.. that we have to report to delivery20:44
weshay|ruckrlandy, we probably now should just use osp-trunk20:44
weshay|ruckif they greenlight that20:44
weshay|ruckI bet there are some things missing20:44
rlandyweshay|ruck: time to pop open the wine then?20:44
weshay|ruckand some things conflicting20:45
weshay|ruckrlandy, aye.. really nice progress this sprint20:45
weshay|ruckFROM IMPOSSIBLE20:45
weshay|ruckto almost there20:45
rlandymission IS possible20:45
weshay|ruckPUNNY!20:45
weshay|ruckI love it20:45
rlandywell - other than the fact we only have 91 containers20:46
rlandyquestion for jjoyce when he returns20:46
weshay|ruckrlandy, John as well20:47
weshay|ruckcan help20:47
rlandyasking about container numbers20:47
*** jmasud has joined #oooq20:52
* fuzzball81 waives at rlandy20:57
*** fuzzball81 is now known as jjoyce20:57
jjoyceSorry my nick got changed20:57
jjoycefuzzball81 is my other nick20:58
weshay|rucklolz21:02
jjoycerlandy: Container number should be ~11621:02
weshay|ruckjjoyce, fooz! you fooz? we must fooz21:02
jjoycelol21:02
rlandyjjoyce: ack - we are shoet fo a few containers as the upstream sole has excludes .. we have 9221:03
rlandyshort of21:03
rlandyrole21:03
jjoyceIf you have a list we can help fill in the gaps.21:04
rlandyjjoyce: https://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/38/check/periodic-tripleo-build-containers-rhel-8-master/483e5bb/logs/containers-expected-to-build.log21:05
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/58/196858/38/check/periodic-tripleo-build-containers-rhel-8-master/483e5bb/logs/containers-successfully-built.log21:05
rlandy75 built successfully21:05
rlandyerrors on mistral and neutron, horizon21:06
rlandysue to package missing/conflicts21:06
rlandydue21:06
weshay|ruckrlandy, we should retry w/ just dlrn and dlrn deps.. and rhos-release w/ RHEL only21:07
weshay|ruckmixing the streams is too messy to debug21:07
rlandyweshay|ruck: yep21:07
weshay|ruckrlandy, rfolco pushing centos-8 train containers :)21:11
weshay|ruckhttps://registry.rdoproject.org:8443/oapi/v1/namespaces/tripleotraincentos8/imagestreamtags/21:11
weshay|ruckwoot21:11
rlandyweshay|ruck++21:11
rlandywoohoo21:11
rfolcoweshay|ruck, :)21:11
rlandyweshay|ruck: guess that makes you "winner takes all"21:11
weshay|ruckmeh.. osp much harder.. but ya.. nanny nanny boo boo21:12
weshay|ruck:P21:12
rlandyosp lives to fight another day - good enough for now21:13
*** ccamacho has quit IRC21:36
rlandyweshay|ruck: tripleo-operator for standalone ... off the top of your head - do you know how to add other options like: --control-virtual-ip={{ control_virtual_ip }}22:22
*** jmasud has quit IRC22:48
*** jmasud has joined #oooq23:00
*** rlandy has quit IRC23:00
*** sanjayu__ has joined #oooq23:05
*** TrevorV has quit IRC23:06
*** jmasud has quit IRC23:15
*** tosky has quit IRC23:20
*** jmasud has joined #oooq23:53

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!