Tuesday, 2021-06-15

rlandyweshay|ruck: hey - around?00:00
rlandyhttps://logserver.rdoproject.org/openstack-periodic-integration-stable1/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-8-multinode-1ctlr-featureset010-wallaby/3946fd7/logs/undercloud/home/zuul/overcloud_deploy.log.txt.gz00:00
rlandy2021-06-14 22:47:48.679 99564 INFO tripleoclient.heat_launcher [-] Pulling conatiner image quay.io/tripleomaster/openstack-heat-api:current-tripleo.[00m00:00
rlandywhy is a periodic job pulling from quay.io????00:00
rlandysent email to ruck/rover00:56
*** ykarel|away is now known as ykarel04:18
*** marios is now known as marios|ruck05:16
*** ysandeep|out is now known as ysandeep06:09
*** jpena|off is now known as jpena06:34
anbanerj|rovermarios|ruck, O/07:18
marios|rucko/ anbanerj|rover 07:20
marios|ruckanbanerj|rover: few mirror related fails this morning filed that just now for tracking 'limestone' related https://bugs.launchpad.net/tripleo/+bug/1931975 07:22
anbanerj|roverthanks marios|ruck 07:25
anbanerj|rovermarios|ruck, https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-scenario010-standalone-train&job_name=periodic-tripleo-ci-centos-8-scenario010-ovn-provider-standalone-train these are still hitting https://bugs.launchpad.net/tripleo/+bug/1881087 Do you have an idea what it might be07:27
marios|ruckanbanerj|rover: ack will check it out in a bit07:29
anbanerj|roverthanks07:29
*** ykarel is now known as ykarel|lunch07:38
marios|ruckanbanerj|rover: fyi https://bugs.launchpad.net/tripleo/+bug/1931995 its the bug that rlandy emailed about 09:48
anbanerj|rovermarios|ruck, thanks09:50
*** ykarel|lunch is now known as ykarel09:57
* marios|ruck food biab10:20
*** sshnaidm|afk is now known as sshnaidm11:21
*** ysandeep is now known as ysandeep|afk11:24
*** jpena is now known as jpena|lunch11:25
zbrmarios|ruck: i am ok to use experiment, the only issue is that zuul does not advertise the existence of optional pipeline jobs when it runs, making this feature discoverability hard for newcomers.11:48
zbrbut I like it more than my placeholder approach11:49
marios|ruckzbr: yeah but not sure what is less discoverable, the check experimental or touching the right file11:50
*** amoralej is now known as amoralej|lunch11:50
marios|ruckzbr: either way you have to check zuul.d/layout i guess :/11:50
zbrin fact there is nothing special about layout.file, zuul treats all files under zuul.d the same way: combines them.11:50
zbrbut i kinda like the split I did between integration and tox. BTW, that feature was already used under zuul-jobs, is not something new at all.11:51
zbrit may prove useful for cases where we have lots of jobs11:51
zbris Not found image: https://registry-1.docker.io/v2/tripleomaster/openstack-ironic-pxe/manifests/ee4aecfe06de7e8ca63aed041b3e42a8 known?11:59
zbrfound it on https://zuul.opendev.org/t/openstack/build/8e04dea97f45428eb2ac5007d418937c/log/logs/undercloud/home/zuul/undercloud_install.log 11:59
rlandymarios: thanks for looking into that bug12:00
rlandymarios|ruck: ^^12:00
marios|ruckrlandy: ack 12:00
marios|ruckrlandy: i am not sure why it isn't on the cix board though what did i miss... checking12:01
* rlandy looks12:01
marios|ruckrlandy: i think we can sell it to DF folks for further work12:01
rlandyadded a milestone12:01
rlandynot sure if that helps12:01
marios|ruckrlandy: ah thanks ;) yeah probably 12:01
rlandyit should be the CIX12:02
rlandyit's blocking 1712:02
rlandymarios|ruck: will ping kevin about it when he comes on line12:02
marios|ruckrlandy: ack 12:02
rlandywas working with him to try get the right 17 containers12:03
rlandycommented on the bug12:04
rlandyweshay|ruck: shoot - we may have just missed promoting 16.212:14
rlandypipeline kicked12:14
rlandymay be the same hash12:14
weshay|ruckrlandy, well.. I'm getting good at proting hashes that are not at tripleo-ci-testing12:14
weshay|ruckif needed12:15
rlandyweshay|ruck: let's see what this hash becomes12:15
rlandyit may be the same12:15
*** jpena|lunch is now known as jpena12:17
rlandymarios|ruck: weshay|ruck: card from https://bugs.launchpad.net/tripleo/+bug/1931995 hit the CIX board12:17
rlandyI would place it in blocked12:17
marios|ruckrlandy: ack saw it thanks12:17
rlandyit's blocking 1712:17
rlandyhel8-osp16-2/promoted-components/delorean.repo.md5 hash will rev12:22
*** ysandeep|afk is now known as ysandeep12:33
weshay|ruckakahat, anbanerj|rover marios|ruck have you folks synced up re: extended_hash and the upstream promoter12:39
marios|ruckweshay|ruck: no 12:39
marios|ruckakahat: i guess i can start digging into that i think akahat is away?12:40
marios|ruckweshay|ruck: i mean^^ 12:40
weshay|ruckakahat, should be here12:42
marios|ruckweshay|ruck: k i recalled someone saying he is out for a few days in one of our calls yesterday b12:42
weshay|ruckbhagyashryi will probably be out.. for some undetermined time...  I'll dig through and see what's up w/ Amol12:44
marios|ruckweshay|ruck: k i can also have a look in a bit 12:45
weshay|ruckmarios|ruck, anbanerj|rover shall I get c8-train promoted via dlrn? would you like me to show you12:45
anbanerj|roverweshay|ruck, I am joining the tmux12:45
marios|ruckweshay|ruck: we should bring this up in retrospective but it isn't clear if we are saying 'only amol and bagyashris will work on the promoter from now on forever and ever' or what12:45
marios|ruckweshay|ruck: i.e. are we not meant to be working/looking at that (like for today's issue, liek for the train promotion we now need to do etc)12:45
marios|ruckweshay|ruck: k joining tmux looking for the ip12:46
weshay|ruckmarios|ruck, ya.. bring it up12:46
marios|ruck:D12:46
marios|ruckweshay|ruck: am i on the right tmux/promoter ?12:49
zbrmarios|ruck: weshay|ruck: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/796278 is now using experimental. Are we ok with this approach or we want to still stick one integration job in check and leave only multinode in experimental?12:50
weshay|ruckhttps://docs.google.com/document/d/1bHVPN-lcXyY7imLdvj2J7I55s-vtF3ffPG-y2Cj8u38/edit12:50
zbrthe cool bit about current config is that it did run in 20mins :D12:50
anbanerj|rovermarios|ruck, can you see me ?12:50
*** amoralej|lunch is now known as amoralej12:51
marios|ruckweshay|ruck: yeah i was rejoining12:51
marios|ruckweshay|ruck: k so later then 12:52
marios|ruckweshay|ruck: we can do when you finish your call 12:52
weshay|ruckmarios|ruck, anbanerj|rover meet.google.com/tws-uddf-qwj13:01
weshay|ruckextended_hash && train quick chat13:01
bhagyashrismarios|ruck, weshay|ruck hey sorry i was out for one hr 13:01
marios|ruckbhagyashris: o/ i think weshay|ruck is in a meeting so we can sync on the community call in half hour i guess13:02
marios|ruckor after13:02
weshay|ruckmarios|ruck, we're live now.. can you join please13:07
marios|ruckweshay|ruck: there13:07
weshay|ruckhttps://meet.google.com/tws-uddf-qwj?authuser=113:07
marios|ruckhttps://review.rdoproject.org/zuul/buildset/faf65db4c8cd42a1a1fd15a91a416645 13:21
bhagyashrisarxcruz, zbr, sshnaidm, rlandy, marios|ruck , ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, weshay, chandankumar, frenzy_friday, dviroel anbanerj|rover 13:27
bhagyashrisCommunity call in 3 mins13:27
bhagyashrishttps://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both#2021-06-15-Community-call13:27
bhagyashrisweshay|ruck, marios|ruck anbanerj|rover community call13:32
bhagyashrisCommunity call done 13:36
marios|ruckbhagyashris: from promoter logs 2021-06-15 12:53:07,029 2665709 DEBUG    promoter Dlrn promote 'aggregate: 4fe8f6193cc25d6ac2778c25cd9e40a7, commit: 8644a02c70a1abec6298623d0734ed3d091c9886, distro: 871628e5c31aafc96db1390050dc1a6d7c62f2a9, extended: None, component: validation, timestamp: 1623693997' from tripleo-ci-testing to current-tripleo: adding 'commit: 92f51eadc73438fe3cb91c608aa47714a02d20e2, 13:40
marios|ruckdistro: 25252426db1f4a969ebd71450bc6625a23fd9eee, extended: None, component: validation,timestamp: 1622793879.0' to promotion list13:40
* bhagyashris brb14:06
anbanerj|roverHey zbr, I am trying to track the limestone timeout on logstash. https://504a6afcff55c8f91b0c-2937bb652abe1e5f0c19a6a7c68db9e1.ssl.cf5.rackcdn.com/796207/1/gate/tripleo-ci-centos-8-containers-multinode/3b786d2/logs/subnode-1/home/zuul/repo_setup.log This is the error log. But query - message:"Timeout was reached for " doesnt return anything , why is that?14:33
dviroelhey folks, don't forget to vote on Y release name14:35
dviroelhttps://www.surveymonkey.com/r/RHR3NCY14:35
marios|ruckyoda surely14:38
zbranbanerj|rover: probably because this errors was not uploaded14:38
anbanerj|roverwhere can I see if repo_setup.log was included in the logs sent to logstash? I found something like this https://504a6afcff55c8f91b0c-2937bb652abe1e5f0c19a6a7c68db9e1.ssl.cf5.rackcdn.com/796207/1/gate/tripleo-ci-centos-8-containers-multinode/3b786d2/logs/undercloud/var/log/extra/logstash.txt14:39
zbrYOLO but they knew not to include that option 14:39
rlandyarxcruz: hey ... the multinode-ips is test is failing as I think it's not running tempest test on master - what should be defined here?14:39
rlandyhttps://github.com/openstack/tripleo-ci/blob/master/zuul.d/standalone-jobs.yaml#L77314:39
rlandyhttps://8e993c4e2eea342df1d7-7bcd0c6ba9078d6047b64bbe393931c0.ssl.cf1.rackcdn.com/789718/11/check/tripleo-ci-centos-8-standalone-on-multinode-ipa/d9c0434/job-output.txt14:39
rlandyarxcruz: ^^ nvm - wrong link14:40
rlandythey are running but failing14:40
rlandynever the less we should probably reconsider what runs there14:41
arxcruzrlandy: checking, but all tests are failing, i believe it's something related to bring up the cirros vm 14:42
arxcruzall images are in ERROR status 14:43
rlandyarxcruz: thanks14:43
dviroeli prefer Ygritte: "you know nothing cloud operator"14:44
arxcruzthe libvirt is not being able to read the certificate and fail to connect so 14:47
*** ykarel_ is now known as ykarel14:56
weshay|ruckmarios|ruck, anbanerj|rover bhagyashris fyi https://softwarefactory-project.io/r/c/DLRN/+/2211215:03
marios|ruckweshay|ruck: thx15:06
*** jpena is now known as jpena|off15:18
ysandeepweshay|ruck, rlandy i think octavia component still need promotion for 16.215:22
ysandeeptripleo-ci-testing: python-octavia-tests-tempest-1.4.1-2.20210611094817.f7718ef.el8osttrunk.src.rpm15:22
ysandeepcurrent-tripleo: python-octavia-tests-tempest-1.4.1-2.20210528115847.f7718ef.el8osttrunk.src.rpm15:22
rlandyysandeep: yep - we know about that15:25
rlandypassed in last night's hash15:25
rlandywe missed promoting the has this morning15:25
rlandywill promoting this afternoon15:25
rlandywatching the line now15:25
ysandeeprlandy: ack, i was reading wes comment on ocatavia component https://docs.google.com/document/d/1dA8m_0g-KRYweL4JXvJ4LsRsn_Yo0rjk8Y4RU3_045Y/edit#heading=h.2u5prbewqce and got confused.15:26
ysandeep~~~15:26
ysandeepthis package is promoted and should be in the compose http://osp-trunk.hosted.upshift.rdu2.redhat.com/rhel8-osp16-2/component/octavia/current-tripleo/python-octavia-tests-tempest-1.4.1-2.20210528115847.f7718ef.el8.src.rpm15:26
rlandywe have the same scen010 error in current line15:27
ysandeepnot sure if this is related: ERROR octavia.api.drivers.driver_factory [-] Unable to load provider driver ovn due to: duplicate option: host: oslo_config.cfg.DuplicateOptError: duplicate option: host 15:32
ysandeephttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2/25494b0/logs/undercloud/var/log/containers/octavia/octavia.log last line15:32
weshay|ruckarxcruz, https://quay.io/organization/tripleowallaby15:40
weshay|ruckrlandy, ^15:41
arxcruzweshay|ruck: ack, i'll add the wallaby on the copy script 15:41
*** ysandeep is now known as ysandeep|out15:41
marios|ruckweshay|ruck: fyi i had a go at the queens/stein draft there https://docs.google.com/document/d/1RfIfs7vxc3Xusn-qkviiHY0HlxTYysH1oIlqe9N_Ibw/edit# but not going to send it now... have a look and if you agree we can get it out soon 15:45
rlandyysandeep|out: yep same error15:47
weshay|ruckk15:48
arxcruzweshay|ruck: organization created \15:52
arxcruzweshay|ruck: but i'm not being able to copy from trunk.rdo 15:52
arxcruzERRO[0013] Failed to copy container image:  Error in copy the image: Error initializing source docker://trunk.registry.rdoproject.org/tripleowallaby/openstack-ironic-conductor:9d6dce6c445f21c730cc3a02339765a0_x86_64: Error reading manifest 9d6dce6c445f21c730cc3a02339765a0_x86_64 in trunk.registry.rdoproject.org/tripleowallaby/openstack-ironic-conductor: manifest unknown: manifest15:52
arxcruzunknown 15:52
*** amoralej is now known as amoralej|off15:53
weshay|ruck      "metadata": {15:54
weshay|ruck        "name": "openstack-ironic-conductor:current-tripleo",15:54
weshay|ruck        "namespace": "tripleowallaby",15:54
weshay|ruck        "selfLink": "/oapi/v1/namespaces/tripleowallaby/imagestreamtags/openstack-ironic-conductor%3Acurrent-tripleo",15:54
weshay|ruck        "uid": "9b8962ef-8ba8-11eb-87c9-fa163e4759f3",15:54
weshay|ruck        "resourceVersion": "76303312",15:54
weshay|ruck        "creationTimestamp": "2021-06-03T17:26:01Z"15:54
weshay|ruckarxcruz, ^15:54
weshay|ruckhttps://registry.rdoproject.org:8443/oapi/v1/namespaces/tripleowallaby/imagestreamtags/15:54
*** marios|ruck is now known as marios|out15:57
weshay|ruckrlandy, I think you may be bumping into emphemeral heat.. in 17 now.. ugh.. bad timing https://sf.hosted.upshift.rdu2.redhat.com/logs/84/195884/103/check/periodic-tripleo-ci-rhel-8-multinode-1ctlr-featureset010-rhos-17/c8f8f95/logs/undercloud/home/zuul/overcloud_deploy.log16:01
weshay|ruckopenstack overcloud deploy: error: unrecognized arguments: --heat-container-api-image docker-registry.upshift.redhat.com/tripleorhos-17/openstack-heat-api:331995955d7fa2dcbe10f812890082e4 --heat-container-engine-image docker-registry.upshift.redhat.com/tripleorhos-17/openstack-heat-engine:331995955d7fa2dcbe10f812890082e416:01
weshay|rucknot sure16:01
weshay|ruckbut that's what it looks like16:01
weshay|ruckarxcruz, moving forward or need eyes?16:01
arxcruzweshay|ruck: checking 16:01
weshay|ruckarxcruz, looks like some things are going https://quay.io/organization/tripleowallaby16:02
rlandyweshay|ruck: there is a bug opne16:06
rlandyopen16:06
rlandyper the email I sent last night16:06
rlandyI tried adding those deploy arguments16:07
rlandybut I guess I did it wrong16:07
rlandyreally that is a hack workaround16:07
rlandydiscussion is on #tripleo16:07
rlandykebin, harold, cedric are on it16:07
rlandyCIX created16:07
rlandyI was just trying those options16:07
rlandybut really even with those it would be in complete16:08
rlandy<hjensas> rlandy: hm, overcloud deploy command seem to only have --heat-container-api-image and --heat-container-engine-image args. Not the --heat-container-image16:08
rlandyweshay|ruck: ^^ still didn't like my options16:08
weshay|ruckrlandy, ya.. we may not have all the patches landed in 1716:09
rlandyweshay|ruck: either way  they need a proper fix ofr it16:09
rlandyI commented in the LP16:09
rlandyweshay|ruck: the same problem exists in master and wallaby16:11
rlandyit's just not being detected16:11
rlandyusing containers from quay current-tripleo in promotion jobs16:11
weshay|ruckthat's a new option I think16:11
weshay|ruckI'd have to compare w/ upstream16:11
rlandywe picked it up in 17 because the registry it defaults to requires auth16:11
rlandyit needs to be fixd upstream as well16:12
rlandythis is  the response I got from asking if I should use those options: rlandy: there are those options, yeah...... still, not sure this is the right way.16:12
rlandylunch - brb16:18
weshay|ruckanbanerj|rover, any updates on the mirror query?16:34
weshay|ruckhttps://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/79600716:35
anbanerj|roverweshay|ruck, no, looks like it is not even being sent to logstash. Do you remember which directory showed the list of files collected by logstash?16:35
anbanerj|roverhttps://504a6afcff55c8f91b0c-2937bb652abe1e5f0c19a6a7c68db9e1.ssl.cf5.rackcdn.com/796207/1/gate/tripleo-ci-centos-8-containers-multinode/3b786d2/logs/undercloud/var/log/extra/logstash.txt I found this but this is different16:36
weshay|ruckya.. if it's a log from dlrn builds only.. we're not getting it16:38
anbanerj|roverIt is in /logs/subnode-1/home/zuul/repo_setup.log 16:40
weshay|ruckanbanerj|rover, not if it's the package being built16:41
weshay|ruckanbanerj|rover, 16:41
weshay|ruckhttps://meet.google.com/vnv-asqt-dnf16:41
weshay|ruckanbanerj|rover, https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_752/796207/1/gate/tripleo-ci-centos-8-standalone-upgrade-ussuri/75236a3/logs/undercloud/home/zuul/repo_setup.log16:45
weshay|ruckanbanerj|rover, https://hackmd.io/5uYRmLaOTI2raTbHWsaiSQ#commit_hash-vs-build_id16:55
* ysandeep|out leaving for the day 16:57
arxcruzweshay|ruck: all set, containers coppied, current-tripleo tagged 17:07
weshay|ruckarxcruz, thank you!17:20
weshay|ruckrlandy, need anything?17:53
rlandyweshay|ruck: a large chocolate cake, thanks for asking17:54
weshay|ruckotw17:54
rlandyweshay|ruck: waiting for baremetal to complete17:54
weshay|ruckyou can get anything in nyc in 20min17:54
rlandywell both in component and in integration17:54
rlandythen we promote integration17:54
weshay|ruckrlandy, ur hungry because you went for a run yesterday17:54
rlandymaybe17:55
* rlandy need to set up dep line with composes17:55
rlandyweshay|ruck: need me to cover anything tomorrow?17:56
weshay|ruckjust the prod-chain-council17:57
rlandyweshay|ruck: marios covering program call?17:58
weshay|ruckaye17:58
rlandywatching octavia17:58
rlandywe may get something there17:58
rlandyweshay|ruck: do you still have powers to kick lines in downstream?17:58
rlandyperiodic-tripleo-ci-rhel-8-ovb-1ctlr_2comp-featureset020-internal-rhos-16.2 success17:59
rlandywow17:59
rlandyshould screenshot that17:59
rlandyfs001 also a success17:59
weshay|ruckvictoria promoted...18:38
weshay|ruckwallaby is promoting now...  18:38
* weshay|ruck steps away for a bit18:38
rlandyweshay|ruck: http://osp-trunk.hosted.upshift.rdu2.redhat.com/api-rhel8-osp16-2/api/civotes_agg_detail.html?ref_hash=7ac5d80bb444f6c089d97fc412b1ac4f18:43
rlandypromoting 16.218:43
rlandyweshay|ruck: pls ping when you are back ... setting up https://github.com/openstack/tripleo-quickstart/blob/master/config/release/dependency_ci/centos8stream/repo_config.yaml for the compose repos19:15
weshay|ruckrlandy, hey19:16
rlandyweshay|ruck: so I am looking at composes19:17
weshay|ruckk19:17
weshay|ruckrlandy, use latest19:17
rlandygive me one sec19:17
rlandyjust checking bm hash on qe jobs19:17
rlandyok good qe job is running now as well19:18
rlandyk - back to composes19:18
rlandyweshay|ruck: so yeah ... that was my question19:18
rlandylatest is what is BEFORE it hits mirrors, right?19:19
rlandynot what is IN mirrors?19:19
rlandyif we run the job as that link changes, we will test every change19:19
weshay|ruckrlandy, latest compose19:20
weshay|rucknot production afaict..19:20
rlandyok19:20
weshay|ruckrlandy, and it's a static url19:20
rlandyweshay|ruck: let's talk timing19:20
rlandyevery day?19:20
rlandywhat when that link updates?19:20
weshay|ruckwelcome to coffee talk ladies and gentleman19:20
rlandymy cake didnt arrive19:21
weshay|ruckonce a day seems good.. especially for c819:21
weshay|ruckwhat's ur addr?19:21
rlandyok - let's start there19:21
weshay|ruckI'll fix that19:21
rlandyjoking19:21
rlandyI live a 5 min walk from the closes bakery19:21
rlandyyou said coffee, reminded me of cake19:21
rlandyalright setting that up19:22
weshay|rucka bisil ruggalah19:22
rlandy:)19:22
rlandynah - I shouldn't be eating all that sugar19:22
weshay|ruckrlandy, anything else?19:23
rlandyweshay|ruck: think that is it19:24
rlandy16.2 promoted19:24
rlandywaiting on BM to complete19:24
rlandythen we need to kick 16.2 again19:24
rlandyweshay|ruck: ^^ you still have those powers?19:24
rlandyI would guess in about two hours or so19:25
rlandyweshay|ruck: ping, ping20:47
rlandytoo eraly20:47
rlandyhttps://sf.hosted.upshift.rdu2.redhat.com/logs/72/190672/49/check/periodic-tripleo-ci-rhel-8-bm_envB-3ctlr_1comp-featureset001-baremetal-rhos-16.2/faa19da/20:48
rlandyjust completed20:48
rlandygoing on line to promote bm20:48
*** sshnaidm is now known as sshnaidm|afk21:15
weshay|ruckrlandy, need me to dequeue?21:44
weshay|ruckrlandy, sorry had to step away for a min21:44
weshay|ruckah I see you got it21:45
rlandyweshay|ruck: got it eventually21:45
rlandyhad to redo my conf21:45
weshay|ruckrlandy ya.. ++21:45
rlandysorry for the panic21:45
weshay|ruck:)21:46
weshay|rucknp21:46
rlandyweshay|ruck: if this queue passes, going to promote again21:49
rlandyeven if it's late21:49
weshay|ruckk21:50
* weshay|ruck grabs kids21:51
weshay|ruckback22:25

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!