Wednesday, 2022-05-04

* rcastillo leaving00:06
rlandydasm|ruck: updated program doc00:33
rlandydasm|ruck: you around?00:33
*** rlandy is now known as rlandy|out00:33
rlandy|outdasm|ruck: tomorrow will explain the diff between lines and promotions00:33
dasm|ruckrlandy|out: ack00:40
dasm|ruckrlandy|out: i think we went through that in the past already. But yeah, another refresher is welcome00:41
*** ysandeep|sick is now known as ysandeep05:02
*** marios is now known as marios|ruck05:05
marios|ruck\o/05:05
marios|ruck\/o\\05:06
marios|ruck---05:06
dasm|ruckmarios|ruck: it's bad :/05:07
dasm|ruckit seems like some of issues went away, but not all of them. swap enablement for tempest didn't resolve the issue05:08
marios|ruckdasm|ruck: morning ....05:11
marios|ruckpretty late for you dasm|ruck :)05:11
marios|ruckdasm|ruck: thanks i'll take it now  go sleep ! :D05:11
marios|ruckdasm|ruck: see you in ~9 hours 05:11
dasm|ruckmarios|ruck: i'm about to go to sleep05:11
dasm|ruckmarios|ruck: good luck mate!05:11
marios|ruckdasm|ruck: 05:11
marios|ruckdasm|ruck: thanks05:12
* marios|ruck prepares sacrifice for zuul05:12
dasm|ruckmarios|ruck: are you gathering a lot of wood and kindling? to fire things up?05:12
marios|ruckdasm|ruck: right :)05:13
dasm|ruck;)05:13
dasm|rucktalk to you tomorrow.05:13
marios|rucko/05:13
*** dasm|ruck is now known as dasm|ruck|off05:13
* pojadhav brb in half 05:37
pojadhavhour05:37
*** jpena|off is now known as jpena06:59
*** pojadhav is now known as pojadhav|lunch07:38
*** pojadhav|lunch is now known as pojadhav07:58
pojadhavfolks, is https://review.opendev.org/ working ?08:08
pojadhavI am getting unreachable...08:08
chandankumarpojadhav: yes08:09
marios|ruckpojadhav: been fine here today so far 08:13
pojadhavchandankumar, marios|ruck : thanks!08:14
pojadhavworking fine now.. may be my network issue.08:15
ysandeepreviewbot, please add in review list: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/405717 08:26
reviewbotI have added your review to the Review list08:26
*** ysandeep is now known as ysandeep|lunch08:27
*** ysandeep|lunch is now known as ysandeep09:42
*** rlandy|out is now known as rlandy10:15
rlandymarios|ruck: hi - let's touch base on OVB 10:17
marios|ruckrlandy: o/10:20
marios|ruckrlandy: you mean about the swap etc (commented https://bugs.launchpad.net/tripleo/+bug/1971465/comments/4 10:21
rlandymarios|ruck: about next stpes10:22
rlandysteps10:22
marios|ruckrlandy: gimme 2mins just writing a comment10:22
marios|ruckrlandy: k whenever you're ready 10:25
rlandymarios|ruck: ack ... https://meet.google.com/tot-cxzq-aqc?pli=1&authuser=010:25
marios|ruckrlandy: joining10:25
jm1arxcruz, frenzy_friday, rcastillo, sshnaidm: hopefully this will fix our tox_extra_args/tags issues.. https://review.opendev.org/c/openstack/ansible-collections-openstack/+/84043710:47
rlandyarxcruz: is there an easy way to split the tempest tests running on fs001?11:02
rlandyso we can run less tests per job11:02
rlandyso maybe a fs001a and fs001b - where we run half the tests in each?11:03
arxcruzrlandy what you mean ? we can reduce yes 11:03
arxcruzrlandy fs001 is already in allow-list 11:03
arxcruzwe just need to setup the group properly 11:04
arxcruzhttps://opendev.org/openstack/openstack-tempest-skiplist/src/branch/master/roles/validate-tempest/vars/tempest_allow.yml#L1711:04
rlandyarxcruz: we'd need some kind of regexp to match half the services on one and half on the other11:07
rlandygood way to do that?11:07
arxcruzrlandy we can split by service11:07
arxcruzlike tempest.api.network on one, and tempest.api.computer on other11:08
arxcruzright now the regex is tempest.api11:08
arxcruzso it run everything 11:08
rlandyarxcruz: ack - like a-m and and m-z?11:08
arxcruzrlandy yes 11:09
rlandyarxcruz: can you help us testproject that? ... so featureset001_periodic_a and featureset001_periodic_b with the right regexps?11:13
*** dviroel|out is now known as dviroel11:20
arxcruzrlandy yes11:38
chandankumarrlandy: https://opendev.org/openstack/python-tripleoclient/src/branch/master/tripleoclient/tests/test_utils.py#L235511:46
chandankumarrlandy: https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_podman/tasks/main.yml#L3211:48
chandankumartripleo_container_registry_login11:48
* dviroel power outage here11:49
* dviroel needs to make coffee to the old ways11:54
*** pojadhav is now known as pojadhav|brb12:04
akahatrlandy, https://review.rdoproject.org/r/c/testproject/+/42434 sc10 jobs green run12:04
rlandyakahat++ thanks  - saw that - success on last run12:18
rlandymarios|ruck: dasm|ruck|off: https://sf.hosted.upshift.rdu2.redhat.com/logs/84/195884/148/check/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17/ea38482/logs/undercloud/var/log/tempest/stestr_results.html.gz and https://sf.hosted.upshift.rdu2.redhat.com/logs/84/195884/148/check/periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset035-internal-rhos-17/031df8d/logs/undercloud/var/log/tempest/stestr_results.ht12:18
rlandyml.gz12:18
rlandyone tempest fail on eacj run12:18
rlandydiff test12:18
rlandyif the current run does not pass, will merge that hash12:18
soniya29kopecmartin, chandankumar, ysandeep, rlandy, arxcruz, please add/edit today's agenda for tempest meeting?12:19
soniya29https://hackmd.io/fIOKlEBHQfeTZjZmrUaEYQ12:19
rlandysoniya29: just the failing OVB jobs12:19
rlandywhich we are debugging with arxcruz and chandankumar 12:20
soniya29rlandy, added to the agenda12:21
*** ysandeep is now known as ysandeep|coffee12:22
kopecmartinsoniya29: i can't attend , i still have a conflict with another meeting, let me bring that up there , i'll try to reschedule the other meeting next time12:22
soniya29kopecmartin, sure12:22
kopecmartinanyway, if you have anything i can help with, ping me and i'll have a look12:22
soniya29kopecmartin, ack12:23
soniya29pojadhav|brb, ^^12:24
rlandysoniya29: we are raising on CIX - so no worries12:30
rlandyc8 just kicked12:31
rlandywatching image build there12:31
soniya29rlandy, okay12:32
arxcruzrlandy https://review.rdoproject.org/r/c/testproject/+/4255412:33
arxcruztesting splited tests 12:33
rlandyarxcruz++ thank you12:39
rlandymarios|ruck: see https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/84044412:39
rlandyhttps://review.rdoproject.org/r/c/testproject/+/4255412:39
marios|ruckrlandy: ack (split jobs?)12:40
marios|ruckwill check in a bit12:40
marios|ruckthanks arxcruz 12:40
rlandymarios|ruck: ack ... temptest test split12:41
rlandygoing to try a train run with that as well12:41
chandankumararxcruz: what is cni?12:45
chandankumarfeatureset001_periodic_cni12:46
arxcruzchandankumar cni stands for computer, network, identity 12:46
arxcruzovi stands for object_storage, volume, image 12:46
chandankumarokok12:47
rlandyarxcruz: chandankumar: trying the same with a train patch12:56
rlandylet's see if it gets us by12:56
chandankumarack12:56
chandankumarysandeep|coffee: please have a look at https://review.rdoproject.org/r/c/rdo-jobs/+/42527 and https://review.rdoproject.org/r/c/config/+/42226 when free, thanks!12:58
soniya29rlandy, tempest meeting?13:01
*** ysandeep|coffee is now known as ysandeep13:01
ysandeepchandankumar, ack for reviews13:01
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.13:01
dpawlikrlandy: hey, copied from #openstack-tc "<fungi> also it's worth reminding, the master branch of e-r is essentially abandoned now, the rdo branch is the only one being maintained as far as i'm aware so it should probably be merged to master and the rdo branch itself deleted"13:02
rlandydpawlik: ok by me13:02
rlandyarxcruz: frenzy_friday: ^^13:02
arxcruzoh, nice 13:03
*** pojadhav|brb is now known as pojadhav13:03
frenzy_fridaydpawlik, yep we can do that. Our only concern was that in the rdo branch we changed a few stuff to make it tripleo-ci specific. We changed stuff like jobs regexes (to match only the jobs we wanted to) , changed gerrit host to rdo etc13:06
frenzy_fridayCan we merge that to master?13:06
pojadhavsoniya29, https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/839588/4/roles/validate-tempest/vars/tempest_allow.yml13:06
dpawlikfrenzy_friday: can not answer. I did not check/compare branches in that project. It can be a good question on #openstack-infra to fungi/clarkb13:07
frenzy_fridayack, thanks13:07
rlandychandankumar: meeting?13:31
rcastilloo/13:39
*** ysandeep is now known as ysandeep|brb13:42
dasm|ruck|offo/13:43
*** dasm|ruck|off is now known as dasm|ruck13:43
rlandydasm|ruck: hey ...13:52
dasm|ruckrlandy: o/13:52
dasm|rucka lot has happened over the night.13:52
marios|ruck\o13:52
dasm|ruckor, actually -- not at all.13:52
marios|ruckdasm|ruck: did you have a good sleep :)13:52
rlandymarios|ruck: ^^ want to sync with dasm|ruck before CIX?13:52
marios|ruckrlandy: sure 13:52
dasm|ruckmarios|ruck: could be better. could use more hours13:52
marios|ruckrlandy: should be a quick one not too much to sync but lets go 13:53
marios|ruckrlandy: dasm|ruck: https://meet.google.com/kew-qjvt-gko?authuser=013:53
rlandydasm|ruck: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/84044413:55
rlandyhttps://review.rdoproject.org/r/c/testproject/+/2185313:55
marios|ruckhttps://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/840444 https://review.rdoproject.org/r/c/testproject/+/4255413:56
*** ysandeep|brb is now known as ysandeep14:20
ysandeepchandankumar, reviewed both of the patches, +wed one and +2 other(need second vote)14:20
chandankumarysandeep: thanks!14:20
chandankumardviroel: needs +w on this https://review.rdoproject.org/r/c/config/+/42226 thanks!14:21
* dviroel looks14:21
ysandeepsoniya29: hey o/ we are next ruck/rovers, if you are around please join cix so that you know what's going on.14:31
dasm|ruckysandeep: ovb is in despair14:32
ysandeepdasm|ruck: thanks, ronelle mentioned that in review call.14:33
dasm|ruckack14:33
ysandeepfinger crossed, hopefully all sort out before my and soniya29 turn as rr :) 14:34
dasm|ruckysandeep: we're hard working to make it better. But our attempts are getting ignored by ZUUL ;)14:41
* ysandeep prepares a sacrifice for zuul :D14:43
rlandydasm|ruck: marios|ruck: https://code.engineering.redhat.com/gerrit/c/tripleo-environments/+/406874 Temp remove fs035 from rhel-9 14:45
marios|ruckysandeep: good idea zuul likes that sort of thing14:46
marios|ruckrlandy: thanks14:46
dasm|ruckHeisen-bugs :)14:46
soniya29ysandeep, i was out for sometime :)14:57
ysandeepsoniya29: no worries o/14:58
soniya29ysandeep, o/14:58
soniya29pojadhav, ack14:58
ysandeepdasm|ruck, this ovb issue - when it started?15:01
dasm|ruckysandeep: it sounds like about few weeks back, after vexx updates to uefi.15:01
dasm|ruckysandeep: i tried to pull up stats, but there are so many issues that it's hard to pin down15:02
ysandeepack15:03
dasm|ruckV(-.o)V15:05
ysandeephttps://logserver.rdoproject.org/openstack-periodic-integration-main/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master/122791b/logs/baremetal_3-console.log 15:05
ysandeepsystemd-journald[748]: Data hash table of /var/log/journal/2993239009c741ca9c8c06dd337e68c3/system.journal has a fill level at 75.0 (3803 of 5070 items, 2920448 file size, 767 bytes per hash table item), suggesting rotation.15:05
ysandeepi wonder if ^^ have something to do with our issue, I can try to dig tomorrow in my morning.15:05
dasm|ruckysandeep: technically it shouldn't be the issue. 15:06
dasm|ruckesp, few secs later it auto-rotated15:06
dasm|rucks/secs/microsecs15:07
dasm|ruckzuul is back...ish15:17
* dviroel lunch15:18
*** dviroel is now known as dviroel|lunch15:18
*** ysandeep is now known as ysandeep|out15:19
chemdasm|ruck: marios|ruck hey ruck people, hope you're having a nice day :) I have a low hanging question about the reproducer that might need new code, but I would like some eyes on the lp before I push the code upstream15:24
chemdasm|ruck: marios|ruck https://bugs.launchpad.net/tripleo/+bug/1971605 when/if you have time.15:24
chemit seems a bit convoluted ... maybe I should have just pushed the code :) ..15:24
rlandyjm1: ^^ could you review pls for chem15:29
chandankumarsee ya!15:30
dasm|ruckchandankumar: o/15:30
rcastillochandankumar: bye o/15:30
dasm|ruckchem: ack. i'll check that in a bit15:31
dasm|ruckchem: push the code too. we might iterate on it15:31
chemdasm|ruck: ack15:32
rlandychandankumar: see you tomorrow - thanks15:32
marios|ruckchem: k will have a look but tomorrow i am going in few15:36
marios|ruckrlandy: dasm|ruck: o/ need something before i am out for the day 15:37
chemmarios|ruck: ack, meanwhile I'll push the code, have a good evening :)15:37
rlandymarios|ruck: a fix for OVB? if you have one in your back pocket :)15:37
rlandyotherwise no, will leave you notes where our investigation goes15:37
marios|ruckrlandy: ack thanks... i'll sleep on it ;)15:38
marios|ruckchem: you too o/15:38
dasm|ruckmarios|ruck: i'm good. thanks15:43
rlandydasm|ruck; put two testprojects running split tempest tests back into check in zuul15:44
rlandydasm|ruck: pls rekick wallaby c8 line15:44
*** marios|ruck is now known as marios|out15:45
rlandywas not restarted after sf upgrade15:45
dasm|ruckrlandy: ack. before i'll do so. by "rekick w cs8 line" do you mean periodic-integration?15:47
rlandyack ... openstack-periodic-integration-stable1-cs815:47
dasm|ruckack, ack.15:47
dasm|ruckon my way15:48
dasm|ruckdone. made mistake by kicking stable2 (victoria). but fixed myself with stable1-cs815:49
jm1rlandy: ack, will have a look later15:50
*** jpena is now known as jpena|off16:00
*** dviroel|lunch is now known as dviroel16:15
rlandylunch - brb16:20
dasm|ruckbrb16:46
dasm|ruckback16:56
rlandydasm|ruck: need help with anything?17:06
dasm|ruckno, everything seems to be fine, rlandy. thanks17:23
rlandydasm|ruck: pls keep an eye on the gate17:43
dasm|ruckack17:48
rlandyugh - deployment fails now17:56
rlandywhen trying to get tempest results17:56
rlandyperiodic-tripleo-ci-build-containers-ubi-8-push-wallabyopenstack/tripleo-ci50faeef35d14ba61a119dbbfe9f0680a3050d799openstack-periodic-integration-stable1-cs8000000017:59
rlandy2 hrs 2 mins 44 secs2022-05-04 15:52:35TIMED_OUT17:59
rlandydasm|ruck: ^^ can you rekick17:59
rlandycontainer builds timed out17:59
dasm|ruckack, yeah, a second ago it was still running18:00
rlandythanks18:07
rlandydasm|ruck: rekicked with newrev18:18
rlandythink it needs that 18:18
dasm|ruckack18:18
rlandydviroel: dasm|ruck; per the review meeting this morning ... 19:01
rlandyhttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/40663119:01
rlandygoing to merge this test review19:01
rlandyto try out new registry login19:01
rlandyshould not kill anything 19:01
rlandynew job definitions19:01
dasm|ruckack19:02
dasm|ruckrlandy: "famous last words" :)19:03
rlandylet's see :)19:03
dasm|ruck"should not kill anything"19:03
rlandyliving on the edge19:03
dasm|ruck"i don't usually test. but when i test, i test on production"19:03
dviroeldasm|ruck: lol19:05
rlandyrebasing julie's job to test19:05
rlandywish me luck19:05
dasm|ruckfingers crossed19:05
jm1rcastillo: zuul cfg patches have been merged, so i have rebased a couple of your patches. Let's see if your tests are executed this time :)19:19
rcastillojm1: ack, keeping an eye on that19:19
* jm1 out, see you tomorrow :)19:25
rlandyhmmm .. ok so that didn't work20:02
dasm|ruckhmmm... cs8 wallaby seems to be stuck again on container building20:07
dasm|ruck> 2022-05-04 18:54:35.243602 | TASK [build-containers : Run image build as ansible user > /home/zuul/workspace/logs/build.log]20:07
dasm|ruckit's for a while already.20:07
rlandydasm|ruck: sis you ping security re: fs039 failure?20:17
dasm|ruckno, forgot20:18
dasm|ruckrlandy: i forgot. just pinged ade on #tripleo20:20
rlandythanks20:22
rlandydasm|ruck: also - please prep the criteria changes for current-tripleo-rdo - if there are any21:12
rlandythe split tempest is a waste21:12
rlandykeep failing deploy21:13
rlandyhttps://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-8-push-wallaby&skip=021:14
dasm|ruckrlandy: current-tripleo-rdo criteria is up to date. no change needed there21:14
rlandydasm|ruck: great ... https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-build-containers-ubi-8-push-wallaby&skip=0 is failing21:15
rlandyhttps://logserver.rdoproject.org/openstack-periodic-integration-stable1-cs8/opendev.org/openstack/tripleo-ci/50faeef35d14ba61a119dbbfe9f0680a3050d799/periodic-tripleo-ci-build-containers-ubi-8-push-wallaby/b5cf215/logs/build.log21:16
rlandy2022-05-04 20:29:52 | 2022-05-04 20:29:52.205 38388 ERROR tripleoclient.v2.tripleo_container_image.Build [-] Buildah failed with the following error: The following jobs were incomplete: ['manila-base', 'novajoin-base', 'nova-base', 'octavia-base', 'tempest', 'glance-api']: SystemError: The following jobs were incomplete: ['manila-base', 'novajoin-base', 'nova-base', 'octavia-base', 'tempest', 'glance-api'][00m21:17
dasm|ruckyes, i'm seeing the same thing21:17
dasm|ruckbut i didn't find yet a root cause21:18
rlandyweird - no error21:19
rlandydasm|ruck: pls bug this21:19
dasm|ruckk21:19
dviroelmerging https://review.rdoproject.org/r/c/config/+/4222621:22
dasm|ruckk21:22
dviroeljust in case something breaks, revert21:22
dasm|ruckdviroel: so maybe wait until tomorrow's morning?21:22
rlandydviroel: do you see something that looks like it broke that>?21:23
dviroelrlandy: dasm|ruck: no - ysandeep did also a good double check on jobs being deleted21:23
dasm|ruck2022-05-04 20:29:52 | level=debug msg="Uploading /v2/tripleowallaby/openstack-manila-base/blobs/uploads/"21:24
rlandydasm|ruck: I think that line was kicked twice21:24
dasm|ruck2022-05-04 20:29:52 | level=debug msg="POST https://trunk.registry.rdoproject.org/v2/tripleowallaby/openstack-manila-base/blobs/uploads/"21:24
dasm|ruck2022-05-04 20:29:52 | level=debug msg="... already exists"21:24
dasm|ruck2022-05-04 20:29:52 | level=debug msg="Trying to mount /v2/tripleowallaby/openstack-manila-base/blobs/uploads/?from=tripleowallaby%2Fopenstack-tempest&mount=sha256%3A1615f362c3b9b1c30262cb5fee1e0a5e0bd193a8bc10f4c77f065faebc2aeb7c"21:24
dasm|ruckrlandy: ^21:24
rlandymaybe there was some overlap21:24
rlandydasm|ruck: let me kick it once on it's own21:24
dasm|ruckack21:24
dasm|ruckrlandy: fyi: https://bugs.launchpad.net/tripleo/+bug/197162921:25
rlandyall the lines are red21:25
rlandy [ERROR]: Container(s) which did not finish after 300 minutes:21:26
rlandy['neutron_db_sync']21:26
rlandy2022-05-04 18:16:58.370187 | fa163ea6-0ffe-261e-ae0c-0000000025bb |      FATAL | Check containers status | standalone | error={"changed": false, "msg": "Failed container(s): ['neutron_db_sync'], check logs in /var/log/containers/stdouts/"}21:26
rlandylet's just leave those lines for now and focus in master and stable 1 wallaby21:30
dasm|ruckmaster one is running here: https://review.rdoproject.org/r/c/testproject/+/4237421:30
dasm|ruckhttps://review.rdoproject.org/zuul/status/change/42374,621:31
* dviroel dinner and out o/21:50
*** dviroel is now known as dviroel|out21:50
dasm|ruckdviroel|out: o/21:51
rlandyugh - the c8 line is  a mess - now image build failed22:25
rlandylet22:25
rlandys leav eit for conatiner builds to complete22:25
* rlandy bbl22:25
*** rlandy is now known as rlandy|bbl22:25
rlandy|bblperiodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset035-masteropenstack/tripleo-cimasteropenstack-periodic-integration-main4 hrs 6 mins 58 secs2022-05-04 17:58:42SUCCESS22:27
rlandy|bblha - fs035 passing now22:27
rlandy|bbldasm|ruck: need to go - can you put fs001 fs020 and other master failures in testproject rerun22:28
rlandy|bblwe have a fs035 pass22:28
dasm|rucklet me see if i don't have it already22:32
dasm|ruckstarted here: https://review.rdoproject.org/r/c/testproject/+/4251822:34
dasm|ruck> 22:49
dasm|ruckperiodic-tripleo-ci-build-containers-ubi-8-push-wallaby22:49
dasm|ruckpush succeeded but overcloud jobs failed22:49

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!