Thursday, 2021-04-15

*** jmasud has joined #oooq00:20
*** jmasud has quit IRC01:03
*** jmasud has joined #oooq01:06
*** matbu has quit IRC01:08
*** matbu has joined #oooq01:09
*** ysandeep|away is now known as ysandeep01:16
*** rlandy|rover|bbl is now known as rlandy|rover01:53
*** jmasud has quit IRC01:55
*** matbu has quit IRC01:57
*** matbu has joined #oooq01:59
*** apetrich has quit IRC02:09
*** rlandy|rover has quit IRC02:26
*** jmasud has joined #oooq02:33
*** jmasud has quit IRC03:34
*** jmasud has joined #oooq03:37
*** ykarel has joined #oooq03:45
*** jmasud has quit IRC03:58
*** skramaja has joined #oooq04:28
*** ratailor has joined #oooq04:30
*** jmasud has joined #oooq04:41
*** marios has joined #oooq05:03
*** ratailor_ has joined #oooq05:10
*** ratailor has quit IRC05:13
*** udesale has joined #oooq05:30
*** sshnaidm|pto has quit IRC05:47
*** sshnaidm|pto has joined #oooq05:48
*** slaweq has quit IRC05:55
*** slaweq_ has joined #oooq05:55
*** slaweq_ is now known as slaweq05:55
*** ykarel has quit IRC06:00
*** ykarel has joined #oooq06:00
*** jpodivin has joined #oooq06:22
*** slaweq_ has joined #oooq06:30
mariosarxcruz: did you get confused there? https://review.opendev.org/c/openstack/tripleo-ci/+/785841/3#message-d2326ca42dcedd8f07160d59c62b631c8588f45506:30
mariosarxcruz: i think you meant to add tempest_test_includelist?06:31
*** saneax has joined #oooq06:31
*** slaweq has quit IRC06:36
*** slaweq_ is now known as slaweq06:36
*** saneax has quit IRC06:36
*** apetrich has joined #oooq06:41
*** ykarel_ has joined #oooq06:49
*** ykarel has quit IRC06:51
*** jpena|off is now known as jpena06:55
*** amoralej|off is now known as amoralej07:05
chandankumarakahat: hello can you update the suggestion here https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/32058 ?07:07
*** jmasud has quit IRC07:07
*** jmasud has joined #oooq07:12
*** ykarel_ is now known as ykarel07:30
*** tosky has joined #oooq07:46
*** ykarel is now known as ykarel|lunch08:33
*** derekh has joined #oooq08:42
*** ysandeep is now known as ysandeep|lunch08:49
*** slaweq has quit IRC08:51
*** slaweq has joined #oooq08:51
*** jmasud has quit IRC08:55
*** saneax has joined #oooq09:29
*** ysandeep|lunch is now known as ysandeep09:32
arxcruzmarios: yes, i fixed it please review when you have chance :)09:38
*** ykarel|lunch is now known as ykarel09:38
mariosack arxcruz will revisit09:41
*** dpawlik9 has quit IRC09:46
arxcruzchandankumar: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/782253 :*09:55
*** dpawlik4 has joined #oooq09:59
*** ratailor__ has joined #oooq10:08
*** ratailor_ has quit IRC10:11
chandankumararxcruz: commented10:25
chandankumarsorry for abusing the tool10:25
*** ykarel_ has joined #oooq10:29
*** ykarel has quit IRC10:32
arxcruzchandankumar: i won ´t forgive you :)10:32
chandankumararxcruz: I will buy you a beer after covid :-)10:34
arxcruzchandankumar: comment there, we can't use yet a real case in the allow yaml file because we need to do it for all the jobs, or groups, I need time for that10:41
chandankumararxcruz: take your time, other wise more people will bug you like me :-)10:42
*** ykarel_ has quit IRC10:44
*** ykarel_ has joined #oooq10:45
arxcruzchandankumar: that will be my next work once this is merged10:51
arxcruz:)10:51
arxcruzchandankumar: but I need it merged to start the testing in some jobs10:51
chandankumarkopecmartin: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/782253 good to go10:54
*** ykarel_ is now known as ykarel11:15
*** jpena is now known as jpena|lunch11:31
*** rlandy has joined #oooq11:46
*** rlandy is now known as rlandy|rover11:46
rlandy|rovermarios: chandankumar: bhagyashris: arxcruz; sent you all email re: yesterday's interview.11:50
rlandy|roverpls PM me with questions/concerns11:50
mariosrlandy|rover: ack thanks11:51
rlandy|roverpojadhav: hey - have to join a call ... wil missing morning meeting11:53
rlandy|roverpls ping me if you want to chat11:53
pojadhavrlandy|rover, sure11:53
*** ykarel_ has joined #oooq11:54
*** ykarel has quit IRC11:56
*** rlandy|rover is now known as rlandy|rover|mtg12:01
chandankumarrlandy|rover|mtg: regarding sc10 train issue I was working here https://review.rdoproject.org/r/c/testproject/+/3326712:02
*** ykarel_ is now known as ykarel12:02
chandankumararxcruz: can we start sc10 case for train and master implementation with new tempest tests skip list utility?12:05
arxcruz chandankumar sure12:05
chandankumararxcruz: if you need more info, I can phone you.12:05
arxcruzchandankumar: here's my number, so call me maybe12:06
chandankumararxcruz: I mean google meet12:06
arxcruzchandankumar: i can't avoid make reference to call me maybe :)12:06
arxcruzchandankumar: https://www.youtube.com/watch?v=O7zdr-82WAo12:07
chandankumar:-)12:10
dviroellol12:14
*** apetrich has quit IRC12:18
*** apetrich has joined #oooq12:19
*** pojadhav is now known as pojadhav|afk12:30
*** jpena|lunch is now known as jpena12:33
*** amoralej is now known as amoralej|lunch12:36
*** rlandy|rover|mtg is now known as rlandy|rover12:42
*** ratailor__ has quit IRC12:47
arxcruzbhagyashris: I might be late for the meeting, I’m having internet issues, just contact my isp provider, right now i’m on cellphone connection12:49
bhagyashrisarxcruz, ok12:50
*** jfrancoa has joined #oooq12:51
rlandy|roverchandankumar: thanks for that work on scenario01012:54
rlandy|roverchandankumar: marios: arxcruz: weshay|ruck: re: tripleo-repos meeting .. sagi is out - would like to take that slot to talk about what we want to cover at PTG on the topic12:56
ysandeeprlandy|rover, chandankumar, is sc010 issue all sorted for train?12:59
rlandy|roverysandeep: chandankumar still working on it ^^12:59
bhagyashrismarios, akahat pojadhav|afk ysandeep weshay|ruck chandankumar scrum time13:00
ysandeepack.. i saw one green run yesterday in train.. testproject with smoketest only failed in 16.2 https://code.engineering.redhat.com/gerrit/#/c/211643/13:00
bhagyashrisdviroel, ^13:01
chandankumarysandeep: what is the failure now?13:01
ysandeepchandankumar, https://sf.hosted.upshift.rdu2.redhat.com/logs/43/211643/39/check/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-16.2/8efad20/job-output.txt13:01
rlandy|roverysandeep: yep - we need to get 16.2 and 17 working again13:01
chandankumarysandeep: for rhos-17 it is ok?13:02
ysandeeplet me crosscheck quickly13:02
ysandeepchandankumar, 17 timed out on tempest run .. no error in tempest logs https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-17/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario010-standalone-rhos-17/75334a9/logs/undercloud/var/log/tempest/tempest_run.log13:04
chandankumarkopecmartin: hello13:10
chandankumarkopecmartin: we need to create a new tag for tempestconf for wallaby13:10
chandankumar?13:11
kopecmartinchandankumar: we have a new tag 3.2.0  .. and wallaby is not pinned : https://github.com/redhat-openstack/rdoinfo/blob/master/tags/wallaby.yml13:12
*** marios is now known as marios|call13:15
chandankumarrlandy|rover: steps for wallaby unblock13:21
chandankumar1. promote this https://trunk.rdoproject.org/centos8-wallaby/component/tempest/tripleo-ci-testing/13:21
chandankumarcurrent tripleo-ci-testing does not contains it.13:21
rlandy|roverchandankumar: k - we need to go consistent -> tripleo-ci-testing13:22
rlandy|rovercomponent ci-testing had same date13:22
*** amoralej|lunch is now known as amoralej13:29
*** marios|call is now known as marios13:55
chandankumarrlandy|rover: weshay|ruck marios|call Do we want to do personas call? since sagi is not around13:55
marioschandankumar: rlandy|rover: weshay|ruck: i think we can/should talk13:56
marioschandankumar: we can chat a bit about our ptg session next week13:56
chandankumarmarios: yes sure13:56
rlandy|roverchandankumar: I would like to discuss out PTG presentation13:56
mariosrlandy|rover: yes13:56
* marios brb13:56
rlandy|roverweshay|ruck: going to promote consistent -> component-ci-testing-> promoted components for tempets wallaby13:59
*** pojadhav|afk is now known as pojadhav14:07
zbrkopecmartin: look at pavel comment on https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/783843/1/roles/collect_logs/defaults/main.yml#214 -- basically you added to fix to the wrong task (the faster one). just add it to the other one *too*, so we avoid surprises14:10
kopecmartinzbr: i got that, so i should just move it to the 3 mentioned tasks and that's it?14:10
zbri think you want a single timeout on a wrapper, not on each command14:13
zbrthat thread is bit funny because more than an year ago i had a CR that was running these tasks in parallel in background, but it did never ship due to lack of support14:14
zbranbanerj|ruck: any chance to re-download the health change, i made lots of changes to it.14:23
anbanerj|ruckzbr, lemme check14:26
weshay|ruckzbr, rlandy|rover tripleo-health14:30
weshay|ruckkopecmartin, think we're having issues w/ infrared here https://zuul.openstack.org/builds?job_name=openstack-tox-molecule&project=openstack%2Fansible-role-collect-logs14:40
kopecmartinweshay|ruck: you mean the failures in openstack-tox-molecule job?14:44
kopecmartinit's installed using py2? hmm, let me check a few things14:44
rlandy|roverweshay|ruck: when back - pls rekick stable1 (wallaby) .. release file merged and tempest component promoted14:53
weshay|ruckrlandy|rover, k14:54
weshay|ruckkopecmartin, ya14:54
weshay|ruckkopecmartin, see it?14:56
mariosarxcruz: please check #tripleo channel when you get a sec14:56
kopecmartinweshay|ruck: i see the traceback in "install Infrared"14:57
kopecmartintrying to reproduce locally14:57
weshay|ruckrlandy|rover, rekicked14:58
weshay|ruckkopecmartin, thanks14:58
rlandy|roverweshay|ruck: thanks14:59
mariosarxcruz: please check http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021853.html and let me know15:04
kopecmartinweshay|ruck: the issue is that infrared is being installed there using pip2, not pip315:13
weshay|ruckhrm k15:13
kopecmartini don't understand why .. checking if we can force pip3 in the code15:14
kopecmartinmaybe the node has py2 set as default15:14
kopecmartinweshay|ruck: i have no idea what caused the usage of py2 .. hopefully this will help, let's see: https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/78646215:21
weshay|ruckthanks I'll watch it15:22
mariosweshay|ruck: 5.54 years of content-provider runtime in the last 6 months ;) https://etherpad.opendev.org/p/tripleo-ptg-retrospective15:31
*** saneax has quit IRC15:34
mariosweshay|ruck: hmmm thinking about it that is probably not accurate one i should pick another15:36
*** ysandeep is now known as ysandeep|dinner15:36
mariosweshay|ruck: because content providers pause while the other jobs run... so it has a mean runtime of 3 hours...15:37
mariosweshay|ruck: which isn't really true15:37
weshay|ruckmarios, looking15:40
mariosweshay|ruck: its ok just thinking out loud re the job runtime but content provider was inaccurate added the standalone instead15:41
marios2.92 years :)15:41
*** jmasud has joined #oooq15:41
chandankumarmarios: you can add about tempest improvements15:42
chandankumarmarios: like openstack-tempest-skiplist15:42
marioschandankumar: you can do that there is a section for each squad15:42
marioschandankumar: these ones at the top are for general/community things15:42
weshay|ruckmarios, /me poking at that15:42
marioschandankumar: under the CI section probably15:42
marioschandankumar: we will fill those during the session15:43
chandankumarmarios: sure, thanks!15:43
rlandy|roverysandeep|dinner: if OVB passes today - promote 16.2?15:45
* zbr frustrated of using opendev gerrit, adding reviewers became a joke. i cannot even leave the browser while gerrit is trying to lookup for someone because popup will disappear if I do15:46
marioszbr: yes i hate this too15:46
marioszbr: sometimes i find the best way is to just type first letter it seems to lookup pretty quickly but even that one not always ;)15:46
zbrplease mention it to #openstack-infra -- i did several times.15:46
*** ykarel has quit IRC15:47
mariosarxcruz: please check when you have time and reply as soon as you can 18:04 < marios> arxcruz: please check http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021853.html and let me know15:47
zbrit used to be hard to find reviewers, now taken to a new level: is even hard to add them to gerrit if you know them!15:47
weshay|ruckmarios, imho.. I think the data folks want to see is pass / fail % over the course of that time15:48
zbri guess power users are adding them using the API,... and admins using SQL15:48
mariosweshay|ruck: ack feel free to add whatever more you think there15:48
weshay|ruckk.. will do15:49
mariosweshay|ruck: thanks15:49
*** jpodivin has quit IRC15:55
*** dviroel is now known as dviroel|lunch15:58
* marios out16:01
*** marios is now known as marios|out16:01
kopecmartinweshay|ruck: it failed with something different now o.O no visible traceback , hmm https://e01559aac4b6db7f5d2c-e03b399dba5d60e417a0a61de1589ca2.ssl.cf2.rackcdn.com/786462/2/check/openstack-tox-molecule/3f843c0/job-output.txt16:05
*** jpena is now known as jpena|off16:05
*** marios|out has quit IRC16:22
*** amoralej is now known as amoralej|off16:48
*** derekh has quit IRC17:03
*** dviroel|lunch is now known as dviroel17:06
*** ysandeep|dinner is now known as ysandeep17:16
*** udesale has quit IRC17:21
rlandy|roverweshay|ruck: arxcruz: https://review.opendev.org/c/openstack/openstack-tempest-skiplist/+/786469 - to unblock scenario007 wallaby17:22
ysandeeprlandy|rover, >> if OVB passes today - promote 16.2 - yes .. we can skip sc10 tempest failure17:22
ysandeeprlandy|rover, fs035 ovb hit infra issue.. lets see how baremetal goes17:23
rlandy|roverack - think we may need tenant clean up17:24
ysandeeprlandy|rover, my script.. hit some real issue.. will reexecute that17:25
ysandeepotherwise will report to infra team17:25
rlandy|roverthanks17:25
ysandeeplooks like gerrit / launchpad integration is working again :)17:34
ysandeepi see comments on launchpad bugs17:34
ysandeepahh.. different error with ovb/ bm jobs in 16.2 , ovb fs001 failed on tempest, fs020 - failed during overcloud node import, fs035 ovb - infra issue, fs035 bm - ssl related error while executing python-tempconf17:39
ysandeeprlandy|rover, internal codesearch is not searching in tripleo-ci-internal-jobs..17:51
ysandeeprlandy|rover, cleaned up infra.. retriggering 16.2 ovb jobs.. if stack creating still hit issue.. I will report to infra.. no stack is running right now.. 2 stack should be created with ease.18:05
ysandeeprlandy|rover, https://code.engineering.redhat.com/gerrit/#/c/211494/ testproject18:06
rlandy|roversec18:06
* ysandeep will need to make a small adjustment to infracleanup script18:08
ysandeepbm failure looks legit.. same issue in second last run as well(when ovb fs035 passed).. will need investigation https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/909db38/job-output.txt .. will check tomorrow morning18:09
ysandeepERROR tempest FileNotFoundError: [Errno 2] No such file or directory: '/opt/cache/files/cirros-0.4.0-x86_64-disk.img'18:10
ysandeep^^ https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/909db38/logs/undercloud/home/zuul/tempest/tempestconf.log18:10
ysandeepcomparing ovb vs bm tempest conf18:11
*** skramaja has quit IRC18:28
rlandy|roverysandeep: sorry - not ignoring you18:32
rlandy|roverjust trying to get wallaby going18:32
rlandy|roverand internal puppet18:32
rlandy|roveryou all set?18:32
ysandeeprlandy|rover, nw.. i saw you speaking with tristian18:32
ysandeeprlandy|rover, reran ovb.. they seems to be hitting transient issue18:32
rlandy|roverwe have puppet jobs running18:32
rlandy|roverno idea what they actually run18:32
rlandy|roverbut something starts18:33
ysandeeprlandy|rover, i got the bm issue.. with tempest conf18:33
ysandeepimage: https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img18:33
ysandeepvs18:33
ysandeepimage: /opt/cache/files/cirros-0.4.0-x86_64-disk.img18:33
ysandeepIn cloud image.. we have cirros at /opt/cache/files .. but not in our undercloud image.. which we are building manually18:33
rlandy|rovernice18:34
ysandeepi think someone changed the behavior of picking default location at this path image: /opt/cache/files/cirros-0.4.0-x86_64-disk.img .. instead of https://download.cirros-cloud.net/0.4.0/cirros-0.4.0-x86_64-disk.img..18:34
rlandy|roverimage should be included now18:35
* ysandeep checking which patch updated the behavior 18:36
ysandeepor we just add below for our bm job:-18:37
ysandeep~~~18:37
ysandeeptempest_tempest_conf_overrides:18:37
ysandeep  image.image_path: 'http://images.rdoproject.org/cirros/cirros-0.4.0-x86_64-disk.img'18:37
ysandeep~~~18:37
ysandeeprlandy|rover, found it https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/77815718:38
rlandy|roveryeah - I recall that change18:39
ysandeeparxcruz, retry from alternate source for cirros image is not working.. https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/909db38/logs/undercloud/home/zuul/tempest/tempestconf.log18:40
ysandeepERROR tempest TypeError: 'Retrying' object is not iterable18:40
ysandeepweshay|ruck, fyi ^^ .. incase you wonder why 16.2 fs035 bm is failing18:41
rlandy|roverweshay|ruck is at vaccine appointment18:42
rlandy|roverwill tell him later18:42
ysandeeprlandy|rover, you okay if I add  tempest_tempest_conf_overrides: image.image_path: in bm jobs definations?18:43
rlandy|roversure18:43
ysandeepwill remove after arxcruz / some else fixes alternate source for cirros image18:44
arxcruzysandeep: it’s already up and running18:47
*** holser has quit IRC18:47
ysandeeparxcruz, when you get some time, could you please take a look at below logs:18:48
ysandeephttps://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-periodic-integration-rhos-16.2/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-bm_envD-3ctlr_1comp-featureset035-rhos-16.2/909db38/logs/undercloud/home/zuul/tempest/tempestconf.log18:48
*** jmasud has quit IRC18:56
arxcruzysandeep: needs a variable18:58
arxcruzretry_image = true18:59
arxcruzysandeep: https://opendev.org/openstack/tripleo-quickstart-extras/src/branch/master/playbooks/tasks/tempest.yml#L21619:01
ysandeepI see you have it here ^^ already, Do we need to add it somewhere else as well?19:02
ysandeeparxcruz, ^^19:06
arxcruzysandeep: i’ll checkt tomorrow19:06
ysandeeparxcruz, thanks! no hurry.. we are applying a workaround : tempest_tempest_conf_overrides image.image_path , Have a great evening :)19:07
ysandeeprlandy|rover, https://code.engineering.redhat.com/gerrit/23642319:27
rlandy|rovervoted19:28
*** rlandy|rover is now known as rlandy|rover|mtg19:28
ysandeeprlandy|rover|mtg, , internal cloud is not in great shape.. atleast for ovb.. i will write a ticket for infra tomorrow19:28
*** holser has joined #oooq19:28
rlandy|rover|mtgsure19:29
rlandy|rover|mtggoing to review 17 after this19:29
ysandeepi trigger 3 ovb job.. all are doing multiple attempts.. out of 8 attempts only 1 passed19:30
ysandeep+Wing https://code.engineering.redhat.com/gerrit/#/c/236423/19:32
*** rlandy|rover|mtg is now known as rlandy|rover19:56
ysandeeprlandy|rover, fyi 16.2 BM retriggered with https://code.engineering.redhat.com/gerrit/#/c/201382/ , internal infra still on hicups.. apparently Need some reboot/live migrations .. If Bm passes we can consider 16.2 promotion.. otherwise lets see tomorrow o/19:59
* ysandeep should be sleeping now, 01:30 am for me :D .. I am out now.. see your tomorrow o/20:00
*** ysandeep is now known as ysandeep|away20:01
*** dviroel is now known as dviroel|afk20:06
rlandy|roverweshay|ruck: we're ready to promote wallaby when you get back20:27
weshay|ruckrlandy|rover, 0/20:31
weshay|ruckrlandy|rover, alight :)20:31
weshay|ruckgood news20:31
weshay|ruckrlandy|rover, you have some test projects?20:31
weshay|ruckrlandy|rover, https://trunk.rdoproject.org/api-centos8-wallaby/api/civotes_agg_detail.html?ref_hash=4d620925b3dfabe6999255d6ee7d82c420:33
rlandy|roverweshay|ruck: hey - ho ware you feeling?20:34
rlandy|roverweshay|ruck; yep - scenario007 ran again and20:34
rlandy|roverpassed20:34
rlandy|roverfs001 an dfs020 are stil in testproject20:34
rlandy|roverfs035 will pass20:34
rlandy|roverweshay|ruck: time to chat quickly?20:35
rlandy|roverwill explain20:35
weshay|rucksure..20:35
rlandy|roveralso we have puppet jobs20:35
weshay|ruckahhh nice20:36
rlandy|roverweshay|ruck: meet.google.com/tsj-yzaj-ozy20:36
*** dviroel|afk is now known as dviroel20:56
*** jbadiapa has quit IRC21:02
*** jfrancoa has quit IRC21:03
*** jmasud has joined #oooq21:33
weshay|ruckrlandy|rover, wallaby promoted.. see you tomorrow 0/21:36
rlandy|roverweshay|ruck: woohoo21:42
*** jmasud has quit IRC22:08
*** rlandy|rover is now known as rlandy|rover|bbl22:33
*** tosky has quit IRC22:34

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!