Thursday, 2021-10-07

*** rlandy|ruck|bbl is now known as rlandy|ruck01:26
*** pojadhav|afk is now known as pojadhav04:34
*** amoralej|off is now known as amoralej06:03
*** ysandeep|out is now known as ysandeep06:39
*** jpena|off is now known as jpena06:47
jbadiapa:1507:20
mariossshnaidm: o/ hey man is there a good/bad time today for you so we can discuss the release job for tripleo-repos? 08:04
sshnaidmmarios, yep08:08
sshnaidmmarios, how about in 1 hour?08:08
mariossshnaidm: great thanks08:08
erbotRequired files not ready after 301.5721483230591s for distgit-cbs-validate-centos8-stream 36037,1,76acc4e7373f4d7080ab34b2e4ad5dba08:20
sshnaidmerbot, who are you?08:21
*** ykarel is now known as ykarel|lunch08:34
erbotRequired files not ready after 301.1625745296478s for distgit-cbs-validate-centos8-stream 36067,1,65ddf0af728d41a7b44b93c8e2f8a51c08:51
erbotRequired files not ready after 301.57702469825745s for distgit-cbs-validate-centos8-stream 36069,1,8599e897e2844a9f819c7d35517d9df608:58
mariossshnaidm: sorry gimme 2 mins09:01
sshnaidmnp09:01
erbotRequired files not ready after 481.22989654541016s for distgit-cbs-validate-centos8-stream 36068,1,bdd9dd800ca44f14a6e31c6c8d3c509409:03
*** jpena is now known as jpena|brb09:05
erbotRequired files not ready after 571.0929400920868s for distgit-cbs-validate-centos8-stream 36071,1,7e84a217d560481f863e7427986c1ed309:08
erbotRequired files not ready after 529.6928639411926s for distgit-cbs-validate-centos8-stream 36073,1,334c60da04ef4ca784aa3d5d36ba68fe09:08
erbotRequired files not ready after 493.31976222991943s for distgit-cbs-validate-centos8-stream 36074,1,ca4de0571f0242fd910e8751d7f57b9a09:08
erbotRequired files not ready after 483.0017969608307s for distgit-cbs-validate-centos8-stream 36075,1,ad7bc7c13e384f3f9dcda69df5a1ef9409:08
erbotRequired files not ready after 476.58645486831665s for distgit-cbs-validate-centos8-stream 36070,1,17c34fe65637481b9204be452b8c8ffa09:08
erbotRequired files not ready after 476.1784830093384s for distgit-cbs-validate-centos8-stream 36079,1,c3e5ffcc48be4c9fb1fa31538365947909:08
erbotRequired files not ready after 466.73684191703796s for distgit-cbs-validate-centos8-stream 36078,1,dd1eac5b5d36458e917b48489c5cde5b09:08
erbotRequired files not ready after 467.3183243274689s for distgit-cbs-validate-centos8-stream 36080,1,df6ef35557404cd1a761845120437afa09:08
erbotRequired files not ready after 363.9725031852722s for distgit-cbs-validate-centos8-stream 36077,1,6f539c223deb4002b2654d646ef3f32a09:08
erbotRequired files not ready after 590.7622880935669s for distgit-cbs-validate-centos8-stream 36072,1,7536af638389428daa7cdeb4a3eef80209:13
erbotRequired files not ready after 590.3224809169769s for distgit-cbs-validate-centos8-stream 36084,1,7baf62ce45d54ff5bcf0af812f870df209:13
erbotRequired files not ready after 547.9445109367371s for distgit-cbs-validate-centos8-stream 36081,1,73d852cb289f46dd98fc519f8807d15e09:13
erbotRequired files not ready after 548.6109676361084s for distgit-cbs-validate-centos8-stream 36085,1,eaa70ab38d614e5d9612c6535950437309:13
erbotRequired files not ready after 441.2600169181824s for distgit-cbs-validate-centos8-stream 36083,1,07372232fcfc4a789fdd85d3a562521009:13
erbotRequired files not ready after 343.82688546180725s for distgit-cbs-validate-centos8-stream 36082,1,cf1b1a0fcf7a45ffa209cff5829bf75c09:13
marioshttps://opendev.org/openstack/releases/src/branch/master/deliverables/_independent/tripleo-operator-ansible.yaml 09:15
erbotRequired files not ready after 476.19757747650146s for distgit-cbs-validate-centos8-stream 36094,1,d9f4afed7ba74fcb8b7d06f4678ad93509:18
erbotRequired files not ready after 475.86432099342346s for distgit-cbs-validate-centos8-stream 36095,1,d1ffcfb1b1a745c0902fe9d1e7e2754e09:18
erbotRequired files not ready after 422.4780993461609s for distgit-cbs-validate-centos8-stream 36093,1,edb03789853b48c9ad854842247e7d2f09:18
erbotRequired files not ready after 423.0610628128052s for distgit-cbs-validate-centos8-stream 36097,1,58ca3b42a91a4248af1b13f31463c72b09:18
erbotRequired files not ready after 422.7059483528137s for distgit-cbs-validate-centos8-stream 36100,1,998fa810d5e34f2489aa4d6bb270989d09:18
erbotRequired files not ready after 423.3480920791626s for distgit-cbs-validate-centos8-stream 36098,1,797b0ab7c5274204a2b090973130782609:18
erbotRequired files not ready after 413.93436336517334s for distgit-cbs-validate-centos8-stream 36099,1,5b16253f4c5943269e1a0a1ad1ade48309:18
erbotRequired files not ready after 390.5136806964874s for distgit-cbs-validate-centos8-stream 36096,1,52e40183942647e9bedd6c03b46c359f09:18
erbotRequired files not ready after 384.10102128982544s for distgit-cbs-validate-centos8-stream 36088,1,d7d154223b6245e9997472bc210b831909:18
erbotRequired files not ready after 310.671422958374s for distgit-cbs-validate-centos8-stream 36090,1,336fcd995dc8439996b7d1d1a931154a09:18
erbotRequired files not ready after 310.2716360092163s for distgit-cbs-validate-centos8-stream 36092,1,4d36f79165a74d1bbe0edaec40a9622909:18
* frenzy_friday checking ^09:19
*** ykarel|lunch is now known as ykarel09:52
*** jpena|brb is now known as jpena10:00
ysandeepfolks o/ Some knows from where we can download the image which nodepool provides for centos9 stream? 10:33
ysandeepwe are hitting an issue "Failed to find required executable virtualenv in paths: /sbin:/bin:/usr/sbin:/usr/bin:/usr/local/sbin" and I want to check if virtualenv package comes on default cs9 image or not?10:33
ysandeephttps://review.rdoproject.org/r/c/testproject/+/35229/4#message-462e4cf13b465e08fe154346d52a1c9764a3030d 10:33
ysandeepykarel: ^^10:34
*** rlandy is now known as rlandy|ruck10:41
rlandy|ruckbhagyashris: arxcruz|ruck: hello - how are things today?10:42
rlandy|ruckbhagyashris: let's chat about PSI - some developments from yesterday's prod chain meeting10:43
arxcruz|ruckrlandy|ruck: all good, i'm  working on the collect-logs molecule fix 10:44
rlandy|ruckakahat: welcome back - how was the hiking trip?10:44
rlandy|ruckarxcruz|ruck: cool - need help with anything else?10:44
arxcruz|rucki think i don't for now 10:45
rlandy|ruckysandeep++ ykarel++ 10:55
rlandy|rucknice https://jenkins-cloudsig-ci.apps.ocp.ci.centos.org/job/tripleo-quickstart-gate-master-tripleo-ci-delorean-full-featureset052/ passing again10:55
ysandeeprlandy|ruck, yeah :D fyi.. rabi patch to fix minimal also merged https://review.opendev.org/c/openstack/tripleo-quickstart/+/812423 , i have closed the bug and cix10:59
rlandy|ruckysandeep: yep - saw that - rerunning10:59
ysandeeprlandy|ruck, https://review.opendev.org/c/openstack/tripleo-quickstart/+/812423/4#message-c5d6a845a30132ba335d845b8cead0901685cb5b minimal check job is also running successfully.. when we post check-ci-minimal :D11:00
rlandy|ruckawesomeness11:01
ykarelysandeep, if you have creds for ci tenant you can download from there the image11:11
ykarelthe centos 9 stream image from c9 stream compose is used + dnf update + virt+customize as define11:12
ysandeepykarel: are you reffering openstack image download/glance download ? 11:12
ykarelthat's a new job which is failing or earlier it was passing?11:13
ykarelysandeep, yes11:13
ysandeepykarel: ack, thanks! 11:13
ykarelysandeep, logs not available to check?11:14
ysandeepFailing at early stage: https://logserver.rdoproject.org/29/35229/2/check/periodic-tripleo-ci-centos-9-standalone-master/b518bf4/job-output.txt 11:15
ykarelohkk11:15
ysandeepI have tried adding https://review.rdoproject.org/r/c/rdo-jobs/+/36064/1/roles/fetch-zuul-cloner-fork/tasks/main.yaml as depends-on but this was not included as called by trusted project pre.yaml11:16
rlandy|ruckysandeep: any progress with the 16.2 neutron fix?11:19
rlandy|rucksee you are rerunning the network component11:20
rlandy|ruckdo we need to trigger the line afterwards?11:20
ysandeeprlandy|ruck: we got the new rpm today, I have triggered network line, if all passes will retrigger integration line.11:21
rlandy|ruckysandeep: great11:21
ykarelysandeep, ok as per virt-customize config i see python3-virtualenv is commented in c9-stream but added in c8-stream11:21
ykarelso it's possible it's not installed11:21
ysandeepykarel, could you please share the link to c9 virt-customize11:21
rlandy|ruckysandeep: need any help with rhos-17? we should try promote tomorrow11:21
ysandeeprlandy|ruck, we don't have any known bug apart from fs35.. lets see how current run goes11:23
*** jpena is now known as jpena|lunch11:23
ykarelysandeep, https://softwarefactory-project.io/r/q/file:nodepool/virt_images/cloud-centos-9-stream.yaml11:24
rlandy|ruckysandeep: k - ping me when you are EoD and I'll watch that or rerun as needed11:24
ysandeeprlandy|ruck, k.. we will ignore fs35 tempest failure for now.. shift team is saying they are because of high load and expected.. 11:24
rlandy|ruckysandeep: yeah - heard that at CIX meeting11:25
bhagyashrisrlandy|ruck, all good11:29
rlandy|ruckbhagyashris: let's chat re: PSI11:30
bhagyashrisrlandy|ruck, yup11:30
bhagyashrisgive me 3 mins11:30
rlandy|rucksure11:30
rlandy|ruckbhagyashris; when you are ready https://meet.google.com/smr-pvrv-wur?pli=1&authuser=011:30
rlandy|ruckbhagyashris: arxcruz|ruck: fyi https://review.rdoproject.org/r/c/config/+/3610911:45
rlandy|ruckputting back SSL verification into dlrn commands11:45
rlandy|ruckysandeep: ^^ in case there is any fallout downstream11:45
ysandeepack 11:46
arxcruz|ruckrlandy|ruck: are the "" missing ?11:46
rlandy|ruckworks with and without11:46
rlandy|ruckat least it should11:47
rlandy|ruckarxcruz|ruck; let me add them to be sure11:47
rlandy|ruckarxcruz|ruck: updated11:49
rlandy|ruckysandeep: arxcruz|ruck: bhagyashris: merging the SSL change - let me know if there is fallout in rdo or downstream12:03
* dviroel ☕12:03
*** amoralej is now known as amoralej|lunch12:12
*** jpena|lunch is now known as jpena12:23
*** raukadah is now known as chandankumar12:28
rlandy|ruckrekicking c7 to test12:32
ysandeepfolks o/ please add in your review list https://review.rdoproject.org/r/c/config/+/3611312:41
ysandeeprlandy|ruck, ^^ need to progress further on c9 standalone work, this role is from zuulv2 times and not needed anymore.12:44
chandankumarysandeep: done12:51
ysandeepchandankumar, you are back from sl :) 12:51
ysandeepthanks!12:51
chandankumarysandeep: yes :-) 12:55
chandankumarysandeep: thank you for picking cs9 work :-)12:55
rlandy|ruckdone12:58
bhagyashrisakahat, chandankumar scrum time..13:00
ysandeepchandankumar, there were comments on your tripleo-common patch for container build.. we checked that but we didn't have details to answer them so we decided to wait for you.13:01
*** amoralej|lunch is now known as amoralej13:07
rlandy|ruckbhagyashris: can you share the scrum board13:09
bhagyashrisreview list 13:13
bhagyashrishttps://review.rdoproject.org/r/c/rdo-jobs/+/35906, https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/81298713:13
bhagyashrishttps://review.opendev.org/q/topic:%22wire-up-yum-config%22+(status:open%20OR%20status:merged)13:15
bhagyashrishttps://review.opendev.org/c/openstack/tripleo-repos/+/80821613:16
dviroelyum-config wire up patches:13:17
dviroelhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/80840813:17
dviroelhttps://review.opendev.org/c/openstack/tripleo-quickstart/+/81267213:17
bhagyashrishttps://review.opendev.org/c/openstack/tripleo-ci/+/808177/9#message-9d7cee6879c8a148ccaca1e42f51e019469a85a013:18
bhagyashrisdviroel, thanks :)13:18
bhagyashrishttps://review.rdoproject.org/r/q/topic:%22update-upgrades-components%22+(status:open%20OR%20status:merged)13:28
*** ykarel_ is now known as ykarel|away13:52
chandankumarysandeep: rlandy|ruck marios akahat dviroel frenzy_friday invite sent for cs9 sync13:54
ysandeepchandankumar, sshnaidm would like to join that as well, he is out tomorrow. On scrum we discussed to move that meeting to Monday.13:55
bhagyashrischandankumar, add me too13:55
chandankumarok moving it to monday13:55
ysandeepchandankumar: thanks!13:57
marioschandankumar: thanks13:58
rlandy|ruckchandankumar: ack - sorry - monday is better for sagi13:58
rlandy|ruckthanks for getting that going13:58
dviroeltks, works for me too14:08
marioschandankumar: did you mention this in the call? 2021-10-07 12:48:07.016416 | primary |     "No package python3-virtualenv available." 14:20
mariosor i am misremembering14:20
chandankumarmarios: nope, 14:20
mariosor maybe dviroel ? ^^ 14:20
marioschandankumar: thanks14:20
chandankumarmarios: we need to tweak pre part 14:20
ade_leedviroel, rlandy|ruck slaweq interesting -- I read up what you guys mentioned yesterday14:21
chandankumarfetching container build chanfes14:21
marioschandankumar: k thanks i will revisit it tomorrow anyway i just saw the fail and remembered someone mentioning something about that in scrum but maybe i am wrong14:21
chandankumarmarios: https://review.opendev.org/c/openstack/tripleo-ci/+/800577/51/playbooks/tripleo-buildcontainers/pre9.yaml#2514:22
ade_leedviroel, rlandy|ruck slaweq I'm curious as to why the secondary nodes and the network setup there is any different than the undercloud/primary nodes 14:22
marioschandankumar: ah i see14:22
marioschandankumar: thanks ;)14:22
ade_leedviroel, rlandy|ruck slaweq - do we know where the net work config is done?  looks like some ansible is invoked14:24
ade_leedviroel, rlandy|ruck slaweq maybe we need to add something to those commands to make the config stick through reboots?14:25
dviroelade_lee: you can try that, but i wouldn't continue in this path, after fixing that, you can face more issues with reboots in these jobs or in the next ones14:30
ade_leedviroel, the code that sets up the networking is here - https://github.com/openstack/tripleo-quickstart-extras/blob/master/roles/undercloud-setup/templates/vxlan_networking.sh.j2#L112 I think14:33
ade_leedviroel, as I understand it, this command only sets up a temporary ip address.14:33
ade_leeso it makes sense this would be wiped out14:33
ysandeepmarios, hey i mentioned missing python3-virtualenv14:34
mariosysandeep: ah 14:34
mariosysandeep: thanks is there a patch in diskimage-builder/or elements yet ... chandankumar pointed me to a workaround i can use for now anyway at 17:22 < chandankumar> marios: 14:35
mariosysandeep: actually that isn't a workaround... anyway relevant (my fail is that there is "No package python3-virtualenv available.")14:36
ysandeepmarios, afaik.. currently we are using virt-customize to create centos 9 stream image.. not disk-image-builder14:36
ysandeephttps://softwarefactory-project.io/r/c/config/+/21896/1/nodepool/virt_images/cloud-centos-9-stream.yaml@4214:36
ysandeeppython3-virtualenv is commented in ^^14:36
mariosysandeep: well we are using elements/dib for the image builds14:36
mariosysandeep: overcloud deploy image builds14:37
rlandy|ruckade_lee: sorry - was in meeting reading back14:37
mariosysandeep: e.g. https://review.opendev.org/c/openstack/tripleo-puppet-elements/+/811712 is one workaround we needed so far 14:37
rlandy|ruckade_lee: we can check with infra (tristanC would know - he is core there) but I don't think we will get support for rebooting14:38
rlandy|ruckade_lee: is there a possibility to go with adding fips to the image and going that way now?14:38
rlandy|ruckade_lee: even if we could spin a test nodepool image in rdo that uses virt-customize or something to add fips support14:39
rlandy|ruckwrt secondary and primary node, there is a bridge set up between them14:39
ysandeepmarios: regarding missing python3-virtualenv yatin and I raised that on internal ops channel.. one of the comment was - why do you we need virtualenv with python3? Can't we use python3 -m venv?14:40
ade_leerlandy|ruck, well -- first off, rebooting does work - its just that we don't set up networking in tripleo-quickstart-extras in a way that is persistent across reboots.14:40
ade_leeif we used soething like ifconfig or somesuch, it would be persistent - but then as dviroel says, we may run into other issues14:41
ade_leeor we may not -- we've been relatively lucky so fay14:41
rlandy|ruckyeah - that may impact on our tests in check14:42
rlandy|ruckade_lee; so ... question - what is the end game here?14:42
ade_leerlandy|ruck, dviroel that said - eventually the plan was always to look into using fips enabled images14:42
ade_leeso maybe this is the time to explore that option more 14:43
rlandy|ruckade_lee: I think so - if that is a dead end, we can look into the network setup in tripleo-quickstart-extras14:43
ysandeepmarios, could you please point me to your failing job logs.. 14:43
rlandy|ruckI am not opposed to improving that code 14:43
rlandy|ruckimproving it is always good14:44
ade_leerlandy|ruck, dviroel trying to think about where to start wrt getting a fips enabled image ..14:44
rlandy|ruckbut this may be the first of other issue if we don't images you need at this point14:44
rlandy|ruckade_lee: as in diskimage-builder?14:44
rlandy|ruckor from the nodepool side?14:45
ysandeepmarios: If its failing on python3-virtualenv install, its possibly because it's not in c9-stream base repos14:45
ade_leerlandy|ruck, at this point, I don't know much about how the images used upstream are built14:45
mariosysandeep: ok will dig more on that tomorrow have switched right now (python3-virtualenv i mean) but logs are there https://logserver.rdoproject.org/65/35465/9/check/tripleo-stream9-development-buildimage-overcloud-full-master/dfe9ea9/job-output.txt (from https://review.rdoproject.org/r/c/testproject/+/3546514:45
mariosysandeep: thanks for hint noting in my logs14:45
ade_leerlandy|ruck, dviroel if you happen to know or to whom I should chat, I can start digging 14:46
rlandy|ruckade_lee: openstack infra usually builds upstream images. for rdo, the rdo-infra team does14:46
ade_leeotherwise I was going to check with infra folks 14:46
rlandy|ruckade_lee: join me on #rhos-ops14:46
dviroel++14:46
dviroelade_lee: i'm also want to understand how it works14:47
rlandy|ruckdviroel: feel free to join us as well14:47
dviroelade_lee: lets work together on this14:47
dviroelrlandy|ruck: ack14:47
ade_leedviroel, cool - we're probably going to need to do both tbh - upstream and rdo14:47
* rlandy|ruck hope internal IRC is functional again14:50
rlandy|ruckotherwise I am just talking to myself14:50
dviroellolz14:52
rlandy|ruckwe could also take this to the prod chain meeting on wednesday if need be14:53
* dviroel manila community meeting15:05
ysandeeprlandy|ruck: need a second review here: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/3409715:18
* rlandy|ruck looks15:18
rlandy|ruckperiodic-tripleo-ci-centos-8-standalone-network-master-validation15:19
rlandy|rucklooks good15:19
rlandy|ruckysandeep : ok to merge that?15:20
ysandeepyes15:20
frenzy_fridayHey rlandy|ruck, what are the jobs that are of interest to us on RDO? I am trying to form a regex that the bot will use. It is checking all jobs with tripleo ot tox in it right now. Is that all?15:24
rlandy|ruckfrenzy_friday: good question15:25
rlandy|ruckyou can consult what the cockpit is tracking15:25
frenzy_fridayrlandy|ruck, thanks, lemme check15:26
rlandy|ruckI's day anything with tripleo and tox is a good start15:26
bhagyashrisrlandy|ruck, leaving for the day15:30
rlandy|ruckbhagyashris: sure - hope you feel good15:30
rlandy|ruckbhagyashris: if you don't feel great tomorrow, just drop arxcruz|ruck and me an email15:31
bhagyashrisrlandy|ruck, sure, thanks :)15:31
*** amoralej is now known as amoralej|off15:46
*** ysandeep is now known as ysandeep|dinner15:48
*** marios is now known as marios|out16:04
*** jpena is now known as jpena|off16:46
frenzy_fridayrlandy|ruck, https://review.rdoproject.org/r/c/testproject/+/36119 - a bit closer to a real use case17:06
frenzy_fridayarxcruz|ruck, rlandy|ruck, when you get some time can we merge https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/812987 ?17:09
rlandy|ruckfrenzy_friday; reading17:12
rlandy|ruckmerging17:13
frenzy_fridayrlandy|ruck, thanks17:13
rlandy|ruckhttps://review.rdoproject.org/r/c/testproject/+/36119 - nicely dine!!17:14
rlandy|ruckdone17:14
frenzy_fridayI have updated the health server with the latest bot patch - https://review.opendev.org/c/opendev/elastic-recheck/+/805638 . Lemme know if the bot goes rogue, I'll kill it17:15
rlandy|ruckfrenzy_friday: no wories17:17
rlandy|ruckworries17:17
*** ysandeep|dinner is now known as ysandeep17:18
*** rlandy|ruck is now known as rlandy|ruck|brb17:36
*** rlandy|ruck|brb is now known as rlandy|ruck18:02
*** ysandeep is now known as ysandeep|out18:39
rlandy|ruckdviroel: fyi ... https://review.rdoproject.org/r/c/testproject/+/3612220:48
dviroeltks20:49
*** dviroel is now known as dviroel|out21:02

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!