Tuesday, 2022-12-06

*** dviroel is now known as dviroel|out00:28
ysandeepgood morning folks o/04:40
*** yadnesh|away is now known as yadnesh04:45
*** ysandeep is now known as ysandeep|afk05:47
*** marios is now known as marios|ruck06:08
marios|rucko/06:13
marios|ruckfrenzy_friday: o/ for container push/promoter creds issue http://promoter.rdoproject.org/promoter_logs/centos8_wallaby.log can we have a quick call when you have a minute06:41
*** ysandeep|afk is now known as ysandeep07:00
*** yadnesh is now known as yadnesh|afk08:13
marios|ruckneeds reviews please https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/86169208:17
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.08:17
ysandeepmarios|ruck, looking08:18
marios|ruckysandeep: thanks its from takashi needs some eyes (not easy/big review but needs a few mins staring thanks)08:18
marios|ruckysandeep: looks like chandankumar voted just now08:19
jm1good morning #oooq :)08:19
jm1what a nice quasi-monday :D08:19
marios|rucko/08:19
ysandeepmarios|ruck, yeah chandankumar already +wed, chandankumar++08:19
frenzy_fridaymarios|ruck, good morning. Lemme check the logs08:26
frenzy_fridayIs it only for wallaby c8 or all the releases?08:26
marios|ruckfrenzy_friday: o/ i see the creds were updated on promoter and it works for me locally for login so not sure why it fails to push 08:26
marios|ruckfrenzy_friday: well wallaby8 is the one trying to promote now so that is why we see it there08:27
frenzy_fridayhm... wallabyc8 is trying to login with tripleozedcentos9+cirobot creds08:27
frenzy_fridayLemme check what recent patches merged08:27
dpawlikhey folks. Yesterday I was generating secrets via zuul encrypt secret script: python3 ./encrypt_secret.py --tenant rdoproject.org --infile secret.data --outfile  test.yml https://review.rdoproject.org/zuul config08:28
dpawlikusually all was working fine08:28
dpawlikbut I remember that last time frenzy_friday was regenerating token after me :>08:29
marios|ruckdpawlik: frenzy_friday: invite sent, lets talk on phone in half hour will be easier than discussing here?08:29
frenzy_fridaydpawlik, yep, it is not an issue with the zuul secrets patch. 08:29
marios|ruckdpawlik: i can login with the new secret (docker login) so seems OK 08:30
dpawlikso you can login to the Quay but push does not work?08:31
marios|ruckdpawlik: yeah but looks to be an issue with our code, probably not creds related? not sure yet 08:31
marios|ruckfrenzy_friday: maybe one of the config patches for the container jobs (rdo registry decommission)08:32
frenzy_fridayyeah, that is my guess. I thought it was https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45898/9/ci-scripts/container-push/roles/containers-promote/defaults/main.yml that was causing trouble but nope. Looking for the other merged patches08:32
dpawlikI made a test: login + pull + push new image -> all was working for  quay.rdoproject.org/tripleozedcentos9/nop:dpawlik   . Checking if it got permissions to create new repository08:35
dpawlikalso got permissions to create new repo: https://quay.rdoproject.org/repository/tripleozedcentos9/dpawlik08:36
dpawlikcould you send me some log url?08:36
marios|ruckdpawlik: wallaby/8 creds are OK (issue currently is pusing to w8) ... logs there http://promoter.rdoproject.org/promoter_logs/centos8_wallaby.log looks like it tries to push with wrong user (using zed pass) 08:37
dpawlikAdded PS for adding new volume: https://softwarefactory-project.io/r/c/software-factory/sf-infra/+/2693308:37
dpawlikoh08:38
frenzy_fridayhey akahat , which role/playbook does the promoter use to promote containers? Is it only ci-scripts/container-push ?08:38
dpawlikdid you check in the project logs history?08:38
marios|ruckfrenzy_friday: yes that play invoked containers-promote role 08:39
dpawlikhttps://quay.rdoproject.org/organization/tripleowallabycentos8?tab=logs -> it seems that it works normally08:40
frenzy_fridaymarios|ruck, https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/dlrnapi_promoter/config_environments/rdo/CentOS-8/wallaby.yaml#L8 this is wrong as well. I'll put up a patch to update. 09:08
frenzy_fridayBut it is not the reason for our current failure09:08
marios|ruckfrenzy_friday: ack 09:08
frenzy_fridayactually it may not be wrong. Sorry, a stupid question: where do we promote wallaby to? Like I know the source registry is quay.rdo, what is the target registry?09:10
*** yadnesh|afk is now known as yadnesh09:20
marios|ruckfrenzy_friday: we 'promote' by tagging the hash as current-tripleo and pushing back to quay.rdoproject.org ... we used to also push to other registries in that code https://github.com/rdo-infra/ci-config/blob/00e78ee915ce8f19ef0661f8c78358f67e3ded8f/ci-scripts/container-push/container-push.yml#L50 (quay.io nowadays only) but i know we have made that even more complicated with an additional copy quay tool09:20
marios|ruckso... 09:20
marios|ruck:/09:20
*** ysandeep is now known as ysandeep|lunch09:21
dpawlikfolks, we are adding new volume to the Quay09:31
dpawlikservice should work normally, but just FYI09:31
akahatfrenzy_friday, we use ci-scripts/container-push role.09:31
frenzy_fridayakahat, hey, need one help - In wallaby c8, promoter is using the wron username09:32
frenzy_fridayIf you check http://promoter.rdoproject.org/promoter_logs/centos8_wallaby.log - unauthorized: Could not find robot with username: tripleozedcentos9+cirobot09:33
akahatfrenzy_friday, yes. i've checked it. looking for the issue.. what exactly caused it.09:33
frenzy_fridayIt is wallaby c8 , it should be using unauthorized: Could not find robot with username: tripleowallabycentos8+cirobot09:33
frenzy_fridaycool, thanks09:33
frenzy_fridaydpawlik, ack, thanks09:33
frenzy_fridaymarios|ruck, akahat this is what I found: try running docker manifest inspect --insecure quay.rdoproject.org/v2/tripleowallabycentos8/openstack-aodh-api:47c581dacb7a7b35ad3aa337c08e1703 from your local (or anywhere)09:46
frenzy_fridayThen try running it from thr promoter (as promoter user)09:46
frenzy_fridayThis is a get call, and should not need authorization. But on the promoter somehow the default docker login is zed09:47
* frenzy_friday trying to find more09:47
akahatfrenzy_friday, this is strange.09:47
marios|ruckfrenzy_friday: looking09:47
marios|ruckfrenzy_friday: yes i see so something in the promoter env is overriding 09:50
frenzy_fridaymarios|ruck, ~/.docker/config.json09:50
frenzy_fridayI renamed this , and now the auth is gone09:51
marios|ruckfrenzy_friday: good, that might be it. but why/how for this config.json 09:51
frenzy_fridaymarios|ruck, did we try docker login manually from the promoter vm?09:52
frenzy_friday* docker login to zed registry09:52
frenzy_fridaythat would have saved the creds in the config09:52
marios|ruckfrenzy_friday: could be 09:54
marios|ruckfrenzy_friday: few mins promoter should try again lets see 09:54
marios|ruckbut :/09:54
frenzy_fridaycan we run the promoter now? 09:55
frenzy_fridayOr will it runin some time?09:55
frenzy_fridayoh, you answered already :P09:55
marios|ruckfrenzy_friday: it will rerun in a few 09:55
marios|ruckfrenzy_friday: looks good running10:01
marios|ruckdamn exception :/10:01
marios|ruckffs10:02
* frenzy_friday looks10:02
frenzy_fridayGenerate list of containers to push passed this time10:05
marios|ruckyeah 10:05
marios|ruckfrenzy_friday: also Login to source registry10:05
marios|ruckattempt a push fails10:05
frenzy_fridayah, bad gateway10:06
marios|ruckhmm10:06
frenzy_fridaysaw a ping from daniel, they are adding a new volume10:07
marios|ruckright maybe related10:07
marios|ruckk thanks frenzy_friday going to get some food and lets see in next run then 10:07
* marios|ruck food biab10:07
frenzy_fridaycool10:07
jpodivin_marios|ruck: Hi. I'm having some issues with the override-checkout parameter. It seems that jobs don't really care for it? Even when it's defined the normal branch matching rules take over.10:13
*** jpodivin_ is now known as jpodivin10:14
jpodivinIt makes work with non-traditional branch names rather painful.10:15
jpodivinthat being said, I can work around it, by either making my own jobs with right nodeset or by changing the matching rule in openstack-zuul-jobs10:16
jpodivinThat second option isn't a what I would go for however.10:17
*** ysandeep|lunch is now known as ysandeep10:18
marios|ruckjpodivin: ok if you give me more information (where are you seeing this, logs, patch etc) and i can try check later10:27
jpodivinmarios|ruck: thanks, see this buildset https://zuul.opendev.org/t/openstack/buildset/5d653e1a3e3149c7a7e5a62ad2bd7110 And the associated patch https://review.opendev.org/c/openstack/validations-common/+/866343/10:28
jpodivinsearch for 'openstack-tox-linters' that's the one I'm trying to bring back on line. 10:29
marios|ruckjpodivin: k 10:30
marios|ruckdpawlik: still adding voluem or done? 10:31
marios|ruckdpawlik: we hit 502 like yesterday (assuming this is why you are adding new vol, out of space)10:31
marios|ruckdamn again frenzy_friday running now... so lets see what dpawlik says10:35
marios|ruckFAILED! => {"changed": true, "cmd": "docker build -t quay.rdoproject.org/tripleowallabycentos8/nop - && docker push quay.rdoproject.org/tripleowallabycentos8/nop\n", "delta": "0:00:00.670944", "end": "2022-12-06 10:34:36.145891", "msg": "non-zero return code", "rc": 1, "start": "2022-12-06 10:34:35.474947", "stderr": "received unexpected HTTP status: 502 Bad Gateway",10:35
dpawliknot yet10:35
marios|ruckdpawlik: ok thanks, we are hitting 502 now for container push ^^^ 10:35
dpawlikthere was small issue 10:35
dpawlikit can be: /dev/mapper/vgimage-image  1.0T  1.0T  128K 100% /var/data10:36
marios|ruckdpawlik: ack 10:36
dpawlikwe are checking one thing due we don't want to loose data10:45
*** dviroel|out is now known as dviroel11:12
dpawlikmarios|ruck: partition resized11:16
dpawlikmarios|ruck: now should be OK11:16
marios|ruckdpawlik: yes, looks like it is pushing now lets see http://promoter.rdoproject.org/promoter_logs/centos8_wallaby.log 11:17
dpawliksorry for delay, one feature were added and took some time to get information if it can be done with my fix11:17
marios|ruckthanks for helping me frenzy_friday ^^^^  11:17
marios|ruckdpawlik: thx, seems it was multiple issues same time (creds leak, disk space, local config on promoter... last one is probably related to debug from creds change)11:18
*** rlandy|out is now known as rlandy|rover11:18
frenzy_friday\0/ Hope this time it promotes11:18
marios|ruckfrenzy_friday: well its been pushing for ~ half hour should be ok 11:19
frenzy_fridayawesome! I'll delete the backup of docker creds json11:19
marios|ruckfrenzy_friday: leave it for now since you renamed should be OK, maybe add 'delme' to the name or something11:20
marios|ruckfrenzy_friday: just incase we want/need debug there11:20
dpawlikI thought I would come back from PTO and it would be so quiet in infra....11:20
marios|ruckfrenzy_friday: hm actually not sure it would be useful we already updated the timestamp on it... k whatever delete is fine 11:20
frenzy_fridaycool11:21
rlandy|rovermarios|ruck: frenzy_friday: hey - reading back on promoter/container push issue11:21
rlandy|rovermarios|ruck: pls ping when you have time to sync11:21
marios|ruckrlandy|rover: sure lets sync in a few11:21
frenzy_fridayrlandy|rover, tldr: it was an issue on the promoter server vm. Solved now11:21
rlandy|rovermarios|ruck: ok - chceking what happened with 17.111:21
marios|ruckrlandy|rover: good there we got green run 11:22
marios|ruckrlandy|rover: was updating that right now thats why need few mins11:22
rlandy|roverfailed periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17.1 - ah O see your rerun11:22
rlandy|rovercool - last night's run passed11:23
rlandy|roveryay11:23
rlandy|roverysandeep: chandankumar: hi - if you have any outstanding travel expenses, pls submit today11:26
chandankumarrlandy|rover: all submitted already.11:26
rlandy|rovergreat11:26
ysandeeprlandy|rover, same for me, all submitted already11:29
rlandy|rovergreat11:30
*** yadnesh is now known as yadnesh|afk12:00
rlandy|roverdpawlik++ frenzy_friday++ marios|ruck++ for working through promoter issue12:00
rlandy|roverysandeep: hey - do we need to chat re: nova operator?12:07
ysandeeprlandy|rover, ack we can and we need a bigger discussion about inrepoconfig, would be good if dviroel can also join.12:08
dviroeli can join now12:09
ysandeepdviroel, rlandy|rover meet.google.com/xta-zxuf-cvn12:10
chandankumardviroel: ysandeep rlandy|rover on a side note, crc takes approx 5 mins to start up in zuul job https://review.rdoproject.org/r/c/testproject/+/46362/1#message-66d25756952f8e35a1fbfb24b4d63b69b2da565612:12
marios|ruckrlandy|rover: especially frenzy_friday helped me a lot today 12:12
*** frenzy_friday is now known as frenzy_friday|food12:21
rlandy|rovermarios|ruck: hmm ... did you recheck the fs001 patch for 17.1?12:49
rlandy|roverlooks like it reset12:49
marios|ruckrlandy|rover: yeah rechecked cos the fail was inconsistent - looking12:49
rlandy|roverah12:50
marios|ruckrlandy|rover: yeah see links https://code.engineering.redhat.com/gerrit/c/testproject/+/437138/1#message-d157ff5f9bb9b31fc59a8e0f1d0513d48536a875 12:50
rlandy|rovermarios|ruck: also - line kicked again12:50
marios|ruckrlandy|rover: may still be a thing but not sure yet 12:50
marios|ruckrlandy|rover: both in provisioning, but different issue in those two runs so far ^^ 12:50
rlandy|roverok - there will be your run plus the line in general12:51
*** yadnesh|afk is now known as yadnesh13:10
marios|ruckjpodivin: had a look - commented there fyi https://review.opendev.org/c/openstack/validations-common/+/866343/12#message-831919147cd28d75c2d6dbf7c98d8eaf04ae4e46 13:18
marios|ruckjpodivin: mainly that branches: != override-checkout (looking at https://zuul.opendev.org/t/openstack/buildset/5d653e1a3e3149c7a7e5a62ad2bd7110 and 'openstack-tox-linters' .. the variants there are determined by branches: 13:18
marios|ruckjpodivin: so it does seem to honour the override-checkout - see https://c978daf28b4ba7352ce1-f4b3e6a1d73b0d4e13079b70a1af7060.ssl.cf2.rackcdn.com/866343/12/check/openstack-tox-linters/f915a35/zuul-info/inventory.yaml projects: e.g. opendev.org/openstack/requirements           checkout: stable/train13:20
*** frenzy_friday|food is now known as frenzy_friday13:30
*** ysandeep is now known as ysandeep|dinner13:34
marios|ruckrlandy|rover: master promoting13:34
*** dasm|off is now known as dasm14:04
dasmo/14:04
rlandy|rovermarios|ruck: zed is promoting14:06
marios|rucknice14:08
marios|ruckrlandy|rover: fs1 is now running tempest (no consistent provision issue then .. we can see if we get it again anyway)14:16
marios|ruckrlandy|rover: (17.1/9)14:16
rlandy|rovermarios|ruck: also jenkins didn't kick again14:16
rlandy|roverso I manually trggered that14:16
marios|ruckrlandy|rover: cool 14:16
jpodivin_marios|ruck: right, but the documentation implies that it also works for job selection14:17
jpodivin_https://zuul-ci.org/docs/zuul/latest/config/job.html#attr-job.override-checkout14:17
jpodivin_the second paragraph 14:17
*** jpodivin_ is now known as jpodivin14:17
marios|ruckjpodivin: reading (zuul docs are always hard to read :/ i mean i usually have to re-read things a few times )14:19
marios|ruckjpodivin: didn't realise this overrides branch for job selection 14:21
marios|ruckjpodivin: in your logs though, the override-checkout does seem to work ok did you see my comment?14:21
marios|ruckjpodivin: it checks out train for required-projects14:21
jpodivinmarios|ruck: yeah, that part is working, but in this context what I really want it to get the "stable/train" version of the job14:23
marios|ruckjpodivin: i see i thought the problem was with required-projects checkout. ok i'll have another pass and see if can spot something else 14:23
jpodivinmarios|ruck: thanks14:28
jpodivinWorst case scenario I'll have to define our own jobs. 14:28
rlandy|roverchandankumar++14:43
rlandy|rovernicely done14:43
chandankumarrlandy|rover:  thank you :-)14:44
chandankumarysandeep|dinner++ thank you for figuring out how to make job required and testing together14:45
*** ysandeep|dinner is now known as ysandeep__14:46
ysandeep__chandankumar++ great work on Zuul POC and today' demo14:47
ysandeep__rlandy|rover, chandankumar dviroel we have gha/prow/zuul already, we should throw some jenkins into the picture too :D 14:49
rlandy|roverysandeep__: don't even :) ... it's coming14:50
rlandy|roverwe don;t know what QE wants yet14:50
ysandeep__hahaha.. true 14:50
dviroel~o~14:52
rlandy|roverpojadhav: you around?14:56
rlandy|rovercommunity call in a few14:57
rlandy|roverchandankumar: hey - we have no agenda items for community call - will you be there? maybe you can show your demo again14:57
chandankumarrlandy|rover:  yes sure, I will be there14:57
rlandy|roverchandankumar: let's see if we get a guest14:58
pojadhavrlandy|rover, yep14:59
rlandy|roverakahat: frenzy_friday: pojadhav: bhagyashris: dviroel: ysandeep__: chandankumar: marios|ruck: rcastillo: jm1: dasm: community call15:00
dasmthx15:00
dasmjoining15:00
pojadhavfolks, https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg#2022-12-06-Community-Call15:00
ysandeep__dasm: o/ your light setup looks great15:01
dasmysandeep__: thanks!15:02
dasmysandeep__: it's ligth strip at the wall in front of my eyes15:02
dpawlikfrenzy_friday, marios|ruck: on the end, what was the issue with authentication to the quay?15:03
pojadhavrlandy|rover, sorry was out for dr appt 15:03
frenzy_fridaydpawlik, we did a docker login -u ... -p ... from the promoter vm (which retags and pushes the containers)15:03
dpawlikaha15:04
frenzy_fridaythis created a file ~/.docker/config.json - which overrides every other auth15:05
rlandy|rovermarios|ruck: going to retrigger wallaby c9 - since that is the only one not promoted15:14
marios|ruckrlandy|rover: k thx15:16
dasmpojadhav: o/ did you get my pm?15:19
pojadhavdasm, yes.. will check after community call 15:20
dasmpojadhav: ack, thx15:20
Tengurlandy|rover: heya! quick headsup: https://review.opendev.org/c/opendev/system-config/+/866175 will be merged shortly, therefore I should be able to finalize the tripleo-ansible change to use the galaxy proxy.15:23
Tengurlandy|rover: also: https://bugzilla.redhat.com/show_bug.cgi?id=2150283   container-selinux issue should hopefully be sorted out shortly.15:24
Tengu(meaning openstack-selinux should properly install, so the vTPM boolean should be good as well)15:24
marios|ruckrlandy|rover: ysandeep__: merge this please https://review.rdoproject.org/r/c/rdo-jobs/+/46334 15:29
marios|ruckrlandy|rover: i'll abandon that one https://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-config/+/436919 15:29
rlandy|rovermarios|ruck: done15:30
*** dviroel is now known as dviroel|lunch15:31
ysandeep__marios|ruck, done double +wed :D15:31
marios|rucktotally merged thanks ysandeep__ 15:31
chandankumarsee ya!15:31
*** pojadhav is now known as pojadhav|afk15:32
* ysandeep__ out, see you all tomorrow, have a great rest of your day/evening o/ 15:37
*** ysandeep__ is now known as ysandeep|out15:50
rlandy|roverdasm: happy RH anniversary!!!!!!!!!!!15:51
*** yadnesh is now known as yadnesh|away15:51
dasmrlandy|rover: thank you :)16:05
dasm4 more years and I'm gonna get a puck! :)16:06
*** dviroel|lunch is now known as dviroel16:34
marios|ruckrlandy|rover: off in a minute need sthing?16:43
marios|ruckrlandy|rover: i think we got all the things, maybe just watch these for me please? https://review.opendev.org/c/openstack/tripleo-ansible/+/866566 for train/cix and https://review.rdoproject.org/r/c/rdo-jobs/+/46334 still waiting in the queue for some reason16:44
rlandy|rovermarios|ruck: should be fine16:44
rlandy|roversure - thanks16:44
rlandy|roverlunch - brb16:49
*** marios|ruck is now known as marios|out16:55
*** pojadhav|afk is now known as pojadhav18:37
rlandy|roverakahat: we tried to review your registry patches ... 19:07
rlandy|roverthere are two sets - one today and one from yesterday19:08
rlandy|roverI added both sets to tomorrow19:08
rlandy|rovercan you present the order and logic at tomorrow's review meeting19:08
rlandy|roverso can go ahead?19:08
dasmoh, review meeting.19:28
dasmi forgot about that.19:28
dasmsorry19:28
*** dviroel is now known as dviroel|biab19:48
* dasm => offline22:51
dasmo/22:51
*** dasm is now known as dasm|off22:51
*** rlandy|rover is now known as rlandy|out23:01

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!