Tuesday, 2022-08-23

*** soniya29 is now known as soniya29|ruck04:12
*** ysandeep|out is now known as ysandeep05:04
jm1hello hello 😎06:29
ysandeepjm1, morning o/ How is everything?06:45
jm1ysandeep: today is going to be another hot day with 30°C, so a perfect day for refactoring ansible modules! oh, wait... XD06:53
jm1ysandeep: how is life in openshift? ^^06:56
ysandeeptemperature here is close your place - ~32°C and I need to write some documentation today for our CI exploration in Openshift world :)06:58
*** ysandeep is now known as ysandeep|afk07:34
*** jpena|off is now known as jpena07:38
jm1ysandeep|afk: oh nice! curious to read that doc :)07:41
*** pojadhav|out is now known as pojadhav09:09
*** rlandy|out is now known as rlandy10:38
*** soniya29|ruck is now known as soniya29|ruck|lunch10:39
rlandyysandeep|afk: Tengu: ok to merge? https://review.opendev.org/c/openstack/tripleo-heat-templates/+/85273410:45
rlandytest looks green10:45
ysandeep|afkrlandy, yes please 10:45
ysandeep|afkrlandy, we also need https://review.opendev.org/c/openstack/tripleo-quickstart/+/852594 along with tht change10:46
*** ysandeep|afk is now known as ysandeep10:46
rlandydone10:46
ysandeeprlandy++ thanks!10:48
rlandypojadhav: frenzyfriday: ok to merge https://code.engineering.redhat.com/gerrit/c/openstack/rrcockpit/+/425563?10:48
frenzyfridayrlandy, yep. Voted10:49
rlandypojadhav: frenzyfriday: merged10:50
rlandypls check cockpit in a  bit to see results10:50
rlandyhmmm ... guess my disk full experiment didn;t work10:50
rlandyysandeep: have a few to chat about podified tests?10:50
rlandypojadhav: frenzyfriday: anything missing now from 17.1 on rhel9?10:51
rlandyI merged the criteria patch10:51
rlandyand fixed multinode10:51
frenzyfridayrlandy, nope, I think we have everything10:51
rlandyfrenzyfriday: anything ready for review on 17.1 on rhel-8?10:51
ysandeeprlandy, sure meet.google.com/fwr-jnri-qpt10:52
frenzyfridayhttps://code.engineering.redhat.com/gerrit/c/openstack/tripleo-ci-internal-jobs/+/424293 is in progress. bhagyashris has a testptoj running for the jobs (container build is passing now)10:53
rlandybhagyashris; hey - welcome back10:53
rlandypls see question to frenzyfriday above10:53
rlandywe merge some rhel-8 stuff10:53
rlandypls send the next set of reviews when you have them10:53
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.10:53
*** soniya29|ruck|lunch is now known as soniya29|ruck10:53
bhagyashrisrlandy, hi10:55
bhagyashrisrlandy, sure10:55
bhagyashrisrlandy, there is problem on standalone deployment and scenarios so looking at them https://sf.hosted.upshift.rdu2.redhat.com/logs/92/424992/14/check/periodic-tripleo-ci-rhel-8-scenario001-standalone-rhos-17.1/5d786d6/logs/undercloud/home/zuul/standalone_deploy.log10:56
pojadhavrlandy, ack11:04
rlandybhagyashris: ok - let's meet and see where you are11:09
Tengurlandy: \o/ and now we'll be able to ensure auditd doesn't blow with next releases :). Thanks!11:09
rlandybhagyashris: https://meet.google.com/mhv-eqwn-zsc?pli=1&authuser=011:11
bhagyashrisrlandy, give me min11:15
rlandyyou have containers pushed11:15
*** dviroel is now known as dviroel}rover11:26
*** dviroel}rover is now known as dviroel|rover11:26
dviroel|rovero/11:29
soniya29|ruckhello11:36
rlandydviroel|rover: soniya29|ruck: hi - need help with anything?11:40
rlandylooking good except the security component on wallaby c911:40
rlandymaster is a bit out11:40
soniya29|ruckrlandy, yupp11:40
dviroel|roverlooking at master fs39 and fs6411:40
soniya29|ruckrlandy, with security we are still struggling with ipa issue 11:41
dviroel|roverfailing to install ansible-macros11:41
rlandyugh11:41
rlandythat comes and goes11:41
rlandydviroel|rover: soniya29|ruck: pls ping if you need help - otherwise we can sync if you prefer11:42
soniya29|ruckrlandy, sure11:42
rlandysoniya29|ruck: dviroel|rover: pls note that 17.1 on rhel-8 jobs are starting to show up11:43
rlandybhagyashris is working on that11:43
soniya29|ruckrlandy, ack11:43
rlandyyou shouldn't have to worry about debugging those yet11:43
rlandyuntil the integration line is up11:43
chandankumarrlandy: dviroel|rover ysandeep https://review.opendev.org/c/openstack/tripleo-ci/+/849806 last patch in series to finish zuul parenting cleanup11:45
rlandyfrenzyfriday: will the 17.1 on rhel-9 page just show up? or do we need a restart/refresh there?11:46
rlandydviroel|rover: soniya29|ruck: ^^ fyi11:46
rlandypojadhav: ^^? don;t see the page yet11:46
frenzyfridayrlandy, it should shouw up on its own, I saw there was a cron job running in the VM. But afuscoar's dashboard also didnt show up after the patch was merged. I am checking if we need extra steps11:47
rlandyfrenzyfriday: thank you 11:47
pojadhavrlandy, i think we need to restart downstream cockpit server to have refreshed changes.11:47
rlandypojadhav: frenzyfriday: ok - pls take care of that so rr have correct tracking11:48
pojadhavack11:48
rlandyakahat: hello - how are you doing with the mixed-rhel reviews?11:48
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.11:48
rlandyleft some comments on friday11:48
rlandylet's touch base on what's left11:48
rlandyand pls add anything else to wed review time so you can be there to explain11:49
ysandeepchandankumar, +wing but just a note that - https://opendev.org/openstack/tripleo-docs/src/branch/master/doc/source/ci/tripleo_ci_job_parenting.rst#L32 - we need to update doc12:04
frenzyfridayfolks, we are trying to pull quay.io/centos/centos:stream9 to rebuild the telegraf_py3 container - but it is failing ith Service 'telegraf_py3' failed to build: missing signature key12:11
frenzyfridayDo we need to do a docker login or something?12:11
chandankumarysandeep: thanks and done here: https://review.opendev.org/c/openstack/tripleo-docs/+/85417712:11
frenzyfriday^ on the internal cockpit server12:11
ysandeepchandankumar, thanks!12:20
akahatrlandy, trying to get this merged... but zuul is not happy: https://review.opendev.org/c/openstack/tripleo-quickstart/+/85381412:27
akahatrlandy, sure, once reviews are ready.. i'll add it.12:27
reviewbotDo you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks.12:27
akahatthis is also ready to merge: https://review.rdoproject.org/r/c/rdo-jobs/+/4459112:28
* pojadhav brb12:39
rlandyakahat: k - can look after meetings - thanks for picking this up12:49
dviroel|rovervexxhost dns issue is killing us12:50
rlandydviroel|rover: it's worse?12:58
rlandydviroel|rover: pls get on #vexxhost and tell them12:58
rlandyping guilhermesp12:58
dviroel|roverrlandy: looks worse than yesterday12:58
rlandyor comment on the card12:58
rlandyok  let me see if I can get the ticket12:58
rlandydviroel|rover: forwarded you email12:59
rlandydviroel|rover: the ticket was closed13:04
rlandydviroel|rover: google dns was the trouble before13:20
dviroel|roverrlandy: good to know, maybe this is the new issue now13:23
pojadhavfolks community meeting in 7 mins : arxcruz, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29|ruck , pojadhav, akahat, chandankumar, frenzy_friday, anbanerj,  dviroel|rover , rcastillo, dasm, jm113:23
rlandypojadhav: do we have anyone scheduled to come to the community call?13:23
pojadhavkindly add agenda if anyone wants to discuss : https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg13:23
pojadhavrlandy, IMO nope, because agenda is empty for today.13:25
*** dasm|off is now known as dasm13:28
dasmo/13:28
chandankumarWho has deployed the downstream cockpit?14:15
chandankumarit is on c714:15
chandankumarwhen we pull cs9 image via docker14:15
chandankumarit fails due to missing signature issue14:15
chandankumarreason: Docker suport manifest v2 but cs9 image is based on cs814:16
ysandeepchandankumar, may be pojadhav knows? 14:16
chandankumarI tried to workaround that time not successed14:16
chandankumarI pulled the image via buildah14:16
pojadhavysandeep, nope I didnt deployed prod d/stream server14:16
chandankumarbut image is not recognized via docker14:16
ysandeepchandankumar, afaik.. its very old deployment , may be Wes deployed it earlier14:17
chandankumarmay be we should move a to a new server14:17
frenzyfriday++ to new server14:17
chandankumarand deploy it there14:17
chandankumarif ok to all?14:17
chandankumarSince we are moving to new server, we can also remove docker command from there and switch to podman or podman compose14:18
ysandeep+1 to move to new server and using podman compose14:18
chandankumarfrenzyfriday: may be we can add acard for new sprint and then someone can pick up and try that out14:19
ysandeepdasm did you already had a patch to use podman compose for cockpit, I recall we discussed using podman - not sure was that just conv or I saw a patch somewhere.14:19
frenzyfridaychandankumar, ack14:19
dasmysandeep: i use podman compose locally, but i didn't submit any patch. i don't have docker on my pc14:22
dasmysandeep: i can do that today. it shouldn't take a long tim14:22
dasme14:22
ysandeepdasm: thanks! 14:25
ysandeepdasm: I have commented on rr patch, https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43307/23#message-8ca7365d9b6f2cc62043c525b91035d4c477c0e014:26
ysandeepsome unrelated jobs showed up in the tables :)14:27
dasmysandeep: i just reran this command and i don't see them14:28
dasmhmm14:28
dasmysandeep: https://privatebin.corp.redhat.com/?12ecf3817f22b027#CpzeTxGW9dQkovoBTNmveFGvxW5MaGJHLsSQWiegPpxt14:28
dasmhmmm... i'll check if i missed something14:29
ysandeepdasm, I think it happens when we don't have any job failing/pending(which is in criteria)14:29
dasmhmm...14:30
* ysandeep cross checking14:30
dasmi might have an idea14:30
dasmysandeep: this might be culprit: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/43307/23/ci-scripts/infra-setup/roles/rrcockpit/files/telegraf_py3/ruck_rover.py#63314:31
dasmbut i'm not sure why i don't see it on my instance14:32
ysandeepdasm, try with older hash now - rr --release wallaby --distro centos-8 --aggregate_hash tripleo-ci-testing/f3/e6/f3e69032a410039458734c3b4f18cf06 14:34
dasmysandeep: thanks for the reproduction14:35
ysandeepdasm: happy to help! 14:35
*** ysandeep is now known as ysandeep|dinner14:36
dasmysandeep|dinner: you're right. "to_promote" is empty14:36
dasmysandeep|dinner: submitted the update.14:39
pojadhavrlandy, dasm, ysandeep|dinner, chandankumar, soniya29|ruck : hey I forgot to setup UA/TC/PM planning meeting today. can I setup for tommorrow @ 1pm UTC ??14:53
rlandypojadhav: ack - pls book that meeting14:54
pojadhavrlandy, sure14:54
rlandy1pm is the tempest meeting14:55
rlandypojadhav: we have retro on wed14:55
rlandyand planning on thurs14:55
rlandyatm14:55
pojadhavyeah14:55
rlandybhagyashris: ^^ pls help pooja the new timing14:55
rlandywhich was I think retro and planning both on thurs14:55
pojadhavwe can do UA/TC sync up @12.30 UTC then tommorrow before tempest meeting14:56
pojadhavif we all agrees to have retro and planning together on thurs then we can utilize time of retro for UA/TC sync up.14:58
*** dviroel|rover is now known as dviroel|rover|lunch14:59
bhagyashrisrlandy, sure15:01
* soniya29|ruck leaving for the day15:06
*** soniya29|ruck is now known as soniya29|out15:06
rlandyjm1: hi - still around?15:18
rlandyrcastillo: hi - have you deployed the downstream cockpit lately?15:26
rcastillorlandy: no15:27
rcastillosomething up with it?15:27
dasmrcastillo: chandankumar noticed it running on c7 not cs915:34
dasm> 14:15:24    chandankumar | it is on c7 15:34
rcastilloindeed it is15:35
rcastillografana, influx, telegraf all run on docker so it's not the end of the world15:37
rcastillobut I'm sure we can look at upgrading if there's any concerns15:37
dasmindeed15:38
rcastillowe're already on cs9 upstream so I don't see why not15:38
chandankumarrcastillo: i have no issue in upgrading the box to cs816:11
rlandychandankumar: spoke with frenzyfriday: 16:12
chandankumarthe main issue is with cs9 container image16:12
rlandyI think the box is a vm on hardware16:12
rlandywe should be using our psi tenant I think16:12
rlandyiiuc16:12
chandankumarok16:12
rlandyso we can create a new vm there16:12
rlandyand try it out16:12
rlandyjm1 can confirm16:12
rlandymaybe we used hardware to get server name16:12
rlandyI can't remember the logic there16:13
rlandyit was also hosting wimp16:13
chandankumarrlandy: please +w it https://review.opendev.org/c/openstack/tripleo-docs/+/85417716:14
rlandydone16:15
rlandylunch brb16:15
*** dviroel|rover|lunch is now known as dviroel|rover16:27
*** ysandeep|dinner is now known as ysandeep|out16:27
*** jpena is now known as jpena|off16:34
rlandydviroel|rover: looking better on vexx??17:14
rlandyalso my bigger node experiment didn't work17:15
rlandystill registered disk full17:15
rlandydoesn't seem to fail the job though17:15
dviroel|roverrlandy: yeah, looking better now17:17
dviroel|roverlets see how main pipeline goes17:18
rlandy\o/17:18
dviroel|roverstill trying to promote master, missing 039 and 06417:18
dviroel|rovertrying on vexx and ibm now17:18
dviroel|roveron vexx with node affinity17:19
rlandynice17:21
rlandylet me know if it makes a diff17:21
rlandygit some admin stuff to take care of17:22
dviroel|roverok17:22
rlandywill look at the big node thing afterwards17:22
dviroel|roverthe good thing is that we have resources for main and testprojects17:23
dviroel|roverno stable4 in the next hours17:23
rlandygreat17:25
rlandyhow was the train run earlier? better?17:25
dviroel|rover3 ovb jobs failing, didn't check the results17:32
dviroel|rovernot bad17:32
rlandyok17:49
rlandyfrenzyfriday: left you updates on the doc18:24
rlandyhmmm ... we're out some components on 16.218:31
dasmdviroel|rover: do you know how we deploy cockpit stuff? i'm looking into change from docker to podman18:32
dviroel|roverdasm: we have a role that does that, in ci-config18:34
dviroel|roverdasm: https://github.com/rdo-infra/ci-config/tree/master/ci-scripts/infra-setup18:35
dasmi saw setup_docker_compose but i'm not sure how it's triggered18:35
dviroel|roverhttps://github.com/rdo-infra/ci-config/blob/master/ci-scripts/infra-setup/servers_setup.yml18:35
dasmso, just "full-run.sh" then? that is not very complicated :)18:35
rlandydviroel|rover: what's the latest on security fox for wallaby?18:35
dasmdviroel|rover++ thanks18:35
dviroel|roverSee that there is different roles for different hosts ^18:35
dviroel|roverrlandy: we need e3afdf14b6 on wallaby18:37
dviroel|roverfor ansible-tripleo-ipa 18:38
dviroel|roverrlandy: sorry, that was master commit, we needd 1c47cb9596 for wallaby18:39
rlandyrekicked downstream jenkins on that component18:39
rlandyhttps://review.opendev.org/c/x/tripleo-ipa/+/853765 merged?18:39
rlandydviroel|rover: ^^18:39
dviroel|roveryes18:40
dviroel|roverhttps://trunk.rdoproject.org/centos9-wallaby/component/security/component-ci-testing/18:40
rlandyok18:40
dviroel|roverwe didn't promoted yet18:40
rlandyok18:40
rlandy periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp_1supp-featureset039-security-wallaby still failed18:41
rlandyI see failed your rerun18:41
dviroel|roveryeah, failed to download packages18:43
rlandycharming18:51
jm1rlandy, rcastillo, chandankumar, ysandeep: downstream cockpit is running in a vm which is hosted on a rh internal server. it is not hosted on vexxhost and it hasn't been moved there because in the past it had quite heavy resource requirements iirc. atm it has 64gb ram, 16vcpus and >100GB storage19:18
rlandyjm1: on hardware box - not psi?19:19
rlandydo you know why we did not use psi19:19
rlandyvexx is public and would not be an option19:19
rlandyatm it has 64gb ram, 16vcpus and >100GB storage - would be ok on psi19:20
jm1rlandy: that vm was created in 2019 if i interpret timestamps correctly, so looong before my time at rh. maybe psi wasnt a thing back then?19:21
jm1rlandy, rcastillo, chandankumar, ysandeep: anyway. as long as psi has a suitable flavor and enough disk space, we can "migrate" that vm to psi. there are several ways we could do this. we can discuss this with the team19:26
jm1rlandy: quickest way to solve chandankumar's issue would be to upgrade that vm from c7 to cs8 to cs9 though19:28
dasmjm1: what's the chandankumar's issue here? is it only: "it's running c7" thing?19:32
dasmif that's the case, i'm eager to keep it running and not touch it.19:32
dasmslowly we can work on moving the stuff over to cs919:32
rcastilloI'd rather have it on psi, we've already had issues with that particular bm machine19:34
rlandyakahat: hey  https://review.opendev.org/c/openstack/tripleo-quickstart/+/853814 is merged - left comment on https://review.rdoproject.org/r/c/rdo-jobs/+/44591 - quick change - then can merge19:38
rcastilloI'd volunteer for doing the migration but I'm not 100% on how we run infra-setup downstream19:39
jm1dasm: chandankumar's issue: https://meetings.opendev.org/irclogs/%23oooq/%23oooq.2022-08-23.log.html#t2022-08-23T14:16:0819:39
jm1rcastillo: inside the vm its the same as with all other vms: ansible-pull on our ci-config repo. the libvirt vm config itself has been defined manually.19:40
dasmjm1: ack, thx19:41
jm1rcastillo: so actually we would benefit from migrating to psi because this would mean that you would define the vm setup itself in our ci-script repo19:42
rcastillojm1: ok nice19:43
dasmvagrant ftw! ;)19:43
jm1rcastillo: actually i doubt its the most pressuring issue atm. but go ahead if you like :) no objection from my side19:45
jm1dasm: in rh we use ansible and aoc XD19:45
rcastillojm1: ack it's not urgent19:49
rcastillomight be faster to upgrade current vm19:49
rcastillobut scared of c7->c8 breakage :/19:50
jm1rcastillo: ping be before you wanna start and we do a backup of the underlying storage volume19:51
* jm1 out for today. have a nice evening!19:53
rcastillobye jm1, have a good one o/19:54
rlandyrcastillo: ananya is starting the move to the vm19:59
rlandybiab20:05
dviroel|roverrlandy: dasm: zuul seems not responding my testproject patches, same on your side?20:59
rlandydon't know - been on confluence - will check now 21:00
rlandydviroel|rover: which zuul?21:00
dviroel|roverrlandy: rdo21:00
rlandydo you have a testproject job you want me to try21:00
rlandycurrent missing jobs in master?21:01
dviroel|roverfs39 on master 21:01
dviroel|roverhttps://review.rdoproject.org/r/c/testproject/+/38646/49/.zuul.yaml21:01
dviroel|roverdidn't start chasing last pipeline run, there was many ovb failures there21:02
rlandydviroel|rover: rr tool is not returning21:02
rlandyoh - now it did21:02
rlandylet me try both21:02
dasmdviroel|rover: i don't have any testproject running atm.21:03
dviroel|rovernobody has :/21:03
rlandyhmm ...21:04
rlandydidn't start21:04
rlandychecking zuul errors21:04
rlandydviroel|rover: time to ping on #rhos-ops21:05
rlandydviroel|rover: what's the bst way to export/save a slack thread?21:23
dviroel|roverrlandy: good question21:25
rlandyI want to add the thread from today to the confluence page21:25
dviroel|roverrlandy: you can copy a link to this thread21:28
dviroel|roveron: More Actions21:28
dviroel|roverdasm: rcastillo: Mexico vs Poland, in world cup, who is going to win this one?22:03
rlandylol - rivalry 22:06
dviroel|roverwe will need to wait until nov 22th22:07
rcastilloI really hope mexico gets 2nd in the group22:09
rcastillono hope for first lol22:09
dviroel|roverrcastillo: must win poland one, because argentina is a hard one22:11
rcastilloyeah I don't have my hopes up for argentina, but we have an ok shot against the other two22:12
dviroel|roverrlandy: yeah, looks like rdo zuul doesn't want to work anymore22:13
rlandydviroel|rover: you can give up for the day22:19
rlandyleave notes for soniya22:20
rlandyto ping rhos-ops in her morning22:20
* rlandy needs to go - will check in later22:20
*** rlandy is now known as rlandy|out22:20
dasmdviroel|rover: mexico :)22:26
dasmdviroel|rover: i'm not a huge fun of soccer. i prefer volleyball ;)22:27
* dasm => offline22:29
dviroel|roverdasm: yeah, thats also good to watch22:29
*** dasm is now known as dasm|off22:29
dviroel|roverdasm: o/22:29
dasm|offdviroel|rover: yup. all Brazil vs. Poland :)22:29
dasm|offit's fun to watch, especially with fast paced game!22:29
* dviroel|rover going off -> dinner22:30
dviroel|rovero/22:31
*** dviroel|rover is now known as dviroel|out22:31
* rcastillo out for today o/23:57

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!