Thursday, 2022-10-13

rlandyysandeep|out: chandankumar: hi - when you get in pls review/merge https://review.opendev.org/c/openstack/tripleo-ci/+/861113 - see gate blocker bug00:58
rlandythank you00:58
* rlandy out00:58
rlandytrain promoted00:58
rlandymaster should as well00:58
rlandy16.2 is just waiting in baremetal job00:58
*** rlandy is now known as rlandy|out01:00
jm1good morning folks :)04:50
marioso/05:08
jm1marios: o/ looks like we got our third gate blocker this week XD05:09
mariosjm1: :) \o/ you mean the upgrade followup from rlandy|out or something else/new ?05:27
jm1marios: oh wait, it has been fixed already05:30
jm1marios: yeah it was this upgrade follow up05:30
jm1marios: was confused by rlandy|out's "fyi - new gate blocker - reporting" immediately followed by https://bugs.launchpad.net/tripleo/+bug/1992668/comments/205:31
mariosjm1: ack forgot to add the gate layout but the fix merged this morning so should be ok05:35
mariosjm1: :/05:35
jm1marios: i love gate blockers which are resolved before i even boot my computer :D05:36
marios:)05:36
mariosbest kind05:36
*** ysandeep|out is now known as ysandeep05:55
jm1gthiemonge: hello :) when building the octavia amphora image on centos9 it fails in diskimage-builder's root.d/08-yum-chroot. any idea what has to be changed? https://bugs.launchpad.net/tripleo/+bug/1992668/comments/5 cc amoralej06:15
gthiemongejm1: hi06:31
*** amoralej is now known as amoralej|off06:33
gthiemongejm1: hmmm we had a similar issue in an upstream job that builds images for c9s, IIRC it was a SELinux issue in diskimage-builder, we disabled it for our job: https://review.opendev.org/c/openstack/octavia/+/84929106:33
*** amoralej|off is now known as amoralej06:37
jm1gthiemonge: interesting, thanks! actually i am wondering why we have zed listed here although i cannot find a job which creates that https://images.rdoproject.org/octavia/06:37
jm1marios: do you know which job creates this path? https://images.rdoproject.org/octavia/zed/06:37
jm1marios: in the past we had two jobs which created and pushed those images: octavia-build-amphora-centos8 and octavia-build-amphora-centos706:38
jm1marios: listed here https://review.rdoproject.org/r/plugins/gitiles/config/+/refs/heads/master/zuul.d/projects.yaml#31106:38
jm1marios: but those do not run on zed06:39
jm1amoralej: ^06:41
mariosjm1: so i asked about that recently for the zed branching... i asked amoralej to create that for us and we copied the master image into zed06:41
mariosjm1: indeed we *used* to have jobs that created it but seems not any more06:42
amoralejhow can i help?06:42
mariosamoralej: sorry o/ not sure something for you to do here i think jm1|ruck was asking about how we created the https://images.rdoproject.org/octavia/zed/ 06:43
amoralejah, i did it manually, as marios said06:43
jm1marios, amoralej: ah ok06:44
jm1marios, amoralej: for c7 and c8 we created the images periodically, maybe we should do for c9 as well? https://review.rdoproject.org/r/c/rdo-jobs/+/4562706:44
mariosjm1: yeah at least it would prevent the image going stale - if i understood the issue of the LP you referenced it was because of an old image used06:45
jm1gthiemonge: with your selinux disable patch we got one step further https://review.rdoproject.org/r/c/rdo-jobs/+/4563207:00
arxcruz|roveroh, so good to wake up and see promotions 07:01
jm1gthiemonge: now its failing with "No package basesystem available." https://logserver.rdoproject.org/28/45628/3/check/octavia-build-amphora-centos9/783933c/job-output.txt07:01
jm1gthiemonge: maybe this is related to https://github.com/openstack/diskimage-builder/commit/e8c7787ec84c436c3a9ef62fc57d6bf128023468 ?07:02
jm1gthiemonge: oh, probably not since this commit is already incorporated07:02
jm1arxcruz|rover: o/ you missed another the gate blocker :D07:02
arxcruz|roverjm1 yeah, i saw ronelle's email 07:03
arxcruz|roveri was huh?07:03
jm1arxcruz|rover: that patch has been merged so maybe you want to send a relief email?07:03
arxcruz|roverjm1 i'm rerunning wallaby cs8 job that fails 07:05
arxcruz|roveronly one failure, hope it passes 07:05
*** jpena|off is now known as jpena07:18
jm1gthiemonge, marios, amoralej: the c9 job for octavia-build-amphora is now failing in the same way as the c8 job. need some help here https://bugs.launchpad.net/tripleo/+bug/1992668/comments/707:29
gthiemongejm1: I'm looking at that07:39
gthiemongejm1: ok I see, "prometheus-proxy.service does not exist", it is called octavia-prometheus-proxy.service in the RPM07:42
gthiemongejm1: I will propose a patch to octavia master to fix that07:42
*** pojadhav is now known as pojadhav|afk07:49
mariosjm1: looking07:52
jm1marios: gthiemonge found the reason :)07:52
jm1gthiemonge: great, thanks :)07:52
mariosk thannks gthiemonge o/ 07:54
mariosi see https://review.opendev.org/c/openstack/octavia/+/861145 07:54
mariosjm1: the build job already providing benefit ;) ^ 07:54
jm1gthiemonge: rekicked testproject with your patch https://review.rdoproject.org/r/c/testproject/+/4562807:57
jm1gthiemonge: although that might not work since we need to rebuild packages.. we will see07:58
jm1amoralej: how can i run role build-test-packages in https://review.rdoproject.org/r/c/testproject/+/45628 to test gthiemonge's patch?08:06
amoralejmmm that's not a oooq job, build-test-packages is oooq specific (i never used it outside), let me try to find some way08:08
*** pojadhav|afk is now known as pojadhav08:16
*** ysandeep is now known as ysandeep|afk08:17
jm1amoralej: if there is no easy way then we can simply wait until gthiemonge's patch has been merged and we rebuild our octavia packages08:26
amoraleji'm trying to get a package with the patch in https://review.rdoproject.org/r/c/openstack/octavia-distgit/+/45639 and then add a temp repo in a DNM patch in rdo-jobs08:27
jm1amoralej: awesome :)08:30
jm1marios: do we want to fast forward this lp from new to done ;) https://trello.com/c/giPXN6TU/2752-cixlp1992699tripleociproa-gate-freeze-job-graph-failure-tripleo-ci-centos-9-undercloud-upgrade-depends-on-tripleo-ci-centos-9-co08:30
mariosjm1: sure go ahead i see you added needed info there about the patch already so fine08:37
mariosysandeep|afk: fyi updated https://review.rdoproject.org/r/c/rdo-jobs/+/45537/5/zuul.d/integration-pipeline-zed-centos9.yaml can you please revote when you get a chance 08:57
mariosfolks please review this when you have time https://review.rdoproject.org/r/c/rdo-jobs/+/45537 needed for zed (ovb job definitions)09:37
* jm1 lunch10:03
chandankumardpawlik: hello10:18
chandankumardpawlik: can you take a look at this bug https://bugs.launchpad.net/tripleo/+bug/199109310:18
chandankumardpawlik: tempest is failing on this job but its internal version of the job passing https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-internal-train&skip=010:21
chandankumardpawlik: I have fired a tp and hold the node also10:21
chandankumardpawlik: anything specific we can collect from the jobs vm to debug it10:22
chandankumar?10:22
*** rlandy|out is now known as rlandy10:24
rlandymarios: arxcruz|rover: jm1: hi - how are things?10:31
rlandymarios: arxcruz|rover: jm1: need help with anything?10:31
mariosrlandy: i believe quieter today :) sorry about the extra gate adventure and thanks for dealing with 10:32
rlandymarios: np - was an easy fix10:32
rlandywe should have spotted that in revies10:33
rlandyreviews10:33
arxcruz|roverrlandy all calm as marios said 10:33
rlandyarxcruz|rover: jm1: anyone dealing with the rhos-17 failures?10:33
arxcruz|roverwe had promtions, i'm trying now get wallab cs8 promoted 10:33
rlandyjm1: thanks for picking up the octavia bug fix10:34
rlandyarxcruz|rover: yea - train, wallaby, master and 16.2 were all promoting yesterday10:34
rlandy17 needs some help10:34
arxcruz|roveryeah, make my morning more pleasant 10:34
jm1rlandy: train and osp16-2 promoted today10:34
jm1rlandy: ok you are right, lets say yesterday since it was the hash from two days ago ;)10:35
rlandyjm1: my yesterday is your today10:35
rlandyie: they were passing yesterday - going to promote today10:35
rlandyCould not resolve host: osp-trunk.10:36
rlandylooks like 17 got hit by that10:37
jm1rlandy: its up again, so shall we simply rekick?10:38
rlandyjm1: yeah - pls rekick the 17 line10:38
rlandyalso some components need help ther10:39
rlandythere10:39
rlandy17.1 components10:39
rlandyarxcruz|rover: hey - anyone working on components?10:42
*** ysandeep|afk is now known as ysandeep10:42
arxcruz|roverrlandy not at the moment, i'll check 10:43
rlandyarxcruz|rover: pls start with 17.1 - then go to wallaby c9 and then master10:44
rlandyarxcruz|rover: you can let me know what you don't get to10:44
rlandyjm1: hi - when scenario010-kvm passes, pls update the criteria file to add it back to master criteria10:46
rlandyit's commented out now10:46
ysandeepmarios: ack, thanks for the update - revoted 10:48
jm1rlandy: that is not going to happen today or tomorrow. we need to get this octavia patch merged and then it has to trickle down all the way to rdo10:49
rlandyarxcruz|rover: jm1: is vexxhost better now? retry failures?10:54
arxcruz|roverrlandy yes, but i got an error trying to download rpm packages in one of my testprojects, i'm reruning now to see if it's a real issue or just random resolv problem 10:55
rlandyarxcruz|rover: ok - great10:56
rlandyarxcruz|rover: lmk and I'll update the ticket10:56
arxcruz|roverok10:56
rlandybhagyashris: hi ... 10:57
rlandybhagyashris: two patches for 17.1 on rhel-8 line were merged.10:58
rlandybhagyashris: can we close out that story now?10:58
rlandyjm1: giving luca's patches another try - here's hoping 11:00
jm1rlandy: one is/was still running 🤷11:02
rlandyfrenzy_friday: pls see pvt11:05
dpawlikhey chandankumar, checking11:06
jm1rlandy: rekicked the galera testproject again. the last failure was an unrelated tempest failure. getting all except one job to pass is very promising11:06
rlandyjm1: ok - great news11:09
dpawlikchandankumar: do you have ip for hold node?11:09
rlandyjm1: once luca's patch merge - there is still my patch outstanding11:09
*** rlandy is now known as rlandy|mtg11:09
chandankumardpawlik: https://review.rdoproject.org/r/c/testproject/+/45624 it has not yet started11:15
chandankumarAny ansible expert here11:15
chandankumarhow can we pass chrony_global_pool_settings: iburst minpoll 6 maxpoll 10 as a extra var to the ansible-playbook11:16
jm1chandankumar, marios: ruck rover tool reports that e.g. periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17 is still running. but i cannot find it on downstream's zuul status page. where else can i look?11:17
chandankumarhttps://paste.opendev.org/raw/bsFfuSs4WIRdSt7aZqSo/11:17
bhagyashrisrlandy|mtg, yes we can 11:17
chandankumaris my reproducer11:17
chandankumarbut it is not working as expected11:18
bhagyashrisrlandy|mtg, i have marked that as Done11:19
rlandy|mtgbhagyashris: thanks11:19
chandankumarsorry my bad I know the issue11:20
soniya29rlandy|mtg, ping11:21
mariosjm1: https://sf.hosted.upshift.rdu2.redhat.com/zuul/t/tripleo-ci-internal/builds?job_name=periodic-tripleo-ci-rhel-9-ovb-3ctlr_1comp-featureset001-internal-rhos-17 11:21
mariosfolks please add to your reviews https://review.rdoproject.org/r/c/rdo-jobs/+/45537 https://review.rdoproject.org/r/c/rdo-jobs/+/4561011:25
mariosshould be most/all of the integration & component jobs now ^^ 11:25
rlandy|mtgjm1: reports that when the job dies in the middel11:25
rlandy|mtgmiddle11:25
rlandy|mtgunless it'd passed, you need to rerun it11:26
rlandy|mtgie: dlrn has not reported so it has no result11:26
rlandy|mtgjm1: you know how to rekick the whole line?11:26
rlandy|mtgmay be easier at this point11:26
jm1rlandy|mtg: ah, i was wondering why dlrn returns None for this combination of aggregate hash and jobname. so "pending" actually means "job might still be running or has died"11:28
jm1rlandy|mtg: how to rekick the whole line?11:28
rlandy|mtgjm1: sending you pvt11:31
*** rlandy|mtg is now known as rlandy11:50
rlandysoniya29: pojadhav: thanks - reviewing now11:54
arxcruz|roverrlandy jm1 we might have some problem in our queue11:56
arxcruz|rovermy testproject is ~2h on queue waiting 11:56
rlandysoniya29: pojadhav: sending feedback on pvt - we are reviewing PTG presentations after scrum today11:57
rlandyarxcruz|rover: rdo?11:57
arxcruz|roverrlandy yes11:57
arxcruz|rovercheck the testproject jobs there11:57
rlandyarxcruz|rover: looks like a lot of stuff running now11:57
rlandyyou can kill the component promotion line11:58
rlandyalso we shoudl check with vexxhost if they replaced all the nodes11:58
rlandythey may still have some out to work on11:58
rlandywill ask them11:58
pojadhavrlandy, ack11:59
* dviroel coffee12:00
rlandyarxcruz|rover: updated the ticket and pinged on channel12:01
mariosarxcruz|rover: you merged that https://github.com/rdo-infra/ci-config/commit/0d86ab8554bebe2bb088beb5d4d714272c705ed8 but we still dont have containers in quay.io @ https://quay.io/organization/tripleozedcentos9 any idea? is there some cron job that executes this which needs update? something else we missed? 12:01
arxcruz|rovermarios checking 12:01
mariosarxcruz|rover: can you check if you get some time otherwise tomorrow when your rover shift finishes please 12:01
mariosarxcruz|rover: thanks, if you have rover stuff can wait until tomorrow 12:02
mariosarxcruz|rover: i was wondering if this https://github.com/rdo-infra/ci-config/blob/master/ci-scripts/copy-containers/copy-rdo.sh was missing zed but that one is to copy *to* rdo registry which isn't what we want12:02
rlandyarxcruz|rover: yeah - just looks like a lot is running now - you can kill some lines if you need to get something through12:05
arxcruz|rovermarios seems to have an error in the toolbox12:07
arxcruz|roverERRO[0002] Failed to copy container image:  Failed to get policy context: invalid policy in "/etc/containers/policy.json": Unknown key "keyPaths"12:07
chandankumararxcruz|rover: downgrade container-commons package12:08
arxcruz|roverchandankumar ty12:08
chandankumararxcruz|rover: https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-9/promotion-testing-hash-master.yml#L19012:09
*** amoralej is now known as amoralej|lunch12:14
mariosthanks arxcruz|rover chandankumar  - so we need it in zed release file too then ^^ arxcruz|rover you taking care ? 12:16
* marios coffee refill brb12:17
rlandydviroel: pls see #vexxhost - thanks12:19
chandankumarmarios: it is already there https://opendev.org/openstack/tripleo-quickstart/src/branch/master/config/release/tripleo-ci/CentOS-9/zed.yml#L22112:20
arxcruz|rovermarios this is in the toolbox that copy the contaienrs 12:23
arxcruz|rovernothing related to zed itself 12:23
marioschandankumar: arxcruz|rover: ah then what are we missing... :/ so we need to downgrade on the box itself then12:33
arxcruz|rovermarios yes, working on it, need to lock in the version 12:33
mariosok thank you arxcruz|rover chandankumar 12:33
dviroelrlandy: ack12:34
soniya29rlandy, ack12:34
*** dasm|off is now known as dasm13:00
dasmo/13:00
dviroelpojadhav: rlandy: connectivity issues from my side, I my join late, or not join the meeting (depends on connection quality)13:01
pojadhavrcastillo, chandankumar : scrum13:01
*** amoralej|lunch is now known as amoralej13:08
dpawlikchandankumar: does the job work on vexxhost?13:14
dpawlikchandankumar: it would help to debug :)13:14
chandankumardpawlik: it will run on vexxhost13:16
chandankumarlet me update the tp13:16
dpawlikchandankumar: and another question: does the tests earlier pass or never tested earlier?13:17
dpawlikchandankumar: thanks13:17
chandankumardpawlik: each tempest tests failures are different13:20
chandankumarwhen this job was running on vexxhost, it was quite stable13:21
dpawlikchandankumar: same test? Do you know when it was last time?13:22
dpawlikWould be good to test it same time in both places13:22
pojadhavdviroel, ack i thought you were on holiday (as per rhos calendar)13:49
dviroelpojadhav: only yesterday, did I added it right?13:52
pojadhavdviroel, it is showing today as well, confusing :)14:01
dviroelpojadhav: humm, might be the TZ difference14:05
pojadhavdviroel, yep14:14
*** ysandeep is now known as ysandeep|dinner14:19
rlandyarxcruz|rover: https://review.rdoproject.org/r/c/rdo-infra/ci-config/+/45648 Add alternative jobs to wallaby c814:28
rlandythat should help you out with c814:28
arxcruz|roveroh, cool 14:29
jm1arxcruz|rover, rlandy: i am out for today. train job and corona..eh..corosync jobs are still running14:29
arxcruz|roveri did not knew we could have this alternative_criteria 14:29
rlandyjm1: ok - will keep watching14:30
rlandyarxcruz|rover: it's been helping promotions all week14:30
rlandyarxcruz|rover: in master, wallaby, train14:30
arxcruz|roveri though it was just coment one and add the internal 14:30
arxcruz|roverdid not knew the alternative_criteria key 14:30
rlandyarxcruz|rover: pls see https://issues.redhat.com/secure/RapidBoard.jspa?rapidView=11751&projectKey=TRIPLEOCI&view=detail&selectedIssue=TRIPLEOCI-123514:31
jm1arxcruz|rover: rlandy really helped us this week with promotions even while she was absent ;) this alternative criteria patch is awesome ^^14:31
* jm1 out for today, have a nice evening #oooq14:32
rlandygood night14:32
tuxcrafterI can not get my openstack.cloud to work .... I been trying diffrent python packages and versions but no luck... 14:36
tuxcrafterpython3-openstacksdk 0.17.2-2 (debian) ansible-galaxy collection install openstack.cloud:1.9.0 >  "msg": "openstacksdk is required for this module"14:37
tuxcrafter>>> import openstack >>> import importlib >>> sdk = importlib.import_module('openstack')14:42
tuxcrafterall these python3 commands work fine from the cli14:42
*** dviroel_ is now known as dviroel14:55
chandankumarsee ya!14:57
rlandyrcastillo: hey - can you help tuxcrafter out?14:59
rlandychandankumar: have a good night14:59
tuxcrafterhtis is the full traceback https://paste.debian.net/plain/1256925 15:02
rcastillotuxcrafter do you have openstacksdk installed in the host where the playbooks actually run? (if you're not running on localhost)15:06
tuxcrafterrcastillo: i am running on my localhost15:10
*** ysandeep|dinner is now known as ysandeep15:18
rcastillohmm, I've never run into that. what ansible version are you using?15:20
rcastillotuxcrafter: ^15:20
*** ysandeep is now known as ysandeep|out15:33
frenzy_fridayfolks pls add to your review list https://review.opendev.org/c/openstack/tripleo-ci-health-queries/+/859297 - to track mysql failure and the vexx hypervisor thing15:34
* ysandeep|out out, Have a great rest of your day folks o/ see you tomorrow15:37
*** marios is now known as marios|out15:38
tuxcrafterI am getting somewhere "msg": "To utilize this module, the installed version of the openstacksdk library MUST be >=0.36."15:39
tuxcrafterI seem to be using python2....15:39
rcastillomay be that15:41
rcastilloand for 1.9.0 we need sdk 0.36 at least. You might be able to get that off pip15:41
tuxcrafterI went to 1.8.0 15:42
tuxcrafterkeystoneauth1.exceptions.auth_plugins.MissingRequiredOptions: Auth plugin requires parameters which were not given: auth_url15:42
tuxcrafterI need to set the environment with the info from the openstack rc 15:45
tuxcrafterI think I will be able to figure that out15:45
* dviroel lunch15:49
rcastillotuxcrafter nice15:53
*** amoralej is now known as amoralej|off16:16
*** jpena is now known as jpena|off16:34
tuxcrafterrcastillo: I got it working!16:39
tuxcrafterrcastillo: thank you16:39
rlandyarxcruz|rover: bhagyashris: frenzy_friday: fyi ... https://redhat.service-now.com/help?id=rh_ticket&is_new_order=true&table=sc_request&sys_id=f465d7c997261950f5d07c7e6253af8516:43
rlandypushing to docker-registry.upshift.redhat.com is timing out16:44
rlandywallaby c8 is promoting16:50
rlandybhagyashris: ^^ fyi16:54
* dviroel back17:13
rlandydviroel: rcastillo: review time?19:01
jm1rlandy: https://review.opendev.org/c/openstack/octavia/+/861145 got merged, so we can now promote octavia component :)19:44
* jm1 fading into background again19:44
rlandyjm1: awesome19:45
rlandywill take care of that19:45
* dviroel biab - ice cream :)19:55
*** dviroel is now known as dviroel|biab19:55
rlandydasm: hey - need your help on rr cockpit20:00
rlandynvm - worked it out20:05
*** dviroel|biab is now known as dviroel20:51
*** dviroel is now known as dviroel|afk21:33
rlandyjm1[m]: luca's patches are in!!21:56
rlandytrain is promoting again22:12
*** rlandy is now known as rlandy|bbl22:12
dasmheh. missed a ping by 2.5h :)22:25
dasmgood you figured that out rlandy|bbl 22:25
*** dasm is now known as dasm|off23:52
dasm|offo/23:52
*** rlandy|bbl is now known as rlandy23:55
*** rlandy is now known as rlandy|out23:59

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!