Friday, 2022-01-07

« Thursday, 2022-01-06 Index Sunday, 2022-01-09 »
rlandy|ruckrcastillo|rover: ok - let's try another testproject00:05
rcastillo|roverfingers crossed00:07
rlandy|ruckrcastillo|rover: here's hoping: https://review.rdoproject.org/r/c/testproject/+/3625600:11
rlandy|ruckif vexxhost doesn't die on us00:11
rlandy|ruckrcastillo|rover: late for  you and you're not well00:11
rlandy|ruckok if you want to knock off00:11
rlandy|ruckand we'll pick this up tomorrow00:12
rcastillo|roverfeeling better actually00:12
rcastillo|roverplus I'm trying to get the full ruck rover experience here00:13
rcastillo|rover:)00:13
rlandy|ruckrcastillo|rover: you can see why everyone LOVES this00:14
rlandy|ruckif the dependencies don't break00:15
rlandy|ruckthe infra does00:15
rlandy|ruckwe call it 'job security'00:15
rlandy|ruck2022-01-06 23:53:31.214921 | TASK [oooci-build-images : Include remove_redfish.yaml]00:16
rlandy|ruck2022-01-06 23:53:41.750349 | primary | ok00:16
rlandy|ruck2022-01-06 23:53:41.783930 | primary | included: /var/lib/zuul/builds/d67e217576534612826a35ec0acf9eb2/untrusted/project_3/opendev.org/openstack/tripleo-ci/roles/oooci-build-images/tasks/remove_redfish.yaml00:16
rlandy|ruckha  that worked00:16
rcastillo|roveryeah it's great00:16
rlandy|ruckon https://logserver.rdoproject.org/46/35646/60/check/periodic-tripleo-centos-8-buildimage-overcloud-hardened-uefi-full-master/d67e217/job-output.txt00:16
rlandy|ruckfs002 is running on that testproject00:17
rlandy|ruckand the exclude on the other00:17
rlandy|ruckhttps://review.rdoproject.org/zuul/stream/7e3ebcbbc5084cf997e43ef33932a6c7?logfile=console.log building images atm00:17
rcastillo|roverwe have twice as many odds of it working then00:19
rlandy|ruckrcastillo|rover: your testproject is also running now00:20
rlandy|ruckfs00200:20
rlandy|ruckwe have three horses in this race!!!00:20
rcastillo|roverlol00:21
rcastillo|rovermaybe I can get compose pinning working in c8... ;)00:22
rlandy|ruckrcastillo|rover: we're all about options00:23
rlandy|ruckif the exclude works, it's our simplest option00:25
rcastillo|roveryeah, seems cleaner to just downgrade00:27
rcastillo|roverhow do we tell when to get rid of the exclude though? Monitor new versions and test the revert?00:28
rcastillo|roveror is there an easier way?00:28
rlandy|ruckthat's basically it00:28
rlandy|ruckwe leave the card open00:28
rlandy|ruckon CIX board in external dependencies00:28
rcastillo|roverI see00:28
rlandy|ruckuntil they close the related bugzilla00:28
rlandy|ruckwe have a launchpad bug for our failure00:29
rlandy|ruckand a bugzilla out to kernel00:29
rlandy|ruckalan usually wathces out for those updates00:29
rcastillo|rover["virt-customize: error: libguestfs error: overcloud-hardened-uefi-full: No ", "such file or directory"00:56
rcastillo|roverno idea why it complains here but not in the buildimage job00:57
rcastillo|roverate the .qcow2 extension for some reason. Oh well00:57
rlandy|ruckhmm ... didn;t see that error01:35
rlandy|ruckthe hash rolled01:35
rlandy|ruckhad to recheck01:35
rcastillo|roverwhat makes the hash change?01:38
rcastillo|rover2022-01-07 02:02:09.724347 | primary | rsync: mkstemp "/var/www/html/images/centos8/master/rdo_trunk/a697b14b948e6954637616410e295450/.overcloud-hardened-uefi-full.qcow2.c641P3" failed: No space left on device (28)02:13
rcastillo|roveralright I think I'm done for the day. I'll leave a note for arx and pick this up tomorrow02:14
*** ysandeep|out is now known as ysandeep02:15
rcastillo|roverthanks for the help rlandy|ruck. See you tomorrow02:15
*** rcastillo|rover is now known as rcastillo02:15
soniya29marios, arxcruz, chandankumar, please add this patch to your review list - https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81565404:29
*** rlandy|ruck is now known as rlandy|out05:39
*** ysandeep is now known as ysandeep|afk06:40
*** amoralej|off is now known as amoralej08:09
*** ysandeep|afk is now known as ysandeep09:38
* bhagyashris stepping out for bit11:03
*** rlandy|out is now known as rlandy|ruck11:15
rlandy|ruckarxcruz|ruck: hello11:18
arxcruz|ruckrlandy|ruck: hello 11:18
rlandy|ruckarxcruz|ruck: any progress on the OVB failure?11:18
rlandy|ruckarxcruz|ruck: let's meet11:19
arxcruz|ruckok 11:20
rlandy|ruckamoralej: hi - do we have a contact on centos stream team? https://bugzilla.redhat.com/show_bug.cgi?id=203099311:20
rlandy|ruckamoralej: ^^ two new reports on that bug11:20
amoralejlemme check11:22
rlandy|ruckarxcruz|ruck: meet.google.com/vzm-nrah-qqf11:22
*** dviroel|afk is now known as dviroel11:25
amoralejrlandy|ruck, i guess the right contacts are the ones in the bz as they are the maintainers for that package11:27
amoraleji see you are applying a workaround https://review.opendev.org/c/openstack/tripleo-ci/+/82369411:27
amoralejthat's probably the best we can do at this point11:27
amoralejif that works11:27
rlandy|ruckamoralej: we're trying workarounds11:27
rlandy|rucknone of which are working at the moment11:28
rlandy|ruckamoralej; we need to either get rid of  fwupd-redfish.conf 11:30
rlandy|ruckor downgrade fwupd11:30
rlandy|ruckneither of which we are successful in now11:30
arxcruz|ruckrlandy|ruck: https://review.rdoproject.org/r/c/testproject/+/36256/11:33
amoralejrlandy|ruck, is there any reproducer i may take a look into?11:33
rlandy|ruckamoralej: yep - tons - all of OVB is failing on c811:33
amoralejsomeone i could ssh to ?11:33
rlandy|ruckyeah ....11:34
amoralejwe used promoted images on ovb?11:34
amoraleji mean, are jobs in check/gate broken?11:34
rlandy|ruckhttps://review.rdoproject.org/zuul/stream/d27857d3bdb04e57a7fc9b005b47ace6?logfile=console.log11:34
amoralejor only promotion pipelines?11:34
rlandy|ruck^^ running right no11:34
rlandy|ruckamoralej: in check also11:34
rlandy|ruckwe have OVB jobs that kick on check11:35
arxcruz|ruckhttps://trello.com/c/p6xDz8CS/2273-cixlp1956083tripleociproa-victoria-error-containers-which-failed-to-be-created-by-podmancontainer-module-novawaitforcomputeservi12:00
arxcruz|ruckrlandy|ruck: https://trello.com/c/p6xDz8CS/2273-cixlp1956083tripleociproa-victoria-error-containers-which-failed-to-be-created-by-podmancontainer-module-novawaitforcomputeservi12:00
rlandy|ruckarxcruz|ruck:  https://review.opendev.org/c/openstack/tripleo-ci/+/82373612:22
amoralejrlandy|ruck, in my tests removing fwupd-redfish.conf worked at least to start the systemd service fine12:26
rlandy|ruckamoralej: great news12:29
rlandy|ruckarxcruz|ruck and I debugged the role we were using12:29
amoralejthere may be an easier one, i'll add a note12:29
rlandy|ruckwe know now why it failed to take12:30
rlandy|ruckso maybe the next run will work 12:30
rlandy|ruckamoralej: note to the LP bug? thank you12:30
amoralejrlandy|ruck, yes, in https://bugs.launchpad.net/tripleo/+bug/195644112:31
arxcruz|ruckrlandy|ruck: updated, at the end we need to run twice because of the md5sum, but i change the variable name to not ahve conflicts with the already defined one 12:35
arxcruz|ruckand recheck your testproject 12:35
rlandy|ruckarxcruz|ruck: thank you12:35
rlandy|ruckpls make sure it works on both fs002 and image builds12:36
soniya29rlandy|ruck, ysandeep, please review this patch whenever you have time - https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/81565413:00
soniya29arxcruz|ruck, can you revote on the above patch? 13:00
ysandeepsoniya29, ack13:01
*** rcastillo is now known as rcastillo|rover13:28
rcastillo|rovero/13:28
chandankumarrlandy|ruck: arxcruz|ruck proposed another fix based on amoralej suggestion https://review.opendev.org/c/openstack/tripleo-ansible/+/82379813:37
amoralejchandankumar, that's executed before doing the deployment?13:39
amoraleji mean, before we are hitting the issue?13:39
chandankumaramoralej: https://opendev.org/openstack/tripleo-ansible/src/branch/master/tripleo_ansible/roles/tripleo_kernel/tasks/main.yml#L5613:39
chandankumarjust before this whether it is breaking13:39
chandankumar*where13:39
amoralejah, then it should be fine i think13:40
chandankumarproposing fs01 testproject13:40
*** amoralej is now known as amoralej|lunch13:40
dasmo/13:40
dasmrlandy|ruck: fyi, i'm around.13:41
rlandy|ruckdasm: hey - on meeting - will chat in a few13:41
dasmok, no worries13:41
rlandy|ruckchandankumar: you back?13:52
rlandy|ruckchandankumar: dasm wanted to start on the wallaby c9 work13:53
chandankumarrlandy|ruck: just lurking here for some time13:53
chandankumarrlandy|ruck: cool, c8s wallaby containers are available now13:53
dasmchandankumar: indeed. i'd like to know where i can start13:54
rlandy|ruckchandankumar: we are going with ykaerl's suggestion13:54
rlandy|ruckchandankumar: amoralej|lunch confirmed it works13:54
rlandy|ruckarxcruz|ruck has the latest patch13:54
chandankumardasm: you can start with standalone job13:54
rlandy|ruckdasm: ^^ ack 13:54
rlandy|ruckwas going to say if you have containers, you can start with standalone13:55
rlandy|ruckrcastillo|rover also had images going13:55
rlandy|ruckchandankumar: so may we can also start a line13:55
dasmchandankumar: ok. any pointers where to look for it? is it opendev or rdoproject?13:55
chandankumardasm: here is the job definition of standalone master https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-jobs-cs9-master.yaml#L2213:55
dasmperfect13:56
rlandy|ruckdasm: we can meet and go through some pointers on this and how to set up a mini line in afternoon13:56
chandankumarand for c8s version fo wallaby standalone is here https://github.com/rdo-infra/rdo-jobs/blob/master/zuul.d/integration-jobs-wallaby.yaml#L5513:56
dasmrlandy|ruck: sounds good13:56
rlandy|ruckchandankumar: thank you13:56
chandankumaryou can create similar job for cs913:56
chandankumarand then send a testprojext13:56
dasmlemme look into that. i'm definitely gonna have some questions :)13:57
rlandy|ruckarxcruz|ruck: I am in a few meetings ... pls sync with rcastillo|rover 13:57
rlandy|ruckabout where you are13:58
rlandy|ruckwith OVB failures13:58
arxcruz|ruckrlandy|ruck: ok13:59
arxcruz|ruckrcastillo|rover: when you're free 13:59
rlandy|ruckrcastillo|rover: arxcruz|ruck: will sync with you guys when out of meetings13:59
chandankumararxcruz|ruck: another fix if file removal does not work https://review.opendev.org/c/openstack/tripleo-ansible/+/82379813:59
arxcruz|ruckchandankumar: i saw it, and i think it's a better approach tbh 14:00
chandankumarone problem is that we need to do multiple backport14:00
chandankumaranother question do we need to adjust this conditional for rhel also14:00
chandankumarsince it is found in RHEL-8.6 i think14:00
ysandeeprlandy|ruck, 1:114:00
rlandy|ruckarxcruz|ruck: chandankumar: if the file removal works, let's get that in14:01
arxcruz|ruckchandankumar: then it is not a better approach lol 14:01
rlandy|ruckand then try ^^14:01
chandankumarhehe14:01
rlandy|ruckysandeep: joining14:01
rlandy|ruckarxcruz|ruck: rcastillo|rover: good reminder though14:01
chandankumararxcruz|ruck: rlandy|ruck we need one functionality in image build to run any command on image 14:01
rlandy|ruckwe need to tetsproject the downstream on this as well14:01
chandankumarfor future14:01
rlandy|ruckchandankumar++14:02
chandankumarwe might hit similar situation in future14:02
rlandy|ruckafter I tried the exclude 14:02
rlandy|ruckI thought the same14:02
rlandy|ruckI could not edit a repo already on the image14:02
ysandeepchandankumar, rlandy|ruck 17  allcontainers are build successfully14:02
rlandy|ruckysandeep: rhel9??14:02
chandankumarysandeep+++14:02
ysandeeprlandy|ruck, yes14:02
rlandy|ruckysandeep+++ you rock!!!!14:02
ysandeepchandankumar, we have a -1 on https://review.opendev.org/c/zuul/zuul-jobs/+/822503 14:03
chandankumarysandeep: I will fix that one14:03
rcastillo|roverarxcruz|ruck: let's sync https://meet.google.com/akc-xvbv-kua14:03
ysandeeprlandy|ruck, https://review.rdoproject.org/r/c/rdo-infra/rdo-infra-playbooks/+/3785414:03
*** akahat_ is now known as akahat14:07
ysandeepchandankumar++ thanks!14:09
chandankumaramoralej|lunch: https://review.rdoproject.org/r/c/rdo-infra/rdo-infra-playbooks/+/37854 please have a look, thanks!14:09
dviroelrcastillo|rover: https://review.rdoproject.org/r/c/testproject/+/36727/36#message-724421b153aa0ec4e1a7a01413327dc866be84e714:14
dviroelrcastillo|rover: with https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/823793 - will run without to double check if is the issue.14:14
*** amoralej|lunch is now known as amoralej14:22
*** ysandeep is now known as ysandeep|dinner14:45
rcastillo|roverdviroel: seems like that was the issue14:46
rcastillo|roverdviroel: same bug seems to be affecting c9 content-provider jobs14:46
dviroel:O14:47
rcastillo|roveroops14:47
rlandy|ruckarxcruz|ruck: you have a success on fs00214:57
arxcruz|ruck rlandy|ruck yeah!14:57
rlandy|ruckawesomeness14:57
rcastillo|rover:D14:57
arxcruz|ruckrlandy|ruck: i'll just update the patch to show which image is being used 14:57
arxcruz|ruckbecause it doesn't show in the log14:57
rlandy|ruckarxcruz|ruck: great - once this reports14:58
rlandy|ruckpls get the patch in mergeable format14:58
rlandy|rucktake out the WIP14:58
rlandy|ruckget the commit message to be omproved14:58
rlandy|ruckgive akahat credit14:58
rlandy|ruckfor the fix 14:58
dviroel++14:58
rlandy|ruckrcastillo|rover: arxcruz|ruck: we should probably give this a run in downstream14:59
rlandy|ruckalthough I don't expect issues14:59
*** ysandeep|dinner is now known as ysandeep15:10
*** dviroel is now known as dviroel|lunch15:37
ysandeepchandankumar: fyi.. https://review.opendev.org/c/openstack/tripleo-common/+/823820 15:45
ysandeeprlandy|ruck, I have cleared cix board a bit for some new triage and already solved issues for container build stuff.15:46
* ysandeep out for the day15:49
*** ysandeep is now known as ysandeep|out15:49
rlandy|ruckarxcruz|ruck: where was your last testproject?15:54
rlandy|ruckoh - you updated15:54
chandankumarysandeep|out: thank you sir :-) ++++++++++++16:10
chandankumarsee ya people, Happy weekend :-)16:14
*** dviroel|lunch is now known as dviroel16:33
rlandy|ruckchandankumar: have a great weekend16:38
* dasm submitted first testproject change. now watching zuul16:48
dviroel\o/16:50
dasmc9-wallaby integration line failed: https://review.rdoproject.org/zuul/build/6e08c6aa1335449382cdf93c41d04902 looking through the logs to find the root cause17:48
dasm> "No package atop available."17:50
*** amoralej is now known as amoralej|off17:50
dasm> "No package atop available."17:50
dasm> Install atop from RDO build-deps repo when configured so (CentOS >= 8)17:50
dviroeldasm: this is a common erro, I think, your real issue should be ahead18:11
dasmoh. in that case, looking further for the problem18:12
dasm(i'm downloading centos9 to check that one)18:12
dasmyou're right18:13
dasm2022-01-07 12:14:20.790746 | primary | fatal: [undercloud]: FAILED! => {"ansible_job_id": "34191650658.47370", "changed": false, "cmd": "openstack tripleo deploy  --templates $DEPLOY_TEMPLATES   --yes --output-dir $DEPLOY_OUTPUT_DIR  --stack $DEPLOY_STACK --standalone-role $DEPLOY_STANDALONE_ROLE --timeout $DEPLOY_TIMEOUT_ARG -e18:13
dasm/usr/share/openstack-tripleo-heat-templates/environments/standalone/standalone-tripleo.yaml -e /home/zuul/containers-prepare-parameters.yaml -e /home/zuul/standalone_parameters.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/low-memory-usage.yaml -r $DEPLOY_ROLES_FILE      --deployment-user $DEPLOY_DEPLOYMENT_USER  --local-ip $DEPLOY_LOCAL_IP --control-virtual-ip18:13
dasm$DEPLOY_CONTROL_VIP          >/home/zuul/standalone_deploy.log 2>&1", "delta": "0:14:25.878047", "end": "2022-01-07 12:14:18.067328", "finished": 1, "msg": "non-zero return code", "rc": 1, "start": "2022-01-07 11:59:52.189281", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}18:13
dasmsorry18:13
dviroelyep, go for standalone_deploy.log18:13
dviroelshuld be there18:13
dviroelnp18:13
rlandy|ruckrcastillo|rover: arxcruz|ruck: w'ed the OVB patch18:25
rlandy|ruckrcastillo|rover; how we looking on the rest of the lines?18:25
rlandy|ruckdasm: rcastillo|rover:  going to dig out car - will be back in a bit o chat with you about wallaby c918:26
dasmrlandy|ruck: ack! good luck18:26
rlandy|ruckdviroel: julia w+'ed the c9 patch on didk image builder18:26
rlandy|ruckdviroel: should be unblocked for OVB on c918:27
dviroelawesome18:32
*** rlandy|ruck is now known as rlandy|ruck|biab18:34
* dasm brb18:38
dviroelrcastillo|rover: fyi, wrt compose c9 testing, dlrn is installed via pip, but the version is too old and does not contains centos9.cfg (yes, there is a base config file that is used in our role)18:59
dviroelrcastillo|rover: so, monday, we need to ask jpena to update pip package for us, so we can try with dlrn build using cento919:00
* dasm back19:01
dviroelrcastillo|rover: i can do that, just sharing with you :)19:02
rcastillo|roverdviroel: ack19:02
rcastillo|roverthanks19:02
*** rlandy|ruck|biab is now known as rlandy|ruck19:13
rlandy|ruckdviroel: rcastillo|rover: jpena has moved teams ... you can try amoralej|off or dpawlik 19:14
rlandy|ruckdasm: rcastillo|rover: pls ping when you are ready to discuss wallaby c919:18
* rlandy|ruck out in 2 hours today19:18
dasmrlandy|ruck: i'm here, i can talk now19:25
dviroelrlandy|ruck: ack, will try to reach them19:26
rlandy|ruckdasm: ok - let' start - rcastillo|rover can join us when he is available19:27
dasmk19:27
dasmhttps://meet.google.com/ccg-gxdk-iau19:27
dasmrlandy|ruck: ^19:28
*** dviroel is now known as dviroel|afk19:37
rcastillo|roverrlandy|ruck: dasm: I had to step away for a bit, srry about that20:15
dasmrcastillo|rover: no worries. we ran over steps to build new c9-wallaby pipeline and whom should i talk to in the case of possible issues20:16
rlandy|ruckrcastillo|rover: no problem20:18
rlandy|ruckwe found your image build jobs20:18
rcastillo|roverdasm: thanks for the summary20:19
rlandy|ruckrcastillo|rover: OVB fix is about to merge in gates21:21
rlandy|ruckhoping to see that all clear over the weekend21:21
dasmi'm done for tonight. have a great weekend team!21:21
rlandy|ruckdasm: rcastillo|rover: I'm out - hope you have a good weekend21:21
rlandy|rucksee you monday21:22
dasmyou too!21:22
« Thursday, 2022-01-06 Index Sunday, 2022-01-09 »

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!