| *** pojadhav is now known as pojadhav|afk | 05:58 | |
| akahat|rover | soniya29|ruck, please take a look when you are free: https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-train | 06:30 |
|---|---|---|
| *** pojadhav|afk is now known as pojadhav | 06:36 | |
| *** ykarel is now known as ykarel|lunch | 09:21 | |
| soniya29|ruck | akahat|rover, sure | 09:58 |
| *** dviroel|afk is now known as dviroel | 10:22 | |
| arxcruz | marios: https://review.opendev.org/c/openstack/tripleo-ci/+/816991 i think what you told makes sense, i forgot about the lookup('env') | 10:25 |
| marios | arxcruz: cool. i was staring at it this mornign and just thinking there *must* be some way.. i didn't test it but i think it should work man. sorry to go on about it it is just one more thing that will be maintenance and then only you and me really know you did it the way you did it etc etc our repos have way too much of that | 10:29 |
| marios | arxcruz: thanks for having another go | 10:29 |
| marios | arxcruz: will you try the test with tht or some other repo ? can be master undercloud upgrade job | 10:30 |
| marios | arxcruz: looks ok from quick look i | 10:30 |
| marios | arxcruz: will review again in my next round | 10:31 |
| arxcruz | marios: but it won't work with a dnm test or am i wrong ? | 10:35 |
| marios | arxcruz: why what will not work /me thinking ? if you depends-on the tripleo-ci patch | 10:37 |
| *** jpena|off is now known as jpena | 10:54 | |
| arxcruz | marios: yeah, i don't think it will work, well, it will work for the upgrade content provider, but not for the undercloud-upgrade | 11:06 |
| arxcruz | isn't this the issue we are facing | 11:06 |
| arxcruz | or you want to test the content-provider ? | 11:06 |
| *** rlandy|out is now known as rlandy|ruck | 11:19 | |
| rlandy|ruck | ysandeep: looking better downstream :) | 11:28 |
| rlandy|ruck | 16.2 promoted | 11:28 |
| rlandy|ruck | did you rerun fs035 on ovb | 11:29 |
| ysandeep | rlandy|ruck, yeah i am tracking component promotions now | 11:29 |
| ysandeep | rlandy|ruck, on 16.2 | 11:29 |
| ysandeep | ? | 11:29 |
| rlandy|ruck | ysandeep: yep on 16.2 | 11:29 |
| rlandy|ruck | otherwise that test is out of criteria | 11:29 |
| ysandeep | rlandy|ruck, we already have a green run on that hash for fs035 - yesterday | 11:30 |
| rlandy|ruck | ok - good | 11:31 |
| rlandy|ruck | ysandeep: need any help on component promotions | 11:31 |
| rlandy|ruck | going to ruck/rover sync now | 11:31 |
| rlandy|ruck | we can pick up the rest | 11:31 |
| rlandy|ruck | so you can carry on | 11:31 |
| ysandeep | rlandy|ruck, nah all good, left notes on tripleo-ci chat channel on what's the status now | 11:31 |
| ysandeep | rlandy|ruck, fyi.. envc have a hardware failure, i have opened a ticket with lab team | 11:32 |
| *** pojadhav is now known as pojadhav|afk | 11:32 | |
| rlandy|ruck | ysandeep: ack - I see thanks | 11:32 |
| rlandy|ruck | akahat|rover: soniya29|ruck: hi- let's ruck/rover sync | 11:33 |
| rlandy|ruck | https://meet.google.com/njw-nrxe-gxk | 11:33 |
| rlandy|ruck | akahat|rover: soniya29|ruck: ^^ | 11:34 |
| rlandy|ruck | arxcruz: are you on the cockpit box? | 11:42 |
| arxcruz | rlandy|ruck: no | 11:42 |
| arxcruz | rlandy|ruck: but ananya left a tmux open | 11:43 |
| arxcruz | don't touch it :D | 11:43 |
| rlandy|ruck | arxcruz: ugh | 11:43 |
| rlandy|ruck | why? | 11:43 |
| rlandy|ruck | do you know what happened here? | 11:43 |
| arxcruz | 12:58:07 <frenzy_friday>I will be on PTO from monday till dec 3. ci health rdo and opensearch both running on the same health server. There is a tmux already with the logs (window 0 and 1) - in case people start using come looking for health | 11:43 |
| rlandy|ruck | arxcruz: can you join https://meet.google.com/njw-nrxe-gxk | 11:44 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1950916 | 11:50 |
| rlandy|ruck | https://review.opendev.org/c/openstack/tripleo-quickstart-extras/+/818229/1/roles/standalone/templates/standalone_config.yaml.j2#b49 | 11:52 |
| rlandy|ruck | akahat|rover: ^^ | 11:52 |
| *** ykarel|lunch is now known as ykarel | 12:05 | |
| rlandy|ruck | AND "result" !='RETRY' AND "result" !='RETRY_LIMIT' | 12:12 |
| *** pojadhav|afk is now known as pojadhav | 12:17 | |
| ysandeep | #Folks retrospective mtg in 2 mins | 12:28 |
| ysandeep | rlandy|ruck, soniya29|ruck akahat|rover rcastillo arxcruz ^^ | 12:31 |
| ysandeep | rlandy|ruck, soniya29|ruck akahat|rover rcastillo arxcruz retro time | 12:33 |
| ysandeep | rlandy|ruck, do you want us to start without you? | 12:34 |
| rlandy|ruck | ysandeep; sorry - joining | 12:34 |
| pojadhav | arxcruz, https://meet.google.com/kkp-bejs-vvo | 12:37 |
| pojadhav | arxcruz, https://miro.com/app/board/o9J_lhP-0mY=/ | 12:37 |
| arxcruz | pojadhav: gracias | 12:37 |
| *** ykarel is now known as ykarel|afk | 12:41 | |
| dviroel | marios: sshnaidm extra doc for libvirt driver https://github.com/bogdando/oooq-warp/blob/master/docs/CI-reproducer.md | 13:16 |
| sshnaidm|afk | dviroel, nice, would be great to have it included in our reproducer.. | 13:17 |
| dviroel | yes, for sure | 13:19 |
| *** sshnaidm|afk is now known as sshnaidm | 13:24 | |
| marios | dviroel: thanks | 13:26 |
| dviroel | wrt 'TODO' in code, it is easier to search for work that you left behind :P | 13:28 |
| dviroel | e.g https://codesearch.opendev.org/?q=TODO%5C(dviroel%5C)&i=nope&literal=nope&files=&excludeFiles=&repos= | 13:28 |
| marios | +1 | 13:28 |
| ysandeep | arxcruz, sshnaidm, rlandy, marios, ysandeep, bhagyashris, svyas, soniya29, pojadhav, akahat, chandankumar, frenzy_friday, anbanerj, dviroel tripleo-ci-community call in 1 min | 13:29 |
| ysandeep | ceph squad will join us today, if you have other items please add in agenda: https://hackmd.io/MMg4WDbYSqOQUhU2Kj8zNg?both | 13:29 |
| soniya29|ruck | rlandy|ruck, i will be joining meeting in 5 mins | 13:58 |
| *** ykarel|afk is now known as ykarel | 14:02 | |
| soniya29|ruck | rlandy|ruck, i am in | 14:02 |
| rlandy|ruck | soniya29|ruck: few minutes | 14:03 |
| soniya29|ruck | rlandy|ruck, okay, no problem | 14:03 |
| rlandy|ruck | just finishing up on community call | 14:04 |
| rlandy|ruck | soniya29|ruck: ^^ | 14:04 |
| *** amoralej|off is now known as amoralej | 14:08 | |
| jfrancoa | hey, bringing the question here: what does DISK_FULL mean in https://review.opendev.org/c/openstack/tripleo-heat-templates/+/817718 ? logs were not collected either, so it's hard to know what happened | 14:10 |
| ysandeep | interesting.. so one the job run gave DISK_FULL - tripleo-ci-centos-8-standalone-upgrade-victoria https://zuul.opendev.org/t/openstack/build/fefdad3293cb40ab9766b696472e07d3 : DISK_FULL in 2h 43m 42s | 14:14 |
| ysandeep | I haven't seen that myself , may be others here have seen that, if not i think worth checking with infra team what does that mean. | 14:14 |
| ysandeep | marios, ykarel sshnaidm ^^ if you have seen this | 14:15 |
| marios | ysandeep: not seen it but sounds like infra issue? | 14:16 |
| ysandeep | jfrancoa, is that consistent with this job? | 14:16 |
| jfrancoa | ysandeep: so, this has been the first time I've got this far | 14:16 |
| jfrancoa | I could relaunch | 14:17 |
| ysandeep | jfrancoa: yes please recheck , lets see if this is consistent | 14:17 |
| jfrancoa | ysandeep: if you suggest it, I'll just recheck and see if it happens again | 14:17 |
| jfrancoa | ysandeep: ack, thanks | 14:17 |
| jfrancoa | I'll keep you posted with the update | 14:17 |
| jfrancoa | ysandeep: thanks! | 14:17 |
| * ysandeep will try to check documenations and see if i get somethings about DISK_FULL | 14:17 | |
| rlandy|ruck | soniya29|ruck:periodic-tripleo-ci-centos-8-standalone-full-tempest-api-master | 14:18 |
| rlandy|ruck | jfrancoa: new zuul status | 14:18 |
| rlandy|ruck | added since last update | 14:18 |
| rlandy|ruck | we hit it occasionally | 14:18 |
| rlandy|ruck | arxcruz: ^^ someone is hitting that status | 14:18 |
| rlandy|ruck | soniya29|ruck: periodic-tripleo-ci-centos-8-ovb-3ctlr_1comp-featureset035-master | 14:19 |
| rlandy|ruck | soniya29|ruck: python3 roles/rrcockpit/files/telegraf_py3/ruck_rover.py --release master | 14:20 |
| ykarel | ysandeep, i have seen that long back, and i think fixed that too for the issue at that time | 14:20 |
| ykarel | let me check can find the patch | 14:20 |
| ykarel | was related to log collectin iirc | 14:20 |
| ykarel | ysandeep, https://review.opendev.org/c/openstack/ansible-role-collect-logs/+/662767 was the fix that time | 14:26 |
| ysandeep | ykarel: thanks! | 14:29 |
| ykarel | ysandeep, upstream i think executor disk limit per job is 5GB, so somehow that job is consuming that much and thus fails | 14:33 |
| ysandeep | ykarel, I have requested jfrancoa to recheck, if we get capture of logs(hopefully not cross 5gb on each run).. we can check what's taking space there. | 14:35 |
| ysandeep | may be we can add a dnm patch to add "du -csh" stats before log collection. | 14:35 |
| ykarel | yes would be good to root cause it | 14:36 |
| ysandeep | jfrancoa: let see how current run goes, otherwise we go that route ^^ | 14:37 |
| ykarel | i had used https://review.opendev.org/c/openstack/tripleo-ci/+/662481/3/roles/run-test/templates/oooq_common_functions.sh.j2 in past | 14:37 |
| jfrancoa | ykarel: cool, I'll add that then | 14:39 |
| jfrancoa | ykarel: thanks a lot | 14:39 |
| ysandeep | ykarel++ looks good we can restore/ use something similiar. | 14:40 |
| ykarel | https://zuul.opendev.org/t/openstack/builds?result=DISK_FULL | 14:42 |
| ysandeep | jfrancoa, you will need to restore that patch first if you want to add as depends on | 14:43 |
| jfrancoa | ysandeep: ohh right..I didn't realize it was abandoned | 14:52 |
| jfrancoa | restored and rebased | 14:52 |
| *** ysandeep is now known as ysandeep|out | 15:01 | |
| *** dviroel is now known as dviroel|lunch | 15:07 | |
| jbadiapa | rlandy|ruck, I think the https://bugs.launchpad.net/tripleo/+bug/1950383 is not exactly the same as https://bugs.launchpad.net/tripleo/+bug/1951260 | 15:48 |
| jbadiapa | after second thoughts it seems to me that they are not related | 15:48 |
| rlandy|ruck | jbadiapa: can remove duplicate | 15:50 |
| rlandy|ruck | done | 15:50 |
| jbadiapa | I'm revieweing the bug regarding UEFI | 15:51 |
| jbadiapa | cos the current issue is the introspection is failing and the VMs on libvirt are set as bios, but configure them as uefi does not work either :/ | 15:52 |
| jbadiapa | rlandy|ruck, thx | 15:53 |
| rlandy|ruck | jbadiapa: trying setting to bios | 15:54 |
| *** ykarel is now known as ykarel|away | 15:55 | |
| jbadiapa | rlandy|ruck: If we use bios the VMs dont start as the overcloud-full image is uefi | 15:56 |
| *** ysandeep|out is now known as ysandeep | 16:05 | |
| ysandeep | Cedric also have to made changes in tripleo-labs to move OC nodes to UEFI | 16:05 |
| ysandeep | https://github.com/cjeanner/tripleo-lab/commit/426b9eff44205d7810439684ceaa219c2d966286 | 16:05 |
| jbadiapa | ysandeep, I've already tried that https://review.opendev.org/c/openstack/tripleo-quickstart/+/818766 | 16:06 |
| jbadiapa | but it didnt work on my lab | 16:06 |
| *** dviroel|lunch is now known as dviroel | 16:08 | |
| Tengu | jbadiapa: we can discuss a bit once I back, next Monday, if you want | 16:08 |
| ysandeep | jbadiapa, ack, I see you already logged the bugs, rlandy|ruck if you are joining hardprov bug scrub today, May be we can discuss these issues there. | 16:08 |
| jbadiapa | Tengu, sure | 16:08 |
| jbadiapa | Tengu, thx.... | 16:09 |
| Tengu | took me a hell of a time (with Harald) to get it working. secure-boot is a terrible thing. | 16:09 |
| ysandeep | rlandy|ruck, imho.. we need to request sbaker to once try the tripleo-quickstart himself and make sure manual deployment are working expected as per his plans. | 16:09 |
| ysandeep | i have triggered check-ci-minimal on https://review.opendev.org/c/openstack/tripleo-quickstart/+/818766 to check where its failing. | 16:10 |
| rlandy|ruck | ack - bringing this all to the scrub | 16:11 |
| ysandeep | jbadiapa, recent hardprov changes around whole_disk_images and some other changes have broke oooq manual deployments, we are trying to better align with them to solve these issues. | 16:12 |
| ysandeep | rlandy|ruck: thanks! | 16:12 |
| jbadiapa | ysandeep, I know... I experienced those changes. | 16:13 |
| jbadiapa | the whole_disk_images but the new movement was to set the uefi images | 16:14 |
| rlandy|ruck | yep | 16:14 |
| rlandy|ruck | we had one pass | 16:14 |
| rlandy|ruck | but now we have the second issue | 16:14 |
| rlandy|ruck | https://bugs.launchpad.net/tripleo/+bug/1951752 | 16:15 |
| rlandy|ruck | two patches to try set back to bios | 16:15 |
| rlandy|ruck | testing that now | 16:15 |
| rlandy|ruck | merged a few days ago | 16:15 |
| rlandy|ruck | it's a never ending joy | 16:15 |
| rlandy|ruck | Tengu: what happened to PTO???? | 16:15 |
| dviroel | marios: check your last testproject run, $HOME should be /home/zuul now, the default user is zuul now | 16:15 |
| marios | dviroel: ah | 16:17 |
| ysandeep | rlandy|ruck, i think Tengu miss us from time to time, so he comes online to see what we are doing :D | 16:17 |
| rlandy|ruck | ysandeep: of course - who wouldn't miss us??? | 16:18 |
| marios | thanks dviroel :) | 16:18 |
| dviroel | ;) | 16:18 |
| *** ysandeep is now known as ysandeep|out | 16:27 | |
| marios | sshnaidm: dviroel: rlandy|ruck: i bumped the zuul-discuss thread there fyi http://lists.zuul-ci.org/pipermail/zuul-discuss/2021-November/001752.html | 16:48 |
| rlandy|ruck | looking | 16:49 |
| marios | dviroel: i held the node again today and confirmed that *without* installing anything explicitly (just the galaxy install) or setting any paths i was able to run th emodule on localhost without problems so it really must be a zuul limitation http://pastebin.test.redhat.com/1010415 | 16:49 |
| sshnaidm | marios, in order to use collections ansible required them to be installed on the controller, installing them on hosts won't help.. | 16:50 |
| sshnaidm | marios, to use collections it should be possible to install them on zuul executor | 16:50 |
| rlandy|ruck | localhost? | 16:50 |
| marios | dviroel: if you see the latest debug https://logserver.rdoproject.org/17/36817/3/check/tripleo-stream9-development-buildimage-overcloud-hardened-uefi-full-master/5c1daf9/job-output.txt it can run it after manual install but then it fails on the include-role task | 16:50 |
| marios | sshnaidm: ok i was going to reach out to you about that tomorrow | 16:51 |
| marios | sshnaidm: you mean, if add a pre which targets the zuul.executor (but in this case, it is only one node anyway?) | 16:51 |
| marios | sshnaidm: and do the ansible-galaxy install there it might work? | 16:51 |
| marios | sshnaidm: but if you see my latest patch, i actually have an explicit install like https://review.opendev.org/c/openstack/tripleo-ci/+/818222/30/roles/tripleo-build-jobs-repos/tasks/main.yaml | 16:52 |
| sshnaidm | marios, I mean the installation should be on zuul executor machine in infra, which triggers and runs all jobs everywhere, it's not under our control | 16:52 |
| marios | sshnaidm: i see ... :/ | 16:52 |
| marios | sshnaidm: then the collection will be available to all zuul you mean | 16:52 |
| marios | sshnaidm: hmm... is that what you are doing with the collection patches i saw something related | 16:53 |
| marios | sshnaidm: like adding them to infra zuul collections? | 16:53 |
| sshnaidm | marios, and it should be "preinstalled" on this zuul executor machine, because ansible can't detect the new collections in the same run, need to respawn it probably | 16:53 |
| sshnaidm | marios, it works well for "nested" ansible, like in my patches | 16:53 |
| marios | sshnaidm: right but here it isn't nested | 16:54 |
| marios | sshnaidm: it is native this is the problem no quickstart | 16:54 |
| sshnaidm | there the "ansible controller" is the host we have | 16:54 |
| marios | cos for nested we are already using that collection | 16:54 |
| marios | sshnaidm: can i catch you tomorrow morning mate | 16:54 |
| sshnaidm | marios, yep, in native collections it won't work | 16:54 |
| marios | sshnaidm: and we can discuss if you have time | 16:54 |
| sshnaidm | marios, sure | 16:54 |
| sshnaidm | ping me | 16:54 |
| marios | sshnaidm: yeah will do but i am confused ... cos 18:54 < sshnaidm> marios, yep, in native collections it won't work | 16:55 |
| marios | sshnaidm: you mean it wont work even if i have it on the zuul executor? | 16:55 |
| marios | of if you have it on the executor then it should work natively | 16:55 |
| sshnaidm | it will work if you have it on executor, but we won't have it on executor, because no one will give such access :) | 16:55 |
| marios | sshnaidm: k thanks | 16:55 |
| sshnaidm | and think that you need to change this collection | 16:56 |
| sshnaidm | executor is just one host, which version of collection to have there | 16:56 |
| sshnaidm | and if you change a collection there, it affects all jobs in this time | 16:56 |
| sshnaidm | so even if we have such access, it won't help much | 16:57 |
| marios | sshnaidm: so there is still something missing on the zuul side right i mean for 'native zuul support for galaxy collections' | 16:57 |
| sshnaidm | marios, yes | 16:57 |
| dviroel | marios: ack, i was taking a look at the latest run | 16:58 |
| marios | sshnaidm: like i'm wondering if my post was not needed but http://lists.zuul-ci.org/pipermail/zuul-discuss/2021-November/001752.html sounds like it is still not lear | 16:58 |
| marios | k thanks sshnaidm lets talk more tomorrow will ping you whenever you have time | 16:58 |
| sshnaidm | marios, yeah, just probably need a clarification of what we seek for exactly | 16:58 |
| marios | dviroel: ack am getting out in a few mins | 16:58 |
| sshnaidm | marios, k | 16:58 |
| dviroel | rlandy|ruck: hey, when you have some time, I updated compose-pinning hackmd with a list of tasks | 16:59 |
| dviroel | rlandy|ruck: https://hackmd.io/qV5mC2FoQX6N0iFt-f8p4A?view#Tasks | 17:00 |
| dviroel | this should help us to create tasks in the board | 17:00 |
| rlandy|ruck | thanks | 17:02 |
| jbadiapa | rlandy|ruck, ysandeep: regarding the issue on the manual deployments. https://paste.opendev.org/show/811245/ | 17:04 |
| *** marios is now known as marios|out | 17:06 | |
| *** jbadiapa is now known as jbadiapa|out | 17:07 | |
| *** jpena is now known as jpena|off | 17:33 | |
| rlandy|ruck | lunch brb | 17:47 |
| * dviroel needs a coffee | 18:18 | |
| *** amoralej is now known as amoralej|off | 19:06 | |
| *** dviroel is now known as dviroel|afk | 19:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!