| *** rlandy is now known as rlandy|out | 00:19 | |
| *** ysandeep|out is now known as ysandeep | 05:18 | |
| jm1 | o/ | 07:15 |
|---|---|---|
| *** amoralej|off is now known as amoralej | 07:20 | |
| *** jpena|off is now known as jpena | 07:37 | |
| *** ysandeep is now known as ysandeep|lunch | 09:05 | |
| jm1 | silence.. 🤨 | 09:23 |
| marios | o/ tripleo-ci please add to your reviews whn you have time https://review.opendev.org/c/openstack/tripleo-ci/+/851516/ & https://review.opendev.org/c/openstack/python-tripleoclient/+/851384 (testing info at https://bugs.launchpad.net/tripleo/+bug/1983004/comments/14) | 09:31 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 09:31 |
| marios | jm1: o/ | 09:34 |
| frenzyfriday|rover | bhagyashris|ruck, hey, the common component failures downstream are also because of the selinux issue right? | 09:52 |
| bhagyashris|ruck | frenzyfriday|rover, yes | 10:06 |
| frenzyfriday|rover | bhagyashris|ruck, ack. This should fix it https://github.com/redhat-openstack/openstack-selinux/pull/96 | 10:06 |
| frenzyfriday|rover | I am rerunning the master and wallaby common to see if we have any other issues | 10:07 |
| bhagyashris|ruck | frenzyfriday|rover, ack | 10:07 |
| *** ysandeep|lunch is now known as ysandeep | 10:21 | |
| bhagyashris|ruck | frenzyfriday|rover, hey looks like it's different issue now | 10:24 |
| bhagyashris|ruck | it's failing on Error running container image prepare | 10:24 |
| *** tosky_ is now known as tosky | 10:24 | |
| frenzyfriday|rover | bhagyashris|ruck, ok, maybe the selinux thing is resolved then | 10:24 |
| bhagyashris|ruck | frenzyfriday|rover, yeah may be | 10:27 |
| bhagyashris|ruck | on upstream it's stiil failing common component? | 10:28 |
| frenzyfriday|rover | testproj is still running | 10:28 |
| * frenzyfriday|rover will be back in 15 mins | 10:29 | |
| *** rlandy|out is now known as rlandy | 10:30 | |
| rlandy | frenzyfriday|rover: bhagyashris|ruck: ty for filling out rr hackmd :) | 10:31 |
| rlandy | frenzyfriday|rover: bhagyashris|ruck: want to sync about open bugs? | 10:32 |
| rlandy | in particular - common | 10:32 |
| frenzyfriday|rover | rlandy, yep, we have a fix for common. It was not exactly the bug we pointed to. https://github.com/redhat-openstack/openstack-selinux/pull/96 will fix it. Testproj is running | 10:33 |
| bhagyashris|ruck | rlandy, sure frenzyfriday|rover is away for 15 min once she back will sync | 10:33 |
| frenzyfriday|rover | yep, lets sync in 10 mins | 10:33 |
| * frenzyfriday|rover is making tea | 10:34 | |
| rlandy | frenzyfriday|rover: pls send invite when you are back | 10:34 |
| rlandy | we'll meet then | 10:34 |
| rlandy | bhagyashris|ruck" pls can you check the components out on 17 on rhel-8 - validation, tripleo, compute, tempest | 10:42 |
| frenzyfriday|rover | rlandy, bhagyashris|ruck I am back | 10:43 |
| frenzyfriday|rover | https://meet.google.com/aci-bygn-zkz | 10:43 |
| rlandy | joining | 10:44 |
| rlandy | bhagyashris|ruck: https://sf.hosted.upshift.rdu2.redhat.com/logs/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-rhel-8-scenario002-standalone-common-rhos-17/ecc4a9b/logs/undercloud/var/log/tripleo-container-image-prepare.log | 10:56 |
| rlandy | nothing provides python3.6dist(lxml) >= 4.5 needed by python3-oslo-vmware-3.8.1-0.20220726102237.27eeee2.el8osttrunk.noarch" | 10:56 |
| dasm|off | o/ | 11:06 |
| *** dasm|off is now known as dasm | 11:06 | |
| rlandy | bhagyashris|ruck: hi - I'm going to book the TC/UA/PM sync meeting | 11:25 |
| rlandy | I know you're busy | 11:25 |
| *** dviroel|out is now known as dviroel | 11:25 | |
| rlandy | ysandeep: ^^ want to be included? | 11:27 |
| rlandy | inviting dasm | 11:27 |
| ysandeep | rlandy, yes I can join as well | 11:28 |
| rlandy | dasm: hey | 11:29 |
| dasm | rlandy: o/ | 11:29 |
| rlandy | dasm: is 6am tomorrow ok for you? | 11:29 |
| dasm | a bit too early. i'm barely able to think at 6am | 11:29 |
| * dasm is writing those words at 4:30am | 11:29 | |
| rlandy | it's 4:30 am now | 11:29 |
| rlandy | ack | 11:29 |
| dviroel | hahaha | 11:29 |
| dasm | i know. i couldn't sleep, so i decided to get up | 11:30 |
| rlandy | bhagyashris|ruck: hey - you have something booked at 3pm UTC today | 11:30 |
| rlandy | is it missable? | 11:30 |
| soniya29 | rlandy, meeting? | 11:31 |
| rlandy | soniya29: will be there in 5 | 11:31 |
| soniya29 | rlandy, ack | 11:31 |
| dasm | rlandy: oh, if you mean one-time today, 6am is fine. i thought you're gonna do this periodically | 11:31 |
| dasm | but 6am today isn't all hands? | 11:31 |
| rlandy | dasm: tomorrow | 11:32 |
| dasm | ack | 11:32 |
| rlandy | dasm: tc/ua/pm sync | 11:33 |
| dasm | +1 | 11:33 |
| pojadhav | rlandy, include me too UA/TC/PM sync | 11:33 |
| bhagyashris|ruck | rlandy, yeah it's Team catalyst meeting but for India people it's time is different | 11:33 |
| rlandy | bhagyashris|ruck: I moved ir | 11:33 |
| rlandy | it | 11:33 |
| rlandy | ysandeep: bhagyashris|ruck: pojadhav: chandankumar: is 3:30 utc to late? | 11:35 |
| rlandy | if yes, then will move back to 1pm | 11:35 |
| rlandy | all tomorrow | 11:35 |
| pojadhav | rlandy, yeah :( | 11:35 |
| rlandy | pojadhav: pls find out who has this tine's retro and ping them | 11:36 |
| pojadhav | rlandy, sure | 11:36 |
| rlandy | pojadhav: also - pls take over the retro meeting from bhagyashris|ruck | 11:36 |
| rlandy | and move it to thurs | 11:36 |
| pojadhav | rlandy, ack | 11:37 |
| bhagyashris|ruck | rlandy, done | 11:37 |
| rlandy | pojadhav: pls add a section for tomorrow's planning in https://hackmd.io/P5sTPMSBREe5kBS_4jDhIg | 11:37 |
| rlandy | bhagyashris|ruck: same time as wed? | 11:37 |
| rlandy | so it's half an hour later | 11:38 |
| bhagyashris|ruck | rlandy, forgot to change time | 11:38 |
| bhagyashris|ruck | rlandy, Now it's done | 11:39 |
| rlandy | bhagyashris|ruck: thanks | 11:53 |
| bhagyashris|ruck | folks plz note today we dont have Tripleo CI community meeting due to conflict with "Products + Global Engineering all-hands" | 12:04 |
| bhagyashris|ruck | arxcruz, akahat chandankumar marios pojadhav frenzyfriday|rover rlandy rcastillo dviroel dasm soniya29 ysandeep ^ | 12:05 |
| marios | thanks bhagyashris|ruck | 12:05 |
| bhagyashris|ruck | jm1, ^ | 12:05 |
| dasm | k | 12:05 |
| pojadhav | bhagyashris|ruck, ack | 12:05 |
| akahat | ack | 12:05 |
| soniya29 | ack | 12:05 |
| frenzyfriday|rover | ack, thanks | 12:05 |
| chandankumar | ack | 12:05 |
| rlandy | pojadhav: hi | 12:06 |
| dviroel | ack - tahnks | 12:06 |
| pojadhav | rlandy, hello | 12:06 |
| rlandy | pojadhav: the retro schedule has soniya29 on aug 3 and aug 24 | 12:06 |
| pojadhav | rlandy, yeah I was about to ask bhagyashris|ruck.. is that mistakenly done or any reason. | 12:07 |
| rlandy | pojadhav: can you remove soniya29 from 24 and move the schedule up | 12:07 |
| pojadhav | bhagyashris|ruck, if that mistakenly happened, I will revise the names. | 12:07 |
| pojadhav | rlandy, ack will do | 12:08 |
| rlandy | soniya29 is also on Nov | 12:08 |
| rlandy | with ysandeep :) | 12:08 |
| rlandy | idk why | 12:08 |
| pojadhav | rlandy, I will change | 12:08 |
| rlandy | pojadhav: ok - so soniya29 is on this week? | 12:08 |
| soniya29 | rlandy, yeah | 12:08 |
| pojadhav | rlandy, yes I pinged her | 12:08 |
| rlandy | ok | 12:08 |
| rlandy | bhagyashris|ruck: frenzyfriday|rover: in a couple hours of meetings now - pls ping if you need help | 12:13 |
| frenzyfriday|rover | rlandy, the common bug is not related to selinux. Spoke to Cedric. I am creaqting a bug for it | 12:14 |
| frenzyfriday|rover | For the tripleo I am making a patch | 12:14 |
| bhagyashris|ruck | rlandy, sure i have filled bug for common component failure and updated hackmd | 12:16 |
| rlandy | great | 12:16 |
| *** amoralej is now known as amoralej|lunch | 12:48 | |
| soniya29 | pojadhav, hi | 12:55 |
| soniya29 | s/hi/hello | 12:55 |
| pojadhav | soniya29, hello | 12:55 |
| rlandy | all: Products + Global Engineering all-hands meeting | 12:59 |
| *** amoralej|lunch is now known as amoralej | 13:18 | |
| Tengu | bhagyashris|ruck: heya! I took the liberty to add a topic for the community CI call - lemme know if it's fine or not. A small update on nftables integration status. | 13:26 |
| bhagyashris|ruck | Tengu, hey we cancelled today's community call due to conflict with Product + Global engg all handsmeeting | 13:27 |
| Tengu | bhagyashris|ruck: oh, np - lemme push it to next week then :) | 13:27 |
| bhagyashris|ruck | Tengu, we will discuss in the next meeting | 13:27 |
| bhagyashris|ruck | Tengu, ack thanks | 13:28 |
| Tengu | sure | 13:28 |
| *** frenzyfriday|rover is now known as frenzyfriday|rover|lunch | 13:28 | |
| rlandy | bhagyashris|ruck: for periodic-tripleo-ci-rhel-8-standalone-on-multinode-ipa-rhos-17 - is it the same set of tempest failure? | 13:48 |
| bhagyashris|ruck | rlandy, ye | 13:49 |
| bhagyashris|ruck | rlandy, yes | 13:49 |
| rlandy | hmmm ... legit issue? | 13:50 |
| bhagyashris|ruck | rlandy, not sure rerunning the job ... waiting for the result | 13:52 |
| rlandy | frenzyfriday|rover|lunch: hey - when you are back - can you add the recent common component bug? | 13:52 |
| rlandy | bhagyashris|ruck: ack - ok | 13:53 |
| chandankumar | ysandeep: I fixed that ovb issue | 13:54 |
| chandankumar | we moved forward | 13:55 |
| ysandeep | chandankumar, with help of that ovb patch? | 13:55 |
| chandankumar | yes | 13:56 |
| ysandeep | chandankumar, great , so that was an issue because of certs | 13:57 |
| chandankumar | ysandeep: https://paste.centos.org/view/raw/d7fd829f | 13:57 |
| chandankumar | ysandeep: https://review.opendev.org/q/topic:ibm_cloud+project:openstack/openstack-virtual-baremetal | 14:01 |
| chandankumar | next error | 14:03 |
| chandankumar | fatal: [localhost]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'undercloud_uuid' is undefined\n\nThe error appears to be in '/home/chandankumar/.ansible/roles/ovb-manage/tasks/ovb-create-stack.yml': line 99, column 7, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n - | 14:03 |
| chandankumar | name: Attach instance to provision OVB network\n ^ here\n"} | 14:03 |
| chandankumar | will dig into this tomorrow | 14:03 |
| chandankumar | it might be related to undercloud metadata service | 14:04 |
| frenzyfriday|rover|lunch | rlandy, created new bug https://bugs.launchpad.net/tripleo/+bug/1983382 | 14:05 |
| *** frenzyfriday|rover|lunch is now known as frenzyfriday|rover | 14:05 | |
| Tengu | is there anything I can help with the CI issues? there are apparently a ton of them lately | 14:05 |
| chandankumar | Tengu: hey, welcome back :-) | 14:06 |
| Tengu | chandankumar: heya :) thanks! | 14:06 |
| rlandy | Tengu: seems that the common component is having troubke | 14:06 |
| rlandy | trouble | 14:06 |
| rlandy | these are the bugs are following | 14:06 |
| chandankumar | Tengu: yw:-) | 14:06 |
| Tengu | rlandy: "great". was anything updated in the "core" packages such as dnf, libselinux (and friends) and so far? | 14:07 |
| Tengu | chandankumar: though seeing the list of CIX, I may extend my PTO a bit :D | 14:07 |
| chandankumar | hehe | 14:07 |
| rlandy | Tengu: and this one from tripleo: https://bugs.launchpad.net/tripleo/+bug/1983281 | 14:07 |
| Tengu | hmm apparently that one is under control | 14:08 |
| rlandy | Tengu: http://pastebin.test.redhat.com/1069416 | 14:08 |
| rlandy | package diff in common | 14:08 |
| rlandy | plus the two in downstream | 14:08 |
| Tengu | rlandy: only those? | 14:08 |
| rlandy | all detailed here: https://hackmd.io/H9CSoXvlTm6nTZ4bsJkeRg | 14:08 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1983382 | 14:09 |
| rlandy | rhos17 on rhel8: https://bugzilla.redhat.com/show_bug.cgi?id=2113940 | 14:09 |
| rlandy | rhos16.2 on rhel8: https://bugzilla.redhat.com/show_bug.cgi?id=2113948 | 14:09 |
| rlandy | pick your poison | 14:09 |
| Tengu | el8 only? nothing on el9? | 14:09 |
| rlandy | el9 - ok so far | 14:12 |
| rlandy | https://bugs.launchpad.net/tripleo/+bug/1983382 | 14:14 |
| rlandy | and that joy | 14:14 |
| rlandy | Tengu: wrt https://bugs.launchpad.net/tripleo/+bug/1983382 - we we have a iptables, permissions issue here? | 14:17 |
| Tengu | yeah so that last one is... weird. | 14:17 |
| rlandy | s::teardown] "Tearing down.."\n[INFO netavark::firewall] Using iptables firewall driver\n[DEBUG netavark::commands::teardown] Setting up network podman with driver bridge\n[DEBUG netavark::network::core_utils] bridge has 2 connected interfaces\n[DEBUG netavark::network::core] Container veth name being removed: "eth0"\n[DEBUG netavark::network::core] Container veth removed: "eth0"\n[DEBUG netavark::commands::teardown] "Teardown | 14:18 |
| rlandy | complete"\nerror building at STEP "RUN dnf install -y openstack-tripleo-common-containers": error while running runtime: exit status 1\nFailed to write to log, | 14:18 |
| Tengu | rlandy: I don't think iptables has anything to do here. | 14:18 |
| Tengu | the actual error is earlier than that, apparently related to the rpm DB or something. | 14:18 |
| Tengu | I'd check bind-mounts and denials actially. | 14:18 |
| Tengu | wokay. not an SELinux issue. | 14:19 |
| Tengu | https://logserver.rdoproject.org/42/44342/2/check/periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/cda9e66/logs/undercloud/var/log/extra/selinux_denials.txt.gz | 14:19 |
| rlandy | frenzyfriday|rover: ^^ how many times did you see this error? | 14:21 |
| rlandy | on multiple runs? | 14:21 |
| frenzyfriday|rover | reading | 14:22 |
| Tengu | that's the actual error: https://paste.openstack.org/show/bznvtFBLRzt5JW9w8ecl/ | 14:22 |
| Tengu | this command is failing. | 14:22 |
| Tengu | but I saw something related to the rpm db thing. | 14:23 |
| frenzyfriday|rover | rlandy, so far we have only one testproj where we see ^ | 14:23 |
| Tengu | Error: Could not run transaction. | 14:23 |
| Tengu | frenzyfriday|rover: weird. reproductable at will? | 14:23 |
| rlandy | frenzyfriday|rover: can you rekick? | 14:23 |
| rlandy | yep | 14:23 |
| rlandy | let's see if this repeats | 14:23 |
| frenzyfriday|rover | yep, reckicking | 14:23 |
| Tengu | I'm really surprised. this error seems like a glitch to me. i.e. weak disk perfs, maybe broken disk even... | 14:24 |
| Tengu | aaand in this specific error, the issue is related to one specific package: openstack-tripleo-common-containers | 14:24 |
| Tengu | nothing more, nothing less. There's apparently no other package involved, not even dependencies. | 14:25 |
| Tengu | https://logserver.rdoproject.org/42/44342/2/check/periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/cda9e66/logs/undercloud/home/zuul/container-builds/876f3388-1ef2-4a6e-8dce-4bff6b47441f/base/base-build.log.txt.gz (at the bottom) | 14:25 |
| frenzyfriday|rover | rechecked https://review.rdoproject.org/r/c/testproject/+/44342 | 14:25 |
| Tengu | will try to keep an eye, thanks! | 14:25 |
| dasm | how the place of job definition is chosen? i see tripleo-ci-base-ovb-required-projects-rdo defined in rdo-jobs, but its child tripleo-ci-ovb-rdo-centos-8 is defined in config. | 14:25 |
| rlandy | dasm: job defs in config generally require secrets | 14:28 |
| dasm | so we can basically jump back and forth between zuul configs? config -> rdo-jobs -> config? | 14:28 |
| dasm | that's confusing | 14:29 |
| rlandy | the less we keep in config the better | 14:34 |
| rlandy | no speculative testing in config | 14:34 |
| rlandy | hence the movement | 14:34 |
| dasm | mhm | 14:36 |
| chandankumar | dasm: due to dependency on ovb-manage role, we need to keep it in config | 14:37 |
| chandankumar | but depends-on from ovb repo will work | 14:37 |
| Tengu | rlandy: wondering about one thing I'm seeing in the container build thing: dnf clean all && rm -rf /var/cache/dnf not sure the last "rm -rf" is really needed/advised.. | 14:40 |
| Tengu | wondering if this couldn't lead to some missing directory, therefore preventing any write, depending on how dnf tries to detect things. | 14:40 |
| Tengu | imho, that rm is OK as a last cleaning step, not really at the beginning though. the "dnf clean all" is safer. | 14:41 |
| Tengu | though it's probably more poking in the dark than anything. | 14:41 |
| dasm | zuul yaml config is pure evil. it's so convoluted dependency hell... i always liked object oriented programming. but this is an example why oop should be avoided :/ | 14:43 |
| *** ysandeep is now known as ysandeep|dinner | 14:47 | |
| bhagyashris|ruck | rlandy, frenzyfriday|rover hey leaving for the day... | 14:49 |
| frenzyfriday|rover | bhagyashris|ruck, cool, anything i should watch? | 14:49 |
| bhagyashris|ruck | multinode ipa rhos17 on rhel8 | 14:49 |
| bhagyashris|ruck | sc010 on 16.2 | 14:50 |
| chandankumar | dasm: if we docouple the ovb-manage and find a way to keep those piece which needs secrets that would be a great win | 14:50 |
| dasm | chandankumar: let's think about that. i'm gonna add something to our planning session | 14:51 |
| rlandy | bhagyashris|ruck: thanks | 14:51 |
| rlandy | Tengu: idk - that's been there for a while | 14:52 |
| Tengu | rlandy: sure. still... I can push a WIP and we can do another testproject. | 14:52 |
| Tengu | IF the current re-kick fails. | 14:52 |
| rlandy | Tengu: ack | 14:53 |
| * dasm is stepping away for a bit | 14:54 | |
| marios | needs reviews please https://review.opendev.org/c/openstack/python-tripleoclient/+/851384 (context & testing @ https://bugs.launchpad.net/tripleo/+bug/1983004/comments/14 ) thank you | 15:09 |
| reviewbot | Do you want me to add your patch to the Review list? Please type something like add to review list <your_patch> so that I can understand. Thanks. | 15:09 |
| *** dviroel is now known as dviroel|lunch | 15:14 | |
| rlandy | frenzyfriday|rover: can you check if the master job has the same issue? | 15:16 |
| rlandy | common with container build | 15:16 |
| frenzyfriday|rover | checking | 15:16 |
| rlandy | frenzyfriday|rover: we need more data here | 15:16 |
| rlandy | pls recheck multiple common jobs | 15:16 |
| rlandy | from master wallaby, c8m c9 | 15:16 |
| rlandy | we need to put various logs in place | 15:17 |
| rlandy | and maybe remove the line Tengu suggests | 15:17 |
| rlandy | in one of those checks | 15:17 |
| frenzyfriday|rover | the testproj failed with the same error. Removing rm -rf and rekicking | 15:18 |
| rlandy | frenzyfriday|rover: yeah - let's see a master kick as well | 15:20 |
| rlandy | we are debugging with too little data here | 15:20 |
| Tengu | I'm on a call - sorry. Should be free in ~40 minutes. | 15:24 |
| frenzyfriday|rover | rlandy, https://review.rdoproject.org/r/c/testproject/+/44374 has all the jobs | 15:24 |
| rlandy | bhagyashris|ruck: pojadhav: the planning meeting needs to come up one hour | 15:24 |
| Tengu | but once I'm free, I can help you for the logging, maybe. | 15:24 |
| frenzyfriday|rover | Tengu, which file were you talking about " dnf clean all && rm -rf /var/cache/dnf" ? | 15:24 |
| rlandy | frenzyfriday|rover: and do we have one with Tengu's suggested depends-on? | 15:24 |
| frenzyfriday|rover | yep, which file is it? I can remove it and try | 15:25 |
| Tengu | frenzyfriday|rover: yeah, just removing the "rm -rf". gimme a minute to find the right location though. | 15:25 |
| rlandy | https://github.com/openstack/tripleo-common/blob/master/container-images/tcib/base/base.yaml | 15:25 |
| rlandy | that one? | 15:25 |
| frenzyfriday|rover | ack, thanks | 15:25 |
| frenzyfriday|rover | meanwhile I found a bugzilla with the same error cannot open Packages index using bdb_ro - Operation not permitted . Checking it | 15:25 |
| Tengu | rlandy: yeah. though it's apparently in ALL container files X( | 15:26 |
| Tengu | but if we can get it removed from the base one, we will know if it builds or not. | 15:26 |
| frenzyfriday|rover | aye | 15:26 |
| Tengu | basically: https://github.com/openstack/tripleo-common/blob/master/container-images/tcib/base/base.yaml#L2 | 15:26 |
| Tengu | just that one here on L2 | 15:26 |
| Tengu | the last one on L54 can stay imho | 15:27 |
| Tengu | want me to provide a WIP patch? | 15:27 |
| frenzyfriday|rover | no, I am putting up the patch | 15:28 |
| frenzyfriday|rover | with dnm | 15:28 |
| Tengu | frenzyfriday|rover: sounds good! | 15:28 |
| frenzyfriday|rover | https://review.opendev.org/c/openstack/tripleo-common/+/851901 | 15:28 |
| Tengu | I'm really not sure it will fix things up. but worth a try. | 15:28 |
| *** ysandeep|dinner is now known as ysandeep | 15:29 | |
| frenzyfriday|rover | https://bugzilla.redhat.com/show_bug.cgi?id=1960991 | 15:30 |
| frenzyfriday|rover | I see this speaks of the same error and says something about selinux denied | 15:31 |
| Tengu | perfect, no selinux denial logged in. | 15:33 |
| Tengu | but I didn't see anything relevant in the job logs related to selinux | 15:33 |
| Tengu | https://logserver.rdoproject.org/42/44342/2/check/periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/cda9e66/logs/undercloud/var/log/extra/selinux_denials.txt.gz nothing related to dnf nor rpm - those 2 are known and related to openvswitch itself. | 15:34 |
| Tengu | there's something else. | 15:34 |
| *** amoralej is now known as amoralej|off | 15:38 | |
| *** ysandeep is now known as ysandeep|out | 15:43 | |
| Tengu | frenzyfriday|rover: stupid question: is the OS up-to-date? I mean, is there a step that actually update the VM before doing anything? | 15:44 |
| frenzyfriday|rover | checking | 15:44 |
| frenzyfriday|rover | rlandy, how can I find out if we run update before the container build steps? release file? | 15:46 |
| Tengu | frenzyfriday|rover: https://logserver.rdoproject.org/42/44342/2/check/periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/cda9e66/logs/undercloud/var/log/dnf.log.txt.gz there's a "dnf update -y" | 15:48 |
| frenzyfriday|rover | oh :D thanks! | 15:48 |
| Tengu | not that many things though.... | 15:48 |
| frenzyfriday|rover | I see here dnf.exceptions.RepoError: Unknown repo: 'delorean' | 15:50 |
| frenzyfriday|rover | Is that supposed to be there? | 15:50 |
| Tengu | uho. | 15:50 |
| Tengu | nope. but it seems to recover a bit after that by passing another version of --enablerepo | 15:51 |
| Tengu | repoquery repoquery --disablerepo=* --enablerepo=delorean -a --qf %{sourcerpm} <- fails, repoquery repoquery --disablerepo=* --enablerepo=delorean-*-deps -a --qf %{sourcerpm} <- good | 15:51 |
| frenzyfriday|rover | oh, right | 15:51 |
| Tengu | and I think this is more related to the log and env data collection | 15:52 |
| Tengu | this is the dnf.log from the host, not the container. | 15:52 |
| Tengu | we actually don't have that log. | 15:52 |
| Tengu | frenzyfriday|rover: fun thing: apparently dnf is working just fine at the beginning of the container build. | 15:54 |
| Tengu | frenzyfriday|rover: basically, this is working: dnf install -y crudini | 15:54 |
| Tengu | it installs the package with its dependencies, but ALSO upgrades a fair amount of packages | 15:55 |
| Tengu | https://logserver.rdoproject.org/42/44342/2/check/periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/cda9e66/logs/undercloud/home/zuul/container-builds/876f3388-1ef2-4a6e-8dce-4bff6b47441f/base/base-build.log.txt.gz | 15:55 |
| frenzyfriday|rover | Tengu, sorry one more stupid question - i am comparing the selinux conf thingy from a passing log - https://logserver.rdoproject.org/openstack-component-common/opendev.org/openstack/tripleo-ci/master/periodic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/a41c0e5//logs/undercloud/var/log/extra/selinux.txt.gz - the mode is permissive here. But in our log it is enforcing https://logserver.rdoproject.org/42/44342/2/check/perio | 15:55 |
| frenzyfriday|rover | dic-tripleo-ci-centos-9-scenario003-standalone-common-wallaby/c3718e2/logs/undercloud/var/log/extra/selinux.txt.gz | 15:55 |
| Tengu | it also replaces a dnf plugin (subscription-manager), but also updates libdnf | 15:56 |
| frenzyfriday|rover | Is the mode set to permissive later on ? (like after the steps where it is failing now passes?) | 15:56 |
| Tengu | frenzyfriday|rover: err... no idea. afaik upstream is in permissive, only downstream is enforcing... can't say more unfortunately :/ | 15:56 |
| Tengu | maybe a question for rlandy -^^ ? | 15:57 |
| Tengu | frenzyfriday|rover: ok. I'm pretty sure the issue is due to an update of some packages within the container that happen when the crudini package is installed. | 15:58 |
| Tengu | especially since some of the upgraded packages are linked to dnf. pretty sure there's something in there. | 15:58 |
| Tengu | ah, yeah. dnf itself is updated. | 15:59 |
| Tengu | yay | 15:59 |
| rlandy | frenzyfriday|rover: what Tengu says | 16:02 |
| rlandy | should be permissive in upstream | 16:02 |
| Tengu | hmmm. and there's apparently something broken with the locales. | 16:02 |
| Tengu | don't UBI remove the locales data ??? | 16:02 |
| rlandy | lunch - brb | 16:07 |
| *** dviroel|lunch is now known as dviroel| | 16:15 | |
| *** dviroel| is now known as dviroel | 16:15 | |
| Tengu | frenzyfriday|rover, rlandy trying to reproduce it on a CS8 VM running the generated script. | 16:23 |
| frenzyfriday|rover | thanks! | 16:23 |
| Tengu | err.... fun... | 16:24 |
| Tengu | it fails, but for another reason :] | 16:24 |
| Tengu | it doesn't like the "if" in the builda run ${CONTAINER} command line. | 16:24 |
| Tengu | great. | 16:24 |
| * dasm back | 16:25 | |
| frenzyfriday|rover | rlandy, I am leaving for the day in some time. I have updated the hackmd | 16:32 |
| *** jpena is now known as jpena|off | 16:34 | |
| Tengu | meh... unable to reproduce here. | 16:35 |
| Tengu | but my image is 100% up-to-date, and was rebooted in my local workflow. | 16:36 |
| Tengu | might be something? | 16:36 |
| rlandy | Tengu: I wonder if we didn't update | 16:37 |
| rlandy | and build from scratch if it would help | 16:37 |
| rlandy | frenzyfriday|rover: thanks - reading back | 16:37 |
| Tengu | rlandy: my env: latest CS-8 image from cloud.centos.org, updated in-place (using virt-sysprep); booted using virt-manager; fetched the buildahfile.sh.txt.gz; uncompressed it; edited the content to NOT rely on the.... oh, wait. | 16:39 |
| Tengu | apparently, my buildah is doing some fancy things. | 16:39 |
| rlandy | fancy? | 16:39 |
| Tengu | yeah - it doesn't pass the whole line when there are "&&" and such. | 16:40 |
| Tengu | so basically, it breaks. | 16:40 |
| Tengu | and doesn't really do the right thing. | 16:40 |
| rlandy | ugh - and real issues on 17 ipa job | 16:40 |
| * Tengu digging | 16:40 | |
| rlandy | new version of buildah? | 16:40 |
| Tengu | buildah version 1.24.2 (image-spec 1.0.2-dev, runtime-spec 1.0.2-dev) | 16:41 |
| Tengu | can we compare with a working job on cs8? | 16:41 |
| Tengu | where does that one come from now. | 16:42 |
| Tengu | wait - cs8... is there the module thingy? | 16:42 |
| Tengu | or stream. or. | 16:42 |
| rlandy | cs8 and cs9 | 16:42 |
| Tengu | erf... that's probably my mistake then. | 16:42 |
| Tengu | what stream are we supposed to use on cs8 again? | 16:42 |
| rlandy | https://review.rdoproject.org/zuul/builds?job_name=periodic-tripleo-ci-centos-9-scenario003-standalone-master&skip=0 | 16:43 |
| rlandy | dies fine | 16:43 |
| rlandy | does | 16:43 |
| rlandy | buildah.x86_64 1:1.26.1-4.el9 @quickstart-centos-appstreams | 16:43 |
| Tengu | yep | 16:43 |
| Tengu | ok. so I don't have the right stream. lemme... dang. I don't remember. | 16:44 |
| Tengu | thoguh tripleo-repos was setting things up | 16:44 |
| rlandy | let's compare master and master | 16:46 |
| Tengu | oh. err. wait. you were pointing to a cs9 job. | 16:47 |
| Tengu | I was looking for a cs8 | 16:47 |
| rlandy | buildah.x86_64 1:1.26.1-4.el9 @quickstart-centos-appstreams | 16:47 |
| rlandy | same | 16:47 |
| Tengu | on cs8 ?? | 16:47 |
| rlandy | diffing out those twp | 16:47 |
| rlandy | no - working with master now | 16:47 |
| Tengu | oh. ok. | 16:48 |
| Tengu | errr are we seeing that container-build issue on master as well now? | 16:48 |
| rlandy | intergration vs common component | 16:48 |
| rlandy | yep | 16:48 |
| Tengu | sh*t... | 16:48 |
| rlandy | yep | 16:48 |
| Tengu | ok, now I'm with you. | 16:48 |
| Tengu | I was stuck on the "it's a cs8 thingy" | 16:48 |
| rlandy | ugh - and I'm in and out of meetings all day | 16:48 |
| Tengu | and it's almost 7pm here... | 16:49 |
| * Tengu shakes fist at TZ | 16:49 | |
| Tengu | :) | 16:49 |
| rlandy | Tengu: I can leave you notes for tomorrow | 16:49 |
| rlandy | let me diff these two on master ... | 16:49 |
| Tengu | rlandy: sounds good. I'll check back once I'm up tomorrow. | 16:50 |
| Tengu | lemme check my agenda though.... since I'm seeing meeting poping | 16:50 |
| rlandy | I already sent the package diff | 16:50 |
| rlandy | per the components | 16:50 |
| Tengu | ok - I just have that program call, and that's all. I can dedicate my day to that cix if needed. | 16:50 |
| rlandy | python3 roles/rrcockpit/files/telegraf_py3/ruck_rover.py --release master --distro centos-9 --component common | 16:50 |
| Tengu | I'll also try to reproduce it locally. but iirc, cs9 doesn't hace container-tools stream anymore. | 16:50 |
| rlandy | openstack-trove-17.1.0-0.20220720123507.4a3ad14.el9 │ openstack-trove-17.1.0-0.20220729095336.ade5ca7.el9 │ | 16:51 |
| rlandy | │ diskimage-builder-3.22.1-0.20220712164907.ba88a12.el9 │ diskimage-builder-3.22.1-0.20220801150041.ac5823c.el9 │ | 16:51 |
| rlandy | │ openstack-selinux-0.8.34-0.20220711202841.a241718.el9 │ openstack-selinux-0.8.34-0.20220727174553.c3061f5.el9 │ | 16:51 |
| rlandy | │ openstack-tobiko-0.6.1-0.20220721215850.648f963.el9 │ openstack-tobiko-0.6.1-0.20220727131836.b303fa4.el9 | 16:51 |
| chandankumar | dasm: rlandy: dpawlik https://review.rdoproject.org/r/c/testproject/+/44151 - periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm https://review.rdoproject.org/zuul/build/6fb69164cdd844fd977e265d421e4bb9 : SUCCESS in 1h 57m 21s | 16:51 |
| chandankumar | https://logserver.rdoproject.org/51/44151/19/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm/6fb6916/logs/undercloud/var/log/tempest/stestr_results.html.gz | 16:51 |
| rlandy | ^^ that's the component diff | 16:51 |
| rlandy | but will do a full diff | 16:51 |
| rlandy | then can try exclude one version | 16:51 |
| rlandy | and see | 16:51 |
| Tengu | hmmm imho we're hitting something from outside of the component. | 16:51 |
| Tengu | i.e. OS level - updated package at some point. | 16:51 |
| rlandy | chandankumar+++++++++++++++++++++++ | 16:51 |
| dasm | chandankumar++ congrats! | 16:52 |
| rlandy | chandankumar: ovb success on IBM cloud?????????????? | 16:52 |
| rlandy | wow - awesomeness | 16:52 |
| chandankumar | it is running check content i think | 16:52 |
| rlandy | this is waaaaaaaaaaaaaaaaayyyyyyy cool | 16:52 |
| Tengu | meaning we'll get some more power for the CI! woohpp | 16:52 |
| chandankumar | https://logserver.rdoproject.org/51/44151/19/check/periodic-tripleo-ci-centos-9-ovb-3ctlr_1comp-featureset001-master-ibm/6fb6916/logs/undercloud/var/log/tempest/stestr_results.html.gz | 16:52 |
| rlandy | Tengu: an alternative to shaking vexx | 16:52 |
| Tengu | anyway. I'm out. rlandy I'll check tomorrow for notes and updates, and try to get a local reproducer. | 16:53 |
| chandankumar | rlandy: let;s run multiple ovb jobs and see the performance | 16:53 |
| Tengu | that's the best thing to do imho: get a local reproducer to be able to debug things. | 16:53 |
| rlandy | Tengu: sur e- will leave you notes | 16:53 |
| rlandy | I want to try lock some versions | 16:53 |
| rlandy | and see if I can narrow it down | 16:53 |
| Tengu | sure | 16:53 |
| rlandy | has to be one of the rpms above | 16:53 |
| Tengu | see you around folks ;). | 16:53 |
| rlandy | the rest are the same | 16:54 |
| rlandy | or would hit all components | 16:54 |
| Tengu | rlandy: would be weird.... | 16:54 |
| rlandy | but will let you know | 16:54 |
| rlandy | Tengu: have a good night | 16:54 |
| Tengu | I don't expect anything selinux related, seeing the logs... | 16:54 |
| Tengu | but I may be wrong. | 16:54 |
| Tengu | anyway. off | 16:54 |
| rlandy | see you | 16:54 |
| chandankumar | dpawlik++++++++++++++++ last night help made it happen | 16:54 |
| rlandy | dpawlik: thank you | 16:55 |
| rlandy | chandankumar: this is really great | 16:55 |
| rlandy | we can try move one line | 16:55 |
| rlandy | maybe train? | 16:55 |
| rlandy | wallaby c8? | 16:55 |
| rlandy | let's decide tomorrow based on quota | 16:56 |
| rlandy | but really great work | 16:56 |
| rlandy | thank you, chandankumar | 16:56 |
| dasm | dviroel: qq, I'm trying to test zuul config change. Can I reuse the same name? https://review.rdoproject.org/r/c/config/+/44377 | 18:19 |
| dasm | Or do I need to redefine entire tree? | 18:19 |
| dasm | trying that here: https://review.rdoproject.org/r/c/testproject/+/42319 | 18:22 |
| dviroel | dasm: sry, not sure if I understand your question | 18:23 |
| dviroel | looking at your tp | 18:24 |
| dasm | sorry if i'm not being clear. | 18:25 |
| dasm | i need to test config change. usually any other change would be done via depends-on, but not "config" (due to secrets) | 18:25 |
| dasm | dviroel: can i just copy-paste "config" change content to overwrite entire dependency tree, or do i need to recreate all leves/dependencies like they're defined | 18:26 |
| dviroel | ah, now i see | 18:26 |
| dasm | > Job tripleo-ci-ovb-rdo-centos-8 in testproject is not permitted to shadow job tripleo-ci-ovb-rdo-centos-8 in config | 18:26 |
| dasm | seems like i can't do that :( | 18:26 |
| dasm | it would be so convenient | 18:26 |
| dasm | ok, i'll try workaround then | 18:27 |
| dviroel | dasm: this will be hard to test, you can test with some specific jobs, but it is hard to say that won't break anything else | 18:30 |
| rlandy | Tengu: frenzyfriday|rover: ok - so the same issue reproduces in the integration line if you try build containers in standalone ... | 18:30 |
| rlandy | openstack-tripleo-common-containers | 18:30 |
| rlandy | RPM: warning: Found bdb_ro Packages database while attempting sqlite backend: using bdb_ro backend. | 18:30 |
| rlandy | RPM: error: cannot open Packages index using bdb_ro - Operation not permitted (1) | 18:30 |
| rlandy | RPM: error: cannot open Packages database in /var/lib/rpm | 18:30 |
| rlandy | The downloaded packages were saved in cache until the next successful transaction. | 18:30 |
| rlandy | You can remove cached packages by executing 'dnf clean packages'. | 18:30 |
| rlandy | Error: Could not run transaction. | 18:30 |
| dviroel | dasm: try next job in tree, which may be on rdo | 18:31 |
| dviroel | dasm: just for sanity check, and with those results we could merge the config one, and see if doesn't break anythin | 18:32 |
| dasm | dviroel: i think i just did that | 18:34 |
| *** kopecmartin_ is now known as kopecmartin | 19:09 | |
| *** dviroel is now known as dviroel|biab | 19:56 | |
| * jm1 out for today, have a nice evening :) | 19:57 | |
| dasm | jm1[m]: o/ | 20:10 |
| *** dviroel|biab is now known as dviroel | 20:31 | |
| dviroel | rlandy: yeah, nodepool is deleting our resources, just tested with floating ip | 20:42 |
| rlandy | dviroel: for podified? | 20:45 |
| dviroel | rlandy: yes | 20:48 |
| rlandy | dviroel: ok - I'll work on getting you a specific compute tomorrow | 20:49 |
| dviroel | rlandy: ack - we can sync about that tomorrow | 20:50 |
| *** dviroel is now known as dviroel|afk | 20:58 | |
| * dasm => off | 21:09 | |
| dasm | o/ | 21:09 |
| *** dasm is now known as dasm|off | 21:09 | |
| rlandy | Tengu: left notes in https://bugs.launchpad.net/tripleo/+bug/1983382/comments/3 | 22:10 |
| rlandy | think it is fixed now | 23:22 |
| *** rlandy is now known as rlandy|out | 23:22 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!