Tuesday, 2023-08-08

opendevreviewJacob Anders proposed openstack/ironic master: Prevent MissingAttribute error when supportedApplyTime missing  https://review.opendev.org/c/openstack/ironic/+/88573902:41
janders^ reported tested this fix for me today, seems to work (in conjunction with BIOS/BMC firmware upgrade that fixes another missing attribute problem), reviews welcome (quick/easy review)02:42
janderss/reported/reporter02:42
TheJuliajanders: reviewed, you'll want to take a look03:06
TheJuliasince sushy can return the property as None afaik03:06
jandersthank you Julia, will look into this more03:07
jandersthank you TheJulia, I somehow got the regression from the first revision back in (!?!). Second revision was good and I broke it again somehow while adding the release note. Still no idea how but I clearly did!03:10
jandersfixing it now03:10
TheJuliaalways pull the latest change down :)03:11
TheJuliaanyway, going back to the evening03:11
TheJuliaHave a wonderful day03:11
jandersthank you TheJulia, have a great evening!03:11
opendevreviewJacob Anders proposed openstack/ironic master: Prevent MissingAttribute error when supportedApplyTime missing  https://review.opendev.org/c/openstack/ironic/+/88573903:16
opendevreviewJacob Anders proposed openstack/ironic master: Prevent MissingAttribute error when supportedApplyTime missing  https://review.opendev.org/c/openstack/ironic/+/88573903:18
opendevreviewTakashi Kajinami proposed openstack/ironic master: Fix missing oslo.versionedobjects library option  https://review.opendev.org/c/openstack/ironic/+/89071506:05
rpittaugood morning ironic! o/06:39
kubajjGood morning rpittau and everyone! o/07:25
rpittauhey kubajj :)07:25
opendevreviewwangjiaqi proposed openstack/ironic-tempest-plugin master: Cleanup py27 support  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/89073207:42
opendevreviewMerged openstack/ironic master: Prevent MissingAttribute error when supportedApplyTime missing  https://review.opendev.org/c/openstack/ironic/+/88573907:59
iurygregorymorning ironic11:45
kubajjgood morning iurygregory 11:53
iurygregorykubajj, o/11:56
dkingGood morning.12:17
iurygregorymorning dking o/12:18
rpittauiurygregory dking o/12:23
rpittauanyone free for a quick approval? :) https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/88775412:23
iurygregorylet me look12:23
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Collect lshw output in json format  https://review.opendev.org/c/openstack/bifrost/+/89040812:24
rpittaubifrost cs9 jbos hsould be fixed, or well at least the new systemd pkg is available12:25
dkingI know this is off topic, but as I'm having trouble finding much help in other places, would anybody here be familiar with pynetbox having a problem with https_proxy? When I try to set mine (in my collector), I get some kind of SSL recursion.12:25
iurygregoryI'm not =( sorry 12:30
opendevreviewMerged openstack/ironic master: Fix missing oslo.versionedobjects library option  https://review.opendev.org/c/openstack/ironic/+/89071512:57
opendevreviewMaksim Malchuk proposed openstack/bifrost stable/2023.1: Remove extra symbols accidentally added  https://review.opendev.org/c/openstack/bifrost/+/89078913:06
JayFdking: DM me the broken output and some basic details and I'll give it a cursory look. I don't have a lot of time to look at it, but maybe I can point you in a direction13:29
opendevreviewMerged openstack/ironic-python-agent-builder master: Build tinyipa with tinycore 14.x  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/88775414:14
opendevreviewVerification of a change to openstack/bifrost master failed: Create the log file for the disk-image-create command  https://review.opendev.org/c/openstack/bifrost/+/82289515:11
rpittaubifrost CI still not happy, probably the new systemd pkg is not in the repo yet15:38
JayFCentOS Stream 9 -- is it infra mirrored?15:43
rpittauI believe it is15:54
JayFif we know it's in upstream mirror but not in infra mirror, I wonder if fungi or one of the infra folks can help us close that gap15:59
fungihttps://static.opendev.org/mirror/centos-stream/timestamp.txt says last mirror update was about 4 hours ago16:02
fungihttps://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/mirror-update/files/centos-stream-mirror-update#L44 says we're currently pulling updates from mirror.rackspace.com16:05
JayFthat can be ... significantly out of date16:05
JayFunless they've improved management of it in the last few years16:05
rpittaufungi: thanks, what's the link to the repo content?16:05
JayFWhen working there; the mirrors could be up to 2-3 days old16:06
fungievery miror we've ever tried can be severely out of date. several times a year, whoever we're mirroring from stops updating and somebody proposes a change to another one16:06
JayFmakes sense16:07
fungirpittau: give me a few, it's lunch time here so i'm trying to type one-handed between bites of leftovers16:07
fungiif somebody can clarify what i'm looking for, that might help16:07
rpittaufungi: ah no worries! no rush :)16:08
rpittaufungi: we'd like to see if systemd package has been updated for cs916:08
JayFdo you know the fixed version rpittau ?16:09
fungihow do i know if the systemd package i find is an updated one?16:09
fungimake that packageS. i see quite a few systemd-252-*.rpm packages at https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/16:09
rpittaupkg with fix is systemd-252-1716:10
rpittaufungi: thanks again, it's not there yet16:10
fungihighest i see there is systemd-252-16.el9.x86_64.rpm16:10
rpittauyeah16:10
rpittauthat's the broken one16:10
fungido other mirrors have a newer version? how long as the fixed package been in circulation?16:10
fungiany idea how out of date this makes the rackspace mirror?16:11
fungiif it's way stale, we can try switching to pull from somewhere else16:11
JayFfixed package landed today16:11
rpittaufungi: I'm not sure the pkg is already out yet, I believe they started syncing it today16:11
fungiokay, yeah i wouldn't expect anything fast based on prior experiences with lack of urgency around stream16:12
rpittauyep16:14
JayFrpittau: gonna be honest, this experience makes me wish we were testing against rocky :/16:14
JayFI keep hearing people say that stream is stable, but like, if this bug landed in a point-release that an enterprise ran ... all hell would break loose :( 16:15
rpittauJayF: probably worth discussing the "rocky" testing in one of the next meetigins or even at PTG16:16
rpittauI'm also very disappointed as it's not the first this s...tuff happens16:17
rpittauanyway, time to rest the eyes, see ya tomorrow! o/16:20
JayFo/16:20
fungiwe do have current rockylinux-9 images available for jobs to use, if you want to try it out16:26
opendevreviewMahnoor Asghar proposed openstack/ironic master: [WIP]: Add non-default inspection hooks (1-4 of 11)  https://review.opendev.org/c/openstack/ironic/+/89081717:17
opendevreviewJakub Jelinek proposed openstack/ironic master: WIP: Introduce default kernel/ramdisks by arch  https://review.opendev.org/c/openstack/ironic/+/89081917:47
JayFI'm going to chuck something at a wall :( the devstack I thought was setup and sane20:17
JayFI took for granted that nova was setup in it. It's not.20:17
TheJuliadoh20:18
JayFI'm outsized levels of upset20:18
JayFbecause I've been trying to get a working devstack for literally a week+ at this point20:18
jelabarre-rhyes, online here20:19
JayFI've been using exclusively bifrost dev environments for so long :/20:19
TheJuliajelabarre-rh: https://github.com/openstack/ironic-tempest-plugin/blob/master/ironic_tempest_plugin/tests/scenario/ironic_standalone/test_ramdisk_iso.py <-- to answer your question20:19
JayFwelcome jelabarre-rh 20:19
JayFI wonder where you work ;) 20:19
jelabarre-rhadded the extension since I've had a similar nick on various IRC systems20:20
JayFoh I'm just making jokes, don't sweat it20:20
JayFwelcome o/20:20
jelabarre-rhI think in the late 90's I was using Gary7 or something like that (Star Trek reference)20:21
JayFI was unfamiliar with that reference, but I'm more of a DS9 guy (and grew up on TNG) :D 20:22
jelabarre-rh'Assignment Earth'20:24
TheJulia.... we need more Supervisors of the Those Old Scientists vain on Earth.20:24
TheJulia... and Picard too...20:24
* TheJulia swears a UFP flag did not suddenly appear20:25
* JayF storms the vulcan ship and steals all the tech20:26
* TheJulia fires up etsy to look for a Terran Empire flag20:26
JayFreal talk: most underrated show and episodes in all of trek: Alternate universe episodes in ST: Enterprise20:27
TheJulia+10000000000020:27
TheJuliaInterestingly... there is a terran emprie flag on amazon but not etsy20:31
* JayF tried hot swapping his local.conf and seeing if it gets him a nova20:31
JayFlooks promising so far20:31
JayFOur devstack docs are not great, but I've failed at coming up with better methods20:32
JayFI wanna find a way, for testing Nova<>Ironic interactions, to configure devstack to create a bunch of fake ironic nodes and register them in ironic20:32
JayFbasically the default tempest config, just with no actual hydrated nodes behind the API of Ironic20:32
JayFthat would be sufficient to test sharding and to test my lessee change20:33
JayFoh, btw TheJulia: https://review.opendev.org/c/openstack/ironic/+/888277/2#message-1313fccebe0052159ed30ab6fee0cdc80449f8ad20:33
JayFsharding passed our ci20:33
TheJuliait is possible, but the stock example devstack config might not be the palce to start with that20:33
TheJulia\o/20:34
JayFonce I get this damn thing working again, I'm going to load up sharding and lessee and play for a bit20:34
TheJuliaokay20:34
JayFmy only real goal this week is to get lesse change in nova mergable and sharding fully tested20:34
JayFI do think there's value in you still looking at sharding testing if you can spare time and get a stack up for something else and can jfdi20:34
JayFjust because this is a big change and more testing is better20:34
TheJuliaOkay, hit me up tomorrow morning, so I can do so with a fresh mind20:35
TheJuliaI'm close on service steps, fwiw20:35
JayFnice20:35
JayFclient for that is my backup plan if I get frustrated doing deeper work lol20:35
JayF++lib/neutron_plugins/services/l3:create_neutron_initial_network:196  oscwrap --os-cloud devstack --os-region RegionOne network create private -f value -c id20:36
JayFError while executing command: HttpException: 503, Unable to create the network. No tenant network is available for allocation.20:36
JayFthis is what happens EVERY TIME20:36
JayFwhen I use the ironic-tempest-plugin config from https://docs.openstack.org/ironic/latest/contributor/dev-quickstart.html20:36
TheJuliahmmmm20:37
TheJuliaummmmm20:37
JayFdbcounter has "mysql server has gone away" errors in proximity to that in the logs20:38
JayFabout 10 seconds later though20:38
TheJuliadisable dbcounter20:38
JayFyou know the devstack slug for that off the top?20:38
TheJuliait is a devstack_service entry afaik20:39
JayFdstat?20:40
JayFlooking  at https://github.com/openstack/ironic/blob/master/zuul.d/ironic-jobs.yaml#L8120:40
TheJuliamaybe20:40
JayFnah I don't think that's it looking at the bug20:40
JayFhttps://github.com/openstack/ironic/commit/8b98dfafd848bc1f1bd31c073bcdd3760608104520:41
JayFwe only disable for grenade20:41
TheJulianah, not dstat20:41
TheJuliayeah, that is the way that you just found20:41
JayFadmittedly, I have little faith that is going to make a difference20:41
JayFprimarily because I'm on beefier gear than our gates right now20:41
JayFand so it's hard for me to believe dbcounter would break locally but work in gate20:42
TheJuliai think your issue might be that your defining local network config and devstack then might be taking a different path on the create networks code in the neutron plugin20:43
TheJuliawell, your networking issue20:44
JayFthat's identical to what I get with that local.conf on a clean install20:44
JayFand the network this is on is 172.16.254.0/24 so it shouldn't conflict with anything devstacky20:44
JayF(I pick that for exactly that reason)20:44
TheJuliathe doc has network_gateway, fixed_network_size, etc.. we have none of that by default in our job definitions unless it is in devstack-base20:44
JayFI've reproduced this error with both those set and unset20:45
JayFI've literally been fighting this for almost the whole week you were gone20:45
JayFwhich is why I feel like every door is already closed on troubleshooting it20:45
TheJuliainteresting20:45
JayFexcept "lol you need OBSCURE_BS=lolcats"20:45
JayFor something like that 20:45
JayFI'm hunting a known unknown 20:45
JayFI started out with stealing LOCALRC from the gate and trying to thin it out and use it20:46
JayFthat didn't work20:46
JayFso I went to the config20:46
JayFthat didn't work, except for apparently the last time when I copy+pasted the !nova version which did work20:46
TheJuliado you get that error wen ironic is trying to run setup?20:46
TheJuliaNEUTRON_CREATE_INITIAL_NETWORKS comes to mind from neutron rooted at https://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L1389C9-L1389C4020:47
TheJuliabut only so it doesn't do it again20:47
JayFlet me see, I didn't save full logs am waiting for reproduction (or not) to happen20:48
JayFit's been hung a while on init_neutron, I think I may have finally worn this machine out lol20:49
JayFoh it broke free, woo20:51
JayFyeah it's not far enough to break yet, I think20:51
JayFokay, lib/ironic ran, now neutron is running20:55
JayFand now it breaks20:55
TheJuliadid it restart neutron?20:57
JayFhttps://home.jvf.cc/~jay/devstacklog.txt20:58
JayFcreate_neutron_initial_network got called in stack.sh20:58
JayFin that method is where it goes boom20:58
JayFdo I need to set NEUTRON_CREATE_INITIAL_NETWORKS=False ? 20:59
JayFthat can't be right20:59
TheJuliaeh, it has to be done once20:59
TheJuliabut it can't be done twice20:59
JayFbut it is doing twice20:59
JayFbut only for me, and not for gate21:00
JayFseemingly21:00
JayFI'm going to rebuild this from scratch and see if it makes a diff21:03
JayFwell, maybe just set that and I get a working one21:03
JayFI'm going to go that route21:03
JayFhmm 21:04
JayFoh, I think I have it, maybe?21:04
TheJuliaso no IP_VERSION... in your log21:04
JayF++lib/neutron_plugins/services/l3:create_neutron_initial_network:196  oscwrap --os-cloud devstack --os-region RegionOne network create private -f value -c id21:04
JayFnote that's coming from the l3 plugin21:04
JayFwhich would not be skipped on second run, because that's a separate flag21:05
JayFnope, that's wrong21:05
JayFthey are managed by the same flag21:05
JayFand l3 is the default21:05
JayFso this is just default21:05
JayFTheJulia: what does that mean?21:05
JayFTheJulia: this is a v4-only box21:05
JayFTheJulia: does that require a specific config?21:05
TheJuliasearch your log, no IP_VERSION, we should have a variable there21:06
TheJuliaI'd explicitly set it to IP_VERSION=4 then21:06
JayFso TheJulia interesting thing21:06
JayFthere are no networks, at least not for network list21:06
JayFeven as admin21:07
JayFoh this makes sense21:07
JayFI should go read the neutron code21:07
JayFI bet it's possible we: 1) assume v6 works and 2) blow up with bad values when it doesn't21:07
mmalchukfolks, what's wrong with bifrost CI ? too many timeouts21:08
JayFmmalchuk: there is a known issue right now, we're waiting for a fixed systemd package to come into centos stream 921:08
mmalchukTheJulia, JayF hi21:08
JayFmmalchuk: rpittau is tracking it, I'd look for comments from him tomorrow as to if it's filtered down to us21:08
mmalchukJayF oh... I need to stop recheck and go to bed)21:08
TheJuliasleep++21:09
JayFmmalchuk: generally speaking, there should never be a need to recheck repeatedly -- if you see a problem come ask21:09
mmalchuksometimes it help, but youre right) nevermind21:10
JayFI mean, you're right too :D 21:10
JayFit can help sometimes21:10
JayFbut we have limited CI resources in the cloud so it's a good idea to only recheck if you know something is fixed or the reason it broke21:10
JayFalso I think we all wanna know anyway if something is busted :D 21:11
JayFTheJulia: I think I'm right21:11
JayFTheJulia: default IP_VERSION=4+621:11
JayFand I never make my VMs v6 compatible21:11
TheJuliahmm21:12
* JayF cries in anguish21:12
mmalchukirc is lack some kind of topic... for example CI status)21:12
TheJuliainteresting my browser didn't find IP_VERSION in the log21:12
JayFmmalchuk: That's usually in the status whiteboard there21:12
JayFmmalchuk: but  honestly we don't do a good job of it21:12
JayFTheJulia: I grepped for the default since I knew I didn't set it21:12
TheJuliammalchuk: you say the current topic odesn't appear?21:12
JayFTheJulia: there's no logic to set it properly21:12
TheJuliammalchuk: to your client?21:12
mmalchuki see.21:12
TheJuliaokay, because when there is not, that is a solid sign of the irc servers being out of sync21:13
mmalchukTheJulia yep21:13
* TheJulia once had a global o line on an irc network many moons ago21:13
JayFmmalchuk: the whiteboard is updated about bifrost CI, but I won't pretend it was set before :D 21:13
mmalchukTheJulia how r u? leave RH?21:14
TheJuliastill at RH21:14
mmalchukwow21:15
mmalchukseen email21:15
TheJulia... what email?21:15
JayFthat's not ominious at all21:15
mmalchukthat you leave21:15
mmalchukam I wrong21:15
TheJuliaI think so, I've not gone anywhere21:16
JayFif Julia is anything like me, she's DM'ing her boss to make sure she still works there 21:16
JayFLOL21:16
TheJulialol21:16
mmalchukcant remember21:16
TheJulianah, I talked to him earlier and put more work on his plate21:16
mmalchuklol21:16
mmalchuknevermind, glad to see you here21:17
mmalchukPierre will update the whiteboard when CI fixed?21:20
mmalchukJayF ?21:20
mmalchukor you will?21:20
JayFrpittau: can you make sure if/when bifrost gate is happy you let mmalchuk know and update the whitebaord21:20
JayFthat's all the power I have ;) 21:20
mmalchukcool21:21
mmalchukJayF he is GMT+3 afaik, along as mee sleep alreay)21:23
JayFof course, but the mirrors move slowly21:23
JayFand he's owned the fix the whole time :)21:23
JayFopenstack is like the tortoise from the fable, we move slow and methodical but get there :D 21:23
mmalchuknice fable)21:24
mmalchukopenstack fable))21:24
JayFTheJulia: IP_VERSION did not change anything21:25
JayFTheJulia: I'm going to assume that's still the problem and go from scratch though21:26
JayFhttps://home.jvf.cc/~jay/local_conf.txt is what I'm going to try21:26
TheJuliaok :(21:27
JayFaight, trying again, this time in script so I have the output if something goes upside down again21:28
* mmalchuk sad because Bifrost CI not happy21:29
JayFthere should be no rush to get your change in friend, it'll be fixed long before release :) 21:29
mmalchuksee you tomorrow folks21:29
JayFo/21:29
mmalchuk\o21:30
JayFoh hell I didn't update the clock on this snapshot lol21:30
JayFaight, that should be happier now lol21:31
JayFTheJulia: thinking more about what we passed over earlier during our chat: do you think there's a world where we could eat less CI resources + get similar coverage with less infra by using not-real-nodes behind Ironic for some of the scenarios?21:45
JayFI guess most of what we're testing *in ironic* is node related, so faking out the nodes sorta just makes it more of a basic ironic<>nova smoke test21:46
TheJuliathe tempest suite uses fake nodes heavily for api integration21:48
TheJuliaI think the key is to elongate and consolidate21:48
TheJuliawe did that with some of the opportunistic changes, let me see if I can find an example in little bit21:49
TheJulia... hmmm21:51
TheJuliawe've got a bunch of different scenarios, but we do lots of setup for them too21:52
JayF+++functions-common:test_with_retry:2333     timeout 20 sh -c 'while ! sudo ip netns exec qdhcp-effb6a1d-07cd-4fb7-9777-c7c370e4df4f ip link list | grep '\'' tap'\'' | cut -d'\'':'\'' -f2 | cut -d'\''@'\'' -f1 | cut -b2-  | grep '\''^tap'\''; do sleep 1; done'21:57
JayFCannot open network namespace "qdhcp-effb6a1d-07cd-4fb7-9777-c7c370e4df4f": No such file or directory21:57
JayFTheJulia: new errs!21:57
TheJuliaso, are you using OVN?21:57
JayFhttps://home.jvf.cc/~jay/local_conf.txt21:58
JayFthat is our doc'd local.conf with additions as discussed at the bottom21:59
JayFthere are no settings around ovs or ovn in local.conf21:59
TheJuliaJayF: I think you need the following:22:07
TheJuliahttps://www.irccloud.com/pastebin/WN1Q9gus/22:07
JayFI created https://etherpad.opendev.org/p/ironic-ptg-october-2023 to allow us to brainstorm topics for the upcoming PTG22:53
iurygregoryRemote Baremetal Deployments using Bifrost https://www.meetup.com/triangle-openstack-meetup/events/295149782/ Tue Aug 22 =) 23:13
iurygregorygood to see people talking about bifrost <323:14
opendevreviewJulia Kreger proposed openstack/ironic master: Adds service steps  https://review.opendev.org/c/openstack/ironic/+/88478323:49
TheJuliaOkay, review away!23:49
TheJuliaI'll work on the other tests/decorating more tomorrow, I've been at it all day23:50

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!