Thursday, 2025-04-03

mnaserJayF: wlp, found it finally - https://bugs.launchpad.net/nova/+bug/2106073 + https://review.opendev.org/c/openstack/nova/+/94622100:44
mnaseri think nova gets away with this because it generates the config drive last minute after running the plug_vifs00:45
opendevreviewSatoshi Shirosaka proposed openstack/ironic-python-agent master: Ironic Config Intergration for IPA  https://review.opendev.org/c/openstack/ironic-python-agent/+/94495100:51
opendevreviewSatoshi Shirosaka proposed openstack/ironic-python-agent master: Implement manual cleaning for ContainerHardwareManager  https://review.opendev.org/c/openstack/ironic-python-agent/+/94586200:53
mnaseri updated it.. it's actually affecting all of nova :) i'll have to think of a bit of a clever solution..02:37
rpittaugood morning ironic! o/07:14
AmarachiOrdor[m]Good morning !! rpittau 07:26
freemanboss[m]rpittau: good morning07:45
Ayo[m]Good morning rpittau 07:46
rpittauAmarachiOrdor[m], freemanboss[m], Ayo[m], good morning!07:47
queensly[m]Good morning rpittau , Amarachi Ordor , Ayo Freeman Boss  and everyone :)07:52
rpittauhey queensly[m] :)07:57
queensly[m]Hey rpittau quick question, I have worked on the comments I received after submitting the patch yesterday. Would it be best to change the branch name before using the command "git review"?. The change I made was to move contents from howto.rst to testenv.rst and troubleshooting.rst. The initial branch name I used was doc/update howto.rst 08:12
rpittauqueensly[m]: you can also just submit your changes and then update the topic afterwards in the gerrit ui08:14
queensly[m]rpittau: Alright, I will do that. Thanks.08:16
opendevreviewluocanhui proposed openstack/ironic-python-agent master: Run UT of metricis_lib  https://review.opendev.org/c/openstack/ironic-python-agent/+/94622908:33
freemanboss[m]rpittau: the 2025.1 is now working fine I used it yesterday with no restrictions 08:36
rpittaufreemanboss[m]: great :)08:36
opendevreviewQueensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Refactor documentation: Move updated instructions to testenv.rst and troubleshooting to troubleshooting.rst  https://review.opendev.org/c/openstack/bifrost/+/94611609:08
opendevreviewQueensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Refactor documentation: Move updated instructions to testenv.rst and troubleshooting to troubleshooting.rst  https://review.opendev.org/c/openstack/bifrost/+/94611609:44
freemanboss[m]masghar: rpittau: I just submitted a Google doc of my second contribution to your mails 09:44
masgharfreemanboss: ack09:59
masghar*acknowledged10:00
freemanboss[m]masghar: Thank you.10:02
opendevreviewPavlo Shchelokovskyy proposed openstack/ironic master: Fix ISO+GPT image handling  https://review.opendev.org/c/openstack/ironic/+/94623510:32
kulsoomsHi everyone, following up on yesterday's discussion. i deleted everything and was able to perform all the commands (power, manage, and provide).10:38
kulsoomsthe enrollment command also worked fine but rn i'm facing an issue with the deploy command itโ€™s kind of stuck and doesn't seem to proceed. 10:38
kulsoomshow long does the deploy command usually take to finish?10:38
AmarachiOrdor[m]Good morning please where is it stuck at10:39
AmarachiOrdor[m]Yeah it takes some time to finish10:39
AmarachiOrdor[m]kulsooms: How long has it been 10:39
kulsoomsAmarachi: good morning, if you can see this image: https://imgur.com/a/dt7EICt10:42
AmarachiOrdor[m]I can see it10:43
AmarachiOrdor[m]So yeah this part takes some time10:44
AmarachiOrdor[m]Just make sure your internet connection is strong so it doesn't affect the process10:45
freemanboss[m]<kulsooms> "how long does the deploy command..." <- It shouldn't be long 10:50
freemanboss[m]<kulsooms> "Amarachi: good morning, if you..." <- Can you run baremetal node list and send screenshot 10:50
freemanboss[m]Also have you created your ssh key pairs?10:50
freemanboss[m]freemanboss[m]: > <@freemanboss:matrix.org> Can you run baremetal node list and send screenshot 10:52
freemanboss[m]> Also have you created your ssh key pairs?10:52
freemanboss[m]You can just create to terminal tab so you can run baremetal node list kulsooms: 10:52
freemanboss[m]<kulsooms> "Hi everyone, following up on..." <- Great. Please before this you actually tried enrolling and it failed? Then perform these steps or went direct to these steps (power, manage and provide)?10:54
AmarachiOrdor[m]Freeman Boss: I think he said it worked fine, it's just the deployment state he is stuck at10:58
freemanboss[m]<AmarachiOrdor[m]> "Freeman Boss: I think he said it..." <- Alright11:12
kulsoomsAmarachi: ok, but its still there and my internet connection seems stable11:13
kulsoomsfreemanboss: yess, i created the ssh keys 11:14
freemanboss[m]Alright run the second suggestion I made11:15
kulsoomswhich one sorry, i'm sending the screenshot11:16
freemanboss[m]<AmarachiOrdor[m]> "Freeman Boss: I think he said it..." <- I'm just trying to understand something in how she overcome it cause I'm testing various ways to troubleshoot errors we do have11:16
freemanboss[m]<freemanboss[m]> "> <@freemanboss:matrix.org..." <- Node list 11:16
kulsoomsfreemanboss: https://imgur.com/a/TZ0sa3f11:17
freemanboss[m]Just control C that running process and redeploy again11:27
freemanboss[m]./bifrost-cli deploy baremetal-inventory.json @baremetal-install-env.json11:27
freemanboss[m]<kulsooms> "freemanboss: https://imgur.com/a..." <- After running this and you're still stuck one way or the other I'll advise you delete the nodes and re-enroll... (full message at <https://matrix.org/oftc/media/v1/media/download/AURwS-oM-Nl9lTcRuMyEyjNvZ2cI-juWNy37whKwbD4mUMJNco2Y35Ij45SUaPM2Yzm6kDitRTJWZBZISxXt-PVCeWQ4LzLAAG1hdHJpeC5vcmcvQ1hKSUNPTnNwS3FFaEppeXFMYmlRbVR5>)11:29
fricklerfreemanboss[m]: I know you likely don't notice this in your matrix client, but I would ask you for two things: a) don't use the reply function, simply reply as a normal message b) don't send multiline messages, only one line at a time. both of these translate badly into IRC and makes it very tiresome to follow the conversation11:33
frickler(the same of course also applies to other matrix users)11:33
frickleryou may want to check https://meetings.opendev.org/irclogs/%23openstack-ironic/latest.log.html to see how your messages look for others11:34
freemanboss[m]frickler: oh thank you I'll adhere to these11:36
fricklerthank you. I hope this didn't sound too negative, it is really great to see so much activity from new people11:38
freemanboss[m]frickler: Not at all. Thank you11:41
kulsoomsfreemanboss: i was using this "./bifrost-cli deploy baremetal-inventory.json" as the deploy command12:02
freemanboss[m]kulsooms: that's where the issue came from12:04
freemanboss[m]During the ./bifrost-cli install --testenv a file which consists all the required variables for deployment was created. You can use this12:04
freemanboss[m]./bifrost-cli deploy baremetal-inventory.json @baremetal-install-env.json12:04
opendevreviewQueensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Move updated instructions to testenv.rst and troubleshooting to troubleshooting.rst  https://review.opendev.org/c/openstack/bifrost/+/94611612:18
iurygregorygood morning ironic12:24
iurygregoryjust to double check, but ilo4 doesn't support redfish virtual media right? it started in ilo5 if I recall..12:25
kulsoomsfreemanboss: https://imgur.com/a/Gy5j0B7 12:27
kulsoomsthen I tried "./bifrost-cli deploy -e @baremetal-install-env.json" and this is the result, i'm assuming the nodes are deployed see: https://imgur.com/a/nHgDVxg12:28
freemanboss[m]kulsooms: great that's a success 12:30
freemanboss[m]The active state there means it's been successfully deployed the os to the intended nodes (testvm1 and testvm2)12:30
kulsoomsyaay :'))12:31
kulsoomsfreemanboss, amarachi, ayo, queensly : thank you guys for you time :)) much appreciated12:32
freemanboss[m]kulsooms: you're welcome 12:33
queensly[m]well done kulsooms ๐ŸŽ‰12:34
AmarachiOrdor[m]Well done kulsooms 12:35
TheJuliagood morning13:08
rpittauiurygregory: ilo4 does have support for redfish virtual media, although it's not greatly implemented and it requires a separate license to activate it 13:15
iurygregoryrpittau, oh interesting13:15
rpittauOpen Infra Live incoming in 13 minutes https://www.youtube.com/watch?v=qX2-8oyFVPk :)13:47
opendevreviewMerged openstack/sushy-tools master: os-vmedia: Add option to delay rebuild on eject  https://review.opendev.org/c/openstack/sushy-tools/+/94580014:29
mnaseralright so14:52
mnasernova + ironic + routed provider networks + bonds / configdrive = boom14:52
mnaseri need some help in how we can figure out how to solve this14:52
mnaserdoes ironic do the actual port binding?14:53
mnasercontext: https://bugs.launchpad.net/nova/+bug/210607314:53
TheJuliayes, ironic does the actual bind because your required to have unique macs in neutron14:54
TheJuliaand we still have to do the actual binding/work to deploy the host overall14:54
mnaserTheJulia: right yea, so with routed networks, the ip address doesn't show up until it's bound, and since ironic does the bind, the configdrive that nova generates is incorrect15:02
mnaserbecause it has no ips there (yet)15:02
TheJuliawell, that makes sense15:02
mnaserand i'm struggling to find a decent fix :(15:02
TheJuliaouch15:03
TheJuliaso... hmm15:03
mnasercause bonds so i need configdrive15:03
TheJuliathere may not be a fix for this15:03
mnaserit might be super stinky but i might just have/need to "rehydrate" the configdrive in ironic :\15:03
TheJuliabecause realistically, if we're relying on neutron to ipam, and we can't get an IP on the interface allocated until the final bind, then the chicken has already hatched15:03
TheJuliayou can't even do that15:03
TheJuliaBecause that point is past deployment15:04
TheJuliawe've had some discussions of doing actions like rewriting the network data in the config drive15:04
TheJuliaso mentally changing gears for a moment15:04
TheJuliaThese are bonds, the network interfaces are flat right?15:04
mnaseryes15:04
TheJuliaokay15:05
TheJuliayeah, static networking15:05
TheJuliaso the way it works is we get a vif15:05
TheJuliaso we bind the vif, but we don't do that until *after* provisioning has started15:06
TheJuliaso still no IP15:06
TheJuliaewwwwww15:06
TheJuliaso... eww15:06
mnasertechnically shouldn't the logic be bind => provision ?15:06
TheJuliaThe only way to fix this is for ironic to rewrite the embedded network metadata which is something I recently discussed with JayF15:07
mnaserso question, doesn't the nova virt manager bind the port when it creates it by default?15:08
TheJuliawell, the logic is: 1) record the vif ( we expect the port to have an ip and all metadata, and nova's config metadata code has that same belief/expectation as well) 2) start provision, regardless of the network interface we take that vif port and "attach" that to the provisioning network with is separate 15:08
TheJuliawe would need to trigger that bind, upfront to the target network for deploy... I guess15:08
JayFDoes "do the binding" imply the interface is up?15:09
TheJuliamnaser: through the driver vif_attach call15:09
JayFNo, correct?15:09
TheJuliawhich just records it with Ironic because we can't provision on untrusted networks15:09
TheJuliaJayF: in routed network contexts, yes apparently15:09
mnaserdo the binding implies that $service has told neutron where the port needs to be bound15:09
mnaserand in routed networks, that's when neutron ipam can action and know which ip to give out15:09
TheJuliamnaser: in workflow, the *final* bind is right before handing off the node after the config drive is written15:09
mnaserbecause based on the binding, it can make sure it gives it an ip that fits the segment15:09
mnaserTheJulia: correct, yes, but i'm wondering if we can get away with the initial nova bind somehow15:10
mnaserif nova does set the binding:host_id in the manager15:10
TheJuliamnaser: there is no initial nova bind15:10
mnaserah15:10
TheJuliait is a vif_attach action to the virt driver15:10
mnaserah right, which is a noop in ironic world15:11
TheJuliabasically, yeah15:11
JayFand especially if for routed networks, binding means onlining the port in some way, we can't do that pre-provisioning 15:11
JayFwithout potential security implications15:11
TheJuliaI guess we might be able to do it, but then we have to tear it down or change the MAC15:11
TheJuliawhich... might be doable15:11
mnaserthe mac gets changed anyways15:11
mnasermany times, from the logs i saw :)15:11
TheJuliamnaser: can you confirm what your network_interface value is set to?15:12
JayFultimately this is a chicken/egg where either us or neutron will have to do something earlier than we're comfy with15:12
TheJuliabecause if your seeing the mac change, that *sounds* like the neutron network_interface is in use15:12
JayFrouted networks + ngs is what mnaser said15:12
JayFso I've assumed neutron the whole way15:12
mnaseryes, sorry, it's neutron, but "flat" network15:12
TheJuliaThere you go15:12
TheJuliathere is your problem15:12
TheJuliaare you doing someting like v6?15:12
mnasernope15:13
TheJuliaor are these v4 networks?15:13
mnaserplain v415:13
TheJuliatry flat instead of neutron15:13
mnaserjust curious how is that going to make any differences in terms of the configdrive still not having the right info? :X15:13
TheJuliaflat is *actually* the interface you want for a flat network15:13
mnaseri mean, yes, i agree, we can switch it to the right vlaue15:14
TheJuliamnaser: a datapoint since the workflow is different under the hood and the port is not reset a bunch of times15:14
TheJuliaso in essence, you do get earlier binding or at least should.15:14
JayFI'm still curious if this will happen in complex networks with a similar setup using neutron15:14
JayFbecause if so this bug still exists in my future :)15:14
mnaserbut i think so long as nova doesn't do it, you'll get that out of date network_info15:14
TheJuliaat least in flat, ther eis also a hope ironic might be abel to do some config metadata updates if  we're just shipped the joson dicts15:15
TheJuliaI think I need to pull the code up on a call to discuss this, can we jump on a call in 15?15:15
mnaseri dont mind :)15:16
TheJuliaJayF: of sorts yes, but because the ip assignment with neutron appears delayed15:16
mnaseri think this will have to involve some sort of hack on the nova driver side15:16
JayFI should be able to get on a call in 1515:16
mnaserfor me since i use flat, i can probably get away with doing the binding there and refreshing network_info15:17
TheJuliahttps://meet.google.com/ovk-edth-amp15:30
opendevreviewMerged openstack/ironic stable/2025.1: [2025.1 only] update devstack config  https://review.opendev.org/c/openstack/ironic/+/94579316:13
opendevreviewQueensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Move updated instructions to testenv.rst and troubleshooting to troubleshooting.rst  https://review.opendev.org/c/openstack/bifrost/+/94611616:47
freemanboss[m]rpittau: masghar: cid: please while creating a patch can I actually upload pictures too? I'm trying to create a patch on series of errors I/we faced when installing bifrost and the quickest way to troubleshoot them.17:44
JayFfreemanboss[m]: you can use etherpad.opendev.org as a good place to put output that's persistent and lets other people comment on them17:45
JayFfreemanboss[m]: if you need to use images; you can but will need to find your own host and just link them in -- but really if you can get them to someone in text-based form that'll be most helpful to them17:45
freemanboss[m]JayF: alright thank you 17:49
freemanboss[m]JayF: but I really don't understand getting them in text-based form?17:49
JayFFYI: we root-caused the unittest failures in IPA; basically the test was counting how many times we went through a loop by counting time.time() executions -- LOG.warning being one of them. New python 3.13 uses time.time_ns() in logging instead of time.time()17:50
JayFfreemanboss[m]: like if it's an error log, copy+paste it into a place like an etherpad or pastebin17:50
JayFfreemanboss[m]: most of the output we need is all in a text terminal typically, so copy+paste without passing thru an image makes it searchable17:50
freemanboss[m]JayF: I'm to write on the chat section right? Also any way to use markdowns?17:57
freemanboss[m]JayF: oh I understand I'll try the copy pasting rather.17:58
JayFfreemanboss[m]: just make yourself a new etherpad.opendev.org and use the main text area there18:06
opendevreviewSatoshi Shirosaka proposed openstack/ironic-python-agent master: Fix test_timeout on Python3.13  https://review.opendev.org/c/openstack/ironic-python-agent/+/94629018:07
JayFfreemanboss[m]: almost nobody use the chat feature on etherpaad18:07
freemanboss[m]Ohhh I should just type on that blank spaces JayF: 18:08
cidYep, exactly.18:12
freemanboss[m]https://etherpad.opendev.org/p/Troubleshooting18:21
freemanboss[m]JayF: cid: masghar: rpittau: this is the link. Thank you. I'll still use markdowns and upload images on this since it's etherpad18:21
cidfreemanboss[m], Cool \o/18:24
freemanboss[m]https://etherpad.opendev.org/p/Quick_start_with_Bifrost.   This is another one. Thank you. cid: 18:25
cid++18:26
opendevreviewJulia Kreger proposed openstack/ironic-specs master: WIP: Trait based port selection and dynamic portgroups  https://review.opendev.org/c/openstack/ironic-specs/+/94564218:52
TheJuliaJayF: mnaser: ^18:57
TheJuliavauge on the details, but the basic problem18:58
TheJuliaand why we need to fix it18:59
JayF++ thank you19:23

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!