| mnaser | JayF: wlp, found it finally - https://bugs.launchpad.net/nova/+bug/2106073 + https://review.opendev.org/c/openstack/nova/+/946221 | 00:44 |
|---|---|---|
| mnaser | i think nova gets away with this because it generates the config drive last minute after running the plug_vifs | 00:45 |
| opendevreview | Satoshi Shirosaka proposed openstack/ironic-python-agent master: Ironic Config Intergration for IPA https://review.opendev.org/c/openstack/ironic-python-agent/+/944951 | 00:51 |
| opendevreview | Satoshi Shirosaka proposed openstack/ironic-python-agent master: Implement manual cleaning for ContainerHardwareManager https://review.opendev.org/c/openstack/ironic-python-agent/+/945862 | 00:53 |
| mnaser | i updated it.. it's actually affecting all of nova :) i'll have to think of a bit of a clever solution.. | 02:37 |
| rpittau | good morning ironic! o/ | 07:14 |
| AmarachiOrdor[m] | Good morning !! rpittau | 07:26 |
| freemanboss[m] | rpittau: good morning | 07:45 |
| Ayo[m] | Good morning rpittau | 07:46 |
| rpittau | AmarachiOrdor[m], freemanboss[m], Ayo[m], good morning! | 07:47 |
| queensly[m] | Good morning rpittau , Amarachi Ordor , Ayo Freeman Boss and everyone :) | 07:52 |
| rpittau | hey queensly[m] :) | 07:57 |
| queensly[m] | Hey rpittau quick question, I have worked on the comments I received after submitting the patch yesterday. Would it be best to change the branch name before using the command "git review"?. The change I made was to move contents from howto.rst to testenv.rst and troubleshooting.rst. The initial branch name I used was doc/update howto.rst | 08:12 |
| rpittau | queensly[m]: you can also just submit your changes and then update the topic afterwards in the gerrit ui | 08:14 |
| queensly[m] | rpittau: Alright, I will do that. Thanks. | 08:16 |
| opendevreview | luocanhui proposed openstack/ironic-python-agent master: Run UT of metricis_lib https://review.opendev.org/c/openstack/ironic-python-agent/+/946229 | 08:33 |
| freemanboss[m] | rpittau: the 2025.1 is now working fine I used it yesterday with no restrictions | 08:36 |
| rpittau | freemanboss[m]: great :) | 08:36 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Refactor documentation: Move updated instructions to testenv.rst and troubleshooting to troubleshooting.rst https://review.opendev.org/c/openstack/bifrost/+/946116 | 09:08 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Refactor documentation: Move updated instructions to testenv.rst and troubleshooting to troubleshooting.rst https://review.opendev.org/c/openstack/bifrost/+/946116 | 09:44 |
| freemanboss[m] | masghar: rpittau: I just submitted a Google doc of my second contribution to your mails | 09:44 |
| masghar | freemanboss: ack | 09:59 |
| masghar | *acknowledged | 10:00 |
| freemanboss[m] | masghar: Thank you. | 10:02 |
| opendevreview | Pavlo Shchelokovskyy proposed openstack/ironic master: Fix ISO+GPT image handling https://review.opendev.org/c/openstack/ironic/+/946235 | 10:32 |
| kulsooms | Hi everyone, following up on yesterday's discussion. i deleted everything and was able to perform all the commands (power, manage, and provide). | 10:38 |
| kulsooms | the enrollment command also worked fine but rn i'm facing an issue with the deploy command it’s kind of stuck and doesn't seem to proceed. | 10:38 |
| kulsooms | how long does the deploy command usually take to finish? | 10:38 |
| AmarachiOrdor[m] | Good morning please where is it stuck at | 10:39 |
| AmarachiOrdor[m] | Yeah it takes some time to finish | 10:39 |
| AmarachiOrdor[m] | kulsooms: How long has it been | 10:39 |
| kulsooms | Amarachi: good morning, if you can see this image: https://imgur.com/a/dt7EICt | 10:42 |
| AmarachiOrdor[m] | I can see it | 10:43 |
| AmarachiOrdor[m] | So yeah this part takes some time | 10:44 |
| AmarachiOrdor[m] | Just make sure your internet connection is strong so it doesn't affect the process | 10:45 |
| freemanboss[m] | <kulsooms> "how long does the deploy command..." <- It shouldn't be long | 10:50 |
| freemanboss[m] | <kulsooms> "Amarachi: good morning, if you..." <- Can you run baremetal node list and send screenshot | 10:50 |
| freemanboss[m] | Also have you created your ssh key pairs? | 10:50 |
| freemanboss[m] | freemanboss[m]: > <@freemanboss:matrix.org> Can you run baremetal node list and send screenshot | 10:52 |
| freemanboss[m] | > Also have you created your ssh key pairs? | 10:52 |
| freemanboss[m] | You can just create to terminal tab so you can run baremetal node list kulsooms: | 10:52 |
| freemanboss[m] | <kulsooms> "Hi everyone, following up on..." <- Great. Please before this you actually tried enrolling and it failed? Then perform these steps or went direct to these steps (power, manage and provide)? | 10:54 |
| AmarachiOrdor[m] | Freeman Boss: I think he said it worked fine, it's just the deployment state he is stuck at | 10:58 |
| freemanboss[m] | <AmarachiOrdor[m]> "Freeman Boss: I think he said it..." <- Alright | 11:12 |
| kulsooms | Amarachi: ok, but its still there and my internet connection seems stable | 11:13 |
| kulsooms | freemanboss: yess, i created the ssh keys | 11:14 |
| freemanboss[m] | Alright run the second suggestion I made | 11:15 |
| kulsooms | which one sorry, i'm sending the screenshot | 11:16 |
| freemanboss[m] | <AmarachiOrdor[m]> "Freeman Boss: I think he said it..." <- I'm just trying to understand something in how she overcome it cause I'm testing various ways to troubleshoot errors we do have | 11:16 |
| freemanboss[m] | <freemanboss[m]> "> <@freemanboss:matrix.org..." <- Node list | 11:16 |
| kulsooms | freemanboss: https://imgur.com/a/TZ0sa3f | 11:17 |
| freemanboss[m] | Just control C that running process and redeploy again | 11:27 |
| freemanboss[m] | ./bifrost-cli deploy baremetal-inventory.json @baremetal-install-env.json | 11:27 |
| freemanboss[m] | <kulsooms> "freemanboss: https://imgur.com/a..." <- After running this and you're still stuck one way or the other I'll advise you delete the nodes and re-enroll... (full message at <https://matrix.org/oftc/media/v1/media/download/AURwS-oM-Nl9lTcRuMyEyjNvZ2cI-juWNy37whKwbD4mUMJNco2Y35Ij45SUaPM2Yzm6kDitRTJWZBZISxXt-PVCeWQ4LzLAAG1hdHJpeC5vcmcvQ1hKSUNPTnNwS3FFaEppeXFMYmlRbVR5>) | 11:29 |
| frickler | freemanboss[m]: I know you likely don't notice this in your matrix client, but I would ask you for two things: a) don't use the reply function, simply reply as a normal message b) don't send multiline messages, only one line at a time. both of these translate badly into IRC and makes it very tiresome to follow the conversation | 11:33 |
| frickler | (the same of course also applies to other matrix users) | 11:33 |
| frickler | you may want to check https://meetings.opendev.org/irclogs/%23openstack-ironic/latest.log.html to see how your messages look for others | 11:34 |
| freemanboss[m] | frickler: oh thank you I'll adhere to these | 11:36 |
| frickler | thank you. I hope this didn't sound too negative, it is really great to see so much activity from new people | 11:38 |
| freemanboss[m] | frickler: Not at all. Thank you | 11:41 |
| kulsooms | freemanboss: i was using this "./bifrost-cli deploy baremetal-inventory.json" as the deploy command | 12:02 |
| freemanboss[m] | kulsooms: that's where the issue came from | 12:04 |
| freemanboss[m] | During the ./bifrost-cli install --testenv a file which consists all the required variables for deployment was created. You can use this | 12:04 |
| freemanboss[m] | ./bifrost-cli deploy baremetal-inventory.json @baremetal-install-env.json | 12:04 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Move updated instructions to testenv.rst and troubleshooting to troubleshooting.rst https://review.opendev.org/c/openstack/bifrost/+/946116 | 12:18 |
| iurygregory | good morning ironic | 12:24 |
| iurygregory | just to double check, but ilo4 doesn't support redfish virtual media right? it started in ilo5 if I recall.. | 12:25 |
| kulsooms | freemanboss: https://imgur.com/a/Gy5j0B7 | 12:27 |
| kulsooms | then I tried "./bifrost-cli deploy -e @baremetal-install-env.json" and this is the result, i'm assuming the nodes are deployed see: https://imgur.com/a/nHgDVxg | 12:28 |
| freemanboss[m] | kulsooms: great that's a success | 12:30 |
| freemanboss[m] | The active state there means it's been successfully deployed the os to the intended nodes (testvm1 and testvm2) | 12:30 |
| kulsooms | yaay :')) | 12:31 |
| kulsooms | freemanboss, amarachi, ayo, queensly : thank you guys for you time :)) much appreciated | 12:32 |
| freemanboss[m] | kulsooms: you're welcome | 12:33 |
| queensly[m] | well done kulsooms 🎉 | 12:34 |
| AmarachiOrdor[m] | Well done kulsooms | 12:35 |
| TheJulia | good morning | 13:08 |
| rpittau | iurygregory: ilo4 does have support for redfish virtual media, although it's not greatly implemented and it requires a separate license to activate it | 13:15 |
| iurygregory | rpittau, oh interesting | 13:15 |
| rpittau | Open Infra Live incoming in 13 minutes https://www.youtube.com/watch?v=qX2-8oyFVPk :) | 13:47 |
| opendevreview | Merged openstack/sushy-tools master: os-vmedia: Add option to delay rebuild on eject https://review.opendev.org/c/openstack/sushy-tools/+/945800 | 14:29 |
| mnaser | alright so | 14:52 |
| mnaser | nova + ironic + routed provider networks + bonds / configdrive = boom | 14:52 |
| mnaser | i need some help in how we can figure out how to solve this | 14:52 |
| mnaser | does ironic do the actual port binding? | 14:53 |
| mnaser | context: https://bugs.launchpad.net/nova/+bug/2106073 | 14:53 |
| TheJulia | yes, ironic does the actual bind because your required to have unique macs in neutron | 14:54 |
| TheJulia | and we still have to do the actual binding/work to deploy the host overall | 14:54 |
| mnaser | TheJulia: right yea, so with routed networks, the ip address doesn't show up until it's bound, and since ironic does the bind, the configdrive that nova generates is incorrect | 15:02 |
| mnaser | because it has no ips there (yet) | 15:02 |
| TheJulia | well, that makes sense | 15:02 |
| mnaser | and i'm struggling to find a decent fix :( | 15:02 |
| TheJulia | ouch | 15:03 |
| TheJulia | so... hmm | 15:03 |
| mnaser | cause bonds so i need configdrive | 15:03 |
| TheJulia | there may not be a fix for this | 15:03 |
| mnaser | it might be super stinky but i might just have/need to "rehydrate" the configdrive in ironic :\ | 15:03 |
| TheJulia | because realistically, if we're relying on neutron to ipam, and we can't get an IP on the interface allocated until the final bind, then the chicken has already hatched | 15:03 |
| TheJulia | you can't even do that | 15:03 |
| TheJulia | Because that point is past deployment | 15:04 |
| TheJulia | we've had some discussions of doing actions like rewriting the network data in the config drive | 15:04 |
| TheJulia | so mentally changing gears for a moment | 15:04 |
| TheJulia | These are bonds, the network interfaces are flat right? | 15:04 |
| mnaser | yes | 15:04 |
| TheJulia | okay | 15:05 |
| TheJulia | yeah, static networking | 15:05 |
| TheJulia | so the way it works is we get a vif | 15:05 |
| TheJulia | so we bind the vif, but we don't do that until *after* provisioning has started | 15:06 |
| TheJulia | so still no IP | 15:06 |
| TheJulia | ewwwwww | 15:06 |
| TheJulia | so... eww | 15:06 |
| mnaser | technically shouldn't the logic be bind => provision ? | 15:06 |
| TheJulia | The only way to fix this is for ironic to rewrite the embedded network metadata which is something I recently discussed with JayF | 15:07 |
| mnaser | so question, doesn't the nova virt manager bind the port when it creates it by default? | 15:08 |
| TheJulia | well, the logic is: 1) record the vif ( we expect the port to have an ip and all metadata, and nova's config metadata code has that same belief/expectation as well) 2) start provision, regardless of the network interface we take that vif port and "attach" that to the provisioning network with is separate | 15:08 |
| TheJulia | we would need to trigger that bind, upfront to the target network for deploy... I guess | 15:08 |
| JayF | Does "do the binding" imply the interface is up? | 15:09 |
| TheJulia | mnaser: through the driver vif_attach call | 15:09 |
| JayF | No, correct? | 15:09 |
| TheJulia | which just records it with Ironic because we can't provision on untrusted networks | 15:09 |
| TheJulia | JayF: in routed network contexts, yes apparently | 15:09 |
| mnaser | do the binding implies that $service has told neutron where the port needs to be bound | 15:09 |
| mnaser | and in routed networks, that's when neutron ipam can action and know which ip to give out | 15:09 |
| TheJulia | mnaser: in workflow, the *final* bind is right before handing off the node after the config drive is written | 15:09 |
| mnaser | because based on the binding, it can make sure it gives it an ip that fits the segment | 15:09 |
| mnaser | TheJulia: correct, yes, but i'm wondering if we can get away with the initial nova bind somehow | 15:10 |
| mnaser | if nova does set the binding:host_id in the manager | 15:10 |
| TheJulia | mnaser: there is no initial nova bind | 15:10 |
| mnaser | ah | 15:10 |
| TheJulia | it is a vif_attach action to the virt driver | 15:10 |
| mnaser | ah right, which is a noop in ironic world | 15:11 |
| TheJulia | basically, yeah | 15:11 |
| JayF | and especially if for routed networks, binding means onlining the port in some way, we can't do that pre-provisioning | 15:11 |
| JayF | without potential security implications | 15:11 |
| TheJulia | I guess we might be able to do it, but then we have to tear it down or change the MAC | 15:11 |
| TheJulia | which... might be doable | 15:11 |
| mnaser | the mac gets changed anyways | 15:11 |
| mnaser | many times, from the logs i saw :) | 15:11 |
| TheJulia | mnaser: can you confirm what your network_interface value is set to? | 15:12 |
| JayF | ultimately this is a chicken/egg where either us or neutron will have to do something earlier than we're comfy with | 15:12 |
| TheJulia | because if your seeing the mac change, that *sounds* like the neutron network_interface is in use | 15:12 |
| JayF | routed networks + ngs is what mnaser said | 15:12 |
| JayF | so I've assumed neutron the whole way | 15:12 |
| mnaser | yes, sorry, it's neutron, but "flat" network | 15:12 |
| TheJulia | There you go | 15:12 |
| TheJulia | there is your problem | 15:12 |
| TheJulia | are you doing someting like v6? | 15:12 |
| mnaser | nope | 15:13 |
| TheJulia | or are these v4 networks? | 15:13 |
| mnaser | plain v4 | 15:13 |
| TheJulia | try flat instead of neutron | 15:13 |
| mnaser | just curious how is that going to make any differences in terms of the configdrive still not having the right info? :X | 15:13 |
| TheJulia | flat is *actually* the interface you want for a flat network | 15:13 |
| mnaser | i mean, yes, i agree, we can switch it to the right vlaue | 15:14 |
| TheJulia | mnaser: a datapoint since the workflow is different under the hood and the port is not reset a bunch of times | 15:14 |
| TheJulia | so in essence, you do get earlier binding or at least should. | 15:14 |
| JayF | I'm still curious if this will happen in complex networks with a similar setup using neutron | 15:14 |
| JayF | because if so this bug still exists in my future :) | 15:14 |
| mnaser | but i think so long as nova doesn't do it, you'll get that out of date network_info | 15:14 |
| TheJulia | at least in flat, ther eis also a hope ironic might be abel to do some config metadata updates if we're just shipped the joson dicts | 15:15 |
| TheJulia | I think I need to pull the code up on a call to discuss this, can we jump on a call in 15? | 15:15 |
| mnaser | i dont mind :) | 15:16 |
| TheJulia | JayF: of sorts yes, but because the ip assignment with neutron appears delayed | 15:16 |
| mnaser | i think this will have to involve some sort of hack on the nova driver side | 15:16 |
| JayF | I should be able to get on a call in 15 | 15:16 |
| mnaser | for me since i use flat, i can probably get away with doing the binding there and refreshing network_info | 15:17 |
| TheJulia | https://meet.google.com/ovk-edth-amp | 15:30 |
| opendevreview | Merged openstack/ironic stable/2025.1: [2025.1 only] update devstack config https://review.opendev.org/c/openstack/ironic/+/945793 | 16:13 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Move updated instructions to testenv.rst and troubleshooting to troubleshooting.rst https://review.opendev.org/c/openstack/bifrost/+/946116 | 16:47 |
| freemanboss[m] | rpittau: masghar: cid: please while creating a patch can I actually upload pictures too? I'm trying to create a patch on series of errors I/we faced when installing bifrost and the quickest way to troubleshoot them. | 17:44 |
| JayF | freemanboss[m]: you can use etherpad.opendev.org as a good place to put output that's persistent and lets other people comment on them | 17:45 |
| JayF | freemanboss[m]: if you need to use images; you can but will need to find your own host and just link them in -- but really if you can get them to someone in text-based form that'll be most helpful to them | 17:45 |
| freemanboss[m] | JayF: alright thank you | 17:49 |
| freemanboss[m] | JayF: but I really don't understand getting them in text-based form? | 17:49 |
| JayF | FYI: we root-caused the unittest failures in IPA; basically the test was counting how many times we went through a loop by counting time.time() executions -- LOG.warning being one of them. New python 3.13 uses time.time_ns() in logging instead of time.time() | 17:50 |
| JayF | freemanboss[m]: like if it's an error log, copy+paste it into a place like an etherpad or pastebin | 17:50 |
| JayF | freemanboss[m]: most of the output we need is all in a text terminal typically, so copy+paste without passing thru an image makes it searchable | 17:50 |
| freemanboss[m] | JayF: I'm to write on the chat section right? Also any way to use markdowns? | 17:57 |
| freemanboss[m] | JayF: oh I understand I'll try the copy pasting rather. | 17:58 |
| JayF | freemanboss[m]: just make yourself a new etherpad.opendev.org and use the main text area there | 18:06 |
| opendevreview | Satoshi Shirosaka proposed openstack/ironic-python-agent master: Fix test_timeout on Python3.13 https://review.opendev.org/c/openstack/ironic-python-agent/+/946290 | 18:07 |
| JayF | freemanboss[m]: almost nobody use the chat feature on etherpaad | 18:07 |
| freemanboss[m] | Ohhh I should just type on that blank spaces JayF: | 18:08 |
| cid | Yep, exactly. | 18:12 |
| freemanboss[m] | https://etherpad.opendev.org/p/Troubleshooting | 18:21 |
| freemanboss[m] | JayF: cid: masghar: rpittau: this is the link. Thank you. I'll still use markdowns and upload images on this since it's etherpad | 18:21 |
| cid | freemanboss[m], Cool \o/ | 18:24 |
| freemanboss[m] | https://etherpad.opendev.org/p/Quick_start_with_Bifrost. This is another one. Thank you. cid: | 18:25 |
| cid | ++ | 18:26 |
| opendevreview | Julia Kreger proposed openstack/ironic-specs master: WIP: Trait based port selection and dynamic portgroups https://review.opendev.org/c/openstack/ironic-specs/+/945642 | 18:52 |
| TheJulia | JayF: mnaser: ^ | 18:57 |
| TheJulia | vauge on the details, but the basic problem | 18:58 |
| TheJulia | and why we need to fix it | 18:59 |
| JayF | ++ thank you | 19:23 |
| iurygregory | tyvm supermicro for following the redfish standard /s | 21:34 |
| JayF | does anyone have like, 5 minutes to help sanity check something for me? | 21:37 |
| JayF | I'm trying to fix a requirements thing, with new pysnmp (frickler mentioned this last week) | 21:38 |
| JayF | https://github.com/lextudio/pysnmp is supposedly the upstream, and the version numbers match | 21:38 |
| JayF | but I have a LOT of files installed in site-packages/pysnmp that do not appear to be in the repo | 21:38 |
| JayF | including the one Ironic imports, that fails: pysnmp.hlapi -- which doesn't have a __init__.py in the repo | 21:39 |
| JayF | and that imports a module that no longer exists in pyasn1 | 21:39 |
| iurygregory | oh wow O.o | 21:40 |
| JayF | I mainly want someone else to observe this | 21:40 |
| iurygregory | my rpittau has some ideas, (if you can wait till tomorrow) | 21:40 |
| JayF | and ensure I'm not missing something or dealing with something weird and environmental | 21:40 |
| JayF | well, I think the outcome will be "file an upstream bug" if I can just get someone else to confirm this weird state | 21:41 |
| iurygregory | JayF, if you share the steps I can try later today (going to the gym to forget about the problems supermicro gave me today) | 21:41 |
| JayF | If people want snmp driver to keep existing, they need to step up QUICK | 21:41 |
| JayF | 21ab5a8e3a947c213603b9fe22ac53117833fc29 it was removed in this but is still in the pypi package | 21:44 |
| JayF | https://github.com/lextudio/pysnmp/commit/31953bc487148852e2eef774125a02028adabdef | 21:46 |
| JayF | they seriously are shipping files in pypi that are not in the repo | 21:47 |
| JayF | yikes this is rough | 21:47 |
| opendevreview | Merged openstack/ironic-python-agent master: Fix test_timeout on Python3.13 https://review.opendev.org/c/openstack/ironic-python-agent/+/946290 | 21:47 |
| JayF | https://github.com/lextudio/pysnmp/issues/175 yikes on bikes for sure | 21:57 |
| JayF | YIKES | 21:57 |
| JayF | ^ https://github.com/lextudio/pysnmp/issues/175#issuecomment-2777209717 no, I was wrong, someone else helped me find out the pattern | 23:31 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!