| janders | Bifrost question: what's the purpose of baremetal vs target in https://opendev.org/openstack/bifrost/src/branch/master/playbooks/inventory/group_vars ? | 01:28 |
|---|---|---|
| janders | and more specifically, what is the origin of the image download URL passed to the IPA? | 01:28 |
| janders | context: I installed bifrost with --network-interface set to eth1 and it still tried to use virbr0's 192.168.122.1 IP address, I suspect it's taking it from network_interface but since it's set in multiple places I do not know which one is causing the problem | 01:29 |
| janders | knowing this should help track down why --network-interface install option doesn't seem to take care of this | 01:30 |
| ikoliveira | ODE_UUID: | 04:08 |
| ikoliveira | <ikoliveira> Received heartbeat for node 878c3113-0035-5033-9f99-46520b89b56d with callback URL https://10.7.0.175:9999. This is not expected, and the heartbeat will not be processed.#033[00m | 04:08 |
| ikoliveira | <ikoliveira> May 3 04:01:14 lsd-srv-238 ironic[665280]: 2024-05-03 04:01:14.041 665280 DEBUG ironic.api.method [req-8fcc726a-6850-44d7-a0a5-69d21354f999 req-41ceeb1c-36c8-48ae-9b9e-f9473cf84c27 - - - - - -] Client-side error: Detected change in ramdisk provided "callback_url" format_exception /opt/stack/bifrost/lib/python3.10/site-packages/ironic/api/method.py:124#033[00m | 04:08 |
| ikoliveira | <ikoliveira> May 3 04:01:14 lsd-srv-238 ironic[665280]: 2024-05-03 04:01:14.043 665280 INFO eventlet.wsgi.server [req-8fcc726a-6850-44d7-a0a5-69d21354f999 req-41ceeb1c-36c8-48ae-9b9e-f9473cf84c27 - - - - - -] 10.7.0.175 "POST /v1/heartbeat/878c3113-0035-5033-9f99-46520b89b56d HTTP/1.1" status: 400 len: 483 time: 0.0156338#033[00m | 04:08 |
| ikoliveira | <ikoliveira> Can someone assist me in resolving this issue? I'm uncertain as to why it's occurring. The node's status after deployment shows as "active," yet the boot process doesn't occur. | 04:08 |
| opendevreview | Jacob Anders proposed openstack/bifrost master: Make boot_interface customisable https://review.opendev.org/c/openstack/bifrost/+/917927 | 06:21 |
| rpittau | good morning ironic! happy friday! o/ | 07:10 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Update supported Python versions https://review.opendev.org/c/openstack/bifrost/+/918029 | 08:18 |
| *** mmalchuk_ is now known as mmalchuk | 08:40 | |
| opendevreview | Adam Rozman proposed openstack/ironic-python-agent master: add quiet cleanup option https://review.opendev.org/c/openstack/ironic-python-agent/+/918039 | 11:23 |
| opendevreview | Adam Rozman proposed openstack/ironic-python-agent master: add quiet cleanup option https://review.opendev.org/c/openstack/ironic-python-agent/+/915825 | 11:26 |
| janders | I'm looking for second +2 for https://review.opendev.org/c/openstack/bifrost/+/917927 if anyone has time for a quick review. Thanks in advance! | 11:33 |
| iurygregory | good morning | 11:40 |
| iurygregory | janders, sure I will add to my list to look at it | 11:41 |
| *** lucasagomes_ is now known as lucasagomes | 12:08 | |
| janders | TY iurygregory | 12:16 |
| dtantsur | TheJulia, hey, has anyone in your team reported or heard reports of memory leaks in Inspector? | 12:25 |
| TheJulia | dtantsur: we had someone mention it in irc 2 or 3 years ago. I remember then we looked at the code and what they indicated before leaving irc and thought it might be something with database interaction, but I think that is about where the search ended. | 13:04 |
| dtantsur | I see, I see.. we have a very "nice" graph in https://issues.redhat.com/browse/OCPBUGS-32261 | 13:06 |
| TheJulia | hmm | 13:15 |
| TheJulia | in the past, we knew it to be introspection data in process such that it looked like it was from the db, is that with sqlalchemy 2.0 or pre 2.0? | 13:15 |
| dtantsur | Sorry, did not parse. | 13:16 |
| dtantsur | I don't think this environment has constant (re-)inspection, so I'm more suspicious of the sync loop. | 13:16 |
| TheJulia | given the progression, that aligns with what I am thinking | 13:17 |
| dtantsur | Interestingly, I don't see anything suspicious on my local environment. | 13:19 |
| TheJulia | crazy thought, since the incline is different (drastically) from the original report. What if it is webserver responses... and that is all the process monitoring | 13:38 |
| dtantsur | hmmm | 13:39 |
| opendevreview | Baptiste Jonglez proposed openstack/networking-generic-switch master: Introduce NGS agent design https://review.opendev.org/c/openstack/networking-generic-switch/+/897047 | 13:51 |
| TheJulia | Looking at items in BZ, we've got https://bugs.launchpad.net/ironic/+bug/1595625 and I'm not sure it makes sense to keep open at this point in time. Thoughts? | 14:15 |
| rpittau | TheJulia: I think we can close it | 14:28 |
| TheJulia | I wonder if https://bugs.launchpad.net/ironic/+bug/1686675 is still the case | 14:34 |
| JayF | If we're still setting that environment variable, the bug that works around was fixed literally years ago | 14:38 |
| TheJulia | I think we removed it.... | 14:39 |
| TheJulia | I've not checked yet | 14:39 |
| JayF | Same, I am going to be very scattered in my availability today and I'm not in front of a desktop right now | 14:41 |
| JayF | I'd suggest 1595625 would likely be superceded by runbooks | 14:47 |
| JayF | and in a world where those exist, we'd likely implement it from a different perspective | 14:47 |
| TheJulia | I'd just close it out at this point because at it's age, the possibility of doing so was *always* "just ask for the step you want" | 14:47 |
| TheJulia | ... I have tons of folks that like to just run the step they want for manual cleaning | 14:48 |
| JayF | I think when I read that (is that my RFE? it sounds like something I'd suggest) I was thinking 'automated_clean' as an alias step anywhere we take a step | 14:48 |
| JayF | that subs the automated steps | 14:48 |
| JayF | but that makes much less sense now than it did in 2016 | 14:48 |
| TheJulia | yeah | 14:48 |
| JayF | I think our use case was, for onmetal | 14:49 |
| JayF | wanting to ensure the node went through cleaning after some kinda maintenance was done | 14:49 |
| JayF | might have even been before we did a clean from manageable -> available | 14:49 |
| TheJulia | really, there is no reason one can't create a step which stubs it anyway in a hardware manager | 14:49 |
| JayF | I wontfix'd it with a comment saying if you want this, reopen iwth a use case | 14:51 |
| TheJulia | https://github.com/openstack/ironic/blob/master/ironic/cmd/__init__.py#L22 | 14:51 |
| TheJulia | cool cool | 14:52 |
| TheJulia | Thanks JayF | 14:52 |
| JayF | zap that from orbit | 14:52 |
| JayF | TheJulia: I'm erradicating that before I step out | 14:53 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Remove eventlet workarounds https://review.opendev.org/c/openstack/ironic/+/918082 | 14:57 |
| JayF | beaker on the burner; bbl | 14:58 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Remove eventlet workaround https://review.opendev.org/c/openstack/ironic-python-agent/+/918087 | 15:19 |
| opendevreview | Merged openstack/ironic-python-agent stable/2023.2: Unmount config drives https://review.opendev.org/c/openstack/ironic-python-agent/+/917734 | 15:20 |
| opendevreview | Merged openstack/ironic-python-agent stable/2024.1: Unmount config drives https://review.opendev.org/c/openstack/ironic-python-agent/+/917733 | 15:27 |
| rpittau | bye everyone, have a great weekend! o/ | 15:33 |
| Sandzwerg[m] | hmmm, so https://docs.openstack.org/ironic/latest/user/deploy.html#capabilities mentions that it's possible to set the root-device hint per instance. But at which step? When I try to set this during the deployment (before IPA booted) I get an error that the node can not be updated while it's in a state transition, but after the deployment a root device hint would not make any sense. I tried to pass this as property during the | 15:35 |
| Sandzwerg[m] | server create but that had no effect. And the linked root device hint page doesn't mention this possibility at all. | 15:35 |
| Sandzwerg[m] | hmm setting it before the deployment sets the property but feels strange. Let's see if it has an effect. | 15:38 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/2023.1: ci: stable-only: explicitly pin centos for builds https://review.opendev.org/c/openstack/ironic-python-agent/+/917903 | 15:43 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/2023.1: Unmount config drives https://review.opendev.org/c/openstack/ironic-python-agent/+/917852 | 15:43 |
| TheJulia | hmmmmm | 15:48 |
| TheJulia | Sandzwerg[m]: looks like it is actually a thing, but it might vary based upon how the data goes in, so if you can get the details of your instance_info field that help wrap our brains around it | 15:55 |
| TheJulia | it might be a bug in how data is handled from nova and hten populated into ironic where it was just not "front of mind" then | 15:56 |
| Sandzwerg[m] | <Sandzwerg[m]> "hmm setting it before the..." <- OK that works, but has the issue that I'm not sure on which node I end up. I'm looking for a way to "modify" the root device hint for one customer. I think I need to look into owner/lessee again | 15:57 |
| Sandzwerg[m] | TheJulia: hmkay, so I could open a bug with nova? | 15:57 |
| TheJulia | Sandzwerg[m]: they would just ignore it | 15:57 |
| TheJulia | We would need to see what and how exactly the data gets posted into the field | 15:58 |
| Sandzwerg[m] | hmk | 15:59 |
| TheJulia | Sandzwerg[m]: how did you create it on a flavor? | 16:00 |
| Sandzwerg[m] | on a flavor? | 16:00 |
| TheJulia | yeah | 16:00 |
| TheJulia | Is that not what your trying to do, have a customer specific flavor? | 16:00 |
| Sandzwerg[m] | that might be another way. Right now we have a generic flavor because the hardware is the same and used by multiple customers. But one wants his OS to end up one the (bigger) second disk instead of the smaller first one, But I don't want to change this for all nodes because it would impact the other customers as well. | 16:02 |
| Sandzwerg[m] | And we have the root device hint on node level, not on the flavor. I was not aware that one could set it on the flavor | 16:02 |
| TheJulia | I'm *not* sure it would work, but basically instance_info data gets posted out to instance_info | 16:03 |
| TheJulia | https://github.com/openstack/ironic/blob/master/ironic/drivers/modules/deploy_utils.py#L1505 | 16:04 |
| TheJulia | https://www.irccloud.com/pastebin/lLQvA0n0/ | 16:05 |
| TheJulia | Sandzwerg[m]: example instance_info field from our tempest test, I'm not sure how much the flavor data gets dropped in | 16:06 |
| Sandzwerg[m] | yeah I now used "openstack baremetal node set $NODE_UUID --instance-info root_device='{"size":"> 400"}'" before deploying to that specific node | 16:06 |
| Sandzwerg[m] | Hmm I need to check that next week, that might work as well. My other thought was to use owner/lessee and assign them some nodes for which I could then switch the hint on node level, assuming other customers wouldn't end up on these nodes and they only end up on these nodes | 16:07 |
| Sandzwerg[m] | But using a custom flavor sounds nice. Great Idea | 16:08 |
| TheJulia | enjoy! | 16:08 |
| Sandzwerg[m] | Thanks, need to drop, have a nice weekend \o/ | 16:11 |
| TheJulia | you too | 16:13 |
| TheJulia | hmm, more changes likely needed, time will tell | 16:32 |
| TheJulia | (regarding 2023.1 | 16:32 |
| TheJulia | ) | 16:32 |
| opendevreview | Merged openstack/ironic-lib master: destroy_disk_metadata: support 4096 sector size https://review.opendev.org/c/openstack/ironic-lib/+/917937 | 16:33 |
| JayF | I'd also note, generally, you'll be disappointed often if you look at Ironic docs for what we can do in some use cases (standalone / metal3 / non-nova openstack) versus what gets exposed via nova driver, it's a bit of a lowest common denominator | 17:10 |
| opendevreview | Merged openstack/ironic master: Remove eventlet workarounds https://review.opendev.org/c/openstack/ironic/+/918082 | 17:53 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.1: ci: stable-only: explicitly pin centos build https://review.opendev.org/c/openstack/ironic/+/918118 | 17:56 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/2023.1: Unmount config drives https://review.opendev.org/c/openstack/ironic-python-agent/+/917852 | 17:57 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent stable/2023.1: Unmount config drives https://review.opendev.org/c/openstack/ironic-python-agent/+/917852 | 17:57 |
| TheJulia | I guess lets see if that clears the issue up | 18:05 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: Additional test state fixes https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/918001 | 18:30 |
| iurygregory | any DIB experts around? | 20:06 |
| iurygregory | ikoliveira is having some trouble after deploying a bm node, ironic shows active but the node seems to hang during boot , I think it might be a problem with the image used in the node | 20:06 |
| TheJulia | o/ | 20:21 |
| TheJulia | ikoliveira: Hi! | 20:21 |
| TheJulia | ikoliveira: where is the node hanging in the boot process? | 20:22 |
| ikoliveira | TheJulia, hi! The node loads BIOS and starts boot process, but doesnt starts the operating system after this. Only shows an black screen with an underscore blinking | 20:25 |
| TheJulia | ikoliveira: so, odds are your bootloader has no console= setting | 20:26 |
| TheJulia | you may need to modify the dib element | 20:26 |
| TheJulia | uhhhh | 20:26 |
| TheJulia | do you see the bootloader screen? | 20:28 |
| iurygregory | it doesn't even reach the bootloader i think when he was showing to me | 20:30 |
| ikoliveira | TheJulia, just like iurygregory said, it doesnt appears to me | 20:30 |
| iurygregory | "super micro" logo loading... black screen | 20:31 |
| TheJulia | hmmm | 20:31 |
| iurygregory | I have the feeling the image is broken or something... I told him to try building a centos8 stream to test, using $ disk-image-create centos bootloader vm -o centos8-wholedisk | 20:32 |
| TheJulia | try rebuilding your image with "DIB_BOOTLOADER_VIRTUAL_TERMINAL=tty0" as a environment setting. If you can share the command line that would be helpful. | 20:32 |
| TheJulia | at this point, don't bother with centos8 | 20:32 |
| TheJulia | okay, so.... | 20:32 |
| ikoliveira | hmmm | 20:32 |
| TheJulia | I'd likely do something like "env DIB_BOOTLOADER_VIRTUAL_TERMINAL=tty0 DIB_RELEASE=9-stream disk-image-create centos block-device-efi vm bootloader -o centos9-wholedisk" | 20:33 |
| iurygregory | he is doing legacy boot, since the machine is old if I recall | 20:34 |
| TheJulia | oh | 20:34 |
| TheJulia | uhhhh | 20:34 |
| iurygregory | yeah =X | 20:34 |
| TheJulia | block-device-mbr then :) | 20:34 |
| iurygregory | university doesn't have a lot of money for new hardware | 20:34 |
| ikoliveira | nice, gonna try those ideas. tks a lot!!!!!! | 20:36 |
| opendevreview | Merged openstack/bifrost master: Make boot_interface customisable https://review.opendev.org/c/openstack/bifrost/+/917927 | 20:48 |
| TheJulia | dtantsur: any thoughts on the json_rpc code in ironic lib and allowing "session" to be able to be client input, thinking for mercury to be able to have a separate configuration section because we're talking about a service "another team" owns which means it could have entirely different parameters | 23:26 |
| TheJulia | server wise, should be "good as-is" | 23:28 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!