| *** akahat is now known as akahat|rover | 05:29 | |
| arne_wiebalck | Good morning, Ironic! | 07:26 |
|---|---|---|
| rpittau | good morning ironic! Happy Friday! o/ | 08:07 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: [DNM] Test tinycore13-based tinyipa https://review.opendev.org/c/openstack/ironic-python-agent/+/827664 | 08:27 |
| *** akahat|rover is now known as akahat|lunch | 08:30 | |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Fix prepare ramdisk for 'wait' states https://review.opendev.org/c/openstack/ironic/+/823311 | 08:42 |
| *** akahat|lunch is now known as akahat|rover | 09:22 | |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Fix Redfish RAID for non-immediate controllers https://review.opendev.org/c/openstack/ironic/+/823312 | 09:35 |
| opendevreview | Merged openstack/ironic bugfix/19.0: Fix resource_url in the remaining resources https://review.opendev.org/c/openstack/ironic/+/827732 | 10:10 |
| opendevreview | Merged openstack/ironic stable/xena: Fix resource_url in the remaining resources https://review.opendev.org/c/openstack/ironic/+/827731 | 10:16 |
| janders | good morning arne_wiebalck rpittau and Ironic o/ | 10:36 |
| rpittau | hey janders :) | 10:36 |
| arne_wiebalck | hey janders o/ | 10:36 |
| * dtantsur finally remembers to open the IRC client | 10:43 | |
| rpittau | well I'm happy I can reproduce the issue with dipipa jobs in bifrost and I can confirm netifs do not get ip or are even woken up -> https://paste.openstack.org/show/812520/ | 11:09 |
| dtantsur | rpittau: is cloud-init even running there? | 11:31 |
| dtantsur | I somewhat suspect our network data format in bifrost is so ancient that it is no longer recognized | 11:31 |
| rpittau | I don't think it's running, but I doubt the problem is on bifrost | 11:32 |
| dtantsur | well, maybe we need to add cloud-init OR simple-init explicitly to the image? | 11:34 |
| dtantsur | is glean running there? | 11:34 |
| rpittau | no, this uses dhcp-all-interfaces | 11:34 |
| rpittau | it stops working when we changed from buster to bullseye | 11:34 |
| dtantsur | hmm | 11:35 |
| rpittau | when the old naming scheme changed to the new predictable one for th netifs | 11:35 |
| rpittau | udev change the names at some point, maybe it confuses the script | 11:35 |
| iurygregory | good morning Ironic o/ | 11:36 |
| rpittau | hey iurygregory :) | 11:36 |
| dtantsur | heyhey iurygregory | 11:36 |
| iurygregory | o// | 11:36 |
| rpittau | time for carbonara and wine, I'll check later, alcohol will help | 11:38 |
| dtantsur | +++ | 11:39 |
| dtantsur | I should have put baileys in my coffee | 11:39 |
| * dtantsur exercising brb | 11:42 | |
| janders | I feel we jumped the gun with a homemade lasagne and a glass of wine a couple hours back :) Gotta love Fridays :) | 12:30 |
| janders | good morning iurygregory o/ | 12:31 |
| iurygregory | hey janders o/ | 12:32 |
| iurygregory | homemade lasagne yummy | 12:32 |
| TheJulia | talk of alcohol so early.... | 13:11 |
| TheJulia | I'm worried | 13:11 |
| iurygregory | depends on your reference for time... | 13:13 |
| TheJulia | touche | 13:13 |
| iurygregory | my reference is always Tokyo <3 so it's 10pm | 13:13 |
| TheJulia | arne_wiebalck: the inspector issue you ran into. Was the issue that the request was not known? are they using a central mysql db or are they just doing local db configuraiton? | 13:23 |
| TheJulia | oh... I think I see what is going on :\ | 13:31 |
| arne_wiebalck | TheJulia: let me get you the exact error | 13:42 |
| dtantsur | seeking opinions: building our own (debian?) partition image for the CI | 13:51 |
| * TheJulia shrugs | 13:51 | |
| dtantsur | we're in a CI deadlock now. we cannot test whole disk images with network boot on UEFI. we cannot test partition images with local boot at all. | 13:52 |
| TheJulia | so we need a partition image with a bootloader | 13:53 |
| TheJulia | makes sense | 13:53 |
| rpittau | I wonder if we will hit the same issue as with debian ipa ramdisk, but probably not as we'll have simple-init | 14:04 |
| dtantsur | rpittau: I think we'll even put cloud-init there | 14:05 |
| rpittau | oh well then no problem | 14:05 |
| dtantsur | but I need to check how quickly the image can be built and how large it is | 14:05 |
| rpittau | btw I'm getting closer to a solution, running dhcp-all-interfaces and ifup manually in the instance works, so it's probably a race condition | 14:06 |
| * TheJulia wonders if https://bugs.launchpad.net/ironic-inspector/+bug/1473024 is still a thing | 14:17 | |
| dtantsur | TheJulia: we have batching, so it's hard to say | 14:20 |
| TheJulia | I guess arne's test will tell us if we need to revisit that fix | 14:21 |
| dtantsur | okay, so 2.5 minutes, the image is 270M | 14:22 |
| dtantsur | it's probably acceptable for the CI | 14:22 |
| TheJulia | dtantsur: ++ | 14:23 |
| dtantsur | on a positive side, we'll be able to remove several jobs | 14:37 |
| TheJulia | \o/ | 14:40 |
| TheJulia | I think I'm going to do my meeting in about 20 minutes and go have a real breakfast before digging into nova's code this morning | 14:40 |
| rpittau | bye everyone, have a great weekend! o/ | 15:23 |
| *** akahat|rover is now known as akahat|dinner | 15:45 | |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: CI: use a custom debian partition image instead of cirros https://review.opendev.org/c/openstack/ironic/+/827882 | 15:54 |
| dtantsur | an experiment ^^^ | 15:54 |
| opendevreview | Bob Fournier proposed openstack/sushy-tools master: Accept non-string types for BIOS settings https://review.opendev.org/c/openstack/sushy-tools/+/827769 | 16:12 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: CI: use a custom debian partition image instead of cirros https://review.opendev.org/c/openstack/ironic/+/827882 | 17:22 |
| dtantsur | have a great weekend folks o/ | 17:23 |
| iurygregory | bye dtantsur o/ | 17:28 |
| arne_wiebalck | bye everyone, see you next week o/ | 17:28 |
| iurygregory | bye arne_wiebalck o/ | 17:32 |
| NobodyCam | Good Morning Ironic'ers | 17:58 |
| NobodyCam | and ofc | 17:58 |
| NobodyCam | TGIF | 17:58 |
| *** akahat|dinner is now known as akahat|rover | 17:59 | |
| iurygregory | morning NobodyCam o/ TGIF \o/ | 18:03 |
| NobodyCam | hey hey iurygregory happy Friday | 18:10 |
| TheJulia | o/ NobodyCam | 18:15 |
| NobodyCam | Happy Friday TheJulia | 18:15 |
| TheJulia | rloo: zer0c00l: either of you around? | 19:27 |
| rloo | hi TheJulia. I am (not zer0c001) | 19:28 |
| rloo | and getting rounder too ;) | 19:28 |
| TheJulia | lol | 19:28 |
| TheJulia | I shouldn't laugh | 19:28 |
| TheJulia | but I am | 19:28 |
| opendevreview | Merged openstack/ironic stable/rocky: Cleanup stable/rocky legacy jobs https://review.opendev.org/c/openstack/ironic/+/827709 | 19:28 |
| rloo | ha ha | 19:28 |
| TheJulia | so I've got someone at rh trying to use the anaconda deploy interface | 19:29 |
| TheJulia | and they ran into an issue | 19:29 |
| rloo | oh oh. there is more than one issue. | 19:29 |
| TheJulia | and I think I need to take away beer from the "due to zer0c00l" column | 19:29 |
| TheJulia | well, okay | 19:29 |
| rloo | thinking. can they wait til eg mid next week? | 19:29 |
| rloo | don't think i can do it today, but i can try to push up the fixes early next week. are they using master branch? | 19:29 |
| TheJulia | I'll try tossing up a patch | 19:30 |
| TheJulia | I *think* I see what is going on | 19:30 |
| TheJulia | xena, and it is the same in wallaby | 19:30 |
| rloo | there is more than one issue/fix... | 19:30 |
| TheJulia | tl;dr we're creating a directory | 19:30 |
| TheJulia | node_uuid/LiveOS/squashfs.img | 19:30 |
| TheJulia | as a directoryu | 19:30 |
| rloo | yeah, i think the squashfs was in the wrong place or something. | 19:31 |
| rloo | ok, give me an hour, and i'll dig through the changes. we haven't yet fixed why we can't ssh but that is possibly downstream stuff, and i didn't want to have to push more than one pr upstream so i've been waiting. but i don't want others to be blocked. | 19:32 |
| rloo | (or 2 hours, just have to finish something and remember how to do upstream work...) | 19:32 |
| TheJulia | okay :) | 19:32 |
| TheJulia | super appreciated! | 19:32 |
| * TheJulia provides provides tea and tasty bread | 19:33 | |
| TheJulia | well, virtual | 19:33 |
| rloo | thx! that'll help speed me up ;) | 19:33 |
| sdanni | Hi! I'm booting a baremetal node with ipxe and get the following error: | 19:46 |
| sdanni | inc: command not found | 19:46 |
| sdanni | pxelinux.cfg/xxx. No such file or directory | 19:46 |
| sdanni | echo PXE boot failed! No configuration found for NIC <xxx>. | 19:46 |
| sdanni | echo Please update your iPXE ROM and retry. | 19:46 |
| sdanni | echo Press any key to reboot... | 19:46 |
| sdanni | is it because the ipxe rom is too old and I need to update it? | 19:46 |
| JayF | Is this an issue with a single node, or with all/most nodes in your deployment? | 19:50 |
| JayF | If it's an issue with a single node, it's extremely likely there's a typo with the mac address of the host in question (I think it's on the Ironic port associated with the node) | 19:50 |
| JayF | If it's an issue with many/all, it's probably a config issue and I'm not super equipped to help with that right now (but others are) | 19:50 |
| JayF | good luck | 19:50 |
| sdanni | JayF: It's an issue with all nodes (3 nodes) | 19:52 |
| TheJulia | sdanni: mellenox nics? | 20:13 |
| sdanni | TheJulia: they are solarflare ethernet nics (10Gig) | 20:24 |
| TheJulia | they *sound* like they have very old and broken ipxe firmware integrated | 20:24 |
| TheJulia | inc is a really good hint in this case | 20:25 |
| TheJulia | and boot.ipxe is actually returning that message saying as such | 20:25 |
| sdanni | TheJulia: yeah, I saw the boot.ipxe file has an error about old rom boot failure. Is it possible to update the ipxe rom? | 20:26 |
| TheJulia | most cards yes, but you likely want to engage the vendor | 20:27 |
| JayF | There are also ways of making the DHCP config detect it's an outdated iPXE and send it an updated pxe rom anyway | 20:27 |
| JayF | I did that at a previous job, and don't remember exactly what I did? I think you can actually ask an ipxe rom about if it supports a modern feature using DHCP client options it sends | 20:28 |
| JayF | rloo: ^^ If you have the time, and wanna gist the relevant section of the dhcp config I modified to fix AIO back when we worked together, it might be helpful for sdanni | 20:28 |
| JayF | https://forum.ipxe.org/showthread.php?tid=21715 this is getting at the idea -- but IDK what featueres are in the embedded ipxe vs the one Ironic sends, so you'd have to figure that part out | 20:29 |
| rloo | JayF: huh? that sounds vaguely familiar. not sure i'll have time to get to that, in the middle of trying to write a pr and can't multitask... will see... :) | 20:29 |
| sdanni | JayF: thanks! | 20:30 |
| JayF | rloo: yeah, I'm mainly annoyed that it's basically, tip-of-the-tongue the exact thing to do, and I can't remember it :-| | 20:30 |
| TheJulia | we do it in bifrost | 20:30 |
| TheJulia | or at least did a long time ago | 20:30 |
| JayF | https://gist.github.com/robinsmidsrod/4008017 is a really good example, too | 20:31 |
| TheJulia | but in that case, it was only when the firmware was doing something super visably bad | 20:31 |
| JayF | In the case I was referencing, we used a qemu VM to test code, kinda like a local devstacky thing | 20:31 |
| JayF | and qemu on modern rhel changed to always use ipxe, which didn't work b/c we were using https, so we had to test for https feature, then if it didn't have it, send the custom ipxe rom over | 20:32 |
| sdanni | JayF: the two links you posted are both not examples to update ipxe with ironic? | 20:35 |
| JayF | sdanni: Hm. You'll have to find some way to modify the DHCP config that Ironic sets up. I don't remember how to do that :| | 20:40 |
| JayF | I also don't know if it's even possible with dnsmasq, but probably is since TheJulia said it was possible | 20:40 |
| JayF | (or attempted at least in bifrost) | 20:40 |
| sdanni | Got it. Thanks! | 20:42 |
| JayF | sdanni: Sorry to be so vague! I'm a bit of an... uh, emeritus contributor? So my knowledge is slowly fading :-( | 20:42 |
| TheJulia | its semi-possible with dnsmasq directly | 20:43 |
| TheJulia | but not neutron integrated | 20:43 |
| TheJulia | this is basically a case of you have a bunch of cards which walk/talk like ipxe, but can't grok 7 year old ipxe commands | 20:44 |
| opendevreview | Ruby Loo proposed openstack/ironic master: More fixes for anaconda deploy interface https://review.opendev.org/c/openstack/ironic/+/827924 | 21:28 |
| rloo | TheJulia: ^^. It might be missing some unit tests and I didn't test on master branch. Also, there are some fixes related to configdrive, which I will do in a separate PR. (because configdrive is in xena, and we're using wallaby and i'd like that ^^ to be backported to wallaby). | 21:30 |
| TheJulia | rloo: absolutely | 21:30 |
| TheJulia | rloo: that looks pretty good, I'm basically happy to merge that if CI passes as is. It does appear to fix the observed issue but I've shared the patch with the downstream user | 21:36 |
| rloo | guess i better do the 2nd pr. not sure how useful it is w/o config drive... | 21:37 |
| TheJulia | okay, I'll do a close review over the weekend. I've reached ENOSPOON a bit earlier than I had hoped today | 21:38 |
| rloo | thx. your downstream user could maybe test it out for us ;) | 21:39 |
| TheJulia | rloo: they indicated they would try to give any patch a spin | 21:45 |
| TheJulia | but likely on monday | 21:45 |
| rloo | sounds good. thx! i might have time to do the configdrive one today too, so they could try both if they are using configdrives (and if the code gets that far!) | 21:46 |
| opendevreview | Ruby Loo proposed openstack/ironic master: Anaconda deploy handles configdrive correctly https://review.opendev.org/c/openstack/ironic/+/827933 | 22:18 |
| rloo | TheJulia: ^^ part 2. And that's it for me today. | 22:19 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!