| rpittau | good morning ironic! o/ | 06:42 |
|---|---|---|
| arne_wiebalck | Good morning rpittau and Ironic! | 06:45 |
| rpittau | hey arne_wiebalck :) | 06:46 |
| dtantsur | morning folks | 08:02 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix names of two jobs https://review.opendev.org/c/openstack/ironic/+/838661 | 08:48 |
| opendevreview | Riccardo Pittau proposed openstack/sushy master: Update jobs names https://review.opendev.org/c/openstack/sushy/+/838662 | 08:50 |
| opendevreview | Riccardo Pittau proposed openstack/sushy master: Update jobs names https://review.opendev.org/c/openstack/sushy/+/838662 | 08:52 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix names of two jobs https://review.opendev.org/c/openstack/ironic/+/838661 | 08:53 |
| opendevreview | Riccardo Pittau proposed openstack/sushy-tools master: Update jobs names https://review.opendev.org/c/openstack/sushy-tools/+/838664 | 08:56 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Fix names of two jobs https://review.opendev.org/c/openstack/ironic/+/838661 | 08:56 |
| dmellado | morning all, forgot to turn on the irc | 09:26 |
| opendevreview | Riccardo Pittau proposed openstack/sushy-tools master: Use python Zed tests https://review.opendev.org/c/openstack/sushy-tools/+/838674 | 09:44 |
| dtantsur | dmellado: I know this feeling. sometimes I forget to turn on slack and keep wondering why the morning is so calm | 09:53 |
| dmellado | dtantsur: heh, totally | 09:55 |
| dmellado | even so 'turn on' is just loading the glowing-bear website | 09:55 |
| dmellado | xD | 09:56 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-python-agent-builder stable/wallaby: Add multipath-io element https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/838549 | 10:47 |
| iurygregory | good morning | 10:48 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-python-agent-builder stable/yoga: Add multipath-io element https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/838690 | 10:48 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-python-agent-builder stable/xena: Add multipath-io element https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/838691 | 10:48 |
| iurygregory | HUH?! Could not perform action: Branch refs/heads/bugfix/8.3 does not exist. O.o | 10:51 |
| dtantsur | wtf | 11:16 |
| iurygregory | yeah, I had this reaction also... | 11:28 |
| iurygregory | it used to be possible to do cherry-pick via UI to bugfix branches right? (Or I'm going crazy...) | 11:28 |
| iurygregory | oh god more meetings in my calendar <O> /me needs more coffee today | 11:30 |
| dtantsur | iurygregory: it was definitely possible a while ago | 11:37 |
| iurygregory | I'm going to try to cherry-pick manually just to see... | 11:37 |
| iurygregory | ok, I was sleepy.... | 11:44 |
| * iurygregory shouldn't open gerrit at 7am | 11:48 | |
| dtantsur | definitely | 11:50 |
| dtantsur | and avoid slack | 11:50 |
| iurygregory | oh, this one I couldn't a lot of message while I was off :D | 11:51 |
| iurygregory | and very urgent ones.. | 11:52 |
| opendevreview | Aija Jauntēva proposed openstack/sushy-oem-idrac stable/yoga: Revert "Update .gitreview after project rename" https://review.opendev.org/c/openstack/sushy-oem-idrac/+/838692 | 12:11 |
| opendevreview | Aija Jauntēva proposed openstack/sushy-oem-idrac stable/xena: Revert "Update .gitreview after project rename" https://review.opendev.org/c/openstack/sushy-oem-idrac/+/838693 | 12:12 |
| ftarasenko | hi, Ironic! Can someone point me to bug with idrac when I get next error while cleaning (raid creation) - Node is in manageable state; should be in one of wait call-back, deploying. Details https://paste.openstack.org/show/bAUHcgWjjjaDQzl6CyiR/ | 12:17 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-python-agent master: Multipath Hardware path handling https://review.opendev.org/c/openstack/ironic-python-agent/+/837039 | 12:17 |
| dtantsur | ajya: do you remember the error ^^^? I *think* it rings some bells, there was something about cleaning reporting.. or maybe it was in iRMC? | 12:27 |
| dmellado | iurygregory: gerrit at 7am is deff a terrible idea xD | 12:31 |
| dmellado | there should be a -1 not_enough_coffee_yet | 12:32 |
| ajya | dtantsur don't recall seeing this before. | 12:32 |
| dtantsur | ftarasenko: do you have https://opendev.org/openstack/ironic/commit/5e4a617b46c713aba762cc74cc0ea0c64c84a0c3? | 12:32 |
| iurygregory | dmellado, yeah! | 12:33 |
| dtantsur | this commit is from year 2020, but who knows | 12:34 |
| dmellado | well, that's a really bad year | 12:34 |
| dmellado | drop it | 12:34 |
| dmellado | dtantsur: | 12:34 |
| dmellado | xD | 12:34 |
| dtantsur | true :) | 12:34 |
| ftarasenko | dtantsur: yep, my W has this commit. | 12:34 |
| dtantsur | ftarasenko: could you grab more logs from this node? from the beginning of cleaning? | 12:35 |
| ftarasenko | dtantsur: Sure. https://paste.openstack.org/show/bSHcKxSmuW4KmMSF4HXI/ | 12:41 |
| ftarasenko | dtantsur: Do you need debug print? It's much bigger, need to share whole file. | 12:43 |
| dtantsur | ftarasenko: I'm looking at what I have now, but I may need debug logs indeed | 12:43 |
| dtantsur | the order of actions is absurd | 12:43 |
| ajya | ftarasenko: so it's indeed cleaning and in logs it says it can't continue deploying? Somehow jumps from cleaning to deploy code? | 12:48 |
| ajya | could be some recent-ish changes around async tasks that causes this | 12:49 |
| dtantsur | ajya: it seems that the continue_polling flag is ignored or not set properly, and the node finishes cleaning on the next heartbeat | 12:49 |
| ftarasenko | dtantsur: ajya: I see that cleaning is finished and have raid ready. but for some reason it says that it wants to start cleaning again. IDRAC is latest one, 5.10.10.00 , I can downgrade to check if it's connected. | 12:51 |
| ftarasenko | Also I have log file with debug output to share. Is google drive ok to share file? | 12:52 |
| TheJulia | good morning | 12:53 |
| ajya | ftarasenko: does it happen with all nodes/all the time? Or from time to time? At the moment, I don't think iDRAC version impacts this. Suspecting something to do with idrac driver cleaning steps. | 12:53 |
| ftarasenko | ajya: it happens from time to time. For example, today I had one node cleaned from the first time, one from second attempt, two nodes from third, and last one still is not cleaned. | 12:56 |
| ajya | thanks, I could try to reproduce this on my systems unless dtantsur sees why this is happening | 12:57 |
| TheJulia | dtantsur: I think irmc, but I think I spotted something in storyboard at one point that might have been exactly this, but we got no details in the bug report | 12:58 |
| ftarasenko | ajya: tnx. it's R350 with integrated H355 controller. I just try to build RAID with 2 drives. | 12:59 |
| TheJulia | dtantsur: the irmc folks are also the only ones that did storage integration with fiberchannel | 12:59 |
| dtantsur | *nod* | 13:00 |
| dtantsur | good morning TheJulia | 13:00 |
| iurygregory | good morning TheJulia | 13:00 |
| rpittau | good morning TheJulia :) | 13:00 |
| dtantsur | ajya, ftarasenko, the fact that it only happens sometimes, can be explained by a race between heartbeats and the background polling in the driver | 13:00 |
| ajya | dtantsur: yup and it seems often enough that it should have been seen previously, could there be something that changed here creating this race on Ironic side? | 13:05 |
| dtantsur | possibly? | 13:06 |
| ajya | maybe something also on idrac steps, have to check | 13:06 |
| TheJulia | dtantsur: truthfully active/passive setups seem quite rare to me at this point, but it can be policy enforced for purposes of A or B storage fabric failover as well. Out of all of the SANs I've managed in my career, only two did hard blocking active/passive by default but some others it could be configured in for A/B fabric scenarios | 13:07 |
| ajya | ftarasenko: to confirm - it's W release and for raid interface it is idrac-wsman and not idrac-redfish? | 13:07 |
| ftarasenko | ajya: that's correct. idrac-wsman and W. Should I retry with idrac-redfish? | 13:08 |
| ajya | ftarasenko: not needed, I'll check that | 13:13 |
| TheJulia | Anyone having issues with github? | 14:30 |
| TheJulia | nvmd, just loaded on my 4th try | 14:30 |
| TheJulia | iurygregory: any estimate on when any sort of summary will go up, trying to create some downstream stories to track work for steve, harald, and I | 14:33 |
| TheJulia | i guess I can just use the etherpad | 14:34 |
| iurygregory | TheJulia, I should have sent yesterday.. but the customer issue took my whole day basically, I wrote part of it, let me just try to get it done before going for lunch o/ | 14:36 |
| TheJulia | no worries, thanks | 14:42 |
| dtantsur | iurygregory: hey, when is our next bugfix release due? | 15:29 |
| iurygregory | dtantsur, interesting that you asked me after I was looking at an email about downstream releases =) | 15:31 |
| dtantsur | iurygregory: it's absolutely not a coincidence :) | 15:31 |
| iurygregory | dtantsur, I was thinking first week of May after the email... | 15:32 |
| dtantsur | we shouldn't plan upstream business around downstream targets.. but if they can line up, that will be great | 15:32 |
| iurygregory | yeah | 15:32 |
| iurygregory | I will try to see if it will be possible to sync things =) | 15:33 |
| iurygregory | so we don't have to break our heads :D | 15:33 |
| dtantsur | cc rpittau | 15:33 |
| rpittau | some unexpected events created an interesting alignment | 15:44 |
| rpittau | we may be able to avoid some headaches | 15:46 |
| iurygregory | we would need before FF downstream right? | 15:46 |
| rpittau | yep | 15:46 |
| dtantsur | or slightly after, if we're ready to do the dirty thing *innocently whistles* | 15:47 |
| rpittau | :( | 15:47 |
| iurygregory | OMG | 15:47 |
| iurygregory | XD | 15:47 |
| dtantsur | see you tomorrow o/ | 16:03 |
| romer | Hello Everyone ! | 16:04 |
| rpittau | bye! o/ | 17:02 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-python-agent stable/wallaby: Multipath Hardware path handling https://review.opendev.org/c/openstack/ironic-python-agent/+/837784 | 17:29 |
| TheJulia | where oh where did I put my brain... | 17:30 |
| admiyo | So, as I learn a little bit about Kolla and Ironic etc, I found it worthwhile to take notes. For any Ironics that are wondering about the docker side of things, maybe this will be useful: http://adam.younglogic.com/2022/04/how-do-you-keep-the-kolla-playing/ | 19:38 |
| admiyo | What do we call people that work on Ironic? I was a Keystoner back in the day.... | 19:39 |
| JayF | I think we're all a bunch of Bare Metal bears, right? | 19:39 |
| * rm_work waves at JayF | 19:43 | |
| JayF | rm_work: o/ | 19:45 |
| JayF | rm_work: they haven't taken away my keys to the joint, wanna go hide out and goof off in the secret cores only party room? /s | 19:46 |
| rm_work | lol | 19:47 |
| TheJulia | NobodyCam: please queue up the bar tender, we'll need tasty beverages :) | 19:56 |
| JayF | **coors only party room /s | 19:56 |
| NobodyCam | wait what | 19:56 |
| TheJulia | lol | 19:57 |
| * TheJulia hates edk2 | 19:57 | |
| NobodyCam | Good Afternoon Ironic folks | 19:57 |
| admiyo | I have to figure out how to get a custom built ipxe.efi with the right drivers into the ipxe container. For aarch64. | 19:58 |
| TheJulia | admiyo: yay! | 19:58 |
| admiyo | Yeah, I got bifrost to work | 19:59 |
| NobodyCam | \o/ | 19:59 |
| admiyo | I did it cutting all the corners and compiling all the tech debt | 19:59 |
| TheJulia | admiyo: generally, I've use "Ironicans" or "Followers of Irony" | 20:00 |
| TheJulia | Which reminds me actually | 20:00 |
| TheJulia | I'm thinking of pixie boots pins | 20:00 |
| admiyo | Base on Mo Duffy's comic? | 20:00 |
| NobodyCam | :take_my_money: | 20:01 |
| JayF | admiyo: TheJulia: Fun story, the word for a group of us is a "Jagged Little Pill" | 20:01 |
| TheJulia | admiyo: Mo Duffy's comic? | 20:02 |
| admiyo | https://i1.wp.com/duffy.fedorapeople.org/blog/drawings/summit2010comics/colonelkernel.png | 20:02 |
| TheJulia | admiyo: oh my | 20:02 |
| JayF | at least they didn't have to use a ramfs in the boot sequence | 20:03 |
| TheJulia | oh jeeze | 20:04 |
| TheJulia | so... one of my coworkers hands me a partition image which is not working, and I want to reproduce the boot | 20:04 |
| TheJulia | so I realize... no deployment. Cool, I can fix the image, I know the magic | 20:04 |
| TheJulia | and I go to do that | 20:04 |
| TheJulia | and edk2 doesn't see the esp partition | 20:04 |
| TheJulia | well | 20:05 |
| TheJulia | It sees it, but says there are zero files | 20:05 |
| TheJulia | so of course, it just won't boot now | 20:05 |
| admiyo | Maybe there really are 0 files in there. | 20:05 |
| TheJulia | take base file before the qcow, and magically there does happen to be files in it when I loopback mount it | 20:06 |
| JayF | TheJulia: I was helping a guy in #gentoo yesterday who had a laptop with a self-encrypting drive that basically was randomly encrypting and decrypting itself (so basically once every "N" boots it stayed locked, and his sda appeared to the OS as a 32k device with no partitions) | 20:06 |
| TheJulia | JayF: yeouch | 20:06 |
| JayF | TheJulia: I'm like "I know things about disk wiping, and you need to throw that disk off a cliff" | 20:07 |
| JayF | lol | 20:07 |
| JayF | TheJulia: loopback mount like, qemu-nbd? | 20:07 |
| TheJulia | loopback raw file | 20:07 |
| TheJulia | with gpt partitions | 20:07 |
| JayF | That's very strange. | 20:07 |
| TheJulia | which EDK2 sees! | 20:07 |
| JayF | So you said this is a partition image, not a whole disk image | 20:08 |
| JayF | yes? | 20:08 |
| JayF | So why does it have partitions? | 20:08 |
| TheJulia | well, I converted it to be a whole disk image | 20:08 |
| TheJulia | and no joy | 20:08 |
| * TheJulia mounts the whole disk image up, mounts hte filesystems, and extracts the files to figure out the headache the harder way | 20:09 | |
| TheJulia | which might just be easier | 20:09 |
| TheJulia | oh joy | 20:11 |
| admiyo | Anyw, to make the AARCH64 ipxe process work, I had to clone https://github.com/ipxe/ipxe.git | 20:15 |
| admiyo | cd ipxe/src/ | 20:15 |
| admiyo | make bin-arm64-efi/snponly.efi ARCH=arm64 | 20:15 |
| admiyo | sudo cp src/bin-arm64-efi/snponly.efi /var/lib/tftpboot/ipxe.efi | 20:15 |
| TheJulia | I feel like I did that ages ago | 20:16 |
| admiyo | I am thinking that the right solution is to get a binary posted somewhere, and always download it | 20:16 |
| admiyo | the dnsmasq config should switch on the arch flag and return a different binary. Not sure how to properly name things. And...I don't think that snponly is the way to go, though it worked for me | 20:17 |
| TheJulia | ++ I would save it someplace and just re-use, tbh | 20:18 |
| TheJulia | admiyo: are you doing only arm or arm and x86_64? | 20:19 |
| admiyo | RIght now only arm | 20:22 |
| admiyo | But I want the solution in upstream | 20:22 |
| admiyo | Even the most recent package from Ubuntu does not have a functional ipxe.efi | 20:22 |
| admiyo | and it does not make sense to limit the ironic server to boot only the devices of the same architecture | 20:23 |
| admiyo | For Kolla, the ipxe files are mounted from the directory where they are installed via the .deb file. | 20:24 |
| admiyo | Are there other cases of binaries being pulled in by openstack projects that are not in packages for the underlying distros? | 20:32 |
| opendevreview | Merged openstack/sushy-oem-idrac stable/yoga: Revert "Update .gitreview after project rename" https://review.opendev.org/c/openstack/sushy-oem-idrac/+/838692 | 20:49 |
| TheJulia | I feel like... for arm specifically, a lot of folks have just kind of had to go down the painful path in the past | 20:51 |
| TheJulia | and yeah, we need to collect all of that and fix it | 20:51 |
| opendevreview | Merged openstack/sushy-oem-idrac stable/xena: Revert "Update .gitreview after project rename" https://review.opendev.org/c/openstack/sushy-oem-idrac/+/838693 | 20:52 |
| admiyo | So...I could try and get the snp and snponly versions of the ipxe.efi built in the Debian and Ubuntu packages, and then use the unstable versions until they are marked stable. User could fetch the non-local-arch version of the packages, unpackathem, and use those files. That seems like the long path. I'm just wondering what the shorter but acceptable path would be. | 21:37 |
| rpioso | dtantsur: Thank you for your message about DHCP-less. I believe I am all set for now. | 21:38 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
