| *** akahat is now known as akahat|ruck | 04:37 | |
| arne_wiebalck | Good morning, Ironic! | 06:22 |
|---|---|---|
| arozman | Good morning Ironic! | 07:08 |
| rpittau | good morning ironic! o/ | 08:34 |
| dtantsur | lecris[m]: given your comment about older vs newer hardware, I suspect the firmware either fails to parse the DHCP options or fails to request the image correctly. | 10:27 |
| dtantsur | RRQ "snponly.efiM-}^DM-@M-(^ELy^N M-)M-~M-)M-~M-@M-(^Ed" is garbage | 10:28 |
| lecris[m] | But it is puzzling that ironic_inspector works fine | 10:28 |
| lecris[m] | Would it be sender or receiver issue? | 10:29 |
| dtantsur | it's curious indeed | 10:30 |
| dtantsur | both use dnsmasq, just different instances of it | 10:30 |
| dtantsur | lecris[m]: if you increase verbosity, you may be able to see what exactly dnsmasq sends in its options | 10:31 |
| arozman | Hi all! If you have time I would really appreciate some feedback for this IPA change https://review.opendev.org/c/openstack/ironic-python-agent/+/855866 . | 10:31 |
| lecris[m] | One difference is one is tagged the other isn't | 10:31 |
| dtantsur | arozman: I still think we should default to lsblk rather than doing this. | 10:32 |
| arozman | dtantsur I made the lsblk default | 10:32 |
| dtantsur | I'm not hardly against working around bugs in kernel/udev/wherever, but we need to put some effort into figuring out the situation still | 10:32 |
| lecris[m] | dtantsur: Is it possible to do that for nova agents? i.e. by editing opts file? | 10:32 |
| dtantsur | arozman: I still see if prop.__contains__("SERIAL"): | 10:33 |
| dtantsur | maybe you got a mix of the two approaches? | 10:33 |
| arozman | dtantsur the udev stuff is just an additional fallback logic | 10:33 |
| dtantsur | arozman: have you tried filing a bug first? | 10:33 |
| dtantsur | as I said, while I'm not against working around issues in other components, we shouldn't do it by default without trying anything else | 10:34 |
| dtantsur | I'm also worried about hitting something like VENDOR_SPECIFIC_UNRELATED_SERIAL | 10:34 |
| dtantsur | lecris[m]: sorry, I meant tcpdump verbosity. at some point it should dump everything it sees. | 10:35 |
| lecris[m] | Also unrelated, but using disk-image-builder with `iscsi-boot` is sufficient for using ironic with cinder? | 10:35 |
| lecris[m] | dtantsur: Ok, let me bring that up really quick | 10:35 |
| dtantsur | re cinder, I don't remember, but there should be some docs | 10:36 |
| dtantsur | lecris[m]: https://docs.openstack.org/ironic/latest/admin/boot-from-volume.html | 10:36 |
| lecris[m] | https://docs.openstack.org/project-deploy-guide/tripleo-docs/latest/features/baremetal_overcloud.html#booting-a-bare-metal-instance-from-a-cinder-volume Here is the doc about it, but it is not ideal | 10:36 |
| dtantsur | I haven't touched tripleo for a while, cannot comment on that (cc hjensas) | 10:37 |
| lecris[m] | It's the part about creating the ramdisk | 10:37 |
| arozman | dtantsur for systemd I have not made a bug report in relation to udev, I am not sure if the root cause is something in systemd, udev, kernel, driver just that the sympthom appears in udev | 10:37 |
| dtantsur | arozman: are you sure that lsblk also does not provide a correct serial& | 10:38 |
| dtantsur | ? | 10:38 |
| lecris[m] | dtantsur: Thank you for the hint, it is indeed that, the tripleo docs should point to that soon | 10:39 |
| arozman | dtantsur based on my limited experience with this issue I have seen 2 incidents in one of them lsblk was correct while udev was not in the other case I only had second hand information that stated that serial was not correct either in lsblk nor in udev, I have detailed data about the first incident that I have shared in the storyboard issue | 10:40 |
| dtantsur | arozman: let's start with fixing the 1st one? chances are high that the 2nd one is a misunderstanding. | 10:41 |
| arozman | dtantsur okay then I will remove the all the udev related changes and leave only lsblk | 10:42 |
| lecris[m] | dtantsur: I've included more verbose tcpdump to the bug https://bugs.launchpad.net/tripleo/+bug/1990028 | 10:44 |
| lecris[m] | Another iscsi question. Is it supposed to be built for the deploy_kernel/ramdisk and/or for the image finally being booted | 10:56 |
| lecris[m] | And to confirm, the deploy_kernel should be having the usual ironic agent, and is it ok to be different than the final booting one? | 10:56 |
| dtantsur | lecris[m]: I don't think you need an agent for iscsi boot? | 10:58 |
| dtantsur | if you wait for TheJulia to wake up, she may remember this stuff better than me | 10:58 |
| lecris[m] | I am on matrix, so if someone can pass the message I can read the response later | 10:59 |
| dtantsur | I think TheJulia aka ashinclouds[m] will see our discussion | 11:00 |
| lecris[m] | 👍️ | 11:00 |
| lecris[m] | But about deploy_kernel, that is only for ipa or does it have to correspond to the production image being booted? | 11:02 |
| dtantsur | only IPA | 11:02 |
| dtantsur | the image is configured using other properties | 11:02 |
| dtantsur | (nova does it for you if using nova) | 11:03 |
| lecris[m] | Ok, then there's another issue I need to hunt down. The booting image does not boot after deploying successfully | 11:03 |
| lecris[m] | Using nova with image and not creating new volume (noop storage and direct deploy) | 11:04 |
| dtantsur | lecris[m]: re bug: I think you pasted a DHCP request, we'd need to see the DHCP offer from the server. | 11:08 |
| lecris[m] | That's the only info being served, after that it crashes | 11:11 |
| dtantsur | hmm, nothing before that? | 11:11 |
| lecris[m] | Just another request identical to that | 11:11 |
| dtantsur | IIRC DHCP is a 3-stage exchange | 11:11 |
| dtantsur | or even 4 | 11:12 |
| * dtantsur cannot think on Monday | 11:12 | |
| lecris[m] | I've added all 3 exchanges that are served | 11:14 |
| iurygregory | good morning Ironic | 11:17 |
| dtantsur | lecris[m]: hmm. it's only stuff that client tells the server, the reverse communication is missing | 11:35 |
| lecris[m] | dtantsur: oh, that's because dhcp fails when it requests the giberish | 11:46 |
| dtantsur | it's the tftp stage, but the giberish or its meaningful part has to come from the initial DHCP response | 11:47 |
| dtantsur | DHCPDISCOVER -> DHCPOFFER (here!) -> DHCPREQUEST -> DHCPACK | 11:47 |
| dtantsur | I only see 1 and 3 | 11:47 |
| lecris[m] | Hmm, let me run a working one for comparison | 11:49 |
| lecris[m] | I remember that the efi file appears 2-3 times | 11:51 |
| lecris[m] | Hmm I'll try the command again in the container | 11:54 |
| lecris[m] | Nvm can't run in the container | 11:57 |
| lecris[m] | The correct one does not show the offer either. Probably my tcpdump command is incorrect | 12:06 |
| dtantsur | it looks correct to me... try -i any? maybe it's going out through a different interface? | 12:13 |
| lecris[m] | Well it goes on tap, but the container does not have tcpdump | 12:17 |
| lecris[m] | Any does not seem to help still only discover and request | 12:19 |
| lecris[m] | Using ovs-tcpdump, hopefully this would show everything | 12:33 |
| * TheJulia begins the consumption of coffeeeeeeee | 13:04 | |
| TheJulia | Good morning everyone! | 13:05 |
| dtantsur | mornign TheJulia | 13:05 |
| iurygregory | good morning TheJulia | 13:05 |
| lecris[m] | dtantsur: lol, I had to resort to running 2 tcpdumps because ovs-tcpdump only showed offer and ack, and the other only discover and request >< | 13:10 |
| dtantsur | O____o | 13:11 |
| dtantsur | bloody computers | 13:11 |
| TheJulia | so, no agent is required for iscsi boot, but agent is required for cleaning after the fact | 13:11 |
| dtantsur | good point Julia | 13:12 |
| lecris[m] | Cleaning local drives that is? | 13:12 |
| TheJulia | yeah | 13:15 |
| TheJulia | Am I reading that lp bug correctly and seeing that it is getting requests back from hosts with incredibly malformed RRQs ? | 13:15 |
| lecris[m] | Mine? | 13:18 |
| lecris[m] | dtantsur: TheJulia: I've updated the bug with the replies. The replies look normal | 13:19 |
| * TheJulia sips coffee faster to wake up faster | 13:19 | |
| dtantsur | TheJulia: yeah, it looks like firmware is requesting absurd file names from TFTP | 13:19 |
| lecris[m] | The weird part is that inspect works fine. I'll add the tcpdump for that next | 13:20 |
| TheJulia | freaky | 13:21 |
| TheJulia | lecris[m]: what is the hardware this is being experienced on? | 13:22 |
| lecris[m] | Some intel nic, the info is in inspector right? | 13:23 |
| TheJulia | well... | 13:26 |
| lecris[m] | Zzz vendor: 0x8086 product: 0x1521 | 13:26 |
| TheJulia | If it is in UEFI mode, it is not the nic loaded firmware which is in use | 13:26 |
| lecris[m] | Dunno if that's searchable | 13:26 |
| TheJulia | it is the system firmware which is in control | 13:26 |
| TheJulia | i.e. EFI runtime | 13:26 |
| lecris[m] | Oh it's a supermicro | 13:26 |
| TheJulia | oh | 13:26 |
| TheJulia | ... Why am I not surprised? | 13:26 |
| TheJulia | What model motherboard? | 13:27 |
| lecris[m] | It's 10 year old so web interface doesn't show | 13:28 |
| lecris[m] | I'll go boot the bios to check | 13:29 |
| TheJulia | oh! | 13:34 |
| TheJulia | so... then it is bios booting only most likely | 13:34 |
| TheJulia | *most likely* | 13:34 |
| TheJulia | umm.... my X8 (or was it an X9) is bios only... | 13:34 |
| TheJulia | X10 and beyond are UEFI machines if memory serves | 13:34 |
| TheJulia | then again, the bios will reveal all! | 13:35 |
| lecris[m] | For pxe it only works on uefi | 13:35 |
| lecris[m] | X9drt | 13:35 |
| TheJulia | ... hmmmmmmm | 13:38 |
| -opendevstatus- NOTICE: As of the weekend, Zuul only supports queue declarations at the project level; if expected jobs aren't running, see this announcement: https://lists.opendev.org/pipermail/service-announce/2022-September/000044.html | 13:38 | |
| lecris[m] | While I'm here, version 3.0 bios 2.15.1236 | 13:38 |
| lecris[m] | Uefi driver of the nic is pro/1000 5.6. | 13:40 |
| lecris[m] | Intel i350 | 13:40 |
| TheJulia | ipmi being used? | 13:40 |
| lecris[m] | Yes | 13:41 |
| TheJulia | So yeah, June 2013 is the date on the docs, so 9-10 years old | 13:41 |
| lecris[m] | I'll check the version from my pc. Too loud down here | 13:41 |
| TheJulia | Interesting! | 13:41 |
| TheJulia | yeah, don't stand in a server room on our account | 13:42 |
| * TheJulia has hearing damage from servers and too many metal tubes | 13:42 | |
| * TheJulia felt old today, went shopping for jeans this weekend and couldn't hear the attendant... :( | 13:43 | |
| TheJulia | err | 13:43 |
| TheJulia | yesterday | 13:43 |
| lecris[m] | What they don't teach you about being a sysadmin :D | 13:44 |
| lecris[m] | I think ipmi version is 02.26 | 13:44 |
| TheJulia | hmm... does the web UI use frames? | 13:46 |
| * TheJulia is trying to place it on the mental timeline of supermicro gear | 13:46 | |
| lecris[m] | Not sure what that, but probably not | 13:47 |
| TheJulia | okay, so similar firmware to my 2013-2014 X10 | 13:48 |
| lecris[m] | Probably, but got a few older ones and slightly newer ones that had no problem | 13:49 |
| lecris[m] | Oh, even on the inspect it looks weird | 13:50 |
| lecris[m] | But it works | 13:50 |
| lecris[m] | So the one that works, tcpdump reports 'snponly.efi^@c | 13:54 |
| lecris[m] | s/'/"/, s/c/"/ | 13:54 |
| TheJulia | ... freaky | 13:57 |
| TheJulia | so... Interestingly enough, this specific platform string it sends... many folks have had problems with | 13:58 |
| TheJulia | "PXEClient:Arch:00007:UNDI:003016" | 13:58 |
| TheJulia | UEFI Bytecode, not 00009 which is x86_64 EFU | 13:58 |
| TheJulia | err, EFI | 13:58 |
| TheJulia | lecris[m]: any chance you can re-capture with more verbose output logging? | 13:59 |
| lecris[m] | That's with -vvvv | 13:59 |
| TheJulia | tcpdump, why do you hate us | 14:00 |
| lecris[m] | Seems the ^@ is specific to inspector, neutron does not give that | 14:01 |
| TheJulia | so your saying dnsmasq is actually repsonding with the appended strings? | 14:04 |
| TheJulia | I guess that was where my brain was going, is the dhcp server returning exactly what it should be | 14:04 |
| TheJulia | which is why I was hoping for more verbose, or even just the pcap file at this point might be helpful | 14:04 |
| opendevreview | Merged openstack/bifrost master: Do not install grub2 and shim on the host system https://review.opendev.org/c/openstack/bifrost/+/857935 | 14:04 |
| lecris[m] | Yes, but only 1 character and only in inspector (which works) | 14:05 |
| lecris[m] | Seems to act like a end of string character for older interfaces? | 14:06 |
| lecris[m] | I.e. on normal without ^@ dhcp client request gibberish efi file | 14:07 |
| TheJulia | I've never seen that before, which has me thinking it is time to get my hands on a pcap file of each and validate the field values | 14:07 |
| lecris[m] | But the dnsmasq file do not look that much different so I don't know what can trigger it | 14:08 |
| TheJulia | it is dynamic with neutron at that level | 14:09 |
| TheJulia | we send it a string to return | 14:09 |
| TheJulia | to the client that is | 14:09 |
| lecris[m] | Isn't it in the opts file? | 14:09 |
| TheJulia | what boot_interface does htis node use? | 14:09 |
| TheJulia | neutron does in a network namespace | 14:09 |
| lecris[m] | Ipmitool I believe | 14:10 |
| TheJulia | can you run `grep bootfile /path/to/ironic.conf` ? | 14:10 |
| lecris[m] | Oh, ipxe | 14:10 |
| TheJulia | okay | 14:10 |
| TheJulia | you will likely need to run that inside of the containre | 14:10 |
| lecris[m] | Neutron or ironic? | 14:11 |
| TheJulia | ironic | 14:11 |
| TheJulia | need to eliminate possibilities | 14:11 |
| *** rcastillo|rover is now known as rcastillo | 14:11 | |
| TheJulia | so first, configuration then we need to ask neutron what it has been configured for on the port | 14:11 |
| lecris[m] | From inspector: | 14:13 |
| lecris[m] | /var/lib/config-data/puppet-generated/ironic_inspector/etc/ironic-inspector/dnsmasq.conf:18:dhcp-option=tag:ipxe6,option6:bootfile-url,http://192.168.5.76:8088/inspector.ipxe | 14:13 |
| lecris[m] | /var/lib/config-data/puppet-generated/ironic_inspector/etc/ironic-inspector/dnsmasq.conf:21:dhcp-option=tag:efi6,tag:!ipxe6,option6:bootfile-url,tftp://192.168.5.76/snponly.efi | 14:13 |
| TheJulia | well, that doesn't look right | 14:13 |
| TheJulia | that is ipv4 urls with the ipv6 tags | 14:14 |
| lecris[m] | That's dnsmasq.conf not ironic.conf | 14:14 |
| TheJulia | i know | 14:14 |
| TheJulia | The lines likely get configured for that regardless of ipv6 or not | 14:14 |
| lecris[m] | Let me show the full dnsmasq.conf | 14:15 |
| TheJulia | ironic has several different pxe and ipxe parameters, all with bootfile in the name | 14:15 |
| TheJulia | that would be excellent, thanks | 14:15 |
| * lecris[m] sent a code block: https://matrix.org/_matrix/media/r0/download/matrix.org/jxARcqGnoEgwGlnWnRVsQXMM | 14:16 | |
| lecris[m] | There is the !ipxe part I guess | 14:17 |
| TheJulia | Yeah, tag:efi,tag:!ipxe to load snponly.efi | 14:19 |
| TheJulia | okay, so what does ironic.conf have in it's config file? | 14:19 |
| lecris[m] | Which parameter? | 14:20 |
| TheJulia | just grep for bootfile | 14:21 |
| TheJulia | it *should* be ipxe_bootfile | 14:21 |
| TheJulia | sorru | 14:21 |
| TheJulia | uefi_ipxe_bootfile | 14:21 |
| lecris[m] | #pxe_bootfile_name = pxelinux.0... (full message at <https://matrix.org/_matrix/media/r0/download/matrix.org/bQsOZrTbLKtaZxVdSVJDtFjh>) | 14:22 |
| lecris[m] | Looks normal | 14:22 |
| TheJulia | okay | 14:24 |
| TheJulia | what state is the machine now in ironic? | 14:24 |
| * TheJulia is now curious what neutron thinks | 14:24 | |
| lecris[m] | The faulty one? Manageable | 14:24 |
| TheJulia | so I guess lets try to trigger cleaning, and then hunt down it's neutron port and see what configuration it shows | 14:25 |
| TheJulia | openstack port show <uuid> will include the dhcp parameters | 14:25 |
| lecris[m] | Otherwise deploy failed at step empty | 14:25 |
| lecris[m] | (Timeout) | 14:25 |
| TheJulia | but, it needs to be in that state while trying to clean/deploy for those parameters to even be set | 14:25 |
| lecris[m] | Ok | 14:25 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost stable/yoga: Simplify bifrost-ironic-install required_defaults https://review.opendev.org/c/openstack/bifrost/+/858389 | 14:28 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost stable/yoga: Consolidate ubuntu/debian required_defaults https://review.opendev.org/c/openstack/bifrost/+/858390 | 14:28 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost stable/yoga: Do not install grub2 and shim on the host system https://review.opendev.org/c/openstack/bifrost/+/858391 | 14:28 |
| lecris[m] | Anything specific or the whole table? | 14:28 |
| dtantsur | had to backport refactoring patches, I hope they're not too invasive | 14:28 |
| lecris[m] | Looks similar to the confs I uploaded in the bug report | 14:29 |
| TheJulia | lecris[m]: I don't remember the fields in particular in neutron | 14:29 |
| lecris[m] | Binding_failed | 14:29 |
| * lecris[m] sent a code block: https://matrix.org/_matrix/media/r0/download/matrix.org/psjrlwFOIHCgOzsyIpfcilkd | 14:30 | |
| * TheJulia reloads bug | 14:31 | |
| TheJulia | lecris[m]: binding failed is to be expected for baremetal if the networking-baremetal agent is not running | 14:31 |
| TheJulia | so... | 14:32 |
| TheJulia | I think we're looking at some sort of firmware bug | 14:32 |
| TheJulia | unless we can absolutely confirm dnsmasq is handing the wrong thing back to the client | 14:32 |
| lecris[m] | The only noteworthy thing is the additional ^@ that seems to resolve the issue | 14:33 |
| TheJulia | it could just be a control character which is out of spec which could be ignored | 14:37 |
| TheJulia | I suspect that is outside of the ascii range permitted | 14:37 |
| lecris[m] | Well dnsmasq seems to append it automatically when using conf file | 14:38 |
| lecris[m] | Going back now, but feel free to @ me, I can read later from matrix | 14:40 |
| TheJulia | @lecris[m]: can we confirm that via the packet captures though? Nothing I've seen indicates that beyond what the card asks for, which seems suspect given the configuration is the same on both services | 14:42 |
| lecris[m] | The tcpdump shows it in the offer | 14:43 |
| lecris[m] | Or you want to confirm somewhere else also? | 14:44 |
| JayF | Is it possible there's somehow a non printable character somewhere in your configuration file? | 14:44 |
| JayF | That's an absolutely wild guess, but it might be worth checking | 14:45 |
| lecris[m] | Hmm, would it show on vi? | 14:45 |
| TheJulia | lecris[m]: oh, I see it uploaded | 14:46 |
| TheJulia | not if your terminal is in unicde | 14:46 |
| TheJulia | unicdoe | 14:46 |
| * TheJulia can't type | 14:46 | |
| TheJulia | I already checked what was pasted, ascii | 14:46 |
| JayF | Try less -R | 14:46 |
| lecris[m] | I'll try in 30 min | 14:47 |
| JayF | Like I said it's a long shot but it doesn't hurt to check... | 14:47 |
| JayF | Trust me when I say you want that to be the problem instead of a firmware breakage lol | 14:47 |
| TheJulia | yeah, both are 11 chars long in the dhcp response | 14:48 |
| lecris[m] | Inpector is 12 though | 14:48 |
| TheJulia | okay, yeah | 14:52 |
| TheJulia | https://launchpadlibrarian.net/624562903/tcpdump_inspector.txt | 14:52 |
| TheJulia | so one reports ^@ which is a null/empty character representing end of string | 14:54 |
| TheJulia | the other doesnt | 14:54 |
| TheJulia | for some... wildly weird reason | 14:54 |
| lecris[m] | Maybe dnsmasq appends it when reading conf file, but not on optsfile | 14:55 |
| TheJulia | maybe | 14:55 |
| TheJulia | but however the firmware behaves, it doesn't grok it and I'm guessing the extra garabage is from other parts of memory | 14:55 |
| TheJulia | @lecris[m] has the firmware on this machine been updated to the latest supermicro indicates as available? | 14:55 |
| lecris[m] | No, don't know if it's still available to download | 14:56 |
| rpittau | need to split, see you tomorrow o/ | 14:57 |
| iurygregory | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon Sep 19 15:00:00 2022 UTC and is due to finish in 60 minutes. The chair is iurygregory. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| iurygregory | o/ | 15:00 |
| matfechner | o/ | 15:00 |
| iurygregory | hello ironicers, welcome to our weekly meeting | 15:00 |
| JayF | o/ | 15:00 |
| TheJulia | o/ | 15:01 |
| ajya | o/ | 15:01 |
| erbarr | o/ | 15:01 |
| iurygregory | you can find our agenda in the wiki page | 15:01 |
| iurygregory | #link https://wiki.openstack.org/wiki/Meetings/Ironic#Agenda_for_next_meeting | 15:01 |
| iurygregory | #topic Announcements / Reminder | 15:02 |
| iurygregory | we have the result for the doodle \o/ | 15:02 |
| iurygregory | #info Check PTG Topics - doodle result Tomorrow Sep 20 at 16 UTC | 15:02 |
| iurygregory | I will send the information in the ML today (I can also send a calendar invite for those who need one =) ) | 15:03 |
| iurygregory | please ping me here or send me a dm if you want the calendar invite | 15:04 |
| iurygregory | #info Zed release by EOW | 15:04 |
| TheJulia | @lecris[m]: https://www.supermicro.com/products/archive/motherboard/x9drt-hf <-- doublecheck that is right | 15:04 |
| iurygregory | we will focus on Ironic, ironic-python-agent, inspector and bifrost this week | 15:05 |
| lecris[m] | Maybe, saw 3.3 is available | 15:05 |
| JayF | iurygregory: if you need any help doing release work, lmk -- I should learn some of this | 15:06 |
| iurygregory | JayF, sure =) | 15:06 |
| iurygregory | does anyone have something that would like to add to our Announcements / Reminder ? | 15:07 |
| TheJulia | has everyone voted? | 15:08 |
| TheJulia | at least, elegeble for voting for PTL/TC? | 15:08 |
| JayF | Voting closes in 7 hours, give or take, yeah? | 15:09 |
| TheJulia | yup | 15:09 |
| iurygregory | great question, I heard a few people voted, other had problems to vote because they didn't have OSF membership etc | 15:09 |
| * TheJulia raises an eyebrow | 15:10 | |
| TheJulia | we can circle back to that I guess | 15:10 |
| iurygregory | I thought it would close earlier lol | 15:10 |
| iurygregory | yeah | 15:10 |
| iurygregory | #topic Review subteam status reports | 15:11 |
| iurygregory | we probably don't have a lot, but would to check the SQLAlchemy 2.0 | 15:12 |
| iurygregory | #link https://etherpad.opendev.org/p/IronicWhiteBoard | 15:12 |
| JayF | I do not know how to run the bug subteam report; it's been ages since it's been properly updated. I did start going through the backlog. ~20 tickets triaged; 5 marked as merged/triaged, 9 closed invalid, rest either left alone or labelled as triage | 15:13 |
| JayF | I'm pretty sure at this pace I'm going to eventually get through the backlog, it just is slow work | 15:13 |
| JayF | Anyone who has done this kind of triage against storyboard who wants to give me advice on how to be more efficient, please reach out | 15:13 |
| iurygregory | not sure how to make more efficient I find it better than jira :D | 15:14 |
| JayF | I don't, and I dislike jira, but hey that's life :P | 15:14 |
| iurygregory | yes! | 15:14 |
| TheJulia | Are there any volunteers for sqlalchemy stuffs on ironic-inspector ? | 15:16 |
| iurygregory | I can help with that | 15:16 |
| kubajj | Am I also able to edit the white board? (line 152 says WIP, but the change has been merged) | 15:16 |
| dtantsur | everyone is | 15:16 |
| iurygregory | kubajj, yeah, everyone can update =) | 15:16 |
| iurygregory | I think we can move on | 15:18 |
| iurygregory | #topic Deciding on priorities for the coming week | 15:19 |
| iurygregory | our focus for this week is important patches for ironic, inspector, ipa, bifrost that we want to include in the Zed release | 15:20 |
| TheJulia | I... think I can mark the RBAC stuffs as basically minus more tempest tests | 15:20 |
| dtantsur | https://review.opendev.org/c/openstack/bifrost/+/858099 is a fix for the last failing bifrost job | 15:20 |
| TheJulia | err, basically done | 15:20 |
| iurygregory | ok now I understand =), I was a bit lost with the minus | 15:21 |
| kubajj | I don't know how important it is, but this one passed zuul after dtantsur changed the grenade test https://review.opendev.org/c/openstack/ironic-inspector/+/853304 | 15:21 |
| iurygregory | tks TheJulia | 15:21 |
| JayF | kubajj: I gave that #ironic-week-prio and a +2 | 15:22 |
| kubajj | thanks JayF | 15:22 |
| ajya | wouldn't mind https://review.opendev.org/c/openstack/ironic/+/855872 getting in Zed - it's already in the prio list | 15:23 |
| TheJulia | https://review.opendev.org/c/openstack/ironic/+/852694 <-- would be nice as well | 15:25 |
| iurygregory | ajya, ack | 15:25 |
| iurygregory | TheJulia, ok | 15:25 |
| iurygregory | ok, moving on | 15:29 |
| iurygregory | #topic Baremetal SIG | 15:29 |
| iurygregory | wondering if we have any updates for it | 15:29 |
| iurygregory | JayF, tks for sending the email | 15:29 |
| JayF | I spoke with some of the devrel experts downstream here at GR-OSS, but didn't get much actionable feedback :| | 15:30 |
| JayF | One thing that was said, that I think we should consider -- is asking cores/contributors of Ironic to give a short talk on a new/lesser known feature when we don't have an external presentation | 15:30 |
| JayF | which I think is a decent idea; except that putting more work on contributors is probably unreasonable | 15:30 |
| TheJulia | We've also done that unless we want to do presentation on super minor things | 15:31 |
| TheJulia | then again... each cycle brings new things | 15:31 |
| iurygregory | it would be interesting, I think we did some presentations on this style... | 15:31 |
| JayF | bluntly, the presentation is just the honey to get folks in ... the real value is in discussion and getting folks in one place | 15:32 |
| JayF | but AIUI there's been little interest in attending a SIG without a talk so... | 15:32 |
| TheJulia | I've had some people tell me it is an issue of a lack of a standing meeting invite | 15:33 |
| TheJulia | and where we *have* gotten invites sent out to end operators, we've tended to have far better participation | 15:33 |
| iurygregory | hummm | 15:34 |
| JayF | I am onboard to create and maintain such an invite, but I have no idea who to initially populate it with | 15:34 |
| TheJulia | cores, if it is on google, we can all edit it with a google account | 15:34 |
| iurygregory | yeah | 15:34 |
| TheJulia | and invite more people *insert evil laugh here* | 15:34 |
| iurygregory | I wouldn't mind doing this | 15:35 |
| TheJulia | granted... the meetings system does icals... but that requires people to go find, and be willing to just add to their own calendar | 15:35 |
| TheJulia | a few... extra steps | 15:35 |
| JayF | It occurs to me both my emails are on weird TLDs (personal = .cc, work = .io) so maybe someone with a three digit TLD should own it anyway lol | 15:35 |
| TheJulia | I could create something, but I think we should discuss time/day | 15:36 |
| JayF | One of us should reply to the on-list thread with a doodle and the outcome of this conversation | 15:36 |
| TheJulia | my limiting factor has been the use of the 2nd or third tuesday in the month which collides with higher level meetings | 15:37 |
| iurygregory | I can create the doodle and answer in the thread JayF | 15:37 |
| JayF | ack, thank you | 15:37 |
| TheJulia | and we could do some sort presentations based upon added features, I think | 15:38 |
| TheJulia | at least from this past cycle | 15:38 |
| iurygregory | wondering if google would be the best thing to use, but we can discuss this after the meeting probably | 15:38 |
| iurygregory | also since at the PTG we will have operators it would be good to raise this during the PTG (I think) | 15:39 |
| JayF | that's actually a really good idea, and not far away | 15:39 |
| JayF | maybe worth just tabling the actual scheduling until post-PTG | 15:40 |
| iurygregory | more thoughts on it? | 15:41 |
| iurygregory | ok, let's do this =) | 15:44 |
| iurygregory | moving on | 15:44 |
| iurygregory | #topic RFE review | 15:44 |
| iurygregory | JayF, you added a topic =) | 15:44 |
| JayF | I mean, I'm "up" theoretically, but this is one of the bugs I ran across in triage | 15:44 |
| JayF | it's a reasonable RFE, we should probably do it, I'm +1 but wanted to put it thru the process | 15:45 |
| JayF | https://storyboard.openstack.org/#!/story/2009872 optionally turn machines off after cleaning failure | 15:46 |
| JayF | Are there any comments or disagreements? If I don't hear any I'm going to make this as RFE-approved. It's a reasonable request, and it's smart to make it configurable so if there's any harm in powering off after some failures (e.g. firmware failure), it'll have been done by a non-default config | 15:47 |
| TheJulia | that seems like a bug as well | 15:47 |
| TheJulia | in that the clean fault should be cleared out upon start | 15:47 |
| TheJulia | what they are describing is behavior where it does not | 15:48 |
| JayF | oh, interesting, I missed that nuance | 15:48 |
| JayF | I'll put that comment there and split the RFE from the bug part, and ask for more details on the bug piece | 15:48 |
| TheJulia | ++ | 15:48 |
| TheJulia | we might have quietly already fixed that without realizing that | 15:48 |
| TheJulia | but worth double checking | 15:48 |
| TheJulia | and fixing if not | 15:48 |
| iurygregory | ++ | 15:49 |
| TheJulia | I think it makes sense to allow operators to say "turn off even if it failed" | 15:49 |
| TheJulia | but we just need a giant warning "if you've bricked things, not our fault" | 15:49 |
| TheJulia | We can do that by conveying "why we leave power on" | 15:49 |
| TheJulia | in the event of a failure | 15:49 |
| JayF | Oh yeah, absolutely | 15:50 |
| iurygregory | yeah, we don't want to receive the blame :D | 15:50 |
| JayF | I'll split off an RFE, marked approved, for that feature and I've already asked the user in that ticket for more details on the bug | 15:50 |
| JayF | you both know I wouldn't implement this in any kind of dangerous way, I'm a hardware-chicken lol | 15:50 |
| iurygregory | tks JayF | 15:51 |
| iurygregory | #topic Open discussion | 15:51 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost stable/yoga: Do not install grub2 and shim on the host system https://review.opendev.org/c/openstack/bifrost/+/858391 | 15:51 |
| iurygregory | does anyone have something that would like to discuss? | 15:52 |
| TheJulia | iurygregory: I've already felt horrible once when someone bricked a bunch of SSDs :( I'm just glad we were able to help them get them recovered | 15:52 |
| TheJulia | I had a thought of something semi-important, then it ran away | 15:52 |
| TheJulia | I think it is standing next to the coffee maker demanding MOAR COFFEEEEEEE | 15:52 |
| iurygregory | oh .-. | 15:52 |
| iurygregory | coffee \o/ | 15:53 |
| * iurygregory is happy because he bought more coffee capsules to use in the machine :D | 15:53 | |
| kubajj | Not to discuss, but I just wanted to let you know that now I have officially started my 4th year project and I will be working on the Integration of inspector into ironic https://storyboard.openstack.org/#!/story/2010275 | 15:53 |
| dtantsur | \o/ | 15:53 |
| iurygregory | kubajj, \o/ nice | 15:54 |
| JayF | Nice to hear it kubajj, it'll be nice to have you staying around \o/ | 15:54 |
| kubajj | So probably expect me asking couple of questions about inspector soon 😄 | 15:55 |
| JayF | as long as you have a comically large magnifying glass while you ask them :P | 15:55 |
| dtantsur | I'm all yours, you can blame me for most of its misdesign :D | 15:55 |
| kubajj | Thanks | 15:55 |
| TheJulia | kubajj: awesome! | 15:56 |
| arne_wiebalck | kubajj: \o/ | 15:56 |
| TheJulia | kubajj: if you want to solve data model stuffs first, sqlalchemy 2.0 compatability is fun! :) | 15:56 |
| * TheJulia should actually look at that this week | 15:56 | |
| dtantsur | "fun" | 15:56 |
| iurygregory | "fun" | 15:57 |
| iurygregory | :D | 15:57 |
| TheJulia | heh | 15:57 |
| TheJulia | Can you blame me?! | 15:57 |
| iurygregory | I think I heard some evil laugh | 15:57 |
| dtantsur | We can even complain to Mr Corgi about your behavior | 15:57 |
| iurygregory | :D | 15:57 |
| iurygregory | dtantsur, lol | 15:57 |
| kubajj | TheJulia: I need to complete the first step in the storyboard first as I defined it as a minimal objective for the project. | 15:57 |
| TheJulia | dtantsur: mr corgi is presently glaring at me... But I think he wants to go for a alk | 15:59 |
| TheJulia | walk | 15:59 |
| iurygregory | corgi wins | 15:59 |
| dtantsur | always | 15:59 |
| iurygregory | thanks everyone o/ | 15:59 |
| iurygregory | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Mon Sep 19 16:00:00 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-09-19-15.00.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-09-19-15.00.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-09-19-15.00.log.html | 16:00 |
| * arne_wiebalck is reading the backlog | 16:00 | |
| arne_wiebalck | JayF: we have done exactly this in the SIG, cores/devs presenting Ironic features | 16:01 |
| JayF | yeah, that's what folks said | 16:01 |
| arne_wiebalck | oh, ok | 16:01 |
| JayF | tbh, and I see this as a positive sign of the maturity of Ironic, we're just getting boring | 16:01 |
| arne_wiebalck | heh | 16:01 |
| JayF | hardware provisioning was supposed to be boring all along, then we came along and started automating it | 16:02 |
| JayF | now it only gets too nuts when some crazy firmware starts appending ^@ ;) | 16:02 |
| arne_wiebalck | there are still people interested which have not even started with Ironic | 16:02 |
| JayF | I wonder if those people know if they came in here, or showed up to something like the BM SIG, that they'd have folks who wanna help | 16:02 |
| arne_wiebalck | I am telling everyone and invite people to the SIG or the channel | 16:03 |
| arne_wiebalck | it works ... sometimes :) | 16:03 |
| arne_wiebalck | at the summit, in TheJulia and my session on the SIG, there were people interested ... but I never heard back from them | 16:04 |
| TheJulia | the... conundrum really is closing the feedback loop | 16:04 |
| TheJulia | for some reason it seems every forum session at the Summit ended up not building the connections necessary | 16:05 |
| TheJulia | it *really* seemed like since we didn't *force* feedback to occur and connections to be built, it fell to the bottom of everyone's radar, and it wasn't just ironic. Several cases where when I was in a packed room resulted in very little feedback/interaction even though many people in the rooms expressed interest | 16:06 |
| TheJulia | I think they didn't know how to connect with us in entirety as well | 16:06 |
| TheJulia | so it becomes a two way disconnect :\ | 16:06 |
| * TheJulia should like... paste irc link of this thought to kendall | 16:09 | |
| JayF | honestly I see it as a perception thing; at least from the folks I've interacted with | 16:14 |
| JayF | OpenStack's dated-at-this-point reputation for being hard to get code into is ingrained in folks who have been around it at that time | 16:15 |
| JayF | and makes people more resistant to participate | 16:15 |
| TheJulia | I'm not even sure it is just that | 16:17 |
| TheJulia | so we've been working to start an environmental sustainability sig | 16:17 |
| TheJulia | while semi-related, it was a bigger picture thing | 16:17 |
| arne_wiebalck | bye everyone, see you tomorrow o/ | 16:18 |
| TheJulia | and we had a packed room in Berlin... and so few people engaged after the fact :( | 16:18 |
| TheJulia | Arrr matties! It be talk like a pirate day! | 16:19 |
| JayF | Yeah, admittedly my observation is skewed by the fact I usually only engage with extreme-scale deployers of Ironic as a result of what my job has been | 16:22 |
| * TheJulia plays the appropriate talking like a pirate training video #1 | 16:27 | |
| TheJulia | ajya: so I'm updating some of the rdoproject.org auto notification emails for the packaging pipeline. For python-dracclient it currently lists cdearborn. I'm guessing that should be updated? | 17:14 |
| opendevreview | Adam Rozman proposed openstack/ironic-python-agent master: use lsblk as a source of device serial numbers https://review.opendev.org/c/openstack/ironic-python-agent/+/855866 | 18:29 |
| opendevreview | Adam Rozman proposed openstack/ironic-python-agent master: prioritize lsblk as a source of device serials https://review.opendev.org/c/openstack/ironic-python-agent/+/855866 | 18:31 |
| JayF | What's the preferred OS to run devstack on these days? | 21:29 |
| JayF | with ironic specifically in mind | 21:29 |
| JayF | going to default to latest ubuntu lts, but if you have suggestions I am still happy to hear mm | 21:29 |
| ashinclouds[m] | I generally just lean towards the LTS since that is what the ci pipeline uses. | 21:36 |
| JayF | ack, that's what I thought | 21:43 |
| JayF | I prefer rhel for binary distros but I prefer things working seamelessly more lol | 21:43 |
| TheJulia | +++++++ | 21:49 |
| JayF | our dev docs also still default to ipmi, we might want to evaluate that | 22:10 |
| JayF | or at least enhance it to include redfish or w/e | 22:10 |
| JayF | IDK if anyone else thinks that is important; but if we do I can take some time and get a more complete devstack config | 22:17 |
| JayF | *config documented | 22:18 |
| JayF | not trying to change stuff just write down what is :D | 22:18 |
| opendevreview | Merged openstack/bifrost master: Do not install EPEL from the internet if it's already installed https://review.opendev.org/c/openstack/bifrost/+/858099 | 22:19 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Concurrent Distructive/Intensive ops limits https://review.opendev.org/c/openstack/ironic/+/840006 | 22:29 |
| JayF | TheJulia: ^ -1, a few valid nits but mainly it looks like you accidentally committed a change from another branch | 22:37 |
| TheJulia | wut... | 23:17 |
| * TheJulia waits for gerrit... | 23:17 | |
| opendevreview | Julia Kreger proposed openstack/ironic master: Concurrent Distructive/Intensive ops limits https://review.opendev.org/c/openstack/ironic/+/840006 | 23:32 |
| TheJulia | JayF: I looked at the query and was like "oh, I will need to re-write that for the sqlalchemy 2.0 stuffs | 23:32 |
| TheJulia | " | 23:32 |
| TheJulia | but not *now* | 23:32 |
| JayF | TheJulia: hm. okay. It's still weird to go in with this change, but it's a comment so I guess it doesn't matter much? | 23:45 |
| JayF | what matters is good stuff ending up in the codebase so... w/e | 23:45 |
| TheJulia | Yup | 23:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!