| *** jph0 is now known as jph | 00:48 | |
| *** jph0 is now known as jph | 07:20 | |
| iurygregory | good morning Ironic | 11:12 |
|---|---|---|
| adam-metal3 | hello Ironic | 11:53 |
| TheJulia | good morning | 13:13 |
| adam-metal3 | Would somone have some time to discuss this with me: https://bugs.launchpad.net/ironic-python-agent/+bug/2061362, I would still like to go forward with this proposal if possible ? | 14:58 |
| TheJulia | adam-metal3: sure, I think I just broke someone's brain with the CRA, so I just need to make coffee to celebrate first | 15:01 |
| TheJulia | adam-metal3: does zoom work for higher bandwidth, or... ? | 15:09 |
| TheJulia | ewww, the spi flash is getting presented as a device | 15:11 |
| TheJulia | adam-metal3: any chance we can get lspci and lsusb output from a host presenting this sort of device? | 15:12 |
| JayF | ++ that's exactly what I was going to ask | 15:13 |
| TheJulia | pci-0000:00:14.0-usb-0:4:1.0-scsi-0:0:0:2 <-- That sure looks like a BMC to me | 15:13 |
| TheJulia | an Aspeed based BMC at that | 15:14 |
| TheJulia | with the virtual usb interconnect | 15:14 |
| TheJulia | I think the answer is "if we can fingerprint it" it makes sense to just go "ugh, yeah, this is a thing" | 15:15 |
| TheJulia | There is a method of using SPI flash modeling to present drivers to a host OS | 15:16 |
| TheJulia | which if this is readable with a filesystem, I bet that is exactly what it has on it | 15:16 |
| TheJulia | if it can't be read, then it is active but not presenting contents, but device IDs are kind of what I'm curious about | 15:21 |
| TheJulia | I've sort of seen something like this in the past and it presents *very* similarly to the way virtual media devices present | 15:21 |
| adam-metal3 | well sure zoom is fine, but related to the device I can't really give more info not that I have not checked but as I wrote there is nothing else about these devices just size , model and the PCI address | 15:22 |
| TheJulia | well, there is an emulated usb bus and a pci bus it is coming across, so there should be device IDs | 15:23 |
| TheJulia | in the lsusb output | 15:23 |
| adam-metal3 | ohh lsusb, that one I have not asked for just lspci, lsblk, udevadm I will ask for the lsusb then not sure when I will get any answer though , do you have something in mind about how to filter out such devices ? | 15:25 |
| TheJulia | so, we have seen similar things, but typically they have presented as a read only device like a CD, and thus get scrubbed out from the list of devices | 15:26 |
| TheJulia | In this case, it looks and smells like a disk, so we would either have to figure out a pattern to identify it upfront as a filtered entity, or a pattern after the fact | 15:27 |
| TheJulia | Because there really is no writing to this device most likely, it is just getting presented in an odd way and we somehow need to handle that if we can "fingerprint" it | 15:28 |
| TheJulia | and if it matches what we expect as a "this is a known thing" I think we're cool maybe even accepting that as a default, a generalized knob is what concerns us | 15:28 |
| JayF | I'm now pondering if you could actually do harm to a device like that trying to wipe it | 15:28 |
| TheJulia | JayF: I don't think they accept writes at all, I think io errors result | 15:28 |
| TheJulia | you can seek/read most likely | 15:29 |
| TheJulia | It is *likely* a BMC firmware bug actually | 15:29 |
| JayF | yep, that's where I was going | 15:29 |
| TheJulia | because a device can express itself as read-only | 15:29 |
| JayF | does it show up as a read only device in lsblk | 15:29 |
| JayF | and if not, we can fix on ironic side, but customer should report a bug to hardware vendor | 15:29 |
| TheJulia | Good question, it might not or it might, we're sort of grasping at straws without the additional info | 15:30 |
| JayF | you know, it gets interesting tho | 15:30 |
| JayF | because it could still be dangerous to skip r/o devices during cleaning | 15:30 |
| JayF | especially since that's a failure mode for many forms of storage | 15:31 |
| TheJulia | yeah, this is why it is nice when CD's are known to be CDs :) | 15:31 |
| TheJulia | or "virtual CDs" | 15:31 |
| TheJulia | yeah, a blanket read-only, I'd still prefer a little deeper of a verification | 15:31 |
| TheJulia | I want to get to "this looks, smells, and feels like it is a fake BMC device" | 15:32 |
| JayF | yep, exactly | 15:32 |
| TheJulia | and, I'm cool with skipping fake bmc devices | 15:32 |
| JayF | ++ | 15:32 |
| TheJulia | (but, do definitely file a bug with a hardware vendor) | 15:33 |
| JayF | something like `lspci -M` + `lspci -mm` might give enough combined info to at least tie it back to the bmc on the pci bus, too | 15:33 |
| JayF | but we really just need more info | 15:33 |
| adam-metal3 | I hae to check the the logs, I remember it presented itself as "disk" I will check the access mode | 15:34 |
| TheJulia | And the line from lsusb representing the device or device's controller | 15:34 |
| TheJulia | I think the issue is it also sort of boils down to how the transport is being handled with the host | 15:35 |
| TheJulia | so it will govern how that reports | 15:35 |
| TheJulia | TIL my desktop has an ISA bridge | 15:35 |
| JayF | most do | 15:35 |
| JayF | that's how coretemp works on intels, for instance | 15:36 |
| JayF | it's ISA straight to the chip | 15:36 |
| * TheJulia blinks | 15:36 | |
| * TheJulia blinks more | 15:36 | |
| JayF | "straight to the chip" [hand waving intensifies] | 15:36 |
| JayF | adam-metal3: TheJulia: the other thing we could consider, if we can't ID the device positively: if a block device is smaller than "X" and can't be wiped, it's probably OK (optionally) | 15:37 |
| JayF | I'm assuming this phantom BMC disk isn't 72 terabytes | 15:37 |
| JayF | obvious hyperbole, but I would hope it'd be small enough to be clearly not a data disk | 15:38 |
| JayF | hmm was that in the bug | 15:38 |
| JayF | nope, it wasn't | 15:39 |
| TheJulia | we've done some similar fingerprinting in the past, it is about building some level of comfort and understanding | 15:44 |
| TheJulia | (and lots of comments) | 15:44 |
| TheJulia | (in the code) | 15:44 |
| adam-metal3 | Thanks for the discussion I will get back with one set of the data but the lsusb will be max next week, since I have provided a patch in the form of this quiet cleanup my downstream was fine with it and run away :D | 15:46 |
| JayF | adam-metal3: so, re: the other half of that bug with an implied question; in metal3 ramdisk builds is there a way for you to install an additional python package alongside IPA in that venv? | 15:48 |
| JayF | adam-metal3: that's basically what you have to do in order to add custom cleaning steps or override existing ones | 15:48 |
| JayF | e.g. https://opendev.org/openstack/ironic-python-agent/src/branch/master/examples if you copied custom-disk-erase out of here into your own repo, overwrite erase_devices_metadata with a version that ignores failure, and installed it in the venv alongside IPA, it'll be used automatically | 15:49 |
| adam-metal3 | JayF, yeah with building custom IPA I can, but my dowsntream req was to use 1 IPA for all envs so if I overwrite the step then is tehre a way to switch back and forth between the default and the overwritten step? or should I kind of "hack" it together something like "put a condition to the overwriting package to check for e.g. a kernel cmdline argument and based on that start the default or the custom cleanup step" ? | 16:04 |
| JayF | if you look at the example, that's what evaluate_hardware_support method is for | 16:04 |
| JayF | there are pretty comprehensive docs on this interface here: https://docs.openstack.org/ironic-python-agent/latest/contributor/hardware_managers.html | 16:05 |
| JayF | it's an old doc but the interface hasn't changed in that time | 16:05 |
| adam-metal3 | okay nice then this seems like a good first option for me but anyways I will try to get more info also about the root cause, thanks! | 16:05 |
| JayF | https://docs.openstack.org/ironic-python-agent/latest/contributor/hardware_managers.html#priority is the specific mechanism you're looking for | 16:06 |
| JayF | Yeah, I suggest having this ability in your toolbox, it'll let you do things that we won't always be cool with upstream (e.g. your existing RFE) | 16:06 |
| adam-metal3 | nice | 16:06 |
| -opendevstatus- NOTICE: There will be a short Gerrit downtime while we update a database and our container image | 17:14 | |
| JayF | I discovered today servicing never got added to the state diagram; was that intentional? (I know we have some issues with tooling to generate those) | 20:15 |
| opendevreview | Tony Breeds proposed openstack/bifrost master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/bifrost/+/919235 | 20:51 |
| opendevreview | Tony Breeds proposed openstack/ironic master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic/+/919264 | 20:53 |
| opendevreview | Tony Breeds proposed openstack/metalsmith master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/metalsmith/+/919275 | 20:54 |
| TheJulia | JayF: likely the image generation never got ran | 20:57 |
| TheJulia | it has to be changed and uploaded | 20:57 |
| JayF | I'll see if I can turn that into a quick win before I start a weekend | 20:57 |
| opendevreview | Tony Breeds proposed openstack/sushy master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/sushy/+/919331 | 20:59 |
| opendevreview | Tony Breeds proposed openstack/tenks master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/tenks/+/919336 | 20:59 |
| opendevreview | Tony Breeds proposed openstack/virtualbmc master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/virtualbmc/+/919340 | 21:00 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: WIP: reboot the node in basic ops tests https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/918462 | 21:08 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Add servicing states to states doc, fix state diagram https://review.opendev.org/c/openstack/ironic/+/919353 | 21:32 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!