| cardoe | JayF: speaking of Ubuntu. Need to add Qemu-Utils to the docs. This is a reminder for me tomorrow. | 02:48 |
|---|---|---|
| masghar | Thanks for the discussion around python-ironicclient and openstacksdk - I see that there is some benefit to keeping python-ironicclient SDK code around, and its not really a community goal to remove it - so I got my answers :) | 10:11 |
| masghar | But long-term, I think it would be beneficial to point all our development efforts and improvements in one direction, really | 10:14 |
| opendevreview | OpenStack Release Bot proposed openstack/tenks stable/2.0: Update .gitreview for stable/2.0 https://review.opendev.org/c/openstack/tenks/+/927159 | 11:07 |
| opendevreview | OpenStack Release Bot proposed openstack/tenks stable/2.0: Update TOX_CONSTRAINTS_FILE for stable/2.0 https://review.opendev.org/c/openstack/tenks/+/927160 | 11:07 |
| opendevreview | OpenStack Release Bot proposed openstack/tenks master: Update master for stable/2.0 https://review.opendev.org/c/openstack/tenks/+/927161 | 11:07 |
| iurygregory | good morning ironic | 11:08 |
| opendevreview | Pierre Riteau proposed openstack/tenks master: Update list of supported Linux distributions https://review.opendev.org/c/openstack/tenks/+/933850 | 11:10 |
| jovial | Any chance of https://review.opendev.org/c/openstack/ironic/+/924887 making it back to 2024.1? It is labelled as a bug, but needs nova changes ported back too (which seem like more of a feature) | 11:35 |
| opendevreview | Merged openstack/tenks master: Add Python 3.12 job https://review.opendev.org/c/openstack/tenks/+/933275 | 12:03 |
| opendevreview | Merged openstack/tenks master: Update master for stable/2.0 https://review.opendev.org/c/openstack/tenks/+/927161 | 12:03 |
| opendevreview | Verification of a change to openstack/tenks master failed: Preserve assigned MAC on subsequent invocations https://review.opendev.org/c/openstack/tenks/+/931816 | 12:13 |
| opendevreview | Will Szumski proposed openstack/tenks master: Adds a release note for mac address bugfix https://review.opendev.org/c/openstack/tenks/+/933856 | 12:13 |
| opendevreview | Merged openstack/tenks master: Update list of supported Linux distributions https://review.opendev.org/c/openstack/tenks/+/933850 | 12:21 |
| opendevreview | Will Szumski proposed openstack/tenks master: Remove deprecated use of include https://review.opendev.org/c/openstack/tenks/+/933858 | 12:28 |
| opendevreview | Will Szumski proposed openstack/tenks master: Adds a release note for mac address bugfix https://review.opendev.org/c/openstack/tenks/+/933856 | 12:47 |
| opendevreview | Will Szumski proposed openstack/tenks master: Preserve assigned MAC on subsequent invocations https://review.opendev.org/c/openstack/tenks/+/931816 | 12:48 |
| TheJulia | good morning | 13:36 |
| TheJulia | masghar: agree, and I think that direction is largely the openstacksdk. There are folks who really want to push for the elimination of all clients for perception reasons, but Ironic is definitely an "odd" project in some respects where it doesn't make as much sense | 13:37 |
| JayF | jovial: that is considered a feature and is ineligible for backport. And you're right that it wouldn't work without the accompanying Nova patch anyway, but we don't even have to get that far with this one because it's a feature on our end too | 14:04 |
| masghar | TheJulia: ironic is definitely not too simple, so its own client for certain use cases does make sense | 14:23 |
| opendevreview | Merged openstack/tenks master: Remove deprecated use of include https://review.opendev.org/c/openstack/tenks/+/933858 | 14:37 |
| jovial | Cheers JayF - I thought that might be the case. Nice feature though :) | 14:38 |
| JayF | Thanks! It's nice to hear other people have that use case too. | 14:39 |
| cardoe | shermanm: I'd definitely be interested in your experience with the custom hardware manager. | 15:38 |
| cardoe | free internet points for reviews and workflows for https://review.opendev.org/q/topic:%22remove-py38%22+hashtag:ironic-week-prio+status:open | 15:42 |
| opendevreview | Pierre Riteau proposed openstack/tenks stable/2.0: Update TOX_CONSTRAINTS_FILE for stable/2.0 https://review.opendev.org/c/openstack/tenks/+/927160 | 16:15 |
| JayF | so update on virtualpdu: Itamar told me he has the actual program working, just down to testsuite fixes now | 16:19 |
| JayF | so the SNMP driver should be removed from the chopping block for the time being \o/ | 16:20 |
| JayF | > This specification document, which someone should take over < gave me a real lol, I think I'm going to put that line in the top of the commit message for every spec I write from now on :D | 16:20 |
| *** edebeste2 is now known as edebeste | 16:35 | |
| opendevreview | Merged openstack/ironic master: Added missing fields in API Ref https://review.opendev.org/c/openstack/ironic/+/933699 | 16:36 |
| JayF | Found a new trick to make pre-commit more useful now that more openstack repos are using it: https://jay.jvf.cc/posts/openstack-pre-commit/ tl;dr you can setup the pre-commit hooks to automatically go into new repos | 16:39 |
| opendevreview | Pierre Riteau proposed openstack/tenks stable/2.0: Update TOX_CONSTRAINTS_FILE for stable/2.0 https://review.opendev.org/c/openstack/tenks/+/927160 | 16:46 |
| clarkb | JayF: one thing to keep in mind is that hooks are not installed automatically by default in git because it would allow repos to perform local code execution | 16:51 |
| clarkb | depending on what the repo is you may not want the hook to automatically get installed | 16:51 |
| JayF | yeah, that's the exception not the rule | 16:51 |
| JayF | so I'd rather remember to delete it then | 16:51 |
| JayF | I can't think of a single time, ever, in my career I've committed code to a repo I haven't been involved enough in to see what it would do though | 16:52 |
| clarkb | ya I dunno I have up my fancy git status stuff in my prompt because it too has been abused for these purposes... But I also sometimes have to fetch random repos/commits to debug things and thats probably not the case for you | 16:55 |
| JayF | I haven't committed to a repo that wasn't openstack, gentoo, or g-research related in ... as long as I can remember? | 16:55 |
| JayF | (or my own, I guess, my blogpost is a repo) | 16:56 |
| cardoe | If you don't wanna sudo stuff JayF. Set core.hooksPath in $HOME/.gitconfig to something in your user | 17:08 |
| JayF | yeah, I def put the heaviest handed method in there | 17:09 |
| JayF | if I was going to take more than 15 minutes to solve my problem+blog I'd probably do something more like a hook that was "if you have a pre-commit config but no pre-commit hook; angry message and exit" | 17:09 |
| JayF | 90% of the value with 5% of the effort is something I'm usually not good at, exercising discipline this time at least :D | 17:10 |
| opendevreview | Merged openstack/ironic-tempest-plugin master: Remove support for Python 3.8 and older https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/933266 | 17:17 |
| opendevreview | Merged openstack/ironic-prometheus-exporter master: docs: Updates configuration documentation https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/932458 | 17:22 |
| opendevreview | Merged openstack/virtualpdu master: Remove Python 3.8 support https://review.opendev.org/c/openstack/virtualpdu/+/933260 | 17:24 |
| opendevreview | Merged openstack/ironic-ui master: Remove Python 3.8 support https://review.opendev.org/c/openstack/ironic-ui/+/933258 | 17:31 |
| opendevreview | Merged openstack/python-ironic-inspector-client master: Remove Python 3.8 support https://review.opendev.org/c/openstack/python-ironic-inspector-client/+/933232 | 17:33 |
| opendevreview | Merged openstack/ironic-inspector master: Remove Python 3.8 support https://review.opendev.org/c/openstack/ironic-inspector/+/933228 | 17:41 |
| opendevreview | Merged openstack/ironic-python-agent master: Remove Python 3.8 support https://review.opendev.org/c/openstack/ironic-python-agent/+/933230 | 17:44 |
| opendevreview | Merged openstack/sushy-tools master: add pyproject.toml to support pip 23.1 https://review.opendev.org/c/openstack/sushy-tools/+/933594 | 17:45 |
| opendevreview | Verification of a change to openstack/ironic-lib master failed: Replace outdated python job template https://review.opendev.org/c/openstack/ironic-lib/+/933242 | 17:47 |
| opendevreview | Verification of a change to openstack/ironic-lib master failed: Remove Python 3.8 support https://review.opendev.org/c/openstack/ironic-lib/+/933229 | 17:47 |
| opendevreview | Merged openstack/sushy master: Make default request timeout configurable on Connector https://review.opendev.org/c/openstack/sushy/+/929055 | 17:54 |
| shermanm | so i'm unfortunately still running into issues with that custom hardware manager (code here https://github.com/ChameleonCloud/chi-hardware-manager/blob/main/chi_hardware_manager/fugaku.py) | 17:55 |
| shermanm | I do see log entries indicating that my manager loaded | 17:55 |
| shermanm | but ipmitool is still running and timing out | 17:56 |
| JayF | please supply full logs? | 17:56 |
| shermanm | sure, let me put a paste somewhere | 17:56 |
| JayF | Also, I'd likely run a test with _detect_hardware() implemented as "Return True" if you haven't already | 17:57 |
| JayF | just to limit the amount of moving parts | 17:57 |
| shermanm | logs are here: https://gist.githubusercontent.com/msherman64/e0b5d1609d2c3effc969e64180114858/raw/eeaae77de0f3d31d3a01f94a7cf9f272ee67b866/agent.log | 17:58 |
| shermanm | and I do see my log entry "found Fugaku node! {'sys_vendor': 'FUJITSU\n', 'product_name': 'FX700\n'}", which is just before the return | 17:59 |
| shermanm | and also get "Hardware manager found: chi_hardware_manager.fugaku:FugakuHardwareManager" | 17:59 |
| JayF | > Oct 31 17:41:28 ironic-agent ironic-python-agent[1095]: 2024-10-31 17:41:28.283 1095 INFO ironic_python_agent.inspector [-] inspection is enabled with collectors ['default', 'extra-hardware', 'pci-devices', 'dmi-decode', 'numa-topology', 'logs'] | 18:00 |
| JayF | makes me wonder if this is inspector, not the other code, that's doing it this time | 18:00 |
| shermanm | that's a good point actually, I can dig into that, or at least test if the behavior is the same with deploy/cleaning | 18:01 |
| JayF | yeah, lets figure out if it's happening with no inspection collectors enabled | 18:03 |
| JayF | if not; then we have problem 1 solves and can move to problem 2 | 18:04 |
| shermanm | +! | 18:04 |
| shermanm | yep, sounds like a plan | 18:04 |
| shermanm | I should probably also add debug logs to my overridden methods, to be sure they're getting executed | 18:05 |
| JayF | ++ | 18:06 |
| opendevreview | Merged openstack/python-ironicclient master: Replace outdated python job template https://review.opendev.org/c/openstack/python-ironicclient/+/933244 | 18:06 |
| opendevreview | Merged openstack/python-ironicclient master: Remove Python 3.8 support https://review.opendev.org/c/openstack/python-ironicclient/+/933231 | 18:06 |
| opendevreview | Itamar Turner-Trauring proposed openstack/virtualpdu master: Work with pysnmp 7.0.1 https://review.opendev.org/c/openstack/virtualpdu/+/933882 | 18:12 |
| opendevreview | Merged openstack/virtualbmc master: Remove Python 3.8 support https://review.opendev.org/c/openstack/virtualbmc/+/933263 | 18:14 |
| JayF | rpittau: ^^^ https://review.opendev.org/c/openstack/virtualpdu/+/933882 line 276 in pysnmp_handler.py makes everything make SO MUCH SENSE | 18:15 |
| JayF | rpittau: seeing itamar's fix made my mental model click into place | 18:16 |
| shermanm | is there an easy way to edit the contents of my initramfs instead of kicking off a dib-build every time? | 18:19 |
| opendevreview | Takashi Kajinami proposed openstack/python-ironicclient stable/2024.2: Replace outdated python job template https://review.opendev.org/c/openstack/python-ironicclient/+/933883 | 18:24 |
| JayF | whether or not it's easier is an exercise for the reader (I do not think it's easier), but you can extract and re-assemble initramfs files | 18:24 |
| JayF | using cpio | 18:24 |
| JayF | I wonder if we have how to do that in contributor docs, probabyl not? | 18:24 |
| JayF | doc/source/admin/troubleshooting.rst:179:39: $ gzip -dc /path/to/the/ramdisk | cpio -id | 18:24 |
| JayF | bingo | 18:25 |
| JayF | https://docs.openstack.org/ironic/latest/admin/troubleshooting.html#patching-the-deploy-ramdisk shermanm | 18:25 |
| shermanm | bam, I thought I'd seen it somewhere, thanks! | 18:26 |
| shermanm | I'm cross-compiling for arm64, so the build process is super-slow | 18:26 |
| JayF | oh yeah, that can be painful | 18:27 |
| JayF | I guess I don't think I've actually "cross-compiled" for arm64 directly; I have used the gentoo trick of qemu + binfmt_misc -> https://wiki.gentoo.org/wiki/Embedded_Handbook/General/Compiling_with_QEMU_user_chroot | 18:28 |
| JayF | basically it sets up your system so any arm64 binary automatically runs under qemu | 18:28 |
| JayF | so you can extract full OS images and chroot into them cross-arch | 18:28 |
| JayF | (those instructions, like many in gentoo wiki, are easily adaptable to other situations) ... not that this is useful for you, it's just cool :D | 18:29 |
| shermanm | I should say that's what I actually meant, using that support that's currently in diskimage-builder for that | 18:29 |
| JayF | I didn't realize dib used that style too | 18:29 |
| JayF | I've not looked at any of the cross-arch stuff in dib | 18:30 |
| shermanm | at least it does with the debootstrap-based sources | 18:30 |
| JayF | that's pretty cool | 18:30 |
| JayF | and makes double sense why it's so slow, too | 18:30 |
| JayF | real cross compilation isn't that bad of a penalty | 18:30 |
| clarkb | you can do similar with qcow2 images too via nbd or you can convert to raw and mount that and then back again etc | 18:38 |
| clarkb | the dib approach is very heavy handed because it gives you a lot of flexibility in what you acn do, but for specific things you can usually take a short cut | 18:39 |
| clarkb | there are image build tools out that there don't do any root and basically byte edit the filesystem out in userspace | 18:40 |
| opendevreview | Verification of a change to openstack/ironic-lib master failed: Drop the unused capabilities module https://review.opendev.org/c/openstack/ironic-lib/+/933029 | 18:45 |
| shermanm | I got debug logging turned on too, and I think I found at least a symptom: "HardwareManager <chi_hardware_manager.fugaku.FugakuHardwareManager object at 0xffff84041690> does not support list_hardware_info dispatch_to_managers" | 18:50 |
| JayF | yeah that's your problem, or at least a big'un | 18:51 |
| JayF | I don't know /why/ it'd think that | 18:51 |
| JayF | shermanm: dalmatian? | 18:52 |
| shermanm | no, we're still back on antelope | 18:52 |
| JayF | ack | 18:53 |
| JayF | btw that's about to turn into a pumpkin (unmaintained) | 18:54 |
| shermanm | yeah, just scrambling to the dev time to pull all of our forks up to caracal | 18:54 |
| JayF | https://opendev.org/openstack/ironic-python-agent/src/branch/stable/2023.1/ironic_python_agent/hardware.py#L3088 | 18:54 |
| JayF | so IncompatibleHardwareMethodError is what it /thinks/ it got returned | 18:55 |
| JayF | because that's the only way to get that log | 18:55 |
| JayF | which means it called that method, that method went boom | 18:55 |
| JayF | OH OH I bet I know | 18:56 |
| JayF | this is crazypants, but I have your answer | 18:56 |
| JayF | lets walk thru it: | 18:56 |
| JayF | in master, GenericHardwareManager(HardwareManager) is what gets called | 18:56 |
| JayF | even though it's using the implementation of list_hardware_info() it inherited | 18:56 |
| JayF | not all of those self.BLAH() methods called for that info are | 18:57 |
| JayF | many of those are coming from the GenericHardwareManager implementation | 18:57 |
| opendevreview | Merged openstack/networking-baremetal master: Remove Python 3.8 support https://review.opendev.org/c/openstack/networking-baremetal/+/933273 | 18:57 |
| JayF | because that's the "self" in that case | 18:57 |
| JayF | if you follow | 18:57 |
| JayF | so, when self.get_cpus() is called from your HWM on line 86 | 18:58 |
| JayF | it's calling this: https://opendev.org/openstack/ironic-python-agent/src/branch/stable/2023.1/ironic_python_agent/hardware.py#L863 | 18:58 |
| JayF | which returns the failure | 18:58 |
| JayF | all this is to say: subclass from GenericHardwareManager instead for this one case | 18:58 |
| shermanm | ugh, yeah, that's making sense now | 18:58 |
| JayF | OR rewrite those self.blah() calls to dispatch_to_manager() style calls so they can be farmed out to other managers | 18:58 |
| JayF | ^^^ that is likely a fix I will write into master IPA if/when I get a chance (and I'm less likely to forget it if you file a bug) | 18:58 |
| shermanm | aand of course my unit tests didn't find those cause I mocked them, wasn't running on real hardware | 18:59 |
| opendevreview | Merged openstack/ironic master: CI: Change 4k block device job to use LVM https://review.opendev.org/c/openstack/ironic/+/931061 | 18:59 |
| JayF | this was a fun little one to debug | 18:59 |
| shermanm | love the excitement when you realized lol | 18:59 |
| JayF | I really don't udnerstand why list_hardware_info looks the way it does | 18:59 |
| JayF | shermanm: history lesson: I'm a converted operator, and this hardwaremanager stuff is literally the first API of any kind I designed end to end | 18:59 |
| JayF | (11 years ago at this point?) | 18:59 |
| JayF | so I was happy to realize it all fit :D (and this is why I'm so annoyed that's not already dispatching those calls) | 19:00 |
| shermanm | riiight there with you on being a converted operator | 19:01 |
| shermanm | side note, HardwareManager.list_hardware_info calls "self.get_system_vendor_info()", but that method is only defined on GenericHardwareManager | 19:02 |
| JayF | +++ yep exactly why it should dispatch | 19:03 |
| JayF | and we likely should add that method to the base class, too | 19:03 |
| JayF | I am going to go grab lunch and work on my SCALE CfP before I get too distracted -- I will try to circle back around to fixing that weirdness in master IPA. | 19:04 |
| shermanm | no worries, I appreciate the help! | 19:04 |
| opendevreview | Merged openstack/ironic-lib master: Replace outdated python job template https://review.opendev.org/c/openstack/ironic-lib/+/933242 | 20:30 |
| shermanm | that pointer totally worked, I'm all set now | 21:14 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-lib master: Drop the unused capabilities module https://review.opendev.org/c/openstack/ironic-lib/+/933029 | 22:01 |
| opendevreview | Merged openstack/ironic-lib master: Remove Python 3.8 support https://review.opendev.org/c/openstack/ironic-lib/+/933229 | 22:07 |
| opendevreview | Jay Faulkner proposed openstack/python-ironicclient master: WIP: Support /v1/shards https://review.opendev.org/c/openstack/python-ironicclient/+/933894 | 22:51 |
| * JayF added support for node.shard + filtering on shard but never for /v1/shards :( | 22:54 | |
| cardoe | What’s shards? | 22:55 |
| JayF | https://specs.openstack.org/openstack/nova-specs/specs/2024.1/implemented/ironic-shards.html although I'm pretty sure we have ironic side docs somewhere | 22:57 |
| JayF | if not, another box I need to check apparently that I missed before | 22:57 |
| JayF | https://files.openstack.org/project/specs.openstack.org/openstack/ironic-specs/specs/approved/shard-key.html | 22:58 |
| JayF | that's the ironic spec | 22:58 |
| JayF | tl;dr just an arbitrary field you set on node objects (and maybe one day things like ports?) that is indexed on | 22:58 |
| JayF | so if you have to do any /v1/nodes (no filter) querying, you can instead do e.g. /v1/nodes?shard=shard1 /v1/nodes?shard=shard2 | 22:58 |
| JayF | useful for things like nova-compute which has a different scaling model than Ironic | 22:59 |
| cardoe | Yeah makes sense. | 22:59 |
| JayF | (an ironic conductor group can outscale a nova-compute by an order of magnitude or more in some use cases) | 22:59 |
| JayF | and basically before this, folks were using conductor groups as psuedo-shards | 22:59 |
| JayF | which sorta eliminates their usefulness for what they are for | 22:59 |
| cardoe | That’s what I’ve done. | 23:00 |
| cardoe | A nova-compute has one group. | 23:00 |
| cardoe | But clearly I need to look at shards. | 23:00 |
| JayF | I mean, if it ain't broke don't fix it, but if you need to fix it, you can now :) | 23:00 |
| opendevreview | Verification of a change to openstack/ironic-lib master failed: Drop the unused capabilities module https://review.opendev.org/c/openstack/ironic-lib/+/933029 | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!