| rpittau | good morning ironic! o/ | 06:45 |
|---|---|---|
| rpittau | as I said yesterday during the meeting, antelope is switching to unmaintained at the end of october | 07:16 |
| rpittau | all patches in 2023.1 are merged, but please double-check as I plan to do the final release before end of this week | 07:16 |
| opendevreview | Junya Noguchi proposed openstack/ironic master: Add image build method for verified OS. https://review.opendev.org/c/openstack/ironic/+/932396 | 09:49 |
| iurygregory | tonyb, hey! we can merge https://review.opendev.org/c/openstack/project-config/+/904012 o/ | 10:43 |
| iurygregory | oh nvm, it's already merged :D | 10:43 |
| iurygregory | good morning ironic | 10:44 |
| * iurygregory needs more coffee :D | 10:44 | |
| TheJulia | Word has it from today’s keynote speaker, nasa Goddard is going to rip out xcat and go to ironic | 13:50 |
| TheJulia | https://usercontent.irccloud-cdn.com/file/C2Qxt0vS/1729000278.JPG | 13:51 |
| rpittau | wow nice | 14:01 |
| TheJulia | That was Jonathan Mills on stage | 14:14 |
| TheJulia | https://usercontent.irccloud-cdn.com/file/jeN1rrjy/1729001774.JPG | 14:16 |
| TheJulia | More mention of things we care about :) | 14:16 |
| rpittau | yep :) | 14:18 |
| opendevreview | Michael Sherman proposed openstack/ironic stable/2023.1: allow disk cleaning during deploy https://review.opendev.org/c/openstack/ironic/+/932418 | 14:20 |
| cardoe | who was the first speaker TheJulia? | 14:31 |
| cardoe | ah Jonathan Mills. Reading comprehension fail. | 14:32 |
| rpittau | :) | 14:36 |
| rpittau | good night! o/ | 16:19 |
| TheJulia | So! University of Chicago has a snapshot thingie they want to make a service step | 16:24 |
| JayF | sounds like fun | 16:26 |
| TheJulia | They are going to try and submit their “horrible bash script” as a starting place | 16:52 |
| TheJulia | 43 people in James Denton's Ironic session.... | 17:37 |
| TheJulia | 4 on the zoom | 17:38 |
| TheJulia | Interesting…. https://usercontent.irccloud-cdn.com/file/MsyDyMMq/1729014383.JPG | 17:46 |
| TheJulia | Raid is separate apparently because of hot spare | 17:46 |
| TheJulia | https://usercontent.irccloud-cdn.com/file/VPBf7J7m/1729014434.JPG | 17:47 |
| iurygregory | There is zoom available?! | 17:49 |
| iurygregory | =O | 17:49 |
| TheJulia | https://usercontent.irccloud-cdn.com/file/M7SNQxnK/trim.EE6C197D-9DBD-4ED0-9541-F2EEB84718CC.mp4 | 17:49 |
| TheJulia | If you registered for NA… | 17:50 |
| TheJulia | :) | 17:50 |
| iurygregory | I didn't =( | 17:50 |
| TheJulia | They only made the hybrid announcement last week | 17:51 |
| TheJulia | And it still flew under the radar some, but it means we’re trying to record the sessions | 17:51 |
| iurygregory | nice =) | 17:51 |
| iurygregory | enjoy the OID NA o/ | 17:52 |
| cardoe | So I can answer stuff about James's piece around Ironic | 18:11 |
| cardoe | That first picture is what we do today. That's what I wanna get away from and do it all of out band inspection | 18:12 |
| cardoe | RAID is problematic because the syntax doesn't allow for expressing the configuration they have. | 18:12 |
| cardoe | We'll also have different users wanting different RAID setups and the make a new flavor scheme gets unwieldy. | 18:13 |
| cardoe | oh you said it... hot spare | 18:14 |
| TheJulia | I think we could declare that with out of band if the driver supports it, but software raid wise I kind of went “huh” | 18:15 |
| TheJulia | Hmm, looks like you just add a device | 18:17 |
| cardoe | So this gets into the sushy / sushy-oem-idrac | 18:17 |
| TheJulia | https://usercontent.irccloud-cdn.com/file/Rtdoj97U/1729016655.JPG | 18:24 |
| TheJulia | Everyone is doing ironic | 18:24 |
| TheJulia | cardoe: knowing the data in advance to generate the raid config? Or at least before the initial cleaning pass? | 18:24 |
| cardoe | So if I recall correctly we need to specify something specific about the disk for Dell to be happy and it's before the initial cleaning | 18:26 |
| cardoe | That's one that I need to get back to messing with. It's been a number of sleeps since I last looked. | 18:27 |
| TheJulia | Figures, your doing hardware with dell then. Yeah, I think the last time I did it, it had to be a spare disk out of the box | 18:34 |
| TheJulia | Which was clean in advance to not have corruption issues | 18:35 |
| cardoe | Believe me. I'd rather hit my fingers with a hammer every day then deal with Dell. | 18:44 |
| cardoe | The only thing he missed is that when we reset the BIOS we lose some settings and Ironic requires it to boot into IPA to change BIOS settings via redfish | 18:46 |
| TheJulia | wwwwwhhhhaaaattttt?!? | 19:05 |
| * TheJulia feels like this has created a RAIFSD | 19:05 | |
| TheJulia | Redundant Array of Independent Finger Smashing Devices | 19:05 |
| TheJulia | regarding OVN, we likely also need to be aware of https://bugs.launchpad.net/neutron/+bug/1995078 | 19:07 |
| cardoe | TheJulia: https://docs.openstack.org/ironic/latest/admin/drivers/idrac.html#pxe-reset-with-factory-reset-bios-clean-step | 19:29 |
| cardoe | https://opendev.org/openstack/ironic/src/commit/c80b8bfdb2eb18d49b049f093c8c79ffd5cac164/ironic/drivers/modules/redfish/bios.py#L189-L190 doesn't have requires_ramdisk=False | 19:34 |
| TheJulia | Easy to fix I guess :( | 19:41 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: devstack: respect USE_VENV in Ironic https://review.opendev.org/c/openstack/ironic/+/930776 | 21:01 |
| JayF | gmann: ^ I think that is updated in a way that it should make your grenade change happy, I'll depends-on it to check after I ensure it passes current-ci/grenade | 21:02 |
| iurygregory | JayF, https://review.opendev.org/c/openstack/ironic/+/930776/15/devstack/lib/ironic#1091 you changed ) for ( | 21:12 |
| JayF | fixed, ty | 21:13 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: devstack: respect USE_VENV in Ironic https://review.opendev.org/c/openstack/ironic/+/930776 | 21:13 |
| iurygregory | yw | 21:13 |
| keekz | hi all, i'm reading through https://docs.openstack.org/ironic/latest/admin/drivers/redfish/metrics.html but it's unclear how to actually enable the sending of the redfish metrics. in ironic config i have enabled sensor_data, configured metrics backend, configured oslo notifications, but nothing is making it to the oslo notifications rabbit queue. i see ironic conductor collecting sensor data, but it's not shipping it, and no | 21:55 |
| keekz | errors in logs | 21:55 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: devstack: respect USE_VENV in Ironic https://review.opendev.org/c/openstack/ironic/+/930776 | 21:57 |
| JayF | keekz: I think you need https://github.com/openstack/ironic-prometheus-exporter | 21:57 |
| JayF | hmm it should come out via notifications per that doc | 21:58 |
| JayF | what is your notification_level? | 21:58 |
| keekz | the docs are light, but i was under the impression the ironic-exporter read from the rabbit queue? | 21:58 |
| keekz | debug | 21:58 |
| JayF | I don't know much about our hardware metrics | 22:00 |
| JayF | I'm digging | 22:00 |
| keekz | before i added [metrics] backend to my ironic.conf, i was getting an error that i've since lost in scroll which said i needed to configure metrics. added the metrics section and no more errors... but no metrics either :) | 22:01 |
| JayF | Can you paste a redacted version of your config? | 22:03 |
| JayF | I suspect metrics config has nothing to do with this but imbw | 22:03 |
| JayF | well, both ways work I mean | 22:03 |
| JayF | also your [oslo_messaging_notifications] is setup, right? | 22:04 |
| keekz | well [metrics] was required or else it gives an error. let me make a gist | 22:07 |
| iurygregory | we had some changes if I recall | 22:08 |
| iurygregory | let me get the right config you should use | 22:08 |
| JayF | well lets look at the config they're using first :P | 22:09 |
| keekz | https://gist.github.com/nicholaskuechler/5b2d7cd183c8dee47d826b16caa53ac1 - pretty simple. fwiw once i get this working i'll update that redfish metrics page with the missing steps / configs | 22:09 |
| JayF | aha the real name did it for me, we worked together at rax, yeah? | 22:09 |
| keekz | and actually - i just found the notifications have arrived in rabbit, but in a different queue than the other notifications :) | 22:10 |
| JayF | keekz: you need transport_url and topics in that oslo_messaging_notifications section | 22:10 |
| iurygregory | driver = messagingv2 I think this is wrong | 22:10 |
| iurygregory | =) | 22:10 |
| iurygregory | yeah | 22:10 |
| keekz | yep that's me, jay :) | 22:10 |
| JayF | it's right, just needs to have topics/transport urls set | 22:10 |
| JayF | in that case I'll send your manager my hourly rate /s | 22:10 |
| iurygregory | [oslo_messaging_notifications] | 22:10 |
| iurygregory | driver = prometheus_exporter | 22:10 |
| iurygregory | transport_url = fake:// | 22:10 |
| iurygregory | location = /opt/stack/node_metrics | 22:10 |
| JayF | still at rax? | 22:10 |
| keekz | yep, and i'm working on doug's team. so you can just send him the bill | 22:10 |
| keekz | speaking of ironic exporter, is there a way to run it without a log file like that? i briefly looked in to it but our ironic is in kubernetes and it appeared to want actual log files, systemd services, etc. | 22:13 |
| iurygregory | we forgot to update the docs when we required the new config [metrics] | 22:13 |
| iurygregory | IPE is basically a flask app that will read all files in the location you have set | 22:14 |
| iurygregory | we receive the data from ironic and turn into a file for each node | 22:15 |
| keekz | yeah, i stumbled through that myself. the https://docs.openstack.org/ironic/latest/admin/drivers/redfish/metrics.html didn't mention any config options needed so i did some spelunking to find them | 22:15 |
| iurygregory | and later you can connect prometheus to scrap the data | 22:15 |
| keekz | sounds not very kubernetes friendly? | 22:15 |
| JayF | iurygregory: can you fix the doc? | 22:15 |
| iurygregory | I can | 22:15 |
| JayF | keekz: it's a k8s-style prometheus exporting flow bolted onto a statsd/message queue based metrics system | 22:16 |
| iurygregory | but keekz mentioned he would submit a patch if I understood | 22:16 |
| JayF | keekz: so ... yeah, it's not a perfect fit, but it's close | 22:16 |
| iurygregory | so I don't want to jump and send if he was planning on submitting the patch | 22:16 |
| iurygregory | keekz, are you using Metal3? | 22:16 |
| keekz | yes i can update docs on https://docs.openstack.org/ironic/latest/admin/drivers/redfish.html - are there other places to add to? | 22:17 |
| iurygregory | https://docs.openstack.org/ironic-prometheus-exporter/latest/configuration.html | 22:17 |
| keekz | no, not using metal3 | 22:17 |
| JayF | iurygregory: keekz is cardoe's environment fwiw :) | 22:17 |
| JayF | if you've heard him talk about his at all, that's where keekz is at | 22:17 |
| iurygregory | oh nice! | 22:18 |
| iurygregory | so rax is rackspace? | 22:18 |
| cardoe | yep | 22:18 |
| iurygregory | I was thinking it was the cloud provider we have in the infra for opendev :D | 22:18 |
| keekz | yep, 'rax' was the old stock ticker | 22:19 |
| iurygregory | because we have one called RAX :D | 22:19 |
| keekz | it is | 22:19 |
| cardoe | iurygregory: it is. | 22:19 |
| iurygregory | OMG | 22:19 |
| iurygregory | =O | 22:19 |
| keekz | different team, same company | 22:19 |
| iurygregory | good to know | 22:19 |
| keekz | although we've done a lot of work over the years on that environment as well :) | 22:19 |
| cardoe | The new stuff is called "flex" I think it's called raxflex or rxtflex in the openinfra configs. | 22:20 |
| iurygregory | keekz, if you have any trouble with the IPE you can directly ping me, since I was the one who wrote it o/ | 22:21 |
| JayF | When I worked at RAX, we got in trouble for calling it RAX | 22:23 |
| JayF | because they said people could accidentally think you're talking about the stock, not the company | 22:23 |
| cardoe | So keekz is doing the needful for kicking the tires and doing some of the short-term functionality. But long term I think we'd probably look to help extend to be a bit more Prometheus native or catching that event stream directly. | 22:23 |
| JayF | now the stock ticker is RXT and they don't pay me a dime, so I can call it RAX all day long :P | 22:23 |
| JayF | cardoe: hooking into the notifications stream is the key there I think | 22:23 |
| JayF | cardoe: you aware of https://github.com/openstack-exporter/openstack-exporter | 22:24 |
| cardoe | yeah keekz is running that now | 22:24 |
| iurygregory | I wasn't aware <eyes> | 22:24 |
| keekz | yeah i set up openstack-exporter, that was pretty straight forward | 22:24 |
| cardoe | speaking of openstack-exporter, I proposed fixing the auth stuff in gophercloud more fully. stephenfin joined in the convo. It'll likely be a v3 thing. | 22:25 |
| cardoe | Basically I said that internally they need to treat everything like a clouds.yaml entry and auth based on that entry. | 22:25 |
| keekz | does that replace ironic-exporter? i haven't really done a deep dive except to get it to do what i wanted, which was to hook in to prometheus for basic api up/down alerts | 22:25 |
| iurygregory | keekz, not that I'm aware ... | 22:25 |
| iurygregory | but it does have ironic metrics | 22:26 |
| iurygregory | but not the ones we collect from the bmc with sensor data | 22:26 |
| cardoe | So another thing we'd want to do is have redfish eventing be part of that pipeline of data. | 22:26 |
| keekz | yeah it's more of what you can see in an `openstack baremetal node list` which is useful, but we were wanting to investigate some of the hardware health info redfish can provide | 22:27 |
| iurygregory | humm interesting | 22:27 |
| cardoe | So if the hardware tells us its bad, put that on the notification stream as well. | 22:27 |
| iurygregory | I added the event subscription to redfish is via vendor passtru (and probably need update if things changed in the redfish schema) | 22:27 |
| iurygregory | but is not integrated with IPE | 22:28 |
| cardoe | yeah I figure we'll need to commit some time to experimenting and then drafting some RFEs and specs around what would work. | 22:29 |
| cardoe | The current hardware monitoring stacks (cause there's many different things) all live outside of Ironic | 22:30 |
| iurygregory | yeah | 22:32 |
| cardoe | But we look at it all and there's commonality between them and with the improvements in redfish the idea is could we do something generic inside of Ironic that can be built on. | 22:32 |
| iurygregory | also, try to not lower the interval to collect data a lot, otherwise you can cause problems to the BMC (but I think you are aware) | 22:32 |
| iurygregory | cardoe, totally agree | 22:33 |
| keekz | yeah some of the bmcs are painfully slow. in some of the ironic vendor docs i did find some performance tuning settings though 👍 | 22:38 |
| keekz | i'm working on updating those docs (both the ironic-exporter and ironic redfish metrics) but it's getting late for me, so i'll have something for review tomorrow | 22:47 |
| opendevreview | Nicholas Kuechler proposed openstack/ironic-prometheus-exporter master: docs: Updates configuration documentation https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/932458 | 23:18 |
| opendevreview | Nicholas Kuechler proposed openstack/ironic-prometheus-exporter master: docs: Updates configuration documentation https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/932458 | 23:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
