| opendevreview | Julia Kreger proposed openstack/sushy master: Retry on ilo state error https://review.opendev.org/c/openstack/sushy/+/880542 | 01:19 |
|---|---|---|
| rpittau | good morning ironic! o/ | 07:48 |
| rpittau | ok, we have yet another problem...... | 08:51 |
| rpittau | pyasn1 update to version 0.5.0 breaks pysnmp compatibility, which for a very sad reason is not maintained since 4 years | 08:51 |
| rpittau | we should probbaly move to pysnmplib https://github.com/pysnmp/pysnmp but I guess that needs to be added to openstack requirements | 08:51 |
| rpittau | I'm getting ahead and working on adding pysnmplib to openstack requirements | 09:01 |
| rpittau | then need to see to update the requirements of our projects from pysnmp to pysnmplib | 09:01 |
| rpittau | if we agree on this of course, but I don't see a lot of alternatives here | 09:01 |
| opendevreview | Riccardo Pittau proposed openstack/virtualpdu master: Migrate to pysnmplib https://review.opendev.org/c/openstack/virtualpdu/+/881538 | 09:05 |
| rpittau | this is the requirements change https://review.opendev.org/c/openstack/requirements/+/881539 | 09:08 |
| opendevreview | Riccardo Pittau proposed openstack/virtualpdu master: Migrate to pysnmplib https://review.opendev.org/c/openstack/virtualpdu/+/881538 | 09:08 |
| dtantsur | rpittau: can we make the snmp job non-voting for the time being? | 09:10 |
| dtantsur | rpittau: we'll probably get into a similar problem with the SNMP driver | 09:11 |
| rpittau | The problem is that we have a lot of unit tests depending on pysnmp that fail because of the recent pyasn1 update | 09:22 |
| dtantsur | rpittau: sign. so we need to do EVERYTHING in a lock-step? | 09:37 |
| rpittau | Yeah.... We could temporarily ask to revert pyasn1 upper constraints to 0.4.8, but that's just postponing the actual fix :/ | 09:39 |
| opendevreview | Vanou Ishii proposed openstack/ironic master: [iRMC] Fix parse_driver_info bug enforcing SNMP v3 under FIPS mode https://review.opendev.org/c/openstack/ironic/+/881358 | 10:41 |
| iurygregory | morning ironic | 11:22 |
| iurygregory | oh jesus regarding pyasn1 | 11:24 |
| rpittau | we probably need to revert to 4.8.0 temporarily anyway because of https://github.com/pyasn1/pyasn1/issues/28 | 12:27 |
| rpittau | or hope for a quick 5.0.1 | 12:27 |
| rpittau | ehm | 12:27 |
| rpittau | that wuold be 0.4.8 and 0.5.1 | 12:27 |
| dtantsur | \o/ | 12:27 |
| dtantsur | in any case, we need to exclude the broken combination from the requirements | 12:28 |
| rpittau | yeah, with pysnmplib at least we should be ok on the snmp side, then we need a new pyasn1 release | 12:29 |
| rpittau | mmmm actually nope, the issue is not fixed there :/ | 12:38 |
| rpittau | going to ask now to pin pyasn1 version to 0.4.8 | 12:39 |
| opendevreview | Riccardo Pittau proposed openstack/virtualpdu master: Migrate to pysnmplib https://review.opendev.org/c/openstack/virtualpdu/+/881538 | 12:44 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Use jammy for base jobs https://review.opendev.org/c/openstack/ironic/+/869052 | 12:49 |
| rpittau | ^ disabling snmp job here to make CI pass until we have pyasn1 0.4.8 again or a fix | 12:50 |
| rpittau | I've requested to revert pyasn1 update https://review.opendev.org/c/openstack/requirements/+/881556 | 13:01 |
| rpittau | pysnmlib and pyasn1 0.4.8 seem to work fine together, at least in https://review.opendev.org/c/openstack/virtualpdu/+/881538 | 13:18 |
| TheJulia | ugh, pysnmplib looks like the splunk fork | 13:38 |
| rpittau | TheJulia: not sure we have an alternative :/ | 13:43 |
| TheJulia | I'm kind of hoping ?lexito? gets control of the pysnmp namespace, but given the lack of reply... dunno | 14:01 |
| TheJulia | well, s/namespace/packages and related packages/ | 14:02 |
| rpittau | lextudio? | 14:02 |
| TheJulia | I think | 14:02 |
| TheJulia | I'm still waking up | 14:02 |
| rpittau | yeah that is actually an alternative :D | 14:02 |
| rpittau | and a good one | 14:02 |
| iurygregory | dtantsur, regarding https://review.opendev.org/c/openstack/ironic-specs/+/878505/9/specs/approved/firmware-interface.rst, basically you mean that each implementation can define if it will be in-band out-of-band or I'm wrong? | 14:03 |
| TheJulia | https://pypi.org/user/lextm | 14:05 |
| rpittau | yep | 14:06 |
| rpittau | I wonder if we could move to its release of pyasn1 | 14:06 |
| TheJulia | I really admire their passion, I wouldn't have said something on the pypi pep 541 request otherwise | 14:06 |
| rpittau | mmmm probably not, pyasn1 is still a requirement for that | 14:07 |
| TheJulia | so your just focused on virtualpdu right? | 14:08 |
| rpittau | that and all snmp at this point | 14:08 |
| TheJulia | so in both proliantutils and the irmc driver? | 14:09 |
| rpittau | anywhere we use pysnmp, yeah | 14:09 |
| TheJulia | yeah :( | 14:09 |
| rpittau | btw this is green now, good sign https://review.opendev.org/c/openstack/virtualpdu/+/881538 | 14:09 |
| TheJulia | That is, just not a real fan of using the splunk fork... although there seems to be some lost in translation happening in the back and forth posts | 14:10 |
| rpittau | if you'd rather go for pysnmp-lextudio I don't really have anything against it, I chose pysnmplib because it's the one I was following, but the other choice looks as good | 14:13 |
| TheJulia | We should likely rope in vanou and also try to reach out to stendulker | 14:14 |
| TheJulia | because we should determine the forward path in unison | 14:14 |
| TheJulia | ... as we don't want namespace conflicts with packages | 14:14 |
| rpittau | right | 14:15 |
| rpittau | I blocked the change in requirements for the time being, if we at least downgrade pyasn1 we should be fine for the time being | 14:18 |
| * TheJulia nods | 14:18 | |
| dtantsur | iurygregory: more of: we have a common in-band implementation that is shared by all implementations | 14:22 |
| dtantsur | so, 'agent' firmware_interface similar to 'agent' RAID which only does in-band (we'll need to settle on IPA-level APIs for that) | 14:22 |
| dtantsur | but then, 'redfish' firmware_interface eventually also supports going to the agent | 14:23 |
| dtantsur | unless we want to start doing interface composition :D | 14:23 |
| * TheJulia is worried we need an Aeva to appear with whiskey | 14:24 | |
| TheJulia | :) | 14:24 |
| rpittau | lol | 14:25 |
| * iurygregory reading | 14:25 | |
| rpittau | TheJulia: I just did a test with lextudio pysnmp and pyasn1, it works just fine, so I think my vote at the moment goes to lextudio implementation | 14:26 |
| dtantsur | TheJulia: great idea :) | 14:26 |
| iurygregory | dtantsur, let me see if I understood, with the firmware_interface we would implement "AgentFirmware" that would be similar to the "AgentRAID" (only does in-band) this part I think its clear for me | 14:29 |
| TheJulia | sounds correct to me | 14:30 |
| iurygregory | I'm trying to understand the redfish firmware_interface supporting going to the agent... | 14:30 |
| iurygregory | <thinking face> | 14:30 |
| TheJulia | so.... | 14:30 |
| TheJulia | well | 14:30 |
| TheJulia | shoot | 14:30 |
| TheJulia | The underlying challenge is what if you need both | 14:31 |
| TheJulia | and maybe that is the rare 10% as opposed to the 90% | 14:31 |
| TheJulia | well, maybe more like 2% | 14:31 |
| TheJulia | agent firmware application could still be wired up through deploy though... | 14:32 |
| TheJulia | feels awful but if someone has the steps and needs to do a one off action, they likely need to do deploy time anyway as a manual action | 14:32 |
| TheJulia | or automated via template action | 14:32 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove use of nomodeset by default https://review.opendev.org/c/openstack/ironic/+/881576 | 14:37 |
| TheJulia | if anyone has any questions w/r/t ^ lmk | 14:37 |
| iurygregory | if the operator wants to run firmware upgrade for one component via in-band and other component via oob, wouldn't it just be a logic in the implementation of redfish firmware interface? like we receive the steps, we can have a flag that would make we call the AgentFirmware ... | 14:37 |
| zigo | Hi there! How experimental is the split ironic-inspector-api / conductor ? Can I try it ? (I'm running under Zed) | 14:37 |
| zigo | If I understand well, I have the choice between: ironic-inspector non-wsgi thing, which includes both api + conductor, or run the api over let's say uwsgi + conductor on a separate service, right? | 14:38 |
| dtantsur | zigo: keeping in mind that we're working on merging inspector into ironic, I think RH OSP folks are using that. | 14:38 |
| TheJulia | iurygregory: I mean, maybe that also makes sense as well/? | 14:38 |
| TheJulia | .. I think we still do single process. I think | 14:38 |
| TheJulia | I've known a few people mention splitting it, but it just adds complexity which many don't need | 14:39 |
| dtantsur | TheJulia, iurygregory, given the list of components, an implementation (say, redfish) can know if the steps are out-of-band (they're in the code) or in-band | 14:39 |
| zigo | TheJulia: I probably do if I want to use uwsgi as a front-end (also for doing the SSL termination). | 14:39 |
| TheJulia | dtantsur: seems reasonable which is basically what jury was kind of raising | 14:39 |
| dtantsur | yeah | 14:41 |
| TheJulia | zigo: I think that was one of the benefit cases, I guess the question becomes how much traffic is a driver of if it makes sense or not otherwise, but offloading ssl makes sense, I just think you'd be an outlier user of it that way. Not a bad thing, just... you may encounter unexpected things. Granted the code path is basically the same *either* way so I wouldn't really be *too* worried | 14:41 |
| dtantsur | zigo: the bifrost approach probably won't work for a proper OpenStack cloud, but we make inspector and ironic listen on a unix socket, with nginx providing TCP+TLS | 14:42 |
| TheJulia | i.e. if it starts and basically functions, I wouldn't be worried about other things suddenly being horribly broken | 14:42 |
| zigo | TheJulia: Thanks. FYI, I'll at least try that path first. | 14:42 |
| zigo | dtantsur: I use haproxy + corosync/pacemaker doing a VIP in front to do the user-side front-end, though I do re-encryption to each individual service using an internal cluster PKI, and then I make uwsgi handle that ... | 14:43 |
| dtantsur | fancy! this is very far from my standalone world :) | 14:44 |
| zigo | It's best if I can use uwsgi, if not, I can fallback to having haproxy in each API node to do the internal PKI decryption, like I did for glance... | 14:44 |
| zigo | Anyways, will try the conductor + api in separate process way, and see how it leads me! :) | 14:45 |
| iurygregory | dtantsur, TheJulia ok, so I think I just need to update https://review.opendev.org/c/openstack/ironic-specs/+/878505/9/specs/approved/firmware-interface.rst removing "Initially we will only support Redfish and updates via OOB." and maybe mentioning the "AgentFirmware" implementation? | 14:50 |
| dtantsur | iurygregory: if you go down this path, you will also need to specify the agent-side API. Which is going to increase the scope of the spec, but I assume JayF would prefer to see the whole scope there? | 14:51 |
| iurygregory | oh right | 14:51 |
| JayF | I want us to make sure a path exists to an in-band implementation. I am 100% OK with not solidifying that path. | 14:51 |
| dtantsur | ah | 14:52 |
| JayF | I just noticed we haven't done an agent-based interface on something this complex before (like I mentioned, we punted on bios interface in-band) | 14:52 |
| dtantsur | iurygregory: then I think you okay if you provide a high-level idea and omit any details | 14:52 |
| iurygregory | ok | 14:52 |
| iurygregory | will do that in my afternoon today | 14:52 |
| zigo | TheJulia: If I understand correctly, standalone=True means separated API + conductor daemons, right? | 14:55 |
| zigo | Ah, no, opposite way. | 14:57 |
| zigo | standalone=False means separated API + conductor daemons, right? | 14:57 |
| TheJulia | zigo: .... I'm not entirely sure, ... I thought we did separate launch binaries | 15:09 |
| TheJulia | well | 15:09 |
| TheJulia | scripts | 15:09 |
| TheJulia | yes, ironic-inspector-conductor is the conductor process | 15:10 |
| TheJulia | I guess that option is if ironic-inspector is its own api surface | 15:11 |
| TheJulia | if you don't use wsgi | 15:11 |
| TheJulia | err | 15:11 |
| TheJulia | hmmmmmmm | 15:11 |
| TheJulia | so, that appears to be a single process binary launcher | 15:12 |
| TheJulia | if true, it uses rpc | 15:12 |
| TheJulia | okay, false == split process | 15:13 |
| TheJulia | ironic-inspector-conductor will not run without it being set to False | 15:13 |
| NobodyCam | Good Morning Ironic Folks | 15:20 |
| NobodyCam | safe to recheck now? | 15:21 |
| zigo | Yeah, saw the binaries, no worries. Though when I launch the api-wsgi, it complains about standalone=False not being set ... | 15:26 |
| zigo | So... | 15:26 |
| zigo | I know what I have to do ... tomorrow ! :) | 15:26 |
| TheJulia | zigo: which aligns with the ironic-inspector-conductor | 15:26 |
| zigo | (going back home) | 15:26 |
| TheJulia | zigo: have a wonderful evening! | 15:26 |
| zigo | Have a nice one too! | 15:26 |
| TheJulia | NobodyCam: no idea yet | 15:28 |
| rpittau | NobodyCam: hey o/ you probably want to wait for https://review.opendev.org/c/openstack/ironic/+/869052 to merge | 15:32 |
| JayF | rpittau: aiui the stuff that required jammy should've all been rolled back at an openstack level | 15:40 |
| JayF | rpittau: we still need/want that for sure, but do we still have jobs failing? | 15:40 |
| rpittau | JayF: the snmp one because of pyasn1 upper constraints update to 0.5.0, I filed a revert for that | 15:41 |
| JayF | aha | 15:41 |
| rpittau | in that patch I moved the snmp job to non-voting at least until we get pyasn1 reverted to 0.4.8 | 15:42 |
| rpittau | JayF: there was a discussion before with TheJulia to move to pysnmp and pyasn1 forks by lextudio | 15:42 |
| JayF | ack yeah I read that but honestly forgot it in the moment | 15:48 |
| TheJulia | rpittau: so! I found a comment by one of the splunk folks back in december pointing people to use the lexstudio fork | 15:52 |
| TheJulia | well, to be more precise, a customer found it | 15:52 |
| rpittau | w00t | 15:52 |
| TheJulia | https://github.com/etingof/pysnmp/pull/437 | 15:53 |
| rpittau | ah! | 15:53 |
| rpittau | ok that makes things simpler, kind of | 15:54 |
| TheJulia | slightly less muddy for those that find it | 15:54 |
| rpittau | I guess we can move toward lextudio versions | 15:54 |
| rpittau | I'll reshape the patches with them | 15:56 |
| rpittau | in the meantime pushing https://review.opendev.org/c/openstack/requirements/+/881556 forward will help us with the current snmp situation | 15:56 |
| rpittau | bye everyone, see you tomorrow o/ | 15:58 |
| iurygregory | JayF, quick question, where it would be better place in the spec to mention the high level idea about the in-band path? | 16:34 |
| iurygregory | maybe a note in Proposed change? | 16:36 |
| JayF | Yeah. "This spec describes the out of band interface. An in-band interface is planned to be implemented later with [a couple sentences on high level design]" | 16:41 |
| iurygregory | ok | 16:42 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic-specs master: Firmware Interface https://review.opendev.org/c/openstack/ironic-specs/+/878505 | 17:32 |
| iurygregory | JayF, if you have more ideas on what I should add, I'm happy to modify, was trying to think but all I could add was that note .-. | 17:38 |
| JayF | Did you see my recent comment? That's really the only interface I can think of that would allow dynamic determination of what things to upgrade based on agent detection of hardware | 17:40 |
| JayF | I just mainly didn't want to hold up your spec for something "future" if that makes sense... I just worry that when we design OOB interfaces, we might prevent the IB one from being as good as possible (e.g. like if we designed an interface that didn't allow a single agent to sanely do firmware upgrades across N hardware configs) | 17:41 |
| JayF | iurygregory: https://review.opendev.org/c/openstack/ironic-specs/+/879381 if you can, I'd love to see this get +2A so I can create the etherpad we review in the meetings... | 17:42 |
| iurygregory | JayF, oh I probably missed your recent comment /facepalm | 17:44 |
| iurygregory | looking now at the Workitems | 17:44 |
| iurygregory | +2 | 17:47 |
| JayF | +A ? | 17:47 |
| iurygregory | happy to +A if we don't need to wait for other feedback | 17:47 |
| iurygregory | done \o/ | 17:48 |
| JayF | If there is other feedback, I'm happy to follow up. At this time I think it's into boring territory though :D | 17:48 |
| iurygregory | agree | 17:48 |
| opendevreview | Merged openstack/ironic-specs master: Add 2023.2 Workitems discussed at Ironic PTG https://review.opendev.org/c/openstack/ironic-specs/+/879381 | 17:58 |
| iurygregory | JayF, adding "This interface would allow dynamic determination of which components can be upgraded based on what the agent detected." would be enough? | 18:06 |
| JayF | Like, I wouldn't object to that, but the purpose I had in asking for us to flesh it out a little bit was to make sure we had a "how" that made sense | 18:07 |
| JayF | that's a nice statement of intent, but doesn't really get to the point of "have we designed an interface that can interface with the dynamic nature of agent hwms?" | 18:07 |
| iurygregory | having a how wouldn't imply more details of the implementation that would be done? | 18:17 |
| JayF | it would, but like I said, my concern was just making sure there's a way to get from "how this interface is designed" to "how we could make it work in-band" | 18:17 |
| JayF | I was worried about this until I thought about it and put that comment on the spec | 18:17 |
| JayF | part of why I am +2 without mentioning it is that I'm fairly certain there's a path to get there | 18:18 |
| iurygregory | i see | 18:18 |
| JayF | Does that make sense? I don't care about which how, just that one exists :P | 18:19 |
| JayF | and I wanted to avoid the obvious-but-limited pattern of "we'll call magic_method() on one hwm interface to perform the upgrades" which doesn't mesh well with dynamically defined hwms | 18:19 |
| JayF | (whereas calling magic_method() on *all* HWM interfaces; similar to what we do in the agent when doing evaluate_hardware_support, it should be OK) | 18:20 |
| iurygregory | yeah I think it makes sense | 18:20 |
| JayF | Since the PTG Workstreams landed, I created https://etherpad.opendev.org/p/IronicWorkstreams2023.2 to track in-progress work this cycle | 18:28 |
| JayF | if you wanna update that with any items you're working on, how it's going, etc it'll allow us to look at each meeting | 18:28 |
| JayF | Heads up ironicers: thanks to rpittau's good work, gate should be fixed. pyasn upgrade in requirements just reverted | 18:38 |
| JayF | FYI https://review.opendev.org/c/openstack/nova-specs/+/881643 Re-Propose "Ironic Shards" for Bobcat/2023.2 [NEW] -- cc TheJulia | 19:32 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: Deprecate LLDP in inventory in favour of a new collector https://review.opendev.org/c/openstack/ironic-python-agent/+/881462 | 19:33 |
| TheJulia | JayF: awesome, thanks | 19:34 |
| JayF | https://review.opendev.org/c/openstack/ironic-python-agent/+/879897 could use some love (patch from Julia; it needs 1 more +2A) | 19:48 |
| JayF | https://review.opendev.org/c/openstack/ironic-python-agent/+/881257 same here (patch from me, trivial, needs +2A) | 19:48 |
| NobodyCam | looks like I am passing all but ironic-grenade | 20:47 |
| opendevreview | Maksim Malchuk proposed openstack/ironic-python-agent-builder master: Extend the DIB_CHECKSUM variable usage https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/881299 | 21:27 |
| JayF | Grenade has been failing off/on with a timeout. We had efforts to land a thing to help us set a timeout; IDK where that is now. | 21:46 |
| JayF | You should be OK to "recheck grenade issue" or similar to see if you can get it to pass | 21:46 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost master: Remove extra symbols accidentally added https://review.opendev.org/c/openstack/bifrost/+/879547 | 22:49 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost master: Remove extra symbols accidentally added https://review.opendev.org/c/openstack/bifrost/+/879547 | 22:49 |
| opendevreview | Merged openstack/ironic-python-agent master: Upgrade to latest hacking - v6 https://review.opendev.org/c/openstack/ironic-python-agent/+/881257 | 23:03 |
| opendevreview | Maksim Malchuk proposed openstack/bifrost master: Remove extra symbols accidentally added https://review.opendev.org/c/openstack/bifrost/+/879547 | 23:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!