| opendevreview | Iury Gregory Melo Ferreira proposed openstack/sushy master: Handle non-default language for registries https://review.opendev.org/c/openstack/sushy/+/872049 | 03:20 |
|---|---|---|
| vanou | good morning ironic | 04:24 |
| opendevreview | Verification of a change to openstack/ironic master failed: Get conductor metric data https://review.opendev.org/c/openstack/ironic/+/865447 | 07:09 |
| opendevreview | Mohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 07:53 |
| opendevreview | Mohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 08:11 |
| opendevreview | Mohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 08:14 |
| opendevreview | Mohammed Boukhalfa proposed openstack/sushy-tools master: Add fake_ipa to fake system https://review.opendev.org/c/openstack/sushy-tools/+/875366 | 08:16 |
| kubajj | Good morning vanou and everyone | 09:07 |
| dtantsur | JayF: auto-TLS was about conductor->IPA connection, not the other way around | 09:20 |
| dtantsur | iurygregory: good morning! the project submission deadline has been extended for outreachy, we have this week still. | 09:49 |
| rpittau | good morning ironic! o/ | 10:00 |
| vanou | hi kubajj | 10:30 |
| rpittau | iurygregory: which patches are we waiting for to release ipe and ngs ? | 10:59 |
| opendevreview | Verification of a change to openstack/ironic master failed: Get conductor metric data https://review.opendev.org/c/openstack/ironic/+/865447 | 11:10 |
| iurygregory | dtantsur, ack | 11:16 |
| iurygregory | rpittau, there is TheJulia patch to include ironic metrics (IPE) and ngs I saw some patches from mgoddard I think... | 11:17 |
| iurygregory | good morning Ironic | 11:17 |
| TheJulia | good morning | 13:55 |
| TheJulia | so many mirror downloads | 13:57 |
| TheJulia | well, download failures | 13:58 |
| rpittau | iurygregory: thanks | 14:00 |
| rpittau | TheJulia mgoddard correct me if I'm wrong, are those the patches missing for the releases? https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/869509 https://review.opendev.org/c/openstack/networking-generic-switch/+/874789 https://review.opendev.org/c/openstack/networking-generic-switch/+/874793 https://review.opendev.org/c/openstack/networking-generic-switch/+/873098 | 14:02 |
| rpittau | https://review.opendev.org/c/openstack/networking-generic-switch/+/743283 | 14:02 |
| TheJulia | iurygregory: can you clarify your comments w/r/t https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/869509 ? | 14:03 |
| TheJulia | If we have to cut today... most likely. The challenge at hand is CI is misbehaving | 14:08 |
| TheJulia | mgoddard: any opinions? | 14:08 |
| iurygregory | TheJulia, most of the comments I added would be some nits I would say (nothing that would really hurt to be done in a follow-up), I removed the +2 because we changed the config options in the ironic side (so we likely need to change https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/869509/9/devstack/plugin.sh ) | 14:25 |
| TheJulia | ahh, right | 14:25 |
| TheJulia | yeah, that would be good to change, although it should still work | 14:25 |
| TheJulia | still don't understand one of your other comments, but that is aside from the config atm | 14:26 |
| TheJulia | oh | 14:27 |
| TheJulia | no, the direct names are used, not the old ones | 14:27 |
| opendevreview | Julia Kreger proposed openstack/ironic-prometheus-exporter master: Support extraction of ironic internal metrics https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/869509 | 14:27 |
| iurygregory | TheJulia, the comment you said you don't understand is https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/869509/9/ironic_prometheus_exporter/parsers/header.py ? | 14:36 |
| TheJulia | I guess https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/869509/9/ironic_prometheus_exporter/parsers/ironic.py are you saying we need to try to consturct more verbose descriptions? I guess the idea is frustrating because reference wise it is in examples, but practical usage I rarely see such in sample payloads | 14:38 |
| iurygregory | no, the descriptions are good, it would be more to keep the same logic we had for other parsers (ipmi/redfish) where we have the json file that would map what is the description for each key | 14:39 |
| TheJulia | ahh, that makes sense to do if we know the description | 14:40 |
| iurygregory | don't worry with this for now, it's like a low-hanging-fruit bug =) | 14:41 |
| *** dansmith_ is now known as dansmith | 14:44 | |
| JayF | hey everyone o/ | 15:00 |
| JayF | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon Feb 27 15:00:58 2023 UTC and is due to finish in 60 minutes. The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| JayF | Who all is around? | 15:01 |
| vanou | o/ | 15:01 |
| matfechner | o/ | 15:01 |
| TheJulia | o/ | 15:01 |
| JayF | #topic Announcements/Reminder | 15:01 |
| JayF | Tag your patches #ironic-week-prio if they need review... if you want them in Antelope release that should be ~nowish :D | 15:02 |
| JayF | Also, if you haven't seen, OIS schedule is out (not forum; just summit). Please check it out | 15:02 |
| JayF | #link https://vancouver2023.openinfra.dev/a/schedule | 15:02 |
| JayF | any other announcements | 15:02 |
| rpittau | o/ | 15:03 |
| JayF | #note TheJulia had an action to get python-ironic-inspector-client CI happy; how did that go? | 15:03 |
| TheJulia | requires a fix to be landed in insepcctor sine it imports the code directly | 15:04 |
| TheJulia | one moment | 15:04 |
| TheJulia | merged, so upon next inspector release the world should be happier for that job | 15:04 |
| TheJulia | zed inspector release to be specific | 15:04 |
| JayF | perfect | 15:04 |
| JayF | that was the only action item last week, moving on | 15:04 |
| TheJulia | err, maybe/maybe not, since it is not constrained | 15:04 |
| TheJulia | either way, the patch needed has merged at this point | 15:05 |
| JayF | well that fits right into | 15:05 |
| JayF | #topic Ironic CI status | 15:05 |
| JayF | how are things? any concerning issues seen over the last week | 15:05 |
| rpittau | bifrost ci still kaput, fix is under review | 15:05 |
| JayF | link? | 15:06 |
| rpittau | https://review.opendev.org/c/openstack/bifrost/+/874650 | 15:06 |
| JayF | that's open here now; I'll have a look post-meeting | 15:06 |
| rpittau | failures are inconsistents, so not easy to fixx all of them at the same time | 15:07 |
| JayF | CI is V-1 right now on that :( | 15:07 |
| rpittau | yeah | 15:07 |
| rpittau | going to recheck | 15:07 |
| JayF | yeah that's always our battle | 15:07 |
| JayF | okay | 15:07 |
| JayF | #topic VirtualPDU | 15:07 |
| JayF | anything new on getting us access? | 15:07 |
| rpittau | well waiting for fungi I guess | 15:08 |
| * iurygregory is late o/ | 15:08 | |
| rpittau | no answers from cores | 15:08 |
| rpittau | so last chance is on opendev team | 15:08 |
| JayF | alright; I know they were all offsite last week so hopefully that moves more now | 15:09 |
| JayF | are we on a timer for that? | 15:09 |
| JayF | do we need to get it flipped before A is cut? | 15:09 |
| rpittau | I think we're good if we move things forward this week | 15:09 |
| JayF | alright | 15:09 |
| JayF | #topic Release countdown: 3 weeks | 15:09 |
| JayF | I owe a revision to cycle highlights; https://review.opendev.org/c/openstack/releases/+/874338 -- I'll do that as soon as this meeting is over | 15:10 |
| JayF | https://etherpad.opendev.org/p/IronicWorkstreams2023.1 looking at this now for anything we can land before A hits | 15:10 |
| JayF | I think we're nearing the point of things being in that are gonna git in, in terms of larger workstreams | 15:10 |
| JayF | moving on since there's no further input | 15:12 |
| JayF | #topic open discussion | 15:13 |
| JayF | vanou: had two items in here | 15:13 |
| vanou | Yes. | 15:13 |
| vanou | First item is about acceptability of backport patch on iRMC driver (sorry for iRMC driver specific) | 15:13 |
| vanou | This backport patch adds logic of logging warning, when it catches incompatible behavior of iRM server firmware | 15:14 |
| TheJulia | through use of a verify step yes? | 15:15 |
| JayF | Can you link the specific patch for context? | 15:15 |
| vanou | Just adds warning, but it adds verify step. So in discussion with TheJulia, we need to ask community if it's backportable | 15:15 |
| vanou | Soryy. This one https://review.opendev.org/c/openstack/ironic/+/870880 | 15:15 |
| vanou | TheJulia: yes | 15:16 |
| JayF | Can we be explicit about the behavior if we don't backport this to Zed? | 15:16 |
| JayF | On the surface I'm in agreement that it's a little much to backport | 15:16 |
| TheJulia | my concern in this case is we're adding basically a feature in the form of a step an operator would need to invoke | 15:16 |
| vanou | If we don't backport this, ironic operator lose chanse to notice iRMC incompatible behavior through ironic log | 15:16 |
| JayF | Yeah; this change reads more like a feature than a bugfix -- even if it is working around/with new firmware behavior | 15:17 |
| JayF | If all we're giving up is an operator getting a logging message; I don't think it should be backported. Instead, could we write a document for how users in these situations can figure out + fix it, outside of Ironic? | 15:17 |
| vanou | JayF: notify user with doc is another reasonable option | 15:18 |
| JayF | I think that's preferable | 15:18 |
| JayF | Is there anyone stable core here who disagrees and wants to fight for #870880? | 15:18 |
| TheJulia | I do not disagree, but I'm also the one who sort of forced this discussion to take place | 15:19 |
| TheJulia | vanou: thank you for being up very late/very early for this meeting | 15:19 |
| JayF | #note https://review.opendev.org/c/openstack/ironic/+/870880 is not permitted to be backported to Zed; instead we will focus on a documentation-based solution for operators in this case. | 15:19 |
| JayF | vanou: I think you also had an item up about the vuln management docs I put a review on | 15:19 |
| JayF | vanou: looking at your agenda item: to clarify my comments; Ironic can only set policy for Ironic-managed projects in the openstack/ namespace | 15:20 |
| vanou | Regarding first item, thanks for feedback :) I'll take that doc way | 15:20 |
| JayF | so vendor tools under x/ like x/proliantutils -- we don't have the authority to set policy for these | 15:20 |
| JayF | one question I've had: why don't we just follow OpenStack VMT standard? | 15:20 |
| JayF | is there a historical reason we're not/ | 15:20 |
| vanou | I felt the need the recommended way to handle vendor library, if that vul is also affect ironic code. | 15:22 |
| TheJulia | JayF: so historical reason I believe was a lack of capacity, but it goes back to the days of Aeva | 15:23 |
| TheJulia | and I think in part it is because of the duality nature at play with things like x/proliantutils being totally out side of our control and we just consume it | 15:24 |
| JayF | Do we have any ironic contributors who'd oppose me syncing up with security group in OpenStack to get us in the VMT? | 15:24 |
| JayF | That will not prevent us from being a 301-redirect for vendor-tools-related security bugs if they come in | 15:24 |
| JayF | I suspect we can talk to the folks involved and they'll deal with us reasonably | 15:24 |
| JayF | and if not, we would then have a specific reason to be different rather than "we just are" :) | 15:24 |
| TheJulia | ++ | 15:24 |
| vanou | I agree with following OpenStack VMT regarding Ironic specific code problem | 15:25 |
| JayF | #action JayF to engage VMT (probably mailing list post) to inquire about getting Ironic in it. | 15:25 |
| JayF | vanou: I think for the non-openstack ironic based code issues; we have two potential paths: 1) the vendor that primarily maintains it discovers and issue, fixes it in the library, and discloses it to us so we can bump versions or | 15:26 |
| JayF | 2) someone external, who uses Ironic, discovers it and reports it through our systems, and we responsibly pass it on to the vendor | 15:26 |
| JayF | both of those things are stuff I would expect/hope would happen just by common sense by folks running things | 15:26 |
| vanou | Yes. These 2 are good option regarding vulnerability on vendor library code. | 15:27 |
| vanou | But I feel we need another guide if that vulnerability needs fix on both ironic and vendor library | 15:27 |
| JayF | In those cases, VMT policy generally allows disclosure to trusted developers/cores needed to fix an issue | 15:28 |
| JayF | in the case of those coordinations, I'd expect/hope people to work together without needing a document on exactly how to do it | 15:28 |
| JayF | but maybe that's wishful thinking? | 15:28 |
| TheJulia | I think the issue is when there is disagreement | 15:29 |
| TheJulia | or a difference of view/opinion | 15:29 |
| JayF | Disagreement about if something is a bug? Or how to fix it? | 15:29 |
| TheJulia | which we've seen recently like with the glance report that has been revised a few times, inherently it is a feature, but the reporter wants it deemed a vulnerability | 15:29 |
| TheJulia | so the challenge is who holds the power to say yes or no in the entire sequence of trying to work through a thing. | 15:30 |
| JayF | I don't see how that problem exists any more or less in Ironic+vendor tools than it does with OpenStack+any-other-non-openstack-library | 15:31 |
| TheJulia | And then codifying such a dynamic in a doc seems to be what is desired, which I think is reasonable, but then not every case is the same... | 15:31 |
| JayF | I default to preferring to not document every single case, because each document comes with a maintenance cost | 15:31 |
| TheJulia | I guess the challenge is there is nuance in all situations | 15:31 |
| JayF | and I don't trust us to do a good job of updating it as things change | 15:31 |
| JayF | ++ I do not want to remove any nuance | 15:32 |
| JayF | Lets go down the path with the VMT | 15:32 |
| JayF | and mention this in the thread | 15:32 |
| JayF | and see how it goes | 15:32 |
| JayF | the folks who do security in openstack-proper might already have some strategies for managing this kind of problem | 15:32 |
| JayF | there's no reason for Ironic to discuss or try to solve it in a vacuum | 15:32 |
| vanou | If we don't write guide on ironic+vendor vul, we need written policy on that because reporter don't know how ironic handle this situation. | 15:34 |
| vanou | ^ just my comment. | 15:34 |
| JayF | I'm saying lets get that question inside the larger conversation aorund Ironic joining VMT | 15:34 |
| JayF | It's extremely possible openstack already has a policy that we can point to aorund that | 15:34 |
| vanou | Ah. I understand | 15:34 |
| JayF | I'll own making that thread on the list today | 15:35 |
| JayF | #action JayF to email list about Ironic joining VMT; will be sure to mention potential vendor:Ironic complications | 15:35 |
| JayF | Is there any other items we'd like to talk about in open discussion? | 15:35 |
| JayF | Oh, I wanted to mention | 15:36 |
| JayF | dtantsur found an issue with api-ref, he mentioned it in channel a couple of times | 15:36 |
| JayF | well, good job there, the issue was found + is pending review to fix it in the theme for all openstack projects | 15:36 |
| dtantsur | a fix has been proposed against openstackdocstheme | 15:36 |
| JayF | https://review.opendev.org/c/openstack/openstackdocstheme/+/874957 | 15:36 |
| JayF | #link https://review.opendev.org/c/openstack/openstackdocstheme/+/874957 | 15:36 |
| JayF | our api-ref looks infinitely better with the change | 15:37 |
| JayF | so thank you dtantsur for not letting that sit \o/ | 15:37 |
| dtantsur | :) | 15:37 |
| JayF | We should probably also mention https://review.opendev.org/c/openstack/releases/+/875396 | 15:37 |
| JayF | #note dtantsur is no longer going to be an Ironic release liason | 15:38 |
| JayF | Thank you for all the things you have done/do/are continuing to do for ironic | 15:38 |
| dtantsur | alas! too much stuff on my shoulders already | 15:38 |
| JayF | happy to lighten the burden a bit :) | 15:38 |
| arne_wiebalck | thanks dtantsur for doing it for so long! | 15:38 |
| vanou | thanks dtantsur! | 15:39 |
| JayF | Also, I need a volunteer to run the meeting 3/13 (meeting-after-next) | 15:39 |
| JayF | I'll be in Southern California presenting at SCALE (with TheJulia) | 15:40 |
| iurygregory | o/ | 15:40 |
| JayF | if anyone is in that area and wants to recieve a high-five and/or have lunch, please reach out | 15:40 |
| iurygregory | I can run the meeting | 15:40 |
| JayF | #action iurygregory to run the meeting 3/13 (2 weeks from today) | 15:40 |
| JayF | Last call for open discussion before I shut it down | 15:40 |
| JayF | #endmeeting | 15:42 |
| opendevmeet | Meeting ended Mon Feb 27 15:42:26 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:42 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-02-27-15.00.html | 15:42 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-02-27-15.00.txt | 15:42 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2023/ironic.2023-02-27-15.00.log.html | 15:42 |
| JayF | thanks everyone; monday time \o/ | 15:42 |
| arne_wiebalck | thanks JayF | 15:42 |
| vanou | Thanks TheJulia JayF and all! Good night | 15:43 |
| fungi | rpittau: i'm going to bring the virtualpdu plan up with the other opendev sysadmins in tomorrow's meeting (19:00 utc in #opendev-meeting) | 15:47 |
| rpittau | fungi: thanks! | 15:47 |
| JayF | https://review.opendev.org/c/openstack/releases/+/874338 is updated; I'm happy with them as they sit. More reviews won't hurt :D | 15:50 |
| opendevreview | Mark Goddard proposed openstack/networking-generic-switch master: Support batching up commands https://review.opendev.org/c/openstack/networking-generic-switch/+/743283 | 15:51 |
| * TheJulia feels like she needs to hold her breath for the CI gate | 15:55 | |
| opendevreview | Mark Goddard proposed openstack/networking-generic-switch master: Add ngs-stress test script https://review.opendev.org/c/openstack/networking-generic-switch/+/874789 | 16:04 |
| JayF | mgoddard: I believe we're going to use sharding to help the case in #743283 | 16:10 |
| JayF | mgoddard: hmm, or actually might make it worse | 16:11 |
| JayF | if we had N ngs processes split across N shards, it might *add* a complication for you there of having >1 process making switch changes | 16:11 |
| TheJulia | disjointed problems | 16:26 |
| TheJulia | and yeah, switch internal locking is sometimes problematic | 16:26 |
| rpittau | good night! o/ | 16:28 |
| JayF | I'm glad to have thought of that though; it's something to avoid when implmenting sharding | 16:28 |
| TheJulia | JayF: different point in the pipelines too | 16:29 |
| TheJulia | n-g-s gets commands based upon port bind sequences | 16:29 |
| TheJulia | networking-baremetal reads ironic's api to seed/sync physnet info | 16:30 |
| rpittau | actually before I leave, JayF TheJulia if you have a moment the bifrost CI fix is finally green! https://review.opendev.org/c/openstack/bifrost/+/874650 | 16:30 |
| rpittau | going for real now o/ | 16:30 |
| TheJulia | batching is to address the already present issue of the existing locking + >1 thing trying to do something at any given time | 16:30 |
| TheJulia | o/ | 16:30 |
| JayF | ack | 16:34 |
| JayF | rpittau: +A | 16:34 |
| opendevreview | Merged openstack/ironic master: Get conductor metric data https://review.opendev.org/c/openstack/ironic/+/865447 | 17:24 |
| TheJulia | \o/ | 17:29 |
| dtantsur | \o/ | 17:51 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org experienced severe performance degradation between 17:50 and 19:45 due to excessive API query activity; the addresses involved are now blocked but any changes missing job results from that timeframe should be rechecked | 19:55 | |
| iurygregory | nice... | 20:04 |
| iurygregory | =( | 20:05 |
| sschmitt | I asked this the other day, but wanted to see what the difference is now between NGS and networking-baremetal. It seems like they both now have functionality to interact with network switches. If I wanted to extend functionality in this area, which project makes sense as a starting point? | 20:50 |
| JayF | sschmitt: gonna be honest; I'm not sure I know hte answer to that either; if you don't get a response here (again), please post it to the openstack-discuss list with [ironic] prefix in the subject | 20:52 |
| JayF | and folks from other timezones can maybe help you out | 20:52 |
| sschmitt | got it, thanks! | 20:52 |
| opendevreview | Julia Kreger proposed openstack/ironic-prometheus-exporter master: Support extraction of ironic internal metrics https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/869509 | 20:53 |
| TheJulia | sschmitt: so Networking-baremetal is where I think we would *love* to see stuff like this evolving, but networking-generic-switch is kind of like the "do the needful" via ssh solution | 21:41 |
| TheJulia | where as networking-baremetal is looking more at netconf | 21:42 |
| JayF | TheJulia: netconf? | 21:42 |
| TheJulia | sschmitt: the person we should likely chat with is hjensas | 21:42 |
| TheJulia | I might be thinking of the wrong word | 21:42 |
| sschmitt | openconf | 21:42 |
| TheJulia | but basically there is an xml standard for network configuration interchanges | 21:42 |
| JayF | I'm not sure I'd know the right word :) | 21:42 |
| TheJulia | yeah, thats it! | 21:42 |
| JayF | yep, haven't heard of that either but it sounds cool | 21:42 |
| jrosser | netconf is all well and good but top of the docs for my switches say they dont support netconf RBAC RFC 6536 so it's anything-goes config wise over that interface | 21:48 |
| jrosser | on the other hand with n-g-s its completely trivial to configure a limited user on the switch which can only issue a subset of commands against a subset of ports, so thats a total no-brainer for me | 21:49 |
| TheJulia | jrosser: that is a very valid data point | 22:05 |
| TheJulia | I think there is a "these things will mature as time moves forward" sort of consideration | 22:05 |
| TheJulia | ... and most network infra admins I've chatted with over the years highly prefer restricted access as opposed to "just give full access to it all" | 22:06 |
| jrosser | thats exactly how i allow n-g-s to work, with a special user on the switch thats deny-all/allow-as-needed control on what it can do | 22:08 |
| jrosser | i wouldnt dare have it different as the attack surface is gigantic especially if you collapse many logical things into the same network hardware | 22:08 |
| iurygregory | humm bifrost CI seems a bit unhappy in the IPE change .-. | 22:13 |
| TheJulia | jrosser: excellent | 22:19 |
| opendevreview | Merged openstack/ironic master: Add configurable delays to the fake drivers https://review.opendev.org/c/openstack/ironic/+/861127 | 23:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!