| *** mdfr3 is now known as mdfr | 02:32 | |
| rpittau | good morning ironic! o/ | 05:58 |
|---|---|---|
| queensly[m] | Good morning o/ | 06:28 |
| abongale | good morning ! | 08:21 |
| rpittau | cardoe: if it's a fix, we can wait after teh bugfix branch and backport it | 09:50 |
| iurygregory | good morning | 10:50 |
| opendevreview | Merged openstack/ironic master: Clear `last_error` on power match/sync https://review.opendev.org/c/openstack/ironic/+/955432 | 12:21 |
| janders | hey folks o/ I am troubleshooting an issue with some machines apparently sending empty-string eTags. Example error in this paste (along with some other data I will touch on in a second): https://paste.openstack.org/show/bdiJfjC55LpJaG3ROsa9/ | 12:36 |
| janders | I wonder if you've seen this before and how do you think we should handle such cases | 12:36 |
| janders | 1) when we get an empty-string eTag, should we try to send them back to BMC with PATCH requests (and possibly retry without eTag on failure) or 2) do we just plain-disregard them and send PATCH requests without eTags in such case? | 12:37 |
| janders | it seems to me that some machines (more likely HPs of various models) have some attributes that have empty eTag fields (paste has some examples) | 12:38 |
| janders | I'm working on unit tests reproducing this problem and code improvements to make them pass, before I go further I wanted to raise the issue and see if anyone experienced this and what you folks think about such case in general | 12:39 |
| opendevreview | Takashi Kajinami proposed openstack/ironic master: Drop redundant geattr https://review.opendev.org/c/openstack/ironic/+/956161 | 12:44 |
| opendevreview | Merged openstack/ironic master: Switch from local RPC to automated JSON RPC on localhost https://review.opendev.org/c/openstack/ironic/+/954755 | 12:48 |
| rpittau | iurygregory, TheJulia, cardoe, cid, when you have a moment can you please check https://review.opendev.org/c/openstack/ironic/+/953477 ? I'd like to include it in the bugfix branch | 13:13 |
| TheJulia | rpittau: there are two release notes there... | 13:19 |
| iurygregory | rpittau, looking now | 13:22 |
| TheJulia | janders: some folks have indicated they have observed some similar behavior, but also a variety of behavior from vendors in this area. I would think if there is an empty etag we would just ignore it. Also looks like this is in OEM regions of remote api surfaces so naturally YMMB | 13:25 |
| TheJulia | err YMMV | 13:25 |
| sdmitriev | 1 | 13:38 |
| TheJulia | ??? | 13:41 |
| rpittau | TheJulia: I didn't think the 2 release notes were a problem | 13:54 |
| TheJulia | I guess typically I would expect to see it a single file per change, not two | 13:56 |
| TheJulia | I guess it should be okay | 13:56 |
| alegacy | friendly reminder. meeting to demo progress on standalone networking in 1 hour from now @ 1500UTC. meet.google.com/ijs-pwev-qhq | 13:56 |
| alegacy | 13:56 | |
| sdmitriev | Hey folks, sorry about the previous message, fat fingers. I wanted to ask a question related to "driver_info". Do I understand correctly that all password fields (e.g., "ipmi_password") are masked when sent to the Ironic Python Agent? So basically, IPA is not supposed to have access to them, right? | 13:58 |
| sdmitriev | Context: we're implementing a custom hardware manager that, unfortunately, requires access to the out-of-band interface (we need to query RestAPI) to perform hardware updates. Any suggestions on how to pass such sensitive information to IPA? | 13:58 |
| sdmitriev | In the perfect world we would just use redfish driver for that, but our ODM provider has "specific" redfish scheme which does not implement all the required attributes, so we have to look into workaround like this | 14:02 |
| TheJulia | sdmitriev: correct the values are not aware. Basically the best pattern we have is to use the secret token value to encrypt the data and enable IPA on the receiving side to decrypt it | 14:11 |
| *** dmellado62 is now known as dmellado6 | 14:13 | |
| iurygregory | rpittau, I think it still requires some changes in https://review.opendev.org/c/openstack/ironic/+/953477 | 14:15 |
| iurygregory | going do add some comments | 14:16 |
| sdmitriev | TheJulia: I see, thanks for the reply. Where would you place the encrypted info then? Into "driver_internal_info"? Any chance you could point me into the code example where similar approach is implemented? | 14:16 |
| TheJulia | sdmitriev: typically we would not save such value to a node object filed, we would be pushed in a call. I guess its a weird issue. Why do you need the driver info fields inbound on the host to run locally? We *generally* advise blocking all host->bmc access for security reasons (so someone can't force reset the bmc or do other naughty things with the BMC) | 14:20 |
| TheJulia | (granted, use cases will vary, but generally we lean towards a case where you might not trust your users. | 14:21 |
| TheJulia | ) | 14:21 |
| sdmitriev | TheJulia: the hardware provider that we use do not implement full Redfish spec, so we can only enrol it with IPMI driver. At the same time, the only way this hardware BIOS/BMC firmware can be updated is with Redfish calls (so out-of-band). So we are trying to figure out the way how to include firmware update steps into the "clean" stage | 14:25 |
| TheJulia | sdmitriev: so the existing firmware update interfaces in Ironic don't work for this specific piece of hardware for OOB updates of the BMC? | 14:26 |
| sdmitriev | TheJulia: Only Redfish driver supports firmware updates currently, and we can not use it unfortunately, we stuck with IPMI at least for now | 14:28 |
| TheJulia | Yeah, IPMI will never support such functionality | 14:28 |
| TheJulia | since really, ipmi should just not be used but... yeah | 14:28 |
| TheJulia | So, when you say required attributes, you mean the attributes inside of the ComputerSystem object? | 14:29 |
| sdmitriev | TheJulia: Can't say for sure which ones. It was tested by my coworkers before me, but the conclusion was that the some of the required Redfish calls would fail due to missing attributes, and ODM provider confirmed that the Redfish spec was not fully implemented for taht devices | 14:32 |
| TheJulia | That is very unfortuante | 14:32 |
| TheJulia | your basically going to need some bit of code with enough information to grab the credentials, be able to identify or consolidate the necessary update information across to the agent as a step call which ironic would need to have code wrapped around to trigger, and then engage a step in the agent to identify the correct inband or potentially out of band (file yourself a security bug if they can do this in your | 14:35 |
| TheJulia | environment), At least to do to be able to perform the interaction. I'd almost wonder if maybe instead we need an "advanced" hardware type where you can, say invoke redfish management but ipmi power, but I don't know if that would bridge the gap enough for you to be able to have a functioning system | 14:35 |
| TheJulia | iurygregory, janders, this may be of interest to you two | 14:35 |
| TheJulia | ^^^ | 14:35 |
| iurygregory | if it's Cisco hardware I know our firmware upgrade won't work | 14:48 |
| iurygregory | normally the only thing we do is a SimpleUpdate call to redfish.. so the UpdateService must be working, without much info is a bit complicated to see if we can provide something that would work for this hardware | 14:50 |
| sdmitriev | Not Cisco, it's Wiwynn. I guess we may need to go over what did not exactly work on our side with redfish driver first | 14:52 |
| TheJulia | That wouldn't be a bad idea, generally if its something on our end or a pattern we're starting to see, we are generally fairly amenable to fixing/patching | 15:01 |
| TheJulia | standalone networking demo? | 15:01 |
| TheJulia | meet.google.com/ijs-pwev-qhq ! | 15:01 |
| mumesan[m] | Hey could I get another eye on this: https://review.opendev.org/c/openstack/networking-generic-switch/+/955798 | 15:13 |
| opendevreview | Verification of a change to openstack/ironic master failed: Drop redundant geattr https://review.opendev.org/c/openstack/ironic/+/956161 | 15:59 |
| iurygregory | Wiwynn never heard about it =( | 16:58 |
| TheJulia | a vendor split off from a fabricator that used to be Asus back in 2000 and now does oem stuffs. | 17:05 |
| cardoe | Can some sanity check me for a sec... node.properties... the "cpus" field... number or string? | 18:27 |
| cardoe | On a box you've got. | 18:27 |
| cardoe | bleh | 18:32 |
| * cardoe table flips. | 18:32 | |
| cardoe | nvm. just another special redfish field. | 18:35 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: fix up redfish inspection mock ethernet interface data https://review.opendev.org/c/openstack/ironic/+/955536 | 19:05 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: fix redfish processor inspection https://review.opendev.org/c/openstack/ironic/+/955537 | 19:05 |
| opendevreview | Doug Goldstein proposed openstack/ironic master: allow running inspection hooks on redfish interface https://review.opendev.org/c/openstack/ironic/+/933066 | 19:05 |
| cardoe | Well hopefully the first two are good enough to pass. | 19:06 |
| rm_work | TheJulia: for the record, he’s on my team 😆 | 20:04 |
| TheJulia | huh?! | 20:04 |
| rm_work | Discussion above about firmware and wiwynn | 20:05 |
| rm_work | So when he says “previous teammates” he means me mostly lol | 20:05 |
| rm_work | Just for context | 20:06 |
| TheJulia | ahhhhh! | 20:06 |
| cardoe | TheJulia: so that Ethernet mock change… is that okay-ish? I feel like that’s better than returning two different pieces of data for the same hardware. The tests put one value for the mock and then expect something different. | 20:08 |
| rm_work | Hey if we’re relying on routed networks right now, is there something that equates to that at all in the ironic standalone networking stuff? | 20:09 |
| cardoe | The processor change fails because we bumped the version of hacking and this is the first time someone is touching it. Despite me simplifying it. | 20:15 |
| TheJulia | cardoe: last I looked at it, it seemed okay to me | 20:16 |
| TheJulia | rm_work: routed networks with neutron? | 20:16 |
| rm_work | Yes | 20:20 |
| TheJulia | ML2 plugins | 20:20 |
| rm_work | But looking at running something more standalone ironic based for undercloud | 20:20 |
| TheJulia | oh, so the standalone thing should align and ultimately support it | 20:20 |
| rm_work | Ok, I just don’t know a lot about the standalone stuff | 20:21 |
| rm_work | It doesn’t directly support ml2 plugins does it? | 20:22 |
| TheJulia | I haven't seen the code, I think it is largely modeled around just re-use of what networking-generic-switch provides | 20:23 |
| rm_work | Hmm 🤔 ok | 20:24 |
| rm_work | I’ll dig deeper | 20:24 |
| TheJulia | we're expecing alegacy to post some initial code next week | 20:24 |
| TheJulia | realistically though, if your using neutron, then the expected model is an neutron ml2 plugin which supports the baremetal vnic type | 20:27 |
| TheJulia | and can manage switches | 20:28 |
| TheJulia | but not all do that | 20:28 |
| rm_work | What do you mean by “manage switches”? | 20:29 |
| rm_work | Like literally do on-switch configuration? | 20:29 |
| TheJulia | yes | 20:29 |
| rm_work | Ok, yeah I think that’s not workable for us 😰 | 20:30 |
| rm_work | Our networking team won’t even give us read-access to switches lol | 20:30 |
| TheJulia | some it is, some it isen't | 20:30 |
| rm_work | I’ve coined the term “adversarial development” since working here 😅 | 20:30 |
| TheJulia | in the standalone stuff, the idea is to delienate it so they can own the service, and the configuration contrat is as small as possible to what is expected/needded | 20:30 |
| cardoe | rm_work: so literally that’s what I’m working on but we’re doing it with neutron still. And trying to teach neutron to be an okay-ish baremetal API | 20:31 |
| cardoe | I’m not using NGS because we also don’t touch the switches. | 20:33 |
| rm_work | Ah ok | 20:37 |
| cardoe | Much like NGS has generic calls to switches, we’re looking to make another ML2 which could call NGS but can also call out to other network automation tools to make the changes | 20:37 |
| cardoe | At this point we’ve got some I | 20:38 |
| cardoe | Implementations and running it. But we’ve got 3 outstanding neutron specs to make the use case documented and part of the test surface. | 20:39 |
| cardoe | We’ve also used OVN to provide cloud-y like features. | 20:40 |
| opendevreview | Verification of a change to openstack/ironic master failed: Drop redundant geattr https://review.opendev.org/c/openstack/ironic/+/956161 | 20:47 |
| cardoe | Like my patched up OVN/neutron natively joins the VXLAN fabric of our Cisco switches and is doing DHCP and DNS. I even got metadata agent to work. But I don’t know of anyway to ensure nobody could spoof. | 20:49 |
| cardoe | I can at least validate that the box is from the right tenant. But can’t ensure it’s not another box in that tenant spoofing the MAC. | 20:51 |
| cardoe | Ultimately I hope this compliments the standalone groups work. | 20:55 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/ironic master: Add manual clean and automated verify steps to set BMC clock via Redfish Manager https://review.opendev.org/c/openstack/ironic/+/953477 | 21:09 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/ironic master: [docs] Update manual clean and verify steps https://review.opendev.org/c/openstack/ironic/+/955730 | 21:09 |
| TheJulia | cardoe: only way to prevent spoofing is to extend the metadata stuff to use the raw data or to operate with raw packet handling or to look at the mac address from which the request comes | 21:23 |
| TheJulia | even then, one could still spoof macs, so then it boils down to switch programming and all | 21:23 |
| cardoe | Well that's what I'm doing. | 21:24 |
| TheJulia | but then if everything is sort of the next problem :) | 21:24 |
| cardoe | How it's done on KVM is that they know the actual virtual port it came in on into OVS from QEMU and they label it there. | 21:24 |
| cardoe | I just know that the request was on the correct VXLAN VNI so I know this request came from that tenant network. | 21:25 |
| cardoe | with L2 VNIs I know that the packet is on the right VLAN but technically that tenant could have 2 boxes on the same VLAN in the same cabinet | 21:27 |
| cardoe | with L3 VNIs / routed stuff I have the same guarantee but then know that the request came in the tenant's VRF | 21:27 |
| cardoe | But if I've got 2 servers on the same cabinet and one is using 00:11:22:33:44:55 and one is using aa:bb:cc:dd:ee:ff. And the 00....55 spoofs aa..ff. I don't know of a way to ensure that cause the packet's already on my VLAN or VRF. | 21:29 |
| cardoe | Maybe there's a way with security groups to limit it down to a specific MAC to a specific port. But then allowed_address_pairs gets busted for failover and if I allow that then I'd be right back at the same place. | 21:30 |
| TheJulia | There was some dsicsusion ages ago about maybe building a version of the agent which could run on the switch devices, but I don't know how practical that would really be | 21:30 |
| cardoe | Maybe there is a way to do it in the switches but I don't know. I'm not a network or switch person. I didn't even stay at a holiday inn express last night... just my RV. | 21:30 |
| TheJulia | in so much the agent would still need to be aware of it all and yeah | 21:30 |
| TheJulia | cardoe: RV's can have way better beds. | 21:31 |
| TheJulia | and worse beds. | 21:31 |
| cardoe | Agreed | 21:32 |
| cardoe | My wife is dragging me to look at fifth wheels tomorrow when we have the truck serviced. | 21:32 |
| TheJulia | We realized the pullout bed on our coach still has the factory plastic wrap on it... it was built in 2015 | 21:33 |
| TheJulia | Enjoy! | 21:33 |
| cardoe | My first goal with all this however is to make it actually all work nicely and get neutron to accept the use case and hopefully patches. | 21:33 |
| TheJulia | its not a bad goal, maybe sometime next week we can get a discussion going with steve, although his current focus is getting security group stuff programmed into switches | 21:34 |
| cardoe | I'm setting up my own hardware and my ML2 will use NGS. | 21:34 |
| cardoe | Yeah I was following his work there. | 21:35 |
| cardoe | He's doing the calls exactly how I'd want to wrap stuff. | 21:35 |
| TheJulia | He is getting tons of time and space... ultimately we have a few customers who are pushing us to enhance these areas and we don't want to get it wrong | 21:36 |
| TheJulia | of course we'll put some big warnings around it, because we're not the arbiter of switch performance. | 21:37 |
| cardoe | yeah that's the thing different models and different builds will behave totally different | 21:40 |
| TheJulia | yup | 21:58 |
| opendevreview | Jacob Anders proposed openstack/sushy master: [WIP] Improve handling of empty-string eTags. https://review.opendev.org/c/openstack/sushy/+/956204 | 22:52 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!