| opendevreview | Yu Zou proposed openstack/ironic master: feat: add verify ca conf support for drivers https://review.opendev.org/c/openstack/ironic/+/947544 | 02:49 |
|---|---|---|
| JayF | I got what I needed, thanks | 03:00 |
| opendevreview | Amarachi Ordor proposed openstack/bifrost master: docs: updated test guide, improve how-to clarity, add troubleshooting tips https://review.opendev.org/c/openstack/bifrost/+/946953 | 03:35 |
| arne_wiebalck | Good morning, Ironic! | 06:36 |
| rpittau | good morning ironic! o/ | 07:13 |
| abongale | Good morning ironic! | 08:15 |
| freemanboss[m] | Good morning abongale: | 08:17 |
| abongale | Good morning freemanboss[m] | 08:18 |
| queensly[m] | Good morning | 08:36 |
| opendevreview | Merged openstack/ironic master: Update some docs https://review.opendev.org/c/openstack/ironic/+/947880 | 11:12 |
| iurygregory | omg arne_wiebalck is around =P | 11:52 |
| iurygregory | good morning ironic o/ | 11:52 |
| arne_wiebalck | Good morning iurygregory o/ | 12:02 |
| dtantsur | cid: https://review.opendev.org/c/openstack/ironic-python-agent/+/946091 passed most integration jobs already, yay! | 12:44 |
| dtantsur | for the next step, please update https://review.opendev.org/c/openstack/requirements/+/947727 per Sean's request (include answers to questions in the commit message) | 12:45 |
| dtantsur | then https://review.opendev.org/c/openstack/ironic-python-agent/+/946091 needs a release note to warn operators that something significant has changed + we can merge https://review.opendev.org/c/openstack/ironic-python-agent/+/947744/ into it | 12:46 |
| cid | dtantsur: And it has passed all \o/ | 13:09 |
| dtantsur | \o/ | 13:23 |
| TheJulia | good morning | 13:30 |
| JayF | cid: dtantsur: One other thing we need to remember to do is to check and see if we need to do anything explicit to pick an oslo.$anything back end that's not enventlet based. Even if that's happening automatically, we need to raise the requirements to only require Oslo versions that don't use eventlet | 13:32 |
| dtantsur | fair point | 13:37 |
| cid | ++ | 13:43 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add several NV GPU types https://review.opendev.org/c/openstack/ironic/+/947213 | 15:22 |
| opendevreview | Kaifeng Wang proposed openstack/ironic-specs master: Flow control on PXE related port requests https://review.opendev.org/c/openstack/ironic-specs/+/947961 | 15:46 |
| opendevreview | Merged openstack/ironic master: Trivial: Fix spelling issue in configuration desc https://review.opendev.org/c/openstack/ironic/+/947881 | 17:36 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: Remove eventlet from Ironic Python Agent https://review.opendev.org/c/openstack/ironic-python-agent/+/946091 | 18:03 |
| JayF | this should be ready for review once the requirements change merges \o/ go cid go | 18:04 |
| JayF | Should https://opendev.org/openstack/molteniron still exist? | 18:07 |
| JayF | cd .. | 18:08 |
| JayF | whoops | 18:08 |
| TheJulia | likely not at this point | 18:09 |
| JayF | So I did an audit of all Ironic managed projects; IPA migration pending review, only remaining ones that need eventlet gone are NGS, NBM, and Ironic-proper | 18:10 |
| TheJulia | cool, that matches my upstream and downstream notes | 18:16 |
| opendevreview | Merged openstack/ironic master: Add several NV GPU types https://review.opendev.org/c/openstack/ironic/+/947213 | 18:34 |
| opendevreview | Jay Faulkner proposed openstack/networking-baremetal master: WIP: Remove eventlet https://review.opendev.org/c/openstack/networking-baremetal/+/947985 | 18:43 |
| JayF | CID and I did some research, it looks like oslo.service does not have a working backend for anything but eventlet right now, and the WIP one didn't work (assuming I used it right). It will not be a good use of our time to pursue further eventlet migrations until this is in a better state. | 18:46 |
| dtantsur | I'm curious how much of oslo.service we'll need once we get rid of its wsgi bits | 18:51 |
| JayF | LoopingCall is a big user | 18:51 |
| JayF | another one too | 18:51 |
| dtantsur | LoopingCall should be reasonably easy to replace | 18:52 |
| JayF | but they have the threading backends for those in progress, I see no need to reimplement it on top of them instead of just waiting | 18:52 |
| dtantsur | cid, JayF, https://review.opendev.org/c/openstack/requirements/+/947727 needs more love still, they won't merge it until we do the needfull | 18:52 |
| dtantsur | yeah, I agree, maybe we just wait at least for some time | 18:53 |
| cid | Yeah, on my list of things to do today. Though, the last time I looked at it, it wasn't very clear. Things always make sense to me the second time, so, I should have addressed any concerns on that today, tomorrow | 18:53 |
| TheJulia | I'd be *super* hesitant to wait beyond the end of this cycle | 18:53 |
| dtantsur | cid: the wiki is a bit vague, but check any previous patch that added a new requirements | 18:53 |
| dtantsur | e.g. https://review.opendev.org/c/openstack/requirements/+/934220 | 18:54 |
| * dtantsur disappears again since it's 9pm | 18:54 | |
| cid | ++ | 18:55 |
| JayF | TheJulia: I already put a comment linking to the NBM change on that oslo.service cchange, I really think it's close | 19:00 |
| JayF | TheJulia: and I suspect with a waiting user it might go even faster | 19:00 |
| TheJulia | ack | 19:01 |
| JayF | yeah literally that threading patch updated this morning and is moving fast | 19:01 |
| JayF | Herve and Daniel on the case :D | 19:01 |
| TheJulia | cool cool | 19:37 |
| Brodey | Hey everyone! I hope it's okay to ask questions in here... If I'm doing this improperly I apologize (first time user). | 20:32 |
| Brodey | We are struggling with an Ironic problem right now where we are trying to plug tenant network vlans on all baremetal node ports (we have 12 ports). 1 is pxe_enabled which works through the cumulus genericswitch plugin, but when booting the machine into the tenant network, we're noticing the ports are never sent to the ml2 driver for plugging. Is this supported somehow? | 20:34 |
| JayF | You're in the (well, a) right place. Although for a question like that you might have better luck on the list. Just hang out a bit and maybe someone will know how to point you | 20:36 |
| Brodey | Thanks Jay, will post in mailing list if there's no luck in here. | 20:37 |
| * TheJulia sees 12 ports and blinks | 20:37 | |
| TheJulia | Brodey: what is the network_interface set to? | 20:38 |
| TheJulia | for the ironic baremetal node | 20:38 |
| Brodey | network_interface is neutron | 20:39 |
| TheJulia | could you provide insight into the vifs your attempting to plug, your saying vlans and I guess we need to try an dunderstand structure and if it matches out or not | 20:40 |
| Brodey | The VIFs we’re attempting to plug are on VLAN-backed provider networks. Each baremetal port right now is mapped to a physical switch/interface using local_link_connection, and we assign VLAN segments to the networks manually. We then boot through nova using --network flag. | 20:43 |
| Brodey | Sorry, hope that is helpful | 20:43 |
| TheJulia | no, that is good | 20:43 |
| TheJulia | does `openstack baremetal node vif list <node_uuid>` work for the deployed node indicating the vifs which are expected ? | 20:44 |
| TheJulia | those should all have segmentation ids already set | 20:44 |
| Brodey | If I list VIFs on the baremetal node, I only see one, which I assume is the VIF for the pxe_enabled port | 20:45 |
| Brodey | Yeah, confirmed. That's the pxe_enabled port. | 20:46 |
| TheJulia | if you ask neutron to show that vif (port) (openstack port show <vif_id>), what do you see? | 20:46 |
| Brodey | It's the active port with the mac_address that maps to the pxe_enabled port on the baremetal node. bound_drivers.0='genericswitch' | 20:47 |
| Brodey | binding_vnic_type baremetal | 20:47 |
| TheJulia | any change you can share the output, redacting mac address and ip addreses ? | 20:48 |
| Brodey | sure, paste the entire json in here? | 20:48 |
| TheJulia | use paste.openstack.org :) | 20:48 |
| TheJulia | and share the link | 20:48 |
| Brodey | roger, one sec | 20:48 |
| Brodey | https://paste.openstack.org/show/btDcoC83X0MpGGue1lqc/ | 20:50 |
| TheJulia | I guess that is a port which is being requested with openstack server create command as well? | 20:51 |
| TheJulia | since it has a name test-port ? | 20:51 |
| Brodey | Yep! | 20:52 |
| TheJulia | okay, so all signs point to only one vif actually being requested | 20:52 |
| TheJulia | And everything there looks good, but your missing everything else your seeking | 20:52 |
| Brodey | Indeed. Should we be attaching vifs manually? | 20:53 |
| TheJulia | which I guess might be a gap to do it based upon networks. (Long story short, there are some fun issues and challenges around how networking can be requested for nova instances and how that can get translated out to nodes, so I guess we need to understand the commands used in a bug and if you can provide reasonably redacted details into a bug in launchpad | 20:53 |
| TheJulia | you can, I'd think nova should be doin git, I'm wondering if it is some case which has a bug or if everyone has just done explict ports | 20:54 |
| TheJulia | (there are literally like 5 ways to get to a nova instance to have networking as well, which doesn't help the overall challenge and result) | 20:54 |
| Brodey | Hmmm, we should be able to create the ports and then attach VIFs for now if that would work? | 20:55 |
| TheJulia | (and turns out, if you request ports with deferred addressing, bad things also happen... which happens to be a bug I'm working on | 20:55 |
| TheJulia | yeah, that *should* work and I'd expect that to work | 20:56 |
| TheJulia | I'm not sure it is really expected by nova for one to mix models of requesting networking either, which might not be helping the overall issue | 20:56 |
| TheJulia | because your obviously getting a deployed instance | 20:56 |
| TheJulia | ironic is only being told 1 vif | 20:56 |
| TheJulia | when it sounds like there should be *12* | 20:57 |
| TheJulia | (btw, impressive!) | 20:57 |
| Brodey | Ok so at nova create time, go through each non-pxe_enabled port on the baremetal node, create a port in nova and then do a baremetal node vif attach --port-uuid | 20:57 |
| Brodey | Heh, yeah we're a GPU cloud provider and hoping to use Ironic, we're at POC stage. 12 ports include gpu fabric etc :) | 20:57 |
| TheJulia | you can just request the instance from nova with all the specific ports you want | 20:57 |
| TheJulia | AHHH | 20:57 |
| TheJulia | well, depending on what your doing, I'm sort of wondering if you even should be using nova in that case | 20:58 |
| Brodey | vs just deploying to ironic directly? | 20:58 |
| TheJulia | you can alternatively attach vifs directly after the fact or separately, you just want to be mindful of the overall flow | 20:58 |
| TheJulia | Sort of depends on what happy path you want to take | 20:58 |
| Brodey | Nova would be nice to have as our direct interface given we want to be able to provision VMs alongside baremetal | 20:59 |
| TheJulia | yeah | 20:59 |
| TheJulia | That is generally why folks tend to do it | 20:59 |
| TheJulia | at least when you have a blend of needs and cases | 21:00 |
| Brodey | So, your recommendation is to create all 12 ports for the nova instance upfront when creating the instance and let Nova/Neutron try to attach the VIFs? Otherwise, create the ports and do the baremetal vif attach manually? | 21:00 |
| Brodey | btw, we also tried neutron.add_all_ports option but that didn't seem to work for us | 21:01 |
| TheJulia | yeah, ideally, the flow which is actually used is nova will tell ironic what the vifs are, and then ironic will attempt to map the vifs to ports on the phyiscal node as part of the deployment/provisioning flow and not before | 21:01 |
| JayF | Brodey: which GPU cloud provider? I work for G-Research (OSS side) and we likely share a lot of problems :) | 21:02 |
| TheJulia | heh | 21:02 |
| JayF | Brodey: While I'm not a network guru, happy to touch brains at some point :) | 21:02 |
| * TheJulia was wondering until someone was going to jump on this discussion | 21:02 | |
| JayF | NGS troubleshooting is just not my bag 😆 | 21:02 |
| * TheJulia still expects her neighbor to remark about burning ears next when she gets home | 21:02 | |
| TheJulia | (although, different networking and use cases) | 21:03 |
| Brodey | Ok, sounds good. I'll try the first option (create all ports when creating the instance) and see what happens. | 21:03 |
| Brodey | Jay, I work for WhiteFiber, a pretty new GPU cloud. :) | 21:03 |
| TheJulia | Brodey: clearly you've brushed up against a bug, do you know where to file bugs? | 21:04 |
| JayF | Brodey: drop me a line at jay@gr-oss.io -- we have a whole group at GR-OSS that loves to sync up with folks solving similar problems, openstack or otherwise (you may have heard of armadaproject.io ?) | 21:05 |
| Brodey | I do not. Do you have a 'how to file a bug' link handy? otherwise I can find one. :) | 21:05 |
| JayF | bugs.launchpad.net | 21:05 |
| JayF | project name is ironic | 21:05 |
| Brodey | heh sounds good, appreciate it | 21:06 |
| Brodey | Jay, sounds good i'll do that | 21:06 |
| TheJulia | https://bugs.launchpad.net/ironic | 21:06 |
| TheJulia | A more direct-ish link ;) | 21:06 |
| JayF | https://bugs.launchpad.net/ironic/+filebug MAXIMAL DIRECT LINK :p | 21:06 |
| TheJulia | lol | 21:06 |
| JayF | honestly, I <3 that launchpad has useful urls like this | 21:06 |
| JayF | one of the benefit of using an older less wizbang tool | 21:07 |
| TheJulia | heh | 21:07 |
| * TheJulia gets text message from the wifey asking for me to stop for cat foot and wonders if it is just time to packup and head out | 21:07 | |
| JayF | TheJulia: btw; I haven't documented this anywhere upstream yet; but I've asked cid to look at port fetching perf in the api and to look at maybe hooking NBM to sharding | 21:09 |
| JayF | Basically I learned today that at some scaled environments it's turned off because that query times out | 21:09 |
| TheJulia | k, eek | 21:09 |
| JayF | yeah, I figured I'd mention to you specifically since you were on the nodes speedup | 21:09 |
| TheJulia | yeah, harald had a change for that someplace along the way | 21:10 |
| JayF | I showed him the benchmark stuff and pointed him at the 2019 blogpost | 21:10 |
| JayF | oooh | 21:10 |
| TheJulia | ... gawd 2019 was so long ago | 21:10 |
| JayF | cid: look at https://review.opendev.org/q/owner:hjensas@redhat.com+status:open for some relevant changes to that port/nbm speedup work | 21:11 |
| JayF | cid: looks like he has changes up for improving benchmarking and trying to setup filtering by conductor group | 21:11 |
| JayF | https://review.opendev.org/c/openstack/ironic/+/862292 in particular will be helpful | 21:12 |
| JayF | hook this up and shard to the node | 21:12 |
| TheJulia | https://review.opendev.org/c/openstack/ironic/+/864781 <-- the one I was thinking of | 21:13 |
| TheJulia | merged in 2023 | 21:13 |
| JayF | that means it's not in the downstream that reported the issue | 21:13 |
| JayF | nice | 21:13 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Patch configdrive metadata https://review.opendev.org/c/openstack/ironic/+/946677 | 21:16 |
| TheJulia | speaking of fun bugs | 21:16 |
| TheJulia | still need to check/validate the pre-mastered ISO image path and write more tests | 21:17 |
| TheJulia | JayF: fwiw, I don't think anyone sat down and benchmarked ports really after that, but yeah, any using any other pattern further away from the DB server makes port listing slow... just like it did with node listing back in the day | 21:18 |
| JayF | yep, it makes sense | 21:18 |
| JayF | I'm just glad to know we have performance to upgrade into | 21:18 |
| TheJulia | the assocation_proxy stuff ends up doing two closely knit db queries and sqlalchemy merges the results before returning the to the caller | 21:19 |
| TheJulia | ++ | 21:19 |
| TheJulia | speaking of networking-baremetal, one of vasyl's chorts posted a neutron patch which we really need to get folks to look at | 21:19 |
| TheJulia | which addresses orphaned agent records | 21:20 |
| TheJulia | well, enables it, adds a agent status call to the rpc method | 21:21 |
| * TheJulia begins to back up to head to the store and then back up the mountain (because triple overtime last night wasn't enough...) | 21:27 | |
| JayF | I gotta be honest, I'm too old for 3OT unless it's like a game 7 for the cup or semis lol | 21:27 |
| TheJulia | JayF: kind of yeah, I was starting to call it the game that never ends since we were like less than 2 minutes from a forth overtime | 21:28 |
| TheJulia | (wranglers v firebirds, so two strong offensive teams too) | 21:29 |
| TheJulia | see y'all in the morning | 21:42 |
| JayF | o/ | 21:43 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!