| opendevreview | Julia Kreger proposed openstack/ironic stable/2023.2: Fix spurious CI job failures around partition images https://review.opendev.org/c/openstack/ironic/+/918646 | 00:00 |
|---|---|---|
| opendevreview | Merged openstack/ironic stable/2024.1: Fix spurious CI job failures around partition images https://review.opendev.org/c/openstack/ironic/+/918645 | 03:15 |
| opendevreview | Michal Nasiadka proposed openstack/ironic-python-agent master: Add LVM based image support to MD scenario https://review.opendev.org/c/openstack/ironic-python-agent/+/816685 | 08:50 |
| opendevreview | Michal Nasiadka proposed openstack/ironic-python-agent master: Add LVM based image support to MD scenario https://review.opendev.org/c/openstack/ironic-python-agent/+/816685 | 09:01 |
| opendevreview | Michal Nasiadka proposed openstack/ironic-python-agent master: Add LVM based image support to MD scenario https://review.opendev.org/c/openstack/ironic-python-agent/+/816685 | 10:22 |
| opendevreview | Michal Nasiadka proposed openstack/ironic-python-agent master: Add LVM based image support to MD scenario https://review.opendev.org/c/openstack/ironic-python-agent/+/816685 | 10:32 |
| iurygregory | good morning Ironic | 11:30 |
| cid | \o | 12:05 |
| mnaser | hmm, shouldn't the root disk partition be the last one on the disk so growpart can work? | 12:08 |
| mnaser | I'm ending up with uefi boot fs part, bios boot part, root os part, then configdrive part | 12:08 |
| mnaser | and this is using a whole disk image because that's what seems to be recommended by the ironic team | 12:09 |
| mnaser | this is kinda what I end up with https://usercontent.irccloud-cdn.com/file/ISfCtkPn/image.png | 12:10 |
| iurygregory | I couldn't access the url (504 HTTP) .-. | 12:12 |
| iurygregory | how did you create the wholedisk image? | 12:13 |
| mnaser | never mind... https://usercontent.irccloud-cdn.com/file/6gVt92xo/image.png | 12:13 |
| mnaser | oh looks like irc cloud is popping the bed | 12:13 |
| mnaser | sorry screenshots are annoying because im in a no-internet no-nothing env lol | 12:13 |
| iurygregory | omg =( | 12:13 |
| mnaser | but pretty much it seems to be related to multipath | 12:13 |
| iurygregory | I'm about to run after reading this word ... | 12:14 |
| iurygregory | multipath is funny... | 12:14 |
| iurygregory | this reminds me I should go back to https://review.opendev.org/c/openstack/ironic-python-agent/+/902012 :facepalm: | 12:14 |
| mnaser | iurygregory: https://imgur.com/tvppCtz | 12:14 |
| mnaser | and https://imgur.com/1WxdMUG | 12:15 |
| mnaser | I feel like I don't know why the root is not using the /dev/mpath | 12:15 |
| iurygregory | I like the fact it complains about not having a dir ..-part3 but it says it ran successfully | 12:16 |
| mnaser | loool | 12:17 |
| mnaser | and also | 12:17 |
| iurygregory | mnaser, in the df -h I don't see /dev/mpath... | 12:17 |
| mnaser | well believe t or not | 12:18 |
| mnaser | /dev/mpatha exists but that long string of things does not | 12:18 |
| iurygregory | I can totally imagine | 12:18 |
| iurygregory | multipath works in another schema of the universe | 12:18 |
| mnaser | yeah :( | 12:19 |
| mnaser | so we need to figure out what is making those names change | 12:19 |
| mnaser | and make it cnsistent | 12:19 |
| mnaser | at least so that it makes cloud-init happy | 12:19 |
| iurygregory | yeah, let me see if I can find something | 12:19 |
| iurygregory | mnaser, can you chack what lsblk shows for the disks? | 12:24 |
| iurygregory | not sure what you already tested, I'm just looking at the logic we have in IPA to see if helps | 12:25 |
| opendevreview | Merged openstack/ironic stable/2023.2: Fix spurious CI job failures around partition images https://review.opendev.org/c/openstack/ironic/+/918646 | 13:04 |
| TheJulia | mnaser: I think the tl;dr is cloud-init is not the end solver of non-cloud edge cases like multipath IO expecting to be working exactly like static disk devices | 13:14 |
| TheJulia | mnaser: I know steve wrote a tool for RHOSP called growvols to do targetted disk partition size expansion as well to address some of that disconnect | 13:15 |
| mnaser | TheJulia oh right. I tried to build an image with that but it didn’t extend. I forgot to dig into why/how it didn’t do its thing | 13:17 |
| TheJulia | mnaser: so we actually trigger it because it turns out people have preferences for how much | 13:18 |
| mnaser | I think in this case the root volume is being mounted without a user friendly name and then config drive later is.. or something renames them down the line. | 13:18 |
| TheJulia | after deploy that is, and that is outside of ironic | 13:18 |
| TheJulia | a... user... friendly name with multipath?! | 13:18 |
| mnaser | I should dig into growvols again I think maybe that might be the best approach | 13:18 |
| TheJulia | what is this heracy! | 13:18 |
| mnaser | mpatha isn’t so bad | 13:18 |
| TheJulia | nope, its not | 13:18 |
| iurygregory | I would be happy if there was a user friendly name with multipath lol | 13:22 |
| iurygregory | we already don't have standards for the output =P | 13:23 |
| TheJulia | It would be nice if multipath-tools had a json output which had a static data model that wouldn't change | 13:28 |
| iurygregory | that would be perfect | 13:32 |
| mnaser | iurygregory: sorry, I got sucked out into a few other things | 15:24 |
| mnaser | and a lot more of these https://usercontent.irccloud-cdn.com/file/xkOgQziP/image.png | 15:25 |
| mnaser | seriously, irc cloud still down? | 15:25 |
| TheJulia | Worked for me :) | 15:42 |
| JayF | it was a 502 when I clicked it before | 15:43 |
| JayF | now it's a 504 timeout | 15:43 |
| mnaser | ok boo | 16:21 |
| mnaser | even if I disable user friendly names | 16:21 |
| mnaser | cc_growpart just doesn't support multipath, it assumes dm => encrypted vols | 16:21 |
| mnaser | growvols is only for lvm but the lvm element is 3 yrs old and can't build me an ubuntu image | 16:21 |
| mnaser | the choices are slowing down lol | 16:22 |
| * JayF has never worked with multipath I/O and wishes you luck :/ | 16:22 | |
| JayF | I will say DIB elements are relatively simple to write, and easy to downstream | 16:23 |
| mnaser | yeah, I am thinking a custom growpart might be the thing here | 16:23 |
| JayF | so if you know what needs to be done, you might be able to write something to plop a grow script in rc.local (or as a oneshot systemd unit, etc) | 16:23 |
| clarkb | mnaser: how is lvm failing with ubuntu? you might need to use ubuntu-minimal instead? In particular ubuntu starts from the upstraem cloud images which may not play nice with custom disk management like lvm | 16:24 |
| clarkb | ubuntu-minimal is built up from scratch using debootstrap and that may work nicer with disk management | 16:24 |
| mnaser | clarkb: it failed with an assertion error wrt disk space pretty much | 16:24 |
| mnaser | https://usercontent.irccloud-cdn.com/file/ZdojFOtD/image.png | 16:25 |
| mnaser | apologies for screenshots, once again crappy vdi env I can't copy out of :< | 16:25 |
| mnaser | ubuntu-minimal maybe a good idea, but the lvm element hasn't been touched for 3 years either so I felt maybe it may have bitrotted | 16:25 |
| clarkb | or its been stable. The things that tend to need changes are "$distro uses this network tool now instead of that one" stuff like filesystems and disks are very stable | 16:26 |
| clarkb | and thats in the block device code which is relatively well exercised | 16:26 |
| clarkb | I believe opendev uses it for some of our images (arm?) | 16:26 |
| mnaser | clarkb: so arm uses block-device-uefi-lvm ? | 16:27 |
| clarkb | mnaser: not lvm but a custom partition setup: (and it was debian not arm) https://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml#L228-L245 | 16:29 |
| clarkb | mnaser: looks like we're using mbr in that partition scheme and you're failing in gpt. Did you provide a custom layout in DIB_BLOCK_DEVICE_CONFIG. Looks like the error is beacuse you haven't provided sufficient space for the layout | 16:30 |
| clarkb | basically I think this isn't an lvm error. This is a sanity check when trying to write out the disk and its saying there isn't enough disk to handle the content. Trying to work backward to determine how the disk size is set | 16:33 |
| mnaser | I think I will have to investigate a bit more on this | 16:42 |
| mnaser | the other fun thing now is figuring out how I can stand up trunked ports (but ideally natively through openstack directly) | 16:42 |
| JayF | I know all the metadata is plumbed through neutron/ml2 drivers for it | 16:45 |
| JayF | (the network config you're after is what we ran for OnMetal, vlans on top of a bond) | 16:45 |
| clarkb | reading the code that your screenshot shows failing what it is doing is indicating the size of your gpt partition is larger than free disk size. The local_loop portion of the config (you can see it in my example block) can specify a disk size. If you don't you get a default size. It will either log "Image size [$size]" or "Using default image size [%s]" | 16:45 |
| clarkb | my read of that is that either you've specified a size that simply isn't big enough or the default isn't large enough for your gpt parition (because you've specified a gpt partition size that is larger than the default base loopback device size) | 16:46 |
| mnaser | JayF: yeah I wonder if just creating a trunk and ports inside using neutron will just ~make it happen~ but I doubt it lol | 16:46 |
| JayF | if you have ml2+ngs setup ... | 16:46 |
| JayF | there is a nonzero chance | 16:46 |
| mnaser | clarkb: yeah, I think that ism next thing to tune, it feels like it had a little bit more space | 16:46 |
| mnaser | JayF: nah, ml2/ovn but also using lace fallback to get IPA to work, so when the bond is up, I can use "the new config" | 16:47 |
| JayF | oh, are you using our OVN VTEP switch support landed last cycle? | 16:48 |
| mnaser | so provisioning network = unlacp-d, then when it finishes deploying it goes into the lacp'd state when it first boots into a trunk port | 16:48 |
| JayF | or something else | 16:48 |
| mnaser | no no, nothing fancy, things are static on the switch side, im just using the fact no lacp up = provisioning/cleaning network, lacp up = final network (all bms live on the same network) | 16:49 |
| opendevreview | Julia Kreger proposed openstack/metalsmith master: Error with clarity when a bad upgrade was encountered https://review.opendev.org/c/openstack/metalsmith/+/918664 | 16:49 |
| JayF | That significantly reduces the security you gain from using a separate provisioning/cleaning network | 16:49 |
| mnaser | I agree, but this is a very single tenant environment and there's no way I can ever login to the switches there ever :) | 16:50 |
| JayF | I figured, just wanted to make sure it was said out loud :D | 16:53 |
| JayF | in this case... it should just work, right? | 16:53 |
| JayF | why wouldn't it just work? cloud-init supports that kinda metadata | 16:53 |
| mnaser | yeah but I'm wondering if when creating the vlans what will "inject" all of these things | 16:54 |
| mnaser | like it would be nice if I created a trunk port in neutron and created the nova instance and it ... magically showed up in the configdrive | 16:54 |
| JayF | as long as it's all structured properly in neutron | 16:54 |
| JayF | it will be correct in network_data.json | 16:54 |
| JayF | and should configure on the machine -- I know bonds with vlans on top are generally supported | 16:54 |
| JayF | if it's truly 100% static on the switch | 16:55 |
| JayF | this is one of those bits where: if it's like you've described, you might be surprised as how smoothly it works | 16:55 |
| mnaser | JayF: I just wasn't sure how ironic mutated the config since I noticed the the bond was in the network_data.json | 16:55 |
| mnaser | so wasn't sure if like.. ironic was slurping it and mutating it somehow | 16:55 |
| clarkb | looks like default image size is based on the size of the content that will go in the image + some small amount of extra room by default (66% more). If the actual image is small but you've requested a largish gpt partition I could see this happening. For example a 600MB image would add up to about 1GB of image space but then if you are requesting a GPT partition of say 20GB | 16:55 |
| clarkb | you'd fail because you're well over the 1GB that was provisioned | 16:55 |
| clarkb | so anyway I don't think ubuntu vs ubuntu-minimal is the problem or even lvm. Its just a matter of getting the numbers to align | 16:56 |
| TheJulia | mnaser: we don't mutate it at all, we just pass it through | 16:56 |
| JayF | Nah, Ironic just passes it through. | 16:56 |
| mnaser | so where is this magic coming from lol https://usercontent.irccloud-cdn.com/file/Pt92g4Za/image.png | 16:57 |
| JayF | nova assembles the configdrive | 16:58 |
| JayF | lemme find the code | 16:58 |
| mnaser | ahh so nova ironic virt driver detects there is a port group I guess | 16:59 |
| JayF | https://opendev.org/openstack/nova/src/branch/master/nova/virt/ironic/driver.py#L1124 | 16:59 |
| JayF | it's in the network metadata | 16:59 |
| JayF | this isn't even really ironic-specific afaict | 16:59 |
| JayF | https://opendev.org/openstack/nova/src/branch/master/nova/virt/ironic/driver.py#L1043 well, we assemble it ourselves | 17:00 |
| mnaser | ok cool cool | 17:00 |
| JayF | like I said, you're using an identical setup to what onmetal used | 17:00 |
| mnaser | yeah so nova's ironic driver takes that and feeds it to to ironic that doesn't touch it | 17:00 |
| JayF | and I'm pretty sure we dotted all the i's and crossed the t's upstream :D | 17:00 |
| JayF | yep exactly | 17:00 |
| JayF | if you're using Ironic standalone, you can pass in JSON instead of a full configdrive and we do some assembly IIRC | 17:01 |
| JayF | but that's not the case for nova integration | 17:01 |
| cid | I'm trying to setup ironic-tempest-plugin testing locally, is that possible? | 19:08 |
| cid | Devstack is already running and I have my patches (ironic and ironic-tempest-plugin) localized as well. I know a systemctl restart|stop&start devstack@ir-.*\.service will pick up my local changes(?). | 19:08 |
| cid | 19:09 | |
| cid | 19:09 | |
| cid | But just running itp at this point skips with a “Neutron not available” reason. | 19:09 |
| cid | 19:11 | |
| cid | Also, I doubt the possibility of getting the ironic patch on ironic-tempest dependence feature locally? | 19:11 |
| opendevreview | Tony Breeds proposed openstack/bifrost master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/bifrost/+/918825 | 19:22 |
| iurygregory | cid, so you added the patches in the local.conf correct? | 19:24 |
| iurygregory | restart | start stop won't pickup code changes (even if you apply in the repos if I recall) | 19:25 |
| cid | iurygregory, No, not in local.conf | 19:26 |
| cid | Changes to that will require rebuilding devstack, I think | 19:27 |
| iurygregory | yeah it would | 19:27 |
| iurygregory | afaik is the only way to pick-up changes that are not merged | 19:28 |
| iurygregory | let me see if I can find the link | 19:28 |
| opendevreview | Tony Breeds proposed openstack/ironic master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic/+/918896 | 19:28 |
| opendevreview | Tony Breeds proposed openstack/ironic-inspector master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic-inspector/+/918897 | 19:28 |
| opendevreview | Tony Breeds proposed openstack/ironic-inspector-specs master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic-inspector-specs/+/918898 | 19:28 |
| opendevreview | Tony Breeds proposed openstack/ironic-lib master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic-lib/+/918899 | 19:28 |
| opendevreview | Tony Breeds proposed openstack/ironic-prometheus-exporter master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/918900 | 19:28 |
| opendevreview | Tony Breeds proposed openstack/ironic-python-agent master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic-python-agent/+/918901 | 19:29 |
| opendevreview | Tony Breeds proposed openstack/ironic-python-agent-builder master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/918902 | 19:29 |
| opendevreview | Tony Breeds proposed openstack/ironic-specs master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic-specs/+/918903 | 19:29 |
| opendevreview | Tony Breeds proposed openstack/ironic-tempest-plugin master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/918904 | 19:29 |
| opendevreview | Tony Breeds proposed openstack/ironic-ui master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/ironic-ui/+/918905 | 19:29 |
| cid | That means JayF, must have done some magic at least once | 19:29 |
| iurygregory | cid, https://docs.openstack.org/ironic/latest/contributor/devstack-guide.html#downloading-an-unmerged-patch-when-stacking | 19:29 |
| iurygregory | from what I remember this would be the way do have a patch that is not merged running in a local devstack | 19:29 |
| iurygregory | it's been a long time I've deployed devstack (maybe something changed?) | 19:30 |
| JayF | Do not assume that whatever I did to get it working in a hurry was the right thing. I think iurygregory pointed you to the official way. Maybe you can teach me the right way | 19:32 |
| cid | This link will go a long way.. | 19:32 |
| opendevreview | Tony Breeds proposed openstack/metalsmith master: [DNM] Testing docs bump with new Sphinx https://review.opendev.org/c/openstack/metalsmith/+/918937 | 19:33 |
| iurygregory | not sure how much unstack.sh works, but basically you would run unstack.sh, update your local.conf to point to the changes in ironic and ironic-tempest-plugin and run stack.sh | 19:34 |
| JayF | There are other, more manual ways to do it. I suspect learning how to use devstack to do it as the better path long term though | 19:35 |
| cid | iurygregory: "...update your local.conf to point to the changes in ironic and ironic-tempest-plugin and run stack.sh" can this be the local repo | 19:38 |
| iurygregory | cid, you mean the cloned repo from ironic and ironic-tempest-plugin? | 19:38 |
| cid | yes | 19:38 |
| iurygregory | I mean, it could work for the tempest-plugin I think.., from what I remember about devstack is that it will clone the upstream repos from the master branch (unless you tell a specific stable/branch, or a patch that you need) | 19:40 |
| cid | ok, thanks | 19:45 |
| cid | I will poke around, I'm looking to do a lot of the testing locally. | 19:46 |
| cid | good night \o | 19:46 |
| JayF | yeah the code itself is in /opt/stack | 19:48 |
| JayF | so what I did originally was just go into /opt/stack/ironic after spinning up a master-branch devstack and manually issue the changes, and then restarted devstack@ir-api & devstack@ir-cond | 19:48 |
| JayF | that's always been my workflow with devstack, but it may not be the best choice for everyone as I'm a bit more comfortable on linux systems than some developers | 19:49 |
| cid | Yeah, that's the approach I was trying to reference earlier :D | 19:50 |
| JayF | I was on mobile having lunch or I would've expounded then :D | 19:50 |
| opendevreview | Igor Kádson de Souza Oliveira proposed openstack/bifrost master: Update deprecated config in troubleshooting https://review.opendev.org/c/openstack/bifrost/+/918948 | 20:58 |
| opendevreview | Merged openstack/bifrost master: Update deprecated config in troubleshooting https://review.opendev.org/c/openstack/bifrost/+/918948 | 21:20 |
| iurygregory | JayF, was fast =P I almost didn't have time to give my +2 | 21:21 |
| JayF | If I'm having a conversation in IRC between work items, and I see a change pop up, I try to get someone a quick review :D | 21:21 |
| JayF | just like when I land something really, really old sometimes ... those are cool moments in OSS (when someone instareviews your stuff, or lands something you wrote years old) so I like enabling those for other people | 21:21 |
| iurygregory | i totally understand | 21:21 |
| iurygregory | I was helping igor to get his first contribution :D he found that issue while deploying bifrost | 21:22 |
| JayF | too late, I already helped him | 21:30 |
| JayF | turn over your gold star ;) | 21:30 |
| JayF | Guess what work I'm currently distracted from (writing those presentations lol) | 21:32 |
| iurygregory | :D | 21:34 |
| iurygregory | yeah | 21:34 |
| iurygregory | it was funny to see an old machine not working because the image was wrong XD | 21:34 |
| JayF | whoops, that was in the wrong channel | 21:34 |
| JayF | Well I'm glad he found+fixed it | 21:35 |
| mnaser | boo | 21:43 |
| mnaser | it seems `vlan_mac_address` and all the other vlan friends are actually not generated/passed/anything in nova.. | 21:44 |
| mnaser | it's in nova's network_data schema.. but I can't find anywhere that it generates with that | 21:44 |
| JayF | I suspect that's a bug? I'm not sure though. | 21:46 |
| mnaser | https://codesearch.opendev.org/?q=vlan_mac_address&i=nope&literal=nope&files=&excludeFiles=&repos= | 21:47 |
| mnaser | Bifrost uses it.. some airship/etc stuff use it, but nova does not | 21:47 |
| mnaser | booo | 21:47 |
| mnaser | that may bring me many steps back now sadly to relying directly to ironic and drop nova to get what I need done | 21:48 |
| JayF | I'd strongly suggest filing a bug with your use case and specific missing fields | 21:50 |
| JayF | it's 100% in the schema as located in the nova api doc | 21:50 |
| JayF | so if they should be passing some metadata thru and are not, this would seem like a bug for me | 21:51 |
| JayF | (at least using vlan_mac_address as an example) | 21:51 |
| clarkb | the former network person in me is struggling with the idea of a vlan mac address | 22:05 |
| clarkb | mac addresses are for interfaces not networks | 22:05 |
| TheJulia | Same… | 22:05 |
| JayF | clarkb: linux will happily spoof a different mac for each vlan | 22:05 |
| clarkb | JayF: ya but thats still an interface address | 22:06 |
| JayF | but in this case, I think it's more to ID VLAN<->physical IF mappings | 22:06 |
| JayF | e.g. the vlan_mac_address isn't saying "this is the mac address to set for this VLAN", it's saying "this is the mac address for the interface this VLAN lives on" iirc | 22:06 |
| clarkb | ah that makes more sense | 22:06 |
| mnaser | Yeah also it seems all the vlan stuff don’t really get passed down like vlan_id etc | 22:21 |
| mnaser | I can’t find anything that generates that in the network data :( | 22:21 |
| JayF | johnthetubaguy: Do we use bonded interfaces with VLANs on top downstream? I think so, yeah? We don't have any patches for it do we? | 22:23 |
| JayF | mnaser: I have a hypothesis, with no data yet, that it may be an OVN v OVS kinda thing | 22:23 |
| JayF | mnaser: but I believe this is possible, and if it's not, it's something that's not acceptable and I'd prioritize fixing | 22:24 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
