| opendevreview | Lajos Katona proposed openstack/project-config master: Remove taas-tempest-plugin and taas-dashboard jobs https://review.opendev.org/c/openstack/project-config/+/938665 | 08:27 |
|---|---|---|
| chandankumar | Thank you frickler sg-core depends-on on opendev is working now. :-) | 08:32 |
| *** jroll01 is now known as jroll0 | 08:45 | |
| *** tosky_ is now known as tosky | 09:33 | |
| *** diablo_rojo_phone is now known as Guest5496 | 09:51 | |
| *** ralonsoh_ is now known as ralonsoh | 09:58 | |
| *** Guest5496 is now known as diablo_rojo_phone | 10:40 | |
| opendevreview | Thierry Carrez proposed openstack/project-config master: Fix release ACL for whitebox-tempest-plugin https://review.opendev.org/c/openstack/project-config/+/938887 | 12:31 |
| opendevreview | Vladimir Kozhukalov proposed zuul/zuul-jobs master: [remove-registry-tag] Fix typo https://review.opendev.org/c/zuul/zuul-jobs/+/938892 | 12:45 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: Remove networking-midonet https://review.opendev.org/c/openstack/project-config/+/919415 | 14:00 |
| opendevreview | Takashi Kajinami proposed openstack/project-config master: oslo: Remove irc notification from oslo-incubator https://review.opendev.org/c/openstack/project-config/+/938897 | 14:03 |
| opendevreview | Lajos Katona proposed openstack/project-config master: Neutron: add review priority to ovn-bgp-agent and ovsdbapp https://review.opendev.org/c/openstack/project-config/+/938908 | 14:45 |
| opendevreview | Lajos Katona proposed openstack/project-config master: Neutron: add review priority to ovn-bgp-agent and ovsdbapp https://review.opendev.org/c/openstack/project-config/+/938908 | 14:54 |
| opendevreview | Lajos Katona proposed openstack/project-config master: Neutron: add review priority to ovn-bgp-agent and ovsdbapp https://review.opendev.org/c/openstack/project-config/+/938908 | 14:55 |
| opendevreview | Lajos Katona proposed openstack/project-config master: Neutron: add review priority to ovn-bgp-agent and ovsdbapp https://review.opendev.org/c/openstack/project-config/+/938908 | 15:05 |
| opendevreview | Lajos Katona proposed openstack/project-config master: Neutron: add review priority to ovn-bgp-agent and ovsdbapp https://review.opendev.org/c/openstack/project-config/+/938908 | 15:09 |
| opendevreview | Merged zuul/zuul-jobs master: [remove-registry-tag] Fix typo https://review.opendev.org/c/zuul/zuul-jobs/+/938892 | 15:36 |
| opendevreview | Clark Boylan proposed opendev/system-config master: Update to Gitea 1.23 https://review.opendev.org/c/opendev/system-config/+/938826 | 16:03 |
| opendevreview | Clark Boylan proposed opendev/system-config master: DNM intentional Gitea failure to hold a node https://review.opendev.org/c/opendev/system-config/+/848181 | 16:03 |
| clarkb | as predicted a .1 release was no far behind the .0. I will rotate the autohold | 16:03 |
| fungi | clarkb: thinking about the proposed pbr beta release, it's going to trigger a constraints update as well, is that something we should try to block, or just more useful testing? | 16:11 |
| clarkb | fungi: will it set the constraint to ===$betarelease? | 16:16 |
| clarkb | fungi: I think that may be useful testing but we should mark the change WIP as we don't actually want that to go into constraints proper | 16:16 |
| fungi | yeah, in theory it will | 16:16 |
| fungi | the propse job is in the pre-release pipeline and rc versions of cycle-with-milestone projects end up in there (ceilometer and tap-as-a-service were examples i saw) | 16:17 |
| fungi | anyway, i'm stepping out for a bit to grab lunch, but shouldn't be more than an hour | 16:18 |
| clarkb | looking at tonyb's homedir on bridge I think the ansible module for openstack cloud images may have been used for the image uploads and I'm guessing that just silently failed? | 16:26 |
| clarkb | I've got an updated vhd image locally that I converted from the current noble image. But I notice disk space is a bit tight on bridge so I don't want to copy it there yet | 16:27 |
| clarkb | I think we may need to use openstack sdk interactively or with a small script to go through the rax image upload process. I'll look at that next I guess | 16:27 |
| clarkb | ya looks like connection.create_image() should automatically figure out the rax upload process for us. We do need to pick a swift container name for the intermediate upload step whcih I guess I should check doesn't conflict | 16:39 |
| clarkb | default is "images" | 16:39 |
| clarkb | looks like we've already been using the images container. it exists with a couple of objects in ORD and IAD (haven't checked dfw yet) | 16:41 |
| clarkb | I'll need to figure out the disk space situation for bridge. Or I could assume tonyb's vhd image is fine to upload and just upload that instead of the one I generated | 16:42 |
| clarkb | I'll think about that a bit. I have to run an errand this morning so will pop out soon | 16:45 |
| clarkb | ya maybe the easiest thing to start is just to upload tonyb's image and see if that works then I don't have to go find things to delete (tonyb's image would be a candidate actually but using it keeps us in sync with the other images in other clouds which is nice) | 16:58 |
| mordred | clarkb: yah- create_image _should_ negotiate all of the magical crazy things needed for you | 17:10 |
| clarkb | mordred: ya looking at this I'm a bit surprised that the ansible stuff apparently didn't work | 17:23 |
| clarkb | but I'm not sure how tonyb acutally ran the ansible if at all | 17:23 |
| clarkb | anyway I'll use create_image() directly which should work. But first I need to drop a car off for new tires | 17:23 |
| fungi | did you check the shell history? | 18:08 |
| fungi | though i just did and am not seeing where it was executed either | 18:10 |
| fungi | i do see a few runs in ~root/.bash_history | 18:11 |
| fungi | e.g. `/usr/ansible-venv/bin/ansible-playbook /home/tonyb/image_upload.yaml` | 18:11 |
| Clark[m] | Ya that is the playbook but the image doesn't exist. So maybe Ansible can't negotiate rax uploads | 18:28 |
| clarkb | ok back after dropping the car off | 18:34 |
| clarkb | going to work on an upload to IAD of tonyb's image now | 18:34 |
| clarkb | that is in progress | 18:38 |
| clarkb | the image uploaded to iad I am now testing boots of it using config drive and will test without config drive after with config drive | 18:46 |
| clarkb | if that looks good then I guess I'll upload it to the other two regions | 18:46 |
| clarkb | there is still an open question on whether or not we want to use the image in tonyb's homedir without direct confirmation from tonyb that it should be safe | 18:47 |
| clarkb | openstack server show doesn't show any ip addresses on the server I created. I thought that was all pretty automagic in rax classic but maybe not? | 18:49 |
| clarkb | hrm network list doesn't do anything which I expected with the behavior that it is all automagic | 18:50 |
| clarkb | fungi: do you know what is necessary to boot with a network in rax iad? | 18:50 |
| clarkb | I'm booting a no config drive test ot see if that changes the behavior | 18:52 |
| clarkb | our nodepool clouds.yaml which is used to successfully boot things for nodepool doesn't appear to have any special network config for rax | 18:53 |
| clarkb | the no config drive boot seems to behave the same way (I expected that but figured it was worth a shot). console log show doesn't work with rax either so I can't tell if the VM thinks it has ip addresses | 18:55 |
| clarkb | if I try to boot a jammy instance using the cloud provided images I get the asme behavior so this likely isn't due to some introspection of image metadata | 18:58 |
| clarkb | I guess I'll test booting in ord and dfw next and if one of them work upload the image there? | 19:00 |
| clarkb | I mean test booting the cloud jammy image | 19:00 |
| clarkb | api requests to ord are very slow but thats what I'm doing now. Trying to create a jammy node in ord and see if I have networking | 19:03 |
| clarkb | seems like the same thing there | 19:04 |
| clarkb | ideas: maybe it is a client version problem? | 19:05 |
| clarkb | I'll try using the launch env openstack client | 19:05 |
| clarkb | just need to cleanup what I did in ord first which is going to take a minute with the api response slowness | 19:10 |
| clarkb | using the launcher env openstack client doesn't seem to have changed the behavior | 19:15 |
| clarkb | infra-root has anyone seen this before? | 19:15 |
| clarkb | I feel like it must be pebkac and I'm issuing the server create with missing information, but I can't network list so there isn't any network info to provide | 19:16 |
| clarkb | server create --auto-network is the default looks like. And the alternative is --nic or --network with specific information. Not understanding how there would be specific information to provide though | 19:20 |
| clarkb | I'm going to try with openstacksdk conn.create_server next I guess | 19:20 |
| clarkb | if in doubt use openstacksdk directly. Which is a terrible user experience but this works | 19:24 |
| clarkb | but that was booting the jammy image still. Time to finally test if I can get into the noble node booted off of tonyb's image | 19:24 |
| clarkb | success! tonyb's image boots but only with config drive set | 19:29 |
| clarkb | the no config drive setup seems to not work (I guess cloud init doesn't understand metadata there?) | 19:29 |
| clarkb | so I think what I will do is clean up what I've done | 19:30 |
| clarkb | then upload this image to ORD and DFW | 19:30 |
| clarkb | then probably Monday figure out launching a new paste | 19:30 |
| clarkb | that gives us time to decide if we want to use a newer image and not trust this one since tonyb hasn't confirmed its good yet and I have some other stuff I can finish up today | 19:30 |
| clarkb | gtema: fyi I am able to boot instances in rackspace classic using openstacksdk conn.create_server but when using osc I get no ip addresses | 19:31 |
| clarkb | gtema: I'm not sure what the difference in the plumbing is to result in that but you can't specify networks in those clouds aiui instead you have to rely on what the cloud assings to you | 19:32 |
| gtema | Clarkb code for managing network for servers is something close to 10% of the overall openstacksdk codebase. CLI is not using the same path and there are lots of possible entry points | 19:40 |
| gtema | I am done for today, but if you want I can have a deeper look on Monday on differences. I would also strongly recommend enable debugging and record all API requests to get a rough idea of what's going wrong | 19:42 |
| clarkb | gtema: oh thats agood idea. I should be able to compare the debug output hetween osc and create_server() too | 19:42 |
| clarkb | not sure I'll get to that before monday but that would be the next step I think | 19:42 |
| clarkb | gtema: enjoy your weekend. This isn't urgent I have a workaround. I just wanted to make sure it wasn't missed with the impending weekend | 19:43 |
| gtema | Great | 19:43 |
| gtema | Same to you | 19:43 |
| clarkb | infra-root to tl;dr we can't use osc to boot instances in rax right now they don't get IP addrs. But using create_server in the sdk directly works. I have also used create_image in the sdk directly to upload tonyb's noble vhd image on bridge to dfw, ord, and iad | 19:48 |
| clarkb | I've also cleaned up all my test instances so we should be in good shape for that now I hope. You do have to use config drive with that image though. I don't expect this to be a problem for us | 19:49 |
| fungi | clarkb: i don't recall having to specify a network the last time i booted a server in rax classic, though i expect we would pin one in our clouds.yaml for nodepool at least if it were necessary | 20:05 |
| fungi | but yeah, you arrived at the same conclusion | 20:05 |
| fungi | looks like lists01.opendev.org was the last server i created there, used the "Ubuntu 22.04 LTS (Jammy Jellyfish) (Cloud)" image. that was a little over 2 years ago (2022-11-22) | 20:10 |
| fungi | oh, no, i also booted keycloak03.opendev.org there less than a year ago, 2024-02-08 | 20:12 |
| fungi | at the time, this was the command i used: | 20:13 |
| fungi | sudo /usr/launcher-venv/bin/launch-node --cloud=openstackci-rax --region=DFW --flavor='4 GB Performance' --image='Ubuntu 22.04 LTS (Jammy Jellyfish) (Cloud)' keycloak03.opendev.org | 20:13 |
| fungi | so no specific network, no bfv | 20:13 |
| comay | list #openstack | 20:19 |
| clarkb | fungi: ya launch node uses openstacksdk so I suspect it will work too | 20:21 |
| clarkb | fungi: I didn't want to launch node since I just wnted to check if ssh and the image functioned. It was a good learning exercise | 20:21 |
| fungi | oh! for some reason i thought we wrapped the cli with it | 20:24 |
| fungi | clarkb: so i double-checked, i can still use launch-node to boot rackspace's jammy image as above, but trying the same exact command with our noble image doesn't work: "While testing ssh access: [Errno None] Unable to connect to port 22..." | 20:39 |
| Clark[m] | fungi: you need to use config drive. But also it's an image I found in tonyb's homedir so not sure if we want to verify it's origins before using it for something more | 20:53 |
| Clark[m] | I did convert my own image but there isn't much room left on bridge to upload it into rax from. But maybe just getting it uploaded is a good idea..I dunno..I'm probably overthinking it. If it's in tonyb's homedir it's probably fine | 20:55 |
| fungi | yeah, i mean the jammy image rackspace provides doesn't seem to need configdrive, while the noble image we uploaded i guess does. probably comes down to the presence of some additional package/agent | 21:14 |
| clarkb | yes I think they run the nova agent stuff in their images maybe? Or maybe its a hacked up cloud-init that understands their metadata service? | 21:16 |
| clarkb | I don't think config drive is a problem for us. But if anyone can think of a reason it is now is a good time to say something | 21:16 |
| clarkb | (I think the primary downside to config drive is you can't update metadata after the server is booted but we never do that) | 21:16 |
| fungi | it should be fine in my opinion, yes | 21:17 |
| clarkb | I can't remember if there were problems iwth live migration too maybe? | 21:19 |
| clarkb | I want to say those were treated as bugs if they existed though | 21:19 |
| fungi | and yes, if i add --config-drive to the launch-node command line, it does bring up that noble image successfully | 21:20 |
| clarkb | ok any concerns with using an image of unconfirmed origin from tonyb's homedir? | 21:22 |
| clarkb | the timestamps all align with when he was uploading images in july so I'm like 98% certain this is that image just in vhd format and should be fine to use | 21:22 |
| fungi | doesn't worry me, no. if someone is able to sneak a compromised image onto our bastion host, we've got bigger issues ;) | 21:25 |
| fungi | for the record, this is what i ran successfully (completed just now): | 21:26 |
| fungi | sudo /usr/launcher-venv/bin/launch-node --cloud=openstackci-rax --region=DFW --flavor='4 GB Performance' --image='Ubuntu Noble OpenDev 20250110' --config-drive fungitest01.opendev.org | 21:26 |
| clarkb | excellent we should be able to boot up a paste server that way monday then | 21:29 |
| fungi | agreed | 21:29 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!