| opendevreview | Merged openstack/nova stable/victoria: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806626 | 00:21 |
|---|---|---|
| opendevreview | melanie witt proposed openstack/nova master: Add stub unified limits driver https://review.opendev.org/c/openstack/nova/+/712137 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Assert quota related API behavior when noop https://review.opendev.org/c/openstack/nova/+/712140 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Make unified limits APIs return reserved of 0 https://review.opendev.org/c/openstack/nova/+/712141 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Add logic to enforce local api and db limits https://review.opendev.org/c/openstack/nova/+/712139 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Enforce api and db limits https://review.opendev.org/c/openstack/nova/+/712142 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits https://review.opendev.org/c/openstack/nova/+/712143 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Update limit APIs https://review.opendev.org/c/openstack/nova/+/712707 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Update quota sets APIs https://review.opendev.org/c/openstack/nova/+/712749 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources https://review.opendev.org/c/openstack/nova/+/713301 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit https://review.opendev.org/c/openstack/nova/+/615180 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits https://review.opendev.org/c/openstack/nova/+/713498 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage https://review.opendev.org/c/openstack/nova/+/713499 | 03:40 |
| opendevreview | melanie witt proposed openstack/nova master: Add reno for unified limits https://review.opendev.org/c/openstack/nova/+/715271 | 03:40 |
| *** bhagyashris|off is now known as bhagyashris | 05:34 | |
| lpetrut | Hi, I have a question about keypairs. Nova allows a single keypair to be associated with a vm, yet in some cases we must inject multiple keys. We're using the k8s CAPO provider so we can't really use the userdata directly to inject additional keys. Now, apparently it's possible to bundle multiple ssh keys with the same keypair. Can we rely on this behavior to remain available? Fwiw, when bundling multiple keypairs, apparently each keypair must | 07:54 |
| lpetrut | have a comment, otherwise nova will fail to generate a fingerprint and reject it. | 07:54 |
| bauzas | good morning Nova | 07:58 |
| bauzas | for the first time during this week, I eventually have a bit time for going upstream... | 07:58 |
| gibi | bauzas: o/ could you please check the comments on the prelude | 07:59 |
| bauzas | lpetrut: the API doesn't look it supports more than one public key for a keypair | 07:59 |
| bauzas | lpetrut: https://docs.openstack.org/api-ref/compute/?expanded=create-or-import-keypair-detail | 07:59 |
| bauzas | gibi: sure, will look | 07:59 |
| gibi | thank you | 08:00 |
| bauzas | I also want to work for the vgpu documentation | 08:00 |
| gibi | also would be nice to land this doc https://review.opendev.org/c/openstack/nova/+/809161 and link it to the prelude | 08:00 |
| gibi | sure, if you push vgpu doc ping me and I will prioritize it | 08:00 |
| bauzas | gibi: ack, will look at it today | 08:01 |
| lpetrut | bauzas: we're passing multiple ssh keys separated by newline. apparently other people rely on it as well: https://help.switch.ch/engines/faq/how-to-use-multiple-ssh-keys/ | 08:01 |
| lpetrut | it's an ugly workaround, but it would be nice if we could continue to allow it until nova gets to support associating multiple keypairs | 08:11 |
| bauzas | lpetrut: well, | 08:13 |
| bauzas | if this works, fine | 08:13 |
| bauzas | but we can't say we would continue to support, given our API doesn't say this | 08:13 |
| lpetrut | makes sense | 08:13 |
| bauzas | but we could create a API microversion for supporting multiple public keys per keypair | 08:14 |
| bauzas | that said, I'm not sure what could be stopping to have public keys | 08:15 |
| bauzas | given we ask for a string | 08:15 |
| bauzas | unless we verify this string | 08:15 |
| lpetrut | there's some validation going on when the fingerprint gets generated | 08:16 |
| lpetrut | but if the first key has a comment, the rest of the payload seems to be treated as a comment and ignored | 08:16 |
| bauzas | hah | 08:23 |
| bauzas | I see | 08:23 |
| bauzas | well, unless the input validation changes... | 08:23 |
| bauzas | lpetrut: https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/keypairs.py#L47 | 08:25 |
| bauzas | well, this is treated from the API as a full string | 08:27 |
| bauzas | so... | 08:27 |
| bauzas | don't be that afraid | 08:27 |
| bauzas | lpetrut: anyway, changing this contract would require a microversion so in case you think you're trampled, you could use an old microversion for importing your multiple-pub keypair | 08:28 |
| lpetrut | great, thanks. just wanted to be sure that others are aware of this situation as well, hopefully we'll be able to improve the API eventually. | 08:28 |
| bauzas | lpetrut: well, an opensource project can't be "aware" of how people use it | 08:28 |
| lpetrut | about the api change, wondering which should be the best option: multiple keys bundled by a single keypair, or multiple keypairs associated with a single vm | 08:28 |
| bauzas | we try to remember exotic usages, but for best effort, we always say that things that aren't tested in CI are unsupported | 08:29 |
| lpetrut | yep, definitely | 08:29 |
| bauzas | as we could break things | 08:29 |
| bauzas | lpetrut: good question about the draft, I'd say this would be discussed in a spec | 08:30 |
| lpetrut | this might require some cloud-init changes as well | 08:30 |
| bauzas | gibi: i'm tempted to rebase the prelude above lyarwood's doc change, thoughts on it ? | 09:16 |
| gibi | bauzas: works for me | 09:16 |
| bauzas | ok, working on it | 09:16 |
| bauzas | anway, needs to provide a new rev for the prelude | 09:17 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: Add the Xena prelude section https://review.opendev.org/c/openstack/nova/+/807786 | 09:25 |
| opendevreview | Merged openstack/nova master: docs: Add nova-volume volume_attachment refresh admin workflow https://review.opendev.org/c/openstack/nova/+/809161 | 09:34 |
| bauzas | woah, the gate is quiet for a RC1 day | 09:43 |
| bauzas | gibi: working now on sean-k-mooney's doc change https://review.opendev.org/c/openstack/nova/+/806412 | 09:45 |
| bauzas | we could merge it soon | 09:45 |
| bauzas | and before the prelude so we could add it in the prelude | 09:45 |
| sean-k-mooney | am i dont know if we need to mention it in the prelude | 10:07 |
| sean-k-mooney | bauzas: have we not already mentioned the mdevs | 10:07 |
| bauzas | sean-k-mooney: yes we told about them in the prelude | 10:08 |
| bauzas | https://review.opendev.org/c/openstack/nova/+/807786 | 10:08 |
| sean-k-mooney | bauzas: so we can proceed with the docs change but i dont think we need to update the prelude for it | 10:10 |
| bauzas | sean-k-mooney: do you want to work on the doc change or do you let me fixing the nits ? | 10:10 |
| sean-k-mooney | ill leave it to you | 10:11 |
| bauzas | ok, will work on it later after lunch | 10:20 |
| bauzas | our lovely customer leaves us quiet for the moment :) | 10:20 |
| sean-k-mooney | oh dont jinx us like that | 10:20 |
| kashyap | Heh | 10:25 |
| opendevreview | OpenStack Release Bot proposed openstack/placement stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/placement/+/809363 | 10:36 |
| opendevreview | OpenStack Release Bot proposed openstack/placement stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/placement/+/809364 | 10:36 |
| opendevreview | OpenStack Release Bot proposed openstack/placement master: Update master for stable/xena https://review.opendev.org/c/openstack/placement/+/809365 | 10:36 |
| opendevreview | OpenStack Release Bot proposed openstack/placement master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/placement/+/809366 | 10:36 |
| *** slaweq__ is now known as slaweq | 13:23 | |
| *** abhishekk is now known as akekane|home | 13:24 | |
| *** akekane|home is now known as abhishekk | 13:25 | |
| opendevreview | Artom Lifshitz proposed openstack/nova stable/victoria: Update SRIOV port pci_slot when unshelving https://review.opendev.org/c/openstack/nova/+/796909 | 13:52 |
| melwitt | gibi: apologies if I missed it but I was wondering what are we doing about placement release note prelude and release? I thought about it since we have a couple of new features this time | 15:05 |
| gibi | melwitt: in the last couple of release we had no placemen release prelude written | 15:06 |
| melwitt | ack | 15:06 |
| gibi | interrestingly https://docs.openstack.org/releasenotes/placement/unreleased.html this is open | 15:06 |
| gibi | s/open/empty/ | 15:06 |
| gibi | that feels bad | 15:06 |
| melwitt | hm... that's weird, I thought we had added renos | 15:07 |
| gibi | yeah I do remember we added | 15:07 |
| gibi | running tox locally now | 15:07 |
| melwitt | yeah, just checked and consumer types definitely had a reno | 15:08 |
| gibi | hm, locally I get a proper releasenotes generated | 15:09 |
| gibi | we have renos for 1.37 and 1.38 | 15:09 |
| gibi | that is the two feature we added in Xena | 15:10 |
| melwitt | hm.. ok I'll try to figure out what's wrong with it. I don't remember off the top of my head how the doc publishing works but I think I could find it | 15:10 |
| gibi | thanks | 15:11 |
| sean-k-mooney | that i think is based on tags | 15:11 |
| sean-k-mooney | or something in the comit | 15:12 |
| sean-k-mooney | that marks the start/end of a releas | 15:12 |
| melwitt | well they are supposed to show up in the "unreleased" area soon after merge | 15:13 |
| melwitt | the creation of the release page like "Xena" is a different thing | 15:13 |
| sean-k-mooney | yes but they might be showing up in a xena section | 15:13 |
| melwitt | there is no xena section. the notes are nowhere | 15:14 |
| bauzas | placement should have an unreleased.rst file | 15:14 |
| bauzas | if not, it's something we could do | 15:14 |
| melwitt | I get why we don't have a xena section, we probably didn't do a release yet. but the notes should be in the unreleased area and there's nothing there | 15:15 |
| gibi | maybe we need this patch https://review.opendev.org/c/openstack/placement/+/809365 | 15:15 |
| bauzas | gibi: no this is the xena stable one | 15:15 |
| bauzas | once we branch with RC1 | 15:15 |
| bauzas | oh wait | 15:15 |
| bauzas | we did merge RC1 | 15:15 |
| melwitt | yeah, I'm saying that normally notes go live on the doc page in "unreleased" soon after merge | 15:15 |
| bauzas | hence the fact you no longer see them | 15:15 |
| gibi | yepp placement has RC1 I think | 15:15 |
| bauzas | gibi: I approved it sooner | 15:16 |
| bauzas | hence why they disappeared | 15:16 |
| bauzas | okay, +2ing the placement stable/xena reno one | 15:16 |
| melwitt | oh.. ok | 15:16 |
| melwitt | thanks for explaining that bauzas | 15:16 |
| bauzas | melwitt: sorry, my fault | 15:16 |
| bauzas | I approved RC1 cut | 15:16 |
| bauzas | but I should have thought about the reno patches | 15:17 |
| bauzas | ideally the release team could make them dependent | 15:17 |
| bauzas | between the release proposal and the reno update | 15:17 |
| gibi | bauzas: this is not a fault. first we need to approve the release then the tooling proposes the stable branch creation and then the stable branch setup patches | 15:17 |
| bauzas | that also explains why gibi was getting the notes | 15:18 |
| bauzas | he didn't cut yet I guess :p | 15:18 |
| gibi | I did not pull from remote | 15:18 |
| gibi | so yes | 15:18 |
| bauzas | et voila | 15:18 |
| bauzas | even, et voilà | 15:18 |
| bauzas | (with an accent) | 15:18 |
| melwitt | fahncy | 15:18 |
| bauzas | gibi: thanks for explaining | 15:19 |
| bauzas | so the patches only appear once we branch ? | 15:19 |
| gibi | I hope I'm correct :D | 15:19 |
| gibi | bauzas: it cannot appeare before as there is no stable/xena to propose them against | 15:19 |
| bauzas | that's an explanation :D | 15:20 |
| * bauzas facepalms | 15:20 | |
| * bauzas rolls eyes at the guy who was release liaison since Liberty | 15:20 | |
| bauzas | that now said, gibi, I didn't had time to work on sean-k-mooney's doc change | 15:21 |
| bauzas | :/ | 15:21 |
| bauzas | I guess we should approve the prelude and cut RC1 | 15:21 |
| gibi | bauzas: then we go with the current reno | 15:21 |
| bauzas | yeah | 15:21 |
| gibi | then doc can be backported | 15:21 |
| bauzas | we could update the prelude later | 15:21 |
| bauzas | that works | 15:22 |
| gibi | ok | 15:22 |
| bauzas | it's just, we need a prelude section before RC1 in order to see it for 24.0.0 | 15:22 |
| bauzas | or the prelude would only appear in 24.0.1 | 15:22 |
| gibi | I see | 15:22 |
| gibi | cool | 15:22 |
| bauzas | but once we merge stuff for 24.0.0, touching the prelude would also update the 24.0.0 (if I'm not wrong) | 15:22 |
| bauzas | that's one of the reno odds | 15:23 |
| bauzas | dansmith has a point, revising the prelude | 15:24 |
| gibi | ok, then merge the prelude | 15:24 |
| bauzas | gibi: hold your approval | 15:24 |
| gibi | ohh, sure | 15:24 |
| bauzas | I mean, don't explicitely approve :p | 15:24 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: Add the Xena prelude section https://review.opendev.org/c/openstack/nova/+/807786 | 15:25 |
| bauzas | gibi: dansmith: melwitt: any other around ^ | 15:26 |
| dansmith | way ahead of you | 15:26 |
| gibi | done | 15:26 |
| bauzas | dansmith: man, you're way too much productive compared to me | 15:26 |
| dansmith | I was just waiting for it after the mention above ;) | 15:27 |
| melwitt | seeya soon, highlights o/ | 15:27 |
| bauzas | ++ | 15:27 |
| dansmith | oh man, an early morning melwitt spotting! | 15:27 |
| dansmith | well, "early" | 15:27 |
| melwitt | uh huh | 15:28 |
| dansmith | :P | 15:29 |
| fungi | elodilles: i noticed yesterday you were pushing forward on some nova stable branch changes (thanks!) and wanted to mention that the vmt is waiting for the rest of https://review.opendev.org/q/I95f68be76330ff09e5eabb5ef8dd9a18f5547866 to merge so we can publish errata for ossa-2021-002 | 16:05 |
| fungi | sean-k-mooney: on that note, the failures on 806629 imply that fix may not be backportable as-is to train? | 16:06 |
| fungi | looks like it could be as simple as a couple of missing imports though | 16:07 |
| melwitt | fungi: hm, looks like the original fix didn't merge yet on train as well | 16:10 |
| fungi | melwitt: oh, interesting yeah | 16:10 |
| fungi | that being 791807 | 16:11 |
| melwitt | fungi: I'll help review the ones that I didn't propose, I didn't realize we needed to wait for ussuri and train, I had thought the original fix notice went out after the victoria change merged | 16:11 |
| melwitt | yeah | 16:12 |
| fungi | melwitt: we need stable/ussuri merged first ideally in this case, since it's still in a maintained state (projected transition to extended maintenance is november) | 16:13 |
| melwitt | fungi: ack, will prioritize | 16:13 |
| fungi | i didn't realize the original train fix never got approved either | 16:13 |
| fungi | but you're right that's less important, i can in theory link to the unmerged patch for train though i'm hesitant to do so unless it's actually passing ci jobs | 16:14 |
| fungi | as i could be inadvertently encouraging someone to merge a change which just causes nova to start crashing | 16:15 |
| fungi | er, to import a change into their deployment i mean | 16:15 |
| opendevreview | Elod Illes proposed openstack/nova stable/ussuri: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806628 | 16:17 |
| melwitt | fungi: yeah makes sense. the train change (the original) is a combo of two patches squashed together bc a test change was needed in order for the test to run on python < 3.6. and the followup should be rebased over it | 16:17 |
| sean-k-mooney | so presumable if we rebase the train one on top of the first patch that might fix it? | 16:18 |
| fungi | aha, that 'splains the errors. i can push that | 16:18 |
| fungi | unless someone else is already on it | 16:18 |
| fungi | ahh, though there are merge conflicts between them as well, so i'll defer to someone familiar with doing nova backports | 16:19 |
| fungi | i don't want to get the conflicts stuff in the commit message wrong | 16:19 |
| sean-k-mooney | im on a call but if melwitt does not get to it i can try and stack them | 16:19 |
| fungi | thanks! | 16:19 |
| elodilles | fungi melwitt : I've updated the commit message of the ussuri patch ^^^ | 16:20 |
| melwitt | thanks elodilles | 16:20 |
| fungi | thanks elodilles!!! | 16:20 |
| elodilles | so that it's nice and clean (& ready to merge) | 16:21 |
| fungi | trying to make sure this ossa update doesn't fall through the cracks | 16:21 |
| melwitt | ++ thanks fungi | 16:23 |
| melwitt | sean-k-mooney: I've got it | 16:32 |
| opendevreview | Merged openstack/placement master: Update master for stable/xena https://review.opendev.org/c/openstack/placement/+/809365 | 16:32 |
| opendevreview | melanie witt proposed openstack/nova stable/train: address open redirect with 3 forward slashes https://review.opendev.org/c/openstack/nova/+/806629 | 16:34 |
| clarkb | Hello, I've got a cloud (the inmotion cloud you might see some of your CI jobs running on) that nodepool is failing to boot instances on. | 16:34 |
| clarkb | the compute log says things like Instance f98ce366-90b1-43ba-8513-bf2ea559c931 has allocations against this compute host but is not found in the database. | 16:35 |
| clarkb | and the conductor log says things like nova.exception_Remote.NoValidHost_Remote: No valid host was found. | 16:35 |
| clarkb | `openstackclient limits show --absolute` shows that we aren't using any quota. | 16:35 |
| clarkb | But it appears that nova is essentially saying the compute hosts are full up and it can't schedule more work? | 16:36 |
| melwitt | clarkb: orphaned allocations in placement :( are making the hosts appear to be using resources when they shouldn't be | 16:36 |
| clarkb | Is there a way to list those without digging into a database? osc server list is empty too | 16:37 |
| melwitt | this is the doc for dealing with that https://docs.openstack.org/nova/latest/admin/troubleshooting/orphaned-allocations.html probably just need to run 'nova-manage placement heal_allocations' tool | 16:37 |
| clarkb | thank you | 16:37 |
| fungi | remember to lay your hands on the server when you chant "heal_allocations" too | 16:39 |
| fungi | it doesn't really make the command work any better, but it looks cool if anyone happens to be walking by | 16:40 |
| melwitt | 😂 | 16:40 |
| melwitt | this is just another reminder to prioritize the healing service (that would run heal_allocations periodically on its own, among other things) that's been in the backlog that I haven't had time to work on | 16:42 |
| opendevreview | Merged openstack/nova master: Add the Xena prelude section https://review.opendev.org/c/openstack/nova/+/807786 | 16:58 |
| melwitt | elodilles: oh no, l-c job failure again https://review.opendev.org/c/openstack/nova/+/806628 | 17:40 |
| sean-k-mooney | The user requested decorator>=3.4.0 | 17:47 |
| sean-k-mooney | The user requested (constraint) decorator==3.4.0 | 17:47 |
| sean-k-mooney | .... | 17:47 |
| sean-k-mooney | that does not feel like it should be a conflict | 17:47 |
| sean-k-mooney | since 3.4.0==3.4.0 is true and 3.4.0>=3.4.0 is also true | 17:48 |
| melwitt | yeah | 17:48 |
| melwitt | there's also this error "error in decorator setup command: use_2to3 is invalid." | 17:48 |
| clarkb | I think fungi said those issues may come up when pypi serves us stale indexes | 17:48 |
| melwitt | I don't know what that means | 17:48 |
| clarkb | the use_2to3 error is due to a new setuptools I think they removed that flag | 17:48 |
| sean-k-mooney | oh error in decorator setup command: use_2to3 is invalid. | 17:48 |
| sean-k-mooney | so this is not deps related | 17:49 |
| sean-k-mooney | ya | 17:49 |
| melwitt | https://zuul.opendev.org/t/openstack/build/4290861d5a464d099ad38165c99b647d/log/job-output.txt#792 | 17:49 |
| sean-k-mooney | the deps are fine but its unhappy with setuptools | 17:49 |
| clarkb | basically python software with modern pypa tools are expected to be python3 and not converted | 17:49 |
| sean-k-mooney | clarkb: is there a compat flag we can enable | 17:50 |
| sean-k-mooney | or jus tmove this to python 3 | 17:50 |
| melwitt | aside: the jump to link doesn't seem to be working for me lately when I link to a line number in a zuul output | 17:50 |
| sean-k-mooney | ya that only worked for me if the file is small | 17:51 |
| sean-k-mooney | it highlights it | 17:51 |
| sean-k-mooney | but does not move to it if its not loaded quick enough | 17:51 |
| melwitt | yeah. hrm. I wonder if something changed. the above ^ link doesn't jump me to the line and it loads really fast | 17:51 |
| clarkb | sean-k-mooney: I think pypa isn't interested in having compat flags. Updating or replacing deps is probably necessary | 17:52 |
| sean-k-mooney | melwitt: if i open it https://zuul.opendev.org/t/openstack/build/4290861d5a464d099ad38165c99b647d/log/job-output.txt#792 then change it to https://zuul.opendev.org/t/openstack/build/4290861d5a464d099ad38165c99b647d/log/job-output.txt#791 | 17:52 |
| sean-k-mooney | it works fine | 17:52 |
| sean-k-mooney | clarkb: ya or droping lower constraints. i just checked and its on py3 | 17:53 |
| sean-k-mooney | we have https://github.com/openstack/nova/blob/stable/train/tox.ini#L10 | 17:54 |
| sean-k-mooney | and we do not override it for lower constraints | 17:54 |
| sean-k-mooney | clarkb: the other option we have woudl be to downgrade setuptools | 17:55 |
| sean-k-mooney | pin it in the tox env to one that works | 17:55 |
| sean-k-mooney | perhaps usign requires https://tox.readthedocs.io/en/latest/config.html#conf-requires | 17:55 |
| sean-k-mooney | that would be a stable only change i guess if we did that. | 17:56 |
| sean-k-mooney | clarkb: do you know if this is affecting anyone else | 17:56 |
| sean-k-mooney | i know a lot of project just deleted there old lower constriants jobs | 17:57 |
| clarkb | the governance repo had problems with pydot2 which hasn't been maintained for years | 17:58 |
| clarkb | I don't think we want ot downgrade setuptools if we can avoid it. We can't really control what version of setuptools others use effectively and wide compatibility is desireable | 17:59 |
| clarkb | (note with the whole pyproject.toml stuff you do get a bit more control but openstack hasn'tdone any of that) | 17:59 |
| sean-k-mooney | ya its more we have 3 options delete the job, update the dep or hack around to make it work | 18:00 |
| sean-k-mooney | pinnign setup tools on an em branch is just that | 18:00 |
| clarkb | can you bump the dep version up such that it works? | 18:00 |
| sean-k-mooney | a hackaround to make it work | 18:00 |
| clarkb | that is all lower constraints is supposed to track iirc. The oldest version that works | 18:00 |
| sean-k-mooney | clarkb: well we are technialy not allow to bump min verison on stable right | 18:01 |
| sean-k-mooney | but pratically speaking yes we could | 18:01 |
| sean-k-mooney | and have to keeep it working already | 18:01 |
| clarkb | ya for stable maybe removing the job entirely makes sense | 18:02 |
| fungi | note the most recent setuptools release was a week ago, not sure if that lines up with when these failures started | 18:02 |
| fungi | however tox updated today | 18:03 |
| fungi | so it may have started vendoring a newer setuptools | 18:03 |
| fungi | this is one of the reasons "pinning" setuptools is complicated | 18:03 |
| sean-k-mooney | ya | 18:05 |
| sean-k-mooney | looking at the releases https://pypi.org/project/decorator/#history | 18:05 |
| sean-k-mooney | we are using 3.4.0 now | 18:05 |
| sean-k-mooney | from october 2012 | 18:05 |
| fungi | we, actually it's virtualenv which vendors setuptools, and that was also released today | 18:05 |
| sean-k-mooney | the next release 3.4.1 is 2015 | 18:05 |
| fungi | v20.8.0 (2021-09-16): upgrade embedded setuptools to 58.0.4 from 57.4.0 and pip to 21.2.4 from 21.2.3 | 18:06 |
| fungi | so my guess is this is setuptools 58.x.x behavior brought in by tox 20.8 | 18:06 |
| fungi | you could try manually downgrading setuptools<58 in the created env to find out, but i don't recommend that as a fix | 18:07 |
| sean-k-mooney | ya im just trying to repodcue it localy now | 18:09 |
| fungi | v58.0.0 Breaking Changes: Removed support for 2to3 during builds. Projects should port to a unified codebase or pin to an older version of Setuptools using PEP 518 build-requires. | 18:09 |
| sean-k-mooney | GLOB sdist-make: /home/sean/repos/nova/setup.py | 18:09 |
| sean-k-mooney | that new to me ^ | 18:09 |
| sean-k-mooney | i have not seen a GLOB sdist-make line before | 18:09 |
| sean-k-mooney | The conflict is caused by: | 18:10 |
| sean-k-mooney | jinja2 2.10 depends on MarkupSafe>=0.23 | 18:10 |
| sean-k-mooney | The user requested (constraint) markupsafe==1.0 | 18:10 |
| sean-k-mooney | so we have other conflict too aparently | 18:10 |
| fungi | and earlier i said tox 20.8 but meant virtualenv 20.8 | 18:10 |
| sean-k-mooney | ImportError: cannot import name 'Feature' from 'setuptools' | 18:11 |
| fungi | sean-k-mooney: yes, this is one of the reasons i contended calculating and maintaining a lower constraints list would be unworkable long term | 18:11 |
| sean-k-mooney | the import error was from markupsafe with tox 3.20.1 | 18:12 |
| sean-k-mooney | althoguh ok i ghet the same on ewith latest tox | 18:12 |
| sean-k-mooney | 3.24.4 | 18:12 |
| fungi | yeah, downgrading tox probably won't help. you more likely need to downgrade setuptools, maybe virtualenv, and possibly pip | 18:13 |
| clarkb | and rebuild the venv | 18:13 |
| sean-k-mooney | yeah am elodilles melwitt how would you feel about makeing it non voting of killing it on stable/train | 18:14 |
| sean-k-mooney | clarkb: yep used -r to rebuild the venv | 18:14 |
| fungi | right. the chain of problems is that these days tox *always* wants to use the latest available version of virtualenv, virtualenv regularly vendors in its own copy of latest setuptools, and setuptools occasionally drops features used by very old packages | 18:14 |
| sean-k-mooney | i could try messing with the other packages but realisticlly we agree that pinning is not what we want to do | 18:14 |
| melwitt | sean-k-mooney: this sounds like a case for removing the l-c job. also I saw this on stable/ussuri (I guess that means it's also happening on stable/train?) | 18:15 |
| sean-k-mooney | oh i assume train train is broke with a diffent error | 18:15 |
| fungi | almost certainly | 18:15 |
| sean-k-mooney | but still broken | 18:15 |
| dansmith | I'm just jumping in here, but yes, whatever $reason is a good case to drop the l-c job :P | 18:15 |
| fungi | er, train almost certainly would suffer from this if it also has a l-c job i mean | 18:15 |
| sean-k-mooney | fungi: https://github.com/openstack/nova/blob/stable/train/.zuul.yaml#L375 | 18:16 |
| sean-k-mooney | so yes | 18:16 |
| sean-k-mooney | we still have the template | 18:17 |
| melwitt | tomorrow we can ask elodilles what he thinks. I think he's likely done for today | 18:17 |
| sean-k-mooney | we fixed it the last time https://github.com/openstack/nova/commit/b2037fc4e356b55949339a1358c16431a9ab8930 | 18:17 |
| fungi | right, i personally never had high hopes for the l-c experiment (mainly because of the forward flow of time and packages from a decade ago not being able to reasonably predict modern systems), but especially not for stable branches. i would cut my losses on l-c if it were me, but it's not my call | 18:17 |
| dansmith | we had some resistance to removing it in glance, | 18:17 |
| dansmith | but then we got to a point where we couldn't add a new dep version or pip would run off into lala land, which helped seal the deal :) | 18:18 |
| sean-k-mooney | it would resolve the disccion im haveing with stephenfin too | 18:18 |
| sean-k-mooney | where he wanted to have it pinend to the oldest python we support for a release | 18:18 |
| sean-k-mooney | which i disagreed with | 18:18 |
| sean-k-mooney | since that was not realistic to how distors worked | 18:19 |
| fungi | well, basically if you head down that road, using old dependencies is not isolated to the python deps you have, those don't exist in a vaccuum and they make assumptions about contemporary systems as a whole | 18:19 |
| sean-k-mooney | speaking of being done for today i likely should finish up soon | 18:20 |
| fungi | i agree it's not a very good way to try to mimic what stable distros experience, because 1. they pin a lot more about the entire environment than we reasonably can, and 2. they backport fixes to things which wouldn't be reflected with this test methodology anyway | 18:20 |
| sean-k-mooney | since this is now efffectily blocking the backport of a cve fix however i think we likely should move forward with droping it so we can do the release | 18:21 |
| fungi | another alternative is early transition out of maintenance mode for stable/ussuri, but that seems like the most drastic choice | 18:21 |
| sean-k-mooney | fungi: well it was more the version of the packate and the version of python are often uncoupled | 18:22 |
| fungi | yes, that too | 18:22 |
| fungi | anyway, lower constraints jobs are not mandated by the pti, so dropping them doesn't affect the maintenance state for the branch | 18:22 |
| fungi | at least not in any official sense | 18:22 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [PoC][yoga] Off-path Networking Backends Support https://review.opendev.org/c/openstack/nova/+/808199 | 19:08 |
| gibi | bauzas: I've update the nova rc1 patch with the prelude hash https://review.opendev.org/c/openstack/releases/+/808706 I think it is good to go now | 19:48 |
| admin1 | hi @all .. how does a nova snapshot work when both glance and nova are backed by ceph ? does it still create a local disk on the hypervisor and uploads ? | 20:34 |
| admin1 | or does everything happen entirely on the ceph side ? | 20:35 |
| admin1 | my snapshot is kind of stuck .. even for a small instance .. and i don't see anything in the hypervisor ( files, or uploads or disk waiting ) so i think its entirely on the ceph side .. just wanted to validate | 20:39 |
| dansmith | admin1: if you have them share a pool, nova just asks ceph to do the snapshot in-place and then nova "tells" glance about it.. no bits fly around like normal | 20:52 |
| dansmith | ("them" meaning glance and nova) | 20:52 |
| opendevreview | melanie witt proposed openstack/nova master: Add section for 'nova-manage placement audit' tool https://review.opendev.org/c/openstack/nova/+/809479 | 20:56 |
| admin1 | nova is on vms pool and glance is on images pool .. so 2 diff pools | 20:56 |
| dansmith | admin1: okay so I think that means it's going to download and then re-upload it, but not totally positive | 20:58 |
| * dansmith has to duck out for a bit | 20:58 | |
| melwitt | admin1: if you have configured your deployment like this https://docs.ceph.com/en/mimic/rbd/rbd-openstack/#configure-openstack-to-use-ceph it should be able to do the fast snapshot | 20:59 |
| admin1 | melwitt , refering to enabling copy-on-write cloning of images ? | 21:00 |
| melwitt | yes | 21:00 |
| admin1 | its not enabled .. | 21:01 |
| admin1 | what does exactly "Note that this exposes the back end location via Glance’s API, so the endpoint with this option enabled should not be publicly accessible." mean ? | 21:01 |
| admin1 | i will try to find more | 21:01 |
| admin1 | melwitt, in absense of this setting, how does the nova snapshot work ? | 21:02 |
| melwitt | another thing that can cause it to not do the CoW is if the image is not in RAW format, but I think glance might automatically convert to RAW now, depending on what release you have | 21:02 |
| admin1 | all images are in raw format | 21:02 |
| admin1 | i converted them before uploading to glance | 21:02 |
| melwitt | that is a warning that the backend url will be shown to users querying glance API so it should not be a url that is publicly accessible (for security reason) | 21:02 |
| melwitt | ok cool. if you don't set up as described in that ceph doc, nova will download the image from glance first and then boot the instance | 21:03 |
| melwitt | if the image is large, it can be slow | 21:04 |
| melwitt | there's some glance setting around chunk size which you might want to adjust to get better throughput, I think it's this? https://docs.openstack.org/glance/latest/configuration/glance_api.html#glance.store.rbd.store.rbd_store_chunk_size | 21:08 |
| melwitt | *settings | 21:09 |
| opendevreview | melanie witt proposed openstack/nova master: Add section for 'nova-manage placement audit' tool https://review.opendev.org/c/openstack/nova/+/809479 | 21:18 |
| admin1 | melwitt, when you meant backend url, is that the url of the ceph ? | 21:26 |
| admin1 | i mean its osd url | 21:27 |
| admin1 | melwitt, what glance command can i use to see the url being exposed .. | 21:29 |
| opendevreview | melanie witt proposed openstack/nova master: Add section for 'nova-manage placement audit' tool https://review.opendev.org/c/openstack/nova/+/809479 | 22:07 |
| melwitt | admin1: yeah ceph url I think. you can see it by doing an 'image show' if you have enabled showing it https://docs.openstack.org/api-ref/image/v2/index.html?expanded=show-image-detail#show-image | 22:19 |
| melwitt | https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/image-v2.html#image-show | 22:20 |
| dansmith | admin1: that location thing means glance will show things like rbd://mycephserver:port/path/to/thing, and so you likely want a different endpoint with that enabled that regular untrusted users can't see | 22:45 |
| dansmith | admin1: without the fast cow snapshot, nova will pause the VM, download its disk, flatten it out, upload it to glance, and then unpause the VM | 22:46 |
| dansmith | admin1: it's far more efficient to just let ceph snapshot it for you (pretty much instantaneously) and then just tell glance it exists | 22:46 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!