| *** martinkennelly has quit IRC | 00:09 | |
| *** whoami-rajat has quit IRC | 01:48 | |
| opendevreview | norman shen proposed openstack/nova master: Saving security group to info_cache https://review.opendev.org/c/openstack/nova/+/786348 | 02:53 |
|---|---|---|
| opendevreview | norman shen proposed openstack/nova master: Saving security group to info_cache https://review.opendev.org/c/openstack/nova/+/786348 | 03:31 |
| opendevreview | norman shen proposed openstack/nova master: Saving security group to info_cache https://review.opendev.org/c/openstack/nova/+/786348 | 04:09 |
| *** ricolin_ has joined #openstack-nova | 04:32 | |
| *** vishalmanchanda has joined #openstack-nova | 04:35 | |
| *** ricolin has quit IRC | 04:36 | |
| *** ricolin_ is now known as ricolin | 04:36 | |
| *** abhishekk has joined #openstack-nova | 04:42 | |
| *** hemanth_n has joined #openstack-nova | 05:07 | |
| *** ralonsoh has joined #openstack-nova | 05:41 | |
| *** LinPeiWen has joined #openstack-nova | 06:16 | |
| *** slaweq has joined #openstack-nova | 06:17 | |
| opendevreview | chengsheng proposed openstack/nova master: libvirt: Use improved guest CPU config APIs https://review.opendev.org/c/openstack/nova/+/762330 | 06:24 |
| *** alex_xu has joined #openstack-nova | 06:37 | |
| opendevreview | norman shen proposed openstack/nova master: Saving security group to info_cache https://review.opendev.org/c/openstack/nova/+/786348 | 06:52 |
| *** hemna has quit IRC | 07:01 | |
| *** hemna has joined #openstack-nova | 07:01 | |
| *** andrewbonney has joined #openstack-nova | 07:19 | |
| frickler | I'm seeing issues with the nova-metadata service on stein, does that ring a bell with someone? from the logs it always seems to start with an eventlet traceback, then some requests timeout. sometimes it recovers, sometimes thing break completely and need a restart of apache2 http://paste.openstack.org/show/sjn1zlp9BYRX4zNGckmg/ | 07:31 |
| *** rpittau|afk is now known as rpittau | 07:44 | |
| *** tosky has joined #openstack-nova | 07:48 | |
| *** tosky has quit IRC | 07:48 | |
| *** jrosser has joined #openstack-nova | 07:48 | |
| *** martinkennelly has joined #openstack-nova | 07:48 | |
| *** lucasagomes has joined #openstack-nova | 08:07 | |
| *** tosky has joined #openstack-nova | 08:08 | |
| *** derekh has joined #openstack-nova | 08:14 | |
| *** abhishekk has quit IRC | 08:35 | |
| *** abhishekk has joined #openstack-nova | 08:35 | |
| *** LinPeiWen has quit IRC | 08:36 | |
| *** stephenfin has quit IRC | 08:49 | |
| *** martinkennelly has quit IRC | 08:50 | |
| *** martinkennelly has joined #openstack-nova | 08:50 | |
| *** stephenfin has joined #openstack-nova | 08:54 | |
| *** abhishekk has quit IRC | 08:56 | |
| *** stephenfin has quit IRC | 08:58 | |
| *** stephenfin has joined #openstack-nova | 08:59 | |
| *** stephenfin has quit IRC | 08:59 | |
| *** stephenfin has joined #openstack-nova | 09:06 | |
| bauzas | frickler: sorry for the late bump, but eventlet and stein are in my mind | 09:10 |
| * bauzas tries to find some bugs we had | 09:10 | |
| bauzas | something like https://bugs.launchpad.net/nova/+bug/1825584 | 09:11 |
| opendevmeet | Launchpad bug 1825584 in OpenStack Compute (nova) stein "eventlet monkey-patching breaks AMQP heartbeat on uWSGI" [Low,In progress] - Assigned to sean mooney (sean-k-mooney) | 09:11 |
| bauzas | frickler: also, fwiw, we deprecated using eventlet for the API by Rocky | 09:15 |
| bauzas | https://review.opendev.org/c/openstack/nova/+/549510 | 09:15 |
| *** amotoki has joined #openstack-nova | 09:36 | |
| *** jangutter_ has joined #openstack-nova | 10:01 | |
| *** jangutter has quit IRC | 10:08 | |
| *** lyarwood has joined #openstack-nova | 10:10 | |
| frickler | bauzas: actually the service is running with wsgi in apache, the traceback is seen within /var/log/apache2/nova-metadata_error.log , I should have noted that | 10:11 |
| bauzas | so a greenthread issue, I think | 10:12 |
| frickler | ah, I just found https://review.opendev.org/c/openstack/nova/+/662095/5/releasenotes/notes/eventlet-monkey-patch-5f734ef581aa550e.yaml in the bug you linked, I need to check on those settings | 10:13 |
| frickler | bauzas: ha, I think that's the issue, we have "threads=1" for nova-api, but =10 for metadata. hopefully that should be the issue already | 10:15 |
| frickler | thx for the pointers | 10:15 |
| jkulik | on that topic^ how does one deal with longrunning rpc-calls done from the API and the number of processes in uWSGI? It should block a whole process right? Thus making a DoS possible by doing a lot of volume-attachments in parallel (which does an RPC which holds a lock on the Instance) | 10:16 |
| lyarwood | Only the initial part of the attachment is a call, the main part is a cast that should yield | 10:18 |
| lyarwood | https://docs.openstack.org/nova/latest/reference/attach-volume.html | 10:19 |
| lyarwood | I haven't had enough coffee to fully answer the impact of threads=1 on that flow however ;) | 10:19 |
| jkulik | if there's an attachment running already, this takes the Instance lock. then all the other calls from nova-api to nova-compute for reserving the block-device-mapping entry (says for k8s workloads attaching > 10 volumes) will wait on the lock | 10:20 |
| jkulik | so would it be recommended to run nova-api with a high number of threads to counter that? | 10:21 |
| lyarwood | right that instance lock will block the remaining attachments | 10:21 |
| lyarwood | that isn't going to help | 10:21 |
| lyarwood | so you can run additional nova-api processes to avoid the lock on that side | 10:21 |
| lyarwood | but nova-compute also takes an instance lock per attachment | 10:22 |
| lyarwood | and another connect_volume lock within os-brick per compute node | 10:22 |
| jkulik | yes, I'm not worried about nova-compute. I'm worried about some k8s projects starving out other users of nova-api | 10:22 |
| lyarwood | I'm not sure if the instance lock actually blocks anything else from being accepted tbh | 10:23 |
| lyarwood | but I get what you're saying | 10:23 |
| *** jangutter has joined #openstack-nova | 10:23 | |
| lyarwood | we've not had any reports of n-api rejecting requests with k8s calling into it tbh but it could be possible | 10:23 |
| jkulik | we're still on eventlet with a low number of process and trying to go to uWSGI, it looks like we have to run a huge number of processes to accommodate for that | 10:25 |
| jkulik | or play around with threads=100 or something | 10:25 |
| lyarwood | would you mind writing this up as a bug https://launchpad.net/nova/+bug ? | 10:26 |
| jkulik | if you think that helps, sure. I just wanted to get input on how to run Nova tbh. | 10:30 |
| *** jangutter_ has quit IRC | 10:30 | |
| lyarwood | It's mostly so others in NA can chime in later today with thoughts | 10:31 |
| jkulik | ok. I'll write the problem statement down | 10:32 |
| lyarwood | thanks | 10:34 |
| *** sean-k-mooney has joined #openstack-nova | 10:34 | |
| sean-k-mooney | i generally try to avoid theads like "X is old and broken and Y is new an shiny" but a recent internal one pointed me to https://github.com/tummychow/git-absorb | 10:41 |
| sean-k-mooney | which looks quite interesting | 10:41 |
| sean-k-mooney | has anyone used it? im not sure how well it would work in pratice | 10:43 |
| kashyap | sean-k-mooney: That project is missing a 1-line summary at the top; instead of a long elevator pitch | 10:43 |
| kashyap | It says it is a port of "Facebook's "hg absorb"; which I don't know what it is | 10:43 |
| sean-k-mooney | well its porting a feature of mecurial to git | 10:43 |
| kashyap | "a feature"? | 10:43 |
| sean-k-mooney | yes | 10:44 |
| kashyap | That's what I mean by a simple 1-line summary of the feature :) | 10:44 |
| gibi | sean-k-mooney: "will automatically identify which commits are safe to modify, and which staged changes belong to each of those commits" this sounds dangerous | 10:44 |
| bauzas | sean-k-mooney: well, I trust in Gerrit for this | 10:44 |
| kashyap | Yeah | 10:44 |
| sean-k-mooney | so what it does is allow you to write one commit at the end of a patch chain that fixes all the review feedback | 10:44 |
| bauzas | and I trust reviewers | 10:44 |
| sean-k-mooney | then it will automaticly decomose it into may commits and aplly them as fixup via an interactive rebase | 10:44 |
| kashyap | That sounds quite invasive to me | 10:44 |
| bauzas | if you create a fixup patch, then the CI wouldn't like it | 10:44 |
| *** abhishekk has joined #openstack-nova | 10:45 | |
| gibi | I mean sometimes I cannot figure out correctly what local change goes to which commit in my branch so I don't trust the machine to do that :D | 10:45 |
| sean-k-mooney | oh ya im not sure this is smart to use | 10:45 |
| sean-k-mooney | but it was an interesting idea | 10:45 |
| bauzas | I'd prefer to just continue to have a master branch that works | 10:45 |
| stephenfin | bauzas: I think you're misunderstanding what this does | 10:45 |
| sean-k-mooney | it basically takes each hunk in the current commit and trys to apply it to the oldest commit that does not cause a marge conflict | 10:46 |
| gibi | I have a nice long commit chain already for pps in nova so I can actually test its smartness | 10:46 |
| bauzas | stephenfin: explain me, then | 10:46 |
| stephenfin | you've got e.g. a 15 patch series and changes are required in a few of them | 10:46 |
| bauzas | yeah that's what i understood | 10:47 |
| stephenfin | in the GH pull request workflow, you'd typically fix them all in one go and commit a single "fix all the comments" patch | 10:47 |
| bauzas | yeah, hence my "I love Gerrit" for this | 10:47 |
| stephenfin | this will decompose that single patch and merge the things back into whatever one of the previous 15 patches they belong to | 10:47 |
| kashyap | stephenfin: I see the use-case here, though. But it requires great care when using it to quadruple-check things | 10:48 |
| kashyap | It sounds very nice on paper :) | 10:48 |
| stephenfin | yeah, it's much less useful for Gerrit where you're forced to think in terms of individual commits and use rebase extensively | 10:48 |
| sean-k-mooney | kashyap: yep that is why by default it create the delta as patch that you then can merge with an interactive rebase | 10:48 |
| lyarwood | yeah it's working around the bork'd nature of the PR workflow | 10:48 |
| sean-k-mooney | yep | 10:48 |
| bauzas | stephenfin: okay, then why it would be nice for our, then ? | 10:48 |
| stephenfin | it wouldn't really, but that doesn't make it a bad tool | 10:49 |
| lyarwood | git stash ftw | 10:49 |
| bauzas | stephenfin: we can already provide a new patch per change | 10:49 |
| * gibi hates stash | 10:49 | |
| sean-k-mooney | yep it kidof reminded me of https://docs.openstack.org/infra/git-restack/ | 10:49 |
| stephenfin | and I don't see what impact this would have on a working master branch | 10:49 |
| sean-k-mooney | although a different approch | 10:49 |
| lyarwood | it's useful when you need to fix HEAD~$something up but at pointing at HEAD | 10:49 |
| lyarwood | but are* | 10:49 |
| stephenfin | lyarwood: yup, agree RE: borked PR workflow | 10:49 |
| bauzas | honestly, I don't see *why* I'd need this new tool | 10:50 |
| jkulik | in our GH workflow, we still rebase the changes into every commit even for multi-commit PRs. no tool necessary. git commit --fixup and git rebase -i | 10:50 |
| sean-k-mooney | bauzas: im not saying you do | 10:50 |
| sean-k-mooney | bauzas: just wondering if people had used it | 10:50 |
| bauzas | stephenfin: I don't like it because it means that it's OK to have a patch having bugs | 10:50 |
| bauzas | if the next patch fixes them | 10:50 |
| gibi | lyarwood: I tend to prepare commits top of HEAD and then do an interactive rebase to meld them into the proper origin commit they belong | 10:50 |
| stephenfin | jkulik: yup, which works for a change to a single patch. Trickier if you have changes that affect multiple patches | 10:50 |
| bauzas | of course, you *can* squash both | 10:50 |
| stephenfin | bauzas: sounds like your issue is with the PR workflow rather than this tool :) | 10:51 |
| bauzas | but heh, you can *not* squash, and that's why I dislike | 10:51 |
| lyarwood | gibi: yeah that's another way, I just find stash a little quicker for small things | 10:51 |
| sean-k-mooney | bauzas: that not what the tool is enabling at all though so that is kind of irrelevent | 10:51 |
| gibi | lyarwood: I guess instead of commits I could do stash | 10:51 |
| bauzas | stephenfin: correct | 10:51 |
| stephenfin | or rather the CI systems built on this workflow | 10:51 |
| sean-k-mooney | gibi: i dont like stash because i have lost work that way | 10:51 |
| bauzas | stephenfin: and that's why i said "I prefer the Gerrit workflow for this" | 10:51 |
| gibi | sean-k-mooney: ditto | 10:51 |
| stephenfin | Ah, okay. Given we were talking about the tool, I thought you were comparing the _tool_ to Gerrit | 10:52 |
| jkulik | lyarwood: here's the bug you requested https://bugs.launchpad.net/nova/+bug/1930406 | 10:52 |
| opendevmeet | Launchpad bug 1930406 in OpenStack Compute (nova) "parallel volume-attachment requests might starve out nova-api for others" [Undecided,New] | 10:52 |
| stephenfin | which doesn't really make sense | 10:52 |
| sean-k-mooney | i much prefer to either do an interactive rebase and fix inline or put patches on the end an move them | 10:52 |
| lyarwood | jkulik: thanks | 10:52 |
| bauzas | sean-k-mooney: gibi: well, I use git reflog in this case | 10:52 |
| gibi | for the git absorb thingy, if it can do a smart split of the local changes then it might help me with the commit creation what I do manually with git add -p | 10:52 |
| sean-k-mooney | bauzas: i dont think that works with stash | 10:52 |
| lyarwood | ah yes the get out of jail free card that is reflog | 10:52 |
| sean-k-mooney | if it does good to know | 10:52 |
| bauzas | sean-k-mooney: nah, I prefer to commit | 10:53 |
| lyarwood | yeah it doesn't with stash | 10:53 |
| sean-k-mooney | ya so do i so i can use reflog if i mess things up | 10:53 |
| stephenfin | trying to parse a reasonably complex reflog is *soo* much fun | 10:53 |
| bauzas | anyway, me needs to lunch | 10:53 |
| lyarwood | stephenfin: always helps when you're looking for something rather important that would take ages to rewrite ^_^ | 10:54 |
| gibi | :D | 10:54 |
| lyarwood | can't say I ever look at reflog when I'm relaxed | 10:54 |
| stephenfin | touché | 10:54 |
| sean-k-mooney | stephenfin: totes fun but 99% of the time when i need it i just need the sha that is a 2-3 lines form the top | 10:54 |
| gibi | have you ever git pulled one repo into another? that is fun to realize later on :D | 10:55 |
| sean-k-mooney | lyarwood: ya when i use it its oftehn to fix an unitential rebase with git reivew | 10:55 |
| sean-k-mooney | gibi: yep | 10:55 |
| sean-k-mooney | gibi: nova has a full copy of the openwrt sorce tree in it somewhere | 10:55 |
| gibi | at some point I had placement back in nova :D | 10:55 |
| sean-k-mooney | at least the gerrit version of it had at one point we may have eventually git gc that out of the public repos | 10:56 |
| gibi | ohh so the central copy of nova had openwrt? nice! | 10:57 |
| *** Luzi has joined #openstack-nova | 10:57 | |
| lyarwood | sounds like some kind of go project repo | 10:57 |
| sean-k-mooney | yep thats something jaypipes told me a long time ago. i assume someone pushed a review where tehy acindtally commited it locally | 10:58 |
| sean-k-mooney | but gerrit would keep that around forever as a result in the gerrit copy of the repo | 10:58 |
| sean-k-mooney | you would have to manually purge the review ref to remove it | 10:59 |
| opendevreview | Stephen Finucane proposed openstack/nova master: Use neutronclient's port binding APIs https://review.opendev.org/c/openstack/nova/+/706295 | 11:08 |
| opendevreview | Stephen Finucane proposed openstack/nova master: docs: Drop references to non-filter scheduler drivers https://review.opendev.org/c/openstack/nova/+/773645 | 11:13 |
| opendevreview | Stephen Finucane proposed openstack/nova master: scheduler: Merge driver into manager https://review.opendev.org/c/openstack/nova/+/773644 | 11:13 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Merge 'test_utils', 'test_scheduler_utils' https://review.opendev.org/c/openstack/nova/+/773646 | 11:13 |
| opendevreview | Stephen Finucane proposed openstack/nova master: conf: Remove deprecated aliases https://review.opendev.org/c/openstack/nova/+/773647 | 11:13 |
| opendevreview | norman shen proposed openstack/nova master: Saving security group to info_cache https://review.opendev.org/c/openstack/nova/+/786348 | 11:38 |
| stephenfin | dead simple Python 3.10 prep patch here if anyone has 2 mins https://review.opendev.org/c/openstack/nova/+/790405 | 11:42 |
| gibi | stephenfin: done | 11:45 |
| stephenfin | ta | 11:45 |
| *** whoami-rajat has joined #openstack-nova | 11:56 | |
| *** abhishekk has quit IRC | 12:04 | |
| *** alex_xu has quit IRC | 12:13 | |
| opendevreview | Lee Yarwood proposed openstack/nova stable/wallaby: hardware: Use image_meta.id within get_mem_encryption_constraint https://review.opendev.org/c/openstack/nova/+/793956 | 12:15 |
| opendevreview | Lee Yarwood proposed openstack/nova stable/victoria: hardware: Use image_meta.id within get_mem_encryption_constraint https://review.opendev.org/c/openstack/nova/+/793957 | 12:16 |
| opendevreview | Lee Yarwood proposed openstack/nova stable/ussuri: hardware: Use image_meta.id within get_mem_encryption_constraint https://review.opendev.org/c/openstack/nova/+/793958 | 12:17 |
| *** tbarron has quit IRC | 12:52 | |
| *** eharney has joined #openstack-nova | 12:54 | |
| *** rpittau is now known as rpittau|afk | 12:57 | |
| *** alex_xu has joined #openstack-nova | 13:00 | |
| *** hemanth_n has quit IRC | 13:00 | |
| *** tbarron has joined #openstack-nova | 13:07 | |
| ganso | Hi nova folks! Not sure if you're familiar with this behavior, whether it is a known bug or limitation (I didn't find any launchpad entry for it), but it is very easy to reproduce: if you try attach a volume with hw_disk_bus property that is different from the root disk's hw_disk_bus one, it gets ignored. Example, root disk's is ide, volume's is virtio, and it gets attached as ide. | 13:07 |
| ganso | using pure kvm through virt-manager, it is possible to have mixed ide+virtio disks, but nova doesn't seem to be allowing that | 13:08 |
| sean-k-mooney | ganso: yes that is expected behavior | 13:08 |
| sean-k-mooney | ganso: the image metadta is only use form the root disk | 13:08 |
| sean-k-mooney | we do not use image metadta form any other disk | 13:08 |
| sean-k-mooney | ganso: you might be able to enable this using the block device mappings api | 13:09 |
| ganso | sean-k-mooney: is there a reason why you would want to keep it this way? | 13:09 |
| sean-k-mooney | well if the bus is not already present in the vm at a minitum we woudl have to also attach a contole for that bus be it ide/stata whatever | 13:10 |
| sean-k-mooney | and then all attach the volume to that new contoler | 13:10 |
| sean-k-mooney | that could fail for a number of reasons | 13:10 |
| sean-k-mooney | in general image metadata was intened to only be used form teh root disk | 13:11 |
| sean-k-mooney | when not using cinder it does not really make sense to have multipel image metadta soruce | 13:11 |
| ganso | sean-k-mooney: hmmm I see, so this is like a new feature, to handle independent disk_bus values for each new volume, something nova doesn't do today | 13:12 |
| sean-k-mooney | if we allowed it to work for cinder there is an issue with confilt between volumens that we would have to determin how to handel | 13:12 |
| ganso | sean-k-mooney: I was able to hack the code just to test it, I replaced the variable with "virtio" here: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2018 | 13:12 |
| sean-k-mooney | ganso: yes it would basiclaly be a new feature | 13:13 |
| ganso | and it worked, upon attach it added a virtio disk beside an ide one (the root disk) | 13:13 |
| sean-k-mooney | yes there is a virtio contoler present by defualt for the nic | 13:13 |
| sean-k-mooney | if you put stata it will fail | 13:13 |
| sean-k-mooney | /stata/sata | 13:13 |
| *** rloo has joined #openstack-nova | 13:14 | |
| ganso | sean-k-mooney: so, adding a controller during attach is something risk you say? would it be 100% safe if it requires the VM to be shutoff? | 13:14 |
| sean-k-mooney | it may or may not fail | 13:15 |
| sean-k-mooney | and it may depend on the version fo libvirt if the contoler is hot pluggable or not | 13:15 |
| ganso | sean-k-mooney: I understand. I will do some testing around this, thank you for the clarification! | 13:16 |
| sean-k-mooney | for example when using q35 machine type libvirt only alloctes 1 spare pcie port for hotplug devices | 13:16 |
| sean-k-mooney | so you would not have enough to add the contoler and a volume | 13:16 |
| *** mdbooth has joined #openstack-nova | 13:16 | |
| sean-k-mooney | there is one way you might be able to do this today | 13:16 |
| sean-k-mooney | but only for new vms | 13:17 |
| sean-k-mooney | https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#create-server | 13:17 |
| sean-k-mooney | server create allows you to specify the block device mappings block_device_mapping_v2 (Optional) | 13:17 |
| sean-k-mooney | "block_device_mapping_v2": [{ | 13:17 |
| sean-k-mooney | "boot_index": "0", | 13:17 |
| sean-k-mooney | "uuid": "ac408821-c95a-448f-9292-73986c790911", | 13:17 |
| sean-k-mooney | "source_type": "image", | 13:17 |
| ganso | sean-k-mooney: oh I see, it would depend on that as well, so depending on how I test it, I may be avoiding a problem (like testing with just 1 extra disk instead of 2 or 3, as it could run out of devices) | 13:17 |
| sean-k-mooney | "volume_size": "25", | 13:17 |
| sean-k-mooney | "destination_type": "volume", | 13:17 |
| sean-k-mooney | "delete_on_termination": true, | 13:17 |
| sean-k-mooney | "tag": "disk1", | 13:17 |
| sean-k-mooney | "disk_bus": "scsi"}] | 13:17 |
| sean-k-mooney | which has the disk bus | 13:17 |
| sean-k-mooney | ganso: yes although we have a config option for how may free port to allcote | 13:18 |
| ganso | sean-k-mooney: I will try that! could just work out for my use case! thank you! | 13:18 |
| sean-k-mooney | ganso: if that works and you can set the disk bus per volume when creatign a new server | 13:18 |
| sean-k-mooney | which the api imples | 13:18 |
| sean-k-mooney | then we could consider if that would be extanded to port attach | 13:19 |
| sean-k-mooney | that coudl be done by either alowing the disk bus to be specified on attach or possibel adding support for a subset of data to be stored in teh block device mapping form the volume metadtata | 13:20 |
| sean-k-mooney | e.g. copy just the hw_disk_bus into the bdm that we create on attach | 13:20 |
| ganso | sean-k-mooney: well, even if that works for the "create" operation, it is a more limited one, as you said, when creating the VM it is a safer scenario (not hotpluggable, can determine the required controllers before-hand). The attach operation is less safe in this regard and could fail in many ways, as you said | 13:21 |
| sean-k-mooney | ganso: this is a feature that would require a spec and someone more expert in the strage layware like lyarwood to review | 13:21 |
| *** Luzi has quit IRC | 13:21 | |
| sean-k-mooney | ganso: the failure could actully be handeled internally in the driver just roleing back and rejecting the attach but we would have to code with that in mind and think about the ux and error message we give to a user | 13:22 |
| sean-k-mooney | whihc is one of the reason i think this woudl need a spec rather then trateing it as a bugfix | 13:23 |
| ganso | sean-k-mooney: yea, just like the hotplugged error message for IDE devices, I was confused at first when the attach didn't work while the request was accepted, but I had no obvious error message | 13:24 |
| sean-k-mooney | ganso: the driver code in general assumes there is only one value for hw_disk_bus for the entire vm so thre are proably edgcases like rescue and move operations that might break. | 13:24 |
| ganso | sean-k-mooney: I agree! +1 | 13:24 |
| *** bhagyashris has joined #openstack-nova | 13:29 | |
| ganso | sean-k-mooney: btw, if you have a minute, that patch you previously reviewed already has a +2. I was wondering if you could please take another look at it when you have a minute, since you're already familiar with it: https://review.opendev.org/c/openstack/nova/+/784166 | 13:30 |
| *** abhishekk has joined #openstack-nova | 13:35 | |
| sean-k-mooney | ah yes am i do not have +2 rights on nova but ill take a look now and maybe poke some of the nova cores to review when im done | 13:36 |
| opendevreview | Merged openstack/nova master: Add unit test for importing ed25519 ssh key https://review.opendev.org/c/openstack/nova/+/789642 | 13:49 |
| opendevreview | Merged openstack/nova master: Change minversion of tox to 3.18.0 https://review.opendev.org/c/openstack/nova/+/791968 | 13:50 |
| sean-k-mooney | bauzas: you like anit-afintiy groups right :P care to review ganso's patch https://review.opendev.org/c/openstack/nova/+/784166 | 13:51 |
| *** belmoreira has joined #openstack-nova | 13:51 | |
| opendevreview | Merged openstack/nova master: Remove unused DeleteFromSelect db api method https://review.opendev.org/c/openstack/nova/+/784528 | 13:51 |
| bauzas | sean-k-mooney: not sure it's correct to say "I like" for the instance groups but sure ;) | 13:52 |
| sean-k-mooney | hehe that was ment to be drowning in sarcasim but i for got to anotate that :) | 13:52 |
| *** hemna has quit IRC | 13:53 | |
| *** hemna has joined #openstack-nova | 13:54 | |
| sean-k-mooney | do we have quota for how may revision of a patch you are allow to have without unit tests passing... | 13:56 |
| sean-k-mooney | i fell like if we did its less then 51 in 6 weeks | 13:57 |
| bauzas | ganso: I provided another +2 but please look at my comment in https://review.opendev.org/c/openstack/nova/+/784166 | 14:01 |
| bauzas | once you look at it, tell me and then I'll +W your change | 14:01 |
| sean-k-mooney | bauzas: yes adding an explaintion would be nice to avoid the tribal knoladge that live migration never reschdules | 14:02 |
| bauzas | can be a follow-ip | 14:02 |
| bauzas | up* | 14:02 |
| sean-k-mooney | also yes i agree it can be a followup | 14:03 |
| bauzas | ganso: do you want to backport your change, actually ? | 14:03 |
| bauzas | if so, maybe we would need to have a new revision for the relnote | 14:03 |
| sean-k-mooney | we can backport the followup too | 14:03 |
| sean-k-mooney | just and fyi | 14:04 |
| bauzas | or squashing both when backporting but... | 14:04 |
| sean-k-mooney | reno is ok with that | 14:04 |
| sean-k-mooney | no need to squash | 14:04 |
| sean-k-mooney | we could but you can edit renos after the fact | 14:04 |
| bauzas | sure, but I don't like to have two patches when backporting when one of them is just a FUP | 14:04 |
| sean-k-mooney | fair | 14:04 |
| bauzas | either way, wanting ganso to reply... | 14:05 |
| ganso | reading scrollback now | 14:05 |
| ganso | sean-k-mooney: loved the term tribal knowledge! didnt know it! will now use it for many things lol! | 14:07 |
| ganso | bauzas: yes, I plan to backport. Will look at the patch now | 14:07 |
| bauzas | ganso: okay, then please provide a new PS and I'll directly +2/+W | 14:08 |
| opendevreview | Balazs Gibizer proposed openstack/placement master: [doc] Redirect people to #openstack-nova https://review.opendev.org/c/openstack/placement/+/793832 | 14:08 |
| ganso | bauzas: great! will work on that now! thank you! | 14:08 |
| bauzas | ganso: ping me when you're done | 14:09 |
| gibi | bauzas, sean-k-mooney, melwitt: when you have time please check back to the placement re-parenting spec. https://review.opendev.org/c/openstack/nova-specs/+/788243 | 14:13 |
| gibi | thanks | 14:13 |
| bauzas | gibi: sure thing | 14:13 |
| * bauzas will try to not bikeshed again | 14:14 | |
| gibi | :) | 14:15 |
| sean-k-mooney | gibi: yes its on my list for today. just looking at the review priorty doc review now | 14:19 |
| sean-k-mooney | gibi: do we have the team meeting here today by the way | 14:19 |
| sean-k-mooney | or are we using #openstack-meeting-3 | 14:19 |
| sean-k-mooney | if that exists here | 14:20 |
| opendevreview | Stephen Finucane proposed openstack/nova master: Deprecate filters that have been replaced by placement filters https://review.opendev.org/c/openstack/nova/+/745605 | 14:23 |
| gibi | sean-k-mooney: we will be on #openstack-meeting-3 | 14:25 |
| sean-k-mooney | ack | 14:26 |
| gibi | it do exists | 14:26 |
| gibi | but I will raise the question if we want to move the meeting to here | 14:26 |
| opendevreview | Rodrigo Barbieri proposed openstack/nova master: Error anti-affinity violation on migrations https://review.opendev.org/c/openstack/nova/+/784166 | 14:31 |
| ganso | bauzas: ^ I improved the reno and commit message | 14:31 |
| *** gouthamr_ has joined #openstack-nova | 14:34 | |
| *** gouthamr has joined #openstack-nova | 14:38 | |
| *** opendevreview has quit IRC | 14:44 | |
| stephenfin | lyarwood: $ openstack server create ... --block-device source_type=volume,uuid=44d317a3-6183-4063-868b-aa0728576f5f,destination_type=volume,delete_on_termination=true --wait test-server | 15:06 |
| stephenfin | Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. | 15:06 |
| stephenfin | <class 'nova.exception.VolumeNotFound'> (HTTP 500) (Request-ID: req-7fe03627-c4ce-4f4b-9d5c-3abd6b88d3e3) | 15:06 |
| *** luksky has joined #openstack-nova | 15:09 | |
| stephenfin | lyarwood: full traceback here http://paste.openstack.org/show/806238/ | 15:09 |
| *** gouthamr_ has quit IRC | 15:12 | |
| *** gouthamr has quit IRC | 15:13 | |
| *** gouthamr has joined #openstack-nova | 15:14 | |
| sean-k-mooney | stephenfin: your create a server with a prexising volumn | 15:15 |
| stephenfin | yes | 15:15 |
| sean-k-mooney | stephenfin: did you intent to create a server wiht a boot volume form an image? | 15:16 |
| stephenfin | nope | 15:16 |
| stephenfin | I'm updating delete_on_termination=true | 15:16 |
| stephenfin | whoops | 15:16 |
| *** opendevreview has joined #openstack-nova | 15:16 | |
| opendevreview | Merged openstack/placement master: [doc] Redirect people to #openstack-nova https://review.opendev.org/c/openstack/placement/+/793832 | 15:16 |
| stephenfin | doc/source/user/launch-instance-from-volume.rst | 15:16 |
| stephenfin | to use OSC | 15:16 |
| sean-k-mooney | well to use your new os command | 15:16 |
| sean-k-mooney | not just osc | 15:16 |
| sean-k-mooney | but that is weird that sould be correct | 15:16 |
| stephenfin | both | 15:16 |
| stephenfin | the command is correct except I did use the wrong UUID | 15:17 |
| stephenfin | that's my image UUID, not the volume | 15:17 |
| stephenfin | but the server should be handling this and spitting out a better error | 15:17 |
| sean-k-mooney | ya that would make sense | 15:17 |
| sean-k-mooney | it should not be a 500 | 15:17 |
| sean-k-mooney | it should be a 400 | 15:17 |
| stephenfin | exactly | 15:17 |
| *** gouthamr is now known as identify | 15:18 | |
| sean-k-mooney | ya so i guess we just need to wrap the volumen not found expciton | 15:18 |
| sean-k-mooney | although | 15:18 |
| stephenfin | I can fix but I'm notifying lyarwood since this seems like something that wouldn't go unnoticed for long, so I suspect it's only broken recently (like, last year) | 15:18 |
| sean-k-mooney | we really should be validating this in the api | 15:18 |
| sean-k-mooney | im not sure about that | 15:19 |
| *** gouthamr has joined #openstack-nova | 15:19 | |
| sean-k-mooney | stephenfin: i would expect most peopple to use "openstack server create --flavor 2 --volume VOLUME_ID myInstanceFromVolume" | 15:20 |
| stephenfin | I'd say that will fail with the same error message | 15:20 |
| stephenfin | since it's doing the same thing under the hood | 15:20 |
| * stephenfin verifies | 15:20 | |
| *** identify has quit IRC | 15:20 | |
| sean-k-mooney | it will but i thik they would be less likely to make the mistake of mixing the uuids | 15:21 |
| sean-k-mooney | am i right that you had just previouly tested "-block-device source_type=image,uuid=44d317a3-6183-4063-868b-aa0728576f5f,destination_type=volume,delete_on_termination=true" | 15:22 |
| stephenfin | nope, I just made a mistake | 15:22 |
| stephenfin | the previous command had been 'openstack image list' | 15:22 |
| sean-k-mooney | and then updated the source_type to volumn with out actully updatign the uuid | 15:22 |
| sean-k-mooney | oh ok | 15:22 |
| stephenfin | so my lazy brain grabbed the first UUID it saw on screen :) | 15:22 |
| sean-k-mooney | stephenfin: well the current error is happing on the compute so this should defintly be caught in the api before we create a server record | 15:23 |
| sean-k-mooney | maybe here https://github.com/openstack/nova/blob/da57eebc9e1ab7e48d4c4ef6ec1eeba80d867d81/nova/api/openstack/compute/servers.py#L470 | 15:27 |
| sean-k-mooney | oh in _validate_and_build_base_options https://github.com/openstack/nova/blob/c0c2888acaaccfa4266874523f62039cb9e143e0/nova/compute/api.py#L968 | 15:30 |
| kashyap | stephenfin: [Off-topic] I have an inkling you'll like this: https://www.merriam-webster.com/words-at-play/can-a-ship-flounder | 15:45 |
| kashyap | stephenfin: Also did you know both noun and verb usage of 'flounder and 'founder'? (Before reading the article :) | 15:46 |
| sean-k-mooney | they are two differnt things | 15:46 |
| sean-k-mooney | a fish can flounder a ship cannot as far as im aware | 15:47 |
| stephenfin | and vice versa, iirc | 15:48 |
| sean-k-mooney | a wale that is beached on land can flounder, when it tries to get back into the water | 15:49 |
| * gibi queued up that etimology article for night time reading.. | 15:49 | |
| gibi | nova meeting starts in 10 minutes in #openstack-meeting-3 on OFTC ;) | 15:50 |
| sean-k-mooney | hum i see how it could be used and it makes sense but it a bit arcic to use flounder for an in animiat object even by irish standard even if its technicaly allowable | 15:52 |
| kashyap | sean-k-mooney: Yeah; they are two different things; but using "founder" as a verb for ship (but not "flounder") was a new thing for me | 15:53 |
| sean-k-mooney | both words honestly have little usage day to day | 15:54 |
| kashyap | Yeah; flounder is more often seen in writng, though | 15:55 |
| opendevreview | Elod Illes proposed openstack/nova stable/wallaby: DNM: test ceph checkout with branch https://review.opendev.org/c/openstack/nova/+/794000 | 15:56 |
| gibi | lyarwood, melwitt: FYI ^^ with elod we might caught a ceph testing issue | 15:56 |
| *** lucasagomes has quit IRC | 16:00 | |
| *** alex_xu has quit IRC | 16:03 | |
| *** hjensas has joined #openstack-nova | 16:15 | |
| opendevreview | Stephen Finucane proposed openstack/nova master: api: Handle invalid volume UUIDs during spawn https://review.opendev.org/c/openstack/nova/+/794006 | 16:41 |
| opendevreview | Stephen Finucane proposed openstack/nova master: docs: Replace 'nova boot' with 'openstack server create' https://review.opendev.org/c/openstack/nova/+/794007 | 16:41 |
| stephenfin | lyarwood: ^ easy fix. I didn't add unit tests because they seemed silly (see reasoning inline) but I can if people insist | 16:43 |
| *** derekh has quit IRC | 17:00 | |
| *** vishalmanchanda has quit IRC | 17:10 | |
| sean-k-mooney | gibi: so we did tweak the realtime handeling to allow alll cores to be used but we required the emulator policy to be used | 17:14 |
| sean-k-mooney | https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/releasenotes/notes/bug-1884231-16acf297d88b122e.yaml | 17:14 |
| sean-k-mooney | gibi: so what we allow today is you can use a specific emulator policy which wont have the downstream issue or you can use a realtime mask | 17:15 |
| gibi | seem good | 17:15 |
| masterpe[m] | Hi, I try to extend a in-use volume on Openstack Train. In the logs I get the following "error: Forbidden: Policy doesn't allow os_compute_api:os-server-external-events:create to be performed" | 17:15 |
| masterpe[m] | Is there an article where it explains how to change the policy so that it is allowed? I searched but I was not able to find it. | 17:15 |
| sean-k-mooney | masterpe[m]: i think this is a know cinder bug | 17:16 |
| sean-k-mooney | masterpe[m]: basicaly when cinder calls back the nova api its incorrectly using the user token to call the admin only external events api | 17:16 |
| *** ralonsoh has quit IRC | 17:20 | |
| *** ralonsoh has joined #openstack-nova | 17:20 | |
| *** ralonsoh has quit IRC | 17:21 | |
| masterpe[m] | sean-k-mooney do you know if there is a patch for this cinder bug? | 17:21 |
| *** abhishekk has quit IRC | 17:22 | |
| sean-k-mooney | masterpe[m]: if you try this this as an admin i think it willl work | 17:31 |
| sean-k-mooney | masterpe[m]: but im not sure | 17:31 |
| sean-k-mooney | lyarwood: do you remember ^ | 17:31 |
| sean-k-mooney | masterpe[m]: what cinder backend are you using | 17:33 |
| masterpe[m] | End ceph | 17:33 |
| masterpe[m] | I wanted to write rbd but auto correct | 17:33 |
| sean-k-mooney | there was this previous bug for nfs https://bugzilla.redhat.com/show_bug.cgi?id=1640443 https://bugs.launchpad.net/tripleo/+bug/1802347 | 17:34 |
| opendevmeet | bugzilla.redhat.com bug 1640443 in openstack-tripleo-heat-templates "volume_extend while attached - Policy doesn't allow os_compute_api:os-server-external-events:create" [High,Closed: errata] - Assigned to abishop | 17:34 |
| opendevmeet | Launchpad bug 1802347 in tripleo "Need ability to configure cinder's settings for nova" [High,Fix released] - Assigned to Alan Bishop (alan-bishop) | 17:34 |
| sean-k-mooney | looks like they hacked around it in ooo to use the admin endpoint https://review.opendev.org/c/openstack/tripleo-heat-templates/+/616669/ | 17:36 |
| sean-k-mooney | masterpe[m]: instead of actully fixing the cinder bug | 17:36 |
| sean-k-mooney | osa also copied the ooo trick https://bugs.launchpad.net/openstack-ansible/+bug/1902914 | 17:38 |
| opendevmeet | Launchpad bug 1902914 in openstack-ansible "cinder extend "Policy doesn't allow os_compute_api:os-server-external-events:create to be performed"" [Medium,Fix released] - Assigned to Dmitriy Rabotyagov (noonedeadpunk) | 17:38 |
| *** pas-ha is now known as Guest486 | 17:38 | |
| masterpe[m] | We are using osa. I will look into that. | 17:42 |
| sean-k-mooney | masterpe[m]: what release did you say you were using? | 17:42 |
| sean-k-mooney | masterpe[m]: oh and to you orignal question you can change the policy for the endpoint by defining a custom policy nova's policy.json or policy.yaml file | 17:44 |
| sean-k-mooney | masterpe[m]: however you should not do that | 17:44 |
| sean-k-mooney | the external events api is intended for interservice comumnication https://docs.openstack.org/api-ref/compute/#create-external-events-os-server-external-events | 17:45 |
| sean-k-mooney | masterpe[m]: not even normal admin should actully call that manually as its very easy to break things and it might even be a secuirty issue in some cases. | 17:45 |
| sean-k-mooney | so the only way to adress your issue is to ensure that cinder uses an admin token to talk to that endpoint | 17:46 |
| masterpe[m] | My question was directly related to the error. But if there are better solution to the problem then .... | 17:47 |
| sean-k-mooney | masterpe[m]: yes that is a generic policy error message | 17:47 |
| sean-k-mooney | in some case adjusting policy would be valid in this case its not | 17:47 |
| *** andrewbonney has quit IRC | 18:10 | |
| *** Guest486 has quit IRC | 18:10 | |
| *** pas-ha[m] has joined #openstack-nova | 18:15 | |
| *** rloo has quit IRC | 18:18 | |
| *** rloo has joined #openstack-nova | 18:18 | |
| frickler | masterpe[m]: you need to configure cinder to use an admin user acc to talk to nova, see e.g. https://bugs.launchpad.net/openstack-ansible/+bug/1902914 | 18:24 |
| opendevmeet | Launchpad bug 1902914 in openstack-ansible "cinder extend "Policy doesn't allow os_compute_api:os-server-external-events:create to be performed"" [Medium,Fix released] - Assigned to Dmitriy Rabotyagov (noonedeadpunk) | 18:24 |
| frickler | oh, that's the same one as cited above | 18:25 |
| *** dklyle has joined #openstack-nova | 18:25 | |
| frickler | fwiw we also copied that solution into openstack-chef, though | 18:26 |
| *** belmoreira has quit IRC | 18:35 | |
| *** Gue______ has quit IRC | 19:02 | |
| opendevreview | Lee Yarwood proposed openstack/nova stable/ussuri: hardware: Use image_meta.id within get_mem_encryption_constraint https://review.opendev.org/c/openstack/nova/+/793958 | 19:06 |
| *** Gue______ has joined #openstack-nova | 19:11 | |
| *** rpioso has joined #openstack-nova | 19:27 | |
| opendevreview | Merged openstack/nova master: Replace getargspec with getfullargspec https://review.opendev.org/c/openstack/nova/+/790405 | 19:33 |
| *** rpioso is now known as rpioso|afk | 19:40 | |
| *** rpioso|afk is now known as rpioso | 19:52 | |
| *** rpioso is now known as rpioso|afk | 20:07 | |
| *** rpioso|afk is now known as rpioso | 20:41 | |
| opendevreview | Lee Yarwood proposed openstack/nova stable/train: Replace 'fake' with a real project ID https://review.opendev.org/c/openstack/nova/+/794041 | 20:50 |
| opendevreview | Lee Yarwood proposed openstack/nova stable/train: image_meta: Provide image_ref as the id when fetching from instance https://review.opendev.org/c/openstack/nova/+/794042 | 20:50 |
| opendevreview | Lee Yarwood proposed openstack/nova stable/train: hardware: Use image_meta.id within get_mem_encryption_constraint https://review.opendev.org/c/openstack/nova/+/794043 | 20:50 |
| *** rpioso is now known as rpioso|afk | 21:08 | |
| *** rpioso|afk is now known as rpioso | 21:46 | |
| *** rpioso is now known as erbarr|afk | 21:47 | |
| *** erbarr|afk is now known as rpioso | 21:48 | |
| *** luksky has quit IRC | 21:59 | |
| opendevreview | Artom Lifshitz proposed openstack/nova master: WIP: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/776250 | 22:31 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: WIP: Rollback live migraiton if cancelled while queued https://review.opendev.org/c/openstack/nova/+/794048 | 22:31 |
| artom | That func test ^^ is disgusting h4xx | 22:31 |
| *** tosky has quit IRC | 23:11 | |
| *** martinkennelly has quit IRC | 23:52 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!