| opendevreview | Wenping Song proposed openstack/nova master: Fill the AcceleratorRequestBindingFailed exception msg info https://review.opendev.org/c/openstack/nova/+/817326 | 02:19 |
|---|---|---|
| opendevreview | Wenping Song proposed openstack/nova master: Fill the AcceleratorRequestBindingFailed exception msg info https://review.opendev.org/c/openstack/nova/+/817326 | 02:52 |
| opendevreview | likui proposed openstack/osc-placement master: Replace deprecated assertRaisesRegexp https://review.opendev.org/c/openstack/osc-placement/+/817365 | 09:09 |
| stephenfin | lyarwood: the py36 tox environment is failing for me locally (F34). I proposed this to fix it https://github.com/powervm/pypowervm/pull/17 | 09:52 |
| stephenfin | We'll need that to merge and a new release of pypowervm be cut before we can get things running again. I suspect we're not seeing issues in the gate yet because an older dependency resolver that doesn't care about these things is being used there | 09:53 |
| stephenfin | Just FYI since you're also using Fedora :) | 09:53 |
| gibi | elodilles, lyarwood: hi! re: kernel panic in stable/victoria with volume detach: https://bugs.launchpad.net/nova/+bug/1950310 yesterday I backported the libvirt event based detach series to victoria but top of that series the kernel panic still happens (links in the bug for the recent failures) | 10:07 |
| gibi | I'm out of ideas now | 10:08 |
| *** melwitt is now known as Guest5508 | 10:12 | |
| sean-k-mooney[m] | gibi i could maybe try to revive my attempts to add support for alpine to dib so that we could use it instead of cirros | 10:24 |
| sean-k-mooney[m] | i started that orignally due to kernel bugs in the cirros kernel | 10:25 |
| gibi | sean-k-mooney[m]: I'm not sure if it is really cirros. i) it happens with both cirros 0.5.1 and 0.5.2 but it does not happen on master just on stable/victoria | 10:26 |
| gibi | s/i)// | 10:26 |
| sean-k-mooney[m] | i see | 10:26 |
| sean-k-mooney[m] | if its still happening with you event based detach patch the only other thing i can think of is changing the qemu version we are using | 10:27 |
| sean-k-mooney[m] | i used to maintain a devstack plugin to compile qemu form git but i have not updated it to support the mason build systme wich they now use. we could try changing the qemu we use either via a ppa or compiling it | 10:29 |
| gibi | hm interesting, I don't know how different the qemu version between master and stable/victoria but I can check that | 10:29 |
| sean-k-mooney[m] | master would be focal is stable victoria on bionic? | 10:29 |
| frickler | gibi: sean-k-mooney[m]: if you want newer qemu, could you try running on bullseye instead ubuntu? | 10:33 |
| sean-k-mooney[m] | frickler: bullseye and centos would be options. its not really about newer it could be older. if the panics are caused by a kvm or qemu bug then just using a diffent disto with different verions might fix the problem | 10:34 |
| lyarwood | gibi: yeah I didn't think it would help tbh, you're still firing device_del down to QEMU and the ACPI requests to the guestOS, are we pinning tempest on stable/victoria? There was an effort recently to force tempest to wait until the guestOS was up before attaching and detaching volumes, I wonder if that logic is there? | 10:36 |
| lyarwood | gibi: I've had my head downstream on something since yesterday but I'll try to help out some more for the rest of the week | 10:36 |
| sean-k-mooney[m] | you mean adding the validation so that we ssh into the vm before atching/detaching | 10:37 |
| sean-k-mooney[m] | or something else | 10:37 |
| lyarwood | sean-k-mooney: yeah that's it | 10:37 |
| lyarwood | stephenfin: ack not seen that personally but will keep it in mind | 10:38 |
| sean-k-mooney[m] | lyarwood has anyone submited patches for that | 10:39 |
| lyarwood | I had thought it already landed in master tbh | 10:39 |
| * lyarwood goes and checks | 10:39 | |
| lyarwood | okay no it hasn't | 10:41 |
| lyarwood | I was sure I saw a change land for this or maybe my dreams are really becoming boring now I'm half way through my 30's | 10:41 |
| sean-k-mooney[m] | we spoke about it in out inernal meeting last week but unless you wrote it i dont think it happened. | 10:42 |
| sean-k-mooney[m] | if i remember correctly we just need to set validation=true on the create server line in the test | 10:43 |
| gibi | lyarwood: do you have a link for the patch that adds that validation? I can add a depends-on somewhere in nova to see if that helps | 11:30 |
| lyarwood | gibi: no sorry I couldn't find it, I was sure I saw something in the scenario manager for attempted to do it, I'll look again after lunch and throw something up on gerrit if I can't find it | 12:10 |
| gibi | ack thanks | 12:11 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/817256 | 13:33 |
| opendevreview | Balazs Gibizer proposed openstack/placement stable/xena: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/817255 | 13:35 |
| gibi | gmann: ^^ added the missing cherry-pick lines in these ^^ | 13:35 |
| *** whoami-rajat__ is now known as whoami-rajat | 14:00 | |
| gmann | gibi: thanks | 14:25 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Fix interference in db unit test https://review.opendev.org/c/openstack/nova/+/817240 | 15:05 |
| opendevreview | Stephen Finucane proposed openstack/nova master: Use unittest.mock instead of third party mock https://review.opendev.org/c/openstack/nova/+/714676 | 15:25 |
| lyarwood | Just when you thought the nova codebase was safe from stephenfin ;) | 15:43 |
| lyarwood | Can we give you the Friday nick of SirChangeALot? | 15:44 |
| stephenfin | someone complained about us still using it recently. blame that person | 15:44 |
| stephenfin | whose name I can't remember | 15:44 |
| lyarwood | ^_^ | 15:44 |
| lyarwood | I'll queue that up later in the week | 15:44 |
| dasm | "it's them, not us" :) | 15:44 |
| sean-k-mooney | lyarwood: for what its worth its really pretty tirvial | 15:49 |
| lyarwood | yeah I know I was only jesting | 15:49 |
| sean-k-mooney | most of it was find replace and one or two other changes which are ease to fined by the change size beside the file | 15:49 |
| sean-k-mooney | it made me sad you cant sort the list of files in the gerrit review by changed lines | 15:50 |
| sean-k-mooney | it would be a nice feature for that type of change | 15:50 |
| EugenMayer | Ah nova backup process was queued, but it was queue since i had not enough space. I removed the queued images in the ui (glance images), but now instances show up a task `image backup` and i cannot see how to remove it. Is that a nove tasks i can list (cannot find anything with the nova cli) or is that rather a glance task? A little confused who | 15:53 |
| EugenMayer | holds that data | 15:53 |
| EugenMayer | `glance task-list` does not show up those tassks | 15:54 |
| EugenMayer | is `nova reset-state --active` the way to solve this | 16:00 |
| opendevreview | Alexey Stupnikov proposed openstack/nova master: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/776250 | 16:41 |
| sean-k-mooney | gibi: good to see the neutron sepc is apporved ill try to rereview https://review.opendev.org/c/openstack/nova-specs/+/787458 this week but i belive its largely the same as v9 and i was ok with the overall design a that point. | 17:37 |
| sean-k-mooney | by the way when was/is the spec review day | 17:37 |
| sean-k-mooney | ah the 16th | 17:38 |
| sean-k-mooney | i tought i migth have missed it ill be looking at specs tomorrow in anycase | 17:38 |
| sean-k-mooney | EugenMayer: reste-state shoudl be used with caution generally as a last resort before do ing a direct db modification | 17:40 |
| sean-k-mooney | if its stuck in the image backup ste a hard reboot might be a better first attempt | 17:40 |
| sean-k-mooney | reststate may leave the instnace in an unclean state | 17:40 |
| opendevreview | Merged openstack/nova master: db: Remove models that were moved to the API database https://review.opendev.org/c/openstack/nova/+/812149 | 17:56 |
| EugenMayer | sean-k-mooney that is what happened, reset state was clearing the ui, but it stuck nevertheless. Tried soft reboot, was not able to, so i rebooted the compute | 17:57 |
| EugenMayer | i cannot say that openstack shines in this areas to be honest. Not used to have such undefined, oncontrolled states created by UI interactions alone | 17:58 |
| EugenMayer | in general, tasks cannot be stopped, cannot be seen and cannot be controlled, while they can stuck. This is the one area i'am kind of frustrated about | 17:58 |
| sean-k-mooney | EugenMayer: that is because they are not independent tasks for the most poart | 17:58 |
| sean-k-mooney | there are a limit set of operation that are modeled as task like live migration that can be aborted | 17:59 |
| sean-k-mooney | but most api action are not premetable once issued | 17:59 |
| sean-k-mooney | EugenMayer: creating a backup for example is not really a task its an imperitive action you have taken that shoudl either complete or put the vm in an error state | 18:02 |
| sean-k-mooney | that error state can be then recovered form vai a hard-reboot or other operation | 18:02 |
| sean-k-mooney | where we can recover form a failed operation nova shoudl and mark the instnace event in error | 18:02 |
| opendevreview | Merged openstack/nova master: objects: Remove 'bandwidth' fields from notifications https://review.opendev.org/c/openstack/nova/+/812148 | 18:02 |
| opendevreview | Merged openstack/nova master: db: Remove models for removed services, features https://review.opendev.org/c/openstack/nova/+/812150 | 18:03 |
| EugenMayer | it's what happens if 'microservice and self contained' is the main principal as an arch without controller the echo that comes with it. In an image backup, you invoke via nova, which then invokes a glance task, set some second state or whatever somewhere. So we have a glance task and state, a second task and state for the server and nothing that is | 18:03 |
| EugenMayer | actually controlling it. And if the task creating that image never reports back and most probably moves the server task/state further, it is stuck forever | 18:03 |
| opendevreview | Dan Smith proposed openstack/nova master: Log instance event wait times https://review.opendev.org/c/openstack/nova/+/817030 | 18:04 |
| opendevreview | Merged openstack/nova master: db: Remove nova-network models https://review.opendev.org/c/openstack/nova/+/812151 | 18:04 |
| EugenMayer | if you decouple that much, the hard thing becomes cascading and controlling events and tasks which involve several subsystem. And for this part, it seems that is where openstack has nothing in place. Nova tasks in nove - great. if it involves other parts, things get stuck very fast. Be it neutron, the storage or similar. | 18:04 |
| EugenMayer | Anyway, sorry. A little frustrated. I get over it :) | 18:05 |
| sean-k-mooney | we do have some recovery logic but asa an operator you really should not be managing any of that internal state | 18:05 |
| sean-k-mooney | https://docs.openstack.org/api-ref/compute/#servers-actions-servers-os-instance-actions is where you can list the action performed on an instance by the way | 18:06 |
| EugenMayer | (i would love to not manage it, but it becomes my duty if queued stuck because it waits to claim storage space, which does not happen and then it rolls forever) | 18:06 |
| sean-k-mooney | it should actully fail if there is not enouch space. is it failing on the image upload | 18:07 |
| sean-k-mooney | or the creation of the inital snapshot | 18:07 |
| EugenMayer | sean-k-mooney interesting, i have see that API but assumed this was rather 'list the actions one can do'. I guess 'nova instance-action-list' is not that, but 'openstack server event list esxi` | 18:08 |
| EugenMayer | well i can assure you, it did not fail but has been queued. The task was started using `nova backup ...` | 18:09 |
| sean-k-mooney | qued in glance | 18:10 |
| EugenMayer | Yes, but 'instance backup' in nova/compute | 18:10 |
| EugenMayer | i deleted the image in glance (not stopping the task, if that is possible at all), which then stuck the compute instance in 'instance backup' forever | 18:11 |
| EugenMayer | A different thing, trying to host an ESXi as a VM on openstack KVM. The sole purpose of that VM is using ESXi native tools to convert qcow2 disks to _proper_ vmdks | 18:12 |
| sean-k-mooney | instance backup... so its no in image uploading or image pendign upload | 18:12 |
| sean-k-mooney | EugenMayer: right we would not expect deleting the image in glance to stop the task | 18:13 |
| sean-k-mooney | that could infact break our cleanup code | 18:13 |
| EugenMayer | I have run this under proxmox with ethernet e1000 but under openstack it starts with virtio, which seems not to be supported by ESXi thus the VM has no interface when booted. Can i switch the interface type? | 18:13 |
| EugenMayer | sean-k-mooney that i understood - after that :) | 18:13 |
| sean-k-mooney | you can specify e1000 | 18:13 |
| EugenMayer | via xml only? | 18:13 |
| sean-k-mooney | you set hw_vif_model=e1000 in the glance image metadata | 18:13 |
| sean-k-mooney | https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L111 | 18:14 |
| EugenMayer | that is pure meta-data,right? | 18:14 |
| sean-k-mooney | it will change the xml that is generated | 18:15 |
| sean-k-mooney | that is how you change the xml in a supported way | 18:15 |
| sean-k-mooney | that and flaovr extra specs but devicve models are genrealy only tunable via the image | 18:15 |
| EugenMayer | ah the flavor can stear that? That's fine, i create a dedicated flavor via terraform anyway | 18:16 |
| sean-k-mooney | a flaovr cannot change that option | 18:16 |
| sean-k-mooney | it can change some others | 18:16 |
| EugenMayer | so it must be an image option only, ok | 18:16 |
| sean-k-mooney | everything in that file is setable via the glance image only | 18:16 |
| sean-k-mooney | yes the image used for the root disk | 18:16 |
| sean-k-mooney | so your esxi image in this case | 18:16 |
| sean-k-mooney | openstack is a cloud plathform rather then virtualisation solution | 18:17 |
| opendevreview | Julia Kreger proposed openstack/nova master: Ironic - Handle instance host on rebalance https://review.opendev.org/c/openstack/nova/+/813897 | 18:17 |
| sean-k-mooney | so you create high levle templatees via the flavor and imave that you then instnaciate many copies of ranter then creatin indivicual instances | 18:17 |
| EugenMayer | i see, i try to adopt that. Can i manipualte an existing glance image or how would i introduce this? | 18:18 |
| sean-k-mooney | modifying an exsiting image will not affect instnace already using it | 18:18 |
| sean-k-mooney | but you can update it | 18:18 |
| EugenMayer | sure, i will redo the instance. That is what i do with terraform anyhhow | 18:18 |
| sean-k-mooney | genreeally if it widely used in production you will not modify it but its really a case by case decision | 18:18 |
| EugenMayer | ok so it is API only i guess | 18:18 |
| EugenMayer | is this an md-property? | 18:19 |
| sean-k-mooney | when we create an image we "snapshot" or copy the exra specs form the flavor and metadta form the image and store a perinstance copy | 18:19 |
| sean-k-mooney | that copy is then used for the lifetime of the instnace | 18:20 |
| mloza | hi folks, i have two kind of computes, first one is configured to store boot disk in ceph and second is configured to store boot disk locally on the host. Both works as intendend but second one can be bypass if the user tries to boot the instance from a volume | 18:20 |
| sean-k-mooney | yes these are image metadata properties | 18:20 |
| sean-k-mooney | mloza: correct that is expect | 18:21 |
| sean-k-mooney | if the user boots from a cinder volume that storage will always be used | 18:21 |
| sean-k-mooney | so if you have a cinier providing storage via say iscsi | 18:21 |
| sean-k-mooney | then the boot form volume instnace can boot on either host by default | 18:22 |
| EugenMayer | sean-k-mooney: glance image-update bbcb16b8-8c84-4b01-92e8-6d66938460bf --property hw_vif_model=e1000 .. is taht what you mean by updating the image? | 18:22 |
| sean-k-mooney | EugenMayer: yes exactly | 18:22 |
| sean-k-mooney | if you create a new vm with that image then it should have e1000 nics instead of virtio | 18:22 |
| mloza | sean-k-mooney: seems what I expected. Thanks for the info | 18:23 |
| EugenMayer | ok learning. Well needless to say, but still much appretiated. Thank you so much! | 18:23 |
| EugenMayer | i guess saving https://github.com/openstack/glance/blob/master/etc/metadefs/compute-libvirt-image.json#L111 and having this file as lookup makes sense. That link i store :) | 18:24 |
| sean-k-mooney | EugenMayer: there is a nova doc for this too | 18:24 |
| sean-k-mooney | im currently looking for it | 18:24 |
| sean-k-mooney | that file is basicaly where horizon and heat get there info | 18:25 |
| sean-k-mooney | to generate uis | 18:25 |
| sean-k-mooney | horizone quiery glance for the set of know image properties | 18:25 |
| sean-k-mooney | via the metadefs api | 18:25 |
| sean-k-mooney | EugenMayer: this is glances docs on the image porperties https://docs.openstack.org/glance/latest/admin/useful-image-properties.html | 18:25 |
| EugenMayer | interesting. Kind of unusual to me to have all this in a map, not a typed object. But i guess that is due to the gazzilion different names/options different hypervisors would need | 18:26 |
| EugenMayer | wow that is a nice one, thanks | 18:26 |
| sean-k-mooney | the actual values are defiend in a concreate object in nova | 18:30 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L158 | 18:30 |
| EugenMayer | sean-k-mooney next time when and glance operation is queued and i want to cancel, should i use `glance image-tasks` and then? i cannot see anything like stop/kill/cancel tasks when i have the ID | 18:31 |
| sean-k-mooney | EugenMayer: theses are the concreate field definiton https://github.com/openstack/nova/blob/master/nova/objects/image_meta.py#L293-L563 | 18:31 |
| EugenMayer | This all got straight into my docs, thank you for digging those gems out | 18:33 |
| EugenMayer | interesting, the interface is now available in ESXi but somehow DHCP does not work. That is unexpected | 18:33 |
| opendevreview | Merged openstack/nova master: Apply common irrelevant_files for centos 8 job https://review.opendev.org/c/openstack/nova/+/817278 | 18:35 |
| EugenMayer | sean-k-mooney up up and away to dinner. Thank you big times. Not sure why DHCP is not working on that ESXi, i will fiddle with that tomorrow. Have a nice evening/moring/whatever :) | 18:42 |
| opendevreview | Julia Kreger proposed openstack/nova master: WIP Ironic - Reconcile compute host records on rebalance https://review.opendev.org/c/openstack/nova/+/817467 | 18:42 |
| sean-k-mooney | EugenMayer: you too enjoy dinner | 18:49 |
| opendevreview | Alexey Stupnikov proposed openstack/nova master: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/776250 | 20:19 |
| opendevreview | Hang Yang proposed openstack/nova master: Support creating servers with RBAC SGs https://review.opendev.org/c/openstack/nova/+/811521 | 21:50 |
| opendevreview | Merged openstack/nova stable/victoria: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810915 | 22:58 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!