| opendevreview | xiaoxin yang proposed openstack/nova master: Secure boot requires SMM feature enabled https://review.opendev.org/c/openstack/nova/+/808126 | 02:12 |
|---|---|---|
| opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | 02:41 |
| *** legochen_ is now known as legochen | 03:36 | |
| opendevreview | Jorhson Deng proposed openstack/nova master: remove some redundant parameters in migrate_server https://review.opendev.org/c/openstack/nova/+/808143 | 06:08 |
| admin1 | seeking info on what happens during a migration .. say i have h1 ( hypervisor1 ) and h2 ( hypervisor2 ) .. when I want to migrate vm1 on h1 -> h2, what makes the ssh conection .. is it h1 nova -> h2 nova .. or h1 libvirt -> h2 libvirt or h1 root -> h2 root .. | 07:00 |
| gibi | bauzas: o/ friendly reminder that we need to land melwitt's placemnt fix in Xena RC1 https://review.opendev.org/c/openstack/placement/+/807014 | 07:24 |
| gibi | melwitt, dansmith: we are out of EU cores for now, so could you please look at a small doc patch before RC1 https://review.opendev.org/c/openstack/nova/+/807564 ? | 07:25 |
| gibi | melwitt, dansmith: also the reno prelude is ready https://review.opendev.org/c/openstack/nova/+/807786 | 07:25 |
| bauzas | gibi: yeah I know and I started to review it | 07:56 |
| bauzas | but y'know how it's a bit difficult | 07:56 |
| bauzas | + loving escalations during the last 2 days :( | 07:56 |
| gibi | bauzas: thanks. feel free to ask questions about the patch I should be able to answer them | 07:57 |
| gibi | ahh escalations. :/ | 07:57 |
| bauzas | don't know what happended | 07:59 |
| bauzas | maybe because of the moon or because of puppets | 07:59 |
| bauzas | but looks like we have a lot of customers asking us to help them | 07:59 |
| gibi | maybe they are back from summer vacation :) | 08:00 |
| bauzas | they're not French :p | 08:01 |
| gibi | then they are just nice and waited for the French to return from their summer vacation :P | 08:08 |
| admin1 | one more question .. if cold migration works but live does not -- ceph backend .. where do i need to start looking .. the nova logs ( debug true ) does not provide any actionable insight .. | 08:28 |
| gibi | bauzas: do you agree with my triage here https://bugs.launchpad.net/nova/+bug/1939856 that it is probably a new feature not a bugfix? | 09:30 |
| * bauzas looks | 09:31 | |
| bauzas | gibi: fwiw, I +wd https://review.opendev.org/c/openstack/nova/+/792357 | 09:34 |
| gibi | bauzas: thanks | 09:35 |
| bauzas | so we can start to use the new label | 09:35 |
| bauzas | but then I'll provide a new change explaining what I'd like to have to helping contributors | 09:35 |
| gibi | OK, I will check that | 09:35 |
| bauzas | gibi: about the bug you asked me | 09:36 |
| bauzas | it's honestly a bug but Wishlist looks good to me | 09:37 |
| bauzas | as you said, given the instance was offloaded from the compute, it's no longer possible to ask nova to wait for deleting it | 09:37 |
| *** elodilles_pto is now known as elodilles | 09:37 | |
| gibi | I would be affraid to change the delete behavior of offloaded instances as the current behavior was there forever and it is not really leaking any resources (I checked ports and volumes are cleaned up) | 09:42 |
| gibi | so probalby people out there are adapted to this | 09:43 |
| bauzas | gibi: we can fix this by documenting :) | 09:51 |
| gibi | I can accept a documentation patch for it sure | 09:51 |
| kevko | Hi folks, in victoria I am hitting this -> /var/log/kolla/nova/nova-compute.log:2021-09-10 09:49:09.815 8 ERROR nova.virt.libvirt.driver [req-8082a663-ca30-4845-9e93-2429c500ba76 31031550a7c94928abbd9dfa8634fba1 7b2ac086026741aa82c6c9c96ded42ee - default default] [instance: cf3ae6f9-a971-4123-ae73-9b597364985a] attaching network adapter failed.: libvirt.libvirtError: internal error: unable to execute QEMU command 'netdev_add': Invalid parameter | 10:13 |
| kevko | type for 'vhost', expected: boolean | 10:13 |
| kevko | is it bug ? | 10:13 |
| sean-k-mooney | kevko: can you past the libvirt xml somewhere so we can see what was generated | 10:25 |
| sean-k-mooney | also what libvirt verions are you useing | 10:25 |
| kevko | sean-k-mooney: https://paste.opendev.org/show/809223/ | 10:28 |
| kevko | debian version | 10:28 |
| kevko | buster one | 10:28 |
| sean-k-mooney | that meets our min libvirt if only barely https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L225 | 10:32 |
| sean-k-mooney | can you provide teh xml that was generated so we can see where the type element was added | 10:32 |
| sean-k-mooney | i assume this was an interface attach? | 10:32 |
| kevko | sean-k-mooney: well, i'm running tempests against stack .. if i turn on debug .. will i see xml ? | 10:33 |
| kevko | sean-k-mooney: yeah, interface attach | 10:33 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [PoC][yoga] Off-path Networking Backends Support https://review.opendev.org/c/openstack/nova/+/808199 | 10:33 |
| sean-k-mooney | yes if you trun on debug you see a about a 8 line segment of xml printed for attach | 10:33 |
| sean-k-mooney | we print the full xml when startign the vm and just the bit we are appending for attach | 10:34 |
| kevko | ok, give me 5 minutes cca | 10:34 |
| kevko | sean-k-mooney: https://paste.opendev.org/show/809224/ | 10:37 |
| kevko | sean-k-mooney: bigger log -> https://paste.opendev.org/show/809225/ | 10:39 |
| sean-k-mooney | that is odd that looks normal to me | 10:39 |
| sean-k-mooney | oh what is your qemu version | 10:40 |
| sean-k-mooney | is it below 4.0.0 | 10:40 |
| kevko | https://paste.opendev.org/show/809226/ | 10:40 |
| kevko | nope, it's 5.2 | 10:40 |
| sean-k-mooney | ok i was wondering if it was older then libvirt suported | 10:41 |
| sean-k-mooney | thye xml we are generating looks corect to me | 10:41 |
| sean-k-mooney | so i suspect the issue is in the qemu args that libvirt is creeating | 10:41 |
| sean-k-mooney | have you looked in the qemu instance long | 10:42 |
| sean-k-mooney | *log | 10:42 |
| sean-k-mooney | in /var/log/libvirt/qemu/instace* | 10:42 |
| sean-k-mooney | is there any lower level error there | 10:42 |
| kevko | let me check | 10:42 |
| kevko | sean-k-mooney: nothing https://paste.opendev.org/show/809227/ | 10:45 |
| sean-k-mooney | odd i dont even see thte attach | 10:46 |
| sean-k-mooney | although it might not log that if it does not work | 10:46 |
| sean-k-mooney | kevko: other then comparing the full xml to the fragment to see if there is any delta (comparing to the exising interface) | 10:49 |
| sean-k-mooney | im not sure we can be much more help this looks like a libvirt bug | 10:49 |
| kevko | i'm trying to replace victoria libvrit container for wallaby | 10:49 |
| kevko | let me check what will happen :P | 10:49 |
| sean-k-mooney | it likely should work nova support a fairly wide range | 10:50 |
| sean-k-mooney | and libvirt has a pretty stable api | 10:50 |
| kevko | sean-k-mooney: success | 10:55 |
| kevko | sean-k-mooney: it's working | 10:55 |
| kevko | nice | 10:55 |
| kevko | so minimal version 5.0.0 is buggy | 10:55 |
| gibi | hm, this sounds like https://bugzilla.redhat.com/show_bug.cgi?id=1835312 | 10:55 |
| gibi | that points to https://bugzilla.redhat.com/show_bug.cgi?id=1832710 | 10:55 |
| gibi | which has a lot of libvirt commits to fix | 10:55 |
| gibi | so yeah, possible libvirt - qemu incompatibility | 10:56 |
| kevko | aaaaa | 10:57 |
| kevko | gibi: yeah, it looks like that is the same thing as i met | 10:57 |
| sean-k-mooney | ya since libvirt dont actully maintian stable branchs upstream frocing the distros to do it your milage might vary on the min version | 10:57 |
| sean-k-mooney | i dont think we actully test our min version in the ci | 10:58 |
| sean-k-mooney | we choose it based on the api feature set | 10:58 |
| opendevreview | Merged openstack/nova master: Describe the use of Review-Priority label https://review.opendev.org/c/openstack/nova/+/792357 | 10:58 |
| sean-k-mooney | and remove any condtional check for api compatiablity below that level | 10:58 |
| sean-k-mooney | most of our testing id done with whatever version is shiped with ubuntu 20.04 | 10:59 |
| sean-k-mooney | which is 6.0.0 | 11:00 |
| sean-k-mooney | kevko: so anything above 5.0.0 should be api compatibale based on teh feature we use but we have only tested with 6.0.0+ for victoria | 11:01 |
| sean-k-mooney | huh https://bugzilla.redhat.com/show_bug.cgi?id=1832710 well that is fun its nice when we are not told about this type of fix... | 11:05 |
| kevko | well, i'm using kolla debian images which are based on buster in victoria (wallaby is already bullseye ..) | 11:14 |
| kevko | so 6+ should be ok | 11:15 |
| kashyap | gibi: FOr that netdev_add thing - so long as you're not mixing "old libvirt and new QEMU" for a given distro, you should not hit that issue. | 12:26 |
| kashyap | gibi: I.e. if you're using CentOS 8.2, just stick to libvirt and QEMU from 8.2; don't mix versions from 8.3 | 12:27 |
| gibi | kashyap: it was kevko who had an env with the issue. I'm not sure about the host OS in kevko's env | 12:42 |
| kashyap | gibi: Sorry, yes; noticed it after you responded. (Didn't read the KM-long scrollback, afraid/) | 12:42 |
| gibi | no worries | 12:42 |
| gibi | kevko solved the issue by using newer libvirt | 12:43 |
| *** legochen_ is now known as legochen | 13:39 | |
| bauzas | melwitt: gibi: eventually +Wd https://review.opendev.org/c/openstack/placement/+/807014 for the create project bugfix, but with some nits | 13:50 |
| *** artom_ is now known as artom | 14:06 | |
| gibi | bauzas: thanks will read shortly | 14:16 |
| gibi | bauzas: I agree with your nits. thanks again | 14:27 |
| bauzas | gibi: ack, no worries, it's just nits | 15:09 |
| * bauzas stops to work for the day | 15:09 | |
| gibi | bauzas: have a nice weekend | 15:10 |
| bauzas | you too | 15:11 |
| * bauzas needs to work for a gate | 15:11 | |
| bauzas | (but not the CI one ;) ) | 15:11 |
| gibi | :) | 15:17 |
| melwitt | bauzas: thanks, I will make a followup patch to fix the nits | 15:19 |
| opendevreview | OpenStack Release Bot proposed openstack/os-vif stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808452 | 15:20 |
| opendevreview | OpenStack Release Bot proposed openstack/os-vif stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808453 | 15:20 |
| opendevreview | OpenStack Release Bot proposed openstack/os-vif master: Update master for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808454 | 15:20 |
| opendevreview | OpenStack Release Bot proposed openstack/os-vif master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/os-vif/+/808455 | 15:20 |
| gibi | melwitt: placment the patch just bounced from the gate with strange functional test failures... | 15:20 |
| opendevreview | OpenStack Release Bot proposed openstack/osc-placement stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808456 | 15:21 |
| opendevreview | OpenStack Release Bot proposed openstack/osc-placement stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808457 | 15:21 |
| melwitt | gibi: 😩 oh no. I will look | 15:21 |
| gibi | https://597ae34c2915b12741a5-9637172b828bef74a2ccde1fe8027212.ssl.cf1.rackcdn.com/807014/6/gate/openstack-tox-functional-py38/bc74f18/testr_results.html | 15:21 |
| opendevreview | OpenStack Release Bot proposed openstack/osc-placement master: Update master for stable/xena https://review.opendev.org/c/openstack/osc-placement/+/808458 | 15:21 |
| opendevreview | OpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808459 | 15:22 |
| opendevreview | OpenStack Release Bot proposed openstack/python-novaclient stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808460 | 15:22 |
| gibi | I can reproduce it on master locally without your patch so this is probably due to requirement change ... :/ | 15:22 |
| opendevreview | OpenStack Release Bot proposed openstack/python-novaclient master: Update master for stable/xena https://review.opendev.org/c/openstack/python-novaclient/+/808461 | 15:22 |
| opendevreview | OpenStack Release Bot proposed openstack/python-novaclient master: Add Python3 yoga unit tests https://review.opendev.org/c/openstack/python-novaclient/+/808462 | 15:22 |
| melwitt | gibi: oh hm | 15:23 |
| gibi | it is not visible in https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly so it happened in the last ~5 days | 15:24 |
| gibi | hm, the global req bumped oslo.db to 11.0.0 two days ago, that is a likely candidate | 15:26 |
| melwitt | agree. aside, there likely isn't a cross-placement job on global bump. I wonder if we should add one | 15:28 |
| gibi | OK I confirmed that those tests fail with oslo.db 11.0.0 but passing with oslo.db 10.0.0 (the previous version) | 15:31 |
| gibi | yeah I don't think we have a placement functional job on the global req repo | 15:31 |
| sean-k-mooney | i dont see one in https://github.com/openstack/requirements/blob/master/.zuul.d/cross-jobs.yaml | 15:31 |
| gibi | but we should | 15:31 |
| sean-k-mooney | we have a nova one but you are correct no placment one https://github.com/openstack/requirements/blob/master/.zuul.d/project.yaml#L42 | 15:32 |
| gibi | ok I failed to figure out how to reformulate those tests with the new sqlalchemy and I have to drop soon. sorry | 15:58 |
| melwitt | thanks gibi. I'm also trying to fix, will let you know if I find any success | 15:59 |
| gibi | thanks | 15:59 |
| gibi | and good luck | 15:59 |
| melwitt | thanks. have a nice weekend o/ | 15:59 |
| gibi | same to you too eventually o/ | 16:00 |
| melwitt | :) | 16:01 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: WIP: Gracefully power off guest on instance delete https://review.opendev.org/c/openstack/nova/+/808474 | 16:03 |
| sean-k-mooney | artom: ok so that jsut add a call self._power_off_instance | 16:29 |
| sean-k-mooney | are you sure we cant remove self._shutdown_instance(context, instance, bdms) | 16:29 |
| artom | sean-k-mooney, maybe? Seems it does more than just power if off though... | 16:29 |
| sean-k-mooney | it delete the volume attachments and detory the domain xml | 16:31 |
| sean-k-mooney | that seam to be about it | 16:31 |
| sean-k-mooney | presumable _cleanup_volumes would do the same | 16:32 |
| sean-k-mooney | ok _cleanup_volumes does not handel the attachments | 16:32 |
| sean-k-mooney | or calling terminat_connection | 16:32 |
| sean-k-mooney | what about powerofof | 16:32 |
| artom | lulz we just suck at reusing code, do we? | 16:32 |
| sean-k-mooney | poweroff instance just calls driver.power off | 16:33 |
| sean-k-mooney | what we proably shoudl do is remove the call to driver.destry in self._shutdown_instance | 16:34 |
| dasm | hey team. i'm trying to wrap my head around placement for nova. When VCPUs number are changed? When VM is stopped? Or only when deleted? I found this old thread: http://lists.openstack.org/pipermail/openstack/2014-June/008022.html but not sure if it's still valid. | 16:34 |
| sean-k-mooney | and have it call _power_off_isntace | 16:34 |
| artom | sean-k-mooney, yeah, maybe. One thing I don't like about my current (admitedly very quickly hacked up) thing is that we're leaking libvirt driver stuff into the compute manager | 16:35 |
| artom | Though I guess it *kinda* makes sense? | 16:35 |
| opendevreview | melanie witt proposed openstack/placement master: Call Engine.execute() in func tests for oslo.db 11.0.0 https://review.opendev.org/c/openstack/placement/+/808477 | 16:36 |
| artom | One _power_off_instance() is graceful shutdown, _shutdown_instance() pulls the plug. | 16:36 |
| artom | That's not really driver-specific... | 16:36 |
| sean-k-mooney | and they are reverst in libvirt too | 16:36 |
| sean-k-mooney | poweroff with virsh pulls the plug | 16:37 |
| sean-k-mooney | if i rememebr correctly | 16:37 |
| sean-k-mooney | i think you can just call power_off here https://github.com/openstack/nova/blob/e276184892250922c96c38415eabbaf7507f7e17/nova/compute/manager.py#L2897-L2898 | 16:37 |
| artom | y haev clear werd wen complixaeted werds maek tings moar fun | 16:37 |
| sean-k-mooney | becasue it will call driver.destroy if the gracefull approch does not work right | 16:37 |
| artom | I think so? _power_off_instance() does the timeout calc for us... | 16:39 |
| sean-k-mooney | yes | 16:40 |
| sean-k-mooney | https://github.com/openstack/nova/blob/e276184892250922c96c38415eabbaf7507f7e17/nova/compute/manager.py#L2833-L2850 | 16:40 |
| sean-k-mooney | huh apparently that is setabel in the image | 16:41 |
| artom | Fun surprises every day | 16:41 |
| sean-k-mooney | that is also not the correct way to get image properties | 16:41 |
| sean-k-mooney | i mean it works | 16:41 |
| sean-k-mooney | but we shoudl be using hte objects | 16:41 |
| sean-k-mooney | oh i see why its not useing the object | 16:42 |
| sean-k-mooney | its not in the object | 16:42 |
| sean-k-mooney | so image_os_shutdown_timeout or os_shutdown_timeout is not a legal image property | 16:43 |
| artom | At this point in the rabbit hole we're going to poke out in Australia | 16:43 |
| sean-k-mooney | this is just code form before we made the image props ovos | 16:44 |
| kashyap | artom: I politely "well-actuallyed" (I strongly avoid this) here on destroy() behaviour: https://review.opendev.org/c/openstack/nova/+/808474. Hope you don't mind. :) | 16:44 |
| sean-k-mooney | that was likely an extenion | 16:44 |
| kashyap | artom: But your broader goal is a good catch -- with volumes, i.e | 16:44 |
| * kashyap needs to head out; will check back in later. Have a good weekend! | 16:45 | |
| artom | kashyap, valid points | 16:45 |
| artom | Well, I guess you're heading out, will continue in the review | 16:45 |
| sean-k-mooney | artom: care to add a todo to remove supprot for image_os_shutdown_timeout or add it to the image proprties properly in the future | 16:46 |
| artom | sean-k-mooney, sure, just stick it in gerrit please | 16:46 |
| sean-k-mooney | artom: ok it should be all there | 16:57 |
| artom | Cheers! | 17:01 |
| melwitt | gibi: here's my patch for fixing the placement func test jobs https://review.opendev.org/c/openstack/placement/+/808477 | 17:40 |
| dansmith | melwitt: +2d that | 17:49 |
| dansmith | melwitt: did the other thing to separate the transactions merge? | 17:49 |
| melwitt | dansmith: no it's stuck behind this recent fail. I'm about to rebase onto ^ | 17:49 |
| dansmith | oh really, okay dang | 17:50 |
| melwitt | also thanks for the +2 | 17:50 |
| melwitt | yeah :( | 17:50 |
| dansmith | was going to say, we were still "recheck nova fail"-ing again yesterday | 17:50 |
| dansmith | who else is around to slam this trivial fix in I wonder? | 17:50 |
| melwitt | dansmith: yeah bauzas +W'ed earlier today (the transaction thing) | 17:51 |
| melwitt | oh maybe gmann | 17:51 |
| melwitt | gmann: are you around to review this gate fix for placement? https://review.opendev.org/c/openstack/placement/+/808477 | 17:52 |
| dansmith | ++ | 17:52 |
| gmann | melwitt: sure, checking | 17:52 |
| melwitt | \o/ | 17:53 |
| melwitt | hm actually I'm not sure whether I will need to rebase on the fix. I get confused about whether/when zuul does it automatically | 17:53 |
| gmann | melwitt: +A | 17:57 |
| melwitt | gmann: thank you!! :) | 17:57 |
| dansmith | melwitt: rebase.. rebase like the wind! | 17:58 |
| gmann | dansmith: melwitt as it is different file it should not require rebase, and zuul should merge it once fix is merged | 17:58 |
| gmann | we can try recheck once 808477 is merged | 17:58 |
| dansmith | gmann: there's another fix that depends on this | 17:58 |
| dansmith | rebase so we can make sure this lands before that | 17:59 |
| gmann | 807014 this one ? | 17:59 |
| dansmith | yeah | 18:00 |
| gmann | I think 807014 should pass with recheck once 808477 is merged | 18:01 |
| dansmith | right, | 18:01 |
| dansmith | but 807014 is breaking lots of *other* gates, so we want that in the queue ASAP | 18:02 |
| gmann | ohk. | 18:03 |
| gmann | or try re/another +w can trigger gate pipeline directly ? | 18:03 |
| gmann | but it might do check also. yeah rebase is more safer to put it queue asap | 18:05 |
| melwitt | gmann: it's this if you want to try https://review.opendev.org/c/openstack/placement/+/807014 | 18:05 |
| melwitt | ok, will do then | 18:05 |
| dansmith | I think rebasing is better, | 18:05 |
| dansmith | because if they get kicked out, we want them to stay in the right order for the next trip | 18:05 |
| gmann | ok | 18:05 |
| opendevreview | melanie witt proposed openstack/placement master: Narrow scope of set allocations database transaction https://review.opendev.org/c/openstack/placement/+/807014 | 18:06 |
| dansmith | melwitt: got it | 18:06 |
| melwitt | thanks! | 18:07 |
| opendevreview | Merged openstack/placement master: Call Engine.execute() in func tests for oslo.db 11.0.0 https://review.opendev.org/c/openstack/placement/+/808477 | 19:00 |
| admin1 | hi all .. is this s known error ? is this a known error coming up when launching new instances "Unexpected API Error. Please report this at http://bugs.launchpad.net/nova/ and attach the Nova API log if possible. <class 'amqp.exceptions.InternalError'> (HTTP 500) (Request-ID: req-17bb244e-46c6-4f2f-a213-3a1ae786afc2)" ? | 19:58 |
| melwitt | admin1: that means an unhandled exception bubbled up to the api. need to check the nova-api.log to see what happened | 20:03 |
| opendevreview | Merged openstack/placement master: Narrow scope of set allocations database transaction https://review.opendev.org/c/openstack/placement/+/807014 | 21:09 |
| opendevreview | Merged openstack/nova master: [doc] port-resource-request-groups not landed in Xena https://review.opendev.org/c/openstack/nova/+/807564 | 23:35 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!