| *** EugenMayer4 is now known as EugenMayer | 01:54 | |
| *** dasm|ruck is now known as dasm|ruck|ff | 04:19 | |
| *** dasm|ruck|ff is now known as dasm|ruck|off | 04:19 | |
| kashyap | gibi: Hi, the 'py310' and 'nova-next' failures seem nothing to do with my patch, yeah? - https://review.opendev.org/c/openstack/nova/+/838926 | 09:00 |
|---|---|---|
| gibi | py310 is definitely not, as it fails at setting up mysql | 09:01 |
| bauzas | wow, python 31.0 release number, time flies | 09:02 |
| bauzas | :) | 09:02 |
| gibi | the nova-next fails with some rsync issue so that is also not your patch | 09:02 |
| gibi | I'm not sure we have a tracking bug for the py310 issue but we should | 09:03 |
| gibi | the nova-next issue might be intermittent, but worth to check if it happens regurarly | 09:03 |
| gibi | bauzas: yeah, I push the problem out to time when py 31.0 will be released until then py310 is py 3.10 for me :D | 09:03 |
| kashyap | I see, yeah - I'll file a tracking bz for py310 'mysql' fail | 09:03 |
| gibi | kashyap: try to check if the same job fails for other repos too maybe there is also a tracking bug somewhere outside of nova | 09:04 |
| kashyap | Right, good point. I'll check before filing | 09:04 |
| kashyap | gibi: Hmm, I checked a random Neutron patch from today, and py310 (non-voting) seem to pass: https://review.opendev.org/c/openstack/neutron/+/840565 | 09:15 |
| gibi | interesting | 09:18 |
| gibi | then we probably need a nova tracking bug for this | 09:18 |
| * kashyap looks if it's failing for other Nova patches too | 09:23 | |
| sean-k-mooney | python 3.10 and eventlest are not partically happy currently. i dont think you can currenlty stack with devstack for example | 10:19 |
| sean-k-mooney | im not sure if we wil see that in unit/functional tests | 10:19 |
| sean-k-mooney | since we dont really use eventlets there the same way | 10:20 |
| sean-k-mooney | just an fyi | 10:20 |
| sean-k-mooney | deian are in the process of moving debian testing to 3.10 and have been having issue with this lately | 10:20 |
| kashyap | sean-k-mooney: Thanks; so it's not just me. | 10:22 |
| sean-k-mooney | kashyap: i think our unit test could pass but i know that there are some issue so i have not looked at your spcific failure but if its a devstack run i woudl expect it to fail currenlty | 10:24 |
| gibi | but the current failue is somewhere in the job setup as it fails on mysql :) | 10:41 |
| frickler | gibi: ah, these nice "unit" tests with external dependencies. py310 currently runs on f35, maybe your setup is ubuntu specific? | 10:50 |
| gibi | frickler: yeah that seems to be the issue | 10:51 |
| frickler | devstack on 22.04 runs the base services mostly without issues except for horizon | 10:51 |
| frickler | https://review.opendev.org/q/topic:add-jammy2 | 10:51 |
| ralonsoh | sean-k-mooney, hi! I'm playing again with live-migration and ceph | 11:26 |
| ralonsoh | and, of course, during the compute installation I had this | 11:26 |
| ralonsoh | 2022-05-05T11:18:22.104+0000 7fcd2a59c700 -1 monclient(hunting): handle_auth_bad_method server allowed_methods [2] but i only support [2] | 11:26 |
| ralonsoh | I've copied the ceph ID from the controller (that is also a compute) | 11:26 |
| ralonsoh | sean-k-mooney, never mind! what I was missing was the keyring files | 11:28 |
| ralonsoh | I've copied them and now it works heheheh | 11:28 |
| sean-k-mooney | cool | 11:31 |
| opendevreview | Merged openstack/nova master: Fix segment-aware scheduling permissions error https://review.opendev.org/c/openstack/nova/+/839361 | 11:32 |
| sean-k-mooney | by the way you might find https://github.com/SeanMooney/ansible_role_devstack interesting/helpful | 11:32 |
| sean-k-mooney | the ceph support curerntly only works on ubuntu since the devstack pluging currently does not have c9s support | 11:33 |
| *** dasm|ruck|off is now known as dasm | 13:07 | |
| *** dasm is now known as dasm|ruck | 13:08 | |
| sean-k-mooney | dansmith: by the way did anything come of the excessivly high keystone and neutron load on the db | 13:16 |
| sean-k-mooney | from your perfoamcne data patches | 13:16 |
| opendevreview | Merged openstack/nova master: Allow claiming PCI PF if child VF is unavailable https://review.opendev.org/c/openstack/nova/+/838555 | 13:34 |
| dansmith | sean-k-mooney: I brought it up to the keystone people a couple days ago, and it was agreed that it was too high, but nothing more than that (yet) | 14:03 |
| sean-k-mooney | ack so they did not see any smoking gun or anything else obviously wrong | 14:04 |
| dansmith | well, nothing obvious off the top of their head, I dunno that anyone has done any real digging yet | 14:11 |
| ralonsoh | sean-k-mooney, sorry again. Did you see something like this https://paste.opendev.org/show/bEMWSLCst6bhd4Ir3zwl/? | 14:41 |
| ralonsoh | one of the nova-compute services is exiting | 14:42 |
| ralonsoh | in a non very elegant way | 14:42 |
| sean-k-mooney | ew segfault | 14:42 |
| ralonsoh | this is the controller-compute node | 14:42 |
| sean-k-mooney | no that is new to me | 14:42 |
| ralonsoh | ok | 14:42 |
| ralonsoh | I'll try to re install everything | 14:43 |
| sean-k-mooney | where was that thrown | 14:43 |
| ralonsoh | during, of course, the live migration | 14:43 |
| sean-k-mooney | was it ci or a local vm/server | 14:43 |
| ralonsoh | when I'm bringing the VM to this host | 14:43 |
| ralonsoh | when the VM evacuates this host, all is OK | 14:43 |
| ralonsoh | no sorry, it is when the VM evacuates the host | 14:43 |
| sean-k-mooney | evacuate or live migrate | 14:44 |
| sean-k-mooney | they are very differnt things | 14:44 |
| ralonsoh | yeah sorry | 14:44 |
| ralonsoh | live-migrate | 14:44 |
| ralonsoh | when the VM is leaving the host | 14:44 |
| sean-k-mooney | so you live migrate and then the source host segfaults? | 14:44 |
| sean-k-mooney | are there any OOM errors | 14:44 |
| sean-k-mooney | or other detail | 14:45 |
| ralonsoh | let me check | 14:45 |
| ralonsoh | nope, I have still 13GB free | 14:45 |
| sean-k-mooney | like the segfault appears to by in python but im wondering if an allcoation failed because you ran out of memory | 14:45 |
| sean-k-mooney | ok | 14:45 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684 | 14:52 |
| gibi | frickler: I try to unblock openstack-tox-py310 job | 14:52 |
| gibi | with ^^ | 14:52 |
| kashyap | gibi: Thank you! | 14:56 |
| opendevreview | Rico Lin proposed openstack/nova-specs master: Add vIOMMU device support for libvirt driver https://review.opendev.org/c/openstack/nova-specs/+/840310 | 14:58 |
| opendevreview | Takashi Natsume proposed openstack/python-novaclient master: Replace old URLs with new ones https://review.opendev.org/c/openstack/python-novaclient/+/840693 | 15:59 |
| gibi | frickler, kashyap: my google foo failed me to figure out why the mysql password change fails on fedora 35 in openstack-tox-py310 So if you have ideas please shoot https://review.opendev.org/c/openstack/nova/+/840684/1#message-ffd5ac00ca235cfaebd12988b65dbc210c2b9ec8 | 16:10 |
| clarkb | gibi: seems like your mysqladmin tool isn't compatible with mariadb | 16:15 |
| clarkb | since it is generating the sql that fails | 16:15 |
| mnaser | hrm | 16:20 |
| gibi | hm, mariadb is on version 10.5 but mysqlclient and mysqladmin is on 8.0.28 but I'm not sure how to map these verison | 16:20 |
| mnaser | i've got a really weird situation where a hypervisor stops getting vms scheduled to it | 16:21 |
| mnaser | i checked `openstack resource provider inventory list 9fe525d9-df51-41c2-8ca7-8344ed0eee39` and that shows the resources available, and `openstack allocation candidate list --resource VCPU=4 --resource DISK_GB=64 --resource MEMORY_MB=2048 | grep 9fe525d9-df51-41c2-8ca7-8344ed0eee39` even returns that | 16:21 |
| mnaser | so its not placement | 16:21 |
| clarkb | gibi: mysqlclient and mysqladmin are probably the mysql tools and not the mariadb tools? Possible that mariadb has alternatives | 16:21 |
| gibi | clarkb: yeah that make sense... try to figure out where are those alternatives | 16:22 |
| mnaser | the filters in use are: "ComputeFilter, AggregateTypeAffinityFilter, ComputeCapabilitiesFilter, PciPassthroughFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter" -- i dont think the rest are relevant in this scenario | 16:22 |
| mnaser | hrm, a bunch of exceptions with relation to libvirt before it fully stopped to deploy new systems | 16:24 |
| mnaser | https://www.irccloud.com/pastebin/n7Ezs7aB/ | 16:24 |
| mnaser | and if i try to provision an instance on them manually (by using `--host` .. it goes up fine) | 16:25 |
| mnaser | and now that i've actually provisioned an instance, the vms have started to flow in the hyperivsor agian | 16:26 |
| clarkb | gibi: side note: https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/840545 | 16:26 |
| gibi | clarkb: yeah that can be a way out :) | 16:27 |
| opendevreview | melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701 | 16:28 |
| opendevreview | melanie witt proposed openstack/placement stable/victoria: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840702 | 16:29 |
| opendevreview | melanie witt proposed openstack/placement stable/ussuri: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840703 | 16:30 |
| opendevreview | melanie witt proposed openstack/placement stable/train: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840704 | 16:30 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684 | 16:40 |
| sean-k-mooney | oh we have 22.04 in nodepool now | 16:45 |
| sean-k-mooney | cool | 16:45 |
| sean-k-mooney | i think the mysql/mariadb changes were also in devstack | 16:45 |
| sean-k-mooney | we use mariadb on most distos i think now | 16:45 |
| clarkb | sean-k-mooney: it is a bit of a slwo rollout while we work through various things, but ya the images are up and mostly work. The last thing we ran into was phased package updates not making sense for us | 16:48 |
| sean-k-mooney | phased package updates? | 16:49 |
| sean-k-mooney | as in replication to mirrors or something else | 16:49 |
| clarkb | something else. Its new functionality in apt that hashes something about your host and then modulo's that against the percentage of users they want to install the package | 16:50 |
| clarkb | which means they can say things like 10% of users get this package update. Then next week change it to 50% and so on until it is 100% | 16:51 |
| clarkb | but reprepro doesn't understand it and it is disabled by default in chroots (which dib uses to make the images) which means you get the latest available packges as if phases didn't exist at all | 16:51 |
| Uggla | sean-k-mooney, can you have a look at my comment on https://review.opendev.org/c/openstack/nova-specs/+/831506/2/specs/zed/approved/unshelve-to-host.rst#42 ? Please let me know what you think about it. | 16:57 |
| sean-k-mooney | clarkb: oh ok | 16:59 |
| sean-k-mooney | clarkb: ya i think we would want to turn that off | 16:59 |
| mnaser | ok, this is most def a thread leak | 16:59 |
| sean-k-mooney | Uggla i saw that breifly | 17:00 |
| mnaser | kill -USR2 <nova-pid> | 17:00 |
| sean-k-mooney | Uggla: so for unshelve ot az it ends up updating the request spec to the requested az | 17:00 |
| mnaser | grep -i 'Green Thread' /tmp/gar | wc -l => 6094 | 17:00 |
| sean-k-mooney | mnaser: those are the userland trhead not real os thread | 17:01 |
| mnaser | a lot of them are this: | 17:01 |
| mnaser | https://www.irccloud.com/pastebin/YBiYPlrp/ | 17:01 |
| mnaser | and i think the root cause is from https://www.irccloud.com/pastebin/n7Ezs7aB/ | 17:01 |
| sean-k-mooney | Uggla: so for unshelve ot host i think it shoudl also update it to the AZ of the host if and only if the request spec is not none | 17:02 |
| mnaser | so with every failure of this, we'd get an extra greenthread that sits and does nothing | 17:02 |
| sean-k-mooney | Uggla: i need to think about that and make sure that is right | 17:02 |
| sean-k-mooney | Uggla: bug basically if the vm orgringaly requested an AZ we want to update it to match the az of the host | 17:03 |
| sean-k-mooney | Uggla: if it did not then we do not want to update it | 17:03 |
| sean-k-mooney | Uggla: i think | 17:03 |
| sean-k-mooney | mnaser: well that implies that nothing is catching the vif creation failure? | 17:04 |
| sean-k-mooney | or its never been resumed | 17:04 |
| mnaser | sean-k-mooney: maybe.. i've seen similar behaviour here https://github.com/eventlet/eventlet/issues/432 and https://github.com/eventlet/eventlet/issues/662 | 17:05 |
| sean-k-mooney | im not really sure why you would end up with multiple dangeling thread like that | 17:05 |
| mnaser | its supposed to log a warning if it hits that exception | 17:06 |
| mnaser | let me see https://github.com/openstack/nova/blob/stable/wallaby/nova/virt/libvirt/driver.py#L7236-L7245 is logged | 17:06 |
| sean-k-mooney | yes which would do io and cause the thread to yeild | 17:07 |
| sean-k-mooney | *greenthread | 17:07 |
| sean-k-mooney | or at least potentially while the python logger processes the logging event | 17:07 |
| sean-k-mooney | mnaser: that looks promising and also depressing | 17:09 |
| sean-k-mooney | i.e. that python logging is broken and has been for ever | 17:10 |
| mnaser | sean-k-mooney: sadly it looks like the logs got rotated out :( | 17:10 |
| *** haleyb_ is now known as haleyb | 17:10 | |
| mnaser | with the kill -USR2 it wiped a bunch of the old logs | 17:10 |
| sean-k-mooney | are you using oslo.log's logrotation feature | 17:10 |
| sean-k-mooney | or using logrotate externally | 17:11 |
| mnaser | no, this is bc we run stuff inside k8s, so the max-log-size feature or whatever its called i believe hit here | 17:11 |
| sean-k-mooney | kill -USR2 usually requires a process restart to recover form | 17:11 |
| mnaser | to me it sounds like there should not be a traceback for this thing to start with | 17:11 |
| sean-k-mooney | ack so dumping the GMR proably caused the pod to be restarted | 17:11 |
| mnaser | this is wallaby blergh | 17:13 |
| Uggla | sean-k-mooney, ok by the way we (with Artom) added some tests to ensure a cold migration after shelve/unshelve to host is moving back the host to the origin host. | 17:14 |
| Uggla | sean-k-mooney, but we can discuss that tomorrow, it will let you think about it. | 17:15 |
| mnaser | ok it looks like timeout was raised, then since `vif_plugging_is_fatal`, that raises another exception again, which bubbles back up to the `except Exception` | 17:15 |
| mnaser | https://github.com/openstack/nova/blob/stable/wallaby/nova/virt/libvirt/driver.py#L7235-L7266 | 17:15 |
| mnaser | then we LOG.error() the whole stack, which seems to add up | 17:16 |
| sean-k-mooney | so its being rasised form here https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7499= | 17:17 |
| sean-k-mooney | so we plug the start a timer to with for the viff plugged event then we plug the vifs then we creat the guest | 17:17 |
| sean-k-mooney | we start waiting in this context manager https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L7491-L7494= | 17:17 |
| sean-k-mooney | that is timeing out after 300 seconds | 17:17 |
| sean-k-mooney | causign this expct block to be taken | 17:17 |
| mnaser | https://eventlet.net/doc/modules/timeout.html | 17:19 |
| mnaser | "If the code block catches and doesn’t re-raise BaseException (for example, with except:), then it will catch the Timeout exception, and might not abort as intended." | 17:19 |
| sean-k-mooney | if you have logs for this in the future you could proably check fo rthis log https://github.com/openstack/nova/blob/7520711a0e3b20354c0a9d46cb1dd62c8f56db24/nova/compute/manager.py#L559-L569= | 17:19 |
| sean-k-mooney | mnaser: i dont think we are incorectly cathching this | 17:20 |
| sean-k-mooney | hum https://github.com/openstack/nova/blob/7520711a0e3b20354c0a9d46cb1dd62c8f56db24/nova/compute/manager.py#L2202-L2242= | 17:24 |
| sean-k-mooney | we are not using a threadpool here so this should be fine | 17:25 |
| mnaser | i believ this issue surfaces if you have a lot of port plugging timeouts, which is why it might not be noticed | 17:25 |
| mnaser | (i could solve this by figuring out why the ports are all timing out being plugged, since that is the cause, but still seems to be an issue) | 17:26 |
| mnaser | i wonder if it can be reproduced by spinning up a bunch of instances with n-ovs-agent shut down for example | 17:26 |
| sean-k-mooney | but it proably woudl be simpler to reproduce in teh func test | 17:27 |
| sean-k-mooney | just mock out the part of the nueton fixture that sends the event | 17:27 |
| sean-k-mooney | what i dont get is why is the thread not being resumed | 17:27 |
| sean-k-mooney | it raising the excption (either timeour or vif plugged) but nothign is resuming the greenthreadn and its just waiting forever | 17:28 |
| sean-k-mooney | i wonder if this because we use spawn_n | 17:28 |
| sean-k-mooney | instead of spawn | 17:28 |
| sean-k-mooney | mnaser: gibi mentioned something abtou this in a differnt context https://review.opendev.org/c/openstack/nova/+/825015/4/nova/healthcheck/manager.py#214= | 17:30 |
| sean-k-mooney | maybe we shoudl be useing spawn here instead https://github.com/openstack/nova/blob/7520711a0e3b20354c0a9d46cb1dd62c8f56db24/nova/compute/manager.py#L2242-L2247 | 17:32 |
| sean-k-mooney | mnaser: this is hte comment that gibi was intening i read i think | 17:35 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/813114/1..4//COMMIT_MSG#b19= | 17:35 |
| sean-k-mooney | https://github.com/eventlet/eventlet/issues/731#issuecomment-953761883 | 17:35 |
| kashyap | gibi: I need to head out, but I see that you've got a new rev up w/ a new package removals. And Zuul seems to be happy. Will check tomm :) | 17:35 |
| sean-k-mooney | melwitt: you might also have context on mnaser greenthread leaking issue | 17:38 |
| sean-k-mooney | melwitt: is there a reason that you are aware of to not remvoe all uses of spawn_n and replace them with spawn? | 17:38 |
| sean-k-mooney | we likely woudl have to add https://github.com/eventlet/eventlet/issues/731#issuecomment-953761883 to our monkey patch file | 17:40 |
| sean-k-mooney | to cater for the thread.start case you noted | 17:40 |
| melwitt | sean-k-mooney: no, temoto recommended not to use spawn_n | 17:40 |
| sean-k-mooney | yep reading the bug that is clear | 17:41 |
| sean-k-mooney | so we coudl replace all the calls and also do | 17:41 |
| melwitt | I did do a patch to change them all but we still had some come up bc for example oslo.messaging uses spawn_n when in eventlet mode | 17:41 |
| sean-k-mooney | import eventlet | 17:41 |
| sean-k-mooney | eventlet.spawn_n = eventlet.spawn | 17:41 |
| sean-k-mooney | eventlet.convenient.spawn_n = eventlet.spawn | 17:41 |
| sean-k-mooney | eventlet.monkey_patch() | 17:41 |
| sean-k-mooney | melwitt: right but if we gloablly repoalce spawn_n with spawn | 17:42 |
| sean-k-mooney | when we monkey patch | 17:42 |
| sean-k-mooney | the oslo will also start using spwan | 17:42 |
| sean-k-mooney | although we probly shoudl fix it in oslo too | 17:42 |
| melwitt | ah, true. I was thinking of when I tried literally changing them all + a hacking rule | 17:42 |
| melwitt | +1 | 17:42 |
| sean-k-mooney | did you ever raise that with oslo? | 17:43 |
| sean-k-mooney | removing there use of spawn_n | 17:43 |
| melwitt | no. at the time I was so mired in trying to sort the eventlet issue that I didn't think to initiate that | 17:43 |
| sean-k-mooney | ack | 17:44 |
| sean-k-mooney | it looks like spawn_n is pretty common in openstack | 17:45 |
| sean-k-mooney | https://codesearch.opendev.org/?q=spawn_n&i=nope&literal=nope&files=&excludeFiles=&repos= | 17:45 |
| melwitt | I had also intended to put together a more concise repro of the spawn_n problematic behavior to do another eventlet issue but lost steam | 17:45 |
| melwitt | yeah I was thinking that might be the case. I probably checked back then and then forgot | 17:45 |
| sean-k-mooney | so this could be a redherring as to why nothing seams to be resuming the greenthread after the excption is raised when there i s a netowk vif plugged | 17:47 |
| sean-k-mooney | event that fails | 17:47 |
| mnaser | so probably the spawn_n here is causing this somehow ? | 17:47 |
| sean-k-mooney | mnaser: thats what im specualting but not sure | 17:47 |
| sean-k-mooney | im wondiering if we just spawned a fucntion taht raise woudl that trigger this | 17:48 |
| sean-k-mooney | anyway i need to go fo today but it might be worth tryign to repoduce this outside fo nova with a simpel script that use eventles swan_n and raises | 17:50 |
| sean-k-mooney | https://eventlet.net/doc/basic_usage.html#eventlet.spawn_n | 17:50 |
| mnaser | yeah.. I don’t know if I have spare cycles right now for that but I think I should probably file a bug in launchpad at least | 17:50 |
| sean-k-mooney | The same as spawn(), but it’s not possible to know how the function terminated (i.e. no return value or exceptions). This makes execution faster. See spawn_n for more details. | 17:50 |
| sean-k-mooney | https://eventlet.net/doc/modules/greenthread.html#eventlet.greenthread.spawn_n | 17:51 |
| melwitt | yeah, I'll try to look into this too | 17:51 |
| sean-k-mooney | If an exception is raised in the function, spawn_n prints a stack trace; the print can be disabled by calling eventlet.debug.hub_exceptions() with False. | 17:51 |
| *** artom__ is now known as artom | 17:52 | |
| sean-k-mooney | so ya i think this is a high likely hood that this is the probelm | 17:52 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Adapt tools/test-setup to Fedora 35 https://review.opendev.org/c/openstack/nova/+/840684 | 17:52 |
| sean-k-mooney | if this was c or c++ spwan_n woudl be annotated with [[noreturn]] | 17:53 |
| sean-k-mooney | ok got to go but let me know if this helps | 17:53 |
| mnaser | sean-k-mooney, melwitt: https://bugs.launchpad.net/nova/+bug/1971760 | 18:27 |
| melwitt | great, thanks | 18:28 |
| opendevreview | melanie witt proposed openstack/nova stable/wallaby: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840717 | 18:34 |
| opendevreview | melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701 | 18:36 |
| opendevreview | melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701 | 18:38 |
| opendevreview | melanie witt proposed openstack/placement stable/wallaby: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840718 | 18:38 |
| mnaser | melwitt, sean-k-mooney: found this https://github.com/openstack/nova/blob/0190d585418f088728533334872820689642a9e3/nova/compute/manager.py#L479 which goes to https://github.com/openstack/nova/blob/0190d585418f088728533334872820689642a9e3/nova/network/model.py#L623 | 19:23 |
| mnaser | which references https://github.com/openstack/nova/blob/0190d585418f088728533334872820689642a9e3/nova/network/model.py#L590 | 19:23 |
| mnaser | so .spawn_n which calls .spawn | 19:23 |
| melwitt | mnaser: I don't see any spawn_n there, or is that what you're pointing out? | 19:28 |
| mnaser | melwitt: oh right, so _locked_do_build_and_run_instance calls spawn().. and down the line that ends up in wait_for_instance_event() which calls .wait() for the event | 19:30 |
| mnaser | i mean, i am trying to create a reproducer but not really able to get something to fail :( | 19:31 |
| mnaser | https://paste.opendev.org/show/b2msfzi2ASYNwN0Fguqp/ | 19:32 |
| mnaser | then kill -USR2 <pid> and no bueno, i dont see those extra threads | 19:32 |
| melwitt | hm ok | 19:35 |
| mnaser | im trying to play with it to reproduce but there's a lot in play i think | 19:35 |
| melwitt | for sure | 19:36 |
| opendevreview | melanie witt proposed openstack/placement stable/yoga: Drop lower-constraints.txt and its testing https://review.opendev.org/c/openstack/placement/+/840728 | 19:41 |
| mnaser | i got a full log of a failure | 19:49 |
| mnaser | https://paste.opendev.org/show/bbd1luUAF8B0rwZnl4Cw/ | 19:50 |
| opendevreview | melanie witt proposed openstack/placement stable/xena: Drop lower-constraints.txt and its testing https://review.opendev.org/c/openstack/placement/+/840756 | 20:32 |
| opendevreview | melanie witt proposed openstack/placement stable/wallaby: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840718 | 21:30 |
| opendevreview | melanie witt proposed openstack/placement stable/wallaby: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840701 | 21:30 |
| opendevreview | melanie witt proposed openstack/nova stable/victoria: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840765 | 21:59 |
| opendevreview | melanie witt proposed openstack/placement stable/victoria: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840702 | 22:04 |
| opendevreview | melanie witt proposed openstack/placement stable/victoria: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840767 | 22:04 |
| opendevreview | melanie witt proposed openstack/nova stable/ussuri: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840771 | 22:07 |
| opendevreview | melanie witt proposed openstack/placement stable/victoria: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840767 | 22:08 |
| opendevreview | melanie witt proposed openstack/placement stable/victoria: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840702 | 22:08 |
| opendevreview | melanie witt proposed openstack/placement stable/ussuri: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840703 | 22:10 |
| opendevreview | melanie witt proposed openstack/placement stable/ussuri: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840773 | 22:11 |
| opendevreview | melanie witt proposed openstack/nova stable/train: Define new functional test tox env for placement gate to run https://review.opendev.org/c/openstack/nova/+/840777 | 22:24 |
| opendevreview | melanie witt proposed openstack/placement stable/train: placement-status: check only consumers in allocation table https://review.opendev.org/c/openstack/placement/+/840704 | 22:25 |
| opendevreview | melanie witt proposed openstack/placement stable/train: Use 'functional-without-sample-db-tests' tox env for placement nova job https://review.opendev.org/c/openstack/placement/+/840778 | 22:25 |
| melwitt | elodilles: just fyi, tried to backport a placement fix to stable branches, found gates broken, so I rebased them onto fixes for the gate ^ (assuming I didn't mess up). and the gate fixes Depends-On nova stable branch changes that I have also posted | 22:29 |
| *** dasm|ruck is now known as dasm|off | 22:55 | |
| gmann | stephenfin: good point on release note of dropping the py3.6/3.7 support in oslo patches, do you think we should have added releasenotes in nova in this thttps://review.opendev.org/c/openstack/nova/+/838943 | 23:13 |
| gmann | stephenfin: I can add one | 23:13 |
| opendevreview | Ghanshyam proposed openstack/nova master: Add releasenote about dropping pythin 3.6|7 support https://review.opendev.org/c/openstack/nova/+/840786 | 23:39 |
| gmann | stephenfin: ^^ | 23:40 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!