| opendevreview | Merged openstack/nova master: Update contributor guide for 2025.1 Epoxy https://review.opendev.org/c/openstack/nova/+/929918 | 10:16 |
|---|---|---|
| opendevreview | Amit Uniyal proposed openstack/nova master: Update Nova bdm with updated swap info https://review.opendev.org/c/openstack/nova/+/929858 | 11:12 |
| opendevreview | Amit Uniyal proposed openstack/nova master: Update Nova bdm with updated swap info https://review.opendev.org/c/openstack/nova/+/929858 | 11:18 |
| opendevreview | Takashi Kajinami proposed openstack/nova master: Remove libvirt tunnelled migration https://review.opendev.org/c/openstack/nova/+/879021 | 11:26 |
| opendevreview | Merged openstack/nova master: Imported Translations from Zanata https://review.opendev.org/c/openstack/nova/+/929719 | 11:44 |
| opendevreview | Merged openstack/python-novaclient master: tox: Drop envdir https://review.opendev.org/c/openstack/python-novaclient/+/932263 | 11:46 |
| opendevreview | Takashi Kajinami proposed openstack/placement master: Drop db migration tool https://review.opendev.org/c/openstack/placement/+/932324 | 11:59 |
| tkajinam | ^^^ I wonder if we can drop the migration script directly or we need deprecation step (I feel like no one may use it from master so direct removal doesn't annoy real users) | 12:02 |
| sean-k-mooney | tkajinam: i assuem we have a placement-manage db_sync command so it shoudl not be in use but im not sure. i have never checked that | 12:22 |
| sean-k-mooney | oh | 12:23 |
| sean-k-mooney | this is the tool to migrate form in nova placment to split out placemnet | 12:23 |
| sean-k-mooney | that a good question we have not requried this for so long that enven the unmaintaiend brances shoudl not need it | 12:23 |
| sean-k-mooney | so i feel like a direct removal would be ok | 12:24 |
| sean-k-mooney | we are talking about somethign that shoudl have been done when upgrading to stien so i dont think deprecation of a shel script really shoudl be required at this point | 12:24 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Revert "Test live migration between hosts with differnet cpu_shared_sets" https://review.opendev.org/c/openstack/nova/+/932327 | 12:42 |
| gibi | sean-k-mooney: ^^ revert to stabilize CI | 12:43 |
| sean-k-mooney | gibi: ah yep thanks i ment to do that for two weeks now and have been putting it off | 12:43 |
| sean-k-mooney | it fiail infrequetly but it will increase as more raxflex host come online | 12:43 |
| gibi | I got hit by it twice in a row so got enough motivation to at least push the revert | 12:43 |
| sean-k-mooney | do we need another to +w i added +2 | 12:44 |
| gibi | bauzas: could you put a +w to ^^ | 12:44 |
| gibi | it is a revert for CI stablization | 12:44 |
| bauzas | okay | 12:44 |
| sean-k-mooney | ill be changign which jobs to what anyway when i get time to work on this again | 12:45 |
| gibi | thanks | 12:45 |
| tkajinam | sean-k-mooney, yeah that's the tool to pull placement tables from old nova db. | 12:55 |
| *** haleyb is now known as haleyb|out | 16:42 | |
| opendevreview | Merged openstack/nova master: Revert "Test live migration between hosts with differnet cpu_shared_sets" https://review.opendev.org/c/openstack/nova/+/932327 | 17:19 |
| melwitt | sean-k-mooney: you had mentioned that scatter_gather_cells is not currently waiting for timeout in the main thread but I thought it is? this is using the eventlet.timeout.Timeout context manager in the main thread right? https://github.com/openstack/nova/blob/741e1b38705efecc083f091c51164b3cafb9697c/nova/context.py#L437 | 18:50 |
| dansmith | melwitt: I think in uwsgi mode it can't since we're not running the eventlet hub right? | 19:07 |
| dansmith | IIRC that's what sean-k-mooney was referring to.. it "waits" but the timeout will never fire in fully uwsgi mode | 19:08 |
| melwitt | dansmith: oh, ok I didn't realize that was why | 19:08 |
| melwitt | I see. thanks | 19:08 |
| sean-k-mooney[m] | it was related to a comment gibi made | 22:11 |
| sean-k-mooney | ... peopel changed my topic | 22:14 |
| sean-k-mooney | melwitt: i was refering to this comment thread https://review.opendev.org/c/openstack/nova/+/905284/comment/ea51fb75_6013c0cd/ | 22:15 |
| sean-k-mooney | im pretty sure that the entire query_wrapper can go away and the timeout logic can move as we only use it for one thing | 22:15 |
| sean-k-mooney | its been a month since i looke at it but i think i was leaning towards puting the timeout here https://github.com/openstack/nova/blob/01b207e50d307a7e7650b5839535fbd3ad40bc1b/nova/context.py#L421 or https://github.com/openstack/nova/blob/01b207e50d307a7e7650b5839535fbd3ad40bc1b/nova/compute/multi_cell_list.py#L262 | 22:17 |
| melwitt | sean-k-mooney: I remember that thread but I didn't realize that today the timeout doesn't work because of no eventlet hub in full uwsgi mode | 22:17 |
| sean-k-mooney | im not sure if that is true or not | 22:18 |
| sean-k-mooney | when run under uwsig today we are monkey patching so i think we have a hub but honestly i have not looked at that in a long time | 22:19 |
| sean-k-mooney | we were not starting the hub at one point | 22:19 |
| sean-k-mooney | but i think matt booth changed that as a bugfix? | 22:19 |
| sean-k-mooney | https://github.com/openstack/nova/commit/3c5e2b0e9fac985294a949852bb8c83d4ed77e04 | 22:20 |
| melwitt | ok well ... if the timeout context manager works then it seems to me the timeout is in the right place in today's code (in the main thread and kills the child threads if the timeout fires). and I'm not making an opinion about where the timeout should go as part of eventlet removal | 22:21 |
| sean-k-mooney | so i htink the issue is the timeout fires on the main thread but does not kill the child threads | 22:22 |
| sean-k-mooney | again i dont currently have the context loaded form gibi's testing but i blieve we need the timeout to fire on the child thread? | 22:23 |
| melwitt | I thought gibi was testing with native threads not eventlet? | 22:23 |
| sean-k-mooney | yes but with this patch we are using native treads too | 22:24 |
| melwitt | yeah, gibi's comment is about the code proposed in your patch | 22:24 |
| sean-k-mooney | right that is what i was brignin up today | 22:24 |
| melwitt | ok, maybe I misunderstood what you said. I thought you said the timeout doesn't work at all today | 22:24 |
| sean-k-mooney | when we remove eventelte we need to ensure the timeout still works for the treads in the tread pool | 22:25 |
| sean-k-mooney | baisicly what we want to have happen is if the timout expires we want the thread to be killed and clean up by the timeout expction when resumed by the kernel | 22:26 |
| melwitt | right. I had thought you were commenting on what's in tree right now. I misunderstood that you were talking about future stuff, my bad | 22:26 |
| sean-k-mooney | by startign the timer in the child thread if we dont get io before it expires then the kernel shoud resume it whtn the timer interupt fires | 22:26 |
| sean-k-mooney | no worries, ya i was talking about the unresolved comments in the current series | 22:27 |
| melwitt | I agree the timer needs to be in the child thread with native threads | 22:27 |
| melwitt | we don't have the ability anyway to kill them from the parent | 22:27 |
| melwitt | *them = native threads | 22:27 |
| melwitt | ok, gotcha. sorry about that | 22:28 |
| sean-k-mooney | right fi we used processes we could but because native thread could be holding locks (even if this is not) it would be unsafe to kill them form the parent even if we coudl do that without hacks | 22:28 |
| melwitt | right | 22:29 |
| sean-k-mooney | the other thing i brought up is https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L169-L172 | 22:30 |
| sean-k-mooney | the nova/cmd moduels for the proxy dont call eventlet but we do import that moduel here https://review.opendev.org/c/openstack/nova/+/904424/13/nova/cmd/standalone/baseproxy.py#29 | 22:31 |
| melwitt | ah, yeah. that one I haven't known what it's for, none of the work I did on the proxy was related to it | 22:32 |
| sean-k-mooney | so it was added by https://github.com/openstack/nova/commit/cb25bc4530323aaa33d5c42eb01f998d463f2106 | 22:32 |
| sean-k-mooney | which is apprently for https://bugs.launchpad.net/nova/+bug/1193031 | 22:33 |
| melwitt | oh, huh. | 22:33 |
| sean-k-mooney | The solution is to either not monkey_patch for /cmd/nova-novncproxy and do the rpc call that validates the token with the consolauth service in a separate process that is monkey_patched (the rpc won't work without it), or to make openstack/common/rpc/amqp/multicall not use green-threads for this particular call (ie. block the caller). | 22:34 |
| sean-k-mooney | so consolauth is dead | 22:34 |
| sean-k-mooney | and we are trying to not monkeypatch | 22:34 |
| sean-k-mooney | so maybe we just delete that and we are good? | 22:34 |
| melwitt | delete all of consoleauth? sure, what could go wrong | 22:35 |
| sean-k-mooney | i ment the hub | 22:36 |
| sean-k-mooney | but also didnt you remove the console auth service | 22:36 |
| sean-k-mooney | we now do auth in the cell db | 22:36 |
| melwitt | oh, the console auth service, yeah that is no longer used | 22:36 |
| melwitt | for some reason I was thinking the db call but yeah that doesn't use rpc | 22:37 |
| sean-k-mooney | well even if it did we could deal with this differently | 22:38 |
| melwitt | oh wait, there is an rpc call for "validating the console port | 22:38 |
| melwitt | https://github.com/openstack/nova/blob/741e1b38705efecc083f091c51164b3cafb9697c/nova/console/websocketproxy.py#L133 | 22:39 |
| melwitt | that's the only one AFAICT | 22:40 |
| sean-k-mooney | so we are changing the oslo messaging executor to threading | 22:40 |
| sean-k-mooney | so im not sure it that will be a problem or not but ok this is somethign we need to test | 22:40 |
| melwitt | ok, yeah. I would think it should work fine if you delete the hub at the same time as the switch to native threads in oslo.messaging. I agree, definitely something to test | 22:42 |
| sean-k-mooney | that is tomorrows problem o/ | 22:43 |
| melwitt | yes o/ | 22:44 |
| sean-k-mooney[m] | so i reinstalled my home openstack and entirly redidi my sotrage to use nova lvm instead of cinder lvm, most of the speedu is form other change but i wend form about an 80 minute devstack install to about 21 minutes | 23:14 |
| sean-k-mooney[m] | im hoping with a liitlme more turning to shave another miniute or so but thats closer to the speed i would expect on this host | 23:15 |
| sean-k-mooney[m] | i used to reliable be able to get a fresh vm deployed in about 18-20 mins so that pretty close already | 23:15 |
| sean-k-mooney[m] | although there isnt much standing out as a bottle neck to be honest although i think i can speed up the disk io slightly more. beyond that i would likely need a local mirror/cache but the install time for package is already pretty good https://paste.opendev.org/show/825791/ | 23:19 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!