| *** brinzhang_ is now known as brinzhang | 05:35 | |
| *** abhishekk is now known as akekane|home | 05:50 | |
| *** akekane|home is now known as abhishekk | 05:50 | |
| gibi | good morning | 08:57 |
|---|---|---|
| lyarwood | Morning all \o | 09:03 |
| gibi | lyarwood: there is a fairly easy bug repro and fix for placement that I'd like to land before RC1. melwitt already +2d it. So if you have time I would appreciate some review https://review.opendev.org/q/topic:story-2009167 | 09:13 |
| lyarwood | ack I'll look this morning | 09:13 |
| gibi | thanks | 09:15 |
| gibi | it is just my devstack or we broke $nova-manage db version ? | 09:39 |
| sean-k-mooney | it now uses alembic | 09:40 |
| sean-k-mooney | so its proably changed | 09:40 |
| gibi | https://paste.opendev.org/show/809276/ | 09:41 |
| sean-k-mooney | that is not what i was expecting | 09:42 |
| gibi | yeah | 09:42 |
| gibi | me neither | 09:42 |
| lyarwood | oh fun | 09:42 |
| sean-k-mooney | Engine' object has no attribute 'get_main_option' | 09:43 |
| sean-k-mooney | that should be a config object | 09:43 |
| sean-k-mooney | not an engine right | 09:43 |
| sean-k-mooney | oh its from alembic_version = alembic_api.current(engine) | 09:43 |
| sean-k-mooney | not script_location = config.get_main_option("script_location") | 09:43 |
| sean-k-mooney | are we passing things in the wrong order | 09:44 |
| sean-k-mooney | https://alembic.sqlalchemy.org/en/latest/api/commands.html?highlight=current#alembic.command.current | 09:45 |
| sean-k-mooney | i think ya we are passing the engine insteasd of the config | 09:45 |
| sean-k-mooney | here https://github.com/openstack/nova/blob/master/nova/db/migration.py#L172-L173 | 09:46 |
| sean-k-mooney | we should probaly factor this out into a function https://github.com/openstack/nova/blob/master/nova/db/migration.py#L127-L139 | 09:48 |
| gibi | sean-k-mooney: thanks for the quick check, I will try it in devstack | 09:49 |
| sean-k-mooney | cool i was going to hack something quickly and push it | 09:49 |
| gibi | your suggestion seem to work | 09:50 |
| gibi | https://paste.opendev.org/show/809277/ | 09:50 |
| opendevreview | Merged openstack/os-vif stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808452 | 09:52 |
| opendevreview | Merged openstack/os-vif stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808453 | 09:52 |
| gibi | sean-k-mooney: is it OK to you if I file a bug and push a fix? | 09:52 |
| sean-k-mooney | sure | 09:53 |
| gibi | OK, working on it | 09:54 |
| sean-k-mooney | this is what i had | 09:54 |
| opendevreview | sean mooney proposed openstack/nova master: [WIP] db version fix https://review.opendev.org/c/openstack/nova/+/808712 | 09:55 |
| sean-k-mooney | feel free to reuse or ignore | 09:55 |
| gibi | sean-k-mooney: OK, thanks | 09:55 |
| gibi | reported a critical bug for it as it affects the execution of db sync as well https://bugs.launchpad.net/nova/+bug/1943436 | 10:01 |
| sean-k-mooney | ah presumable because db sync checks the version first | 10:03 |
| sean-k-mooney | do we know why this happens now | 10:03 |
| sean-k-mooney | looking at git blame this has not changed for 3 months | 10:03 |
| gibi | I think this is changed when we switched to alembic. was it 3 months ago? | 10:04 |
| sean-k-mooney | yep | 10:04 |
| gibi | I think we don't have test coverage on this path | 10:04 |
| sean-k-mooney | well that is thet commit date | 10:04 |
| sean-k-mooney | we should for sync however right | 10:04 |
| gibi | as it only happens if the db is migrated to alembic control and then you wan to migrate it futher | 10:04 |
| sean-k-mooney | ah | 10:05 |
| gibi | so grenade will only hit it after we have Xena -> Yoga run | 10:05 |
| sean-k-mooney | and since we have no migrations since then we would not see it | 10:05 |
| sean-k-mooney | i was wondering if it was related to the alembic version bump 12 days ago | 10:06 |
| sean-k-mooney | form 1.6.5 to 1.7.1 | 10:06 |
| sean-k-mooney | e.g. if the engine was previously valid and nolonger is | 10:06 |
| gibi | I did not see any job failing do to this. I just noticed this problem when run the CLI manually this morning | 10:06 |
| gibi | hm I was mistaken, the db sync part is not affected | 10:10 |
| sean-k-mooney | its still a regerssion in either case | 10:16 |
| gibi | yepp | 10:16 |
| gibi | trying to add repro / test coverage | 10:17 |
| opendevreview | Merged openstack/os-vif master: Update master for stable/xena https://review.opendev.org/c/openstack/os-vif/+/808454 | 10:24 |
| opendevreview | Merged openstack/placement master: Reproduce 404 when allocation queried with 1.38 https://review.opendev.org/c/openstack/placement/+/807154 | 10:42 |
| gibi | sean-k-mooney: alembic_api.current does not return the version alembic_script.ScriptDirectory.from_config( self.config) | 10:57 |
| gibi | sorry | 10:57 |
| gibi | wrong buffer | 10:57 |
| gibi | so here https://github.com/openstack/nova/blob/master/nova/db/migration.py#L173 | 10:57 |
| gibi | the alembic_api.current does not return anything | 10:57 |
| gibi | that function actually prints to stdout | 10:58 |
| sean-k-mooney | hehe lovely | 11:02 |
| sean-k-mooney | so 2 bugs | 11:02 |
| sean-k-mooney | https://alembic.sqlalchemy.org/en/latest/api/commands.html#alembic.command.current right Display the current revision for a database. | 11:03 |
| sean-k-mooney | we likely dont want to call the function form the command module | 11:04 |
| sean-k-mooney | we likely want to get the saem from the engin or similar | 11:04 |
| gibi | yepp | 11:10 |
| gibi | https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.migration.MigrationContext | 11:10 |
| gibi | I think that is the right way | 11:10 |
| gibi | there is get_current_revision() on the MigrationContext object | 11:10 |
| sean-k-mooney | https://alembic.sqlalchemy.org/en/latest/api/runtime.html#alembic.runtime.migration.MigrationContext.get_current_revision | 11:10 |
| sean-k-mooney | ya | 11:10 |
| sean-k-mooney | i saw that before when i first went looking for current | 11:11 |
| sean-k-mooney | Return the current revision, usually that which is present in the alembic_version table in the database. | 11:11 |
| sean-k-mooney | that ussaully is presumable becasue before you run the migration it may or may not match? | 11:11 |
| sean-k-mooney | its likely good enough for our use case | 11:12 |
| sean-k-mooney | zzzeek: if your around if we want to get teh current db version is get_current_revision() the correct function to call | 11:14 |
| opendevreview | Merged openstack/placement master: Fix adding 'unknown' to the ConsumerTypeCache https://review.opendev.org/c/openstack/placement/+/807155 | 11:57 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug/1943436 https://review.opendev.org/c/openstack/nova/+/808753 | 12:11 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Fix nova-manage db version https://review.opendev.org/c/openstack/nova/+/808712 | 12:13 |
| gibi | sean-k-mooney: ^^ | 12:13 |
| gibi | I dropped your refactoring as the config object is not needed any more | 12:13 |
| gibi | and the engine object was already created in scope | 12:14 |
| gibi | bauzas, lyarwood: it is an RC critical fix ^^ | 12:14 |
| bauzas | gibi: ack, will look | 12:14 |
| sean-k-mooney | cool | 12:14 |
| sean-k-mooney | ill take a look although it proably does not need the co-authoured by since you drop most of what i did | 12:15 |
| bauzas | gibi: wow, thanks for finding the issue | 12:16 |
| gibi | sean-k-mooney: you was the one that point out where the code failed so you earned the mention :) | 12:16 |
| sean-k-mooney | yeah strack traces :) | 12:17 |
| gibi | bauzas: I just tried to run nova-manage db version in my devstack this morning, and found things to for today :) | 12:17 |
| bauzas | gibi: I just wonder why we had no find the issue with the existing tests | 12:17 |
| bauzas | hadn't* | 12:18 |
| sean-k-mooney | bauzas: because we would be monkey patching the calls to the alembic lib | 12:18 |
| sean-k-mooney | so it would only show up in integration style test or maybe functional | 12:19 |
| sean-k-mooney | not in unit test | 12:19 |
| sean-k-mooney | bauzas: also i think gibi mentioned it only happens the second time you call sync | 12:20 |
| sean-k-mooney | when its using alembic? | 12:20 |
| gibi | it only fails if you db already under alembic control | 12:20 |
| sean-k-mooney | right so with an empty db it wont fail | 12:20 |
| gibi | so grenade did not detect it as it upgrades from legacy to alembic not alembic to alembiv | 12:20 |
| bauzas | sean-k-mooney: we have functional tests for nova-manage, nope ? | 12:20 |
| gibi | bauzas: not for db version | 12:20 |
| bauzas | ah right shit | 12:21 |
| gibi | becuase that is "simple" it just calls alembic ;) | 12:21 |
| bauzas | mmm, looks like we no longer generate guru meditation reports using USR2 signal ? | 12:54 |
| bauzas | it's just recycling the service | 12:54 |
| bauzas | gibi: can you test ? | 12:55 |
| bauzas | (context, other escalation meanwhile) | 12:55 |
| sean-k-mooney | bauzas: it shoudl dump it into the log | 12:55 |
| sean-k-mooney | bauzas: so when you do the kill --USR2 it should output the gmr to the logs | 12:56 |
| sean-k-mooney | not to standard out of the terminal you run kill in | 12:56 |
| sean-k-mooney | its possibel it coudl end up in the container standar out but im 99% sure it went to the logs | 12:57 |
| bauzas | sean-k-mooney: gibi: no, it's recycling | 12:57 |
| bauzas | sean-k-mooney: gibi: https://paste.opendev.org/show/809280/ | 12:57 |
| bauzas | it should dump the stateinto the logs | 12:57 |
| bauzas | but rather, it gets the signal and then dies | 12:57 |
| sean-k-mooney | that is to the wsgi server | 12:58 |
| bauzas | I guess this is uswsgi related | 12:58 |
| sean-k-mooney | ya | 12:58 |
| sean-k-mooney | have you tried it with nova-compute | 12:58 |
| sean-k-mooney | for uwsgi i dont know if it ever worked but you might need to do it to the worker process | 12:58 |
| bauzas | sean-k-mooney: I can run it against n-cpu for sure | 12:59 |
| sean-k-mooney | that is what the customer need to do anyway | 13:00 |
| sean-k-mooney | not nova-api | 13:00 |
| * gibi was off lunching | 13:00 | |
| bauzas | sean-k-mooney: sure but I wanted to test it against the API | 13:00 |
| sean-k-mooney | bauzas: i dont know if we have ever tested GMR when the api is not runnign under python | 13:00 |
| sean-k-mooney | bauzas: with nova-api directly it proably works | 13:00 |
| sean-k-mooney | with nova-api under uwsgi i have never seen it documented how to make that work | 13:01 |
| bauzas | sean-k-mooney: okay, this works with nova-compute service directly | 13:01 |
| gibi | nova-compute gmr works for me | 13:01 |
| gibi | nova-api doesnt | 13:03 |
| bauzas | gibi: okay, so I guess it's PEBKAC as I issued the signal against the main PID on the wsgi server | 13:03 |
| bauzas | ah | 13:03 |
| bauzas | so, yeah, we should configure uwsgi to *not* handle this signal | 13:03 |
| bauzas | a devstack change I guess | 13:03 |
| gibi | nova-api works if I run it under python directly | 13:04 |
| sean-k-mooney | SIGUSR2 -USR2 prints worker status | 13:05 |
| sean-k-mooney | that is what uWSGI does by default | 13:05 |
| bauzas | gibi: yeah, so this is because of the server | 13:06 |
| sean-k-mooney | https://uwsgi-docs-additions.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | 13:06 |
| sean-k-mooney | we may or may not be able to change that | 13:06 |
| bauzas | https://uwsgi-docs.readthedocs.io/en/latest/Management.html#signals-for-controlling-uwsgi | 13:07 |
| bauzas | this | 13:07 |
| bauzas | jinxed by sean-k-mooney | 13:07 |
| opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | 13:09 |
| bauzas | sean-k-mooney: looks like Stackalytics gives me a clue https://uwsgi-docs.readthedocs.io/en/latest/Options.html?#py-call-osafterfork | 13:09 |
| opendevreview | Federico Ressi proposed openstack/nova master: Check Nova project changes with Tobiko scenario test cases https://review.opendev.org/c/openstack/nova/+/806853 | 13:09 |
| sean-k-mooney | maybe | 13:10 |
| bauzas | anyway, not an urgent problem but we should somehow track this | 13:10 |
| sean-k-mooney | https://www.techatbloomberg.com/blog/configuring-uwsgi-production-deployment/ | 13:11 |
| sean-k-mooney | This feature should be enabled by default because we expect processes to respond to signals that are sent to them. Without enabling this feature, the first developer to attempt to trap signals in a uWSGI-hosted service is going to be confused until they find this option. That could cause a delay of anywhere from 0 minutes to days, depending on how good someone is at Googling | 13:12 |
| sean-k-mooney | or asking coworkers for their opinions. | 13:12 |
| sean-k-mooney | ^ | 13:12 |
| gibi | yupp that feels like a doc bug to suggest using py-call-osafterfork = true | 13:15 |
| bauzas | I'm proud of my Google fu's then :) | 13:16 |
| bauzas | I should ask for a promotion "Senior Principal Software Googler" I guess | 13:17 |
| * gibi grants bauzas the SPSG title | 13:17 | |
| bauzas | I'm used to practice it for administrative and legal paperwork | 13:18 |
| bauzas | anyway, back to the regression fix you asked me to look a | 13:18 |
| *** abhishekk is now known as akekane|home | 13:37 | |
| *** akekane|home is now known as abhishekk | 13:39 | |
| *** tosky is now known as Guest7131 | 13:40 | |
| *** tosky_ is now known as tosky | 13:40 | |
| noonedeadpunk | hey there! | 13:49 |
| noonedeadpunk | I see weird behaviour of nova-manage when I'm trying to do db_sync when ca_file is provided for mysql connection string | 13:49 |
| noonedeadpunk | s/ca_file/ssl_ca/ | 13:50 |
| noonedeadpunk | So eventually that's stack trace https://paste.opendev.org/show/809281/ | 13:50 |
| noonedeadpunk | And all other services treat same connection string correctly | 13:51 |
| noonedeadpunk | So for me it feels like there's somewhere urlencode appllied or smth like that | 13:51 |
| artom | noonedeadpunk, that smells like a thing we fixed recently... | 13:52 |
| noonedeadpunk | If I add quotes in the connection line - this prevents file patch from being converted, but then wrong path is passed | 13:52 |
| noonedeadpunk | oh, I'm running master but with head several weeks before... | 13:52 |
| noonedeadpunk | will try to update sha used | 13:52 |
| artom | noonedeadpunk, https://review.opendev.org/c/openstack/nova/+/805663/9 | 13:53 |
| noonedeadpunk | well, I came to smth simmilar :) thanks | 13:53 |
| noonedeadpunk | artom: works nicely! | 13:59 |
| artom | \o/ | 13:59 |
| gibi | interestingly multiple people independently reported bugs about nova/placement does not allow migrating VMs out from overallocated computes https://bugs.launchpad.net/nova/+bug/1943191 | 14:18 |
| gibi | I think we have that situation for a long time, sice we use migration allocations | 14:19 |
| gibi | hm, even before | 14:19 |
| gibi | since we use placement actually | 14:19 |
| opendevreview | Alban Lecorps proposed openstack/nova master: VMware: Support volumes backed by VStorageObject https://review.opendev.org/c/openstack/nova/+/808791 | 15:03 |
| gibi | it is fun to see how bauzas consistently clicks on Review-Priority +1 instead of Workflow +1 then lyarwood goes and fix is up for him. this is real teamwork :) | 15:09 |
| gibi | btw thanks for the reviews on the nova-manage db version fix | 15:10 |
| lyarwood | ^_^ np | 15:10 |
| dansmith | in bauzas' defense, it breaks a decade of muscle memory, so I sympathize | 15:23 |
| dansmith | I usually catch myself, but not always | 15:24 |
| bauzas | gibi: holy shit | 15:27 |
| bauzas | dansmith: and yeah, it was that for at least 6 years for Nova and 8 years for another project ;) | 15:27 |
| bauzas | wow, time flies | 15:28 |
| gibi | yeah I also guessed that it was muscle memory | 15:28 |
| bauzas | gibi: if I was a astronaut, I could then have created a problem with my spacecraft :p | 15:46 |
| * bauzas clicked on the wrong button | 15:46 | |
| opendevreview | Merged openstack/nova master: Reproduce bug/1943436 https://review.opendev.org/c/openstack/nova/+/808753 | 15:48 |
| gibi | I hope spacex freezing the UI layout at least for each individual flight :) | 15:48 |
| opendevreview | Merged openstack/nova master: Fix nova-manage db version https://review.opendev.org/c/openstack/nova/+/808712 | 16:43 |
| coreycb | hello, are there any near-term plans for new stable releases of nova? | 16:53 |
| elodilles | coreycb: I'm not aware of plans but it is possible to release for wallaby, victoria and ussuri as well as there are merged patches | 17:02 |
| lyarwood | coreycb: elodilles normally does these, I can't see any open but we can raise it tomorrow during the team meeting | 17:02 |
| lyarwood | ah there we go | 17:03 |
| lyarwood | :) | 17:03 |
| elodilles | :) | 17:03 |
| coreycb | elodilles: lyarwood: thanks for the response and raising it at the meeting | 17:33 |
| opendevreview | Alban Lecorps proposed openstack/nova master: VMware: Support volumes backed by VStorageObject https://review.opendev.org/c/openstack/nova/+/808791 | 19:33 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!