| bauzas | good morning Nova | 08:20 |
|---|---|---|
| *** thuvh1 is now known as thuvh | 09:22 | |
| tobias-urdin | good morning o/ | 09:52 |
| tobias-urdin | let's say I have hyperthreding enabled, is there any logic to schedule an dedicated pCPU instance to use one core and it's thread sibbling? to ensure no two instances are running on the same thread sibbling and shares that | 09:53 |
| tobias-urdin | i can only find that we can select cpu thread policy but not to isolate actual usage of thread sibblings | 09:53 |
| sean-k-mooney | tobias-urdin: only if you are not usign cpu in placement | 09:55 |
| opendevreview | Ilya Popov proposed openstack/nova stable/xena: compute: Update volume_id within connection_info during swap_volume https://review.opendev.org/c/openstack/nova/+/874921 | 09:55 |
| opendevreview | Ilya Popov proposed openstack/nova stable/xena: Add regression test for bug #1943431 https://review.opendev.org/c/openstack/nova/+/875374 | 09:55 |
| sean-k-mooney | we deprecated that functionality a few release ago | 09:55 |
| sean-k-mooney | ~train ish | 09:55 |
| sean-k-mooney | well actully it depend on what you want | 09:56 |
| sean-k-mooney | if you use thread policy require then we require hyperthread siblings to be used | 09:56 |
| sean-k-mooney | i.e a 4 core vm will use 2 phsyical cores and use both hyperthread form each core to provided the 4 pinned cpus | 09:56 |
| sean-k-mooney | that still works today even with placmeent | 09:57 |
| sean-k-mooney | the isolate poicy is what chagned. | 09:57 |
| sean-k-mooney | without pcpu in placementit made each flavor.vcpu be pinnded to a speerate core (and if it had hyperthreadig it reserved the hyperthread so nothing could use it.) | 09:57 |
| sean-k-mooney | with pcpus in placement isolate means find cores without hyperthread | 09:58 |
| tobias-urdin | sean-k-mooney: hm, so if thread policy=required and using even numbers (4 cores, 8 cores) etc we always assume that thread sibblings for those 2 or 4 cores are assigned to only one instance? | 09:58 |
| tobias-urdin | and that would be honored for live migration as well if we set vcpu_pin_set to same on all nodes? | 09:58 |
| sean-k-mooney | tobias-urdin: yes i belive so | 09:58 |
| sean-k-mooney | well it does not have to be the same | 09:59 |
| sean-k-mooney | we update the pinning on migration as of train | 09:59 |
| tobias-urdin | ah | 09:59 |
| sean-k-mooney | so they can be slightly differnt vcpu_pin_sets | 09:59 |
| sean-k-mooney | just make sure you include both hypertherad siblings in vcpu_pin_set or cpu_dedicated_set | 10:00 |
| sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/numa-aware-live-migration.html | 10:00 |
| sean-k-mooney | ^ that fixed live migration | 10:01 |
| sean-k-mooney | https://specs.openstack.org/openstack/nova-specs/specs/train/implemented/cpu-resources.html | 10:01 |
| sean-k-mooney | and that explains how placement aware cpu pinning works if you use it | 10:01 |
| tobias-urdin | ack, I think you answered my question but will read above! thanks, wanted to make sure that we could safetly have ht enabled if using dedicated | 10:03 |
| sean-k-mooney | tobias-urdin: the old way of doign cpu pinning with vcpu_pin_set has been deprecated for a couple of years now. we will likely delete it in 2023.2 or 2024.1 | 10:03 |
| sean-k-mooney | you want to prevent inter tenant side channel attacks | 10:03 |
| sean-k-mooney | if so then yes use required/isolate in all flavors to ensure to vms form different tenants dont overlap | 10:04 |
| tobias-urdin | so i will set cpu_policy=dedicated cpu_thread_policy=required on flavor and make sure cpu_dedicated_set is set and including sibblings | 10:04 |
| opendevreview | Merged openstack/os-vif master: Update master for stable/2023.1 https://review.opendev.org/c/openstack/os-vif/+/875097 | 10:04 |
| tobias-urdin | make sure flavors use even amount of cores, so it gets distributed between core and thread (4 = 2 core, 2 thread, 8 = 4 core, 4 thread etc) | 10:04 |
| tobias-urdin | that sounds reasonable? | 10:04 |
| sean-k-mooney | more or less but read the upgrade section fo the cpu tracking in placemnt doc carefully | 10:05 |
| tobias-urdin | we're on xena moving to yoga in a couple of weeks btw if that makes any difference | 10:05 |
| sean-k-mooney | by the way i generally recommend seting hw:cpu_sockets=<the number of numa nodes> and hw:cpu_thread=2 | 10:06 |
| sean-k-mooney | not really just that we planned to delete the old pinning code around wallaby but stephenfin moved team and we just didnt get around to it | 10:06 |
| sean-k-mooney | the upgrade impact is the same regardless of when you actuly do the upgrade form old pining to placment aware pinning | 10:07 |
| sean-k-mooney | with the slight simplifciation that at least you know all your nodes already have support for it in code | 10:08 |
| tobias-urdin | sean-k-mooney: not sure i follow about what is legacy and not | 10:14 |
| tobias-urdin | it should be tracked in placement already but do you we mean we should probably disable the fallback path for pcpu? | 10:14 |
| tobias-urdin | or do you mean you want to get rid of the hw: aliases and use PCPU etc resource directly in flavor properties to affect placement? | 10:15 |
| sean-k-mooney | useing vcpu_pin_set for pinning is deprecated | 10:21 |
| sean-k-mooney | we will likely remove that config option sooner rather then later | 10:22 |
| sean-k-mooney | PCUPs are only tracked in placment as a sperate inventory if you use cpu_dedicated_set | 10:22 |
| tobias-urdin | that makes sense, thanks | 10:22 |
| sean-k-mooney | there will be no changes to the hw: extra specs | 10:22 |
| tobias-urdin | ack | 10:22 |
| sean-k-mooney | once we remvoe the vcpu_pin_set we can remove the old code for the legacy isolate behavior (asinging one ht siblibing to the vm and reserving the other) | 10:23 |
| sean-k-mooney | that will allow use to simplify that code a little | 10:24 |
| sean-k-mooney | right now any time we want to extend pinned cpus in some way we have to desgin arond the placement natiave and pre placement behavior which is a bit of a pain | 10:25 |
| sean-k-mooney | the code is pretty stable at this point but its still a maintance burden becuase we have to think about both sets anythime we are changing anything | 10:26 |
| tobias-urdin | sean-k-mooney: i see, but using cpu_dedicated_set should that also honor numa_nodes? i understand, a lof of complexity involved with this amount of options that affect each other | 10:26 |
| tobias-urdin | i.e i set hw:num_nodes=1 on a flavor | 10:27 |
| bauzas | cores, needs a quick slap on https://review.opendev.org/c/openstack/nova/+/874103 (max microversion for Antelope) | 10:27 |
| sean-k-mooney | -2 it is then :p | 10:28 |
| sean-k-mooney | lets see | 10:28 |
| sean-k-mooney | yep makes sense | 10:28 |
| sean-k-mooney | that was trivial enough that you could have single core approved that | 10:28 |
| bauzas | sean-k-mooney: we need it every cycle end | 10:28 |
| sean-k-mooney | right but its a mechanical patch and pretty easy to see its correct | 10:29 |
| bauzas | gmann: saw your ping, when you're up, let's discuss about the RBAC stuff for the prelude | 10:29 |
| sean-k-mooney | anyway its on its way | 10:29 |
| sean-k-mooney | although the main delta this time is 2023.1 Antelope | 10:30 |
| bauzas | sean-k-mooney: sure, but I prefer to ask for another review, and in case I don't have a second core, I can +W it | 10:30 |
| sean-k-mooney | i.e. using both the number and the code name | 10:30 |
| bauzas | sean-k-mooney: not sure I understand your concern, takashi used both the number and the code name | 10:31 |
| sean-k-mooney | its not a concern | 10:31 |
| sean-k-mooney | i prefer having both | 10:31 |
| sean-k-mooney | rather then just 2023.1 or antelope | 10:31 |
| sean-k-mooney | that was the one thing i wanted to make sure was done in that patch | 10:32 |
| sean-k-mooney | so the patch is correct | 10:32 |
| sean-k-mooney | we listed the offical release name "2023.1" and the code name "Antelope" and listed teh offical release name "2023.1" first. | 10:33 |
| bauzas | don't disagree here https://governance.openstack.org/tc/reference/release-naming.html | 10:34 |
| sean-k-mooney | we are very close to being in violent agreement i think :) | 10:34 |
| sean-k-mooney | tobias-urdin: cpu_dedicated_set supports numa yes | 10:37 |
| sean-k-mooney | hw:numa_nodes was unaffected by the cpu tracking in placement spec | 10:38 |
| sean-k-mooney | we do eventually want to supprot modeling numa in placment but its quite difficult | 10:38 |
| sean-k-mooney | we have instead decided to track all resouces we can in placement first without numa | 10:39 |
| sean-k-mooney | and then add numa at the end | 10:39 |
| tobias-urdin | sean-k-mooney: ack, thanks for all your help – extremely helpful :) | 10:39 |
| sean-k-mooney | the pci devices in placement spec added one of the builing block features we will need for numa in placment (the ablity for filters to filter on allcoations candiates) | 10:40 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: Add the 2023.1 Antelope prelude section https://review.opendev.org/c/openstack/nova/+/875380 | 10:40 |
| sean-k-mooney | lol i tought you had written that but that was the release highlights | 10:41 |
| sean-k-mooney | did we land the spice console compressiong feature | 10:41 |
| bauzas | yes | 10:42 |
| opendevreview | Jorge San Emeterio proposed openstack/nova master: Moving privsep profiles to nova/__init__.py https://review.opendev.org/c/openstack/nova/+/872010 | 10:42 |
| sean-k-mooney | good good | 10:42 |
| * bauzas needs to get his kid from school, bbiab | 10:42 | |
| sean-k-mooney | i reviewed the spec for that and planned to review the feature i just ran out of time to do that im glad it landed | 10:44 |
| opendevreview | Merged openstack/nova master: doc: mark the max microversion for 2023.1 Antelope https://review.opendev.org/c/openstack/nova/+/874103 | 11:28 |
| *** bhagyashris is now known as bhagyashris|ruck | 11:51 | |
| opendevreview | Merged openstack/python-novaclient stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/python-novaclient/+/875101 | 12:23 |
| opendevreview | Rajesh Tailor proposed openstack/nova stable/xena: Fix rescue volume-based instance https://review.opendev.org/c/openstack/nova/+/875343 | 12:26 |
| opendevreview | Rajesh Tailor proposed openstack/nova stable/wallaby: Fix rescue volume-based instance https://review.opendev.org/c/openstack/nova/+/875344 | 12:27 |
| opendevreview | Merged openstack/os-vif stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/os-vif/+/875095 | 12:28 |
| opendevreview | Merged openstack/os-vif stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/os-vif/+/875096 | 12:28 |
| opendevreview | Merged openstack/osc-placement stable/2023.1: Update .gitreview for stable/2023.1 https://review.opendev.org/c/openstack/osc-placement/+/875098 | 12:29 |
| opendevreview | Rajesh Tailor proposed openstack/nova stable/xena: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/875345 | 12:29 |
| opendevreview | Rajesh Tailor proposed openstack/nova stable/wallaby: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/875346 | 12:30 |
| opendevreview | Merged openstack/python-novaclient stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/python-novaclient/+/875102 | 12:30 |
| opendevreview | Rajesh Tailor proposed openstack/nova stable/victoria: Fix rescue volume-based instance https://review.opendev.org/c/openstack/nova/+/875347 | 12:32 |
| opendevreview | Rajesh Tailor proposed openstack/nova stable/victoria: Handle InstanceInvalidState exception https://review.opendev.org/c/openstack/nova/+/875348 | 12:33 |
| opendevreview | Merged openstack/nova master: Doc: update live-migration cmd https://review.opendev.org/c/openstack/nova/+/875043 | 12:40 |
| opendevreview | Merged openstack/osc-placement stable/2023.1: Update TOX_CONSTRAINTS_FILE for stable/2023.1 https://review.opendev.org/c/openstack/osc-placement/+/875099 | 12:46 |
| opendevreview | Jorge San Emeterio proposed openstack/nova master: WIP: Creating an example of the refactor privileged functions will go through. https://review.opendev.org/c/openstack/nova/+/875497 | 14:23 |
| artom | bauzas, dansmith_, is there an etherpad or something for all the CI issues currently hitting us? | 14:36 |
| artom | jsanemet ^^ | 14:36 |
| dansmith_ | artom: there's this at least: https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure | 14:36 |
| *** haleyb_ is now known as haleyb | 14:36 | |
| artom | Do we still have an ELK/opensearch instance somewhere? | 14:37 |
| * dansmith_ artom: https://opensearch.logs.openstack.org/_dashboards/app/discove | 14:39 | |
| artom | That needs a login? | 14:43 |
| dansmith_ | openstack/openstack | 14:44 |
| dansmith_ | it's apparently not option-able anymore | 14:44 |
| *** dansmith_ is now known as dansmith | 14:44 | |
| artom | https://opensearch.logs.openstack.org/_dashboards/app/discover, you were missing an 'r' :) | 14:48 |
| artom | jsanemet, ^^ so yeah, that allows you to search through all the logs we keep from our Zuul jobs | 14:52 |
| artom | openstack/openstack to log in | 14:52 |
| artom | There's a query language that I always get wrong to search for specific things | 14:52 |
| jsanemet | cool, that will be useful | 14:53 |
| artom | Apparently you can use https://opensearch.org/docs/latest/opensearch/query-dsl/index/, or Lucene directly | 14:53 |
| artom | I don't know a lot about Lucene, https://www.lucenetutorial.com/lucene-query-syntax.html I guess? | 14:56 |
| jsanemet | ok, i will play with these a little bit, see what can be done | 14:56 |
| artom | So yeah, jsanemet, if you're interested in helping fix gate bugs, ^^ is a useful tool, and I'm sure dansmith's will be more than happy to take you under his wing and show you everything he knows ;) | 14:57 |
| jsanemet | i need to take a look at the reported bugs from before | 14:57 |
| jsanemet | but i will see if i can pick up something | 14:58 |
| jsanemet | will be great to contribute | 14:58 |
| opendevreview | Merged openstack/nova master: Transport context to all threads https://review.opendev.org/c/openstack/nova/+/827467 | 15:11 |
| Uggla | bauzas, I need your opinion on that bug https://bugs.launchpad.net/nova/+bug/2008461, the bug seems legit, but the method is marked deprecated and might be removed at some point. How do you treat this bug ? | 15:33 |
| Uggla | bauzas, I mean should I set it to valid in this case ? | 15:33 |
| Uggla | s/valid/confirmed | 15:34 |
| lowercase | Hey guys, i think i found an issue with https://review.opendev.org/c/openstack/nova/+/812145/2/nova/db/api/migrations/versions/b30f573d3377_remove_unused_build_requests_columns.py#30 . I'm currently upgrading from wallaby to yoga using openstack-ansible, and during the upgrade process i get the error, sqlalchemy.exc.OperationalError: (pymysql.err.OperationalError) (1091, "Can't DROP COLUMN `vm_state`; check that it exists"). [SQL: | 16:05 |
| lowercase | ALTER TABLE build_requests DROP COLUMN vm_state] The entire table, build_requests does not exist. Additionally, as a troubleshooting measure i droped the nova database, created a blank one and ran api_db sync again to see the same issue. | 16:05 |
| lowercase | It would seem a lacking test to confirm if the table exists before attempting to drop it. | 16:06 |
| lowercase | Opened this bug report to address this issue. https://bugs.launchpad.net/nova/+bug/2008716 | 16:32 |
| sean-k-mooney | lowercase: you cant directly upgrade form wallaby to yoga | 17:45 |
| lowercase | i miss remembered the version names, i went from xena to yoga | 17:45 |
| lowercase | I corrected the bug as well | 17:45 |
| lowercase | *i corrected the bug report to accuratly reflect my upgrade path. | 17:46 |
| sean-k-mooney | so based on https://github.com/openstack/nova/commit/9657297dd6c63e7a1e0c84c3e943b26f1795d388 | 17:47 |
| sean-k-mooney | these were never used | 17:47 |
| sean-k-mooney | so im wondering | 17:47 |
| sean-k-mooney | how you got entries into that db table | 17:48 |
| lowercase | I do not have entries in the db table. My issue is that I do not have build_requests table and nova-manage api_db sync attempts to remove them, resulting in a python exception. | 17:48 |
| sean-k-mooney | oh ok | 17:49 |
| sean-k-mooney | sound like an issue tha twould only happne if you created the db after it was removed form the schema | 17:50 |
| sean-k-mooney | which happned in https://github.com/openstack/nova/commit/98a05bc637e4c2e485d5aa5b62945102ea71d08b | 17:50 |
| sean-k-mooney | so was your cloud/db created in newton or later | 17:50 |
| lowercase | mmm.. this cloud could be traced to around train. | 17:51 |
| lowercase | regardless. I dropped the entire nova database, and created a fresh one. No tables present, and the error persisted. | 17:51 |
| sean-k-mooney | stephenfin: this sounds like it might be relate to swaping to alembic | 17:52 |
| lowercase | I have an env that is about 30 servers or so that is exclusive for dev testing releases before upgrading our other environments. | 17:52 |
| sean-k-mooney | lowercase: so in yoga we have two ways to upgrade the db | 17:54 |
| sean-k-mooney | we have the legacy migration based on sqlalchemy migrate | 17:54 |
| sean-k-mooney | and we have alembic which is what we use now | 17:55 |
| sean-k-mooney | the base schema we use for train + has the build_requests table https://github.com/openstack/nova/blob/stable/yoga/nova/db/api/legacy_migrations/versions/067_train.py#L159-L209 | 17:55 |
| sean-k-mooney | and then in yoga we removed it | 17:56 |
| sean-k-mooney | so somehow you seam to have droped it already | 17:56 |
| stephenfin | lowercase: there should be no reason to check if it's present because we would have created it in the initial migration | 17:57 |
| stephenfin | and if someone dropped it outside of the context of nova then all bets are off | 17:58 |
| stephenfin | lowercase: you said you dropped the table and ran api_db sync again to see what happened. Did | 17:58 |
| stephenfin | *dropped the database | 17:59 |
| lowercase | What is happening right now, is openstack-ansible includes an api sync during its installation/upgrade path. | 17:59 |
| stephenfin | Did you drop the database or just the tables? | 17:59 |
| lowercase | so if i was to run the playbook multiple times.. could result in a situation where the api_sync has already removed the build_requests table being removed. | 17:59 |
| stephenfin | No, alembic records the migrations it has already applied. It won't reapply them | 18:00 |
| lowercase | i dropped the database as a test, and i confirmed the table was already dropped. I then created the table and continued my testing. | 18:00 |
| stephenfin | if you look, you should see an 'alembic_version' table with this information in it | 18:01 |
| lowercase | Running upgrade d67eeaabee36 -> b30f573d3377, Remove unused build_requests columns | 18:01 |
| stephenfin | I then created the table and continued my testing. <-- how? | 18:01 |
| stephenfin | The thing I'm trying to figure out is how much stuff, if anything, you are doing outside of nova-manage/alembic | 18:02 |
| lowercase | Let me explain the order of events. I ran the nova playbook. resulted in an error. I logged intot he nova databases and confirmed the build_requests column was indeed absent. To isolate the issue, i created a dummy build_requests with vm_state. Error persisted. I dropped build_requests again, as expected. The error persisted. I then, backed up, dropped and recreated the nova database entirely. Resulting in the same issue. | 18:04 |
| lowercase | I've been picking at this issue for about two weeks, only today was i able to find the resulting code that resulted in my issue. | 18:04 |
| sean-k-mooney | right but it sound like you are startign form an invalid db state | 18:06 |
| sean-k-mooney | im jus tlooking at the osa code now | 18:06 |
| sean-k-mooney | to see how its creatign the db | 18:06 |
| stephenfin | lowercase: can you drop the database again then run the following: 'nova-manage api_db sync d67eeaabee36' | 18:06 |
| stephenfin | ...and share both the output of the command and the schema of the table after doing so | 18:06 |
| stephenfin | that will apply all migrations up to the offending one but not that migration itself | 18:07 |
| lowercase | show databases; -> confirmed nova was present. drop database nova; and then show databases; confirmed nova is no longer present. | 18:07 |
| sean-k-mooney | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/tasks/nova_db_setup.yml and https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/tasks/nova_db_post_setup.yml look ok | 18:07 |
| lowercase | create database nova; show databases; nova is now present | 18:08 |
| sean-k-mooney | lowercase: nova is the cell db | 18:08 |
| sean-k-mooney | lowercase: not the api db | 18:09 |
| sean-k-mooney | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/defaults/main.yml#L93 | 18:09 |
| sean-k-mooney | https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/defaults/main.yml#L107 | 18:09 |
| lowercase | /openstack/venvs/nova-25.2.1.dev3/bin/nova-manage api_db sync d67eeaabee36 | 18:09 |
| lowercase | Modules with known eventlet monkey patching issues were imported prior to eventlet monkey patching: urllib3. This warning can usually be ignored if the caller is only importing and not executing nova code. | 18:09 |
| lowercase | echo $? | 18:09 |
| lowercase | 0 | 18:09 |
| sean-k-mooney | lowercase: you shoudl be droping and recreateing nova_api | 18:09 |
| stephenfin | sean-k-mooney: good spot :) | 18:09 |
| lowercase | okay, since we are in a different spot. please confirm you want me to repeat the steps with nova_api databases. | 18:10 |
| stephenfin | yup | 18:10 |
| stephenfin | though before you do | 18:11 |
| lowercase | okay, stoping. | 18:11 |
| stephenfin | can you dump the contents of the 'alembic_version' table from nova_api | 18:11 |
| lowercase | kk | 18:11 |
| stephenfin | and perhaps the migrate_version table too, if there's one | 18:11 |
| stephenfin | paste.openstack.org | 18:11 |
| sean-k-mooney | nova is slightly differnt in that by default it has 3 dbs nova_api nova_cell0 and nova_cell1 and osa used ot use nova for nova_cell1 i think but changed at some point | 18:12 |
| sean-k-mooney | so here nova is the cell1 db https://github.com/openstack/openstack-ansible-os_nova/blob/abac462dc218b08656b56c57e5a401aa639a0f0b/tasks/main.yml#L69-L80 | 18:14 |
| lowercase | sorry, my mysql isn't very strong. show columns from alembic_version; shows me what the key and type everything is. how do i dump the data from a field? | 18:15 |
| sean-k-mooney | select * from alembic_versions; | 18:15 |
| sean-k-mooney | i think its actully just one row | 18:16 |
| lowercase | select * from alembic_version; results in d67eeaabee36 | 18:16 |
| lowercase | which is what we expect | 18:16 |
| stephenfin | is there a migrate_version table? | 18:16 |
| lowercase | yes | 18:17 |
| lowercase | | nova_api | /openstack/venvs/nova-21.2.0/lib/python3.6/site-packages/nova/db/sqlalchemy/api_migrations/migrate_repo | 87 | | 18:17 |
| stephenfin | cool, that's the most recent version of the legacy migrations, so everything has been applied | 18:19 |
| lowercase | Good to proceed with the dropping of the db? | 18:19 |
| stephenfin | one last thing | 18:19 |
| sean-k-mooney | well i would make a backup first | 18:20 |
| lowercase | actually, since i was in the wrong db. there is a build_requests here. | 18:20 |
| lowercase | select * from build_requests; | 18:20 |
| lowercase | Empty set (0.000 sec) | 18:20 |
| lowercase | sean-k-mooney: i backed up before all changes, we are good. | 18:20 |
| sean-k-mooney | you could do a "show create table build_request;" | 18:20 |
| stephenfin | can you dump the schema from that table? | 18:20 |
| lowercase | show create table build_request; | 18:21 |
| lowercase | ERROR 1146 (42S02): Table 'nova_api.build_request' doesn't exist | 18:21 |
| sean-k-mooney | that what the command above does by the way | 18:21 |
| sean-k-mooney | missing an s at the end sorry | 18:21 |
| sean-k-mooney | show create table build_requests; | 18:21 |
| lowercase | let me paste this | 18:21 |
| lowercase | my fault, i should have caught that | 18:21 |
| sean-k-mooney | i typo things constantly | 18:22 |
| lowercase | https://paste.centos.org/view/cf6648bc | 18:22 |
| lowercase | Still holding off on blowing away the db. | 18:24 |
| stephenfin | You can blow it away now | 18:24 |
| lowercase | kk | 18:25 |
| sean-k-mooney | so that db is missing the filed in from the train schema | 18:25 |
| sean-k-mooney | so it has already been migrated | 18:25 |
| sean-k-mooney | or contracted/truncated in this case | 18:26 |
| lowercase | drop nova_api database; show database ; -> confirmed nova-api is no longer present. | 18:26 |
| lowercase | create database nova_api ; and /openstack/venvs/nova-25.2.1.dev3/bin/nova-manage api_db sync now results in echo $? == 0 | 18:27 |
| sean-k-mooney | ok so it succeeded | 18:27 |
| sean-k-mooney | you shoudl be able to run "nova-manage api_db sync" again | 18:28 |
| sean-k-mooney | and i basically shoudl do nothing | 18:28 |
| stephenfin | Cool. So something has gone wrong the first time you applied that. My guesses would be that it was already applied but the alembic_version table wasn't updated for some reason. The other guess is that someone manually dropped these at some point in the past | 18:29 |
| lowercase | Correct, i ran again and no error | 18:29 |
| sean-k-mooney | its also possible that you ran this pointing at the wrong db | 18:32 |
| sean-k-mooney | the api db migratoins wont work on the cell dbs | 18:32 |
| sean-k-mooney | so if you had the wrong db in the config file then you woudl see the issue you had | 18:32 |
| stephenfin | that only makes sense if the config is wrong though, yeah | 18:32 |
| lowercase | no chance. openstack-ansible performed the operation. And i ran a dozen playbooks which all require sql connections successfully before this one | 18:32 |
| sean-k-mooney | ack | 18:33 |
| stephenfin | then I guess you need to restore the backup and try again | 18:33 |
| lowercase | okay, restoring the database | 18:33 |
| stephenfin | you can dummy apply the migration is needed, updating the 'alembic_version' record, but you should figure out why this happened first. Did you keep logs from OSA? | 18:33 |
| lowercase | That seems like a highly desirable feature. but i dont think it does for me. And I didn't go out of my way to save them | 18:34 |
| stephenfin | ack, then you might be stuck with never knowing and just fixing the issue | 18:35 |
| sean-k-mooney | this looks right to me too by the way https://github.com/openstack/openstack-ansible-os_nova/blob/stable/yoga/templates/nova.conf.j2#L216-L217 | 18:36 |
| sean-k-mooney | i do not see an obvious bug in osa | 18:36 |
| sean-k-mooney | if you hit this again and find a repoducer let us know | 18:36 |
| sean-k-mooney | but it seams to be working ok again | 18:37 |
| lowercase | okay, i restored the database. d67eeaabee36 is the alembic_version again. | 18:38 |
| lowercase | /openstack/venvs/nova-25.2.1.dev3/bin/nova-manage api_db sync results in [SQL: ALTER TABLE build_requests DROP COLUMN vm_state] exception again. | 18:38 |
| stephenfin | and the schema's the same before you run that? i.e. those columns are missing? | 18:39 |
| stephenfin | lowercase: If you need to dummy apply the migration, you can create a file like this https://paste.opendev.org/show/818911/ and call alembic with 'alembic -c path/to/file stamp head' | 18:40 |
| stephenfin | If alembic isn't available, 'virtualenv .venv; source .venv; pip install alembic; {commands}; deactivate; rm -rf .venv' | 18:40 |
| * stephenfin needs to drop, unfortunately | 18:40 | |
| lowercase | Yes, build_requests is present with nothing in it. | 18:42 |
| lowercase | and the schema is identical to the paste i shared previosuly. | 18:42 |
| lowercase | stephenfin: alembic resulted in the following error: https://paste.opendev.org/show/beYkgfrwuHYduGCDnGcE/ | 18:57 |
| -opendevstatus- NOTICE: The Gerrit service on review.opendev.org experienced severe performance degradation between 17:50 and 19:45 due to excessive API query activity; the addresses involved are now blocked but any changes missing job results from that timeframe should be rechecked | 19:55 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!