| *** auniyal0 is now known as auniyal | 07:45 | |
| noonedeadpunk | Hey folks! Completely forgot to ask that during PTG last week :( But wdyt about this bug report? For me it sounds more like a nova issue, that service fails and do not try to re-auth? https://bugs.launchpad.net/openstack-ansible/+bug/2023370 | 10:56 |
|---|---|---|
| noonedeadpunk | So what's hapenning in OSA here, is that we reset a password (in 99.9% to exactly same one). While I kinda agree we should not do that at the first place, but this should not break service either? | 10:57 |
| *** Continuity__ is now known as Continuity | 11:24 | |
| stephenfin | melwitt: Sorry for the delay, reviewed that DB archive series. I addressed your feedback on https://review.opendev.org/c/openstack/nova/+/880669/ also. Thanks for that | 11:26 |
| opendevreview | Merged openstack/nova master: testing: Fix and robustify archive_deleted_rows test https://review.opendev.org/c/openstack/nova/+/877055 | 11:49 |
| sean-k-mooney | noonedeadpunk: reset the password where | 12:52 |
| sean-k-mooney | noonedeadpunk: you update the db password? rabbit? | 12:52 |
| noonedeadpunk | sean-k-mooney: in keystone. | 12:52 |
| noonedeadpunk | so issued previously tokens are invalidated | 12:52 |
| noonedeadpunk | despite password is kinda the same | 12:53 |
| sean-k-mooney | oh that sound like a keystone issue then | 12:53 |
| sean-k-mooney | password or what by the way | 12:53 |
| noonedeadpunk | but it happens kinda only when asking neutron api by nova service, and only first request fails, and then it re-asks for the token I believe | 12:53 |
| sean-k-mooney | the password of the nova user? or the enduser creating the vm | 12:53 |
| noonedeadpunk | of the nova user | 12:53 |
| noonedeadpunk | and system service user per say | 12:54 |
| sean-k-mooney | right so if you udpate it in keystoen you need to update it in the nova.conf and then restart the nova service | 12:54 |
| sean-k-mooney | unless this is marked as mutable a sighup wont be enough | 12:54 |
| noonedeadpunk | yeah, but it's reseted to the same value | 12:54 |
| noonedeadpunk | So you kinda don't need to update conf | 12:55 |
| noonedeadpunk | And thus - don't restart service | 12:55 |
| sean-k-mooney | regardless of if its updateed to the same or diffent value i woudl expect keystone to acppet existing non expired tokens | 12:55 |
| noonedeadpunk | And it wasn't an issue in X and even in Y I believe | 12:55 |
| sean-k-mooney | so that why i said it sounded like a keystone issue | 12:56 |
| sean-k-mooney | it sounds like they have made updating the password invalidation all current tokens | 12:56 |
| sean-k-mooney | which would be a major api brake | 12:56 |
| sean-k-mooney | as in it should requrie a major api bump as that woudl completely change the meaning of the token experation | 12:57 |
| noonedeadpunk | I'm not sure that's exactly the case, was more guessing what hapens | 12:58 |
| noonedeadpunk | As I can hardly imagine any other reason for such behaviour | 12:58 |
| noonedeadpunk | The interesting part also, that according to the report, only neutron connection breaks | 12:59 |
| sean-k-mooney | well if the user and password is the same when we create the admin client it shoudl get a new token | 12:59 |
| sean-k-mooney | this is presumable port binding? | 12:59 |
| noonedeadpunk | Yeah, I believe so. | 13:00 |
| sean-k-mooney | we generally dont use nova for anything else we use the end useers token | 13:00 |
| sean-k-mooney | ya so portbidning is admin only so we use the admin clinet withthe user/password form teh config | 13:00 |
| noonedeadpunk | And it fails with `nova.exception.NeutronAdminCredentialConfigurationInvalid` | 13:00 |
| noonedeadpunk | (which is weird exception kinda) | 13:01 |
| noonedeadpunk | One thing ofc, is that we will just stop resetting password to the same value. This was added quite some time ago to ensure that password is correct | 13:02 |
| noonedeadpunk | but even though I'd say it shouldn't fail that way. So maybe just having some retry or try/except would be good to get new token when needed | 13:08 |
| anskiy | I'm the reporter of the https://bugs.launchpad.net/openstack-ansible/+bug/2023370. I see this behaviour in Y too, except that in Z it does recover, whilst in Y one needs to restart Nova API service. I haven't tested this in X, but I can do it if that would help. | 13:08 |
| noonedeadpunk | anskiy: I think latest Y should be quite the same? | 13:08 |
| noonedeadpunk | As OSSA-2023-003 was backported to Y | 13:09 |
| anskiy | noonedeadpunk: oh, yes, true. | 13:09 |
| noonedeadpunk | no idea though | 13:09 |
| * bauzas reading back the channel | 13:25 | |
| bauzas | noonedeadpunk: sean-k-mooney: I don't know how Neutron ensures Keystone authd, but at least for Nova, all our meat is provided by keystonemiddleware | 13:27 |
| bauzas | I assume the same for Neutron | 13:27 |
| bauzas | so yeah, sounds a Keystone issue, probably in middleware | 13:27 |
| bauzas | I don't see why we would need to recycle the nova-api services, but if middleware does some sort of caching, this may explain | 13:28 |
| noonedeadpunk | yeah, ok, will try to bother keystone folks then | 13:29 |
| bauzas | noonedeadpunk: https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html | 13:30 |
| bauzas | particularly https://docs.openstack.org/keystonemiddleware/latest/middlewarearchitecture.html#improving-response-time | 13:30 |
| noonedeadpunk | yeah, I'm pretty sure memcached is being configured | 13:31 |
| bauzas | https://blog.benhall.me.uk/wp-content/uploads/2015/01/cache-invalidation.png | 13:32 |
| bauzas | pretty well known stance, I assume | 13:32 |
| noonedeadpunk | the werid thing though, is that on subsequent request it jsut passes | 13:32 |
| noonedeadpunk | which made me thinking if it's jsut some simple retry that can be added somewhere | 13:33 |
| bauzas | well, sounds a design problem to me, which is never simple to solve :) | 13:34 |
| bauzas | like, memcached has to be flushed against a user if that user password changed | 13:34 |
| bauzas | unrelated, was the f... is going on with nova-live-migration jobs ? I got two CI failures on the same patch with the same error https://review.opendev.org/c/openstack/nova/+/886232 | 13:36 |
| stephenfin | bauzas: melwitt commented about that last week | 13:40 |
| stephenfin | <melwitt> fyi just got two CI failures on the same patch in the nova-live-migration and nova-grenade-multinode jobs https://bugs.launchpad.net/neutron/+bug/1940425 and it reminded me I have been seeing this failure often lately | 13:40 |
| stephenfin | <melwitt> we likely need to look into this to improve gate fail rate | 13:40 |
| bauzas | stephenfin: ack and the story is ? | 13:40 |
| bauzas | ok | 13:40 |
| * bauzas grabs gibi's opensearch | 13:40 | |
| bauzas | https://zuul.openstack.org/builds?job_name=nova-live-migration&skip=0 sounds pretty bad if you want MHO | 13:41 |
| bauzas | ralonsoh: lajoskatona: you'll be happy to hear we have some kind of neutron hard failure on nova-live-migration jobs | 13:42 |
| ralonsoh | bauzas, please! | 13:42 |
| bauzas | what a nice day, isn't it ? :) | 13:43 |
| lajoskatona | bauzas: this one perhaps: https://bugs.launchpad.net/neutron/+bug/2024160 ? | 13:43 |
| ralonsoh | do you know what is happening? | 13:43 |
| bauzas | ralonsoh: one of the failing logs https://ffd910c1fba2c4e789e5-da0bf4b2b0817e3979adc64c27f459a5.ssl.cf5.rackcdn.com/880719/5/check/nova-live-migration/08a7455/testr_results.html | 13:43 |
| ralonsoh | so this is related only with the trunk ports, right? | 13:43 |
| bauzas | yeah | 13:43 |
| ralonsoh | ok, we have the triage meeting in 20 mins | 13:43 |
| bauzas | tempest.api.compute.admin.test_live_migration.LiveMigrationTest is failing | 13:43 |
| bauzas | due to a failed assert | 13:43 |
| ralonsoh | I'll raise it as urgent | 13:43 |
| ralonsoh | can you disable this test temporarily? | 13:44 |
| ralonsoh | hold on | 13:44 |
| bauzas | lajoskatona: sounds related | 13:44 |
| ralonsoh | is this CI in a gate/check queue? | 13:44 |
| ralonsoh | I mean, is this blocking you from merging patches? | 13:44 |
| lajoskatona | ralonsoh, bauzas: yeah by the logs seems bug/2024160 , and to the OVN version bump | 13:45 |
| ralonsoh | bauzas, is this in a gate/check queue? is this blocking you from merging patches? | 13:46 |
| ralonsoh | if so, I'll disable this test temporarily | 13:46 |
| bauzas | ralonsoh: (sorry, was elsewhere) yeah this is a voting job | 13:48 |
| bauzas | on both check and gate | 13:48 |
| bauzas | Tempest seems impacted too | 13:48 |
| ralonsoh | ok, I'll disable make this test unstable temporarily | 13:48 |
| ralonsoh | but we'll start working on this right now | 13:49 |
| bauzas | I can add a skip test in zuul | 13:50 |
| bauzas | ideally, we should have a tempest bug report against it | 13:50 |
| bauzas | want me to declare it ? | 13:50 |
| ralonsoh | bauzas, isn't enough with the Neutron LP bug? | 13:51 |
| ralonsoh | I'm sending a patch right now for tempest | 13:51 |
| bauzas | ralonsoh: sorry I misread your last comment, I thought you said the opposite | 13:52 |
| ralonsoh | broken phone, I think | 13:52 |
| ralonsoh | sorry if I wasn't clear | 13:52 |
| ralonsoh | --> https://review.opendev.org/c/openstack/tempest/+/886496 | 13:56 |
| bauzas | ralonsoh++ | 13:56 |
| lajoskatona | +1 | 13:59 |
| bauzas | reminder : nova meeting in 1 hour-ish here | 15:02 |
| * gibi lurk | 16:01 | |
| bauzas | sorry | 16:01 |
| bauzas | #startmeeting nova | 16:01 |
| opendevmeet | Meeting started Tue Jun 20 16:01:36 2023 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:01 |
| opendevmeet | The meeting name has been set to 'nova' | 16:01 |
| bauzas | #topic Bugs (stuck/critical) | 16:01 |
| bauzas | #info No Critical bug | 16:01 |
| bauzas | whoops, forgot to tell howdy :) | 16:02 |
| bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | 16:02 |
| bauzas | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 27 new untriaged bugs (+9 since the last meeting) | 16:03 |
| bauzas | but last week was hectic :) | 16:03 |
| bauzas | so, let's try to look again this week | 16:03 |
| bauzas | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | 16:05 |
| bauzas | I'll try to look at them this week | 16:05 |
| bauzas | so, | 16:05 |
| bauzas | #info bug baton is being passed to bauzas | 16:05 |
| bauzas | any bug to raise ? | 16:05 |
| bauzas | ok, moving on | 16:06 |
| bauzas | #topic Gate status | 16:06 |
| bauzas | #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure Nova gate bugs | 16:06 |
| bauzas | we have a new winner this week | 16:06 |
| bauzas | in case you haven't seen it, | 16:06 |
| bauzas | #link https://lists.openstack.org/pipermail/openstack-discuss/2023-June/034176.html Hold your rechecks ! | 16:06 |
| bauzas | I'd appreciate if tempest cores could quickly approve https://review.opendev.org/c/openstack/tempest/+/886496 | 16:07 |
| bauzas | #link https://etherpad.opendev.org/p/nova-ci-failures | 16:07 |
| bauzas | people wanting to discuss it now ? | 16:08 |
| * gibi is wondering if it is only bauzas and gibi is in the room | 16:09 | |
| bauzas | meh | 16:09 |
| bauzas | at least I'll write this for people looking at our meeting logs: p | 16:09 |
| bauzas | :p | 16:09 |
| bauzas | but yeah, quickly moving then | 16:10 |
| bauzas | #link https://zuul.openstack.org/builds?project=openstack%2Fnova&project=openstack%2Fplacement&pipeline=periodic-weekly Nova&Placement periodic jobs status | 16:10 |
| * gmann here but in another meeting in parallel | 16:10 | |
| bauzas | all greens, nothing to relate | 16:10 |
| bauzas | #info Please look at the gate failures and file a bug report with the gate-failure tag. | 16:10 |
| bauzas | #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures | 16:10 |
| bauzas | there we go | 16:10 |
| bauzas | #topic Release Planning | 16:10 |
| bauzas | #link https://releases.openstack.org/bobcat/schedule.html | 16:10 |
| bauzas | #info Nova deadlines are set in the above schedule | 16:10 |
| bauzas | #info Nova spec review day next Tuesday (June 27th) | 16:10 |
| bauzas | #action bauzas to notify the ML about the spec review day | 16:11 |
| bauzas | and as a reminder : | 16:11 |
| bauzas | #link https://releases.openstack.org/bobcat/schedule.html#b-nova-spec-freeze Nova Spec Freeze on July 6th ! | 16:11 |
| bauzas | so, basically folks, we have 3 weeks for approving specs | 16:11 |
| bauzas | that said, iirc the cinder spec freeze is this week :) | 16:11 |
| bauzas | nothing to tell about the above ? | 16:12 |
| gibi | I see some movement on the SRIOV spec but not ready yet | 16:13 |
| bauzas | indeed | 16:14 |
| bauzas | anyway, moving on | 16:14 |
| bauzas | #topic Forum/PTG feedback | 16:15 |
| bauzas | #link https://lists.openstack.org/pipermail/openstack-discuss/2023-June/034152.html | 16:15 |
| bauzas | in case you haven't read it yet :) | 16:15 |
| bauzas | we won't do a retrospective, we don't have a quorum here :) | 16:15 |
| bauzas | and actually, all the folks around were on the Summit too :p | 16:15 |
| bauzas | so, next topic | 16:16 |
| gibi | :) | 16:17 |
| bauzas | #topic Review priorities | 16:18 |
| bauzas | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2) | 16:18 |
| bauzas | #info As a reminder, cores eager to review changes can +1 to indicate their interest, +2 for committing to the review | 16:18 |
| bauzas | #topic Stable Branches | 16:18 |
| bauzas | I haven't seen elodilles_pto since a while and his nick tells me something | 16:18 |
| bauzas | maybe "pto" means something like "people to obligate" :) | 16:19 |
| bauzas | but meh | 16:19 |
| bauzas | let's say he's not here | 16:20 |
| bauzas | so I won't tell anything about the stable branches | 16:20 |
| bauzas | maybe just | 16:20 |
| bauzas | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | 16:20 |
| bauzas | but I also have | 16:20 |
| bauzas | #info train-eol patch proposed https://review.opendev.org/c/openstack/releases/+/885365 | 16:20 |
| bauzas | so, there was a question in ^ | 16:21 |
| bauzas | should we just abandon the existing changes in this branch ? | 16:21 |
| bauzas | gibi: would be nice if you could vote about it | 16:21 |
| bauzas | ok, last topic | 16:23 |
| bauzas | #topic Open discussion | 16:23 |
| bauzas | (bauzas) Specless blueprint approval request for https://blueprints.launchpad.net/nova/+spec/num-instances-weigher | 16:23 |
| bauzas | given we don't have quorum today, I'll ask him again next week | 16:23 |
| bauzas | I'm done for me, anything else? | 16:24 |
| * gibi voten on the EOL patch now | 16:24 | |
| gibi | I have nothing extra for today | 16:25 |
| bauzas | ok, so, | 16:26 |
| bauzas | thanks folks | 16:26 |
| bauzas | #endmeeting | 16:26 |
| opendevmeet | Meeting ended Tue Jun 20 16:26:15 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:26 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2023/nova.2023-06-20-16.01.html | 16:26 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2023/nova.2023-06-20-16.01.txt | 16:26 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2023/nova.2023-06-20-16.01.log.html | 16:26 |
| bauzas | was quick and silent :) | 16:27 |
| Uggla_ | yep but nothing special this week I guess | 16:28 |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (db) https://review.opendev.org/c/openstack/nova/+/831193 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (objects) https://review.opendev.org/c/openstack/nova/+/839401 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction) https://review.opendev.org/c/openstack/nova/+/831194 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part) https://review.opendev.org/c/openstack/nova/+/833090 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Mounting the shares as part of the initialization process https://review.opendev.org/c/openstack/nova/+/880075 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Deletion of associated share mappings on instance deletion https://review.opendev.org/c/openstack/nova/+/881472 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add metadata for shares https://review.opendev.org/c/openstack/nova/+/850500 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/854823 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/854824 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add share_info parameter to resume method for each driver (driver part) https://review.opendev.org/c/openstack/nova/+/860284 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Support resuming an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860285 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add helper methods to rescue/unrescue shares https://review.opendev.org/c/openstack/nova/+/860286 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares (driver part) https://review.opendev.org/c/openstack/nova/+/860287 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Support rescuing an instance with shares (compute manager part) https://review.opendev.org/c/openstack/nova/+/860288 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Allow to mount manila share using Cephfs protocol https://review.opendev.org/c/openstack/nova/+/883862 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Check shares support (compute manager) https://review.opendev.org/c/openstack/nova/+/885751 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Check shares support (only API exception) https://review.opendev.org/c/openstack/nova/+/885752 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (API) https://review.opendev.org/c/openstack/nova/+/836830 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Check shares support (API) https://review.opendev.org/c/openstack/nova/+/850499 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add helper methods to attach/detach shares https://review.opendev.org/c/openstack/nova/+/885753 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach notification https://review.opendev.org/c/openstack/nova/+/850501 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach notification https://review.opendev.org/c/openstack/nova/+/851028 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add shares to InstancePayload https://review.opendev.org/c/openstack/nova/+/851029 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_attach_error notification https://review.opendev.org/c/openstack/nova/+/860282 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add instance.share_detach_error notification https://review.opendev.org/c/openstack/nova/+/860283 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working. https://review.opendev.org/c/openstack/nova/+/852086 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Add virt/libvirt error test cases https://review.opendev.org/c/openstack/nova/+/852087 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Docs about Manila shares API usage https://review.opendev.org/c/openstack/nova/+/871642 | 16:34 |
| opendevreview | ribaudr proposed openstack/nova master: Allow config to support virtiofs (driver) https://review.opendev.org/c/openstack/nova/+/886522 | 16:34 |
| gibi | it was almost like the PTG :) | 16:35 |
| sean-k-mooney | ouch | 16:36 |
| sean-k-mooney | i can feel the heat form that burn form here | 16:36 |
| gibi | :) | 16:43 |
| dansmith | heh | 16:54 |
| opendevreview | melanie witt proposed openstack/nova master: database: Archive parent and child rows "trees" one at a time https://review.opendev.org/c/openstack/nova/+/877056 | 19:34 |
| opendevreview | melanie witt proposed openstack/nova master: database: Archive parent and child rows "trees" one at a time https://review.opendev.org/c/openstack/nova/+/877056 | 20:04 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
