| auniyal | Hi | 04:37 |
|---|---|---|
| auniyal | please review these - | 04:37 |
| auniyal | https://review.opendev.org/c/openstack/nova/+/853811 | 04:37 |
| auniyal | https://review.opendev.org/c/openstack/nova/+/853812 | 04:37 |
| *** tkajinam|off is now known as tkajinam | 05:48 | |
| sean-k-mooney[m] | gibi: i have mostly figured out the issue with my test. its not the migration stub im calling _detach_passthough_vifs in this code path and that was not being mocked by the fixture | 07:43 |
| gibi | ahh, OK | 07:43 |
| sean-k-mooney[m] | if i mock that the migration succeed although my post condtions fail | 07:43 |
| sean-k-mooney[m] | so i need to mock it better | 07:43 |
| sean-k-mooney[m] | or perhaps just the detach call | 07:44 |
| sean-k-mooney[m] | im still not sure why this fails in my case and not the sriov case | 07:44 |
| sean-k-mooney[m] | or in my detach test | 07:44 |
| sean-k-mooney[m] | but thats the issue | 07:45 |
| opendevreview | sean mooney proposed openstack/nova master: Add VDPA support for suspend and livemigrate https://review.opendev.org/c/openstack/nova/+/853704 | 08:10 |
| sean-k-mooney[m] | im just running ^ now locally | 08:12 |
| sean-k-mooney[m] | that should resolve the issue with the tests and i have dropped the service version release mapping | 08:12 |
| gibi | sean-k-mooney[m]: I'm +2 and fingers crossed for the test | 08:13 |
| sean-k-mooney[m] | the issue once i mocked it was the _live_migrate helper did not return the server object | 08:14 |
| sean-k-mooney[m] | so i fixed that and then the assert worked | 08:14 |
| sean-k-mooney[m] | hum that apprently broke one functional test im going to leave the unit test finish then ill check that again | 08:16 |
| sean-k-mooney[m] | im not sure why wapping the order of the waits would have that effect | 08:16 |
| sean-k-mooney[m] | i can restore the order of the waits and store teh server responce in a local var | 08:17 |
| gibi | or return the server from the _live_migrate helper | 08:18 |
| sean-k-mooney[m] | well i am returning it from _live_migrate | 08:19 |
| gibi | ahh yes | 08:19 |
| sean-k-mooney[m] | https://review.opendev.org/c/openstack/nova/+/853704/6/nova/tests/functional/integrated_helpers.py | 08:19 |
| gibi | would be nice to check the code which state transtion happens last, the migration or the server | 08:19 |
| gibi | and encode that order in the helper | 08:20 |
| sean-k-mooney[m] | the order should not break other test right? | 08:20 |
| gibi | if that make a test case fail then we probably have a bug somwhere | 08:20 |
| gibi | yes, it should not | 08:20 |
| gibi | except if the returned server is asserted | 08:20 |
| gibi | but it is not yet in the expected strate | 08:20 |
| gibi | state | 08:20 |
| gibi | or vice versa | 08:20 |
| gibi | hm | 08:21 |
| gibi | actually what you do is the correct one I think | 08:21 |
| gibi | as you return the server after the server state is changed | 08:21 |
| gibi | so I guess the bug is in that other func test case | 08:21 |
| sean-k-mooney[m] | ya that is why i reordered it | 08:21 |
| sean-k-mooney[m] | i think so ya just getting its name | 08:22 |
| sean-k-mooney[m] | i think its an ovs hardware offload test | 08:22 |
| sean-k-mooney[m] | ya test_rollback_pre_live_migration | 08:23 |
| sean-k-mooney[m] | ya its a regression but its expected the server state to be migrating and migration state to be failed | 08:29 |
| sean-k-mooney[m] | but the server state is actully now active | 08:29 |
| sean-k-mooney[m] | because i am now waitign until after the migation has failed and we have rolled back | 08:29 |
| sean-k-mooney[m] | so if you are doing fault injection the order actully can matter but this is the only test out of all the unit and fuctional tests that fail | 08:30 |
| sean-k-mooney[m] | so im just going to update it | 08:30 |
| opendevreview | sean mooney proposed openstack/nova master: Add VDPA support for suspend and livemigrate https://review.opendev.org/c/openstack/nova/+/853704 | 08:32 |
| sean-k-mooney[m] | ok i have been looking at that since 6 am and i have not had coffee this morning so im going to go correct that | 08:34 |
| sean-k-mooney[m] | brb | 08:34 |
| gibi | yeah, that is a valid fix for the test case | 08:41 |
| Uggla | Hi gibi , do we have the nova meeting this evening ? | 08:44 |
| gibi | Uggla: it is tuesday so yes | 08:44 |
| gibi | I will chair it | 08:44 |
| gibi | as the last two | 08:44 |
| Uggla | ok cool | 08:44 |
| sean-k-mooney[m] | still embing caffine but assuming im done with vdpa for today. what seriese should i review first | 08:47 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Basics for PCI Placement reporting https://review.opendev.org/c/openstack/nova/+/846187 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Extend device_spec with resource_class and traits https://review.opendev.org/c/openstack/nova/+/846218 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reject PCI dependent device config https://review.opendev.org/c/openstack/nova/+/846435 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reject mixed VF rc and trait config https://review.opendev.org/c/openstack/nova/+/846436 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Ignore PCI devs with physical_network tag https://review.opendev.org/c/openstack/nova/+/846219 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reject devname based device_spec config https://review.opendev.org/c/openstack/nova/+/846466 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Support [pci]device_spec reconfiguration https://review.opendev.org/c/openstack/nova/+/846470 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Stop if tracking is disable after it was enabled before https://review.opendev.org/c/openstack/nova/+/847009 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Move provider_tree RP creation to PciResourceProvider https://review.opendev.org/c/openstack/nova/+/850546 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Retry /reshape at provider generation conflict https://review.opendev.org/c/openstack/nova/+/851358 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Heal missing simple PCI allocation in the resource tracker https://review.opendev.org/c/openstack/nova/+/851359 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Heal PCI allocation during resize https://review.opendev.org/c/openstack/nova/+/852396 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Handle PCI dev reconf with allocations https://review.opendev.org/c/openstack/nova/+/852397 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement https://review.opendev.org/c/openstack/nova/+/850468 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Generate request_id for Flavor based InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/853835 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Request filter for PCI in placement https://review.opendev.org/c/openstack/nova/+/852771 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias https://review.opendev.org/c/openstack/nova/+/853316 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs https://review.opendev.org/c/openstack/nova/+/854118 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters https://review.opendev.org/c/openstack/nova/+/854119 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation https://review.opendev.org/c/openstack/nova/+/854120 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/854121 | 09:06 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling https://review.opendev.org/c/openstack/nova/+/854122 | 09:07 |
| gibi | sean-k-mooney[m]: the pci stuff can be reviewed up until https://review.opendev.org/c/openstack/nova/+/851358/ and even further if you have the willpower | 09:09 |
| gibi | the top of the series missing unit test coverage and has spotty functional coverage though | 09:09 |
| sean-k-mooney | ack | 09:09 |
| sean-k-mooney | ok so that is everything up to just before the allocation healling | 09:10 |
| opendevreview | Jan Hartkopf proposed openstack/python-novaclient master: add support for microversion 2.93 https://review.opendev.org/c/openstack/python-novaclient/+/816158 | 09:14 |
| Uggla | gibi, sean-k-mooney may I request you to check https://review.opendev.org/c/openstack/nova/+/853986 and merge it if ok. | 09:25 |
| gibi | Uggla: done, and thanks | 10:18 |
| sean-k-mooney | gibi: by the way i have started to notice tempest.api.volume.test_volumes_extend.VolumesExtendAttachedTest.test_extend_attached_volume failing semi often in nova-next | 10:20 |
| Uggla | gibi, thank you ! | 10:20 |
| gibi | sean-k-mooney: do you have an example failure? | 10:21 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/5e64ad2283a24653a424ad91d99a3004 | 10:21 |
| gibi | ack, I will quickly check | 10:21 |
| sean-k-mooney | it failed on teh vdpa patch | 10:21 |
| sean-k-mooney | but i have seen it on several others recnetly | 10:21 |
| sean-k-mooney | do you know where the new elastic search deployment is hosted | 10:21 |
| sean-k-mooney | im kind of wonderign how often this happens | 10:22 |
| gibi | interesting | 10:23 |
| gibi | https://zuul.opendev.org/t/openstack/build/5e64ad2283a24653a424ad91d99a3004/log/compute1/logs/screen-n-cpu.txt#52040 | 10:23 |
| gibi | libvirt.libvirtError: invalid argument: disk '/dev/sda' was not found in the domain config | 10:23 |
| sean-k-mooney | ya | 10:23 |
| sean-k-mooney | thats the same failure | 10:23 |
| gibi | sean-k-mooney: https://opensearch.logs.openstack.org/_dashboards/app/login?nextUrl=%2F_dashboards%2Fapp%2Fdiscover%2F%3Fsecurity_tenant%3Dglobal this is the logstash | 10:24 |
| sean-k-mooney | we dont guarentee the adress of th disk | 10:24 |
| sean-k-mooney | ack i was trying http://logstash.openstack.org/ | 10:24 |
| sean-k-mooney | which is obviously dead but maybe infra coudl add a redirect to the new location | 10:24 |
| gibi | yah a redirect would be nice | 10:25 |
| gibi | hm from the last 10 novan-next failure 9 was due to the above error | 10:27 |
| sean-k-mooney | do you know the login info? | 10:27 |
| sean-k-mooney | i think they sent one ot the mail list but i cant find it in my email | 10:27 |
| sean-k-mooney | gibi: ya so we have a log message that says "the libvirt driver cannot honner the request path" or somethign like that when we attach so im supresed we care when detaching | 10:28 |
| gibi | openstack openstack | 10:28 |
| sean-k-mooney | thanks | 10:29 |
| gibi | it is almost like it started failing on 08.19. | 10:30 |
| gibi | and we merged https://review.opendev.org/c/openstack/nova/+/843700/ in 19th | 10:30 |
| sean-k-mooney | yep | 10:31 |
| gibi | so I think we enabled an unstable test | 10:31 |
| gibi | I'm proposing a revert | 10:31 |
| sean-k-mooney | i can revert that i guess | 10:32 |
| sean-k-mooney | ah go for it | 10:32 |
| sean-k-mooney | so its calling | 10:33 |
| sean-k-mooney | def attach_detach_volume(self, server, volume): | 10:33 |
| sean-k-mooney | attached_volume = self.nova_volume_attach(server, volume) | 10:33 |
| sean-k-mooney | self.nova_volume_detach(server, attached_volume) | 10:33 |
| sean-k-mooney | if that is not waiting for sshable then that is l ikely the issue | 10:33 |
| sean-k-mooney | https://review.opendev.org/c/openstack/tempest/+/836065/4/tempest/scenario/test_encrypted_cinder_volumes.py | 10:34 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Revert "Test attached volume extend actions in the nova-next job" https://review.opendev.org/c/openstack/nova/+/854132 | 10:34 |
| sean-k-mooney | i wonder is this another case of that type of error | 10:34 |
| gibi | yeah that can be one issue yes | 10:35 |
| sean-k-mooney | we might have removed it form teh persitent tomain but not the active domain | 10:35 |
| sean-k-mooney | and fail with not found on a retry or something | 10:35 |
| sean-k-mooney | i wonder if the logs shows anything like that | 10:35 |
| sean-k-mooney | hum no its the resize that is failing | 10:36 |
| sean-k-mooney | but perhaps the attach has not completed | 10:36 |
| gibi | I'm table this for now. we can merge the revert or somebody can propose a fix to the tempest test but I let that be done by others :) | 10:38 |
| sean-k-mooney | sure :) | 10:38 |
| gibi | (we actually did a naked recheck that showed this error to merge the original patche enabling this test case) | 10:41 |
| sean-k-mooney | thats unfortunet | 10:42 |
| opendevreview | Merged openstack/nova master: Alphabetizes objects https://review.opendev.org/c/openstack/nova/+/853986 | 10:43 |
| gibi | that shows why we are against naked rechecks :D | 10:44 |
| sean-k-mooney | i left a comment on the tempest chagne https://review.opendev.org/c/openstack/tempest/+/836065/4#message-a8c9c9794c75555728517230fcab3b7beb8427de | 10:46 |
| gibi | thanks | 10:46 |
| sean-k-mooney | is logsearch a cli for logstash/opensearch ? | 10:48 |
| sean-k-mooney | that looks like its quite nice to use vs the ui | 10:49 |
| sean-k-mooney | the ui is not bad bu the cli output is very short and too the point | 10:49 |
| gibi | nope it is my own tool grepping logs locally | 10:50 |
| gibi | https://github.com/gibizer/zuul-log-search | 10:50 |
| sean-k-mooney | oh ok | 10:50 |
| sean-k-mooney | so its pulling the logs locally | 10:50 |
| gibi | that is one of my hack and hustle target | 10:50 |
| gibi | yes it pulls the logs locally based on the zuul query you make | 10:51 |
| gibi | caches it locally | 10:51 |
| gibi | and can grep in it | 10:51 |
| gibi | also you can define bug signatures | 10:51 |
| sean-k-mooney | ah nice | 10:51 |
| gibi | and it can try to classify failures by stored signatures | 10:51 |
| gibi | it was my answer to shutting down the old logsearch instance | 10:52 |
| sean-k-mooney | ya | 10:52 |
| gibi | *logstash | 10:52 |
| sean-k-mooney | only downside is disk space really | 10:52 |
| sean-k-mooney | but that is relitivly cheap | 10:53 |
| sean-k-mooney | when you do --file compute1/logs/screen-n-cpu.txt --file controller/logs/screen-n-cpu.txt | 10:53 |
| sean-k-mooney | is that going to just pull those files | 10:53 |
| gibi | yes | 10:53 |
| gibi | by default it only pulls the job-output.txt | 10:54 |
| gibi | to limit disk space usage :) | 10:54 |
| sean-k-mooney | ok so as logn as you knwo what file you want it wont take that mauch space at all | 10:54 |
| sean-k-mooney | ill have to find some time to set it up and try it out | 10:54 |
| gibi | my query for the above failure in nova-next in the last 25 days takes ~1G local disk space | 10:55 |
| * sean-k-mooney is slightly sad we had 1G of failure logs in 25days | 10:55 | |
| gibi | the compute service logs are pretty big | 10:56 |
| gibi | 53 compute service logs == 1G | 10:57 |
| sean-k-mooney | ya i guess they are in debug | 10:57 |
| gibi | but I can put the cache dir to a compressed volume and that can safe a lot of space | 10:57 |
| gibi | yes fortunately gate use debug :D | 10:57 |
| gibi | s/safe/save/ | 10:57 |
| sean-k-mooney | i rememebr the intel nfv ci used to use a 300G log share that we sotred gziped logs too | 10:58 |
| sean-k-mooney | that was enough for about 60 days fo ci logs | 10:58 |
| sean-k-mooney | once you compressed everything | 10:58 |
| gibi | yes these logs are highl compressable | 10:59 |
| sean-k-mooney | like db dumps. the db dump i was working with for the downstream escalation was about 900MB raw or about 60 mb compressed | 11:00 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Retry /reshape at provider generation conflict https://review.opendev.org/c/openstack/nova/+/851358 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Heal missing simple PCI allocation in the resource tracker https://review.opendev.org/c/openstack/nova/+/851359 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Heal PCI allocation during resize https://review.opendev.org/c/openstack/nova/+/852396 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Handle PCI dev reconf with allocations https://review.opendev.org/c/openstack/nova/+/852397 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement https://review.opendev.org/c/openstack/nova/+/850468 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Generate request_id for Flavor based InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/853835 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Request filter for PCI in placement https://review.opendev.org/c/openstack/nova/+/852771 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias https://review.opendev.org/c/openstack/nova/+/853316 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs https://review.opendev.org/c/openstack/nova/+/854118 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters https://review.opendev.org/c/openstack/nova/+/854119 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation https://review.opendev.org/c/openstack/nova/+/854120 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/854121 | 11:44 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling https://review.opendev.org/c/openstack/nova/+/854122 | 11:44 |
| gibi | sean-k-mooney, stephenfin: solved the todo in https://review.opendev.org/c/openstack/nova/+/851358 so now the pci series is ready for review up until https://review.opendev.org/c/openstack/nova/+/853835/ the rest of the series only ready for high level architectural review, as those patches does not have proper test coverage. The last patch of the series adds some functional tests that proves that | 11:46 |
| gibi | the scheduling logic is sane though | 11:46 |
| sean-k-mooney | ack | 11:48 |
| sean-k-mooney | i was looking at some other stuff but ill start on it soon | 11:48 |
| gibi | ack | 11:48 |
| gibi | stephenfin, sean-k-mooney: I'm really thankful for all these reviews. I know that this is a lot of code and logic | 11:49 |
| gibi | ... and now a quick lunch | 11:50 |
| opendevreview | Amit Uniyal proposed openstack/nova master: Adds check for VM snapshot fail while quiesce https://review.opendev.org/c/openstack/nova/+/852171 | 13:02 |
| opendevreview | Elod Illes proposed openstack/nova stable/stein: DNM: check CI health https://review.opendev.org/c/openstack/nova/+/854184 | 13:12 |
| *** dasm|off is now known as dasm | 13:28 | |
| sean-k-mooney | stephenfin: o/ | 14:40 |
| sean-k-mooney | stephenfin: just an fyi i figured out the issue with the live migration test and that is now passing | 14:41 |
| sean-k-mooney | melwitt: are you around | 14:41 |
| sean-k-mooney | melwitt: if so can you take a look at this revert https://review.opendev.org/c/openstack/nova/+/854132 | 14:41 |
| sean-k-mooney | its failing pretty consitently | 14:42 |
| sean-k-mooney | im holding off recheckign the base patch in teh vdpa seriese until that is merged because it will likely get hit by it again | 14:42 |
| gibi | elodilles: I've updated the nova meeting agenda for today | 15:22 |
| elodilles | gibi: i've just updated it :D | 15:22 |
| gibi | cool, thanks | 15:22 |
| elodilles | gibi: hopefully i did not overwrite any of your changes :S | 15:23 |
| gibi | It does not seem so | 15:24 |
| elodilles | \o/ | 15:27 |
| gibi | nova meeting in 2 minutes | 15:58 |
| gibi | #startmeeting nvoa | 16:00 |
| opendevmeet | Meeting started Tue Aug 23 16:00:12 2022 UTC and is due to finish in 60 minutes. The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'nvoa' | 16:00 |
| gibi | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Tue Aug 23 16:00:17 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nvoa/2022/nvoa.2022-08-23-16.00.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nvoa/2022/nvoa.2022-08-23-16.00.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nvoa/2022/nvoa.2022-08-23-16.00.log.html | 16:00 |
| gibi | #startmeeting nova | 16:00 |
| opendevmeet | Meeting started Tue Aug 23 16:00:19 2022 UTC and is due to finish in 60 minutes. The chair is gibi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'nova' | 16:00 |
| gibi | o/ | 16:00 |
| elodilles | o/ | 16:01 |
| sean-k-mooney | o/ | 16:01 |
| gibi | so lets start | 16:03 |
| Uggla | o/ | 16:03 |
| gibi | #topic Bugs (stuck/critical) | 16:03 |
| gibi | #info One Critical bug | 16:03 |
| gibi | https://bugs.launchpad.net/nova/+bug/1986545 | 16:03 |
| gibi | this is the same from last week | 16:03 |
| gibi | about the novncproxy | 16:03 |
| gibi | melwitt has a fix for the unit test | 16:03 |
| gibi | and I'm +2 | 16:03 |
| gibi | https://review.opendev.org/c/openstack/nova/+/853379 | 16:04 |
| gibi | so we need another core on it | 16:04 |
| gibi | #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New 8 new untriaged bugs (0 since the last meeting) | 16:05 |
| gibi | #link https://storyboard.openstack.org/#!/project/openstack/placement 27 open stories (+0 since the last meeting) in Storyboard for Placement | 16:05 |
| gibi | #info Add yourself in the team bug roster if you want to help https://etherpad.opendev.org/p/nova-bug-triage-roster | 16:06 |
| gibi | the next on the roster is artom | 16:06 |
| gibi | artom: will you take the triage baton? | 16:06 |
| artom | Yes | 16:06 |
| gibi | cool thanks | 16:06 |
| gibi | and thanks for the triages melwitt | 16:06 |
| gibi | #info bug baton is being passed to artom | 16:07 |
| gibi | any specific bug we need to discuss here/ | 16:07 |
| gibi | ? | 16:07 |
| gibi | #topic Gate status | 16:09 |
| gibi | we have an unstable test in nova-next enabled recently. We are reverting the patch enabled it #link https://review.opendev.org/c/openstack/nova/+/854132 | 16:09 |
| gibi | this also needs a second core | 16:09 |
| gibi | until that revert lands nova-next will be fairly but not totally red | 16:10 |
| gibi | #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly Placement periodic job status | 16:11 |
| gibi | it is green | 16:11 |
| gibi | #link https://zuul.openstack.org/builds?job_name=tempest-integrated-compute-centos-9-stream&project=openstack%2Fnova&pipeline=periodic-weekly&skip=0 Centos 9 Stream periodic job status | 16:11 |
| gibi | green too | 16:11 |
| gibi | #link https://zuul.opendev.org/t/openstack/builds?job_name=nova-emulation&pipeline=periodic-weekly&skip=0 Emulation periodic job runs | 16:11 |
| gibi | and green as well | 16:11 |
| gibi | #info Please look at the gate failures and file a bug report with the gate-failure tag. | 16:11 |
| gibi | #info STOP DOING BLIND RECHECKS aka. 'recheck' https://docs.openstack.org/project-team-guide/testing.html#how-to-handle-test-failures | 16:11 |
| gibi | as a note the current nova-next instability could have been caught if we had stopped using blind rechecks :/ | 16:12 |
| gibi | anything else about the master gate status? | 16:12 |
| JayF | I have a change, https://review.opendev.org/c/openstack/nova/+/853529, which should help with Ironic changes by not running nova-live-migration tests if only Ironic driver is modified. This should help conserve resources and prevent rechecks by avoiding the tests that can't be broken by Ironic. It's my intention to backport this if/when merged. It only needs one additional | 16:12 |
| JayF | core vote. | 16:12 |
| gibi | JayF: cool added to my list, I will check it after the meeting | 16:13 |
| JayF | thank you gibi, I appreciate it. | 16:13 |
| gibi | anything else gate related? | 16:13 |
| sean-k-mooney | just yor revert | 16:13 |
| sean-k-mooney | https://review.opendev.org/c/openstack/nova/+/854132 | 16:13 |
| sean-k-mooney | if we can merge that before it wedges the gate it would be good | 16:13 |
| sean-k-mooney | we have 40 hits today | 16:13 |
| gibi | sean-k-mooney: yes, we need melwitt or gmann or stephenfin to look at it and approve the revert | 16:14 |
| sean-k-mooney | or dansmith if your about | 16:15 |
| dansmith | looking | 16:15 |
| gibi | thanks | 16:16 |
| gibi | anything else about the gate? | 16:16 |
| dansmith | got it | 16:16 |
| sean-k-mooney | i dont think so | 16:16 |
| sean-k-mooney | i have not see any other recurrent failures | 16:17 |
| gibi | ack, moving on | 16:17 |
| gibi | #topic Release Planning | 16:17 |
| gibi | #link https://releases.openstack.org/zed/schedule.html | 16:17 |
| gibi | #info Zed-3 is in 1 weeks | 16:17 |
| gibi | #link Zed tracking etherpad: https://etherpad.opendev.org/p/nova-zed-blueprint-status | 16:17 |
| gibi | #link https://etherpad.opendev.org/p/nova-zed-microversions-plan | 16:17 |
| gibi | please update the https://etherpad.opendev.org/p/nova-zed-blueprint-status etherpad with the status of your work | 16:18 |
| gibi | and if you have someting that blocking your progress then raise it | 16:19 |
| gibi | is there any specific bp we want to talk about today? | 16:20 |
| gibi | I see that https://review.opendev.org/q/topic:bp/volume-backed-server-rebuild+status:open is green but lacking core review | 16:22 |
| dansmith | I initially did a bunch of review on it, | 16:23 |
| dansmith | but was hoping to get another core to have fresh eyes on it before we merge | 16:23 |
| sean-k-mooney | the tempet test is red is that related to teh testcases they are adding | 16:23 |
| dansmith | so I'm good to +W that if someone else looks it over, but I had my hands all in it, so I think that'd be better | 16:23 |
| dansmith | sean-k-mooney: is it? it was passing at one point | 16:23 |
| opendevreview | Merged openstack/nova stable/ussuri: [ironic] Minimize window for a resource provider to be lost https://review.opendev.org/c/openstack/nova/+/853540 | 16:24 |
| gibi | it is behind https://review.opendev.org/q/topic:bp/update-userdata+status:open where I need to go back and re-review. after that I might be able to look at the rebuild sets but no hard promises | 16:24 |
| sean-k-mooney | well some of the jobs on the tempest repo are red but i dont know if the new ones worked | 16:24 |
| dansmith | the runs logs are too old, so it needs a recheck | 16:24 |
| sean-k-mooney | ack shall i do it or have you already done it | 16:25 |
| dansmith | I done did it | 16:25 |
| sean-k-mooney | cool | 16:25 |
| dansmith | seems unlikely to merge at this late date if it's behind something else for a microversion, | 16:25 |
| dansmith | but that kinda sucks because a lot was done on it | 16:25 |
| dansmith | so if someone has time to look that'd be nice, maybe we can FFE it if we need | 16:26 |
| sean-k-mooney | well it coudl be merge ahead of the other patch | 16:26 |
| sean-k-mooney | we might want to swap them since there is an openquestion on the update user data pathc | 16:26 |
| gibi | I have the intention to go back the the update user_data patch (that is the next microversion) and also melwitt reviewe that before | 16:26 |
| dansmith | it's a cross-project thing where cinder has done a bunch of work to enable it, so it definitely sucks to cause them to miss again | 16:26 |
| gibi | so we might be able to unblock the it | 16:27 |
| dansmith | already missed last cycle, IIRC | 16:27 |
| sean-k-mooney | dansmith: there is only one patch in the other fature to merge | 16:27 |
| sean-k-mooney | but yes it woudl be shame if it missed | 16:27 |
| dansmith | okay well if we can get someone to look it over while we decide or work on the other, that'd be cool | 16:28 |
| dansmith | maybe melwitt would want to look at it | 16:28 |
| gibi | ack | 16:28 |
| gibi | any other bp to raise? | 16:29 |
| sean-k-mooney | am jsut i think the teanat id one is likely not going to land | 16:30 |
| gibi | it is in merge conflict since we merged the unittest.mock change ~3weeks ago | 16:30 |
| sean-k-mooney | dansmith: i kind of lost track of the SRBAC stuff are there any reverts that still need to land for that | 16:30 |
| dansmith | um | 16:30 |
| sean-k-mooney | its not really a blue print but basically is there any work we need to have in zed | 16:31 |
| dansmith | I think it merged underneath something from gmann | 16:31 |
| sean-k-mooney | ok | 16:31 |
| dansmith | but I'm not seeing it actually | 16:32 |
| dansmith | nope: https://review.opendev.org/c/openstack/nova/+/848021 | 16:32 |
| dansmith | that one and the one above | 16:32 |
| sean-k-mooney | ack so those should be set as a review priorty to land for zed | 16:32 |
| dansmith | maybe we can get gmann to +2 that and someone else to ack it, yeah | 16:32 |
| sean-k-mooney | is gmann on pto i have not spoken to them in a few days | 16:33 |
| dansmith | I think something is keeping him busy, but he was around the last few days a little | 16:33 |
| gibi | I added that rbac stuff to the tracking etherpad | 16:34 |
| dansmith | gibi: thanks | 16:34 |
| sean-k-mooney | i think thats the main set that we needed to talk about | 16:34 |
| gibi | ack then moving on | 16:35 |
| gibi | #topic Review priorities | 16:35 |
| gibi | #link https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement+OR+project:openstack/os-traits+OR+project:openstack/os-resource-classes+OR+project:openstack/os-vif+OR+project:openstack/python-novaclient+OR+project:openstack/osc-placement)+(label:Review-Priority%252B1+OR+label:Review-Priority%252B2) | 16:35 |
| gibi | looks fairly relevant | 16:35 |
| gibi | #topic Stable Branches | 16:36 |
| gibi | elodilles: any news from stable/ | 16:36 |
| gibi | ? | 16:36 |
| elodilles | yepp | 16:36 |
| elodilles | #info stable/train was unblocked last week (with the backport of the patch that migrated jobs to zuul v3) | 16:36 |
| elodilles | #info now only stable/stein (and older) are blocked: grenade and other devstack based jobs fail with the same timeout issue as stable/train was previously | 16:36 |
| elodilles | #info stable branch status / gate failures tracking etherpad: https://etherpad.opendev.org/p/nova-stable-branch-ci | 16:36 |
| sean-k-mooney | elodilles i think there is an issue with cliff on yoga and older by the way | 16:37 |
| sean-k-mooney | more for cilometer then us | 16:37 |
| elodilles | sean-k-mooney: is it? :-o | 16:37 |
| sean-k-mooney | but cliff 4. something droped python 3.6 support | 16:37 |
| sean-k-mooney | so centos 8 stream jobs are failing | 16:37 |
| sean-k-mooney | i did see it on stable/victoria on nova however if i am rememebring correctly | 16:38 |
| elodilles | isn't that non-voting? | 16:38 |
| elodilles | i mean in that case it's not blocking the gate | 16:38 |
| sean-k-mooney | not for nova | 16:38 |
| sean-k-mooney | i work with peopel that work on ceilometer that are trigng to fix it for there jobs | 16:39 |
| sean-k-mooney | just mentioning it as an fyi incase you see stabel review related to it | 16:39 |
| elodilles | sean-k-mooney: ack, thanks, i'll look into that | 16:39 |
| sean-k-mooney | https://review.opendev.org/c/openstack/requirements/+/854130 | 16:40 |
| elodilles | sean-k-mooney: ++ | 16:40 |
| elodilles | i've noted that patch | 16:40 |
| gibi | anything else about stable? | 16:41 |
| elodilles | nothing else from me | 16:41 |
| gibi | thanks elodilles | 16:41 |
| elodilles | np | 16:41 |
| gibi | #topic Open discussion | 16:42 |
| gibi | nothing on the agenda. | 16:42 |
| gibi | does anybody has a topic still for today? | 16:42 |
| gibi | I take the silence as no | 16:43 |
| gibi | so thanks for joining | 16:43 |
| gibi | #endmeeting | 16:43 |
| opendevmeet | Meeting ended Tue Aug 23 16:43:35 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:43 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2022/nova.2022-08-23-16.00.html | 16:43 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2022/nova.2022-08-23-16.00.txt | 16:43 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2022/nova.2022-08-23-16.00.log.html | 16:43 |
| elodilles | thanks o/ | 16:43 |
| sean-k-mooney | elodilles: i put the details in the etherpad | 16:47 |
| sean-k-mooney | https://etherpad.opendev.org/p/nova-stable-branch-ci#L22 | 16:47 |
| JayF | btw; thanks to you all for the help with the stable reviews, trying to get the ironic driver as good as it can be in the stable branch. https://review.opendev.org/c/openstack/nova/+/821350 still needs another core vote in victoria | 16:50 |
| JayF | I know you rarely need us, but I'm happy to help if you all run into Ironic shenanigans in the future :D | 16:50 |
| gibi | sean-k-mooney: fyi I created a test case and now I can confirm that different PFs are tracked as separate PCI pools even if they have same kind of VFs. Fortunately both the apply and the consume logic in stats able to consume from two pools for a single pci request. So everything works \o/ even with splitted pools | 16:51 |
| sean-k-mooney | well there is one open spec that we really need your import on | 16:51 |
| sean-k-mooney | gibi: nice | 16:52 |
| sean-k-mooney | JayF: if you have not read https://review.opendev.org/c/openstack/nova-specs/+/842015 | 16:52 |
| sean-k-mooney | JayF: we dont currently have a good way forward on that | 16:52 |
| JayF | I've absolutely had backchannel chats with Julia about that (and felt the pain it is trying to solve). I will scratch my brain for it again but I'm not sure I know how to break out of the logjam | 16:53 |
| sean-k-mooney | we were hopign to have a face to face session on this at the ptg before it went virtual | 16:54 |
| sean-k-mooney | it really feels like it needs a white board and some time to talk thing through | 16:54 |
| JayF | I have $800 of Alaska Airlines credit proving my intention to have showed up to that :( | 16:54 |
| JayF | hopefully we can find time in virtual summit, anyway, although you're right it's often easier to solve things irl | 16:55 |
| sean-k-mooney | +2w on that victoria backport | 16:56 |
| * sean-k-mooney it may have taken me longer to workout what baranch comes before and after V then to review | 16:57 | |
| JayF | wonderful, thanks. I'll have probably three fresh in the morning as I pull everything down another level. | 16:57 |
| JayF | sean-k-mooney: https://etherpad.opendev.org/p/JayFTODO I put all the branches on this so I'd stop doing that | 16:57 |
| JayF | sean-k-mooney: it's worse for Ironic, we have mid-stream bugfix releases that are numbered, and they might as well be named after colors for all the meaning the names have for me lolsob | 16:57 |
| sean-k-mooney | i see :) | 16:58 |
| sean-k-mooney | ok im gong to more or less call it a day. | 17:01 |
| sean-k-mooney | ill likely do some code review on my ipad while i cook/order dinner but im going to drop off irc ffor the day | 17:01 |
| gibi | sean-k-mooney: have a nice evening o/ | 17:01 |
| opendevreview | Elod Illes proposed openstack/nova stable/train: [ironic] Minimize window for a resource provider to be lost https://review.opendev.org/c/openstack/nova/+/853546 | 17:15 |
| JayF | elodilles: thanks, I had that on my todo list but I appreciate it \o/ | 17:16 |
| elodilles | JayF: np, just re-applied the cherry-pick to have the right format of the commit message, otherwise it seems right, so +2'd it already | 17:22 |
| JayF | So, what you want in the commit is | 17:22 |
| elodilles | it's not the commit message in general o:) | 17:23 |
| JayF | cherry-picked-from (master SHA)\ncherry-picked-from (master-1 sha)\n etc etc | 17:23 |
| JayF | which is achieved by a clean cherry pick from stable N -> stable N-1 | 17:23 |
| elodilles | but to do the cherry-picking branch-by-branch, | 17:23 |
| elodilles | so that one can see that the cherry picking was in the right order, from the right (latest) patch sets, etc. | 17:24 |
| JayF | so for my most recent master change, I'd cherry-pick it to stable/yoga, then cherry pick *the yoga change* onto xena, and so on | 17:25 |
| JayF | and if at any point one patch in the cycle is changed or rebased (whoops), the ones after it have to be re-cherry-picked | 17:25 |
| elodilles | JayF: exactly :) | 17:26 |
| JayF | is this a nova-ism? Or something generally that most OpenStack projects do that Ironic has never really enforced? | 17:26 |
| elodilles | the backports should be done branch-by-branch according to stable policy | 17:27 |
| JayF | ack, so I'll update how I do it in Ironic world too | 17:28 |
| elodilles | but yes, probably only nova has this check as (we) stable cores were picky about this one o:) | 17:28 |
| JayF | Yeah, I've been focused on stable reviews in Ironic for a long time, and this is *never* something I've checked for | 17:29 |
| elodilles | to avoid wrong/not latest/missing backports | 17:29 |
| JayF | frankly, I doubt I'd -1 a change for this now -- but I do always manually check the earlier branches | 17:29 |
| JayF | and I'll start doing it right for me ;D | 17:29 |
| JayF | **my changes | 17:29 |
| elodilles | :) | 17:30 |
| elodilles | also note, that the rebase is unnecessary as Zuul is always doing it. the only time when it is needed, when there's a merge conflict | 17:32 |
| JayF | tbh rebase button is my shortcut for "clear verify and re-run tests on this" except now I realize there are negative side effects in this context | 17:43 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Request filter for PCI in placement https://review.opendev.org/c/openstack/nova/+/852771 | 17:59 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias https://review.opendev.org/c/openstack/nova/+/853316 | 17:59 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs https://review.opendev.org/c/openstack/nova/+/854118 | 17:59 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters https://review.opendev.org/c/openstack/nova/+/854119 | 17:59 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation https://review.opendev.org/c/openstack/nova/+/854120 | 17:59 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest https://review.opendev.org/c/openstack/nova/+/854121 | 17:59 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling https://review.opendev.org/c/openstack/nova/+/854122 | 17:59 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Test PCI scheduling during move operations https://review.opendev.org/c/openstack/nova/+/854247 | 17:59 |
| opendevreview | Merged openstack/nova master: Revert "Test attached volume extend actions in the nova-next job" https://review.opendev.org/c/openstack/nova/+/854132 | 18:08 |
| opendevreview | Elod Illes proposed openstack/nova stable/queens: WIP: [stable-only] Remove grenade jobs https://review.opendev.org/c/openstack/nova/+/854252 | 18:23 |
| opendevreview | Merged openstack/nova master: nova-live-migration tests not needed for Ironic https://review.opendev.org/c/openstack/nova/+/853529 | 18:47 |
| opendevreview | Merged openstack/nova stable/victoria: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/821350 | 18:47 |
| opendevreview | Jay Faulkner proposed openstack/nova stable/ussuri: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/821351 | 19:14 |
| JayF | https://review.opendev.org/c/openstack/nova/+/853546 (to stable/train) needs one more core review, then this one will be backported as far as I plan to take it | 19:15 |
| opendevreview | Jay Faulkner proposed openstack/nova stable/yoga: nova-live-migration tests not needed for Ironic https://review.opendev.org/c/openstack/nova/+/854257 | 19:16 |
| opendevreview | Jay Faulkner proposed openstack/nova stable/yoga: nova-live-migration tests not needed for Ironic https://review.opendev.org/c/openstack/nova/+/854257 | 19:17 |
| opendevreview | Jay Faulkner proposed openstack/nova stable/ussuri: Ignore plug_vifs on the ironic driver https://review.opendev.org/c/openstack/nova/+/821351 | 19:18 |
| JayF | elodilles: took a couple of tries, but I think I got them right now ^^ had to do two rounds, first used `-x` then I realized I needed the `-X` :D | 19:20 |
| JayF | ty for the guidance | 19:20 |
| JayF | https://review.opendev.org/c/openstack/nova/+/821351 and https://review.opendev.org/c/openstack/nova/+/854257 should both be good for reviews now, no-change-backports of stuff already merged, shouldn't be controversial :D | 19:22 |
| JayF | thank you all again for helping me plow thru this ironic-driver-backport tech debt | 19:22 |
| melwitt | sean-k-mooney, gibi: I'm off today and tomorrow, thanks for getting the revert done and sorry for the trouble ☹️ | 22:08 |
| *** dasm is now known as dasm|off | 22:29 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!