Wednesday, 2022-07-27

opendevreviewmelanie witt proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption  https://review.opendev.org/c/openstack/nova/+/82675506:10
gibio/06:49
opendevreviewTakashi Natsume proposed openstack/nova-specs master: Create specs directory for 2023.1 Antelope  https://review.opendev.org/c/openstack/nova-specs/+/85100707:00
bauzas\o07:25
gibibauzas: I left feedback in https://review.opendev.org/c/openstack/nova/+/83897607:42
bauzasgibi: will look soon08:11
opendevreviewwangkuntian proposed openstack/nova master: Modify the url of openstack client commands.  https://review.opendev.org/c/openstack/nova/+/85119709:02
gibistephenfin, sean-k-mooney: we have valid py310 failures in the mock switch patch09:39
gibiwe can debate that py310 is non voting, so we can merge regardless but as the failure is valid we just pushing the problem to our future slef09:41
gibiself09:41
kashyapgibi: Morning, can you give this another go? - I've addressed your concerns: https://review.opendev.org/c/openstack/nova/+/85103409:42
gibisure09:42
gibidone +209:43
gibithanks for the releasenotes09:43
kashyapgibi: Thank you!  Now to figure out the backports09:44
gibiI expect it to be painless as you have a well contained change09:44
kashyapYeah, checking09:48
kashyapsean-k-mooney: or bauzas - Can you put it through, pls - https://review.opendev.org/c/openstack/nova/+/85103409:49
bauzasack09:49
kashyapbauzas: Just for info: the above code was functionally (in a real env) tested by Red Hat support folks09:51
kashyapgibi: Heh, Yoga backport itself has conflicts :D /me goes to look09:54
kashyapIt's simple, though09:54
opendevreviewKashyap Chamarthy proposed openstack/nova stable/yoga: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85120209:58
opendevreviewStephen Finucane proposed openstack/nova master: Remove the PowerVM driver  https://review.opendev.org/c/openstack/nova/+/85034610:00
bauzaskashyap: -1 just for the relnote https://review.opendev.org/c/openstack/nova/+/85103410:09
bauzasI could accept this change, but I'd change the note to use 'feature' and not 'upgrade'10:10
bauzasso given we're not close to the deadline, that's why I'm asking just for this nit10:10
bauzasat least for operators looking at our releasenotes10:11
kashyapbauzas: FWIW, I've actually seen other rel-notes for such things and chose the 'upgrade' thing10:11
kashyapbauzas: As this is mostly useful during that scenario10:11
sean-k-mooney[m]both points are valid10:12
opendevreviewKashyap Chamarthy proposed openstack/nova stable/xena: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85120510:12
bauzaskashyap: https://docs.openstack.org/nova/latest/contributor/releasenotes.html?highlight=reno#when-a-release-note-is-needed10:12
sean-k-mooney[m]it is a new feature but its also a workaround that is only usefule for upgrades and limited other cases10:13
kashyapsean-k-mooney[m]: bauzas: I could use "other" flag, then10:13
bauzaskashyap: honestly, my thought is just about operators looking at our notes10:13
bauzasI know some of them10:13
kashyapbauzas: Yeah, I get that10:13
bauzasin general, they look at two sections10:13
kashyapbauzas: How about "other", then?10:13
bauzaskashyap: if you want10:14
bauzasmy only concern was about the 'upgrade' section10:14
kashyapbauzas: Just vote - you and Sean :)10:14
bauzasas I was saying, operators look at two sections10:14
kashyap"feature" or "other"10:14
kashyapI'll respin right away (inncluding the backports)10:14
bauzasthey look at the prelude section (for knowing what we have) and at the upgrade section (to know what they need to verify)10:15
bauzashere, they don't need to verify anything10:15
bauzashence my concern10:15
bauzaskashyap: about 'feature' or 'other', meh.10:16
kashyapbauzas: Sure, I'll go w/ 'feature'10:17
bauzasthanks10:18
bauzasappreciated10:18
opendevreviewKashyap Chamarthy proposed openstack/nova master: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85103410:20
opendevreviewKashyap Chamarthy proposed openstack/nova stable/yoga: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85120210:20
opendevreviewKashyap Chamarthy proposed openstack/nova stable/xena: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85120510:21
kashyapbauzas: --^10:31
bauzaskashyap: you're late10:31
kashyapHehe10:31
kashyapThank you!10:31
kashyapbauzas: Can you also ACK the backports, then, please?10:32
bauzasI can10:32
kashyapThx10:34
kashyapsean-k-mooney[m]: Unrelated - I see this is still active, can you pls ACK this too: "[nova/libvirt] Support for checking and enabling SMM when needed10:34
kashyap"10:34
kashyaphttps://review.opendev.org/c/openstack/nova/+/84961010:34
kashyap(Cc: gibi)10:35
* kashyap apologizes for several pings today10:35
opendevreviewKashyap Chamarthy proposed openstack/nova stable/wallaby: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85120610:36
sean-k-mooneykashyap: i wasnt going to review teh wallaby backport until the xena one had merged10:40
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/84967610:40
kashyapSure10:40
kashyapBut it's the same thing10:40
sean-k-mooneyyes but it cant merge until that one does so i review them in order10:41
kashyapSigh, also this stable/yoga backport is not merged yet - https://review.opendev.org/c/openstack/nova/+/845045 (libvirt: Add a workaround to skip compareCPU() on destination10:41
kashyap)10:41
kashyapsean-k-mooney: Oh, sure10:41
opendevreviewMerged openstack/nova master: Allow unshelve to a specific host (Compute API part)  https://review.opendev.org/c/openstack/nova/+/83150711:44
opendevreviewMerged openstack/nova master: Allow unshelve to a specific host (REST API part)  https://review.opendev.org/c/openstack/nova/+/84589711:45
gibi\o/11:45
sean-k-mooneystephenfin: gibi  how do we want to proceed with the mock change?11:46
gibisean-k-mooney: good question11:46
sean-k-mooneyim kind of sad that we chose to delay it last cycle11:47
sean-k-mooneyto drop 3.611:47
gibisean-k-mooney: I'm not sure I will have enough time today to try to fix the duplicate mock issue11:47
sean-k-mooneyand now we added 3.10 which is causing other issues11:47
gibiI don't know if stephenfin looked at it yet or not11:47
gibiI can be convinced to let it land and fix the 310 issue separately I just affraid that we will forget about it and only realize it again when we want to make the py310 job votinh11:49
gibig11:49
sean-k-mooneyya thats a valid concen11:50
sean-k-mooneythe 3.10 fixes shoudl be a seperate patch in my opipion11:50
sean-k-mooneyit would be nice ot have it in the seriese11:50
gibiI agree with that11:50
sean-k-mooneybut not sure how much work it is11:50
sean-k-mooneyim also not partically happy with the decision they made in 3.1011:51
gibime neither but I feel like it is not a oneliner11:51
sean-k-mooneydouble mocking is perhaps ineffenct but i think it should be allowed11:51
sean-k-mooneyso im sad they made that breakign change11:51
gibilets wait for stephenfin to chime in. Also I can try to look into it tomorrow to see how big is the issue11:52
gibithen we can reconsider11:52
sean-k-mooneywe might need to add a helper funciton that check if its mocked and only create a new mock if not11:52
gibiI would go and remove the duplicates if possible11:52
gibibut I don't know how hard that will be11:53
sean-k-mooneyyes if we can do that once11:53
sean-k-mooneythen we will never hit this again because it will fail11:53
gibithe helper feels like a coverup11:53
sean-k-mooneyim just worried about loading the context of is this mocked 11:54
sean-k-mooneyagain with the new behaivor it shoudl be obvious i guess because it will fail11:54
gibiyeah11:54
sean-k-mooneyi just hope this does not cause  shared global state11:56
sean-k-mooneythe fixture should be set up for every test11:57
sean-k-mooneybut we likely will have to replace part of them in the test where its double mocking11:57
sean-k-mooneyas i feel like we double mock somethimes when the fixture does not provide exactly what we want11:58
sean-k-mooneyi know i have done that at least once11:58
gibiyes, I think that is the "normal" case when we double mock. I did that before11:58
gibiso in this case the fixture should expose the mock11:58
gibiso the test case can use it instead of remocking it11:58
sean-k-mooneywell we can jsut create a new mock object and assign it11:59
sean-k-mooneyinstaed of having it patch it again11:59
gibiwe cannot create the new Mock as autospec will fail11:59
gibias it will try to spec what is already mocked11:59
sean-k-mooneywe can with mock.Mock()12:00
gibihm12:00
sean-k-mooneywe dont tend to use autospec much12:00
sean-k-mooneywe do in somecases but i rarely do12:00
gibiA().foo = mock.Mock() is a monkeypatch that is not reverted automatically so that is a bad pattern12:01
gibialso I think there is automatic autospecing in the mock lib (obviously not for mock.Mock() but for mock.patch decorator)12:01
sean-k-mooneyi dont think there is12:02
sean-k-mooneyor at least12:02
sean-k-mooneyits behavior shoudl be the same as the standard lib12:02
sean-k-mooneyso if there is it shoudl be in both12:02
sean-k-mooneygibi: in the A().foo = mock.Mock()12:02
sean-k-mooneywe shoudl not need to revert that automaticlly if A is a mock already12:03
gibiahh, yeah, I ment there is automatic autospec in both lib and stdlib12:03
gibisean-k-mooney: yeah, if A().foo is already a mocked then that mock's patcher.stop() will revert the change. But the caller doing the A().foo = mock.Mock() does not know if foo is a Mock. So we establish a dangerous pattern12:04
gibiif that pattern is used to other than a mocked field the we will leak global state between tests12:05
sean-k-mooneygibi: right but we would only do that for the double mock case12:06
sean-k-mooneybut i agreee in general its bad12:06
sean-k-mooneywe have leaked state that way in the past12:06
gibiyes, now, but then the next new dev came look at the code and think that A().foo = mock.Mock() is a generic pattern12:06
sean-k-mooneywhere a mock was assinged to a stdlib function12:07
sean-k-mooneyso that is why i was wondering if we need a nova helper fucniton12:07
sean-k-mooneyto basiclaly always do the right thing12:07
gibibut that mean all our mocking need to go through that helper12:07
gibithat will be a massive change12:07
sean-k-mooneyperhaps i need to think about what pattern we can actully use that will work12:08
gibianyhow I'd like to do a non theorethicaly investigation with this12:08
sean-k-mooneyperhaps using the mock as a context manager12:08
sean-k-mooneybut not sure if that would still fail for the double mocking case12:08
gibibtw this is why I remembered automatic autospeccing https://review.opendev.org/q/message:autospec+project:openstack/nova+status:merged12:09
sean-k-mooney sure but that not automatic12:09
sean-k-mooneythat was exiplcitly enabling it and new right12:10
gibiyeah, I missremembered12:10
sean-k-mooneyoh ok12:10
gibiI will look at this mock thing tomorrow12:10
gibinow I'm in a middle fighting with reshape + provider generation conflict12:11
gibiI'm in the denial phase where I think my code is correct and reshape is broken :D12:12
sean-k-mooney:)12:12
sean-k-mooneymaybe but if thats the case you will probly have to fix it in placment yes12:13
sean-k-mooneyso hopefully the issue is in your current code12:13
gibiI know :D12:13
sean-k-mooneythat feels like less work12:13
gibihence "denial"12:13
gibianyhow I think this is the first case when we both create new RPs and reshape allocations at the same time12:13
gibifrom nova12:14
gibiand RP creation is outside of the reshape code12:14
sean-k-mooneyif nova has not done it its likely no one else has12:15
sean-k-mooneyi dont think neutron or cyborg have used reshapes before12:15
gibime neither12:16
bauzasI just capture the discussion12:50
bauzasis it because unittest.mock is not on par with our mock lib ?12:51
gibiit is that in py310 unittest.mock introduced a breaking change. The mock lib is a rolling backport of the upstream unittest.mock so at some point this change will appera in the standalone mock lib too 12:52
kashyapbauzas: Please put the stable/wallaby backport through: https://review.opendev.org/c/openstack/nova/+/85120612:52
kashyapgibi ACKed it too; and you've ACKed the upstream main12:52
gibikashyap: we need to land the backports in order12:53
gibiso we land the yoga first12:53
gibithen xena12:53
gibithen wallaby12:53
kashyapgibi: Oh, yes.  I mean to also implicitly look at 'em.  :)12:54
gibiack12:54
bauzasgibi: which breaking change ?12:54
gibiif you mock a function twice, the second mocking attempt will fail12:54
gibias autospecing a mock is disallowed now12:55
bauzasthen we should wait for our mock lib 12:55
gibiwhat? why?12:55
gibiwhy it is better to get the same breaking change later?12:56
bauzaswe should change the related tests first due to this 12:56
bauzasgibi: well, once our mock lib would be modified, then the CI wouldn't work and we would see it so we could not accept the new release12:57
bauzasthen we could modify the tests12:57
bauzasfor this new release12:57
bauzasand then accepting the new release12:57
gibithis is the same thing. If we want to accept py310 it means we need to accept all the breaking changes there. 12:58
gibiso we are only debating when to fix those duplicated mocks12:59
bauzasat least we need to modify the only needed tests12:59
gibiand as the mock switch is ready now I think we should fix it now12:59
bauzasbefore changing to use unittest.mock12:59
gibiwhy before? we can switch to unittest.mock and then fix the duplicated mock to get the py310 non voting job gren13:00
gibigreen again13:00
gibiswitching first make py310 job telling us where are the duplicates13:01
admin1i have nova using local disk , but glance and cinder uses ceph .. when using snapshots, i get broken pipe from nova compute .. is there any special way ceph needs to be configured for snapshots to work ? 13:01
sean-k-mooneyadmin1: no in that config nova should upload the snapshot via the glace http api13:12
sean-k-mooneyso it will create a snapshot using qemu-image then upload that disk13:12
opendevreviewSylvain Bauza proposed openstack/nova master: api: Drop generating a keypair and add special chars to naming  https://review.opendev.org/c/openstack/nova/+/84913313:19
bauzasrebased my keypair api microversion due to unshelve_to_host merge ^13:20
bauzaspeople can review it13:20
opendevreviewKashyap Chamarthy proposed openstack/nova stable/victoria: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85122313:27
gibibauzas: +2, do you have an python binding patch and or osc patch with the client side change?13:31
bauzasnot yet13:31
opendevreviewKashyap Chamarthy proposed openstack/nova stable/victoria: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85122313:31
gibiUggla: similar question to you too, do you have a client patch for the unshelve to host change?13:32
gibibauzas: ack13:32
sean-k-mooneytechnically we dont need thos to merge the nova change but ya woudl be nice to see them sooner rather then later13:32
Ugglagibi, yes give you the ids in one sec13:32
gibiin the past we said we would at least like to see the client patches proposed (not merged) before we approve a microversion 13:32
gibiUggla: awesome13:32
sean-k-mooneyill be dropping soon but ill see if i have time to review before i do13:32
opendevreviewKashyap Chamarthy proposed openstack/nova stable/ussuri: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85122413:33
Ugglagibi: https://review.opendev.org/c/openstack/python-novaclient/+/831651 and https://review.opendev.org/c/openstack/python-openstackclient/+/83190213:34
gibiUggla: thanks13:34
Ugglagibi, there is also a related tempest test: 841088: Tempest test for checking unshelve to host | https://review.opendev.org/c/openstack/tempest/+/84108813:35
gibiack13:35
opendevreviewKashyap Chamarthy proposed openstack/nova stable/train: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85122613:38
kashyapbauzas: gibi: If you have the stomach for it today still (other backports): https://review.opendev.org/q/Iec387dcbc49ddb91ebf5cfd188224eaf6021c0e113:40
bauzasgibi: kashyap: was working on the novaclient change13:40
kashyapNo worries13:40
opendevreviewMerged openstack/nova master: Add a workaround to skip hypervisor version check on LM  https://review.opendev.org/c/openstack/nova/+/85103413:46
gibiUggla: I left feedback in the python-novaclient patch and +1d the osc patch. nice work!14:04
Ugglagibi, ok I'll have a look ASAP.14:05
*** akekane_ is now known as abhishekk14:12
opendevreviewSylvain Bauza proposed openstack/python-novaclient master: Add support for 2.92 : keypair import mandatory  https://review.opendev.org/c/openstack/python-novaclient/+/85123114:12
bauzasgibi: ^14:13
gibibauzas: thanks14:13
bauzasnow working on the OSC patch14:13
gibibauzas: you probably want to put your novaclient patch top of Uggla's14:14
bauzasgibi: there is a depends-on my api patch14:14
bauzasgibi: but I can rebase it14:14
gibiyeah, keep the depends on, but rebase your client patch on Uggla's client patch 14:15
gibiotherwise you will be in merge conflict14:15
bauzasgibi: cool, yeah I know about the merge conflict but the fix will be simple14:16
gibiyepp14:16
gibiothervise your client patch looks good to me14:16
opendevreviewSylvain Bauza proposed openstack/python-novaclient master: Add support for 2.92 : keypair import mandatory  https://review.opendev.org/c/openstack/python-novaclient/+/85123114:20
bauzasgibi: done14:20
gibibauzas: thanks, reapplied my +@14:20
gibi+214:20
bauzasthanks14:21
* bauzas needs to have a home haircut14:21
* bauzas is back in 20 mins14:21
gibiyour home has hair? ;)14:21
*** dasm|off is now known as dasm14:42
bauzasgibi: sorry, haircut *at* home :p14:45
bauzasmy wife and I have a hairdresser visiting uds14:46
gibiohh, nice14:46
gibimy wife tend to cut mine14:46
bauzascheaper and better14:46
gibithat is the cheapest :)14:46
bauzasyeah but if I use a hair mower, then my hair will have issues :)14:47
bauzasI prefer to have a pro using scissors14:48
artomgibi, so rollback_live_migration_at_destination() is the cast that contains the mutated migrated context, right?15:48
artomSo what if we just change it to a call instead of cast to make it blocking?15:48
artomThat way, we're sure that by the time we drop the claim, we're out of the mutated context?15:48
gibithat is a dirty thing that will mean we will have a short time window where the db contains the wrong info, then then the second call will return the db content to the correct state15:49
gibiso we will have a smaller race window15:49
gibibut we still have the race15:49
artomYou're assuming that mutating the context saves to the DB?15:50
artomI don't think that's the case15:50
gibiwe mutate the context and then save the instance today15:50
gibijust the mutate is not an issue15:50
gibithe instance.save happens in _cleanup in the driver15:50
gibiand that is run under the mutated context15:51
gibilinks to the code are here https://bugs.launchpad.net/nova/+bug/1982497/comments/215:52
artomgibi, oh, right15:53
artomWe mutate the context in the compute manager, and in that context we eventually call down to instance.save() in the libvirt driver15:53
gibihence my second patch that forbids instance.save under the mutated context to avoid this in the fututre15:53
gibiartom: yes15:53
artomWait, so why did you jump through the sync loops in your reproducer test?15:54
artomThe Condition(), for example?15:54
gibiI needed a special serialization order to make sure that we avoid the case when we save the wrong state, then right after we save the good state. 15:54
artomAh15:55
gibi_rollback_live_migration makes the DB invalid then drop_move_claim_at_destination fixes it. 15:55
gibiso in that order it is just a small race window (same as if we would make _rollback_live_migration a call instead of a cast)15:56
gibiinterestingly in a real environment _rollback_live_migration tend to take longer and finishing _after_  drop_move_claim_at_destination probably due to _rollback_live_migration doing disk IO that can be slow15:57
artomI mean we could just remove the instance.save() from the cleanrup()...15:58
artomAnd count on the compute manager doing it later...15:58
artomSounds fragile15:58
gibiartom: yepp that is one way, we can do that and then figure out the fallout 15:59
gibiit seems we have that instance.save as delete_instance_files() can fail and we want to retry16:00
gibilater in a periodic if that fails16:00
gibiunfortunately _cleanup not just set instance.cleaned but also init instance.system_metadata['clean_attempts']  that is used in _run_pending_deletes16:02
gibiand honestly not 100% sure if that is the only instance update we do during the full destroy codepath16:02
gibiI think that is the only instance.save but maybe other pieces of code also changes the instance that we would loose if we remove the instance.save16:03
melwittgmann: fyi the tempest tests for the volume extend coverage have merged, so I think the nova patch to start running the tests is ready for review https://review.opendev.org/c/openstack/nova/+/84370016:08
gmannmelwitt: checking16:08
melwittthanks!16:08
sean-k-mooneyim going to go nap soon i think but i might review the keypair change before or after16:57
sean-k-mooneyis that the next microverion in the queue16:57
gibit is17:14
gibiit is17:14
bauzasI'm just about to upload the OSC patch in 5 mins17:14
bauzasafter creating the relnote17:14
bauzasso we'll have all the meat for 2.92 except the tempest tests17:15
sean-k-mooney[m]ok cool17:15
bauzasWTF 17:21
bauzas[sbauza@sbauza python-openstackclient]$ git review -s17:22
bauzasProblems encountered installing commit-msg hook17:22
bauzasThe following command failed with exit code 25517:22
bauzas    "scp -P29418 sbauza@review.opendev.org:hooks/commit-msg .git/hooks/commit-msg"17:22
bauzasok, fixed by following https://www.mediawiki.org/wiki/Gerrit/Alternatives_to_git-review17:26
bauzasthere it goes : 17:27
bauzasgibi: sean-k-mooney: https://review.opendev.org/q/topic:bp%252Fkeypair-generation-removal17:27
bauzasall the changes are up17:27
* bauzas disappears for now17:28
sean-k-mooney[m]gmann:  holding +w to leave bauzas respond to your comments17:42
sean-k-mooney[m]i think they could be adressed in a followup17:42
*** dasm is now known as dasm|off21:02

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!