Friday, 2022-08-26

opendevreviewMerged openstack/nova master: Reject PCI dependent device config  https://review.opendev.org/c/openstack/nova/+/84643504:42
opendevreviewMerged openstack/nova master: Reject mixed VF rc and trait config  https://review.opendev.org/c/openstack/nova/+/84643604:46
opendevreviewMerged openstack/nova master: Ignore PCI devs with physical_network tag  https://review.opendev.org/c/openstack/nova/+/84621904:51
opendevreviewMerged openstack/nova master: Reject devname based device_spec config  https://review.opendev.org/c/openstack/nova/+/84646604:58
opendevreviewMerged openstack/nova master: Support [pci]device_spec reconfiguration  https://review.opendev.org/c/openstack/nova/+/84647004:58
opendevreviewMerged openstack/nova master: Stop if tracking is disable after it was enabled before  https://review.opendev.org/c/openstack/nova/+/84700904:59
opendevreviewMerged openstack/nova master: Move provider_tree RP creation to PciResourceProvider  https://review.opendev.org/c/openstack/nova/+/85054605:01
opendevreviewAmit Uniyal proposed openstack/nova master: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/85449905:30
opendevreviewRajesh Tailor proposed openstack/nova master: Fix rescue volume-based instance  https://review.opendev.org/c/openstack/nova/+/85273706:54
opendevreviewRajesh Tailor proposed openstack/nova master: Fix rescue volume-based instance  https://review.opendev.org/c/openstack/nova/+/85273706:59
gibigood morning07:02
gibiwow all the approved PCI patch merged. the gate is in a pretty good shape then07:02
Ugglagood morning07:39
sean-k-mooney1gibi: by the way you know how cursed the gate is going to be for the rest of the cycle ya11:45
*** sean-k-mooney1 is now known as sean-k-mooney11:47
opendevreviewAmit Uniyal proposed openstack/nova master: Adds a repoducer for post live migration fail  https://review.opendev.org/c/openstack/nova/+/85449912:14
opendevreviewAmit Uniyal proposed openstack/nova master: [compute] always set instnace.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/79113512:14
gibiI'm hopefule about the gate :)12:14
sean-k-mooneyi reviewed the bfv reuild patches today12:15
sean-k-mooneythe last 2 are mostly ok but the last one needs docs for the api change12:16
sean-k-mooneythe first needs some changes12:16
sean-k-mooneyis it intentional that we did not implemtne this for ironic?12:16
gibiI reviewed a buncs of manila today12:16
gibibunch12:16
sean-k-mooneywhich do you think makes sense for me to look at next manilla or encypted storage12:17
sean-k-mooneyi was going to look that encypted12:17
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (db)  https://review.opendev.org/c/openstack/nova/+/83119312:21
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (objects)  https://review.opendev.org/c/openstack/nova/+/83940112:21
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction)  https://review.opendev.org/c/openstack/nova/+/83119412:21
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part)  https://review.opendev.org/c/openstack/nova/+/83309012:21
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (api)  https://review.opendev.org/c/openstack/nova/+/83683012:21
opendevreviewribaudr proposed openstack/nova master: Bump compute version and check shares support  https://review.opendev.org/c/openstack/nova/+/85049912:21
opendevreviewribaudr proposed openstack/nova master: Add metadata for shares  https://review.opendev.org/c/openstack/nova/+/85050012:21
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach notification  https://review.opendev.org/c/openstack/nova/+/85050112:21
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach notification  https://review.opendev.org/c/openstack/nova/+/85102812:21
opendevreviewribaudr proposed openstack/nova master: Add shares to InstancePayload  https://review.opendev.org/c/openstack/nova/+/85102912:21
opendevreviewribaudr proposed openstack/nova master: Add instance.power_on_error notification  https://review.opendev.org/c/openstack/nova/+/85208412:21
opendevreviewribaudr proposed openstack/nova master: Add instance.power_off_error notification  https://review.opendev.org/c/openstack/nova/+/85227812:21
opendevreviewribaudr proposed openstack/nova master: Add helper methods to attach/detach shares  https://review.opendev.org/c/openstack/nova/+/85208512:21
opendevreviewribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working.  https://review.opendev.org/c/openstack/nova/+/85208612:21
opendevreviewribaudr proposed openstack/nova master: Add virt/libvirt error test cases  https://review.opendev.org/c/openstack/nova/+/85208712:21
opendevreviewribaudr proposed openstack/nova master: Change microversion to 2.93  https://review.opendev.org/c/openstack/nova/+/85208812:21
gibisean-k-mooney: encrypted12:21
gibisean-k-mooney: I can check the bfv after I finish with the manila api patach12:21
sean-k-mooneyi kicked the first 4 that are remaining into the queue 12:22
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/82675212:22
sean-k-mooneyis where melwitt is adding the encyption options which she has set -w on12:22
sean-k-mooneyill review some of the later patches but that will need ot be updated12:22
gibisean-k-mooney: how do you feel about this https://review.opendev.org/c/openstack/nova/+/836830/10/nova/api/openstack/compute/server_shares.py#54 ?12:26
gibisean-k-mooney: nova-api tries to resolve the compute hostname to ip address12:26
sean-k-mooneythat sound incorrect12:26
gibiyeah it feels strange12:27
sean-k-mooney for one there might be multipel interfaces and the compute host might connect to the starge backend via a differnt interface the we use for our inter service managment trafffic12:27
sean-k-mooneythe api also may not be able to resolve the compute ip in an iolsated networkign config12:28
gibiyes, also this socket call only works for ipv412:28
sean-k-mooneyyep so we shoudl not do this in the api and im not sure if we should do it in the comptue12:28
gibiunfortunately manila only takes IP as per https://docs.openstack.org/api-ref/shared-file-system/?expanded=grant-access-detail#grant-access12:29
sean-k-mooneywhat is this bieing used for 12:29
gibiit is to allow mounting the share to the compute12:29
sean-k-mooneywell we dont have to use that grant type12:29
sean-k-mooneyi was hoping to move to the cert one12:29
gibiahh I see12:29
sean-k-mooneyi think generating a tls cert for each share attachmetn woudl be the better approch longterm12:30
opendevreviewAmit Uniyal proposed openstack/nova master: [compute] always set instnace.host in post_livemigration  https://review.opendev.org/c/openstack/nova/+/79113512:31
sean-k-mooneyso for now i guess if we are using ip we shoudl do this in the compute12:32
sean-k-mooneynot the api12:32
sean-k-mooneyalthough we might need a new config option12:32
sean-k-mooneya share_access_network_ip12:32
sean-k-mooneywhich we can default based on a lookup using the hypervioer_hostname12:32
sean-k-mooneybut for those that have need to have a seperate network for this they can overried it12:33
sean-k-mooneylike we  do for the live migratieon inbound adress12:33
sean-k-mooneygibi: Uggla  ^ what do you think12:33
gibiso define one IP per compute for manila shares12:33
gibithat make sense12:33
sean-k-mooneyyes and default to socket.getIpform host or whatehever12:33
sean-k-mooneythere is a socket fucntion that gives your an ip form a hostname12:34
sean-k-mooneyso use that by default but allow the operator to provide an alternitive12:34
sean-k-mooneyif they have a dedicated storage network12:34
gibiyeah, although I might avoid that defaulting to avoid FQDN issues. and just document that if you need manila you need to define an IP there12:34
gibipush the issue up in the stack to the deployment engine to provide a proper IP ther12:34
gibie12:35
sean-k-mooneyok i was suggesting default ing becuase i think that is what we do for the live migration one but lets check12:35
sean-k-mooneywe have https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.my_ip12:35
sean-k-mooneyand https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.my_block_storage_ip12:35
sean-k-mooneyso i guess it should use the blockstorage one by default?12:35
sean-k-mooneyhum looks like that is only used in hyperv12:37
gibihm, that is already an IP, so I'm OK with that12:37
sean-k-mooneyso lets reuse that for maniall 12:37
sean-k-mooneyand just document that its now used by libvirt for this usecase too?12:37
sean-k-mooneyit also already has a default12:38
sean-k-mooneyhttps://github.com/openstack/nova/blob/0d84833e9688e0df97f3d24e06025e512bca3ce3/nova/conf/netconf.py#L24-L5112:38
sean-k-mooney=netutils.get_my_ipv4(12:38
gibiI don't like the defaulting, but I won't block on that12:38
sean-k-mooneywell i guess the intent is ot have it work out of the box12:39
sean-k-mooneyhttps://docs.openstack.org/nova/latest/configuration/config.html#libvirt.live_migration_inbound_addr was what i was thinking of orginally12:39
sean-k-mooneythat is not default because we use the hostname if not set12:40
gibiyeah I know we have these defaulting already, hence I'm accepting it, I just don't like that we guess what would be good instead of asking the deployer to provide what is good12:42
gibibut that is just a philosophy I can supress :)12:43
sean-k-mooneyi generally like things to work out of the box so i dont need to look at ooo to figure out how to set things12:43
sean-k-mooneybut ya i understand why you would like it to be explict12:44
gibiOK, I left comment in the API patch and linked to this IRC log for Uggla to look at12:44
gibisean-k-mooney: thanks12:44
Ugglasorry was looking at something else. I have just quicky read the chat. I think I get the idea.12:46
sean-k-mooneytl;dr to the grant in the compute and read CONF.my_block_storage_ip12:48
gibicool12:48
sean-k-mooneyand in the futre maybe we can use the cert grant instead but not sure how that works12:48
sean-k-mooneythe cert grant would limit the connection to that speicic vm and would not need to be updated if we move the vm12:49
sean-k-mooneyso more secure and less work in the long rune but we woudl need to figure out how to create the certs and set it up12:49
sean-k-mooneywhich is not for this cycle12:50
Ugglasean-k-mooney, agree.13:22
gibiI'm done with the bfv rebuild, I had some additional questions top of what you sean-k-mooney had13:35
sean-k-mooneyoh ya i forgot we have not drop the 5.x proxy code yet13:51
sean-k-mooneyalso good point on the functional test disbaing much of the code13:53
gibihonestly I did not check the unit testts13:53
sean-k-mooneywhile this techincaly coudl be libvirt indepentite it proably woudl be simmpler to use use the libvrit functional tests to test this more compeltely13:54
gibiI think the key there would be to assert cinder interaction13:54
gibiI don't care which driver fixture we use for it13:55
sean-k-mooneyright but we should not mock         with mock.patch.object(self.compute.manager,13:55
sean-k-mooney                               '_rebuild_volume_backed_instance'), \13:55
sean-k-mooneyin a functional test13:55
sean-k-mooneywe shoudl acutlly create a ocmpute service create an instnace and rebuild it as you said13:55
gibiyes13:57
gibiand assert the cinder fixture state after the rebuild13:57
gibiprobably need a some extra state in the cinder fixture to store what image is on th volume13:57
*** amorin_ is now known as amorin14:00
*** dasm|off is now known as dasm14:04
dansmithgibi: sean-k-mooney: I think the tempest test for the happy path is better than a functional, especially given the way it's verified,14:07
dansmiththe functional for testing the failure cases is better though obviously14:07
opendevreviewMerged openstack/nova master: virt: Add block_device_info helper to find encrypted disks  https://review.opendev.org/c/openstack/nova/+/82652914:08
sean-k-mooneywell its more that the functional test is closer to a unit test as written14:11
sean-k-mooneyi agree that the tempest test is going to be more valueable14:11
dansmithsean-k-mooney: you know the tempest test is written and working right? (just in case you didn't see it)14:12
gibiI agree too. I left some error case questing inline that would benefit from a functional test14:12
gibi*questiong14:12
gibi*question14:12
gibi(it is Friday)14:12
sean-k-mooneydansmith: yes i have seen it14:12
*** hemna6 is now known as hemna14:12
gibilike I'm not sure what will happen with the instance if the reimage fails 14:13
sean-k-mooneythe -1 i left on the last patch was more for the lack of docs by the way14:13
sean-k-mooneyspecifica the api ref 14:13
sean-k-mooneywe need to update that note that calls out how rebuidl for bfv is differnt14:13
dansmithI just hate that we do this to people.. they're fairly responsive for two cycles and then a week before the deadline, we finally review and play them the sad trombone14:13
dansmithI dunno how to fix that14:13
sean-k-mooneydansmith: actuly i didnt think they were responsive14:14
dansmithI spent a lot of time with him on it, but had several interruptions this cycle14:14
sean-k-mooneyits one of the reason i was not really looking at the patch beccause the comment i left on the tempset one had not been adress i did not end up looking at the nova ones14:14
dansmithsean-k-mooney: really? for also being the PTL of cinder I think he did pretty well and did everything I asked of him for a pretty complicated feature14:14
sean-k-mooneybaisicaly i check back on the tempest one a few times and didnt see much change and never got around to looking at the nova ones14:15
gibisorry for not reviewing it earlier but I hope it is better to review it late than never14:15
sean-k-mooneydansmith: the patches are in merge conflict one way or another so they need to actuly be update14:16
dansmithgibi: yes of course and definitely appreciated, it just sucks and I wish we (all) could do better.. more deadlines instead of fewer, I always say14:16
sean-k-mooneyi tought most of the issue were pretty minor that i pointed too and would not take that long14:16
gibidansmith: I agree on the more deadlines14:17
sean-k-mooneyalthoghg is it intentional that they did not implement ironic support14:17
dansmithsean-k-mooney: yeah, we knew it was going to need updates, and I'm not complaining about the comments (at all)14:17
dansmithsean-k-mooney: I think that's probably unintentional and I didn't even catch it14:18
gibidansmith: I think I could do better if there would be clear and agreed priority which feature to review first. As I think i filled my time with plenty of reviews in general14:18
sean-k-mooneyack we can just call that out as a limiation in the docs and fix it next cycle14:18
dansmithgibi: oh I know, I'm certainly not complaining about the amount of reviews being done :)14:18
sean-k-mooneyits just because ironic does not use the default implementation of the rebuild fucntion14:18
sean-k-mooneyit has its own 14:18
dansmithsean-k-mooney: yeah I know, but I also didn't know we had bfv with ironic :)14:19
JayFGood morning folks o/. Just wanted to bump my three outstanding stable ironic driver patches for review. https://review.opendev.org/c/openstack/nova/+/853546 https://review.opendev.org/c/openstack/nova/+/821351 https://review.opendev.org/c/openstack/nova/+/854257 all three are clean backports, and all but one already have one +214:21
gibidansmith: I slept on your comment about split the PCI feautre at the point where all the compute related part is ready. I figured that the current patch order does not really matches with that. I currently I have the other i) inventory healing, ii) allocation healing, ii) scheduling. But the scheduling part of the feauture needs compute side changes: 1) to driver the PCI claim based on the placment 14:23
gibiallocation 2) the pci and numa fitting logic is shared between the scheduler and the compute14:23
dansmithgibi: I wasn't really suggesting a reorder, I was more just commenting on what seams are flexible for backports and which aren't :)14:24
gibiso even if we could merge only the inventory and allocation healing without the scheduling support, backporting the scheduling support later is not really feasible14:24
gibior at least pretty shaky business14:25
gibifortunately I don't have to think about feature backport in this chat window :D14:25
opendevreviewMerged openstack/nova master: blockinfo: Add encryption details to the disk_info mappings when provided  https://review.opendev.org/c/openstack/nova/+/77227214:26
opendevreviewMerged openstack/nova master: imagebackend: Add disk_info_mapping as an optional attribute of Image  https://review.opendev.org/c/openstack/nova/+/82653014:26
opendevreviewMerged openstack/nova master: libvirt: Consolidate create_cow_image and create_image  https://review.opendev.org/c/openstack/nova/+/84624614:26
dansmithgibi: :)14:26
opendevreviewMerged openstack/nova stable/wallaby: add regression test case for bug 1978983  https://review.opendev.org/c/openstack/nova/+/85381114:27
dansmithgibi: sean-k-mooney: isn't there a planned train for service/microversions somewhere? I wonder if I could at least rebase this on the right thing to get it lined up for whoami-rajat 14:28
gibidansmith:  https://etherpad.opendev.org/p/nova-zed-microversions-plan14:33
dansmithyeah, thanks14:33
gibiI'm not against to reorder the next to microversion14:34
dansmithis that 2.93 one likely to get the review and attention it needs?14:34
gibiif the rebuild bfv become ready before the user_data update14:34
gibidansmith: I'm actively helping 2.39 and melwitt too14:35
dansmithlooks like it's getting attention.. yeah okay cool14:35
gibiI suggest to make rebuild bfv ready independently from the fact which microversion will it get. and it is ready before the user_data feature then we can switch the order in couple of hours14:36
dansmithoh yeah for sure,14:36
dansmithI just wanted to make sure that this was still the ordering before I rebase14:36
gibiI have no reason to reorder now as both feautre needs work14:36
dansmithyup14:37
gibialso don't want to step over our PTL :)14:37
dansmithagain, I just wanted to make sure I knew the right thing to rebase on, nothing more :)14:38
gibino worried :)14:38
gmanngibi: do you know till when bauzas is on PTO?14:40
gibilet me double check14:40
gibihe is back on the 30th14:41
gibibased on the RH internal calendar14:41
gmannok, just one day before PTL nomination close. let me rebase his nomination patch which he added before going to PTO.14:41
gibiack14:42
dansmithlooks like user_data is pretty far behind master at this point too, so I hope that gets rebased when it is updated14:46
gibiahh you already commented that to review so I don't need to :)14:48
dansmithyeah just did, typo and all14:49
opendevreviewMerged openstack/nova stable/wallaby: For evacuation, ignore if task_state is not None  https://review.opendev.org/c/openstack/nova/+/85381214:55
dansmithsean-k-mooney: your comment here says 64: https://review.opendev.org/c/openstack/nova/+/820368/32/nova/objects/service.py14:59
dansmithah, nevermind15:00
* dansmith gets out his calculator to add 63 and 115:00
sean-k-mooneynice power of 215:01
gibinow we can infer that your brain uses 6 bit ints internally15:01
dansmithokay I rebased that stack plus user data on master, but won't push user data of course15:02
dansmithbut this should apply cleanly once the user data author does15:03
gibicool15:03
dansmithsome conflicts in exception.py too15:04
dansmithis that author on irc?15:04
gibithe author's IRC nick was jhartkopf in the past15:05
gibihe was on nova before but not at the moment15:06
dansmithah15:06
sean-k-mooneythe os traits release i think is still pending but that should happen soon i hope15:07
gibisean-k-mooney: I talked about that with elodilles and he expect it to happen today15:07
sean-k-mooneycool15:08
sean-k-mooneydansmith: so the docs change i reqestied in the last patch could be in a followup which can merge after FF15:09
* dansmith nods15:09
sean-k-mooneyam i might even be able to jsut write that for them15:09
sean-k-mooneywe can call out this does not work for ironic htere too15:09
sean-k-mooneyand adress that next cycle15:09
sean-k-mooneyif you have fixed the compute service version then i think that was the main thing that need to be adress to resolve the merge conflict15:10
dansmithtbh, I'm not sure we even need to pass that to the virt driver anymore,15:10
dansmiththat might be a holdover from a previous approach to this15:10
dansmithoh right, envermind,15:11
dansmithbecause the default impl is in compute manager15:11
sean-k-mooneyyep15:11
dansmithI was eye-grepping for virt/*15:11
sean-k-mooneyironic has an imple of it15:11
sean-k-mooneybut i think only it does15:11
sean-k-mooneyso it does not fallback15:11
dansmiththat bottom patch fails unit tests with a missing trait, but does not depends-on anything else.. I assume that's because we're waiting for the traits release?15:12
sean-k-mooneyits why ironic supprot rebuild witout erasing the epmeeral disk while reimaing the root disk15:12
sean-k-mooneyyes15:12
dansmithhaving this behave differently for one virt driver does not seem like a very good user experience, even though ironic is weird15:13
sean-k-mooneyright ironic already had specific beahivor15:13
dansmithsigh15:13
sean-k-mooneyim not sure it would take much to add ironci supprot but testin gwould be hard15:13
sean-k-mooneythe ironic specific behavior is that it has an api parmater that allows the ephemerl disk to not be erased when you rebild with a new image15:14
sean-k-mooneywe could support that for libvirt too but we dotn since the default impl does not support it15:14
dansmithSIGH15:15
dansmithso yeah, so much of rebuild is done by ironic, perhaps just refusing to do rebuild on bfv if we're instructed to wipe the root disk is the right approach15:16
dansmithbut we'll really need that implemented 15:16
sean-k-mooneywell we can check that in the api fairly simplely15:17
sean-k-mooneywe can check the hypervior type i belive15:17
dansmithwhich would be terrible15:17
sean-k-mooneyand just reject it with a 40015:17
dansmithwe should not have the API behaving differently depending on the virt driver15:17
sean-k-mooneyit also should not behave differntly based on if the instnace is boot form volume15:18
dansmiththat's compute stuff, and while it sucks to fail late like that, I much prefer that than building virt-specific stuff into the api15:18
sean-k-mooneybut that the situration we are in15:18
sean-k-mooneydue to the hacky (only update metadat if the image is the same and its BFV) legacy15:18
dansmiththe api has to behave differently for bfv because it historically did, but we shouldn't be building *new* stuff that drags virt specifics into the api15:19
sean-k-mooneyack i agree with that15:19
sean-k-mooneythe preserve_ephemeral filed looks like it predates microveriosn by the way15:19
dansmithbasically all the functionals on that user data patch fail because of that missing trait,15:19
sean-k-mooneyi was trying to figure out when we added that15:19
dansmithwhich makes it hard to work on the functional tests15:20
sean-k-mooneyi belvie the pased in the previsous version of the patch that did not have the trait15:20
sean-k-mooneylocally we can just pip install the os-traits git repo15:20
sean-k-mooneybut ya that why i was hopign to do the release yesterday15:20
dansmithyeah, I'll have to do that15:20
sean-k-mooneyelodilles: any chance we can merge https://review.opendev.org/c/openstack/releases/+/85461715:21
sean-k-mooneyto adress ^15:21
elodillessean-k-mooney: ack, will look into it15:22
opendevreviewMerged openstack/nova master: Add locked_memory extra spec and image property  https://review.opendev.org/c/openstack/nova/+/77834715:22
sean-k-mooneyelodilles++ thanks very much :)15:22
elodillessean-k-mooney: do you need to release it ASAP? I'm asking because if really needed then I'll +W with single +2 (i'm the only release manager on duty today o:)) otherwise we need to wait till Monday15:24
sean-k-mooneyam ideally yes. it could wait but we have 2 api changes blocked by it currently.15:27
sean-k-mooneyits at your disgression ultimately dansmith or gibi might have a stonger prefernce15:28
gibiI'm fine releasing that today with a single release mgmt +215:28
dansmithyeah, it'd be a lot better if it could be today, unfortunately15:29
* gibi wondering if we can move the next summer Feature Freeze to Sept15:29
gibiPTO + FF is not a good combination15:29
elodillessean-k-mooney gibi dansmith: ack, os-traits release is on the way16:06
dansmithelodilles: thanks!16:06
gibielodilles: thank oyu16:06
gibiyou16:06
elodillesnp16:06
sean-k-mooneyonce that is release we shuld bump our min required version  https://github.com/openstack/nova/blob/master/requirements.txt#L56 right in the change that adds the userdata feature16:07
sean-k-mooneylike i know we  will get it automaticaly but we should raise our dep  in the feature that requires the new trait16:07
opendevreviewDan Smith proposed openstack/nova master: DNM: Test for rosmaita  https://review.opendev.org/c/openstack/nova/+/85481516:09
opendevreviewDan Smith proposed openstack/nova master: DNM: Test for rosmaita  https://review.opendev.org/c/openstack/nova/+/85481516:10
gibithe gate queue is 53 patches long, nice16:10
elodilles:-o16:12
sean-k-mooneyoh the oauth 2 feature is mergin in keystone16:12
sean-k-mooneythats cool16:12
dansmithsean-k-mooney: I'm getting functional fails on missing api samples for 2.9316:15
dansmithlooks like the author dropped some of them from a previous version?16:15
sean-k-mooneyon hte user data patch lets see16:15
dansmithyeah16:16
sean-k-mooney... i jsut realsied i review version 8 not 9 just now16:16
sean-k-mooneyand yes16:16
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova/+/816157/8..916:16
sean-k-mooneythere were more removed then added16:16
sean-k-mooneyim stongly tempeted to say we shoudl swap those two if the author of the user data one has not updated it by monday16:17
sean-k-mooneyim tempeted to say we do it now but its kind of a pain to update all the micorversion refrences16:17
sean-k-mooneyya so it passed on 8 https://review.opendev.org/c/openstack/nova/+/816157/9#message-87c965d558c8c7b25b2c4d1ebd3232c2627f444a and likely because of os-traits not being released they did not realise it failed on v916:20
sean-k-mooneythats a pian16:20
dansmithyeah16:21
opendevreviewMerged openstack/nova master: Retry /reshape at provider generation conflict  https://review.opendev.org/c/openstack/nova/+/85135817:07
dansmithgibi: still around?17:36
gibidansmith: yes, but with limited brain power17:37
dansmithgibi: on the error path.. if we created the new attachment, saved it to the bdm, but failed to delete the old one, is there any reason not to just keep rolling?17:37
dansmithlike what does restoring the old attachment_id and aborting get us there?17:38
sean-k-mooneyso the db save failed17:38
dansmithno17:38
dansmiththe db save of the new one completed17:38
dansmiththe delete of the old one failed (in cinderclient)17:38
sean-k-mooneybut the delete in cinder failed17:38
sean-k-mooneyah ok17:39
sean-k-mooneyam well we could leak it but other then that i think its ok17:39
sean-k-mooneyif we have our side in order17:39
dansmithright, we leak an attachment, but presumably we're already sunk if we fail to delete17:39
gibidansmith: please point me the case in the code 17:39
dansmithgibi: https://review.opendev.org/c/openstack/nova/+/820368/32/nova/compute/manager.py in L343717:40
sean-k-mooneydansmith: at that point have we "bound" the new attachmet propelry and got all the connector info so we can update the xml and proceed17:40
sean-k-mooneypresuamable deleteing the attcoemnt woudl only fail if cidner was unaviable or something like that17:41
dansmithsean-k-mooney: well this is before the reimage, but yeah, I'm saying if we've recorded the new attachment and can't delete the old one, I'm not sure it gets us much to revert our own db record to the old attachment and then try to delete the new one17:41
sean-k-mooneyah17:41
gibidansmith: so the root_bdm.attachment_id points to the new attachement_id and we just deleted that in cinder17:41
dansmithreverting to the old one and deleting the new one is a more complicated cleanup, which I can do, but I just want to make sure it's worth it17:42
gibiby L344117:42
dansmithgibi: that's what I'm saying.. I think if we hit a cinder error in L3437, we should NOT do L3441 and abort17:42
gibidansmith: OK, that make sense17:43
dansmiththe clientexception will catch errors on L3432 *and* 343717:43
gibikeep the bdm to use the new attachment id17:43
dansmith++ okay, it's much cleaner (on our side) to do that, but just wanted to make sure that was reasonable17:43
gibibut we still need to handle the fact that we might detached the volume from the guest already at 343417:44
sean-k-mooney i kind of feel like it migh make more sesen to break up that try17:44
dansmithsean-k-mooney: s'what I'm doing (and said in the review)17:45
sean-k-mooneyack17:45
dansmithgibi: yeah, so I'm going to delete the new attachment if we fail to save, but otherwise, I'm going to log both and the situation and plow on17:45
gibiand we say that the instance can be recovered with a hard reboot from this case? 17:46
gibias at that point when the save fail we removed the volume from the guest17:46
dansmithI don't know that we need even that17:47
sean-k-mooneyso the exception.InstanceNotFound is coming form the detac potieintally or can it come form the create too17:47
sean-k-mooneygibi: if the bdms are correct and the atachment we created on line 343217:47
dansmithsean-k-mooney: only the bdm save, AFAIK17:48
sean-k-mooneyis correct then a hard reboot might work17:48
dansmithwhy do we need a hard reboot?17:48
dansmithwe've fixed the instance to point to the new attachment, we can just move on with the reimage17:48
dansmithwe'17:48
dansmithwe will have leaked an attachment that cinder didn't let us delete but... that doesn't impact the instance anymore right?17:48
gibiahh OK17:48
gibiI see now17:48
gibiI thought we would abort17:48
dansmithI wish I could push this up, but I'll rebase the userdata one if I do :/17:49
sean-k-mooneyya that was what i was workign my way thoguht we abort and go to error17:49
dansmithwe abort all cases, except the last delete of the old attachment17:49
sean-k-mooneybut if we move on17:49
gibiOK, that is fine17:49
dansmithbecause it doesn't matter at that point17:49
sean-k-mooneyas you suggest no reboot needed17:49
sean-k-mooneyactully17:49
sean-k-mooneyif we get instance not found form our own db17:49
sean-k-mooneywell no form libvirt17:50
sean-k-mooneyprefumably that means we raced with a delete?17:50
dansmithon bdm save, yeah17:50
sean-k-mooneyya so nothing to try and recover17:50
sean-k-mooneydansmith: you can do git review -R by the way to avoid a rebase17:51
dansmithsean-k-mooney: no, I've already rebased user data locally so I could get this rebased and ready17:52
sean-k-mooneyah ok well it would not be the end fo the world if it was rebased. it needs to be updated for  the api sample issue anyway17:52
dansmithI just don't want to step on the other author if they already have things in the middle17:53
sean-k-mooneyack, but yes it sounds like you can just proceed and leak the attacment and may just log it so an op could clean it up later17:54
sean-k-mooneyso whats left is gettin gth image form galnce. using that to calualte the size, setting up the wait for the external event and then calling cinder to do the reimage17:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Handle PCI dev reconf with allocations  https://review.opendev.org/c/openstack/nova/+/85239717:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Allow enabling PCI tracking in Placement  https://review.opendev.org/c/openstack/nova/+/85046817:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Generate request_id for Flavor based InstancePCIRequest  https://review.opendev.org/c/openstack/nova/+/85383517:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Create RequestGroups from InstancePCIRequests  https://review.opendev.org/c/openstack/nova/+/85277117:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Support resource_class and traits in PCI alias  https://review.opendev.org/c/openstack/nova/+/85331617:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Split PCI pools per PF  https://review.opendev.org/c/openstack/nova/+/85444017:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Map PCI pools to RP UUIDs  https://review.opendev.org/c/openstack/nova/+/85411817:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Make allocation candidates available for scheduler filters  https://review.opendev.org/c/openstack/nova/+/85411917:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Filter PCI pools based on Placement allocation  https://review.opendev.org/c/openstack/nova/+/85412017:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Store allocated RP in InstancePCIRequest  https://review.opendev.org/c/openstack/nova/+/85412117:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Func test for PCI in placement scheduling  https://review.opendev.org/c/openstack/nova/+/85412217:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Support cold migrate and resize with PCI tracking in placement  https://review.opendev.org/c/openstack/nova/+/85424717:56
opendevreviewBalazs Gibizer proposed openstack/nova master: Support evacuate with PCI in placement  https://review.opendev.org/c/openstack/nova/+/85461517:57
opendevreviewBalazs Gibizer proposed openstack/nova master: Support unshelve with PCI in placement  https://review.opendev.org/c/openstack/nova/+/85461617:57
opendevreviewBalazs Gibizer proposed openstack/nova master: Support same host resize with PCI in placement  https://review.opendev.org/c/openstack/nova/+/85444117:57
opendevreviewBalazs Gibizer proposed openstack/nova master: Test reschedule with PCI in placement  https://review.opendev.org/c/openstack/nova/+/85462617:57
opendevreviewBalazs Gibizer proposed openstack/nova master: Support multi create with PCI in placement  https://review.opendev.org/c/openstack/nova/+/85466317:57
opendevreviewBalazs Gibizer proposed openstack/nova master: Heal allocation for same host resize  https://review.opendev.org/c/openstack/nova/+/85482217:57
dansmithugh, that causes another failure later, which may be a quirk of our cinder fixture17:58
dansmiththe attachment isn't deleted, so much later it complains that the instance is double-attached17:58
sean-k-mooneywell form the cinder side i guess it is17:58
sean-k-mooneywe could try and recover at that point and delete the old one again17:59
sean-k-mooneywhich would possisble stop the leak17:59
dansmiththat's what I'm trying to avoid because it ends up a pretty nested mess17:59
sean-k-mooneyah ok17:59
dansmithand it's likely to fail in reality if we just failed to delete17:59
sean-k-mooneywe are not doing the attchment delete before the save to prevent the other case right18:00
dansmithI'm also not sure I know why we're detaching and re-attaching here, just to do the reimage18:00
sean-k-mooneywhere our db is now out of sync if we fail to save18:00
dansmithmust be some cinder reason why that happens, to reset the state while the instance is powered off or something18:00
dansmithyeah I think we have to update our db before the delete for races18:00
sean-k-mooneyya i think so too.18:01
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (db)  https://review.opendev.org/c/openstack/nova/+/83119318:01
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (objects)  https://review.opendev.org/c/openstack/nova/+/83940118:01
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (manila abstraction)  https://review.opendev.org/c/openstack/nova/+/83119418:01
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (drivers and compute manager part)  https://review.opendev.org/c/openstack/nova/+/83309018:02
opendevreviewribaudr proposed openstack/nova master: Attach Manila shares via virtiofs (api)  https://review.opendev.org/c/openstack/nova/+/83683018:02
opendevreviewribaudr proposed openstack/nova master: Bump compute version and check shares support  https://review.opendev.org/c/openstack/nova/+/85049918:02
opendevreviewribaudr proposed openstack/nova master: Add metadata for shares  https://review.opendev.org/c/openstack/nova/+/85050018:02
opendevreviewribaudr proposed openstack/nova master: Add instance.share_attach notification  https://review.opendev.org/c/openstack/nova/+/85050118:02
opendevreviewribaudr proposed openstack/nova master: Add instance.share_detach notification  https://review.opendev.org/c/openstack/nova/+/85102818:02
opendevreviewribaudr proposed openstack/nova master: Add shares to InstancePayload  https://review.opendev.org/c/openstack/nova/+/85102918:02
opendevreviewribaudr proposed openstack/nova master: Add instance.power_on_error notification  https://review.opendev.org/c/openstack/nova/+/85208418:02
opendevreviewribaudr proposed openstack/nova master: Add instance.power_off_error notification  https://review.opendev.org/c/openstack/nova/+/85227818:02
opendevreviewribaudr proposed openstack/nova master: Add helper methods to attach/detach shares  https://review.opendev.org/c/openstack/nova/+/85208518:02
opendevreviewribaudr proposed openstack/nova master: Add libvirt test to ensure metadata are working.  https://review.opendev.org/c/openstack/nova/+/85208618:02
opendevreviewribaudr proposed openstack/nova master: Add virt/libvirt error test cases  https://review.opendev.org/c/openstack/nova/+/85208718:02
opendevreviewribaudr proposed openstack/nova master: Change microversion to 2.93  https://review.opendev.org/c/openstack/nova/+/85208818:02
opendevreviewribaudr proposed openstack/nova master: Add share_info parameter to reboot method for each driver (driver part)  https://review.opendev.org/c/openstack/nova/+/85482318:02
opendevreviewribaudr proposed openstack/nova master: Support rebooting an instance with shares (compute and API part)  https://review.opendev.org/c/openstack/nova/+/85482418:02
sean-k-mooneyi dont recall why but i rememebr lee disucsing this at some point18:02
sean-k-mooneyi dont see this dicussed in the ptg18:06
sean-k-mooneyah18:07
sean-k-mooneyhttps://review.opendev.org/c/openstack/nova-specs/+/840155/5/specs/zed/approved/volume-backed-server-rebuild.rst#5818:07
sean-k-mooney#. Create an empty (no connector) volume attachment for the volume and18:07
sean-k-mooney   server. This ensures the volume remains ``reserved`` through the next18:07
sean-k-mooney   step.18:07
sean-k-mooney#. Delete the existing volume attachment (the old one).18:07
sean-k-mooney#. Save the new attachment UUID to the BDM.18:07
sean-k-mooney#. The above two steps are needed to keep the volume in ``reserved`` state18:07
sean-k-mooney   as a management state which is required by cinder to perform re-image18:07
sean-k-mooney   operation on it.18:07
sean-k-mooney#. Call the new ``os-reimage`` cinder API.18:07
dansmithyeah, to reset the state basically I assume18:07
sean-k-mooneywe need to move it form in-use to reserved to reimage18:08
sean-k-mooneyhonestly i feel like the way out of this if everything explodes is just call rebuild again18:11
sean-k-mooneylike wait for cinder to be back18:11
sean-k-mooneyand then the vm that is precumsble in error if we coudl not comptle the reimage later18:11
sean-k-mooneycould just be rebuilt again18:11
dansmithyeah I assume that will work, as long as it's still in a reasonable state18:12
sean-k-mooneykind of like what we do with a failed evacuate we just evacuate again18:12
dansmithso I guess I'm just going to put it somewhat back the way it was, let it fail for all cases, but only call delete if we created the new attachment18:13
dansmithright now it'll call delete(None) which I assume will fail18:13
sean-k-mooneyya presumabley with a type/atibute error18:13
sean-k-mooneyNone type has no attirbute uuid or something like that18:14
gibirebuild after a failed rebuild make sense to me too18:15
sean-k-mooneyi dont know if the precondtions on rebuild allow that18:16
dansmithyeah, I'm not so concerned about that, I just want to make sure we haven't left things in too bad of a state where that's not possible18:16
sean-k-mooneyor if you would have to do reset state first18:16
dansmithsounds to me like we're going to have leaked an attachment regardless18:16
sean-k-mooneyyes18:16
dansmithbecause we have nowhere to stash it and try to delete it later18:16
sean-k-mooneyalthough the user can actully delete that if they wanted too18:16
sean-k-mooneyit would be a bit painful however18:17
sean-k-mooneysince they have no way to lsit teh bdms today18:17
sean-k-mooneyits not part of server detail show18:18
sean-k-mooneyi mena they coudl try deleteing all the attachments on a voluem but then nova and cinder are out of sync18:18
sean-k-mooneyso i think we woudl want to make sure they coudl juse rebuild again18:19
sean-k-mooneyrather then trying to repair it themselves18:19
dansmithI think that will fail because of the double attachment18:20
dansmiththe same thing as when I failed the delete18:20
dansmithbut I dunno there's much we can do about that18:20
opendevreviewRajat Dhasmana proposed openstack/nova master: add support for updating server's user_data  https://review.opendev.org/c/openstack/nova/+/81615718:56
opendevreviewRajat Dhasmana proposed openstack/nova master: Add support for volume backed server rebuild  https://review.opendev.org/c/openstack/nova/+/82036818:56
opendevreviewRajat Dhasmana proposed openstack/nova master: Add conductor RPC interface for rebuild  https://review.opendev.org/c/openstack/nova/+/83121918:56
opendevreviewRajat Dhasmana proposed openstack/nova master: Add API support for rebuilding BFV instances  https://review.opendev.org/c/openstack/nova/+/83088318:56
ozzzo_workWe use the AggregateMultiTenancyIsolation filter for some aggregates, and for those aggregates we specify "filter_tenant_id = '(list of projects)'19:00
ozzzo_workToday I'm hitting that filter for aggregates that do not have filter_tenant_id set19:00
ozzzo_workThis aggregate has no properties set at all19:01
ozzzo_work2022-08-26 17:48:49.879 33 INFO nova.filters [req-f57f505f-b3c8-42f5-acf7-5e5efe10ef83 a9219bb013c944edea6ba612b8fa704adc28d676f3ad1e326094350b8ff0c9c0 efc7139e50db483991df846c69b42477 - 8793b235debf49e6aba6bd1e2bf65360 8793b235debf49e6aba6bd1e2bf65360] Filtering removed all hosts for the request with instance ID 'e9f9292a-f1c0-409b-a47f-9fa596fd0965'. Filter results: ['ComputeFilter: (start: 50, end: 50)', 'RetryFilter: (start: 50, end19:01
ozzzo_worklooks like the line is too long19:01
ozzzo_workhttps://paste.openstack.org/show/b5UFNxMB3oIlduDz1nat/19:02
ozzzo_workWhat could be causing that?19:02
dansmithoye19:02
dansmithwhoami-rajat: did you make changes to that set or just rebase?19:02
dansmithwhoami-rajat: I should have made it more clear -- I was doing some work there, but was trying to avoid rebasing the other person's patch19:03
whoami-rajatdansmith, just rebased to resolve merge conflict, now working on changes19:03
dansmithfigured I was safe since you said monday19:03
dansmithwhoami-rajat: hold on, I have a number of changes19:03
whoami-rajatdansmith, oh sorry about that19:03
dansmithI might as well push them up now that you rebased it19:03
whoami-rajatack, will wait then19:03
whoami-rajatthis microversion dependency doesn't seem good ...19:04
dansmithah, whoami-rajat I think you reverted the other person's latest rev with your push just now :(19:08
dansmithyeah19:09
whoami-rajatoh ...19:09
dansmithokay so I'll just push up my stuff with a rebased version of their latest19:09
whoami-rajati think it's just better to break the dependency chain19:10
whoami-rajatwill need to do an extra rebase when their change is merged19:10
whoami-rajati can see how it happened, their change was not latest and my patches were not rebased on their latest19:11
whoami-rajatshould've been more careful there :/19:11
dansmithwhoami-rajat: right, I had it all set locally,19:11
dansmithand breaking the chain just means we'll have to do more version switching if we do that19:12
dansmithjust a sec19:12
opendevreviewDan Smith proposed openstack/nova master: add support for updating server's user_data  https://review.opendev.org/c/openstack/nova/+/81615719:12
opendevreviewDan Smith proposed openstack/nova master: Add support for volume backed server rebuild  https://review.opendev.org/c/openstack/nova/+/82036819:12
opendevreviewDan Smith proposed openstack/nova master: Add conductor RPC interface for rebuild  https://review.opendev.org/c/openstack/nova/+/83121919:12
opendevreviewDan Smith proposed openstack/nova master: Add API support for rebuilding BFV instances  https://review.opendev.org/c/openstack/nova/+/83088319:12
whoami-rajatyeah 2.94 after 2.92 will break something surely19:12
ozzzo_workIs this the right channel to ask about nova filters?19:12
dansmithozzzo_work: this is really for nova development not help, and it's friday afternoon19:13
dansmithwhoami-rajat: so in the main patch I broke apart that error handler for the issue we noted there,19:13
ozzzo_workok I'll try emailing the list19:13
dansmithwhoami-rajat: and in the last one I made the test less mock-heavy and added some negative cases19:13
dansmithozzzo_work: that'd be better19:13
whoami-rajatdansmith, great, i was just thinking about how to break those exception blocks, thanks19:15
dansmithwhoami-rajat: I don't think we need to be fully covered in the functional tests, but I included several error paths, so I hope that's good enough, assuming I didn't break any unit tests.. more unit tests for covering all the potential exit paths is still good of course19:16
whoami-rajatack, will take a look19:17
dansmithall the brought-forward comments are referring to the wrong lines in the latest PS instead of being collected up at the top, that's very weird19:18
dansmithIMHO you can resolve those and let people re-comment if they want19:19
* dansmith hates that "feature" of gerrit.19:19
whoami-rajatyeah that happens, I'm on the same patchset when the comments were left and it looks fine like that19:19
whoami-rajatleft view: old PS with comments, right view: new PS19:20
dansmithright, but if you have base on the left and current on the right, it is supposed to show the comments on unchanged lines, or at the top if they've been changed19:22
dansmithand this is showing them on the wrong lines19:22
dansmithactually hard refresh seems to have fixed it,19:23
dansmithso maybe just a caching bug or something19:23
whoami-rajatyep, I've also been struggling with that recently19:23
whoami-rajatah really?19:23
whoami-rajatyep that fixed it, nice19:24
dansmithhaven't seen that before19:24
sean-k-mooneyozzzo_work: quick glance not sure 19:27
sean-k-mooneydansmith: ya i often have to go to the patset the comment was left on to make any use out of that feature19:29
sean-k-mooneyok im starting to get hungery so im going to finish there for today.19:30
sean-k-mooneyi will keep an eye on those patches19:31
sean-k-mooneyi assume at this point however that it will be monday for the base patch to be ready19:31
whoami-rajatsean-k-mooney, if you're still around, I've a question19:54
whoami-rajatsean-k-mooney, regarding your comment here https://review.opendev.org/c/openstack/nova/+/820368/comments/711f86cd_9c3731a419:54
whoami-rajatsean-k-mooney, if we want to support it for the ironic use case, we would have to provide a generic driver implementation and also implemented it for the libvirt driver right?19:55
dansmithwhoami-rajat: ironic is the only one that provides its own rebuild implementation, AFAIK19:55
whoami-rajatlooking19:55
dansmithwhoami-rajat: I think what you need to do is modify ironic's rebuild to check for the flag and fail if it's requested (for now)19:55
dansmithand obviously pass the flag to it as sean mentions19:55
dansmithwhoami-rajat: getting it to work with ironic will require work on the ironic side, which obviously isn't going to happen19:56
dansmithironic is already "weird" about rebuilds, so I think we just have to document that and hope we can get it resolved later19:56
whoami-rajatdansmith, yeah, that's what my concern was, if we just need to fail in that case then it's good19:56
dansmithit's not good, but..yeah :)19:56
whoami-rajatcool, that clears things19:56
whoami-rajatwas kind of worried there19:57
opendevreviewRajat Dhasmana proposed openstack/nova master: Add support for volume backed server rebuild  https://review.opendev.org/c/openstack/nova/+/82036820:59
opendevreviewRajat Dhasmana proposed openstack/nova master: Add conductor RPC interface for rebuild  https://review.opendev.org/c/openstack/nova/+/83121920:59
opendevreviewRajat Dhasmana proposed openstack/nova master: Add API support for rebuilding BFV instances  https://review.opendev.org/c/openstack/nova/+/83088320:59
*** dasm is now known as dasm|off21:07
opendevreviewRico Lin proposed openstack/nova master: libvirt: Add vIOMMU device to guest  https://review.opendev.org/c/openstack/nova/+/83064621:16
opendevreviewRico Lin proposed openstack/nova master: Add traits for viommu model  https://review.opendev.org/c/openstack/nova/+/84450721:16
ricolinsean-k-mooney: thanks, just update patches, will try to work on traits asap21:17

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!