Friday, 2021-10-01

gmannsure. but I am sure we will continue spending time every time and always find solution :P just joking 00:00
gmannwe means except me :)00:00
melwitthehe00:00
*** gmann is now known as gmann_afk00:07
*** gmann_afk is now known as gmann01:11
opendevreviewmelanie witt proposed openstack/nova master: Add logic to enforce local api and db limits  https://review.opendev.org/c/openstack/nova/+/71213905:49
opendevreviewmelanie witt proposed openstack/nova master: Enforce api and db limits  https://review.opendev.org/c/openstack/nova/+/71214205:49
opendevreviewmelanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits  https://review.opendev.org/c/openstack/nova/+/71214305:49
opendevreviewmelanie witt proposed openstack/nova master: Update limit APIs  https://review.opendev.org/c/openstack/nova/+/71270705:49
opendevreviewmelanie witt proposed openstack/nova master: Update quota sets APIs  https://review.opendev.org/c/openstack/nova/+/71274905:49
opendevreviewmelanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources  https://review.opendev.org/c/openstack/nova/+/71330105:49
opendevreviewmelanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit  https://review.opendev.org/c/openstack/nova/+/61518005:49
opendevreviewmelanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits  https://review.opendev.org/c/openstack/nova/+/71349805:49
opendevreviewmelanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage  https://review.opendev.org/c/openstack/nova/+/71349905:49
opendevreviewmelanie witt proposed openstack/nova master: Add reno for unified limits  https://review.opendev.org/c/openstack/nova/+/71527105:49
opendevreviewmelanie witt proposed openstack/nova master: Enforce api and db limits  https://review.opendev.org/c/openstack/nova/+/71214206:53
opendevreviewmelanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits  https://review.opendev.org/c/openstack/nova/+/71214306:53
opendevreviewmelanie witt proposed openstack/nova master: Update limit APIs  https://review.opendev.org/c/openstack/nova/+/71270706:53
opendevreviewmelanie witt proposed openstack/nova master: Update quota sets APIs  https://review.opendev.org/c/openstack/nova/+/71274906:53
opendevreviewmelanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources  https://review.opendev.org/c/openstack/nova/+/71330106:53
opendevreviewmelanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit  https://review.opendev.org/c/openstack/nova/+/61518006:53
opendevreviewmelanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits  https://review.opendev.org/c/openstack/nova/+/71349806:53
opendevreviewmelanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage  https://review.opendev.org/c/openstack/nova/+/71349906:53
opendevreviewmelanie witt proposed openstack/nova master: Add reno for unified limits  https://review.opendev.org/c/openstack/nova/+/71527106:53
opendevreviewmelanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits  https://review.opendev.org/c/openstack/nova/+/71349806:56
opendevreviewmelanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage  https://review.opendev.org/c/openstack/nova/+/71349906:56
opendevreviewmelanie witt proposed openstack/nova master: Add reno for unified limits  https://review.opendev.org/c/openstack/nova/+/71527106:56
bauzasgood morning Nova07:21
*** frickler_ is now known as frickler07:25
gibibauzas: good Friday 07:30
bauzasindeed07:30
bauzasgibi: looks like the nvme osbrick issue is now fixed 07:48
gibibauzas: os-brick 5.0.1 with the nvme fix and global requirements has been bumped https://review.opendev.org/c/openstack/releases/+/812048/1/deliverables/xena/os-brick.yaml07:48
gibiheh07:49
gibiwe looked at it at the same time :D07:49
bauzasthe xena backport is now merged https://review.opendev.org/c/openstack/os-brick/+/81201007:49
bauzasyeah07:49
bauzasI call that "friends" :p07:49
gibi:)07:49
bauzasok, so should we do something with our requirements ?07:50
bauzaselodilles: thoughts on touching our reqs so close to GA ?07:50
gibiupper constraints are coming from global 07:50
gibian as far as I remember reqs repo hasn't been branched to xena yet07:51
gibiso the merged global req bump is applied to Xena automatically07:52
gibithe only question is, do we want to blacklist os-brick 5.0.007:52
bauzasah good point07:52
gibiand I think we agreed with lyarwood that this is a narrow scoped issue so we don't need to blacklist07:52
bauzasabout the merge07:52
bauzasgibi: yeah I think so07:53
gibithen I think we are done07:53
bauzasso let it be07:53
bauzasyeah07:53
gibiit was a nice bug as we needed to do nothing to fix it :D07:53
gibi<3 cinder 07:53
bauzas++07:53
opendevreviewDmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling  https://review.opendev.org/c/openstack/nova/+/80819908:59
opendevreviewDmitrii Shcherbakov proposed openstack/nova master: [yoga] Support remote-managed SmartNIC DPU ports  https://review.opendev.org/c/openstack/nova/+/81211108:59
elodillesbauzas: sorry, missed your question. but yes, as gibi said, it comes from openstack/requirements and RFE was granted for os-brick ( http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025155.html )09:34
lyarwoodbauzas: would you mind hitting https://review.opendev.org/c/openstack/nova/+/811713 and https://review.opendev.org/c/openstack/nova/+/811716 today?09:41
opendevreviewElod Illes proposed openstack/nova stable/ussuri: [stable-only] Pin virtualenv and setuptools  https://review.opendev.org/c/openstack/nova/+/81046109:47
kashyapdmitriis: Hi, from #virt here :) Thanks for reworking the patch and splitting it out.09:57
kashyapReviewer time is a bit limited these days, so please don't hesitate to re-ping here if you don't get any input in a bit09:58
dmitriiskashyap: ack, will do. Appreciate that it's Xena release time so I'll circle back some time after it's out.10:02
opendevreviewMerged openstack/os-traits master: Change minversion of tox to 3.18.0  https://review.opendev.org/c/openstack/os-traits/+/79197310:10
opendevreviewBalazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next  https://review.opendev.org/c/openstack/nova/+/81174810:24
lyarwoodtgif++ side_effect != side_effects10:52
gibi+1 on Friday10:53
opendevreviewLee Yarwood proposed openstack/nova master: Add regression test for bug #1937084  https://review.opendev.org/c/openstack/nova/+/81212611:32
opendevreviewLee Yarwood proposed openstack/nova master: block_device: Ignore VolumeAttachmentNotFound during detach  https://review.opendev.org/c/openstack/nova/+/81212711:32
opendevreviewBalazs Gibizer proposed openstack/nova master: Enable min pps tempest testing in nova-next  https://review.opendev.org/c/openstack/nova/+/81174812:04
opendevreviewTakashi Natsume proposed openstack/nova master: Update min supported service version for Yoga  https://review.opendev.org/c/openstack/nova/+/80993213:09
gmannmelwitt: nova-grenade-multinode passing on stable/train with the devstack fix but devsdtack fix hitting the novnc failure in stable/stein https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c3f/812092/1/check/tempest-full-py3/c3fde5e/testr_results.html13:53
gmannare those known one?13:53
gmannor gibi lyarwood elodilles ^^ you know about novnc issue ? 14:17
gibidoes not ring a bell but let me grep a bit14:18
elodilleshaven't seen that before14:19
gmannit seems circular deps, nova stable/stein hitting jsonschema issue which devstack fix but there we are hitting novnc14:19
opendevreviewGhanshyam proposed openstack/nova stable/stein: DNM: testing jsonschema version fix  https://review.opendev.org/c/openstack/nova/+/81214214:21
gibigmann: I found on other hits recently about this14:23
gibigmann: I see that novncproxy says it is invalid / expired token https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c3f/812092/1/check/tempest-full-py3/c3fde5e/controller/logs/screen-n-novnc-cell1.txt14:25
gibibut I don't know why14:26
gibithe token should not be expired based on the timestamps in the nova-api and novnc proxy logs14:26
gmannk14:29
gibiI don't see anything else14:31
opendevreviewStephen Finucane proposed openstack/nova master: db: Enable auto-generation of API DB migrations  https://review.opendev.org/c/openstack/nova/+/81214414:34
opendevreviewStephen Finucane proposed openstack/nova master: db: Remove unused build_requests columns  https://review.opendev.org/c/openstack/nova/+/81214514:34
opendevreviewStephen Finucane proposed openstack/nova master: db: Remove legacy placement models  https://review.opendev.org/c/openstack/nova/+/81214614:34
opendevreviewStephen Finucane proposed openstack/nova master: objects: Stop querying the main DB for keypairs  https://review.opendev.org/c/openstack/nova/+/81214714:34
opendevreviewStephen Finucane proposed openstack/nova master: objects: Remove 'bandwidth' fields from notifications  https://review.opendev.org/c/openstack/nova/+/81214814:34
opendevreviewStephen Finucane proposed openstack/nova master: db: Remove models that were moved to the API database  https://review.opendev.org/c/openstack/nova/+/81214914:34
opendevreviewStephen Finucane proposed openstack/nova master: db: Remove models for removed services, features  https://review.opendev.org/c/openstack/nova/+/81215014:34
opendevreviewStephen Finucane proposed openstack/nova master: db: Remove nova-network models  https://review.opendev.org/c/openstack/nova/+/81215114:34
opendevreviewStephen Finucane proposed openstack/nova master: WIP: objects: Prepare for Instance, InstanceList v3  https://review.opendev.org/c/openstack/nova/+/81215214:34
opendevreviewStephen Finucane proposed openstack/nova master: WIP: objects: Add Instance, InstanceList v3  https://review.opendev.org/c/openstack/nova/+/81215314:34
stephenfinFYI, those patches are me stress testing the alembic infrastructure and fixing things I find along the way. Would be useful to merge early in Yoga (i.e. starting next week?) to give us loads of time to resolve any potential fallout :)14:42
gibistephenfin: ack, good point14:48
*** tbachman is now known as Guest150314:54
spatelFolk, not sure what i should look for but we have same hardware on both cloud but local storage showing different result - https://paste.opendev.org/show/809727/ 15:06
spatelI don't have any option configure here on both clouddisk_cachemodes =15:09
artom_bauzas, around? Does https://bugs.launchpad.net/nova/+bug/1944947 and the fix at https://review.opendev.org/c/openstack/nova/+/810849 qualify as a review-priority bug?15:12
bauzasstill there15:12
artom_Seems pretty important to be able to restart hw:cpu_policy=dedicated instances after an upgrade :)15:12
*** artom_ is now known as artom15:13
bauzasartom: any bug seems to me nice to review15:13
bauzasso +1 on R-P labeling15:13
artompriteau, btw, are you good to continue the patch above? ^^ It's basically just missing unit tests15:13
artomOr would you want someone more experienced with Nova to take over?15:13
bauzaswell, we can mentor priteau, I'm sure :)15:14
priteauHi artom. Sorry, been quite busy so I haven't looked into it yet.15:14
bauzasany new contributor is appreciated :)15:14
bauzas(even if pierre isn't exactly new to me :D )15:14
priteauI can try to add unit tests, if I can't figure it out I'll ask for help15:15
artombauzas, yep, I'm happy either way, just interested in keeping things moving15:15
bauzaspriteau: sure, you are 1 hour away from my usual time, so you can ping me for help15:16
bauzasI could redirect15:16
priteaubauzas: Actually I live in France now ;-)15:16
bauzasheh, I see the French mafia growing15:16
priteauI've actually thought about an issue with the whole migration approach (including the existing function)15:16
priteauIt updates the object but not the version number field15:17
priteauIs this likely to be an issue?15:17
priteauWith the existing code, a pinned instance started on Ussuri would have a 1.4 InstanceNUMACell object15:18
artompriteau, that's an excellent question, actually. I suspect there's magic in the ovo code that does it automatically?15:18
* artom calls forth dansmith 15:18
* dansmith reads15:19
bauzaswell, I need more context15:19
dansmithI'm not sure what the question is15:20
bauzaswe only update the object is something remotable is added15:20
bauzasthe object version*15:20
artomdansmith, the context is https://review.opendev.org/c/openstack/nova/+/810849/2/nova/objects/instance_numa.py, and the question is, won't that leave the InstanceNUMACell as version 1.4 in the DB, even if we added the pcpuset and migrated to 1.515:20
bauzasartom: is the pcpuset attribute a ovo field ?15:21
artomIt's a SetOfIntegersField15:21
bauzaslemme find this object15:21
bauzasthe whole module is too large15:22
bauzasok, I see it15:22
bauzasso, yeah this field exists15:22
bauzaswhy should we bump the object version ?15:22
dansmithartom: you're asking if you deserialize, tweak and re-serialize, if you'll serialize the 1.4 version that came from the original thing?15:22
artomdansmith, aye15:23
artomTweak specifically by adding the field that was added in the 1.5 version15:23
bauzasartom: then the remote object won't get it15:23
dansmithno, when you deserialize an object, it becomes a 1.5, which is why you have to be compatible with your object changes15:24
dansmiththe new field will be unset because it came from a 1.4, but when you re-serialize it, you'll use the current version unless you ask for an older one (like conductor does when it backports changes for you)15:24
priteauI've just looked at a production system with pinned-instances launched on Train, the system now being on Victoria. The InstanceNUMACell objects are actually showing "nova_object.version": "1.6"15:24
priteauSo it looks OK15:25
bauzaswait15:25
bauzasyou lost me15:25
artompriteau, cool, so you confirmed in a real system the mechanism that dansmith explained \o/15:25
bauzaswho's on 1.5 (or later) and who's on 1.4 ?15:25
dansmithbauzas: Who is on first15:25
* bauzas gets an headache15:26
bauzasdansmith: that's what I'm asking15:26
bauzasbut it looks it works, so meh15:26
priteau1.4 are pre-Victoria objects. 1.5, actually none because 1.6 was also introduced in Victoria. Any instance launched or restarted on Victoria gets 1.6 (but only for cpu_policy=dedicated, which is the problem)15:27
dansmithartom: I might actually be wrong here, I'15:27
priteauAnyway, there is no problem with the version number, I just wasn't aware of the nova magic15:27
dansmithI'm looking hang on15:27
bauzaspriteau: so, you're asking about what would happen with an old compute and a Victoria conductor or compute, I guess15:28
bauzasdansmith explained it very pretty well15:28
bauzaswhen you serialize, you add the object version15:28
bauzasso any reader getting the serialized object would know the version and creates a new object based on the compatibility methods we provide15:29
bauzaskeeping in mind that obj 1.X doesn't know what to do with 1.y if y > x15:30
dansmithsoooo,15:30
dansmithI'm actually not sure this behaves the way I thought.. I don't actually know that it matters, but I think it might be different15:30
dansmithit looks like we _will_ preserve the inbound version that we got from the serialized form when we re-serialize,15:31
dansmithwhich I think we do for the purposes of not up-leveling objects if we're newer than others in the cluster15:31
dansmithi.e. to avoid a new conductor up-leveling an old version that an api sent to it, knowing an old compute will need the old version15:32
dansmithand also I guess to avoid upleveling something in the database when we load/save it if we're not ready to roll everyone forward yet15:32
dansmithbut, the behavior you notice is not the version, it's whether or not the field sticks, which is what I was focusing on, and I think it will, even though that doesn't match the version15:32
dansmithmeaningm15:32
artomdansmith, wouldn't upleveling not be a problem because we have code to make it compatible to old versions?15:33
dansmithif you store the object with the new field, it will serialize the new field even if the version is wrong, and a newer node will still pull it out because it doesn't know which version the field belongs to15:33
artomSo if the conductor uplevels something, then an old compute loads, it will automatically downlevel for that compute?15:33
dansmithartom: well, I picked conductor because it was easy, but imagine two computes talking15:34
dansmithso I'm not actually sure I can explain priteau's observation of the new versions15:35
dansmiththat's how I recalled it working, but I can't back that up with references to the code purely in the deserialization/serialization stuff15:36
priteauThese instances were live migrated after the Victoria upgrade, could it explain it?15:36
dansmithdo we recreate the numa objects as part of another operation? yeah, like live migration15:36
dansmiththis is the test that proves that we don't automatically uplevel it: https://github.com/openstack/oslo.versionedobjects/blob/e7b6d52aa4b1b40e68a21a122c09b968d5959b0e/oslo_versionedobjects/tests/test_objects.py#L778-L78515:38
dansmithwhich isn't how I remembered it, but re-reading that test jogs my memory about why15:38
dansmithso I think technically that operation should do something like object.VERSION = ObjectClass.VERSION when it goes to save it out15:40
artomSo the upleveling happens at serialization then?15:46
artomOr some other operation like live migration, as you've said15:46
dansmithit doesn't15:47
artomBut not at de-serialization, at any rate15:47
dansmithI'm thinking live migrations create new numa stuff for the new host and that's how those got updated15:47
dansmithin the past, we've done things like "load and save the object to update it" which would run routines to do updates, but they are probably saving the objects back with the old version but including the updates :/15:48
dansmithI can't find any VERSION updating going on in the tree right now15:48
dansmithlike for online data migrations I mean15:48
dansmithtechnically it probably hasn't mattered, because who (besides artom and priteau) would notice that the version was wrong if the data was right? :)15:49
artomYeah, maybe it's an academic debate - if it does the right thing...15:49
artomWhich it appears to do15:50
priteauI just wanted to check it wouldn't break anything if the version number was not updated15:50
dansmithwell, we shouldn't store the version incorrectly just because that's going to be confusing, but yeah the object would still load with the data we wanted so in practice it hasn't mattered15:50
dansmithpriteau: right, but we should fix it to update the version though15:50
priteauI will leave that to you if you don't mind :D15:52
dansmithartom: do you want me to do that or will you?15:52
artomdansmith, do what? What priteau said, "fix it to update the version though"?15:53
artomWouldn't that be within priteau own's patch?15:53
artomWell, maybe not, since the on-load migration is pre-existing...15:53
dansmithartom: I said it, but no, the code that claims to migrate is already in the tree15:53
dansmithright15:53
artomHe's just fixin it15:53
dansmithright15:53
dansmithlooks like there are several _migrate_* cases in instance_numa.py15:54
artomAnd probably elsewhere...15:54
dansmithmaybe, not sure15:54
dansmithif you do it I can review/+2 but ... :)15:54
artomI mean, the hint of a +2 from dansmith on a PS1 is pretty alluring...15:55
dansmithwell, if you get it right in PS1.. :P15:55
artomCrazier things have happened15:56
dansmithlet me write you a diff to start, hang on15:56
dansmithI think this should be unified instead of done in multiple places like it is, so something like this: https://termbin.com/iawkf15:58
dansmithoh, you know what,15:59
dansmiththe other migrate routine is for non-ovo migrations,15:59
dansmithwhich is different, so probably no need to break out that helper function anyway15:59
dansmithso just the one line VERSION thing plus an assert in whatever tests this and you're probably good16:00
artomOh I see (I think), your initial thought was "have any on-load migration call to a helper that updates version to latest"16:02
dansmithyeah, because I thought there were at least two,16:02
dansmithbut that's because I didn't read16:02
* bauzas gave up and TGIF so bye and thanks all for the fish16:02
artomYeah makes sense16:04
artomMy stomach is requiring food, I think lunch first, patch after16:04
dansmithI +2 that ordering16:04
artom\o/16:05
*** tbachman is now known as Guest151016:16
melwittgmann: no I was not aware of the novnc fail, I will look at it16:48
gmannmelwitt: thanks. 16:52
-opendevstatus- NOTICE: The review.opendev.org Gerrit server has become unreachable as of approximately 19:10 UTC due to a networking issue in the provider, but should be reachable again shortly19:45
melwittgmann: just to update you, I know what is wrong but I don't find how it's happening yet. tl;dr is the "path" in the vnc proxy is wrong and doesn't contain the token (so it's not even validating a token). according to all of the config files I looked at, it should be working. so there is something I'm missing so far20:00
melwittthis is an example of how the path is supposed to look: https://zuul.opendev.org/t/openstack/build/2a183ce6e15f4ab08d31190d4d286a7b/log/controller/logs/screen-n-novnc-cell1.txt#1120:01
melwittand this is the bad one that's failing: https://zuul.opendev.org/t/openstack/build/9234f33acf1247cdb995416ca9389240/log/controller/logs/screen-n-novnc-cell1.txt#1320:01
melwittthis is where it would pick up the "path" variable https://github.com/novnc/noVNC/blob/v0.4/vnc_auto.html#L10620:04
opendevreviewmelanie witt proposed openstack/nova master: Add logic to enforce local api and db limits  https://review.opendev.org/c/openstack/nova/+/71213922:45
opendevreviewmelanie witt proposed openstack/nova master: Enforce api and db limits  https://review.opendev.org/c/openstack/nova/+/71214222:45
opendevreviewmelanie witt proposed openstack/nova master: Update quota_class APIs for db and api limits  https://review.opendev.org/c/openstack/nova/+/71214322:45
opendevreviewmelanie witt proposed openstack/nova master: Update limit APIs  https://review.opendev.org/c/openstack/nova/+/71270722:45
opendevreviewmelanie witt proposed openstack/nova master: Update quota sets APIs  https://review.opendev.org/c/openstack/nova/+/71274922:45
opendevreviewmelanie witt proposed openstack/nova master: Tell oslo.limit how to count nova resources  https://review.opendev.org/c/openstack/nova/+/71330122:45
opendevreviewmelanie witt proposed openstack/nova master: Enforce resource limits using oslo.limit  https://review.opendev.org/c/openstack/nova/+/61518022:45
opendevreviewmelanie witt proposed openstack/nova master: Add legacy limits and usage to placement unified limits  https://review.opendev.org/c/openstack/nova/+/71349822:45
opendevreviewmelanie witt proposed openstack/nova master: Update quota apis with keystone limits and usage  https://review.opendev.org/c/openstack/nova/+/71349922:45
opendevreviewmelanie witt proposed openstack/nova master: Add reno for unified limits  https://review.opendev.org/c/openstack/nova/+/71527122:45
opendevreviewmelanie witt proposed openstack/nova master: DNM Run against unmerged oslo.limit changes  https://review.opendev.org/c/openstack/nova/+/81223622:45

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!