Wednesday, 2020-10-28

« Tuesday, 2020-10-27 Index Thursday, 2020-10-29 »
*** k_mouza has quit IRC00:09
*** zzzeek has quit IRC00:16
*** zzzeek has joined #openstack-nova00:17
*** hamalq has quit IRC00:26
*** k_mouza has joined #openstack-nova00:32
*** k_mouza has quit IRC00:37
*** LinPeiWen21 has joined #openstack-nova00:37
*** macz_ has joined #openstack-nova00:42
*** brinzhang has joined #openstack-nova00:46
*** macz_ has quit IRC00:47
brinzhangbauzas:hi, yesterday was my wife's brithday, I am sorry replied later00:48
*** LinPeiWen21 has quit IRC00:53
*** recyclehero has quit IRC01:40
*** k_mouza has joined #openstack-nova02:04
*** k_mouza has quit IRC02:08
*** swp20 has joined #openstack-nova02:29
*** LinPeiWen22 has joined #openstack-nova02:47
*** gyee has quit IRC02:57
*** tkajinam has quit IRC03:10
*** macz_ has joined #openstack-nova03:18
*** macz_ has quit IRC03:22
*** arne_wiebalck has quit IRC03:22
*** luyao has quit IRC03:22
*** luyao has joined #openstack-nova03:23
*** arne_wiebalck has joined #openstack-nova03:23
*** yonglihe has quit IRC03:23
*** yonglihe has joined #openstack-nova03:25
*** rchurch has quit IRC03:33
*** rchurch has joined #openstack-nova03:34
*** psachin has joined #openstack-nova03:38
*** brinzhang_ has joined #openstack-nova03:45
*** recyclehero has joined #openstack-nova03:46
*** rcernin has quit IRC03:48
*** brinzhang has quit IRC03:48
*** rcernin has joined #openstack-nova03:53
*** bbowen has quit IRC04:11
*** sean-k-mooney has quit IRC04:16
*** sean-k-mooney has joined #openstack-nova04:16
*** JamesBenson has quit IRC04:25
*** JamesBenson has joined #openstack-nova04:26
*** links has joined #openstack-nova04:29
*** JamesBenson has quit IRC04:31
*** brinzhang0 has joined #openstack-nova04:55
*** brinzhang_ has quit IRC04:58
*** macz_ has joined #openstack-nova05:11
*** k_mouza has joined #openstack-nova05:14
*** macz_ has quit IRC05:16
*** k_mouza has quit IRC05:18
*** jamesdenton has quit IRC05:23
*** jamesdenton has joined #openstack-nova05:24
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-nova05:33
*** ratailor has joined #openstack-nova05:46
*** openstackgerrit has quit IRC05:46
*** ratailor has quit IRC05:55
*** ratailor has joined #openstack-nova05:58
*** sapd1_x has joined #openstack-nova06:03
*** openstackgerrit has joined #openstack-nova06:20
openstackgerritsean mooney proposed openstack/nova master: enable host shuffeling by default  https://review.opendev.org/76005506:20
*** ratailor_ has joined #openstack-nova06:25
*** ratailor has quit IRC06:28
*** dklyle has quit IRC06:32
openstackgerritJorhson Deng proposed openstack/nova master: Reject to attach and detach volume while task_state is not None  https://review.opendev.org/75891806:39
*** links has quit IRC06:45
*** tkajinam has joined #openstack-nova06:53
*** sapd1_x has quit IRC07:03
*** sapd1_x has joined #openstack-nova07:05
*** Yumeng has joined #openstack-nova07:09
*** mkrai has joined #openstack-nova07:23
*** macz_ has joined #openstack-nova07:25
*** k_mouza has joined #openstack-nova07:25
*** k_mouza_ has joined #openstack-nova07:27
openstackgerritBrin Zhang proposed openstack/nova-specs master: Re-proposes 'Proposal for a safer remote console with password authentication  https://review.opendev.org/75982807:27
*** macz_ has quit IRC07:29
*** rcernin has quit IRC07:30
*** k_mouza has quit IRC07:30
*** k_mouza_ has quit IRC07:31
*** sapd1_x has quit IRC07:33
*** mkrai has quit IRC07:33
*** mkrai has joined #openstack-nova07:33
*** martinkennelly has quit IRC07:40
*** martinkennelly has joined #openstack-nova07:41
*** ralonsoh has joined #openstack-nova07:45
*** mkrai has quit IRC07:47
*** mkrai has joined #openstack-nova07:47
*** rpittau|afk is now known as rpittau07:51
*** rcernin has joined #openstack-nova07:54
*** slaweq has joined #openstack-nova08:00
openstackgerritBrin Zhang proposed openstack/nova-specs master: Re-proposes 'Proposal for a safer remote console with password authentication  https://review.opendev.org/75982808:00
*** mkrai has quit IRC08:02
*** sapd1_x has joined #openstack-nova08:07
*** tesseract has joined #openstack-nova08:07
*** andrewbonney has joined #openstack-nova08:10
*** lpetrut has joined #openstack-nova08:11
bauzasbrinzhang0: absolutely no worries, priorities go to the family08:13
bauzasgood morning Nova08:14
*** rcernin has quit IRC08:15
brinzhang0bauzas: good morning, thanks08:16
*** mkrai has joined #openstack-nova08:17
*** mkrai has quit IRC08:23
*** mkrai has joined #openstack-nova08:27
lyarwoodwell the gate is fun at the moment08:31
lyarwoodmelwitt++ thanks for the ER query and other bug you raised, looks like we have hit *more* issues overnight08:31
*** sapd1_x has quit IRC08:32
bauzaslyarwood:  /o\08:33
bauzaslyarwood: mmmm, the gate looks all good for me08:35
lyarwoodhttps://review.opendev.org/#/c/759831/ check the failures here, a random collection of unrelated failures08:36
lyarwoodthe gate isn't jammed, jobs are just failing08:37
* lyarwood grabs a coffee before going over the failures08:38
*** ociuhandu has joined #openstack-nova08:38
*** sapd1_x has joined #openstack-nova08:38
*** ociuhandu has quit IRC08:48
bauzaslyarwood: I see08:50
bauzaswe're just lucky then08:50
*** Luzi has joined #openstack-nova08:55
kashyaplyarwood: Mornin; will check08:57
*** tosky has joined #openstack-nova08:57
* kashyap notes that today also starts KVM Forum (where the libvirt / QEMU / KVM and related infra project devs get together). So I'm torn between different things this week08:58
lyarwoodkashyap: ack, is that streaming somewhere?09:01
* lyarwood remembers something about a paywall this year09:01
kashyaplyarwood: It is per-email-based login; costs $2509:01
kashyaplyarwood: Yeah, your recollection is right09:01
kashyapBTW, don't worry; the full talks will be available on YouTube soon09:02
lyarwoodkk no issues, I'll watch it once it's posted there09:02
kashyap(Even the panel discussion was preprecorded)09:02
kashyaplyarwood: So the test that's failing is live migration _without_ shared storage?09:04
* kashyap feels this is coming from the 'blockdev'-related improvements ... goes to look at the libvirtd filter logs09:05
fricklergibi: lyarwood: since https://review.opendev.org/759831 failed once more on unrelated issues, how about force-merging it in order to unblock things? I think we can be pretty sure that the patch itself is working fine by now09:06
lyarwoodkashyap: https://opendev.org/openstack/tempest/src/branch/master/tempest/api/compute/admin/test_live_migration.py#L15109:06
lyarwoodfrickler: I'd be happy with that if it unblocks everyone, gibi should have the final say as PTL however.09:07
lyarwoodI'm going to spend some time this morning going over the other failures and making sure we have bugs / ER queries etc for each09:07
* kashyap clicks09:08
lyarwoodkashyap: hmm why is this even listed as live block migration?09:09
lyarwoodkashyap: it's just a paused instance09:09
kashyaplyarwood: So it is migrating a paused instance but _with_ storage, isn't it?09:09
lyarwoodkashyap: yeah, it's just making assumptions about the env that storage isn't shared09:10
lyarwoodI guess that's what block_migration_for_live_migration is checking09:10
kashyapAah, so it might well be a shared storage setup, but the test is dubious09:10
kashyapI wonder if that's why the NBD copy was cancelled - it must've detected "hey, it's shared storage, no need for copy, hence I'm cancelling it"09:11
lyarwoodkashyap: no I don't think the storage is in this job09:11
kashyapSo, it is non-shared storage?09:11
lyarwoodyeah sorry I'm just confusing the situation09:11
kashyapNo prob09:12
lyarwoodoh you know what this is09:12
*** brinzhang0 has quit IRC09:12
lyarwoodit's the back and forth LM of a paused instance issue we saw downstream09:12
kashyapOhh, wait - that rings a faint bell09:13
lyarwood2020-10-27 16:59:16.022725 | primary | 2020-10-27 16:59:16.022 | 2020-10-27 16:59:06,664 22654 INFO     [tempest.api.compute.admin.test_live_migration] Live migrate back to source ubuntu-bionic-rax-iad-002108267409:13
kashyapIIRC, we ruled it out as a bogus test?09:13
lyarwoodyeah the QEMU devs said it was fubar09:14
kashyapDo you recall that other bug no. by any chance, from your browser history?09:14
kashyapHang on, I found it09:14
kashyapBecause I recall "fixing" the bug description myself :D09:15
lyarwoodhaha09:15
kashyaplyarwood: https://bugzilla.redhat.com/show_bug.cgi?id=171300909:21
openstackbugzilla.redhat.com bug 1713009 in qemu-kvm "[QEMU] Back-n-forth live migration of a paused VM results in QEMU crash with: "bdrv_inactivate_recurse: Assertion `!(bs->open_flags & BDRV_O_INACTIVE)' failed."" [Unspecified,New] - Assigned to kwolf09:21
kashyap"Back-n-forth live migration of a paused VM results in QEMU crash ..."09:21
lyarwoodyou know what, I didn't even check the QEMU logs for this failure upstream09:21
*** songwenping_ has joined #openstack-nova09:21
*** ociuhandu has joined #openstack-nova09:21
kashyapIf we see the QEMU crash, then it's this.  Let's see09:21
lyarwoodI don't see this09:21
kashyap1 sec, on the phone09:21
*** swp20 has quit IRC09:21
*** openstack has quit IRC09:21
*** openstack has joined #openstack-nova09:23
*** ChanServ sets mode: +o openstack09:23
kashyaplyarwood: So the libvirtd log in the bug is from destination host, yeah?09:24
kashyap(I think so, based on the "migrate-incoming" bit in the log)09:24
lyarwoodkashyap: yeah09:25
*** logan- has quit IRC09:27
*** logan- has joined #openstack-nova09:27
kashyaplyarwood: So, to recap: should we care about this test that is "invalid" - what is the point of doing back-n-forth LM of a paused instance?09:28
kashyapAlso, have you got the QEMU command-line of the instance log in question?  /me goes searching09:28
* kashyap notes that his earlier speculation was wrong: this setup is not using the new -blockdev based infrastructure09:30
kashyapHm, which of the instances is it here - https://api.us-east.open-edge.io:8080/swift/v1/AUTH_e02c11e4e2c24efc98022353c88ab506/zuul_opendev_logs_30d/759831/4/gate/nova-grenade-multinode/30d8eb1/logs/libvirt/qemu/index.html09:31
lyarwoodinstance-0000001e09:31
* lyarwood wishes we just used the instance UUID for all of this stuff09:31
kashyapOkay, I looked at the "right one", then09:31
kashyap(Also, the "-incoming defer" confirms it's the dest host)09:32
kashyaplyarwood: It's also in the instance log, as you surely must know - 4d564e22-8ba4-48fb-ac93-27bea660fd7709:32
*** mkrai has quit IRC09:33
lyarwoodkashyap: right my point was that the log should use the UUID as the filename, IOW we should use it as the domain name09:34
lyarwoodkashyap: but I think there's an upgrade issue with that so we've never tried09:34
kashyapAh, guessed as much; consistency across the board09:34
*** mkrai has joined #openstack-nova09:35
kashyaplyarwood: Source host logs are under subnode-2/ dir, right?09:35
kashyapYep09:35
*** brinzhang has joined #openstack-nova09:36
*** rcernin has joined #openstack-nova09:36
*** k_mouza has joined #openstack-nova09:36
kashyaplyarwood: So, I collated the useful bits from the logs here, to check w/ the QEMU/libvirt folks - https://kashyapc.fedorapeople.org/LP-1901739/09:44
*** songwenping__ has joined #openstack-nova09:45
lyarwoodack thanks09:47
kashyaplyarwood: So, is this an apt descrption: back-n-forth live migration (w/o shared storage) of a paused instance causes this missing block job data?09:48
*** macz_ has joined #openstack-nova09:48
*** songwenping_ has quit IRC09:48
*** martinkennelly has quit IRC09:49
*** martinkennelly has joined #openstack-nova09:50
lyarwoodfrickler: any idea how your last recheck of https://review.opendev.org/#/c/759831/ has ended up with it in both the check and gate queues at the same time?09:50
* lyarwood thought it had to pass the check queue first09:51
*** macz_ has quit IRC09:53
fricklerlyarwood: I manually enqueued it into the gate directly09:53
lyarwoodfrickler: ah my bad09:53
fricklerlyarwood: I did the recheck because I didn't see the earlier ones, only the -2 from gate09:54
*** sapd1_x has joined #openstack-nova09:54
*** rcernin has quit IRC09:56
kashyaplyarwood: BTW, ignore my earlier wondering about the use-case for back-n-forth LM of a paused guest.  I just thought of one plausible case:10:08
kashyapYou migrate a paused a instance; it succeeds.  But then you suddenly realize: "oh, I actually need to do some maint work on this dest, so I need to migrate this VM _back_ to the source again"10:08
* kashyap only now saw your comment in the bug10:17
*** psachin has quit IRC10:18
lyarwoodfrickler: ack no issues, I was just confused how it ended up in both queues at the same time10:18
*** psachin has joined #openstack-nova10:27
gibifrickler: I have no objection force merging https://review.opendev.org/75983110:29
*** mkrai has quit IRC10:38
*** mkrai has joined #openstack-nova10:40
openstackgerritMerged openstack/nova master: libvirt: Only ask tpool.Proxy to autowrap vir* classes  https://review.opendev.org/75983110:40
fricklergibi: ^^ done10:41
*** psachin has quit IRC10:45
lyarwoodmany thanks10:46
*** sapd1_x has quit IRC10:47
*** psachin has joined #openstack-nova10:48
*** psachin has quit IRC10:49
*** slaweq has quit IRC10:51
*** psachin has joined #openstack-nova10:51
gibifrickler: thank you10:51
lyarwoodstephenfin: https://review.opendev.org/#/c/758971/ updated btw10:52
lyarwoodgibi: https://review.opendev.org/#/q/topic:bug/1899649 would you mind taking a look at this if you have time ahead of the ptg10:53
*** slaweq has joined #openstack-nova10:55
stephenfinlyarwood: +2; thanks for that10:56
gibilyarwood: sure, looking11:01
* gibi is already caffeinating for the ptg11:01
*** JamesBenson has joined #openstack-nova11:01
openstackgerritStephen Finucane proposed openstack/nova master: tox: Stop linting as part of docs target  https://review.opendev.org/73517611:05
*** mkrai has quit IRC11:07
stephenfinlyarwood: Think you could take a gawk at https://review.opendev.org/#/q/topic:bug/1898272+(status:open+OR+status:merged) too?11:10
lyarwoodstephenfin: sure11:10
*** rcernin has joined #openstack-nova11:16
gibilyarwood: I left a question in https://review.opendev.org/#/c/757894/2/nova/compute/manager.py@a223311:25
*** lpetrut has quit IRC11:25
*** lpetrut has joined #openstack-nova11:26
*** lpetrut has quit IRC11:26
*** lpetrut has joined #openstack-nova11:27
kashyapgibi: [Maybe I'll get kicked off this channel for sayin it] Speaking of caffeine, I switched to decaf-only after 10:00 AM (for my afternoon coffee.11:27
*** sapd1_x has joined #openstack-nova11:28
kashyapgibi: Because of a certain sleep scientist; check this 2-min msg from him: https://www.youtube.com/watch?v=KGfdR7TSJo411:28
gibikashyap: we are inclusive, no kicks due to decaf consumption :)11:28
kashyapHaha11:28
kashyapgibi: "If you have a cup of coffee at noon, then a quarter of that caffeine is still in your brain at _midnight_."11:28
kashyapBut --^ that was startling to me11:28
* gibi watching the video11:29
kashyap(That said, PTG demands more caffeine)11:29
gibinow I'm wondering about 5 o'clock tea too11:31
gibifor me tea has a lower but more elongated effect11:32
kashyapYeah, I've been decaf in the afternoon for 4 days in a row.  And my sleep is steadily improving already11:32
kashyapgibi: BTW, his book (https://www.penguin.co.uk/books/295/295665/why-we-sleep/9780141983769.html) that is referenced earlier is excellent.  (I've also suggested it to Eric Fried in the past, he said he thoroughly enojyed it, FWIW)11:33
lyarwoodgibi: ack thanks looking11:33
gibikashyap: thanks11:36
*** sapd1_x has quit IRC11:39
*** zzzeek has quit IRC11:49
*** zzzeek has joined #openstack-nova11:52
openstackgerritStephen Finucane proposed openstack/nova master: db: Clean up base migration  https://review.opendev.org/75815511:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Merge in unique constraints  https://review.opendev.org/75815611:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Drop "remove" of index  https://review.opendev.org/75815711:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Icehouse database migrations  https://review.opendev.org/75839411:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Juno database migration  https://review.opendev.org/75839511:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Kilo database migrations  https://review.opendev.org/75839611:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Liberty database migrations  https://review.opendev.org/75839711:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Mitaka database migrations  https://review.opendev.org/75839811:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Newton database migrations  https://review.opendev.org/75908511:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Ocata database migrations  https://review.opendev.org/75908611:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Pike database migrations  https://review.opendev.org/75908711:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Queens database migrations  https://review.opendev.org/75908811:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Rocky database migrations  https://review.opendev.org/75908911:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Stein database migrations  https://review.opendev.org/75909011:57
openstackgerritStephen Finucane proposed openstack/nova master: db: Compact Train database migrations  https://review.opendev.org/75909111:57
stephenfingibi: probably want to get this in sooner rather than later https://review.opendev.org/#/c/754396/11:59
gibistephenfin: good point12:03
kashyapstephenfin: Hey, for later, you might want to bookmark this talk from SUSE, ongoing at KVM Forum, on vCPU topology, pinning, etc - https://kvmforum2020.sched.com/event/eE1y/virtual-topology-for-virtual-machines-friend-or-foe-dario-faggioli-suse12:06
kashyap(Slides are in link above; so you can already skim the content)12:06
gibistephenfin: can we skip adding db migration placeholders in the future if no new migration is added during W? now we have a big chunk of placeholders after the last db migration12:07
*** macz_ has joined #openstack-nova12:27
openstackgerritBalazs Gibizer proposed openstack/nova master: WIP: add ovo-mypy-plugin to type hinting o.vos  https://review.opendev.org/75885112:31
*** macz_ has quit IRC12:31
*** raildo has joined #openstack-nova12:41
*** nweinber has joined #openstack-nova12:41
*** rcernin has quit IRC12:43
*** spatel has joined #openstack-nova12:54
gibinova PTG starts in 10 minutes with nova-cyborg corss project session12:56
gibinova PTG starts in 4 minutes with nova-cyborg corss project session12:56
*** jangutter_ has joined #openstack-nova12:56
lyarwoodI'm going to be attending the cinder session this afternoon FYI, I don't think I'm of any use in the cross projects today but ping me if something comes up that I can help with12:57
gibilyarwood: ck12:57
gibilyarwood: ack12:57
*** jangutter has quit IRC12:58
*** jangutter has joined #openstack-nova12:58
*** sapd1_x has joined #openstack-nova13:02
*** jangutter_ has quit IRC13:02
openstackgerritGhanshyam Mann proposed openstack/nova master: DNM: testing copy image with non admin policy  https://review.opendev.org/76012813:12
*** bbowen has joined #openstack-nova13:12
*** ratailor_ has quit IRC13:23
*** psachin has quit IRC13:40
bauzaswow, this was violent13:46
kashyapWhat was violent?13:46
bauzasgibi: my laptop crashed but I'm rejoining13:46
*** brinzhang_ has joined #openstack-nova13:52
*** Liang__ has joined #openstack-nova13:55
*** Liang__ has quit IRC14:03
*** mlavalle has joined #openstack-nova14:07
*** brinzhang_ has quit IRC14:17
*** dklyle has joined #openstack-nova14:24
*** JamesBenson has quit IRC14:25
*** songwenping__ has quit IRC14:39
*** songwenping__ has joined #openstack-nova14:39
*** macz_ has joined #openstack-nova14:43
*** psachin has joined #openstack-nova14:54
lyarwoodzoom--14:55
lyarwoodhaha finally get it to work and it's time for a break, typical.14:56
*** dklyle has quit IRC14:58
*** dklyle has joined #openstack-nova15:08
melwittlyarwood: I saw, the gate is super angry :(15:16
lyarwoodmelwitt: yup indeed thanks for checking things overnight btw15:19
*** lpetrut has quit IRC15:19
lyarwoodmelwitt: it's merged now finally15:19
melwittlyarwood: yeah, I saw it got force merged if I understood correctly. and yeah, want to get a handle on the gate bugs and figure out some root causes. things have been crazy lately15:21
*** gyee has joined #openstack-nova15:24
*** songwenping_ has joined #openstack-nova15:40
*** raildo has quit IRC15:42
*** raildo has joined #openstack-nova15:42
*** songwenping__ has quit IRC15:43
*** takamatsu has quit IRC15:45
*** amodi has quit IRC15:52
*** takamatsu has joined #openstack-nova15:56
lyarwoodbauzas: mute :)15:56
bauzasshit15:57
lyarwoodnp15:57
bauzasif anyone knows whether zoom supports space tabs for muting/unmuting automatically, I'm your man15:57
*** amodi has joined #openstack-nova16:02
bauzasF32--16:11
bauzaskilled my zoom and a couple of apps16:11
gibi#nova now Deprecate os-hypervisors API16:26
openstackgerritMerged openstack/nova master: Add placeholder migrations for Victoria backports  https://review.opendev.org/75439616:28
*** k_mouza has quit IRC16:41
bauzasgibi: wrong channel ^ ;)16:44
gibi:D16:44
melwittgibi, elod, lyarwood: been meaning to ask for awhile ... would appreciate your opinion about these two open stable-only changes whether you think it's appropriate given the oldness of queens https://review.opendev.org/#/q/topic:bug/173166816:44
melwitttl;dr is there was a bug in some transitionary code that is triggered by lots of parallel requests to placement (example: running Rally). impact (if I read the code correctly) is instances go into ERROR state for failure to create placement allocations16:45
gibimelwitt: I have to get back to these tomorrow morning16:45
melwittnp thanks16:45
*** Luzi has quit IRC16:47
elodmelwitt: sure, I'll try to understand the whole case and figure out whether it fits to be backported or not. but I'm a bit lagging behind myself, so I won't promise anything right now o:)16:50
elodand thanks for the tl;dr :]16:50
*** hamalq has joined #openstack-nova16:51
melwittelod: np. yeah it's just a bug, if you try to create a bunch of instances in parallel in queens, you can hit this bug and cause lots of instances to fall into ERROR state as they fail during the scheduling step16:52
lyarwoodmelwitt: I'm slightly confused, why is this stable-only?16:52
melwittlyarwood: bug is in transitionary code that no longer exists16:52
lyarwoodmelwitt: ah sorry didn't gork that correctly in your tldr16:52
lyarwoodmelwitt: and just to be clear, it no longer exists in Placement right?16:53
lyarwoodmelwitt: not just Nova16:53
melwittlyarwood: np, I'm sure I didn't explain it well. also, note the func test needs real MySQL to repro the bug because it has to do with database transaction isolation which sqlite has none of16:53
melwittlyarwood: correct, no longer in placement16:53
lyarwoodmelwitt: ack thanks16:53
*** ociuhandu has quit IRC17:03
*** rpittau is now known as rpittau|afk17:09
*** sapd1_x has quit IRC17:09
*** dtantsur is now known as dtantsur|afk17:10
lyarwoodmelwitt: so the change LGTM as a stable only change but I'd like other master cores with placement/db exp to take a look before we merge this17:12
lyarwoodmelwitt: could you also add the context about why this is stable only in the commit message itself?17:12
lyarwoodactually both of them as they are both marked as stable only17:12
*** mlavalle has quit IRC17:14
*** mlavalle has joined #openstack-nova17:17
*** tosky has quit IRC17:20
openstackgerritMerged openstack/nova stable/victoria: libvirt: 'video.vram' property must be an integer  https://review.opendev.org/75761617:20
melwittlyarwood: I tried to do that as "As part of data migration code" but I acknowledge that's not good enough17:42
*** andrewbonney has quit IRC17:57
*** ralonsoh has quit IRC18:00
mnaserhmm, it looks like live migration when using routed networks can result in the scheduler picking the wrong hosts (systems that are on a different segment?)18:01
mnaseram I missing something or this seems to be something that's not yet implemetned18:01
lyarwoodmnaser: https://review.opendev.org/#/q/topic:bp/routed-networks-scheduling - I think that's still a WIP18:03
rm_workre: the way nova migrations / DB stuff is handled -- if the DB is upgraded from say, rocky->stein, and then we try to run stein but have some issue... could the rocky service run on the stein DB schema? or would it explode violently? or maybe more subtly break?18:13
sean-k-mooneymnaser: ya as lyarwood said that is not implemented nova currenly has not awareness of segments18:16
sean-k-mooneymnaser: bauzas will be working on it for wallaby18:17
*** psachin has quit IRC18:22
melwittrm_work: yes it can, this is what happens during a rolling upgrade from N to N+1, see details https://docs.openstack.org/nova/latest/user/upgrade.html18:22
rm_workohh18:23
rm_work"These DB schema changes are done in a way that both the N and N+1 release can perform operations against the same schema."18:23
rm_workA++++18:23
rm_workthanks!18:23
*** tesseract has quit IRC18:24
melwittyw18:24
rm_workfollowup -- placement is microversioned, right? so ... we could feasibly run nova at Stein and placement at ... Master? :P18:28
rm_work(once we've done the initial migration to standalone placement service/DB, in stein)18:28
*** k_mouza has joined #openstack-nova18:30
rm_workthere is only a dependency on the NOVA side that placement must be at least at the same version as nova18:30
rm_workthere's no such reverse dependency on the placement side?18:30
sean-k-mooneyrm_work: correct18:31
melwittyeah. as mentioned in the doc, placement must be same or ahead of nova bc of the dependency but other than that, it is designed to work independently by way of the API microversions18:31
sean-k-mooneynova has a min placment version it works with18:31
rm_workour cloud is about to look like: Nova(Stein), Placement(Master), Neutron(Train), Octavia(Master)18:31
sean-k-mooneybut placement does not have any depency on nova18:31
sean-k-mooneyrm_work: that should be fine at least Nova(Stein), Placement(Master), Neutron(Train)18:31
rm_workany specific neutron version dependencies?18:31
rm_workIf we jump Nova to Victory with Neutron still at Train, could we see problems?18:32
sean-k-mooneyit depends on the feature you are using but we disable feature if neutron is not new enough18:32
rm_workok, so it handles it18:32
sean-k-mooneyrm_work: that is not tested. in thory it might be ok generally you want to upgrade neutorn before nova18:32
rm_workassuming that'd only be for newer features anyway, and if we're JUST upgrading nova, we wouldn't already be relying on them18:33
rm_workok, noted18:33
sean-k-mooneybut it shoudl work for those versions18:33
rm_workit's probably fine to plan to "be safe" and just upgrade neutron first, not really a huge issue18:33
rm_workhow common do you think it is for folks to run clouds with services at completely different versions?18:33
sean-k-mooneyhttps://github.com/openstack/grenade is what we use for upgrade testing in the ci18:33
*** nweinber has quit IRC18:33
sean-k-mooneyso usign the same or a similar orderign is likely to work better18:34
*** nweinber has joined #openstack-nova18:34
rm_workI haven't seen it discussed a ton, but I've been giving a lot of thought to how Openstack at this point really shouldn't have a ton of requirements as far as versions go, in many cases, so running a cloud with patchwork versioning seems actually quite reasonable18:34
rm_workso we've moved in that direction18:35
*** k_mouza has quit IRC18:35
rm_workstarting to wonder if that might be worth a presentation next summit ;)18:35
sean-k-mooneyit might18:35
sean-k-mooneythis summerises how upgrades should work in openstack https://github.com/openstack/grenade#theory-of-upgrade18:36
sean-k-mooneythats specific to singel version upgrades18:36
rm_workcool thanks, I THINK i've gone through this but it's been a while so will give it a re-read18:36
rm_workyeah, seems no matter what we should probably avoid fast-forward / skip-level18:37
sean-k-mooneyrm_work: this is the order of upgrades done by grenade https://github.com/openstack/grenade/tree/master/projects18:37
sean-k-mooneyi acutlly expected cinder to go before nova but i guess it does not18:38
rm_workI have been studying https://docs.openstack.org/placement/latest/admin/upgrade-to-stein.html EXTENSIVELY, heh18:38
rm_workthat's our current challenge18:38
sean-k-mooneyits more a case of manageing dependices.18:38
sean-k-mooneykeystone has none outside oslo18:39
melwitt+1 running services at different versions is reasonable. it's designed to work that way. there are likely limitations (like placement needs to be >= nova) but prior to that things were supposed to work with mixed versions18:39
sean-k-mooneyplacemnt only depneds on keystone18:39
sean-k-mooneyectra18:39
sean-k-mooneyrm_work: are you using a containerised install18:39
sean-k-mooneyor package install18:39
rm_workmixed :P18:39
rm_workwe create our own portable venvs for most services18:39
rm_workbut placement will be our first actual container18:39
sean-k-mooneyah perfect18:39
melwittfast-forward is ok but skip-level is not, you could get screwed if data migration code was removed in the middle of what you're skipping18:40
sean-k-mooneyi was going to point out the bigest obstical to different version will be oslo requirements18:40
sean-k-mooneyvenvs or container solve that18:40
rm_workyeah, the thought of installing multiple openstack services in the SAME PYTHON ENV is just ... i know it's halloween, but that's TOO scary18:40
sean-k-mooneyrm_work: well we test that in the gate18:41
sean-k-mooneybut only on a singel version18:41
sean-k-mooneyat a minium you would want 1 python env per release18:41
rm_workyeah honestly I think a lot of stuff will become a lot simpler once people are willing to accept that it just isn't good practice to do IMO T_T18:41
sean-k-mooneybut if you are already doing that per servicce then cool18:41
rm_workor rather, if we were willing to make the decision to say "no, one env per service", think of how much simpler G-R management would be, heh18:42
sean-k-mooneywell co installablity is currenlty a goal/requirement18:42
rm_workright, but imagine if it weren't! :P18:42
sean-k-mooneypackaging would be a pain and you would have to use containers18:43
sean-k-mooneywell or venvs18:43
sean-k-mooneyso it would make distro lives harder18:43
rm_workwell, I guess to be fair packaging venvs is ... not STRICTLY recommended by the pip/venv folks18:43
rm_workseeing as how they keep trying to break it / make it harder to create portable venvs18:43
rm_workand saying "this was never actually intended to work"18:43
sean-k-mooney:)18:44
sean-k-mooneybut ya do think about shareing your experince with multiple versions at some point18:44
*** psyton has quit IRC18:47
*** niedbalski has quit IRC19:07
*** mtreinish has quit IRC19:19
*** mtreinish has joined #openstack-nova19:20
*** nweinber has quit IRC19:25
*** nweinber has joined #openstack-nova19:25
*** mtreinish has quit IRC19:29
*** mtreinish has joined #openstack-nova19:29
*** rchurch has quit IRC19:36
*** rchurch has joined #openstack-nova19:40
*** vesper has quit IRC20:02
*** vesper11 has joined #openstack-nova20:21
*** vesper has joined #openstack-nova20:27
*** Yumeng has quit IRC20:29
*** vesper11 has quit IRC20:30
*** nweinber has quit IRC20:38
*** tosky has joined #openstack-nova20:39
*** mlavalle has quit IRC20:40
*** vesper11 has joined #openstack-nova20:48
*** vesper has quit IRC20:48
openstackgerritmelanie witt proposed openstack/nova stable/queens: [stable-only] Add functional test for bug 1731668  https://review.opendev.org/75663621:44
openstackbug 1731668 in OpenStack Compute (nova) queens "placement: claim allocations fails with IndexError in _ensure_lookup_table_entry" [Low,In progress] https://launchpad.net/bugs/1731668 - Assigned to melanie witt (melwitt)21:44
openstackgerritmelanie witt proposed openstack/nova stable/queens: [stable-only] Use a separate transaction for reading after race  https://review.opendev.org/75663721:44
*** spatel has quit IRC21:46
*** slaweq has quit IRC21:48
*** purplerbot has quit IRC21:59
*** dansmith has quit IRC21:59
*** rm_work has quit IRC21:59
*** jrosser has quit IRC21:59
*** Alon_KS has quit IRC21:59
*** zer0c00l has quit IRC21:59
*** zer0c00l has joined #openstack-nova21:59
*** rm_work has joined #openstack-nova21:59
*** jrosser has joined #openstack-nova21:59
*** dansmith has joined #openstack-nova21:59
*** purplerbot has joined #openstack-nova22:00
*** k_mouza has joined #openstack-nova22:00
*** Alon_KS has joined #openstack-nova22:00
*** slaweq has joined #openstack-nova22:04
*** k_mouza has quit IRC22:05
*** slaweq has quit IRC22:14
*** raildo has quit IRC22:37
*** rcernin has joined #openstack-nova22:46
*** k_mouza has joined #openstack-nova22:53
*** k_mouza has quit IRC22:58
*** ociuhandu has joined #openstack-nova23:05
*** ociuhandu has quit IRC23:09
*** macz_ has quit IRC23:47
« Tuesday, 2020-10-27 Index Thursday, 2020-10-29 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!