*** rcernin has quit IRC | 00:08 | |
*** rcernin has joined #openstack-nova | 00:16 | |
openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Improve policy doc for supported scope info https://review.opendev.org/762013 | 00:31 |
---|---|---|
*** mlavalle has quit IRC | 00:33 | |
openstackgerrit | MaAoyu proposed openstack/os-traits master: bump py37 to py3 in tox.ini https://review.opendev.org/757432 | 00:44 |
*** gyee has quit IRC | 00:55 | |
*** k_mouza has joined #openstack-nova | 01:01 | |
*** k_mouza has quit IRC | 01:06 | |
*** brinzhang has joined #openstack-nova | 01:08 | |
openstackgerrit | Merged openstack/nova stable/victoria: Update pci stat pools based on PCI device changes https://review.opendev.org/761700 | 01:09 |
*** Liang__ has joined #openstack-nova | 01:24 | |
*** Liang__ has quit IRC | 01:32 | |
*** Liang__ has joined #openstack-nova | 01:35 | |
*** brinzhang_ has quit IRC | 01:37 | |
*** LinPeiWen has quit IRC | 01:40 | |
*** LinPeiWen has joined #openstack-nova | 02:04 | |
*** zzzeek has quit IRC | 02:28 | |
*** zzzeek has joined #openstack-nova | 02:33 | |
*** brinzhang_ has joined #openstack-nova | 02:38 | |
*** brinzhang has quit IRC | 02:41 | |
*** hamalq has quit IRC | 02:44 | |
*** zzzeek has quit IRC | 02:56 | |
*** zzzeek has joined #openstack-nova | 02:57 | |
*** Liang__ has quit IRC | 03:19 | |
*** Liang__ has joined #openstack-nova | 03:20 | |
*** psachin has joined #openstack-nova | 03:22 | |
*** zzzeek has quit IRC | 03:31 | |
*** mkrai has joined #openstack-nova | 03:31 | |
*** zzzeek has joined #openstack-nova | 03:32 | |
*** JamesBenson has quit IRC | 03:41 | |
*** dklyle has quit IRC | 03:43 | |
*** rcernin has quit IRC | 04:12 | |
*** dave-mccowan has quit IRC | 04:19 | |
*** dave-mccowan has joined #openstack-nova | 04:20 | |
*** zzzeek has quit IRC | 04:26 | |
*** macz_ has joined #openstack-nova | 04:28 | |
*** zzzeek has joined #openstack-nova | 04:29 | |
*** ratailor has joined #openstack-nova | 04:30 | |
*** macz_ has quit IRC | 04:33 | |
*** rcernin has joined #openstack-nova | 04:35 | |
openstackgerrit | jichenjc proposed openstack/nova master: Print more helpful info when qemu validation failed https://review.opendev.org/762035 | 04:52 |
*** vishalmanchanda has joined #openstack-nova | 04:57 | |
*** viks____ has joined #openstack-nova | 04:59 | |
*** JamesBenson has joined #openstack-nova | 05:04 | |
*** amodi has quit IRC | 05:05 | |
*** LinPeiWen has quit IRC | 05:08 | |
*** evrardjp has quit IRC | 05:33 | |
*** evrardjp has joined #openstack-nova | 05:33 | |
*** k_mouza has joined #openstack-nova | 05:34 | |
*** JamesBenson has quit IRC | 05:36 | |
*** k_mouza has quit IRC | 05:38 | |
*** brinzhang_ has quit IRC | 06:01 | |
*** brinzhang_ has joined #openstack-nova | 06:01 | |
*** Yumeng has joined #openstack-nova | 06:02 | |
*** brinzhang_ has quit IRC | 06:02 | |
*** brinzhang_ has joined #openstack-nova | 06:03 | |
*** brinzhang_ has quit IRC | 06:04 | |
*** brinzhang_ has joined #openstack-nova | 06:04 | |
*** LinPeiWen has joined #openstack-nova | 06:05 | |
*** LinPeiWen has quit IRC | 06:24 | |
*** LinPeiWen has joined #openstack-nova | 06:26 | |
openstackgerrit | Xinran WANG proposed openstack/nova-specs master: SRIOV Enabled Nic Support Specification https://review.opendev.org/742785 | 06:28 |
*** zzzeek has quit IRC | 06:28 | |
*** zzzeek has joined #openstack-nova | 06:33 | |
*** mkrai has quit IRC | 06:58 | |
openstackgerrit | Xinran WANG proposed openstack/nova-specs master: SRIOV Enabled Nic Support Specification https://review.opendev.org/742785 | 07:03 |
*** mkrai has joined #openstack-nova | 07:06 | |
*** links has joined #openstack-nova | 07:13 | |
*** links has quit IRC | 07:14 | |
*** rcernin has quit IRC | 07:14 | |
*** brinzhang0 has joined #openstack-nova | 07:21 | |
*** ralonsoh has joined #openstack-nova | 07:22 | |
*** brinzhang_ has quit IRC | 07:24 | |
*** brinzhang_ has joined #openstack-nova | 07:29 | |
*** mkrai has quit IRC | 07:30 | |
*** mkrai has joined #openstack-nova | 07:30 | |
*** tosky has joined #openstack-nova | 07:32 | |
*** brinzhang0 has quit IRC | 07:33 | |
*** slaweq has joined #openstack-nova | 07:44 | |
*** zzzeek has quit IRC | 07:48 | |
*** zzzeek has joined #openstack-nova | 07:50 | |
*** tesseract has joined #openstack-nova | 08:01 | |
*** rpittau|afk is now known as rpittau | 08:08 | |
*** andrewbonney has joined #openstack-nova | 08:11 | |
*** tosky has quit IRC | 08:23 | |
*** xek has joined #openstack-nova | 08:23 | |
*** rcernin has joined #openstack-nova | 08:32 | |
*** xek_ has joined #openstack-nova | 08:35 | |
*** xek has quit IRC | 08:38 | |
*** luksky has joined #openstack-nova | 08:40 | |
*** xek_ has quit IRC | 08:42 | |
*** xek_ has joined #openstack-nova | 08:43 | |
*** rcernin has quit IRC | 08:56 | |
*** dtantsur|afk is now known as dtantsur | 09:01 | |
bauzas | good morning Nova | 09:05 |
*** mkrai has quit IRC | 09:08 | |
*** luksky has quit IRC | 09:09 | |
*** martinkennelly has joined #openstack-nova | 09:12 | |
*** xek__ has joined #openstack-nova | 09:29 | |
*** tosky has joined #openstack-nova | 09:31 | |
*** derekh has joined #openstack-nova | 09:31 | |
*** xek_ has quit IRC | 09:32 | |
*** ociuhandu has joined #openstack-nova | 09:33 | |
*** mkrai has joined #openstack-nova | 09:33 | |
lyarwood | Morning | 09:41 |
*** ociuhandu has quit IRC | 09:49 | |
*** xek__ has quit IRC | 09:50 | |
*** xek__ has joined #openstack-nova | 09:52 | |
*** lifeless has joined #openstack-nova | 09:53 | |
*** xek_ has joined #openstack-nova | 10:06 | |
*** lpetrut has joined #openstack-nova | 10:08 | |
*** xek__ has quit IRC | 10:09 | |
*** xek_ has quit IRC | 10:17 | |
*** ociuhandu has joined #openstack-nova | 10:18 | |
*** xek_ has joined #openstack-nova | 10:19 | |
*** masayukig has quit IRC | 10:24 | |
*** NobodyCam has quit IRC | 10:25 | |
*** xek__ has joined #openstack-nova | 10:25 | |
*** flaviof has quit IRC | 10:25 | |
*** NobodyCam has joined #openstack-nova | 10:26 | |
*** TheJulia has quit IRC | 10:26 | |
*** rpittau has quit IRC | 10:26 | |
*** johnsom has quit IRC | 10:26 | |
*** masayukig has joined #openstack-nova | 10:26 | |
*** jangutter_ has joined #openstack-nova | 10:26 | |
*** flaviof has joined #openstack-nova | 10:26 | |
*** rpittau has joined #openstack-nova | 10:27 | |
*** johnsom has joined #openstack-nova | 10:28 | |
*** TheJulia has joined #openstack-nova | 10:28 | |
*** xek_ has quit IRC | 10:28 | |
*** jangutter has quit IRC | 10:30 | |
*** Liang__ has quit IRC | 10:32 | |
gibi | o/ | 10:38 |
*** k_mouza has joined #openstack-nova | 10:49 | |
*** jangutter_ is now known as jangutter | 11:03 | |
*** alex_xu has joined #openstack-nova | 11:07 | |
*** k_mouza has quit IRC | 11:10 | |
*** k_mouza has joined #openstack-nova | 11:11 | |
gibi | "Delay in Elastic Search: Up to date" | 11:22 |
gibi | hm, did infra cleared the 144 hours of queue? | 11:22 |
*** zzzeek has quit IRC | 11:29 | |
*** zzzeek has joined #openstack-nova | 11:31 | |
stephenfin | lyarwood, gibi, kashyap: Could you folks cast your eye over https://review.opendev.org/#/q/topic:bp/smarter-usb-devices this week? | 11:32 |
gibi | added to my list, but it is now behind the cyborg shelve/unshelve patch where I'm really late already. | 11:34 |
kashyap | stephenfin: On a phone; will queue, sir | 11:35 |
*** luksky has joined #openstack-nova | 11:36 | |
*** rcernin has joined #openstack-nova | 11:40 | |
*** brinzhang0 has joined #openstack-nova | 11:45 | |
*** dtantsur is now known as dtantsur|brb | 11:45 | |
*** brinzhang_ has quit IRC | 11:49 | |
*** Luzi has joined #openstack-nova | 11:51 | |
*** rcernin has quit IRC | 11:53 | |
*** tbachman has quit IRC | 11:54 | |
*** k_mouza has quit IRC | 11:55 | |
*** k_mouza has joined #openstack-nova | 11:57 | |
*** JamesBenson has joined #openstack-nova | 12:02 | |
*** JamesBenson has quit IRC | 12:03 | |
*** JamesBenson has joined #openstack-nova | 12:04 | |
sean-k-mooney | lyarwood: is "libvirt.libvirtError: internal error: missing block job data for disk 'vda'" something that is currently happening on bionic | 12:04 |
sean-k-mooney | it looks like that is what is calling the grenade multi node job to fail during a paused live migratrion | 12:05 |
*** ociuhandu has quit IRC | 12:06 | |
lyarwood | sean-k-mooney: yeah https://bugs.launchpad.net/nova/+bug/1901739 - I should move this back to open | 12:07 |
openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,Fix released] - Assigned to Lee Yarwood (lyarwood) | 12:07 |
lyarwood | updated | 12:09 |
* lyarwood looks at https://review.opendev.org/#/c/742056/ | 12:09 | |
sean-k-mooney | ok do w ehave a workaround e.g moving master to focal for grenade | 12:09 |
sean-k-mooney | victoria should have been focal anyway right | 12:10 |
lyarwood | sean-k-mooney: correct | 12:10 |
lyarwood | sean-k-mooney: and we are going to backport moving these jobs to focal to stable/victoria AFAIK | 12:10 |
lyarwood | brb | 12:11 |
*** k_mouza has quit IRC | 12:16 | |
sean-k-mooney | oh you removed the parent of the grenade job by mistake but https://review.opendev.org/#/c/742056/ corrects it and movs to v3 | 12:16 |
*** xek__ has quit IRC | 12:17 | |
*** xek__ has joined #openstack-nova | 12:18 | |
*** Yumeng has quit IRC | 12:18 | |
*** ratailor has quit IRC | 12:19 | |
*** k_mouza has joined #openstack-nova | 12:22 | |
*** xek_ has joined #openstack-nova | 12:28 | |
sean-k-mooney | lyarwood: could we make the multinode grenade job nonvoting until https://review.opendev.org/#/c/742056/ is merged | 12:31 |
*** xek__ has quit IRC | 12:32 | |
sean-k-mooney | i think we also will need to use the cloud archive to get libvirt 6.0 for some of the other stable branches | 12:32 |
sean-k-mooney | that have to run on bionic | 12:32 |
sean-k-mooney | am i correct in assuming ussuri proably has support for blockdev too or was that added in victoria | 12:34 |
sean-k-mooney | https://bugs.launchpad.net/nova/+bug/1901739/comments/6 is the root cause right? | 12:34 |
openstack | Launchpad bug 1901739 in OpenStack Compute (nova) " libvirt.libvirtError: internal error: missing block job data for disk 'vda'" [High,In progress] - Assigned to Lee Yarwood (lyarwood) | 12:34 |
*** ociuhandu has joined #openstack-nova | 12:40 | |
sean-k-mooney | i think im going to repopose https://opendev.org/openstack/devstack/commit/7f7f488bc385dd707a3a6d8dae7859bbe72182e5 with victoria instead | 12:40 |
*** ociuhandu has quit IRC | 12:48 | |
*** dtantsur|brb is now known as dtantsur | 12:59 | |
*** k_mouza has quit IRC | 13:05 | |
*** k_mouza has joined #openstack-nova | 13:05 | |
kashyap | sean-k-mooney: Yeah; the workaround is mentioned in the bug as a comment | 13:06 |
* kashyap hopes he "committed' the comment | 13:06 | |
sean-k-mooney | using libvirt 6.0.0 | 13:07 |
kashyap | Yep | 13:07 |
sean-k-mooney | really that just means we are not fixing the issue form openstack and raising the min libvirt | 13:07 |
sean-k-mooney | which is not really a good thing | 13:07 |
kashyap | sean-k-mooney: It's using the legacy "-drive" approach; and the modern one ("-blockdev") should fix it | 13:07 |
kashyap | sean-k-mooney: It's not an OpenStack issue | 13:08 |
sean-k-mooney | yep i know | 13:08 |
sean-k-mooney | and i also know we cant force libvirt to only use drive or blockdev | 13:08 |
sean-k-mooney | which is why we can workaournd it form nova | 13:08 |
* kashyap nods | 13:09 | |
sean-k-mooney | my point is for all deployments that cant use libvirt 6 there is no way for them to work around this | 13:09 |
sean-k-mooney | well excpet upgrade | 13:09 |
sean-k-mooney | anmyway i think the victoia cloud archive had 6.0.0 | 13:09 |
kashyap | Yeah; the whole backports / how far back should upstream support is a tricky thing | 13:10 |
sean-k-mooney | so im going to bump the version we use in devstack on the older branches | 13:10 |
kashyap | The answer is: "if you want such backported fixes", use an "enterprise" distro | 13:10 |
kashyap | (So goes the argument) | 13:10 |
sean-k-mooney | well no | 13:10 |
sean-k-mooney | the anser is that libvirt could actully maintain branches and do backports | 13:10 |
kashyap | Well, they do that | 13:10 |
sean-k-mooney | they dont which forces distros to do it | 13:10 |
kashyap | But how you have to backport is an upstream decision | 13:11 |
sean-k-mooney | yep its just a strange one | 13:11 |
kashyap | They maintain several "stable" branches | 13:11 |
sean-k-mooney | very few work the way they do | 13:11 |
sean-k-mooney | anyway this is simple tweek | 13:11 |
sean-k-mooney | since it packaged in the uca | 13:12 |
kashyap | I never claimed it's all perfect :) | 13:12 |
sean-k-mooney | if it means i dont have to keep rechecking stuff then thats good enough | 13:12 |
lyarwood | sorry just back from lunch | 13:13 |
sean-k-mooney | lyarwood: im assuming the zuulv3 patch will take a while to merge | 13:13 |
lyarwood | hmm that's a good point about the UCA, why isn't the grenade bionic job using it? | 13:13 |
sean-k-mooney | lyarwood:so im just going to swap to victoria uca which has 6.0.0 | 13:13 |
sean-k-mooney | it is but train | 13:13 |
lyarwood | ah | 13:13 |
lyarwood | right kk | 13:14 |
sean-k-mooney | we when with train when we tought the other qemu detach thing was a focal issue | 13:14 |
lyarwood | and yeah either way the multinode grenade change isn't simple | 13:14 |
sean-k-mooney | that gives us 5.2.0 | 13:14 |
sean-k-mooney | on tain i think | 13:14 |
lyarwood | even calling the old scripts is borked as it assumes we are using devstack-gate etc | 13:14 |
lyarwood | sean-k-mooney: ack | 13:14 |
lyarwood | http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/bionic-updates/ I don't see Victoria listed here however | 13:16 |
*** ociuhandu has joined #openstack-nova | 13:18 | |
kashyap | stephenfin: Really nice rework here - https://review.opendev.org/#/c/756551/ | 13:18 |
kashyap | (Also the commit message :)) | 13:18 |
sean-k-mooney | i kind fo feel like using tabels in a commit is cheating but ya it explains things well | 13:19 |
*** k_mouza has quit IRC | 13:23 | |
*** tbachman has joined #openstack-nova | 13:25 | |
*** k_mouza has joined #openstack-nova | 13:30 | |
kashyap | sean-k-mooney: Hehe; what else would you use? | 13:30 |
kashyap | In my books, it's perfectly fair game to see tables in a commit message :) | 13:31 |
*** macz_ has joined #openstack-nova | 13:33 | |
*** macz_ has quit IRC | 13:37 | |
*** derekh has quit IRC | 13:40 | |
*** derekh has joined #openstack-nova | 13:42 | |
*** ociuhandu has quit IRC | 13:42 | |
stephenfin | gibi: Have you seen the comment on https://review.opendev.org/#/c/738482/ ? | 13:42 |
*** ociuhandu has joined #openstack-nova | 13:42 | |
stephenfin | gibi: There's a bug report filed for it here https://bugs.launchpad.net/tripleo/+bug/1903655 | 13:46 |
openstack | Launchpad bug 1903655 in tripleo "Compute component jobs in master branch are failing with ERROR nova nova.exception.DBNotAllowed: nova-compute attempted direct database access which is not allowed by policy" [Critical,Triaged] | 13:46 |
*** ociuhandu has quit IRC | 13:47 | |
gibi | stephenfin: thanks for the notification, I haven't seen it | 13:50 |
gibi | yet | 13:50 |
gibi | I have to be on a call, but after It I will look into it | 13:58 |
*** mlavalle has joined #openstack-nova | 13:59 | |
*** ociuhandu has joined #openstack-nova | 14:04 | |
*** lbragstad has joined #openstack-nova | 14:05 | |
*** xek__ has joined #openstack-nova | 14:08 | |
*** xek_ has quit IRC | 14:10 | |
*** xek has joined #openstack-nova | 14:11 | |
openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: WIP - Image and flavor defined ephemeral storage encryption https://review.opendev.org/752284 | 14:11 |
lyarwood | ^ reviews welcome on that spec now btw, still WIP but hopefully ready for serious reviews. I'll have PoC code updated this week once I've written the func tests. | 14:13 |
*** xek__ has quit IRC | 14:13 | |
*** brinzhang_ has joined #openstack-nova | 14:15 | |
*** brinzhang0 has quit IRC | 14:18 | |
*** elod has quit IRC | 14:21 | |
*** nweinber has joined #openstack-nova | 14:22 | |
*** elod has joined #openstack-nova | 14:23 | |
*** artom has quit IRC | 14:25 | |
*** artom has joined #openstack-nova | 14:26 | |
*** jamesdenton has quit IRC | 14:36 | |
* bauzas is about to cry about duplicate code in the unittests for build_and_run_instance() method :) | 14:53 | |
sean-k-mooney | that does a lot | 14:56 |
sean-k-mooney | im guessing the duplicaiton is a result of that but havent looke at the test for it in a long time | 14:57 |
*** xek has quit IRC | 15:02 | |
*** amodi has joined #openstack-nova | 15:02 | |
*** xek has joined #openstack-nova | 15:03 | |
*** Luzi has quit IRC | 15:07 | |
*** rpittau is now known as rpittau|bbl | 15:14 | |
gibi | stephenfin dansmith: I've looked into https://bugs.launchpad.net/tripleo/+bug/1903655 and it seems that tripleoo configures [api_database]/connection for the nova-compute service and that causes that the service version check assumes that we are in a top level controller service which can access the api database https://github.com/openstack/nova/blob/master/nova/utils.py#L1064-L1072 | 15:16 |
openstack | Launchpad bug 1903655 in tripleo "Compute component jobs in master branch are failing with ERROR nova nova.exception.DBNotAllowed: nova-compute attempted direct database access which is not allowed by policy" [Critical,Triaged] | 15:16 |
dansmith | gibi: yeah I saw your analysis and I'm sure you're right | 15:17 |
gibi | is ther a smarter way to decided if we are inside a cell? | 15:17 |
dansmith | we have other such checks I think, so this is probably just the first time they've hit something fatal to even notice | 15:17 |
dansmith | no, I think this is a good thing | 15:17 |
dansmith | although, hmm | 15:17 |
dansmith | er, yeah, this is just compute that's failing | 15:18 |
dansmith | so yeah, I think this is good | 15:18 |
gibi | same could happen with a cell controller as well | 15:18 |
gibi | I mean a cell conductor | 15:18 |
*** nweinber has quit IRC | 15:18 | |
dansmith | yep, but that's fine | 15:18 |
*** nweinber has joined #openstack-nova | 15:19 | |
dansmith | a cell conductor shouldn't be, but can be configured with api_connection (for the last few reasons why you might need it), but it'll just survey the whole cluster instead of the cell, but that should be fine | 15:19 |
gibi | I see, the remaining upcalls | 15:20 |
dansmith | right | 15:20 |
stephenfin | gibi: That sounds reasonable. What's the remedy though? I assume we can't simply say "don't do this", at least not without a deprecation window? | 15:22 |
gibi | OK then now I feel better about asking tripleoo folks to check and fix there nova-cpu.conf | 15:22 |
gibi | stephenfin: if this is an invalid config then nova-compute is allowed to fail on it | 15:23 |
dansmith | gibi: definitely. we don't deploy it that way in devstack, and I'm pretty sure we've had and fixed this issue downstream | 15:23 |
dansmith | I'm surprised the upstream tripleo is still doing it | 15:23 |
stephenfin | Okay, can we add a check to nova-compute startup to hard fail on that config being present? | 15:23 |
stephenfin | So the error is more obvious? | 15:23 |
dansmith | sure | 15:23 |
gibi | stephenfin: that is a good point, but I think the current service level check runs first and fail first | 15:24 |
gibi | stephenfin: so I cannot add this to the _inithost | 15:24 |
gibi | or I can but it is pointless | 15:24 |
*** lpetrut has quit IRC | 15:24 | |
gibi | buuut I might be able add someting to the cmd | 15:24 |
dansmith | so, we have another such check in rpcapi, and a catch for them doing it wrongly, with a warning to explain | 15:25 |
dansmith | https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L439-L457 | 15:26 |
dansmith | so, I'm guessing they've also not configured upgrade_levels=auto, | 15:26 |
dansmith | else they would be exploding there too | 15:26 |
gibi | could be, my tripleoo knowledge is close to 0 | 15:26 |
sean-k-mooney | i think we have a down stream bug for this | 15:27 |
dansmith | point being, this is not a new behavior or requirement | 15:27 |
sean-k-mooney | owalsh was fixing it | 15:27 |
dansmith | sean-k-mooney: I thought he did a while ago, which is why I'm surprised we're still seeing it | 15:27 |
sean-k-mooney | we hilighted that there nova.conf was wrong | 15:27 |
gibi | dansmith: thanks for the pointer I will add that to my comment in the bugreport asking tripleoo to fix the conf | 15:27 |
sean-k-mooney | im not sure if it has merged | 15:27 |
dansmith | gibi: ++ | 15:28 |
sean-k-mooney | he was rewriting the config gen to not put the db creds in the compute nodes | 15:28 |
sean-k-mooney | we merged a hack to make 16.0 work but i dont know if the real fix is merged | 15:28 |
sean-k-mooney | ill check | 15:28 |
sean-k-mooney | https://review.opendev.org/#/c/718552/ | 15:29 |
dansmith | gibi: we have another check for api.connection in conductor, which is how it knows if it's a cell or global conductor. doesn't affect compute, but just FYI for prior art | 15:29 |
sean-k-mooney | that is the ooo fix | 15:29 |
sean-k-mooney | its not merged yet | 15:29 |
dansmith | sean-k-mooney: ah yeah, from april.. hence my assuming it was done | 15:29 |
dansmith | 2020 has been a bit of a time warp | 15:29 |
sean-k-mooney | dansmith: they used to config auto | 15:29 |
sean-k-mooney | the hack was to not | 15:29 |
owalsh | hey... | 15:29 |
dansmith | sean-k-mooney: ah, heh | 15:30 |
sean-k-mooney | owalsh: o/ | 15:30 |
sean-k-mooney | this is the bug https://bugs.launchpad.net/tripleo/+bug/1871482 | 15:30 |
openstack | Launchpad bug 1871482 in tripleo "nova-compute must not configure api database" [High,In progress] - Assigned to Oliver Walsh (owalsh) | 15:30 |
sean-k-mooney | this was the hack https://review.opendev.org/#/c/737287/ | 15:30 |
sean-k-mooney | which is train only | 15:30 |
sean-k-mooney | not on master | 15:30 |
sean-k-mooney | that is why tehy are hitting this now | 15:31 |
sean-k-mooney | victoria and train dont have it | 15:31 |
sean-k-mooney | *victoria and master | 15:31 |
owalsh | ok, they need to stop doing that. RPC version pinning is pointless with FFU | 15:31 |
dansmith | owalsh: note that this new failure is unrelated to pinning, | 15:32 |
dansmith | but to a check for service versions which fails on the compute if they have api_database.connection set on the compute | 15:32 |
sean-k-mooney | yep that was the root cause of the orginal issue | 15:32 |
sean-k-mooney | so its the same thing | 15:33 |
dansmith | sean-k-mooney: right, they just can't work around this new one by disabling =auto | 15:33 |
sean-k-mooney | ah ok | 15:33 |
sean-k-mooney | well i think they only get this check if we have auto set | 15:33 |
dansmith | no | 15:33 |
dansmith | this is now done at compute startup | 15:33 |
sean-k-mooney | no? | 15:33 |
dansmith | always | 15:34 |
sean-k-mooney | yes | 15:34 |
sean-k-mooney | oh instead of condtionally | 15:34 |
sean-k-mooney | got it | 15:34 |
owalsh | ok, this is master only? | 15:34 |
dansmith | the fail is master only, | 15:35 |
sean-k-mooney | so ya not a nova bug https://review.opendev.org/#/c/718552/ just need to be finished | 15:35 |
dansmith | we backported it to V but as a warning only | 15:35 |
gibi | dansmith: I'm affraid the backport also blows | 15:35 |
sean-k-mooney | we shoudl be backporting the ooo change in anycase | 15:35 |
dansmith | ohh, right right | 15:35 |
dansmith | because it tries and explodes before it warns | 15:35 |
gibi | yes | 15:35 |
dansmith | gibi: we could catch DBNotAllowed and warn on stable | 15:35 |
gibi | dansmith: yes, I will do that | 15:35 |
dansmith | gibi: maybe just do that on master too, to satisfy stephenfin's concern and then we can backport it cleanly | 15:36 |
sean-k-mooney | well do we need to | 15:36 |
gibi | dansmith: good idea | 15:36 |
gibi | thanks | 15:36 |
sean-k-mooney | we can but we shoudl be fixing ooo | 15:36 |
sean-k-mooney | they are genering an invalid config | 15:36 |
dansmith | sean-k-mooney: we're just talking about logging more helpful messages, | 15:36 |
dansmith | and avoiding a breakage on stable | 15:36 |
dansmith | but yes, this is a ooo thing to fix | 15:37 |
sean-k-mooney | yep and it was orginally a 16.1 relase blocker | 15:37 |
gibi | yes, I want to keep stable stable, or else elod will haunt me | 15:37 |
dansmith | sean-k-mooney: we backported this as a warning, but we'll explode before we warn now, if they have this config, so we need to also catch this error and warn for that too to avoid introducing a new startup fail on stable | 15:37 |
gibi | ^^++ | 15:37 |
sean-k-mooney | sure that a good thing to do | 15:38 |
sean-k-mooney | im just going to find the downstream bz and bump it with a blocker flag request before 16.2 | 15:38 |
dansmith | yup | 15:39 |
owalsh | ack, I think https://review.opendev.org/718552 is pretty much done but I wanted to run some manual tests for corner cases before removing the -W | 15:39 |
stephenfin | Going with a warning rather than exploding on master would be less aggressive, for sure | 15:42 |
dansmith | we still need to abort startup on master, | 15:43 |
stephenfin | yup, agreed | 15:43 |
dansmith | we can just catch the DB error, and log something relevant before we then abort | 15:43 |
gibi | yes | 15:43 |
stephenfin | gotcha | 15:44 |
stephenfin | I know you already know, but the addition of the abort needs a reno | 15:44 |
dansmith | and since the error is compute specific, we can do that in utils and know that it will only happen on computes, for the log | 15:44 |
*** dklyle has joined #openstack-nova | 15:44 | |
dansmith | gibi already added one yeah? | 15:44 |
dansmith | https://review.opendev.org/#/c/761923/1/releasenotes/notes/warn-when-services-started-with-old-compute-fc80b4ff58a2aaea.yaml | 15:45 |
dansmith | this is going to abort for the same reason | 15:45 |
gibi | yepp, reno is there | 15:46 |
gibi | it is a bit different reno on the master | 15:46 |
stephenfin | I thought it would abort because we had API DB configuration on a compute node? | 15:46 |
gibi | this is the master one https://review.opendev.org/#/c/738482/10/releasenotes/notes/prevent-starting-services-with-old-compute-fc80b4ff58a2aaea.yaml | 15:46 |
bauzas | dansmith: just a quick question, amirite to delete https://review.opendev.org/#/c/761452/1/nova/compute/manager.py@a5115 for the RPC API 6.0 ? | 15:46 |
gibi | stephenfin: I can extend the reno to explain the api config issue too | 15:47 |
dansmith | gibi: you could catch the db exception and then do the single-cell check if you get it, | 15:47 |
gibi | dansmith: that is a good point too | 15:47 |
dansmith | gibi: avoiding the different behavior, although that won't help get people to stop configuring their computes wrong | 15:47 |
*** ociuhandu has quit IRC | 15:48 | |
stephenfin | gibi: I think that would be good | 15:48 |
dansmith | so I guess decide whether you want to lump that in or not | 15:48 |
gibi | I think if the rule is not to configure api db for the compute service, then it is clearer to fail on it | 15:48 |
bauzas | dansmith: because even if the wallaby computes would use 5.11 (as it's the victoria API version) in case of a rolling upgrade, then they will only get objects ? | 15:48 |
dansmith | gibi: so, you could make this not explode, and then follow up with a separate check of just that config thing with a separate abort, separate reno, etc.. just saying you _can_ separate them I think, if you want | 15:49 |
sean-k-mooney | gibi: you should not configre any db creds on teh compute not just api | 15:49 |
dansmith | sean-k-mooney: indeed | 15:49 |
sean-k-mooney | since we removed the local conductor in icehouose they have not been used | 15:49 |
gibi | dansmith: ahh, so just split the change into two commit | 15:49 |
sean-k-mooney | *icehouse ish | 15:49 |
gibi | dansmith: I can do that | 15:49 |
dansmith | gibi: if you want.. make the existing check smart if it hits the compute error, knowing it should fall back to the single-cell check, | 15:50 |
dansmith | gibi: and then a separate patch to check that no db creds are set | 15:50 |
dansmith | gibi: I'm happy to write that second one if you want | 15:50 |
gibi | dansmith: thanks, I think I can do both | 15:50 |
gibi | and I will make them separate | 15:50 |
stephenfin | so in one patch, we're catching that error and logging a warning instead (or falling back to the second call), and then in another we hard fail if DB configuration is present on a compute node? | 15:50 |
dansmith | gibi: okay | 15:50 |
stephenfin | maybe I should just wait for the patches :) | 15:51 |
dansmith | stephenfin: yes | 15:51 |
stephenfin | lovely :-) | 15:51 |
gibi | :) | 15:51 |
dansmith | bauzas: you're asking if you can effectively only support 5.11 and not 5.0 in the bridge release that also supports 6.0? | 15:52 |
*** mkrai has quit IRC | 15:52 | |
bauzas | dansmith: yeah, or whether we could wait until X | 15:52 |
*** xek_ has joined #openstack-nova | 15:52 | |
bauzas | tbh, we would need to wait | 15:52 |
*** xek has quit IRC | 15:53 | |
*** xek_ has quit IRC | 15:53 | |
sean-k-mooney | bauzas: well 6.0 would be 5.11 right | 15:53 |
dansmith | bauzas: I think technically that works, but it's cheating of course... I guess I'd prefer to wait just to make it fully clean, | 15:53 |
bauzas | but given the 5.1 version was for something like Queens, then I'm pretty sure that none of the clients would pass a legacy dict | 15:53 |
sean-k-mooney | that what we normally do | 15:53 |
dansmith | but if it's something very hard to hold across the line, then maybe worth it | 15:53 |
bauzas | dansmith: okay, then I'll remove it | 15:53 |
bauzas | and we would remove those for X | 15:53 |
sean-k-mooney | dansmith: isnt that what we did for the 5.0 and 4.0 bumps | 15:54 |
bauzas | in case people look at my change later, maybe they wouldn't understand why we would remove this | 15:54 |
bauzas | sean-k-mooney: we still need to support 5.x in Wallaby | 15:54 |
bauzas | at least 5.11 | 15:54 |
*** xek_ has joined #openstack-nova | 15:54 | |
*** rpittau|bbl is now known as rpittau | 15:54 | |
dansmith | bauzas: I'm not sure what you've decided.. are you going to leave the 5.0 stuff until X or strip it down to only be 5.11? | 15:54 |
bauzas | dansmith: tbh, I don't have a opinion | 15:54 |
sean-k-mooney | i think we stiped it for 5.0 | 15:55 |
bauzas | dansmith: but after what you told, let's not delete the 5.0 support in my change | 15:55 |
*** macz_ has joined #openstack-nova | 15:55 | |
sean-k-mooney | we could jsut check the patch | 15:55 |
dansmith | if you do it now, then you've got a lot of change, right? de-5.0ing the existing code, supporting 6.0, plus anything else we add between now and then, followed by a de-5.11ing in X | 15:55 |
bauzas | we could discuss about whether we would delete the <5.11 support before X | 15:55 |
dansmith | whereas if you leave it all, then you just de-5.x the whole thing at once | 15:55 |
bauzas | dansmith: yzah | 15:55 |
bauzas | anyway, nevermind | 15:55 |
bauzas | I'll return it | 15:56 |
dansmith | sean-k-mooney: I don't think I stripped the older 4.x support when we added 5.0 until after we dropped 4.x, but I could be wrong | 15:56 |
sean-k-mooney | this was the 5.0 bump https://github.com/openstack/nova/commit/a761e57368280b6d3e931831ecd393fd5787b3ef#diff-91f79786d7e3744c39926c88bbafe3b727630fa4eb48e845686d7f12f876d067 | 15:56 |
bauzas | sean-k-mooney: dansmith: nevermind, again | 15:57 |
sean-k-mooney | so it loks like queens supported both | 15:57 |
dansmith | sean-k-mooney: that's the drop of 4.x not the bump | 15:57 |
sean-k-mooney | then we droped in rocky | 15:57 |
dansmith | right | 15:57 |
bauzas | for the main change, I'll just provide a new 6.0 RPC version but we will support 5.0 | 15:57 |
sean-k-mooney | so i guess we do that | 15:57 |
sean-k-mooney | so drop in X | 15:57 |
dansmith | and you notice all the 4.x intermediate version parameter defaults are still there | 15:58 |
bauzas | and then we could do a remove in a later change that we could drop for 5.0 to 5.10, or just wait for X | 15:58 |
dansmith | bauzas: wait for X IMHO | 15:58 |
bauzas | yeah | 15:58 |
bauzas | it would be simplier | 15:58 |
*** LinPeiWen has quit IRC | 16:00 | |
*** ociuhandu has joined #openstack-nova | 16:02 | |
*** vishalmanchanda has quit IRC | 16:06 | |
*** hemna has quit IRC | 16:09 | |
*** hemna has joined #openstack-nova | 16:10 | |
*** xek__ has joined #openstack-nova | 16:11 | |
*** jmlowe has quit IRC | 16:13 | |
*** xek_ has quit IRC | 16:14 | |
*** jmlowe has joined #openstack-nova | 16:16 | |
openstackgerrit | Sylvain Bauza proposed openstack/nova master: WIP: Bump the Compute RPC API to version 6.0 https://review.opendev.org/761452 | 16:25 |
*** xek__ has quit IRC | 16:30 | |
*** gyee has joined #openstack-nova | 16:31 | |
*** xek__ has joined #openstack-nova | 16:31 | |
*** xek_ has joined #openstack-nova | 16:33 | |
*** xek__ has quit IRC | 16:36 | |
*** iurygregory has quit IRC | 16:42 | |
*** luksky has quit IRC | 16:45 | |
*** ociuhandu_ has joined #openstack-nova | 16:45 | |
*** ociuhandu has quit IRC | 16:48 | |
*** ociuhandu_ has quit IRC | 16:49 | |
*** recyclehero has quit IRC | 16:51 | |
*** recyclehero has joined #openstack-nova | 16:53 | |
*** dave-mccowan has quit IRC | 16:56 | |
*** tesseract has quit IRC | 16:58 | |
*** recyclehero has quit IRC | 17:01 | |
*** hemna has quit IRC | 17:01 | |
*** hemna has joined #openstack-nova | 17:02 | |
*** recyclehero has joined #openstack-nova | 17:03 | |
*** hamalq has joined #openstack-nova | 17:10 | |
*** hamalq has quit IRC | 17:10 | |
*** hamalq has joined #openstack-nova | 17:11 | |
*** k_mouza has quit IRC | 17:14 | |
*** k_mouza has joined #openstack-nova | 17:15 | |
*** hoonetorg has quit IRC | 17:17 | |
*** hoonetorg has joined #openstack-nova | 17:18 | |
*** k_mouza has quit IRC | 17:19 | |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Improve error handling during service level check https://review.opendev.org/762175 | 17:21 |
openstackgerrit | Balazs Gibizer proposed openstack/nova master: Prevent nova-compute start with DB credentials https://review.opendev.org/762176 | 17:21 |
gibi | stephenfin, dansmith, sean-k-mooney: ^^ I followed what we discussed (or I hope so) | 17:21 |
*** takamatsu has quit IRC | 17:22 | |
gibi | I will leave for today now but tomorrow I can fix comments | 17:22 |
gibi | o/ | 17:22 |
*** takamatsu has joined #openstack-nova | 17:23 | |
stephenfin | gibi: ta | 17:25 |
*** hoonetorg has quit IRC | 17:26 | |
*** nweinber has quit IRC | 17:31 | |
*** nweinber has joined #openstack-nova | 17:31 | |
*** hoonetorg has joined #openstack-nova | 17:32 | |
*** xek_ has quit IRC | 17:32 | |
*** xek_ has joined #openstack-nova | 17:33 | |
*** iurygregory has joined #openstack-nova | 17:33 | |
*** xek__ has joined #openstack-nova | 17:46 | |
*** ralonsoh has quit IRC | 17:49 | |
*** xek_ has quit IRC | 17:49 | |
*** tbachman has quit IRC | 17:50 | |
*** tbachman has joined #openstack-nova | 17:50 | |
sean-k-mooney | gibi: yep that is what we discussed or at least what i understood we said we should do. | 18:03 |
*** derekh has quit IRC | 18:10 | |
*** psachin has quit IRC | 18:15 | |
*** rpittau is now known as rpittau|afk | 18:20 | |
*** takamatsu has quit IRC | 18:22 | |
*** takamatsu has joined #openstack-nova | 18:24 | |
*** dtantsur is now known as dtantsur|afk | 18:25 | |
*** viks____ has quit IRC | 18:39 | |
*** jamesdenton has joined #openstack-nova | 18:52 | |
*** andrewbonney has quit IRC | 18:59 | |
*** xek__ has quit IRC | 19:09 | |
*** xek__ has joined #openstack-nova | 19:10 | |
*** xek__ has quit IRC | 19:17 | |
*** xek__ has joined #openstack-nova | 19:18 | |
*** jangutter_ has joined #openstack-nova | 19:26 | |
*** jangutter has quit IRC | 19:29 | |
*** gmann is now known as gmann_lunch | 19:29 | |
openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: WIP libvirt: Allow the default machine type to be changed https://review.opendev.org/762199 | 19:44 |
openstackgerrit | Lee Yarwood proposed openstack/nova-specs master: WIP libvirt: Allow the default machine type to be changed https://review.opendev.org/762199 | 19:45 |
*** k_mouza has joined #openstack-nova | 20:01 | |
*** k_mouza has quit IRC | 20:05 | |
*** martinkennelly has quit IRC | 20:09 | |
*** iurygregory has quit IRC | 20:25 | |
*** xek__ has quit IRC | 20:32 | |
*** xek__ has joined #openstack-nova | 20:33 | |
*** k_mouza has joined #openstack-nova | 20:36 | |
*** xek_ has joined #openstack-nova | 20:40 | |
*** k_mouza has quit IRC | 20:40 | |
*** tosky has quit IRC | 20:42 | |
*** xek__ has quit IRC | 20:42 | |
*** tosky has joined #openstack-nova | 20:42 | |
*** xek_ has quit IRC | 20:43 | |
*** rcernin has joined #openstack-nova | 21:10 | |
*** rcernin has quit IRC | 21:10 | |
*** rcernin has joined #openstack-nova | 21:11 | |
*** rcernin has quit IRC | 21:33 | |
*** rcernin has joined #openstack-nova | 21:37 | |
*** bbowen has quit IRC | 21:43 | |
*** slaweq has quit IRC | 21:47 | |
*** iurygregory has joined #openstack-nova | 21:51 | |
*** nweinber has quit IRC | 21:54 | |
*** raildo has quit IRC | 22:01 | |
*** gmann_lunch is now known as gmann | 22:27 | |
*** ociuhandu has joined #openstack-nova | 22:46 | |
*** iurygregory has quit IRC | 22:47 | |
*** iurygregory has joined #openstack-nova | 22:49 | |
*** ociuhandu has quit IRC | 22:50 | |
*** tosky has quit IRC | 22:55 | |
*** zzzeek has quit IRC | 23:13 | |
*** zzzeek has joined #openstack-nova | 23:14 | |
*** bbowen has joined #openstack-nova | 23:14 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!