| *** lbragstad has quit IRC | 00:14 | |
| *** cdent has quit IRC | 00:36 | |
| *** gyee has quit IRC | 00:48 | |
| *** ileixe has joined #openstack-nova | 00:56 | |
| *** igordc has quit IRC | 01:16 | |
| *** Kevin_Zheng has joined #openstack-nova | 01:17 | |
| *** imacdonn has quit IRC | 01:19 | |
| *** mriedem has quit IRC | 01:26 | |
| *** lbragstad has joined #openstack-nova | 01:35 | |
| *** tbachman has joined #openstack-nova | 01:40 | |
| *** BjoernT has joined #openstack-nova | 02:01 | |
| *** BjoernT has quit IRC | 02:03 | |
| *** markvoelker has joined #openstack-nova | 02:43 | |
| *** janki has quit IRC | 02:44 | |
| *** cfriesen has quit IRC | 02:54 | |
| *** wolverineav has joined #openstack-nova | 02:55 | |
| *** tzumainn has quit IRC | 03:00 | |
| *** zhubx has quit IRC | 03:04 | |
| *** zhubx has joined #openstack-nova | 03:06 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec for API policy updates https://review.openstack.org/547850 | 03:29 |
|---|---|---|
| openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec for API policy updates https://review.openstack.org/547850 | 03:29 |
| *** zhubx has quit IRC | 03:38 | |
| *** zhubx007 has joined #openstack-nova | 03:38 | |
| *** zhubx007 has quit IRC | 03:41 | |
| *** zhubx has joined #openstack-nova | 03:41 | |
| *** nicolasbock has quit IRC | 03:53 | |
| *** janki has joined #openstack-nova | 04:00 | |
| *** cfriesen has joined #openstack-nova | 04:04 | |
| *** whoami-rajat has joined #openstack-nova | 04:10 | |
| *** betherly has joined #openstack-nova | 04:19 | |
| *** ileixe has quit IRC | 04:21 | |
| *** betherly has quit IRC | 04:24 | |
| *** marst has joined #openstack-nova | 04:28 | |
| *** ileixe has joined #openstack-nova | 04:34 | |
| *** marst has quit IRC | 04:45 | |
| *** lpetrut has joined #openstack-nova | 04:54 | |
| openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/627765 | 04:55 |
| *** wolverineav has quit IRC | 05:02 | |
| *** wolverineav has joined #openstack-nova | 05:02 | |
| *** wolverineav has quit IRC | 05:08 | |
| *** phasespace has quit IRC | 05:14 | |
| openstackgerrit | Merged openstack/python-novaclient stable/stein: Update .gitreview for stable/stein https://review.openstack.org/644181 | 05:15 |
| openstackgerrit | Merged openstack/python-novaclient stable/stein: Update UPPER_CONSTRAINTS_FILE for stable/stein https://review.openstack.org/644182 | 05:15 |
| *** lpetrut has quit IRC | 05:15 | |
| *** ivve has quit IRC | 05:18 | |
| *** janki has quit IRC | 05:20 | |
| *** wolverineav has joined #openstack-nova | 05:34 | |
| *** sridharg has joined #openstack-nova | 05:39 | |
| openstackgerrit | Merged openstack/os-vif stable/stein: Update .gitreview for stable/stein https://review.openstack.org/644034 | 05:42 |
| *** janki has joined #openstack-nova | 05:51 | |
| *** lbragstad has quit IRC | 06:06 | |
| *** wolverineav has quit IRC | 06:07 | |
| *** pcaruana has joined #openstack-nova | 06:11 | |
| *** ivve has joined #openstack-nova | 06:22 | |
| *** brinzhang has joined #openstack-nova | 06:28 | |
| *** wolverineav has joined #openstack-nova | 06:41 | |
| *** Luzi has joined #openstack-nova | 06:44 | |
| *** slaweq has quit IRC | 06:45 | |
| *** sridharg has quit IRC | 06:50 | |
| *** sridharg has joined #openstack-nova | 06:53 | |
| *** ivve has quit IRC | 06:58 | |
| *** lpetrut has joined #openstack-nova | 07:02 | |
| *** ivve has joined #openstack-nova | 07:03 | |
| *** liuyulong_ has joined #openstack-nova | 07:17 | |
| *** cfriesen has quit IRC | 07:17 | |
| *** phasespace has joined #openstack-nova | 07:18 | |
| *** sridharg has quit IRC | 07:21 | |
| *** sridharg has joined #openstack-nova | 07:23 | |
| *** rcernin has quit IRC | 07:24 | |
| *** dpawlik_ is now known as dpawlik | 07:30 | |
| *** xek_ has joined #openstack-nova | 07:41 | |
| *** slaweq has joined #openstack-nova | 07:41 | |
| *** luksky has joined #openstack-nova | 07:46 | |
| kashyap | sean-k-mooney: Hi, just saw the ping in the scrollback; there's a lot of text. Hope klindgren's problem is resolved :-) | 07:51 |
| *** wolverineav has quit IRC | 07:54 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Use 'writeback' QEMU cache mode when 'none' is not viable https://review.openstack.org/641981 | 08:00 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: vzstorage: Use 'writeback' QEMU cache mode https://review.openstack.org/643376 | 08:00 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: smbfs: Use 'writeback' QEMU cache mode https://review.openstack.org/643377 | 08:00 |
| *** tesseract has joined #openstack-nova | 08:02 | |
| kashyap | efried: When you get a moment, lost yours / SylvainB's +2s & +W; appreciate if you can re-ACK|NACK it. | 08:04 |
| *** liuyulong_ has quit IRC | 08:06 | |
| *** ttsiouts has joined #openstack-nova | 08:06 | |
| *** tssurya has quit IRC | 08:12 | |
| *** openstackgerrit has quit IRC | 08:17 | |
| *** dtantsur|afk is now known as dtantsur | 08:24 | |
| *** tssurya has joined #openstack-nova | 08:24 | |
| *** tosky has joined #openstack-nova | 08:27 | |
| *** helenafm has joined #openstack-nova | 08:31 | |
| *** slaweq_ has joined #openstack-nova | 08:36 | |
| *** slaweq has quit IRC | 08:38 | |
| kashyap | bauzas: ^ When you get a moment :-) | 08:39 |
| kashyap | (It already had +W, just lost the ACK due to rebase) | 08:40 |
| * bauzas looks | 08:41 | |
| *** slaweq_ has quit IRC | 08:47 | |
| *** mdbooth has joined #openstack-nova | 08:49 | |
| *** ralonsoh has joined #openstack-nova | 08:51 | |
| *** mdbooth_ has quit IRC | 08:51 | |
| *** whoami-rajat has quit IRC | 09:10 | |
| *** whoami-rajat has joined #openstack-nova | 09:18 | |
| *** sridharg has quit IRC | 09:19 | |
| *** IvensZambrano has joined #openstack-nova | 09:19 | |
| kashyap | bauzas: Docs will be fixed in a separate change | 09:24 |
| kashyap | To not let the core patch be blocked | 09:24 |
| bauzas | cool with me then | 09:24 |
| kashyap | bauzas: Please see the review comments :-) MattR and MattB both agreed on the change | 09:24 |
| kashyap | That to let this merge, and I fix the rest of the documenation | 09:24 |
| bauzas | just comment and say you'll add a fup | 09:24 |
| bauzas | sorry, fup == follow-up change | 09:24 |
| kashyap | Yeah, it's actually added | 09:24 |
| kashyap | bauzas: Also at the end of the commit message I noted as such | 09:24 |
| kashyap | [quote] | 09:25 |
| kashyap | Do the minimum required update to the `disk_cachemodes` config help text. (In a future patch, rewrite the cache modes documentation to fix confusing fragments and outdated information.) | 09:25 |
| kashyap | [/quote | 09:25 |
| *** SASAA has joined #openstack-nova | 09:27 | |
| bauzas | kashyap: sure, but I still feel we need to document in the relnote why we now recommend writeback as default | 09:27 |
| kashyap | bauzas: Rel Note is also there | 09:27 |
| bauzas | basically because modern OSes fsync | 09:27 |
| kashyap | bauzas: The default depends on what file system one is using | 09:27 |
| *** sridharg has joined #openstack-nova | 09:27 | |
| bauzas | but if operators are scary about that, they should just keep writethrough and not use default | 09:27 |
| bauzas | kashyap: yup, but you're not telling it in the relnote | 09:28 |
| kashyap | bauzas: The documentation fix follow right after that. | 09:28 |
| bauzas | operators litterally have to load a lot of context | 09:28 |
| kashyap | bauzas: No need for the operators to be "scary" here. There are lots of myths about cache modes | 09:28 |
| bauzas | again, I'm talking about relnotes | 09:28 |
| kashyap | Let me re-read it | 09:28 |
| bauzas | tbc, my -1 is due to https://review.openstack.org/#/c/641981/13/releasenotes/notes/writeback-cache-mode-for-guests-a7e4d2806c956164.yaml@8 | 09:29 |
| kashyap | bauzas: Okay, saw your comment on the rel note -- again, not required to add your comment | 09:29 |
| kashyap | bauzas: Really, that's not required. The guest OSes that don't fsync() are really really really horribly old and we don't support | 09:29 |
| bauzas | kashyap: I mean, we're making an opinionated choice, right? If so, let's just capture it in the relnote | 09:30 |
| bauzas | and not just in the commit msg, which is bad for operators | 09:30 |
| bauzas | and technically, I accepted it as a fix, but this isn't really a fix | 09:30 |
| kashyap | bauzas: Huh. That's what the rel note says | 09:30 |
| bauzas | it's a performance improvement | 09:30 |
| kashyap | bauzas: I don't follow you. What is not really a fix? | 09:30 |
| kashyap | Well, it's both. | 09:31 |
| bauzas | let's take an example | 09:31 |
| bauzas | I have a car which is running | 09:31 |
| kashyap | (1) Fixing wrong assumption in Nova; (2) And (1) brings performance improvement | 09:31 |
| bauzas | for sure, the engine is running | 09:31 |
| bauzas | but I can make sure that I can improve the engine by changing some, say, injector | 09:32 |
| bauzas | that will possibly make my engine having trouble when I stop, for example, but that's depending on my driving | 09:32 |
| bauzas | so, I like to know that I can improve my engine by changing my driving | 09:32 |
| bauzas | kashyap: again, the relnote doesn't mention a bit about guest OSes | 09:33 |
| bauzas | it's just litterally saying "heh, we're changing a default value" | 09:33 |
| bauzas | there is no other contextual information | 09:33 |
| bauzas | if you say "ok, I'm going to amend the relnote in a fup", fine with me | 09:34 |
| bauzas | or I could just do it | 09:34 |
| bauzas | but I just want to make sure operators know about the reasons we decided | 09:34 |
| kashyap | bauzas: Mentioning guest OSes really confuses at this point. | 09:34 |
| kashyap | If an operator has longer attention span, they will carefully read the documentation | 09:35 |
| sean-k-mooney | kashyap: if it was just for perfromance we would not be changing the default however right | 09:35 |
| sean-k-mooney | that is just a side benifit | 09:36 |
| kashyap | Huh, who said it was just for performance? | 09:36 |
| bauzas | kashyap: don't expect operators to load contect on every relnote | 09:36 |
| bauzas | context* | 09:36 |
| bauzas | it's a massive amount of information they have to digest | 09:36 |
| bauzas | and we suffered some pain in the past by overexpecting their knowledge | 09:37 |
| kashyap | bauzas: I know very well about that. I take great care in not assuming people will have context. | 09:37 |
| gibi | stephenfin: do you have any suggestion about the table issue in https://review.openstack.org/#/c/644810/1/specs/stein/approved/bandwidth-resource-provider.rst@624 | 09:37 |
| bauzas | anyway, I don't want to spend more time on it, will spin a new revision | 09:37 |
| *** priteau has joined #openstack-nova | 09:38 | |
| bauzas | gibi: stephenfin is mostly eaten by meetings all week | 09:38 |
| gibi | bauzas: thanks I will try to fight sphinx alone then :) | 09:38 |
| bauzas | kashyap: let me just provide a new revision | 09:38 |
| kashyap | bauzas: Hang on | 09:38 |
| sean-k-mooney | kashyap could you update the release not with the explaning you gave to me about the 3 booleans that the cachemodes set to explain why its safe to ues writeback | 09:38 |
| kashyap | bauzas: Just post your new text as a comment | 09:39 |
| sean-k-mooney | kashyap: that is the context that ops will be missing. it was the context i was missing when i saw the commit first | 09:39 |
| sean-k-mooney | ops wont see the commit message so all they have is the release note | 09:39 |
| bauzas | sean-k-mooney: yeah I agree | 09:40 |
| * kashyap needs to be AFK; will address it later | 09:40 | |
| bauzas | sean-k-mooney: kashyap did a great work on documenting it in a docstring https://review.openstack.org/#/c/641981/13/nova/virt/libvirt/driver.py@410 | 09:41 |
| bauzas | but we don't take the opportunity to explain it to ops | 09:41 |
| bauzas | while it could be scary "heh, wtf, now I could loose data integrity?" | 09:41 |
| sean-k-mooney | right well quickly looking at that it could be more or less copied into the release note | 09:42 |
| sean-k-mooney | it might be a little verbose but that is not a bad thing | 09:42 |
| *** derekh has joined #openstack-nova | 09:43 | |
| sean-k-mooney | i think all of the context is already in the patch just need to duplicate some of it into the release note | 09:43 |
| *** pcaruana has quit IRC | 09:45 | |
| *** pcaruana has joined #openstack-nova | 09:46 | |
| *** wolverineav has joined #openstack-nova | 09:51 | |
| bauzas | mdbooth: kashyap: sean-k-mooney: so, I provided an example of what I'd like to see in the relnote | 09:55 |
| bauzas | if operators don't want to expect modern OSes, it's their choices | 09:55 |
| mdbooth | bauzas: I saw that. Honestly, though, this hasn't been an issue for at least a decade, maybe 2. | 09:55 |
| bauzas | mdbooth: not really, you can enable caching on some OSes that don't flush, right? | 09:56 |
| mdbooth | I think it just increases cognitive load on the reader for no practical benefit. | 09:56 |
| sean-k-mooney | mdbooth: there are still people running centos 5 | 09:56 |
| bauzas | from my recall, W2K is one of those | 09:56 |
| mdbooth | sean-k-mooney: Centos 5 definitely flushes! | 09:56 |
| sean-k-mooney | mdbooth: didnt it have issue with flushing | 09:56 |
| sean-k-mooney | ok | 09:56 |
| sean-k-mooney | that was the distro that i always worried about for this kind of thing | 09:57 |
| mdbooth | If your OS doesn't flush data on request, you're going to find out about that real soon. | 09:57 |
| sean-k-mooney | i know it does not partcally like virtio disks as i useulaly had to revert to useing other disk bus like ide to avoid kernel issues | 09:58 |
| mdbooth | I remember having to be very careful shutting down my old 286-clone Tandy running DOS, though. | 09:58 |
| bauzas | oh, I remember some PITAs I got with Win2K guests in the past with OpenStack Essex | 09:58 |
| bauzas | I'm pretty sure they weren't flushing | 09:59 |
| sean-k-mooney | im sure that at least 50% of all openstack workloads :P | 09:59 |
| bauzas | but this could have been fixed in a SP or something else | 09:59 |
| sean-k-mooney | dos gaming in the cloud | 09:59 |
| bauzas | actually, that's a good question now we have containerizing OSes like Atomic | 10:00 |
| bauzas | do we still flush on disk ? | 10:00 |
| sean-k-mooney | with containers? | 10:00 |
| sean-k-mooney | that shoudl not change | 10:00 |
| mdbooth | bauzas: It's really an application issue: your application needs to flush. | 10:01 |
| mdbooth | If the application flushes, the OS should also flush. | 10:01 |
| mdbooth | The only thing writeback vs writethrough is going to fix is if your application flushes but the OS does not. | 10:01 |
| mdbooth | If the application doesn't flush, the OS can't flush it anyway, as it may not have the data. | 10:02 |
| *** slaweq_ has joined #openstack-nova | 10:03 | |
| bauzas | hum, reading https://lifehacker.com/do-i-really-need-to-eject-usb-drives-before-removing-th-5863810 | 10:03 |
| bauzas | man, why is this so complicated ? | 10:04 |
| bauzas | anyway, I agree, it's maybe just a strawman | 10:04 |
| bauzas | but I still feel we need to explain it | 10:04 |
| mdbooth | bauzas: USB drives don't need O_SYNC either if your application flushes and you wait for your application to finish before unplugging. | 10:05 |
| mdbooth | Eject flushes the whole disk. | 10:05 |
| *** cdent has joined #openstack-nova | 10:08 | |
| *** zbr has quit IRC | 10:08 | |
| bauzas | yeah, just reading http://msdn.microsoft.com/en-us/library/windows/desktop/aa364218%28v=vs.85%29.aspx | 10:08 |
| bauzas | anyway, I'm balanced | 10:08 |
| bauzas | see, we're talking for 30 mins about something needing more than just a glance | 10:09 |
| bauzas | the fact that mdbooth and kashyap know the situation makes reasonable the change | 10:09 |
| bauzas | but I just feel we need to capture this knowledge | 10:09 |
| bauzas | and explain it thru the relnote | 10:10 |
| *** zbr has joined #openstack-nova | 10:10 | |
| *** zbr has quit IRC | 10:16 | |
| *** slaweq_ is now known as slaweq | 10:18 | |
| *** zbr has joined #openstack-nova | 10:22 | |
| *** zbr has quit IRC | 10:22 | |
| *** wolverineav has quit IRC | 10:25 | |
| *** IvensZambrano has quit IRC | 10:28 | |
| stephenfin | gibi: Looks like you solved it :) | 10:34 |
| gibi | stephenfin: yeah, I managed. | 10:35 |
| sean-k-mooney | bauzas: for the record we can talk for 30 mins about just about any topic :) | 10:51 |
| *** IvensZambrano has joined #openstack-nova | 10:52 | |
| *** jaosorior has quit IRC | 10:53 | |
| *** nicolasbock has joined #openstack-nova | 10:58 | |
| *** eharney has quit IRC | 11:16 | |
| *** priteau has quit IRC | 11:19 | |
| *** ileixe has quit IRC | 11:20 | |
| *** wolverineav has joined #openstack-nova | 11:23 | |
| *** whoami-rajat has quit IRC | 11:30 | |
| *** whoami-rajat has joined #openstack-nova | 11:34 | |
| *** ttsiouts has quit IRC | 11:37 | |
| *** ttsiouts has joined #openstack-nova | 11:37 | |
| *** zbr has joined #openstack-nova | 11:38 | |
| *** wolverineav has quit IRC | 11:40 | |
| kashyap | bauzas: Just back; ah, I mistook your formatting comment | 11:41 |
| *** luksky has quit IRC | 11:41 | |
| *** ttsiouts has quit IRC | 11:42 | |
| *** eharney has joined #openstack-nova | 11:43 | |
| *** dave-mccowan has joined #openstack-nova | 11:50 | |
| *** jaosorior has joined #openstack-nova | 11:51 | |
| *** openstackgerrit has joined #openstack-nova | 11:52 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Fix links to neutron QoS minimum bandwidth doc https://review.openstack.org/645084 | 11:52 |
| *** panda is now known as panda|lunch | 11:52 | |
| *** dave-mccowan has quit IRC | 11:53 | |
| *** IvensZambrano has quit IRC | 11:56 | |
| *** tbachman has quit IRC | 11:57 | |
| *** IvensZambrano has joined #openstack-nova | 11:57 | |
| kashyap | sean-k-mooney: CentOS-5 is perfectly fine w.r.t. flushes. Again, as noted only *horribly* old OSes that don't matter (like RHEL-4 or even older versions of Windoze) | 12:01 |
| kashyap | ... that had problems. And they're not relevant. | 12:01 |
| sean-k-mooney | kashyap: ya | 12:02 |
| kashyap | sean-k-mooney: /me reworks the rel note; and you're right - some of the info I could add there | 12:02 |
| kashyap | But as Matt noted, I was being careful of not adding needless cognitive load to the reader. | 12:02 |
| sean-k-mooney | i just have had data curption issue with centos 5 and centos 5 based vnf in the past and was not sure if that was due to flushing | 12:02 |
| *** cdent has quit IRC | 12:04 | |
| sean-k-mooney | that said it could equally have been down to the use of ext2/3? i cant rememebr but all i recal was we had to be very care to ensure we shutdown cleanly or we woudl have to do a rebuild with a clean image and/or do a rescue to fix the file system | 12:05 |
| * kashyap nods | 12:06 | |
| *** eharney has quit IRC | 12:09 | |
| *** ttsiouts has joined #openstack-nova | 12:13 | |
| kashyap | sean-k-mooney: bauzas: Before I upload, does this read better? -- https://kashyapc.fedorapeople.org/PS14-writeback-cache-mode-for-guests-a7e4d2806c956164.yaml.txt | 12:15 |
| kashyap | mdbooth: ^ | 12:15 |
| kashyap | (Hit refresh, if you already clicked it :D) | 12:17 |
| sean-k-mooney | haha i had | 12:17 |
| kashyap | Sorry, removed a duplication. (/me is obsessed with words) | 12:18 |
| sean-k-mooney | you also just fixed an issue i ws goint to raise | 12:19 |
| sean-k-mooney | we are not ment to use phases like "we're" | 12:19 |
| kashyap | Which one? | 12:19 |
| kashyap | Yeah, I know. Contractions, etc. | 12:19 |
| kashyap | I find it a bit irritating, though. I'd assume people (even non-native speakers) are competent enough to parse "we're" as "we are" | 12:20 |
| kashyap | But I see where "I'd" can can confusion, as it can read: "I had", or "I would" :D | 12:20 |
| sean-k-mooney | actully its not the contractrions | 12:20 |
| *** luksky has joined #openstack-nova | 12:20 | |
| kashyap | Oh? | 12:20 |
| sean-k-mooney | we are ment to say nova in its the wrong perspective | 12:21 |
| *** markvoelker has quit IRC | 12:22 | |
| kashyap | tox -e releasenotes | 12:23 |
| sean-k-mooney | https://docs.openstack.org/doc-contrib-guide/writing-style/general-writing-guidelines.html#write-in-second-person | 12:23 |
| kashyap | Err, wrong window | 12:23 |
| sean-k-mooney | "Use first person plural pronouns (we, our) judiciously." we are allowed to use "we" in some cases but its generally not teh perfered pharsing | 12:24 |
| kashyap | Right, I normally avoid using "we", beceause it is so loaded | 12:24 |
| kashyap | Thanks for the pointer, though :-) | 12:24 |
| sean-k-mooney | kashyap: am so to your origninal question i would remove "And OSes do that." | 12:25 |
| *** tbachman has joined #openstack-nova | 12:25 | |
| sean-k-mooney | i dont think its needed but otherswise that looks ok | 12:25 |
| kashyap | sean-k-mooney: Not quite sure, otherwise it leaves the reader in the dark. They'll wonder: "will the OS do or not". But all OSes "that matter" do that | 12:26 |
| sean-k-mooney | perhaps | 12:26 |
| kashyap | I'll say "all modern OSes" | 12:27 |
| sean-k-mooney | ya that read better how about "All modern OSes flush data as requried." | 12:28 |
| sean-k-mooney | "And OSes do that." i a sentence fragment which is why i originly suggested removing it | 12:28 |
| kashyap | It follows the previous sentence :-) But point noted. | 12:28 |
| kashyap | I'll do that quick lunch | 12:28 |
| sean-k-mooney | yes which is what makes it a sentence fragment :) it only makes sense when read in the context of the previous sentence. when read in isolate what does "that" refer to :) | 12:29 |
| sean-k-mooney | anyway enjoy lunch | 12:30 |
| kashyap | Okido, will join them :-) I'm all for clarity | 12:30 |
| *** zhubx has quit IRC | 12:35 | |
| *** zhubx007 has joined #openstack-nova | 12:35 | |
| *** mriedem has joined #openstack-nova | 12:38 | |
| *** dtantsur is now known as dtantsur|brb | 12:52 | |
| *** lbragstad has joined #openstack-nova | 12:54 | |
| *** eharney has joined #openstack-nova | 12:56 | |
| mriedem | bauzas: want to help get this in for queens? https://review.openstack.org/#/c/611945/ | 12:57 |
| * bauzas clicks | 13:06 | |
| *** panda|lunch is now known as panda|sick | 13:07 | |
| *** irclogbot_1 has quit IRC | 13:07 | |
| bauzas | mriedem: done | 13:08 |
| *** irclogbot_1 has joined #openstack-nova | 13:09 | |
| mriedem | thanks | 13:09 |
| *** brinzhang has quit IRC | 13:11 | |
| *** yaawang has quit IRC | 13:11 | |
| *** yaawang has joined #openstack-nova | 13:12 | |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: Use 'writeback' QEMU cache mode when 'none' is not viable https://review.openstack.org/641981 | 13:18 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: vzstorage: Use 'writeback' QEMU cache mode https://review.openstack.org/643376 | 13:18 |
| openstackgerrit | Kashyap Chamarthy proposed openstack/nova master: libvirt: smbfs: Use 'writeback' QEMU cache mode https://review.openstack.org/643377 | 13:18 |
| *** cdent has joined #openstack-nova | 13:20 | |
| *** BjoernT has joined #openstack-nova | 13:20 | |
| *** marst has joined #openstack-nova | 13:21 | |
| *** awaugama has joined #openstack-nova | 13:22 | |
| *** altlogbot_3 has quit IRC | 13:23 | |
| *** altlogbot_0 has joined #openstack-nova | 13:24 | |
| *** takashin has joined #openstack-nova | 13:27 | |
| *** janki has quit IRC | 13:28 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Re-propose cross-cell-resize spec for Train https://review.openstack.org/642807 | 13:31 |
| mriedem | dansmith: i tried to wordsmith the personality files limitation in the re-proposed cross-cell resize spec ^ | 13:31 |
| *** brinzhang has joined #openstack-nova | 13:32 | |
| dansmith | okay I guess I'll have to diff it to see the changed part | 13:32 |
| *** brinzhang has quit IRC | 13:33 | |
| sean-k-mooney | mriedem: were we recreating the config dirve during a cross cell migration or something that would cause issue for personality files? | 13:35 |
| mriedem | we are | 13:36 |
| *** wolverineav has joined #openstack-nova | 13:37 | |
| mriedem | we spawn https://review.openstack.org/#/c/635080/13/nova/compute/manager.py@5145 | 13:37 |
| sean-k-mooney | ok then back to the conversation we had yesterday. if its oke to do it for cross cell resize it logically means we may be able to do the same in general. perhaps contoled by a config option to addes klindgren concers regarding ssh keys ans scp | 13:38 |
| cdent | scanning across the big window and seeing "mriedem: we spawn" is chilling and worrying. Somewhere in the basement the alien mriedem's are laying eggs | 13:38 |
| *** gaoyan has joined #openstack-nova | 13:38 | |
| mriedem | muwahaha | 13:38 |
| mriedem | don't worry, i only harvest their organs | 13:39 |
| *** altlogbot_0 has quit IRC | 13:39 | |
| *** whoami-rajat has quit IRC | 13:40 | |
| mriedem | sean-k-mooney: i'm not necessarily saying it's ok, but i don't have a good solution right now for dealing with those during a cross-cell resize | 13:40 |
| mriedem | without glance or nova-compute running it's own http server... | 13:41 |
| *** altlogbot_1 has joined #openstack-nova | 13:41 | |
| efried | mriedem: Didn't you (or somebody? gibi?) propose the docs for bug 1821015 ? | 13:41 |
| openstack | bug 1821015 in OpenStack Compute (nova) "Attaching virtual GPU devices to guests in nova - libvirt reshaping" [Medium,Triaged] https://launchpad.net/bugs/1821015 | 13:41 |
| mriedem | efried: no, i mentioned it to bauzas yesterday | 13:41 |
| efried | okay, thought I saw a patch come through. | 13:41 |
| mriedem | what i'd like to see in those docs is an example of the expected resource providers (2 of them) and inventory (VGPU on the child provider) in the osc command output, but since i don't have the hardware, i'd have to fake out anything about the child provider | 13:42 |
| sean-k-mooney | mriedem: ok, well if for no other reason then documentation and haveing a place to discuss if its ok i can write up a short strawman spec to propose recreating config drives on migration. | 13:42 |
| *** marst has quit IRC | 13:42 | |
| efried | mriedem: ack, thanks. | 13:42 |
| gibi | efried: I don't have VGPU hardware either | 13:43 |
| gibi | the closest I can have is to extract examples from the functional test we have | 13:43 |
| sean-k-mooney | mriedem: but in any case yes for cross cell resize the option are limited | 13:43 |
| *** irclogbot_1 has quit IRC | 13:45 | |
| bauzas | mriedem: gibi: don't worry, I'm working on it | 13:46 |
| openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add request-filter-image-types spec https://review.openstack.org/644625 | 13:46 |
| *** jaypipes has quit IRC | 13:46 | |
| gibi | bauzas: cool, thanks. Hit me with the review when it is ready | 13:46 |
| *** irclogbot_3 has joined #openstack-nova | 13:46 | |
| dansmith | sean-k-mooney: yeah, I don't think it's okay either | 13:47 |
| dansmith | sean-k-mooney: the user doesn't know they are getting a cross-cell resize | 13:47 |
| sean-k-mooney | dansmith: that is true but what i was thinking was proposing. if file injection is disable we always recreate the config drive | 13:48 |
| sean-k-mooney | dansmith: since that is the default the behavior sould be the same for cross cell and normal resizes | 13:49 |
| dansmith | sean-k-mooney: as I said yesterday, file injection being disabled at the compute has nothing to do with whether or not the files end up in config drive (at least as I understand it) | 13:49 |
| sean-k-mooney | oh i was not aware of that | 13:49 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Override the 'get' method in DriverBlockDevice class https://review.openstack.org/638821 | 13:49 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153 | 13:50 |
| sean-k-mooney | ok. that is partly why i was just going to creat a strawman spec as i honestly have not tought it true propely | 13:50 |
| *** hemna has joined #openstack-nova | 13:50 | |
| openstackgerrit | Takashi NATSUME proposed openstack/python-novaclient master: Remove deprecated options https://review.openstack.org/642312 | 13:50 |
| dansmith | I have not chased it down fully, but I believe this is where we pass the injected files to the configdrive creation process: https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L3632-L3635 | 13:50 |
| sean-k-mooney | i was pretty sure other would find holes in it | 13:51 |
| *** jistr is now known as jistr|call | 13:52 | |
| mriedem | dansmith: fix the footnotes docs issue in your spec and i'm +2 | 13:54 |
| mriedem | dansmith: i also mentioned a rebuild thing which you can document or omit, so long as we remember it during implementation | 13:54 |
| dansmith | gah, okay | 13:54 |
| dansmith | mriedem: can I leave those footnotes as bullets? | 13:55 |
| mriedem | they aren't really footnotes then | 13:55 |
| dansmith | i.e. "* .. [1] Glance..." | 13:55 |
| *** marst has joined #openstack-nova | 13:56 | |
| mriedem | i mean, yo'ure using the [1]_ format above, so use .. [1] below | 13:56 |
| dansmith | Well, I have that one caps-as-traits thing | 13:56 |
| gibi | nova meeting starts in 5 minutes on openstack-meeting | 13:56 |
| mriedem | i think it's fine to mix them | 13:56 |
| dansmith | okay will push it up and check the render | 13:56 |
| mriedem | on stephenfin will lose sleep over that | 13:56 |
| mriedem | tox -e docs :) | 13:56 |
| dansmith | too hard, let jenkins do the work | 13:56 |
| openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add request-filter-image-types spec https://review.openstack.org/644625 | 13:56 |
| mriedem | actually i only noticed it because i threw it in http://rst.ninjs.org/# | 13:57 |
| mriedem | and it complained | 13:57 |
| mriedem | i always forget how to do footnotes in rst | 13:57 |
| dansmith | me too, I looked at one of your specs to cheat, | 13:57 |
| dansmith | but then copied wrong :D | 13:57 |
| sean-k-mooney | dansmith looks like you are right. the only config value that i see that effect this is the injected_files quota and changing the behavior based on that seams wrong | 13:57 |
| mriedem | the docs builder doesn't actually fail on busted footnotes i don't think | 13:57 |
| dansmith | mriedem: but I can look at the render and see if it looks right | 13:58 |
| mriedem | regarding personality files, i will say, they are best effort and broken in lots of ways | 13:58 |
| mriedem | as noted in my spec known issue, you lose them on unshelve, you lose them on evacuate, and you lose them if you're only relying on the metadata api and get migrated | 13:58 |
| sean-k-mooney | mriedem: ya which is part of the reaosn we deprecated them. | 13:58 |
| *** phasespace has quit IRC | 13:58 | |
| dansmith | I know, but it does kinda suck | 13:59 |
| mriedem | i'm not disagreeing | 13:59 |
| mriedem | it also seems like we kind of punted on fixing that long ago | 13:59 |
| mriedem | when we talked for years about deprecating personality files, and then finally did it | 13:59 |
| dansmith | and it sucks because we suck, not because throwing files in a disk image that we generate is icky for any other reason | 13:59 |
| sean-k-mooney | on a pure policy basis do we/should we allow behavior changes that may break deprecated api features | 13:59 |
| *** jistr|call is now known as jistr | 14:00 | |
| mriedem | meeting time | 14:00 |
| dansmith | deprecated api features are no different than regular ones if they're still available, | 14:00 |
| dansmith | and especially if they're still the default for the base api version | 14:00 |
| dansmith | (IMHO) | 14:00 |
| dansmith | no nova meeting for me, I have a conflict | 14:00 |
| mriedem | agree ^ | 14:00 |
| sean-k-mooney | ok | 14:00 |
| mriedem | people can and do still use all this stuff on 2.1 | 14:00 |
| mriedem | because that's what osc defaults to | 14:00 |
| sean-k-mooney | ya true. maybe we shoudl fix osc to not default to that first :) | 14:01 |
| *** mlavalle has joined #openstack-nova | 14:05 | |
| *** lpetrut has quit IRC | 14:06 | |
| *** lpetrut has joined #openstack-nova | 14:07 | |
| *** wolverineav has quit IRC | 14:12 | |
| *** jaosorior has quit IRC | 14:15 | |
| bauzas | mriedem: just to understand correctly, you want some examples using osc-placement ? | 14:19 |
| bauzas | so I need to use my hardware then | 14:19 |
| * bauzas checks whether he still has it | 14:20 | |
| bauzas | yay | 14:20 |
| mriedem | bauzas: yeah | 14:20 |
| mriedem | because the release note is just a lot of words, | 14:20 |
| mriedem | but you really need to see an example on the command line to drive this home i think | 14:20 |
| bauzas | okay, I'll try to get some commands | 14:20 |
| bauzas | I still have a devstack node | 14:20 |
| *** whoami-rajat has joined #openstack-nova | 14:23 | |
| *** gaoyan has quit IRC | 14:25 | |
| *** hongbin has joined #openstack-nova | 14:29 | |
| *** gaoyan has joined #openstack-nova | 14:35 | |
| *** altlogbot_1 has quit IRC | 14:35 | |
| *** altlogbot_0 has joined #openstack-nova | 14:36 | |
| *** dtantsur|brb is now known as dtantsur | 14:37 | |
| *** ttsiouts has quit IRC | 14:37 | |
| *** irclogbot_3 has quit IRC | 14:38 | |
| *** ttsiouts has joined #openstack-nova | 14:38 | |
| *** BjoernT has quit IRC | 14:38 | |
| *** irclogbot_0 has joined #openstack-nova | 14:39 | |
| *** ttsiouts has quit IRC | 14:41 | |
| *** ttsiouts has joined #openstack-nova | 14:41 | |
| *** jackding has joined #openstack-nova | 14:42 | |
| jackding | mriedem: Hi Matt, could you or other cores have a look at this please? It's been sitting there for quite a while. https://review.openstack.org/#/c/621646/ | 14:43 |
| mriedem | efried: you'll want to jump in #openstack-release at some point | 14:47 |
| mriedem | i added comments and re-word suggestions to the highlight patch https://review.openstack.org/#/c/644697/ | 14:47 |
| sean-k-mooney | jackding: am we are cutting RC1 today so still in code freeze mode | 14:48 |
| jackding | sean-k-mooney: ok didn't know that | 14:49 |
| efried | mriedem: ack; what's the timeline on this, do I need to hit it or can we wait for melwitt? | 14:49 |
| sean-k-mooney | jackding: master will open for train tomorow | 14:49 |
| mriedem | efried: i think it's due today | 14:49 |
| sean-k-mooney | jackding: not sure if this backportable but ill take a look again | 14:49 |
| jackding | sean-k-mooney: will hassle cores later then :) thanks. | 14:49 |
| mriedem | http://lists.openstack.org/pipermail/openstack-discuss/2019-March/003952.html | 14:49 |
| mriedem | efried: ^ | 14:49 |
| mriedem | jackding: i can try to look when i'm not doing with release issues | 14:50 |
| mriedem | *dealing with | 14:50 |
| jackding | mriedem: great, thanks | 14:50 |
| *** panda|sick is now known as panda|drappt | 14:50 | |
| *** gaoyan has quit IRC | 14:51 | |
| bauzas | hmmm, I thought we were tagging milestones automatically? | 14:51 |
| bauzas | I don't see a 19.0.0.0b1 tag | 14:52 |
| bauzas | mriedem: ^ | 14:53 |
| mriedem | no | 14:54 |
| bauzas | or are we done with milestones ? | 14:54 |
| mriedem | not anymore | 14:54 |
| bauzas | ok I missed this crucial info | 14:54 |
| mriedem | bauzas: https://releases.openstack.org/reference/release_models.html#cycle-with-rc | 14:54 |
| bauzas | I knew we weren't needing to provide a tag | 14:54 |
| bauzas | gotcha, thanks | 14:55 |
| bauzas | I was just wanting to checkout some m-1 tag for my devstack | 14:55 |
| bauzas | but I'll rather checkout stable/rocky | 14:55 |
| mriedem | if you check out stable/rocky you won't get the nested/reshaper stuff but ok | 14:56 |
| bauzas | well, I want to verify the reshape using osc-placement | 14:57 |
| bauzas | so I can provide some examples | 14:57 |
| *** takashin has left #openstack-nova | 15:01 | |
| *** snevi has joined #openstack-nova | 15:02 | |
| *** cfriesen has joined #openstack-nova | 15:02 | |
| *** IvensZambrano has quit IRC | 15:03 | |
| *** wolverineav has joined #openstack-nova | 15:08 | |
| mriedem | cdent: looks like the vmware 3rd party CI is skipping live migration tests... http://207.189.188.190/logs/52/626952/9/check-vote/ext-nova-zuul/744e2c6/tempest_results.html.gz | 15:11 |
| mriedem | kind of sucks to report in our prelude release notes that vmware supports live migration now but it's not tested | 15:12 |
| mriedem | http://207.189.188.190/logs/52/626952/9/check-vote/ext-nova-zuul/744e2c6/tempest.log.gz#_2019-03-19_10_59_00_155 | 15:12 |
| mriedem | i shall post to the ML so it's not your issue :) | 15:12 |
| *** dpawlik is now known as dpawlik_ | 15:13 | |
| NobodyCam | Good Morning nova folks, can someone point me in the right direction to troubleshoot A instance not starting? I see the instance get scheduled I see a entry in the selected hosts compute log like: "Instance <BLAH> has been scheduled to this compute host, the scheduler has made an allocation against this compute node but the instance has yet to start" | 15:16 |
| sean-k-mooney | NobodyCam: if you have got that far your next step is to check the nova compute agent log and see if it ever recived a boot request | 15:16 |
| sean-k-mooney | if it did not then you shoudl check the nova-conductor log to ensure the down call from the conductor to the compute node was made. | 15:17 |
| *** liuyulong|away is now known as liuyulong | 15:17 | |
| dansmith | sean-k-mooney: that message is from the compute log | 15:18 |
| sean-k-mooney | if it was then the issue is related to rabbitmq and you shoudl check the memsage queue | 15:18 |
| sean-k-mooney | dansmith: oh ok | 15:18 |
| NobodyCam | sean-k-mooney: Thank you :) | 15:18 |
| sean-k-mooney | NobodyCam: from dansmith reply given the message you quated is form the nova compute agent are there any errors? | 15:20 |
| *** sapd1_x has joined #openstack-nova | 15:21 | |
| NobodyCam | no errors | 15:21 |
| NobodyCam | Rabbit appears up and running | 15:21 |
| NobodyCam | I see services connecting | 15:21 |
| *** eharney has quit IRC | 15:22 | |
| openstackgerrit | Merged openstack/nova stable/queens: Don't persist RequestSpec.requested_destination https://review.openstack.org/611945 | 15:22 |
| sean-k-mooney | NobodyCam: if you grep for the instace uuid in the logs of the nova compute agent are there any results? if so coudl you share the last couple so we know howfar it got | 15:23 |
| NobodyCam | just that message repeating itself several times | 15:24 |
| dansmith | NobodyCam: that message is not an error | 15:24 |
| *** Luzi has quit IRC | 15:25 | |
| NobodyCam | yea I not seeing nt errors | 15:25 |
| dansmith | NobodyCam: it means that the scheduler picked that host but never got so far as to give it to the host to start, so the message is saying that it's ignoring that assuming the boot request will be coming in soon | 15:25 |
| dansmith | that means it's finding it in the DB | 15:25 |
| NobodyCam | conductor only has two messages with that uuis | 15:25 |
| dansmith | so look elsewhere for logs, likely conductor | 15:25 |
| NobodyCam | uuids | 15:25 |
| NobodyCam | http://paste.openstack.org/show/hrQOgRtlb6g0GbZN1mfa/ | 15:26 |
| sean-k-mooney | NobodyCam: what state is the instance in by the way? is it still building or in error? | 15:26 |
| NobodyCam | building | 15:27 |
| sean-k-mooney | and the task is still block_device_mapping? | 15:27 |
| sean-k-mooney | or something similar | 15:27 |
| NobodyCam | task-state is scheduling | 15:28 |
| NobodyCam | http://paste.openstack.org/show/g2rPJbBSCo4Aj2KufVFN/ | 15:29 |
| sean-k-mooney | that does not really make sense. https://docs.openstack.org/nova/rocky/reference/vm-states.html#create-instance-states | 15:31 |
| sean-k-mooney | the condoctor is showing that it is creating the block device mappings in this case you were booting with epherial storage form an image | 15:31 |
| dansmith | we really should have a condoctor service | 15:32 |
| sean-k-mooney | so the vm shoudl be in in task state Block_device_mapping or spawning | 15:32 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova-specs master: Support server power state update through external event https://review.openstack.org/636132 | 15:33 |
| sean-k-mooney | and a super condoctor | 15:33 |
| NobodyCam | yea, there is not ethereal requested with at flavor | 15:33 |
| NobodyCam | http://paste.openstack.org/show/y7eD1LLJIlvkldWKOqFd/ | 15:33 |
| NobodyCam | *ephemeral | 15:34 |
| sean-k-mooney | NobodyCam: no what i ment is you are not usign boot from volume as the block device mappings show the root disk is local | 15:34 |
| NobodyCam | ahh yes! | 15:35 |
| sean-k-mooney | and from the flavor i can now see you have a root disk of 10G | 15:35 |
| NobodyCam | yes! | 15:35 |
| tssurya | dansmith: whenever you have some time, your opinion on https://review.openstack.org/#/c/636132/ would be highly valued/needed :) thanks in advance | 15:36 |
| sean-k-mooney | anyway form the sequence diagarm of the boot state transition there is definetly somthing wrong. the only time the task state shoudl be in Schduling after the conductor started setting up the disk would be a rechedule after a failed spawn i think | 15:38 |
| sean-k-mooney | unfortunetly the diagram does not capture the error paths | 15:38 |
| *** wolverineav has quit IRC | 15:42 | |
| *** ivve has quit IRC | 15:42 | |
| efried | stephenfin: You seem like an obvious choice for nova/docs liaison https://wiki.openstack.org/wiki/CrossProjectLiaisons#Documentation -- can I put you down? | 15:44 |
| dansmith | tssurya: ack | 15:44 |
| stephenfin | efried: I've already been appointed the oslo one so why not | 15:44 |
| efried | stephenfin: you mean the oslo/docs liaison or the oslo/nova liaison? Cause I could totally put you down for that too. | 15:45 |
| NobodyCam | thank you sean-k-mooney :) I will attempt to redeploy nova | 15:45 |
| stephenfin | oslo/docs | 15:45 |
| stephenfin | but sure, oslo/nova maybe makes sense too | 15:45 |
| efried | k | 15:45 |
| efried | Kevin_Zheng, nicolasbock: Y'all are currently down as the nova/docs liaisons https://wiki.openstack.org/wiki/CrossProjectLiaisons#Documentation -- are you wanting to continue in that capacity? | 15:47 |
| mriedem | efried: i think you could replace | 15:48 |
| efried | ight | 15:49 |
| efried | it seems pretty clear this list isn't used much for some of the categories. Not sure how much I ought to care about it. | 15:50 |
| *** gyee has joined #openstack-nova | 15:57 | |
| *** belmoreira has quit IRC | 15:58 | |
| openstackgerrit | Merged openstack/nova stable/rocky: ironic: check fresh data when sync_power_state doesn't line up https://review.openstack.org/640772 | 15:59 |
| openstackgerrit | Merged openstack/nova stable/rocky: pass endpoint interface to Ironic client https://review.openstack.org/643098 | 15:59 |
| efried | jackding: Did you send out a ML post for https://review.openstack.org/#/c/621646/ warning of the ComputeDriver interface change? | 15:59 |
| openstackgerrit | Merged openstack/nova master: Documentation for bandwidth support https://review.openstack.org/642064 | 15:59 |
| openstackgerrit | Merged openstack/nova master: Add a prelude release note for the 19.0.0 Stein GA https://review.openstack.org/644412 | 15:59 |
| efried | mriedem: looks like ready to cut rocky --^ ? | 15:59 |
| mriedem | efried: yup i'll propose a release | 16:00 |
| efried | k | 16:00 |
| sean-k-mooney | jackding: the change you ping about before still looks good to me by the way | 16:02 |
| gibi | efried: do you need anything else regarding https://review.openstack.org/#/c/645084/ ? | 16:02 |
| jackding | efried: no. Do I need to? it's an optional parameter that has a default value. | 16:03 |
| *** BjoernT has joined #openstack-nova | 16:03 | |
| jackding | efried: not invoked by default | 16:03 |
| efried | gibi: nope, just forgot. +A now | 16:03 |
| gibi | efried: coolio | 16:04 |
| efried | jackding: Any time the signature changes, even for an optional kwarg, it breaks OOT drivers. | 16:04 |
| efried | jackding: We're not supposed to care about them, but I still have a soft spot in my heart. | 16:04 |
| efried | gibi: Should we wait for that --^ for RC1? | 16:05 |
| efried | I guess so, since it's renos | 16:05 |
| efried | mriedem: you see anything else we should hold rc1 for? | 16:07 |
| mriedem | efried: https://review.openstack.org/#/c/642863/ | 16:08 |
| gibi | efried: yeah it is fixing the renos | 16:08 |
| gibi | efried: so I would wait for that | 16:08 |
| mriedem | the release patch can depends-on the known stuff you want to hold up for and then just adjust the hash when they land | 16:10 |
| efried | don't see a lot of value in the depends-on if the content needs to be updated anyway. Just to stop them from automerging it? | 16:11 |
| mriedem | yup | 16:11 |
| *** wwriverrat has quit IRC | 16:13 | |
| *** eharney has joined #openstack-nova | 16:14 | |
| *** sapd1_x has quit IRC | 16:16 | |
| *** imacdonn has joined #openstack-nova | 16:19 | |
| *** jaosorior has joined #openstack-nova | 16:27 | |
| *** ttsiouts has quit IRC | 16:30 | |
| *** ttsiouts has joined #openstack-nova | 16:31 | |
| *** lpetrut has quit IRC | 16:32 | |
| *** tobias-urdin has quit IRC | 16:32 | |
| *** tssurya has quit IRC | 16:34 | |
| *** ttsiouts has quit IRC | 16:35 | |
| *** BjoernT has quit IRC | 16:36 | |
| openstackgerrit | Merged openstack/nova master: Fix links to neutron QoS minimum bandwidth doc https://review.openstack.org/645084 | 16:41 |
| *** mvkr has quit IRC | 16:48 | |
| *** ivve has joined #openstack-nova | 16:54 | |
| *** rpittau is now known as rpittau|afk | 16:55 | |
| *** helenafm has quit IRC | 16:58 | |
| mdbooth | Could this be a source of random SSH failures in the gate: https://bugzilla.redhat.com/show_bug.cgi?id=1689933 ? | 17:01 |
| openstack | bugzilla.redhat.com bug 1689933 in openstack-nova "[RFE] Add a simple way to enable random number generator on all flavor/image" [Unspecified,New] - Assigned to nova-maint | 17:01 |
| *** luksky has quit IRC | 17:02 | |
| *** tesseract has quit IRC | 17:02 | |
| kashyap | bauzas: mdbooth: efried: When you get a minute, please put this one out of its misery: https://review.openstack.org/#/c/641981/ (libvirt: Use 'writeback' QEMU cache mode when 'none' is not viable) | 17:04 |
| efried | kashyap: did you want that in RC1? | 17:05 |
| bauzas | I'm just wrapping my head with devstack atm | 17:05 |
| efried | Cause that's all we're gonna merge today. | 17:05 |
| kashyap | efried: Would certainly be useful to get some initial testing | 17:05 |
| mdbooth | kashyap: That's a no ;) | 17:05 |
| efried | mriedem: RC1 candidate? | 17:05 |
| kashyap | The sooner the better, as we'd be doing a bit of a disservice to the users | 17:05 |
| kashyap | mdbooth: Hehe | 17:05 |
| *** igordc has joined #openstack-nova | 17:06 | |
| * kashyap will be off the next week; and don't want to cache all this stuff in his brain when he comes back the week after, and still seeing a Zuul +1 staring at me :-) | 17:06 | |
| mdbooth | I think it's a backport candidate, though. We'll almost certainly take it downstream. It's not a regression, though, and it won't cause dataloss or any kind of error. It's just unnecessarily slow. | 17:06 |
| mdbooth | s/I *don't* think/ | 17:06 |
| mdbooth | Erm, I do think. | 17:06 |
| * mdbooth is hungry and confused | 17:07 | |
| * kashyap gives mdbooth a glass of ... $beverage_of_choice :-) | 17:07 | |
| * mdbooth gets food | 17:07 | |
| sean-k-mooney | kashyap: that is not a regression so it normally would not qualify for an rc right? | 17:08 |
| kashyap | mdbooth: Thanks for the review so far! Appreciate it | 17:08 |
| kashyap | sean-k-mooney: Not a regression; but why inflict needless slowness? This patch can (positively) impact all new instances | 17:09 |
| sean-k-mooney | kashyap: im also not sure its technically a bug. it sound more like a specless blueprint. | 17:09 |
| sean-k-mooney | kashyap: well you can just set the config option too | 17:09 |
| sean-k-mooney | anyway this just feels like a feature that is filed as a bug which can be worked around via a config already so im not sure if it qualifes for RC incution but i also dont think you will be breaking anything | 17:11 |
| kashyap | This is fixing existing tech debt. The more it is delayed, the more I lose motivation to fix it. | 17:11 |
| sean-k-mooney | sure but when i looked at this earilar i assume this was targeting train not stien | 17:11 |
| kashyap | sean-k-mooney: Also, Huh? This is not a "specless blueprint". This is actually a misuse/bug. | 17:11 |
| sean-k-mooney | everything working but being slow is not a bug | 17:12 |
| kashyap | sean-k-mooney: True. But I won't cry into my pillow if it doesn't make it to Stein, though | 17:12 |
| kashyap | Just that I'd "strongly suggest" to get this in -- also because the "other patch" (for `qemu-img convert` case) merged. | 17:12 |
| sean-k-mooney | kashyap: didnt that merge before feature freeze | 17:13 |
| kashyap | And it's not nice to have these splatter over two releases. | 17:13 |
| kashyap | sean-k-mooney: Yep | 17:13 |
| kashyap | Frankly, I was just so buried in actually digging the issue, that I lost track of all this time. Most deadlines are arbitrary in my mind. | 17:13 |
| kashyap | (As in, wasn't paying attention whether it was merged before or after RC1) | 17:14 |
| mriedem | efried: i do not think that writeback patch is an rc1 candidate no | 17:14 |
| mriedem | it's extremely latent | 17:14 |
| mriedem | damn it where is jaypipes | 17:14 |
| efried | k, swhat I figured. | 17:14 |
| mriedem | dansmith: edleafe: cdent: i've dumped comments on surya's 'locked_reason' spec - specific feedback on (1) the data model changes and (2) rest api payload format would be appreciated https://review.openstack.org/#/c/638629/ | 17:15 |
| sean-k-mooney | kashyap: once rc1 is cut tonight it can be merged tommorow after the stable/stien branch is cut | 17:15 |
| * mriedem gets a haircut | 17:15 | |
| kashyap | mriedem: Yes, it is latent. | 17:15 |
| sean-k-mooney | the real question is is it backportable | 17:16 |
| cdent | mriedem: thanks queued | 17:16 |
| *** mriedem is now known as mriedem_grooming | 17:16 | |
| kashyap | I definitely think it is backport-worthy. See mdbooth's comment earlier | 17:16 |
| *** mvkr has joined #openstack-nova | 17:16 | |
| sean-k-mooney | kashyap: in which case it can be proposed for backport to stable/stien and the other stable branches | 17:17 |
| kashyap | mriedem_grooming: I don't agree with your emphatic "no". But whatever. I have 1000 other things that need my attention | 17:17 |
| sean-k-mooney | so its not really that much of an issue | 17:17 |
| kashyap | sean-k-mooney: Yep, so it's fine, if it's merged tomm. | 17:17 |
| * mdbooth disagrees. Everything working but being *unnecessarily* slow is a bug, imho. | 17:17 | |
| sean-k-mooney | mdbooth: do you consider qemu vert type to be a bug | 17:18 |
| kashyap | sean-k-mooney: But, I'd prefer it to be in Stein. I didn't "plan" this work. | 17:18 |
| sean-k-mooney | mdbooth: you can say yes :) | 17:18 |
| mdbooth | kashyap: I don't think it's worthy of Stein at this stage. | 17:18 |
| kashyap | mdbooth: Okido... | 17:18 |
| mdbooth | kashyap: Focus on the backport. | 17:18 |
| kashyap | I don't know; I'll focus on dinner first | 17:19 |
| mdbooth | kashyap: Good call :) | 17:19 |
| kashyap | sean-k-mooney: mdbooth: But as noted, The two volume drivers patches should _also_ merge at the same time | 17:19 |
| kashyap | For the "minimum possible doc" to make sense. | 17:19 |
| *** psachin has joined #openstack-nova | 17:23 | |
| mriedem_grooming | listen guys, i didn't say it wasn't a bug, i sait it wasn't a regression in stein, it's latent, therefore it doesn't need to be held up for rc1 which is already stressful | 17:24 |
| mriedem_grooming | so we'll deal with backports | 17:24 |
| mriedem_grooming | if people don't like my input on it then don't ask | 17:24 |
| kashyap | mriedem_grooming: Hey, I do like your input very much :-) | 17:25 |
| efried | I was content at :14:52 | 17:26 |
| kashyap | mriedem_grooming: But yep, please go ahead with RC1 stuff (after your grooming) | 17:26 |
| sean-k-mooney | efried: what change ad 14:53 | 17:27 |
| sean-k-mooney | * at | 17:27 |
| kashyap | sean-k-mooney: If I have to guess it's the "not an RC1 candidate" remark :-) | 17:27 |
| efried | :14:52, as in 14 mins 52s after whatever hour you're in (Unless you're in one of those bizarre time zones that's off by a half hour) | 17:27 |
| efried | when I said "k, swhat I figured". | 17:28 |
| sean-k-mooney | oh ha i see | 17:28 |
| efried | which was shorthand for, "I was pretty sure this wasn't a candidate for rc1, but I wasn't positive, so I asked mriedem. He voiced a similar opinion. So that is what we shall go with." | 17:29 |
| *** psachin has quit IRC | 17:29 | |
| bauzas | honestly, not a big deal (again) | 17:29 |
| bauzas | we always cut RC1 after things we wanna wait | 17:29 |
| *** dtantsur is now known as dtantsur|afk | 17:30 | |
| bauzas | the question is : should the bug we discuss be in the waitlist ? obviously not in my mind, period | 17:30 |
| sean-k-mooney | efried: by the way have you looked at the nova ptg etherpad to see what should be moved to placemetn or nova placement cross project stuff | 17:30 |
| efried | sean-k-mooney: Yes, I did that yesterday, but it will need constant mriedem_groominging | 17:30 |
| efried | could have been Tuesday | 17:31 |
| *** psachin has joined #openstack-nova | 17:31 | |
| efried | no, yesterday: http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004030.html | 17:31 |
| sean-k-mooney | efried: i originally adde the resource provider yaml but topic but is that something that anyone is going to pursue in train? | 17:32 |
| efried | sean-k-mooney: IMO that's a straight-up Nova topic, as it doesn't ask for any changes to the placement API. | 17:32 |
| sean-k-mooney | oh sorry yes it is | 17:32 |
| efried | I haven't felt a whole lot of excitement behind that effort, so I haven't been pushing it. | 17:32 |
| efried | but if you're excited, I'll get excited with you. | 17:33 |
| efried | since I proposed the thing | 17:33 |
| sean-k-mooney | i partly context switch and was wondering if there was still a desire to discuss it | 17:33 |
| *** psachin has quit IRC | 17:34 | |
| sean-k-mooney | well i think there is a pain point there but im not sure if that is the solution | 17:34 |
| efried | Had this revelation recently when talking about https://etherpad.openstack.org/p/placement-inter-provider-affinity -- would rather do small, simple, easy-to-understand things that have limited scope and power than try to do a super generic one-size-fits-all thing. | 17:34 |
| efried | and that yaml spec leans much more toward the latter. | 17:34 |
| * efried bbiab | 17:35 | |
| sean-k-mooney | ok. im going to grab lunch too. | 17:35 |
| bauzas | oh man, I'm getting mad at devstack deployments | 17:37 |
| sean-k-mooney | bauzas: something specificlly? | 17:37 |
| bauzas | neutron fails to install | 17:39 |
| bauzas | but meh, I'll fix it | 17:39 |
| sean-k-mooney | ah ok | 17:39 |
| bauzas | yeah, it's a reqs thing | 17:40 |
| bauzas | let's go for a RECLONE=yes | 17:41 |
| *** gmann is now known as gmann_afk | 17:43 | |
| NobodyCam | fyi incase anyone ever runs in to this again: issue was bad rabbit ip in nova_api/cell_mappings.transport_url | 17:45 |
| *** mvkr has quit IRC | 17:46 | |
| *** tssurya has joined #openstack-nova | 17:47 | |
| dansmith | efried: cdent: what's the status on this eventlet monkeypatch thing? I saw some discussion this morning, but when I looked the other day it had changed since my review for reasons I wasn't sure about | 17:52 |
| efried | dansmith: tldr we think it's good, but wait for after rc1 | 17:52 |
| dansmith | efried: ah right, because they said it was okay in train, cool | 17:53 |
| efried | dansmith: was gonna bug you tomorrow (after rc1 cut) to re-up your +2 | 17:53 |
| efried | well, they want it backported. We just don't want to drop it into rc1 | 17:53 |
| dansmith | right | 17:53 |
| *** ivve has quit IRC | 17:54 | |
| melwitt | mriedem_grooming, efried: thanks for getting the cycle highlights updated | 17:59 |
| efried | yw | 17:59 |
| *** derekh has quit IRC | 18:02 | |
| *** ivve has joined #openstack-nova | 18:02 | |
| *** mrch_ has quit IRC | 18:04 | |
| efried | our last rc1 patch just hit the 2h mark unstarted in the check queue :( | 18:09 |
| *** whoami-rajat has quit IRC | 18:23 | |
| *** ralonsoh has quit IRC | 18:28 | |
| openstackgerrit | Merged openstack/nova master: Move libvirt calculation of machine type to utils.py https://review.openstack.org/644554 | 18:29 |
| *** ivve has quit IRC | 18:36 | |
| *** cfriesen has quit IRC | 18:39 | |
| *** gmann_afk is now known as gmann | 18:40 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Move creation of rpcapi.ComputeAPI object from __init__ https://review.openstack.org/644998 | 18:41 |
| openstackgerrit | melanie witt proposed openstack/nova master: Move create of ComputeAPI object in websocketproxy https://review.openstack.org/644998 | 18:44 |
| jroll | afaik there is no way to limit the project(s) which can boot an instance on a given set of compute nodes, right? surely someone has proposed this before, is it a hard no? | 18:45 |
| dansmith | tenant isolation yo! | 18:49 |
| dansmith | jroll: new placementy way to do it: https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/placement-req-filter.html | 18:50 |
| jroll | oooo | 18:50 |
| dansmith | jroll: also an old filter-based approach (google for filter_tenant_id) | 18:50 |
| * jroll reads | 18:50 | |
| jroll | dansmith: this should work perfectly, thanks! | 18:51 |
| dansmith | finally, FINALLY, I do something to make jroll happy. | 18:52 |
| jroll | it only took 5 years | 18:52 |
| dansmith | better late than never | 18:52 |
| sean-k-mooney | the tenant isolation filter has some edgecases | 18:53 |
| *** mriedem_grooming is now known as mriedem | 18:53 | |
| sean-k-mooney | the main one being it limits the set of host a project can boot instnace on but does not prevent other project form booting instnaces on those same host | 18:54 |
| dansmith | sean-k-mooney: you should read that spec | 18:54 |
| dansmith | there's a provision (unlike the filter approach) that will reject if the tenant is not confined, so you can avoid that situation | 18:55 |
| *** itlinux has quit IRC | 18:55 | |
| mriedem | melwitt: did you want to re-propose the counting quotas spec so we can line that up for landing early in train? | 18:56 |
| sean-k-mooney | oh cool. well it was not really a edgecase in the old filter i guess as a desing choice | 18:56 |
| mriedem | the code i mean, after the spec is re-approved | 18:56 |
| sean-k-mooney | dansmith: nice to know the placement version accounts fot that | 18:56 |
| melwitt | mriedem: oh, yes, I do. thanks for reminding me | 18:56 |
| melwitt | let me do it now so I don't forget a third time | 18:57 |
| jroll | dansmith: the use case mentions segregate users into cells, but it sounds like the solution only depends on aggregates, not cells, am I reading that right? | 18:58 |
| mriedem | the scheduler does might be more clear than the spec https://docs.openstack.org/nova/latest/admin/configuration/schedulers.html#tenant-isolation-with-placement | 18:59 |
| mriedem | *docs | 19:00 |
| jroll | mriedem: thanks, that answers it | 19:00 |
| dansmith | jroll: correct | 19:00 |
| *** snevi has quit IRC | 19:01 | |
| sean-k-mooney | jroll: the original usecase was a cellv1 -> cellv2 perfromcne regration which is why the spec focused on cells but ya it just needs aggreates | 19:01 |
| jroll | sean-k-mooney: right, figured as much | 19:01 |
| openstackgerrit | melanie witt proposed openstack/nova-specs master: Re-propose count quota usage from placement https://review.openstack.org/645302 | 19:01 |
| amodi | dansmith: mriedem additional question related to this: im currently developing a test case on this, im getting a Scheduling error, No Hosts found. Is tempest generated project/tenant not able to execute this? cz i can run the same from admin project and it works | 19:03 |
| amodi | sorry this error: "No valid host was found. Scheduling failed: No hosts available for tenant" | 19:03 |
| dansmith | amodi: if the tenant is not allowed to any aggregate and the "require" flag is set, then that's expected yes | 19:04 |
| dansmith | sean-k-mooney: not really performance regression, but replicating the inate behavior/feature of being able to schedule to specific cells, yes | 19:05 |
| amodi | yes thats set placement_aggregate_required_for_tenants=True | 19:05 |
| amodi | dansmith: i do set the filter_tenant_id metadata to the aggregate | 19:06 |
| dansmith | amodi: and is the tempest tenant mentioned in there? | 19:07 |
| dansmith | amodi: I'd have to see what all you have and the logs, but you can see from the functional tests how it's supposed to work (and that it does) so it must be config-related | 19:07 |
| mriedem | amodi: why do we need a tempest test for that placement request filter? we have in-tree functional testing which should be sufficient | 19:07 |
| sean-k-mooney | dansmith: well cern were using the tenant isolation filter to window a set of host down to a cell based on the teant then runing the rest of the filters on that reduced set or somthing like that right. so when the went the cells v2 + placement they had some scaling issue where by if the use the placement limit feature they could get no valid host so you impmented this feature to give the same | 19:08 |
| sean-k-mooney | fucntionaliy the previously had in a placement native way. at lest that is more or less my memory of events | 19:08 |
| amodi | dansmith: yeah the filter_tenant_id= $tempest_generated_tenant_id | 19:08 |
| dansmith | sean-k-mooney: because they look for a small number of placement results it was just not working, not really scale-related, although this does/should perform better yes | 19:09 |
| dansmith | amodi: yep, well, should be working then, if the host(s) are in that aggregate.. see the functional test for details :) | 19:10 |
| amodi | mriedem: well im checking functional tests | 19:10 |
| amodi | dansmith: sure | 19:10 |
| sean-k-mooney | dansmith: didnt the look at a small number of host because when they did not limit the got too much data back form placment and it was slowing down schduling | 19:10 |
| *** awalende has joined #openstack-nova | 19:11 | |
| dansmith | sean-k-mooney: okay, yes, they limit placement to a very small number for performance reasons. And then that affects the correctness of the multi-tenant filter. | 19:11 |
| sean-k-mooney | basically the "this tiny vm fits on all host so return all host proablem" | 19:11 |
| sean-k-mooney | ya | 19:12 |
| dansmith | I'm *so* glad we hashed that out. | 19:12 |
| sean-k-mooney | sorry but this is likely to become more important in an edge cloud deployment too | 19:13 |
| sean-k-mooney | or similar issues at anyrate | 19:13 |
| *** cdent has quit IRC | 19:13 | |
| *** mvkr has joined #openstack-nova | 19:13 | |
| dansmith | most edge cloud configs I know of are set up so tenants can boot things in geo-local areas, and thus limiting them to specific aggregates by project isn't as common | 19:14 |
| dansmith | unless maybe you have really big edge sites with overlapping vertical AZs and horizontal non-AZ aggregates or something | 19:15 |
| *** cdent has joined #openstack-nova | 19:15 | |
| sean-k-mooney | i was more thinking of the schduler prefilters in general rather then this one implmenation. as the number of host in singel cloud grows with edge deployment the mechanium you intoduce to solve this problem will like become more useful going forward | 19:16 |
| dansmith | indeed, and hence my desire to do the image type filtering in placement instead of after the fact | 19:18 |
| cdent | prefilters++ | 19:19 |
| mriedem | speaking of, i think i've got a hack for pre-filtering hosts that support multiattach volumes now | 19:21 |
| sean-k-mooney | mriedem: im also working on a spec for prefilting host based on virtual device modesl they support | 19:22 |
| mriedem | let us all work on pre-filters | 19:22 |
| sean-k-mooney | i mean they are the new placement that will solve all proablems that and traits | 19:23 |
| sean-k-mooney | this is what i have so far but ill have a first draft up on gerrit tommorow or monday | 19:24 |
| sean-k-mooney | https://etherpad.openstack.org/p/image-metadata-prefilters | 19:24 |
| dansmith | sean-k-mooney: and is this for places where you've passed a port in and it specifies some host model? | 19:25 |
| dansmith | I would expect the normal "just hook me up" request to not need such filtering | 19:25 |
| *** whoami-rajat has joined #openstack-nova | 19:26 | |
| sean-k-mooney | no this is based on image metadata so if you select an image that says it ned hw_vif_type=e1000 then we should not schdule you to a hyperv host that does not support it | 19:27 |
| dansmith | oh from | 19:27 |
| dansmith | yeah | 19:27 |
| * dansmith just read | 19:27 | |
| *** luksky has joined #openstack-nova | 19:29 | |
| *** cfriesen has joined #openstack-nova | 19:31 | |
| sean-k-mooney | dansmith: i havent spell checked anything so before i submit it as a spec ill be running it true grammerly | 19:33 |
| dansmith | sean-k-mooney: oh I don't think you need to spell check it | 19:34 |
| dansmith | sean-k-mooney: =P | 19:34 |
| sean-k-mooney | :) your right i just need to add stephenfin or jay to the review and wait for the wall of comments | 19:35 |
| dansmith | hah, make gerrit OOM | 19:36 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Pre-filter hosts based on multiattach volume support https://review.openstack.org/645316 | 19:40 |
| mriedem | efried: cdent: dansmith: sean-k-mooney: ^ pretty hacky but it's also pretty easy building on top of the requested_resources stuff gibi added in stein | 19:40 |
| dansmith | eeeeeverybody wants into the prefilter game | 19:41 |
| cdent | some people can follow trends but only dansmith can make them possible | 19:41 |
| mriedem | psh i was talking about this way back in https://review.openstack.org/#/c/538498/ | 19:41 |
| cdent | now that efried is ptl do we have to roll to resolve disputes? | 19:42 |
| mriedem | gonna go for a walk with the mrs while (1) it's nice out and (2) the neighbors and their annoying kids aren't outside mingling | 19:42 |
| *** BjoernT has joined #openstack-nova | 19:45 | |
| *** sridharg has quit IRC | 19:46 | |
| openstackgerrit | Merged openstack/python-novaclient master: Remove deprecated options https://review.openstack.org/642312 | 19:47 |
| * efried perks up | 19:51 | |
| efried | I've long thought the center of the square of tables would make a good "cage". | 19:52 |
| mnaser | should _local_delete() be called with instance.host == None ? | 19:55 |
| mnaser | "instance's host None is down, deleting from database" with "Ignoring volume cleanup failure due to 'NoneType' object has no attribute 'get'" afterwards, i'm assuming that volume clean up has to do with the fact that host is None | 19:56 |
| mnaser | https://github.com/openstack/nova/blob/5ca858eaa72acd0513e27a4c9518980b769f5d6e/nova/compute/api.py#L1967 | 19:57 |
| mnaser | i can only imagine that not possibly being evaluated properly because its hitting the local delete path | 19:57 |
| sean-k-mooney | mnaser: proably not | 19:57 |
| sean-k-mooney | instance.host==None would only happen if the inscate was in cell0 right | 19:57 |
| mnaser | or not scheduled yet | 19:57 |
| mnaser | if not instance.host and not may_have_ports_or_volumes <= the first part evaluates to True (as per the obvious warnings) | 19:58 |
| mnaser | may_have_ports_or_volumes = compute_utils.may_have_ports_or_volumes(instance) is probably evaluating the true (say this is a bfv instance) | 19:58 |
| sean-k-mooney | ya just read the code around it for context | 19:59 |
| mnaser | so it may seem that _local_delete() assumes that a host exists | 20:00 |
| sean-k-mooney | i might be blind but wher is _local_delete() called in that function | 20:01 |
| mnaser | https://github.com/openstack/nova/blob/5ca858eaa72acd0513e27a4c9518980b769f5d6e/nova/compute/api.py#L2128-L2129 | 20:02 |
| mnaser | the NoneType get happens inside here somewhere https://github.com/openstack/nova/blob/5ca858eaa72acd0513e27a4c9518980b769f5d6e/nova/compute/api.py#L2174-L2199 | 20:02 |
| sean-k-mooney | ah that funciton is a lot longer then i tought it was | 20:02 |
| sean-k-mooney | proably form compute_utils.get_stashed_volume_connector | 20:05 |
| mnaser | i think so | 20:06 |
| sean-k-mooney | yep https://github.com/openstack/nova/blob/a5e3054e1d6df248fc4c00b9abd7289dde160393/nova/compute/utils.py#L1262 | 20:06 |
| mnaser | will LOG.error() print out the traceback or whats the best way to do it | 20:06 |
| mnaser | sean-k-mooney: yeah but there is 'if connector:' before that | 20:06 |
| mnaser | which if connector is None then it should evaluate to false right off the bat) | 20:07 |
| sean-k-mooney | i think there is a log.exception() prints a nice traceback | 20:07 |
| mnaser | connector = jsonutils.loads(bdm.connection_info).get('connector') | 20:11 |
| mnaser | AttributeError: 'NoneType' object has no attribute 'get' | 20:11 |
| mnaser | the interesting thing is there is an if statement `bdm.connection_info is not None` | 20:12 |
| sean-k-mooney | ok so jsonutils.loads(bdm.connection_info) did not produce a dictionary | 20:12 |
| mnaser | yeah, i will log it's value | 20:12 |
| sean-k-mooney | it could be the empty sting | 20:12 |
| sean-k-mooney | *string | 20:12 |
| mnaser | LOG.debug(bdm.connection_info) => null ? | 20:13 |
| sean-k-mooney | null | 20:14 |
| sean-k-mooney | as in that is what it litally printed | 20:14 |
| mnaser | and then json parsing null gives... None | 20:14 |
| mnaser | so bdm.connection_info is the literal value 'null' | 20:14 |
| sean-k-mooney | ya that would make sense | 20:14 |
| mnaser | weird | 20:14 |
| sean-k-mooney | well i dont know why bdm.connection_info is null | 20:14 |
| mnaser | well i see it in the db equal to null too hmm | 20:15 |
| sean-k-mooney | well the litral null is a valid json type | 20:16 |
| sean-k-mooney | https://www.json.org/ | 20:16 |
| mnaser | hmm | 20:17 |
| sean-k-mooney | so id the db colume is null becuae no conection info exists for this incance becasue it has never been schduled then it makes sense that it woudl be null when we parse it there | 20:17 |
| mnaser | Quota exceeded for 86bbbcfa8ad043109d2d7af530225c72, tried to create 80G volume (6080G of 6144G already consumed).: OverQuota: Quota exceeded for resources: ['gigabytes'] | 20:17 |
| mnaser | so im wondering if when it tries to create bfv vol, it fails, instance tries to delete, poof | 20:18 |
| sean-k-mooney | ya that sound plasible | 20:18 |
| sean-k-mooney | we could jsut add a none check there | 20:18 |
| sean-k-mooney | so store teh value of jsonutils.loads(bdm.connection_info) then do a "if connection_info is not None: ... | 20:19 |
| mnaser | does nova get connection_info right away or does it poll till it gets it | 20:19 |
| mnaser | im assuming it polls | 20:19 |
| *** awaugama has quit IRC | 20:20 | |
| mnaser | looks like we do json loading often so it would be good for us to store {} instead of null i guess | 20:20 |
| sean-k-mooney | i wont have connection info untill after it has schduled to a host | 20:20 |
| sean-k-mooney | we proably dont store {} to avoid updating all old instances when we first load them | 20:21 |
| sean-k-mooney | in anycase we proably should not be doing random json loads in the code like that | 20:22 |
| mnaser | its happening a tons tho :X | 20:22 |
| sean-k-mooney | sure but that does not mean its a good thing :) | 20:23 |
| mnaser | ++ yeah i agree | 20:24 |
| *** cdent has quit IRC | 20:24 | |
| mnaser | https://github.com/openstack/nova/blob/6efa3861a5a829ba5883ff191e2552b063028bb0/nova/volume/cinder.py#L575-L605 | 20:24 |
| edmondsw | I don't think openstack-tox-lower-constraints is working properly... I just had to put up a fix for nova-powervm that would appear to be needed for nova as well based on the lower-constraints.txt, but that check is passing... | 20:25 |
| sean-k-mooney | if | 20:25 |
| mnaser | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L677-L687 | 20:25 |
| edmondsw | and it looks like it's passing because the job is actually using a newer version of psycopg2 (2.7.7) than what the lc txt file indicates (2.6.2) | 20:26 |
| sean-k-mooney | edmondsw: cdent fixed some things in lower contratins a week or two ago | 20:26 |
| edmondsw | here's the nova-powervm fix for reference: https://review.openstack.org/#/c/645330 | 20:26 |
| sean-k-mooney | edmondsw: it may be using the system installed version | 20:26 |
| edmondsw | whatever it's doing, it's not right | 20:26 |
| mnaser | sean-k-mooney: from oslo_serialization import jsonutils; jsonutils.dumps(None) => 'null' | 20:27 |
| efried | edmondsw: This may be a better question for #openstack-requirements, but I think whatever you have in l-c is not necessarily what the lower-constraints job will install, as it has to take the highest-low from all the projects it's installing. | 20:27 |
| sean-k-mooney | mnaser: ya mapping None to 'null' is what i would expect | 20:27 |
| mnaser | so what we should do is make sure that we do something like | 20:28 |
| mnaser | self.get('connection_info', {}) instead | 20:28 |
| efried | edmondsw: iow if nova-powervm has l-c for foo==1.5 and bar==X and bar at X has l-c for foo==1.7, you're gonna get 1.7 | 20:28 |
| edmondsw | efried this is just an FYI... do with it what you will | 20:28 |
| sean-k-mooney | edmondsw: if the issue is the one i think it is related to postgree it could also be donw to the desto the lower constratits job is using | 20:30 |
| edmondsw | and whatever you do, you may want to bump to at least 2.7 for psycopg2 in nova's lc file, since 2.6.2 isn't going to work if it ever was to get used | 20:30 |
| sean-k-mooney | edmondsw: well it will if you are not useing postgres | 20:31 |
| edmondsw | if you're not using postgres then you don't need psycopg2 :) | 20:31 |
| sean-k-mooney | well that was basically my point | 20:31 |
| edmondsw | but anyway... this is just a public service announcement | 20:32 |
| sean-k-mooney | im not sure we can raise our lower constrants at this point can we? | 20:32 |
| *** igordc has quit IRC | 20:32 | |
| *** mrch_ has joined #openstack-nova | 20:32 | |
| efried | for stein? It would take an emergency. | 20:32 |
| sean-k-mooney | edmondsw: we have see this issue downstream by the way too but we just used 2.7 instead | 20:33 |
| mnaser | https://bugs.launchpad.net/nova/+bug/1821244 im gonna work on this now. it should be a simple fix. | 20:33 |
| openstack | Launchpad bug 1821244 in OpenStack Compute (nova) "Failed volume creation can result in invalid `connection_info` field" [Undecided,New] | 20:33 |
| edmondsw | sean-k-mooney negating the purpose of lower-constraints | 20:33 |
| sean-k-mooney | edmondsw: well this show up in older release too | 20:34 |
| sean-k-mooney | the root casue was the presence of postgres 10.X that broke the compatiblity | 20:34 |
| sean-k-mooney | if you use an older postgres it also works | 20:35 |
| sean-k-mooney | but ya we should bump it but tommorow | 20:35 |
| *** tssurya has quit IRC | 20:38 | |
| sean-k-mooney | efried: the lower constratins job should use nova lower constratin as the upper constratin | 20:38 |
| efried | sean-k-mooney: even if it installs some other dep that has a higher lower constraint? That doesn't make sense. | 20:39 |
| sean-k-mooney | pip transitive depencies via requirement.txt will not look at there upper or lower constratints | 20:39 |
| sean-k-mooney | efried: it will fail to install if our constratin file say 2.0.0 and they request >= 3.0.0 | 20:40 |
| sean-k-mooney | upper/lower constratins.txt is an openstack convention so pip/setuptools does not know about them | 20:41 |
| sean-k-mooney | pip just looks ate the requiremets.txt files | 20:42 |
| efried | sure, but I would expect the l-c job to know about them. | 20:42 |
| efried | ...about the lower constraints for the projects that have lower constraints. | 20:42 |
| sean-k-mooney | the lower constratin job does pip install -r requriements.txt -c lower-constratints.txt | 20:43 |
| sean-k-mooney | only one constratint file. nova is passed when installing the deps | 20:43 |
| efried | mm, okay, then yeah, I guess it should only be observing my own l-c huh | 20:43 |
| * efried shrugs | 20:44 | |
| sean-k-mooney | for what its worth the job does look broken | 20:46 |
| sean-k-mooney | http://logs.openstack.org/46/621646/20/check/openstack-tox-lower-constraints/8b49dc9/tox/lower-constraints-1.log | 20:46 |
| sean-k-mooney | we are passing bot an upper-constratins.txt and lower-constraints.txt to pip | 20:46 |
| sean-k-mooney | ok i see what going on | 20:48 |
| sean-k-mooney | ill see if i can fix that | 20:48 |
| sean-k-mooney | our install command include upper-constratints | 20:49 |
| *** mdbooth_ has joined #openstack-nova | 20:49 | |
| efried | sean-k-mooney: you mean L18 of tox.ini? | 20:51 |
| sean-k-mooney | yes | 20:51 |
| sean-k-mooney | im overriding hte install_command local to not do that in the lower-constraints env and ill see what hapens | 20:52 |
| sean-k-mooney | i can see it endup using 2.7.7 in the gate | 20:52 |
| sean-k-mooney | it should have been using 2.6.8 or whatever | 20:52 |
| *** mdbooth has quit IRC | 20:52 | |
| sean-k-mooney | well that failed instantly | 20:53 |
| efried | so there's nothing tox-magical about the "lower-constraints" name like there is for pyXX that would be smart enough to use a fresh install_command? | 20:53 |
| efried | failed instantly because a l-c is too low? | 20:53 |
| sean-k-mooney | no lower-constraints is an openstack thing | 20:53 |
| efried | okay. | 20:53 |
| sean-k-mooney | it faild on the postgressql issue | 20:54 |
| efried | so we've been borked for a while and real l-cs have been creeping past us. | 20:54 |
| efried | edmondsw: Thanks for the find there! | 20:54 |
| sean-k-mooney | ya for basicly ever i think looking at the git blame | 20:54 |
| efried | well, we've done quite a bit to tox.ini in the last release or two. | 20:54 |
| sean-k-mooney | the install command predate adding lower constratints i think | 20:54 |
| efried | ...but not sure if those have been touched. | 20:55 |
| sean-k-mooney | ill bump the lower constatin to 2.7 and push a patch shortly | 20:55 |
| efried | sean-k-mooney: Well, that might not be the only transgressor | 20:55 |
| efried | but I guess you'll find out :) | 20:55 |
| sean-k-mooney | i was trying not to say that out loud but yes | 20:55 |
| sean-k-mooney | i should proably file a bug for this. ya the next issue is the zvm sdk | 20:57 |
| sean-k-mooney | i think i fixed this error actully it complingin about the python version support | 20:57 |
| sean-k-mooney | i shoudl proably file a bug for this. shoudl this be a RC2 thing | 20:58 |
| openstackgerrit | Merged openstack/nova-specs master: Re-propose count quota usage from placement https://review.openstack.org/645302 | 21:00 |
| mnaser | im struggling to trace the path of failure but i have a fix for this :\ | 21:04 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: wip: bdm: store empty object as connection_info by default https://review.openstack.org/645352 | 21:07 |
| mnaser | if someone has a few cycles to spare, i'm going to keep digging but i don't know the code that well to know why it's being saved with 'null' | 21:07 |
| sean-k-mooney | mnaser: if the value is ever set to None it will be serialised to 'null' | 21:08 |
| sean-k-mooney | i can try and take a look at it in a bit but i also dont know the bdm code that well | 21:08 |
| mnaser | sean-k-mooney: yeah.. but i am trying to figure out why it would either be a) set to None or b) .get('connection_info') fails and returns None | 21:08 |
| mnaser | oh interesting | 21:09 |
| sean-k-mooney | if its trying to delete it before the block device mappings are constructed then that might be the issue | 21:09 |
| mnaser | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L284-L288 | 21:10 |
| mnaser | by default, the connection_info field defaults to 'NULL' | 21:10 |
| mnaser | so that means self['connection_info'] = None .. | 21:10 |
| mriedem | isn't the instance.host set before that though? | 21:10 |
| mriedem | the instance_claim sets the instance.host | 21:10 |
| sean-k-mooney | ya which just porpagates that default | 21:10 |
| mriedem | so you shouldn't get a local delete flow in the API | 21:10 |
| mnaser | mriedem: this is a case where instance.host = None so i think what happens is that the build aborts a few times on the compute host | 21:11 |
| mnaser | and then when it fails to build it, instance.host goes to None | 21:11 |
| mnaser | because cinder blows up when it tries to create a volume and you're out of quota | 21:11 |
| mnaser | this isnt a case of instance.host = None because we're in cell0, it's because we failed to build the instance N times and gave up | 21:11 |
| mriedem | anything failing in _prep_block_device automatically aborts the build | 21:12 |
| mriedem | there is no reschedule | 21:12 |
| mriedem | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/manager.py#L2317 | 21:12 |
| mriedem | especially if you hit overquota | 21:12 |
| mnaser | yeah im def hitting overquota because of the error logs | 21:12 |
| mnaser | "Failed to create block device for instance due to exceeding volume related resource quota. Error: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 80G, quota is 6144G and 6080G has been consumed.: OverQuota: VolumeSizeExceedsAvailableQuota: Requested volume or snapshot exceeds allowed gigabytes quota. Requested 80G, quota is 6144G and 6080G has been consumed." | 21:13 |
| mnaser | so that code is getting exercised | 21:13 |
| mriedem | that should reset the instance.host to None here https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/manager.py#L2018 | 21:13 |
| mriedem | this is booting a volume from an image yeah? | 21:13 |
| mnaser | which means a follow up delete will result in a local delete path? | 21:13 |
| mnaser | yeah | 21:13 |
| mriedem | so it blows here https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L741 | 21:14 |
| mnaser | yup, and the block device connection_info is set to null at this point | 21:14 |
| mnaser | note if i quickly hit the db, i see it flap from NULL (actual SQL null, when a new record is created) to null (json-ified null, when the save happens, with a missing connection_info) | 21:15 |
| mriedem | probably b/c of this https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L681 | 21:15 |
| mriedem | if connection_info_string != self._bdm_obj.connection_info: | 21:15 |
| mriedem | if '' != None: | 21:15 |
| mnaser | mriedem: more accurately if 'null' != None afaik | 21:16 |
| mriedem | oh right yeah | 21:16 |
| mnaser | hence i think this will fix it https://review.openstack.org/#/c/645352/ | 21:16 |
| mriedem | ok and then this blows up right? https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/utils.py#L1096 | 21:16 |
| mriedem | i mean, that is true, | 21:16 |
| mriedem | then jsonutils.loads(bdm.connection_info).get('connector') | 21:16 |
| mriedem | blows up | 21:16 |
| mnaser | yeah | 21:17 |
| mnaser | that's as far as i got, but i don't know what triggers a save() which pushes the json stuff in the db in the failing path | 21:17 |
| mriedem | that code is rife with tricky decorators that save changes on methods | 21:18 |
| mriedem | so could be one of those | 21:18 |
| mnaser | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L645 | 21:18 |
| mnaser | so if `refresh_connection_info` is called at any given moment | 21:18 |
| mnaser | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L648-L650 | 21:18 |
| mnaser | will return | 21:18 |
| mnaser | and then it will call the decorator to save | 21:18 |
| mnaser | self['connection_info'] by default is None | 21:19 |
| mriedem | i don't think refresh_connection_info would be getting called during spawn | 21:19 |
| mnaser | the only place i noticed calls it is refresh_conn_infos | 21:19 |
| mriedem | but yeah that update_db decorator is what could be doing it | 21:19 |
| sean-k-mooney | efried: .. we pin our lower constating for warlock to one explcitly blacklisted by python-glance client... | 21:20 |
| mnaser | ah | 21:20 |
| sean-k-mooney | the fun continues | 21:20 |
| mriedem | mnaser: i think that's only called here https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/manager.py#L1779 | 21:20 |
| mriedem | on like move ops and such | 21:20 |
| mnaser | indeed thats the only reference to it that i find | 21:20 |
| efried | sean-k-mooney: Presumably it doesn't hurt to elevate the warlock l-c to... whatever glanclient is using | 21:21 |
| sean-k-mooney | it is useing an older version and there is no newer version | 21:21 |
| mnaser | ah shit | 21:21 |
| sean-k-mooney | we pin to 1.3.0 which was relwased in 2016 | 21:21 |
| sean-k-mooney | they use 1.2.0 | 21:22 |
| mriedem | mnaser: which release is this? | 21:22 |
| mnaser | rocky | 21:22 |
| mriedem | it could be somewhere in here that the changes to connection_info get saved https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/compute/manager.py#L2554 | 21:23 |
| sean-k-mooney | efried: im goint to try lowerering it | 21:23 |
| sean-k-mooney | the constratin we have is from when we intoduced lower-constratits | 21:23 |
| sean-k-mooney | so im not sure its actully the min version we need | 21:23 |
| mnaser | terminate_instance seems to get bdms | 21:24 |
| mriedem | mnaser: you wouldn't get there when the server is deleted, | 21:24 |
| mriedem | because the instance.host is None | 21:24 |
| mnaser | yeah | 21:24 |
| sean-k-mooney | efried: actully looking at requirements.txt we dont actully depend on it directly | 21:24 |
| efried | Not surprising. | 21:25 |
| *** SASAA has quit IRC | 21:27 | |
| mriedem | mnaser: anyway, it's probably not worth trying to figure out where in that pile of spaghetting the bdm.save is getting called, | 21:28 |
| mriedem | we shouldn't save 'null' in the connection_info | 21:28 |
| mriedem | so your change is probably sane on it's own | 21:28 |
| mnaser | mriedem: yeah.. i was just trying to find a way to have some sort of tests but yeah | 21:29 |
| mnaser | also i just realized that wouldn't even fix it | 21:29 |
| mriedem | get_stashed_volume_connector is very old so i'm kind of surprised someone is just hitting this now | 21:29 |
| mnaser | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L284-L288 | 21:29 |
| mnaser | where i thought we were getting 'None' because .get("connection_info") was returning an empty field, it was actually set, but to 'null'.. so yeah | 21:30 |
| mriedem | i'm not sure what that has to do with it - that's code that runs before things blow up | 21:30 |
| mnaser | the assumption in my fix that self['connection_info'] doesn't exist and the fact we get None is because there is no 'connection_info' key | 21:30 |
| mriedem | _transform is called to convert the BlockDeviceMapping object into a DriverVolImageBlockDevice object before the compute manager calls attach() on it | 21:31 |
| mriedem | from _prep_block_device | 21:31 |
| mriedem | it's extremely confusing | 21:31 |
| mnaser | but it seems that 'connection_info' key always exists, but it can be set to None. so the 'None' value comes from the fact that it's *actually* equal to 'None' | 21:31 |
| mriedem | loads(None) should raise the TypeError though | 21:31 |
| *** pcaruana has quit IRC | 21:33 | |
| mnaser | ok so the ideal fix would be | 21:33 |
| mnaser | https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L288 change that default to {} | 21:33 |
| mnaser | so if its null, it blows up and returns {}, if its an actual object, we keep it | 21:33 |
| sean-k-mooney | dumps(None) should return 'null' loads('null) shoudl return None | 21:33 |
| sean-k-mooney | loads(None) shoudl be a type error yes | 21:34 |
| mriedem | _validate_existing_attachment_ids | 21:34 |
| mriedem | oops | 21:34 |
| mriedem | mnaser: connection_info can be None, and get_stashed_volume_connector handles it, so i'm not following you | 21:34 |
| mnaser | mriedem: but if update_db happened, connection_info now is a literally 'null' (the string) | 21:34 |
| mriedem | i though https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L681 was the problem | 21:34 |
| mnaser | and json loads converts 'null' to 'None' | 21:35 |
| mriedem | but we can avoid the 'null' by changing https://github.com/openstack/nova/blob/67ac57b3258d4d7476fa544b3dfef7296e220ad2/nova/virt/block_device.py#L682 to default to {} yeah? | 21:35 |
| sean-k-mooney | mnaser: it convets the sting 'null' to the object None | 21:35 |
| mnaser | yes | 21:35 |
| mriedem | so you'd get '{}' | 21:35 |
| mnaser | yep, and that would fix it | 21:35 |
| mriedem | right, hence me saying "anyway, it's probably not worth trying to figure out where in that pile of spaghetting the bdm.save is getting called" | 21:36 |
| mriedem | "we shouldn't save 'null' in the connection_info" | 21:36 |
| sean-k-mooney | {} is an empty json object which is technically not the same thing but it will fix this case | 21:36 |
| mnaser | mriedem: ah because my original 'fix' didn't actually fix it -- https://review.openstack.org/#/c/645352/1/nova/virt/block_device.py | 21:36 |
| mnaser | so i was referring to that, but, i will fix that now | 21:36 |
| mnaser | and will try to have some unit test for _transform | 21:36 |
| mnaser | to make sure it doesn't output connection_info = None | 21:37 |
| mriedem | i would think this could be easily reproducible if we just hit overquota when booting from volume and then try to delete the server | 21:37 |
| mnaser | probably, but not sure if i can write that level of functional tests from my skillset so far | 21:38 |
| mnaser | unless you wanna a similar functional test that i can read and implement similar to | 21:38 |
| mriedem | i could write one | 21:38 |
| mnaser | sure, let me push up this with a unit test too | 21:38 |
| *** BjoernT has quit IRC | 21:40 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Pre-filter hosts based on multiattach volume support https://review.openstack.org/645316 | 21:41 |
| mriedem | efried: i went through and marked the deferred train blueprints that dont have specs re-approved as pending approval https://blueprints.launchpad.net/nova/train | 21:44 |
| mriedem | i forgot to do that when deferring from stein | 21:44 |
| mriedem | so we re-approve in LP once the spec is re-approved | 21:44 |
| *** idlemind has joined #openstack-nova | 21:44 | |
| efried | mriedem: ack, thank you. I can see you saw I approved the quota one. | 21:44 |
| mriedem | yeah, i thought about it because you said on the review: | 21:45 |
| mriedem | "Blueprint in approved state [3] ✔" | 21:45 |
| mriedem | technically the blueprint shouldn't have been approved before the spec | 21:45 |
| mriedem | but i forgot to change those when deferring | 21:45 |
| *** whoami-rajat has quit IRC | 21:45 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Trivial: remove unused var from policies.base.py https://review.openstack.org/645380 | 21:47 |
| mriedem | dansmith: i've got this pretty much done, without using a pre-request filter https://review.openstack.org/#/c/645316/ but not sure if i need a blueprint or not, since i'm not adding a configurable filter, it's just going to always be required if you're booting from volume | 21:49 |
| mriedem | cuz it's kind of dumb to even try to boot with a multiattach volume if the scheduler isn't going to filter out hosts that won't support them | 21:50 |
| efried | does that mean they would break further on down the line? | 21:51 |
| efried | I mean, would initial spawn succeed or fail? | 21:51 |
| mriedem | it would fail, | 21:51 |
| mriedem | it's arguably a bug fix | 21:51 |
| mriedem | the bare minimum we do today is check if the computes are new enough, but that was minimum of queens | 21:51 |
| mriedem | that doesn't say bfv wo'nt work for vmware for exapmle | 21:52 |
| mriedem | if you're using vmware exclusively, you'd just have to never enable multiattach volumes on the cinder side via policy | 21:52 |
| mriedem | that's a pain if you're doing multi-hypervisor though | 21:52 |
| mriedem | you'd have to setup aggregates and filter using some flavor extra spec or something | 21:52 |
| *** BjoernT has joined #openstack-nova | 21:54 | |
| *** xek_ has quit IRC | 21:55 | |
| mriedem | anywho, idk, it's janky, i suppose you can't reasonably do bfv with vmware and libvirt mixed anyway because you have different volume types, and the libvirt driver isn't going to support vmware volumes and vice versa, so i'm not sure what if anything people do for mixed hypervisor - definitely at least aggregates/AZs | 21:55 |
| sean-k-mooney | efried: ... this change https://github.com/openstack/nova/commit/6b844af57eebda6317b1ed3bf5a2f85dcba0bc13 does not work after i fix the lower constrating so im going to have to change it | 21:56 |
| efried | mriedem: I say no bp, bug good, reno optional | 21:57 |
| efried | sean-k-mooney: I'm on board with that. Long as you don't have to go changing the prod code, I say do what you gotta do to get this fixed. | 21:58 |
| sean-k-mooney | im actully not sure why this is even working upstream honestly | 21:59 |
| sean-k-mooney | i think it is depending on something having previously loaded messaging._drivers.impl_fake | 22:00 |
| sean-k-mooney | *imported | 22:00 |
| sean-k-mooney | ya it is | 22:02 |
| sean-k-mooney | >>> import oslo_messaging as messaging | 22:02 |
| sean-k-mooney | >>> messaging._drivers.impl_fake.FakeExchangeManager._exchanges | 22:02 |
| sean-k-mooney | Traceback (most recent call last): | 22:02 |
| sean-k-mooney | File "<stdin>", line 1, in <module> | 22:02 |
| sean-k-mooney | AttributeError: 'module' object has no attribute 'impl_fake' | 22:02 |
| sean-k-mooney | >>> import oslo_messaging._drivers.impl_fake | 22:02 |
| sean-k-mooney | >>> messaging._drivers.impl_fake.FakeExchangeManager._exchanges | 22:02 |
| sean-k-mooney | {} | 22:02 |
| sean-k-mooney | so there is a race based on the execution order of the tests | 22:03 |
| *** yankcrime has quit IRC | 22:04 | |
| *** irclogbot_0 has quit IRC | 22:05 | |
| *** mgoddard has quit IRC | 22:08 | |
| *** marst has quit IRC | 22:09 | |
| *** mgoddard has joined #openstack-nova | 22:09 | |
| *** ivve has joined #openstack-nova | 22:11 | |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for cyborg access https://review.openstack.org/631242 | 22:16 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Add Cyborg device profile groups to request spec. https://review.openstack.org/631243 | 22:16 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Create and bind Cyborg ARQs. https://review.openstack.org/631244 | 22:16 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: WIP: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.openstack.org/631245 | 22:16 |
| *** rcernin has joined #openstack-nova | 22:19 | |
| *** BjoernT has quit IRC | 22:24 | |
| dansmith | mriedem: why is that not in request_filter.py, just without a conf flag? | 22:26 |
| dansmith | that way all these "check a thing about the request and add a trait requirement" things are in one place | 22:27 |
| sean-k-mooney | efried: so just as an fyi we have other dodgy test too that are wrong upstream :( | 22:27 |
| efried | sean-k-mooney: I am not surprised at all. | 22:27 |
| efried | sean-k-mooney: I wonder if we need to file a bug for this. | 22:27 |
| sean-k-mooney | oh we do | 22:27 |
| sean-k-mooney | but that is not what im concerned about | 22:28 |
| sean-k-mooney | the test issue im finding shoudl be broken in other jobs | 22:28 |
| sean-k-mooney | https://github.com/openstack/nova/blob/d2dc7549ce9a0c979fd319cf0332a861b9cd2000/nova/tests/unit/api/openstack/compute/test_serversV21.py#L5129 | 22:28 |
| sean-k-mooney | self.stub_out('uuid.uuid4', lambda: FAKE_UUID) | 22:28 |
| sean-k-mooney | that stubs out uuid.uuid4 to return FAKE_UUID | 22:29 |
| sean-k-mooney | FAKE_UUID is a string | 22:29 |
| sean-k-mooney | oslo messaging does unique_id = uuid.uuid4().hex' | 22:29 |
| sean-k-mooney | .hex is not defiend on a string | 22:29 |
| sean-k-mooney | so it explodes | 22:29 |
| efried | why isn't it exploding now? | 22:30 |
| sean-k-mooney | oslo.messaging still does unique_id = uuid.uuid4().hex on master | 22:30 |
| sean-k-mooney | i think its another race in the tests | 22:31 |
| *** erlon has joined #openstack-nova | 22:32 | |
| sean-k-mooney | hum running that test on its own on the py36 target passes but files in lower constratits | 22:32 |
| efried | are you running stop-on-first-failure or do you have a sense of how many tests are broken? | 22:34 |
| sean-k-mooney | ok so my theory is that thing have moved and we are not mocking the right things when usign the older version of oslo.messaging | 22:34 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: bdm: store empty object as connection_info by default https://review.openstack.org/645352 | 22:34 |
| sean-k-mooney | i was running all the test but while i was getting many of the test to pass the lower-constratits job was hannging after running for a bit | 22:35 |
| mnaser | mriedem: ^ this worked locally running the unit tests | 22:36 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: bdm: store empty object as connection_info by default https://review.openstack.org/645352 | 22:37 |
| cfriesen | mriedem: I assume the vTPM blueprint status change is because it didn't make it in for Stein? | 22:38 |
| *** phasespace has joined #openstack-nova | 22:39 | |
| sean-k-mooney | cfriesen: mriedem mentioned he set the state back to pending approval for any blupritn that was approved last cycle but needs to have the spec reapproved | 22:39 |
| cfriesen | thx, figured as much | 22:39 |
| mriedem | cfriesen: right, re-propose the spec for train | 22:40 |
| mriedem | i forgot to switch the bp status when deferring | 22:40 |
| mriedem | mnaser: want to report a bug for this? | 22:40 |
| mnaser | mriedem: oh dang i had one but i didnt add closes-bug | 22:45 |
| openstackgerrit | Mohammed Naser proposed openstack/nova master: bdm: store empty object as connection_info by default https://review.openstack.org/645352 | 22:45 |
| openstackgerrit | sean mooney proposed openstack/nova master: make lower-constratints env use lower-constratins https://review.openstack.org/645392 | 22:46 |
| mnaser | ok just noticed your reviews too ill try to address them after foodz | 22:46 |
| sean-k-mooney | im going to keep working on ^ | 22:46 |
| efried | sean-k-mooney: Thanks for doing that. | 22:46 |
| mriedem | mnaser: ok i left some comments in PS3, not really worth -1ing over though | 22:47 |
| sean-k-mooney | efried: i can fix the FAKE_UUID issue but i still suspect that our mocking is wrong in some of the tests and code that should not be run actully is being run | 22:48 |
| mriedem | dansmith: it's not a request filter because the request spec doesn't store the bdms, and during server create the bdms won't have multiattach information in them anyway, so to figure it out, i'd have to, within the filter, pull the list of bdms for the instance and for all with a volume_id, query cinder to figure out if the volume is multiattach or not | 22:48 |
| sean-k-mooney | if the lower constraitn job explodes or times out then that is likely why | 22:48 |
| mriedem | if we actually had a BDM.multiattach field, that could maybe work - but we'd still have to get the bdms during scheduling...which might suck | 22:49 |
| *** mriedem is now known as mriedem_afk | 22:50 | |
| dansmith | mriedem_afk: okay, seemed like it was just examining the reqspec to determine what to do, which is exactly like the other filters, but I admittedly was skimming | 22:56 |
| dansmith | can look again tomorrow | 22:56 |
| *** tkajinam has joined #openstack-nova | 22:56 | |
| *** hongbin has quit IRC | 22:57 | |
| *** tosky has quit IRC | 22:59 | |
| *** igordc has joined #openstack-nova | 23:04 | |
| *** ivve has quit IRC | 23:27 | |
| *** awalende has quit IRC | 23:30 | |
| *** luksky has quit IRC | 23:32 | |
| sean-k-mooney | mriedem_afk: dansmith melwitt i have been talking to erric for the last hour or two about the fact our lower constrats job has been broken since we created it | 23:32 |
| sean-k-mooney | im assuming we are not going to fix this in RC1 i can likely have it fixed tomorow or early next week once i figure out where we are not mocking the rpc bus in some test but is this something we would fix in RC2 or in train? | 23:34 |
| sean-k-mooney | i assume we would want to fix lower constraitn on the stable branches too? | 23:34 |
| sean-k-mooney | https://review.openstack.org/#/c/645392/1 | 23:34 |
| melwitt | sean-k-mooney: I think you'll want to open a bug for that, yeah? | 23:36 |
| sean-k-mooney | ya i have left a todo in the review for that | 23:37 |
| melwitt | I'm not sure this is an RC thing because it's our test job not working, not a user deliverable? | 23:37 |
| sean-k-mooney | but ill do it when i start on this again tomorow | 23:37 |
| sean-k-mooney | true but the minium requirement we had listed were wrong | 23:37 |
| melwitt | and then yes, I think we'd want to fix it on stable too. probably not pike since it's bound for EM in the next few weeks | 23:38 |
| melwitt | sean-k-mooney: oh, I see. yeah, so maybe it would be RC | 23:38 |
| sean-k-mooney | lower-constrtis has only been a thing since queens i think so its only the last 2 release + stien | 23:39 |
| melwitt | RC2, that is. I'll add it to the etherpad | 23:39 |
| sean-k-mooney | ya its not rc1 in anycase. but cool ill keep workign on this again tomorow | 23:40 |
| *** wolverineav has joined #openstack-nova | 23:40 | |
| melwitt | sean-k-mooney: sounds good, thank you | 23:42 |
| *** wolverineav has quit IRC | 23:45 | |
| sean-k-mooney | crap.. this same issue is in a bunch of repos | 23:46 |
| sean-k-mooney | http://codesearch.openstack.org/?q=install_command%20%3D%20pip%20install%20-c%7Benv%3AUPPER_CONSTRAINTS_FILE%3A&i=nope&files=&repos= | 23:46 |
| sean-k-mooney | i should also send a mail to the list about this tomorow | 23:46 |
| *** lbragstad has quit IRC | 23:46 | |
| melwitt | sean-k-mooney: ++ good idea | 23:50 |
| sean-k-mooney | does this fall under the remite or the requireemtn team, qa team or relase team. | 23:50 |
| sean-k-mooney | i think its the same people more or less on all of them but ill chat to them on irc too | 23:51 |
| melwitt | sean-k-mooney: I would think #openstack-requirements | 23:52 |
| sean-k-mooney | ya so apparently lower-constraints is not actully requried by the PTI which explaing why many of the proejct with the prolematic install command have not seen this issue as they are not testing with lower constratints | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!