| *** rcernin_ has joined #openstack-nova | 00:05 | |
| *** bnemec has quit IRC | 00:05 | |
| *** bnemec has joined #openstack-nova | 00:06 | |
| *** rcernin has quit IRC | 00:09 | |
| *** takashin has joined #openstack-nova | 00:10 | |
| *** ociuhandu has joined #openstack-nova | 00:10 | |
| *** bnemec has quit IRC | 00:14 | |
| *** ociuhandu has quit IRC | 00:21 | |
| *** gyee has quit IRC | 00:33 | |
| *** macz has quit IRC | 00:36 | |
| *** slaweq__ has joined #openstack-nova | 00:40 | |
| *** slaweq has joined #openstack-nova | 00:43 | |
| *** slaweq__ has quit IRC | 00:45 | |
| *** TxGirlGeek has quit IRC | 00:45 | |
| *** JamesBenson has joined #openstack-nova | 00:46 | |
| *** ricolin has joined #openstack-nova | 00:47 | |
| *** Liang__ has joined #openstack-nova | 00:48 | |
| *** slaweq_ has joined #openstack-nova | 00:49 | |
| *** slaweq has quit IRC | 00:50 | |
| *** JamesBenson has quit IRC | 00:50 | |
| *** Liang__ has quit IRC | 00:54 | |
| *** ociuhandu has joined #openstack-nova | 00:54 | |
| *** ricolin has quit IRC | 00:56 | |
| *** slaweq_ has quit IRC | 01:06 | |
| *** slaweq_ has joined #openstack-nova | 01:08 | |
| *** bnemec has joined #openstack-nova | 01:13 | |
| *** ociuhandu has quit IRC | 01:13 | |
| *** slaweq_ has quit IRC | 01:20 | |
| *** slaweq has joined #openstack-nova | 01:20 | |
| *** slaweq has quit IRC | 01:20 | |
| *** slaweq has joined #openstack-nova | 01:21 | |
| *** slaweq has quit IRC | 01:22 | |
| *** slaweq has joined #openstack-nova | 01:22 | |
| *** ricolin has joined #openstack-nova | 01:30 | |
| *** ileixe has quit IRC | 01:32 | |
| *** maciejjozefczyk has joined #openstack-nova | 01:33 | |
| *** ileixe has joined #openstack-nova | 01:33 | |
| *** nanzha has joined #openstack-nova | 01:37 | |
| *** ociuhandu has joined #openstack-nova | 01:40 | |
| *** openstackstatus has joined #openstack-nova | 01:44 | |
| *** ChanServ sets mode: +v openstackstatus | 01:44 | |
| *** ociuhandu has quit IRC | 01:45 | |
| *** chason has joined #openstack-nova | 01:47 | |
| *** ociuhandu has joined #openstack-nova | 01:48 | |
| *** chason has quit IRC | 01:49 | |
| *** ociuhandu has quit IRC | 01:52 | |
| *** ociuhandu has joined #openstack-nova | 01:54 | |
| *** aarents has joined #openstack-nova | 02:00 | |
| *** abaindur has joined #openstack-nova | 02:01 | |
| *** ociuhandu has quit IRC | 02:02 | |
| *** maciejjozefczyk has quit IRC | 02:06 | |
| *** maciejjozefczyk has joined #openstack-nova | 02:07 | |
| *** mkrai has joined #openstack-nova | 02:10 | |
| *** ricolin has quit IRC | 02:19 | |
| *** liuyulong has joined #openstack-nova | 02:20 | |
| *** tkajinam has joined #openstack-nova | 02:24 | |
| openstackgerrit | Archit Modi proposed openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | 02:37 |
|---|---|---|
| *** bnemec has quit IRC | 02:38 | |
| gibi_ptg | the nova ptg table: | 02:40 |
| gibi_ptg | http://vogon.no-ip.org/gibizer/ptg/signal-attachment-2019-11-06-103414_001.jpeg | 02:40 |
| gibi_ptg | http://vogon.no-ip.org/gibizer/ptg/signal-attachment-2019-11-06-103414_002.jpeg | 02:40 |
| gibi_ptg | so far we can fit | 02:40 |
| *** ricolin has joined #openstack-nova | 02:52 | |
| *** liuyulong has quit IRC | 02:57 | |
| *** ricolin has quit IRC | 03:08 | |
| *** sapd1_x has joined #openstack-nova | 03:13 | |
| *** Liang__ has joined #openstack-nova | 03:30 | |
| *** psachin has joined #openstack-nova | 03:32 | |
| *** psachin has quit IRC | 03:33 | |
| *** psachin has joined #openstack-nova | 03:35 | |
| *** ricolin has joined #openstack-nova | 03:37 | |
| *** TxGirlGeek has joined #openstack-nova | 03:41 | |
| *** artom has quit IRC | 03:43 | |
| *** tkajinam has quit IRC | 03:45 | |
| *** bhagyashris has joined #openstack-nova | 03:55 | |
| *** maciejjozefczyk has quit IRC | 04:03 | |
| *** Liang__ has quit IRC | 04:04 | |
| *** sapd1_x has quit IRC | 04:15 | |
| *** slaweq has quit IRC | 04:19 | |
| *** ricolin_ has joined #openstack-nova | 04:22 | |
| *** ricolin has quit IRC | 04:25 | |
| eandersson | We just deployed the latest nova patches for Rocky | 04:28 |
| eandersson | and are getting | 04:28 |
| eandersson | > Could not retrieve compute node resource provider XXX and therefore unable to error out any instances stuck in BUILDING state. | 04:28 |
| eandersson | On hundreds of nodes on startup | 04:28 |
| *** bhagyashris has quit IRC | 04:31 | |
| *** mkrai has quit IRC | 04:34 | |
| *** mkrai_ has joined #openstack-nova | 04:35 | |
| *** ricolin_ has quit IRC | 04:35 | |
| eandersson | pretty sure what happens is that in Rocky with latest stable it will error if you restart a compute node with no instances | 04:36 |
| eandersson | https://github.com/openstack/nova/commit/13bb7ed701121955ba015103c2e44429927e78d4#diff-77f9348ab09642ba46409b6828af4af0R1327 | 04:36 |
| eandersson | actually it happens on non-empty computes as well | 04:44 |
| *** TxGirlGeek has quit IRC | 04:50 | |
| *** ociuhandu has joined #openstack-nova | 05:05 | |
| *** slaweq has joined #openstack-nova | 05:06 | |
| *** bhagyashris has joined #openstack-nova | 05:07 | |
| eandersson | Interesting | 05:07 |
| eandersson | An error'd compute due to quota still takes up an allocation | 05:07 |
| eandersson | Preventing heal_allocation from progressing | 05:08 |
| *** ociuhandu has quit IRC | 05:10 | |
| eandersson | Turns out placement isn't in a great place in that region. Probably why we are hitting errors. | 05:15 |
| *** ratailor has joined #openstack-nova | 05:27 | |
| *** ociuhandu has joined #openstack-nova | 05:30 | |
| *** irclogbot_3 has quit IRC | 05:34 | |
| *** bhagyashris_ has joined #openstack-nova | 05:34 | |
| *** takashin has quit IRC | 05:34 | |
| *** bhagyashris has quit IRC | 05:35 | |
| *** takashin has joined #openstack-nova | 05:35 | |
| *** bnemec has joined #openstack-nova | 05:35 | |
| *** ociuhandu has quit IRC | 05:36 | |
| *** bhagyashris has joined #openstack-nova | 05:36 | |
| *** maciejjozefczyk has joined #openstack-nova | 05:36 | |
| *** Liang__ has joined #openstack-nova | 05:37 | |
| *** irclogbot_1 has joined #openstack-nova | 05:38 | |
| *** tkajinam has joined #openstack-nova | 05:39 | |
| eandersson | I wish heal_allocations would have an option to skip failed computes. | 05:39 |
| *** bhagyashris_ has quit IRC | 05:39 | |
| eandersson | When you are in a bad state you basically need to do some mysql magic to figure out "bad" hosts. Otherwise you are fixing them sequentially. | 05:40 |
| *** mdbooth has joined #openstack-nova | 05:42 | |
| *** links has joined #openstack-nova | 05:47 | |
| *** mkrai_ has quit IRC | 05:53 | |
| *** sorrison has joined #openstack-nova | 05:53 | |
| *** mkrai has joined #openstack-nova | 05:54 | |
| *** abhishekk has joined #openstack-nova | 05:57 | |
| *** pcaruana has joined #openstack-nova | 05:59 | |
| *** takashin has quit IRC | 05:59 | |
| *** abhishekk has quit IRC | 06:21 | |
| *** nanzha has quit IRC | 06:24 | |
| *** takashin has joined #openstack-nova | 06:25 | |
| *** mdbooth has quit IRC | 06:25 | |
| *** nanzha has joined #openstack-nova | 06:26 | |
| *** mdbooth has joined #openstack-nova | 06:32 | |
| *** takashin has left #openstack-nova | 06:40 | |
| *** abhishekk has joined #openstack-nova | 06:43 | |
| *** mkrai has quit IRC | 06:47 | |
| *** bnemec has quit IRC | 06:49 | |
| *** dpawlik has joined #openstack-nova | 06:54 | |
| *** bnemec has joined #openstack-nova | 06:57 | |
| *** nanzha has quit IRC | 06:58 | |
| *** abhishekk has quit IRC | 07:00 | |
| *** nanzha has joined #openstack-nova | 07:00 | |
| *** ricolin_ has joined #openstack-nova | 07:02 | |
| *** tkajinam has quit IRC | 07:08 | |
| *** mkrai has joined #openstack-nova | 07:09 | |
| *** mdbooth has quit IRC | 07:11 | |
| *** Liang__ has quit IRC | 07:12 | |
| *** Liang__ has joined #openstack-nova | 07:14 | |
| *** bnemec has quit IRC | 07:14 | |
| *** bnemec has joined #openstack-nova | 07:16 | |
| *** Liang__ has quit IRC | 07:17 | |
| *** jawad_axd has joined #openstack-nova | 07:17 | |
| *** Liang__ has joined #openstack-nova | 07:17 | |
| *** fungi has quit IRC | 07:20 | |
| *** mdbooth has joined #openstack-nova | 07:21 | |
| *** lpetrut has joined #openstack-nova | 07:22 | |
| *** fungi has joined #openstack-nova | 07:24 | |
| *** lpetrut has quit IRC | 07:33 | |
| *** luksky has joined #openstack-nova | 07:34 | |
| *** slaweq_ has joined #openstack-nova | 07:34 | |
| *** mkrai has quit IRC | 07:36 | |
| *** slaweq has quit IRC | 07:36 | |
| *** bnemec has quit IRC | 07:38 | |
| *** bnemec has joined #openstack-nova | 07:38 | |
| *** damien_r has joined #openstack-nova | 07:52 | |
| *** mkrai has joined #openstack-nova | 07:52 | |
| *** ygk_12345 has joined #openstack-nova | 08:01 | |
| *** ygk_12345 has quit IRC | 08:02 | |
| *** rcernin_ has quit IRC | 08:06 | |
| *** bhagyashris has quit IRC | 08:07 | |
| *** tesseract has joined #openstack-nova | 08:08 | |
| *** abaindur has quit IRC | 08:08 | |
| *** lpetrut has joined #openstack-nova | 08:14 | |
| *** bnemec has quit IRC | 08:20 | |
| *** arxcruz|off is now known as arxcruz | 08:21 | |
| *** maciejjozefczyk has quit IRC | 08:25 | |
| *** lpetrut has quit IRC | 08:26 | |
| *** dpawlik has quit IRC | 08:28 | |
| *** ricolin_ is now known as ricolin | 08:28 | |
| *** dpawlik has joined #openstack-nova | 08:29 | |
| *** nanzha has quit IRC | 08:35 | |
| *** nanzha has joined #openstack-nova | 08:36 | |
| *** maciejjozefczyk has joined #openstack-nova | 08:45 | |
| *** ralonsoh has joined #openstack-nova | 08:46 | |
| *** maciejjozefczyk has quit IRC | 08:50 | |
| *** slaweq_ has quit IRC | 08:55 | |
| *** Liang__ has quit IRC | 08:56 | |
| kashyap | gibi_ptg: Good luck :-) | 08:57 |
| *** ricolin has quit IRC | 08:57 | |
| *** ociuhandu has joined #openstack-nova | 09:02 | |
| *** ociuhandu has quit IRC | 09:07 | |
| *** dpawlik has quit IRC | 09:25 | |
| *** ociuhandu has joined #openstack-nova | 09:38 | |
| *** mdbooth has joined #openstack-nova | 09:56 | |
| *** nanzha has quit IRC | 09:57 | |
| *** nanzha has joined #openstack-nova | 09:57 | |
| *** dpawlik has joined #openstack-nova | 09:58 | |
| *** slaweq_ has joined #openstack-nova | 10:00 | |
| *** dpawlik has quit IRC | 10:03 | |
| *** dpawlik has joined #openstack-nova | 10:04 | |
| *** slaweq__ has joined #openstack-nova | 10:04 | |
| *** slaweq_ has quit IRC | 10:05 | |
| *** slaweq__ has quit IRC | 10:10 | |
| *** mdbooth has quit IRC | 10:13 | |
| *** derekh has joined #openstack-nova | 10:13 | |
| *** xek has joined #openstack-nova | 10:13 | |
| openstackgerrit | Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825 | 10:36 |
| openstackgerrit | Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072 | 10:36 |
| *** priteau has joined #openstack-nova | 10:51 | |
| *** rcernin_ has joined #openstack-nova | 10:55 | |
| *** luksky has quit IRC | 10:56 | |
| *** ociuhandu has quit IRC | 11:20 | |
| *** mkrai has quit IRC | 11:27 | |
| *** rcernin_ has quit IRC | 11:31 | |
| *** artom has joined #openstack-nova | 11:41 | |
| *** mkrai has joined #openstack-nova | 11:43 | |
| *** luksky has joined #openstack-nova | 11:45 | |
| *** ratailor has quit IRC | 11:47 | |
| *** bhagyashris has joined #openstack-nova | 11:48 | |
| *** tbachman has quit IRC | 11:50 | |
| *** lpetrut has joined #openstack-nova | 11:52 | |
| *** luksky has quit IRC | 11:56 | |
| *** sapd1_x has joined #openstack-nova | 12:02 | |
| *** eharney has quit IRC | 12:12 | |
| sean-k-mooney | ... so the random number generator has been broken in nova for a while... | 12:23 |
| sean-k-mooney | and by a while i mean since https://github.com/openstack/nova/commit/be548a3396ee5f3a78b2b8b59321eb4c4530f7fb merged in queens | 12:24 |
| sean-k-mooney | the comments is in the wrong place | 12:25 |
| sean-k-mooney | https://github.com/openstack/nova/commit/be548a3396ee5f3a78b2b8b59321eb4c4530f7fb#diff-f4019782d93a196a0d026479e6aa61b1L4037-L4071 | 12:25 |
| sean-k-mooney | after teh refactor the rng is only enabled if you also enable the qemu guest agent | 12:25 |
| sean-k-mooney | before it was eneabled if you enabled it in the image and the host was qemu or kvm | 12:26 |
| sean-k-mooney | kashyap: ^ we proably need to fix that but we talked before about enableing the rng by default. it should be two seperate patches but do you think we shoudl still enable it by default | 12:27 |
| *** mkrai has quit IRC | 12:30 | |
| *** sapd1_x has quit IRC | 12:34 | |
| *** jaosorior has joined #openstack-nova | 12:35 | |
| *** bhagyashris has quit IRC | 12:35 | |
| sean-k-mooney | i have filed a bug for it here https://bugs.launchpad.net/nova/+bug/1851497 | 12:38 |
| openstack | Launchpad bug 1851497 in OpenStack Compute (nova) "rng is only enabled if qemu guest agent is enabled" [Medium,Triaged] - Assigned to sean mooney (sean-k-mooney) | 12:38 |
| *** rouk has quit IRC | 12:38 | |
| *** mkrai has joined #openstack-nova | 12:45 | |
| *** xek has quit IRC | 12:53 | |
| *** xek has joined #openstack-nova | 12:53 | |
| sean-k-mooney | efried: hi so http://paste.openstack.org/show/785846/ seams to now work. im going to kick it off on a clean vm to test but that stacked and i could boot vms | 12:54 |
| *** luksky has joined #openstack-nova | 13:03 | |
| *** eharney has joined #openstack-nova | 13:15 | |
| efried | sean-k-mooney: cool beans | 13:16 |
| efried | sean-k-mooney: I just need to update that enable_plugin line for my purposes, right? The rest ought to be unchanged from yesterday? | 13:21 |
| sean-k-mooney | yep | 13:22 |
| sean-k-mooney | ill proably merge the patch today or tomorow but i want to do a clean install first to make sure i have not missed something | 13:22 |
| sean-k-mooney | i need to actully set up a gate job to actully test it rather then manually testing it every time i update it | 13:23 |
| efried | okey. Were you able to repro the last problem I had yesterday where nova thought I didn't have a recent enough qemu? | 13:23 |
| efried | like, it was looking in the wrong place or something | 13:23 |
| sean-k-mooney | not exactly but i noticed i was not unistalling qemu from the package manager before installing it for source | 13:24 |
| sean-k-mooney | so i fixed that | 13:24 |
| sean-k-mooney | i had that for libvirt but not qemu | 13:24 |
| *** derekh has quit IRC | 13:24 | |
| efried | ah | 13:25 |
| efried | sounds like that would probably do it. | 13:25 |
| efried | k, I'm running now. Cleaned first to get rid of compiled libvirt/qemu -- I needed that because of new compile flaggage, rite? | 13:25 |
| efried | for apparmor | 13:26 |
| sean-k-mooney | actully no i have not added that yet | 13:26 |
| sean-k-mooney | but i found the bits i needed to set to disable the use of polkit | 13:27 |
| sean-k-mooney | a clean does not hurt it just will be slower | 13:27 |
| efried | ight. We'll see how this goes :) | 13:27 |
| sean-k-mooney | by the way to go back to normal you just need to unstack then comment out the pluging. it should remove the libvirt and qemu it installed on unstack but wont delete the source in /opt/stack | 13:29 |
| efried | cool | 13:29 |
| efried | I'll never go back to normal | 13:29 |
| efried | This is my new normal | 13:29 |
| sean-k-mooney | :) | 13:29 |
| *** tbachman has joined #openstack-nova | 13:36 | |
| *** mkrai has quit IRC | 13:37 | |
| *** sridharg has joined #openstack-nova | 13:38 | |
| *** slaweq__ has joined #openstack-nova | 13:39 | |
| *** slaweq has joined #openstack-nova | 13:44 | |
| *** slaweq__ has quit IRC | 13:44 | |
| *** jmlowe has quit IRC | 13:54 | |
| efried | sean-k-mooney: no idea why this should be failing differently on my otherwise-identical VMs, but one is hitting the libvirt-python install failure again, and the other is hitting the /var/run/libvirt/libvirt-sock ENOENT again. | 14:00 |
| *** derekh has joined #openstack-nova | 14:07 | |
| *** sridharg has quit IRC | 14:07 | |
| *** davee_ has joined #openstack-nova | 14:11 | |
| sean-k-mooney | so the libvirt python failure is complaingin about the libvit.so right with pkg-config or somthing like that and the other case libvirt is not running? | 14:12 |
| efried | sean-k-mooney: Maybe I'm not looking at the right patch set. I don't see a delta where you're purging qemu for instance. | 14:13 |
| sean-k-mooney | its here https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@102 | 14:14 |
| efried | ah, k | 14:14 |
| efried | well, now I got both to fail the same way. The one that was failing with ENOENT is now failing on the libvirt-python thing. | 14:15 |
| efried | I cut it over to USE_PYTHON3 | 14:15 |
| sean-k-mooney | ok. am if you look in /usr/lib is there a libvirt.so | 14:16 |
| *** jmlowe has joined #openstack-nova | 14:16 | |
| efried | yes | 14:17 |
| sean-k-mooney | ok if you unstack it should remove it | 14:17 |
| efried | and that's what I want? | 14:17 |
| sean-k-mooney | if it does not run " sudo rm -f /urs/lib/libvirt* | 14:17 |
| sean-k-mooney | well we want to make sure its the libvirt.so you compiled for a start | 14:18 |
| sean-k-mooney | it shoudl be | 14:18 |
| efried | fwiw the date stamp implies that it is. | 14:18 |
| efried | everything /usr/lib/libvirt* has a now-ish date stamp | 14:19 |
| efried | and yes, unstacking removes everything /usr/lib/libvirt* | 14:19 |
| sean-k-mooney | ok so if you run "pkg-config libvirt ; echo $?" | 14:19 |
| sean-k-mooney | im guessin git now prints a non 0 value right | 14:20 |
| sean-k-mooney | if so can you cd in /opt/stack/libvit and do sudo make install | 14:20 |
| sean-k-mooney | then run it again | 14:20 |
| efried | yes, confirm all of that. | 14:21 |
| sean-k-mooney | so if you run it after you do make install is it 0 or non 0 | 14:22 |
| efried | 0 | 14:22 |
| sean-k-mooney | ok and if you manually to the pip install of libvirt-python? | 14:22 |
| efried | and the checksum of the lib is the same as on the other system where I didn't unstack and manually `make install` | 14:22 |
| sean-k-mooney | ya it should be | 14:22 |
| efried | same error: | 14:23 |
| efried | /usr/bin/pkg-config --print-errors --atleast-version=0.9.11 libvirt | 14:23 |
| efried | /usr/bin/python generator.py libvirt /usr/share/libvirt/api/libvirt-api.xml | 14:23 |
| efried | Found 475 functions in /usr/share/libvirt/api/libvirt-api.xml | 14:23 |
| efried | Found 0 functions in libvirt-override-api.xml | 14:23 |
| efried | Generated 385 wrapper functions | 14:23 |
| efried | Missing type converters: | 14:23 |
| efried | virTypedParameterPtr:1 | 14:23 |
| efried | ERROR: failed virConnectSetIdentity | 14:23 |
| efried | error: command '/usr/bin/python' failed with exit status 1 | 14:23 |
| sean-k-mooney | if you do pip show libvirt-python | 14:24 |
| sean-k-mooney | what version do you see | 14:24 |
| sean-k-mooney | im using 5.7.0 | 14:24 |
| efried | When we hit this yesterday, it went away magically by itself | 14:24 |
| efried | 5.5.0 | 14:24 |
| efried | it's trying to install 5.7.0 that breaks | 14:25 |
| sean-k-mooney | if you try and install 5.5.0 does it work | 14:25 |
| sean-k-mooney | e.g. if you remove it and add it | 14:25 |
| efried | here's a thing: I'm running py2 pip | 14:26 |
| sean-k-mooney | so am i | 14:27 |
| sean-k-mooney | pip version 9.0.3 | 14:27 |
| sean-k-mooney | the one devstack installs | 14:27 |
| efried | if I uninstall 5.5.0 and try to reinstall, it fails (at 5.5.0 and at 5.7.0) the same way | 14:28 |
| efried | also with pip3 | 14:28 |
| efried | what if I tried setting the plugin to build libvirt 5.7.0? | 14:30 |
| sean-k-mooney | oh... | 14:30 |
| sean-k-mooney | yes you could do that | 14:30 |
| sean-k-mooney | but its passing form me because its using the cached wheel | 14:30 |
| sean-k-mooney | if i try to use 5.5.0 i get your error | 14:31 |
| efried | like you already had 5.7.0 installed before you started? | 14:31 |
| sean-k-mooney | actully i get | 14:31 |
| sean-k-mooney | ERROR: failed virConnectSetIdentity | 14:31 |
| sean-k-mooney | ERROR: failed virDomainGetGuestInfo | 14:31 |
| sean-k-mooney | error: command '/usr/bin/python' failed with exit status 1 | 14:31 |
| efried | yes, that's what I'm seeing. | 14:31 |
| *** mmethot_ has joined #openstack-nova | 14:31 | |
| efried | it's slightly different with 5.5 vs 5.7 | 14:31 |
| *** slaweq has quit IRC | 14:31 | |
| sean-k-mooney | so i proably had libvirt python 5.7 alredy install as that is the lates version | 14:32 |
| sean-k-mooney | and i had previously stacked | 14:32 |
| sean-k-mooney | ill ask downstream one sec | 14:32 |
| *** mmethot has quit IRC | 14:34 | |
| sean-k-mooney | efried: it might be this? https://github.com/libvirt/libvirt-python/commit/873e0ca7db3ae43bd514354997cdfa7ea5b7f70a | 14:35 |
| efried | you mean that's the fix for same | 14:36 |
| efried | kinda looks that way, yeah. | 14:36 |
| efried | It wouldn't surprise me that libvirt-python's version has to match that of libvirt itself. | 14:36 |
| efried | lemme clean and try building 5.7.0 | 14:36 |
| sean-k-mooney | efried: so apparently the libvirt python version needs to be newer or the same as the libvir tyou are compiling | 14:38 |
| efried | I guess libvirt-python hasn't been published to pypi yet or something? Why wouldn't `pip install -U` pick up the latest? | 14:38 |
| efried | tried asking explicitly for 5.8.0 and it bounces | 14:39 |
| sean-k-mooney | ya let me check pypy you can install it from souce if you need too | 14:39 |
| efried | nah, I'll just bust down to libvirt 5.7.0 | 14:40 |
| efried | I only need 5.6.0 min | 14:40 |
| efried | but I guess an improvement for your script would be a version qualifier on libvirt-python >= whatever version of libvirt you're compiling. | 14:41 |
| sean-k-mooney | yep | 14:41 |
| efried | I don't know if that failure ("can't find libvirt-python 5.8.0") would be more obvious or not. | 14:41 |
| sean-k-mooney | i proably can also do a safty check or install it form souce. | 14:43 |
| sean-k-mooney | it would not be hard to just clone it and check out the same tag | 14:43 |
| sean-k-mooney | then pip install it | 14:43 |
| efried | depends what the plugin user wants it for. If it's not available on pypi that means it's not released yet, right? | 14:43 |
| efried | Though I thought tags don't move once they're set. | 14:44 |
| *** lennyb has quit IRC | 14:44 | |
| sean-k-mooney | efried: no its released just talking to danpb he though it was released he is fixing it now | 14:44 |
| efried | o | 14:44 |
| sean-k-mooney | efried: its there now | 14:45 |
| *** lennyb has joined #openstack-nova | 14:46 | |
| efried | cool. I've already got both of these stacking at 5.7.0; if they fail I'll bump one up. | 14:46 |
| *** xek has quit IRC | 14:46 | |
| *** xek has joined #openstack-nova | 14:46 | |
| efried | I guess now we'll see if libvirt-python 5.8.0 works with libvirt 5.7.0 :P | 14:47 |
| sean-k-mooney | i manulaly installed it and it did at least on my host | 14:48 |
| *** slaweq has joined #openstack-nova | 14:51 | |
| sean-k-mooney | melwitt: by the way have you seen migrate.exceptions.VersionNotFoundError: \"Database schema file with version 72 doesn't exist.\" before | 14:52 |
| sean-k-mooney | melwitt: were you looking at somthing similar a while ago | 14:52 |
| sean-k-mooney | that is form a "nova-manage api_db sync" | 14:53 |
| *** slaweq has quit IRC | 14:55 | |
| *** mriedem has joined #openstack-nova | 15:00 | |
| *** xek has quit IRC | 15:07 | |
| *** xek_ has joined #openstack-nova | 15:07 | |
| *** links has quit IRC | 15:08 | |
| *** xek_ has quit IRC | 15:09 | |
| *** xek__ has joined #openstack-nova | 15:09 | |
| *** tbachman has quit IRC | 15:14 | |
| mriedem | gibi_ptg: when it's not past 11pm in china and you're free, you were +2 on this cross cell change https://review.opendev.org/#/c/635668/ and now dan is too after some updates if you can hit it again | 15:16 |
| *** tbachman has joined #openstack-nova | 15:16 | |
| *** dpawlik has quit IRC | 15:19 | |
| *** jmlowe has quit IRC | 15:22 | |
| *** jmlowe has joined #openstack-nova | 15:24 | |
| sean-k-mooney | mriedem: you generally know what direction to look to root casue thing with db migrations | 15:24 |
| sean-k-mooney | mriedem: does http://paste.openstack.org/show/785852/ look familar | 15:24 |
| sean-k-mooney | igrate.exceptions.VersionNotFoundError: \"Database schema file with version 72 doesn't exist.\"", | 15:24 |
| sean-k-mooney | im not sure where the version 72 comes form in the code | 15:24 |
| sean-k-mooney | oh its here https://github.com/openstack/nova/tree/master/nova/db/sqlalchemy/api_migrations/migrate_repo/versions | 15:26 |
| *** tbachman has quit IRC | 15:26 | |
| *** TxGirlGeek has joined #openstack-nova | 15:26 | |
| *** tbachman has joined #openstack-nova | 15:28 | |
| *** lbragstad_ is now known as lbragstad | 15:33 | |
| dansmith | sean-k-mooney: yeah, those are where the indexes are | 15:38 |
| dansmith | sean-k-mooney: usually that error happens when you have a pyc of the migration but not the py of it, because you have a dirty working directory or something | 15:38 |
| sean-k-mooney | dansmith: in this case its becaue i deploy kolla with master after you added the placeholders | 15:39 |
| sean-k-mooney | but the current master contaienr are now clamped to traing | 15:39 |
| sean-k-mooney | *train | 15:39 |
| sean-k-mooney | while they are preparing an rc | 15:39 |
| sean-k-mooney | so the api sysnc is trying to do a downgrade | 15:39 |
| dansmith | sure, but.. the only thing that causes it to know about migration 72 is the fact that that file exists on disk | 15:39 |
| sean-k-mooney | which obviously wont work | 15:39 |
| dansmith | downgrade? | 15:40 |
| sean-k-mooney | they dont exist on stable train right | 15:40 |
| dansmith | that shouldn't cause that error | 15:40 |
| dansmith | unless you have a database with version 72 and are trying to run downgrade I guess | 15:40 |
| mriedem | downgrade isn't around anymore | 15:40 |
| mriedem | sdague ripped it out a long time ago | 15:41 |
| sean-k-mooney | ya | 15:41 |
| dansmith | mriedem: it's just a pass and technically runs IIRC | 15:41 |
| sean-k-mooney | its fine | 15:41 |
| mriedem | the method doesn't exist in the scripts anymore | 15:41 |
| sean-k-mooney | its just my home openstack | 15:41 |
| dansmith | but I dunno why anything would be trying to do it | 15:41 |
| *** JamesBenson has joined #openstack-nova | 15:41 | |
| dansmith | mriedem: we know that | 15:42 |
| sean-k-mooney | it was bad timing that i deploy master just after nova rc1 when that was merged | 15:42 |
| dansmith | sean-k-mooney: so yeah must be that sqla-migrate sees the db version as 72 and is looking for the downgrade script I guess? but..that'd be your problem | 15:42 |
| sean-k-mooney | but the current "master" kolla image are stable train as they have not done there rc1 yet | 15:42 |
| mriedem | heh so you time traveled | 15:43 |
| sean-k-mooney | ya i can just build the container form master nova and continue or redeploy with stable train | 15:43 |
| sean-k-mooney | its not a big deal | 15:43 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add --dry-run option to heal_allocations CLI https://review.opendev.org/693199 | 15:44 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add --instance option to heal_allocations https://review.opendev.org/693200 | 15:44 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/stein: Add BFV wrinkle to TestNovaManagePlacementHealAllocations https://review.opendev.org/693201 | 15:44 |
| *** gyee has joined #openstack-nova | 15:46 | |
| sean-k-mooney | well now i know where to look and how that works. im off to lunch brb | 15:46 |
| *** amodi has quit IRC | 15:48 | |
| *** igordc has joined #openstack-nova | 15:50 | |
| openstackgerrit | Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825 | 15:53 |
| openstackgerrit | Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072 | 15:53 |
| *** amodi has joined #openstack-nova | 15:56 | |
| *** xek_ has joined #openstack-nova | 16:00 | |
| *** slaweq has joined #openstack-nova | 16:02 | |
| *** xek__ has quit IRC | 16:02 | |
| *** lpetrut has quit IRC | 16:03 | |
| openstackgerrit | Boris Bobrov proposed openstack/nova master: Move rng device checks to the appropriate method https://review.opendev.org/693210 | 16:04 |
| *** macz has joined #openstack-nova | 16:04 | |
| *** tbachman has quit IRC | 16:06 | |
| efried | sean-k-mooney: back to n-cpu failing to start because of | 16:07 |
| efried | '/var/run/libvirt/libvirt-sock': No such file or directory. | 16:07 |
| efried | The libvirt-bin service is inactive. There's no journal, but systemctl shows these as the last two lines: | 16:07 |
| efried | Nov 06 09:19:25 breakfast dnsmasq-dhcp[27199]: read /var/lib/libvirt/dnsmasq/default.hostsfile | 16:07 |
| efried | Nov 06 09:22:55 breakfast dnsmasq[27199]: exiting on receipt of SIGTERM | 16:07 |
| efried | I don't see anything in the stack logs that looks like it ought to have shut the service down. | 16:07 |
| *** slaweq has quit IRC | 16:07 | |
| *** damien_r has quit IRC | 16:07 | |
| *** artom has quit IRC | 16:07 | |
| efried | On both stacks the systemctl status shows that ~3.5min gap between loading default.hostsfile and SIGTERM | 16:08 |
| *** jmlowe has quit IRC | 16:10 | |
| *** derekh has quit IRC | 16:11 | |
| efried | sean-k-mooney: I was able to manually start libvirt-bin and it hasn't stopped itself, so clearly something else was hitting it... | 16:13 |
| efried | ...but with the service running, n-cpu still fails to come up in the same way. | 16:14 |
| *** tbachman has joined #openstack-nova | 16:15 | |
| *** derekh has joined #openstack-nova | 16:17 | |
| *** derekh has quit IRC | 16:17 | |
| openstackgerrit | Boris Bobrov proposed openstack/nova master: Also enable iommu for virtio controllers and video in libvirt https://review.opendev.org/684825 | 16:21 |
| openstackgerrit | Boris Bobrov proposed openstack/nova master: Create a controller for qga when SEV is used https://review.opendev.org/693072 | 16:21 |
| mriedem | amodi: melwitt: looks like we can keep the iso file without changes https://review.opendev.org/#/c/693065/7 | 16:22 |
| mriedem | it was already fixed in the manuals repo before i imported that doc https://github.com/openstack/openstack-manuals/commit/1206b9b727594ae04f0d2a4818f183e95253b464 | 16:22 |
| openstackgerrit | Merged openstack/nova master: Pass exception through TaskBase.rollback https://review.opendev.org/692689 | 16:22 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | 16:25 |
| mriedem | this is why i'm not as free willy on accepting any old docs change since it's a pain in the butt to fix them if they are wrong | 16:27 |
| melwitt | haha free willy | 16:30 |
| *** bnemec has joined #openstack-nova | 16:30 | |
| melwitt | and yeah, understood. thanks for pointing out those previous fixes and updating it | 16:31 |
| dansmith | mriedem: query | 16:34 |
| dansmith | was just looking at your heal_instance_mappings thing | 16:34 |
| dansmith | and was thinking. | 16:34 |
| *** ivve has quit IRC | 16:34 | |
| *** amodi has quit IRC | 16:35 | |
| dansmith | I wonder if it wouldn't make sense for us to have another totally optional service with the sole purpose of looking for things that need healing.. instance mappings, maybe allocations, host mappings, maybe even archive and purge | 16:35 |
| dansmith | we could make it a daemon, or something you could run from cron single-shot if you just wanted to run it at midnight or something | 16:36 |
| melwitt | I think operators would love that | 16:36 |
| dansmith | instead of a ton of cron entries for all those things, and not knowing you need new cron entries to run new things on a new release, | 16:37 |
| dansmith | you just run one nova-audit in one place, no HA, just low and slow burn of audity things | 16:37 |
| melwitt | yeah, would be nice | 16:37 |
| mriedem | i'm not opposed | 16:38 |
| mriedem | would that also run the discover host mappings thing that the scheduler can run today? | 16:39 |
| mriedem | i remember glance has several things like that https://github.com/openstack/glance/blob/master/setup.cfg#L34 | 16:40 |
| mriedem | though i'm not sure if they are all daemons | 16:40 |
| dansmith | mriedem: yeah, hence the host mappings in my list | 16:40 |
| *** nanzha has quit IRC | 16:40 | |
| mriedem | i will admit the number of heal/discover/sync things is getting a bit overwhelming | 16:41 |
| dansmith | right | 16:41 |
| dansmith | impossible to track for the average operator I'm sure | 16:42 |
| mriedem | and we have talked about a periodic archive/purge thing in the past | 16:42 |
| dansmith | and we put the host mappings thing in scheduler mostly because the cardinality was small | 16:42 |
| dansmith | (and then I subsequently solved their problem a different way) | 16:42 |
| mriedem | and i think b/c api and conductor didn't have periodic tasks already but scheduler did | 16:42 |
| dansmith | maybe, but the point was to run one or few of those since they'd step all over each other, and there'd be too many conductors and apis even if they did have periodics | 16:43 |
| mriedem | maybe you want to kick that idea out into the mailing list tagged with [nova][ops] to get initial feedback? and/or just start a spec and kick that into the ML to get attention from operators. | 16:43 |
| mriedem | we do run nproc/2 scheduler works by default now i think.. | 16:44 |
| mriedem | but yeah | 16:44 |
| dansmith | yeah, I could start with a spec | 16:44 |
| dansmith | I was kinda thinking of just starting with code | 16:44 |
| mriedem | now that you've gotten a taste from the image cache stuff you're addicted! | 16:45 |
| dansmith | JUST ONE MORE HIT MAN | 16:45 |
| *** jaosorior has quit IRC | 16:48 | |
| efried | sean-k-mooney: The libvirt-bin service seems to not want to stay alive. When I start it manually, it runs for a couple minutes and then dies, but doesn't log anything. | 16:48 |
| efried | If I start n-cpu in that window, I run into the qemu version check. I added some debug and find it's coming from: | 16:48 |
| efried | Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host File "/usr/local/lib/python3.6/dist-packages/libvirt.py", line 4190, in getVersion | 16:48 |
| efried | Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host if ret == -1: raise libvirtError ('virConnectGetVersion() failed', conn=self) | 16:48 |
| efried | Nov 06 10:45:03 breakfast nova-compute[15654]: ERROR nova.virt.libvirt.host libvirt.libvirtError: internal error: Cannot find suitable emulator for x86_64 | 16:48 |
| sean-k-mooney | is qemu-system-x86_64 | 16:50 |
| sean-k-mooney | in your path | 16:50 |
| sean-k-mooney | also do you have permission denied errors in dmesg | 16:50 |
| openstackgerrit | Merged openstack/nova master: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693065 | 16:51 |
| efried | you mean like this | 16:51 |
| efried | [73117.887250] audit: type=1400 audit(1573058703.103:556): apparmor="DENIED" operation="exec" profile="/usr/sbin/libvirtd" name="/usr/local/bin/qemu-system-microblazeel" pid=15694 comm="libvirtd" requested_mask="x" denied_mask="x" fsuid=0 ouid=0 | 16:51 |
| sean-k-mooney | yes | 16:51 |
| sean-k-mooney | so apparmor is blocking it | 16:51 |
| efried | (yes, qemu-[system-]x86_64 is in /usr/local/bin, which is in my PATH, at least on the CLI) | 16:51 |
| efried | oh, I thought that libvirtd.conf hack was supposed to get around apparmor | 16:52 |
| sean-k-mooney | ya so if we fix the apparmor issue it shoudl be working | 16:52 |
| sean-k-mooney | no it fixs the fact libvirt expect polkit to be configure to use apparmor | 16:52 |
| sean-k-mooney | i really should just add teh compile flag | 16:52 |
| efried | is that easy? | 16:52 |
| sean-k-mooney | i think its just https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@93 | 16:53 |
| sean-k-mooney | add --with-apparmor_profiles after --system | 16:53 |
| sean-k-mooney | how im working around it at the moment was https://review.opendev.org/#/c/693094/3/README.rst | 16:54 |
| sean-k-mooney | sudo ln -s /etc/apparmor.d/usr.sbin.libvirtd /etc/apparmor.d/disable/usr.sbin.libvirtd | 16:54 |
| sean-k-mooney | then reboot for it to take effect. | 16:54 |
| efried | ah | 16:54 |
| efried | I'll try it both ways... | 16:54 |
| *** ioni has joined #openstack-nova | 16:55 | |
| sean-k-mooney | apparmor is the security sandboxing tool ubuntu use. its not happy that the binaries are in a different location then normal e.g. /usr/local/ | 16:55 |
| sean-k-mooney | hence teh error | 16:55 |
| efried | can't we `make install` with the right location? | 16:56 |
| sean-k-mooney | yes we can just change the prefix to /usr | 16:57 |
| sean-k-mooney | i have done that for libvirt actully | 16:57 |
| sean-k-mooney | just not qemu | 16:57 |
| mriedem | huh, i just saw the 'timed out waiting for response from cell' error in a functional test https://zuul.opendev.org/t/openstack/build/63001bbd58c244cea70c995f1ebf61fb/log/job-output.txt#3092 | 17:00 |
| sean-k-mooney | https://review.opendev.org/#/c/693094/3/devstack/libs/libvirt@113 this would have to be updated to "./configure --prefix=/usr" | 17:00 |
| sean-k-mooney | in test_evacuate_with_soft_anti_affinity | 17:01 |
| mriedem | i don't think that matters, | 17:02 |
| mriedem | we're just calling GET /servers/server_id to get the status probably | 17:02 |
| mriedem | WARNING [oslo.service.loopingcall] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 59.77 sec | 17:02 |
| sean-k-mooney | likely not but just was wondering which one | 17:02 |
| mriedem | chances are this is another infra node that hit some crazy io issue | 17:02 |
| sean-k-mooney | mriedem: oh that reminds me https://bugs.launchpad.net/nova/+bug/1851497 | 17:03 |
| openstack | Launchpad bug 1851497 in OpenStack Compute (nova) "rng is only enabled if qemu guest agent is enabled" [Medium,Triaged] - Assigned to sean mooney (sean-k-mooney) | 17:03 |
| sean-k-mooney | the random number generator code has apparently been partly broken since queens ... | 17:03 |
| sean-k-mooney | mriedem: did ye nail down the gate issue to io | 17:04 |
| mriedem | sean-k-mooney: https://review.opendev.org/#/c/129325/ was kilo | 17:04 |
| sean-k-mooney | if the vm runs out of entrapy it will have issue connecting to ssl sockets | 17:04 |
| mriedem | oh wait | 17:04 |
| mriedem | wrong change | 17:04 |
| mriedem | oh no that's right | 17:05 |
| mriedem | yeah kilo | 17:05 |
| mriedem | who would have guessed that a 373 LOC refactor would break something.... | 17:05 |
| sean-k-mooney | well no one noticed till now so ya | 17:06 |
| *** psachin has quit IRC | 17:06 | |
| sean-k-mooney | i was review the sev virtio viommu fix and found it odd that the qemu guest agent was adding a rng | 17:06 |
| melwitt | mriedem: that reminds me, I was chatting with penick about the gate io issues and he suggested infra/tempest using a flavor where the iops qos/limit has been set. are you familiar with that? (I'm not) | 17:07 |
| mriedem | the trend here is kind of interesting http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Function%20'nova.servicegroup.drivers.db.DbDriver._report_state'%20run%20outlasted%20interval%20by%5C%22&from=7d | 17:10 |
| mriedem | noon and midnight | 17:10 |
| mriedem | mostly on ovh | 17:10 |
| *** amodi has joined #openstack-nova | 17:10 | |
| efried | noisy cron job? | 17:10 |
| mriedem | melwitt: i think we were talking about that in here the other day a bit | 17:10 |
| *** efried is now known as efried_rollin | 17:11 | |
| amodi | mriedem++ melwitt++ thanks both!! | 17:14 |
| mriedem | amodi: happy backporting | 17:14 |
| amodi | lol sure :) | 17:14 |
| mriedem | i posted to the ML about the ovh + spike cron | 17:15 |
| mriedem | i was wondering about the dip there, but that's the weekend | 17:15 |
| *** TxGirlGeek has quit IRC | 17:15 | |
| mriedem | so that query is kind of annoying because there are a lot that are just a couple of seconds, but those 50+ ones are indicative of something really bad on the node | 17:16 |
| mriedem | mehinks it would be good to maybe have some config in oslo that sets a threshold for that and if it goes above the threshold then it's a WARNING but below it's just INFO? | 17:16 |
| mriedem | right now everything is WARNING which is kind of just noise | 17:16 |
| mriedem | bnemec: ^ what do you think about that? | 17:16 |
| *** igordc has quit IRC | 17:16 | |
| *** igordc has joined #openstack-nova | 17:17 | |
| mriedem | like i'd like to make that a warning if something outlasts 30 seconds in devstack/gate and then we can trace it in e-r | 17:17 |
| dansmith | mriedem: hmm, do we still need null_instance_uuid_scan? wasn't that like kilo and we have a constraint now? | 17:18 |
| mriedem | heh yeah i think we can deprecate and remove that https://review.opendev.org/#/c/97946/ | 17:19 |
| mriedem | https://review.opendev.org/#/q/topic:bp/db2-database+(status:open+OR+status:merged) !!! | 17:19 |
| mriedem | classic riderman | 17:19 |
| *** sapd1_x has joined #openstack-nova | 17:20 | |
| *** spatel has joined #openstack-nova | 17:21 | |
| *** artom has joined #openstack-nova | 17:21 | |
| *** JamesBen_ has joined #openstack-nova | 17:22 | |
| *** jawad_axd has quit IRC | 17:23 | |
| *** igordc has quit IRC | 17:23 | |
| *** JamesBenson has quit IRC | 17:26 | |
| *** jawad_axd has joined #openstack-nova | 17:29 | |
| *** melwitt is now known as jgwentworth | 17:31 | |
| *** jawad_axd has quit IRC | 17:32 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: DNM: break live migration to trigger bug 1788014 https://review.opendev.org/693045 | 17:37 |
| openstack | bug 1788014 in OpenStack Compute (nova) "when live migration fails due to a internal error rollback is not handeled correctly." [Medium,In progress] https://launchpad.net/bugs/1788014 - Assigned to Matt Riedemann (mriedem) | 17:37 |
| *** dtantsur is now known as dtantsur|afk | 17:38 | |
| jgwentworth | is there any way to discover whether force_config_drive=True in nova config from tempest? asking because of this tempest change https://review.opendev.org/693096 | 17:38 |
| *** TxGirlGeek has joined #openstack-nova | 17:40 | |
| openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226 | 17:43 |
| artom | jgwentworth, nothing's discoverable in Tempest, it's all configured | 17:43 |
| dansmith | mriedem: jgwentworth ^ wanna skim that for obvious booboos before I blast to the ML? | 17:43 |
| jgwentworth | artom: yeah, I was thinking whether there's some kinda api discoverability from nova that I'm not aware of | 17:44 |
| artom | jgwentworth, in this case, the [compute-features-enabled]/config_drive option | 17:44 |
| jgwentworth | right | 17:44 |
| artom | jgwentworth, ah, doesn't ring a bell | 17:44 |
| dansmith | jgwentworth: isn't that per compute node? | 17:44 |
| jgwentworth | yeah, I'm pretty sure there's not | 17:45 |
| jgwentworth | dansmith: oh, yeah I guess it can be | 17:45 |
| artom | jgwentworth, I suppose you could always boot a VM, ssh to it, and check | 17:45 |
| artom | But... | 17:45 |
| artom | You're meant to be using the config option :) | 17:45 |
| jgwentworth | sigh. ok. just thinking, the test could be useful if force_config_drive=False but it's not in the gate so it would be a dupe test | 17:45 |
| artom | And skip tests that need config drive if it's not set | 17:45 |
| jgwentworth | and is only not a dupe in envs where force_config_drive=False | 17:46 |
| jgwentworth | so at least in the gate, it should be disabled by default. maybe it should actually check the opposite, only run if [compute-features-enabled]/config_drive=False | 17:48 |
| jgwentworth | because tempest seems to be using that to mean whether config drive is being forced | 17:48 |
| artom | What does passing config_drive=True on server creation even do? | 17:49 |
| artom | "Indicates whether a config drive enables metadata injection." | 17:49 |
| artom | That's... no the same as "a config drive will be present" | 17:49 |
| artom | Is it? | 17:49 |
| larsks | artom: attaches a config drive to your instance that will be used for getting instance metadata instead of the network metadata service. | 17:50 |
| artom | larsks, see, I had no idea that was user-controllable in that way | 17:50 |
| larsks | Used e.g. by cloud-init for retrieving ssh keys, network confguration, etc. | 17:50 |
| artom | larsks, right, I know what a config drive *is* :) | 17:50 |
| artom | I just didn't think the user had that level of control over its presence | 17:51 |
| artom | "When this option is set to true config drive functionality will be forced | 17:52 |
| artom | enabled by default, otherwise users can still enable config drives via the REST | 17:52 |
| artom | API or image metadata properties. Launched instances are not affected by this | 17:52 |
| artom | option." | 17:52 |
| artom | Oh OK. | 17:52 |
| jgwentworth | ok, I commented accordingly on the review about inverting the logic of the skip | 17:53 |
| mriedem | because not everyone wants to run the metadata api service | 17:53 |
| artom | mriedem, so I heard | 17:53 |
| mriedem | and the metadata api is only as good as the stuff we persist, which isn't everything that you can use, like personality files | 17:53 |
| mriedem | i.e. you might get your goodies on boot but they are gone after resizing | 17:54 |
| dansmith | config drive is a big benefit for edge peeps | 17:55 |
| dansmith | because hitting the metadata api is a trip over the wan, either via http if it's central, or by it to the db which likely is | 17:55 |
| dansmith | configdrive keeps working even if you're isolated | 17:55 |
| openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226 | 18:00 |
| *** sapd1_x has quit IRC | 18:00 | |
| sean-k-mooney | you can use metadta service if you have isolated neutron networks too | 18:02 |
| sean-k-mooney | but you need to configure neturon to proxy the metadta api via the dhcp agent | 18:03 |
| sean-k-mooney | instead fo the router | 18:03 |
| dansmith | sean-k-mooney: to be clear, I meant partitioned as in...the wan is down | 18:03 |
| sean-k-mooney | but ya config drive woks regardless of the network topology | 18:03 |
| sean-k-mooney | ah right | 18:03 |
| sean-k-mooney | the only draw back with the config drive is that it will not be updated if you attach interface ectra at least while the vm is running | 18:04 |
| sean-k-mooney | in general i works well for what it does | 18:04 |
| artom | jgwentworth, I think gmann and the other tempest folks need to weigh in on https://review.opendev.org/#/c/693096/ - I'm not even sure it's necessary at all | 18:05 |
| jgwentworth | yes, I assumed tempest ppl will review it :) | 18:06 |
| artom | There's... latency, sometimes :) | 18:07 |
| *** JamesBen_ has quit IRC | 18:09 | |
| jgwentworth | yeah, I just meant it's a tempest patch so it's gonna be up to them | 18:09 |
| *** JamesBenson has joined #openstack-nova | 18:10 | |
| mriedem | dansmith: per usual you updated while i was leaving comments :/ | 18:13 |
| *** mriedem is now known as mriedem_lunch | 18:15 | |
| *** xek__ has joined #openstack-nova | 18:15 | |
| dansmith | mriedem_lunch: sorry man, just fixing pep8 | 18:16 |
| *** xek_ has quit IRC | 18:18 | |
| jgwentworth | dansmith: spec looks sane to me and ok for ML input | 18:18 |
| dansmith | jgwentworth: thanks, I'll make some of the changes mriedem_lunch suggested and then blast | 18:22 |
| *** lpetrut has joined #openstack-nova | 18:25 | |
| *** lpetrut has quit IRC | 18:26 | |
| *** igordc has joined #openstack-nova | 18:28 | |
| *** jgwentworth is now known as melwitt | 18:36 | |
| openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226 | 18:38 |
| openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add nova-audit spec https://review.opendev.org/693226 | 18:46 |
| * dansmith rocsk | 18:46 | |
| dansmith | omg the gate, the gate | 18:56 |
| dansmith | 5 minute turnaround time.... <3 | 18:56 |
| *** mriedem_lunch is now known as mriedem | 18:57 | |
| melwitt | we're living in gate luxury | 19:01 |
| sean-k-mooney | if only there was ptg every month | 19:02 |
| sean-k-mooney | and we did not have to travel | 19:03 |
| artom | PTG in my basement, hell yeah | 19:04 |
| *** tbachman has quit IRC | 19:21 | |
| *** tbachman has joined #openstack-nova | 19:27 | |
| *** tbachman has quit IRC | 19:27 | |
| *** tbachman has joined #openstack-nova | 19:28 | |
| *** spatel has quit IRC | 19:28 | |
| openstackgerrit | Archit Modi proposed openstack/nova stable/pike: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693236 | 19:31 |
| artom | Do we have a middleware-knowing person? | 19:32 |
| artom | I'm trying to figure out a 413 error for a downstream, and it's bugging me | 19:32 |
| artom | Even with debug enabled, the only log line I see is "POST /v2.1/servers" status: 413" | 19:32 |
| artom | Nothing else that's interesting for that request ID | 19:32 |
| artom | So... could it be oslo_middleware's sizelimit doing that? How would we make sure? | 19:33 |
| openstackgerrit | Archit Modi proposed openstack/nova stable/queens: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693239 | 19:37 |
| sean-k-mooney | artom: byond a print statemtent in the function im not really sure | 19:38 |
| artom | sean-k-mooney, or I guess removing it from the paste config | 19:38 |
| *** damien_r has joined #openstack-nova | 19:38 | |
| *** damien_r has quit IRC | 19:38 | |
| *** damien_r has joined #openstack-nova | 19:38 | |
| artom | But paste scares me | 19:38 |
| sean-k-mooney | ya although that might fail somewhere else then | 19:39 |
| mriedem | artom: efried_rollin has dorked around in a lot of that code lately | 19:39 |
| openstackgerrit | Archit Modi proposed openstack/nova stable/rocky: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693240 | 19:39 |
| sean-k-mooney | im sure there is a way to congirue custom midelware to log extra stuff in the 413 case but ya i didnt really want to have to learn that | 19:39 |
| sean-k-mooney | cdent know that code pretty well too i think | 19:40 |
| mriedem | amodi: i see you're doing backports backwards, | 19:40 |
| mriedem | amodi: note that you should be doing train -> stein -> rocky -> queens -> pike | 19:40 |
| mriedem | in that order | 19:40 |
| amodi | mriedem: yes | 19:40 |
| amodi | mriedem: oh i dint know it mattered | 19:40 |
| artom | mriedem, can I blow your mind? | 19:40 |
| mriedem | https://docs.openstack.org/project-team-guide/stable-branches.html#processes | 19:40 |
| amodi | mriedem: im still cherry picking from the master | 19:40 |
| artom | amodi is Red Hat QE! | 19:40 |
| sean-k-mooney | amodi: the patch has to merge on all the newer branches first | 19:41 |
| mriedem | artom: that does not blow my mind | 19:41 |
| artom | He's upstream! | 19:41 |
| sean-k-mooney | amodi: same as downstream | 19:41 |
| mriedem | amodi: the point of that is if we merged the rocky backport, someone got it, and then upgraded to train where the backport isn't merged yet, they'd lose the fix | 19:41 |
| mriedem | and have to re-fix it | 19:41 |
| mriedem | that's why you go latest to oldest | 19:41 |
| amodi | mriedem: i see, yes makes sense | 19:41 |
| melwitt | recently I struggled with locating where an exception was being raised to nova api, all the log said was "INFO [nova.api.openstack.wsgi] HTTP exception thrown: Maximum number of ports exceeded" I'm not sure whether more would be shown at DEBUG | 19:43 |
| melwitt | and for this 413 thing, you said there's zero logging of it.. I can't recall if I've seen that before or how/why it happens | 19:44 |
| artom | melwitt, well, there's just the 1 line | 19:44 |
| artom | And then the request ID appears in a bunch of OPTIONS / requests | 19:45 |
| melwitt | yeah, sorry, I meant other than the POST line with the status code | 19:45 |
| artom | Which I assume are load balancers poking the API | 19:45 |
| sean-k-mooney | artom: no the load blancer would not reuse a request id like that | 19:45 |
| artom | sean-k-mooney, this is queens, did we still have that request ID sharing bug back then? | 19:46 |
| artom | Though that was only for periodic tasks | 19:46 |
| openstackgerrit | Archit Modi proposed openstack/nova stable/train: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693241 | 19:46 |
| openstackgerrit | Archit Modi proposed openstack/nova stable/stein: Revert "openstack server create" to "nova boot" in nova docs https://review.opendev.org/693242 | 19:47 |
| sean-k-mooney | ha proxy is dumb and know nothing about the request id. iw we are seeing multipel request with the same request id its form heat/novaclient reusing the request id i woudl suspect | 19:47 |
| sean-k-mooney | its a vm create so it might use the same request id to poll for its completion | 19:48 |
| artom | sean-k-mooney, wait, so heat would be creating the whole stack with a single request ID? | 19:48 |
| artom | Fun | 19:48 |
| mriedem | nobody calls haproxy dumb and gets away with it | 19:48 |
| artom | sean-k-mooney, but... OPTIONS / ? | 19:48 |
| artom | That's obviously health check stuff | 19:48 |
| *** eharney has quit IRC | 19:48 | |
| artom | I've like we've gotten sidelined :) | 19:48 |
| sean-k-mooney | well a http options request is use to discover info about the endpoint | 19:49 |
| mriedem | heh, oh boy, someone just reply-all'ed to 400 people internally | 19:49 |
| sean-k-mooney | yes we may have internal maining lists that people dont understand how to unsubsribe form | 19:49 |
| melwitt | unsubsquirrel | 19:50 |
| sean-k-mooney | and they may or may not like to reply all to them to ask for help to the point ^ is now a thing | 19:50 |
| *** ivve has joined #openstack-nova | 19:51 | |
| sean-k-mooney | as is http://www.unsubsquirrel.com/ | 19:51 |
| artom | Oh common | 19:52 |
| mriedem | great now i'm playing that game | 19:52 |
| sean-k-mooney | https://scratch.mit.edu/projects/327361863/ is the beeter version | 19:52 |
| artom | I can't believe the obvious "Unsubscribe you? That's nuts!" pun | 19:52 |
| artom | *they missed | 19:52 |
| melwitt | open a PR | 19:53 |
| artom | Ah no, if you refresh enough times you get it | 19:54 |
| melwitt | oh yeah, I see https://github.com/joellord/subsquirrel/blob/master/index.html#L33 | 19:55 |
| *** eharney has joined #openstack-nova | 20:01 | |
| *** nanzha has joined #openstack-nova | 20:07 | |
| *** nanzha has quit IRC | 20:12 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Regression test for bug 1849657 https://review.opendev.org/693248 | 20:13 |
| openstack | bug 1849657 in OpenStack Compute (nova) " allocation key is missing from the binding:profile of the neutron qos port when the server is created by a non-admin user" [Medium,In progress] https://launchpad.net/bugs/1849657 - Assigned to Balazs Gibizer (balazs-gibizer) | 20:13 |
| *** xek__ has quit IRC | 20:17 | |
| *** xek_ has joined #openstack-nova | 20:17 | |
| *** ralonsoh has quit IRC | 20:19 | |
| *** abaindur has joined #openstack-nova | 20:20 | |
| *** abaindur has joined #openstack-nova | 20:20 | |
| *** efried_rollin is now known as efried | 20:24 | |
| efried | artom: paste? | 20:24 |
| efried | sean-k-mooney: So the one where I added --with-apparmor_profiles and --prefix=/usr failed, but the one where I added that symlink and rebooted succeeded. | 20:26 |
| sean-k-mooney | efried: ok so i need to work on proper apparmor support in a follow up | 20:27 |
| sean-k-mooney | i may have spent the last 3 hour fixing my openstack that i broke but im testing it on a clean vm right now | 20:27 |
| efried | sean-k-mooney: the symptoms are all the same, except dmesg in addition to those apparmor="DENIED" messages also shows things like | 20:28 |
| efried | [75800.085042] netlink: 'ovs-vswitchd': attribute type 5 has an invalid length. | 20:28 |
| efried | [75800.085251] device br-int entered promiscuous mode | 20:28 |
| *** priteau has quit IRC | 20:28 | |
| sean-k-mooney | the other message are normal | 20:28 |
| sean-k-mooney | but you still got the denie message with the profiles comipled in ya | 20:29 |
| efried | well, I'm taking it on faith that "profiles are compiled in". | 20:29 |
| efried | - ./autogen.sh --system | 20:30 |
| efried | + ./autogen.sh --system --with-apparmor_profiles | 20:30 |
| efried | and | 20:30 |
| efried | - ./configure | 20:30 |
| efried | + ./configure --prefix=/usr | 20:30 |
| sean-k-mooney | im not sure if you have to pass that flage to autogen or configure | 20:30 |
| efried | iiuc either one of those should have worked by itself, no? | 20:30 |
| *** tbachman has quit IRC | 20:30 | |
| sean-k-mooney | the profies are hardcoded to no | 20:30 |
| efried | because apparmor's snit was with libvirt not being in the right dir? | 20:30 |
| sean-k-mooney | so if you dont explcitly pass the flag they wont be generated | 20:31 |
| sean-k-mooney | it was unhapy with qemu not being in the correct location | 20:31 |
| sean-k-mooney | the --system puts libvirt in the correct place | 20:31 |
| efried | so if I do that symlink thing and reboot, I just have to unstack and restack, not clean, right? | 20:32 |
| sean-k-mooney | yep | 20:32 |
| sean-k-mooney | you dont need to modify the build either | 20:32 |
| sean-k-mooney | if you add the symlink that should be enough with the patch i have pushed up | 20:32 |
| *** eharney has quit IRC | 20:32 | |
| sean-k-mooney | and the local.conf i shared earlier | 20:33 |
| sean-k-mooney | atleast that is what im testing | 20:33 |
| efried | still at ps3 yah? | 20:33 |
| sean-k-mooney | yep | 20:33 |
| sean-k-mooney | i littally created the sysmlink rebooted, cloned devstack, and created the local.conf http://paste.openstack.org/show/785846/ | 20:34 |
| sean-k-mooney | and now im waiting for it to finish stacking | 20:34 |
| sean-k-mooney | i think that should be all that is required | 20:34 |
| *** ceryx has quit IRC | 20:36 | |
| efried | what's the incantation to create a VM I can actually ssh into (from the hypervisor, don't care about external stuff) | 20:41 |
| sean-k-mooney | i normally use openstack or virt-manager to do it | 20:42 |
| sean-k-mooney | what os do you want to do it on | 20:43 |
| efried | I mean create via openstack. --network public doesn't seem to be sufficient, or I'm missing something else. Console log shows the big cirros splash, so I think the thing is up. But trying the IPs I see from `server show` is getting me nothing. | 20:44 |
| sean-k-mooney | oh devsteack? | 20:44 |
| sean-k-mooney | if so the public network has dhcp disabled | 20:44 |
| sean-k-mooney | so you have to trun that on first | 20:45 |
| efried | or get to the console. I don't really even need networking for what I'm doing. | 20:45 |
| efried | (It's kind of embarrassing that I've basically never done this) | 20:45 |
| sean-k-mooney | you have never done a nova boot? | 20:45 |
| sean-k-mooney | or actully tried to ssh in after | 20:46 |
| efried | the latter | 20:46 |
| artom | efried, paste? | 20:47 |
| efried | artom: As in, I might be able to help you with your paste/middleware | 20:47 |
| efried | sean-k-mooney: never cared about the VM itself, just about the bits that get it to ACTIVE state. | 20:47 |
| sean-k-mooney | to enable dhcp do "openstack subnet set --dhcp public" | 20:49 |
| sean-k-mooney | you then need to open ssh in the security groups | 20:49 |
| sean-k-mooney | "openstack security group rule create --ingress --protocol tcp --remote-ip 0.0.0.0/0 default" | 20:51 |
| sean-k-mooney | if you reboot the vm you should then be able to ssh in with | 20:51 |
| sean-k-mooney | openstack server ssh --private -l cirros <vm name> | 20:52 |
| artom | efried, ah, thanks! Tomorrow? If it's still bugging me then | 20:52 |
| efried | artom: sure thing | 20:52 |
| artom | I'm on the hook for the school run now | 20:52 |
| efried | me too in 20 mins | 20:52 |
| * artom -> off | 20:52 | |
| *** artom has quit IRC | 20:52 | |
| eandersson | Fighting heal_allocations is never fun | 20:53 |
| eandersson | I almost wish it was out of tree | 20:53 |
| efried | sean-k-mooney: is it gonna hurt me that my VM currently doesn't think it has an ipv4 address? | 20:54 |
| sean-k-mooney | you dont see one in openstack server show our you cant ping it | 20:54 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use admin neutron client to query ports for binding https://review.opendev.org/690999 | 20:57 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use admin neutron client to gather port resource requests https://review.opendev.org/691005 | 20:57 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use admin neutron client to see if instance has qos ports https://review.opendev.org/691900 | 20:57 |
| sean-k-mooney | efried: if you just want the console url you can do "openstack console url <vm name>" by the way | 20:57 |
| efried | sean-k-mooney: per the console running ifconfig -a | 20:57 |
| sean-k-mooney | oh did you enable dhcp like i said and reboot | 20:58 |
| efried | yeah, the console via browser is pretty clunky, I could make do with that but would rather be able to ssh | 20:58 |
| efried | for one thing, it seems to be bypassing my keyboard driver | 20:58 |
| efried | I can't type qwerty to save my life | 20:58 |
| sean-k-mooney | davorak? | 20:58 |
| efried | there were three security groups, so I set that up for all of them, cause I didn't know which one was the right one. rebooting the vm now | 20:59 |
| efried | yes, dvorak | 20:59 |
| efried | for nearly 20 years. | 20:59 |
| sean-k-mooney | ... i see | 20:59 |
| sean-k-mooney | anyway am if your in the vm | 20:59 |
| efried | can't imagine how a console through a browser is using its own kb driver | 20:59 |
| mriedem | note that those 3 fixes above are gibi's and pretty straight-forward | 20:59 |
| efried | but whatever | 20:59 |
| mriedem | i'm +2 on all 3 | 20:59 |
| sean-k-mooney | efried: so anyway your in the vm right. have you enabled dhcp on the subnet? if so you can reboot and it should get an ip or you can run cirros dhcpclient directly | 21:01 |
| sean-k-mooney | efried: then you should be able to ssh in | 21:01 |
| efried | rebooting, waiting for it to come up | 21:01 |
| sean-k-mooney | efried: i usually put the command to do this in a local.sh file beside my local.conf | 21:02 |
| mriedem | efried: are you just trying to figure out how to ssh into a cirros guest in devstack? | 21:02 |
| sean-k-mooney | i just cant find it | 21:02 |
| mriedem | i've got a recording on youtube that does that with a floating ip | 21:02 |
| sean-k-mooney | devstack will auto run the local.sh if present | 21:02 |
| efried | I did | 21:02 |
| efried | openstack subnet set --dhcp public-subnet | 21:02 |
| efried | openstack security group rule create --ingress --protocol tcp --remote-ip 0.0.0.0/0 $group for all three groups | 21:02 |
| efried | openstack server reboot | 21:02 |
| efried | mriedem: yes, that's what I'm trying to do. | 21:02 |
| sean-k-mooney | yep that should be enough | 21:02 |
| mriedem | heh, it's been like 3 days :) | 21:02 |
| mriedem | sec | 21:02 |
| mriedem | efried: it's a bit old, but this shows setting up a guest in devstack with a floating ip to ssh into it https://www.youtube.com/watch?v=hZg6wqxdEHk | 21:03 |
| sean-k-mooney | ... my clean vm failed because i forget to mount /opt on a cinder volume and ran out of space :( | 21:03 |
| efried | thanks mriedem, will try that on my other devstack | 21:04 |
| efried | sean-k-mooney: | 21:04 |
| efried | stack@nucle:/opt/stack/devstack$ openstack server ssh --private -l cirros test1 | 21:04 |
| efried | ssh: connect to host 2001:db8::109 port 22: No route to host | 21:04 |
| sean-k-mooney | add -4 | 21:04 |
| mriedem | start about 3 minutes in | 21:04 |
| mriedem | ignore all of the volume multiattach stuff | 21:05 |
| sean-k-mooney | unless you have routable ipv6 | 21:05 |
| sean-k-mooney | if you do you need to change teh --remote-ip to allow ipv6 trafic | 21:05 |
| sean-k-mooney | efried: you cans also just use ssh directly if you know the ip | 21:05 |
| efried | ah, but I can ssh to the IP... yes | 21:06 |
| efried | I don't think I can ipv6 at all | 21:06 |
| ioni | hello guys, i'm using openstack in production and also sadly still use openvz and i plan to move away from it in the near future. is there a project that i can use with openstack to orchestrate containers like openvz. lxd seems the same but nova-lxd is not maintained anymore and everything that i can find related to openstack and containers is mostly related to docker and kubernets but not somethin | 21:06 |
| ioni | g like lxd/openvz | 21:06 |
| efried | adding -4 also made the openstack command work | 21:07 |
| sean-k-mooney | efried: the firewall is blocking all ipv6 into the vm by default | 21:07 |
| mriedem | ioni: zun | 21:07 |
| efried | sokay, I'm pretty sure ipv6 is never going to catch on anyway. I'll just use v4 | 21:07 |
| efried | thanks for the help, again. | 21:07 |
| sean-k-mooney | haha i spend like a full weekend getting it working at home like a month ago | 21:08 |
| sean-k-mooney | i dont have native ipv6 at home so had to set up tunneling and then route it into openstack so that my vms get public ipv6 adresses | 21:08 |
| mriedem | "how do i ssh into a guest in devstack" <3 days later> "wait, you don't need ipv6 with numa and dpdk? oh, well then..." | 21:08 |
| sean-k-mooney | :) | 21:09 |
| sean-k-mooney | mriedem: actully regarding multi attach | 21:09 |
| mriedem | stop | 21:09 |
| sean-k-mooney | what file system do you use? | 21:09 |
| mriedem | in this demo? | 21:10 |
| sean-k-mooney | in general | 21:10 |
| mriedem | i don't | 21:10 |
| sean-k-mooney | i dont think we mention it in the docs | 21:10 |
| sean-k-mooney | ok | 21:10 |
| mriedem | multiattach docs are mostly in cinder | 21:10 |
| sean-k-mooney | ah o should read those at some point | 21:10 |
| mriedem | https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiattach.html | 21:11 |
| mriedem | doesn't really go into details either | 21:11 |
| ioni | mriedem, zun seems that it uses docker and is not something that i can replace openvz with | 21:11 |
| sean-k-mooney | i was thinking of using it for k8s on openstack but it seamed complicated | 21:11 |
| mriedem | efried: so about 9 minutes into that demo i'm creating a floating ip for the port attached to the server | 21:11 |
| mriedem | might be useful if i just wrote this up into devstack docs | 21:13 |
| sean-k-mooney | mriedem: i think there used to be an exersie that coverd it | 21:13 |
| sean-k-mooney | but i cant find it | 21:13 |
| mriedem | ioni: nova isn't really big on containers, it's basically vms and a front end for ironic baremetal, | 21:14 |
| mriedem | nova-lxd is a thing out of tree as you know, openvz is in who knows what kind of maintenance state, and then there is libvirt+lxc which is also - who really knows if it works very well right now | 21:14 |
| mriedem | of course with cirros you can just login with a password | 21:15 |
| efried | sean-k-mooney: presumably I could have done all that dhcp and security group nonsense before creating the vm | 21:15 |
| sean-k-mooney | efried: yes and you only have to do it once | 21:15 |
| efried | k, writing myself a cheat sheet | 21:15 |
| sean-k-mooney | efried: also i normally do it all via horizon as its faster | 21:15 |
| efried | ew, gui | 21:16 |
| sean-k-mooney | the cli is faster for most things but security groups are a pain on the cli | 21:16 |
| efried | cheat sheet line for that: | 21:16 |
| efried | openstack security group list -f value -c ID | xargs -n1 openstack security group rule create --ingress --protocol tcp --remote-ip 0.0.0.0/0 | 21:16 |
| sean-k-mooney | efried: anyway as i said once you have the command you need to run. just put them in a file called local.sh and it will be run at the right time | 21:17 |
| sean-k-mooney | oh is in the default local.sh in samples/local.sh | 21:17 |
| sean-k-mooney | just copy that up one directory | 21:17 |
| efried | hm, I thought I did all the right things on my other setup, but I'm still getting no route to host. | 21:18 |
| efried | aaand gotta run get a kid | 21:18 |
| *** efried is now known as efried_afk | 21:19 | |
| melwitt | ioni: I see mention of openvz in the original proposal for the magnum project https://specs.openstack.org/openstack/magnum-specs/specs/pre-ocata/implemented/containers-service.html but I don't know if it supports it today | 21:21 |
| sean-k-mooney | melwitt: ioni magnum is a different usage model | 21:23 |
| sean-k-mooney | nova libvirt/lxc woudl be the closest | 21:23 |
| melwitt | oh ok | 21:24 |
| ioni | i just want to stop using openvz and migrate the current containers to something orchestrated by openstack with good support | 21:24 |
| ioni | nova-lxd was something i could use but that was dropped recently by canonical | 21:24 |
| sean-k-mooney | ioni: well there isnt really a good path forward if you also want the nova api | 21:24 |
| sean-k-mooney | zun will work with docker style container but not system contianers | 21:25 |
| sean-k-mooney | as far as i know | 21:25 |
| ioni | right, that is want i found as well | 21:25 |
| sean-k-mooney | nova libvirt wiht lxc could work but it does not have good support a the momemnt | 21:25 |
| sean-k-mooney | if there was a lot of interest in it it would not take that much to get it working well again | 21:25 |
| ioni | everybody told me that lxc support is not that great and was too scared to try it | 21:26 |
| sean-k-mooney | its not but its not that far form being ok | 21:26 |
| *** gshippey has quit IRC | 21:27 | |
| sean-k-mooney | the openvz driver is proably the best supproted fo all the nova conatiner drivers at the moment | 21:27 |
| sean-k-mooney | it has not reaaly been a focus of nova of late | 21:27 |
| *** pcaruana has quit IRC | 21:27 | |
| openstackgerrit | Merged openstack/python-novaclient master: doc: Fix supported version descriptions https://review.opendev.org/693097 | 21:29 |
| mriedem | i'm guessing the openvz 3rd party ci is still busted | 21:29 |
| *** damien_r has quit IRC | 21:30 | |
| sean-k-mooney | was that the ci that had a hardcoded patch that is now in conclict because we fixed the bug | 21:30 |
| mriedem | yes | 21:30 |
| mriedem | looks like it doesn't even report anymore | 21:31 |
| sean-k-mooney | ya i was just going to say that | 21:31 |
| sean-k-mooney | i have not seen it in a while | 21:31 |
| ioni | sean-k-mooney, alright thanks. i'll take a look at openvz 7 and nova | 21:31 |
| sean-k-mooney | ioni: if you need contianer i would not bet on that long term | 21:32 |
| ioni | sean-k-mooney, indeed, i mostly think that i'll do a container to full kvm migration, but that really sucks | 21:33 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Use ListOfUUIDField from oslo.versionedobjects https://review.opendev.org/693258 | 21:34 |
| sean-k-mooney | if you are interested in system contaienr there are afew other like donnyd who also share that interest | 21:34 |
| sean-k-mooney | so we could try and fix/test libvit lxc upstream but its currntly on lifesupprot/hobbyist support | 21:35 |
| sean-k-mooney | kvm will have less desitiny but its deffently the best tested solution | 21:36 |
| sean-k-mooney | night all o/ | 21:37 |
| ioni | thanks for your feedback sean-k-mooney | 21:37 |
| *** efried_afk is now known as efried | 21:41 | |
| *** takashin has joined #openstack-nova | 21:48 | |
| *** igordc has quit IRC | 21:51 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused 'nova-dsvm-base' job https://review.opendev.org/688389 | 22:01 |
| *** kisahm has joined #openstack-nova | 22:01 | |
| *** abaindur has quit IRC | 22:02 | |
| *** abaindur has joined #openstack-nova | 22:02 | |
| kisahm | HI together, i have the problem, that 'openstack availability zone list' receives no output. the error message is ConnectFailure: Unable to establish connection to https://nova-url:8774/v2.1/os-availability-zone/detail: ('Connection aborted.', BadStatusLine('No status line received - the server has closed the connection',)) | 22:03 |
| kisahm | nova api/placement shows no errors. | 22:03 |
| kisahm | the request timed out after 90 sec | 22:04 |
| kisahm | openstack queens release, deployed with Juju | 22:05 |
| *** xek has joined #openstack-nova | 22:05 | |
| *** xek_ has quit IRC | 22:06 | |
| *** xek has quit IRC | 22:06 | |
| *** xek has joined #openstack-nova | 22:07 | |
| mriedem | melwitt: dansmith: this isn't bad, but i'm not sure i want to pull the trigger on it - see if you have opinions https://review.opendev.org/#/c/688122/ | 22:07 |
| mriedem | i like the idea of not hitting the synchronized stuff if we have the entry in the cache | 22:07 |
| *** eharney has joined #openstack-nova | 22:10 | |
| melwitt | I'm pretty sure Kevin_Zheng proposed the same thing in the past in a simpler way and got nacked. lemme see if I can find | 22:10 |
| kisahm | the nova-api-os-compute.log shows this error: Timed out waiting for response from cell 79275f0e-86b1-4859-aeac-642382585ef4: CellTimeout: Timeout waiting for response from cell | 22:12 |
| mriedem | kisahm: known http://status.openstack.org/elastic-recheck/#1844929 | 22:12 |
| *** xek has quit IRC | 22:12 | |
| melwitt | mriedem: ^ https://review.opendev.org/#/c/595530/2/nova/context.py | 22:13 |
| mriedem | uh oh, do i see dansmith talking about eventlet + locks in there?! | 22:16 |
| mriedem | dropping that on booth might start a war | 22:16 |
| melwitt | hah | 22:16 |
| kisahm | mriedem does the bug also exists in openstack releases >queens? | 22:27 |
| *** xek has joined #openstack-nova | 22:28 | |
| mriedem | kisahm: logstash says it started in the upstream gate in train, but mostly master jobs | 22:28 |
| *** tesseract has quit IRC | 22:28 | |
| mriedem | do people see any issue with this? https://review.opendev.org/#/c/540962/ - if not i'll rev it to add the short release note | 22:30 |
| mriedem | tl;dr allow CONF.enable_new_services to be configured per-host rather than globally | 22:30 |
| mriedem | so you can start some computes as disabled to start but not others | 22:30 |
| *** TxGirlGeek has quit IRC | 22:31 | |
| *** TxGirlGeek has joined #openstack-nova | 22:32 | |
| efried | mriedem: so previously you would put that in your conductor conf and it would disable all computes? | 22:33 |
| mriedem | yup | 22:33 |
| mriedem | well, newly created computes | 22:33 |
| *** TxGirlGeek has joined #openstack-nova | 22:33 | |
| efried | right | 22:34 |
| efried | hm | 22:34 |
| efried | it's a pretty serious semantic change, really. | 22:35 |
| *** dklyle has quit IRC | 22:35 | |
| *** JamesBenson has quit IRC | 22:36 | |
| mriedem | as in, if you used different configs for your controllers and computes, and you had this set in your controllers but not computes, you might now have new enabled comptues where you'd expect them to be disabled b/c you didn't move the config? | 22:36 |
| *** dklyle has joined #openstack-nova | 22:36 | |
| efried | let me make sure I'm understanding: you would have the option set to False in your compute-side conf, and when you bootstrap your compute, it'll start out disabled; but once you re-enable it, it will stay enabled from that point on -- like, the conf opt will not have an effect on subsequent starts of the service | 22:36 |
| *** dklyle has quit IRC | 22:36 | |
| *** dklyle has joined #openstack-nova | 22:36 | |
| mriedem | it would be ignored if set in compute conf | 22:37 |
| mriedem | as the bug notes | 22:37 |
| efried | I mean in the new world | 22:37 |
| melwitt | yeah, I was thinking similar. I thought usually that sort of thing would have hit the ML. I can't think of any issue with it other than possible confusion, which the release note would address | 22:37 |
| mriedem | correct | 22:37 |
| mriedem | it's only when starting a new service and the record is created | 22:37 |
| efried | different levels of conductor and compute are supported though, right? | 22:38 |
| melwitt | I'd want to ask dansmith if there's any upgrade concerns I'm missing | 22:38 |
| efried | which one upgrades first? | 22:38 |
| mriedem | the whole point is, i want to deploy some new computes but not make them scheduable until i can kick their tires | 22:38 |
| mriedem | then add them to the pool | 22:38 |
| mriedem | efried: doesn't really have anything to do with rpc upgrade levels if that's what you mean | 22:38 |
| efried | no, I mean train conductor, ussuri compute. If the option is set differently, what happens? | 22:39 |
| mriedem | for an upgrade, if you care about this, you'd probably leave it to False in conductor until you upgrade all computes to the new code, then you can remove it from conductor config | 22:39 |
| mriedem | train conductor ussuri compute isn't really a thing | 22:39 |
| efried | okay, vice versa. | 22:39 |
| mriedem | if you had that for some reason, obviously the train conductor code would override | 22:39 |
| efried | "if you care about this" and you remembered that you set it years ago in your conductor conf | 22:39 |
| *** Kevin_Zheng has quit IRC | 22:40 | |
| efried | Have we considered instead creating a *new* opt that's compute-only? Because then I could have the conductor "default" and the per-compute "override". | 22:40 |
| mriedem | once everything is upgraded, having it set in your conductor conf would be a noop | 22:40 |
| efried | yeah, I get that. | 22:40 |
| mriedem | i mentioned moving the option to the [compute] group in a follow up | 22:40 |
| mriedem | i didn't think about 2 options | 22:41 |
| mriedem | i also don't know how much this is even used, so i don't know that we should boil the ocean over it | 22:42 |
| mriedem | but 2 options could be a transition plan | 22:42 |
| mriedem | DEFAULT.enable_new_services gets deprecated, compute.enable_new_services gets added | 22:43 |
| efried | on compute it makes more sense to be singular | 22:44 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve error log when snapshot fails https://review.opendev.org/356020 | 22:53 |
| *** rcernin has joined #openstack-nova | 22:53 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve error log when snapshot fails https://review.opendev.org/356020 | 22:54 |
| *** tbachman has joined #openstack-nova | 22:57 | |
| *** TxGirlGeek has quit IRC | 22:59 | |
| *** TxGirlGeek has joined #openstack-nova | 23:00 | |
| mriedem | stephenfin: remember that when you rebase a patch that contains a bug fix, the bug is automatically assigned to you in launchpad and it's probably good to go back into launchpad and re-assign it to the owner, | 23:06 |
| mriedem | looks like during your old review scan/cleanup you did a lot of rebases for old patches (like the one above) and now you own all of the bugs | 23:07 |
| *** threestrands has joined #openstack-nova | 23:07 | |
| *** artom has joined #openstack-nova | 23:07 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Regression test for bug 1849657 https://review.opendev.org/693248 | 23:22 |
| openstack | bug 1849657 in OpenStack Compute (nova) " allocation key is missing from the binding:profile of the neutron qos port when the server is created by a non-admin user" [Medium,In progress] https://launchpad.net/bugs/1849657 - Assigned to Balazs Gibizer (balazs-gibizer) | 23:22 |
| *** jmlowe has joined #openstack-nova | 23:29 | |
| *** mriedem has quit IRC | 23:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
