| *** panda has quit IRC | 00:02 | |
| *** panda has joined #openstack-nova | 00:05 | |
| *** dviroel has quit IRC | 00:08 | |
| *** tkajinam has joined #openstack-nova | 00:16 | |
| *** brinzhang_ has joined #openstack-nova | 00:41 | |
| *** brinzhang has quit IRC | 00:44 | |
| *** ricolin has quit IRC | 00:45 | |
| *** pots has quit IRC | 00:54 | |
| *** pots has joined #openstack-nova | 00:55 | |
| *** ricolin has joined #openstack-nova | 00:57 | |
| *** Liang__ has joined #openstack-nova | 01:03 | |
| *** nanzha has joined #openstack-nova | 01:07 | |
| openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | 01:16 |
|---|---|---|
| *** aloga has quit IRC | 01:24 | |
| *** brinzhang has joined #openstack-nova | 01:29 | |
| *** brinzhang_ has quit IRC | 01:32 | |
| *** nanzha has quit IRC | 01:39 | |
| *** nanzha has joined #openstack-nova | 01:39 | |
| *** dviroel has joined #openstack-nova | 01:45 | |
| *** Xuchu has joined #openstack-nova | 01:46 | |
| *** Dinesh_Bhor has quit IRC | 02:10 | |
| *** slaweq has joined #openstack-nova | 02:13 | |
| *** brinzhang has quit IRC | 02:13 | |
| *** brinzhang_ has joined #openstack-nova | 02:13 | |
| *** slaweq has quit IRC | 02:17 | |
| *** slaweq has joined #openstack-nova | 02:22 | |
| *** slaweq has quit IRC | 02:27 | |
| *** brinzhang has joined #openstack-nova | 02:28 | |
| *** brinzhang_ has quit IRC | 02:31 | |
| *** slaweq has joined #openstack-nova | 02:37 | |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: ksa auth conf and client for Cyborg access https://review.opendev.org/631242 | 02:41 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add Cyborg device profile groups to request spec. https://review.opendev.org/631243 | 02:41 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Create and bind Cyborg ARQs. https://review.opendev.org/631244 | 02:41 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Get resolved Cyborg ARQs and add PCI BDFs to VM's domain XML. https://review.opendev.org/631245 | 02:41 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Delete ARQs for an instance when the instance is deleted. https://review.opendev.org/673735 | 02:41 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova master: Add cyborg tempest job. https://review.opendev.org/670999 | 02:41 |
| *** slaweq has quit IRC | 02:52 | |
| *** yaawang has quit IRC | 02:55 | |
| *** yaawang has joined #openstack-nova | 02:56 | |
| *** slaweq has joined #openstack-nova | 02:57 | |
| *** brinzhang has quit IRC | 03:03 | |
| *** brinzhang has joined #openstack-nova | 03:04 | |
| *** brinzhang has quit IRC | 03:08 | |
| *** brinzhang has joined #openstack-nova | 03:09 | |
| *** brinzhang has quit IRC | 03:10 | |
| *** brinzhang has joined #openstack-nova | 03:10 | |
| *** slaweq has quit IRC | 03:12 | |
| *** mdbooth has quit IRC | 03:12 | |
| *** slaweq has joined #openstack-nova | 03:13 | |
| *** mdbooth has joined #openstack-nova | 03:14 | |
| openstackgerrit | Merged openstack/nova master: Make nova-next multinode and drop tempest-slow-py3 https://review.opendev.org/683988 | 03:14 |
| *** slaweq has quit IRC | 03:22 | |
| *** slaweq_ has joined #openstack-nova | 03:23 | |
| openstackgerrit | Qiu Fossen proposed openstack/nova-specs master: Support fuzzy querying instance by tag https://review.opendev.org/691651 | 03:24 |
| openstackgerrit | Sundar Nadathur proposed openstack/nova-specs master: Updated Nova-Cyborg interaction spec. https://review.opendev.org/684151 | 03:25 |
| *** slaweq_ has quit IRC | 03:27 | |
| *** slaweq_ has joined #openstack-nova | 03:32 | |
| *** belmoreira has quit IRC | 03:33 | |
| *** slaweq_ has quit IRC | 03:37 | |
| *** psachin has joined #openstack-nova | 03:37 | |
| *** slaweq_ has joined #openstack-nova | 03:38 | |
| *** slaweq_ has quit IRC | 03:47 | |
| *** slaweq_ has joined #openstack-nova | 03:48 | |
| *** mkrai has joined #openstack-nova | 03:48 | |
| *** nweinber__ has joined #openstack-nova | 03:52 | |
| *** slaweq_ has quit IRC | 03:57 | |
| *** dave-mccowan has quit IRC | 04:02 | |
| *** nweinber__ has quit IRC | 04:15 | |
| *** dviroel has quit IRC | 04:23 | |
| *** brinzhang_ has joined #openstack-nova | 04:25 | |
| *** brinzhang has quit IRC | 04:29 | |
| *** abaindur has joined #openstack-nova | 04:31 | |
| *** udesale has joined #openstack-nova | 04:35 | |
| *** brinzhang has joined #openstack-nova | 04:48 | |
| *** brinzhang has quit IRC | 04:49 | |
| *** brinzhang has joined #openstack-nova | 04:50 | |
| *** brinzhang_ has quit IRC | 04:51 | |
| *** brinzhang has quit IRC | 04:51 | |
| *** brinzhang has joined #openstack-nova | 04:51 | |
| *** abaindur has quit IRC | 04:52 | |
| *** brinzhang has quit IRC | 05:00 | |
| *** brinzhang has joined #openstack-nova | 05:00 | |
| *** janki has joined #openstack-nova | 05:17 | |
| *** brinzhang_ has joined #openstack-nova | 05:24 | |
| *** brinzhang has quit IRC | 05:28 | |
| *** brinzhang has joined #openstack-nova | 05:31 | |
| *** brinzhang_ has quit IRC | 05:34 | |
| *** brinzhang has quit IRC | 05:35 | |
| *** brinzhang has joined #openstack-nova | 05:35 | |
| *** larainema has joined #openstack-nova | 05:39 | |
| *** ratailor has joined #openstack-nova | 05:47 | |
| *** links has joined #openstack-nova | 05:51 | |
| *** ileixe has quit IRC | 05:56 | |
| *** ivve has joined #openstack-nova | 05:57 | |
| *** ileixe has joined #openstack-nova | 05:59 | |
| openstackgerrit | Andreas Jaeger proposed openstack/nova stable/train: Switch to opensuse-15 nodeset https://review.opendev.org/692032 | 06:04 |
| *** jawad_axd has quit IRC | 06:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack/nova stable/stein: Switch to opensuse-15 nodeset https://review.opendev.org/692033 | 06:07 |
| *** ivve has quit IRC | 06:15 | |
| *** Xuchu has quit IRC | 06:34 | |
| *** jawad_axd has joined #openstack-nova | 06:36 | |
| *** Xuchu has joined #openstack-nova | 06:36 | |
| *** igordc has quit IRC | 06:37 | |
| *** sridharg has joined #openstack-nova | 06:43 | |
| *** mkrai has quit IRC | 06:43 | |
| *** brinzhang has quit IRC | 06:47 | |
| *** luksky has joined #openstack-nova | 06:57 | |
| *** dpawlik has joined #openstack-nova | 06:58 | |
| *** dpawlik has quit IRC | 07:03 | |
| *** pcaruana has joined #openstack-nova | 07:09 | |
| *** ircuser-1 has quit IRC | 07:14 | |
| *** dpawlik has joined #openstack-nova | 07:18 | |
| *** mkrai has joined #openstack-nova | 07:20 | |
| *** Luzi has joined #openstack-nova | 07:22 | |
| *** jawad_axd has quit IRC | 07:32 | |
| *** lpetrut has joined #openstack-nova | 07:35 | |
| openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | 07:39 |
| *** jawad_axd has joined #openstack-nova | 07:42 | |
| *** sridharg has quit IRC | 07:44 | |
| *** tkajinam has quit IRC | 08:03 | |
| *** maciejjozefczyk has joined #openstack-nova | 08:12 | |
| *** mkrai has quit IRC | 08:13 | |
| *** ratailor has quit IRC | 08:16 | |
| *** ratailor has joined #openstack-nova | 08:18 | |
| *** damien_r has joined #openstack-nova | 08:20 | |
| *** jraju__ has joined #openstack-nova | 08:21 | |
| *** links has quit IRC | 08:21 | |
| *** jraju__ has quit IRC | 08:22 | |
| *** links has joined #openstack-nova | 08:26 | |
| *** slaweq_ has joined #openstack-nova | 08:30 | |
| *** janki has quit IRC | 08:46 | |
| *** jchhatbar has joined #openstack-nova | 08:47 | |
| alex_xu | gibi: good morning | 08:50 |
| gibi | alex_xu: good morning | 08:50 |
| alex_xu | gibi: just want to check the plan of project update | 08:51 |
| gibi | alex_xu: for me the slides looks OK | 08:51 |
| alex_xu | yes, I think it is ok also | 08:51 |
| alex_xu | do we need to expand the U release, I guess there are more specs | 08:51 |
| alex_xu | but can't sure which is for sure can be accepted | 08:52 |
| alex_xu | gibi: so who responsible for which pages? | 08:52 |
| gibi | alex_xu: let me check what are the freshly approved spec... | 08:53 |
| *** ivve has joined #openstack-nova | 08:53 | |
| gibi | alex_xu: I think we should only mention already approved specs | 08:53 |
| alex_xu | cool | 08:53 |
| gibi | alex_xu: regarding spliting the work. I'm OK if you or stephenfin talks about the generic part of the beginning. I can talk about couple of features on slides 5 - 9. I think we can note who talk about what in the speaker notes | 08:55 |
| *** jangutter has joined #openstack-nova | 08:56 | |
| alex_xu | gibi: ok, cool, I guess all we familar with different features | 08:56 |
| gibi | alex_xu: yes definitly | 08:56 |
| alex_xu | gibi: looks like stephenfin is on vacation already, probably we need to catch him at Shanghai :) | 08:57 |
| gibi | alex_xu: I've just checked we can less specs approved than what is already mentioned on the Ussuri slides so I think we dont need to extend that | 08:58 |
| *** rpittau|afk is now known as rpittau | 08:58 | |
| *** ralonsoh has joined #openstack-nova | 08:58 | |
| gibi | alex_xu: yes stephenfin is already on vacation. I will travell on Friday. Lets try to sit together at some point during Monday next week | 08:59 |
| alex_xu | gibi: ok, no problem. the features in slides looks like a feature begin from previous release, and looks like more ensure to be accepted | 08:59 |
| gibi | alex_xu: I will add some speaker notes now about what I can talk about | 08:59 |
| alex_xu | gibi: cool | 09:00 |
| *** dtantsur|afk is now known as dtantsur | 09:01 | |
| *** vesper has joined #openstack-nova | 09:08 | |
| *** vesper11 has quit IRC | 09:09 | |
| *** slaweq_ is now known as slaweq | 09:18 | |
| *** trident has quit IRC | 09:26 | |
| *** dpawlik has quit IRC | 09:33 | |
| alex_xu | gibi: it is time to assign everything else to Stephen :) | 09:34 |
| gibi | alex_xu: yeah. I guess that is the default behavior :) | 09:34 |
| *** trident has joined #openstack-nova | 09:34 | |
| alex_xu | hah | 09:34 |
| gibi | alex_xu: I will drop a mail to you and stephen to summarize what we talked here now, and ask stephenfin about a sitdown on Monday | 09:35 |
| alex_xu | gibi: cool, thanks | 09:35 |
| *** Liang__ has quit IRC | 09:36 | |
| *** elod has quit IRC | 09:37 | |
| *** ratailor_ has joined #openstack-nova | 09:38 | |
| *** tesseract has joined #openstack-nova | 09:38 | |
| *** ratailor has quit IRC | 09:41 | |
| *** dpawlik has joined #openstack-nova | 09:48 | |
| *** cgoncalves has quit IRC | 09:49 | |
| *** nanzha has quit IRC | 09:51 | |
| *** Xuchu has quit IRC | 09:55 | |
| *** elod has joined #openstack-nova | 09:57 | |
| *** cgoncalves has joined #openstack-nova | 09:58 | |
| *** nanzha has joined #openstack-nova | 10:02 | |
| *** aloga has joined #openstack-nova | 10:22 | |
| *** gryf has quit IRC | 10:27 | |
| *** rcernin has quit IRC | 10:27 | |
| *** gryf has joined #openstack-nova | 10:29 | |
| *** tbachman has quit IRC | 10:34 | |
| *** pcaruana has quit IRC | 10:35 | |
| *** tbachman has joined #openstack-nova | 10:38 | |
| *** openstackstatus has quit IRC | 10:44 | |
| *** mgoddard has quit IRC | 10:46 | |
| *** tbachman has quit IRC | 10:46 | |
| *** mgoddard has joined #openstack-nova | 10:47 | |
| *** sulo has joined #openstack-nova | 10:55 | |
| *** sulo has left #openstack-nova | 10:56 | |
| *** panda is now known as panda|pto | 11:00 | |
| *** arxcruz is now known as arxcruz|lunch | 11:10 | |
| *** jchhatbar has quit IRC | 11:14 | |
| openstackgerrit | Silvan Kaiser proposed openstack/nova master: Move Nova Quobyte driver to LibvirtMountedFileSystemVolumeDriver https://review.opendev.org/687066 | 11:19 |
| *** brinzhang has joined #openstack-nova | 11:34 | |
| *** dpawlik has quit IRC | 11:41 | |
| *** dviroel has joined #openstack-nova | 11:51 | |
| *** ratailor_ has quit IRC | 11:56 | |
| *** brinzhang_ has joined #openstack-nova | 11:57 | |
| *** pcaruana has joined #openstack-nova | 12:00 | |
| *** brinzhang has quit IRC | 12:01 | |
| *** brinzhang has joined #openstack-nova | 12:01 | |
| *** brinzhang_ has quit IRC | 12:02 | |
| *** rouk has quit IRC | 12:02 | |
| *** brinzhang has quit IRC | 12:03 | |
| *** brinzhang has joined #openstack-nova | 12:03 | |
| *** dpawlik has joined #openstack-nova | 12:08 | |
| *** larainema has quit IRC | 12:09 | |
| *** tbachman has joined #openstack-nova | 12:09 | |
| *** dpawlik has quit IRC | 12:12 | |
| *** arxcruz|lunch is now known as arxcruz | 12:18 | |
| *** AdamMork has joined #openstack-nova | 12:18 | |
| *** jawad_axd has quit IRC | 12:24 | |
| *** jawad_axd has joined #openstack-nova | 12:24 | |
| *** jawad_axd has quit IRC | 12:25 | |
| *** markvoelker has quit IRC | 12:25 | |
| *** markvoelker has joined #openstack-nova | 12:25 | |
| *** jawad_axd has joined #openstack-nova | 12:26 | |
| *** AJaeger has joined #openstack-nova | 12:26 | |
| *** udesale has quit IRC | 12:26 | |
| AJaeger | nova stable cores, please review https://review.opendev.org/692032 to update train opensuse job so that Infra can retire openSUSE 15.0 | 12:27 |
| *** brinzhang_ has joined #openstack-nova | 12:28 | |
| *** jawad_ax_ has joined #openstack-nova | 12:28 | |
| *** brinzhang has quit IRC | 12:30 | |
| *** jawad_axd has quit IRC | 12:31 | |
| *** jawad_ax_ has quit IRC | 12:33 | |
| *** nweinber__ has joined #openstack-nova | 12:35 | |
| *** jmlowe has quit IRC | 12:36 | |
| *** Sundar has joined #openstack-nova | 12:36 | |
| *** mvkr has quit IRC | 12:38 | |
| *** brinzhang has joined #openstack-nova | 12:38 | |
| *** brinzhang_ has quit IRC | 12:41 | |
| *** READ10 has joined #openstack-nova | 12:50 | |
| *** jawad_axd has joined #openstack-nova | 12:51 | |
| *** jawad_axd has quit IRC | 12:53 | |
| *** jawad_ax_ has joined #openstack-nova | 12:53 | |
| *** jawad_axd has joined #openstack-nova | 12:54 | |
| *** jawad_ax_ has quit IRC | 12:57 | |
| *** eharney has quit IRC | 12:57 | |
| *** jawad_axd has quit IRC | 13:00 | |
| *** jawad_axd has joined #openstack-nova | 13:01 | |
| *** jawad_axd has quit IRC | 13:01 | |
| *** jawad_axd has joined #openstack-nova | 13:02 | |
| *** psachin has quit IRC | 13:02 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support re-configure deleted_on_termination in server https://review.opendev.org/580336 | 13:10 |
| *** jmlowe has joined #openstack-nova | 13:11 | |
| *** mvkr has joined #openstack-nova | 13:12 | |
| *** xek has joined #openstack-nova | 13:13 | |
| *** luksky has quit IRC | 13:19 | |
| *** ygk_12345 has joined #openstack-nova | 13:20 | |
| *** tbachman has quit IRC | 13:20 | |
| *** mriedem has joined #openstack-nova | 13:22 | |
| *** tbachman has joined #openstack-nova | 13:23 | |
| *** brinzhang_ has joined #openstack-nova | 13:27 | |
| *** brinzhang_ has joined #openstack-nova | 13:28 | |
| *** brinzhang has quit IRC | 13:30 | |
| *** READ10 has quit IRC | 13:30 | |
| mriedem | smcginnis: queens-em tag patch is up https://review.opendev.org/#/c/692142/ | 13:46 |
| mriedem | elod: ^ | 13:46 |
| *** artom has quit IRC | 13:48 | |
| *** tbachman has quit IRC | 13:49 | |
| smcginnis | Thanks mriedem | 13:50 |
| smcginnis | Did that last one make it, or still stuck in recheck hell? | 13:50 |
| *** tbachman has joined #openstack-nova | 13:51 | |
| *** luksky has joined #openstack-nova | 13:52 | |
| *** dpawlik has joined #openstack-nova | 13:54 | |
| mriedem | merged late last night | 13:56 |
| smcginnis | Nice! | 13:58 |
| *** dave-mccowan has joined #openstack-nova | 14:01 | |
| *** eharney has joined #openstack-nova | 14:03 | |
| *** tbachman has quit IRC | 14:04 | |
| elod | mriedem: thanks, i'm a bit sad that gibi's patch sets couldn't make it, but of course, those are not crucial to be part of the final release o:) | 14:06 |
| *** tssurya has joined #openstack-nova | 14:06 | |
| elod | so, the final and em patch looks good, thanks! | 14:06 |
| *** tbachman has joined #openstack-nova | 14:07 | |
| *** tbachman has quit IRC | 14:20 | |
| *** jaosorior has joined #openstack-nova | 14:21 | |
| *** tbachman has joined #openstack-nova | 14:21 | |
| openstackgerrit | Merged openstack/nova master: Log some stats for image pre-cache https://review.opendev.org/688173 | 14:26 |
| *** mlavalle has joined #openstack-nova | 14:29 | |
| *** ganso has quit IRC | 14:29 | |
| *** dasp has quit IRC | 14:32 | |
| *** artom has joined #openstack-nova | 14:34 | |
| *** ygk_12345 has quit IRC | 14:35 | |
| mriedem | someone want to take a pass at mel's host_status UNKNOWN policy change patch? i'm +2 on it. https://review.opendev.org/#/c/679181/ | 14:39 |
| mriedem | same with gibi's evacuate support for qos ports series https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports-ussuri+status:open | 14:39 |
| mriedem | dansmith: when you get a chance it'd be cool if you could take a pass at my re-worked fix for the cross_az_attach=False bug fix https://review.opendev.org/#/c/469675/ | 14:40 |
| mriedem | should be 35% less gross than before | 14:41 |
| dansmith | mriedem: enqueued | 14:41 |
| dansmith | 35.0%? pretty precise grossness measurement | 14:41 |
| mriedem | i was in the lab all last night | 14:41 |
| dansmith | heh | 14:41 |
| *** dasp has joined #openstack-nova | 14:42 | |
| efried | mriedem: I was working through that | 14:42 |
| *** links has quit IRC | 14:42 | |
| efried | I have some questions, so dansmith if you wouldn't mind holding off approving it... | 14:42 |
| mriedem | efried: which? the az one? | 14:43 |
| efried | sorry, the host_status-unknown one | 14:43 |
| mriedem | oh | 14:43 |
| mriedem | dan and i are talking about the az one | 14:43 |
| efried | ack | 14:43 |
| sean-k-mooney | dansmith: thanks for reviewing the image metadta prefilter series. ill try and adress your comments this week | 14:55 |
| dansmith | sean-k-mooney: np | 14:56 |
| mriedem | easy multi-cell functional test-only change that's been sitting a long time https://review.opendev.org/#/c/452006/ | 14:56 |
| mriedem | 2.5 years... | 14:56 |
| dansmith | let's not rush that one | 14:57 |
| *** JamesBenson has joined #openstack-nova | 15:00 | |
| efried | mriedem, melwitt: I'm +2 on the host_status UNKNOWN policy patch, but would like some help understanding a thing before +Wing. (No hurry) | 15:00 |
| *** xek_ has joined #openstack-nova | 15:03 | |
| efried | mriedem: how come those other release patches are stacked on the queens-em one? | 15:03 |
| dansmith | mriedem: for N volumes your patch also avoids making N volume.get() calls instead opting for the bulk query once yeah? | 15:03 |
| mriedem | no particular reason, laziness | 15:03 |
| efried | ight | 15:04 |
| mriedem | dansmith: yeah - which is where i mentioned i could split that part out if it helps | 15:04 |
| dansmith | oh wait | 15:04 |
| dansmith | it actually just does volume.get() on each one in _get_volumes_for_bdms() | 15:05 |
| *** xek has quit IRC | 15:05 | |
| dansmith | I was thinking this was an improvement, but it just moves where you make all the individual calls | 15:05 |
| *** eharney has quit IRC | 15:05 | |
| *** jaosorior has quit IRC | 15:06 | |
| *** READ10 has joined #openstack-nova | 15:06 | |
| *** efried1 has joined #openstack-nova | 15:07 | |
| mriedem | efried: i replied in https://review.opendev.org/#/c/679181/ | 15:07 |
| *** efried has quit IRC | 15:08 | |
| *** efried1 is now known as efried | 15:08 | |
| mriedem | dansmith: yeah, i mentioned using GET /volumes with a list but as commented i don't think that would work and i should remove that comment | 15:08 |
| *** READ10 has quit IRC | 15:08 | |
| dansmith | yeah, sorry, just pulling all this context back in | 15:08 |
| mriedem | it does mean we'd avoid multiple GETs on the same volume, but you can't really boot from volume with multiple servers and the same pre-existing volume anyway | 15:09 |
| mriedem | unless it's a multiattach volume anyway | 15:09 |
| melwitt | efried: let me fix it, I think I just spaced bc gmann added those two in a patch below mine | 15:10 |
| mriedem | melwitt: do it in a follow up | 15:10 |
| mriedem | it's hard enough getting things through the gate right now | 15:11 |
| melwitt | ok | 15:11 |
| efried | I just want to understand why the tests work | 15:11 |
| efried | still reading mriedem's reply... | 15:11 |
| mriedem | efried: the policy docs are just an omission | 15:11 |
| mriedem | note that those are just docs...not code | 15:11 |
| efried | oh, that's just a doc thing? (/me doesn't really understand policy) | 15:11 |
| efried | got it. | 15:11 |
| mriedem | right, the policy rule is real, the API method / route stuff is docs | 15:11 |
| *** Sundar has quit IRC | 15:12 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.opendev.org/452006 | 15:12 |
| mriedem | ah gdi, fake_nodes | 15:13 |
| efried | melwitt, mriedem: +W. If you want to hit any of that other stuff in the fup too, feel free, but none of it was super important. | 15:13 |
| melwitt | ok, thanks | 15:14 |
| gibi | mriedem: fix is on the way, and then I will +2 it | 15:14 |
| efried | melwitt: also, wouldn't hurt to update the bp text for at least these two things: | 15:14 |
| efried | - the name of the policy rule | 15:14 |
| efried | - the fact that the host_status field is omitted, not included as "" | 15:14 |
| efried | (there turned out to be no spec for this, right?) | 15:15 |
| melwitt | ah, right, will do | 15:15 |
| efried | thx | 15:15 |
| melwitt | yeah no spec | 15:15 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add functional test for two-cell scheduler behaviors https://review.opendev.org/452006 | 15:15 |
| *** dpawlik has quit IRC | 15:16 | |
| dansmith | mriedem: question for you in there, and figure I'm standing by for you to remove that comment and split the patches if you're going to | 15:18 |
| *** eharney has joined #openstack-nova | 15:19 | |
| *** elod has quit IRC | 15:19 | |
| *** jawad_axd has quit IRC | 15:19 | |
| mriedem | i'll split the patches if needed but not keen to if i can help it, at least for the volume GET stuff since that touches a lot of unit tests. the config option docs and such are easy to split out though, but also minor. | 15:19 |
| dansmith | mriedem: yep I figured, hence my comment about it :) | 15:19 |
| *** elod has joined #openstack-nova | 15:19 | |
| dansmith | it's easy enough to see what test changes are related to the refactor so I'm not too concerned | 15:20 |
| dansmith | mriedem: are you planning to backport this? | 15:20 |
| *** JamesBen_ has joined #openstack-nova | 15:21 | |
| *** lpetrut has quit IRC | 15:22 | |
| *** ganso has joined #openstack-nova | 15:23 | |
| *** JamesBenson has quit IRC | 15:23 | |
| *** gyee has joined #openstack-nova | 15:24 | |
| *** Luzi has quit IRC | 15:25 | |
| mriedem | dansmith: replied, it's about pinning the instance to the default 'nova' zone which is a no-no | 15:26 |
| mriedem | dansmith: backporting it would probably be difficult, at least past train | 15:27 |
| *** brinzhang_ has quit IRC | 15:27 | |
| mriedem | i guess it depends on how much people/distros need it, but it's extremely latent and from talking to smorrison about this he says his users just have to always specify an az | 15:28 |
| dansmith | mriedem: okay, I didn't think about the cinder default being 'nova' | 15:28 |
| dansmith | mriedem: sure, was just curious | 15:28 |
| *** brinzhang_ has joined #openstack-nova | 15:28 | |
| *** brinzhang_ has quit IRC | 15:29 | |
| *** brinzhang_ has joined #openstack-nova | 15:29 | |
| mriedem | actually it won't even backport cleanly to train b/c of https://review.opendev.org/#/c/667133/ | 15:29 |
| *** brinzhang_ has quit IRC | 15:29 | |
| mriedem | that's pretty trivial to resolve though | 15:29 |
| *** brinzhang_ has joined #openstack-nova | 15:30 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add support matrix for Delete (Abort) on-going live migration https://review.opendev.org/625781 | 15:33 |
| *** brinzhang_ has quit IRC | 15:34 | |
| artom | mriedem, can https://review.opendev.org/#/c/649419/ be expected to merge? I want to know if I can backport it internally to Newton | 15:35 |
| dansmith | mriedem: yeah, it's cool, I just kinda thought you'd be wanting that, is all | 15:35 |
| artom | Or should I stick to the merged pike version | 15:35 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Cleanup libvirt test_mount unit tests https://review.opendev.org/692173 | 15:37 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Allow alternate implementations of mount/umount in _HostMountState https://review.opendev.org/692174 | 15:37 |
| mriedem | artom: ask your stable core brethren | 15:38 |
| artom | lyarwood, around? plz to be lookink at https://review.opendev.org/#/c/649419/ :) | 15:39 |
| mriedem | dansmith: i'm cool with not backporting it and letting it bake on master. if let's say some edge distro product thing needed it then i could be convinced to backport... | 15:39 |
| artom | lyarwood, it would facilitate the backport for https://bugzilla.redhat.com/show_bug.cgi?id=1633909, which we want for OSP10z14 | 15:39 |
| openstack | artom: Error: Error getting bugzilla.redhat.com bug #1633909: NotPermitted | 15:39 |
| dansmith | mriedem: heh | 15:39 |
| artom | openstack, apologies | 15:39 |
| artom | Ah, he took the whole day off as PTO | 15:40 |
| mriedem | there are others | 15:40 |
| mriedem | there is one in france always saying he's happy to review backports | 15:40 |
| *** jmlowe has quit IRC | 15:41 | |
| artom | He's on PTO as well | 15:41 |
| mriedem | jesus | 15:42 |
| dansmith | artom: I can hit that if you want | 15:42 |
| artom | dansmith, thank you :) | 15:42 |
| * melwitt slips under the radar | 15:42 | |
| dansmith | seems reasonable, it's a nasty issue, if we still care about it | 15:42 |
| artom | melwitt, you were next on the list! But dansmith preempted tht | 15:42 |
| artom | *that | 15:42 |
| melwitt | woohoo | 15:42 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Only allow one scheduler service in tests https://review.opendev.org/682486 | 15:42 |
| dansmith | artom: I just couldn't stand to see where it was going with mriedem calling out all the team | 15:44 |
| artom | dansmith, ... it was going to melwitt :P | 15:44 |
| dansmith | artom: right, who I figured was stranded with no services in the third world country we call California | 15:45 |
| artom | And on fire. | 15:45 |
| melwitt | haha | 15:46 |
| mriedem | you can always bug tonyb https://review.opendev.org/#/admin/groups/540,members | 15:46 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Only allow one scheduler service in tests https://review.opendev.org/682486 | 15:47 |
| melwitt | I have learned to greatly appreciate when I have internet access | 15:48 |
| mriedem | my grandpappy always said internet is a privilege not a right | 15:48 |
| mriedem | you know this has to be done: https://southpark.cc.com/clips/x2wsii/is-there-internet-here | 15:49 |
| melwitt | lol | 15:51 |
| *** jaosorior has joined #openstack-nova | 15:51 | |
| dansmith | that's pretty much how I picture california in my head | 15:51 |
| *** luksky has quit IRC | 15:52 | |
| melwitt | that was me driving to different coffee shops and libraries looking for wifi. first three places I tried didn't have internet | 15:54 |
| melwitt | "is there internet here??" | 15:55 |
| artom | melwitt, actually, you might still be on the hook for https://review.opendev.org/#/c/649419/ and the patch above it, looks like dansmith didn't +W | 15:55 |
| artom | Two RH +2s for a non-RH backport is cool, right? | 15:55 |
| melwitt | yeah | 15:55 |
| dansmith | artom: yeah I figured if she was around she'd be hitting it after I'm done | 15:55 |
| artom | I misled her by explicitly told her you'd preempted her | 15:56 |
| artom | Mea culpa | 15:56 |
| melwitt | :) | 15:56 |
| *** mkrai has joined #openstack-nova | 15:56 | |
| dansmith | mriedem: aren't you respinning the az one to remove the bulk comment? that's why I didn't +2 | 16:02 |
| *** davee_ has joined #openstack-nova | 16:03 | |
| mriedem | dansmith: yeah i will, was busy abandoning old patches from my dashboard | 16:08 |
| *** ivve has quit IRC | 16:10 | |
| *** nanzha has quit IRC | 16:12 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Reset vm_state to original value if rebuild claim fails https://review.opendev.org/692185 | 16:15 |
| melwitt | mriedem: do you remember how is REQUIRES_LOCKING related to NeutronFixture? I couldn't find a note about it in nova/test.py https://review.opendev.org/649419 | 16:19 |
| mriedem | melwitt: i think it's related to https://review.opendev.org/#/c/649385/ and how if you're using nova-net you have to require locking b/c of the network manager | 16:21 |
| *** igordc has joined #openstack-nova | 16:21 | |
| mriedem | but it's been a long time since i did that backport | 16:21 |
| melwitt | ok, was just curious | 16:21 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Default AZ for instance if cross_az_attach=False and checking from API https://review.opendev.org/469675 | 16:22 |
| mriedem | dansmith: ^ | 16:22 |
| *** mvkr has quit IRC | 16:31 | |
| *** ricolin has quit IRC | 16:41 | |
| *** jawad_axd has joined #openstack-nova | 16:44 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for add/remove instance info from node https://review.opendev.org/659691 | 16:46 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use SDK for getting network metadata from node https://review.opendev.org/670213 | 16:46 |
| *** jawad_axd has quit IRC | 16:49 | |
| *** links has joined #openstack-nova | 16:50 | |
| *** links has quit IRC | 16:50 | |
| *** damien_r has quit IRC | 16:51 | |
| AJaeger | nova stable cores, please review https://review.opendev.org/692032 to update train opensuse job so that Infra can retire openSUSE 15.0 | 16:57 |
| *** rpittau is now known as rpittau|afk | 16:59 | |
| *** mdbooth has quit IRC | 16:59 | |
| *** openstackstatus has joined #openstack-nova | 17:03 | |
| *** ChanServ sets mode: +v openstackstatus | 17:03 | |
| *** jmlowe has joined #openstack-nova | 17:05 | |
| mriedem | done | 17:09 |
| *** tssurya has quit IRC | 17:12 | |
| *** mdbooth has joined #openstack-nova | 17:13 | |
| *** dlbewley has joined #openstack-nova | 17:15 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Reset vm_state to original value if rebuild claim fails https://review.opendev.org/692185 | 17:17 |
| artom | melwitt, dansmith thanks you folks for the backport reviews! | 17:18 |
| *** mvkr has joined #openstack-nova | 17:21 | |
| *** JamesBen_ has quit IRC | 17:23 | |
| *** dpawlik has joined #openstack-nova | 17:23 | |
| *** JamesBenson has joined #openstack-nova | 17:24 | |
| *** Garyx_ has quit IRC | 17:25 | |
| *** xek_ has quit IRC | 17:29 | |
| *** mdbooth has quit IRC | 17:33 | |
| *** mdbooth has joined #openstack-nova | 17:34 | |
| *** dtantsur is now known as dtantsur|afk | 17:49 | |
| *** Garyx has joined #openstack-nova | 17:51 | |
| *** mlavalle has quit IRC | 17:53 | |
| *** macz has joined #openstack-nova | 18:04 | |
| mriedem | dansmith: i think i found a justification for https://review.opendev.org/#/c/669545/ now - https://bugs.launchpad.net/nova/+bug/1850682 | 18:09 |
| openstack | Launchpad bug 1850682 in OpenStack Compute (nova) "functional tests in rocky randomly fail with "Build of instance was re-scheduled: Cannot modify readonly field uuid"" [Undecided,New] | 18:09 |
| dansmith | aight | 18:10 |
| *** pcaruana has quit IRC | 18:10 | |
| *** nweinber__ has quit IRC | 18:10 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Nova compute: add in log exception to help debug failures https://review.opendev.org/669545 | 18:10 |
| *** eharney has quit IRC | 18:14 | |
| *** mkrai has quit IRC | 18:14 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Use annotated ddt for test_cpu_policy_constraint https://review.opendev.org/692205 | 18:18 |
| * artom can't seem to POST a bug to launchpad | 18:18 | |
| artom | Some sort of timeout | 18:19 |
| artom | And their stupid interface means hitting the back button loses all you bug text | 18:19 |
| artom | It's still saved by firefox as part of the POST request though, so if I just keep hitting refresh and re-sending... | 18:19 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Avoid error 500 on shelve task_state race https://review.opendev.org/692206 | 18:22 |
| artom | There we go | 18:24 |
| artom | mdbooth, sean-k-mooney ^^ keep me honest | 18:24 |
| artom | There was a definite lack of hairy tentacles | 18:24 |
| artom | Did I miss something? | 18:24 |
| *** tbachman has quit IRC | 18:25 | |
| *** luksky has joined #openstack-nova | 18:26 | |
| *** dpawlik has quit IRC | 18:26 | |
| mriedem | artom: yeah just refresh on lp until it goes through | 18:30 |
| AJaeger | thanks, mriedem | 18:31 |
| *** tbachman has joined #openstack-nova | 18:34 | |
| mriedem | artom: question on your patch | 18:34 |
| *** dpawlik has joined #openstack-nova | 18:37 | |
| *** AdamMork has quit IRC | 18:40 | |
| artom | mriedem, cheers, replied | 18:41 |
| artom | (To one of the things, at any rate) | 18:43 |
| mriedem | yeah, this is just whack a mole | 18:44 |
| mriedem | which i'm not really against but | 18:44 |
| mriedem | we could also just have a generic decorator in the api methods that handles this exception and returns HTTPConflict so we don't have to do the dance in the wsgi controller route handler code | 18:45 |
| artom | Yep | 18:45 |
| artom | I wasn't aware of the widespreaded-ness of this | 18:45 |
| artom | Such a decorator would also remain backportable | 18:45 |
| mriedem | yeah and no maybe, | 18:46 |
| mriedem | not all route handlers have 409 in their @wsgi.expected_errors((404, 409)) decorator | 18:46 |
| mriedem | i don't know how much that matters honestly - i wouldn't be surprised if we leak things through in some cases | 18:47 |
| artom | We could always do the wait and see thing | 18:47 |
| artom | Wack this mole, and if another mole pops up, think about a more generic appraoch | 18:47 |
| mriedem | that's what i'd probably do | 18:48 |
| mriedem | no point overengineering this now | 18:48 |
| artom | Ack | 18:48 |
| * artom continues whacking the mole | 18:48 | |
| *** mlavalle has joined #openstack-nova | 18:53 | |
| umbSublime | Is there any reference documentation to help fill up a new blueprint (to avoid wasting peoples time) | 19:00 |
| *** abaindur has joined #openstack-nova | 19:01 | |
| mriedem | umbSublime: https://docs.openstack.org/nova/latest/contributor/blueprints.html | 19:03 |
| *** tesseract has quit IRC | 19:03 | |
| *** eharney has joined #openstack-nova | 19:03 | |
| *** pcaruana has joined #openstack-nova | 19:05 | |
| *** ivve has joined #openstack-nova | 19:06 | |
| efried | mriedem: added comment as requested to https://review.opendev.org/#/c/682486/ and gibi is +2, wanna send it? | 19:14 |
| mriedem | hmmm, maybe | 19:15 |
| dansmith | cah-ripes | 19:15 |
| dansmith | change failed in the gate and rechecked 4.5h ago, still hasn't started a single job. | 19:15 |
| mriedem | the gate is not happy | 19:16 |
| mriedem | buttloads of ssh fails in guests for different reasons, | 19:16 |
| *** dpawlik has quit IRC | 19:16 | |
| mriedem | the timed out talking to cell db thing, | 19:16 |
| mriedem | https read timeout errors | 19:16 |
| artom | b'Expected: save(expected_tast_state=[None])' | 19:16 |
| artom | b'Actual: save(expected_task_state=[None])' | 19:16 |
| artom | *sigh* | 19:16 |
| artom | Oh! | 19:16 |
| * artom fails | 19:16 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova master: Avoid error 500 on shelve task_state race https://review.opendev.org/692206 | 19:19 |
| * artom -> school run | 19:21 | |
| *** artom has quit IRC | 19:21 | |
| *** jaosorior has quit IRC | 19:21 | |
| *** abaindur has quit IRC | 19:21 | |
| *** abaindur has joined #openstack-nova | 19:21 | |
| *** tbachman has quit IRC | 19:23 | |
| openstackgerrit | Merged openstack/nova master: api-ref: remove mention about os-migrations no longer being extended https://review.opendev.org/682102 | 19:28 |
| *** pcaruana has quit IRC | 19:29 | |
| *** jawad_axd has joined #openstack-nova | 19:29 | |
| efried | mriedem: are there other projects that are running tempest-slow that could stop doing that, similar to https://review.opendev.org/683988 ? | 19:32 |
| *** jawad_axd has quit IRC | 19:34 | |
| *** READ10 has joined #openstack-nova | 19:36 | |
| mriedem | efried: don't know | 19:36 |
| *** pcaruana has joined #openstack-nova | 19:39 | |
| mriedem | efried: i'd like to remove that part of the comment that gibi pointed out and then approve, is that ok with you? | 19:43 |
| efried | sure | 19:43 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Only allow one scheduler service in tests https://review.opendev.org/682486 | 19:44 |
| openstackgerrit | Merged openstack/nova master: Refactor volume connection cleanup out of _post_live_migration https://review.opendev.org/682741 | 19:54 |
| openstackgerrit | Merged openstack/nova master: Move pre-3.44 Cinder post live migration test to test_compute_mgr https://review.opendev.org/683597 | 19:55 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Avoid error 500 on shelve task_state race https://review.opendev.org/692206 | 19:57 |
| *** pcaruana has quit IRC | 20:09 | |
| *** abaindur has quit IRC | 20:13 | |
| *** abaindur has joined #openstack-nova | 20:13 | |
| *** abaindur_ has joined #openstack-nova | 20:17 | |
| *** abaindur has quit IRC | 20:18 | |
| *** abaindur_ has quit IRC | 20:19 | |
| *** abaindur has joined #openstack-nova | 20:19 | |
| *** READ10 has quit IRC | 20:27 | |
| *** adriant has quit IRC | 20:31 | |
| *** aloga has quit IRC | 20:36 | |
| *** artom has joined #openstack-nova | 20:45 | |
| *** ralonsoh has quit IRC | 20:46 | |
| *** tbachman has joined #openstack-nova | 20:47 | |
| dansmith | efried: asking because I think I've seen some notes from you around this code in the past, | 20:52 |
| dansmith | but do you have strong opinions on what we should do if we have, say, three glance endpoints and we get an error from one trying to do something? | 20:52 |
| efried | dansmith: we should not have three glance endpoints | 20:52 |
| dansmith | looks like we round-robin the available endpoints, but don't move onto the next one after a failure | 20:52 |
| efried | unless you mean three interfaces | 20:53 |
| dansmith | efried: three urls | 20:53 |
| efried | f, did we not get rid of that api_servers thing yet? | 20:53 |
| efried | stand by | 20:53 |
| dansmith | no, the round-robin-ing you mean? that's still there | 20:53 |
| dansmith | I honestly thought that we had pushed responsibility for that down into the glance client, but ... the round-robin stuff is there | 20:54 |
| efried | omg we haven't even deprecated it yet! | 20:54 |
| efried | we should do that. | 20:54 |
| dansmith | efried: use more words.. what are the people currently using that supposed to do? | 20:54 |
| efried | sorry | 20:54 |
| efried | okay | 20:54 |
| *** markvoelker has quit IRC | 20:55 | |
| efried | people using api_servers were theoretically stuffing multiple endpoints in there, and then they were responsible for making sure all of those endpoints pointed to the same images. | 20:55 |
| *** markvoelker has joined #openstack-nova | 20:55 | |
| dansmith | right | 20:55 |
| efried | nova would, on a given call, pick "the next one" and use it. | 20:55 |
| efried | afaik nova never had any logic to, say, "try the next one" if one failed. | 20:56 |
| dansmith | right, it doesn't, but if we can have multiples we should, but.. go on | 20:56 |
| efried | Because nova doesn't keep track of how many there are. It just loads them up and cycles over them. | 20:56 |
| efried | so if we had only one, we still "cycle" over just the one. | 20:56 |
| efried | and if we tried to add a "try next" thing, we would just end up trying... the same one. | 20:56 |
| dansmith | well, all we'd have to do is retry N-1 times for an endpoint count of N and then we'd hit them all and not dupe one or more | 20:57 |
| efried | if api_servers was a thing we still wanted people to use, I could see enhancing our logic to do that properly - keep track of how many there are and, on certain classes of error, try the next and so on until we succeed or come full circle | 20:57 |
| efried | but | 20:57 |
| efried | we don't want people to use api_servers. | 20:57 |
| dansmith | and that is because why? | 20:57 |
| efried | They should use a single endpoint that's a front for a load balancer | 20:57 |
| efried | and use standard ksa options like every other service | 20:58 |
| dansmith | yeah, so the problem there is that you have to have a HA LB for that | 20:58 |
| dansmith | cloud client software needs to handle failures like this to be robust, I'm not sure why nova shouldn't | 20:59 |
| efried | I thought there was a project that did that for you | 20:59 |
| efried | do we try to HA other things? cinder, keystone, neutron...? | 20:59 |
| efried | (trick question) | 20:59 |
| *** markvoelker has quit IRC | 20:59 | |
| *** aloga has joined #openstack-nova | 20:59 | |
| dansmith | well, glance and keystone would be much easier to retry than the others | 20:59 |
| smcginnis | Not the API, but cinder services can be run active/active. | 20:59 |
| smcginnis | And I've seen people use the active/passive pacemaker setup for things too. | 21:00 |
| dansmith | yeah | 21:00 |
| efried | smcginnis: behind a single catalog endpoint tho? | 21:00 |
| smcginnis | Yeah | 21:00 |
| efried | melwitt: would you mind giving this another look? https://review.opendev.org/#/c/615704/ | 21:00 |
| dansmith | efried: well, point is, if you're going to deprecate it, I'd appreciate you getting that patch up so I can point the people I'm talking to about it | 21:00 |
| dansmith | efried: I remember you talking about this a while ago, had kinda assumed it had already happened | 21:01 |
| efried | dansmith: are you playing devil's advocate or do you really think nova should be responsible for load balancing glance? | 21:01 |
| dansmith | tripleo is using api_servers, so they're going to need some notice | 21:01 |
| dansmith | efried: not load balancing, but being able to have multiple endpoints and find one that works, yeah | 21:02 |
| efried | yeah, "everyone" is probably still using api_servers, because we only added the standard ksa stuff in, I think, queens. | 21:02 |
| dansmith | having to have a floating VIP and HA'ing services behind that VIP for everything really sucks | 21:02 |
| efried | is tripleo using multiple endpoints to api_servers? | 21:02 |
| dansmith | efried: just to be clear, "the ksa stuff" doesn't imply anything about multiple endpoints right? | 21:03 |
| dansmith | efried: they are | 21:03 |
| melwitt | efried: whoa, that is old. I can look at it again once I figure out what it's about again | 21:03 |
| dansmith | efried: for glance, but we don't do the failover stuff, so not really for much gain | 21:03 |
| efried | melwitt: thanks, I got a nudge from a downstream (not $employer) who needs it. | 21:03 |
| melwitt | ack | 21:03 |
| efried | dansmith: right, the way nova handles it really isn't HA-ish, it's more load-balance-ish. | 21:04 |
| dansmith | efried: currently, right | 21:04 |
| efried | dansmith: and no, "the ksa stuff" doesn't have anything for mulitiple endpoints... I don't think. (Can you put multiple endpoints in the service catalog?) | 21:04 |
| dansmith | gah | 21:05 |
| efried | dansmith: I can tell you this, cutting over to sdk to talk to glance, api_servers will have to be gone first. | 21:05 |
| dansmith | brb | 21:05 |
| *** ociuhandu has joined #openstack-nova | 21:05 | |
| dansmith | efried: I dunno about the service catalog, probably not | 21:06 |
| dansmith | not saying that's a good thing, but.. :) | 21:06 |
| dansmith | efried: okay so, can you put up a patch to deprecate it real quick so I can point people to it? | 21:07 |
| efried | I'm going to have to nail down mordred for the more philosophical aspects of this. | 21:07 |
| dansmith | efried: -W it if you want for completeness and comments or something | 21:07 |
| efried | I can do that, yeah, though now you've got me doubting if we *should*. | 21:07 |
| dansmith | efried: ack, we can blame lots of stuff on him like this | 21:07 |
| dansmith | efried: that's fine, a place to collect feedback | 21:07 |
| efried | ... | 21:07 |
| dansmith | efried: the tripleo people can go voice their hate there if they want, or they may say "meh, okay, we can VIP this too" I honestly don't know | 21:08 |
| efried | cool | 21:08 |
| dansmith | efried: can you do that, like, uh, quickly so I can reply to someone quickly? :D | 21:08 |
| efried | on it rn | 21:08 |
| dansmith | spanks. | 21:08 |
| efried | if people would stop bugging me in IRC. | 21:08 |
| efried | dansmith: while you're waiting, I found this (search for api_servers) http://specs.openstack.org/openstack/nova-specs/specs/queens/implemented/use-ksa-adapter-for-endpoints.html#proposed-change | 21:12 |
| *** ircuser-1 has joined #openstack-nova | 21:12 | |
| dansmith | "The exception is [glance]api_servers, which will continue to be supported." | 21:13 |
| *** panda|pto has quit IRC | 21:14 | |
| dansmith | I wonder if "valid_interfaces" is an arbitrary list, such that you could have internal1, internal2, internal3? | 21:14 |
| openstackgerrit | Joshua Cornutt proposed openstack/nova master: Moving get_hash_str() from md5 to sha-256 https://review.opendev.org/615704 | 21:15 |
| dansmith | I also don't know how LBs (or HAproxy) handle a failed http session post-request.. meaning if I do a GET against glance and then the connection is reset after I've sent the GET, does the LB retry and re-play that against the next server? | 21:16 |
| mriedem | dansmith: i think it might be, we've had to dig into the ksa code a few times to answer that | 21:16 |
| dansmith | I'd kindof doubt it.. figure it'd fail the client, mark the server, next request doesn't include the bad server | 21:16 |
| dansmith | which means if we don't retry we propagate the failure up | 21:16 |
| dansmith | which is fine in the trivial case, but not so much when we're making multiple calls as part of a complex operation, like against cinder or neutron | 21:17 |
| dansmith | mriedem: might be what, arbitrary? | 21:17 |
| dansmith | or might be constrained? | 21:17 |
| *** panda has joined #openstack-nova | 21:18 | |
| dansmith | based on the description in that spec, though, it tries them in order, so you'd hammer internal1 until it fails, and then try internal2, it sounds like | 21:18 |
| dansmith | which wouldn't be what you'd want really | 21:18 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Deprecate [glance]api_servers https://review.opendev.org/692227 | 21:21 |
| efried | dansmith: ^ | 21:21 |
| mriedem | dansmith: i think valid_interfaces is arbitrary but i've had to look this up multiple times in the ksa code | 21:21 |
| dansmith | efried: thanks | 21:21 |
| efried | dansmith: no, valid_interfaces is public/private/internal only | 21:21 |
| efried | I think they can have 'URL' suffixes for legacy reasons | 21:21 |
| mriedem | also, on that note in the spec for api_servers, i want to say the godaddy or smorrison made a point that they still really needed that when the spec was being reviewed | 21:22 |
| *** maciejjozefczyk has quit IRC | 21:22 | |
| dansmith | mriedem: yeah, I *think* people are going to be quite unhappy about it, but we'll see | 21:22 |
| mriedem | might have been in the ML, i don't see it in review comments | 21:22 |
| efried | "really needed" meaning it was easier for us to continue to support it than it was for them to stop relying on our shoddy load balancing hack | 21:22 |
| efried | I'm having trouble finding sympathy for "wah, I don't want to set up a load balancer in my cloud" from someone who has enough resources to have multiple glance API endpoints in the first place. | 21:24 |
| efried | Isn't there an openstack project that does load balancing for you? | 21:24 |
| dansmith | efried: a six-node mini cloud would still have three controllers in our setups | 21:24 |
| mriedem | efried: are you thinking of octavia? | 21:25 |
| dansmith | efried: we don't deploy less than three controllers (and thus glance apis) no matter how big or small your cloud is, which means it's *always* an issue | 21:25 |
| efried | ...yes, it looks like I may be thinking of octavia | 21:25 |
| mriedem | i don't know much about octavia, but doesn't that spawn VMs to run the load balancer? | 21:25 |
| mriedem | and thus kind of a chicken and egg issue? | 21:25 |
| efried | I dunno how it works | 21:25 |
| dansmith | yeah, octavia is not a solution to this problem | 21:25 |
| mriedem | johnsom would know | 21:26 |
| dansmith | it's a thing you use on top of a functioning cloud to LB an application | 21:26 |
| mriedem | right | 21:26 |
| dansmith | (afaik) | 21:26 |
| efried | it's not a chicken/egg when you're using it as a *load balancer*. It would be chicken/egg if you were using it for HA. | 21:27 |
| mriedem | e.g. we don't use trove instances to run the nova db... | 21:27 |
| efried | which, again, this is not and never has been. | 21:27 |
| *** ociuhandu has quit IRC | 21:27 | |
| dansmith | efried: it's totally chicken and egg for nova to use it between it and glance, for whatever reason :D | 21:28 |
| efried | but at this point I need to stfu and get back to unwedging osc. I'll look for discussion on that patch I guess. | 21:28 |
| dansmith | yep, thanks pointing people at that soonly | 21:28 |
| efried | and probably expect dansmith to start a ML thread for his people | 21:28 |
| efried | yeah | 21:28 |
| dansmith | yeah, octavia requires a functioning nova to work against | 21:29 |
| dansmith | so nova using octavia to talk to glance is not a thing :) | 21:29 |
| mriedem | here is the thread btw http://lists.openstack.org/pipermail/openstack-dev/2017-April/116028.html | 21:29 |
| *** ociuhandu has joined #openstack-nova | 21:29 | |
| mriedem | which also bled into may | 21:29 |
| mriedem | http://lists.openstack.org/pipermail/openstack-dev/2017-May/116574.html is godaddy | 21:30 |
| dansmith | yeah | 21:32 |
| dansmith | I know tripleo and similar things HA certain endpoints, but AFAIK, glance and rabbit have always been multiple servers | 21:32 |
| *** JamesBenson has quit IRC | 21:33 | |
| dansmith | especially for fundamental things like that, it's far easier to set that kind of thing in config vs. something you need an api to do, | 21:33 |
| dansmith | for the same reason that cells in the database is problematic | 21:33 |
| dansmith | anyway | 21:33 |
| johnsom | o/ Not all of the Octavia drivers use nova, but I still would not recommend using Octavia load balancers for internal API traffic. There are many chickens and eggs to fight with for that. I.e. keystone, neutron, potentially glance/nova, etc. | 21:34 |
| mriedem | note that a lot of that ML thread is about removing the endpoint_override options, which we haven't done i don't think | 21:34 |
| mriedem | ksa has an override option | 21:34 |
| mriedem | http://lists.openstack.org/pipermail/openstack-dev/2017-May/116149.html sums up what dan was saying | 21:35 |
| dansmith | well, if I'm understanding this right, once it's down in ksa then we lose the ability to retry smartly right? | 21:35 |
| mriedem | i'm just saying the thread diverged a bit, | 21:35 |
| dansmith | yup | 21:35 |
| mriedem | between overrding the endpoint (don't use the catalog) and multiple api servers for load balancing | 21:35 |
| mriedem | something something openstack ironic > k8s ingress > nova|glance > profit!!! | 21:37 |
| dansmith | and godaddy and blair there actually mention failover more than load-balancing | 21:37 |
| dansmith | heh, well, k8s actually handles this as part of the infra, AFAIK, so ... yeah ;) | 21:37 |
| mriedem | yeah, that's blair's "I think it'd be useful if all OpenStack APIs and their clients actively handled this poor-man's HA without having to resort to haproxy" | 21:38 |
| mriedem | i.e. would be nice if openstack just handled this natively | 21:38 |
| dansmith | right | 21:40 |
| sean-k-mooney | honestly k8s with the nginx ingress control + cert manger + lets encrypt is a nice way to get free tls and do load balancing or at least revers proxying | 21:40 |
| dansmith | because it's kinda dumb for us to reject all kinds of pet features and then not be elastic and failure-handly ourselves | 21:40 |
| sean-k-mooney | johnsom: does octavia have any support for metallb? | 21:41 |
| johnsom | sean-k-mooney: not at the moment | 21:42 |
| johnsom | You can use Octavia for you ingress in k8s though | 21:42 |
| sean-k-mooney | yes | 21:42 |
| *** ociuhandu has quit IRC | 21:43 | |
| sean-k-mooney | but that does seam well like more effort | 21:43 |
| johnsom | The other thing to think about when replacing an LB tier is rate limiting, WAF, and TLS offload | 21:43 |
| sean-k-mooney | you do not nessisarly want your load balancer to have to do those things | 21:44 |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#glance.api_servers isn't an LB tier :) | 21:44 |
| sean-k-mooney | but yes | 21:44 |
| johnsom | I guess that is the context I missed. That seems duplicate to keystone, why not just deprecate it? | 21:45 |
| mriedem | keystone doesn't have a round robin concept like what this thing is | 21:45 |
| mriedem | it's just a very dumb, oh endpoint 1 failed, try endpoint 2 | 21:46 |
| johnsom | DNS provides round robin | 21:46 |
| dansmith | DNS doesn't provide smart round robin unless you resolve the name to all the addresses and then try them yourself | 21:47 |
| dansmith | smart meaning "know when to stop retrying" | 21:47 |
| dansmith | dumb use of DNS RR gets you poor-man's LB, which is not what we're talking about | 21:47 |
| dansmith | since we're off on a tangent, I'm gonna drop off | 21:48 |
| sean-k-mooney | dansmith: so i have scolled back to the point where you have strong opions on how this should work. what was the orginial question? | 21:52 |
| openstackgerrit | Merged openstack/nova master: Mark "block_migration" arg deprecation on pre_live_migration method https://review.opendev.org/682963 | 21:53 |
| openstackgerrit | Merged openstack/nova master: Move Destination object tests to their own test class https://review.opendev.org/683017 | 21:53 |
| openstackgerrit | Merged openstack/nova master: Remove 'test_cold_migrate_with_physnet_fails' test https://review.opendev.org/683961 | 21:53 |
| sean-k-mooney | dansmith: was it solely how to support glance active active | 21:54 |
| *** ociuhandu has joined #openstack-nova | 21:55 | |
| sean-k-mooney | and then the load balancer converstation was a tangent form [glance]/api_servers not being a loadblancing teir or even really a ha solution | 21:56 |
| eandersson | Is there a way to make nginx retry when uwsgi (nova-metadata-api) gets overwhelmed? | 21:56 |
| eandersson | Seeing nova-metadata-api falling behind and causing nginx to throw 502s | 21:57 |
| sean-k-mooney | am can you just increase the process count on the uwsgi config | 21:58 |
| eandersson | We have 5x 24 nova-metadata-plus processes and feels like it should be high enough | 21:58 |
| sean-k-mooney | ah ok | 21:58 |
| eandersson | Kinda feel like 120 should be enough. | 21:58 |
| sean-k-mooney | have you truned on memcache caching for those too | 21:58 |
| eandersson | I'll probably end up bumping it, but feel like I am missing something | 21:59 |
| eandersson | [cache] has memcached enabled, and keystone auth too | 21:59 |
| donnyd | melwitt: thank you :) | 21:59 |
| sean-k-mooney | ok because if you have lots of worker but use the normal in memeory caceh you end up rebuilding stuff in each worker over and over again | 22:00 |
| eandersson | Gonna bump the worker count to like 200, but will dig deeper | 22:03 |
| eandersson | Might do something like https://stackoverflow.com/questions/44581719/resource-temporarily-unavailable-using-uwsgi-nginx as well to see if that helps a bit | 22:03 |
| eandersson | since it looks like a ddos, but kinda unavoidable when you have... many many thousands vms :p | 22:03 |
| *** ociuhandu has quit IRC | 22:04 | |
| sean-k-mooney | ya its possible your hitting connection limits. how is memory looking on those nodes. you dont see OOM event or any other red flags | 22:04 |
| *** ociuhandu has joined #openstack-nova | 22:10 | |
| eandersson | We had to beef them up due to neutron. | 22:10 |
| eandersson | So got plenty of memory :D | 22:10 |
| eandersson | Might be stacking too many services. Could be hitting limits on context switching. | 22:11 |
| sean-k-mooney | im glad but also sad. "whats using all the ram? our cloud software :(" | 22:11 |
| eandersson | Yea I think I mentioned this already, but each neutron worker peaks out at 8.2gb memory per process when under heavy load :'( | 22:12 |
| sean-k-mooney | ya that is much more hten i was expecting | 22:12 |
| eandersson | I do wonder what kind of hardware people use for their larget clusters | 22:13 |
| eandersson | for the "control plane" | 22:13 |
| mriedem | are you sure it's using the cache properly? do you see "Using cached metadata for" in the metadata API debug logs? | 22:13 |
| *** jawad_axd has joined #openstack-nova | 22:13 | |
| eandersson | I'll check in our lab setup sec | 22:14 |
| sean-k-mooney | enabling memcache instead of the dict cache makes a huge difference so if its not useing it properly you could definetly reduce teh amount of workers as a resutl of fixing it | 22:15 |
| *** ociuhandu has quit IRC | 22:15 | |
| mriedem | fwiw cern and huawei public cloud run a metadata api per cell with direct access to the cell db rather than a huge since metadata api that spans the entire deployment | 22:16 |
| mriedem | s/since/single/ | 22:16 |
| mriedem | https://docs.openstack.org/nova/latest/user/cellsv2-layout.html#nova-metadata-api-service | 22:16 |
| sean-k-mooney | mriedem: ya that seams like a better way to approch scaling | 22:16 |
| mriedem | blizzard is only 1 cell though right? | 22:16 |
| sean-k-mooney | dont you have to modify neutorn to proxy to the local instance in that case too | 22:17 |
| eandersson | Yea seeing | 22:17 |
| eandersson | > Using cached metadata for X get_metadata_by_remote_address | 22:17 |
| eandersson | Yea one cell many regions | 22:18 |
| *** jawad_axd has quit IRC | 22:18 | |
| mriedem | sean-k-mooney: yes that's in that doc | 22:19 |
| sean-k-mooney | woudl it make sense even in a singel cell deployment to configure different neutron metadata proxies to hit different apis directly rather then loadbalnce? | 22:20 |
| *** mlavalle has quit IRC | 22:20 | |
| *** ociuhandu has joined #openstack-nova | 22:20 | |
| sean-k-mooney | eandersson: out of interest do you knwo how long the resonce form the metadata api take when you dont get teh 502s | 22:21 |
| eandersson | ~0.004 | 22:22 |
| sean-k-mooney | ok that is normal. we have seen up to 10s of seconds in the gate randomly before we enabeld the memcache caching | 22:23 |
| eandersson | I might have overtuned nginx tbh combined with the nature of metadata | 22:23 |
| eandersson | I set worker count to auto in nginx which ends up creating processors based on the vcpu count | 22:24 |
| eandersson | but uwsgi worker is set to 2/vcpu | 22:24 |
| *** igordc has quit IRC | 22:25 | |
| sean-k-mooney | so i honestly dont know how much load you have but i would try a smaller value like 8 | 22:25 |
| sean-k-mooney | instead of 24 | 22:25 |
| sean-k-mooney | i certely would not have more then 1 per hyper tread | 22:25 |
| eandersson | Yea exactly got one per core now (48/2) | 22:25 |
| eandersson | for nova-api at least | 22:26 |
| sean-k-mooney | ya if your resonce time is in the order of 4ms i would bring that down as i siad to like 8-16 and see if that help rahter then go from 24 to 30 | 22:27 |
| eandersson | We did a lot of benchmarking, but it's difficult to tell how the system performs in real enviornments | 22:27 |
| *** dviroel has quit IRC | 22:29 | |
| *** ociuhandu has quit IRC | 22:31 | |
| *** ociuhandu has joined #openstack-nova | 22:33 | |
| *** ociuhandu has quit IRC | 22:37 | |
| *** ociuhandu has joined #openstack-nova | 22:38 | |
| *** mriedem has quit IRC | 22:40 | |
| *** ociuhandu has quit IRC | 22:43 | |
| *** adriant has joined #openstack-nova | 22:51 | |
| *** jawad_axd has joined #openstack-nova | 22:55 | |
| *** markvoelker has joined #openstack-nova | 22:56 | |
| *** jawad_axd has quit IRC | 22:59 | |
| *** markvoelker has quit IRC | 23:00 | |
| *** tkajinam has joined #openstack-nova | 23:01 | |
| *** tkajinam has quit IRC | 23:01 | |
| *** macz has quit IRC | 23:01 | |
| *** tkajinam has joined #openstack-nova | 23:04 | |
| *** slaweq has quit IRC | 23:05 | |
| *** slaweq has joined #openstack-nova | 23:10 | |
| sean-k-mooney | artom: i left a comment for you here https://review.opendev.org/#/c/692206/3/nova/api/openstack/compute/shelve.py | 23:14 |
| sean-k-mooney | artom: i think that would also fix the issue more generally | 23:14 |
| sean-k-mooney | but i have not tested it | 23:14 |
| sean-k-mooney | anyway night all o/ | 23:15 |
| *** slaweq has quit IRC | 23:16 | |
| *** jawad_axd has joined #openstack-nova | 23:16 | |
| artom | sean-k-mooney, nice idea | 23:17 |
| artom | I'd kinda be afraid of any side effects | 23:17 |
| artom | It'd a wide-reaching change | 23:17 |
| *** jawad_axd has quit IRC | 23:21 | |
| *** ivve has quit IRC | 23:21 | |
| *** jawad_axd has joined #openstack-nova | 23:37 | |
| *** jawad_axd has quit IRC | 23:41 | |
| openstackgerrit | Merged openstack/nova master: fixtures: Add support for security groups https://review.opendev.org/686802 | 23:43 |
| *** ociuhandu has joined #openstack-nova | 23:43 | |
| openstackgerrit | Merged openstack/nova master: nova-net: Migrate 'test_floating_ips' functional tests https://review.opendev.org/684344 | 23:47 |
| *** ociuhandu has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!