| eandersson | interesting | 00:00 |
|---|---|---|
| mriedem | oh wait... | 00:00 |
| eandersson | terraform had the same issue | 00:00 |
| eandersson | and we double checked that we had a version that supports it | 00:00 |
| eandersson | brb | 00:00 |
| mriedem | can you run the osc command with --debug? | 00:00 |
| mriedem | after you're rb | 00:00 |
| *** tosky has quit IRC | 00:00 | |
| *** tetsuro has joined #openstack-nova | 00:02 | |
| *** wolverineav has quit IRC | 00:04 | |
| openstackgerrit | Merged openstack/nova master: Ensure that bandwidth and VF are from the same PF https://review.openstack.org/623543 | 00:08 |
| openstackgerrit | Merged openstack/nova master: api-ref: typo service.disable_reason https://review.openstack.org/640956 | 00:08 |
| openstackgerrit | Merged openstack/nova master: Handle missing exception in instance creation code https://review.openstack.org/641082 | 00:08 |
| *** ttsiouts has quit IRC | 00:11 | |
| mriedem | eandersson: i'm not seeing it by eyeball but maybe something subtle regressed here https://review.openstack.org/#/c/567534/ | 00:13 |
| *** hongbin has quit IRC | 00:13 | |
| *** sapd1 has quit IRC | 00:16 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: WIP: use ironicclient AdapterClient https://review.openstack.org/641170 | 00:17 |
| *** wolverineav has joined #openstack-nova | 00:22 | |
| *** hoonetorg has quit IRC | 00:24 | |
| *** hoonetorg has joined #openstack-nova | 00:26 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova stable/rocky: Fix an error when generating a host ID https://review.openstack.org/641173 | 00:29 |
| *** macza_ has quit IRC | 00:30 | |
| *** brinzhang has joined #openstack-nova | 00:31 | |
| *** mlavalle has quit IRC | 00:33 | |
| *** luksky has quit IRC | 00:35 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Fix RT usage issues in cross-cell resize functional tests https://review.openstack.org/641176 | 00:36 |
| mriedem | Kevin_Zheng: figured out another problem ^ | 00:36 |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support for changing deleted_on_termination after boot https://review.openstack.org/580336 | 00:39 |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Support delete_on_termination in volume attach api https://review.openstack.org/612949 | 00:40 |
| *** artom has joined #openstack-nova | 00:42 | |
| *** wolverineav has quit IRC | 00:48 | |
| *** markvoelker has joined #openstack-nova | 00:49 | |
| mriedem | cfriesen: does starlingx by chance already have a fix for this? https://review.openstack.org/#/c/641176/1/nova/compute/resource_tracker.py@504 | 00:49 |
| mriedem | because that todo left from https://review.openstack.org/#/c/370374/ is the exact issue i'm hitting, | 00:50 |
| mriedem | resource usage on the source node is not correct until update_available_resource runs | 00:50 |
| *** wolverineav has joined #openstack-nova | 00:51 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Fix RT usage issues in cross-cell resize functional tests https://review.openstack.org/641176 | 00:53 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179 | 00:53 |
| *** ileixe has joined #openstack-nova | 00:56 | |
| mriedem | cfriesen: sob it does | 00:57 |
| mriedem | https://github.com/starlingx-staging/stx-nova/blob/master/nova/compute/resource_tracker.py#L728 | 00:57 |
| *** itlinux has joined #openstack-nova | 00:58 | |
| yonglihe | about this topology api, does it had block issue or we just run out of bandwidth? https://review.openstack.org/#/c/621476/ | 01:00 |
| *** mriedem has quit IRC | 01:00 | |
| *** wolverineav has quit IRC | 01:03 | |
| *** wolverineav has joined #openstack-nova | 01:04 | |
| *** whoami-rajat has joined #openstack-nova | 01:09 | |
| Kevin_Zheng | mriedem: for cross cell resize? ACK | 01:09 |
| *** wolverineav has quit IRC | 01:09 | |
| yonglihe | Sean's several suggest directly confilict with current spec, what can i do for that? help me. | 01:13 |
| *** oanson has quit IRC | 01:13 | |
| Kevin_Zheng | yonglihe I will try to check | 01:14 |
| yonglihe | Kevin_Zheng, thanks | 01:15 |
| *** mmethot has joined #openstack-nova | 01:15 | |
| *** gyee has quit IRC | 01:21 | |
| *** mrch has joined #openstack-nova | 01:21 | |
| *** markvoelker has quit IRC | 01:22 | |
| openstackgerrit | guang-yee proposed openstack/nova master: pass endpoint interface to Ironic client https://review.openstack.org/640879 | 01:23 |
| *** mrch_ has quit IRC | 01:24 | |
| *** sdake has quit IRC | 01:29 | |
| *** tetsuro has quit IRC | 01:32 | |
| *** wolverineav has joined #openstack-nova | 01:36 | |
| *** itlinux_ has joined #openstack-nova | 01:39 | |
| *** itlinux_ has quit IRC | 01:40 | |
| *** itlinux has quit IRC | 01:41 | |
| openstackgerrit | Merged openstack/nova master: Improve libvirt image and snapshot handling https://review.openstack.org/616692 | 01:44 |
| eandersson | I tried the latest openstack client in a lab version of Rocky | 01:45 |
| eandersson | and soft-anti-affinity does not work there either | 01:45 |
| *** sdake has joined #openstack-nova | 01:45 | |
| *** mikal has joined #openstack-nova | 01:47 | |
| *** jhesketh has quit IRC | 01:57 | |
| *** lbragstad has quit IRC | 01:58 | |
| *** jhesketh has joined #openstack-nova | 01:58 | |
| *** wolverineav has quit IRC | 02:00 | |
| *** wolverineav has joined #openstack-nova | 02:00 | |
| *** sdake has quit IRC | 02:05 | |
| *** ileixe has quit IRC | 02:05 | |
| *** ileixe has joined #openstack-nova | 02:08 | |
| *** mdbooth_ has quit IRC | 02:15 | |
| *** markvoelker has joined #openstack-nova | 02:19 | |
| *** wolverineav has quit IRC | 02:22 | |
| *** hongbin has joined #openstack-nova | 02:26 | |
| *** wolverineav has joined #openstack-nova | 02:27 | |
| *** sdake has joined #openstack-nova | 02:29 | |
| *** tbachman has quit IRC | 02:33 | |
| *** wolverineav has quit IRC | 02:40 | |
| *** sdake has quit IRC | 02:42 | |
| *** psachin has joined #openstack-nova | 02:43 | |
| *** lbragstad has joined #openstack-nova | 02:51 | |
| *** markvoelker has quit IRC | 02:52 | |
| *** mdbooth has joined #openstack-nova | 03:09 | |
| *** sdake has joined #openstack-nova | 03:20 | |
| *** sdake has quit IRC | 03:23 | |
| *** sdake_ has joined #openstack-nova | 03:24 | |
| *** wolverineav has joined #openstack-nova | 03:43 | |
| *** wolverineav has quit IRC | 03:44 | |
| *** zhubx has quit IRC | 03:46 | |
| *** zhubx has joined #openstack-nova | 03:47 | |
| *** janki has joined #openstack-nova | 03:49 | |
| *** markvoelker has joined #openstack-nova | 03:49 | |
| *** janki has quit IRC | 03:50 | |
| *** janki has joined #openstack-nova | 03:50 | |
| openstackgerrit | Boxiang Zhu proposed openstack/nova master: Raise error when force_raw_images and images_type are incompatible https://review.openstack.org/640271 | 03:56 |
| *** zhubx has quit IRC | 03:57 | |
| *** sdake_ has quit IRC | 03:58 | |
| *** zhubx has joined #openstack-nova | 03:58 | |
| *** wolverineav has joined #openstack-nova | 03:58 | |
| *** markvoelker has quit IRC | 04:22 | |
| openstackgerrit | Merged openstack/nova master: Check hosts have no instances for AZ rename https://review.openstack.org/509206 | 04:25 |
| *** ratailor has joined #openstack-nova | 04:27 | |
| *** hongbin has quit IRC | 04:42 | |
| openstackgerrit | Merged openstack/nova master: Move set_vf_interface_vlan to be with its only caller. https://review.openstack.org/624229 | 04:59 |
| openstackgerrit | Merged openstack/nova master: Move DHCP releasing to privsep. https://review.openstack.org/624230 | 05:00 |
| openstackgerrit | Merged openstack/nova master: Convert additional IP management calls to privsep. https://review.openstack.org/624231 | 05:00 |
| openstackgerrit | Merged openstack/nova master: Move route management to privsep. https://review.openstack.org/624232 | 05:00 |
| *** sdake has joined #openstack-nova | 05:03 | |
| *** sdake has quit IRC | 05:03 | |
| *** dave-mccowan has quit IRC | 05:03 | |
| *** tetsuro has joined #openstack-nova | 05:04 | |
| *** sdake has joined #openstack-nova | 05:07 | |
| *** marst has joined #openstack-nova | 05:08 | |
| *** bhagyashris_ has joined #openstack-nova | 05:11 | |
| *** sridharg has joined #openstack-nova | 05:13 | |
| *** igordc has joined #openstack-nova | 05:14 | |
| *** marst has quit IRC | 05:18 | |
| *** igordc has quit IRC | 05:27 | |
| *** hoonetorg has quit IRC | 05:33 | |
| *** sdake has joined #openstack-nova | 05:39 | |
| *** wolverineav has quit IRC | 05:40 | |
| *** bhagyashris_ has quit IRC | 05:43 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.openstack.org/641200 | 05:43 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Add a live migration regression test https://review.openstack.org/641200 | 05:45 |
| *** johnthetubaguy has quit IRC | 05:49 | |
| *** hoonetorg has joined #openstack-nova | 05:50 | |
| *** johnthetubaguy has joined #openstack-nova | 05:56 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Add pmem_namespaces field to the NUMACell obj https://review.openstack.org/634547 | 05:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: object: Add virtual_pmems fields to the InstanceNUMACell obj https://review.openstack.org/634548 | 05:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Enable driver configures PMEM namespace when initiating libvirt driver https://review.openstack.org/634549 | 05:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: Update PMEM namespaces info and usage https://review.openstack.org/634550 | 05:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: report pmem namespaces resources by provider tree https://review.openstack.org/634551 | 05:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: API: parse pmem related flavor extra spec https://review.openstack.org/634552 | 05:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: scheduler: translate virtual pmems request to placement request group https://review.openstack.org/634553 | 05:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: update _numa_fit_instance_cell for the support of virtual_pmems https://review.openstack.org/634554 | 05:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: libvirt: boot VM with vpmems and cleanup vpmems when destroying https://review.openstack.org/634555 | 05:58 |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: support VM resizing with vpmem data migration https://review.openstack.org/634556 | 05:58 |
| *** igordc has joined #openstack-nova | 06:07 | |
| *** udesale has joined #openstack-nova | 06:07 | |
| *** igordc has quit IRC | 06:07 | |
| *** wolverineav has joined #openstack-nova | 06:14 | |
| *** wolverineav has quit IRC | 06:18 | |
| *** wolverineav has joined #openstack-nova | 06:18 | |
| openstackgerrit | Merged openstack/nova master: Flavor extra spec and image properties validation from API https://review.openstack.org/620706 | 06:23 |
| *** wolverineav has quit IRC | 06:32 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Update --max-rows parameter description for archive_deleted_rows https://review.openstack.org/606995 | 06:35 |
| *** markvoelker has joined #openstack-nova | 06:49 | |
| openstackgerrit | Abhishek Kekane proposed openstack/nova-specs master: Support multiple backend of Glance https://review.openstack.org/641210 | 06:52 |
| *** Luzi has joined #openstack-nova | 06:53 | |
| *** ivve has joined #openstack-nova | 07:00 | |
| openstackgerrit | Boxiang Zhu proposed openstack/nova master: Raise error when force_raw_images and images_type are incompatible https://review.openstack.org/640271 | 07:04 |
| openstackgerrit | melanie witt proposed openstack/nova master: Count instances from mappings and cores/ram from placement https://review.openstack.org/638073 | 07:04 |
| openstackgerrit | melanie witt proposed openstack/nova master: Use instance mappings to count server group members https://review.openstack.org/638324 | 07:04 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: api-ref: Add description for BDM volume_size https://review.openstack.org/641255 | 07:06 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add user_id column to the instance_mappings table https://review.openstack.org/633349 | 07:10 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add user_id field to InstanceMapping https://review.openstack.org/633350 | 07:10 |
| openstackgerrit | melanie witt proposed openstack/nova master: Populate InstanceMapping.user_id during migrations and schedules https://review.openstack.org/638574 | 07:10 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add online data migration for populating user_id https://review.openstack.org/633351 | 07:10 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add get_counts() to InstanceMappingList https://review.openstack.org/638072 | 07:10 |
| openstackgerrit | melanie witt proposed openstack/nova master: Count instances from mappings and cores/ram from placement https://review.openstack.org/638073 | 07:10 |
| openstackgerrit | melanie witt proposed openstack/nova master: Use instance mappings to count server group members https://review.openstack.org/638324 | 07:10 |
| *** cfriesen has quit IRC | 07:14 | |
| openstackgerrit | Abhishek Kekane proposed openstack/nova-specs master: Support multiple backend of Glance https://review.openstack.org/641210 | 07:16 |
| *** lbragstad has quit IRC | 07:17 | |
| *** markvoelker has quit IRC | 07:21 | |
| *** luksky has joined #openstack-nova | 07:35 | |
| *** bhagyashris has joined #openstack-nova | 07:38 | |
| *** luksky has quit IRC | 07:39 | |
| *** ccamacho has joined #openstack-nova | 07:41 | |
| yonglihe | sean-k-mooney: ping | 07:47 |
| yonglihe | https://docs.google.com/document/d/1kRRZFq_ha0T9mFDOEzv0PMvXgtnjGm5ii9mSzdqt1VM/edit#heading=h.kn16u3i4dj85 | 07:56 |
| yonglihe | I collected your commets in google docs and anyone could edit or comment it, i given my opion about them, some of them needs discuss. | 07:57 |
| *** rpittau|afk is now known as rpittau | 08:02 | |
| *** dtantsur|afk is now known as dtantsur | 08:03 | |
| *** dtantsur is now known as dtantsur|mtg | 08:03 | |
| *** wolverineav has joined #openstack-nova | 08:03 | |
| *** tesseract has joined #openstack-nova | 08:07 | |
| *** wolverineav has quit IRC | 08:08 | |
| *** tkajinam has quit IRC | 08:09 | |
| *** wolverineav has joined #openstack-nova | 08:10 | |
| *** awalende has joined #openstack-nova | 08:15 | |
| *** wolverineav has quit IRC | 08:17 | |
| *** wolverineav has joined #openstack-nova | 08:18 | |
| *** phasespace has joined #openstack-nova | 08:19 | |
| *** markvoelker has joined #openstack-nova | 08:19 | |
| *** helenafm has joined #openstack-nova | 08:23 | |
| *** tosky has joined #openstack-nova | 08:29 | |
| *** pcaruana has joined #openstack-nova | 08:29 | |
| *** takashin has left #openstack-nova | 08:31 | |
| *** tssurya has joined #openstack-nova | 08:33 | |
| *** ralonsoh has joined #openstack-nova | 08:38 | |
| openstackgerrit | Merged openstack/nova master: Validate PCI aliases early in resize https://review.openstack.org/641131 | 08:42 |
| *** xek has joined #openstack-nova | 08:49 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Should not skip volume_size check for bdm.image_id == image_ref case https://review.openstack.org/641282 | 08:50 |
| *** markvoelker has quit IRC | 08:52 | |
| *** dpawlik has joined #openstack-nova | 08:57 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Use Selection object to fill request group mapping https://review.openstack.org/638711 | 09:05 |
| *** ttsiouts has joined #openstack-nova | 09:08 | |
| *** ttsiouts has quit IRC | 09:13 | |
| *** dpawlik has quit IRC | 09:17 | |
| *** wolverineav has quit IRC | 09:19 | |
| *** tetsuro has quit IRC | 09:35 | |
| *** derekh has joined #openstack-nova | 09:37 | |
| *** markvoelker has joined #openstack-nova | 09:49 | |
| *** moshele has joined #openstack-nova | 09:50 | |
| *** bhagyashris has quit IRC | 09:54 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Warn if group_policy is missing from flavor https://review.openstack.org/641289 | 10:05 |
| gibi | simple patch ^^ | 10:06 |
| bauzas | gibi: +2d | 10:19 |
| openstackgerrit | Merged openstack/nova master: Move additional IP address management to privsep. https://review.openstack.org/624591 | 10:21 |
| *** markvoelker has quit IRC | 10:22 | |
| gibi | bauzas: thanks | 10:23 |
| gibi | bauzas: this the that API patch for bandwidht, turning the functionalty on https://review.openstack.org/#/c/636360/ | 10:24 |
| gibi | bauzas: mriedem already +2-d it | 10:24 |
| bauzas | I know, I was planning to vote on it soon | 10:24 |
| gibi | bauzas: cool. thanks | 10:24 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Test macvtap port with resource request https://review.openstack.org/641309 | 10:33 |
| gibi | it is a functional test only patch ^^ | 10:33 |
| gibi | sean-k-mooney: ^^ | 10:34 |
| *** moshele has quit IRC | 10:40 | |
| *** cdent has joined #openstack-nova | 10:49 | |
| openstackgerrit | Neil Jerram proposed openstack/nova master: Allow utime call to fail on Flat image base file https://review.openstack.org/641315 | 10:52 |
| *** wolverineav has joined #openstack-nova | 10:53 | |
| *** udesale has quit IRC | 10:57 | |
| *** wolverineav has quit IRC | 10:57 | |
| *** xek has quit IRC | 10:58 | |
| *** xek_ has joined #openstack-nova | 10:58 | |
| *** zhubx has quit IRC | 10:59 | |
| *** zhubx has joined #openstack-nova | 10:59 | |
| *** thgcorrea has joined #openstack-nova | 11:00 | |
| *** FlorianFa has quit IRC | 11:01 | |
| *** FlorianFa has joined #openstack-nova | 11:02 | |
| *** helenafm_ has joined #openstack-nova | 11:04 | |
| *** pcaruana has quit IRC | 11:04 | |
| *** mikal has quit IRC | 11:04 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: objects: Store InstancePCIRequest.numa_policy in DB https://review.openstack.org/624444 | 11:06 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: fakelibvirt: Add ability to generate fake PCI devices https://review.openstack.org/640409 | 11:06 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Create PCI tests for NUMA'y tests https://review.openstack.org/640826 | 11:06 |
| *** helenafm has quit IRC | 11:06 | |
| stephenfin | gibi, bauzas: Any chance you could take a look at those today (or at least the bottom one)? | 11:06 |
| bauzas | stephenfin: sure, I dedicated my time today for reviews | 11:07 |
| *** moshele has joined #openstack-nova | 11:07 | |
| bauzas | spread the word | 11:07 |
| gibi | stephenfin: looking | 11:08 |
| *** erlon has joined #openstack-nova | 11:09 | |
| *** panda|ruck|off is now known as panda|ruck|flu | 11:10 | |
| gibi | stephenfin: I'm +2 on the whole series. It really made those sriov tests more readable at the end | 11:14 |
| stephenfin | gibi: Sweet. Thanks. | 11:14 |
| stephenfin | gibi: I'd been thinking about maybe using Jinja2 templates for that entire file, but I'll tackle that later/separately :) | 11:15 |
| stephenfin | gibi: If it's not too much, this might also be of interest to you https://review.openstack.org/#/c/636383/ (another clean up'y one) | 11:16 |
| *** erlon has quit IRC | 11:16 | |
| *** erlon has joined #openstack-nova | 11:17 | |
| gibi | stephenfin: I will check that after launch | 11:18 |
| gibi | lunch | 11:18 |
| stephenfin | ta :) | 11:18 |
| *** markvoelker has joined #openstack-nova | 11:19 | |
| *** pcaruana has joined #openstack-nova | 11:32 | |
| *** dave-mccowan has joined #openstack-nova | 11:45 | |
| openstackgerrit | Neil Jerram proposed openstack/nova master: Allow utime call to fail on qcow2 image base file https://review.openstack.org/641315 | 11:46 |
| *** markvoelker has quit IRC | 11:52 | |
| *** xek_ has quit IRC | 11:54 | |
| *** ratailor has quit IRC | 11:54 | |
| *** erlon has quit IRC | 12:03 | |
| *** erlon has joined #openstack-nova | 12:04 | |
| *** mvkr has joined #openstack-nova | 12:13 | |
| *** ttsiouts has joined #openstack-nova | 12:16 | |
| *** mdbooth has quit IRC | 12:21 | |
| openstackgerrit | Merged openstack/nova master: Fix WeighedHost logging regression https://review.openstack.org/641143 | 12:25 |
| *** tetsuro has joined #openstack-nova | 12:31 | |
| *** brinzhang has quit IRC | 12:31 | |
| *** udesale has joined #openstack-nova | 12:38 | |
| *** markvoelker has joined #openstack-nova | 12:49 | |
| *** mchlumsky has joined #openstack-nova | 13:02 | |
| *** mdbooth has joined #openstack-nova | 13:05 | |
| *** TheJulia_sick is now known as TheJulia | 13:05 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Warn if group_policy is missing from flavor https://review.openstack.org/641289 | 13:15 |
| *** ttsiouts has quit IRC | 13:17 | |
| *** ttsiouts has joined #openstack-nova | 13:18 | |
| *** liuyulong_zzz is now known as liuyulong | 13:19 | |
| *** markvoelker has quit IRC | 13:22 | |
| *** ttsiouts has quit IRC | 13:23 | |
| frickler | eandersson: check whether your compute endpoint is v2 or v2.1. the former doesn't support microversions. | 13:23 |
| frickler | probably osc should do some better guarding against that | 13:24 |
| sean-k-mooney | frickler: osc does not support auto microversions either but yes if you pass one explcitly i coudl check that you are not using the v2 endpoint | 13:25 |
| frickler | sean-k-mooney: see the backlog from yesterday. "openstack --os-compute-api-version 2.15 server group create anti-affinity-test --policy soft-anti-affinity" fails when I change my compute endpoint url from /v2.1 to /v2 | 13:26 |
| sean-k-mooney | frickler: sure you are using an explcitly microverions it could guard against that but osc doese not really have any special logic to handel microversions | 13:27 |
| *** ttsiouts has joined #openstack-nova | 13:28 | |
| frickler | sean-k-mooney: hmm, o.k., maybe then nova should throw an error when the client requests 2.15 instead of silently falling back to 2.0 behaviour | 13:29 |
| sean-k-mooney | that woudl be a backwards incompatiable chagne | 13:30 |
| sean-k-mooney | and would break those that expect it | 13:30 |
| sean-k-mooney | the v2 endpoint is unversioned so by deffinion it shoudl not be doing version checks | 13:30 |
| yonglihe | sean-k-mooney: hi, good morning | 13:30 |
| openstackgerrit | Andrey Volkov proposed openstack/nova stable/rocky: Check hosts have no instances for AZ rename https://review.openstack.org/641351 | 13:31 |
| frickler | sean-k-mooney: hmm, o.k., let's wait for eandersson to confirm whether I located his issue before continuing that discussion ;) | 13:32 |
| *** tacco is now known as oekotaco | 13:33 | |
| *** mriedem has joined #openstack-nova | 13:34 | |
| mriedem | sean-k-mooney: https://blueprints.launchpad.net/nova/+spec/libvirt-neutron-sriov-livemigration is next in the runways slot but is that going to make stein in the next 24 hours? | 13:38 |
| mriedem | my guess is no looking at the state of the series | 13:38 |
| yonglihe | mriedem, sean-k-mooney: any suggest on https://review.openstack.org/#/c/621476/? sean-k-mooney had quiet a lot concern about the sockets/threads/cores that reported to user. | 13:40 |
| mriedem | yonglihe: the contents of that regarding numa are not my strong area so i will have to defer to sean-k-mooney and stephenfin and others | 13:41 |
| yonglihe | ok, i collected them into https://docs.google.com/document/d/1kRRZFq_ha0T9mFDOEzv0PMvXgtnjGm5ii9mSzdqt1VM/edit# , might be easier to work on. sean-k-mooney , stephenfin. | 13:43 |
| mriedem | tssurya: i copied you on an email to jose about https://review.openstack.org/#/c/613039/ since he's at cern as well - just fyi | 13:43 |
| yonglihe | but seems time for stein is quite tight. | 13:43 |
| mriedem | tssurya: since we've got <24 hours to land that | 13:43 |
| mriedem | yonglihe: yes it is | 13:43 |
| tssurya | mriedem: ack, thanks | 13:43 |
| mriedem | tssurya: i'm of course assuming cern is very small and everyone knows everyone else and what they are working on... | 13:44 |
| sean-k-mooney | yonglihe: o/ | 13:44 |
| tssurya | mriedem: yep I am going to pass by his office now :) | 13:44 |
| mriedem | tssurya: oh heh i was just joking | 13:44 |
| mriedem | but cool | 13:44 |
| openstackgerrit | Merged openstack/nova master: Validate bandwidth configuration for other VIF types https://review.openstack.org/636383 | 13:45 |
| openstackgerrit | Merged openstack/nova master: Further de-dupe os-vif VIF tests https://review.openstack.org/636384 | 13:45 |
| sean-k-mooney | yonglihe: so overall i dont think the datamodel is quite right. given the time constraitns i personally would be more comfortable waiting for train to finalise it but i can take a look at the google doc. etherpad thens to be a good choice for this kind of thing also | 13:46 |
| *** josecastroleon has joined #openstack-nova | 13:47 | |
| sean-k-mooney | yonglihe: one of the main issues is the current proposed api is coupeling cpu topology and numa topology but they are independnt fo each other which is why im concerned the current data model is missleading | 13:47 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Fix WeighedHost logging regression https://review.openstack.org/641355 | 13:48 |
| yonglihe | sean-k-mooney, yeah, i get that. it's mass. how about delete the cpu topology toltally. | 13:49 |
| *** tetsuro has quit IRC | 13:50 | |
| bauzas | gibi: +Wd with a comment on https://review.openstack.org/#/c/636360/23 | 13:51 |
| yonglihe | considerating the time factor, it's up to you choosing what we can do on the stein, or postpone to next release. both ok for me. | 13:51 |
| sean-k-mooney | you could or you could group it seperatly. e.g. in the toployg endpoint you get back a dict with two fields {cpu_toplolgoy:{}, numa_toplogy[{},{}]} | 13:51 |
| *** pcaruana has quit IRC | 13:51 | |
| *** tetsuro has joined #openstack-nova | 13:51 | |
| *** eharney has joined #openstack-nova | 13:51 | |
| gibi | bauzas: thanks, replied | 13:51 |
| tssurya | mriedem: josecastroleon is working on it | 13:52 |
| bauzas | gibi: great, that works with me | 13:52 |
| gibi | cool | 13:52 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/queens: Fix WeighedHost logging regression https://review.openstack.org/641359 | 13:52 |
| bauzas | for docs, I dunno where to land docs | 13:52 |
| bauzas | given that's both neutron/nova | 13:52 |
| bauzas | stephenfin: thoughts for doc'ing bandwith RPs feature ? | 13:53 |
| bauzas | in nova, neutron or both ? | 13:53 |
| gibi | bauzas: from nova perspective only the API limitations and the vrit driver dependency are externally visible, the rest is neutron configuration and neutron API | 13:53 |
| bauzas | stephenfin: how can we write some admin notes given docs are now per project ? | 13:53 |
| * gibi needs to step away for a bit | 13:53 | |
| stephenfin | bauzas: Depends on where most of the work has to be done. It _feels_ like more of a neutron thing, IMO | 13:53 |
| bauzas | oki doki, I'm good with this | 13:54 |
| stephenfin | bauzas: SR-IOV networking is probably the closest thing we have and that is in the neutron guide, though we do have references to it in our docs | 13:54 |
| *** sdake has quit IRC | 13:54 | |
| mriedem | tssurya: thanks | 13:54 |
| bauzas | that's my point, we probably still need references to it | 13:54 |
| stephenfin | sean-k-mooney: what was the reply to mriedem's question about the libvirt-neutron-sriov-livemigration spec? You think it's reasonable to land | 13:55 |
| stephenfin | (FWIW, I have reviewed it myself multiple times and I don't _think_ any of jaypipes' comments were too serious) | 13:56 |
| yonglihe | sean-k-mooney: good idea, then seems I could not do much on that tonight, and I would like to hear more from you. so after I had a good sleep, than I can catch all comment from you, and trying to change the patch and to see what happens then. anyway, thanks a lot. have a good one. | 13:56 |
| sean-k-mooney | stephenfin: i did not see that question | 13:56 |
| sean-k-mooney | mriedem: ah you were asking if it would make stien in the next 24 hours | 13:57 |
| sean-k-mooney | the main issue i think has been the lack of review so i dont know if people will raise issue | 13:58 |
| sean-k-mooney | mriedem: i think it could land but if you would prefer to defer to train to give it more scurtiny then i can live with that too | 13:58 |
| sean-k-mooney | the neuton depency has merged so its all on the nova side at this point | 13:59 |
| mriedem | i personally think it's probably high risk at this point but i also haven't reviewed it | 13:59 |
| *** sdake has joined #openstack-nova | 14:00 | |
| sean-k-mooney | ya thats fair. its less high risk then the numa stuff since sriov live migration never work before in any scenairo so we cant break it more then it was but obviosly we want it to be right | 14:01 |
| *** pcaruana has joined #openstack-nova | 14:01 | |
| *** mlavalle has joined #openstack-nova | 14:02 | |
| sean-k-mooney | mriedem: artom has modifed the openlab request to see if we could get sriov capably servers so we can test it as part of that effort too | 14:02 |
| stephenfin | mriedem: I would suggest taking a glance. There's one patch that is rather bulky but the rest seem compact/grokable. Definitely would benefit from someone with a deep knowledge of the live migration flow too | 14:03 |
| stephenfin | The NUMA stuff is far more involved, yeah | 14:03 |
| *** marst has joined #openstack-nova | 14:06 | |
| sean-k-mooney | ... i need to resovle a merge conflict with gibi's stuff ill work on that now | 14:07 |
| sean-k-mooney | it should be small but we merged a few thing in the last 36 hours that conflicted with this code | 14:08 |
| *** wolverineav has joined #openstack-nova | 14:08 | |
| * alex_xu can't find one which can chime in easily and have chance in last 24 hours | 14:08 | |
| *** FlorianFa has quit IRC | 14:09 | |
| mriedem | yeah honestly i'm going to be focusing in 2 blueprints today most likely, the rbd extend volume one and maybe we can get the data migration part of melwitt's counting quotas from placement in stein, but i don't know about the rest of it | 14:09 |
| mriedem | *on | 14:10 |
| mriedem | i've taken about a week off from the cross-cell resize stuff and despite it not getting in stein our product team needs it by end of the month so i have to start working on that more | 14:10 |
| sean-k-mooney | mriedem: sure no worries. ill resolve the merge confilcit anyway and redeploy locally to test. | 14:10 |
| mriedem | alex_xu: if i find something that does not require a lot of prior context i will ping you | 14:11 |
| mriedem | although it's late now | 14:11 |
| mriedem | gibi: i left some comments on that neutron docs patch, | 14:11 |
| mriedem | gibi: it reminded me - we don't have any sort of minimum compute service version check from the api for min bw provider support right? | 14:12 |
| alex_xu | mriedem: cool, I empty tomorrow for help something | 14:12 |
| *** wolverineav has quit IRC | 14:12 | |
| mriedem | gibi: and maybe we don't because of what we talked about the other day with bauzas - if the compute/neutron agent are upgraded to stein then they report inventory, otherwise they don't and the scheduler shouldn't pick them for these types of workloads | 14:13 |
| gibi | mriedem: we dont have compute version checks for the reason you described | 14:14 |
| mriedem | yeah ok | 14:14 |
| *** jaypipes has quit IRC | 14:14 | |
| *** jaypipes has joined #openstack-nova | 14:15 | |
| *** sdake has quit IRC | 14:17 | |
| *** markvoelker has joined #openstack-nova | 14:19 | |
| *** lbragstad has joined #openstack-nova | 14:20 | |
| *** sdake has joined #openstack-nova | 14:22 | |
| sean-k-mooney | jaypipes: thanks for taking the time to review the sriov stuff yesterday just seeing it now ill sync with adrianc to adress all the feedback and we will respin. | 14:24 |
| *** sdake has quit IRC | 14:24 | |
| adrianc | already addressed the comments will upload a PS soon | 14:25 |
| sean-k-mooney | adrianc: ah cool. i was distracted in neutron land the last 2 days | 14:25 |
| adrianc | sean-k-mooney, shall i rebase the direct and indirect patches on top ? | 14:25 |
| sean-k-mooney | am sure | 14:25 |
| adrianc | promise not to loose a PS :) | 14:25 |
| sean-k-mooney | hehe i trust you not to :) i generally work my way from the bottom up and cherrypick the later patches when working on a chain like this | 14:27 |
| sean-k-mooney | that or use interactive rebases if its just my own patches | 14:27 |
| jaypipes | ok, thanks adrianc and sean-k-mooney. will review it as soon as I see the new patches. | 14:28 |
| sean-k-mooney | :) | 14:28 |
| adrianc | sean-k-mooney: gotcha thanks ! | 14:30 |
| adrianc | jaypipes: thanks for the inputs | 14:30 |
| *** tetsuro has quit IRC | 14:34 | |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: Sep methods to free claimed and allocated devs https://review.openstack.org/616120 | 14:36 |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: Allow per-port modification of vnic_type and profile https://review.openstack.org/607365 | 14:36 |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: Add get_instance_pci_request_from_vif https://review.openstack.org/619929 | 14:36 |
| *** moshele has quit IRC | 14:37 | |
| *** sdake has joined #openstack-nova | 14:38 | |
| *** awaugama has joined #openstack-nova | 14:38 | |
| mriedem | https://www.youtube.com/watch?v=jk8SToEQPGw | 14:39 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Fix WeighedHost logging regression https://review.openstack.org/641398 | 14:40 |
| mtreinish | mriedem: I like that the top comments on that are trying to explain the joke... | 14:42 |
| mriedem | mtreinish: i like that you are gone for weeks at a time and only show up, in this channel of all places, when i drop a simpsons video | 14:43 |
| mtreinish | I think that I have my priorities straight | 14:45 |
| mriedem | i don't disagree | 14:45 |
| sean-k-mooney | adrianc: by the way i am assuming you are crurrently rebasing https://review.openstack.org/#/c/620115 on the ohter changes. the final change in the seriese does not use any of the funcitions you modified so that should be a straight cherrypick at the end | 14:46 |
| *** tbachman has joined #openstack-nova | 14:46 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/rocky: Handle missing exception in instance creation code https://review.openstack.org/641401 | 14:46 |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: SR-IOV Live migration indirect port support https://review.openstack.org/620115 | 14:46 |
| adrianc | sean-k-mooney: yes | 14:47 |
| sean-k-mooney | :) | 14:47 |
| mriedem | takashin: when you get a chance can you backport https://review.openstack.org/#/c/636271/ please? | 14:48 |
| mriedem | mtreinish: while you're here, see how this job runs tempest with --concurrency=4 http://logs.openstack.org/72/638072/14/check/nova-next/c8ecf61/job-output.txt.gz#_2019-03-06_08_18_20_021174 | 14:52 |
| mriedem | but yet it looks like a lot of the tests are running serially | 14:52 |
| *** markvoelker has quit IRC | 14:52 | |
| mriedem | the first 7 are on the same worker | 14:52 |
| mriedem | unless that just means the other workers were running slower tests at the same time? | 14:53 |
| mriedem | yeah i suppose that's all it is | 14:53 |
| mtreinish | mriedem: yeah I think that's what's going on | 14:54 |
| mriedem | man there are tests in tempest that really just don't belong there | 14:54 |
| mriedem | tempest.api.compute.servers.test_list_server_filters.ListServerFiltersTestJSON.test_list_servers_filter_by_shutoff_status [72.172009s] ... ok | 14:54 |
| mtreinish | the stackviz view is good for visualizing that: http://logs.openstack.org/72/638072/14/check/nova-next/c8ecf61/logs/stackviz/#/stdin/timeline | 14:54 |
| mriedem | create a server, stop it, wait for it to be stopped | 14:54 |
| HD|Laptop | hey all | 14:55 |
| openstackgerrit | Jose Castro Leon proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039 | 14:55 |
| mriedem | we could test shutoff server filtering in functional tests | 14:55 |
| mtreinish | I'd support moving that test out of tempest | 14:56 |
| HD|Laptop | how do I convince nova-compute to not include the NFS storage under "local disk store" in Horizon? | 14:56 |
| mriedem | cripes these dualnet scenario tests are also long as hell | 14:56 |
| mriedem | 3+ minutes | 14:56 |
| *** ivve has quit IRC | 14:57 | |
| openstackgerrit | Adrian Chiris proposed openstack/nova master: Sep methods to free claimed and allocated devs https://review.openstack.org/616120 | 14:57 |
| mriedem | HD|Laptop: you mean a single compute is showing local storage for the entire nfs share rather? | 14:58 |
| mriedem | that's not really a horizon thing | 14:58 |
| mriedem | e.g. compute1 is showing like 2TB of "local" disk | 14:58 |
| HD|Laptop | mriedem: under horizon/admin/hypervisors/, each hypervisor shows with 68TB local storage | 14:58 |
| HD|Laptop | probably this is because its instance_path is set to a shared folder on said NFS server | 14:59 |
| mriedem | yes | 14:59 |
| HD|Laptop | and horizon seems to sum up all the disks... to "Lokale Festplattenbelegung | 14:59 |
| mtreinish | mriedem: iirc those tests make multiple servers and networks, login to them with an ipv4 network and test connectivity between servers over different ipv6 configurations. It takes forvever just waiting for things to come up | 14:59 |
| HD|Laptop | 120GB von 582,8TB benutzt | 14:59 |
| mriedem | HD|Laptop: that comes from the libvirt driver in here https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/virt/libvirt/driver.py#L5747 | 15:00 |
| mriedem | https://github.com/openstack/nova/blob/dfaa513fe1329aabe6e5d83b6ac42e0e25ad9eb6/nova/virt/libvirt/driver.py#L5762 | 15:00 |
| mriedem | since instances_path is on shared storage, it's going to show the entire storage pool for that path | 15:00 |
| mriedem | i'm not aware of a way around that | 15:00 |
| *** sdake has quit IRC | 15:00 | |
| HD|Laptop | ah okay. damn :'D | 15:00 |
| mriedem | mtreinish: time for an "extra-slow" tag soon | 15:01 |
| mriedem | mtreinish: will openstack health show me how long on average the entire TestGettingAddress class takes? | 15:01 |
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: auto detach/attach sriov ports on migration https://review.openstack.org/629589 | 15:01 |
| mtreinish | it doesn't let you filter the averages by class iirc | 15:02 |
| mtreinish | but if you give me a sec I can hack together a graph for you manually | 15:02 |
| mriedem | took ~35 minutes in logs.openstack.org/72/638072/14/check/nova-next/c8ecf61/job-output.txt.gz | 15:05 |
| mriedem | 8 tests | 15:06 |
| *** sdake has joined #openstack-nova | 15:06 | |
| *** cfriesen has joined #openstack-nova | 15:07 | |
| *** moshele has joined #openstack-nova | 15:07 | |
| *** Luzi has quit IRC | 15:09 | |
| *** agopi has joined #openstack-nova | 15:14 | |
| efried | mriedem: where/how can I run a CI that exercises ironic-in-nova? | 15:14 |
| mriedem | efried: we have a non-voting job on all nova changes | 15:14 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Do not run tempest.scenario.test_network* tests in nova-next https://review.openstack.org/641416 | 15:14 |
| mriedem | ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa | 15:14 |
| mriedem | and i think there are a couple in the experimental queue as well for multinode and neutron stuff | 15:14 |
| efried | aha, okay. | 15:14 |
| efried | mriedem: background: I finally bit the bullet and started hacking up a way for ironicclient to use the ksa adapter we go through all the trouble of building up on the nova side, only to throw it away. | 15:15 |
| sean-k-mooney | efried: the other way to do it is a DNM change to ironic that depends on the nova change | 15:15 |
| *** awalende has quit IRC | 15:15 | |
| sean-k-mooney | that will runn the ironcis jobset and pull in the relevent nova patch | 15:15 |
| efried | sean-k-mooney: and that exercises the nova side? | 15:16 |
| *** awalende has joined #openstack-nova | 15:16 | |
| sean-k-mooney | assuming they have jobs that boot ironic instances via nova | 15:16 |
| sean-k-mooney | it just depends on what you want to test | 15:16 |
| efried | ironicclient: https://review.openstack.org/641169 | 15:16 |
| efried | nova: https://review.openstack.org/#/c/641170/ | 15:16 |
| efried | in case you care. | 15:16 |
| mriedem | melwitt: looks like functional test failures at the end of your counting quotas series are real https://review.openstack.org/#/c/638324/ | 15:16 |
| mriedem | efried: the one thing i'm not sure about is if that nova change will pull in the ironicclient code dependency or if it will pull in ironicclient from upper-constraints | 15:17 |
| mriedem | might depend on if the job is zuul v3-ified | 15:17 |
| efried | o | 15:18 |
| mriedem | http://logs.openstack.org/70/641170/1/check/ironic-tempest-ipa-wholedisk-bios-agent_ipmitool-tinyipa/0e5cd32/controller/logs/pip2-freeze.txt.gz indicates it doesn't | 15:18 |
| *** agopi has quit IRC | 15:18 | |
| mriedem | python-ironicclient==2.6.0 | 15:18 |
| sean-k-mooney | efried: mriedem looks like the ironic base job is a zullv3 native job but it does not list openstack/nova as a required project | 15:18 |
| mriedem | doesn't pull in the source dep | 15:18 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Block automatic transport_url update for cell0 https://review.openstack.org/605414 | 15:18 |
| efried | well crap | 15:19 |
| mriedem | so, | 15:19 |
| mriedem | the "legacy" way to do this is a devstack change which depends on the nova/ironicclient changes and you configure devstack to pull in the client from source | 15:19 |
| mriedem | assuming devstack has an ironic job... | 15:19 |
| efried | and if it doesn't, make the devstack change and then make a nova change that depends on it. | 15:20 |
| mriedem | or, your nova change depends on a devstack change that pulls in ironicclient from source which depends on the ironicclient patch | 15:20 |
| mriedem | yeah | 15:20 |
| efried | yeah | 15:20 |
| efried | whee | 15:20 |
| *** awalende has quit IRC | 15:20 | |
| mriedem | https://github.com/openstack-dev/devstack/blob/master/stackrc#L345 | 15:20 |
| sean-k-mooney | efried: i can mock up a quick test patch that ill pull in one of the ironic jobs and force it to use nova and ironcic client form source if iyou like | 15:21 |
| *** sridharg has quit IRC | 15:21 | |
| efried | sean-k-mooney: If you have that at your fingertips, that would be great. It would take me a while to figure out. | 15:21 |
| sean-k-mooney | efried: if you list a project in required-project and it auto adds it to LIBS_FROM_GIT and zuul follow the depends-on stuff to pull in the requried patches | 15:22 |
| sean-k-mooney | efried: it should only take me a minute to create one | 15:23 |
| *** tbachman has quit IRC | 15:28 | |
| *** moshele has quit IRC | 15:29 | |
| *** xek_ has joined #openstack-nova | 15:30 | |
| *** moshele has joined #openstack-nova | 15:30 | |
| *** xek__ has joined #openstack-nova | 15:32 | |
| *** xek_ has quit IRC | 15:35 | |
| *** Sundar has joined #openstack-nova | 15:37 | |
| *** udesale has quit IRC | 15:37 | |
| openstackgerrit | Jose Castro Leon proposed openstack/nova master: Extend volume for libvirt network volumes (RBD) https://review.openstack.org/613039 | 15:38 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] tempest client testing jobs https://review.openstack.org/641421 | 15:39 |
| sean-k-mooney | efried: i think ^ will do what you want | 15:39 |
| *** moshele has quit IRC | 15:39 | |
| efried | ... | 15:40 |
| sean-k-mooney | i proably should have also dropt the tox jobs but they are quick | 15:41 |
| efried | sean-k-mooney: Cool. How is that pulling in the ironicclient patch? | 15:41 |
| sean-k-mooney | it is based on your nova patch an your nova patch has a depends on | 15:42 |
| efried | right, but I thought we established that the depends-on wasn't working? | 15:42 |
| sean-k-mooney | so because i added the ironic client to the require projects it will pull it in when zuul prepares the git repos and add it ot devstack LIBS_FROM_GIT list | 15:42 |
| efried | or is that magic instated by virtue of this having been done in .zuul.yaml? | 15:42 |
| efried | okay, nice. | 15:43 |
| sean-k-mooney | we should still check when it actully runs just to be sure but it should work | 15:43 |
| sean-k-mooney | i might quickly resping to drop the tempest full and greade job too since i has not started in zuul yet | 15:44 |
| sean-k-mooney | well its ther ebut no vms yet | 15:45 |
| efried | sean-k-mooney: Thanks for doing that. I have plenty of work left to do on the code side, but if I can at least get it running, then I have a way to debug. | 15:45 |
| openstackgerrit | sean mooney proposed openstack/nova master: [DNM] tempest client testing jobs https://review.openstack.org/641421 | 15:46 |
| sean-k-mooney | there we go i missed doce but that will be much more efficent | 15:48 |
| efried | thanks sean-k-mooney | 15:48 |
| sean-k-mooney | and no worries hopefully that works the way we want it too | 15:48 |
| sean-k-mooney | if it does i can move those jobs into the experimental pipline or replace the current non voting jobs we have | 15:49 |
| mtreinish | mriedem: hmm, well getting the time series graph took longer than I thought (I must be rusty): https://blog.kortar.org/wp-content/uploads/2019/03/v6_network_total-1.png | 15:49 |
| *** lpetrut has joined #openstack-nova | 15:50 | |
| *** markvoelker has joined #openstack-nova | 15:50 | |
| mtreinish | we also must have dropped a test (or skip it in some configurations) which made the aggregation difficult after Sept 2018 | 15:50 |
| *** dtantsur|mtg is now known as dtantsur|afk | 15:50 | |
| mtreinish | I also box and whiskered each individual test: https://blog.kortar.org/wp-content/uploads/2019/03/v6_network_scenario_run_times.png | 15:50 |
| mtreinish | it's definitely way too slow | 15:51 |
| *** pcaruana has quit IRC | 15:53 | |
| mriedem | geez yeah | 15:53 |
| mriedem | that aligns with the 2100 seconds i saw in that timed out job | 15:53 |
| mriedem | so i posted https://review.openstack.org/#/c/641416/ as a short-term workaround for nova-next | 15:53 |
| mriedem | i can't really decide if we should whitelist the scenario modules we care about in the nova-next job, because then new ones could be added and we'd miss them, but at the same time there is probably more we should blacklist | 15:54 |
| *** janki has quit IRC | 15:55 | |
| mriedem | i remember the good old days when tempest took 45 minutes | 15:55 |
| mriedem | the whole job | 15:55 |
| mtreinish | man, that was a long time ago | 15:55 |
| dansmith | stephenfin: can you answer my questions about the test assertions in this? https://review.openstack.org/#/c/624592 | 15:55 |
| stephenfin | dansmith: sure | 15:55 |
| mriedem | granted, the gate was fubar'd on a weekly basis, but still | 15:55 |
| mriedem | it was fubar'ed FAST | 15:56 |
| dansmith | stephenfin: I was thinking it would be nice to merge another good chunk of those this cycle, but... | 15:56 |
| sean-k-mooney | mriedem for what its work im hoping to get the os-vif gates down to that by droping the keystone and cinder apis test. | 15:56 |
| mtreinish | keystone tests take like zero time | 15:57 |
| mtreinish | dropping cinder will save a lot of time | 15:57 |
| mtreinish | sean-k-mooney: if there is no integration between os-vif and cinder I was just disable cinder all together in those jobs | 15:57 |
| sean-k-mooney | mtreinish: ya but os-vif is a lib that never talk to any other openstack service | 15:57 |
| mtreinish | s/was/would/g | 15:57 |
| mriedem | that's why i dropped the non-compute api tests from the nova-next job | 15:57 |
| sean-k-mooney | mtreinish: yes im going ot | 15:58 |
| sean-k-mooney | i think in the os-vif jobs i might keep the compute and networking api tests + the senario test for now and see what the job times look like | 15:59 |
| sean-k-mooney | if they are under an hour ill be happy if i can get down to 45 mins or so even more so | 16:00 |
| *** erlon has quit IRC | 16:00 | |
| *** thgcorrea has quit IRC | 16:01 | |
| mtreinish | well if you disable cinder (not just blacklist the cinder api tests) it won't run a good chunk of the scenarios | 16:02 |
| mtreinish | and those scenarios are individually among the slowest tests in a job | 16:02 |
| *** erlon has joined #openstack-nova | 16:03 | |
| *** pcaruana has joined #openstack-nova | 16:06 | |
| *** hongbin has joined #openstack-nova | 16:07 | |
| *** ttsiouts has quit IRC | 16:08 | |
| *** ttsiouts has joined #openstack-nova | 16:09 | |
| *** sdake has quit IRC | 16:09 | |
| *** ttsiouts has quit IRC | 16:11 | |
| stephenfin | dansmith: Done-zo https://review.openstack.org/#/c/624592/8 | 16:11 |
| *** ttsiouts has joined #openstack-nova | 16:11 | |
| *** sdake has joined #openstack-nova | 16:13 | |
| *** IvensZambrano has joined #openstack-nova | 16:14 | |
| melwitt | mriedem: ack, will fix. I think it's bc that patch changed things from counting instances => instance mappings, that test is probably only creating instance records and not mapping records | 16:17 |
| *** _alastor_ has joined #openstack-nova | 16:17 | |
| *** tssurya has quit IRC | 16:18 | |
| *** mrch has quit IRC | 16:21 | |
| *** markvoelker has quit IRC | 16:22 | |
| dansmith | mriedem: probably need someone to make a call on breaking potentially n-net-only tests with this: https://review.openstack.org/#/c/624592 | 16:23 |
| mriedem | i know some of the converted privsep stuff is going untested because testing the privsep stuff is hard... | 16:24 |
| mriedem | i remember wresting with trying to mock around that stuff at one point | 16:24 |
| mriedem | i don't remember the details | 16:24 |
| mriedem | *wrestling even | 16:24 |
| mriedem | i'd rather not make that call today... | 16:25 |
| dansmith | just seemed like some feature-y patches we could make independent progress on, and they were just refreshed, but.. I got stopped at the first one | 16:26 |
| sean-k-mooney | dansmith: even in the nova-net case some of that code is likely dead code as os-vif will be used in most cases | 16:27 |
| *** ccamacho has quit IRC | 16:27 | |
| dansmith | sean-k-mooney: understand, but it's just hard to get on board with orphaning code with tests.. if it's dead code, then maybe we should remove it | 16:28 |
| mriedem | in other fun news, when post live migration fails we don't set the instance to error status or reset the task_state to None apparently http://logs.openstack.org/43/635343/4/check/tempest-slow-py3/a2497ae/job-output.txt.gz#_2019-03-06_05_29_44_910801 | 16:28 |
| sean-k-mooney | the gateway code is likely still used | 16:28 |
| sean-k-mooney | mriedem: ya i have seen that happen a few times | 16:30 |
| sean-k-mooney | dansmith: by the way we were waiting mainly for cern to move to cellsv2 only before droping nova-networks | 16:31 |
| *** yonglihe has quit IRC | 16:31 | |
| sean-k-mooney | so can we drop it entirly in train? | 16:31 |
| dansmith | sean-k-mooney: they're not fully off it, AFAIK | 16:31 |
| sean-k-mooney | we are not going to froward port it to python3 im assuming | 16:32 |
| sean-k-mooney | dansmith: ok we should check with them and see. | 16:32 |
| *** ileixe has quit IRC | 16:33 | |
| stephenfin | There was a mail about this on openstack-discuss a few weeks back | 16:33 |
| stephenfin | I recall they said they were planning to migrate *everything* to neutron, but I can't recall the timeline | 16:33 |
| dansmith | yeah, that's been the plan for a while, I just haven't heard that they have | 16:34 |
| dansmith | either way, removal is non-trivial so cern being off it isn't the only barrier :) | 16:34 |
| stephenfin | Sounds like a challenge | 16:34 |
| stephenfin | Does anyone know if config drives rely on the metadata service being up? I'm assuming 'nova-compute' handles stuff by itself but haven't dived too deep yet | 16:35 |
| melwitt | mriedem: I have a osc-placement change up for a batch command for setting allocation ratios per aggregate in placement, if you're interested in reviewing https://review.openstack.org/640898 | 16:35 |
| sean-k-mooney | well i think we might be able to delegate some of the logic to os-vif in the interim | 16:35 |
| * stephenfin would ask mikal if he were around | 16:35 | |
| sean-k-mooney | for example i want to pull the trusted vf stuff into a sriov plugin in os-vif | 16:36 |
| dansmith | sean-k-mooney: if you think those gateway things are still used can you comment? | 16:36 |
| *** imacdonn has quit IRC | 16:36 | |
| *** imacdonn has joined #openstack-nova | 16:36 | |
| dansmith | sean-k-mooney: I think there's also some default firewalling of nova services that still use a lot of the linux_net ip.* commands, right? like punching holes for nova-api, metadata, etc | 16:36 |
| *** macza has joined #openstack-nova | 16:39 | |
| sean-k-mooney | dansmith: ya there is a bunch of firwalling stuff. i would prefer not to migrate that code to os-vif but we migth be able to clean it up abit | 16:42 |
| dansmith | sean-k-mooney: yeah, not saying we should, I'm saying we shouldn't just be randomly dropping tests for stuff we convert to privsep because "it's hard" and "it might be only for n-net" | 16:43 |
| openstackgerrit | melanie witt proposed openstack/nova master: Use instance mappings to count server group members https://review.openstack.org/638324 | 16:44 |
| sean-k-mooney | ill take a look and see if its used and ya. | 16:44 |
| sean-k-mooney | i can take a look at some to the privisep changes and see if i can convert them or help if needed too | 16:44 |
| sean-k-mooney | but that will be after ff | 16:45 |
| sean-k-mooney | i am happy to work with stephen on paying donw that technical debt | 16:45 |
| dansmith | I was just thinking that these feature-y patches might be a few easy wins to get in before FF, several were +2 already and just refreshed | 16:46 |
| dansmith | but clearly I was wrong :) | 16:46 |
| sean-k-mooney | actully they are more or less indepenate so maybe we can change the odering of some of the patches that dont drop tests/nova-net stuff | 16:47 |
| *** sdake has quit IRC | 16:48 | |
| sean-k-mooney | e.g. merge the stuff that is a driect port/obviously correct and deffer the rest | 16:48 |
| *** sdake has joined #openstack-nova | 16:48 | |
| dansmith | yep, I +W'd one above that first one | 16:49 |
| artom | stephenfin, the config drive is independant of the metadata API | 16:50 |
| stephenfin | artom: So put configuration in nova-compute? | 16:50 |
| stephenfin | I'm thinking on the vendordata stuff, specifically | 16:50 |
| dansmith | stephenfin: what does that mean? | 16:51 |
| artom | Yeah, what do you want to put in nova-compute? | 16:51 |
| dansmith | stephenfin: config for how to reach the vendordata server/ | 16:51 |
| stephenfin | dansmith: If I was using DynamicJSON, yeah | 16:51 |
| dansmith | stephenfin: config_drive literally just imports the python module that generates the metadata response and uses it to dump things into a disk image | 16:52 |
| dansmith | stephenfin: then yeah, nova-compute would need that config | 16:52 |
| sean-k-mooney | artom: you can add force_config drive or something but you can also just enable config drive in the flavor/image | 16:52 |
| stephenfin | dansmith++ ta | 16:52 |
| stephenfin | I ended up down a rabbithole yesterday and ended up doing this https://review.openstack.org/#/c/640730/ and now I'm more confused than ever. Back to reviews | 16:52 |
| *** pcaruana has quit IRC | 16:55 | |
| mriedem | https://bugs.launchpad.net/nova/+bug/1818873 | 16:56 |
| openstack | Launchpad bug 1818873 in OpenStack Compute (nova) "When post_live_migration_at_destination fails the instance is not put into ERROR/None vm_state/task_state" [Medium,Triaged] | 16:56 |
| *** sdake has quit IRC | 16:56 | |
| mriedem | so if post_live_migration_at_destination fails early, the guest is on the dest but the instance db state is all f'ed up | 16:56 |
| mriedem | mdbooth: don't you have a similar bug to this ^? | 16:56 |
| sean-k-mooney | mriedem: yes | 16:57 |
| *** panda|ruck|flu is now known as panda|ruck|off | 16:57 | |
| mdbooth | The one I recall was the other way round. The issue is when post_live_migration fails on the *source* | 16:58 |
| artom | mriedem, maybe https://bugs.launchpad.net/nova/+bug/1628606? | 16:58 |
| openstack | Launchpad bug 1628606 in OpenStack Compute (nova) "live migration does not clean up at target node if a failure occurs during post migration" [Low,In progress] - Assigned to Artom Lifshitz (notartom) | 16:58 |
| sean-k-mooney | i brought that up in https://review.openstack.org/#/c/603844/ | 16:58 |
| artom | mriedem, I've been meaning to refactor the flow to make sure we save the instance state correctly in all cases | 16:59 |
| artom | I addressed the Cinder API fail case, there are more (like Neutron) | 16:59 |
| sean-k-mooney | artom: didnt you partial fix this already. we just 1 addional try finally block effectily to make sure we always set the instance host | 16:59 |
| openstackgerrit | Merged openstack/nova master: Support server create with ports having resource request https://review.openstack.org/636360 | 16:59 |
| sean-k-mooney | artom: you didnt address all the cinder issues | 16:59 |
| openstackgerrit | Merged openstack/nova master: objects: Store InstancePCIRequest.numa_policy in DB https://review.openstack.org/624444 | 16:59 |
| artom | sean-k-mooney, yeah https://review.openstack.org/#/c/611093/ | 16:59 |
| *** ivve has joined #openstack-nova | 17:00 | |
| sean-k-mooney | you missed on place where we call cinder that can raise an exception | 17:00 |
| artom | sean-k-mooney, yeah, it's entirely possible | 17:00 |
| artom | sean-k-mooney, it was meant as a bandaid until a more correct fix is done | 17:01 |
| *** ccamacho has joined #openstack-nova | 17:02 | |
| mriedem | this is clearly not...enterprise ready | 17:02 |
| mriedem | one could spend days cleaning up this mess | 17:02 |
| sean-k-mooney | artom: ya i know | 17:03 |
| sean-k-mooney | https://github.com/openstack/nova/blob/38c96993fd7163afe1a7c5f380af0d57a1bd2903/nova/compute/manager.py#L6736 | 17:03 |
| mdbooth | It seems more intuitive to me to update instance.host to the destination *immediately* we're actually running there, i.e. from the source at the top of _post_live_migration, but IIRC there are weird gotchas with that. | 17:03 |
| sean-k-mooney | that is that call im thinking of | 17:03 |
| mriedem | another clunker is we pass the Migration object around via the MigrateData which is pretty brittle, | 17:03 |
| artom | mriedem, what mdbooth said | 17:03 |
| mriedem | i'd like to just partial that like what mdbooth did with source_bdms | 17:03 |
| sean-k-mooney | it will hit the cinder attachments api and can raise an exception which means the vm is running on the dest but we never update the db | 17:04 |
| artom | I think just making sure we always update instance state the moment it's irreversibly on the destination is a good first step | 17:04 |
| sean-k-mooney | ya i wrote it down on my whiteboard after reviewing the other chage earlier in the week | 17:04 |
| *** eharney has quit IRC | 17:06 | |
| *** gyee has joined #openstack-nova | 17:06 | |
| mdbooth | You know how sometimes you don't understand a thing, but you remember understanding a thing while somebody was explaining it? I vaguely recall dansmith explaining why instance.host needs to be updated later. | 17:06 |
| dansmith | mdbooth: during migration? | 17:07 |
| mdbooth | dansmith: Yeah. | 17:07 |
| sean-k-mooney | dansmith: later in post_live_migration_at_destination to be percise | 17:07 |
| dansmith | you know how sometimes you don't understand a thing, but you remember that every time someone asks about it, you go re-re-re-research it and re-remind yourself why that thing is true? | 17:07 |
| mdbooth | dansmith: haha | 17:07 |
| artom | Presumably things count on instance.host still being the source | 17:08 |
| mdbooth | artom: Right. dansmith can be very convincing at times. | 17:08 |
| sean-k-mooney | it was releated to cleaning up the cinder and neutron port properly on the souces | 17:08 |
| sean-k-mooney | in anycase we can likely resolve that and make sure the db is consitent if we spend a little itme on it | 17:09 |
| artom | sean-k-mooney, heh, that's true for literally *all* of the bugs :D | 17:09 |
| sean-k-mooney | a good first step is proably jsut to break the fucntion into smaller functions so its eaier to grock | 17:10 |
| artom | sean-k-mooney, that too | 17:10 |
| artom | It's actually on our downstream squad backlog | 17:10 |
| sean-k-mooney | ya i know its on my personal white board backlog too for after i get done with sriov. | 17:10 |
| sean-k-mooney | i didnt wnat to go messing with the post live migrtation stuff in parallel | 17:11 |
| *** eharney has joined #openstack-nova | 17:13 | |
| *** lpetrut has quit IRC | 17:14 | |
| *** mrch has joined #openstack-nova | 17:15 | |
| *** ccamacho has quit IRC | 17:15 | |
| *** awalende has joined #openstack-nova | 17:17 | |
| *** lpetrut has joined #openstack-nova | 17:17 | |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/nova master: Add requested_networks to RequestSpec https://review.openstack.org/570201 | 17:19 |
| openstackgerrit | Theodoros Tsioutsias proposed openstack/nova master: Enable rebuild for instances in cell0 https://review.openstack.org/570203 | 17:19 |
| *** ccamacho has joined #openstack-nova | 17:19 | |
| *** markvoelker has joined #openstack-nova | 17:19 | |
| *** awalende has quit IRC | 17:22 | |
| *** ttsiouts has quit IRC | 17:24 | |
| *** ttsiouts has joined #openstack-nova | 17:25 | |
| *** ttsiouts has quit IRC | 17:29 | |
| sean-k-mooney | adrianc: mind if i fix https://review.openstack.org/#/c/620115/27/nova/compute/manager.py@89 and rebase everyting | 17:31 |
| *** lpetrut has quit IRC | 17:32 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: De-cruft compute manager live migration https://review.openstack.org/641449 | 17:33 |
| *** helenafm_ has quit IRC | 17:34 | |
| sean-k-mooney | is it my imaginaiton or does devstack no longer delete exsiting libvirt instances when it stacks? | 17:42 |
| sean-k-mooney | i keep hitting domain already exists errors lately if i dont delete instance before i unstack and stack | 17:42 |
| *** mmethot has quit IRC | 17:44 | |
| *** wolverineav has joined #openstack-nova | 17:44 | |
| *** mmethot has joined #openstack-nova | 17:45 | |
| cdent | sean-k-mooney: I haven't experienced that, I rarely created actual vms from devstack, but I'm not surprised if it is happening: maintenance of devstack as a _dev_ tool, rather than the thing that happens in the gate is getting nowhere near the attention it used to | 17:48 |
| sean-k-mooney | i think someone removed it | 17:49 |
| *** wolverineav has quit IRC | 17:49 | |
| sean-k-mooney | ill check the logs but after 5+ years of knowing that stack will clean up any vms you forgot about its hard to unlearn | 17:50 |
| dansmith | sean-k-mooney: I experienced that a while ago.. probably more than a year | 17:50 |
| dansmith | tbh, I'm not sure why it would do that for you, and it'd need to be real careful about any non-nova vms that were there | 17:50 |
| sean-k-mooney | well i always found it strange it did it on stack not unstack but i have only noticed it lately | 17:50 |
| dansmith | unstack is really not maintained, AFAIK | 17:50 |
| melwitt | meanwhile, I never knew it cleaned up anything because I always deleted mine or destroyed the entire env and start over. not much unstack/stack | 17:50 |
| dansmith | it's missing a ton of crap | 17:51 |
| sean-k-mooney | dansmith: it used to use the instance name prefix that you have cofigred to delete any vm that matched | 17:51 |
| dansmith | sean-k-mooney: okay, that's surprising to me, but ... | 17:51 |
| sean-k-mooney | dansmith: ya i have a todo to fix wsgi config cleanup | 17:51 |
| sean-k-mooney | there are a few other unstack/clean things i need to do. | 17:52 |
| sean-k-mooney | dansmith: it woudl be nice to keep it but make it conditional | 17:52 |
| bauzas | sean-k-mooney: yup, I saw the same behaviour, even with a ./clean.sh | 17:52 |
| *** markvoelker has quit IRC | 17:52 | |
| sean-k-mooney | one of the hack we did in the intel nfv ci to make it more reliable back when i was invoved in runing it is we never called unstack | 17:53 |
| sean-k-mooney | nodepool was going to delete the vm so it saved time and sometime the cinder loop device clean woudl randomly fail | 17:54 |
| *** wolverineav has joined #openstack-nova | 17:54 | |
| sean-k-mooney | if i find a commit that removed it for a reason i might revert and make it condional but off by default. | 17:55 |
| openstackgerrit | guang-yee proposed openstack/nova master: pass endpoint interface to Ironic client https://review.openstack.org/640879 | 17:55 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor pre-live-migration work out of _do_live_migration https://review.openstack.org/641453 | 17:57 |
| openstackgerrit | Ken'ichi Ohmichi proposed openstack/nova master: doc: cleanup pci.alias references https://review.openstack.org/640882 | 17:57 |
| *** lchabert has quit IRC | 17:58 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: doc: cleanup pci.alias references https://review.openstack.org/640882 | 17:59 |
| *** lchabert has joined #openstack-nova | 17:59 | |
| mriedem | efried: you want to just fast approve ^ since ken'ichi updated it and +2ed but his updated need tweaking | 17:59 |
| *** ccamacho has quit IRC | 18:00 | |
| *** lchabert has quit IRC | 18:01 | |
| *** wolverineav has quit IRC | 18:01 | |
| *** lchabert has joined #openstack-nova | 18:01 | |
| *** derekh has quit IRC | 18:03 | |
| sean-k-mooney | huh its still there https://github.com/openstack-dev/devstack/blob/master/lib/nova#L230-L239 | 18:04 |
| sean-k-mooney | i must be broken | 18:04 |
| *** psachin has quit IRC | 18:05 | |
| dansmith | heh that's a pretty brute-force thing.. would be wrong if you ever redefined the pattern to not be a prefix | 18:05 |
| dansmith | not that anyone would, but.. | 18:05 |
| sean-k-mooney | well people sometimes change the pattern to just use the uuid but ya its pretty dumb code | 18:06 |
| *** wolverineav has joined #openstack-nova | 18:06 | |
| *** wolverineav has quit IRC | 18:06 | |
| sean-k-mooney | that sad its 2-7 years old and mostly just worked so its not terible | 18:06 |
| *** wolverineav has joined #openstack-nova | 18:06 | |
| melwitt | mriedem: I'm going to review the rbd volume extend patch. anything else you'd recommend for review today? | 18:06 |
| sean-k-mooney | ill debug this later. | 18:07 |
| dansmith | I mean unlikely to change it on devstack | 18:07 |
| sean-k-mooney | right operators change it for reasons but devs dont really have a need too | 18:07 |
| eandersson | frickler, thanks will double check | 18:08 |
| sean-k-mooney | i was getting migration failure because i had stoped domains and got an error saying the domain already existed | 18:08 |
| sean-k-mooney | nova rolled back the live-migration and everything was fine | 18:09 |
| sean-k-mooney | ok finished testing the sriov migration stuff again. im going rebase it all to the head of master but it seams to work locally | 18:21 |
| *** ralonsoh has quit IRC | 18:28 | |
| *** Sundar has quit IRC | 18:31 | |
| mriedem | melwitt: not really, i'm just waiting for another run of the ceph change on that one | 18:40 |
| eandersson | frickler, that was indeed it | 18:41 |
| *** IvensZambrano has quit IRC | 18:42 | |
| melwitt | mriedem: ok, thanks. I just went through it, also lgtm | 18:43 |
| *** jaypipes has quit IRC | 18:44 | |
| mriedem | melwitt: the rebuild from cell0 stuff could probably use more eyes but i've already been over it in parts and continued to find issues, so i'm just not really comfortable with pushing for that on the last day | 18:45 |
| openstackgerrit | Merged openstack/nova master: Warn if group_policy is missing from flavor https://review.openstack.org/641289 | 18:46 |
| melwitt | mriedem: got it | 18:46 |
| openstackgerrit | sean mooney proposed openstack/nova master: Sep methods to free claimed and allocated devs https://review.openstack.org/616120 | 18:46 |
| openstackgerrit | sean mooney proposed openstack/nova master: Allow per-port modification of vnic_type and profile https://review.openstack.org/607365 | 18:46 |
| openstackgerrit | sean mooney proposed openstack/nova master: Add get_instance_pci_request_from_vif https://review.openstack.org/619929 | 18:46 |
| openstackgerrit | sean mooney proposed openstack/nova master: SR-IOV Live migration indirect port support https://review.openstack.org/620115 | 18:46 |
| openstackgerrit | sean mooney proposed openstack/nova master: libvirt: auto detach/attach sriov ports on migration https://review.openstack.org/629589 | 18:46 |
| dansmith | melwitt: what's the party line on the privsep stuff? | 18:46 |
| sean-k-mooney | stephenfin: ^ they are all rebased and typos adressed. | 18:47 |
| sean-k-mooney | stephenfin: i tested it before the rebase so it should all be working but ill restack and test again | 18:47 |
| melwitt | dansmith: I don't know the details but from earlier chat it looked like there's a lot of questions/issues. I'll re-read | 18:49 |
| dansmith | melwitt: well, there are questions on a class of them, but that's a small subset | 18:49 |
| dansmith | melwitt: re-ordering the patches would free up a bunch that are already +W | 18:49 |
| dansmith | I'm mostly interested in whether or not we care about the current half-baked situation we're in (I do) | 18:50 |
| *** markvoelker has joined #openstack-nova | 18:50 | |
| dansmith | they're not fixes, so kinda seem like they should go in front of FF to me, and they're not no-risk | 18:50 |
| dansmith | it'd be nice to make some more progress, especially since mikal just updated them (IMHO) | 18:50 |
| sean-k-mooney | there is what 7-10 left | 18:51 |
| melwitt | dansmith: ah, I see. I agree they should go in front of FF and I think it would be good to merge the straightforward ones by ore-ordering | 18:51 |
| *** spatel has joined #openstack-nova | 18:51 | |
| dansmith | melwitt: okay I hadn't seen you in that stack much so wasn't sure if you were ignoring them for specific reasons or just because they're not interesting.. maybe a ptl statement on the bottom one about "let's try to get a chunk of these merged before FF tomorrow" then ? | 18:52 |
| sean-k-mooney | oh 18 but 3 are +w and could merge if we reorder | 18:52 |
| dansmith | I guess I could also reorder a few for mikal and we could do some before he wakes up even | 18:52 |
| melwitt | dansmith: no, I recently went in there and approved one, which unblocked a small stack of them. haven't been ignoring them purposely | 18:52 |
| dansmith | sean-k-mooney: yeah and more proabably I just stopped walking up the stack | 18:52 |
| dansmith | okay | 18:52 |
| dansmith | lemme reorder them and see what I can free up easily | 18:53 |
| eandersson | mriedem, the soft-anti-affinity issue was because the keystone endpoint was still v2/ and not v2.1/ | 18:53 |
| eandersson | just fyi | 18:53 |
| mriedem | melwitt: why create a new test class for this? https://review.openstack.org/#/c/633350/12/nova/tests/functional/db/test_instance_mapping.py | 18:54 |
| melwitt | dansmith: I think they're valuable and want to get what progress we can there. I'll add a comment | 18:54 |
| mriedem | eandersson: ah..yeah we even have a troubleshooting entry for that in our docs from when oath upgraded to ocata | 18:54 |
| mriedem | eandersson: https://docs.openstack.org/nova/latest/admin/support-compute.html#requested-microversions-are-ignored | 18:54 |
| mriedem | eandersson: heh for that exactly same problem :) | 18:54 |
| mriedem | *exact | 18:55 |
| eandersson | It was just an odd one, as it worked before. =] | 18:55 |
| eandersson | Plus does not help that we haven't had a weekend yet :p | 18:55 |
| eandersson | Upgrading from Mitaka to Rocky... not easy lol | 18:55 |
| melwitt | mriedem: don't have a great reason. at first, I had modeled it after the bdm uuid addition/migration, which used a separate class for the test cases. and I wasn't sure if it was sufficiently different from the rest of the tests and should be separated or not. I can lump it with the rest, no problem | 18:56 |
| eandersson | Moving to Cells V2 has been super smooth so far thou | 18:56 |
| mriedem | eandersson: that's good to hear | 18:56 |
| mriedem | the cells thing that is | 18:56 |
| eandersson | :p | 18:56 |
| eandersson | We still have a lot of unaccounted performance issues that we are troubleshooting. | 18:56 |
| mriedem | too bad you're late to submit a "upgrading from mitaka to rocky at blizzzzard" talk at the summit | 18:56 |
| eandersson | haha | 18:56 |
| mriedem | its not just cells v2 you're biting off, it's also placement | 18:57 |
| mriedem | so you might be hitting some perf issues due to not scaling that out enough, | 18:57 |
| *** ivve has quit IRC | 18:57 | |
| mriedem | or nova-compute being way too chatty | 18:57 |
| eandersson | It actually looks pretty good so far on the nova side, but something is talking to Neutron a lot more. | 18:58 |
| dansmith | yeah that shouldn't be cells-related I don't think | 18:58 |
| eandersson | Yea - don't think so either | 18:58 |
| mriedem | are you doing a lot of live migrations from rocky? | 18:58 |
| dansmith | eandersson: glad to hear things aren't terrible :) | 18:58 |
| *** marst has quit IRC | 18:58 | |
| eandersson | We haven't dared to test live migration yet :p | 18:58 |
| mriedem | ok because the only big new neutron-related thing in rocky i can think of is https://specs.openstack.org/openstack/nova-specs/specs/rocky/implemented/neutron-new-port-binding-api.html | 18:59 |
| dansmith | eandersson: are you trying to move to multiple cells or just upgrading to the point where it's required for a single one? | 18:59 |
| eandersson | single one for now | 18:59 |
| dansmith | gotcha | 18:59 |
| eandersson | we want to move to multiple cells | 18:59 |
| eandersson | but wasn't a priority | 18:59 |
| dansmith | one thing at a time, for sure | 19:00 |
| melwitt | mriedem: replied on the review. I think maybe I was thinking since it's special and temporary handling, maybe keep it separate for more obvious removal later. but none of that is strong reasoning. just wasn't sure what to do | 19:00 |
| *** hongbin has quit IRC | 19:00 | |
| melwitt | eandersson: ++ also glad to hear things aren't terrible with cells :P | 19:01 |
| *** hongbin has joined #openstack-nova | 19:02 | |
| mriedem | melwitt: there is something that needs fixing here https://review.openstack.org/#/c/638574/ | 19:08 |
| melwitt | mriedem: ok, on it | 19:08 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move arping to privsep. https://review.openstack.org/624593 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move cleaning conntrack to privsep. https://review.openstack.org/624594 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move killing processes to privsep. https://review.openstack.org/635432 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Cleanup the exec_ebtables code a little. https://review.openstack.org/635434 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move slight bonkers IP management to privsep. https://review.openstack.org/624592 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move FP device creation and deletion to privsep. https://review.openstack.org/635433 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move iptables rule fetching and setting to privsep. https://review.openstack.org/636508 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move dnsmasq restarts to privsep. https://review.openstack.org/639280 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move router advertisement daemon restarts to privsep. https://review.openstack.org/639281 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move calls to ovs-vsctl to privsep. https://review.openstack.org/639282 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move setting of device trust to privsep. https://review.openstack.org/639283 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Move final bridge commands to privsep. https://review.openstack.org/639580 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Cleanup the _execute shim in nova/network. https://review.openstack.org/639581 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438 | 19:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Cleanup no longer required filters and add a release note. https://review.openstack.org/639826 | 19:09 |
| mriedem | makes you feel like a BIG MAN | 19:10 |
| * dansmith puffs out his chest | 19:10 | |
| sean-k-mooney | :) | 19:10 |
| *** marst has joined #openstack-nova | 19:10 | |
| melwitt | lol. | 19:11 |
| sean-k-mooney | efried: do you want the good news or the bad news | 19:12 |
| dansmith | I have some draft comments on one of melwitt's in that stack too, I need to go find and finish that | 19:12 |
| mriedem | i got you a puppy on the way home from work, | 19:12 |
| mriedem | but then i got hungry and ate it | 19:12 |
| dansmith | melwitt: okay bottom four of that privsep set are +W | 19:13 |
| dansmith | assuming I didn't break anything.. the rebases were not bad, but not trivial | 19:13 |
| melwitt | dansmith: cool, thanks for doing that | 19:13 |
| dansmith | aye | 19:13 |
| melwitt | I made a comment on the old bottom patch to explain | 19:13 |
| dansmith | oh okay | 19:13 |
| sean-k-mooney | efried: the good news is your change passed the fucntional job that runs the ironic api test and and python-ironclinet was installed from git | 19:14 |
| sean-k-mooney | efried: the bad news is the tempest job that ran teh senario tests failed with this error http://logs.openstack.org/21/641421/2/check/nova-ironic-client-tempest/8faf490/controller/logs/screen-n-cpu.txt.gz?level=TRACE#_Mar_06_18_04_20_985307 | 19:14 |
| *** wolverineav has quit IRC | 19:14 | |
| sean-k-mooney | efried: anyway you should be able to use that to test your chagnes looks like your just need to set the ironic api version | 19:17 |
| *** markvoelker has quit IRC | 19:22 | |
| openstackgerrit | Chris Friesen proposed openstack/nova master: Split up _is_storage_shared_with() in libvirt driver https://review.openstack.org/639933 | 19:24 |
| openstackgerrit | Chris Friesen proposed openstack/nova master: Add emulated TPM support to Nova https://review.openstack.org/631363 | 19:24 |
| *** Sundar has joined #openstack-nova | 19:24 | |
| cfriesen | ignore these, debugging something that fails in zuul but works locally. ^ | 19:24 |
| *** wolverineav has joined #openstack-nova | 19:25 | |
| mriedem | melwitt: ok i think dansmith and i are both up through the instance mapping user_id data migration patch, so you can probably start addressing stuff and pushing | 19:26 |
| *** wolverineav has quit IRC | 19:26 | |
| *** wolverineav has joined #openstack-nova | 19:26 | |
| melwitt | mriedem: ack, reading through | 19:27 |
| cfriesen | mriedem: just wanted to say thanks for all the reviews in the past week. you're a machine. | 19:29 |
| *** wolverineav has quit IRC | 19:29 | |
| *** wolverineav has joined #openstack-nova | 19:30 | |
| mriedem | cfriesen: yw | 19:31 |
| mriedem | i'm also having a hard time not making an inappropriate machine-related comment here | 19:32 |
| mriedem | feels like a trap | 19:32 |
| dansmith | haha | 19:32 |
| dansmith | I've been thinking | 19:32 |
| dansmith | all I came up with that was appropriate was "that's why his hugs are so cold" | 19:32 |
| mriedem | but i'd have to hug for anyone to know that | 19:32 |
| dansmith | I can tell from a distance | 19:32 |
| melwitt | pff lol | 19:34 |
| mriedem | we'll also need to think about what we do about counting for instance mappings that have queued_for_delete=None (not yet migrated) | 19:35 |
| mriedem | i think i would tend to count those since we do'nt know if the instance is deleted, so it's safer to be conservative | 19:35 |
| *** wolverineav has quit IRC | 19:35 | |
| melwitt | yeah, was just reading through that | 19:36 |
| dansmith | I haven't gotten to any of the actual counting stuff | 19:36 |
| dansmith | I figure best case, | 19:36 |
| dansmith | we merge the data model and migration stuff | 19:36 |
| dansmith | so we can get the things migrated in stein and then we're looking at train for the actual counting stuff | 19:36 |
| dansmith | as I haven't looked over any of that other than to go answer one question for myself | 19:36 |
| melwitt | +1 my initial thought is to be conservative and count the queued_for_delete=None ones | 19:37 |
| mriedem | yeah i haven't gotten into the api changes that actually use the new counting stuff, and at this rate it's too late for me to be comfortable with that, | 19:37 |
| mriedem | so i'd be happy to get the data model change and migration in stein | 19:38 |
| mriedem | so there is less to migrate in train | 19:38 |
| dansmith | yeah, having it migrated in stein before makes that stuff a lot better | 19:38 |
| melwitt | I can't argue with that. it's a bummer, but I take responsibility for not having this all ready for review earlier in the cycle | 19:38 |
| melwitt | I appreciate the review you're doing now and glad to be able to land some of it because of it | 19:39 |
| melwitt | s/some of it/the migration stuff/ | 19:40 |
| *** wolverineav has joined #openstack-nova | 19:41 | |
| *** xek has joined #openstack-nova | 19:42 | |
| melwitt | I don't know why, but the whole unset non-nullable ovo field thing really confuses me :x | 19:44 |
| *** xek has quit IRC | 19:44 | |
| *** xek_ has joined #openstack-nova | 19:44 | |
| dansmith | I don't know why either :) | 19:45 |
| *** xek__ has quit IRC | 19:45 | |
| melwitt | :P | 19:45 |
| *** wolverineav has quit IRC | 19:47 | |
| mriedem | melwitt: i'm +2 on the rbd extend volume change now https://review.openstack.org/#/c/613039/ | 19:50 |
| melwitt | +W | 19:51 |
| dansmith | melwitt: comment for you on the base patch.. I dare not -1 against that raft of positive votes, but.. genuine question | 19:56 |
| *** wolverineav has joined #openstack-nova | 19:56 | |
| * melwitt looks | 19:56 | |
| melwitt | dansmith: good question -- I think probably so. we added indexes for the quotas re-work in pike | 19:58 |
| dansmith | YES | 19:58 |
| dansmith | oh sweet sweet -1 vote | 19:58 |
| melwitt | hah | 19:58 |
| melwitt | I'd ask jaypipes to confirm but he's not around atm | 19:58 |
| *** zhubx has quit IRC | 19:59 | |
| *** zhubx007 has joined #openstack-nova | 19:59 | |
| *** jding1__ has joined #openstack-nova | 19:59 | |
| *** cdent_ has joined #openstack-nova | 19:59 | |
| dansmith | it should be free to add since it's defaulted to null at add time, so .. I can't really imagine we don't want one | 19:59 |
| *** lbragstad has quit IRC | 20:00 | |
| *** tosky has quit IRC | 20:00 | |
| *** lbragstad has joined #openstack-nova | 20:00 | |
| *** tosky has joined #openstack-nova | 20:01 | |
| melwitt | yeah...I'll see what the indexes look like for the current counting stuff we have and model after that. I don't know that much about database stuff | 20:01 |
| *** jding1_ has quit IRC | 20:02 | |
| *** cdent has quit IRC | 20:02 | |
| *** cdent_ is now known as cdent | 20:02 | |
| dansmith | should be fairly straightforward..nothing really to configure afaik | 20:04 |
| openstackgerrit | Merged openstack/nova master: Test live migration with config drive https://review.openstack.org/567860 | 20:06 |
| melwitt | dansmith: IIUC, we would need two right? one on project_id and one that is on project_id and user_id? | 20:06 |
| dansmith | I'm not sure you need both, but maybe.. I guess I would think one on each is as good as one on (a) and one on (a,b) but not positive | 20:07 |
| dansmith | I was focused on the userid bit, but since I haven't really looked at how this is used, I wasn't thinking about it being queried as (a,b) | 20:08 |
| dansmith | so yeah maybe.. Guess we do need some expertise | 20:08 |
| melwitt | I think I'm thinking that because of what jaypipes recommended to be done on the consumers table: https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/api_models.py#L646-L648 | 20:08 |
| melwitt | if the queries are going to be project_id and project_id + user_id | 20:09 |
| dansmith | right, but there isn't a bare one on project there, just composite ones | 20:09 |
| dansmith | but yeah I dunno | 20:09 |
| melwitt | yeah, true | 20:09 |
| *** igordc has joined #openstack-nova | 20:10 | |
| melwitt | I'm not sure there's another example. but yeah the pattern is going to be (a) and (a,b) | 20:10 |
| efried | mriedem: done | 20:16 |
| efried | sean-k-mooney: not at all surprised it failed, good deal that it's hitting the changes. Thanks again. | 20:16 |
| * melwitt casts summon jaypipes | 20:18 | |
| sean-k-mooney | efried: it looks like the functional job that runs the ironic api tempest test dose not help so you could drop that job | 20:18 |
| efried | sean-k-mooney: I guess it "proves" (or at least suggests) that I didn't f up backward compatibility | 20:19 |
| efried | assuming of course that the func tests actually exercise the paths I dorked with. | 20:19 |
| *** Sundar has quit IRC | 20:19 | |
| *** markvoelker has joined #openstack-nova | 20:19 | |
| sean-k-mooney | ya i guess. | 20:19 |
| sean-k-mooney | im just going to go grab some gorceries. ill be back online a bit later | 20:21 |
| *** wolverineav has quit IRC | 20:21 | |
| efried | wow, grocery stores are open after 6pm? Things have changed since I lived over there:) | 20:22 |
| cdent | efried: the best part is that the sign says "open 24 hours" | 20:22 |
| cdent | but it means except saturday evening or sunday | 20:22 |
| efried | heh | 20:23 |
| cdent | I have anger about that because there was this one, not at band camp, when I needed to pee | 20:23 |
| cdent | and because I saw the "24 hours" I had let myself relax that little bit you do when you know that safety approaches | 20:23 |
| cdent | but no | 20:23 |
| efried | mriedem: Popping https://review.openstack.org/#/c/639854/ off the stack - did you say you were going to address both issues in a fup (and therefore consider this one done)? | 20:25 |
| efried | if you're loaded I can do it, no biggie | 20:26 |
| efried | (and by loaded I mean...) | 20:26 |
| mriedem | i can't hardly stand | 20:27 |
| mriedem | so i was going to start working on a fup patch for some of my comments in the earlier reshaper patches and then stack yours on top of that, but then ran out of steam to start doing that yesterday, | 20:27 |
| mriedem | and really kind of need to be working more on the cross-cell stuff | 20:27 |
| mriedem | so if you're looking for fun things to fup on, go nuts | 20:28 |
| efried | fup my nuts | 20:29 |
| *** betherly has joined #openstack-nova | 20:31 | |
| openstackgerrit | Chris Friesen proposed openstack/nova master: Add missing libvirt exception during device detach https://review.openstack.org/641480 | 20:33 |
| cfriesen | melwitt: this is for the bug you took a look at a few weeks ago. ^ | 20:34 |
| mriedem | whoa | 20:34 |
| melwitt | cfriesen: ack | 20:35 |
| melwitt | on my list | 20:35 |
| *** igordc has quit IRC | 20:35 | |
| *** wolverineav has joined #openstack-nova | 20:37 | |
| *** betherly has quit IRC | 20:41 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Test proper allocation of devices during reshape https://review.openstack.org/639854 | 20:42 |
| *** cdent has quit IRC | 20:42 | |
| efried | mriedem: There's that one with the implicit comma thingy removed. | 20:42 |
| efried | now fupping helper method for mdev name | 20:42 |
| mriedem | efried: did you see my comment about that earlier in the series? | 20:44 |
| mriedem | https://review.openstack.org/#/c/599208/18/nova/virt/libvirt/driver.py@6835 | 20:45 |
| efried | yes | 20:45 |
| mriedem | i think we should just make _get_mediated_device_information handle the dash to underscore conversion | 20:45 |
| mriedem | so the caller (like your test) doesn't need to care | 20:45 |
| efried | ack | 20:45 |
| mriedem | do you want to just also fix those temporarily/PR typos in the test code in the same change as your test wrinkle or a new generic follow up? | 20:46 |
| efried | mriedem: one caller only has the name. Want the method to be smart and figure out which one it got, or take one of two kwargs, or... | 20:46 |
| mriedem | i don't follow | 20:47 |
| efried | _get_mediated_device_information is called in two places. In one of them, the one you pointed out, I have the UUID and convert to the name, which is the part that sucks. In the other, I only have the name and would have to do the reverse if I wanted the arg to be uuid. | 20:48 |
| mriedem | i guess you could add a helper method if that's more clear | 20:48 |
| *** mikal has joined #openstack-nova | 20:48 | |
| mriedem | i figured at the start of _get_mediated_device_information it would just dev_name = dev_name.replace('-', '_') | 20:49 |
| mriedem | and then it doesn't matter | 20:49 |
| efried | the method uses both the name and the uuid. I think a pair of helpers might be clearest. | 20:50 |
| mriedem | oh you're talking about this right? https://review.openstack.org/#/c/599208/18/nova/virt/libvirt/driver.py@6161 | 20:50 |
| mriedem | whatever, a simple _get_mdev_name(uuid) is also easy | 20:51 |
| *** betherly has joined #openstack-nova | 20:51 | |
| *** markvoelker has quit IRC | 20:53 | |
| *** tesseract has quit IRC | 20:54 | |
| *** betherly has quit IRC | 20:55 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: FUP for test_reshape https://review.openstack.org/641482 | 21:01 |
| efried | mriedem: gathered them together ^ | 21:01 |
| efried | 'cept I seem to have f'ed that up somehow. | 21:02 |
| mriedem | dos2unix nice | 21:03 |
| efried | whaaa | 21:04 |
| efried | oh | 21:04 |
| efried | thought you were saying my line feeds were wrong | 21:04 |
| *** erlon has quit IRC | 21:05 | |
| mriedem | so is this ok or are tests failing? | 21:06 |
| efried | failing. We must be monkey patching libvirt/utils somewhere | 21:07 |
| efried | f it, I'll move the utils into the driver. | 21:07 |
| mriedem | fakelibvirt... | 21:07 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/tests/unit/virt/libvirt/fake_libvirt_utils.py | 21:07 |
| efried | that's... crazy | 21:09 |
| mikal | There's heaps of that stuff from back before we understood how unit testing worked. | 21:10 |
| mikal | Most of its removable with conversion to mock patches now. | 21:10 |
| mikal | (i.e. that's now been done for most of nova-net because I like doomed projects) | 21:10 |
| *** betherly has joined #openstack-nova | 21:11 | |
| efried | clearly | 21:11 |
| mikal | *shrug* That code is 8 years old and no one has ever been offended enough to clean it up, so we're all complicit in it being like that. | 21:12 |
| openstackgerrit | Eric Fried proposed openstack/nova master: FUP for test_reshape https://review.openstack.org/641482 | 21:12 |
| efried | oh ffs | 21:12 |
| mikal | It made sense at the time with the tools available then. | 21:12 |
| openstackgerrit | Eric Fried proposed openstack/nova master: FUP for test_reshape https://review.openstack.org/641482 | 21:12 |
| efried | okay mriedem, hopefully that one is okay. | 21:13 |
| efried | I'm pretty offended mikal. But yeah, I don't know if I'm offended enough to clean it up. | 21:13 |
| mriedem | we're talking about fake_libvirt_utils being offensive right? | 21:15 |
| efried | yes | 21:15 |
| mriedem | not just mikal showing up and randomly offending people | 21:15 |
| efried | hahaha | 21:15 |
| dansmith | also, | 21:15 |
| dansmith | mikal is in his PJs currently, which may offend you | 21:15 |
| *** betherly has quit IRC | 21:15 | |
| efried | for some values of "PJs". | 21:16 |
| dansmith | and for all we know, australian PJs might be like american birthday suits | 21:16 |
| efried | swhat I'm sayin | 21:16 |
| efried | ...with a pouch | 21:16 |
| mikal | Look. I've only told one of you to fuck off so far today. I'm really not being _unusually_ offensive. | 21:17 |
| mriedem | i was going to say "as is his wont" but figured that might offend him, but i see it wouldn't | 21:18 |
| *** whoami-rajat has quit IRC | 21:18 | |
| efried | We can hug it out, mikal, but let's get one thing perfectly clear: | 21:19 |
| efried | No pants, no hug. | 21:19 |
| mriedem | i'll hug a nude man | 21:19 |
| mriedem | i don't care | 21:19 |
| mriedem | i'm told it will be cold on the receiving end though | 21:20 |
| dansmith | last week mriedem told me he prefers sweaty guys | 21:20 |
| dansmith | I don't really like where this is going | 21:20 |
| mriedem | i'll go back to the fire i just started in the tc channel | 21:20 |
| dansmith | to warm your icy embrace? | 21:21 |
| melwitt | dansmith: going through the change to add a test case for an instance mapping that has no instance (manual cleanup or purge) and was thinking, what should we do in that case? if we don't migrate the user_id, the record would keep getting found. should we migrate it to some sentinel value? or bother trying to get it from request spec? or other? | 21:28 |
| * mikal wanders off to get into his coding pants. | 21:29 | |
| *** eharney has quit IRC | 21:29 | |
| *** betherly has joined #openstack-nova | 21:31 | |
| cfriesen | are there docs on how to add new privsep commands in nova? | 21:31 |
| efried | cfriesen: You are right on time, mikal just put on pants | 21:32 |
| efried | if anyone has that answer, it's him. | 21:32 |
| mikal | Well, I actually _left_ to put on pants, but you get the idea. | 21:32 |
| mikal | cfriesen: do you want a pre-pants possibly grumpy answer, or a post pants probably still grumpy answer? | 21:32 |
| cfriesen | :) | 21:33 |
| cfriesen | I'm easy | 21:34 |
| mikal | cfriesen: https://www.madebymikal.com/adding-oslo-privsep-to-a-new-project-a-worked-example/ is an overly complete answer with way more detail than you care about. | 21:34 |
| mikal | cfriesen: but the short answer is its pretty trivial. Decide where in nova/privsep to put the method. Add the method with the decorator. Call the method from other places. Profit. | 21:34 |
| mikal | cfriesen: the only real warts are around style (don't pass in command lines, pass in args that are used to generate a command line); and unit testing hoops if efried notices that test coverage has dropped. | 21:35 |
| cfriesen | sweet, thanks. | 21:35 |
| mikal | cfriesen: I'd be happy to do a worked example in a blog post, but you're the first person to ever ask because no one loves me and my life is meaningless. | 21:35 |
| *** betherly has quit IRC | 21:35 | |
| cfriesen | (I'm trying to resolve a problem with emulated TPM and cold migration...the software TPM daemon runs as root but nova doesn't, and I need to copy the root-owned file to the dest node.) | 21:36 |
| mikal | cfriesen: although when I think of it, https://review.openstack.org/#/c/624593/ is probably close enough to documentation for now. Copy that. | 21:36 |
| mikal | Oh, so you just want to read a file with escalated permissions? | 21:36 |
| mikal | That's already implemented IIRC. | 21:36 |
| mikal | cfriesen: http://git.openstack.org/cgit/openstack/nova/tree/nova/privsep/path.py#n27 is what you want to read a file as root. | 21:37 |
| cfriesen | I suspect I need to modify SshDriver.copy_file() and RsyncDriver.copy_file() to optionally run with root privileges. | 21:37 |
| mikal | cfriesen: with maybe a http://git.openstack.org/cgit/openstack/nova/tree/nova/privsep/path.py#n73 thrown in for the destination file. | 21:38 |
| mikal | Ummmm. Why not just change the permissions on the file, sync, then change the permissions at the destination? | 21:39 |
| mikal | Instead of running the sync as root? | 21:39 |
| cfriesen | might work. It's tricky because root might not even be set up for passwordless ssh, and the resize code path doesn't do RPCs to set things up the way live migration does. | 21:40 |
| cfriesen | still studying the flow to figure out how to make it work | 21:40 |
| mikal | Ok, well I'm going to put on pants and stuff, but ping me if you need more random pointers. | 21:41 |
| *** betherly has joined #openstack-nova | 21:46 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: WIP: Make fake_libvirt_utils DIAF https://review.openstack.org/641489 | 21:50 |
| *** markvoelker has joined #openstack-nova | 21:50 | |
| efried | mikal, mriedem: let's see how that goes ^ | 21:50 |
| efried | it's stage 1 of ? btw | 21:50 |
| *** takashin has joined #openstack-nova | 21:50 | |
| efried | Probably better to explicitly mock only what's necessary to mock in only the places it's necessary. | 21:51 |
| efried | But I probably *am* too lazy for that. | 21:51 |
| *** betherly has quit IRC | 21:51 | |
| dansmith | melwitt: I think we should include it in found, not in done, so it's clear that it's not migrated.. a sentinel doesn't make sense, IMHO, and it really should be cleaned up via archive/purge I think | 21:53 |
| dansmith | melwitt: this is exactly why I wanted a test for it, so we make some decision and commit to it :) | 21:53 |
| *** igordc has joined #openstack-nova | 21:53 | |
| *** mchlumsky has quit IRC | 21:53 | |
| sean-k-mooney | so instead of mocking nova.virt.libvirt.driver.libvirt_utils you are now mocking a buch of functions in that module | 21:53 |
| sean-k-mooney | efried: ^ | 21:53 |
| efried | sean-k-mooney: right, instead of mocking a whole module with a whole module, I'm mocking the individual methods that are actually mocked. | 21:54 |
| sean-k-mooney | most of which were just pass.... | 21:54 |
| efried | which is fine | 21:54 |
| sean-k-mooney | ya | 21:54 |
| efried | sean-k-mooney: what pissed me off enough to do this was having to add new passthrough utils in fake_libvirt_utils at the same time as adding them to utils - see the predecessor patch. | 21:55 |
| efried | this makes it so you don't have to do that. | 21:55 |
| efried | which is a win. | 21:55 |
| sean-k-mooney | why not mock them in the testfucntion where they are actuly needed | 21:55 |
| efried | i.e. by default anything you write in utils is not mocked - but also won't break the world if you try to use it. | 21:55 |
| sean-k-mooney | e.g. instaad of a blanket mock | 21:55 |
| efried | Yeah, sean-k-mooney agree with that. Not having looked, that sounds like a huge effort. | 21:56 |
| efried | I assume the test surface of libvirt is enormous | 21:56 |
| efried | though from the little I've seen, we're nowhere near majority coverage. | 21:56 |
| efried | I guess we have a 'cover' job to tell us that, huh | 21:56 |
| sean-k-mooney | efried: you should assume that but it is not nessicaialy true. | 21:56 |
| efried | in any case, yes, stage 2-N might be to remove the mocks one at a time, see which tests break, and explicitly mock the removed thingy in those tests. | 21:57 |
| sean-k-mooney | well i guess we could do that first then localise the mocs | 21:57 |
| efried | just so. | 21:57 |
| mriedem | cfriesen: wrote this up based on what i pinged you about last night https://bugs.launchpad.net/nova/+bug/1818914 | 21:58 |
| openstack | Launchpad bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,Triaged] | 21:58 |
| melwitt | dansmith: oh, gotcha. consider it part of a "not completed" migration and operator must clean up to finish migration. yep, thank you | 21:58 |
| *** spatel has quit IRC | 21:59 | |
| sean-k-mooney | efried: also fun fact https://github.com/openstack/nova/search?q=assert_called_once&unscoped_q=assert_called_once | 21:59 |
| melwitt | dansmith, mriedem: something that sucks that I realize now is that if we don't land the counting part of the series, we need to restore the policy change behavior thing to account for what to do with server creates in the face of down cells... | 21:59 |
| dansmith | melwitt: yeah, you could also make a nova-status check for those I guess, and/or just do a warn-once on the migrate to say "yo dawg, this might never end if you don't do something" | 21:59 |
| sean-k-mooney | i leaned yesterday that assert_called_once was added in py36 | 21:59 |
| cfriesen | mriedem: oh yeah, meant to bring that up. | 21:59 |
| mriedem | melwitt: i don't know that we *need* to | 22:00 |
| *** phasespace has quit IRC | 22:00 | |
| mriedem | it's a latent issue since multi-cell support in pike right? | 22:00 |
| dansmith | melwitt: we already changed that? but it's no different than if we land none of this right? | 22:00 |
| mriedem | and counting quotas since pike | 22:00 |
| efried | sean-k-mooney: Yes, in base python, BUT we use mock-the-lib, which had it added/backported to much earlier. | 22:00 |
| melwitt | dansmith: ++ appreciate the ideas, as my brain is about fried | 22:00 |
| dansmith | right, nothing has changed since _that_ point right? | 22:00 |
| melwitt | mriedem: ok. I wasn't sure if it's need or not. ok | 22:00 |
| efried | sean-k-mooney: I remember learning that a couple years ago when I freaked out about same. | 22:00 |
| sean-k-mooney | efried: oh ok | 22:00 |
| sean-k-mooney | efried: ya so i freaked out a bit yesterday when i found that neutron have a hacking check that prevent you from using ig | 22:01 |
| sean-k-mooney | *it | 22:01 |
| melwitt | dansmith: yeah... yeah, that's true, current behavior is to ignore cell timeouts and such | 22:01 |
| melwitt | so it's no worse than today | 22:01 |
| dansmith | yeah, I thought you were saying we had landed something very out of order | 22:01 |
| dansmith | from where we are today, | 22:02 |
| dansmith | landing the data bit won't change anything I don't think | 22:02 |
| melwitt | no, just... I don't know, I was thinking did down cell support change something about that. but I don't think it did | 22:02 |
| dansmith | I surely hope not | 22:02 |
| melwitt | haha. just paranoid about things changing things | 22:03 |
| sean-k-mooney | change is scary | 22:03 |
| melwitt | it is :) especially in nova where you can change something and some weird side effect pops up somewhere else because of it | 22:04 |
| cfriesen | mriedem: might want to check whether the last paragraph in https://bugs.launchpad.net/nova/+bug/1498126 is still applicable | 22:05 |
| openstack | Launchpad bug 1498126 in OpenStack Compute (nova) "Inconsistencies with resource tracking in the case of resize operation." [Medium,Confirmed] | 22:05 |
| cfriesen | mriedem: that bug is old enough that the code may be fixed by now | 22:06 |
| dansmith | melwitt: sean-k-mooney: that's whyfor we have careful review and don't just slam everything in, IMHO | 22:06 |
| sean-k-mooney | change is scary? or so that we can all check our understading of the change and its wider effect so that changin a thing does not change other tings unless we ment too | 22:07 |
| dansmith | sean-k-mooney: no, the fear of weird side effects part.. that's why review from a heterogenous pool of experienced reviewers is important | 22:10 |
| mriedem | cfriesen: you mean this? "drop_move_claim looks up the new_flavor to compare it with the flavor that was used to track the migration, but on the source node it's certain to be the old_flavor. Thus as it stands now drop_move_claim (only ran on source nodes) doesn't do anything. Not a big deal, but we should probably fix it." | 22:10 |
| cfriesen | mriedem: yeah | 22:10 |
| cfriesen | I haven't checked whether it still applies | 22:10 |
| mriedem | i don't think that applies anymore, i'm not really sure what it's talking about | 22:10 |
| mriedem | when confirm_resize calls drop_move_claim, it's from the source and dropping usage for the old_flavor | 22:11 |
| sean-k-mooney | dansmith: yes because we all have different experice and approce solutions in different ways so we tend to see different things. | 22:11 |
| mriedem | when revert_resize (on the dest) calls rt.drop_move_claim, it's dropping the new_flavor usage from the dest host | 22:11 |
| mriedem | anyway, the bug i reported is something i hit in my cross-cell resize functional tests for confirm resize and should be really easy to recreate in general | 22:13 |
| mriedem | auto-confirm the resize and set the update_available_resource interval to a high number, like 10 minutes | 22:13 |
| cfriesen | mriedem: I think you're right and it doesn't apply anymore. | 22:16 |
| sean-k-mooney | dansmith: by the way the horrible code in https://review.openstack.org/#/c/624592/9/nova/network/linux_net.py | 22:17 |
| sean-k-mooney | we have the same code more or less in os-vif https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L115-L153 | 22:17 |
| *** betherly has joined #openstack-nova | 22:17 | |
| *** hongbin has quit IRC | 22:18 | |
| sean-k-mooney | we also have iptables code we inheritied form nova networks in the linux bridge plugin. | 22:20 |
| *** phasespace has joined #openstack-nova | 22:21 | |
| *** betherly has quit IRC | 22:22 | |
| *** markvoelker has quit IRC | 22:22 | |
| sean-k-mooney | i wouldnt be too surpisidn if we could replace alot of this code wide a call to plug on the linux bridge driver | 22:23 |
| sean-k-mooney | *surprised | 22:23 |
| mikal | efried: I effectively did this sort of thing with the nova-net privsep thing. I'd much rather end up with mock decorators on the methods which actually need them. Otherwise its super hard to untangle what calls what when something fails. I can take a look at this later, but only if people say one nice thing about me each day. Today's topic can be how pleasing my hairstyling is. Go. | 22:27 |
| efried | I saw a shampoo commercial the other day and thought of you. | 22:28 |
| efried | mikal: Did you see conversation above where I said: this is stage 1, so we don't have to have "passthrough mocks" in the fake module. Stage 2-N might be to remove the remaining mocks one at a time, see which tests break, and explicitly mock the removed thingy in those tests. | 22:29 |
| mikal | No not really. I was trying to remember how jeans work. | 22:29 |
| mikal | I didn't read all the scrollback, it looked long and boring. | 22:30 |
| efried | it was | 22:31 |
| sean-k-mooney | mikal: fyi dansmith reorderd some of you privsep removal code so the 4 patchs with +w can merge without being blocked by https://review.openstack.org/#/c/624592 | 22:31 |
| efried | mikal: that's why I extracted the salient bit of it for you. So you wouldn't have to mess up your hair. | 22:32 |
| efried | or... something. | 22:32 |
| mikal | sean-k-mooney: careful. Some of us said regrettable things when we noticed that but hadn't yet had our valium for the day. | 22:32 |
| *** hongbin has joined #openstack-nova | 22:33 | |
| *** macza has quit IRC | 22:36 | |
| sean-k-mooney | mikal: i assumed you would be happy that 4 more of them are currently makeing there way through the gate | 22:38 |
| sean-k-mooney | mikal: but on a releated note we have similar code in os-vif which i kindof want to kill at some point once i figure out why we do it | 22:38 |
| sean-k-mooney | https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_net.py#L115-L153 | 22:38 |
| *** awaugama has quit IRC | 22:39 | |
| efried | mriedem: Are we waiting for train for https://review.openstack.org/#/c/633855/ series? (SEV stuff) | 22:49 |
| mriedem | efried: yes | 22:51 |
| *** rcernin has joined #openstack-nova | 22:52 | |
| efried | mriedem: procedural hold so I don't keep looking at it and trying to decide whether to approve it? | 22:52 |
| *** wolverineav has quit IRC | 22:52 | |
| mikal | Oh wow, this is terrible. Did you know we sometimes mock away the fake for that fake_libvirt_utils thing? Why? Who did this?!? | 22:53 |
| mriedem | efried: i put a -W on the 2nd patch in the series which adds the capability which will now auto-generate the trait on the compute node resource provider, | 22:54 |
| mriedem | i avoided the -2 to not blow up jay's beautiful stats | 22:54 |
| efried | mriedem: ack | 22:54 |
| sean-k-mooney | mikal: do you know that we somtimes import the fake_libvirt direver for the unit tests in too fucntional tests | 22:54 |
| mriedem | efried: as for the one below that, i don't have an opinion | 22:54 |
| efried | mikal: You mean mock.patch(restore original method)? | 22:55 |
| efried | that's pretty fantastic | 22:55 |
| mikal | efried: no, its even more useless. We do a mock.patch to a noop for a function THAT IS ALREADY A NOOP. | 22:55 |
| efried | wow | 22:55 |
| sean-k-mooney | mikal: well actully that is proably new test that were written without knowing that ere was a global mock | 22:56 |
| sean-k-mooney | so those test are actully proably mocking just the things they need to mock | 22:56 |
| sean-k-mooney | which is good | 22:56 |
| sean-k-mooney | the fact it was already mocked is wasteful but it does not hurt anything | 22:57 |
| sean-k-mooney | by the way is there a reason the fake libvirt dirver is not with the other virt dirvers like the the Fake virt diriver is? | 22:58 |
| *** wolverineav has joined #openstack-nova | 22:58 | |
| mikal | sean-k-mooney: please don't ask logical questions, it just makes use feel even more inadequate | 22:59 |
| *** tkajinam has joined #openstack-nova | 23:01 | |
| *** dave-mccowan has quit IRC | 23:06 | |
| mikal | Herm. Turns out we apply the fake_libvirt_utils fixture in many places it isn't even used. | 23:08 |
| mikal | This rathole is nice. I like it. It feels like home. | 23:08 |
| *** wolverineav has quit IRC | 23:13 | |
| *** mlavalle has quit IRC | 23:15 | |
| *** wolverineav has joined #openstack-nova | 23:17 | |
| *** markvoelker has joined #openstack-nova | 23:19 | |
| mikal | Wait what? nova.virt.libvirt.utils.file_open is definitely my favourite part of all this. | 23:23 |
| cfriesen | mikal: did you intentionally omit "mv" or "cp" from privsep? | 23:25 |
| mikal | cfriesen: privsep only has things which had callers. So an absence either means that the user hasn't merged yet, or there is no user in nova until you came along. | 23:27 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add user_id field to InstanceMapping https://review.openstack.org/633350 | 23:28 |
| openstackgerrit | melanie witt proposed openstack/nova master: Populate InstanceMapping.user_id during migrations and schedules https://review.openstack.org/638574 | 23:28 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add online data migration for populating user_id https://review.openstack.org/633351 | 23:28 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add get_counts() to InstanceMappingList https://review.openstack.org/638072 | 23:28 |
| openstackgerrit | melanie witt proposed openstack/nova master: Count instances from mappings and cores/ram from placement https://review.openstack.org/638073 | 23:28 |
| openstackgerrit | melanie witt proposed openstack/nova master: Use instance mappings to count server group members https://review.openstack.org/638324 | 23:28 |
| cfriesen | mikal: I need to copy a subtree under /var/lib/libvirt/swtpm/<uuid> and recursively chmod or chown it. On the dest I need to do the opposite. | 23:29 |
| cfriesen | I'm wondering whether from a security standpoint it'd make more sense to create these under libvirt, or as generic operations in path.py | 23:29 |
| mikal | cfriesen: I think I'd put it under path, it seems generically useful. | 23:30 |
| mikal | cfriesen: there's no security difference between either location, its more about who uses it. | 23:31 |
| *** hongbin has quit IRC | 23:32 | |
| *** hongbin has joined #openstack-nova | 23:34 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in virt/test_block_device.py https://review.openstack.org/566153 | 23:37 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova stable/queens: Add description of custom resource classes https://review.openstack.org/619125 | 23:37 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add online data migration for populating user_id https://review.openstack.org/633351 | 23:45 |
| openstackgerrit | melanie witt proposed openstack/nova master: Add get_counts() to InstanceMappingList https://review.openstack.org/638072 | 23:45 |
| openstackgerrit | melanie witt proposed openstack/nova master: Count instances from mappings and cores/ram from placement https://review.openstack.org/638073 | 23:45 |
| openstackgerrit | melanie witt proposed openstack/nova master: Use instance mappings to count server group members https://review.openstack.org/638324 | 23:45 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521 | 23:52 |
| openstack | bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,Triaged] https://launchpad.net/bugs/1818914 | 23:52 |
| mriedem | cfriesen: ^ recreate test at least | 23:52 |
| *** markvoelker has quit IRC | 23:53 | |
| melwitt | mriedem: I think I'm not going to be able to be around tomorrow... :( on that note, I'm not sure how helpful it'd be to land the migration without the user-impacting changes, | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
