Monday, 2020-05-25

« Sunday, 2020-05-24 Index Tuesday, 2020-05-26 »
*** rcernin has quit IRC00:06
*** rcernin has joined #openstack-nova00:25
*** hongbin_ has joined #openstack-nova00:52
*** hongbin has quit IRC00:52
*** xiaolin has joined #openstack-nova00:58
*** Liang__ has joined #openstack-nova01:01
*** songwenping_ has joined #openstack-nova01:04
*** songwenping__ has quit IRC01:07
*** jawad_axd has joined #openstack-nova01:18
*** jawad_axd has quit IRC01:23
*** lvbin01 has quit IRC01:49
*** yaawang has joined #openstack-nova02:17
*** ociuhandu has joined #openstack-nova02:27
*** ociuhandu has quit IRC02:32
*** sapd1 has joined #openstack-nova03:08
*** songwenping__ has joined #openstack-nova04:21
*** songwenping_ has quit IRC04:24
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-nova04:33
*** threestrands has joined #openstack-nova04:38
*** sapd1 has quit IRC04:39
*** hongbin_ has quit IRC05:16
*** nightmare_unreal has joined #openstack-nova05:32
*** ociuhandu has joined #openstack-nova05:36
*** links has joined #openstack-nova05:38
*** ociuhandu has quit IRC05:46
*** ociuhandu has joined #openstack-nova05:47
*** ociuhandu has quit IRC05:52
*** sapd1 has joined #openstack-nova05:54
*** breizhkoala has joined #openstack-nova05:54
*** jawad_axd has joined #openstack-nova06:18
*** jsuchome has joined #openstack-nova06:30
*** tosky has joined #openstack-nova06:30
*** maciejjozefczyk has joined #openstack-nova06:38
*** damien_r has joined #openstack-nova06:39
*** ociuhandu has joined #openstack-nova06:39
*** udesale has joined #openstack-nova06:40
openstackgerritsean mooney proposed openstack/os-vif master: Use unittest.mock instead of third party mock  https://review.opendev.org/72496506:41
openstackgerritsean mooney proposed openstack/os-vif master: update tox envs and support pdf docs  https://review.opendev.org/72803706:45
*** slaweq has joined #openstack-nova06:46
*** ociuhandu has quit IRC06:46
*** damien_r has quit IRC06:47
*** tesseract has joined #openstack-nova06:56
*** links has quit IRC07:01
*** jhesketh has quit IRC07:03
*** jhesketh has joined #openstack-nova07:05
*** damien_r has joined #openstack-nova07:13
*** sean-k-mooney has joined #openstack-nova07:18
*** links has joined #openstack-nova07:18
*** ttsiouts has joined #openstack-nova07:19
*** rpittau|afk is now known as rpittau07:21
*** jhesketh has quit IRC07:25
*** jhesketh has joined #openstack-nova07:25
*** ralonsoh has joined #openstack-nova07:26
*** links has quit IRC07:33
*** links has joined #openstack-nova07:33
openstackgerritBalazs Gibizer proposed openstack/nova stable/train: Reserve DISK_GB resource for the image cache  https://review.opendev.org/73032407:34
openstackgerritBalazs Gibizer proposed openstack/nova stable/train: Add admin doc information about image cache resource accounting  https://review.opendev.org/73032607:34
*** arxcruz|off is now known as arxcruzz07:37
*** arxcruzz is now known as arxcruz07:37
*** xek has joined #openstack-nova07:38
*** ttsiouts has quit IRC07:42
*** ttsiouts has joined #openstack-nova07:42
*** ociuhandu has joined #openstack-nova07:46
*** ttsiouts has quit IRC07:52
*** ccamacho has joined #openstack-nova07:53
*** ttsiouts has joined #openstack-nova07:57
*** tkajinam has quit IRC08:01
openstackgerritBalazs Gibizer proposed openstack/nova stable/rocky: Update scheduler instance info at confirm resize  https://review.opendev.org/73034408:03
gibielod: fixed the commit message accordingly ^^08:04
*** breizhkoala has quit IRC08:04
openstackgerritMerged openstack/os-vif master: Switch to newer openstackdocstheme and reno versions  https://review.opendev.org/72861208:06
*** kashyap has joined #openstack-nova08:08
*** martinkennelly has joined #openstack-nova08:11
*** dtantsur|afk is now known as dtantsur08:12
*** xiaolin has quit IRC08:24
*** xiaolin has joined #openstack-nova08:28
*** adrianc has joined #openstack-nova08:34
*** ttsiouts has quit IRC08:43
*** ttsiouts has joined #openstack-nova08:44
*** ociuhandu has quit IRC08:49
bauzasgood morning Nova08:52
bauzasgibi: sorry, saw your ping, was on PTO08:52
elodgibi: thx, I'll look into it soon09:02
*** links has quit IRC09:13
openstackgerritWenping Song proposed openstack/nova master: WIP: Cyborg suspend/resume support  https://review.opendev.org/72994509:14
*** links has joined #openstack-nova09:15
gibibauzas: no problem09:29
bauzasgibi: also, I'll try to help for the routed networks09:30
gibibauzas: there I have started something but I don't think I will have time to continue in V cycle09:31
bauzasgibi: I provided an email for it09:31
gibiahh OK, I will read09:32
bauzasgibi: sean-k-mooney will work on NUMA topology in Placement instead of me09:32
bauzasso I could have time for this09:32
*** links has quit IRC09:32
bauzasgibi: also, I'll volunteer for working on https://specs.openstack.org/openstack/nova-specs/specs/stein/approved/reshape-provider-tree.html#offline-upgrade-script09:33
gibibauzas: cool, thanks09:34
sean-k-mooneywell thats the plan at least. ill try to work on the numa stuff as the primary thing im aiming to get done this cycle and hopefully allowing bauzas to complete other work09:35
bauzassure09:35
sean-k-mooneyif i manage to get the numa stuff done early in the cycle ill move on to something else of if not ill move on to bug fixes and focus on get it done09:35
bauzasanyway, it's a plan :)09:36
bauzaslike for me09:36
bauzas(and like any other)09:36
bauzasif anyone has other priority, then of course it can change09:36
bauzasgibi: btw. do you think we could have a spec for the routed networks ?09:39
sean-k-mooneyok i woke up around 4:45 this morning so im actully starting to get hungery. im going to go grab something for a deli and see if i can pick up a few bits and pieces. ill be back on irc in an hour or so09:39
gibibauzas: we could have a doc that described the scenarios we need to support09:40
sean-k-mooneybauzas: we should have a spec for routed networks if for nothing else to define the scope.09:40
gibi+1 ^^09:40
bauzasok, I'll work on this then09:41
sean-k-mooneyi would like to see use eventually get to full network aware schduleing but if you want to target basic routed network support for this cycle that would be fine with me. just define the rest as out of scope09:41
*** links has joined #openstack-nova09:42
*** links has quit IRC09:42
*** ociuhandu has joined #openstack-nova09:45
bauzassean-k-mooney: the latter for Victoria honestly09:47
*** belmoreira has joined #openstack-nova09:47
*** jhesketh has quit IRC09:48
openstackgerritMerged openstack/os-vif master: Use unittest.mock instead of third party mock  https://review.opendev.org/72496509:49
*** jhesketh has joined #openstack-nova09:49
gibibauzas: thanks.09:51
bauzasgibi: I need to get more context (since I'm not a network expert) before providing the spec09:51
bauzasbut what I like is that I'll try to understand more about networks :)09:51
gibibauzas: the functional test in this patch has a lot of comments that I think helps understanding the goal of the feature https://review.opendev.org/#/c/656885/7/nova/tests/functional/test_servers.py09:53
bauzasahah thanks !09:53
gibialso there was a mail from me during U http://lists.openstack.org/pipermail/openstack-discuss/2020-February/012846.html09:53
bauzasexcellent, ta :)09:54
gibi:)09:54
kashyapsean-k-mooney: When you get a min, I'm trying to see what are the "real Q35" failures, but I don't see any that is directly related to Q35 here - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_725/708701/8/check/devstack-platform-fedora-latest-virt-preview-q35/7259849/testr_results.html10:03
kashyapI only see "Request timed out"s10:03
* kashyap combs through all the failures to see if there are any others10:04
*** breizhkoala has joined #openstack-nova10:10
*** rpittau is now known as rpittau|bbl10:12
*** Liang__ has quit IRC10:19
* gibi goes for food10:32
openstackgerritWenping Song proposed openstack/nova master: WIP: Cyborg suspend/resume support  https://review.opendev.org/72994510:34
sean-k-mooney kashyap in theory there should be no "real q35" failure but there might be some test configuration issues10:36
sean-k-mooneykashyap: we have been allowing customer to use q35 in production since osp 13 if not eairler so i would hope they would have reported any real issues long before now10:37
kashyapsean-k-mooney: For test configuration, right: wherever 'ide' is used, that needs to be replaced (or better: conditionalized).  E.g. https://review.opendev.org/#/c/721555/10:39
* kashyap back in a few; need to print something, and the printer is broken here10:39
sean-k-mooney ERROR cinder.volume.api [None req-a5e3e7e0-f8ce-4a58-bd74-53517edc5fd5 tempest-VolumeQuotasNegativeTestJSON-777903610 tempest-VolumeQuotasNegativeTestJSON-777903610] Quota exceeded for a769409cccff4ea48d0fe2efea551c8a, tried to extend volume by 1G, (1G of 1G already consumed).: cinder.exception.OverQuota: Quota exceeded for resources: ['gigabytes']10:39
sean-k-mooneyi think the issue are mostly down to that10:40
sean-k-mooneykashyap: i dont think ide will be used anyhwere in the  test and that is not the reason for the failures as far as i can tell10:40
kashyapsean-k-mooney: IDE _is_ used in some places, see the link I posted :-)10:41
kashyap[But be back in a few...]10:41
sean-k-mooneywell most of the test failures are down to exceeding quota i belive10:41
sean-k-mooneywhy did lee us ide for hw_rescue_bus?10:42
sean-k-mooneythat specificly was only added last cycle so that should have used ide form the start10:42
stephenfinyo, bauzas - care to look at https://review.opendev.org/#/c/728968/ and surrounding patches?10:48
*** derekh has joined #openstack-nova10:51
*** ociuhandu has quit IRC11:00
*** ociuhandu has joined #openstack-nova11:01
*** ttsiouts has quit IRC11:01
openstackgerritStephen Finucane proposed openstack/nova master: quota: remove get_quota_classes() driver method  https://review.opendev.org/61561511:04
openstackgerritStephen Finucane proposed openstack/nova master: quota: remove Context.quota_class  https://review.opendev.org/61561611:04
openstackgerritStephen Finucane proposed openstack/nova master: quota: remove FakeContext from quota unit tests  https://review.opendev.org/61561711:04
openstackgerritStephen Finucane proposed openstack/nova master: quota: remove _no_class tests  https://review.opendev.org/61561811:04
openstackgerritStephen Finucane proposed openstack/nova master: quota: clean up DbQuotaDriver unit tests  https://review.opendev.org/61561911:04
openstackgerritStephen Finucane proposed openstack/nova master: quota: rename arguments to clarify they are limits  https://review.opendev.org/61563311:04
*** ociuhandu has quit IRC11:05
*** udesale_ has joined #openstack-nova11:06
*** udesale has quit IRC11:09
*** threestrands has quit IRC11:11
*** ttsiouts has joined #openstack-nova11:23
openstackgerritBalazs Gibizer proposed openstack/nova stable/train: Add admin doc information about image cache resource accounting  https://review.opendev.org/73032611:27
*** sapd1 has quit IRC11:32
sean-k-mooneystephenfin: i forget tox -e py3 worked11:54
gibisean-k-mooney: I  know that you have a better view than me regarding network outage during live migration. there is a new bug 1880389 reported about it. Could you check it if you have time?12:03
openstackbug 1880389 in OpenStack Compute (nova) "lost net connection when live migration" [Undecided,New] https://launchpad.net/bugs/188038912:03
sean-k-mooneysure i can take a look shortly just grabing coffee but i have it open for when i get back12:09
*** ociuhandu has joined #openstack-nova12:09
*** raildo has joined #openstack-nova12:10
sean-k-mooneyalso without looking is sure bad things proably happen if there is a network outage while live migrating12:10
*** rpittau|bbl is now known as rpittau12:11
*** sapd1 has joined #openstack-nova12:15
sean-k-mooneyok so the issue is with the vm losing network connectivyt not a network partion at the host level causeing the migration to fail in weird ways which is less scary12:16
sean-k-mooneygibi: i need to dig deaper but its proably a duplicate of https://bugs.launchpad.net/neutron/+bug/1815989 which is cause by the root cause of https://bugs.launchpad.net/neutron/+bug/173432012:18
openstackLaunchpad bug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress]12:18
openstackLaunchpad bug 1734320 in OpenStack Compute (nova) "Eavesdropping private traffic" [Undecided,In progress] - Assigned to sean mooney (sean-k-mooney)12:18
bauzasstephenfin: ack will look12:19
sean-k-mooneygibi: https://review.opendev.org/#/c/724386/ is a potential workaround which some have said fixes the issue12:19
sean-k-mooneygibi: this is how i actully wanted to fix it https://review.opendev.org/#/q/status:open++branch:master+topic:bug/173432012:19
sean-k-mooneythat said they are using linux bridge so it could be something else12:20
sean-k-mooney"the ping was broken down. and the ping package from the physic switch was still set to the source."12:22
sean-k-mooneyso they were observing that pings are switched to the souce host which implies the mac learning frames were lost12:22
sean-k-mooneywhich is bug: #181598912:22
openstackbug 1815989 in OpenStack Compute (nova) "OVS drops RARP packets by QEMU upon live-migration causes up to 40s ping pause in Rocky" [Medium,In progress] https://launchpad.net/bugs/181598912:22
sean-k-mooneyalthough in this case they are reportign the same behavior for linux bridge12:23
*** sapd1 has quit IRC12:23
sean-k-mooneyah last paragraph "And I found the time of 5 RARP from VM is before the time of the vlan sub interface insert to the linuxbridge bridge when live migration. maybe this will help us to make sure the problem I found."12:23
sean-k-mooneyso yes its the same underlying issue12:24
*** ttsiouts has quit IRC12:24
sean-k-mooneythe issue happens in the ovs case when libvirt recreated the port that os-vif has allready created resulting in a race with the neutron l2 agent12:25
sean-k-mooneyi did not think we had the same race for linux bridge but ill take a look12:25
gibisean-k-mooney: thanks12:26
*** sapd1 has joined #openstack-nova12:28
*** artom has joined #openstack-nova12:28
*** avolkov has joined #openstack-nova12:29
*** ttsiouts has joined #openstack-nova12:44
sean-k-mooneyso we use interface type bridge for linux bridge which is not very surpising12:48
sean-k-mooney <interface type='bridge'>12:48
sean-k-mooney    <source bridge='br1'/>12:48
sean-k-mooney    <target dev='vnet7'/>12:48
sean-k-mooney    <mac address="00:11:22:33:44:55"/>12:48
sean-k-mooney  </interface>12:48
sean-k-mooneyand se the target, mac  and source to the values provided by neutron12:48
sean-k-mooneybefore the vm is spawned on the destination host we invoke os-vif to ensure that the bridge exists12:49
sean-k-mooneyso when we define the libvirt domian there should be nothing for libvirt to do it should be alredy configured12:50
sean-k-mooneythis is all os-vif does https://github.com/openstack/os-vif/blob/master/vif_plug_linux_bridge/linux_bridge.py#L97-L11212:51
openstackgerritMerged openstack/os-vif master: Remove .testr.conf  https://review.opendev.org/72915612:52
sean-k-mooneyi should proably go clean up that code since nova net is now dead and we should not really be doing any ip filtering in os-vif even in neutron is not anymore since that was really for nova net12:55
gibisean-k-mooney: thanks for triageing that bug13:00
sean-k-mooneyim debating if im goint to try and repoduce it or not13:05
sean-k-mooneyi would need to check the libvirt sorce code but i dont think it should have the same race13:06
sean-k-mooneyit might but i did not think it should13:06
openstackgerritStephen Finucane proposed openstack/nova master: virt: Add 'context', drop 'network_info' parameters for 'unrescue'  https://review.opendev.org/73038213:32
openstackgerritStephen Finucane proposed openstack/nova master: Add emulated TPM support to Nova  https://review.opendev.org/63136313:32
openstackgerritStephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files  https://review.opendev.org/63993413:32
openstackgerritsean mooney proposed openstack/os-vif master: update tox envs and support pdf docs  https://review.opendev.org/72803713:51
*** rpittau is now known as rpittau|brb13:56
sean-k-mooneystephenfin: hopefully you are more happy with ^14:02
sean-k-mooneybut also that is more a merge when convenient patch. as long as we merge it before m3 (perferably m1) its fine14:02
sean-k-mooneyso no rush14:02
stephenfinbetter but still not correct :)14:03
sean-k-mooneyhehe ok im goint to update my osc patch for evacuate ill take a look when that is done14:04
sean-k-mooneyjust looked. ya ill do some of the chagne in a follow up and adress some in that patch. ill push both patches togeter this evening or tommorow14:09
sean-k-mooneythat patch is getting a bit long  but the change you suggest make sense14:09
sean-k-mooney'Wait for instance evacuation to complete' sound less worng then 'Wait for evacuation to complete' right?14:18
stephenfinThe operation affects multiple instances, right?14:20
sean-k-mooneyno14:20
sean-k-mooneyonly one instance14:20
stephenfinthen the first one is better, yes14:20
sean-k-mooneyim porting evacuate not host-evacuate14:20
stephenfinalthough I don't know if osc uses the term "instance"14:20
sean-k-mooneyhost-evacuate it the clinet side command that enumarate the instance on the host and evacuates all of them14:20
sean-k-mooneyi think it might use server but that is kind of wronge14:21
sean-k-mooneyi could use server14:21
sean-k-mooneyserver is proably more inline with the other messages in teh file so ill use that14:22
*** ociuhandu has quit IRC14:25
*** ttsiouts has quit IRC14:26
*** ociuhandu has joined #openstack-nova14:27
*** rpittau|brb is now known as rpittau14:30
*** ociuhandu has quit IRC14:32
*** sapd1 has quit IRC14:32
openstackgerritMerged openstack/python-novaclient master: Remove Babel requirement  https://review.opendev.org/72313914:33
*** breizhkoala has quit IRC14:50
*** hamzy has quit IRC15:06
*** ociuhandu has joined #openstack-nova15:27
openstackgerritBalazs Gibizer proposed openstack/nova-specs master: Make Feature Liaison optional  https://review.opendev.org/73063815:37
*** sapd1 has joined #openstack-nova15:45
*** ociuhandu has quit IRC15:46
*** ociuhandu has joined #openstack-nova15:47
*** ociuhandu has quit IRC15:53
*** elod has quit IRC15:59
*** elod has joined #openstack-nova16:01
*** udesale_ has quit IRC16:14
*** ociuhandu has joined #openstack-nova16:23
*** ircuser-1 has joined #openstack-nova16:26
*** dtantsur is now known as dtantsur|afk16:36
*** ociuhandu has quit IRC16:44
*** ociuhandu has joined #openstack-nova16:45
*** rpittau is now known as rpittau|afk16:48
*** mgariepy has quit IRC16:49
openstackgerritStephen Finucane proposed openstack/nova master: virt: Add 'context', drop 'network_info' parameters for 'unrescue'  https://review.opendev.org/73038216:57
openstackgerritStephen Finucane proposed openstack/nova master: Add emulated TPM support to Nova  https://review.opendev.org/63136316:57
openstackgerritStephen Finucane proposed openstack/nova master: WIP: Add support for resize and cold migration of emulated TPM files  https://review.opendev.org/63993416:57
*** ociuhandu has quit IRC17:00
*** mgariepy has joined #openstack-nova17:03
*** sapd1 has quit IRC17:14
*** derekh has quit IRC17:16
*** damien_r has quit IRC17:25
*** derekh has joined #openstack-nova17:30
*** sapd1 has joined #openstack-nova17:32
*** derekh has quit IRC17:34
*** mgariepy has quit IRC17:42
*** mgariepy has joined #openstack-nova17:43
*** damien_r has joined #openstack-nova17:46
*** mgariepy has quit IRC17:58
*** sapd1 has quit IRC18:02
*** avolkov has quit IRC18:06
*** ociuhandu has joined #openstack-nova18:15
*** belmoreira has quit IRC18:16
*** mgariepy has joined #openstack-nova18:17
*** ociuhandu has quit IRC18:23
*** nightmare_unreal has quit IRC18:31
*** ociuhandu has joined #openstack-nova18:33
*** damien_r has quit IRC18:35
*** tesseract has quit IRC18:38
*** jsuchome has quit IRC18:38
*** maciejjozefczyk has quit IRC18:41
*** ociuhandu has quit IRC19:01
*** ociuhandu has joined #openstack-nova19:02
*** ralonsoh has quit IRC20:26
*** maciejjozefczyk has joined #openstack-nova20:51
*** maciejjozefczyk has quit IRC20:58
*** raildo has quit IRC21:00
*** sapd1_y has joined #openstack-nova21:02
*** jamesdenton has quit IRC21:02
*** sapd1_x has quit IRC21:02
*** jamesdenton has joined #openstack-nova21:05
*** xek has quit IRC21:22
*** ccamacho has quit IRC21:47
*** slaweq has quit IRC21:54
*** slaweq has joined #openstack-nova22:00
*** slaweq has quit IRC22:05
*** derekh has joined #openstack-nova22:31
*** martinkennelly has quit IRC22:34
*** rcernin has quit IRC22:49
*** rcernin has joined #openstack-nova22:50
*** tkajinam has joined #openstack-nova22:53
*** tosky has quit IRC23:18
*** derekh has quit IRC23:21
*** ab-a has quit IRC23:24
openstackgerritMerged openstack/nova stable/queens: Revert "nova shared storage: rbd is always shared storage"  https://review.opendev.org/72686423:28
« Sunday, 2020-05-24 Index Tuesday, 2020-05-26 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!