Wednesday, 2021-03-03

*** whoami-rajat has quit IRC00:15
*** johanssone has quit IRC00:16
*** tosky has quit IRC00:17
*** johanssone has joined #openstack-nova00:19
*** macz_ has quit IRC00:49
*** martinkennelly has quit IRC00:50
*** spatel has joined #openstack-nova00:52
openstackgerritsean mooney proposed openstack/nova master: add hw:mlock extra spec  https://review.opendev.org/c/openstack/nova/+/77834700:54
*** spatel has quit IRC00:57
*** songwenping_ has joined #openstack-nova01:01
*** mlavalle has quit IRC01:02
*** spatel has joined #openstack-nova01:07
*** Underknowledge has quit IRC01:08
*** Underknowledge has joined #openstack-nova01:08
*** gyee has quit IRC01:17
*** songwenping__ has joined #openstack-nova01:47
*** songwenping_ has quit IRC01:51
*** Underknowledge2 has joined #openstack-nova01:51
*** Underknowledge has quit IRC01:54
*** Underknowledge2 is now known as Underknowledge01:54
*** rcernin has quit IRC02:00
*** mkrai has joined #openstack-nova02:14
openstackgerritsean mooney proposed openstack/nova master: add vdpa vnic to pci request mapping and filtering.  https://review.opendev.org/c/openstack/nova/+/77835002:24
*** hamalq has quit IRC02:25
*** dtantsur has quit IRC02:25
*** dtantsur has joined #openstack-nova02:29
*** rcernin has joined #openstack-nova02:29
*** rcernin has quit IRC02:29
*** rcernin has joined #openstack-nova02:30
*** Underknowledge has quit IRC02:38
*** rcernin has quit IRC02:44
*** macz_ has joined #openstack-nova02:46
*** hemanth_n has joined #openstack-nova02:49
*** macz_ has quit IRC02:50
*** jamesdenton has quit IRC02:58
*** jamesdenton has joined #openstack-nova02:58
*** dtantsur has quit IRC03:14
*** dtantsur has joined #openstack-nova03:15
*** rcernin has joined #openstack-nova03:16
*** LinPeiWen has quit IRC03:21
*** dasp has quit IRC03:34
*** dasp has joined #openstack-nova03:45
*** psachin has joined #openstack-nova03:51
*** k_mouza has joined #openstack-nova04:00
*** k_mouza has quit IRC04:05
*** zzzeek has quit IRC04:17
*** zzzeek has joined #openstack-nova04:17
*** zzzeek has quit IRC04:22
*** zzzeek has joined #openstack-nova04:23
*** mkrai has quit IRC04:30
*** mkrai_ has joined #openstack-nova04:30
*** spatel has quit IRC04:48
*** ratailor has joined #openstack-nova04:54
*** dviroel has quit IRC04:55
*** whoami-rajat has joined #openstack-nova04:58
*** LinPeiWen has joined #openstack-nova05:02
*** ratailor_ has joined #openstack-nova05:26
*** vishalmanchanda has joined #openstack-nova05:27
*** songwenping__ has quit IRC05:27
*** songwenping__ has joined #openstack-nova05:28
*** ratailor has quit IRC05:29
*** ratailor__ has joined #openstack-nova05:34
*** ratailor_ has quit IRC05:37
*** brinzhang has joined #openstack-nova05:41
*** zzzeek has quit IRC06:06
*** jamesdenton has quit IRC06:27
*** jamesdenton has joined #openstack-nova06:28
*** zzzeek has joined #openstack-nova06:41
*** bnemec has quit IRC06:41
*** bnemec has joined #openstack-nova06:49
*** mkrai_ has quit IRC06:55
*** links has joined #openstack-nova06:59
songwenping__melwitt: hi, your process is right, the guest xml is live update but it doesnot work unless we reboot the guest. this is really a problem.07:06
*** luksky has joined #openstack-nova07:19
*** ralonsoh has joined #openstack-nova07:37
*** khomesh24 has joined #openstack-nova07:39
*** belmoreira has joined #openstack-nova07:44
*** rcernin has quit IRC07:44
*** rpittau|afk is now known as rpittau07:52
*** dklyle has quit IRC08:03
brinzhanggibi, bauzas: replied in https://review.opendev.org/c/openstack/nova/+/761452/9/nova/compute/manager.py#210108:08
brinzhanggibi, bauzas: do I need to register a bug, then submit the fix?08:09
*** ociuhandu has joined #openstack-nova08:13
*** andrewbonney has joined #openstack-nova08:14
*** iurygregory_ has joined #openstack-nova08:17
*** iurygregory has quit IRC08:18
*** sapd1 has quit IRC08:21
*** sapd1 has joined #openstack-nova08:21
*** ccstone has quit IRC08:22
*** ccstone has joined #openstack-nova08:22
bauzasbrinzhang: good afternoon for you08:26
brinzhangbauzas: good morning^08:27
bauzasbrinzhang: yes, please file a bug and work on a patch if you want to fix it08:27
brinzhangbauzas: ack, I will file a bug, then fix it with a seperate patch08:28
bauzasthanks08:28
bauzasbrinzhang: then, ping me08:28
brinzhangbp, thanks point this bug, this is my ignore08:28
bauzasand I'll see whether I can rebase my RPC API change above your change08:29
brinzhangbauzas: ack, cool08:29
bauzasbecause if not, you couldn't backport it to stable08:29
brinzhangyeah08:29
*** ociuhandu has quit IRC08:30
*** tosky has joined #openstack-nova08:34
*** ociuhandu has joined #openstack-nova08:35
*** gokhani has joined #openstack-nova08:38
*** mkrai has joined #openstack-nova08:41
gokhaniHi folks, ı have a development environment which is installed by OSA Ussuri. ı have problems on nova-api side. I can observe rabbitmq timeout errors > http://paste.openstack.org/show/803175/ and rarely I am getting connection failed errors to rabbitmq on api side > http://paste.openstack.org/show/803176/. And ıt is strange behaviour now we don't08:43
gokhanihave any uwsgi processes on nova api side. I doubt about uwsgi + nova. this is expected behaviour or there is a problem on nova, I am not sure. do you have any ideas about this ? nova = 21.1.1 and oslo.messaging version is 12.1.408:43
*** jdillaman has quit IRC08:44
*** jdillaman has joined #openstack-nova08:45
*** tbachman has quit IRC08:53
*** tbachman has joined #openstack-nova08:54
kashyapstephenfin: gibi: Morning; I was away earlier yesterday.  Reading the scroll; I see your question is answered on the +/- thing08:57
kashyapgibi: Indeed; thanks for the ACK; the separate options would've made sense if we were doing it from scratch.08:58
kashyapThanks for bearing with me.08:59
*** mkrai has quit IRC09:04
*** mkrai has joined #openstack-nova09:04
*** Underknowledge has joined #openstack-nova09:11
*** lucasagomes has joined #openstack-nova09:12
*** spatel has joined #openstack-nova09:15
*** derekh has joined #openstack-nova09:25
songwenping__stephenfin: morning; there are two problems for the noVNC patch. 1. the password works after you reboot the guest 2. if you donnot pass the password when get-vnc-console, you need config `[vnc]auth_schemes = none, vnc` in /etc/nova/nova-cell1.conf for noVNC guest.09:35
songwenping__for the first issue, we can clarify in the docs that user need to reboot the server if set/reset password for vnc console. we can improve if the libvirt/qemu support live set/reset password.09:38
*** khomesh24 has quit IRC09:39
*** spatel has quit IRC09:39
*** ociuhandu has quit IRC09:39
stephenfinsongwenping__: to be honest, it sounds like we shouldn't allow users to change the password if that's not supported so09:41
stephenfinI don't think asking users to restart their instance is a reasonable request09:41
openstackgerritLucas Alvares Gomes proposed openstack/nova master: [OVN] Explicitly set nova-next job to ML2/OVS  https://review.opendev.org/c/openstack/nova/+/77694409:44
songwenping__for the second issue, we plan to update the release note and the description of nova/conf/vnc.py for `auth_schemes`.09:44
songwenping__stephenfin: the libvirt docs[1] said it already support live update password, but i donnot know why it failed in my local ven. [1] https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainUpdateDeviceFlags09:46
*** links has quit IRC09:46
*** links has joined #openstack-nova09:49
brinzhangstephenfin, songwenping__:IMHO, support reset VNC password is good for the user, it can be protected  thire instance opening by unknown user, if it must be reboot to take effect, can we do the first mode to support [vnc]auth_schemes=vnc?09:55
lyarwoodstephenfin / gibi ; https://review.opendev.org/c/openstack/nova/+/778177 should be good now btw, I missed that the requirements job actually forces you to sync the in-direct LC deps to requirements that bloats the change a little09:59
stephenfinbrinzhang: The issue is that _set_ VNC password also requires a reboot09:59
lyarwoodas discussed with sean-k-mooney yesterday I've added a note to the PTG to discuss this mess09:59
stephenfinBasically to touch anything to do with VNC passwords, the instance must be restarted09:59
*** ociuhandu has joined #openstack-nova10:04
brinzhangstephenfin: can you check this libvirt docs? https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainUpdateDeviceFlags does it means it cannot be reset password if flags=LIVE(1)?10:05
brinzhang*E.g. the hypervisor driver will return failure if LIVE is specified but it only supports modifying the persisted device allocation.*, so the VNC password isnot blongs to the *persisted device*?10:06
*** xek has joined #openstack-nova10:07
songwenping__stephenfin: we just need reboot first time to set password, if we reset the password again, we donnot need reboot the server any more.10:09
gibibrinzhang: I agree to have a bug reported and fixing it in a separate patch10:10
songwenping__from none -> vnc need to reboot, from vnc -> vnc donnot need to reboot10:10
brinzhanggibi: I have filed the bug, and will be submited the fix later ^, pls see bug 191759210:11
openstackbug 1917592 in OpenStack Compute (nova) "Missed 'accel_uuids' when we the 'shelved_offload_time' time out in shelving instance periodic task" [Medium,New] https://launchpad.net/bugs/1917592 - Assigned to Brin Zhang (zhangbailin)10:11
gibibrinzhang: thanks10:11
brinzhanggibi: np, thanks for your mind too10:11
gibilyarwood: thanks I]10:13
gibilyarwood: thanks I'm +210:13
lyarwoodcheers10:19
*** lpetrut has joined #openstack-nova10:22
lyarwoodrandom question, does anyone know why we don't use the instance UUID as the domain name within libvirt?10:22
lyarwoodtrying to debug a live migration failure on stable/ussuri and hate having to lookup the domain name in the logs10:22
gibilyarwood: good question, we probably need a historian to answer itr10:26
kashyapYeah, I don't remember that answer either; and wondered the same in the past10:28
kashyapBut the guest XML records both instance-YYYYYYY and the UUID10:28
stephenfinbrinzhang: Yeah, looking now10:28
*** ociuhandu has quit IRC10:28
stephenfinbrinzhang: songwenping__: IMO we need to fix this and figure out how to get VNC password working with TLS. At the moment, enabling this means (a) no TLS, (b) you have to restart the instance to use the feature, and (c) you have to do some funky nova.conf changes for things to work10:30
stephenfinAlso (d) you're stuck with 8 characters (not our fault, but unfortunate all the same)10:30
stephenfinNot a good UX /o\10:30
stephenfinbrinzhang: I know it's a lot of work that you said you don't want to do it, but I would still seriously suggest switching the order of the VNC password and tenant_id -> project_id series10:31
lyarwoodkashyap: yeah thats fine in live envs, just a PITA when you only have logs to debug things10:32
lyarwoodkashyap: I think it might be that libvirt couldn't fit the entire UUID in the name or something?10:32
kashyaplyarwood: Yeah, no denying; I just let the steam out by cussing10:32
*** nightmare_unreal has joined #openstack-nova10:32
kashyaplyarwood: Hmm, that I don't remember.  But lemme quickly test by creating a domain name with the UUID :)10:32
stephenfinThe VNC series is at huge risk due to all those concerns. I can't in good faith merge what I consider to be a broken feature :-( Hopefully we can address them before feature freeze but it will prevent the other series landing until we do10:33
kashyaplyarwood: Ah, so: libvirt seems to be smart here:10:34
lyarwoodaaaaaaaaaah it's instance.name10:34
kashyaplyarwood: When I tried to live-modify a guest's name from s/cvm1/64ba587c-57a7-44c0-94ec-5444e5268c4e/ --> libvirt tells me: "well, the guest already has a UUID"10:35
kashyapI tried removing dashes and plug the dash-less UUID into <name/>, it still detects.10:35
lyarwoodokay, we can't really change this anyway, it's just annoying10:36
*** k_mouza has joined #openstack-nova10:36
kashyapYeah, libvirt explicitly rejects a UUID for a domain "name"10:37
*** ociuhandu has joined #openstack-nova10:37
kashyaps/Yeah,/Yeah, and/10:37
* lyarwood wonders if it would be worth adding some log collection that lists instance uuids and instance names somewhere10:38
lyarwoodnot sure how we'd do that after the test run10:38
kashyaplyarwood: You mean, you want a file logged a mapping of UUIDs with instance names?10:38
lyarwoodyeah but I don't think that's going to be possible *after* tempest has already deleted everything10:40
kashyapHmm, true10:40
*** jangutter has joined #openstack-nova10:41
*** jangutte_ has joined #openstack-nova10:43
openstackgerritStephen Finucane proposed openstack/nova master: console: Improve logging  https://review.opendev.org/c/openstack/nova/+/77840710:43
*** jangutter_ has quit IRC10:44
*** jangutter has quit IRC10:46
*** mkrai has quit IRC10:47
lyarwoodgibi: https://review.opendev.org/q/topic:spec/libvirt-default-machine-type+status:open should be ready for review this week btw, not sure if you want to move it into a runway slot or not.10:55
*** dviroel has joined #openstack-nova11:05
gibilyarwood: I added to my queue, I see stephenfin is already +2 so I prioritize this11:06
*** jangutter has joined #openstack-nova11:07
*** luksky has quit IRC11:07
lyarwoodgibi: yup thanks, I'm around all day for respins etc so let me know if you have any questions11:07
gibiOK11:07
*** luksky has joined #openstack-nova11:08
*** jangutte_ has quit IRC11:09
*** luksky has quit IRC11:09
*** zoharm has joined #openstack-nova11:11
*** iurygregory_ is now known as iurygregory11:19
slaweqhi nova team :)11:23
gibislaweq: hi!11:23
slaweqI need Your help with some ci issue11:23
slaweqwe saw, mostly in stable/train errors like in https://a574f9c0fd4ca92b7603-2045be852d43868eb95da6cc3429b40d.ssl.cf2.rackcdn.com/777334/2/check/neutron-tempest-dvr-ha-multinode-full/44d0207/testr_results.html recently11:23
slaweqdo You have any idea why there is no host for cold-migration or resize in that job?11:23
slaweqhi gibi :)11:25
* gibi clicks11:25
bauzasstephenfin: saw my weak -1 on your apidb change for Ocata ? https://review.opendev.org/c/openstack/nova/+/759402/411:25
*** luksky has joined #openstack-nova11:25
bauzasI don't really to vote -1 on a commit msg miss, but I feel this is important to comment the fact that we don't care about a specific DB migration11:25
bauzasreally like*11:26
bauzasand a commit msg can't be patched by a FUP...11:26
gibislaweq: is it happens all the time or just sometimes?11:28
slaweqgibi: on train I think that all the time11:29
slaweqor at least very often11:29
slaweqgibi: in scheduler log I see something like:11:30
*** bhagyashris is now known as bhagyashris|rove11:30
slaweqMar 03 10:32:31.142569 ubuntu-bionic-rax-ord-0023272867 nova-scheduler[24545]: INFO nova.scheduler.host_manager [None req-657ecf60-4fba-4168-9613-3c57d01bdcb7 tempest-MinBwAllocationPlacementTest-1587030703 tempest-MinBwAllocationPlacementTest-1587030703-project-admin] Host filter ignoring hosts: ubuntu-bionic-rax-ord-002327286711:30
*** bhagyashris|rove is now known as bhagyashri|rover11:30
gibislaweq: ack, give me some time to dive into the logs11:30
slaweqand then next message that " There are 0 hosts available but 1 instances requested to build."11:30
slaweqgibi: sure, I will open LP for that to track it11:30
slaweqthx for help11:30
gibislaweq: thanks for reporting it11:31
slaweqit's not very urgent for us as this is non-voting job :)11:31
slaweqso take Your time11:31
brinzhangstephenfin: *(songwenping_)from none -> vnc need to reboot, from vnc -> vnc donnot need to reboot* I still think from docs or a bug trace this is a good choice11:31
slaweqgibi: https://bugs.launchpad.net/nova/+bug/191761011:40
openstackLaunchpad bug 1917610 in OpenStack Compute (nova) "Migration and resize tests from tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest failing in neutron-tempest-dvr-ha-multinode-full" [Undecided,New]11:40
slaweqreported11:40
*** ociuhandu has quit IRC11:41
openstackgerritBrin Zhang proposed openstack/nova master: Add missed accel_uuids for _poll_shelved_instances  https://review.opendev.org/c/openstack/nova/+/77844011:54
stephenfinbrinzhang: I don't :(11:57
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations  https://review.opendev.org/c/openstack/nova/+/75940211:57
stephenfinbauzas: Is that okay? ^11:58
stephenfinI'll rebase the rest of the series on top if so11:58
brinzhangbauzas: fixed that bug, pls see https://review.opendev.org/c/openstack/nova/+/77844011:58
*** legochen has quit IRC12:00
*** ociuhandu has joined #openstack-nova12:12
*** ociuhandu has quit IRC12:17
*** martinkennelly has joined #openstack-nova12:18
brinzhangstephenfin: we will re-search, and make sure whether can reset passwrod and without restart the server12:22
brinzhangs/and//12:22
gibislaweq: responded in https://bugs.launchpad.net/nova/+bug/1917610 with alternatives to either fix the job definition in neutron or fix the test precoditions in tempest12:28
openstackLaunchpad bug 1917610 in OpenStack Compute (nova) "Migration and resize tests from tempest.scenario.test_minbw_allocation_placement.MinBwAllocationPlacementTest failing in neutron-tempest-dvr-ha-multinode-full" [Undecided,New]12:28
*** hemanth_n has quit IRC12:32
*** hemanth_n has joined #openstack-nova12:36
slaweqgibi: thx12:37
slaweqgibi: proposed fix https://review.opendev.org/c/openstack/tempest/+/778451 and dnm test patch https://review.opendev.org/c/openstack/neutron/+/77845212:37
*** ociuhandu has joined #openstack-nova12:37
slaweqlet's check if that will help12:37
gibitht was fast :)12:38
gibiwondering why this is only hit on stable/train. It should fail on any newer than train too12:38
*** hemanth_n has quit IRC12:40
*** legochen_ has joined #openstack-nova12:42
*** legochen_ is now known as legochen12:43
gibilyarwood: I see an issue in https://review.opendev.org/c/openstack/nova/+/769548/12/nova/cmd/manage.py#263112:47
lyarwoodah sorry, I thought I had sorted that out12:53
gibithe update command looks good, you only missed the get12:54
lyarwoodyeah my bad12:56
*** psachin has quit IRC13:01
*** ratailor__ has quit IRC13:19
*** jangutter_ has joined #openstack-nova13:34
*** jangutter has quit IRC13:37
*** spatel has joined #openstack-nova13:40
*** hemanth_n has joined #openstack-nova13:41
*** hemanth_n has quit IRC13:45
openstackgerritBalazs Gibizer proposed openstack/nova master: Fix ImageBackendFixture not to support direct_snapshot  https://review.opendev.org/c/openstack/nova/+/77846213:47
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Simplify device_path check in _detach_encryptor  https://review.opendev.org/c/openstack/nova/+/77846313:49
gibilyarwood: also could you check my question in https://review.opendev.org/c/openstack/nova/+/774898/7/nova/tests/functional/libvirt/test_machine_type.py#388 and the follow up I pushed?13:49
lyarwoodgibi: I was about to ask if that was related13:50
lyarwoodlooking now13:50
lyarwoodjust had to push the above fix before switching back to this13:50
gibiyepp it is a fixture fix13:51
gibito enable the unshelve test13:51
lyarwoodwonderful13:53
lyarwoodgibi: would you mind if I just pulled in that fix ahead of the new func test13:54
gibilyarwood: sure13:54
lyarwoodthanks13:54
*** ociuhandu has quit IRC13:58
gibilyarwood: overall I'm +2 on the machine type series I will quickly re-review it once you respin it13:59
*** ociuhandu has joined #openstack-nova13:59
lyarwoodgibi: thanks, just waiting on a local func run now and I'll push14:00
gibiack14:00
*** ociuhandu has quit IRC14:04
openstackgerritLee Yarwood proposed openstack/nova master: nova-manage: Add libvirt get_machine_type command  https://review.opendev.org/c/openstack/nova/+/76954814:04
openstackgerritLee Yarwood proposed openstack/nova master: nova-manage: Add libvirt update_machine_type command  https://review.opendev.org/c/openstack/nova/+/77489614:04
openstackgerritLee Yarwood proposed openstack/nova master: nova-manage: Add libvirt list_unset_machine_type command  https://review.opendev.org/c/openstack/nova/+/77489714:04
openstackgerritLee Yarwood proposed openstack/nova master: nova-status: Add hw_machine_type check for libvirt instances  https://review.opendev.org/c/openstack/nova/+/77064314:04
openstackgerritLee Yarwood proposed openstack/nova master: Fix ImageBackendFixture not to support direct_snapshot  https://review.opendev.org/c/openstack/nova/+/77846214:04
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Add a config update workflow test for [libvirt]hw_machine_type  https://review.opendev.org/c/openstack/nova/+/77489814:04
openstackgerritLee Yarwood proposed openstack/nova master: docs: Add admin docs for configuring and updating machine types  https://review.opendev.org/c/openstack/nova/+/77489914:04
gibion it ^^14:05
*** ociuhandu has joined #openstack-nova14:07
gibistephenfin: you blessing is needed on some of the patches in the machine_type series as lyarwood respined it14:13
stephenfinack, will take a look shortly14:13
stephenfin(in a meeting atm :()14:13
*** links has quit IRC14:15
openstackgerritLee Yarwood proposed openstack/nova master: libvirt: Simplify device_path check in _detach_encryptor  https://review.opendev.org/c/openstack/nova/+/77846314:16
*** jangutter_ has quit IRC14:16
*** jangutter has joined #openstack-nova14:16
*** takamatsu has joined #openstack-nova14:17
*** jdillaman has quit IRC14:18
*** jdillaman has joined #openstack-nova14:18
stephenfingibi: Need your +2 again on https://review.opendev.org/c/openstack/nova/+/759402/ too. I just changed the commit message per bauzas' request14:20
stephenfinPlease and thanks :)14:20
gibion it14:20
*** songwenping__ has quit IRC14:20
bauzasgibi: stephenfin: fast approved as this is only a commit msg update14:20
stephenfinsweet14:20
stephenfinI'll rebase the rest of that series on top of that shortly so14:21
gibistading down...14:21
gibianyhow it failed in zuul with a valid reason. unused import /o\14:25
stephenfinoh, weird14:27
*** iurygregory has quit IRC14:27
*** ociuhandu has quit IRC14:27
*** ociuhandu has joined #openstack-nova14:28
*** iurygregory has joined #openstack-nova14:29
stephenfinI think I broke Gerrit14:30
stephenfinStuck on publishing edit for at least 2 minutes /o\14:31
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Ocata database migrations  https://review.opendev.org/c/openstack/nova/+/75940214:31
stephenfinphew14:31
stephenfingibi: bauzas: Removed that import. Conflict was due to I532c7918a8e2c887f29d2f0e1e33b80f2b3a750714:31
bauzas+Wd14:32
gibiyepp14:33
*** ociuhandu has quit IRC14:33
gibistephenfin: this needs a +A (there is 2 +2) https://review.opendev.org/c/openstack/nova/+/757615/814:35
gibialso if somebody could +2 the https://review.opendev.org/c/openstack/nova/+/778177/2 that would be nice to unblock the hyperv rbd series14:39
*** ociuhandu has joined #openstack-nova14:40
gmanngibi: done on 77817714:42
gibithanks!14:42
gmannremoving indirect deps will help in these kind of constraints issue14:42
gibigmann: we have a PTG topic around that https://etherpad.opendev.org/p/nova-xena-ptg L3914:44
gmanngibi: +1 and I think by this or next week I will be able to get consensus in TC also and have some reference document too.14:45
gibicool14:45
gmannstephenfin: do you need help on this? you might have forget this policy change - https://review.opendev.org/c/openstack/nova/+/76579814:46
gmannlast one for that BP as client one is also +A now14:46
stephenfingmann: Yes, please. I've gotten dragged into UEFI secure boot reviews and haven't been able to round back on that14:46
gmannstephenfin: sure.14:47
*** zoharm has quit IRC14:47
stephenfingibi: +W on https://review.opendev.org/c/openstack/nova/+/757615/ In general I have no issues with you +W your own patches after a rebase and I do it myself all the time. I doubt anyone would disagree :)14:57
*** zoharm has joined #openstack-nova14:57
gibiack, thanks14:57
*** jdillaman has quit IRC15:00
*** mkrai has joined #openstack-nova15:02
*** LinPeiWen has quit IRC15:06
*** lpetrut has quit IRC15:09
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Pike database migrations  https://review.opendev.org/c/openstack/nova/+/75940315:11
*** happyhemant has joined #openstack-nova15:13
ralonsohhi stephenfin, I have one quick question. When you migrated to privsep, did you have any long lived process to be executed as root?15:16
ralonsohin other words, do you spawn any daemon/server/etc that requires root permissions?15:16
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Queens database migrations  https://review.opendev.org/c/openstack/nova/+/75940415:17
stephenfinralonsoh: I don't think so. We just escalate when we need to15:17
ralonsohstephenfin, without rootwrap, of course15:18
stephenfiniirc privsep is run under a rootwrap daemon15:18
stephenfinso rootwrap is still there15:18
ralonsohyes, but only privsep15:19
*** luksky has quit IRC15:19
*** luksky has joined #openstack-nova15:19
*** luksky has quit IRC15:19
ralonsohI mean, the privsep daemon. In anycase, you still use rootwrap15:19
stephenfinMostly. I recall there being some rootwrap filters left for some os-brick stuff15:20
atmarkhello, is there to another option in the config to move the cold migration target to different interface? I know for live you can set live_migration_inbound_addr but how about for cold migration?15:21
ralonsohstephenfin, thanks!!15:21
stephenfinatmark: I'm not aware of such a mechanism, no15:21
atmarkThe cold migration still uses the management interface even if I have live_migration_inbound_addr15:23
atmarkin a different interface15:24
sean-k-mooneyatmark: yes it will15:33
sean-k-mooneylive_migration_inbound_addr is only for live migration15:33
sean-k-mooneyyou can adres this via kernel routes15:34
*** luksky has joined #openstack-nova15:36
*** hemna has quit IRC15:39
*** hemna has joined #openstack-nova15:40
*** dklyle has joined #openstack-nova15:40
*** jangutter_ has joined #openstack-nova15:51
*** jangutter has quit IRC15:54
*** zoharm has quit IRC15:55
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Rocky database migrations  https://review.opendev.org/c/openstack/nova/+/75940516:02
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Stein database migrations  https://review.opendev.org/c/openstack/nova/+/75940616:03
openstackgerritStephen Finucane proposed openstack/nova master: apidb: Compact Train database migrations  https://review.opendev.org/c/openstack/nova/+/77142016:05
*** mkrai has quit IRC16:05
*** mkrai has joined #openstack-nova16:05
*** takamatsu has quit IRC16:11
*** ociuhandu has quit IRC16:15
openstackgerritSylvain Bauza proposed openstack/nova master: Bump the Compute RPC API to version 6.0  https://review.opendev.org/c/openstack/nova/+/76145216:18
bauzasdansmith: gibi: stephenfin: artom: updated based on your feedbacks ^16:19
*** gokhani has quit IRC16:19
gibibauzas: looking16:19
bauzastl;dr: removed the unused args from the RPC methods stephenfin told + changed the numa tests to pin to 5.max instead16:20
*** mkrai has quit IRC16:23
openstackgerritMerged openstack/python-novaclient master: Add support for microversion v2.88  https://review.opendev.org/c/openstack/python-novaclient/+/77057316:23
belmoreirarecently found this behaviour in Nova: https://bugs.launchpad.net/nova/+bug/1917645 and I'm not sleeping well since :) not sure if this should be Nova or Oslo. I would appreciate some guidance.16:25
openstackLaunchpad bug 1917645 in OpenStack Compute (nova) "Nova can't create instances if RabbitMQ notification cluster is down" [Undecided,New]16:25
*** smcginnis has quit IRC16:25
*** macz_ has joined #openstack-nova16:25
bauzasbelmoreira: if the rabbit is down, how the conductor and scheduler could help the nova-api service to tell which host ?16:29
bauzasthat actually reminds me preemptible instances16:29
belmoreirawe have an independent rabbit for everything16:29
bauzassome kind of instance that would be pre-created16:29
belmoreirabauzas each cell has it's own rabbit, including one for the support conductor/scheduler. Then we have a rabbit for the notifications16:30
belmoreirasupport/super16:31
bauzasoh sorry, I missed the fact you were mentioning the notifications rabbit16:31
*** ociuhandu has joined #openstack-nova16:31
bauzasand not the API MQ or the cell MQs16:31
bauzasI guess we hold on emitting notifications16:32
belmoreiraI was expecting that behaviour (maybe with an error msg), but in reality instances can't be created16:33
belmoreirawas looking into the config options that I think this is not handled at all16:33
gibibelmoreira: I think we need to warp the notification sending with some exception handler and log a WARNING if the notification sending failed but not block the actual work16:33
*** smcginnis has joined #openstack-nova16:34
belmoreiragibi +116:35
gibibelmoreira: I think we simply did not have those error handled properly in the current code16:35
gibione can argue that not delivering a notification could mean some external system become desynced16:36
belmoreiragibi I agree, but in that case should be configurable16:36
gibibelmoreira: hm, that could work. Something like notification_failure_is_fatal config option16:37
gibiso if somebody use the notification interface for charging customers based on usage then that deployer would likely make this config True16:38
belmoreiramakes sense to me16:38
gibime to16:41
gibio16:41
belmoreiraI'm not familiar with the notifications code. Is this something that someone can have a look?16:44
gibiI can take a look but my backlog is pretty long so it will take time to reach that bug16:44
*** macz_ has quit IRC16:46
belmoreirathanks gibi, meanwhile I can have a look but definitely I will need some guidance16:46
gibibelmoreira: sure, let me know if you have questions16:47
*** macz_ has joined #openstack-nova16:47
belmoreirathanks a lot16:48
*** Madkiss has quit IRC16:48
bauzasgibi: belmoreira: sorry wrapped into a meeting, but the approach looks good to me16:49
*** mlavalle has joined #openstack-nova16:51
openstackgerritBalazs Gibizer proposed openstack/nova master: Remove non-libguestfs file injection for libvirt  https://review.opendev.org/c/openstack/nova/+/32472016:57
openstackgerritBalazs Gibizer proposed openstack/nova master: Remove VFSLocalFS  https://review.opendev.org/c/openstack/nova/+/77850616:58
*** luksky has quit IRC16:59
kashyapgibi: Interesting that you revived it (I agree).  Anything in particular that made you revive?16:59
gibikashyap: the security bug behind it become public a week ago17:00
gibikashyap: and also we removed Xen support so one less complication17:00
kashyapArgh, rotting security bugs :-(17:00
gibiyepp17:00
*** luksky has joined #openstack-nova17:00
*** hemna has quit IRC17:01
*** ralonsoh has quit IRC17:01
*** pmannidi has quit IRC17:01
*** hoonetorg has quit IRC17:01
*** stand has quit IRC17:01
*** etp has quit IRC17:01
*** lifeless_ has quit IRC17:01
*** mgoddard has quit IRC17:01
*** hack-char has quit IRC17:01
kashyapgibi: Right; fair enough.  That problem is real ...17:01
*** lucasagomes has quit IRC17:02
*** openstackgerrit has quit IRC17:02
*** spatel has quit IRC17:03
bauzasgibi: urgent review needs on them, I guess ?17:03
gibibauzas: no, it is a really old security bug17:04
gibiso no need to rush17:04
gibiit just become stuck in private state until the secu team did a spring cleaning recently17:04
bauzasgibi: okay, focusing on blueprints reviews, but I can take a look at them later17:04
gibiand made the bug public17:04
gibibauzas: sure, thanks17:05
*** ralonsoh has joined #openstack-nova17:05
*** stand has joined #openstack-nova17:05
*** hemna has joined #openstack-nova17:07
*** pmannidi has joined #openstack-nova17:07
*** hoonetorg has joined #openstack-nova17:07
*** etp has joined #openstack-nova17:07
*** lifeless_ has joined #openstack-nova17:07
*** mgoddard has joined #openstack-nova17:07
*** hack-char has joined #openstack-nova17:07
*** spatel has joined #openstack-nova17:07
sean-k-mooneythats the one we talked about 2 weeks ago in teh meeting right17:10
sean-k-mooneyi assume you revied the old patches17:10
gibisean-k-mooney: yes, I revived, rebased, and realized the we deleted Xen since so I put a cleanup top of it17:11
gibisean-k-mooney: but the basic idea is the same17:11
gibisean-k-mooney: fail to boot if file injection is requested but libguestfs is not available on the compute17:11
sean-k-mooneycool ill try and review this this week17:11
gibisean-k-mooney: thanks17:11
sean-k-mooneyby the way i saw your question on the port numa patches. ill hopefully get time to rebase that tomorrow to address it i just need to fix that env i was using it for something else but ill do that in the morning17:12
sean-k-mooneythanks for taking a look17:12
gibiack17:13
bauzassean-k-mooney: just a quick q, why do we need to pass the list of ARQs when shelving an instance ? I guess this is for the cyborg-agent to free up the resources ?17:19
bauzascontext : https://review.opendev.org/c/openstack/nova/+/778440/1/nova/compute/manager.py17:19
*** ociuhandu_ has joined #openstack-nova17:21
*** nightmare_unreal has quit IRC17:23
*** ociuhandu has quit IRC17:25
sean-k-mooneybauzas: we need to free them yes17:25
sean-k-mooneyso its for unbinding them17:25
sean-k-mooneytechnially its only needed for the shelve_offload part17:25
*** ociuhandu_ has quit IRC17:26
bauzasyup, that's what I guessed17:26
*** macz_ has quit IRC17:27
*** ociuhandu has joined #openstack-nova17:32
sean-k-mooneyi commented in line but i dont think this is a ddos vector really17:36
sean-k-mooneythe new api query only happens if the instance has cyborg resoucs. and it will happen only once per shleved instance17:36
*** ociuhandu has quit IRC17:37
sean-k-mooneybauzas: we also prefilter the list by the timeout and only do this for instance that have exceed the time out so we wont check this on every iteration of the perodic17:38
sean-k-mooneypulling the client out of the loop is not a bad idea17:39
bauzassure, but I wonder whether some malicious user could create 10000 small instances by one and wait for 3600secs17:39
sean-k-mooneyi mean they would hit there instance quota right17:40
sean-k-mooneyshelved instances still count to that17:40
bauzasthe problem is that we call N times the cyborg api17:40
bauzasat the same time17:40
bauzasand you multiply by the periodic value17:40
sean-k-mooneyya but you cant avoid that wihout caching the info in nova which we do not do intentionally17:41
bauzasmaybe not an attack vector but some performance impact for sure17:41
sean-k-mooneyi dont think it will be17:41
bauzason a large cloud with 10000 instances being shelved at the same time from the same tenant, cyborg will face 10000 times a connection roundtrip17:42
sean-k-mooneyif the nova api is beefy enought to handel the 10000 shleve api calls then the cyborg one should be able to handel 10000 arq lookups17:42
bauzasfrom different tenants*17:42
bauzasthat's a periodic17:42
bauzasnot an API straight call17:42
sean-k-mooneysure i know17:42
bauzasduring those 3600 secs, you can create and shelve as much instances as you want17:42
sean-k-mooneyright but we defualt to 017:43
bauzasbut once the periodic runs, it will pick all the shelved instances during this window17:43
sean-k-mooneye.g. offloading without a delay17:43
sean-k-mooneyso for it to be an issue the operator has to opt in to offloading after a period of time and increase it enouch for the shelved instance to build up enough to ddos the cyborg api17:44
sean-k-mooneypragmatically i dont think we will enough user of cyborg+shelve +that non default config for this to realisticlly happen17:44
sean-k-mooneyit could but the instance.save() before this would propably ddos the db before the cyborg issue was hit17:45
sean-k-mooneyim not saying it not a valid concern i just dont think it makes it substantailly worse then it would be already17:46
bauzassean-k-mooney: I'm just saying "doc it"17:46
sean-k-mooneywell we should doc the instance.save then too right17:46
bauzasbecause shelving has a very specific implication now if you use cyborg17:46
sean-k-mooneywell shelve offload already calls out to cinder17:47
bauzaswell, I'm pretty sure ops size the DB write performance on the nova database17:47
sean-k-mooneyto remove the volumn attaments right?17:47
sean-k-mooneyand it should be calling neutron to unbind the neutron ports17:47
sean-k-mooneyso im wonderign wny cyborg is special in this case17:48
bauzasactually, you're making a point17:48
bauzasbut that doesn't infirm my theory17:48
bauzaswhich is that this periodic task can harm our dependent projects17:49
sean-k-mooneywell im not disagreeing that it could cause issue if you misconfigred things17:49
sean-k-mooneyyep17:49
sean-k-mooneyi think we just have differnet levels of concners as to the effect17:49
bauzasanyway, I don't wanna hold this17:50
bauzasbut I was just taking my operator hat, being afraid of my internal APIs17:50
sean-k-mooneymoving the client out of the loop will at least stop the load on keystone17:50
sean-k-mooneyso that proably shoudl be done in anycase17:50
bauzasand while I trust cinder and neutron in terms of sizing, I'd certainly take much attention to cyborg API if I know this17:51
sean-k-mooneyya there is less op experince with cyborg for sure17:51
bauzasanyway, this is nearly 7pm for me and I really worked late yesterday night, so I'll bail out17:51
sean-k-mooneythat said at least with osp donwstream we tend to use one db server for all the dbs17:52
sean-k-mooneyenjoy your evnening o/17:52
* bauzas remembers when he was designing different DBs for different read/write needs :)17:52
bauzaslike, hello keystone17:52
bauzasbear with me17:53
bauzasand give me a token that you'll store for a long period of time17:53
bauzasglance ? be quiet and just give me the images17:53
bauzasbut meh, I'm diverging17:54
bauzas\o17:54
sean-k-mooneyim sure cern use different dbs and rabbitmq instance for different services but many smaller clouds wont17:55
*** derekh has quit IRC18:05
*** rpittau is now known as rpittau|afk18:11
*** andrewbonney has quit IRC18:23
*** ralonsoh has quit IRC18:23
*** lpetrut has joined #openstack-nova18:38
*** xek has quit IRC18:47
*** lpetrut has quit IRC18:49
*** tbachman_ has joined #openstack-nova19:27
*** tbachman has quit IRC19:31
*** tbachman_ is now known as tbachman19:31
*** whoami-rajat has quit IRC19:50
*** lifeless_ is now known as lifeless19:56
*** spatel has quit IRC19:58
*** spatel has joined #openstack-nova20:00
*** slaweq has quit IRC20:04
*** happyhemant has quit IRC20:13
*** spatel has quit IRC20:35
*** spatel has joined #openstack-nova20:38
*** rcernin has joined #openstack-nova20:46
*** k_mouza has quit IRC20:49
*** openstackgerrit has joined #openstack-nova20:56
openstackgerritMerged openstack/nova master: requirements.txt: Bump os-brick to 4.2.0  https://review.opendev.org/c/openstack/nova/+/77817720:56
*** slaweq has joined #openstack-nova20:57
*** gyee has joined #openstack-nova21:01
openstackgerritMerged openstack/nova master: Remove unused Instance method  https://review.opendev.org/c/openstack/nova/+/75761521:01
*** slaweq has quit IRC21:07
*** rcernin has quit IRC21:17
*** gary_perkins has joined #openstack-nova21:27
*** irclogbot_3 has quit IRC21:33
*** bauzas has quit IRC21:33
*** sean-k-mooney has quit IRC21:33
*** ianw has quit IRC21:33
*** tristanC has quit IRC21:33
*** DinaBelova has quit IRC21:33
*** amotoki has quit IRC21:33
*** zer0c00l has quit IRC21:33
*** atmark has quit IRC21:33
*** johnthetubaguy has quit IRC21:33
*** irclogbot_3 has joined #openstack-nova21:34
*** bauzas has joined #openstack-nova21:34
*** sean-k-mooney has joined #openstack-nova21:34
*** ianw has joined #openstack-nova21:34
*** atmark has joined #openstack-nova21:34
*** tristanC has joined #openstack-nova21:34
*** DinaBelova has joined #openstack-nova21:34
*** amotoki has joined #openstack-nova21:34
*** zer0c00l has joined #openstack-nova21:34
*** johnthetubaguy has joined #openstack-nova21:34
openstackgerritMerged openstack/nova master: hyper-v rbd volume support  https://review.opendev.org/c/openstack/nova/+/76355021:38
*** zzzeek has quit IRC21:44
*** zzzeek has joined #openstack-nova21:45
*** rcernin has joined #openstack-nova22:05
*** rcernin has quit IRC22:11
*** rcernin has joined #openstack-nova22:11
*** jamesdenton has quit IRC22:15
*** jamesdenton has joined #openstack-nova22:15
*** spatel has quit IRC22:28
*** belmoreira has quit IRC22:35
*** spatel has joined #openstack-nova22:39
*** spatel has quit IRC22:39
*** k_mouza has joined #openstack-nova22:49
*** efried has quit IRC22:53
*** k_mouza has quit IRC22:54
*** efried has joined #openstack-nova22:54
*** rcernin has quit IRC22:56
*** rcernin has joined #openstack-nova23:01
*** k_mouza has joined #openstack-nova23:17
*** k_mouza has quit IRC23:22
*** rcernin has quit IRC23:26
*** k_mouza has joined #openstack-nova23:27
*** k_mouza has quit IRC23:31
*** k_mouza has joined #openstack-nova23:36
*** luksky has quit IRC23:39
*** zzzeek has quit IRC23:40
*** k_mouza has quit IRC23:41
*** mgariepy has quit IRC23:42
*** zzzeek has joined #openstack-nova23:42
*** vishalmanchanda has quit IRC23:44
*** mgariepy has joined #openstack-nova23:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!