| artom | alex_xu, sure, I'll take another look tomorrow | 00:08 |
|---|---|---|
| *** tetsuro has joined #openstack-nova | 00:10 | |
| *** tetsuro has quit IRC | 00:12 | |
| *** tetsuro has joined #openstack-nova | 00:13 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add new default roles in os-flavor-access policies https://review.opendev.org/713697 | 00:18 |
| *** tetsuro has quit IRC | 00:19 | |
| *** tetsuro has joined #openstack-nova | 00:20 | |
| *** sean-k-mooney2 has joined #openstack-nova | 00:26 | |
| *** sean-k-mooney1 has quit IRC | 00:27 | |
| *** nicolasbock has quit IRC | 00:28 | |
| *** tetsuro_ has joined #openstack-nova | 00:30 | |
| *** tetsuro has quit IRC | 00:33 | |
| *** mlavalle has quit IRC | 00:40 | |
| *** martinkennelly has quit IRC | 00:48 | |
| *** tosky has quit IRC | 00:50 | |
| *** brinzhang has joined #openstack-nova | 00:52 | |
| *** sean-k-mooney2 has quit IRC | 00:53 | |
| *** TxGirlGeek has quit IRC | 00:53 | |
| *** sean-k-mooney2 has joined #openstack-nova | 00:55 | |
| *** brinzhang has quit IRC | 00:57 | |
| *** brinzhang has joined #openstack-nova | 00:57 | |
| openstackgerrit | Merged openstack/nova master: Refine and introduce correct parameters for test_get_guest_config_numa_host_instance_topo_cpu_pinning https://review.opendev.org/713351 | 01:05 |
| *** sean-k-mooney2 has quit IRC | 01:08 | |
| *** Liang__ has joined #openstack-nova | 01:15 | |
| openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | 01:18 |
| openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | 01:18 |
| *** brinzhang_ has joined #openstack-nova | 01:20 | |
| *** brinzhang has quit IRC | 01:23 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470 | 01:30 |
| *** liuyulong has quit IRC | 01:33 | |
| *** vishalmanchanda has joined #openstack-nova | 01:33 | |
| openstackgerrit | Brin Zhang proposed openstack/nova master: Add new default roles in os-instance-actions policies https://review.opendev.org/706470 | 01:36 |
| *** mgoddard has quit IRC | 01:49 | |
| *** sapd1 has joined #openstack-nova | 01:50 | |
| *** sapd1_x has joined #openstack-nova | 01:52 | |
| *** mgoddard has joined #openstack-nova | 01:57 | |
| *** ociuhandu has joined #openstack-nova | 02:04 | |
| *** ociuhandu has quit IRC | 02:09 | |
| *** zhanglong has joined #openstack-nova | 02:28 | |
| *** psachin has joined #openstack-nova | 02:30 | |
| *** mkrai has joined #openstack-nova | 03:17 | |
| *** tetsuro_ has quit IRC | 03:51 | |
| *** gyee has quit IRC | 04:08 | |
| *** tetsuro has joined #openstack-nova | 04:16 | |
| *** udesale has joined #openstack-nova | 04:32 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Synchronize sqlalchemy models with migrations for alembic 1.4.1 https://review.opendev.org/713778 | 04:39 |
| *** tetsuro_ has joined #openstack-nova | 04:49 | |
| *** tetsuro has quit IRC | 04:52 | |
| *** ratailor has joined #openstack-nova | 04:54 | |
| *** tetsuro has joined #openstack-nova | 05:03 | |
| *** tetsuro_ has quit IRC | 05:06 | |
| *** dave-mccowan has quit IRC | 05:10 | |
| *** tetsuro_ has joined #openstack-nova | 05:15 | |
| *** tetsuro has quit IRC | 05:18 | |
| *** macz_ has joined #openstack-nova | 05:19 | |
| *** macz_ has quit IRC | 05:24 | |
| *** links has joined #openstack-nova | 05:32 | |
| *** evrardjp has quit IRC | 05:36 | |
| *** evrardjp has joined #openstack-nova | 05:36 | |
| *** zhanglong has quit IRC | 05:57 | |
| *** zhanglong has joined #openstack-nova | 05:57 | |
| *** macz_ has joined #openstack-nova | 06:28 | |
| *** macz_ has quit IRC | 06:33 | |
| *** mkrai has quit IRC | 06:35 | |
| *** mkrai has joined #openstack-nova | 06:40 | |
| *** yedongcan has joined #openstack-nova | 06:47 | |
| *** damien_r has quit IRC | 06:51 | |
| *** dpawlik has joined #openstack-nova | 06:53 | |
| *** lbragstad has quit IRC | 07:01 | |
| *** mkrai has quit IRC | 07:02 | |
| *** mkrai_ has joined #openstack-nova | 07:02 | |
| openstackgerrit | Kevin Zhao proposed openstack/nova master: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/712607 | 07:10 |
| *** ociuhandu has joined #openstack-nova | 07:28 | |
| *** amoralej|off is now known as amoralej | 07:32 | |
| *** ociuhandu has quit IRC | 07:33 | |
| *** damien_r has joined #openstack-nova | 07:39 | |
| *** damien_r has quit IRC | 07:44 | |
| *** damien_r has joined #openstack-nova | 07:47 | |
| *** tetsuro_ has quit IRC | 07:49 | |
| *** tesseract has joined #openstack-nova | 07:51 | |
| *** jawad_axd has joined #openstack-nova | 07:51 | |
| *** maciejjozefczyk has joined #openstack-nova | 07:58 | |
| *** klindgren has quit IRC | 08:01 | |
| *** klindgren has joined #openstack-nova | 08:01 | |
| *** threestrands has quit IRC | 08:01 | |
| gibi | good morning nova | 08:02 |
| gibi | stephenfin: hi! dansmith +2 all over the qos remaining patches, could you check back to those? https://review.opendev.org/#/q/topic:bp/support-move-ops-with-qos-ports-ussuri | 08:03 |
| gibi | stephenfin: the major change since you looked at is a compute service version check in the API | 08:04 |
| gibi | to ensure the computes are on Ussuri version before we start moving the servers around as the feauture needs support from the compute service | 08:05 |
| gibi | due to the PCI claim magic | 08:05 |
| *** slaweq has joined #openstack-nova | 08:06 | |
| *** portdirect has quit IRC | 08:08 | |
| *** kevinz has quit IRC | 08:08 | |
| *** guilhermesp has quit IRC | 08:09 | |
| *** fyx has quit IRC | 08:09 | |
| *** portdirect has joined #openstack-nova | 08:10 | |
| *** mnaser has quit IRC | 08:10 | |
| *** kevinz has joined #openstack-nova | 08:11 | |
| *** fyx has joined #openstack-nova | 08:11 | |
| *** mnaser has joined #openstack-nova | 08:11 | |
| *** rpittau|afk is now known as rpittau | 08:11 | |
| *** guilhermesp has joined #openstack-nova | 08:11 | |
| *** tkajinam has quit IRC | 08:12 | |
| *** lpetrut has joined #openstack-nova | 08:18 | |
| *** tosky has joined #openstack-nova | 08:21 | |
| *** zhanglong has quit IRC | 08:22 | |
| *** zhanglong has joined #openstack-nova | 08:27 | |
| *** ygk_12345 has joined #openstack-nova | 08:28 | |
| *** macz_ has joined #openstack-nova | 08:29 | |
| *** klindgren has quit IRC | 08:31 | |
| *** klindgren has joined #openstack-nova | 08:31 | |
| *** macz_ has quit IRC | 08:33 | |
| *** tetsuro has joined #openstack-nova | 08:34 | |
| *** klindgren has quit IRC | 08:38 | |
| *** klindgren has joined #openstack-nova | 08:39 | |
| *** tetsuro_ has joined #openstack-nova | 08:40 | |
| *** tetsuro has quit IRC | 08:43 | |
| *** ralonsoh has joined #openstack-nova | 08:53 | |
| *** xek has joined #openstack-nova | 08:55 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: gate: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | 08:55 |
| nightmare_unreal | hello is there specific channel for python-openstackclient | 08:56 |
| brinzhang_ | It seems there is not, you can send to ML, and add [openstackclient] [your_project] tag | 09:00 |
| brinzhang_ | nightmare_unreal | 09:00 |
| *** psachin has quit IRC | 09:01 | |
| *** psachin has joined #openstack-nova | 09:02 | |
| *** yoctozepto has quit IRC | 09:07 | |
| *** klindgren has quit IRC | 09:10 | |
| *** ociuhandu has joined #openstack-nova | 09:10 | |
| *** klindgren has joined #openstack-nova | 09:11 | |
| *** yoctozepto0 has joined #openstack-nova | 09:16 | |
| *** ociuhandu has quit IRC | 09:22 | |
| *** yoctozepto9 has joined #openstack-nova | 09:27 | |
| *** yoctozepto0 has quit IRC | 09:27 | |
| nightmare_unreal | thanks | 09:28 |
| *** dtantsur|afk is now known as dtantsur | 09:39 | |
| *** jangutter has joined #openstack-nova | 09:40 | |
| *** tetsuro has joined #openstack-nova | 09:40 | |
| *** zhanglong has quit IRC | 09:42 | |
| *** tetsuro_ has quit IRC | 09:43 | |
| luyao | dansmith, artom: Thanks for your comments on https://review.opendev.org/#/c/687856, I replied on them, and thanks for alex_xu helping me clarifying. | 09:43 |
| frickler | nightmare_unreal: brinzhang_: #openstack-sdks | 09:43 |
| *** derekh has joined #openstack-nova | 09:43 | |
| luyao | artom: FYI. I also run your live migration functional tests on top of my patch, they all passed | 09:44 |
| *** Liang__ has quit IRC | 09:45 | |
| *** ociuhandu has joined #openstack-nova | 09:54 | |
| *** tetsuro has quit IRC | 09:56 | |
| *** vishalmanchanda has quit IRC | 10:03 | |
| *** jangutter has quit IRC | 10:07 | |
| *** amoralej is now known as amoralej|afk | 10:13 | |
| *** kaisers_ has joined #openstack-nova | 10:16 | |
| *** mkrai_ has quit IRC | 10:19 | |
| openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | 10:20 |
| *** macz_ has joined #openstack-nova | 10:20 | |
| *** vishalmanchanda has joined #openstack-nova | 10:22 | |
| openstackgerrit | Arthur Dayne proposed openstack/nova master: libvirt:driver:Disallow AIO=native when 'O_DIRECT' is not available https://review.opendev.org/682772 | 10:22 |
| *** macz_ has quit IRC | 10:25 | |
| *** mkrai_ has joined #openstack-nova | 10:30 | |
| openstackgerrit | Merged openstack/nova master: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713035 | 10:30 |
| *** mkrai has joined #openstack-nova | 10:32 | |
| *** mkrai_ has quit IRC | 10:35 | |
| *** tesseract has quit IRC | 10:36 | |
| *** yoctozepto9 is now known as yoctozepto | 10:37 | |
| *** tesseract has joined #openstack-nova | 10:42 | |
| *** tesseract has quit IRC | 10:46 | |
| *** tesseract has joined #openstack-nova | 10:48 | |
| *** ivve has joined #openstack-nova | 10:55 | |
| *** ociuhandu has quit IRC | 10:58 | |
| *** tesseract-RH has joined #openstack-nova | 11:00 | |
| lyarwood | wow it finally landed | 11:01 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: gate: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | 11:01 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | 11:03 |
| lyarwood | ^ sorry just trying to get this some additional runs before review to ensure we don't need the VIF timeout again | 11:03 |
| *** tesseract has quit IRC | 11:03 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713836 | 11:03 |
| *** ociuhandu has joined #openstack-nova | 11:03 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713837 | 11:04 |
| *** sapd1_x has quit IRC | 11:04 | |
| *** ociuhandu has quit IRC | 11:04 | |
| *** ociuhandu has joined #openstack-nova | 11:05 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713839 | 11:05 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/rocky: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713840 | 11:05 |
| *** sapd1 has quit IRC | 11:05 | |
| *** nicolasbock has joined #openstack-nova | 11:06 | |
| kashyap | lyarwood: Why sorry? | 11:07 |
| kashyap | lyarwood: You're doing the donkey work, people better be grateful ;-) | 11:07 |
| kashyap | I mean to say it in a _good sense_, BTW! | 11:08 |
| lyarwood | thanks :) | 11:10 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Stop setting glance_api_version in cinder.conf in nova-live-migration https://review.opendev.org/713843 | 11:10 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713844 | 11:10 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/queens: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713845 | 11:10 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036 | 11:15 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713848 | 11:15 |
| *** dtantsur is now known as dtantsur|afk | 11:20 | |
| *** klindgren has quit IRC | 11:23 | |
| *** klindgren has joined #openstack-nova | 11:24 | |
| *** rpittau is now known as rpittau|bbl | 11:30 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | 11:34 |
| brinzhang_ | damsmith: I have replied your question in https://review.opendev.org/#/c/693828/ | 11:34 |
| *** ccamacho has quit IRC | 11:38 | |
| *** ccamacho has joined #openstack-nova | 11:44 | |
| *** ociuhandu has quit IRC | 11:50 | |
| *** udesale_ has joined #openstack-nova | 12:19 | |
| *** udesale has quit IRC | 12:20 | |
| *** ociuhandu has joined #openstack-nova | 12:22 | |
| openstackgerrit | Johannes Kulik proposed openstack/nova master: Don't recompute weighers' minval/maxval attributes https://review.opendev.org/713863 | 12:27 |
| *** amoralej|afk is now known as amoralej | 12:28 | |
| *** PetrTuma has joined #openstack-nova | 12:28 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: Replace ansible --sudo with --become in live_migration/hooks scripts https://review.opendev.org/713848 | 12:28 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036 | 12:28 |
| *** ociuhandu has quit IRC | 12:28 | |
| openstackgerrit | Johannes Kulik proposed openstack/nova master: Don't recompute weighers' minval/maxval attributes https://review.opendev.org/713863 | 12:34 |
| *** PetrTuma has quit IRC | 12:39 | |
| *** jraju__ has joined #openstack-nova | 12:43 | |
| *** links has quit IRC | 12:44 | |
| *** PetrTuma has joined #openstack-nova | 12:46 | |
| *** sean-k-mooney2 has joined #openstack-nova | 12:53 | |
| openstackgerrit | Merged openstack/nova master: images: Allow the output format of qemu-img info to be controlled https://review.opendev.org/706898 | 12:53 |
| *** sean-k-mooney2 is now known as sean-k-mooney | 12:53 | |
| *** rpittau|bbl is now known as rpittau | 12:59 | |
| *** jangutter has joined #openstack-nova | 13:07 | |
| *** nweinber has joined #openstack-nova | 13:07 | |
| *** mkrai has quit IRC | 13:09 | |
| lyarwood | sean-k-mooney: when you're around, PetrTuma had some quesitons about https://bugs.launchpad.net/nova/+bug/1763766 and stable/rocky | 13:12 |
| openstack | Launchpad bug 1763766 in OpenStack Compute (nova) queens "nova needs to disallow resource consumption changes on image rebuild" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | 13:12 |
| sean-k-mooney | lyarwood: i can take a look now | 13:12 |
| PetrTuma | hello, hold on :) | 13:12 |
| PetrTuma | I'm wondering about state of https://bugs.launchpad.net/nova/+bug/1763766 in Rocky release, from the bug report it would seem Rocky should be affected by this (there is fix proposed to Rocky). I'm trying to reproduce the issue, but no matter what combination of flavors / images with different sets of properties I use, the resource consumption is | 13:12 |
| PetrTuma | not changed at all during rebuild (all properties on image I'm rebuilding to are ignored). This is follow up on https://bugs.launchpad.net/nova/+bug/1804502. We had to include the fix in our Rocky (disable the filter on rebuild), because we were having the issue similar to the one described in the bug report (i.e. rebuild failing on compute host | 13:12 |
| openstack | Launchpad bug 1804502 in OpenStack Compute (nova) rocky "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood) | 13:12 |
| PetrTuma | nearly filled up to its capacity by NUMA instances, because the NUMATopologyFilter returns empty set of hosts). I'm not sure what I might be doing wrong that I don't see the resource change during rebuild or did I miss some special condition? | 13:12 |
| sean-k-mooney | PetrTuma: during a rebuil you should _NOT_ be able to change the resouce usage | 13:13 |
| *** lbragstad has joined #openstack-nova | 13:13 | |
| sean-k-mooney | so if you should not be able to rebuild from an image with 1 numa node to 2 numa nodes | 13:14 |
| sean-k-mooney | or form a pinned immage to an unpined one | 13:14 |
| sean-k-mooney | so the fact that the topology you are rebuilding too is ignored is partly correct | 13:14 |
| sean-k-mooney | the fix on master is to block any rebuild attpemnt that would alter the numa toplogy requested | 13:15 |
| sean-k-mooney | so with this backported the expected behavior would be that the rebuild is rejected by the api | 13:15 |
| sean-k-mooney | we can only fix https://bugs.launchpad.net/nova/+bug/1804502 | 13:16 |
| openstack | Launchpad bug 1804502 in OpenStack Compute (nova) rocky "Rebuild server with NUMATopologyFilter enabled fails (in some cases)" [Medium,In progress] - Assigned to Lee Yarwood (lyarwood) | 13:16 |
| sean-k-mooney | by adressing https://bugs.launchpad.net/nova/+bug/1763766 and explictily blocking the case where the toplogy would change | 13:17 |
| openstack | Launchpad bug 1763766 in OpenStack Compute (nova) queens "nova needs to disallow resource consumption changes on image rebuild" [Medium,In progress] - Assigned to sean mooney (sean-k-mooney) | 13:17 |
| sean-k-mooney | in doing so we can disabel the numa toplogy filter safely on rebuild | 13:17 |
| sean-k-mooney | it is not correct to disabel the numa toplogy filter on rebuild without blocking numa toplogy changes first | 13:17 |
| PetrTuma | well, that's the thing, if I try to rebuild instance with hw_numa_node=1 to image with hw_numa_node=2 I have same result, resources of the instance are unchanged (this is the case where I don't have the API change to block this in my nova) | 13:17 |
| sean-k-mooney | PetrTuma: does ^ make sense | 13:17 |
| sean-k-mooney | PetrTuma: yes that is becasue rebuild uses a noop claim | 13:18 |
| sean-k-mooney | so the resouces calimed on the host are not updated | 13:18 |
| *** nicolasbock has quit IRC | 13:18 | |
| sean-k-mooney | but we considerd it to be a bug to allow you to requst a numa toplogy and not honour that requst | 13:19 |
| sean-k-mooney | which is why we block it | 13:19 |
| PetrTuma | yes, I understand that, I was trying to actually somehow recreate the issue, but so far I failed utterly | 13:19 |
| sean-k-mooney | well recreating is simple you just boot a vm with 1 numa node and rebuild to 2 | 13:20 |
| sean-k-mooney | on rocky without any backport that will result in the image stating the vm has 2 numa nodes but it will only have 1 | 13:20 |
| sean-k-mooney | until you cold migrate and then it will change to two | 13:20 |
| *** nicolasbock has joined #openstack-nova | 13:21 | |
| sean-k-mooney | if you live migrate it will schdule based on 2 claime 2 and pin to one. | 13:21 |
| *** psachin has quit IRC | 13:21 | |
| sean-k-mooney | so live migration wont fix the inconsetnecy and will like cause others | 13:21 |
| PetrTuma | ah, so that's what I misunderstood | 13:22 |
| PetrTuma | it's not the issue with rebuild itself, but any subsequent action that might be run on the instance | 13:22 |
| sean-k-mooney | yes | 13:23 |
| sean-k-mooney | cold migate and shelve/unshevel could fix the instnace. as would evacuate i guess | 13:23 |
| sean-k-mooney | but live migate, hard reboot rescue would not help | 13:23 |
| PetrTuma | ok, thanks for clarification, I'm going to run some more tests then (I have the fix backported to my nova) | 13:24 |
| sean-k-mooney | sure let me know if you have any specific questions. | 13:24 |
| sean-k-mooney | we technically could allow rebuild to change resouce usage but its invaive as we would have to alter rebuild to not use a noop claim and instead use a move claim. basicaly allowing a rebuild to change host | 13:25 |
| sean-k-mooney | like a resize | 13:25 |
| lyarwood | sean-k-mooney: https://review.opendev.org/#/c/713674/ is ready for review again btw, had to remember to escapte a few things. | 13:26 |
| *** derekh has quit IRC | 13:26 | |
| sean-k-mooney | that is not something that would be backportable and im not sure its something we want to do | 13:27 |
| sean-k-mooney | lyarwood: cool | 13:27 |
| sean-k-mooney | so ya the virsh list was not evaluated on the target host | 13:27 |
| sean-k-mooney | i was assuming it would not be | 13:27 |
| sean-k-mooney | look like the job passed which is a good sign | 13:28 |
| lyarwood | yeah I haven't seen it fail yet since we stopped all of the devstack@* services on the subnode | 13:28 |
| lyarwood | I also don't see the ceph errors in n-cpu on the primary host | 13:29 |
| sean-k-mooney | presumably because the vm is actully stopped on the source host | 13:29 |
| lyarwood | yeah | 13:29 |
| lyarwood | just confirming | 13:29 |
| sean-k-mooney | lyarwood: ya so i think that looks good | 13:31 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Use domain capabilities to get supported device models https://review.opendev.org/666915 | 13:32 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add transform_image_metadata request filter https://review.opendev.org/665775 | 13:32 |
| sean-k-mooney | https://review.opendev.org/#/c/713342/1 isnt doing what i wanted it too. i might take a look at it again but i think im going to rebase it on top of your change | 13:33 |
| sean-k-mooney | im seeing some port not fond messages with it | 13:33 |
| stephenfin | gibi, alex_xu: Addressed your comments in that disk image metadata series ^ | 13:33 |
| sean-k-mooney | which makes me think its doing more then removing the binding host | 13:34 |
| gibi | stephenfin: ack | 13:34 |
| alex_xu | stephenfin: got it | 13:35 |
| alex_xu | sean-k-mooney: stephenfin just replied https://review.opendev.org/#/c/665775/25/nova/conf/scheduler.py@203, maybe I missunderstand sean-k-mooney's comment | 13:35 |
| sean-k-mooney | also jsut replied | 13:35 |
| alex_xu | sean-k-mooney: yes, that is what I said | 13:36 |
| stephenfin | alex_xu: Yeah, you're correct. That's what I've said in the new patchset I think | 13:36 |
| sean-k-mooney | you are saying we should not set it to true until all host are upgraded yes | 13:36 |
| alex_xu | yes | 13:36 |
| stephenfin | In addition to saying this only works with libvirt | 13:36 |
| stephenfin | for now, anyway | 13:36 |
| alex_xu | at least not suggest to set as true | 13:36 |
| sean-k-mooney | alex_xu: ya so it default to false for two reasons. | 13:37 |
| sean-k-mooney | one not all virt drivers support it | 13:37 |
| alex_xu | ah, right | 13:37 |
| sean-k-mooney | and second because on upgrade you want to finish the upgrade before enableing it | 13:37 |
| alex_xu | ack | 13:38 |
| sean-k-mooney | we wont be able to change that default untill all the in tree virt drivers supprot this | 13:38 |
| sean-k-mooney | so it will proably always default to false | 13:38 |
| sean-k-mooney | at least in the short to medium term | 13:38 |
| alex_xu | agree with that | 13:38 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: nova-live-migration: Wait for n-cpu services to come up after configuring Ceph https://review.opendev.org/713036 | 13:39 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: pike-only: remove broken non-voting ceph jobs https://review.opendev.org/700072 | 13:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: Init HostState.failed_builds https://review.opendev.org/669538 | 13:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: Mask the token used to allow access to consoles https://review.opendev.org/708876 | 13:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: Remove exp legacy-tempest-dsvm-full-devstack-plugin-nfs https://review.opendev.org/702061 | 13:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/pike: Improve metadata server performance with large security groups https://review.opendev.org/697523 | 13:46 |
| lyarwood | gibi / stephenfin: https://review.opendev.org/#/c/713674/ - can you take a look at this nova-live-migration fix today please. | 13:47 |
| *** mkrai has joined #openstack-nova | 13:48 | |
| *** StevenK has quit IRC | 13:49 | |
| gibi | lyarwood: do you need to make sure that the stopped services started later? or the whole subnode will be rebooted? | 13:50 |
| *** StevenK has joined #openstack-nova | 13:50 | |
| lyarwood | gibi: evacuation is the final thing that happens in the job at the moment | 13:50 |
| sean-k-mooney | gibi: we do the evacuation tests after all the others | 13:50 |
| gibi | OK. thanks. then it make sense | 13:51 |
| lyarwood | yeah the previous behaviour was to leave libvirt stopped on the subnode anyway | 13:51 |
| lyarwood | I'm still plannig to break this out into seperate jobs btw | 13:51 |
| *** ygk_12345 has left #openstack-nova | 13:51 | |
| lyarwood | Just want to get it stable again before I start with that | 13:51 |
| sean-k-mooney | yep makes sense | 13:51 |
| gibi | lyarwood: thank you for taking care of this. +2 | 13:52 |
| lyarwood | np it has stopped my stuff from landing so this is the least I could do ;) | 13:52 |
| *** amoralej is now known as amoralej|lunch | 13:53 | |
| *** mkrai has quit IRC | 13:54 | |
| gibi | lyarwood: you are unblocking everyone and that is highly appreciated :) | 13:54 |
| *** udesale_ has quit IRC | 14:00 | |
| *** derekh has joined #openstack-nova | 14:00 | |
| *** dklyle has joined #openstack-nova | 14:15 | |
| *** amoralej|lunch is now known as amoralej | 14:22 | |
| sean-k-mooney | gibi: is the nova team meeing happing now by the way | 14:33 |
| sean-k-mooney | oh its later i need to update my calandar | 14:33 |
| gibi | sean-k-mooney: each occasion moved to 16:00 UTC | 14:33 |
| sean-k-mooney | yep | 14:33 |
| sean-k-mooney | ical file for anyone else http://eavesdrop.openstack.org/calendars/nova-team-meeting.ics | 14:37 |
| *** jraju__ has quit IRC | 14:42 | |
| *** mriedem has joined #openstack-nova | 14:45 | |
| *** macz_ has joined #openstack-nova | 14:55 | |
| gibi | dansmith: I went through the cyborg patch series. left comments. I did not find any ground breaking problem. | 15:02 |
| dansmith | gibi: okay I haven't even looked at my reviews box yet this morning but will try to do that soon | 15:02 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834 | 15:03 |
| gibi | dansmith: ack | 15:04 |
| gmann | sean-k-mooney: a bug related to flavor. id with name or vice versa are not unique. If we do some change in flavor things what we discussed yesterday, i think we can stop taking flavor id in request and always generate one- https://bugs.launchpad.net/nova/+bug/1858019 | 15:08 |
| openstack | Launchpad bug 1858019 in OpenStack Compute (nova) "The flavor id is not limited when creating a flavor" [Wishlist,Triaged] - Assigned to Choi-Sung-Hoon (knu-cse) | 15:08 |
| gmann | because i do not think fixing 1858019 alone is worth for microversion bump | 15:08 |
| sean-k-mooney | gmann: im not sure about that | 15:08 |
| gmann | any specific reason for flavor id in request ? | 15:08 |
| sean-k-mooney | well people often just use the id | 15:08 |
| sean-k-mooney | i know we frequently do that in ci | 15:09 |
| sean-k-mooney | so we know that the m1.nano flaovr is 42 and that is just used in ci jobs | 15:09 |
| sean-k-mooney | gmann: the flavor id is always uniqute | 15:09 |
| sean-k-mooney | but the name is not | 15:09 |
| gmann | name is also unique to each other but id with name is not | 15:10 |
| sean-k-mooney | yes but you can have two flaovr with the same name | 15:10 |
| sean-k-mooney | you cant have two flavors with the same id | 15:10 |
| sean-k-mooney | right? | 15:10 |
| gmann | no, it will be 409 i think | 15:10 |
| gmann | in name case also | 15:10 |
| sean-k-mooney | i would have to go check | 15:11 |
| gmann | let me confirm quickly | 15:11 |
| sean-k-mooney | i know we look for deleted status | 15:11 |
| sean-k-mooney | gmann: i guess we could stop taking the id in the new microverion i just know i often specify the id if i am using ci jobs or locally so i dont know how wide spread that is | 15:12 |
| sean-k-mooney | gmann: you suggesign blocking it on flavor create right | 15:12 |
| openstackgerrit | Kevin Zhao proposed openstack/nova master: fix scsi disk unit number of the attaching volume when cdrom bus is scsi https://review.opendev.org/712607 | 15:12 |
| sean-k-mooney | not for other operations | 15:12 |
| gmann | yeah, in create | 15:12 |
| gmann | create request only | 15:12 |
| sean-k-mooney | https://github.com/openstack/nova/blob/master/nova/db/sqlalchemy/models.py#L415-L418 | 15:13 |
| sean-k-mooney | im not sure if we still use the instance_types table | 15:13 |
| sean-k-mooney | but the unique constratif for that is just flavor id and deleted | 15:14 |
| sean-k-mooney | oh and name | 15:14 |
| gmann | only thing will be create will generate the uuid. current id in request can be just anything easy to remember like number of string | 15:15 |
| gmann | number or string | 15:15 |
| sean-k-mooney | well its ment to be an int or a uuid | 15:15 |
| sean-k-mooney | but a sting can work | 15:16 |
| sean-k-mooney | *string | 15:16 |
| sean-k-mooney | the idiomatic usage is uuid or int | 15:16 |
| sean-k-mooney | i guess we would make it just a uuid if we auto generated it. i think that is what we do today correct | 15:17 |
| gmann | yeah, if people find uuid difficult to use than int | 15:17 |
| gmann | yes, https://github.com/openstack/nova/blob/master/nova/compute/flavors.py#L69 | 15:17 |
| sean-k-mooney | ya if i set it i alwasy set it to an int | 15:17 |
| sean-k-mooney | if i dont set it i use the name | 15:17 |
| sean-k-mooney | the only reason i have ever used the id is when its shorter then typing the name | 15:18 |
| sean-k-mooney | i guess since you can always use an older microversin if you need to set the id then i would be ok with the change | 15:18 |
| gmann | same. i do not know why bug repartee want to give exiting flavor's name as id for new one | 15:19 |
| gmann | i do not think that is normal usage that is why i marked that bug as wishlist but we could say would not fix or somethings. | 15:19 |
| sean-k-mooney | so the microverion would make the flavor_id auto generated and then either make is_public mutable or remove is_public right? | 15:19 |
| sean-k-mooney | those are the two changes you were suggesting combining? | 15:20 |
| gmann | yeah and anything else there in flavor side cleanup. other way is to do another single cleanup microversion we did in stein or train and combine other API cleanup which are worth to do. | 15:21 |
| *** ratailor has quit IRC | 15:21 | |
| sean-k-mooney | ya i think that makes sense. im not sure if there are other change we were thinking of for the flavor | 15:21 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: virt: Provide block_device_info during rescue https://review.opendev.org/700811 | 15:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Add support for stable device rescue https://review.opendev.org/700812 | 15:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Report COMPUTE_RESCUE_BFV and check during rescue https://review.opendev.org/701429 | 15:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: api: Introduce microverion 2.83 allowing boot from volume rescue https://review.opendev.org/701430 | 15:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Extract _get_bdm_image_metadata into nova.utils https://review.opendev.org/705212 | 15:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Support boot from volume stable device instance rescue https://review.opendev.org/701431 | 15:22 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: DNM - Test stable device rescue tests with BFV instances https://review.opendev.org/710050 | 15:22 |
| sean-k-mooney | we have talked about things like removing rxtx_factor and vcpu_weight in the past | 15:22 |
| sean-k-mooney | i think those are both for vmware but they should have been extraspecs | 15:22 |
| gmann | let me find the etherpad of cleanup tihngs if we have those already | 15:22 |
| *** lpetrut has quit IRC | 15:23 | |
| sean-k-mooney | the disabled field also has some overlap with private/public https://github.com/openstack/nova/commit/f371198b843ba17ad6a6e4bc77a58afb006ab677 | 15:23 |
| gmann | https://etherpad.openstack.org/p/nova-api-cleanup | 15:25 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: Add config option for neutron client retries https://review.opendev.org/712226 | 15:25 |
| gmann | yeah, only admin in access list make it disable | 15:26 |
| sean-k-mooney | ah its an internal field | 15:26 |
| sean-k-mooney | i was looking at the client and did not see how to set it | 15:27 |
| gmann | I will add these two candidates in etherpad. | 15:28 |
| sean-k-mooney | ok. i assume this would all be done in Victoria too right | 15:29 |
| sean-k-mooney | e.g. we cant approve a spec for this at this point and this would not qualify for a bug fix right | 15:29 |
| *** jawad_axd has quit IRC | 15:31 | |
| gmann | yeah, not before Vistoria. or i will say wait till W cycle also to collect more. | 15:32 |
| openstackgerrit | Kevin Zhao proposed openstack/nova master: fix unit test error block info on non x86 architecture https://review.opendev.org/713905 | 15:36 |
| lyarwood | hmm so zuul doesn't kick a failing change out of the gate until all jobs have returned? | 15:36 |
| *** mriedem has quit IRC | 15:37 | |
| gmann | lyarwood: yeah. | 15:38 |
| lyarwood | gmann: kk, this might take some time then :D | 15:38 |
| sean-k-mooney | zuul will kick it out if you propsoe a new version | 15:39 |
| lyarwood | it's not my change | 15:39 |
| gmann | one trick is if you have Depends-On then failure or change on Depends-On | 15:39 |
| lyarwood | https://review.opendev.org/#/c/704759/ is first in the queue but failing | 15:40 |
| lyarwood | I was just checking what zuul was going to do in this case | 15:40 |
| gibi | lyarwood: yeah that is mine, failing on some neutron tempest test case :/ | 15:41 |
| sean-k-mooney | lyarwood: is this related to your evacuate change | 15:41 |
| *** jawad_axd has joined #openstack-nova | 15:42 | |
| *** vishalmanchanda has quit IRC | 15:43 | |
| lyarwood | sean-k-mooney: no, my change is below it | 15:44 |
| lyarwood | actually that doesn't matter does it | 15:44 |
| lyarwood | gibi: was that the subnet creation bug I linked in my email? | 15:44 |
| * lyarwood looks | 15:44 | |
| gibi | lyarwood: there was some timeout but did not look deeper | 15:45 |
| gibi | lyarwood: nvm, I looked at a different job | 15:46 |
| lyarwood | https://4cc1d8431535199f530e-fcfc85786424b7f5a3fcadff9da35f94.ssl.cf2.rackcdn.com/704759/10/gate/grenade-py3/9dc93d2/logs/grenade.sh.txt - looks like grenade failed to update libvirt/qemu | 15:47 |
| gibi | yep | 15:47 |
| *** spatel has joined #openstack-nova | 15:48 | |
| gibi | nova team meeting in 10 minutes on #openstack-meeting-3 | 15:50 |
| *** spatel has quit IRC | 15:51 | |
| stephenfin | Can someone != gibi take a look at this, the final remove-nova-network patch? https://review.opendev.org/#/c/703974/ | 15:52 |
| * gibi will look elswhere :) | 15:52 | |
| stephenfin | I don't want your kind round here :P | 15:52 |
| gibi | nova meeting is running on #openstack-meeting-3 | 16:01 |
| *** ociuhandu has joined #openstack-nova | 16:04 | |
| *** ociuhandu has quit IRC | 16:06 | |
| *** ociuhandu_ has joined #openstack-nova | 16:06 | |
| *** maciejjozefczyk has quit IRC | 16:10 | |
| *** yedongcan has left #openstack-nova | 16:15 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Fix unit test error block info on non x86 architecture https://review.opendev.org/713905 | 16:24 |
| *** tesseract-RH has quit IRC | 16:31 | |
| stephenfin | CI stability patch here looking for another core https://review.opendev.org/#/c/713611/ | 16:46 |
| *** TxGirlGeek has joined #openstack-nova | 16:47 | |
| openstackgerrit | Merged openstack/nova master: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713674 | 16:48 |
| openstackgerrit | Merged openstack/nova master: virt: Pass request context to extend_volume https://review.opendev.org/706899 | 16:48 |
| lyarwood | stephenfin: nice! | 16:50 |
| lyarwood | stephenfin: did you end up fixing that upstream? | 16:50 |
| stephenfin | Yup, thankfully | 16:50 |
| sean-k-mooney | so they accpeted your change related to async stream handeling? | 16:50 |
| stephenfin | My original fix was correct. It just took a while to figure out why | 16:51 |
| lyarwood | awesome | 16:51 |
| stephenfin | It wasn't anything to do with async. Turns out 'read(N)' on anything inheriting from 'RawIOBase' doesn't guarantee N bytes | 16:52 |
| stephenfin | unlike 'BufferedIOBase' | 16:52 |
| stephenfin | lyarwood: is this a real issue? https://zuul.opendev.org/t/openstack/build/9dc93d24e9634d7e95ee6f2763b38613/log/logs/screen-n-cpu.txt#5506 | 16:54 |
| stephenfin | trying to figure out why https://review.opendev.org/#/c/704759/10 failed | 16:54 |
| * lyarwood clicks | 16:57 | |
| stephenfin | actually, nvm, it's not that. It's a simple 404 https://zuul.opendev.org/t/openstack/build/9dc93d24e9634d7e95ee6f2763b38613/log/logs/grenade.sh.txt#46542 | 16:57 |
| lyarwood | stephenfin: no that's fine | 16:57 |
| lyarwood | stephenfin: yeah I spoke about it earlier, apt-get install failing right | 16:58 |
| stephenfin | yuup | 16:58 |
| stephenfin | from http://mirror.ord.rax.opendev.org/ | 16:58 |
| sean-k-mooney | stephenfin: it returns upto N bytes correct | 16:58 |
| stephenfin | sean-k-mooney: correct | 16:59 |
| sean-k-mooney | stephenfin: i think that is the same behavior as the read syscall | 16:59 |
| sean-k-mooney | e.g. if you were to invoke it form c | 16:59 |
| sean-k-mooney | so you have to check how many bytes you actully got | 16:59 |
| stephenfin | also correct | 17:00 |
| sean-k-mooney | im guessign that was causing the header or soemthign to be tuncated | 17:00 |
| sean-k-mooney | or other strange hard to dignose error | 17:00 |
| sean-k-mooney | in anycase im glad you managed to fix it | 17:01 |
| efried_gone | sean-k-mooney: re rootwrap, I think there's some vestigial code that we needed to keep because of a weird os-brick thing. I don't remember the details exactly. But we never figured out whether we could safely remove it, or how. | 17:03 |
| *** adrianc_ has quit IRC | 17:06 | |
| *** adrianc has joined #openstack-nova | 17:06 | |
| sean-k-mooney | efried_gone: did we not remove all usage in nova itself | 17:10 |
| sean-k-mooney | you might be very well right about os-brick | 17:10 |
| sean-k-mooney | but we merged all the patches in the remove root wrap series | 17:11 |
| stephenfin | sean-k-mooney: https://github.com/openstack/nova/tree/master/etc/nova/rootwrap.d/compute.filters | 17:12 |
| efried_gone | yeah | 17:17 |
| *** gyee has joined #openstack-nova | 17:19 | |
| sean-k-mooney | right so that is so it can launch its own privsep deamon right | 17:20 |
| sean-k-mooney | os-vif also launches its own privsep deamon and we dont have a rootwrap entry | 17:21 |
| *** mlavalle has joined #openstack-nova | 17:23 | |
| *** evrardjp has quit IRC | 17:36 | |
| *** evrardjp has joined #openstack-nova | 17:36 | |
| sean-k-mooney | is there a reason that os-brick needs to have rootwap to start privsep? | 17:40 |
| *** lbragstad_ has joined #openstack-nova | 17:41 | |
| *** ociuhandu_ has quit IRC | 17:42 | |
| *** ociuhandu has joined #openstack-nova | 17:42 | |
| sean-k-mooney | ok so it still uses rootwap directly https://github.com/openstack/os-brick/blob/master/os_brick/privileged/rootwrap.py | 17:43 |
| *** lbragstad has quit IRC | 17:44 | |
| sean-k-mooney | stephenfin: so ya i think nova nolonger need to have that rootwap filter. it could be move to os-brick but basically if that is the only usage i satand by my comment that nova does not use rootwap os-brick does | 17:46 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: libvirt: Use virDomainBlockCopy to swap volumes when using -blockdev https://review.opendev.org/696834 | 17:47 |
| sean-k-mooney | it looks like os-brick only has 4 usages of rootwap remining too | 17:47 |
| *** ociuhandu has quit IRC | 17:48 | |
| sean-k-mooney | actully 3 | 17:49 |
| sean-k-mooney | actully no its 7 i didnt scroll im going to check if those need to be elevated or not | 17:50 |
| sean-k-mooney | ya so all of them pass run_as_root=true which means they actully use privsep to run them | 17:51 |
| sean-k-mooney | because we hit here https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f28b7e6ff/os_brick/privileged/rootwrap.py#L169 | 17:52 |
| sean-k-mooney | and execute_root is a privsep function https://github.com/openstack/os-brick/blob/9649f17228203186b523e400080a300f28b7e6ff/os_brick/privileged/rootwrap.py#L190-L194 | 17:52 |
| sean-k-mooney | it looks like oslo rootwrap was removed as a os-bricks depency in liberty too https://github.com/openstack/os-brick/commit/7d341f154397ef38c480bdd9cf6d2ccad6502f13 but https://github.com/openstack/os-brick/commit/dbf77fba1061cb4e93b3db5f8117d6ccc689f702#diff-0d141267b46cdfd7a9dfe6100d79fe33 removed it as an actul depency in newton | 17:59 |
| *** ociuhandu has joined #openstack-nova | 18:01 | |
| sean-k-mooney | the way os-bricks is using privsep is not secure but i dont think its usign the filters at all | 18:01 |
| *** derekh has quit IRC | 18:05 | |
| *** rpittau is now known as rpittau|afk | 18:06 | |
| *** jangutter has quit IRC | 18:20 | |
| *** ociuhandu has quit IRC | 18:27 | |
| *** ociuhandu has joined #openstack-nova | 18:28 | |
| *** ociuhandu has quit IRC | 18:33 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/train: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713961 | 18:42 |
| openstackgerrit | Lee Yarwood proposed openstack/nova stable/stein: nova-live-migration: Ensure subnode is fenced during evacuation testing https://review.opendev.org/713962 | 18:43 |
| *** gyee has quit IRC | 18:44 | |
| *** mriedem has joined #openstack-nova | 18:48 | |
| *** gyee has joined #openstack-nova | 18:51 | |
| *** maciejjozefczyk has joined #openstack-nova | 19:02 | |
| artom | dansmith, quick chat about https://review.opendev.org/#/c/705545/4/nova/tests/functional/integrated_helpers.py@370 ? | 19:03 |
| *** jawad_axd has quit IRC | 19:04 | |
| dansmith | artom: just back to my desk, have to do something first and then yeah | 19:10 |
| artom | dansmith, so, I looked around some more, thought some more, and I think I've come around to your way of thinking | 19:12 |
| artom | I gotta go walk the kiddos | 19:12 |
| dansmith | heh okay | 19:13 |
| *** TxGirlGeek has quit IRC | 19:14 | |
| openstackgerrit | Merged openstack/nova stable/rocky: Fix os-keypairs pagination links https://review.opendev.org/711898 | 19:34 |
| *** maciejjozefczyk has quit IRC | 19:40 | |
| *** maciejjozefczyk has joined #openstack-nova | 19:41 | |
| *** mlavalle has quit IRC | 19:49 | |
| *** bbowen has quit IRC | 19:57 | |
| *** bbowen has joined #openstack-nova | 19:58 | |
| *** mlavalle has joined #openstack-nova | 19:58 | |
| *** gyee has quit IRC | 20:03 | |
| *** maciejjozefczyk has quit IRC | 20:06 | |
| *** ralonsoh has quit IRC | 20:19 | |
| *** bbowen has quit IRC | 20:45 | |
| *** nweinber has quit IRC | 20:46 | |
| artom | dansmith, hrmm, so in the end maybe talking about https://review.opendev.org/#/c/705545/4/nova/tests/functional/integrated_helpers.py@370 could help | 21:06 |
| artom | So, places that redefine _setup_compute_service just don't inherit from the ServicesMixing | 21:07 |
| artom | I think I brought about some confusion when I called it the ServicesHelper mixin | 21:08 |
| artom | It's really the PeriodicHelperMixin | 21:08 |
| artom | Ie - you only get run_periodics in it | 21:08 |
| *** ivve has quit IRC | 21:08 | |
| artom | But because run_periodics depends on self.computes, you get _start_compute as well, which populates self.compute | 21:08 |
| openstackgerrit | Merged openstack/nova stable/train: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713739 | 21:09 |
| openstackgerrit | Merged openstack/nova stable/queens: Avoid circular reference during serialization https://review.opendev.org/713132 | 21:09 |
| openstackgerrit | Merged openstack/nova stable/ocata: Add functional regression test for bug 1849409 https://review.opendev.org/690746 | 21:09 |
| openstack | bug 1849409 in OpenStack Compute (nova) ocata "openstack server list --deleted --limit -1 hangs" [High,In progress] https://launchpad.net/bugs/1849409 - Assigned to Matt Riedemann (mriedem) | 21:09 |
| openstackgerrit | Merged openstack/nova stable/ocata: Fix listing deleted servers with a marker https://review.opendev.org/690747 | 21:09 |
| openstackgerrit | Merged openstack/nova stable/ocata: Fix incompatible version handling in BuildRequest https://review.opendev.org/660456 | 21:09 |
| openstackgerrit | Merged openstack/nova stable/ocata: Hook resource_tracker to remove stale node information https://review.opendev.org/676467 | 21:09 |
| *** tbachman_ has joined #openstack-nova | 21:13 | |
| *** tbachman has quit IRC | 21:13 | |
| *** tbachman_ is now known as tbachman | 21:13 | |
| openstackgerrit | melanie witt proposed openstack/nova stable/stein: Lowercase ironic driver hash ring and ignore case in cache https://review.opendev.org/713982 | 21:14 |
| openstackgerrit | Merged openstack/nova master: Switching new default roles in os-volumes-attachments policies https://review.opendev.org/710190 | 21:17 |
| *** xek has quit IRC | 21:19 | |
| *** mriedem has left #openstack-nova | 21:32 | |
| *** ociuhandu has joined #openstack-nova | 21:33 | |
| melwitt | zzzeek: I've been looking at some debug logs in gate runs and I notice a lot of ROLLBACK logged, would you think it's concerning? https://zuul.opendev.org/t/openstack/build/b4f6a11e4b0940e0a9513f7113bfb95c/log/logs/screen-n-sch.txt#1691 | 21:48 |
| *** PetrTuma has quit IRC | 21:48 | |
| zzzeek | melwitt: when a connection is done being used, it is returned to the pool where a ROLLBACK is emitted. if you have lots of short single-statement transactions then you'd see a lot of these | 21:50 |
| melwitt | zzzeek: oh, I see. thanks | 21:50 |
| zzzeek | melwitt: these can be a problem performance wise | 21:50 |
| zzzeek | but only if yo're observing that | 21:51 |
| melwitt | zzzeek: yeah ... not yet sure what we're seeing, but this is the file where we have the gate bug where we wait 60 seconds without getting rows back from the database. could be a bug on our side. but I'm studying the logs looking for anything that seems like it could cause trouble. what would be a way of avoiding the rollbacks? for the sake of experiment | 21:53 |
| *** gyee has joined #openstack-nova | 22:00 | |
| zzzeek | melwitt: there's a parameter "rollback_on_return" to craete_engine | 22:02 |
| zzzeek | oops, that's not it, one sec | 22:02 |
| melwitt | rollback_reader_sessions ? | 22:03 |
| zzzeek | reset_on_return | 22:03 |
| zzzeek | https://docs.sqlalchemy.org/en/13/core/engines.html?highlight=reset_on_return#sqlalchemy.create_engine.params.pool_reset_on_return | 22:03 |
| melwitt | oh k thanks | 22:03 |
| zzzeek | but...it's dangerous to use this | 22:03 |
| zzzeek | in fact it can't actually work correctly, because there is always a transaction with MySQL InnoDB | 22:04 |
| zzzeek | so you will have very poor effects from using htis | 22:04 |
| *** tbachman has quit IRC | 22:04 | |
| zzzeek | melwitt: your only option is to try the NullPool and turn off pooling entirely | 22:04 |
| zzzeek | melwitt: unfortunately oslo.db has no option for that so if you wanted to test it you'd need to monkeypatch oslo.db | 22:04 |
| *** tbachman has joined #openstack-nova | 22:05 | |
| melwitt | oh. good to know. I wouldn't want to use it then. I was just curious if there was any good/recommended way to address the performance problem with rollbacks if there were one | 22:05 |
| melwitt | it's weird. in the scheduler, there's this constant flow of service select/update/commit happening for the service heartbeats, and when things fail, all of the other selects (for services, compute_nodes) just disappear entirely | 22:08 |
| melwitt | and all I see are the steady select/update/commit for the heartbeats | 22:08 |
| *** ociuhandu has quit IRC | 22:09 | |
| *** ociuhandu has joined #openstack-nova | 22:10 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Fix unit test error block info on non x86 architecture https://review.opendev.org/713905 | 22:10 |
| *** lbragstad_ has quit IRC | 22:13 | |
| melwitt | ohhh dang I actually see now that the debug logging I added in my DNM patch is _not_ showing up, meaning we're not even trying to query for compute_nodes. that's bad. /me keeps digging | 22:15 |
| *** ociuhandu has quit IRC | 22:15 | |
| *** zigo has quit IRC | 22:22 | |
| *** bbowen has joined #openstack-nova | 22:27 | |
| zzzeek | melwitt: sorry I dont know that code very well, keep me in the loop :) | 22:31 |
| zzzeek | (dont know it at all really ... :) ) | 22:31 |
| *** zigo has joined #openstack-nova | 22:31 | |
| melwitt | oh, derp, false alarm I think. I only put the debug statements in the master branch and this is a grenade job so if it failed on the old side, that would be stable/train which doesn't have the debug logging on it. I need to cherry pick this DNM to train again and run with Depends-On | 22:37 |
| *** dpawlik has quit IRC | 22:38 | |
| openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | 22:42 |
| openstack | bug 1844929 in OpenStack Compute (nova) "grenade jobs failing due to "Timed out waiting for response from cell" in scheduler" [High,Confirmed] https://launchpad.net/bugs/1844929 | 22:42 |
| openstackgerrit | melanie witt proposed openstack/nova stable/train: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/713116 | 22:42 |
| openstackgerrit | melanie witt proposed openstack/nova master: DNM: try to get some debug info for bug 1844929 https://review.opendev.org/701478 | 22:43 |
| *** slaweq has quit IRC | 22:45 | |
| *** CeeMac has quit IRC | 22:50 | |
| *** macz_ has quit IRC | 23:01 | |
| *** spatel has joined #openstack-nova | 23:42 | |
| *** spatel has quit IRC | 23:46 | |
| *** CeeMac has joined #openstack-nova | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!