| opendevreview | melanie witt proposed openstack/nova master: WIP Enable unified limits in the nova-next job https://review.opendev.org/c/openstack/nova/+/789963 | 00:26 |
|---|---|---|
| *** akekane_ is now known as abhishekk | 05:23 | |
| frickler | good morning nova, it seems you are running the l-c job in gate as non-voting, which doesn't make sense to me, see e.g. https://review.opendev.org/809955 | 06:27 |
| *** rpittau|afk is now known as rpittau | 06:41 | |
| bauzas | good Friday, Nova | 07:31 |
| bauzas | frickler: tell me, we had issues with this job due to some dep | 07:31 |
| bauzas | frickler: saw the mailing thread from gibi ? can find it if you want | 07:32 |
| bauzas | actually, the problem is on placement, not nova | 07:36 |
| elodilles | well, the problem is there in several projects | 07:39 |
| bauzas | elodilles: I don't disagree | 07:40 |
| elodilles | bauzas: good to hear that :) | 07:40 |
| bauzas | but I'd somehow appreciate that we could only make the job non-voting only when needed | 07:40 |
| bauzas | for nova, ussuri and later provide decorator>=4.0.0 | 07:40 |
| elodilles | and nova has it too in ussuri and older branches | 07:40 |
| bauzas | oh, strange https://github.com/openstack/nova/blob/stable/ussuri/lower-constraints.txt#L21 | 07:41 |
| bauzas | oh shit | 07:41 |
| bauzas | 22 to 24 is 3 releases | 07:41 |
| bauzas | xena being the 24th, ussuri is 4 releases older | 07:42 |
| bauzas | I forgot about victoria :D | 07:42 |
| bauzas | probably the effect of not having the PTG physically located for the first time :) | 07:42 |
| elodilles | anyway, gibi has a solution, which actually solves the situation: https://review.opendev.org/c/openstack/nova/+/810461 | 07:42 |
| elodilles | maybe we should use that one directly and not merge the 'set l-c as non-voting' patch (which meant to be as 'temporary') | 07:43 |
| bauzas | maybe | 07:44 |
| gibi | morning | 08:01 |
| kashyap | lyarwood: Morning; that error about "Could not allocate dynamic translator buffer" -- I recall seeing it in the past (in OpenStack CI) -- it was due to out-of-memory mostly | 08:10 |
| lyarwood | ACK, just getting my haircut now, I'll look again at the memory tracking log in the job when I get back | 08:30 |
| kashyap | Have a good one | 08:40 |
| bauzas | gibi: does that ring a bell to you if I say that when the virt driver returns a list of inventories, we don't remove the existing RPs that are no longer used ? | 09:24 |
| bauzas | context : https://bugs.launchpad.net/nova/+bug/1944031 | 09:24 |
| bauzas | hmmmm, looking at https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L8226 | 09:25 |
| bauzas | we update the provider tree | 09:26 |
| bauzas | but I guess if we have resource providers, we don't verify whether they're still used | 09:26 |
| * bauzas needs to disappear for going to the gym | 09:37 | |
| opendevreview | Pierre Riteau proposed openstack/nova master: Create empty pcpuset for unpinned instances https://review.opendev.org/c/openstack/nova/+/810849 | 09:47 |
| gibi | bauzas: hm, it could be that we never had to delete any RP before VGPU move to its own RP | 10:11 |
| gibi | from nova-compute | 10:11 |
| gibi | so we might missed that case | 10:11 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810763 | 11:06 |
| *** bhagyashris is now known as bhagyashris|rover | 11:09 | |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Walk database migrations in correct order https://review.opendev.org/c/openstack/nova/+/810291 | 11:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: db: Add migration to resolve shadow table discrepancies https://review.opendev.org/c/openstack/nova/+/805738 | 11:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Address some nits with database migration series https://review.opendev.org/c/openstack/nova/+/810856 | 11:14 |
| opendevreview | Stephen Finucane proposed openstack/nova master: tests: Silence noise from database tests https://review.opendev.org/c/openstack/nova/+/810857 | 11:14 |
| opendevreview | Lee Yarwood proposed openstack/nova master: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/790519 | 11:41 |
| gibi | sean-k-mooney: do you remember where we are with mixing numa aware live migration with sriov live migration? Does src_compute(PF-numa0) -> dest_compute(PF-numa1) works? | 12:05 |
| gibi | I do remember that we had issues but I don't if we solved them or just punted them for later | 12:05 |
| gibi | and I only have lab nodes where both PF on numa1 so I cannot test it | 12:06 |
| sean-k-mooney | gibi: yes it should | 12:10 |
| sean-k-mooney | although most of the testing i did was with nic that did not report numa affinity | 12:10 |
| sean-k-mooney | but i did force cross numa migration viat the cpu_dedicated_set and test that with sriov | 12:11 |
| sean-k-mooney | but i would have been using the prefer policy effectivly by relaying on the fact my nics did not report numa affinity | 12:12 |
| sean-k-mooney | gibi: you can try testing it with the prefer policy in your case | 12:12 |
| sean-k-mooney | the ohter way to test it is technially the numa filed in /sys is writable | 12:13 |
| gibi | hm, interesting :D | 12:13 |
| sean-k-mooney | so if you echo 0 into the file and restart libvirt ... | 12:13 |
| sean-k-mooney | i have done that in the past to fake it too with my hardware | 12:13 |
| gibi | OK, I can try that, thanks for the idea | 12:13 |
| opendevreview | Lee Yarwood proposed openstack/nova-specs master: Repropose flavour and image defined ephemeral storage encryption https://review.opendev.org/c/openstack/nova-specs/+/810867 | 12:14 |
| opendevreview | Lee Yarwood proposed openstack/nova-specs master: Repropose Add libvirt support for flavor and image defined ephemeral encryption https://review.opendev.org/c/openstack/nova-specs/+/810868 | 12:14 |
| sean-k-mooney | gibi: since i have you have you seen nova.exception.PortNotUsable: Port 123c94fa-a71a-48c1-9195-ec2a1d6d2f40 not usable for instance 36441284-b272-439d-b8cf-f7c62efc9151. before | 12:15 |
| gibi | hm I saw but I have to dig, I think there are multiple reasons for that | 12:15 |
| sean-k-mooney | i must have been lucky to never hit it till now | 12:16 |
| sean-k-mooney | im currently trying to test vdpa with ovn | 12:16 |
| gibi | one reason is that the port is in a different project than the instance | 12:16 |
| sean-k-mooney | ill dig into it on the neutron side and see whats going on | 12:16 |
| gibi | nova explicitly checks that | 12:16 |
| sean-k-mooney | it is failing in _validate_requested_port_ids so maybe | 12:17 |
| gibi | https://github.com/openstack/nova/blob/c8940f9d60f1b0290ebea94fb6174efac9a1632e/nova/network/neutron.py#L737 | 12:17 |
| gibi | yepp that will be it | 12:17 |
| sean-k-mooney | yep https://github.com/openstack/nova/blob/master/nova/network/neutron.py#L736-L738 | 12:17 |
| sean-k-mooney | ya ok prot is owned by demo and vm is owned by admin | 12:19 |
| sean-k-mooney | ok so horrizon does not work with vdpa ports | 12:19 |
| sean-k-mooney | they dont show up in the port list in the instance create as demo | 12:20 |
| sean-k-mooney | maybe i shoudl just recreated it and test this form the cli | 12:20 |
| sean-k-mooney | oh active :) | 12:22 |
| sean-k-mooney | i was not expecting that :P | 12:22 |
| opendevreview | Pierre Riteau proposed openstack/nova master: Create empty pcpuset for unpinned instances https://review.opendev.org/c/openstack/nova/+/810849 | 12:31 |
| bauzas | gibi: I guess we would have the same concern (deleting old and not used RPs) for vpmems, nope ? | 13:06 |
| gibi | vpmems has its own RP? | 13:08 |
| gibi | or is it tracked on the compute RP? | 13:08 |
| opendevreview | sean mooney proposed openstack/nova master: [DMN] test removal of CAP_DAC_OVERRIDE https://review.opendev.org/c/openstack/nova/+/810906 | 13:21 |
| bauzas | gibi: my bad, those are just inventories from the same RP | 13:22 |
| sean-k-mooney | gibi: i think its on the compute node currently | 13:22 |
| gibi | yepp that what was I assumed | 13:22 |
| bauzas | sean-k-mooney: yeah, https://github.com/openstack/nova/blob/62406b5728077afa9cd38d5c5d510bba64c43bd7/nova/virt/libvirt/driver.py#L8379 | 13:22 |
| sean-k-mooney | with 1 inventory per namespace | 13:22 |
| sean-k-mooney | *namespace size | 13:22 |
| bauzas | then, maybe bandwidth-aware resource providers could be impacted or is that only a VGPU thing ? | 13:23 |
| bauzas | actually, this doesn't come from the virt driver | 13:23 |
| gibi | bauzas: bwm RPs are managed by neutron agents | 13:23 |
| bauzas | yeah | 13:23 |
| sean-k-mooney | what are you currently talking about by the way | 13:23 |
| bauzas | so I guess this is maybe unrelate | 13:23 |
| gibi | to be precies RP and inventory is managed by neutron, allocation managed by nova | 13:23 |
| bauzas | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1944031 | 13:24 |
| sean-k-mooney | for bandwith and pps inventories? | 13:24 |
| gibi | sean-k-mooney: https://bugs.launchpad.net/nova/+bug/1944031 | 13:24 |
| bauzas | shoooooot | 13:24 |
| sean-k-mooney | ah right that | 13:24 |
| sean-k-mooney | well for things on the root rp its simple to fix | 13:24 |
| sean-k-mooney | for nested RPs without an owner trait on the RP its harder | 13:25 |
| sean-k-mooney | e.g. if RP had ownwer_nova and its not in the tree nova just compute then it should be removed | 13:25 |
| sean-k-mooney | without that its very hard to figure out it the RP is owned by nova or by cyborg/neutorn | 13:25 |
| sean-k-mooney | if vgpus were tracked in the db we woudl also have another way to figure this out but since we dont have them in the resouce table or pci_devices table | 13:27 |
| sean-k-mooney | we cant use the nova db to determin if it was an rp we created | 13:27 |
| bauzas | sean-k-mooney: yeah tbh, I think it should be the responsibility of the virt driver to compare the provider tree before and after the update and eventually remove unused RPs | 13:27 |
| sean-k-mooney | bauzas: it should but we dont have the info in placment to be able to do that correctly today | 13:27 |
| bauzas | sean-k-mooney: this could be done in https://github.com/openstack/nova/blob/62406b5728077afa9cd38d5c5d510bba64c43bd7/nova/virt/libvirt/driver.py#L8472 | 13:28 |
| sean-k-mooney | bauzas: not today | 13:28 |
| bauzas | of course | 13:28 |
| bauzas | but we have the actual state in provider_tree | 13:28 |
| sean-k-mooney | bauzas: you can know if the nested RP is own by neutron or cyborg | 13:29 |
| bauzas | that's given | 13:29 |
| sean-k-mooney | so it not safe to remove it | 13:29 |
| bauzas | sean-k-mooney: we namespace the RPs | 13:29 |
| sean-k-mooney | we do not | 13:29 |
| gibi | what we can do is to assume that if the RP had VGPU inventory before then it is owned by nova | 13:29 |
| bauzas | for GPUs ? we totally do | 13:29 |
| gibi | :) | 13:29 |
| gibi | or we use the name of the RP | 13:30 |
| sean-k-mooney | gibi: only if we intend to make it so cyborg cannot use the vgpu inventory type | 13:30 |
| bauzas | the problem is that this naming could be used by *something else* | 13:30 |
| bauzas | but if we consider that https://github.com/openstack/nova/blob/62406b5728077afa9cd38d5c5d510bba64c43bd7/nova/virt/libvirt/driver.py#L8505 is only used by the virt driver, then you can look at all existing RPs matching this | 13:31 |
| sean-k-mooney | any resilance on nameing or RC usage is fragile | 13:31 |
| gibi | yep it is fragile I agree | 13:31 |
| bauzas | agreed tho | 13:31 |
| gibi | RP naming is a bit better than RC I think | 13:31 |
| gibi | so fixing this as a bug I would go with relying on RP naming | 13:32 |
| sean-k-mooney | if we prefixed them all with nova_ then i would strongly agree | 13:32 |
| sean-k-mooney | fight now however we may have collitions although its unlikely | 13:32 |
| sean-k-mooney | *right | 13:32 |
| sean-k-mooney | gibi: as a bug fix using the name template | 13:32 |
| sean-k-mooney | would proably be ok if we implement a proper solution in yoga | 13:32 |
| bauzas | honestly, operators are not intended to change their config everyday | 13:33 |
| sean-k-mooney | e.g. in nova prefix all nova RPs with nova_ or add a mapping table in our db where we store the uuids of all RPs we own | 13:33 |
| gibi | sean-k-mooney: yeah, thinking about it in Yoga PTG would be a good first step | 13:34 |
| bauzas | we could somehow use the audit command to find the orphaned RPs | 13:34 |
| sean-k-mooney | a nova manage command ya would work | 13:34 |
| bauzas | it would have my preference honestly | 13:34 |
| sean-k-mooney | using osc they can also fix it today | 13:34 |
| sean-k-mooney | via osc-placment | 13:34 |
| bauzas | sure, hence the low | 13:34 |
| bauzas | the low status I mean | 13:34 |
| bauzas | this is absolutely not a problem | 13:34 |
| bauzas | this is just, you're creating garbage | 13:35 |
| sean-k-mooney | well it can break schduling | 13:35 |
| sean-k-mooney | so it can be a problem | 13:35 |
| bauzas | give me a sec, verifying the inventories | 13:35 |
| sean-k-mooney | but we do not nessisaly need to automically fix it if we provide a tool to do it when you change the config | 13:35 |
| sean-k-mooney | gibi: just an fyi im on PTO next week just in case your looking for me | 13:36 |
| bauzas | hmmm, you're right | 13:36 |
| bauzas | inventories are left | 13:36 |
| bauzas | [root@hab-19 devstack]# openstack resource provider inventory list 7853e09b-5f5c-4b79-99c6-a550c56eb7e0 | 13:36 |
| bauzas | +----------------+------------------+----------+----------+----------+-----------+-------+------+ | 13:36 |
| bauzas | | resource_class | allocation_ratio | min_unit | max_unit | reserved | step_size | total | used | | 13:36 |
| bauzas | +----------------+------------------+----------+----------+----------+-----------+-------+------+ | 13:36 |
| bauzas | | VGPU | 1.0 | 1 | 1 | 0 | 1 | 1 | 0 | | 13:36 |
| bauzas | +----------------+------------------+----------+----------+----------+-----------+-------+------+ | 13:36 |
| bauzas | dammit | 13:36 |
| gibi | sean-k-mooney: thanks for the heads up, enjoy your time off | 13:36 |
| bauzas | so instances could pick those resources | 13:37 |
| sean-k-mooney | bauzas: ya the inventories would because when you remove the confi the virt driver will just not update the rp or its inventories again | 13:37 |
| bauzas | yup | 13:37 |
| sean-k-mooney | this is also a problem for provider.yaml by the way | 13:37 |
| sean-k-mooney | if we remove something form provider.yaml it will never get removed form placment | 13:37 |
| bauzas | definitely worth a PTG discussion then | 13:38 |
| sean-k-mooney | bauzas: we could perhaps write a file to disk in the nova state dir | 13:38 |
| bauzas | sean-k-mooney: fancy adding a 10th bullet point or want me to write it ? | 13:38 |
| bauzas | I mean, I couldf | 13:38 |
| sean-k-mooney | that way the agent could read it on start up and use that to figure out if something was removed | 13:38 |
| sean-k-mooney | if we did not want to store it in the db | 13:39 |
| sean-k-mooney | bauzas: am i can i guess i have another to add anyway | 13:39 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810909 | 13:39 |
| sean-k-mooney | just assume the stuff i add is an aggreate of stuff we have mentioned in downstream or upstream meetings/discussions | 13:39 |
| gibi | bauzas, sean-k-mooney: I will check what today's neutron does if I remove the bw config. I guess we have the same missing cleanup problem in neutron too. If we have, then I would argue for a solution that is not nova specific or at least easy to apply to neutron too | 13:42 |
| bauzas | gibi: that's why a audit tool seems important | 13:44 |
| bauzas | but the working logic is owned by neutron or the virt driver | 13:44 |
| bauzas | so I don't know how a tool could be prescriptive | 13:45 |
| gibi | bauzas: RP cleanup is OK to be left for the nova-manage but inventory cleanup is needed to be automatic to avoid wrong scheduling | 13:45 |
| bauzas | agreed | 13:45 |
| bauzas | and RP cleanup would be easily manageable in the audit command | 13:46 |
| bauzas | if inventories are left empty | 13:46 |
| gibi | yepp | 13:47 |
| gibi | no inventory is a good indication to a RP to be deleted | 13:47 |
| gibi | there are exceptions though :D | 13:48 |
| gibi | a neturn sriov agent without any PF configured | 13:48 |
| gibi | creates an agent RP and no PF RPs | 13:48 |
| gibi | but the agent RP has no inventory | 13:48 |
| gibi | so the audit tool would delete it | 13:48 |
| gibi | then neutron would re-create it :D | 13:48 |
| gibi | there is no harm, just noise | 13:48 |
| gibi | and having an sriov agent without any PF is a pretty useless config :D | 13:49 |
| gibi | btw if we have a proper logic to null out inventory on an RP then the agent nulling out the last inventory could delete the RP automatically too | 13:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810910 | 13:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810911 | 13:50 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810912 | 13:53 |
| gibi | one less escalation ^^ | 13:53 |
| gibi | I'm pushing this back to victoria without delay as I need this upstream on victoria | 13:54 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810913 | 13:54 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: Reproduce bug 1944759 https://review.opendev.org/c/openstack/nova/+/810914 | 13:56 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: Store old_flavor already on source host during resize https://review.opendev.org/c/openstack/nova/+/810915 | 13:56 |
| sean-k-mooney | i added two more itmes to the ptg adgenda | 14:00 |
| * sean-k-mooney planned to not have any this cycle... | 14:00 | |
| sean-k-mooney | lines 182 for the inventory cleanup https://etherpad.opendev.org/p/nova-yoga-ptg | 14:01 |
| gibi | sean-k-mooney: thanks! | 14:01 |
| sean-k-mooney | can you update that with what you find for neturon | 14:02 |
| gibi | sure | 14:02 |
| gibi | I will do that today | 14:02 |
| sean-k-mooney | no rush we have time before the ptg | 14:03 |
| sean-k-mooney | bauzas: speaking of which i assuem you are going to take the etherpad and turn it into an adgenda at some point | 14:04 |
| gibi | sean-k-mooney: I just closed an escalation I have a lot of motivation :D | 14:04 |
| sean-k-mooney | thats alwasy a nice feeling espically on a friday | 14:05 |
| sean-k-mooney | gibi: by the way i have only been skimming what happing with setuptool. are we any closer to a decision on what to do for the stable branches | 14:05 |
| bauzas | sean-k-mooney: yup will work on it | 14:07 |
| bauzas | (the agenda) | 14:07 |
| * bauzas needs to dogwalk for getting her kid from school | 14:07 | |
| gibi | sean-k-mooney: I have a solution based on you idea of [tox]requires config that proved to be working. It seems others are more for dropping lower constraints testing althogheter which is fine by me. | 14:07 |
| gibi | sean-k-mooney: however a recent mail might means that we have the same issue outside of lower constraint testing http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025040.html | 14:08 |
| sean-k-mooney | ok and are we going to look into the project.yaml or what ever the new thing is for yoga on? | 14:08 |
| gibi | pyproject.yaml is the thing but honestly I has not time to look into that as a direction | 14:08 |
| gibi | I assume it also needs some common agreement in OpenStack | 14:09 |
| sean-k-mooney | ya we proably should not be using any deps that rely on use_2to3 realistically in ussuri | 14:09 |
| sean-k-mooney | gibi: yep we would | 14:09 |
| sean-k-mooney | it would fundemtally change how we do dep managment | 14:09 |
| sean-k-mooney | its a replacment for requirements.txt and tox.ini all wrapped into one file | 14:09 |
| sean-k-mooney | i think it can still use the other but its basically one file to rule them all | 14:10 |
| gibi | OK then it is definitely bigger that I can chew right now | 14:12 |
| sean-k-mooney | yep it is | 14:13 |
| gibi | and it is Friday :) | 14:15 |
| priteau | Hello. Who has rights to update the Nova bug template on LP? I noticed an issue with the rpm command. | 14:17 |
| priteau | See https://wiki.openstack.org/wiki/Nova/BugsTeam/BugReportTemplate | 14:17 |
| priteau | `rpm -ql | grep <projectname>` should be `rpm -qa | grep <projectname>` | 14:17 |
| gibi | priteau: I think anybody can edit the wiki I'm not sure about LP | 14:18 |
| priteau | I can make the wiki edit of course, but what matters is syncing with LP ;-) | 14:19 |
| gibi | priteau: I found I can update the LP | 14:19 |
| gibi | let me know what needs to be fixed and I will do it in LP | 14:20 |
| gibi | ahh I see what is needed | 14:20 |
| priteau | I've updated the wiki page | 14:20 |
| gibi | hm, do we need both -a and -l ? | 14:21 |
| gibi | is -l a shortcut for --last ? | 14:23 |
| gibi | (I'm on debian :D)_ | 14:23 |
| sean-k-mooney | we proably should jsut remove it from the template | 14:23 |
| sean-k-mooney | i think for deb packages -l is for list | 14:23 |
| sean-k-mooney | i can check | 14:23 |
| priteau | Just -qa | 14:24 |
| priteau | -ql is for listing files inside a package | 14:24 |
| priteau | -l, --list | 14:24 |
| priteau | List files in package. | 14:24 |
| gibi | ohh OK then -a it is | 14:24 |
| gibi | fixed the LP | 14:24 |
| gibi | priteau: thanks for reporting it | 14:24 |
| sean-k-mooney | yes -qa | 14:24 |
| sean-k-mooney | but really we dont care about the package version most of the time we just care about the openstack version | 14:25 |
| priteau | This is an easy way to find the version when using binary packages | 14:25 |
| sean-k-mooney | we care for libvirt and qemu sometimes but we porably could make it more generic | 14:26 |
| sean-k-mooney | priteau: right but upstream that normally not useful | 14:26 |
| sean-k-mooney | since we cant easially map it to the source code | 14:26 |
| sean-k-mooney | even downstream the pacakge version is not very useful since that mapping is hard to do | 14:27 |
| sean-k-mooney | we often have to pull the srouce rpm and check if a patch is in it which is a pain | 14:27 |
| sean-k-mooney | priteau: by the way the template also tells you to run udo sosreport -o openstack_nova --batch | 14:29 |
| sean-k-mooney | which i dont think i have ever seen peopl actully do | 14:29 |
| priteau | Who follows instructions? :) | 14:29 |
| priteau | Would you like people to use `pip3 list | grep nova` as an alternative? | 14:29 |
| sean-k-mooney | not nessisarly but its helpful if they clearly state that they used train or the serise name | 14:30 |
| opendevreview | Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uwsgi https://review.opendev.org/c/openstack/nova/+/810922 | 14:30 |
| sean-k-mooney | knowing the disto and or package version is nice too | 14:30 |
| priteau | I imagine knowing the release tag can be quite useful | 14:32 |
| opendevreview | Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uwsgi https://review.opendev.org/c/openstack/nova/+/810922 | 14:32 |
| gibi | just yesterday I troubleshooted a deployment with nova_compute version 22.2.3 :D | 14:32 |
| lpetrut | hi, I'm hitting some nova api deadlocks and noticed that oslo.reports isn't enabled when using uwsgi so I've submitted a small commit: https://review.opendev.org/c/openstack/nova/+/810922 | 14:33 |
| gibi | (note that we only released 22.2.2 upstream) | 14:33 |
| sean-k-mooney | gibi: ya i was going to say was this in the gate :) | 14:33 |
| sean-k-mooney | cause otherwise they are going to have fun when we do the next stable release | 14:34 |
| gibi | sean-k-mooney: it was downstream. I think what they did is they took what was unreleased from stable/victoria and created 22.2.3 out of it downstream | 14:34 |
| sean-k-mooney | i see | 14:35 |
| gibi | which is problematic as you said | 14:35 |
| gibi | lpetrut: seems useful | 14:35 |
| gibi | lpetrut: thanks | 14:35 |
| sean-k-mooney | huh | 14:36 |
| sean-k-mooney | maybe that is why the GMR were not working instead of what we tought with the signal being intercpted by uwsgi/mod_wsgi | 14:37 |
| gibi | sean-k-mooney: or we need both :) | 14:37 |
| sean-k-mooney | lpetrut: i assume you tested this and it logs the GMR to the log properly on kill -usr2 | 14:37 |
| sean-k-mooney | gibi: ya i was wonderign if that would only work if you set the signal to the python interpreter instnace | 14:38 |
| gibi | unfortunately this is pretty complicated to test upstream. | 14:39 |
| lpetrut | sean-k-mooney: the signal still gets intercepted but I'm using a file listener | 14:39 |
| gibi | I mean automatically testing it | 14:39 |
| sean-k-mooney | lpetrut: intercepted by uwsgi and not passed to nova right | 14:39 |
| sean-k-mooney | lpetrut: oh are you poking a file to trigger it | 14:40 |
| sean-k-mooney | instead of sig_usr2 or soemthing | 14:40 |
| lpetrut | yep, I'm setting something like oslo_reports.file_event_handler = /opt/stack/logs/trigger | 14:40 |
| sean-k-mooney | gibi: we could proably add a func test but we would have to expand the test scope | 14:40 |
| sean-k-mooney | lpetrut: ok i dont think we technially support that in nova | 14:40 |
| sean-k-mooney | but it certenly works around the issue | 14:41 |
| lpetrut | it already works with most nova services, they key is to pass the config when calling the gmr hook | 14:41 |
| sean-k-mooney | so this is really a mini feature rather then a bug | 14:41 |
| gibi | sean-k-mooney: do we run nova-api in uwsgi in func test? | 14:41 |
| sean-k-mooney | gibi: no but we coudl do somehting like neutorn fullstack tests | 14:41 |
| gibi | sean-k-mooney: ack, that is a possibility | 14:42 |
| sean-k-mooney | it would be a different type of test | 14:42 |
| gibi | sean-k-mooney: or add this to nova-next post test hook | 14:42 |
| sean-k-mooney | gibi: ya that too | 14:42 |
| gibi | bauzas, sean-k-mooney: btw I confirm that neutron also leaks inventories if the bw or pps config is removed | 14:43 |
| sean-k-mooney | lpetrut: did you want to backport udo sosreport -o openstack_nova --batch | 14:43 |
| sean-k-mooney | * https://review.opendev.org/c/openstack/nova/+/810922/2/nova/api/openstack/wsgi_app.py | 14:43 |
| sean-k-mooney | to me this is really a specless blueprint | 14:43 |
| sean-k-mooney | so not something we woudl backport | 14:43 |
| sean-k-mooney | i think its a nice change to merge so no real objection to the patch | 14:44 |
| sean-k-mooney | just not sure this is a bug and a spec is way to heavy weight so not sure how to track this | 14:45 |
| sean-k-mooney | to me its really just a trivial fix but it proably should have a release note | 14:45 |
| lpetrut | yeah, it's hard to label it as a bug in order to allow backports but that's ok. a release note makes sense, I can also mention the fact that uwsgi may intercept SIGUSR2, in which case a file trigger may be configured | 14:46 |
| sean-k-mooney | lpetrut: ya if you add a release note and maybe add a doc for the intercept i would be +1 on it | 14:47 |
| lpetrut | awesome, thanks. is there a specific doc that you have in mind? | 14:48 |
| sean-k-mooney | we have a doc for GMR i think in the contibutor section | 14:48 |
| sean-k-mooney | https://github.com/openstack/nova/blob/50fdbc752a9ca9c31488140ef2997ed59d861a41/doc/source/reference/gmr.rst | 14:49 |
| sean-k-mooney | ah its in refernce | 14:49 |
| sean-k-mooney | can you update https://github.com/openstack/nova/blob/master/doc/source/reference/gmr.rst#generating-a-gmr | 14:49 |
| sean-k-mooney | with the exmaple of the file trigger | 14:50 |
| lpetrut | definitely, thanks for the link | 14:50 |
| lpetrut | the "TextGuruMeditation.setup_autorun(version)" hook sample should also be updated. if we don't pass the config, gmr will not be aware of the [oslo_reports] config opts | 14:51 |
| sean-k-mooney | ya i dont think we have really updated it since it was added | 14:56 |
| sean-k-mooney | i would suggest updating the existig singal based exmaple to use nova-compute and then adding the file example for nova-api and makeign any other changes that you think are needed | 14:57 |
| lpetrut | sounds good | 14:59 |
| *** yonglihe_ is now known as yonglihe | 15:08 | |
| bauzas | gibi: ack, so we need to discuss this during the PTG | 15:15 |
| gibi | yepp | 15:15 |
| gibi | added notes to the bad | 15:15 |
| gibi | pad | 15:16 |
| lpetrut | one minor nit: gmr.setup_autorun takes a "service_name" parameter which is used when constructing the report filename. when missing, it's trying to retrieve it from the stack trace but it seems to always end up with "thread.py", so the reports are named something like "thread.py_gurumeditation_20210924141722". since none of the other service pass this parameter, I'm thinking about doing the same for nova-uwsgi for consistency reasons. | 15:16 |
| sean-k-mooney | lpetrut: or you could fix them all | 15:19 |
| sean-k-mooney | lpetrut: you should be able to use the service binary name | 15:20 |
| sean-k-mooney | so service_obj.binary | 15:20 |
| lpetrut | that works as well | 15:20 |
| sean-k-mooney | that way if you have multiple service on the same host using the same file tirrger they wont overright | 15:21 |
| sean-k-mooney | although the timestamp is unlikely to collide in anycase | 15:21 |
| lpetrut | right, it's also more user friendly since it's easier to tell which is the originating service | 15:24 |
| sean-k-mooney | well when not using the file backend it dumps to the service log so that not been an issue before but for dumping the GMR to a file its something we shoudl definetly address | 15:25 |
| gibi | sean-k-mooney: btw, I tried your echo 0 > numa_node trick and it works like a charm. I can now confirm that live migration with SRIOV + NUMA works and the numa topology is properly recalculated | 15:41 |
| sean-k-mooney | thats an old trick i have been using for ever | 15:44 |
| gibi | this knowledge is gold | 15:45 |
| sean-k-mooney | if you ignore the horrible hack that it is you can actully write udev rules to allwo you to assocaite the device with other numa nodes on the same socket if you enabel cluster on die or amds numa_per_socket>1 | 15:48 |
| opendevreview | Lucian Petrut proposed openstack/nova master: api: enable oslo.reports when using uwsgi https://review.opendev.org/c/openstack/nova/+/810922 | 15:48 |
| sean-k-mooney | ok going to finish there o/ talk to ye in a week | 15:55 |
| fungi | elodilles: has there been any progress on discussions of whether to discontinue lower-constraints jobs on nova's stable/ussuri branch? at this point, nothing (including an outstanding security fix) can merge there, so it's probably time to start talking about early eol instead | 16:36 |
| fungi | at least if we're up front with users that we're no longer fixing known vulnerabilities on that branch, the vmt can go forward with announcing fixes on the nova branches which are still receiving patches | 16:37 |
| *** tbachman is now known as Guest882 | 17:02 | |
| opendevreview | Rodolfo Alonso proposed openstack/nova master: Set "cache_ok=True" in "TypeDecorator" inheriting classes https://review.opendev.org/c/openstack/nova/+/807359 | 17:08 |
| *** tbachman is now known as Guest883 | 17:17 | |
| opendevreview | Artom Lifshitz proposed openstack/nova master: Gracefully power off guest on instance delete https://review.opendev.org/c/openstack/nova/+/808474 | 17:58 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: "Regression" test for server delete https://review.opendev.org/c/openstack/nova/+/810951 | 17:58 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: WIP: Make os_shutdown_timer a proper image property https://review.opendev.org/c/openstack/nova/+/810952 | 17:58 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: "Regression" test for server delete https://review.opendev.org/c/openstack/nova/+/810951 | 18:02 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: Gracefully power off guest on instance delete https://review.opendev.org/c/openstack/nova/+/808474 | 18:02 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: WIP: Make os_shutdown_timer a proper image property https://review.opendev.org/c/openstack/nova/+/810952 | 18:02 |
| elodilles | fungi: there is a patch that sets the lower-constraints job non-voting, but unfortunately it needed an impressive amount of rechecks and it is still not merged so far ( https://review.opendev.org/c/openstack/nova/+/809955 ) | 18:57 |
| elodilles | also there is another patch from gibi that pins the setuptools ( https://review.opendev.org/c/openstack/nova/+/810461 ) | 18:59 |
| fungi | note that the lower-constraints job was already not passing on stable/ussuri when setuptools updated | 19:04 |
| elodilles | how do you mean? | 19:12 |
| elodilles | what I remember is that the gate was not blocked until the gate started to use setuptools 58.0.4. | 19:45 |
| elodilles | even lower-constraints jobs were passing | 19:45 |
| elodilles | and the passing setuptools pinning patch also proves that there is no other issue (for now) than the one that was introduced by the latest setuptools release | 19:48 |
| fungi | elodilles: oh, you're right, we got setuptools 58 in tox jobs when virtualenv 20.8.0 was released on 2021-09-16 at 12:52 utc and change 806628 was unlucky enough to be approved a few hours later at 16:23 utc that same day | 20:06 |
| fungi | change 810461 could probably instead do requires = virtualenv<20.8 if you wanted to be more flexible | 20:10 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!