| *** Uggla|afk is now known as Uggla | 08:45 | |
| *** ministry is now known as __ministry | 09:48 | |
| opendevreview | Frode Nordahl proposed openstack/nova master: Ensure binding profile keys added by Nova are cleared https://review.opendev.org/c/openstack/nova/+/826601 | 10:10 |
|---|---|---|
| gibi | bauzas: can we land this? https://review.opendev.org/q/topic:bug%252F1952941 I'm getting pressure from downstream... | 10:41 |
| *** bhagyashris_ is now known as bhagyashris|ruck | 11:05 | |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Include pf mac and vf num in port updates https://review.opendev.org/c/openstack/nova/+/824833 | 11:14 |
| guesswhat | is xen ( xcp-ng ) supported in openstack as nova driver? seems its deprecated, right ? | 11:31 |
| dmitriis | sean-k-mooney: o/ question about https://review.opendev.org/c/openstack/nova/+/824834/1/nova/pci/stats.py#530. Looking at pci_passthrough_whitelist alone without runtime checks we can't tell if a device is a PF or not. There may be a case where a device is present in the config but not yet hot-plugged into the host or deployment tooling may apply | 11:44 |
| dmitriis | the same config for various PCI devices which may or may not be present in all servers. So I am trying to decide whether implementing runtime checks during pci_passthrough_whitelist parsing is a good idea or not. | 11:44 |
| sean-k-mooney | dmitriis: its not that tere wont be any runtime checks its that when we first inistalise the pci tracker and iterate over the pci devcied on the host we can check once and either raise an error that prevent the agent form starting or strip out the pf depending on the configurtion | 12:51 |
| sean-k-mooney | dmitriis: you do not use the same config on all servers | 12:53 |
| sean-k-mooney | the pci whitelist is differnt per host | 12:53 |
| sean-k-mooney | i dont enforce this today but its logcially an error to list a device in the pci whitelist if its not present on the host. | 12:54 |
| sean-k-mooney | the only reason we dont enforce it is it could alredy be pass through to a vm | 12:55 |
| dmitriis | sean-k-mooney: What about the hot-plug case? I know it's a niche one and I haven't actually seen this to be used | 12:55 |
| sean-k-mooney | dmitriis: we do not support hotplug of pci device on the host | 12:55 |
| sean-k-mooney | libvirt caches the pci device list and you woudl have ot restart it for libvirt to work and then restart the nova agent | 12:56 |
| dmitriis | sean-k-mooney: ah, ok, so then it's not a problem and I can make a runtime check when whitelist is parsed | 12:56 |
| sean-k-mooney | so for the lifetime of the nova agent hot-plug on the host is not supported | 12:56 |
| sean-k-mooney | yes | 12:56 |
| sean-k-mooney | we can ensure the PFs never actully make it to the db in the first plance | 12:56 |
| sean-k-mooney | and therefore can never be a candiate for selection | 12:57 |
| dmitriis | sean-k-mooney: right, presumably the compute node will filter out PCI devices obtained from libvirt based on the whitelist | 12:57 |
| dmitriis | sean-k-mooney: what about the "if PF is specified in a request with a remote_managed tag"? Do I need to support this? | 12:58 |
| sean-k-mooney | well there wont be any pf with remote managed in the db because we filtered them at start up so it will be handeled automaticaly | 12:59 |
| sean-k-mooney | vnic-type direct-phsyical will only match PFs without remote-managed=true | 12:59 |
| sean-k-mooney | because that is all tha will exist | 13:00 |
| sean-k-mooney | and vnic-type smartnic will only match VFs | 13:00 |
| sean-k-mooney | so i think that is covered | 13:00 |
| sean-k-mooney | you do not need to specificlyu request remote_mannaged=false in the driect-physical case | 13:00 |
| dmitriis | sean-k-mooney: ok, so I can just drop the check here https://review.opendev.org/c/openstack/nova/+/824834/1/nova/pci/stats.py#505 | 13:00 |
| dmitriis | basically we assume that those devices won't get into the DB so no additional filtering will be needed | 13:01 |
| sean-k-mooney | yes that is more or less why im suggesting. you jst need to move the logic to the inital startup | 13:01 |
| sean-k-mooney | so its done once per host rather then once per request | 13:02 |
| sean-k-mooney | * once per host start up | 13:02 |
| dmitriis | sean-k-mooney: ok, makes sense, I'll have a look at how/where to put those and resubmit | 13:02 |
| dmitriis | sean-k-mooney: regarding os-traits, looks like the 2.7.0 package got built and pushed out but 2.6.0 is still referenced in upper-constraits https://github.com/openstack/requirements/blob/master/upper-constraints.txt#L382 | 13:04 |
| dmitriis | I've seen a change to the release repo and the requirements repo | 13:05 |
| dmitriis | ah, looks like this hasn't been merged yet https://review.opendev.org/c/openstack/requirements/+/826447 while it's approved | 13:06 |
| sean-k-mooney | right i had to make a change to placment and rebase that to depend on it | 13:06 |
| sean-k-mooney | https://review.opendev.org/c/openstack/placement/+/826486 | 13:06 |
| sean-k-mooney | still need to be approved. | 13:06 |
| sean-k-mooney | bauzas: can you quickly hit ^ | 13:07 |
| dmitriis | sean-k-mooney: ack, I see | 13:07 |
| sean-k-mooney | dmitriis: we have some packaging issue with os-traits and placment that dont really work with how our ci is | 13:08 |
| sean-k-mooney | we may need to merge os-traits into placment to fix it | 13:08 |
| sean-k-mooney | so i had to hack around it for now | 13:08 |
| sean-k-mooney | until recently we did not have ci in the requirements repo that tested placement | 13:08 |
| sean-k-mooney | so this is the first tiem we are seeign the cirular depency problem | 13:09 |
| sean-k-mooney | dmitriis: by the way you and chateaulav both need the os-trait bump | 13:10 |
| sean-k-mooney | the best way to handel that is to have a singel patch to nova that increases our lower-constirati and requiremets.txt to 2.7.0 and then have both of ye rebase your series ontop of that patch | 13:11 |
| sean-k-mooney | i can prably go file that but if you or chateaulav want to file that patch to nova feel free too | 13:11 |
| dmitriis | sean-k-mooney: ack, I'll propose a patch and make my change depend on it | 13:12 |
| sean-k-mooney | cool can you ping chateaulav when you do or add them as a reviewer so they see it | 13:12 |
| dmitriis | sean-k-mooney: will do. | 13:13 |
| dmitriis | Just trying to figure out why my local unit test runs are failing with | 13:13 |
| dmitriis | File "/home/<user>/src/opendev/nova/.tox/py39/lib/python3.9/site-packages/oslo_concurrency/lockutils.py", line 183, in _get_lock_path | 13:13 |
| dmitriis | raise cfg.RequiredOptError('lock_path') | 13:13 |
| dmitriis | oslo_config.cfg.RequiredOptError: value required for option lock_path in group [DEFAULT] | 13:13 |
| sean-k-mooney | your not adding any locks are you | 13:15 |
| sean-k-mooney | that seam unrelated | 13:15 |
| sean-k-mooney | try reruning with tox -r | 13:16 |
| sean-k-mooney | to regenerate teh tox venv | 13:16 |
| sean-k-mooney | incase you have some out of date pacakges | 13:16 |
| sean-k-mooney | or just rm the tox enve manually without -r and let it regenerate it | 13:16 |
| dmitriis | sean-k-mooney: no, and the test cases aren't related to what I've changed. Tried nuking .tox before. Let's see if -r helps. | 13:18 |
| sean-k-mooney | if you nuked . tox it wont | 13:18 |
| sean-k-mooney | i can try running it locally what patch have you checked out | 13:19 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Include pf mac and vf num in port updates https://review.opendev.org/c/openstack/nova/+/824833 | 13:20 |
| dmitriis | sean-k-mooney: ^ https://review.opendev.org/c/openstack/nova/+/824833 the latest here | 13:20 |
| sean-k-mooney | funcitonal or unit | 13:21 |
| dmitriis | sean-k-mooney: unit: `tox -e py39` | 13:21 |
| sean-k-mooney | ack just running them now under py38 | 13:22 |
| sean-k-mooney | if they pass ill install py39 and test again | 13:22 |
| dmitriis | sean-k-mooney: ack, I'll spin up a container and do a clean check as well. | 13:23 |
| sean-k-mooney | we do ocationally get unit test breakages when a new lib is released | 13:23 |
| sean-k-mooney | although the last release was Oct 21, 2021 | 13:24 |
| sean-k-mooney | so that is likely not the issue here | 13:24 |
| dmitriis | ack | 13:26 |
| sean-k-mooney | there was an oslo.limits release and olslo utils release in the last few days but also does not look like the woudl be at fault | 13:28 |
| sean-k-mooney | so far i think things are pass ing but im gong to grab a drink | 13:28 |
| sean-k-mooney | brb | 13:28 |
| dmitriis | sean-k-mooney: thanks for checking, will try to figure out what's on my env that's causing it | 13:28 |
| *** dasm|off is now known as dasm | 13:34 | |
| sean-k-mooney | got the same error here | 13:38 |
| bauzas | sean-k-mooney: gibi: sorry folks, was on discussion with someone you know | 13:38 |
| sean-k-mooney | https://paste.opendev.org/show/812399/ | 13:38 |
| bauzas | looking at both your changes | 13:38 |
| sean-k-mooney | dmitriis: so its in the block device tests so not related to your patch | 13:39 |
| gibi | bauzas: no worries | 13:39 |
| sean-k-mooney | this is failing here https://github.com/openstack/oslo.concurrency/blob/master/oslo_concurrency/lockutils.py#L172-L185 | 13:40 |
| dmitriis | sean-k-mooney: yeah, just confusing. Got 1 failure only with py39 in a clean 21.10 container https://paste.opendev.org/show/812400/ | 13:42 |
| dmitriis | hopefully it will just work in CI :^) | 13:42 |
| sean-k-mooney | ah there was a fasterners release 2 days ago | 13:43 |
| sean-k-mooney | https://pypi.org/project/fasteners/ | 13:43 |
| sean-k-mooney | which we picked up | 13:44 |
| sean-k-mooney | https://github.com/openstack/requirements/commit/c086fff066e40f3a87ff81fe72c8c16cdefe7c64 | 13:44 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Introduce remote_managed tag for PCI devices https://review.opendev.org/c/openstack/nova/+/824834 | 13:54 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835 | 13:54 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111 | 13:54 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Bump os-traits to 2.7.0 https://review.opendev.org/c/openstack/nova/+/826675 | 13:54 |
| dmitriis | sean-k-mooney: looking | 13:56 |
| dmitriis | sean-k-mooney, chateaulav: os-traits version bump change: https://review.opendev.org/c/openstack/nova/+/826675 | 13:57 |
| chateaulav | thanks | 13:57 |
| sean-k-mooney | gibi: bauzas we might have an impending gate failure https://paste.opendev.org/show/812401/ | 14:07 |
| sean-k-mooney | so looks like even with latest master we now have lock issues | 14:07 |
| sean-k-mooney | looks like os-brick was depending on poslition for calling kw args with is always dangrous | 14:11 |
| sean-k-mooney | https://github.com/openstack/os-brick/blob/master/os_brick/initiator/utils.py#L45-L46 | 14:11 |
| gibi | sean-k-mooney: I cannot reproduce the lockpath failure locally with fresh env on nova master | 14:12 |
| sean-k-mooney | odd i could after checking out dmitriis code and then still could after checking out master | 14:13 |
| sean-k-mooney | ill try cleaning my env and pyc files again | 14:13 |
| sean-k-mooney | i can repoduce ti consitently on my laptop im trying it on my server now too | 14:16 |
| dmitriis | sean-k-mooney: oddly, the latest run on CI https://zuul.opendev.org/t/openstack/build/daf00a97b54f4e349ef609240a956347/log/job-output.txt doesn't have it | 14:17 |
| sean-k-mooney | gibi: odd it passed there | 14:20 |
| bauzas | gibi: sean-k-mooney: do we have logstash yet ? | 14:20 |
| bauzas | to see whether it's a large race condition | 14:20 |
| gibi | bauzas: I used logstash today | 14:20 |
| sean-k-mooney | im going to see if it only happens in a repo after usign dmitriis patch | 14:21 |
| sean-k-mooney | i wonder if we are leaking locks or somehting | 14:21 |
| sean-k-mooney | dmitriis: you patch is still pending in the gate right | 14:22 |
| dmitriis | sean-k-mooney: yes but I can see here https://zuul.opendev.org/t/openstack/status#nova that it already finished running unit tests | 14:23 |
| sean-k-mooney | odd that it passed | 14:24 |
| sean-k-mooney | and we can both repoduce the issue locally | 14:24 |
| sean-k-mooney | dmitriis: what operating system are you using by the way | 14:25 |
| dmitriis | sean-k-mooney: yes, I'm trying to compare how many test cases fail each time and whether it depends on timing | 14:25 |
| opendevreview | Merged openstack/placement master: disable traits count check to allow os-traits 2.7.0 https://review.opendev.org/c/openstack/placement/+/826486 | 14:25 |
| dmitriis | sean-k-mooney: Ubuntu 21.10 | 14:25 |
| sean-k-mooney | for me it was consitent | 14:26 |
| sean-k-mooney | well mostly | 14:26 |
| sean-k-mooney | i coudl re run with --failing | 14:26 |
| dmitriis | sean-k-mooney: in a 21.10 container I consistently get 1 failure | 14:26 |
| sean-k-mooney | that went form 9 to 8 then staied with 8 | 14:26 |
| sean-k-mooney | hum clean repo one failure but its differetn | 14:28 |
| sean-k-mooney | RuntimeError: Reader <_MainThread(MainThread, started 140118297270080)> to writer privilege escalation not allowed | 14:28 |
| dmitriis | sean-k-mooney: yes, that's the one I am getting in a clean container with the patch applied | 14:28 |
| dmitriis | RuntimeError: Reader <_MainThread(MainThread, started 139926055308288)> to writer privilege escalation not allowed | 14:29 |
| sean-k-mooney | ya i have your patch aplied too | 14:29 |
| sean-k-mooney | gibi: bauzas i guess we are not seeing this in the gate | 14:30 |
| sean-k-mooney | but its just odd behavior | 14:30 |
| sean-k-mooney | the issue seam to be related to the fasteners release i think | 14:31 |
| sean-k-mooney | im going to downgrade it to confirm | 14:31 |
| dmitriis | sean-k-mooney: ack | 14:31 |
| dmitriis | sean-k-mooney: tried with the previous (VPD) patch only https://paste.opendev.org/show/812404/ | 14:32 |
| sean-k-mooney | well a uuid is not an rfc1738 url | 14:33 |
| sean-k-mooney | hum same issue with older fasteners | 14:34 |
| dmitriis | sean-k-mooney: the other failures are odd to see because that same patch passed on CI https://review.opendev.org/c/openstack/nova/+/808199. Checking with older fasteners as well. | 14:36 |
| sean-k-mooney | dmitriis: master seam to work ok for me | 14:37 |
| sean-k-mooney | dmitriis: have you treid rebaseing all your patches | 14:37 |
| sean-k-mooney | we might have fix this already on master and you patches might be missing it | 14:37 |
| dmitriis | sean-k-mooney: ack, let me try that | 14:37 |
| *** artom__ is now known as artom | 14:50 | |
| bauzas | sean-k-mooney: cool, interesting | 14:53 |
| bauzas | thanks for the follow-up | 14:53 |
| sean-k-mooney | bauzas: gibi by the way we likely need to have a dicussion about what to do about placment,os-traits and os-resouce-providres at some point | 14:54 |
| gibi | sean-k-mooney: ack. I read the scrollback from the release channel | 14:54 |
| sean-k-mooney | perhaps a ptg topic. we coudl start it on the ML or in an etherpad | 14:55 |
| gibi | I do believe that os-traits being a separate lib helps for the other consumers like nova and neutron. So I would not want to merge that back to placement | 14:55 |
| gibi | So I'm on the side to relax the testing and add a release step where we check that the placement is released with the latest os-traits | 14:56 |
| sean-k-mooney | ya long term that is proably the most operator freindly too | 14:57 |
| sean-k-mooney | it would be a lot of work to reabsorb os-trait into placment | 14:57 |
| sean-k-mooney | since we woudl have to modify nova,neutron,cyborg and possibel others + the deployment tooling | 14:58 |
| sean-k-mooney | gibi: i can look at relaxing the test and push a patch for people to consider | 15:01 |
| gibi | sean-k-mooney: OK, thanks | 15:01 |
| sean-k-mooney | i dont know if cdent or others are aroudn that shoudl review | 15:05 |
| sean-k-mooney | are any of the placment core team actully still active | 15:05 |
| dmitriis | sean-k-mooney: hmm, tried master locally, getting the same failures with locking. Tried master on a server machine (21.04) - no failures. | 15:05 |
| gibi | sean-k-mooney: I think we only have those that are active in nova too | 15:06 |
| dmitriis | will try a few more combinations | 15:06 |
| sean-k-mooney | perhaps tetsuro? although i have not seen them in a while | 15:06 |
| sean-k-mooney | ill add the patch to open discuss for next weeks meeting once i figure out what to do | 15:07 |
| gibi | ack | 15:07 |
| ade_lee__ | sean-k-mooney, hey -- could you take a look at https://zuul.opendev.org/t/openstack/build/554ffdeb6c3a4cdc8c53c55dddd6c36e please and let me know whats missing? | 15:24 |
| ade_lee__ | sean-k-mooney, this is for https://review.opendev.org/c/openstack/tempest/+/826580 as discussd the other day | 15:24 |
| sean-k-mooney | sure ill take a look quickly and see if anything jumps out | 15:26 |
| sean-k-mooney | you might need to set swap on the subnode too but ill check that | 15:26 |
| sean-k-mooney | ade_lee__: actuly devstack may not have finished on the compute | 15:29 |
| sean-k-mooney | ya ok | 15:29 |
| sean-k-mooney | so on the compute devstack failed to install pip | 15:29 |
| sean-k-mooney | https://zuul.opendev.org/t/openstack/build/554ffdeb6c3a4cdc8c53c55dddd6c36e/log/compute1/logs/devstacklog.txt | 15:29 |
| dmitriis | sean-k-mooney: on 21.04 server tests are passing even with patches applied like it happens on the CI. Also not getting any failures in a 21.04 container locally with patches applied (with a rebase to master). Without a rebase I seem to be getting only "writer privilege escalation not allowed". So probably something in 21.10 libs is different and | 15:30 |
| dmitriis | results in the breakage with locks. This hasn't happened recently so I'll try to explore further which update might have caused it. | 15:30 |
| dmitriis | but at least one issue should be covered with rebasing | 15:31 |
| sean-k-mooney | ack | 15:31 |
| sean-k-mooney | if we dont see it in ci for now i would proceed with the rest of your work and we can keep an eye out for it | 15:31 |
| sean-k-mooney | ade_lee__: this might just be an intermient failure on centos 8 | 15:34 |
| sean-k-mooney | so we shoudl recheck but ade_lee__ any plans to move that to centos-9-stream | 15:35 |
| sean-k-mooney | if we dont do that this cycle we shoudl do it next cycle | 15:35 |
| ade_lee__ | sean-k-mooney, maybe -- the relevant code seems to be here -- https://opendev.org/openstack/devstack/src/branch/master/tools/install_pip.sh#L121-L134 | 15:35 |
| sean-k-mooney | it looks like hte rax mirror just did not have a cenots8 compatiable packages | 15:36 |
| ade_lee__ | sean-k-mooney, but yes, the plan is absolutely to move to centos-9-stream | 15:36 |
| sean-k-mooney | the issue with 8 is its using python 3.6 | 15:37 |
| sean-k-mooney | which is now end of life upstream | 15:37 |
| sean-k-mooney | so we are going to start to see pip not working on it sonner rather then later | 15:37 |
| ade_lee__ | sean-k-mooney, the only problem right now is that fips+python+centos-9 is broken | 15:37 |
| ade_lee__ | sean-k-mooney, but its getting fixed .. | 15:37 |
| sean-k-mooney | so we might need to expand that to use pip form packages for cenos 8 stream too | 15:38 |
| ade_lee__ | yup | 15:39 |
| sean-k-mooney | https://github.com/pypa/pip/commit/0252c04a16cd93fe422cebf0b48453b559a2e404 | 15:40 |
| sean-k-mooney | pip droped support for 3.6 | 15:40 |
| sean-k-mooney | https://github.com/pypa/pip/blob/main/setup.py#L83 | 15:40 |
| ade_lee__ | sean-k-mooney, as soon as https://bugzilla.redhat.com/show_bug.cgi?id=1942527 is fixed, I plan to move all these to centos-9-stream | 15:40 |
| sean-k-mooney | so ya that is the issue | 15:40 |
| sean-k-mooney | this will break all centos 8 jobs so we should likely fix devstack to work around it for now | 15:41 |
| ade_lee__ | which is supposed to happen maybe by next week .. | 15:41 |
| ade_lee__ | yeah - I'll expand the conditional and see if that works | 15:41 |
| * sean-k-mooney clicks | 15:41 | |
| sean-k-mooney | just have it check is_fedroa | 15:42 |
| sean-k-mooney | drop the rest | 15:42 |
| sean-k-mooney | and it shoudl be correct | 15:42 |
| sean-k-mooney | we just need to ensure python3-pip is in the rpm files in devstack | 15:43 |
| ade_lee__ | sean-k-mooney, ok - trying .. | 15:43 |
| sean-k-mooney | which it is https://github.com/openstack/devstack/blob/master/files/rpms/general#L29 | 15:44 |
| sean-k-mooney | ade_lee__: by the way that is in post but has to go through qe ectra and then be published and buil in centos | 15:45 |
| sean-k-mooney | so it will proably take more then a week | 15:45 |
| ade_lee__ | sigh .. yeah | 15:45 |
| sean-k-mooney | i mean it wont take that much longer there is a merge request up | 15:46 |
| sean-k-mooney | we can still do it this cycle | 15:46 |
| ade_lee__ | sean-k-mooney, oh we definitely want to do it this cycle | 15:46 |
| sean-k-mooney | am so pip installed fine on one of the host | 15:46 |
| sean-k-mooney | you can proably just recheck and see if you get looking and avoid rackspace | 15:47 |
| sean-k-mooney | but i think infra/qa are aware of the centos 8 failure | 15:47 |
| ade_lee__ | I'll throw up the dependent patch to avoid the issue for now | 15:48 |
| sean-k-mooney | elodilles: ^ pip being broken on centos 8 on some clouds, is that new to you | 15:48 |
| ade_lee__ | with my luck, I'll get rax three times in a row | 15:48 |
| sean-k-mooney | ade_lee__: well staticially they used to be one of the larges ci providers so proably | 15:51 |
| ade_lee__ | sean-k-mooney, seems like others have run into this -- https://review.opendev.org/c/openstack/devstack/+/779806 | 15:53 |
| sean-k-mooney | ya | 15:54 |
| sean-k-mooney | i was conisdering addign a env var to disable it a few weeks ago | 15:54 |
| sean-k-mooney | actully did i push my one | 15:55 |
| sean-k-mooney | i have been using a modifed install_pip on one of my hosts | 15:55 |
| sean-k-mooney | ade_lee__: https://termbin.com/k0x9 | 15:57 |
| sean-k-mooney | ade_lee__: i was modifying it to only install pip if it was not already there | 15:59 |
| ade_lee__ | sean-k-mooney, makes sense -- anyways, for now added this as depends -- https://review.opendev.org/c/openstack/devstack/+/826714 - lets see how it goes | 16:00 |
| pmonteir | Hi everybody! I was checking the "live_migration_downtime" parameter and I there's a function that supposedly configures the max_downtime, but this function (migrate_configure_max_downtime in https://opendev.org/openstack/nova/src/commit/ca03fe87371d8c13fc2895f8ff3e7e1ca88cfe79/nova/virt/libvirt/guest.py) calls another one which is just a "pass", has anyone ever used this parameter? I think it's not working properly, | 16:03 |
| pmonteir | should this be implemented? | 16:03 |
| pmonteir | ps: this function that's just a pass that I mentioned above, belongs to a "tests" folder (which makes me think that it shouldn't be used outside of a testcase). | 16:12 |
| opendevreview | Balazs Gibizer proposed openstack/placement master: Add any-traits support for listing resource providers https://review.opendev.org/c/openstack/placement/+/826491 | 16:15 |
| opendevreview | Balazs Gibizer proposed openstack/placement master: Add any-traits support for allocation candidates https://review.opendev.org/c/openstack/placement/+/826492 | 16:15 |
| opendevreview | Balazs Gibizer proposed openstack/placement master: Remove unused compatibility code https://review.opendev.org/c/openstack/placement/+/826493 | 16:15 |
| opendevreview | Balazs Gibizer proposed openstack/placement master: Add microversion 1.39 to support any-trait queries https://review.opendev.org/c/openstack/placement/+/826719 | 16:15 |
| gibi | pmonteir: what you see is the libvirt test fixture we use for functional testing | 16:17 |
| gibi | pmonteir: in production we use the real python binding for libvirt | 16:18 |
| gibi | https://libvirt.org/python.html | 16:18 |
| prometheanfire | uefi guests are failing to boot after upgrade to xena :| nova.exception.UEFINotSupported: UEFI is not supported | 16:18 |
| prometheanfire | trying to find what in particular is triggering it | 16:18 |
| gibi | pmonteir: so the the migrateSetMaxDowntime call is sent to libvirt | 16:20 |
| prometheanfire | introduced in faad45b6323d7c52d35b7ccc45eacb5580b3b4d3 | 16:21 |
| pmonteir | gibi: ohhh, I think I got it know... | 16:21 |
| pmonteir | but shouln't it be virDomainMigrateGetMaxDowntime then? Don't know if I'm looking at the right place here > https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainMigrateSetMaxDowntime | 16:22 |
| pmonteir | I meant virDomainMigrateSetMaxDowntime* | 16:23 |
| gibi | pmonteir: look at the naming rules at https://libvirt.org/python.html I think the python bindig call is generated to be MigrateSetMaxDowntime on teh domain object | 16:24 |
| gibi | sorry migrateSetMaxDowntime | 16:25 |
| prometheanfire | spatel: think this is a nova issue? (if so we should probably talk here) :D | 16:26 |
| spatel | prometheanfire did you check your vm xml file and see what are the path you are seeing? | 16:27 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: DNM: run nova tests with any--traits placement feature https://review.opendev.org/c/openstack/nova/+/825914 | 16:28 |
| prometheanfire | arch not specified, so hits the continue | 16:33 |
| prometheanfire | spatel: the vm was shut down, so it's failing to boot on being defined | 16:33 |
| prometheanfire | instance id is not listed in /etc/libvirt/qemu/ | 16:34 |
| pmonteir | gibi: I think I got it now, thanks! | 16:35 |
| prometheanfire | if I had the xml it'd be easier to debug | 16:35 |
| gibi | pmonteir: cool | 16:35 |
| gmann | dansmith: gibi can you check this backport centos8 nodeset usage update for stable/xena https://review.opendev.org/c/openstack/nova/+/825930 | 16:44 |
| gmann | after that I will recheck on devstack patch and see if need to backport it for other stable too https://review.opendev.org/c/openstack/devstack/+/824290 | 16:45 |
| dansmith | gmann: by "check" you mean "rubber stamp" right? | 16:45 |
| gmann | dansmith: yes :) | 16:45 |
| gibi | gmann: looks good to me, but I only have the small stamp on stable | 16:45 |
| gmann | oh did not know | 16:46 |
| dansmith | gibi: any concern with me ninjaing? | 16:46 |
| gibi | dansmith: go for it | 16:46 |
| prometheanfire | I think nova uefi booting only works with redhat now | 16:51 |
| sean-k-mooney | prometheanfire: it shoudl work on ubuntu 20. | 17:11 |
| sean-k-mooney | 20.04 | 17:11 |
| sean-k-mooney | since that is what the dev and testign was down with | 17:11 |
| prometheanfire | debian buster doesn't seem to be working at least, still trying to figure out what in particular is throwing the flag | 17:12 |
| prometheanfire | sean-k-mooney: if you have suggestions on how to debug https://github.com/openstack/nova/commit/faad45b6323d7c52d35b7ccc45eacb5580b3b4d3#diff-67d0163175a798156def4ec53c18fa2ce6eba79b6400fa833a9219d3669e9a11R1465 I'd appreciate it :D | 17:13 |
| prometheanfire | atm I'm about to put in a bunch of debug statements | 17:14 |
| sean-k-mooney | what is the errror you are seeing | 17:16 |
| prometheanfire | traceback nova.exception.UEFINotSupported: UEFI is not supported | 17:16 |
| prometheanfire | which tracks down to just that get_loader function | 17:17 |
| gmann | dansmith: one more easy one https://review.opendev.org/c/openstack/osc-placement/+/819203 | 17:17 |
| sean-k-mooney | prometheanfire: but that is coming form use parsing libvirt | 17:17 |
| sean-k-mooney | prometheanfire: so that impleis qemu is sayign secure boot is not supproted on debian | 17:17 |
| sean-k-mooney | prometheanfire: can you paste the output of vrish capablities adn vrish domcaps | 17:18 |
| prometheanfire | the ovmf files are there, it used to work, I'll look some more, I didn't see any errors on the libvirt-daemon side | 17:18 |
| prometheanfire | k | 17:18 |
| dansmith | gmann: done | 17:18 |
| gmann | dansmith: thanks | 17:18 |
| prometheanfire | sean-k-mooney: capabilities https://gist.github.com/prometheanfire/adb89403ef70a11301b9cc45b040cd4e | 17:22 |
| sean-k-mooney | ack looks liek the loader info is not in the main caps proably dom caps | 17:23 |
| prometheanfire | sean-k-mooney: domcaps https://gist.github.com/prometheanfire/49bf656542cacbf8e37c275cb08ca27b | 17:23 |
| sean-k-mooney | ok yes | 17:23 |
| sean-k-mooney | https://gist.github.com/prometheanfire/49bf656542cacbf8e37c275cb08ca27b#file-gistfile1-txt-L10-L25 | 17:24 |
| sean-k-mooney | so ya libvirt said secure no | 17:24 |
| sean-k-mooney | https://gist.github.com/prometheanfire/49bf656542cacbf8e37c275cb08ca27b#file-gistfile1-txt-L23 | 17:24 |
| prometheanfire | but I don't want secure boot | 17:24 |
| sean-k-mooney | because the files it looked at are not the secure boot ones | 17:24 |
| sean-k-mooney | oh right | 17:24 |
| sean-k-mooney | am let me reead the patch | 17:24 |
| prometheanfire | is nova only supporting secure boot for uefi now? | 17:24 |
| prometheanfire | ack | 17:24 |
| sean-k-mooney | well yes secure boot is only supprote with uefi but no you shoudl be able to use uefi without secure boot | 17:25 |
| prometheanfire | cool | 17:25 |
| prometheanfire | so, domcaps says I should support uefi guests, but only without secure boot, nova is saying that's not good enough, I think | 17:26 |
| prometheanfire | (to summarize) | 17:26 |
| sean-k-mooney | ya still readign the code it was out of my cache | 17:28 |
| sean-k-mooney | but that i belvie is incorect | 17:28 |
| sean-k-mooney | we should not require secure boot unless you ask for it | 17:28 |
| opendevreview | Merged openstack/osc-placement master: Updating python testing as per Yoga testing runtime https://review.opendev.org/c/openstack/osc-placement/+/819203 | 17:29 |
| prometheanfire | elsewhere in debug logs it says UEFI support detected | 17:29 |
| sean-k-mooney | prometheanfire: ok so i don tthink the get_loader part is the issue | 17:32 |
| sean-k-mooney | it _get_loaders i think | 17:32 |
| sean-k-mooney | https://github.com/openstack/nova/commit/faad45b6323d7c52d35b7ccc45eacb5580b3b4d3#diff-67d0163175a798156def4ec53c18fa2ce6eba79b6400fa833a9219d3669e9a11R97 | 17:32 |
| sean-k-mooney | you have <value>/usr/share/OVMF/OVMF_CODE.fd</value> | 17:32 |
| sean-k-mooney | we are checking | 17:33 |
| sean-k-mooney | '/usr/share/qemu/firmware', | 17:33 |
| sean-k-mooney | '/etc/qemu/firmware', | 17:33 |
| sean-k-mooney | well maybe that is not the issue actuly | 17:33 |
| prometheanfire | you are only checking json there though? | 17:33 |
| prometheanfire | logs don't show the error message either | 17:34 |
| sean-k-mooney | prometheanfire: bacicly im curently looking back in that patch to see where we determin if its supported or not | 17:36 |
| sean-k-mooney | uefi that is | 17:36 |
| prometheanfire | ack | 17:36 |
| sean-k-mooney | prometheanfire: we really should have test that assert this behavior using xml by the way | 17:37 |
| sean-k-mooney | if we dont we can use your gist as the sameple data and and see if we can repoduce | 17:37 |
| prometheanfire | ack | 17:37 |
| prometheanfire | I think this is a problem in OSA for xena on buster (and maybe bullseye), ovmf in debian buster-backports includes json and secboot files | 17:43 |
| prometheanfire | instance booted with that | 17:43 |
| prometheanfire | sean-k-mooney: https://gist.github.com/prometheanfire/1e2414828d7904daa2c568653c70cd15 | 17:44 |
| prometheanfire | spatel: ^ I think ovmf needs to be installed from backports on buster at least | 17:44 |
| spatel | hmm | 17:45 |
| spatel | sean-k-mooney i had this issue in nova error logs when i was trying to do secure boot - nova error Secure boot requires SMM feature enabled | 17:47 |
| sean-k-mooney | well SMM is enabled by default be libvirt | 17:48 |
| sean-k-mooney | and we do not specify it at all | 17:48 |
| sean-k-mooney | so that seams like a libvirt bug | 17:48 |
| sean-k-mooney | what prometheanfire is reporting is more extream | 17:48 |
| sean-k-mooney | e.g. uefi just does not work on non redhat distos | 17:49 |
| EugenMayer | i have an instance which has a 'stuck image backup task' - is there any way to clean this up? | 17:49 |
| prometheanfire | sean-k-mooney: I THINK that the missing json file may be a cause | 17:49 |
| prometheanfire | /usr/share/qemu/firmware/60-edk2-x86_64.json in the backports version vs not in the main version | 17:50 |
| sean-k-mooney | prometheanfire: ya i think kashyap had a converation with the ubuntu deves at somepoint | 17:50 |
| sean-k-mooney | looking at the code it does seam to be tryign to pasrse that instead fo gettign suff from libvirt | 17:51 |
| sean-k-mooney | but im not really sure | 17:51 |
| opendevreview | Lee Yarwood proposed openstack/nova master: func: Allow compute_driver to be set and used by _IntegratedTestBase https://review.opendev.org/c/openstack/nova/+/764484 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: block_device_info: Add swap to inline https://review.opendev.org/c/openstack/nova/+/826523 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: libvirt: Improve creating images INFO log https://review.opendev.org/c/openstack/nova/+/826524 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: libvirt: Remove defunct comment https://review.opendev.org/c/openstack/nova/+/826525 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: imagebackend: default by_name image_type to config correctly https://review.opendev.org/c/openstack/nova/+/826526 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: image_meta: Add ephemeral encryption properties https://review.opendev.org/c/openstack/nova/+/760454 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: BlockDeviceMapping: Add encryption fields https://review.opendev.org/c/openstack/nova/+/760453 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: BlockDeviceMapping: Add is_local property https://review.opendev.org/c/openstack/nova/+/764485 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: compute: Update bdms with ephemeral encryption details when requested https://review.opendev.org/c/openstack/nova/+/764486 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: virt: Add ephemeral encryption flag https://review.opendev.org/c/openstack/nova/+/760455 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: scheduler: Add an ephemeral encryption pre filter https://review.opendev.org/c/openstack/nova/+/760456 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: block_device: Add DriverImageBlockDevice to block_device_info https://review.opendev.org/c/openstack/nova/+/826527 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: block_device: Add encryption attributes to image and ephemeral disks https://review.opendev.org/c/openstack/nova/+/826528 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: virt: Add block_device_info helper to find encrypted disks https://review.opendev.org/c/openstack/nova/+/826529 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: blockinfo: Add encryption details to the disk_info mappings when provided https://review.opendev.org/c/openstack/nova/+/772272 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: imagebackend: Add disk_info_mapping as an optional attribute of Image https://review.opendev.org/c/openstack/nova/+/826530 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: libvirt: Introduce support for qcow2 with LUKS https://review.opendev.org/c/openstack/nova/+/772273 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: privsep: Move qemu-img create calls under nova.privsep.qemu https://review.opendev.org/c/openstack/nova/+/826750 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: privsep: Return QemuImgInfo objects from qemu-img info calls https://review.opendev.org/c/openstack/nova/+/826751 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: privsep: Add encryption support to qemu-img create command https://review.opendev.org/c/openstack/nova/+/826752 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: libvirt: Report ephemeral encryption traits based on imagebackend https://review.opendev.org/c/openstack/nova/+/826753 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: libvirt: Configure and teardown ephemeral encryption secrets https://review.opendev.org/c/openstack/nova/+/826754 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: imagebackend: Add support to libvirt_info for LUKS based encryption https://review.opendev.org/c/openstack/nova/+/826755 | 18:03 |
| opendevreview | Lee Yarwood proposed openstack/nova master: imagebackend: Cache the key manager when disk is encrypted https://review.opendev.org/c/openstack/nova/+/826756 | 18:03 |
| sean-k-mooney | prometheanfire: so can you see if adding the file will actully resolve this | 18:05 |
| prometheanfire | sean-k-mooney: fix already deployed :| | 18:05 |
| sean-k-mooney | ok so you are fixing via packaging | 18:06 |
| prometheanfire | yep, ovmf from buster-backports | 18:06 |
| sean-k-mooney | i think we might stil need to update the doc | 18:06 |
| prometheanfire | probably | 18:06 |
| EugenMayer | to 'unstuck from a stuck iamge backup task' i used 'nova reset-state` .. eventhough the task did not show up, i can neither restart the instance soft or hard. So somehow it is still in an undefined state. Any hint how to get out of this (alive :) ) | 18:10 |
| ganso | Hi folks. I see that since Wallaby the option --live has been removed from the migrate command, therefore not providing a way to bypass the scheduler. I have a customer that is trying to evacuate a host for maintenance and is not able to evacuate because of the anti-affinity policy, therefore the scheduler needs to be bypassed. I've looked at the evacuate commands but I strongly suspect that they will not bypass the scheduler | 18:13 |
| ganso | Is there any other way to bypass the scheduler? | 18:13 |
| EugenMayer | ganso there was a hack i got told, but i forgot about it (had the same question). I ended up backing up the instance and restoring it with the other affinity policy, the moving it to the new host during the restore | 18:16 |
| ganso | EugenMayer: thanks! unfortunately the instance cannot be shutdown :\ | 18:17 |
| EugenMayer | All in one, comparing to what i know from k8s, affinity policies in openstack are really clunky since changing the basically means 'rebuilding the instance' | 18:17 |
| EugenMayer | ganso well, no clue then, sorry | 18:17 |
| ganso | EugenMayer: yea the instance cannot be removed from the group, it is only added to the group when it is created and removed only when deleted... I'm trying to avoid DB surgery | 18:18 |
| EugenMayer | ah i remember what it was, i think you just downgrade the API level in the cli and force the parameter anyway - i think that was it | 18:19 |
| EugenMayer | you should test that though - do not trust me :) | 18:19 |
| rosmaita | bauzas: i posted these to deal with the openstacksdk-functional-devstack failures in the stable branches: https://review.opendev.org/q/topic:fix-sdk-func-job ... want me to do a set for nova? | 18:23 |
| ganso | EugenMayer: that is still not working, but it is not working for other reason (I am to blame), just a min | 18:25 |
| ganso | EugenMayer: this: https://review.opendev.org/c/openstack/nova/+/800114 | 18:25 |
| ganso | EugenMayer: apparently this might work CONF.workarounds.disable_group_policy_check_upcall | 18:27 |
| EugenMayer | interesting | 18:28 |
| sean-k-mooney | ganso: you can use the old microversion but we really dont like supproting forced live migrations | 18:29 |
| sean-k-mooney | ganso: if yo just want to specify the host you can now do that | 18:29 |
| sean-k-mooney | but it wont bypass the schduler | 18:29 |
| sean-k-mooney | evacuate does not use live migration by the way | 18:30 |
| ganso | sean-k-mooney: yea but apparently it is not a matter of bypassing the scheduler anymore, I think, but I'm confused to how this worked in this past... my patch broke the functionality, but I can't see exactly how the forced host could also bypass the anti-affinity check before | 18:31 |
| ganso | sean-k-mooney: oh it doesn't? I won't run pre_live_migration checks if it is a live-evacuation? | 18:31 |
| EugenMayer | sean-k-mooney i recall you helped me unstack an instance which had an endless/stack task (image backup) running. I tried to reset state but i still cannot restart the instance or check the logs/console. Are there any leftovers? | 18:31 |
| sean-k-mooney | ganso: it wont bypass the late affintiy check on the compute | 18:31 |
| sean-k-mooney | you can only bypass that by disabling it on the host temporaly | 18:32 |
| sean-k-mooney | ganso: live evacuation are not a thing at the api level its a nova client thing | 18:33 |
| sean-k-mooney | so nova host-evacuate-live | 18:33 |
| sean-k-mooney | is just a clisnt side for loop that does live migrations | 18:33 |
| ganso | sean-k-mooney: oh, therefore it will use live-migration code at the backend | 18:34 |
| sean-k-mooney | ganso: the workaroudn you can do for new is select a singel host, temperally disable the upcall then migrate teh host to the host useing a force live migratio nwith the old microversion | 18:34 |
| sean-k-mooney | then renable the upcall | 18:34 |
| sean-k-mooney | ganso: yes | 18:34 |
| ganso | sean-k-mooney: thanks! | 18:35 |
| sean-k-mooney | lol im glad you coudl read that. i am slightly distracted and that had more typos then i usually have | 18:36 |
| ganso | sean-k-mooney: lol np it isn't hard to spellcheck while reading xD | 18:38 |
| elodilles | sean-k-mooney: sorry for the late answer. so, no, i didn't know about any pip issue in centos job. though my local devstack started to fail with the very same point (pip bootstrap; ubuntu focal host), so it's weird :S | 19:05 |
| sean-k-mooney | elodilles: pip upsteream now require py3.7+ | 19:06 |
| sean-k-mooney | since 3.6 is eol | 19:06 |
| sean-k-mooney | my local hack which i have not push is to skip installing pip form pypi if pip is already installed | 19:06 |
| elodilles | it's interesting, because py3.8 is on the host | 19:07 |
| sean-k-mooney | is that you default python | 19:07 |
| elodilles | sean-k-mooney: i did exactly the same as a temporary fix | 19:07 |
| elodilles | yepp, 3.8 is the default | 19:07 |
| sean-k-mooney | ya so we used to install pip form pypi in the past becasue the default was old | 19:07 |
| sean-k-mooney | but im thinking maybe we shoudl rever to the package manager version | 19:08 |
| sean-k-mooney | or at least have an option too | 19:08 |
| elodilles | good question | 19:08 |
| elodilles | i need to leave now... but my feeling is that we will see the issue in other jobs as well in the coming days, so some kind of general fix might be needed | 19:21 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add PCI VPD Capability Handling https://review.opendev.org/c/openstack/nova/+/808199 | 21:25 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Include pf mac and vf num in port updates https://review.opendev.org/c/openstack/nova/+/824833 | 21:25 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Introduce remote_managed tag for PCI devices https://review.opendev.org/c/openstack/nova/+/824834 | 21:25 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835 | 21:25 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Bump os-traits to 2.7.0 https://review.opendev.org/c/openstack/nova/+/826675 | 21:25 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111 | 21:25 |
| *** dasm is now known as dasm|off | 22:58 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!