| *** amoralej|off is now known as amoralej | 07:28 | |
| *** hemna3 is now known as hemna | 07:37 | |
| gibi | sean-k-mooney: hi! could you upgrade my vote from +1 to +2 on https://review.opendev.org/q/topic:bug%252F1952941 ? thanks! | 11:10 |
|---|---|---|
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835 | 11:11 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111 | 11:11 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Add supports_remote_managed_ports capability https://review.opendev.org/c/openstack/nova/+/827839 | 11:11 |
| sean-k-mooney[m] | gibi: ah the requestspec fix sure | 11:12 |
| dmitriis | gibi: apologies for an extra respin of https://review.opendev.org/c/openstack/nova/+/812111. I moved the addition of the compute capability to a separate change https://review.opendev.org/c/openstack/nova/+/827839 and made an extra check in the whitelist. This way we can check if Libvirt has the right version and reject remote_managed ports if it | 11:12 |
| dmitriis | doesn't early at startup. | 11:12 |
| sean-k-mooney[m] | ill review the series shortly | 11:12 |
| gibi | dmitriis: no worry I will check after lunch | 11:16 |
| dmitriis | gibi: ack, tyvm | 11:16 |
| opendevreview | sean mooney proposed openstack/nova master: [WIP] add initial healthcheck support https://review.opendev.org/c/openstack/nova/+/825015 | 11:18 |
| opendevreview | sean mooney proposed openstack/nova master: [WIP] add healthcheck manager to manager base https://review.opendev.org/c/openstack/nova/+/827844 | 11:18 |
| sean-k-mooney | stephenfin: by th way im also good with your unittest patch https://review.opendev.org/c/openstack/nova/+/714676 gibi was previously +2 on it | 12:10 |
| sean-k-mooney | i just approved his patches which will conflict with it | 12:10 |
| sean-k-mooney | and rechecked yours | 12:11 |
| sean-k-mooney | so waht i woudl propose is if we need to rebase your mock removal patch we shoudl try and fast approve it when you resubmit to avoid conflict with other patches | 12:11 |
| gibi | dmitriis: left comments in https://review.opendev.org/c/openstack/nova/+/827839 | 12:17 |
| sean-k-mooney | artom_: did you put up a patch to skip the flaky test in nova-next | 12:19 |
| sean-k-mooney | if not i might quickly do that | 12:19 |
| gibi | sean-k-mooney: I don't see such patch from artom | 12:20 |
| sean-k-mooney | we are only seeing tempest.api.compute.servers.test_device_tagging.TaggedAttachmentsTest.test_tagged_attachment fail so ill limit it to that | 12:21 |
| gibi | sean-k-mooney: yepp | 12:21 |
| sean-k-mooney | ah https://bugs.launchpad.net/nova/+bug/1959899 is the bug cool | 12:24 |
| gibi | yes | 12:24 |
| sean-k-mooney | i was just looking for it for the topic/commit message | 12:24 |
| opendevreview | sean mooney proposed openstack/nova master: skip test_tagged_attachment in nova-next https://review.opendev.org/c/openstack/nova/+/827851 | 12:35 |
| sean-k-mooney | ok ya i think ^ is all that is needed | 12:36 |
| opendevreview | Merged openstack/nova master: Reproduce bug 1952941 https://review.opendev.org/c/openstack/nova/+/820121 | 12:42 |
| gibi | sean-k-mooney: thanks | 12:44 |
| gibi | sean-k-mooney: I will fast approve it as soon as zuul is green on the patch | 12:44 |
| artom_ | gibi, sean-k-mooney, I hadn't, so thanks! | 12:52 |
| opendevreview | Attila Fazekas proposed openstack/nova master: Document the floating ip behavior in short https://review.opendev.org/c/openstack/nova/+/827856 | 12:58 |
| dmitriis | gibi: looking | 13:28 |
| tobias-urdin | i have a weird issue, back in december 2021 a resize that failed, also failed to cleanup /var/lib/instances/<uuid>_resize directory, when resize was retried now (earlier today) it failed because os.rename got File exists when it tried to rename that https://github.com/openstack/nova/blob/stable/victoria/nova/virt/libvirt/driver.py#L10402 -- see | 13:31 |
| tobias-urdin | logs https://paste.opendev.org/show/812522/ | 13:31 |
| tobias-urdin | the first issue was because of a libvirt issue (out of fd and socket issues) | 13:31 |
| tobias-urdin | don't have logs to understand if for example SELinux blocked the file deletion, does it make sense to insert something that would catch the issue and remove the _resize directory before trying the rename? since the API would ensure there can't be an race condition for resizes | 13:32 |
| gibi | tobias-urdin: do I understand correctly that the original delete of the dir faild and this now prevents the retry of the resize? I think we can robustify nova by retry the delete during the next resize if it sees the stale dir there | 13:34 |
| *** dasm|off is now known as dasm | 13:36 | |
| tobias-urdin | gibi: yes, resize failed because of leftover <uuid>_resize directory in instances directory | 13:51 |
| opendevreview | Tobias Urdin proposed openstack/nova master: Cleanup _resize directory in instances dir on resize https://review.opendev.org/c/openstack/nova/+/827865 | 14:02 |
| tobias-urdin | gibi: i guess it could be as simple as ^ | 14:03 |
| gibi | tobias-urdin: that could be it, but would be nice to add a test around it | 14:12 |
| gibi | tobias-urdin: and bug report for tracking purposes :) | 14:13 |
| tobias-urdin | yeah i'll fix everything up if it feels acceptable (bug report, test, release note etc) - should be very low risk backport as well | 14:16 |
| gibi | tobias-urdin: I agree this feels a safe thing to do | 14:16 |
| sean-k-mooney | i have not beeing following but you will need to ensuer its not a shared file system | 14:17 |
| sean-k-mooney | otherwise the cleanup would delete the vm | 14:17 |
| gibi | sean-k-mooney: it is the _resize dir I think that is being created there, so deleting the dir before we creating it feels save | 14:17 |
| gibi | safe | 14:17 |
| opendevreview | Merged openstack/nova master: Migrate RequestSpec.numa_topology to use pcpuset https://review.opendev.org/c/openstack/nova/+/820153 | 14:18 |
| sean-k-mooney | ah ok we have a parallel dir that we then move into the final location later? | 14:18 |
| sean-k-mooney | os.rename(inst_base, inst_base_resize) | 14:18 |
| sean-k-mooney | so that is moving inst_base to inst_base_resize | 14:19 |
| gibi | yepp, to save it if we want to roll back | 14:19 |
| sean-k-mooney | ok so ya that should be safe | 14:19 |
| gibi | tobias-urdin: thanks for taking care of | 14:19 |
| gibi | it | 14:20 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Reproduce bug 1952941 https://review.opendev.org/c/openstack/nova/+/827868 | 14:30 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Migrate RequestSpec.numa_topology to use pcpuset https://review.opendev.org/c/openstack/nova/+/827869 | 14:30 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Reproduce bug 1952941 https://review.opendev.org/c/openstack/nova/+/827870 | 14:34 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Migrate RequestSpec.numa_topology to use pcpuset https://review.opendev.org/c/openstack/nova/+/827871 | 14:34 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: Reproduce bug 1952941 https://review.opendev.org/c/openstack/nova/+/827872 | 14:37 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/victoria: Migrate RequestSpec.numa_topology to use pcpuset https://review.opendev.org/c/openstack/nova/+/827873 | 14:37 |
| opendevreview | Merged openstack/nova master: Revert "Revert resize: wait for events according to hybrid plug" https://review.opendev.org/c/openstack/nova/+/819494 | 15:23 |
| opendevreview | Merged openstack/nova stable/wallaby: [rt] Apply migration context for incoming migrations https://review.opendev.org/c/openstack/nova/+/820555 | 15:43 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Add supports_remote_managed_ports capability https://review.opendev.org/c/openstack/nova/+/827839 | 16:08 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: [yoga] Add support for VNIC_TYPE_SMARTNIC https://review.opendev.org/c/openstack/nova/+/824835 | 16:08 |
| opendevreview | Dmitrii Shcherbakov proposed openstack/nova master: Filter computes without remote-managed ports early https://review.opendev.org/c/openstack/nova/+/812111 | 16:08 |
| dmitriis | gibi: re-uploaded with the cases mentioned covered | 16:17 |
| dmitriis | gibi: hopefully the approach in general is OK | 16:17 |
| gibi | dmitriis: thanks. I will check it on monday | 16:36 |
| gibi | have a nice weekend | 16:36 |
| dmitriis | gibi: same to you, cheers o/ | 16:37 |
| gibi | o/ | 16:37 |
| * gibi leaves the building | 16:42 | |
| *** amoralej is now known as amoralej|off | 17:41 | |
| opendevreview | Federico Ressi proposed openstack/nova master: Debug Nova APIs call failures https://review.opendev.org/c/openstack/nova/+/806683 | 17:42 |
| chateaulav | has anyone run into issues with libvirt in rgards to `error: failed to get CPU model names` | 18:32 |
| sean-k-mooney | i hit it yesterday when i set an incorrct value on an arm system | 18:37 |
| sean-k-mooney | i put virt for [libvirt]/cpu_model | 18:38 |
| sean-k-mooney | it should have been max or host | 18:38 |
| sean-k-mooney | so if you use virt-type=qemu and cpu_model=virt it will raise that error | 18:38 |
| sean-k-mooney | i shoudl have had virt_type=qemu, cpu_mode=custom and cpu_model=max with machine_type=aarch64:virt or set on the image | 18:39 |
| chateaulav | sean-k-mooney: it relates to riscv currently. I can run emulation using the qemu binary directly; however, run into issue because virsh doesnt see any available cpu models | 18:39 |
| chateaulav | https://www.irccloud.com/pastebin/SoDG0HXa/error-risc-v | 18:40 |
| sean-k-mooney | if you do qemu-system-riscv64 --cpu ? | 18:42 |
| sean-k-mooney | it will list the supported ones | 18:42 |
| sean-k-mooney | home ok libvirt may not have support for that yet | 18:42 |
| sean-k-mooney | you could try rv64 | 18:42 |
| sean-k-mooney | for the cpu_model | 18:43 |
| sean-k-mooney | and see if it work if you spcify one | 18:43 |
| chateaulav | yeah no luck, been going back and forth and though i may have missed something. | 18:43 |
| sean-k-mooney | no it sound like libvirt is missing supprt | 18:43 |
| chateaulav | yeah thats what i was thinking to | 18:45 |
| sean-k-mooney | it might work in a later libvirt | 18:45 |
| sean-k-mooney | what version are you using | 18:45 |
| sean-k-mooney | the most recent release is 8.0.0 i belive | 18:45 |
| chateaulav | libvirtd (libvirt) 6.0.0 | 18:45 |
| sean-k-mooney | i mean its ment to have supprot | 18:46 |
| sean-k-mooney | i see stuff as far back as 4.7.0 | 18:46 |
| sean-k-mooney | in the release notes | 18:46 |
| chateaulav | it seems like partial support, gonna test along that route | 18:47 |
| sean-k-mooney | maybe they dont support emulation with riscv | 18:47 |
| sean-k-mooney | just native supprot? | 18:47 |
| chateaulav | I feel that as well, a couple last checks and then ill put a nail in it | 18:49 |
| opendevreview | Artom Lifshitz proposed openstack/nova master: DNM: Run Nova gate on Neutron notifier patch https://review.opendev.org/c/openstack/nova/+/827917 | 19:56 |
| opendevreview | Ade Lee proposed openstack/nova stable/xena: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827895 | 19:59 |
| opendevreview | Ade Lee proposed openstack/nova stable/xena: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827895 | 20:00 |
| opendevreview | Ade Lee proposed openstack/nova stable/wallaby: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827896 | 20:01 |
| melwitt | dansmith: hi :D I have been working on removing tracebacks in the logs in nova-multi-cell that happen when a neutron network-changed event is received after the instance has moved to another cell during a cross-cell migration, | 20:03 |
| * sean-k-mooney note to self you still hate debuging networkign issues. | 20:04 | |
| melwitt | this seemed simple at first until the fact that the exception is being raised from the db access on nova-conductor but the tracing is happening in nova-compute, | 20:04 |
| sean-k-mooney | it helps if you enable forwarding on the host if you want nat to work for your vms.... sudo sysctl net.ipv4.conf.all.forwarding=1 | 20:05 |
| dansmith | melwitt: because of indirection I assume | 20:05 |
| melwitt | I'm seeing tracing still from the rpc messaging layer, so I tried this https://review.opendev.org/c/openstack/nova/+/826942/5/nova/objects/instance_info_cache.py#77 but it doesn't work as expected. I get oslo_messaging.rpc.client.RemoteError: Remote error: ExpectedException on the client side instead of InstanceNotFound | 20:06 |
| melwitt | I had thought when oslo.messaging receives ExpectedException on the client side it translates it into the original exception type that it wrapped on the server side, but that is not occurring | 20:06 |
| dansmith | no, I think that just avoids the traceback on the conductor side | 20:07 |
| melwitt | hm ok | 20:08 |
| sean-k-mooney | melwitt: i assume you are not checkign if the instance exists then updating because it could get deleted in between | 20:08 |
| sean-k-mooney | which is why you are trying to catch the excpiton | 20:08 |
| melwitt | sean-k-mooney: yeah | 20:08 |
| melwitt | well, I'm catching DBReferenceError and then raising InstanceNotFound to be handled by upper layers | 20:09 |
| sean-k-mooney | do the upper layers need to do anything with the not found | 20:09 |
| sean-k-mooney | the instance has been moved to the other cell yes | 20:09 |
| melwitt | most things handle not found, like "the instance disappeared while <whatever>" | 20:10 |
| melwitt | if someone deletes while it's in the middle of <whatever> | 20:11 |
| melwitt | it's working fine except I'm still getting tracebacks in nova-compute coming from oslo.messaging and I can't get it to stop | 20:11 |
| sean-k-mooney | ack | 20:11 |
| melwitt | it sends over the longass chain of exception tracebacks even though I caught DBReferenceError on the conductor side | 20:13 |
| dansmith | melwitt: I think oslo.messaging wraps exceptions in a special class called Original_Remote, IIRC | 20:29 |
| dansmith | not sure you can catch that directly though because I think it's dynamic | 20:29 |
| dansmith | if you catch it with except Exception, then try to print e.__class__ | 20:29 |
| melwitt | dansmith: yeah, it's weird. it's as if it's logging it somewhere I can't get to (like in oslo.messaging). this is an example from PS3 before I added any @expected_exceptions, it logs at INFO what I wanted and then directly after that it logs the supertraceback https://zuul.opendev.org/t/openstack/build/7c3c755868b848ed841d37457004d99b/log/controller/logs/screen-n-cpu.txt#10027 | 20:33 |
| melwitt | I don't think? it's just appended to the exception message bc it's logged in a different color but since I'm desperate I might try truncating str(e) to see what happens | 20:33 |
| dansmith | yeah I expect it's logged explicitly somewhere, but not sure | 20:36 |
| dansmith | nothing specific jumps out at me though | 20:36 |
| melwitt | dansmith: ok, I'll try some more things. thanks for looking :) | 20:39 |
| opendevreview | Jonathan Race proposed openstack/nova master: Adds Pick guest CPU architecture based on host arch in libvirt driver support https://review.opendev.org/c/openstack/nova/+/822053 | 21:01 |
| opendevreview | Ade Lee proposed openstack/nova stable/wallaby: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827896 | 21:28 |
| opendevreview | melanie witt proposed openstack/nova master: Raise InstanceNotFound on fkey constraint fail saving info cache https://review.opendev.org/c/openstack/nova/+/826942 | 22:09 |
| opendevreview | Ade Lee proposed openstack/nova stable/wallaby: Add check job for FIPS https://review.opendev.org/c/openstack/nova/+/827896 | 22:12 |
| *** dasm is now known as dasm|off | 22:13 | |
| *** Corwin is now known as Guest1826 | 23:39 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!