| *** dasm is now known as dasm|off | 01:49 | |
| opendevreview | Yusuke Okada proposed openstack/nova master: Fix failed count for anti-affinity check https://review.opendev.org/c/openstack/nova/+/873216 | 05:04 |
|---|---|---|
| *** blarnath is now known as d34dh0r53 | 06:47 | |
| opendevreview | Tobias Urdin proposed openstack/nova master: libvirt: set remaining to 0 when no disk to migrate https://review.opendev.org/c/openstack/nova/+/873846 | 07:36 |
| bauzas | good morning Nova | 08:14 |
| bauzas | gibi: I thought we merged most of the wait_for_ssh series for volume attachments issues https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6f9/868236/5/gate/nova-next/6f9f3d0/testr_results.html | 08:28 |
| gibi | bauzas: good morning | 08:35 |
| gibi | bauzas: what you see there is that the wait for ssh step we merged times out. based on the guest log, the guest is still waiting for the DHCP to finish when the tempest times out | 08:36 |
| gibi | slow node? slow dhcp? | 08:36 |
| bauzas | oh so this the udhchpd issue | 08:37 |
| bauzas | https://bugs.launchpad.net/nova/+bug/2006467 | 08:37 |
| gibi | no | 08:37 |
| gibi | this is the last message from the guest in your case | 08:37 |
| gibi | udhcpc: sending discover | 08:37 |
| gibi | in the bug you linked the discover fails | 08:38 |
| bauzas | yup, now I see the difference | 08:39 |
| gibi | maybe it would have failed the discover there as well if we the test waited enough | 08:39 |
| bauzas | the tempest test finishes sooner before the dhcp lease is attributed | 08:39 |
| gibi | yepp | 08:39 |
| gibi | bumping the ssh timeout in tempest could be an option | 08:40 |
| gibi | also I suggest to look at a successfull case and check the guest log to see how fast the guest boots in a successful case | 08:40 |
| gibi | in the failed case it needs more than 12 sec to cirros to get to dhcp | 08:41 |
| bauzas | hmmm | 08:41 |
| bauzas | the problem here is that I don't know how to identify whether this issue is happening a lot or just unfortunate | 08:42 |
| gibi | we have to check for `wait_for_ssh_or_ping` and remove the cases where the dhcp lease failes as per https://bugs.launchpad.net/nova/+bug/2006467 | 08:43 |
| gibi | probably manual work | 08:43 |
| gibi | there is a work in progress to add multiple regex support to logsearch to filter for more than one independent pattern in a file but it is not on main branch yet | 08:44 |
| bauzas | I'm also looking at the job output | 08:46 |
| bauzas | trying to identify the timings | 08:46 |
| bauzas | from what I can read, I agree with you, it only calls the dhcp lease after more than 12 secs | 08:47 |
| bauzas | Feb 14 18:18:56.240226 np0033093378 nova-compute[83239]: INFO nova.compute.manager [None req-053318ab-09ad-4a3a-8ddb-633cc0002c3e tempest-AttachVolumeNegativeTest-1605485622 tempest-AttachVolumeNegativeTest-1605485622-project] [instance: 6a265379-ebfd-4aea-a081-8b271f32c0ea] Took 6.36 seconds to spawn the instance on the hypervisor. | 08:53 |
| bauzas | so we waited for the instance to be fully spawned before ssh'ing to it | 08:54 |
| bauzas | 2023-02-14 18:22:39.102680 | controller | 2023-02-14 18:18:59,630 92653 INFO [tempest.lib.common.ssh] Creating ssh connection to '172.24.5.161:22' as 'cirros' with public key authentication | 08:54 |
| bauzas | and that's only 2m30s after this that we give up to connect thru ssh | 08:56 |
| bauzas | 2023-02-14 18:22:39.103394 | controller | 2023-02-14 18:22:31,398 92653 ERROR [tempest.lib.common.ssh] Failed to establish authenticated ssh connection to cirros@172.24.5.161 after 16 attempts. Proxy client: no proxy client | 08:56 |
| bauzas | so, even if the guest took 12 secs to fully boot and being sshable, we still have enough delay here | 08:57 |
| gibi | I only have timestamp until the guest kernel boots | 08:58 |
| gibi | after that the guest logs has no timestamps | 08:58 |
| gibi | [ 12.638156] sr 0:0:0:0: Attached scsi generic sg0 type 5 | 08:58 |
| gibi | currently loaded modules: 8139cp 8390 9pnet 9pnet_virtio ahci drm drm_kms_helper e1000 failover fb_sys_fops hid hid_generic ip_tables isofs libahci mii ne2k_pci net_failover nls_ascii nls_iso8859_1 nls_utf8 pcnet32 qemu_fw_cfg syscopyarea sysfillrect sysimgblt ttm usbhid virtio_blk virtio_gpu virtio_input virtio_net virtio_rng virtio_scsi x_tables | 08:58 |
| gibi | info: copying initramfs to /dev/vda1 | 08:59 |
| gibi | info: initramfs loading root from /dev/vda1 | 08:59 |
| gibi | info: /etc/init.d/rc.sysinit: up at 18.84 | 08:59 |
| gibi | info: container: none | 08:59 |
| gibi | currently loaded modules: 8139cp 8390 9pnet 9pnet_virtio ahci drm drm_kms_helper e1000 failover fb_sys_fops hid hid_generic ip_tables isofs libahci mii ne2k_pci net_failover nls_ascii nls_iso8859_1 nls_utf8 pcnet32 qemu_fw_cfg syscopyarea sysfillrect sysimgblt ttm usbhid virtio_blk virtio_gpu virtio_input virtio_net virtio_rng virtio_scsi x_tables | 08:59 |
| gibi | Initializing random number generator... done. | 08:59 |
| gibi | Starting acpid: OK | 08:59 |
| bauzas | yup, that is what's in the console | 08:59 |
| gibi | Starting network: udhcpc: started, v1.29.3 | 08:59 |
| gibi | udhcpc: sending discover | 08:59 |
| gibi | udhcpc: sending discover | 08:59 |
| gibi | udhcpc: sending discover | 08:59 |
| gibi | so it take at least 12.6 sec to reach this point | 08:59 |
| gibi | but there are things without timestamp that can take time | 08:59 |
| bauzas | gibi: I checked n-cpu to get when the instance was spawned | 08:59 |
| gibi | like re-sending the discover | 08:59 |
| bauzas | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6f9/868236/5/gate/nova-next/6f9f3d0/controller/logs/screen-n-cpu.txt | 09:00 |
| bauzas | with instance uuid 6a265379-ebfd-4aea-a081-8b271f32c0ea | 09:00 |
| bauzas | Feb 14 18:19:02.741562 np0033093378 nova-compute[83239]: DEBUG nova.network.neutron [req-1396e9c5-b2f1-4ceb-ac85-97edccc3919a req-5a2c93ac-f84a-44aa-9f96-5cca954570de service nova] [instance: 6a265379-ebfd-4aea-a081-8b271f32c0ea] Updated VIF entry in instance network info cache for port 77cc62f3-8d3f-4a4f-bbef-ac68fdcd9c9e. {{(pid=83239) _build_network_info_model /opt/stack/nova/nova/network/neutron.py:3459}} | 09:00 |
| bauzas | at 18:19:02, the instance network info cache was updated | 09:01 |
| bauzas | so we're still in the ssh connnection attempts window, which started at 18:18:59 and tried until 18:22:31 | 09:02 |
| gibi | my point is if the 3rd discover would succeed then the test could pass if waiting more for ssh | 09:07 |
| gibi | but we don't know that the 3rd will succeed | 09:07 |
| gibi | or fail with https://bugs.launchpad.net/nova/+bug/2006467 as we don't wait enough to see | 09:07 |
| bauzas | gibi: the problem is that we don't know *when* that 3rd discovery was made given it was on the console | 09:08 |
| bauzas | if we assume that the guest starts after the spawn, then the 3rd discover was at 18:18:56+12s-ish | 09:08 |
| bauzas | and we were still trying to connect thru ssh | 09:09 |
| * bauzas tries to correlate the tempest timings with n-api calls | 09:09 | |
| gibi | do you suggest the that after the 3rd discover request the udhcp freeze and never time out? | 09:10 |
| bauzas | yeah, confirmed, the timings match between the job-output and the n-api log | 09:13 |
| bauzas | in n-api I can see Feb 14 18:18:57.185675 np0033093378 devstack@n-api.service[74363]: DEBUG nova.api.openstack.wsgi [None req-de265685-f44e-4d63-98a7-2e38964ce455 tempest-AttachVolumeNegativeTest-1605485622 tempest-AttachVolumeNegativeTest-1605485622-project] Calling method '<bound method ServersController.show of <nova.api.openstack.compute.servers.ServersController object at 0x7f3bd48f7dc0>>' {{(pid=74363) _process_stack /opt/s | 09:13 |
| bauzas | tack/nova/nova/api/openstack/wsgi.py:513}} | 09:13 |
| bauzas | that correlates with job-output : 2023-02-14 18:22:39.100791 | controller | 2023-02-14 18:18:57,782 92653 INFO [tempest.lib.common.rest_client] Request (AttachVolumeNegativeTest:test_attach_attached_volume_to_different_server): 200 GET https://10.176.197.146/compute/v2.1/servers/6a265379-ebfd-4aea-a081-8b271f32c0ea 0.621s | 09:13 |
| gibi | one thing we can do before bump the ssh timeout is to add "os-getConsoleOutput" nova API call to tempest when ssh timeouts to see the guest log at that point in time | 09:14 |
| bauzas | gibi: I dunno what happens but yeah, the discover happens way earlier than the ssh timeout AFAICU | 09:14 |
| bauzas | or | 09:15 |
| bauzas | between the last console timestamp that says [12...] and the 3rd discover, then 2min15s lasted | 09:15 |
| * bauzas looks at udhcpc client to say how long lasts a discover | 09:16 | |
| bauzas | https://bugs.launchpad.net/cirros/+bug/1273159 | 09:17 |
| bauzas | "It only sends up to 3 DHCP discover packets with a 60 second pause between." :) | 09:18 |
| bauzas | so yeah, that explains | 09:18 |
| bauzas | that explains why we don't see the dhcp lease failing, but that doesn't explain why after 2mins, the dhcp server wasn't providing a lease | 09:18 |
| * gibi_ lost the cable internet | 09:24 | |
| gibi_ | bauzas: this was the last thing I saw | 09:25 |
| gibi_ | 10:18 < bauzas> so yeah, that explains | 09:25 |
| bauzas | (10:18:57) bauzas: that explains why we don't see the dhcp lease failing, but that doesn't explain why after 2mins, the dhcp server wasn't providing a lease | 09:25 |
| gibi_ | yeah | 09:25 |
| gibi_ | so it might be a neutron dhcp issue | 09:26 |
| bauzas | gibi: I'm inclined to say that increasing the ssh timeout wouldn't help | 09:26 |
| gibi_ | bauzas: it might prove that the 3rd discover fails too and then udhcp gives up | 09:26 |
| gibi_ | but I agree that it is unlikely that the 3rd discover will succeed | 09:27 |
| gibi_ | (except if this is a slow worker node situation) | 09:27 |
| bauzas | well | 09:28 |
| bauzas | I wouldn't say that a ssh connection successful after 3 mins would be a good user experience | 09:28 |
| bauzas | we asked to bind the port way earlier | 09:28 |
| bauzas | and the network cache was updated | 09:29 |
| bauzas | so this doesn't look a vif plugging issue | 09:29 |
| bauzas | gibi: welcome back :) | 09:29 |
| bauzas | anyway, I gonna drop my investigations | 09:29 |
| bauzas | we identified the problem | 09:29 |
| bauzas | now we now it can somehow be kinda-related to https://bugs.launchpad.net/nova/+bug/2006467 | 09:30 |
| gibi_ | bauzas: would it make sense to ping #openstack-neutron with ^^ | 09:32 |
| bauzas | gibi_: seems to me a good thing to do + adding neutron to the list of projects impacted by that bug | 09:33 |
| * bauzas grabbing a coffee and then writing some bug comment | 09:33 | |
| opendevreview | Merged openstack/nova-specs master: Amend FQDN in hostname spec to reflect implementation https://review.opendev.org/c/openstack/nova-specs/+/872422 | 09:43 |
| sean-k-mooney | bauzas: its not really a openstack bug is it | 10:15 |
| sean-k-mooney | bauzas: like its not a nova bug we have nothing to do with dhcp | 10:15 |
| sean-k-mooney | bauzas: perhaps we can finally bump the verion of cirros in devstack | 10:16 |
| sean-k-mooney | to a 6.x version | 10:16 |
| sean-k-mooney | that should fix some of the kernel panics we see too | 10:16 |
| stephenfin | Do folks think dropping the legacy migrations and fully supporting SQLA 2.0 is viable during the A cycle? https://review.opendev.org/c/openstack/nova/+/872428/ | 10:16 |
| sean-k-mooney | you mean by tommorow right | 10:17 |
| stephenfin | oof, I thought we had longer for services | 10:17 |
| stephenfin | then I guess that answers my question 0:) | 10:17 |
| sean-k-mooney | i saw you submit that and also use sdk seriese | 10:17 |
| sean-k-mooney | well we might be able too but i have not looked at the patch yet | 10:18 |
| stephenfin | yeah, the use SDK series is a nice-to-have and highlights some SDK gaps. This one's a _little_ more urgent | 10:18 |
| bauzas | stephenfin: while I understand your concern, I'm afraid of merging it 2 days before FF, given the huge number of CI failures we havbe | 10:18 |
| sean-k-mooney | i skimed it when you first submitted it and the second patch failed zuul | 10:18 |
| sean-k-mooney | look liek it passed after a recheck | 10:19 |
| sean-k-mooney | bauzas: why | 10:19 |
| sean-k-mooney | this code is not used currently | 10:19 |
| sean-k-mooney | if i recall correctly its only used if your coming form pre train | 10:20 |
| sean-k-mooney | actully pre wallaby based on the commit message | 10:20 |
| sean-k-mooney | so thsi wont impact any ci jobs | 10:21 |
| sean-k-mooney | stephenfin: for what its worth we can land them early after RC1 if we dont land them now | 10:21 |
| bauzas | sean-k-mooney: because we already have a lot of changes to be merged | 10:21 |
| bauzas | and I'm done with the CI failures | 10:22 |
| stephenfin | yeah, early in RC1 would work also. We just don't want to drag our feet on this. | 10:22 |
| sean-k-mooney | well i ment after RC1 so in bobcat | 10:22 |
| sean-k-mooney | i would be ok with proceedign with this in Antelope however | 10:23 |
| sean-k-mooney | if ci does not object | 10:23 |
| sean-k-mooney | what is the sate of ci in general currently. has the tempest issue been fixed | 10:25 |
| bauzas | yes | 10:25 |
| bauzas | but now we also have other issues | 10:25 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/845754 | 10:29 |
| sean-k-mooney | stephenfin: im +2 on both of those if others care to review | 10:30 |
| sean-k-mooney | otherwise we can try and land it again in 2 weeks or so | 10:30 |
| sean-k-mooney | bauzas: is https://review.opendev.org/c/openstack/nova/+/873584 just there so we can merge it before RC1 or have ye found the issue in the tests? | 11:35 |
| bauzas | sean-k-mooney: we merged it yesterday | 11:35 |
| bauzas | I mean the logging patch | 11:35 |
| bauzas | so I'd prefer to keep it logging for more than one day | 11:35 |
| sean-k-mooney | ack i was just asking if they issue had been found or not | 11:37 |
| opendevreview | Rajesh Tailor proposed openstack/nova master: Fix case-sensitivity for metadata keys https://review.opendev.org/c/openstack/nova/+/873901 | 11:37 |
| sean-k-mooney | we can leave it there for a week or so until we get clsoe to rc1 | 11:37 |
| opendevreview | Merged openstack/nova master: cpu: interfaces for managing state and governor https://review.opendev.org/c/openstack/nova/+/868236 | 12:25 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Cleanup old resize instances dir before resize https://review.opendev.org/c/openstack/nova/+/864730 | 13:50 |
| *** dasm|off is now known as dasm | 14:01 | |
| bauzas | gibi: another interesting usual suspect kinda related https://ae59d1e8526fa7671728-240e4b572b6f89b26c1b0e70b1c00c17.ssl.cf1.rackcdn.com/872413/3/check/nova-multi-cell/5e89e48/testr_results.html | 14:09 |
| bauzas | somethink looks wrong to me with the userplane | 14:09 |
| bauzas | this time, we got a lease directly | 14:09 |
| bauzas | but when adding the route, it failed | 14:09 |
| bauzas | and when calling the metadata API, we got a failure | 14:10 |
| bauzas | gibi: oh, ralonsoh did an update on the udhcpc bug https://review.opendev.org/c/openstack/neutron/+/871272 | 14:25 |
| bauzas | I guess we need to update our zuul config, lemme check | 14:26 |
| gibi | bauzas: I don't know if the root of this is the warning in the route add, or that just a red herring and we have a failure in metadata request handling in neutron or in nova | 14:26 |
| bauzas | the 'route add default gw' command failed | 14:26 |
| gibi | with a WARN :D | 14:27 |
| bauzas | so basically there is no default route set | 14:27 |
| sean-k-mooney | bauzas: that happens somethimes | 14:27 |
| sean-k-mooney | its normal | 14:27 |
| bauzas | that's why I guess the call to the metadata API is failing | 14:27 |
| bauzas | since the route to the network isn't told | 14:27 |
| sean-k-mooney | adding the default route often fails because its already there | 14:27 |
| bauzas | sean-k-mooney: I'm just checking the 'normality' in logsearch | 14:28 |
| bauzas | route: SIOCADDRT: File exists WARN: failed: route add -net "0.0.0.0/0" gw "10.1.0.1" | 14:28 |
| sean-k-mooney | i litrally have been seeing that for years | 14:28 |
| bauzas | technically there is indeed a default route set | 14:28 |
| bauzas | because of the 'file exists' | 14:28 |
| sean-k-mooney | i dont think this is related | 14:29 |
| sean-k-mooney | as i said im used to seeing htat warning. its possible however i think its unlikely | 14:30 |
| bauzas | well, OK | 14:30 |
| bauzas | I already spent my month investigating | 14:31 |
| bauzas | and we still have the udhcpc fix to merge, working on it | 14:31 |
| opendevreview | Sylvain Bauza proposed openstack/nova master: upgrade nova-next to use dhcpcd client w/ cirros-0.6.1 guests https://review.opendev.org/c/openstack/nova/+/873934 | 14:42 |
| bauzas | gibi: sean-k-mooney: ralonsoh: reviews appreciated ^ | 14:42 |
| ralonsoh | let me check | 14:44 |
| bauzas | ralonsoh: thanks a lot | 14:45 |
| gibi | bauzas: about the cirros image version change. Does our common jobs switched to 0.6.1 already? I.e. do we know that such cirros guest change is safe from the other tests perspective? | 15:01 |
| bauzas | gibi: that's a very good point | 15:01 |
| bauzas | I don't really want this change to be merged *now* | 15:02 |
| ralonsoh | we have been using this image for 1 month now | 15:02 |
| bauzas | but at least this is nova-next | 15:02 |
| ralonsoh | but maybe now this is not the time for nova, maybe in some weeks | 15:02 |
| bauzas | so we can somehow test cirros-6 in that job | 15:02 |
| bauzas | ralonsoh: tbh, I'm a bit reluctant to do any job changes while we are so close to FF as this is already a terrible month of CI failures but we somehow need to balance the benefits vs. the risks | 15:04 |
| ralonsoh | I agree | 15:04 |
| ralonsoh | maybe it should be better to avoid this change for now | 15:05 |
| gibi | bauzas: if neutron using this image for a month now then maybe we can take the risk and merge it in nova-next too | 15:05 |
| gibi | it is not like we would be the first to switch then | 15:05 |
| *** d34dh0r53 is now known as d34dh0r5- | 15:24 | |
| *** d34dh0r5- is now known as d34dh0r53 | 15:27 | |
| *** d34dh0r53 is now known as blarnath | 15:28 | |
| *** blarnath is now known as d34dh0r53 | 15:28 | |
| opendevreview | Kashyap Chamarthy proposed openstack/nova stable/train: libvirt: At start-up rework compareCPU() usage with a workaround https://review.opendev.org/c/openstack/nova/+/873722 | 15:39 |
| bauzas | sean-k-mooney: if you wanna get some status on the accepted blueprints https://etherpad.opendev.org/p/nova-antelope-blueprint-status | 17:17 |
| sean-k-mooney | thanks | 17:18 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/victoria: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/845754 | 17:18 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Test aborting queued live migration https://review.opendev.org/c/openstack/nova/+/873575 | 17:21 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Add functional tests to reproduce bug #1960412 https://review.opendev.org/c/openstack/nova/+/873576 | 17:21 |
| bauzas | gmann: every morning, I'm opening https://review.opendev.org/c/openstack/nova/+/864594 to check its status, I guess we gonna defer it to Bobcat unfortunately ? | 17:21 |
| sean-k-mooney | bauzas: https://pypi.org/project/python-novaclient/#history so is that still broken | 17:23 |
| bauzas | yes | 17:23 |
| bauzas | the requirements gate is mostly broken | 17:23 |
| bauzas | oh the novaclient | 17:23 |
| bauzas | no, we haven't released yet a client patch | 17:24 |
| bauzas | elodilles: planning to propose a releases patch for novaclient soon or want me to do it ? | 17:25 |
| sean-k-mooney | ok so nova-client is blocking the patch to osc | 17:29 |
| sean-k-mooney | so i tough we were going to do a release last week i guess not | 17:29 |
| sean-k-mooney | bauzas: in relation to https://review.opendev.org/c/openstack/python-openstackclient/+/872420 | 17:30 |
| opendevreview | Alexey Stupnikov proposed openstack/nova stable/ussuri: Clean up when queued live migration aborted https://review.opendev.org/c/openstack/nova/+/873577 | 17:30 |
| bauzas | no, last week was a nonclient lib featurefreeze | 17:30 |
| sean-k-mooney | yep | 17:30 |
| bauzas | and we just merged 2.95 support in novaclient yesterday :) | 17:30 |
| sean-k-mooney | but i tought wee were also going to do a release of novaclinet to unblock artom patch | 17:30 |
| bauzas | we did | 17:31 |
| sean-k-mooney | oh ok what does that do | 17:31 |
| bauzas | and it was merged yesterday | 17:31 |
| bauzas | now we only have OSC patch up | 17:31 |
| sean-k-mooney | oh 2.95 is the evacuate one | 17:32 |
| sean-k-mooney | oh right that has been blocked in ci | 17:32 |
| sean-k-mooney | ok that makes sense | 17:32 |
| bauzas | iirc, 2.94 and 2.95 were noop on the client side | 17:32 |
| bauzas | this is just for saying that novaclient actually supports those microversion calls, that's it | 17:33 |
| sean-k-mooney | its adding help text and a devstack functional test | 17:36 |
| sean-k-mooney | but its not strictly required | 17:36 |
| bauzas | the OSC patch ? ya | 17:36 |
| sean-k-mooney | yep | 17:36 |
| bauzas | and I think sahid abandoned his OSC patch | 17:36 |
| bauzas | since no really support was required | 17:37 |
| sean-k-mooney | i asked artom to add docs saying that as of microverison ... fqdns are supported | 17:37 |
| bauzas | but that can be addressed after ff | 17:37 |
| sean-k-mooney | if the osc team is fine wiht that yes | 17:37 |
| sean-k-mooney | as i said its just help text really | 17:38 |
| bauzas | yeah it's out of our scope anyway | 17:38 |
| bauzas | fwiw, if someone tells me gate is great, I just send that https://review.opendev.org/c/openstack/nova/+/872413 | 17:39 |
| bauzas | like I already said, I'm no longer a core reviewer | 17:39 |
| bauzas | my name should be "core rechecker" | 17:39 |
| bauzas | https://zuul.openstack.org/status#openstack/nova | 17:40 |
| bauzas | sean-k-mooney: -1d https://review.opendev.org/c/openstack/releases/+/873537 | 17:44 |
| bauzas | (and +1d https://review.opendev.org/c/openstack/releases/+/873515 ) | 17:45 |
| gmann | bauzas: ok, agree to move 864594 to bobcat. do not want to rush at last movement. | 17:45 |
| bauzas | gmann: ack and thanks for your dedication | 17:46 |
| sean-k-mooney | bauzas: im ok with the feature bump for novaclient instead of bugfix | 17:46 |
| sean-k-mooney | so makes sense | 17:46 |
| sean-k-mooney | osc-placement also makes sesne | 17:47 |
| spatel | sean-k-mooney quick question, I have ceph storage backed openstack today ceph went down because of power | 17:55 |
| spatel | now ceph is back but when i start VMs its getting stuck in boot script no available boot disk | 17:55 |
| spatel | SeaBIOS POST script getting that message. | 17:56 |
| spatel | logs are clean no errors related ceph in nova logs | 17:56 |
| spatel | it stuck here - https://paste.opendev.org/show/bPcyI5UPi0PWoN7iUX2J/ | 17:57 |
| spatel | I found this when searching - https://bugs.launchpad.net/kolla/+bug/1688496 | 17:59 |
| elodilles | bauzas: ok, i see you have found the generated release patches :) | 18:30 |
| elodilles | bauzas: btw, requirements gate fix has landed | 18:30 |
| elodilles | i've rechecked the os-traits. now we have to wait for the ci jobs to finish | 18:32 |
| mnaser | question: do you have to enable the new scope enforcement for the reader role to work? | 19:28 |
| opendevreview | Merged openstack/nova master: Factor out a mixin class for candidate aware filters https://review.opendev.org/c/openstack/nova/+/854929 | 19:44 |
| gmann | mnaser: we enabled the scope/new-defaults by default which enforce reader role by default https://review.opendev.org/c/openstack/nova/+/866218 | 19:47 |
| gmann | mnaser: it is on master which is not yet released. for older release yes you need to enable these two flags in nova.conf to enforce reader role . [oslo_policy] enforce_scope = True enforce_new_defaults = True | 19:48 |
| mnaser | gmann: so it’s not possible to just have enforce new defaults only without scope? I’m mostly worried about Horizon :) | 19:52 |
| gmann | mnaser: scope for all the policy rule is 'project' now so enabling the scope is to reject the system scope token at early stage with 403. existing project scope token keep working whether scope is enabled or disabled | 19:54 |
| gmann | but yes, you can keep only new defaults enable which will work fine unless system scoped token is used | 19:55 |
| gmann | with scope disabled system scope token may fail (for project level resources) later on as they do not have project id | 19:56 |
| gmann | I think horizon is using project scoped token (they have not changed it for srabc) so scope enable/disable should not impact there | 19:58 |
| gmann | kolla was the one who were using system scope for nova which is now fixed https://review.opendev.org/c/openstack/kolla-ansible/+/870879 | 19:59 |
| gmann | other than that no body reported if new defaults or scope enable has impacted them | 20:00 |
| mnaser | gmann: oh I see, so as of now, as long as we don’t have custom policies, we should be able to flip it on? | 20:10 |
| gmann | mnaser: yeah | 20:13 |
| mnaser | gmann: ok we’ll flip that and see how it goes! Thank you. | 20:13 |
| *** nicolasbock_ is now known as nicolasbock | 20:47 | |
| *** dasm is now known as dasm|off | 21:56 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!