| opendevreview | yangzhipeng proposed openstack/nova master: when evacuate is performing, and restart compute node, if get instance info early, the instance state is not latest. this will reset instance task to error incorrectly, so refresh instance when modify instance state. https://review.opendev.org/c/openstack/nova/+/866960 | 06:46 |
|---|---|---|
| opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. https://review.opendev.org/c/openstack/nova/+/865362 | 06:53 |
| opendevreview | yzp proposed openstack/nova master: Refresh instance when init instance in rebuilding https://review.opendev.org/c/openstack/nova/+/866960 | 07:54 |
| opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. Signed-off-by: yangzhipeng <yangzhipeng@cmss.chinamobile.com> https://review.opendev.org/c/openstack/nova/+/865362 | 07:58 |
| opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. Signed-off-by: yangzhipeng <yangzhipeng@cmss.chinamobile.com> https://review.opendev.org/c/openstack/nova/+/865362 | 08:00 |
| opendevreview | yzp proposed openstack/nova master: Remove all tag if instance has beed hard deleted. https://review.opendev.org/c/openstack/nova/+/865362 | 08:03 |
| opendevreview | norman shen proposed openstack/nova master: Skip deleting instance info for same host migration https://review.opendev.org/c/openstack/nova/+/866521 | 08:04 |
| opendevreview | yzp proposed openstack/nova master: Refresh instance when init instance in rebuilding https://review.opendev.org/c/openstack/nova/+/866960 | 09:48 |
| opendevreview | yzp proposed openstack/nova master: Refresh instance when init instance. https://review.opendev.org/c/openstack/nova/+/866960 | 10:27 |
| *** dasm|off is now known as dasm | 13:00 | |
| gibi | do we have a blocked nova gate? https://zuul.opendev.org/t/openstack/build/32c3c4a56eec4d7eac956a629abdee362 | 13:24 |
| gibi | it looks we have constant failure in https://zuul.opendev.org/t/openstack/builds?job_name=nova-ceph-multistore&project=openstack/nova | 13:24 |
| gibi | since yesterday | 13:25 |
| gibi | also in https://zuul.opendev.org/t/openstack/builds?job_name=nova-grenade-multinode&project=openstack/nova | 13:25 |
| gibi | "venv: failed with tempest is not allowed, use allowlist_externals to allow it" | 13:26 |
| sean-k-mooney | that sound like we are not installing tempet in the venv properly | 13:47 |
| gibi | yeah | 13:51 |
| gibi | grenade fails a bit differently but maybe related too | 13:51 |
| gibi | this seems related https://review.opendev.org/c/openstack/tempest/+/865314/1/tox.ini | 13:52 |
| gibi | but merged couple weeks ago | 13:53 |
| sean-k-mooney | ya so there was a mail about that | 13:53 |
| sean-k-mooney | the allowlist_external=* is problematic | 13:53 |
| sean-k-mooney | gibi: https://lists.openstack.org/pipermail/openstack-discuss/2022-November/031343.html | 13:57 |
| sean-k-mooney | clarkb: ^ so that tempest fix may have broken grenade? | 13:59 |
| sean-k-mooney | gibi: my guess is the tempest venv is not being recreated | 14:02 |
| sean-k-mooney | so on zed it would have had allowlist_external=* and used global pip | 14:03 |
| sean-k-mooney | as a result tempest was likely insalled outside of the venv? | 14:03 |
| sean-k-mooney | and then with the master tox file it would have failed? | 14:03 |
| gibi | it fails in nova-ceph-multistore too so it is not grenade specific | 14:04 |
| sean-k-mooney | not sure if that actully what is happening as i tough we used master tempetst regradels of the branch but i would guess its soemthign like that | 14:04 |
| sean-k-mooney | oh ok | 14:04 |
| sean-k-mooney | so may somethign related to our jobs | 14:04 |
| frickler | yes, that allowlist needs to include tempest | 14:05 |
| * frickler can do a patch | 14:06 | |
| sean-k-mooney | frickler: im not sure it should | 14:06 |
| sean-k-mooney | tempest shoudl be installed in teh venv | 14:06 |
| sean-k-mooney | we shoudl not be using tempest form teh host right? | 14:06 |
| sean-k-mooney | we install tempest in a venv because it is often not the same version of openstack | 14:06 |
| gibi | this is where we fail https://github.com/openstack/devstack/blob/master/lib/tempest#L731 | 14:06 |
| sean-k-mooney | and we do it that way to avoid the possibel depency conflcit that woudl create | 14:06 |
| sean-k-mooney | so tempest should __not__ be in the allowlist | 14:07 |
| sean-k-mooney | ya so that shoudl be using tempst form the venv | 14:07 |
| sean-k-mooney | shoudl it be tox -evenv-tempest https://github.com/openstack/devstack/blob/master/lib/tempest#L709 | 14:08 |
| sean-k-mooney | instead of venv | 14:08 |
| sean-k-mooney | tox -evenv-tempest seams to be what we use on the other lines | 14:08 |
| gibi | hm, good point | 14:08 |
| gibi | that seems like the wrong env | 14:09 |
| gibi | I will push a patch | 14:09 |
| frickler | oh, cool, so this found an actual bug, nice | 14:09 |
| ralonsoh | sean-k-mooney, thanks! I saw this issue in out CI too | 14:11 |
| ralonsoh | our* | 14:11 |
| gibi | devstack fix is up https://review.opendev.org/c/openstack/devstack/+/866997 | 14:14 |
| opendevreview | Konrad Gube proposed openstack/nova-specs master: Use extend volume completion action https://review.opendev.org/c/openstack/nova-specs/+/855490 | 14:28 |
| opendevreview | Bence Romsics proposed openstack/nova master: doc: soft delete and shadow tables https://review.opendev.org/c/openstack/nova/+/867001 | 14:44 |
| opendevreview | Bence Romsics proposed openstack/nova master: doc: soft delete and shadow tables https://review.opendev.org/c/openstack/nova/+/867001 | 15:06 |
| elendrys | Hello here, is this channel open to nova related questions or is it dedicated to developers ? | 15:19 |
| frickler | elendrys: this is a bit of a grey area, #openstack is what is meant for general questions, but the chance to get an answer there is low, so you might as well try here directly | 15:45 |
| fungi | also the openstack-discuss@lists.openstack.org mailing list has a much higher chance of getting you an answer eventually | 15:47 |
| elendrys | Ok thank you | 15:54 |
| sean-k-mooney | elendrys: did you have a question in general | 15:54 |
| sean-k-mooney | while the channel is not for supprot we will try an point you in the right direction in genreal | 15:55 |
| elendrys | Yes | 15:55 |
| elendrys | I have servers with cinder volumes, backed in PureStorage array over iscsi | 15:56 |
| sean-k-mooney | ok so the vm is using host mount iscsi block devices passthroug to the vms | 15:57 |
| elendrys | When i resize volumes attached to running servers, I can see in nova's log that it rescans the array to get the new size when cinder notifies | 15:57 |
| elendrys | But in the vm, the size always have 1 notification late size | 15:57 |
| elendrys | I mean, a 10Gb vol expended to 20 stills show as 10, but if I extend to 30, it shows 20 | 15:58 |
| elendrys | I can't get why (probably libvirt) it is missing something | 15:58 |
| sean-k-mooney | hum interesting. | 15:58 |
| sean-k-mooney | am im not sure this is libvirt/qemu related | 15:59 |
| elendrys | If I stop then start the server it shows 30 as intended | 15:59 |
| sean-k-mooney | i woudl suppect this is more likely to be related to iscsid or simialr | 15:59 |
| sean-k-mooney | elendrys: did you check on the host if the host kernel sees the correct volume size | 15:59 |
| sean-k-mooney | you should be able to find the dev path form the vm xml and check with lsblk or a similar too to see if it sees the correct current size | 16:00 |
| sean-k-mooney | if the hosts sees the correct size then the issue is likely at the qemu level | 16:01 |
| sean-k-mooney | elendrys: i dont think libvirt is invovled for voulme resize in this case | 16:01 |
| elendrys | It does | 16:02 |
| sean-k-mooney | im wondering is there anything you can do at the gues os level | 16:05 |
| elendrys | It is queued somewhere by nova | 16:05 |
| elendrys | I tried to force scsci bus scan but it doesn't change | 16:06 |
| elendrys | but when I extend an extra time, the previous size is seen instantaneously by the guest | 16:06 |
| elendrys | it's just 1 extend late | 16:07 |
| *** ministry is now known as __ministry | 16:07 | |
| sean-k-mooney | well nova as far as i am aware will not need to notify qemu but perhaps we do | 16:17 |
| clarkb | and those are sizes reported by lsblk? | 16:20 |
| sean-k-mooney | unfortunetly i dont currently have the low level details loaded in my brain | 16:20 |
| sean-k-mooney | clarkb: ya | 16:20 |
| sean-k-mooney | the size of the block device repoted to the guest | 16:20 |
| sean-k-mooney | which should eventualy be show in lsblk on the host and guest | 16:21 |
| clarkb | ya I'm wondering if the guest does an explicit check via lsblk if it sees correct info and the only problem is in propogating the not on demand info | 16:21 |
| elendrys | If it is better I can open the discussion on the mailing list | 16:25 |
| kgube_ | the size seems to come from os-brick LinuxSCSI which calls "blockdev --getsize64" after a rescan | 16:26 |
| kgube_ | so maybe the rescan in LinuxSCSI.extend_volume is not working | 16:31 |
| kgube_ | elendrys, have you tried it with debug logging? | 16:32 |
| elendrys | nope, but I can do that quickly | 16:33 |
| kgube_ | there should be "Starting size: " and "volume size after scsi device rescan " in the log | 16:35 |
| elendrys | Starting size: 76235669504 extend_volume /usr/lib/python3/dist-packages/os_brick/initiator/linuxscsi.py:555 | 16:36 |
| elendrys | volume size after scsi device rescan 80530636800 extend_volume | 16:36 |
| elendrys | hem | 16:39 |
| elendrys | - /dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de has shown up. wait_for_path | 16:39 |
| elendrys | mpath(/dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de) current size 76235669504 | 16:39 |
| elendrys | mpath(/dev/disk/by-id/dm-uuid-mpath-3624a9370cebbf98ad4534b81000386de) new size 76235669504 | 16:39 |
| elendrys | Extend iSCSI Volume /dev/dm-28; new_size=76235669504 extend_volume | 16:40 |
| elendrys | Resizing target device /dev/dm-28 to 76235669504 _resize_attached_volume | 16:40 |
| elendrys | I think that it rely on multipath but it is not refreshed quickly enough | 16:40 |
| kgube_ | yeah that seems to come from the multipath code | 16:43 |
| kgube_ | https://github.com/openstack/os-brick/blob/master/os_brick/initiator/linuxscsi.py#L639 | 16:43 |
| elendrys | it explains why I see the previous size in the guest | 16:45 |
| kgube_ | I don't know enough about scsi to have any idea what to do about this, though | 16:47 |
| elendrys | Should I fill an issue on the tracker or just ask the mailing list ? | 16:47 |
| elendrys | Thank you for your time :) | 16:48 |
| kgube_ | hm, I'm not sure. I'm very new to Nova dev | 16:53 |
| elodilles | sean-k-mooney: sorry for pinging, but if you have some free minutes to review this old stable backport, that would be awesome: https://review.opendev.org/c/openstack/nova/+/764504 | 16:55 |
| elodilles | (it has a lot of +1 on it) | 16:55 |
| kgube_ | elendrys, maybe try the mailing list first there is also a lot of openstack users reading it | 16:57 |
| kgube_ | elendrys, maybe someone has seen this before and found a workaround | 16:58 |
| sean-k-mooney | oh ya that sure happy to see that merge | 17:08 |
| sean-k-mooney | elodilles: give me 5 mins to finsih somehting and ill review it then | 17:09 |
| elodilles | thanks in advance o/ | 17:12 |
| sean-k-mooney | elodilles: once that merges im happy to continue moving this back to train so ill keep an eye out for that over then few days | 17:27 |
| sean-k-mooney | elodilles: how are the stabel gates at the moment have they been affected by the tempest tox issue on master or the tox 4.0 issues | 17:28 |
| elodilles | sean-k-mooney: thanks, i'll try to keep an eye on them, too :) good question about the gate, I haven't checked that yet :S but i guess we will see :/ | 17:32 |
| *** dasm is now known as dasm|off | 21:50 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!