| opendevreview | Mounika Sreeram proposed openstack/cinder master: [SVf] Delete/Resize issue in reverse replication https://review.opendev.org/c/openstack/cinder/+/831776 | 05:30 |
|---|---|---|
| *** bhagyashris_ is now known as bhagyashris | 05:52 | |
| rubasov | whoami-rajat: good morning, I'm back today, when you have time, please let me know how you would like to continue yesterday's debugging | 07:35 |
| whoami-rajat | rubasov, good morning, sure we can continue | 07:36 |
| whoami-rajat | not sure where we left though | 07:36 |
| whoami-rajat | IIRC you're facing issues when saving multiple images? | 07:37 |
| rubasov | that with two parallel image saves we still seem to have someting similar as in bug https://bugs.launchpad.net/cinder/+bug/1940466 | 07:37 |
| rubasov | though the error message in my reproduction is a bit different, this: cinderclient.exceptions.BadRequest: Invalid volume: Volume e5ad5acd-c20f-4c9e-a6c5-0771af3056ac status must be available or in-use or downloading to reserve, but the current status is reserved. (HTTP 400) | 07:38 |
| rubasov | we tested too, that parallel image creates do not trigger the same error | 07:39 |
| rubasov | the glance image saved is backed by a multiattach volume type | 07:40 |
| whoami-rajat | strange as we also have a retry mechanism for this case | 07:41 |
| whoami-rajat | https://github.com/openstack/glance_store/blob/77919e15d281e908de7b687e2af927b59d3dc8a8/glance_store/common/cinder_utils.py#L112-L114 | 07:41 |
| whoami-rajat | do you see the above errors multiple times in your g-api logs? ^^ | 07:41 |
| rubasov | and we tried adding 'cinder_volume_type = lvmdriver-1-multiattach' to glance-api.conf but that does not seemed to have changed the behavior | 07:41 |
| whoami-rajat | rubasov, IIUC, after changing the conf file, you created a new image and trying to download THAT NEW IMAGE in parallel right? | 07:42 |
| whoami-rajat | or are you trying that on an old image? (before the conf changes) | 07:42 |
| whoami-rajat | s/that/download | 07:43 |
| rubasov | I did not create a new image, but I'll try that now | 07:46 |
| rubasov | by the way I just double checked and the 'status must be available ...' error message is present only once in the logs, and then a few seconds later I have this: FileNotFoundError: [Errno 2] No such file or directory: '/opt/stack/data/glance/cache/incomplete/88d4a9ac-1529-4f3d-8b18-a9eef8a93155' -> '/opt/stack/data/glance/cache/88d4a9ac-1529-4f3d-8b18-a9eef8a93155' | 07:47 |
| rubasov | but this was still with the old image | 07:47 |
| whoami-rajat | rubasov, sure, I assume you've the config option already changed, just create a new image and try downloading that in parallel? | 07:47 |
| rubasov | testing it... | 07:48 |
| whoami-rajat | rubasov, hmm, that seems to be a glance specific issue, i think we should continue the discussion in glance channel | 07:48 |
| whoami-rajat | sadly i don't see the usual people in glance channel... | 07:49 |
| lpetrut | hi, we'd need someone to take a look over this multipath resize bug fix: https://review.opendev.org/c/openstack/os-brick/+/762776 | 07:52 |
| rubasov | I double checked that I left yesterday the cinder_volume_type set to a multiattach type, uploaded a new image (openstack image create --public --disk-format qcow2 --file ~/cirros-0.5.2-x86_64-disk.img image0) and downloading this image twice in parallel, one image save still fails, while having the same error message in glance logs: status must be available or in-use or downloading to reserve, | 07:57 |
| rubasov | but the current status is reserved | 07:57 |
| whoami-rajat | rubasov, can you do ``cinder list --all`` and and check if the volume containing the image exists? or just dump the output in a paste https://paste.opendev.org/ | 08:03 |
| rubasov | whoami-rajat: I believe the volume got created properly, please see here: https://paste.opendev.org/show/b046N88xSrNLSx8HNzyo/ | 08:08 |
| whoami-rajat | rubasov, yep it did, I would require the glance logs for further debugging as I'm not sure what should be the reason, I tested this code with simultaneous request to create 10 volumes from same image and it worked fine | 08:13 |
| whoami-rajat | if you can provide logs for this particular request ID | 08:13 |
| rubasov | sure, will collect and share | 08:15 |
| whoami-rajat | thanks | 08:17 |
| rubasov | whoami-rajat: 127 kilobytes proved too big for pastebin, so I added the logs as an attachment in LP here: https://bugs.launchpad.net/cinder/+bug/1940466/comments/6 | 09:27 |
| rubasov | also I'm doing one more thing to double check my bug reproduction: in the vm I'm using I previously built a stable/victoria devstack and now I'm throwing this vm away and building a fresh one from master to make sure devstack/pip could not leave some victoria dependencies behind | 09:28 |
| rubasov | whoami-rajat: I built a master devstack in a fresh vm, in which I never built a victoria devstack before and the parallel image save still fails, so I think leftover dependencies cannot be the cause | 10:22 |
| rubasov | regarding your previous question about whether I created the image before or after the glance change, I realized that it may be relevant that I always tested with an image whose volume I retyped to a multiattach type for example: openstack volume set --type lvmdriver-1-multiattach "$( openstack volume list --all --user glance --name image-$( openstack image show cirros-0.5.2-x86_64-disk -f value | 10:24 |
| rubasov | -c id ) -f value -c ID )" | 10:24 |
| rubasov | I'll be away for a bit , be back around 13:00 utc | 10:37 |
| whoami-rajat | rubasov, oh, the image volume is not supposed to be retyped... you just set the conf option in glance and it automatically creates the right multiattach type volume | 10:39 |
| whoami-rajat | to store the image | 10:39 |
| rubasov | I mean I retyped it before I knew about the glance config option, but not after I uploaded new image with the changed glance config | 10:40 |
| rubasov | sorry for the confusion | 10:40 |
| whoami-rajat | ah ok | 10:41 |
| opendevreview | Rajat Dhasmana proposed openstack/cinder master: Update docstring for copy_image_to_volume method https://review.opendev.org/c/openstack/cinder/+/836769 | 10:56 |
| opendevreview | Rajat Dhasmana proposed openstack/cinder master: Add missing copy_image_to_encrypted_volume in interface https://review.opendev.org/c/openstack/cinder/+/836804 | 10:56 |
| opendevreview | Rajat Dhasmana proposed openstack/cinder master: Update docstring for copy_volume_to_image https://review.opendev.org/c/openstack/cinder/+/836822 | 10:56 |
| yuval | Guys, how do we test flows related to reboot and abrupt reboot - tempest have something for that? | 10:57 |
| opendevreview | Brian Rosmaita proposed openstack/cinder master: Add info about code coverage job https://review.opendev.org/c/openstack/cinder/+/836930 | 11:06 |
| rosmaita | dc_arun__kv: ^^ | 11:11 |
| *** dviroel|afk is now known as dviroel\ | 11:13 | |
| *** dviroel\ is now known as dviroel | 11:13 | |
| opendevreview | Harsh Ailani proposed openstack/cinder master: [SVf]:Fix retype failure for replication volume-type https://review.opendev.org/c/openstack/cinder/+/836944 | 11:38 |
| opendevreview | Harsh Ailani proposed openstack/cinder master: [SVf]:Fix retype failure for replication volume-type https://review.opendev.org/c/openstack/cinder/+/836944 | 12:11 |
| opendevreview | Alexander Malashenko proposed openstack/cinder master: Add check if size exist in model_update and update size in model_update https://review.opendev.org/c/openstack/cinder/+/836973 | 12:53 |
| whoami-rajat | Courtesy reminder Cinder PTG happening at https://bluejeans.com/556681290 | 12:58 |
| whoami-rajat | jungleboyj rosmaita smcginnis tosky whoami-rajat m5z e0ne geguileo eharney walshh_ jbernard sfernand enriquetaso hemna fabiooliveira yuval | 12:58 |
| fabiooliveira | thanks | 12:58 |
| whoami-rajat | np | 12:59 |
| tosky | coming on while | 13:22 |
| bingyanh__ | hi dear all can help to take a look at this cherry pick:https://review.opendev.org/c/openstack/cinder/+/830050 | 13:48 |
| *** dviroel is now known as dviroel|ruck | 14:14 | |
| whoami-rajat | rosmaita we are discussing Update on Image Encryption | 14:37 |
| rosmaita | whoami-rajat: ty, looks like i will miss it | 14:37 |
| whoami-rajat | ok, np | 14:37 |
| ade_lee | whoami-rajat, hey - thanks for putting fips topic on for tomorrow morning -- I do have a session planned with the manila folks on friday 13:55-14:25 UTC, can we move my topic to earlier in the day? (maybe second in the day) | 14:49 |
| ade_lee | or later .. | 14:50 |
| ade_lee | just concerned about overlap | 14:50 |
| whoami-rajat | ade_lee, sure, will ask Gorka (as he has the first topic) if he's fine moving it for later | 14:51 |
| ade_lee | whoami-rajat, thanks | 14:52 |
| ade_lee | whoami-rajat, I anticipate we won't take more than 15 mins most likely | 14:53 |
| whoami-rajat | ade_lee, cool, that helps with covering rest of the topics | 14:53 |
| ade_lee | whoami-rajat, of course, I don't know how verbose people are :) | 14:54 |
| whoami-rajat | VERY! but let's see | 14:55 |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:28 | |
| sean-k-mooney | looking at https://docs.openstack.org/api-ref/block-storage/v3/?expanded=show-a-volume-s-details-detail#show-a-volume-s-details= | 15:39 |
| sean-k-mooney | the host specific info could just be filtered based on project-admin vs project member | 15:40 |
| sean-k-mooney | for example "os-vol-host-attr:host": null, | 15:40 |
| sean-k-mooney | you could also remove the host info at a later micorversion and optionally expose it via a system admin api if required | 15:41 |
| sean-k-mooney | what was the acul concern that cinder had with this api with regards to rbac | 15:41 |
| sean-k-mooney | we have similar things in nova for servers for the host its resides on which are admin only | 15:42 |
| sean-k-mooney | and will onely be shown if you are a project_admin going forward | 15:42 |
| sean-k-mooney | since we redefiend project admin to be an operator acting on a project scoped resoures | 15:43 |
| sean-k-mooney | rosmaita: ^ that was for you by the way but not urgent just wondering what the context was | 15:48 |
| tosky | whoami-rajat: are we in the cinder or in the glance room now? | 15:52 |
| whoami-rajat | tosky, in cinder room | 15:52 |
| tosky | oh, I see now, thanks | 15:52 |
| whoami-rajat | should've cleared that before ... | 15:52 |
| whoami-rajat | glance cinder cross project happening in cinder meeting room at 1600 UTC jungleboyj rosmaita smcginnis tosky whoami-rajat m5z e0ne geguileo eharney walshh_ jbernard sfernand enriquetaso hemna fabiooliveira yuval | 15:56 |
| whoami-rajat | meeting room: https://bluejeans.com/556681290 | 15:56 |
| whoami-rajat | just doing it so people are aware ^^ | 15:56 |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:13 | |
| opendevreview | DataCore Software proposed openstack/cinder master: Reintroduce DataCore driver https://review.opendev.org/c/openstack/cinder/+/836996 | 16:21 |
| opendevreview | Francesco Pantano proposed openstack/devstack-plugin-ceph master: Deploy with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 | 16:28 |
| geguileo | rosmaita: thank you for the survey and the results, they were helpful to reach a decission on today's PTG :-) | 17:05 |
| rosmaita | geguileo: yw! | 17:06 |
| *** dviroel|ruck is now known as dviroel|afk | 20:39 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!