| opendevreview | YuehuiLei proposed openstack/cinderlib master: Update python testing as per zed cycle teting runtime https://review.opendev.org/c/openstack/cinderlib/+/845881 | 06:34 |
|---|---|---|
| opendevreview | Peter Penchev proposed openstack/cinder master: Send the correct location URI to the Glance v2 API https://review.opendev.org/c/openstack/cinder/+/845178 | 09:01 |
| opendevreview | Gorka Eguileor proposed openstack/cinder master: Revert "Cleanup code duplication in cinder.cmd.backup module" https://review.opendev.org/c/openstack/cinder/+/845806 | 09:18 |
| opendevreview | Gorka Eguileor proposed openstack/cinder master: Doc: Mention Skip Level upgrades https://review.opendev.org/c/openstack/cinder/+/830283 | 09:23 |
| *** tosky_ is now known as tosky | 09:42 | |
| Roamer` | at the May 25th meeting I brought up an issue with Cinder refusing to clone a volume containing an image because (due to historical implementation details) it was reported as being in a different pool. I believe there was a consensus there that this would be handled best by adding a new driver capability; does "clone_across_pools" sound good as a capability name? ...maybe I should have asked | 09:48 |
| Roamer` | here before I registered https://blueprints.launchpad.net/cinder/+spec/clone-across-pools :) | 09:48 |
| *** dviroel|afk is now known as dviroel | 11:28 | |
| *** tosky_ is now known as tosky | 11:55 | |
| *** tosky__ is now known as tosky | 13:17 | |
| *** tosky is now known as Guest2169 | 13:29 | |
| *** tosky__ is now known as tosky | 13:29 | |
| opendevreview | Peter Penchev proposed openstack/cinder master: Send the correct location URI to the Glance v2 API https://review.opendev.org/c/openstack/cinder/+/845178 | 13:43 |
| opendevreview | Peter Penchev proposed openstack/cinder master: StorPool: drop _attach_volume() and _detach_volume() https://review.opendev.org/c/openstack/cinder/+/787296 | 13:43 |
| opendevreview | Peter Penchev proposed openstack/cinder master: StorPool driver: remove the obsolete backup_volume() https://review.opendev.org/c/openstack/cinder/+/803823 | 13:43 |
| opendevreview | Peter Penchev proposed openstack/cinder master: StorPool: create_cloned_volume() improvements https://review.opendev.org/c/openstack/cinder/+/843277 | 13:43 |
| opendevreview | Peter Penchev proposed openstack/cinder master: StorPool: really detach after copy to/from image. https://review.opendev.org/c/openstack/cinder/+/845990 | 13:43 |
| *** tosky__ is now known as tosky | 13:58 | |
| whoami-rajat | Cinder meeting in #openstack-meeting-alt at 1400 UTC | 13:58 |
| whoami-rajat | jungleboyj rosmaita smcginnis tosky whoami-rajat m5z e0ne geguileo eharney walshh_ jbernard sfernand enriquetaso hemna fabiooliveira yuval tobias-urdin | 13:58 |
| enriquetaso | thanks | 13:58 |
| jungleboyj | o/ | 13:58 |
| whoami-rajat | np | 13:58 |
| opendevreview | Merged openstack/os-brick master: Fix Unit Test failures due to host's ANA support https://review.opendev.org/c/openstack/os-brick/+/845149 | 14:02 |
| opendevreview | Sofia Enriquez proposed openstack/os-brick stable/yoga: Fix encryption symlink issues https://review.opendev.org/c/openstack/os-brick/+/845845 | 14:07 |
| opendevreview | Jean Pierre Roquesalane proposed openstack/cinder master: WIP: Add Cinder NFS driver for Dell EMC PowerStore https://review.opendev.org/c/openstack/cinder/+/797608 | 14:47 |
| HappyStacker1 | Hey cinder community | 14:54 |
| HappyStacker1 | Can someone explain to me why this patch doesn't make any progress? https://review.opendev.org/c/openstack/cinder/+/796286 We need a workflow +1 | 14:54 |
| enriquetaso | HappyStacker1 reviewed! | 14:57 |
| HappyStacker1 | thanks! | 14:58 |
| HappyStacker1 | we have some other in the same case | 15:01 |
| HappyStacker1 | like this one : https://review.opendev.org/c/openstack/cinder/+/821749 | 15:01 |
| HappyStacker1 | and this one https://review.opendev.org/c/openstack/cinder/+/816705 | 15:01 |
| HappyStacker1 | and the last one for now https://review.opendev.org/c/openstack/cinder/+/797333 | 15:02 |
| HappyStacker1 | all need workflow +1 | 15:02 |
| enriquetaso | OK, I'll review after the cinder bug meeting HappyStacker1 | 15:03 |
| HappyStacker1 | swwet! thanks enriquetaso | 15:03 |
| enriquetaso | #startmeeting cinder_bs | 15:03 |
| opendevmeet | Meeting started Wed Jun 15 15:03:56 2022 UTC and is due to finish in 60 minutes. The chair is enriquetaso. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:03 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:03 |
| opendevmeet | The meeting name has been set to 'cinder_bs' | 15:03 |
| enriquetaso | Welcome back to the Cinder Bug Meeting. | 15:04 |
| rosmaita | HappyStacker1: can't tell who you are from your nick, have you been reviewing patches yourself? | 15:04 |
| enriquetaso | good question :P | 15:04 |
| rosmaita | (i mean other peoples' patches) | 15:04 |
| enriquetaso | We have a few bugs today, so let us get started. | 15:04 |
| enriquetaso | Full list of bugs: | 15:04 |
| enriquetaso | #link http://lists.openstack.org/pipermail/openstack-discuss/2022-June/029056.html | 15:04 |
| HappyStacker1 | thats' jp from dell rosmalta | 15:04 |
| rosmaita | ok, thanks | 15:05 |
| enriquetaso | :) | 15:05 |
| enriquetaso | #topic [NFS/NetApp] No feedback from Nova when extending attached Volumes | 15:05 |
| enriquetaso | #link https://bugs.launchpad.net/cinder/+bug/1978294 | 15:05 |
| enriquetaso | In summary, n-cpu is able to extend an attached NFS volume by itself, but c-vol is preventing this from happening when it tries to perform `qemu-img resize`. | 15:05 |
| enriquetaso | This seems to be a problem not just for the NFS driver, since drivers using os-brick may fail, and Nova also handles extending the LUKS structure of attached encrypted volumes, which may also fail. | 15:06 |
| enriquetaso | This bug affects Quobyte, NetApp Cinder NFS drivers as far as I know | 15:06 |
| enriquetaso | No fix proposed to master yet. | 15:06 |
| rosmaita | "Lucio Seki, who first worked on [2] and [3], collected some thoughts on this in an etherpad [4] together with other potential issues of the changes." | 15:07 |
| rosmaita | i thought this sounded familiar | 15:07 |
| enriquetaso | yes, Lucio work on this as well | 15:08 |
| enriquetaso | #link https://review.opendev.org/c/openstack/cinder/+/739079 | 15:08 |
| enriquetaso | Etherpad link | 15:09 |
| enriquetaso | #link https://etherpad.opendev.org/p/fix-nfs-online-extend | 15:09 |
| enriquetaso | The original bug is #link https://bugs.launchpad.net/cinder/+bug/1870367 | 15:09 |
| enriquetaso | As far as I understood the new bug is for a related issue but we are going to track it on 1978294 | 15:10 |
| enriquetaso | Maybe kgube would like to add something to the bug report :) | 15:11 |
| enriquetaso | Currently, the solution we have for this is to disallow extension of attached volumes for NFS & Quobyte drivers | 15:12 |
| enriquetaso | #link https://review.opendev.org/c/openstack/cinder/+/725805/ | 15:12 |
| enriquetaso | OK, moving on | 15:13 |
| rosmaita | yeah, that was a short term "fix" | 15:13 |
| enriquetaso | do you think the second bug should be close as duplicated? rosmaita | 15:13 |
| rosmaita | not sure | 15:14 |
| enriquetaso | OK, i'll re review it and double check but looks like a separate bug | 15:14 |
| rosmaita | i agree | 15:15 |
| enriquetaso | #topic Fails to extend in-use (non LUKS v1) encrypted volumes | 15:15 |
| enriquetaso | #link https://bugs.launchpad.net/os-brick/+bug/1967157 | 15:15 |
| enriquetaso | Summary: Patch fixing bug #1861071 resolved the issue of extending LUKS v1 volumes when nova connects them via libvirt instead of through os-brick, but nova side still fails to extend in-use volumes when they don't go through libvirt (i.e., LUKS v2). | 15:15 |
| enriquetaso | geguileo, worked on the nova part of the bug that appears as Fix Released. | 15:15 |
| enriquetaso | now we have a cinder patch for this | 15:16 |
| enriquetaso | Fix proposed to master | 15:16 |
| enriquetaso | #link https://review.opendev.org/c/openstack/os-brick/+/836059 | 15:16 |
| enriquetaso | OK, feel free to review ^ | 15:18 |
| enriquetaso | moving on | 15:18 |
| enriquetaso | #topic Old-style Glance location URI sent when uploading a volume to an image | 15:18 |
| *** dviroel is now known as dviroel|lunch | 15:18 | |
| enriquetaso | #link https://bugs.launchpad.net/cinder/+bug/1978020 | 15:18 |
| enriquetaso | whoami-rajat, discussed it with the reporter | 15:19 |
| enriquetaso | Fix proposed to master | 15:19 |
| enriquetaso | #link https://review.opendev.org/c/openstack/cinder/+/845178 | 15:19 |
| enriquetaso | So, feel free to review it as well ^ | 15:20 |
| rosmaita | that looks ok to me, i thought glance did a lazy-migrate of old-style URIs, but maybe that was before you could have multiple cinder glance_stores | 15:21 |
| enriquetaso | good to know | 15:22 |
| rosmaita | the only thing i want to check is what kind of validation we have around the store_id in a volume-type extra-specs | 15:22 |
| enriquetaso | mm, not sure about that | 15:22 |
| enriquetaso | maybe we need some coverage there | 15:22 |
| rosmaita | i think we have something, just can't remember what | 15:23 |
| rosmaita | i will look into it and leave a comment on the patch | 15:23 |
| enriquetaso | thank you rosmaita ! | 15:23 |
| enriquetaso | OK, last two bugs: | 15:23 |
| enriquetaso | #topic cinder-backup context.message_action is None on errors | 15:23 |
| enriquetaso | Not sure about this one | 15:23 |
| enriquetaso | #link https://bugs.launchpad.net/cinder/+bug/1978729 | 15:24 |
| enriquetaso | If a driver raises an exception another exception is thrown from https://opendev.org/openstack/cinder/src/commit/3dfc519301aeb92ed4595788450b57e0be74db12/cinder/message/api.py#L104 because AFAICT the action is None. (also checked above https://opendev.org/openstack/cinder/src/commit/3dfc519301aeb92ed4595788450b57e0be74db12/cinder/message/api.py#L101 for setting action_id. | 15:24 |
| enriquetaso | looks like valid and the reporter added some logs to the case | 15:25 |
| enriquetaso | I'll try to reproduce and try the possible solutions | 15:26 |
| enriquetaso | #action (enriquetaso): try the fixes for ^ | 15:26 |
| enriquetaso | OK, last one! I'm running out of time | 15:26 |
| enriquetaso | #topic [IBM Storwize] Optimize SSH calls in create replicated volume | 15:27 |
| enriquetaso | #link https://bugs.launchpad.net/cinder/+bug/1978290 | 15:27 |
| enriquetaso | No fix proposed to master yet but probably IBM will proposed a fix :) | 15:27 |
| enriquetaso | OK, that's all I have for today's meeting | 15:27 |
| enriquetaso | thanks rosmaita | 15:28 |
| enriquetaso | thanks everyone | 15:28 |
| enriquetaso | #endmeeting | 15:28 |
| opendevmeet | Meeting ended Wed Jun 15 15:28:24 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:28 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/cinder_bs/2022/cinder_bs.2022-06-15-15.03.html | 15:28 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/cinder_bs/2022/cinder_bs.2022-06-15-15.03.txt | 15:28 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/cinder_bs/2022/cinder_bs.2022-06-15-15.03.log.html | 15:28 |
| rosmaita | thanks enriquetaso! | 15:28 |
| whoami-rajat | enriquetaso, cool, thanks for taking a look | 15:48 |
| *** dviroel|lunch is now known as dviroel | 16:15 | |
| opendevreview | Gorka Eguileor proposed openstack/cinderlib master: Update Ceph version on CI job https://review.opendev.org/c/openstack/cinderlib/+/846034 | 16:38 |
| opendevreview | Merged openstack/cinder master: PowerMax Driver - Improve error handling around deletes https://review.opendev.org/c/openstack/cinder/+/796286 | 16:45 |
| opendevreview | Gorka Eguileor proposed openstack/cinder master: Fix cinderlib RBD keyring issue https://review.opendev.org/c/openstack/cinder/+/846054 | 18:09 |
| dansmith | geguileo: do you know about performance.json | 18:34 |
| dansmith | ? | 18:34 |
| geguileo | dansmith: no, no idea | 18:35 |
| dansmith | geguileo: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_80f/845805/2/check/tempest-full-py3/80f2af1/controller/logs/performance.json | 18:35 |
| dansmith | from your "reduce memory footprint" run | 18:35 |
| geguileo | ooooooooooh, nice!!! | 18:36 |
| geguileo | dansmith: I did a bunch of local test runs to ensure that this was actually better | 18:36 |
| geguileo | but it's great to be able to reference something at the gate | 18:36 |
| dansmith | geguileo: we've been generating that for a few weeks now, | 18:36 |
| dansmith | so you can compare that run to other runs with opensearch | 18:37 |
| dansmith | we're ideally going to get that graphed at some point, but at the moment it's just .. that | 18:37 |
| dansmith | I have a little tool locally which you can use to get this: https://termbin.com/kjvb | 18:38 |
| dansmith | i.e. the top offenders on memory | 18:38 |
| dansmith | https://termbin.com/v5n6k | 18:38 |
| dansmith | takes an url so you can just run it against jobs pretty easy.. --mem is the only thing it does right now, but that's the important one | 18:39 |
| geguileo | dansmith: thanks!! | 18:42 |
| HappyStacker1 | enriquetaso if you can take some time to review our patches, it'd be great! thanks in advance | 18:45 |
| *** dviroel is now known as dviroel|afk | 18:47 | |
| opendevreview | Jean Pierre Roquesalane proposed openstack/cinder master: PowerMax Driver -- Add support for Unisphere for PowerMax 10.0 https://review.opendev.org/c/openstack/cinder/+/844563 | 18:50 |
| geguileo | dansmith: mmmm, I don't know if it's my patch's fault, but n-cpu consumes a lot more memory on that run... | 18:51 |
| dansmith | oh yeah ~660mb, that's a lot | 18:52 |
| dansmith | things vary per service quite a bit from run to run, but that's way higher than I've seen | 18:52 |
| geguileo | it was 390 on another run without that patch | 18:53 |
| dansmith | maybe a couple rechecks are in order | 18:53 |
| geguileo | I'll investigate it tomorrow | 18:53 |
| dansmith | so, there's one more thing: | 18:53 |
| dansmith | https://review.opendev.org/c/openstack/devstack/+/838947 | 18:53 |
| dansmith | this adds something that compares two files and tries to highlight surges | 18:53 |
| dansmith | it's a lot harder for even countable things than I was expecting, not sure where all the variability comes from | 18:53 |
| dansmith | however, you could use that grok_performance.py there to compare multiple runs maybe | 18:54 |
| geguileo | dansmith: from what I've been seeing the variability is caused by memory fragmentation | 18:55 |
| geguileo | both in the malloc arenas as well as in the Python internal arenas | 18:56 |
| dansmith | in memory yeah, I mean countable things like api requests and db accesses | 18:56 |
| dansmith | geguileo: we also cycle workers on some of the services, so we'll inflate, then cycle and drop a bunch and the memory at the end of a tempest run depends on how many times we did that | 18:56 |
| dansmith | uwsgi has a max_requests sort of thing I think | 18:56 |
| geguileo | yeah, in my tests for cinder-backup the high water mark memory issue gets fixed | 18:57 |
| dansmith | yeah that's awesome because that sucker was big :) | 18:57 |
| geguileo | so I did 10 concurrent processes each doing 200 backup then restore cycles | 18:57 |
| dansmith | presumably c-bak must have been reading decent-sized chunks of data into memory to even cause that though right? | 18:58 |
| geguileo | and without the patch the high water memory mark was 2GB (it would not go down from there even idle) | 18:58 |
| geguileo | with the patch it went down immediately to 140MB | 18:58 |
| dansmith | yeah, that's awesome | 18:58 |
| geguileo | dansmith: yes, reading chunks and they don't get immediately garbage collected | 18:58 |
| dansmith | now if you can just avoid making nova huge :P | 18:59 |
| geguileo | rofl rofl | 18:59 |
| dansmith | right, okay | 18:59 |
| geguileo | I'll look into it tomorrow | 18:59 |
| dansmith | cool, thanks for doing this | 18:59 |
| dansmith | not enough people care about our waistline :) | 18:59 |
| geguileo | because there's a second run on that patch going on now, so I'll be able to compare both | 18:59 |
| dansmith | ++ | 18:59 |
| geguileo | lol | 18:59 |
| geguileo | thanks for all the good tools you've just given me | 19:00 |
| dansmith | np :) | 19:00 |
| enriquetaso | rosmaita, geguileo feel free to review this backport https://review.opendev.org/c/openstack/os-brick/+/845845 :) | 20:34 |
| enriquetaso | we need it in wallaby | 20:34 |
| opendevreview | Sofia Enriquez proposed openstack/cinder master: RBD: Flattening of child volumes during deletion https://review.opendev.org/c/openstack/cinder/+/835384 | 21:22 |
| opendevreview | Peter Penchev proposed openstack/cinder master: StorPool: really detach after copy to/from image. https://review.opendev.org/c/openstack/cinder/+/845990 | 23:37 |
| opendevreview | Peter Penchev proposed openstack/cinder master: StorPool driver: remove the obsolete backup_volume() https://review.opendev.org/c/openstack/cinder/+/803823 | 23:37 |
| opendevreview | Peter Penchev proposed openstack/cinder master: StorPool: create_cloned_volume() improvements https://review.opendev.org/c/openstack/cinder/+/843277 | 23:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!