| opendevreview | 王昊 proposed openstack/cinder master: Support multiple filters together for volume time comparison filter https://review.opendev.org/c/openstack/cinder/+/740146 | 02:01 |
|---|---|---|
| opendevreview | Victoria Martinez de la Cruz proposed openstack/devstack-plugin-ceph master: Deploy with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 | 10:50 |
| opendevreview | Victoria Martinez de la Cruz proposed openstack/devstack-plugin-ceph master: Deploy with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/826484 | 10:51 |
| *** dviroel|out is now known as dviroel|ruck | 11:12 | |
| *** dviroel is now known as dviroel|ruck | 12:20 | |
| hemna | arg more capacity issues. | 13:47 |
| hemna | also looks like the volume manager doesn't update the allocated_capacity_gb for migrated volumes. | 13:48 |
| hemna | which is also another reason why that can go negative | 13:48 |
| hemna | migrate a volume from host a to b | 13:48 |
| hemna | delete the volume. | 13:48 |
| hemna | I'm seeing thick provisioned pools being over provisioned and I think this is one of the reasons it's happening. | 13:50 |
| hemna | also a side effect of the vmware driver allowing lazy creates | 13:51 |
| hemna | cinder asks it to create a 1TB volume, the driver says ok, but does nothing. | 13:51 |
| hemna | it creates the actual volume when someone wants to attach it. | 13:52 |
| hemna | since the scheduler makes decisions based upon the reported free space for thick provisioned volumes, you get over provisioned thick pool. :( | 13:52 |
| eharney | shouldn't the driver account for those lazily created volumes in its stats reporting? | 13:55 |
| hemna | It's not currently, and if it was kept in memory, a restart would lose that info | 13:55 |
| hemna | the vmware driver is a mess | 13:56 |
| hemna | and vmware behind it is a mess | 13:56 |
| hemna | the vmware driver would have to go through every volume created against it and see if there is a backing shadow vm associated with it, if and only if that volume hasn't been attached ever. | 13:57 |
| hemna | that would be crazy to do at startup | 13:57 |
| hemna | https://github.com/openstack/cinder/blob/9a8f86732131c4c5b4b647514496155598faf561/cinder/volume/drivers/vmware/vmdk.py#L485-L505 | 13:58 |
| hemna | that's all it does during lazy create | 13:58 |
| rosmaita | hemna: can you lead a discussion about the issues you are finding at the PTG? i think eharney is correct that some issues seem to be scheduler related but others seem more driver concerns | 14:07 |
| rosmaita | and like you point out, the vmware driver is kind of pathological | 14:07 |
| rosmaita | we'll need to get agreement on how this all should be fixed | 14:07 |
| hemna | sure | 14:07 |
| hemna | when is the PTG ? | 14:08 |
| rosmaita | april 4-8 | 14:08 |
| rosmaita | cinder is meeting tuesday-friday | 14:08 |
| hemna | ok do we still need to register for it and all that? | 14:08 |
| rosmaita | yeah, the usual deal | 14:08 |
| rosmaita | https://etherpad.opendev.org/p/zed-ptg-cinder-planning | 14:08 |
| * hemna registers | 14:09 | |
| hemna | ok thanks for the reminder | 14:10 |
| opendevreview | Brian Rosmaita proposed openstack/cinder master: Add yoga release note prelude https://review.opendev.org/c/openstack/cinder/+/833365 | 14:20 |
| rosmaita | whoami-rajat: abishop: let me know if i described your features correctly ^^ | 14:21 |
| abishop | rosmaita: ack, left a +1 for mine | 14:27 |
| whoami-rajat | rosmaita, i think it's worth mentioning that it's only applicable for ``available`` and ``error`` volumes for now (we would not want to highlight it for for reserved volumes until the nova part is merged) | 14:28 |
| whoami-rajat | but maybe that's more detail for a prelude so not sure ^ | 14:28 |
| rosmaita | whoami-rajat: yeah, maybe we should update the new feature part of your release note for that feature? | 14:29 |
| * whoami-rajat checking the feature releasenote | 14:30 | |
| whoami-rajat | rosmaita, yeah it does mention about reimaging reserved volumes although apart from nova, i don't think any end user/operator might use it | 14:31 |
| whoami-rajat | but we can modify it to remove the reserved part | 14:31 |
| rosmaita | ok, you can propose a patch to stable/yoga (as soon as it's cut) and we can discuss there whether we need to mention it or not | 14:32 |
| whoami-rajat | I'm thinking again about when the feature merges in nova in Zed, will we propose an empty releasenote mentioning it in cinder? maybe the releasenote is fine and I'm just thinking too much | 14:33 |
| whoami-rajat | rosmaita, in any case, i think the prelude looks fine, +1 | 14:33 |
| rosmaita | cool, ty | 14:34 |
| opendevreview | OpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/yoga: Update .gitreview for stable/yoga https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/833374 | 14:34 |
| opendevreview | OpenStack Release Bot proposed openstack/devstack-plugin-ceph stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/833375 | 14:34 |
| opendevreview | Sofia Enriquez proposed openstack/cinder-tempest-plugin master: [tests] Deleting Volumes Which are consumed as a basis for others https://review.opendev.org/c/openstack/cinder-tempest-plugin/+/823719 | 14:35 |
| hemna | ultimately, I think cinder needs to fix the reporting of it's allocated space. and then use that to determine what it thinks a pool has free, and then compare that to what's being reported, and then use the smaller number between the 2. | 14:37 |
| hemna | because also, something else besides cinder could be allocated space on that pool. | 14:38 |
| hemna | even my new calculate_capacity_factors here: https://review.opendev.org/c/openstack/cinder/+/831247/10/cinder/utils.py uses the reported free space for thick provisioning | 14:39 |
| opendevreview | Sofia Enriquez proposed openstack/cinder master: rbd: fix full restore ceph backups https://review.opendev.org/c/openstack/cinder/+/750782 | 15:23 |
| *** hemna1 is now known as hemna | 15:27 | |
| dansmith | rosmaita: can you look at a CI failure? https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_25f/831245/2/check/nova-ceph-multistore/25f37fb/job-output.txt | 15:31 |
| rosmaita | dansmith: sure | 15:32 |
| dansmith | AFAICT, a lot of tests got stuck in trying to clean up volume backups which takes so long that the job timeout fired and killed it | 15:32 |
| dansmith | this is on a simple devstack patch to to change how we generate our tls certs, so I wouldn't expect that to be anything related | 15:32 |
| dansmith | this is just a one-off failure while running that job, but wondering if it looks like something you recognize.. I can't find anything obvious in cinder logs | 15:33 |
| rosmaita | ok, i am in a meeting for a bit, but will take a look | 15:34 |
| dansmith | ack thanks | 15:34 |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:48 | |
| opendevreview | Ade Lee proposed openstack/cinder master: WIP/DNM: Add fips check jobs https://review.opendev.org/c/openstack/cinder/+/790535 | 16:20 |
| opendevreview | Merged openstack/cinder master: Add yoga release note prelude https://review.opendev.org/c/openstack/cinder/+/833365 | 16:32 |
| rosmaita | finally | 16:32 |
| rosmaita | whoami-rajat: can you verify the hash on https://review.opendev.org/c/openstack/releases/+/832367 ? | 16:40 |
| whoami-rajat | looking | 16:41 |
| whoami-rajat | looks good | 16:45 |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:46 | |
| whoami-rajat | did my +1 trigger PTL-Approved +1 ? didn't know that was in place | 16:50 |
| rosmaita | welcome to the big time! | 16:53 |
| opendevreview | Eric Harney proposed openstack/cinder master: RBD: Use static methods where possible https://review.opendev.org/c/openstack/cinder/+/833443 | 17:12 |
| opendevreview | Merged openstack/devstack-plugin-ceph stable/yoga: Update .gitreview for stable/yoga https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/833374 | 18:09 |
| opendevreview | Brian Rosmaita proposed openstack/cinder master: Update doc8 ignore-path https://review.opendev.org/c/openstack/cinder/+/830674 | 18:21 |
| opendevreview | Merged openstack/devstack-plugin-ceph stable/yoga: Update TOX_CONSTRAINTS_FILE for stable/yoga https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/833375 | 18:25 |
| hemna | another deficiency I noticed that there is a bit of a race condition between the time when a volume is created and the next time the get_volume_stats is called. | 20:40 |
| hemna | the allocated_capacity_gb doesn't update | 20:40 |
| hemna | and you can create a bunch of volumes between then and the next update. | 20:40 |
| hemna | *sigh* | 20:41 |
| hemna | I think the scheduler should automatically decrement that space in it's local setting until the next update comes | 20:43 |
| hemna | this seems quite bad :( | 20:43 |
| *** dviroel|ruck is now known as dviroel|out | 21:19 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!