| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: WIP: Cleanup ceph install on cs9 https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/877147 | 01:54 |
|---|---|---|
| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: WIP: Cleanup ceph install on cs9 https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/877147 | 04:37 |
| opendevreview | Takashi Kajinami proposed openstack/manila master: Remove TripleO job https://review.opendev.org/c/openstack/manila/+/877335 | 05:26 |
| opendevreview | Takashi Kajinami proposed openstack/manila master: Remove TripleO job https://review.opendev.org/c/openstack/manila/+/877335 | 05:29 |
| carthaca | periodic tasks are giving me headaches https://bugs.launchpad.net/oslo.service/+bug/2011587, related bug in manila is https://bugs.launchpad.net/manila/+bug/2008497 - I appreciate any thoughts on this | 14:15 |
| gouthamr | hey carthaca - the log for one is annoying if we're not picking up the decorated function's name: https://opendev.org/openstack/oslo.service/src/commit/43413775fca9431bcb3b7aae91c58f7e885f27a3/oslo_service/loopingcall.py#L188 | 15:15 |
| gouthamr | carthaca: to check the scale of things, does the backend have a lot of replicas? | 15:18 |
| carthaca | gouthamr: no, not many replicas - I see a 700 sec delay for a backend that has roughly 30 share instances... ah but 1200 deleted ones, let me check the share servers | 15:25 |
| carthaca | gouthamr: nope, just 3 share servers overall | 15:26 |
| carthaca | gouthamr: all our shares are replication enabled (replication_type dr), but real replicas i.e. replication_state!='active' are only 10 out of those 1200 | 15:36 |
| gouthamr | carthaca: the task ran longer than the interval by 700 seconds? | 15:42 |
| carthaca | yes, and my interval is configured at 300, so it took 1000 seconds | 15:43 |
| gouthamr | carthaca: :| wow that's bad.. | 15:49 |
| gouthamr | there are 11 periodic tasks in the share manager, and 1 of them is disabled by default (think there's another one we can disable by default too) | 15:50 |
| gouthamr | i wonder if the db query (https://github.com/openstack/manila/blob/74ae712d58a01daf2a6a1ca76ea8a43cfeaff4e7/manila/share/manager.py#L2672-L2679) is inefficient | 15:51 |
| gouthamr | then there is work being done to repair replicas in the netapp driver during the update call: https://github.com/openstack/manila/blob/29db644db5a60a93356a7837da3d9f7551bb95f4/manila/share/drivers/netapp/dataontap/cluster_mode/lib_base.py#L2555 | 15:52 |
| gouthamr | we could try factoring those out into separate threads | 15:53 |
| carthaca | re db query: the filtering should move to the db layer probably | 15:58 |
| carthaca | but the other place, the netapp driver on replica update, is more likely to consume that many time | 15:58 |
| carthaca | is it wise to spawn threads within the periodic tasks? probably this would need a bigger rewrite, then | 16:01 |
| * gouthamr re-reads the netapp code | 16:01 | |
| gouthamr | agree, feels like a minimal gain there if any | 16:03 |
| gouthamr | > is it wise to spawn threads within the periodic tasks? probably this would need a bigger rewrite, then | 16:04 |
| gouthamr | there're separate periodic threads in the netapp driver | 16:05 |
| gouthamr | currently used for deferred cleanups; that pattern is good for reconciling driver's resources since won't overload the looping call that we use in the share manager | 16:06 |
| opendevreview | Merged openstack/manila stable/zed: update capacity filter during extend share https://review.opendev.org/c/openstack/manila/+/876227 | 16:07 |
| opendevreview | Martin Kopec proposed openstack/devstack-plugin-ceph master: DNM testing out https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/877388 | 16:12 |
| carthaca | gouthamr: seems like I just needed you to be my rubber duck, thanks | 16:20 |
| carthaca | gouthamr: the db calls are the slow thing https://paste.opendev.org/show/bSdsJTqVpArecXhTLjiu/ | 16:20 |
| gouthamr | carthaca: :P happy to help | 16:20 |
| gouthamr | did you prune the deleted shares? | 16:21 |
| carthaca | gouthamr: 10 minutes to query all the stuff until it goes from 'Updating status of share replica instances' to 'Updating status of share share_replica 12..' | 16:21 |
| gouthamr | ah :O | 16:22 |
| carthaca | same for periodic_share_replica_snapshot_update | 16:23 |
| carthaca | I'm already pruning after 7 days | 16:23 |
| gouthamr | ack, the code attempts the same query there and looks for snapshots of non-active replicas | 16:24 |
| gouthamr | ack carthaca; so our bug here would be to optimize the db queries in those two methods | 16:27 |
| gouthamr | still interested in someone looking at the oslo.service bug and telling us if we're missing some config opts | 16:28 |
| carthaca | I fully agree | 16:29 |
| carthaca | deleted object should be already ignored https://github.com/openstack/manila/blob/74ae712d58a01daf2a6a1ca76ea8a43cfeaff4e7/manila/db/sqlalchemy/api.py#L1948 | 16:30 |
| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: WIP: Cleanup ceph install on cs9 https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/877147 | 18:33 |
| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: WIP: Cleanup ceph install on cs9 https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/877147 | 18:45 |
| *** dviroel_ is now known as dviroel | 20:21 | |
| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: Cleanup installation and revive cephfs-nfs job https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/877147 | 23:17 |
| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: Cleanup installation and revive cephfs-nfs job https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/877147 | 23:21 |
| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: Remove branch specific code from repo https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/877435 | 23:42 |
| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: Cleanup installation and revive cephfs-nfs job https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/877147 | 23:58 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!