| *** itlinux has joined #openstack-cinder | 00:06 | |
| *** _alastor_ has joined #openstack-cinder | 00:11 | |
| *** dklyle has quit IRC | 00:21 | |
| *** itlinux_ has joined #openstack-cinder | 00:25 | |
| *** itlinux has quit IRC | 00:28 | |
| *** lixiaoy1 has joined #openstack-cinder | 00:29 | |
| *** dklyle has joined #openstack-cinder | 00:34 | |
| *** tpsilva has quit IRC | 00:37 | |
| *** itlinux_ has quit IRC | 00:49 | |
| *** dklyle has quit IRC | 01:04 | |
| *** brinzhang has joined #openstack-cinder | 01:09 | |
| *** dave-mccowan has joined #openstack-cinder | 01:15 | |
| *** mriedem has quit IRC | 01:23 | |
| *** _alastor_ has quit IRC | 01:25 | |
| *** erlon has quit IRC | 01:35 | |
| *** markvoelker has quit IRC | 01:41 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 02:01 | |
| *** pots has joined #openstack-cinder | 02:05 | |
| *** brinzhang has quit IRC | 02:10 | |
| *** brinzhang has joined #openstack-cinder | 02:10 | |
| *** bhagyashris_ has joined #openstack-cinder | 02:21 | |
| *** Dinesh_Bhor has quit IRC | 02:27 | |
| *** mhen has quit IRC | 02:28 | |
| *** mhen has joined #openstack-cinder | 02:31 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 02:33 | |
| *** hedvig_01 has quit IRC | 02:33 | |
| *** dhineshb has quit IRC | 02:52 | |
| openstackgerrit | Yikun Jiang proposed openstack/cinder master: Add x_project_id, accepted to transfers https://review.openstack.org/615487 | 03:13 |
|---|---|---|
| openstackgerrit | Yikun Jiang proposed openstack/cinder master: Improve volume transfer records https://review.openstack.org/615716 | 03:13 |
| *** psachin has joined #openstack-cinder | 03:27 | |
| *** Dinesh_Bhor has quit IRC | 03:34 | |
| *** dave-mccowan has quit IRC | 03:52 | |
| *** lixiaoy1 has quit IRC | 03:53 | |
| *** lixiaoy1 has joined #openstack-cinder | 03:54 | |
| *** v12aml has quit IRC | 04:10 | |
| *** udesale has joined #openstack-cinder | 04:17 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 04:42 | |
| *** pots has quit IRC | 04:43 | |
| *** pots has joined #openstack-cinder | 04:44 | |
| openstackgerrit | Yingxin Cheng proposed openstack/cinder master: [wip] Cinder RSD Driver https://review.openstack.org/621465 | 05:10 |
| *** dhellmann has quit IRC | 05:19 | |
| *** rtjure has quit IRC | 05:19 | |
| *** dhellmann has joined #openstack-cinder | 05:20 | |
| *** itlinux has joined #openstack-cinder | 05:21 | |
| *** rtjure has joined #openstack-cinder | 05:22 | |
| *** dklyle has joined #openstack-cinder | 05:51 | |
| *** itlinux has quit IRC | 05:51 | |
| openstackgerrit | Brin Zhang proposed openstack/cinder master: Add user_id in backup list and show API https://review.openstack.org/615129 | 05:52 |
| *** dklyle has quit IRC | 05:56 | |
| *** _alastor_ has joined #openstack-cinder | 06:06 | |
| *** Dinesh_Bhor has quit IRC | 06:13 | |
| *** alkhodos has quit IRC | 06:14 | |
| *** lpetrut has joined #openstack-cinder | 06:30 | |
| openstackgerrit | Yikun Jiang proposed openstack/cinder master: Add x_project_id, accepted to transfers https://review.openstack.org/615487 | 06:45 |
| openstackgerrit | Yikun Jiang proposed openstack/cinder master: Improve volume transfer records https://review.openstack.org/615716 | 06:45 |
| *** _alastor_ has quit IRC | 06:45 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 06:47 | |
| *** lixiaoy1 has quit IRC | 06:54 | |
| *** Luzi has joined #openstack-cinder | 07:01 | |
| *** e0ne has joined #openstack-cinder | 07:13 | |
| openstackgerrit | Yikun Jiang proposed openstack/cinder master: Add x_project_id, accepted to transfers https://review.openstack.org/615487 | 07:22 |
| openstackgerrit | Yikun Jiang proposed openstack/cinder master: Improve volume transfer records https://review.openstack.org/615716 | 07:22 |
| *** openstackgerrit has quit IRC | 07:29 | |
| *** dklyle has joined #openstack-cinder | 07:29 | |
| *** rcernin has quit IRC | 07:30 | |
| *** Liang__ has joined #openstack-cinder | 07:34 | |
| *** lpetrut has quit IRC | 07:46 | |
| *** trident has quit IRC | 07:58 | |
| *** trident has joined #openstack-cinder | 08:00 | |
| *** openstackgerrit has joined #openstack-cinder | 08:13 | |
| openstackgerrit | Yingxin Cheng proposed openstack/cinder master: [wip] Cinder RSD Driver https://review.openstack.org/621465 | 08:13 |
| *** imacdonn has quit IRC | 08:23 | |
| *** imacdonn has joined #openstack-cinder | 08:23 | |
| *** Liang__ has quit IRC | 08:27 | |
| *** Liang__ has joined #openstack-cinder | 08:28 | |
| *** e0ne has quit IRC | 08:31 | |
| *** dklyle has quit IRC | 08:32 | |
| *** Liang__ has quit IRC | 08:36 | |
| *** Liang__ has joined #openstack-cinder | 08:39 | |
| *** markvoelker has joined #openstack-cinder | 08:44 | |
| *** markvoelker has quit IRC | 08:49 | |
| *** Liang__ has quit IRC | 09:32 | |
| openstackgerrit | Gorka Eguileor proposed openstack/cinder stable/rocky: Ensure image utils don't block greenthreads https://review.openstack.org/624497 | 09:33 |
| *** psachin is now known as psachin|session | 09:35 | |
| *** markvoelker has joined #openstack-cinder | 09:39 | |
| *** lpetrut has joined #openstack-cinder | 09:41 | |
| *** lpetrut has quit IRC | 09:45 | |
| *** bhagyashris_ has quit IRC | 09:53 | |
| *** gkadam has joined #openstack-cinder | 10:05 | |
| *** gkadam has quit IRC | 10:05 | |
| *** e0ne has joined #openstack-cinder | 10:08 | |
| *** ganso has joined #openstack-cinder | 10:14 | |
| openstackgerrit | wang yong proposed openstack/cinder master: Cinder volume driver for Inspur AS13000 series https://review.openstack.org/562977 | 10:16 |
| *** luizbag has joined #openstack-cinder | 10:16 | |
| *** Dinesh_Bhor has quit IRC | 10:24 | |
| *** Dinesh_Bhor has joined #openstack-cinder | 10:49 | |
| *** erlon has joined #openstack-cinder | 10:57 | |
| *** udesale has quit IRC | 11:02 | |
| *** udesale has joined #openstack-cinder | 11:03 | |
| *** lpetrut has joined #openstack-cinder | 11:03 | |
| openstackgerrit | Gorka Eguileor proposed openstack/cinder master: Add cinderlib functional tests https://review.openstack.org/620671 | 11:11 |
| *** v12aml has joined #openstack-cinder | 11:28 | |
| *** markvoelker has quit IRC | 11:30 | |
| *** e0ne has quit IRC | 11:30 | |
| *** Dinesh_Bhor has quit IRC | 11:33 | |
| *** yikun_ has quit IRC | 11:51 | |
| *** tpsilva has joined #openstack-cinder | 11:57 | |
| *** markvoelker has joined #openstack-cinder | 12:05 | |
| *** dave-mccowan has joined #openstack-cinder | 12:40 | |
| *** abishop has joined #openstack-cinder | 12:59 | |
| *** brinzhang has quit IRC | 13:00 | |
| openstackgerrit | Merged openstack/cinder master: Tintri: Inherit tests from BaseDriverTestCase https://review.openstack.org/600442 | 13:25 |
| *** markvoelker has quit IRC | 13:28 | |
| *** eharney has joined #openstack-cinder | 13:40 | |
| *** markvoelker has joined #openstack-cinder | 14:01 | |
| *** mriedem has joined #openstack-cinder | 14:13 | |
| openstackgerrit | Sofia Enriquez proposed openstack/cinder stable/rocky: Ensure image utils don't block greenthreads https://review.openstack.org/624497 | 14:16 |
| mszwed | /q lennyb | 14:33 |
| mszwed | oops :) | 14:33 |
| *** irclogbot_0 has quit IRC | 14:36 | |
| *** e0ne has joined #openstack-cinder | 14:38 | |
| *** mmethot has quit IRC | 14:43 | |
| *** mmethot has joined #openstack-cinder | 14:46 | |
| *** irclogbot_0 has joined #openstack-cinder | 14:51 | |
| *** Miouge has quit IRC | 14:55 | |
| openstackgerrit | Eric Harney proposed openstack/cinder master: Rename BackupNFSSwiftBasedTestCase https://review.openstack.org/611137 | 14:55 |
| *** Miouge has joined #openstack-cinder | 14:57 | |
| noonedeadpunk | Hi everyone. It seems, that read/write_iops_sec_per_gb_min do not work at rocky. So I have a disk type with the following limits http://paste.openstack.org/show/737138/ | 15:09 |
| noonedeadpunk | And have 2 disks, which are connected to the instance - 10Gb VDC and 200GB VDD: http://paste.openstack.org/show/737139/ | 15:10 |
| noonedeadpunk | And min value is always applied, not depending on size of the disk | 15:11 |
| *** markvoelker has quit IRC | 15:18 | |
| *** lpetrut has quit IRC | 15:22 | |
| *** mchlumsky has quit IRC | 15:38 | |
| *** mchlumsky has joined #openstack-cinder | 15:40 | |
| *** lixiaoy1 has joined #openstack-cinder | 15:48 | |
| *** lpetrut has joined #openstack-cinder | 15:48 | |
| *** alkhodos has joined #openstack-cinder | 15:49 | |
| *** Liang__ has joined #openstack-cinder | 15:51 | |
| *** Liang__ is now known as LiangFang | 15:51 | |
| *** lpetrut has quit IRC | 15:56 | |
| *** Luzi has quit IRC | 15:57 | |
| whoami-rajat | Hi smcginnis jungleboyj , can you please take a look at https://review.openstack.org/#/c/619676/ . Thanks! | 15:58 |
| *** yikun has joined #openstack-cinder | 16:04 | |
| *** udesale has quit IRC | 16:19 | |
| *** _alastor_ has joined #openstack-cinder | 16:32 | |
| *** _alastor_ has quit IRC | 16:40 | |
| *** itlinux has joined #openstack-cinder | 16:53 | |
| *** lixiaoy1 has quit IRC | 16:56 | |
| openstackgerrit | Woojay Poynter proposed openstack/cinder master: Update DRBD (drbdmanagedrv) Driver to LINSTOR (linstordrv) https://review.openstack.org/624233 | 17:11 |
| *** LiangFang has quit IRC | 17:16 | |
| *** psachin|session has quit IRC | 17:20 | |
| *** e0ne has quit IRC | 17:21 | |
| *** canori01 has joined #openstack-cinder | 17:22 | |
| canori01 | hey guys, does the lvm driver currently support lvm striping? | 17:23 |
| eharney | canori01: i think that would be up to how you constructed the lvm volume group outside of cinder | 17:31 |
| eharney | oh, it's per LV, so... no | 17:33 |
| *** ianychoi has joined #openstack-cinder | 17:38 | |
| canori01 | eharney: ah, ok. Thanks! | 17:42 |
| *** _alastor_ has joined #openstack-cinder | 18:03 | |
| *** mriedem is now known as mriedem_lunch | 18:21 | |
| *** markguz_ has joined #openstack-cinder | 18:23 | |
| markguz_ | jungleboyj: i got it mostly working as you described. can't seem to actually attach the volume tho. nova throws an exception complaining about "KeyError: 'target_wwn'" | 18:23 |
| openstackgerrit | Merged openstack/cinder master: An alternate way to fix retype notifier test case https://review.openstack.org/618499 | 18:28 |
| jungleboyj | markguz_: Hmmm, that is interesting. So something isn't totally working obviously. | 18:35 |
| markguz_ | jungleboyj: yeah. http://paste.openstack.org/show/737161/ i think it can't locate the FC device... | 19:00 |
| jungleboyj | Odd that it is coming through as a KeyError. Seems like that is a bug. | 19:03 |
| jungleboyj | Are there any errors in the volume log? | 19:04 |
| *** markguz_ has quit IRC | 19:06 | |
| *** mriedem_lunch is now known as mriedem | 19:11 | |
| *** e0ne has joined #openstack-cinder | 19:20 | |
| *** luizbag has quit IRC | 19:24 | |
| *** e0ne_ has joined #openstack-cinder | 19:27 | |
| *** e0ne has quit IRC | 19:27 | |
| *** e0ne_ has quit IRC | 19:30 | |
| *** erlon has quit IRC | 19:33 | |
| *** e0ne has joined #openstack-cinder | 19:34 | |
| *** e0ne_ has joined #openstack-cinder | 19:39 | |
| mriedem | this is a pretty old gate bug at this point https://bugs.launchpad.net/cinder/+bug/1763712 | 19:39 |
| openstack | Launchpad bug 1763712 in Cinder "Unable to update the attachment.: MessagingTimeout" [Medium,Confirmed] | 19:39 |
| mriedem | would be good to get cinder people to take a look at that again | 19:39 |
| mriedem | not sure if that's just a problem on slow nodes or what, or if something is inefficient during volume attachment update in cinder-volume | 19:40 |
| mriedem | the rpc call from c-api to c-vol definitely doesn't help | 19:40 |
| *** e0ne has quit IRC | 19:41 | |
| *** e0ne has joined #openstack-cinder | 19:44 | |
| mriedem | also, this was marked as fixed but something must have regressed it http://status.openstack.org/elastic-recheck/#1745168 | 19:44 |
| *** e0ne_ has quit IRC | 19:45 | |
| *** e0ne has quit IRC | 19:45 | |
| *** e0ne has joined #openstack-cinder | 19:46 | |
| *** markguz_ has joined #openstack-cinder | 19:48 | |
| *** e0ne has quit IRC | 19:54 | |
| canori01 | With the lvm driver, is thin the only supported configuration currently? | 19:55 |
| smcginnis | canori01: No, thick or thin. See the lvm_type config option: https://docs.openstack.org/cinder/latest/configuration/block-storage/drivers/lvm-volume-driver.html | 19:59 |
| canori01 | smcginnis: Ah, I was just looking at that. So default=thick? Because it says on there that auto defaults to thin | 20:01 |
| smcginnis | canori01: Yeah, if you want thick then set lvm_type=thick | 20:02 |
| canori01 | smcginnis: Thanks. That's not one of the keywordslisted on the doc, but I'll give that a shot | 20:03 |
| smcginnis | "keywords listed on the doc"? | 20:04 |
| canori01 | yeah, it says "(String(choices=[‘default’, ‘thin’, ‘auto’]))" "thick" is not listed there as a choice | 20:04 |
| canori01 | but if "thick" works, I'll go with that | 20:05 |
| smcginnis | Oh! Sorry, thick is indeed not an option there. | 20:06 |
| canori01 | yeah, that's why I was wondering if "default" was thick. Or if thick is supported at all | 20:07 |
| smcginnis | I'd have to look closer (which I'm not able to do right now), but IIRC, default or auto will give you thick if the VG has not been set up as thinly provisioned. | 20:08 |
| smcginnis | eharney: You around? I know you probably know all that off the top of your head. ^ | 20:08 |
| *** kmalloc is now known as notmorgan | 20:10 | |
| *** notmorgan is now known as morgan | 20:10 | |
| *** itlinux_ has joined #openstack-cinder | 20:14 | |
| *** e0ne has joined #openstack-cinder | 20:15 | |
| *** itlinux has quit IRC | 20:15 | |
| *** e0ne has quit IRC | 20:16 | |
| eharney | lvm_type = default is thick | 20:16 |
| *** _alastor_ has quit IRC | 20:32 | |
| *** dklyle has joined #openstack-cinder | 20:39 | |
| mriedem | jungleboyj: dansmith: on that attachment update rpc timeout, | 20:39 |
| mriedem | 99% of the 60 seconds spent are on this call: | 20:40 |
| mriedem | Dec 10 22:03:24.340740 ubuntu-xenial-inap-mtl01-0001107482 cinder-volume[31110]: DEBUG oslo_concurrency.processutils [req-2627d823-75bc-4092-bb8e-f079761d732f req-fec531fa-7200-4e15-a820-18ea7e5e6214 tempest-ServerActionsTestJSON-1162184781 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-b62c0c8d-b3cd-46d0-b009-cabc13115f07" returned: 0 in 58.435s {{(pid=3120 | 20:40 |
| mriedem | xecute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409}} | 20:40 |
| dansmith | that seems really odd | 20:40 |
| dansmith | lvchange should be pretty easy/fast, | 20:40 |
| jungleboyj | Sheesh. Why would an LVChange take that long? | 20:41 |
| dansmith | although I think it does require doing a block device refresh | 20:41 |
| eharney | which job is that on? | 20:41 |
| mriedem | http://logs.openstack.org/81/624181/1/check/nova-multiattach/c1589ae/logs/screen-c-vol.txt#_Dec_10_22_03_24_340740 | 20:41 |
| mriedem | eharney: nova-multiattach job, but not a multiattach volume or test | 20:41 |
| smcginnis | There are some retries around lvchange since every 6 months or so there's some update that causes the call to fail the first few times. | 20:41 |
| mriedem | this isn't a retry, unless it's not logging it | 20:42 |
| dansmith | but I think mriedem is saying this one call blocks for that long | 20:42 |
| mriedem | returned: 0 in 58.435s | 20:42 |
| dansmith | mriedem: also, that's 96%.. come on.. FAKE NEWS | 20:42 |
| mriedem | ha | 20:42 |
| mriedem | hey i'll shut down this whole gd government | 20:43 |
| dansmith | is this a manhood thing for you? | 20:43 |
| jungleboyj | If that is even possible for him. | 20:43 |
| dansmith | okay that's enough before I barf. | 20:44 |
| smcginnis | So if lvchange is taking its sweet time, not sure how we can improve that situation in the cinder code. | 20:44 |
| mriedem | fwiw a couple of seconds before that another volume is hitting lvchange | 20:44 |
| mriedem | http://logs.openstack.org/81/624181/1/check/nova-multiattach/c1589ae/logs/syslog.txt.gz#_Dec_10_22_02_22 | 20:44 |
| dansmith | mriedem: and how long does that one take? | 20:45 |
| dansmith | smcginnis: do you guys @synchronized your calls to lvm stuff? | 20:45 |
| mriedem | Dec 10 22:03:24.317836 ubuntu-xenial-inap-mtl01-0001107482 cinder-volume[31110]: DEBUG oslo_concurrency.processutils [req-9e16b85d-5662-4c70-aa13-88cd9c6a03fc req-ba539360-fea6-4538-88dd-afa948120e1c tempest-ServerRescueNegativeTestJSON-1204517625 None] CMD "sudo cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-c48c90f2-bf42-473b-b9a4-48f7e6f97f24" returned: 0 in 61.526s {{(p | 20:45 |
| mriedem | 1207) execute /usr/local/lib/python2.7/dist-packages/oslo_concurrency/processutils.py:409}} | 20:45 |
| mriedem | that one takes over 61 seconds | 20:45 |
| dansmith | nice | 20:45 |
| smcginnis | dansmith: I seem to remember it being synchronized. | 20:45 |
| dansmith | if not, it would be interesting to see if serializing all lvm ops make things overall go faster | 20:46 |
| dansmith | I thought they had some of their own internal locking, but if they're fighting and thrashing, that might really go slow on an io-limited instance | 20:46 |
| jungleboyj | dansmith: That would be my guess as to what is happening. | 20:47 |
| dansmith | jungleboyj: could you throw up a patch we could hammer on a bit? | 20:47 |
| jungleboyj | Especially given that we have started seeing more issues like this since the size of our infra cloud has decreased. | 20:47 |
| mriedem | smcginnis: this code right? http://git.openstack.org/cgit/openstack/cinder/tree/cinder/brick/local_dev/lvm.py#n656 | 20:48 |
| smcginnis | Hmm, yeah. activate_lv and deactivate. | 20:48 |
| smcginnis | Does NOT appear to have synchronization. | 20:48 |
| smcginnis | We could try that and see if it makes a difference. | 20:48 |
| dansmith | no lockutils in there at all | 20:48 |
| dansmith | we'd want to sync all ops, not just all lvchange, so something more than just a decorator would be needed | 20:49 |
| eharney | do we know if this happens on lvm jobs that don't use the scsi-target-utils target? | 20:49 |
| smcginnis | I think when I've seen these before there weren't any signs of other operations going on at the same time, but shouldn't hurt to try. | 20:49 |
| dansmith | mriedem: so that code is in brick.. can I change it in their tree and make them use the change, or do I have to do complicated library things? | 20:51 |
| jungleboyj | dansmith: What do you mean as far as a patch? | 20:51 |
| dansmith | jungleboyj: I'll start | 20:51 |
| jungleboyj | Ok. Cool. | 20:51 |
| smcginnis | The LVM bits aren't in os-brick. That's still local in o/cinder | 20:51 |
| mriedem | dansmith: it's in cinder | 20:51 |
| eharney | this message in syslog is interesting: Dec 10 22:02:31 ubuntu-xenial-inap-mtl01-0001107482 kernel: device-mapper: thin: 252:2: reached low water mark for data device: sending event. | 20:52 |
| jungleboyj | smcginnis: Are you thinking that the activate and deactivate might be blocking each other? | 20:52 |
| openstackgerrit | Dhinesh Balasubramaniam proposed openstack/cinder master: Hedvig Cinder driver implementation https://review.openstack.org/433341 | 20:52 |
| *** hedvig_01 has joined #openstack-cinder | 20:53 | |
| smcginnis | jungleboyj: Or multiple lv ops slower each other down. | 20:53 |
| smcginnis | mriedem: It does appear there were two there in the logs you linked to above. | 20:53 |
| jungleboyj | Ok. That makes sense. | 20:53 |
| smcginnis | Two overlapping calls that is. | 20:53 |
| mriedem | Dec 10 22:02:22 ubuntu-xenial-inap-mtl01-0001107482 sudo[5460]: stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-c48c90f2-bf42-473b-b9a4-48f7e6f97f24 | 20:55 |
| mriedem | Dec 10 22:02:25 ubuntu-xenial-inap-mtl01-0001107482 sudo[5504]: stack : TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/local/bin/cinder-rootwrap /etc/cinder/rootwrap.conf lvchange -a y --yes -K stack-volumes-lvmdriver-1/volume-b62c0c8d-b3cd-46d0-b009-cabc13115f07 | 20:55 |
| dansmith | I'll push a patch in a sec | 20:55 |
| mriedem | 3 seconds apart | 20:55 |
| mriedem | note also that cinder-backup calls initialize_connection in cinder-volume which is the same thing that attachment_update does, so likely the same source of the cinder backup messaging timeouts | 20:56 |
| jungleboyj | Hmm, interesting. | 20:56 |
| hedvig_01 | @eharney: Thank you for the comments. We have pushed the code to openstack repository. https://review.openstack.org/#/c/433341/ | 20:57 |
| openstackgerrit | Dan Smith proposed openstack/cinder master: WIP: Synchronize all lvm operations https://review.openstack.org/624809 | 20:57 |
| dansmith | does that look like it'll do it? ^ | 20:57 |
| smcginnis | dansmith: Yeah, I think that looks right. Thanks | 20:58 |
| mriedem | dansmith: the theory being that 2 lvchange -a happening at the same time are causing both to be extra slow? | 20:58 |
| dansmith | mriedem: yeah, so a couple possibilities: | 20:59 |
| jungleboyj | hedvig_01: Have you gotten the boot pattern test running yet? | 20:59 |
| dansmith | 1. access to dm is one at a time, IIRC, which means they'll fight to setup dm devices to some degree, and | 20:59 |
| dansmith | 2. If they're doing anything that causes buffer cache to flush, two running at once on an io-limited system will slow things down | 21:00 |
| dansmith | IIRC, there is a long history of issues with lvm concurrency | 21:00 |
| smcginnis | Totally plausible it could cause some sort of internal deadlock/thrashing to happen. | 21:00 |
| jungleboyj | dansmith: I know you are right on the concurrency issues. | 21:02 |
| dansmith | mriedem: see here, and look at "locking_type" and below: https://linux.die.net/man/5/lvm.conf | 21:02 |
| hedvig_01 | @jungleboyj: No we did not. As mentioned earlier, we do not take responsibility for quiescing. If IOs are done on a volume and immediately a snapshot is invoked without quiescing, data will be inconsistent. | 21:02 |
| hedvig_01 | I was waiting for a response from you guys for the same. | 21:03 |
| mriedem | dansmith: aha | 21:03 |
| jungleboyj | eharney: Wasn' | 21:03 |
| mriedem | and wait_for_locks defaults to 1 | 21:03 |
| mriedem | so yeah by default we'll lock on /var/lock/lvm for concurrent operations | 21:04 |
| jungleboyj | t that the big discussion earlier in the week or last week that quiescing isn't a requirement for any other drivers? | 21:04 |
| jungleboyj | You just get a crash-consistent snapshot without quiescing ? | 21:04 |
| dansmith | which means it is using its own locking and should be providing protection, but we could be having dining philosopher type interaction | 21:04 |
| eharney | jungleboyj: yes, that test not working due to concerns about quiescing doesn't make sense IMO, i think something else is going on there | 21:05 |
| jungleboyj | eharney: Ok. I agree. | 21:05 |
| *** rcernin has joined #openstack-cinder | 21:06 | |
| jungleboyj | hedvig_01: Does the test not run correctly or are you disabling it because you are concerned with the results of doing that operation? | 21:06 |
| *** jmlowe has quit IRC | 21:10 | |
| *** jmlowe has joined #openstack-cinder | 21:11 | |
| *** redcavalier has joined #openstack-cinder | 21:11 | |
| redcavalier | Hi, we made a bug report yesterday and we were wondering if somebody could have a look at it, to at least confirm that it's valid. | 21:12 |
| redcavalier | Here's the report : https://bugs.launchpad.net/cinder/+bug/1808083 | 21:12 |
| openstack | Launchpad bug 1808083 in Cinder "DellEMC SC Storage Center API "Unable to find result where folderPath is" - Duplicate folders" [Undecided,New] | 21:12 |
| *** jmlowe has quit IRC | 21:13 | |
| *** jmlowe has joined #openstack-cinder | 21:14 | |
| hedvig_01 | @jungleboyj , @eharney: I can get a crash-consistent snapshot without quiescing. Does openstack follow this . Send IOs, Wait for current IOs to succeed and then issue snapshot? | 21:18 |
| jungleboyj | hedvig_01: I can't speak as to how this is handled on the Nova side, but I know other drivers are able to create snapshots without any type of quiescing happening. | 21:22 |
| eharney | hedvig_01: what protocol does your driver attach with? | 21:22 |
| *** ganso has quit IRC | 21:25 | |
| hedvig_01 | @eharney: block protocol | 21:28 |
| jungleboyj | hedvig_01: So, lets make this simple. The boot pattern test is required. Other storage backends using iSCSI work fine as a sync is done and if the iSCSI cache flush is done properly all the data should be there. | 21:28 |
| jungleboyj | If your driver can't handle that, then it isn't ready to be merged. | 21:28 |
| markguz_ | jungleboyj: digging around. it seems that ubuntu 16.04 creates a /dev/disk/by-path/pci-0000:01:00.1-fc-0x500507605ebff681-lun-1 | 21:38 |
| markguz_ | but nova on mount is looking for /dev/disk/by-path/fc-0x500507605ebff681-lun-1 | 21:39 |
| markguz_ | i guess i need to ask the nova folks about this | 21:39 |
| markguz_ | seems weird | 21:39 |
| jungleboyj | That does seem weird. | 21:39 |
| jungleboyj | hemna: You around? | 21:40 |
| jungleboyj | ^^ | 21:40 |
| imacdonn | that's actually a os_brick thing (which nova and cinder both utilise) | 21:40 |
| jungleboyj | Right. | 21:40 |
| jungleboyj | Did something change in ubuntu that that isn't working? | 21:41 |
| hedvig_01 | @jungleboyj , @eharney : I see most of them skipping this test too. I had sent a link earlier and also some other vendor also raised the same concern last week while we were discussing this . Let me try running this testcase in loop and check if a sync is performed before we receive a snapshot request. If that has happened, and we acknowledged all IOs before snapshot came in, then definitely it's something we are missing on our | 21:41 |
| hedvig_01 | end. | 21:41 |
| markguz_ | jungleboyj: the only diff here is that this is ppc64el | 21:41 |
| markguz_ | or le if you're not ubuntu | 21:41 |
| hedvig_01 | I'll keep you guys posted | 21:41 |
| jungleboyj | markguz_: Oh, interesting. I ahve seen differences like that before when working with PPC64 back in my IBM days. | 21:43 |
| markguz_ | jungleboyj: i might need to modify udev rules somehow | 21:43 |
| *** lpetrut has joined #openstack-cinder | 21:44 | |
| *** _alastor_ has joined #openstack-cinder | 21:46 | |
| hedvig_01 | @eharney: https://review.openstack.org/#/c/433341/65/cinder/volume/drivers/hedvig/hedvig_cinder.py Thank you for the comments. check_for_export , get_Export etc are not deleted because it throws an error if we delete these functions. I guess it's a part of the interface. Log level are changed to info because we got a few comments saying we do not have info logs and it's difficult to trace in production env. So I made a few logs i | 21:56 |
| hedvig_01 | nfo, so that the code flow can be traced | 21:56 |
| *** itlinux_ has quit IRC | 22:00 | |
| *** dklyle has quit IRC | 22:00 | |
| *** itlinux has joined #openstack-cinder | 22:00 | |
| *** dcdamien has joined #openstack-cinder | 22:02 | |
| *** abishop has quit IRC | 22:08 | |
| markguz_ | jungleboyj: so it seems that os-brick thinks ppc64 doesn't need the prepended pci-path. there's separate fibre_channel_ppc64.py where _get_host_devices does not include pci_num | 22:19 |
| markguz_ | i'm running pike, so maybe this is fixed already | 22:19 |
| markguz_ | pike is getting old i guess | 22:19 |
| jungleboyj | markguz_: Ok. | 22:19 |
| markguz_ | jungleboyj: i hacked the ppc64.py file and the volume attaches no problems | 22:20 |
| jungleboyj | Interesting. | 22:22 |
| jungleboyj | What OS are you running? | 22:22 |
| markguz_ | ubuntu 16.04 LTS | 22:22 |
| jungleboyj | Ok, you said that earlier. | 22:22 |
| markguz_ | little endian | 22:22 |
| jungleboyj | I am guessing that the IBMers made an assumption of RedHat and RedHat may handle things differently? | 22:23 |
| jungleboyj | It would seem to me that it would be valid to try update os-brick to try and handle both cases. | 22:23 |
| jungleboyj | May want to check with Chhavi Agarawal and get her input though. | 22:24 |
| markguz_ | well it could be that os-brick rocky has the changes. ihavn't check | 22:24 |
| jungleboyj | hhagarw@in.ibm.com | 22:24 |
| markguz_ | k. will do | 22:24 |
| jungleboyj | Sorry, chhagarw@in.ibm.com | 22:24 |
| jungleboyj | Well, I just looked at master and I don't seen an update that obviously handles that. | 22:25 |
| markguz_ | i'll drop her a line. I'm an ibmer too so no problems | 22:25 |
| markguz_ | ok | 22:25 |
| jungleboyj | markguz_: Ah, that is why you are using ppc64. | 22:25 |
| *** obre_ has joined #openstack-cinder | 22:25 | |
| markguz_ | jungleboyj: lol because noone else does? | 22:25 |
| jungleboyj | Well, I remember from being there ... not as many as I hoped. | 22:26 |
| *** _alastor_ has quit IRC | 22:26 | |
| markguz_ | jungleboyj: the openpower systems are nice. not had a chance to play with Power9 yet, but the P8 systems are sweet | 22:26 |
| jungleboyj | Were you looking at You looking at _get_host_devices ? | 22:26 |
| markguz_ | def _get_host_devices | 22:26 |
| markguz_ | yeah | 22:26 |
| jungleboyj | Yeah, it is still just looking at fc-%s-lun%s | 22:27 |
| markguz_ | had to change the host_device filter and add the pci_num param to get it to work | 22:27 |
| jungleboyj | Ok. So, it hasn't been fixed. | 22:27 |
| markguz_ | essentially making it the same as intel | 22:27 |
| jungleboyj | I am guessing it is RedHat specific. | 22:27 |
| markguz_ | perhaps. | 22:28 |
| jungleboyj | markguz_: Where you based? | 22:28 |
| markguz_ | Houston TX | 22:28 |
| markguz_ | jungleboyj: Houston TX | 22:28 |
| jungleboyj | Cool. I was and still am in Rochester, MN. Work for Lenovo now though. | 22:28 |
| *** uberjay has joined #openstack-cinder | 22:29 | |
| *** andreykurilin has quit IRC | 22:29 | |
| *** obre has quit IRC | 22:29 | |
| *** uberjay_ has quit IRC | 22:29 | |
| *** smcginnis has quit IRC | 22:29 | |
| *** logan- has quit IRC | 22:29 | |
| markguz_ | jungleboyj: ex systemX division? | 22:30 |
| *** logan- has joined #openstack-cinder | 22:31 | |
| *** trident has quit IRC | 22:32 | |
| *** gouthamr has quit IRC | 22:32 | |
| jungleboyj | I was in iSeries, then HPC with Blue Gene, then Cloud for OpenStack. | 22:32 |
| *** lpetrut has quit IRC | 22:32 | |
| jungleboyj | When they moved me to SoftLayer I voluntarily went to Lenovo to help get their cloud initiatives going. | 22:33 |
| markguz_ | cool | 22:33 |
| jungleboyj | Was originally in Research but I am now in DCG which was the xSeries business from IBM. | 22:33 |
| jungleboyj | We are doing some great stuff. | 22:33 |
| markguz_ | yeah the lenovo systemXs are nice as are the thinkSystems | 22:34 |
| *** trident has joined #openstack-cinder | 22:34 | |
| jungleboyj | Yeah, I have an old 3650-m3 at home now. Would love an sr650 | 22:34 |
| markguz_ | we just got a bunch in. | 22:35 |
| markguz_ | i use ironic to provision the labs systems here | 22:35 |
| *** itlinux has quit IRC | 22:35 | |
| markguz_ | i kinda use openstack to manage the whole lab. | 22:35 |
| *** gouthamr has joined #openstack-cinder | 22:35 | |
| jungleboyj | Sweet. Don't tell the SoftLayer people. ;-) | 22:35 |
| markguz_ | hehe. well we also use bluemix for stuff. but for running the local lab infra openstack is perfect. lets me set stuff up self service for most folks. | 22:36 |
| jungleboyj | Yep. That is the way to go for the future. | 22:39 |
| markguz_ | yep :-) | 22:40 |
| woojay | jungleboyj: Would you have a moment to help me understand some of the failed tests on https://review.openstack.org/#/c/624233/ ? | 22:46 |
| jungleboyj | woojay: Looking. | 22:48 |
| woojay | Thank you. I goofed on pylint for sure. Just trying to see how to fix tox-docs, and how grenade-py3 andnova-multiattach are failing. | 22:49 |
| jungleboyj | woojay: So, this test case is failing: tempest.api.image.v2.test_images.BasicOperationsImagesTest.test_update_image[id-f66891a7-a35c-41a8-b590-a065c2a1caa6,smoke] | 22:50 |
| jungleboyj | tempest.lib.exceptions.UnexpectedContentType: Unexpected content type provided | 22:51 |
| jungleboyj | nova-multiattach also failed but we ahve been seeing issues there. | 22:54 |
| jungleboyj | Elastic recheck seems to think it knows why the grenade-py3 failure happened. | 22:55 |
| woojay | I figured on multiattach, since it passed on the first time. | 22:55 |
| *** redcavalier has left #openstack-cinder | 22:55 | |
| woojay | I did see that notice about elastic recheck. I do see it's not just me now http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22502%20PUT%5C%22%20AND%20message:%5C%22%2Fimage%2Fv2%2Fimages%2F%5C%22%20AND%20message:%5C%22%2Ffile%5C%22%20AND%20tags:%5C%22console%5C%22&from=7d | 22:55 |
| woojay | I saw a potential fix for that grenade bug merged to devstack a few days ago so I wasn't sure if that would still affect mine. | 22:56 |
| jungleboyj | Ah, and there is a problem with docs. | 22:56 |
| jungleboyj | You did or did not know what that was about. | 22:56 |
| *** mchlumsky has quit IRC | 22:57 | |
| woojay | I only just read about the potential fix regarding grenade just moments ago. | 22:57 |
| woojay | And I definitely am not sure how to fix the failure on the docs. | 22:58 |
| jungleboyj | Ok. So you need a recheck on the grenade failure but lets figure out the docs failure. | 22:58 |
| woojay | yes, that would be great. | 22:58 |
| *** smcginnis has joined #openstack-cinder | 23:07 | |
| *** ChanServ sets mode: +o smcginnis | 23:07 | |
| jungleboyj | have recreated the failure locally. Will take me a little bit to figure out what is wrong. It is a problem with the way your docstrings are. Have seen it before but can't remember what it is. | 23:09 |
| woojay | I see. I can also try to removing them and convert to regular comments, if that may help. | 23:10 |
| jungleboyj | Eh, give me a little time to play. | 23:11 |
| woojay | Thank you so much. | 23:11 |
| jungleboyj | Oh man. | 23:17 |
| jungleboyj | Found the problem. | 23:18 |
| jungleboyj | I will update the review. | 23:18 |
| woojay | I must have been super silly. | 23:18 |
| *** dave-mccowan has quit IRC | 23:19 | |
| jungleboyj | Typo in the support-matrix.ini file. | 23:19 |
| woojay | I see the comment. Thank you so much for that. | 23:19 |
| jungleboyj | Welcome. I knew I had seen it somewhere before. | 23:20 |
| *** dklyle has joined #openstack-cinder | 23:35 | |
| *** dklyle has quit IRC | 23:45 | |
| *** dcdamien has quit IRC | 23:47 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!