Friday, 2017-12-22

*** salv-orlando has quit IRC00:00
openstackgerritSean McGinnis proposed openstack/python-cinderclient master: Follow the new PTI for document build  https://review.openstack.org/52899700:07
*** david-lyle has quit IRC00:12
*** david-lyle has joined #openstack-cinder00:12
Roamer`mriedem, for the record, yes, Tempest's test_volumes_list.py was indeed born in May 2012 as a test for nova-volumes :)00:19
mriedemheh00:20
openstackgerritMerged openstack/python-cinderclient master: Remove setting of version/release from releasenotes  https://review.openstack.org/52064500:20
Roamer`mriedem, so would you actually prefer that I upload a new patchset with the method renamed w/o the camel case?00:21
Roamer`(I could do that if you'd like me to)00:21
mriedemdoesn't matter to me - then it would be inconsistent with the other method; up to tempest core i guess00:21
Roamer`ok00:21
Roamer`thanks for looking at it00:21
mriedemcould push a patch before the fix that renames the method and fixes the nova-volume reference00:22
mriedemthen put your fix on top with the non-camel case00:22
mriedemthen you get double the stackalytics hits!00:22
*** harlowja has quit IRC00:23
openstackgerritSean McGinnis proposed openstack/python-cinderclient master: Follow the new PTI for document build  https://review.openstack.org/52899700:23
Roamer`hm, I was going to (very selfishly) say that this would make it a bit harder to cherry-pick it for our CI, but of course I can cherry-pick two or three patches in quick succession just as easily as one00:24
openstackgerritJohn Griffith proposed openstack/cinder master: Enable fail back in SolidFire driver  https://review.openstack.org/52918200:27
*** yangyapeng has quit IRC00:28
*** chhavi__ has joined #openstack-cinder00:32
*** AlexeyAbashkin has joined #openstack-cinder00:33
*** chhavi__ has quit IRC00:36
*** AlexeyAbashkin has quit IRC00:37
*** wanghao has joined #openstack-cinder00:38
*** edmondsw has joined #openstack-cinder00:52
*** salv-orlando has joined #openstack-cinder00:56
*** edmondsw has quit IRC00:57
*** markvoelker has quit IRC00:58
openstackgerritTommyLike proposed openstack/cinder-specs master: Support restore backup to volumes simultaneously  https://review.openstack.org/52444101:02
*** salv-orlando has quit IRC01:02
*** markvoelker has joined #openstack-cinder01:10
*** ntpttr_laptop has quit IRC01:13
*** daidv has quit IRC01:25
*** daidv has joined #openstack-cinder01:25
openstackgerritYingzhe Zeng proposed openstack/cinder master: Huawei driver refactor  https://review.openstack.org/51412601:26
*** Apoorva has quit IRC01:29
openstackgerritYingzhe Zeng proposed openstack/cinder master: Huawei driver refactor  https://review.openstack.org/51412601:30
*** psachin has joined #openstack-cinder01:31
*** yangyapeng has joined #openstack-cinder01:32
*** AlexeyAbashkin has joined #openstack-cinder01:34
*** AlexeyAbashkin has quit IRC01:38
*** wanghao_ has joined #openstack-cinder01:48
*** wanghao has quit IRC01:51
*** liverpooler has joined #openstack-cinder01:54
openstackgerritTommyLike proposed openstack/cinder-specs master: Support restore backup to volumes simultaneously  https://review.openstack.org/52444101:54
*** salv-orlando has joined #openstack-cinder01:58
*** chhavi__ has joined #openstack-cinder02:01
*** salv-orlando has quit IRC02:03
*** zhenguo has joined #openstack-cinder02:13
*** linkmark has quit IRC02:19
*** liverpooler has quit IRC02:20
*** Apoorva has joined #openstack-cinder02:30
*** itlinux has quit IRC02:32
*** AlexeyAbashkin has joined #openstack-cinder02:34
*** Apoorva has quit IRC02:35
*** AlexeyAbashkin has quit IRC02:38
*** psachin has quit IRC02:40
*** mriedem has quit IRC02:40
*** erlon has quit IRC02:42
*** dalgaaf has quit IRC02:45
*** dalgaaf has joined #openstack-cinder02:47
*** psachin has joined #openstack-cinder02:54
*** salv-orlando has joined #openstack-cinder02:59
*** salv-orlando has quit IRC03:03
*** zhurong has joined #openstack-cinder03:09
*** diablo_rojo has joined #openstack-cinder03:12
*** lkuchlan has joined #openstack-cinder03:21
*** itlinux has joined #openstack-cinder03:21
*** crushil has joined #openstack-cinder03:29
*** crushil has quit IRC03:33
*** zhaochao has quit IRC03:36
openstackgerritKushal Wathore proposed openstack/cinder master: HPE 3PAR - Implement Tiramisu feature on 3PAR  https://review.openstack.org/52826603:38
*** wanghao_ has quit IRC03:53
*** salv-orlando has joined #openstack-cinder04:00
*** salv-orlando has quit IRC04:04
*** wanghao has joined #openstack-cinder04:08
*** markvoelker has quit IRC04:13
*** rcernin has quit IRC04:26
*** rcernin has joined #openstack-cinder04:27
*** edmondsw has joined #openstack-cinder04:28
*** threestrands_ has quit IRC04:29
*** zhurong has quit IRC04:32
*** edmondsw has quit IRC04:33
*** links has joined #openstack-cinder04:43
*** gmann has joined #openstack-cinder04:46
*** rcernin has quit IRC04:47
*** rcernin has joined #openstack-cinder04:49
*** Apoorva has joined #openstack-cinder04:56
*** salv-orlando has joined #openstack-cinder04:58
*** pgadiya has joined #openstack-cinder04:58
*** lbragstad has quit IRC04:59
*** imran_ansari has joined #openstack-cinder05:12
*** yangyapeng has quit IRC05:17
*** yangyapeng has joined #openstack-cinder05:17
*** diablo_rojo has quit IRC05:25
*** daidv has quit IRC05:26
*** daidv has joined #openstack-cinder05:26
*** lkuchlan has quit IRC05:31
*** yangyapeng has quit IRC05:36
*** yangyapeng has joined #openstack-cinder05:37
*** yangyapeng has quit IRC05:41
*** Apoorva has quit IRC05:43
*** gcb has quit IRC05:44
*** armax has quit IRC05:54
*** vivsoni has joined #openstack-cinder05:56
vivsoniHi Team, zuul job timed out06:03
vivsoniPOST-RUN START: [untrusted : git.openstack.org/openstack-infra/zuul-jobs/playbooks/sphinx/post.yaml@mas06:03
*** wanghao has quit IRC06:05
vivsoni2017-12-22 05:20:53.501951 | ubuntu-xenial | ERROR06:05
vivsoni2017-12-22 05:20:53.503254 | ubuntu-xenial | {06:05
vivsoni2017-12-22 05:20:53.503356 | ubuntu-xenial |   "failed": true,06:05
vivsoni2017-12-22 05:20:53.503442 | ubuntu-xenial |   "msg": "ssh: connect to host 158.69.88.212 port 22: Connection timed out\r\nrsync: connection unexpectedly closed (0 bytes received so far) [Receiver]\nrsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.1]\n",06:05
vivsoni2017-12-22 05:20:53.503531 | ubuntu-xenial |   "rc": 25506:05
*** gouthamr has quit IRC06:07
*** yangyapeng has joined #openstack-cinder06:07
*** markvoelker has joined #openstack-cinder06:14
*** edmondsw has joined #openstack-cinder06:16
*** edmondsw has quit IRC06:20
openstackgerritYingzhe Zeng proposed openstack/cinder master: Huawei driver refactor  https://review.openstack.org/51412606:28
*** alexchadin has joined #openstack-cinder06:31
*** zengyingzhe has joined #openstack-cinder06:31
*** felipemonteiro__ has joined #openstack-cinder06:35
-openstackstatus- NOTICE: Zuul.openstack.org is currently under heavy load and not starting new jobs. We're waiting for an admin to restart Zuul.06:42
*** markvoelker has quit IRC06:49
openstackgerritNeha Alhat proposed openstack/cinder master: V2/V3 jsonschema validation: snapshots  https://review.openstack.org/52099106:55
*** hemna_ has quit IRC06:56
openstackgerritDou Rui Yuan proposed openstack/cinder master: Add microversion and volume status limit to revert api ref doc  https://review.openstack.org/52979907:00
*** rcernin has quit IRC07:04
*** imran_ansari has quit IRC07:04
*** jdurgin has quit IRC07:11
openstackgerritYingzhe Zeng proposed openstack/cinder master: Huawei driver refactor  https://review.openstack.org/51412607:32
*** zhaochao has joined #openstack-cinder07:39
openstackgerritBrin Zhang proposed openstack/python-cinderclient master: Add origin to the pike branch of release-notes  https://review.openstack.org/52980307:51
*** AlexeyAbashkin has joined #openstack-cinder07:53
*** armaan has joined #openstack-cinder07:54
*** edmondsw has joined #openstack-cinder08:04
*** giodeco has joined #openstack-cinder08:06
*** edmondsw has quit IRC08:09
*** amoralej|off is now known as amoralej08:13
*** wanghao has joined #openstack-cinder08:13
*** AlexeyAbashkin has quit IRC08:15
*** stakeda has joined #openstack-cinder08:20
*** AlexeyAbashkin has joined #openstack-cinder08:21
*** daidv has quit IRC08:22
*** daidv has joined #openstack-cinder08:22
*** pcaruana has joined #openstack-cinder08:24
*** AlexeyAbashkin has quit IRC08:25
*** niraj_singh has joined #openstack-cinder08:26
*** AlexeyAbashkin has joined #openstack-cinder08:26
*** hoangcx has quit IRC08:26
*** hoangcx has joined #openstack-cinder08:26
*** abhishekk has joined #openstack-cinder08:27
openstackgerritBrin Zhang proposed openstack/cinder master: Add instructions for the Error response codes  https://review.openstack.org/52734508:27
niraj_singhsmcginnis:service token for cinder to nova patch is merged ref:https://review.openstack.org/#/c/524497/. There is another patch for cinder to glance interaction using service token ref: https://review.openstack.org/#/c/526611/. Please look into this.08:32
*** markvoelker has joined #openstack-cinder08:46
*** zhurong has joined #openstack-cinder08:54
*** wanghao_ has joined #openstack-cinder09:16
*** wanghao has quit IRC09:16
*** markvoelker has quit IRC09:19
*** felipemonteiro__ has quit IRC09:21
*** wanghao_ has quit IRC09:26
*** erlon has joined #openstack-cinder09:26
*** wanghao has joined #openstack-cinder09:26
*** wanghao has quit IRC09:27
*** wanghao has joined #openstack-cinder09:27
*** wanghao has quit IRC09:28
*** wanghao has joined #openstack-cinder09:28
*** wanghao has quit IRC09:28
*** wanghao has joined #openstack-cinder09:29
*** wanghao has quit IRC09:29
*** wanghao has joined #openstack-cinder09:29
*** wanghao has quit IRC09:30
*** wanghao has joined #openstack-cinder09:30
*** wanghao has quit IRC09:31
*** wanghao has joined #openstack-cinder09:31
*** wanghao has quit IRC09:31
*** wanghao has joined #openstack-cinder09:32
*** wanghao has quit IRC09:32
*** wanghao has joined #openstack-cinder09:33
*** wanghao has quit IRC09:33
*** wanghao has joined #openstack-cinder09:33
*** wanghao has quit IRC09:34
*** wanghao has joined #openstack-cinder09:34
*** imran_ansari has joined #openstack-cinder09:39
*** Jack_Iv has joined #openstack-cinder09:43
*** Jack_Iv has quit IRC09:45
*** Jack_Iv has joined #openstack-cinder09:45
*** edmondsw has joined #openstack-cinder09:52
*** edmondsw has quit IRC09:57
*** owalsh_ is now known as owalsh10:00
*** zhurong has quit IRC10:04
*** e0ne has joined #openstack-cinder10:04
*** daidv has quit IRC10:10
*** e0ne has quit IRC10:11
*** psachin has quit IRC10:12
*** e0ne has joined #openstack-cinder10:13
-openstackstatus- NOTICE: zuul has been restarted, all queues have been reset. please recheck your patches when appropriate10:15
*** markvoelker has joined #openstack-cinder10:16
*** imran_ansari has quit IRC10:25
*** stakeda has quit IRC10:34
*** hoangcx has quit IRC10:35
*** imran_ansari has joined #openstack-cinder10:35
openstackgerritImran Ansari proposed openstack/cinder master: 3PAR - Fix temp snapshot that remains post online copy  https://review.openstack.org/52418510:46
*** markvoelker has quit IRC10:50
*** abhishekk has quit IRC10:57
*** armaan has quit IRC11:05
*** armaan has joined #openstack-cinder11:06
openstackgerritBrin Zhang proposed openstack/cinder master: add enough notification for QoS  https://review.openstack.org/52248211:07
*** AlexeyAbashkin has quit IRC11:13
*** itlinux has quit IRC11:16
*** pgadiya has quit IRC11:31
*** imran_ansari has quit IRC11:31
*** AlexeyAbashkin has joined #openstack-cinder11:31
*** dalgaaf has quit IRC11:37
*** dalgaaf has joined #openstack-cinder11:38
*** edmondsw has joined #openstack-cinder11:41
*** edmondsw has quit IRC11:45
*** markvoelker has joined #openstack-cinder11:47
*** lkuchlan has joined #openstack-cinder11:50
*** lkuchlan has quit IRC11:56
*** imran_ansari has joined #openstack-cinder11:57
*** alexchadin has quit IRC11:58
*** alexchad_ has joined #openstack-cinder12:02
*** alexchad_ has quit IRC12:06
*** alexchadin has joined #openstack-cinder12:08
*** alexchadin has quit IRC12:13
*** erlon has quit IRC12:16
openstackgerritChandan Kumar proposed openstack/cinder-tempest-plugin master: Refactored cinder tempest tests to cinder_tempest_plugin  https://review.openstack.org/52982412:17
*** markvoelker has quit IRC12:20
*** zhaochao has quit IRC12:29
*** gmann has quit IRC12:33
*** takedakn has joined #openstack-cinder12:43
*** armaan_ has joined #openstack-cinder12:45
*** imran_ansari has quit IRC12:46
*** Jack_Iv has quit IRC12:46
*** armaan has quit IRC12:47
*** linkmark has joined #openstack-cinder12:47
*** Jack_Iv has joined #openstack-cinder12:47
*** Jack_Iv has quit IRC12:52
*** dave-mccowan has joined #openstack-cinder12:57
openstackgerritMerged openstack/cinder master: Add service_token for cinder-glance interaction  https://review.openstack.org/52661113:00
*** Jack_Iv has joined #openstack-cinder13:08
*** armaan_ has quit IRC13:08
*** armaan has joined #openstack-cinder13:09
*** Jack_Iv has quit IRC13:10
*** Jack_Iv has joined #openstack-cinder13:11
*** gnufied has quit IRC13:14
*** Jack_Iv has quit IRC13:15
*** dave-mcc_ has joined #openstack-cinder13:16
*** markvoelker has joined #openstack-cinder13:17
*** imran_ansari has joined #openstack-cinder13:18
openstackgerritChandan Kumar proposed openstack/cinder-tempest-plugin master: Refactored cinder tempest tests to cinder_tempest_plugin  https://review.openstack.org/52982413:19
*** dave-mccowan has quit IRC13:19
*** owalsh is now known as owalsh_afk13:23
*** Jack_Iv has joined #openstack-cinder13:23
*** giodeco has quit IRC13:26
*** Jack_Iv has quit IRC13:30
*** Jack_Iv has joined #openstack-cinder13:31
*** imran_ansari has quit IRC13:31
*** imran_ansari_ has joined #openstack-cinder13:31
*** yangyapeng has quit IRC13:36
openstackgerritMerged openstack/cinder master: Remove deprecated HP 3PAR config options  https://review.openstack.org/52830013:37
*** yangyapeng has joined #openstack-cinder13:37
*** Jack_Iv has quit IRC13:37
*** Jack_Iv has joined #openstack-cinder13:38
*** mriedem has joined #openstack-cinder13:38
*** markvoelker has quit IRC13:40
*** markvoelker has joined #openstack-cinder13:40
*** yangyapeng has quit IRC13:42
*** stephenfin is now known as finucannot13:42
*** links has quit IRC13:45
*** takedakn has quit IRC13:59
*** imran_ansari_ has quit IRC14:09
*** amoralej is now known as amoralej|lunch14:10
*** dalgaaf has quit IRC14:11
*** stvnoyes has joined #openstack-cinder14:13
*** _ix has joined #openstack-cinder14:13
_ixGood morning friends... I'm looking for a little help with my cinder implementation.14:17
openstackgerritxing-yang proposed openstack/cinder master: Bump up the API microversion in cinder.rc  https://review.openstack.org/52983514:18
_ixIt's a little bit older, to be fair, but I've got a number of cinder nodes, and one is unhealthy right now. I've got some machines that are using a volume mounted as /dev/vda. Networking connectivity was lost to the iscsi network, and the targets cannot be found.14:20
_ixCan someone point me in the right direction to get back on the right track?14:20
smcginnis_ix: Is this using the LVM backend?14:20
_ix| e667f807-624c-4db1-becf-1e9930cedbd6 | vltrader01-1              | error     |    8 | Attached to vltrader01 on /dev/vda           |14:21
_ix| d985d40e-bce1-44fe-9312-88a5c9f2d1ce | vlproxy01-1               | error     |    8 | Attached to vlproxy01 on /dev/vda            |14:21
_ixSorry!14:21
_ixYes, it's lvm-backed.14:21
smcginnisOn the LVM host, you can run "sudo tgt-admin -s" to get a list of iSCSI targets that are exposed.14:22
smcginnis_ix: Did you try something on the volumes that put them in error state?14:23
_ixSo far, we've been avoiding doing anything to the volume host itself... largely out of fear.14:24
_ixShould the tgt-admin tool be generally available on this host in a standard installation?14:24
_ix(There's some history here... this is an inherited mess.)14:24
smcginnis_ix: Unless it was configured to use a different target driver, the default LVM backend uses the tgt-admin tool to set up and manage the iSCSI targets for the LVM volumes.14:25
smcginnis_ix: The "-s" will just list what is configured. It won't modify anything.14:25
*** yangyapeng has joined #openstack-cinder14:25
smcginnisIf that is not there, then we need to check your /etc/cinder/cinder.conf file to see what is configured as the target driver.14:26
*** yangyapeng has quit IRC14:26
smcginnis_ix: Do you know what version this is?14:26
*** yangyapeng has joined #openstack-cinder14:26
_ixIt's a liberty release.... 7x14:26
_ixI see that the volume service logs the command cinder-rtstool14:26
smcginnis_ix: Oh, OK. Then it's using LIO, not tgt. Let me check the equivalent there.14:27
_ixThanks so much for any help you can lend.14:27
smcginnis_ix: OK, I don't have as much experience with LIO, but looks like you would need to run "sudo cinder-rtstool get-targets"14:29
_ixOK. And this will show me what targets actually exist.14:30
_ixOr rather, are exposed?14:30
smcginnis_ix: I believe so.14:30
_ixOK.14:30
_ixsmcginnis: Can we still call these targets, or is there a different term here?14:31
_ixThe _target_ if you'd like that I'm looking for appears in this output.14:31
smcginnis_ix: OK, if it lists that target, then the nova host should be able to log in to it and access the volume.14:32
_ixnova-compute is complaining that the NovaException: iSCSI device not found at /dev/disk/by-path/ip-10.12.128.14:3260-iscsi-iqn.2010-10.org.openstack:volume-cd58909d-230b-4350-8757-78b222f89463-lun-0 -- that file does exist on the nova host, but there is still no success in getting that guest to boot from that volume.14:33
openstackgerritxing-yang proposed openstack/cinder master: Deprecate CG APIs  https://review.openstack.org/52693214:34
smcginnis_ix: On the nova host, you can run "iscsiadm -m node" to get a list of sessions.14:34
_ixI spoke out of turn. That file no longer exists?14:34
_ixsmcginnis: Running that command shows the target that I'm looking for in session.14:35
smcginnis_ix: Try "iscsiadm -m node -l" to make sure it's logged in.14:35
_ixNo records found.14:36
smcginnis_ix: Hmm. You could try manually logging out, perform discover again, then manually logging in.14:38
smcginnis_ix: Make sure you have that target info.14:38
_ixOK. So, if I maintain the list of targets from the previous commmand...14:39
_ix`iscsi -m node` for sessions with the targets.14:39
smcginnis_ix: Then do "iscsiadm -m node -u -T iqn.2010-10.org.openstack:<volname> -p 172.23.10.240:326014:39
smcginnis_ix: Then, iscsiadm -m discovery -t st -p 172.23.10.240:326014:39
smcginnis_ix: Then, iscsiadm -m node -l14:40
_ixWhere do you get that 172.23.10.240 address? Would that be replaced with my 10.12.128.14 address?14:41
_ix(cinder host)14:41
*** mchlumsky has joined #openstack-cinder14:41
smcginnis_ix: Yes, that would be whatever your target iSCSI port is.14:41
_ixOK14:41
*** kwathore__ has joined #openstack-cinder14:45
_ixAlright. I was mistaken. I reported that there were no records found. I was paged some 8 hours ago to start looking at these problems, and only getting around to the volume portion. Forgive me.14:45
_ixThere are records found above.14:46
smcginnis_ix: No worries. I know how it is to inherit someone else's old mess. ;)14:46
smcginnis_ix: So there is an active iSCSI session to your volume's target?14:47
_ixI don't know if I should be happy that I'm in good company, or feel bad for you.14:47
_ixIt looks that way.14:47
_ixRunning with the -l option says that it can't login, at least on that target.14:47
_ixinitiator reported error (24 - iSCSI login failed due to authorization failure)14:47
smcginnis_ix: Oh, that is a problem then.14:48
_ixGo on.14:48
*** kwathore_ has quit IRC14:48
smcginnis_ix: It looks like it must be set up to use a CHAP password.14:49
_ixA CHAP password you say.14:49
_ixSeems fine on some other targets, though.14:50
smcginnis_ix: I actually haven't dealt much with CHAP, but there's some details here: https://wiki.debian.org/SAN/iSCSI/open-iscsi14:50
smcginnisDoes the previous cinder-rtstool output show CHAP credentials?14:51
_ixNot that I can see.14:52
smcginnis_ix: Unfortunately I have a meeting coming up. Let's see if anyone else shows up with some advice. If I get the time I'll try to look for some references.14:54
_ixHey, that's cool. I think I might have enough to get started here. Thanks very much. I'm largely unaware of how this service works, so it's fine opportunity to learn more.14:55
openstackgerritMerged openstack/cinder master: Storwize: disable creating volume with non cg_snapshot group_id  https://review.openstack.org/51756114:55
_ixEnjoy your meeting.14:55
_ixThanks very much all the same.14:55
smcginnis_ix: No problem, good luck!14:56
*** edmondsw has joined #openstack-cinder14:56
*** edmondsw has quit IRC14:57
*** amoralej|lunch is now known as amoralej14:58
_ixFriends, could my initiator name be completely wrong?15:04
*** Jack_Iv has quit IRC15:08
*** KeithMnemonic has joined #openstack-cinder15:13
smcginnis_ix: It cause problems if the host's IQN was modified. Do you have reason to believe that might have happened?15:16
_ixThe authorization failures, namely.15:16
_ixI don't believe there are any CHAP credentials at play here.15:17
kwathore__@Team: Please review : https://review.openstack.org/#/c/528266/15:18
_ixAlso, is the target service supposed to be running?15:18
*** armax has joined #openstack-cinder15:18
_ix`targetcli status` hangs on the troubled host.15:20
*** mriedem has quit IRC15:21
Roamer`jungleboyj, maybe others: was the procedural -2 placed on https://review.openstack.org/#/c/220155/ based on the December 6th milestone?  I'm asking since there was some vague hope of treating this as a reintroduction of a driver, not a new one... and I know that right now is probably not a good time for a discussion, holidays and people being away and all, so just asking whether it was based on t15:26
Roamer`he Dec 6 milestone15:26
jungleboyjRoamer` smcginnis:  Had mentioned that you had reached out about the driver but then things went silent again around Milestone 2 and the goals hadn't been met.15:27
Roamer`well, the truth is that we took a bit longer than expected to set up the CI system, but it became pretty much operational several days ago (although right now I'm dealing with some weird slowdown that makes some runs time out)15:29
*** itlinux has joined #openstack-cinder15:35
*** mriedem has joined #openstack-cinder15:38
*** armaan has quit IRC15:44
*** armaan has joined #openstack-cinder15:44
*** gouthamr has joined #openstack-cinder15:56
jungleboyjRoamer`:  Understood.15:56
jungleboyjSorry, in another meeting as well right now.15:57
_ixAny danger in bouncing a cinder host that's been failing all morning?16:05
_ixWill it be in anyway out of sync with the rest of the openstack infrastructure?16:06
_ix(I suppose, any further than it already is)16:06
smcginnis_ix: If no hosts are accessing the volumes from it, should be fine. Might be good at this point.16:06
jungleboyjsmcginnis:  ++16:07
_ixsmcginnis: I think there is maybe... one host.16:07
_ixYes, somehow, one host is still able to talk to it correctly.16:07
_ixOne volume is mounted correctly.16:08
smcginnis_ix: If there are any instances using a volume from there you should shut it down first. Then after the cinder node bounces you can start it up again.16:08
_ixOK.16:08
jungleboyjRoamer`:  So, your CI is working now?16:10
*** arnewiebalck has quit IRC16:11
*** itlinux has quit IRC16:12
Roamer`jungleboyj, yes, still in silent mode; I'll make it post comments in the next day or so after I've figured out the slowdown thing16:14
*** armax has quit IRC16:16
*** Jack_Iv has joined #openstack-cinder16:18
jungleboyjRoamer`:  So, my concern here is that we are way beyond the deadline and things have been very quiet from your end.  I am concerned  if we let this back in we are going to end back up where we were before with your CI not working.16:18
*** Jack_Iv has quit IRC16:23
jgriffithtommylikehu: I added some comments/questions and I guess maybe a suggestion here:  https://review.openstack.org/#/c/524441/16:28
*** imran_ansari has joined #openstack-cinder16:37
*** AlexeyAbashkin has quit IRC16:39
openstackgerritMerged openstack/cinder master: Follow the new PTI for document build  https://review.openstack.org/52900616:41
*** Apoorva has joined #openstack-cinder16:44
*** armaan has quit IRC16:55
*** chhavi__ has quit IRC16:56
*** armaan has joined #openstack-cinder16:56
*** mchlumsky has quit IRC16:59
*** itlinux has joined #openstack-cinder17:00
openstackgerritMerged openstack/cinder master: Bump up the API microversion in cinder.rc  https://review.openstack.org/52983517:04
*** itlinux has quit IRC17:05
*** itlinux has joined #openstack-cinder17:05
*** imran_ansari has quit IRC17:08
*** imacdonn has quit IRC17:12
*** imacdonn has joined #openstack-cinder17:12
*** _ix has quit IRC17:14
*** imran_ansari has joined #openstack-cinder17:14
*** ntpttr_laptop has joined #openstack-cinder17:20
*** ntpttr_laptop has quit IRC17:21
*** ntpttr_laptop__ has joined #openstack-cinder17:21
*** rmcall has joined #openstack-cinder17:22
openstackgerritMerged openstack/cinder master: HPE 3PAR - Implement Tiramisu feature on 3PAR  https://review.openstack.org/52826617:22
*** rmcall has quit IRC17:26
*** Apoorva has quit IRC17:26
openstackgerritMerged openstack/cinder master: Add online data migration routine for attachment_specs  https://review.openstack.org/52916017:31
*** mriedem is now known as bauzas17:40
*** bauzas is now known as mriedem17:42
*** AlexeyAbashkin has joined #openstack-cinder17:59
*** AlexeyAbashkin has quit IRC18:03
*** _ix has joined #openstack-cinder18:11
*** diablo_rojo has joined #openstack-cinder18:14
*** _ix has quit IRC18:16
*** _ix has joined #openstack-cinder18:16
*** _ix_ has joined #openstack-cinder18:22
*** _ix has quit IRC18:23
*** jgriffith is now known as jgriffith-merryC18:34
*** jgriffith-merryC is now known as jgriffith18:34
*** imran_ansari has quit IRC18:35
*** armax has joined #openstack-cinder18:41
*** dave-mcc_ has quit IRC18:41
*** rmcall has joined #openstack-cinder18:46
*** felipemonteiro__ has joined #openstack-cinder18:48
_ix_OK. And now we're at the data center after bouncing the cinder box in question.18:49
_ix_After the reboot, we were generally in good shape, but there's one volume that refuses to behave.18:50
jgriffith_ix_: what does *behave* mean?18:53
*** imran_ansari has joined #openstack-cinder18:53
_ix_From earlier, the logs were chock full of messages about the local initator not being authorized to access certain targets.18:54
_ix_Now, we have everything humming along except for one machine and its two volumes, including its OS disk.18:54
jgriffith_ix_: can you `telnet <tgt-ip> 3260` from that one node?18:56
_ix_cinder -> nova or nova -> cinder?18:57
jgriffithnova-->cinder18:57
jgriffithbut it's not really *cinder* it's the configured tgt IP address18:57
jgriffithwhich isn't necessarily (and in production envs should NOT be) the same as the cinder node itself18:57
jgriffithof course I'm assuming based on what you've described you're using LVM driver?18:58
* jgriffith wasn't around for your earlier conversations18:58
_ix_We've got an iscsi vlan setup and I believe it's lvm driver with lio helper, if that sounds right.18:58
jgriffith_ix_: yup, and vlans is probably what's not configured properly from your compute node would be my guess18:59
_ix_No big deal. Just an on-call dude on the 13th hour of an issue after an hour or so of sleep. I apologize if I'm a bit slow to think through some of these things.18:59
_ix_jgriffith: It looks like 3260 is generally available.18:59
jgriffithnah, lots of moving parts and variables.. even after a full nights sleep :)18:59
_ix_no more moving parts18:59
_ix_Just blankets that move to cover my head.19:00
jgriffithhaha19:00
jgriffith_ix_: if it were me I'd try creating a new volume and attaching it to that compute node19:00
*** harlowja has joined #openstack-cinder19:01
jgriffithif that works that could mean the model info on the failing volumes is honked up19:01
jgriffithit it doesn't, it means you still may have a basic connectivity issue19:01
_ix_Thanks for the tip.19:01
_ix_Alright. I've got a colleague working on that. Meanwhile, `iscsiadm -m Node` on the hypervisor doesn't include the target that it's fialing on. I forogt to inquire, I shouldn't expect to see that until it's fully attached, right?19:03
_ix_s/Node/node19:03
*** harlowja has quit IRC19:04
jgriffith_ix_: I find it easier to just look in /etc/iscsi/nodes19:04
jgriffithare you hung in *attaching*?19:04
_ix_hey, that does sound easier, but... I've got no /etc/iscsi/nodes19:05
*** ntpttr_laptop__ has quit IRC19:06
_ix_I think it's effectively failing to attach... it's a volume-based node, so, it won't boot with /dev/vda not found.19:07
*** imran_ansari has quit IRC19:07
*** ntpttr_laptop has joined #openstack-cinder19:09
*** gouthamr has quit IRC19:20
*** felipemonteiro__ has quit IRC19:27
_ix_It looks like cinder thinks a volume is attached, but the rest of openstack thinks that it's not attached.19:28
_ix_How do I bring reality to cinder's world?19:29
smcginnis_ix_: Did you try just detaching it?19:29
*** amoralej is now known as amoralej|off19:30
_ix_I have not, but perhaps a colleague did?19:30
*** armax has quit IRC19:32
*** AlexeyAbashkin has joined #openstack-cinder19:33
_ix_smcginnis: Felt like we were rigt where we wanted to be, but... this guy is a big fan of circumventing the comfort of the openstack wrapping and using the commands that openstack depends on.19:33
_ix_It leads to nice side effects like this.19:34
smcginnis_ix_: Slap his hands. :)19:34
smcginnis_ix_: Unless you need to do some fixing up or troubleshooting, if it's being managed by OpenStack then it's best to only use OpenStack.19:34
* _ix_ nods19:34
smcginnisAnd not much around with things and getting things out of sync.19:35
smcginniss/much/muck/19:35
_ix_That's how I like doing it, but indiscipline got us into this, could it also be the way out?19:35
smcginnis_ix_: Might be at this point.19:35
smcginnis_ix_: What happens if you try to detach the volume?19:35
_ix_OpenStack doesn't think it's attached.19:35
_ix_There's nothing to detach.19:36
smcginnis_ix_: I thought you said you had a volume and Cinder thought it was attached, but the host does not see the volume.19:36
_ix_I must not be typing well.19:36
_ix_cinder thinks it attached, openstack does not, nor does the end node.19:37
smcginnisI've been a bit distracted by other things, so I probably didn't read it right.19:37
_ix_That's just fine. I appreciate your support.19:37
*** AlexeyAbashkin has quit IRC19:37
smcginnis_ix_: OK, can you explain what you mean by cinder thinks it's attached but openstack does not. Cinder is part of openstack, so not sure what that means.19:37
_ix_I mean, it's like a body. When my foot falls asleep, it's still there... and what about phantom limbs?19:38
_ix_Perhaps the analogy wears thin.19:39
smcginnis_ix_: Haha, you're getting to abstract for me. :)19:39
*** dave-mccowan has joined #openstack-cinder19:39
_ix_Let's make it more concrete. Nova doesn't think it's attached. The node doesn't think it's attached. Cinder doesn't know about the attachment. But the underlying lvm says it's busy.19:39
jgriffith_ix_: WTF?19:40
_ix_jgriffith: You don't like my body analogy?19:40
*** gouthamr has joined #openstack-cinder19:40
jgriffith_ix_: what does `cinder list` return as the status of said volume?19:40
jgriffithno riddles please :)19:41
_ix_It's available.19:42
jgriffithOk, then what do you mean "cinder thinks it's attached"?19:42
_ix_I ammended my previous statement, I apologize.19:43
_ix_^^ Cinder doesn't know about the attachment. But the underlying lvm says it's busy.19:43
jgriffith_ix_: can you run "nova volume-attach <volume-id> <instance-id>"19:43
smcginnis_ix_: OK, so cinder-rtstool list-targets has the volume exported, but Cinder does not know about that?19:43
jgriffithsmcginnis: ahh19:43
jgriffithhanging connection outside of the control plane... naughty naughty19:44
_ix_smcginnis: I don't think I get list-targets with my version of cinder.19:46
smcginnis_ix_: I thought you had that this morning when we checked. Is this a different one?19:47
_ix_Let me check what you gave me to run.19:47
smcginnisOh, sorry, get-targets, not list-targets.19:47
_ix_Just `iscsiadm -m node`19:48
_ix_Oh, get.19:48
_ix_Yeah, that target is not in the list.19:49
smcginnisSo... next point of confusion I guess. What do you mean by "underlying lvm says it's busy".19:49
smcginnisCinder doesn't have it attached and there's no target created for it, so on the Cinder side it sounds like things are as they should be.19:50
_ix_`dmsetup info /dev/vg_cinder/volume-4d47c12a-d944-4031-aae2-84c503e55df4` seems to show that it's attached to something.19:50
_ix_I don't know what kind of underlying commands were run by this other engineer, but he was recommending that I potentially run a force unmount with the dmsetup tools.19:52
_ix_dmsetup remove?19:52
_ix_I feel like this is what it would be like if Ken M. was looking for OpenStack help, you just replace every instance of 'colleague' and 'other engineer' with 'My grandson'19:53
smcginnis_ix_: What about the dmsetup output are you seeing that's a problem?19:53
*** salv-orlando has quit IRC19:54
_ix_Well, my grandson told me that this output indicates that it's in use19:55
_ix_https://paste.ubuntu.com/26234647/19:55
*** salv-orlando has joined #openstack-cinder19:55
smcginnis_ix_: Looks fine to me.19:55
_ix_Yeah? OK. Let me give it a go and we'll see about the error output.19:56
*** kwathore__ has quit IRC19:56
*** vivsoni has quit IRC19:56
*** jmccrory has quit IRC19:59
*** jmccrory has joined #openstack-cinder19:59
*** salv-orlando has quit IRC19:59
_ix_OK... it fails silently in the console:20:00
_ix_annot configure StorageObject because device /dev/vg_cinder/volume-4d47c12a-d944-4031-aae2-84c503e55df4 is already in use20:00
*** gouthamr has quit IRC20:00
_ix_How do I make it not in use?20:00
jgriffith_ix_: you're seeing that in the Nova logs?20:00
_ix_That's in the cinder log.20:01
jgriffithcan you paste said logs?20:01
_ix_nova throws the unhelpful ClientException: The server has either erred or is incapable of performing the requested operation.20:02
smcginnisLooks like an error coming back from rtslib, but I don't know enough about LIO to know what that means.20:02
_ix_Yeah, I ca do that.20:02
jgriffith_ix_: BTW, did you try as I mentioned earlier to just create a *new* volume and try and attach it?20:02
smcginnisWould be a good test. ^^20:02
_ix_Oh, thanks for reminding me.20:02
_ix_I asked my manager to do it, and he said it didn't work, but he wasn't sure if he did it correctly.20:02
jgriffithgiven there's been some "out of band" messing with targets etc it would be good to isolate a few things20:02
_ix_Let me go give this a shot.20:02
_ix_I have a number of cinder servers though. How can I target this cinder host for volumes?20:03
jgriffithif you have types setup you can use those, if you don't you're SOL20:03
_ix_So, I might have to create a bunch of 1gb volumes until one landed on this host.20:04
imacdonncould use AZs, perhaps ? (not sure what the context is here)20:04
_ix_NEat.20:04
jgriffithor disable services on other nodes temporarily, or create a type for that specific backend, or create a unique capability, or create it manually and migrate it in, or.....20:05
jgriffithimacdonn: unfortunately in most multi c-vol deployments with cinder/lvm they're all going to be in the same AZ20:06
imacdonnOK... thought maybe we were just looking for a way to isolate a test case20:07
jgriffithimacdonn: yeah... and that might work too20:08
_ix_Just created 10 volumes and they all landed on the same host.20:09
_ix_Not good.20:09
smcginnis_ix_: I'm assuming you have data you need on this volume that says in-use, right?20:11
_ix_Well, yes.20:11
*** niraj_singh has quit IRC20:11
smcginnis_ix_: I thought so, but worth asking.20:11
_ix_Haha20:11
_ix_We're into a 14 hour page at this point.20:12
*** gaurangt has quit IRC20:13
*** niraj_singh has joined #openstack-cinder20:13
*** pcaruana has quit IRC20:13
*** e0ne has quit IRC20:16
*** mvk has quit IRC20:17
_ix_I've to take a break from this nonsense.20:17
*** gaurangt has joined #openstack-cinder20:19
imacdonnspeaking of volume types .... anyone familiar with the way devstack/tempest use them? I'm trying to make sense of it for my third party driver CI20:20
imacdonnseems that default_volume_type defaults to lvmdriver-120:20
imacdonnif I set it to something else in local.conf (for my driver), how do I make sure that that type gets created ?20:21
jgriffithimacdonn: do you have a section kinda like this:  https://github.com/j-griffith/sos-ci/blob/master/sos-ci/templates/localconf.base#L4420:22
smcginnis_ix_: I googled that error and it sent me here: https://bugzilla.redhat.com/show_bug.cgi?id=113944120:22
openstackbugzilla.redhat.com bug 1139441 in lvm2 "LVM should not autoactivate nested LVs (see comment 6)" [Medium,Closed: notabug] - Assigned to lvm-team20:22
smcginnis_ix_: Is this maybe an old RHEL server? Comment 8 looks like it might be a possible solution.20:23
imacdonnjgriffith: sorta ... I don't have all of that ... some other useful clues in there (for other issues I've been having)20:23
jgriffithimacdonn: essentially you need your own version of everything from L#44 on down20:23
jgriffithand don't have any LVM config (unless you want to do multi-backend testing)20:23
imacdonnjgriffith: right ... so ... then ... what part of that determines the volume_type used in the tests ?20:24
jgriffithmulti-backend testing requires *other* special things20:24
jgriffithimacdonn: that setup I showed you there will create a default that points to your driver20:24
*** edmondsw has joined #openstack-cinder20:25
jgriffithimacdonn: and as long as you don't do multi-backend the default-type is kind of irrelevant anyway20:25
jgriffithas far as Tempest is concerned at least20:25
_ix_smcginnis: Hey. That looks neat.20:25
_ix_Changes to lvm.conf... what kind of services need to restart under those circumstances?20:25
imacdonnjgriffith: well, it's apparently not irrelevant, since I had tests failing due to the default volume_type not existing20:25
smcginnisjgriffith: So you know what needs to be restarted after editing lvm.conf?20:26
smcginniss/So/Do/20:26
smcginnisDo si do20:26
jgriffithimacdonn: ok, well I tried to help20:27
jgriffithimacdonn: I pointed you to what you need for a conf file20:27
imacdonnjgriffith: sorry, didn't mean to sound unappreciative20:27
jgriffithnah, no worries20:28
jgriffithmy point was you're configuring multiple backends in your ci20:28
jgriffithif you fix that, this isn't a big deal20:28
jgriffithif you use the template I pointed you to, you'll see your cinder.conf ends up looking like this:  http://54.164.167.86/solidfire-ci-logs/refs-changes-82-529182-3/logs/etc/cinder/cinder.conf20:28
jgriffithwhich check out "default_volume_type"20:29
imacdonnjgriffith: What determines if it's multibackend or not20:29
imacdonnjgriffith: ?20:29
jgriffithand enabled_backends20:29
jgriffithimacdonn: look at your CI's cinder.conf20:29
jgriffithwhat do you have under "enabled_backends"20:29
imacdonnjgriffith: I have the name of my backend ..as you do ("solidfire" in your case)20:30
jgriffithand do you have an [lvm-driver-1] (or something like it) stanza20:30
jgriffithhmm.. but no default_volume_type entry?20:30
*** ntpttr_laptop has quit IRC20:30
*** _ix_ has quit IRC20:30
jgriffithimacdonn: can you paste your local.conf and cinder.conf20:31
imacdonnjgriffith: I'm really trying to understand what magic is happening under the covers right now ... e.g. does something (devstack/tempest) create a volume_type matching the volume_backend_name of the enabled backend? or something20:34
jgriffithimacdonn: https://github.com/openstack-dev/devstack/blob/master/lib/cinder#L26720:35
jgriffithdefault_name which is set from CINDER_ENABLED_BACKENDS in your local.conf20:36
smcginnisimacdonn: Yeah, devstack should create that default volume type on setup.20:36
imacdonnOK, trying stack.sh with a new local.conf now .. will see what I get .. thanks20:37
jgriffithwell it's been a hell of a year...20:37
jgriffithThanks everyone!  Merry Christmas!!20:37
imacdonnMerry Christmas to you too :)20:38
imacdonnjenkins@slcai477:~$ openstack volume type list20:50
imacdonn+--------------------------------------+-------------+-----------+20:50
imacdonn| ID                                   | Name        | Is Public |20:50
imacdonn+--------------------------------------+-------------+-----------+20:50
imacdonn| ca4afd85-536e-43e9-85e5-ac8d4f7eba2b | zfssa_iscsi | True      |20:50
imacdonn+--------------------------------------+-------------+-----------+20:50
imacdonnjenkins@slcai477:~$ grep default_volume_type /etc/cinder/cinder.conf20:50
imacdonndefault_volume_type = zfssa_iscsi20:50
imacdonnjenkins@slcai477:~$20:50
imacdonnI guess the magic worked .. somehow ;)20:50
*** salv-orlando has joined #openstack-cinder20:55
*** edmondsw has quit IRC20:58
*** wanghao_ has joined #openstack-cinder20:59
*** salv-orlando has quit IRC21:00
*** wanghao has quit IRC21:02
*** e0ne has joined #openstack-cinder21:13
*** markvoelker has quit IRC21:20
*** e0ne has quit IRC21:43
*** e0ne has joined #openstack-cinder21:46
*** salv-orlando has joined #openstack-cinder21:50
*** e0ne has quit IRC21:54
*** edmondsw has joined #openstack-cinder22:12
*** edmondsw has quit IRC22:17
Roamer`jungleboyj, sorry for disappearing like that, there was a family dinner that I had to run to22:23
Roamer`jungleboyj, I understand your concerns. Thing is, the commitment from our side is different now, much firmer, partly because of demand from our customers22:25
Roamer`I am not sure what I have to offer to prove it, other than promises to really keep track of our CI and our driver now, and to try to take more part in the whole community process22:26
jungleboyjThat is good22:26
jungleboyjIt was appreciated that you pulled your driver out voluntarily when you were unable to continue maintaining at22:26
jungleboyjsmcginnis:  what is the date for milestone 322:29
Roamer`according to https://releases.openstack.org/queens/schedule.html that would be January 22-2622:30
*** itlinux has quit IRC22:35
*** dave-mccowan has quit IRC22:35
*** salv-orlando has quit IRC22:40
*** stvnoyes has quit IRC22:40
jungleboyjRoamer`: thank you. Not where I could look that up right now.22:40
jungleboyjWhen do you think your C I will be stable enough to be reporting?22:41
smcginnisjungleboyj: Yep, Jan 25 to be exact.22:42
jungleboyjsmcginnis: thank you.22:43
Roamer`jungleboyj, I think it'll start reporting within the next day or two, a week at the most, but most probably tomorrow22:43
jungleboyjRoamer`: okay. Please let me know when it is stable. I will take a look at your driver. We can talk next week about possibly getting it in since it is a driver that is being added back in.22:45
Roamer`jungleboyj, thanks a lot!22:45
jungleboyjsmcginnis: that okay with you?22:45
smcginnisjungleboyj: Sure, seems reasonable.22:47
jungleboyjRoamer`: things need to look good though in the next week or so. Do not want to push anything closer to mile Stone 322:48
jungleboyjsmcginnis: good. Thank you for the input.22:49
Roamer`jungleboyj, acknowledged; a very reasonable position, thanks again22:49
jungleboyjYou are welcome. Just call me Santa Claus. I try to keep everyone on the nice list.22:51
Roamer`hm, and, yeah, sorry if this "very reasonable position" reply sounded a bit weird - I did *not* mean "it is very reasonable of you to finally come around to the idea of letting our driver back in", I meant "it is very reasonable of you to place a deadline", of course :)22:52
*** KeithMnemonic has quit IRC22:56
smcginnisHaha22:56
jungleboyjI didn't read anything weird into your response. It is all good.23:12
jungleboyjsmcginnis:  what are you laughing and want coal in your stocking23:19
*** markvoelker has joined #openstack-cinder23:21
jungleboyjsmcginnis: what are you laughing at? Do you want coal in your stocking?23:29
jungleboyjOops. Didn't mean for that to be sent twice.23:30
*** salv-orlando has joined #openstack-cinder23:41
*** salv-orlando has quit IRC23:46
*** markvoelker has quit IRC23:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!