Friday, 2019-09-06

« Thursday, 2019-09-05 Index Saturday, 2019-09-07 »
*** lseki has quit IRC00:33
*** rosmaita has quit IRC00:38
hemnaRoamer`: yes, that way when you do pip install[all]  (that installs ALL deps)00:46
hemnaand when you do pip install[purestorage] only installs the purestorage deps00:47
*** rosmaita has joined #openstack-cinder00:51
*** markvoelker has joined #openstack-cinder00:51
*** markvoelker has quit IRC00:56
*** brinzhang_ has joined #openstack-cinder00:57
*** brinzhang has quit IRC01:00
*** whfnst has joined #openstack-cinder01:06
*** spsurya has joined #openstack-cinder01:28
*** enriquetaso has joined #openstack-cinder02:30
*** mvkr has joined #openstack-cinder03:34
*** vishakha has joined #openstack-cinder04:22
*** enriquetaso has quit IRC04:23
*** Luzi has joined #openstack-cinder04:52
*** abishop has quit IRC04:54
*** abishop has joined #openstack-cinder05:07
*** markvoelker has joined #openstack-cinder05:15
*** markvoelker has quit IRC05:20
*** pcaruana has joined #openstack-cinder05:25
*** lpetrut has joined #openstack-cinder05:49
*** udesale has joined #openstack-cinder05:54
*** itlinux has joined #openstack-cinder06:01
*** sahid has joined #openstack-cinder06:02
*** markvoelker has joined #openstack-cinder06:06
*** markvoelker has quit IRC06:10
*** brinzhang_ has quit IRC06:14
*** brinzhang_ has joined #openstack-cinder06:15
*** brinzhang_ has quit IRC06:16
*** lpetrut has quit IRC06:16
*** lpetrut has joined #openstack-cinder06:16
openstackgerritPeter Penchev proposed openstack/cinder master: StorPool: update the driver requirements.  https://review.opendev.org/68053006:23
Roamer`hemna, ah, oops, my bad, thanks... so ^^^^ is an updated version of the storpool one06:24
Roamer`smcginnis, https://spfactory.storpool.com/zuul/t/local/build/b96c3192253245d691d8b39d6c8a7a9c - there's your failure :)06:24
*** markvoelker has joined #openstack-cinder06:36
*** jawad_axd has joined #openstack-cinder06:38
*** markvoelker has quit IRC06:41
openstackgerritRaghavendra Tilay proposed openstack/cinder master: 3PAR: Add Peer Persistence support  https://review.opendev.org/67794506:42
*** whfnst has quit IRC06:48
*** trident has quit IRC06:53
openstackgerritrenminmin proposed openstack/cinder master: Fix 'detach.end' notify with old volume status  https://review.opendev.org/68058807:03
*** trident has joined #openstack-cinder07:04
*** markvoelker has joined #openstack-cinder07:06
*** itlinux has quit IRC07:09
*** threestrands has quit IRC07:10
*** rcernin|brb has quit IRC07:10
*** tesseract has joined #openstack-cinder07:11
*** markvoelker has quit IRC07:11
*** markvoelker has joined #openstack-cinder07:16
*** brinzhang has joined #openstack-cinder07:17
*** markvoelker has quit IRC07:20
*** wangzhiguang has joined #openstack-cinder07:57
wangzhiguangPlease help me review the change 675801, Thank you07:58
*** pcaruana has quit IRC08:08
*** ociuhandu has joined #openstack-cinder08:13
*** pcaruana has joined #openstack-cinder08:13
*** whfnst has joined #openstack-cinder08:13
*** ociuhandu has quit IRC08:18
*** e0ne has joined #openstack-cinder08:24
*** whfnst has quit IRC08:25
*** whfnst has joined #openstack-cinder08:27
*** tkajinam has quit IRC08:32
*** raghavendrat has joined #openstack-cinder08:34
*** trident has quit IRC08:40
*** whfnst has quit IRC08:45
openstackgerritNaoki Saito proposed openstack/cinder master: NEC Driver: Support multi-attach  https://review.opendev.org/67527908:46
*** tosky has joined #openstack-cinder08:49
*** wangzhiguang has quit IRC08:50
*** virendra-sharma has joined #openstack-cinder08:51
*** trident has joined #openstack-cinder08:51
*** zigo has quit IRC08:57
*** zigo has joined #openstack-cinder09:03
*** markvoelker has joined #openstack-cinder09:10
*** markvoelker has quit IRC09:20
*** davidsha has joined #openstack-cinder09:29
*** spatel has joined #openstack-cinder09:31
*** spatel has quit IRC09:35
*** ociuhandu has joined #openstack-cinder10:15
*** jawad_axd has quit IRC10:19
*** ociuhandu has quit IRC10:19
openstackgerritMerged openstack/cinderlib master: Don't run functional gates on doc-only changes  https://review.opendev.org/68007010:21
dr_gogeta86morning all10:22
dr_gogeta86hemna, i've drill down a lil bit10:22
dr_gogeta86is like cinder-volume doesn't trigger rescan-scsi-bus.sh10:22
*** ociuhandu has joined #openstack-cinder10:29
*** ociuhandu has quit IRC10:30
openstackgerritGorka Eguileor proposed openstack/cinder master: Fix online data migrations  https://review.opendev.org/67246610:48
*** ociuhandu has joined #openstack-cinder10:55
*** carloss has joined #openstack-cinder10:58
*** spatel has joined #openstack-cinder11:01
*** spatel has quit IRC11:05
*** brinzhang has quit IRC11:19
openstackgerritPawel Kaminski proposed openstack/cinder master: target/spdknvmf: Add max_queue_depth configuration parameter  https://review.opendev.org/67206411:36
*** rcernin|brb has joined #openstack-cinder12:03
*** ociuhandu has quit IRC12:14
*** ociuhandu has joined #openstack-cinder12:15
*** jawad_axd has joined #openstack-cinder12:17
*** ociuhandu has quit IRC12:19
*** virendra-sharma has quit IRC12:20
*** dviroel has joined #openstack-cinder12:20
*** ociuhandu has joined #openstack-cinder12:21
*** spsurya has quit IRC12:26
*** dave-mccowan has joined #openstack-cinder12:29
*** rcernin|brb has quit IRC12:30
*** dave-mccowan has quit IRC12:31
*** dave-mccowan has joined #openstack-cinder12:34
*** dave-mccowan has quit IRC12:38
*** n-saito has quit IRC12:41
*** markvoelker has joined #openstack-cinder12:45
*** mriedem has joined #openstack-cinder12:46
*** raghavendrat has left #openstack-cinder12:49
*** jawad_axd has quit IRC13:15
*** jawad_axd has joined #openstack-cinder13:16
*** eharney has joined #openstack-cinder13:18
*** jawad_ax_ has joined #openstack-cinder13:19
*** jawad_a__ has joined #openstack-cinder13:21
*** jawad_axd has quit IRC13:21
*** jawad_ax_ has quit IRC13:23
*** jawad_a__ has quit IRC13:25
*** jawad_axd has joined #openstack-cinder13:38
*** jawad_axd has quit IRC13:42
*** ociuhandu has quit IRC13:47
*** udesale has quit IRC13:56
*** udesale has joined #openstack-cinder13:57
*** lseki has joined #openstack-cinder13:59
*** Luzi has quit IRC13:59
*** lbragstad has quit IRC14:04
*** ociuhandu has joined #openstack-cinder14:06
*** markvoelker has quit IRC14:07
openstackgerritJay Bryant proposed openstack/cinder master: Mark Virtuozzo Driver Unsupported  https://review.opendev.org/68070114:13
hemnajungleboyj: rosmaita, so FWIW, I just had another call with HPE about their CI14:19
hemnathey are actively working on it14:19
rosmaita\o/14:19
hemnathey are close to getting py37 working14:19
hemnabut are still running into issues14:19
hemnathey tried to upgrade the providers to ubuntu 16, but it didn't work14:19
hemnaso they are sticking with newton for now.14:19
hemnajust to get the py37 tests working14:20
rosmaitawhen you say "sticking with newton", what does that mean exactly?14:21
hemnaafter Train, they will look into upgrading to Zuul v3 etc14:21
jungleboyjhemna:  Thanks for the update.  Worried they will run into other issues with running such an old level, but ok.14:21
jungleboyjhemna:  Ah, ok ++14:21
hemnarosmaita: that's the old old way of doing CI with jenkins, etc14:21
rosmaitagotcha14:21
hemnathe baremetal are providers14:22
hemnathat run devstack14:22
hemnathat stand up VMs that run the real tests14:22
hemnathose providers are running newton devstack14:22
hemnato stand up master devstack vms14:22
rosmaitathat's not complicated at all14:22
hemnathose run the real tests14:22
hemnayah, it's pain14:22
*** ociuhandu has quit IRC14:23
rosmaitaok, so the lowdown is that the actual tests are being run on master, so that's cool14:23
hemnayes14:23
hemnathis is years old stuff14:23
hemnathey might get it working this weekend14:23
hemnaso, lets hold off on merging the patch for HPE until the deadline14:23
hemnaThey are all in India14:24
hemnaso, there is a timezone difference and why they aren't in irc much14:24
hemna:(14:24
hemnaI told them to first contact us here, but also post to the mailing list on their status.14:24
rosmaitaok, great14:24
rosmaitawell even if the patch gets merged, it can still be reverted before the stable branch is cut14:25
hemnayah, that's what I told them too14:25
rosmaitamy point isn't that we necessarily *want* to merge it, my point is that they shouldn't quit if they don't get everything done by wednesday14:26
rosmaitagreat, thatnks for working with them on this14:26
hemnayah, that's what I told them.  That even if we miss the 12th, that the patch can get reverted, so keep working on it no matter what14:28
hemnaCI is such a PITA14:28
*** raghavendrat has joined #openstack-cinder14:37
raghavendrathemna: whoami-rajat: smcginnis: eharney: jungleboyj: may i request review on https://review.opendev.org/#/c/677945/14:38
*** sapd1_x has joined #openstack-cinder14:45
*** markvoelker has joined #openstack-cinder14:51
Roamer`jungleboyj, smcginnis, hemna, actually anyone else: if you have a minute or two, I'd be really grateful if you could look at a couple of really trivial StorPool-related patches: https://review.opendev.org/679676 https://review.opendev.org/679785 (this one already has a +2 from smcginnis) https://review.opendev.org/68053014:52
*** jawad_axd has joined #openstack-cinder14:53
*** jawad_axd has quit IRC14:53
*** ociuhandu has joined #openstack-cinder14:54
*** markvoelker has quit IRC14:55
hemnaraghavendrat: the CI has to be working first14:56
raghavendratok14:57
*** ociuhandu has quit IRC14:59
raghavendratwe are working on CI front. its in progress; i thought ... may be code review can get +1 if reviewers find it good. Later after CI is up, +2 can be given15:01
raghavendratalso in parallel, i can work on review comments [as & when they are received]15:03
*** lpetrut has quit IRC15:03
*** ociuhandu has joined #openstack-cinder15:07
*** snecklifter has joined #openstack-cinder15:16
*** snecklifter has quit IRC15:22
*** snecklifter has joined #openstack-cinder15:23
*** raghavendrat has quit IRC15:31
*** trident has quit IRC15:37
*** snecklifter has quit IRC15:43
*** trident has joined #openstack-cinder15:46
*** virendra-sharma has joined #openstack-cinder15:46
virendra-sharmaHi, Devstack failing with below error... any suggestions ?15:50
virendra-sharmaFailed to discover available identity versions when contacting http://172.21.74.3/identity. Attempting to parse version from URL.Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Internal Server Error (HTTP 500)15:50
hemna looks like the wrong port ?15:51
hemnafor keystone15:51
virendra-sharmado i need to give any port explicitly ?... or may be in local.conf ?15:52
*** tesseract has quit IRC15:52
hemnahrmm, my cinder.conf doesn't have a port either15:53
hemnaso maybe keystone is down?15:53
Roamer`if it says "internal server error", my first thought would be to check the apache and keystone logs15:53
Roamer`yeah, it ought to be the apache logs, since keystone runs there15:54
hemnayah, looks like it errored out15:54
openstackgerritPeter Penchev proposed openstack/cinder master: Advertise some of the StorPool driver's capabilities.  https://review.opendev.org/67967615:55
openstackgerritPeter Penchev proposed openstack/cinder master: Mark the StorPool driver as supported again.  https://review.opendev.org/67978515:55
openstackgerritPeter Penchev proposed openstack/cinder master: StorPool: update the driver requirements.  https://review.opendev.org/68053015:55
virendra-sharmaapache2 is running and there is no keystone log in /var/log/16:00
virendra-sharma[mpm_event:notice] [pid 3279:tid 140365982181248] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/3.5.2 configured -- resuming normal operations[core:notice] [pid 3279:tid 140365982181248] AH00094: Command line: '/usr/sbin/apache2'[mpm_event:notice] [pid 3279:tid 140365982181248] AH00491: caught SIGTERM, shutting down[wsgi:warn] [pid 4711:tid16:01
virendra-sharma 140304871200640] mod_wsgi: Compiled for Python/3.5.1+.[wsgi:warn] [pid 4711:tid 140304871200640] mod_wsgi: Runtime using Python/3.5.2.[mpm_event:notice] [pid 4711:tid 140304871200640] AH00489: Apache/2.4.18 (Ubuntu) mod_wsgi/4.3.0 Python/3.5.2 configured -- resuming normal operations[core:notice] [pid 4711:tid 140304871200640] AH00094: Command lin16:01
virendra-sharmae: '/usr/sbin/apache2'16:01
virendra-sharmahere is apache2 error.log16:01
hemnajournalctl -u devstack@keystone16:02
hemna?16:02
hemnavirendra-sharma: please use pastebin for large pastes16:02
virendra-sharmaStarted Devstack devstack@keystone.service.Traceback (most recent call last):Traceback (most recent call last):  File "/usr/local/bin/keystone-wsgi-public", line 6, in <module>  File "/usr/local/bin/keystone-wsgi-public", line 6, in <module>    from keystone.server.wsgi import initialize_public_applicationModuleNotFoundError: No module named 'keyst16:04
virendra-sharmaone'unable to load app 0 (mountpoint='') (callable not found or import error)*** no app loaded. going in full dynamic mode ***16:04
hemnamaybe ask in the keystone channel?16:05
hemnaI've never seen that16:06
*** sapd1_x has quit IRC16:08
virendra-sharmado we need to make entry for pip too in local.conf to point it to pip3.7 ?16:09
*** sahid has quit IRC16:09
virendra-sharmaMy CI running on python3.7 where as pip installing some of packages in pip2.716:09
*** mriedem is now known as mriedem_lunch16:11
hemnaUSE_PYTHON3=True16:11
hemnain local.conf16:11
Roamer`virendra-sharma, if you want to run with Pyhton 3 (and, yeah, I guess you should want to), you need to set USE_PYTHON3=True and PYTHON3_VERSION=3.7 in the devstack localrc file16:12
Roamer`right, local.conf would work too, sorry, I'm already too much used to the zuulv3 settings :)16:13
hemnaheh16:13
Roamer`hemna, btw I updated https://review.opendev.org/#/c/679676 with unit tests, oof, thanks!16:14
virendra-sharmaI have done those setting in local.conf, is that enough for pip to point 3.7 ?16:14
hemnaRoamer`: sweet16:14
hemnathanks for that16:14
hemnaI guess you probably should add a release note for it as well, since this enables multiattach16:15
Roamer`for a moment I wondered if the "No module named keystone" problem could be related to the python3-then-2-then-3-again issue, but, well, it would only matter if keystone.server.wsgi and bin/keystone-wsgi-public were installed by different Python modules, which seems unlikely (haven't checked though)16:17
Roamer`virendra-sharma, do you preserve the host for a while after a failed run? can you check what is the first line of the /usr/local/bin/keystone-wsgi-public file - is it #!/path/to/python3.7 or is it #!/path/to/python2.something?16:19
virendra-sharmayes, I noticed some of packages used pip2.716:19
virendra-sharmaI do have node with me on which devstack failed16:19
virendra-sharmait is "#!/usr/bin/python3.7"16:22
Roamer`ok, so it's not that, so I honestly have no idea what could lead to this file being present and keystone.server.wsgi not being found16:23
Roamer`I mean, this file ought to be installed by a "pip3.7 install keystone", which certainly ought to install the module, too16:24
Roamer`maybe ask in #openstack-keystone or #openstack-qa or something16:24
virendra-sharmaroot@d-p-c-p74-4423578:/opt/stack/new/devstack# pip3.7 list | grep keystonekeystone (15.1.0.dev282, /opt/stack/new/keystone)keystoneauth1 (3.17.0)keystonemiddleware (7.0.0)python-keystoneclient (3.20.0)16:25
openstackgerritMerged openstack/cinder master: Support Incremental Backup Completion In RBD  https://review.opendev.org/62794116:25
virendra-sharma#openstack-keystone yet to get response16:26
Roamer`(and BTW, yeah, the "haven't checked though" from above does not apply, I did check and the keystone-wsgi-public script is indeed installed by the keystone library)16:27
Roamer`well, maybe it was installed in a virtual environment and Apache does not know how to invoke it... sorry, I don't really know quite enough about the way keystone's wsgi interface works :(16:31
*** davidsha has quit IRC16:32
*** pcaruana has quit IRC16:37
*** e0ne has quit IRC16:39
smcginnishemna: Did you find a bugfix to backport for your ceph issue?16:45
*** vesper11 has quit IRC16:48
*** vesper11 has joined #openstack-cinder16:48
*** pcaruana has joined #openstack-cinder16:51
*** tosky has quit IRC16:58
*** ociuhandu has quit IRC17:03
hemnawell it looks like the excessive ImageNotFound exceptions was removed already17:06
hemnaI've been looking at Pike17:06
hemnaso I'll find that patch and backport it to pike17:06
hemnathe scaled create volume problem is still around17:06
hemnafor all drivers I Think17:07
hemnaI was able to reproduce that with lvm17:07
jungleboyjrosmaita:  Thanks for sending out the Forum Planning note.  I have todos from the mid-cycle to get some of those up.17:21
*** markvoelker has joined #openstack-cinder17:25
*** udesale has quit IRC17:27
*** qqmber has joined #openstack-cinder17:31
qqmberhi.. quick question.. if I want to "fsck" a volume.. but not from the instance, but from the host where Cinder runs... (the volume is unmounted), should be like this?: fsck /dev/cinder-volumes/volume-63a86d1d-a32d-43f3-a3f5-6c5d6c1ecc517:32
*** mriedem_lunch is now known as mriedem17:38
jbernardqqmber: please dont do that while the volume is attached to a guest17:38
qqmberit is not17:38
qqmberthe problem is fsck "think" this is a ext2 device..17:39
qqmberand if I run this:17:39
qqmber# blkid | grep 6c5d6c1ecc5f17:39
qqmber/dev/mapper/cinder--vol--iqq--d-volume--63a86d1d--a32d--43f3--a3f5--6c5d6c1ecc5f: PTTYPE="gpt"17:39
qqmberjbernard: do you know what command to run in order to check the LV?17:49
qqmberthe volume has ex4... but the LV... GPT? There I'm getting lost...17:50
jbernardqqmber: there are several questions, much to say, and none of it is on-topic for this channel, you're welcome to email me if you lik17:53
qqmberoh, sorry...17:53
*** enriquetaso has joined #openstack-cinder17:53
jbernardqqmber: it's okay, but your questions are basic block device questions, partition table formats, etc17:54
qqmberi know.17:54
qqmber:(17:54
jbernardqqmber: im happy to help, but it's not within the domain of cinder17:54
*** jawad_axd has joined #openstack-cinder17:55
*** mvkr has quit IRC17:55
*** markvoelker has quit IRC17:57
qqmberjbernard: let me tell you something about Cinder then... I have an app running (ElasticSearch) in three instances... I was loading a lot of data unto it... and suddenly one of the volumes (where the ElasticSearch data was writing on) remunted as Read-only (Centos 7). I stop everything, umount the volume, fsck it, it repaired some journal stuff, re munted, start all over and as soon as I start ElasticSearch and the other nodes start se17:57
qqmbernding data to this one... it remounts as read-only again... 10 seconds has elapsed.. tops.17:57
jbernardqqmber: ok, the guest will surely log the reason for remount, what was it?17:58
*** e0ne has joined #openstack-cinder17:58
qqmberI know I have to do some tunning on stuff, like 10G interfaces (and network in general), MTU Jumbo frames, etc.. but now I just want to make it work17:59
qqmberjbernard: at the inscance?18:00
jbernardqqmber: the best thing you can probably do now is gather all of the info you have collected and create a bug at https://bugs.launchpad.net/cinder/+filebug18:00
qqmberjbernard: ok, I'll do it. Thanks.18:01
jbernardqqmber: there's a reasonable chance that it's not a bug in cinder, but that's probably the best place to work through it18:02
smcginnisI would recommend checking what the error was that caused it to make it read only in the first place. That might (should?) go a long way to pointing to what the actual issue is.18:03
*** markvoelker has joined #openstack-cinder18:03
*** virendra-sharma has quit IRC18:04
qqmbersmcginnis: everythime it happens is when I put millions of log lines into a DB, or thousands of files into the filesystem in a short period of time... I change volumes (different physical disks), and so on... same result..18:05
qqmber*(every time)18:06
qqmberso, basically I/O18:06
smcginnisYeah, so you said. But you haven't mentioned what the system logs as the reason for doing so.18:06
qqmbersmcginnis: here it is what I see on the syslog:18:08
qqmberSep  5 19:19:18 elastic kernel: blk_update_request: I/O error, dev vdb, sector 85592240018:08
qqmberSep  5 19:19:18 elastic kernel: Aborting journal on device vdb1-8.18:08
qqmberSep  5 19:19:18 elastic kernel: EXT4-fs error (device vdb1): ext4_journal_check_start:56: Detected aborted journal18:08
qqmberSep  5 19:19:18 elastic kernel: EXT4-fs (vdb1): Remounting filesystem read-only18:08
*** markvoelker has quit IRC18:12
*** e0ne has quit IRC18:15
*** markvoelker has joined #openstack-cinder18:15
*** markvoelker has quit IRC18:26
*** markvoelker has joined #openstack-cinder18:28
*** enriquetaso has quit IRC18:34
*** markvoelker has quit IRC18:38
*** jawad_axd has quit IRC18:42
*** enriquetaso has joined #openstack-cinder18:44
*** abishop_ has joined #openstack-cinder19:10
*** markvoelker has joined #openstack-cinder19:11
*** abishop has quit IRC19:12
*** markvoelker has quit IRC19:15
openstackgerritEric Harney proposed openstack/cinder master: Rekey volume on clone  https://review.opendev.org/66497619:30
*** markvoelker has joined #openstack-cinder19:41
*** markvoelker has quit IRC19:45
openstackgerritWalter A. Boring IV (hemna) proposed openstack/cinder stable/pike: RBD: remove redundant exception log to reduce noise  https://review.opendev.org/68076419:51
*** markvoelker has joined #openstack-cinder20:19
*** markvoelker has quit IRC20:29
*** senrique_ has joined #openstack-cinder20:35
*** senrique_ has quit IRC20:35
*** enriquetaso has quit IRC20:35
*** markvoelker has joined #openstack-cinder20:50
*** markvoelker has quit IRC20:54
*** ociuhandu has joined #openstack-cinder21:04
*** ociuhandu has quit IRC21:15
*** rcernin|brb has joined #openstack-cinder21:18
*** rcernin|brb has quit IRC21:33
*** KeithMnemonic1 has quit IRC21:40
*** mriedem has quit IRC21:41
*** e0ne has joined #openstack-cinder22:02
*** markvoelker has joined #openstack-cinder22:15
*** carloss has quit IRC22:19
*** markvoelker has quit IRC22:20
*** e0ne has quit IRC22:34
*** ociuhandu has joined #openstack-cinder23:15
*** ociuhandu has quit IRC23:20
*** qqmber has quit IRC23:41
« Thursday, 2019-09-05 Index Saturday, 2019-09-07 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!