Monday, 2015-02-23

*** markvoelker has joined #openstack-operators00:16
*** markvoelker has quit IRC00:20
*** markvoelker has joined #openstack-operators00:50
*** markvoelker has quit IRC00:55
*** VW_ has quit IRC01:27
*** markvoelker has joined #openstack-operators01:51
*** markvoelker has quit IRC01:57
*** markvoelker has joined #openstack-operators02:00
*** signed8bit has joined #openstack-operators02:13
*** markvoelker has quit IRC02:26
*** markvoelker has joined #openstack-operators02:27
*** markvoelker has quit IRC02:31
*** blair has joined #openstack-operators02:55
*** Marga_ has joined #openstack-operators03:37
*** signed8bit has quit IRC04:20
*** Marga_ has quit IRC04:25
*** Marga_ has joined #openstack-operators04:25
*** zerda has joined #openstack-operators04:39
*** Marga_ has quit IRC05:16
*** pcaruana has quit IRC05:45
*** blair has quit IRC05:52
*** blair has joined #openstack-operators06:22
*** zerda has quit IRC06:33
*** blair has quit IRC06:35
*** turnerg has quit IRC07:09
*** belmoreira has joined #openstack-operators07:10
*** turnerg has joined #openstack-operators07:44
*** Miouge has joined #openstack-operators07:51
*** matrohon has joined #openstack-operators08:09
*** Miouge_ has joined #openstack-operators08:37
*** Miouge has quit IRC08:38
*** Miouge_ is now known as Miouge08:38
*** sanjayu has joined #openstack-operators08:41
*** blair has joined #openstack-operators08:41
*** VW_ has joined #openstack-operators09:02
*** subscope has joined #openstack-operators09:15
*** blair has quit IRC09:25
*** blair has joined #openstack-operators09:25
*** bvandenh has joined #openstack-operators09:55
*** bvandenh has quit IRC10:28
*** bvandenh has joined #openstack-operators10:33
*** Marga_ has joined #openstack-operators10:56
*** jaypipes has joined #openstack-operators11:05
*** Marga_ has quit IRC11:06
*** Miouge_ has joined #openstack-operators12:06
*** Miouge has quit IRC12:06
*** Miouge_ is now known as Miouge12:06
*** Miouge has quit IRC12:09
*** VW_ has quit IRC12:10
*** Miouge has joined #openstack-operators12:10
*** dmsimard_away is now known as dmsimard12:10
*** Miouge has quit IRC12:20
*** Miouge has joined #openstack-operators12:21
*** Miouge has quit IRC12:27
*** VW_ has joined #openstack-operators12:31
*** Miouge has joined #openstack-operators12:44
*** derekh has joined #openstack-operators12:45
*** zigo has quit IRC12:52
*** zigo has joined #openstack-operators12:56
*** pboros has joined #openstack-operators13:02
*** zerda has joined #openstack-operators13:19
*** turnerg has quit IRC13:23
*** cpschult has joined #openstack-operators13:37
*** j05h1 has quit IRC13:57
*** Guest28285 is now known as mgagne14:13
*** mgagne has joined #openstack-operators14:13
*** j05h has joined #openstack-operators14:18
*** mtanino has quit IRC14:22
*** sanjayu has quit IRC14:26
*** turnerg has joined #openstack-operators14:29
*** cpschult has quit IRC14:55
*** signed8bit has joined #openstack-operators14:56
*** pboros has quit IRC15:00
*** lascii is now known as alaski15:09
mgagnemfisch: working at TWC? Do you know who is giving the talk? https://www.openstack.org/vote-vancouver/Presentation/vdev-a-virtual-development-environment-in-an-openstack-cloud15:13
mgagnemfisch: before I swear I thought it was someone from my workplace ;)15:14
*** Miouge has quit IRC15:15
*** Miouge has joined #openstack-operators15:20
*** Miouge has quit IRC15:23
*** Miouge has joined #openstack-operators15:26
*** Piet has quit IRC15:27
*** zigo_ has joined #openstack-operators15:37
*** radez_g0n3 is now known as radez15:38
*** Miouge has quit IRC15:39
mfischmgagne: no, it's Adam Vinsh, for some reason the speaker info vanished15:40
mfischhe's ex-HP15:40
*** radez is now known as radez_g0n315:43
*** Miouge has joined #openstack-operators15:45
*** jawed has joined #openstack-operators15:45
*** mdorman has joined #openstack-operators15:47
*** Miouge has quit IRC15:50
*** Piet has joined #openstack-operators16:00
*** Miouge has joined #openstack-operators16:10
*** Miouge has quit IRC16:14
*** belmoreira has quit IRC16:17
*** devlaps has joined #openstack-operators16:21
*** zerda has quit IRC16:25
*** devlaps has quit IRC16:42
*** Miouge has joined #openstack-operators16:42
*** Miouge has quit IRC16:48
*** devlaps has joined #openstack-operators16:48
*** alop has joined #openstack-operators16:50
*** reed has joined #openstack-operators16:57
*** matrohon has quit IRC17:02
*** Miouge has joined #openstack-operators17:09
*** jawed has quit IRC17:12
*** Miouge has quit IRC17:14
*** Miouge has joined #openstack-operators17:15
*** csoukup has joined #openstack-operators17:16
*** Miouge has quit IRC17:16
*** jlk has quit IRC17:25
*** jlk has joined #openstack-operators17:25
*** bvandenh has quit IRC17:29
*** radez_g0n3 is now known as radez17:30
*** Marga_ has joined #openstack-operators17:34
*** Marga_ has quit IRC17:34
*** Marga_ has joined #openstack-operators17:35
*** derekh has quit IRC17:36
mfischAnyone have any context on the BoD choosing Austin in a secret ballot discussion I see on twitter?17:41
mfischI'm not even sure what the vote was about17:43
*** cpschult has joined #openstack-operators17:44
j05hmfisch: just about where the next board meeting is.17:46
mfischthanks17:47
mfischI guess thats a long trip for Tristan17:47
j05hi *think* the bigger deal is not doing it in the open. doesn't bug me one way or another, but i understand how folks want as much as possible as open as possible.17:48
j05hthe portrayal of some secret kabal nefariously trying to meet in Austin seems a bit overblown.17:48
mfischyes but its not a great precedent17:49
j05hno argument there.17:50
mfischthe odd thing is that it was secret, its not like they were voting to make someone king of openstack for life or something else equally strange and ridicilous17:50
j05htrue…but its also an easy thing to slip up on, because it doesn't really matter.17:51
*** Marga_ has quit IRC17:53
*** Piet has quit IRC17:53
jlknobody probably would have noticed if it wasn't called out as a secret meeting in the announcement email.17:54
*** Marga_ has joined #openstack-operators17:54
mdormani haven’t read all the ML posts to understand the full contexnt.  but my unfiltered opinion is that somebody just got their feelings hurt about their preferred location not being picked, and is not on a crusade about it ( there, i said it)17:55
mdormanpersonally i dont’ think openness solely for the sake of openness is always an absolute necessity17:55
mdormans/not/now/17:56
mdormani donno.  bit of a soapbox issue for me.  i just think there are more important pragmatic things to focus on17:56
mgagnemdorman: =)18:00
j05hmdorman: agreed, i mean, if there are concerns about lots of backchannel on the board, then fine… but reality is it will always exist.18:01
mgagneI don't know if the procedure or way of doing has always been the same since the beginning or not. If so, it's "fun" to see how NOW it's a problem.18:02
mdormanyeah.  in any case.  back to real work of manually fixing up nova quotas!18:03
mdorman*eyeroll*18:03
mgagnemdorman: we have to talk about theoretical problems with galera first18:03
jlkwhy bother with theoretical, there are plenty of actual ones :)18:04
mdormanjlk:  +118:07
j05hjlk: :D18:07
*** david-lyle_afk is now known as david-lyle18:09
jlkAny of you know if it's possible to convince python-novaclient (the nova cli itself) or openstack client to paginate instance listing? We're getting 500s trying to list over 200 instances.18:19
jlk(listing with --minimal succeeds)18:20
klindgrenhrm18:20
klindgrenwe have over 1000 instances and ours is fine18:21
jlkI suspect it's nova talking to neutron to get more data that's timing out and sending a 500 up18:21
klindgrenthough we do get random 500's18:21
klindgrenwas goign to say18:21
klindgrenwe bumped neutron workers18:21
jlkyeah, maybe we have a timeout to increase.18:21
klindgrenhonestly fighting that right now18:21
klindgrendoesn't seem to help18:21
klindgrenwe went from 30 second to 12018:21
klindgrenstill get the same errors18:22
jlkours is a tiny cluster, 3 nodes. We've got neutron-API on two of those nodes, and I think we have 3 API workers, Metadata workers is hilariously at CPU count right now, 30~18:23
jlkyou think maybe we need more neutron-api workers?18:23
mgagnebut then someone will try to convince you that you want filtering, not pagination18:25
mgagnewhich makes no sense if you really want to fetch the whole list for other purposes than UI.18:26
jlkyeah...18:26
jlkwhen filtering still returns over 200....18:27
*** VW_ has quit IRC18:31
klindgrenjlk we only use to see it when doing --all-tenants which would give us back everything18:33
klindgrenbut that seemed to have improved18:33
*** harlowja has joined #openstack-operators18:33
mdormanjlk:  there is also the general purpose “openstack client” which is supposed to be “better” than the individual service cli tools.  however i don’t have much experience w/ it so i can’t speak to how good it is.  but i have heard it has more advanced output/filtering options.18:33
jlkI've been using it a bunch lately18:34
mgagneI don't think a client can improve the server side performance :P18:34
jlkbecause I'm mapping supernova over to it, so that it it can be used with a bunch of clouds18:34
jlkmgagne: it does if it can paginate18:34
jlkbut I don't recall that being ported over18:34
jlkalso, hilariously, looks like openstackclient is coded for cinder v1 only, which makes it rather difficult to test my move to v2, since v1 is deprecated.18:35
mgagnehilarious18:35
mdormanheh18:35
mdormansomehow, i am not surprised18:36
mdormanwhat’s supernova?18:36
mgagnelets try python-openstacksdk then18:36
jlkmdorman: supernova was written a long time ago to wrap around novaclient and set the credentials and endpoints stuff per-execution18:41
jlkso you could use it against multiple clouds or regions18:41
jlkhttps://github.com/major/supernova18:42
mdormanah i see18:42
jlkI'm working on a PR for it that changes it to "superstack" and wraps openstack client instead18:42
jlkso you could do superstack raxdfw server list18:42
jlkor superstack bbg-iad image list18:42
*** Marga_ has quit IRC18:44
mgagnemeta talk: https://www.openstack.org/vote-vancouver/presentation/the-openstack-summit-talk-selection-process-is-broken18:51
*** priteau has joined #openstack-operators18:53
jlkhahah18:54
jlkwonder if that person has ever been a track chair18:54
jlkti's certainly a hard job18:54
mdormanyeah i don’t envy those guys18:54
jlkI did it for operators track in Paris, and then did it again for my own conference here in Seattle. I had to read through so many shit pitches.18:56
mdormani bet18:56
jlkI even had some duplicates across the two shows, which was interesting18:57
mgagnealright, finally done voting on all talks18:57
jlkas a hint, at least in the operators track, we didn't look at the votes at all really18:57
jlkbecause astroturfing18:57
mdormanhah18:57
jlkit's pretty apparent that the big companies use employee force to vote up talks. There were some pretty crap talks with LOTS of votes.18:57
mgagnewhich is unfortunate18:58
mdormanfor sure18:58
mdormanyeah i wish there was a btter way to filter out the sales pitch ones18:58
dmsimardjlk: That's actually comforting (that the talks are not decided solely on votes)19:07
*** Marga_ has joined #openstack-operators19:08
klindgrenI typically try to look at who the speaker is - i they are head of product engineering or something that sounds like a salesy title - I typically dont go/vote for those19:14
*** Rockyg has joined #openstack-operators19:17
*** mdorman has quit IRC19:23
jlkSo here's a fun one. Recently when doing nova volume attach, it's doing the work, but when it tries to get volume info, I get back a 404 error, even though it successfully mounts the volume. What the hell client?19:23
dvorakare the vote totals actually published?19:24
jlkDEBUG (connectionpool:364) "GET /v2/7efae1d1945f49ed9fea4af0604d2236/volumes/9d716b7a-9843-42e8-bec6-0aaecd63b09e HTTP/1.1" 404 5219:24
jlkI can't for the life of me figure out why19:24
*** VW_ has joined #openstack-operators19:25
jlkoh interesting19:25
jlkINTERESTING19:25
* mgagne is listening19:27
jlkwhen nova is doing it, it's trying to connect to 8777, when cinder does it, it's connecting to 8778.19:28
mgagnejlk: using cells? =)19:28
jlkmy catalog for cinder shows 877819:28
mgagnewut19:28
jlkcompute has 877719:28
jlkso nova is trying to hit v2 volumes on it's own port.19:28
*** Piet has joined #openstack-operators19:29
jlkDEBUG (session:190) REQ: curl -g -i --cacert "/opt/stack/ssl/openstack.crt" -X GET https://openstack.example.com:8777/v2/7efae1d1945f49ed9fea4af0604d2236/volumes/9d716b7a-9843-42e8-bec6-0aaecd63b09e19:29
jlkI don't even.19:30
mgagnethe "volume info" using which client? novaclient?19:30
jlkthe get on 8777 is coming from python-novaclient, doing a volume-attach19:32
mgagnecould it be novaclient thinking it can proxies the request through nova to cinder?19:32
jlkmaybe, wonder if that's a paste api thing19:33
jlknope19:33
jlkhrm.19:33
jlkdoing a show, I see hits to 8777/<tenant>/images19:34
mgagnenova proxying again19:34
jlkoh, compute supports images/flavors19:34
jlkbut not volume19:34
mgagnewhat's in your catalog cinderv2 and volumev2 or is v1 still available too?19:35
jlkI have both cinder and volume just set to the v2 API point19:35
jlkbecause nova-api wasn't picking up on volumev219:35
jlkand kept trying to use volume19:36
mgagnecould it be novaclient not knowing about cinder v2 and fallbacking to nova proxying?19:36
mgagnewhat's the service type and service name for cinder?19:39
mgagnetrying to reproduce your issue19:39
jlkgeez, nova volume-list and nova volume-show both work19:39
jlkoh huh, I DO have both volume and volumev219:40
jlkcinder and cinderv2 respectively too19:40
jlkeven odder.19:45
jlknova --debug volume-show <id> makes a request to 8778, not 877719:45
mgagnelist and show looks to go to the right port on my side. trying to reproduce your attach reqeust19:46
jlkalright19:46
mgagnewhich logs did you find your entry in?19:49
mgagnelog file19:49
mgagnegetting 500 here19:49
jlkit's client side19:49
jlkthe GET call on 8777 is all from client --debug19:50
jlkthe client is trying to do the get19:50
mgagnemine (volume-attach) returns right after POST to http://compute:8774/v2/<tenant>/servers/<id>/os-volume_attachments with volume/attachment info19:54
mgagnebtw, isn't 8777 the telemetry service port?19:55
jlkinteresting. Wonder if it's client code version19:55
jlkno, in our env we frontend nova on 8777 with https to haproxy, haproxy backs it to 877419:56
mgagneI think your got your endpoint wrong and changed the port for other reason19:56
jlkcatalog lists 8777 for nova19:56
mgagneyou trying to confuse me! :O19:56
*** VW_ has quit IRC19:56
mgagnecould be ceilometer responding with 404 :D19:56
jlkwe aren't running ceilometer19:56
*** VW_ has joined #openstack-operators19:57
mgagneyou are blessed19:57
mgagnenovaclient from master?19:58
jlkI'm going to try that next19:59
mgagnewhich version then?19:59
jlkdoing a fresh pip install in a venv19:59
jlkum...19:59
*** cpschult has quit IRC20:00
jlkactually I think we're just taking whatever is in pip20:00
mgagnehow convenient... https://github.com/openstack/python-novaclient/commits/master/novaclient/v2/volumes.py20:00
mgagneOH! I got the 404!20:05
mgagnebefore: python-novaclient==2.20.0  // after: python-novaclient==2.21.020:05
mgagnesomething went wrong between those versions20:05
jlkah hah! nice grab20:06
mgagnebut we lost the goddamn history20:06
jlk....20:06
mgagneI bet someone copied the files instead of moving them20:06
*** markvoelker has joined #openstack-operators20:07
jlkgit log --follow doesn't help?20:08
*** Rockyg has quit IRC20:09
*** c_soukup has joined #openstack-operators20:10
*** csoukup has quit IRC20:11
jlkmgagne: goddamn, you're right, 2.20.0 totes works.20:12
jlkdamn, follow doesn't help20:13
*** Marga__ has joined #openstack-operators20:14
*** Marga_ has quit IRC20:14
*** c_soukup has quit IRC20:14
mgagneI think I found the commit20:16
mgagnelemme check again20:16
jlkdon't share, I want to see if I get the same one20:16
mgagnewas 2ms from sending20:16
jlkheh, 75727c2d37eb616fc2d0573200db44552e106e91 ?20:17
mgagnenope20:17
jlkdrats20:17
mgagnewell, I could be wrong too. I took the git log and bisect20:17
jlkyeah I was guessing just on timestamp on changes20:17
mgagneyou are too far down the history20:18
jlkI think the only other one it could be (if it's the 1_1 changes, is ae6c39397e9ae60a4e987d0f1af3086b7001957120:19
jlkoh you think it's after the v2 move?20:19
mgagneit's not the move itself20:19
mgagnebut yes, after that20:19
mgagnewhat I did: attach -> detach -> pip install git -> rinse and repeat20:19
*** matrohon has joined #openstack-operators20:21
mgagnewhat's hilarious is that the change looks to be totally unrelated to volume management20:21
jlkof course, it always is20:22
*** csoukup has joined #openstack-operators20:22
jlkwell, don't let me waste more time, what change did you find?20:22
mgagnehttps://github.com/openstack/python-novaclient/commit/578390ee7e74c0998b9590f0327449ae19a5c8f020:23
jlkI... what?20:24
mgagnehttp://paste.openstack.org/show/180757/20:25
mgagnethe cache is properly used somewhere else for other purposes than completion20:25
jlkwhat about with caching disabled?20:25
mgagnehaha, properly -> probably20:25
jlkwe have export OS_NO_CACHE=True20:26
mgagneand no, it's not properly used :D20:26
mgagnejust try with the commit and the previous one20:26
jlkcould it be that at that change your cached data is no longer good20:26
mgagneoh...20:27
mgagneyou could be very right20:27
mgagnehow about I clear my cache and try again20:27
jlkhowever20:27
jlkI do get a 404 with that commit20:27
jlkeven after clearing cache20:28
mgagnesame here20:28
mgagneI tried deleting cinderclient cache too20:28
jlkand the previous one works20:29
jlkso it's definitely the cache, even though I have cache disabled20:29
jlkmaybe nothing reads that anymore20:29
mgagnedoes .novaclient get recreated?20:29
jlkyeah20:29
jlkit's still caching20:29
*** matrohon has quit IRC20:29
mgagnewhy if cache is disabled? =)20:29
jlkmaybe because this isn't auth token cache20:30
jlkit's a different cache20:30
mgagnecan't grep OS_NO_CACHE in novaclient20:30
jlkit's caching human-id, server-id, volume-id20:30
mgagnehow about we question the fact that no test caught that one =)20:30
jlkI saw something on this recently20:31
jlkit's certainly not caching the token any where20:31
*** Piet has quit IRC20:31
mgagnehahaha https://github.com/openstack/python-novaclient/commit/ac6636a54d72ba76d0adca76e07d1d26d9ea35c320:33
mgagnelatest commit in master20:33
mgagneshould have tested master too20:33
*** Marga__ has quit IRC20:34
*** harlowja has quit IRC20:34
jlkthose appear to be written out for bash completion20:34
*** harlowja_ has joined #openstack-operators20:34
jlkoh geez20:34
mgagneand it's working fine with this commit20:34
mgagneThis reverts commit https://github.com/openstack/python-novaclient/commit/4c8cefb98a425382204df2f38f24e6b5b71520dd.20:34
jlkyeah20:34
mgagnewe are too late to the show20:34
jlksigh20:34
jlkfigured somebody would find this.20:34
*** Marga_ has joined #openstack-operators20:34
jlk4 days ago tho20:35
*** Marga_ has quit IRC20:35
jlkmake with the pip update already20:35
*** Marga_ has joined #openstack-operators20:35
*** Marga_ has quit IRC20:35
*** Marga_ has joined #openstack-operators20:36
jlkseriously, api grabs for bash completion.20:36
jlkfuck that20:36
*** Marga_ has quit IRC20:36
mgagnethanks alaski for that one20:36
*** Marga_ has joined #openstack-operators20:37
*** Marga_ has quit IRC20:38
*** Marga_ has joined #openstack-operators20:38
mgagneyep... what about non-interactive usage. doesn't need autocompletion20:38
jlkI can't see a way to turn it off20:39
jlkeven if we move it to /dev/null20:39
*** matrohon has joined #openstack-operators20:43
alaskimgagne: :)20:43
jlkalaski: is there a way to flat out turn off tab completion caching?20:44
alaskijlk: I think a new client will get pushed out soon, the revert just merged earlier today20:44
alaskijlk: not without modifying code (that I'm aware of)20:44
*** priteau has quit IRC20:45
alaskijlk: best bet for now is to go back to 2.2020:45
jlknod20:45
*** Marga_ has quit IRC20:46
*** csoukup has quit IRC20:49
*** blair has quit IRC20:51
*** markvoelker has quit IRC20:51
*** VW__ has joined #openstack-operators20:56
*** VW_ has quit IRC20:59
*** sandywalsh_ has joined #openstack-operators21:02
*** sandywalsh has quit IRC21:05
*** hj-hp has joined #openstack-operators21:09
*** markvoelker has joined #openstack-operators21:15
*** markvoelker has quit IRC21:15
*** markvoelker has joined #openstack-operators21:16
*** Marga_ has joined #openstack-operators21:17
*** cpschult has joined #openstack-operators21:18
*** radez is now known as radez_g0n321:23
*** matrohon has quit IRC21:27
*** cpschult has quit IRC21:28
*** Marga_ has quit IRC21:32
*** Marga_ has joined #openstack-operators21:32
*** matrohon has joined #openstack-operators21:35
*** VW__ has quit IRC21:41
*** VW_ has joined #openstack-operators21:41
*** j05h has quit IRC21:47
*** turnerg has quit IRC21:48
*** VW_ has quit IRC21:58
*** VW_ has joined #openstack-operators21:58
*** csoukup has joined #openstack-operators22:10
*** radez_g0n3 is now known as radez22:16
*** j05h has joined #openstack-operators22:20
*** VW_ has quit IRC22:24
*** VW_ has joined #openstack-operators22:25
*** j05h has quit IRC22:26
*** j05h has joined #openstack-operators22:26
*** hj-hp has quit IRC22:27
*** harlowja_ has quit IRC22:29
*** harlowja has joined #openstack-operators22:30
*** matrohon has quit IRC22:36
*** hj-hp has joined #openstack-operators22:44
*** j05h has quit IRC22:47
*** dmsimard is now known as dmsimard_away22:47
*** j05h has joined #openstack-operators22:48
*** Marga_ has quit IRC22:50
*** VW_ has quit IRC22:51
*** radez is now known as radez_g0n322:55
*** Marga_ has joined #openstack-operators22:55
*** Marga_ has quit IRC22:56
*** Marga_ has joined #openstack-operators22:56
*** VW_ has joined #openstack-operators23:03
*** harlowja has quit IRC23:04
*** blair has joined #openstack-operators23:08
*** devlaps has quit IRC23:14
*** devlaps has joined #openstack-operators23:14
*** harlowja has joined #openstack-operators23:16
mfischanyone here switched to the cinderv2 api?23:20
*** mfisch has quit IRC23:36
*** csoukup has quit IRC23:46
*** VW_ has quit IRC23:47
*** VW_ has joined #openstack-operators23:48
*** j05h has quit IRC23:51
*** mfisch has joined #openstack-operators23:54
*** mfisch has quit IRC23:54
*** mfisch has joined #openstack-operators23:54

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!