Tuesday, 2015-02-24

*** hj-hp has quit IRC00:06
*** Piet has joined #openstack-operators00:07
*** VW_ has quit IRC00:28
*** VW_ has joined #openstack-operators00:28
*** VW_ has quit IRC00:39
*** Marga_ has quit IRC00:40
*** Marga_ has joined #openstack-operators00:41
*** Marga_ has quit IRC00:41
*** VW_ has joined #openstack-operators00:44
*** devlaps has quit IRC01:11
*** david-lyle is now known as david-lyle_afk01:19
*** radez_g0n3 is now known as radez01:22
*** VW_ has quit IRC01:33
*** signed8bit is now known as signed8bit_ZZZzz01:50
*** signed8bit_ZZZzz is now known as signed8bit02:04
*** signed8bit has quit IRC02:16
*** markvoelker has quit IRC02:25
*** markvoelker has joined #openstack-operators02:26
*** markvoelker has quit IRC02:30
*** markvoelker has joined #openstack-operators03:08
*** Marga_ has joined #openstack-operators03:30
*** Marga_ has quit IRC03:43
*** Marga_ has joined #openstack-operators03:44
*** blairo has joined #openstack-operators03:48
*** blair has quit IRC03:51
jlkmfisch: I am doing that today03:56
jlkmfisch: or rather have been working on that most of today, and got lost in that novaclient bug03:56
jlkother than that, seems okay if you just change your endpoint record from v1 to v2 in the catalog03:57
mfischjlk: let me forward you this email I'm prepping for the ml03:57
mfischthats what Im considering03:57
jlkand set the client ENV var03:57
jlkhttps://github.com/blueboxgroup/ursula/pull/81203:57
mfischsure you can do that but thats bs03:57
mfischthe client should just fugure it out03:57
jlkI consider most the client stuff to be BS03:58
mfischI have v1 and v2 in the cat03:58
jlknovaclient is deficient, and deprecated. openstackclient doesn't seem to even support cinder v203:58
mfischmy biggest worry is that cinder v2 is really keystone v303:59
jlkhttps://github.com/openstack/python-openstackclient/tree/master/openstackclient/volume03:59
mfischmeaning unused, untested03:59
jlkhrm, I expose keystone v3, as it's own named endpoint03:59
jlkbut I don't have any services using v303:59
mfischright03:59
jlkand things still seem to work with veinderv203:59
jlkah I see03:59
mfischthats what I dont want cinder v2 to be but I dont know much about it03:59
jlkI thought I saw something that in kilo dev nova would use v2 cinder in the gate03:59
mfischI dont consider setting --api-version a valid solution03:59
jlkmy biggest concern is that openstackclient doesn't even support v2 yet04:00
mfischwell then I dont know how they feel okay with deprecating it04:00
jlkbecause when that decision was made, openstackclient likely didn't exist04:01
jlkand cinderclient could easily handle v204:01
mfischit can04:01
jlkit's not really cinder's vault that openstackclient is slow to the game04:01
mfischbut I dont plan on telling all my users to pass in an argument becuase the client is dumb04:01
jlkyah04:05
jlkI'm not sure how we're going to play that one04:05
jlkhttps://bugs.launchpad.net/python-openstackclient/+bug/142492004:06
openstackLaunchpad bug 1424920 in python-openstackclient "Support volume v2 API" [Undecided,New]04:06
mfischthats not my largest worry about this04:07
mfischbut its an annoyance04:07
jlkour users are small and relatively easy to train. We might look into patching horizon to stick the volume version thing in the openrc it'll generate.04:08
mfischthats doable too04:09
mfischI think swapping the endpoint is a valid solution too but it wont work if anyone has the endpoint saved off in a script for some reason04:09
mfischnot sure if that is a real worry04:09
mfischthe bottom line is that I dont like deprecations ;)04:09
jlkwell04:10
jlkwe leave the v1 working in paste.ini, because upstream hasn't disabled it yet04:10
jlkbut we just don't /publish/ it in the catalog any more04:10
jlkI don't like deprecations either. More things should work with a discovery on the unversioned endpoint URL04:11
jlklike keystone, but christ if I can make clients figure that out04:11
mfischThat option makes sense04:11
jlkfully adopting keystone v3 is going to be a nightmare.04:11
mfischwould you also rename the v1 endpoint?04:11
jlkauth_url is littered everywhere.04:11
mfischI still plan on winning the powerball before that one04:12
jlklol04:12
jlkmaybe I'll work it out and pitch that for Tokyo. That should get me in.04:12
mfischjlk: I think I might send that mail I sent you to the list since its a topic other folks are thinking about04:12
jlkyeah, I think it's a good post. I'll respond in the morning with our work in progress04:12
jlkMy PR failed tests for some reason, other than broken nova client.04:12
mfischthx for the chat04:13
jlkoh lordy04:13
jlkapparently you NEED a volumev2 endpoint?04:14
jlkI don't understand that04:14
jlkbut alright.04:14
mfischI didnt try removing that04:14
mfischthanks for the info04:14
*** harlowja is now known as harlowja_away04:14
mfischhowabout developers never change anything ever again04:14
jlksrsly04:14
mfischwhere's the option04:14
mfisch#operatorlife04:14
mfischthen again I'm still employed04:14
jlkI'm afraid that if I leave volume v1 in place, nova-api won't use volumev2, because there is no way to /direct/ nova to use it04:15
jlkit just looks for volume or volumev2 in context04:15
jlkso maybe I have to set volume to the v2 path and volumev2 to the v2 path. because... developers.04:15
*** harlowja_away is now known as harlowja04:15
*** devlaps has joined #openstack-operators04:17
mfischmaybe some cinder-ellas will weigh in04:17
mfischkeystoners ended up with a much better name04:17
jlkhah04:18
mfischjlk: I didnt even start testing other services so please mention that when you reply04:19
jlkah okay04:19
mfischI also noticed our cinder scheduler was still the old Simple one04:19
jlkoff the top of my head, I think just heat, nova, maybe cellometer make use of it?04:19
jlkwe don't run heat or cellometer, so all we would test is nova and horizon04:19
mfischI dont think we need any features in it right now04:20
mfischbut we're going to have different volume types soon04:20
jlkI've also been playing with encrypted volumes04:20
jlkwhere nova does the encryption, cinder is aware of it04:20
mfischwe've not tried that, well I havent anyway04:22
*** radez is now known as radez_g0n304:24
jlkwe have it working with v1, haven't tried with v204:28
*** csoukup has joined #openstack-operators04:32
dvorakmfisch: seems like someone should have validated all this worked before planning to have different volume types.04:35
mfischdvorak: the v2 thing isnt the volume type issue, but our scheduler is04:39
mfischI've sent that info to the right people on our team anyway04:39
*** markvoelker has quit IRC04:39
*** markvoelker has joined #openstack-operators04:39
*** markvoelker has quit IRC04:44
*** c_soukup has joined #openstack-operators05:09
*** csoukup_ has joined #openstack-operators05:10
*** markvoelker has joined #openstack-operators05:10
*** klindgren_ has joined #openstack-operators05:11
*** klindgren has quit IRC05:13
*** csoukup has quit IRC05:13
*** c_soukup has quit IRC05:14
*** devlaps has quit IRC05:17
*** markvoelker has quit IRC05:17
*** zerda has joined #openstack-operators05:18
*** csoukup_ has quit IRC05:32
*** harlowja is now known as harlowja_away06:11
*** markvoelker has joined #openstack-operators06:13
*** markvoelker has quit IRC06:17
*** blairo has quit IRC06:42
*** reed has quit IRC06:45
*** andrewbogott has joined #openstack-operators06:59
andrewbogottI just had a whole compute node’s worth of VMs drop off the network.  The node itself seems fine…  anyone have troubleshooting suggestions?06:59
*** klindgren_ has quit IRC07:05
*** blair has joined #openstack-operators07:15
*** Miouge has joined #openstack-operators07:27
*** blair has quit IRC07:27
*** belmoreira has joined #openstack-operators07:38
*** klindgren_ has joined #openstack-operators07:44
*** Marga_ has quit IRC07:45
*** Marga_ has joined #openstack-operators07:46
*** klindgren__ has joined #openstack-operators07:46
*** klindgren_ has quit IRC07:49
*** Marga_ has quit IRC07:59
*** VW_ has joined #openstack-operators08:04
*** VW_ has quit IRC08:13
*** klindgren__ has quit IRC08:15
*** matrohon has joined #openstack-operators08:23
*** priteau has joined #openstack-operators08:27
*** blair has joined #openstack-operators08:51
*** sanjayu has joined #openstack-operators08:55
*** jawed has joined #openstack-operators09:00
*** andrewbogott is now known as andrewbogott_afk09:01
*** VW_ has joined #openstack-operators09:06
*** markvoelker has joined #openstack-operators09:17
*** derekh has joined #openstack-operators09:20
*** markvoelker has quit IRC09:22
*** VW_ has quit IRC09:22
*** bvandenh has joined #openstack-operators09:41
*** VW_ has joined #openstack-operators09:46
*** harlowja_away has quit IRC10:04
*** markvoelker has joined #openstack-operators10:18
*** markvoelker has quit IRC10:23
*** blair has quit IRC10:53
*** jawed has quit IRC10:56
*** blair has joined #openstack-operators11:00
*** subscope has quit IRC11:05
*** subscope has joined #openstack-operators11:17
*** markvoelker has joined #openstack-operators11:19
*** markvoelker has quit IRC11:24
*** VW_ has quit IRC11:36
*** blair has quit IRC11:49
*** zerda has quit IRC11:58
*** markvoelker has joined #openstack-operators12:05
*** markvoelker has quit IRC12:11
*** VW_ has joined #openstack-operators12:25
*** VW_ has quit IRC12:32
*** Miouge has quit IRC12:53
*** Miouge has joined #openstack-operators12:54
*** markvoelker has joined #openstack-operators13:06
*** markvoelker has quit IRC13:11
*** radez_g0n3 is now known as radez13:25
*** sanjayu has quit IRC13:29
*** jaypipes has quit IRC13:36
*** turnerg has joined #openstack-operators13:42
*** turnerg has quit IRC13:47
*** turnerg1 has joined #openstack-operators13:47
*** csoukup has joined #openstack-operators13:50
*** emagana has joined #openstack-operators13:56
*** Miouge_ has joined #openstack-operators13:57
*** turnerg1 has quit IRC13:58
*** Miouge has quit IRC13:58
*** Miouge_ is now known as Miouge13:58
*** radez is now known as radez_g0n313:59
*** turnerg has joined #openstack-operators14:00
*** Miouge has quit IRC14:06
*** Miouge has joined #openstack-operators14:06
*** markvoelker has joined #openstack-operators14:07
*** zerda has joined #openstack-operators14:15
*** markvoelker has quit IRC14:32
*** markvoelker has joined #openstack-operators14:33
*** markvoelker has quit IRC14:38
*** dmsimard_away is now known as dmsimard14:38
*** emccormick has joined #openstack-operators14:42
*** zerda has quit IRC14:55
*** turnerg has quit IRC14:55
*** turnerg has joined #openstack-operators14:57
*** cpschult has joined #openstack-operators15:03
*** j05h has joined #openstack-operators15:04
*** emagana has quit IRC15:04
*** emagana has joined #openstack-operators15:05
*** klindgren__ has joined #openstack-operators15:07
*** Piet has quit IRC15:08
*** emagana has quit IRC15:10
*** signed8bit has joined #openstack-operators15:10
*** mdorman has joined #openstack-operators15:14
*** devlaps has joined #openstack-operators15:24
*** devlaps has quit IRC15:24
*** csoukup has quit IRC15:43
*** david-lyle_afk is now known as david-lyle15:47
*** markvoelker has joined #openstack-operators15:49
*** hj-hp has joined #openstack-operators15:55
*** reed has joined #openstack-operators16:09
*** csoukup has joined #openstack-operators16:11
*** emagana has joined #openstack-operators16:12
*** j05h has quit IRC16:22
*** j05h has joined #openstack-operators16:23
*** SimonChung1 has quit IRC16:40
*** belmoreira has quit IRC16:42
*** radez_g0n3 is now known as radez16:42
*** emagana has quit IRC16:58
*** emagana has joined #openstack-operators16:59
*** esker has joined #openstack-operators17:03
*** emagana has quit IRC17:03
*** emagana has joined #openstack-operators17:05
mgagnejlk: so cinder v1 is deprecated and novaclient doesn't support v2 yet?17:09
klindgren__Argh - for those running juno (specifically neutron) in a medium sized environment and have a large number of workers and are getting timeouts - dial back the number of workers.17:12
mgagnejlk: just read the email, ignore me17:14
jlkklindgren__: haha, whut?17:14
jlkklindgren__: which workers?17:14
klindgren__jlk since upgrading to juno we have been getting errors usually around metadata but others as well17:15
klindgren__about timeouts talking to neutron17:15
jlkandrewbogott_afk: I'd expect that the network bridge devices went kaboom.17:15
klindgren__we have 3 app servers running 60 api/rpc workers for neutron17:15
klindgren__each.  I cut those workers down from 60 to 8 and the timeouts so far have bascially stopped17:16
jlkah. Our clusters are much smaller, 3 nodes total for many, 6~10 for others.17:16
jlkwe have our workers turned down to maybe 5 per control node (2 control nodes)17:16
mdormanmore workers not always better :)17:16
mdormanapparently17:16
jlkklindgren__: I wonder if you're running into sql contention17:16
jlkor whatever else those workers do to talk back to the API17:17
klindgren__yea - I had read some stuff around that17:18
klindgren__our issues in a way is self imposed17:19
klindgren__we store stuff in metadata to change some stuff on the vm's17:19
klindgren__so the vm's poll metadata ~every 10 minutes17:19
*** emagana has quit IRC17:19
jlkah17:19
klindgren__we staggered them to not slams the api's all at once17:20
klindgren__but it does def drive up load on services17:20
klindgren__and since every metadata request is a request to enutron....17:20
klindgren__neutron*17:20
*** klindgren__ is now known as klindgren17:21
*** matrohon has quit IRC17:23
*** emagana has joined #openstack-operators17:28
*** derekh has quit IRC17:44
*** Marga_ has joined #openstack-operators17:48
*** hj-hp has quit IRC17:48
*** tuvenen1 has joined #openstack-operators17:49
*** hj-hp has joined #openstack-operators17:49
*** signed8bit is now known as signed8bit_ZZZzz17:52
*** SimonChung has joined #openstack-operators17:55
*** tuvenen1 has quit IRC17:56
*** bvandenh has quit IRC18:05
*** andrewbogott_afk is now known as andrewbogott18:06
*** devlaps has joined #openstack-operators18:11
*** hj-hp has quit IRC18:19
*** hj-hp has joined #openstack-operators18:19
*** harlowja has joined #openstack-operators18:20
*** harlowja has quit IRC18:24
*** harlowja has joined #openstack-operators18:24
mgagneAnyone running Icehouse from UCA on Precise? I found the ipxe-precise package to be missing from UCA which looks to be required with live migration from old QEMU (1.0-1.5) to QEMU 2.018:26
mgagneotherwise live migration fails because it's looking for /usr/share/qemu/pxe-virtio.rom.12.04 which is missing18:28
*** signed8bit_ZZZzz is now known as signed8bit18:30
*** turnerg has quit IRC18:30
*** Marga_ has quit IRC18:32
*** Marga_ has joined #openstack-operators18:32
*** turnerg has joined #openstack-operators18:33
*** priteau has quit IRC18:36
jlkI read about that18:40
jlkI can't recall where, but there was a good write up on how to get that all working, and yes it does include grabbing that file from an old package18:40
mgagnefound the email from Joe Topjian and replied =)18:41
mgagneI flagged the email months ago since we had the same issue. But we just ended up pinning the package to avoid dealing with this mess.18:43
jlkhah18:52
jlkhilarious18:53
jlkput v2 in the endpoint for volume. nova volume-* works.18:53
jlkcinder bitches about there not being a volumev2 endpoint18:53
mgagnebtw, glad to see that email in openstack-dev about novaclient testing18:53
mgagnejlk: didn't Mike Perez send an email about fixing it? (cinder v2 in novaclient)18:55
jlkyeah, fixing it "next month"18:56
mgagnejlk: what I find concerning though is the fact that some deprecation path aren't fully thought or working.18:56
jlkwhich doesn't help me with issues today :/18:56
mgagnejlk: it shows a serious lack of concerns for people actually using the software18:56
mgagnejlk: or understanding of the whole openstack picture and its moving parts18:57
*** VW_ has joined #openstack-operators18:58
*** VW_ has quit IRC18:59
*** VW_ has joined #openstack-operators18:59
*** emagana has quit IRC19:03
*** emagana has joined #openstack-operators19:04
*** csoukup has quit IRC19:11
klindgrenmgagne, you mean - people actually use these things together?!?!  People run prod on this sh*t?!19:12
mgagneklindgren: unfortunately yes19:12
klindgrenI saw someone on the mailing list - something about ITAR and PCI compliance19:13
mgagneklindgren: packaging is currently a one man show here =)19:13
klindgrenand I was thinking - you know I wouldn't wantto be running openstack in a PCI environment19:13
klindgrenthat would not be cool19:13
mgagnejlk: I propose a challenge: all OpenStack projects/services should show the example and use openstackclient in M release =)19:16
*** hj-hp has quit IRC19:19
*** hj-hp has joined #openstack-operators19:24
jlkFFFFFFfffff19:26
*** j05h has quit IRC19:29
*** hj-hp has quit IRC19:30
*** hj-hp has joined #openstack-operators19:31
*** VW__ has joined #openstack-operators19:33
*** devlaps has quit IRC19:33
*** SimonChung has quit IRC19:35
*** SimonChung has joined #openstack-operators19:35
*** VW_ has quit IRC19:36
jlkoh I get it. You have to define catalog_info in [cinder] to convince nova-api to use the v2 end point instead of the v1.19:44
*** Miouge has quit IRC19:44
*** hj-hp has quit IRC19:44
mgagnejlk: yes or use cinder_endpoint_template19:45
jlkaaand I bet that default is in oslo somewhere19:46
mgagneafaik, no19:46
mgagnehttps://github.com/openstack/nova/blob/master/nova/volume/cinder.py#L40-L4719:47
jlkthat defines the option, but not hte default19:48
jlktrying to follow the code, I can't figure out what the default is19:48
jlkoh christ, I'm blind.19:48
mgagnedidn't want to say it =)19:48
jlkyeah, in juno the default is volume:cinder:publicURL19:49
*** matrohon has joined #openstack-operators19:50
*** sandywalsh_ is now known as sandywalsh20:04
*** csoukup has joined #openstack-operators20:11
*** hj-hp has joined #openstack-operators20:13
*** hj-hp has quit IRC20:14
*** turnerg has quit IRC20:15
*** turnerg has joined #openstack-operators20:16
*** turnerg has quit IRC20:18
*** Marga_ has quit IRC20:18
*** hj-hp has joined #openstack-operators20:19
*** Marga_ has joined #openstack-operators20:19
*** turnerg has joined #openstack-operators20:23
*** SimonChung1 has joined #openstack-operators20:27
*** SimonChung has quit IRC20:27
*** turnerg has quit IRC20:28
*** georgem1 has joined #openstack-operators20:28
*** hj-hp has quit IRC20:30
*** Marga_ has quit IRC20:30
*** j05h has joined #openstack-operators20:34
*** turnerg has joined #openstack-operators20:34
*** andrewbogott is now known as andrewbogott_afk20:36
georgem1anybody running 2 x 10 Gb bonded and trunked with openvswitch and can give some feedback on performance?20:44
*** VW__ has quit IRC20:46
*** VW_ has joined #openstack-operators20:47
*** georgem1 has quit IRC20:58
*** hj-hp has joined #openstack-operators20:58
*** turnerg has quit IRC21:11
*** sandywalsh has quit IRC21:22
*** sandywalsh has joined #openstack-operators21:23
klindgrengeorgem1... I guess he left21:29
*** blair has joined #openstack-operators21:29
*** Marga_ has joined #openstack-operators21:32
*** andrewbogott_afk is now known as andrewbogott21:39
*** Marga_ has quit IRC21:40
*** Marga_ has joined #openstack-operators21:41
*** matrohon has quit IRC21:41
*** Marga_ has quit IRC21:42
*** Marga_ has joined #openstack-operators21:42
*** Marga_ has quit IRC21:42
*** Marga_ has joined #openstack-operators21:43
*** Marga_ has quit IRC21:44
*** Marga_ has joined #openstack-operators21:44
*** georgem1 has joined #openstack-operators21:45
*** Marga_ has quit IRC21:45
*** Marga_ has joined #openstack-operators21:46
*** Marga_ has quit IRC21:47
*** Marga_ has joined #openstack-operators21:47
*** Marga_ has quit IRC21:51
*** Marga_ has joined #openstack-operators21:52
*** Marga_ has quit IRC21:53
*** Marga_ has joined #openstack-operators21:53
*** Marga_ has quit IRC21:54
*** Marga_ has joined #openstack-operators21:54
*** c_soukup has joined #openstack-operators22:02
*** csoukup has quit IRC22:04
*** delattec has joined #openstack-operators22:08
*** radez is now known as radez_g0n322:16
*** georgem1 has quit IRC22:33
*** Marga_ has quit IRC22:35
*** Marga_ has joined #openstack-operators22:36
*** VW_ has quit IRC22:40
klindgrenmgagne, was your rabbitmq issue a version of oslo.messaging leaking connectors?22:48
*** Marga_ has quit IRC22:49
mgagneklindgren: yes, we are getting it by the other team, thanks =)22:49
*** Marga_ has joined #openstack-operators22:49
klindgrenwhat version of oslo.messaging are they using?22:50
klindgrenwas looking at trying to package the latest tage (1.6) to see if it solves some issues for us22:50
klindgrenwant to make sure in that same vein I dont totally blow myself up :-)22:51
mgagnelooks to be 1.5.122:52
mgagneI think the other team complained that they had to pin to a commit as no recent release were available on pypi22:54
klindgrenthanks - much22:54
klindgrenyea looks like branch stable/juno is more update to date than the latest tag :-/22:54
*** hj-hp has quit IRC22:55
*** c_soukup has quit IRC22:55
mgagneabout the bug with live migration I had earlier, it looks like the source isn't running pc-1.0 but pc-i440fx-1.5. And I'm getting this error: Length mismatch: 0000:00:03.0/virtio-net-pci.rom: 20000 in != 4000022:56
mgagneso the compat patch precise -> trusty isn't working for us =(22:56
mgagneit expects pc-1.022:57
*** hj-hp has joined #openstack-operators22:57
*** Piet has joined #openstack-operators23:01
*** harlowja is now known as harlowja_away23:02
*** cpschult has quit IRC23:03
*** harlowja_away is now known as harlowja23:05
*** emagana has quit IRC23:08
*** blair has quit IRC23:08
*** andrewbogott is now known as andrewbogott_afk23:14
*** emagana has joined #openstack-operators23:17
*** hj-hp has quit IRC23:19
*** andrewbogott_afk is now known as andrewbogott23:43
*** j05h has quit IRC23:45
*** blair has joined #openstack-operators23:47
*** emagana has quit IRC23:47
*** emagana has joined #openstack-operators23:48
*** emagana has quit IRC23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!