Tuesday, 2015-06-23

*** markvoelker has joined #openstack-ansible00:00
*** darrenc_afk is now known as darrenc00:20
*** abitha has quit IRC00:24
*** stevemar has joined #openstack-ansible01:19
*** JRobinson__ has joined #openstack-ansible01:22
*** sdake has joined #openstack-ansible01:24
*** davidself has joined #openstack-ansible01:25
*** dkalleg has quit IRC01:27
*** sdake_ has joined #openstack-ansible01:27
*** sdake has quit IRC01:31
*** georgem1 has joined #openstack-ansible01:36
*** javeriak has quit IRC01:40
*** sdake_ has quit IRC01:58
*** sdake has joined #openstack-ansible01:58
*** javeriak has joined #openstack-ansible02:05
*** tlian has joined #openstack-ansible02:07
*** sdake has quit IRC02:07
*** javeriak has quit IRC02:23
*** javeriak has joined #openstack-ansible02:32
*** javeriak has quit IRC02:35
*** annashen has joined #openstack-ansible02:42
openstackgerritMiguel Grinberg proposed stackforge/os-ansible-deployment: Support SSL certs for Keystone  https://review.openstack.org/19447402:48
openstackgerritMiguel Grinberg proposed stackforge/os-ansible-deployment: Support SSL certs for Keystone  https://review.openstack.org/19447402:51
*** tlian has quit IRC03:16
*** tlian has joined #openstack-ansible03:17
*** annashen has quit IRC03:19
*** sdake has joined #openstack-ansible03:25
*** sdake_ has joined #openstack-ansible03:27
*** sdake has quit IRC03:31
*** sdake__ has joined #openstack-ansible03:31
*** sdake_ has quit IRC03:35
*** tlian has quit IRC03:44
*** abitha has joined #openstack-ansible03:54
*** georgem1 has quit IRC03:59
*** andreb has joined #openstack-ansible04:01
*** JRobinson__ is now known as JRobinson__afk04:03
andrebstevelle : you around ?04:05
stevelleyes04:05
stevellewelcome04:05
andrebstevelle: any idea what the costing for support is ?04:06
stevelleno idea04:06
andrebstevelle : ok... i am going over some docs from the rackspace site and teh stuff you shared with me earlier... going to see if i can get a full understanding of it all...  if not i hope i can find a vendor out there that is not going to screw me over with charges to do a setup and training04:07
stevelleandreb: good plan04:09
andrebstevelle : the dell reps in my country are off that list :(04:10
andrebstevelle : thanks for all the help so far :)04:16
stevelleyeah, hope you figure something out04:17
andrebstevelle : i will reading some docs now... and also searching for a vendor that does installs and training04:18
*** abitha has quit IRC04:18
*** sdake__ has quit IRC04:24
*** JRobinson__afk is now known as JRobinson__04:28
*** andreb has quit IRC04:29
*** bcoca has quit IRC04:45
*** jmccrory has quit IRC05:15
*** jmccrory has joined #openstack-ansible05:16
*** OldCrowEW has joined #openstack-ansible05:28
OldCrowEWhello05:29
*** fawadkhaliq has joined #openstack-ansible05:38
*** shausy has joined #openstack-ansible05:53
*** stevemar has quit IRC05:54
*** stevemar has joined #openstack-ansible05:54
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Config memcached connections limit and threads  https://review.openstack.org/19449905:57
*** JRobinson__ has quit IRC05:59
*** annashen has joined #openstack-ansible06:01
stevelleHello OldCrowEW06:04
OldCrowEWhi06:04
OldCrowEWhows it going?06:04
stevellefair I guess.  about time for me to depart though06:04
OldCrowEWi figured everyone was sleeping06:04
OldCrowEWhave a good night? :)06:04
stevellewe should have some of the folks in here become active in the next two hours or so06:05
OldCrowEWoh, cool06:05
OldCrowEWi'll be around06:05
stevelleI figured out what I was doing wrong, so yeah it went ok06:05
OldCrowEWi am just updating all of my images to use the local repos i setup06:05
stevellehow are things for you?06:05
OldCrowEWpretty good06:06
OldCrowEWwish i understood openstack dns better06:06
stevelleas in designate?06:06
OldCrowEWyup06:07
OldCrowEWi am in that channel but no one appears to be around06:07
OldCrowEWi dont really want to pull the "hi i'm new, answer all my questions"06:08
OldCrowEWi'll lurk for a bit06:08
stevellefair enough. I don't know it06:08
stevelleof it, but that's all06:08
OldCrowEWi was hoping it behaved like AWS06:08
OldCrowEWnot sure that it does or doesnt at this point :D06:08
stevelleI won't speculate, but if it worked like Route53 that would be pretty nice06:09
*** stevemar2 has joined #openstack-ansible06:10
*** stevemar has quit IRC06:12
*** stevemar2 is now known as stevemar06:17
*** markvoelker has quit IRC06:40
*** javeriak has joined #openstack-ansible06:41
*** fawadkhaliq has quit IRC06:55
*** fawadkhaliq has joined #openstack-ansible06:55
*** annashen has quit IRC07:16
*** javeriak has quit IRC07:23
OldCrowEWagreed.07:24
*** fawadkhaliq has quit IRC07:39
*** markvoelker has joined #openstack-ansible07:41
*** stevemar has quit IRC07:43
*** fawadkhaliq has joined #openstack-ansible07:44
*** markvoelker has quit IRC07:46
*** persia has quit IRC07:53
*** persia has joined #openstack-ansible07:53
*** persia has quit IRC07:53
*** persia has joined #openstack-ansible07:53
*** vdo_ has joined #openstack-ansible08:00
evrardjpgood morning everyone08:08
svghi evrardjp08:08
svghow's it going08:08
evrardjpatm everything is fine :) and for you?08:09
svgstill struggling with dozens of issues08:09
svgI take it you didn;t start deploying osad yet?08:10
evrardjpI'm configuring OSAD08:10
evrardjpwe have an ipv6 issue right now08:11
evrardjpagain08:11
evrardjpbut we should have the first part of the playbook runs today08:11
svgomg, I wouldn't even think of trying ipv6 too08:11
evrardjp:)08:11
*** shausy has quit IRC08:20
*** shausy has joined #openstack-ansible08:20
vincent_vdkevrardjp: living on the wild side..08:25
evrardjp:)08:26
*** shausy has quit IRC08:34
svgvincent_vdk: what are you doing here, you renegade08:38
*** markvoelker has joined #openstack-ansible09:17
*** shausy has joined #openstack-ansible09:18
*** OldCrowEW has quit IRC09:20
*** markvoelker has quit IRC09:22
odyssey4melol09:24
*** mancdaz has quit IRC09:25
*** mancdaz has joined #openstack-ansible09:25
*** OldCrowEW has joined #openstack-ansible09:44
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Add Keystone SSL key/cert generation & distribution  https://review.openstack.org/19447410:01
*** ctgriffiths_ has quit IRC10:05
*** ctgriffiths has joined #openstack-ansible10:05
*** fawadkhaliq has quit IRC10:15
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Keystone idp configuration  https://review.openstack.org/19425910:25
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Keystone SP configuration  https://review.openstack.org/19439510:34
evrardjphey odyssey4me: the 194474... is there a reason why self generated certificates are hardcoded to be from Texas? ;)10:35
odyssey4meevrardjp existing convention, really... self-signed certs are not expected to be used for anything other than testing10:36
*** OldCrowEW has quit IRC10:36
evrardjpTrue10:36
odyssey4meevrardjp we really need to revise all the SSL stuff to implement SSL offloading on haproxy instead of at Apache10:37
odyssey4methat would be more production-like... at least for us where we do the SSL offloading on the f5 load balancers10:37
*** vdo_ is now known as vdo10:58
*** markvoelker has joined #openstack-ansible11:06
*** markvoelker has quit IRC11:11
*** sdake has joined #openstack-ansible11:46
*** sdake_ has joined #openstack-ansible11:47
*** sdake has quit IRC11:50
*** sdake_ has quit IRC11:54
*** fawadkhaliq has joined #openstack-ansible11:57
evrardjpodyssey4me: I completely agree on this, although I understand the use case to have SSL everywhere, having for example EV certificates on load balancers, and self-signed everywhere else local11:57
odyssey4meevrardjp self-signed certs give you absolutely no security - what is the point?11:58
odyssey4methe only option for internally signed certs would be to have an internal CA and issue certs from there - that makes more sense11:59
evrardjpI'm not sure to get what you mean... self-signed aren't validated so it's not secure but it seems it's not inherently insecure: you still encrypt what's going over the wire, to the end component...12:02
evrardjpBTW, I though with an internal CA, not really self-signed standalones...12:03
evrardjpI meant not validated by an external provider12:03
evrardjpif it's your CA, you're still signing them yourself ... (Sorry if english is not my mother tongue, I don't subtilities)12:04
odyssey4meevrardjp if you're using self-signed certs, what prevents someone impersonating the server and thereby gaining the credentials?12:05
*** markvoelker has joined #openstack-ansible12:05
odyssey4meif you're using an internal CA which is on another server, then you still have a chain of trust in order to verify that the server is who they say they are.12:06
evrardjpthat's true, but it also mean you have other problems... Eavesdropping is doable when you don't have encryption, without impersonating the server. If you impersonate the server, that's a complete different story, that's about the complete chain of trust12:06
evrardjpI agree with you12:06
odyssey4meevrardjp so I prefer to take the approach of coming from the outside and coming in - start at the edge, get that right, then work your way in12:08
*** willemgf has joined #openstack-ansible12:09
odyssey4mebut yeah, we don't have the haproxy ssl offloading right, nor do we have the apache ssl bits done well... and we should really start with getting the keystone bits right - that's where the primary need for encryption is as that traffic holds the keys to the kingdom12:09
evrardjpindeed12:10
evrardjpwhat's the relative priorities on these items12:10
evrardjpfirst haproxy ssl offloading?12:10
odyssey4mewithin the project thus far, haproxy is used for dev/test only12:11
evrardjpyeah I read that... my question was wrongly written12:11
odyssey4meso I expect that perhaps we should get keystone ssl right first12:11
odyssey4methe haproxy setup could do with an overhaul by someone who's interested in using it for production12:12
odyssey4meor at least for some sort of long-lived cluster, not necessarily for anything too serious12:12
evrardjpso a summary: keystone > apache ssl bits > haproxy (if someone wants it in prod)12:12
odyssey4meevrardjp simply, keystone's apache ssl bits > haproxy ssl offloading for anyone who cares12:14
evrardjp :)12:14
evrardjpAre there other hardware load-balancer vendors than f5 looking for integration in OSAD?12:15
evrardjpI guess rackspace did the f5 integration12:16
evrardjpMaybe vendors are willing to do it too...12:16
odyssey4meevrardjp no offers just yet - it's still early days though12:17
evrardjpok thanks for all that info12:18
odyssey4meevrardjp of course you can quite capably improve the ssl bits :)12:20
evrardjpYeah I brought the contributor's agreement on the topic today to my management12:20
evrardjpif we look into the ssl things, we'll also look at haproxy first12:22
odyssey4meevrardjp either way works :)12:22
odyssey4meis there resistance to the CLA in your work environment?12:23
*** fawadkhaliq has quit IRC12:32
evrardjpnot the CLA particularily, just needs to go through our legal unit (which is a bottleneck atm)12:43
evrardjpshould it?12:43
evrardjpI mean, did you experience resistance to the CLA in the past?12:44
odyssey4meevrardjp it shouldn't - the CLA just basically says that things you contribute to the project are not something you can try and sue anyone for later - essentially if you contribute IP, the IP is part of the public domain and no longer yours12:45
odyssey4meit should only incur resistance if you materially produce patentable IP every day - and I do question any patents registered in operations software these days12:45
evrardjp:)12:46
evrardjpYeah, that's what I also thought12:47
odyssey4mefyi hughsaunders the log file you're looking for is /var/log/shibboleth/shibd.log and if the xml files are changed, the shibd service needs to be restarted to pick up the changes12:47
evrardjpI didn't read the paper so I was curious when you mentionned resistance12:47
odyssey4meevrardjp I was just wondering why it was taking so long :)12:47
*** jwagner is now known as jwagner_away12:52
cloudnullMorning12:55
evrardjpgood morning cloudnull12:56
cloudnullHow goes it?12:56
cloudnullSvg did you ever get the token issues resolved ? (Too many sql connections) ?12:57
* cloudnull was rather occupied yesterday and didn't check back with you , sorry. 12:58
svgI changed the config to mysql backend, but since then we again had several issues - testing a heat deploy that went well with memcached tokens + everything up, nof miserably failss12:58
svgto the point we can't manage to delete the stacks12:59
svgso far not sure what the causing issue is now..12:59
cloudnullIs it an api error?13:00
cloudnullOr something else?13:01
svgclient side point of view, no errors, pretty much 50% of the stacks end in a fail status13:01
svgso far I haven't been able to pinpoint something specific that caused it13:01
svgcoworker told me there were some amqp issues, and he couldnt restart the rabbit, had to force kill it....13:02
svgall in all this is getting frustrating over here I'm afraid13:02
cloudnullI'd imagine.13:02
*** davidself has quit IRC13:03
cloudnullMiguelgrinberg has worked a lot with heat , when he wakes up maybe he can help if you've not nailed it down by then.13:04
svgSo far we still didn;t manage to get an agreement for support, but Kevin is doing a bit of suppor tby mail for now13:05
svgHe suggested it wasn't a good idea to try running the kilo release, and that we should stick to the better tested juno/10/rpc stack13:05
svgDo you have thoughts on that?13:05
*** tlian has joined #openstack-ansible13:06
cloudnullRax support doesn't run kilo yet.13:06
*** yaya has joined #openstack-ansible13:06
cloudnullThey need to train everyone up before they ad a new product.13:07
svgsure13:08
cloudnullFrom a tested / stability standpoint IMO kilo is where you should be. But others might have thoughts on that.13:08
svgok13:08
cloudnullThe memcached tokens thing is a problem though. And we've redoubled our efforts on fernet.13:17
*** KLevenstein has joined #openstack-ansible13:17
svgYes, I noticed that, thanks for that.13:17
cloudnullIn kilo you could run fernet , just like you can run sql, you can cherry pick the review from master to enable the functionality in the roles. But most of the big players are switching over to it. Due to the endless issues with the other drivers.13:17
cloudnullHowever that wouldn't be tested at this point.13:17
svgmight be an alternate path here and now, ut yes, perhaps too soon13:17
*** openstack has quit IRC13:17
*** openstack has joined #openstack-ansible13:20
*** yaya has quit IRC13:25
cloudnullIt's exactly what you have now with the change to sql.13:25
*** sdake_ has joined #openstack-ansible13:32
*** fawadkhaliq has joined #openstack-ansible13:33
svgah13:36
*** fawadkhaliq has quit IRC13:39
*** yaya has joined #openstack-ansible13:49
*** willemgf has quit IRC14:02
*** sigmavirus24_awa is now known as sigmavirus2414:07
*** jmccrory has quit IRC14:10
*** jmccrory has joined #openstack-ansible14:11
*** sdake_ has quit IRC14:12
openstackgerritgit-harry proposed stackforge/os-ansible-deployment: Fix errors when enabling SSL for apache  https://review.openstack.org/19467214:14
*** jrniemijr has joined #openstack-ansible14:14
*** jwagner_away is now known as jwagner14:42
*** stevemar has joined #openstack-ansible14:46
*** sdake has joined #openstack-ansible14:51
*** alextricity_h has joined #openstack-ansible14:55
*** galstrom_zzz is now known as galstrom14:56
*** shausy has quit IRC15:12
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Add Keystone SSL key/cert generation & distribution  https://review.openstack.org/19447415:21
cloudnullso this is a new thing in master http://logs.openstack.org/44/193844/4/check/os-ansible-deployment-dsvm-check-commit/b7b622c/console.html#_2015-06-22_23_39_01_75315:34
cloudnullit seems there have been changes in tempest / nova v2.1 that we're going to need to account for, luckily that a liberty thing.15:35
miguelgrinbergsvg: heat problems resolved, or still have trouble?15:37
cloudnull^ heat guru =)15:38
svgnot resolved, but scratched the heat db...15:40
*** sdake has quit IRC15:42
svgthough I tried redeploying some stacks, and some go in failed state again15:43
*** Mudpuppy has joined #openstack-ansible15:44
svgso far I just checked the nova logs, but seems they don;t hit that yet15:44
svgthis started happening just after we changed the keystone token backend from memcache to mysql, so chances are big that is related15:45
miguelgrinbergsvg: what do the heat logs say?15:46
svgI didn't had the chance to check those yet, heading home by train right now15:47
svglet me see if I can access that15:47
miguelgrinbergany time a stack fails the heat-engine service should say something15:47
miguelgrinbergok15:47
miguelgrinbergI suspect if this is token related you may see 401s15:48
svgConflict: Unable to complete operation on subnet 9664e1e1-5efb-494b-b882-3f30da65fb6b. One or more ports have an IP allocation from this subnet.15:49
svgbut this is from later, after I tried deleting a failed stack15:50
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource InternalServerError: Request Failed: internal server error while processing your request.15:52
miguelgrinbergis there a stack trace on the internal server error?15:52
miguelgrinbergactually no, the stack trace is going to be on the other side15:53
svg hold on flacky network :)15:53
miguelgrinbergdoes it tell you what resource type gave that error?15:53
svg2015-06-23 17:08:15.917 3099 INFO heat.engine.resource [-] CREATE: Port "node1_port" Stack "test-stack1" [e6e5bc1b-dc2b-4da8-b494-2870bbc40163]15:53
*** fawadkhaliq has joined #openstack-ansible15:53
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource Traceback (most recent call last):15:53
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 489, in _action_recorder15:53
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     yield15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 559, in _do_action15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     yield self.action_handler_task(action, args=handler_args)15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/scheduler.py", line 296, in wrapper15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     step = next(subtask)15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resource.py", line 530, in action_handler_task15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     handler_data = handler(*args)15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/heat/engine/resources/openstack/neutron/port.py", line 279, in handle_create15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     port = self.neutron().create_port({'port': props})['port']15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 99, in with_params15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     ret = self.function(instance, *args, **kwargs)15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 507, in create_port15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     return self.post(self.ports_path, body=body)15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 295, in post15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     headers=headers, params=params)15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 208, in do_request15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     self._handle_fault_response(status_code, replybody)15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 182, in _handle_fault_response15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     exception_handler_v20(status_code, des_error_body)15:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource   File "/usr/local/lib/python2.7/dist-packages/neutronclient/v2_0/client.py", line 67, in exception_handler_v2015:54
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     status_code=status_code)15:55
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource InternalServerError: Request Failed: internal server error while processing your request.15:55
svg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource15:55
svgoops15:55
palendaeGah15:55
svgsorry, that should have been http://paste.ubuntu.com/11762898/15:55
miguelgrinbergokay, so neutron returned a 50015:55
miguelgrinbergnow you need to check the neutron logs :)15:55
openstackgerritHugh Saunders proposed stackforge/os-ansible-deployment: [WIP] Keystone SP configuration  https://review.openstack.org/19439515:58
svgI seem to only find errors on the delete operations15:58
miguelgrinbergthis was specifically on a port creation call16:00
miguelgrinbergthis call:16:01
miguelgrinberg2015-06-23 17:08:15.917 3099 TRACE heat.engine.resource     port = self.neutron().create_port({'port': props})['port']16:01
b3rnard0we having any bug triaging?16:01
cloudnullmeeting cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, mancdaz, dolphm, _shaps_, BjoernT, claco, echiu, dstanek, jwagner16:02
* dstanek is lurking from my phone16:02
palendaePresent16:03
odyssey4meo/ although mainly lurking :p16:03
andymccro/16:03
b3rnard0hello16:04
sigmavirus24hi16:04
*** sdake has joined #openstack-ansible16:05
cloudnullso there are only three new items16:06
cloudnullfisrt up https://bugs.launchpad.net/openstack-ansible/+bug/146693016:06
openstackLaunchpad bug 1466930 in openstack-ansible "Ensure user task happens before LDAP config is in place" [Undecided,New]16:06
cloudnullthis looks incomplete16:07
*** Bjoern__ has joined #openstack-ansible16:08
*** Bjoern__ is now known as BjoernT16:08
svgmiguelgrinberg: db deadlock : http://paste.ubuntu.com/11762977/16:08
svgdarn16:08
BjoernTeven with active/passive load balancing for galera ?16:08
odyssey4mecloudnull I would agree. BjoernT may be right that an attribute wasn't set to ensure that stuff was in place.16:08
cloudnullthis is likely however paui hasn't replied back to BjoernT comment so at this point im inclined to mark it as incomplete.16:09
cloudnullor invalid16:10
andymccrid go with incomplete16:11
andymccrgive it a bit for paul to reply then invalid if nothing comes up :)16:11
BjoernTwhich bug are we talking off ?16:11
cloudnullhttps://bugs.launchpad.net/openstack-ansible/+bug/146693016:11
openstackLaunchpad bug 1466930 in openstack-ansible "Ensure user task happens before LDAP config is in place" [Undecided,Incomplete]16:11
BjoernTthanks16:12
cloudnullNext https://bugs.launchpad.net/openstack-ansible/+bug/146711816:13
openstackLaunchpad bug 1467118 in openstack-ansible "Missing packages after ./scripts/teardown.sh" [Undecided,New]16:13
cloudnullit looks like the issue was an intermitent failure with the "http://ppa.launchpad.net/adiscon/v8-stable/ubuntu/dists/utopic/main/binary-amd64/Packages" repo .16:13
cloudnulli think we should just remove that ppa16:14
cloudnullin master/kilo we dont need it16:14
*** Mudpuppy has quit IRC16:14
odyssey4mecloudnull I would agree to that - it's an external dependancy which adds no value16:15
cloudnull+116:15
andymccr+116:16
*** Mudpuppy has joined #openstack-ansible16:16
andymccrif its an external dependency adding nothing why is it even there?!16:16
cloudnullwe had the v8 repo ppa in juno16:17
cloudnullit was used for newer versions of rsyslog16:17
openstackgerritMerged stackforge/os-ansible-deployment: Add support for deploying Keystone with Fernet  https://review.openstack.org/19419416:17
andymccrahh got you16:17
cloudnullwhich was required for all our old logging bits16:17
andymccryeh i remember16:17
cloudnullok16:18
cloudnulltriaged to remove that ppa16:18
cloudnullLast https://bugs.launchpad.net/openstack-ansible/+bug/146777316:19
openstackLaunchpad bug 1467773 in openstack-ansible "python-openstackclient install failed in kilo 11.0.4" [Undecided,New]16:19
odyssey4methat looks like it's using the upstream repo sync, and that's missing something16:20
cloudnullyea16:21
*** annashen has joined #openstack-ansible16:22
cloudnullit may not be a recent build though, looking at the report http://rpc-repo.rackspace.com/reports/kilo.json16:22
cloudnullall of the entries for "oslo.config" are the same ">=1.11.0"16:22
cloudnullwhich includes openstackclient16:23
cloudnullso ill ask for more data and to see if a rerun helps.16:28
andymccrsounds good - if we can recreate it'd be easy to confirm but we havnt seen that issue on any of the gates?16:31
odyssey4meandymccr yeah, the gate builds its own repo - so this is only to do with the sync from rpc-repo16:31
cloudnullok. we;re done here16:33
cloudnullunless we want to talk about more things16:33
odyssey4mebetter to add 'more things' to the agenda for thu16:34
cloudnullwe got time16:35
cloudnulltheres no need to stand on ceremony :)16:35
*** daneyon has left #openstack-ansible16:40
*** fawadk has joined #openstack-ansible16:42
cloudnullsvg:  going back to your deadlocks, is your galera cluster running in all active mode from the lb ?16:42
*** fawadkhaliq has quit IRC16:42
miguelgrinbergcloudnull: svg is offline for about an hour16:43
cloudnullif so that'll make neutron very unhappy. in liberty i believe that upstream (liberty) has fixed most of those issus however i dont think those fixes will ever make it back to kilo.16:44
cloudnullmiguelgrinberg:  ah16:44
miguelgrinbergcloudnull: absolutely no relation to the token storage changes he's made, right?16:45
cloudnull i dont believe so .16:46
cloudnullbut would explain some of the other issues that have been talked about.16:46
miguelgrinbergright16:48
*** vdo has quit IRC16:48
cloudnullwhat is the channel tempest is developed in ?16:51
cloudnullopenstack-tempest seems empty16:51
cloudnull#openstack-qa im thinking16:53
palendaeA heads up - http://lists.openstack.org/pipermail/openstack-dev/2015-June/067795.html16:55
openstackgerritMiguel Alejandro Cantu proposed stackforge/os-ansible-deployment: Implement Ceilometer[WIP]  https://review.openstack.org/17306716:56
alextricity_hInteresting...16:57
alextricity_hMight be the reason behind this: + echo 'TEMPEST FAIL scenario heat_api cinder_backup (15 tests)'16:57
alextricity_hAh..nevermindm e16:59
*** dontalton has joined #openstack-ansible17:01
*** fawadk has quit IRC17:06
*** alextricity_h has quit IRC17:11
openstackgerritDavid Alfano proposed stackforge/os-ansible-deployment: Rename group rpc to openstack  https://review.openstack.org/19474917:13
*** sacharya has joined #openstack-ansible17:18
*** yaya has quit IRC17:18
svgcloudnull: miguelgrinberg should be, but yes, one of the nextthings to check17:26
svgthough 'deadlock' seems to imply something else17:26
svgin the mean time, since all troubles we had and all manual sql cleanups my colleague had to do, it might be frackup too17:27
svgso I'm considering retesting from a clean plate17:27
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: [WIP] Keystone SP configuration  https://review.openstack.org/19439517:29
*** javeriak has joined #openstack-ansible17:29
odyssey4memiguelgrinberg ^ I've added some more bits after hughsaunders' work putting together much of what we figured out today. I've begun testing using https://www.testshib.org to give a known working IDP/SP to test against, which you may also wish to do.17:30
*** dkalleg has joined #openstack-ansible17:31
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Revert "changed container bind mounts to use abspath"  https://review.openstack.org/19475917:38
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Revert "changed container bind mounts to use abspath"  https://review.openstack.org/19476017:38
cloudnullguys. these two commits went in and sadly they're breaking the bindmounts for logging.17:39
cloudnullexample http://logs.openstack.org/74/194474/4/check/os-ansible-deployment-dsvm-check-commit/6c0a1eb/logs/17:39
cloudnullwe have no gate logs except whats on the host17:39
cloudnulland in test lxc accepts the abs path but does not actually bind mount the things.17:40
cloudnullso we need to revert those commits with a quickness. :\17:43
miguelgrinbergodyssey4me: awesome. I'm now trying to figure out how to do the haproxy bit for Keystone, so that we can have the SSL keystone working on our AIOs.17:45
*** fawadkhaliq has joined #openstack-ansible17:54
annashenanyone knows where can i find sample dynamic inventory scripts mentioned in this page http://docs.ansible.com/intro_dynamic_inventory.html ?17:55
*** Mudpuppy_ has joined #openstack-ansible18:05
*** yaya has joined #openstack-ansible18:05
*** TheIntern has joined #openstack-ansible18:07
*** Mudpuppy has quit IRC18:08
cloudnullannashen: https://github.com/ansible/ansible/tree/devel/plugins/inventory18:09
cloudnullthats a large collection of dyn inv scripts18:09
*** Mudpuppy has joined #openstack-ansible18:09
annashenthanks cloudnull!18:09
cloudnullanytime18:09
*** Mudpuppy_ has quit IRC18:09
cloudnullodyssey4me: miguelgrinberg: we'll need to upgrade haproxy to 1.5 which will need to come from source or from some other repo .18:12
*** fawadkhaliq has quit IRC18:16
*** jwagner is now known as jwagner_away18:22
jmccroryIs haproxy 1.5 required for SSL support?18:28
*** annashen has quit IRC18:33
*** jwagner_away is now known as jwagner18:34
cloudnullyes18:41
cloudnullssl termination didnt land in haproxy until 1.5.x18:41
cloudnullversion 1.5 : the most featureful version, supports SSL, IPv6, keep-alive, DDoS protection, etc18:44
cloudnullversion 1.4 : the most stable version for people who don't need SSL. Still provides client-side keep-alive18:44
cloudnullfrom http://www.haproxy.org/18:44
jmccroryhmm good to know, working through PCI readiness for next couple months at least...18:48
cloudnullthat sounds like a lot of fun18:48
jmccroryheh a whole lot18:49
*** KLevenstein has quit IRC18:59
*** davi8784 has joined #openstack-ansible18:59
*** TheIntern has quit IRC19:00
*** davi8784 is now known as TheIntern19:00
*** yaya has quit IRC19:03
palendaePCI is always fun19:07
sigmavirus24PCI is more fun than PKI19:11
sigmavirus24Just saying19:11
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Remove the v8 repos because they're not needed  https://review.openstack.org/19479019:12
*** KLevenstein has joined #openstack-ansible19:13
cloudnullcores please make these go https://review.openstack.org/#/c/194760/ https://review.openstack.org/#/c/194759/19:15
palendaecloudnull: A clarification - this is version 8 of rsyslog, not v8 the runtime?19:16
*** yaya has joined #openstack-ansible19:16
cloudnullyes19:16
palendaeOk19:17
cloudnullthe repo is "v8-stable"19:17
*** yaya has quit IRC19:19
sigmavirus24cloudnull: why are we reverting those?19:21
*** openstackgerrit has quit IRC19:21
cloudnullpalendae: its a revert pr i guess we could change the pr commit message.19:22
*** yaya has joined #openstack-ansible19:22
*** openstackgerrit has joined #openstack-ansible19:22
palendaesigmavirus24: There's a note in the ticket that 'it breaks' :p19:22
palendaeEr bug, issue...thing19:22
sigmavirus24which one?19:22
palendaehttps://bugs.launchpad.net/openstack-ansible/+bug/146206819:23
openstackLaunchpad bug 1462068 in openstack-ansible trunk "lxc container create bind mounts should use the full path" [Medium,In progress] - Assigned to Kevin Carter (kevin-carter)19:23
cloudnullsigmavirus24:  compare http://logs.openstack.org/60/194760/1/check/os-ansible-deployment-dsvm-check-commit/0f191a7/logs/ and http://logs.openstack.org/74/194474/4/check/os-ansible-deployment-dsvm-check-commit/6c0a1eb/logs/19:23
palendaeBut your questions are exactly why I was asking19:23
cloudnullthe bind mounts are broken19:23
palendaeHad to go to commit -> reverted commit -> LP19:23
sigmavirus24got it19:23
cloudnullpalendae:  thats the commit message generated from the gerrit revert button19:23
palendaeAh19:23
palendaegfj gerrit19:23
sigmavirus24Yeah, it looks like a typically git revert message19:23
palendaeOh, yeah, that's bad19:23
sigmavirus24+ Change-Id19:24
sigmavirus24not going to lie, the gerrit flow has really grown on me19:24
palendaeOk, then I'll ignore it19:24
palendaeSince it's what the tooling does, and wasn't a manual message19:24
sigmavirus24Typically git gives you the option to append to the message to19:24
sigmavirus24or rewrite it19:24
sigmavirus24git revert --no-edit is probably what gerrit is using19:24
palendaeProbably19:24
sigmavirus24or whatever the equivalent is in the JGit bindings19:24
palendaeMakes it push button19:24
* sigmavirus24 thinks it's still called JGit19:25
sigmavirus24pssst, people should look at https://review.openstack.org/#/c/181007/ because it'll be crucial for our ADFS/Keystone federation work19:26
stevemarsigmavirus24, ask in #openstack-keystone :\19:28
* sigmavirus24 was hoping people here could look at it too19:28
sigmavirus24Has a whole bunch of +1s and figured I'd bug our people before bugging the Keystone folk19:28
sigmavirus24In case there is something someone picks up on19:28
sigmavirus24Just in case19:28
sigmavirus24;)19:28
sigmavirus24stevemar: I like to outsource the nitpicking to other projects =P19:29
*** annashen has joined #openstack-ansible19:34
*** annashen has quit IRC19:40
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Added a nova.conf option for instance_passwords  https://review.openstack.org/19479619:42
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated master to the latest SHAs - 06.20.2015  https://review.openstack.org/19384419:44
*** sdake has quit IRC19:54
odyssey4mesigmavirus24 stevelle miguelgrinberg dolphm dstanek Do you guys know how it is that the exaple/sample config files are generated? (eg: keystone.cfg, etc?)19:54
sigmavirus24tox -e genconfig19:54
odyssey4me*example19:54
sigmavirus24What's up odyssey4me ?19:54
odyssey4mesigmavirus24 so it's simply git clone the project, then 'tox -e genconfig' ?19:55
sigmavirus24Yessir19:55
odyssey4methanks sigmavirus24 - I'm taking some personal time to look into doing the tunable configs a different way... our vars are growing and getting out of hand - there has to be a better way of catering fo rthe generally obscure and ever-changing options available19:56
stevelleodyssey4me: unless it's cinder in which case lol19:56
odyssey4mestevelle oh? is that because of all the drivers?19:57
sigmavirus24drivers + quotas19:57
dolphmodyssey4me: in keystone i think it's tox -e config19:57
sigmavirus24the quotas are pretty dynamic19:57
stevelleodyssey4me: yes, including postgres19:57
dolphmodyssey4me: check the project's tox.ini file19:57
sigmavirus24dolphm: why does keystone have to be so unique? =P19:58
stevellecan't generate a config file w/o postgres installed19:58
dolphmsigmavirus24: i guess ours is genconfig now https://github.com/openstack/keystone/blob/master/tox.ini#L112-L11319:58
dstanekdolphm: yeah, it was changed to conform19:59
sigmavirus24I bet swift has it as gifnocneg just to stick it to the conformance checker19:59
cloudnullodyssey4me:  neutron doesnt use the genconfig either.20:00
cloudnullall of there example files are created by hand20:00
odyssey4mecloudnull *sigh*20:00
sigmavirus24artisinal20:00
sigmavirus24glance doesn't exactly use genconfig either20:00
sigmavirus24although it exists20:00
cloudnullhand crafted config files for the win20:00
sigmavirus24kragniz tried to fix that last cycle but there wasn't much attention paid to that effort20:01
odyssey4mewell, with what I have in mind it won't matter too much - let's see if I can make it go20:01
cloudnulli think Sam-I-Am worked on the neutron side too20:01
cloudnulltc meeting for bigtent today in #openstack-meeting20:03
cloudnullstarting now20:03
*** alextricity_h has joined #openstack-ansible20:14
*** KLevenstein has quit IRC20:19
stevemardolphm, we changed it to genconfig so we can match up with nova20:20
stevemari had a patch to get the proposal bot to propose new config changes20:21
dolphmstevemar: around the same time we switch to oslo-config-generator?20:21
stevemardolphm, no, well after20:21
openstackgerritMerged stackforge/os-ansible-deployment: Revert "changed container bind mounts to use abspath"  https://review.openstack.org/19476020:21
stevemarearly liberty20:21
stevemardolphm, https://review.openstack.org/#/c/177620/20:21
*** KLevenstein has joined #openstack-ansible20:21
alextricity_hHey, has anybody had any issues with the tempest cinder_backup tests on the AIO? I'm running  the gate-check-commit.sh script on my code but recieving this: http://pastebin.com/GPP7JYsP20:30
alextricity_hCan somebody take a quick look to see if it looks familiar?20:31
*** dontalton has quit IRC20:32
*** dontalton has joined #openstack-ansible20:33
sigmavirus24alextricity_h: looking20:34
alextricity_h@sigmavirus24 thanks20:34
*** KLevenstein has quit IRC20:34
sigmavirus24alextricity_h: that's weird that it's timing out20:34
*** yaya has quit IRC20:35
alextricity_hhmm..it's just a standard AIO build on a public cloud VM20:35
alextricity_hI didn't do anything fancy20:35
*** jwagner is now known as jwagner_away20:36
sigmavirus24What happens if you create a cinder volume from the CLI20:36
alextricity_hI created one about 10 min ago. It's stilli in the 'creating' status20:37
alextricity_hI'm going to verify my cinder services20:37
alextricity_hNothing significant in the cinder logs :/20:39
alextricity_hand the services are up and running20:39
alextricity_hHmm..I don't see my logical volumes being created in the cinder_volumes_container20:41
odyssey4mealextricity_h if you're kicking the tires and experimenting, gate-check-commit is not the best tool... it's built for a one-time-run20:46
alextricity_hEven after a teardown?20:46
palendaeI don't think teardown accounts for tempest20:46
palendaeAlso, teardown requires a reboot if you rebuild20:47
odyssey4meit's better to clone the repo, checkout the branch/tag, run the bootstrap-aio and bootstrap-ansible scripts... then use run-playbooks to kick off the playbooks20:47
alextricity_hodyssey4me I typically run scripts/teardown.sh before running gate-check-commit20:47
palendaeBecause the kernel won't let go of the volume groups20:47
odyssey4meteardown does a reasonable job, but you only need to do it if you actually need to destroy everything - and with a cloud server you're better off just rebuilding the cloud server20:48
alextricity_hpalendae oh i didn't know that. Well right now cinder_volumes_container is showing I have the cinder-volumes volume group, but no logical volmues20:48
palendaealextricity_h: Yeah, it gets...weird.20:48
palendaeI've spent a few hours trying to force the kernel to let go without a reboot, but to no avail20:48
palendaeSo I generally do what odyssey4me just suggested20:48
odyssey4mepalendae we should probably drop the cloudserver-aio script and change the docs to recommend using this method instead20:49
alextricity_hpalendae, odyssey4me: I'm seeing this error on the jenkin slaves though. So it must be a problem with the addition of ceilometer20:49
alextricity_hI'm also seeing it on my local AIO cloud VM20:50
palendaealextricity_h: Is this master?20:50
alextricity_hmy cloud VM*20:50
cloudnullbig tent here we come - https://review.openstack.org/#/c/191105/220:50
alextricity_hpalendae: i rebased master this morning.20:50
alextricity_hso yes20:50
palendaeOk20:50
palendaealextricity_h: But master with changes?20:51
alextricity_hpalendae: What do you mean?20:51
*** annashen has joined #openstack-ansible20:51
odyssey4mealextricity_h so, if you're using RAX cloud and the standard 8cpu-8G RAM image then it probably has one disk and uses a loopback image for the cinder-volumes20:51
*** KLevenstein has joined #openstack-ansible20:52
palendaealextricity_h: Was it master, or your own change set rebased on master?20:52
odyssey4mealextricity_h oh yes, you should look at all the changes in master - a ton of restructures of where configs live and stuff have gone in20:52
alextricity_hodyseey4me: right. I didn't change anything in the gate scripts that would affect those configurations20:52
alextricity_hpalendae: I git pulled/fetch from master, then rebased my bp/ceilometer branch with master20:53
alextricity_hThat was this morning20:53
palendaeOk20:53
odyssey4mecloudnull woohoo! :)20:53
palendaeI ask because, afaik, the gate is passing on commit checks20:53
odyssey4mepalendae alextricity_h but the ceilometer review has not been: https://review.openstack.org/17306720:54
palendaeRight, kind of what i was getting at20:54
palendaeWonder if there's something in that patch that's throwing a wrench in cinder tests somehow20:54
*** yaya has joined #openstack-ansible20:56
alextricity_hpalendae. That's what i'm trying to pin point :/ I'll try this on a fresh VM just in case20:58
alextricity_hBut as of now my cinder volumes are just hanging on the 'creating' status20:59
palendaeYeah, I would say first thing I'd check is a fresh one without doing teardown.sh20:59
*** Mudpuppy has quit IRC21:00
odyssey4mealextricity_h I've also added some review comments which I'd advise looking into as they may help21:02
*** Mudpuppy has joined #openstack-ansible21:03
*** Mudpuppy has quit IRC21:03
*** Mudpuppy has joined #openstack-ansible21:04
alextricity_hodyssey4me: Thanks :)21:04
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Remove the adiscon/v8 ppa because it's not needed  https://review.openstack.org/19479021:12
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Remove the adiscon/v8 ppa  https://review.openstack.org/19479021:16
*** fawadkhaliq has joined #openstack-ansible21:17
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Remove the adiscon/v8 ppa  https://review.openstack.org/19479021:17
*** fawadkhaliq has quit IRC21:21
*** abitha has joined #openstack-ansible21:28
openstackgerritMerged stackforge/os-ansible-deployment: Revert "changed container bind mounts to use abspath"  https://review.openstack.org/19475921:30
*** jrniemijr has quit IRC21:38
*** tlian2 has joined #openstack-ansible21:38
*** tlian has quit IRC21:41
*** sacharya has quit IRC21:45
*** JRobinson__ has joined #openstack-ansible21:49
sigmavirus24alextricity_h: try a performance 1-1521:50
sigmavirus24I've been using those for AIOs instead21:50
palendae2-1521:50
palendaeIf you're using RAX ones21:50
palendae1 maxes at 821:50
cloudnull++ 2-15 is what i dev on21:51
alextricity_hOkay. I was running the run_playbooks.sh script just now and it keeps failing because my containers are unreachable :/21:51
*** annashen has quit IRC21:51
alextricity_hI did what you suggested, odyssey4me21:51
palendaeyeah, 2-15 seems most comfortable. Too bad it's not 'standard'21:51
sigmavirus24er yeah21:51
sigmavirus242-15 sorry21:51
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Configurable memcached connections limit & threads  https://review.openstack.org/19449921:58
*** yaya has quit IRC21:58
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: Updated tempest isolation options  https://review.openstack.org/19434421:59
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: [WIP] Updated keystone to use fernet as the default  https://review.openstack.org/19372921:59
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Configurable memcached connections limit & threads  https://review.openstack.org/19449922:00
*** Mudpuppy has quit IRC22:09
*** alextricity_h has quit IRC22:11
*** tlian2 has quit IRC22:11
*** annashen has joined #openstack-ansible22:13
*** annashen has joined #openstack-ansible22:13
*** dontalton2 has joined #openstack-ansible22:19
*** KLevenstein has quit IRC22:24
*** sigmavirus24 is now known as sigmavirus24_awa22:26
*** TheIntern has quit IRC22:30
*** yaya has joined #openstack-ansible22:34
*** annashen has quit IRC22:35
*** galstrom is now known as galstrom_zzz22:41
*** annashen has joined #openstack-ansible22:58
*** sdake has joined #openstack-ansible23:07
*** dontalton2 has quit IRC23:16
*** dontalton has quit IRC23:16
*** sdake_ has joined #openstack-ansible23:23
*** BjoernT has quit IRC23:25
*** sdake has quit IRC23:26
*** stevemar has quit IRC23:29
*** sdake_ has quit IRC23:40
*** annashen has quit IRC23:42

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!