Wednesday, 2015-08-26

*** arbrandes has joined #openstack-ansible00:09
*** sdake_ has joined #openstack-ansible00:13
*** sdake has quit IRC00:16
*** BjoernT has quit IRC00:17
*** openstack has joined #openstack-ansible00:35
*** woodard has quit IRC00:41
*** cloudtrainme has joined #openstack-ansible00:46
*** sigmavirus24 is now known as sigmavirus24_awa00:46
*** shoutm has quit IRC00:47
*** shoutm has joined #openstack-ansible00:49
*** sdake has joined #openstack-ansible00:56
*** sdake_ has quit IRC00:59
*** shoutm has quit IRC01:00
*** shoutm has joined #openstack-ansible01:00
*** shoutm has quit IRC01:07
*** shoutm has joined #openstack-ansible01:09
*** shoutm has quit IRC01:27
*** cloudtrainme has quit IRC01:33
*** shoutm has joined #openstack-ansible01:40
openstackgerritMerged stackforge/os-ansible-deployment: Read affinity from environment  https://review.openstack.org/21590301:57
*** javeriak has joined #openstack-ansible02:57
*** javeriak has quit IRC03:15
*** fawadkhaliq has joined #openstack-ansible03:58
*** tlian has quit IRC04:21
*** shoutm_ has joined #openstack-ansible04:32
*** shoutm has quit IRC04:34
openstackgerritMerged stackforge/os-ansible-deployment: Removed default lxc profile on container create  https://review.openstack.org/21630104:52
openstackgerritMerged stackforge/os-ansible-deployment: Fixes loops for bashate  https://review.openstack.org/21590405:38
*** shausy has joined #openstack-ansible05:41
*** shausy has quit IRC05:53
*** shausy has joined #openstack-ansible05:54
*** javeriak has joined #openstack-ansible06:43
evrardjpxar-: znc is indeed nice :)06:46
evrardjpgood morning everyone06:46
matttevrardjp: morning2u06:52
*** javeriak has quit IRC07:14
*** fawadkhaliq has quit IRC07:15
*** cristicalin has joined #openstack-ansible07:22
cristicalinanybody else seeing kernel traces about corrupt packets and bad offloading when setting up containers for OSAD with ubuntu 14.04 ?07:25
cristicalinI managed to repro this with all 3 official kernel generations for 14.04 on different hardware07:25
cristicalinas far as I can tell the trace is generated my the internal interfaces (veths or linux bridges) not the external interface07:26
cristicalinhttp://pastebin.com/5jgXVCRd here's a kern.log for anybody interested and knowledgable to look into the issue07:27
*** gparaskevas has joined #openstack-ansible07:30
*** fawadkhaliq has joined #openstack-ansible07:34
*** fawadkhaliq has quit IRC07:38
odyssey4mecristicalin is this for an AIO or for a multinode deployment? also, is it on real hardware/vm's/?07:40
odyssey4methis is something we've seen, and it seems that it relates to container veths which shouldn't be there any more - Apsu put together a script to clean up after containers when you restart them: https://gist.github.com/Apsu/7947a3347fcc86bb45a707:42
odyssey4methat said, what you're seeing may not be the same thing - would you mind registering a bug for this so that we can look into it properly and perhaps also report it upstream if necessary07:42
cristicalinodyssey4me, it's an AIO running in an openstack instance (so VM)07:43
cristicalinsure, grad to register a bug, but where ?07:43
odyssey4mecristicalin ok, so it may be the same thing that I've often seeing - but it has no impact on general functionality07:43
cristicalinon github ?07:43
odyssey4mecristicalin https://bugs.launchpad.net/openstack-ansible07:44
cristicalinodyssey4me, yes, looks like it completes even with the traces so I guess it's harmless07:44
odyssey4mecristicalin I'd like us to follow through with it though, just in case it becomes a problem later - so a bug registered would be greatly appreciated :)07:44
*** gparaskevas_ has joined #openstack-ansible07:48
*** gparaskevas has quit IRC07:49
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Removed default lxc profile on container create  https://review.openstack.org/21701407:54
*** fawadkhaliq has joined #openstack-ansible07:54
odyssey4memattt andymccr hughsaunders please review the horde of backports and key patches in flight - only two days to go before we release 11.2.0: https://review.openstack.org/#/q/starredby:%22Jesse+Pretorius%22+project:stackforge/os-ansible-deployment,n,z07:55
odyssey4mehughsaunders please check if your query has been adequately covered in https://review.openstack.org/20793907:56
odyssey4mecloudnull it appears that https://review.openstack.org/215905 is causing some trouble in the container setup process - please look into it07:59
*** vdo has joined #openstack-ansible08:01
*** benwh4 has joined #openstack-ansible08:02
benwh4hello08:02
cristicalinodyssey4me, https://bugs.launchpad.net/openstack-ansible/+bug/1488815 done08:02
openstackLaunchpad bug 1488815 in openstack-ansible "Kernel traces with skb_warn_bad_offload showing up during an AIO deployment on Ubuntu 14.04" [Undecided,New]08:02
odyssey4meo/ benwh408:03
odyssey4methanks cristicalin :)08:03
cristicalinodyssey4me, I'm still waiting for that deployment to finish so I'm not 100% sure about that not impacted part08:04
*** fawadkhaliq has quit IRC08:04
benwh4in the bug I think you mean dist-upgrade not diet-upgrade08:04
cristicalin:) true08:05
cristicalinit was not exactly a copy & paste08:05
cristicalinbut it would be nice as a feature to apt-get, wouldnt it?08:06
benwh4a diet-upgrade ! sure it would be dope08:07
*** javeriak has joined #openstack-ansible08:07
odyssey4mehahaha08:07
odyssey4meI've added the note about the leftover veth's and a link to the clean-up script - once Apsu's online later perhaps he can put a little more time into figuring out the root cause. It may have to wait though as there is more focus on ensuring that upgrades are better right now.08:09
cristicalinso upgrade will be supported from kilo to liberty ?08:09
cristicalinI mean fully ?08:09
cristicalinhmm, I think I just borked my deployment with that fix script08:10
odyssey4mecristicalin the current work is to improve the juno to kilo upgrades as we've found that it causes some down-time for neutron routing, which isn't great08:10
cristicalinI tried to run it during the deployment ...08:10
odyssey4mebut yes, once liberty goes into code freeze we'll be finalising changes to release liberty on the same day as the upstream projects and part of the testing will be to ensure that upgrades work08:11
odyssey4mecristicalin oops - I haven't tried the script myself, so I can't vouch for it, but that wasn't perhaps the greatest of ideas :p08:12
cristicalinyeah, figured that one out the hard way08:12
cristicalinoh well , terminate and relaunch08:12
odyssey4mecristicalin you may wish to use the process outlined here instead of your method: https://github.com/stackforge/os-ansible-deployment/blob/master/development-stack.rst08:12
cristicalinok, I'll use that, might as well learn the right way to do it08:14
odyssey4mecristicalin there isn't really anything wrong with the method you used, but it's a once-off run method - whereas the method outlined there breaks it down a little more to understand things better08:15
*** fawadkhaliq has joined #openstack-ansible08:17
benwh4does this error break the deployment or is it just impacting ssh ? and why the message return 127.0.0.1 and no other IP address ?08:19
benwh4TASK: [Wait for ssh to be available] ******************************************08:19
benwh4failed: [infra1_utility_container-1aa5c074 -> 127.0.0.1] => {"elapsed": 302, "failed": true}08:19
benwh4msg: Timeout when waiting for search string OpenSSH in 10.200.239.61:2208:19
benwh4 ??08:19
odyssey4mebenwh4 I thought we'd covered this the last time you asked?08:22
matttodyssey4me: WELL COVER IT AGAIN08:22
matttsheesh08:22
mattt:P08:22
benwh4yes but I changed the value in my user_config but still the message appear08:23
odyssey4mebenwh4 can you post your updated config to pastebin please :)08:24
odyssey4melast I recall you have both a container and management network, and I suggested that you consolidate them08:24
odyssey4meI also noticed that your LB addresses weren't in the list of used addresses, so your containers may conflict with them08:25
cloudnullmornings08:25
odyssey4meyou may wish to refer to the logs near the top of http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2015-08-24.log.html :)08:25
odyssey4megparaskevas_ when you're around it'd be great if you could backport https://review.openstack.org/215019 to kilo - just cherry-pick it to kilo from the gerrit interface :)08:27
benwh4odyssey4me hter is the config file : http://pastebin.com/eaAB8UXH08:28
gparaskevas_odyssey4me: hey there! sure no problem08:28
odyssey4mebenwh4 thanks - it looks better08:29
odyssey4methe indent on line 7 is wrong - it needs to line up with the others in the same section08:29
cloudnullodyssey4me: on https://review.openstack.org/#/c/215905/ seems odd that the change would be causing issues in kilo as that change has been in master for some time now.08:29
odyssey4mecloudnull yep - it might just be a flaky gate, but kilo's gate has been pretty stable and that patch is failing every time (although in different places) - it may just warrant a test build to see if there's anything going on to be concerned about08:30
cloudnulli have that going presently08:30
benwh4odyssey4me the line 7 is a paste bug it is ok in the config file though08:31
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment: [WIP] Adds the crud_template to ceilometer  https://review.openstack.org/21703008:31
odyssey4mebenwh4 the affinity settings can also be removed - the default is 1, so you're implementing an unnecessary override08:32
*** javeriak has quit IRC08:32
cloudnullalso https://review.openstack.org/#/c/216790/ replaces the copy_update module and https://review.openstack.org/#/c/217030/ shows how it can be used to allow configurations files that are json, yaml, or in ini format to be dyanmically updated.08:32
cloudnulli did ceilometer first as it uses all three for config08:33
odyssey4mecloudnull awesome - looks good, I think it's time to restore https://review.openstack.org/168976 and update it with this method :)08:37
cloudnullyea ill try and bang on that today08:37
odyssey4mebenwh4 ok, so I don't see anything else in the openstack_user_config that is funky08:40
*** shoutm_ has quit IRC08:40
odyssey4mebenwh4 once you'd fixed up the environment, did you tear it down and restart - or have you tried to re-use the existing environment?08:42
*** javeriak has joined #openstack-ansible08:48
gparaskevas_odyssey4me: I get an error : code review error cherry pick failed09:00
odyssey4megparaskevas_ ah, there must be a dependent patch that hasn't merged into kilo09:01
gparaskevas_odyssey4me: ok then!09:02
*** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible"09:02
odyssey4megparaskevas_ hmm, are you sure - it seems to cherry-pick cleanly for me09:04
gparaskevas_odyssey4me: let me see gain09:04
odyssey4megparaskevas_ note that you must select the right branch to cherry pick into - and it must all be lower case09:05
gparaskevas_odyssey4me: i select from the latest patch(7) and press cherry pick, then on the modal window i type-select kilo and i leave the commit message as is, I can see that i has the cherry pick from id.09:06
odyssey4megparaskevas_ odd, i see that it fails via the browser09:06
odyssey4meok well, do you have an appropriate connection to be able to do it via cli?09:07
gparaskevas_yes let me try from chrome as well maybe firefox is the issue09:07
gparaskevas_then cli09:07
benwh4odyssey4me I teardown and rebuild09:10
gparaskevas_git fetch https://gparask@review.openstack.org/stackforge/os-ansible-deployment refs/changes/19/215019/7 && git cherry-pick FETCH_HEAD09:10
odyssey4megit fetch https://review.openstack.org/stackforge/os-ansible-deployment refs/changes/19/215019/7 && git cherry-pick -x FETCH_HEAD09:11
benwh4odyssey4me the heck is that the setup-hosts is ok for the target hosts but not for the container inside and the ssh error abort the playbook09:11
odyssey4meie add the '-x' just before the end09:11
gparaskevas_ok09:11
odyssey4methen git review09:11
odyssey4mebenwh4 please remind me - are you executing ansible from a workstation on another network, or from one of the hosts?09:12
gparaskevas_do i need to be on the desired repo folder? or it will clone it?09:12
*** shoutm has joined #openstack-ansible09:12
odyssey4megparaskevas_ you need to get into the repo folder, git fetch to update the refs, then git checkout origin/kilo09:12
odyssey4methen execute your cherry-pick09:13
benwh4odyssey4me I execute ansible from a sevrer in the same network as my targets 10.200.0.0 for the management09:14
benwh4odyssey4me it was my pxe server for the targets as well09:15
odyssey4mebenwh4 ok, so let's step through where you're at09:16
*** gparaskevas_ has quit IRC09:16
odyssey4meyou've pxe installed your hosts, bootstrapped ansible (which version are you using?) on your pxe host, cloned the repo, implemented /etc/openstack_deploy/{openstack_user_config.yml,user_variables.yml,user_secrets.yml} ?09:17
odyssey4meand the /etc/openstack_deploy/{conf.d,env.d} directories with all the contents of env.d from the repo clone09:18
odyssey4mebenwh4 does that sound right so far?09:19
benwh4odyssey4me I didn't touched the env.d,conv.d directory but yes it sound familiar09:25
benwh4odyssey4me I use ansible 1.9.209:27
*** misc_ has joined #openstack-ansible09:27
*** mhayden has quit IRC09:28
*** jmccrory has quit IRC09:28
*** misc has quit IRC09:28
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment-specs: Tunable OpenStack Configuration Specification  https://review.openstack.org/16897609:28
*** jmccrory has joined #openstack-ansible09:28
cloudnullodyssey4me: ^09:30
benwh4do you think I can go on even if I have this after the play recap ...09:38
benwh4log1                       : ok=46   changed=7    unreachable=0    failed=009:38
benwh4log1_rsyslog_container-c5d3f7e7 : ok=16   changed=10   unreachable=0    failed=109:39
benwh4all of my containers have encouter error during the task wait for ssh available task09:39
*** mhayden has joined #openstack-ansible09:40
*** gparaskevas has joined #openstack-ansible09:50
*** shoutm has quit IRC09:50
gparaskevasodyssey4me: hey sorry for late reply we are having connection issues at the office. i did git fetch to update my refs, i did git checkout origin/kilo and now i have detatched head(head shows kilo) should i git review?09:51
*** cristicalin has quit IRC10:02
*** shausy has quit IRC10:10
*** gparaskevas has quit IRC10:12
*** gparaskevas has joined #openstack-ansible10:15
*** benwh4 has quit IRC10:32
cloudnullthis is ithe error that is consistently happening within kilo per the recent patches http://paste.openstack.org/show/42816910:35
*** fawadkhaliq has quit IRC10:40
*** javeriak has quit IRC10:46
odyssey4mebenwh4 can you please pastebin a copy of your sshd_config from one of the hosts?10:48
odyssey4meit sounds to me like the sshd config isn't right10:48
odyssey4megparaskevas after git checkout origin/kilo, do the cherry pick, then git review10:48
*** fawadkhaliq has joined #openstack-ansible10:50
openstackgerritGeorge Paraskevas proposed stackforge/os-ansible-deployment: Enable HAProxy Stats Web UI  https://review.openstack.org/21709411:08
*** misc_ is now known as misc11:14
odyssey4megparaskevas +1 :)11:16
gparaskevasodyssey4me: thanks! :) always pleasure11:18
*** javeriak has joined #openstack-ansible11:23
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Updated juno to include fix for CVE-2015-3241 - 26 Aug 2015  https://review.openstack.org/21709811:27
*** fawadkhaliq has quit IRC11:30
cloudnullodyssey4me:  fixes openstack problems https://gist.github.com/cloudnull/936146111:32
*** fawadkhaliq has joined #openstack-ansible11:33
cloudnullif youre curious how the tool works i have a tool for that too https://pypi.python.org/pypi/AdvancedSearchDiscovery11:35
*** javeriak has quit IRC11:35
*** javeriak has joined #openstack-ansible11:35
*** javeriak has quit IRC11:36
*** javeriak has joined #openstack-ansible11:37
*** javeriak has quit IRC11:41
*** javeriak has joined #openstack-ansible11:42
odyssey4mehughsaunders cloudnull mattt andymccr please work through these reviews asap: https://review.openstack.org/#/q/starredby:%22Jesse+Pretorius%22+project:stackforge/os-ansible-deployment,n,z11:45
odyssey4mehughsaunders please check if your question has been appropriately addressed? https://review.openstack.org/20793911:45
openstackgerritHugh Saunders proposed stackforge/os-ansible-deployment: Add ebtables to neutron agent configuration  https://review.openstack.org/21710311:51
odyssey4mecloudnull hughsaunders https://bugs.launchpad.net/neutron/+bug/127403411:56
openstackLaunchpad bug 1274034 in neutron "Neutron firewall anti-spoofing does not prevent ARP poisoning" [High,Fix released] - Assigned to Kevin Benton (kevinbenton)11:56
*** benwh4 has joined #openstack-ansible12:00
evrardjpouch12:03
evrardjpthis is a bad one12:03
evrardjpI've seen we have another CVE today12:04
evrardjphttps://bugs.launchpad.net/nova/+bug/138754312:04
openstackLaunchpad bug 1387543 in OpenStack Compute (nova) "[OSSA 2015-015] Resize/delete combo allows to overload nova-compute (CVE-2015-3241)" [High,Fix committed] - Assigned to Abhishek Kekane (abhishek-kekane)12:04
odyssey4meevrardjp it's an upstream CVE for juno12:04
evrardjpodyssey4me: which one?12:05
evrardjpArp poisoning?12:05
odyssey4meah, no that one isn't a CVE - but it is s security issue across all branches12:05
odyssey4methe reviews are still in progress upstream though12:05
odyssey4meah, I see the one you linked needs a patch update for us - let me do that quickly12:06
evrardjpI didn't got the chance to do it myself, sorry12:07
*** rward has quit IRC12:08
evrardjpstill the arp poisoning is a bad one, I'll check which it hasn't been merged12:08
evrardjpwhy*12:08
evrardjpmmm a lot of work12:09
evrardjpI guess12:09
evrardjphttps://review.openstack.org/#/c/209705/12:10
evrardjpany of your customers has asked this?12:10
odyssey4meevrardjp we hit that when the changes merged to master and stuff didn't work any more - it turns out that the implementation right now is incomplete12:12
odyssey4mewe have therefore registered https://bugs.launchpad.net/neutron/+bug/148331512:12
openstackLaunchpad bug 1483315 in neutron "ebtables ARP rules don't account for floating IPs on LinuxBridge" [Undecided,In progress] - Assigned to Kevin Benton (kevinbenton)12:12
odyssey4meonce that's fixed, we can turn it on in master - for now we've disabled it in https://review.openstack.org/21059312:13
evrardjpok12:15
evrardjplinux bridge really deserves more love.12:17
evrardjpcloudnull: nice spec12:18
evrardjphttps://review.openstack.org/#/c/168976/2/specs/kilo/tunable-openstack-configuration.rst12:18
*** woodard has joined #openstack-ansible12:21
*** shoutm has joined #openstack-ansible12:23
*** javeriak_ has joined #openstack-ansible12:23
odyssey4meevrardjp linuxbridge is no voting in the gate, so it's an equal to ovs from a voting standpoint12:23
odyssey4methe issue with the arp filtering things is that they added the feature, but explicity turned off the tests for both ovs and linuxbridge12:24
odyssey4meso whoever added that, and whoever approved that, needs their heads checked :p12:24
*** javeriak has quit IRC12:25
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Updated juno to include fix for CVE-2015-3241 - 26 Aug 2015  https://review.openstack.org/21711412:35
openstackgerritJesse Pretorius proposed stackforge/os-ansible-deployment: Updated kilo to include fix for CVE-2015-3241 - 26 Aug 2015  https://review.openstack.org/21711412:35
*** pradk has joined #openstack-ansible12:37
mhaydenodyssey4me / palendae: my bouncer blew up last night and i wasn't sure where the APLv2 vs MIT licensing discussion went12:57
mhaydeni just tossed some notes into https://review.openstack.org/#/c/216849/12:58
openstackgerritJean-Philippe Evrard proposed stackforge/os-ansible-deployment: Adds the ability to provide user certificates to HAProxy  https://review.openstack.org/21552513:04
*** scarlisle has joined #openstack-ansible13:12
*** tlian has joined #openstack-ansible13:16
ApsuMorning.13:47
evrardjpgood morning Apsu13:53
*** fawadkhaliq has quit IRC13:56
*** shoutm has quit IRC13:56
*** jmckind has joined #openstack-ansible14:00
*** fawadkhaliq has joined #openstack-ansible14:02
*** shoutm has joined #openstack-ansible14:03
*** KLevenstein has joined #openstack-ansible14:04
*** sigmavirus24_awa is now known as sigmavirus2414:05
*** spotz_zzz is now known as spotz14:08
odyssey4memhaydenI saw the notes, thanks - will read them more thoroughly in a bit once this CVE is dealt with14:09
*** cloudtrainme has joined #openstack-ansible14:10
*** Mudpuppy has joined #openstack-ansible14:15
evrardjpis gerrit slow today or it's just me?14:17
cloudnullits slow today...14:18
cloudnull:(14:18
evrardjpI'll leave earlier then14:20
evrardjphave a nice day!14:20
*** shoutm has quit IRC14:22
benwh4how do I used the new patch impacting regex ssh ? does I need to re-clone the git repo ?14:24
*** cristicalin has joined #openstack-ansible14:25
cristicalinanybody have a idea how I can set horizon to use the keystone v3 api instead of the default v2 ?14:26
cristicalinI see the template for local_settings.py contains some logic to do it based on the API endpoint URL14:27
cristicalinbut the catalog looks weird the publicURL for identity is v3 but the internal and admin are v2 and horizon is using the internalURL14:27
odyssey4mecristicalin yes, it adapts based on what you enter as the keystone api14:27
odyssey4meand you'll need to make sure that you have a catalog entry to match it14:28
cristicalinand where is the catalog configured ?14:28
odyssey4mecristicalin so it all depends on what you really want to achieve14:30
odyssey4mesome warnings - keystone v3 endpoints are not yet fully supported for all service to service comms14:30
cristicalinodyssey4me, my end goal is to get Horizon to show me keystone domains and be able to manage them in Horizon14:30
odyssey4meso start with just setting https://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/roles/os_horizon/defaults/main.yml#L8614:30
cristicalinok, that could work14:31
cristicalincan I override that in user_variables.yml ?14:31
odyssey4meie in user_variables, set - horizon_keystone_endpoint: "{{ keystone_service_internalurl_v3 }}"14:32
odyssey4meor you can simply set it to the appropriate url instead of using a variable14:32
*** gparaskevas has quit IRC14:32
odyssey4mecristicalin this will then result in the template adding the right bits when it's dropped on the horizon hosts: https://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/roles/os_horizon/templates/horizon_local_settings.py.j2#L45-L5414:34
cristicalinhttps://github.com/stackforge/os-ansible-deployment/blob/master/playbooks/roles/os_horizon/templates/horizon_local_settings.py.j2#L56-L5814:35
cristicalinthis should also be changed to True14:35
cristicalinso I have to change the template to support multiple domains14:35
*** fawadkhaliq has quit IRC14:35
cristicalinwould be nice to have that into a variable14:35
odyssey4meah, so yes - can you register a bug to request the bits you need for horizon multi-domain support14:36
odyssey4mewe'll add it as a wishlist and I can prep a patch quickly, unless you'd like to prep a patch for it?14:36
cristicalinI can do the patch if you accept it14:37
cristicalinwill try to do it tomorrow14:37
odyssey4mecristicalin sure - we may need to do some to and fro initially around style, etc - but if you're comfortable doing so, we love patches :)14:37
odyssey4mewe're doing a release tomorrow, then the next will be in two weeks (unless we hit a CVE) - bear that in mind if this is essential for your requirements :)14:38
cristicalinodyssey4me, I'm just learning the tools at the moment so I'm in no hurry14:39
odyssey4mecristicalin if you can register bugs as you find them, then someone may pick them up along the way - if you find things that are missing, register bugs for those too and we'll assess them as wishlist items14:40
odyssey4mewe discuss new bugs in the tuesday meetings, then longer term plans in the thu meetings14:41
* odyssey4me points at the channel topic :)14:41
openstackgerritKevin Carter proposed stackforge/os-ansible-deployment-specs: Tunable OpenStack Configuration Specification  https://review.openstack.org/16897614:42
*** gparaskevas has joined #openstack-ansible14:42
*** gparaskevas_ has joined #openstack-ansible14:42
cristicalinok, will keep that in mind14:43
*** javeriak_ has quit IRC14:47
*** yaya has joined #openstack-ansible14:51
*** gparaskevas has quit IRC14:52
*** gparaskevas_ has quit IRC14:52
cristicalinodyssey4me, one more question if it's not too much pestering14:54
cristicalinis changing haproxy_ssl from no to yes supported ?14:54
odyssey4mecristicalin you may pester as much as you like :) if we're so busy that we can't answer, we won't :p14:55
cristicalindo I need to re-run all the playbooks or just the haproxy one and the os-keystone to update the catalog14:55
odyssey4me(or we may not be here)14:55
*** javeriak has joined #openstack-ansible14:55
odyssey4methe answer is yes :)14:55
cristicalinok, dumb question, too late in the evening for me14:55
cristicalinso which one ? run all or just the haproxy and keystones ?14:56
odyssey4mecristicalin more info in the commit message: https://github.com/stackforge/os-ansible-deployment/commit/36640a8f436fae8d0957d92f033dd4baf9e8af3f14:56
odyssey4methere is also a review in flight to add user provided certificate support: https://review.openstack.org/21552514:57
cristicalinself signed works for me atm14:58
*** sdake has quit IRC14:58
odyssey4mecristicalin so you can use haproxy to do ssl offloading for keystone, or you can do ssl on keystone itself (see https://github.com/stackforge/os-ansible-deployment/commit/4b35b3e929cbc728b903bf19d8d169e376920832 ) and you can theoretically also do ssl on both if you want14:58
*** CheKoLyN has joined #openstack-ansible14:59
odyssey4mebe warned though, haproxy is primarily a dev/test tool for the project - it hasn't been fine tuned for production use14:59
cristicalinI prefer the haproxy approach at the moment it's cleaner14:59
odyssey4meevrardjp has been submitting patches to improve it for production use and will continue to improve it over time15:00
cristicalinalso I can separate and tune it outside the main node15:00
*** javeriak has quit IRC15:01
odyssey4memost production deployments using the project are using F5's for load balancing and SSL offloading - they scale better for heavy loads15:01
*** daneyon has joined #openstack-ansible15:01
*** fawadkhaliq has joined #openstack-ansible15:02
cristicalinmine is chugging along just fine with haproxy but it's a small one <20 nodes15:03
*** yaya has quit IRC15:03
*** gparaskevas has joined #openstack-ansible15:04
cristicalinodyssey4me, is that patch backported to the kilo branch ?15:08
*** cristicalin has quit IRC15:16
*** yaya has joined #openstack-ansible15:19
xar-evrardjp: yes it is ;)15:20
xar-morning everyone15:20
*** mhayden has left #openstack-ansible15:26
odyssey4meandymccr mattt cloudnull hughsaunders sigmavirus24 please review: https://review.openstack.org/#/q/starredby:%22Jesse+Pretorius%22+project:stackforge/os-ansible-deployment,n,z15:26
*** mhayden has joined #openstack-ansible15:26
*** gparaskevas has quit IRC15:27
*** jwagner is now known as jwagner_away15:31
cloudnullanyone else getting a 503 from os infra ?15:43
*** fawadkhaliq has quit IRC15:43
-openstackstatus- NOTICE: restarting gerrit due to a slow memory leak15:43
sigmavirus24cloudnull: ^?15:44
odyssey4meo/ cloudnull15:44
cloudnullyup thats a thing15:45
odyssey4mexar- an early morning to you15:45
cloudnullxor not an early morning any longer15:45
*** benwh4 has quit IRC15:55
*** Bjoern_ has joined #openstack-ansible15:56
*** yaya has quit IRC15:56
*** Bjoern_ is now known as BjoernT15:57
*** alop has joined #openstack-ansible15:57
*** vdo has quit IRC16:02
*** k_stev has joined #openstack-ansible16:03
*** k_stev has quit IRC16:03
*** k_stev has joined #openstack-ansible16:04
*** jwagner_away is now known as jwagner16:11
*** yaya has joined #openstack-ansible16:18
*** sdake has joined #openstack-ansible16:20
*** fawadkhaliq has joined #openstack-ansible16:23
*** Mudpuppy_ has joined #openstack-ansible16:27
*** Mudpupp__ has joined #openstack-ansible16:29
*** Mudpuppy has quit IRC16:29
*** jmckind has quit IRC16:29
*** Mudpuppy has joined #openstack-ansible16:31
*** Mudpuppy_ has quit IRC16:32
*** Mudpupp__ has quit IRC16:33
*** cloudtrainme has quit IRC16:33
xar-:)16:37
*** jwagner is now known as jwagner_away16:39
*** jwagner_away is now known as jwagner16:41
openstackgerritMerged stackforge/os-ansible-deployment: Use dict args for ceph_config slurp  https://review.openstack.org/21660816:51
openstackgerritHugh Saunders proposed stackforge/os-ansible-deployment: Add variable for cirros url  https://review.openstack.org/21731017:08
*** k_stev has quit IRC17:21
*** daneyon has quit IRC17:27
*** k_stev has joined #openstack-ansible17:30
openstackgerritMerged stackforge/os-ansible-deployment: Set iptables-persistent install execution to append to log  https://review.openstack.org/21549517:39
*** cloudtrainme has joined #openstack-ansible17:56
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Add sorting_method to swift proxy config as needed  https://review.openstack.org/20881718:13
*** openstackgerrit has quit IRC18:17
*** openstackgerrit has joined #openstack-ansible18:17
*** yaya has quit IRC18:39
*** javeriak has joined #openstack-ansible18:42
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Ensure rsync restarts fully during swift setup  https://review.openstack.org/21734118:45
*** javeriak has quit IRC18:50
*** javeriak has joined #openstack-ansible18:53
openstackgerritMerged stackforge/os-ansible-deployment: Trigger restart after adding user to cephkeys  https://review.openstack.org/21632018:57
openstackgerritMerged stackforge/os-ansible-deployment: Enable admin level on the haproxy stats socket  https://review.openstack.org/21589918:58
*** jmckind has joined #openstack-ansible19:27
*** daneyon has joined #openstack-ansible19:29
*** javeriak has quit IRC19:33
*** daneyon has quit IRC19:33
*** javeriak has joined #openstack-ansible19:34
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Set default container apparmor profile to uncontained  https://review.openstack.org/21736719:46
*** daneyon has joined #openstack-ansible19:48
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Set default container apparmor profile to unconfined  https://review.openstack.org/21736719:49
*** klindgren has joined #openstack-ansible19:53
*** daneyon has quit IRC19:54
klindgrenSam-I-Am, who was it that you said had a branch to run OSAD on top of redhat/cent?19:55
*** daneyon has joined #openstack-ansible19:55
*** KLevenstein has quit IRC19:58
*** BjoernT has quit IRC19:58
Sam-I-Amklindgren: cloudnull20:04
Sam-I-Amklindgren: let me find it...20:04
Sam-I-Amklindgren: https://github.com/cloudnull/os-ansible-deployment/tree/master-rhel20:04
*** KLevenstein has joined #openstack-ansible20:06
klindgrenSam-I-Am, thanks20:08
*** daneyon has quit IRC20:09
Sam-I-Amklindgren: you didnt hear it from me :)20:10
klindgrenSam who?  I found it on github ;-)20:10
*** cloudtrainme has quit IRC20:12
*** k_stev has quit IRC20:12
*** cloudtrainme has joined #openstack-ansible20:13
*** k_stev has joined #openstack-ansible20:17
*** k_stev1 has joined #openstack-ansible20:18
Sam-I-Amklindgren: lol20:18
*** javeriak has quit IRC20:19
*** k_stev has quit IRC20:19
*** javeriak has joined #openstack-ansible20:19
*** daneyon has joined #openstack-ansible20:24
*** fawadkhaliq has quit IRC20:25
*** k_stev1 has quit IRC20:29
*** yaya has joined #openstack-ansible20:41
*** javeriak has quit IRC20:42
*** openstackgerrit has quit IRC21:01
*** k_stev has joined #openstack-ansible21:02
*** openstackgerrit has joined #openstack-ansible21:02
*** woodard has quit IRC21:11
*** daneyon_ has joined #openstack-ansible21:16
*** k_stev1 has joined #openstack-ansible21:18
*** daneyon has quit IRC21:19
*** k_stev1 has quit IRC21:19
*** k_stev1 has joined #openstack-ansible21:19
*** k_stev has quit IRC21:19
odyssey4meklindgren that work will be submitted into osad at some point soon - if you're interested in maintaining it then perhaps you could become that person21:21
*** k_stev1 has quit IRC21:35
*** d34dh0r53 has quit IRC21:38
*** d34dh0r53 has joined #openstack-ansible21:39
*** alejandrito has joined #openstack-ansible21:39
stevellemiguelgrinberg: any idea why I might be getting issues with the heat config when turning on rsync on the host in an aio?21:40
*** KLevenstein has quit IRC21:40
miguelgrinbergstevelle: can you be more specific? what issues are you seeing?21:40
stevellehttp://paste.openstack.org/show/9OR5JgmodVC3yY1kUgrt/21:40
miguelgrinbergstevelle: doesn't seem related to rsync. You have an undefined var.21:41
stevellemiguelgrinberg: I don't.  I didn't change anything related to heat in that diff21:42
*** Mudpuppy has quit IRC21:42
stevellemy builds work fine, this just misbehaving in gate.  gate has an undefined var :)21:42
stevellemy hunch was maybe heat also uses rsync or something21:42
stevellenot that this makes any sense21:43
cooljpalendae: odyssey4me does 1488315 make sense? need more info? seems to be effecting all 10.1.11+ deployments21:43
miguelgrinbergstevelle: so this var is not a regular var21:43
miguelgrinbergstevelle: it is registered internally: https://github.com/stackforge/os-ansible-deployment/blob/f665c58d38c58ee395229c566d3aab77d94e1a6e/playbooks/roles/os_heat/tasks/heat_domain_setup.yml#L11821:43
miguelgrinbergor actually, on line 126 of that file, a few lines below the one I highlighted21:44
miguelgrinbergstevelle: try running the openstack cli command manually and see what you get21:44
stevelle"openstack: 'keystone_service_adminurl_v3' is not an openstack command. See 'openstack --help'."21:47
stevellestupid substitution21:47
odyssey4mecoolj it makes sense, but the workaround is specified in the bug comments21:47
odyssey4methose commands should ideally be implemented from the utility container until a suitable fix can be found21:48
miguelgrinbergstevelle: yeah, that should be "host_ip:5000/v3"21:48
stevellemiguelgrinberg: I have another 6 flags to resolve or something like that21:49
miguelgrinbergare you doing this on the utility container?21:49
stevelleyes21:50
stevellepruned all the noise, command resolves just fine21:51
miguelgrinbergstevelle: you can just list all domains and find the one for heat21:51
stevellebut this is in my testing aio21:51
miguelgrinbergstevelle: source the openrc, then "openstack domain list"21:51
stevelleopenstack domain show heat21:51
stevelleworks fine21:51
*** alejandrito has quit IRC21:51
miguelgrinbergdo you have a domain with name "heat"?21:51
stevelleyes21:52
stevelleid bdf9e15718e84a8bb78c1679692c9a9b21:52
miguelgrinbergif you do "openstack domain show heat" and then pipe it to the grep command that we use in the playbook do you get the id captured?21:52
miguelgrinbergstevelle: this thing: grep -oE -m 1 "[0-9a-f]{32}"21:53
stevellemiguelgrinberg: I do21:53
miguelgrinbergthen I guess we have to assume that task was skipped for some reason? Do you have the ansible output to check if this task executed?21:54
stevelleurl for the review is in the paste21:55
stevellesays it skipped the task21:55
miguelgrinbergstevelle: it was skipped. I wonder why.21:56
stevellemiguelgrinberg: would seem to be because "inventory_hostname == groups['heat_all'][0]" was false21:58
miguelgrinbergstevelle: was this a first run?21:58
stevellehttps://review.openstack.org/#/c/217341/ miguelgrinberg21:59
stevelleit was the gate check there21:59
miguelgrinbergstevelle: the post_install task runs for all heat containers, but as you noticed, the domain setup task runs for the first22:00
miguelgrinbergI'm guessing it is an unrelated bug22:00
miguelgrinbergstevelle: the other heat container is bad it seems22:01
miguelgrinbergsearch for "fatal: [aio1_heat_apis_container-d847bf57]" in your log22:01
stevelleSSH Error: data could not be sent to the remote host. Make sure this host can be reached over ssh22:02
miguelgrinbergright, so that's your first heat container, the one that should have set that domain id thing22:03
stevellethanks, I got caught on the wrong error22:03
miguelgrinbergso go take a look at that container22:03
stevelledoes anything else with the review present an issue?  seems like your initial comments are addressed22:04
miguelgrinbergstevelle: my only problem with it is that the restart rsync just stops rsync22:04
stevelleyou do understand that currently that play just stops rsync?22:05
miguelgrinbergI understand what you are saying, but before this change, restart was a restart, and now it is a stop22:05
stevellebefore this change "restart" didn't work. it never triggered the start handler22:06
miguelgrinbergstevelle: yes, I do. So maybe this was broken before, I get that.22:06
miguelgrinbergBut now that we understand the problem, shouldn't we rename it to stop, if that is all it does?22:06
stevelleFrankly I think it was named wrong before, under the pretense that it worked.22:06
miguelgrinbergstevelle: well, I think it tried to do the right thing, right? It stopped, then notified the other handler that does the start22:07
stevelleIn my eyes it tried to pawn part of it's job off on another handler. I can change the name though.22:10
miguelgrinbergstevelle: we have many other restart handlers that do actually a restart22:13
stevelleagreed22:13
miguelgrinbergif we can't do a clean restart with rsync, then to make it clear that we have this limitation the handlers should be called stop and start, or something like that22:13
openstackgerritIan Cordasco proposed stackforge/os-ansible-deployment: Remove temporary upgrade task that removes profile  https://review.openstack.org/21736722:13
stevellemiguelgrinberg: I'll rename it in another revision, just digging a little further into this gate failure I see there are no logs for that container, but the container was created and given a backing store etc.22:15
*** shoutm has joined #openstack-ansible22:15
*** sdake_ has joined #openstack-ansible22:15
stevellethe missing container logs is a bit troubling22:15
miguelgrinbergstevelle: before you invest a lot of time, I would do a recheck to see if it is repeatable22:17
miguelgrinbergstevelle: and regarding the handlers, the other thing to double check is that ansible invokes multiple handlers one after the other, and not in parallel. That could also break your rsync restarts.22:18
*** sdake has quit IRC22:18
stevellemiguelgrinberg: I did test it, n=9, to ensure consistent execution order and service always up at the end22:19
*** jmckind has quit IRC22:19
miguelgrinbergstevelle: ok, sounds good22:23
*** cloudtrainme has quit IRC22:31
*** shoutm has quit IRC22:32
openstackgerritSteve Lewis proposed stackforge/os-ansible-deployment: Ensure rsync restarts fully during swift setup  https://review.openstack.org/21734122:32
stevellethat is another way to recheck ^22:32
*** spotz is now known as spotz_zzz22:33
*** shoutm has joined #openstack-ansible22:36
*** scarlisle has quit IRC22:37
cooljodyssey4me: sorry i had to step away for a meeting. the issue is effecting nova-compute on all the hosts--when it tries to get image info from glance it gets back an empty Image object and raises an AttributeError22:38
*** sdake_ is now known as sdake22:41
*** yaya has quit IRC22:42
*** markvoelker has quit IRC22:46
*** jwagner is now known as jwagner_away22:53
*** markvoelker has joined #openstack-ansible22:54
cooljanother new 10.1.11 deployment hit23:07
cooljhttps://bugs.launchpad.net/openstack-ansible/+bug/148831523:07
openstackLaunchpad bug 1488315 in openstack-ansible trunk "The python-requests package is pulled in by apt via dependency" [Medium,Confirmed]23:07
miguelgrinbergstevelle: regarding order of handler execution. Looks like ansible executes handlers not in the order they are given in the notify line, they run in the order in which they are defined in the handlers file. Something to keep in mind.23:14
stevellemiguelgrinberg: good to know23:15
openstackgerritMerged stackforge/os-ansible-deployment: Enable tempest testing of ceilometer  https://review.openstack.org/20956823:16
openstackgerritMerged stackforge/os-ansible-deployment: Add configurable ssh_delay  https://review.openstack.org/21642923:16
openstackgerritMerged stackforge/os-ansible-deployment: Remove read only disks from lvm candidates  https://review.openstack.org/21632723:16
openstackgerritMerged stackforge/os-ansible-deployment: Updated kilo to include fix for CVE-2015-3241 - 26 Aug 2015  https://review.openstack.org/21711423:16
*** CheKoLyN has quit IRC23:23
*** alop has quit IRC23:54
*** shoutm_ has joined #openstack-ansible23:54
*** klindgren has quit IRC23:57
*** shoutm has quit IRC23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!