Tuesday, 2015-11-24

*** sacharya has quit IRC00:00
*** gouthamr has quit IRC00:09
*** galstrom is now known as galstrom_zzz00:14
*** leifmadsen has joined #openstack-ansible00:15
*** cemmason has joined #openstack-ansible00:16
*** openstackgerrit has quit IRC00:16
*** openstackgerrit has joined #openstack-ansible00:17
*** sacharya has joined #openstack-ansible00:21
*** mss_ has quit IRC00:22
*** sacharya has quit IRC00:23
*** niknakpaddywak has joined #openstack-ansible00:24
*** cemmason has quit IRC00:33
*** rebase has quit IRC00:39
*** sdake_ has joined #openstack-ansible00:45
*** sdake has quit IRC00:45
openstackgerritMatthew Oliver proposed openstack/openstack-ansible: Add statsd configuration for swift  https://review.openstack.org/24618200:57
*** cemmason has joined #openstack-ansible01:01
*** elo has joined #openstack-ansible01:03
*** eil397 has quit IRC01:06
*** sdake_ is now known as sdake01:12
*** galstrom_zzz is now known as galstrom01:15
*** jhesketh has joined #openstack-ansible01:15
*** rebase has joined #openstack-ansible01:15
*** jhesketh has quit IRC01:16
*** skamithi has joined #openstack-ansible01:16
*** skamithi has quit IRC01:17
*** skamithi14 has joined #openstack-ansible01:21
*** skamithi13 has quit IRC01:24
*** galstrom is now known as galstrom_zzz01:26
*** cemmason has quit IRC01:42
*** baker has joined #openstack-ansible01:43
*** baker has quit IRC01:44
cloudnullevenings01:45
*** baker has joined #openstack-ansible01:46
*** baker has quit IRC01:48
*** mss has joined #openstack-ansible01:52
*** jhesketh has joined #openstack-ansible01:58
*** jhesketh has quit IRC02:03
*** Grey_A has joined #openstack-ansible02:03
*** niknakpaddywak has quit IRC02:05
*** jhesketh has joined #openstack-ansible02:06
*** skamithi14 has quit IRC02:11
*** skamithi13 has joined #openstack-ansible02:11
*** Grey_A has left #openstack-ansible02:17
*** niknakpaddywak has joined #openstack-ansible02:19
bgmccollumis this thing on?02:25
cloudnullnope02:26
*** gouthamr has joined #openstack-ansible02:34
*** neillc_away is now known as neillc02:36
*** sacharya has joined #openstack-ansible02:53
*** eil397 has joined #openstack-ansible02:56
*** rebase has quit IRC02:56
*** jthorne has joined #openstack-ansible02:57
bgmccollumhah02:57
bgmccollumworking on clouds.yml config for openstack client02:58
*** eil397 has quit IRC02:59
*** eil397 has joined #openstack-ansible02:59
openstackgerritByron McCollum proposed openstack/openstack-ansible: Implement support for OpenStack client clouds.yml configuration file  https://review.openstack.org/24898603:14
openstackgerritByron McCollum proposed openstack/openstack-ansible: Implement OpenStack client clouds.yml configuration file  https://review.openstack.org/24898603:15
*** eil397 has quit IRC03:18
*** pcaruana has quit IRC03:27
*** skamithi13 has quit IRC03:48
*** alkari has joined #openstack-ansible04:09
*** alkari has quit IRC04:13
openstackgerritMerged openstack/openstack-ansible: IRR - Implemented for setup-hosts  https://review.openstack.org/24639204:13
*** sacharya has quit IRC04:18
openstackgerritNeill Cox proposed openstack/openstack-ansible: Stop configure_diskspace from using small partitions  https://review.openstack.org/21688004:23
cloudnullYAY the first IRR patch landed :)04:32
cloudnullnow i need https://review.openstack.org/#/c/241483/ to continue down the IRR path04:33
*** sacharya has joined #openstack-ansible04:40
*** alkari has joined #openstack-ansible04:45
openstackgerritKevin Carter proposed openstack/openstack-ansible: Functional backport for the new repo-build process  https://review.openstack.org/24421504:48
*** jmccrory has quit IRC04:58
*** jmccrory has joined #openstack-ansible05:00
*** sdake_ has joined #openstack-ansible05:09
*** gouthamr has left #openstack-ansible05:10
*** sdake has quit IRC05:10
*** ashishb has joined #openstack-ansible05:14
*** sdake_ has quit IRC05:26
*** shausy has joined #openstack-ansible05:41
*** javeriak has joined #openstack-ansible05:43
*** pcaruana has joined #openstack-ansible05:49
*** mss has quit IRC05:54
*** javeriak has quit IRC05:57
*** mss has joined #openstack-ansible06:09
*** gouthamr has joined #openstack-ansible06:10
openstackgerritNeill Cox proposed openstack/openstack-ansible: Stop configure_diskspace from using small partitions  https://review.openstack.org/21688006:13
*** javeriak has joined #openstack-ansible06:21
*** javeriak has quit IRC06:23
*** javeriak has joined #openstack-ansible06:23
*** alkari has quit IRC06:25
*** pcaruana has quit IRC06:28
*** javeriak has quit IRC06:29
*** javeriak_ has joined #openstack-ansible06:29
*** javeriak has joined #openstack-ansible06:32
*** javeriak_ has quit IRC06:34
*** phiche has joined #openstack-ansible06:43
*** phiche has quit IRC06:46
*** phiche has joined #openstack-ansible06:51
*** phiche has quit IRC06:56
*** henrymy has joined #openstack-ansible07:01
*** henrymy has quit IRC07:02
*** phiche has joined #openstack-ansible07:02
*** phiche has quit IRC07:04
*** phiche has joined #openstack-ansible07:05
*** phiche has quit IRC07:06
*** phiche has joined #openstack-ansible07:17
*** sacharya has quit IRC07:27
*** mss has quit IRC07:35
evrardjphello everyone07:54
evrardjpbgmccollum: you're there?07:54
*** javeriak has quit IRC07:55
*** sacharya has joined #openstack-ansible07:56
*** sacharya has quit IRC08:03
*** mpavone has joined #openstack-ansible08:16
*** mss has joined #openstack-ansible08:31
*** subscope has joined #openstack-ansible08:36
*** sdake has joined #openstack-ansible08:45
*** sdake has quit IRC08:48
*** subscope has quit IRC08:48
*** cemmason has joined #openstack-ansible08:48
*** sdake has joined #openstack-ansible08:48
*** subscope has joined #openstack-ansible08:52
*** tiagogomes has joined #openstack-ansible08:53
*** sdake has quit IRC09:01
*** subscope has quit IRC09:09
*** subscope has joined #openstack-ansible09:10
*** mattoliverau has quit IRC09:10
*** matt6434 has joined #openstack-ansible09:11
*** javeriak has joined #openstack-ansible09:16
*** openstackgerrit has quit IRC09:16
*** openstackgerrit has joined #openstack-ansible09:17
*** slotti has joined #openstack-ansible09:19
*** javeriak has quit IRC09:27
*** javeriak has joined #openstack-ansible09:28
*** javeriak_ has joined #openstack-ansible09:28
*** javeriak has quit IRC09:32
*** ashishb has quit IRC09:35
*** subscope has quit IRC09:37
*** ashishb has joined #openstack-ansible09:41
*** adac has joined #openstack-ansible10:02
adacDid someone of you manage to install AIO build?10:03
adacwith this script: curl https://raw.githubusercontent.com/openstack/openstack-ansible/master/scripts/run-aio-build.sh | sudo bash10:03
adacI'm running into trouble with the connection to mongodb10:04
adacrestarting the script then results in an error that the user on mongodb already exists10:04
*** sacharya has joined #openstack-ansible10:08
adacok I can clean the messs up with the teardown.sh script and then restart it again10:09
adachowever still I have trouble to connect to mongodb it seems10:09
*** gparaskevas has joined #openstack-ansible10:12
*** sacharya has quit IRC10:12
*** javeriak_ has quit IRC10:25
*** javeriak has joined #openstack-ansible10:26
*** prometheanfire has quit IRC10:27
*** prometheanfire has joined #openstack-ansible10:29
*** subscope has joined #openstack-ansible10:30
*** javeriak_ has joined #openstack-ansible10:34
*** javeriak has quit IRC10:37
*** subscope has quit IRC10:41
odyssey4meadac we use that process for gate testing, so yes it works10:43
odyssey4meadac I would recommend doing it step by step instead as per http://docs.openstack.org/developer/openstack-ansible/developer-docs/quickstart-aio.html#running-a-customized-aio-build10:44
odyssey4meif you disable the ceilometer deployment then you will bypass mongo issues10:44
adacodyssey4me, this is probably a good idea, since I have no plan what "ceilometer" does ;-)10:47
odyssey4meadac I hope you're starting with a minimal Ubuntu install base - otherwise you may hit other trouble10:47
adacodyssey4me, I try the teardown script, that worked just fine the first time at least10:48
adacnow it is running since ages the scond time10:48
adacI think I have to reinstall ubuntu then...10:48
*** javeriak_ has quit IRC10:53
*** matt6434 has quit IRC10:54
*** javeriak has joined #openstack-ansible10:54
*** matt6434 has joined #openstack-ansible10:55
*** javeriak_ has joined #openstack-ansible10:57
kmARCodyssey4me: No matching distribution found for pycrypto10:59
kmARCI start feeling I'm running into every possible bugs :)10:59
odyssey4mekmARC hmm, that's odd - at what stage are you seeing this?11:01
*** javeriak has quit IRC11:02
odyssey4mehughsaunders this is the review I mentioned earlier: https://review.openstack.org/24148311:02
hughsaundersodyssey4me: cool, will have a look when I'm done testing mancdaz's mass pr11:03
kmARCsetup-infrastructure.yml, galera_container11:03
kmARCbrb11:04
odyssey4mekmARC alright, did the repo-build run successfully?11:04
*** phiche has quit IRC11:04
odyssey4mehughsaunders thanks11:04
odyssey4meadac yeah the teardown script is ok, but not foolproof11:05
*** javeriak_ has quit IRC11:10
evrardjphello everyone11:15
*** tiagogomes has quit IRC11:18
*** tiagogomes_ has joined #openstack-ansible11:18
*** toddnni has quit IRC11:19
*** toddnni has joined #openstack-ansible11:22
odyssey4meo/ evrardjp11:25
odyssey4meour gate check updates in -infra have completed but the puppet jobs haven't quite finished for some reason... busy following that up11:25
evrardjpI had nice scripts to apply all these commits to all the repos, now I realize that I destroyed the irr11:26
evrardjpcause I thought it was going to merge directly...11:26
evrardjpthe 2 line change on the .gitignore will cause me pain :p11:26
odyssey4meevrardjp yeah, we should have fixed that a while ago - sorry11:26
evrardjpthe problem is that .gitreview is not there, so when you redownload, you can't git review stuff done :p11:27
*** phiche has joined #openstack-ansible11:34
odyssey4meevrardjp if you git review -d for each review, then you'll be able to edit and amend the commit11:34
evrardjptrue, if there is .gitreview to have git review work :)11:35
evrardjpelse the git review -s won't work :p11:35
odyssey4meevrardjp ah yes - I just added a .gitreview before doing git review -s11:36
evrardjpthat's what in my commit :p11:36
evrardjphave the git dotfiles :)11:37
evrardjpbut you're making me doubt11:37
kmARCodyssey4me: yes, the repo containers look fine to me11:37
evrardjpyeah there is a .gitreview in the trees11:37
odyssey4mekmARC ok, but the pip wheel it's trying to install should be in the repo containers11:38
kmARCmakes sense11:39
odyssey4mekmARC go to /opt/openstack-ansible/playbooks11:39
odyssey4methen execute: ansible repo_all -m shell -a 'ls -al /var/www/repo/os-releases/'11:39
odyssey4methere should be one directory there - likely 12.0.1 ?11:39
kmARCyes11:40
odyssey4mekmARC ok, so execute: ansible repo_all -m shell -a 'ls -al /var/www/repo/os-releases/12.0.1/pycrypto*'11:40
kmARCwow, lot of red stuff :)11:41
kmARC/bin/sh: 2: /var/www/repo/os-releases/12.0.1/pycrypto-2.6.1-cp27-none-linux_x86_64.whl: Permission denie11:41
odyssey4mekmARC oh? can you pastebin the output somewhere?11:41
odyssey4mehmm, that's odd11:42
kmARCok, sorry it was a line break problem11:42
kmARCnow pasted again, looks fine11:42
odyssey4mekmARC is there more than one container?11:42
kmARC311:43
odyssey4meok, did all containers give a good response?11:43
kmARCyes11:43
odyssey4mealright, now let's try something else11:44
odyssey4mecan you pastebin the results of this somewhere? ansible galera_all -m shell -a 'cat ~/.pip/pip.conf'11:44
kmARCOK here are the problems http://paste.openstack.org/show/IDFAXZMTvjoWq9pdUe70/11:45
evrardjpin advance: sorry for the spam, and I hope this script works.11:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-apt_package_pinning: New git dotfiles for independant repository  https://review.openstack.org/24863211:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-lxc_container_create: New git dotfiles for independant repository  https://review.openstack.org/24863411:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-lxc_hosts: New git dotfiles for independant repository  https://review.openstack.org/24863511:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-openstack_hosts: New git dotfiles for independant repository  https://review.openstack.org/24863611:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-pip_install: New git dotfiles for independant repository  https://review.openstack.org/24863711:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-py_from_git: New git dotfiles for independant repository  https://review.openstack.org/24863811:47
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-rsyslog_client: New git dotfiles for independant repository  https://review.openstack.org/24863911:47
kmARCodyssey4me: should I destroy the galera_all containers and redeploy them?11:48
odyssey4mekmARC ok - that's probably due to a failure in executing the playbooks on the first container... try this now: ansible galera_all -m shell -a 'curl http://192.168.0.1:8181/os-releases/12.0.1/ | grep pycrypto'11:48
kmARChm11:48
kmARCneither of the containers can connect at all11:48
odyssey4mekmARC none of them?11:49
kmARCoh wait11:49
kmARCthis is a wrong IP11:49
odyssey4meah11:49
kmARCor not?11:49
odyssey4mewell, that depends on the config you set out11:49
kmARCthis is the IP of the first infra box11:49
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-lxc_hosts: Removed undesired .DS_Store file  https://review.openstack.org/24864311:49
kmARC(and the LB)11:50
odyssey4mekmARC yes, it should be the lb11:50
kmARClooks like I cannot telnet either11:50
odyssey4methe lb address you set out as 'internal_lb_vip_address' in /etc/openstack_deploy/openstack_user_config.yml11:50
odyssey4meright, so on your lb host confirm whether haproxy is running?11:51
odyssey4meie: service haproxy status11:51
kmARCit's not. Strange11:51
odyssey4meaha, try and restart it11:51
odyssey4meif it has a config error, it should tell you which one is a problem11:52
kmARCit's not even installed11:52
kmARChow is that possible11:52
odyssey4meah11:53
odyssey4methat is normal if you didn't run it manually11:53
odyssey4methe sequence for running the playbooks would be:11:53
odyssey4me1 - setup-hosts11:53
odyssey4me2 - haproxy-install11:53
odyssey4me3 - setup-infrastructure11:53
odyssey4me4 - setup-openstack11:53
kmARCyepp, this is the sequence I usually do11:54
odyssey4mehaproxy is not included as a default in the meta playbooks because some deployers use an F5 load balancer instead11:54
kmARCwhat can be is that in the multiple try-teardown cycles I missed this one11:54
odyssey4meyeah, entirely possible11:54
evrardjppabelanger: are you there?11:54
odyssey4merun the haproxy-install playbook and verify that it actually installed it11:55
kmARCodyssey4me: thanks, hope that this was the problem. Let me check back later now I need to run11:55
odyssey4meevrardjp unlikely - pabelanger will only show up in our afternoon, normally11:55
odyssey4mehmm, the functional jobs are still showing as 'NOT_REGISTERED'11:56
odyssey4meinfra is looking into that11:57
*** sdake has joined #openstack-ansible11:58
*** subscope has joined #openstack-ansible12:00
*** sdake has quit IRC12:03
evrardjpthat was my question :)12:05
*** mpavone has quit IRC12:07
*** javeriak has joined #openstack-ansible12:09
*** sdake has joined #openstack-ansible12:09
adacodyssey4me, what is missing in my eyes in the start script for the AIO is a check if an openssh-server is installed12:12
odyssey4meadac well, I guess that could be a bad assumption... but we are assuming that you have an sshd installed, otherwise you wouldn't be able to get to your server ;)12:13
adacodyssey4me, I'm testing it on my laptop currently :-)12:13
adacdirectly!12:13
*** subscope has quit IRC12:13
*** subscope has joined #openstack-ansible12:14
odyssey4meadac bad idea12:15
*** sacharya has joined #openstack-ansible12:15
odyssey4meyou need to make use of something that you can trash and rebuild - best to use virtualbox or some other vm mechanism12:16
odyssey4meotherwise you'll end up with a pretty dirty machine12:16
evrardjpmaybe his laptop is trash/rebuild? ;)12:16
odyssey4melol12:16
adacodyssey4me, yeah I see. But actually It is an old notbook which I'm not using anymore ;)12:16
evrardjpahah! touché!12:17
adacactually it is trash :D12:17
adachaha12:17
*** ChanServ sets mode: +o odyssey4me12:17
evrardjpno issues with hardware requirements?12:17
evrardjplaptops are not usually powerhouses12:17
adacevrardjp, well I just wanted to test the installation part12:18
adacnot really run it there12:18
adacbut yeah the memory is to weak in any case12:18
*** ChanServ sets mode: +o odyssey4me12:18
adacbut I run into trouble with mogodb as I told odyssey4me before12:18
evrardjpyup it could cause some lxc containers to crash (that happened to me)12:18
adacevrardjp, ok that fully explains it12:19
adacevrardjp, odyssey4me how much disk space do I need for testing the AIO?12:19
evrardjpif your system is OOM, he eventually needs to get ram12:19
adac80 it seems is that correct?12:19
evrardjpdepends on your vg12:19
evrardjp80 is a bare minimum12:20
*** mpavone has joined #openstack-ansible12:20
adacthat also makes it difficult to test it via virtualbox if it is really 8012:20
*** sacharya has quit IRC12:20
evrardjpmaybe we should work on having CoW of lxc systems with btrfs/overlayfs12:21
adacevrardjp, is 16GB Ram rally neede or can i also go with les?12:21
adac*less12:21
adacmean just for testing the installation12:21
evrardjpon an AIO?12:21
adacexactly12:21
evrardjpthat's tough question, maybe ask odyssey4me, he knows more in AIO than me :p12:21
evrardjp4GB is definitely not enough12:22
*** ChanServ sets mode: +o hughsaunders12:22
evrardjp8GB... I don't know12:22
odyssey4meadac it's covered in http://docs.openstack.org/developer/openstack-ansible/developer-docs/quickstart-aio.html#quick-start12:22
odyssey4mealthough you can do it with 8GB RAM, but it mmight behave a little poorly12:23
adacodyssey4me, yeah it says 16 but I do not have that many ond wondering if the installation will crash when I have about 8 only12:23
adacbut I see it should be fine but slow :)12:23
evrardjpmaybe reduce your requirements, by removing swift and ceilometer?12:23
odyssey4meadac as long as you're not using an lvm vg you'll get away with around 65GB of free space on the disk12:23
adacevrardjp, yes odyssey4me suggested me that. Should try it out12:24
odyssey4meyeah, if you cut down requirements it'll help12:24
adacodyssey4me, ok thanks12:24
*** ChanServ sets mode: +o odyssey4me12:24
evrardjpyou need to cut in either scope, budget or time, that's the triforce!12:24
odyssey4meadac this patch will help with that: https://review.openstack.org/24888712:24
odyssey4meadac I'm doing some work at the moment to implement more focused use-cases which will require less too... but I don't have patches up for review yet12:25
adacodyssey4me, evrardjp  The admin just messaged me. He has 8 gigs of ram for me in a present box :-) So I think I will give virtualbox a try again now. with 8GB of ram and about 80 gigs of space :)12:27
evrardjphaha :D12:28
adacguys I'm back soon! With new horse powers and surely some more questions! :D12:29
adacthank you!12:29
*** adac has quit IRC12:30
*** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible"12:31
*** ChanServ changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://goo.gl/tTmdgs"12:32
*** alejandrito has joined #openstack-ansible12:34
*** ashishb has quit IRC12:38
odyssey4meevrardjp fyi, I think I know what the issue is with the gate jobs - but I need to chat to pabelanger to figure out a suitable solution12:39
evrardjpgood12:39
evrardjpI'm thinking to the best way to approach policy files distribution during that time, don't worry, I have other stuff to do :p12:39
*** javeriak has quit IRC12:41
*** subscope has quit IRC12:44
*** subscope has joined #openstack-ansible12:46
*** javeriak has joined #openstack-ansible12:49
*** mpavone has quit IRC13:00
*** gouthamr has left #openstack-ansible13:00
*** jhesketh has quit IRC13:06
*** jhesketh has joined #openstack-ansible13:07
*** mpavone has joined #openstack-ansible13:12
*** javeriak has quit IRC13:13
*** javeriak has joined #openstack-ansible13:14
*** dolpher has quit IRC13:16
*** dolpher has joined #openstack-ansible13:17
*** adac has joined #openstack-ansible13:21
adacarrrg. He has only time tomorrow to upgrade my RAM memory13:22
*** shausy has quit IRC13:23
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: AIO bootstrap in Ansible  https://review.openstack.org/23952513:25
bgmccollumodyssey4me: is there a way move that review to target master? or just abandon and open a new review?13:28
odyssey4mebgmccollum cherry-pick it to master, I guess :)13:29
bgmccollumok, ill push the fixes, then cheery pick to master13:29
*** josh6627 has joined #openstack-ansible13:29
*** josh6627 has quit IRC13:29
*** jhesketh has quit IRC13:30
openstackgerritByron McCollum proposed openstack/openstack-ansible: Implement OpenStack client clouds.yml configuration file  https://review.openstack.org/24920913:33
*** markvoelker has quit IRC13:35
bgmccollumodyssey4me: went the opposite route. abandoned liberty review, opened new review targeting master, and manually cherry-picked + fixes. will make things cleaner in the future.13:40
*** markvoelker has joined #openstack-ansible13:53
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-specs: Added spec for policy-files-distribution  https://review.openstack.org/24922713:59
*** subscope has quit IRC14:00
*** woodard has joined #openstack-ansible14:01
*** woodard has quit IRC14:01
*** woodard has joined #openstack-ansible14:02
*** jhesketh has joined #openstack-ansible14:02
*** karimb has joined #openstack-ansible14:03
*** jhesketh has quit IRC14:03
*** jhesketh has joined #openstack-ansible14:04
*** subscope has joined #openstack-ansible14:04
mhaydenodyssey4me: thanks for the aio heads up14:09
mhaydeni'll hopefully adjust that later today14:09
mhaydenany more thoughts on the switch to paramiko for the gate14:09
*** subscope has quit IRC14:10
*** javeriak has quit IRC14:10
*** subscope has joined #openstack-ansible14:11
*** subscope has quit IRC14:13
*** subscope has joined #openstack-ansible14:13
*** subscope has quit IRC14:16
*** subscope has joined #openstack-ansible14:17
*** sacharya has joined #openstack-ansible14:45
*** greg_a has joined #openstack-ansible14:47
*** sacharya has quit IRC14:49
*** mgoddard_ has joined #openstack-ansible14:59
lbragstadcloudnull miguelgrinberg fyi - https://review.openstack.org/#/c/244853/15:02
*** mgoddard has quit IRC15:03
cloudnullmorning15:05
*** spotz_zzz is now known as spotz15:05
odyssey4memhayden so there are two approaches - disable ssh pipelining https://review.openstack.org/248739 or switch to paramico https://review.openstack.org/24836115:07
odyssey4meboth do seem to work15:07
cloudnullany cores about that can do a review on https://review.openstack.org/#/c/241483/ -- i need that to continue the irr work15:09
odyssey4mecloudnull hughsaunders will try to review that today after some other work he's busy with15:10
odyssey4memaybe mattt could too - not sure?15:10
cloudnullkk.15:10
cloudnullwhats the state of the irr gate? did we disable the functional tests?15:11
matttcloudnull: i can have a look15:11
*** jwagner_away is now known as jwagner15:13
openstackgerritKevin Carter proposed openstack/openstack-ansible: Update Kilo SHA's - 20.11.2015  https://review.openstack.org/24596815:14
cloudnull^ re-updated the kilo shas to pull in the backport lbragstad worked on for fernet15:15
odyssey4mecloudnull no, the functional tests are broken - busy working with infra to figure out why15:15
odyssey4mecloudnull does that kilo update need an updated date?15:16
openstackgerritKevin Carter proposed openstack/openstack-ansible: Update Kilo SHA's - 24.11.2015  https://review.openstack.org/24596815:17
cloudnullsure15:17
odyssey4mecloudnull  and perhaps updates to playbooks/defaults/repo_packages/openstack_other.yml too?15:17
cloudnullodyssey4me: nothing to update15:17
*** markvoelker has quit IRC15:18
cloudnulli I could move the date forward, but its all the same bits at this point15:18
*** markvoelker has joined #openstack-ansible15:18
cloudnullodyssey4me: are the functional tests busted because of something in the role or is it an infra thing?15:19
spotzLet me know if there's anything small to do or reviews needed, if today is like yesterday should have some free time15:19
odyssey4mespotz we always have reviews to do :) see the channel topic for a dashboard15:21
odyssey4mecloudnull it's an issue somewhere between infra's puppet jobs failing a bit, and likely also some additional job configuration required - pabelanger is on it :)15:22
spotzOkie, I'll hit that up. Its help you guys or go look for soemthing over in barbican:)15:22
cloudnullodyssey4me: sweet15:22
cloudnullpabelanger: if i  can help in any way please let me know15:22
cloudnullspotz: so you have to choose the lesser of evils i see15:23
cloudnull:p15:23
pabelangerodyssey4me: cloudnull: Ya, working in -infra15:23
pabelangerbut I don't think the issue was the patchset15:23
pabelangerJJB just failing for another reason15:23
odyssey4mespotz well, a new barbican role for OSA could be nice to see :)15:23
spotzheheh well if there's nothing there I just make my own bug and do  localization:)15:24
*** subscope has quit IRC15:26
cloudnull++ barbican in OSA would be cool for sure15:35
*** subscope has joined #openstack-ansible15:36
*** kmARC_2 has joined #openstack-ansible15:36
*** kmARC has quit IRC15:37
mhaydenodyssey4me: i'm fine with either so long as it makes ssh more reliable in the gate :P15:38
*** kmARC has joined #openstack-ansible15:38
odyssey4memhayden , cloudnull , evrardjp any thoughts on disabling ssh pipelining versus switching to paramico ?15:38
openstackgerritMajor Hayden proposed openstack/openstack-ansible: Use Ansible AIO playbook for run-aio-build.sh  https://review.openstack.org/24470815:38
odyssey4messh pipeline disable: https://review.openstack.org/24873915:38
evrardjpI had more chance in the past with paramiko15:38
openstackgerritMajor Hayden proposed openstack/openstack-ansible: Updating AIO docs for Ansible playbook  https://review.openstack.org/24472015:38
odyssey4meswitch to paramico: https://review.openstack.org/24836115:38
evrardjpdisabling pipelining is slow, paramiko too15:39
evrardjpI found paramiko better overall, but's maybe just an impression15:40
mhaydeni've used paramiko for other projects and it's always been nice15:40
*** subscope has quit IRC15:40
mhaydenand i believe it respects ~/.ssh/config as well as keys15:40
mhayden(as the ssh client does)15:40
odyssey4mebgmccollum any thoughts from you, considering that you did some digging into ssh issues before?15:41
mhaydenwe could try flipping pipelining off since it's a smaller change15:41
mhaydenbut i think controlmaster might be causing problems, too15:41
mhaydeni could be wrong15:41
mhaydenansible does some crazy stuff around sending data over ssh15:41
evrardjpmhayden: IIRC, controlmaster and pipelining are connected together15:42
evrardjpI don't remember15:42
mhaydenoh, okay15:42
mhaydenperhaps we shouldn't specify ssh command options at all and turn pipelining off15:42
evrardjpI even tried to specifically define controlmaster path etc...15:43
cloudnullmhayden:  on your ML post the code you linked back to was for ansible 2.0, have you had a chance to look into the 1.9 bits to see if we can fix the issue?15:43
odyssey4mewell, something recent changed which is causing this - I suspect it's something in base ubuntu itself... perhaps a default sshd config change, or a change in sshd or ssh itself?15:43
mhaydencloudnull: not yet -- didn't 2.0 go stable ?15:43
mhaydeni haven't had a chance to look at it yet15:44
cloudnullno15:44
evrardjpnot stable yet15:44
cloudnullits in rc1 i think15:44
cloudnullhttps://github.com/ansible/ansible/blob/v1.9.4-1/lib/ansible/runner/connection_plugins/ssh.py15:44
mhaydenoh, nvm15:44
cloudnull^ thats the connection plugin we've been using15:44
mhaydenah, it's in _communicate() in 1.915:44
mhaydenso it looks like it can't write data to stdin when that happens15:45
mhaydenwhich is still really similar to what i saw in the 2.0 code15:45
odyssey4meanother thing to consider is that it seems that the -nv job succeeds more often than the dsvm job... so perhaps it's a base set of packages or some sort of config difference...15:45
odyssey4mewe could switch wholesale to the base image used in the -nv job15:45
mhaydenweird15:46
odyssey4mebut that doesn't work for juno, so I'd have to setup different jobs based on branch15:46
mhaydenodyssey4me: is -nv just plain vanilla ubuntu 14.04?15:46
evrardjpfor pipelining, IIRC you could disable it only for the run by setting an env variable15:46
mhaydenevrardjp: i thought you put pipelining=False in ansible.cfg?15:46
evrardjpyou could15:46
evrardjpI think you can also have ANSIBLE_SSH_PIPELINING=015:47
odyssey4memhayden -nv is a minimal image built by disk-image-builder and deployed across all cloud providers for infra... the dsvm job is d-i-b based in hpcloud and snapshot based in rax cloud15:47
evrardjpto avoid editing config files15:47
mhaydeninteresting, odyssey4me15:47
evrardjpif it helps15:47
mhaydeni like the nv idea then15:47
mhayden:)15:47
mhaydenalso nv has almost zero ssh issues15:47
odyssey4meevrardjp so my concern here is that this issue will hit production environments15:47
odyssey4methe gate is our canary15:47
mhaydenthe ansible aio bootstrap would always make it much further in the nv15:47
*** javeriak has joined #openstack-ansible15:48
mhaydengood point, odyssey4me15:49
mhaydeni never get those ssh erorrs when i do a build on a plain ubuntu 14.04 rax image15:49
mhaydenokay, i'm going to tear myself away from the computer for a bit15:50
mhayden:)15:50
evrardjpodyssey4me: that's why I mentionned it's possible to use ENV variables15:50
evrardjpstick to safe defaults for everyone15:50
evrardjpand tweak for gate checks15:51
odyssey4meok, I'm going to work with pabelanger to switch the integration tests to use the 'ubuntu-trusty' image, as we have in https://github.com/openstack-infra/project-config/blob/master/zuul/layout.yaml#L677715:52
evrardjpdon't forget that ppl can also tweak their ansible.cfg for production (while not recommended, because it's in repo code)15:52
odyssey4meI suspect that the switch over, and some cruft removal, will also resolve issues for the role functional tests15:52
evrardjp(but this could also move because we are using a bootstrap ansible that could copy this file...)15:53
evrardjpok15:53
evrardjpnice15:54
pabelangernice15:54
pabelangerodyssey4me: what does that job do today?15:54
odyssey4mepabelanger is there a reason to prefer using the devstack-ubuntu image over using the ubuntu-trusty image for role functional tests?15:55
*** phiche has quit IRC15:55
odyssey4mepabelanger gate-openstack-ansible-commit-nv and gate-openstack-ansible-dsvm-commit are exactly the same in execution, but use different images - the nv job uses ubuntu-trusty and the dsvm job uses devstack-trusty15:56
odyssey4meubuntu-trusty is a newer d-i-b based image in all cloud providers, whereas devstack-trusty is prepared differently in each provider15:57
odyssey4methe images look different15:57
*** subscope has joined #openstack-ansible15:57
pabelangerodyssey4me: eventually all trusty images (minus devstack-) will be removed.  So, using devstack-* nodes is for future proofing15:57
pabelanger-infra will basically have a single image for trusty, centos7, fedora-21, etc15:58
pabelangerthey are a little different15:58
pabelangerdifference is being build using diskimage-builder and native cloud imagtes15:59
pabelangerimages*15:59
odyssey4mepabelanger odd, I was told otherwise - I heard that ubuntu-trusty was the way forward15:59
*** subscope has quit IRC15:59
matttcloudnull: going to have to test that one as i'm not familiar enough with the python code to simply review15:59
pabelangerodyssey4me: Ya, it is confusing.  You might be right, either way, they are condensing to a single image.  I should confirm again to see what it is16:00
odyssey4mepabelanger condensing to a d-i-b based image that's consistent between clouds helps... you're always testing on the same thing16:01
pabelangerodyssey4me: right. Last I looked not may were using ubuntu-trusty, should check again and see how is using it16:02
pabelangerdownside, there is no centos-7 dib yet16:02
pabelangershould try and get it going16:02
pabelangerodyssey4me: haha, looks like you are the only team that is currently using it :)16:03
odyssey4mepabelanger yeah, afaik the new images were still being 'beta' tested - we dived in to try them out, hence the -nv job :)16:03
pabelangereverything else is experimental16:03
*** kmARC_2 has quit IRC16:03
odyssey4meas you know, no-one will use it until they're forced tp16:03
odyssey4me*to16:03
pabelangerright16:03
odyssey4meeveryone ready for a quick triage meeting? cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, dolphm, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak16:04
Sam-I-Amyo16:04
pabelangerso, that's why I defaulted to devstack-* nodes,dibs were not ready16:04
d34dh0r53\o/16:04
*** Bjoern_ has joined #openstack-ansible16:04
mattt\o16:04
odyssey4meany volunteers to lead the triage?16:05
cloudnullo/ present16:05
odyssey4mepabelanger I'd like to chat a bit later around how better to implement our jobs if you can spare the time?16:05
odyssey4mein an hour or so16:05
Sam-I-Amodyssey4me: looks like you're it :)16:07
odyssey4mealright, let's do it: https://bugs.launchpad.net/openstack-ansible/+bug/150628516:07
openstackLaunchpad bug 1506285 in python-openstackclient "11.2.1 : openstack client with V3 auth causes usability issues" [Undecided,New]16:07
openstackgerritMerged openstack/openstack-ansible: Remove duplicated Aodh host var  https://review.openstack.org/24869616:07
pabelangerodyssey4me: Sounds good. Should be back from walking daughter to school16:08
Sam-I-Amthis bug... did something change in the openstack client?16:09
odyssey4meok, so this bug was a mess16:10
Sam-I-Amit is16:10
odyssey4meit has not been particularly clear what the issue is16:10
odyssey4mebut this was registered, as per the last comment: https://bugs.launchpad.net/openstack-ansible/+bug/151835116:11
openstackLaunchpad bug 1518351 in openstack-ansible trunk "Playbooks stop when enabling keystone ldap backend " [Medium,In progress] - Assigned to Kevin Carter (kevin-carter)16:11
Bjoern_the issue is clearly stated16:11
*** Bjoern_ is now known as BjoernT16:11
odyssey4meso that has a patch in play which resolves the playbook issue, I believe - I've not yet tested it16:11
BjoernTthe openstack client did not change, the issue getting introduces when enabling multi domain in keystone16:11
jthornei thought the issue was pretty clear, with ldap enabled, you have to run "openstack user list --domain default" instead of "openstack user list". if you run "openstack user list" the user gets a 401.16:12
BjoernTyeah among other thing like playbooks won16:12
BjoernT't work16:12
BjoernTbut I moved that into a different bug16:12
BjoernT151835116:12
odyssey4meagreed, BjoernT - as stated above :)16:12
odyssey4mejthorne can you please put that into https://bugs.launchpad.net/openstack-ansible/+bug/1506285 so that the keystone team can respond, as that is the exact info they need16:13
openstackLaunchpad bug 1506285 in python-openstackclient "11.2.1 : openstack client with V3 auth causes usability issues" [Undecided,New]16:13
cloudnulljthorne: BjoernT: https://review.openstack.org/#/c/248900/16:13
cloudnullsorry in two meetings16:14
BjoernTodyssey4me everything is in the bug16:14
odyssey4meI suspect this is not exclusive to an ldap back-end, this is probably a more general multi-domain issue16:14
BjoernTsee #1216:14
jthornecloudnull: oh my god i just clicked "open side-by-side all" in your review16:14
BjoernTyes correct multi domain in general16:14
jthorneso i submitted this which is similar to cloudnull's https://bugs.launchpad.net/openstack-ansible/+bug/151917416:15
openstackLaunchpad bug 1519174 in openstack-ansible "Set OpenStack service usernames and passwords to reference their own variables and work with LDAP Keystone Identity Backend" [Undecided,New]16:15
*** gparaskevas has quit IRC16:16
odyssey4meBjoernT so are you asking openstackclient to be modified to assume the default domain if no domain is given?16:16
*** mpavone has quit IRC16:17
BjoernTNo all i'm asking is to see if we can do something about  it since someone Mirantis claimed it might possibly work with a policy change16:17
BjoernTAgain, please READ #1216:17
odyssey4meright, I'm reading #12 and there is nothing actionable for openstack-ansible there as far as I can see16:18
BjoernTyes when we decide to roll out a policy change16:18
BjoernTPlease assign it to Dolph and go on16:18
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/151730416:20
openstackLaunchpad bug 1517304 in openstack-ansible "os-swift is missing configuration for the object-reconstructor" [Undecided,New]16:20
odyssey4meandymccr any thoughts on whether you'd consider this a bug or a wishlist item?16:21
andymccrodyssey4me: id say wishlist16:22
odyssey4mewishlist, alright16:23
*** mgoddard_ has quit IRC16:24
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/151816216:24
openstackLaunchpad bug 1518162 in openstack-ansible "Set openrc_os_username to keystone_admin_user_name variable" [Undecided,New]16:24
jthorneodyssey4me: ignore that16:24
jthorneodyssey4me: i merged that into another bug report16:24
*** mgoddard_ has joined #openstack-ansible16:24
evrardjpk16:24
jthornei merged that into this one: https://bugs.launchpad.net/openstack-ansible/+bug/151917416:25
openstackLaunchpad bug 1519174 in openstack-ansible "Set OpenStack service usernames and passwords to reference their own variables and work with LDAP Keystone Identity Backend" [Undecided,New]16:25
jthornewhich may overlap with cloudnull's submission16:25
*** rromans is now known as rromans_afk16:26
odyssey4mecloudnull ping?16:28
cloudnullyes?16:29
cloudnull sorry totally split brained16:29
odyssey4mecloudnull have you looked at the patch in https://bugs.launchpad.net/openstack-ansible/+bug/1519174 and how does it relate to https://review.openstack.org/248900 ?16:29
openstackLaunchpad bug 1519174 in openstack-ansible "Set OpenStack service usernames and passwords to reference their own variables and work with LDAP Keystone Identity Backend" [Undecided,New]16:29
*** rebase has joined #openstack-ansible16:30
cloudnulljthorne: this was already happening16:30
cloudnullthats why i didnt add it to the patch to add the ldap bool16:31
cloudnullIR https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_ceilometer/tasks/ceilometer_service_setup.yml16:31
cloudnullcalls https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_ceilometer/tasks/ceilometer_service_add.yml16:31
jthorneoh i see16:31
cloudnullso the service usernames and passwords are consistent16:32
jthornei gotcha16:32
jthornethe other piece in my bug report is the "when: keystone_ldap not set" which has the same outcome as your patch16:34
cloudnullthis https://review.openstack.org/#/c/248900  and https://review.openstack.org/#/c/238509/ is all we should need to fix the ldap deployment as I can see so far16:34
*** rebase has quit IRC16:34
cloudnullyea I made it configurable per role and globally controlled in the group_vars16:34
jthornegotcha.16:34
odyssey4meok cloudnull  shall we add this bug in the commit messages of those reviews, or shall we mark the bug as a dup?16:36
cloudnullno their different issues but are required for our ldap configs16:36
cloudnulltheir in the same thread, but solve different problems for sure.16:37
odyssey4meok, so I'll tweak the commit message of the reviews to include this one16:37
cloudnullif you think its needed16:38
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/151848516:39
openstackLaunchpad bug 1518485 in openstack-ansible " lxcbr0 disappeared run-playbooks.sh " [Undecided,New] - Assigned to Anton Haldin (anton-haldin)16:39
evrardjpdisappeared :)16:39
evrardjpnot enough info, we don't even know the lxc installed version16:40
odyssey4meyeah, the only way I can see this happening is if lxc was removed using a teardown or something16:40
evrardjpdiscard my remark16:41
evrardjpmmm this line should still happen: https://github.com/openstack/openstack-ansible/blob/master/scripts/run-playbooks.sh#L5616:41
evrardjpbut it's definitely not enough info16:41
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/151849216:43
openstackLaunchpad bug 1518492 in openstack-ansible "Flawed volume delete behavior in teardown script" [Undecided,New]16:43
odyssey4methis is a scope issue again - in my view the teardown should only remove things implemented by the playbooks... it should not destroy things prepared on the host by the bootstrap (assuming the bootstrap-aio script was used).16:44
odyssey4meany volunteers to add a warning, or to shift the scope?16:45
evrardjpI'll take it16:46
odyssey4methanks evrardjp16:46
evrardjpnp16:46
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/151884516:47
openstackLaunchpad bug 1518845 in openstack-ansible "cinder doesn't set a default volume type" [Undecided,New]16:47
odyssey4meas I recall, Richard is planning to fix this himself16:48
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/151905316:49
openstackLaunchpad bug 1519053 in openstack-ansible "container backend not set correctly with Ansible v2" [Undecided,New] - Assigned to Jimmy McCrory (jimmy-mccrory)16:49
evrardjplow because ansible216:49
evrardjpbut important for us to follow16:50
odyssey4meyeah, jmccrory has self allocated it16:50
odyssey4meI'm sure there'll be plenty more issue found.16:50
evrardjp:D16:50
odyssey4menext: https://bugs.launchpad.net/openstack-ansible/+bug/151910516:51
openstackLaunchpad bug 1519105 in openstack-ansible "Glance has wrong rpc_backend configured when ceilometer is enabled" [Undecided,New] - Assigned to Miguel Alejandro Cantu (miguel-cantu)16:51
*** ashishb has joined #openstack-ansible16:51
odyssey4meit looks like alextricity is on it already16:52
alextricityyup16:52
odyssey4meimportance?16:52
alextricityI've uncovered more problems though16:52
alextricitymedium16:52
*** javeriak has quit IRC16:53
odyssey4mealextricity more issues?16:53
alextricityYes. Serious ones too. The ceilometer-agent-central service never comes up, even though the playbooks successfully finish16:54
matttodyssey4me: regarding https://bugs.launchpad.net/openstack-ansible/+bug/1518845, you can set a default, but we don't automatically pick one for the deployment16:54
openstackLaunchpad bug 1518845 in openstack-ansible "cinder doesn't set a default volume type" [Medium,Confirmed] - Assigned to Richard Jones (r1chardj0n3s)16:54
alextricitySeemingly it looks like the ceilometer service is running, however.16:54
alextricityAnd upstart gives me *no info* as to why it doesn't come up16:54
alextricityWhenever I run 'start ceilometer-agent-central', the services comes up, but then dies16:55
alextricityAnd no log file is generated16:55
alextricityI'll make a bug once I figure out the issue16:55
odyssey4mealextricity ugh, I see that on one of my recent builds16:56
odyssey4melet's rather register a bug and assign it to you16:56
alextricityOkay. I'll do that16:57
odyssey4meI'm doing some gate work and may be able to assist you there16:57
alextricityThanks bud :)16:57
odyssey4mealextricity ping me the bug ref when you've registsred it please16:57
alextricitySure will16:57
odyssey4mealright, that's the new bug list - any others?16:57
*** javeriak has joined #openstack-ansible16:57
odyssey4memattt you wanted to go back to https://bugs.launchpad.net/openstack-ansible/+bug/1518845 ?16:58
openstackLaunchpad bug 1518845 in openstack-ansible "cinder doesn't set a default volume type" [Medium,Confirmed] - Assigned to Richard Jones (r1chardj0n3s)16:58
matttodyssey4me: yeah, just pointing out that it is possible to set that16:58
matttodyssey4me: as i htink richard is suggesting we do that in our role16:58
odyssey4memattt yeah, as I recall he wanted to fix it to get familiar with the code16:59
matttodyssey4me: oh!  ok great16:59
odyssey4meperhaps you could add some triage info to the bug report to help him along the way? :)16:59
matttsure16:59
*** javeriak_ has joined #openstack-ansible17:00
*** javeriak has quit IRC17:00
evrardjpdone?17:00
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible: Improve message of the teardown.sh  https://review.openstack.org/24932517:00
*** mgoddard_ has quit IRC17:00
*** slotti has quit IRC17:01
*** mgoddard has joined #openstack-ansible17:01
odyssey4mealright everyone, thank you all for your time!17:01
spotzevrardjp question on that patch you just submitted17:02
evrardjpmmm yes?17:02
spotzAre you talking about a single lxc or multiple when you're talking about the lvm group?17:03
evrardjplxc volume group17:03
spotzJust trying to figure out if the ' is in the right place:)17:04
evrardjpyeah english is not my mother language17:04
spotzI think you want lxcs' if you're talking in the plural17:04
evrardjplet me rephrase that17:04
spotzk17:04
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible: Improve message of the teardown.sh  https://review.openstack.org/24932517:06
evrardjpavoided the 's ;)17:06
*** sdake_ has joined #openstack-ansible17:07
evrardjpspotz: it's one vg, so it should be lxcs'17:07
*** javeriak_ has quit IRC17:07
evrardjpbut I think it's more clear now17:07
*** sdake has quit IRC17:07
evrardjpclearer17:07
*** javeriak has joined #openstack-ansible17:09
spotzyeah but if you're talking about multiple lxcs it's lxcs' vs lxc's. Hence the question:)17:09
spotzsweet perfect:)17:10
spotzgrammar just sucks, I personally love run on sentances17:11
*** javeriak_ has joined #openstack-ansible17:13
*** javeriak has quit IRC17:13
*** adac has quit IRC17:15
*** eil397 has joined #openstack-ansible17:15
*** alkari has joined #openstack-ansible17:16
*** sdake_ has quit IRC17:17
*** javeriak_ has quit IRC17:19
*** javeriak has joined #openstack-ansible17:23
*** bangfrog has joined #openstack-ansible17:45
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Fixes playbook runtime issues with ldap  https://review.openstack.org/24890017:48
*** javeriak_ has joined #openstack-ansible17:48
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Keystone domain fix  https://review.openstack.org/23850917:49
*** karimb has quit IRC17:49
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Keystone domain fix  https://review.openstack.org/23850917:50
bgmccollumodyssey4me: most of the SSH issues i saw were when ansible forks were > sshd max sessions / max startups...17:50
*** javeriak has quit IRC17:51
bgmccollumbeen meaning to work on figuring out the best forks values based on sshd config, but it requires fetching the lesser value across all the infra nodes.17:54
odyssey4mebgmccollum yeah, this time it's different - it seems to relate to pipelining17:54
*** sacharya has joined #openstack-ansible18:00
cloudnullodyssey4me: this is the package diff between dsvm and nv http://paste.openstack.org/show/479906/18:02
*** sacharya has quit IRC18:04
*** elgertam has joined #openstack-ansible18:06
bgmccollumalextricity: when ive seen services start, immediately die, and have no logs, its been a permission issue on the logging path...might be something to check18:09
cloudnullodyssey4me: i think we may have broken pipelining18:10
cloudnulli found this18:11
cloudnullhttps://github.com/ansible/ansible/pull/924618:11
cloudnullwhich was a closed issue by dehan18:11
cloudnullbut documented that if you have ssh_args pipelining may not work18:11
cloudnulland we have https://github.com/openstack/openstack-ansible/blob/master/playbooks/ansible.cfg#L2518:11
openstackgerritKevin Carter proposed openstack/openstack-ansible: Remove ssh args to ensure pipelining is working  https://review.openstack.org/24934818:15
cloudnull^ testing the theory.18:16
*** elgertam1 has joined #openstack-ansible18:20
bgmccollumcloudnull: did you see this comment? -- https://github.com/ansible/ansible/issues/7251#issuecomment-5788822318:21
cloudnullyes18:22
*** elgertam has quit IRC18:22
cloudnulli've been diving into the connection plugin and found https://github.com/ansible/ansible/blob/v1.9.4-1/lib/ansible/runner/connection_plugins/ssh.py#L68-L7218:22
cloudnullwhich then lead me to good that found that issue18:23
bgmccollumcloudnull : i see ControlMaster and ControlPersist defined, but not ControlPath18:23
bgmccollumin the ansible.cfg ssh_args18:23
cloudnullyes, it was not being used18:23
cloudnullbut with that line removed it would be18:23
cloudnullthis little bit may be biting us https://github.com/ansible/ansible/blob/v1.9.4-1/lib/ansible/runner/connection_plugins/ssh.py#L65-L6818:24
*** elgertam1 has quit IRC18:24
cloudnulli guess we could simply add the ControlPath but im questioning what benifit we get from the other bits18:25
odyssey4mecloudnull yeah, it seems that we're pretty much using the defaults anyway18:28
bgmccollumwere all these ssh options added when trying to diagnose the other ssh failure from the past?18:29
odyssey4mecloudnull fyi the other bits were added to combat the forks issue before we knew it related to forks18:29
odyssey4mebgmccollum ++18:29
bgmccollumbingo18:29
odyssey4meso let's rip them out!18:29
bgmccollumnuke it18:29
cloudnullso we'll see if the test works18:29
cloudnulli was digging through the ansible code before I found that nonsense. . . While I love ansible I HATE their coding style (or lack there of)...18:30
odyssey4methe only other issue plaguing the gate is failed tempest runs, which I expect should be resolved by https://review.openstack.org/248887 for now while I work on further tweaks to split the gate up18:30
odyssey4mepabelanger you mean like this? https://review.openstack.org/24877418:36
pabelangerodyssey4me: Ya, I suspect that is the issue.  I currently use sudo: yes in my roles, but need to convert to newer syntax18:37
pabelangerI should test that too18:37
odyssey4mepabelanger that just failed too :/18:37
pabelangerodyssey4me: how did you run your roles before, under root directly or via sudo?18:43
cloudnullpabelanger: in the new roles they were being executed directly as root18:48
pabelangerya18:48
pabelangerso that's the difference18:48
cloudnulltox -e ...18:48
pabelangerwe are using non-root for tox -e18:48
pabelangerso: https://github.com/openstack/openstack-ansible-apt_package_pinning/blob/master/tasks/main.yml#L16 needs sudo: yes18:49
pabelangeror what ever the new syntax is18:49
cloudnullso we can, in our inventory add a sudo option to escalate18:49
pabelangerya18:49
cloudnullha great minds18:49
pabelangerdon't know which why is better honestly :)  Should read up on it and see what people default too18:49
pabelangersudo: yes was pretty easy to do18:49
cloudnullwe should be able to do that in inventory without effecting the role18:49
cloudnullhttps://github.com/openstack/openstack-ansible-apt_package_pinning/blob/master/tests/inventory18:50
cloudnull^ adding it to there should be all we need to do18:50
cloudnullIE what evrardjp did here https://review.openstack.org/#/c/248774/1/tests/inventory,cm18:50
pabelangerya, so this should work: https://review.openstack.org/#/c/248774/18:51
cloudnulldo we have a place to test without impacting the jobs ?18:52
pabelangernot sure I follow18:53
*** eil397 has quit IRC18:53
cloudnullfor the gate18:53
cloudnulli mean i can spin up an env add a user with sudo no pass and test18:54
cloudnullbut i didnt know if we had an instance to beat on while debuging the gate18:54
*** sacharya has joined #openstack-ansible18:56
pabelangerAh19:05
pabelangernothing yet19:05
pabelangerfor me, I have a simple playbook to fire up a node on my local cloud19:05
evrardjpcloudnull odyssey4me bgmccollum, would you mind explaining your ansible findings, I'm not sure to have understood19:07
evrardjpat some point we added ssh_args, and it broke pipelining, that's what you mean?19:08
*** skamithi13 has joined #openstack-ansible19:09
*** alejandrito has quit IRC19:10
cloudnullevrardjp: o/19:11
cloudnullso yes a long while back we had added ssh_args to the system19:11
bgmccollumwell technically, it may have broken Control* settings...19:11
evrardjpI went through that19:11
evrardjpdidn't I speak about it on this chan?19:11
cloudnullidk. and if you did i missed it ... :(19:12
evrardjpanyway, pipelining is the foe of reliability19:12
evrardjplet me find an article for you19:13
evrardjpI can't find anything better than that for the moment: http://www.anchor.com.au/blog/2010/02/ssh-controlmaster-the-good-the-bad-the-ugly/19:17
evrardjpmaybe I'm completely out of scope19:18
evrardjpI should read how pipelining is implemented19:18
*** vCloudernBeer has joined #openstack-ansible19:21
bgmccollumevrardjp: my understanding is that pipelining is an ansible thing...where as controlmaster is an ssh thing19:26
evrardjpisn't pipelining a fancy name to put controlmaster controlpath ... arguments for connections?19:27
bgmccollumevrardjp: https://github.com/ansible/ansible/issues/7251#issuecomment-5788822319:27
evrardjpok my bad19:29
bgmccollumno worries...its all very confusing19:29
evrardjpso, wouldn't it be better for the future, to remove our specific ansible.cfg configuration changes (ssh_args and pipelining True), to be with safe defaults, and enable pipelining for the gate checks?19:30
evrardjpor for AIOs for example19:30
bgmccollumevrardjp: cloudnull has a patch up with just the ssh_args removal. i think after a couple rechecks to confirm no ssh failures. if there are still failure, we might look at disabling pipelining, or both...19:31
odyssey4meevrardjp you'll notice that the sudo method din't work in https://review.openstack.org/248774 - are you sure that's done right?19:33
odyssey4mecloudnull pabelanger ^19:33
evrardjpI'm not sure19:35
*** javeriak has joined #openstack-ansible19:35
evrardjpI followed the docs19:35
*** ashishb has quit IRC19:36
*** javeriak_ has quit IRC19:36
odyssey4meevrardjp are you sure that become_ is not an ansible 2 thing?19:37
palendaePretty sure become is a 1.9+ thing19:38
evrardjpI'll check on my host19:38
odyssey4meok, I'm out for the night - enough is enough19:39
odyssey4mecloudnull FYI it seems that the upsteam LXC bug related to https://review.openstack.org/196702 has been fixed, so I've restored the patch - time to give that a try again19:40
evrardjpI'll check tomorrow, sorry for that19:42
cloudnullodyssey4me: we'll likely need to make sure we have a newer kernel too19:44
bgmccollumanyone have some examples of recent tempest failures in the gate?19:44
odyssey4mecloudnull perhaps just make it run only if the kernel is newer than x19:44
odyssey4mein my cloud server spun up a day or two ago, it was present19:45
odyssey4mebgmccollum several, most of the -v's are related to either tempest or ssh19:45
odyssey4mebgmccollum that's why I added https://review.openstack.org/248887 to reduce resource usage19:46
bgmccollumodyssey4mel: tempest failures specifically...i know there is an issue on RAX pub cloud where libvirt cant identify the CPU (based of a know set of cpu flags)...so nova compute doesn't start...vms won't boot...just wanted to see if any of those failures were related to that...i have a modified libvirt cpu_map.xml that allows libvirt to identify the cpu make / model and fully startup...19:48
bgmccollumodyssey4me: ^19:48
bgmccollumnot all hosts exhibit the problem though...19:49
odyssey4meoh, that's interesting19:49
*** sdake has joined #openstack-ansible19:49
odyssey4meso we do have the occasional scheduler failures due to no host found19:49
bgmccollumthats probably it19:50
odyssey4meif you think those are related to libvirt, then perhaps that's worth adding for gate checks19:50
*** adac has joined #openstack-ansible19:51
odyssey4mebgmccollum can you push up a WIP review which I can work from19:51
odyssey4meI'll rearrange where it should go and how it can be applied - we can give that a go and see.19:51
*** sdake_ has joined #openstack-ansible19:52
bgmccollumodyssey4me: this is what needs to be added to the cpu_map.xml -- https://github.com/cloud-training/rpc-heat-ansible/blob/master/playbooks/files/usr/share/libvirt/cpu_map.xml#L337-L38119:52
bgmccollumthen you just bounce libvirt and nova-compute19:52
bgmccollumhttps://github.com/cloud-training/rpc-heat-ansible/blob/master/playbooks/rpc-11.0-playbook.yml#L481-L49619:52
pabelangerodyssey4me: worked for me: https://review.openstack.org/#/c/249376/219:53
odyssey4mecool, thanks - I'll factor that in19:53
bgmccollumodyssey4me: ill cook up something19:53
odyssey4mepabelanger thanks - I'll have to look into it tomorrow19:54
odyssey4methanks for confirming though19:54
odyssey4meand bgmccollum thanks for the heads up - that's super useful!19:54
*** sdake has quit IRC19:54
*** jwagner is now known as jwagner_away19:55
bgmccollumnp19:55
odyssey4menow I really am outta here19:55
bgmccollumgo19:55
vCloudernBeermeeting done?19:58
*** BjoernT is now known as Bjoern_zZzZzZzZ20:03
*** ashishb has joined #openstack-ansible20:04
cloudnullvCloudernBeer: yup, we meet at 1600 utc for bug triage20:05
*** sacharya has quit IRC20:06
vCloudernBeer<cloudnull> thanks.20:06
bgmccollumcloudnull: mind if i throw a couple rechecks at the ssh_args patch? see if we can get it to tip over...20:06
cloudnullsorry20:06
cloudnullsure by all means20:06
bgmccollumdone20:06
bgmccollumill keep an eye on it20:06
*** sdake_ has quit IRC20:15
*** Bjoern_zZzZzZzZ is now known as BjoernT20:15
*** sdake has joined #openstack-ansible20:15
*** greg_a has quit IRC20:16
*** skamithi13 has quit IRC20:17
*** javeriak has quit IRC20:22
*** vCloudernBeer has quit IRC20:22
*** jwagner_away is now known as jwagner20:32
*** ashishb has quit IRC20:34
*** sacharya has joined #openstack-ansible20:44
*** eil397 has joined #openstack-ansible20:47
*** greg_a has joined #openstack-ansible20:50
*** matt6434 is now known as mattoliverau20:52
*** b3rnard0 is now known as b3rnard0_away20:53
*** sdake has quit IRC20:59
*** sdake has joined #openstack-ansible20:59
*** phiche has joined #openstack-ansible21:01
*** karimb has joined #openstack-ansible21:11
*** adac has quit IRC21:16
*** adac has joined #openstack-ansible21:30
*** sacharya has quit IRC21:41
*** sacharya has joined #openstack-ansible21:43
*** elo has quit IRC21:46
*** jaypipes has quit IRC21:47
*** sacharya has quit IRC21:47
*** alkari has quit IRC21:54
*** elo has joined #openstack-ansible21:54
evrardjpFYI I did a quick check of the variables in the inventory, it works on my vagrant machine21:56
evrardjpbut it doesn't seem to work with current tox configuration21:57
evrardjpor playbook21:57
evrardjpor whatever21:57
evrardjpI'll check further tomorrow21:57
evrardjpthe ansible_become_ variables*21:58
evrardjpoh another quick check seem that setting a remote_tmp folder in ansible.cfg makes the ansible-playbook run by tox working on my local box22:01
*** adac has quit IRC22:15
*** zaina has joined #openstack-ansible22:19
*** jthorne has quit IRC22:30
*** zaina has quit IRC22:43
*** zaina has joined #openstack-ansible22:45
*** b3rnard0_away is now known as b3rnard022:56
*** sacharya has joined #openstack-ansible22:57
*** phiche has quit IRC23:04
*** spotz is now known as spotz_zzz23:06
*** tlian has joined #openstack-ansible23:26
*** tlian has quit IRC23:44
*** woodard_ has joined #openstack-ansible23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!