| thetrav | so, am I the only one hitting issues with too many open files? | 00:00 |
|---|---|---|
| thetrav | when running the setup-hosts.yml playbook? | 00:00 |
| thetrav | seems to hit around the time it's hitting the rsyslog_client tasks | 00:01 |
| thetrav | varies depending on where I set the ulimit to | 00:01 |
| thetrav | appears to be related to an ansible 2.x bug? | 00:01 |
| *** oneswig has joined #openstack-ansible | 00:04 | |
| *** oneswig has quit IRC | 00:09 | |
| *** v1k0d3n has joined #openstack-ansible | 00:13 | |
| *** aernhart has quit IRC | 00:23 | |
| *** aernhart has joined #openstack-ansible | 00:23 | |
| *** pacoblox has joined #openstack-ansible | 00:24 | |
| *** thorst has joined #openstack-ansible | 00:27 | |
| mattt | thetrav: yeah, this project is still on ansible 1.9 | 00:31 |
| mattt | thetrav: so you're in unchartered territory | 00:32 |
| thetrav | so I checked | 00:32 |
| thetrav | 1.9.4 | 00:32 |
| thetrav | I'm not using 2.0 | 00:32 |
| thetrav | also the bug didn't just hit 2.0 | 00:33 |
| thetrav | it hit heaps of places | 00:33 |
| thetrav | however | 00:33 |
| thetrav | it looks like it's only for when there's tons and tons of targets | 00:33 |
| thetrav | so I don't really understand why it's hitting me when I've got only the 3 infra hosts and one compute host | 00:33 |
| *** thorst has quit IRC | 00:36 | |
| mattt | thetrav: that is odd, because i know openstack-ansible has been used to deploy quite big clusters -- hundreds of nodes | 00:36 |
| *** thorst has joined #openstack-ansible | 00:37 | |
| thetrav | yeah | 00:39 |
| thetrav | so what I'm thinking | 00:39 |
| thetrav | something in my config is causing it to open files | 00:39 |
| thetrav | likely a range | 00:40 |
| thetrav | I did at one point specify used_ips incorrectly and it blew up trying to enumerate from 10.4.0.0 to 256.0.10.4 | 00:41 |
| thetrav | interestingly enough that causes a segfault | 00:41 |
| thetrav | ok, so I don't really understand how to read lsof output, but there's tons of rows with ansible-p opening /dev/null | 00:42 |
| thetrav | I'm thinking that's the problem | 00:42 |
| *** thorst has quit IRC | 00:44 | |
| mattt | thetrav: i don't have any deploys in front of you, i'm not going to be of much use :( | 00:44 |
| mattt | s/of you/of me/ | 00:45 |
| *** pacoblox has quit IRC | 00:45 | |
| * thetrav shrugs rubber ducking is of use :D | 00:45 | |
| *** pacoblox has joined #openstack-ansible | 00:45 | |
| *** aernhart has quit IRC | 00:46 | |
| mattt | thetrav: swing back tomorrow, we have far more people online during the week | 00:47 |
| thetrav | heh, it's Monday for me, but yes I'll be around tomorrow | 00:47 |
| mattt | thetrav: you are using ubuntu 14.04 right? | 00:47 |
| thetrav | yep | 00:47 |
| *** aernhart has joined #openstack-ansible | 00:47 | |
| thetrav | shouldn't matter, but the targets are bare metal Cisco UCS, the deploy host is a vm running on an openstack I deployed with my own ansible scripts | 00:48 |
| thetrav | hoping to decommission my own scripts and move to something community supported | 00:48 |
| *** furlongm has joined #openstack-ansible | 00:48 | |
| mattt | thetrav: what exactly is cisco ucs? | 00:49 |
| thetrav | UCS is the first three letters in their product code | 00:49 |
| thetrav | generic compute servers | 00:49 |
| thetrav | which so far I've been fairly unimpressed with | 00:50 |
| thetrav | that however, is largely irrelevant | 00:50 |
| mattt | thetrav: so that's just the chassis type? they're running 14.04? | 00:50 |
| thetrav | yeah | 00:50 |
| mattt | ok cool | 00:51 |
| thetrav | host OS is 14.04 everywhere | 00:51 |
| *** markvoelker has joined #openstack-ansible | 00:51 | |
| *** markvoelker has quit IRC | 00:56 | |
| mattt | thetrav: so you're saying it's failing on https://github.com/openstack/openstack-ansible-rsyslog_client/blob/master/tasks/rsyslog_client_install-apt.yml#L34-L39 ? | 00:56 |
| thetrav | so that was the first place it failed | 00:56 |
| thetrav | however I can cause it to fail in different places by changing the allowed open files | 00:57 |
| mattt | thetrav: you are running as root right ? | 00:57 |
| thetrav | yes | 00:57 |
| thetrav | right now for example it is failing at rsyslog_client | Write log rotate file | 00:57 |
| thetrav | I believe I am unable to allow it to go further because of a hard limitation on the select function in the OS | 00:58 |
| thetrav | that's way outside my comfort zone though | 00:58 |
| thetrav | ere be dragons | 00:58 |
| thetrav | so I'm in the process of monkeypatching the ansible code to log the output of lsof and do some horrible bisection | 00:58 |
| thetrav | to try and find where the open file handles is increasing | 00:59 |
| thetrav | on the plus side, because it's interpreted and not compiled, this is something I am confident I can actually debug with my horrible brute force method | 00:59 |
| *** woodard has quit IRC | 01:04 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh: Remove duplicate usage of the galera role in aodh func test https://review.openstack.org/334068 | 01:06 |
| *** smatzek has joined #openstack-ansible | 01:23 | |
| *** pacoblox has quit IRC | 01:25 | |
| *** aernhart has quit IRC | 01:25 | |
| *** pacoblox has joined #openstack-ansible | 01:25 | |
| thetrav | ok, so from watching lsof it gains about 10k file handles as soon as it tries to install pip | 01:25 |
| *** aernhart has joined #openstack-ansible | 01:25 | |
| thetrav | it falls over at just under 12k so I guess I'll look at why pip install is picking up so many file handles and not putting them down | 01:26 |
| thetrav | did I mention that almost all of the handles are /dev/null ? | 01:31 |
| *** pacoblox has quit IRC | 01:31 | |
| *** pacoblox has joined #openstack-ansible | 01:31 | |
| *** smatzek has quit IRC | 01:32 | |
| *** pacoblox has quit IRC | 01:36 | |
| *** pacoblox has joined #openstack-ansible | 01:36 | |
| mattt | thetrav: what version of pip out of curiosity? | 01:39 |
| thetrav | which version is it installing to the target? | 01:40 |
| *** pacoblox1 has joined #openstack-ansible | 01:40 | |
| thetrav | pip 8.1.2 from /usr/local/lib/python2.7/dist-packages (python 2.7) | 01:40 |
| *** pacoblox has quit IRC | 01:41 | |
| thetrav | which in fact is the same as the version on the deployment node | 01:41 |
| *** thorst has joined #openstack-ansible | 01:42 | |
| mattt | ok that's good then :) | 01:43 |
| thetrav | ok, so do you by any chance know where openstack-ansible keeps its role files? | 01:45 |
| thetrav | on the deployment host? | 01:45 |
| thetrav | it's /etc/ansible | 01:49 |
| *** thorst has quit IRC | 01:50 | |
| *** smatzek has joined #openstack-ansible | 01:50 | |
| *** markvoelker has joined #openstack-ansible | 01:52 | |
| *** markvoelker has quit IRC | 01:57 | |
| *** pacoblox has joined #openstack-ansible | 02:05 | |
| *** pacoblox1 has quit IRC | 02:05 | |
| *** thorst has joined #openstack-ansible | 02:08 | |
| *** pacoblox has quit IRC | 02:08 | |
| *** pacoblox has joined #openstack-ansible | 02:11 | |
| *** thorst has quit IRC | 02:12 | |
| *** pacoblox has quit IRC | 02:13 | |
| *** thorst has joined #openstack-ansible | 02:13 | |
| *** smatzek has quit IRC | 02:22 | |
| *** thorst has quit IRC | 02:22 | |
| *** sdake_ has quit IRC | 02:23 | |
| *** pacoblox has joined #openstack-ansible | 02:24 | |
| *** weezS has joined #openstack-ansible | 02:41 | |
| *** pacoblox has quit IRC | 02:52 | |
| *** markvoelker has joined #openstack-ansible | 02:53 | |
| *** pacoblox has joined #openstack-ansible | 02:55 | |
| *** markvoelker has quit IRC | 02:57 | |
| *** mhayden_ is now known as mhayden | 02:59 | |
| *** mhayden has quit IRC | 02:59 | |
| *** mhayden has joined #openstack-ansible | 03:00 | |
| thetrav | ok, so the files are opened before it does the first task in the pip_install play | 03:01 |
| thetrav | >_< | 03:01 |
| *** pacoblox has quit IRC | 03:06 | |
| *** pacoblox has joined #openstack-ansible | 03:06 | |
| *** pacoblox has quit IRC | 03:11 | |
| *** v1k0d3n has quit IRC | 03:20 | |
| *** thorst has joined #openstack-ansible | 03:22 | |
| *** phalmos has joined #openstack-ansible | 03:23 | |
| *** winggundamth has quit IRC | 03:23 | |
| *** Tridde has quit IRC | 03:24 | |
| *** Trident has joined #openstack-ansible | 03:25 | |
| *** mennie has quit IRC | 03:26 | |
| *** mennie has joined #openstack-ansible | 03:27 | |
| *** thorst has quit IRC | 03:27 | |
| *** phalmos has quit IRC | 03:28 | |
| *** zerda2 has joined #openstack-ansible | 03:33 | |
| *** pacoblox has joined #openstack-ansible | 03:34 | |
| *** pacoblox1 has joined #openstack-ansible | 03:40 | |
| *** pacoblox has quit IRC | 03:40 | |
| *** sacharya has joined #openstack-ansible | 03:41 | |
| *** pacoblox has joined #openstack-ansible | 03:46 | |
| *** pacoblox1 has quit IRC | 03:46 | |
| *** markvoelker has joined #openstack-ansible | 03:54 | |
| *** markvoelker has quit IRC | 03:58 | |
| *** pacoblox has quit IRC | 03:59 | |
| *** pacoblox has joined #openstack-ansible | 03:59 | |
| thetrav | ok, so I realised I had 10 forks running | 04:04 |
| thetrav | lowering it to 1 gives the same failure but it's only around 1k file handles | 04:04 |
| thetrav | which is more consistent with the limitations of select (1024?) | 04:05 |
| *** oneswig has joined #openstack-ansible | 04:06 | |
| *** pacoblox1 has joined #openstack-ansible | 04:06 | |
| *** pacoblox has quit IRC | 04:06 | |
| *** oneswig has quit IRC | 04:10 | |
| *** pacoblox1 has quit IRC | 04:18 | |
| *** sguduru has joined #openstack-ansible | 04:23 | |
| *** thorst has joined #openstack-ansible | 04:25 | |
| *** thorst has quit IRC | 04:32 | |
| *** sguduru has quit IRC | 04:36 | |
| *** markvoelker has joined #openstack-ansible | 04:54 | |
| *** markvoelker has quit IRC | 04:59 | |
| *** thorst has joined #openstack-ansible | 05:29 | |
| *** saneax_AFK is now known as saneax | 05:29 | |
| *** bryan_att has quit IRC | 05:33 | |
| *** winggundamth has joined #openstack-ansible | 05:34 | |
| *** thorst has quit IRC | 05:36 | |
| *** lunarlamp is now known as mariusv | 05:36 | |
| *** mariusv has quit IRC | 05:43 | |
| *** markvoelker has joined #openstack-ansible | 05:55 | |
| *** kylek3h has joined #openstack-ansible | 05:56 | |
| *** markvoelker has quit IRC | 06:00 | |
| *** kylek3h has quit IRC | 06:01 | |
| *** bootsha has joined #openstack-ansible | 06:05 | |
| openstackgerrit | Darren Chan proposed openstack/openstack-ansible: [docs] Address tox errors https://review.openstack.org/334298 | 06:18 |
| openstackgerrit | Darren Chan proposed openstack/openstack-ansible: [docs] Address tox errors https://review.openstack.org/334298 | 06:19 |
| *** neilus has joined #openstack-ansible | 06:19 | |
| *** zerda2 has quit IRC | 06:29 | |
| *** zerda2 has joined #openstack-ansible | 06:30 | |
| *** zerda2 has quit IRC | 06:30 | |
| *** M00nr41n has joined #openstack-ansible | 06:33 | |
| *** thorst has joined #openstack-ansible | 06:34 | |
| *** jiteka has joined #openstack-ansible | 06:35 | |
| *** v1k0d3n has joined #openstack-ansible | 06:35 | |
| *** v1k0d3n has quit IRC | 06:40 | |
| *** weezS has quit IRC | 06:40 | |
| *** pcaruana has joined #openstack-ansible | 06:41 | |
| *** thorst has quit IRC | 06:41 | |
| *** jiteka has quit IRC | 06:41 | |
| *** zhangjn has quit IRC | 06:54 | |
| *** zhangjn has joined #openstack-ansible | 06:54 | |
| *** sacharya has quit IRC | 06:55 | |
| *** markvoelker has joined #openstack-ansible | 06:56 | |
| *** markvoelker has quit IRC | 07:01 | |
| *** jamielennox is now known as jamielennox|away | 07:01 | |
| *** persia has quit IRC | 07:08 | |
| *** persia has joined #openstack-ansible | 07:11 | |
| *** oneswig has joined #openstack-ansible | 07:12 | |
| *** fishcried1 has joined #openstack-ansible | 07:17 | |
| *** fishcried1 has quit IRC | 07:18 | |
| *** zerda2 has joined #openstack-ansible | 07:23 | |
| *** Oku_OS-away is now known as Oku_OS | 07:28 | |
| *** pacoblox has joined #openstack-ansible | 07:31 | |
| *** javeriak has joined #openstack-ansible | 07:33 | |
| *** pacoblox has quit IRC | 07:36 | |
| *** thorst has joined #openstack-ansible | 07:40 | |
| *** marcoamorales is now known as ameobapox | 07:40 | |
| *** thorst has quit IRC | 07:46 | |
| *** sacharya has joined #openstack-ansible | 07:56 | |
| *** permalac has joined #openstack-ansible | 07:56 | |
| *** sacharya has quit IRC | 08:01 | |
| *** neilus has quit IRC | 08:03 | |
| *** thetrav has quit IRC | 08:03 | |
| *** neilus has joined #openstack-ansible | 08:06 | |
| *** bsv has joined #openstack-ansible | 08:11 | |
| evrardjp | good morning | 08:23 |
| *** admin0 has joined #openstack-ansible | 08:25 | |
| admin0 | good morning all :) | 08:26 |
| kamsz | mornin | 08:26 |
| bsv | admin0: mornings | 08:27 |
| admin0 | morning .. | 08:28 |
| admin0 | whats cooking today :D | 08:28 |
| bsv | Big news, it was decided overnight to start over from scratch with all OpenStack projects. All will be rewritten in erlang. Even ansible playbooks! ;) | 08:31 |
| bsv | So nothing big... :) | 08:31 |
| *** electrofelix has joined #openstack-ansible | 08:32 | |
| admin0 | if you had said go :) would be believeable :) | 08:32 |
| admin0 | erlang gave it away :) | 08:32 |
| bsv | :D | 08:32 |
| bsv | Damn... | 08:32 |
| bsv | Erhmm, I meant to say Haskell. | 08:32 |
| admin0 | i recall threads trying to use go language for swift :) | 08:32 |
| *** fawadkhaliq has joined #openstack-ansible | 08:39 | |
| *** thorst has joined #openstack-ansible | 08:44 | |
| evrardjp | erlang could actually make sense | 08:50 |
| evrardjp | and good morning admin0 bsv kamsz | 08:50 |
| admin0 | morning evrardjp | 08:50 |
| *** thorst has quit IRC | 08:52 | |
| kamsz | evrardjp: mornin | 08:52 |
| bsv | moin evrardjp | 08:52 |
| *** sacharya has joined #openstack-ansible | 08:57 | |
| *** neilus has quit IRC | 09:00 | |
| *** neilus has joined #openstack-ansible | 09:01 | |
| *** sacharya has quit IRC | 09:01 | |
| *** asettle has joined #openstack-ansible | 09:12 | |
| *** jamielennox|away is now known as jamielennox | 09:26 | |
| *** kylek3h has joined #openstack-ansible | 09:33 | |
| *** kylek3h has quit IRC | 09:38 | |
| *** weezS has joined #openstack-ansible | 09:39 | |
| *** admin0 has quit IRC | 09:46 | |
| *** bootsha has quit IRC | 09:47 | |
| *** neilus has quit IRC | 09:47 | |
| *** neilus has joined #openstack-ansible | 09:48 | |
| *** neilus has quit IRC | 09:48 | |
| *** neilus has joined #openstack-ansible | 09:49 | |
| *** neilus has quit IRC | 09:49 | |
| *** thorst has joined #openstack-ansible | 09:49 | |
| *** neilus has joined #openstack-ansible | 09:50 | |
| *** javeriak has quit IRC | 09:50 | |
| *** javeriak has joined #openstack-ansible | 09:55 | |
| *** thorst has quit IRC | 09:57 | |
| *** admin0 has joined #openstack-ansible | 10:00 | |
| *** javeriak has quit IRC | 10:03 | |
| *** javeriak has joined #openstack-ansible | 10:03 | |
| *** neilus has quit IRC | 10:07 | |
| *** asettle has quit IRC | 10:10 | |
| *** asettle has joined #openstack-ansible | 10:11 | |
| openstackgerrit | Merged openstack/openstack-ansible-security: Docs: Fix rendering of :orphan: https://review.openstack.org/332983 | 10:12 |
| *** bootsha has joined #openstack-ansible | 10:26 | |
| *** neilus has joined #openstack-ansible | 10:28 | |
| *** weezS has quit IRC | 10:28 | |
| *** fawadkhaliq has quit IRC | 10:39 | |
| *** pacoblox has joined #openstack-ansible | 10:49 | |
| *** thorst has joined #openstack-ansible | 10:54 | |
| *** pacoblox has quit IRC | 10:56 | |
| *** sacharya has joined #openstack-ansible | 10:58 | |
| *** markvoelker has joined #openstack-ansible | 10:59 | |
| *** thorst has quit IRC | 11:02 | |
| *** sacharya has quit IRC | 11:02 | |
| *** markvoelker has quit IRC | 11:03 | |
| *** wadeholler has joined #openstack-ansible | 11:05 | |
| *** bjornar has quit IRC | 11:17 | |
| *** fawadkhaliq has joined #openstack-ansible | 11:20 | |
| *** bjornar has joined #openstack-ansible | 11:21 | |
| *** weshay_afk is now known as weshay | 11:26 | |
| admin0 | who from here will be in barcelona ? | 11:27 |
| admin0 | in the summit ? | 11:27 |
| *** wadeholler has quit IRC | 11:39 | |
| *** neilus1 has joined #openstack-ansible | 11:39 | |
| *** wadeholler has joined #openstack-ansible | 11:39 | |
| *** neilus has quit IRC | 11:40 | |
| *** deverter has joined #openstack-ansible | 11:43 | |
| *** agireud has quit IRC | 11:45 | |
| *** neilus1 has quit IRC | 11:47 | |
| *** thorst has joined #openstack-ansible | 11:50 | |
| *** thorst_ has joined #openstack-ansible | 11:51 | |
| *** javeriak has quit IRC | 11:54 | |
| *** thorst has quit IRC | 11:55 | |
| *** asettle has quit IRC | 11:59 | |
| *** admin0 has quit IRC | 11:59 | |
| *** wadeholler has quit IRC | 12:00 | |
| *** aernhart has quit IRC | 12:00 | |
| *** aernhart has joined #openstack-ansible | 12:01 | |
| *** aernhart has quit IRC | 12:02 | |
| *** aernhart has joined #openstack-ansible | 12:02 | |
| *** zerda2 has quit IRC | 12:03 | |
| *** asettle has joined #openstack-ansible | 12:03 | |
| *** neilus has joined #openstack-ansible | 12:04 | |
| *** GMAzrael has joined #openstack-ansible | 12:05 | |
| *** markvoelker has joined #openstack-ansible | 12:14 | |
| evrardjp | it's a little to early to tell I think | 12:15 |
| evrardjp | obviously odyssey4me will go to the Barcelona summit | 12:15 |
| *** agireud has joined #openstack-ansible | 12:16 | |
| kamsz | i'm getting too many open files errors in cinder - will changing ulimit at infra solve the issue? im not very familiar with lxc | 12:19 |
| *** wadeholler has joined #openstack-ansible | 12:20 | |
| *** admin0 has joined #openstack-ansible | 12:20 | |
| *** psilvad_ has joined #openstack-ansible | 12:22 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 12:30 | |
| *** fawadkhaliq has quit IRC | 12:31 | |
| *** fawadkhaliq has joined #openstack-ansible | 12:32 | |
| *** fawadkhaliq has quit IRC | 12:37 | |
| *** Zucan has joined #openstack-ansible | 12:38 | |
| *** bootsha has quit IRC | 12:41 | |
| *** kylek3h has joined #openstack-ansible | 12:46 | |
| *** woodard has joined #openstack-ansible | 12:52 | |
| *** woodard has quit IRC | 12:52 | |
| *** woodard has joined #openstack-ansible | 12:53 | |
| evrardjp | cinder volumes are on metal by default IIRC | 12:56 |
| evrardjp | nothing linked to lxc | 12:56 |
| evrardjp | at least IIRC | 12:56 |
| *** smatzek has joined #openstack-ansible | 12:57 | |
| admin0 | yep | 12:59 |
| *** klamath has joined #openstack-ansible | 13:01 | |
| *** klamath has quit IRC | 13:01 | |
| *** klamath has joined #openstack-ansible | 13:02 | |
| automagically | Morning all | 13:02 |
| automagically | admin0: I’m hoping to be in Barcelona | 13:04 |
| asettle | o/ morn | 13:04 |
| admin0 | morning all | 13:04 |
| admin0 | oh wait .. afternoon here :D | 13:04 |
| automagically | o/ asettle | 13:04 |
| *** wadeholler has quit IRC | 13:05 | |
| admin0 | i am hoping for that as well :) | 13:06 |
| admin0 | hopefully :) | 13:06 |
| asettle | +1 to that | 13:06 |
| asettle | Since I sadly missed Austin :( | 13:06 |
| automagically | You did make a brief virtual appearance IIRC | 13:06 |
| asettle | I sure did. In my gym gear. Hahaha. Winnerrrrr | 13:06 |
| asettle | That was a happy coincidence. | 13:07 |
| admin0 | :) | 13:07 |
| *** messy has joined #openstack-ansible | 13:07 | |
| *** fawadkhaliq has joined #openstack-ansible | 13:09 | |
| asettle | You guys going to the midcycle? | 13:09 |
| *** sdake has joined #openstack-ansible | 13:10 | |
| automagically | I just booked my travel to SAT this weekend. | 13:10 |
| automagically | Never been, so looking forward to it. | 13:10 |
| asettle | Awesome :) | 13:10 |
| admin0 | what is a SAT :D ? | 13:11 |
| automagically | San Antonio, TX | 13:11 |
| asettle | San Antonio :) | 13:11 |
| admin0 | :) | 13:11 |
| automagically | Its the airport code | 13:11 |
| admin0 | got it :) | 13:11 |
| *** fawadkhaliq has quit IRC | 13:12 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_aodh: Remove mongodb vars from the aodh func tests https://review.openstack.org/334070 | 13:13 |
| alextricity25 | Where can I find the mid-cycle details and notes? | 13:17 |
| asettle | alextricity25: do you mean details on *how* to go? Or details on what we're doing there? | 13:17 |
| *** pacoblox has joined #openstack-ansible | 13:17 | |
| automagically | https://etherpad.openstack.org/p/osa-midcycle-newton | 13:17 |
| asettle | What automagically said | 13:18 |
| alextricity25 | thanks :) | 13:18 |
| *** sdake_ has joined #openstack-ansible | 13:20 | |
| *** sdake has quit IRC | 13:21 | |
| *** ManojK has joined #openstack-ansible | 13:24 | |
| asettle | automagically: do you know if we have the midcycle tickets out yet and how much they'll be costing? | 13:25 |
| automagically | Tickets? | 13:25 |
| automagically | Probably a better question for mhayden | 13:25 |
| *** pacoblox has quit IRC | 13:26 | |
| automagically | Who has taken point on organizing it | 13:26 |
| asettle | Yeah, where is he? He's disappeareddd! | 13:26 |
| asettle | I'll ask one of the guys :) cheers | 13:26 |
| messy | mhayden is attending the Red Hat summit in San Francisco this week | 13:27 |
| evrardjp | hey automagically | 13:28 |
| automagically | o/ evrardjp | 13:28 |
| admin0 | there are more “software” summits now then “real” summits in Nepal :D | 13:28 |
| evrardjp | he is on holiday this week I think | 13:29 |
| evrardjp | talking about mhayden | 13:29 |
| evrardjp | or off at least | 13:29 |
| automagically | Ah yes, didn’t realize | 13:29 |
| evrardjp | how come? | 13:29 |
| evrardjp | :p | 13:29 |
| asettle | How dare he do holiday when I just come back from holiday | 13:29 |
| asettle | messy you know things | 13:29 |
| evrardjp | oh yes true, | 13:30 |
| messy | asettle: just a few | 13:30 |
| evrardjp | the RH summit | 13:30 |
| evrardjp | he has a presentation there, right ? | 13:30 |
| admin0 | for someone from outside UK, so this referendum is not binding, and needs a vote from parliament (where majority want to vote no) .. if they vote yes, they vote against wishes of scotland which can then veto or ask another referendum to break away from UK .. so if they proceed, there will be 2 breakups ? sounds complicated | 13:31 |
| asettle | messy you doing work things these days rather than letting the evil mhayden push you around? :p | 13:31 |
| asettle | admin0: hmm yeah, it is complicated. | 13:31 |
| *** weshay is now known as weshay_mtg | 13:31 | |
| asettle | There's also the point that the conservatives promised this referendum as a part of their election promise... which is, a bit tricky in itself. They don't follow through, they lose the vote, and eventually lose their seat. | 13:31 |
| evrardjp | I'll skip the polics talk for today | 13:31 |
| asettle | It's a bit topsy turvy here at the mo. | 13:31 |
| *** wadeholler has joined #openstack-ansible | 13:31 | |
| asettle | I'm so glad I just moved to the UK *rolls eyes* | 13:32 |
| * admin0 goes back to reading theguardian :D | 13:32 | |
| asettle | admin0: good luck. That ain't happy reading :( | 13:32 |
| asettle | Things are not nice here at the mo. | 13:32 |
| admin0 | anyone from poland here ? | 13:32 |
| *** wadeholler has quit IRC | 13:32 | |
| asettle | admin0: in this chat, or in this office | 13:32 |
| asettle | ? * | 13:32 |
| admin0 | in this chat :) or office :) | 13:33 |
| *** wadeholler has joined #openstack-ansible | 13:33 | |
| evrardjp | you want to remember them they lost against switzerland? | 13:33 |
| admin0 | during the weekend, i was trying to dig through on the “why” .. you know like a incident root cause analysis and it led me to the migration from poland :) | 13:33 |
| asettle | I don't know of anyone in our dev group, wider office, definitely. | 13:33 |
| asettle | Yeahhhh there's a lot of concerning bigotry towards the Polish at the moment. | 13:34 |
| asettle | I'm trying to figure that out myself, tbh. | 13:34 |
| asettle | I only moved over 2 months ago. | 13:34 |
| admin0 | i am still waiting for my residence permit to actually apply for a visitor visa to the UK | 13:34 |
| asettle | admin0: where are you based? | 13:34 |
| evrardjp | oh also, if the OSA cores have time to look at bugs before tomorrow's meeting, it could be cool | 13:35 |
| admin0 | Netherlands | 13:35 |
| evrardjp | or not even cores, anyone | 13:35 |
| evrardjp | https://etherpad.openstack.org/p/osa-bugtriage | 13:35 |
| automagically | evrardjp: I’m about to open one I think, so I’ll review what’s there when I do. | 13:35 |
| asettle | admin0: ahhh gotcha. If you have an EU passport, you should be good to go? But I don't know much about other passports, to be completely honest. Australian passport is pretty much a free reign. | 13:35 |
| admin0 | evrardjp: i see one or two that i canvalidate/test .. especially on documentation part | 13:36 |
| admin0 | asettle: still on my Nepal passport :) | 13:36 |
| evrardjp | cool thanks admin0 | 13:36 |
| admin0 | need 1 more year to be eligible to apply for the dutch passport | 13:37 |
| evrardjp | cool automagically | 13:37 |
| asettle | admin0: ahhhh is there a lot of restrictions on that? Oh a dutch passport would be ace. | 13:37 |
| admin0 | Nepali passport :) yes | 13:37 |
| evrardjp | I'll update this list tomorrow anyway, but we've got quite a long list this week | 13:37 |
| admin0 | which is why i live in the hague .. home to 150+ embassies | 13:37 |
| evrardjp | so I'd like to be fully prepared | 13:37 |
| admin0 | so that applying for visa is easier :) | 13:37 |
| admin0 | no long drives :) just a walk/cycle away | 13:37 |
| asettle | admin0: sounds amazing. How long have you been living in the hague? | 13:38 |
| admin0 | asettle: do you also work for rackspace ? | 13:39 |
| asettle | admin0: like everyone else ;) | 13:39 |
| admin0 | 3 years 9 months | 13:39 |
| asettle | I was WFH AU, and just transferred to the UK. Which is why I suddenly now appear in chat :p | 13:39 |
| *** michaelgugino has joined #openstack-ansible | 13:39 | |
| admin0 | need to be here 5 years legally without i think €840 in legal fine over the last 5 years and pass a dutch exam to be eligible for citizenship | 13:40 |
| kamsz | evrardjp: i've cinder-volume container, should i set somewhere to make it deploy cinder-volume on metal if im using ceph? | 13:41 |
| asettle | admin0: that's not too bad! But damn, you must be walking on eggshells a bit :p | 13:41 |
| admin0 | me, Nepal => Singapore => Netherlands ( i think not moving further to anywhere else ) | 13:41 |
| kamsz | och, yeah | 13:41 |
| kamsz | my bad then | 13:41 |
| asettle | Oh very cool :) how'd you find singapore? | 13:41 |
| *** Ashana has joined #openstack-ansible | 13:41 | |
| admin0 | kamsz: if you use ceph, make cinder in container | 13:41 |
| admin0 | so need for bare metal | 13:41 |
| admin0 | its just the APIs | 13:41 |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Flush memcached on first listen IP only https://review.openstack.org/334484 | 13:42 |
| admin0 | well, singapore was good for the first 6 months — done everything twice .. but 5 years was bit too long and too limited | 13:42 |
| admin0 | limited by the size of the country :) | 13:42 |
| evrardjp | kamsz: if you are using ceph you can use cinder in a container | 13:42 |
| kamsz | then i'm hitting ulimit issue | 13:42 |
| asettle | admin0: yeah I visited and honestly? didn't find it particularly enchanting. But I have had plenty of people say how much they *love* living there. | 13:42 |
| admin0 | but allowed me to go to thailand, hongkong, philippines, malayasia, indoneasia, cambodia etc :) | 13:43 |
| admin0 | singapore is good for a visit and maybe stay for 6 months :) after that, things start to get a bit boring | 13:43 |
| admin0 | my cupboard were only t-shirts, flipflops and shorts .. | 13:43 |
| admin0 | nothing else | 13:43 |
| admin0 | even the weather is predictable and the same | 13:44 |
| admin0 | here in netherlands, its supposed to be summer and feels like winter .. no sun for days . :) | 13:44 |
| michaelgugino | hi all | 13:44 |
| admin0 | its a small country .. fun initially, but starts to get borin | 13:44 |
| admin0 | hello michaelgugino | 13:44 |
| *** _d34dh0r53_ is now known as d34dh0r53 | 13:45 | |
| openstackgerrit | Logan V proposed openstack/openstack-ansible: Flush memcached on first listen IP only https://review.openstack.org/334487 | 13:45 |
| evrardjp | hi michaelgugino | 13:46 |
| *** bsv has quit IRC | 13:46 | |
| automagically | o/ michaelgugino | 13:46 |
| admin0 | what * as a service ( apart from that provided by neutron) is in the roadmap for ansible ? | 13:48 |
| admin0 | database, dns, containers, filesystem ? | 13:48 |
| admin0 | and if anyone is using it now, how are they doing it ? run ansible playbooks first, then run additional playbooks > | 13:49 |
| *** KLevenstein has joined #openstack-ansible | 13:50 | |
| *** ManojK has quit IRC | 13:54 | |
| *** ManojK has joined #openstack-ansible | 13:55 | |
| *** bootsha has joined #openstack-ansible | 13:56 | |
| *** M00nr41n has quit IRC | 13:56 | |
| *** ametts has joined #openstack-ansible | 14:00 | |
| evrardjp | not sure to understand the question admin0 | 14:00 |
| evrardjp | ansible is still working on new modules for managing/using openstack if that's what you meant | 14:01 |
| admin0 | it was more of “what more is out there that I can do” with the openstack ansible | 14:01 |
| admin0 | https://www.openstack.org/software/project-navigator/ — on top of core services, ansible is also doing horizon, heat, ceilometer, ironic | 14:02 |
| admin0 | so manlia looks mature enough .. someone workign to integrate it ? | 14:02 |
| admin0 | or can i work to integrate it ? | 14:02 |
| admin0 | so just trying to look ahead | 14:02 |
| automagically | admin0: I’m sure folks would appreciate a Manila role | 14:03 |
| evrardjp | I didn't hear anyone looking at manila | 14:03 |
| automagically | I’m not aware of anyone working on it | 14:03 |
| admin0 | i see it “green” :) | 14:03 |
| admin0 | in the maturity list | 14:03 |
| bkreitch | actually i was thinking on checking manila too:) | 14:04 |
| automagically | Newest roles seem to be rally, cloudkitty, magnum and ironic | 14:04 |
| evrardjp | designate too, more or less | 14:04 |
| *** jayc has joined #openstack-ansible | 14:04 | |
| evrardjp | I'd say plus less than more | 14:04 |
| *** mfisch has joined #openstack-ansible | 14:06 | |
| *** mfisch has quit IRC | 14:06 | |
| *** mfisch has joined #openstack-ansible | 14:06 | |
| *** jthorne has joined #openstack-ansible | 14:12 | |
| admin0 | bkreitch: you have not started it yet ? | 14:12 |
| *** bootsha has quit IRC | 14:13 | |
| bkreitch | admin0: not yet, just was looking if someone is doing it already | 14:16 |
| admin0 | me too :) | 14:16 |
| admin0 | whats your plan of approach in integrating this ? | 14:16 |
| cloudnull | mornings | 14:18 |
| admin0 | mornings | 14:18 |
| bkreitch | wanted to borrow something from tripleo manila patch | 14:18 |
| asettle | o/ | 14:18 |
| admin0 | my approach is to install and test manilla over a multi node install, then note down the changes required to do and start with ansible | 14:18 |
| *** jmckind has joined #openstack-ansible | 14:18 | |
| automagically | Makes good sense admin0 | 14:20 |
| automagically | o/ cloudnull | 14:20 |
| cloudnull | hows it automagically? | 14:20 |
| automagically | it goes | 14:21 |
| cloudnull | i guess thats better than "not" :) | 14:22 |
| *** spotz_zzz is now known as spotz | 14:22 | |
| asettle | cloudnull: how was your hols?! | 14:26 |
| asettle | I saw Llama. | 14:26 |
| cloudnull | It was a good time | 14:26 |
| cloudnull | there were llamas, hiking, family. | 14:27 |
| cloudnull | generally good time | 14:27 |
| bkreitch | admin0: i'm just building aio first | 14:28 |
| admin0 | aha ..different approaches :) | 14:28 |
| admin0 | i never did aio .(might not do it ) :D | 14:29 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible: Fix HAProxy config and install version when ssl is disabled https://review.openstack.org/334506 | 14:29 |
| bkreitch | how do you do multi node install - physical servers, VMs? | 14:30 |
| evrardjp | hey cloudnull | 14:32 |
| cloudnull | bkreitch: if you have a physical node to play with you can build a multi-node env on that single node using something like https://github.com/cloudnull/osa-multi-node-aio | 14:32 |
| bkreitch | cloudnull: thx | 14:35 |
| *** jmckind_ has joined #openstack-ansible | 14:37 | |
| *** weshay_mtg is now known as weshay | 14:39 | |
| *** jthorne has quit IRC | 14:40 | |
| *** jmckind has quit IRC | 14:40 | |
| *** jthorne has joined #openstack-ansible | 14:41 | |
| *** thorst__ has joined #openstack-ansible | 14:41 | |
| evrardjp | in the future, I'll write a full ansible multi node deploy for devs, when I'll have time, so probably not before 2 week-ends | 14:43 |
| evrardjp | on clouds* | 14:43 |
| evrardjp | so a different use case than cloudnull's | 14:44 |
| evrardjp | at the moment cloudnull's multi node is quite the only way to automate it, the rest is build your own | 14:44 |
| *** thorst_ has quit IRC | 14:45 | |
| *** Mudpuppy has joined #openstack-ansible | 14:45 | |
| cloudnull | evrardjp: the RPC crew has a bunch of heat templates to do this on an existing OpenStack cloud. | 14:45 |
| * cloudnull goes to look | 14:45 | |
| evrardjp | I know | 14:45 |
| *** jthorne has quit IRC | 14:45 | |
| evrardjp | but it's based on heat | 14:45 |
| cloudnull | https://github.com/rcbops/rpc-heat | 14:45 |
| evrardjp | I think the latest version is elsewhere | 14:45 |
| evrardjp | https://github.com/cloud-training/rpc-heat-ansible | 14:45 |
| evrardjp | but still | 14:46 |
| cloudnull | ah thats the more updated version | 14:46 |
| evrardjp | yup | 14:47 |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_server: Add CentOS 7 support https://review.openstack.org/332229 | 14:47 |
| evrardjp | you'll have to update your github stars :D | 14:47 |
| bkreitch | i'm having an issue with aio - ssh key doesn't seems to be copied into containers - is it in lxc_hosts role? | 14:48 |
| openstackgerrit | Merged openstack/openstack-ansible-security: Docs: Add developer guide for security role https://review.openstack.org/332992 | 14:49 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh: Remove mongodb vars from the aodh func tests https://review.openstack.org/334070 | 14:55 |
| *** Mudpuppy has quit IRC | 14:55 | |
| *** Mudpuppy has joined #openstack-ansible | 14:55 | |
| *** sacharya has joined #openstack-ansible | 14:57 | |
| *** javeriak has joined #openstack-ansible | 14:59 | |
| *** sdake has joined #openstack-ansible | 15:01 | |
| *** jthorne has joined #openstack-ansible | 15:02 | |
| *** aernhart has quit IRC | 15:02 | |
| *** sdake_ has quit IRC | 15:03 | |
| *** aernhart has joined #openstack-ansible | 15:03 | |
| *** weezS has joined #openstack-ansible | 15:03 | |
| *** krotscheck is now known as krotscheck_dcm | 15:05 | |
| *** TxGirlGeek has joined #openstack-ansible | 15:09 | |
| *** javeriak_ has joined #openstack-ansible | 15:09 | |
| *** javeriak has quit IRC | 15:09 | |
| *** TxGirlGeek has quit IRC | 15:09 | |
| *** TxGirlGeek has joined #openstack-ansible | 15:09 | |
| *** thorst__ is now known as thorst_ | 15:09 | |
| *** john51 has joined #openstack-ansible | 15:09 | |
| *** ManojK has quit IRC | 15:11 | |
| *** john51_ has quit IRC | 15:13 | |
| *** ManojK has joined #openstack-ansible | 15:18 | |
| *** TxGirlGeek has quit IRC | 15:18 | |
| *** TxGirlGeek has joined #openstack-ansible | 15:19 | |
| *** raddaoui has joined #openstack-ansible | 15:21 | |
| *** saneax is now known as saneax_AFK | 15:24 | |
| *** jorge_munoz has joined #openstack-ansible | 15:25 | |
| *** weshay is now known as weshay_brb | 15:25 | |
| *** BjoernT has joined #openstack-ansible | 15:26 | |
| michaelgugino | are the 16.04 nova, neutron services being run in a virtualenv? | 15:29 |
| *** daneyon has joined #openstack-ansible | 15:31 | |
| *** weshay_brb is now known as weshay | 15:31 | |
| automagically | I sure hope so | 15:31 |
| *** TxGirlGeek has quit IRC | 15:32 | |
| *** TxGirlGeek has joined #openstack-ansible | 15:33 | |
| *** daneyon_ has quit IRC | 15:34 | |
| *** TxGirlGeek has quit IRC | 15:36 | |
| *** TxGirlGeek has joined #openstack-ansible | 15:36 | |
| *** Drago has joined #openstack-ansible | 15:38 | |
| *** pcaruana has quit IRC | 15:38 | |
| michaelgugino | it doesn't really look like it based on the systemd service files. I don't see anything about activating a venv | 15:39 |
| *** M00nr41n has joined #openstack-ansible | 15:41 | |
| *** cloader89_wfh has joined #openstack-ansible | 15:45 | |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_swift: Implement Xenial Support https://review.openstack.org/334543 | 15:51 |
| *** BjoernT has quit IRC | 15:54 | |
| *** TxGirlGeek has quit IRC | 15:59 | |
| *** TxGirlGeek has joined #openstack-ansible | 15:59 | |
| *** TxGirlGeek has quit IRC | 16:01 | |
| *** TxGirlGeek has joined #openstack-ansible | 16:02 | |
| cloudnull | michaelgugino: yes. | 16:03 |
| *** admin0 has quit IRC | 16:03 | |
| cloudnull | the venv only needed to be activated in upstart init files. w/ the systemd variant we only need to call the binary from within the venv | 16:03 |
| *** Drago has quit IRC | 16:06 | |
| *** TxGirlGeek has quit IRC | 16:06 | |
| *** TxGirlGeek has joined #openstack-ansible | 16:07 | |
| *** sacharya has quit IRC | 16:09 | |
| mhayden | asettle: tickets for the midcycle? entry to the midcycle will be free as far as i know | 16:09 |
| asettle | Woah mhayden lives. Man it's been quiet today. | 16:10 |
| asettle | Gotcha, thanks :) | 16:10 |
| asettle | I just need to be able to tell the hoomans above me. | 16:10 |
| mhayden | asettle: i'm out in SF for the red hat summit | 16:10 |
| *** cloader89_wfh is now known as cloader89 | 16:10 | |
| asettle | mhayden: so I was informed thanks to messy | 16:11 |
| asettle | Having a blast? | 16:11 |
| *** eil397 has joined #openstack-ansible | 16:11 | |
| *** Drago has joined #openstack-ansible | 16:12 | |
| *** sacharya has joined #openstack-ansible | 16:12 | |
| mhayden | hah, it's just getting underway shortly... tired from running this morning! | 16:12 |
| mhayden | although running in SF is quite refreshing | 16:12 |
| *** BjoernT has joined #openstack-ansible | 16:13 | |
| *** BjoernT is now known as Bjoern_zZzZzZzZ | 16:13 | |
| cloudnull | bkreitch: did you get the aio going ? | 16:13 |
| cloudnull | generally my process for an AIO is to, make VM /w >=8GB of ram & 150GB storage, install git tmux vim, start tmux session, clone repo, cd to repo, run ./scripts/gate-check-commit.sh, go get coffee. | 16:15 |
| *** johnmilton has quit IRC | 16:15 | |
| alextricity25 | I like the coffee part the best | 16:16 |
| cloudnull | that script will setup the env like we do in the gate which in the RAX public cloud takes about 45 min. | 16:16 |
| cloudnull | alextricity25: ++ the coffee part is my favorite too . | 16:16 |
| openstackgerrit | Merged openstack/openstack-ansible: Fix config generated for rsyslog_client_log_files https://review.openstack.org/331546 | 16:16 |
| *** KLevenstein has quit IRC | 16:20 | |
| *** Bjoern_zZzZzZzZ is now known as BjoernT | 16:23 | |
| *** weezS has quit IRC | 16:25 | |
| *** weezS has joined #openstack-ansible | 16:28 | |
| *** TxGirlGeek has quit IRC | 16:34 | |
| *** jthorne has quit IRC | 16:39 | |
| *** jthorne has joined #openstack-ansible | 16:39 | |
| *** asettle has quit IRC | 16:44 | |
| *** jthorne has quit IRC | 16:44 | |
| *** KLevenstein has joined #openstack-ansible | 16:49 | |
| *** fawadkhaliq has joined #openstack-ansible | 16:59 | |
| *** ManojK has quit IRC | 17:05 | |
| *** weezS has quit IRC | 17:14 | |
| *** admin0 has joined #openstack-ansible | 17:15 | |
| BjoernT | Hey does anyone know a good reason why we only add the root device to the LVM filter ? Was it because of locks when using cinder lvm ? | 17:17 |
| *** messy_ has joined #openstack-ansible | 17:27 | |
| *** messy_ has quit IRC | 17:28 | |
| *** messy_ has joined #openstack-ansible | 17:29 | |
| *** messy has quit IRC | 17:30 | |
| *** messy has joined #openstack-ansible | 17:33 | |
| *** messy_ has quit IRC | 17:33 | |
| bkreitch | cloudnull: i think the problem was in 32-bit ubuntu, don't why still had it, but check can be added in supported os | 17:34 |
| *** admin0 has quit IRC | 17:37 | |
| *** TxGirlGeek has joined #openstack-ansible | 17:37 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-security: Docs: Fix rendering of :orphan: https://review.openstack.org/334586 | 17:42 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-security: Docs: Fix rendering of :orphan: https://review.openstack.org/334587 | 17:43 |
| spotz | geez mhayden again?:) | 17:47 |
| mhayden | spotz: backports! | 17:49 |
| *** psilvad has joined #openstack-ansible | 17:54 | |
| *** psilvad_ has quit IRC | 17:58 | |
| *** albertcard has joined #openstack-ansible | 17:58 | |
| *** ManojK has joined #openstack-ansible | 18:01 | |
| *** jthorne has joined #openstack-ansible | 18:03 | |
| *** jthorne_ has joined #openstack-ansible | 18:06 | |
| *** jthorne has quit IRC | 18:06 | |
| *** weezS has joined #openstack-ansible | 18:28 | |
| *** admin0 has joined #openstack-ansible | 18:30 | |
| *** admin0 has quit IRC | 18:33 | |
| *** jayc has quit IRC | 18:37 | |
| *** admin0 has joined #openstack-ansible | 18:41 | |
| *** pacoblox has joined #openstack-ansible | 18:43 | |
| *** pacoblox has quit IRC | 18:47 | |
| *** pacoblox has joined #openstack-ansible | 18:49 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon: Add enable_lb option back to horizon https://review.openstack.org/333317 | 18:50 |
| *** jayc has joined #openstack-ansible | 18:51 | |
| *** Mudpuppy has quit IRC | 18:51 | |
| *** Mudpuppy has joined #openstack-ansible | 18:52 | |
| *** pacoblox has quit IRC | 18:57 | |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_swift: Implement Xenial Support https://review.openstack.org/334543 | 18:58 |
| *** messy has quit IRC | 19:01 | |
| *** weezS_ has joined #openstack-ansible | 19:01 | |
| *** weezS has quit IRC | 19:02 | |
| *** weezS_ is now known as weezS | 19:02 | |
| *** sacharya_ has joined #openstack-ansible | 19:04 | |
| *** sacharya has quit IRC | 19:07 | |
| *** pacoblox has joined #openstack-ansible | 19:07 | |
| *** messy has joined #openstack-ansible | 19:09 | |
| *** pacoblox has quit IRC | 19:09 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Dynamic Neutron OVS agent tunnel_types and local_ip config https://review.openstack.org/332957 | 19:11 |
| *** berendt has joined #openstack-ansible | 19:20 | |
| *** javeriak_ has quit IRC | 19:21 | |
| *** sacharya has joined #openstack-ansible | 19:24 | |
| *** sacharya_ has quit IRC | 19:28 | |
| openstackgerrit | Merged openstack/openstack-ansible: Confirm container destroys https://review.openstack.org/334199 | 19:28 |
| *** Zucan has quit IRC | 19:33 | |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_neutron: Add missing xenial to meta.yml https://review.openstack.org/334631 | 19:34 |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-os_nova: Add missing xenial to meta.yml https://review.openstack.org/334633 | 19:35 |
| openstackgerrit | Michael Gugino proposed openstack/openstack-ansible-openstack_hosts: Add missing xenial to meta.yml https://review.openstack.org/334635 | 19:38 |
| *** sdake has quit IRC | 19:38 | |
| *** sdake has joined #openstack-ansible | 19:40 | |
| *** jayc has quit IRC | 19:41 | |
| *** admin0 has quit IRC | 19:42 | |
| *** admin0 has joined #openstack-ansible | 19:44 | |
| *** fawadkhaliq has quit IRC | 19:51 | |
| *** fawadkhaliq has joined #openstack-ansible | 19:51 | |
| *** jayc has joined #openstack-ansible | 19:53 | |
| *** fawadkhaliq has quit IRC | 19:56 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible: Remove _net_address_search from dynamic inventory https://review.openstack.org/334639 | 19:56 |
| *** jthorne_ has quit IRC | 19:57 | |
| *** jthorne has joined #openstack-ansible | 19:58 | |
| *** ManojK has quit IRC | 19:59 | |
| *** ManojK has joined #openstack-ansible | 20:01 | |
| *** jthorne has quit IRC | 20:02 | |
| *** TxGirlGeek has quit IRC | 20:03 | |
| *** weezS has quit IRC | 20:03 | |
| *** weezS has joined #openstack-ansible | 20:05 | |
| *** cloader89 has quit IRC | 20:07 | |
| *** alan_ has joined #openstack-ansible | 20:17 | |
| *** alan_ is now known as Guest52772 | 20:17 | |
| *** sdake_ has joined #openstack-ansible | 20:19 | |
| *** aernhart has quit IRC | 20:19 | |
| *** sdake has quit IRC | 20:22 | |
| *** jthorne has joined #openstack-ansible | 20:28 | |
| *** jthorne has quit IRC | 20:28 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Add missing xenial to meta.yml https://review.openstack.org/334631 | 20:29 |
| *** jthorne has joined #openstack-ansible | 20:29 | |
| *** smatzek has quit IRC | 20:30 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-specs: Add spec for only supporting venv installs https://review.openstack.org/334649 | 20:30 |
| *** appprod0 has joined #openstack-ansible | 20:32 | |
| *** jmckind_ has quit IRC | 20:35 | |
| *** smatzek has joined #openstack-ansible | 20:35 | |
| appprod0 | hello. anyone know if the horizon lbaas v2 panels work in liberty? one doc says yes, other says no. testing it out i don't see the panel in horizon after enabling it | 20:35 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova: Add missing xenial to meta.yml https://review.openstack.org/334633 | 20:38 |
| *** fawadkhaliq has joined #openstack-ansible | 20:39 | |
| *** fawadkhaliq has quit IRC | 20:39 | |
| alextricity25 | appprod0: It should work. We had doc patches merge last week that detailed how to add the lbaas v2 horizon panel | 20:40 |
| openstackgerrit | Amy Marrich (spotz) proposed openstack/openstack-ansible-security: Docs: Fix rendering of :orphan: https://review.openstack.org/334587 | 20:40 |
| alextricity25 | Did you follow the steps exactly? Did you re-run the neutron and horizon playbooks after enabling lbaas v2 and the horizon panel | 20:40 |
| appprod0 | i set horizon_enable_neutron_lbaas: True and reran os-horizon-install.yml | 20:40 |
| *** fawadkhaliq has joined #openstack-ansible | 20:40 | |
| appprod0 | the lbaas agent is running, just seems like horizon isn't showing the panel. there should be a LBaaS button in horizon afterwards right? | 20:42 |
| *** asettle has joined #openstack-ansible | 20:42 | |
| openstackgerrit | Amy Marrich (spotz) proposed openstack/openstack-ansible-security: Docs: Fix rendering of :orphan: https://review.openstack.org/334586 | 20:42 |
| alextricity25 | appprod0: I'm not sure to be honest. i've never tried this before. Maybe it's under the networking tab? | 20:43 |
| *** weezS has quit IRC | 20:43 | |
| alextricity25 | appprod0: mhayden should know about the horizon lbaas v2 panels. But he is out of town | 20:44 |
| alextricity25 | Maybe he'll pop in later | 20:44 |
| appprod0 | thanks alextricity25 | 20:45 |
| openstackgerrit | Merged openstack/openstack-ansible-openstack_hosts: Add missing xenial to meta.yml https://review.openstack.org/334635 | 20:45 |
| *** fawadkhaliq has quit IRC | 20:45 | |
| mhayden | appprod0: fix proposed for that -> https://review.openstack.org/#/c/333010/ | 20:45 |
| spotz | mhayden: No more :orphan:s! | 20:46 |
| *** pacoblox has joined #openstack-ansible | 20:46 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible: Remove _net_address_search from dynamic inventory https://review.openstack.org/334639 | 20:46 |
| appprod0 | cool, thanks mhayden | 20:47 |
| *** Drago has quit IRC | 20:48 | |
| *** sdake has joined #openstack-ansible | 20:49 | |
| *** timrc has quit IRC | 20:49 | |
| *** sdake_ has quit IRC | 20:53 | |
| *** timrc has joined #openstack-ansible | 20:53 | |
| *** berendt has quit IRC | 20:53 | |
| mhayden | spotz: yay! | 20:54 |
| *** catintheroof has joined #openstack-ansible | 20:55 | |
| spotz | mhayden: Going through those is a full time job | 20:55 |
| *** smatzek has quit IRC | 20:58 | |
| *** thorst_ has quit IRC | 20:58 | |
| *** asettle has quit IRC | 20:58 | |
| spotz | mhayden look like mitaka failed but the same changes passed liberty | 21:00 |
| *** pacoblox has quit IRC | 21:01 | |
| *** fawadkhaliq has joined #openstack-ansible | 21:01 | |
| *** Mudpuppy has quit IRC | 21:05 | |
| *** woodard has quit IRC | 21:06 | |
| *** TxGirlGeek has joined #openstack-ansible | 21:08 | |
| *** jmckind has joined #openstack-ansible | 21:11 | |
| *** sdake_ has joined #openstack-ansible | 21:12 | |
| *** Guest52772 is now known as aernhart | 21:12 | |
| *** jmckind has quit IRC | 21:12 | |
| *** sdake has quit IRC | 21:13 | |
| *** sdake has joined #openstack-ansible | 21:16 | |
| *** sdake_ has quit IRC | 21:17 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible: Remove doc and example config references to host_bind_override https://review.openstack.org/334670 | 21:19 |
| *** asettle has joined #openstack-ansible | 21:24 | |
| *** fawadkhaliq has quit IRC | 21:24 | |
| *** asettle has quit IRC | 21:26 | |
| *** aernhart has quit IRC | 21:26 | |
| *** aernhart has joined #openstack-ansible | 21:26 | |
| *** aernhart has quit IRC | 21:27 | |
| *** aernhart has joined #openstack-ansible | 21:28 | |
| *** Drago has joined #openstack-ansible | 21:34 | |
| *** thorst_ has joined #openstack-ansible | 21:34 | |
| *** permalac_ has joined #openstack-ansible | 21:35 | |
| *** permalac has quit IRC | 21:36 | |
| *** smatzek has joined #openstack-ansible | 21:37 | |
| *** thorst_ has quit IRC | 21:38 | |
| *** johnmilton has joined #openstack-ansible | 21:39 | |
| *** fawadkhaliq has joined #openstack-ansible | 21:39 | |
| *** fawadkhaliq has quit IRC | 21:43 | |
| *** smatzek has quit IRC | 21:43 | |
| *** johnmilton has quit IRC | 21:45 | |
| *** fawadkhaliq has joined #openstack-ansible | 21:45 | |
| *** fawadkhaliq has quit IRC | 21:49 | |
| *** kylek3h has quit IRC | 21:52 | |
| *** michaelgugino has quit IRC | 21:54 | |
| *** catintheroof has quit IRC | 21:55 | |
| *** messy has quit IRC | 21:56 | |
| *** johnmilton has joined #openstack-ansible | 21:58 | |
| *** ManojK has quit IRC | 22:00 | |
| *** ManojK has joined #openstack-ansible | 22:01 | |
| *** KLevenstein has quit IRC | 22:07 | |
| *** fawadkhaliq has joined #openstack-ansible | 22:10 | |
| *** deverter_ has joined #openstack-ansible | 22:12 | |
| *** oneswig has quit IRC | 22:13 | |
| *** fawadkhaliq has quit IRC | 22:14 | |
| *** deverter has quit IRC | 22:14 | |
| *** raddaoui has quit IRC | 22:17 | |
| *** neilus has quit IRC | 22:21 | |
| *** jamielennox is now known as jamielennox|away | 22:21 | |
| *** weezS has joined #openstack-ansible | 22:25 | |
| *** weezS has quit IRC | 22:27 | |
| *** ametts has quit IRC | 22:28 | |
| *** neilus has joined #openstack-ansible | 22:29 | |
| *** Ashana has quit IRC | 22:29 | |
| *** Drago has quit IRC | 22:30 | |
| *** ManojK has quit IRC | 22:30 | |
| *** admin0 has quit IRC | 22:33 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible: Fail Fast when trying to upgrade with LBaaS v1 enabled https://review.openstack.org/324728 | 22:35 |
| *** TxGirlGeek has quit IRC | 22:35 | |
| *** spotz is now known as spotz_zzz | 22:39 | |
| *** ametts has joined #openstack-ansible | 22:41 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible: Fail Fast when trying to upgrade with LBaaS v1 enabled https://review.openstack.org/324728 | 22:43 |
| *** BjoernT has quit IRC | 22:43 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-openstack_hosts: Added the ip_vs kernel module to all openstack hosts https://review.openstack.org/334698 | 22:45 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Added the ip_vs kernel module to all openstack hosts https://review.openstack.org/334699 | 22:46 |
| *** psilvad has quit IRC | 22:47 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Added the ip_vs kernel module to all openstack hosts https://review.openstack.org/334699 | 22:47 |
| *** ametts has quit IRC | 22:48 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-openstack_hosts: Added the ip_vs kernel module to all openstack hosts https://review.openstack.org/334701 | 22:50 |
| *** ManojK has joined #openstack-ansible | 22:57 | |
| *** jayc has quit IRC | 23:03 | |
| *** evilt0ne has quit IRC | 23:08 | |
| *** thetrav has joined #openstack-ansible | 23:19 | |
| thetrav | so, still having toruble pinning down my problem with too many open files | 23:20 |
| thetrav | it appears that security hardening may be to blame | 23:20 |
| thetrav | before starting the security-hardening.yml play, there are about 100 open files | 23:20 |
| thetrav | before starting the lx-hosts-setup.yml file, there are about 900 | 23:21 |
| thetrav | ansible bombs out at 1024 due to limitations of the OS select function | 23:21 |
| thetrav | I will double check, but I think I've left security hardening to its default value (False) so the fact taht it's hopening files at all is odd | 23:22 |
| thetrav | oh, the other thing to note is that the majority of file handles are to /dev/null which implies to me that it's some sort of "run a script and trash the stdout or stderr stream" that doesn't close afterwards | 23:22 |
| openstackgerrit | Merged openstack/openstack-ansible-lxc_hosts: Implement base container for overlayfs-backed containers https://review.openstack.org/329029 | 23:23 |
| *** raddaoui has joined #openstack-ansible | 23:23 | |
| cloudnull | thetrav: sorry to ask as you've likely been asked this before, but there is it bombing ? | 23:23 |
| cloudnull | s/there/where/g | 23:24 |
| thetrav | where specifically does it give me an error? trying to install rsyslog packages | 23:24 |
| thetrav | I've spent a long time investigating though and believe that's a red herring | 23:24 |
| thetrav | rsyslog packages only opens a handfull of files | 23:24 |
| cloudnull | env on a physical host w/ containers on the root fs or in LVs | 23:25 |
| thetrav | the problem seems to be that it doesn't have any headroom left after the initial 900 ish files are open | 23:25 |
| thetrav | oh, what's my install setup? | 23:25 |
| thetrav | 4 bare metal target hosts 1 virtual machine as the deployment host | 23:25 |
| thetrav | all running ubuntu 14.04 | 23:25 |
| cloudnull | ok . | 23:25 |
| thetrav | the vm is hosted on an existing openstack, kilo, using kvm and gre tunnels | 23:25 |
| cloudnull | cool | 23:25 |
| thetrav | I'm 90% certain the error occurs on the deploy_host not on the target machines | 23:26 |
| thetrav | I've been debugging by dumping lsof at strategic parts in the playbooks | 23:26 |
| thetrav | effectively a bisect to locate the point at which things go weird | 23:26 |
| cloudnull | what version of ansible ? | 23:27 |
| thetrav | 1.9.4 | 23:27 |
| cloudnull | deploying mitaka or liberty ? | 23:27 |
| thetrav | 2277aa8cc19140d9313fa6f3363db966312ffdc3 which was stable/mitaka when I started | 23:27 |
| cloudnull | ok. | 23:28 |
| thetrav | yesterday that was one commit behind, however that commit did not have anything relevant | 23:28 |
| cloudnull | sorry was just catching up | 23:28 |
| thetrav | no worries | 23:28 |
| thetrav | incredibly pleased for someone to be paying attention :D | 23:28 |
| thetrav | oh | 23:28 |
| thetrav | the other thing worth mentioning | 23:28 |
| thetrav | initially I had an error that I sought to fix by raising the allowed open files | 23:28 |
| thetrav | I forget specifically waht it was | 23:29 |
| thetrav | but editing /etc/security/limits.conf pushed past it | 23:29 |
| cloudnull | so setting ulimit -Hn doesn't help ? | 23:29 |
| *** bjornar has quit IRC | 23:29 | |
| thetrav | yeah that's what /etc/security/limits.conf does | 23:29 |
| thetrav | so it lets you get a tiny bit further | 23:29 |
| thetrav | but there's some sort of underlying function called select that has a hard coded limit of 1024 | 23:29 |
| thetrav | actually hard coded may be the wrong word | 23:30 |
| thetrav | It's a limit that I have not been able to figure out how to increase | 23:30 |
| *** fawadkhaliq has joined #openstack-ansible | 23:30 | |
| cloudnull | I cant remember exactly but I do remember seeing issues similar to what your describing when setting the limits.conf for all contexts. | 23:30 |
| cloudnull | maybe not relevant . . . | 23:31 |
| cloudnull | laso fs.file-max in sysctl.conf may be able to be used to move that limit higher | 23:31 |
| thetrav | nope | 23:31 |
| thetrav | so I've adjusted that limit | 23:31 |
| thetrav | to something quite high | 23:31 |
| thetrav | https://github.com/pexpect/pexpect/issues/47 <- kind of what I'm talking about | 23:32 |
| thetrav | different project and issue | 23:32 |
| cloudnull | here are my current settings http://cdn.pasteraw.com/tk6p63qfxzvc0h58vjsb72ll1r1nunm | 23:32 |
| thetrav | but that's the limitation on select.select() | 23:32 |
| cloudnull | thats my deployment node on a ~500 node build | 23:32 |
| thetrav | yep | 23:33 |
| thetrav | so my deployment is only 4 nodes | 23:33 |
| cloudnull | w/ http://cdn.pasteraw.com/k77mrvema9cvwk2zoxx8gej7x1813nd | 23:33 |
| cloudnull | so it should work :) | 23:33 |
| thetrav | indeed it should | 23:33 |
| thetrav | and yet it is not | 23:33 |
| thetrav | my best guess is that it's enumerating a range somewhere, and I've configured the range to be quite large | 23:33 |
| cloudnull | that said, that build is running liberty /w-out the security role (it wasn't implemented at that time as the default). | 23:33 |
| thetrav | oh | 23:34 |
| thetrav | so security is turned on by default now? | 23:34 |
| cloudnull | so not quite apples to apples | 23:34 |
| cloudnull | it is | 23:34 |
| cloudnull | in mitaka | 23:34 |
| thetrav | ok, I must have been reading the wrong docs then | 23:34 |
| * cloudnull thought it was , goes to look | 23:34 | |
| *** fawadkhaliq has quit IRC | 23:35 | |
| cloudnull | do you happen to have a stacktrace saved somewhere when it decides to die ? | 23:35 |
| thetrav | I can generate one | 23:36 |
| cloudnull | if you wouldn't mind ? | 23:36 |
| thetrav | not at all, it'll take me a few minutes is all, just rebooting the node to re-set the file limits | 23:36 |
| thetrav | I'm actually invoking openstack-ansible using ansible, so it's reasonably easy to reproduce everything | 23:37 |
| cloudnull | https://github.com/ansible/ansible/issues/12259 -- ulimit related, but is it something like that TB ? | 23:37 |
| thetrav | yes | 23:37 |
| thetrav | that is the error message | 23:38 |
| thetrav | OSError: [Errno 24] Too many open files | 23:38 |
| thetrav | however the default limit in ubuntu is 1024 | 23:38 |
| thetrav | I confirmed that when I first came across that issue | 23:38 |
| thetrav | the other factor is that I'm not using ansible 2 | 23:39 |
| thetrav | I'm using the version of ansible that the bootstrap script installs | 23:39 |
| *** bjornar has joined #openstack-ansible | 23:39 | |
| cloudnull | ansible 2 or 2.1 ? | 23:39 |
| thetrav | neither | 23:39 |
| thetrav | you asked above | 23:39 |
| thetrav | 1.9.4 | 23:39 |
| cloudnull | sorry i read those last updates wrong. | 23:40 |
| cloudnull | "not" being the operitive word that i missed | 23:40 |
| *** sacharya has quit IRC | 23:40 | |
| cloudnull | is UsePAM set within your sshd_confi g? | 23:41 |
| cloudnull | maybe the pam_limits are not being loaded? | 23:42 |
| thetrav | yeah so, it's not 2 or 2.1 on my deploy host, the issue seems strongly related, however they're focussing on a newer version and osx as the deploy host, wheras I'm using ubuntu | 23:42 |
| thetrav | I thought sshd_config would only impact inbound ssh sessions | 23:43 |
| thetrav | this is outbound | 23:43 |
| thetrav | UsePAM is set to yes | 23:43 |
| thetrav | so the question is moot | 23:43 |
| cloudnull | hum... | 23:43 |
| *** jamielennox|away is now known as jamielennox | 23:43 | |
| cloudnull | was thinking maybe ansible calling openstack-ansible was getting "other" limits due to the connection . | 23:44 |
| cloudnull | but maybe not. | 23:44 |
| thetrav | ahh right | 23:46 |
| thetrav | no, I don't think so | 23:46 |
| *** aernhart has quit IRC | 23:46 | |
| thetrav | should have a stacktrace for you shortly | 23:46 |
| thetrav | then I'm going to have a go with security_hardening set to false | 23:46 |
| thetrav | http://cdn.pasteraw.com/8srl4ujpjinlni641pviz9hgrz1n107 | 23:47 |
| thetrav | that's one of the hosts, I have a matching exception for each host | 23:47 |
| *** aernhart has joined #openstack-ansible | 23:48 | |
| thetrav | TASK: [rsyslog_client | Write log rotate file] ******************************** | 23:48 |
| thetrav | that's the closest ansible output | 23:48 |
| cloudnull | could you remove these lines from the ansible.cfg file in your playbooks directory and try again ? https://github.com/openstack/openstack-ansible/blob/master/playbooks/ansible.cfg#L17-L19 | 23:48 |
| openstackgerrit | Darren Chan proposed openstack/openstack-ansible: [docs] Address tox errors https://review.openstack.org/334298 | 23:49 |
| cloudnull | looks like it may be spooling opening and closing cached facts | 23:49 |
| cloudnull | well in this case, not closing | 23:49 |
| thetrav | ok, I'll try once I've completed this run with security_hardening disabled (and will of course re-enable security hardening) | 23:50 |
| cloudnull | also I wonder if you have a bazillion files in the directory "/etc/openstack_deploy/ansible_facts" for some strange reason ? | 23:51 |
| cloudnull | if so you can nuke them and try again | 23:51 |
| cloudnull | all that is stored there are cached facts. | 23:51 |
| thetrav | only 5 files | 23:52 |
| thetrav | one per host | 23:52 |
| cloudnull | thats what it should be | 23:52 |
| cloudnull | so hum . | 23:52 |
| thetrav | ok, so setting that security_hardening variable to False did not solve anything | 23:53 |
| thetrav | still have too many files open | 23:53 |
| thetrav | hasn't crashed yet but I can see it in LSOF | 23:53 |
| thetrav | using lsof | grep '^ansible-p' | wc -l | 23:54 |
| thetrav | trying again with those lines you suggested commented out of ansible.cfg | 23:55 |
| thetrav | interestingly enough setting the security_hardening variable to False did not actually skip the role all together, it looked like it was stepping through everything and skipping individual tasks | 23:56 |
| thetrav | so if this doesn't work I may try butchering setup-hosts.yml and commenting out the include | 23:56 |
| cloudnull | yea, sadly thats ansible. | 23:56 |
| thetrav | that will at least tell me if it is security_hardening at fault or some metadata loading thinger | 23:56 |
| cloudnull | sadly ansible will not reverse state once a change has been made. so if you disable after its been enabled the change remains. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!