Thursday, 2015-10-01

*** palendae has joined #openstack-ansible00:07
*** kerwin_bai has joined #openstack-ansible00:14
*** abitha has quit IRC00:15
*** sdake has quit IRC00:23
*** itsuugo has quit IRC00:40
*** alextricity-mobi has joined #openstack-ansible01:08
*** spotz_zzz is now known as spotz01:20
openstackgerritMerged openstack/openstack-ansible: Allow pip extra arguments to be passed to playbooks  https://review.openstack.org/22938301:27
openstackgerritMerged openstack/openstack-ansible: Added DEBIAN_FRONTEND=noninteractive to bootstrap scripts  https://review.openstack.org/22878601:27
*** spotz is now known as spotz_zzz01:28
openstackgerritMerged openstack/openstack-ansible: Fixes log rotate issue  https://review.openstack.org/22795701:29
*** k_stev has joined #openstack-ansible02:13
*** k_stev has quit IRC02:13
*** markvoelker has quit IRC02:15
*** itsuugo has joined #openstack-ansible02:37
*** skamithi13 has quit IRC02:59
*** itsuugo has quit IRC03:09
*** skamithi13 has joined #openstack-ansible03:12
*** itsuugo has joined #openstack-ansible03:13
*** markvoelker has joined #openstack-ansible03:15
*** itsuugo has quit IRC03:18
*** markvoelker has quit IRC03:20
*** _hanhart has joined #openstack-ansible03:48
*** kerwin_bai has quit IRC03:51
*** sdake has joined #openstack-ansible04:00
*** galstrom_zzz is now known as galstrom04:02
*** _hanhart has quit IRC04:02
*** markvoelker has joined #openstack-ansible04:16
*** abitha has joined #openstack-ansible04:19
*** markvoelker has quit IRC04:20
*** sdake_ has joined #openstack-ansible04:24
*** sdake has quit IRC04:28
openstackgerritKevin Carter proposed openstack/openstack-ansible: Implement horizon venv support  https://review.openstack.org/22922604:35
prometheanfireand so it begins04:39
*** shausy has joined #openstack-ansible04:48
*** galstrom is now known as galstrom_zzz04:53
*** sdake_ is now known as sdake05:21
*** shausy has quit IRC05:35
*** shausy has joined #openstack-ansible05:36
*** skamithi14 has joined #openstack-ansible05:52
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Fixes log rotate issue  https://review.openstack.org/22946305:55
*** skamithi13 has quit IRC05:55
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Fixes log rotate issue  https://review.openstack.org/22946305:56
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Allow pip extra arguments to be passed to playbooks  https://review.openstack.org/22954205:58
*** shausy has quit IRC06:10
*** shausy has joined #openstack-ansible06:10
*** markvoelker has joined #openstack-ansible06:17
*** markvoelker has quit IRC06:21
*** abitha has quit IRC06:35
*** willemgf has joined #openstack-ansible06:47
*** Burgosz has joined #openstack-ansible07:16
*** Burgosz has quit IRC07:22
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Adjust default Keystone httpd processes and threads  https://review.openstack.org/22927507:23
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Updates ansible role requirements script name  https://review.openstack.org/22954007:26
*** openstackgerrit has quit IRC07:46
*** openstackgerrit has joined #openstack-ansible07:46
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update all SHA's (except Horizon) to Liberty RC1  https://review.openstack.org/22838507:49
*** gparaskevas has joined #openstack-ansible07:50
willemgfHi, In horizon -> Project -> Orchestration -> Stacks I get "Error: Unable to retrieve stack list." Also with CLI-command heat service list I have 120 services which have the status down. What could be wrong?07:55
evrardjpcloudnull: thanks for the rebase... The attention is more about whether we should distribute the ansible galaxy requirements files like that or not08:02
*** Burgosz has joined #openstack-ansible08:04
*** itsuugo has joined #openstack-ansible08:11
*** markvoelker has joined #openstack-ansible08:18
*** markvoelker has quit IRC08:22
odyssey4meevrardjp yeah, I was discussing the patch with cloudnull last night08:24
odyssey4meon the one hand it's pretty clean and works nicely - but on the other hand it puts every deployer into the position where that extra role is downloaded whether they want it or not08:25
odyssey4meas you know, I'm more in favor of having deployments more pluggable and that features or roles are added when desired, but only the core stuff is ever compulsory08:26
odyssey4mebut this is a pattern we need to figure out as a group - what is important to us08:26
evrardjpodyssey4me: indeed, that was my opinion too08:28
evrardjpmaybe worth a discussion today?08:28
*** gparaskevas_ has joined #openstack-ansible08:40
*** shausy has quit IRC08:40
*** gparaskevas has quit IRC08:42
odyssey4mefor sure08:43
odyssey4mecan you add it to the agenda?08:44
*** vdo has quit IRC09:01
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Limit the number of Ansible forks used to 10  https://review.openstack.org/22978609:16
evrardjpodyssey4me: doing it right now :)09:18
evrardjp(even it's maybe a little late...)09:18
*** shausy has joined #openstack-ansible09:19
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Limit the number of Ansible forks used to 10  https://review.openstack.org/22978609:21
odyssey4meandymccr mattt fyi - priority reviews for the next kilo tag: https://review.openstack.org/229275 / https://review.openstack.org/229542 / https://review.openstack.org/229463 / https://review.openstack.org/22898309:25
*** mgoddard_ has joined #openstack-ansible09:25
*** sdake_ has joined #openstack-ansible09:38
*** mgoddard_ has quit IRC09:40
*** mgoddard_ has joined #openstack-ansible09:40
*** sdake has quit IRC09:42
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Ensure that all pip wheels are reinstalled during major upgrades  https://review.openstack.org/22980309:53
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Ensure that all pip wheels are reinstalled during major upgrades  https://review.openstack.org/22980310:00
*** shausy has quit IRC10:15
*** markvoelker has joined #openstack-ansible10:19
tiagogomes_https://github.com/evrardjp/ansible-keepalived/blob/master/tasks/main.yml is missing a step to create the /etc/keepalived directory10:19
evrardjpthis should be created  by apt10:20
evrardjpI can ensure it's presence, but normally it should work by default10:21
evrardjpits*10:21
tiagogomes_ah, I see10:21
tiagogomes_I removed the directory without uninstalling keepalived. Another thing, on ansible-role-requirements.yml, the path shouldn't be `playbook/roles`?10:23
*** markvoelker has quit IRC10:23
tiagogomes_If you run it with the bootstrap-ansible, it will end in the wrong dir10:25
openstackgerritMerged openstack/openstack-ansible: Fixes log rotate issue  https://review.openstack.org/22946310:29
odyssey4metiagogomes_ no, it should push it into the ansible namespace - not into the code tree10:30
odyssey4meie it'll end up in /etc/ansible/10:30
tiagogomes_odyssey4me not on my system10:31
tiagogomes_I don't even have a /etc/ansible/10:31
tiagogomes_(using kilo branch)10:31
odyssey4metiagogomes_ where does it end up?10:36
openstackgerritMatt Thompson proposed openstack/openstack-ansible: Update ceph key hash  https://review.openstack.org/22889410:36
tiagogomes_on the repository rootdir, /opt/os-ansible/deployment/roles/keepalived10:36
odyssey4metiagogomes_ urgh, that sucks10:37
*** agireud has quit IRC10:48
openstackgerritMerged openstack/openstack-ansible: Add galera monitoring user during upgrades  https://review.openstack.org/22898310:51
openstackgerritMerged openstack/openstack-ansible: Allow pip extra arguments to be passed to playbooks  https://review.openstack.org/22954210:51
*** skamithi14 has quit IRC11:04
*** skamithi13 has joined #openstack-ansible11:04
*** itsuugo has quit IRC11:06
*** itsuugo has joined #openstack-ansible11:13
*** gparaskevas_ has quit IRC11:28
*** itsuugo has quit IRC11:31
openstackgerritMatt Thompson proposed openstack/openstack-ansible: Update ceph key hash  https://review.openstack.org/22889411:32
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Enable Horizon multidomain configuration  https://review.openstack.org/22983611:34
*** markvoelker has joined #openstack-ansible11:34
*** markvoelker has quit IRC11:39
*** itsuugo has joined #openstack-ansible11:51
mhaydenbuenos dias11:53
odyssey4meo/ mhayden11:55
*** skamithi14 has joined #openstack-ansible11:55
*** skamithi13 has quit IRC11:58
evrardjpodyssey4me: that's my fault12:01
*** markvoelker has joined #openstack-ansible12:01
evrardjpabout the roles/keepalived12:01
odyssey4meevrardjp oh?12:02
evrardjpbut it's WIP12:02
evrardjpyeah, the new keepalived role is in the ansible requirements.yml file12:02
evrardjpand I manually set up a path12:02
evrardjpnormally it's not needed12:02
evrardjpI set the path to roles/12:02
evrardjpI'll remove it12:03
odyssey4meperhaps bootstrap-ansible should ensure that /etc/ansible exists?12:03
evrardjpI don't think it's needed12:03
*** errr has quit IRC12:04
evrardjpI just think that roles have to be downloaded wherever the user decides to12:04
evrardjpby default it could be in playbooks/roles/12:04
evrardjpor following the ansible configuration12:04
odyssey4meevrardjp yeah, but it shouldn't go in the code tree - that's the point12:04
evrardjpoh that's what you mean?12:05
evrardjpthat's gonna be hard then12:05
odyssey4meby default ansible looks for roles in /etc/ansible as I recall?12:05
evrardjpbecause by default, it will download the galaxy role in your default role folder12:05
evrardjpok it will not be hard, because we didn't define it in ansible.cfg12:06
evrardjpI thought we did it12:06
*** errr has joined #openstack-ansible12:06
evrardjpodyssey4me: you're right12:06
evrardjpmore or less12:07
odyssey4meyeah, we don't need to because the first path it checks is the current directory, and all playbooks are executed from the playbooks directory which contains the roles12:07
mgariepygood morning everyone12:08
evrardjpI'll need to test more, but the path I set in the current [WIP] commit is wrong12:09
odyssey4medefault role locations: https://github.com/ansible/ansible/blob/v1.9.3-1/lib/ansible/constants.py#L10312:10
evrardjpyeah that's exactly what I thought :)12:11
odyssey4mewhich should mean that if you set no path, it'll download it to /etc/ansible/roles I would hope?12:11
evrardjpwe didn't define a role_path in ansible.cfg so yes12:11
evrardjpI think12:11
evrardjpI have to test ;)12:11
*** woodard has joined #openstack-ansible12:14
evrardjpyes, it's by default in /etc/ansible/roles12:16
*** KLevenstein has joined #openstack-ansible12:16
odyssey4meevrardjp as long as no path is set in the role information?12:16
evrardjpas long as there is no path in the requirements file12:17
evrardjpso if we want to put the role in the /opt/os-ansible-deployment/playbooks/roles, we'll have to set a path in the requirements file12:19
odyssey4meevrardjp yes, and we don't want to do that for any roles that are no in the repo12:19
odyssey4mein fact down the line as roles become more independent - the role path for everything should be wherever the deployer wants it to be12:20
evrardjpso that's something harder12:22
evrardjpat least harder to automate12:22
tiagogomes_so if I removed `path` from the requirements file all good?12:22
evrardjptiagogomes_: yes, or take the latest patchset from the commit12:22
tiagogomes_new patchet? \o/12:23
evrardjpodyssey4me: because ansible.cfg is in /opt/os-ansible-deployment/playbooks/, and that's where you'll need to put the custom role path (if the deployer decides to puts the role elsewhere)12:23
evrardjp(ofc the exception would be that there is another ansible.cfg consuming /opt/os-ansible-deployment, but that's another topic)12:24
evrardjpso it's really up to the deployer12:24
evrardjpbut he needs to understand ansible :p12:24
odyssey4meevrardjp yes, if you stretch your mind to a place where openstack-ansible is a collection of roles in their own repositories... and the openstack-ansible repository itself is simply a representation of playbooks which show how you could use the roles in various use-cases... then things would be quite different ;)12:25
evrardjpodyssey4me: it seems in that case more useful to have all the roles in the same folder as the playbooks12:26
evrardjpit's more obvious12:26
evrardjpeven if they are fetched from outside12:26
evrardjpbecause you'll have all your tools at your disposal in the "work folder"12:27
odyssey4meevrardjp yes, but in that case the location of the roles would be wherever the deployer choose to have them - as they'd all be pulled down using ansible-galaxy12:27
evrardjpbut that's only my personal preference ;)12:27
*** alejandrito has joined #openstack-ansible12:27
odyssey4mein other words, we would never need to set ansible.cfg in the repo12:27
*** cristicalin has joined #openstack-ansible12:28
odyssey4meansible.cfg could live in /etc/ansible/ and the deployer would control it12:28
*** pradk has joined #openstack-ansible12:28
odyssey4mebut I am talking way, way down the line here.... if it even happens12:28
evrardjpit has pros and cons ;)12:29
evrardjpbut in all cases, it needs to be properly documented12:30
odyssey4meas does everything :p12:30
evrardjpI guess we'll continue this discussion later today, because I'll need to know if I have to edit the ansible-requirements file directly, or let the user manage this12:30
evrardjpthere will be impact here I guess: http://docs.openstack.org/developer/openstack-ansible/install-guide/deploymenthost-osa.html12:31
*** skamithi13 has joined #openstack-ansible12:34
*** tlian has quit IRC12:36
*** skamithi14 has quit IRC12:37
*** mat425 has joined #openstack-ansible12:38
mat425hi guys, i'm installing devstack and monasca with the github repo monasca-installer using ansible. i've an issue, when i set the apt-get installer cmd, with with_items in a variable, it takes only the first argument, here is the pastebin, can you help me? http://pastebin.com/tk396pek12:43
*** tlian has joined #openstack-ansible12:47
openstackgerritMiguel Alejandro Cantu proposed openstack/openstack-ansible: Removes /opt/*openstack*/ pattern from teardown.  https://review.openstack.org/22855812:50
openstackgerritMiguel Alejandro Cantu proposed openstack/openstack-ansible: Adds insecure vars to configure-sslcertificate  https://review.openstack.org/22952512:57
*** jaypipes has joined #openstack-ansible12:58
*** k_stev has joined #openstack-ansible13:06
*** skamithi13 has quit IRC13:07
*** skamithi13 has joined #openstack-ansible13:08
cloudnullmat425: are you sure its only running the first item ? it seems like it should iterate through all items. what version of ansible are you using ? as an aside: you might want to add update_cache=yes in that command to make sure you have the most updated cache before running that apt commands13:12
cloudnullmorning btw o/13:12
cloudnullmat425:  additionally its worth noting that this channel is dedicated to the openstack-ansible project RE: https://github.com/openstack/openstack-ansible13:13
cloudnullhappy to help if we can, but you might want to reach out to the #ansible channel for general ansible help if thats more of what you're looking for.13:14
mat425cloudnull using ansible 1.9, i'm sure, cause i added a new debug task before, which print only the first entry of the list. know this is the wrong channel, but is quite related (i guess :D ) to my issue... i just though it was a common issue with fast solution, that's it13:15
*** skamithi has joined #openstack-ansible13:15
mat425i'm using that repo which is a wrapper of a bunch of different roles among github, is there the possibility that the var "dependencies" is not overwritten by the current role?13:16
cloudnullso the apt module should squash the list and run all the items ad the same time (since ansible 1.7 i think)13:16
cloudnullyou might want to add your deps into the role as a var or default . and I'd namespace the deps so you dont run into collisions.13:16
mat425how to do namespace that? just naming? like depencencies --> moduleX_dependencies13:17
*** skamithi has quit IRC13:17
cloudnullyes13:18
cloudnullwe follow that patter for all of our roles https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_cinder/defaults/main.yml#L19213:19
mat425ok thanks, using this fix :)13:21
*** skamithi has joined #openstack-ansible13:24
*** mnestheu1 has joined #openstack-ansible13:39
*** mnestheu1 is now known as scarlisle13:39
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Ensure that all pip wheels are reinstalled during major upgrades  https://review.openstack.org/22980313:45
*** cristicalin has quit IRC13:54
*** phalmos has joined #openstack-ansible13:58
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Cinder to Liberty RC1  https://review.openstack.org/22720514:05
*** Mudpuppy has joined #openstack-ansible14:05
*** Mudpuppy has quit IRC14:05
*** Mudpuppy has joined #openstack-ansible14:06
openstackgerritKevin Carter proposed openstack/openstack-ansible: Break apart and document the upgrade process  https://review.openstack.org/22413714:09
openstackgerritKevin Carter proposed openstack/openstack-ansible: Ensure that all pip wheels are reinstalled during major upgrades  https://review.openstack.org/22980314:09
cloudnullprometheanfire odyssey4me rebased off current head of kilo14:09
*** dolpher has joined #openstack-ansible14:10
cloudnulland added https://review.openstack.org/#/c/229803/3 as a dep14:10
*** Burgosz has quit IRC14:11
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Keystone to Liberty RC1  https://review.openstack.org/22691714:12
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update all SHA's (except Horizon) to Liberty RC1  https://review.openstack.org/22838514:12
*** sigmavirus24_awa is now known as sigmavirus2414:12
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Cinder to Liberty RC1  https://review.openstack.org/22720514:13
*** mat425 has quit IRC14:20
*** willemgf has quit IRC14:22
*** spotz_zzz is now known as spotz14:31
openstackgerritKevin Carter proposed openstack/openstack-ansible: Implement horizon venv support  https://review.openstack.org/22922614:36
*** gparaskevas has joined #openstack-ansible14:38
gparaskevashello there14:39
gparaskevasi am facing again the issue with locale settings when heat playbooks is running14:39
gparaskevasand cannot create heat domain due to false locale settings14:39
gparaskevasthe issue gets resolved with export LC_ALL=en_US.UTF-814:40
gparaskevason the heat api container14:40
*** phalmos has quit IRC14:48
cloudnullgparaskevas: what are the base local setting on your hosts ?14:53
cloudnulland what release ?14:53
gparaskevasone moment14:56
gparaskevasen_US.UTF-814:57
gparaskevasthe  LC_AL variable lacks a value14:57
gparaskevasthe  LC_ALL* variable lacks a value14:58
*** aojea_ has joined #openstack-ansible14:59
*** jmckind has joined #openstack-ansible14:59
*** jaypipes has quit IRC14:59
cloudnullis this on deployment ?15:00
gparaskevasyes15:00
gparaskevaskilo15:00
gparaskevasactualy 11.2.215:00
*** itsuugo has quit IRC15:00
gparaskevasif i set the export LC_ALL=en_US.UTF-8 on the host then the playbooks goes all well15:02
cloudnullwhat is `echo $LANG` on your host and within the heat api container ?15:02
gparaskevasen_US.UTF-815:03
gparaskevasen_US.UTF-8 on container and same on host15:04
cloudnullim curious because i have a box stood up deployed w/ kilo from branch and heat seems to be functional. LC_ALL and LANGUAGE are  blank and LANG=en_US.UTF-815:04
gparaskevasmm15:04
*** jmckind has quit IRC15:05
gparaskevaswell there must be something eith my installation then15:05
gparaskevasi select greece as country15:05
gparaskevasbut us lang as language15:05
*** jmckind_ has joined #openstack-ansible15:05
gparaskevasso some locale settngs are greek15:05
gparaskevasmaybe thats the issue?15:05
gparaskevaslocale: Cannot set LC_ALL to default locale: No such file or directory LANG=en_US.UTF-8 LANGUAGE=en_US:en LC_CTYPE="en_US.UTF-8" LC_NUMERIC=el_GR.UTF-8 LC_TIME=el_GR.UTF-8 LC_COLLATE="en_US.UTF-8" LC_MONETARY=el_GR.UTF-8 LC_MESSAGES="en_US.UTF-8" LC_PAPER=el_GR.UTF-8 LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT=el_GR.UTF-8 LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=15:06
*** sdake has joined #openstack-ansible15:06
gparaskevasthats what 'locale' gives me15:06
*** jmckind_ has quit IRC15:07
*** jmckind has joined #openstack-ansible15:07
cloudnulllooks like fuel ran into this issue before https://bugs.launchpad.net/fuel/+bug/140893515:08
openstackLaunchpad bug 1408935 in Fuel for OpenStack "LC_ALL variable not set" [Medium,Fix committed] - Assigned to Bartlomiej Piotrowski (bpiotrowski)15:08
cloudnulland resolved it by setting the local setting within the client rc15:08
cloudnullhttps://review.openstack.org/#/c/147469/15:08
*** agireud has joined #openstack-ansible15:08
openstackgerritMerged openstack/openstack-ansible: Add confirmation to teardown.sh  https://review.openstack.org/22890715:08
gparaskevasyes thats what i did the last time if you mean changing the openrc right?15:09
cloudnullyes15:09
gparaskevasfrom withing the playbooks?15:09
gparaskevaswithin*15:09
openstackgerritMerged openstack/openstack-ansible: Fix broken link in documentation  https://review.openstack.org/22962415:09
openstackgerritMerged openstack/openstack-ansible: Update documentation for rename in cidr_networks  https://review.openstack.org/22963515:09
openstackgerritMerged openstack/openstack-ansible: Adds insecure vars to configure-sslcertificate  https://review.openstack.org/22952515:09
cloudnullgparaskevas: seems like we could add it here https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/openstack_openrc/templates/openrc15:10
*** sdake_ has quit IRC15:10
cloudnullfirst line being `export LC_ALL=C`15:10
gparaskevasyes!15:10
cloudnullto piggyback on their success .15:10
gparaskevasi think so15:10
cloudnullthat would specifically make any client call from your shell happy.15:11
gparaskevasyes15:12
gparaskevasi thiknk i can submit that15:12
gparaskevasshould i create a bug entry?15:13
openstackgerritKevin Carter proposed openstack/openstack-ansible: Added LC_ALL to openrc  https://review.openstack.org/22995515:15
tiagogomes_Hi, I so I created two containers with galera in two different nodes and I shutdown one node. The virtual IP changed to the other machine fine and I can run some openstack commands (HTTP). However `nova agent-list` and `neutron agent` list report that everything is dead /o\15:15
cloudnullgparaskevas: i added our project to their bug report15:15
tiagogomes_On the logs of e.g. l3 agent I see 2015-10-01 16:15:26.242 4766 ERROR oslo_messaging._drivers.impl_rabbit [req-71aad0ca-8600-4920-b3cd-fba080d68359 ] AMQP server on 10.3.171.23:5672 is unreachable: [Errno 113] EHOSTUNREACH. Trying again in 2 seconds.15:15
cloudnulland PR'd that fix if you can give it a go and let us know that would be much appreciated.15:15
tiagogomes_ 10.3.171.23 is the IP of the container that had rabbitmq running in the node that was shutdown15:16
tiagogomes_Any ideas how to fix the problem?15:16
tiagogomes_Shouldn't the connections to the rabbitmq server use internal loadbalancer IP?15:17
cloudnulltiagogomes_:  when you say "The virtual IP changed to the other machine fine" do you mean that the vip failed over from one node to another?15:18
tiagogomes_cloudnull yep15:18
cloudnullrabbitmq does not like using vips and when a vip moves from one node to another the connections to the old rabbit server go into a sleep / wait state, which according to the RFC can last a day or so. Apsu can fact check me on that.15:19
cloudnullin our base setup we use a multi master config15:19
cloudnullno need to fail over between hosts in almost all cases.15:19
tiagogomes_cloudnull I am using the default config, whatever it is15:20
cloudnullthe openstack services directly connect to a deterministicly sorted list of rabbit servers15:20
cloudnullso the connection should go directly to the rabbitmq containers.15:20
Apsutiagogomes_: What VIP?15:20
*** palendae has left #openstack-ansible15:21
*** palendae has joined #openstack-ansible15:21
cloudnullbut if your services are using a vip and it fails over , it leads to a pit of dispair.15:21
*** palendae has left #openstack-ansible15:21
ApsuWhere are VIPs involved here? If you're using a load-balancer, unless you're running multiple load-balancers and a VIP between them, and fail one of those LBs, there shouldn't be any VIP movement.15:21
ApsuFailing Galera containers doesn't involve VIPs moving.15:21
ApsuIt involves a pool member going down in the LB, and it skipping it in the balance rotation15:22
cloudnullApsu:  i think he has his own vip failover between two hosts.15:22
cloudnulland it sounds like he's testing failure scenarios.15:22
Apsucloudnull: Right, that's what I'm asking about. Because if they do... excitement.15:22
ApsuVIPs are hard.15:22
ApsuThough, the pit of despair only occurs when you have services connecting to each other on the same node, and you move a VIP.15:23
tiagogomes_does this mean that I have to have more than 1 node active at a time: http://paste.openstack.org/show/475087/ ?15:23
ApsuThere's still plenty of nasty without that situation, but15:23
ApsuThat's the worst one15:23
tiagogomes_I am using keepalived for moving the VIP15:23
ApsuThere it is15:24
cloudnulltiagogomes_:  that looks like failover rabbit sadness15:24
ApsuWhich VIP?15:24
ApsuAre you using LVS to load-balance services?15:24
ApsuOr haproxy/some other LB?15:24
tiagogomes_haproxy + keepalived15:24
tiagogomes_I am making use of this https://review.openstack.org/#/c/21881815:25
gparaskevassure i will give a try tomorow15:25
odyssey4metiagogomes_ to cut this a little shorter - check the contents of nova.conf for the rabbit server list15:25
odyssey4meif you're using openstack-ansible, then the load balancer and keepalived are not involved15:26
cloudnulltiagogomes_:  is haproxy running on the same host as the rest of your openstack infra ?15:26
cloudnulland then failing over between the two hosts?15:26
tiagogomes_cloudnull yes15:26
cloudnullwe used to do exactly that @ RAX in the havana timeframe which suffered from similar issues.15:27
Apsutiagogomes_: Did you shutdown the previous master node entirely or just failover keepalived?15:28
tiagogomes_Apsu entirely15:28
tiagogomes_nova.conf looks good : rabbit_hosts = 10.3.213.143:5672,10.3.171.23:567215:28
odyssey4meok, and where you saw the logs showing the loss of connection to rpc - was there a reconnect afterwards?15:29
odyssey4meif so, that's exactly what should be going on15:30
*** jaypipes has joined #openstack-ansible15:30
tiagogomes_no there was no re connection. You were right, it tries to connect to the other rabbitmq server but it fails15:33
Apsutiagogomes_: Note that those aren't VIPs.15:33
ApsuThose are the rabbit container IPs.15:33
tiagogomes_yes, I got that15:34
ApsuIf you want to load-balance rabbit yourself, you'll need to change from our current multi-master layout to using the your VIP in those config lines.15:34
ApsuAlrighty15:34
tiagogomes_I don't care much about load balancing it, I just want high-availability15:34
odyssey4metiagogomes_ ok, that means that your rabbit cluster is fubar - if you look into that you'll find out why15:34
Apsurabbitmqctl cluster_status15:35
ApsuOn all the nodes15:35
ApsuMake sure none are partitioned and all of them can see each other -- minus the dead box of course15:35
*** jwagner_away is now known as jwagner15:35
tiagogomes_Cluster status of node 'rabbit@bl002-test1_rabbit_mq_container-b2d92330' ...15:36
tiagogomes_[{nodes,[{disc,['rabbit@bl002-test0_rabbit_mq_container-2ba5447d',15:36
tiagogomes_                'rabbit@bl002-test1_rabbit_mq_container-b2d92330']}]}]15:36
tiagogomes_bl002-test1_rabbit_mq_container-b2d92330 is the only one that I have active15:36
*** daneyon has joined #openstack-ansible15:37
*** persia has quit IRC15:38
*** skamithi14 has joined #openstack-ansible15:38
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Glance to Liberty RC1  https://review.openstack.org/22996715:38
ApsuRight, so that 'disc' portion means that's what rabbit understands is the cluster configuration15:38
*** skamithi14 has quit IRC15:38
Apsutiagogomes_: What does running_nodes say?15:38
*** skamithi14 has joined #openstack-ansible15:39
*** daneyon__ has joined #openstack-ansible15:39
*** skamithi13 has quit IRC15:39
*** skamithi has quit IRC15:39
tiagogomes_Apsu that command appears to not exist15:40
*** skamithi13 has joined #openstack-ansible15:40
Apsutiagogomes_: No... in the cluster_status output15:41
*** daneyon has quit IRC15:41
tiagogomes_Apsu, this is what I only have http://paste.openstack.org/show/475090/15:42
*** aojea_ has quit IRC15:43
*** skamithi14 has quit IRC15:43
Apsuhttps://www.rabbitmq.com/clustering.html#creating is what the output should look like for a running cluster15:43
*** phalmos has joined #openstack-ansible15:44
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Glance to Liberty RC1  https://review.openstack.org/22996715:44
*** mgoddard__ has joined #openstack-ansible15:45
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Glance to Liberty RC1  https://review.openstack.org/22996715:46
*** itsuugo has joined #openstack-ansible15:47
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Glance to Liberty RC1  https://review.openstack.org/22996715:48
*** mgoddard_ has quit IRC15:48
*** alop has joined #openstack-ansible15:49
*** phalmos has quit IRC15:50
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update Keystone to Liberty RC1  https://review.openstack.org/22691715:51
openstackgerritKevin Carter proposed openstack/openstack-ansible: Implement horizon venv support  https://review.openstack.org/22922615:53
*** phalmos has joined #openstack-ansible15:55
*** alejandrito has quit IRC15:58
odyssey4mecommunity meeting in #openstack-meeting-4 in 1 minute: cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, mancdaz, dolphm, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle15:59
*** Bjoern_ has joined #openstack-ansible15:59
*** daneyon has joined #openstack-ansible16:00
*** woodard has quit IRC16:01
*** mgoddard has joined #openstack-ansible16:02
*** woodard has joined #openstack-ansible16:02
*** mgoddard__ has quit IRC16:03
*** daneyon__ has quit IRC16:03
*** dolpher has quit IRC16:04
*** alejandrito has joined #openstack-ansible16:05
*** Bjoern_ is now known as BjoernT16:07
*** dolpher has joined #openstack-ansible16:12
tiagogomes_Apsu I run `rabbitmqctl start_app` and things are working now16:13
*** woodard has quit IRC16:18
*** jmckind has quit IRC16:18
*** jaypipes has quit IRC16:18
*** dolpher has quit IRC16:18
*** BjoernT has quit IRC16:19
*** woodard has joined #openstack-ansible16:22
*** Bjoern_ has joined #openstack-ansible16:37
*** woodard has quit IRC16:42
*** Bjoern_ is now known as Bjoernt16:45
*** Bjoernt is now known as BjoernT16:45
*** woodard has joined #openstack-ansible16:53
*** jwagner is now known as jwagner_away16:55
*** markvoelker has quit IRC16:55
*** markvoelker has joined #openstack-ansible16:56
*** persia has joined #openstack-ansible16:57
*** jmccrory_ has joined #openstack-ansible16:58
*** k_stev has quit IRC16:58
*** k_stev has joined #openstack-ansible16:58
odyssey4meright, let's get the discussion about https://review.openstack.org/224137  through the door - as it's a little more related to here and now17:00
cloudnullscarlisle BjoernT woodard ^17:01
odyssey4meI'm guessing that the issue here has been time constraints, but with 11.2.3 out the door it'll give a little more time for d34dh0r53 and co to give it a run through and confirm that it's ok. Right?17:01
scarlislecool, thanks odyssey4me17:02
odyssey4meBjoernT scarlisle have you guys done any testing with it?17:02
*** sdake_ has joined #openstack-ansible17:02
scarlisleI have not.  cloudnull told me about it last night and that was the first I've heard of it17:02
scarlisleI'm getting our 10.1.15 lab back up and running, so I'm just about in a position to try and test it there, if possible17:03
odyssey4meok, so essentially time is the biggest issue - there's a lot going on.... so if you'd really like to help it move along then the best thing to do is test it, then provide feedback for your testing in the review17:03
odyssey4meso if it worked for you and you're happy with it, then add a comment saying how you tested and what you results were to the +1 vote17:03
*** sdake_ has quit IRC17:04
odyssey4meif we see plenty of testers give positive feedback, then the cores will feel more comfortable voting it through without necessarily testing it themselves17:04
scarlislethis will be a nice way to also verify the --force-reinstall flag added in the dep17:04
odyssey4meagreed17:04
odyssey4meI've worked through most of it with my eyes, so I'm happy to add my vote as soon as I see more than one tester give a positive review.17:05
odyssey4meif those testers happen to be from a support background, even better. :)17:05
*** sdake has quit IRC17:05
cloudnulli'd also like to point out that presently without that change, we have no documention on upgrades besides the script directly. so regardless of the functionality or consistent experience with the script the imporvements are to the general deployer experience are worth pushing it through. - my2 cents.17:06
Apsutiagogomes_: Good to hear.17:06
*** skamithi13 has quit IRC17:06
*** skamithi13 has joined #openstack-ansible17:06
scarlislecloudnull +117:06
scarlislewe currently have really no documentation on it.  Its a black box.  Put Juno environment in, Kilo environment may come out, or it will fail with generic playbook output (if it got that far)17:07
odyssey4mecloudnull yeah, I'd like to just get 11.2.3 through the door - so I'm happy to add my vote on monday17:07
*** skamithi14 has joined #openstack-ansible17:07
odyssey4meif d34dh0r53 and andymccr choose to vote it through based on their testing between today and 11.2.3 releasing, then we're golden :)17:08
cloudnullsounds good to me17:09
scarlisleand I should be able to test it later today as well17:09
*** skamithi13 has quit IRC17:11
*** woodard has quit IRC17:11
odyssey4meexcellent, I'll look out for more feedback tomorrow17:11
odyssey4meevrardjp ping17:12
evrardjppong17:12
odyssey4meyou wanted to discuss the keepalived role inclusion for haproxy?17:12
evrardjpit's not really mine to decide, I agree with the way I did things, and cloudnull too...17:12
evrardjpI'd like to have your opinion, because you expressed the concerns17:13
evrardjpwhich are legitimate concerns :)17:13
evrardjpdownloading roles aren't that bad per se17:13
evrardjpand the role will only run when needed17:13
odyssey4meso ideally I'd prefer to have any extras optional - the battery inclusion should be a choice which can be enabled or disabled17:13
odyssey4methat said, I've worked through what you've done and I can't think of a better way to do it unless we can get haproxy's config to be more pluggable17:14
evrardjpI'm working on a new haproxy role btw17:14
evrardjp;)17:14
*** elo1 has quit IRC17:15
odyssey4meso I'm sort-of inclined at this stage to let it get in as it gives valuable functionality, then we can try and figure out a better and more pluggable model afterwards17:15
evrardjphaproxy doesn't really need to be more pluggable, it just needs to be correctly modifiable17:15
evrardjpok17:15
evrardjpthen I'll continue that way, and will give you a working version soon17:16
cloudnullmy only nit is the multiple var files which i think could be smashed into one, and the two calls to the plays can be combined as well using role variables / role when clause. that said i too like the way it is .17:16
*** skamithi14 has quit IRC17:16
*** skamithi13 has joined #openstack-ansible17:16
evrardjpcloudnull, I completely agree with the double file :p17:17
evrardjpI think there is room for improvement, and this is just the first version, that will work :p17:17
odyssey4meone way to prevent the download of the extra role for those not using it would simply be to name the role requirements file something different, then to include the instructions to rename it and to bootstrap ansible before running the haproxy play17:19
evrardjpodyssey4me, indeed17:19
evrardjpthat's exactly what we are talking about :)17:20
evrardjphowever if you do that, it will (by default) break your keepalived+haproxy run17:20
odyssey4methe role doesn't get used unless there is more than one haproxy host, right?17:20
evrardjpyup17:20
evrardjpit will only break stuff with multiple hosts17:20
evrardjpI think it's better for the future to have this inside a file that we'll use with bootstrap ansible17:21
odyssey4meok, then I think that's perfectly fine - if it's not compulsory to use, and the role isn't downloaded unless you mean it to then we're all good - my concerns are covered17:21
evrardjpif the deployer doesn't want it, then he could comment it17:22
odyssey4meyeah, but if the deployer wants it he can uncomment it ;)17:22
evrardjpthe advantage for the future is: if we already have a file in use, we could just drop more galaxy roles there, that could be requirement17:22
evrardjprequirements*17:22
cloudnullevrardjp: https://gist.github.com/cloudnull/02805f70edf00d80ff90 example diff17:23
evrardjpfor also AIO deployments17:23
odyssey4meactually, leave it on by default17:23
*** elo1 has joined #openstack-ansible17:23
odyssey4mefor the gate check we can script the removal of it so that we don't end up with doanload failures unnecessarily17:23
evrardjpthanks cloudnull ! I know it was easy to do it, but I was lazy ;)17:24
evrardjpI'll include your changes tomorrow17:24
cloudnulli updated the gist with the new file output too17:24
cloudnullsounds good. also its just a suggestion17:24
evrardjpI planned to do it anyway ;)17:25
cloudnulllike i said before, its good as is for now.17:25
evrardjpyou just pushed it so it will come earlier :D17:25
evrardjpodyssey4me, ok I'll leave it on, and if it brings problems, we will change the default behaviour17:25
evrardjpI'll include these changes tomorrow and test them before giving it for review17:26
evrardjpI have to go for today, sorry17:26
*** abitha has joined #openstack-ansible17:26
cloudnulli think download failures are inevitable as we move to a more galaxy based setup. we might want to add some retry logic around the capabilities. additionally if we find consistent failures this might be a good opportunity to go upstream and help make the general galaxy better.17:27
odyssey4mecloudnull sure, although for gate tests the preference is to, where possible, use resources available on the inside of the infra17:28
odyssey4mebut yeah, things are going to get more complicated - but I hope that we can split the roles out by then so that not every commit into every role has to do everything17:28
*** alejandrito has quit IRC17:36
*** alejandrito has joined #openstack-ansible17:38
*** alejandrito has quit IRC17:46
*** alejandrito has joined #openstack-ansible17:47
*** woodard has joined #openstack-ansible17:49
openstackgerritMerged openstack/openstack-ansible: Adjust default Keystone httpd processes and threads  https://review.openstack.org/22927517:52
*** woodard has quit IRC17:54
*** jwagner_away is now known as jwagner17:54
*** itsuugo has quit IRC17:55
*** itsuugo has joined #openstack-ansible17:56
*** galstrom_zzz is now known as galstrom17:57
*** CheKoLyN has joined #openstack-ansible18:03
*** tiagogomes_ has quit IRC18:05
*** javeriak has joined #openstack-ansible18:14
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update all SHA's (except Horizon) to Liberty RC1  https://review.openstack.org/22838518:18
*** woodard has joined #openstack-ansible18:27
*** gparaskevas has quit IRC18:30
*** openstackgerrit has quit IRC18:31
*** openstackgerrit has joined #openstack-ansible18:32
*** KLevenstein has quit IRC18:38
*** aslaen has joined #openstack-ansible18:44
*** KLevenstein has joined #openstack-ansible18:46
BjoernTodysset4me : no I did not play with that PR yet18:53
*** javeriak has quit IRC18:58
*** javeriak has joined #openstack-ansible18:59
*** javeriak_ has joined #openstack-ansible19:01
*** jmccrory_ has quit IRC19:03
*** aslaen has quit IRC19:03
*** javeriak has quit IRC19:03
*** itsuugo has quit IRC19:04
openstackgerritJaveria Khan proposed openstack/openstack-ansible: Add isolated flag to pip fall back installation  https://review.openstack.org/22228519:05
openstackgerritJaveria Khan proposed openstack/openstack-ansible: Add isolated flag to pip fall back installation  https://review.openstack.org/22228519:07
*** mgoddard has quit IRC19:25
*** mgoddard has joined #openstack-ansible19:25
*** mgoddard has quit IRC19:32
*** mgoddard has joined #openstack-ansible19:37
openstackgerritJesse Pretorius proposed openstack/openstack-ansible: Update ceph key hash  https://review.openstack.org/22889419:39
*** itsuugo has joined #openstack-ansible19:40
openstackgerritMerged openstack/openstack-ansible: Use of old name in deployment guide  https://review.openstack.org/22839219:40
*** shanecc has joined #openstack-ansible19:41
*** itsuugo has quit IRC19:44
*** itsuugo has joined #openstack-ansible19:46
*** logan2 has quit IRC19:47
*** javeriak_ has quit IRC19:51
*** mgoddard has quit IRC19:52
*** logan2 has joined #openstack-ansible19:55
*** mgoddard has joined #openstack-ansible19:56
*** mgoddard has quit IRC20:02
*** mgoddard has joined #openstack-ansible20:02
*** mgoddard has quit IRC20:08
*** elo1 has quit IRC20:10
*** mgoddard has joined #openstack-ansible20:11
*** Mudpuppy_ has joined #openstack-ansible20:16
*** mgoddard has quit IRC20:17
*** Mudpuppy has quit IRC20:19
prometheanfireosad works with armv8 you think?20:24
prometheanfiremight test it this weekend if I get access to a friends hw20:24
*** mgoddard has joined #openstack-ansible20:25
*** elo has joined #openstack-ansible20:28
openstackgerritMerged openstack/openstack-ansible: Add policy changes required for OSSA-2015-018 / CVE-2015-5240  https://review.openstack.org/22687220:38
*** k_stev1 has joined #openstack-ansible20:40
*** k_stev1 has quit IRC20:40
*** k_stev has quit IRC20:43
*** subsecond has joined #openstack-ansible20:43
*** markvoelker has quit IRC20:46
*** markvoelker has joined #openstack-ansible20:46
*** markvoelker has quit IRC20:47
*** markvoelker has joined #openstack-ansible20:47
*** k_stev has joined #openstack-ansible20:57
*** mgoddard has quit IRC20:59
*** subsecond has quit IRC21:00
*** woodard has quit IRC21:02
*** KLevenstein_ has joined #openstack-ansible21:03
*** Mudpuppy has joined #openstack-ansible21:04
*** KLevenstein has quit IRC21:04
*** KLevenstein_ is now known as KLevenstein21:04
*** Mudpuppy_ has quit IRC21:07
*** mgoddard has joined #openstack-ansible21:07
*** mgoddard has quit IRC21:12
*** mgoddard has joined #openstack-ansible21:17
*** Mudpuppy has quit IRC21:22
*** phalmos has quit IRC21:32
*** mgoddard has quit IRC21:32
*** mgoddard has joined #openstack-ansible21:33
*** svg has joined #openstack-ansible21:33
*** svg has left #openstack-ansible21:34
*** KLevenstein has quit IRC21:38
mgagneI'm looking for general Ansible experience here: How can I have a with_items clause and skip some of the item if a file does not exist on the source host. My intention is to be able to call the synchronize module and skip some item ONLY IF it's marked optional and it doesn't exist on the source.21:40
mgagneI tried to create my own synchronize module but found out that ansible also has a Synchronize action plugin doing black magic before calling the actual synchronize module :-/21:43
openstackgerritKevin Carter proposed openstack/openstack-ansible: Added documentation for config_template works  https://review.openstack.org/23016121:45
*** skamithi13 has quit IRC21:46
*** daneyon_ has joined #openstack-ansible21:51
*** daneyon_ has quit IRC21:51
*** mgoddard has quit IRC21:53
*** daneyon has quit IRC21:54
mgagnenooooo include + with_items is a removed deprecated feature =(21:55
*** skamithi13 has joined #openstack-ansible21:58
*** CheKoLyN has quit IRC21:59
openstackgerritKevin Carter proposed openstack/openstack-ansible: Added documentation for config_template works  https://review.openstack.org/23016122:00
*** mgoddard has joined #openstack-ansible22:01
*** jwagner is now known as jwagner_away22:11
cloudnullmgagne:  yup include with_items is dead . i think its coming back in ansible 222:15
mgagneyea, just read about it22:15
mgagneback to N include then =(22:15
*** skamithi13 has quit IRC22:16
mgagne /set mode +troll // I'll just rewrite it in Chef // /set mode -troll22:17
*** mgoddard has quit IRC22:18
cloudnulleverything is better in chef...22:20
* cloudnull runs22:20
cloudnull:)22:20
openstackgerritKevin Carter proposed openstack/openstack-ansible: Added documentation for config_template usage  https://review.openstack.org/23016122:22
*** mgoddard has joined #openstack-ansible22:22
mgagneand now I can't have variable in task name =(22:22
cloudnullno task vars are gone too. but from the souds of it they too will return in ansible222:22
mgagneI feel like ansible2 is the key to a lot of problems in this world22:23
cloudnullhttps://review.openstack.org/#/c/230161 <- evrardjp odyssey4me d34dh0r53 scarlisle BjoernT -> documentation on how to use the config_template22:23
cloudnullonce in master, we'll need to cherry-pick to kilo22:23
*** itsuugo has quit IRC22:26
*** spotz is now known as spotz_zzz22:27
*** galstrom is now known as galstrom_zzz22:33
*** mgoddard has quit IRC22:34
*** mgoddard has joined #openstack-ansible22:38
*** alextricity-mobi has quit IRC22:40
*** tlian has quit IRC22:42
*** alextricity-mobi has joined #openstack-ansible22:42
*** mgoddard has quit IRC22:44
openstackgerritIan Cordasco proposed openstack/openstack-ansible: Remove containers from /etc/hosts during teardown  https://review.openstack.org/21535722:46
*** scarlisle has quit IRC22:48
*** mgoddard has joined #openstack-ansible22:48
*** mgoddard has quit IRC22:53
mgagneso I managed to make it work :D22:53
mgagnehere is a stripped down version (and untested due to renamed variables) of what I wanted to achieve: https://gist.github.com/mgagne/4f0012b0804d8541e36222:55
mgagnehmm might be a wrong logic somewhere but base in there22:55
openstackgerritKevin Carter proposed openstack/openstack-ansible: Added documentation for config_template works  https://review.openstack.org/23016122:57
*** mgoddard has joined #openstack-ansible22:57
*** pradk has quit IRC22:59
*** sigmavirus24 is now known as sigmavirus24_awa22:59
mgagneimproved with better boolean logic included :D https://gist.github.com/mgagne/4f0012b0804d8541e36223:02
*** alejandrito has quit IRC23:03
*** mgoddard has quit IRC23:03
mgagnecloudnull: ^ :D23:04
*** mgoddard has joined #openstack-ansible23:07
*** mgoddard has quit IRC23:13
*** dolpher has joined #openstack-ansible23:16
*** mgoddard has joined #openstack-ansible23:16
*** dolpher has quit IRC23:22
cloudnullmgagne: nice !23:23
*** k_stev has quit IRC23:23
*** alop has quit IRC23:25
*** mgoddard has quit IRC23:28
*** dolpher has joined #openstack-ansible23:30
*** markvoelker has quit IRC23:30
openstackgerritKevin Carter proposed openstack/openstack-ansible: Removes /opt/*openstack*/ pattern from teardown.  https://review.openstack.org/22855823:34
*** dolpher has quit IRC23:36
openstackgerritKevin Carter proposed openstack/openstack-ansible: Implement keystone venv support  https://review.openstack.org/22951323:36
*** mgoddard has joined #openstack-ansible23:36
*** dolpher has joined #openstack-ansible23:37
*** mgoddard has quit IRC23:42
*** mgoddard has joined #openstack-ansible23:43
*** dolpher has quit IRC23:43
*** dolpher has joined #openstack-ansible23:44
*** agireud has quit IRC23:47
*** mgoddard has quit IRC23:49
*** dolpher has quit IRC23:50
*** mgoddard has joined #openstack-ansible23:52
*** mgoddard has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!