| *** mma has quit IRC | 00:05 | |
| *** gyee has quit IRC | 00:29 | |
| *** mattoliverau has joined #openstack-ansible | 00:37 | |
| openstackgerrit | Alex Redinger proposed openstack/openstack-ansible-os_keystone master: Add memcache flushing handler on db migrations https://review.openstack.org/608066 | 00:43 |
|---|---|---|
| *** mma has joined #openstack-ansible | 01:02 | |
| *** mma has quit IRC | 01:06 | |
| *** cshen has joined #openstack-ansible | 01:14 | |
| *** mmercer has quit IRC | 01:15 | |
| *** cshen has quit IRC | 01:19 | |
| *** jonher has quit IRC | 01:26 | |
| *** jonher has joined #openstack-ansible | 01:26 | |
| *** faizy98 has joined #openstack-ansible | 01:29 | |
| *** faizy_ has quit IRC | 01:32 | |
| *** spatel has joined #openstack-ansible | 01:34 | |
| *** mma has joined #openstack-ansible | 01:40 | |
| *** fatdragon has quit IRC | 01:41 | |
| *** mma has quit IRC | 01:44 | |
| *** macza has joined #openstack-ansible | 01:58 | |
| *** macza has quit IRC | 02:02 | |
| *** francois has quit IRC | 02:31 | |
| *** francois has joined #openstack-ansible | 02:31 | |
| openstackgerrit | Alex Redinger proposed openstack/openstack-ansible-os_keystone master: Add memcache flushing handler on db migrations https://review.openstack.org/608066 | 02:40 |
| *** mma has joined #openstack-ansible | 02:41 | |
| *** ram5391 has joined #openstack-ansible | 02:43 | |
| *** mma has quit IRC | 02:45 | |
| ram5391 | is there a known issue in the keystone deployment phase where haproxy only accepts https, but the deployment tests use http? | 02:45 |
| *** lbragstad has joined #openstack-ansible | 02:45 | |
| ram5391 | or some config I'm missing that sets what to use? | 02:46 |
| *** cshen has joined #openstack-ansible | 02:57 | |
| ram5391 | hoping I found the solution using keystone_service_publicuri_proto as per: https://docs.openstack.org/openstack-ansible-os_keystone/ocata/ | 02:59 |
| *** fatdragon has joined #openstack-ansible | 02:59 | |
| *** cshen has quit IRC | 03:02 | |
| *** fatdragon has quit IRC | 03:08 | |
| *** macza has joined #openstack-ansible | 03:18 | |
| *** macza has quit IRC | 03:22 | |
| *** mma has joined #openstack-ansible | 03:22 | |
| *** mma has quit IRC | 03:27 | |
| *** jonher has quit IRC | 03:30 | |
| *** jonher has joined #openstack-ansible | 03:30 | |
| *** vnogin has joined #openstack-ansible | 03:31 | |
| ram5391 | It seems like the tests for keystone out of the box are configured to check for 'http' not 'https' whereas the otb config for keystone is to use 'https' I can verify that the service is running both on the container and via haproxy via 'https' but not 'http' If this isn't a known issue, I'll create a ticket for it | 03:31 |
| *** vnogin has quit IRC | 03:36 | |
| *** fatdragon has joined #openstack-ansible | 03:38 | |
| *** fatdragon has quit IRC | 03:49 | |
| *** ram5391 has quit IRC | 03:52 | |
| *** udesale has joined #openstack-ansible | 03:54 | |
| *** spatel has quit IRC | 04:00 | |
| *** macza has joined #openstack-ansible | 04:04 | |
| *** canori01 has quit IRC | 04:06 | |
| *** faizy_ has joined #openstack-ansible | 04:07 | |
| *** macza has quit IRC | 04:10 | |
| *** macza has joined #openstack-ansible | 04:10 | |
| *** faizy98 has quit IRC | 04:11 | |
| *** macza has quit IRC | 04:15 | |
| *** macza has joined #openstack-ansible | 04:16 | |
| *** fatdragon has joined #openstack-ansible | 04:17 | |
| *** macza has quit IRC | 04:20 | |
| *** lbragstad has quit IRC | 04:21 | |
| *** mma has joined #openstack-ansible | 04:24 | |
| *** mma has quit IRC | 04:28 | |
| *** fatdragon has quit IRC | 04:29 | |
| *** defionscode has quit IRC | 04:30 | |
| *** yetiszaf has quit IRC | 04:30 | |
| *** defionscode has joined #openstack-ansible | 04:33 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro https://review.openstack.org/591424 | 04:56 |
| *** cshen has joined #openstack-ansible | 04:57 | |
| *** macza has joined #openstack-ansible | 04:57 | |
| *** fatdragon has joined #openstack-ansible | 04:57 | |
| *** macza has quit IRC | 05:02 | |
| *** cshen has quit IRC | 05:03 | |
| openstackgerrit | Alex Redinger proposed openstack/openstack-ansible-os_keystone master: Add memcache flushing handler on db migrations https://review.openstack.org/608066 | 05:09 |
| *** fatdragon has quit IRC | 05:09 | |
| *** cshen has joined #openstack-ansible | 05:13 | |
| *** pcaruana has joined #openstack-ansible | 05:15 | |
| *** cshen has quit IRC | 05:18 | |
| *** mma has joined #openstack-ansible | 05:23 | |
| *** olivierb has joined #openstack-ansible | 05:30 | |
| *** chkumar|off is now known as chandankumar | 05:34 | |
| *** fatdragon has joined #openstack-ansible | 06:11 | |
| chandankumar | odyssey4me: Good morning | 06:16 |
| chandankumar | odyssey4me: http://logs.openstack.org/24/591424/21/check/openstack-ansible-functional-centos-7/2045287/job-output.txt.gz#_2018-10-12_06_00_11_015357 | 06:16 |
| chandankumar | odyssey4me: during instaling python packages in venv it is failing | 06:17 |
| openstackgerrit | Alex Redinger proposed openstack/openstack-ansible-os_keystone master: Add memcache flushing handler on db migrations https://review.openstack.org/608066 | 06:17 |
| chandankumar | please have a look | 06:18 |
| *** cshen has joined #openstack-ansible | 06:21 | |
| *** fatdragon has quit IRC | 06:22 | |
| *** DanyC has quit IRC | 06:34 | |
| *** hamzaachi has joined #openstack-ansible | 06:37 | |
| *** hamzaachi has quit IRC | 06:40 | |
| *** hamzaachi has joined #openstack-ansible | 06:45 | |
| *** hamzaachi has quit IRC | 06:46 | |
| *** hamzaachi has joined #openstack-ansible | 06:46 | |
| *** shardy has joined #openstack-ansible | 07:18 | |
| *** hamzaachi has quit IRC | 07:25 | |
| *** fatdragon has joined #openstack-ansible | 07:29 | |
| *** faizy98 has joined #openstack-ansible | 07:37 | |
| *** faizy_ has quit IRC | 07:41 | |
| *** fatdragon has quit IRC | 07:41 | |
| deployer2 | Hi! odyssey4me, mgariepy, chandankumar something very similar here - markers 'python_version == "3.4"' don't match your environment | 07:43 |
| deployer2 | I have applied patch https://review.openstack.org/#/c/608042/3 to 18.0.0.0rc3, now failing on TASK [repo_build : Create OpenStack-Ansible requirement wheels] https://pastebin.com/raw/20jaHnUT | 07:44 |
| *** tosky has joined #openstack-ansible | 07:45 | |
| *** mma has quit IRC | 07:45 | |
| *** mma has joined #openstack-ansible | 07:45 | |
| odyssey4me | deployer2: you need to apply that patch to the head of stable/rocky, not just to 10.0.0.0rc3 | 07:46 |
| *** mma has quit IRC | 07:47 | |
| odyssey4me | morning folks, looks like opensuse is broken again :/ | 07:48 |
| chandankumar | odyssey4me: for this one centos failure https://review.openstack.org/#/c/591424/ | 07:48 |
| chandankumar | what to do? | 07:48 |
| deployer2 | ok, will try to figure out how to do that. Meanwhile - has anyone got successful run of all playbooks for rocky on ubuntu 18.04? | 07:49 |
| chandankumar | odyssey4me: how to fix this part https://review.openstack.org/#/c/591424/21/tasks/tempest_install.yml@128 ? if we pass tempest_install_method = distro it is going to execute all the steps | 07:51 |
| chandankumar | for source will I move it a seperate yaml? | 07:52 |
| odyssey4me | chandankumar: reviewed, much the same comments as done before | 07:56 |
| chandankumar | odyssey4me: updating thanks :-) | 07:59 |
| odyssey4me | chandankumar: also, the niclude_vars has gone - that needs to be returned | 08:00 |
| *** electrofelix has joined #openstack-ansible | 08:04 | |
| *** rpittau has quit IRC | 08:09 | |
| *** rpittau has joined #openstack-ansible | 08:10 | |
| *** rpittau has quit IRC | 08:10 | |
| *** rpittau has joined #openstack-ansible | 08:11 | |
| *** macza has joined #openstack-ansible | 08:18 | |
| *** macza has quit IRC | 08:19 | |
| *** macza has joined #openstack-ansible | 08:19 | |
| *** faizy_ has joined #openstack-ansible | 08:20 | |
| *** faizy98 has quit IRC | 08:20 | |
| *** cshen has quit IRC | 08:23 | |
| *** macza has quit IRC | 08:24 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Use the compute kit + horizon for all distros https://review.openstack.org/609329 | 08:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Restore OpenSUSE voting jobs https://review.openstack.org/609353 | 08:29 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Use the compute kit + horizon for all distros https://review.openstack.org/609329 | 08:31 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Restore OpenSUSE voting jobs https://review.openstack.org/609353 | 08:34 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Restore bionic/ceph voting jobs https://review.openstack.org/609965 | 08:34 |
| *** devx has quit IRC | 08:39 | |
| *** devx has joined #openstack-ansible | 08:40 | |
| *** cshen has joined #openstack-ansible | 08:51 | |
| chandankumar | odyssey4me: by the way python-tempestconf and stackviz package are now available in openstack/rpm-packaging :-) | 08:53 |
| *** DanyC has joined #openstack-ansible | 08:55 | |
| *** cshen has quit IRC | 08:55 | |
| openstackgerrit | Vieri proposed openstack/openstack-ansible-memcached_server master: fix tox python3 overrides https://review.openstack.org/609973 | 09:03 |
| openstackgerrit | Vieri proposed openstack/openstack-ansible-os_gnocchi master: fix tox python3 overrides https://review.openstack.org/609975 | 09:06 |
| openstackgerrit | Vieri proposed openstack/openstack-ansible-os_heat master: fix tox python3 overrides https://review.openstack.org/609976 | 09:11 |
| openstackgerrit | Vieri proposed openstack/openstack-ansible-os_almanach master: fix tox python3 overrides https://review.openstack.org/609977 | 09:13 |
| openstackgerrit | Vieri proposed openstack/openstack-ansible-os_ceilometer master: fix tox python3 overrides https://review.openstack.org/609979 | 09:16 |
| evrardjp | thanks for the rechecks odyssey4me | 09:18 |
| openstackgerrit | Vieri proposed openstack/openstack-ansible-openstack_hosts master: fix tox python3 overrides https://review.openstack.org/609980 | 09:18 |
| odyssey4me | evrardjp: yeah, no worries - it's a bit frustrating :/ | 09:18 |
| evrardjp | I am sorry I should be on the ball on this -- I just have realised my notifications where to be improved :p | 09:19 |
| evrardjp | I see what goes to gating, not what fails in gating | 09:20 |
| evrardjp | so I always assume it's passing which is not the case :p | 09:20 |
| *** cshen has joined #openstack-ansible | 09:29 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro https://review.openstack.org/591424 | 09:30 |
| *** cshen has quit IRC | 09:34 | |
| *** faizy98 has joined #openstack-ansible | 09:35 | |
| *** faizy_ has quit IRC | 09:37 | |
| odyssey4me | chandankumar: almost there, just a few edits to go | 09:38 |
| chandankumar | odyssey4me: sure | 09:38 |
| chandankumar | odyssey4me: in openstack-ansible do we use lxc containers or kolla containers? | 09:40 |
| odyssey4me | chandankumar: lxc by default for now, we're switching to nspawn soon | 09:40 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Tidy up README image use instructions https://review.openstack.org/609983 | 09:42 |
| chandankumar | odyssey4me: one more thing does this one https://github.com/openstack/openstack-ansible-os_tempest/blob/master/meta/main.yml can be switched to centos also? | 09:42 |
| odyssey4me | chandankumar: it already has EL 7, which covers CentOS and RHEL | 09:43 |
| chandankumar | odyssey4me: so apt_package_pinning is just for ubunut | 09:44 |
| chandankumar | na? | 09:44 |
| chandankumar | I mean ansible galaxy dependencies | 09:44 |
| odyssey4me | oh, but that role will do nothing on any platform other than ubuntu anyway - it'll just skip it | 09:44 |
| odyssey4me | in fact, in another patch, I think we can remove it because that role doesn't do any pinning | 09:45 |
| chandankumar | odyssey4me: sure | 09:45 |
| *** cshen has joined #openstack-ansible | 09:51 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro https://review.openstack.org/591424 | 09:54 |
| *** fatdragon has joined #openstack-ansible | 09:59 | |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Remove apt_package_pinning dependency from os_tempest role https://review.openstack.org/609992 | 10:00 |
| chandankumar | odyssey4me: ^^ | 10:00 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Remove apt_package_pinning dependency from os_tempest role https://review.openstack.org/609992 | 10:01 |
| *** vnogin has joined #openstack-ansible | 10:03 | |
| *** fatdragon has quit IRC | 10:03 | |
| *** suggestable has joined #openstack-ansible | 10:06 | |
| deployer2 | odyssey4me btw is it expected that those who are now on 18.0.0.0rc3 will be able to upgrade to 18.0.0 stable when it is released or full reinstall will be needed? | 10:07 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Remove apt_package_pinning dependency from os_tempest role https://review.openstack.org/609992 | 10:11 |
| *** udesale has quit IRC | 10:36 | |
| *** fatdragon has joined #openstack-ansible | 10:39 | |
| *** cshen has quit IRC | 10:39 | |
| *** pabelanger has quit IRC | 10:40 | |
| *** fatdragon has quit IRC | 10:44 | |
| odyssey4me | deployer2: it should work, but we don't test it so YMMV depending on what your deployment looks like | 10:45 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Tidy up README image use instructions https://review.openstack.org/609983 | 10:58 |
| *** cshen has joined #openstack-ansible | 11:12 | |
| *** fatdragon has joined #openstack-ansible | 11:13 | |
| *** cshen has quit IRC | 11:16 | |
| *** fatdragon has quit IRC | 11:18 | |
| *** faizy_ has joined #openstack-ansible | 11:20 | |
| *** faizy98 has quit IRC | 11:24 | |
| *** vnogin has quit IRC | 11:31 | |
| *** dave-mccowan has joined #openstack-ansible | 11:38 | |
| *** vnogin has joined #openstack-ansible | 11:39 | |
| deployer2 | I have defined 3 repo containers, but only 2 of them deploy successfully, third one fails with "Failed to establish a new connection: [Errno 111] Connection refused". | 11:46 |
| deployer2 | TASK [repo_server : Install pip packages (from repo)]. Have destroyed container with containers-lxc-destroy.yml + deleted facts & recreated, but the same. Tested with wget - rest of repo containers get http this one connection refused. | 11:46 |
| deployer2 | what else to delete to make third container to be identical to others? Does haproxy has some accesslist ? | 11:47 |
| deployer2 | It cannot het http service from haproxy VIP:8181 | 11:48 |
| *** juhak has left #openstack-ansible | 11:48 | |
| odyssey4me | deployer2: is haproxy running, are the backends available? also, are you sure that you don't have bad CIDR's or conflicting IP's? | 11:49 |
| suggestable | Hi everyone! We've managed to deploy OSA without any warnings during the playbook runs, but are struggling to get l3ha routing working. Running on stable/queens on Ubuntu 16.04. Three controller nodes handling all infrastructure/network/storage services. We've even tried running neutron-server on metal, and encountered an issue where the bind host is hard set in the j2 template. Anyone able to point us in the right direction? | 11:50 |
| deployer2 | odyssey4me haproxy runs and VIP is only up on 1 infra node - that should be so. The rest of repo containers get the service, only one gets refused | 11:51 |
| *** vnogin has quit IRC | 11:51 | |
| *** vnogin has joined #openstack-ansible | 11:52 | |
| odyssey4me | deployer2: ok, and in the playbook output that ran, did anything happen there to show a failure? | 11:52 |
| *** cshen has joined #openstack-ansible | 11:55 | |
| chandankumar | odyssey4me: for setting default to distro for centos where can i make changes? | 11:55 |
| deployer2 | odyssey4me this is the output https://pastebin.com/raw/GHv5WSWN | 11:55 |
| *** vnogin has quit IRC | 11:58 | |
| deployer2 | cannot understand how one container is different from the rest even after complete recreation | 11:58 |
| *** vnogin has joined #openstack-ansible | 11:58 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: Update MNAIO to deploy systemd-networkd https://review.openstack.org/609826 | 12:02 |
| deployer2 | odyssey4me to recreate container is it enough to destroy with with containers-lxc-destroy.yml + deleted facts or there could be any other remnanats somewhere? Or maybe I need to reinstall haproxy also if repo container is recreated from scratch? | 12:05 |
| odyssey4me | deployer2: no, using the destroy playbook is just fine - and haproxy is already configured and the container names & ip's aren't changing, so it shouldn't need to be setup again | 12:08 |
| odyssey4me | chandankumar: the default must not be distro, the default must be source - the tox env sets the override to distro for the distro test | 12:09 |
| odyssey4me | deployer2: what release are you using to test with? | 12:09 |
| *** vnogin has quit IRC | 12:09 | |
| odyssey4me | deployer2: also, that task has a fallback - did the fallback not work? https://github.com/openstack/openstack-ansible-repo_server/blob/master/tasks/repo_install.yml#L90-L124 | 12:11 |
| *** sawblade6 has joined #openstack-ansible | 12:13 | |
| odyssey4me | deployer2: if it's queens or later, you'll also need to run the repo-use playbook after deleting those containers | 12:13 |
| odyssey4me | (I think) | 12:13 |
| deployer2 | odyssey4me Rocky on bionic. Using git clone 18.0.0.0rc3 from docs, then git checkout stable/rocky, then applying patch https://review.openstack.org/#/c/608042/3 and going from there | 12:13 |
| odyssey4me | deployer2: ok, did the fallback task not work? your log only shows the first task failing | 12:14 |
| deployer2 | need to check logs | 12:15 |
| *** fatdragon has joined #openstack-ansible | 12:18 | |
| deployer2 | odyssey4me what should I look for? not finding "Install pip packages (from pypi mirror)" from rescue task name | 12:18 |
| odyssey4me | deployer2: that's odd, but if there's a /root/.pip/pip.conf in those containers, remove it | 12:19 |
| odyssey4me | this is unusual, and I'm not sure how you got into this mess | 12:20 |
| *** fatdragon has quit IRC | 12:23 | |
| *** ansmith has joined #openstack-ansible | 12:24 | |
| deployer2 | odyssey4me will try, thanks. This is already n-th attemt to bring up rocky on bionic. First had mistake in my user_variables.yml regarding VIP netmask and had to bring internal VIP up manually but then corrected it. Dont know, need to rethink | 12:27 |
| odyssey4me | deployer2: rather than destroy and recreate containers next time, try to properly figure out the cause of the fail - there's a fair chance that you have some sort of bad networking config if there was a comms failure | 12:28 |
| odyssey4me | networking config is always the stumbling block for a first-time deployer | 12:28 |
| odyssey4me | also, you'll save yourself a lot of pain by restarting from scratch with a fresh host if you change any network configs | 12:29 |
| *** sawblade6 has quit IRC | 12:42 | |
| *** sawblade6 has joined #openstack-ansible | 12:43 | |
| *** sawblade6 has quit IRC | 12:46 | |
| *** sawblade6 has joined #openstack-ansible | 12:46 | |
| *** sawblade6 has quit IRC | 12:49 | |
| *** sawblade6 has joined #openstack-ansible | 12:49 | |
| *** sawblade6 has quit IRC | 12:51 | |
| *** sawblade6 has joined #openstack-ansible | 12:51 | |
| *** fatdragon has joined #openstack-ansible | 12:53 | |
| *** sawblade6 has quit IRC | 12:53 | |
| *** sawblade6 has joined #openstack-ansible | 12:53 | |
| *** sawblade6 has quit IRC | 12:54 | |
| *** sawblade6 has joined #openstack-ansible | 12:54 | |
| *** sawblade6 has quit IRC | 12:55 | |
| *** sawblade6 has joined #openstack-ansible | 12:55 | |
| *** sawblade6 has quit IRC | 12:56 | |
| *** fatdragon has quit IRC | 12:57 | |
| *** vnogin has joined #openstack-ansible | 13:03 | |
| *** thuydang has joined #openstack-ansible | 13:08 | |
| *** markvoelker has quit IRC | 13:15 | |
| *** munimeha1 has joined #openstack-ansible | 13:22 | |
| *** lbragstad has joined #openstack-ansible | 13:23 | |
| *** fatdragon has joined #openstack-ansible | 13:26 | |
| *** fatdragon has quit IRC | 13:31 | |
| chandankumar | odyssey4me: http://logs.openstack.org/24/591424/23/check/openstack-ansible-functional-ubuntu-bionic/6e73cfa/job-output.txt.gz#_2018-10-12_10_53_09_755379 at this part it is failing at all places | 13:33 |
| chandankumar | odyssey4me: https://review.openstack.org/#/c/591424/ | 13:34 |
| odyssey4me | chandankumar: commented in review | 13:36 |
| mgariepy | hwoarang, evrardjp : opensuse-423, gives Retry limit in 1m 08s for the my patch in nova. is that something that you are aware ? https://review.openstack.org/#/c/605789 | 13:38 |
| *** munimeha1 has quit IRC | 13:38 | |
| *** vrobert has joined #openstack-ansible | 13:42 | |
| vrobert | hi | 13:43 |
| vrobert | this calculation caused troubles to me in nova-api: nova_wsgi_processes: "{{ [[ansible_processor_vcpus|default(1), 1] | max * 2, nova_wsgi_processes_max] | min }}" | 13:43 |
| vrobert | caused slowdown and timeouts in nova-api | 13:44 |
| vrobert | caused slowdowns and timeouts in nova-api | 13:44 |
| openstackgerrit | Chandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro https://review.openstack.org/591424 | 13:44 |
| vrobert | Are you sure the max*2 is correct in the formula? | 13:45 |
| vrobert | In the calculations of nova_api_threads: "{{ [[ansible_processor_vcpus|default(2) // 2, 1] | max, nova_api_threads_max] | min }}" seems correct for me. | 13:51 |
| vrobert | Here is a divison by 2 causing these threads are set to 10 if I have 20 vcpus. | 13:51 |
| vrobert | But nova_wsgi_processes ended with 16 which is the default for nova_wsgi_processes_max. | 13:52 |
| mgariepy | you can set the variables to change it if you need to | 13:53 |
| vrobert | Of course and I did it. | 13:53 |
| vrobert | Just want to be shared with you. | 13:53 |
| vrobert | Maybe I am wrong and my controller nodes are overloaded... | 13:53 |
| mgariepy | I guess it depends on the hw you have for you control plane. | 13:53 |
| vrobert | Yes its hardly depends on you are right. | 13:53 |
| *** rpittau has quit IRC | 13:54 | |
| vrobert | But I think this calculation for nova_wsgi_processes: "{{ [[ansible_processor_vcpus|default(1), 1] | max * 2, nova_wsgi_processes_max] | min }}" not correct. | 13:54 |
| vrobert | Why is the max*2 multiplier at all? | 13:54 |
| vrobert | I think lots of deployments out there running a lot of things in the controller nodes. | 13:55 |
| vrobert | I tested it on a 59 nodes cluster with 3 controller nodes. | 13:55 |
| mgariepy | in my case i have 24 vcpus , and end up with 16 which is ok for me. | 13:55 |
| vrobert | It was okay for me in the beginning. | 13:55 |
| vrobert | But when I started to do full tempest runs it was slowing down. | 13:56 |
| vrobert | After 2-3 tempest run its started to give me timeouts. | 13:56 |
| vrobert | Its happened after all my 16 uwsgi workers are connected to rabbitmq. | 13:57 |
| vrobert | Its happened after all my 16 uwsgi workers are connected to rabbitmq on all nodes. | 13:57 |
| *** munimeha1 has joined #openstack-ansible | 13:57 | |
| vrobert | So it was a bit hiding problem for me. | 13:57 |
| vrobert | But right now, with nova_wsgi_processes_max: 10 solved the problem for me. | 13:57 |
| mgariepy | my config is : thread 1, process 16 on a 24 thread cpu | 13:58 |
| vrobert | Yes it is the default. | 13:58 |
| vrobert | I found that each workers start 5-6 threads if there is a load on the nova-api. | 13:59 |
| vrobert | 3 rabbitmq-heartbeats, 1 other, 1 rabbitmq read() which is a blocking read. | 14:01 |
| mgariepy | lol, the nova_api_threads calculation doesn't make much sense.. haha | 14:01 |
| vrobert | why? | 14:02 |
| *** sum12 has quit IRC | 14:02 | |
| mgariepy | ok i'm confused about it. nova-api thread != wsgi thread haha | 14:02 |
| vrobert | yes. | 14:02 |
| vrobert | yes, I think so. | 14:03 |
| vrobert | there is always 1 wsgi thread and others are nova threads | 14:03 |
| vrobert | there is always 1 wsgi thread per wsgi processes and others are nova threads | 14:03 |
| vrobert | there is always 1 wsgi thread per wsgi processes and others are nova threads I think... | 14:03 |
| *** sum12 has joined #openstack-ansible | 14:03 | |
| mgariepy | yeah | 14:03 |
| mgariepy | so each wsgi process will start 16 nova-api threads | 14:04 |
| vrobert | No, I don't sad that. | 14:05 |
| mgariepy | is the issue with the number of wsgi process or the number of thread in nova-api ? | 14:05 |
| vrobert | No, I didn't sad that. | 14:05 |
| vrobert | I had issue with the number of max wsgi processes. | 14:06 |
| vrobert | I tried to modify nova api threads, rpc pool threads but the only thing wich is worked for me is that I reduces to max nova-api wsgi processes from 16 to 10. | 14:07 |
| *** fatdragon has joined #openstack-ansible | 14:07 | |
| vrobert | I tried to modify nova api threads, rpc pool threads but the only thing wich is worked for me is that I reduced the max nova-api wsgi processes from 16 to 10. | 14:07 |
| mgariepy | maybe a physical core count would be better ? | 14:08 |
| mgariepy | like, the min between 16 or the # of core ? | 14:09 |
| vrobert | yes, its makes sense. | 14:09 |
| mgariepy | can you write a patch ? | 14:10 |
| vrobert | This problem was hiding until I started to stresstest my cloud. | 14:10 |
| vrobert | So maybe there are a lots of deployments out there where this problems is hiding... | 14:10 |
| mgariepy | yeah indeed. | 14:11 |
| odyssey4me | cloudnull: when you're in, I have a slightly perplexing issue in downstream CI to figure out relating to systemd-resolved when executing an AIO build | 14:11 |
| vrobert | Because in the beginning where not all my wsgi processes was utilized everything was ok. | 14:11 |
| *** weezS has joined #openstack-ansible | 14:11 | |
| *** fatdragon has quit IRC | 14:11 | |
| vrobert | Okay lets say that I modify only one thing from vcpu to cpu count in the formula: {{ [[ansible_processor_cores|default(1), 1] | max * 2, nova_wsgi_processes_max] | min }} | 14:13 |
| openstackgerrit | weezer su proposed openstack/openstack-ansible master: Add one test case to the TestMergeDictUnit for same key dict merge https://review.openstack.org/609745 | 14:13 |
| vrobert | It would be min(10*2,16) which would be 16 again, not good again... | 14:14 |
| vrobert | If I remove the *2 multiplier: {{ [[ansible_processor_cores|default(1), 1] | max 2, nova_wsgi_processes_max] | min }} | 14:14 |
| vrobert | It would be min(10,16) which would be 10 which is good for me | 14:15 |
| vrobert | It would be min(10,16) which would be 10 which is good for me on a busy controller node | 14:15 |
| vrobert | It would be min(10,16) which would be 10 which is good for me on a busy controller nodes | 14:15 |
| vrobert | But yes Its hardly depends on the load and the busy processes and threads on the actual controller node so its very hard to prove it... | 14:16 |
| mgariepy | i would not multiply per 2. | 14:18 |
| vrobert | I think the *2 multiplier on vcpus doesn't make sense | 14:21 |
| vrobert | maybe they wanted to do it on cpu cores not vcpu cores... | 14:21 |
| mgariepy | i would do something like: "{{ [[ansible_processor_count * ansible_processor_cores, 1] | max, nova_wsgi_processes_max] | min }}" | 14:21 |
| mgariepy | or ansible_processor_vcpus / ansible_processor_threads_per_core | 14:22 |
| vrobert | yes, seems scorrect to me | 14:22 |
| vrobert | I like it! | 14:22 |
| mgariepy | can you create the patch and comment the why in the commit message please ? | 14:23 |
| vrobert | Yes, I can try that. | 14:23 |
| *** thuydang has quit IRC | 14:23 | |
| *** lbragstad is now known as elbragstad | 14:25 | |
| *** sawblade6 has joined #openstack-ansible | 14:27 | |
| mgariepy | if you need help let us know. | 14:27 |
| vrobert | Okay thank you mgariepy! | 14:31 |
| *** munimeha1 has quit IRC | 14:32 | |
| mgariepy | vrobert, have you submitted a patch in the past ? | 14:33 |
| *** faizy_ has quit IRC | 14:35 | |
| vrobert | No I didn't. | 14:36 |
| vrobert | If you can help me to do this this it can save me a lot of time. | 14:37 |
| mgariepy | spotz, you you still have that handy doc on how to start submitting code ? | 14:38 |
| *** fatdragon has joined #openstack-ansible | 14:39 | |
| spotz | mgariepy: You mean the git and gerrit stuff? | 14:39 |
| mgariepy | yep to help vrobert getting started :D | 14:39 |
| vrobert | thanks in advance :) | 14:39 |
| *** ansmith has quit IRC | 14:40 | |
| *** markvoelker has joined #openstack-ansible | 14:41 | |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-repo_server stable/rocky: Replace Chinese punctuation with English punctuation https://review.openstack.org/610062 | 14:43 |
| mgariepy | vrobert, do you have a gerrit account ? | 14:43 |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_neutron stable/rocky: Replace Chinese punctuation with English punctuation https://review.openstack.org/610063 | 14:43 |
| vrobert | No, I dont. | 14:43 |
| mgariepy | https://docs.openstack.org/infra/manual/developers.html | 14:43 |
| vrobert | thx | 14:43 |
| mgariepy | follow this to setup your account. | 14:44 |
| *** fatdragon has quit IRC | 14:45 | |
| *** canori01 has joined #openstack-ansible | 14:49 | |
| openstackgerrit | jacky06 proposed openstack/openstack-ansible-os_swift master: Revert "use include_tasks instead of include" https://review.openstack.org/610069 | 14:52 |
| *** spatel has joined #openstack-ansible | 14:55 | |
| *** sawblade6 has quit IRC | 14:55 | |
| vrobert | woo its a bit too much for me at this time for a one line change | 14:56 |
| vrobert | maybe I should need to open a discussion around this with opening a thread somewhere | 14:56 |
| *** ansmith has joined #openstack-ansible | 14:57 | |
| vrobert | Somebody needs to verify I am right when I sad that nova_wsgi_processes calculation is not correct. :) | 14:57 |
| *** weezS has quit IRC | 14:57 | |
| *** sawblade6 has joined #openstack-ansible | 14:58 | |
| vrobert | mgariepy what do you think where should I open a discussion for this? | 14:59 |
| *** sawblade6 has quit IRC | 14:59 | |
| *** weezS has joined #openstack-ansible | 14:59 | |
| *** weezS has joined #openstack-ansible | 15:00 | |
| mgariepy | cloudnull, any idea about this ? ^^ | 15:01 |
| *** weezS has joined #openstack-ansible | 15:01 | |
| mgariepy | vrobert, you can open a bug in LP | 15:01 |
| mgariepy | then it will be reviewed | 15:01 |
| *** sawblade6 has joined #openstack-ansible | 15:01 | |
| *** sawblade6 has quit IRC | 15:03 | |
| *** thuydang has joined #openstack-ansible | 15:04 | |
| *** ansmith has quit IRC | 15:05 | |
| openstackgerrit | Andy Smith proposed openstack/openstack-ansible master: Add documentation for hybrid messaging configuration https://review.openstack.org/610079 | 15:05 |
| vrobert | Okay, I will follow that way. Thanks for your help and thx for your time. | 15:05 |
| *** sawblade6 has joined #openstack-ansible | 15:05 | |
| *** sawblade6 has quit IRC | 15:08 | |
| *** sawblade6 has joined #openstack-ansible | 15:08 | |
| *** sawblade6 has quit IRC | 15:09 | |
| *** sawblade6 has joined #openstack-ansible | 15:09 | |
| *** sawblade6 has quit IRC | 15:11 | |
| *** cshen has quit IRC | 15:15 | |
| *** vrobert has left #openstack-ansible | 15:17 | |
| *** ansmith has joined #openstack-ansible | 15:18 | |
| *** gyee has joined #openstack-ansible | 15:19 | |
| *** fatdragon has joined #openstack-ansible | 15:20 | |
| *** vnogin has quit IRC | 15:21 | |
| *** fatdragon has quit IRC | 15:24 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Do not use apt-cacher-ng on the MNAIO host https://review.openstack.org/610083 | 15:30 |
| *** thuydang has quit IRC | 15:31 | |
| *** thuydang has joined #openstack-ansible | 15:31 | |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_masakari stable/rocky: use include_tasks instead of include https://review.openstack.org/610084 | 15:31 |
| openstackgerrit | caoyuan proposed openstack/openstack-ansible-os_horizon stable/rocky: Fix the UI Panel name of ironic https://review.openstack.org/610085 | 15:32 |
| *** fatdragon has joined #openstack-ansible | 15:34 | |
| *** thuydang has quit IRC | 15:35 | |
| *** thuydang has joined #openstack-ansible | 15:36 | |
| *** vollman has quit IRC | 15:38 | |
| *** goldenfri has joined #openstack-ansible | 15:39 | |
| benkohl | odyssey4me: after checking out stable/rocky and cherry-pick https://review.openstack.org/#/c/608042/ I get this: https://snag.gy/qKFcA7.jpg | 15:40 |
| *** vnogin has joined #openstack-ansible | 15:40 | |
| odyssey4me | benkohl: take a look at the wheel build log in the repo container to see what broke | 15:41 |
| benkohl | maybe I should wait until osa rocky is stable enough... the semester started and I haven't enough time for all the issues :/ | 15:42 |
| *** vnogin has quit IRC | 15:44 | |
| odyssey4me | benkohl: either use queens, which is stable, or wait for rocky's release | 15:47 |
| benkohl | odyssey4me: queens is no option because I want to use ubuntu 18.04, so I will wait... Thanks anyway :) | 15:49 |
| *** markvoelker has quit IRC | 15:51 | |
| *** ansmith has quit IRC | 15:51 | |
| *** markvoelker has joined #openstack-ansible | 15:52 | |
| *** ansmith has joined #openstack-ansible | 15:57 | |
| spatel | Quick question how does anti-affinity understand application should spread out other hypervisor ? | 15:57 |
| spatel | Does it based on instance name ? | 15:57 |
| spatel | if i create web1, web2 , web3 and db1, db2, db3 so how does that fit in anti-affinity server group? | 15:58 |
| *** deployer2 has quit IRC | 16:00 | |
| *** cshen has joined #openstack-ansible | 16:01 | |
| *** skiedude has joined #openstack-ansible | 16:08 | |
| skiedude | So i've finally figured out what container it is thats causing me 504s, I currently have a 3 infra cluster, and when I only shut down the memcached container on the 3rd infra host, everything slows to a crawl | 16:12 |
| skiedude | Looking at the other 2 up memcached containers, their is a log folder in /var/log/memecached, however there is no log file to watch errors for | 16:12 |
| odyssey4me | skiedude: memcache doesn't cluster, so if there's a switch from one to the other, everything cached has to be re-cached | 16:15 |
| *** suggestable has quit IRC | 16:15 | |
| skiedude | so in theory once everything gets recached, it should speed back up | 16:15 |
| skiedude | but it seems to never do that | 16:16 |
| *** mmercer has joined #openstack-ansible | 16:16 | |
| odyssey4me | skiedude: have you checked whether the openstack services saw the disconnected and found the next in line? although I think this may go through haproxy - I can't recall | 16:19 |
| *** dariko has joined #openstack-ansible | 16:19 | |
| skiedude | how exactly would I go about checked that? | 16:21 |
| skiedude | I don't see any vips or data in haproxy for memcache, looking through the roles, it appears most services just have the list of IPs | 16:21 |
| skiedude | for the memcache containers | 16:21 |
| *** fatdragon has quit IRC | 16:24 | |
| *** dariko has quit IRC | 16:25 | |
| skiedude | so I'm thinking its more the other containers trying to to use the downed memcached container, then the container itself being the issue | 16:28 |
| odyssey4me | skiedude: yep, so that means the services themselves are supposed to handle failover | 16:32 |
| odyssey4me | so in their debug logs you might find some clues | 16:32 |
| odyssey4me | it might be a misconfig | 16:32 |
| spatel | odyssey4me: is it possible to do --limit compute1,compute2,compute3 multiple values in openstack-ansible command? | 16:33 |
| *** irclogbot_0 has joined #openstack-ansible | 16:35 | |
| *** flaviosr_ has quit IRC | 16:36 | |
| *** shardy has quit IRC | 16:37 | |
| odyssey4me | spatel: yes, you can also use groups - eg 'compute_hosts' and exclusions like 'compute_hosts:!compute1' | 16:40 |
| odyssey4me | spatel: https://docs.ansible.com/ansible/2.6/user_guide/intro_patterns.html | 16:41 |
| *** irclogbot_0 has quit IRC | 16:42 | |
| *** cshen has quit IRC | 16:43 | |
| *** cshen has joined #openstack-ansible | 16:45 | |
| *** spatel has quit IRC | 16:48 | |
| *** spatel has joined #openstack-ansible | 16:48 | |
| spatel | perfect! | 16:48 |
| spatel | I just want to run playbook on set of compute machine | 16:48 |
| *** DanyC has quit IRC | 16:49 | |
| spatel | This should work for me --limit compute1,compute2,compute3 | 16:49 |
| goldenfri | I am adding some compute nodes they are getting stuck on Add keys (primary keyserver) and the fallback is not working either. Its showing the urls as: u'hkp://keyserver.ubuntu.com:80' is that normal? | 17:06 |
| cloudnull | o/ all | 17:07 |
| *** Bhujay has joined #openstack-ansible | 17:07 | |
| cloudnull | mgariepy vrobert if there's something that we can do to improve that calculation (the nova_wsgi_processes calculation) it'd be wonderful | 17:08 |
| cloudnull | odyssey4me im around now | 17:08 |
| cloudnull | still seeing that resolved issue? | 17:08 |
| odyssey4me | cloudnull: I managed to work it out, thanks. | 17:09 |
| cloudnull | what was it ? | 17:09 |
| mgariepy | cloudnull, 16 process if you have only 10 cores might be a bit too much ? basing the count on cores instead of thread would be probably a better default. | 17:11 |
| odyssey4me | Well, in nodepool we use glean so that we can write out the network/resolver configs from config-drive... and glean is massively lighter than cloud-init + nova-agent | 17:11 |
| cloudnull | mgariepy ++ | 17:11 |
| cloudnull | that probably makes a lot more sense | 17:11 |
| cloudnull | ah that's cool! | 17:11 |
| odyssey4me | but in bionic systemd-resolved is turned on by default, and when glean writes out /etc/resolv.conf, systemd-resolved replaces it... so I had to make our image builds disable systemd-resolved to prevent that | 17:11 |
| cloudnull | odyssey4me ^ | 17:12 |
| odyssey4me | otherwise sh*t don't work, yo | 17:12 |
| cloudnull | can you get glean to update the `/etc/systemd/resolved.conf` file ? | 17:12 |
| cloudnull | via glean ? | 17:12 |
| odyssey4me | cloudnull: well, prometheanfire is working on making glean detect and do the right things | 17:13 |
| cloudnull | https://www.freedesktop.org/software/systemd/man/resolved.conf.html | 17:13 |
| cloudnull | ah cool | 17:13 |
| odyssey4me | thanks prometheanfire :) | 17:14 |
| prometheanfire | odyssey4me: cloudnull: https://review.openstack.org/610105 is part 1 | 17:16 |
| prometheanfire | too bad no providers do that though | 17:16 |
| prometheanfire | I suppose I should only write out to /etc/systemd/resolved.conf if that file already exists (aka if resolved is installed) | 17:17 |
| cloudnull | prometheanfire = Add icons in the PWA manifest ? | 17:17 |
| prometheanfire | cloudnull: wat | 17:17 |
| goldenfri | please continue to ignore my question, I'm dumb and figured it out. :) | 17:17 |
| cloudnull | that review number | 17:18 |
| odyssey4me | lol | 17:18 |
| prometheanfire | wrong link :| | 17:18 |
| prometheanfire | https://review.openstack.org/610107 | 17:18 |
| *** fatdragon has joined #openstack-ansible | 17:18 | |
| prometheanfire | that's just if using glean with networkd (which you can force with 'glean --distro networkd') | 17:19 |
| *** fatdragon has quit IRC | 17:19 | |
| prometheanfire | the generic stuff will follow thisafternoon | 17:19 |
| cloudnull | yes that resolved.conf file is part of systemd | 17:19 |
| cloudnull | and systemd-resolved will use that for all of its config as needed | 17:19 |
| prometheanfire | it wasn't always, forget when it was added, but trusty may not have it, which means I should only write the file if /etc/systemd/resolved.conf exists (and then only set DNS= within that file, leaving other values) | 17:21 |
| prometheanfire | glean runs on more than one OS | 17:21 |
| *** faizy98 has joined #openstack-ansible | 17:25 | |
| *** mmercer has quit IRC | 17:33 | |
| *** mmercer has joined #openstack-ansible | 17:44 | |
| *** Bhujay has quit IRC | 17:50 | |
| mgariepy | anyone here tested neutron networking-ovn ? | 17:53 |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: MNAIO: Do not use apt-cacher-ng on the MNAIO host https://review.openstack.org/610083 | 17:54 |
| *** mmercer has quit IRC | 17:57 | |
| *** mmercer has joined #openstack-ansible | 17:58 | |
| odyssey4me | mgariepy: I think jamesdenton has. | 18:00 |
| *** skiedude has quit IRC | 18:07 | |
| *** mmercer has quit IRC | 18:21 | |
| *** olivierb has quit IRC | 18:21 | |
| jamesdenton | yo | 19:36 |
| jamesdenton | mgariepy define "testing" | 19:36 |
| *** electrofelix has quit IRC | 19:37 | |
| mgariepy | did you tryed it ? | 19:39 |
| mgariepy | is it working ? | 19:39 |
| mgariepy | jamesdenton, ^^ | 19:39 |
| jamesdenton | Needs this: https://review.openstack.org/#/c/584069/ | 19:40 |
| jamesdenton | which, surprising just passed checks this week | 19:40 |
| jamesdenton | (thanks odyssey4me) | 19:40 |
| jamesdenton | i would also say that it does not fully address NB DB HA | 19:40 |
| jamesdenton | but it should be functional, albeit not production ready | 19:41 |
| mgariepy | what kind of test did you do with it ? did any load test on it ? | 19:41 |
| jamesdenton | no, i did not perform any official load testing, only functional tests (i.e. connectivity) | 19:42 |
| jamesdenton | and security groups IIRC | 19:42 |
| mgariepy | ok | 19:43 |
| jamesdenton | are you interested in deploying it? | 19:43 |
| mgariepy | maybe, i was mostly wondering is somebody here tested it a bit. | 19:47 |
| jamesdenton | i gotcha. I'm hoping to dedicate some cycles to it soon, if that helps | 19:48 |
| mgariepy | ok if i get time to test it a bit, i'll probalby ping you then. but it seems interesting to simplify the network a bit. | 19:51 |
| jamesdenton | agreed. getting rid of DHCP and L3 agents would be nice. | 19:52 |
| *** DanyC has joined #openstack-ansible | 20:00 | |
| *** vnogin has joined #openstack-ansible | 20:22 | |
| *** vnogin has quit IRC | 20:26 | |
| *** ianychoi has joined #openstack-ansible | 20:35 | |
| *** ansmith has quit IRC | 20:40 | |
| jrosser | odyssey4me: you around? | 20:46 |
| *** dave-mccowan has quit IRC | 20:57 | |
| *** spatel has quit IRC | 20:57 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Add a guard so we don't allow for duplicate config https://review.openstack.org/610162 | 20:58 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-nspawn_container_create master: Add a guard so we don't allow for duplicate config https://review.openstack.org/610162 | 20:59 |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: Upgrade ceph to mimic release https://review.openstack.org/610165 | 21:00 |
| *** spatel has joined #openstack-ansible | 21:03 | |
| *** DanyC_ has joined #openstack-ansible | 21:03 | |
| *** cshen has quit IRC | 21:04 | |
| *** strattao has joined #openstack-ansible | 21:06 | |
| *** DanyC has quit IRC | 21:07 | |
| *** nsmeds has joined #openstack-ansible | 21:24 | |
| *** jamesdenton has quit IRC | 21:24 | |
| *** DanyC_ has quit IRC | 21:32 | |
| *** thuydang has quit IRC | 21:41 | |
| *** thuydang has joined #openstack-ansible | 21:41 | |
| *** ansmith has joined #openstack-ansible | 21:43 | |
| *** thuydang has quit IRC | 21:45 | |
| *** thuydang has joined #openstack-ansible | 21:46 | |
| *** tosky has quit IRC | 22:02 | |
| openstackgerrit | Merged openstack/openstack-ansible-nspawn_container_create master: Add a guard so we don't allow for duplicate config https://review.openstack.org/610162 | 22:08 |
| *** spatel has quit IRC | 22:11 | |
| *** pcaruana has quit IRC | 22:20 | |
| openstackgerrit | Merged openstack/openstack-ansible-ops master: add lxc3 support https://review.openstack.org/609800 | 22:25 |
| *** strattao has quit IRC | 22:34 | |
| *** strattao has joined #openstack-ansible | 22:41 | |
| *** strattao has quit IRC | 22:41 | |
| *** weezS has quit IRC | 22:45 | |
| *** elbragstad has quit IRC | 22:56 | |
| *** spatel has joined #openstack-ansible | 22:57 | |
| *** nsmeds has quit IRC | 23:01 | |
| *** spatel has quit IRC | 23:01 | |
| *** spatel has joined #openstack-ansible | 23:03 | |
| *** spatel has quit IRC | 23:08 | |
| *** elbragstad has joined #openstack-ansible | 23:15 | |
| *** elbragstad has quit IRC | 23:15 | |
| *** gyee has quit IRC | 23:52 | |
| *** spatel has joined #openstack-ansible | 23:55 | |
| *** spatel has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
