| sdake | cleanup-images -f | 00:04 |
|---|---|---|
| sdake | will do the trick | 00:04 |
| sdake | but yu ha eto stop the containers first | 00:04 |
| sdake | if your cleaning up in this way i'd recommend a reboot as well | 00:05 |
| *** salv-orl_ has quit IRC | 00:05 | |
| sdake | neutron leaves around some junk | 00:05 |
| sdake | that cleanup-hosts doesn't get iiuc | 00:05 |
| manjeets | sdake thanks tools/cleanup -f helped | 00:13 |
| sdake | np | 00:13 |
| * manjeets works on neutron and he started neutron 6 months ago | 00:13 | |
| manjeets | lol | 00:14 |
| *** achanda has quit IRC | 00:19 | |
| *** unicell has quit IRC | 00:19 | |
| *** weiyu has joined #kolla | 00:27 | |
| *** jtriley has joined #kolla | 00:37 | |
| *** ssurana has quit IRC | 00:39 | |
| *** ccesario_ has joined #kolla | 00:40 | |
| *** sdake has quit IRC | 00:41 | |
| openstackgerrit | Merged openstack/kolla-mesos: Attach external NIC to a NAT-Network if on Wi-Fi https://review.openstack.org/295584 | 00:54 |
| *** vhosakot has joined #kolla | 00:54 | |
| openstackgerrit | Vikram Hosakote proposed openstack/kolla: Revert libvirt change in heka-openstack.toml.j2 in master https://review.openstack.org/296649 | 00:54 |
| *** vhosakot has quit IRC | 01:13 | |
| *** vhosakot has joined #kolla | 01:14 | |
| *** Jeffrey4l has joined #kolla | 01:15 | |
| *** achanda has joined #kolla | 01:19 | |
| *** achanda has quit IRC | 01:23 | |
| *** alisonh has joined #kolla | 01:24 | |
| *** vhosakot has quit IRC | 01:25 | |
| *** vhosakot has joined #kolla | 01:25 | |
| *** vishwanathj has quit IRC | 01:26 | |
| *** vhosakot has quit IRC | 01:33 | |
| *** achanda has joined #kolla | 01:34 | |
| *** vhosakot has joined #kolla | 01:34 | |
| *** jtriley has quit IRC | 01:43 | |
| *** jtriley has joined #kolla | 01:43 | |
| *** iceyao has joined #kolla | 01:53 | |
| *** sdake has joined #kolla | 01:57 | |
| *** Jeffrey4l has quit IRC | 01:58 | |
| sdake | evening | 01:58 |
| *** achanda has quit IRC | 01:59 | |
| vhosakot | evening... weird mysql gate failures today... | 02:02 |
| *** achanda has joined #kolla | 02:05 | |
| *** vhosakot has quit IRC | 02:07 | |
| *** vhosakot has joined #kolla | 02:07 | |
| sdake | ya lots of misbehavior | 02:09 |
| vhosakot | sdake: http://logs.openstack.org/18/296818/3/check/gate-kolla-dsvm-deploy-centos-binary/cb8ae3a/console.html#_2016-03-23_22_43_02_531 | 02:13 |
| vhosakot | sdake: the python mysql package has gone missing in gate ? | 02:13 |
| sdake | not sure what that problem is | 02:24 |
| vhosakot | sdake: my recheck saw the same error again... | 02:26 |
| sdake | vhosakot that is the toolbox container missing the dependency | 02:27 |
| sdake | not the gate | 02:28 |
| vhosakot | ah ok, I see kolla_toolbox is failing to create haproxy mysql user in gate | 02:29 |
| sdake | there was a recent change to the openstack-base | 02:33 |
| sdake | i python-muranoclient \ | 02:34 |
| sdake | - python-PyMySQL \ | 02:34 |
| sdake | + python-netaddr \ | 02:35 |
| *** cemason has quit IRC | 02:38 | |
| *** vhosakot has quit IRC | 02:39 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix regression https://review.openstack.org/296868 | 02:39 |
| sdake | asalkeld note i'm not sure if that actually fixes a regression or not | 02:50 |
| asalkeld_ | k | 02:50 |
| *** vhosakot has joined #kolla | 02:58 | |
| *** cemason has joined #kolla | 03:02 | |
| *** vishwanathj has joined #kolla | 03:02 | |
| *** jtriley has quit IRC | 03:02 | |
| *** jtriley has joined #kolla | 03:03 | |
| *** jasonsb has joined #kolla | 03:03 | |
| openstackgerrit | Gerard Braad proposed openstack/kolla: Include docker python install and set shared flag https://review.openstack.org/290973 | 03:04 |
| vhosakot | sdake: I see you added python-PyMySQl back to openstack-base that was removed in https://review.openstack.org/#/c/295048/5/docker/openstack-base/Dockerfile.j2....looks like that was the issue.. let us wait for gate results... | 03:04 |
| patchbot | vhosakot: patch 295048 - kolla - Improve openstack-base image for centos binary (MERGED) | 03:04 |
| vhosakot | https://review.openstack.org/#/c/295048/5/docker/openstack-base/Dockerfile.j2 i meant | 03:05 |
| patchbot | vhosakot: patch 295048 - kolla - Improve openstack-base image for centos binary (MERGED) | 03:05 |
| sdake | i dont kno if it was the issue or not | 03:05 |
| sdake | know | 03:05 |
| sdake | but it is suspicious | 03:06 |
| vhosakot | cool | 03:06 |
| *** iceyao has quit IRC | 03:08 | |
| sdake | centos binary nad source are failing the gat ein my patches though | 03:10 |
| sdake | the gates are slow atm | 03:12 |
| sdake | need more capacity ;) | 03:12 |
| sdake | asalkeld you want to throw me a bone on heat config | 03:13 |
| asalkeld | What's the issue? | 03:14 |
| asalkeld_ | sdake: ? what help do you need... | 03:15 |
| sdake | asalkeld creating a heat stack with kolla fails | 03:16 |
| sdake | authorization failure | 03:16 |
| sdake | however heat api responds to the request | 03:16 |
| asalkeld_ | logs? | 03:16 |
| sdake | so its heat api tha is suffering an authorization failure | 03:16 |
| sdake | the logs are not hepful | 03:16 |
| sdake | i can turn on debug if you would like to see those though | 03:17 |
| sdake | asalkeld https://bugs.launchpad.net/kolla/+bug/1553565 | 03:17 |
| openstack | Launchpad bug 1553565 in kolla "heat is DOA in mitaka-3" [Critical,In progress] - Assigned to Steven Dake (sdake) | 03:17 |
| asalkeld_ | yeah, that may help | 03:18 |
| sdake | meoment let me deploy with debug | 03:18 |
| *** sdake_ has joined #kolla | 03:21 | |
| *** sdake has quit IRC | 03:22 | |
| sdake_ | asalkeld heatpai: http://paste.fedoraproject.org/344469/78992314/ -> http://paste.fedoraproject.org/344469/78992314 | 03:25 |
| sdake_ | heat engine http://paste.fedoraproject.org/344470/14587900/ -> http://paste.fedoraproject.org/344470/14587900 | 03:26 |
| sdake_ | note we may not be seting up trusts correctly | 03:26 |
| asalkeld_ | sdake_: can you do a "heat event-list" | 03:27 |
| sdake_ | hea tengine contains a authorization failure | 03:27 |
| asalkeld_ | ERROR heat.common.keystoneclient [req-bebc5995-52e2-4776-a9e8-1c62085c0eec admin admin - default default] Domain admin client authentication failed | 03:27 |
| sdake_ | | steak | 78ad761f-8480-4f9f-88bf-10d8a090c6d3 | Authorization failed. | CREATE_FAILED | 2016-03-24T03:24:28 | | 03:27 |
| asalkeld_ | sdake_: so api is fine, it's the login to the admin domain | 03:28 |
| asalkeld_ | (done by the engine) | 03:28 |
| sdake_ | https://github.com/openstack/kolla/blob/master/docker/heat/heat-api/extend_start.sh#L7 | 03:29 |
| sdake_ | that is what bootstraps the domain | 03:29 |
| sdake_ | i am pretty sure there is a domain issue | 03:29 |
| sdake_ | i just dont know hte magic incantation required by heat to get it to register prpoerly | 03:29 |
| sdake_ | there are alot of bugs filed on this around the internets | 03:30 |
| asalkeld_ | looking... | 03:30 |
| sdake_ | what tht code does is register a heat domain if there is not one already registered | 03:31 |
| sdake_ | we have a similar domain problem with magnum | 03:31 |
| sdake_ | there is probably something missing from this file: https://github.com/openstack/kolla/blob/master/ansible/roles/heat/templates/heat.conf.j2 | 03:33 |
| asalkeld_ | sdake_: can you edit this: https://github.com/openstack/kolla/blob/master/ansible/roles/heat/templates/heat.conf.j2#L12 | 03:35 |
| asalkeld_ | stack_user_domain_name = heat | 03:35 |
| sdake_ | let me try that | 03:35 |
| asalkeld_ | that is the domain for stack users | 03:35 |
| asalkeld_ | confusing config name | 03:35 |
| sdake_ | [sdake@minime-03 tools]$ time sudo ./kolla-ansible reconfigure | 03:37 |
| sdake_ | real1m3.461s | 03:38 |
| sdake_ | create in progress!! | 03:38 |
| asalkeld_ | nice | 03:38 |
| sdake_ | wow that was easy | 03:39 |
| sdake_ | asalkeld_ i dont want to take credit for your work | 03:39 |
| sdake_ | do you want to submit the patch | 03:39 |
| asalkeld_ | haha | 03:39 |
| asalkeld_ | sure, you tested tho | 03:39 |
| sdake_ | ok i'll coauthor you then | 03:39 |
| sdake_ | asalkeld@miranits.com? | 03:40 |
| asalkeld_ | yar | 03:40 |
| sdake_ | asalkeld_ does the rest of that config look correct to you? | 03:41 |
| asalkeld_ | the trust bit looks good | 03:41 |
| *** tfukushima has joined #kolla | 03:42 | |
| asalkeld_ | seems ok | 03:42 |
| openstackgerrit | Gerard Braad proposed openstack/kolla: Include docker python install and set shared flag https://review.openstack.org/290973 | 03:42 |
| *** salv-orlando has joined #kolla | 03:43 | |
| *** ccesario_ has quit IRC | 03:43 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Make heat work https://review.openstack.org/296878 | 03:44 |
| sdake_ | asalkeld_ that is a bad variable name imo :) | 03:44 |
| *** ccesario_ has joined #kolla | 03:45 | |
| asalkeld_ | yeah, you can understand the intent only once you know what it is ment for | 03:45 |
| asalkeld_ | :-O | 03:45 |
| sdake_ | vhosakot are you able to get vms to boot? | 03:46 |
| vhosakot | yes... worked 3 days ago... haven't booted in three days since I'm fixing lua parser issue.. I run init_runonce, and do nova boot | 03:47 |
| *** achanda has quit IRC | 03:49 | |
| sdake_ | vhosakot yes see if it works now pleae | 03:49 |
| sdake_ | my horizon doesn't work and my nova vms are not booting | 03:49 |
| *** salv-orlando has quit IRC | 03:49 | |
| sdake_ | but it was working about 10 days ago when i did a demo | 03:49 |
| vhosakot | latest master ? when did you pull ? both horizon and nova vm booted fine for tme 3 days ago for sure... | 03:50 |
| sdake_ | my images are 2 days old | 03:50 |
| sdake_ | centos source | 03:50 |
| vhosakot | ok, i use ubuntu source.. let me rebuild not.. are all you containers up ? can you curl the VIP, auth-url ? | 03:55 |
| vhosakot | not=now | 03:55 |
| sdake_ | 2016-03-24 03:54:31.609 33 WARNING nova.image.glance [req-3fb6bf04-3a2f-4249-9eb4-7797e5f4069a f1981f1ece3b4718b68722b6ecd3b226 f79efb64634a422ebfc0a5aeba673bb6 - - -] No protocol specified in for api_server 'http://192.168.1.103:9292', please update [glance] api_servers with fully qualified url including scheme (http / https) | 03:56 |
| vhosakot | ah, 9292 is glance api port... you could try "netstat -pan | grep 9292" and see if glance api's PID is listening | 03:58 |
| sdake_ | no | 03:58 |
| sdake_ | that isn't the problem | 03:58 |
| vhosakot | oh ok | 03:58 |
| *** Marga_ has quit IRC | 03:58 | |
| *** Marga_ has joined #kolla | 03:59 | |
| *** Marga_ has quit IRC | 03:59 | |
| *** Marga_ has joined #kolla | 03:59 | |
| openstackgerrit | Gerard Braad proposed openstack/kolla: Include docker python install and set shared flag https://review.openstack.org/290973 | 04:00 |
| openstackgerrit | Gerard Braad proposed openstack/kolla: Include docker python install and set shared flag https://review.openstack.org/290973 | 04:06 |
| openstackgerrit | Gerard Braad proposed openstack/kolla: Include docker python install https://review.openstack.org/290973 | 04:07 |
| *** absubram has joined #kolla | 04:07 | |
| *** absubram_ has joined #kolla | 04:08 | |
| *** absubram has quit IRC | 04:12 | |
| *** absubram_ is now known as absubram | 04:12 | |
| *** jtriley has quit IRC | 04:22 | |
| *** vishwanathj is now known as vishwanathj_zzz | 04:24 | |
| *** sdake_ is now known as sdake | 04:29 | |
| *** absubram has quit IRC | 04:36 | |
| *** dave-mccowan has quit IRC | 04:45 | |
| *** weiyu has quit IRC | 04:46 | |
| *** achanda has joined #kolla | 04:59 | |
| *** salv-orlando has joined #kolla | 05:02 | |
| *** weiyu_ has joined #kolla | 05:14 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix inconsistancy in globals.yml https://review.openstack.org/296892 | 05:15 |
| *** sdake has quit IRC | 05:16 | |
| *** salv-orlando has quit IRC | 05:17 | |
| *** sdake has joined #kolla | 05:18 | |
| *** sdake_ has joined #kolla | 05:20 | |
| *** sdake has quit IRC | 05:23 | |
| *** ssurana has joined #kolla | 05:33 | |
| *** weiyu_ has quit IRC | 05:34 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] This fixes a gate failure https://review.openstack.org/296868 | 05:36 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] This fixes a gate failure https://review.openstack.org/296868 | 05:38 |
| sdake_ | akwasnie around | 05:38 |
| sdake_ | any core reiewers around for that matter? | 05:38 |
| asalkeld_ | o/ | 05:42 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix consistently failing CentOS gate https://review.openstack.org/296868 | 05:44 |
| sdake_ | need someone to ack the reno patch | 05:45 |
| *** sdake_ is now known as sdake | 05:45 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix consistently failing CentOS gate https://review.openstack.org/296868 | 05:51 |
| *** vhosakot_ has joined #kolla | 06:09 | |
| *** weiyu_ has joined #kolla | 06:09 | |
| *** vhosakot has quit IRC | 06:10 | |
| *** salv-orlando has joined #kolla | 06:21 | |
| *** vhosakot_ has quit IRC | 06:21 | |
| *** salv-orlando has quit IRC | 06:24 | |
| *** coolsvap has joined #kolla | 06:26 | |
| *** daneyon_ has quit IRC | 06:34 | |
| *** daneyon has joined #kolla | 06:34 | |
| sdake | asalkeld_ can you ack https://review.openstack.org/#/c/296868/5 | 06:36 |
| patchbot | sdake: patch 296868 - kolla - Fix consistently failing CentOS gate | 06:36 |
| sdake | fixes broken gate | 06:36 |
| asalkeld_ | k | 06:36 |
| nihilifer | \o/ | 06:37 |
| sdake | thanks asalkeld | 06:37 |
| * nihilifer will look at patches too, when gertty will end refreshing | 06:37 | |
| sdake | morning nihilifer | 06:37 |
| sdake | nihilifer i need an ack on https://review.openstack.org/#/c/296125/ but i want the gate passing first | 06:39 |
| patchbot | sdake: patch 296125 - kolla - Add reno support | 06:39 |
| nihilifer | ok | 06:39 |
| openstackgerrit | Merged openstack/kolla: Fix consistently failing CentOS gate https://review.openstack.org/296868 | 06:40 |
| sdake | nihilifer i have done a recheck on that reno patch | 06:45 |
| sdake | once it works, i'll let you know | 06:45 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix consistently failing CentOS gate https://review.openstack.org/296911 | 06:45 |
| nihilifer | sdake: isn't a rebase needed? | 06:46 |
| sdake | nope | 06:46 |
| sdake | the gate runs against master plus whatever patches are in the path stream | 06:46 |
| openstackgerrit | Gerard Braad proposed openstack/kolla: Change CentOS in documentation https://review.openstack.org/296913 | 06:46 |
| nihilifer | ah, ok | 06:46 |
| sdake | i was wrong oirginally when i posted that email | 06:46 |
| sdake | and infra (I think Jeremy) corrected me | 06:46 |
| sdake | or i the case of stable/mitaka, runs against stable/mitaka | 06:47 |
| nihilifer | good to know | 06:47 |
| sdake | nihilifer dont jusst rubber stamp the reno change - please review it appropriately :) | 06:48 |
| openstackgerrit | Merged openstack/kolla-mesos: Fix auto doc generation https://review.openstack.org/295074 | 06:50 |
| openstackgerrit | Merged openstack/kolla: Include docker python install https://review.openstack.org/290973 | 06:50 |
| *** unicell has joined #kolla | 06:57 | |
| sdake | nihilifer this pretty much needs a rubber stamp: https://review.openstack.org/#/c/296878/1 | 06:58 |
| patchbot | sdake: patch 296878 - kolla - Make heat work | 06:58 |
| openstackgerrit | Gerard Braad proposed openstack/kolla: Change CentOS in documentation https://review.openstack.org/296913 | 06:59 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Parse and render yaml files in the proper order https://review.openstack.org/295831 | 07:01 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Fix OpenStack services registration https://review.openstack.org/291251 | 07:01 |
| sdake | kolla is #9 on reviews for mitaka and #5 on commits | 07:03 |
| sdake | busy busy project | 07:03 |
| *** ssurana has quit IRC | 07:03 | |
| sdake | interestingly kolla is #1 for commits and reviews for cisco http://stackalytics.com/?metric=commits&company=cisco%20systems | 07:04 |
| sdake | and #18 on commits for lifetime of openstack | 07:07 |
| sdake | #17 on reviews | 07:09 |
| *** mikelk has joined #kolla | 07:11 | |
| coolsvap | :) | 07:13 |
| sdake | asalkeld i was looking at heat - I'm still i the top 10 contributors even though I havent worked on that project for nearly 3 years | 07:18 |
| *** cristicalin has joined #kolla | 07:19 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Fix string parameter in mesos-dns.conf https://review.openstack.org/295830 | 07:26 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Parse and render yaml files in the proper order https://review.openstack.org/295831 | 07:26 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Encode the ifname before passing to struct https://review.openstack.org/295829 | 07:26 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Fix OpenStack services registration https://review.openstack.org/291251 | 07:26 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Override Mesos config in configuration tests https://review.openstack.org/296924 | 07:26 |
| *** Marga_ has quit IRC | 07:27 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add note about heat being busted for development https://review.openstack.org/290276 | 07:30 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add note about heat being busted for development https://review.openstack.org/290276 | 07:33 |
| *** achanda has quit IRC | 07:33 | |
| *** achanda has joined #kolla | 07:37 | |
| *** Marga_ has joined #kolla | 07:40 | |
| sdake | nihilifer this pased a rechek https://review.openstack.org/#/c/296878/ | 07:41 |
| patchbot | sdake: patch 296878 - kolla - Make heat work | 07:41 |
| sdake | nihilifer this is a cherrypick required for reno in mitaka - needs an ack :) https://review.openstack.org/#/c/296662/1 | 07:44 |
| patchbot | sdake: patch 296662 - kolla (stable/mitaka) - Add PBR generation of version information | 07:44 |
| *** Marga_ has quit IRC | 07:45 | |
| nihilifer | sdake: um... sorry if i'm missing something, but are you sure with change from heat_stack_owner to heat? | 07:46 |
| nihilifer | i see these roles created during bootstrap https://github.com/openstack/kolla/blob/master/docker/heat/heat-api/extend_start.sh#L13-L14 | 07:46 |
| sdake | nihilifer i teste dit and it works | 07:46 |
| sdake | prior it didnt 'work | 07:46 |
| sdake | asalkeld suggested the change | 07:46 |
| sdake | and is a modern het expert | 07:46 |
| sdake | so yes, 100% positive | 07:47 |
| nihilifer | strange :/ but ok | 07:47 |
| sdake | the variable is misnomer | 07:47 |
| sdake | what it mens is the domain user | 07:47 |
| sdake | asalkeld isn't that correct? | 07:47 |
| nihilifer | that worked some moths ago and setting this role name is what openstack docs are suggesting, but ok, i believe you and don't insist ;) | 07:48 |
| nihilifer | +2 | 07:48 |
| sdake | well i want to get it right | 07:49 |
| sdake | so it doesn't break later | 07:49 |
| sdake | what do the docs suggest again? | 07:49 |
| sdake | note I read all kinds of bug reports that the docs in upstream are wrong | 07:49 |
| sdake | nihilifer heat has never worked once [trusts] section was added | 07:50 |
| sdake | once authentication type was changed from password heat was broken | 07:50 |
| sdake | sam did that, but he didn't fix it, he just broke it so it would be consistent | 07:50 |
| sdake | don't ask me the logic of that ;) | 07:50 |
| sdake | its been broken for atleast 3 months | 07:51 |
| *** achanda has quit IRC | 07:56 | |
| *** salv-orlando has joined #kolla | 07:57 | |
| *** salv-orlando has quit IRC | 08:02 | |
| *** sdake has quit IRC | 08:02 | |
| *** sdake has joined #kolla | 08:06 | |
| *** stvnoyes has quit IRC | 08:06 | |
| sdake | hey sorry network dropped out | 08:07 |
| *** cristicalin has quit IRC | 08:07 | |
| *** stvnoyes has joined #kolla | 08:08 | |
| *** athomas has joined #kolla | 08:09 | |
| *** achanda has joined #kolla | 08:10 | |
| *** akwasnie1 has joined #kolla | 08:16 | |
| *** akwasnie1 has left #kolla | 08:16 | |
| *** sdake has quit IRC | 08:18 | |
| akwasnie | sdake: hi | 08:18 |
| *** salv-orl_ has joined #kolla | 08:21 | |
| *** weiyu_ has quit IRC | 08:24 | |
| *** kproskurin has joined #kolla | 08:25 | |
| *** mbound has joined #kolla | 08:26 | |
| *** shardy has joined #kolla | 08:27 | |
| *** weiyu has joined #kolla | 08:28 | |
| *** jmccarthy has quit IRC | 08:32 | |
| *** jmccarthy has joined #kolla | 08:35 | |
| openstackgerrit | Merged openstack/kolla: Add reno support https://review.openstack.org/296125 | 08:39 |
| openstackgerrit | Merged openstack/kolla: Add PBR generation of version information https://review.openstack.org/296662 | 08:39 |
| *** macsz has joined #kolla | 08:40 | |
| *** achanda has quit IRC | 08:45 | |
| *** Jeffrey4l has joined #kolla | 08:50 | |
| *** achanda has joined #kolla | 08:56 | |
| *** coolsvap has quit IRC | 09:18 | |
| *** achanda has quit IRC | 09:26 | |
| *** jasonsb has quit IRC | 09:28 | |
| *** salv-orl_ has quit IRC | 09:37 | |
| *** salv-orlando has joined #kolla | 09:37 | |
| *** coolsvap has joined #kolla | 09:43 | |
| *** athomas has quit IRC | 09:45 | |
| *** eandersson_ has quit IRC | 09:48 | |
| *** eandersson_ has joined #kolla | 09:49 | |
| *** athomas has joined #kolla | 09:50 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Only copy the necessary config files for Glance https://review.openstack.org/296982 | 09:58 |
| *** Jeffrey4l has quit IRC | 10:06 | |
| *** jasonsb has joined #kolla | 10:06 | |
| *** kproskurin has quit IRC | 10:08 | |
| *** Serlex has joined #kolla | 10:09 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add Ubuntu binary build and deploy gate https://review.openstack.org/292930 | 10:09 |
| openstackgerrit | Merged openstack/kolla: Make heat work https://review.openstack.org/296878 | 10:12 |
| *** pbourke has quit IRC | 10:15 | |
| *** pbourke has joined #kolla | 10:15 | |
| *** Jeffrey4l has joined #kolla | 10:18 | |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Make heat work https://review.openstack.org/296992 | 10:18 |
| *** ccesario_ has quit IRC | 10:20 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Upgrade elasticsearch https://review.openstack.org/296993 | 10:20 |
| *** Jeffrey4l has quit IRC | 10:35 | |
| *** akwasnie1 has joined #kolla | 10:56 | |
| *** akwasnie1 has left #kolla | 10:56 | |
| *** iceyao has joined #kolla | 11:01 | |
| *** weiyu has quit IRC | 11:08 | |
| *** brad[] has quit IRC | 11:08 | |
| *** brad[] has joined #kolla | 11:09 | |
| *** coolsvap has quit IRC | 11:33 | |
| *** jtriley has joined #kolla | 11:34 | |
| *** asalkeld_ has quit IRC | 11:38 | |
| *** mlima has joined #kolla | 11:40 | |
| *** jtriley has quit IRC | 11:46 | |
| openstackgerrit | Andrey Pavlov proposed openstack/kolla-mesos: Fix marathon framework autodetection https://review.openstack.org/297054 | 11:52 |
| *** rhallisey has joined #kolla | 11:52 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Parse and render yaml files in the proper order https://review.openstack.org/295831 | 11:55 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Fix OpenStack services registration https://review.openstack.org/291251 | 11:55 |
| mlima | SamYaple, atm storage_interface only works with ceph, but for any other will not work. I still do not know how, but it must have a way to get it "flexible" and independent of the backend used. (Ceph, lvm, any other) | 11:59 |
| *** salv-orl_ has joined #kolla | 12:02 | |
| *** salv-orl_ has quit IRC | 12:02 | |
| *** kproskurin has joined #kolla | 12:02 | |
| *** salv-orlando has quit IRC | 12:05 | |
| *** dwalsh has joined #kolla | 12:07 | |
| *** mbound has quit IRC | 12:16 | |
| *** dave-mccowan has joined #kolla | 12:17 | |
| *** mbound has joined #kolla | 12:18 | |
| SamYaple | mlima: if it doesnt work for "others" thats because they arent specifying the storage backend. what merged code does it not work for | 12:23 |
| kproskurin | Hi guys, I trying to rebuild keystone and horizon from master, centos as a base, and got this error: | 12:23 |
| kproskurin | INFO:kolla.cmd.build:horizon:Error unpacking rpm package httpd-2.4.6-40.el7.centos.x86_64 | 12:23 |
| mlima | i know SamYaple | 12:24 |
| openstackgerrit | Merged openstack/kolla: Reduce log errors from heka https://review.openstack.org/296818 | 12:29 |
| *** salv-orlando has joined #kolla | 12:34 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Revert "Fix consistently failing CentOS gate" https://review.openstack.org/297084 | 12:37 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Revert "Fix consistently failing CentOS gate" https://review.openstack.org/297084 | 12:39 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Revert "Make heat work" https://review.openstack.org/297090 | 12:47 |
| SamYaple | asalkeld: you still around? | 12:48 |
| SamYaple | rhallisey: pbourke can you please give priority to reviewing the two reverts above? I have blocked the backports to the mitaka branch, but those are still bugs in mitaka until the revert and subsequent proper fix happen | 12:51 |
| pbourke | looking now | 12:51 |
| *** akwasnie1 has joined #kolla | 12:51 | |
| *** akwasnie1 has left #kolla | 12:51 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Revert "Make heat work" https://review.openstack.org/297090 | 12:53 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Revert "Make heat work" https://review.openstack.org/297090 | 12:54 |
| SamYaple | dont like that web editor at all | 12:54 |
| *** mbound has quit IRC | 12:58 | |
| *** mbound has joined #kolla | 12:58 | |
| pbourke | do we need bug id for reverts? | 12:59 |
| pbourke | not that I dont trust you but there's no concrete info in the commit msg to justify the revert | 12:59 |
| SamYaple | no not for reverts | 13:00 |
| *** dwalsh has quit IRC | 13:00 | |
| SamYaple | the revert is for incorectly merged patch | 13:00 |
| SamYaple | if you look, there is also no justification for merging the patch ('i suspect a new version', 'i dont know the cause, this seems to fix it') | 13:01 |
| SamYaple | the worst part is this was commented on and dismissed | 13:02 |
| SamYaple | oh i should clarify rhallisey pbourke , please to +W these patches just yet, just review them | 13:02 |
| SamYaple | we dont want to go reverting each others work all willynilly | 13:03 |
| SamYaple | its just the proper fix for these patches cant be posted without these reverts | 13:03 |
| *** jtriley has joined #kolla | 13:05 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image https://review.openstack.org/288936 | 13:09 |
| *** jtriley has quit IRC | 13:10 | |
| *** dwalsh has joined #kolla | 13:11 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: make kolla-toolbox build on debian https://review.openstack.org/297104 | 13:12 |
| *** gfidente has joined #kolla | 13:13 | |
| *** absubram has joined #kolla | 13:33 | |
| *** absubram_ has joined #kolla | 13:34 | |
| *** cemason has quit IRC | 13:34 | |
| *** absubram has quit IRC | 13:38 | |
| *** absubram_ is now known as absubram | 13:38 | |
| *** dwalsh has quit IRC | 13:41 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: Make swift containers build with debian https://review.openstack.org/297121 | 13:45 |
| *** achanda has joined #kolla | 13:50 | |
| *** achanda has quit IRC | 13:50 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: Make nova containers build with debian https://review.openstack.org/297169 | 13:56 |
| *** dtturner has quit IRC | 13:59 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: Make swift containers build with debian https://review.openstack.org/297121 | 13:59 |
| sbezverk | elemoine_ ping | 14:01 |
| elemoine_ | sbezverk: moment busy | 14:01 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: make kolla-toolbox build with debian https://review.openstack.org/297104 | 14:01 |
| *** mgoddard_ has joined #kolla | 14:01 | |
| sbezverk | elemoine_ np whenever you have a spare moment | 14:02 |
| *** inc0 has joined #kolla | 14:03 | |
| inc0 | good morning | 14:04 |
| *** mgoddard has quit IRC | 14:04 | |
| SamYaple | morning inc0 | 14:08 |
| *** haplo37 has joined #kolla | 14:12 | |
| *** mlima has quit IRC | 14:13 | |
| *** mgoddard_ has quit IRC | 14:23 | |
| *** mgoddard has joined #kolla | 14:23 | |
| *** aginwala has joined #kolla | 14:24 | |
| ccesario | morning!!! | 14:28 |
| *** aginwala has quit IRC | 14:29 | |
| *** dwalsh has joined #kolla | 14:30 | |
| *** vishwanathj_zzz is now known as vishwanathj | 14:33 | |
| *** aginwala has joined #kolla | 14:35 | |
| *** sdake has joined #kolla | 14:37 | |
| sdake | morning | 14:37 |
| elemoine_ | sbezverk: hi | 14:40 |
| sbezverk | elemoine_ hi, thank you for pinging back.. | 14:41 |
| sbezverk | elemoine_ I am trying to get in touch with this person Marian Horban <mhorban@mirantis.com> do you know by any chance iirc? | 14:42 |
| elemoine_ | I can find this email in the company directory | 14:43 |
| elemoine_ | can't | 14:43 |
| openstackgerrit | Merged openstack/kolla: Re-set external_network_bridge for l3 agent https://review.openstack.org/296697 | 14:43 |
| sbezverk | elemoine_ I see, thank you for letting me know.. | 14:44 |
| ccesario | SamYaple, did you tested!? | 14:44 |
| *** absubram has quit IRC | 14:45 | |
| SamYaple | pbourke: info preovided | 14:47 |
| SamYaple | sdake: for your review, two reverts https://review.openstack.org/#/c/297084/ https://review.openstack.org/#/c/297090/ | 14:47 |
| patchbot | SamYaple: patch 297084 - kolla - Revert "Fix consistently failing CentOS gate" | 14:47 |
| patchbot | SamYaple: patch 297090 - kolla - Revert "Make heat work" | 14:47 |
| sdake | SamYaple i think both of those rverts rae incorrect | 15:00 |
| sdake | especially the mysql revert since it breaks the gate | 15:00 |
| SamYaple | sdake: it breaks the gate because it was a bad fix | 15:01 |
| SamYaple | i cant make your bad fix work in one patchset | 15:01 |
| SamYaple | but it only breaks centos gate | 15:01 |
| sdake | it is proven that myql 1.2.5 doesn't work on centos | 15:01 |
| *** mbound has quit IRC | 15:01 | |
| SamYaple | thats not true at all.... | 15:01 |
| sdake | do a stream which fixes the gate, then i'd consider revert | 15:01 |
| SamYaple | you need to make your packages install better. we arent merging that into mitaka | 15:01 |
| SamYaple | im not working on centos | 15:01 |
| SamYaple | i dont know centos | 15:02 |
| openstackgerrit | Merged openstack/kolla: Change CentOS in documentation https://review.openstack.org/296913 | 15:02 |
| SamYaple | centos has this issue and that fix reverts a fix from preliberty for ubuntu | 15:02 |
| *** dencaval has quit IRC | 15:02 | |
| SamYaple | make your change not affect ubuntu | 15:02 |
| sdake | i can do that | 15:02 |
| SamYaple | ok revert has to happen first | 15:02 |
| SamYaple | which mean gate goes back to failing | 15:03 |
| sdake | i prefer just to fix it with working gate | 15:03 |
| SamYaple | we can hold off until you have a patchset ready of course | 15:03 |
| SamYaple | thats munging up a revert | 15:03 |
| sdake | ok, well i'll need to take over your revert then | 15:03 |
| sdake | because it needs to be a patch stream | 15:03 |
| SamYaple | you can rebase without taking over | 15:04 |
| SamYaple | just do a checkout insteado cherry-pick | 15:04 |
| SamYaple | then you dont actually take over revert at all | 15:04 |
| sdake | ok i'll try that | 15:04 |
| sdake | still waking up - 8am | 15:04 |
| SamYaple | no prolem sdake | 15:04 |
| SamYaple | ill submit a patch ontop of revert for heat fix too | 15:04 |
| sdake | i am certain that heat ix is correct , because heat works afterwords with a 1 liner ;) | 15:04 |
| SamYaple | you are not correct there | 15:05 |
| SamYaple | because it breaks liberty -> mitaka | 15:05 |
| SamYaple | it was a bad trusts patch | 15:05 |
| SamYaple | ill fix it right | 15:05 |
| sdake | make sure its a strea please | 15:05 |
| *** mgoddard_ has joined #kolla | 15:05 | |
| sdake | so it can be evalluated as one unit | 15:05 |
| SamYaple | as always | 15:05 |
| *** ssurana has joined #kolla | 15:08 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Change heat domain to heat_user_domain https://review.openstack.org/297200 | 15:08 |
| sdake | sam regardign mysql packaging from ubuntu - what is precisely the issue | 15:09 |
| *** mgoddard has quit IRC | 15:09 | |
| sdake | wich 1.2.3 | 15:09 |
| SamYaple | we had several issue with 1.2.3, all bugs fixed in 1.2.5 | 15:09 |
| SamYaple | i posted a link to one such bug | 15:09 |
| *** ruhe2 has joined #kolla | 15:11 | |
| *** jtriley has joined #kolla | 15:12 | |
| *** aginwala has quit IRC | 15:13 | |
| *** inc0 has quit IRC | 15:13 | |
| *** dave-mccowan has quit IRC | 15:13 | |
| *** iceyao has quit IRC | 15:13 | |
| *** yuanying has quit IRC | 15:13 | |
| *** ruhe has quit IRC | 15:13 | |
| *** clayton has quit IRC | 15:13 | |
| *** dmsimard has quit IRC | 15:13 | |
| *** ruhe2 is now known as ruhe | 15:13 | |
| openstackgerrit | Merged openstack/kolla: Fix inconsistancy in globals.yml https://review.openstack.org/296892 | 15:13 |
| *** shardy has quit IRC | 15:14 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: Make kibana container build with debian https://review.openstack.org/297206 | 15:14 |
| *** mgoddard_ has quit IRC | 15:15 | |
| *** mgoddard has joined #kolla | 15:16 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Re-set external_network_bridge for l3 agent https://review.openstack.org/297209 | 15:16 |
| *** akwasnie1 has joined #kolla | 15:20 | |
| *** akwasnie1 has left #kolla | 15:20 | |
| SamYaple | sdake: we should not tag rc2 on the 25th. i thought of a reason | 15:21 |
| SamYaple | sdake: that reason is all other projects tag on the 25th. we should consume the rc2 packages | 15:22 |
| SamYaple | which means we need to tag 26th-ish | 15:22 |
| SamYaple | or later | 15:22 |
| SamYaple | we want to be consuming the rc2 fixes in our rc1 for stability (and to make sure nothing new breask) | 15:22 |
| *** vhosakot has joined #kolla | 15:25 | |
| *** dmsimard has joined #kolla | 15:25 | |
| *** sdake_ has joined #kolla | 15:25 | |
| *** sdake has quit IRC | 15:25 | |
| dmsimard | My bouncer silently disconnected (net split?) so I'm not sure you guys saw that | 15:27 |
| dmsimard | SamYaple: I was looking at that CentOS mysql-python revert and was wondering why mysql-python was even still installed, I thought the transition to pymysql was completed everywhere ? | 15:28 |
| *** inc0 has joined #kolla | 15:29 | |
| *** dave-mccowan has joined #kolla | 15:29 | |
| *** yuanying has joined #kolla | 15:29 | |
| *** clayton has joined #kolla | 15:29 | |
| SamYaple | dmsimard: this is ansible module mysql | 15:31 |
| SamYaple | so we need mysqldb for that | 15:31 |
| dmsimard | oh... | 15:31 |
| vhosakot | SamYaple: sdake_: https://review.openstack.org/#/c/296649/ and https://review.openstack.org/#/c/296012/ need to merge so I can continue lua parser work for libvirtd.log | 15:37 |
| patchbot | vhosakot: patch 296649 - kolla - Revert libvirt change in heka-openstack.toml.j2 in... | 15:37 |
| patchbot | vhosakot: patch 296012 - kolla (stable/mitaka) - Make heka collect libvirtd.log in nova_libvirt con... | 15:37 |
| vhosakot | one fixes broken toml is master and the other backports a fix (copy libvirtd.log to /var/log/kolla/libvirt) to mitaka | 15:38 |
| *** absubram has joined #kolla | 15:39 | |
| *** shardy has joined #kolla | 15:42 | |
| *** mgoddard_ has joined #kolla | 15:45 | |
| *** shardy has quit IRC | 15:46 | |
| *** shardy has joined #kolla | 15:47 | |
| *** shardy has quit IRC | 15:48 | |
| *** mgoddard has quit IRC | 15:49 | |
| *** shardy has joined #kolla | 15:49 | |
| vhosakot | SamYaple: sdake_: ^^^^ | 15:50 |
| sdake_ | vhosakot just getting going dude - and its school hour - give me a bit o ftime plz | 15:51 |
| vhosakot | sdake_: cool.... thanks! say hi to kids :) | 15:51 |
| *** mbound has joined #kolla | 15:55 | |
| sdake_ | sbezverk https://bootstrap.pypa.io/get-pip.py | 15:59 |
| sdake_ | run that | 15:59 |
| *** mikelk has quit IRC | 16:00 | |
| sdake_ | sbezverk after running that | 16:00 |
| sdake_ | pip install -U setuptools tox | 16:00 |
| sdake_ | then try tox -e setuptoolsl | 16:00 |
| sdake_ | rather tox -e releasenotes | 16:00 |
| sbezverk | sdake http://paste.openstack.org/show/491748/ | 16:01 |
| sdake_ | looks good to me | 16:01 |
| sdake_ | type pip show pip | 16:02 |
| sdake_ | SamYaple the assertion about not using heat in the domains in liberty is incorrect | 16:03 |
| sdake_ | we use heat in the domains in liberty - the code hasn't changed in that area except with the checking for a domain | 16:03 |
| *** Kennan has quit IRC | 16:04 | |
| sbezverk | sdake_ http://paste.openstack.org/show/491750/ | 16:04 |
| sdake_ | yup latest version | 16:05 |
| sdake_ | sudo pip install -U setuptools tox | 16:05 |
| sdake_ | sudo pip install -U setuptools tox virtualenv | 16:05 |
| sbezverk | sdake_ http://paste.openstack.org/show/491751/ | 16:05 |
| *** Kennan has joined #kolla | 16:05 | |
| SamYaple | sdake_: i responded to you, you were wrong | 16:06 |
| *** inc0 has quit IRC | 16:06 | |
| sbezverk | sdake_ it is done | 16:07 |
| sdake_ | SamYaple ok i'll check it out | 16:08 |
| sdake_ | sbezverk ok tox -e releasenotes | 16:08 |
| sbezverk | sdake_ generates the same error | 16:09 |
| sdake_ | sbezverk pip install python-extras | 16:09 |
| sdake_ | it may be called some other package | 16:10 |
| sdake_ | sbezverk pip install wheel as well | 16:11 |
| sdake_ | run pip show into a paste | 16:11 |
| sbezverk | sdake_ http://paste.openstack.org/show/491753/ | 16:13 |
| sbezverk | all these are already isntalled | 16:13 |
| sbezverk | extras and wheel and virtenv is already there | 16:14 |
| sdake_ | sudo pip install -U testtools | 16:14 |
| sdake_ | -U = upgrade | 16:14 |
| *** mgoddard_ has quit IRC | 16:14 | |
| *** mgoddard has joined #kolla | 16:14 | |
| sdake_ | pip install -U extras | 16:15 |
| SamYaple | uggghhh | 16:15 |
| SamYaple | they turned on the rax gates and its busting ubuntu | 16:15 |
| sdake_ | SamYaple yess i filed a bug for that | 16:15 |
| SamYaple | sdake_: when? | 16:16 |
| sdake_ | last night at midnight | 16:16 |
| sbezverk | sdake_ http://paste.openstack.org/show/491754/ | 16:16 |
| sdake_ | sbezverk try gen release notess again | 16:16 |
| SamYaple | ah ok. im working it in a different bug | 16:17 |
| SamYaple | ill track it down | 16:17 |
| *** mbound has quit IRC | 16:17 | |
| *** mgoddard_ has joined #kolla | 16:18 | |
| *** Pierre_Carette has joined #kolla | 16:21 | |
| *** mgoddard has quit IRC | 16:21 | |
| *** Marga_ has joined #kolla | 16:21 | |
| *** Marga_ has quit IRC | 16:22 | |
| *** Marga_ has joined #kolla | 16:23 | |
| sbezverk | sdake_ http://paste.openstack.org/show/491756/ | 16:23 |
| sdake_ | sbezverk not sure- ubuntu i a pile of shit ;) | 16:24 |
| *** tfukushima has quit IRC | 16:24 | |
| sdake_ | sbezverk hit up the release channel | 16:25 |
| sdake_ | with a pip show | 16:25 |
| sdake_ | paste | 16:25 |
| sdake_ | and a paste of that last log | 16:25 |
| sdake_ | and ask for advice | 16:25 |
| sdake_ | i am not a python wizard | 16:25 |
| sdake_ | the folks in release = dhellman wrote the book on python | 16:25 |
| jpeeler | those pip modules are looking for headers that aren't there. python-devel and libyaml-devel are the fedora package names | 16:27 |
| jpeeler | probably just -dev on ubuntu | 16:27 |
| sdake_ | jpeeler ht is an optimization step | 16:28 |
| sdake_ | but ya it wouldn't hurt to install the devel libs | 16:28 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix /etc/hosts in gate https://review.openstack.org/297247 | 16:30 |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla: Add kibana authentication using HAproxy https://review.openstack.org/296650 | 16:32 |
| SamYaple | sbezverk: you are missing python-dev pacakge. that problem is common and not part of this-or-that distro is better. | 16:32 |
| SamYaple | sbezverk: youre probably missing a few more libs too, all of these are documented in quickstart guide | 16:32 |
| jpeeler | ok yeah... it works fine here, not sure how it's broken | 16:33 |
| *** sdake_ is now known as sdake | 16:34 | |
| vhosakot | sbezverk: if Ubuntu, can you do "sudo apt-get -y install python-pip python-dev"... wfm on a brand new vm | 16:37 |
| sdake | vhosakot try running tox -e releasenotes | 16:39 |
| sdake | with master | 16:39 |
| vhosakot | ah, let me do it | 16:39 |
| *** akwasnie1 has joined #kolla | 16:42 | |
| *** akwasnie1 has left #kolla | 16:42 | |
| *** jasonsb has quit IRC | 16:42 | |
| vhosakot | sdake: works fine for me.. on Ubuntu 14.04.... on latest master pulled 2 mins ago... http://paste.openstack.org/show/491760/ | 16:43 |
| vhosakot | tox said congratulations :) at the end | 16:43 |
| *** unicell has quit IRC | 16:44 | |
| manjeets | I have done doing all the config but i have question ssh key is not injected to all my nodes can add hostname and password in ~/.ssh config will it work | 16:45 |
| manjeets | ? | 16:45 |
| *** mbound has joined #kolla | 16:46 | |
| *** tfukushima has joined #kolla | 16:46 | |
| *** mbound has quit IRC | 16:46 | |
| sdake | you hve to add your pubic key to uahtoried_keys file in .ssh directory | 16:46 |
| sdake | files needs to be mode 600 | 16:46 |
| *** jtriley has quit IRC | 16:46 | |
| sdake | .ssh directory needs to be mode 700 | 16:47 |
| manjeets | will password not work ? | 16:47 |
| *** gokrokve has joined #kolla | 16:47 | |
| *** achanda has joined #kolla | 16:47 | |
| *** mgoddard_ has quit IRC | 16:47 | |
| vhosakot | did you use --key-name when doing "nova boot" ? have you setup a keypair using "nova keypair-add" ? | 16:47 |
| *** mgoddard has joined #kolla | 16:47 | |
| manjeets | I mean giving host ip and password | 16:47 |
| vhosakot | I think cirros is the only distro that has a default username/password which is cirros/cubswin:).... other distros don't have default user and password... so, either inject ssh keys at boot time, or create a new user in cloud-init using nova's --user-data argument | 16:49 |
| Pierre_Carette | manjeets: are you talking about ssh key for multinode deployment? this is required for ansible to work I believe | 16:50 |
| vhosakot | sdake: so, "tox -e releasenotes" worked for me... http://paste.openstack.org/show/491760/ | 16:50 |
| *** tfukushima has quit IRC | 16:50 | |
| *** dwalsh has quit IRC | 16:51 | |
| *** dwalsh has joined #kolla | 16:53 | |
| manjeets | sdake thanks gonna inject public key using copy of ansible | 16:53 |
| *** vhosakot_ has joined #kolla | 16:54 | |
| *** mbound has joined #kolla | 16:55 | |
| *** macsz has quit IRC | 16:56 | |
| *** vhosakot has quit IRC | 16:57 | |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla: Add kibana authentication using HAproxy https://review.openstack.org/296650 | 16:57 |
| *** vhosakot_ has quit IRC | 16:57 | |
| *** vhosakot has joined #kolla | 16:58 | |
| sdake | akwasnie nice ^^ | 16:59 |
| vhosakot | is https://review.openstack.org down ? | 17:00 |
| vhosakot | working now | 17:00 |
| *** kproskurin has quit IRC | 17:00 | |
| *** openstackgerrit has quit IRC | 17:01 | |
| *** openstackgerrit has joined #kolla | 17:01 | |
| *** mbound has quit IRC | 17:02 | |
| sdake | sbezverk lets debug over here plz | 17:05 |
| sbezverk | sdake sure | 17:07 |
| sdake | akwasnie i disagree with eric about the haproxy_ prefix | 17:08 |
| sdake | akwasnie can you fix patch and i'll +2 it thanks :) | 17:08 |
| *** dwalsh has quit IRC | 17:08 | |
| sdake | akwasnie see this is much simpler then introducing nginx | 17:08 |
| sdake | SamYaple good suggestion on the use of haproxy | 17:08 |
| sbezverk | sdake what else do you want me to try? | 17:11 |
| sdake | sbezverk add extras to test-requirements.txt | 17:11 |
| sbezverk | sdake with a specific version or just the name? | 17:12 |
| sdake | just the name | 17:12 |
| sbezverk | sdake done | 17:12 |
| sdake | just append test-requirements.txt with extas | 17:12 |
| sdake | rm -rf .tox | 17:12 |
| sdake | and tox -e releasenotes | 17:12 |
| *** dwalsh has joined #kolla | 17:13 | |
| *** inc0 has joined #kolla | 17:13 | |
| sbezverk | sdake itdoes not complain now about extras but for ImportError: No module named pyrsistent | 17:15 |
| sdake | pip show tesettools | 17:15 |
| sdake | i have version 2.0.0 | 17:15 |
| sbezverk | sdake I had 0.9 just updated it to 2.0.0 | 17:17 |
| sdake | rm -rf .tox | 17:17 |
| sdake | and run tox -e releasenotes | 17:17 |
| sdake | results? | 17:18 |
| sbezverk | sdake same thing :-( | 17:19 |
| sdake | paste log of pyristent problem | 17:19 |
| sbezverk | sdake can we try to add pyrsistent | 17:19 |
| sdake | i need to see what imports it | 17:19 |
| sdake | these aren't solutions we can merge in to the repository | 17:20 |
| sdake | i want to get to the root caue of your problem | 17:20 |
| sdake | since it could omply our packaging is broken | 17:20 |
| sbezverk | sdake http://paste.fedoraproject.org/344736/45884004/ | 17:20 |
| sbezverk | sdake looks like setup.py is calling it if I am not mistaken | 17:22 |
| sdake | testtools is calling it | 17:22 |
| sdake | see line 1003 | 17:22 |
| sbezverk | sdake ok | 17:23 |
| sdake | oris testtools in your test-requirements.txt? if so paste it | 17:24 |
| sbezverk | testtools>=1.4.0 # MIT | 17:24 |
| sdake | change the >= to == | 17:24 |
| sbezverk | sdake do you want me to keep 1.4.0? | 17:25 |
| sdake | yes | 17:25 |
| sdake | rm -rf .tox and tox -e again | 17:25 |
| sbezverk | sdake failed but with new module name http://paste.fedoraproject.org/344742/40436145/ | 17:27 |
| *** unicell has joined #kolla | 17:27 | |
| sdake | ok well that confirms testtools is being installed in your virtualenv | 17:28 |
| sdake | revert the == change | 17:28 |
| sbezverk | sdake ok | 17:29 |
| sdake | add the pyresistent to test-requiremetns | 17:29 |
| sbezverk | Downloading/unpacking pyresistent (from -r /nobackup/sbezverk/iscsi/kolla/test-requirements.txt (line 7)) | 17:31 |
| sbezverk | Could not find any downloads that satisfy the requirement pyresistent (from -r /nobackup/sbezverk/iscsi/kolla/test-requirements.txt (line 7)) | 17:31 |
| sdake | spell it properly :) | 17:31 |
| sdake | look at previous logs for what it wants | 17:31 |
| sdake | you may have to look at pip.org for the package name | 17:31 |
| sbezverk | sdake ok | 17:31 |
| sbezverk | sdake it looks like it is installed http://paste.fedoraproject.org/344747/84079714/ | 17:33 |
| sdake | sbezverk there is this thign in python called a virtualenv | 17:33 |
| sdake | tox runs in a virutalenv | 17:33 |
| sdake | virutalenv's cretaea completely new python environment to run within | 17:34 |
| sbezverk | sdake I see | 17:34 |
| sdake | pip show virtualenv | 17:35 |
| sbezverk | ImportError: No module named pytz | 17:35 |
| sdake | i'vegot 15.0.0 | 17:35 |
| sbezverk | I have 15.0.1 | 17:35 |
| sdake | let me try tht version | 17:35 |
| sbezverk | sdake should I add pytz? | 17:35 |
| akwasnie | sdake: hi, I am changing it now to kibana_user, kibana_password | 17:36 |
| inc0 | akwasnie, so, auth is based on haproxy? | 17:36 |
| sdake | virtualenv 15.0.1 works for me | 17:37 |
| akwasnie | inc0: yes | 17:37 |
| inc0 | ok, cool | 17:37 |
| akwasnie | as sdake said, much simpler | 17:37 |
| akwasnie | than nginx | 17:37 |
| sdake | its more consistent | 17:37 |
| akwasnie | yes | 17:37 |
| sdake | then haproxy_* | 17:37 |
| sdake | sbezverk hae you rebased on master recently | 17:39 |
| inc0 | akwasnie, but don't add external bind | 17:39 |
| sdake | pytz is in my virtualenv | 17:39 |
| sbezverk | sdake now it succeeded here is the list I had to add: extras pyrsistent pytz docutils | 17:39 |
| sdake | cd .tox/releaenotes/lib/pythoon2.7 | 17:40 |
| sdake | ls | 17:40 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Parse and render yaml files in the proper order https://review.openstack.org/295831 | 17:40 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Fix OpenStack services registration https://review.openstack.org/291251 | 17:40 |
| sdake | you should see a bunch of stuff | 17:40 |
| sbezverk | sdake yes I see | 17:41 |
| sbezverk | paste the content? | 17:42 |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla: Add kibana authentication using HAproxy https://review.openstack.org/296650 | 17:42 |
| manjeets | i tried installing docker using this command through ansible curl -sSL https://get.docker.io | bash | 17:42 |
| sdake | no | 17:42 |
| manjeets | it fails | 17:42 |
| sdake | this is for your education ;) | 17:42 |
| sdake | ls -l setuptools* | 17:43 |
| sdake | ls -ld setuptools | paste | 17:43 |
| sdake | ls -ld setuptools* | paste | 17:43 |
| manjeets | I am trying install dependencies on all the nodes | 17:43 |
| sbezverk | sdake no setup* there | 17:43 |
| SamYaple | back from lunch | 17:44 |
| sdake | manjeets install just docker-py and docker-engine | 17:44 |
| sdake | manjeets those are only requirements for target nodes | 17:44 |
| sdake | sbezverk [sdake@minime-03 site-packages]$ ls -ld setuptools* | 17:44 |
| sdake | drwxrwxr-x. 4 sdake sdake 4096 Mar 24 13:36 setuptools | 17:44 |
| sdake | drwxrwxr-x. 2 sdake sdake 4096 Mar 24 13:36 setuptools-20.3.1.dist-info | 17:45 |
| sdake | this is in my to directory | 17:45 |
| manjeets | cool thanks | 17:45 |
| sbezverk | sbezverk@sbezverk-lnx01:/nobackup/sbezverk/iscsi/kolla/.tox/releasenotes/lib/python2.7$ ls -ld setuptools* | 17:45 |
| sbezverk | ls: cannot access setuptools*: No such file or directory | 17:45 |
| sdake | cd site-packages | 17:45 |
| sdake | and try same thing | 17:45 |
| sbezverk | ls -ld setuptools* | 17:46 |
| sbezverk | drwxrwxr-x 4 sbezverk sbezverk 4096 Mar 24 13:38 setuptools | 17:46 |
| sbezverk | drwxrwxr-x 2 sbezverk sbezverk 4096 Mar 24 13:38 setuptools-20.3.1.dist-info | 17:46 |
| sdake | run pip list | fpaste | 17:46 |
| sdake | setuptools is not installing its dependencies properly | 17:46 |
| manjeets | apt-get install docker-engine and pip install docker-enginer does not works | 17:47 |
| *** tfukushima has joined #kolla | 17:47 | |
| sbezverk | sdake sudo pip list or just pip list, they show different results | 17:47 |
| sdake | manjeets you need to use the curl method to install docker-engine | 17:47 |
| sdake | what ind of idferent results? | 17:47 |
| sdake | do you have two pip binaries installed? | 17:47 |
| sdake | one in /usr/sbin and one in /usr/bin? | 17:48 |
| sbezverk | sdake both outputs are there http://paste.fedoraproject.org/344761/14588416/ | 17:48 |
| manjeets | curl is not working in my ansible playbook | 17:49 |
| sbezverk | sdake "/usr/local/bin/pip" | 17:49 |
| sdake | sbezverk line 24 is a problem | 17:50 |
| sdake | sbezverk do you have other versions of pip somewhere? | 17:50 |
| sbezverk | sdake yes, there are two and they are not the same I did diff | 17:50 |
| sdake | where is the other pip | 17:51 |
| *** tfukushima has quit IRC | 17:51 | |
| sdake | run /usr/local/bin/pip --version | 17:51 |
| sbezverk | http://paste.fedoraproject.org/344766/45884189/ | 17:51 |
| sdake | ok now we are getting somewhere | 17:52 |
| sdake | use apt to remove python-pip | 17:52 |
| sbezverk | sdake ok now there is just one | 17:54 |
| sbezverk | sdake more recent | 17:54 |
| sdake | run pip list | 17:54 |
| sdake | and paste it | 17:54 |
| sbezverk | sdake http://paste.fedoraproject.org/344769/42090145/ | 17:54 |
| sdake | ok now remove that stuf you added to test-requirements, remove .tox, and tox -e releaenotes | 17:55 |
| sbezverk | sdake ok | 17:55 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix /etc/hosts in gate https://review.openstack.org/297247 | 17:56 |
| vhosakot | sdake: you asked me last night if I can successfully boot a VM with latest master.. yes, I can boot a VM an attach it to private network with no issues in latest master... | 17:56 |
| sdake | vhosakot ok | 17:56 |
| vhosakot | sdake: http://paste.openstack.org/show/491780/ and horizon screenshot is at http://picpaste.com/pics/vm-sHpjWkse.1458842072.png | 17:56 |
| sdake | vhosakot i reubilt my images before i headed to bed last night | 17:56 |
| sdake | so i'll try it today | 17:57 |
| vhosakot | sdake: ah, cool... | 17:57 |
| sdake | maybe master was temporarily broken | 17:57 |
| sbezverk | sdake it is working now | 17:57 |
| sdake | your welcome | 17:58 |
| sdake | ok now onto vhosakot 's issue | 17:58 |
| sbezverk | sdake thank you | 17:58 |
| vhosakot | sdake: so, only two new things I observed it, use "default" for domain in the horizon login screen, and use the UUID4 random password generated for keystone_admin_password in /etc/kolla/passwords.yml as the password in the horizon login scrren | 17:58 |
| vhosakot | sdake: I got the horizon password using "grep keystone_admin_password /etc/kolla/passwords.yml" | 17:58 |
| SamYaple | sdake: did you get my messages about tagging kolla this friday? | 17:59 |
| sdake | SamYaple no | 18:00 |
| vhosakot | the same password also shows up for the field OS_PASSWORD (as OS_PASSWORD=suxwkSOL03M7Q3Qx0XIiwfrWMdZtAbuUfyE4JrSP) when we source /etc/kolla/admin-openrc.sh | 18:00 |
| SamYaple | 15:22:52 < SamYaple> sdake: we should not tag rc2 on the 25th. i thought of a reason | 18:00 |
| SamYaple | 15:23:06 < SamYaple> sdake: that reason is all other projects tag on the 25th. we should consume the rc2 packages | 18:00 |
| SamYaple | 15:23:17 < SamYaple> which means we need to tag 26th-ish | 18:00 |
| SamYaple | 15:23:19 < SamYaple> or later | 18:01 |
| SamYaple | 15:23:39 < SamYaple> we want to be consuming the rc2 fixes in our rc1 for stability (and to make sure nothing new breask) | 18:01 |
| sdake | other projects release rc2 at their discrtetion i think | 18:01 |
| sdake | not on the 25th in general | 18:01 |
| SamYaple | the openstack projects release on the 25th | 18:02 |
| SamYaple | we still have some b3 stuff instead of rc1 for OUR rc1, thats not the ones im talkign about | 18:02 |
| *** salv-orl_ has joined #kolla | 18:02 | |
| SamYaple | i mean like nova, neutron etc | 18:02 |
| SamYaple | we need thier rc2 in our rc2 | 18:02 |
| *** salv-orlando has quit IRC | 18:05 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: iscsi container with lvm2 support https://review.openstack.org/291285 | 18:06 |
| *** inc0 has quit IRC | 18:07 | |
| *** sdake_ has joined #kolla | 18:07 | |
| vhosakot | SamYaple: I'll update docs about how the operator can get horizon password after doing kolla-genpwd | 18:09 |
| *** sdake has quit IRC | 18:10 | |
| *** shardy has quit IRC | 18:10 | |
| *** inc0 has joined #kolla | 18:12 | |
| *** athomas has quit IRC | 18:13 | |
| sbezverk | sdake SamYaple do I need release notes for iscsi patch or not? | 18:18 |
| sbezverk | my undersatnding was that it was -2 because of that.. | 18:19 |
| SamYaple | vhosakot: just fyi, the way this should go is the operator _should_ be creating a new domain and user and project forthemseleves | 18:20 |
| SamYaple | vhosakot: then using that with a password they set | 18:20 |
| SamYaple | sbezverk: only sdake can remove his -2 | 18:21 |
| *** vhosakot has quit IRC | 18:29 | |
| manjeets | just finished kolla deploy ended up in issues | 18:36 |
| manjeets | http://paste.openstack.org/show/491784/ | 18:36 |
| *** Serlex has left #kolla | 18:36 | |
| manjeets | sdake sbezverk | 18:36 |
| SamYaple | manjeets: looks like you might have somethign wrong in your configs | 18:37 |
| SamYaple | manjeets: mind posting those? | 18:37 |
| manjeets | which one specifically ? | 18:38 |
| manjeets | its always failing at creating volume task | 18:42 |
| inc0 | manjeets, which version of docker you're running? | 18:42 |
| inc0 | and which version of ansible | 18:43 |
| inc0 | ? | 18:43 |
| manjeets | 1.10.3 | 18:44 |
| manjeets | i guess I am running wrong version of ansible its 2.0.1.0 | 18:44 |
| manjeets | inc0: | 18:44 |
| inc0 | you need 1.9.4 | 18:44 |
| inc0 | ansible | 18:44 |
| inc0 | and 1.10 for docker | 18:44 |
| inc0 | 1.10+ so docker is good | 18:45 |
| inc0 | but ansible has to be 1.9+ | 18:45 |
| inc0 | not 2.0 | 18:45 |
| *** Marga_ has quit IRC | 18:46 | |
| manjeets | inc0: just corrected ansible and re ran | 18:47 |
| manjeets | thanks | 18:47 |
| *** gokrokve has quit IRC | 18:48 | |
| *** inc0 has quit IRC | 18:51 | |
| manjeets | http://paste.openstack.org/show/491787/ now I am getting this issue | 18:52 |
| *** vhosakot has joined #kolla | 18:52 | |
| vhosakot | SamYaple: ok, thanks... agreed... so, should I update any docs or you think it is not needed... just that the usual passowrd 'password' does not work for horizon | 18:53 |
| *** achanda has quit IRC | 18:53 | |
| SamYaple | vhosakot: maybe update the docs with "create new user/domain/project" type thing | 18:54 |
| SamYaple | vhosakot: or point to the official docs | 18:54 |
| vhosakot | SamYaple: ah ok... point yo official keystone docs ? | 18:55 |
| vhosakot | to* | 18:55 |
| SamYaple | docs.openstack.org should talk about it | 18:55 |
| SamYaple | but i dont recall | 18:55 |
| vhosakot | cool | 18:55 |
| *** Kennan has quit IRC | 19:02 | |
| *** Kennan has joined #kolla | 19:03 | |
| manjeets | I am gettig api error while starting heka container task | 19:06 |
| manjeets | how do i make sure my docker registry i reachable by nodes ? | 19:06 |
| vhosakot | curl <registry IP>:4000 | 19:07 |
| *** dwalsh has quit IRC | 19:08 | |
| vhosakot | or, "curl -v <registry IP>:4000" and see if you get back a "HTTP/1.1 200 OK" status code | 19:08 |
| *** gfidente has quit IRC | 19:09 | |
| manjeets | yes I got 200 ok from one of my node | 19:10 |
| manjeets | vhosakot | 19:10 |
| manjeets | http://paste.openstack.org/show/491792/ | 19:11 |
| manjeets | this is waht i am getting | 19:11 |
| *** Marga_ has joined #kolla | 19:12 | |
| vhosakot | manjeets: on UBuntu or centos ? | 19:21 |
| *** rhallisey has quit IRC | 19:22 | |
| *** dwalsh has joined #kolla | 19:23 | |
| *** achanda has joined #kolla | 19:24 | |
| *** rhallisey has joined #kolla | 19:25 | |
| vhosakot | is your /run shared | 19:26 |
| vhosakot | mount --make-shared /run | 19:26 |
| vhosakot | on ubuntu | 19:26 |
| *** mgoddard has quit IRC | 19:29 | |
| *** inc0 has joined #kolla | 19:36 | |
| *** inc0 has quit IRC | 19:36 | |
| *** inc0 has joined #kolla | 19:36 | |
| daneyon | All: I pulled from master last night, successfully built images, kolla-deploy, kolla post-deploy, but I can't nova boot. Please see this gist for more details: https://gist.github.com/danehans/2b752f0c2885353fd791 | 19:37 |
| daneyon | sdake_ SamYaple when you have a moment, I'd appreciate your help with ^ | 19:38 |
| inc0 | daneyon, and what was the error with boot? | 19:38 |
| inc0 | ahh | 19:39 |
| inc0 | see it now | 19:39 |
| inc0 | hmm | 19:39 |
| vhosakot | daneyon: I am able to successfully do "nova boot" with last night's master with no issue... http://paste.openstack.org/show/491780/ and horizon screenshot is at http://picpaste.com/pics/vm-sHpjWkse.1458842072.png | 19:39 |
| inc0 | glance image-list daneyon | 19:39 |
| daneyon | inc0 please see the gist for all the details | 19:40 |
| inc0 | so nova doesn't allow you to do insecure connections by default afair | 19:41 |
| daneyon | vhosakot can i see what your rc file looks like? | 19:42 |
| inc0 | you may try nova --insecure | 19:42 |
| vhosakot | daneyon: sure, 1 sec | 19:42 |
| inc0 | or kolla-ansible certificates | 19:42 |
| daneyon | inc0 let me try --insecure | 19:42 |
| vhosakot | $ cat /etc/kolla/admin-openrc.sh | 19:42 |
| vhosakot | export OS_PROJECT_DOMAIN_ID=default | 19:42 |
| vhosakot | export OS_USER_DOMAIN_ID=default | 19:42 |
| vhosakot | export OS_PROJECT_NAME=admin | 19:42 |
| vhosakot | export OS_TENANT_NAME=admin | 19:42 |
| vhosakot | export OS_USERNAME=admin | 19:42 |
| vhosakot | export OS_PASSWORD=suxwkSOL03M7Q3Qx0XIiwfrWMdZtAbuUfyE4JrSP | 19:42 |
| vhosakot | export OS_AUTH_URL=http://192.168.122.202:35357/v3 | 19:42 |
| vhosakot | export OS_IDENTITY_API_VERSION=3 | 19:42 |
| vhosakot | daneyon: did you run kolla-genpwd... the default OS_password (export OS_PASSWORD=password) does not work | 19:43 |
| daneyon | inc0 i added --insecure and got an error. I updated the gist with the details, pls refresh. | 19:43 |
| vhosakot | daneyon: search for kolla-genpwd in http://docs.openstack.org/developer/kolla/quickstart.html#deploying-kolla | 19:43 |
| daneyon | vhosakot that might be it... 1 moment | 19:44 |
| inc0 | yeah, I can't remember this stuff, there was https insecure way to boot an instance afair | 19:44 |
| vhosakot | do you see empty passwords in /etc/kolla/passwords.yml | 19:44 |
| daneyon | vhosakot I did not, let me do a redeploy. | 19:45 |
| daneyon | vhosakot I am going to use tools/clean-containers and clean-host, then run the pswd-gen and deploy | 19:45 |
| *** Marga_ has quit IRC | 19:46 | |
| vhosakot | cool, run tools/cleanup-images as well | 19:46 |
| daneyon | vhosakot passwords in /etc/kolla/passwords.yml are NOT empty. | 19:46 |
| *** Marga_ has joined #kolla | 19:46 | |
| vhosakot | ah... is keystone_admin_password set to password in /etc/kolla/passwords.yml ? | 19:47 |
| daneyon | vhosakot yes | 19:48 |
| vhosakot | daneyon: I think --insecure must come before "boot"... "nova --insecure boot blah..." | 19:49 |
| vhosakot | ah.. can you try insecure boot by putting --insecure before nova and boot ? | 19:49 |
| daneyon | vhosakot sure, 1 moment | 19:54 |
| manjeets | vhosakot : ubuntu | 19:55 |
| *** achanda has quit IRC | 19:55 | |
| vhosakot | manjeets: can you do "mount --make-shared /run" and redeploy the heka container | 19:55 |
| manjeets | i dod that on all nodes and hosts as well before but re did it and re randeploy | 19:57 |
| daneyon | vhosakot same error: Unable to establish connection to https://172.18.151.92:8774/v2/f72b02396e924434bdc9b59fa4df12a4/servers | 19:59 |
| vhosakot | hmm.. | 19:59 |
| daneyon | vhosakot could it have anything to do with the contents of /etc/kolla/certificates/ | 19:59 |
| vhosakot | i dont know.. ask dave-mccowan | 19:59 |
| daneyon | does Kolla manage/update the /etc/kolla/certificates/ for redeploys? Do the redeploy tools manage this dir? | 20:00 |
| daneyon | vhosakot I think he's away. Let me try rebuilding from scratch and not use the https stuff. | 20:01 |
| vhosakot | dont know.... ask dave-mccowan.... I comment out kolla_enable_tls_external in my /etc/kolla/globals.yml.... I dont use TLS in my test | 20:01 |
| vhosakot | daneyon: you have kolla_enable_tls_external turned on ? | 20:02 |
| sdake_ | was afk | 20:03 |
| vhosakot | daneyon: I see https used when kolla_enable_tls_external is true in https://github.com/openstack/kolla/blob/master/ansible/group_vars/all.yml#L139 | 20:03 |
| sdake_ | daneyon if you have tls enabled | 20:04 |
| sdake_ | if you need to run kolla-ansible certifictes | 20:05 |
| sdake_ | then you neeed to use an openrc like this one: | 20:05 |
| *** sdake has joined #kolla | 20:07 | |
| *** sdake_ has quit IRC | 20:07 | |
| daneyon | vhosakot I do: /etc/kolla/globals.yml:kolla_enable_tls_external: "yes" | 20:09 |
| *** sdake_ has joined #kolla | 20:10 | |
| vhosakot | ah, then, see the steps above sdake mentioned to generate the right certs when auth'ing en-points | 20:10 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: iscsi container with lvm2 support https://review.openstack.org/291285 | 20:11 |
| *** tfukushima has joined #kolla | 20:11 | |
| daneyon | sdake_ when should kolla-ansible certificates be run in the deployment process? | 20:11 |
| *** sdake has quit IRC | 20:12 | |
| daneyon | or vhosakot if you know ^ | 20:12 |
| vhosakot | daneyon: https://github.com/openstack/kolla/blob/master/tools/kolla-ansible#L46 | 20:13 |
| vhosakot | daneyon: to geenrate certs for TLS auth | 20:13 |
| vhosakot | certificates.yml will be used... https://github.com/openstack/kolla/blob/master/tools/kolla-ansible#L148-L150 | 20:14 |
| daneyon | vhosakot understood. I'm trying to understand when you run the cert command. Is it before or after kolla deploy, build, etc.. | 20:14 |
| vhosakot | I want to say before... dont know.. the certs need to be there before the end-points are created... so, I'd say before "kolla-ansible deploy" | 20:15 |
| daneyon | vhosakot i think you're right. Let me redeploy | 20:15 |
| *** tfukushima has quit IRC | 20:15 | |
| manjeets | vhisakot it was me did not restart docker for all nodes after insecure thing | 20:16 |
| sdake_ | daneyon firt thing | 20:16 |
| manjeets | but now getting a new issue | 20:16 |
| manjeets | http://paste.openstack.org/show/491796/ | 20:16 |
| sdake_ | daneyon vhosakot is correct | 20:17 |
| inc0 | manjeets, docker ps | 20:17 |
| inc0 | plz | 20:17 |
| ccesario | maybe SamYaple can give more informations about certificates, I think he uses let's encrypt cert in your environment | 20:17 |
| inc0 | and docker logs mariadb | 20:17 |
| vhosakot | sdake_: may be kolla-ansible finds out if certs are not created when TLS is enabled and errors out... would be agood enhancement | 20:17 |
| manjeets | inc0 it shows the registry 2.3 | 20:17 |
| inc0 | on the node which failed mariadb | 20:18 |
| manjeets | http://paste.openstack.org/show/491797/ | 20:18 |
| SamYaple | hey guys | 20:21 |
| sdake_ | daneyon kolla-ansible create self-signed certs | 20:21 |
| vhosakot | manjeets: id' restart mariadb container and redeploy | 20:21 |
| SamYaple | whats happening? | 20:21 |
| vhosakot | daneyon has cert Qs | 20:22 |
| manjeets | vhosakot on a particular node ? | 20:22 |
| manjeets | or on all nodes ? | 20:22 |
| SamYaple | ah if its about generating self-signed certs idk. i never used that playbook | 20:22 |
| vhosakot | manjeets: whever it is down | 20:23 |
| SamYaple | i always did it by hand (or have real certs) | 20:23 |
| manjeets | docker restart mariadb ? | 20:24 |
| vhosakot | SamYaple: I think we need good docs about the deployment workflow when kolla_enable_tls_external is enabled... how to genereate certs and stuff | 20:24 |
| vhosakot | manjeets: yes... | 20:24 |
| manjeets | vhosakot do i need to koll whatever containers are already running before i do deploy | 20:25 |
| manjeets | or its fine to do now ? | 20:25 |
| manjeets | kill ** | 20:26 |
| vhosakot | manjeets: not needed.... just restart ones that are bad | 20:27 |
| manjeets | cool just did that | 20:27 |
| *** achanda has joined #kolla | 20:28 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Fix /etc/hosts in gate https://review.openstack.org/297247 | 20:29 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Use debian repos for debian base docker image https://review.openstack.org/288936 | 20:30 |
| manjeets | vhosakot it failed again i restarted container before i did deploy | 20:30 |
| *** jmccarthy1 has joined #kolla | 20:30 | |
| vhosakot | hmmm | 20:30 |
| *** sdake has joined #kolla | 20:30 | |
| *** ccesario has quit IRC | 20:31 | |
| *** achanda has quit IRC | 20:31 | |
| *** haplo37 has quit IRC | 20:31 | |
| dave-mccowan | daneyon vhosakot looks like you figured it out. all good on certificates? | 20:32 |
| sdake | daneyon re magnum, jeffrey4l is already working on making magnum work - pleae work with him on this topic | 20:32 |
| sdake | or sync up with taking over the work | 20:32 |
| *** sdake_ has quit IRC | 20:33 | |
| sdake | sam testing your heat change | 20:33 |
| * manjeets just restarted again and re ran the deploy | 20:33 | |
| sdake | SamYaple please add asalkeld and myself as coauthors since we figured out the root cause thanks | 20:33 |
| *** inc0 has quit IRC | 20:35 | |
| manjeets | cannot access /var/lib/mysql/mariadb.pid: No such file or directory | 20:36 |
| manjeets | but i see mariadb in list when i docker ps | 20:37 |
| daneyon | sdake will do | 20:37 |
| vhosakot | manjeets: is this AIO or multi ? | 20:37 |
| manjeets | multi | 20:38 |
| manjeets | i restarted in all control nodes | 20:38 |
| manjeets | because i have maridb [control] | 20:38 |
| sdake | SamYaple your heat patch is no beuno | 20:38 |
| vhosakot | how many controllers ? 3 ? | 20:39 |
| sdake | SamYaple fwiw, there are numerous reports onteh internet that the docuenation is wrong | 20:39 |
| manjeets | 7 or 8 | 20:39 |
| sdake | please use launch to tesst a stack creation | 20:39 |
| manjeets | let me double check | 20:39 |
| sdake | its a simple script that can be run in the demo directory | 20:39 |
| vhosakot | SamYaple: is a multinode deployment, mariadb runs on each of three controllers.. is this right ? | 20:40 |
| manjeets | I have actually 6 controllers | 20:40 |
| sbezverk | majeet would you consider just to for training purposes to complete at least one all in one?? | 20:41 |
| vhosakot | manjeets: do you see mariadb running on all 6 controllers ? | 20:41 |
| manjeets | vhosakot: yes | 20:41 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: Make kibana container build with debian https://review.openstack.org/297206 | 20:41 |
| manjeets | sbezverk i did it once on aio last week | 20:42 |
| vhosakot | manjeets: because mariadb's children in the [control] group in the multinode inventory... so, I'd restart on the controller on which mariadb is down | 20:42 |
| sbezverk | manjeets ok, great.. I did not know that | 20:42 |
| manjeets | restart you mean reboot the controllers ? | 20:43 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: Make nova containers build with debian https://review.openstack.org/297169 | 20:43 |
| ntpttr | sbezverk vhosakot: gmmaha and I are running into the same issue as manjeets on a fresh deploy after cleaning up containers and hosts. We had a working cloud last week using the same config | 20:43 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: Make swift containers build with debian https://review.openstack.org/297121 | 20:43 |
| manjeets | i restarted maridb on all controllers | 20:43 |
| ntpttr | but it only happened once, we tried again and it worked | 20:43 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: make kolla-toolbox build with debian https://review.openstack.org/297104 | 20:43 |
| vhosakot | same issue ? maridb pid file not found ? | 20:43 |
| manjeets | yes he had issue | 20:44 |
| vhosakot | hmmm | 20:44 |
| sdake | i've got a dentist appointment, back in 2-3 hours | 20:45 |
| gmmaha | vhosakot: ntpttr just did a cleanup and a ddeploy again seems to be workign | 20:45 |
| manjeets | vhosakot i manually did docker ps on all control | 20:45 |
| manjeets | it shows mariadb up | 20:45 |
| vhosakot | gmmaha: ntpttr: multinode with 6 controllers ? and redeploy worked ? | 20:46 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make mesosphere build with debian https://review.openstack.org/297358 | 20:46 |
| manjeets | they have 3 node | 20:46 |
| ntpttr | vhosakot: multinode with 3 controllers | 20:46 |
| ntpttr | manjeets: 10 nodes, 3 controllers | 20:46 |
| manjeets | here it is 64 nodes and its 6 controll | 20:47 |
| sbezverk | manjeets could you paste docker ps -a on all controlers | 20:47 |
| manjeets | http://paste.openstack.org/show/491800/ | 20:48 |
| *** openstackgerrit has quit IRC | 20:48 | |
| *** openstackgerrit has joined #kolla | 20:48 | |
| sbezverk | ntpttr on your controller, do you see keepalived and haproxy containers running? | 20:49 |
| ntpttr | sbezverk: yes | 20:50 |
| sbezverk | manjeets please confirm you run "docker ps -a" and not just "docker ps" | 20:50 |
| *** dwalsh has quit IRC | 20:50 | |
| sbezverk | ntpttr thank you | 20:50 |
| sbezverk | manjeets in your deployment log do you see haproxy and keepalived get deployed?? | 20:51 |
| manjeets | sbezverk i see same out put docker ps -a as well | 20:51 |
| manjeets | http://paste.openstack.org/show/491801/ | 20:51 |
| *** ccesario_ has joined #kolla | 20:52 | |
| *** Marga_ has quit IRC | 20:53 | |
| manjeets | sbezverk deployment log ? | 20:53 |
| sbezverk | manjeets when you run kolla-deploy it generates log | 20:53 |
| *** Marga_ has joined #kolla | 20:53 | |
| sbezverk | can you check if you see entries for keepalived and haproxy | 20:54 |
| vhosakot | the output looks like docker ps... can you send docker ps -a that shows _all_ containers including dead/restarting ones | 20:54 |
| manjeets | http://paste.openstack.org/show/491804/ | 20:56 |
| *** rhallisey has quit IRC | 20:56 | |
| manjeets | from console output while deploy i see coying over happening for ha proxy and keepalived but i don't those getting deployed | 20:57 |
| sbezverk | can you paste whole log so we could analyze it? | 20:57 |
| vhosakot | manjeets: yes, haproxy must be deplyed... | 20:58 |
| vhosakot | manjeets: can you post your entire multinode inventory file | 20:58 |
| sbezverk | vhosakot I suspect that processes tries to check for mariadb but virtual IP is not up | 20:58 |
| vhosakot | ah ok | 20:59 |
| sbezverk | so it fails | 20:59 |
| manjeets | vip is not a active ip i just gave an unused ip from subnet | 20:59 |
| sbezverk | manjeets that is correct but when haproxy is up this ip will become active and pingable | 21:00 |
| manjeets | http://paste.openstack.org/show/491806/ | 21:01 |
| sbezverk | manjeets if this ip is not pingable we should stop troublehooting mariadb and start traoublehooting haproxy | 21:01 |
| *** sdake has quit IRC | 21:01 | |
| manjeets | that ip is pingable | 21:02 |
| sbezverk | manjeets kolla-ansible deploy logs please as well. and can you check your nodes to see where haproxy is running? | 21:04 |
| vhosakot | manjeets: ok, in your inventory file, I see mariadb on all nodes in the [control] group and haproxy on all nodes in the [network] group.. so, the gouping looks correct | 21:04 |
| SamYaple | sdake my heat patch works correclty, it is you who have not done it correctly. recommend redoing it | 21:04 |
| manjeets | sbezverk are logs saved in some file or you just need console out while running deploy ? | 21:06 |
| manjeets | console output ** | 21:07 |
| sbezverk | manjeets console output is good enough | 21:07 |
| manjeets | http://paste.openstack.org/show/491807/ | 21:09 |
| manjeets | i lost some on top | 21:09 |
| manjeets | let me if it is not enogh i can re run | 21:09 |
| *** ccesario_ has quit IRC | 21:10 | |
| sbezverk | manjeets could you try to add controllers ip address to network group as well and re-run deployment | 21:12 |
| sbezverk | majeets under [haproxy:children] remove network and replace it for "control" | 21:14 |
| manjeets | ok | 21:17 |
| manjeets | sbezverk should i do both or just majeets under [haproxy:children] remove network and replace it for "control" | 21:18 |
| sbezverk | majeets just replace network with control and re-run deployment | 21:19 |
| * manjeets just did that and started re deplloying | 21:21 | |
| manjeets | ise ha proxy container getting started now | 21:21 |
| vhosakot | ntpttr: gmmaha: in your deployment, you have haproxy under control group, right ? | 21:23 |
| manjeets | it again failed | 21:23 |
| sbezverk | manjeets same place? | 21:23 |
| vhosakot | can you send the error | 21:23 |
| gmmaha | vhosakot: yes.. | 21:23 |
| sbezverk | and dockker ps -a from controllers | 21:23 |
| manjeets | http://paste.openstack.org/show/491810/ | 21:24 |
| vhosakot | gmmaha: cool... | 21:24 |
| ntpttr | vhosakot: yes | 21:24 |
| sbezverk | manjeets ip addresses in the log correspond to ip addresses of you controllers right? | 21:25 |
| manjeets | yes | 21:25 |
| manjeets | i guess ntpttr is running control as network as well | 21:25 |
| sbezverk | manjeets please check that now haproxy and keepalived are running on these hosts | 21:26 |
| manjeets | it is http://paste.openstack.org/show/491811/ | 21:27 |
| sbezverk | majeets sorry but could you clean up all these hosts and run deployment again, you see you have some containers runing for 1 hour some for just few minutes | 21:28 |
| manjeets | ok | 21:29 |
| vhosakot | I see mariadb up on 7 nodes... | 21:29 |
| manjeets | 24 to 30 | 21:29 |
| manjeets | 7 nodes control | 21:30 |
| vhosakot | manjeets: you have 7 controllers ? 172.23.0.24 - 172.23.0.30 | 21:30 |
| vhosakot | ok | 21:30 |
| * manjeets started over to deploy after cleaning containers from all controll nodes | 21:33 | |
| *** ayoung has quit IRC | 21:34 | |
| manjeets | http://paste.openstack.org/show/491812/ | 21:36 |
| manjeets | should i clean up hosts as well | 21:36 |
| manjeets | vhosakot: it failed again | 21:37 |
| manjeets | this time i did cleanup on all control | 21:37 |
| sbezverk | manjeets can you connect to 172.23.0.24 and run this command: docker exec mariadb ls -l /var/lib/mysql | 21:38 |
| *** vhosakot has quit IRC | 21:39 | |
| *** openstack has joined #kolla | 21:49 | |
| *** openstackgerrit has joined #kolla | 21:49 | |
| vhosakot | sorry my wifi dropped | 21:50 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: WIP: Make marathon container build with debian https://review.openstack.org/297378 | 21:51 |
| *** tfukushima has joined #kolla | 21:53 | |
| manjeets | vhosakot sbezverk just told me to get docker logs for mariadb | 21:56 |
| manjeets | I am still hitting the same issue | 21:56 |
| sbezverk | manjeets I was looking at the log | 21:56 |
| vhosakot | ah ok | 21:57 |
| sbezverk | I do not see why it exists, log looks ok until the very last line | 21:57 |
| gmmaha | sbezverk: is this normal.. i see it in the log. | 21:57 |
| gmmaha | ERROR 1008 (HY000) at line 1: Can't drop database 'test'; database doesn't exist^M | 21:57 |
| gmmaha | ... Failed! Not critical, keep moving...^M | 21:57 |
| *** tfukushima has quit IRC | 21:58 | |
| sbezverk | no this one: mysqld from pid file /var/lib/mysql/mariadb.pid ended | 21:58 |
| manjeets | sbezverk ntpttr have network and control same | 21:58 |
| manjeets | i have that separate hope that should not be an issue | 21:58 |
| vhosakot | manjeets: so, you have network and control separate, and happroxy on control... is this rihgt? | 21:59 |
| manjeets | yes | 21:59 |
| manjeets | i now have ha proxy on control and network nodes are separate | 22:00 |
| *** yuanying has joined #kolla | 22:00 | |
| *** clayton has joined #kolla | 22:00 | |
| sbezverk | manjeets can you please on all controllers: docker stop mariadb, docker rm mariadb, docker rmi {image id of mariadb container} | 22:00 |
| sbezverk | manjeets I am running out of ideas, some folks on the internet reported similar problem with mariadb, but their solution was clean up mariadb binaries and reinstall it.. | 22:02 |
| *** DuncanT has quit IRC | 22:03 | |
| manjeets | thats only thing left at this moment to do for me | 22:03 |
| sbezverk | manjeets please make sure there is no any mariadb images left on any controllers | 22:04 |
| *** DuncanT has joined #kolla | 22:05 | |
| vhosakot | manjeets: do "docker volume rm mariadb" as well and redeploy | 22:05 |
| *** jtriley has joined #kolla | 22:06 | |
| * manjeets started re deploying after removing all the containers from all the nodes | 22:08 | |
| sbezverk | manjeets and images right? docker images was empty? | 22:08 |
| manjeets | not images | 22:08 |
| manjeets | using same | 22:08 |
| manjeets | ah | 22:08 |
| manjeets | cleanup-containers | 22:09 |
| sbezverk | on ALL controllers ALL images should be removed | 22:09 |
| sbezverk | not on your deployment server though | 22:09 |
| manjeets | ok | 22:09 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make mesos containers build with debian https://review.openstack.org/297385 | 22:10 |
| manjeets | ok cleared all containers and images from all control nodes | 22:12 |
| manjeets | sbezverk ? | 22:12 |
| manjeets | anything should i do before deploy | 22:12 |
| sbezverk | manjeets, ok start deployment | 22:12 |
| * manjeets crosses fingers for 6th time | 22:13 | |
| *** dave-mccowan has joined #kolla | 22:14 | |
| manjeets | sbezverk same result | 22:16 |
| sbezverk | manjeets one last thing comes to my mind is to rebuild mariadb image | 22:17 |
| sbezverk | manjeet make sure you use --no-cache option | 22:18 |
| manjeets | kolla-build mariadb --no-cache ? | 22:18 |
| sbezverk | manjeets you need to add ip address of your registry and --push keyword | 22:19 |
| *** jtriley has quit IRC | 22:20 | |
| sbezverk | kolla-build --registry registry_ip_address:registry_ip_port --push --no-cache mariadb | 22:20 |
| sbezverk | when it is done an all controllers make sure docker ps -a and docker images return nothing and re-deploy | 22:21 |
| manjeets | kolla-build mariadb --registry 172.23.x.x:4000 --push | 22:21 |
| sbezverk | where is --no-cache?? | 22:21 |
| manjeets | ok just did that | 22:22 |
| *** ccesario_ has joined #kolla | 22:24 | |
| manjeets | rebuild mariab deleted old one | 22:28 |
| manjeets | new have no container on any node | 22:28 |
| manjeets | have no images on any node | 22:29 |
| sbezverk | manjeets just to be on safe side | 22:29 |
| manjeets | ready too go ? | 22:30 |
| sbezverk | could you stop docker on your controllers and remove /var/lib/docker/* | 22:30 |
| manjeets | ok | 22:30 |
| sbezverk | it is overkill but that would be the last thing | 22:30 |
| sbezverk | manjeets do not forget to start docker when you are done cleaning | 22:31 |
| manjeets | one idea i am getting is to restart docker on all nodes | 22:31 |
| sbezverk | manjeets ok | 22:32 |
| manjeets | should i restart docker after stop on control nodes | 22:32 |
| manjeets | or will deploy will start it ? | 22:33 |
| sbezverk | ntpttr which flavor you were deploying centos or ubuntu? | 22:33 |
| manjeets | ubuntu | 22:33 |
| ntpttr | ubuntu | 22:33 |
| sbezverk | ok | 22:33 |
| sbezverk | manjeets I am not sure deploy will start docker | 22:33 |
| manjeets | will restart before i deploy | 22:34 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Pass extra arguments through kolla-ansible to ansible-playbook https://review.openstack.org/297394 | 22:34 |
| * manjeets ran into same issue again | 22:40 | |
| *** jmccarthy1 has quit IRC | 22:44 | |
| manjeets | how do i manually start mariadb in each control node | 22:44 |
| manjeets | ? | 22:44 |
| sbezverk | manjeets if mariadb container is running then you do not start it manually, if container it is not running you can do docker start mariadb, but it is not supposed to be this way.. | 22:48 |
| manjeets | i was thinking of running maridb manually inside each control and then retry | 22:49 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make swift-base container build with debian https://review.openstack.org/297121 | 22:50 |
| sbezverk | manjeets I would try to do all in one installation on one of your contoller nodes just to prove that your images are ok and it is something else prevents mariadb to start properly | 22:52 |
| manjeets | sbezverk what i rebuild images as well and then try | 22:52 |
| manjeets | ? | 22:52 |
| manjeets | i have 122 images right now | 22:53 |
| sbezverk | manjeets do not see reason to rebuild all 122, since you hit everytime mariadb issue no matter what we try.. | 22:54 |
| *** vishwanathj has quit IRC | 22:55 | |
| *** sdake has joined #kolla | 22:58 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make nova-libvirt container build with debian https://review.openstack.org/297169 | 22:59 |
| *** cfarquhar has quit IRC | 23:00 | |
| *** cfarquhar has joined #kolla | 23:00 | |
| *** cfarquhar has quit IRC | 23:00 | |
| *** cfarquhar has joined #kolla | 23:00 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make mesos-slave container build with debian https://review.openstack.org/297385 | 23:02 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make container dind build with debian https://review.openstack.org/297434 | 23:03 |
| manjeets | sbezverk mariadb just worked for AIO | 23:06 |
| manjeets | withh all the old images | 23:07 |
| Pierre_Carette | I am having a volume permission error on a new deployment stable/mitaka with ubuntu source right at the heka_bootstrap level | 23:07 |
| Pierre_Carette | ERROR:__main__:The source to copy does not exist: /var/lib/kolla/config_files/heka-haproxy.toml | 23:08 |
| *** ccesario_ has quit IRC | 23:08 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make mesos-slave container build with debian https://review.openstack.org/297385 | 23:08 |
| manjeets | AIO succeded sbezverk | 23:08 |
| sbezverk | manjeets great in a sense, so we do not need to worry about images | 23:09 |
| manjeets | it was so smooth and fast i guess only maridb is giving me hard time on multinode | 23:10 |
| manjeets | and ntpttr has control and network same | 23:10 |
| manjeets | but that should not be an issue ? | 23:11 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make mesosphere-base build with debian https://review.openstack.org/297358 | 23:12 |
| sbezverk | manjeets I do not think it is an issue, especially since in case of ntpttr it worked, do you want to give similar config a try? control and network the same hosts? | 23:13 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make container kolla-toolbox build with debian https://review.openstack.org/297104 | 23:14 |
| manjeets | nopes i really network separate to test things out on network | 23:14 |
| manjeets | but i can give it a try if thats causing an issue | 23:14 |
| sbezverk | manjeets it is just to collect extra data point.. | 23:15 |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make kibana container build with debian https://review.openstack.org/297206 | 23:16 |
| *** sdake_ has joined #kolla | 23:17 | |
| *** sdake has quit IRC | 23:18 | |
| openstackgerrit | Benedikt Trefzer proposed openstack/kolla: Make marathon container build with debian https://review.openstack.org/297378 | 23:20 |
| *** tfukushima has joined #kolla | 23:24 | |
| manjeets | sbezverk ran into same issue for same network as control | 23:25 |
| sbezverk | manjeets you will need then help from senior kolla gurus.. | 23:27 |
| manjeets | sbezverk thanks i appreciate | 23:27 |
| sbezverk | manjeets sorry it is really weird issue :-( | 23:28 |
| *** sdake has joined #kolla | 23:28 | |
| *** sdake_ has quit IRC | 23:28 | |
| *** tfukushima has quit IRC | 23:28 | |
| *** tfukushima has joined #kolla | 23:32 | |
| *** tfukushima has quit IRC | 23:36 | |
| *** sbezverk has quit IRC | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
