| openstackgerrit | Steve Lewis (stevelle) proposed openstack/openstack-ansible-os_gnocchi: Enabling bashate and pep8 lint checks https://review.openstack.org/304328 | 00:04 |
|---|---|---|
| openstackgerrit | Steve Lewis (stevelle) proposed openstack/openstack-ansible-os_gnocchi: Enable ansible lint and syntax tests https://review.openstack.org/304343 | 00:04 |
| *** fawadkhaliq has quit IRC | 00:07 | |
| *** fawadkhaliq has joined #openstack-ansible | 00:08 | |
| *** ChrisBenson has joined #openstack-ansible | 00:27 | |
| *** ChrisBenson has quit IRC | 00:28 | |
| *** ChrisBenson has joined #openstack-ansible | 00:28 | |
| *** fawadkhaliq has quit IRC | 00:28 | |
| *** fawadkhaliq has joined #openstack-ansible | 00:29 | |
| *** markvoelker has joined #openstack-ansible | 00:35 | |
| *** ChrisBenson1 has joined #openstack-ansible | 00:35 | |
| *** ChrisBenson has quit IRC | 00:35 | |
| openstackgerrit | Merged openstack/openstack-ansible: Fix idempotency bug in AIO bootstrap https://review.openstack.org/304227 | 00:37 |
| *** keedya has quit IRC | 00:37 | |
| *** fawadkhaliq has quit IRC | 00:40 | |
| *** markvoelker has quit IRC | 00:41 | |
| *** automagically has joined #openstack-ansible | 00:42 | |
| *** daneyon__ has joined #openstack-ansible | 00:42 | |
| *** daneyon has quit IRC | 00:45 | |
| *** ChrisBenson1 has quit IRC | 00:49 | |
| *** ChrisBenson has joined #openstack-ansible | 00:49 | |
| *** busterswt has quit IRC | 00:50 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Enable SSL termination for all services https://review.openstack.org/277199 | 00:51 |
| *** elgertam has joined #openstack-ansible | 00:52 | |
| *** woodard has joined #openstack-ansible | 00:53 | |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_swift: Resolve issues with swift tests https://review.openstack.org/304350 | 00:54 |
| *** woodard has quit IRC | 00:55 | |
| openstackgerrit | Merged openstack/openstack-ansible-repo_build: Remove global-requirements build from the build process https://review.openstack.org/300589 | 00:56 |
| *** woodard has joined #openstack-ansible | 00:56 | |
| openstackgerrit | Andy McCrae proposed openstack/openstack-ansible-os_swift: Resolve issues with swift tests https://review.openstack.org/304350 | 00:56 |
| *** daneyon has joined #openstack-ansible | 01:00 | |
| *** sdake_ has joined #openstack-ansible | 01:02 | |
| *** sdake has quit IRC | 01:02 | |
| *** fishcried has quit IRC | 01:03 | |
| *** daneyon__ has quit IRC | 01:04 | |
| openstackgerrit | Alexandra Settle proposed openstack/openstack-ansible: Minor fix to correct passive to active voice https://review.openstack.org/304351 | 01:07 |
| *** busterswt has joined #openstack-ansible | 01:07 | |
| *** sdake_ is now known as sdake | 01:08 | |
| *** woodard has quit IRC | 01:14 | |
| *** woodard has joined #openstack-ansible | 01:15 | |
| *** thorst has quit IRC | 01:16 | |
| *** thorst has joined #openstack-ansible | 01:17 | |
| *** elgertam has quit IRC | 01:18 | |
| *** elgertam has joined #openstack-ansible | 01:24 | |
| *** thorst has quit IRC | 01:25 | |
| *** phalmos has joined #openstack-ansible | 01:28 | |
| *** weezS has quit IRC | 01:29 | |
| *** phalmos has quit IRC | 01:39 | |
| *** weezS has joined #openstack-ansible | 01:40 | |
| *** automagically has quit IRC | 01:46 | |
| *** asettle has quit IRC | 01:57 | |
| *** asettle has joined #openstack-ansible | 02:05 | |
| *** bapalm has quit IRC | 02:06 | |
| *** bapalm has joined #openstack-ansible | 02:13 | |
| *** Bjoern has joined #openstack-ansible | 02:18 | |
| *** Bjoern has quit IRC | 02:19 | |
| *** fishcried has joined #openstack-ansible | 02:21 | |
| *** fishcried has quit IRC | 02:21 | |
| *** thorst has joined #openstack-ansible | 02:23 | |
| *** gfa is now known as gfa_ | 02:24 | |
| *** gfa_ is now known as gfa | 02:24 | |
| *** thorst has quit IRC | 02:30 | |
| mhayden | cloudnull: i like seeing 'ssl' and 'all' in the same commit | 02:33 |
| mhayden | i'll gander in the morning | 02:33 |
| *** markvoelker has joined #openstack-ansible | 02:37 | |
| *** markvoelker has quit IRC | 02:42 | |
| *** asettle has quit IRC | 03:01 | |
| *** fishcried has joined #openstack-ansible | 03:06 | |
| *** furlongm has quit IRC | 03:07 | |
| *** fishcried has quit IRC | 03:07 | |
| *** sdake_ has joined #openstack-ansible | 03:12 | |
| *** elgertam has quit IRC | 03:13 | |
| *** sdake has quit IRC | 03:15 | |
| *** sdake has joined #openstack-ansible | 03:18 | |
| *** sdake_ has quit IRC | 03:20 | |
| *** thorst has joined #openstack-ansible | 03:28 | |
| *** mongo2 has quit IRC | 03:29 | |
| *** jayc has joined #openstack-ansible | 03:30 | |
| *** mongo2 has joined #openstack-ansible | 03:31 | |
| *** asettle has joined #openstack-ansible | 03:33 | |
| *** thorst has quit IRC | 03:34 | |
| *** mongo2 has quit IRC | 03:36 | |
| *** asettle has quit IRC | 03:38 | |
| *** mongo2 has joined #openstack-ansible | 03:38 | |
| *** javeriak has joined #openstack-ansible | 03:45 | |
| *** furlongm has joined #openstack-ansible | 03:45 | |
| *** asettle has joined #openstack-ansible | 03:58 | |
| *** woodard has quit IRC | 04:14 | |
| *** woodard has joined #openstack-ansible | 04:15 | |
| *** woodard has quit IRC | 04:19 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-plugins: Update the config_template plugin https://review.openstack.org/304385 | 04:23 |
| prometheanfire | cloudnull: nn | 04:24 |
| *** busterswt has quit IRC | 04:31 | |
| *** thorst has joined #openstack-ansible | 04:33 | |
| *** markvoelker has joined #openstack-ansible | 04:38 | |
| *** thorst has quit IRC | 04:40 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-plugins: Update the config_template plugin https://review.openstack.org/304385 | 04:41 |
| *** markvoelker has quit IRC | 04:42 | |
| openstackgerrit | Merged openstack/openstack-ansible-ironic: Add tests for the ironic CLI https://review.openstack.org/303104 | 04:48 |
| *** furlongm has left #openstack-ansible | 04:51 | |
| *** saneax_AFK is now known as saneax | 04:59 | |
| *** LiftedKilt has quit IRC | 05:08 | |
| *** admin0 has joined #openstack-ansible | 05:10 | |
| *** LiftedKilt has joined #openstack-ansible | 05:13 | |
| *** javeriak has quit IRC | 05:16 | |
| *** admin0 has quit IRC | 05:21 | |
| *** asettle has quit IRC | 05:27 | |
| *** asettle has joined #openstack-ansible | 05:30 | |
| *** thorst has joined #openstack-ansible | 05:38 | |
| *** markvoelker has joined #openstack-ansible | 05:38 | |
| *** fishcried has joined #openstack-ansible | 05:40 | |
| *** markvoelker has quit IRC | 05:43 | |
| *** thorst has quit IRC | 05:45 | |
| *** fawadkhaliq has joined #openstack-ansible | 05:49 | |
| *** javeriak has joined #openstack-ansible | 05:51 | |
| *** woodard has joined #openstack-ansible | 06:07 | |
| *** woodard has quit IRC | 06:13 | |
| *** fawadkhaliq has quit IRC | 06:26 | |
| *** fawadkhaliq has joined #openstack-ansible | 06:29 | |
| *** thorst has joined #openstack-ansible | 06:43 | |
| *** weezS has quit IRC | 06:45 | |
| *** czunker has joined #openstack-ansible | 06:50 | |
| *** thorst has quit IRC | 06:51 | |
| *** asettle has quit IRC | 07:00 | |
| *** mikelk has joined #openstack-ansible | 07:13 | |
| *** fishcried has quit IRC | 07:15 | |
| *** clsacramento has quit IRC | 07:19 | |
| *** clsacramento has joined #openstack-ansible | 07:22 | |
| *** sdake_ has joined #openstack-ansible | 07:24 | |
| *** sdake has quit IRC | 07:26 | |
| *** sdake has joined #openstack-ansible | 07:27 | |
| *** admin0 has joined #openstack-ansible | 07:29 | |
| *** asettle has joined #openstack-ansible | 07:30 | |
| *** sdake_ has quit IRC | 07:30 | |
| *** thorst has joined #openstack-ansible | 07:33 | |
| evrardjp | good morning everyone | 07:34 |
| *** asettle has quit IRC | 07:34 | |
| *** markvoelker has joined #openstack-ansible | 07:39 | |
| *** thorst has quit IRC | 07:40 | |
| *** jamielennox is now known as jamielennox|away | 07:42 | |
| *** markvoelker has quit IRC | 07:44 | |
| *** Oku_OS-away is now known as Oku_OS | 07:47 | |
| *** ChrisBenson1 has joined #openstack-ansible | 07:51 | |
| *** ChrisBenson has quit IRC | 07:51 | |
| *** neilus has joined #openstack-ansible | 07:55 | |
| *** fawadkhaliq has quit IRC | 08:01 | |
| mancdaz | jmccrory no, odyssey4me had asked me to take a look at the issues with secondary and tertiary nodes joining the new cluster in the gate. Your patch addresses a definite problem, but I'm not sure it addresses this specific problem | 08:03 |
| *** javeriak_ has joined #openstack-ansible | 08:10 | |
| *** javeriak has quit IRC | 08:10 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-security: [WIP] Unattended upgrades https://review.openstack.org/304096 | 08:12 |
| *** ChrisBenson1 has quit IRC | 08:24 | |
| *** thorst has joined #openstack-ansible | 08:39 | |
| *** agireud has quit IRC | 08:39 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance: Update min_ansible_version to 1.9 https://review.openstack.org/304040 | 08:40 |
| *** neilus has quit IRC | 08:41 | |
| *** neilus has joined #openstack-ansible | 08:41 | |
| *** agireud has joined #openstack-ansible | 08:42 | |
| *** agireud has quit IRC | 08:44 | |
| *** thorst has quit IRC | 08:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone: Update min_ansible_version to 1.9 https://review.openstack.org/304044 | 08:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: If a global pip.conf file exists, let the AIO use it for containers https://review.openstack.org/304452 | 08:46 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: If a global pip.conf file exists, let the AIO use it for containers https://review.openstack.org/304453 | 08:46 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Update min_ansible_version to 1.9 https://review.openstack.org/304045 | 08:47 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder: Update min_ansible_version to 1.9 https://review.openstack.org/304039 | 08:47 |
| openstackgerrit | Merged openstack/openstack-ansible: Minor fix to correct passive to active voice https://review.openstack.org/304351 | 08:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat: Update min_ansible_version to 1.9 https://review.openstack.org/304042 | 08:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon: Update min_ansible_version to 1.9 https://review.openstack.org/304043 | 08:52 |
| *** agireud has joined #openstack-ansible | 08:52 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/openstack-ansible: Fix idempotency bug in AIO bootstrap https://review.openstack.org/304456 | 08:52 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins: Update the config_template plugin https://review.openstack.org/304385 | 08:55 |
| *** nhadzter has joined #openstack-ansible | 09:10 | |
| *** thorst has joined #openstack-ansible | 09:12 | |
| *** openstackgerrit has quit IRC | 09:17 | |
| *** openstackgerrit has joined #openstack-ansible | 09:18 | |
| *** javeriak_ has quit IRC | 09:18 | |
| *** asettle has joined #openstack-ansible | 09:18 | |
| *** asettle has quit IRC | 09:23 | |
| *** javeriak has joined #openstack-ansible | 09:27 | |
| *** thorst has quit IRC | 09:29 | |
| *** yatin has joined #openstack-ansible | 09:30 | |
| *** Oku_OS is now known as Oku_OS-away | 09:31 | |
| *** markvoelker has joined #openstack-ansible | 09:40 | |
| *** yatin has quit IRC | 09:44 | |
| *** markvoelker has quit IRC | 09:45 | |
| *** Oku_OS-away is now known as Oku_OS | 09:49 | |
| *** openstackstatus has quit IRC | 09:57 | |
| *** openstack has joined #openstack-ansible | 10:01 | |
| *** yatin has joined #openstack-ansible | 10:07 | |
| *** yatin has quit IRC | 10:07 | |
| *** yatin has joined #openstack-ansible | 10:08 | |
| odyssey4me | mattt could you take a peek at https://review.openstack.org/303770 please? | 10:09 |
| *** sdake has quit IRC | 10:11 | |
| *** yatin has quit IRC | 10:19 | |
| *** yatin has joined #openstack-ansible | 10:22 | |
| *** javeriak has quit IRC | 10:36 | |
| *** javeriak has joined #openstack-ansible | 10:39 | |
| *** pjm6 has joined #openstack-ansible | 10:58 | |
| pjm6 | morning all | 10:58 |
| *** asettle has joined #openstack-ansible | 10:59 | |
| *** asettle has quit IRC | 10:59 | |
| *** yatin has quit IRC | 11:04 | |
| *** yatin has joined #openstack-ansible | 11:07 | |
| *** pjm6 has quit IRC | 11:10 | |
| *** johnmilton has joined #openstack-ansible | 11:13 | |
| *** pjm6 has joined #openstack-ansible | 11:14 | |
| *** jaypipes has joined #openstack-ansible | 11:28 | |
| *** clickboom has joined #openstack-ansible | 11:38 | |
| *** markvoelker has joined #openstack-ansible | 11:41 | |
| mattt | odyssey4me: sure, just going to grab a bite then i'll review | 11:42 |
| *** yatin has quit IRC | 11:44 | |
| *** retreved has joined #openstack-ansible | 11:44 | |
| *** markvoelker has quit IRC | 11:45 | |
| *** jayc has quit IRC | 11:46 | |
| *** Oku_OS is now known as Oku_OS-away | 11:51 | |
| *** asettle has joined #openstack-ansible | 11:53 | |
| *** neilus has quit IRC | 11:57 | |
| *** Oku_OS-away is now known as Oku_OS | 12:04 | |
| *** asettle has quit IRC | 12:04 | |
| *** neilus has joined #openstack-ansible | 12:04 | |
| *** openstack has quit IRC | 12:04 | |
| *** openstack has joined #openstack-ansible | 12:08 | |
| *** pjm6 has quit IRC | 12:12 | |
| *** markvoelker has joined #openstack-ansible | 12:12 | |
| *** tlbr has quit IRC | 12:14 | |
| *** retreved has joined #openstack-ansible | 12:14 | |
| Bofu2U | morning | 12:15 |
| *** tlbr has joined #openstack-ansible | 12:15 | |
| *** pjm6 has joined #openstack-ansible | 12:17 | |
| *** klamath has joined #openstack-ansible | 12:19 | |
| *** klamath has quit IRC | 12:19 | |
| *** klamath has joined #openstack-ansible | 12:20 | |
| *** jamielennox|away is now known as jamielennox | 12:26 | |
| *** v1k0d3n has joined #openstack-ansible | 12:26 | |
| *** Oku_OS is now known as Oku_OS-away | 12:26 | |
| *** pjm6 has quit IRC | 12:27 | |
| *** b3rnard0_away is now known as b3rnard0 | 12:28 | |
| *** severion has joined #openstack-ansible | 12:29 | |
| *** v1k0d3n has quit IRC | 12:29 | |
| *** thorst has joined #openstack-ansible | 12:32 | |
| *** chhavi has joined #openstack-ansible | 12:32 | |
| mhayden | buenos dias | 12:32 |
| *** saneax is now known as saneax_AFK | 12:35 | |
| *** thorst has quit IRC | 12:36 | |
| *** thorst has joined #openstack-ansible | 12:37 | |
| *** gregfaust has joined #openstack-ansible | 12:37 | |
| *** thorst has quit IRC | 12:41 | |
| *** tlbr has quit IRC | 12:43 | |
| *** keedya has joined #openstack-ansible | 12:43 | |
| *** tlbr has joined #openstack-ansible | 12:43 | |
| mhayden | who broke gerrit? :P | 12:45 |
| *** neilus1 has joined #openstack-ansible | 12:47 | |
| gregfaust | mhayden: possibly a team effort: https://etherpad.openstack.org/p/gerrit_server_replacement | 12:49 |
| *** neilus has quit IRC | 12:50 | |
| mattt | mhayden: can you peep https://review.openstack.org/#/c/304096/ when you get a minute? | 12:51 |
| *** elgertam1 has joined #openstack-ansible | 12:51 | |
| mattt | mhayden: i'm not that familiar w/ ubuntu auto update best practices, so feedback welcome :) | 12:51 |
| mhayden | mattt: last time i looked at it, i wanted to cry | 12:52 |
| * mhayden will gander | 12:52 | |
| mhayden | thanks for taking that on! | 12:52 |
| *** neilus1 has quit IRC | 12:53 | |
| *** tlbr has quit IRC | 12:54 | |
| *** Oku_OS-away is now known as Oku_OS | 12:56 | |
| *** elgertam1 has quit IRC | 12:58 | |
| *** javeriak has quit IRC | 13:00 | |
| *** briancubed has joined #openstack-ansible | 13:07 | |
| *** tlbr has joined #openstack-ansible | 13:08 | |
| *** neilus has joined #openstack-ansible | 13:09 | |
| admin0 | feature request: https://cloudplatform.googleblog.com/2016/04/OpenStack-users-backup-your-Cinder-volumes-to-Google-Cloud-Storage.html :D | 13:11 |
| *** automagically has joined #openstack-ansible | 13:13 | |
| automagically | morning | 13:16 |
| *** pjm6 has joined #openstack-ansible | 13:16 | |
| evrardjp | morning automagically | 13:17 |
| mgariepy | good morning all | 13:17 |
| *** Bjoern_ has joined #openstack-ansible | 13:19 | |
| *** Bjoern_ is now known as Bjoern_zZzZzZzZ | 13:19 | |
| *** galstrom_zzz is now known as galstrom | 13:22 | |
| *** tlbr has quit IRC | 13:23 | |
| *** tlbr has joined #openstack-ansible | 13:24 | |
| *** yatin has joined #openstack-ansible | 13:25 | |
| briancubed | Are admin0, cloudnull, and odyssey4me around? | 13:28 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-security: WIP] Unattended upgrades https://review.openstack.org/304096 | 13:28 |
| * admin0 is hiding under the bed | 13:28 | |
| admin0 | briancubed: you found me .. whats up | 13:29 |
| briancubed | just wanted to report that I found the root cause for the ssh timeouts I was seeing last week. It's, well, silly. | 13:29 |
| briancubed | I had used the wrong network mask on the internal network for containers. 255.255.255.0 instead of the correct 255.255.252.0 | 13:30 |
| briancubed | Once I found and fixed it, lxc_container_create playbook ran to completion without error. | 13:30 |
| admin0 | oh .. not in the config but in your interfaces in ubuntu ? | 13:30 |
| briancubed | yes | 13:30 |
| admin0 | :D | 13:31 |
| admin0 | how many days spent ? | 13:31 |
| admin0 | on that ? | 13:31 |
| briancubed | happy that it works, frustrated that it was such a silly mistake. | 13:31 |
| admin0 | how big was the smile on your face when you found out :D ? | 13:32 |
| briancubed | It feels good to have it behind me, for sure. Days? Calendar days 5. But I was time-slicing between this and 3 other activities. So may 2 days of effort. | 13:32 |
| admin0 | :) | 13:33 |
| briancubed | I wanted to let you know that it's fixed. Thanks for listening | 13:33 |
| admin0 | ok .. thanks for the feedback .. we will know what to check next when someone else reports the same issue | 13:33 |
| *** busterswt has joined #openstack-ansible | 13:34 | |
| *** Bjoern_zZzZzZzZ is now known as Bjoern_ | 13:37 | |
| *** Oku_OS is now known as Oku_OS-away | 13:38 | |
| lbragstad | mhayden I hear you're looking to rework a bunch of the ssl cert stuff in osa? | 13:38 |
| *** Oku_OS-away is now known as Oku_OS | 13:39 | |
| admin0 | lbragstad: we want to have all endpoints in SSL - i think :D | 13:39 |
| mhayden | lbragstad: i did? :) | 13:39 |
| mhayden | ol' cloudnull is working that angle | 13:39 |
| mhayden | i need to gander at his patch | 13:39 |
| *** czunker has quit IRC | 13:39 | |
| lbragstad | mhayden dolphm must have volu-told you | 13:39 |
| *** jthorne has joined #openstack-ansible | 13:40 | |
| *** asettle has joined #openstack-ansible | 13:42 | |
| mhayden | haha | 13:42 |
| lbragstad | mhayden ah - this must be the patch you're talking about https://review.openstack.org/#/c/277199/ | 13:43 |
| admin0 | lbragstad: https://review.openstack.org/#/c/277199/ — this is the one you are referring to ? | 13:43 |
| mhayden | i've wanted it for a while, but i've been hacking more on the internals (like rabbitmq) | 13:43 |
| admin0 | that patch is in one of my Must haves :D | 13:43 |
| *** ametts has joined #openstack-ansible | 13:43 | |
| lbragstad | I'm only curious because I'm trying to deploy keystone with self-signed certs | 13:43 |
| lbragstad | but it doesn't look like cloudnull's patch will affect what I'm doing | 13:44 |
| *** rohanp_ has joined #openstack-ansible | 13:45 | |
| *** mgoddard_ has joined #openstack-ansible | 13:45 | |
| *** asettle has quit IRC | 13:46 | |
| *** mgoddard has quit IRC | 13:49 | |
| *** sanjay__u has joined #openstack-ansible | 13:51 | |
| *** yatin_ has joined #openstack-ansible | 13:54 | |
| *** neilus has quit IRC | 13:55 | |
| cloudnull | morning | 13:56 |
| *** yatin has quit IRC | 13:57 | |
| cloudnull | briancubed: you pinged ping <-> pong | 13:57 |
| *** stian__ has quit IRC | 13:57 | |
| admin0 | hello cloudnull | 13:58 |
| admin0 | yes … we are all asking for SSL :D | 13:58 |
| admin0 | ;) | 13:59 |
| cloudnull | ha | 13:59 |
| *** Mudpuppy has joined #openstack-ansible | 13:59 | |
| admin0 | remember he had a SSH connection issue | 13:59 |
| *** Mudpuppy has quit IRC | 14:00 | |
| admin0 | he was seekign us to tell that his netmask in interfaces was /24 while in the config was /22 | 14:00 |
| admin0 | solved and he is happy | 14:00 |
| *** Mudpuppy has joined #openstack-ansible | 14:00 | |
| *** yatin_ has quit IRC | 14:00 | |
| cloudnull | ah . | 14:01 |
| briancubed | yes, happy is the word, admin0 | 14:01 |
| cloudnull | that will do it :) | 14:01 |
| * cloudnull is reading scroll back | 14:01 | |
| briancubed | thank you all for the assistance | 14:02 |
| adreznec | Hey odyssey4me, thanks for the reviews on https://review.openstack.org/#/c/302941 so far. In the comments you mention pivoting all nova.conf settings on nova_virt_type key going forward. Does having the driver-specific config included only on a nova_virt_type conditional (like in the Ironic patch referenced) fulfill that requirement? | 14:02 |
| cloudnull | briancubed: hows it going btw (besides the now fixed cidr problems) ? | 14:06 |
| briancubed | Good, cloudnull. thanks for asking. I'm moving on to the rest of the playbooks for deployment this morning (EDT). I am very close to complete. | 14:07 |
| cloudnull | sweet! | 14:07 |
| admin0 | cloudnull: what release do I need to be in for this ? https://review.openstack.org/#/c/277199/ — mitaka ? | 14:08 |
| admin0 | so i need to checkout mitaka, pull in this change .. and do a deploy and cross fingers :D ? | 14:08 |
| cloudnull | admin0: master which would be newton | 14:09 |
| cloudnull | we may be able to backport to mitaka | 14:09 |
| cloudnull | but at present its master(newton) | 14:09 |
| admin0 | if i have to wait another 6 months for a SSL, its bye bye ansible :D | 14:09 |
| admin0 | i can help backporting to mitaka :D | 14:09 |
| *** thorst_ has joined #openstack-ansible | 14:10 | |
| cloudnull | so everything to do ssl is in mitaka | 14:10 |
| cloudnull | this patch just turns it on | 14:10 |
| cloudnull | so if you clone openstack-ansible you can cherry pick this patch and run and all will be well | 14:11 |
| cloudnull | git fetch https://git.openstack.org/openstack/openstack-ansible refs/changes/99/277199/21 && git cherry-pick FETCH_HEAD | 14:11 |
| admin0 | my test bed is currently liberty .. i need to be in at least mitaka for that right ? | 14:11 |
| cloudnull | i think so | 14:12 |
| cloudnull | i mean you can implement the horizon keystone and nova scheme pass through using the config_template in liberty | 14:12 |
| cloudnull | then pull in the patch and same thing | 14:12 |
| cloudnull | but mitaka is the first to "officially" support it | 14:12 |
| admin0 | i will start all new deployments in mitaka now | 14:13 |
| *** spotz_zzz is now known as spotz | 14:13 | |
| mancdaz | odyssey4me have we lost all git commit history from anything moved into separate repo/roles? | 14:13 |
| cloudnull | cool | 14:13 |
| admin0 | if that is our current “stable” relase | 14:13 |
| cloudnull | mancdaz: some yes, some no. | 14:13 |
| mancdaz | :sadface: | 14:13 |
| cloudnull | it is the stable current release | 14:13 |
| cloudnull | mancdaz: looking for anything in particular? | 14:14 |
| mattt | cloudnull: i thought most was retained | 14:14 |
| admin0 | maybe i will do a upgrade first and see if it upgrades . by just checking out to the mitaka branch :D | 14:14 |
| cloudnull | mattt: in the os_* roles we were able to keep it all | 14:14 |
| cloudnull | in the rest no | 14:14 |
| mattt | cloudnull: ok | 14:14 |
| cloudnull | that was my fault | 14:15 |
| mancdaz | cloudnull yeah, wondering why we create mysql data dirs with ansible when mysql will do it itself. I'm guessing it's a timing thing since we prevent mysql starting up until we've dropped config into place, or something | 14:15 |
| cloudnull | i hadnt figured out how to keep the history until we were at that point | 14:15 |
| cloudnull | mancdaz: i believe its exactly that | 14:15 |
| mancdaz | cloudnull trying to debuf second/third node join fails | 14:15 |
| mancdaz | debug | 14:15 |
| mattt | cloudnull: yep :( but i'm glad we were able to keep a bunch of history | 14:16 |
| *** woodard has joined #openstack-ansible | 14:16 | |
| *** woodard has quit IRC | 14:16 | |
| cloudnull | itll do the data dir create on startup LIC , so we create it. | 14:16 |
| cloudnull | mancdaz: is it throwing an error there? | 14:16 |
| *** woodard has joined #openstack-ansible | 14:17 | |
| cloudnull | is it an sst problem maybe ? | 14:17 |
| mancdaz | cloudnull yeah something weird is happening on the initial SST that causes a failure. If you log in later and remove the .sst dir, it will work. But the .sst dir is not there in the first place so that can't be the initial problem... | 14:17 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:18 | |
| cloudnull | so weve seen that before | 14:18 |
| cloudnull | we added https://github.com/openstack/openstack-ansible-galera_server/blob/master/handlers/main.yml | 14:18 |
| cloudnull | which should clean that up | 14:19 |
| cloudnull | however master is a little different than liberty | 14:19 |
| mancdaz | cloudnull yeah, it cleans it up after the first 3x fail, then the fallback restart also fails so it's not fixing it | 14:19 |
| mancdaz | but a manual removal of .sst later does fix it | 14:19 |
| mancdaz | so still trying to work it out | 14:19 |
| cloudnull | git-harry: did some tune up here https://github.com/openstack/openstack-ansible-galera_server/commit/72a1dfb4d71eceae43ad6a4fb0bd986205c92484 -- so maybe you need to pull that in and try ? | 14:19 |
| mancdaz | cloudnull this is in master | 14:20 |
| odyssey4me | briancubed good to see that you found it - it's crazy how often we stumble on the basics :) | 14:20 |
| *** Bjoern_ is now known as BjoernT | 14:20 | |
| cloudnull | ah . | 14:20 |
| cloudnull | mancdaz: nevermind me :) | 14:20 |
| mancdaz | cloudnull see Jimmy comment on https://review.openstack.org/#/c/303770/2 | 14:21 |
| *** pjm6 has quit IRC | 14:21 | |
| briancubed | odyssey4me yeah. Wanted to do the dance of joy with my head down in shame. ;-) | 14:21 |
| cloudnull | ok . | 14:22 |
| cloudnull | briancubed: shit happens :) | 14:22 |
| briancubed | :-) | 14:23 |
| * cloudnull if i had a nickle every time i broke a cloud... | 14:23 | |
| *** michaelgugino has joined #openstack-ansible | 14:23 | |
| *** pjm6 has joined #openstack-ansible | 14:23 | |
| odyssey4me | adreznec perhaps, I guess that it'll come down to the details in review - I don't think the spec should spell too much out, but I think that both cloudnull and I were trying to ensure that the spec did make reference to related work and existing patterns which the spec should make reference to and that the spec work should try to use as far as possible, or evolve to something better | 14:23 |
| odyssey4me | mancdaz commit history has been lost for the first set of roles - but all the openstack roles have kept the history thanks to a trick jmccrory gave to cloudnull :) | 14:24 |
| *** andrei_ has quit IRC | 14:24 | |
| cloudnull | mancdaz: hum... thats an odd one. happy to help / debug where i can. | 14:25 |
| adreznec | odyssey4me: Yeah ok, totally agree. Wanted to make sure I wasn't completely off-base before I pushed up my next spec iteration | 14:25 |
| spotz | Morning gang | 14:26 |
| cloudnull | adreznec: ++ Im excited to see OpenPower as a compute option. | 14:26 |
| odyssey4me | ++ :) | 14:27 |
| odyssey4me | ok, caught up on scrollback - time for some coffee | 14:27 |
| cloudnull | morning spotz | 14:27 |
| cloudnull | thanks for the review on the ssl patch yesterday :) | 14:27 |
| *** phalmos has joined #openstack-ansible | 14:29 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 14:29 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:30 | |
| *** sdake has joined #openstack-ansible | 14:32 | |
| *** flwang has quit IRC | 14:32 | |
| *** pjm6 has quit IRC | 14:33 | |
| evrardjp | admin0 if you really want to have all this SSL sorted out before the next release, you could run your own configuration for haproxy, it would solve all your problems | 14:35 |
| *** flwang has joined #openstack-ansible | 14:36 | |
| admin0 | evrardjp: i can manually do that, but i would prefer if its managed by ansible :) | 14:36 |
| admin0 | so if cherry picking the review from cloudnull works, that would be great as well | 14:36 |
| evrardjp | you mean openstack-ansible? | 14:37 |
| admin0 | which i plan to do this evening | 14:37 |
| admin0 | yep | 14:37 |
| cloudnull | ++ that should work just fine, all of the framework for enabling ssl termination is already there in mitaka | 14:38 |
| admin0 | “mitaka .. here i come” :D | 14:39 |
| mancdaz | jmccrory ping | 14:39 |
| *** pjm6 has joined #openstack-ansible | 14:39 | |
| *** weezS has joined #openstack-ansible | 14:41 | |
| evrardjp | cloudnull what I also liked with my haproxy role, it's the ssl_termination option, to let the deployer easily chose what he wants. but I guess if you do SSL all the way, life is going to be simpler | 14:42 |
| evrardjp | good commit | 14:43 |
| *** mgoddard_ has quit IRC | 14:43 | |
| *** elgertam has joined #openstack-ansible | 14:44 | |
| *** mgoddard has joined #openstack-ansible | 14:44 | |
| *** sdake_ has joined #openstack-ansible | 14:51 | |
| adreznec | cloudnull: Same here, it should be pretty cool to get running | 14:54 |
| *** sdake has quit IRC | 14:55 | |
| *** ametts has quit IRC | 14:59 | |
| prometheanfire | which review is that one? | 14:59 |
| prometheanfire | the ssl terminiation one | 14:59 |
| admin0 | say yes :D | 15:01 |
| *** thorst_ has quit IRC | 15:02 | |
| *** galstrom is now known as galstrom_zzz | 15:02 | |
| *** metral is now known as metral_zzz | 15:07 | |
| *** Brew has joined #openstack-ansible | 15:08 | |
| *** phalmos has quit IRC | 15:09 | |
| *** sdake has joined #openstack-ansible | 15:12 | |
| *** metral_zzz is now known as metral | 15:15 | |
| *** sdake_ has quit IRC | 15:16 | |
| openstackgerrit | Rohan Parulekar proposed openstack/openstack-ansible-os_nova: Nuage nova configuration ansible changes https://review.openstack.org/296538 | 15:16 |
| *** openstackgerrit has quit IRC | 15:18 | |
| *** openstackgerrit has joined #openstack-ansible | 15:18 | |
| *** galstrom_zzz is now known as galstrom | 15:19 | |
| *** jayc has joined #openstack-ansible | 15:19 | |
| *** TxGVNN has joined #openstack-ansible | 15:24 | |
| *** phalmos has joined #openstack-ansible | 15:28 | |
| *** asettle has joined #openstack-ansible | 15:30 | |
| *** logan- has quit IRC | 15:34 | |
| *** asettle has quit IRC | 15:35 | |
| *** yarkot_ has joined #openstack-ansible | 15:38 | |
| *** phalmos has quit IRC | 15:39 | |
| *** logan- has joined #openstack-ansible | 15:40 | |
| *** sdake_ has joined #openstack-ansible | 15:41 | |
| *** eil397 has joined #openstack-ansible | 15:42 | |
| *** galstrom is now known as galstrom_zzz | 15:43 | |
| *** sdake has quit IRC | 15:43 | |
| *** phalmos has joined #openstack-ansible | 15:43 | |
| *** mikelk has quit IRC | 15:43 | |
| *** fawadkhaliq has joined #openstack-ansible | 15:55 | |
| briancubed | My deployment joy was short-lived. I am hitting an error deploying galera on infra1. (infra2 and infra3 passed...) You can find a snippet from the log here: http://pastebin.com/bqcHVtQS | 15:55 |
| *** sdake has joined #openstack-ansible | 15:55 | |
| *** sdake_ has quit IRC | 15:55 | |
| *** gregfaust has quit IRC | 15:55 | |
| odyssey4me | briancubed it looks like you're using an old tag - which tag /branch is it? | 15:55 |
| briancubed | i'll need to check with rohan on that. This is his set of playbooks from his fork/PR. | 15:56 |
| briancubed | this is the nuage integration work... | 15:56 |
| briancubed | When you say old tag, are you referring to the os-releases/12.0.8 that appears in the log? | 15:58 |
| *** galstrom_zzz is now known as galstrom | 15:58 | |
| briancubed | FWIW, the error in the log is the second. The first was the same error on python-memcached. I was able to attach to the container and do a pip install. The next time I ran the playbook it hit this next error. | 16:00 |
| odyssey4me | hmm, 'No route to host' definitely seems a little simpler - networking from the container to the LB or from the LB to the repo container isn't working | 16:01 |
| odyssey4me | I expect that it's more likely to be the first | 16:01 |
| odyssey4me | as the second would give you a 404 error | 16:01 |
| odyssey4me | bug triage here cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak, automagically, spotz, vdo, jmccrory, alextricity25, jasondotstar, KLevenstein, admin0, michaelgugino, ametts, v1k0d3n, severion, bgmccollum | 16:02 |
| briancubed | Sorry. What's LB? | 16:02 |
| izaakk | o/ | 16:02 |
| spotz | \o/ | 16:02 |
| odyssey4me | briancubed LB = load balancer | 16:02 |
| *** jmccrory_ has joined #openstack-ansible | 16:03 | |
| odyssey4me | briancubed make sure that host network configs are right, and all containers too | 16:03 |
| odyssey4me | https://bugs.launchpad.net/openstack-ansible/+bugs?search=Search&field.status=New | 16:03 |
| briancubed | okay. something seems inconsistent in the config because the galera containers on infra2 and infra3 didn't experience this error. | 16:04 |
| *** admin0 has quit IRC | 16:04 | |
| odyssey4me | first up: https://bugs.launchpad.net/openstack-ansible/+bug/1568029 | 16:04 |
| openstack | Launchpad bug 1568029 in openstack-ansible "Security: Disable role during major version upgrades" [Wishlist,New] | 16:04 |
| odyssey4me | briancubed we'll get back to it after bug triage, but I suspect that this action is only run against that container and skipped on the others? | 16:05 |
| odyssey4me | mhayden this seems like a documentation itsem, but perhaps also an edit to the upgrade automation to disable the hardening when executing the playbooks | 16:05 |
| *** michaelgugino_ has joined #openstack-ansible | 16:06 | |
| evrardjp | makes sense but the security role isn't applied by default right? | 16:06 |
| michaelgugino_ | here | 16:06 |
| odyssey4me | can you self assign and look into this? | 16:06 |
| automagically | o/ | 16:06 |
| odyssey4me | evrardjp yeah, but in the upgrade scripts it makes better sense to be certain | 16:06 |
| *** michaelgugino has quit IRC | 16:07 | |
| evrardjp | assign to mhayden for the upgrade scripts part? | 16:07 |
| odyssey4me | I see that https://bugs.launchpad.net/openstack-ansible/+bug/1568070 has been picked up by Ala | 16:07 |
| openstack | Launchpad bug 1568070 in openstack-ansible "Security: Identify which changes require a reboot" [Wishlist,New] - Assigned to Ala Raddaoui (raddaoui-ala) | 16:07 |
| raddaoui | 0/ | 16:07 |
| mhayden | sorry, running behind | 16:07 |
| mhayden | odyssey4me: would probably be a doc | 16:07 |
| odyssey4me | ah, I'll mark that as confirmed raddaoui | 16:08 |
| odyssey4me | mhayden yeah, makes sense to me | 16:08 |
| odyssey4me | next up https://bugs.launchpad.net/openstack-ansible/+bug/1566629 | 16:08 |
| openstack | Launchpad bug 1566629 in openstack-ansible "Missing insecure flag for [neutron] section of nova.conf" [Undecided,New] - Assigned to Ala Raddaoui (raddaoui-ala) | 16:08 |
| odyssey4me | this same issue may also be in the heat conf templates and other places where keystone auth is used from a client standpoint | 16:09 |
| odyssey4me | this seems pretty high importance to me? thoughts anyone? | 16:09 |
| *** fawadkhaliq has quit IRC | 16:09 | |
| automagically | agreed | 16:10 |
| raddaoui | noted | 16:10 |
| odyssey4me | thanks raddaoui - marked as confirmed after inspection, and high importance | 16:11 |
| odyssey4me | next up https://bugs.launchpad.net/openstack-ansible/+bug/1566985 | 16:11 |
| openstack | Launchpad bug 1566985 in openstack-ansible "Policies do not support multi domain setups" [Undecided,New] | 16:11 |
| odyssey4me | this is a wishlist item, but not something we should support exactly | 16:12 |
| odyssey4me | our stance is to only use the upstream defaults | 16:12 |
| odyssey4me | if there's anything we should be doing, is to allow deployers to upload their own custom policies | 16:12 |
| automagically | That sounds reasonable | 16:12 |
| *** fawadkhaliq has joined #openstack-ansible | 16:13 | |
| odyssey4me | right now we allow a config_override, but if the policy changes upstream the config override may result in a broken policy file | 16:14 |
| odyssey4me | I'm inclined to say that as the bug is written now, this is invalid. We would accept a feature to allow a deployer to upload a custom policy file, but we'll not be changing the default policy file. | 16:15 |
| odyssey4me | thoguhts? | 16:15 |
| michaelgugino_ | bug is not really specific in what they're asking | 16:18 |
| stevelle | think I agree with that | 16:18 |
| stevelle | also, would be nice if there was a way to validate a policy file format | 16:19 |
| evrardjp | odyssey4me I agree | 16:20 |
| evrardjp | stevelle isn't it standard json ? | 16:20 |
| stevelle | evrardjp: not all json is a valid policy | 16:20 |
| evrardjp | k | 16:21 |
| palendae | evrardjp: Might need to validate the included keys | 16:21 |
| odyssey4me | marking as won't fix - but I've added a note | 16:21 |
| odyssey4me | yeah, validating a policy is something that should perhaps be included in the cross project discussion around the config classification | 16:22 |
| stevelle | +1 | 16:22 |
| evrardjp | stevelle a tool for validating json policies should be done by keystone guys, right? | 16:22 |
| evrardjp | or that, yes | 16:22 |
| odyssey4me | we've asked in that for devops tooling to validate config files, and perhaps policy files should be included in that initiative - or as a follow-on initiative | 16:22 |
| odyssey4me | evrardjp I expect that something should be done in oslo to provide the tool, but validation would have to be done by the projects as each policy file would need to reflect the API for the project | 16:23 |
| odyssey4me | anyway, we digress | 16:23 |
| odyssey4me | next up: https://bugs.launchpad.net/openstack-ansible/+bug/1569171 | 16:23 |
| openstack | Launchpad bug 1569171 in openstack-ansible "Logging not enabled for memcached" [Undecided,New] | 16:23 |
| *** weezS has quit IRC | 16:24 | |
| *** javeriak has joined #openstack-ansible | 16:24 | |
| odyssey4me | it seems valid as a wishlist item - we'd need to ensure that log rotation and rsyslog redirection is also implemented | 16:24 |
| odyssey4me | thoughts? | 16:24 |
| automagically | Seems like a good wishlist item | 16:24 |
| evrardjp | it seems right | 16:25 |
| evrardjp | I'll take it | 16:25 |
| odyssey4me | ok cool, thanks evrardjp | 16:26 |
| *** pjm6 has quit IRC | 16:26 | |
| odyssey4me | next up https://bugs.launchpad.net/openstack-ansible/+bug/1569446 | 16:26 |
| openstack | Launchpad bug 1569446 in openstack-ansible "Secondary nodes fail to join galera cluster" [Undecided,New] | 16:26 |
| odyssey4me | YEah, I've seen this - it's very evident in the galera_server role. | 16:26 |
| odyssey4me | I know that mancdaz and jmccrory have been poking at it. | 16:26 |
| odyssey4me | I'm inclined to call this a critical bug. Thoughts? | 16:27 |
| odyssey4me | If not critical - then High. | 16:27 |
| evrardjp | critical | 16:27 |
| automagically | If its intermittent, perhaps High | 16:27 |
| evrardjp | "Secondary nodes fail to join galera cluster" sounds really bad | 16:27 |
| automagically | But, I don’t have strong feelings either way | 16:28 |
| *** TxGVNN has quit IRC | 16:28 | |
| jmccrory | between that and the 30+ min runtime of the role. it's extremely hard to get commits in galera_server | 16:28 |
| mancdaz | odyssey4me yes I'm continuing to try and decipher exactly what's going on | 16:28 |
| mancdaz | odyssey4me so far all my theories have been disproved | 16:29 |
| odyssey4me | ++ jmccrory | 16:29 |
| *** flwang has quit IRC | 16:29 | |
| mancdaz | but I will continue | 16:29 |
| odyssey4me | I'm waiting for the Ceph mirrors to be added to OpenStack-CI's mirrors as it sets a precedent, after which I can follow with a MariaDB mirror which will likely make a big difference to the run time. | 16:30 |
| odyssey4me | But alas the ceph mirror patch is languishing. Maybe I should step that up and try and get something going there. Let me see what I can do. | 16:30 |
| odyssey4me | mancdaz / jmccrory who will self assign that one? | 16:30 |
| *** pjm6 has joined #openstack-ansible | 16:30 | |
| mancdaz | odyssey4me I'll take it for now if you like | 16:31 |
| michaelgugino_ | perhaps there needs to be a check in the play that validates the secondary host has joined the cluster | 16:31 |
| mancdaz | might need to hand it off if my day job takes over | 16:31 |
| jmccrory | i'll keep helping wherever i can on it as well | 16:31 |
| odyssey4me | mancdaz jmccrory perhaps the best is to each take over the bug when you're working on it, and to add your daily notes at the end of each day? | 16:33 |
| jmccrory | michaelgugino_ the problem seems to be the join itself. state transfer is failing for some reason | 16:33 |
| *** flwang has joined #openstack-ansible | 16:33 | |
| mancdaz | odyssey4me sure I can add notes later | 16:33 |
| jmccrory | odyssey4me works for me | 16:34 |
| odyssey4me | awesome, thanks | 16:34 |
| odyssey4me | that's it for the new bug list (the others are all waiting for confirmation and have been previously discussed) | 16:35 |
| openstackgerrit | Steve Lewis (stevelle) proposed openstack/openstack-ansible-os_gnocchi: Enable ansible lint and syntax tests https://review.openstack.org/304343 | 16:35 |
| michaelgugino_ | I know I've seen mysql fail to replicate if a server is started as id 0, and then restarted under another id. | 16:36 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh: Updated role using the Multi-Distro framework https://review.openstack.org/295620 | 16:36 |
| michaelgugino_ | so, the correct server id needs to be in my.cnf before the server is started initially, or things aren't going to go well for replication. | 16:36 |
| mancdaz | michaelgugino_ this is different from normal mysql replication so the server id doesn't matter | 16:36 |
| michaelgugino_ | I understand that it's different, but each server appears to have an id in the logs | 16:37 |
| mancdaz | galera/wsrep does things differently - on initial start of a secondary node, it connects to the primary server and does a full SST (State snapshot transfer) by streaming the entire contents of the mysql data dir from the primary node to the secondary node | 16:38 |
| mancdaz | this SST is what's failing, but it's not to do with server IDs | 16:38 |
| *** jwagner is now known as jwagner_lunch | 16:39 | |
| *** eil397 has quit IRC | 16:40 | |
| *** Oku_OS is now known as Oku_OS-away | 16:41 | |
| *** eil397 has joined #openstack-ansible | 16:41 | |
| *** ChrisBenson has joined #openstack-ansible | 16:44 | |
| *** woodard has quit IRC | 16:45 | |
| michaelgugino_ | looks like from the paste logs that a transfer was started and failed at some point | 16:46 |
| *** fawadkhaliq has quit IRC | 16:50 | |
| briancubed | odyssey4me still there? | 16:52 |
| *** fawadkhaliq has joined #openstack-ansible | 16:53 | |
| *** pjm6 has quit IRC | 16:54 | |
| *** elgertam has quit IRC | 16:59 | |
| michaelgugino_ | everyone leave? | 17:01 |
| *** b3rnard0 is now known as b3rnard0_away | 17:05 | |
| *** elgertam has joined #openstack-ansible | 17:06 | |
| *** ggillies has quit IRC | 17:08 | |
| *** michaelgugino_ has quit IRC | 17:10 | |
| *** ggillies has joined #openstack-ansible | 17:10 | |
| *** michaelgugino has joined #openstack-ansible | 17:10 | |
| *** mgoddard has quit IRC | 17:11 | |
| *** admin0 has joined #openstack-ansible | 17:11 | |
| *** mgoddard has joined #openstack-ansible | 17:12 | |
| odyssey4me | briancubed back - popped away from the desk for a bit | 17:12 |
| *** admin0 has quit IRC | 17:14 | |
| *** admin0 has joined #openstack-ansible | 17:14 | |
| briancubed | hey, odyssey4me | 17:24 |
| * stevelle is almost expecting a knock knock joke | 17:24 | |
| briancubed | i want to poke a bit at your assertion about LB. I don't have an LB. The docs say I can use haproxy, instead. | 17:24 |
| briancubed | so i'm thinking i have the config wrong | 17:25 |
| briancubed | when I did the openstack_user_config.yml, I didn't know what IP addresses to assign to the vips | 17:26 |
| briancubed | setting up a pastebin... | 17:27 |
| briancubed | snippet of user config: http://pastebin.com/scsMXa3C | 17:28 |
| odyssey4me | briancubed yep, so haproxy is the lb | 17:28 |
| briancubed | Right, so what should internal_lb_vip_address be set to? | 17:29 |
| odyssey4me | briancubed is this an AIO, or a multi-node environment? | 17:29 |
| briancubed | odyssey4me multi node | 17:29 |
| briancubed | 3 infra, 1 log, 1 compute | 17:30 |
| odyssey4me | ok, and is that address reachable by the containers and hosts? | 17:30 |
| briancubed | (all vms running on KVM) | 17:30 |
| odyssey4me | do you have the hosts setup in openstack_user_config? one of the groups set there should be haproxy_hosts, for example: https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio#L130-L132 | 17:31 |
| briancubed | the address in the file at the moment (I just changed it 10 minutes ago) is the container-network address of infra1 | 17:31 |
| briancubed | so, yes, it can be reached | 17:31 |
| odyssey4me | ok, in that case you should ensure that haproxy_hosts has one member and that is infra1 | 17:32 |
| briancubed | but I don't think it's correct. With this config, the error has changed to "too many 503 errors" | 17:32 |
| odyssey4me | if that's done, then run the haproxy-install.yml playbook and haproxy will actually be setup on infra1 | 17:32 |
| briancubed | ah, so maybe that's the problem. When I ran haproxy-install.yml the first time I had the vip addr set to an open address, one not used by any server or node | 17:33 |
| *** woodard has joined #openstack-ansible | 17:34 | |
| odyssey4me | so external_lb_vip_address is only used for the keystone public endpoints when the endpoints are setup | 17:34 |
| odyssey4me | internal_lb_vip_address is used for almost everything as the internal address to connect to for services, and that is meant to point at the internal vip for an environment's load balancer | 17:35 |
| odyssey4me | the load balancer may be a hardware LB, or haproxy | 17:35 |
| briancubed | excellent. good to know. I was wondering about that | 17:35 |
| odyssey4me | to actually setup haproxy, you have to tell the playbooks where to put it, which is why something like this is needed: https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio#L130-L132 | 17:35 |
| briancubed | so if I set internal_lb_vip_address to the container network address of infra1, re-run haproxy-install, then my galera install will work???? | 17:36 |
| odyssey4me | by 'container', you mean 'vm' right? | 17:36 |
| briancubed | right, VM. Sorry. | 17:37 |
| odyssey4me | ok, so if you have the group 'haproxy_hosts' in 'openstack_user_config.yml', with 'infra1' as the key, and the key:value pair of 'ip: <infra1's ip address>' | 17:38 |
| odyssey4me | and then you have 'internal_lb_vip_address: <infra1's ip address>' in global_overrides | 17:38 |
| odyssey4me | then you run the haproxy-install playbook (to actually setup haproxy) | 17:38 |
| odyssey4me | then the setup-infrastructure playbook will progress beyond the repo build | 17:39 |
| odyssey4me | the internal and external lb address can be the same, as long as you're not hoping to do SSL then it will just work | 17:40 |
| *** sdake_ has joined #openstack-ansible | 17:40 | |
| briancubed | I can do that. Thank you for the explanation. That goes a long way to aid my understanding. | 17:40 |
| odyssey4me | sure, no problem :) | 17:41 |
| *** chhavi has quit IRC | 17:41 | |
| *** sdake has quit IRC | 17:43 | |
| *** ametts has joined #openstack-ansible | 17:45 | |
| *** tricksters has joined #openstack-ansible | 17:48 | |
| *** tricksters is now known as elopez | 17:49 | |
| *** eric_lopez has quit IRC | 17:52 | |
| *** sdake has joined #openstack-ansible | 17:54 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:55 | |
| *** sdake_ has quit IRC | 17:57 | |
| *** yarkot_ has quit IRC | 17:57 | |
| *** javeriak_ has joined #openstack-ansible | 17:58 | |
| *** javeriak has quit IRC | 18:01 | |
| *** mkrish004c has joined #openstack-ansible | 18:01 | |
| mkrish004c | hi guys, i am trying out ceilometer installation via openstack ansible, i am not getting any notification from glance or any service. Do i need to install any agent in other service containers as well ? | 18:04 |
| mkrish004c | i just installed mongo DB on the node and ceilometer API and ceilometer collector in the respective containers | 18:05 |
| *** jwagner_lunch is now known as jwagner | 18:05 | |
| mkrish004c | and run aodh playbook as well | 18:06 |
| *** jayc has quit IRC | 18:06 | |
| palendae | mkrish004c: I don't know a ton about ceilometer, but I do know there's per-service variables to enable or disable it. Not sure what their defaults are | 18:07 |
| palendae | A downstream project disables them in its user_variables.yml file: https://github.com/rcbops/rpc-openstack/blob/master/rpcd/etc/openstack_deploy/user_variables.yml#L80-L87 | 18:08 |
| mkrish004c | @palendae, i have enabled notification in all the containers, but i am running this ceilometer playbook separately | 18:09 |
| mkrish004c | will that work | 18:09 |
| palendae | I believe the ceilometer playbook running by itself will work, so long as the right variables are set | 18:10 |
| mkrish004c | what is the purpose of aodh services, do i need to run that as well ? | 18:10 |
| palendae | Yeah, aodh is a new metrics gathering service I think | 18:10 |
| palendae | I know ceilometer depends on it as of liberty | 18:10 |
| *** weezS has joined #openstack-ansible | 18:12 | |
| *** javeriak_ has quit IRC | 18:12 | |
| *** javeriak has joined #openstack-ansible | 18:12 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:12 | |
| *** elgertam has quit IRC | 18:16 | |
| *** clickboom has quit IRC | 18:17 | |
| *** falanx has quit IRC | 18:17 | |
| *** javeriak_ has joined #openstack-ansible | 18:18 | |
| *** jayc has joined #openstack-ansible | 18:19 | |
| *** javeriak has quit IRC | 18:19 | |
| stevelle | aodh is the alarm engine for ceilometer | 18:20 |
| *** clickboom has joined #openstack-ansible | 18:20 | |
| stevelle | it was extracted from the ceilometer code base | 18:21 |
| mkrish004c | if i need to remove the ceilometer containers and re create this service alone without disturbing other services, is that possible ? | 18:21 |
| *** kukacz has quit IRC | 18:24 | |
| stevelle | mkrish004c: that should not disturb other services. | 18:25 |
| admin0 | git clone -b mitaka https://github.com/openstack/openstack-ansible.git /opt/openstack-ansible — we do not have mitaka yet :D ? | 18:26 |
| admin0 | how do I checkout/test it out ? | 18:26 |
| *** javeriak_ has quit IRC | 18:26 | |
| admin0 | hmm.. so 13.0.0 | 18:27 |
| admin0 | sorry :) | 18:27 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-os_barbican: Enable functional convergence testing https://review.openstack.org/301422 | 18:27 |
| *** falanx has joined #openstack-ansible | 18:27 | |
| palendae | admin0: Looks like it's named 'stable/mitaka' | 18:27 |
| palendae | http://git.openstack.org/cgit/openstack/openstack-ansible/log/?h=stable/mitaka | 18:28 |
| admin0 | so we will also have unstable/neuton ? | 18:28 |
| admin0 | or why the stable prefix ? | 18:28 |
| palendae | Pretty sure that's the convention across OpenStack. openstack-ansible used to have them a couple cycles back, I don't remember why they were removed | 18:29 |
| palendae | I think because we weren't being as strict as the rest of OpenStack about backports | 18:29 |
| admin0 | going to give mitaka a spin | 18:29 |
| *** javeriak has joined #openstack-ansible | 18:30 | |
| *** lihg has joined #openstack-ansible | 18:30 | |
| *** clickboom has quit IRC | 18:32 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 18:35 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 18:35 | |
| LiftedKilt | with the openstack-ansible neutron configuration, are there any known host scaling limits? | 18:36 |
| mkrish004c | thanks stevelle, i will give a try, how many container it should create if we run ceilometer alone? | 18:36 |
| LiftedKilt | I've got between 1.5-2k physical servers - do I need Calico/Opencontrail/etc or will the stability tweaks in OSA be sufficient? | 18:37 |
| *** fawadkhaliq has quit IRC | 18:38 | |
| stevelle | mkrish004c: not sure how to answer that. | 18:38 |
| *** fawadkhaliq has joined #openstack-ansible | 18:38 | |
| *** skamithi has joined #openstack-ansible | 18:38 | |
| *** admin0 has quit IRC | 18:40 | |
| *** skamithi has quit IRC | 18:42 | |
| *** skamithi has joined #openstack-ansible | 18:42 | |
| *** skamithi has quit IRC | 18:43 | |
| *** fawadkhaliq has quit IRC | 18:43 | |
| *** fawadkhaliq has joined #openstack-ansible | 18:43 | |
| *** skamithi has joined #openstack-ansible | 18:45 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server: Fix handlers https://review.openstack.org/303770 | 18:45 |
| *** yarkot_ has joined #openstack-ansible | 18:46 | |
| *** jthorne has quit IRC | 18:47 | |
| *** jthorne has joined #openstack-ansible | 18:47 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-security: Security: flake8 fixes in conf.py https://review.openstack.org/304815 | 18:50 |
| *** javeriak_ has joined #openstack-ansible | 18:52 | |
| *** fawadkhaliq has quit IRC | 18:52 | |
| *** fawadkhaliq has joined #openstack-ansible | 18:54 | |
| *** keedya has quit IRC | 18:55 | |
| *** javeriak has quit IRC | 18:56 | |
| *** admin0 has joined #openstack-ansible | 18:58 | |
| *** fawadkhaliq has quit IRC | 18:58 | |
| *** ametts has quit IRC | 18:58 | |
| *** fawadkhaliq has joined #openstack-ansible | 18:59 | |
| *** fawadkhaliq has quit IRC | 19:00 | |
| *** fawadkhaliq has joined #openstack-ansible | 19:01 | |
| *** javeriak_ has quit IRC | 19:07 | |
| *** javeriak has joined #openstack-ansible | 19:07 | |
| *** admin0 has quit IRC | 19:08 | |
| *** admin0 has joined #openstack-ansible | 19:08 | |
| *** mkrish004c has quit IRC | 19:11 | |
| *** admin0 has quit IRC | 19:13 | |
| *** admin0 has joined #openstack-ansible | 19:14 | |
| evrardjp | LiftedKilt 2k is quite different than standards deployments | 19:16 |
| palendae | The OSIC cluster is 1k nodes, pretty sure it uses OSA | 19:16 |
| evrardjp | ;) | 19:17 |
| evrardjp | hardware load balancers for the OSIC cluser? I don't remember | 19:17 |
| *** admin0 has quit IRC | 19:17 | |
| palendae | evrardjp: I think so | 19:18 |
| evrardjp | it makes sense | 19:18 |
| evrardjp | LiftedKilt for calico/contrail/standard it really depends on what you need as features | 19:18 |
| *** jmccrory_ has quit IRC | 19:19 | |
| evrardjp | imo | 19:19 |
| LiftedKilt | evrardjp: how so? | 19:19 |
| *** fawadkhaliq has quit IRC | 19:20 | |
| *** daledude has joined #openstack-ansible | 19:20 | |
| *** fawadkhaliq has joined #openstack-ansible | 19:20 | |
| LiftedKilt | lxd for hypervisor, ceph for block/object, software LBs | 19:20 |
| daledude | is it yet possible to upgrade from 12.x to the new 13.x? | 19:20 |
| evrardjp | LiftedKilt software lb would probably require fine tuning | 19:22 |
| evrardjp | a lot of fine tuning | 19:22 |
| LiftedKilt | currently we are running everything on haproxy/heartbeat on dedicated servers | 19:22 |
| evrardjp | ok | 19:23 |
| evrardjp | everything means? openstack? | 19:23 |
| *** admin0 has joined #openstack-ansible | 19:23 | |
| LiftedKilt | no - everything meaning 1500 servers running openvz containers all managed solely with a frankenstein of bash scripts to provision and manage everything | 19:24 |
| LiftedKilt | while the number of compute hosts is high, each host is only running 4-6 containers so the aggregate load is low | 19:25 |
| LiftedKilt | it's just terribly utilized | 19:25 |
| evrardjp | openstack neutron will be quite verbose compared to standard flat openvz, that's my point of view | 19:32 |
| stevelle | daledude: we don't have upgrades for that yet, I expect it to come together before Milestone 1 of Newton. | 19:32 |
| evrardjp | LiftedKilt I didn't try on that scale, but I'd give it a go | 19:33 |
| evrardjp | just to test if you can :D | 19:33 |
| LiftedKilt | yeah right now there's no traffic from openvz since they aren't connected to each other - I'm ok with deploying something like calico, I was just wondering if it was necessary | 19:33 |
| evrardjp | osic cluser works with a large amount of nodes, so why not | 19:33 |
| *** javeriak has quit IRC | 19:34 | |
| LiftedKilt | evrardjp: these nodes are also all dual gig nic only | 19:34 |
| LiftedKilt | I assume osic has 10gb nics? | 19:34 |
| evrardjp | i was afraid of large vxlan meshings of tunnels | 19:36 |
| evrardjp | but it should scale anyway | 19:36 |
| evrardjp | and you're not forced to use vxlan | 19:36 |
| palendae | LiftedKilt: I think so, but not totally sure. busterswt or jthorne would know better | 19:36 |
| LiftedKilt | it's probably worthwhile to just deploy calico then? I'm thinking I'll need to squeeze every ounce of efficiency I can get out of this network, especially at this scale | 19:37 |
| admin0 | LiftedKilt: do not use a single cluster .. break into regions .. say 400 nodes each .. should do fine | 19:38 |
| admin0 | you said 4-6 containers per host so thats around 9000 max now .. | 19:38 |
| admin0 | should do fine .. but not in a single region | 19:38 |
| LiftedKilt | admin0: can multiple regions work together seamlessly? | 19:38 |
| evrardjp | LiftedKilt admin0 has a good advice there | 19:38 |
| admin0 | seamlessly — depends on how you make it .. :) | 19:39 |
| LiftedKilt | meaning like live migration betweek regions | 19:39 |
| admin0 | nope | 19:39 |
| admin0 | but migration is possible | 19:39 |
| LiftedKilt | hmm | 19:39 |
| evrardjp | availability zones, and maybe cells are what you would be looking for | 19:39 |
| admin0 | use federated swift , glance backend to swift | 19:39 |
| logan- | when you say dual gig im more scared of running ceph than the network setup | 19:39 |
| evrardjp | I don't know cells 'though | 19:39 |
| LiftedKilt | about 750 of the servers are for jenkins | 19:39 |
| admin0 | availability zones are just logical seperations .. does not account for network issues | 19:39 |
| evrardjp | and region is better when you have large amount | 19:40 |
| admin0 | need to use regions at scale | 19:40 |
| evrardjp | precisely | 19:40 |
| LiftedKilt | logan- the plan was to use a single 1tb ssd in each node | 19:40 |
| LiftedKilt | logan-: is that a terrible idea? | 19:40 |
| admin0 | LiftedKilt: i run aound 5000 vms now on a single region, 2 avaibaility zones .. .. next cluster we are building is planned for 20k vms | 19:40 |
| admin0 | i am dumping avaibality zones, moving into regions | 19:41 |
| logan- | you are stretching the throughput pretty thin doing that without considering any non-storage traffic going over those links | 19:41 |
| admin0 | LiftedKilt: at your scale, you can use a single ceph clsuter and do live migration | 19:41 |
| LiftedKilt | admin0: so multiple openstack regions on top of a single ceph cluster? | 19:42 |
| evrardjp | but logan has a point too, ceph is usually requiring more than one gig | 19:42 |
| admin0 | LiftedKilt: we have 3 sepearte datacenters .. but you can have all, different racks per region to keep latency local .. and then if you are going to use ceph, you can pretty much do inter region migrations | 19:43 |
| admin0 | however :D | 19:43 |
| admin0 | there is a however :D | 19:43 |
| admin0 | at scale, even we see 96 gbps saturated due to ceph .. at scale, disks break .. and ceph would like to fix it at the earliest .. saturating every bit of network | 19:43 |
| LiftedKilt | right now we are using moosefs on platter drives - I imagine that ceph can't be that much more network intensive that moosefs, right? | 19:44 |
| evrardjp | once again a good advice from admin0, LiftedKilt | 19:45 |
| admin0 | ceph is awesome when things do not break .. when disks break .. there would be a saturation point when ceph is rebalncing | 19:45 |
| LiftedKilt | evrardjp: I'm going to print all of this and read it a few times to try and absorb it all haha | 19:45 |
| evrardjp | come on admin0 just stop being faster at typing than me | 19:45 |
| admin0 | :D | 19:45 |
| evrardjp | :p | 19:45 |
| logan- | how write heavy is your environment | 19:46 |
| admin0 | i am also watching a movie and making a coffee for my wife :D | 19:46 |
| LiftedKilt | admin0: we'll only be keeping 2 copies of a lot of the data, so that should help a little | 19:46 |
| evrardjp | I'm with a new keymap admin0 | 19:46 |
| evrardjp | :p | 19:46 |
| LiftedKilt | 750 of the nodes are jenkins slaves doing CI builds | 19:46 |
| logan- | i guess it will be similar to moose probably, i haven't used it but iirc its similar replicated storage so similar network requirements id expect | 19:46 |
| LiftedKilt | on which we only need one backup copy of their data | 19:47 |
| evrardjp | it's fire and forget right? | 19:47 |
| evrardjp | why should you need live migration and all these things? | 19:47 |
| evrardjp | just build cinder lvm and tada! | 19:47 |
| LiftedKilt | evrardjp: the jenkins is, yeah | 19:47 |
| admin0 | no :D | 19:47 |
| evrardjp | ;) | 19:47 |
| admin0 | no cinder lvm :D | 19:47 |
| admin0 | very bad advice | 19:47 |
| michaelgugino | our ceph guys have tuned auto-healing all the way to the lowest setting to avoid network sat | 19:48 |
| evrardjp | it was a joke | 19:48 |
| admin0 | :D | 19:48 |
| LiftedKilt | but I think they output their results to a central place for some reason or another - I really have no idea what it's doing | 19:48 |
| LiftedKilt | then the rest of the nodes are going to be containers holding liferay + mariadb | 19:48 |
| LiftedKilt | nodes will hold containers of liferay + mariadb, that is | 19:49 |
| admin0 | LiftedKilt: use regions, 400 nodes per region .. use a federated keystone and swift .. use swift as backend for glance, use ceph for cinder .. use local storage .. for people who want live migration, maybe have a different region with nova backend to ceph .. live life happy :D | 19:49 |
| admin0 | you have enough machines to have local storage and live-migration regions | 19:49 |
| admin0 | just sell as premium for those who still insist to have pets on cloud and not cattles | 19:50 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:50 | |
| admin0 | most people will want faster iops .. don’t care as long as it runs .. with multiple regions, make it their burden to do HA and ensure they are covered in event of a breakdown .. | 19:50 |
| LiftedKilt | admin0: this is all great advice | 19:52 |
| LiftedKilt | thank you guys so much | 19:53 |
| admin0 | 90% of customers will happily run 2 mysql databases on local SSDs and do replication themself between regions, 10% will need live migration and all the belss and wishles and even automatic fallback ..etc etc .. so do not do a global ceph at your scale and be sad 90% of the time .. with regions you can isolate issues, offerings etc | 19:53 |
| admin0 | bells* | 19:53 |
| LiftedKilt | admin0: oh this is an entirely private cloud | 19:54 |
| LiftedKilt | it's for our internal developers | 19:54 |
| evrardjp | you really like your developers then | 19:54 |
| evrardjp | :D | 19:54 |
| admin0 | well in terms of openstack @ scale .. issues are issues … public and private is if you bill it or not :) | 19:54 |
| LiftedKilt | admin0: fair enough | 19:55 |
| admin0 | network issues/cpe will not know its running in private cloud :D | 19:55 |
| LiftedKilt | evrardjp: I don't, but management does | 19:55 |
| *** yarkot_ has quit IRC | 19:55 | |
| LiftedKilt | evrardjp: haha | 19:55 |
| evrardjp | :D | 19:55 |
| admin0 | devs to management — “but we are nto getting enough iops .. a disk dies and the whole system crawls “ — you to manaagement “ yes, its not on local ssds.. its on awesome ceph" | 19:55 |
| evrardjp | admin0 one could say that you will bill internally, but I'm becoming picky ;) | 19:55 |
| admin0 | before this public cloud job, i worked on a fairly large private cloud ( gaming company, 120million gamers served per month ) .. | 19:56 |
| admin0 | and issues are common :_) | 19:57 |
| evrardjp | riot? | 19:57 |
| evrardjp | blizzard? | 19:57 |
| admin0 | nah .:D | 19:57 |
| admin0 | look up my linkedin :D | 19:57 |
| admin0 | https://www.linkedin.com/in/sashidahal | 19:57 |
| evrardjp | oh yeah | 19:57 |
| admin0 | LiftedKilt: what is yuor use case ? dbs ? vms ? that would matter how you need to design | 19:58 |
| admin0 | boo.. my spellings :D | 19:59 |
| falanx | admin0: dbs, logs, flat data, no vms | 20:00 |
| *** asettle has joined #openstack-ansible | 20:00 | |
| falanx | <--- works with LiftedKilt | 20:00 |
| LiftedKilt | admin0: what he said ^ | 20:00 |
| admin0 | falanx: evrardjp , evrardjp falanx | 20:00 |
| admin0 | introducing both of you to each other :D | 20:00 |
| admin0 | \o/ | 20:01 |
| admin0 | :D | 20:01 |
| evrardjp | redundancy | 20:01 |
| falanx | o/ | 20:01 |
| evrardjp | you should have that in two lines :p | 20:01 |
| jthorne | LiftedKilt: you were asking about OSIC hardware? | 20:02 |
| LiftedKilt | jthorne: we were discussing scalability of osa with out of the box networking, and I said that OSIC probably has 10gb or greater nics on the nodes | 20:03 |
| jthorne | this is true | 20:03 |
| LiftedKilt | jthorne: we're looking to build a 1.5k node openstack cluster on commmodity hardware with dual gig nics | 20:04 |
| *** jayc has quit IRC | 20:04 | |
| evrardjp | falanx nice to meet you | 20:04 |
| jthorne | LiftedKilt: so Cloud 1 is only 352 nodes. the rest of the environment is broken up into pure bare metal environments due to community requests | 20:04 |
| jthorne | LiftedKilt: this is the design of Cloud 1: http://public.thornelabs.net/osic-cloud-1-rpc-physical-connectivity-and-specs-diagram.pdf | 20:04 |
| *** keedya has joined #openstack-ansible | 20:04 | |
| *** asettle has quit IRC | 20:05 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 20:05 | |
| LiftedKilt | jthorne: Yeah we are in a whole different world - I've got "enterprise" netgear TOR switches | 20:06 |
| admin0 | :D | 20:06 |
| admin0 | netgeat .. i have one for my home lab . | 20:06 |
| admin0 | they work good actually | 20:06 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Isolate Ansible from the deployment host https://review.openstack.org/304840 | 20:06 |
| LiftedKilt | they have 10gb uplinks, which I was planning on connecting to a pair of cumulous linux 10gb spines | 20:06 |
| LiftedKilt | I have 66 netgear switches, split up into 22 stacks of three | 20:07 |
| admin0 | LiftedKilt: so when you want to design yoru new openstack .. be here and we can offer advice and help | 20:08 |
| evrardjp | I guess now it's always broadcom silicons anyway :D | 20:09 |
| spotz | gerrit hates me! | 20:10 |
| *** jayc has joined #openstack-ansible | 20:10 | |
| evrardjp | yeah, LiftedKilt, don't hesitate to come here | 20:10 |
| LiftedKilt | admin0: I started looking at openstack beginning of this year - tried fuel and juju, and now I'm here - I feel like ansible is probably the only tool that's going to give me the flexibility to make something this janky actually work | 20:10 |
| evrardjp | spotz ? | 20:10 |
| spotz | I get 503 on login since the server change | 20:10 |
| LiftedKilt | evrardjp admin0: I really appreciate all your guys' input | 20:10 |
| admin0 | !gerrit-- | 20:10 |
| openstack | admin0: Error: "gerrit--" is not a valid command. | 20:10 |
| admin0 | no karma points here :( | 20:11 |
| spotz | No reviews until tonight:( | 20:11 |
| admin0 | \o/ — party time spotz ? | 20:11 |
| spotz | hah admin0 - real work time:) | 20:11 |
| *** fawadkhaliq has quit IRC | 20:11 | |
| *** fawadkhaliq has joined #openstack-ansible | 20:12 | |
| *** Nepoc has quit IRC | 20:12 | |
| admin0 | LiftedKilt: i have tried all .. but sticking to ansible for the flexibility | 20:12 |
| admin0 | spotz: you are not in the UK ? | 20:12 |
| * admin0 thinks everyone of rackspace is in the UK | 20:12 | |
| admin0 | at least people in this channel | 20:13 |
| spotz | Nope I'm US with cloudnull, mhayden, prometheanfire, etc | 20:13 |
| admin0 | ok | 20:13 |
| *** Nepoc has joined #openstack-ansible | 20:13 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Isolate Ansible from the deployment host https://review.openstack.org/304840 | 20:14 |
| *** b3rnard0_away is now known as b3rnard0 | 20:14 | |
| admin0 | shouldn’t gerrit also be redundant deployed on the cloud :D | 20:14 |
| admin0 | we must implement what we build | 20:14 |
| admin0 | for redundancy :) | 20:14 |
| cloudnull | anyone around that can give this a review https://review.openstack.org/#/c/304385/ ? | 20:15 |
| cloudnull | also is anyone is wanted to test something new, I'd appreciate feedback on https://review.openstack.org/#/c/304840/ | 20:16 |
| cloudnull | ^ -cc automagically jmccrory admin0 -- we've talked about being able to do multiple deploys from a single host and that can help to get it going. | 20:17 |
| cloudnull | that is w/out having to much dirs/files about | 20:17 |
| cloudnull | 's/much/munge/g' | 20:18 |
| automagically | cloudnull: will take a look | 20:18 |
| cloudnull | no rush | 20:18 |
| admin0 | cloudnull: i have 25 servers ready to be fiddled with .. anything you want me to test, i can test :) | 20:18 |
| cloudnull | sweet! | 20:18 |
| cloudnull | :) | 20:18 |
| cloudnull | fiddle away :) | 20:18 |
| admin0 | i was planning to first do the mitaka install ( without the SSL ) and then run again with the SSL patch and see if it breaks stuff | 20:19 |
| admin0 | before moving into other stuff | 20:19 |
| cloudnull | cool. that'd be super useful | 20:19 |
| admin0 | i repurpose my env once per day, so i can test things at scale | 20:20 |
| cloudnull | mattt: https://review.openstack.org/#/c/296839/ -cc odyssey4me | 20:21 |
| cloudnull | 1.9.5 is busted | 20:22 |
| *** weezS has quit IRC | 20:22 | |
| cloudnull | when 1.9.6 comes out we should give that a go however I think 1.9.5 should be avoided. | 20:22 |
| eil397 | cloudnull: 304840 . I was talking about this thing " scripts/scripts-library.sh: line 202: tracepath: command not found" ? | 20:23 |
| eil397 | s/I/you/d | 20:23 |
| cloudnull | eil397: whats that ? | 20:23 |
| eil397 | cloudnull: https://review.openstack.org/#/c/304840/ failed | 20:24 |
| eil397 | gate-openstack-ansible-dsvm-commit with one of errors about tracepath not found | 20:25 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-security: Fix flake8 violation in conf.py https://review.openstack.org/304815 | 20:25 |
| jmccrory | cloudnull: cool, ansible change looks good after quick glance. only thing with deploying multiple environments would be making sure that deployment host is able to reach each container network | 20:25 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Isolate Ansible from the deployment host https://review.openstack.org/304840 | 20:27 |
| cloudnull | eil397: ^ that should fix that oversight | 20:27 |
| cloudnull | afk a min | 20:27 |
| eil397 | cloudnull: cool : - ) I've not seen all changes but it is great thing to ahve | 20:29 |
| spotz | cloudnull I'm trying to review:( | 20:30 |
| spotz | I blame mhayden | 20:30 |
| * mhayden wats | 20:31 | |
| *** pjm6 has joined #openstack-ansible | 20:31 | |
| spotz | ok even an /etc/hosts to review.openstack.org for 104.130.246.91 isn't helping me | 20:32 |
| *** yarkot_ has joined #openstack-ansible | 20:35 | |
| admin0 | https://review.openstack.org works well from here (Netherlands) .. signed out, signed in | 20:35 |
| admin0 | same ip - 104.130.246.91 | 20:36 |
| palendae | cloudnull: Is there a spec fo that? | 20:36 |
| spotz | If all else fails change VPNs:) | 20:36 |
| mhayden | i use Cat-6 cable, that helps | 20:38 |
| spotz | cloudnull got into the editor, you mind if I fix grammar while here?:) | 20:39 |
| admin0 | spotz: you are very keen on grammatical correctness :) | 20:40 |
| spotz | It's why they keep me around | 20:40 |
| admin0 | :D | 20:40 |
| admin0 | well no complaints there .. my patches are of a better quality | 20:41 |
| admin0 | due to you | 20:41 |
| * admin0 sends spotz a pizza :D | 20:41 | |
| spotz | Thanks;) | 20:41 |
| cloudnull | spotz: ++ | 20:46 |
| cloudnull | please do | 20:46 |
| cloudnull | palendae: nope | 20:47 |
| spotz | thanks cloudnull. I would do it more often but I always get in the editor by accident:) | 20:48 |
| *** weezS has joined #openstack-ansible | 20:48 | |
| openstackgerrit | Amy Marrich (spotz) proposed openstack/openstack-ansible: Isolate Ansible from the deployment host https://review.openstack.org/304840 | 20:54 |
| admin0 | cloudnull: acually i will start the mitaka install using your patch .. i see no use to do mitaka and not have those SSls in | 21:00 |
| *** asettle has joined #openstack-ansible | 21:00 | |
| *** phalmos has quit IRC | 21:02 | |
| *** Brew1 has joined #openstack-ansible | 21:02 | |
| *** Brew1 has quit IRC | 21:03 | |
| *** fawadkhaliq has quit IRC | 21:03 | |
| *** Brew1 has joined #openstack-ansible | 21:03 | |
| *** fawadkhaliq has joined #openstack-ansible | 21:03 | |
| *** Brew has quit IRC | 21:04 | |
| mattt | cloudnull: cool, i figured something there wasn't right :) | 21:04 |
| *** johnmilton has quit IRC | 21:05 | |
| *** sdake has quit IRC | 21:05 | |
| *** ametts has joined #openstack-ansible | 21:06 | |
| admin0 | cloudnull: i did a git clone -b stable/mitaka .. followed by: git fetch https://git.openstack.org/openstack/openstack-ansible/refs/changes/99/277199/21 && git cherry-pick FETCH_HEAD | 21:08 |
| admin0 | now can you give me an example gist on how to pass the SSL certs that i have | 21:08 |
| *** sdake has joined #openstack-ansible | 21:08 | |
| admin0 | so that i can start this and validate | 21:09 |
| *** rohanp_ has quit IRC | 21:10 | |
| *** skamithi has quit IRC | 21:10 | |
| *** Mudpuppy has quit IRC | 21:12 | |
| *** gregfaust has joined #openstack-ansible | 21:14 | |
| palendae | ^ such an example should probably go in docs :) | 21:14 |
| admin0 | cloudnull: for liberty, to have horizon and keystone in ssl, i had these in the variables | 21:15 |
| admin0 | https://gist.github.com/a1git/3c2b3b3faa3bd631d5c6d936f77cafa2 | 21:15 |
| admin0 | so i have the patch now .. what should go in the variables ? | 21:16 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible: Isolate Ansible from the deployment host https://review.openstack.org/304840 | 21:17 |
| cloudnull | ^ palendae updated based on your initial review . thanks for that btw :) | 21:18 |
| *** daledude has quit IRC | 21:18 | |
| cloudnull | admin0: you have a cert you want to push out ? | 21:18 |
| *** fawadkhaliq has quit IRC | 21:18 | |
| *** fawadkhaliq has joined #openstack-ansible | 21:18 | |
| *** michaelgugino has quit IRC | 21:20 | |
| *** galstrom is now known as galstrom_zzz | 21:21 | |
| *** asettle has quit IRC | 21:22 | |
| admin0 | i have a real cert | 21:25 |
| *** skamithi has joined #openstack-ansible | 21:28 | |
| *** elgertam has joined #openstack-ansible | 21:29 | |
| admin0 | cloudnull: could not find anywhere how to use this patch :D | 21:31 |
| admin0 | https://review.openstack.org/#/c/277199/21/releasenotes/notes/haproxy_ssl_terminiation-cdf0092a5bfa34b5.yaml — was hoping it would be there | 21:31 |
| admin0 | especially on how to get/set the user_variables file | 21:31 |
| mrda | Morning OSA | 21:32 |
| admin0 | morning mrda | 21:33 |
| palendae | 'lo mrda | 21:34 |
| cloudnull | ah admin0 -- http://docs.openstack.org/developer/openstack-ansible/install-guide/configure-haproxy.html?highlight=haproxy#securing-haproxy-communication-with-ssl-certificates | 21:36 |
| cloudnull | nothing there has changed | 21:36 |
| cloudnull | our haproxy role has supported ssl for some time | 21:36 |
| cloudnull | its the openstack services that did not | 21:36 |
| admin0 | so cloudnull , i should just keep using this : https://gist.github.com/a1git/3c2b3b3faa3bd631d5c6d936f77cafa2 and it should work ? | 21:36 |
| admin0 | and i can set the internal and admin also to true i guess | 21:37 |
| mrda | cloudnull: Just replied to https://review.openstack.org/#/c/301712 and change my vote. | 21:37 |
| cloudnull | admin0: you should only need https://gist.github.com/a1git/3c2b3b3faa3bd631d5c6d936f77cafa2#file-gistfile1-txt-L15-L17 | 21:37 |
| admin0 | just those 3 lines and nothing else :D ? | 21:37 |
| admin0 | \o/ | 21:37 |
| admin0 | will give it a try | 21:37 |
| cloudnull | i do believe so | 21:37 |
| cloudnull | mrda: cool | 21:37 |
| * cloudnull looking now | 21:37 | |
| mrda | cloudnull: your welcome :) | 21:38 |
| cloudnull | mrda: do you really not have the username option in the keystone_auth section? and its working? | 21:43 |
| cloudnull | keystone auth should throw an exception because it cant validate tokens | 21:43 |
| cloudnull | https://bugs.launchpad.net/ironic/+bug/1418341 | 21:43 |
| openstack | Launchpad bug 1418341 in Ironic "keystone_authtoken configuration error in ironic.conf from devstack" [Medium,In progress] - Assigned to Pavlo Shchelokovskyy (pshchelo) | 21:43 |
| mrda | cloudnull: uh-huh | 21:43 |
| cloudnull | that would seem suspect to me. | 21:44 |
| cloudnull | but if thats the case ill pull it | 21:44 |
| mrda | or, if you think it's required, and it's not hurting anything, leave it | 21:45 |
| cloudnull | I had seen this in my logs which is why i put it , WARNING ironic.conductor.manager [-] Error in deploy of node 08a45b46-f123-4f19-a10d-faff54c8342b: Could not authorize in Keystone: A username and password or token is required. , but maybe i have something else going on | 21:45 |
| cloudnull | ill pull it for now and see how it goes. | 21:46 |
| mrda | cloudnull: See Yuki Nishiwaki's comment on that bug | 21:46 |
| cloudnull | i see it however i have my logs telling me otherwise. | 21:46 |
| cloudnull | ill redeploy without it | 21:47 |
| cloudnull | and see what happens | 21:47 |
| mrda | cloudnull: See Dmitry's comment. | 21:47 |
| mrda | "So, correct ones according to http://docs.openstack.org/developer/keystonemiddleware/middlewarearchitecture.html and https://github.com/openstack/keystonemiddleware/blob/b562b04ee5db309268716e0e1b8270f30bdf1a76/keystonemiddleware/auth_token.py#L645-L661 are ones with admin_ prefix" | 21:47 |
| *** briancubed has quit IRC | 21:47 | |
| mrda | cloudnull: ^^^ | 21:47 |
| cloudnull | right but thats old | 21:48 |
| cloudnull | https://github.com/openstack/keystonemiddleware/tree/master/keystonemiddleware | 21:48 |
| cloudnull | doesnt exist in master. | 21:49 |
| cloudnull | so maybe i was ahead | 21:49 |
| cloudnull | ? | 21:49 |
| *** automagically has quit IRC | 21:49 | |
| * mrda thinks keystone_auth is a bit of a mess | 21:49 | |
| cloudnull | see https://github.com/openstack/keystonemiddleware/blob/6e58f8620ae60eb4f26984258d15a9823345c310/keystonemiddleware/tests/unit/auth_token/test_auth_token_middleware.py | 21:50 |
| cloudnull | rather https://github.com/openstack/keystonemiddleware/blob/6e58f8620ae60eb4f26984258d15a9823345c310/keystonemiddleware/tests/unit/auth_token/test_auth_token_middleware.py#L558-L563 | 21:50 |
| mrda | cloudnull: since you are more likely to be within physical reach of a keystone core, feel free it poke hiome with a wet fish, but make sure you do it ironically. | 21:50 |
| cloudnull | and https://github.com/openstack/keystonemiddleware/blob/6e58f8620ae60eb4f26984258d15a9823345c310/keystonemiddleware/tests/unit/auth_token/test_auth_token_middleware.py#L2287-L2293 | 21:50 |
| cloudnull | im not, working from home these dats | 21:51 |
| cloudnull | *days | 21:51 |
| *** asettle has joined #openstack-ansible | 21:51 | |
| mrda | cloudnull: who would be crazy enough to work from home? | 21:51 |
| *** aludwar has quit IRC | 21:52 | |
| cloudnull | hahaa | 21:53 |
| *** aludwar has joined #openstack-ansible | 21:53 | |
| mrda | ok, there's enough confusion here to leave the admin_ and non-admin version of these vars in place. | 21:53 |
| *** sdake_ has joined #openstack-ansible | 21:53 | |
| mrda | So cloudnull, I'd suggest leaving the review as is. | 21:53 |
| cloudnull | "auth_plugin: This is the plugin used for authentication, such as password and token. For example, if the auth_plugin configuration option is set to password then set username, password, project_name, project_domain_name, user_domain_name and auth_url accordingly." | 21:53 |
| cloudnull | dolphm lbragstad dstanek ^ | 21:53 |
| cloudnull | is that still true? | 21:54 |
| *** Brew1 is now known as Brew | 21:54 | |
| *** sdake has quit IRC | 21:54 | |
| cloudnull | you guys mind having a look at https://review.openstack.org/#/c/301712/21/templates/ironic.conf.j2 specifically the keystone_auth section | 21:54 |
| *** asettle has quit IRC | 21:55 | |
| *** asettle has joined #openstack-ansible | 21:55 | |
| *** Brew1 has joined #openstack-ansible | 21:57 | |
| *** Brew has quit IRC | 21:57 | |
| *** Brew1 is now known as Brew | 21:57 | |
| lbragstad | cloudnull I think that looks right | 21:57 |
| cloudnull | there's been some back and forth on if we need the admin_.* and not vars. | 21:58 |
| cloudnull | being that keystone is partially your fault i figured id ask | 21:58 |
| cloudnull | :p | 21:58 |
| *** elgertam has quit IRC | 21:59 | |
| *** woodard has quit IRC | 22:00 | |
| mrda | thanks lbragstad for the clarification! | 22:01 |
| *** KLevenstein has joined #openstack-ansible | 22:01 | |
| *** woodard has joined #openstack-ansible | 22:01 | |
| admin0 | cloudnull: this https://gist.github.com/a1git/86a05ba025680c0aa69d2d7ed6ce54bd is what I have now in my /etc/openstack_deploy/user_variables.yml .. i did a openstack-ansible haproxy-install.yml .. i checked inside the haproxy setup .. I do not see SSL anywhere :D | 22:03 |
| admin0 | so i think i am missing to enable 1/more important variable | 22:03 |
| admin0 | help please | 22:03 |
| *** sdake has joined #openstack-ansible | 22:04 | |
| openstackgerrit | Amy Marrich (spotz) proposed openstack/openstack-ansible: Isolate Ansible from the deployment host https://review.openstack.org/304840 | 22:04 |
| *** sdake_ has quit IRC | 22:06 | |
| cloudnull | admin0: yes my fault you also need `haproxy_ssl: true` | 22:07 |
| *** busterswt has quit IRC | 22:08 | |
| *** woodard has quit IRC | 22:10 | |
| admin0 | cloudnull: did that, ran the playbooks again .. except horizon, no other files even mention crt | 22:12 |
| cloudnull | all of the ssl is terminated at the hap. is that working ? | 22:13 |
| admin0 | how to check ? | 22:14 |
| admin0 | in haproxy configs, i see only horizon doing ssl | 22:14 |
| admin0 | none others are | 22:14 |
| openstackgerrit | Adam Reznechek proposed openstack/openstack-ansible-specs: PowerVM Virt Driver Support https://review.openstack.org/302941 | 22:14 |
| admin0 | updated config: https://gist.github.com/a1git/86a05ba025680c0aa69d2d7ed6ce54bd | 22:15 |
| *** fawadkhaliq has quit IRC | 22:16 | |
| cloudnull | when you rerun the haproxy play it should enable ssl termination at haproxy | 22:16 |
| *** fawadkhaliq has joined #openstack-ansible | 22:16 | |
| cloudnull | can you curl https://$VIP:5000 ? | 22:16 |
| cloudnull | so something similar ? | 22:16 |
| cloudnull | *or | 22:16 |
| admin0 | without SSL = response .. with SSL = no response | 22:17 |
| admin0 | cloudnull: https://gist.github.com/anonymous/95a7a353a5b1a337c8739694c98fc834 | 22:19 |
| admin0 | the whole haproxy ansible run | 22:19 |
| cloudnull | 404 | 22:21 |
| admin0 | cloudnull: https://gist.github.com/a1git/dc339cf45ae40ffd43b40543ff08677a — updated one with the relevant config and run | 22:21 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:23 | |
| cloudnull | admin0: can you restart hap -- ansible haproxy_hosts -m shell -a 'service haproxy restart' | 22:23 |
| cloudnull | maybe its not being triggered ? | 22:23 |
| cloudnull | idk | 22:23 |
| cloudnull | sadly i have to run in a couple of mins | 22:24 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 22:24 | |
| logan- | that play output shows haproxy_ssl false on all endpoints | 22:24 |
| admin0 | cloudnull: https://gist.github.com/a1git/6cdf0129dfa8b8553bc67f47e380d71d — that is what I get | 22:24 |
| admin0 | on the restart command | 22:24 |
| *** Brew has quit IRC | 22:25 | |
| *** sdake has quit IRC | 22:25 | |
| admin0 | should True be in caps or inside ‘ ‘ or “ “ ? | 22:25 |
| admin0 | in haproxy_ssl: true | 22:26 |
| admin0 | logan-: do you know what i need to do to fix it ? | 22:27 |
| *** spotz is now known as spotz_zzz | 22:27 | |
| logan- | i am looking at the role atm, but i think you might have to override the entire haproxy services dict like that gist I sent you a week or two back | 22:27 |
| logan- | looks like haproxy_ssl is pulled into the service config only for horizon, keystone, and nova console endpoints | 22:30 |
| logan- | but even then, for some reason your play is passing haproxy_ssl: false for those endpoints | 22:31 |
| admin0 | logan- i did the following: git clone -b stable/mitaka .. followed by: git fetch https://git.openstack.org/openstack/openstack-ansible/refs/changes/99/277199/21 && git cherry-pick FETCH_HEAD .. and thus run the playbook with that config | 22:33 |
| admin0 | also i get this strange restart message: https://gist.github.com/a1git/6cdf0129dfa8b8553bc67f47e380d71d | 22:34 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:34 | |
| logan- | yeah, it doens't look like haproxy_ssl is being picked up from the config though. try openstack-ansible -e 'haproxy_ssl=true' haproxy-install.yml | 22:35 |
| admin0 | ok | 22:35 |
| *** elgertam has joined #openstack-ansible | 22:36 | |
| *** ametts has quit IRC | 22:37 | |
| *** elgertam has quit IRC | 22:41 | |
| stevelle | I assume this is something others have seen as well on master: http://paste.openstack.org/show/6wF4OOnb3dRfUupsduGk/ would love to get a confirmation | 22:41 |
| admin0 | logan-: https://gist.github.com/a1git/cb178caea150e49e7e31ba848cd7f39e | 22:41 |
| admin0 | still the same output .. | 22:41 |
| logan- | horizon shows haproxy_ssl true hmm | 22:42 |
| logan- | oh, because that is the new default with that patch anyway. | 22:43 |
| admin0 | logan-: https://gist.github.com/a1git/cb178caea150e49e7e31ba848cd7f39e — i updated that with the first 2 lines of what cloudnull said i need to do | 22:44 |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ironic: Update ironic.conf for swift and keystone compat https://review.openstack.org/301712 | 22:44 |
| logan- | yeah | 22:44 |
| cloudnull | sorry , have a hard stop today . | 22:44 |
| cloudnull | mrda: i updated that review to match your config, well see how it goes. | 22:44 |
| cloudnull | other wise, cheers everyone | 22:45 |
| admin0 | i need to get off as well .. 12:45 AM .. need to sleep .. hopefully cloudnull or logan- or someelse else can read this https://gist.github.com/a1git/cb178caea150e49e7e31ba848cd7f39e from thier chat log and try to see where it goes wrong | 22:45 |
| openstackgerrit | Steve Lewis (stevelle) proposed openstack/openstack-ansible-os_gnocchi: WIP Initial convergence testing https://review.openstack.org/304887 | 22:45 |
| admin0 | thanks all.. see ya tomorrow | 22:46 |
| *** admin0 has quit IRC | 22:47 | |
| logan- | set keystone_service_publicuri_proto to https | 22:47 |
| logan- | https://github.com/openstack/openstack-ansible/blob/stable/mitaka/playbooks/vars/configs/haproxy_config.yml#L92 | 22:47 |
| *** galstrom_zzz is now known as galstrom | 22:51 | |
| *** b3rnard0 is now known as b3rnard0_away | 22:54 | |
| *** sanjay__u has quit IRC | 22:55 | |
| *** elgertam has joined #openstack-ansible | 22:57 | |
| *** BjoernT has quit IRC | 23:03 | |
| *** retreved has quit IRC | 23:03 | |
| *** gregfaust has quit IRC | 23:03 | |
| *** jamielennox is now known as jamielennox|away | 23:09 | |
| *** galstrom is now known as galstrom_zzz | 23:09 | |
| *** elgertam has quit IRC | 23:11 | |
| *** jamielennox|away is now known as jamielennox | 23:13 | |
| *** fawadkhaliq has quit IRC | 23:13 | |
| *** fawadkhaliq has joined #openstack-ansible | 23:14 | |
| *** elgertam has joined #openstack-ansible | 23:15 | |
| *** sdake has joined #openstack-ansible | 23:18 | |
| *** klamath has quit IRC | 23:19 | |
| *** KLevenstein has quit IRC | 23:19 | |
| *** weezS has quit IRC | 23:29 | |
| *** pjm6 has quit IRC | 23:46 | |
| *** asettle has quit IRC | 23:47 | |
| *** jayc has quit IRC | 23:48 | |
| *** sdake has quit IRC | 23:50 | |
| *** busterswt has joined #openstack-ansible | 23:51 | |
| *** eil397 has quit IRC | 23:52 | |
| *** yarkot_ has quit IRC | 23:55 | |
| *** jauyeung has joined #openstack-ansible | 23:58 | |
| *** skamithi has left #openstack-ansible | 23:58 | |
| *** elopez has quit IRC | 23:59 | |
| *** jauyeung has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!