| *** Nepoc has joined #openstack-ansible | 00:02 | |
| *** thorst_ has joined #openstack-ansible | 00:02 | |
| *** sdake_ has joined #openstack-ansible | 00:02 | |
| *** sdake has quit IRC | 00:03 | |
| *** sdake has joined #openstack-ansible | 00:05 | |
| *** sdake_ has quit IRC | 00:07 | |
| *** raddaoui has quit IRC | 00:08 | |
| *** Mudpuppy has quit IRC | 00:13 | |
| *** Mudpuppy has joined #openstack-ansible | 00:14 | |
| *** raddaoui has joined #openstack-ansible | 00:16 | |
| *** scarlisle has quit IRC | 00:19 | |
| *** fawadkhaliq has joined #openstack-ansible | 00:23 | |
| *** jguy_ has joined #openstack-ansible | 00:25 | |
| *** Nepoc has quit IRC | 00:25 | |
| *** Nepoc has joined #openstack-ansible | 00:26 | |
| *** raddaoui has quit IRC | 00:30 | |
| *** woodard has quit IRC | 00:31 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Add radvd to neutron containers https://review.openstack.org/289561 | 00:33 |
|---|---|---|
| *** fawadkhaliq has quit IRC | 00:35 | |
| openstackgerrit | Michael Davies proposed openstack/openstack-ansible-ironic: Setup tftp for all scenarios https://review.openstack.org/290206 | 00:37 |
| *** bapalm has quit IRC | 00:37 | |
| *** ShannonM has quit IRC | 00:38 | |
| *** Mudpuppy has quit IRC | 00:46 | |
| *** Nepoc has quit IRC | 00:49 | |
| *** Nepoc has joined #openstack-ansible | 00:49 | |
| *** bapalm has joined #openstack-ansible | 00:54 | |
| *** Nepoc has quit IRC | 01:05 | |
| *** Nepoc has joined #openstack-ansible | 01:05 | |
| *** johnmilton has joined #openstack-ansible | 01:14 | |
| *** izaakk has quit IRC | 01:17 | |
| *** johnmilton has quit IRC | 01:19 | |
| *** sdake has quit IRC | 01:20 | |
| *** asettle has quit IRC | 01:20 | |
| *** weezS has quit IRC | 01:21 | |
| *** shanec has quit IRC | 01:25 | |
| *** Nepoc has quit IRC | 01:30 | |
| *** asettle has joined #openstack-ansible | 01:30 | |
| *** Nepoc has joined #openstack-ansible | 01:30 | |
| *** johnmilton has joined #openstack-ansible | 01:31 | |
| *** lykinsbd has quit IRC | 01:33 | |
| *** lykinsbd has joined #openstack-ansible | 01:34 | |
| *** asettle has quit IRC | 01:35 | |
| *** cloudtrainme has joined #openstack-ansible | 01:55 | |
| *** Nepoc has quit IRC | 01:56 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-repo_server: Trim apt package list https://review.openstack.org/289510 | 01:59 |
| *** Nepoc has joined #openstack-ansible | 02:06 | |
| *** asettle has joined #openstack-ansible | 02:08 | |
| *** johnmilton has quit IRC | 02:11 | |
| *** b3rnard0 is now known as b3rnard0_away | 02:21 | |
| *** gfa_ is now known as gfa | 02:24 | |
| *** john51 has quit IRC | 02:31 | |
| *** john51 has joined #openstack-ansible | 02:33 | |
| *** sdake has joined #openstack-ansible | 02:33 | |
| *** cemmason has quit IRC | 02:39 | |
| *** cemmason has joined #openstack-ansible | 02:39 | |
| *** network__ has quit IRC | 02:40 | |
| *** networkninja has joined #openstack-ansible | 02:43 | |
| *** jguy_ has quit IRC | 02:49 | |
| *** jguy_ has joined #openstack-ansible | 02:52 | |
| *** thorst_ has quit IRC | 03:05 | |
| *** thorst_ has joined #openstack-ansible | 03:06 | |
| *** thorst_ has quit IRC | 03:14 | |
| *** shanec has joined #openstack-ansible | 03:19 | |
| *** markvoelker has joined #openstack-ansible | 03:30 | |
| *** jguy_ has quit IRC | 03:32 | |
| *** sdake has quit IRC | 03:37 | |
| *** cloudtrainme has quit IRC | 03:43 | |
| *** sdake has joined #openstack-ansible | 03:43 | |
| *** cloudtrainme has joined #openstack-ansible | 03:44 | |
| *** raddaoui has joined #openstack-ansible | 03:46 | |
| *** raddaoui has quit IRC | 03:48 | |
| *** cloudtrainme has quit IRC | 03:49 | |
| *** jamielennox is now known as jamielennox|away | 03:54 | |
| *** shanec has quit IRC | 03:57 | |
| *** shausy has joined #openstack-ansible | 04:09 | |
| *** thorst_ has joined #openstack-ansible | 04:12 | |
| *** asettle has quit IRC | 04:14 | |
| *** thorst_ has quit IRC | 04:19 | |
| *** asettle has joined #openstack-ansible | 04:22 | |
| *** lykinsbd has quit IRC | 04:34 | |
| *** asettle has quit IRC | 04:36 | |
| *** asettle has joined #openstack-ansible | 04:44 | |
| *** asettle has quit IRC | 04:48 | |
| *** skamithi13 has quit IRC | 04:56 | |
| *** skamithi13 has joined #openstack-ansible | 04:57 | |
| *** jmckind_ is now known as jmckind | 05:17 | |
| *** thorst_ has joined #openstack-ansible | 05:17 | |
| *** jmckind has quit IRC | 05:19 | |
| *** thorst_ has quit IRC | 05:23 | |
| *** sdake_ has joined #openstack-ansible | 05:24 | |
| *** skamithi14 has joined #openstack-ansible | 05:27 | |
| *** sdake has quit IRC | 05:27 | |
| *** skamithi14 has quit IRC | 05:29 | |
| *** skamithi14 has joined #openstack-ansible | 05:30 | |
| *** skamithi13 has quit IRC | 05:31 | |
| *** skamithi13 has joined #openstack-ansible | 05:34 | |
| *** skamithi13 has quit IRC | 05:36 | |
| *** skamithi13 has joined #openstack-ansible | 05:37 | |
| *** skamithi14 has quit IRC | 05:37 | |
| *** skamithi14 has joined #openstack-ansible | 05:39 | |
| *** daneyon_ has quit IRC | 05:41 | |
| *** skamithi13 has quit IRC | 05:41 | |
| *** asettle has joined #openstack-ansible | 05:43 | |
| *** cemmason has quit IRC | 05:44 | |
| *** cemmason has joined #openstack-ansible | 05:45 | |
| *** javeriak has joined #openstack-ansible | 05:45 | |
| *** javeriak has quit IRC | 05:46 | |
| *** fawadkhaliq has joined #openstack-ansible | 05:47 | |
| *** asettle has quit IRC | 05:48 | |
| *** javeriak_ has joined #openstack-ansible | 05:49 | |
| *** TheIntern has joined #openstack-ansible | 05:56 | |
| *** gfa is now known as gfa_ | 05:58 | |
| *** asettle has joined #openstack-ansible | 06:02 | |
| *** asettle has quit IRC | 06:07 | |
| *** networkninja has quit IRC | 06:07 | |
| *** javeriak has joined #openstack-ansible | 06:10 | |
| *** TheIntern has quit IRC | 06:11 | |
| *** admin0 has joined #openstack-ansible | 06:11 | |
| *** TheIntern has joined #openstack-ansible | 06:12 | |
| *** javeriak_ has quit IRC | 06:13 | |
| *** thorst_ has joined #openstack-ansible | 06:21 | |
| *** admin0 has quit IRC | 06:23 | |
| *** phiche has joined #openstack-ansible | 06:25 | |
| *** TheIntern has quit IRC | 06:26 | |
| *** thorst_ has quit IRC | 06:28 | |
| *** McMurlock1 has joined #openstack-ansible | 06:40 | |
| *** javeriak has quit IRC | 06:45 | |
| *** javeriak has joined #openstack-ansible | 06:46 | |
| *** phiche has quit IRC | 06:55 | |
| *** phiche has joined #openstack-ansible | 07:05 | |
| *** mathlin1 has joined #openstack-ansible | 07:06 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift: [DOCS] Cleaned up Swift README https://review.openstack.org/287840 | 07:14 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Fix auth_url for neutron configuration https://review.openstack.org/287411 | 07:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tempest: Removing unneeded with_items usage for clarity https://review.openstack.org/289080 | 07:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron: Rootwrap configuration should be owned by root https://review.openstack.org/289077 | 07:17 |
| openstackgerrit | Merged openstack/openstack-ansible: Adding Networking Information to Install Guide https://review.openstack.org/290145 | 07:21 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova: Fix auth_url for nova configuration https://review.openstack.org/288642 | 07:22 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh: Fix auth_url for aodh configuration https://review.openstack.org/288655 | 07:22 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer: Fix auth_url for ceilometer configuration https://review.openstack.org/288693 | 07:22 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova: Conditionally set UID/GID for nova user/group https://review.openstack.org/290110 | 07:24 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder: Enable developer mode https://review.openstack.org/289854 | 07:24 |
| *** thorst_ has joined #openstack-ansible | 07:26 | |
| *** fawadkhaliq has quit IRC | 07:28 | |
| openstackgerrit | Merged openstack/openstack-ansible-rsyslog_server: Multi-distro support for rsyslog-server role https://review.openstack.org/274290 | 07:29 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server: Removing unneeded with_items usage for clarity https://review.openstack.org/289062 | 07:33 |
| *** mathlin1 has quit IRC | 07:33 | |
| *** thorst_ has quit IRC | 07:33 | |
| *** jwagner has quit IRC | 07:35 | |
| *** eglute has quit IRC | 07:35 | |
| *** d34dh0r53 has quit IRC | 07:36 | |
| *** palendae has quit IRC | 07:36 | |
| *** sigmavirus24_awa has quit IRC | 07:36 | |
| *** odyssey4me has quit IRC | 07:36 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder: Enable functional convergence testing https://review.openstack.org/289856 | 07:36 |
| *** cloudnull has quit IRC | 07:38 | |
| *** asettle has joined #openstack-ansible | 07:40 | |
| *** b3rnard0_away has quit IRC | 07:41 | |
| *** bogeyon18 has quit IRC | 07:41 | |
| *** odyssey4me has joined #openstack-ansible | 07:41 | |
| *** asettle has quit IRC | 07:47 | |
| *** admin0 has joined #openstack-ansible | 08:00 | |
| *** phiche has quit IRC | 08:00 | |
| *** sdake_ is now known as sdake | 08:01 | |
| *** javeriak has quit IRC | 08:01 | |
| *** admin0 has quit IRC | 08:02 | |
| *** phiche has joined #openstack-ansible | 08:03 | |
| *** admin0 has joined #openstack-ansible | 08:04 | |
| *** asettle has joined #openstack-ansible | 08:05 | |
| admin0 | \o/ i managed to make 2 nice overview diagrams yesterday — one if physical, one if logical | 08:07 |
| admin0 | how to you backup/sync objects from the object store ? | 08:08 |
| admin0 | nvm | 08:12 |
| admin0 | good morning all | 08:12 |
| *** asettle has quit IRC | 08:13 | |
| bsv | admin0: mornings | 08:13 |
| *** mikelk has joined #openstack-ansible | 08:18 | |
| openstackgerrit | Merged openstack/openstack-ansible: Run Liberty high level playbooks in upgrade script https://review.openstack.org/289540 | 08:19 |
| *** mgoddard has joined #openstack-ansible | 08:22 | |
| bsv | admin0: so, did you finish that blog? ;) :D | 08:24 |
| *** mgoddard has quit IRC | 08:29 | |
| *** thorst_ has joined #openstack-ansible | 08:31 | |
| *** lykinsbd has joined #openstack-ansible | 08:31 | |
| admin0 | almost sir :) | 08:32 |
| admin0 | it will be up today :) | 08:32 |
| stevelle | odyssey4me: I might be able to put some time in on the ceilometer testing today | 08:36 |
| *** lykinsbd has quit IRC | 08:36 | |
| bsv | admin0: really looking forward to it. | 08:38 |
| *** thorst_ has quit IRC | 08:39 | |
| bsv | A simple thing like a working openstack_user_conf.yml would be nice to see. | 08:39 |
| *** asettle has joined #openstack-ansible | 08:41 | |
| admin0 | it will have it :) | 08:44 |
| admin0 | a working lxc, kickstart, vyos config, nat, dhcp :D | 08:44 |
| admin0 | a complete end-to-end setup | 08:44 |
| admin0 | with floating and public ips | 08:44 |
| admin0 | :D | 08:44 |
| openstackgerrit | Merged openstack/openstack-ansible: Restart containers on network configuration change https://review.openstack.org/289904 | 08:46 |
| openstackgerrit | Merged openstack/openstack-ansible: Restart containers on network configuration change https://review.openstack.org/289905 | 08:46 |
| openstackgerrit | Merged openstack/openstack-ansible: Fix cinder client --insecure call when adding device types https://review.openstack.org/289615 | 08:46 |
| *** openstackgerrit has quit IRC | 08:47 | |
| *** openstackgerrit has joined #openstack-ansible | 08:48 | |
| openstackgerrit | git-harry proposed openstack/openstack-ansible: Update user variables to Liberty names https://review.openstack.org/287438 | 08:49 |
| *** asettle has quit IRC | 08:49 | |
| *** mgoddard has joined #openstack-ansible | 08:51 | |
| openstackgerrit | git-harry proposed openstack/openstack-ansible: Revert "Exit early from unfinished upgrade script" https://review.openstack.org/290333 | 08:57 |
| bsv | admin0: I cant even run the basic playbooks without errors, so something weird is going on. Would be nice to see a working setup to compare against. | 09:05 |
| admin0 | bsv, it will be there today :) | 09:06 |
| bsv | Good, spend 17 hours yesterday trying to track down a single error in the playbook. | 09:11 |
| bsv | Didn't find a solution >.< | 09:11 |
| bsv | But I did learn an awfull lot about other parts of OpenStack ;) | 09:11 |
| *** jguy_ has joined #openstack-ansible | 09:13 | |
| *** bapalm has quit IRC | 09:14 | |
| *** admin0 has quit IRC | 09:23 | |
| *** d34dh0r53 has joined #openstack-ansible | 09:23 | |
| *** b3rnard0 has joined #openstack-ansible | 09:23 | |
| *** admin0 has joined #openstack-ansible | 09:24 | |
| *** jwagner has joined #openstack-ansible | 09:24 | |
| *** eglute has joined #openstack-ansible | 09:24 | |
| *** palendae has joined #openstack-ansible | 09:24 | |
| *** bogeyon18 has joined #openstack-ansible | 09:25 | |
| *** sigmavirus24_awa has joined #openstack-ansible | 09:26 | |
| *** cloudnull has joined #openstack-ansible | 09:26 | |
| *** bapalm has joined #openstack-ansible | 09:27 | |
| mattt | bsv: 17 hours? jeez, that's some persistence right there | 09:30 |
| *** electrofelix has joined #openstack-ansible | 09:34 | |
| *** thorst_ has joined #openstack-ansible | 09:36 | |
| *** gfa_ is now known as gfa | 09:43 | |
| *** thorst_ has quit IRC | 09:44 | |
| *** neilus has joined #openstack-ansible | 09:45 | |
| *** neilus has quit IRC | 09:49 | |
| *** neilus has joined #openstack-ansible | 09:51 | |
| *** cemmason has quit IRC | 09:59 | |
| *** cemmason1 has joined #openstack-ansible | 09:59 | |
| odyssey4me | stevelle excellent, that would be appreciated | 10:03 |
| odyssey4me | bsv I think we need to go back to basics here as I think you've been doing some things wonky | 10:04 |
| odyssey4me | bsv first things first - what tag are you using for the build? | 10:04 |
| odyssey4me | bsv you should be checking out 12.0.7 as it's the latest liberty tag | 10:05 |
| odyssey4me | the tags are an indicator of us setting a point at which we think the release is stable and ready for consumption | 10:06 |
| odyssey4me | the 12.x.x series is based on the OpenStack Liberty release, which is the latest stable release | 10:06 |
| odyssey4me | secondly, you asked for a working openstack_user_config - there's one in the repo that's used for the AIO: https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio | 10:07 |
| odyssey4me | openstack_release and openstack_repo_url are not settings you should be messing with at all... until you understand the consequences of doing so - start simple | 10:09 |
| *** javeriak has joined #openstack-ansible | 10:09 | |
| odyssey4me | bsv, so please pastebin your openstack_user_config and user_variables file when you get a chance and we can help you debug it | 10:10 |
| bsv | odyssey4me: sure, almost have a scratched environment ready for trying again. | 10:15 |
| *** johnmilton has joined #openstack-ansible | 10:20 | |
| *** johnmilton has quit IRC | 10:26 | |
| odyssey4me | andymccr / mattt / hughsaunders if you can assist with helping get the config right once the configs are posted, please do... | 10:28 |
| odyssey4me | pabelanger congrats on your appointment to infra-core! | 10:29 |
| mattt | yep | 10:29 |
| openstackgerrit | Hugh Saunders proposed openstack/openstack-ansible: Retry tempest cirros image creation https://review.openstack.org/290381 | 10:30 |
| openstackgerrit | Hugh Saunders proposed openstack/openstack-ansible: Retry tempest cirros image creation https://review.openstack.org/290382 | 10:32 |
| *** lykinsbd has joined #openstack-ansible | 10:32 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova: Add support for the nova_api db https://review.openstack.org/287945 | 10:33 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Add support for the nova_api db https://review.openstack.org/274932 | 10:35 |
| *** lykinsbd has quit IRC | 10:37 | |
| *** mgoddard has quit IRC | 10:37 | |
| *** johnmilton has joined #openstack-ansible | 10:38 | |
| *** johnmilton has quit IRC | 10:40 | |
| bsv | odyssey4me: configs pasted | 10:41 |
| *** thorst_ has joined #openstack-ansible | 10:42 | |
| odyssey4me | ok, first up - I see you're referencing hostvars all over the place, but I see no host entry for 'burger', for instance | 10:43 |
| bsv | odyssey4me: https://etherpad.openstack.org/p/imhavingosadnightmares | 10:44 |
| odyssey4me | bsv, ok building from the head of liberty is ok - although as mentioned before, the appropriate convention for a production environment deployment is to use a tag | 10:45 |
| bsv | can you give an example of how that would look? | 10:46 |
| odyssey4me | bsv: http://docs.openstack.org/developer/openstack-ansible/install-guide/deploymenthost.html#installing-source-and-dependencies | 10:47 |
| bsv | odyssey4me: sry, i just brainfarted... that is the config before hitting the deployment server >.< | 10:47 |
| odyssey4me | in your case you already have the clone done, so just execute 'git checkout 12.0.7' | 10:47 |
| odyssey4me | bsv, ok can you pastebin the result as what's there won't work as-is | 10:48 |
| mattt | where is a gist of the specific error you're hitting? | 10:48 |
| mattt | coming in late in the game | 10:48 |
| odyssey4me | mattt he's hitting a failure on the first pip packages to install | 10:48 |
| bsv | pad updated. | 10:49 |
| odyssey4me | yesterday we discovered that his repo servers weren't synchronising their contents - bsv did you ever get to a point where they were? | 10:49 |
| bsv | no, they never synced | 10:49 |
| *** thorst_ has quit IRC | 10:49 | |
| mattt | ok so kill a repo server for the time being | 10:49 |
| mattt | or just ensure one is up | 10:49 |
| odyssey4me | ok mattt andymccr hughsaunders the etherpad has the config | 10:50 |
| *** javeriak_ has joined #openstack-ansible | 10:52 | |
| hughsaunders | bsv did you find out why the repos weren't syncing? could you login as the nginx user from one to the other? Is the lsyncd process running on the primary repo container? | 10:52 |
| odyssey4me | bsv I see that you're using a high mtu on the storage network - I assume that your switches are set to handle the high MTU? | 10:52 |
| bsv | odyssey4me: mattt: Started this morning by wiping the entire setup, reinstalling fresh Ubuntu 14.04.4 on all servers. | 10:52 |
| bsv | Storage network and vxlan network is configured for jumboframes. | 10:53 |
| hlkv6 | the repos were probably not syncing due to network issues, had pings working strangely, containers could ping deployment host, but not each other | 10:53 |
| * hlkv6 is bsv's colleague | 10:53 | |
| bsv | :) | 10:53 |
| odyssey4me | ah | 10:54 |
| odyssey4me | ok, so the config looks fine - I think - I can't speak to the network side of the config, but I don't see anything glaringly horrible | 10:54 |
| odyssey4me | the key issue here is the repo servers not syncing | 10:54 |
| bsv | good thing :) | 10:54 |
| hlkv6 | we have done multiple ansible -i hosts.odn1 -u root -a "ping -c 1 10.50.41.4" all, green for all blades pinging deployment host, on all VLANs 401,402,403,404 | 10:54 |
| odyssey4me | now, instead of running setup-infrastructure.yml every time - let's focus the work until each step gets it right | 10:55 |
| hlkv6 | odyssey4me: currently we dont have any repo servers, since we reinstalled clean ubuntu this morning on all 6 blades | 10:55 |
| odyssey4me | have you done the ansible bootstrap? | 10:55 |
| *** javeriak has quit IRC | 10:55 | |
| odyssey4me | is the user_secrets.yml file populated | 10:56 |
| bsv | odyssey4me: yes | 10:56 |
| bsv | both :) | 10:56 |
| bsv | confirmed. | 10:56 |
| odyssey4me | can you post your user_variables.yml file as well please? | 10:56 |
| bsv | done | 10:58 |
| *** mgoddard has joined #openstack-ansible | 10:59 | |
| *** javeriak has joined #openstack-ansible | 11:01 | |
| bsv | odyssey4me: just to verify, we should use git checkout 12.0.7 and NOT git checkout liberty (for stable)? | 11:02 |
| odyssey4me | bsv yes please - use the tag | 11:03 |
| bsv | ok, updated pad | 11:03 |
| *** javeriak_ has quit IRC | 11:03 | |
| odyssey4me | while the branch is usually stable too, sometimes we hit factors which break it - the tag is a specific mark where we believe it's ready to be used for production | 11:03 |
| hlkv6 | rerun scripts/bootstrap-ansible.sh then? | 11:04 |
| odyssey4me | yep, that just ensures that you have everything at the expected levels | 11:05 |
| odyssey4me | question - do you already have a mongodb setup on 10.50.41.16 ? | 11:05 |
| hlkv6 | nope, but we have a small playbooks that does this, ceilometer-prepare.yml and aodh-prepare.yml added to bottom of etherpad | 11:06 |
| hlkv6 | can run them now, if ppl are ready | 11:07 |
| *** cemmason1 has quit IRC | 11:07 | |
| *** cemmason has joined #openstack-ansible | 11:07 | |
| openstackgerrit | Merged openstack/openstack-ansible-rabbitmq_server: Upgrade RabbitMQ Server to 3.6.1-1 https://review.openstack.org/288617 | 11:08 |
| odyssey4me | ok, go ahead and setup the MongoDB - it'll need to actually be present. | 11:09 |
| hlkv6 | Create the ceilometer database and user, and Create the aodh database and user, done | 11:10 |
| odyssey4me | the glance store on the file system is not optimal, but you're not installing swift or ceph so I guess this is your only option | 11:10 |
| hlkv6 | we have tried ceph once, it looks very nice, but huge - we ended up doing RIAK with S3 instead, another project | 11:11 |
| hlkv6 | if we can avoid this for our current PoC it would be great | 11:11 |
| odyssey4me | you should remove 'glance_notification_driver: noop' from user_variables as it will break the ceilometer metrics | 11:11 |
| odyssey4me | yeah, the glance file store should ideally be on shared storage of some sort | 11:12 |
| odyssey4me | but that's something that can be worked on later | 11:12 |
| hlkv6 | line 'glance_notification_driver: noop' - deleted | 11:12 |
| odyssey4me | ok, are the appropriate bridges setup on your hosts? do you have time sync setup on all your hosts? | 11:14 |
| odyssey4me | ie have all the bits in http://docs.openstack.org/developer/openstack-ansible/install-guide/targethosts-prepare.html been done? | 11:14 |
| hlkv6 | date command on all hosts reported same to a second, bridges setup, and pinged back and forth | 11:14 |
| odyssey4me | is your deployment host on the same mgmt network? | 11:15 |
| hlkv6 | Yes, OS is 14.04.4 - which is latest LTS but quite new, bonding modules etc. done | 11:15 |
| odyssey4me | is sshd_config on all the hosts setup to allow root access via a ssh key | 11:16 |
| hlkv6 | yes, deployment host is on 10.50.{41|42|43|44}.4 and all 6 blades can ping that IP on each VLAN | 11:16 |
| odyssey4me | if you go into /opt/openstack-ansible/playbooks and execute: ansible hosts -m ping | 11:17 |
| odyssey4me | does that work? | 11:17 |
| hlkv6 | pong from 6 bladesd | 11:18 |
| hlkv6 | amigo | success >> {samuel | success >> {china | success >> {burger | success >> {bacon | success >> {solo | success >> { | 11:19 |
| odyssey4me | ok, looking good | 11:20 |
| hlkv6 | names and layout/roles added in pad, from my local ansible hosts | 11:21 |
| odyssey4me | have you checked the host partitions - is there enough tmp space, space in the other partitions for the data? | 11:21 |
| hlkv6 | all blades have 100G SAN volume, used for root VG, default install gave /home 86G - we changed so /home is 12G, and / is 50G | 11:22 |
| *** cemmason1 has joined #openstack-ansible | 11:23 | |
| odyssey4me | you might want to actually rather have /home just be part of the root partition | 11:23 |
| *** cemmason has quit IRC | 11:23 | |
| odyssey4me | but 50G may be enough to work with for a PoC - I think it'll be fine | 11:24 |
| hlkv6 | do you want me to change it now, or just make a note? | 11:24 |
| odyssey4me | make a note | 11:25 |
| odyssey4me | effectively the containers will be create in /var/lib/lxc | 11:25 |
| odyssey4me | unless you setup a vg for them | 11:25 |
| odyssey4me | if it uses a file system, then they'll be sparse files - if it uses lvm backed containers then they will be thick | 11:26 |
| odyssey4me | then the containers bind mount into subfolders of /openstack on the hosts | 11:26 |
| odyssey4me | that's where all the crucial data (like your DB data), logs files, etc are kept | 11:26 |
| odyssey4me | so in terms of partitioning those are the things you should probably care about | 11:27 |
| odyssey4me | at least off the top of my head | 11:27 |
| odyssey4me | but for a PoC we can ignore that | 11:27 |
| hlkv6 | sorry, got interrupted | 11:27 |
| hlkv6 | samuel, the storage host has two VGs lxc and cinder-volumes as described in docs | 11:28 |
| hlkv6 | 500G and 1Tb | 11:28 |
| odyssey4me | ok, so the storage host doesn't need the lxc vg | 11:28 |
| odyssey4me | only the cinder-volumes vg | 11:28 |
| hlkv6 | ok | 11:29 |
| odyssey4me | we switched running cinder-volume for lvm-backed storage to running on the host as there were all sorts of issues hit when running it in a container | 11:29 |
| odyssey4me | note that for production I would definitely recommend against using lvm-backed storage for cinder - it's a single point of failure | 11:29 |
| *** admin0 has quit IRC | 11:30 | |
| odyssey4me | it'd be better to make use of some sort of shared storage that you can upgrade online | 11:30 |
| hlkv6 | noted | 11:30 |
| odyssey4me | otherwise you'd have to suspend any instances using it every time you want to upgrade your storage host | 11:30 |
| hlkv6 | yeah, sounds like pain | 11:30 |
| odyssey4me | so ceph, or any nfs/iscsi shared storage back-end is best | 11:31 |
| odyssey4me | netapp, emc, ibm, etc - see the openstack docs for supported drivers | 11:31 |
| *** admin0 has joined #openstack-ansible | 11:31 | |
| odyssey4me | right, we're ready to get onto http://docs.openstack.org/developer/openstack-ansible/install-guide/install-foundation.html | 11:31 |
| odyssey4me | change to /opt/openstack-ansible/playbooks | 11:32 |
| odyssey4me | then execute the setup-hosts.yml playbook | 11:32 |
| odyssey4me | using the 'openstack-ansible' (not ansible-playbook) | 11:32 |
| odyssey4me | ie execute: openstack-ansible setup-hosts.yml | 11:32 |
| hlkv6 | I will let bsv run, and I will eat here at the kbd | 11:32 |
| odyssey4me | :) | 11:33 |
| bsv | back :) | 11:34 |
| bsv | just had to read the backlog ;) | 11:34 |
| odyssey4me | hughsaunders FYI this is a blocker fixerator for the security role: https://review.openstack.org/290071 | 11:34 |
| *** skamithi13 has joined #openstack-ansible | 11:35 | |
| bsv | running setup-hosts.yml now. | 11:36 |
| *** skamithi14 has quit IRC | 11:36 | |
| *** skamithi13 has quit IRC | 11:38 | |
| *** skamithi13 has joined #openstack-ansible | 11:39 | |
| openstackgerrit | Merged openstack/openstack-ansible-security: Security: Disable AppArmor check https://review.openstack.org/290071 | 11:39 |
| mattt | odyssey4me: bleh, can no longer build keystone in IRR :P | 11:40 |
| mattt | odyssey4me: we need to use upper constraints like you said | 11:40 |
| odyssey4me | mattt yeah :/ | 11:41 |
| mattt | odyssey4me: can you just pass the upper constraints file to pip ? | 11:42 |
| bsv | pip_install | Install pip is failing, but moving on. | 11:42 |
| odyssey4me | mattt I think the developer mode would have to git clone both the keystone and the requirements repositories, then install using the keystone requirements with the upper-constraints as a constraint | 11:44 |
| *** johnmilton has joined #openstack-ansible | 11:44 | |
| odyssey4me | bsv is that the fail that's expected? the task doesn't fail | 11:44 |
| mattt | odyssey4me yeah let me try that now, the sucky thing is that we'll now have to jam a bunch of SHAs in these roles | 11:44 |
| odyssey4me | mattt lemme find an example | 11:44 |
| bsv | odyssey4me: correct | 11:44 |
| odyssey4me | mattt I think it'll likely be safe to use the branch for upper constraints, but we'll have to see | 11:45 |
| mattt | odyssey4me: maybe the best thing is to clone openstack-ansible itself and then grab the SHAs from there | 11:45 |
| odyssey4me | mattt that might be a pretty nifty way of doing it | 11:45 |
| odyssey4me | then we manage the SHA's in one place | 11:45 |
| mattt | odyssey4me: or we work off master and fix bugs as we go | 11:45 |
| odyssey4me | well, my intent is to get a proposal bot to maintain the SHA's for us - but that's going to take time to figure it out | 11:46 |
| mattt | k | 11:46 |
| mattt | let me try the openstack-ansible clone route first | 11:46 |
| bsv | odyssey4me: fall back mode for pip worked. | 11:46 |
| odyssey4me | mattt so essentially you want to clone the two repositories, then do this: https://github.com/openstack/openstack-ansible-repo_build/blob/master/tasks/repo_build_install.yml#L17-L28 | 11:47 |
| *** thorst_ has joined #openstack-ansible | 11:47 | |
| openstackgerrit | Hugh Saunders proposed openstack/openstack-ansible-repo_server: Disable slave repo servers while syncing https://review.openstack.org/279452 | 11:47 |
| odyssey4me | I expect that in the case of building keystone, you'd want to feed it the keystone requirements file, with the upper-constraints file as a constraint | 11:48 |
| odyssey4me | mattt I would suggest keeping it simple and on-point now... just have a SHA in the repo | 11:48 |
| odyssey4me | cloning openstack-ansible is quite a bit of complexity to be adding right now | 11:48 |
| odyssey4me | but it's up to you | 11:48 |
| *** neilus has quit IRC | 11:49 | |
| odyssey4me | bsv yes, so before the repo container is there the initial pip task will fail because the initial task tries to install from the repo | 11:49 |
| odyssey4me | so that's an expected failure - we should try to make that more elegant | 11:49 |
| odyssey4me | bsv it seems to be taking pretty long? | 11:51 |
| bsv | meh, it usualy takes about 5 minutes. | 11:51 |
| *** neilus has joined #openstack-ansible | 11:53 | |
| *** thorst_ has quit IRC | 11:53 | |
| bsv | Creating containers now. | 11:54 |
| odyssey4me | bsv as a side note, you may wish to implement a local apt mirror for production | 11:55 |
| odyssey4me | one that you update when you're ready to update | 11:55 |
| bsv | odyssey4me: we have that now, but did not want to use it for this PoC | 11:55 |
| odyssey4me | ok cool, that's fine | 11:56 |
| bsv | i assume you are talking about the rackspace repo? | 11:56 |
| bsv | not standard ubuntu packages. | 11:56 |
| odyssey4me | bsv the rackspace ubuntu mirror is just a mirror - nothing special about it | 11:57 |
| odyssey4me | we use it as a point of reference | 11:57 |
| odyssey4me | you could use any ubuntu archive mirror | 11:57 |
| odyssey4me | the mariadb mirror is quite important - we use the rackspace one because other mirrors are rate limited | 11:58 |
| bsv | we have a mirror for dev, staging and production | 11:58 |
| odyssey4me | that's good practise | 11:58 |
| bsv | but right now, for this test, we are not using any local mirror. | 11:58 |
| bsv | odyssey4me: good practice, and will save your ass at some point ;) | 11:59 |
| hlkv6 | usually we debmirror stuff, reprepro to a current mirror, and then rsync to staging, and sometimes rsync staging to prod mirror | 11:59 |
| hlkv6 | odyssey4me: how can we mirror the "container stuff"? or is that not needed? | 12:00 |
| bsv | odyssey4me: playbook done, no errors | 12:02 |
| odyssey4me | hlkv6 for the moment you can mirror http://rpc-repo.rackspace.com/container_images/rpc-trusty-container.tgz to a local web server - we'll be changing how the image is prepped some time soon, hopefully for the Mitaka release | 12:03 |
| odyssey4me | but it may slip | 12:03 |
| hlkv6 | ok, I wont worry now then | 12:03 |
| bsv | odyssey4me: so, next one would be haproxy? | 12:03 |
| *** cloudtrainme has joined #openstack-ansible | 12:03 | |
| odyssey4me | ok, now instead of executing the setup-infrastructure playbook, we'll execute each playbook in there in turn | 12:04 |
| odyssey4me | but yeah, haproxy first | 12:04 |
| odyssey4me | openstack-ansible haproxy-install.yml | 12:04 |
| bsv | running now. | 12:04 |
| bsv | done | 12:05 |
| odyssey4me | ok, you'll see each playbook to run in sequence here: https://github.com/openstack/openstack-ansible/blob/liberty/playbooks/setup-infrastructure.yml | 12:06 |
| odyssey4me | first memcache, then repo | 12:06 |
| odyssey4me | once the repo is done, we'll need to confirm that it's all working right | 12:06 |
| bsv | ok | 12:06 |
| bsv | running memcached-install now | 12:07 |
| bsv | done | 12:08 |
| bsv | running repo-install now | 12:08 |
| bsv | odyssey4me: just passed the expected pip-error | 12:10 |
| *** cloudtrainme has quit IRC | 12:13 | |
| *** jguy_ has quit IRC | 12:16 | |
| bsv | odyssey4me: I think Akamai is using a filter on us ;) slow fetcing repo | 12:22 |
| bsv | been doing this multiple times each day for the last 14 days :D | 12:22 |
| odyssey4me | bsv the git clones take around 5 mins (some of the repositories have a lot of history) | 12:23 |
| *** cemmason1 has quit IRC | 12:23 | |
| odyssey4me | the first repo build takes the longest though, so yeah - we're trying to find ways of making it quicker | 12:23 |
| *** toddnni_ has joined #openstack-ansible | 12:24 | |
| hughsaunders | palendae: mhayden: I would appreciate it if you could have a look at https://review.openstack.org/#/c/279452/6 | 12:25 |
| *** eric_lopez has joined #openstack-ansible | 12:25 | |
| *** MCoLo__ has joined #openstack-ansible | 12:26 | |
| *** dank__ has joined #openstack-ansible | 12:26 | |
| *** toddnni has quit IRC | 12:27 | |
| *** alextricity25_ has quit IRC | 12:27 | |
| *** dank_ has quit IRC | 12:27 | |
| *** bogeyon18 has quit IRC | 12:27 | |
| *** elo has quit IRC | 12:27 | |
| *** neillc has quit IRC | 12:27 | |
| *** MCoLo___ has quit IRC | 12:27 | |
| *** toddnni_ is now known as toddnni | 12:27 | |
| *** dank__ is now known as dank_ | 12:27 | |
| *** bapalm has quit IRC | 12:27 | |
| *** neillc has joined #openstack-ansible | 12:27 | |
| *** bapalm has joined #openstack-ansible | 12:27 | |
| *** alextricity25 has joined #openstack-ansible | 12:31 | |
| *** bogeyon18 has joined #openstack-ansible | 12:31 | |
| *** neilus has quit IRC | 12:33 | |
| tiagogomes | Hi, why is rabbitmq installed from a deb packaged instead of using apt. Can't you pin the the package to a certain version? | 12:33 |
| *** lykinsbd has joined #openstack-ansible | 12:33 | |
| tiagogomes | Same for mariadb | 12:34 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_keystone: [WIP] Use contraints file in developer mode https://review.openstack.org/290446 | 12:35 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Make the purpose of the example user_variables file clearer https://review.openstack.org/290447 | 12:35 |
| odyssey4me | tiagogomes the apt repository provided at https://www.rabbitmq.com/ is a testing repository which only ever has the latest version | 12:36 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_keystone: [WIP] Use contraints file in developer mode https://review.openstack.org/290446 | 12:37 |
| odyssey4me | so you can't pin to that version, nor can you be guaranteed to be at a certain version because it changes whenever they want it to change | 12:37 |
| odyssey4me | for mariadb and the associated packages there is a similar issue, which is why packages are downloaded from fixed pin points | 12:37 |
| *** lykinsbd has quit IRC | 12:38 | |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_keystone: [WIP] Use contraints file in developer mode https://review.openstack.org/290446 | 12:38 |
| *** neilus has joined #openstack-ansible | 12:39 | |
| *** thorst_ has joined #openstack-ansible | 12:39 | |
| *** admin0 has quit IRC | 12:44 | |
| bsv | odyssey4me: status= repo_build | Move wheels into place and ensure permissions, is taking a loooooong time | 12:45 |
| *** admin0 has joined #openstack-ansible | 12:46 | |
| odyssey4me | bsv yeah, the repo build process is very IO bound | 12:46 |
| odyssey4me | how fast is the storage access? | 12:46 |
| bsv | 8Gbps FC | 12:47 |
| odyssey4me | mattt comment in https://review.openstack.org/290446 :) | 12:47 |
| odyssey4me | bsv it may be interesting to check where the bottleneck is (CPU, RAM, Disk IO, etc) | 12:47 |
| *** cemmason has joined #openstack-ansible | 12:48 | |
| mattt | odyssey4me: yeah, saw that ... i'm not entirely sure either, that was added by cloudnull | 12:48 |
| mattt | odyssey4me: but this change does seem to allow keystone to build at least | 12:48 |
| *** retreved has joined #openstack-ansible | 12:49 | |
| odyssey4me | it seems odd to me that the requirements are fed using the --constraints parameter | 12:49 |
| bsv | odyssey4me: 2xCPU w/16 cores, 128 GB RAM, should suffice | 12:49 |
| odyssey4me | although it's probably likely that anything given in those two options is merged into a single set anyway | 12:50 |
| odyssey4me | bsv should do - I suspect that the underlying tooling probably doesn't multithread very well | 12:50 |
| odyssey4me | bsv, interestingly though - our last gate check shows this sort of time for the repo_build on a heavily constrained cloud instance: http://logs.openstack.org/15/289615/2/gate/gate-openstack-ansible-dsvm-commit/e4f8efa/console.html#_2016-03-09_07_56_54_243 | 12:52 |
| bsv | you think? :) its major overkill for the role, but we wanted to test on real hardware. | 12:52 |
| odyssey4me | bsv, that's a good spec for production I think | 12:52 |
| bsv | yeah, for compute-nodes i would agree | 12:54 |
| odyssey4me | mattt I see that it works :) it's likely that it'll also work with --requirements... and I think that makes what it's doing clearer, don't you? | 12:54 |
| bsv | repo-install done | 12:54 |
| *** skamithi14 has joined #openstack-ansible | 12:54 | |
| odyssey4me | bsv bear in mind that your control plane is running a lot of stuff, so it needs some grunt | 12:54 |
| odyssey4me | ok, now check whether the rsync copies between the repo servers are executing successfully | 12:55 |
| bsv | playbook complete, no errors | 12:55 |
| bsv | look at the pad. | 12:55 |
| bsv | unless im testing it in a wrong way, there is no sync happening. | 12:57 |
| odyssey4me | see my comparison in the etherpad | 12:57 |
| odyssey4me | interestingly - I see you have no lsync process | 12:57 |
| *** skamithi13 has quit IRC | 12:58 | |
| odyssey4me | ok, let's figure out how that happened | 12:58 |
| odyssey4me | this is the crux of the issues you've been experiencing | 12:58 |
| bsv | i saw something about aptitude and lsync flying by at some point during the previous playbooks. | 12:58 |
| bsv | not an error... but | 12:58 |
| bsv | green text ;) | 12:58 |
| odyssey4me | bsv, ok hang on a minute | 12:59 |
| odyssey4me | bsv, execute ansible repo_all[0] -m shell -a 'ps aux | grep sync' | 13:01 |
| odyssey4me | record the results of each in the etherpad | 13:01 |
| odyssey4me | execute: ansible repo_all[0] -m shell -a 'df -h' | 13:01 |
| odyssey4me | execute: ansible repo_all[0] -m shell -a 'cat /etc/lsyncd/lsyncd.conf.lua' | 13:02 |
| bsv | odyssey4me: pad updated | 13:02 |
| *** jguy_ has joined #openstack-ansible | 13:03 | |
| *** retreved has quit IRC | 13:04 | |
| odyssey4me | pastebin the result of this one: execute: ansible repo_all[0] -m shell -a 'cat /var/log/lsyncd/lsyncd.log' | 13:04 |
| *** retreved has joined #openstack-ansible | 13:04 | |
| bsv | Bingo | 13:05 |
| bsv | unexplained error | 13:05 |
| bsv | see output in pad | 13:05 |
| odyssey4me | ok, to the containers can't ssh to each other | 13:05 |
| odyssey4me | go to the host of that container, lxc-attach -n <container name> to attach to the console, then test connectivity | 13:06 |
| odyssey4me | if that's fine, try and restart the lsync service and check the log again | 13:06 |
| odyssey4me | I see thatyou already verified connectivity | 13:07 |
| bsv | yes, ill check ssh from within the container | 13:07 |
| *** markvoelker_ has joined #openstack-ansible | 13:08 | |
| bsv | odyssey4me: ssh just hangs. | 13:09 |
| bsv | running with debugging | 13:09 |
| bsv | so | 13:11 |
| odyssey4me | bsv check MTU's? | 13:11 |
| *** retreved has quit IRC | 13:11 | |
| odyssey4me | the container may be using a higher MTU than the host or something like that | 13:11 |
| bsv | MTU 1500 | 13:11 |
| odyssey4me | this is where we need Apsu :p | 13:11 |
| *** retreved has joined #openstack-ansible | 13:11 | |
| *** markvoelker has quit IRC | 13:11 | |
| bsv | odyssey4me: so, the container is using mtu 1500 | 13:14 |
| odyssey4me | I'm not quite sure what the best course of action is to narrow down the ssh connectivity issue, but that's a blocker before we can continue. | 13:14 |
| bsv | odyssey4me: the br-mgmt is also using mtu 1500 | 13:14 |
| bsv | the underlying bond (bond0.402) is mtu 1500 | 13:14 |
| mhayden | morning folks | 13:14 |
| mhayden | hughsaunders: i'll gander | 13:15 |
| hughsaunders | mhayden: thanks :) | 13:15 |
| bsv | odyssey4me: the physical nic's are also set at mtu 1500 | 13:15 |
| mhayden | hughsaunders: this idea intrigues me | 13:15 |
| bsv | What type of encapsulation does the LXC use? | 13:15 |
| odyssey4me | bsv it does not - the container plugs directly into the bridge on the host | 13:16 |
| mhayden | whoa, that's a lotta lua | 13:16 |
| odyssey4me | lol | 13:17 |
| mhayden | enough for a luau | 13:17 |
| odyssey4me | bsv so ping works between the repo containers, in both directions? | 13:18 |
| odyssey4me | maybe it's worth tcpdumping the connections to see where the data gets to? | 13:18 |
| *** weshay has joined #openstack-ansible | 13:19 | |
| bsv | odyssey4me: now ping is not working anymore. from primary to any of the two slaves. | 13:19 |
| bsv | odyssey4me: already dumping :) | 13:19 |
| *** markvoelker_ has quit IRC | 13:20 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [DO NOT MERGE] Diagnostics https://review.openstack.org/290486 | 13:28 |
| *** woodard has joined #openstack-ansible | 13:29 | |
| *** neilus1 has joined #openstack-ansible | 13:36 | |
| *** neilus has quit IRC | 13:36 | |
| *** v1k0d3n has quit IRC | 13:38 | |
| bsv | odyssey4me: mysterious ARP going on | 13:38 |
| *** markvoelker has joined #openstack-ansible | 13:39 | |
| mhayden | hughsaunders: as nolan said, a release note in there would be appreciated | 13:40 |
| *** woodard has quit IRC | 13:41 | |
| *** woodard has joined #openstack-ansible | 13:41 | |
| bsv | odyssey4me: in my notes, only networks for VXLAN and Storage should use >1500 mtu | 13:42 |
| bsv | can you confirm that the container-network only use default (mtu 1500) settings? | 13:42 |
| odyssey4me | bsv, yes I think so | 13:43 |
| mattt | odyssey4me: does the cross role dependency stuff work yet ? | 13:44 |
| odyssey4me | mattt nope | 13:44 |
| odyssey4me | that's next on my hit list | 13:44 |
| bsv | so, status. If you generate IP traffic from the repo-container towards the repo_slave, the primary repo_container will drop its arp entry for that slave. | 13:45 |
| hlkv6 | user_variables.yml: apply_security_hardening: true <- does this have any influence on this? | 13:45 |
| odyssey4me | hlkv6 nope, that playbook hasn't run yet | 13:46 |
| hlkv6 | ok | 13:46 |
| odyssey4me | in fact, that playbook doesn't exist in the liberty branch :p | 13:46 |
| odyssey4me | speaking of which, mhayden I see that only run-playbooks executes that playbook? it's not included in any of the setup- playbooks at all | 13:46 |
| mhayden | odyssey4me: should we include it automatically? | 13:47 |
| odyssey4me | mhayden I'm thinking for mitaka we should include it automatically in one of the meta playbooks - it's controlled by a setting anywa | 13:47 |
| odyssey4me | mhayden also, I think we should be ebaling it by default | 13:48 |
| odyssey4me | *enabling | 13:48 |
| bsv | odyssey4me: okay, just discovered something weird. | 13:48 |
| odyssey4me | bsv that's odd, but this stuff is out of my depth | 13:48 |
| mhayden | odyssey4me: i'm not opposed to either -- want me to create a bug or do you want to? | 13:49 |
| bsv | the above scenario with dropping ARP, only applies if I try to communicate with another LXC host. | 13:49 |
| odyssey4me | I'm not sure if mhayden has the skills to help, but I'm pretty sure that Apsu does. | 13:49 |
| odyssey4me | mhayden no need for a bug, just put in the review :) | 13:49 |
| mhayden | haha, i am reviewing legal addendums to contracts right now -- i'm sure i'll forget :P | 13:49 |
| odyssey4me | mhayden also, see the change in https://review.openstack.org/290447 - the default should be set in group_vars, with a commented note on how to change the default in user_variables | 13:50 |
| *** keedya has joined #openstack-ansible | 13:50 | |
| *** javeriak has quit IRC | 13:50 | |
| odyssey4me | that patch clearly isn't working, so I'm going to need to work through whatever issues arise | 13:50 |
| bsv | odyssey4me: Status: sending IP-traffic between containers will cause the initiating container to drop its arp entry for the target container. | 13:51 |
| bsv | any changes in LXC for the 14.04.4 release? mm | 13:51 |
| odyssey4me | bsv that is very strange behaviour, and not behaviour we're seeing anywhere | 13:53 |
| bsv | Maybe some arp-spoofing feature... | 13:54 |
| bsv | arp-filtering | 13:55 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Docs/Reno: Make default_schedule_zone configurable https://review.openstack.org/290161 | 13:57 |
| odyssey4me | afk - meeting | 13:58 |
| mhayden | odyssey4me: ah, good idea on the user_variables.yml | 13:58 |
| *** markvoelker has quit IRC | 13:58 | |
| *** automagically_ has joined #openstack-ansible | 13:59 | |
| *** automagically_ has left #openstack-ansible | 14:02 | |
| automagically | morning all | 14:03 |
| bsv | o/ | 14:04 |
| mhayden | odyssey4me: would it make sense for the security hardening to happen as part of the setup-hosts.yml playbook? | 14:15 |
| *** automagically_ has joined #openstack-ansible | 14:17 | |
| git-harry | odyssey4me: d34dh0r53 Please review https://review.openstack.org/#/c/287438/ the only change since your +2/+W was to fix a linting failure. | 14:23 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Include security role in setup-hosts.yml https://review.openstack.org/290526 | 14:25 |
| *** b3rnard0 is now known as b3rnard0_away | 14:26 | |
| mhayden | odyssey4me: i'm not sure i can enable OSAS by default until we get rid of the "fail" tasks in there | 14:26 |
| *** b3rnard0_away is now known as b3rnard0 | 14:26 | |
| *** skamithi14 has quit IRC | 14:26 | |
| mhayden | i'd hate to derail a deployment because the base OS isn't compliant | 14:26 |
| *** skamithi13 has joined #openstack-ansible | 14:27 | |
| odyssey4me | mhayden I guess so? | 14:27 |
| mhayden | hah | 14:27 |
| mhayden | what's your take? | 14:27 |
| *** asettle has joined #openstack-ansible | 14:30 | |
| mhayden | if the environment isn't configured properly, we need some way to let the user know that | 14:32 |
| mhayden | but i don't really want to derail the playbook | 14:32 |
| mhayden | with a hard failure | 14:32 |
| mhayden | i think we had an email thread on this | 14:33 |
| * mhayden searches | 14:33 | |
| mhayden | http://lists.openstack.org/pipermail/openstack-dev/2016-January/083988.html | 14:34 |
| *** asettle has quit IRC | 14:35 | |
| *** KLevenstein has joined #openstack-ansible | 14:36 | |
| *** jthorne has joined #openstack-ansible | 14:36 | |
| bsv | odyssey4me: pick a bug https://bugs.launchpad.net/ubuntu?field.searchtext=arp&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_package= | 14:37 |
| *** TheIntern has joined #openstack-ansible | 14:37 | |
| bsv | it seems that Ubuntu have very serious issues with ARP and bridges/bonding interface. | 14:37 |
| *** scarlisle has joined #openstack-ansible | 14:39 | |
| odyssey4me | bsv, the thing is that RAX deploys on bonded interfaces, I think, and it works just fine | 14:41 |
| odyssey4me | also I think logan- also has this working | 14:42 |
| logan- | yes br-mgmt on my controllers is attached to a bond | 14:42 |
| *** automagically_ has left #openstack-ansible | 14:42 | |
| bsv | I'll try to "draw" our network to the host on the pad. | 14:43 |
| odyssey4me | bsv perhaps sharing the content of the network interfaces file from the hosts and the repo containers would help? | 14:44 |
| *** markvoelker has joined #openstack-ansible | 14:48 | |
| *** cemmason has quit IRC | 14:48 | |
| logan- | yeah a bunch of pastes with like brctl show, interface files from host/container, ip route and ip addr output, cat /proc/net/bonding/bond*, etc would help to look thru | 14:49 |
| *** sdake_ has joined #openstack-ansible | 14:50 | |
| *** ShannonM has joined #openstack-ansible | 14:52 | |
| *** sdake has quit IRC | 14:53 | |
| bsv | logan-: the weird thing is, the "bug" seems to be triggered if a container tries to communicate with IP to another container (on another host). | 14:53 |
| *** admin0 has quit IRC | 14:54 | |
| *** cloudtrainme has joined #openstack-ansible | 14:54 | |
| bsv | logan-: it will drop the arp-entry of the remote container | 14:54 |
| logan- | can you ping between those 2 repo containers at all? or is it only happening when you start generating traffic ie rsync/ssh | 14:54 |
| bsv | Ping works (once the arp-tables is populated correct, it will fix itself after a while), then you can ping all you want... | 14:55 |
| *** admin0 has joined #openstack-ansible | 14:56 | |
| bsv | the moment you send IP traffic from the container, it will drop the arp-entry of the destination container (this is only a container-to-container issue). | 14:56 |
| bsv | Then the container will send arp who-has request, the remote container see it, reply with a correct arp-reply. | 14:57 |
| bsv | The arp-reply will leave the remote host, travel over the physical network, into the server of the container who sent the arp who-has request. | 14:58 |
| *** sdake has joined #openstack-ansible | 14:58 | |
| bsv | tcpdump confirms the arp-reply on the bond-interface, bond0.402 interface, br-mgmt bridge.. and then.. pufff gone. | 14:59 |
| *** cloudtrainme has quit IRC | 14:59 | |
| bsv | the arp-reply never enters the tap-interface of the container. | 14:59 |
| logan- | interesting | 14:59 |
| *** sdake_ has quit IRC | 15:00 | |
| bsv | From what I see, the only change to interfaces we have done, is putting the host-management on a tagged vlan. | 15:01 |
| bsv | But this is happening over the container-network. | 15:01 |
| logan- | i wonder if there is some sysctl setting on the host causing it to filter that arp for some reason | 15:01 |
| *** cloudtrainme has joined #openstack-ansible | 15:01 | |
| bsv | logan-: precise... | 15:01 |
| bsv | But then everyone else using Ubuntu would have the same problem. | 15:02 |
| bsv | imho | 15:02 |
| *** cemmason has joined #openstack-ansible | 15:03 | |
| odyssey4me | logan- FYI https://etherpad.openstack.org/p/imhavingosadnightmares | 15:03 |
| *** kencjohnston has joined #openstack-ansible | 15:03 | |
| bsv | odyssey4me: we are going to remove the bond from the equation. | 15:04 |
| *** xek_ is now known as xek | 15:05 | |
| odyssey4me | bsv wait a minute, perhaps logan- can spot a problem? | 15:05 |
| logan- | thanks odyssey4me. bsv: not necessarily.. I can't speak for everyone's setup but my deploy is different in that I am using lacp as bonding proto and no dot1q tags.. I may be an edge case on that. I ran across this right now https://serverfault.com/questions/518254/linux-container-bridge-filters-arp-reply so I wonder if it is something specific to the tagging | 15:05 |
| logan- | i don't see anythign specifically wrong with that config though | 15:05 |
| logan- | everything seems mapped correctly, it just seems like some kernel setting (ie netfilter) is filtering the arps. removing the bonds to simplify things for now isnt a bad idea | 15:06 |
| *** sigmavirus24_awa is now known as sigmavirus24 | 15:09 | |
| bsv | lets just make it work, then tweak it later ;) | 15:09 |
| bsv | That would make the nightmares go away, I think ;) | 15:10 |
| *** Mudpuppy has joined #openstack-ansible | 15:13 | |
| bsv | What OS do you use logan- ? | 15:14 |
| Apsu | Morning. | 15:14 |
| Apsu | This looks fun | 15:14 |
| bsv | >.< | 15:14 |
| Apsu | bsv: So, which neighbor table are you looking in that you see an entry disappearing? | 15:16 |
| logan- | bsv: 14.04.4 | 15:17 |
| bsv | Apsu: im executing a arp -an on the container itself (lxc-attach) | 15:17 |
| *** networkninja has joined #openstack-ansible | 15:19 | |
| bsv | I have to run now, workday ended 20 mins ago, but I will be back again in 3-4 hours. Do you expect to be online at that time Apsu? | 15:19 |
| bsv | logan-: okay, all in sync then ;) | 15:19 |
| Apsu | bsv: Would strongly recommend using "ip neigh" instead, but I'd like to see the "before" table, when you can ping the other container, and the "after" when you send IP traffic. | 15:20 |
| Apsu | And also "ip neigh" on the host. Filtered for the container IPs in question | 15:20 |
| Apsu | But yeah I'll be here | 15:20 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_keystone: [WIP] Use contraints file in developer mode https://review.openstack.org/290446 | 15:20 |
| andrei_ | hi, can someone explain conditions at https://github.com/openstack/openstack-ansible/blob/eca0f5ddfba5ac12fc0199d854b3f18504f9ccd3/playbooks/roles/ceph_client/tasks/ceph_auth.yml#L20-L40 ? If one has cinder-backup and cinder-volume in one container keyring for cinder-backup was not copied. Is it a bug or a feature? | 15:21 |
| cloudnull | morning | 15:21 |
| bsv | cloudnull o/ | 15:22 |
| Apsu | bsv: I suspect you have duplicate MACs or IPs somewhere, and are getting bridge port MAC mappings swapping around | 15:22 |
| Apsu | brctl showmacs $bridge can help track that kind of issue down | 15:22 |
| Apsu | When the port # for a MAC switches | 15:23 |
| mattt | andrei_: sec | 15:23 |
| Apsu | Sometimes after container creating/destroying or even stopping/starting, on particular versions of linux/lxc, you can get some dangling veths leftover with the same MACs | 15:23 |
| Apsu | I made a script to check for that | 15:23 |
| hlkv6 | Apsu: more like, broadcast packets getting repeated/looped around and then getting that swap around? | 15:24 |
| odyssey4me | Apsu this issue is on a fresh install | 15:24 |
| *** spotz_zzz is now known as spotz | 15:24 | |
| logan- | Apsu: I don't know much about the non-lacp bonding options in linux but maybe the active-backup mode he is using created a bridging loop | 15:24 |
| Apsu | hlkv6: logan-: Could easily be something like that, yes. Bonding/bridging/VEPA/reflection issues causing MAC->port mappings to move. | 15:25 |
| logan- | yep | 15:25 |
| logan- | will be interesting to see if anything changes when the bonds are removed | 15:25 |
| Apsu | Linux bridges are actually pretty serious switches, in more ways than are immediately obvious. | 15:25 |
| Apsu | Gotta dig in there deep :P | 15:25 |
| spotz | o/ | 15:26 |
| *** phalmos has joined #openstack-ansible | 15:26 | |
| mattt | andrei_: this was all deliberate, i vaguely remember writing it :) trying to recall the details | 15:27 |
| cloudnull | bsv: did you make it work ? | 15:28 |
| cloudnull | RE: the repo bits? | 15:28 |
| hlkv6 | cloudnull: he probably left work now, I left before him, and in train now | 15:29 |
| hlkv6 | blades are reboted | 15:29 |
| cloudnull | ah. | 15:29 |
| hlkv6 | currently testing with ansible -i hosts.odn1 -u root all -a "ping -c 1 10.50.41.4" | 15:30 |
| hlkv6 | on all VLANs | 15:30 |
| odyssey4me | cloudnull my brain has stopped working and I need an idea | 15:31 |
| odyssey4me | I want to read a list of files from the deployment host, then put together a dict based on that | 15:31 |
| odyssey4me | thoughts on how best to do it? | 15:31 |
| cloudnull | using python? | 15:32 |
| cloudnull | also a list of files? | 15:32 |
| *** skamithi13 has quit IRC | 15:32 | |
| odyssey4me | cloudnull I'd rather not implement another filter, so preferably ansible | 15:32 |
| cloudnull | ok | 15:32 |
| hlkv6 | all blades can ping deployment host .4 and .20 future compute node on all 4 VLANs without bond | 15:32 |
| palendae | hughsaunders: Will take a look at that today hopefully | 15:32 |
| cloudnull | odyssey4me: whats the key in the dict | 15:33 |
| odyssey4me | but basically I'd like to grab a list of files in a directory, then create a list of dicts which uses the file names | 15:33 |
| hlkv6 | all lxc seem to be STOPPED? just wait or start manually? | 15:33 |
| hlkv6 | server been up for 10min | 15:33 |
| *** phalmos has quit IRC | 15:33 | |
| *** raddaoui has joined #openstack-ansible | 15:33 | |
| cloudnull | hlkv6: the lxc container start is slow, you can manually start them if you like. | 15:33 |
| *** mgoddard has quit IRC | 15:33 | |
| odyssey4me | cloudnull a simple list of dicts with: { src: /foo/path/file1, dest: /foo/path/file1 } | 15:33 |
| cloudnull | ah. | 15:34 |
| *** mgoddard has joined #openstack-ansible | 15:34 | |
| cloudnull | odyssey4me: so you can use a lookup with a * | 15:34 |
| cloudnull | is foo/path always known in this case ? | 15:34 |
| cloudnull | well... thatt wont work because you cant add to a dict in an ansible task . | 15:35 |
| odyssey4me | cloudnull yes, it's a fixed path | 15:35 |
| cloudnull | you can do a command call to: ls -1 | 15:36 |
| cloudnull | register that | 15:36 |
| odyssey4me | and the source and dest values will be the same | 15:36 |
| odyssey4me | so yeah, that's what I've got so far | 15:36 |
| odyssey4me | then the stdout needs to be split somehow into a list | 15:36 |
| cloudnull | and if there is more than one dir you can use the union filter to merge the lists. | 15:36 |
| cloudnull | use stdout_lines | 15:37 |
| cloudnull | register: things.stdout_lines | 15:37 |
| cloudnull | ls -1 will put all files on a newline | 15:37 |
| cloudnull | which will work if the file has a \s in it | 15:37 |
| odyssey4me | you know, I thought of that but it wasn't showing in a verbose output | 15:38 |
| odyssey4me | silly hidden feature | 15:38 |
| cloudnull | yup... | 15:38 |
| odyssey4me | ok, then I expect that this can be converted into a list of dicts using something like the map filter? | 15:38 |
| odyssey4me | the trick is that I want to set one fact as a result | 15:39 |
| hlkv6 | ok, it seems lxcbr0 is not running, when I try to start an LXC, ifup lxcbr0 does not start it | 15:39 |
| odyssey4me | ie one var, which is a list of dicts | 15:39 |
| cloudnull | hlkv6: can you ifup lxcbr0 ? | 15:39 |
| hlkv6 | Ignoring unknown interface lxcbr0=lxcbr0. | 15:40 |
| cloudnull | it should have an interface file in the /etc/network/interfaces.d | 15:40 |
| hlkv6 | there is a config in /etc/network/interfaces.d/lxc-net-bridge.cfg | 15:40 |
| cloudnull | hlkv6: do you have https://github.com/openstack/openstack-ansible/blob/master/etc/network/interfaces#L7 in the main interfaces file? | 15:40 |
| cloudnull | odyssey4me: i've had mixed success with the map filter. | 15:41 |
| cloudnull | but that should work | 15:41 |
| hlkv6 | cloudnull: nope, will add | 15:41 |
| *** sdake_ has joined #openstack-ansible | 15:41 | |
| *** sdake has quit IRC | 15:41 | |
| mattt | andrei_: that should copy it, if i'm reading correctly -- do you have cinder-backup fully enabled ? | 15:44 |
| hlkv6 | cloudnull: TY, now ifup lxcbr0 works | 15:45 |
| *** weezS has joined #openstack-ansible | 15:46 | |
| *** raddaoui has quit IRC | 15:49 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_heat: [WIP] Enable functional testing https://review.openstack.org/289454 | 15:49 |
| hlkv6 | cloudnull: and I can see lxc being started now | 15:51 |
| cloudnull | nice! | 15:51 |
| *** raddaoui has joined #openstack-ansible | 15:52 | |
| *** phalmos has joined #openstack-ansible | 15:52 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible: Docs/Reno: Make default_schedule_zone configurable https://review.openstack.org/290161 | 15:52 |
| palendae | cloudnull: RE you comment on https://review.openstack.org/#/c/289540/ about the ignore cluster state - git-harry tested without it and it worked. I originally had it in there, but can easily be added back | 15:52 |
| cloudnull | i wasnt sure if it was needed. | 15:52 |
| cloudnull | i just thought id mention it | 15:52 |
| palendae | Yeah | 15:52 |
| palendae | I was copying it from your master work | 15:53 |
| palendae | I don't know that it hurts to be there | 15:53 |
| palendae | But this next week should be about testing that stuff | 15:53 |
| *** raddaoui has quit IRC | 15:54 | |
| *** michaelgugino has joined #openstack-ansible | 15:55 | |
| michaelgugino | hello all. Thanks for merging that rsyslog commit, and thanks for the kind words. | 15:55 |
| automagically | o/ | 15:56 |
| cloudnull | hi michaelgugino | 15:57 |
| *** cemmason has quit IRC | 15:57 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-rabbitmq_server: Removing unused default role variable https://review.openstack.org/290612 | 15:58 |
| *** ametts has joined #openstack-ansible | 16:01 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-rabbitmq_server: Removing unneeded with_items usage for clarity https://review.openstack.org/290615 | 16:01 |
| *** raddaoui has joined #openstack-ansible | 16:03 | |
| *** thorst_ is now known as thorst_afk | 16:03 | |
| automagically | This is the last of what I believe was 4 patchs to various roles that all exhibited the same security issue: https://review.openstack.org/#/c/289079/ Any opinions on whether this is relevant to backport into Liberty? | 16:05 |
| openstackgerrit | Hugh Saunders proposed openstack/openstack-ansible-repo_server: Disable slave repo servers while syncing https://review.openstack.org/279452 | 16:05 |
| cloudnull | automagically: idk that we should backport it. | 16:07 |
| cloudnull | its a good sec improvement. | 16:07 |
| cloudnull | i can go both ways on that. | 16:07 |
| automagically | I’ve been reading through the guidelines for stable branches and it seems like their is a definite bias against backporting anything | 16:08 |
| palendae | automagically: Yep. | 16:08 |
| palendae | OSA's pretty liberal | 16:08 |
| automagically | Which TBH, worries me only because I’m going to be using OSA to install Liberty and yet I’m going to want a bunch of work that is currently only available in master | 16:09 |
| automagically | The SSL termination stuff for example | 16:09 |
| hlkv6 | Apsu: without bond I see icmp pings, but was not immediately and did loose some packets on testing | 16:09 |
| palendae | automagically: That's been our ongoing problem internally :) | 16:09 |
| cloudnull | automagically: +1 | 16:09 |
| palendae | People want the new hotness, but don't want to upgrade the major version | 16:10 |
| cloudnull | im on both sides of that argument, so lots of arguing with myself. | 16:10 |
| Apsu | cloudnull: At least you always win when you argue with yourself | 16:10 |
| automagically | cloudnull: I can definitely appreciate both sides. Stability _is_ king afterall | 16:10 |
| cloudnull | Apsu: this is not always true :P_ | 16:10 |
| automagically | Would just like to see us reach a point where we can more comfortably guarantee that backported changes are not impacting stability | 16:10 |
| Apsu | cloudnull: Well sometimes it's a war of attrition. Nobody wins ;P | 16:11 |
| *** andrei__ has joined #openstack-ansible | 16:11 | |
| michaelgugino | stability and new features at the same time? That's what the whole world wants. If you can figure out a way, you'll be the first ;) | 16:12 |
| cloudnull | hahaha, i have this bridge... | 16:12 |
| automagically | michaelgugino: Well, to some extent, its about strict definitions of compatibility | 16:13 |
| automagically | mhayden is running into this with the proposed backport of the security role | 16:13 |
| automagically | Making the feature available, but feature flagging it in such a way that backward compatibility is maintained by default | 16:14 |
| palendae | Yep | 16:14 |
| *** sdake has joined #openstack-ansible | 16:14 | |
| cloudnull | automagically: I'm pro-backport in most cases, which is one reason that the branches dont have "stable/" in-front of them. however thats not something shared by most. | 16:14 |
| cloudnull | but changes can cause instability | 16:15 |
| michaelgugino | I think part of the issue is trying to tightly couple to an upstream version | 16:15 |
| *** cemmason has joined #openstack-ansible | 16:15 | |
| *** sdake_ has quit IRC | 16:15 | |
| spotz | Who can answer a cinder question for documentation? Need to know if you are required to configure the cinder api or just the cinder volume so I can figure out wording | 16:16 |
| cloudnull | michaelgugino: i've pondered running Master Ansible code w/ liberty SHAs | 16:16 |
| *** cemmason has quit IRC | 16:16 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_heat: Enable functional testing https://review.openstack.org/289454 | 16:16 |
| michaelgugino | Say, OSA v1 supports Kilo, Liberty, Mitaka with a certain feature set. OSA v2 supports Liberty, Mitaka with a certain feature set. OSA v3 supports a different feature set, etc | 16:16 |
| cloudnull | in most cases it should work | 16:16 |
| automagically | Hmm…I like that notion | 16:16 |
| *** shausy has quit IRC | 16:16 | |
| michaelgugino | but, that sounds like a lot more work ;) | 16:17 |
| *** sdake_ has joined #openstack-ansible | 16:17 | |
| *** b3rnard0 is now known as b3rnard0_away | 16:18 | |
| *** asettle has joined #openstack-ansible | 16:19 | |
| automagically | Reviews on https://review.openstack.org/#/c/289454 very much welcome. Last gate check succeeded, so I’m no longer considering it WIP. Link to the check log if you don’t want to wait ~20 minutes: http://logs.openstack.org/54/289454/6/check/gate-openstack-ansible-os_heat-ansible-func-ubuntu-trusty/777aeef/ | 16:19 |
| *** sdake has quit IRC | 16:19 | |
| *** sdake_ is now known as sdake | 16:21 | |
| spotz | automagically just looked, now you need to answer my cinder api question:) | 16:22 |
| automagically | spotz: I would if I could | 16:22 |
| spotz | heheh | 16:22 |
| *** asettle has quit IRC | 16:23 | |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Gate: Use OpenStack-CI apt mirrors https://review.openstack.org/290486 | 16:25 |
| spotz | example yml says API required, I wonder why it's missing out of the doc totally | 16:25 |
| automagically | Question about the OSA main repo checks/gate…is there any way for us to not run the full AIO build piece when the only modified files in a patch are rst files? We have been churning through a bunch of doc updates lately and we are hammering infra building AIOs when the outcome is completely irrelevant for an RST only commit | 16:26 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_keystone: [WIP] Use contraints file in developer mode https://review.openstack.org/290446 | 16:27 |
| *** javeriak has joined #openstack-ansible | 16:28 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova: Rootwrap configuration should be owned by root https://review.openstack.org/289079 | 16:28 |
| *** javeriak has quit IRC | 16:29 | |
| *** javeriak has joined #openstack-ansible | 16:29 | |
| mattt | automagically: i thought docs changes didn't trigger a full test | 16:29 |
| odyssey4me | automagically hmm, .rst files should not trigger a full gate job | 16:30 |
| odyssey4me | can you share an example? | 16:30 |
| odyssey4me | they may have busted up our rules again | 16:30 |
| automagically | Let me double-check, thought I saw it happening yesterday | 16:30 |
| *** skamithi13 has joined #openstack-ansible | 16:30 | |
| automagically | Ah, this was it: https://review.openstack.org/#/c/290075/ and its a .example file | 16:31 |
| odyssey4me | yep, that will trigger a full built | 16:31 |
| automagically | bummer | 16:31 |
| openstackgerrit | Ala Raddaoui proposed openstack/openstack-ansible: Test if dynamic inventory catches some bad configs https://review.openstack.org/290011 | 16:32 |
| odyssey4me | so be it - those are few and far between and we should be rid of them :p | 16:32 |
| odyssey4me | on backports, if we can improve our gate checks to the degree that we have more confidence that they're testing many more code paths... then we can confidently allow backporting | 16:33 |
| odyssey4me | right now we're not testing enough, so we need to err on the side of caution | 16:33 |
| *** cloudtrainme has quit IRC | 16:34 | |
| *** lykinsbd has joined #openstack-ansible | 16:35 | |
| odyssey4me | Also, I'd rather drive people forward. | 16:35 |
| palendae | raddaoui: https://review.openstack.org/#/c/290011/1..2/tests/test_inventory.py appears to have removed all your code; I left a comment on there, but checking to see if that was intentional | 16:36 |
| *** javeriak has quit IRC | 16:36 | |
| odyssey4me | Another things is that backporting often creates a pretty severe review backlog. It would be nice if we could put together some sort of stable-maint team that focuses on testing and approving backports so that the core team can focus on driving improvement in the master branch instead of backport testing/reviews. | 16:36 |
| *** galstrom_zzz is now known as galstrom | 16:39 | |
| *** lykinsbd has quit IRC | 16:40 | |
| odyssey4me | something else that would also be really good is if the major stakeholders of the project implement external CI to provide feedback on patches - the external CI should test their specific use-cases to verify that the patch works for their needs... if that was in place for the major stakeholders, then I'd be far happier to support proactive backporting | 16:41 |
| automagically | Nice idea that ^ | 16:41 |
| automagically | I’d have to think through what that could look like for my org | 16:41 |
| *** javeriak has joined #openstack-ansible | 16:41 | |
| palendae | I've been thinking about that for rackspace | 16:42 |
| palendae | Upgrades in particular could benefit from an external CI that has a longer time out | 16:42 |
| -cloudnull- these patches are still in "merge conflict" after the great IRR migration. If you have a patch that needs to be updated after the migration please update it so that itll be considered / reviewed. If you need help updating the patch please let us know as many of us would be happy to lend a hand to resolve the issues: https://review.openstack.org/#/q/project:%255Eopenstack/openstack-ansible.*+-is:mergeable+is:open,n,z | 16:44 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_client: Implement human readable logging for functional tests https://review.openstack.org/289335 | 16:44 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_keystone: Use contraints file in developer mode https://review.openstack.org/290446 | 16:44 |
| openstackgerrit | Merged openstack/openstack-ansible-galera_client: [DOCS] Cleanup the role docs for consistency and clarity https://review.openstack.org/289044 | 16:44 |
| openstackgerrit | Merged openstack/openstack-ansible-plugins: [DOCS] Cleanup the role docs for consistency and clarity https://review.openstack.org/289598 | 16:44 |
| *** admin0 has quit IRC | 16:45 | |
| *** mikelk has quit IRC | 16:45 | |
| *** cloudtrainme has joined #openstack-ansible | 16:46 | |
| odyssey4me | palendae git-harry and others, excellent work on getting the kilo>liberty upgrades working | 16:47 |
| odyssey4me | it's great that we have the last two patches merging a week ahead of the next tag, so there's time to test them | 16:47 |
| palendae | odyssey4me: Yep. Hopefully any changes will be small | 16:47 |
| odyssey4me | it'd be great if we could put a basic framework in place in master so that we can incrementally add stuff as we go instead of having to do it as a block of work | 16:48 |
| palendae | Still need to get the manual process documented, but that's mostly running down the upgrade script and putting a little bit of prose around it | 16:48 |
| palendae | That'd be swell :) | 16:48 |
| openstackgerrit | Merged openstack/openstack-ansible-security: Add curl to bindep requirements https://review.openstack.org/288691 | 16:48 |
| odyssey4me | heh, I'm surprised this made it all the way through to tempest before flaking out: https://review.openstack.org/290447 | 16:50 |
| palendae | git-harry: Looks like we'll need to wait for those last 2 patches to merge - they were based on liberty before it got the bug fix patch merged. e.g. https://review.openstack.org/#/c/290333/1/scripts/run-upgrade.sh still says "SCRIPTS", not "SCRIPTS_PATH" on L151/141 | 16:50 |
| palendae | Actually nm, I'll do a cherry pick onto liberty instead of a checkout | 16:52 |
| *** izaakk has joined #openstack-ansible | 16:52 | |
| palendae | Maybe I should actually install Kilo on this AIO before running the upgrade scripts >.> | 16:54 |
| spotz | heheh | 16:54 |
| *** mathlin1 has joined #openstack-ansible | 16:56 | |
| odyssey4me | it's good to see that https://review.openstack.org/290486 is doing what it was intended to do :) | 16:57 |
| *** mgoddard_ has joined #openstack-ansible | 16:58 | |
| *** cluster444 has joined #openstack-ansible | 16:58 | |
| mattt | cloudnull: was hoping you would have a look at https://review.openstack.org/290446 before it merged | 17:01 |
| mattt | which is about to happen any second :) | 17:01 |
| mattt | mind having a quick look? | 17:01 |
| *** mgoddard has quit IRC | 17:02 | |
| cluster444 | Hi, having some trouble getting my compute nodes going. Did a fresh install of kilo, run-playbooks seemed to complete ok. Each compute node has nova-compute and neutron-linuxbridge-agent running, but nova service-list says the state is down | 17:02 |
| * cloudnull looking | 17:02 | |
| openstackgerrit | Merged openstack/openstack-ansible-galera_server: Initial commit for bindep requirements https://review.openstack.org/287870 | 17:02 |
| * odyssey4me removes +w so that cloudnull can look and approve | 17:02 | |
| odyssey4me | cluster444 do you have an ntp service on all nodes? | 17:03 |
| cluster444 | Another wierd thing is that glance fails when trying to add an image, it gets to 2%, hangs and then errors with a Broken Pipe error | 17:03 |
| *** raddaoui has quit IRC | 17:03 | |
| *** fawadkhaliq has joined #openstack-ansible | 17:03 | |
| openstackgerrit | Ala Raddaoui proposed openstack/openstack-ansible: Test if dynamic inventory catches some bad configs https://review.openstack.org/290011 | 17:03 |
| cluster444 | I'll check that | 17:03 |
| odyssey4me | cluster444 check that your time is all in sync and that your rabbitmq and mariadb clusters are in a healthy state | 17:03 |
| *** raddaoui has joined #openstack-ansible | 17:03 | |
| *** fawadkhaliq has quit IRC | 17:03 | |
| odyssey4me | cloudnull should the pip install not have the requirements implemented through --requirements and the upper-contraints through --constraints? | 17:04 |
| odyssey4me | it doesn't appear to make a difference, functionally | 17:05 |
| cluster444 | rabbitmq cluster_status looks good, how do i check that mariadb cluster is healthy? | 17:05 |
| odyssey4me | cluster444 some hints here: http://docs.openstack.org/developer/openstack-ansible/install-guide/ops-galera-recovery.html | 17:05 |
| cluster444 | mariadb cluster looks ok, checking on the time sync | 17:07 |
| openstackgerrit | Matt Thompson proposed openstack/openstack-ansible-os_keystone: Use contraints file in developer mode https://review.openstack.org/290446 | 17:08 |
| *** cloudtra_ has joined #openstack-ansible | 17:08 | |
| *** thorst_afk is now known as thorst_ | 17:09 | |
| *** fawadkhaliq has joined #openstack-ansible | 17:10 | |
| *** cloudtrainme has quit IRC | 17:10 | |
| mattt | cloudnull: what are your thoughts? are we approaching this incorrectly? | 17:11 |
| mattt | afk, back online in a bit | 17:14 |
| *** andrei__ has quit IRC | 17:15 | |
| *** asettle has joined #openstack-ansible | 17:15 | |
| *** asettle has quit IRC | 17:20 | |
| cloudnull | mattt: LGTM -- sorry I was pondering how to do this better. however I cant think of one. | 17:20 |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Clarify run-upgrade.sh failure message https://review.openstack.org/290707 | 17:21 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Set apt sources to use a configured list of components https://review.openstack.org/289835 | 17:22 |
| cloudnull | i dont like the additional dep and the requirement to track yet another branch/sha however it kinda looks like its required. | 17:22 |
| openstackgerrit | Jesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Set apt sources to use a configured list of components https://review.openstack.org/289833 | 17:23 |
| *** keedya has quit IRC | 17:25 | |
| odyssey4me | cloudnull yeah, I think it's pretty essential - hopefully we can get a proposal bot to update the sha's from time to time | 17:25 |
| cluster444 | ok, got ntp installed on compute nodes, compute nodes are all +/- 4ms now | 17:26 |
| cluster444 | they weren't really off by much to begin with, maybe 200ms on the worst node | 17:27 |
| *** mathlin1 has quit IRC | 17:29 | |
| *** raddaoui has quit IRC | 17:32 | |
| *** raddaoui has joined #openstack-ansible | 17:32 | |
| openstackgerrit | git-harry proposed openstack/openstack-ansible-galera_server: Re-initialise when restarting 1-node cluster https://review.openstack.org/290716 | 17:33 |
| odyssey4me | odd - two different patches, both producing nova API errors in the tempest phase: https://review.openstack.org/290486 / https://review.openstack.org/290447 | 17:34 |
| odyssey4me | git-harry good patch, can I ask whether it's possible to actually have the cluster automatically initialise if the container is restarted? | 17:35 |
| odyssey4me | git-harry ie perhaps --wsrep-new-cluster can go into the initi script if there's only one node | 17:35 |
| odyssey4me | that will be a massive help to the usability in a single galera node (usually test environment) use-case | 17:36 |
| *** admin0 has joined #openstack-ansible | 17:36 | |
| palendae | raddaoui: Ok, I'll slow down with the reviews on that patch now so you can get a chance to look at them :) | 17:36 |
| git-harry | odyssey4me: if we add it to the init script we make that node special | 17:36 |
| odyssey4me | git-harry but if there's only one node, then it is special :) | 17:37 |
| git-harry | odyssey4me: True. I guess it just seems like we're doing something unexpected to work around something that isn't an issue, it's just the way it works. | 17:39 |
| *** cloudtra_ has quit IRC | 17:39 | |
| git-harry | odyssey4me: I'd be opposed to that but I guess we can discuss it in the review and see what everyone else thinks | 17:39 |
| odyssey4me | git-harry sure, but it would be quite useful to have a developer AIO that's cut down as a simple use-case for the entry-level | 17:39 |
| cluster444 | is there something i do to enable a compute node? or does it just auto-enable itself when nova-compute starts on that node? | 17:40 |
| spotz | odyssey4me I was working https://bugs.launchpad.net/openstack-ansible/+bug/1550041 and the more I read the configure-cinder documentation page and the more it seems the page is only about setting up netapp. There is a section talking about API but it's the netapps | 17:40 |
| openstack | Launchpad bug 1550041 in openstack-ansible "Chapter 4, configure Block Storage missing storage-infra_hosts" [Low,Confirmed] - Assigned to Amy Marrich (amy-marrich) | 17:40 |
| odyssey4me | cluster444 it registers itself and its state | 17:40 |
| raddaoui | yes palendae, I will look at it after lunch | 17:40 |
| palendae | raddaoui: Yep, no rush. | 17:40 |
| odyssey4me | spotz yeah, we should actually improve the page to show examples for ceph, netapp and perhaps an NFS back-end | 17:40 |
| cluster444 | so if its not doing that, then it must not be able to talk to whatever it registers with? | 17:41 |
| odyssey4me | cluster444 it communicates through rabbitmq, the conductor service and to the db | 17:41 |
| spotz | odyssey4me As long as I'm not going nuts as I wasn't seeing a clean way to just insert the storage-infra_hosts in and talk about cinder api | 17:41 |
| git-harry | odyssey4me: yeah, I get what you're aiming for. It's a reasonable goal but I think I'd prefer better documentation than doing something that wouldn't exist in 'real life'. | 17:42 |
| *** cloudtrainme has joined #openstack-ansible | 17:42 | |
| cluster444 | and all of which seem to be running fine, but the computes are not coming up | 17:43 |
| odyssey4me | spotz you're not going nuts - I was looking at that the other day and it made me sad | 17:44 |
| openstackgerrit | Merged openstack/openstack-ansible: Update user variables to Liberty names https://review.openstack.org/287438 | 17:44 |
| openstackgerrit | Merged openstack/openstack-ansible: Revert "Exit early from unfinished upgrade script" https://review.openstack.org/290333 | 17:44 |
| openstackgerrit | Merged openstack/openstack-ansible: Retry tempest cirros image creation https://review.openstack.org/290381 | 17:44 |
| openstackgerrit | Merged openstack/openstack-ansible: Minor fixes for App-PLUMgrid doc entry https://review.openstack.org/289900 | 17:44 |
| odyssey4me | heh, let the games begin :) | 17:45 |
| spotz | odyssey4me ok, I'll see what I can do to get it in order and add more info | 17:45 |
| *** raddaoui has quit IRC | 17:45 | |
| odyssey4me | thanks spotz | 17:45 |
| *** skamithi14 has joined #openstack-ansible | 17:47 | |
| odyssey4me | ok, time for me to go offline for the night | 17:47 |
| odyssey4me | see y'all tomorrow! | 17:47 |
| automagically | Later odyssey4me | 17:48 |
| *** skamithi14 has quit IRC | 17:48 | |
| *** skamithi14 has joined #openstack-ansible | 17:49 | |
| *** skamithi13 has quit IRC | 17:49 | |
| openstackgerrit | Andrew Mitry proposed openstack/openstack-ansible: Fix: AODH sort rabbitmq hosts pre-task should use 'aodh_all' group https://review.openstack.org/290725 | 17:53 |
| *** kencjohnston has quit IRC | 17:54 | |
| cluster444 | should there be more logging after the line 'Auditing locally available compute resources' in the nova-compute output? | 17:55 |
| openstackgerrit | Andrew Mitry proposed openstack/openstack-ansible: Fix: AODH sort rabbitmq hosts pre-task should use 'aodh_all' group https://review.openstack.org/290725 | 17:55 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_horizon: Enable functional convergence testing https://review.openstack.org/290034 | 18:00 |
| openstackgerrit | git-harry proposed openstack/openstack-ansible-galera_server: Use var galera_cluster_members to identify cluster https://review.openstack.org/290729 | 18:00 |
| cluster444 | i added some extra logging into nova compute resource tracker, it seems the call to get_available_resources never returns | 18:05 |
| *** rohanp_ has joined #openstack-ansible | 18:05 | |
| rohanp_ | Hi folks, i am facing an issue wherein i am trying to test my ansible changes using AIO..i am failing at bootstrap ansible step itself..i am using liberty | 18:06 |
| rohanp_ | the bootstrap ansible doesnot thrown any errors though it just stops executing | 18:06 |
| rohanp_ | https://usercontent.irccloud-cdn.com/file/SrtxuR8j/ | 18:07 |
| rohanp_ | it doesnot run to completion and hence i cannot run next steps of bootstrap aio | 18:07 |
| *** TheIntern has quit IRC | 18:09 | |
| *** andyhky` has joined #openstack-ansible | 18:11 | |
| *** sdake has quit IRC | 18:11 | |
| *** andyhky` is now known as andyhky | 18:11 | |
| *** flaviosr has joined #openstack-ansible | 18:11 | |
| *** michaelgugino has quit IRC | 18:12 | |
| *** michaelgugino has joined #openstack-ansible | 18:12 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-rabbitmq_server: [WIP] Multi-distro support for rabbitmq_server role https://review.openstack.org/286282 | 18:14 |
| *** iurygregory has joined #openstack-ansible | 18:14 | |
| *** mgoddard_ has quit IRC | 18:15 | |
| openstackgerrit | Nolan Brubaker proposed openstack/openstack-ansible: Move inventory environment loading to function https://review.openstack.org/290740 | 18:15 |
| flaviosr | Hi people, i'm testing ansible to deploy a cloud and i'm facing this bug https://bugs.launchpad.net/openstack-ansible/+bug/1555256 | 18:15 |
| openstack | Launchpad bug 1555256 in openstack-ansible "os_neutron: no service or tool found for: neutron-lbaas-agent" [Undecided,New] - Assigned to Flávio Ramalho (flaviosr) | 18:15 |
| flaviosr | i've fixed locally, this bug is valid upstream? | 18:16 |
| cloudnull | flaviosr: looks like a real bug | 18:19 |
| cloudnull | seems valid | 18:20 |
| cloudnull | i assume its because the lbaas agent isint installed at that point and the check is failing. | 18:20 |
| cloudnull | mhayden: ^ | 18:20 |
| flaviosr | exactly | 18:21 |
| *** cloudtrainme has quit IRC | 18:22 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-rabbitmq_server: [WIP] Multi-distro support for rabbitmq_server role https://review.openstack.org/286282 | 18:22 |
| cloudnull | odyssey4me: looking into the keystone issues you we're mentioning earlier. | 18:22 |
| cloudnull | im not seeing any issues | 18:22 |
| cloudnull | i moved my install to keystone using the head of master | 18:22 |
| cloudnull | and its working fine | 18:22 |
| cloudnull | I even applied it to an existing env to upgrade it and see if something was there and nothing. | 18:23 |
| cloudnull | maybe the reporter was using keystone client and not openstack client which caused the issues? | 18:23 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_heat: Account for oslo_config deprecations in heat.conf template https://review.openstack.org/290748 | 18:23 |
| flaviosr | thanks cloudnull, i will send a patch asap! | 18:24 |
| cloudnull | tyvm flaviosr | 18:24 |
| cloudnull | thats great | 18:24 |
| *** phiche has quit IRC | 18:25 | |
| jmccrory | mattt looks like requirements capped pysaml2 a few weeks after the SHAs in openstack-ansible https://review.openstack.org/#/c/280968/ | 18:26 |
| *** keedya has joined #openstack-ansible | 18:26 | |
| *** cloudtrainme has joined #openstack-ansible | 18:27 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-rabbitmq_server: [WIP] Multi-distro support for rabbitmq_server role https://review.openstack.org/286282 | 18:30 |
| *** McMurlock1 has quit IRC | 18:33 | |
| *** leifmadsen has left #openstack-ansible | 18:40 | |
| *** electrofelix has quit IRC | 18:40 | |
| *** grumpycatt has quit IRC | 18:43 | |
| *** gayathri has joined #openstack-ansible | 18:43 | |
| *** grumpycatt has joined #openstack-ansible | 18:45 | |
| *** b3rnard0_away is now known as b3rnard0 | 18:45 | |
| *** bsv has quit IRC | 18:50 | |
| *** bsv has joined #openstack-ansible | 18:53 | |
| openstackgerrit | Steve Lewis proposed openstack/openstack-ansible-os_ceilometer: Remove unused galera_client and related vars https://review.openstack.org/290770 | 18:54 |
| bsv | admin0: url? :D | 18:54 |
| admin0 | https://www.openstackfaq.com/ | 18:55 |
| admin0 | under private cloud | 18:55 |
| gayathri | Hi All. I m trying to setup ovs manually after installing all the services using community playbook. when i install openvswitch package m getting only 2.0.2 which is not suitable for liberty | 18:55 |
| admin0 | BUT it not yet published :D | 18:55 |
| admin0 | workign on it sir | 18:55 |
| admin0 | working on it | 18:55 |
| admin0 | it wil be today :D | 18:55 |
| admin0 | allow me to work :D | 18:55 |
| gayathri | from the repo which is been pointed (i.e.rackspace) only this version is getting downloaded.. | 18:56 |
| *** McMurlock1 has joined #openstack-ansible | 18:56 | |
| gayathri | How cani get the latest version 2.4.0 for ovs | 18:56 |
| *** sdake has joined #openstack-ansible | 18:57 | |
| openstackgerrit | Flávio Ramalho proposed openstack/openstack-ansible-os_neutron: Ignore error when neutron-lbaas-agent is not installed https://review.openstack.org/290772 | 18:57 |
| gayathri | which sources.list I need to update | 18:57 |
| gayathri | Can anybody please help on my query | 18:57 |
| openstackgerrit | Steve Lewis proposed openstack/openstack-ansible: Remove unused galera_address from playbook https://review.openstack.org/290773 | 18:57 |
| cloudnull | gayathri: we dont support OVS. be let me have a look to see what is recommended. | 18:57 |
| *** mgoddard has joined #openstack-ansible | 18:58 | |
| gayathri | sure..thanks cloudnull.. please let me know your suggestion | 18:59 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_horizon: Enable functional convergence testing https://review.openstack.org/290034 | 19:00 |
| bsv | admin0: you wont even notice me, I will leave you the lonest of alones... :p | 19:01 |
| admin0 | :D | 19:01 |
| *** markvoelker has quit IRC | 19:01 | |
| cloudnull | gayathri: looks like cloud archive has 2.4 in it for their liberty release | 19:02 |
| cloudnull | https://wiki.ubuntu.com/ServerTeam/CloudArchive | 19:02 |
| cloudnull | run: add-apt-repository cloud-archive:liberty && apt-get update | 19:02 |
| cloudnull | and you should be able to install 2.4.0 | 19:02 |
| cloudnull | actual package 2.4.0-0ubuntu4~cloud0 | 19:02 |
| cloudnull | docs on cloud archive https://wiki.ubuntu.com/ServerTeam/CloudArchive | 19:03 |
| *** markvoelker has joined #openstack-ansible | 19:03 | |
| *** kencjohnston has joined #openstack-ansible | 19:04 | |
| *** asettle has joined #openstack-ansible | 19:04 | |
| gayathri | after installing neutron service..inside that container i tried adding repo and apt-get update..in the node I was able to get 2.4.0 | 19:04 |
| gayathri | but not inside the container :( | 19:04 |
| gayathri | i got some error with respect to version | 19:06 |
| gayathri | ssl sync error | 19:06 |
| *** unlaudable has joined #openstack-ansible | 19:07 | |
| *** asettle has quit IRC | 19:08 | |
| *** metral_zzz is now known as metral | 19:09 | |
| *** phiche has joined #openstack-ansible | 19:09 | |
| openstackgerrit | Hugh Saunders proposed openstack/openstack-ansible: Fix command not found in scripts lib https://review.openstack.org/290778 | 19:09 |
| *** metral is now known as metral_zzz | 19:09 | |
| cloudnull | gayathri: can you paste that ? | 19:10 |
| gayathri | sorry.. i do not have the error log now.. | 19:10 |
| cloudnull | ah sorry for the late reply | 19:11 |
| *** skamithi14 has quit IRC | 19:12 | |
| gayathri | Thanks for your suggestions cloudnull.. I will try again and get back | 19:13 |
| *** gayathri has quit IRC | 19:13 | |
| cloudnull | let me know if theres something i can help with :) | 19:15 |
| *** markvoelker has quit IRC | 19:17 | |
| *** markvoelker has joined #openstack-ansible | 19:18 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_ceilometer: Account for oslo_config deprecation in ceilometer.conf template https://review.openstack.org/290787 | 19:19 |
| stevelle | hah, I thought I did that already automagically | 19:19 |
| stevelle | guess not | 19:19 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_horizon: Enable functional convergence testing https://review.openstack.org/290034 | 19:26 |
| *** sdake_ has joined #openstack-ansible | 19:30 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_cinder: Account for oslo_config deprecation in cinder.conf template https://review.openstack.org/290799 | 19:30 |
| *** sdake has quit IRC | 19:32 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_aodh: Account for oslo_config deprecations in aodh.conf template https://review.openstack.org/290801 | 19:35 |
| *** thorst__ has joined #openstack-ansible | 19:36 | |
| *** michaelgugino has quit IRC | 19:38 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_ceilometer: Account for oslo_config deprecation in ceilometer.conf template https://review.openstack.org/290787 | 19:39 |
| *** thorst_ has quit IRC | 19:40 | |
| automagically | Can I get some +2s on https://review.openstack.org/#/c/287571/ please | 19:40 |
| *** raddaoui has joined #openstack-ansible | 19:43 | |
| automagically | Or, rather reviews, vote as you wish clearly | 19:44 |
| hughsaunders | haha three-line whip | 19:45 |
| *** cloudtrainme has quit IRC | 19:48 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 19:48 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_glance: Account for oslo_config deprecations in glance config templates https://review.openstack.org/290811 | 19:50 |
| hughsaunders | Anyone else seen gate failures on "ensure service tenant"? I'm pretty sure its unrelated to my patch as that code isn't executed before the failure. kibana tells me that other jobs aren't having the same error though. (kibana query: build_name:"gate-openstack-ansible-dsvm-commit" AND message:"keystoneclient.openstack.common.apiclient.exceptions.ServiceUnavailable") https://review.openstack.org/#/c/290382/1 | 19:51 |
| automagically | I’ve seen maybe one in the last week | 19:52 |
| automagically | Don’t recall where | 19:52 |
| automagically | It rings a bell though | 19:52 |
| *** skamithi13 has joined #openstack-ansible | 19:55 | |
| automagically | Here’s a puzzler. Check fails due to http://logs.openstack.org/34/290034/5/check/gate-openstack-ansible-os_horizon-ansible-func-ubuntu-trusty/82d32ef/console.html#_2016-03-09_19_48_43_143, and yet: http://logs.openstack.org/34/290034/5/check/gate-openstack-ansible-os_horizon-ansible-func-ubuntu-trusty/82d32ef/console.html#_2016-03-09_19_47_52_404 | 19:56 |
| automagically | TLDR - keystone module complains that python-keystoneclient isn’t installed, but log shows Ansible installing python-keystoneclient just prior | 19:57 |
| hughsaunders | automagically: hmm, so is the keystone module activating the venv properly? | 19:59 |
| automagically | Ah right, Ansible itself isn’t running in that venv | 20:01 |
| automagically | I see that the Keystone role handles this with a var keystone_requires_pip_packages which get installed in the container outside of the venv | 20:03 |
| automagically | Guess I’ll give that a shot with Horizon | 20:03 |
| hughsaunders | sounds good, I couldn't think of a way to get ansible to activate a venv before executing a module. | 20:04 |
| automagically | Hmm, its already doing that…which would make sense | 20:04 |
| mhayden | cloudnull / flaviosr: i'll check on that bug | 20:05 |
| automagically | http://logs.openstack.org/34/290034/5/check/gate-openstack-ansible-os_horizon-ansible-func-ubuntu-trusty/82d32ef/console.html#_2016-03-09_19_46_20_587 | 20:05 |
| flaviosr | k, thanks mhayden :) | 20:05 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_horizon: Enable functional convergence testing https://review.openstack.org/290034 | 20:07 |
| mhayden | flaviosr: i see your patch there in gerrit, but it may make more sense to reorder the includes in main.yml | 20:09 |
| spotz | Fell behind on the channel poke me with links for any reviews I'm needed on | 20:09 |
| mhayden | or well, i guess i am confused... the actual neutron lbaas service scripts should be in place by the time neutron_lbaas.yml is executed | 20:09 |
| automagically | All the reviews spotz, all the reviews. Otherwise how are you going to stay ahead of mhayden | 20:09 |
| mhayden | so many reviews, i'm seeing spotz | 20:10 |
| flaviosr | hahaha | 20:10 |
| spotz | true that automagically. Need to do something in *gasp* chef then I'll look for reviews:) | 20:10 |
| *** raddaoui has quit IRC | 20:11 | |
| mhayden | when i hear chef, i /quit | 20:11 |
| flaviosr | mhaydenm you mean by the time the task to stop lbaas-agent run, it should have already installed the agent? | 20:12 |
| *** mgoddard has quit IRC | 20:12 | |
| flaviosr | I think this task is just for the case that you already have an lbaas-agentv1 running and is installing a lbass-agentv2, so you need to stop the other agent | 20:13 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova: Add support for the nova_api db https://review.openstack.org/287945 | 20:16 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer: Account for oslo_config deprecation in ceilometer.conf template https://review.openstack.org/290787 | 20:18 |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh: Account for oslo_config deprecations in aodh.conf template https://review.openstack.org/290801 | 20:18 |
| *** sdake has joined #openstack-ansible | 20:21 | |
| *** kencjohnston has quit IRC | 20:21 | |
| *** sdake_ has quit IRC | 20:23 | |
| mhayden | flaviosr: oh, i think i see what the problem is | 20:27 |
| mhayden | service_en ends up being false, so the whole service doesn't get deployed | 20:27 |
| mhayden | i think might adjust the neutron_lbass.yml playbook to check for the service and take action *if* it exists | 20:28 |
| mhayden | so that the appropriate action is taken if the deployer has deployed that service in the past | 20:28 |
| flaviosr | i see | 20:29 |
| flaviosr | I think i can do that, i just start learning ansible, so it will be good to learn | 20:29 |
| mhayden | flaviosr: ah, okay -- you're welcome to! | 20:30 |
| flaviosr | mhayden: so i should check if the service exist, if so stop it right? | 20:30 |
| openstackgerrit | Steve Lewis proposed openstack/openstack-ansible-os_nova: Resolve oslo_config deprecations in nova.conf https://review.openstack.org/290834 | 20:31 |
| mhayden | flaviosr: right -- i just put comments in your review | 20:31 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder: Account for oslo_config deprecation in cinder.conf template https://review.openstack.org/290799 | 20:32 |
| flaviosr | thats great! thanks mhayden! | 20:32 |
| mhayden | you're welcome | 20:32 |
| mhayden | thanks for finding the bug i (accidentally) created, flaviosr! :P | 20:33 |
| flaviosr | thats fine :) | 20:33 |
| automagically | one hopes you aren’t creating bugs on purpose mhayden ;) | 20:33 |
| *** kencjohnston has joined #openstack-ansible | 20:33 | |
| * mhayden does like smashing them | 20:33 | |
| mhayden | perhaps i thought the services would be deployed automagically | 20:34 |
| mhayden | but now it's turned into an odyssey4me | 20:34 |
| automagically | As one does... | 20:34 |
| mhayden | but i'm learning new spotz of the OSA code | 20:34 |
| mhayden | right now, i seem to be beating a d34dh0r53 | 20:34 |
| spotz | hehehe | 20:34 |
| d34dh0r53 | ba dum cha | 20:34 |
| *** lykinsbd has joined #openstack-ansible | 20:36 | |
| *** cloudtrainme has joined #openstack-ansible | 20:41 | |
| *** lykinsbd has quit IRC | 20:41 | |
| bsv | Anyone using Emulex NIC's with their OpenStack setup? | 20:41 |
| *** raddaoui has joined #openstack-ansible | 20:43 | |
| *** asettle has joined #openstack-ansible | 20:46 | |
| mattt | pysaml2 released a fast follow to 4.0.4 which broke keystone in master | 20:50 |
| mattt | so thankfully our IRR tests are passing now | 20:50 |
| *** asettle has quit IRC | 20:50 | |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible: OpenStack services should reach Glance via the internal LB VIP https://review.openstack.org/290844 | 20:51 |
| palendae | Is anyone else getting consistent apt failures, or is it just https://review.openstack.org/#/c/290707/ ? | 20:56 |
| *** raddaoui has quit IRC | 20:57 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 20:58 | |
| spotz | https://review.openstack.org/#/q/status:open+openstack-ansible is my friend:) | 21:00 |
| cloudnull | palendae: oddly enough, gayathri was talking about apt failures | 21:01 |
| cloudnull | maybe their repo is syncing ? | 21:01 |
| palendae | cloudnull: Hm, ok | 21:01 |
| mattt | did odyssey4me flip over to infra's apt servers yet? | 21:02 |
| palendae | mattt: I think so | 21:02 |
| mattt | wonder if it's related to that | 21:02 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_nova: OpenStack services should reach Glance via the internal LB VIP https://review.openstack.org/290851 | 21:02 |
| stevelle | automagically: https://review.openstack.org/#/c/290834/1 | 21:05 |
| stevelle | I'm curious if you get a deprecation msg for api_servers as well | 21:05 |
| automagically | Ah, whoops | 21:06 |
| stevelle | lets see what the gate tells us on yours | 21:06 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-rabbitmq_server: [WIP] Multi-distro support for rabbitmq_server role https://review.openstack.org/286282 | 21:06 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-rabbitmq_server: [WIP] Multi-distro support for rabbitmq_server role https://review.openstack.org/286282 | 21:08 |
| bsv | cloudnull Apsu: https://access.redhat.com/solutions/774743 | 21:10 |
| bsv | perhaps? | 21:10 |
| Apsu | Got Emulex NICs? | 21:11 |
| openstackgerrit | Travis Truman (automagically) proposed openstack/openstack-ansible-os_cinder: OpenStack services should reach Glance via the internal LB VIP https://review.openstack.org/290856 | 21:11 |
| bsv | Apsu: yep | 21:11 |
| Apsu | bsv: Worth checking into, then! Not familiar with them | 21:12 |
| bsv | More from the same dept.; http://support.citrix.com/article/CTX137598 | 21:12 |
| bsv | OpenVS tho | 21:12 |
| rohanp_ | Folks, 1 question regrading providing an option for own neutron plugin similar to plumgrid..while testing this shud the overriding neutron plugin be specified in "/etc/openstack_deploy/user_variables.yml" file or it can be any file like user_xxx_vars.yml under /etc/openstack_deploy | 21:13 |
| *** Mudpuppy has quit IRC | 21:13 | |
| automagically | Done for the day, see you all tomorrow | 21:13 |
| mhayden | night automagically | 21:13 |
| rohanp_ | the goal here is to override the ml2 plugin option and provide own neutron option similar to plumgrid | 21:13 |
| bsv | Apsu: I just contacted HPE and asked them for a set of Intel NIC's to test with. Expect to have them tomorrow or the day after. | 21:13 |
| mhayden | darn -- just tried a master build and discovered i need a user secret from https://review.openstack.org/#/c/274932/12 :P | 21:14 |
| bsv | Would be nice to find out if this is the source of all our pain and agony ;) | 21:14 |
| Apsu | bsv: Indeed! | 21:14 |
| *** cloudtrainme has quit IRC | 21:14 | |
| Apsu | mhayden: First rule of user secrets, don't talk about them | 21:15 |
| * mhayden hugs Apsu | 21:15 | |
| bsv | I asked in #openstack-neutron if anyone was using Emulex, but the silence is strong on that channel ;) | 21:16 |
| Apsu | mhayden: d'awww | 21:16 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-rabbitmq_server: [WIP] Multi-distro support for rabbitmq_server role https://review.openstack.org/286282 | 21:16 |
| mhayden | bsv: what's unique about the emulex cards? | 21:17 |
| mhayden | Apsu: imma pour out a vuka in your honor | 21:17 |
| bsv | mhayden: the chipet is supported by OpenBSD. | 21:17 |
| *** jguy_ has quit IRC | 21:17 | |
| openstackgerrit | Steve Lewis proposed openstack/openstack-ansible-os_nova: Resolve oslo_config deprecations in nova.conf https://review.openstack.org/290834 | 21:18 |
| *** raddaoui has joined #openstack-ansible | 21:18 | |
| bsv | mhayden: and we use OpenBSD... ;) | 21:18 |
| *** markvoelker has quit IRC | 21:20 | |
| Apsu | mhayden: Do it for the homies | 21:22 |
| Apsu | bsv: Wait, you're running OpenStack on OBSD? | 21:22 |
| bsv | Apsu: no... | 21:22 |
| bsv | ...not yet... | 21:22 |
| Apsu | Or just saying you use OBSD in your org... gotcha | 21:22 |
| bsv | :D | 21:22 |
| Apsu | Same NICs across the board. Makes sense | 21:23 |
| bsv | Yeah, all servers are identical | 21:23 |
| bsv | Apsu: but I would really like to have OpenBSD as my 'Network Node' for OpenStack, that would be nice. | 21:24 |
| bsv | With the changes comming up, that goal will be very close. | 21:25 |
| Apsu | I prefer Linux for networking these days, but I'm a lot more equipped and informed about the minutia of Linux Networking than most, so yeah | 21:25 |
| *** cloudtrainme has joined #openstack-ansible | 21:25 | |
| admin0 | bsv: https://www.openstackfaq.com/openstack-liberty-private-cloud-howto/ | 21:26 |
| admin0 | just the basics are up | 21:26 |
| bsv | admin0: (thumbsup) | 21:27 |
| bsv | :D | 21:27 |
| admin0 | tomorrow i will add how to make that develppment box . with just run.sh and it will delete and re-create c11 .. c25 for re-runs | 21:28 |
| bsv | Apsu: im only touching linux networking because of OpenStack. I'd rather stay in BSD land ;) | 21:28 |
| bsv | But if I have to learn OpenStack, I need to know Linux networking.. end of story. | 21:29 |
| *** thorst__ is now known as thorst_ | 21:30 | |
| admin0 | i wanted to use pfsense insetad of vyos, but pfsense put my 24 core 128gb mem test machine to lockup . so vyos :D | 21:30 |
| Apsu | Just make sure you learn the appropriate tools. ip, bridge, ss, lnstat | 21:30 |
| bsv | Apsu: And most maps directly over, except you have to install stuff to make anything work on Linux :) | 21:30 |
| Apsu | Rather than ifconfig/brctl/route/netstat | 21:30 |
| Apsu | The latter are broken, abandoned and lie to you | 21:30 |
| bsv | Hey wait a minute, the OpenStack doc says 'use brctl' | 21:30 |
| bsv | :) | 21:30 |
| palendae | Apsu has Opinions :) | 21:31 |
| admin0 | bsv: curl gw.am .. this is one domain i have that i use to check if my outgoing ips are correct | 21:31 |
| Apsu | I'm sure it does. But the common distros still haven't fully abandoned net-tools or bridge-utils, despite the code itself correctly having moved on. | 21:31 |
| admin0 | youc an do that to check your outbound ip to verify if you have a real nat working and the outgoing ip is the ip you were expecting | 21:31 |
| admin0 | cirros has curl, so perfect to test | 21:32 |
| *** keedya has quit IRC | 21:32 | |
| bsv | Apsu: noted. | 21:32 |
| *** raddaoui has quit IRC | 21:33 | |
| bsv | admin0: i have a setup for that too :) nice to have. | 21:33 |
| admin0 | i added geoip stuff there ..sometimes i am in NL and i see my ip is RU :D | 21:33 |
| bsv | geoip is broken :) | 21:34 |
| admin0 | i made network name as DHCP and FLOATING .. youcan add ranges like test, office, dev, prod, guests | 21:34 |
| *** raddaoui has joined #openstack-ansible | 21:34 | |
| admin0 | vyos will allow you to do site-2-site vpn, so everything on DHCP could be seamlessly accessible from office | 21:35 |
| openstackgerrit | Ala Raddaoui proposed openstack/openstack-ansible: Test if dynamic inventory catches some bad configs https://review.openstack.org/290011 | 21:35 |
| bsv | Apsu: do you have some tool/way to quickly verify container inter-connectivity? | 21:37 |
| Apsu | bsv: We usually run ad-hoc ping checks between all the IPs via ansible. | 21:38 |
| Apsu | I think our support team has some scripts for it... | 21:38 |
| Apsu | cloudnull: You have anything along those lines that could be linked? | 21:38 |
| cloudnull | like can the neutron container get to the galera container? | 21:39 |
| cloudnull | or like can the deploy host get to everythoing? | 21:39 |
| Apsu | Think the former is what they mean | 21:39 |
| cloudnull | nothing that im aware of. | 21:40 |
| Apsu | Could probably grab the IP list from inventory-manage.py and do something clever | 21:42 |
| spotz | Anyone have a cinder NFS example? | 21:43 |
| *** sdake has quit IRC | 21:44 | |
| bsv | cloudnull: just a way to verify that all containers indeed can reach each other. | 21:45 |
| bsv | i would have saved us a few days ;) | 21:46 |
| cloudnull | that sounds very useful. however i dont have anything in my tool belt for that right now. | 21:46 |
| cloudnull | we could make it :) | 21:47 |
| Apsu | bsv: You could workup a playbook to do it. Target the host group of all containers, then use a loop in the task over the same group, except when the iteration matches the current host | 21:47 |
| Apsu | Run a ping against it | 21:47 |
| Apsu | Limit to 3 pings or so, fail the task when it errors | 21:47 |
| Apsu | Should be pretty short and quick | 21:48 |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-rabbitmq_server: [WIP] Multi-distro support for rabbitmq_server role https://review.openstack.org/286282 | 21:49 |
| bsv | Apsu: I would make it myself, if you did not have any exising :D | 21:52 |
| bsv | existing* | 21:52 |
| Apsu | I bet our support guys have stuff, just not sure where it'd be or if I could share, offhand lol | 21:53 |
| Apsu | Seems easy to do | 21:53 |
| bsv | But a simple ICMP echo-reply test is not enough, actual IP packets have to be exchanged. | 21:53 |
| bsv | In our case, all containers can ping each other, but if you try to establish an TCP stream.. boom, drop arp, all dead. | 21:54 |
| bsv | A basic ssh $remote-container "time" would do it. | 21:55 |
| admin0 | bsv: mtu ? | 21:55 |
| openstackgerrit | Amy Marrich (spotz) proposed openstack/openstack-ansible: Added configuration information for using Ceph with Cinder https://review.openstack.org/290873 | 21:55 |
| bsv | admin0: mtu is 1500 all over, and no headers added | 21:56 |
| *** cloudtrainme has quit IRC | 21:57 | |
| Apsu | bsv: ICMP still requires ARP and rides on IP. If you're specifically having TCP issues, sounds like packet size, so I'm wondering about MTUs and checksums now. | 21:58 |
| Apsu | ICMP is IP protocol #1 | 21:58 |
| Apsu | Could be an offloading issue | 21:59 |
| *** cloudtrainme has joined #openstack-ansible | 21:59 | |
| bsv | Apsu: my fault, when I wrote IP... i meant to write tcp. | 22:00 |
| Apsu | No worries | 22:00 |
| Apsu | Have you tried very large payload pings? | 22:00 |
| bsv | if the container have the arp-entry for the remote host, you can ping all day long...it works. | 22:00 |
| stevelle | mhayden: you think that tempest flavors thing is transient? | 22:01 |
| Apsu | If those fail, MTU is more likely. If not, TCP offloading is suspect | 22:01 |
| *** skamithi14 has joined #openstack-ansible | 22:01 | |
| bsv | But try and do ex. ssh, the arp-entry gets dropped right away, and arp who-has is issued at once. | 22:01 |
| *** javeriak has quit IRC | 22:02 | |
| openstackgerrit | Jimmy McCrory proposed openstack/openstack-ansible-rabbitmq_server: [WIP] Multi-distro support for rabbitmq_server role https://review.openstack.org/286282 | 22:02 |
| Apsu | Sounds like duplicate MACs or IPs or bridging loops or Something... | 22:02 |
| *** thorst_ has quit IRC | 22:02 | |
| bsv | if you try and ssh to a host (not container host), it all works fine. | 22:02 |
| Apsu | The ARP entry shouldn't "drop" immediately, ever, but it could be replaced if the IP stays the same but MAC changes on seeing the traffic | 22:02 |
| *** skamithi15 has joined #openstack-ansible | 22:03 | |
| mhayden | stevelle: beginning to wonder if it isn't | 22:04 |
| bsv | But it does :) intriguing huh? :) | 22:04 |
| *** skamithi13 has quit IRC | 22:04 | |
| Apsu | bsv: Sounds loopy or duplicatey is all :) | 22:04 |
| stevelle | mhayden: I found a critical error in nova api when I was hunting, and https://review.openstack.org/#/c/290834/ might fix it but I haven't jiggled the parts yet to see | 22:05 |
| *** thorst_ has joined #openstack-ansible | 22:05 | |
| stevelle | critical msg said it couldn't find auth_type in config | 22:05 |
| mhayden | stevelle: ah okay -- i got bitten by the missing nova api user password earlier :| | 22:05 |
| *** skamithi14 has quit IRC | 22:05 | |
| bsv | (╯°□°）╯︵ ┻━┻ | 22:06 |
| * mhayden ducks to avoid the table | 22:06 | |
| stevelle | (°_°) ┬─┬ please respect tables | 22:06 |
| mhayden | little bobby tables? | 22:06 |
| stevelle | nah, you can drop him | 22:07 |
| mhayden | https://xkcd.com/327/ | 22:07 |
| *** retreved has quit IRC | 22:07 | |
| Apsu | mhayden: <3 | 22:07 |
| mhayden | stevelle: actually, i got the tempest failure here with something totally unrelated -> https://review.openstack.org/#/c/290526/ | 22:08 |
| openstackgerrit | Steve Lewis proposed openstack/openstack-ansible: Remove unused galera_address from playbook https://review.openstack.org/290773 | 22:09 |
| *** thorst_ has quit IRC | 22:09 | |
| *** Mudpuppy has joined #openstack-ansible | 22:10 | |
| stevelle | mhayden: I think nova is going to fail all tempest until this is fixed "http://logs.openstack.org/26/290526/1/check/gate-openstack-ansible-dsvm-commit/db5b788/logs/aio1_nova_api_os_compute_container-3bfadf28/nova-api-os-compute.log | 22:10 |
| stevelle | search for CRITICAL nova [-] MissingRequiredOptions | 22:10 |
| stevelle | we just need to get that auth_url in which seems to be missing for some reason | 22:11 |
| *** raddaoui has quit IRC | 22:17 | |
| openstackgerrit | Steve Lewis proposed openstack/openstack-ansible-os_nova: Revert "Fix auth_url for nova configuration" https://review.openstack.org/290881 | 22:18 |
| *** phalmos has quit IRC | 22:19 | |
| openstackgerrit | Steve Lewis proposed openstack/openstack-ansible-os_nova: Revert "Fix auth_url for nova configuration" https://review.openstack.org/290881 | 22:20 |
| *** raddaoui has joined #openstack-ansible | 22:20 | |
| openstackgerrit | Steve Lewis proposed openstack/openstack-ansible: Remove unused galera_address from playbook https://review.openstack.org/290773 | 22:21 |
| *** jguy_ has joined #openstack-ansible | 22:23 | |
| *** admin0 has quit IRC | 22:23 | |
| *** thorst_ has joined #openstack-ansible | 22:24 | |
| flaviosr | mhayden, can you give me a hand? | 22:25 |
| *** McMurlock1 has quit IRC | 22:27 | |
| *** automagically_ has joined #openstack-ansible | 22:28 | |
| mhayden | stevelle: ugh that's weird | 22:29 |
| mhayden | flaviosr: yes sir | 22:29 |
| mhayden | flaviosr: i have 5-10 minutes until i must go, though | 22:29 |
| flaviosr | oh thats fine, i got it nvm. thanks anyway for being reachable :) | 22:30 |
| *** b3rnard0 is now known as b3rnard0_away | 22:30 | |
| *** KLevenstein has quit IRC | 22:32 | |
| *** kencjohnston has quit IRC | 22:32 | |
| *** woodard has quit IRC | 22:33 | |
| mhayden | haha okay | 22:33 |
| *** asettle has joined #openstack-ansible | 22:34 | |
| stevelle | cloudnull: d34dh0r53 any other cores? https://review.openstack.org/290881 | 22:36 |
| *** johnmilton has quit IRC | 22:37 | |
| *** lykinsbd has joined #openstack-ansible | 22:38 | |
| *** galstrom is now known as galstrom_zzz | 22:38 | |
| *** asettle has quit IRC | 22:39 | |
| *** lykinsbd has quit IRC | 22:42 | |
| *** aslaen has quit IRC | 22:45 | |
| *** aslaen has joined #openstack-ansible | 22:46 | |
| *** aslaen has quit IRC | 22:46 | |
| *** aslaen has joined #openstack-ansible | 22:46 | |
| *** aslaen has quit IRC | 22:48 | |
| *** aslaen has joined #openstack-ansible | 22:49 | |
| *** aslaen has quit IRC | 22:49 | |
| *** raddaoui has quit IRC | 22:50 | |
| *** thorst_ has quit IRC | 22:51 | |
| *** thorst_ has joined #openstack-ansible | 22:52 | |
| *** weezS has quit IRC | 22:52 | |
| *** networkninja has quit IRC | 22:57 | |
| *** skamithi15 has quit IRC | 23:01 | |
| *** skamithi13 has joined #openstack-ansible | 23:01 | |
| *** lunarlamp is now known as mariusv | 23:01 | |
| *** weezS has joined #openstack-ansible | 23:01 | |
| *** thorst_ has quit IRC | 23:01 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!