Thursday, 2016-10-20

*** asettle has joined #openstack-ansible00:00
*** LiYuenan has joined #openstack-ansible00:04
*** asettle has quit IRC00:04
*** jmckind has joined #openstack-ansible00:12
*** jmckind has quit IRC00:20
*** dxiri has quit IRC00:28
*** wadeholler has joined #openstack-ansible00:30
*** agrebennikov has quit IRC00:31
*** agrebennikov_ has joined #openstack-ansible00:31
*** wadeholler has quit IRC00:43
*** wadeholler has joined #openstack-ansible00:44
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible: Add missing Ironic auth plugin variable  https://review.openstack.org/38894600:47
*** wadeholler has quit IRC00:47
*** wadeholler has joined #openstack-ansible00:47
*** david-lyle_ has joined #openstack-ansible00:51
*** david-lyle has quit IRC00:54
*** wadeholler has quit IRC00:54
*** gouthamr has joined #openstack-ansible00:55
*** wadeholler has joined #openstack-ansible00:55
*** wadeholler has quit IRC00:59
*** jmckind has joined #openstack-ansible01:02
*** cathrich_ has joined #openstack-ansible01:08
*** cathrichardson has quit IRC01:08
*** agrebennikov_ has quit IRC01:15
*** chandanc has joined #openstack-ansible01:17
*** chandanc has quit IRC01:23
*** thorst_ has quit IRC01:24
*** thorst_ has joined #openstack-ansible01:25
*** thorst_ has quit IRC01:33
*** javeriak has quit IRC01:36
*** maeker has quit IRC01:38
*** hughmFLEXin has joined #openstack-ansible01:38
*** Mudpuppy has joined #openstack-ansible01:47
*** Mudpuppy has quit IRC01:49
*** Mudpuppy has joined #openstack-ansible01:49
*** fops has joined #openstack-ansible01:50
jwitkoHey all,  I have pip installs failing on my galera container because they can't reach the haproxy IP on port 818101:53
jwitkoI'm wondering what I screwed up... I thought the playbooks installed HA proxy and configured a load balanced setup with the specified IP from the config?01:53
*** fops_ has quit IRC01:53
*** Mudpuppy has quit IRC01:54
jwitkonm i see I'm missing some user_vars02:02
cloudnulljwitko: did you get it to go?02:11
*** thorst_ has joined #openstack-ansible02:14
*** thorst_ has quit IRC02:14
*** thorst_ has joined #openstack-ansible02:14
*** thorst_ has quit IRC02:23
*** poopcat has quit IRC02:28
*** poopcat has joined #openstack-ansible02:28
*** chhavi has joined #openstack-ansible02:29
*** thorst_ has joined #openstack-ansible02:32
*** LiYuenan has quit IRC02:32
*** thorst_ has quit IRC02:33
*** poopcat has quit IRC02:36
jwitkoI moved on to a new error  :)02:38
*** thorst_ has joined #openstack-ansible02:39
*** thorst_ has quit IRC02:39
jwitkois there a script to generate random entries into user_secrets.yml ?02:41
openstackgerritWang Qing wu proposed openstack/openstack-ansible-os_nova: Identify virt type of PowerVM and KVM on Power  https://review.openstack.org/38857102:44
*** jcrst has quit IRC02:46
*** chhavi has quit IRC02:47
*** Hosam has quit IRC02:48
*** chandanc has joined #openstack-ansible02:49
*** poopcat has joined #openstack-ansible02:49
cooljjwitko: is pw-token-gen.py what you're looking for? http://docs.openstack.org/developer/openstack-ansible/install-guide/configure-creds.html02:49
jwitkoyes!  ty!02:51
cooljnp!02:51
openstackgerritMerged openstack/openstack-ansible-os_tempest: Use upper constraints for all tox targets  https://review.openstack.org/38844802:53
*** Mudpuppy has joined #openstack-ansible02:56
*** rmelero has quit IRC02:59
*** Mudpuppy has quit IRC03:01
*** david-lyle_ has quit IRC03:23
*** david-lyle has joined #openstack-ansible03:23
*** phalmos has quit IRC03:29
*** jcrst has joined #openstack-ansible03:34
jwitkook new error on the last playbook03:36
jwitkosetup-openstack.yml03:36
jwitkoTASK [os_keystone : Get remote venv checksum] **********************************03:36
jwitkofatal: [infra3_keystone_container-161bc34a]: FAILED! => {"changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: An unknown error occurred: ''", "redirected": false, "status": -1, "url": "http://10.89.103.25:8181/venvs/14.0.0/ubuntu/keystone-14.0.0-x86_64.checksum"}03:36
jwitkofatal: [infra1_keystone_container-4a9ca527]: FAILED! => {"changed": false, "content": "", "failed": true, "msg": "Status code was not [200]: An unknown error occurred: ''", "redirected": false, "status": -1, "url": "http://10.89.103.25:8181/venvs/14.0.0/ubuntu/keystone-14.0.0-x86_64.checksum"}03:36
jwitkook: [infra2_keystone_container-be81ce54]03:36
jwitkofailed on 2/3 nodes03:36
jwitkolooks like it just needed time to propogate?  ran it again, 2/3 passed, ran it a 3rd time, 3/3 passed03:39
*** thorst_ has joined #openstack-ansible03:40
jwitkonevermind... its still erroring on 1/303:41
jwitkoyea every time on the keystone 14.0.0 version03:44
*** thorst_ has quit IRC03:48
*** chhavi has joined #openstack-ansible03:51
jwitkofinally got all 3 to pass... just by executing again03:52
jwitkonew error though on cache03:52
jwitkofatal: [infra1_keystone_container-4a9ca527]: FAILED! => {"changed": false, "failed": true, "msg": "Source '/var/cache/keystone-14.0.0-x86_64.tgz' does not exist"}03:52
*** kamtamtun has joined #openstack-ansible04:00
*** poopcat has quit IRC04:04
*** dxiri has joined #openstack-ansible04:09
*** dxiri has quit IRC04:11
*** dxiri has joined #openstack-ansible04:12
jwitkonow the same error for glance... any ideas whats going on ?04:12
jwitko2/3 glance containers fail finding /var/cache file04:12
jwitkofatal: [infra2_glance_container-b70a3ad3]: FAILED! => {"changed": false, "failed": true, "msg": "Source '/var/cache/glance-14.0.0-x86_64.tgz' does not exist"}04:12
*** maeker has joined #openstack-ansible04:16
*** hj-hpe has quit IRC04:17
*** jmckind_ has joined #openstack-ansible04:22
*** hybridpollo has quit IRC04:22
*** jmckind has quit IRC04:23
*** weezS has joined #openstack-ansible04:25
*** chandanc has quit IRC04:25
*** jmckind_ has quit IRC04:27
*** jmckind has joined #openstack-ansible04:30
*** jmckind has quit IRC04:34
*** chhavi has quit IRC04:36
*** janki has joined #openstack-ansible04:41
*** thorst_ has joined #openstack-ansible04:48
*** markvoelker_ has quit IRC04:49
*** Hosam has joined #openstack-ansible04:50
jwitkolooks like every once in a while my haproxy is responding with no data received04:52
jwitkohttp://paste.ubuntu.com/23352068/04:52
jwitkobut retries and it works04:52
*** javeriak has joined #openstack-ansible04:53
*** thorst_ has quit IRC04:54
*** galstrom is now known as galstrom_zzz04:55
jwitkoi'm seeing a lot of "Oct 20 04:57:46 localhost haproxy[6559]: 10.89.103.166:56794 [20/Oct/2016:04:57:46.174] repo_all-front-1/1: SSL handshake failure04:58
jwitkoO" in the logs.  is there some setting I should be using to ignore ssl certs?04:58
jwitkoall of the /etc/ssl/private/haproxy.pem files have different md5sums ?05:09
*** weezS has quit IRC05:13
*** weezS has joined #openstack-ansible05:14
dxirijwitko: I am having that same problem :)05:17
dxirihaproxy seems to be intermittent05:17
dxiriare you by any chance using the same vip for internal and external?05:17
*** weezS has quit IRC05:18
*** Hosam has quit IRC05:19
*** Hosam has joined #openstack-ansible05:19
*** maeker has quit IRC05:20
*** Hosam_ has joined #openstack-ansible05:22
*** thetrav has joined #openstack-ansible05:22
thetravis setup-infrastructure idempotent?05:22
thetravI'm trying to set up automatic haproxy certificate renewal.  Looks like that's the top level task used to make it happen05:23
jwitkodxiri, I am!05:24
dxirijwitko: then same boat as me :)05:24
jwitkodxiri interesting.  should I change them?05:25
*** Hosam has quit IRC05:25
dxiriin my case, haproxy spun up only the https instance first (without any http instance) and the fix was to change the external vip to a hostname (that I set it to resolve to the same ip)05:26
jwitkooh, I already had that05:26
jwitkothe external was the hostname05:26
dxirithat got me past the setup-infrastructure.yml05:26
jwitkoand the internal was the IP of the resolved hostname05:26
dxiribut now I am stuck with this intermitent stuff05:26
dxirisometimes works in the first try, sometimes on the 7th try05:27
dxiriyep, that's exactly how I set that up as well05:27
jwitkoI am going to change internal to a different subnet05:28
jwitkoand re-run infra playbook05:28
jwitkoand then setup playbook05:28
jwitkosee what happens05:28
dxirilet's wait and see if someone can chime in on this :) I need some sleep but will check on the history and soon as I get up to work05:28
dxiriif you do that, please let me know the result :)05:28
*** markvoelker_ has joined #openstack-ansible05:30
*** LiYuenan has joined #openstack-ansible05:31
*** McMurlock1 has joined #openstack-ansible05:32
*** dxiri has quit IRC05:33
*** javeriak has quit IRC05:44
*** markvoelker_ has quit IRC05:46
*** thorst_ has joined #openstack-ansible05:52
thetravweird, it decided to skip my certificates05:53
*** thorst_ has quit IRC05:59
*** kamtamtun has quit IRC06:05
*** javeriak has joined #openstack-ansible06:08
*** dxiri has joined #openstack-ansible06:12
*** jcrst has quit IRC06:14
*** thetrav has quit IRC06:14
*** jcrst has joined #openstack-ansible06:16
*** dxiri has quit IRC06:16
*** pcaruana has joined #openstack-ansible06:18
*** Hosam_ has quit IRC06:37
*** dxiri has joined #openstack-ansible06:40
*** Hosam has joined #openstack-ansible06:40
*** Hosam has quit IRC06:41
*** Hosam_ has joined #openstack-ansible06:41
*** dxiri has quit IRC06:44
*** omiday has quit IRC06:50
*** jcrst has quit IRC06:57
*** Mudpuppy has joined #openstack-ansible06:58
*** thorst_ has joined #openstack-ansible06:58
*** jcrst has joined #openstack-ansible06:59
*** Mudpuppy has quit IRC07:02
*** cathrich_ has quit IRC07:03
*** cathrichardson has joined #openstack-ansible07:03
*** thorst_ has quit IRC07:04
*** automagically has quit IRC07:06
*** automagically has joined #openstack-ansible07:06
*** fxpester has joined #openstack-ansible07:10
*** Hosam_ has quit IRC07:36
*** haad1 has joined #openstack-ansible07:41
*** haad1 has quit IRC07:41
*** haad1 has joined #openstack-ansible07:41
*** fxpester has quit IRC07:51
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Automatically source the rc file for ansible/ansible-playbook  https://review.openstack.org/38887707:52
*** javeriak has quit IRC07:53
gaudenzMatias: I have now reported a bug on ceilometer in launchpad: https://bugs.launchpad.net/ceilometer/+bug/163514807:56
openstackLaunchpad bug 1635148 in Ceilometer "SSLError in connection from ceilometer-collector to RabbitMQ" [Undecided,New]07:56
gaudenzMatias: Is this the same issue you had? If so, can you add your information so we might find out what fixed it and backport that to mitaka.07:57
*** haad1 has quit IRC07:59
*** haad1 has joined #openstack-ansible07:59
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Adds support for disks on HP Smart Array Controllers  https://review.openstack.org/38905308:03
*** thorst_ has joined #openstack-ansible08:03
*** asettle has joined #openstack-ansible08:04
*** david-lyle_ has joined #openstack-ansible08:07
*** Hosam has joined #openstack-ansible08:07
*** Hosam_ has joined #openstack-ansible08:08
*** karimb has joined #openstack-ansible08:08
*** david-lyle has quit IRC08:09
*** thorst_ has quit IRC08:09
*** Hosam has quit IRC08:12
odyssey4meandymccr mrda noticed this? https://review.openstack.org/388946 - it seems odd that this would be required in group_vars given that this is the default and should only be used in the role08:13
andymccrodyssey4me: its in the nova role08:14
andymccri think it was added as part of the auth --> v3 change and we didn't pick up the group_vars change08:15
odyssey4meah08:15
odyssey4meI don't actually see why we have vars for the auth plugin.08:15
odyssey4meWe should just hard set it in the templates, and for the rare use-case where people want to change it - there's config template.08:16
*** markvoelker has joined #openstack-ansible08:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Work around bad libvirt-python wheel  https://review.openstack.org/38906408:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Work around bad libvirt-python wheel  https://review.openstack.org/38906408:41
openstackgerritAndy McCrae proposed openstack/openstack-ansible-galera_server: Fix MYSQLD_STARTUP_TIMEOUT for systemd  https://review.openstack.org/38906608:43
*** mpotdar has joined #openstack-ansible08:44
*** electrofelix has joined #openstack-ansible08:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts: Add ability to set upper constraints file  https://review.openstack.org/38906708:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts: Add ability to set upper constraints file  https://review.openstack.org/38906808:46
*** karimb has quit IRC08:58
*** McMurlock1 has quit IRC09:01
evrardjpandymccr: thanks for the bug cleanup :D09:02
evrardjptest-vars I mean09:03
andymccrevrardjp: ahh np! yeah should probably go through some of those bugs and close out ones that no longer apply, but will add it to my todo list09:03
evrardjpremember they are likely to be tagged already09:04
evrardjpI will help you through it if you want09:04
odyssey4meevrardjp FYI for any bugs that are marked fix released, the tags should actually be removed09:04
odyssey4mewell, that's the traditional management method that was used in OpenStack. How we decide to use tags is entirely up to us.09:05
evrardjpodyssey4me: agreed09:05
evrardjpbut if they are fix released, they are not listed as with tags IIRC09:05
evrardjpI will go over them anyway09:05
odyssey4memancdaz does this seem sane? perhaps you can verify that this is a good idea? https://review.openstack.org/38906609:05
evrardjpandymccr: are you busy on this? https://bugs.launchpad.net/openstack-ansible/+bug/163209809:05
openstackLaunchpad bug 1632098 in openstack-ansible "Liberty to Mitaka upgrade, rabbitmq queues are not cleaned up" [High,Confirmed] - Assigned to Andy McCrae (andrew-mccrae)09:05
mancdazodyssey4me I was chatting with andymccr about this earlier09:06
mancdazthis is essentially to fix a regression in behaviour09:06
mancdazit seems we decided to stop placing the defaults file on disk when we are using systemd09:06
odyssey4mesure, my only cause for concern is that systemd can't just 'source' vars like upstart does09:06
*** thorst_ has joined #openstack-ansible09:07
mancdazodyssey4me that is apparently what it does09:07
odyssey4mewell, that's what I believe based on some distant memory of a conversation about how horrible it is09:07
mancdazodyssey4me on that I don't realy know09:07
evrardjpI thought we said we're gonna trust the startup scripts delivered with packages09:07
mancdazassuming that it does, this is the correct fix09:07
*** jcrst has quit IRC09:07
evrardjpbecause it was supposed to be fine09:07
odyssey4meevrardjp sure, until the package-based startups don't work :p09:07
evrardjpshould we work with packagers? We are probably not the only ones09:08
mancdazevrardjp except that we are not placing the defaults file on disk for the package scripts to source09:08
odyssey4meevrardjp IIRC the issue we fix with all that is a known bug fixed in MariaDB 10.109:08
odyssey4mebut idk for sure09:08
evrardjpyup that's what I recall too odyssey4me09:08
mancdazodyssey4me what bug?09:09
evrardjpI thought I heard that09:09
evrardjpthe startup timing issue09:09
mancdazno09:09
odyssey4mewhen it comes to MariaDB, I believe everything that mancdaz tells me because he's awesome :)09:09
evrardjpodyssey4me: +109:09
mancdazthe problem in 10.0 was that the package provided initscript was not sourcing /etc/defaults/mysql09:09
mancdazso we added a oine to make it do that09:09
mancdaz*line09:09
evrardjpmancdaz: agreed09:10
odyssey4memancdaz ok, if it wasn't sourcing that file then perhaps we shouldn't be putting that file down and we should be implementing config somewhere else?09:10
mancdazthe problem we have with xenial is that someone decided not to place /etc/defaults/mysql on disk becuase we're using systemd09:10
mancdazthe 'fix' in mariadb10.1 is that the initscript does properly source /etc/defaults/mariadb (note, not mysql)09:10
odyssey4meie instead of forcing it to work the way we work, perhaps we should learn how it's supposed to be used and change how we use it09:10
mancdazbut either way, if you're not putting it down on disk it can't be sourced09:11
pjm6good morning all09:11
mancdazodyssey4me this *is* the way it's meant to work09:11
mancdazthe defaults file is a standard mechanism09:11
mancdazbut you have to put it there!09:11
evrardjpmancdaz: standard for ubuntu09:11
evrardjp:p09:11
odyssey4mekylek3h_away when you're in, it'd be great to know if https://review.openstack.org/389066 solves the problem you reported :)09:11
mancdazevrardjp pretty sure that's what we're using09:11
evrardjpmancdaz: so it works with xenial?09:11
mancdazevrardjp so I am told09:12
evrardjpjust saying there are many standards09:12
odyssey4meevrardjp apparently so - see gate test09:12
evrardjphttps://xkcd.com/927/09:12
*** karimb has joined #openstack-ansible09:13
mancdazodyssey4me to be fair that gate test is not actually showing you anything09:13
evrardjpmaybe I misunderstood09:13
odyssey4memancdaz yeah I know, I was being facetious09:13
evrardjpso if I understand correctly, per distribution we should define if we use /etc/defaults/mysql or mariadb09:13
odyssey4meno, not as far as I understand it09:14
odyssey4meper mariadb version perhaps09:14
odyssey4mebut not per districution09:14
*** thorst_ has quit IRC09:14
odyssey4mebah, need more coffee09:14
*** Hosam_ has quit IRC09:18
andymccrevrardjp: im looking at it now -but if you are already busy with it/have an idea feel free to get it done09:18
*** jcrst has joined #openstack-ansible09:23
mancdazandymccr https://mariadb.com/kb/en/mariadb/systemd/09:25
mancdazhttps://jira.mariadb.org/browse/MDEV-920209:25
andymccrmancdaz: odyssey4me: I just tested using systemctl and adding an "echo x > tmp.file" to ensure it was sourcing/performing the actions in the mysql init.d script - and it is.09:26
andymccrso imo that will work but perhaps there is a better way of doing it09:26
mancdazandymccr I'd say if it's sourcing that file, then it's sufficient09:27
evrardjpandymccr: I planned to look at it, and I have an upgrade ready09:27
odyssey4meandymccr yeah, if we can get confirmation of the issue being solved from kylek3h_away then great - per we can then implement an improvement later09:28
*** karimb has quit IRC09:28
andymccrevrardjp: honestly it'd be great if you took it off me and did it :) but no pressure - I'll take a look if you can't.09:28
evrardjpI will do it09:34
*** markvoelker has quit IRC09:39
*** karimb has joined #openstack-ansible09:39
*** gaudenz has quit IRC09:53
*** jcrst has quit IRC09:55
*** Hosam has joined #openstack-ansible10:00
*** Hosam has quit IRC10:05
*** winggundamth has quit IRC10:08
*** winggundamth has joined #openstack-ansible10:10
*** thorst_ has joined #openstack-ansible10:13
*** thorst_ has quit IRC10:19
*** wadeholler has joined #openstack-ansible10:35
*** markvoelker has joined #openstack-ansible10:37
*** dxiri has joined #openstack-ansible10:41
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Ensure that upper constraints are always applied  https://review.openstack.org/38822010:41
*** gaudenz has joined #openstack-ansible10:42
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Ensure that upper constraints are always applied  https://review.openstack.org/38822010:43
*** karimb has quit IRC10:44
*** karimb has joined #openstack-ansible10:46
*** dxiri has quit IRC10:47
*** wadeholler has quit IRC10:50
*** janki has quit IRC10:52
*** asettle has quit IRC10:54
*** asettle has joined #openstack-ansible10:58
*** asettle has quit IRC10:59
*** Mudpuppy has joined #openstack-ansible11:00
*** Hosam has joined #openstack-ansible11:01
*** Mudpuppy has quit IRC11:04
*** hughmFLEXin has quit IRC11:05
*** Hosam has quit IRC11:06
*** hughmFLEXin has joined #openstack-ansible11:07
*** markvoelker_ has joined #openstack-ansible11:07
*** markvoelker has quit IRC11:11
*** thorst_ has joined #openstack-ansible11:17
*** thorst_ has quit IRC11:17
*** thorst_ has joined #openstack-ansible11:17
*** Hosam has joined #openstack-ansible11:19
hughsaunderslogan-: this is excellent https://github.com/openstack/openstack-ansible/commit/d651ed70f4d2298ba1a372a6b9906e1942a871e811:19
odyssey4mehughsaunders :)11:20
*** Hosam_ has joined #openstack-ansible11:22
*** Hosam has quit IRC11:25
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Ensure that upper constraints are always applied  https://review.openstack.org/38822011:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Ensure that upper constraints are always applied  https://review.openstack.org/38822011:26
*** retreved has joined #openstack-ansible11:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_aodh: Use centralised Ansible test scripts  https://review.openstack.org/38912811:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cinder: Use centralised Ansible test scripts  https://review.openstack.org/38912911:34
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance: Use centralised Ansible test scripts  https://review.openstack.org/38913111:36
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_gnocchi: Use centralised Ansible test scripts  https://review.openstack.org/38913211:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_heat: Use centralised Ansible test scripts  https://review.openstack.org/38913311:41
*** gouthamr has quit IRC11:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_horizon: Use centralised Ansible test scripts  https://review.openstack.org/38913411:44
*** johnmilton has quit IRC11:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ironic: Use centralised Ansible test scripts  https://review.openstack.org/38913611:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone: Use centralised Ansible test scripts  https://review.openstack.org/38913711:49
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_neutron: Use centralised Ansible test scripts  https://review.openstack.org/38913911:51
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova: Use centralised Ansible test scripts  https://review.openstack.org/38914111:55
*** kylek3h_away has quit IRC11:55
*** gouthamr has joined #openstack-ansible11:55
odyssey4meandymccr we need to move os_trove into the combined deliberable now that it's part of the integrated build: https://github.com/openstack/governance/blob/master/reference/projects.yaml#L3390-L339411:56
odyssey4me*deliverable11:56
odyssey4meotherwise we won't be able to tag it as part of the milestone 1 release11:57
andymccrodyssey4me: so basically remove it from there and add it to the openstack-ansible list above?11:57
odyssey4meandymccr yep11:57
andymccrany other special thing i ahve to do, or just make the PR?11:57
odyssey4mejust push up the review with a commit message explaining that it's now part of the combined release11:57
odyssey4methat's it11:57
andymccrsweet :) thanks11:58
andymccrwill do now11:58
odyssey4meI think we'll have to tag a liberty/mitaka release today too thanks to the percona debacle11:58
odyssey4meso a bit later I'll guide you through how to go about doing that :)11:59
*** gouthamr has quit IRC11:59
*** gouthamr has joined #openstack-ansible12:00
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_rally: Use centralised Ansible test scripts  https://review.openstack.org/38914312:01
andymccrok cool yeah that'd be appreciated!12:01
andymccrI assume Trove would only be integrated from Ocata(master onwards)12:01
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_sahara: Use centralised Ansible test scripts  https://review.openstack.org/38914412:03
odyssey4meandymccr yep12:03
odyssey4meI suppose there is the option of pulling it back to Newton if we want to, but until we have scenario testing for the integrated gate, I would be resistant to the backport12:03
andymccragree12:03
andymccrhttps://review.openstack.org/#/c/389145/ think its all good12:04
odyssey4meandymccr another thing - I think any non-integrated repo should rather be tagged 'release:none' instead of 'release:cycle-trailing' - the cycle-trailing tag has much more governance around it and results in us not being able to create branches or tags whenever we want to12:06
*** johnmilton has joined #openstack-ansible12:06
andymccrodyssey4me: makes sense12:06
odyssey4meI think we might want to have more flexibility until we integrate them. Once they're integrated then we stick to the stricter governance.12:06
andymccryeah im not sure why we'd want to have strict rules on repos that arent yet ready at all12:07
odyssey4meexactly12:07
odyssey4meI tagged them with the cycle-trailing tag before really understanding the repurcussions of that12:07
odyssey4meI'd suggest doing a follow up patch (on top of the previous one) which changes the tags.12:07
andymccrodyssey4me: will do12:08
odyssey4meThe governance reviews sit around for a week before merging, so don't worry about chasing them.12:08
odyssey4meI think there's a rule around giving them a week to provide the TC opportunity to review and object. If there are no objections, then it gets merged.12:08
*** Hosam has joined #openstack-ansible12:08
*** Hosam__ has joined #openstack-ansible12:09
*** Hosam_ has quit IRC12:10
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_swift: Use centralised Ansible test scripts  https://review.openstack.org/38914712:12
*** markvoelker has joined #openstack-ansible12:12
*** Hosam has quit IRC12:13
*** Hosam__ has quit IRC12:14
*** markvoelker_ has quit IRC12:16
andymccrhttps://review.openstack.org/389150 - odyssey4me done.12:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_tempest: Use centralised Ansible test scripts  https://review.openstack.org/38915112:18
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-pip_install: Use centralised Ansible test scripts  https://review.openstack.org/38915212:20
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins: Use centralised Ansible test scripts  https://review.openstack.org/38915312:23
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-rabbitmq_server: Use centralised Ansible test scripts  https://review.openstack.org/38915512:25
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build: Use centralised Ansible test scripts  https://review.openstack.org/38915612:27
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_server: Use centralised Ansible test scripts  https://review.openstack.org/38915712:29
*** karimb has quit IRC12:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-rsyslog_client: Use centralised Ansible test scripts  https://review.openstack.org/38916212:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-rsyslog_server: Use centralised Ansible test scripts  https://review.openstack.org/38916412:34
*** Jeffrey4l has quit IRC12:35
*** thorst_ has quit IRC12:37
*** fops has quit IRC12:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-security: Use centralised Ansible test scripts  https://review.openstack.org/38916612:43
*** fops has joined #openstack-ansible12:44
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests: Use upper constraints for all tox targets  https://review.openstack.org/38916712:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests: Fix neutron_agent IP to be neutron_server IP  https://review.openstack.org/38917012:52
*** schwicht has joined #openstack-ansible12:52
odyssey4meok, andymccr with https://review.openstack.org/#/q/topic:bp/central-test-repository+status:open we have all the tests using the central repo for all the bits12:53
odyssey4mewe're going to have to be vigilent about keeping the two branches in sync with changes for as long as possible12:54
odyssey4mealso, these need attention: https://review.openstack.org/#/q/project:%255Eopenstack/openstack-ansible.*+topic:use-upper-constraints+status:open12:54
*** gaudenz has quit IRC12:55
*** dxiri has joined #openstack-ansible12:57
openstackgerritMerged openstack/openstack-ansible: Add missing Ironic auth plugin variable  https://review.openstack.org/38894612:58
openstackgerritAndy McCrae proposed openstack/openstack-ansible-os_neutron: [WIP] Move to full tempest run for Neutron  https://review.openstack.org/38745712:58
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Add missing Ironic auth plugin variable  https://review.openstack.org/38917412:58
*** dxiri has quit IRC13:02
*** schwicht has quit IRC13:03
*** schwicht has joined #openstack-ansible13:07
*** klamath has joined #openstack-ansible13:08
odyssey4meandymccr ok, for the sake of expediency let me handle today's releases - but I'm thinking that perhaps we should move the sources-branch-updater and related release scripts into the ops repo13:09
odyssey4megenerally I think it'll be easier to merge changes there and it'll keep it out of the changelog for the production release13:09
odyssey4methoughts?13:09
andymccrodyssey4me: sounds good to me13:09
*** admin0 has joined #openstack-ansible13:12
*** gaudenz has joined #openstack-ansible13:12
*** adrian_otto has joined #openstack-ansible13:14
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [docs] Remove openstack-ansible.rc sourcing  https://review.openstack.org/38918513:15
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [docs] Remove openstack-ansible.rc sourcing instruction  https://review.openstack.org/38918513:16
*** cathrich_ has joined #openstack-ansible13:17
*** cathrichardson has quit IRC13:17
*** cathrich_ is now known as cathrichardson13:19
* mhayden toots13:26
*** dxiri has joined #openstack-ansible13:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update all SHAs for 13.3.7  https://review.openstack.org/38919813:27
*** karimb has joined #openstack-ansible13:30
*** dxiri has quit IRC13:31
odyssey4meandymccr release requests: https://review.openstack.org/389191 / https://review.openstack.org/38920213:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update all SHAs for 12.2.7  https://review.openstack.org/38920413:33
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Update all SHAs for 12.2.7  https://review.openstack.org/38920413:33
*** schwicht has quit IRC13:34
*** asettle has joined #openstack-ansible13:43
andymccrodyssey4me:  on the mitaka sha bump, the repo_build and lxc_container_create repos seem wrong13:45
andymccror atleast not head of stable/mitaka13:45
*** asettle has quit IRC13:47
odyssey4meandymccr are you sure you're checking it right? https://github.com/openstack/openstack-ansible-lxc_container_create/commits/stable/mitaka shows d319546 which matches the sha bump?13:48
*** schwicht has joined #openstack-ansible13:49
odyssey4mesame goes for the repobuild role13:49
*** jheroux has joined #openstack-ansible13:51
*** galstrom_zzz is now known as galstrom13:54
andymccr      - repo: openstack/openstack-ansible-lxc_container_create13:54
andymccr        hash: a837705bb20aaddaec56dd3319b8a359c96296e413:54
andymccrthats in the PR13:54
andymccram i reading it wrong?13:54
*** galstrom is now known as galstrom_zzz13:54
*** asettle has joined #openstack-ansible13:55
andymccr      - repo: openstack/openstack-ansible-repo_build13:55
andymccr        hash: 6e2989d73907d4e01573b1fd6359b146364a75cd13:55
andymccrwhich i think should be 22c55a72b75c17df84529e09aaa51be9909c5b1513:55
*** kjw3 has joined #openstack-ansible14:01
*** adrian_otto has quit IRC14:02
mgariepygood morning every one14:03
logan-hughsaunders: thanks. :)14:06
odyssey4meandymccr ah, you're looking at the release request - not the sha bump14:08
odyssey4methe release request is on the current SHA's in https://github.com/openstack/openstack-ansible/blob/stable/mitaka/ansible-role-requirements.yml14:09
andymccrodyssey4me: ahh got you14:09
odyssey4mewhen we release, we release based on the SHA's bumped immediately after the previous release14:09
odyssey4methat way those SHA's are tested for two weeks before we release14:09
andymccrmakes sense14:09
odyssey4meon occasion we do things like https://github.com/openstack/openstack-ansible/commit/2c2cfbb1924ccb15e3a35fc4ccf7181c0fcf3bf4 to bump the role SHA's14:10
odyssey4mebut that's not the norm14:10
mgariepyany master on lxb and vxlan here ? i'm having some issue whit the network on centos7 ..14:10
mgariepywith**14:11
jwitkohey odyssey4me, I screwed up last night and deleted /etc/ssl on a controller in a fresh setup.  What can I run to re-generate that directory and its contents?14:11
odyssey4memgariepy well, if jamesdenton or evrardjp are around they could possibly help14:11
odyssey4mejwitko argh, that's not fun14:11
jwitko;\14:11
jwitkoDo I have to reinstall ubuntu ?14:11
odyssey4mejwitko you should just be able to copy the directory from another controller14:12
evrardjpjwitko: the complete ssl folder?14:12
odyssey4meI think14:12
evrardjpyeah copy from another source I guess14:12
mgariepyevrardjp, do you have some time to help me out a bit ?14:12
jwitkooh, that folder isn't unique to the machine?14:12
evrardjpstill there are certs for the host there maybe14:12
odyssey4meour SSL process copies the same private key and public key combo for the services we deploy to all the controller containers14:12
evrardjpsure I can try :D14:12
evrardjpmgariepy: ^14:12
jwitkoodyssey4me, so the reason I deleted it is i was having lots of HA proxy SSL errors in my logs14:12
odyssey4mejwitko there may be some sort of unique cert that ubuntu made, but not OSA14:13
jwitkoi noticed that the servers did not share the same haproxy.pem14:13
jwitkoit was unique on each server14:13
jwitkoand I thought they needed to be identical14:13
odyssey4meoh, that is not right14:13
jwitkomy assumption?14:13
odyssey4mehmm, what release?14:13
jwitkonewton14:13
jwitkolastest rc14:13
odyssey4meno, your assumption is right - that is a bug if it is repeatable14:14
jwitkoso if I copy over the ssl directory and re-run setup-hosts14:14
jwitkoit should end up with the same haproxy.pem everywhere?14:14
odyssey4meoh bother - this is a bug14:15
jwitkoyay i found a bug  :D14:15
odyssey4mewe generate unique certs on each server instead of generating on the first and replicating14:15
odyssey4mehttps://github.com/openstack/openstack-ansible-haproxy_server/blob/master/tasks/haproxy_ssl_configuration.yml#L7414:15
odyssey4meplease report the bug14:15
jwitkois there a quick instruction set on how to do that14:16
odyssey4mereport it with the symptoms you saw - while you do that I'll figure out a fix14:16
*** agrebennikov_ has joined #openstack-ansible14:17
odyssey4mejwitko it's not all that complicated - just pop to https://bugs.launchpad.net/openstack-ansible/+filebug14:17
jwitkothanks I'm on it14:18
*** Mudpuppy has joined #openstack-ansible14:18
jwitkowould you consider this a security vuln?14:18
jwitkono right ?14:18
odyssey4meadd the version of OSA you're using, the symptoms and any information that can help replicate the issue14:18
odyssey4meno, I don't think so personally14:18
odyssey4memhayden ^14:18
odyssey4methis is more of a functional bug in my eyes14:19
jwitkohttps://bugs.launchpad.net/openstack-ansible/+bug/163527414:19
openstackLaunchpad bug 1635274 in openstack-ansible "Certificates are not being distributed across all controllers and containers" [Undecided,New]14:19
odyssey4mejwitko I'll tweak it a bit, thanks14:19
mhaydenodyssey4me: ah, interesting find14:20
mhaydenseems more like a bug than a security vulnerability14:20
*** haad1 has quit IRC14:21
*** DanyC has joined #openstack-ansible14:21
*** DanyC has left #openstack-ansible14:21
jwitkoodyssey4me, thanks!  one more quick question, is it an issue for the external and internal LB vips to be identical ?14:22
jwitkoor for one to be a the hostname the other to be the IP the hostname resolves to14:22
odyssey4mejwitko if you use DNS names, the names must be resolvable by the keepalived/haproxy hosts14:22
odyssey4mebut yeah, no issue if they're the same14:23
*** Mudpuppy_ has joined #openstack-ansible14:23
odyssey4megenerally I'd say use DNS for external (those become the public endpoints) and IP's for internal (internal endpoints are used by services)14:23
odyssey4mebut I know that some feel that internal should also use DNS14:23
jwitkook great thats what I was doing.  but the SSL errors made me question things14:24
jwitkoI think the SSL errors were a symtpom of each server having different certs though14:24
jwitkobut after I matched all the certs haproxy was still throwing shade my way with those SSL errors14:24
*** TxGirlGeek has joined #openstack-ansible14:26
*** Mudpuppy has quit IRC14:26
jwitkois there a way to destroy all containers and start fresh?14:28
jwitkonow I'm getting SSL certificate errors eaching out to github  ;\14:28
jwitkofatal: [infra3_repo_container-00f006e4]: FAILED! => {"changed": false, "failed": true, "msg": "Failed to validate the SSL certificate for raw.githubusercontent.com:443. Make sure your managed systems have a valid CA certificate installed. You can use validate_certs=False if you do not need to confirm the servers identity but this is unsafe and not recommended. Paths checked for this14:28
jwitkoplatform: /etc/ssl/certs, /etc/pki/ca-trust/extracted/pem, /etc/pki/tls/certs, /usr/share/ca-certificates/cacert.org, /etc/ansible"}14:28
cloudnullmornings14:31
jwitkogood morning cloudnull14:32
cloudnullo/ jwitko14:32
*** Mudpuppy_ is now known as Mudpuppy14:32
*** omiday has joined #openstack-ansible14:34
*** Jeffrey4l has joined #openstack-ansible14:35
*** smatzek has joined #openstack-ansible14:35
cloudnulljwitko: did you get the ssl to github going?14:37
jwitkocloudnull, i destroyed all my containers and am running setup-hosts.yml again14:37
cloudnullok14:37
openstackgerritMerged openstack/openstack-ansible: [docs] Update all links to refer to Newton published docs  https://review.openstack.org/38862414:37
jwitkowill keep you updated14:38
*** karimb has quit IRC14:38
cloudnulli'll be around today .14:38
odyssey4meandymccr here's hoping this one will go through :) https://review.openstack.org/38923214:39
*** chris_hultin|AWA is now known as chris_hultin14:39
*** karimb has joined #openstack-ansible14:40
*** dxiri has joined #openstack-ansible14:48
*** smatzek has quit IRC14:51
openstackgerritMerged openstack/openstack-ansible: [docs] Remove openstack-ansible.rc sourcing instruction  https://review.openstack.org/38918514:54
*** phalmos has joined #openstack-ansible14:55
jwitkocloudnull, setup-hosts.yml completed successfully.  moving on to setup-infrastructure.yml14:56
mgariepyjwitko, I saw a similar issue with github certs, at the end it was a mtu issue.14:57
*** marst has quit IRC14:58
*** Jack_Iv has joined #openstack-ansible15:00
mhaydenodyssey4me / andymccr: i need to leave about halfway through our weekly meeting today -- would one of y'all be able to finish it out?15:01
*** galstrom_zzz is now known as galstrom15:01
*** weezS has joined #openstack-ansible15:02
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-haproxy_server: Ensure self-signed certificates are distributed  https://review.openstack.org/38924015:02
andymccrmhayden: sometimes i doubt your commitment to sparkle motion15:02
odyssey4mejwitko ^ patch to test out15:02
mhaydenandymccr: WAT15:02
andymccrmhayden: its a Donnie Darko ref!15:04
*** smatzek has joined #openstack-ansible15:06
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-haproxy_server: Ensure self-signed certificates are distributed  https://review.openstack.org/38924015:06
palendaeandymccr, I thought that was Little Miss Sunshine15:07
*** smatzek has quit IRC15:07
openstackgerritAndy McCrae proposed openstack/openstack-ansible-os_ironic: [WIP] Enable virtualBMC testing  https://review.openstack.org/38104915:09
andymccrpalendae: now you've got me double checking haha15:09
*** marst has joined #openstack-ansible15:10
openstackgerritMerged openstack/openstack-ansible-pip_install: Add ability to set upper constraints file  https://review.openstack.org/38809715:10
openstackgerritMerged openstack/openstack-ansible-pip_install: Add ability to set upper constraints file  https://review.openstack.org/38810915:11
*** thorst_ has joined #openstack-ansible15:11
*** thorst_ has quit IRC15:11
*** thorst_ has joined #openstack-ansible15:13
jwitkoodyssey4me, thanks!15:15
odyssey4mejwitko if it works for you, please comment and vote in the review15:16
jwitkoodyssey4me can you help me apply it to my repo and test?  do i need to delete anything or do something to test it ?15:16
odyssey4mejwitko on the deployment host do this:15:17
odyssey4mecd /etc/ansible/roles; rm -rf haproxy_server; git clone https://github.com/openstack/openstack-ansible-haproxy_server.git haproxy_server15:17
odyssey4methen: git fetch https://git.openstack.org/openstack/openstack-ansible-haproxy_server refs/changes/40/389240/2 && git cherry-pick FETCH_HEAD15:18
odyssey4methen: cd /opt/openstack-ansible/playbooks; openstack-ansible haproxy-install.yml15:18
odyssey4methen check if it worked15:19
odyssey4meyou may need to do this to regen the keys15:19
odyssey4meopenstack-ansible haproxy-install.yml -e 'haproxy_ssl_self_signed_regen=yes'15:20
jwitkoshould I do that before running the haproxy-install.yml playbook ?15:21
*** hughmFLEXin has quit IRC15:21
odyssey4mejwitko that will run the playbook and regen the keys at the same time15:21
odyssey4mebut the other bits to update the role, yes do them in that order15:21
jwitkooh, doh.15:22
jwitkoty!  will report back15:22
*** deepak_jon has joined #openstack-ansible15:22
openstackgerritMarc Gariépy proposed openstack/openstack-ansible-tests: Fix post-up and post-down script on CentOS  https://review.openstack.org/38924715:23
mgariepyandymccr, ^^ ;)15:23
andymccrmgariepy: ahh nice - i thought i had tested that one :P but obviously not that well15:24
*** kylek3h has joined #openstack-ansible15:25
openstackgerritAndy McCrae proposed openstack/openstack-ansible-os_neutron: [WIP] Fix Calico testing  https://review.openstack.org/38652615:27
*** admin0 has left #openstack-ansible15:28
*** admin0 has quit IRC15:28
*** deepak_jon1 has joined #openstack-ansible15:31
*** deepak_jon has quit IRC15:32
*** deepak_jon1 is now known as deepak_jon15:32
openstackgerritMerged openstack/openstack-ansible-tests: Ensure that pip_install uses upper-constraints  https://review.openstack.org/38870015:33
*** weezS has quit IRC15:35
*** hj-hpe has joined #openstack-ansible15:37
openstackgerritMerged openstack/openstack-ansible-os_heat: Use centralised Ansible test scripts  https://review.openstack.org/38913315:38
*** karimb has quit IRC15:43
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: Security: Add tasks for RHEL-07-010010  https://review.openstack.org/38394315:44
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: Security: Add tasks for RHEL-07-010020  https://review.openstack.org/38558315:44
openstackgerritMajor Hayden proposed openstack/openstack-ansible-security: Security: Add tasks for RHEL-07-010260  https://review.openstack.org/38558715:44
*** deepak_jon1 has joined #openstack-ansible15:45
*** karimb has joined #openstack-ansible15:46
jwitkook odyssey4me, setup-infrastructure has completed successfully.  currently all servers hold haproxy.pem that I generated and distributed.  I will execute your commands now and see if the md5 sum is new and matches on all servers afterwards.15:47
*** deepak_jon has quit IRC15:47
*** deepak_jon1 is now known as deepak_jon15:47
kylek3handymccr: Hey thanks for patch for galera_server.  I'm tried to recreate the issue but hit other problems.  Now the host system is being reloaded.  I probably won't be able to get to another recreate until at least Monday.15:48
jwitkoodyssey4me, looks like it worked15:49
jwitkokeys are new and identical15:49
andymccrkylek3h: no problem! hopefully that solves it - I believe it should, we hit the same bug in trusty a while back, but i think we thought the systemd files worked slightly differently - it turns out it's still using the old mysql init file15:51
*** kylek3h has quit IRC15:51
*** DanyC has joined #openstack-ansible15:53
*** DanyC has quit IRC15:55
jwitkoodyssey4me, I left a comment on the ticket15:55
mhaydenOpenStack-Ansible weekly meeting coming up soon in #openstack-meeting-4! :)15:55
mhayden^^ cloudnull, mattt, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, Sam-I-Am, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak, automagically, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, v1k0d3n, severion,15:55
mhaydenbgmccollum, darrenc, JRobinson__, asettle, colinmcnamara, thorst, adreznec, eil397, qwang,nishpatwa_15:55
*** DanyC has joined #openstack-ansible15:56
palendaemhayden, I'm gonna edit the page real quick15:56
mhaydenpalendae: THE AGENDA HAS BEEN SET FIRMLY IN STONE15:56
mhaydenIT IS IMMUTABLE15:56
* mhayden cackles15:56
*** karimb has quit IRC15:56
palendaeIt sure is - that's been the agenda for the last 3 weeks :)15:56
asettlemhayden: yeah yeah yeah15:56
mhaydenoh burn15:57
andymccrhahaha15:57
palendaeOk my edit's done15:57
jwitkodamn, even after implementing the SSL cert fix I am still seeing SSL handshake failures come across haproxy15:58
jwitkoOct 20 15:57:46 localhost haproxy[20172]: 10.89.103.147:46118 [20/Oct/2016:15:57:46.447] repo_cache-front-1/1: SSL handshake failure15:58
* asettle raises hand15:58
asettlemhayden:15:58
asettleis it going to be a logn one or a short one?15:58
*** DanyC has quit IRC15:58
asettleI have a craving for a chicken schnitzel15:58
palendaeLooked short to me15:58
mhaydenasettle: i have to leave at half past15:59
asettleExcellent. I c'mon for you schnitty15:59
mhaydenso i hope it's short!15:59
asettlecomin'*15:59
asettlemhayden: that's what she said?15:59
palendae>.>15:59
mhaydenasettle: i am so offended15:59
* asettle exist stage left15:59
asettleexits*15:59
asettleomg I can't even type15:59
asettleWho hired me?15:59
* mhayden toots15:59
*** david-lyle_ is now known as david-lyle16:01
*** BjoernT has joined #openstack-ansible16:02
openstackgerritAndy McCrae proposed openstack/openstack-ansible-os_ironic: [WIP] Enable virtualBMC testing  https://review.openstack.org/38104916:02
mgariepycan I get some review on this please ? :D https://review.openstack.org/#/c/389247/16:03
openstackgerritMerged openstack/openstack-ansible-os_aodh: Use centralised Ansible test scripts  https://review.openstack.org/38912816:05
*** TxGirlGeek has quit IRC16:07
*** TxGirlGeek has joined #openstack-ansible16:07
*** pjm6 has quit IRC16:07
*** kjw3 has quit IRC16:10
*** pjm6 has joined #openstack-ansible16:11
*** TxGirlGeek has quit IRC16:12
*** TxGirlGeek has joined #openstack-ansible16:13
jwitkohey guys, I can not get passed the setup-openstack.yml playbook.  It looks like a couple steps that are dependant on the haproxy VIP for checking venvs are consistently getting SSL errors in HAProxy and the ansible step fails.  I've created http://paste.ubuntu.com/23354425/ to show you16:14
jwitkothis is 14.0.0.rc416:14
*** weezS has joined #openstack-ansible16:15
*** kjw3 has joined #openstack-ansible16:15
*** TxGirlGeek has quit IRC16:15
*** TxGirlGeek has joined #openstack-ansible16:15
stevellejwitko: using any overrides for the keystone service (nginx or uwsgi enabled?) or providing your own cert in overrides?16:16
jwitkostevelle, I am using no overrides and not providing my own certs.16:17
jwitkoI had an issue where the certs were not being distributed but odyssey4me submitted a patch that fixed that issue16:17
jwitkoI can also see the issue in a simple wget, although wget retries a bunch of times16:18
*** TxGirlGeek has quit IRC16:18
*** openstackgerrit has quit IRC16:18
*** TxGirlGeek has joined #openstack-ansible16:19
*** maeker has joined #openstack-ansible16:19
*** openstackgerrit has joined #openstack-ansible16:19
Adri2000trying to deploy ceilometer with gnocchi as a backend in newton, it seems ceilometer still wants to connect to a mongo database, am I correct?16:19
openstackgerritchandanc proposed openstack/openstack-ansible-os_neutron: Add support for Neutron FWaaS v2  https://review.openstack.org/38854816:21
openstackgerritLogan V proposed openstack/openstack-ansible-pip_install: Fix same-host offline install race  https://review.openstack.org/38926916:21
*** phalmos has quit IRC16:24
openstackgerritMerged openstack/openstack-ansible: Automatically source the rc file for ansible/ansible-playbook  https://review.openstack.org/38887716:26
openstackgerritMerged openstack/openstack-ansible: Work around bad libvirt-python wheel  https://review.openstack.org/38906416:26
openstackgerritMerged openstack/openstack-ansible: Passing eth0 to br-vlan  https://review.openstack.org/38893216:26
openstackgerritMerged openstack/openstack-ansible: Do not extend lists in AIO config  https://review.openstack.org/38893316:26
*** TxGirlGeek has quit IRC16:27
*** TxGirlGeek has joined #openstack-ansible16:27
*** asettle has quit IRC16:29
*** TxGirlGeek has quit IRC16:30
palendaeIf people have the time, https://review.openstack.org/#/c/383920/ adds some more checking to the inventory check mode16:33
*** hughmFLEXin has joined #openstack-ansible16:34
Adri2000bug #1628952 - that seems to be what I was talking about16:38
openstackbug 1628952 in openstack-ansible "os_ceilometer role missing functional gate testing of gnocchi storage option" [Medium,In progress] https://launchpad.net/bugs/1628952 - Assigned to Travis Truman (travis-truman)16:38
*** TxGirlGeek has joined #openstack-ansible16:39
jwitkodamn I can not get past this, I even attempted to add retries but its not retrying on failure with the uri module16:39
*** hughmFLEXin has quit IRC16:40
*** hughmFLEXin has joined #openstack-ansible16:41
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Work around bad libvirt-python wheel  https://review.openstack.org/38893416:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [docs] Remove openstack-ansible.rc sourcing instruction  https://review.openstack.org/38927416:47
openstackgerritMerged openstack/openstack-ansible-lxc_container_create: Use centralised Ansible test scripts  https://review.openstack.org/38879216:48
*** kjw3 has quit IRC16:59
*** mgariepy has quit IRC16:59
*** TxGirlGeek has quit IRC17:00
*** TxGirlGeek has joined #openstack-ansible17:00
openstackgerritLogan V proposed openstack/openstack-ansible-lxc_hosts: Restore configurability of container resolvers  https://review.openstack.org/38927617:00
openstackgerritJimmy McCrory proposed openstack/openstack-ansible-galera_server: Remove duplicate when conditions  https://review.openstack.org/38927817:00
*** hughmFLE_ has joined #openstack-ansible17:01
openstackgerritJimmy McCrory proposed openstack/openstack-ansible-galera_server: Remove duplicate when conditions  https://review.openstack.org/38927817:01
*** hughmFLEXin has quit IRC17:03
openstackgerritSteve Lewis (stevelle) proposed openstack/openstack-ansible-os_ceilometer: Roll-up of multiple fixes for os_ceilometer role  https://review.openstack.org/38569717:05
openstackgerritMerged openstack/openstack-ansible-galera_server: [DOCS] Added release-name as a watermark to Docs.  https://review.openstack.org/38816317:09
openstackgerritMerged openstack/openstack-ansible-os_keystone: Simplify pip options/constraints mechanism  https://review.openstack.org/38823017:09
*** karimb has joined #openstack-ansible17:14
*** karimb has quit IRC17:15
*** karimb has joined #openstack-ansible17:16
*** hughmFLE_ has quit IRC17:16
*** karimb has quit IRC17:17
*** karimb has joined #openstack-ansible17:17
*** phalmos has joined #openstack-ansible17:18
*** hughmFLEXin has joined #openstack-ansible17:19
*** hughmFLEXin has quit IRC17:22
*** hughmFLEXin has joined #openstack-ansible17:23
openstackgerritLogan V proposed openstack/openstack-ansible-lxc_hosts: Restore configurability of container resolvers  https://review.openstack.org/38927617:30
openstackgerritMerged openstack/openstack-ansible-plugins: Remove deprecated first_available_file  https://review.openstack.org/38884517:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [TEST] Update to Ansible v2.1.3.0-0.1.rc1  https://review.openstack.org/38200117:40
*** omiday has quit IRC17:43
jwitkoHey if your guys meeting is over and anyone is around I'm a bit stuck on these intermittent SSL handshake failures coming across my HAProxy logs and stopping setup-openstack.yml from succeeding17:43
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: [TEST] Update to Ansible v2.2.0.0-0.2.rc2  https://review.openstack.org/38929217:44
jwitkothere was a guy last night experiencing the same issue as well17:44
jwitkothought the SSL cert fix would get rid of them but it doesn't look like that is true17:45
openstackgerritweezer su proposed openstack/openstack-ansible-ops: Enhanced the setup-cobbler.sh  https://review.openstack.org/38929617:46
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer: [DOCS] Added release-name as a watermark to Docs.  https://review.openstack.org/38813417:48
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer: Remove 'ignore_errors: true' in favor of 'failed_when: false'  https://review.openstack.org/38680217:48
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer: Use upper constraints for all tox targets  https://review.openstack.org/38842917:48
*** alij has joined #openstack-ansible17:49
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer: Changed the home-page link  https://review.openstack.org/38222017:49
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer: Make ceilometer sample interval configurable  https://review.openstack.org/38205717:49
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer: Update paste, policy and rootwrap configurations 2016-10-12  https://review.openstack.org/38554317:50
odyssey4mejmccrory if you have a little free time, https://review.openstack.org/#/q/topic:bp/central-test-repository+status:open needs an extra eye :)17:52
odyssey4mealso https://review.openstack.org/#/q/project:%255Eopenstack/openstack-ansible.*+topic:use-upper-constraints+status:open17:52
jmccrorysure, i'll start going through them17:53
dxiriguys, I am having a problem with haproxy where its kind of intermitent and making the playbook run fail cause it fails to retrieve some files17:56
*** markvoelker has quit IRC17:56
dxirireading trought history, its the same exact problem as jwitko17:57
dxiri*trough17:57
cloudnullssl related?17:57
evrardjpjwitko: dxiri is it an horizon+haproxy issue?18:00
dxiricloudnull: I haven't done anything regarding ssl, its just the default config18:00
dxiriand it seems the task tries over plain http and not on https18:00
dxiriif I do a wget it works, but sometimes on the first try, other times upon retrying18:01
jwitkohere is an example of the playbook failure and the resutling /var/log/haproxy.log entry18:01
jwitkohttp://paste.ubuntu.com/23354425/plain/18:01
jwitkoat first I thought it was because all the haproxy.pem files were different, but we fixed that here https://review.openstack.org/#/c/389240/218:02
jwitkoand it is still happening18:02
openstackgerritLogan V proposed openstack/openstack-ansible-lxc_hosts: Restore configurability of container resolvers  https://review.openstack.org/38927618:02
dxirievrardjp: I don't believe horizon has anything to do here yet, is it even configured at this point?18:02
*** karimb has quit IRC18:03
openstackgerritMerged openstack/openstack-ansible-os_ceilometer: Roll-up of multiple fixes for os_ceilometer role  https://review.openstack.org/38569718:03
dxiriits the repo http://172.25.190.251:8181/venvs/14.0.0/ubuntu/glance-14.0.0-x86_64.checksum18:03
openstackgerritMerged openstack/openstack-ansible-lxc_hosts: Add ability to set upper constraints file  https://review.openstack.org/38906718:03
jwitkoAye I do not believe this is horizon related,  most errors are coming at the repo level on port 818118:03
dxiriabove its a example url I took from the task18:03
evrardjpdxiri: just asking, I don't know where your problem is yet :p18:03
dxirievrardjp: sure, I am just giving as much info as possible :D18:04
evrardjpcan one of you show the issue?18:04
dxirijwitko pasted it above18:04
evrardjpok18:04
evrardjpcould you drop your haproxy config file somewhere?18:05
*** jcrst has joined #openstack-ansible18:05
evrardjpI'm surprised it worked for the first and not for the other ones18:05
evrardjpdxiri: you got the same issue?18:05
dxiriyes18:06
evrardjpis lsyncd running?18:06
evrardjp(just checking everything to make sure the most standard errors are verified)18:06
jwitkoevrardjp, it is intermittent (the SSL handshake failure)  it will happen in a wget attempt as well18:06
jwitkobut wget handles failures beter than uri module18:07
jwitkoand retries18:07
dxirilsyncd running where?18:07
dxirijwitko: why SSL handshake? url is fetched using plain http18:07
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is being restarted now in an attempt to resolve some mismatched merge states on a few changes, but should return momentarily.18:07
jwitkoevrardjp the haproxy config would be the default install of a 5 node cluster18:07
evrardjpdxiri: on the repo container nodes18:07
jwitkooh wow...18:07
jwitkothats a good point18:07
dxiri:P18:07
jwitkoi mean you can literally see though18:07
evrardjpjwitko: I'd be happy if you could show it to me18:07
jwitkothe failure is directly related18:08
evrardjpjwitko: well not 100%18:08
evrardjpdepending on how haproxy is configured it has to (or doesn't have to) terminate connections18:09
dxirievrardjp: here: http://pastebin.com/xNnqgYZr18:09
dxirihaproxy config18:09
evrardjpthanks18:09
openstackgerritMerged openstack/openstack-ansible-lxc_hosts: Add ability to set upper constraints file  https://review.openstack.org/38906818:09
dxirilsyncd:    Active: active (exited) since Wed 2016-10-19 23:44:40 UTC; 18h ago18:10
dxirilooks running ok18:10
evrardjpdxiri: thanks18:10
*** javeriak has joined #openstack-ansible18:10
evrardjpwell 18h ago is far, but let's say it's running for the example18:10
evrardjpcould you curl into 172.25.190.222:8181/venvs/14.0.0/ubuntu/glance-14.0.0-x86_64.checksum correctly ?18:11
*** phalmos has quit IRC18:11
*** electrofelix has quit IRC18:11
evrardjpor just 172.25.190.222:8181/venvs/18:11
dxiriyou mean 251?18:12
evrardjpfrom the inside18:12
dxiri172.25.190.25118:12
evrardjpnope18:12
dxiriwhere did you got 222 from?18:12
evrardjpfrom your paste :p18:12
dxirilol18:12
dxirihaven't seen it18:12
evrardjpI told you I was doing on a step by step18:13
*** alij has quit IRC18:13
*** alij has joined #openstack-ansible18:13
jwitkoevrardjp, my config is the same as dxiri18:13
jwitkojust fyi18:13
jwitkolooked it over18:13
jwitkoexcept IPs of course18:13
evrardjpgood 2 birds 1 stone18:13
openstackgerritMerged openstack/openstack-ansible-os_glance: Add os-brick to glance_pip_packages  https://review.openstack.org/38801418:13
evrardjpnobody found the solution yet, right?18:13
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance: Add os-brick to glance_pip_packages  https://review.openstack.org/38930518:14
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance: Add os-brick to glance_pip_packages  https://review.openstack.org/38930618:14
dxiri(I ran the playbook again before you told me to curl, will do as soon as it fails)18:14
evrardjpOMG I just realized for one work I'm doing on mitaka I used a module rabbitmq_queue that is available starting on ansible 2.018:15
jwitkoevrardjp,  please see here18:15
jwitkohttp://paste.ubuntu.com/23354996/18:15
evrardjpFAIL18:15
*** karimb has joined #openstack-ansible18:15
evrardjpjwitko: could you show which ip does that refer to?18:16
evrardjpjwitko: is 10.89.103.25 your repo container ip?18:16
jwitkothat is my VIP18:16
*** phalmos has joined #openstack-ansible18:16
jwitkohaproxy hosted VIP18:16
evrardjpthen please do on the container :D18:16
evrardjpfirst container IP, then internal VIP then external VIP18:17
evrardjpprogressive :p18:17
openstackgerritLogan V proposed openstack/openstack-ansible-lxc_hosts: Restore configurability of container resolvers  https://review.openstack.org/38927618:17
evrardjpis there someone here familiar with rabbitmqadmin ?18:17
dxiriroot@sjociara03:/opt/openstack-ansible/playbooks# curl 172.25.190.222:8181/venvs/14.0.0/ubuntu/glance-14.0.0-x86_64.checksum18:18
dxiria39a1e2ea6281d9c86a39f275149dcf72c382d8f18:18
dxirioh look a that...it got past that part this time...I got really lucky...haha18:18
evrardjpdxiri: so it works, and you never have problems when accessing directly on the repo container18:18
*** phalmos has quit IRC18:18
dxiriworks everytime yes, no retries18:19
*** phalmos has joined #openstack-ansible18:19
dxiribut when passing trough haproxy18:19
evrardjpdxiri: ok18:19
dxiriroot@sjociara03:/opt/openstack-ansible/playbooks# curl 172.25.190.251:8181/venvs/14.0.0/ubuntu/glance-14.0.0-x86_64.checksum18:19
dxiricurl: (52) Empty reply from server18:19
dxiriroot@sjociara03:/opt/openstack-ansible/playbooks# curl 172.25.190.251:8181/venvs/14.0.0/ubuntu/glance-14.0.0-x86_64.checksum18:19
dxiria39a1e2ea6281d9c86a39f275149dcf72c382d8f18:19
dxiriand the next time works18:19
evrardjphehe18:19
dxirisometimes its more than 1 try, sometimes works right away, sometimes its about 7 tries :P18:20
evrardjpdxiri: yup not surprised18:20
evrardjplet's do some haproxy then18:20
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests: Use upper constraints for all tox targets  https://review.openstack.org/38916718:21
evrardjpwell I'm gonna pollute the chan if I continue18:21
jwitkosame results here18:21
*** phalmos_ has joined #openstack-ansible18:24
*** phalmos has quit IRC18:27
openstackgerritMerged openstack/openstack-ansible: Implement upper constraints for Ansible bootstrap  https://review.openstack.org/38805118:31
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible: Ensure that upper constraints are always applied  https://review.openstack.org/38822018:36
evrardjpjwitko: dxiri18:41
evrardjpI've got your solution18:42
jwitko:D!18:42
jwitko!!!18:42
openstackjwitko: Error: "!!" is not a valid command.18:42
evrardjp:D18:42
*** karimb has quit IRC18:42
evrardjpyour external vip and your internal vip are identical18:42
jwitkoyes18:42
jwitkoI was told this is not an issue18:42
evrardjpwell it is18:42
jwitko:O18:42
evrardjpyou cannot bind two times on the same ip/port18:43
evrardjpand then haproxy doesn't know what to do18:43
evrardjp:p18:43
evrardjphave two different ips18:43
evrardjpOR18:43
evrardjpbetter18:43
*** karimb has joined #openstack-ansible18:43
evrardjphave a real external net18:43
evrardjpand have boundaries18:44
dxiriwell...shit :)18:44
dxirican I use an ip from the lxcbridge?18:44
dxiri    inet 10.0.3.1/24 brd 10.0.3.255 scope global lxcbr018:44
evrardjpnope18:44
dxirisay...that one? as internal?18:44
evrardjpfor the ease of use18:44
evrardjpinternal is part of the br-mgmt18:44
evrardjpeasier18:44
*** deepak_jon has quit IRC18:45
evrardjpit avoids many redirections18:45
evrardjpexternal you could use an ip on the same net if you want18:45
evrardjpit doesn't really make sense in terms of security, but I'm not your CSO, right :p18:45
dxirimmmm, and can I just leave the external_vip blank? I can reach the IP I set on internal from anywhere on my network18:45
evrardjpso you could 2 2 different ips of mgmt net18:46
dxiriso I don't really have a need for the external18:46
dxiriyou mean have 2 ips on br-mgmt?18:46
evrardjpyup18:46
evrardjpone for the internal facing18:46
evrardjpand one for the external facing18:46
evrardjpand then reach from the external facing for external facing operations18:46
*** markvoelker has joined #openstack-ansible18:46
dxiriand when VMs need to use the bridge, which IP will they use to go out the bridge?18:46
evrardjpthat's entirely up to you18:47
evrardjpthat's how you define your networks18:47
*** markvoelker has quit IRC18:47
*** markvoelker has joined #openstack-ansible18:47
evrardjpI'd suggest you to isolate tenant traffic and openstack control plane traffic18:47
jwitkoevrardjp, so do you know what I have to do to make this work now?  I've edited the config file with the internal vip18:47
jwitkodo I need to create all containers?  or just run the infra setup again ?18:48
evrardjpwell there is one big problem jwitko dxiri of the change of internal_lb_vip_address or external_lb_vip_address18:48
evrardjpwe don't force a change of endpoints18:48
evrardjpso the easier for you18:49
evrardjpis to redeploy18:49
evrardjpand redestroy every container :p18:49
evrardjpyou could do it smarter, but it's tedious18:49
evrardjp:p18:49
jwitkoso18:49
jwitkoopenstack-ansible lxc-containers-destroy.yml,  followed by setup-hosts, setup-infra, setup-openstack ?18:49
evrardjpjwitko: could you show me your o_u_c ?18:49
evrardjpyup18:50
jwitkosure one moment18:50
evrardjpoh also18:50
evrardjpdelete your /etc/haproxy/conf.d folder18:50
jwitkoon the bare metal controllers right ?18:51
jwitkothe haproxy_infra_hosts ?18:51
evrardjpon tha haproxy hosts yes18:51
evrardjpansible -m shell -a "rm -rf /etc/haproxy/conf.d" haproxy_all18:51
evrardjpif I'm not mistaken18:51
jwitkothat is correct18:52
*** tobias_ has joined #openstack-ansible18:52
odyssey4memhayden with https://review.openstack.org/388799 merged it might be time to do an osa-differ and tweet it :) ?18:53
odyssey4meunfortunately not all the emails have come through to the announce list - they're working on it18:54
openstackgerritMerged openstack/openstack-ansible-security: Use centralised Ansible test scripts  https://review.openstack.org/38916618:54
dxiriwait up...in my case18:55
dxiriI had a hostname, what If I just change the record to point to the new IP and restart haproxy?18:55
dxirishouldn't that do it?18:55
openstackgerritMerged openstack/openstack-ansible-rsyslog_client: Use centralised Ansible test scripts  https://review.openstack.org/38916218:56
openstackgerritMerged openstack/openstack-ansible-rsyslog_server: Use centralised Ansible test scripts  https://review.openstack.org/38916418:56
*** hybridpollo has joined #openstack-ansible18:56
dxiriso ansible -m shell -a "systemctl restart haproxy" haproxy_all18:56
evrardjpodyssey4me: cool18:57
openstackgerritMerged openstack/openstack-ansible-plugins: Use centralised Ansible test scripts  https://review.openstack.org/38915318:57
evrardjpdxiri: yes18:58
evrardjpthat would be fine18:58
evrardjpdon't forget to change your hosts file etc18:58
dxiriI have my infra hosts pointing to a DNS server on the same subnet18:59
*** tobias_ has quit IRC18:59
evrardjpI have to go guys18:59
evrardjpbut I think you're closer to a solution18:59
dxirievrardjp: thanks a lot dude!18:59
jwitkoty, i'm redeploying now with new IP18:59
jwitkoin the same subnet18:59
evrardjpdrop message on the channel here, I'll read that tomorrow morning18:59
jwitkowill let you know how it goes19:00
evrardjpjwitko: cool19:00
openstackgerritMerged openstack/openstack-ansible-os_swift: Use centralised Ansible test scripts  https://review.openstack.org/38914719:01
openstackgerritMerged openstack/openstack-ansible-repo_server: Use centralised Ansible test scripts  https://review.openstack.org/38915719:02
dxiri [WARNING]: Host file not found: /etc/ansible/hosts19:02
dxiri [WARNING]: provided hosts list is empty, only localhost is available19:02
dxiriwhat am I missing here?19:02
dxiritrying to run the ansible command above19:02
openstackgerritweezer su proposed openstack/openstack-ansible-ops: Remove the expired key and opensuse repo after cobbler being installed  https://review.openstack.org/38890619:05
*** asettle has joined #openstack-ansible19:05
openstackgerritMerged openstack/openstack-ansible-repo_build: Use centralised Ansible test scripts  https://review.openstack.org/38915619:06
openstackgerritMerged openstack/openstack-ansible-os_rally: Use centralised Ansible test scripts  https://review.openstack.org/38914319:08
odyssey4medxiri that is cosmetic for newton onwards - all config is done via environment variables instead of an ansible.cfg file19:08
odyssey4mehmm, if there's a way to turn those off then we should include the env vars to do that19:09
openstackgerritMerged openstack/openstack-ansible-rabbitmq_server: Use centralised Ansible test scripts  https://review.openstack.org/38915519:09
*** jcrst has quit IRC19:10
dxiriodyssey4me: so the command actually went trough?19:10
*** TxGirlGeek has quit IRC19:10
dxiricause I got no ansible output back19:10
dxirino success/failure...not nothing19:10
*** TxGirlGeek has joined #openstack-ansible19:10
odyssey4medxiri that sounds very odd19:11
dxiriI am on the deployment host trying to run ansible -m shell -a "systemctl restart haproxy" haproxy_all19:11
dxiriand I am inside /opt/openstack-ansible/playbooks19:11
Matiasdxiri: add -i inventory to the command line19:12
mhaydenodyssey4me: oh nice19:12
odyssey4medxiri before executing that, execute: . /usr/local/bin/openstack-ansible.rc19:13
odyssey4mewe have a patch coming in to resolve that19:13
mhaydenodyssey4me: a diff from 13.0.0 to 14.0.0? that'd be, uh, large19:13
odyssey4memhayden yeah, apparently that's why the release announcement got held ;)19:13
dxiriinfra1 | SUCCESS | rc=0 >>19:13
dxirithere we go!19:13
odyssey4memebbe there's a way to slim it down to some essentials19:13
mhaydenodyssey4me: i'll give it a try :)19:13
*** TxGirlGeek has quit IRC19:13
mhaydenlet's hope i don't start raiding swap memory on my system19:14
*** TxGirlGeek has joined #openstack-ansible19:14
openstackgerritMerged openstack/openstack-ansible-os_gnocchi: Use centralised Ansible test scripts  https://review.openstack.org/38913219:14
openstackgerritMerged openstack/openstack-ansible-ops: Enhanced the setup-cobbler.sh  https://review.openstack.org/38929619:14
openstackgerritMerged openstack/openstack-ansible-ops: Add remove known_hosts in the instruction for Multi-Node AIO  https://review.openstack.org/38889819:14
openstackgerritMerged openstack/openstack-ansible-os_glance: Use centralised Ansible test scripts  https://review.openstack.org/38913119:15
openstackgerritMerged openstack/openstack-ansible-tests: Use upper constraints for all tox targets  https://review.openstack.org/38916719:15
openstackgerritMerged openstack/openstack-ansible-tests: Fix neutron_agent IP to be neutron_server IP  https://review.openstack.org/38917019:15
*** TxGirlGeek has quit IRC19:16
*** TxGirlGeek has joined #openstack-ansible19:16
odyssey4memhayden w00t! http://lists.openstack.org/pipermail/openstack-announce/2016-October/001855.html19:16
mhaydenwow19:17
mhaydenlengthy19:17
openstackgerritMerged openstack/openstack-ansible-os_keystone: Use centralised Ansible test scripts  https://review.openstack.org/38913719:18
*** asettle has quit IRC19:18
mhaydenodyssey4me: done tweeted it19:18
odyssey4memhayden done retweeted your tweet19:18
mhaydengonna run down my Pebble battery19:19
openstackgerritMerged openstack/openstack-ansible-os_neutron: Use centralised Ansible test scripts  https://review.openstack.org/38913919:19
*** johnmilton has quit IRC19:21
*** alij has quit IRC19:21
mhaydenodyssey4me: oh, got goosed with the openstack_other.yml not being there :)19:22
odyssey4memhayden eh?19:22
openstackgerritMerged openstack/openstack-ansible-os_cinder: Use centralised Ansible test scripts  https://review.openstack.org/38912919:23
mhaydeni rely on playbooks/defaults/repo_packages/openstack_other.yml19:23
mhaydenbut now that's split up a bit19:23
odyssey4memhayden ah, yes - you should just iterate through playbooks/defaults/repo_packages/*.yml ?19:23
* odyssey4me is going to stop computering now19:23
odyssey4merelease done, mic dropped - andymccr it's your baby now :p19:23
*** asettle has joined #openstack-ansible19:24
mhaydenawww19:24
mhaydentime for some scotch, odyssey4me? :)19:24
odyssey4memhayden most definitely19:24
*** jheroux has quit IRC19:24
odyssey4menight all - catch y'all tomorrow19:24
*** TxGirlGeek has quit IRC19:24
dxiriodyssey4me: see ya19:24
jwitkoare direct pull requests to github ignored for sauce-ansible ?19:25
*** TxGirlGeek has joined #openstack-ansible19:25
mhaydeni heard andymccr talking about converting this project into openstack-cfengine19:26
andymccrmhayden: i trusted you19:26
openstackgerritMerged openstack/openstack-ansible-os_ceilometer: Changed the home-page link  https://review.openstack.org/38222019:26
mhaydenandymccr: first mistake19:26
andymccrnow i know :(19:26
openstackgerritMerged openstack/openstack-ansible-os_tempest: Use centralised Ansible test scripts  https://review.openstack.org/38915119:27
openstackgerritMerged openstack/openstack-ansible-os_nova: Use centralised Ansible test scripts  https://review.openstack.org/38914119:32
openstackgerritMerged openstack/openstack-ansible-os_ironic: Use centralised Ansible test scripts  https://review.openstack.org/38913619:35
openstackgerritMerged openstack/openstack-ansible-tests: Fix post-up and post-down script on CentOS  https://review.openstack.org/38924719:35
*** phalmos_ has quit IRC19:36
*** jcrst has joined #openstack-ansible19:37
jwitkoHey guys, does HA proxy not set up a floating IP for the internal VIP address as it does for the external?19:38
bazSo what's the desired default for Horizon (Newton) image creation access? I ask because the out-of-box OSA has missing config in /etc/ansible/roles/os_horizon/templates/horizon_local_settings.py.j2 that prevents image creation.19:38
*** Jack_Iv has quit IRC19:38
*** johnmilton has joined #openstack-ansible19:39
openstackgerritMerged openstack/openstack-ansible-os_tempest: [DOCS] Added release-name as a watermark to Docs.  https://review.openstack.org/38811119:40
bazjwitko you need to define both in internal and external in your global_overrides and add the config mentioned in the openstack_user_config example to your user_variables.yml19:41
jwitkowait... I need the global_overrides config in two places?19:41
bazand of course they need to be in your excluded address range since a container doesn't try to pick it.19:41
bazglobal_overrides is in openstack_user_config.yml19:42
jwitkoso I have this in user_variables.yml19:43
jwitkohaproxy_keepalived_external_vip_cidr: "{{external_lb_vip_address}}/32"19:43
jwitkohaproxy_keepalived_internal_vip_cidr: "{{internal_lb_vip_address}}/32"19:43
jwitkohaproxy_keepalived_external_interface: br-mgmt19:43
jwitkohaproxy_keepalived_internal_interface: br-mgmt19:43
jwitkokeepalived_use_latest_stable: True19:43
bazthat's what I have in my testing deploy as well.19:44
bazhowever you need in openstack_user_config.yml you need:19:44
bazglobal_overrides:19:44
baz   internal_lb_vip_address: 1.2.3.419:45
*** admin0 has joined #openstack-ansible19:45
baz  external_lb_vip_address: 5.6.7.819:45
jwitkoyes19:45
jwitkoi have that as well19:45
jwitkoglobal_overrides:19:46
jwitko  internal_lb_vip_address: 10.89.103.2419:46
jwitko  external_lb_vip_address: openstack.lab.mgmt.domain.net19:46
jwitkoand that hostname resolves to 10.89.103.2519:46
jwitkohowever haproxy only creates and listens on the floating .25 IP,  there is no IP on the servers listening for 10.89.103.2419:46
bazit should be VRRP'ing both of those addresses with keepalived19:46
bazand there should be bind statements in /etc/haproxy.cfg for each of those addresses.19:47
jwitkothere are bind statements in haproxy.cfg19:47
*** spotz_zzz is now known as spotz19:47
jwitkofrontend repo_all-front-219:48
jwitko    bind 10.89.103.24:818119:48
bazand keepalived is alive? and do you see VRRP for both vips with tcpdump?19:48
jwitkokeepalived is running19:49
jwitkostatus is happy19:49
jwitkotcp        0      0 10.89.103.24:8181       0.0.0.0:*               LISTEN      1803/haproxy19:50
jwitkotcp        0      0 10.89.103.25:8181       0.0.0.0:*               LISTEN      1803/haproxy19:50
jwitkoyet I can't telnet to .24,  i can telnet to .25 though19:50
openstackgerritMerged openstack/openstack-ansible-os_glance: Add os-brick to glance_pip_packages  https://review.openstack.org/38930519:50
bazwhat about the acl lines for the frontends in haproxy.cfg?19:51
bazand one of them should be ssl'd19:52
openstackgerritMerged openstack/openstack-ansible-os_ceilometer: Update paste, policy and rootwrap configurations 2016-10-12  https://review.openstack.org/38554319:53
openstackgerritMerged openstack/openstack-ansible-os_ceilometer: [DOCS] Added release-name as a watermark to Docs.  https://review.openstack.org/38813419:53
bazso you'd need to use openssl s_client -connect host-or-ip:818119:53
openstackgerritMerged openstack/openstack-ansible-os_ceilometer: Use upper constraints for all tox targets  https://review.openstack.org/38842919:53
jwitkobaz, there should still be an open connection via telnet with or without ssl19:57
*** omiday has joined #openstack-ansible20:00
bazand both 24 and 25 show up in 'ip a show br-mgmt'?20:03
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer: Remove 'ignore_errors: true' in favor of 'failed_when: false'  https://review.openstack.org/38680220:08
jwitkobaz, no neither does20:18
*** thorst_ has quit IRC20:22
*** phalmos has joined #openstack-ansible20:25
*** spotz is now known as spotz_zzz20:25
dxiriguys, my openstack is running! yay!20:27
dxiriinstalling some images now20:27
dxiribut I noticed this: The current Horizon settings indicate no valid image creation methods are available. Providing an image location and/or uploading from the local file system must be allowed to support image creation.20:27
dxiriwhere can I change that?20:27
*** galstrom is now known as galstrom_zzz20:33
*** weezS has quit IRC20:36
*** phalmos has quit IRC20:36
openstackgerritMerged openstack/openstack-ansible: [docs] Add os_trove to Advanced Config section  https://review.openstack.org/38862720:40
*** askb has joined #openstack-ansible20:47
jwitkoso I'm changing the internal VIP to an IP on my vxlan network20:49
jwitkoto be on a different network than my external VIP20:49
jwitkodo I need to do anything to make sure the internal VIP uses the correct br-vxlan interface?20:49
jwitkoI think just set haproxy_keepalived_internal_interface right?20:50
openstackgerritMerged openstack/openstack-ansible-os_ceilometer: Remove 'ignore_errors: true' in favor of 'failed_when: false'  https://review.openstack.org/38680220:53
*** weezS_ has joined #openstack-ansible20:54
*** thorst_ has joined #openstack-ansible20:56
mrdaMorning OSA21:00
*** jeh has joined #openstack-ansible21:06
*** jeh has quit IRC21:07
*** jeh has joined #openstack-ansible21:12
cloudnullbaz: RE: Image create. personally I think that's an oversight21:13
cloudnulldo you know which sections are needed?21:13
openstackgerritweezer su proposed openstack/openstack-ansible-ops: Remove the expired key and opensuse repo after cobbler being installed  https://review.openstack.org/38890621:15
cloudnulldxiri:  -- baz mentioned that horizon is not allowing image creation.21:15
cloudnullin general I use the CLI to do all of that.21:16
cloudnullwhich is likely why the horizon config is missing.21:16
cloudnullas I imagine that most of us use the CLI for day to day ops21:17
cloudnullI'm looking into that now.21:17
dxiricloudnull: awesome! thank you :)21:19
dxirialso found another problem, I can't create instances21:19
dxiriError: Failed to perform requested operation on instance "inst1", the instance has an error status: Please try again later [Error: No valid host was found. There are not enough hosts available.].21:19
dxirithis is using a cirros test image...with 1gb ram and 5gb disk21:19
dxirireally small21:19
cloudnullare the compute-nodes checking in  ?21:20
*** jeh has quit IRC21:20
cloudnullbaz: dxiri: it would seem we need to set https://github.com/openstack/horizon/blob/master/openstack_dashboard/local/local_settings.py.example#L369-L37821:22
c-martdxiri, check nova-compute.log for your compute host(s)21:23
*** retreved has quit IRC21:23
dxiriif I go to system-information, I see all compute services enabled and ip, last updated 0 minutes21:23
c-marti had the "no valid host was found" issue and it was due to something completely unrelated to the compute service -- Nova couldn't talk to another API, and that was apparent in nova-compute.log21:24
dxirihere is the error from nova-compute.log21:28
dxirihttp://pastebin.com/WgnM4aSP21:28
*** hybridpollo has quit IRC21:31
*** schwicht has quit IRC21:33
*** Jeffrey4l has quit IRC21:34
admin0dxiri: check your libvirt logs21:34
admin02016-10-20 16:26:52.637 3573 ERROR nova.compute.manager PortBindingFailed: Binding failed for port 5bc482f5-7c8b-4496-bdee-230165294653, please check neutron logs for more information.21:34
admin0evrardjp: around ?21:34
admin0evrardjp: when you read through the logs next, tomorrow i will have 3 new environments iwth 3 controlles each .. will test/validate the haproxy/keepalive issue and confirm21:36
c-martis anyone here familiar with using Ceph as a storage back-end for cinder and Glance? I have a question about how OSA is configured to talk to Ceph21:36
cloudnullbaz: dxiri: so nevermind . https://bugs.launchpad.net/horizon/+bug/163238321:37
openstackLaunchpad bug 1632383 in OpenStack Dashboard (Horizon) "The current Horizon settings indicate no valid image creation methods are available" [Undecided,New]21:37
cloudnullthe image create via horizon seems to be an upstream bug21:37
admin0c-mart it works :)21:39
dxiricloudnull: so all I need to do is wait ?21:39
cloudnullyup21:39
cloudnullsadly21:39
dxiriadmin0: checking neutro logs now21:39
*** weezS_ has quit IRC21:39
admin0check libvrit as well21:40
admin0sometimes no logs give any error, you still get host not found, but real reason in libvirt log21:40
*** spotz_zzz is now known as spotz21:40
c-martadmin0, do you know why we need to define storage_hosts in openstack_user_config.yml, if we'll just be talking to Ceph as the storage back-end? what do the storage_hosts do in that case?21:41
cloudnulldxiri: baz: i've poked a few folks and ill let you know what shakes out.21:42
admin0c-mart, i actually wrote that part i the ansible docs21:42
admin0they do nothing21:42
admin0so you need to set the is_metal: false21:42
admin0so that the api’s run in the controller nodes themselves21:42
admin0c-mart: i wrote a blog on this: http://www.openstackfaq.com/openstack-ansible-ceph/21:43
admin0i am working to expand the current blog to be with newton and 3 ceph backends/pools:   SSD, SATA and Hybrid21:43
*** hughmFLEXin has quit IRC21:43
c-martI found that blog post admin0 :)21:44
admin0i already tested 3 backends for volumes so when you create volume, in dropdown you can select sata vs ssd21:44
admin0that will go up .. maybe will write after the summit21:44
c-martcool. so in that blog post, when you define three storage hosts, you're saying those containers are just created on the controller nodes??21:45
*** hybridpollo has joined #openstack-ansible21:46
admin0yes21:48
admin0in that way, the storage-api/containers are load balanced21:48
admin0and de-coupled from your ceph21:48
admin0they are just like any other api21:48
c-martah, now that makes so much more sense. thank you21:48
*** schwicht has joined #openstack-ansible21:49
admin0right now i am doing something complex .. i have OSA as region2, but on a keystone db that is from region1 and on juno .. so its upgraded to newton, region 2 uses the v3, while i am chaning the v2 to this new newton keystone so that my existing region1 (still on icehoise)  works fine as usual with 1 single keystone serving region2 and region1 :D21:50
admin0\o/21:50
admin0chaning/fixing*21:51
dxiriso, this may have something to do with my problem...I created a new network on horizon (made up the subnet) and set it shared21:51
admin0dxiri: it works :)21:52
*** chris_hultin is now known as chris_hultin|AWA21:52
dxiridoes that subnet needs to be the subnet for the br-vxlan or br-vlan I configured when setting openstack up21:52
admin0dxiri:  something like this: http://www.openstackfaq.com/openstack-add-direct-attached-dhcp-ip/21:52
admin0no21:52
admin0it needs to be different21:52
admin0there http://www.openstackfaq.com/openstack-add-direct-attached-dhcp-ip/ , i add a different range as shared, and a different range for floating ip21:53
admin0br-vxlan is just one subnet you need for the vtep and on its own vlan/tag/isolation21:53
admin0br-vlan has no ips21:54
admin0so whatever ips you add as floating or shared , and if its under a vlan tag, it carries/forwards that21:54
*** spotz is now known as spotz_zzz21:55
dxiriok so If I want an internal network between the vms only (for now) how should I create the network? I just want the VM to have an internal IP, it doesn't matter if it can't reach anything else for now21:56
*** thorst_ has quit IRC21:56
*** thorst_ has joined #openstack-ansible21:57
*** hughmFLEXin has joined #openstack-ansible21:57
openstackgerritMerged openstack/openstack-ansible: Add missing Ironic auth plugin variable  https://review.openstack.org/38917421:58
*** spotz_zzz is now known as spotz22:01
*** weezS has joined #openstack-ansible22:02
admin0dxiri:  in that case, create a private network ..  and just spawn those vms int he private net22:02
cloudnull^ admin0 beat me too it :)22:02
admin0if you are using osa and defaults, they will use some X vxlan22:02
* admin0 goes back to meditation :D22:03
admin0cloudnull: i suddenly have 4 multi-node environments to test stuff out :D22:03
admin0so been very occupied22:03
c-mart*bow* thank you admin022:03
admin0will you be in the summit ?22:03
cloudnullsadly I will not be at the summit.22:04
cloudnull:'(22:04
dxiriadmin0: I am using OSA and defaults yes, but *how* do I create that private network? that's my question, do I just make stuff up and create a network with whatever range I want and whatever vlan ID I want?22:04
admin0skype link for people to meetup in openestack summit barcelona: https://join.skype.com/fnzd1JCc4Vxp22:04
admin0dxiri:22:04
admin0you do not worry about the id22:04
admin0please feel free to create whatever valid IP you want to have22:05
*** spotz is now known as spotz_zzz22:05
admin01.1.1.0/8  is perfectly valid as well :)22:05
*** thorst_ has quit IRC22:05
*** asettle has quit IRC22:05
admin0because its internal and tied to some vxlan22:05
admin0i use 1.1.1.0/8; 2.2.2.0/24 — those are nice :D22:06
admin011.11.11.0/24 :D22:06
admin0if you create a router, then you need to enter the router namespace before you can ssh in or reach it internally22:06
admin0if you don’t then console is your friend :)22:06
bazcloudnull the "missing" config is https://github.com/openstack/horizon/blob/master/openstack_dashboard/local/local_settings.py.example#L754-L75622:10
bazan additional problem relates to haproxy if you set 'haproxy_stats_enabled: true' in user_variables.yml22:11
cloudnullbaz: so adding that list fixed it ?22:13
c-martadmin0 a suggested change to your Ceph blog post: "disable is_metal from /etc/openstack_deploy/env.d/cinder.yml" is now a different folder path, should be /opt/openstack-ansible/env.d/cinder.yml, I believe.22:13
admin0yes22:13
admin0that was for the old release22:13
admin0newton one is coming up22:13
bazhttps://github.com/openstack/openstack-ansible-haproxy_server/blob/master/templates/haproxy.cfg.j2#L2622:14
cloudnullbaz: also what's the other issue when stats are enabled?22:14
cloudnullha.22:14
cloudnullis that not working?22:14
bazthe part after listen stats needs to be on a separate line as a 'bind'22:14
bazjust like normal services22:14
bazadditionally might want to add 'stats refresh 60s' to have it auto-reload22:15
bazas for the horizon thing, yes that's what was ultimately needed to fix the Horizon image upload issue after digging through the code.22:15
cloudnullok22:16
cloudnulldo you have a moment to submit a PR?22:17
dxirik I created the net...same problem :(22:19
*** thorst_ has joined #openstack-ansible22:20
bazthose required REST APIs were added either in commits on Feb 10th and Apr 15th of this year.22:20
dxirimmm22:20
dxiri2016-10-20 22:16:20.832 5199 WARNING stevedore.named [req-41fc2818-9a7a-4e91-ba89-2f7fc7f11941 1ead964a48754da2822f0fafd8be3890 98b68cdb2f834a06b56733f57db7f8e5 - - -] Could not load neutron.agent.linux.interface.BridgeInterfaceDriver22:20
cloudnullbaz: regatding haproxy -- http://104.130.20.56:1936/ its working for me as is22:20
* cloudnull removed auth just FYI22:20
cloudnullbut the horizon bits seem to fix it22:20
*** admin0 has quit IRC22:20
bazcloudnull which version of haproxy?22:21
cloudnull1.5.1822:21
bazHA-Proxy version 1.6.3 2015/12/25 fails if the line is 'listen stats 1.2.3.4:1936'22:21
bazthe config changed in 1.6 and stats looks just like services in that it takes a bind statement22:22
bazso there'll need to be ansible conditionals based on which version of haproxy is used.22:22
*** admin0 has joined #openstack-ansible22:23
cloudnullit looks like the 1.6 syntax format works in 1.522:23
cloudnullI changed my config to http://cdn.pasteraw.com/nol2eyn1mld7nod3wslw4sahqlahf8s22:24
cloudnulland the stats dash is still working22:24
*** thorst_ has quit IRC22:24
*** schwicht has quit IRC22:24
bazit likely works on 1.5.x but fails in 1.4.x, but does anything OSA supports as the host distribution ship 1.4.x?22:25
jwitkohey cloudnull, can you help me understand how to work the internal/external vip where I don't have a separate network?22:28
jwitkoit doesn't seem to work with both ips on the same subnet22:28
jwitkoand definitely doesn't work where both IPs are identical22:28
admin0jwitko: its supposed to be different on production setting22:28
admin0i use that a lot22:28
admin0the vip addresses22:29
jwitkoalthough it seems like it would work in the case of identical IPs if there was a conditional that only generated HAProxy output once instead of twice22:29
jwitkoadmin0, so can you help me ?22:29
cloudnullbaz: i think we're always using 1.5 and above in newton +22:29
admin0use different ips and endpoints jwitko22:29
cloudnulljwitko: it should work with different IPs on the same subnet22:30
jwitkoadmin0, "ips and endpoints"  isn't the IP the endpoint ?22:30
jwitkocloudnull, only one of them is ever pingable22:30
admin0nope22:30
bazcloudnull then it's a safe change. 1.6 just made it a requirement22:30
admin0endpoint traditionally refers to URL :)22:30
admin0URL is something that points to IP22:30
cloudnullbaz: agreeded.22:30
jwitkolike right now I have an IP for internal and a hostname in a different subnet for external22:30
jwitkoi can ping the IP and telnet to the ports22:30
jwitkobut the external hostname I can not connect to any of the ports or ping it22:31
jwitkoand it is on the same network as br-mgmt22:31
jwitkoi can see the servers listening on that IP with the ports open22:31
cloudnulli think if you bind to two ips on the same subnet you'll need to add an alias IP on the network interface of the host.22:31
jwitkobut can't connect22:31
cloudnullbaz: do you have a moment to make the change to hap ?22:32
admin0jwitko:  i have this on config:     internal_lb_vip_address: cloud101int.stack31.com    external_lb_vip_address: cloud101.stack31.com     && and on variables      haproxy_keepalived_external_vip_cidr: "10.11.12.3/22" haproxy_keepalived_internal_vip_cidr: "172.29.236.3/22"   haproxy_keepalived_external_interface: ens2   haproxy_keepalived_internal_interface: br-mgmt22:32
cloudnullor horizon. if not I'd be happy to, I just don't want to take your commit.22:32
bazcloudnull give me a second to create a pull22:33
admin0so the VIP addresses gets  used as endpoints22:33
admin0from DNS i map the IPs to the addresses22:33
*** TxGirlGeek has quit IRC22:33
jwitkoadmin0, cloudnull, ok thanks I think that helps.  if I want to change the IP/endpoints is there any easy way or do i have to destroy containers?22:33
admin0truncate table keystone :D22:33
admin0if its new one22:33
admin0sorry22:33
cloudnulljwitko: change the IPs of all containers ?22:34
*** weezS has quit IRC22:34
cloudnullor just the endpoints?22:34
admin0no .. its just the VIP/endpoint22:34
admin0truncate the endpoints22:34
jwitkocloudnull, if I'm changing my VIPs and CIDRs for haproxy22:34
bazsince there's already a launchpad bug for the Horizon issue I'll just attach a patch to that.22:34
jwitkojust to reset those throughout the environment22:34
cloudnullcool thanks baz22:34
admin0jwitko: you need to re-do the endpoints22:35
admin0delete the existing one from d22:35
admin0db22:35
admin0and then re-run the os-keystone22:35
openstackgerritJimmy McCrory proposed openstack/openstack-ansible: Add upgrade playbook for ceilometer-api  https://review.openstack.org/38937822:35
openstackgerritJimmy McCrory proposed openstack/openstack-ansible: Add upgrade playbook for ceilometer-api  https://review.openstack.org/38937822:36
cloudnulljwitko: i'd make the change to config and then change the DB like so : update endpoint set url = REPLACE(url, '172.19.8.100', '172.19.0.21') where interface='public';22:37
openstackgerritJimmy McCrory proposed openstack/openstack-ansible: Add upgrade playbook for ceilometer-api  https://review.openstack.org/38937822:37
jwitkoi'd have to change the interface it binds too as well22:37
jwitkoI think i'll just destroy the containers and the haproxy configs22:38
jwitkothat would work yes?22:38
admin0yesp22:38
admin0for i in `lxc-ls`; do lxc-destroy -f -n $i ; done :D22:38
cloudnullthe nuclear option works too.22:38
cloudnullthe lxc-container-destroy.yml play will do it to all the things22:39
admin0:D22:39
admin0that too22:39
cloudnullhttps://www.youtube.com/watch?v=aCbfMkh940Q22:39
admin0those coming to barcelona and want to join in https://join.skype.com/fnzd1JCc4Vxp22:39
admin0its a skype chat group22:39
admin0to arrange a meeting22:39
dxiriadmin0: libvirt logs are blank22:45
admin0:(22:45
admin0strace :D22:45
dxiriunless I'm looking at the wrong place22:45
admin0well, figure out why the VM does not create22:45
admin0could be neutron, if not neutron, libvirt22:45
dxirilooking in /var/log/libvirt on the compute node22:46
dxiri/var/log/libvirt/libxl:22:46
dxiritotal 022:46
dxiri/var/log/libvirt/lxc:22:46
dxiritotal 022:46
dxiri/var/log/libvirt/qemu:22:46
dxiritotal 022:46
dxiri/var/log/libvirt/uml:22:46
dxiritotal 022:46
dxiriah!22:47
dxiri2016-10-20 17:47:14.742 5433 ERROR neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent [-] Interface eth12 for physical network flat does not exist. Agent terminated!22:47
dxiribut I don't recall creating any flat networks22:48
dxiriso wtf22:48
admin0yeah22:49
admin0see :D22:49
BjoernTit's probably listed inside the provider networks section22:49
admin0well22:49
admin0i know this22:49
admin0its the defaults22:49
admin0they list eth1222:49
admin0you have to remove the eth12 to yoru br-pxe or your pxe/standard interface and retry22:50
BjoernThttps://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio#L4722:50
BjoernTremove that whole flat block22:50
admin0dunno who ( yes developers, i am pointing to you) .. thought eth12 will exist in a normal box22:50
admin0     - network:22:51
admin0         group_binds:22:51
admin0           - neutron_linuxbridge_agent22:51
admin0         container_bridge: "br-vlan"22:51
admin0         container_type: "veth"22:51
admin0         container_interface: "eth12"22:51
admin0         host_bind_override: "ens2"22:51
admin0         type: "flat"22:51
admin0         net_name: "flat"22:51
admin0boo22:51
admin0sorry22:51
BjoernTLol you ad admin need to make sure that you have proper config, OSA doesn't deploy clean without that. Unless you have AIO22:51
admin0 host_bind_override: "ens2" in my case22:51
BjoernTyepp22:51
admin0dxiri: its a consipracy from the devs to ensure we operators are awake .. so need to change  host_bind_override: "eth12” to whatver you have in reality22:52
admin0else the first deployment fails22:52
admin0due to this22:52
admin0#nastydevs :D22:53
* admin0 is an operator 22:53
jwitkolooks like i need to reset bare metal pip configs as well..  they're still trying to reach the old vip22:53
*** marst has quit IRC22:53
admin0well, you have 00apt-proxy everywhere which  you need to rm -f as well22:54
admin0else the playbooks fail22:54
dxiriis that the physical interface itself? not the br-vlan bridge?22:55
dxiriIn my case I am using a single nic (eth0)22:55
admin0put eth0 and yuo are done :)22:56
jwitkoadmin0, any others I should know about ?22:56
admin0lots :)22:56
admin0depends on your deployment22:57
admin0greenfield or brown ?22:57
admin0cinder, ceph,22:57
jwitkogreenfield, cinder22:57
admin0well, apart from this change, and if hte VIps are correctly defined, it should work without issues22:58
dxiriadmin0: k changed...now...do I have to destroy everything and start over?22:59
admin0yes please22:59
dxirithat is great!22:59
* admin0 loves destruction and chaos 22:59
dxiri:P22:59
dxiriby the way I am following appendix A of the documentation23:00
dxiriwould be good if this stuff get mentioned there23:00
admin0i have 3 virtual and 2 phiysical  multi-node envs.. i think i have destroyed and ran 500+ times already23:00
dxiriI also see yet another eth11 for neutron23:00
dxiri- network:23:01
dxiri        container_bridge: "br-vlan"23:01
dxiri        container_type: "veth"23:01
dxiri        container_interface: "eth11"23:01
dxiri        type: "vlan"23:01
dxiri        range: "1:1"23:01
dxiri        net_name: "vlan"23:01
dxiri        group_binds:23:01
dxiri          - neutron_linuxbridge_agent23:01
dxirido I leave that alone?23:01
admin0its for the container23:01
admin0yes23:01
admin0leave that alone please23:01
dxiricool23:01
dxiriguess I will have to redownload and recreate the images after I destroy everything right?23:02
admin0https://gist.github.com/a1git/af22cce0d39e389d689835e4a0d6aa4c - this is for a normal one23:02
dxiriAre you sure you want to destroy the LXC container data? [no]:23:04
dxiriwhat happens if I don't destroy data?23:04
dxiribad thing will happen?23:04
admin0and the variables https://gist.github.com/a1git/9831617c5d0933a53c8e346482e0f93523:04
admin0its greenfield :) and new23:04
admin0you can destroy everything23:04
dxiriopenstack-ansible lxc-containers-destroy.yml23:06
dxirijust that one right?23:06
admin0yes23:06
admin0and the ansible facts as well23:06
dxiridestroy the ansible facts?23:07
*** schwicht has joined #openstack-ansible23:07
admin0remove the ansible facts if you remove the containers23:07
dxiriwhere? how?23:07
dxirisry for being this noob :P23:07
admin0 rm -rf /etc/openstack_deploy/ansible_facts/   ## on the deploy hosts23:07
admin0i am a noob as well23:07
dxirik thanks!23:07
admin0there is no shame in asking what you dont know23:08
admin0dxiri:  rnage 1:1 in vxlan23:10
admin0it means you cannot have more than 1 private network23:10
admin0look at mine:  range: "10000:100000"23:11
admin0br-vlan23:12
admin0is 1:!23:12
admin01:123:12
admin0means you can have just 1 network23:13
admin0thati is wrong23:13
admin0you need at least a few23:13
admin0one for floating, one for test, one for dev, one for prod — all share23:13
admin0shared23:13
* admin0 needs sleep 23:17
admin0good night all23:17
BjoernTrnage 1:1 in vxlan: only means that you can't create networks over horizon beyond one vlan23:17
admin01:17 AM23:17
admin0its in his vlan23:17
BjoernTin neutron cli you can just override the segmentation_id23:17
admin0i have at least 3 vlans for floating, 3 vlans for shared , all /22 each23:18
jwitkodamn my deploy is still trying to go through the old VIPs23:19
jwitkosomething with pip is still telling it to go to the old IP23:19
*** admin0 has left #openstack-ansible23:19
*** admin0 has quit IRC23:20
*** c-mart has quit IRC23:21
*** weezS has joined #openstack-ansible23:21
jwitko"/root/.pip "23:24
jwitkohaha23:24
*** thorst_ has joined #openstack-ansible23:24
*** gouthamr has quit IRC23:27
*** hughmFLEXin has quit IRC23:27
*** maeker has quit IRC23:28
*** fops_ has joined #openstack-ansible23:30
*** Jeffrey4l has joined #openstack-ansible23:31
*** fops has quit IRC23:32
*** thorst_ has quit IRC23:33
openstackgerritMerged openstack/openstack-ansible: Work around bad libvirt-python wheel  https://review.openstack.org/38893423:35
*** weezS has quit IRC23:35
*** BjoernT has quit IRC23:36
*** markvoelker has quit IRC23:37
*** agrebennikov_ has quit IRC23:46
openstackgerritJimmy McCrory proposed openstack/openstack-ansible: Add upgrade playbook for ceilometer-api  https://review.openstack.org/38937823:46
*** schwicht has quit IRC23:47
openstackgerritJameson Finney proposed openstack/openstack-ansible: Adds support for disks on HP Smart Array Controllers  https://review.openstack.org/38905323:56
*** javeriak has quit IRC23:58
dxirihaving some problem with apt now, trying to run setup-hosts again23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!