Tuesday, 2018-03-06

*** esberglu has quit IRC00:02
*** hw_wutianwei has quit IRC00:04
*** hw_wutianwei has joined #openstack-ansible00:04
*** esberglu has joined #openstack-ansible00:17
*** chyka_ has quit IRC00:25
*** chyka has joined #openstack-ansible00:25
*** sawblade6 has quit IRC00:30
*** TxGirlGeek has joined #openstack-ansible00:45
*** chyka has quit IRC00:47
*** woodard has joined #openstack-ansible00:50
*** vishwana_ has quit IRC00:53
*** lihi has quit IRC00:54
*** oanson has quit IRC00:56
*** oanson has joined #openstack-ansible01:03
*** lihi has joined #openstack-ansible01:03
jmccroryodyssey4me I think I found the issue with nova breaking in master, it was the recent RPC changes that you were talking about. they dropped support for everything pre-queens. conductor defaults to 'mitaka' and compute never gets any messages. added a comment on one of their patches: https://review.openstack.org/#/c/546256/01:26
*** TxGirlGeek has quit IRC01:59
*** woodard has quit IRC02:04
*** woodard has joined #openstack-ansible02:04
*** SmearedBeard has quit IRC02:08
*** SmearedBeard has joined #openstack-ansible02:12
*** chyka has joined #openstack-ansible02:13
*** chyka has quit IRC02:18
*** radeks has joined #openstack-ansible02:26
*** radeks has quit IRC02:33
*** ANKITA_ has joined #openstack-ansible02:38
*** gaoyan has joined #openstack-ansible02:50
*** kstev has joined #openstack-ansible02:56
*** esberglu has quit IRC03:26
*** chyka has joined #openstack-ansible03:28
*** kstev has quit IRC03:31
*** chyka has quit IRC03:33
*** masuberu has quit IRC03:34
*** esberglu has joined #openstack-ansible04:00
*** idlemind has quit IRC04:11
*** esberglu has quit IRC04:21
*** masber has joined #openstack-ansible04:23
*** udesale has joined #openstack-ansible04:44
*** pmannidi has quit IRC05:01
*** pmannidi has joined #openstack-ansible05:15
*** chyka has joined #openstack-ansible05:15
*** dariko has joined #openstack-ansible05:17
*** chyka has quit IRC05:20
*** dariko has quit IRC05:23
*** poopcat has quit IRC05:23
*** dariko has joined #openstack-ansible05:24
*** markvoelker has quit IRC05:27
*** woodard has quit IRC05:29
*** pmannidi has quit IRC05:39
*** pmannidi has joined #openstack-ansible05:43
*** ivve has quit IRC06:05
*** dariko has quit IRC06:07
*** threestrands has quit IRC06:14
*** markvoelker has joined #openstack-ansible06:27
*** ANKITA_ has quit IRC06:34
*** ANKITA_ has joined #openstack-ansible06:38
*** radeks has joined #openstack-ansible06:41
*** taseer1 has joined #openstack-ansible06:49
*** woodard has joined #openstack-ansible06:52
*** fusmu has joined #openstack-ansible06:52
*** Taseer has quit IRC06:52
*** woodard has quit IRC06:56
*** taseer1 is now known as Taseer\06:58
*** Taseer\ is now known as Taseer06:58
Taseerevrardjp: good morning ! Are you feeling better today ?06:59
Taseerare all 'openstack-ansible-deploy-aio_lxc-ubuntu-xenial' jobs timing out ?07:02
openstackgerritTaseer Ahmed proposed openstack/openstack-ansible master: [WIP] Integrate Blazar with OpenStack Ansible  https://review.openstack.org/54995607:04
nsinghANKITA: thanks, running the playbook from /opt/openstack-ansible/playbooks works.07:04
nsinghMy playbook is failing with error:07:04
nsinghTASK [os_masakari : Retrieve checksum for venv download] ***************************************************************************************************************************07:04
nsinghTuesday 06 March 2018  11:01:37 +0530 (0:00:12.339)       0:00:45.959 *********07:04
nsinghfatal: [aio1]: FAILED! => {"changed": false, "content": "", "msg": "Status code was not [200]: Request failed: <urlopen error [Errno 111] Connection refused>", "redirected": false, "status": -1, "url": "http://127.0.0.1/venvs/untagged/ubuntu/masakari.checksum"}07:04
nsinghi am not sure about this URL. Any help07:05
*** pmannidi has quit IRC07:07
Taseernsingh: it is trying to get venv from galera server I guess. which apparently is not present07:12
*** PTO has joined #openstack-ansible07:14
*** Sha000000 has quit IRC07:32
*** jwitko has quit IRC07:32
*** PTO has quit IRC07:32
*** PTO has joined #openstack-ansible07:38
PTOI'm trying to update the ELK playbooks in the openstack-ansible-ops repo: https://github.com/openstack/openstack-ansible-ops/tree/master/elk_metrics. Im alsmost complete, but currently im stuck at a strange bug. When i run the install-haproxy.yml, the dynamic inventory will update the target container with a faulty ip address. I have tried to manualle change it and even delete it. Same result. Any ideas on what is wrong?07:44
*** lvdombrkr has joined #openstack-ansible07:46
*** armaan has joined #openstack-ansible07:46
*** armaan_ has joined #openstack-ansible07:48
*** PaulusNL has joined #openstack-ansible07:49
*** PaulusNL has left #openstack-ansible07:50
*** armaan has quit IRC07:51
*** woodard has joined #openstack-ansible07:53
*** woodard has quit IRC07:57
*** armaan_ has quit IRC07:57
*** epalper has joined #openstack-ansible08:05
evrardjpnsingh: which branch are you on?08:06
evrardjpTaseer: not really08:06
evrardjpbut let's see in the next hours/day08:06
evrardjpPTO you should ask d34dh0r5308:06
*** epalper has quit IRC08:12
*** epalper has joined #openstack-ansible08:13
PTOevrardjp: Just realize that my entire ansible inventory is screwed *cry* - it happend after running install-haproxy.yml to add a kibana service. Wrong ip addresses in the for all containers. All containers have multiple ip addresses on eth1. Is it possible to reset and run osa_gather_facts again?08:13
*** armaan has joined #openstack-ansible08:14
*** gaoyan has quit IRC08:20
evrardjpPTO that's concerning. I'd roll back to a known inventory08:20
evrardjpCould you file a bug with all what you did and what happened ? That would help d34dh0r53 fixing things.08:20
PTOevrardjp: By overwriting the /etc/openstack/openstack_hostnames_ips.yml andopenstack_inventory.json files?08:24
evrardjpPTO no, by restoring a backup (or at least checking the tar file present in the same folder)08:25
PTOevrardjp: Thanks.08:26
*** sar has joined #openstack-ansible08:26
*** olivierbourdon38 has joined #openstack-ansible08:31
*** gkadam has joined #openstack-ansible08:33
*** hw_wutianwei has quit IRC08:39
*** hw_wutianwei has joined #openstack-ansible08:40
*** gameon has joined #openstack-ansible08:41
*** oanson has quit IRC08:41
gameonHi all - I am having an issue with a newly deployed Pike cluster where instances are being given a DNS suffix of .novalocal by cloud-init - however the dhcp_domain setting in Nova seems to have been deprecated so I can't see how I can change this setting. Any ideas please?08:42
*** lihi has quit IRC08:42
*** pcaruana has joined #openstack-ansible08:44
nsinghevrardjp: master08:44
*** electrofelix has joined #openstack-ansible08:45
*** gkadam_ has joined #openstack-ansible08:46
*** gkadam_ has quit IRC08:46
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-openstack_hosts stable/queens: SUSE: Switch OpenStack repo to Queens  https://review.openstack.org/54997708:47
*** gkadam has quit IRC08:48
PTOevrardjp: After the restore, the container ips are still wrong. Any other suggestions on how to update? I have rebooted the containers, which fixed the multiple ip addresses08:48
evrardjp[m]nsingh please use our healthcheck playbooks ?08:49
*** oanson has joined #openstack-ansible08:50
*** lihi has joined #openstack-ansible08:50
odyssey4mehwoarang you have a circular dependency between https://review.openstack.org/#/c/549764/3 and https://review.openstack.org/#/c/549878/208:51
evrardjp[m]pto that sounds bad... all the inventories are busted there ?08:51
*** chyka has joined #openstack-ansible08:51
odyssey4mePTO is this a test environment, or a live one?08:52
hwoarangodyssey4me: thank you i fixed it08:52
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_container_create master: templates: networkd: Drop Link=ARP from networkd configuration  https://review.openstack.org/54987808:52
PTOevrardjp[m]: It seems so *cry*. All containers exists on the hosts and they work, they exists in the inventory-manage.py file, but the ip mapping is wrong. E.g. cloudbc1bl01_keystone_container-58166ebd: 192.168.10.9 in inventory and the containers real ip is 192.168.9.39/2008:52
*** woodard has joined #openstack-ansible08:53
odyssey4mehwoarang on that one, IIRC the arp flush was important to ensure that restarts worked right - as restarts in lxc often end up with a new mac address - are we sure we're not inviting that regression back in?08:53
PTOodyssey4me: Its not in production - yet.08:53
odyssey4mePTO best to fiddle in a test environment that you can throw away, but that ship has sailed08:53
hwoarangodyssey4me: This ARP=yes thing is not supported by systemd anyway :/ if we had such a workaround before we should prob bring it back08:54
PTOodyssey4me: I can still throw it away08:54
odyssey4mePTO perhaps best to inspect the inventory.json to figure how how the mapping is all screwed up - you may have to hand edit it08:54
hwoarang*is not supported by old-ish systemd08:54
*** pcaruana has quit IRC08:54
odyssey4meotherwise, assuming you've figured out the right config now - it may be easier to just rekick your hosts and start again08:54
ANKITAodyssey4me THANKS08:55
*** chyka has quit IRC08:56
odyssey4mehwoarang we did some sort of post-down arp flush IIRC... I can't seem to find reference to it, but I've not grepped yet08:57
PTOodyssey4me: I dont know how or why, but all my containers have got assigned an ekstra ip address, so they have a primary and secondary. Which seems to have screwed up the inventory. A reboot of the container has fixed the multiple ip addresses. Now i just need to fix the inventory by updating the ips again08:57
odyssey4mePTO hmm, on which interface do they have extra IP's? - their eth0 attached to lxcbr0 is assigned via dnsmasq... those aren't tracked in inventory at all08:57
*** woodard has quit IRC08:57
*** gkadam has joined #openstack-ansible08:57
PTOodyssey4me: eth108:58
odyssey4mehwoarang https://github.com/openstack/openstack-ansible-lxc_container_create/blob/stable/queens/vars/ubuntu-16.04.yml#L29-L3308:59
hwoaranghmm08:59
odyssey4mePTO hmm, eth1 is inventory tracked... which means it will show up in inventory.json and you can remove it (assuming there are two)09:00
odyssey4meif there is only one in there, then take a look at the container network configs themselves - both in lxc (on the host) and inside... compare that to inventory09:00
PTOIm trying to build a 25 node openstack cluser. Started on 3 nodes and was success full. Then expanded and was successfull on all 25 nodes. Then i needed an ELK, and used the openstack-ansible-ops elk_metrics repo. These plays were obsolete and i have updated these and it works. Finally, i needed to update haproxy endpoints to kibana, which crashed it all09:00
odyssey4mehwoarang so it may be best to re-implement that in the oneshot implemenentation...09:01
hwoarangyeah09:01
hwoarangi will look into that after the dns thing because it's driving me nuts09:01
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-rabbitmq_server stable/queens: [DNM] - Testing rabbitmq - Queens OBS repo  https://review.openstack.org/54998009:03
*** ivve has joined #openstack-ansible09:08
*** masber has quit IRC09:09
odyssey4mehwoarang I've seen cases where a boolean is not interpreted properly unless the bool filter is used, so I think it'd be best to tweak https://review.openstack.org/#/c/549877/109:10
odyssey4mejust like this it might be interpreted as "if this variable exists, then..."09:11
odyssey4meor more accurately - "if this variable is not None..."09:12
*** ivve has quit IRC09:13
*** kmARC has quit IRC09:15
*** Sha0000 has joined #openstack-ansible09:16
hwoarangok makes sense will fix it09:16
openstackgerritAndy McCrae proposed openstack/openstack-ansible master: Avoid putting rgw conf on ALL hosts  https://review.openstack.org/54984009:16
*** gkadam has quit IRC09:18
PTOIs it enough to copy a inventrou backup and overwrite  /etc/openstack_deploy/openstack_inventory.json?09:18
PTO*inventory09:19
evrardjp[m]could someone make a quick summary of what's going on ?09:19
*** hw_wutianwei has quit IRC09:20
*** hw_wutianwei1 has joined #openstack-ansible09:20
evrardjp[m]master/Queens related. it seems queens is passing gates but i see some commits... should I wait for them to land before tagging our first queens release? has someone checked on the upgrades ? i will check during my commute if need be, but I'd like to tag soonish this week.09:21
*** hw_wutianwei1 is now known as hw_wutianwei09:23
hwoarangevrardjp[m]: for queens i have just switched the suse repo from queens to pike09:26
hwoarang*eh the other way around09:26
hwoarangcan you have 10 min for gates to pass?09:26
evrardjp[m]hwoarang: i saw that patch09:26
*** mbuil has joined #openstack-ansible09:26
evrardjp[m]not sure what you mean with the 10 minutes. you want to extend gate time ?09:27
odyssey4meevrardjp[m] master is a bit broken, hwoarang is unwinding it - queens appears to be fine09:28
hwoarangevrardjp[m]: no im just waiting for https://review.openstack.org/#/c/549980/ to pass09:28
odyssey4meI think that spectre/meltdown is affecting out gate times quite badly for master09:28
hwoarangwhich tests queens/rabbitmq on suse09:28
odyssey4meevrardjp[m] we might need to bring in some role patches to ensure that queens is using the queens repositories for centos/suse... they've merged, but I don't think the role SHA's are using them09:29
evrardjp[m]meltdown mitigations applies to all branches, not sure to understand what you mean.09:29
odyssey4meevrardjp[m] our gates are very slow, timing out a lot - that's a problem for our integrated build09:29
evrardjp[m]only for master ?09:30
odyssey4menope, but especially on master where all the action is :p09:30
odyssey4menothing gets to the stable branches without going through master ;)09:30
evrardjp[m]well I've seen queens doing well, so I am wondering why only master.09:31
evrardjp[m]maybe i am wrong and queens isn't well.09:31
evrardjp[m]but i am now more worried about pike, which was already slower than q09:31
odyssey4mePTO adding extra haproxy endpoints should not have done that - it's very odd that it did... there may have been some sort of data error in the config which got things mis-aligned somehow09:31
odyssey4meevrardjp[m] yep, me too09:32
odyssey4meI haven't checked into upgrades. Will take a look later today.09:32
evrardjp[m]i would be happy if we could check the docs together odyssey4me09:33
evrardjp[m]I've got a few more patches in.09:33
evrardjp[m]if we are good in terms of practices, we can backport.09:33
odyssey4meja, saw those - not sure how back portable some of those changes are as I noticed some were using the new play layout introduced by the nspawn patches - and those are master only09:34
odyssey4mebbiab - time for me to relocate09:34
evrardjp[m]yeah indeed, this is the only concern i have for those.09:35
*** shardy has joined #openstack-ansible09:36
*** Sha0000 has quit IRC09:37
*** ANKITA_ has quit IRC09:38
PTOodyssey4me: i have rolled back to a very old backup of the inventory file and it seems to have fixed the problem.09:39
PTOBy the way, is openstack-ansible-ops part of the openstack-ansible project?09:39
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: SUSE: Do not wipe DNS resolver entries from base image  https://review.openstack.org/54999309:47
hwoarangevrardjp[m]: queens/suse is fine. thanks for waiting09:47
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_container_create master: templates: networkd: UseDNS requires systemd-resolved  https://review.openstack.org/54987709:49
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_container_create master: templates: networkd: Drop Link=ARP from networkd configuration  https://review.openstack.org/54987809:49
*** Sha0000 has joined #openstack-ansible09:56
openstackgerritGaudenz Steinlin proposed openstack/openstack-ansible-haproxy_server master: Options to enable SSL for backend connections  https://review.openstack.org/54983409:58
*** gkadam has joined #openstack-ansible10:08
*** lvdombrkr89 has joined #openstack-ansible10:09
evrardjpthx.10:09
*** gkadam has quit IRC10:09
*** gkadam has joined #openstack-ansible10:10
*** SmearedBeard has quit IRC10:10
*** lvdombrkr has quit IRC10:11
*** gkadam has quit IRC10:13
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_neutron master: Provide a dynamic repo for ODL  https://review.openstack.org/54928810:13
*** gkadam has joined #openstack-ansible10:14
openstackgerritGaudenz Steinlin proposed openstack/openstack-ansible-haproxy_server master: Options to enable SSL for backend connections  https://review.openstack.org/54983410:15
*** gkadam has quit IRC10:15
*** gkadam has joined #openstack-ansible10:16
*** SmearedBeard has joined #openstack-ansible10:18
*** gkadam has quit IRC10:19
*** Sha0000 has quit IRC10:20
*** gkadam has joined #openstack-ansible10:20
*** gkadam has quit IRC10:21
*** gkadam has joined #openstack-ansible10:21
*** gkadam has quit IRC10:23
*** gkadam has joined #openstack-ansible10:23
*** gkadam has quit IRC10:26
*** gkadam has joined #openstack-ansible10:27
*** masber has joined #openstack-ansible10:28
*** gkadam has quit IRC10:28
*** gkadam has joined #openstack-ansible10:29
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_container_create master: tests: Use domain names for external network testing  https://review.openstack.org/54981710:29
openstackgerritManuel Buil proposed openstack/openstack-ansible-os_neutron master: Move the roles defined in meta to tasks  https://review.openstack.org/55000410:30
*** gkadam_ has joined #openstack-ansible10:32
*** gkadam has quit IRC10:34
*** gkadam_ has quit IRC10:35
*** gkadam_ has joined #openstack-ansible10:35
*** gkadam_ has quit IRC10:38
odyssey4mePTO yes it is - it's a place where anything goes, really - we don't scrutinise the content much - the idea is to incubate ideas or simply to share operations tools10:52
odyssey4meandymccr you around? if so, could you give this a bump? https://review.openstack.org/54997710:54
andymccrsure10:54
andymccrlemme look10:55
andymccrLGTM10:55
odyssey4methanks10:56
odyssey4mehwoarang do you mean to still -w on https://review.openstack.org/#/c/549784/ with all those other patches depending on it?10:57
hwoarangodyssey4me: it should be OK to go in but it doesn't fix the problem for suse yet.10:57
hwoarangneed one more patch10:58
*** woodard has joined #openstack-ansible10:59
*** woodard has quit IRC11:04
openstackgerritMerged openstack/openstack-ansible-openstack_hosts stable/queens: SUSE: Switch OpenStack repo to Queens  https://review.openstack.org/54997711:10
*** ANKITA has quit IRC11:34
hwoarangodyssey4me: did you get anywhere with the connection plugin issues11:37
hwoarangi couldn't follow up the discussion on that front11:37
odyssey4mehwoarang We're waiting for cloudnull on that front.11:39
hwoarangok11:39
aspiershi all :)11:44
odyssey4meo/ aspiers :)11:45
aspiersanyone who was at the PTG: please ping me if you have any questions about https://github.com/aspiers/ansible-viz11:45
*** stuartgr has joined #openstack-ansible11:45
*** gkadam has joined #openstack-ansible11:50
*** sep has joined #openstack-ansible11:50
odyssey4meevrardjp it looks like newton->ocata upgrades are completing successfully, ocata->pike are timing out, and pike->queens/queens->master are not running at all11:52
*** woodard has joined #openstack-ansible11:53
odyssey4meevrardjp ah ok, the job was renamed11:54
odyssey4mepike->queens is timing out, queens->master is failing11:54
*** hw_wutianwei has quit IRC11:54
*** mrch has joined #openstack-ansible11:55
*** hw_wutianwei has joined #openstack-ansible11:55
*** MeltedLux has quit IRC11:56
*** radeks has quit IRC11:56
*** MeltedLux has joined #openstack-ansible11:56
mbuilfunctional-centos-7 and functional-opensuse423 gates are failing right?11:56
odyssey4membuil in master, quite a few are failing due to some recent changes - hwoarang is digging into what's going on there11:57
*** woodard has quit IRC11:57
mbuilodyssey4me: ok. We are in good hands then11:57
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Change timeout for upgrade periodics from 3hrs to 4hrs  https://review.openstack.org/55003112:02
*** hw_wutianwei has quit IRC12:05
*** mrch has quit IRC12:08
*** hw_wutianwei has joined #openstack-ansible12:08
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Remove meta job map for aio->aio_lxc  https://review.openstack.org/55004012:13
hwoarangodyssey4me: if you want to start voting on stuff in order to get them in that would be fine. bottom point is that without systemd-resolved, networkd is unable to get dns info over dhcp12:15
hwoarangso we need to hardcode them to the image. i will see if i can bring resolved to leap otherwise i may have to manually query the dhcp server or do another ugly hack12:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Remove meta job map for aio->aio_lxc  https://review.openstack.org/55004212:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/queens: Replace meta job map for aio->aio_lxc with simpler mechanism  https://review.openstack.org/55004012:21
odyssey4mehwoarang ok, that makes sense - so resolved is not currently in opensuse?12:22
hwoarangnope. i am talking to the maintainer to see how to get it in12:22
*** ansmith has quit IRC12:23
PTOodyssey4me: crewl. I plan to push my changes to the repo when it works. I would love to contribute to the project as much as I can12:24
odyssey4mePTO we would love that too!12:26
*** SmearedBeard has quit IRC12:26
*** chyka has joined #openstack-ansible12:27
*** chyka has quit IRC12:32
*** SmearedBeard has joined #openstack-ansible12:34
*** olivierbourdon38 has quit IRC12:36
openstackgerritOpenStack Proposal Bot proposed openstack/ansible-hardening master: Updated from global requirements  https://review.openstack.org/55004912:41
hwoaranglol this one is scary http://logs.openstack.org/93/549993/1/check/openstack-ansible-lxc-overlayfs-ubuntu-xenial/7162b81/job-output.txt.gz#_2018-03-06_12_25_36_69131712:45
*** olivierbourdon38 has joined #openstack-ansible12:49
nsinghhow can i fetch venv onfiguration if my role is on my system only(localhost)12:50
*** woodard has joined #openstack-ansible12:51
*** kstev has joined #openstack-ansible12:53
*** SmearedBeard has quit IRC12:53
evrardjp[m]aspiers : The installation seems straightforward, I wonder how long would it take in gates, and if it would be reliable in gates. If it's reliable (not too many external dependencies), I'd be enclined to add the generated html in our documentation.12:54
*** stuartgr has quit IRC12:55
*** woodard has quit IRC12:55
aspiersevrardjp[m]: it generally only takes a few seconds to run. The most time consuming bit would be installing Ruby and the gems, but I guess that could be cached somehow12:56
aspiersevrardjp[m]: +1 for auto-including in the docs! There could even be a Sphinx extension to do that?12:56
evrardjp[m]it would be a great doc page in our reference guide or, with the appropriate filters, in an appendix of our deploy guide.12:57
evrardjp[m]aspiers yes thats what i meant :)12:57
evrardjp[m]we do not really need an extension, we can run subprocess already ;)12:57
evrardjp[m]it would be cleaner though.12:58
evrardjp[m]I'll analyse this12:58
*** kstev1 has joined #openstack-ansible12:58
*** Sha0000 has joined #openstack-ansible12:59
*** kstev has quit IRC13:00
*** SmearedBeard has joined #openstack-ansible13:00
*** woodard has joined #openstack-ansible13:01
*** woodard has quit IRC13:02
*** woodard has joined #openstack-ansible13:03
aspiersevrardjp[m]: cool thanks!13:06
*** SmearedBeard has quit IRC13:07
*** gameon has quit IRC13:12
*** pylot has joined #openstack-ansible13:23
*** markvoelker has quit IRC13:24
*** markvoelker has joined #openstack-ansible13:24
*** udesale has quit IRC13:29
*** udesale_ has joined #openstack-ansible13:29
*** SmearedBeard has joined #openstack-ansible13:36
*** ansmith has joined #openstack-ansible13:39
*** szaher has quit IRC13:42
*** SmearedBeard has quit IRC13:44
*** szaher has joined #openstack-ansible13:49
*** SmearedBeard has joined #openstack-ansible13:51
*** esberglu has joined #openstack-ansible13:55
evrardjpaspiers: do you think it would be possible to have a different output than html? Or would that require too much work? http://www.sphinx-doc.org/en/stable/ext/graphviz.html13:58
evrardjpit looks like the viz.js is also very important14:00
*** szaher has quit IRC14:00
evrardjpI thought it was just rendering, but it's very detailed14:00
evrardjpMmm checking viz.js it looks like it should be possible14:01
*** szaher has joined #openstack-ansible14:07
evrardjpbecause it's the same tech14:09
evrardjpit is very weird to me that it's written in ruby, while the resolution could be done with python and ansible natively... :p14:12
*** electrofelix has quit IRC14:14
*** SmearedBeard has quit IRC14:14
*** Sha0000 has quit IRC14:18
*** sar has quit IRC14:18
*** admin0 has joined #openstack-ansible14:20
*** mrch has joined #openstack-ansible14:21
openstackgerritPerToft proposed openstack/openstack-ansible-ops master: Updatev version of ELK stack for openstack ansible  https://review.openstack.org/55008914:24
PTOI hope i have added the ELK stack correct for review, otherwise let me knwo14:26
evrardjpdear cloudnull, DimGR, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__, asettle,14:37
evrardjpcolinmcnamara, thorst, adreznec, eil397, qwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking14:38
evrardjpthe osa bug triage will start in 1h30'. Please have a look at the bug list before starting: https://etherpad.openstack.org/p/osa-bugtriage14:38
prometheanfireneat14:38
evrardjphey prometheanfire how are things?14:39
evrardjpare you on holiday for recovery, or are you already working?14:39
evrardjpPTO*14:39
prometheanfireI was back in yesterday :p14:40
*** idlemind has joined #openstack-ansible14:43
*** mrch has quit IRC14:46
openstackgerritMajor Hayden proposed openstack/openstack-ansible master: Use RDO Queens repo  https://review.openstack.org/54864114:49
openstackgerritMajor Hayden proposed openstack/openstack-ansible-openstack_hosts master: [WIP] Make role idempotent  https://review.openstack.org/55010214:53
evrardjpI was wrong it's not in 1h30 but 2h30.14:55
fusmuguys i want to assign role to user on different domain via os_user_role but it says that couldn't find such user, when I execute command locally everything is okey, could you give me some hints ? :)14:56
cloudnullmornings14:58
cloudnullhwoarang: odyssey4me: what are you waiting on me fore?15:00
odyssey4mecloudnull my memory is now hazy, but since the networkd and connection plugin changes, there's a bit of havok around - hwoarang is probably more up to speed and needs advice/help15:01
odyssey4mejmccrory FYI thanks to git-harry we know why nova/master is failing - there's a patch to nova for it: https://review.openstack.org/#/c/549737/115:01
* odyssey4me is trying to chat about that in nova to determine a suitable way forward15:02
hwoarangcloudnull: one thing is that without systemd-resolved  networkd can't fetch DNS stuff from dhcp or even use hardcoded DNS= entries from .network file15:03
hwoarangcloudnull: second thing is the new connection plugin for example http://logs.openstack.org/93/549993/1/check/openstack-ansible-lxc-overlayfs-ubuntu-xenial/7162b81/job-output.txt.gz#_2018-03-06_12_25_36_69131715:03
openstackgerritMerged openstack/ansible-hardening master: Updated from global requirements  https://review.openstack.org/55004915:04
openstackgerritMerged openstack/openstack-ansible-lxc_hosts stable/queens: Implement always tag for required fact  https://review.openstack.org/54980815:04
hwoarangcloudnull: re DNS, i have workarounded this now on SUSE by hardcoding dns entries and talking to systemd people to find out if we can get resolved on Leap. but until then i need to come up with a smarter way to figure out the right DNS for the containers.15:04
openstackgerritMerged openstack/openstack-ansible master: [Docs] Updating managing networks documentation  https://review.openstack.org/54929215:05
openstackgerritMerged openstack/openstack-ansible master: [Docs] Update scale environment  https://review.openstack.org/54962915:05
cloudnullhwoarang: interesting. I will look at the connection plugin issues today and try and work out a solution there.15:06
cloudnullhwoarang: re DNS, is that an issue with the gate because they use unbound locally ?15:06
hwoarangyep15:07
openstackgerritMajor Hayden proposed openstack/ansible-hardening master: Replace Fedora 26 with 27  https://review.openstack.org/55010915:10
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: test-log-collect.sh: Collect networkd information  https://review.openstack.org/54976415:11
cloudnullhwoarang: do we know where the unbound config is being generated ?15:12
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: templates: lxc-system-manage: Send correct DNS over dhcp  https://review.openstack.org/54978415:13
openstackgerritMajor Hayden proposed openstack/openstack-ansible-openstack_hosts master: [WIP] Make role idempotent  https://review.openstack.org/55010215:13
hwoarangcloudnull: over here i think https://github.com/openstack-infra/project-config/blob/master/nodepool/elements/nodepool-base/finalise.d/89-unbound15:14
hwoarangif we can find a way for containers to not end up with 'nameserver 127.0.0.1' that would be great :)15:15
hwoarangin centos/ubuntu i am not even sure if they get dns from teh dhcp. i think they simply get lucky by using 127.0.0.1 and they have systemd-resolved running which is the local resolver15:15
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: SUSE: Do not wipe DNS resolver entries from base image  https://review.openstack.org/54999315:16
openstackgerritMerged openstack/openstack-ansible-openstack_hosts master: Revert "Revert "SUSE: Switch OpenStack repo to Queens""  https://review.openstack.org/54974015:16
*** fusmu has quit IRC15:19
cloudnullthe containers on ubuntu and centos are pulling the dns names from resolvd.15:20
cloudnullwhich should have the same dns resolvers as the host15:20
hwoarangbut host has 127.0.0.115:20
hwoarangso i guess they just pull 127.0.0.1 in the end15:20
cloudnullit should be the actually forward lookup addresses.15:21
hwoaranghmm15:21
cloudnullso maybe in the event we detect nodepool we can capture the nameservers using something like15:22
cloudnull`sed -e 's/^  //g' -e 's/forward-addr:/nameserver/g' /tmp/forwarding.conf | grep -w '^nameserver'`15:22
cloudnullwe should also be able to set the resolver within the container to the gateway of the lxcbr0 interface15:23
hwoarangcloudnull: let me see what ubuntu gets as dns15:23
hwoarangcloudnull: i thought about it but i was not sure if lxcbr0 ip is always expected to be able to act as dns15:23
hwoarangnot sure what kind of setups are out there15:23
cloudnullsure15:23
hwoarangif you think that the bridge will also be able to do dns lookups then we can hardcode that in the base image i suppose15:24
hwoarangor we can do that during container setup using 'networkctl status eth0' or something15:25
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: templates: networkd: UseDNS requires systemd-resolved  https://review.openstack.org/54987715:25
cloudnullwe have dnsmasq running on the host and bound to lxcbr0 so it should just work15:26
hwoarangok i can bake a patch then15:28
*** masber has quit IRC15:28
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_hosts master: SUSE: Use the lxcbr0 IP as the initial DNS resolver  https://review.openstack.org/55012915:33
odyssey4mejmccrory git-harry I'll implement a workaround in os_nova as a temporary way of getting around the nova bug to buy time to allow the nova patch to get to a place where it merges.15:34
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create master: Use static MAC by default  https://review.openstack.org/55013115:35
odyssey4mehwoarang cloudnull that may work, but it's a dirty nodepool hack - what about deployers who do something similar... I think we should consider something more universal15:36
cloudnullodyssey4me: ++ the nodepool hack is not being used.15:36
odyssey4mecloudnull hwoarang question in review15:37
*** armaan has quit IRC15:40
*** armaan has joined #openstack-ansible15:41
*** jwitko has joined #openstack-ansible15:42
openstackgerritMajor Hayden proposed openstack/openstack-ansible-tests master: Improve idempotency  https://review.openstack.org/55013515:46
*** jbadiapa has quit IRC15:59
evrardjpoh great, forced reboot of my mac!16:03
evrardjpI'm glad the meeting is in an hour now :)16:04
openstackgerritMajor Hayden proposed openstack/openstack-ansible-openstack_hosts master: Remove kernel modules: scsi_dh + x_tables  https://review.openstack.org/54975416:04
openstackgerritMajor Hayden proposed openstack/openstack-ansible-openstack_hosts master: [WIP] Make role idempotent  https://review.openstack.org/55010216:04
cloudnullevrardjp: feeling better?16:05
*** jbadiapa has joined #openstack-ansible16:06
*** Sha0000 has joined #openstack-ansible16:06
evrardjp[m]no fever for now, but I still feel dizzy.16:13
evrardjp[m]how was your trip back ?16:14
evrardjp[m]my mac has decided to feel dizzy too, and needs rebooting and upgrades now :(16:14
*** lihi has quit IRC16:15
*** lihi has joined #openstack-ansible16:15
*** holser__ has joined #openstack-ansible16:17
*** Sha0000 has quit IRC16:18
*** gkadam has quit IRC16:20
*** throwsb1 has joined #openstack-ansible16:20
openstackgerritMerged openstack/openstack-ansible-os_nova stable/queens: Remove systemd conditionals  https://review.openstack.org/54623216:21
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/queens: Set the MTU regardless if an address is presnet  https://review.openstack.org/55014516:22
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: SUSE: Use the lxcbr0 IP as the initial DNS resolver  https://review.openstack.org/55012916:23
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/queens: Set the MTU regardless if an address is presnet  https://review.openstack.org/55014516:23
*** armaan has quit IRC16:26
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/ocata: Set the MTU regardless if an address is presnet  https://review.openstack.org/55014616:27
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/newton: Set the MTU regardless if an address is presnet  https://review.openstack.org/55014716:27
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/pike: Set the MTU regardless if an address is presnet  https://review.openstack.org/55014816:28
cloudnullif folks have a moment it'd be great to see this through https://review.openstack.org/#/q/topic:bug/1753543+(status:open+OR+status:merged)16:36
cloudnullsomething we've run into setting jumbo frames on a couple of our customer environments16:37
*** gkadam has joined #openstack-ansible16:37
*** gkadam has quit IRC16:37
hwoarangcloudnull: are the changes in line 9 and 11 intentional?16:38
hwoarangim curious it's a prefered coding style for jinja216:38
hwoarangsame for 17,18,20,2116:38
cloudnullyes. I just spaced out the jinja because it easier for me to read16:38
hwoarangok16:38
cloudnullthe meat is just the one line change, I could get rid of the indentation in jinja if folks think it best16:39
hwoarangi dont mind im just curious. and is it only debian that needs the mtu change?16:41
hwoaranganwering to myself: yes it is16:41
cloudnullyes the other OS's already do the right thing16:41
*** kevinbenton has quit IRC16:41
spotzPoor Debian....16:41
cloudnullas is the networkd bits in master.16:41
cloudnullso it's just poor, poor debian :)16:42
spotzDebian needs love you know... everyone says bad things about it:(16:42
cloudnullhaha16:42
evrardjpI propose we fix up things in newton (the mtu + the nova RPC thing for upgrades), then we can EOL it. Okay for everyone?16:43
openstackgerritMajor Hayden proposed openstack/openstack-ansible-tests master: Improve idempotency  https://review.openstack.org/55013516:43
cloudnullevrardjp: ++16:43
hwoarangfine by me16:43
hwoarangpoor newton16:43
spotzevrardjp: Let me know when you EOL it, need to make a doc change for it16:43
cloudnullchao newton, thanks for all the fish !16:43
*** kevinbenton has joined #openstack-ansible16:43
evrardjpspotz: this week? Could you include that doc patch for that time?16:44
spotznewton is still a thing for a little bit though, COA won't move off of it until April/May but they've got a mirror16:44
spotzevrardjp: Yeah just need to find the change:) I had it all prepped for a while then had to ditch it as git was complaining about my stash methods16:44
cloudnullhttps://youtu.be/N_dUmDBfp6k?t=8316:45
spotzevrardjp: As soon as you pull the plug and send to to EOL I'll get the patch up. It'll take me longer to find it then to do it16:46
openstackgerritMerged openstack/openstack-ansible-plugins master: Reduce interactions by nearly 50%  https://review.openstack.org/54969216:46
spotzevrardjp: I think I'd prepped it back in Sydney:)16:46
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-plugins stable/queens: Reduce interactions by nearly 50%  https://review.openstack.org/55015616:48
logan-very nice cloudnull16:49
openstackgerritMajor Hayden proposed openstack/openstack-ansible-openstack_hosts master: Make role idempotent  https://review.openstack.org/55010216:49
*** fusmu has joined #openstack-ansible16:50
spotzevrardjp: Ok I have the page but will need the date of EOL to update it16:52
spotzso just ping when done16:52
*** udesale_ has quit IRC16:53
evrardjpspotz: oh you need something done AFTER eol16:56
spotzevrardjp: Yeah I need to patch one doc page after we're EOL16:56
*** chyka has joined #openstack-ansible17:01
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-lxc_container_create master: Always create containers with fixed MAC addresses  https://review.openstack.org/55016017:02
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-plugins stable/pike: Reduce interactions by nearly 50%  https://review.openstack.org/55016117:02
hwoarangmhayden: sir, centos is a bit sick today http://logs.openstack.org/04/549204/4/check/openstack-ansible-functional-centos-7/27b04bb/job-output.txt.gz#_2018-03-06_16_47_28_84899317:03
hwoarangi've seen many jobs failing on this particular task17:03
hwoarang*centos jobs17:03
evrardjphey everyone17:04
evrardjpBug triage cloudnull, DimGR, andymccr, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, mhayden, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__,17:04
evrardjpasettle, colinmcnamara, thorst, adreznec, eil397, qwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking17:04
evrardjpHere is our bug list for today https://etherpad.openstack.org/p/osa-bugtriage17:04
evrardjp#startmeeting openstack_ansible_meeting17:04
openstackMeeting started Tue Mar  6 17:04:28 2018 UTC and is due to finish in 60 minutes.  The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot.17:04
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.17:04
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"17:04
openstackThe meeting name has been set to 'openstack_ansible_meeting'17:04
spotzo/17:05
evrardjp#topic focus of the week17:05
*** openstack changes topic to "focus of the week (Meeting topic: openstack_ansible_meeting)"17:05
*** chyka_ has joined #openstack-ansible17:05
evrardjpThis week's focus will be on releasing queens first17:05
evrardjpthen releasing all the other branches, as it should have been done during end of last month, which was ptg17:06
*** lvdombrkr89 has quit IRC17:06
*** chyka has quit IRC17:06
evrardjp#topic previous week's conversations17:06
*** openstack changes topic to "previous week's conversations (Meeting topic: openstack_ansible_meeting)"17:06
andymccro/17:06
andymccrsorry im late17:06
hwoarango/17:06
evrardjpwe had many conversations at the PTG, and it's too long to write them down here, so I'll try to write up a blog or ML post. Anyone willing to help is welcome17:07
spotzandymccr: You're back!!!!17:07
evrardjpwelcomed*17:07
andymccrspotz: its like i never left :D17:07
evrardjpok next topic17:07
spotz:)17:07
odyssey4meo/17:07
evrardjp#topic bug triage17:07
andymccrevrardjp: if you need help on the blog let me know - happy to read over it or w/e17:07
*** openstack changes topic to "bug triage (Meeting topic: openstack_ansible_meeting)"17:07
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/175354317:08
openstackLaunchpad bug 1753543 in openstack-ansible "LXC interface template does not set MTU for provider interface" [Undecided,New]17:08
odyssey4meI think cloudnull just pushed up a slew of patches for that17:08
evrardjplooks like it17:08
andymccryeah17:09
andymccri guess we can move to next17:09
andymccrand review the patches as usual17:09
evrardjpconfirmed at least :)17:09
evrardjpok next then17:10
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/175312517:10
openstackLaunchpad bug 1753125 in openstack-ansible "os_cinder : Ensure cinder api is available timeout" [Undecided,New]17:10
odyssey4meneed more info17:10
evrardjpIncomplete17:11
evrardjpindeed17:11
evrardjpnext17:11
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/175293217:11
openstackLaunchpad bug 1752932 in openstack-ansible "hostvars breaks on Queens" [Undecided,New]17:11
evrardjpwe've seen this at the PTG.17:12
evrardjpI'd like to debug this in more details. It sounds serious -- high or critical17:12
odyssey4meyeah, nice that we have a full config to work with :)17:12
evrardjpbut the conditions are not clear to me -- python/jinja/ansible version dependant17:12
*** dariko has joined #openstack-ansible17:12
andymccryeah that seems pretty critical17:12
evrardjpwe weirdly don't see that in gates.17:13
andymccryeah17:13
evrardjplet me leave it as unconfirmed now17:13
*** armaan has joined #openstack-ansible17:13
odyssey4memight have to use vagrant, or the mnaio to reproduce17:13
evrardjpif next week I haven't reported17:13
evrardjpthen we can continue17:14
evrardjpI'll assign that to myself17:14
evrardjpok for everyone?17:14
hwoarangοκ17:14
hwoarangok17:14
evrardjphaha17:14
evrardjpnext17:14
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/175289817:14
openstackLaunchpad bug 1752898 in openstack-ansible "gateway option does not work" [Undecided,New]17:14
*** dariko1 has joined #openstack-ansible17:15
*** dariko has quit IRC17:15
evrardjpI see this listed in all branches up to queens, probably master too.17:16
evrardjpso let me check if that's a docs bug or a code bug :)17:16
andymccrhmm17:17
evrardjphttps://github.com/openstack/openstack-ansible-lxc_container_create/blob/aee117fc09981930ac1500e6375905cea2bfaab9/templates/container_network.network.j2#L3117:17
*** olivierbourdon38 has quit IRC17:17
evrardjplooks okay to me17:17
spotzI blame mhayden....17:18
odyssey4meso it should work, but apparently doesn't17:18
evrardjpbut our test coverage doesn't test it17:18
evrardjphttps://github.com/openstack/openstack-ansible-lxc_container_create/blob/dc034ff5c32e373efeab127128327e361430b2c0/tests/group_vars/all_containers.yml#L2717:18
odyssey4meit might be because the containers use dhcp on eth0, which gives it a route17:18
odyssey4meso it's plausible that the option there was something that worked when we hard coded everything - newton onwards we rely more on auto config17:19
odyssey4meso perhaps we should just remove those docs and suggest static routes17:19
odyssey4meit would seem a better approach to me anyway17:19
odyssey4meactually, this is for a secondary interface17:20
odyssey4me...17:20
* odyssey4me needs to think about this more17:21
*** radeks has joined #openstack-ansible17:22
evrardjpdocs issue17:23
odyssey4meadding that extra gateway is only useful if you have disabled lxcbr0, otherwise eth0 has a default gw and that will be there first17:23
odyssey4meyou can't have two17:23
*** epalper has quit IRC17:24
evrardjpI think the code is fine indeed, on all branches.17:24
evrardjpMarked it invalid17:25
evrardjpTahvok: we might need your review on the docs, because you submitted this bug17:25
evrardjpok next17:27
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/175284817:27
openstackLaunchpad bug 1752848 in openstack-ansible "Cinder backup on ceph when ceph backend not on all cinder-volume" [Undecided,New]17:27
*** mbuil has quit IRC17:30
evrardjpDefinitely need more eyes17:30
hwoarangi can't parse it right now17:30
* hwoarang doesn't understand ceph that much yet17:30
evrardjphaha17:30
evrardjpI think what we can summarize from that is that we are checking what cinder backend is used for cinder_backend_rbd_inuse17:31
evrardjpbut we should probably have cinder_service_backup_driver17:31
evrardjpI mean cinder_service_backup_rbd_inuse17:31
evrardjpand rbd_in_use if either is true17:32
odyssey4mecurrently we only automagically set the right vars when there is swift in the env.. we haven't done the same for ceph yet, you have to set them yourself... and if you have a mixed env, then using global overrides in user_*.yml will break things17:33
odyssey4meso we should perhaps look at either simplifying the implementation and doing some clever things in the role, like we did for nova for the different settings used per hypervisor17:34
odyssey4meor we should wire ceph up in the group vars like we've done with swift17:34
*** rpittau has quit IRC17:34
pabelangerafternoon, any change I can get a few eyes on https://review.openstack.org/549887/ Remove fedora-26 nodes from testing (migrates to fedora-27)17:35
evrardjpI asked the overrides to make sure we can triage this appropriately17:36
evrardjpbut I expect this to be indeed a lack of user friendliness on our side -- poor variable wiring17:36
odyssey4mepabelanger if mhayden says it's fine, then it's fine to me - he likes breaking things :)17:36
evrardjppabelanger: I will have a look after the bug triage :)17:36
evrardjpok next bug17:37
pabelangerodyssey4me: evrardjp: thanks17:37
spotzodyssey4me: hhehe17:37
evrardjpok next17:37
*** dariko1 has quit IRC17:37
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/175207317:37
openstackLaunchpad bug 1752073 in openstack-ansible "Neutron firewall_v2 driver missing" [Undecided,New]17:37
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: templates: networkd: Drop Link=ARP from networkd configuration  https://review.openstack.org/54987817:38
evrardjpthat's pike17:38
hwoaranginteresting17:39
evrardjpour docs seem wrong if it doesn't work: https://docs.openstack.org/openstack-ansible-os_neutron/pike/configure-network-services.html#deploying-fwaas-v217:39
evrardjpdo we have a scenario for that?17:39
evrardjpnot sure17:39
odyssey4menope17:39
evrardjpwe should probably add a scenario to validate that17:40
evrardjpand update the docs17:40
evrardjpwho has cycle to work on that?17:40
odyssey4meyeah, hopefully it's not broken most of the time due to upstream bitrot17:40
evrardjpfirewalling, that must be interesting for some ppl I guess??17:40
odyssey4methis is accurate according to upstream docs: https://docs.openstack.org/neutron/queens/admin/fwaas-v2-scenario.html17:41
odyssey4mebut maybe we're missing something17:41
hwoarangi think that we prob forget to pull in some package or something17:41
evrardjpI think we should probably exercice it before confirming/infirming it.17:42
evrardjpdeny*17:42
odyssey4meyeah, a new scenario would be good17:43
evrardjpleaving it as is17:43
hwoarangi can look into that maybe i will ask mbuil or someone else from opnfv17:43
odyssey4meI'm afraid my networking knowledge is too sketchy.17:43
odyssey4meah yes, goo dplan17:43
evrardjphwoarang: oh that would be great17:43
hwoarangi will assign it to myself17:43
evrardjpgreat17:43
hwoarangand i will delegate :D17:43
evrardjphaha17:43
evrardjpnext17:44
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/175172217:44
openstackLaunchpad bug 1751722 in openstack-ansible "ceph_mon check fails" [Undecided,New]17:44
evrardjpandymccr: :D17:45
evrardjpI think we need to know more17:46
evrardjpabout the environment17:46
evrardjpis the ceph mon not authorized with current deploy keys?17:46
evrardjpI think it would be worth knowing what were the expectations and what's the failyre17:47
evrardjpso marking it as incomplete17:47
logan-ssh: connect to host 192.168.10.30 port 22: Connection timed out17:48
logan-doesn't look like a key issue17:48
odyssey4mewell, jrosser and boxrick discovered that there's a port check, then it fails back to an ssh connection17:49
odyssey4meI'm sorry - it's port check, then a "ping" check using the ansible module17:49
*** Sha0000 has joined #openstack-ansible17:49
evrardjpomg17:49
logan-yeah it just needs to turn into a wait_for_connection and get rid of the whole rest of the stuff17:50
odyssey4methis is fixed in ansible 2.4 I think, but a problem in 2.3 - and they work through a bastion so the initial check always fails17:50
logan-:/17:50
odyssey4melogan- I think that bit is in wait_for_connection too for 2.3, but in 2.4 it's done right... but yeah, perhaps some of the ceph stuff is still using older methods17:51
odyssey4mein their case they are maintaining a 2.3 fork with patches from newer ansible, which is less than ideal, but aapparently 2.3 is near EOL17:51
boxrickWe ended up backporting the fix to a fork of 2.3x we point at for now17:52
boxrickDoh connection fall17:52
boxrickFail17:52
jrosserthats all a bit confusing, if copying the priv key around allegedly fixes it, suggests its not really a connection issue?17:53
evrardjpYeah exactly.17:53
evrardjpI vote for incomplete17:54
evrardjpDid you see my comment I just posted?17:54
odyssey4mehttps://github.com/openstack/openstack-ansible-ceph_client/blob/159d8164f6080db68d641f0ff5a54b0fd92449d8/tasks/ceph_get_mon_host.yml#L16-L2817:54
odyssey4meit's actually doing ssh17:54
odyssey4methat;d be why17:54
evrardjpyeah.17:55
odyssey4mewow, it used to be even more interesting: https://github.com/openstack/openstack-ansible-ceph_client/commit/f30ee47ed0e423a0555888b45b14999e7da517bd17:55
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/newton: Set the MTU regardless if an address is present  https://review.openstack.org/55014717:56
odyssey4meah, and logan- is to blame :p https://github.com/openstack/openstack-ansible-ceph_client/commit/a039741521ae90b5003c21284c93dc887baa76df17:56
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/queens: Set the MTU regardless if an address is present  https://review.openstack.org/55014517:56
openstackgerritMerged openstack/ansible-hardening master: Update to fedora-27 for testing  https://review.openstack.org/54988717:56
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/pike: Set the MTU regardless if an address is present  https://review.openstack.org/55014817:56
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-lxc_container_create stable/ocata: Set the MTU regardless if an address is present  https://review.openstack.org/55014617:56
odyssey4meit would seem that we have many more tools in the box now to do this better - perhaps logan- could pick up doing this better?17:56
logan-yes please assign it to me17:56
evrardjpthat would be nice -- but I still think we should be aware of the expectations first.17:56
evrardjplogan-: done!17:57
evrardjpthanks logan- !17:57
evrardjpok last one17:58
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/175084117:58
openstackLaunchpad bug 1750841 in openstack-ansible "playbook fails if /opt is missing" [Undecided,New]17:58
odyssey4mepossible due to https://github.com/openstack/openstack-ansible-pip_install/blob/26a5868fff0f52db977e6be52dcc2786cf1b8c43/tasks/install_online.yml#L20 and many other places where we assume it's there17:59
odyssey4meit's a fair bug, I'd say confirmed17:59
odyssey4memedium/low, because clearly most environments have that folder otherwise we'd have resolved that by now18:00
evrardjpI'd like to see the task that fails18:01
evrardjpbecause https://github.com/openstack/openstack-ansible/blob/stable/pike/playbooks/common-tasks/set-upper-constraints.yml#L4118:01
evrardjpis the first I see18:01
evrardjpand it should be fairly standard18:01
evrardjpit's not like /opt/openstack-ansible18:01
*** holser__ has quit IRC18:01
evrardjpit's the whole damn /opt18:01
*** olivierbourdon38 has joined #openstack-ansible18:02
evrardjpok I will ask for more info, mark it incomplete18:02
evrardjpok for everyone?18:02
odyssey4mewe could add a docs entry which says that /opt, /var, /bin, etc must exist :p18:02
*** dariko has joined #openstack-ansible18:02
evrardjphahahaha18:02
evrardjpok on that fun note18:02
evrardjplet's wrap up18:03
evrardjpthanks everyone!18:04
evrardjp#endmeeting18:04
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: https://goo.gl/tTmdgs"18:04
openstackMeeting ended Tue Mar  6 18:04:46 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)18:04
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-03-06-17.04.html18:04
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-03-06-17.04.txt18:04
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-03-06-17.04.log.html18:04
jmccroryodyssey4me wouldn't we just need a workaround for 'queens' in only the master branch? https://review.openstack.org/#/c/547072/18:06
mhaydenhwoarang: looking at your centos shenanigans18:09
jmccroryservices should get bumped up to a new minimum version as they upgrade and restart, that's why the upgrade test is the only thing passing right now18:09
mhaydenhwoarang: weird, it fails with no stdout/stderr output18:09
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: tests: Use domain names for external network testing  https://review.openstack.org/54981718:11
mnaserhi osa friends18:16
mnaserwe use vendor data in nova but looks like it doesn't have much integration in the role18:16
mnaseris there some sort of pattern of 'upload this file into the container' that a role uses which i can take advantage of?18:16
mnaserok, not take advantage of, but replicate18:17
mnaserso that user can supply a local file or string and that would be the vendor data uploaded into the api and compute containers18:17
odyssey4meevrardjp https://review.openstack.org/#/c/550160/118:17
odyssey4memnaser hmm, we have some stuff in the horizon role - not sure how great it is, but there is something18:18
mnaseroh yeaaah18:18
mnaserthe logo and stuff like that18:18
odyssey4meyup18:18
*** dariko has quit IRC18:24
*** dariko has joined #openstack-ansible18:26
openstackgerritMerged openstack/openstack-ansible-lxc_hosts master: zuul: Add missing CentOS and openSUSE jobs  https://review.openstack.org/54924718:26
*** jamesdenton has joined #openstack-ansible18:27
goldenfriI am attempting to install an AIO using 17.0.0.0rc1 and getting an error running the setup-hosts.yml during the pip_install I don't see anything obviously wrong in big error block, if anyone has time can you take a look? http://paste.openstack.org/show/693409/18:27
openstackgerritMerged openstack/openstack-ansible-lxc_hosts stable/pike: Implement always tag for required fact  https://review.openstack.org/54980918:29
*** shardy has quit IRC18:32
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_nova master: Added vendor_data.json support  https://review.openstack.org/55018918:35
mnaserodyssey4me: i started a pattern and then realized.. oh hey!  i could do this better!18:35
cloudnullanyone around want to give this a love nudge https://review.openstack.org/#/c/55016018:36
*** chyka_ has quit IRC18:38
*** chyka has joined #openstack-ansible18:38
openstackgerritAndreas Jaeger proposed openstack/openstack-ansible-os_octavia stable/pike: Zuul: Remove project name  https://review.openstack.org/54193218:50
*** armaan has quit IRC18:56
openstackgerritMajor Hayden proposed openstack/openstack-ansible-openstack_hosts master: Make role idempotent  https://review.openstack.org/55010219:00
*** ivve has joined #openstack-ansible19:01
openstackgerritMajor Hayden proposed openstack/openstack-ansible-os_nova master: Simplify SELinux check  https://review.openstack.org/55020219:02
openstackgerritMajor Hayden proposed openstack/openstack-ansible-os_swift master: Simplify SELinux check  https://review.openstack.org/55020519:04
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_hosts master: Simplify SELinux check  https://review.openstack.org/55020619:07
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_tempest master: update for changes needed to pass refstack  https://review.openstack.org/55020719:09
prometheanfiremhayden cloudnull: ^19:09
*** poopcat has joined #openstack-ansible19:14
openstackgerritMajor Hayden proposed openstack/openstack-ansible-haproxy_server master: Simplify SELinux check  https://review.openstack.org/55020919:15
openstackgerritMajor Hayden proposed openstack/openstack-ansible-os_neutron master: Simplify SELinux check  https://review.openstack.org/55021019:16
openstackgerritMajor Hayden proposed openstack/openstack-ansible-os_cinder master: Simplify SELinux check  https://review.openstack.org/55021119:16
openstackgerritMajor Hayden proposed openstack/ansible-hardening master: Simplify SELinux check  https://review.openstack.org/55021319:18
openstackgerritMajor Hayden proposed openstack/openstack-ansible master: Simplify SELinux check  https://review.openstack.org/55021519:19
mhaydenprometheanfire: dude, NICE19:19
prometheanfiremhayden: yarp19:30
odyssey4memnaser heh, I wasn't aware that vendor data was a nova.conf thing19:40
*** zul has quit IRC19:41
*** portdirect has quit IRC19:41
*** zul has joined #openstack-ansible19:41
*** portdirect has joined #openstack-ansible19:42
hwoarangcloudnull: let me know if you make any progress with the connection plugin issue so i can continue the investigation after you go to sleep :)19:44
hwoarangbecause i think this is the no1 blocking thing righ tnow19:45
odyssey4meevrardjp cloudnull hwoarang could do with a vote on https://review.openstack.org/550042 & https://review.openstack.org/550040 so that we see whether our upgrade periodics work again19:56
openstackgerritMohammed Naser proposed openstack/openstack-ansible-os_nova master: Added vendor_data.json support  https://review.openstack.org/55018919:57
mnaserwho broke da gate19:57
evrardjphaha19:58
odyssey4memnaser nova ;) https://review.openstack.org/54973719:58
evrardjpI want blames!19:58
mnaserih u renenber thus oatch19:58
mnaserpatch19:58
odyssey4mebut it is, to be fair, related to how we configure things19:58
mnaserwsgi19:58
mnaserbut also it was failing in os_keystone db check19:59
odyssey4menope, we're applying the upgrade section of nova.conf to greenfield deploys19:59
odyssey4meyes, saw that - no idea what's going on there19:59
odyssey4methat may need some digging19:59
mnaseri can dig lets see20:00
odyssey4meI'll work up a patch to get us moving on nova tomorrow, although it looks like that patch for nova is very likely to merge quite soon - we'll need a workaround to help our stable branches while we negotiate whether that nova patch is back portable.20:00
odyssey4methanks mnaser - I'm out for the night20:00
mnasero/ odyssey4me20:00
*** dariko has quit IRC20:06
*** dariko has joined #openstack-ansible20:06
*** olivierbourdon3- has joined #openstack-ansible20:13
*** olivierbourdon3- has quit IRC20:17
*** olivierbourdon3- has joined #openstack-ansible20:18
*** olivierbourdon38 has quit IRC20:18
*** olivierbourdon38 has joined #openstack-ansible20:19
*** dariko1 has joined #openstack-ansible20:21
*** olivierbourdon38 has quit IRC20:21
*** dariko has quit IRC20:21
*** oneswig has joined #openstack-ansible20:24
mhaydenso keystone db_sync is failing because keystone can't talk to mariadb on centos 720:26
prometheanfiremhayden: was just about to ask, thanks20:28
openstackgerritMajor Hayden proposed openstack/openstack-ansible-openstack_hosts master: Make role idempotent  https://review.openstack.org/55010220:29
* mhayden is looking into lawgs20:29
mhaydenso mariadb is up20:30
mhaydenand it has no errors on startup20:31
*** hw_wutianwei has quit IRC20:31
mhayden"/usr/sbin/mysqld: ready for connections."20:32
*** hw_wutianwei has joined #openstack-ansible20:32
mhaydenhmm20:32
*** hamza21 has joined #openstack-ansible20:35
mnasermhayden: selinux?20:41
mhaydenthat command is run inside the container, so selinux shouldn't get involved...20:42
mhaydenwe're using permissive in the gates right now anyway20:42
mhaydenin other news, i found a nice little nugger:20:44
mhaydennugget:20:44
mhaydenlxc_confile - confile.c:set_config_pivotdir:2262 - lxc.pivotdir is ignored.  It will soon become an error.20:44
mhaydeni guess we ought to stpo using pivotdir :P20:44
mhaydenwow, why can i not spell?20:44
mhaydenstpo!20:45
mhayden:P20:45
openstackgerritMerged openstack/openstack-ansible-lxc_container_create stable/queens: Set the MTU regardless if an address is present  https://review.openstack.org/55014520:51
openstackgerritMerged openstack/openstack-ansible-lxc_container_create stable/ocata: Set the MTU regardless if an address is present  https://review.openstack.org/55014620:51
openstackgerritMerged openstack/openstack-ansible-lxc_container_create stable/newton: Set the MTU regardless if an address is present  https://review.openstack.org/55014720:51
mhaydencloudnull: i think something in this patch might have messed up lxc networking in centos -> https://github.com/openstack/openstack-ansible-lxc_hosts/commit/bf143155f4758025be6767ace0638a06626effad20:53
mhaydenhaven't pinned it down yet20:53
openstackgerritMerged openstack/openstack-ansible-lxc_container_create stable/pike: Set the MTU regardless if an address is present  https://review.openstack.org/55014820:55
*** dariko1 has quit IRC21:02
*** prometheanfire has quit IRC21:10
*** ansmith has quit IRC21:13
-openstackstatus- NOTICE: The infrastructure team is aware of replication issues between review.openstack.org and github.com repositories. We're planning a maintenance to try and address the issue. We recommend using our official supported mirrors instead located at https://git.openstack.org.21:17
*** hamza21 has quit IRC21:22
*** dariko has joined #openstack-ansible21:22
*** yifei has quit IRC21:23
*** yifei has joined #openstack-ansible21:23
*** hamza21 has joined #openstack-ansible21:24
*** prometheanfire has joined #openstack-ansible21:28
*** yifei has quit IRC21:38
*** yifei has joined #openstack-ansible21:39
*** afred312 has quit IRC21:41
*** dave-mccowan has quit IRC21:49
*** threestrands has joined #openstack-ansible21:54
*** admin0 has quit IRC21:55
openstackgerritRussell Tweed proposed openstack/ansible-hardening master: Add check to ClamAV task to prevent simeltaneous content updates  https://review.openstack.org/55026822:00
openstackgerritMajor Hayden proposed openstack/openstack-ansible-lxc_container_create master: Flush entire interface rather than just routes  https://review.openstack.org/55026922:01
openstackgerritMajor Hayden proposed openstack/openstack-ansible-tests master: Improve idempotency  https://review.openstack.org/55013522:01
mnaserhmm.22:13
mnaserhttps://github.com/openstack/openstack-ansible/commit/4e6e87afdf8c69dc29269f46d39fa049af4dc45c22:13
mnaseris it possible that hostvars[physical_host] is empty because no tasks have ran on this host?22:13
mnaserrunning this playbook https://github.com/openstack/openstack-ansible/blob/stable/pike/playbooks/os-nova-install.yml22:13
mnaserit doesn't run anything against physical infra, so it wont collect any facts from the physical host22:13
*** esberglu has quit IRC22:13
mnaserso when it reaches the conductor installation.. it blows up22:14
mnaserhwoarang: ^ maybe of interest to you?22:14
* mhayden hopes that https://review.openstack.org/550269 will please centos' container networking again22:15
*** hamza21 has quit IRC22:16
*** oneswig has quit IRC22:16
openstackgerritRussell Tweed proposed openstack/ansible-hardening master: Add check to ClamAV task to prevent simeltaneous content updates  https://review.openstack.org/55026822:20
pabelangerspeaking of hostvars[physical_host], is there a good sample inventory file that shows what that looks like? I haven't really looked for it as an FYI22:22
*** ansmith has joined #openstack-ansible22:25
*** radeks has quit IRC22:34
mnaserpabelanger: i think there's a whole layout that is generated by dynamic_inventory.py22:34
*** admin0 has joined #openstack-ansible22:40
openstackgerritRussell Tweed proposed openstack/ansible-hardening master: Add check to ClamAV task to prevent simeltaneous content updates  https://review.openstack.org/55026822:44
openstackgerritRussell Tweed proposed openstack/ansible-hardening master: Add check to ClamAV task to prevent simeltaneous content updates  https://review.openstack.org/55026822:46
*** masber has joined #openstack-ansible22:50
*** masuberu has joined #openstack-ansible22:55
*** masber has quit IRC22:59
pabelangermnaser: okay, I'll have to poke at it23:05
pabelangerand see if I can get zuulv3 to use that connection pluging23:05
*** admin0 has quit IRC23:12
*** jwitko_ has joined #openstack-ansible23:13
*** jwitko has quit IRC23:17
*** Sha0000 has quit IRC23:18
*** jwitko_ has quit IRC23:21
*** throwsb1 has quit IRC23:34
*** jafeha has quit IRC23:34
*** jafeha has joined #openstack-ansible23:35
*** jwitko has joined #openstack-ansible23:45
*** dariko has quit IRC23:46
*** woodard has quit IRC23:58
*** woodard has joined #openstack-ansible23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!