Wednesday, 2018-05-09

*** mma has joined #openstack-ansible00:01
*** mma_ has joined #openstack-ansible00:08
*** mma has quit IRC00:09
*** MasterofJOKers has quit IRC00:17
*** MasterofJOKers has joined #openstack-ansible00:18
*** ebbex has quit IRC00:18
*** ebbex has joined #openstack-ansible00:20
*** gyee has quit IRC00:36
*** sawblade6 has joined #openstack-ansible00:46
*** nicolasbock has quit IRC00:57
*** cjloader has joined #openstack-ansible01:14
*** armaan has quit IRC01:15
*** armaan has joined #openstack-ansible01:16
*** cjloader has quit IRC01:19
*** username_ has joined #openstack-ansible01:21
*** ianychoi has joined #openstack-ansible01:21
rschulmanAfter making a change to the openstack_user_config.yml file, is rerunning the ansible playbooks from the beginning (setup-hosts.yml, that is) the right idea?01:21
*** username_ is now known as username__01:23
cloudnullrschulman kinda01:40
cloudnulldepends on the change01:40
cloudnullto be safe yes, once you get a handle on everything maybe no01:41
rschulmanThat makes sense, thanks.01:41
cloudnullif you changed something like the provide_networks section you should rerun setup-hosts (container-create.yml)01:41
cloudnullif you just changed some host_vars or host specific propertie, like cinder overrides, then just the os-cinder-install.yml playbook would be all thats needed to enact that change.01:42
cloudnullsame goes with nova-compute, etc01:42
cloudnullthe openstack_user_config is a doorway into inventory01:43
cloudnullrunning setup-everything.yml is an easy way to make sure nothing is missed, though it can be time consuming01:43
* cloudnull knows that answer is super nebulous sorry.01:44
rschulmanNo, it makes total sense.01:53
rschulmanI changed cinder backends, so I'm just rerunning everything from the top to be sure. :)01:54
rschulmanand also its a test lab so nobody is using it yet.01:54
*** coolj has quit IRC01:57
*** coolj has joined #openstack-ansible01:58
*** mwarad has joined #openstack-ansible02:09
*** vnogin has joined #openstack-ansible02:11
*** cjloader has joined #openstack-ansible02:15
*** vnogin has quit IRC02:16
*** cjloader has quit IRC02:19
*** username__ has quit IRC02:35
*** mwarad has quit IRC03:35
*** klamath has quit IRC03:35
*** udesale has joined #openstack-ansible03:36
*** threestrands has joined #openstack-ansible03:38
*** threestrands has quit IRC03:38
*** threestrands has joined #openstack-ansible03:38
*** dave-mccowan has quit IRC03:48
*** klamath has joined #openstack-ansible03:50
*** udesale has quit IRC04:00
*** udesale has joined #openstack-ansible04:01
*** osnaya has quit IRC04:04
*** klamath has quit IRC04:27
*** udesale_ has joined #openstack-ansible04:36
openstackgerritTaseer Ahmed proposed openstack/openstack-ansible-os_blazar master: Introduce os_blazar role in gerrit  https://review.openstack.org/56711204:38
*** udesale has quit IRC04:39
nsinghhi all good morning04:51
nsinghGetting "ERROR! Ansible sub-program not implemented: openstack-ansible" while running any playbook.04:51
nsinghAny clue?04:52
*** mma_ has quit IRC04:55
*** klamath has joined #openstack-ansible05:00
cloudnullnsingh sounds like ansible is not quite right05:02
cloudnulli'd rerun the bootstrap-ansible.sh script05:02
cloudnullin ./scripts/bootstrap-ansible.sh05:03
*** radeks__ has joined #openstack-ansible05:05
*** tomtom001 has quit IRC05:05
*** tomtom001 has joined #openstack-ansible05:06
*** xar- has quit IRC05:12
*** klamath has quit IRC05:12
*** cjloader has joined #openstack-ansible05:13
*** cjloader has quit IRC05:18
*** udesale_ has quit IRC05:22
*** klamath has joined #openstack-ansible05:28
*** klamath has quit IRC05:32
*** chas_ has joined #openstack-ansible05:38
*** chas_ has quit IRC05:38
*** radeks_ has joined #openstack-ansible05:40
*** radeks__ has quit IRC05:46
*** evin has quit IRC05:47
*** udesale_ has joined #openstack-ansible05:51
nsinghcloudnull: ooh ok.05:51
*** udesale_ has quit IRC05:53
*** udesale has joined #openstack-ansible05:53
*** threestrands has quit IRC05:54
*** pcaruana has joined #openstack-ansible05:57
*** bhujay has joined #openstack-ansible05:59
*** armaan has quit IRC06:05
*** armaan has joined #openstack-ansible06:05
*** cjloader has joined #openstack-ansible06:13
*** armaan_ has joined #openstack-ansible06:13
*** evin has joined #openstack-ansible06:15
*** armaan has quit IRC06:16
*** cjloader has quit IRC06:18
*** dxiri has quit IRC06:20
*** dxiri has joined #openstack-ansible06:21
*** dxiri has quit IRC06:25
*** dxiri has joined #openstack-ansible06:26
*** mma has joined #openstack-ansible06:32
*** MasterofJOKers_ has joined #openstack-ansible06:33
*** armaan_ has quit IRC06:33
*** armaan has joined #openstack-ansible06:34
*** MasterofJOKers has quit IRC06:34
*** MasterofJOKers_ is now known as MasterofJOKers06:34
*** mbuil has joined #openstack-ansible06:35
evrardjpnsingh: if you're doing role test,that shouldn't be required06:49
nsinghevrardjp: i am working on masakari-monitor in my env. Thats why i need that.06:51
nsinghevarardjp: and i updated https://review.openstack.org/#/c/566538/06:52
odyssey4meevrardjp could you go through these please - this is a finalization based on our engineering ops group feedback and my own testing to validate: https://review.openstack.org/#/q/I196f4e29cfef5f0eb0fcacf898ded877672d99f907:12
odyssey4meGiven that feedback, I think the merged fixes may also be a little too brutal, so I'll push some patches up to correct the solution for Queens and below.07:13
*** epalper has joined #openstack-ansible07:13
hwoaranggood morning.07:21
hwoarangodyssey4me: evrardjp re https://review.openstack.org/#/c/562606/ the experimental job is not passing due to unrelated error. is the experimental job a requirement for this review to get in?07:22
hwoarangmaybe we can create a DNM review and Depend-On this patchset instead?07:22
odyssey4mehwoarang sure, can do that - the test is to verify whether merging it will break the integrated gate07:23
odyssey4methat's a pretty odd failure07:24
hwoarangok07:25
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: [DNM] - Testing AIO on openSUSE  https://review.openstack.org/50745107:27
hwoarangi dont understand the failure to be honest and I have no time to dig into it right now07:27
*** bertys_ has joined #openstack-ansible07:27
*** pmannidi has quit IRC07:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/queens: Fix LXC cache resolvers more appropriately  https://review.openstack.org/56713707:30
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/queens: Fix LXC cache resolvers more appropriately  https://review.openstack.org/56713707:31
odyssey4meyeah, me neither unfortunately07:31
*** masber has quit IRC07:36
*** tosky has joined #openstack-ansible07:37
jrosserwow pip_install : Install PIP -------- 6567.58s07:43
*** armaan has quit IRC07:50
*** armaan has joined #openstack-ansible07:51
*** electrofelix has joined #openstack-ansible07:51
odyssey4meouch07:53
jrosseri guess this doesnt hurt too much in the CI but if you have any reasonable number of nodes its gonna hurt badly07:54
jrosserparticulary if ansible forks < number of nodes07:55
*** rpittau has joined #openstack-ansible07:57
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/queens: Fix LXC cache resolvers more appropriately  https://review.openstack.org/56713708:00
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_tempest master: Prefer distribution packages for Tempest dependencies  https://review.openstack.org/56260608:03
jrosseroh no it's doing another round of pip_install in setup_infrastrcuture :(08:04
*** DanyC has joined #openstack-ansible08:05
*** armaan has quit IRC08:05
evrardjpnsingh: yes08:06
*** armaan has joined #openstack-ansible08:06
evrardjpnsingh: it's good for me as is08:06
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible stable/ocata: Prepare gate-check-commit.sh for newton EOL  https://review.openstack.org/56658708:09
odyssey4mejrosser the long time taken is due to the repo server not having the wheels yet?08:09
evrardjphwoarang: it's not a requirement, yes a DNM and depends on is fine08:09
jrosserodyssey4me: looks like its another thing trying to hit the repo, which isnt there yet http://paste.openstack.org/show/720641/08:10
jrosseri wonder if i should just burn the whole deployment and apply your patches you linked to yesterday?08:11
odyssey4meah yes, that's what this fix is for: https://review.openstack.org/#/c/564422/08:12
jrosserooh another one :)08:12
odyssey4methat whole chain has struggled to gate, and I haven't had much time to dig into what's going on to try and get it resolved08:13
jrosserso far i've got 3 i think? https://review.openstack.org/#/c/563114/6 https://review.openstack.org/549012 https://review.openstack.org/#/c/564422/08:13
odyssey4meI am strongly considering reverting the pypiserver/pypi-reverse proxy patches in Queens given the trouble they've created... but it was a complex chain of patches, and I don't want to break it any worse either08:14
odyssey4meyep08:14
jrosserhappy to test those out for you08:14
odyssey4me#3 should be applied first, then #1, then #208:14
odyssey4meif you loko at 'related patches' for the first one, you'll see them in their reverse order08:15
odyssey4methe one on the bottm is applied first, then the others on top in the order shown from bottom to top08:15
odyssey4meevrardjp hwoarang if you have a gap to inspect https://review.openstack.org/567137 I'd appreciate it... it's a more subtle approach to solving the resolver issue in the base cache, rather than the current hammer approach which may break things if people are depending on resolvconf in any way08:19
hwoarangodyssey4me: i was looking at it earlier on. one point to consider is that not all distros download the same image so the /etc/resolv.conf symlink may not exist everywhere. suse for example is not using resolvd08:20
hwoarangso i am still thinking if the new solution will handle these cases as well08:20
odyssey4mehwoarang that's true for Rocky only, but not for the stable branches08:20
odyssey4mebut yes - the new solution works better anyway - because it tests for a symlink or static file08:21
hwoarangi think at least for suse that's true everywhere because resolvd is not available i believe08:21
hwoarangyeah08:21
odyssey4meas far as I know it's systemd-resolvd that's not available... whereas resolvd is08:21
hwoarangoh that's what i though you meant by 'resolvd'08:22
hwoarangnot sure what resolvd is :p08:23
odyssey4meI might be using the wrong name... there's that service that checks the network interface config and dynamically builds the resolv.conf file08:23
hwoarangresolvconf ?08:23
odyssey4meyes!08:24
*** dxiri has quit IRC08:24
*** dxiri has joined #openstack-ansible08:25
*** udesale_ has joined #openstack-ansible08:28
*** udesale has quit IRC08:28
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/queens: Fix LXC cache resolvers more appropriately  https://review.openstack.org/56713708:32
odyssey4methanks evrardjp and hwoarang  - updated08:32
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: [DNM] - Testing AIO on openSUSE  https://review.openstack.org/50745108:43
TahvokGuys, what are your working hours? I see you here 24/7 XD08:43
TahvokAnyway, good morning08:43
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-repo_build master: tasks: repo_build_wheels: Collect more information from the wheel build  https://review.openstack.org/56715608:50
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: [DNM] - Testing AIO on openSUSE  https://review.openstack.org/50745108:51
*** tomtom001 has quit IRC08:51
jrosserodyssey4me: if https://review.openstack.org/#/c/564422/ requires the rax mirror to work to pass the gate then it might need trusted-host:... adding to the pip config?08:52
*** tomtom001 has joined #openstack-ansible08:52
odyssey4mejrosser hmm, I'm not following?08:53
odyssey4mewhat're you seeing?08:53
jrossermirrors are added, and ignored08:53
jrosser"The repository located at mirror.ord.rax.openstack.org is not a trusted or secure host and is being ignored."08:54
jrosseri don't know if that matters or not, clearly it's being configured to be used but then is deemed untrusted08:54
odyssey4methat may be coming from the global host /etc/pip.conf08:55
jrosseri have to explicity trust my local pip mirror via trusted-host: or it doesnt get used08:55
odyssey4melemme look again08:55
jrosseri don't know if it actually breaks anything in this case, or just means that a more optimum mirror is ignored08:56
*** nicolasbock has joined #openstack-ansible08:56
odyssey4methat's happening because the default index is set to the repo server in http://logs.openstack.org/22/564422/4/check/openstack-ansible-deploy-aio_lxc-ubuntu-xenial/31c6c21/logs/ara-report/result/5039b12d-c112-42ac-90fe-5906e621ce80/08:58
odyssey4meand the global pip.conf still has the infra mirror08:58
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-repo_build master: tasks: repo_build_wheels: Collect more information from the wheel build  https://review.openstack.org/56715608:58
*** udesale_ has quit IRC08:59
*** udesale_ has joined #openstack-ansible08:59
*** udesale_ has quit IRC09:00
*** udesale has joined #openstack-ansible09:00
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Do not sync pre-configured /etc/pip.conf file  https://review.openstack.org/56715909:04
odyssey4mejrosser ^ that may help reduce confusion, and things have changed since we implemented that anyway now that we use the mirror script09:04
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_tempest master: DNM - Testing AIO jobs with Tempest  https://review.openstack.org/56716009:05
jrosserodyssey4me: anything that reduces the pip related noise currently will be good09:06
*** gkadam__ has joined #openstack-ansible09:06
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: [DNM] - Testing AIO on openSUSE  https://review.openstack.org/50745109:06
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/pike: Fix LXC cache resolvers more appropriately  https://review.openstack.org/56716109:07
openstackgerritChristian Zunker proposed openstack/openstack-ansible-os_cloudkitty master: Configure cloudkitty-api to use Apache + mod_wsgi  https://review.openstack.org/56716809:16
openstackgerritMerged openstack/ansible-hardening master: Use absolute path for aide binary in cronjob  https://review.openstack.org/56596009:26
*** Taseer has quit IRC09:30
openstackgerritlei zhang proposed openstack/openstack-ansible master: Fix the incorrect cirros default password  https://review.openstack.org/56717209:30
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Adjust inside/outside openstack-ci bootstrap-host tasks  https://review.openstack.org/56715909:36
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: [DNM] - Testing AIO on openSUSE  https://review.openstack.org/50745109:38
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_tempest master: Prefer distribution packages for Tempest dependencies  https://review.openstack.org/56260609:38
*** kysse has joined #openstack-ansible09:43
hwoarangodyssey4me: got a sec?10:07
odyssey4mehwoarang sure10:07
hwoarangre the wheel build failures for experimental jobs10:07
hwoarangso the failing file is this http://logs.openstack.org/51/507451/61/check/openstack-ansible-deploy-aio_metal-ubuntu-xenial/b1264db/logs/host/repo/requirements.txt10:07
odyssey4mehmm, that underscore10:07
hwoarangnotice the '-' right before 'a' in the first times on the list10:07
odyssey4meyeah, also the 'a', 'c', 'e'... etc10:08
hwoarangi am guessing we hit some  sort of error in the script that generates that thing10:08
odyssey4meit looks like a var being fed there isn't being interpreted correctly10:08
hwoarangdo you know how it's being generated?10:08
odyssey4meah, I see the {} too10:08
odyssey4meso yeah, it's likely some sort of jinja thing - let's check the py_pkgs output10:08
hwoarangok so that's the thing that messes things up10:09
odyssey4meinteresting - that's coming from the py_pkgs lookup: http://logs.openstack.org/51/507451/61/check/openstack-ansible-deploy-aio_metal-ubuntu-xenial/b1264db/logs/ara-report/result/1c802389-70b3-4346-a901-0aae4a70ce90/10:09
odyssey4melxc_cache_distro_packages10:10
odyssey4meof course, it reads that jinja var as a string and treats it as a list10:10
evrardjpmmm10:10
evrardjpI guess that's for me?10:10
odyssey4meoh hang on - distro_packages shouldn't get blended into the pip packages10:11
evrardjpwe need a rename10:11
evrardjpthat's not in pip packages10:12
evrardjpit's in role_distro_packages10:12
odyssey4meaha - it's os_tempest actually - see the pip packages in that list10:12
hwoarangso this only happens when we test this patchset https://review.openstack.org/#/c/562606/10:12
hwoarangwhich is in fact tempest yes, but i don't see why it would mess things up10:13
odyssey4meah, that's because of tempest_requires_pip_packages: "{{ _tempest_requires_pip_packages }}"10:13
hwoarangwhy would that be a problem?10:13
odyssey4mepy_pkgs doesn't interpret jinja, and reads the pip_packages as a list, always10:13
evrardjpyup10:13
odyssey4meif you fix that to be a list, not a string, it should be fine10:13
evrardjpit's time to get rid of this10:14
odyssey4meyes, working on that10:14
evrardjp(Never heard that before)10:14
hwoarangodyssey4me: like | list filter?10:14
evrardjphwoarang: the lookup is just a python script10:14
evrardjpwe can probably make it work with the {{10:14
hwoarangoh i see you mean that it does not understand what "{{ foo }}" is10:14
hwoarangit thinks it's a string10:15
evrardjpit just parses the yaml file, so we can say if it's jinja, ignore10:15
hwoarang:(10:15
evrardjpyes10:15
hwoarangok fair enough10:15
odyssey4mehwoarang well, either we need to recognise a string in the lookup and not think the string is a list... or that var needs to be tempest_requires_pip_packages: [ "{{ _tempest_requires_pip_packages }}" ] - but I think that'll break other things10:15
evrardjpbecause it expects a list, it just sees the list as string, and boom you have your elements! :)10:15
odyssey4meanother way to do it would be to not have the default var, and just do this in the package task: tempest_requires_pip_packages | default(_tempest_requires_pip_packages)10:16
evrardjpa string is always a list! :P10:16
evrardjpodyssey4me: that's quite elegant but you remove the documented part10:16
evrardjpI am not sure that needs overriding though10:17
evrardjpdoes it?10:17
odyssey4mepersonally I'd just get rid of the defaults file entry and adjust the task file - if documenting the available var to override packages is important, just make the var commented10:17
evrardjpfor the requires packages I mean10:17
odyssey4meI don't think it needs documenting personally10:17
odyssey4methat's not really something we want people to muck with10:17
evrardjpyeah, on this case, yes10:17
evrardjpso I'd say we can simply have tempest_requires_pip_packages somewhere in a vars file10:18
evrardjpand nothing in defaults10:18
hwoarangso if i move that exact line in vars/main.yml will it blow up again?10:18
odyssey4methat's an option too10:18
odyssey4meI think the lookup checks defaults and vars.10:18
odyssey4meSo yes.10:18
evrardjpI think in the future, when we need an user override, it needs to 1) go into defaults, loading standard "vars/", 2) Add the documentation in the defaults/ saying WHY we needed this override.10:19
evrardjp(a comment)10:19
evrardjpodyssey4me: it does10:19
hwoarangso better move that line to var/$distro.yml ?10:19
evrardjpyes10:19
odyssey4meyep10:19
evrardjpremoving the default10:19
hwoarangok10:19
hwoaranganother day another bug10:19
evrardjp:)10:20
evrardjpwell10:20
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Add missing api_paste sections  https://review.openstack.org/56655210:20
evrardjpnow that I saw the py_pkg output, I realize we have to do it in many other places10:20
evrardjp:)10:20
evrardjpthanks hwoarang !10:20
evrardjp:D10:20
odyssey4mehttps://review.openstack.org/#/c/564841/ would simplify the output, but unfortunately it causes some sort of module failure10:20
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_tempest master: Prefer distribution packages for Tempest dependencies  https://review.openstack.org/56260610:21
odyssey4meevrardjp if we get https://review.openstack.org/#/c/564841/ done then no, we don't - because all those other places are distro package lists10:21
odyssey4meheh, I think I see the problem10:21
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins master: Remove distro package indexing from py_pkgs lookup  https://review.openstack.org/56484110:24
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Use tags only once  https://review.openstack.org/56654710:24
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Use two seperate service variables  https://review.openstack.org/56654810:24
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Loosen up the sqlalchemy requirements  https://review.openstack.org/56654510:24
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Use cloudkitty_collector also for section config  https://review.openstack.org/56655110:24
evrardjpodyssey4me: yes on that patch you need to change some things :)10:25
evrardjpok you've done it10:26
evrardjpgreat10:26
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_almanach stable/queens: Be clear about python3: Remove its support  https://review.openstack.org/56659910:34
openstackgerritMerged openstack/openstack-ansible-os_masakari master: Add Maskari base jobs  https://review.openstack.org/56653810:38
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-os_almanach stable/queens: Be clear about python3: Remove its support  https://review.openstack.org/56659910:39
*** Taseer has joined #openstack-ansible10:41
openstackgerritJean-Philippe Evrard proposed openstack/ansible-role-systemd_networkd master: Add Bionic testing  https://review.openstack.org/56701610:42
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-lxc_hosts stable/ocata: Fix LXC cache resolvers more appropriately  https://review.openstack.org/56718410:43
jrosserodyssey4me: pip_install : Install PIP --------- 21.71s10:50
jrosserbetter :)10:50
odyssey4mejrosser hopefully for rocky we'll be doing that a *lot* less given that we're shifting to using packages on the host where possible, and only doing pip installs where we must on the host10:53
odyssey4mewe'll still do it in the venvs, but that'll be much more isolated10:53
jrosserbtw thats those 3 patches on top of stable/queens10:54
odyssey4meah, so it works - the tests just keep getting stuck on bad hosts then :/10:54
jrosseri think so yes - there seems to be a huge amount of random fail right now10:54
odyssey4meyep, we got no favours thanks to spectre/meltdown10:55
odyssey4meeveryting has slowed down quite a lot10:55
*** jwitko_ has joined #openstack-ansible10:58
*** gkadam_ has joined #openstack-ansible10:59
*** gkadam__ has quit IRC10:59
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-openstack_hosts master: Add Bionic testing  https://review.openstack.org/56703810:59
nsinghodyssey4me: Add Maskari base jobs  https://review.openstack.org/566538 got merged. After this what should be my nect step? Should i upload masakari-monitor patch after test?10:59
odyssey4mensingh well, getting them into a working state would be good, so they can be marked voting11:00
odyssey4mebut yes, any additional patches can go up independently now11:00
odyssey4meonly in the role for now - nothing to the integrated build until the role's tests are passing and voting11:00
*** jwitko has quit IRC11:01
evrardjpyup that's the deal11:01
nsinghok. are you going to add other jobs for role test.11:01
evrardjpthe functional test is the one we should start with11:01
evrardjpof the role11:02
evrardjpbut yes, bionic will probably be added, but don't you worry about it for now :)11:02
nsingh:) ok11:02
odyssey4meevrardjp odd https://review.openstack.org/#/q/I11d069e2d0cb3becad067689fa8f1e4c9391f368 works fine for ocata and pike, but not for queens - I didn't think that queens had changed much, but it looks like it's failing to resolve DNS11:03
nsinghevrardjp: so if i want to test masakari-monitor with masakari. I need your help in configuration as monitor services will be install on compute host.11:04
evrardjpsmall steps at a time, first ensure the role works, then integrate it, document it, and ... then https://memegenerator.net/img/instances/59895613/whats-the-plan-world-domination.jpg11:05
nsinghIf i get this part that how i can install masakari-monitor services on all compute node then i can work on testing part for masakari-monitor and masakari11:05
nsinghOk ok.11:05
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Add Bionic testing  https://review.openstack.org/56719211:06
evrardjpodyssey4me: is resolveconf included in the base image, so that it can start?11:07
odyssey4mensingh if you need integration with other roles and different groups, take a look at something like the nova role which consumes neutron and cinder11:07
evrardjpbecause the image changed in Q right?11:07
odyssey4meevrardjp I thought the  image changed in R, not Q11:07
evrardjpnsingh: let's first fix the current issues, which is a difference between existing roles and your role. Then we can think of tweaking tests/inventory.ini11:08
evrardjpI thought it was, but I might be wrong11:08
odyssey4mensingh yeah, at least try to get the API service up and exercise some sort of test to validate that it's up if that's possible11:08
odyssey4mesome roles just do a port check11:08
odyssey4meit was in master, afaik, that we switched to using OCI images11:10
odyssey4meevrardjp ^11:10
evrardjpmmm ok11:11
openstackgerritMerged openstack/openstack-ansible stable/newton: Update release note for resolvconf cache prep failure  https://review.openstack.org/56696811:11
evrardjpodyssey4me: I don't have any machine to test . It would be super nice to see if the symlink/file we put back is good.11:12
nsinghodyssey4me: ok. i added that test https://github.com/openstack/openstack-ansible-os_masakari/blob/master/tests/test-masakari-functional.yml that checks masakari-api service11:12
evrardjpmaybe ensuring the resolvconf is well started11:12
nsinghi will test it in my env and let you know11:13
openstackgerritMerged openstack/openstack-ansible stable/queens: Update release note for resolvconf cache prep failure  https://review.openstack.org/56705211:19
openstackgerritMerged openstack/openstack-ansible stable/ocata: Update release note for resolvconf cache prep failure  https://review.openstack.org/56705011:19
odyssey4meevrardjp the trouble is that I'm getting inconsistent results - my ocata test failed (integrated build) and my queens test passed11:21
odyssey4mesomething is hinky11:21
*** ygl has joined #openstack-ansible11:21
yglhi11:21
yglI am unable to upload images from horizon in queens version of openstack-ansible11:21
yglcan someone help m eplease11:21
openstackgerritMerged openstack/openstack-ansible stable/pike: Update release note for resolvconf cache prep failure  https://review.openstack.org/56705111:22
evrardjpyes11:22
evrardjpygl:11:22
yglI also found this on google: https://bugs.launchpad.net/openstack-ansible/+bug/174768411:22
openstackLaunchpad bug 1747684 in openstack-ansible "Default Values Do Not Allow Image Uploads to Glance from Horizon" [Medium,In progress] - Assigned to Kevin Carter (kevin-carter)11:22
yglevrardjp: what is this actually about ? can you explain me a bit please11:22
evrardjpyes so11:22
evrardjpI voted on the patch to improve the usability right now :)11:22
yglevrardjp: can you please explain me a bit what this bug is about ?11:23
evrardjpbut before people were using their own user_variables.yml before11:23
evrardjps/before$//11:23
evrardjpso in the meantime the bug is fixed, you can have something like this in your environment:11:23
evrardjp(just a sec, trying to find my example)11:24
yglevrardjp: sure11:24
evrardjphttps://gist.github.com/evrardjp/5f1ae8d1afdf7b23d3224bc55388adab11:24
evrardjpygl that should help you ^11:25
yglevrardjp: can you please explain what this is about ?11:25
evrardjpI am not fully familiar with it:11:26
evrardjphttps://en.wikipedia.org/wiki/Cross-origin_resource_sharing11:26
evrardjpbut basically saying you allow url x to do y.11:26
evrardjpso here x is horizon calls11:26
yglevrardjp: ok then ?11:27
evrardjpand y is glance API calls (different url)11:27
yglevrardjp: so why is it not working in this case ?11:27
evrardjpit depends on your configuration11:27
evrardjpdepends on what you used as urls11:27
yglevrardjp: but my horizon is working fine11:27
yglevrardjp: where is this CORS configuration stored ? in glance.conf ?11:28
evrardjpnot sure to understand your question11:29
openstackgerritMerged openstack/ansible-hardening master: Ensure that comments are not counted  https://review.openstack.org/56693611:29
yglevrardjp: where is this user_cors.yml file located ?11:30
evrardjpCORS is a standard spec11:30
evrardjpygl: oh you must create it for your deploy11:30
evrardjpin /etc/openstack_deploy/user_<whatever>.yml11:30
evrardjpuser_cors.yml was just the name for me to remember it.11:30
evrardjpbut it's possible you don't need that at all11:31
yglevrardjp: then in which files, the changes take place in openstack ? in horizon or in glance ?11:31
evrardjphttps://gist.github.com/evrardjp/5f1ae8d1afdf7b23d3224bc55388adab#file-user_cors-yml-L21 would impact glance.conf11:31
evrardjpbut https://gist.github.com/evrardjp/5f1ae8d1afdf7b23d3224bc55388adab#file-user_cors-yml-L19 would impact cinder.conf for example11:32
evrardjpI think for us to understand better what your use case is, you should paste on a website your openstack_user_config and your user variables.11:32
yglevrardjp: so basically what does it do when you say cors_override ?11:33
evrardjpcors_override in itself doesn't do anything11:33
evrardjpbut glance_glance_api_conf_overrides changes the glance config11:34
evrardjpand you pass it what you override in the glance config11:34
evrardjpso here, a "cors" section in the ini file, with https://gist.github.com/evrardjp/5f1ae8d1afdf7b23d3224bc55388adab#file-user_cors-yml-L11-L14 as key/values11:34
yglevrardjp: what is overridden in glance exactly ?11:34
yglevrardjp: ok got it11:34
yglevrardjp: what is the ETA for this bug fix ?11:35
evrardjpyou can have my workaround in the meantime11:37
evrardjpthe bugfix simply helps you, and allows you to not define this variable yourself11:37
yglevrardjp: ok, thanks11:40
*** armaan has quit IRC11:41
*** ygl has quit IRC11:41
*** armaan has joined #openstack-ansible11:41
ionihello guys11:45
ionii have a situation and i don't know why it happend11:45
ionii updated to 16.0.13 and when running os-neutron playbook, compute nodes have now l3 and metadata-agent got installed11:46
ionii don't have compute nodes defined as network_hosts11:47
ionibut not all compute nodes have neutron agents installed11:47
*** jwitko_ has quit IRC11:48
evrardjpioni: what the version ebfore?11:50
ioni16.0.1011:50
ionievrardjp 16.0.1011:51
ionibut the thing is, only on one region does this11:51
evrardjpwhy not updating to latest btw ?11:52
evrardjp:)11:52
evrardjpmmm11:52
ioniit's the latest from pike?11:52
evrardjpit looks like a bad inventory11:52
evrardjpioni: yup11:52
evrardjphttps://docs.openstack.org/project-deploy-guide/openstack-ansible/pike/deploymenthost.html#install-the-source-and-dependencies11:52
ionievrardjp i don't understand what you mean11:53
ionii'm already on latest version from pile11:53
ionipike11:53
ioni16.0.1311:54
*** bhujay has quit IRC11:55
evrardjp16.0.12 I guess :)11:55
ionihttps://paste.xinu.at/p3YlT/11:55
ioninot sure how this happened11:55
evrardjpoh yes I've seen that in some inventories11:56
ionihow can i get rid of it?11:56
evrardjpthat's a good question11:56
evrardjplet me just ask you more questions first11:56
evrardjpcould you paste your inventory?11:57
evrardjp(make sure you remove public ips etc.)11:57
ionii could do it in private11:57
ioniit's fine?11:57
evrardjpfine for me11:57
openstackgerritAndy McCrae proposed openstack/ansible-config_template master: Allow default section in ini to be specified  https://review.openstack.org/56629211:57
hwoarangodyssey4me: evrardjp tempest finally passes in experimental https://review.openstack.org/#/c/562606/11:58
hwoaranghow can I convince you to get it in? :)11:58
hwoarangit's a dependency for all the roles which are converted for the source/distro blueprint11:58
ionievrardjp you have it in private11:59
ionievrardjp should i just manually edit it?12:01
evrardjpwell the problem to edit the json file manally is that it's error prone12:02
evrardjpand it can have disastrous results.12:02
evrardjpbut the inventory-manage tool here won't help you.12:02
evrardjpSo... I guess you have no real choice.12:03
*** bertys_ has quit IRC12:03
evrardjpHowever, I have a tool (in beta) that manipulate json inventories.12:03
evrardjpif you want to try it :)12:03
ionii'll edit it manually, i can manage json :)12:03
jrosserthere is backup_openstack_inventory.tar if you want to diff back through what has changed12:04
evrardjpyes.12:05
evrardjpthat could be good to know what happened between 16.0.10 to 16.0.1312:06
jrosserthat would well have been around the time my inventory got scrambled too12:07
evrardjpandymccr: if you want to help on config_template, I'd pretty much love to see its role included as meta dependency everywhere. Or cloned into an extraplugins folder12:07
ionijrosser, well, i did run again setup hosts and the backup is the one that is broken12:08
andymccrevrardjp: yeah i'd put it in a plugins folder personally, but yeah im putting some changes in there. using it for fio currently causes issues so wanted a way to specify the top group. although perhaps being able to specify a sorted list would be even better12:09
jrosserioni: each time the dynamic inventory script is run it should drop a copy of the current inventory into that tar file12:09
jrosserit should be possible to find a good copy of the inventory in there12:10
evrardjpioni: and no change in the config files?12:11
evrardjpandymccr: I think the priority would be to be arbitrary, including None :)12:11
evrardjp(making it work with no headers)12:12
evrardjpthat would be even cooler!12:12
evrardjp(and fix a bug)12:12
evrardjphahah12:12
andymccrsome of the problem is that we use python 2's config parser which doesnt have a bunch of stuff that is in py3's.12:12
evrardjpI am enclined to add it into meta/ dependencies because it comes for free12:12
evrardjpno need to configure stuff12:13
evrardjpbut I agree it's easier with the folder12:13
evrardjptell me about python3 :)12:13
evrardjp:p12:13
andymccrfor e.g. stripping spaces is possible in py3 directly (not in py2), specifying the default_section is possible in py3, not in py2 directly etc etc etc.12:13
evrardjphttps://wiki.openstack.org/wiki/Python3#Python_3_Status_of_OpenStack_projects12:13
*** astellwag has quit IRC12:15
evrardjpbetween 16.0.10 and head of pike, there is NO change in the inventory code12:16
*** armaan has quit IRC12:16
evrardjpit must be a hidden remnant of a previous edit.12:17
evrardjpour worst ones.12:17
evrardjpor a problem with the tar file or something.12:17
ioniindeed12:17
ionii looked in the inventory from last month and l3 with computes is there12:18
ionimaybe is from a mistake from a long time ago12:18
ioniand only know it takes care of l3 in the right way12:18
*** pcaruana has quit IRC12:21
evrardjpdon't worry the future is brighter: We'll move to static inventories, easy to edit!12:29
*** armaan has joined #openstack-ansible12:30
odyssey4mehwoarang yay! lgtm!12:31
odyssey4meandymccr I thought there was already a way to not have the general group - is that not enough?12:32
andymccrodyssey4me: no for e.g. in fio [global] has to be at the top, if it's not it won't read those conf options correctly - the way config_template orders sections could mean its not at the top12:32
mgariepymorning everyone.12:33
andymccrthe top section will only ever be [DEFAULT] or if thats not specified some random other section which is fine if you don't care what is at the top of the file.12:33
andymccrin py3 we could specify the "default_section" option and just pass it through to config parser and it'd sort the rest, but in py2 that option doesnt exist12:33
odyssey4meoh I see, so to make that work in py2 we'd have to output that section, then output the rest - two steps12:34
andymccryeah which is the PR i put in12:34
andymccrkinda annoying :D12:34
andymccri have a manual hack to remove spacing too - which would be possible in py3 with another flag.12:34
odyssey4meandymccr evrardjp mind reviewing https://review.openstack.org/567159 - I'd like to port that back to Queens too to help reduce the pip noise about infra mirrors not being trusted12:36
odyssey4meI'll do an adapted version of it for pike and back given the changes in the way we do things, but the use of the block still being relevant.12:36
odyssey4meoh yay, there's also this patch now passing the integrated build! https://review.openstack.org/564841 :)12:42
odyssey4meno more distro package noise in the output: http://logs.openstack.org/41/564841/3/experimental/openstack-ansible-integrated-deploy-aio/5b25b77/logs/ara-report/result/e0c850a4-3fe8-4752-af51-7ecdab21a434/12:43
*** dave-mccowan has joined #openstack-ansible12:43
mnasermorning everyone12:46
mgariepymorning mnaser12:47
openstackgerritMohammed Naser proposed openstack/ansible-hardening stable/queens: Ensure that comments are not counted  https://review.openstack.org/56721212:49
openstackgerritMohammed Naser proposed openstack/ansible-hardening stable/pike: Ensure that comments are not counted  https://review.openstack.org/56721312:49
openstackgerritMohammed Naser proposed openstack/ansible-hardening stable/ocata: Ensure that comments are not counted  https://review.openstack.org/56721412:49
mnaserhttps://review.openstack.org/#/c/565894/ simple cherry picked patch12:53
*** armaan has quit IRC12:54
*** arbrandes has quit IRC13:06
*** arbrandes has joined #openstack-ansible13:07
openstackgerritMerged openstack/openstack-ansible-openstack_hosts stable/queens: Retry restarting systemd-journald  https://review.openstack.org/56589413:08
*** klamath has joined #openstack-ansible13:13
cloudnullMorning13:14
mgariepymorning cloudnull13:15
*** yolanda has joined #openstack-ansible13:16
*** klamath has quit IRC13:17
*** yolanda_ has quit IRC13:18
*** throwsb1 has joined #openstack-ansible13:19
openstackgerritMatt Riedemann proposed openstack/openstack-ansible-os_nova master: Remove ironic_host_manager usage  https://review.openstack.org/56582913:21
openstackgerritMerged openstack/openstack-ansible master: Configure cors for glance for additional usability  https://review.openstack.org/56596113:29
*** armaan has joined #openstack-ansible13:34
cloudnullHow's it mgariepy?13:36
*** cjloader has joined #openstack-ansible13:38
*** jwitko has joined #openstack-ansible13:38
*** pcaruana has joined #openstack-ansible13:40
*** jwitko has quit IRC13:41
*** jwitko has joined #openstack-ansible13:42
*** kaiokmo has joined #openstack-ansible13:44
*** weezS has joined #openstack-ansible13:49
*** klamath has joined #openstack-ansible13:54
*** evin has quit IRC13:55
*** esberglu has joined #openstack-ansible13:58
*** pcaruana has quit IRC14:05
mgariepynot too bad, yourself? fighting to network connectivity ...14:05
*** kstev has joined #openstack-ansible14:07
*** pcaruana has joined #openstack-ansible14:08
evrardjpmorning mnaser cloudnull and mgariepy14:20
mnaserbonjour14:20
mgariepyon pourrait faire un take-over du channel en francais !14:22
evrardjpmnaser: you got the french touch in you, do you?14:22
mgariepyhaha14:22
evrardjpoh my god14:22
mnaser:p14:22
evrardjpwhen I am tired or just weirdly context switching, I sometimes speak french to odyssey4me14:22
mnasereveryone seems to forget i speak fluient french :X14:22
mnaserfluent*14:23
evrardjpask him how he feels about it :p14:23
jrossercould i get advice on this? http://paste.openstack.org/show/720684/14:23
*** dxiri has quit IRC14:23
evrardjpmnaser: een probleem dat ik niet heb.14:24
*** dxiri has joined #openstack-ansible14:24
evrardjpthen Tahvok and hwoarang come in, and we are happy that utf-8 was invented.14:24
evrardjp:D14:25
evrardjphow are things everyone ?14:25
*** pcaruana has quit IRC14:25
*** pcaruana has joined #openstack-ansible14:26
*** dxiri_ has joined #openstack-ansible14:26
TahvokIf I search badly I probably can find some hebrew characters not supported by utf-8 ;)14:27
mgariepywhen are you all arriving in Vancouver ?14:29
*** dxiri has quit IRC14:29
mgariepymy plane should land at 7h30 pm on sunday may 20.14:29
odyssey4melol, yeah - evrardjp really has done that... literally 2-3 minutes of him talking in french with me looking at him blankly, wondering whether he's trying to play a joke or whether he's about to translate... then he doesn't14:29
mgariepyodyssey4me, lol.14:30
mgariepyodyssey4me, to help him you should learn french.14:30
*** dxiri_ has quit IRC14:30
odyssey4memgariepy that would probably help, but then he'd never have learned so much english ;)14:31
evrardjpodyssey4me: but do you REALLY speak English?14:32
evrardjp:p14:32
evrardjpodyssey4me: and it only happened like twice.14:33
mnasermgariepy: i'm coming in saturday at 9am14:36
*** evin has joined #openstack-ansible14:37
evrardjpmgariepy: saturday evening14:39
prometheanfireis anyone using radosgw as an object backend to store glance images on AND using ironic in that setup with a ocata or newer deployment (swiftclient-3.3.0 or up)14:44
openstackgerritMatt Riedemann proposed openstack/openstack-ansible-os_nova master: Remove ironic_host_manager usage  https://review.openstack.org/56582914:45
*** udesale has quit IRC14:46
*** srihas has quit IRC14:52
jrosserwhat would make neutron_fwaas==12.0.1.dev3 be missing from my repo server? http://paste.openstack.org/show/720684/14:53
cloudnullI believe the selective wheel build only builds what's enabled. So if fwaas wasnt enabled when the repobuild happened it would skip it. I think.14:56
cloudnullodyssey4me ^14:57
jrosserit looks like there are other versions there, which is a little odd14:58
*** mma has quit IRC15:01
jrossercloudnull: i find this neutron_fwaas-12.0.1.dev3-py2.py3-none-any.whl -> /var/www/repo/pools/ubuntu-16.04-x86_64/neutron_fwaas/neutron_fwaas-12.0.1.dev3-py2.py3-none-any.whl15:02
*** DanyC has quit IRC15:04
*** mma has joined #openstack-ansible15:04
*** DanyC has joined #openstack-ansible15:05
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Fix links to figures  https://review.openstack.org/56725115:05
*** Leo_m has joined #openstack-ansible15:05
*** DanyC has quit IRC15:06
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Fix links to figures  https://review.openstack.org/56725115:06
*** mma has quit IRC15:09
idlemindthe stable/pike centos patches might need: https://github.com/openstack/openstack-ansible-galera_server/commit/dbbcb56a62e4a97af3f63f970a34a3c815352ced#diff-0dc9e326c64f4a15341a43e16b94ee6015:16
idlemindhmm maybe not; we'll see15:21
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-openstack_hosts master: Add Bionic testing  https://review.openstack.org/56703815:22
*** armaan has quit IRC15:25
*** armaan has joined #openstack-ansible15:26
dmsimardcloudnull: hey, if you'd like to join #ara, I have a question about the virtualenv stuff :)15:28
*** prometheanfire has quit IRC15:32
cloudnulldmsimard sure15:32
*** gyee has joined #openstack-ansible15:38
*** epalper has quit IRC15:41
*** evin has quit IRC15:42
odyssey4mejrosser so the wheel was there, but installing it didn't work?15:48
*** tosky has quit IRC15:48
*** tosky has joined #openstack-ansible15:48
odyssey4mecloudnull should https://review.openstack.org/565961 be ported back? it looks like it may be relevant to stable branches? (not sure which)15:49
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible stable/queens: Configure cors for glance for additional usability  https://review.openstack.org/56726415:49
*** dxiri has joined #openstack-ansible15:49
*** openstacking_123 has joined #openstack-ansible15:49
cloudnullodyssey4me yes15:49
jrosserodyssey4me: yes thats right15:57
jrosserodyssey4me: i wonder if there is an unfortunate interaction with me having a local pip mirror and its looking there instead15:57
jrosserjust travelling but i will dig into this later15:58
jrossereverything went completely fine with those three patches we discussed earlier until it got to neutron15:59
odyssey4mejrosser bother, that's the exact problem they were trying to solve :/16:01
odyssey4meI unfortunately haven't had the time to dig into it again since pushing those up.16:01
jrosserhey no problem :) i'll help get to the bottom of it if youre about16:02
jrosseri am slightly suspicious as i expect noone else enables the fwaas stuff, i only recently patched that to work at all16:03
*** openstacking_123 has quit IRC16:06
evrardjpjrosser: did you bump the fwaas sha?16:07
jrosserevrardjp: no16:08
evrardjpk16:08
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Add Bionic testing  https://review.openstack.org/56719216:08
odyssey4mejrosser well, the issue will show up for any of the optional_pip_packages, including lbaas and perhaps others16:09
odyssey4methey're not put into the venv in repo build, but added in a task later16:09
odyssey4methat's why it relies on the pypiserver serving them16:10
jrosserah right i see16:10
odyssey4mepip -> haproxy -> repo server -> nginx -> pypiserver16:10
odyssey4meif pypiserver has a package index for the request package, then it returns only the versions it has - effectively locking down the deployment to only the versions on the repo server16:11
odyssey4meif it doesn't, then it returns the pypi index which has everything... but it won't have the pre-release packages (...dev...)16:12
jrosserand if i have a bogus pip ->  my_mirror thats going to end badly16:12
odyssey4meif you have your own mirror, that'w what pip_links was meant for - it will ensure that's also looked at for packages16:12
odyssey4mehowever, that's primarily for proprietary packages which the repo can never build16:13
odyssey4meif you have just a normal pypi mirror, it's the repo build that's supposed to be given that information16:13
odyssey4methat should be provided as the repo build default index so that the repo build uses it: https://github.com/openstack/openstack-ansible-repo_build/blob/master/defaults/main.yml#L7716:14
*** dxiri_ has joined #openstack-ansible16:16
jrosserfab thanks, i'll check what weve done16:17
*** dxiri_ has quit IRC16:17
*** dxiri_ has joined #openstack-ansible16:18
*** dxiri has quit IRC16:20
*** hamzy has quit IRC16:28
*** DanyC has joined #openstack-ansible16:28
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-galera_client master: Add Bionic testing  https://review.openstack.org/56702916:31
*** DanyC_ has joined #openstack-ansible16:31
*** udesale has joined #openstack-ansible16:32
*** DanyC has quit IRC16:33
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible-lxc_hosts master: Add Bionic testing  https://review.openstack.org/56703416:35
*** prometheanfire has joined #openstack-ansible16:36
*** weezS has quit IRC16:36
*** masber has joined #openstack-ansible16:40
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-plugins master: Remove test-requirements files from py_pkgs lookup  https://review.openstack.org/56728516:40
odyssey4meI'm out for the evening folks. Chat tomorrow!16:49
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible master: [Docs] Fix links to figures  https://review.openstack.org/56725116:51
*** armaan has quit IRC16:55
*** armaan has joined #openstack-ansible16:55
*** tosky has quit IRC16:57
*** evin has joined #openstack-ansible16:58
*** tosky has joined #openstack-ansible16:59
evrardjpodyssey4me: have a good day16:59
*** prometheanfire has quit IRC17:06
openstackgerritMerged openstack/openstack-ansible-tests master: Vagrantfile: Increase VM memory requirements to 6GB  https://review.openstack.org/56415317:25
*** mbuil has quit IRC17:25
*** hamzy has joined #openstack-ansible17:27
*** gkadam_ has quit IRC17:43
*** tasker has joined #openstack-ansible17:45
taskerI have need to move a few LXCs made with OSA to another physical host within the same cluster. is it possible to destroy and recreate the containers using OSA?17:46
taskeri found: https://docs.openstack.org/openstack-ansible/12.2.6/install-guide/ops-galera-recoverycontainer.html , but that's galera specific and suggests that the host name needs to be the same.17:46
taskerin my case the target hosts have different names.17:46
taskerI've used `lxc-containers-destroy` to get rid of the old containers and modified /etc/openstack_deploy/openstack_user_config.yml to point the services to their new homes.17:47
*** osnaya has joined #openstack-ansible17:48
taskerhowever, the inventory still thinks that the original hosts are necessary and a part of the deploy. I can't move the inventory out of the way, or else the whole set of containers is recreated.17:48
taskerdid I miss or forget a step?17:48
taskerI've deleted the facts and run "ansible -m setup all" and get a key error during inventory that references the old hosts.17:49
*** udesale has quit IRC17:50
mnaserevrardjp, jrosser: https://bugs.launchpad.net/openstack-ansible/+bug/1768231 fyi i just hit this17:52
openstackLaunchpad bug 1768231 in openstack-ansible "Cinder filter scheduler not enabled with multiple storage back ends" [Medium,Confirmed] - Assigned to Mohammed Naser (mnaser)17:52
mnaserso i'm going to be diving in to figuring out a) why it goes down17:52
mnaserand b) seeing if we have to make that switch17:52
osnayaHitting an issue upon executing setup-hosts.yml related to lxc cache ... chroot /var/lib/machines/ubuntu-xenial-amd64 /usr/local/bin/cache-prep-commands.sh. Log excerpt @http://paste.openstack.org/show/720609/17:52
osnayaHas anyone seen this issue? any pointers?17:53
*** mma has joined #openstack-ansible17:57
*** mma has quit IRC17:59
mgariepyosnaya, https://review.openstack.org/#/c/566298/17:59
*** armaan has quit IRC18:00
*** tasker has quit IRC18:00
*** armaan has joined #openstack-ansible18:00
mgariepyosnaya, https://review.openstack.org/#/c/566046/18:01
*** tasker has joined #openstack-ansible18:03
osnaya@mgariepy thanks for the pointer. let me check.18:05
mnaserevrardjp, jrosser: added comments, please have a look :)18:06
taskeri guess, really, I can just treat "moving" the LXCs as adding new infrastructure? https://docs.openstack.org/openstack-ansible/pike/admin/maintenance-tasks/scale-environment.html#add-a-new-infrastructure-host18:09
taskerbut it's not all of the LXCs from a host I want to move. just a subset. "network_hosts" in particular.18:17
*** armaan has quit IRC18:17
*** armaan has joined #openstack-ansible18:17
*** armaan has quit IRC18:18
*** armaan has joined #openstack-ansible18:18
*** spine55 has joined #openstack-ansible18:36
jrossermnaser: i did a deploy today which i expect the same cinder issue to be present, i'll see if theres anything interesting in the log18:39
mnaserjrosser: cool thanks18:39
*** weezS has joined #openstack-ansible18:39
mnaserjrosser: ok i noticed something i'll log too18:52
mnaserchecking journalctl18:52
mnasercinder-volume[567641]: terminate called after throwing an instance of 'std::runtime_error'18:53
mnasercinder-volume[567641]: what():  random_device::random_device(const std::string&)18:53
mnaserkilled one of the agents18:53
mnaserand restarted it18:53
jrossermnaser: mine is essentially a log of what happens during the deploy18:59
jrosserso there are many many of these cinder-volume.service: Main process exited, code=exited, status=1/FAILURE18:59
mnaserjrosser: anything with this? journalctl | fgrep 'cinder-volume['19:00
jrosserand the cinder-volumes log is piled with errors becasue of the database tables not existing19:00
jrosserno nothing matching that - but this cloud hasnt been used at all yet19:01
*** hamzy has quit IRC19:01
mnaserinteresting19:04
*** hamzy has joined #openstack-ansible19:18
throwsb1Hi, I am having failure with tenant vm getting an IP from DHCP servers on tenant network.  I believe the issue is that there is not a eth10 on the 3 infra nodes in my build.19:21
throwsb1This looks to be a difference between pike and queens.  Where pike would install all neutron services in a container and queens puts services on baremetal.19:22
throwsb1would some be able to verify that yes I need to create a veth eth10 for DHCP on tenant networks to work?  I can verify with a tcpdump that the request is seen on all node on the br-vxlan physical nic.19:23
jrosseri don't think you need to do that19:24
jrosserip netns <- that should show you the dhcp namespace19:24
throwsb1jrosser: I don't see dhcp request coming in on the vxlan-72, which is associated with the tenant network on the infra nodes19:25
jrosser(i don't have my neutron book to hand sadly)19:25
throwsb1I was going by the OSA troubleshooting guide where it mentions eth1019:25
jrosserjust to clarify, you see the request on the compute node, but not at all on the infra node?19:26
throwsb1and eth10 is listed in the user config19:26
throwsb1I only see it broadcasted on infra nodes on the physical nic19:26
throwsb1actually I also see it on br-vxlan on all nodes too19:27
jrosserdid 'ip netns' show you anything related to dhcp?19:28
jrosseryou should know which infra nodes (all maybe) that the dhcp agents are running on19:29
throwsb1qdhcp-dc79c87f-098d-4458-87df-8b28fd1b7b74 (id: 15)19:29
throwsb1qrouter-82c6957d-d19c-4f38-92f6-708b91925b5f (id: 14)19:29
throwsb1from infra19:29
throwsb1last deploy was single infra, this one is 3 nodes.  little harder to trace.  I was trying to at least trace the traffic with a tcpdump19:31
jrosserip netns exec qdhcp-<blah> tcpdump <options>19:33
jrosser^ run tcpdump within the dhcp agent network namespace19:34
throwsb1k19:34
jrosserthrowsb1: as i don't have my usual neutron reference book to hand the best i can offer you is this https://dischord.org/2015/03/09/troubleshooting-openstack-neutron-networking-part-one/19:35
jrosserwhich admittedly is OVS based so will be a little off topic, but covers the use of tcpdump in network namespaces which should be the same19:36
throwsb1cool, thanks.  and btw the dhcp is not coming in19:36
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-apt_package_pinning master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56732419:38
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-ceph_client master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56732519:38
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-galera_client master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56732619:38
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-galera_server master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56732719:38
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-haproxy_server master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56732819:38
osnaya@cloudnull @odyssey4me @mgariepy looked at issue "lxc-cache"... workaround @https://review.openstack.org/#/c/566298/ and fix in 16.0.13 @https://review.openstack.org/#/c/566046/. So since I have 16.0.12 on my host, I applied workaround. Questions is ... after the workaorund - Do I just rerun setup-hosts.yml? Please clarify.19:38
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_container_create master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56732919:38
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-lxc_hosts master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733019:38
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-memcached_server master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733119:39
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-nspawn_container_create master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733219:39
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-nspawn_hosts master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733319:39
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-openstack_hosts master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733419:39
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-openstack_openrc master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733519:39
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_almanach master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733619:39
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_aodh master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733719:39
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_barbican master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733819:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_blazar master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56703619:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56733919:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_cinder master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734019:40
throwsb1jrosser: thanks for the guide.  looks like some good troubleshooting steps.19:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_cloudkitty master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734119:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_congress master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734219:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_designate master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734319:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_glance master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734419:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_gnocchi master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734519:40
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_heat master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734619:41
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_horizon master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734719:41
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734819:41
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_keystone master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56734919:41
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_magnum master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735019:41
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_masakari master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735119:41
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_monasca master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735219:41
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735319:42
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_nova master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735419:42
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735519:42
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_panko master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735619:42
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_rally master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735719:42
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735819:42
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_searchlight master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56735919:42
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56736019:43
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_tacker master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56736119:43
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_tempest master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56736219:43
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_trove master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56513319:43
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_watcher master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56736319:43
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_zaqar master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56736419:43
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-pip_install master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56736519:43
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-plugins master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56736619:43
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-rabbitmq_server master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56736719:43
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-repo_build master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56736819:44
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-repo_server master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56737019:44
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-rsyslog_client master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56737119:44
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-rsyslog_server master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56737219:44
openstackgerritOpenStack Proposal Bot proposed openstack/ansible-config_template master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56737319:44
openstackgerritOpenStack Proposal Bot proposed openstack/ansible-hardening master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56737419:44
openstackgerritOpenStack Proposal Bot proposed openstack/ansible-role-python_venv_build master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56737519:44
openstackgerritOpenStack Proposal Bot proposed openstack/ansible-role-systemd_mount master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56737619:45
openstackgerritOpenStack Proposal Bot proposed openstack/ansible-role-systemd_networkd master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56737719:45
openstackgerritOpenStack Proposal Bot proposed openstack/ansible-role-systemd_service master: Updated from OpenStack Ansible Tests  https://review.openstack.org/56737819:45
*** kysse has quit IRC19:51
*** kysse has joined #openstack-ansible19:58
*** DanyC_ has quit IRC20:03
*** prometheanfire has joined #openstack-ansible20:09
*** DanyC has joined #openstack-ansible20:28
*** DanyC has quit IRC20:30
*** DanyC has joined #openstack-ansible20:31
osnaya@cloudnull @odyssey4me @mgariepy please clarify on the workaround question - on deployment host I did workaround step.... Do I need to re-execute setup-hosts.yml?20:35
*** DanyC has quit IRC20:36
cloudnullosnayawhats up ?20:42
*** radeks_ has quit IRC20:46
cloudnullosnaya if you have that commit within the lxc hosts role, all you need to do is run lxc-hosts20:46
cloudnullhowever you can rerun setup-everything if you want to be sure20:47
osnaya@cloudnull thanks20:49
*** kstev has quit IRC20:54
*** DanyC has joined #openstack-ansible20:59
*** pcaruana has quit IRC20:59
*** vnogin has joined #openstack-ansible21:01
*** hamzy has quit IRC21:01
*** prometheanfire has quit IRC21:02
*** osa_tacker has joined #openstack-ansible21:04
osa_tackerhey guys, I'm running into some issues trying to deploy tacker using openstack ansible. I currently have queens deployed, but when I copy over the default variables from https://docs.openstack.org/openstack-ansible-os_tacker/latest/ and then copy the tacker.yml into the env.d directory, and run the setup-hosts playbook, I cannot see any tacker-server container being created, any suggestions?21:04
*** throwsb1 has quit IRC21:07
*** hamzy has joined #openstack-ansible21:11
*** hamzy has quit IRC21:15
osnaya@cloudnull @odyssey4me @mgariepy hmmm... re-running setup-hosts.yml upon the workaround still gives the same error... Am I missing something? Please see @http://paste.openstack.org/show/720715/ Please advise21:15
*** dave-mccowan has quit IRC21:17
*** osa_tacker has quit IRC21:18
osnaya@cloudnull @odyssey4me @mgariepy please note, since I have 16.0.12 .... I changed the config (user_config.yml) as specified in https://git.openstack.org/cgit/openstack/openstack-ansible/commit/?id=0f88f27268a7913349aaa3df393ec5566874685121:18
cloudnullosnaya do you have this commit https://github.com/openstack/openstack-ansible-lxc_hosts/commit/e77c3a923bc8aae4a264917f592b58b5d1c79aed ?21:21
cloudnullif youre going to use the method specifiied in the commit your referenced you will need to save that override in the user_variables file21:22
cloudnulllike /etc/openstack_deploy/user_variables.yml21:22
*** dave-mccowan has joined #openstack-ansible21:22
osnaya@cloudnull nope... I don't have the commit. I used the workaround and updated /etc/openstack_deploy/user_variables.yml specified @https://review.openstack.org/#/c/566298/ Please see @http://paste.openstack.org/show/720715/ for exact details from my setup21:27
*** DanyC has quit IRC21:28
*** DanyC has joined #openstack-ansible21:28
*** kstev has joined #openstack-ansible21:29
osnaya@cloudnull did the workaround from upstream as I interpreted... exact details of the workaround step @@http://paste.openstack.org/show/720715/..... Is it either workaround or commit .....Or Do I need both workaround and the commit?21:29
*** hamzy_ has joined #openstack-ansible21:29
*** hamzy_ is now known as hamzy21:30
*** DanyC has quit IRC21:33
*** cjloader has quit IRC21:38
*** cjloader_ has joined #openstack-ansible21:38
osnaya@cloudnull @odyssey4me @mgariepy modified /etc/openstack_deploy/user_variables.yml already---> added a line [lxc_cache_prep_pre_commands: "rm -f /etc/resovl.conf || true"] Am I missing anything in the workaround step.... re-running setup-hosts.yml log has the same error.... as if it didn't take the change added line21:39
*** jwitko_ has joined #openstack-ansible21:40
*** prometheanfire has joined #openstack-ansible21:41
*** jwitko_ has quit IRC21:41
*** jwitko_ has joined #openstack-ansible21:41
*** cjloader_ has quit IRC21:43
*** jwitko has quit IRC21:43
*** prometheanfire has quit IRC21:45
*** jwitko_ has quit IRC21:47
*** weezS has quit IRC21:51
*** Leo_m has quit IRC21:53
cloudnullIt's hard to say... I'd likely cherry pick the change in and go from there.21:56
*** wat89 has joined #openstack-ansible22:00
wat89hey guys, I'm running into some issues trying to deploy tacker using openstack ansible. I currently have queens deployed, but when I copy over the default variables from https://docs.openstack.org/openstack-ansible-os_tacker/latest/ and then copy the tacker.yml into the env.d directory, and run the setup-hosts playbook, I cannot see any tacker-server container being created, any suggestions?22:00
*** prometheanfire has joined #openstack-ansible22:02
*** weezS has joined #openstack-ansible22:10
*** vnogin has quit IRC22:10
*** esberglu has quit IRC22:10
*** sawblade6 has quit IRC22:14
*** prometheanfire has quit IRC22:15
*** sawblade6 has joined #openstack-ansible22:15
*** cjloader has joined #openstack-ansible22:17
*** vnogin has joined #openstack-ansible22:21
*** cjloader has quit IRC22:21
*** vnogin has quit IRC22:31
*** threestrands has joined #openstack-ansible22:33
*** threestrands has quit IRC22:33
*** threestrands has joined #openstack-ansible22:33
*** threestrands_ has joined #openstack-ansible22:36
*** threestrands has quit IRC22:38
*** tomcsi has quit IRC22:38
*** hamzy has quit IRC22:39
*** tomcsi has joined #openstack-ansible22:39
*** mstrohl has joined #openstack-ansible22:39
openstackgerritMerged openstack/openstack-ansible-galera_server stable/queens: Updated from OpenStack Ansible Tests  https://review.openstack.org/56661322:43
*** prometheanfire has joined #openstack-ansible22:43
*** prometheanfire has quit IRC22:47
*** prometheanfire has joined #openstack-ansible22:48
*** mstrohl has quit IRC22:49
osnaya@cloudnull ok... my local host git describe --tags is "16.0.12" How do I get this new commit with the fix on my host?.... Not sure how to do that. Can you please provide pointers?23:00
cloudnullOsnaya you can do a fetch and cherry pick of commit e77c3a923:02
*** prometheanfire has quit IRC23:09
*** weezS has quit IRC23:13
*** cjloader has joined #openstack-ansible23:13
*** cjloader has quit IRC23:17
*** vnogin has joined #openstack-ansible23:19
osnaya@cloudnull ok... so once I cherry pick the commit on my local host, then I undo the workaround step, right?23:23
*** SamYaple has quit IRC23:32
*** SamYaple has joined #openstack-ansible23:33
*** tosky has quit IRC23:38
*** dxiri_ has quit IRC23:39
*** prometheanfire has joined #openstack-ansible23:39
*** prometheanfire has quit IRC23:43
*** vnogin has quit IRC23:44
osnaya@cloudnull @odyssey4me Cherrypick a commit is new for me...please confirm if the command to get that commit is right? git fetch origin e77c3a923bc8aae4a264917f592b58b5d1c79aed:refs/remotes/origin/lxccacheprefix-commit23:47

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!