| nurdie | guilhermesp: I didn't rerun run-upgrade.sh | 00:06 |
|---|---|---|
| nurdie | I continued where it left off with the manual steps in the guide listed above | 00:07 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_nova master: Eliminate nova_novnc_distro_packages https://review.openstack.org/648886 | 00:52 |
| *** mino_ has joined #openstack-ansible | 00:59 | |
| *** nurdie has quit IRC | 01:03 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-tests master: Stop deleting-and-creating RabbitMQ account on deploy https://review.openstack.org/648887 | 01:07 |
| *** mino_ has quit IRC | 01:10 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_nova master: Add privsep-helper to nova sudoers file https://review.openstack.org/648888 | 01:24 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_neutron master: Add privsep-helper to nova sudoers file https://review.openstack.org/648890 | 01:28 |
| *** cshen has joined #openstack-ansible | 01:46 | |
| *** cshen has quit IRC | 01:50 | |
| *** irclogbot_1 has quit IRC | 02:19 | |
| *** nurdie has joined #openstack-ansible | 02:59 | |
| *** mkuf has joined #openstack-ansible | 03:34 | |
| openstackgerrit | Guilherme Steinmuller Pimentel proposed openstack/openstack-ansible-os_horizon master: Drop horizon_keystone_host variable https://review.openstack.org/648898 | 03:37 |
| *** udesale has joined #openstack-ansible | 04:02 | |
| *** ChosSimbaOne has quit IRC | 04:54 | |
| *** ChosSimbaOne has joined #openstack-ansible | 04:54 | |
| *** minops has joined #openstack-ansible | 04:57 | |
| *** minops has joined #openstack-ansible | 04:58 | |
| *** minops has quit IRC | 05:21 | |
| *** ivve has joined #openstack-ansible | 05:22 | |
| *** ianychoi has quit IRC | 05:23 | |
| *** minops has joined #openstack-ansible | 05:24 | |
| *** ianychoi has joined #openstack-ansible | 05:24 | |
| *** chandankumar has joined #openstack-ansible | 05:33 | |
| *** minops has quit IRC | 05:54 | |
| *** minops has joined #openstack-ansible | 05:55 | |
| *** minops has quit IRC | 06:05 | |
| *** minops has joined #openstack-ansible | 06:12 | |
| *** DanyC has joined #openstack-ansible | 06:22 | |
| *** minops has quit IRC | 06:25 | |
| *** minops has joined #openstack-ansible | 06:26 | |
| *** minops has quit IRC | 06:26 | |
| *** cshen has joined #openstack-ansible | 06:28 | |
| *** kopecmartin|off is now known as kopecmartin | 06:30 | |
| *** DanyC has quit IRC | 06:45 | |
| fnpanic | good morning | 06:55 |
| fnpanic | quick question in the morning. can i add files to local_settings.d with OSA? | 06:56 |
| fnpanic | doing this with a script is not that easy because the venv are OSA version specific. | 06:56 |
| *** pcaruana has joined #openstack-ansible | 06:58 | |
| *** pcaruana has quit IRC | 07:02 | |
| *** pcaruana has joined #openstack-ansible | 07:02 | |
| *** rgogunskiy has joined #openstack-ansible | 07:07 | |
| *** fghaas has joined #openstack-ansible | 07:08 | |
| *** DanyC has joined #openstack-ansible | 07:13 | |
| *** tosky has joined #openstack-ansible | 07:13 | |
| evrardjp | fnpanic what do you want to achieve ? | 07:20 |
| evrardjp | Can't <what you are looking for> be done with config_template ? | 07:20 |
| *** DanyC has quit IRC | 07:22 | |
| *** luksky has joined #openstack-ansible | 07:23 | |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-lxc_hosts stable/rocky: tasks: lxc_cache_preparation: Ensure the /opt directory exists in the container https://review.openstack.org/648908 | 07:25 |
| openstackgerrit | Dmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-lxc_hosts stable/rocky: Fix image fetching process for leap 15 https://review.openstack.org/648854 | 07:26 |
| *** jbadiapa has joined #openstack-ansible | 07:54 | |
| *** hamzaachi has joined #openstack-ansible | 07:56 | |
| *** gkadam has joined #openstack-ansible | 08:02 | |
| openstackgerrit | Jonathan Rosser proposed openstack/openstack-ansible master: [WIP] Use the Mitogen connection method in OSA https://review.openstack.org/591236 | 08:02 |
| *** phasespace has joined #openstack-ansible | 08:05 | |
| *** miloa has joined #openstack-ansible | 08:11 | |
| *** trikorasolns has joined #openstack-ansible | 08:20 | |
| fnpanic | evrardjp: not sure about config_template. i wanted to enable domain dropdown in rocky and also limit the list of glance images to upload | 08:27 |
| fnpanic | patching local_settings.py works but is really bad | 08:29 |
| *** gkadam has quit IRC | 08:32 | |
| *** DanyC has joined #openstack-ansible | 08:35 | |
| *** cshen has quit IRC | 08:37 | |
| evrardjp | oh so you are changing horizon you mean? | 08:39 |
| evrardjp | not really sure what you are looking for. If there is a feature we don't expose, you can expose it indeed | 08:40 |
| *** fghaas has quit IRC | 08:40 | |
| evrardjp | For the "limit of x to upload" isn't that a policy thing? | 08:40 |
| *** nurdie has quit IRC | 08:40 | |
| evrardjp | What do you mean there? | 08:40 |
| fnpanic | no it is the dropdown in glance image upload panel | 08:44 |
| fnpanic | do not want to confuse users | 08:44 |
| fnpanic | they can only upload raw and iso | 08:44 |
| CeeMac | morning | 09:04 |
| fnpanic | hi | 09:08 |
| CeeMac | fnpanic: for the image upload list you could use horizon_config_overrides and configure the image_formats? https://docs.openstack.org/horizon/latest/configuration/settings.html#openstack-image-backend | 09:12 |
| CeeMac | and for domain choices: https://docs.openstack.org/horizon/latest/configuration/settings.html#openstack-keystone-domain-choices? | 09:13 |
| CeeMac | I think that would work? | 09:13 |
| CeeMac | with the https://docs.openstack.org/horizon/latest/configuration/settings.html#openstack-keystone-domain-dropdown | 09:14 |
| CeeMac | configured true | 09:14 |
| *** electrofelix has joined #openstack-ansible | 09:14 | |
| *** med_ has quit IRC | 09:15 | |
| fnpanic | CeeMac: yes i know | 09:22 |
| fnpanic | but i do not know how to do it the "osa" way | 09:23 |
| fnpanic | patching the jinja template is not nice | 09:23 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: Copy over YAML policy files https://review.openstack.org/648841 | 09:24 |
| CeeMac | fnpanic: hmm i thought using the horizon_config_override: config template in user_variables would work for these settings? | 09:28 |
| CeeMac | fnpanic, I'm no expert though :) | 09:28 |
| *** gkadam has joined #openstack-ansible | 09:28 | |
| CeeMac | i've definitely seen another page with the settable options on it, can't find it again though :( | 09:29 |
| *** noonedeadpunk has joined #openstack-ansible | 10:10 | |
| *** noonedeadpunk has quit IRC | 10:13 | |
| *** noonedeadpunk[h] has joined #openstack-ansible | 10:14 | |
| *** luksky has quit IRC | 10:17 | |
| *** noonedeadpunk[h] has quit IRC | 10:20 | |
| *** noonedeadpunk has joined #openstack-ansible | 10:21 | |
| noonedeadpunk | evrardjp - I was trying to backport https://review.openstack.org/#/c/648854/ and strange thing happens for suse. It seems, that /var/lib/machines/opensuse-15.0-amd64 is just empty here http://logs.openstack.org/54/648854/2/check/openstack-ansible-lxc-dir-opensuse-150/8cfbfe9/job-output.txt.gz#_2019-04-01_07_40_46_432101 so chroot fails | 10:30 |
| noonedeadpunk | I suppose, that downloaded imgae should be extracted at this stage? As image is present at machinectl... | 10:31 |
| evrardjp | I know nothing about machinectl | 10:34 |
| evrardjp | if you don't mind let's not backport this for now | 10:35 |
| *** ivve has quit IRC | 10:35 | |
| noonedeadpunk | evrardjp: yep, sure. just wanted to get suse working on rocky, but ok:0 | 10:36 |
| evrardjp | me too :) | 10:36 |
| jrosser | it's not really to do with machinectl | 10:36 |
| jrosser | it's just where the rootfs gets unpacked | 10:37 |
| evrardjp | the problem I have right now is with other packages, so I will focus on lxc_image_create later | 10:37 |
| evrardjp | and lxc_hosts | 10:37 |
| evrardjp | jrosser: I suppose the download is file, else it would have failed before, but I haven't digged into the details | 10:37 |
| noonedeadpunk | jrosser so yeah, it seems that it's not get unpacked. But i'm not really sure that it should get unpacked before preparation script... | 10:38 |
| jrosser | yeah, it just happens to use the same dir that machinectl would use for it's images iirc | 10:38 |
| evrardjp | I don't have time to dig into that quite yet | 10:38 |
| fnpanic | CeeMac: thanks | 10:39 |
| evrardjp | as I do this on my spare time :) | 10:39 |
| noonedeadpunk | so am I:) | 10:39 |
| fnpanic | do you have an example for modifing this with config_template? was going through the docs but not sure. an example would help :-) | 10:39 |
| *** nurdie has joined #openstack-ansible | 10:40 | |
| CeeMac | fnpanic: not to hand. I'm sure i did something with it, but i can't find my notes, and its not in my user_varibles anymore :( | 10:41 |
| jrosser | fnpanic: i'm not sure I have a proper answer, but do you need to do more than whats possible with this? https://github.com/openstack/openstack-ansible-os_horizon/blob/master/templates/horizon_local_settings.py.j2#L826 | 10:43 |
| noonedeadpunk | jrosser actually machinectl list-images shows downloaded image, so I'm not sure that it's only about directory | 10:44 |
| *** noonedeadpunk[h] has joined #openstack-ansible | 10:45 | |
| *** noonedeadpunk[h] has quit IRC | 10:45 | |
| *** noonedeadpunk[h] has joined #openstack-ansible | 10:46 | |
| *** nurdie has quit IRC | 10:46 | |
| fnpanic | jrosser: i need a few variables altered | 10:47 |
| fnpanic | for example this one OPENSTACK_IMAGE_BACKEND | 10:47 |
| fnpanic | and these two OPENSTACK_KEYSTONE_DOMAIN_DROPDOWN andOPENSTACK_KEYSTONE_DOMAIN_CHOICES | 10:48 |
| *** noonedeadpunk has quit IRC | 10:48 | |
| *** noonedeadpunk[h] is now known as noonedeadpunk | 10:49 | |
| fnpanic | i think this is not possible with horizon_config_overrides: | 10:49 |
| jrosser | so this sort of thing is not good enough? https://github.com/openstack/openstack-ansible-os_horizon/blob/4283200534eafa444efd9bb408ddcb5c98a1d442/tests/os_horizon-overrides.yml#L38-L39 | 10:49 |
| fnpanic | i changed the bug and help url and this works | 10:49 |
| fnpanic | so this would work? | 10:50 |
| fnpanic | then it comes to how to format the lists in yaml correctly :-) it was a pain for the help and bug url :-) | 10:51 |
| jrosser | yes this is horrible really | 10:52 |
| fnpanic | http://paste.openstack.org/show/748655/ | 10:52 |
| fnpanic | for the bools it is easy :-) | 10:52 |
| fnpanic | not sure how to do these two lists.... | 10:52 |
| CeeMac | fnpanic, i'm struggling with this lists as well. | 10:55 |
| jrosser | well like you said earlier, it may be easier to provide a way to drop files from the deploy host into local_settings.d/ instead | 10:55 |
| jrosser | becasue trying to turn yaml into python is never going to work out neatly | 10:56 |
| fnpanic | jrosser: yes +1 | 10:57 |
| fnpanic | that was my inital idea :-) | 10:57 |
| CeeMac | jrosser: do you think the mod evrardjp wrote for dropping custom image files etc into horizon could be adopted for this? | 10:57 |
| jrosser | i'm sure there is a pattern somewhere in the existing code you can copy | 10:58 |
| fnpanic | great :-) | 10:58 |
| CeeMac | horizon_custom_uploads: | 10:58 |
| jrosser | it's a very minor adjustment to this https://github.com/openstack/openstack-ansible-os_horizon/blob/master/templates/horizon_local_settings.py.j2#L826 | 11:00 |
| jrosser | copy/paste/adjust | 11:00 |
| CeeMac | makes it sound so easy when you say it like that>_> | 11:03 |
| evrardjp | it kinda is :) | 11:05 |
| jrosser | this is an ideal time to try making a patch | 11:05 |
| jrosser | becasue you've got a real issue to solve and can test it locally | 11:05 |
| jrosser | plus there is almost-right code there already as an example | 11:05 |
| evrardjp | jrosser: did you follow what was going on this week-end? | 11:06 |
| jrosser | sort of | 11:06 |
| evrardjp | I am not sure if mnaser wants to branch today | 11:06 |
| jrosser | merge merge merge mostly :) | 11:06 |
| evrardjp | (not talking about brexit) | 11:06 |
| evrardjp | haha yes I saw that | 11:06 |
| jrosser | ah well that is fork fork fork mostly | 11:06 |
| evrardjp | but I want to make sure we are ready before proposing a branch | 11:06 |
| noonedeadpunk | evrardjp: what's the next rocky tag is going to be? 18.1.6? and are you going to base it on https://github.com/openstack/openstack-ansible/commit/9558b79e1a33b76b2e591964e395d497e6b92367 ? Just probably I may checkout to this commit and place tag locally... | 11:07 |
| evrardjp | as long as you dont s/or/uc/ | 11:07 |
| evrardjp | don't say* | 11:07 |
| evrardjp | the releases are done before the upstream sha bump | 11:07 |
| jrosser | CeeMac: fnpanic if either if you fancy making a patch please do :) we can help if you get stuck | 11:08 |
| evrardjp | noonedeadpunk: the idea is that we have two weeks to find out issues | 11:08 |
| evrardjp | so next release will indeed include this bump | 11:08 |
| evrardjp | but not the next one | 11:08 |
| jrosser | evrardjp: from saturday "mnaser> evrardjp: we can branch stable/stein and release rc :)" | 11:09 |
| evrardjp | woot I have missed that | 11:09 |
| evrardjp | let me propose that then | 11:09 |
| jrosser | all the way up there ^^^^^^^ somewhere | 11:09 |
| evrardjp | like it always is :) | 11:09 |
| evrardjp | mnaser: I proposed this, I would love seeing your review. | 11:16 |
| evrardjp | https://review.openstack.org/#/c/648947/ | 11:16 |
| *** udesale has quit IRC | 11:22 | |
| *** ivve has joined #openstack-ansible | 11:25 | |
| *** trikorasolns has quit IRC | 11:26 | |
| *** luksky has joined #openstack-ansible | 11:26 | |
| *** ansmith_ has quit IRC | 11:37 | |
| *** vollman has joined #openstack-ansible | 11:43 | |
| *** vollman has quit IRC | 11:56 | |
| fnpanic | is someone using ara in osa? | 11:57 |
| fnpanic | when yes how it the best way to install it? setting it setup_ara true in bootstrap-ansible.sh? | 11:58 |
| *** weshay|rover is now known as weshay | 12:03 | |
| *** priteau has joined #openstack-ansible | 12:11 | |
| *** vollman has joined #openstack-ansible | 12:14 | |
| admin0 | hi guys .. i have a strange issue in my hypervisors .. https://pastebin.com/9Gmf6d1h -- the root disk is / 100% full .. but there is nothing that i can see what will help identify why this is the case | 12:17 |
| *** markvoelker has quit IRC | 12:32 | |
| masterpe | admin0: what do you see when you run: du -h -x --max-depth=1 / | 12:34 |
| mgariepy | admin0, maybe you have some process that have a/some file open, until the handle is release the data is not deleted on disk. | 12:34 |
| admin0 | mgariepy, running that command now | 12:34 |
| admin0 | mgariepy, is there a way to find such process ? | 12:34 |
| masterpe | the -x option specifies that it only collect the diskuse with that disk | 12:35 |
| mgariepy | lsof | 12:35 |
| *** ansmith_ has joined #openstack-ansible | 12:35 | |
| mgariepy | 1.5T var << what's in this folder ? | 12:36 |
| masterpe | var is 1.5TB, probably your /var/log folder is full | 12:37 |
| masterpe | with du -x it doesn't count your /var/lib/nova | 12:37 |
| admin0 | mgariepy, https://pastebin.com/7LLxT977 <- https://pastebin.com/7LLxT977 | 12:37 |
| mgariepy | 70G /var | 12:38 |
| mgariepy | << | 12:38 |
| odyssey4me | mnaser I think https://review.openstack.org/636192 may be of interest to you with regards to the repo server selection - it would seem that perhaps there's a need for improvement there to try and make it use the same repo server as the one lsyncd is setup on | 12:38 |
| *** vollman has quit IRC | 12:40 | |
| *** nurdie has joined #openstack-ansible | 12:43 | |
| CeeMac | does anyone know if the vpnaas dashboard plugin should be enabled/installed automatically when vpnaas is added to neutron_plugin_base (as per fwaas and lbaas)? | 12:44 |
| CeeMac | also, if the kernel modules area 'skipped' does this mean some changes are made to the underlying host to support the functionality? | 12:45 |
| CeeMac | ^skipped during the playbook run | 12:45 |
| *** nicolasbock has joined #openstack-ansible | 12:45 | |
| *** nicolasbock has quit IRC | 12:47 | |
| *** nicolasbock has joined #openstack-ansible | 12:47 | |
| *** nurdie has quit IRC | 12:48 | |
| jrosser | CeeMac: see this http://git.openstack.org/cgit/openstack/openstack-ansible/tree/inventory/group_vars/horizon_all.yml#n30 | 12:51 |
| *** priteau has quit IRC | 12:51 | |
| *** priteau has joined #openstack-ansible | 12:53 | |
| CeeMac | thansk jrosser I couldn't see that in the main docs, and the install doc didn't say either way :) | 12:59 |
| jrosser | CeeMac: codesearch is great http://codesearch.openstack.org/?q=horizon_enable_neutron_vpnaas&i=nope&files=&repos= | 12:59 |
| CeeMac | ah, thats neat, thanks | 13:00 |
| jrosser | i was wondering if we had an opinion on this, as a deployment project http://lists.openstack.org/pipermail/openstack-discuss/2019-March/004329.html | 13:02 |
| guilhermesp | mornings | 13:04 |
| openstackgerrit | Merged openstack/openstack-ansible-tests master: Stop deleting-and-creating RabbitMQ account on deploy https://review.openstack.org/648887 | 13:09 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_aodh master: Updated from OpenStack Ansible Tests https://review.openstack.org/648996 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_barbican master: Updated from OpenStack Ansible Tests https://review.openstack.org/648997 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_blazar master: Updated from OpenStack Ansible Tests https://review.openstack.org/639522 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.openstack.org/648998 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cinder master: Updated from OpenStack Ansible Tests https://review.openstack.org/648999 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_cloudkitty master: Updated from OpenStack Ansible Tests https://review.openstack.org/639525 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_congress master: Updated from OpenStack Ansible Tests https://review.openstack.org/649000 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_designate master: Updated from OpenStack Ansible Tests https://review.openstack.org/649001 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_glance master: Updated from OpenStack Ansible Tests https://review.openstack.org/649002 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_heat master: Updated from OpenStack Ansible Tests https://review.openstack.org/649003 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/649004 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_keystone master: Updated from OpenStack Ansible Tests https://review.openstack.org/649005 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_magnum master: Updated from OpenStack Ansible Tests https://review.openstack.org/649006 | 13:18 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_manila master: Updated from OpenStack Ansible Tests https://review.openstack.org/649007 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_masakari master: Updated from OpenStack Ansible Tests https://review.openstack.org/649008 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_mistral master: Updated from OpenStack Ansible Tests https://review.openstack.org/649009 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests https://review.openstack.org/649010 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_nova master: Updated from OpenStack Ansible Tests https://review.openstack.org/649011 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.openstack.org/649012 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.openstack.org/649014 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests https://review.openstack.org/649015 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_tacker master: Updated from OpenStack Ansible Tests https://review.openstack.org/649016 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_trove master: Updated from OpenStack Ansible Tests https://review.openstack.org/649017 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_watcher master: Updated from OpenStack Ansible Tests https://review.openstack.org/649018 | 13:19 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/openstack-ansible-os_zun master: Updated from OpenStack Ansible Tests https://review.openstack.org/649019 | 13:19 |
| *** irclogbot_3 has joined #openstack-ansible | 13:25 | |
| *** nurdie has joined #openstack-ansible | 13:30 | |
| *** nurdie has quit IRC | 13:34 | |
| *** jroll has quit IRC | 13:39 | |
| *** jroll has joined #openstack-ansible | 13:39 | |
| *** nurdie has joined #openstack-ansible | 13:40 | |
| *** CeeMac_ has joined #openstack-ansible | 13:41 | |
| *** CeeMac has quit IRC | 13:45 | |
| nurdie | I don't have a dedicated network host in my OSA environment. In Pike, neutron-agents were running in containers on 3 controllers. After the Queens upgrade, they are gone and and I have no neutron agents L3, DHCP, etc | 13:47 |
| nurdie | Can someone help me figure out why the OSA plays are ignoring my agents and not installing them on the controllers like they are supposed to? | 13:47 |
| nurdie | I must be missing some user_variables entry or something | 13:48 |
| jrosser | nurdie: i would start with ./scripts/inventory_manage.py -G to look at the ansible groups | 13:49 |
| jrosser | and see if you've got groups for neutron_agent and neutron_dhcp_agent that contain your infra hosts | 13:50 |
| nurdie | jrosser: I do. Both of those groups say infra1, infra2, and infra3, which should be correct. Those are my controllers | 13:52 |
| mnaser | ho | 13:55 |
| mnaser | ,e has finished [ittomg pit fores | 13:55 |
| mnaser | dfgosfg | 13:55 |
| * mnaser has finished putting out fires | 13:55 | |
| mnaser | evrardjp: I would like some release advice from you | 13:56 |
| evrardjp | yup? | 13:56 |
| evrardjp | if I can | 13:56 |
| mnaser | evrardjp: : we have https://review.openstack.org/#/q/topic:stein-deploy which actually are things that result in a broken deployment and all the OpenStack Ansible tests updated from patches which result in breaking deployments. do you think we should release the rc anyways, back port and do another rc, or hold off for those? | 13:57 |
| jrosser | nurdie: i am looking a bit suspiciously at this https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-neutron-install.yml#L49 | 13:57 |
| evrardjp | We should wait for an ok rc. | 13:57 |
| evrardjp | People are actively trying RCs, or at least they were in the past | 13:58 |
| jrosser | nurdie: and i'm wondering what neutron_all actually is there | 13:58 |
| jrosser | openstack-ansible playbooks/os-neutron-install.yml --limit 'neutron_all:!neutron_linuxbridge_agent:!neutron_openvswitch_agent:!neutron_server' --list-hosts | 14:01 |
| jrosser | ERROR! Specified hosts and/or --limit does not match any hosts | 14:01 |
| noonedeadpunk | I was rolling 1 prod on rocky rc:) | 14:01 |
| jrosser | ^ so what is going on here? i have no idea | 14:01 |
| nurdie | jrosser: Yep | 14:02 |
| nurdie | That's what I'm seeing as well | 14:02 |
| nurdie | Should I change that to neutron_agent instead of neutron_all and see what happens? | 14:03 |
| mnaser | evrardjp: ok, so in that case, let's land those o-a-t changes and stein-deploy topic | 14:03 |
| *** priteau has quit IRC | 14:03 | |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible-os_neutron master: Add privsep-helper to neutron sudoers file https://review.openstack.org/648890 | 14:04 |
| nurdie | jrosser: That doesn't help either | 14:04 |
| mnaser | evrardjp: left a comment for you @ https://review.openstack.org/#/c/648886/1 | 14:04 |
| *** mrhillsman_wkend is now known as mrhillsman | 14:04 | |
| evrardjp | mnaser: ok I won't remove my -w until we update the appropriate sha (if necessary) | 14:04 |
| mnaser | evrardjp: thanks, I think this is the last of it, I haven't shaken up neutron a lot.. so that might result in more patches | 14:05 |
| evrardjp | mnaser: I trust you | 14:05 |
| evrardjp | it looks okay anyway | 14:05 |
| evrardjp | :) | 14:05 |
| mnaser | \o/ | 14:05 |
| mnaser | im exicted | 14:05 |
| evrardjp | it's just we are not testing this, so I am careful | 14:05 |
| mnaser | yeah, I dunno why we default to spice | 14:06 |
| mnaser | when the upstream default is novnc | 14:06 |
| mgariepy | nurdie, i think the default changed in queens the agent would run on metal. | 14:06 |
| mnaser | I have this on the list of "let's reimagine our roles in a perfectly world" | 14:06 |
| noonedeadpunk | +1 for novnc by default | 14:06 |
| mnaser | "zero overrides." | 14:06 |
| nurdie | mgariepy: Yes I know that | 14:07 |
| noonedeadpunk | spice is disgusting:) | 14:07 |
| mnaser | as in like, we give you the ability to override things, but, we don't change things by default. | 14:07 |
| cloudnull | mornings | 14:07 |
| nurdie | The problem I'm having is that the plays aren't installing the agents on my controllers | 14:07 |
| noonedeadpunk | and it's not working really good with graphic interfaces, like windows - to laggy | 14:07 |
| jrosser | mgariepy: do you know what neutron_all is supposed to be here https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-neutron-install.yml#L49 | 14:07 |
| jrosser | i just don't understand how that works :/ | 14:08 |
| evrardjp | noonedeadpunk: spice used to be okayish | 14:08 |
| evrardjp | but now spice is kinda dead | 14:08 |
| evrardjp | and novnc is kinda back alive | 14:08 |
| evrardjp | jrosser: ok so | 14:08 |
| evrardjp | jrosser: the idea is that we run this in the order for the upgrades | 14:09 |
| evrardjp | in the playbook | 14:09 |
| noonedeadpunk | the problem is, that you can't you some spice client, as they do not support hashes in url | 14:09 |
| evrardjp | so it means neutron server run first | 14:09 |
| evrardjp | L33 | 14:09 |
| evrardjp | then you run the agents, but you don't need to reapply on neutron_server (41) | 14:09 |
| evrardjp | then to all the rests of the agents (because neutron advised order) which havent't run already | 14:10 |
| noonedeadpunk | like nova provides during url creation. so the only way seemed to me is use spice console. And customize it to get at least ctrl+alt+del working | 14:10 |
| mgariepy | jrosser, maybe for other network stuff, like calico or others. i don't have any host that match that on my install. | 14:10 |
| noonedeadpunk | *spice html5 console | 14:10 |
| evrardjp | this runs like this because when we run neutron on a machine that has two roles, both roles will be applied at the same time. | 14:10 |
| nurdie | evrardjp: That all makes sense but the common-playbooks/neutron.yml doesn't end up installing the L3, DHCP, etc (other agents) on my controllers | 14:10 |
| evrardjp | jrosser: ^ | 14:10 |
| mgariepy | jrosser, but my agent is in container. | 14:10 |
| evrardjp | nurdie: because the same node is used for many things then? | 14:10 |
| evrardjp | check your inventory | 14:11 |
| evrardjp | mgariepy: it totally depends on what you put on your nodes/how you split your ivnentory | 14:11 |
| evrardjp | :) | 14:11 |
| mgariepy | computing is hard. | 14:12 |
| evrardjp | :) | 14:13 |
| nurdie | My inventory was created by OSA o_0 | 14:14 |
| evrardjp | computers too, don't shoot in one | 14:14 |
| evrardjp | nurdie: with your input :) | 14:14 |
| nurdie | You know I doubt it updated them from containers to onmetal for me | 14:14 |
| nurdie | Well, yes, but I had nothign neutron specific in my user_variables | 14:14 |
| evrardjp | the best way is to check the inventory.json | 14:15 |
| evrardjp | or th openstack_user_config too | 14:15 |
| noonedeadpunk | or use scripts/inventory-manage.py :) | 14:17 |
| evrardjp | you can create a little playbook that shows the inventory too, if you prefer | 14:17 |
| evrardjp | wow I wouldn't use that :p | 14:17 |
| evrardjp | in fact it's okay | 14:17 |
| evrardjp | I am just bad mouthing here | 14:17 |
| evrardjp | hahah | 14:17 |
| nurdie | My inventory-manage.py already shows that I do have neutron_agents and neutron_dhcp_agents groups set to my controllers | 14:18 |
| mgariepy | that script never broke my inventory, but i keep it in git, and commit every changes ;) haha | 14:18 |
| nurdie | The plays still ignore that | 14:18 |
| jrosser | nurdie: do you have hosts in the group neutron_l3_agent? | 14:21 |
| *** phasespace has quit IRC | 14:22 | |
| nurdie | jrosser: Yes, same. infra1, infra2, infra3 | 14:26 |
| openstackgerrit | Merged openstack/openstack-ansible-os_horizon master: Drop horizon_keystone_host variable https://review.openstack.org/648898 | 14:28 |
| nurdie | fatal: [infra1]: FAILED! => {"failed": true, "msg": "The task includes an option with an undefined variable. The error was: {{ properties.is_metal | default(false) }}: 'properties' is undefined\n\nThe error appears to have been in '/opt/openstack-ansible/playbooks/common-tasks/dynamic-address-fact.yml': line 16, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be | 14:28 |
| nurdie | :\n\n\n- name: Gather address fact\n ^ here\n\nexception type: <class 'ansible.errors.AnsibleUndefinedVariable'>\nexception: {{ properties.is_metal | default(false) }}: 'properties' is undefined"} | 14:28 |
| nurdie | Sorry for the line break | 14:28 |
| nurdie | I was also having that issue with the haproxy play. I had to --extra_vars 'is_metal: true' for haproxy during the upgrade | 14:29 |
| nurdie | But I can't do that with the whole neutron play | 14:29 |
| openstackgerrit | Merged openstack/openstack-ansible-os_manila master: Updated from OpenStack Ansible Tests https://review.openstack.org/649007 | 14:30 |
| nurdie | Should I define is_metal int he inventory for neutron_dhcp_agent, neutron_l3_agent, etc manually? | 14:31 |
| odyssey4me | nurdie do you happen to have files in /etc/openstack_deploy/env.d ? | 14:35 |
| odyssey4me | nurdie those will override the defaults that are in-tree, are are likely what's causing a problem in your inventory, because whatever's there is old | 14:35 |
| nurdie | yes I do | 14:37 |
| nurdie | odyssey4me: I have a neutron file too: https://pastebin.com/gWMHexik | 14:38 |
| nurdie | I just added properties: is_metal: true to the "neutron_agent" component_skel. That had no effect | 14:38 |
| *** med_ has joined #openstack-ansible | 14:39 | |
| odyssey4me | nurdie do you need any user-space overrides of the defaults - for example, are you trying to ensure that cinder-volume goes into a container or something? or are they there simply because the upgrade process put them there? | 14:40 |
| odyssey4me | you could, for example, do https://github.com/openstack/openstack-ansible/blob/master/scripts/upgrade-utilities/deploy-config-changes.yml#L23-L41 to eliminate anything that matches the in-tree versions, then inspect the rest to see how they differ from the in-tree versions and ask why | 14:40 |
| nurdie | odyssey4me: https://pastebin.com/QhnyQ16G <-- my user_variables.yml | 14:41 |
| odyssey4me | nurdie I'm not speaking of user_variables.yml... I'm speaking of env.d files | 14:41 |
| nurdie | odyssey4me: Ah. So just run that for loop in a script to clean any default stuff | 14:42 |
| odyssey4me | nurdie make a backup first, of course | 14:42 |
| odyssey4me | although you should still have one from the run-upgrade script | 14:42 |
| nurdie | ok i backed it up and ran that script | 14:45 |
| nurdie | odyssey4me: Should I try a os-neutron-install run now? | 14:45 |
| nurdie | It deleted almost everything in there | 14:46 |
| *** CeeMac_ is now known as CeeMac | 14:47 | |
| odyssey4me | nurdie ok, now what is left there - and do you know why it's left there? | 14:47 |
| nurdie | cinder, heat, ironic, neutron, nova, trove | 14:49 |
| nurdie | I know why cinder is there | 14:50 |
| odyssey4me | ok, use diff to check how they differ from the in-tree file and then figure out if you made those change or not - if not, then it's likely a legacy of the past and you can remove it | 14:52 |
| odyssey4me | That may mean that you'll need to then clean up some old containers, and then clean the inventory afterwards. | 14:52 |
| openstackgerrit | Merged openstack/openstack-ansible-os_watcher master: Updated from OpenStack Ansible Tests https://review.openstack.org/649018 | 14:53 |
| odyssey4me | nurdie you're running queens, right? | 14:54 |
| nurdie | odyssey4me: Yes | 14:54 |
| odyssey4me | nurdie ok, so I'm guessing you still have members in the inventory for neutron_agents_container ? | 14:55 |
| odyssey4me | possibly also for cinder_scheduler_container ? | 14:56 |
| nurdie | odyssey4me: I did. I just deleted it to see what would happen with the stock neutron envd | 14:57 |
| nurdie | We had no special neutron directives | 14:57 |
| nurdie | overrides* | 14:57 |
| odyssey4me | ok, but that's not the whole story | 14:58 |
| odyssey4me | just cleaning up the env.d files does nothing - we also have to clean up the inventory entries | 14:58 |
| nurdie | ok | 14:58 |
| odyssey4me | you should be able to either run https://github.com/openstack/openstack-ansible/blob/stable/queens/scripts/upgrade-utilities/playbooks/cleanup-neutron.yml or execute the commands it runs by hand, whichever you prefer | 14:58 |
| nurdie | I ran that play | 14:59 |
| odyssey4me | you ran it after the env.d file removal? | 14:59 |
| nurdie | That's what fucked me. I ran that not knowing that the agents weren't running on the controllers. I didn't check first | 15:00 |
| nurdie | I have not! | 15:00 |
| nurdie | I will rerun | 15:00 |
| nurdie | I apologize for my language. I forgot I was on Freenode :) | 15:00 |
| odyssey4me | ok, are you still missing agents? if so, run the os-neutron-install.yml playbook | 15:01 |
| nurdie | I just tried. I still get the is_metal error | 15:01 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cloudkitty master: Updated from OpenStack Ansible Tests https://review.openstack.org/639525 | 15:02 |
| openstackgerrit | Merged openstack/openstack-ansible-os_trove master: Updated from OpenStack Ansible Tests https://review.openstack.org/649017 | 15:02 |
| odyssey4me | hmm, so that sounds like something is funky in the inventory itself | 15:02 |
| odyssey4me | unfortunately I don't have a healthy inventory to reference right now - I wonder if anyone else online does to help compare things | 15:02 |
| nurdie | odyssey4me: The cleanup play did modify my inventory. I'm rerunning os-neutron-install now | 15:03 |
| *** ivve has quit IRC | 15:06 | |
| nurdie | Yes I must have something messy in the inventory file | 15:06 |
| *** priteau has joined #openstack-ansible | 15:08 | |
| odyssey4me | nurdie I wonder if it'd make sense to use a clean vm with the same config files, generate the inventory, then compare the two - obviously the various container IP's and names will differ, but there will be some content related to is_metal that will differ and may be key to figuring it out | 15:08 |
| jrosser | nurdie: you also have /etc/openstack_deploy/backup_openstack_inventory.tar if you want to see what has changed in your inventory | 15:09 |
| jrosser | although total bonus points for managing /etc/openstack_deploy with git so it's just a git diff away..... | 15:10 |
| CeeMac | jrosser, tried using that variable, and the dashboard still isnt' showing. I've checked the horizon container and sure enough, the dashboard py file isn't in the local/enabled area | 15:10 |
| CeeMac | jrosser, do you know if it relies in a 'present' check against the neutron servers for if the functionality is actually there? | 15:10 |
| CeeMac | i'm wondering if the 'skipped' kernel modules have prevented to role from being enabled/installed | 15:11 |
| CeeMac | jrosser, for vpnaas that is | 15:11 |
| jrosser | CeeMac: which variable again? | 15:11 |
| CeeMac | horizon_enable_neutron_vpnaas | 15:11 |
| CeeMac | set that to true and reran the horizon playbook | 15:11 |
| CeeMac | vpnaas is listed under the service_plugins in neutron.conf on the neutron-server container | 15:13 |
| nurdie | odyssey4me, jrosser: I'm currently reviewing https://bugs.launchpad.net/openstack-ansible/+bug/1785592 | 15:16 |
| openstack | Launchpad bug 1785592 in openstack-ansible "dynamic inventory doesn't handle is_metal: false->true change" [High,Fix released] - Assigned to Jesse Pretorius (jesse-pretorius) | 15:16 |
| nurdie | Says it was fixed in 17.1.7. We're using 17.1.9 o_0 | 15:16 |
| CeeMac | and the [vpnagent] | 15:16 |
| CeeMac | is showing in l3_agent.ini | 15:16 |
| CeeMac | so I'm assuming its installed on the network node ok, just resisting the dashboard plugin | 15:16 |
| jrosser | CeeMac: here is the condition http://git.openstack.org/cgit/openstack/openstack-ansible/tree/inventory/group_vars/horizon_all.yml#n30 | 15:17 |
| fnpanic | CeeMac: i am not sure if the dashboard is implemented in osa | 15:18 |
| odyssey4me | nurdie ok, so let's verify if it's the same issue - are there still containers in the neutron_all group? which container names are they | 15:18 |
| fnpanic | i think it just does not copy the files | 15:18 |
| *** gillesMo has joined #openstack-ansible | 15:19 | |
| fnpanic | https://bugs.launchpad.net/openstack-ansible/+bug/1750789 | 15:19 |
| openstack | Launchpad bug 1750789 in openstack-ansible "VPNaaS dashboard not copied to horizon container" [Wishlist,Confirmed] | 15:19 |
| nurdie | odyssey4me: [root@deployhost openstack-ansible]# ./scripts/inventory-manage.py -G | grep 'neutron_all' | 15:19 |
| nurdie | That returns nothing | 15:19 |
| CeeMac | jrosser, ok so both of this conditions are true | 15:19 |
| odyssey4me | nurdie well, no - try this rather: ansible -m ping neutron_all | 15:19 |
| CeeMac | fnpanic, oh? is this something you've come across too? | 15:19 |
| nurdie | odyssey4me: That pings all of my compute hosts and my 3 controllers | 15:20 |
| fnpanic | sure | 15:20 |
| nurdie | odyssey4me: And my 3 neutron server containers | 15:20 |
| CeeMac | jrosser, that test is overridden if we manually specify horizon_enable_neutron_vpnaas as true? | 15:20 |
| nurdie | odyssey4me: There are no neutron_agents containers in it | 15:20 |
| odyssey4me | nurdie ok, so nothing else - alright, and how about: ansible -m ping neutron_dhcp_agent | 15:21 |
| jrosser | well i think you need to follow the code, like to here https://github.com/openstack/openstack-ansible-os_horizon/blob/master/tasks/horizon_install_source.yml#L61 | 15:21 |
| nurdie | odyssey4me: It pings my 3 controllers, infra1, infra2 and infra3 | 15:21 |
| CeeMac | jrosser, yeah that was going to be my next step. Was hoping for a lazy answer though :D | 15:21 |
| jrosser | so if the files are not in the venv then it needs to be rebuilt, i think | 15:22 |
| CeeMac | getting close to end of play for me, i'll check again in the morning | 15:22 |
| odyssey4me | nurdie ok, and yet running the neutron playbook doesn't make the neutron agent services run on the host? | 15:22 |
| CeeMac | rebuilt how? | 15:22 |
| nurdie | odyssey4me: That is correct | 15:22 |
| nurdie | I'll show you one moment | 15:22 |
| *** luksky has quit IRC | 15:23 | |
| odyssey4me | nurdie ok, on any one of the controllers, can you list the neutron-* services running on them? | 15:24 |
| odyssey4me | nurdie then also list the systemd units configured | 15:24 |
| nurdie | odyssey4me: They aren't there | 15:25 |
| odyssey4me | nurdie systemctl list-unit-files --type=service | grep neutron | 15:25 |
| odyssey4me | no services at all? | 15:25 |
| nurdie | odyssey4me: Correct. None. | 15:25 |
| nurdie | There is one dhcp agent on controller1 that I'm trying to configure manually lol | 15:26 |
| odyssey4me | nurdie so there're absolutely no neutron agents whatsoever? | 15:26 |
| nurdie | odyssey4me: https://pastebin.com/JrhtqPmf | 15:26 |
| nurdie | odyssey4me: Correct. No neutron agents whatsoever | 15:27 |
| nurdie | odyssey4me: The only neutron agents currently in my stack are the linuxbridge-agents running on my compute nodes | 15:27 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Eliminate nova_novnc_distro_packages https://review.openstack.org/648886 | 15:27 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Add privsep-helper to nova sudoers file https://review.openstack.org/648888 | 15:27 |
| odyssey4me | nurdie ok, and you do have a network_hosts section in openstack_user_config.yml ? | 15:28 |
| openstackgerrit | Merged openstack/openstack-ansible-os_blazar master: Updated from OpenStack Ansible Tests https://review.openstack.org/639522 | 15:29 |
| nurdie | odyssey4me: Yes. It is set to "*infrastructure_hosts", and that is defined as "&infrastructure_hosts" and has all 3 of our controllers listed | 15:29 |
| odyssey4me | nurdie ok, and you don't have an /etc/openstack_deploy/env.d file with any content related to neutron any more? | 15:30 |
| openstackgerrit | Merged openstack/openstack-ansible-os_congress master: Updated from OpenStack Ansible Tests https://review.openstack.org/649000 | 15:30 |
| nurdie | odyssey4me: yes. I backed it up and deleted it | 15:30 |
| *** shananigans has joined #openstack-ansible | 15:31 | |
| nurdie | Should I just try directly setting network_hosts to my controllers? | 15:31 |
| nurdie | It's the is_metal property that's failing it | 15:31 |
| odyssey4me | nurdie nope, that won't affect things, unless you happen to have a yaml typo | 15:32 |
| nurdie | I haven't seen any syntax related failures | 15:32 |
| odyssey4me | nurdie ok, if you open inventory.json with less or something like that, can you find the key word 'properties' for your infra hosts? if you compare that to the compute hosts, it seems that 'properties' exists there... | 15:33 |
| odyssey4me | nurdie perhaps you could extract and sanitise a sample from an infra host and a compute host and share it? | 15:34 |
| nurdie | odyssey4me: https://pastebin.com/htv0ZjVi | 15:37 |
| odyssey4me | nurdie well, that's odd - the missing property appears to be there? | 15:39 |
| nurdie | That's from /etc/openstack_deploy/openstack_inventory.json | 15:39 |
| nurdie | odyssey4me: yes >_< | 15:39 |
| odyssey4me | nurdie ok, if you execute /opt/openstack-ansible/inventory/dynamic_inventory.py then does it spit out the same contents as /etc/openstack_deploy/openstack_inventory.json ? | 15:40 |
| nurdie | odyssey4me: https://pastebin.com/rSem1swH | 15:42 |
| odyssey4me | nurdie hmm, ok one more thing to try - can you do: rm -rf /etc/openstack_deploy/ansible_facts | 15:43 |
| *** fabstao has joined #openstack-ansible | 15:44 | |
| nurdie | odyssey4me: Ok, now try os-neutron-install? | 15:44 |
| odyssey4me | then: cd /opt/openstack-ansible/playbooks; ansible -m debug -a 'var=properties' neutron_agents | 15:44 |
| openstackgerrit | Merged openstack/openstack-ansible-os_zun master: Updated from OpenStack Ansible Tests https://review.openstack.org/649019 | 15:45 |
| nurdie | odyssey4me: https://pastebin.com/0xcP0mEY | 15:45 |
| nurdie | Nothing! | 15:45 |
| odyssey4me | wait a minute - I think that's not a correct group name, use neutron_all instead | 15:46 |
| nurdie | For neutron_all, I see great properties for all EXCEPT infra1 2 and 3. Those all say "properties": "VARIABLE IS NOT DEFINED!" | 15:47 |
| odyssey4me | that's infuriating | 15:48 |
| *** priteau has quit IRC | 15:48 | |
| nurdie | odyssey4me: hahahaha Yessiiirr! | 15:48 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ironic master: Updated from OpenStack Ansible Tests https://review.openstack.org/649004 | 15:49 |
| odyssey4me | nurdie are there also warnings about the names? | 15:49 |
| nurdie | odyssey4me: Yes | 15:50 |
| odyssey4me | hmm, I wonder if that's an issue here | 15:50 |
| odyssey4me | '[WARNING]: Found both group and host with same name: infra1' | 15:50 |
| nurdie | odyssey4me: https://pastebin.com/u1AsxMhT | 15:50 |
| nurdie | I can't find any groups with the name infra1 though | 15:50 |
| odyssey4me | I suspect that's the issue, although I don't know how it is that there's a group called infra{1,2,3} | 15:51 |
| nurdie | The only group I can find is infra1-host_containers | 15:51 |
| nurdie | ./scripts/inventory-manage.py -G |grep infra1 | 15:51 |
| odyssey4me | ansible -m debug -a 'var=groups' localhost | 15:52 |
| openstackgerrit | Merged openstack/openstack-ansible-os_tacker master: Updated from OpenStack Ansible Tests https://review.openstack.org/649016 | 15:52 |
| nurdie | There aarreeeee | 15:52 |
| nurdie | There are infra1 infra2 and infra3 groups! | 15:52 |
| odyssey4me | nurdie right, ok - are those in the json file? | 15:53 |
| odyssey4me | jrosser if you're able to, can you check your inventory to see if you have any groups with the exact same name as any hosts | 15:53 |
| nurdie | odyssey4me: Yes, I think I found them | 15:55 |
| nurdie | "infra1": { "children": [], "hosts: [] } | 15:55 |
| nurdie | Should I delete those for infra1 2 and 3? | 15:55 |
| odyssey4me | nurdie I don't think they're supposed to exist, but it'd be good to get feedback from someone else with an inventory | 15:56 |
| nurdie | I can try it and see what happens. I have that inventory file backed up | 15:56 |
| odyssey4me | sure, why not - nothing to lose, right? ;) | 15:56 |
| admin0 | cloudnull, around ? | 15:57 |
| admin0 | hoping to test the ELK on bionic today | 15:58 |
| nurdie | odyssey4me: How do I get those changes to update? | 15:58 |
| nurdie | ansible -m debug -a 'var=groups' localhost still shows them | 15:58 |
| odyssey4me | nurdie just run /opt/openstack-ansible/inventory/dynamic_inventory.py afterwards and check the output | 15:58 |
| nurdie | ok | 15:58 |
| nurdie | odyssey4me: They are still there | 15:59 |
| jrosser | odyssey4me: i don't appear to have groups == hostname anywhere | 15:59 |
| odyssey4me | nurdie ok, I suspect there's some sort of formatting issue in openstack_user_config... can you share a sanitised version of it without all the network parts? | 16:02 |
| odyssey4me | jrosser thanks for confirming that | 16:02 |
| odyssey4me | nurdie also, do you have any /etc/openstack_deploy/conf.d files? | 16:02 |
| nurdie | odyssey4me: https://pastebin.com/VSMDeuW0 | 16:03 |
| jrosser | odyssey4me: heres "ansible -m debug -a 'var=groups' localhost" https://pasted.tech/pastes/e4cfd412b0004d9aeb7c4a4d83372eb63814bfaf | 16:03 |
| nurdie | odyssey4me: yes, but none of them look like they have anything custom | 16:04 |
| odyssey4me | jrosser looks like you do have a host called 'infra1' and a group by the same name? | 16:04 |
| odyssey4me | oh, I misread - that's a member of the 'all' group | 16:04 |
| nurdie | odyssey4me: Except for haproxy.yml is in there | 16:04 |
| nurdie | Where the HECK is my infra1,2,3 empty groups coming from >_< | 16:05 |
| nurdie | I can't find it | 16:05 |
| nurdie | I don't think they should be there either | 16:06 |
| nurdie | I never set them and I feel like those groups came from the Ocata to Pike upgrade | 16:06 |
| openstackgerrit | Merged openstack/openstack-ansible-os_nova master: Updated from OpenStack Ansible Tests https://review.openstack.org/649011 | 16:06 |
| odyssey4me | nurdie ok, I wonder if those groups can be listed with inventory-manage? | 16:07 |
| odyssey4me | oh, but they're not deletable via inventory-manage so that's not useful | 16:07 |
| openstackgerrit | Merged openstack/openstack-ansible-os_sahara master: Updated from OpenStack Ansible Tests https://review.openstack.org/649014 | 16:07 |
| nurdie | odyssey4me: Those groups don't show up in inventory-manage.py -G | 16:08 |
| openstackgerrit | Merged openstack/openstack-ansible-os_magnum master: Updated from OpenStack Ansible Tests https://review.openstack.org/649006 | 16:08 |
| openstackgerrit | Merged openstack/openstack-ansible-os_swift master: Updated from OpenStack Ansible Tests https://review.openstack.org/649015 | 16:12 |
| openstackgerrit | Merged openstack/openstack-ansible-os_cinder master: Updated from OpenStack Ansible Tests https://review.openstack.org/648999 | 16:12 |
| odyssey4me | nurdie if you can use a fresh vm with that config to validate whether those groups exist in a clean environment, then we know there's a bug in the queens code... if they aren't there, then they have somehow been inherited from something done previously and we'll have to figure out how to remove them | 16:13 |
| odyssey4me | in an ideal world, it would be nice if the dynamic inventory tested for this situation and simply failed if it finds this | 16:13 |
| openstackgerrit | Merged openstack/openstack-ansible-os_glance master: Updated from OpenStack Ansible Tests https://review.openstack.org/649002 | 16:14 |
| openstackgerrit | Merged openstack/openstack-ansible-os_heat master: Updated from OpenStack Ansible Tests https://review.openstack.org/649003 | 16:15 |
| nurdie | odyssey4me: Like try a fresh deployhost? | 16:15 |
| openstackgerrit | Merged openstack/openstack-ansible-os_barbican master: Updated from OpenStack Ansible Tests https://review.openstack.org/648997 | 16:16 |
| odyssey4me | I'm building one now to check. | 16:16 |
| odyssey4me | Assuming my battery lasts. ;) | 16:16 |
| nurdie | odyssey4me: Thank you so much. Just in case your battery doesn't last, is that what you're talking about? Start a new deployhost with our config and see what the inventory file does? | 16:17 |
| odyssey4me | nurdie yep - just a throwaway container/VM prepared like a deployment host, just to run the inventory script to see what the resulting jason looks like and how it compares | 16:17 |
| *** cmart has joined #openstack-ansible | 16:18 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Updated from OpenStack Ansible Tests https://review.openstack.org/649010 | 16:18 |
| jrosser | nurdie: if you've not commited /etc/openstack_deploy to a git repo now would be a really good time to do that | 16:18 |
| nurdie | jrosser: I have! :) | 16:19 |
| jrosser | excellent :) | 16:19 |
| odyssey4me | nurdie I wonder if your previous attempt to get rid of the group in the json file was perhaps incomplete? | 16:19 |
| nurdie | I was searching for "infra": | 16:20 |
| nurdie | "infraN": | 16:20 |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Updated from OpenStack Ansible Tests https://review.openstack.org/649005 | 16:20 |
| nurdie | Even when I delete those groups in the inventory file, the install play still fails with the is_metal property missing | 16:22 |
| *** cmart has quit IRC | 16:22 | |
| odyssey4me | nurdie well, ceph.ceph.* are giving invalid address errors, because those aren't IP's | 16:23 |
| nurdie | lol sorry | 16:23 |
| nurdie | That was my sanitization | 16:23 |
| odyssey4me | ok, it helps to know that | 16:25 |
| odyssey4me | alright, it's confirmed that a clean env doesn't generate those groups - so it must be a cause of some other config, or an existing part of the json file | 16:26 |
| openstackgerrit | Merged openstack/openstack-ansible-ceph_client master: Remove unnecessary GPG keys setting for YUM repos. https://review.openstack.org/648877 | 16:27 |
| nurdie | I'm creating a VM right now to test with my whole openstack_deploy directory | 16:27 |
| odyssey4me | nurdie with the whole dir will generate the same results - so that's a bit pointless | 16:28 |
| nurdie | odyssey4me: Ah | 16:28 |
| odyssey4me | are there conf.d files? | 16:28 |
| odyssey4me | is there any mention of infra1 in any of the env.d files? | 16:28 |
| openstackgerrit | Merged openstack/openstack-ansible-os_octavia master: Updated from OpenStack Ansible Tests https://review.openstack.org/649012 | 16:29 |
| nurdie | There are conf.d files. The only one out of place in there is haproxy.yml, which just has infra1 2 and 3's IP listed | 16:29 |
| odyssey4me | nurdie do any of the conf.d files duplicate content from openstack_user_config.yml ? | 16:29 |
| *** kopecmartin is now known as kopecmartin|off | 16:30 | |
| *** hwoarang has quit IRC | 16:31 | |
| odyssey4me | nurdie by content I mean do they assign any information to the same host groups? | 16:33 |
| odyssey4me | nurdie FYI you can consolidate everything into openstack_user_config - conf.d files are just a convenience... it all gets merged into a single result anyway | 16:34 |
| openstackgerrit | Merged openstack/openstack-ansible-os_mistral master: Updated from OpenStack Ansible Tests https://review.openstack.org/649009 | 16:34 |
| *** hwoarang has joined #openstack-ansible | 16:36 | |
| nurdie | odyssey4me: I think it's working now | 16:37 |
| nurdie | I think it was having cinder info duplicated in env.d and openstack_user_config | 16:38 |
| nurdie | inventory was creating groups | 16:38 |
| odyssey4me | nurdie aha! | 16:38 |
| openstackgerrit | Merged openstack/openstack-ansible-os_masakari master: Updated from OpenStack Ansible Tests https://review.openstack.org/649008 | 16:39 |
| nurdie | odyssey4me: Is that a bug? Should I create one? | 16:39 |
| nurdie | It has always been like that, since our Ocata install I guess, so this queens would be the first time that it has effected us | 16:39 |
| odyssey4me | nurdie if the intent with your env.d changes was to make cinder-volume go into a container, then the content there should be very, very mininmal | 16:40 |
| odyssey4me | see the expected config here: https://docs.openstack.org/openstack-ansible-os_cinder/latest/configure-cinder.html#configuring-cinder-to-use-ceph | 16:40 |
| odyssey4me | notice that it's 4 lines | 16:40 |
| nurdie | odyssey4me: I think that's what we had. I'm going to compare our previous config after this meeting in a few | 16:41 |
| odyssey4me | nurdie ok, I suspect the other env.d files you have there aren't needed either | 16:42 |
| *** cmart has joined #openstack-ansible | 16:42 | |
| odyssey4me | with all that sorted, you can make use of the queens cleanup playbooks from the upgrade to ensure you don't have any leftover incorrect containers or inventory entries... then re-run setup-openstack and everything should all be in the right places | 16:42 |
| openstackgerrit | Merged openstack/openstack-ansible-os_ceilometer master: Updated from OpenStack Ansible Tests https://review.openstack.org/648998 | 16:43 |
| odyssey4me | if anything, perhaps a bug to add a check for the condition of a group and host having the same name would be useful for the inventory - although ansible did kinda warn about it | 16:44 |
| odyssey4me | anyway, I'm happy we finally got to the bottom of it - I need to go offline now... enjoy the rest of your day! | 16:45 |
| *** cmart has quit IRC | 16:47 | |
| *** gillesMo has quit IRC | 16:51 | |
| mnaser | cores: can I get a +2 to kick this off to gate https://review.openstack.org/#/c/648890/ | 16:56 |
| mnaser | cc guilhermesp | 16:56 |
| logan- | done mnaser | 16:58 |
| mnaser | thanks logan- | 16:59 |
| mnaser | shaping up to be one heck of a stable release imho | 16:59 |
| *** cmart has joined #openstack-ansible | 16:59 | |
| *** gillesMo has joined #openstack-ansible | 17:01 | |
| *** gillesMo has quit IRC | 17:02 | |
| *** DanyC has quit IRC | 17:03 | |
| *** goldenfri has joined #openstack-ansible | 17:05 | |
| *** gkadam has quit IRC | 17:05 | |
| *** partlycloudy has joined #openstack-ansible | 17:06 | |
| *** hamzaachi has quit IRC | 17:12 | |
| *** hamzaachi has joined #openstack-ansible | 17:14 | |
| *** gillesMo has joined #openstack-ansible | 17:16 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_aodh master: Updated from OpenStack Ansible Tests https://review.openstack.org/648996 | 17:16 |
| *** miloa has quit IRC | 17:20 | |
| *** DanyC has joined #openstack-ansible | 17:27 | |
| *** DanyC has quit IRC | 17:32 | |
| *** phasespace has joined #openstack-ansible | 17:38 | |
| *** electrofelix has quit IRC | 18:12 | |
| *** jonher has quit IRC | 18:29 | |
| *** jonher has joined #openstack-ansible | 18:29 | |
| nurdie | odyssey4me: Thank you for your help! | 18:29 |
| nurdie | I got another fun one | 18:29 |
| *** DanyC has joined #openstack-ansible | 18:29 | |
| nurdie | On controllers, there's a weird bug that isn't allowing the neutron agents to be run as user neutron. SELinux in permissive. CentOS 7. I suspect this is a security hardening bug or a CentOS bug. | 18:30 |
| nurdie | I finally got one of our sets of agents running by editing the unitfiles to run the services as the root user | 18:31 |
| *** CobHead has quit IRC | 18:34 | |
| mnaser | poop | 18:45 |
| mnaser | I found another blocker | 18:45 |
| mnaser | novnc is broken. | 18:45 |
| mnaser | they ripped out some behaviour in 2018 August which broke OpenStack nova | 18:45 |
| mnaser | We will have to pin novnc | 18:46 |
| mnaser | https://github.com/novnc/noVNC/commit/51f9f0098d306bbc67cc8e02ae547921b6f6585c#diff-1d6838e3812778e95699b90d530543a1L173 | 18:46 |
| *** BjoernT has joined #openstack-ansible | 18:55 | |
| *** DanyC has quit IRC | 18:56 | |
| *** DanyC has joined #openstack-ansible | 18:57 | |
| *** DanyC has quit IRC | 19:08 | |
| *** hamzaachi has quit IRC | 19:18 | |
| *** luksky has joined #openstack-ansible | 19:35 | |
| *** spsurya has quit IRC | 19:36 | |
| openstackgerrit | Merged openstack/openstack-ansible master: Automatically prune the inventory backup https://review.openstack.org/565950 | 19:52 |
| *** DanyC has joined #openstack-ansible | 19:57 | |
| *** fabstao has quit IRC | 19:58 | |
| *** cmart has quit IRC | 20:02 | |
| *** mmercer has quit IRC | 20:11 | |
| *** mmercer has joined #openstack-ansible | 20:12 | |
| serverascode | can I set LXC_NETWORK if I don't want the default with openstack-ansible? | 20:13 |
| *** hamzaachi has joined #openstack-ansible | 20:18 | |
| *** cmart has joined #openstack-ansible | 20:20 | |
| *** cmart has quit IRC | 20:28 | |
| *** cmart has joined #openstack-ansible | 20:29 | |
| *** DanyC has quit IRC | 20:34 | |
| * mnaser is not sure about that | 20:47 | |
| mnaser | evrardjp: around? | 20:47 |
| openstackgerrit | Mohammed Naser proposed openstack/openstack-ansible master: Use non-broken commit for NoVNC https://review.openstack.org/649186 | 20:51 |
| *** ansmith_ has quit IRC | 20:59 | |
| *** ivve has joined #openstack-ansible | 21:02 | |
| evrardjp | mnaser going to bed. feel free to ping I will read tomorrow | 21:03 |
| mnaser | evrardjp: ok cool, just wanted to point at that change above ^ novnc broke nova and because we track master we got affected, so just wanted you to check the change to not break your tooling | 21:03 |
| *** eglute has joined #openstack-ansible | 21:04 | |
| *** ivve has quit IRC | 21:09 | |
| *** vnogin has joined #openstack-ansible | 21:13 | |
| *** pcaruana has quit IRC | 21:17 | |
| *** hamzaachi has quit IRC | 21:35 | |
| *** samueldmq has joined #openstack-ansible | 21:36 | |
| *** hamzaachi has joined #openstack-ansible | 21:36 | |
| *** hamzaachi has quit IRC | 21:46 | |
| *** mrhillsman is now known as mrhillsman_afk | 21:46 | |
| *** BjoernT has quit IRC | 22:01 | |
| *** shananigans has quit IRC | 22:11 | |
| *** nurdie has quit IRC | 22:17 | |
| *** nurdie has joined #openstack-ansible | 22:18 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_neutron master: Add privsep-helper to neutron sudoers file https://review.openstack.org/648890 | 22:18 |
| *** vnogin has quit IRC | 22:20 | |
| *** partlycloudy has quit IRC | 22:23 | |
| *** nurdie_ has joined #openstack-ansible | 22:45 | |
| *** nurdie has quit IRC | 22:48 | |
| *** ansmith_ has joined #openstack-ansible | 22:49 | |
| *** nurdie_ has quit IRC | 22:50 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Add pip overrides to basic bootstrap https://review.openstack.org/649200 | 22:56 |
| NobodyCam | Good afternoon folks dumb question: should glance sync images between the glance containers when there are more then one? | 23:27 |
| *** tosky has quit IRC | 23:28 | |
| *** hwoarang has quit IRC | 23:35 | |
| *** hwoarang has joined #openstack-ansible | 23:37 | |
| *** nurdie has joined #openstack-ansible | 23:43 | |
| *** nurdie has quit IRC | 23:47 | |
| *** BjoernT has joined #openstack-ansible | 23:50 | |
| *** BjoernT has quit IRC | 23:54 | |
| *** partlycloudy has joined #openstack-ansible | 23:55 | |
| *** openstackgerrit has quit IRC | 23:56 | |
| *** BjoernT has joined #openstack-ansible | 23:57 | |
| *** timburke has quit IRC | 23:58 | |
| *** antonym has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!