| *** cmart has quit IRC | 00:14 | |
| *** foutatoro has quit IRC | 00:16 | |
| *** sbra has joined #openstack-ansible | 00:22 | |
| *** openstacking_123 has joined #openstack-ansible | 00:35 | |
| *** openstacking_123 has quit IRC | 00:40 | |
| *** rmcall has quit IRC | 00:46 | |
| *** rmcall has joined #openstack-ansible | 00:46 | |
| *** sbra has quit IRC | 00:46 | |
| *** yifei1 has joined #openstack-ansible | 00:48 | |
| *** threestrands has quit IRC | 01:03 | |
| *** pgreg has joined #openstack-ansible | 01:15 | |
| *** pgreg has quit IRC | 01:15 | |
| *** pgreg has joined #openstack-ansible | 01:15 | |
| *** pgreg has quit IRC | 01:28 | |
| *** askb has quit IRC | 01:41 | |
| *** cjloader_ has joined #openstack-ansible | 01:41 | |
| *** askb has joined #openstack-ansible | 01:43 | |
| *** cjloader_ has quit IRC | 01:46 | |
| *** askb has quit IRC | 01:48 | |
| *** askb has joined #openstack-ansible | 01:49 | |
| *** markvoelker has joined #openstack-ansible | 01:50 | |
| *** TxGirlGeek has quit IRC | 01:53 | |
| *** yifei1 has quit IRC | 02:13 | |
| *** rmcall has quit IRC | 02:20 | |
| *** markvoelker has quit IRC | 02:36 | |
| *** cjloader_ has joined #openstack-ansible | 02:41 | |
| *** woodard has joined #openstack-ansible | 02:42 | |
| *** cjloader_ has quit IRC | 02:46 | |
| *** threestrands has joined #openstack-ansible | 02:47 | |
| *** masber has quit IRC | 03:02 | |
| *** masber has joined #openstack-ansible | 03:05 | |
| *** rmcall has joined #openstack-ansible | 03:10 | |
| *** cjloader_ has joined #openstack-ansible | 03:41 | |
| *** sbra has joined #openstack-ansible | 03:42 | |
| *** cjloader_ has quit IRC | 03:45 | |
| *** germs has quit IRC | 03:48 | |
| *** sbra has quit IRC | 03:54 | |
| *** yifei1 has joined #openstack-ansible | 03:59 | |
| *** gkadam has joined #openstack-ansible | 04:03 | |
| *** germs has joined #openstack-ansible | 04:09 | |
| *** esberglu has quit IRC | 04:10 | |
| cloudnull | evenings | 04:12 |
|---|---|---|
| *** yifei1 has quit IRC | 04:17 | |
| *** germs1 has quit IRC | 04:18 | |
| *** bhujay has joined #openstack-ansible | 04:33 | |
| *** prasen has joined #openstack-ansible | 04:41 | |
| *** chhavi__ has joined #openstack-ansible | 04:46 | |
| *** rmcall has quit IRC | 04:54 | |
| *** poopcat has quit IRC | 05:11 | |
| *** gkadam has quit IRC | 05:16 | |
| *** germs has quit IRC | 05:23 | |
| *** nshetty has joined #openstack-ansible | 05:35 | |
| *** markvoelker has joined #openstack-ansible | 05:47 | |
| *** pradiprwt has joined #openstack-ansible | 05:52 | |
| *** jbadiapa has quit IRC | 06:06 | |
| *** hybridpollo has quit IRC | 06:14 | |
| *** armaan has quit IRC | 06:17 | |
| *** taseer1 has joined #openstack-ansible | 06:35 | |
| *** Taseer has quit IRC | 06:35 | |
| *** taseer2 has joined #openstack-ansible | 06:35 | |
| *** drifterza has joined #openstack-ansible | 06:36 | |
| *** taseer1 has quit IRC | 06:40 | |
| *** markvoelker has quit IRC | 06:40 | |
| *** markvoelker has joined #openstack-ansible | 06:42 | |
| jrosser | cloudnull: is it worth me spending time looking if the 6aec task minimum time can be reduced? | 06:56 |
| jrosser | as it seems to apply to everything modest gains would reflect directly in the deploy time seemingly | 06:56 |
| *** threestrands has quit IRC | 07:00 | |
| *** sbra has joined #openstack-ansible | 07:00 | |
| *** threestrands has joined #openstack-ansible | 07:01 | |
| *** arbrandes has joined #openstack-ansible | 07:01 | |
| *** threestrands has quit IRC | 07:02 | |
| *** threestrands has joined #openstack-ansible | 07:03 | |
| *** threestrands has quit IRC | 07:03 | |
| *** threestrands has joined #openstack-ansible | 07:03 | |
| *** threestrands has quit IRC | 07:04 | |
| *** arbrandes1 has quit IRC | 07:04 | |
| *** threestrands has joined #openstack-ansible | 07:04 | |
| *** armaan has joined #openstack-ansible | 07:13 | |
| *** pcaruana has joined #openstack-ansible | 07:21 | |
| jafeha__ | good morning | 07:23 |
| *** jafeha__ is now known as jafeha | 07:23 | |
| *** nshetty is now known as nshetty|lunch | 07:26 | |
| *** afred312 has quit IRC | 07:27 | |
| *** afred312 has joined #openstack-ansible | 07:30 | |
| *** armaan has quit IRC | 07:41 | |
| *** armaan has joined #openstack-ansible | 07:41 | |
| *** armaan has quit IRC | 07:46 | |
| *** armaan has joined #openstack-ansible | 07:47 | |
| *** SmearedBeard has joined #openstack-ansible | 07:48 | |
| *** epalper has joined #openstack-ansible | 07:52 | |
| *** SmearedBeard has quit IRC | 07:54 | |
| *** drifterza has quit IRC | 07:56 | |
| *** flaviosr has joined #openstack-ansible | 07:58 | |
| *** taseer2 is now known as Taseer | 08:02 | |
| *** mbuil has joined #openstack-ansible | 08:04 | |
| *** evin has joined #openstack-ansible | 08:04 | |
| *** sbra has quit IRC | 08:06 | |
| *** threestrands has quit IRC | 08:08 | |
| *** SmearedBeard has joined #openstack-ansible | 08:09 | |
| *** SmearedBeard has quit IRC | 08:13 | |
| *** gkadam has joined #openstack-ansible | 08:18 | |
| *** rgogunskiy has joined #openstack-ansible | 08:19 | |
| *** nshetty|lunch is now known as nshetty | 08:19 | |
| *** shardy has joined #openstack-ansible | 08:22 | |
| *** nshetty is now known as nshetty|mtg | 08:24 | |
| *** taseer1 has joined #openstack-ansible | 08:33 | |
| *** Taseer has quit IRC | 08:33 | |
| *** taseer2 has joined #openstack-ansible | 08:33 | |
| pradiprwt | Hi Everyone, I am getting "PermissionError: error unprotecting snapshot" error while deleting glance images, after changing osd permission to "osd 'allow *"" it is working fine ... Anyone can tell me what rule I am missing. It is not able to delete on this rule which is provided by official docs " caps: [osd] allow class-read object_prefix rdb_children, allow rwx pool=images" | 08:34 |
| *** exodusftw has quit IRC | 08:35 | |
| *** taseer1 has quit IRC | 08:38 | |
| *** jbadiapa has joined #openstack-ansible | 08:40 | |
| *** pcaruana has quit IRC | 08:44 | |
| openstackgerrit | Nguyen Hung Phuong proposed openstack/openstack-ansible master: Remove empty file https://review.openstack.org/532767 | 08:44 |
| *** armaan has quit IRC | 08:47 | |
| *** armaan has joined #openstack-ansible | 08:48 | |
| *** taseer2 is now known as Taseer | 08:51 | |
| *** pbandark has joined #openstack-ansible | 09:02 | |
| *** cjloader_ has joined #openstack-ansible | 09:03 | |
| *** cjloader_ has quit IRC | 09:08 | |
| *** flaper87 has quit IRC | 09:17 | |
| *** flaper87 has joined #openstack-ansible | 09:21 | |
| gameon | Morning everyone - I'm trying to use Ceph as my backend storage and can upload and create volumes through Horizon no problem | 09:23 |
| *** flaper87 has joined #openstack-ansible | 09:23 | |
| gameon | However, when I try and create an instance with a disk using Ceph, the volume creation fails. Has anyone seen this befoer? | 09:23 |
| *** armaan has quit IRC | 09:27 | |
| *** armaan has joined #openstack-ansible | 09:27 | |
| *** armaan has quit IRC | 09:36 | |
| *** SmearedBeard has joined #openstack-ansible | 09:41 | |
| *** prasen has quit IRC | 09:43 | |
| *** prasen has joined #openstack-ansible | 09:44 | |
| *** prasen has quit IRC | 09:45 | |
| *** prasen has joined #openstack-ansible | 09:45 | |
| *** SmearedBeard has quit IRC | 09:47 | |
| *** nshetty|mtg has quit IRC | 09:51 | |
| *** nshetty has joined #openstack-ansible | 09:51 | |
| *** sxc731 has joined #openstack-ansible | 10:00 | |
| *** armaan has joined #openstack-ansible | 10:03 | |
| *** pcaruana has joined #openstack-ansible | 10:04 | |
| *** indistylo has joined #openstack-ansible | 10:04 | |
| *** SmearedBeard has joined #openstack-ansible | 10:07 | |
| *** markvoelker has quit IRC | 10:13 | |
| *** Smeared_Beard has joined #openstack-ansible | 10:15 | |
| *** SmearedBeard has quit IRC | 10:18 | |
| *** Taseer has quit IRC | 10:22 | |
| *** taseer2 has joined #openstack-ansible | 10:22 | |
| *** taseer2 is now known as Taseer | 10:32 | |
| *** sxc731 has quit IRC | 10:56 | |
| openstackgerrit | Jakob Hasselmann proposed openstack/openstack-ansible-os_neutron master: Adds Neutron OVS DVR Documentation https://review.openstack.org/532798 | 11:01 |
| *** sxc731 has joined #openstack-ansible | 11:11 | |
| ashak | mgariepy: so I know what's up with haproxy logging on CentOS (actually logging in general on CentOS). I know ways to fix it, but i'm a little conflicted on what would be the best approach. | 11:14 |
| *** rgogunskiy has quit IRC | 11:17 | |
| *** shardy has quit IRC | 11:22 | |
| *** shardy has joined #openstack-ansible | 11:24 | |
| *** prasen has quit IRC | 11:30 | |
| *** prasen has joined #openstack-ansible | 11:31 | |
| *** stuartgr has joined #openstack-ansible | 11:35 | |
| *** Jasem[m] has joined #openstack-ansible | 11:42 | |
| *** Jasem[m] has left #openstack-ansible | 11:43 | |
| *** bhujay has quit IRC | 11:45 | |
| *** jrosser has quit IRC | 11:47 | |
| *** jwitko_ has quit IRC | 11:53 | |
| *** prasen has quit IRC | 12:00 | |
| *** pcaruana has quit IRC | 12:02 | |
| *** pcaruana|afk| has joined #openstack-ansible | 12:02 | |
| *** markvoelker has joined #openstack-ansible | 12:14 | |
| *** jrosser has joined #openstack-ansible | 12:19 | |
| *** smatzek has joined #openstack-ansible | 12:21 | |
| *** portante has quit IRC | 12:24 | |
| *** redondo-mk has quit IRC | 12:24 | |
| *** redondo-mk has joined #openstack-ansible | 12:24 | |
| *** sxc731 has quit IRC | 12:25 | |
| *** mbuil has quit IRC | 12:26 | |
| *** portante has joined #openstack-ansible | 12:27 | |
| *** jose-phillips has quit IRC | 12:27 | |
| *** jose-phi_ has joined #openstack-ansible | 12:27 | |
| *** mbuil has joined #openstack-ansible | 12:28 | |
| *** markvoelker has quit IRC | 12:48 | |
| *** pcaruana|afk| has quit IRC | 12:49 | |
| *** pcaruana|afk| has joined #openstack-ansible | 13:02 | |
| hwoarang | odyssey4me: you around? do you know if this line https://github.com/openstack/openstack-ansible/blob/master/scripts/gate-check-commit.sh#L164 performs any fact caching? | 13:15 |
| hwoarang | i guess not right? | 13:15 |
| hwoarang | and do you know why we are sending stderr to the log but on line 176 we send stdout? | 13:16 |
| odyssey4me | hwoarang yep, the point of that is to pre-populate the fact cache | 13:16 |
| odyssey4me | heh, good question - I think that's an oversight | 13:16 |
| hwoarang | but where is that cache? according to the docs, facts are being wiped between playbook runs | 13:16 |
| odyssey4me | the redirect was likely to cut down on all the facts coming into the log | 13:16 |
| hwoarang | unless you have a static place to put them like redis, json file tc | 13:17 |
| hwoarang | *etc | 13:17 |
| hwoarang | and i dont see us having any fact_caching thing anywhere | 13:17 |
| hwoarang | so im confused | 13:17 |
| odyssey4me | we have a static one setup: https://github.com/openstack/openstack-ansible/blob/master/scripts/openstack-ansible.rc#L36 | 13:17 |
| hwoarang | ah | 13:17 |
| hwoarang | ok makes sense now | 13:18 |
| hwoarang | thank you | 13:18 |
| odyssey4me | so we're pre-populating the fact cache there so that the playbook runs only have to *update* the cache, making execution a bit faster | 13:18 |
| hwoarang | yep | 13:18 |
| *** chas has joined #openstack-ansible | 13:21 | |
| *** indistylo has quit IRC | 13:23 | |
| mhayden | good morning | 13:30 |
| mgariepy | morning everyone | 13:33 |
| mgariepy | ashak, do you know how to submit patches ? | 13:34 |
| hwoarang | o/ | 13:36 |
| jrosser | what i doing wrong? i cant see the build logs on here https://review.openstack.org/#/c/525113/ "file not found" | 13:37 |
| ashak | mgariepy: not unless it's via github pull requests... but I haven't fixed anything yet,like I said, i'm conflicted on how I should fix it. Was hoping someone could help me work out an appropriate solution. | 13:39 |
| *** markvoelker has joined #openstack-ansible | 13:44 | |
| mgariepy | ok. | 13:46 |
| mgariepy | do you have the list of server that don;t log correclty ? | 13:46 |
| mgariepy | or it's just haproxy for now ? | 13:47 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_neutron master: Add SELinux policies for bare metal agents https://review.openstack.org/532646 | 13:51 |
| *** exodusftw has joined #openstack-ansible | 13:52 | |
| *** masber has quit IRC | 13:52 | |
| mgariepy | mhayden, commented on 532646 | 13:55 |
| mhayden | yay | 13:55 |
| mhayden | oh, good catch sir | 13:56 |
| mgariepy | i've been biten by that once or twice :) | 13:56 |
| ashak | mgariepy: i've only looked at a controller node, that's where I wanted to view haproxy logs. But it's basically all logging from rsyslog is broken. CentOS's /var/log has differing permissions than Ubuntu. The rsyslog config seems to be a stock Ubuntu one, so it drops permissions to the syslog user. None of the CentOS system is prepared for that, the CentOS default config is quite different. | 13:57 |
| mgariepy | mhayden, is logging working on you centos cluster ? | 13:58 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_neutron master: Add SELinux policies for bare metal agents https://review.openstack.org/532646 | 14:01 |
| ashak | The reason I am conflicted is because it looks like the rsyslog config has simply been taken from a stock ubuntu install. So it could be made to work by fixing up a bunch of permissions around the time that the rsyslog config is changed and rsyslog restarted. But that seems a bit wrong. Mostly because it doesn't look like there's been any effort yet to make rsyslog things consider anything other | 14:01 |
| mhayden | yeah, neutron agent was really grumpy about its limitations | 14:01 |
| ashak | than Ubuntu. CentOS for example would normally use /var/log/messages instead of /var/log/syslog. | 14:01 |
| mhayden | ashak: that's entirely possible since Ubuntu was the first supported OS :) centos support still needs a little love | 14:02 |
| ashak | indeed | 14:02 |
| mhayden | in other news, my CentOS 7 cloud is running OSA master | 14:02 |
| mhayden | and things seem to work | 14:02 |
| *** shardy has quit IRC | 14:02 | |
| *** jwitko_ has joined #openstack-ansible | 14:02 | |
| mgariepy | wow you love trouble :) | 14:03 |
| mhayden | trouble is my middle name | 14:03 |
| ashak | mhayden: ooh, how fresh? I still have mariadb pacakge problems. | 14:03 |
| mhayden | oddly enough, most of the time spent on debugging the deployment was due to a mis-trunked VLAN and default security groups | 14:03 |
| mhayden | i swear, default security groups make me want to throw a chair -- i always forget about them :/ | 14:04 |
| mhayden | ashak: oh, on master? | 14:04 |
| ashak | As of a couple of days ago yeah. I could try to manually fix them I guess. | 14:04 |
| mhayden | i ran into the health check shenanigans with galera | 14:05 |
| mhayden | haproxy was hitting port 9200 on galera from an IP that wasn't in the allowed list | 14:05 |
| mhayden | i wish we could come up with a secure, but more user friendly way to figure that out | 14:05 |
| *** cshen has quit IRC | 14:09 | |
| *** cshen has joined #openstack-ansible | 14:12 | |
| *** dave-mccowan has joined #openstack-ansible | 14:16 | |
| *** markvoelker has quit IRC | 14:18 | |
| *** evin has quit IRC | 14:24 | |
| openstackgerrit | Merged openstack/openstack-ansible-specs master: blueprint spec for opendaylight bgpvpn support https://review.openstack.org/523171 | 14:26 |
| *** esberglu has joined #openstack-ansible | 14:28 | |
| *** ricardoas has joined #openstack-ansible | 14:29 | |
| *** woodard has quit IRC | 14:31 | |
| *** woodard has joined #openstack-ansible | 14:31 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_nova master: Adjust deprecated nova console variables https://review.openstack.org/532836 | 14:37 |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible master: scripts: openstack-ansible.rc: Check if additional inventory file exists https://review.openstack.org/532837 | 14:40 |
| hwoarang | odyssey4me: evrardjp what do you think^ | 14:40 |
| hwoarang | i dont quite understand the inventory changes that went in due to ansible-2.4 to be honet | 14:41 |
| hwoarang | *honest | 14:41 |
| hwoarang | for whatever reason /etc/openstack_deploy does not have group_vars or host_vars in OPNFV/XCI and that breaks because no inventory.ini is created | 14:41 |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-os_nova master: Adjust deprecated nova placement variables https://review.openstack.org/532838 | 14:43 |
| gameon | Hi guys - is there a way I can install OSA with the linuxbridge agent for neutron? | 14:59 |
| gameon | I cant see that there is a way to do this having picked through the neutron repos | 14:59 |
| *** markvoelker has joined #openstack-ansible | 15:02 | |
| *** markvoelker has quit IRC | 15:02 | |
| *** vnogin has joined #openstack-ansible | 15:05 | |
| odyssey4me | gameon by default OSA uses the linuxbridge agent | 15:06 |
| gameon | Shit - OK, I think I've misconfigured something :( | 15:06 |
| gameon | Thanks odyssey4me: Can I re-run the playbook against my containers or will that cause issues, | 15:06 |
| odyssey4me | gameon you can re-run at your leisure | 15:07 |
| gameon | Thanks again odyssey4me, appreciated | 15:07 |
| *** shardy has joined #openstack-ansible | 15:08 | |
| *** sxc731 has joined #openstack-ansible | 15:11 | |
| *** nshetty has quit IRC | 15:12 | |
| gameon | As I inadvertently deployed using ml2.ovs - do I need to scrub anything from my containers or compute nodes or would a re-run of the neutron playbook suffice? | 15:13 |
| *** hamzy has quit IRC | 15:17 | |
| cloudnull | g'morning | 15:18 |
| cloudnull | jrosser: ++ I think its worth looking into | 15:18 |
| cloudnull | if there's something we can do to improve the skip task speed I think it would make things A LOT faster. | 15:18 |
| *** Smeared_Beard has quit IRC | 15:20 | |
| jrosser | cloudnull: we've been doing some poking, and the short story is even the most trivial task takes 6 seconds | 15:22 |
| cloudnull | no matter if it skips right ? | 15:22 |
| jrosser | and the same playbook run either with a local connection or outside the world of the openstack-ansible wrapper and plugins is quick | 15:23 |
| cloudnull | six seconds for the round trip (skip|change|success) | 15:23 |
| jrosser | we just did 'date' with shell: | 15:23 |
| cloudnull | ok. | 15:23 |
| *** markvoelker has joined #openstack-ansible | 15:24 | |
| cloudnull | could that time be related to the addition of the ansible inventory/group_vars/host_vars? | 15:24 |
| hwoarang | cloudnull: Hello. Do you have a few minutes to talk about "Add basic plugin support for nspawn type containers" commit in the plugins directory? | 15:24 |
| cloudnull | sure. | 15:24 |
| jrosser | i was trying to look back through gate logs for the openstack-ansible-plugins to see what the execution times there were | 15:24 |
| hwoarang | i noticed a strange change in our setups between november and today and i think that commit may have broken something | 15:24 |
| jrosser | but there dont seem to be any useful logs kept there | 15:24 |
| hwoarang | so before when you were trying to use ansible to gather facts from a container the IP of the real host was used | 15:25 |
| hwoarang | but now the hostname is being used | 15:25 |
| hwoarang | so if /etc/hosts is not setup or if your dns is blahblah then you can't manage a container anymore | 15:25 |
| *** rmcall has joined #openstack-ansible | 15:26 | |
| hwoarang | cloudnull: look at this http://paste.opensuse.org/57560131 . before the IP of controller00 was used, but now the 'controller00' hostname is used | 15:26 |
| hwoarang | and i suspect the connection plugin is responsible for it | 15:27 |
| *** SmearedBeard has joined #openstack-ansible | 15:27 | |
| hwoarang | i am still bisecting :/ | 15:27 |
| jrosser | cloudnull: skipped tasks are very quick 0.00 something, anything not skipped seems to have a ~6sec floor on execution time | 15:27 |
| *** openstacking_123 has joined #openstack-ansible | 15:28 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible master: Add user agent for haproxy health checks https://review.openstack.org/532863 | 15:29 |
| cloudnull | hwoarang: hum . well that's unexpected/ | 15:29 |
| *** rromans has quit IRC | 15:29 | |
| * cloudnull looking | 15:29 | |
| *** afred312 has quit IRC | 15:31 | |
| cloudnull | jrosser: I wonder if there's something in our plugins that hangs isn't efficiently closing a process. | 15:31 |
| *** afred312 has joined #openstack-ansible | 15:32 | |
| cloudnull | **hangs or isn't | 15:32 |
| cloudnull | hwoarang: still walking back through that patch | 15:32 |
| jrosser | it certainly feels related to the plugins - i've just failed to run the openstack-ansible-plugins tests locally for the last couple of hours :( | 15:32 |
| *** phalmos has joined #openstack-ansible | 15:32 | |
| jrosser | it really really doesnt like being behind an http proxy | 15:33 |
| hwoarang | i am comparing the inventories from November and January to find out if they differ | 15:33 |
| cloudnull | :( | 15:33 |
| jrosser | cloudnull: i'd hoped to be able to look back through logs on review.openstack.org to see the results, but it seems they arent kept | 15:34 |
| *** gaudenz has joined #openstack-ansible | 15:34 | |
| cloudnull | yea they're not kept for all that long | 15:34 |
| cloudnull | infra is constantly running cleanup | 15:35 |
| *** shardy has quit IRC | 15:35 | |
| hwoarang | yeah might not be that patch :/ | 15:36 |
| cloudnull | https://review.openstack.org/#/c/490689/12/connection/ssh.py@183 | 15:37 |
| cloudnull | it looks like the lxc command / hostname pass through is the same | 15:38 |
| cloudnull | however maybe something else | 15:38 |
| *** chas has quit IRC | 15:39 | |
| *** cjloader_ has joined #openstack-ansible | 15:41 | |
| *** vnogin has quit IRC | 15:42 | |
| *** vnogin has joined #openstack-ansible | 15:45 | |
| *** hamzy has joined #openstack-ansible | 15:48 | |
| *** Smeared_Beard has joined #openstack-ansible | 15:50 | |
| *** esberglu has quit IRC | 15:52 | |
| *** SmearedBeard has quit IRC | 15:53 | |
| cloudnull | hwoarang: hum. did you find anything on the inventory front that might be causing this ? | 15:54 |
| hwoarang | cloudnull: i think it might be this line self.host = self._play_context.remote_addr = self.physical_host | 15:55 |
| hwoarang | self.host maps to physical_host and not physical_host_addrs as it used to | 15:56 |
| hwoarang | no clue why this is working in the gates but not for us | 15:56 |
| *** woodard has quit IRC | 15:59 | |
| *** amine has joined #openstack-ansible | 15:59 | |
| cloudnull | is that a change in the core ansible behavior ? | 15:59 |
| amine | Hi guys | 15:59 |
| hwoarang | not sure i am still looking :/ | 16:00 |
| cloudnull | https://review.openstack.org/#/c/490689/12/connection/ssh.py@85 | 16:00 |
| *** fuatu has joined #openstack-ansible | 16:00 | |
| logan- | ive noticed some weirdness in the physical host lookup stuff since moving to 2.4 in some projects | 16:00 |
| cloudnull | orly | 16:00 |
| hwoarang | haha | 16:00 |
| hwoarang | sounds similar :) | 16:00 |
| logan- | in some cases it fails to lookup ansible_host and uses inventory_hostname | 16:01 |
| hwoarang | logan-: welcome to the club :) i was just talking to cloudnull about it :D | 16:01 |
| logan- | haha | 16:01 |
| amine | i am trying sahara on OSA ocata release and facing issu launching clusters. the cluster always stuck at ssh_accessibility step. any help on this matter would be appreciated thank you | 16:01 |
| logan- | hwoarang: i havent looked at it since before the holidays but i believe i got as far as confirming the strategy plugin is still populating the physical host addr list, but maybe something changed in the magic vars handling and it is not being passed to the connection plugin as it used to be | 16:03 |
| *** armaan has quit IRC | 16:04 | |
| *** vnogin has quit IRC | 16:04 | |
| hwoarang | that thing is black magic to me | 16:05 |
| *** sbra has joined #openstack-ansible | 16:05 | |
| *** jbadiapa has quit IRC | 16:08 | |
| hwoarang | logan-: yeah so to me it seems that physical_host_addr contains the hostname and not the ip address | 16:11 |
| hwoarang | at least the ssh connection plugin doesn't see an ip address there | 16:11 |
| logan- | :/ | 16:11 |
| logan- | https://github.com/openstack/openstack-ansible-plugins/blob/c665f9f2a2dea2ac532402d41b68b6132fd774bc/strategy/linear.py#L121-L132 is where the list comes from | 16:13 |
| logan- | so that makes sense | 16:13 |
| logan- | it is failing to get ansible_host on 129 i guess | 16:13 |
| hwoarang | actually physical_host_addrs is empty | 16:13 |
| logan- | ah | 16:13 |
| hwoarang | so it defaults to empty dict {} and because it's emtpy the plugins' logic is to use the self.physical_host which is the hostname | 16:13 |
| hwoarang | pff | 16:14 |
| logan- | yup | 16:14 |
| logan- | if we can find a way to access any host's 'ansible_host' inside the connection plugin this whole thing could be much cleaner | 16:15 |
| hwoarang | we can simply do hostvars.get('ansible_host') i suppose but i presume there is a reason it's written like that | 16:17 |
| logan- | if the task it not delegated then that should work fine, but i think that breaks with delegation | 16:17 |
| *** armaan has joined #openstack-ansible | 16:17 | |
| hwoarang | yep | 16:17 |
| hwoarang | so overall you are right. the var is set in the strategy plugin but the connection plugin doesn't see it | 16:21 |
| *** armaan has quit IRC | 16:21 | |
| hwoarang | so i blame ansible ;p | 16:21 |
| jrosser | cloudnull: well i'm appreciating the irony of this. https://github.com/openstack/openstack-ansible/blob/a6575b4dd0b4a975dffacb5e5f9d5c31dac1f0e6/playbooks/security-hardening.yml#L27 | 16:21 |
| jrosser | comment that out and it is waaaay faster | 16:22 |
| jrosser | we just picked the security playbook as an example but it quite likley is the same across everything | 16:22 |
| *** phalmos has quit IRC | 16:25 | |
| *** esberglu has joined #openstack-ansible | 16:26 | |
| *** fuatu has quit IRC | 16:30 | |
| *** shardy has joined #openstack-ansible | 16:32 | |
| amine | any clue on sahara issue ? in fact can access the instance through FIP but can not do it when using private addresses .... | 16:38 |
| Bico_Fino | Hello. | 16:38 |
| Bico_Fino | https://bugs.launchpad.net/openstack-ansible/+bug/1669125 , this supposed to be solved on 16.0.5, correct? | 16:39 |
| openstack | Launchpad bug 1669125 in openstack-ansible "rabbitmq-server fails so start with ubuntu16.04" [Undecided,Fix released] - Assigned to Alexandra Settle (alexandra-settle) | 16:39 |
| *** phalmos has joined #openstack-ansible | 16:45 | |
| cloudnull | jrosser: oh ! | 16:50 |
| odyssey4me | heh, jrosser that is funny :) | 16:50 |
| cloudnull | so if we drop the environment addition it should be faster? | 16:50 |
| jrosser | well, it seems to be wierd | 16:52 |
| cloudnull | jrosser: mind testing changing environment: "{{ deployment_environment_variables | default({}) }}" to environment: "{{ deployment_environment_variables | default(omit) }}" and seeing if it is still fast? | 16:52 |
| jrosser | just fiddling with it now | 16:52 |
| *** vnogin has joined #openstack-ansible | 16:54 | |
| *** armaan has joined #openstack-ansible | 16:55 | |
| *** chyka has joined #openstack-ansible | 16:55 | |
| *** woodard has joined #openstack-ansible | 16:56 | |
| *** pcaruana|afk| has quit IRC | 16:57 | |
| *** vnogin has quit IRC | 16:59 | |
| *** RandomTech has joined #openstack-ansible | 17:05 | |
| mgariepy | anyone here know if it's possible to migrate shutdown instances to a specific host ? | 17:06 |
| *** phalmos has quit IRC | 17:06 | |
| cloudnull | mgariepy: you should be able to do that with the nova cli | 17:09 |
| cloudnull | doing an offline migration | 17:10 |
| mgariepy | when doing live i can specify the destination | 17:10 |
| mgariepy | but didn't saw same option for offline one. | 17:10 |
| *** woodard has quit IRC | 17:11 | |
| cloudnull | hum. | 17:12 |
| *** woodard has joined #openstack-ansible | 17:12 | |
| cloudnull | IDK about that one | 17:12 |
| cloudnull | i'd have to look more :\ | 17:12 |
| RandomTech | hey anyone here familiar with uploading images? | 17:12 |
| RandomTech | weve run into a strange error | 17:13 |
| *** germs has joined #openstack-ansible | 17:14 | |
| mgariepy | i don't have a lot of vms in that stage so i'll just migrate them a few time until they get a better host.. | 17:15 |
| *** electrofelix has left #openstack-ansible | 17:16 | |
| *** epalper has quit IRC | 17:18 | |
| *** phalmos has joined #openstack-ansible | 17:19 | |
| RandomTech | cloudnull: are you familar with how glance works? | 17:22 |
| *** chhavi__ has quit IRC | 17:26 | |
| *** electrofelix has joined #openstack-ansible | 17:26 | |
| openstackgerrit | Markos Chandras (hwoarang) proposed openstack/openstack-ansible-plugins master: connection: ssh: Use individuals' host variables for physical host IP https://review.openstack.org/532917 | 17:28 |
| hwoarang | logan-: cloudnull^ that should fix it but I need to investigate what exactly changes in ansible 2.4 | 17:28 |
| hwoarang | i am not happy with the commit message there is something fishy going on | 17:28 |
| *** pedja has quit IRC | 17:31 | |
| *** rromans has joined #openstack-ansible | 17:31 | |
| jrosser | cloudnull: ANISBLE_DEBUG=1 results in 30k lines per task when things are slow, and ~400 when fast | 17:31 |
| RandomTech | hwoarang: do you know where glance stores its error logs? | 17:32 |
| hwoarang | not off the top of my head | 17:33 |
| RandomTech | we cant find what is causing this error is anywhere, thanks though | 17:34 |
| logan- | nice fix hwoarang | 17:34 |
| *** gkadam has quit IRC | 17:37 | |
| logan- | hwoarang do you know if the plugins tests fail without your patch currently? looks like nothing has merged there for a while | 17:43 |
| logan- | i think one of the existing test cases may cover that bug | 17:43 |
| jrosser | i have just run the plugins tests locally today (for other things) and they pass | 17:44 |
| logan- | thanks | 17:44 |
| jrosser | i do have an interested problem though, with deployment_environment_variables set i have 30,000 lines of this in my ANSIBLE_DEBUG output http://paste.openstack.org/show/642930/ | 17:46 |
| -openstackstatus- NOTICE: Due to an unexpected issue with zuulv3.o.o, we were not able to preserve running jobs for a restart. As a result, you'll need to recheck your previous patchsets | 17:46 | |
| jrosser | it just goes on and on and on like that | 17:46 |
| *** indistylo has joined #openstack-ansible | 17:47 | |
| jrosser | with environment: in the playbook unset everything is quick and the whole thing is over and done with in ~400 debug lines | 17:47 |
| jrosser | cloudnull: odyssey4me any opinion on that would be welcome | 17:47 |
| *** germs has quit IRC | 17:55 | |
| *** indistylo has quit IRC | 17:58 | |
| *** indistylo has joined #openstack-ansible | 17:59 | |
| *** mbuil has quit IRC | 18:00 | |
| *** jbadiapa has joined #openstack-ansible | 18:04 | |
| *** kysse has quit IRC | 18:06 | |
| *** indistylo has quit IRC | 18:16 | |
| *** rmcall has quit IRC | 18:21 | |
| cloudnull | RandomTech: yes. I'm familiar with glance | 18:21 |
| RandomTech | were having a really stange error when trying to create images | 18:22 |
| RandomTech | whenever we fill out the form and hit create image it says [object Object] and "unable to create image" | 18:23 |
| SamYaple | cloudnull: you?! familiar with glance?! unheard of! | 18:23 |
| cloudnull | haha | 18:23 |
| SamYaple | RandomTech: so i did ask for the actually error (stack trace) in the #openstack channel to help you | 18:24 |
| RandomTech | i finally just found one and pinged cloudnull before you asked for anything so i was filling him in | 18:24 |
| RandomTech | i just found these errors SamYaple: http://paste.openstack.org/show/642962/ | 18:25 |
| RandomTech | sorry for the waity | 18:25 |
| cloudnull | jrosser: did setting "omit" instead of "{}" make any difference ? | 18:26 |
| cloudnull | RandomTech: does uploading the image via the CLI work ? | 18:27 |
| RandomTech | not sure how to do that | 18:27 |
| jrosser | cloudnull: omit made no difference | 18:27 |
| jrosser | it would be very interesting to know if this can be reproduced | 18:28 |
| jrosser | just set any arbitrary env var in deployment_environment_variables | 18:28 |
| jrosser | i cant trigger it to do this in the openstack-ansible-plugins run_tests.sh for example | 18:29 |
| SamYaple | RandomTech: it does say 201, which is image created. that log doesnt appear to contain errors at passing glance | 18:29 |
| jrosser | but it does happen on a deploy host with a properly set up osa | 18:29 |
| RandomTech | when refresh an image just stays queued seemingly indefinately SamYaple | 18:30 |
| SamYaple | RandomTech: how large is said image? | 18:30 |
| cloudnull | jrosser: http://paste.openstack.org/show/642969/ | 18:31 |
| cloudnull | yup | 18:31 |
| RandomTech | its just a blank ubuntu 16 server we got from ubuntus website, so about 826 MB | 18:31 |
| cloudnull | I ran the utility playbook with and without the env var set. | 18:31 |
| cloudnull | ~30 sec w/ it set ~26 without | 18:32 |
| SamYaple | RandomTech: queued is basically a "im waiting for data" state. i would defientely try to upload it from the CLI first, since through horizon is a second stage of buffering. with debug glance logs you should more info (could be transfering to the backend *very* slowly) | 18:33 |
| cloudnull | ^ | 18:33 |
| cloudnull | jrosser: multiple runs shows the same results. | 18:33 |
| *** germs has joined #openstack-ansible | 18:33 | |
| jrosser | im seeing 6s per non skipped task | 18:33 |
| RandomTech | how would we go about trying that? | 18:34 |
| jrosser | not quite sure what im looking at in your paste in that regard? | 18:34 |
| *** phalmos_ has joined #openstack-ansible | 18:34 | |
| *** germs1 has joined #openstack-ansible | 18:35 | |
| cloudnull | Just confirming that when setting an arbitrary key=value in deployment_environment_variables the run is slower. | 18:35 |
| cloudnull | I have a multi-node cloud I was testing that against. | 18:35 |
| cloudnull | I'm not seeing the same 6s slowdown | 18:35 |
| cloudnull | but i am seeing a ~4 second difference between the runs consistently | 18:36 |
| jrosser | the other thing i see is vastly more output in ANSIBLE_DEBUG | 18:36 |
| jrosser | by a factor of about 75x | 18:36 |
| *** phalmos has quit IRC | 18:37 | |
| cloudnull | testing that now | 18:38 |
| *** germs has quit IRC | 18:38 | |
| *** phalmos_ has quit IRC | 18:39 | |
| cloudnull | I captured all of the debug output between the runs. I'm not seeing a massive size difference in the log file. | 18:41 |
| jrosser | ok thats interedting to know | 18:41 |
| cloudnull | http://paste.openstack.org/show/642981/ | 18:42 |
| cloudnull | thats the size of the files and the line count | 18:42 |
| jrosser | basically tomorrow we break out python debugger and try to get to the bottom of it | 18:42 |
| jrosser | thanks for having a poke in your env, helps having another view | 18:42 |
| cloudnull | which playbook are you running? | 18:43 |
| cloudnull | maybe I can try and retest 1:1 | 18:43 |
| jrosser | security | 18:43 |
| cloudnull | running that now | 18:44 |
| jrosser | i think we just slipped a trivial pre-task in there | 18:44 |
| jrosser | and that was enough, didnt make it run the actual bulk of the stuff | 18:44 |
| *** stuartgr has left #openstack-ansible | 18:45 | |
| *** openstacking_123 has quit IRC | 18:52 | |
| cloudnull | I let the entire play run, but the results are simial | 18:55 |
| cloudnull | time: 2m47.560s without the env var set, 3m10.685s with it set | 18:55 |
| cloudnull | not a big difference in captured logging | 18:55 |
| jrosser | thats really interesting :) | 18:56 |
| jrosser | wonder what we have messing ours up | 18:56 |
| cloudnull | My cloud isn't all that big and the vars are super stock. | 18:57 |
| cloudnull | I'd be happy to share all of teh inventory/config if it'd be helpful to compare | 18:57 |
| RandomTech | cloudnull: SamYaple when i used the CLI to create an image from volume (i didnt know where to put a file to use but i was able to create a volume) It is in a saving state instead of the queued state the other one is | 18:59 |
| RandomTech | now the cli volume one is active | 18:59 |
| jrosser | cloudnull: this is super mimimal, its just in a few kvm hosts as a test bed / CI environment | 19:01 |
| *** shardy has quit IRC | 19:01 | |
| jrosser | i think we dive in with pdb tomorrow, shouldnt be massively hard | 19:02 |
| *** poopcat has joined #openstack-ansible | 19:02 | |
| *** sxc731 has quit IRC | 19:06 | |
| *** poopcat1 has joined #openstack-ansible | 19:07 | |
| *** poopcat has quit IRC | 19:09 | |
| *** indistylo has joined #openstack-ansible | 19:15 | |
| *** gkadam has joined #openstack-ansible | 19:17 | |
| *** smatzek has quit IRC | 19:27 | |
| *** smatzek has joined #openstack-ansible | 19:28 | |
| *** smatzek has quit IRC | 19:33 | |
| *** vnogin has joined #openstack-ansible | 19:45 | |
| *** indistylo has quit IRC | 19:54 | |
| *** smatzek has joined #openstack-ansible | 19:57 | |
| openstackgerrit | Merged openstack/openstack-ansible-os_keystone master: Use a venv with previous ansible version for upgrades https://review.openstack.org/532470 | 19:58 |
| *** armaan has quit IRC | 20:02 | |
| *** armaan has joined #openstack-ansible | 20:03 | |
| *** phalmos has joined #openstack-ansible | 20:03 | |
| *** kysse has joined #openstack-ansible | 20:08 | |
| *** gkadam has quit IRC | 20:12 | |
| *** pedja has joined #openstack-ansible | 20:18 | |
| *** phalmos has quit IRC | 20:19 | |
| *** sxc731 has joined #openstack-ansible | 20:23 | |
| *** denta has quit IRC | 20:29 | |
| *** phalmos has joined #openstack-ansible | 20:56 | |
| *** kenperkins has joined #openstack-ansible | 20:59 | |
| *** openstacking_123 has joined #openstack-ansible | 20:59 | |
| *** RandomTech has quit IRC | 21:01 | |
| *** vnogin has quit IRC | 21:04 | |
| *** masber has joined #openstack-ansible | 21:10 | |
| *** exodusftw has quit IRC | 21:27 | |
| *** cmart has joined #openstack-ansible | 21:31 | |
| cmart | I <3 OpenStack-Ansible | 21:31 |
| *** exodusftw has joined #openstack-ansible | 21:33 | |
| mhayden | and i bet it loves you back | 21:34 |
| mhayden | haha | 21:34 |
| mhayden | how are things, cmart ? | 21:34 |
| cmart | good mhayden! just recovered from some serious cluster issues in production.. clocks had drifted on 2 of the control plane hosts and RabbitMQ was really unhappy, new instances were launching but didn't have network connectivity | 21:36 |
| mhayden | oof | 21:36 |
| mhayden | how bad was the drift? | 21:36 |
| cmart | 5-10 mins | 21:37 |
| cmart | NTP was busted for some reason | 21:37 |
| *** sbra has quit IRC | 21:38 | |
| cmart | anyhow, between this and patching for Spectre/Meltdown, I'm really happy with the way my OpenStack is architected, thanks to OSA. as basically "the one person supporting OpenStack and a bunch of other things" at my shop, I would never have the time to set up OpenStack as highly available services running in LXCs otherwise. | 21:39 |
| cmart | OSA basically lets me run what I would consider a sophisticated, highly available and fault-tolerant deployment, without a ton of effort | 21:40 |
| cmart | and the various services are so well-organized that when things break, troubleshooting is no more difficult than it absolutely needs to be. | 21:41 |
| *** sxc731 has quit IRC | 21:41 | |
| mhayden | i like it for the same reason | 21:42 |
| * mhayden is deploying it on centos 7 now | 21:42 | |
| *** smatzek has quit IRC | 21:42 | |
| ivve | oi guys, no possiblity to override neutron-lbaas.ini yet? it defaults "user_group = nogroup" should be haproxy | 21:43 |
| cmart | mhayden how's the CentOS support these days? | 21:53 |
| mhayden | quite good | 21:53 |
| mhayden | working through some selinux things in queens | 21:53 |
| *** woodard has quit IRC | 21:54 | |
| *** threestrands has joined #openstack-ansible | 21:58 | |
| *** threestrands has quit IRC | 21:58 | |
| *** threestrands has joined #openstack-ansible | 21:58 | |
| *** hamzy has quit IRC | 21:58 | |
| *** threestrands has quit IRC | 21:59 | |
| *** threestrands has joined #openstack-ansible | 21:59 | |
| *** threestrands has quit IRC | 22:00 | |
| *** threestrands has joined #openstack-ansible | 22:01 | |
| *** threestrands has quit IRC | 22:02 | |
| *** threestrands has joined #openstack-ansible | 22:02 | |
| *** dave-mccowan has quit IRC | 22:16 | |
| *** dave-mccowan has joined #openstack-ansible | 22:17 | |
| *** NostawRm has quit IRC | 22:23 | |
| *** NostawRm has joined #openstack-ansible | 22:25 | |
| *** openstacking_123 has quit IRC | 22:25 | |
| *** dave-mccowan has quit IRC | 22:25 | |
| *** openstacking_123 has joined #openstack-ansible | 22:25 | |
| *** esberglu has quit IRC | 22:36 | |
| cloudnull | cmart: very nice! | 22:42 |
| *** kenperkins has quit IRC | 22:45 | |
| *** openstacking_123 has quit IRC | 22:46 | |
| *** jbadiapa has quit IRC | 22:47 | |
| *** markvoelker has quit IRC | 22:49 | |
| *** markvoelker has joined #openstack-ansible | 22:49 | |
| *** Jeffrey4l has quit IRC | 22:53 | |
| *** markvoelker has quit IRC | 22:54 | |
| *** Jeffrey4l has joined #openstack-ansible | 22:56 | |
| *** markvoelker has joined #openstack-ansible | 22:56 | |
| *** woodard has joined #openstack-ansible | 22:58 | |
| *** markvoelker has quit IRC | 22:59 | |
| *** markvoelker has joined #openstack-ansible | 23:00 | |
| *** markvoelker has quit IRC | 23:05 | |
| *** jbadiapa has joined #openstack-ansible | 23:14 | |
| *** chhavi__ has joined #openstack-ansible | 23:22 | |
| *** chhavi__ has quit IRC | 23:26 | |
| *** phalmos has quit IRC | 23:26 | |
| *** jbadiapa has quit IRC | 23:32 | |
| *** smatzek has joined #openstack-ansible | 23:43 | |
| *** smatzek has quit IRC | 23:47 | |
| *** armaan has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!