Tuesday, 2018-12-11

*** hwoarang has quit IRC00:06
*** hwoarang has joined #openstack-ansible00:08
*** macza has quit IRC00:09
*** macza_ has joined #openstack-ansible00:09
*** weezS has quit IRC00:10
*** strattao has quit IRC00:24
*** cshen has joined #openstack-ansible00:35
*** cshen has quit IRC00:40
*** cshen has joined #openstack-ansible01:08
*** cshen has quit IRC01:12
*** ianychoi has quit IRC01:20
*** DanyC has joined #openstack-ansible01:34
*** hwoarang has quit IRC01:36
*** hwoarang has joined #openstack-ansible01:37
*** DanyC has quit IRC01:39
*** mma has joined #openstack-ansible01:52
*** mma has quit IRC01:57
*** vnogin has joined #openstack-ansible02:13
jamesdentonredkrieg finally had a chance to look at your config. I think if you set host_bind_override to enp4s0f0, get rid of the br-vlan-veth<->eth12 veth pair, and move enp4s0f0 out of br-vlan, you should be set. As it stands now, the veth pair and br-vlan are unnecessary L2 hops. The br-vlan bridge can stay, but won't really be necessary. The reason your previous config of [flat:br-vlan] didn't work, is because the system02:16
jamesdenton was trying to add the br-vlan bridge to the neutron brq-* bridge, which isn't allowed.02:16
*** vnogin has quit IRC02:17
redkriegso, I can't use the interface name because my compute nodes will have a number of different interface names02:18
jamesdentonin that case, you can do what you're doing now, or setup a series of host overides02:23
jamesdentonbut yes, that definitely makes things tricky to manage02:23
redkriegshould I have to manually change the config file though?  I would expect host_bind_override's value to be placed in that config file02:23
jamesdentonhost_bind_override should result in the label being mapped to the respective interface. Like "flat:eth12", "vlan:eth12", "physnet1:eth1", etc02:24
jamesdentonultimately, you need a consistent "label" (e.g. flat, vlan, physnet1) and can have inconsistent interfaces based on the machine02:25
jamesdentonwhen you create the network with neutron CLI, you specify the label. The agent on the host does the rest based on the mapping02:25
*** mma has joined #openstack-ansible02:25
jamesdentonsomething like this can be done: http://paste.openstack.org/show/736940/02:27
jamesdentonnotice the network_mappings value is different based on the host. same label, different interface02:28
redkriegI would expect br-vlan in this example to be eth12 based on the value of host_bind_override: http://paste.openstack.org/show/736942/02:29
redkriegI'm not sure why it wouldn't be02:29
redkriegsorry, http://paste.openstack.org/show/736942/02:30
*** mma has quit IRC02:30
redkriegno idea what that first one is02:30
jamesdentoni see what you're saying. And yes, I agree. What version again?02:30
redkrieghttp://paste.openstack.org/show/736943/02:31
jamesdentonk02:32
*** macza_ has quit IRC02:33
*** macza has joined #openstack-ansible02:37
jamesdentonSo what did you do to get it working?02:39
jamesdentonmanually change the file and restart the agent?02:39
redkriegyep02:40
*** macza has quit IRC02:41
redkriegI set it to the value of host_bind_override02:42
jamesdentonHere is the code for that logic: https://github.com/openstack/openstack-ansible-plugins/blob/d94217eb0d19acf3de7b82e0287e994a74a8b15a/library/provider_networks#L248-L25702:43
jamesdentonWhile you do have that network block defined in openstack_user_config.yml, it looks like it's being overridden by the neutron_provider_networks overide is user_variables02:45
jamesdenton*override in02:45
jamesdentonhave these files been modified since you originally had the issue? Just want to make sure I'm looking at the same thing that was implemented at the time02:46
redkriegI'll paste the latest user_variables02:47
redkrieghttp://paste.openstack.org/show/736944/02:49
jamesdentonSo based on that, i would expect to see the mapping in linuxbridge_agent.ini be "flat:br-provider", despite what you might have in openstack_user_config.yml.02:51
jamesdentonthere are playbooks (e.g. OVS) that will create that br-provider bridge and put the attach the network_interface to it. But in this case, the template would simply refer to that override here: https://github.com/openstack/openstack-ansible-os_neutron/blob/stable/rocky/templates/plugins/ml2/linuxbridge_agent.ini.j2#L702:54
jamesdentonAre you in a position to re-run the playbook against a single host to see what happens?02:54
redkriegI can do that tomorrow03:03
jamesdentoncool. i'd appreciate that.03:04
openstackgerritMerged openstack/openstack-ansible-plugins master: missing document start "---"  https://review.openstack.org/61413403:05
openstackgerritMerged openstack/openstack-ansible-plugins master: 'os' imported but unused  https://review.openstack.org/61611903:05
openstackgerritMerged openstack/openstack-ansible-repo_server stable/rocky: Enable git automatic thread count detection  https://review.openstack.org/62252203:12
openstackgerritMerged openstack/openstack-ansible-os_blazar stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62253603:17
openstackgerritMerged openstack/openstack-ansible-os_ceilometer master: Use the distro package for libvirt-python  https://review.openstack.org/62404903:17
openstackgerritMerged openstack/openstack-ansible-os_panko stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62252003:22
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62253503:22
openstackgerritMerged openstack/openstack-ansible-os_masakari stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62251903:24
openstackgerritMerged openstack/openstack-ansible-os_tacker stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62252103:24
openstackgerritMerged openstack/openstack-ansible-os_congress stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62253203:24
openstackgerritMerged openstack/openstack-ansible-os_heat stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62402703:24
openstackgerritMerged openstack/openstack-ansible-os_trove stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62251803:33
*** spatel has joined #openstack-ansible03:34
*** spatel has quit IRC03:35
openstackgerritMerged openstack/openstack-ansible master: [DOCS] Fix user guide to source admin openrc  https://review.openstack.org/62357603:46
*** udesale has joined #openstack-ansible03:48
openstackgerritMerged openstack/openstack-ansible-os_swift stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62251603:52
openstackgerritMerged openstack/openstack-ansible-os_sahara stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62251703:52
openstackgerritMerged openstack/openstack-ansible-os_tempest stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62251503:52
*** gyee has quit IRC03:53
*** jamesden_ has joined #openstack-ansible03:54
*** jamesdenton has quit IRC03:55
openstackgerritMerged openstack/openstack-ansible-os_magnum stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62250503:56
*** markvoelker has joined #openstack-ansible03:57
*** markvoelker has quit IRC04:02
*** udesale has quit IRC04:08
*** udesale has joined #openstack-ansible04:43
*** cshen has joined #openstack-ansible04:45
*** cshen has quit IRC04:50
*** hwoarang has quit IRC05:10
*** hwoarang has joined #openstack-ansible05:12
*** chandan_kumar has joined #openstack-ansible05:21
*** fnpanic has quit IRC05:47
*** cshen has joined #openstack-ansible05:49
*** cshen has quit IRC05:53
openstackgerritMerged openstack/openstack-ansible stable/queens: Docs: Fix VLAN bridge interface in example setup  https://review.openstack.org/62396906:04
openstackgerritMerged openstack/openstack-ansible stable/rocky: Add bionic to Vagrantfile  https://review.openstack.org/62250206:07
*** hamzaachi has joined #openstack-ansible06:10
*** macza has joined #openstack-ansible06:28
*** ahosam has joined #openstack-ansible06:29
*** macza has quit IRC06:32
*** jmccrory has quit IRC06:34
*** sdake has quit IRC06:35
*** jmccrory has joined #openstack-ansible06:40
*** sdake has joined #openstack-ansible06:40
*** hamzaachi has quit IRC06:49
*** mkuf_ has joined #openstack-ansible06:58
*** mkuf has quit IRC07:03
*** ahosam has quit IRC07:08
*** cshen has joined #openstack-ansible07:21
*** macza has joined #openstack-ansible07:30
*** macza has quit IRC07:34
*** mma has joined #openstack-ansible07:37
*** fnpanic has joined #openstack-ansible07:37
*** trident has quit IRC07:42
*** ahosam has joined #openstack-ansible07:43
*** trident has joined #openstack-ansible07:44
*** ahosam has quit IRC07:49
*** hamzaachi has joined #openstack-ansible07:50
*** mkuf_ has quit IRC07:52
*** mma has quit IRC07:53
*** radeks has joined #openstack-ansible07:53
*** mkuf_ has joined #openstack-ansible07:54
*** oanson has quit IRC07:54
*** aedc has joined #openstack-ansible07:54
*** tosky has joined #openstack-ansible07:58
evrardjpo/07:59
jrosserMorning08:08
evrardjpmorning jrosser08:12
fnpanicmorning08:14
*** shardy has joined #openstack-ansible08:15
evrardjpmorning fnpanic and others08:18
evrardjphow are things?08:18
openstackgerritMerged openstack/openstack-ansible master: Only enable OVN VIPs when ml2.ovn plugin is used  https://review.openstack.org/62247308:20
jrosserBusy, but good :)08:20
jrosserfnpanic: did you sort out the issues you were having the other day?08:21
*** shardy has quit IRC08:28
*** hwoarang has quit IRC08:28
*** shardy has joined #openstack-ansible08:29
*** hwoarang has joined #openstack-ansible08:30
*** ahosam has joined #openstack-ansible08:32
*** ahosam has quit IRC08:32
*** priteau has joined #openstack-ansible08:39
*** aedc has quit IRC08:42
*** aedc has joined #openstack-ansible08:42
*** hamzaachi has quit IRC08:46
*** hamzaachi has joined #openstack-ansible08:46
*** goldenfri has quit IRC08:54
*** goldenfri has joined #openstack-ansible08:54
*** ahosam has joined #openstack-ansible09:02
*** eumel8 has joined #openstack-ansible09:05
*** ahosam has quit IRC09:05
*** hamzaachi_ has joined #openstack-ansible09:13
*** hamzaachi_ has quit IRC09:13
*** hamzaachi has quit IRC09:15
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/rocky: nspawn: correct task names  https://review.openstack.org/62218309:16
*** rgogunskiy has joined #openstack-ansible09:17
openstackgerritMerged openstack/openstack-ansible stable/rocky: Update lxc_hosts SHA for CentOS systemd fix  https://review.openstack.org/62404109:18
*** jhonni99 has quit IRC09:20
*** DanyC has joined #openstack-ansible09:25
*** hamzaachi has joined #openstack-ansible09:30
openstackgerritMerged openstack/openstack-ansible-os_cinder master: remove loop control when running include_role  https://review.openstack.org/60781409:32
*** rodolof has joined #openstack-ansible09:47
openstackgerritMerged openstack/openstack-ansible-os_octavia stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62250309:49
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Ensure that systemd-resolved is present in the Centos container  https://review.openstack.org/62433609:49
*** rodolof has quit IRC09:57
*** rodolof has joined #openstack-ansible09:57
odyssey4memorning folks10:03
odyssey4mejrosser should we perhaps make sure systemd-resolved is present for all distributios as a preventative measure?10:03
jrosserodyssey4me: good idea, i'll fix that up10:04
jrosserodyssey4me: that patch is a result of this http://logs.openstack.org/43/620343/1/check/openstack-ansible-functional-centos-7/d099dac/job-output.txt.gz#_2018-12-11_09_32_49_17115710:04
odyssey4meI'm guessing that suddenly the deps have changed in the packages, or the contents of the container have changed10:06
*** rodolof has quit IRC10:09
*** rodolof has joined #openstack-ansible10:10
jrosseroddly, as far as i can see systemd-resolved is part of the systemd package itself on ubuntu10:12
jrosserand appears similar for suse though the package search tool isnt very helpful there10:12
jrossercan anyone with a suse box confirm that?10:13
*** aedc has quit IRC10:18
*** priteau has quit IRC10:24
*** electrofelix has joined #openstack-ansible10:28
*** hamzaachi has quit IRC10:28
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Add ability to symlink host python packages into venv  https://review.openstack.org/62405810:28
*** hamzaachi has joined #openstack-ansible10:28
odyssey4mejrosser ok, if that's the case then perhaps just note it in the commit msg so that no-one else gets the same idea I had ;)10:29
odyssey4mejrosser evrardjp could I get eyes on https://review.openstack.org/#/c/623250/ please10:30
evrardjpodyssey4me: makes sense10:32
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Ensure that systemd-resolved is present in the Centos container  https://review.openstack.org/62433610:32
evrardjpodyssey4me: did we remove the experimental jobs?10:33
odyssey4meevrardjp nope, no need to - those should still work10:33
evrardjpyes it looks like it10:36
noonedeadpunkmorning everyone10:42
*** gisak has joined #openstack-ansible10:50
gisakgood day guys10:51
gisakfacing an issue with networking on my new openstack env, I managed to create external network, and a router10:51
gisakbut unfortunately, the external_gateway port status is down10:52
gisakand floating IP's or dhcp agent's IP are unreachable..10:52
noonedeadpunkI have strange behavior of cinder-api service. I have 2 containers with cinder-api, and both of them has different service status - http://paste.openstack.org/show/736930/Has anyone has symptoms like that?10:53
noonedeadpunk* http://paste.openstack.org/show/736930/10:53
noonedeadpunkthe same staff is happening inside cinder-volume container10:56
*** udesale has quit IRC10:56
noonedeadpunkr/staff/stuff10:56
fnpanicnoonedeadpunk: which backend are you using for cinder?11:02
noonedeadpunkceph11:02
fnpanici guess maybe a problem of the cinder containers contacting the ceph cluster11:03
fnpanicplease verify that this is working11:03
noonedeadpunkbut it's operating normally. For some reason, containers on one host see only services, placed on it. And the other container see both of them11:03
noonedeadpunkSo, as you may see, even scheduler on another container is marked as down - not oncly cinder-volumes11:04
noonedeadpunkAnd when I'm using openstack client, which connects through haproxy, I have 50% chance to get the correct status11:05
fnpaniclooks like a network issue from what i see here.11:05
fnpaniccheck the config of switches and host11:06
fnpanicLACP?11:06
fnpanicbonding?11:06
noonedeadpunkShouldn't it retrieve data from rabbitmq?11:06
noonedeadpunkyep, LACP on host is configured11:06
fnpanicplease verify everyting is setup properly11:06
noonedeadpunkat least ping is ok.... http://paste.openstack.org/show/736974/11:08
jrosserit's quite important what haproxy view of things is - enabling the stats page or using hatop to check all the backends look up is useful11:12
noonedeadpunkjrosser: haproxy says everything is up - I'm monitoring it's output:)11:17
noonedeadpunkFor me it seems, like rabbitmq related issue - as service list should be relying on it, but I'm not quite sure about that11:18
noonedeadpunkI had a guess, that cinder needs rabbit_ha_queues to be enabled, but then everyone using OSA should have such problem. Ok, will try to debug this stuff by reading cinder code... probably, I'll see smth related...11:22
noonedeadpunkLike https://docs.openstack.org/ha-guide/storage-ha-block.html#configure-block-storage-api-service11:22
openstackgerritMerged openstack/openstack-ansible stable/rocky: Fix "configure container (other services)" when statement  https://review.openstack.org/62250711:27
*** hamzaachi has quit IRC11:31
*** rodolof has quit IRC11:35
*** rodolof has joined #openstack-ansible11:35
*** ahosam has joined #openstack-ansible11:57
*** ahosam has quit IRC12:01
*** macza has joined #openstack-ansible12:02
*** macza has quit IRC12:06
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer stable/rocky: Use the distro package for libvirt-python  https://review.openstack.org/62437912:08
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_ceilometer stable/rocky: Enable overriding the service setup host python interpreter  https://review.openstack.org/62253012:08
jrossernoonedeadpunk: fwiw i get the same as this http://paste.openstack.org/show/736931/ - not my area of expertise though....12:09
chandan_kumarodyssey4me: jrosser anything more to add on this https://review.openstack.org/622999 will i enable the same for susu distro job also as temepstconf is already packaged there?12:21
noonedeadpunkjrosser: I think, that the problem is, that my 2 hosts have different time, but I thought, that chrony is supposed to be configured... So I guess, that cinder-scheduler see that last message was too late, and mark service as down.12:21
*** macza has joined #openstack-ansible12:23
*** mkuf has joined #openstack-ansible12:26
*** fresta_ is now known as fresta12:27
*** macza has quit IRC12:28
jamesden_mornin'12:29
*** jamesden_ is now known as jamesdenton12:29
admin0\o12:29
*** mkuf_ has quit IRC12:30
noonedeadpunkAny guess, why chrony has more that 1 min difference? http://paste.openstack.org/show/736979/12:31
*** hamzaachi has joined #openstack-ansible12:31
jamesdentonnot really my wheelhouse, but what does /etc/chrony.conf look like for each of those?12:34
odyssey4meevrardjp jrosser what do you think of https://review.openstack.org/619526 ?12:34
jrosserodyssey4me: i don't think i've ever used the logs in the log container for debugging a CI job12:35
odyssey4mejrosser so the only reason I can think of to keep it is to exercise that code path, but honestly it's barely worth it IMO12:38
odyssey4meThe AIO is quite artificial, so I'm of the opinion that we should slim it down - and perhaps later introduce multi-node testing instead.12:39
*** rodolof has quit IRC12:41
*** rodolof has joined #openstack-ansible12:41
noonedeadpunkjamesdenton: its the same and looks like this http://paste.openstack.org/show/736980/12:43
*** ansmith has quit IRC12:45
jrosserodyssey4me: everything logs as DEBUG too so it's really loads of traffic12:47
openstackgerritMerged openstack/openstack-ansible master: Zuul: Simplify the integrated test playbooks  https://review.openstack.org/62325012:47
jrosseri was looking at the rather out-of-date set of periodic jobs, perhaps these would be a good place to put a more full fat config12:48
odyssey4meevrardjp jrosser mind also taking a peek at https://review.openstack.org/619732 ?12:51
jamesdentonnoonedeadpunk that ref time seems to be the last time it checked the source. You can force a sync with >> chronyc -a 'burst 4/4'12:52
jamesdentonnoonedeadpunk between my two nodes there was nearly a 9 min difference in that ref time, but the time reflected by the 'date' command was within 0.5 secs of each other12:52
*** dave-mccowan has joined #openstack-ansible12:53
jamesdentonnoonedeadpunk "chronyc -a 'burst 4/4'; chronyc -a makestep" might be the trick.12:53
noonedeadpunkjamesdenton yep, it did the trick! Thanks!12:54
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Zuul: Simplify the integrated test playbooks  https://review.openstack.org/62438512:54
noonedeadpunkdate had also more than 1 min difference.12:55
jamesdentonThank you, Oracle docs: https://docs.oracle.com/cd/E90981_01/E90982/html/chrony.html12:55
noonedeadpunkYeah. And it has fixed my problem with cinder status:)12:56
jamesdentonoh, good deal!12:57
noonedeadpunkThe question is - why chrony hasn't synced the time itself... And maybe it's worth considering to use timedatectl, in terms of global movement to systemctl services?12:58
*** vnogin has joined #openstack-ansible12:59
jrosserodyssey4me: won't this still be "distro" and miss a bunch of tempest source related vars in playbooks/defaults/source_install.yml ?12:59
jrosserhttps://github.com/openstack/openstack-ansible/blob/master/playbooks/os-tempest-install.yml#L2112:59
*** dave-mccowan has quit IRC13:01
openstackgerritMerged openstack/openstack-ansible-ceph_client master: tasks: python: Add support for python3 environments  https://review.openstack.org/61660813:08
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Refactor ceph server playbooks  https://review.openstack.org/61114613:08
openstackgerritJonathan Rosser proposed openstack/openstack-ansible master: Refactor ceph server playbooks  https://review.openstack.org/61114613:09
openstackgerritMerged openstack/openstack-ansible stable/rocky: Remove unnecessary octavia scenario AIO bootstrap  https://review.openstack.org/61979213:14
evrardjpjrosser: so?13:18
evrardjpogh13:18
evrardjpoh*13:18
evrardjpyeah it's maybe better to set fact ahead of time in this play for ubuntu instead13:18
odyssey4meevrardjp jrosser well, hmm... I think the trouble is that install_method is set as an extra var - so nothing else will trump it13:19
evrardjpset_fact13:19
jrosserit just felt a bit ugly where some things would be using source_install.yml and others distro_install, felt like a trap waiting to get us later13:19
*** aedc has joined #openstack-ansible13:19
evrardjpset_fact trumps the included vars/13:20
evrardjpoh you meant extra13:20
evrardjpgarg13:20
odyssey4meyep13:20
odyssey4meI'm going to have to take a better look at all this and see what we can do.13:20
evrardjpwell at the same time it passed testing13:21
odyssey4meI think most of those vars are unnecessary to set, so I think it can be simplified/removed.13:21
jrosseryeah i don't have a specific thing it breaks, its just not totally nice13:21
evrardjpsimplification can help there13:21
odyssey4mejrosser could you register your -1 on the patch so that I remember what the issue is.13:21
jrossersure13:21
odyssey4methanks - I'll revisit it some time soon13:23
*** macza has joined #openstack-ansible13:25
jamesdentonis there a magic command that will show me the inventory and "is_metal" status?13:28
odyssey4mejamesdenton well, is_metal is effectively a host var I think - so just using the debug module and an ad-hoc ansible command may show it13:29
*** macza has quit IRC13:29
jrosseris there a release note for this? https://review.openstack.org/#/c/622216/13:35
odyssey4membuil_ ?13:35
jrossertheres suggestion in the comments that there is but i'm having a hard time finding it13:36
jamesdentonthanks odyssey4me13:36
odyssey4mejrosser hmm, I see where a reno would be good13:37
openstackgerritMerged openstack/openstack-ansible stable/rocky: Ensure AIO container_tech/install_method vars are namespaced  https://review.openstack.org/62027813:41
openstackgerritMerged openstack/openstack-ansible stable/rocky: Implement documentation changes for translations  https://review.openstack.org/62027913:41
openstackgerritMerged openstack/openstack-ansible stable/rocky: [docs] Clean up the AIO user story  https://review.openstack.org/62028013:41
*** vnogin has quit IRC13:43
jrossermnaser evrardjp can you take a look at this? https://review.openstack.org/#/c/624336/13:48
openstackgerritMerged openstack/openstack-ansible-os_heat stable/rocky: Switch to using public endpoint for URIs  https://review.openstack.org/61935513:53
openstackgerritMerged openstack/openstack-ansible-os_glance master: Fix permissions on /var/lib/glance/cache  https://review.openstack.org/61934313:55
*** ansmith has joined #openstack-ansible13:55
evrardjpjrosser: makes sense to me. I will not +W though, waiting for mnaser13:55
jrosserta13:56
openstackgerritMerged openstack/openstack-ansible stable/rocky: Implement AIO changes to allow test usage by roles  https://review.openstack.org/62251313:57
pabelangerevrardjp: Is there plan to propose an update to stable/rocky git sha1 sometime this week for roles?14:02
evrardjppabelanger: it was on the agenda of last friday, I am a little late as you can see.14:04
evrardjpI will do that now14:05
openstackgerritMerged openstack/openstack-ansible-os_cinder stable/rocky: Disable cinder api apache config in distro install  https://review.openstack.org/61697814:05
*** macza has joined #openstack-ansible14:06
*** mkuf_ has joined #openstack-ansible14:06
*** mkuf has quit IRC14:09
*** macza has quit IRC14:10
*** mkuf has joined #openstack-ansible14:14
jrosserodyssey4me: that failed job with lxc cache prep did not complete was happening a whole bunch a week or so ago14:17
*** goldenfri has quit IRC14:17
*** mkuf_ has quit IRC14:17
MiougeIt would appreciate to get reviews on those 2 backport for Queens and Pike: https://review.openstack.org/#/q/f8187bda2cb7bdc306e09f369ecbed15fba1e04114:18
Miouges/It/I :)14:18
odyssey4mecores - it'd be good to get this in so that I can backport it - it affects anyone using LDAP: https://review.openstack.org/62057414:18
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible stable/rocky: Update all SHAs for 18.1.2  https://review.openstack.org/62440714:19
*** rodolof has quit IRC14:19
*** rodolof has joined #openstack-ansible14:19
odyssey4meevrardjp any chance that you could rebase that sha bump on top of https://review.openstack.org/624043 ?14:20
evrardjpwill do when my x meetings are over :)14:21
evrardjpbecause brain power required14:21
evrardjphahah14:21
openstackgerritMerged openstack/openstack-ansible-os_nova stable/rocky: Add support for NFS  https://review.openstack.org/61328514:22
openstackgerritMerged openstack/openstack-ansible-galera_server master: Increase Galera self-signed SSL CA expiration  https://review.openstack.org/62423414:22
pabelangerevrardjp: +1, thanks!14:22
odyssey4meevrardjp it will require a rebase after that one merges anyway, which will happen soon-ish14:23
jrosseralso today is the first day for a week+ that we've been able to merge any volume of patches14:24
jrosserso might just need some looking at what needs to be in a release14:24
evrardjpodyssey4me: yeah that was my understanding when I so the votes :)14:24
evrardjpsaw*14:25
odyssey4mejrosser fair statement, although I think https://review.openstack.org/624043 should cover most bases14:25
odyssey4meit updates all the AIO installed roles :p14:25
admin0heat does not work in 18.1.0 in my test stack ( new one) and says: Unable to connect to AMQP server on 172.29.239.110:5671 after None tries: (0, 0): (403) ACCESS_REFUSED - Login was refused using authentication mechanism AMQPLAIN14:25
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: gnocchi_resources override fixed  https://review.openstack.org/62295614:25
admin0how do I force heat user/pass creation on rabbithq14:26
odyssey4meadmin0 just run the playbook again - but make sure that you have the secrets correctly populated14:26
*** aedc has quit IRC14:27
admin0the secrets are there .. i reran the playbook ..even deleted the rabbitmq container14:27
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for running tempestconf from rpm  https://review.openstack.org/62299914:27
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Update messaging notification configuration  https://review.openstack.org/60411714:28
admin0odyssey4me, secrets has heat_service_password and heat_stack_domain_admin_password14:29
admin0also heat_auth_enc_key, container_mysql and heat_oslomsg_rpc14:29
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: gnocchi_resources override fixed  https://review.openstack.org/62295614:30
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_ceilometer master: Add ability to override meters.yaml and event_defenitions.yaml  https://review.openstack.org/62323914:30
odyssey4meadmin0 the rabbtmq passwords are the oslomsg vars14:30
admin0yes and those are correct int he config as well14:31
admin0heat.conf14:31
odyssey4meadmin0 check if these tasks ran successfully: https://github.com/openstack/openstack-ansible-os_heat/blob/stable/rocky/tasks/mq_setup.yml14:32
openstackgerritMerged openstack/openstack-ansible-os_magnum stable/rocky: Drop identity_uri usage in keystone_authtoken  https://review.openstack.org/61927014:33
openstackgerritMerged openstack/openstack-ansible-os_magnum stable/queens: Drop identity_uri usage in keystone_authtoken  https://review.openstack.org/61927114:33
admin0rerunning the rabbit install playbook with logging to a file .. will grep and check after this14:33
admin0will do the same for heat14:33
odyssey4meadmin0 the rabbitmq install play only installs rabbit, it doesn't do any of the users - the service plays do that14:33
*** udesale has joined #openstack-ansible14:33
admin0loggin both :) just in case i may catch any errror14:34
odyssey4meadmin0 also, there is already a log file by default in https://github.com/openstack/openstack-ansible/blob/master/scripts/openstack-ansible.rc#L1914:34
admin0TASK [os_heat : Add RPC Qdrouterd user]  - has no OK14:35
admin0just blank14:35
odyssey4meadmin0 yes, because you're not using qrouterd - you're using rabbitmq14:36
admin0TASK [os_heat : Add Notify RabbitMQ user]  - also blank14:36
admin0no OK .. nothing14:36
*** rodolof has quit IRC14:37
odyssey4meadmin0 that'll only be actioned if it's different to the rpc user/password14:37
odyssey4mewhich is not the case here, you're using the default14:37
*** rodolof has joined #openstack-ansible14:37
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible stable/queens: Update all SHAs for 17.1.6  https://review.openstack.org/62441414:38
*** rodolof has quit IRC14:38
*** rodolof has joined #openstack-ansible14:38
admin0odyssey4me, so if i change the user_pass to 1 bit something else in user_secrets, that should re-add and re-create the files and users14:39
rjgibsonFollowing up on my cinder problem from yesterday... I had some leftover config in env.d/cinder.yml which specified cinder_scheduler_container in container_skel14:39
odyssey4meadmin0 no, the user/pass are always created every time14:39
odyssey4meadmin0 as long as that task runs, it should show changed when running the heat role - if it only shows ok, then I'd be surprised, because most user/pass tasks aren't idempotent14:40
*** cshen has quit IRC14:41
admin0well, the issue is it does not even show OK14:41
admin0the config has the correct pass14:41
admin0but rabbitmq somehow does not have that user/pass setup correctly14:41
odyssey4meadmin0 what does the task result show ?14:41
admin0https://pastebin.com/vZWzEXPD14:41
odyssey4meadmin0 it's showing changed14:42
odyssey4meonly two tasks are running - one to create the vhost, and one to create the user and set the password - that's how it's supposed to work14:42
admin0hmm.. how to fix this then .. that rabbit denies the user14:43
odyssey4meadmin0 try using the user/password through the cli or something to verify it - is this a new env which had passwords previously? if so, check if the old one works?14:43
admin0this is a brand new env14:44
*** kopecmartin has joined #openstack-ansible14:45
*** aedc has joined #openstack-ansible14:46
odyssey4mewell then I dunno, you'll have to try it via CLI to verify whether it's a user/pass issue or some other connectivity issue14:46
*** jawad_axd has joined #openstack-ansible14:46
admin0odyssey4me, i changed the user_secret by hand, reran the heat playbook and found something .. http://paste.openstack.org/show/737020/  - maybe it can say something14:49
odyssey4meadmin0 what does it tell you?14:50
admin0i am trying out to figure out which of the conditional set it to faluse14:51
admin0lots of OR (|) in there14:51
*** jawad_axd has quit IRC14:51
odyssey4meadmin0 you're not reading it right - it's running against two hosts, only one should actually run it14:51
odyssey4mebecause the rabbitmq user can only be created in rabbitmq once, otherwise there'd be a race condition14:52
odyssey4meso if you read it carefully, it's not running on the first host - but is on the second14:52
odyssey4meso it is doing the right thing14:52
*** jawad_axd has joined #openstack-ansible14:52
odyssey4meadmin0 as suggested before, use the rabbitmq cli to see whether the user is there and the password is right - that helps to diagnose whether that is actually the issue or whether it's a connectivity one14:54
admin0i think i have a runaway container :D14:55
odyssey4mehuh?14:55
*** jawad_ax_ has joined #openstack-ansible14:56
*** jawad_axd has quit IRC14:56
*** jawad_ax_ has quit IRC15:00
*** markvoelker has joined #openstack-ansible15:00
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible stable/rocky: Document how to use the utility container for service setup  https://review.openstack.org/62404315:01
jamesdentonQQ: I just performed a P->Q upgrade, expecting to see the behavior outlined in the release notes here: "Neutron connectivity agents will now be deployed on baremetal within the “network_hosts” defined within the openstack_user_config.yml". But that didn't occur. I do have network_hosts defined as my (single) infra node, but no indication that agents were deployed on baremetal15:17
*** oanson has joined #openstack-ansible15:17
*** mkuf has quit IRC15:22
neithHello, I am facing an issue where it seems like instances randomly freeze while connected over ssh. I'm trying to determine if its a network problem related to ssh transport or a real freeze on the instance side. How would you test to eliminate the suspicion regarding ssh. Our setup is an openstack deployed with osa/queens using linux bridge.15:22
*** rodolof has quit IRC15:27
*** rodolof has joined #openstack-ansible15:27
openstackgerritMerged openstack/openstack-ansible-ops master: MNAIO: Fix warning for guestfish workaround task  https://review.openstack.org/62059915:28
*** mkuf has joined #openstack-ansible15:28
openstackgerritMerged openstack/openstack-ansible-ops master: MNAIO: If there is no secondary data disk, use the root disk  https://review.openstack.org/62060015:30
jamesdentonneith it's locking up while using it? Any consistency (i.e. is it after some idle period) or can you be actively using it and it just stops responding?15:30
openstackgerritJonathan Rosser proposed openstack/openstack-ansible-lxc_hosts master: Use libeatmydata during LXC cache prep  https://review.openstack.org/62150915:33
*** ivve has joined #openstack-ansible15:37
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible stable/pike: Update all SHAs for 16.0.24  https://review.openstack.org/62442715:38
noonedeadpunkjamesdenton: so you don't have /openstack/venvs/neutron-17-*/ on your network node?15:41
*** weezS has joined #openstack-ansible15:41
jamesdentonno15:41
noonedeadpunkAnd don't you have somewhere is_metal: false for network_containers?15:46
noonedeadpunk*neutron_agents_container15:46
noonedeadpunkAs for queens is_metal: true https://github.com/openstack/openstack-ansible/blob/stable/queens/inventory/env.d/neutron.yml#L56-L7115:47
noonedeadpunkFolks, can you kindly review these 2 patches? https://review.openstack.org/#/q/topic:template_overrides+(status:open+OR+status:merged)15:49
jamesdentonnoonedeadpunk i confirmed the env.d skel has is_metal: true. I am not sure if I should expect changes to the inventory or not15:49
noonedeadpunkI think, that inventory is like a war - it never changes:)15:50
noonedeadpunkSo I had to change inventory manually, once I some record in it is already present.15:51
jamesdentoni might try greenfield queens deploy and compare.15:51
cjloadercan I get some eyes on the designate spec https://review.openstack.org/#/c/556586/15:51
cjloadermnaser: you mentioned to bug you at the ptg15:52
admin0odyssey4me, fixed finally .. had to remove the heat completely from inventory and ips, have it create new one, change passwords and finally working15:53
admin0jamesdenton, doesn't pci passthrough whitelist need the pci address https://docs.openstack.org/nova/pike/admin/pci-passthrough.html  -- while the ansible way we did only passes the devname and not the address ?15:55
*** markvoelker has quit IRC15:55
jamesdentonadmin0 https://docs.openstack.org/nova/pike/configuration/config.html and check passthrough_whitelist¶15:56
*** markvoelker has joined #openstack-ansible15:56
jamesdentonhttps://docs.openstack.org/nova/pike/configuration/config.html#pci.passthrough_whitelist15:56
admin0https://docs.openstack.org/nova/pike/admin/pci-passthrough.html  -- we have 2 docs that say different15:57
admin0is there a way to test if this is working ?15:57
neithjamesdenton its locking while using it yes, and I did not see any pattern15:57
neithjamesdenton: its like a fsfreeze15:58
jamesdentonadmin0 well it does say, For more information about the syntax of passthrough_whitelist, refer to Configuration Options ;D15:58
admin0yeah . i think it can be said any way15:58
admin0config file accepts any way i think15:58
admin0"Alternatively the [pci] passthrough_whitelist parameter also supports whitelisting by:" --15:58
admin0read this :)15:58
admin0i meant to say i read that .. and not read this to you16:00
*** aedc has quit IRC16:01
*** markvoelker has quit IRC16:01
admin0past tense of read is read  :D16:01
*** aedc has joined #openstack-ansible16:01
*** cshen has joined #openstack-ansible16:01
evrardjpo/16:02
admin0\o16:02
redkriegjamesdenton: finished a playbook run with flat:eth12 in network_mappings.  everything is correct now.  I don't know why that config was set up that way, I borrowed config examples from all over the place :(16:03
redkriegthanks again for your help16:04
jamesdentonok, so host_bind_override worked then? good deal16:04
evrardjpis there a meeting today?16:05
cjloaderthx odyssey4me for the review16:05
neithjamesdenton: the only thing I noticed is that the duration of the freeze varies from a few seonds to 30sec16:06
neithjamesdenton: I should try to tcpdump on the tap device of an instance to see if network flow is suspended, what do you think?16:07
jamesdentonnot a bad idea. is it restoring on its own?16:08
jamesdentonCheck virsh to see if instance is becoming paused16:08
neithjamesdenton: yes its restoring on its own16:09
*** jawad_axd has joined #openstack-ansible16:11
openstackgerritMerged openstack/openstack-ansible-os_keystone master: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62057416:11
*** udesale has quit IRC16:14
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone stable/rocky: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62443016:14
*** jawad_axd has quit IRC16:15
*** rodolof has quit IRC16:15
*** rodolof has joined #openstack-ansible16:16
mgariepyadmin0, https://github.com/openstack/openstack-ansible-os_nova/blob/master/defaults/main.yml#L648-L65916:17
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone stable/queens: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62443316:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone stable/pike: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62443416:19
mgariepyadmin0, I do it like: the devname is there i guess only for network devices.16:20
mgariepynova_pci_passthrough_whitelist: '[{"vendor_id": "10de", "product_id": "1b02"}]'16:21
mgariepythen i set an alias for the devices and use that in the flavors16:21
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_keystone stable/ocata: Ensure that LDAP config is deployed on all keystone hosts  https://review.openstack.org/62443616:21
*** rodolof has quit IRC16:23
*** rodolof has joined #openstack-ansible16:24
*** rodolof has quit IRC16:44
*** rodolof has joined #openstack-ansible16:44
*** macza has joined #openstack-ansible16:49
ArchiFleKsHi, I'm trying to deploy on AIO with scenario aio_nspawn and it failed at http://paste.openstack.org/show/737036/ It tries t connect to 172.29.236.100, but I have no bidge created automatically, I have only 'mv-*'16:53
ArchiFleKscloudnull: I have no interface created with this IP16:56
*** aedc has quit IRC17:06
*** hamzaachi has quit IRC17:07
openstackgerritJean-Philippe Evrard proposed openstack/openstack-ansible stable/rocky: Update all SHAs for 18.1.2  https://review.openstack.org/62440717:15
*** kopecmartin is now known as kopecmartin|off17:16
*** aedc has joined #openstack-ansible17:20
*** gyee has joined #openstack-ansible17:22
openstackgerritAndy Smith proposed openstack/openstack-ansible master: Add qdrouterd role for rpc messaging backend deployment  https://review.openstack.org/62418417:27
*** vakuznet has joined #openstack-ansible17:30
*** DanyC has quit IRC17:32
*** DanyC has joined #openstack-ansible17:32
*** rodolof has quit IRC17:33
*** rodolof has joined #openstack-ansible17:33
*** DanyC has quit IRC17:34
*** hamzaachi has joined #openstack-ansible17:50
openstackgerritMerged openstack/openstack-ansible-galera_server stable/rocky: Fix Galera self-signed SSL functionality  https://review.openstack.org/61597117:52
*** gyee has quit IRC17:56
*** rodolof has quit IRC17:58
*** rodolof has joined #openstack-ansible17:59
openstackgerritMerged openstack/openstack-ansible-os_nova stable/rocky: Cleanup nova distro package install lists  https://review.openstack.org/61698518:02
nsmedsmorning/evening guys. Question about something - I saw that a change was merged from master into stable/rocky ( https://review.openstack.org/#/c/615971/ ), which is sweet. Will it also be merged into stable/queens?18:12
openstackgerritAntony Messerli proposed openstack/openstack-ansible-ops master: Better handling for resuming leapfrog upgrade  https://review.openstack.org/62445518:16
*** Adri2000 has quit IRC18:43
gisakguys can you help me with external network issue on my new openstack env18:44
gisakfighting this one from 1 week so far, cant understand why gateway is unreachable ...18:44
gisakthe port status from public-network is Down for some reason18:46
gisakbut binding VIF type is : bridge (ok as I understand), still unreachable network anyways ((18:47
gisakhttp://paste.openstack.org/show/737043/18:48
*** Adri2000 has joined #openstack-ansible18:48
*** gisak has quit IRC18:54
jrosserWell, I’d probably be starting on the network node / controller with all the neutron logs18:57
jrosserLooking for error messages, there’s usually something, or at the worse a stack trace if something had gone really horribly wrong18:58
jrosserThen start poking in the relevant network namespace to ping the instance and your external gateway ip18:59
mgariepynsmeds, you can cherry pick the change to queens branch19:08
nsmedsmgariepy: ok, cool. time to learn something new again (i'll give it a shot in a bit)19:09
jonhernsmeds: just click cherry pick on the link you sent and select the stable/queens branch and you're good :)19:10
*** cshen has quit IRC19:12
*** cshen has joined #openstack-ansible19:13
nsmedsthat'd have been too easy - need to deal with merge conflict19:13
jonheroh :)19:15
*** cshen has quit IRC19:17
*** electrofelix has quit IRC19:26
*** lbragstad has quit IRC19:30
mgariepyhaha :D19:30
*** lbragstad has joined #openstack-ansible19:31
*** jawad_axd has joined #openstack-ansible19:35
*** shardy has quit IRC19:37
*** cshen has joined #openstack-ansible19:38
mgariepynsmeds, if you need help let us know.19:39
nsmedsjust struggling with something else right now, but I'll give it a shot today - useful to learn19:40
nsmedsappreciate it19:40
mgariepyok19:40
*** jawad_axd has quit IRC19:40
mgariepybasicaly you git clone the repo, checkout stable/queens, cherry-pick the wanted patch, resolv the conflict and then submit for review19:41
*** ansmith has quit IRC19:41
nsmeds:thumbsup"19:41
*** ansmith has joined #openstack-ansible19:41
*** cshen has quit IRC19:43
*** bouncy has quit IRC19:46
*** macza has quit IRC19:51
*** macza has joined #openstack-ansible19:53
*** cshen has joined #openstack-ansible19:53
openstackgerritMerged openstack/openstack-ansible stable/rocky: Document how to use the utility container for service setup  https://review.openstack.org/62404320:02
openstackgerritNiko Smeds proposed openstack/openstack-ansible-galera_server stable/queens: Fix Galera self-signed SSL functionality  https://review.openstack.org/62448220:17
nsmedsmaybe?20:17
nsmedsexcept I didn't make a branch for it... derp. Does it matter? was following https://docs.openstack.org/project-team-guide/stable-branches.html#proposing-fixes20:20
*** DanyC has joined #openstack-ansible20:20
jrosseryou can fix that by editing the topic in gerrit ui20:33
*** jawad_axd has joined #openstack-ansible20:38
nsmedsdone20:41
*** jawad_axd has quit IRC20:42
admin0jamesdenton, what was the git for the fail-mode again :)20:50
admin0i forgot to bookmark where i cherry-picked it from20:50
*** jawad_axd has joined #openstack-ansible20:58
*** ThiagoCMC has joined #openstack-ansible21:00
*** rodolof has quit IRC21:01
*** rodolof has joined #openstack-ansible21:01
*** jawad_axd has quit IRC21:03
*** cshen has quit IRC21:08
*** ansmith has quit IRC21:10
*** aedc has quit IRC21:22
*** aedc has joined #openstack-ansible21:22
*** markvoelker has joined #openstack-ansible21:45
*** cshen has joined #openstack-ansible21:45
*** markvoelker has quit IRC21:49
*** DanyC has quit IRC21:52
*** jawad_axd has joined #openstack-ansible22:00
*** jawad_axd has quit IRC22:04
openstackgerritJonathan Rosser proposed openstack/openstack-ansible stable/rocky: nspawn: correct task names  https://review.openstack.org/62218322:11
*** radeks has quit IRC22:18
*** markvoelker has joined #openstack-ansible22:24
jonheradmin0: what's the review URL for the cherry picked version?22:25
admin0jonher, https://review.openstack.org/#/c/613286/1/tasks/providers/ovs_config.yml22:26
jonheradmin0: this? https://review.openstack.org/#/c/611874/22:30
admin0yep22:31
*** rjgibson has quit IRC22:45
*** aedc has quit IRC22:54
*** jawad_axd has joined #openstack-ansible23:23
*** jawad_axd has quit IRC23:27
openstackgerritMerged openstack/openstack-ansible stable/rocky: Update playbook to newer syntax.  https://review.openstack.org/62056923:30
*** vakuznet has quit IRC23:37
*** jawad_axd has joined #openstack-ansible23:44
*** jawad_axd has quit IRC23:49
*** rodolof has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!