Tuesday, 2018-10-23

openstackgerritJimmy McCrory proposed openstack/openstack-ansible-tests master: Restart lxc-dnsmasq after clearing iptables rules  https://review.openstack.org/61253900:04
*** chhagarw has joined #openstack-ansible00:47
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Update ansible to latest stable 2.6.x  https://review.openstack.org/61206200:47
*** chhagarw has quit IRC00:51
*** mmercer has quit IRC01:00
*** spatel has joined #openstack-ansible01:22
*** markvoelker has joined #openstack-ansible01:25
*** idlemind has joined #openstack-ansible01:53
*** canori01 has quit IRC01:54
*** dave-mccowan has quit IRC02:57
*** munimeha1 has quit IRC03:01
*** fatdragon has joined #openstack-ansible03:32
*** fatdragon has quit IRC03:43
*** chhagarw has joined #openstack-ansible03:58
*** udesale has joined #openstack-ansible03:58
*** faizy98 has joined #openstack-ansible04:14
*** faizy_ has quit IRC04:16
*** faizy98 has quit IRC04:34
openstackgerritzhulingjie proposed openstack/ansible-config_template master: use include_tasks instead of include  https://review.openstack.org/61256904:40
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro  https://review.openstack.org/59142404:47
*** fatdragon has joined #openstack-ansible04:57
openstackgerritcaoyuan proposed openstack/openstack-ansible-os_cinder stable/rocky: Switch include to include_tasks or import_playbook  https://review.openstack.org/61259604:59
*** spatel has quit IRC05:02
*** fatdragon has quit IRC05:09
*** fatdragon has joined #openstack-ansible05:40
openstackgerritVictor Palma proposed openstack/openstack-ansible-ops master: fix osquery filesystem logging  https://review.openstack.org/61259805:40
*** ThiagoCMC has quit IRC05:46
*** spsurya has joined #openstack-ansible05:48
*** fatdragon has quit IRC05:51
openstackgerritFrank Kloeker proposed openstack/openstack-ansible master: Add new tox doc and publishing job  https://review.openstack.org/61251005:52
*** faizy98 has joined #openstack-ansible05:57
*** fatdragon has joined #openstack-ansible06:22
*** fatdragon has quit IRC06:29
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro  https://review.openstack.org/59142406:43
*** pcaruana has joined #openstack-ansible06:56
*** fatdragon has joined #openstack-ansible07:02
*** fatdragon has quit IRC07:06
*** rpittau has quit IRC07:07
*** rpittau has joined #openstack-ansible07:07
*** cshen has joined #openstack-ansible07:16
*** shardy has joined #openstack-ansible07:19
*** cshen has quit IRC07:21
openstackgerritJimmy McCrory proposed openstack/openstack-ansible-os_neutron master: Include dependency packages in distro install list  https://review.openstack.org/61261407:22
openstackgerritJimmy McCrory proposed openstack/openstack-ansible-os_nova master: Cleanup nova distro package install lists  https://review.openstack.org/61206107:22
*** fatdragon has joined #openstack-ansible07:40
*** cshen has joined #openstack-ansible07:47
*** fatdragon has quit IRC07:52
*** cshen has quit IRC07:52
*** tosky has joined #openstack-ansible07:57
mnaserodyssey4me: as our resident repo build expert, it looks like i'm getting problems rebuilding vensv in rocky08:01
*** thuydang has joined #openstack-ansible08:01
mnaserodyssey4me: http://paste.openstack.org/show/732689/08:02
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: SUSE: Add support for openSUSE Leap 15  https://review.openstack.org/60436608:03
*** thuydang has quit IRC08:06
*** cshen has joined #openstack-ansible08:12
*** pvradu has joined #openstack-ansible08:14
*** shardy has quit IRC08:17
*** cshen has quit IRC08:17
*** cshen has joined #openstack-ansible08:18
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61262908:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61262908:27
odyssey4memnaser: oh? have you got a failure log handy?08:28
*** michelv has joined #openstack-ansible08:28
*** michelv has quit IRC08:29
*** electrofelix has joined #openstack-ansible08:29
*** michelv has joined #openstack-ansible08:31
*** michelv has left #openstack-ansible08:33
mnaserodyssey4me: paste above, not sure if it made it08:36
mnaserhttps://review.openstack.org/60436608:36
odyssey4memnaser: oh duh - sorry08:36
mnaseroops yeah i mean http://paste.openstack.org/show/732689/08:36
mnaseralso tstrangely enough08:36
odyssey4memnaser: ok, I saw that the other day and made a note to look into it, so I'll do that today08:36
mnaserodyssey4me: it only happens on a rebuild, initial build seemed to be ok08:36
odyssey4meyep, it makes sense - there is a bug, and I'll patch it shortly08:37
mnaserodyssey4me: awesome, i've been trying to do as much as i can in terms of bringing stuff upstream, this is all gonna be covering rocky/bionic deployment which is nice coverage08:38
odyssey4methat's odd https://github.com/openstack/openstack-ansible-repo_build/blob/stable/rocky/templates/venv-build-script.sh.j2#L76-L87 shows it using the upstream index, so it should reach out to pypi and find whatever it needs if it's not available locally... I guess I'll have to fire up a build instead08:40
odyssey4mecould you enable the debug callback and fire that off again, then paste the stdout/stderr?08:41
mnaserodyssey4me: i ran it with -vvv and stdout had "Building <whatever>..." but stderr was empty08:42
mnaserand rc was 108:42
mnaserodyssey4me: OH also08:42
odyssey4meoh of course, we push the logs to a log file08:42
mnaseri ran it manually08:42
mnaserand it worked.08:42
odyssey4meok, let me fire up a build to try this out08:42
mnaserso somehow when ansible runs it, it doesnt work, but when i ran the script manually08:42
mnaseraka /opt/venv-builder-foo ..../cinder.. in shell as root08:43
mnaserBUT i didnt check the exit code08:43
mnaserso it could be failing there too08:43
mnaserlet me confirm08:43
odyssey4meso you ran /opt/venv-build-script.sh /var/www/repo/os-releases/<tag>/venv-build-options_<service>.txt (or something like that08:43
mnaseryeah08:44
mnaserrerunning it08:44
odyssey4mehmm, that is odd08:44
mnaserto check exit code08:44
mnaserit might be dying too08:44
*** priteau has joined #openstack-ansible08:45
mnaserodyssey4me: hmm, it worked08:46
mnaserodyssey4me: but i wonder if its because i didnt do sudo -Hs before the lxc-attach08:47
mnaserso HOME=/home/mnaser08:47
mnasertherefore it's ignoring all our pip tooling config08:47
mnaserlet me try again with sudo -Ha08:47
mnaserHs*08:47
odyssey4meinteresting - yeah, I do think it has something to do with that08:47
odyssey4meok, I've got a patch to push up to try08:48
mnaserodyssey4me: yep, failed with sudo -Hs so when HOME=/root and it parsed the pip stuff08:49
*** shardy has joined #openstack-ansible08:49
odyssey4meyep, I'm just looking for the right place to put --isolated so that it ignores pip.conf, because the repo build shouldn't be using it08:50
mnaserodyssey4me: i guess it wasnt caught the first time around because repo-use runs after repo-server right?08:50
odyssey4mecorrect08:50
mnaserso the first build doesnt have the pip settings, then after that it breaks08:50
odyssey4meso this would be an issue on re-run or on upgrade08:50
* mnaser is slowly learning more about this08:50
mnaserexact behaviour yup08:50
odyssey4meI'm busy trying to eliminate it all entirely so that all options are on the CLI and in the task - more obvious that way08:51
*** vnogin has joined #openstack-ansible08:51
*** fatdragon has joined #openstack-ansible08:56
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build master: Isolate repo wheel builds from pip.conf  https://review.openstack.org/61263908:58
odyssey4memnaser: ^ that's only partial for rocky - lemme do a patch for rocky now08:58
*** vnogin has quit IRC08:58
*** fatdragon has quit IRC09:00
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build stable/rocky: Isolate repo wheel builds from pip.conf  https://review.openstack.org/61264009:01
odyssey4memnaser: try that one?09:01
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_almanach master: Set container_name as host_var in test inventory  https://review.openstack.org/61234909:04
mnaserodyssey4me: let me pull that and try it09:04
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_cloudkitty master: Set container_name as host_var in test inventory  https://review.openstack.org/61235409:05
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_watcher master: Set container_name as host_var in test inventory  https://review.openstack.org/61236109:07
*** sm806 has joined #openstack-ansible09:07
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible stable/rocky: scripts: sources-branch-updater: Make regexp for master branch stricter  https://review.openstack.org/61090209:09
mnaserodyssey4me: same thing09:09
*** rgogunskiy has joined #openstack-ansible09:09
mnaserodyssey4me: want me to run the venv builder as root with set -x ?09:09
odyssey4memnaser: I can have a test env running shortly.09:09
mnaserodyssey4me: ok cool, no worries then, im around if you need anything09:10
odyssey4memnaser: I see why that didn't work - the problem is when creating the venv, not when upgrading the stuff in it09:16
odyssey4megimme a sec to track that09:16
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: SUSE: Add support for openSUSE Leap 15  https://review.openstack.org/60436609:22
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-repo_build stable/rocky: Create virtualenv without downloading  https://review.openstack.org/61264009:23
odyssey4memnaser: get rid of that previous patch - this one should do the trick09:24
*** tosky has quit IRC09:24
mnaserodyssey4me: cool, ill reset to HEAD~1 and try that09:24
*** tosky has joined #openstack-ansible09:24
mnaserodyssey4me: fixed it!09:27
odyssey4mebrilliant :)09:28
mnaserthank you very much for your help09:28
chandankumarodyssey4me: Hello09:34
*** vnogin has joined #openstack-ansible09:34
*** fatdragon has joined #openstack-ansible09:35
*** suggestable has joined #openstack-ansible09:35
odyssey4mechandankumar: o/09:35
*** FuzzyFerric has joined #openstack-ansible09:36
chandankumarodyssey4me: I need some help http://logs.openstack.org/24/591424/39/check/openstack-ansible-functional-distro_install-centos-7/25dc0c2/job-output.txt.gz#_2018-10-23_08_09_28_366857 and http://logs.openstack.org/24/591424/39/check/openstack-ansible-functional-distro_install-opensuse-423/04dfb50/job-output.txt.gz#_2018-10-23_07_42_59_51665609:37
chandankumarodyssey4me: https://review.openstack.org/#/c/591424/09:37
chandankumarodyssey4me: centos one is failing with different error and for other distro I am not sure what is the exact package name09:38
suggestableHey folks!09:38
suggestableTrying to get LDAP integration working with stable/rocky on Bionic.09:38
suggestableI've got this at the moment: http://paste.openstack.org/show/jVlCEgEE0ID5DBnQeT3B/09:38
*** fatdragon has quit IRC09:40
odyssey4memnaser or any other cores: if you have a bit of time, I'd appreciate some reviews for https://goo.gl/k9cCfF09:41
odyssey4mechandankumar: alright, let me do a complete review09:43
chandankumarodyssey4me: sure09:44
chandankumarodyssey4me: I have used recursive so that /opt/<blah>/<blah> dir does not then it should get created09:45
odyssey4mechandankumar: that's not what recurse does... form the ansible docs: 'recursively set the specified file attributes (applies only to directories)'09:49
odyssey4mechandankumar: ansible always creates directories in the same way as mkdir -p <dir>09:49
chandankumarodyssey4me: thanks for highlighting :-)09:50
*** udesale has quit IRC09:51
*** udesale has joined #openstack-ansible09:52
*** shardy has quit IRC09:57
*** rgogunskiy has quit IRC09:57
*** rgogunskiy has joined #openstack-ansible09:58
*** shardy has joined #openstack-ansible09:58
*** rgogunskiy has quit IRC09:58
*** pvradu has quit IRC10:00
*** pvradu has joined #openstack-ansible10:00
*** pvradu_ has joined #openstack-ansible10:05
*** pvradu has quit IRC10:09
*** fatdragon has joined #openstack-ansible10:15
*** fatdragon has quit IRC10:20
*** ianychoi has quit IRC10:22
suggestableAnyone able to help with my LDAP issue?10:22
suggestablehttp://paste.openstack.org/show/jVlCEgEE0ID5DBnQeT3B/10:22
odyssey4mechandankumar: reviewed10:25
*** ianychoi has joined #openstack-ansible10:25
chandankumarodyssey4me: thanks :-)10:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests stable/rocky: Update Ansible to 2.5.10  https://review.openstack.org/61240510:26
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_almanach master: Set container_name as host_var in test inventory  https://review.openstack.org/61234910:29
odyssey4meevrardjp: I can't see any glance logs here, can you find them? https://review.openstack.org/#/c/611508/10:35
*** cshen has quit IRC10:40
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Use distro_install project-template and bionic/lxd  https://review.openstack.org/60843110:46
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: test-log-collect: Disable repositories when collecting installed on SUSE  https://review.openstack.org/61266110:49
*** rgogunskiy has joined #openstack-ansible10:50
*** fatdragon has joined #openstack-ansible10:51
mnaserodyssey4me: i assume this should be -W so it isnt part of your list? https://review.openstack.org/#/c/612425/ :)10:52
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-tests master: test-log-collect: Disable repositories when collecting installed pkgs on SUSE  https://review.openstack.org/61266110:52
*** rgogunskiy has quit IRC10:54
odyssey4memnaser: yep, thanks10:55
*** fatdragon has quit IRC10:56
openstackgerritMerged openstack/openstack-ansible-os_cinder stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61234010:57
openstackgerritMerged openstack/openstack-ansible-os_keystone stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61162610:58
openstackgerritMerged openstack/openstack-ansible-os_magnum stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61162410:59
openstackgerritMerged openstack/openstack-ansible-os_aodh stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61231710:59
openstackgerritMerged openstack/openstack-ansible-os_glance stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61232010:59
openstackgerritMerged openstack/openstack-ansible-os_ceilometer stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61231910:59
openstackgerritMerged openstack/openstack-ansible-os_designate stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61232111:00
openstackgerritMerged openstack/openstack-ansible-os_sahara stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61233911:00
openstackgerritMerged openstack/openstack-ansible-os_barbican stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61234111:01
*** cshen has joined #openstack-ansible11:05
openstackgerritMerged openstack/openstack-ansible-os_heat stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61162811:05
*** rgogunskiy has joined #openstack-ansible11:07
*** rgogunskiy has quit IRC11:07
*** vnogin has quit IRC11:08
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro  https://review.openstack.org/59142411:10
openstackgerritMerged openstack/openstack-ansible-os_trove stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61233711:11
openstackgerritMerged openstack/openstack-ansible-os_nova stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61233811:11
openstackgerritMerged openstack/openstack-ansible-os_ironic stable/rocky: releasenotes: oslo-messaging-separate-backends add project name  https://review.openstack.org/61233511:11
*** udesale has quit IRC11:14
*** tosky has quit IRC11:18
*** tosky has joined #openstack-ansible11:18
*** fghaas has quit IRC11:20
*** fghaas has joined #openstack-ansible11:21
*** fatdragon has joined #openstack-ansible11:25
*** vnogin has joined #openstack-ansible11:29
*** fatdragon has quit IRC11:29
openstackgerritMerged openstack/openstack-ansible-os_watcher master: Set container_name as host_var in test inventory  https://review.openstack.org/61236111:35
hwoarangfyi a centos failures which i dont have time to deal right now http://logs.openstack.org/66/604366/7/check/openstack-ansible-deploy-aio_lxc-centos-7/1c27c64/job-output.txt.gz#_2018-10-23_11_10_01_08994511:39
hwoarangsounds like a regression11:39
hwoarangok every distro is affected by this11:41
openstackgerritMerged openstack/openstack-ansible-tests stable/rocky: Set container_name as host_var in test inventory  https://review.openstack.org/61262911:42
*** strobelight has joined #openstack-ansible11:43
openstackgerritMerged openstack/openstack-ansible-os_cloudkitty master: Set container_name as host_var in test inventory  https://review.openstack.org/61235411:44
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-repo_build master: SUSE: Add support for openSUSE Leap 15  https://review.openstack.org/61267111:44
*** markvoelker has quit IRC11:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-tests master: Add bionic py3 test and switch linters/integrated to bionic  https://review.openstack.org/61267211:46
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: vars: Add missing netcat package  https://review.openstack.org/61267311:48
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_keystone master: vars: Add missing netcat package  https://review.openstack.org/61267311:48
openstackgerritOpenStack Proposal Bot proposed openstack/openstack-ansible-os_blazar stable/rocky: Updated from OpenStack Ansible Tests  https://review.openstack.org/61267411:49
openstackgerritMerged openstack/openstack-ansible-tests master: Set container_name as host_var in test inventory  https://review.openstack.org/61236511:50
*** setuid has left #openstack-ansible11:50
*** ansmith has quit IRC11:52
*** fatdragon has joined #openstack-ansible12:02
*** spatel has joined #openstack-ansible12:04
*** hwoarang has quit IRC12:04
*** hwoarang has joined #openstack-ansible12:06
*** fatdragon has quit IRC12:06
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_tempest master: Added support for installing tempest from distro  https://review.openstack.org/59142412:07
*** dave-mccowan has joined #openstack-ansible12:08
*** spatel has quit IRC12:08
openstackgerritMerged openstack/openstack-ansible-os_octavia master: Ensure that the flavor/network facts are available to all targets  https://review.openstack.org/61138912:11
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_octavia stable/rocky: Ensure that the flavor/network facts are available to all targets  https://review.openstack.org/61267812:15
benkohlI did only updated the openstack-ansible 18.0.0 os_ceilometer role to a never commit and get this: https://gist.github.com/benkuly/69a8735deb574988c6060c056c651a0012:19
*** tobberydberg has quit IRC12:30
*** udesale has joined #openstack-ansible12:31
odyssey4mecores, I have some reviews ready and waiting that need more eyes please: http://bit.ly/2JbHyki12:33
*** markvoelker has joined #openstack-ansible12:36
*** markvoelker has quit IRC12:37
odyssey4mebenkohl: in the repo container's /var/log/repo you'll find each venv's log file which will have the ceilometer log file and probably the error in it12:38
odyssey4mebenkohl: it's possible that https://review.openstack.org/612001 will resolve it12:38
*** fatdragon has joined #openstack-ansible12:39
*** jamesdenton has joined #openstack-ansible12:39
*** fatdragon has quit IRC12:43
odyssey4mecores - here's a better list which includes patches from others :) http://bit.ly/2PNVSSn12:43
*** vakuznet has joined #openstack-ansible12:57
vakuznetjmccrory, please review https://review.openstack.org/#/c/612011/12:59
*** vakuznet has quit IRC13:02
*** ansmith has joined #openstack-ansible13:04
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible master: SUSE: Add support for openSUSE Leap 15  https://review.openstack.org/60436613:05
nicolasbockHi xgerman_ (IRC) are you there? I have an octavia question... Is the `vip_address` of the loadbalancer supposed to be on the `lbaas_mgmt` network?13:06
noonedeadpunkfolks, can you point me please to mechanism of adding new backend/frontend to haproxy? I'd like to add custom frontend there, and as for now found only one handler, but I do not fully understand how it works...13:09
*** fatdragon has joined #openstack-ansible13:10
odyssey4menoonedeadpunk: there is an haproxy_extra_services var for that13:12
suggestableHi guys. Trying to work out if it's possible to have per-host overrides in place for neutron_linuxbridge_agent_ini_overrides. If so, how? I saw mnaser replied to a mailing list question about this topic in the past, but the original question was not about per-host overrides.13:12
*** fatdragon has quit IRC13:15
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Use distro_install project-template  https://review.openstack.org/60843113:16
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_nova master: Switch LXC test job from xenial to bionic  https://review.openstack.org/61269413:16
odyssey4mesuggestable: this may be handy: https://evrard.me/group-and-host-variables-overriding-in-openstack-ansible13:17
noonedeadpunkodyssey4me: and how are default ones appear in haproxy? I'm just trying to configure masakari api service "manually" with existing half of the role, andlooking for mechanism to add it to haproxy13:17
odyssey4menoonedeadpunk: https://github.com/openstack/openstack-ansible/tree/master/inventory/group_vars/haproxy13:18
noonedeadpunkodyssey4me: oh, got it. thanks!13:19
noonedeadpunkit was exactly what I was looking for13:20
suggestableodyssey4me: Thank you. That's helpful for main services, but I'm trying to override one line in the linuxbridge_agent.ini file. I'm having to accommodate using different server vendors in the environment, and the NICs get different names. I'm trying to set the physical interface mappings.13:20
suggestableodyssey4me: I've used http://www.420web.co/openstack/msg39867.html to get the basic implementation working, but now need to expand it to allow for per-host overrides.13:21
odyssey4mesuggestable: so the var you'd need to set in the host/group vars is https://github.com/openstack/openstack-ansible-os_neutron/blob/master/defaults/main.yml#L17013:22
odyssey4mesuggestable: and you'd be using this mechanism: https://docs.openstack.org/openstack-ansible/rocky/reference/configuration/using-overrides.html#overriding-openstack-configuration-defaults13:22
suggestableodyssey4me: Thank you. Would this work, potentially? http://paste.openstack.org/show/9BJm2BnVR4sIg4w0HIIe/13:26
evrardjpfolks, I've +w the changes of odyssey4me for bumping ansible version (latest patch version) should you see issues, please let us know :)13:29
*** strattao has joined #openstack-ansible13:37
suggestableodyssey4me: Corrected, but am I right? I'm adding this to openstack_user_config.yml http://paste.openstack.org/show/yjBswaRQ245XYW2U4TTL/13:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Install required devel packages for source installs  https://review.openstack.org/61270413:45
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Only install the appropriate distro packages  https://review.openstack.org/61150813:46
odyssey4mesuggestable: why all_hosts?13:48
suggestableodyssey4me: That's the only bit we're not sure about. I need to set the overrides on all the hosts. Is there an alternative I should use? Perhaps infra_hosts: followed by compute_hosts: ?13:49
odyssey4mesuggestable: you already have each host in your openstack_user_config, right?13:50
suggestableodyssey4me: Correct.13:51
odyssey4meunder each host, you have the IP, so under that add 'container_vars:' as a heading, then the vars you want to set under them13:51
odyssey4mealthough if you're doing a universal override - then just set it in user_variables.yml13:52
odyssey4meas far as I can see in your paste, all the values are the same?13:52
suggestableodyssey4me: Would this affect containers, too? I only need to override the ini file on the machines running linuxbridge-agent.13:52
odyssey4mesuggestable: 'container_vars' is misleading - that applies to hosts/containers depending on the service that uses it and where it's running13:53
jungleboyjmbuil: Morning.13:53
suggestableodyssey4me: The values are the same for these particular hosts, as they're all the same OEM. The older machines we're migrating across to this new environment are from different vendors.13:53
odyssey4mesuggestable: ah, I see - ok, so it's only a subset of the hosts13:54
suggestableodyssey4me: Essentially, yes. When we add the older machines to the environment, we'll add their particular override to the openstack_user_config.yml file in the same way as we're doing here.13:54
odyssey4mesuggestable: ok, could you paste/gist your existing openstack_user_config, sanitised appropriately13:55
odyssey4mesuggestable: you can send it privately if you'd prefer not to make it too open13:58
*** Leo_m has joined #openstack-ansible13:58
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-os_swift master: vars: Do not pull devel packages on distro installations  https://review.openstack.org/61270913:59
*** _NiC has joined #openstack-ansible14:01
_NiCI'm looking at the openstack os_volume/os_server modules, and I'm trying to find a way to create a volume, set it bootable, and then create an instance using that volume. However I can't find a way to make the volume bootable, causing the os_server invocation to fail with "Block Device <id> is not bootable". Any hints?14:01
*** ThiagoCMC has joined #openstack-ansible14:02
ThiagoCMCHey guys, I'm facing a network problem with OSA Rocky, LinuxBridge+VXLAN, guests between different compute nodes can't ping each other! But they can ping guests on the same host, as well as the Internet... What's happening?14:03
ThiagoCMCI found this: https://access.redhat.com/solutions/130079314:03
ThiagoCMCAny solution?14:03
ThiagoCMCMy blades have 2 10G NIC cards (eno1 + eno2), in a bond0 channel -> bond0.12 (vlan for clan) -> br-vxlan on top of it.14:04
ThiagoCMCBasically, I did what it is documented here: https://docs.openstack.org/project-deploy-guide/openstack-ansible/rocky/targethosts.html14:05
ThiagoCMCI'm seeing the following message on `tcpdump -evnni tapxxxxx-xx` interface: "fa:16:3e:49:37:f0 > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.10.9 tell 10.10.10.23, length 28" :-(14:07
ThiagoCMCWhere 10.10.10.9 is guest A, compute A, and 10.10.10.23 is guest b, compute b. They can both `ping 8.8.8.8` but, not each other.14:08
xgerman_nicolasbock: you rang?14:08
jamesdentonThiagoCMC and they're pinging via the neutron router? is l2pop enabled?14:08
ThiagoCMCyes, via neutron l3, yes, l2pop enabled.14:09
jamesdentonand your compute nodes can ping each other via the bond0.12 interface, right?14:09
ThiagoCMCyes14:09
ThiagoCMCBond0 -> bond0.12 -> br-vxlan (IP 172.29.240.XX here on the bridge)14:10
jamesdentonWhat NICs? What OS?14:11
jamesdentonand - do you mind setting a static ARP inside the instances and see if the pings work then?14:12
ThiagoCMCNIC: Intel Corporation C610/X99 series chipset MEI Controller #1 (rev 05) / Ubuntu 18.04 everywhere (from maas to OSA and blades)14:14
ThiagoCMCHmm... I14:14
ThiagoCMCI'll try to set a static ARP! Didn't tried this...14:14
openstackgerritMichael Vollman proposed openstack/openstack-ansible master: Add os_manila role deployment plays  https://review.openstack.org/61271114:16
_NiCLooks like the openstack cli has a --bootable option to set a volume bootable, so from what I can tell, the support for that is missing in the os_volume module.14:17
jamesdentonThiagoCMC can you run 'ethtool -i <nic>14:19
jamesdentonfor the members of that bond?14:19
ThiagoCMC1 sec14:20
ThiagoCMCHere: http://paste.openstack.org/show/732827/14:20
jamesdentonthx14:21
ThiagoCMCThank you!14:22
jamesdentonwhat blade server is this?14:23
jamesdentonAlso, when you get a sec 'lspci -vvv | grep 03:00.0'14:25
ThiagoCMCSupermicro, Mainboard product: X10DRT-PT14:26
ThiagoCMClspci: 03:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10-Gigabit X540-AT2 (rev 01)14:26
jamesdentonk14:26
ThiagoCMCBIOS and BMC updated last month14:27
ThiagoCMCI'll try to add the static ARP entry now.14:28
jamesdentonk14:28
suggestablejamesdenton: odyssey4me sent me your way. Do you have time to help with an l3_ha issue?14:32
jamesdentoni'll do my best14:32
jamesdentonwhats up14:32
suggestablejamesdenton: When creating a router, using high availability, it stays down, with all HA agents in standby mode. When we create a router with HA mode off, it works just fine. Any ideas?14:32
suggestableAll other HA services appear to work just fine.14:33
jamesdentoni assume the router doesn't work in the first instance?14:33
suggestableIt does not, no.14:33
jamesdentonthe agent creates a special HA network for each tenant that is used for HA routers. It uses the default tenant network type14:34
jamesdentonit's possible the routers cant communicate on this network14:34
suggestableWe're using VXLAN internally, and flat to the outside.14:34
jamesdentoncan you spin up a new HA router, and enter the respective namespaces and see if the routers can communicate amongst themselves?14:35
jamesdentonis your vxlan working?14:35
ThiagoCMCjamesdenton, I manually added the ARP inside of each Instance, still doesn't work. I'll try to add static ARP at the hosts themselves...14:35
jamesdentonnaw, you shouldn'14:35
*** canori01 has joined #openstack-ansible14:35
suggestableWhen using non-HA mode, we can get connectivity right through to the Internet, so I believe VXLAN is indeed working.14:36
jamesdentonk lemme think about this for a sec. in a meeting at the moment14:36
jamesdentonoh right.14:36
jamesdentonwhat version/14:36
suggestablestable/rocky14:36
suggestableOn bionic14:36
jamesdentonk14:36
ThiagoCMCyep14:37
suggestableHow does one check that the routers can communicate amongst themselves? Can you log into a router and ping from it, like a hardware device would let you?14:37
jamesdentonIIRC, yes. pop into the namespace and there should be an interface starting with 'ha'. The IP is 169.254.x.x. Each namespace would have its own IP. Trying pinging between them,14:38
openstackgerritMerged openstack/openstack-ansible-ops master: fix osquery filesystem logging  https://review.openstack.org/61259814:39
ThiagoCMCjamesdenton, no rush...   =)14:41
*** skiedude has joined #openstack-ansible14:41
ThiagoCMCAt the KVM host, I'm not seeing that "who as ..." Missing ARP anymore, however, I noticed something...14:41
ThiagoCMC`tcpdump -evnni brq9d8ef233-4e` (bridge attached to the Instance that is pinging the other), I can see: "10.10.10.6 > 10.10.10.15: ICMP echo request, id 1376, seq 280, length 64"14:42
jamesdentonok yeah wiuth the static arp you won't see the arp request anymore14:42
ThiagoCMCHowever, `tcpdump -evnni br-vxlan` shows nothing! Neither bond0.12...14:42
ThiagoCMCThe packets aren't leaving the compute node...14:43
jamesdentonit will be encapsulated there, so you may need to show more of the packet14:43
ThiagoCMCBut `tcpdump -evnni br-vxlan` is totally silent!14:44
ThiagoCMCI thought that I would be seeing something...14:44
yetiszafovs-bridge?14:44
ThiagoCMCLinuxBridges+VXLAN14:44
ThiagoCMCGuests in diff computes can't ping each other.14:45
ThiagoCMC"physical_interface_mappings = external:bond0.10,vxlan:br-vxlan"14:45
jamesdentonok14:45
yetiszafHmm. I've had something similar last week and we were never able to see the traffic on the br-tun that the vxlans are attached to, but we could see it on the vxlan-interfaces themselves.14:45
jamesdentonso you definitely don't need to the vxlan:br-vxlan mapping. that's being treated like a vlan provider mapping14:45
openstackgerritMerged openstack/openstack-ansible-ops master: Fixes related to installation on Trusty  https://review.openstack.org/61242414:46
ThiagoCMCjamesdenton, Hmm...14:46
jamesdentonwhat does your network look like for the vxlanb net?14:46
jamesdenton'openstack network show' output14:47
suggestablejamesdenton: What's the correct syntax for pinging between the routers? We're trying with "ip netns exec router-<uuid> ping <IP address of other router>" and getting nothing.14:47
jamesdentonthat would be correct. can you provide 'ip netns exec router-<uuid> ip neighbor show' output?14:48
ThiagoCMCFirst one: http://paste.openstack.org/show/732830/14:49
suggestablejamesdenton: Lots of stuff. Apparently all IPv6, and all marked as STALE.14:49
suggestable(We're using IPv4)14:50
ThiagoCMCThe neighbor show: http://paste.openstack.org/show/732831/14:50
openstackgerritMerged openstack/openstack-ansible-ops master: Update the test scripts  https://review.openstack.org/61249314:50
*** aredinger has joined #openstack-ansible14:53
ThiagoCMCBTW, my network nodes (2) are bare_metal (yes), running at the same compute nodes (2). There are no neutron-*-agent containers here.14:54
jamesdentonThiagoCMC can you run that same ip neighbor show command on the infra node hosting the router?14:55
jamesdentonok yeah those containers are gone14:55
ThiagoCMCI don't want the neutron-*-agents within containers to simplify the bridge setup and try to minimize the bottlenecks.14:55
jamesdentonsuggestable you're on Rocky, too?14:56
ThiagoCMCyep14:56
ThiagoCMCBrand new deployment14:56
suggestableWe are indeed, jamesdenton14:56
ThiagoCMCOh, sorry. lol14:56
jamesdentonand these are both new deploys for ThiagoCMC and suggestable? not upgrades?14:56
suggestableCorrect. New deployment.14:56
ThiagoCMCNot upgrades14:56
ThiagoCMCfresh14:56
jamesdentonkk14:56
ThiagoCMCI'm using Ubuntu 18.04 everywhere, from MaaS, to OSA Ansible node and all targets.14:57
jamesdentoni haven't done a Rocky deploy, and i stay away from vxlan these days. But i can try a deploy and see if there's an issue14:57
jamesdentonsuggestable using 18.04 too14:57
jamesdenton?14:57
suggestableWe're using Bionic, too, so yes.14:57
jamesdentonkk14:57
ThiagoCMCsuggestable, you have the same problem as I do?14:58
suggestableThiagoCMC: L3HA router not working? If so, yes.14:58
jamesdentonit's a little different, but still related to vxlan14:58
ThiagoCMCI see14:58
benkohlodyssey4me: The log said that :8181/simple/pkg-resources/ returns a 404. Is that related to https://review.openstack.org/612001 ?14:59
suggestablejamesdenton: In our configs, we're overriding "physical_interface_mappings" to provider:eno2 to get non-HA routing working.14:59
suggestable(In linuxbridge_agent.ini)14:59
jamesdentonsuggestable can you post the 'openstack network show'14:59
jamesdentonfor both provider (external) and project networks?15:00
suggestablejamesdenton: http://paste.openstack.org/show/sYlv3nH0Gb2Brk3wjsDG/15:02
*** Leo_m_ has joined #openstack-ansible15:02
cloudnullodyssey4me ping - on python_venv_build we don't install `python-vritualenv` within that role. I'm trying to use the role kinda outside of the "normal" use case and am curious if this is intended.15:02
cloudnullwe expect that the package will be installed by deployer outside of that role itself?15:03
odyssey4mecloudnull: yep, the README states that cirtualenv is expected to be there already15:03
odyssey4methe reason for that is openstack_hosts installs it, and it doesn't make sense to keep installing it - better to be clear about expectations, and you get it there however you want before using the role15:03
odyssey4me*virtualenv15:04
cloudnullhwoarang so we need to install "nc" everywhere?15:04
hwoarangcloudnull: hmm? just on the keystone containers. but i already sent a patch for that15:04
odyssey4mecloudnull: nope, just the keystone role - there's a patch in flight already15:04
*** Leo_m has quit IRC15:04
*** hamzy has quit IRC15:04
cloudnullodyssey4me: ++15:04
cloudnullthanks !15:05
odyssey4mecloudnull: for the systemd_service role, is there a reason we're using 'simple' rather than 'forking'? With forking it checks the service start return code and reports that back... sindle always returns success15:05
odyssey4me*simple15:05
jamesdentonsuggestable thanks. i saw you had that override in place. that's just to use a different physical interface for the provider network on some hosts?15:05
logan-it looks like that task is delegated to the memcached container.. i don't think nc in keystone is going to help?15:05
suggestablejamesdenton: That's correct, yet. We're going to be migrating across some older machines into the cluster, and their NICs are named differently.15:06
odyssey4mecloudnull: so we're getting false success when restarting a service which is broken15:06
cloudnullodyssey4me nope, no reason. the simple service should not always return success, unless the process is executed with a dash in front of it.15:06
odyssey4melogan-: oh, good point - let's revisit that more closely15:06
jamesdentongot it. what you're doing should be fine. and i don't think affecting this at all. i will work on a Rocky/Bionic deploy and see if there's something going on with vxlan.15:07
logan-maybe delegation issues with connection plugin updates15:07
odyssey4mecloudnull: it definitely does return success to systemctl restart - although inspecting the status shows the fail15:07
suggestablejamesdenton: Thank you!15:07
cloudnullodyssey4me does "forking" return the correct code ?15:08
ThiagoCMCjamesdenton, maybe you can access my Rocky / Bionic env via teamviewer, it's fully isolated and fresh. I guess that it would save you some time!   ;-)15:08
odyssey4mewell, not sure if this may be better - but it seems the memcache flush is specifically a newton->ocata upgrade issue, so perhaps we should revert and just patch ocata?15:08
odyssey4mecloudnull: yep15:08
cloudnullif so, lets change it15:08
odyssey4mecloudnull: I'll post up some logs now - gimme a minute15:08
*** Leo_m_ has quit IRC15:09
*** Leo_m has joined #openstack-ansible15:09
*** strattao has quit IRC15:13
odyssey4mecloudnull: https://gist.github.com/odyssey4me/b8f6a13ffc48b99cc837761cf4e4b06415:14
odyssey4mecloudnull: https://serverfault.com/a/833237 gives a nice explanation why15:15
*** ThiagoCMC has quit IRC15:17
odyssey4mecloudnull: I can push a patch up to change the default in the role if you think that's best? Alternatively we'd have to patch all the roles which consume it.15:19
*** fatdragon has joined #openstack-ansible15:19
*** strattao has joined #openstack-ansible15:19
odyssey4meit looks to me like most just use the default15:19
openstackgerritMerged openstack/openstack-ansible-os_keystone master: vars: Add missing netcat package  https://review.openstack.org/61267315:21
mbuiljamesdenton: hello! I am trying to help jungleboyj but his topology is a bit different than what I have and I am not sure what is the best way forward, maybe you know better. He has a AIO deployment and neutron is using linux bridges. He has a private network and a flat public network connected through a router. The bridge mapping is flat:eth1215:22
*** ThiagoCMC has joined #openstack-ansible15:23
mbuiljamesdenton: as far as I know, eth12 is a veth connected to br-vlan-veth. This is how his bridges look like ==> https://www.irccloud.com/pastebin/LR36Tdup/15:23
jamesdentonyes, eth12 is, or was, the interface inside the neutron-agents-container15:24
mbuilajmesdenton: that AIO VM has one interface (eno3) which he is using to connect to the VM, so if we attach eno3 to br-vlan, he'll lose connectivity to it15:25
jungleboyjmbuil:  ens3.  :-)15:25
jamesdentonk - i'll have to circle back to this. tied up with a couple of other issues at the moment15:26
jungleboyjhttps://www.irccloud.com/pastebin/MLa7DL2h/15:26
mbuilyesterday, he tried to ping 8.8.8.8 from the VM connected to the private neutron network, and we were seeing ARP messages in br-vlan but of course those do not leave the VM as ens3 is not connected to br-vlan. I am not sure what are the best practices in this case to connect VMs to the internet15:26
openstackgerritKevin Carter (cloudnull) proposed openstack/ansible-role-systemd_mount stable/rocky: Add mount state rescue block  https://review.openstack.org/61273115:29
cloudnullodyssey4me makes sense.15:30
cloudnullsorry, just looking at a sample of roles. https://github.com/openstack/openstack-ansible-os_glance/blob/f075899894463807134a40674b4f5482dac4708a/tasks/glance_install.yml#L104-L13015:32
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-systemd_service master: Switch default service type to 'forking'  https://review.openstack.org/61273215:32
cloudnullit looks like we dont set the service type in the role15:32
cloudnullso changing it globally makes the most sense to me15:33
odyssey4mecloudnull: yep, so it takes the default15:33
cloudnull++15:33
odyssey4mehttps://review.openstack.org/612732 is up to do that15:33
openstackgerritMerged openstack/openstack-ansible stable/rocky: Update Ansible to 2.5.10  https://review.openstack.org/61240315:33
cloudnullodyssey4me https://github.com/openstack/ansible-role-systemd_service/blob/master/templates/systemd-service.j2#L2615:33
cloudnullwe need to change the service type check to `(service_type in ['simple', 'forking'])`15:34
odyssey4meah yeah, let me do a more thorough sweep15:34
cloudnull++15:34
*** ianychoi_ has joined #openstack-ansible15:36
*** hamzy has joined #openstack-ansible15:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-systemd_service master: Switch default service type to 'forking'  https://review.openstack.org/61273215:39
openstackgerritMerged openstack/openstack-ansible-repo_build master: SUSE: Add support for openSUSE Leap 15  https://review.openstack.org/61267115:39
*** ianychoi has quit IRC15:40
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Only install the appropriate distro packages  https://review.openstack.org/61150815:41
*** ivve has joined #openstack-ansible15:42
openstackgerritKevin Carter (cloudnull) proposed openstack/ansible-role-systemd_networkd master: Move link options to a link file  https://review.openstack.org/61248915:43
*** nsmeds has joined #openstack-ansible15:44
odyssey4mecould I get one more core review for https://review.openstack.org/612062 please15:46
d34dh0r53odyssey4me: lgtm15:46
openstackgerritMarkos Chandras (hwoarang) proposed openstack/openstack-ansible-rabbitmq_server master: SUSE: Make epmd listen to all addresses  https://review.openstack.org/61273615:48
mnasercloudnull, DimGR, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp, arbrandes, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__, colinmcnamara, thorst, adreznec, eil397,15:50
mnaserqwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking, mnaser, nicolasbock, jrosser, cjloader15:50
mnaseranyone around? :)15:51
spotz\o/15:51
odyssey4meo/15:51
hwoarangi am15:51
mnasercools15:51
hwoaranguntil the baby says otherwise15:51
mnaseri'll gladly run this one unless someone wants to volunteer15:51
mnaserahaha15:51
d34dh0r53o/15:51
mnaseri might not be able to run the entire hour in the place where i'm located currently though15:52
mnaserso if anyone wants to run it, i'll let them :) if not expect a MAYBE 5-10 minute disappearance as i relocate15:53
evrardjpthere are not too many bugs open today, just fyi15:53
mnaserits cause we ship flawless software15:53
d34dh0r53:)15:53
spotzheheh15:53
openstackgerritMerged openstack/ansible-role-systemd_mount master: Add mount state rescue block  https://review.openstack.org/61247615:55
evrardjpmnaser: sure!15:55
mnaserevrardjp: sure we ship awesome software or sure you can help run the meeting :p15:56
evrardjphas anyone updated the wiki of this week's highlights or topics?15:56
mnaseri see some berlin stuff15:57
evrardjphwoarang: jrosser you still have topics there15:57
spotzBerlin etherpad, sign up if you're coming so I can make the hangout!!15:57
evrardjpcloudnull: should we still discuss about unit file conflicts? It's been a few times now :)15:57
hwoarangi will wipe mine15:58
evrardjpmnaser: I don't mind, I prefer when you run it though. You're better at it.15:58
evrardjp:D15:58
cloudnullo/15:58
mnaserevrardjp: learned from the best, but i'd really appreciate this week.  it's been a rooough one here15:58
cloudnullwe can, though i've not spent much time finding more of the conflicts.15:58
*** skiedude has quit IRC15:59
evrardjpmnaser: ok will do then15:59
jrosserevrardjp: my topic was put in just too late and missed last week15:59
mnaserthanks i really appreciate it evrardjp15:59
evrardjpmnaser: no worries... team team team!15:59
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Add legacy os-infra_hosts group back  https://review.openstack.org/61273715:59
evrardjpso...16:00
evrardjphttps://www.youtube.com/watch?v=pGFGD5pj03M for those who don't know it16:00
evrardjpcloudnull, DimGR, d34dh0r53, hughsaunders, b3rnard0, palendae, odyssey4me, serverascode, rromans, erikmwilson, mancdaz, _shaps_, BjoernT, claco, echiu, dstanek, jwagner, ayoung, prometheanfire, evrardjp,16:00
evrardjparbrandes, scarlisle, luckyinva, ntt, javeriak, spotz, vdo, jmccrory, alextricity25, jasondotstar, admin0, michaelgugino, ametts, bgmccollum, darrenc, JRobinson__, colinmcnamara, thorst, adreznec, eil397,16:00
evrardjpqwang,nishpatwa_, cathrichardson, drifterza, hwoarang, cshen, ullbeking, mnaser, nicolasbock, jrosser, cjloader16:00
evrardjpmeeting time!16:00
nicolasbocko/16:01
evrardjp#startmeeting openstack_ansible_meeting16:01
openstackMeeting started Tue Oct 23 16:01:31 2018 UTC and is due to finish in 60 minutes.  The chair is evrardjp. Information about MeetBot at http://wiki.debian.org/MeetBot.16:01
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.16:01
*** openstack changes topic to " (Meeting topic: openstack_ansible_meeting)"16:01
openstackThe meeting name has been set to 'openstack_ansible_meeting'16:01
evrardjp#topic rollcall16:01
*** openstack changes topic to "rollcall (Meeting topic: openstack_ansible_meeting)"16:01
evrardjpo/16:01
d34dh0r53o/16:02
nicolasbocko/16:02
hwoarango/16:02
mnasero/16:02
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Add legacy os-infra_hosts group back  https://review.openstack.org/61273716:02
odyssey4meo/16:02
evrardjp#topic Last week highlights16:03
*** openstack changes topic to "Last week highlights (Meeting topic: openstack_ansible_meeting)"16:03
cloudnullo/16:03
evrardjphwoarang: added "Proposed distro job for ceph deployments and needs votes to get it in. Also restored lxc and ceph deployments for SUSE. Also needs votes :)"16:03
mnaserany link to that review?16:03
evrardjpjrosser 's highlight was "bionic timouts - fixed by removing repo cach server"16:03
hwoarangmy highlight is old i've removed it from the wiki. i think it's all done but i am still catching up. iirc some opensuse jobs have been reverted and/or switched to non-voting16:04
hwoarangso trying to figure out what happened :(16:04
evrardjpok sorry for that it seems my browser might have caused a reference to an old highlight then16:05
mnaserafaik we went nv after a lot of timeout and failure, some might be infra related and some being mirror related and some just being some os thing16:05
spotz\o/16:05
openstackgerritMerged openstack/openstack-ansible-os_swift master: RedHat: Use monolithic openstack-swift package  https://review.openstack.org/61239716:06
openstackgerritMerged openstack/openstack-ansible-os_swift master: zuul: Switch to distro package installation template  https://review.openstack.org/60605616:06
*** udesale has quit IRC16:06
evrardjpshould we discuss this further in the open discussion, or is there a bug referencing this?16:06
hwoarangopen discussion16:06
evrardjpgreat16:06
evrardjplet's move to bug triage first though16:06
evrardjp#topic bugtriage16:07
*** openstack changes topic to "bugtriage (Meeting topic: openstack_ansible_meeting)"16:07
evrardjpPlease see our usual etherpad16:07
evrardjphttps://etherpad.openstack.org/p/osa-bugtriage16:07
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/179807916:07
openstackLaunchpad bug 1798079 in openstack-ansible "Test environment example in openstack-ansible" [Undecided,New]16:07
*** aedc has quit IRC16:07
evrardjpyup16:08
evrardjpseems fair16:08
evrardjphttps://docs.openstack.org/openstack-ansible/rocky/user/test/example.html doesn't talk about bounds16:08
evrardjpbonds*16:08
odyssey4melooks like low hanging fruit16:08
evrardjpclassification?16:09
odyssey4melow/confirmed?16:09
*** aedc has joined #openstack-ansible16:09
evrardjplgtm16:10
evrardjpanyone new wants to take this low hanging fruit?16:10
nicolasbockI like low hanging fruit :)16:10
nicolasbockI could give it a try16:11
evrardjpthanks!16:11
evrardjp#link https://bugs.launchpad.net/openstack-ansible/+bug/179749916:11
openstackLaunchpad bug 1797499 in openstack-ansible "keystone default deploy test uses http not https" [Undecided,New]16:11
odyssey4meI'm pretty sure this was discussed at length last week in the channel.16:12
evrardjpnot reflected in the bug, although the bug was updated 3 times.https://bugs.launchpad.net/openstack-ansible/+bug/1797499/+activity16:13
openstackLaunchpad bug 1797499 in openstack-ansible "keystone default deploy test uses http not https" [Undecided,New]16:13
odyssey4meThe test mentioned is a localhost test against the local keystone service. It does not need to use https because the keystone container only listens on http by default.16:13
odyssey4meYeah, I didn't know there was a bug.16:13
odyssey4meThis is partially fixed by something that merged recently.16:13
odyssey4meThe actual issue the guy had was he was trying to use the same IP for the external and internal endpoints.16:14
evrardjpI think the title of the bug is very confusing and incorrect16:15
odyssey4meThe fact that http didn't work right when the settings were applied is now fixed in https://review.openstack.org/#/q/I823f2f949258157e306dbf80570abe53373da0c316:15
evrardjpI remember said patch odyssey4me16:16
evrardjpgood16:16
evrardjpso we can close this one then, as invalid?16:16
evrardjpif it's an incorrect classification we'll see this one open16:16
odyssey4methere is still an issue in that if keystone is set to use client <-https-> haproxy <-https-> keystone then it will fail mmiserably16:16
*** gyee has joined #openstack-ansible16:16
spotzYou should be able to use the same IP for internal and external endpoints, its the user_variables.yml.example16:17
odyssey4meI think we may already have a bug or two for that condition16:17
odyssey4mespotz: yes, but it never worked - but thanks to your patch it now will16:17
evrardjpspotz: I would not recommend it though16:17
evrardjpbut that's another topic16:17
odyssey4meMy suggestion to the reporter was to use different IP's for the different endpoints.16:17
odyssey4meThere was a report of success after that.16:18
evrardjpok16:18
evrardjpso do we agree on the invalid classification?16:18
evrardjplet's move to the open discussion16:19
evrardjp#topic open discussion16:20
*** openstack changes topic to "open discussion (Meeting topic: openstack_ansible_meeting)"16:20
evrardjphwoarang: before going to your topic I will prio on those already written there, as there are topics that might be skipped16:20
spotzlet me know your email to add your to the hangout if coming to summit!16:20
evrardjp(due to their recurrent/unupdated) nature16:21
*** FuzzyFerric has left #openstack-ansible16:21
evrardjpthat's indeed first topic16:21
*** suggestable has quit IRC16:21
evrardjp#link https://etherpad.openstack.org/p/OSA-berlin-planning16:21
evrardjpanything else on that topic?16:21
evrardjpok next16:22
evrardjp"Openvswitch configuration does not handle configuration properly on compute nodes. It should be configured with different interfaces on neutron agent container and compute hosts"16:22
evrardjpwasn't this one already present last week?16:22
jamesdentondo you have the bug for that?16:22
evrardjpjamesdenton: all I can see on said topic is two links: http://eavesdrop.openstack.org/irclogs/%23openstack-ansible/%23openstack-ansible.2018-07-30.log.html#t2018-07-30T15:33:48 and https://drive.google.com/file/d/1ebmQFx4w7W6G9KJGLuj82VDH5xOwXEhW/view16:23
jamesdentonthx16:24
evrardjpjamesdenton: but I think it's an old topic16:24
jamesdentoni did OVS install recently and don't recall any issues16:24
evrardjpok16:24
evrardjpthanks for the feedback there16:24
evrardjpTahvok is not there to talk about that, so let's move towards another topic then!16:24
evrardjpIntegration between os_tempest and tripleo validate-tempest16:24
evrardjpis there anything to say there?16:24
evrardjpanything new?16:25
evrardjpchandankumar and arxcruz?16:25
chandankumarevrardjp: I am working on https://review.openstack.org/59142416:25
evrardjpI think I saw a commit today, please review!16:25
chandankumardistro support16:25
evrardjpchandankumar: great!16:25
arxcruzevrardjp: well, i'll work to enable python-tempestconf16:25
arxcruzthis week/sprint16:26
evrardjpthat sounds very nice!16:26
chandankumarit is almost done few breaking changes16:26
evrardjpif you need help don't hesitate to ping16:26
arxcruzwe are still reformulating how our internal sprint works, but now things are get track16:26
*** hamzaachi has joined #openstack-ansible16:26
evrardjparxcruz: agile!16:26
chandankumarevrardjp: I am in sync with odyssey4me lots of stuff going on from that changes16:26
evrardjp:D16:26
arxcruzI should send the first wip this week16:27
arxcruzevrardjp: hehe, i wish :D16:27
evrardjpchandankumar: cool, you are in good hands! other cores, don't hesitate to help there :)16:27
mnaseri can do reviews if need be :)16:27
evrardjparxcruz: hahah. Thanks for the first WIP then! :)16:27
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-ops master: MNAIO: Add legacy os-infra_hosts group back  https://review.openstack.org/61273716:27
evrardjparxcruz: chandankumar do you mind if we keep this item on the agenda, so that I know to ping you, and we can track things next week?16:28
jungleboyjspotz: mbuil Opened a bug for AIO networking not working:  https://bugs.launchpad.net/openstack-ansible/+bug/179950716:28
openstackLaunchpad bug 1799507 in openstack-ansible "AIO deployment instances do not have network connectivity" [Undecided,New]16:28
chandankumarevrardjp: sure16:28
arxcruzevrardjp: sure16:28
evrardjpgreat16:28
arxcruzevrardjp: https://etherpad.openstack.org/p/openstack-ansible-tempest is our plan16:29
openstackgerritMerged openstack/openstack-ansible-tests stable/rocky: Update Ansible to 2.5.10  https://review.openstack.org/61240516:29
arxcruzso far16:29
arxcruzfeel free to comment :)16:29
evrardjpI will have a look :)16:29
*** vnogin has quit IRC16:29
evrardjpanything else on that topic?16:29
evrardjpIf not, I'd like to leave the mic to hwoarang16:29
hwoarangok16:30
hwoarangso what i would like to talk about is this job revert/non-voting situation. what we normally do when a job brakes one day, then we normally move it to non-voting16:30
hwoaranghowever, nobody remembers to bring it back to voting so the testing matrix is different every other day16:30
hwoarangand not sure what we can do about that16:31
hwoarangit all feels a bit random right now16:31
odyssey4mehwoarang: typically I try to push an immediate patch to revert the non-voting change, and then recheck it from time to time16:31
odyssey4meI find that works best16:31
evrardjpZuul v3 brought us the ability to be more flexible, but indeed I don't like this kind of expectations issues personally16:31
odyssey4mebut, quite honestly, all the job changes are making it hard for me to get work done to switch the role to use the integrated build properly16:32
hwoarangwe are fortunate enough to have distro people around so maybe we can reach out to them, give them like 2 days to fix stuff before we revert or something?16:32
evrardjpshould we decide a rule?16:32
hwoarangbecause moving from non-voting to voting normally has lower prio16:32
hwoarangso things can stay in non-voting for days or weeks16:32
odyssey4meit seems like the mirror issues are better now, that's definitely been more stable since nicolasbock upped the mirror refresh frequency16:32
evrardjphwoarang: which can lead to bad things16:32
odyssey4methe only issue now is that broken packages hit the repositories relatively often on master - especially for the distro builds16:33
hwoarangbut a broken master is expected from time to time :/16:33
hwoarangkeepig jobs as voting actually puts pressure on upstream people for a quick fix16:34
evrardjpmay I suggest we move towards a non-voting in master, and keep jobs stable on stable branches, until 2 days are passed without improvements?16:34
odyssey4meperhaps then distro builds should remain non-voting for master until after m3, then work gets done to make it all work right until the RC time frame? that sucks though because it puts tons of pressure on everyone working on that then16:34
odyssey4memore ideally, the work should be spread out16:34
odyssey4meso it'd be far nicer if we could use a more stable repo somehow16:34
odyssey4mesomething that gets testing before promoting16:34
evrardjpit maybe doesn't need to wait for m316:35
hwoarangthat might work since packages are changing quite often before M points16:35
hwoarangi am fine with stabilizing distro jobs after branching too16:35
odyssey4mecould we perhaps rather use an infra specific mirror that's updated only after the package updates are tested and validated?16:36
hwoarangwhat i dont like is this flip-flot because it's hard to keep track of it on all the repos16:36
odyssey4meyep, definitely agreed for that16:36
evrardjpalso the jobs should stay in checks and we should all together not merge things blindly if a -nv fails -- really check at the failure16:36
evrardjpodyssey4me: hwoarang agreed on the no flip-flop16:36
*** vakuznet has joined #openstack-ansible16:37
nicolasbockWe currently don't have that for Master odyssey4me16:37
odyssey4meok, but perhaps there'd be a way to implement it in openstack-infra?16:37
*** pvradu_ has quit IRC16:37
hwoarangso far it seems that distro jobs are causing the big trouble so we can make them non-voting. but the source based ones should remain voting and try to fix them instead of moving to non-voting. fixes normally arrive in less than 48h16:37
odyssey4mesome sort of periodic job to test the 'proposed' set, then if it passes copy the tested set into the infra mirrors16:38
hwoarangwell ideally upstreams should CI their packages ;p16:38
evrardjphwoarang: and publish them right? :p16:39
odyssey4methat seems fair to me - we aim to switch the distro jobs to voting before the new release, and all stable branches have them voting16:39
hwoarang:)16:39
evrardjpso could we sum it up?16:39
hwoarangthat sounds like the easier solution to keep 'master' happy16:39
hwoarang*easiest16:39
evrardjpmaster: source -- wait for 2 days, packages -- see what we can do with more stability + making it non voting until milestone x // stable branches -- always wait for 2 days16:39
hwoarangmore or less. at least check with the appropriate $distro channel for ETA16:40
hwoarangmaybe the problem is not known to them at all16:40
odyssey4mewell, centos and ubuntu don't update their packages without testing - so those distro jobs could perhaps remain voting the entire time... but it seems that opensuse is not testing the master packages after prepping them, so this may have to be suse specific?16:41
evrardjphwoarang: indeed16:41
odyssey4meright now the ubuntu distro installs are incomplete, so they'd need to be non-voting anyway16:41
odyssey4mebut I think centos is fine as far as I've seen16:41
evrardjpodyssey4me: and we are not running bleeding edge PPAs16:42
hwoarangodyssey4me: honestly i dont know how the suse cloud team is testing the packages16:42
odyssey4meevrardjp: yep, if that changes then we'd likely have to apply the same rule16:42
nicolasbockWe don't hwoarang (IRC) ;)16:42
* evrardjp whistles16:42
*** ThiagoCMC has quit IRC16:42
hwoarangah ok then16:42
nicolasbockWe don't spend a lot of time on master unfortunately16:42
evrardjpwe do but... OMG you don't want to know16:43
hwoarangso stabilizting after branching seems the most sensible thing for suse16:43
nicolasbockOnce it's branched it's a different story16:43
nicolasbockI think so16:43
hwoarangok then so be it16:43
nicolasbockUnless we can convince the right people to add some more vetting to master ;)16:43
odyssey4meok, so we're all happy for Ubuntu/SUSE distro jobs to remain non-voting until the RC period where work ramps up to get them working - any work done during the cycle is appreciated and advised, but it may break routinely16:43
evrardjpnicolasbock: isn't that what I suggested?16:43
evrardjp:p16:43
hwoarangodyssey4me: ok16:43
spotzhehe16:44
odyssey4memnaser: you happy with that?16:44
odyssey4meIt seems that CentOS is the model to follow for the rest. ;)16:44
nicolasbockYes, I don't think we need to convince you evrardjp (IRC) ;)16:44
openstackgerritMerged openstack/openstack-ansible-tests master: Update ansible to latest stable 2.6.x  https://review.openstack.org/61206216:44
evrardjpok are we done on this topic? A new bug was raised16:45
evrardjpfrom jungleboyj16:45
evrardjphttps://bugs.launchpad.net/openstack-ansible/+bug/179950716:45
openstackLaunchpad bug 1799507 in openstack-ansible "AIO deployment instances do not have network connectivity" [Undecided,New]16:45
mnaseri mean, i don't think it's ideal, but i dont think we can do better.16:45
odyssey4mehwoarang: FYI https://review.openstack.org/612391 is up, but isn't passing yet :/16:45
openstackgerritMerged openstack/openstack-ansible-os_glance master: Make glance cache management cron task idempotent  https://review.openstack.org/61206516:45
hwoarangodyssey4me: yes but it's not suse who is failing :)16:46
mnaserbut yes, packagers: please CI your stuff, that'd be awesomeeeEe16:46
spotzI can poke jungleboyj if we need him16:46
hwoarangso...16:46
jungleboyji am here16:46
evrardjpsorry I might have pulled jungleboyj a little too early in the conversation if you are all still talking about that16:47
evrardjp:p16:47
odyssey4mehwoarang: ah yes, I remember now :p16:47
evrardjpjungleboyj: can you give us your openstack_user_config.yml and eventual user variables?16:47
mnaseri mean, we cant force people to do things but it would REALLY be nice if they cared about down stream users, because if they break us we can break them and vice versa16:48
evrardjpor did you use gate_check_commit?16:48
mnaserbut yeah.  moving on.16:48
evrardjpsorry to keep the clock in there16:48
evrardjp:p16:48
evrardjpjungleboyj: I propose continue discussing your bug after the meeting, would that be okay for you? It would let you the time to publish said configuration or say how you reached said state (which process did you run for example)16:49
evrardjp(it's only in 10 minutes)16:49
evrardjpso for the last 10 minutes: Are there other topics ?16:49
jungleboyjevrardjp:  Sure.  Happy to provide any data you need.  :-)  Just let me know how I can help.16:50
evrardjpthanks jungleboyj16:50
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance stable/rocky: Make glance cache management cron task idempotent  https://review.openstack.org/61275216:51
jungleboyjthank you16:51
jrosserbionic/ceph another -nv thing that might fester. I have it working but only by upgrading ceph to mimic16:51
evrardjpoh16:51
evrardjpthat's quite something to track too16:51
evrardjpjrosser: do you have help there? like logan- ?16:51
jrosseri want a second opinion really16:52
jrosseruca and dl.ceph.com only seem to provide mimic for bionic anyway16:52
logan-o/16:52
logan-we should use mimic then16:52
odyssey4mejrosser: sounds sensible to me then16:52
jrosserand a discussion point might be if we prefer uca or ceph.com packages16:52
odyssey4mejrosser.doit()16:52
evrardjpsounds reasonable to me16:52
evrardjpodyssey4me: :D16:53
jrosseri prefer ceph.com becasue they work :)16:53
*** fghaas has quit IRC16:53
jrosseri can't make the uca ones pass gate check16:53
odyssey4mesounds fine to me, I just hate the extra moving part :p16:53
evrardjpjrosser: you mean the packages or the people?16:53
logan-^ ive hit bugs in radosgw with uca packages because they were pending SRE in launchpad16:53
logan-so i always use ceph.com packages personally16:53
mnaserisnt ceph mirrored in infra btw16:53
jrosserceph.com also provide debug symbol packages which is ++ good thing16:53
odyssey4memnaser: hammer is :p16:53
mnaserhttp://mirror.sjc1.vexxhost.openstack.org/ceph-deb-mimic/16:54
mnaserhttp://mirror.sjc1.vexxhost.openstack.org/ceph-deb-luminous/16:54
mnaserhttp://mirror.sjc1.vexxhost.openstack.org/ceph-deb-jewel/16:54
mnaser:D16:54
mnaserproblem solved?16:54
evrardjpmnaser: :D16:54
jrossermnaser: look at the env var passed into a job though16:54
evrardjpjrosser: agreed16:54
jrosserthats not so useful16:54
mnasernot sure, as in when to use uca and when not to?16:55
odyssey4meok, do we need to switch from mirroring to a reverse proxy instead? or perhaps expose the right env vars?16:55
* jrosser school half term so not managed to follow this up16:55
odyssey4meok, so we keep using the ceph packages, but need to switch to using the right mirror - and figure out how we get the right mirror path16:56
jrosseranyway - i have patches in for switching to mimic and also some general tidy up of the ceph server install16:56
mnaserodyssey4me: agreed16:56
odyssey4mejrosser: thanks for that!16:56
jrosserreviews please on those and i can chase them up when i'm back at work later in the week16:56
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_masakari master: Basic implementation of masakari-monitors  https://review.openstack.org/58462916:56
*** ThiagoCMC has joined #openstack-ansible16:57
odyssey4meFYI I've got a URL that may be useful for reviewers in a hurry: http://bit.ly/2NVPFCg16:57
odyssey4meThose are mergable, have passed CI, submitted by cores, and have no negative reviews.16:57
evrardjpthanks odyssey4me16:57
evrardjponly two minutes remaining for your last topics!16:57
* mnaser cant wait for gerrit shared dashboards16:59
evrardjpok thanks everyone!16:59
mnaserodyssey4me: will be our goto dashboard-ian16:59
evrardjpmnaser: agreed16:59
evrardjphaha16:59
evrardjp#endmeeting17:00
spotzhe does make it easy17:00
*** openstack changes topic to "Launchpad: https://launchpad.net/openstack-ansible || Weekly Meetings: https://wiki.openstack.org/wiki/Meetings/openstack-ansible || Review Dashboard: http://bit.ly/2xA1eZC"17:00
openstackMeeting ended Tue Oct 23 17:00:09 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)17:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-10-23-16.01.html17:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-10-23-16.01.txt17:00
openstackLog:            http://eavesdrop.openstack.org/meetings/openstack_ansible_meeting/2018/openstack_ansible_meeting.2018-10-23-16.01.log.html17:00
noonedeadpunkFolks, can you also take a look at https://review.openstack.org/#/c/611585/ ? These quotes really make pain for me...17:00
spotzodyssey4me and prometheanfire - please DM me your emails for the hangout, I think I have everyone else and as always feel free to add others17:00
logan-oh btw circling back on the ceph stuff jrosser, etc. please take a look at https://review.openstack.org/#/c/504094/ when you have time17:01
evrardjpcool17:02
evrardjpjungleboyj: what did you run on your machine to get to said state in https://bugs.launchpad.net/openstack-ansible/+bug/1799507 ?17:03
openstackLaunchpad bug 1799507 in openstack-ansible "AIO deployment instances do not have network connectivity" [Undecided,New]17:03
noonedeadpunklogan- that's great patch to be honest17:03
jungleboyjevrardjp: https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html17:03
noonedeadpunkI'm really lokking forward to see it merged)17:03
jungleboyjFollowed the AIO directions in there.17:04
*** pvradu has joined #openstack-ansible17:04
evrardjpokay so everything should work out of the box17:05
jungleboyjevrardjp:  That was my expectation.  :-)17:05
evrardjpwell gates tell me "positive" :D17:05
evrardjpdid you use a scenario? like ceph?17:05
jungleboyjI would love to debug/fix it but I get totally lost with Neutron.17:06
spotzevrardjp: If it's any help had one in Rackspace cloud, made an instance opened up the security groups and it couldn't ping out either17:06
jungleboyjOnly thing I added was Barbican.17:06
jungleboyjevrardjp:  That was why I opened the bug.  Not the only person seeing it.17:06
evrardjpspotz: that could be for a different reason though: nested clouds are not well supported due to port security on your cloud provider17:07
mnaserbtw: anyone has experience in how well the playbooks for galera and rabbit react to adding new nodes to scale out?17:07
evrardjpmnaser: to what I recall: bad17:07
evrardjp:D17:07
spotzmnaser: might be a cloudnull question17:07
mnaserdamnit17:07
evrardjpbut there is a process though17:07
evrardjpyou should have a look17:07
mnaseri remember reading it somewhere17:08
mnaserill dig17:08
evrardjphttps://docs.openstack.org/openstack-ansible/latest/admin/maintenance-tasks.html17:08
evrardjpIIRC17:08
evrardjpjungleboyj: It's a little late for me, but I'd be happy to help you. Could you add your env.d/conf.d/openstack_user_config.yml somewhere? Like paste.openstack.org?17:09
*** pvradu has quit IRC17:09
odyssey4meOK folks, time for me to conclude my day - chat again tomorrow!17:09
evrardjpI would like to make sure the inventory is alright after the inclusion of barbican. Network wise it should be okay though, as barbican has no networking impact (as far as I am aware)17:10
evrardjpjamesdenton: cloudnull could you help the kind Jay there? I should really stop working :p17:11
jungleboyjevrardjp:  Sure.  I will add that info to the bug.17:12
jamesdentonevrardjp of course. jungleboyj bear with me justa few min17:12
*** hamzy has quit IRC17:12
evrardjpgreat thanks everyone, and have all a nice day!17:12
jungleboyjHad reached out to jamesdenton but he was busy.17:12
*** hamzy has joined #openstack-ansible17:12
jungleboyjWasn't opening the bug to be irritating but thinking I might have a problem worth documenting.17:12
jungleboyj:-)17:12
jungleboyjjamesdenton:  Thanks.  Whenever you have time it is greatly appreciated.17:13
spotzhave a good night evrardjp and odyssey4me!17:14
openstackgerritMerged openstack/openstack-ansible-os_horizon master: Make horizon DB management cron task idempotent  https://review.openstack.org/61206617:14
jungleboyjevrardjp:  Night.  Hope you have some time to relax before bed.  :-)17:14
jamesdentonok jungleboyj - you have an AIO, flat provider network and a vxlan project network. instance is unable to get connectivity out thru the router?17:15
jungleboyjjamesdenton:  Yes sir.  I think that is accurate.  It was just a default AIO deployment on ubuntu 16.04 and the only external network interface on the node is ens3 at 192.168.1.8/2417:17
jamesdentonand did i read this is a single NIC deploy, too?17:17
jungleboyjYes.17:17
*** lbragstad is now known as lbragstad_f00d17:17
jamesdentoncool. are you just looking to get outbound connectivity from the VM?17:17
jungleboyjWell, Ideally I would like to know how to be able to assign 192.168.1.150 to 192.168.1.170 to the VMs.17:18
jungleboyjNeed to start with at least outbound connectivity though.17:18
jamesdentonoh i see17:18
jamesdentondo you have some kind of console to this server?17:19
jamesdentonyou'll want to move the mgmt IP off ens3 and onto br-vlan. Then move ens3 into br-vlan17:19
jungleboyjjamesdenton:  Yes, I can get to the console.17:20
jamesdentongreat. when you make those adjustments you'll lose access. but once ens3 is in the br-vlan bridge, and the mgmt IP has been moved from ens3 to br-vlan, ip connectivity should return17:20
*** hamzaachi has quit IRC17:21
jungleboyjOk.  So I have the console up.17:21
openstackgerritMerged openstack/openstack-ansible-plugins master: Fix issues with and enable Python 3 job  https://review.openstack.org/61190917:22
jamesdentonso the process would be something like: ip a d 192.168.1.8/24 dev ens3; ip a a 192.168.1.8/24 dev br-vlan; brctl addif br-vlan ens317:22
jungleboyjjamesdenton:  Ok.  Let me give that a try.17:22
openstackgerritMerged openstack/openstack-ansible-openstack_openrc master: Add default for openrc_os_domain_name and update example  https://review.openstack.org/61241617:24
jungleboyjjamesdenton:  You are amazing!?!17:27
jungleboyjjamesdenton:  Help me understand what happened here.17:27
jamesdentonwhats working17:28
*** hamzaachi has joined #openstack-ansible17:28
jungleboyjWell, I now have network access from the VM to the internet.17:28
jungleboyjI have not tried adding in the network configured to give IP addresses from my home network.17:28
jungleboyjBut this is the furthest I have been yet.17:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_horizon stable/rocky: Make horizon DB management cron task idempotent  https://review.openstack.org/61276317:29
cloudnulljungleboyj I'll be back online in 3-4 hours-ish. let me know if there's still issues with the environment or if I can help at all.17:29
* cloudnull just saw evrardjp ping17:29
jungleboyjcloudnull:  Will do.  Thanks.  I think I might be good.17:29
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-openstack_openrc stable/rocky: Add default for openrc_os_domain_name and update example  https://review.openstack.org/61276417:29
jamesdentonso, ens3 is needed to serve as the 'provider' interface linking neutron (virtual) networking to the physical infrastructure. Once the interface is a member of a bridge, any IP configured on it no longer works. So, if you move the IP to the bridge, you're good.17:30
cloudnullmnaser spotz: adding new nodes to galera / rabbitmq is all contained within the roles themselves. IE: they should be able to add without issues.17:30
jamesdentonthat was likely the missing piece from work with mbuil17:30
jungleboyjjamesdenton:  Yep,  mbuil almost had me there.  We didn't move the IP from ens3 to br-vlan though.  I think that was the missing piece.17:31
jungleboyjSo, let me reboot with the network for external IPs.17:31
jamesdentonmake sure you update netplan or interfaces file accordingly17:32
openstackgerritMerged openstack/openstack-ansible stable/rocky: Set an upper pin for Cython to please gnocchi  https://review.openstack.org/61200117:32
jamesdentonif you reboot the host the changes will get reverted17:32
jungleboyjOk, so is this an issue with the OSA deployment or is it working as expected?17:34
openstackgerritVadim Kuznetsov proposed openstack/openstack-ansible-os_octavia master: octavia container network address pool  https://review.openstack.org/61201117:34
jamesdentonwell, it's working as designed AFAIK.17:36
jamesdentonjust takes some tweaks for different environments17:36
vakuznetodessy4me I updated commit message ^17:36
*** fghaas has joined #openstack-ansible17:37
jungleboyjOk.17:37
jungleboyjWell, I will document what needed to be done in the bug so that others have a point of reference.17:37
jamesdentonk17:38
*** pvradu has joined #openstack-ansible17:42
*** vnogin has joined #openstack-ansible17:43
jrosserjungleboyj: the AIO is primarily a testing tool used in CI where there is only one IP assigned for the VM it runs in17:45
*** lbragstad_f00d is now known as lbragstad17:45
jrosserit sets up a flat network here, 'untagged' on br-vlan https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_networking.yml#L124-L13417:45
jrosserand then sets up some NAT here https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/tasks/prepare_networking.yml#L17517:46
jungleboyjjrosser:  Gotcha so I am kind-of mis-using the AIO then.17:46
jrosserperhaps - you just need to undo/replace some of what the AIO setup has done17:46
jrosserbringing your wider network onto br-vlan has done just that17:47
jungleboyjjrosser:  Ok.  Gotcha.17:48
jrosserthe network you saw the other day was from tempest testing, thats all defined here https://github.com/openstack/openstack-ansible-os_tempest/blob/master/defaults/main.yml#L11617:48
openstackgerritMerged openstack/ansible-role-systemd_mount stable/rocky: Add mount state rescue block  https://review.openstack.org/61273117:52
*** pvradu has quit IRC17:53
jungleboyjjrosser: jamesdenton  Looks like everything is working now with that change.17:57
jungleboyjThank you so much!17:57
jamesdentonnice work.17:58
*** hamzy has quit IRC17:59
jungleboyj:-)  You guys did the nice work.  ;-)17:59
*** hamzy has joined #openstack-ansible18:00
openstackgerritJimmy McCrory proposed openstack/openstack-ansible master: Use file module instead of shell to remove roles  https://review.openstack.org/61276718:00
*** hamzaachi has quit IRC18:00
*** ThiagoCMC has quit IRC18:01
*** vnogin has quit IRC18:02
openstackgerritMerged openstack/ansible-role-systemd_mount master: Set container_name as host_var in test inventory  https://review.openstack.org/61234218:02
*** skiedude has joined #openstack-ansible18:07
openstackgerritMerged openstack/openstack-ansible-ops master: MNAIO: Add legacy os-infra_hosts group back  https://review.openstack.org/61273718:19
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Include dependency packages in distro install list  https://review.openstack.org/61261418:22
*** hamzy has quit IRC18:23
*** hamzy has joined #openstack-ansible18:24
*** fghaas has quit IRC18:24
*** strattao has quit IRC18:25
*** aredinger has quit IRC18:26
*** aredinge1 has joined #openstack-ansible18:26
*** strattao has joined #openstack-ansible18:27
*** irclogbot_0 has joined #openstack-ansible18:35
openstackgerritAntony Messerli proposed openstack/openstack-ansible master: Adds option to enable ara during upgrades  https://review.openstack.org/61277518:36
skiedudeTrying out ubuntu hosts intead of Centos. Running the setup-hosts playbook went just fine, but setup-infrastructure fails on setting the state in haproxy. Trying to manually start haproxy shows cannot bind to socket errors18:37
skiedudeI've commented out everything except the stats part of the config, and it still can't bind it to port 9000, although I don't see anything else using that port18:37
*** chandankumar is now known as chkumar|off18:37
nicolasbockHi. I am trying to get Octavia to work on an AIO based off of master. I found that the amphora is not starting up and is failing with what looks like some ugly kernel backtrace.18:39
nicolasbockIf it helps I have a screenshot of at least the last part of that backtrace18:39
*** ThiagoCMC has joined #openstack-ansible18:52
*** chhagarw has quit IRC18:57
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_masakari master: Basic implementation of masakari-monitors  https://review.openstack.org/58462919:02
skiedudeTo fix my haproxy, I had to add net.ipv4.ip_nonlocal_bind=1 to /etc/sysctl.conf19:03
skiedudenot sure why, or if its supposed to be in there by default19:03
jrosserskiedude: it should be if the 'when' on this task is true https://github.com/openstack/openstack-ansible-haproxy_server/blob/master/tasks/haproxy_post_install.yml#L1619:11
skiedudewhich is only true if the groups.haproxy|length > 119:13
skiedudewhich right now I'm running single infra, so ist not19:13
skiedudeI suppose on a single infra host, I can just make the internal/external_LB ips the ip of the infra host?19:13
skieduderather than using external IPs which would be mnore used in multiple host setup19:14
*** cshen has quit IRC19:15
*** cshen has joined #openstack-ansible19:15
jrosserinternal/external vip should be different IP otherwise you'll probably have all sorts of trouble later19:17
skiedudebut its only going to enable that bind option if I have multiple haproxy hsots19:22
skiedudeso whats the setup for a single infra node19:23
*** pcaruana has quit IRC19:23
*** chandraprakashDw has quit IRC19:26
skiedudeI guess according to https://docs.openstack.org/openstack-ansible/queens/user/test/example.html I just manually bind the external ip in the interfaces conf19:29
skiedudewhen on a single infra19:29
openstackgerritMerged openstack/openstack-ansible stable/rocky: Fix rgw playbook pre_tasks  https://review.openstack.org/61161119:31
openstackgerritMerged openstack/openstack-ansible stable/rocky: scripts: sources-branch-updater: Make regexp for master branch stricter  https://review.openstack.org/61090219:35
*** strattao has quit IRC19:38
*** fghaas has joined #openstack-ansible19:42
*** lbragstad has quit IRC19:50
*** lbragstad has joined #openstack-ansible19:53
*** openstackgerrit has quit IRC20:06
*** aredinge1 has quit IRC20:07
*** spatel has joined #openstack-ansible20:12
spatelcloudnull: ^^20:12
spatelyou there?20:13
spatelI want to add some custom stuff in /etc/rsyslog.d/51-remote-logging.conf  file where i should edit them ?20:13
spatelI am setting up graylog with openstack and want to add some extra stuff in that file20:14
jrosserspatel: have you seen this? https://github.com/openstack/openstack-ansible-rsyslog_client/blob/master/defaults/main.yml#L66-L8320:29
*** nsmeds has quit IRC20:30
spatelnice! let me look20:30
*** ansmith has quit IRC20:37
*** hamzy has quit IRC20:40
*** hamzy has joined #openstack-ansible20:40
*** ivve has quit IRC20:42
*** nsmeds has joined #openstack-ansible20:45
*** spatel has quit IRC20:52
*** aredinger has joined #openstack-ansible21:03
*** aredinger has quit IRC21:07
*** priteau has quit IRC21:09
*** FuzzyFerric has joined #openstack-ansible21:11
*** smoke87 has quit IRC21:32
*** strobelight has quit IRC21:32
*** spsurya has quit IRC21:38
*** vakuznet has quit IRC21:44
*** mmercer has joined #openstack-ansible21:44
*** aredinger has joined #openstack-ansible22:03
*** skiedude has quit IRC22:05
*** aredinger has quit IRC22:07
*** priteau has joined #openstack-ansible22:09
*** priteau has quit IRC22:14
*** vnogin has joined #openstack-ansible22:16
*** ansmith has joined #openstack-ansible22:29
*** openstackgerrit has joined #openstack-ansible22:31
openstackgerritAndy Smith proposed openstack/openstack-ansible master: Add qdrouterd role for oslo.messaging backend deployment  https://review.openstack.org/57026122:31
*** vnogin has quit IRC22:43
*** cshen has quit IRC22:45
*** vnogin has joined #openstack-ansible22:55
*** hamzy has quit IRC22:59
*** aredinger has joined #openstack-ansible23:02
*** threestrands has joined #openstack-ansible23:02
*** tosky has quit IRC23:02
*** aredinger has quit IRC23:07
*** vnogin has quit IRC23:29
*** vnogin has joined #openstack-ansible23:30
*** FuzzyFerric has quit IRC23:34
ThiagoCMCcloudnull, hey man, I'm trying to use nspawn but, the TASK "nspawn_container_create : Start new container (enable)" is failing, error: http://paste.openstack.org/show/732925/ - any idea?23:34
ThiagoCMCThere are 4 nspawn containers that failed to start.23:35
ThiagoCMCFresh OSA Rocky23:35
*** idlemind has quit IRC23:38
ThiagoCMCFull Ansible output of TASK: "nspawn_container_create : Start new container (enable)" -> http://paste.openstack.org/show/732926/23:39
*** vnogin has quit IRC23:41
*** vnogin has joined #openstack-ansible23:42
*** gyee has quit IRC23:46
ThiagoCMCBoth cinder_api_container and glance_container failed to start under nspawn.23:54
ThiagoCMCFresh deployment OSA/Rocky23:54
ThiagoCMCTrying a second run of containers-deploy.yml...23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!