Friday, 2019-02-15

*** hwoarang has quit IRC00:03
*** hwoarang has joined #openstack-ansible00:06
*** markvoelker has joined #openstack-ansible00:09
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Fix dashboards and possible port conflicts  https://review.openstack.org/63708200:15
*** macza has joined #openstack-ansible00:27
*** macza has quit IRC00:32
*** gyee has quit IRC00:41
*** markvoelker has quit IRC00:42
prometheanfirecloudnull: you running a 4.20 kernel?00:46
prometheanfiresee if you can run a 'iptables-save -c' for me00:47
prometheanfireprobably a gentoo regression, but maybe not00:47
cloudnullprometheanfire I am01:14
* cloudnull needs to go get on my laptop01:14
prometheanfirecloudnull: seems like a kconfig issue01:15
*** ArchiFleKs has quit IRC01:35
*** zul has quit IRC01:37
openstackgerritMerged openstack/openstack-ansible-ops master: Fix dashboards and possible port conflicts  https://review.openstack.org/63708201:38
*** markvoelker has joined #openstack-ansible01:39
*** hwoarang has quit IRC01:43
*** hwoarang has joined #openstack-ansible01:44
*** ArchiFleKs has joined #openstack-ansible01:53
*** hwoarang has quit IRC01:54
*** hwoarang has joined #openstack-ansible01:55
*** markvoelker has quit IRC02:12
*** KeithMnemonic has quit IRC02:18
*** KeithMnemonic has joined #openstack-ansible02:19
openstackgerritKevin Carter (cloudnull) proposed openstack/openstack-ansible-ops master: Increment nginx check port  https://review.openstack.org/63711502:44
*** partlycloudy has joined #openstack-ansible02:49
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Use the new services names for sfc  https://review.openstack.org/62221602:57
openstackgerritJames Denton proposed openstack/openstack-ansible-ops master: Correct formatting in README.rst  https://review.openstack.org/63711703:06
*** markvoelker has joined #openstack-ansible03:09
partlycloudyhello folks, i need a little help to follow through the OVS scenario documented at https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-openvswitch.html.03:10
jamesdentonsure, whatcha got03:10
partlycloudythanks james! i am confused about br-vlan and br-provider. is the br-provider in ovs the same as br-vlan in the linuxbridge version?03:11
jamesdentonahh ok03:12
jamesdentonso, i agree that the docs need to be updated in that regard. It used to be that the architecture called for something like br-provider (ovs) -> br-vlan (lxb) -> eth103:13
jamesdentonBUT, now we can avoid br-vlan altogether03:14
jamesdentonso you can ignore that bit03:14
jamesdentonIf you look midway down where it gives the example of br-provider and br-provider2. Those are OVS bridges that the playbooks would create on your behalf. and the 'network_interface' defined there would be an available interface on your system, like eth1 or bond1, that the playbooks would connect to the respective bridge on your behalf03:15
*** priteau has joined #openstack-ansible03:16
partlycloudyoh i see! Then, what is the relation between br-vxlan and br-tun? is it br-tun -> br-vxlan?03:16
jamesdentonwell, not exactly. Neutron will create br-tun and connect it to br-int. Any 'tunnel' established between hosts will use the address configured on br-vxlan as the VTEP address03:17
jamesdentonso br-vxlan -> eth1.100 (dedicated vlan for overlay traffic) and the VTEP addr is configured on br-vxlan. When OVS passes vxlan traffic, it will traverse eth1.100 via br-vxlan (as thats where the IP is configured)03:18
*** shyamb has joined #openstack-ansible03:19
jamesdentonboth br-vxlan and br-vlan bridges are important when neutron agents are deployed in containers, as the LXC containers connect to those bridges -> physical infrastructure. But as of Queens(?) the agents run on bare metal by default instead of LXC, so the bridges don't serve the same role03:20
*** priteau has quit IRC03:20
partlycloudyso for example, tunnel traffics would go from br-int -> br-tun -> br-vxlan (VTEP)?03:21
jamesdentonslowly trying to update the docs to reflect that.03:21
jamesdentonyes, but the connection from br-tun to br-vxlan is logical for our sake. OVS is using the IP configured on br-vxlan - it could just as easily be configured on eth1.100. BUT there is logic in the playbooks, i believe, that may expect it to be configured on br-vxlan so it's best to just leave it there03:23
jamesdentonThis was recently added, and may help: https://docs.openstack.org/openstack-ansible/latest/user/network-arch/example.html03:24
jamesdentondoesn't call out OVS specifically, but gets close03:24
partlycloudythat was a great help! thank you james!03:26
jamesdentonSure! Please let me know how the docs can be improved03:26
partlycloudyso with ovs, it is not necessarily to create br-vlan, if neutron agent is running on metal, right?03:27
jamesdentonthat should be the case, yes03:27
jamesdentonWhat version are you deploying?03:28
partlycloudyi see. i was pulling my hairs off today… now i'm clear! :-)03:28
*** shyam89 has joined #openstack-ansible03:28
jamesdentonsorry bout that!03:28
partlycloudytag 18.1.303:28
jamesdentonok - just be aware that 'latest' docs usually reflect the master branch (v19 in this case)03:29
jamesdentonI cannot guarantee what's on that page will work w/o issue on v1803:30
jamesdentonso the Rocky docs are here: https://docs.openstack.org/openstack-ansible-os_neutron/rocky/app-openvswitch.html03:30
jamesdentonbut give the latest ones a go03:30
jamesdentonand if you do, be sure to add a 'container_interface: "<ethX>"' to the br-provider blocks that you create03:30
jamesdentonthe eth interface you specify won't be used but will need to be there. eth100 is fine03:31
*** shyamb has quit IRC03:31
partlycloudycopy that! thank you so much!03:33
jamesdentongood luck! feel free to pop in - i'll be around tomorrow03:34
partlycloudythank you James. you've been a great help!03:36
*** markvoelker has quit IRC03:43
*** shyam89 has quit IRC03:47
*** dave-mccowan has quit IRC04:03
jamesdentoncloudnull you around?04:11
cjloaderhttps://review.openstack.org/#/c/636992/04:27
cjloadercan we get some eyes? cloudnull mnaser04:28
cjloaderodyssey4me ^04:28
cjloaderGJ mgariepy for fixing the master bootstrap issue =)04:29
cjloaderty mnaser04:30
cjloaderd34dh0r53 ^04:38
openstackgerritMerged openstack/openstack-ansible-ops master: Increment nginx check port  https://review.openstack.org/63711504:39
*** markvoelker has joined #openstack-ansible04:40
*** markvoelker has quit IRC05:12
*** pabelanger has quit IRC05:15
*** hwoarang has quit IRC05:21
*** hwoarang has joined #openstack-ansible05:26
*** shyam89 has joined #openstack-ansible05:30
*** priteau has joined #openstack-ansible05:39
*** priteau has quit IRC05:43
*** shyam89 has quit IRC06:06
*** shyam89 has joined #openstack-ansible06:07
*** markvoelker has joined #openstack-ansible06:09
*** markvoelker has quit IRC06:43
*** shyam89 has quit IRC06:49
*** shyam89 has joined #openstack-ansible06:55
*** kopecmartin|off is now known as kopecmartin07:15
*** shyam89 has quit IRC07:23
*** shyam89 has joined #openstack-ansible07:25
*** chkumar|pto is now known as chkumar|ruck07:29
*** markvoelker has joined #openstack-ansible07:40
fnpanichi07:44
fnpanicdo i need to do something more then setting setup-ara true in bootstrap-ansible.sh to get it setup?07:47
fnpaniclooks like ara-manage is not on the system07:47
*** hamzaachi has joined #openstack-ansible07:51
*** shyam89 has quit IRC07:57
openstackgerritChandan Kumar proposed openstack/openstack-ansible-os_nova master: Use venv_packages_to_symlink to symlink to import libvirt-python  https://review.openstack.org/63347408:01
*** hamzaachi_ has joined #openstack-ansible08:04
*** hamzaachi has quit IRC08:05
*** KeithMnemonic has quit IRC08:08
*** KeithMnemonic has joined #openstack-ansible08:08
*** markvoelker has quit IRC08:12
*** rgogunskiy has joined #openstack-ansible08:16
*** tosky has joined #openstack-ansible08:38
*** shyam89 has joined #openstack-ansible08:44
*** aedc has joined #openstack-ansible08:59
*** DanyC has joined #openstack-ansible09:08
*** markvoelker has joined #openstack-ansible09:10
*** shardy has joined #openstack-ansible09:11
*** priteau has joined #openstack-ansible09:19
odyssey4mefnpanic that will install it into the ansible runtime venv, not the system - so if it's there, it should be in /opt/ansible-runtime/bin09:25
odyssey4meyou can activate the venv to use it more easily: source /opt/ansible-runtime/bin/activate; ara-manage ...09:25
odyssey4mejrosser so we have too many file watches in the repo container then?09:28
odyssey4meCan we overcome that, or do we need to look at removing pypiserver?09:29
jrosserodyssey4me: pypi-server seems to consume tons09:32
*** DanyC has quit IRC09:32
jrosserand i expect that as the number of upgrades you do builds up the contents of the repo server gets larger and larger (?)09:32
*** DanyC has joined #openstack-ansible09:33
*** hamzaachi_ has quit IRC09:36
*** ArchiFleKs has quit IRC09:38
odyssey4mejrosser probably, so now we have both lsync and pypiserver chewing up the watches :/09:40
jrosserit seems surprisingly hard to find out what process has a watch on which files to understand what is happening09:40
jrosserbut i expect this may explain a number of cases of failed lsync09:40
odyssey4meok, I think most of the scaffolding is in place for us to back out of using pypiserver if we need to for rocky - that's what I'm planning anyway for master... however, that would be a bit of a distraction - can we work around it for now to buy me some time to figure it out?09:41
odyssey4meperhaps for now we can just do a known issue release note, explaining the issue and how to work around it09:41
jrosserjust making the number of inotify large like my patch does should be totally aqequate for now09:41
jrosser*adequate09:41
jrosserit is a shame that the number of inotify is a system wide thing and exhasting them from one container will hose the whole system09:42
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_gnocchi master: Convert systemd services to common role(s)  https://review.openstack.org/63715509:42
*** markvoelker has quit IRC09:42
odyssey4memgariepy cjloader I've provided feedback in https://review.openstack.org/636992 - good attempt, but we can keep the host cleaner.09:43
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_gnocchi master: Convert systemd services to common role(s)  https://review.openstack.org/63715509:43
*** ArchiFleKs has joined #openstack-ansible09:48
jrosserodyssey4me: another upgrade related thing i see in an environment that has gone from Q->R is that the repo build fails every time because there is still some remnant of molterniron hanging aroung which it tries to git clone, and fails09:50
odyssey4mejrosser hmm, that's odd - https://github.com/openstack/openstack-ansible/commit/e94c56ebe56559a2b2940cd2ef5378378905ac5e should be taking care of getting rid of the role, and it should no longer be in the py_pkgs lookup09:52
jrosseri'll circle back on that next week and get the actual error in a paste, i've not really dug into it yet09:52
odyssey4mejrosser could you type up a known issue for the issue with the repo container to accompany the patch you pushed up?09:55
jrosserah right, yes and the ansible log should say if that role removal playbook got run09:55
odyssey4mea release note, I mean09:55
jrosserin a seperate patch, or to add to that?09:55
odyssey4meseperate patch is fine for master, given it's merged - but for the backports we should perhaps combine them09:55
jrosserok sure i'll take a look later09:55
fnpanicodyssey4me: /opt/ansible-runtime/bin has no ara-manage10:13
fnpanicmaybe i made a mistake10:13
fnpanici did change this in the bootstrap script10:14
fnpanicexport SETUP_ARA=${SETUP_ARA:-"True"}10:14
*** hamzaachi has joined #openstack-ansible10:14
fnpanici guess this is wrong then10:14
odyssey4mefnpanic are you sure that it should - the ARA version being used may not have that command10:14
fnpanicoh10:14
fnpanicok10:14
*** shyam89 has quit IRC10:15
odyssey4mefnpanic if you want to see whether it's installed, activate the venv and do 'pip freeze | grep ara'10:16
CeeMac_morning10:20
*** CeeMac_ is now known as CeeMac10:20
fnpanicmorning10:20
fnpanicnope10:21
fnpanicparamiko==2.4.110:21
CeeMacI ran into a small issue running the minor upgrade yesterday. Is there a specific requirement for which roles tp update in what order if I wanted to be more controlled about it?10:21
*** DanyC has quit IRC10:23
*** DanyC has joined #openstack-ansible10:23
odyssey4mefnpanic no ARA? if it's not there, then it seems there may be a bug in the implementation10:30
*** priteau has quit IRC10:33
odyssey4mefnpanic perhaps when antonym comes online he can share whether he's verified that it works10:33
odyssey4meit does appear to be working in our gate tests though10:33
*** DanyC has quit IRC10:37
fnpanicmhh ok so as i said it is most likely my mistake10:38
*** markvoelker has joined #openstack-ansible10:39
*** DanyC has joined #openstack-ansible10:43
fnpanici wanted to setup octaivua10:46
fnpanicit added the provider network according to this doc10:46
fnpanichttps://docs.openstack.org/openstack-ansible-os_octavia/rocky/configure-octavia.html#setup-a-neutron-network-for-use-by-octavia10:46
fnpanicit is named octavia like in the example10:46
fnpanicbut afterwards it states in the docs:10:47
fnpanicThere are a couple of variables which need to be adjusted if you don’t use lbaas for the provider network name and lbaas-mgmt for the neutron name.10:47
fnpanicso is just the example wrong?10:47
fnpanicalso according to this doc octaiva is setup different10:47
fnpanichttps://docs.openstack.org/openstack-ansible-os_neutron/rocky/configure-network-services.html#load-balancing-service-optional10:48
openstackgerritMerged openstack/openstack-ansible-os_nova master: Use venv_packages_to_symlink to symlink to import libvirt-python  https://review.openstack.org/63347410:56
*** shyamb has joined #openstack-ansible11:06
*** DanyC has quit IRC11:07
*** shyamb has quit IRC11:12
*** markvoelker has quit IRC11:13
odyssey4mefnpanic that would be for xgerman or johnsom to comment11:21
*** shyamb has joined #openstack-ansible11:37
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Remove glance_developer_mode and glance_venv_download  https://review.openstack.org/62034011:42
phasespaceIn an upgrade from pike to queens the hostname of the hypervisors are changed as to add '.openstack.local'. This breaks live migration for old instances. The node column of the instances table in the nova database will then include '.openstack.local' for new instances, but not old ones. When trying to live migrate old instances it fails because it apparently can't find the hypervisor named 'hostname' when it expects11:57
phasespace'hostname.openstack.local'.11:57
*** electrofelix has joined #openstack-ansible11:57
*** shyamb has quit IRC11:59
*** shyamb has joined #openstack-ansible11:59
*** hamzaachi has quit IRC11:59
*** hamzaachi has joined #openstack-ansible11:59
*** markvoelker has joined #openstack-ansible12:10
*** kmadac has joined #openstack-ansible12:12
odyssey4mephasespace could you register a bug for that? that's not good12:27
jamesdentonmornin'12:29
*** tosky has quit IRC12:30
chkumar|ruckodyssey4me: is there a way to run this job locally http://logs.openstack.org/94/630694/12/check/openstack-ansible-deploy-aio_metal_heat-centos-7/062e8b9/ ?12:33
phasespaceodyssey4me: will do12:33
chkumar|ruckodyssey4me: http://logs.openstack.org/94/630694/12/check/openstack-ansible-deploy-aio_metal_heat-centos-7/062e8b9/logs/openstack/aio1-utility/stestr_results.html and this job is creating tempest venv 18.0.0 any idea why?12:33
odyssey4mechkumar|ruck yes - clone openstack/openstack-ansible, then export SCENARIO=aio_metal_heat, then run through the basic AIO steps12:34
*** nyloc has quit IRC12:34
odyssey4mechkumar|ruck more details here: https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html12:34
*** nyloc has joined #openstack-ansible12:35
*** shyamb has quit IRC12:35
*** shyamb has joined #openstack-ansible12:36
*** tosky has joined #openstack-ansible12:36
odyssey4mechkumar|ruck the version there is probably because that was the last tag implemented on the master branch? the version now comes from https://github.com/openstack/openstack-ansible/commit/c44fbcbbe3b6d88e9393503e92ac670f9e81114812:38
phasespaceodyssey4me: https://bugs.launchpad.net/openstack-ansible/+bug/181603712:40
openstackLaunchpad bug 1816037 in openstack-ansible "Pike->Queens upgrade adds ".openstack.local" to hypervisor hostname breaking live migration for old instances" [Undecided,New]12:40
odyssey4mephasespace thanks, hopefully someone will pick that up in the next triage session - or before12:42
*** markvoelker has quit IRC12:42
odyssey4meI'm surprised that hasn't come up in the work antonym has done - or maybe it has and it's not been resolved yet.12:42
*** miloa has joined #openstack-ansible12:45
openstackgerritMarc Gariépy (mgariepy) proposed openstack/openstack-ansible master: Use ansible-runtime to run setup.py  https://review.openstack.org/63699212:50
mgariepyodyssey4me, ^^12:50
odyssey4memgariepy mind if I tweak the commit msg?12:52
mgariepynot at all :D12:52
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Use runtime venv to determine OSA version  https://review.openstack.org/63699212:55
odyssey4memgariepy I think ^ will please jrosser  more than the previous PS :)12:56
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Use runtime venv to determine OSA version  https://review.openstack.org/63699212:56
odyssey4meapologies for the last iteration - the extra whitespace was bugging me :p12:57
odyssey4methanks mgariepy !12:58
*** shyamb has quit IRC13:03
mgariepythanks to you :D13:04
mgariepywhitespace are the worst.13:04
mgariepybut not as bad as tabs hha13:05
openstackgerritMichael Vollman proposed openstack/openstack-ansible-os_neutron master: Avoid distro installing unused services  https://review.openstack.org/63327713:08
*** shyamb has joined #openstack-ansible13:12
*** electrofelix has quit IRC13:14
*** flaviosr has quit IRC13:18
*** dave-mccowan has joined #openstack-ansible13:20
*** tosky has quit IRC13:25
*** tosky has joined #openstack-ansible13:26
*** flaviosr has joined #openstack-ansible13:27
*** markvoelker has joined #openstack-ansible13:39
*** phasespace has quit IRC13:44
openstackgerritMerged openstack/openstack-ansible-os_neutron master: Set appropriate default for vxlan multicast ttl  https://review.openstack.org/63667613:56
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_gnocchi master: Convert systemd services to common role(s)  https://review.openstack.org/63715513:58
*** markvoelker has quit IRC14:12
*** kmadac has quit IRC14:18
openstackgerritMichael Vollman proposed openstack/openstack-ansible-os_neutron master: Avoid distro installing unused services  https://review.openstack.org/63327714:23
*** shyamb has quit IRC14:28
*** shyamb has joined #openstack-ansible14:35
*** shyamb has quit IRC14:40
*** jorti has joined #openstack-ansible14:48
openstackgerritFlorian Haas proposed openstack/openstack-ansible-os_cinder master: cinder.conf: add [nova] section, override interface defaults  https://review.openstack.org/63719814:56
*** flaviosr has quit IRC15:00
*** sum12 has quit IRC15:00
*** flaviosr has joined #openstack-ansible15:01
*** itandops has joined #openstack-ansible15:04
*** jorti has quit IRC15:05
*** jorti has joined #openstack-ansible15:05
*** DanyC has joined #openstack-ansible15:07
*** markvoelker has joined #openstack-ansible15:10
*** sum12 has joined #openstack-ansible15:15
fnpanicwhat is the plan for the heat ssl verify error15:15
fnpanichttps://bugs.launchpad.net/openstack-ansible/+bug/181490915:15
openstackLaunchpad bug 1814909 in openstack-ansible "Ssl verify error in heat -> keystone communication" [Undecided,New]15:15
fnpanicwill the patch be merged ?15:15
*** itandops has quit IRC15:15
jamesdentonIs there an official patch for that yet?15:19
openstackgerritFlorian Haas proposed openstack/openstack-ansible-os_cinder master: Introduce cinder_scheduler_default_filters  https://review.openstack.org/63720115:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Apply constraints when building the venv  https://review.openstack.org/62033115:23
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Apply constraints when building the venv  https://review.openstack.org/62033115:30
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Use the repo if available  https://review.openstack.org/62033915:33
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_nova stable/rocky: Update deprecated option for pci passthrough  https://review.openstack.org/63639015:35
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_nova stable/rocky: Update the pci config for nova.  https://review.openstack.org/63639115:35
*** rgogunskiy has quit IRC15:36
*** chkumar|ruck is now known as chandankumar15:38
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Update role for new source build process  https://review.openstack.org/62034015:39
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_nova stable/queens: Update deprecated option for pci passthrough  https://review.openstack.org/63639615:40
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_nova stable/queens: Update the pci config for nova.  https://review.openstack.org/63639715:40
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_nova stable/pike: Update deprecated option for pci passthrough  https://review.openstack.org/63639815:42
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_nova stable/pike: Update the pci config for nova.  https://review.openstack.org/63639915:42
*** markvoelker has quit IRC15:42
openstackgerritFlorian Haas proposed openstack/openstack-ansible-os_cinder master: Introduce cinder_scheduler_default_filters  https://review.openstack.org/63720115:43
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_gnocchi master: Convert systemd services to common role(s)  https://review.openstack.org/63715515:48
openstackgerritMatthew Thode proposed openstack/openstack-ansible-os_nova master: update pci_passthrough template variables  https://review.openstack.org/63721015:50
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_gnocchi master: Convert systemd services to common role(s)  https://review.openstack.org/63715515:51
*** gillesMo has joined #openstack-ansible15:56
*** francois has quit IRC16:17
antonymodyssey4me: i think we worked around it internally in newton to just leave the hostname and drop the domainname but i think we have a bug internally for it for the queens upgrade... i'd guess the proper way would check for stand alone hostnames and then convert them to openstack.local if a custom domain wasn't set since greenfield assumes openstack.local is set or overridden16:26
*** TxGirlGeek has joined #openstack-ansible16:28
*** macza has joined #openstack-ansible16:39
*** markvoelker has joined #openstack-ansible16:39
openstackgerritDmitriy Rabotjagov (noonedeadpunk) proposed openstack/openstack-ansible-os_gnocchi master: Convert systemd services to common role(s)  https://review.openstack.org/63715516:40
partlycloudyjamesdenton: hello! are you there and have a moment?16:56
*** gyee has joined #openstack-ansible17:01
*** phasespace has joined #openstack-ansible17:06
openstackgerritMerged openstack/openstack-ansible master: Use runtime venv to determine OSA version  https://review.openstack.org/63699217:07
*** markvoelker has quit IRC17:13
jamesdentonhi im here17:14
partlycloudythank you james! still about OVS.. can i use a bridge instead of a nic or bond for the 'network_interface' for br-provider in openstack_user_variables.yml?17:18
jamesdentonwhat bridge did you have in mind?17:18
jamesdentona linux bridge?17:18
partlycloudy i tried to use br-vlan (linuxbridge) that I created during provisioning.17:19
guilhermespdoes someone know a good way to deal with this? http://paste.openstack.org/show/745173/17:20
guilhermespthat's probably the reason for that bug https://bugs.launchpad.net/openstack-ansible/+bug/181590217:20
openstackLaunchpad bug 1815902 in openstack-ansible "Error galera_server : Download extra packages" [Undecided,New]17:20
jamesdentonyou should be able to use a bridge, however, i don't know that network_interface works in rocky. You can always try 'ovs-vsctl add-if br-provider br-vlan'17:20
*** gillesMo has quit IRC17:21
*** TxGirlGeek has quit IRC17:21
partlycloudyi tried to use br-vlan, but according to 'ovs-vsctl show', the br-provider only had two ports: br-provider and phy-br-provider17:21
jamesdentontry the ovs-vsctl command ^^^17:22
partlycloudyok. let me try it17:22
evrardjpodyssey4me: mnaser how is master nowadays? should I try a alpha release? We are not tied to anything anymore, and I like to either do that, or do a pbr bump17:22
evrardjpI would prefer releasing an alpha17:23
evrardjpmnaser: we should also discuss the releasing of ansible-config_template17:23
partlycloudyjamesdenton: ok. i used 'ovs-vsctl add-port br-provider br-vlan' and it worked.17:24
jamesdentoncool.17:25
partlycloudydoing it on all controller(network)nodes now.. see if i can get packets out after.17:26
partlycloudyjamesdenton: hmm… i added br-vlan to br-provider and restarted neutron on all controller(network) hosts. still i can't reach vms using floating ip. actually i can't even reach the router.17:33
jamesdentoncan you paste the output of 'ovs-vsctl show' into pastebin?17:33
partlycloudyjamesdenton: https://pasted.tech/pastes/5ea040cfd60a767955d23be833697405097b9ce217:35
partlycloudyi found pasted.tech is nicer :-)17:35
jamesdenton:D17:36
jamesdentonand just to confirm, br-vlan is setup and UP with some physical interface?17:36
*** miloa has quit IRC17:38
partlycloudyjamesdenton: i see the non-ha router gateway interface is on infra1. so here is the 'ovs-vsctl show' from infra1: https://pasted.tech/pastes/dc125bb1a93f316818754ffd5974df43c9e3c7d017:39
partlycloudyyes, br-vlan is up and has eno2 enslaved to it.17:40
jamesdentonif you pop into the qrouter namespace on infra1, can it ping its gateway?17:40
partlycloudylet me try. ip netns exec ….. right?17:40
jamesdentonyep17:40
partlycloudyyes, i can.17:42
jamesdentonok, so that tells me that br-vlan is indeed working for you17:42
jamesdentonat least on that infra17:42
jamesdentonand your tenant networks are vxlan? or vlan?17:42
partlycloudyusing vxlan for overlay17:43
*** TxGirlGeek has joined #openstack-ansible17:43
jamesdentonok - can you confirm in a vm console log if dhcp worked?17:45
jamesdentonalso, did you update sec group rules to allow the connectivity?17:45
partlycloudyyes, i've added Allow all ICMP to sec group.17:46
jamesdentoncan you confirm connectivity between infra1 and some compute via the br-vxlan IP? Just to make sure that works17:47
jamesdenton172.29.240.11 to 172.29.240.12, for example17:48
partlycloudyfrom infra1 (172.29.240.11) i can reach all br-vxlan end points on other infras and computes17:50
jamesdentonkk17:50
partlycloudylet me try to recreate all networks, routers and vms.17:51
jamesdentonk17:52
odyssey4meevrardjp I dunno if you noticed, but https://review.openstack.org/636992 was needed to make the versioning work outside the gate.17:54
odyssey4meguilhermesp I think https://bugs.launchpad.net/openstack-ansible/+bug/1815902 was fixed - added a comment to the bug.17:56
openstackLaunchpad bug 1815902 in openstack-ansible "Error galera_server : Download extra packages" [Undecided,New]17:56
odyssey4meevrardjp releasing an alpha/beta doesn't matter much to me - I guess it depends on precedence in openstack as a whole17:56
odyssey4meevrardjp but yeah, I think master is relatively releasable right now - although it'd need a role pin17:57
partlycloudyjamesdenton: after rebuild, both interfaces on the router remain down.18:02
partlycloudyfrom horizon interface status page it says "VIF Type binding failed"18:02
jamesdentonhmm, well there will be an error in the openvswitch-agent log, hopefully, that coincides with that18:03
guilhermespodyssey4me: yeah, was fixed but raised another error. The fact that when it tries to get the url item ansible says it is undefined https://github.com/openstack/openstack-ansible-galera_server/blob/5e78067f409a438411c5c563811ac6edba358229/tasks/galera_install_download_extra_packages.yml#L2518:03
jamesdentoncheck the log and see if there's anything interesting ther18:03
guilhermespI was debugging locally and I saw that the values are lists and then ansible considers url as undefined variable18:04
guilhermespthat's a part of my debugging http://paste.openstack.org/show/745173/18:04
guilhermespitem.value is a list, so I can't access the 'url' attribute directly. because we're looping over a dict at the higher level18:05
jamesdentonpartlycloudy i need to grab a bite, but will be back in a few18:06
partlycloudyjamesdenton: yeah of course! take ur time. thanks so much.18:07
odyssey4meguilhermesp that's weird, because https://github.com/openstack/openstack-ansible-galera_server/blob/5e78067f409a438411c5c563811ac6edba358229/tasks/galera_install.yml#L3418:09
*** markvoelker has joined #openstack-ansible18:10
guilhermespyeah and then the task gets the whole dictionary https://github.com/openstack/openstack-ansible-galera_server/blob/5e78067f409a438411c5c563811ac6edba358229/tasks/galera_install_download_extra_packages.yml#L2318:10
guilhermespthen item in this task is a list18:10
guilhermespand boom, breaks with undefined variable18:11
odyssey4meguilhermesp Those tasks are meant to only execute for arm/PPC, so I think it's best to focus on what's going wrong there first18:12
odyssey4meactually, based on https://github.com/openstack/openstack-ansible-galera_server/blob/5e78067f409a438411c5c563811ac6edba358229/tasks/galera_install.yml#L34 - it should only execute on PPC hosts18:12
*** hamzaachi has quit IRC18:15
*** hamzaachi has joined #openstack-ansible18:15
guilhermespyes odyssey4me I agree. If we look at this aspect, cool, it works. The thins is just the logic of "Download extra packages". I think the structure to iterate over the values needs to be refactored so we can get all the urls and checksums we need to install the PPC packages18:17
guilhermespguys at #ansible are making some suggestions here. To create a list of those values to iterate using with_items18:18
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible master: Include package pins for glance playbook  https://review.openstack.org/63724018:18
prometheanfiremessage-id CAJCXu8ep1ER3AH0S=ukxyqpi0=kkDPWstr_voLQ-Gn=v-z3LCQ@mail.gmail.com had a good comment about security updates for constraints on stable branches18:18
prometheanfireI'd like to see OSA's perspective18:18
guilhermespas it runs only for PPC we don't need to worry about the keys of the dict18:18
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_glance master: Update role for new source build process  https://review.openstack.org/62034018:18
odyssey4meprometheanfire um, I think a link would be more useful than a mail ID18:19
odyssey4meguilhermesp maybe we just need to backstop it with a conditional of whether the item exists? although I'm curious why it even executes the task in the first place18:20
*** DanyC has quit IRC18:20
prometheanfireisn't online yet18:20
prometheanfirerefresh :D http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002802.html18:21
prometheanfireand fungi's response http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002803.html18:21
odyssey4melogan- was http://logs.openstack.org/31/632131/2/check/openstack-ansible-functional-ubuntu-xenial/9addd27/job-output.txt.gz#_2019-01-22_14_37_40_350931 ever sorted out?18:21
odyssey4meprometheanfire if I read that right, requests had a CVE - from our standpoint, as long as upper-constraints updates the version to include the fix, we inherit it18:23
openstackgerritKevin Carter (cloudnull) proposed openstack/monitorstack master: Update monitorstack to use updated openstacksdk and es  https://review.openstack.org/63724218:23
prometheanfireodyssey4me: right, but what's being argued is that reqs should not be expected to update stable branches for security things18:24
odyssey4meprometheanfire well, I think that packagers and us all rely on that mechanism - however, if that's the stance we can work around it because we have our own override mechanisms... but those are meant to only be for temporary purposes18:25
odyssey4meI would hope that even for devstack tests the desire would be to test with the same thing that everyone is using18:26
prometheanfireya, I kinda think we should backport stuff as we fix it but not be trusted to constantly watch for CVEs18:27
openstackgerritMichael Vollman proposed openstack/openstack-ansible-os_manila master: Basic working os_manila role  https://review.openstack.org/61193018:27
odyssey4meprometheanfire I think 'best-effort' from a requirements standpoint is totally suitable18:27
prometheanfirefungi: ^ that sound good?18:28
odyssey4mevollman wrt the distro test - perhaps it'd be best to just get the source builds right, get that merged, then follow on with distro builds?18:28
odyssey4mevollman I'm concerned that the patch is growing to be even more difficult to review than it already is18:29
prometheanfireodyssey4me: also, update the reviews to add cherry-picked-from18:29
vollmanodyssey4me: I'm happy to make the distro gate non-voting.  source gate is working consistently18:29
odyssey4mevollman many of the roles don't even have the distro builds setup, so there's no requirement to make it work... although thanks to you for improving those builds!18:30
guilhermespodyssey4me: I came up with a solution to access the values transforming the dicts values in a list and then iterate using with_items18:31
guilhermespI can submit a PR so you can see how it goes18:31
odyssey4meguilhermesp okie dokey, thanks!18:31
vollmanodyssey4me: np.  thanks for all the reviews.  I'll update the os_manila tests18:34
odyssey4meprometheanfire I'll respond to the email.18:36
prometheanfireodyssey4me: I just did as well18:37
jamesdentonodyssey4me https://github.com/logan2211/ansible-etcd/commit/fa1c447b6a979a614fc024725b5ecad215261c4a18:39
openstackgerritKevin Carter (cloudnull) proposed openstack/monitorstack master: Update monitorstack to use updated openstacksdk and es  https://review.openstack.org/63724218:40
jamesdentonpartlycloudy any luck in the logs?18:41
partlycloudythanks for coming back! the downed router interfaces have been solved. somehow the ovs daemon died on all 3 infra nodes.18:43
*** markvoelker has quit IRC18:43
partlycloudynow i can ping the router external gateway from internet.18:44
jamesdentonok, so you're back where you started18:44
partlycloudyas you mentioned momentarily, dhcp may be the cause of the problem.18:46
*** kopecmartin is now known as kopecmartin|off18:46
jamesdentonwell dhcp is a symptom usually18:47
jamesdentonor lack of dhcp, that is18:47
partlycloudyi checked the vm log and it contains 'checking http://169.254.169.254/2009-04-04/instance-id … failed 1/20: up 210.38. request failed' all the way to 20/2018:47
*** kmadac has joined #openstack-ansible18:47
jamesdentoncan you paste the entire log?18:47
openstackgerritMichael Vollman proposed openstack/openstack-ansible-os_manila master: Basic working os_manila role  https://review.openstack.org/61193018:48
partlycloudysure. and btw i may have some difficult time to explain to cloudnull as to why pasted.tech is down….18:48
jamesdentonyou broke it?18:48
partlycloudyi never abused it… only made a few pastes…18:48
jamesdentonlol18:49
jamesdentonhe'll make it better. stronger. faster18:49
partlycloudyno doubt about it!18:50
cloudnullmoved to a new clouds sever. one sec18:50
partlycloudyhere is the full boot up log: http://paste.openstack.org/show/745180/18:50
partlycloudycloudnull: kudo!18:51
*** kmadac has quit IRC18:51
openstackgerritMichael Vollman proposed openstack/openstack-ansible-os_manila master: Basic working os_manila role  https://review.openstack.org/61193018:51
cloudnullits back !18:52
cloudnullsorry. seems my VMs were down, they've now been rebuilt18:52
partlycloudyhigh 5!18:52
jamesdentonpartlycloudy Sending discover... repeats 3 times then moves on. so dhcp is not working. it is most likely related to vxlan18:53
jamesdentondo you have l2pop enabled?18:53
jrossermutlicast......18:53
jrosserdepending on your switch, not having a querier running would do that18:54
jrossernxos for sure18:54
jamesdentonand juniper, too18:54
jamesdentonenabled l2pop, if it isn't enabled, may be fastest path to success18:54
partlycloudyjamesdenton: l2_population=false18:54
*** DanyC has joined #openstack-ansible18:55
jamesdentonok, so if you can, delete the routers and networks, then add an override for 'neutron_l2_population: true' and rerun the neutron playbooks.18:55
partlycloudywill do. i pretty much followed the sample configurations. does that mean there should be a line added into the user_variables.yml for OVS Scenario?18:57
odyssey4meok folks, I'm out for the w/end - have a fantastic one!19:06
jamesdentonnot necessarily, but the docs need to be updated to reflect multicast/non-multicast scenarios19:06
jamesdentonand caveats to each19:06
*** TxGirlGeek has quit IRC19:07
*** TxGirlGe_ has joined #openstack-ansible19:07
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-galera_server master: Iterate over list of values of PPC packages dict  https://review.openstack.org/63727519:10
guilhermespodyssey4me: ^^19:10
*** shananigans has joined #openstack-ansible19:13
shananigansI'm unfamiliar with the backport process. Do we need to do anything to get this guy back ported to rocky and queens? https://review.openstack.org/#/c/636071/19:15
*** colin- has left #openstack-ansible19:16
partlycloudyjamesdenton: it's working now 8-)19:17
jamesdentonwoot!19:17
openstackgerritShannon Mitchell proposed openstack/openstack-ansible-os_cinder stable/rocky: Add missing CLI_OPTIONS when setting up qos volume types  https://review.openstack.org/63727919:19
partlycloudyso, what caused this issue in the first place? something related to my switch? (i saw jrosser mentioned it, but i didn't get it)19:19
openstackgerritShannon Mitchell proposed openstack/openstack-ansible-os_cinder stable/queens: Add missing CLI_OPTIONS when setting up qos volume types  https://review.openstack.org/63728119:19
fungiprometheanfire: odyssey4me: the counterargument we get from distros packaging out of our stable branches is that things shouldn't be changing unless they directly relate to fixes in the software. this particular requests update may not require patches to stable branches depending on the library, but other similar situations the the future could and we need to be able to continue supporting the19:27
fungidistros who are backporting fixes to our dependencies and not break them too. also as noted our security focus is on fixing vulnerabilities within our source code so we shouldn't give downstream consumers the false impression we track vulnerabilities in our dependencies19:27
prometheanfirethat false impression is the main issue I have with taking it19:28
fungias for devstack testing "the same thing that everyone is using" sticking with the frozen version of requests is likely closer to testing against an lts distro's contemporary python-requests package with a backported fix than testing against the latest upstream release of requests is19:28
jamesdentonpartlycloudy Depending on the switch, you have to configure IGMP snooping for multicast to work, or have a multicast router setup. L2 population relies on the neutron agent pre-populating the forwarding tables on all hosts. lot more overhead and places where things can go wrong19:31
*** gyee has quit IRC19:32
openstackgerritFlorian Haas proposed openstack/openstack-ansible-os_cinder master: Introduce cinder_scheduler_default_filters  https://review.openstack.org/63720119:34
partlycloudyjamesdenton: so, by default, l2population should be enabled and dhcp request from vms should be diverted directly to the dhcp agent, right?19:37
partlycloudyi need to go back and re-read your book now. :-)19:38
jamesdentonwell, l2pop should be enabled/disabled based on the capabilities of a given environment (and maybe documented why you would choose one or the other). When enabled, the compute node knows exactly which host to forward traffic to, and vice-versa. When disabled, the hosts rely on multicast to populate their fdb19:39
jamesdentonhah, well i don't know how well i documented it there, either19:39
*** markvoelker has joined #openstack-ansible19:41
*** shardy has quit IRC19:43
partlycloudyi see. thanks so very much for helping and explaining everything. i am really appreciated!19:44
*** gyee has joined #openstack-ansible19:46
openstackgerritFlorian Haas proposed openstack/openstack-ansible-os_cinder master: Introduce cinder_scheduler_default_filters  https://review.openstack.org/63720119:58
-openstackstatus- NOTICE: The StoryBoard service on storyboard.openstack.org is offline momentarily for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-February/002666.html20:02
*** TxGirlGe_ has quit IRC20:11
*** markvoelker has quit IRC20:13
*** TxGirlGeek has joined #openstack-ansible20:13
*** ArchiFleKs has quit IRC20:44
*** TxGirlGeek has quit IRC20:45
*** shananigans has quit IRC20:45
*** TxGirlGeek has joined #openstack-ansible20:45
*** shananigans has joined #openstack-ansible20:47
*** ArchiFleKs has joined #openstack-ansible21:06
*** markvoelker has joined #openstack-ansible21:10
openstackgerritMichael Vollman proposed openstack/openstack-ansible-os_manila master: Basic working os_manila role  https://review.openstack.org/61193021:31
*** markvoelker has quit IRC21:43
*** jra has joined #openstack-ansible21:46
jraso we're running a queens openstack deploy with OSA 17.0.8, all of which was originally installed on centos 7.5. Now 7.6 is out, and it's causing significant problems: there's no co 7.5 LXC base image available anymore, so I basically can't install a new infrastructure node unless I'm ready to move at least some of the deployment up to 7.6, which means rebuilding the repo21:48
jraHow have other folks dealt with this?21:48
*** markvoelker has joined #openstack-ansible22:40
*** shananigans has quit IRC22:41
*** aedc has quit IRC22:45
*** dave-mccowan has quit IRC22:45
*** marst has joined #openstack-ansible22:47
*** marst has quit IRC23:05
*** markvoelker has quit IRC23:13
openstackgerritMichael Vollman proposed openstack/openstack-ansible-os_manila master: Basic working os_manila role  https://review.openstack.org/61193023:22
*** gyee has quit IRC23:50
*** hwoarang has quit IRC23:50
*** gyee has joined #openstack-ansible23:54
*** hwoarang has joined #openstack-ansible23:54
cloudnulljra you have two options, 1 rerun the lxc-hosts playbook, 2 copy the lxc image cache to the new version name23:56
*** DanyC has quit IRC23:56
cloudnullthe cache should in under /var/cache/lxc23:56
*** DanyC has joined #openstack-ansible23:56
cloudnullI guess the distro version according to ansible also includes the minor release on centos ?23:57
*** hamzaachi has quit IRC23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!