Friday, 2019-04-12

*** ianychoi has joined #openstack-ansible00:01
snadgeand from what I can see, vmware (vsphere) doesn't appear to support QinQ or double vlan tagging.. i didn't know that would be required?00:02
*** hamzy has joined #openstack-ansible00:14
kplantno, i was thinking outloud if it did not support trunks00:14
*** marst has joined #openstack-ansible00:15
*** gyee has quit IRC00:15
mnaseryou wouldn't necessarily require it00:23
mnaserif you do vxlan networks only, that might be ok00:23
mnaserbut you won't ever be able to access your VMs in there00:23
mnaser:P00:23
kplantyou could do vxlan with an sdn00:37
kplantthen plop a bastion on the vxlan(s) and a real l2/l3 interface00:38
kplantthat was a joke, please don't do that00:51
*** marst has quit IRC00:59
*** cshen has joined #openstack-ansible01:31
*** cshen has quit IRC01:35
snadgeim slowly learning about vlan, vxlan, qinq etc.. im a systems guy who feels horribly overwhelmed by all these new networking concepts02:00
snadgei mean.. I obviously understand vlans to the extent that I need to.. but it appears the rabbit hole goes very deep02:00
*** markvoelker has joined #openstack-ansible02:00
snadgeso far neutron seems to cause me mentally the most amount of problems.. i feel fairly comfortable with everything else.. so i just need to get over this aversion, or do a course on it or something02:03
kplanti would maybe start with ccent/jncia02:06
kplantyou'll want a solid base understanding of routing and switching02:07
kplantnot even getting the certifications, they mean nothing, but the material for them if you learn it02:07
*** jsquare has quit IRC02:19
*** jra has quit IRC02:19
*** jsquare has joined #openstack-ansible02:20
*** nurdie has joined #openstack-ansible02:20
*** nurdie has quit IRC02:24
*** markvoelker has quit IRC02:35
*** irclogbot_2 has quit IRC03:01
*** irclogbot_2 has joined #openstack-ansible03:01
*** kplant has quit IRC03:12
*** nurdie has joined #openstack-ansible03:14
*** kplant has joined #openstack-ansible03:16
*** nurdie has quit IRC03:20
*** partlycloudy has quit IRC03:21
*** raukadah is now known as chandankumar03:25
*** nicolasbock has quit IRC03:25
*** cshen has joined #openstack-ansible03:31
*** markvoelker has joined #openstack-ansible03:32
*** cshen has quit IRC03:40
*** hwoarang has quit IRC03:59
*** hwoarang has joined #openstack-ansible04:00
*** markvoelker has quit IRC04:04
*** udesale has joined #openstack-ansible04:13
*** marst has joined #openstack-ansible04:16
*** _d34dh0r53_ has joined #openstack-ansible04:29
*** marst has quit IRC04:35
*** prometheanfire has joined #openstack-ansible04:37
*** cloudkiller has joined #openstack-ansible04:57
*** markvoelker has joined #openstack-ansible05:02
*** markvoelker has quit IRC05:34
*** cshen has joined #openstack-ansible05:36
*** cshen has quit IRC05:40
*** cshen has joined #openstack-ansible06:21
*** markvoelker has joined #openstack-ansible06:31
fnpanicgood morning06:33
*** pcaruana has joined #openstack-ansible06:41
*** ivve has joined #openstack-ansible06:42
*** DanyC has joined #openstack-ansible06:42
*** DanyC has quit IRC06:50
spotzMorning06:51
*** luksky has joined #openstack-ansible06:58
*** hamzy has quit IRC07:05
*** markvoelker has quit IRC07:05
*** hamzy has joined #openstack-ansible07:05
*** mnaser has quit IRC07:06
*** gundalow has quit IRC07:06
*** gundalow has joined #openstack-ansible07:06
*** mnaser has joined #openstack-ansible07:06
*** tosky has joined #openstack-ansible07:43
*** phasespace has joined #openstack-ansible07:46
*** hamzaachi has joined #openstack-ansible07:53
evrardjpmnaser: as promised: https://review.openstack.org/#/c/651998/107:54
evrardjpnow dealing with EM07:54
*** ivve has quit IRC07:56
CeeMacmorning07:58
*** markvoelker has joined #openstack-ansible08:02
evrardjpmorning08:03
evrardjp(but I am on holidays, so it's just a courtesy greeting, sorry :p )08:03
CeeMac:)08:05
*** ivve has joined #openstack-ansible08:11
*** luksky has quit IRC08:12
fnpanicdoes anyone know if it is possible to not store the password on openrc as plaintext?08:18
fnpanicor another solution because having a service using openstack apis is a common scenario08:19
fnpanicbecause of the ad backend putting the password there is not a good idea08:19
spotzfnpanic You can not have it there but would then be prompted for it, not sure that helps08:20
fnpanicspotz: thanks but this does not help :-)08:22
*** dxiri has joined #openstack-ansible08:23
spotzCould you pass in the password from a script as you go?08:27
*** dxiri has quit IRC08:28
evrardjpfnpanic: you can use a passwords file to remove the password from your clouds.yaml, but not 100% sure what you can do to encrypt it from that on. Have you checked openstack client documentation ?08:28
evrardjpfnpanic: is there a reason you are afraid of having this in plain text in some *trusted* host?08:28
spotzfnpanic evrardjp: Yeah I'm not sure we can encrypt decrypt with Keystone... I don't think Barbican would help either but that might be a suggestion08:30
evrardjpI think I misunderstood the question08:30
spotzevrardjp: No I think you got it:) He doesn't want OS_PASSWORD to be plain text in openrc08:31
evrardjpbut in which context for openstack-ansible?08:31
spotzI was just thinking in his deployment not OSA specific?08:32
evrardjpthat's where I am confused08:32
evrardjpok08:32
spotzmight be wrong though08:32
*** markvoelker has quit IRC08:34
evrardjpfnpanic: unix file permissions is not enough for you?08:36
evrardjp(just asking)08:36
spotzodyssey4me and anyone else if you're coming to summit let me know and your google hangout info on channel or PM08:36
evrardjpthe usual one for me :)08:36
*** electrofelix has joined #openstack-ansible08:39
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/openstack-ansible-os_designate master: Updated from OpenStack Ansible Tests  https://review.openstack.org/64900108:49
CeeMacspotz, people still use google hangouts?08:50
* CeeMac goes to put it back on his phone08:51
chandankumarodyssey4me: hello08:52
chandankumarodyssey4me: Hello08:53
spotzCeeMac: just at summit for communication between the team for like dinner and such08:53
CeeMacspotz, i just presumed they'd shut that down along with google+08:53
chandankumarodyssey4me: please have a look at this error http://logs.openstack.org/54/650054/7/check/openstack-ansible-functional-tempestconf-centos-7/d0f58e9/logs/ara-report/result/3bc8be06-8924-4ace-9add-c3403be12ae5/ when free, thanks!08:54
CeeMacSo, I'm building a follow on environment from the RC I've been testing in and I'm planning on implementing untagged management vlan for host management08:56
CeeMacpresumably, the deployment node still needs to be homed on the br-mgmt vlan so it can talk direct to the containers?08:57
CeeMachost=physical host08:57
jrosserCeeMac: deploy node does not need to be on br-mgmt08:58
CeeMacas long as their is a route to br-mgmt network right?08:58
CeeMacand hi jrosser :)08:58
jrosserIt needs to be able to ssh to the ip of the physical hosts you put in user_config08:58
jrosserNothing to do with mgmt net really at all08:58
CeeMacoh, so when it runs plays against the containers it communicates with them via the underlying host not direct?08:59
jrosserOnce on the host the ansible connection plugin does its thing, without ssh being involved08:59
CeeMactheres an ansible connection plugin?08:59
CeeMaci missed that one08:59
jrosserOh yes08:59
CeeMacpresumably thats a connection plugin for lxc then?09:00
jrosserAnd nspawn too09:00
CeeMacright ==containers09:00
CeeMacthats clever :D09:00
jrosserLook in the plugins repo :)09:00
* CeeMac goes to look09:01
* jrosser away for the rest of today....09:01
spotzCeeMac: nope still working:)09:01
*** jsquare has quit IRC09:05
*** jsquare has joined #openstack-ansible09:05
*** ivve has quit IRC09:10
*** luksky has joined #openstack-ansible09:13
CeeMaci seem to recall an issues around multiple repo nodes, was that in rocky? Is it better just to deploy 1?09:17
odyssey4meCeeMac That's stein, for now.09:20
*** ivve has joined #openstack-ansible09:21
CeeMacodyssey4me, ah ok09:21
CeeMaci'll try with 3 then and see what happens :009:22
CeeMac:)09:22
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible-os_tempest master: Switch to import_task in os_tempest  https://review.openstack.org/65005409:27
*** hamzaachi has quit IRC09:27
*** hamzaachi has joined #openstack-ansible09:28
*** markvoelker has joined #openstack-ansible09:31
CeeMachmm09:34
CeeMacits only the compute and network nodes that need br-vlan and br-vxlan right?09:34
*** hamzaachi has quit IRC09:42
*** hamzaachi has joined #openstack-ansible09:43
*** hamzaachi has quit IRC09:51
*** hamzaachi has joined #openstack-ansible09:52
openstackgerritMerged openstack/openstack-ansible-os_ironic master: Replace usage of netloc filters to urlsplit  https://review.openstack.org/64819009:58
openstackgerritMerged openstack/openstack-ansible-os_nova master: Replace usage of netloc filters to urlsplit  https://review.openstack.org/64818910:00
*** markvoelker has quit IRC10:04
*** Kurlee has joined #openstack-ansible10:10
openstackgerritMerged openstack/openstack-ansible-os_cinder master: Replace usage of netloc filters to urlsplit  https://review.openstack.org/64818810:11
*** hamzaachi has quit IRC10:19
*** hamzaachi has joined #openstack-ansible10:22
*** pcaruana has quit IRC10:31
*** nicolasbock has joined #openstack-ansible10:44
*** markvoelker has joined #openstack-ansible11:02
*** udesale has quit IRC11:05
*** dxiri has joined #openstack-ansible11:06
*** dxiri has quit IRC11:11
*** ivve has quit IRC11:21
*** cshen has quit IRC11:25
*** cshen has joined #openstack-ansible11:25
*** ivve has joined #openstack-ansible11:30
*** markvoelker has quit IRC11:35
*** pcaruana has joined #openstack-ansible11:36
*** hamzaachi has quit IRC11:44
openstackgerritMerged openstack/openstack-ansible-os_designate master: Updated from OpenStack Ansible Tests  https://review.openstack.org/64900111:58
openstackgerritVadim Kuznetsov proposed openstack/openstack-ansible-os_octavia master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/65204911:59
*** dxiri has joined #openstack-ansible12:01
openstackgerritVadim Kuznetsov proposed openstack/openstack-ansible-os_octavia master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/65204912:03
*** starborn has joined #openstack-ansible12:07
*** dhellmann has left #openstack-ansible12:10
*** partlycloudy has joined #openstack-ansible12:28
*** darkomenz has joined #openstack-ansible12:31
darkomenzHello, is there any way to reduce memory consumption when running a multi node deploy. I have a machine that has used 8gb's of physical ram and and 12gb's of swap and is our of memory. As I understand 8GB should be the minimum.12:33
darkomenzouot of memory *12:33
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible-os_tempest master: Use tempest_run_smoke for running smoke tests  https://review.openstack.org/65206012:40
*** sm806 has quit IRC12:45
*** DanyC has joined #openstack-ansible12:45
*** timburke has quit IRC12:46
*** cjloader has quit IRC12:46
*** timburke has joined #openstack-ansible12:48
*** DanyC has quit IRC12:50
openstackgerritChandan Kumar (raukadah) proposed openstack/openstack-ansible-os_tempest master: Use tempest_run_smoke for running smoke tests  https://review.openstack.org/65206012:51
*** dxiri has quit IRC13:03
*** altlogbot_3 has joined #openstack-ansible13:03
*** dave-mccowan has joined #openstack-ansible13:07
jamesdentonAnsible pros... what would cause Ansible to hang when gathering facts or perform any task against a host involving SSH, yet, SSH works fine from the CLI?13:10
*** vnogin has joined #openstack-ansible13:11
ironfootis it ssh'ing into the right user? Use -vvvv to see all the debug information from ansible13:12
*** dave-mccowan has quit IRC13:17
odyssey4mejamesdenton it's always the MTU ;)13:17
jamesdenton:)13:17
ironfoot+inf13:17
ironfootIt's like the selinux of networking13:18
*** hamzaachi has joined #openstack-ansible13:19
jamesdentonodyssey4me you were not wrong13:22
*** marst has joined #openstack-ansible13:23
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-nspawn_hosts master: debian: add support  https://review.openstack.org/65207113:23
kplantsomeone needs to write a windowing standard for ethernet13:24
kplantmtu/mfs mismatches are super annoying :-(13:24
jamesdentonugh, i'm all over the place and forgot to set jumbo on the physical NIC. yikes. thanks everyone.13:27
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-nspawn_container_create master: debian: add support  https://review.openstack.org/65207513:30
openstackgerritVadim Kuznetsov proposed openstack/openstack-ansible-os_octavia master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/65204913:31
*** altlogbot_3 has quit IRC13:32
*** altlogbot_2 has joined #openstack-ansible13:33
*** eglute has quit IRC13:33
*** cloudnull has quit IRC13:34
*** cloudkiller is now known as cloudnull13:34
*** d34dh0r53 has quit IRC13:35
*** phasespace has quit IRC13:38
*** altlogbot_2 has quit IRC13:38
*** altlogbot_2 has joined #openstack-ansible13:38
guilhermespcloudnull: around?13:50
*** hamzaachi has quit IRC13:51
*** hamzaachi has joined #openstack-ansible13:52
CeeMacjamesdenton, hi :)13:52
CeeMacis there any reason why you couldnt wire a bond.vlan interface to an openvswitch bridge?13:53
jamesdentonIf you're asking what I think you're asking, it would limit you to a single flat network from neutron's perspective, since you're already tagging a vlan on that interface13:57
*** altlogbot_2 has quit IRC14:00
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-nspawn_hosts master: debian: add support  https://review.openstack.org/65207114:04
jamesdentonCeeMac Does that make sense?14:06
CeeMacjamesdenton, it does14:07
mnasermorning all14:07
CeeMaci was thinking from the br-vxlan perspective14:08
mnaserevrardjp: I left some reviews, I can update your openstack/releases change, or you can if you want :)14:08
CeeMacalthough i'm still unclear on if/which the other bridges are required alongside openvswitch14:08
CeeMacas in all of the docs it only seems to reference br-provider14:08
CeeMacbut there is br-int and br-tun - they get autocreated maybe?14:09
jamesdentonyes, br-int and br-tun are automatically created14:09
jamesdentonyou simply need to create a provider bridge - and newish versions of OSA may create that for you, too14:09
CeeMachmm14:10
jamesdentonbr-vxlan is a vestige of containerized network agents, but it serves as an interface on top of which the VTEP address (for vxlan) is configured14:10
CeeMacand does that need to ovs as well for tenant networks?14:11
CeeMacoh wait, you're saying if the network agents are running on metal (which they do in rocky) you don't necessarily need br-vxlan?14:12
CeeMacin user_config br-vxlan binds to the neutron_linuxbridge_agent group by default14:13
odyssey4mejamesdenton that.is.hilarious.14:15
jamesdentonodyssey4me you took a stab in the dark, and were right. lol14:15
jamesdentonCeeMac There's an expectation that br-vxlan exists, as the provider definition for the vxlan network type is used in a playbook to dynamically fetch the VTEP IP from inventory, IIRC. Even for OVS.14:18
CeeMacright14:18
CeeMacif i understand things correctly, with ovs you still create seperate bridges, each with their own port14:19
CeeMacso i could bind bond.vlan to the br-vxlan bridge, and a dedicated/seperate bond for br-provider?14:19
CeeMacthey're the only 2 bridges requied in ovs i think?  the others (br-mgmt br-storage) would remain lxb?14:20
jamesdentonWith OSA OVS, br-vxlan would also be a linux bridge containing bond.vlan. You only need to create br-provider as an OVS bridge14:21
*** fdegir has joined #openstack-ansible14:22
CeeMacoh14:22
CeeMacwell14:22
CeeMacthat makes things straightforward :)14:23
CeeMacand br-vlan would not then be required?  I'm pure vxlan for tenant networks14:24
jamesdentonYou wouldn't need br-vlan as a linux bridge, no. But you will need a provider bridge for OVS, and that can be called br-provider or br-vlan or br-ex or br-whatever14:24
*** cshen has quit IRC14:25
CeeMacright14:25
jamesdentonBut it wouldn't be necessary on a compute, per say, if you're only doing overlay14:25
CeeMacunless dvr?14:25
jamesdentonHowever, because the docs bind that network to neutron_openvswitch_agent group, it might want it there regardless.14:25
jamesdentonRight, DVR14:25
*** altlogbot_0 has joined #openstack-ansible14:26
jamesdentonYou *can* limit the group to 'network_hosts' instead of 'neutron_openvswitch_agent', in which case it ought to only be configured on the infras/network nodes14:26
CeeMacis it possible to inobtrusivley implement DVR at a future point if i start with ovs and legacy routers?14:26
CeeMacsorry. jumping around a bit i know14:27
jamesdentonThere may be one or two overrides for it, IIRC. You may be better off rolling it out initially, then you can set an override to force tenant routers to legacy or HA when created by non-admins14:28
CeeMacmakes sense14:29
*** altlogbot_0 has quit IRC14:29
CeeMacincidentally, i've abandoned netplan for these new server builds and am trying pure systemd-networkd14:29
*** altlogbot_1 has joined #openstack-ansible14:30
jamesdentoni went pure networkd, too. they all suck.14:30
CeeMachaha, true dat14:30
CeeMacits not as bad as i initially thought14:30
CeeMacalthough it took me a while to work out why my bond.vlan interface wasn't being created14:31
jamesdentondeath by micro-segmentation14:31
*** nurdie has joined #openstack-ansible14:31
*** altlogbot_1 has quit IRC14:33
CeeMacjamesdenton, if i wanted to provision 2 provider switches, is that configurable through osa?14:39
jamesdentonyes14:39
jamesdentonjust define two different 'vlan' blocks with their respective attributes14:40
CeeMaccool14:40
CeeMacwant to seperate out internet facing traffic to transport/mpls traffic14:41
*** vnogin has quit IRC14:41
jamesdentonhttp://paste.openstack.org/show/749243/14:41
CeeMacthanks14:43
CeeMacand that is the same regardless of lxb/ovs right?14:45
CeeMacand can be on the same host group14:46
jamesdentonyeah, the group bind could change (from neutron_linuxbridge_agent to neutron_openvswitch_agent) but in my example, those networks apply to 'compute_hosts' regardless of plugin14:46
CeeMacawesome, thanks14:47
*** altlogbot_1 has joined #openstack-ansible14:48
CeeMacout of interest are you using and sdn controller with ovs?14:49
jamesdentoni am not, no.14:51
*** altlogbot_1 has quit IRC14:51
*** kplant has quit IRC14:51
*** altlogbot_0 has joined #openstack-ansible14:52
*** altlogbot_0 has quit IRC14:55
*** altlogbot_0 has joined #openstack-ansible14:56
*** altlogbot_0 has quit IRC14:56
*** kplant has joined #openstack-ansible14:57
CeeMachmm14:58
CeeMacif the dvrs exist in ovs14:58
*** altlogbot_3 has joined #openstack-ansible14:58
CeeMachow does the tenant vxlan network integrate with the dvr and break out via the lxb br-vxlan14:59
CeeMacthink that's a monday question15:01
CeeMacok, i'm out. have a great weekend osa people :)15:04
noonedeadpunkThanks, you too CeeMac :)15:08
jamesdentonCeeMac It doesn't have to break out of br-vxlan in that way. OVS will use the IP address that is configured on br-vxlan, and thus traffic leaves bond.vlan connected to br-vxlan. There is no 'plumbing connection' between the lxb and ovs bridge, per say15:09
chandankumarguilhermesp: after turning all smoke tests few tests failed, we will fix it on monday!15:10
*** tosky has quit IRC15:11
*** tosky has joined #openstack-ansible15:12
*** ivve has quit IRC15:20
*** electrofelix has quit IRC15:23
*** luksky has quit IRC15:24
dmsimardodyssey4me, mnaser: heads up, things like http://git.openstack.org/cgit/openstack/openstack-ansible/tree/scripts/scripts-library.sh#n158 and http://git.openstack.org/cgit/openstack/openstack-ansible-tests/tree/test-ansible-env-prep.sh#n191 are eventually going to fail with the incoming opendev move15:28
mnaserdmsimard: ugh, that is going to be pretty bad15:30
mnaserI worry we will break existing releases15:30
dmsimardmnaser: src/git.openstack.org/openstack/ara should only be used in Zuul itself, though, right ?15:31
mnaserdmsimard: im wondering if there is other places we might depend on things15:31
*** chandankumar is now known as raukadah15:36
logan-for external dependencies the redirects should work fine15:38
logan-in zuul where we hard code src paths we'll have to fix that, but that can only work in the gate anyway, so we should never break a release due to it15:38
dmsimardmnaser: re-reading that code, it should probably not break after all15:45
dmsimardit checks if the directory exists (depends-on) and if it doesn't it installs from pypi15:45
noonedeadpunkIs there any rabbitmq expert? Preferably among cores:)15:47
noonedeadpunkJust need advice about some idea15:48
*** hamzaachi has quit IRC15:49
*** hamzaachi has joined #openstack-ansible15:50
noonedeadpunkok, whatever. So yestarday phasespace pointed that we don't have ha enabled for all rabbitmq vhosts - only for /. It seems ,that the policies are not inherited from / iirc. Than my thought was to move rabbitmq_openstack_policies to all group_vars.15:52
noonedeadpunkAnd apply policies like https://github.com/openstack/openstack-ansible-rabbitmq_server/blob/master/tasks/rabbitmq_post_install.yml#L87-L9815:52
noonedeadpunkFor each role which is uses rabbit (almost every actually). So I was going to patch one role (probably ceilometer) during this weekends as an exmaple and to check the concept.15:54
noonedeadpunkSo does anyone have any thoughts about it?15:54
noonedeadpunkAnd, probably, this all should be backported at least to stein?15:55
*** cmart has joined #openstack-ansible15:58
openstackgerritVadim Kuznetsov proposed openstack/openstack-ansible-os_octavia master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/65204916:05
*** gyee has joined #openstack-ansible16:15
*** cshen has joined #openstack-ansible16:19
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Ensure venv_wheel_build_enable is evaluated as boolean  https://review.openstack.org/65210816:22
*** cshen has quit IRC16:23
*** hamzaachi has quit IRC16:29
*** hamzaachi has joined #openstack-ansible16:29
mnasernoonedeadpunk: I think it would be good to actually move the policy setup inside the role rather than in rabbitmq_server IMHO16:33
noonedeadpunkmnaser: That excatly what I was going to achieve - set policy inside every affected role16:35
noonedeadpunkBut as the policy will be the same I though about adding rabbitmq_openstack_policies into gloabl group_vars16:35
noonedeadpunksomewhere here https://github.com/openstack/openstack-ansible-os_ceilometer/blob/master/tasks/mq_setup.yml#L3516:36
noonedeadpunkand probably we don't need policy for / at all (not sure wether smth utilize / vhost)16:38
*** tosky has quit IRC16:38
partlycloudyHello everyone, i got a rookie question on using Clos network with openstack. (continued from yesterday)16:39
partlycloudynow I understand the subnetting for br-mgmt, br-vxlan, br-storage networks for every leaf.16:39
partlycloudywhat i don't know is how to bring the external network (which sits in a dedicate leaf) to every other leaf.16:39
partlycloudysorry if that sounds like i duno what i'm doing (cos that exactly is the case here...)16:40
*** bgmccollum has quit IRC16:41
partlycloudyhow can i give every leaf the access to the default gateway for the external network which is located on a different leaf?16:42
jrosserpartlycloudy: hello :)16:43
partlycloudyjrosser: hello there!16:44
jrosserso i have a set of leaves that the external network comes to, running whichever routing protocol is needed for the upstream router16:45
jrosserand thats as far as the external networks go. network nodes and so on have an interface on those leaves to allow floating IP16:45
jrosserif you want an effective L2 network over your L3 fabric you'll need an overlay of some sort16:46
jrosseror you can split you external network into smaller subnets and make a "segmented provider network"  which may be excellent/terrible fit for your use case16:49
jrosserreally depends what you want to achieve16:49
partlycloudyjrosser: if i understand it correctly, the first solution is to connect a set of leaves (which contain network nodes or whatever requires external access) to the same L2 layer as the external router?16:50
*** bgmccollum has joined #openstack-ansible16:51
jrosserif all you are interested in is floating ip via neutron l3 agent then that would do it16:51
jrosserbut if you want to attach instances directly to the external net, it wouldn't do it16:52
partlycloudyjrosser: yes, i actually wish to be able to connect instances to external network directly (bypass the network nodes).16:53
*** cjloader has joined #openstack-ansible16:53
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Ensure the build host is the first repo server  https://review.openstack.org/65211016:54
jrosserok so that is fundamentally hard if you also want to build a L3 fabric, it's a sort of two orthoganol things16:54
kplantthat's kind of scary isn't it?16:54
odyssey4memnaser jrosser ^ that should solve the build host selection for a homogenous env... I'll try to figure something out for multi-arch/multi-distro envs16:54
partlycloudyjrosser: would something like EVPN/VXLAN help to connect every leaf together on the same L2 level with the external router?16:56
noonedeadpunkodyssey4me but in my case lsyncd seems to be configured on the last host....16:57
odyssey4menoonedeadpunk oh? how the heck did that happen?16:57
odyssey4menote that it's the first host from the ansible inventory standpoint, not alphanumerically sorted16:58
noonedeadpunkso I have the following http://paste.openstack.org/show/749252/ and lsyncd is running on the uacloud-mgmt0316:58
odyssey4menoonedeadpunk can you paste the output of: ansible -m ping repo_all16:58
noonedeadpunkYep, you're right, they got reversed in inventory http://paste.openstack.org/show/749253/16:59
noonedeadpunkSo disregard:)16:59
odyssey4menoonedeadpunk yeah, that's the dynamic inventory as far as I know17:00
*** cmart has quit IRC17:00
mnaserodyssey4me: cloudnull was trying to look into that afaik17:02
mnasersos just cc on that message :)17:02
*** openstacking_123 has joined #openstack-ansible17:03
openstacking_123Anyone have an issue with the latest stable openstack-ansible rocky deploy having issues deleting images or restoring snapshots?17:03
openstacking_123specifically seeing ile "rbd.pyx", line 2456, in rbd.Image.unprotect_snap .  PermissionError: [errno 1] error unprotecting snapshot17:05
openstacking_123In the glance api log17:05
*** bgmccollum has quit IRC17:05
jrosserpartlycloudy: you should look at two things "routed provider networks" and like you say an alternative is EVPN or similar to make a virtual L2. Both of these have different pros/cons - also double check you really do need a L3 underlay for this :)17:05
*** openstacking_123 has quit IRC17:06
*** hamzaachi has quit IRC17:08
partlycloudyjrosser: thank you so much for showing me the way! i'll do my homework now and come back for help if i get jammed :-)17:10
*** noonedeadpunk has quit IRC17:11
odyssey4megood call on https://review.openstack.org/651598 by the way mnaser  - just need another core to +2+w17:12
mnaserodyssey4me: slowly getting my hang on python_venv_build :)17:12
mnaserthose are stuff that bubbled up to me from our stein deploy17:12
odyssey4memnaser yeah, I do see that we're not using the global pins anywhere - so that will need to be instrumented in, otherwise we lose out on the pip/setuptools/wheel pins17:13
*** cmart has joined #openstack-ansible17:14
odyssey4mewe're also not actually building the wheels for pip/setuptools/wheel anywhere, so we need to sort that out to help speed things up17:15
raukadahguilhermesp: what is the etherpad link for osa train planning?17:20
raukadahodyssey4me: is there a way to skip the python_venv_build handler here http://logs.openstack.org/54/650054/8/check/openstack-ansible-functional-tempestconf-centos-7/5cc3faa/job-output.txt.gz#_2019-04-12_10_31_43_727975?17:22
raukadahit is constantly failing17:23
odyssey4meraukadah it is failing because of something else there, not because of the handler17:23
odyssey4meeither a var is not properly imported, or not evaluating properly17:23
raukadahodyssey4me: or is this one https://github.com/openstack/openstack-ansible-os_tempest/blob/master/tests/test-tempest-functional.yml#L32?17:24
partlycloudyjrosser: sorry to bother again. for the solution on "subneting provider networks", that would give each leaf a different external gateway, right?17:24
odyssey4meraukadah I dunno, but I haven't looked deeply into the specific test details and don't have the time to either.17:26
raukadahodyssey4me: I will check tomorrow!17:27
*** ivve has joined #openstack-ansible17:27
*** dxiri has joined #openstack-ansible17:37
*** hamzaachi has joined #openstack-ansible17:38
*** darkomenz has quit IRC17:39
openstackgerritAntony Messerli proposed openstack/openstack-ansible-lxc_hosts stable/stein: Use pkill for lxc-dnsmasq systemd unit file  https://review.openstack.org/65211917:42
*** openstacking_123 has joined #openstack-ansible17:45
openstacking_123Another thing I notice. Is all images are being created as protected in Ceph and a snapshot is made17:45
jrosserpartlycloudy: for a routed provider network leaf would have its own gateway17:47
logan-openstacking_123: yeah, that is standard since ceph requires a protected snapshot to do copy on write image layering. are there any children of the snapshot existing still? that would prevent unprotecting it i'd think17:47
openstacking_123Well I just uploaded a non snap shot brand new image and hit the same issue.17:49
openstacking_123Unproctecting it in ceph allows it to delete17:49
openstackgerritAntony Messerli proposed openstack/openstack-ansible-lxc_hosts stable/queens: Use pkill for lxc-dnsmasq systemd unit file  https://review.openstack.org/65212017:49
partlycloudyjrosser: i see. i'll dig into it. thank you so much for helping!17:49
logan-openstacking_123: are you using the same cephx key to do your manual unprotect as openstack is using? or are you using the admin key? i wonder if the openstack key lacks some permissions17:52
openstacking_123Yeah using cephx with glance key from a glance container17:53
openstackgerritAntony Messerli proposed openstack/openstack-ansible-lxc_hosts stable/pike: Use pkill for lxc-dnsmasq systemd unit file  https://review.openstack.org/65212317:55
openstacking_123logan https://gist.github.com/fritzstauff/9e0d64cdeb09d504b935e6965f30bc6c17:55
openstacking_123Is the the error ^17:55
openstacking_123But as an admin on ceph no problem17:56
*** Kurlee has quit IRC17:56
openstacking_123Seems like it is trying to parse some rados gw buckets first. Which I would assume is unrelated since the snapshots are rbd based17:57
logan-just to verify your 'ceph auth list' looks something like https://pasted.tech/pastes/f90aa96ea1f109b2fb291c6e3a8d4024408c337b for the glance key right?17:59
*** spatel has joined #openstack-ansible18:01
openstacking_123loga-- I think yours uses the older model? I have https://gist.github.com/fritzstauff/9e0d64cdeb09d504b935e6965f30bc6c . Which when compared to the ceph docs looks right18:02
openstacking_123at http://docs.ceph.com/docs/mimic/rbd/rbd-openstack/18:04
logan-ya profile rbd should be fine18:04
openstacking_123So we have another Openstack cloud deployed on rocky from a few months ago and same thing.18:05
*** openstacking_123 has quit IRC18:06
*** openstacking_123 has joined #openstack-ansible18:07
guilhermespraukadah: https://etherpad.openstack.org/p/osa-train-ptg18:08
*** irclogbot_2 has quit IRC18:08
*** irclogbot_2 has joined #openstack-ansible18:10
*** dave-mccowan has joined #openstack-ansible18:12
*** openstac_ has joined #openstack-ansible18:12
openstac_logan- Seems like this issue to me http://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-January/015682.html18:13
openstac_Will let you know18:13
logan-yeah I was just looking at that thread too. kind of weird their docs seem to differ18:13
openstac_Indeed18:13
logan-i checked another cluster which uses the 'profile rbd' caps18:14
logan-https://pasted.tech/pastes/fad4d37056604932b961c3e9c66cc2539f82f1db18:14
logan-the difference from yours is that my caps apply 'profile rbd' to mgr18:15
*** openstacking_123 has quit IRC18:15
openstac_Excellent will test that now18:16
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-nspawn_hosts master: debian: add support  https://review.openstack.org/65207118:18
spatelstein - should i put that in production ?18:18
*** tosky has joined #openstack-ansible18:18
*** cshen has joined #openstack-ansible18:19
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-nspawn_hosts master: debian: add support  https://review.openstack.org/65207118:20
*** cshen has quit IRC18:23
raukadahguilhermesp: thanks!18:24
openstac_logan- looks like same issue18:25
openstackgerritJesse Pretorius (odyssey4me) proposed openstack/ansible-role-python_venv_build master: Ensure the build host is the first repo server  https://review.openstack.org/65211018:28
openstac_logan- so it seems like all rgw buckets had issues. We added them as rbd profiles for now. We don't know what the proper fix is yet. But this will do for now18:40
openstac_On thing openstack ansible makes the rgw buckets toward the end. So there maybe some differnt procedure in place for those.18:41
logan-yeah that's weird. i have rgw running on the 'profile rbd' cluster and no issues like this :/18:42
openstac_interesting will check that18:44
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-nspawn_hosts master: debian: add support  https://review.openstack.org/65207118:46
*** openstac_ has quit IRC18:54
*** openstacking_123 has joined #openstack-ansible18:57
openstacking_123logan- ceph auth caps client.glance  mon "profile rbd" osd "allow class-read object_prefix rbd_children, profile rbd pool=volumes, profile rbd pool=images"18:58
openstacking_123Looks like maybe the long term fix for mimic18:59
logan-nice18:59
openstacking_123Anyways that is working for us. Thanks again for all the help18:59
logan-np, good to know for my luminous->mimic notes. thanks19:00
kplantno matter how many different ways i try, my multinode deployment of stable/rocky ends up corrupting the database during setup-openstack.yml: http://paste.openstack.org/show/749260/ i cleaned up the output with jq19:00
kplantnot sure where i'm going wrong here19:00
openstackgerritMerged openstack/ansible-role-python_venv_build master: Delete constraints and requirements files on build fail  https://review.openstack.org/65159819:02
*** openstacking_123 has quit IRC19:04
openstackgerritVadim Kuznetsov proposed openstack/openstack-ansible-os_octavia master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/65204919:07
*** openstacking_123 has joined #openstack-ansible19:08
kplantos is centos 7, using lxc containers and an install from source19:09
openstackgerritMerged openstack/ansible-role-python_venv_build master: Ensure venv_wheel_build_enable is evaluated as boolean  https://review.openstack.org/65210819:17
openstacking_123kplant you doing any bonding. Can you describe your network setup between nodes.19:17
kplantno bonding, it's a very simple poc environment. there are 10 kvm guests (1 deploy, 3 infra, 3 ceph, 3 compute)'19:18
kplanteach machine has a nic mapped to a specific network: external, mgmt, vxlan, vlan and storage19:18
openstacking_123kplant what tag are you on?19:19
openstacking_123git tag19:19
kplant18.1.6-6-g7aff56619:20
openstacking_123kplant I would guess that is the issue. I would run  18.1.5  per the docs19:21
openstacking_123Unless you are doing development you are going to run into trouble doing antying besides stable 18.1.519:22
openstacking_123https://docs.openstack.org/project-deploy-guide/openstack-ansible/rocky/deploymenthost.html19:22
kplantokay, i can definitely give that a try19:23
*** phasespace has joined #openstack-ansible19:23
kplantdoes it make sense for something in a stable branch to purposefully not be stable though?19:23
*** nurdie has quit IRC19:24
openstacking_123kplant speaking from experince just stable will give you issues19:24
kplantunderstood. i'm definitely going to give it a try, something i haven't done yet19:25
mnaseropenstacking_123: let's not spread fud19:28
mnaserstable branches are fine.  they're tested.  tagged releases are nothing but stable branches that are tagged19:28
mnaseryou can track stable with no problems19:28
*** cshen has joined #openstack-ansible19:29
kplant^ that's what i was thinking19:29
*** spatel has quit IRC19:30
openstacking_123mnaser not meaning to but.  I have done a lot of deploys and always hit issues if I am not using the one in doc.19:31
mnaserright.  I'd be happy to look into those issues, but there's no reason to call stable problematic, tags have failed in the past due to third parties also failing19:32
kplantmnaser: do you have any guesses as to what keeps blowing up the database?19:34
*** openstacking_123 has quit IRC19:38
*** openstacking_123 has joined #openstack-ansible19:39
logan-check the mysql logs in /var/log/mysql_logs on the galera container. also check for system issues like OOM on the node hosting the galera container. from your log, you lost the connection to mysql.. now you have to figure out why that happened19:40
jsquarewe're using the nfs driver for cinder, stable/rocky, instances don't boot up, no errors whatsoever, console shows no boot device found, fdisk -l on the image files show no partitions19:41
kplanti did check out the galera error logs but didn't even think to check something as simple as memory19:41
jsquareanyone seen this?19:41
logan-kplant: if you do find that galera oomed or some resource issue like that, you might take a look at https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2 for some ideas on tuning for test systems. all of the tuning there allows for an OSA deploy on a 8vcpu/8gb test node in the openstack ci.19:43
kplanti do have each infra guest as 8c/16g but.. with ceph mons and mgrs on there19:46
kplantit might be pushing it over19:46
*** openstacking_123 has quit IRC19:48
logan-it would not surprise me. i've had issues running 16gb infra nodes before. 32gb is the smallest i've run recently, no issues there.19:48
openstackgerritMerged openstack/openstack-ansible-ceph_client stable/rocky: Remove unnecessary GPG keys setting for YUM repos.  https://review.openstack.org/65052819:49
kplanti already toredown the env but i'm going to rebuild it with 32g infra nodes19:50
kplantthat's a good idea, i hope you're right19:50
mnaserjsquare: do you actually see the files there?19:53
jsquaremnaser: yes, the files are there19:57
*** openstacking_123 has joined #openstack-ansible19:57
*** starborn has quit IRC19:58
jsquareif we try to list the partitions on the volume, there are none, I suspect that is the problem, don't know why19:59
mnaserjsquare: so /var/lib/nova/instances/<whatever> is there right?19:59
mnaserwait this is cinder19:59
mnaserso I mean your compute node has that mounted19:59
jsquareyes19:59
jsquarethe compute node has it mounted19:59
mnaserso are you doing boot from volume? did you make sure you create a volume from an image?19:59
jsquareyes, launch instance, from image/create volume20:00
jsquareother deployments, non-OSA, don't show this behavior, not saying it's OSA's fault though20:01
*** openstacking_123 has quit IRC20:05
openstackgerritMerged openstack/openstack-ansible-lxc_container_create master: Updated from OpenStack Ansible Tests  https://review.openstack.org/64590420:06
*** luksky has joined #openstack-ansible20:07
*** partlycloudy has quit IRC20:23
*** pcaruana has quit IRC20:26
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-nspawn_hosts master: debian: add support  https://review.openstack.org/65207120:28
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-nspawn_container_create master: debian: add support  https://review.openstack.org/65207520:29
*** Darcidride has joined #openstack-ansible20:30
*** aludwar has quit IRC20:37
mnaserjsquare: maybe try creating a volume on nfs manually from image20:41
mnaserand see if that imag has the right data there?20:41
openstackgerritGuilherme  Steinmuller Pimentel proposed openstack/openstack-ansible-nspawn_hosts master: debian: add support  https://review.openstack.org/65207120:50
*** cmart has quit IRC20:51
*** cshen has quit IRC20:52
openstackgerritVadim Kuznetsov proposed openstack/openstack-ansible-os_octavia master: Add support for using distribution packages for OpenStack services  https://review.openstack.org/65204921:02
*** cmart has joined #openstack-ansible21:04
guilhermespcores, could I get votes here https://review.openstack.org/#/c/652075/ ? needs to merge this in order to restore job voting here https://review.openstack.org/#/c/652071/21:12
*** cmart has quit IRC21:30
*** nurdie has joined #openstack-ansible21:32
*** cmart has joined #openstack-ansible21:33
*** kplant has quit IRC21:36
*** kplant has joined #openstack-ansible21:36
*** nurdie has quit IRC21:37
*** cmart has quit IRC21:38
*** cmart has joined #openstack-ansible21:41
*** marst has quit IRC21:46
*** hamzaachi has quit IRC21:56
*** Darcidride has quit IRC22:06
*** Darcidride has joined #openstack-ansible22:08
openstackgerritMohammed Naser proposed openstack/ansible-role-python_venv_build stable/stein: Delete constraints and requirements files on build fail  https://review.openstack.org/65216122:29
djhankbHow is the rsyslog configured when its deployed? In my deployment I had configured a host/ip in the "log_hosts" section - but I'm not exactly sure where the syslog data is supposed to go. Looking in the rsyslog container I don't have any logs and rsyslog doesn't seem to be configured at all. Are there some variables in user_variables.yml that would be required to get this working?"22:35
*** cshen has joined #openstack-ansible22:49
*** cshen has quit IRC22:53
openstackgerritMerged openstack/openstack-ansible-nspawn_hosts master: debian: add support  https://review.openstack.org/65207122:57
*** dave-mccowan has quit IRC23:00
*** partlycloudy has joined #openstack-ansible23:09
*** tosky has quit IRC23:11
*** luksky has quit IRC23:19
*** nicolasbock has quit IRC23:30

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!