| *** openstack has joined #openstack-ansible | 15:46 | |
| *** leifmadsen_ has joined #openstack-ansible | 15:46 | |
| *** woodard has quit IRC | 15:47 | |
| *** woodard has joined #openstack-ansible | 15:47 | |
| cloudnull | morning | 15:47 |
|---|---|---|
| odyssey4me | adac this would tell you the running state: ansible galera_container -m shell -a "cat /var/lib/mysql/grastate.dat" | 15:47 |
| adac | odyssey4me, exactly yes | 15:48 |
| *** cfarquhar has joined #openstack-ansible | 15:49 | |
| *** cfarquhar has quit IRC | 15:49 | |
| *** cfarquhar has joined #openstack-ansible | 15:49 | |
| *** Mudpuppy has quit IRC | 15:51 | |
| *** alkari has joined #openstack-ansible | 15:51 | |
| *** alextricity has joined #openstack-ansible | 15:53 | |
| *** Mudpuppy has joined #openstack-ansible | 15:53 | |
| *** agireud has joined #openstack-ansible | 15:53 | |
| *** grumpycatt has quit IRC | 15:54 | |
| *** leifmadsen has quit IRC | 15:55 | |
| openstackgerrit | Major Hayden proposed openstack/openstack-ansible-security: Updating tests for openstack-ansible-security https://review.openstack.org/251430 | 15:55 |
| *** leifmadsen_ is now known as leifmadsen | 15:56 | |
| *** jasondotstar has joined #openstack-ansible | 15:58 | |
| mhayden | if someone could check my tox work there ^^ i'd be much obliged | 16:01 |
| mhayden | that'll unblock some of my future work | 16:01 |
| adac | Even tough I restarted the mysql deamons on the lxc cotainers (one with "/etc/init.d/mysql start --wsrep-new-cluster" and the other with "/etc/init.d/mysql start") A 'cat' still shows me this: https://gist.github.com/anonymous/eaaa3d336a6671c15520 Any ideas? | 16:01 |
| *** mss has joined #openstack-ansible | 16:03 | |
| *** stevelle_ is now known as stevelle | 16:03 | |
| *** woodard has quit IRC | 16:08 | |
| *** woodard has joined #openstack-ansible | 16:08 | |
| *** openstackstatus has joined #openstack-ansible | 16:14 | |
| *** ChanServ sets mode: +v openstackstatus | 16:14 | |
| adac | hmm nope no luck. tried it again. The clsuter does not come up | 16:15 |
| adac | *cluster | 16:16 |
| openstackgerrit | Miguel Alex Cantu proposed openstack/openstack-ansible: Add documentation for HA ceilometer https://review.openstack.org/248905 | 16:17 |
| spotz | adac I think this might help http://galeracluster.com/documentation-webpages/restartingcluster.html | 16:19 |
| *** alkari has quit IRC | 16:22 | |
| adac | spotz, thanks! looks promising! | 16:22 |
| spotz | adac you're welcome. Haven't done a lot with galera but I'll help if I can | 16:23 |
| adac | spotz, Yes this looks like the solution! :-) | 16:23 |
| spotz | sweet! | 16:24 |
| *** alkari has joined #openstack-ansible | 16:24 | |
| *** pabelanger has joined #openstack-ansible | 16:24 | |
| *** metral_zzz has joined #openstack-ansible | 16:25 | |
| *** metral_zzz is now known as metral | 16:25 | |
| adac | spotz, ultrasweet :-) | 16:26 |
| alextricity | Is OSA going to eventually use submodules pointing to the new role repos? | 16:26 |
| kysse | odyssey4me: PortBindingFailed: Binding failed for port e97bcaef-1750-45f5-9a37-76e973c1f0d6, please check neutron logs for more information. sounds familiar? | 16:26 |
| spotz | :) | 16:26 |
| kysse | nova's log when creating instance. | 16:26 |
| Sam-I-Am | odyssey4me: you around? | 16:27 |
| *** woodard has quit IRC | 16:28 | |
| *** woodard has joined #openstack-ansible | 16:29 | |
| palendae | alextricity: The plan was galaxy roles | 16:31 |
| palendae | Instead of straight submodules | 16:32 |
| cloudnull | alextricity: ++ what palendae said | 16:32 |
| alextricity | then just ansible-galaxy <role-name> what you need? | 16:34 |
| alextricity | or 'ansible-galaxy *install*, rather | 16:34 |
| cloudnull | we're building out the ansible-rolerequirements.yml file | 16:36 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/master/ansible-role-requirements.yml | 16:37 |
| cloudnull | whis is resolved https://github.com/openstack/openstack-ansible/blob/master/scripts/bootstrap-ansible.sh#L88-L90 | 16:37 |
| *** grumpycatt has joined #openstack-ansible | 16:38 | |
| alextricity | ohh i see. cool | 16:42 |
| alextricity | thanks :) | 16:42 |
| *** sdake has joined #openstack-ansible | 16:42 | |
| odyssey4me | Sam-I-Am sort-of :) | 16:42 |
| odyssey4me | kysse from what I saw in #openstack you likely have a config error of some sort - it may relate to some missing config on the hosts involved... without some depth of knowledge in your openstack_user_config and user_variables, it's hard to tell... I'm unfortunately on a train heading home, so I don't have time to assist right now - perhaps someone else can help? Apsu ? | 16:44 |
| Sam-I-Am | odyssey4me: i was looking, but the config looked ok | 16:44 |
| odyssey4me | alextricity if you're working with master, you'll now find the roles in /etc/ansible/roles/ | 16:44 |
| *** mgoddard_ has joined #openstack-ansible | 16:44 | |
| odyssey4me | alextricity and no, no submodules - we're trying to work with ansible in the way it was designed | 16:45 |
| *** sdake_ has joined #openstack-ansible | 16:46 | |
| *** sdake has quit IRC | 16:46 | |
| alextricity | Are there any plans to move the openstack roles as well? | 16:47 |
| *** mgoddard__ has quit IRC | 16:48 | |
| Apsu | What am I helping with? port binding failures... | 16:48 |
| Apsu | kysse: What's the Neutron log say? | 16:49 |
| *** woodard has quit IRC | 16:49 | |
| *** woodard has joined #openstack-ansible | 16:50 | |
| stevelle | alextricity: yes, all in increments | 16:52 |
| *** sacharya has joined #openstack-ansible | 16:52 | |
| *** adac has quit IRC | 16:53 | |
| kysse | Apsu: compute nodes or neutron server nodes log? | 16:54 |
| Apsu | kysse: Compute. | 16:55 |
| Sam-I-Am | kysse: do you see any errors in the agent logs in the agent containers or just compute? | 16:55 |
| kysse | lets look | 16:55 |
| Apsu | It was a port binding error on starting an instance, and directed them to the compute node's Neutron log. I suspect it's a vif binding failure | 16:55 |
| kysse | yes but what causes vif binding failure? | 16:56 |
| Sam-I-Am | kysse: usually bad bridge mappings | 16:56 |
| Apsu | The log usually has some info about that, but yeah | 16:56 |
| Sam-I-Am | at least one of the more common problems | 16:56 |
| *** sacharya has quit IRC | 16:56 | |
| Apsu | Usually bad ml2 config | 16:56 |
| kysse | ok, I try to get a real info packet for you guys. | 16:57 |
| *** sacharya has joined #openstack-ansible | 16:58 | |
| *** Mudpuppy has quit IRC | 17:00 | |
| *** mgoddard_ has quit IRC | 17:02 | |
| *** mgoddard has joined #openstack-ansible | 17:02 | |
| *** Mudpuppy has joined #openstack-ansible | 17:06 | |
| *** sdake_ has quit IRC | 17:06 | |
| *** woodard has quit IRC | 17:10 | |
| *** woodard has joined #openstack-ansible | 17:11 | |
| odyssey4me | alextricity yep, all the roles are moving - it just takes time to get it done and cloudnull is doing them in small groups | 17:15 |
| *** rebase has joined #openstack-ansible | 17:19 | |
| cloudnull | hughsaunders: still around ? | 17:20 |
| kysse | Sam-I-Am: Apsu http://sprunge.us/iFKS :') | 17:22 |
| Apsu | kysse: Is the network you created a flat one? | 17:23 |
| kysse | vlan | 17:24 |
| Apsu | Ok. Is p1p2 already in a bridge? That's why I'm asking really | 17:24 |
| Apsu | ip link show p1p2, look for "master ___" | 17:25 |
| *** tiagogomes has quit IRC | 17:25 | |
| kysse | http://paste.nerv.fi/54101194.txt | 17:25 |
| Apsu | Doesn't matter here for making a vlan network, but if it is.. won't work if you try to make a flat later | 17:26 |
| Apsu | Yeah | 17:26 |
| Apsu | Can't do it that way. | 17:26 |
| kysse | oh, is there a real reason why? | 17:26 |
| Apsu | Neutron creates bridges and puts interfaces you map into them. | 17:26 |
| Apsu | You can't put the same interface into two bridges. | 17:26 |
| kysse | yes you can | 17:26 |
| kysse | hmh. lol. you may be right | 17:27 |
| Apsu | :) | 17:27 |
| kysse | well.. | 17:28 |
| Sam-I-Am | Apsu: we're not automagically doing that, right? | 17:28 |
| Apsu | You can cheat a little if you need both flat and vlan | 17:28 |
| Apsu | Sam-I-Am: Nope | 17:28 |
| Sam-I-Am | i thought we included some magic to make this possible | 17:28 |
| kysse | I need only vxlan and vlan with same interface ;-) and vxlan in different vlan | 17:28 |
| Apsu | Sam-I-Am: There's an example of it in the example network configs | 17:28 |
| Apsu | But we don't do host network config for you | 17:28 |
| Apsu | Except the AIO does some of the magic for you | 17:28 |
| Sam-I-Am | Apsu: i thought our host network config covered this situation because people generally want to use flat and vlan networks on the same underlying interface | 17:29 |
| Apsu | kysse: Ok, cool. Well that's pretty easy | 17:29 |
| Sam-I-Am | and on a regular host i think this works... except we have br-vlan to deal with | 17:29 |
| Apsu | Just make a subinterface for the VXLAN vlan, and put it in br-vxlan | 17:29 |
| Apsu | And take the flat mapping out | 17:29 |
| kysse | ah, lol, so I'm bridging bridge to bridge | 17:30 |
| Apsu | So fun fact, the interface that you see when you create a bridge is *not* a bridge. | 17:30 |
| Apsu | It's a port on that bridge, and it can itself be bridged. | 17:30 |
| Apsu | Or subinterfaced with a VLAN tag | 17:31 |
| *** woodard has quit IRC | 17:31 | |
| Apsu | Just spin a top to see what level of bridging you're in ;) | 17:31 |
| kysse | hmh. can you make me a example interface file (just something) to clear my head | 17:31 |
| *** woodard has joined #openstack-ansible | 17:31 | |
| *** sacharya has quit IRC | 17:31 | |
| Apsu | Well, what the result looks like as I describe is something like this | 17:32 |
| Apsu | p1p2 ... master br-vlan | 17:32 |
| Apsu | p1p2.401 ... master br-vxlan | 17:32 |
| Apsu | And in the ml2, vlan: br-vlan, vxlan: br-vxlan | 17:32 |
| *** daneyon_ has joined #openstack-ansible | 17:32 | |
| kysse | ok | 17:32 |
| Apsu | Using 401 as an example VLAN ID for VXLAN traffic, of course | 17:33 |
| Apsu | The only tricky or unexpected thing here is just that folks are used to assuming that the bridge interface they can see *is* the bridge | 17:33 |
| Apsu | And it's true they're linked in some ways | 17:34 |
| Apsu | But it's also just an interface | 17:34 |
| Apsu | So you can do interface-oriented things, like bridge it | 17:34 |
| Apsu | I think of these things like switches and cables. | 17:34 |
| Apsu | Making a bridge creates a switch and attaches a cable, to start | 17:34 |
| Apsu | Putting other interfaces in it attaches more cables | 17:34 |
| Apsu | You can use the first cable to hook into another switch, of course | 17:35 |
| *** daneyon__ has joined #openstack-ansible | 17:35 | |
| Sam-I-Am | Apsu: a mapping for vxlan in ml2? | 17:35 |
| *** daneyon has quit IRC | 17:36 | |
| Apsu | Sam-I-Am: A mapping for the bridge for it. Oh, right, it doesn't work that way. It just needs the endpoint IP | 17:36 |
| Sam-I-Am | yarp | 17:36 |
| Apsu | You still need to bridge the subinterface, but don't have to specify the vxlan mapping, because you don't specify --provider:physical_network on VXLAN | 17:36 |
| Apsu | @ kysse | 17:36 |
| Sam-I-Am | the ansible magic uses that, but ml2 itself does not | 17:36 |
| Apsu | Sam-I-Am is right | 17:37 |
| Sam-I-Am | it sometimes happens | 17:37 |
| Apsu | Sometimes. | 17:37 |
| kysse | Apsu: http://paste.nerv.fi/69548198.txt running configuration like this now. So i should change.. | 17:38 |
| Apsu | kysse: So if you take the 'flat' mapping out and restart the neutron-linuxbridge-agent, do the things work? In theory it's unrelated to your problem since you were using a VLAN type network... but... | 17:38 |
| Apsu | Neutron is picky | 17:38 |
| *** daneyon_ has quit IRC | 17:38 | |
| Apsu | That's actually fine and correct. | 17:38 |
| *** karimb has quit IRC | 17:38 | |
| kysse | ... | 17:38 |
| Apsu | Just take the flat mapping out of the ml2 | 17:38 |
| kysse | ok | 17:38 |
| cloudnull | any cores about (or anyone for that matter) that would | 17:39 |
| Apsu | Wait, hold on... | 17:39 |
| cloudnull | be able to test https://review.openstack.org/#/c/241483/ ? | 17:39 |
| Apsu | http://sprunge.us/WJgh your ml2_conf.ini | 17:39 |
| Apsu | You don't have any mappings specified here... I guess it's using the ones from the network type sections | 17:40 |
| Apsu | No, because they're just the identifiers, not interface names | 17:40 |
| *** egonzalez has quit IRC | 17:40 | |
| Apsu | Oh, they're in linuxbridge_agent.ini | 17:41 |
| Apsu | derp | 17:41 |
| kysse | yes, ansible does tha :') | 17:41 |
| kysse | how derp is that?! | 17:41 |
| Apsu | lol. I will withhold my commentary | 17:42 |
| Apsu | Huh. We should probably be using bridge_mappings instead of interface_mappings. | 17:43 |
| Apsu | Since we're providing bridges | 17:43 |
| kysse | so if I change it to bridge_mappings = vxlan:br-vxlan,vlan:br-vlan | 17:44 |
| Sam-I-Am | Apsu: bridge_mappings is an ovs thing | 17:44 |
| kysse | ah | 17:44 |
| Apsu | Sam-I-Am: Not according to the docs (source code) | 17:44 |
| Sam-I-Am | Apsu: linky | 17:44 |
| Apsu | https://github.com/openstack/neutron/blob/c8a7d9bfdba9cab82cc29f563a387d8c3088c630/neutron/plugins/ml2/drivers/linuxbridge/agent/linuxbridge_neutron_agent.py#L98 | 17:44 |
| Apsu | Uses it throughout the agent | 17:44 |
| Apsu | Basically for when you have bridges already, and Neutron will just use them. | 17:45 |
| Apsu | kysse: Don't think it should matter for your case, because it works either way for everyone | 17:45 |
| kysse | lets try | 17:45 |
| Apsu | Just commenting, bridge_mappings is more correct seemingly | 17:45 |
| Sam-I-Am | Apsu: thats sort of a new not-well-tested thing | 17:46 |
| Apsu | Neat | 17:47 |
| kysse | aaaaaand failed! what a suprise! | 17:47 |
| Apsu | Sounds like "Stable" in Neutron land | 17:47 |
| kysse | surprise* | 17:47 |
| Sam-I-Am | i think the idea was that people did not like brq### stuff, so there's an option for neutron to use existing bridges | 17:47 |
| *** javeriak has quit IRC | 17:48 | |
| Sam-I-Am | Apsu: because creating brq### and plugging the bare interface into it effectively moves the ip configuration of the bare interface to the bridge | 17:48 |
| Sam-I-Am | frankly i see it as a corner case thats solvable in better ways | 17:48 |
| Sam-I-Am | but alas, neutron | 17:49 |
| Sam-I-Am | no one really understands networking, so you get patches like this | 17:49 |
| kysse | its a real problem | 17:49 |
| kysse | coders are not network engineers | 17:49 |
| Apsu | Indeed. | 17:51 |
| *** javeriak has joined #openstack-ansible | 17:51 | |
| Apsu | kysse: So what we really need to see are neutron logs when you try to create an instance, showing the failure from its perspective | 17:51 |
| kysse | from compute neutron bridge agent? | 17:51 |
| Apsu | Might need to turn on verbose and/or debug to see useful info | 17:51 |
| Apsu | Yeah | 17:51 |
| Apsu | afk 5-10 min, grabbing some food | 17:51 |
| kysse | I'll turn debug option on.. | 17:51 |
| kysse | ye ok | 17:52 |
| *** woodard has quit IRC | 17:52 | |
| *** woodard has joined #openstack-ansible | 17:52 | |
| kysse | http://sprunge.us/IXcO no failures :/ | 17:56 |
| *** harlowja has joined #openstack-ansible | 18:10 | |
| *** b3rnard0 is now known as b3rnard0_lunch | 18:10 | |
| *** phiche1 has quit IRC | 18:10 | |
| *** woodard has quit IRC | 18:12 | |
| *** woodard has joined #openstack-ansible | 18:13 | |
| *** karimb has joined #openstack-ansible | 18:15 | |
| *** neillc has quit IRC | 18:17 | |
| *** neillc has joined #openstack-ansible | 18:18 | |
| *** stevelle has quit IRC | 18:18 | |
| *** odyssey4me has quit IRC | 18:19 | |
| *** dalees has quit IRC | 18:20 | |
| *** b3rnard0_lunch has quit IRC | 18:20 | |
| *** daneyon__ has quit IRC | 18:20 | |
| *** mss has quit IRC | 18:20 | |
| *** daneyon has joined #openstack-ansible | 18:21 | |
| *** finchd-also has quit IRC | 18:21 | |
| *** Guest83268 has quit IRC | 18:23 | |
| *** erikwilson has quit IRC | 18:23 | |
| *** xar- has quit IRC | 18:23 | |
| *** karimb has quit IRC | 18:23 | |
| *** mss has joined #openstack-ansible | 18:24 | |
| *** stevelle has joined #openstack-ansible | 18:25 | |
| *** odyssey4me has joined #openstack-ansible | 18:25 | |
| *** b3rnard0 has joined #openstack-ansible | 18:25 | |
| *** erikmwilson has joined #openstack-ansible | 18:27 | |
| *** mgagne has joined #openstack-ansible | 18:28 | |
| *** mgagne is now known as Guest63453 | 18:28 | |
| *** xar- has joined #openstack-ansible | 18:29 | |
| *** finchd has joined #openstack-ansible | 18:30 | |
| Apsu | kysse: Is that log including trying to boot an instance and it failing? | 18:31 |
| *** phiche has joined #openstack-ansible | 18:32 | |
| kysse | yes | 18:32 |
| *** woodard has quit IRC | 18:33 | |
| *** woodard has joined #openstack-ansible | 18:34 | |
| Apsu | Neat. | 18:35 |
| Apsu | Maybe restart nova-compute | 18:36 |
| *** adac has joined #openstack-ansible | 18:37 | |
| *** Guest63453 has quit IRC | 18:38 | |
| *** Guest63453 has joined #openstack-ansible | 18:38 | |
| *** Guest63453 is now known as mgagne | 18:39 | |
| *** eil397 has joined #openstack-ansible | 18:39 | |
| kysse | done already :-) | 18:44 |
| *** woodard_ has joined #openstack-ansible | 18:45 | |
| *** dalees has joined #openstack-ansible | 18:45 | |
| *** woodard has quit IRC | 18:47 | |
| *** woodard_ has quit IRC | 18:54 | |
| *** woodard has joined #openstack-ansible | 18:55 | |
| *** sdake has joined #openstack-ansible | 19:10 | |
| *** woodard has quit IRC | 19:15 | |
| *** woodard has joined #openstack-ansible | 19:15 | |
| *** mancdaz has quit IRC | 19:20 | |
| *** adac has quit IRC | 19:20 | |
| *** xek has quit IRC | 19:21 | |
| *** mancdaz has joined #openstack-ansible | 19:22 | |
| kysse | I just can't understand what's going on. Maybe I should go to the dark side and test ovs and or with packstack | 19:25 |
| Sam-I-Am | kysse: have you installed openstack before? | 19:25 |
| *** javeriak has quit IRC | 19:25 | |
| kysse | nope | 19:26 |
| Sam-I-Am | i recommend starting out with something a bit simpler that you can learn from | 19:27 |
| Sam-I-Am | trying to deploy openstack the first time using a complicated deployment tool that installs openstack in a production (read: complicated) way may not be beneficial | 19:27 |
| kysse | (compilated deployment tool that i'm familiar with) | 19:28 |
| Sam-I-Am | so there's that... | 19:28 |
| kysse | and compilicated networking wich was correct | 19:28 |
| Sam-I-Am | but using containers and stuffing neutron into them adds a lot of complexity to the picture | 19:28 |
| kysse | and compilcated setup where I dont get _any_ red color when installing :') and everything works smooth and correct but those compute nodes | 19:29 |
| cloudnull | kysse: sorry im late to the party , but reading the scroll back , are you still getting the vif type binding failure? | 19:30 |
| kysse | yes. | 19:30 |
| cloudnull | can we see you openstack_user_config.yml ? | 19:30 |
| kysse | yes. | 19:30 |
| cloudnull | idk if you pasted that before ... | 19:30 |
| * cloudnull reading more | 19:30 | |
| d9k | http://sprunge.us/iFKS | 19:31 |
| Sam-I-Am | kysse: you might get it working, but you really need to learn how openstack itself works because it'll break someday. | 19:31 |
| kysse | Sam-I-Am: you are right, and yes I'm learning how it works right now. | 19:32 |
| kysse | cloudnull: d9k's url contains all information. | 19:33 |
| cloudnull | kysse: remove http://cdn.pasteraw.com/af3bx8jx934rr761r4l7fziongg69gd from the user config | 19:33 |
| kysse | err, okey | 19:34 |
| cloudnull | w/out the host bind override neutron will not support an ml2 mapping using the same interface | 19:34 |
| cloudnull | and on compute nodes that interface would be br-vlan | 19:34 |
| cloudnull | which based on your user config would be shared with vlan and flat network types | 19:35 |
| *** adac has joined #openstack-ansible | 19:35 | |
| *** woodard has quit IRC | 19:36 | |
| *** sacharya has joined #openstack-ansible | 19:36 | |
| cloudnull | once you do that rerun the os-neutron-install.yml play | 19:36 |
| cloudnull | and you can limit the command with the tag "neutron-config" | 19:36 |
| cloudnull | at least i think you can... | 19:36 |
| *** woodard has joined #openstack-ansible | 19:36 | |
| * cloudnull brain is full | 19:36 | |
| kysse | i'm running it now, lets see. | 19:37 |
| cloudnull | in our gate test we use both flat and vlan networks together. | 19:37 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio#L40-L48 | 19:37 |
| cloudnull | but set host_bind_override to be the eth12 device | 19:38 |
| cloudnull | which is a veth https://github.com/openstack/openstack-ansible/blob/master/etc/network/interfaces.d/aio_interfaces.cfg#L53-L58 | 19:38 |
| cloudnull | kysse: also i'd recommend disabling sg-offload for your bridge devices. its been known to cause issues with network perfofrmance and general LXC we do that like so: https://github.com/openstack/openstack-ansible/blob/master/etc/network/interfaces.d/aio_interfaces.cfg#L42 | 19:41 |
| * cloudnull was reading http://paste.nerv.fi/69548198.txt | 19:41 | |
| kysse | it's not my full interfaces file, but thanks! Good advices :) | 19:42 |
| cloudnull | for sure. I figured there was more, but i thought i mention it | 19:43 |
| *** sdake has quit IRC | 19:43 | |
| *** javeriak has joined #openstack-ansible | 19:43 | |
| kysse | I ran it against compute without that part you told me, still same error. Do I need to change physical mappings? | 19:49 |
| kysse | ah, lets see. hmhm | 19:49 |
| cloudnull | the physical mappings shouldve been updated | 19:50 |
| kysse | yes they were. physical_interface_mappings = vlan:br-vlan | 19:50 |
| cloudnull | you will need to run it everywhere | 19:50 |
| cloudnull | agents need to be restarted / updated across the cluster | 19:51 |
| kysse | ah, okey | 19:51 |
| *** woodard has quit IRC | 19:56 | |
| *** woodard has joined #openstack-ansible | 19:57 | |
| *** sacharya has quit IRC | 20:05 | |
| cloudnull | kysse: anything ? | 20:05 |
| kysse | 1 sec .. or 2 .. :P | 20:06 |
| cloudnull | ok . no worries | 20:06 |
| *** woodard has quit IRC | 20:06 | |
| kysse | nope. I just rebooted all neutron containers and compute node. I'm missing something clearly. | 20:12 |
| cloudnull | so the mapping is consistent ? | 20:14 |
| cloudnull | and the agents are running i assume | 20:14 |
| cloudnull | does `neutron agent-list` return anything intersting ? | 20:14 |
| cloudnull | and do you see the neutron-lxb-agent running on the compute node? | 20:15 |
| kysse | what does :-) mean? | 20:15 |
| cloudnull | also I assume your running liberty. is that right? | 20:16 |
| kysse | yes | 20:16 |
| cloudnull | thats OpenStack for its working ... | 20:16 |
| cloudnull | which i hate ! | 20:16 |
| kysse | no I dont see neutron-lxb-agent on my compute node :-( | 20:16 |
| cloudnull | sorry neutron-linuxbridge-agent | 20:16 |
| kysse | ah, it's running yes | 20:16 |
| * cloudnull was being a lazy person | 20:16 | |
| kysse | what interfaces I should see in neutron-agent and server containers? | 20:18 |
| cloudnull | its still the "PortBindingFailed: Binding failed for port <UUID>, please check neutron logs for more information" error? | 20:19 |
| cloudnull | in the containers you should see eth0,1,11 | 20:19 |
| cloudnull | sorry | 20:20 |
| cloudnull | in the containers you should see eth0,1,10,11 | 20:20 |
| cloudnull | the networking containers that is. | 20:20 |
| cloudnull | and the host should have br-vlan,vxlan,mgmt | 20:20 |
| kysse | error is sam eyes | 20:21 |
| kysse | hmm | 20:21 |
| cloudnull | when you boot your VM are you using a vxlan network type ? | 20:22 |
| kysse | no, vlan | 20:22 |
| cloudnull | when you created the neutron networks did you do so with the range "200:400" ? | 20:23 |
| cloudnull | can you try and build a vm that is only on a vxlan network ? | 20:23 |
| kysse | ok, i'll try. | 20:23 |
| cloudnull | just to see if it completes . | 20:23 |
| cloudnull | also how were the neutron networks created ? horizon? neutron CLI ? | 20:24 |
| Apsu | Would like to see "neutron net-show $yournetwork" and "neutron subnet-show $yoursubnet" | 20:24 |
| cloudnull | ^ that | 20:24 |
| Apsu | To see more details around the VLAN ID, CIDR, provider flags, etc | 20:24 |
| kysse | I used horizon.. also tried with cli | 20:25 |
| cloudnull | w/in horizon did you set the segmentation ID of your network to be the same as your vlan range ? | 20:26 |
| kysse | same error with vxlan | 20:26 |
| cloudnull | ok so neutron is still mad | 20:26 |
| Apsu | kysse: Let's get a paste with the details on the network and subnet | 20:26 |
| kysse | yes ok | 20:26 |
| Apsu | Also, take a look at dmesg on the compute node | 20:26 |
| Apsu | In case there's some kernel messages that might shed some light | 20:27 |
| cloudnull | also is there anything else being set in the neutron-linux-bridge-agent.log within the compute node and the neutron agent container ? | 20:28 |
| kysse | http://paste.nerv.fi/27203893.txt | 20:28 |
| cloudnull | kysse: the provider:segmentation_id needs to be within the 200:400 range . | 20:29 |
| Apsu | ^^^ | 20:29 |
| Apsu | Was just pasting the lines that conflict, lol | 20:29 |
| Apsu | network_vlan_ranges = vlan:200:400, in your ml2_conf | 20:29 |
| kysse | wtf | 20:29 |
| cloudnull | what Apsu said | 20:29 |
| kysse | aaah! but hey, it didnt work with vxlan either. | 20:30 |
| Sam-I-Am | cloudnull: as an admin you can use any arbitrary segmentation id | 20:30 |
| kysse | and it was between allowed vxlans | 20:30 |
| Sam-I-Am | the range is just there for non-priv networks that cant choose vlan ids | 20:30 |
| kysse | I can try to create a network between 200:400, but I'm just saying that it's not gonn work any better. | 20:30 |
| Apsu | That rings a bell, actually. | 20:31 |
| cloudnull | kysse: this is likely, theres something else happening thats making neutron not happy. | 20:31 |
| kysse | indeed. | 20:31 |
| Apsu | Ok, let's look at dmesg on the compute. Also.... if you run "ip a", do you see any ipv6 addresses on interfaces? | 20:31 |
| cloudnull | just pointing out that the vlan network needs a segmentation id within your set vlan range . | 20:31 |
| Apsu | Or, did you purposefully unload the ipv6 module and blacklist it on your compute box? | 20:32 |
| kysse | there was no errors in dmesg. | 20:32 |
| kysse | nothing unusual | 20:32 |
| kysse | I can see link local addresses yes | 20:32 |
| Apsu | Ok | 20:32 |
| Apsu | Neutron has an obscure bug when the ipv6 module is blacklisted, so guess it's not that | 20:33 |
| cloudnull | and nothing in the neuttron-linuxbridge-agent log file ? | 20:33 |
| kysse | nope, not even with debug mode. | 20:34 |
| Apsu | cloudnull: kysse posted this earlier, it's a collection of pastes of various things http://sprunge.us/iFKS | 20:34 |
| Apsu | kysse: What about turning on debug for nova-compute and checking it on the failure | 20:34 |
| Apsu | Kind of wondering if it's not neutron but something libvirt/nova related, and its just blaming neutron | 20:35 |
| cloudnull | yea but the log is mostly empty, so was curious if its gotten more data since the original paste | 20:35 |
| cloudnull | i wonder if the issue is "WARNING oslo_config.cfg [-] Option "username" from group "neutron" is deprecated. Use option "user-name" from group "neutron"." | 20:36 |
| Sam-I-Am | cloudnull: nah, thats all jamie lennox | 20:37 |
| Sam-I-Am | in other words, bs no one wants to fix | 20:37 |
| cloudnull | it says deprecated but do they mean "removed" :) | 20:37 |
| Sam-I-Am | turns out user-name doesnt even work | 20:37 |
| cloudnull | ok nevermind then | 20:38 |
| *** KLevenstein has quit IRC | 20:40 | |
| kysse | http://sprunge.us/WcCO nova debug log when creating instance | 20:41 |
| *** mfisch` has quit IRC | 20:56 | |
| *** mfisch has joined #openstack-ansible | 20:57 | |
| *** mfisch is now known as Guest7150 | 20:57 | |
| kysse | no comments? :P | 20:57 |
| cloudnull | sorry was looking elsewhere | 20:58 |
| cloudnull | the raised exception is here https://github.com/openstack/nova/blob/stable/liberty/nova/network/neutronv2/api.py#L342 | 20:59 |
| cloudnull | kysse: when you reran the os-neutron-install.yml play did you do so with or without a tag / limit ? | 21:01 |
| *** Guest7150 has quit IRC | 21:01 | |
| *** javeriak has quit IRC | 21:01 | |
| kysse | openstack-ansible setup-everything.yml --tags neutron-config | 21:02 |
| cloudnull | i hate to ask however, you can you run ``openstack-ansible os-neutron-install.yml`` and then try to create the vm using a vxlan network ? | 21:02 |
| kysse | sure. sec. | 21:03 |
| cloudnull | maybe the tag i told you is missing a step ... | 21:03 |
| cloudnull | but the only thing i can think of that would cause that is a busted ml2 config | 21:03 |
| *** metral is now known as metral_zzz | 21:03 | |
| *** metral_zzz is now known as metral | 21:04 | |
| cloudnull | and it could be in your agent container(s), or the compute node(s) | 21:04 |
| Apsu | It seems a little light to me, but then I know we split out some LB things into its own file... | 21:04 |
| kysse | maybe I and d9k should contribute openstack-ansible's documentation and stuff, we saw that there is lots of information missing from documentation. | 21:05 |
| kysse | btw. | 21:05 |
| cloudnull | kysse: d9k that would be awesome | 21:05 |
| Apsu | Patches welcome! Encouraged. Lauded. | 21:05 |
| cloudnull | doc updates help everyone, and we've done our best to make the docs what they are however they could use some more love thats for sure. | 21:07 |
| Sam-I-Am | plenty of love needed | 21:07 |
| Sam-I-Am | especially around that finicky host_bind_override thing | 21:07 |
| cloudnull | back in a min, making food | 21:11 |
| kysse | cloudnull: no luck. | 21:11 |
| *** adac has quit IRC | 21:14 | |
| *** phiche1 has joined #openstack-ansible | 21:17 | |
| *** phiche has quit IRC | 21:17 | |
| cloudnull | kysse: hum... | 21:21 |
| Apsu | kysse: cloudnull: Sam-I-Am: "binding:vif_type": "binding_failed" | 21:23 |
| Apsu | This is the standard vif_type binding_failed error | 21:23 |
| kysse | I see that when I query mysql neutron something.. | 21:23 |
| Apsu | Also... from the debug output of nova-compute... | 21:24 |
| Apsu | I see neutron.admin_* values are None | 21:24 |
| kysse | i pasted debug output of nova-compute long time ago | 21:24 |
| Apsu | Which means nova can't log into neutron's endpoint | 21:24 |
| Apsu | That's probably the whole problem. Nova needs creds for Neutron in the nova.conf | 21:25 |
| *** sdake has joined #openstack-ansible | 21:25 | |
| Apsu | This doesn't look right to me either: neutron.admin_auth_url = http://localhost:5000/v2.0 | 21:25 |
| Apsu | Guessing keystone isn't running on your compute node | 21:26 |
| Sam-I-Am | Apsu: so i was noticing earlier, working with kysse, that neutron.conf did not have a [keystone_authtoken] section | 21:26 |
| Sam-I-Am | which i found a bit odd | 21:26 |
| Apsu | Yeah seems like there's some auth shenanigans here. | 21:26 |
| Sam-I-Am | pretty sure thats a) needed and b) something o-a has stuffed into that file for a LONG time | 21:26 |
| cloudnull | what should be "internal_lb_vip_address: 10.0.8.4" based on the user_config | 21:28 |
| *** adac has joined #openstack-ansible | 21:29 | |
| cloudnull | kysse: did you by chance run the nova-compute play w/ ``ansible-playbook`` ? or ``openstack-ansible`` also w/ the updates we made earlier in the openstack_user_config file we might want to rerun the os-nova-install.yml (openstack-ansible os-nova-install.yml) | 21:33 |
| cloudnull | but that would be odd if the keystone auth secions are missing from the nova-compute nodes | 21:33 |
| kysse | with openstack-ansible | 21:34 |
| kysse | hmh. Maybe I should reinstall whole compute node or something.. | 21:34 |
| *** karimb has joined #openstack-ansible | 21:34 | |
| cloudnull | just rerun ``openstack-ansible os-nova-install.yml --tags nova-config`` | 21:34 |
| kysse | ok | 21:35 |
| cloudnull | the adminurl shouldve been defined here " https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_nova/templates/nova.conf.j2#L167" | 21:35 |
| cloudnull | while thats running you mind openning the nova.conf file on the compute node and seeing if auth_url is in the [neutron] section | 21:36 |
| Apsu | cloudnull: And what about the admin_username and tenant_id and all that for neutron, too? | 21:39 |
| Apsu | I assume that's still required for nova these days | 21:39 |
| cloudnull | https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_nova/templates/nova.conf.j2#L154-L169 | 21:40 |
| cloudnull | all that should be in the file | 21:40 |
| * Apsu nods | 21:40 | |
| Apsu | So either the template didn't run, or the variables are blank or something | 21:40 |
| Apsu | All blank/missing would mean it'd use defaults, and I bet the auth URL default is localhost | 21:41 |
| *** sdake_ has joined #openstack-ansible | 21:47 | |
| *** sdake has quit IRC | 21:47 | |
| kysse | cloudnull: no luck. We also checked those neutron.admin_auth urls etc. | 21:47 |
| *** mfisch has joined #openstack-ansible | 21:48 | |
| *** adac has quit IRC | 21:48 | |
| *** mfisch has quit IRC | 21:48 | |
| *** mfisch has joined #openstack-ansible | 21:48 | |
| *** KLevenstein has joined #openstack-ansible | 21:51 | |
| *** stevelle has quit IRC | 21:53 | |
| cloudnull | do they say localhost ? | 21:53 |
| *** stevelle has joined #openstack-ansible | 21:56 | |
| kysse | inner lb vip | 21:57 |
| *** alkari has quit IRC | 21:57 | |
| cloudnull | kysse: and you have [keystone_authtoken] in the nova.conf | 22:00 |
| cloudnull | and its auth_ur* entries are using the internal vip too ? | 22:00 |
| *** coolj has left #openstack-ansible | 22:01 | |
| *** sdake_ has quit IRC | 22:02 | |
| cloudnull | kysse: on the compute node, are you restarting the service w/ "service nova-compute restart" | 22:04 |
| cloudnull | have you started it w/ backgrounding the command nova-compute ? | 22:04 |
| *** markvoelker has joined #openstack-ansible | 22:07 | |
| Apsu | Yeah seems like maybe it's not reading the configs | 22:11 |
| cloudnull | whats odd is I see "2015-11-30 22:39:45.641 5513 DEBUG oslo_service.service [req-4aab2824-7df8-4159-9914-ce087783fabc - - - - -] oslo_messaging_rabbit.rabbit_host = localhost" | 22:12 |
| cloudnull | and 2015-11-30 22:39:45.644 5513 DEBUG oslo_service.service [req-4aab2824-7df8-4159-9914-ce087783fabc - - - - -] neutron.admin_auth_url = http://localhost:5000/v2.0 log_opt_values /openstack/venvs/nova-12.0.1/lib/python2.7/site-packages/oslo_config/cfg.py:2233 | 22:12 |
| cloudnull | the rabbit one makes sense | 22:12 |
| cloudnull | because we use rabbit_hosts | 22:12 |
| cloudnull | which has the correct values of the rabbit nodes ['10.0.9.97:5671', '10.0.9.82:5671'] | 22:12 |
| cloudnull | but "neutron.admin_auth_url = http://localhost:5000/v2.0" seems wrong... | 22:13 |
| cloudnull | unless we've missed something | 22:13 |
| cloudnull | kysse: i have a new liberty build going on now | 22:13 |
| cloudnull | and will be able to see if i can test the same things here in a min | 22:14 |
| *** spotz is now known as spotz_zzz | 22:14 | |
| cloudnull | kysse: Apsu im thinking that https://github.com/openstack/openstack-ansible/blob/master/playbooks/roles/os_nova/templates/nova.conf.j2#L167 | 22:14 |
| cloudnull | should be admin_auth_url = ... | 22:15 |
| Apsu | yeah | 22:16 |
| kysse | hmh. | 22:16 |
| Apsu | pgrep -fa nova-compute | 22:16 |
| Apsu | See what parameters its running with | 22:16 |
| *** spotz_zzz is now known as spotz | 22:19 | |
| *** daneyon has quit IRC | 22:23 | |
| *** mancdaz has quit IRC | 22:26 | |
| *** mancdaz has joined #openstack-ansible | 22:28 | |
| cloudnull | kysse: so this is not your fault. you wouldve had the same issue because of the openstack_user_config change we updated a while back however the config options in the nova.conf file have recently changed... | 22:29 |
| cloudnull | and i think its a change in keystonemiddleware that is causing the issue | 22:29 |
| kysse | ah, so I'm not crazy afterall. | 22:30 |
| cloudnull | not at all | 22:31 |
| cloudnull | its just taken me a while to tracck the issue down | 22:31 |
| cloudnull | sorry about that | 22:31 |
| kysse | no problems. | 22:32 |
| *** lkoranda_ has joined #openstack-ansible | 22:48 | |
| *** Mudpuppy has quit IRC | 22:49 | |
| *** lkoranda has quit IRC | 22:50 | |
| *** lkoranda_ has quit IRC | 22:52 | |
| *** metral has quit IRC | 22:56 | |
| cloudnull | kysse: can you try something for me | 22:56 |
| cloudnull | can you add "auth_plugin = password" to the [neutron] section in your nova.conf | 22:56 |
| cloudnull | on the compute node | 22:56 |
| cloudnull | run: service nova-compute restart" and then try again | 22:56 |
| kysse | well, I'd have to do all that stuff you told me a while ago. | 22:57 |
| cloudnull | this time try booting an instance w/ a vxlan network | 22:57 |
| *** metral_zzz has joined #openstack-ansible | 22:57 | |
| *** metral_zzz is now known as metral | 22:57 | |
| cloudnull | whats that ? | 22:57 |
| *** lkoranda has joined #openstack-ansible | 22:58 | |
| *** phiche1 has quit IRC | 22:58 | |
| kysse | ammmm. nothing. I'll test service nova-com... sec. | 22:58 |
| kysse | it's already there! but the password is password | 22:59 |
| cloudnull | auth_plugin = password is already there ? | 22:59 |
| kysse | yes indeed. | 23:00 |
| cloudnull | ok | 23:00 |
| kysse | /etc/nova/nova.conf @compute | 23:00 |
| cloudnull | in the neutron section right ? | 23:00 |
| *** tlian2 has joined #openstack-ansible | 23:00 | |
| kysse | y | 23:01 |
| *** tlian has quit IRC | 23:03 | |
| cloudnull | ok last thing, try adding "admin_" to the auth_url, password, username to the opt in the [neutron[ section | 23:04 |
| cloudnull | restart nova-compute and build a vm | 23:05 |
| Apsu | Did we verify it's actually reading these configs? | 23:06 |
| cloudnull | it is | 23:06 |
| Apsu | pgrep -fa nova-compute, make sure the confs are in the commandline? | 23:06 |
| Apsu | ok | 23:06 |
| cloudnull | however this https://github.com/openstack/nova/blob/stable/liberty/nova/network/neutronv2/api.py#L182-L208 seems to be loading the plugin | 23:07 |
| cloudnull | which has all of the deprecated ops https://github.com/openstack/nova/blob/stable/liberty/nova/network/neutronv2/api.py#L51-L53 | 23:07 |
| cloudnull | and is not using the regular auth_plugin like it should | 23:07 |
| *** spotz is now known as spotz_zzz | 23:08 | |
| cloudnull | jamielennox: question for you when your around . | 23:08 |
| cloudnull | [neutron] auth_plugin seems to be ignoring the values of the keystone_authtoken section when auth_plugin = password | 23:09 |
| cloudnull | from the [neutron] section | 23:09 |
| Sam-I-Am | this was kilo? | 23:09 |
| Sam-I-Am | or is... | 23:09 |
| cloudnull | liberty | 23:09 |
| Sam-I-Am | cloudnull: did you see the related sections in here? http://docs.openstack.org/liberty/install-guide-ubuntu/ | 23:10 |
| cloudnull | Sam-I-Am: yes thats what we have | 23:11 |
| cloudnull | however thats not what is being loaded in nova.conf | 23:11 |
| Sam-I-Am | sure this is liberty? because it was different in kilo. | 23:12 |
| kysse | hmh | 23:12 |
| cloudnull | Sam-I-Am: yes, liberty | 23:12 |
| kysse | we're running neweset one. | 23:13 |
| kysse | newest* | 23:13 |
| Sam-I-Am | also wondering why this would have all of the sudden broke | 23:13 |
| Sam-I-Am | kysse: which tag? | 23:13 |
| cloudnull | Sam-I-Am: 12.0.1 (based on the log files) | 23:13 |
| kysse | 12.0.1 | 23:13 |
| *** baker has quit IRC | 23:14 | |
| cloudnull | Sam-I-Am: http://sprunge.us/WcCO | 23:14 |
| cloudnull | has neutron.admin_auth_url = http://localhost:5000/v2.0 | 23:14 |
| kysse | 01:04 cloudnull| ok last thing, try adding "admin_" to the auth_url, password, username to the opt in the [neutron[ section | 23:15 |
| kysse | trying this now | 23:15 |
| cloudnull | Sam-I-Am: i'd expect to see "neutron.admin_auth_url" or neutron.auth_url set to the keystone auth endpoint | 23:17 |
| kysse | ahmh. | 23:17 |
| kysse | Could not clean up failed build, not rescheduling | 23:17 |
| cloudnull | but the only two uses of port 5000 are set to localhost | 23:17 |
| kysse | http://sprunge.us/CjZG | 23:18 |
| cloudnull | kysse: with all of these tests you may need to delete a bunch of the dead vms | 23:18 |
| kysse | ok | 23:18 |
| cloudnull | well thats an odd error | 23:19 |
| kysse | http://paste.nerv.fi/77158297.txt neutron section | 23:20 |
| cloudnull | Sam-I-Am jamielennox do you know if the keystone_authtoken will simply not show up in the debug output when the config is loaded ? | 23:22 |
| cloudnull | because maybe this is a red harring ? | 23:23 |
| Sam-I-Am | cloudnull: in that [neutron] stuff, the usual username/password (rather than admin_username) should work | 23:23 |
| Sam-I-Am | cloudnull: what parts of keystone_authtoken are you looking for? | 23:24 |
| cloudnull | any of it | 23:24 |
| Sam-I-Am | doubt it | 23:24 |
| cloudnull | if i restart nova-compute that section does not show up in the running output | 23:24 |
| cloudnull | same for the neutron section | 23:25 |
| Sam-I-Am | oh, i think i've seen that | 23:25 |
| cloudnull | however the old opts do | 23:25 |
| Sam-I-Am | yeah because he never updated those | 23:25 |
| cloudnull | :'( | 23:25 |
| *** spotz_zzz is now known as spotz | 23:26 | |
| Sam-I-Am | the whole cfg.CONF is broken for keystone middleware | 23:27 |
| Sam-I-Am | and should not be used | 23:27 |
| Sam-I-Am | so you sort of just figure it out by reading code, or a blog post | 23:28 |
| cloudnull | kysse: i have to run for the evening however I may be back online later on . | 23:31 |
| cloudnull | you can rerun the os-nova-install.yml --tags nova-config | 23:31 |
| cloudnull | to restore the configs back to the way they were prior to all of the messing about. | 23:31 |
| kysse | ok. :) | 23:32 |
| Sam-I-Am | i think this went down the wrong rabbit hole | 23:32 |
| cloudnull | as for the issues, i still dont know, however I have my 10 node cluster I'm going to beat on to see if i can recreat the issues. | 23:32 |
| Sam-I-Am | just a hunch | 23:32 |
| cloudnull | Sam-I-Am: that may be however kysse still has a broken cluster which would be good to figure out why . | 23:33 |
| cloudnull | it could be something to do with our liberty code, or a random misconfiguration | 23:33 |
| cloudnull | IDK . | 23:33 |
| Sam-I-Am | given the stuff i saw that was missing earlier | 23:33 |
| Sam-I-Am | i wasnt around if y'all were solving (or solved) that | 23:34 |
| Sam-I-Am | but curious what else might be missing | 23:34 |
| cloudnull | kysse: are all of the bridges up on all of the network and compute nodes ? | 23:37 |
| cloudnull | and do you see the corresponding devices in the containers | 23:37 |
| openstackgerrit | Merged openstack/openstack-ansible: Allow ramdisk_id, kernel_id to be null on schema https://review.openstack.org/246503 | 23:38 |
| kysse | hmh. | 23:39 |
| *** sigmavirus24 is now known as sigmavirus24_awa | 23:44 | |
| *** spotz is now known as spotz_zzz | 23:49 | |
| kysse | corresponding devices as? | 23:50 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!