| opendevreview | James Denton proposed openstack/openstack-ansible master: [WIP] Track ML2 plugin for octavia-ovn-provider driver https://review.opendev.org/c/openstack/openstack-ansible/+/868461 | 03:57 |
|---|---|---|
| opendevreview | James Denton proposed openstack/openstack-ansible-os_octavia master: [WIP] Implement support for octavia-ovn-provider driver https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/868462 | 04:13 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_octavia master: [WIP] Implement support for octavia-ovn-provider driver https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/868462 | 04:18 |
| *** akahat|ruck is now known as akahat | 07:05 | |
| moha7 | I got tired of getting locked in the deployment stage: https://p.teknik.io/41RQK | 10:21 |
| moha7 | Now, I'm switching to an AIO env, but would you please share your config files: user_variables.yml and openstack_user_config.yml by hiding sensitive data for a working multinode environment | 10:23 |
| *** dviroel|out is now known as dviroel | 11:02 | |
| jrosser | moha7: there are examples in the openstack-ansible repo https://github.com/openstack/openstack-ansible/tree/master/etc/openstack_deploy | 11:07 |
| jrosser | yuo do not need really anything more for a deployment that follows the 'reference architecture' for openstack-ansible | 11:08 |
| jrosser | moha7: your paste does not render correctly here https://p.teknik.io/41RQK | 11:12 |
| moha7 | Oh, the correct link: http://ix.io/4jli | 11:15 |
| moha7 | Are those examples out of the box? Actually I'm using the multibone.example for my env: https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.multibond.example | 11:17 |
| moha7 | Here is mine: http://ix.io/4jlj <-- just replacing ovs with the default linuxbridge network backend | 11:18 |
| moha7 | And here the user_variables file: https://pastebin.com/raw/LEejpWsq | 11:21 |
| moha7 | Also, the netplan config: http://ix.io/4jlm | 11:22 |
| moha7 | Oops, the above link is the netplan conf file for the deployment server! This one is the settings on the controller/compute nodes: http://ix.io/4jln | 11:24 |
| jrosser | you should do really basic tests | 11:24 |
| jrosser | can you ping the infra node bridges from the deploy host and each other | 11:25 |
| jrosser | can you ping the containers similarly | 11:25 |
| jrosser | do you have connectivity from inside one container to the others | 11:25 |
| jrosser | most “getting started” troubles are network related, especially when multinode labs are built in VM environments | 11:26 |
| * noonedeadpunk trying to setup lxc networking with OVS at the same time :D | 11:27 | |
| jrosser | and which branch do you use? master is the development branch for the *next* release, you should be using a stable branch | 11:27 |
| noonedeadpunk | the more I go the more I feel I shouldn't do that.... | 11:31 |
| jrosser | infra node lxc + bridges seems so simple / obvious | 11:32 |
| jrosser | somehow i miss the attraction of ovs for this | 11:32 |
| kleini | noonedeadpunk: thanks for that insight. I already felt, I need to migrate my infra hosts to OVS. Now you're saying this and I will stick to compute/network nodes with OVS and infra with LB. | 11:32 |
| noonedeadpunk | kleini: well, I don't have any good reasons now. Just some feeling.... | 11:33 |
| noonedeadpunk | Eventually my main reason behind that to ease life with octavia network a bit | 11:34 |
| kleini | I trust your gut feeling! I am trusting mine very often and I mostly it proves me right. | 11:34 |
| noonedeadpunk | (behind trying out ovs) | 11:34 |
| noonedeadpunk | And also I'm not sure how/if that will help indeed if/when we will decide to move to OVN | 11:35 |
| noonedeadpunk | as imagine I will set octavia net as vxlan on controller, which is nice, with moving to OVN that will quickly can become a nightmare | 11:37 |
| noonedeadpunk | not saying that ovs or glibc upgrade causing interruptions for the controller overall... | 11:38 |
| jrosser | we did octavia net as vlan even though project networks are vxlan | 11:40 |
| jrosser | then it is easier on the controller | 11:40 |
| noonedeadpunk | but if controller is ovs... then vxlan on controller should be super easy, as they're in the same mesh? | 11:40 |
| jrosser | but was then necessary to use real vxlan on the switches to span it as L2 to all the computes | 11:41 |
| noonedeadpunk | kleini: what were your reasons for ovs on controllers? | 11:41 |
| noonedeadpunk | I thought it would be enough to pass L2 with vxlan networks to controller, not only net nodes? | 11:42 |
| noonedeadpunk | ah, yes, maybe you're right | 11:43 |
| moha7 | Yup, all of them are pingable from the other sides | 11:43 |
| kleini | I have LB on controllers/infra hosts and OVS on compute and network hosts. | 11:44 |
| noonedeadpunk | so just to align worloads? | 11:44 |
| moha7 | > and which branch do you use? ---> I'm testing on both the master and also v25.2.20 | 11:45 |
| noonedeadpunk | jrosser: well, we have that as vlan as well today... I was just trying to check if there's easier or neater way of doing that | 11:46 |
| jrosser | noonedeadpunk: from another angle we have been doing switch firmware upgrades and it has been really quite the nightmare | 11:47 |
| kleini | I think OVS ML2 integrates better with OVS bridges for the host and it requires less context switches and packet transfers. So it provides better throughput and less packet loss, is easier to debug as everything is in OVS. With OVS firewall driver it is even less networking mess on computes. | 11:48 |
| jrosser | even though it is all mlag/bonds we have had bad bugs in both nic firmware and switch firmware which are pretty catastrophic failures | 11:48 |
| jrosser | we are thinking that it would be really nice to have L3-to-the host with bgp or something and remove all the fragile L2 stuff completely | 11:49 |
| noonedeadpunk | yeah, I was also thinking why I would need l2. and given keepalived in unicast it should work in l3 | 11:51 |
| jrosser | we lost the whole bond/mlag setup in the whole control plane due to cascading bugs in nic and switch | 11:51 |
| jrosser | and even with it all fixed now the run book for how to upgrade switch firmware is really tricky | 11:52 |
| noonedeadpunk | kleini: the thing is I'm not sure how neat it would be with migration to ovn/ despite it's still ovs underneath, but not sure how that will work together | 11:53 |
| noonedeadpunk | but I'm quite bad in OVN now | 11:53 |
| noonedeadpunk | jrosser: so you think ovs on control worth a shot? | 11:55 |
| kleini | I have no idea yet about OVN, just read about it and was scared, that it should be the new default. | 11:55 |
| noonedeadpunk | as I'm not sure how ovs/lxb changing fact of having l2... | 11:55 |
| noonedeadpunk | kleini: upgrade script should take care of overrides. But the thing is that both cannoncical and redhat migrated most of their clients to ovn already. And I've heard that ovs is deprecated in redhat platform already | 11:56 |
| kleini | This information does not make me less scared :-D | 11:58 |
| noonedeadpunk | But I'm quite tired of net nodes and namespaces and how painful it always is to reboot a net node | 11:58 |
| noonedeadpunk | or even upgrade ovs there or restart l3 agent... | 11:58 |
| jrosser | noonedeadpunk: I am kind of happy with lxb on controllers | 11:58 |
| jrosser | but then I have separate network nodes anyway | 11:58 |
| noonedeadpunk | yeah, me to | 11:59 |
| jrosser | for controllers I would rather move to L3 | 11:59 |
| noonedeadpunk | net nodes are not co-located on controllers | 11:59 |
| jrosser | frr or something with hack to advertise the LXC ip | 11:59 |
| jrosser | still there is ugly mess with things like octavia and ironic networks | 12:00 |
| noonedeadpunk | in our case we have multiple AZs, so each controller is in independent datacenter and they're still routed (as have own network spaces) | 12:01 |
| kleini | I have no issues with network nodes. Due to crappy SuperMicro hardware it happens already too often, that network nodes die completely and end users don't even notice this. | 12:01 |
| jrosser | all of this seems to be about choosing the least worst place to put some complexity | 12:01 |
| kleini | noonedeadpunk: What is you problem in detail with network namespaces on network nodes? | 12:01 |
| noonedeadpunk | I'm not completely sure, but it seems that failover happens too slowly when too many namespaces needs to be failedovered | 12:02 |
| noonedeadpunk | And I think rabbit is involved there | 12:02 |
| jrosser | that’s keepalived again isn’t it? | 12:02 |
| jrosser | I don’t think we have much trouble failover with a node failing, just very long time to rebuild all the namespaces when a node comes back | 12:03 |
| noonedeadpunk | well, yes. but also neutron tries to spawn or verify namespaces or smth. So out of like 1000 l3 namespaces we usually get around 5 that don't go properly on their own | 12:04 |
| jrosser | hmm we are in low 100s rather than 1000 | 12:04 |
| noonedeadpunk | but to have that said - we have also vpnaas in env | 12:05 |
| noonedeadpunk | not sure it has smth to do directly with l3 namspaces, but maybe slowing down neutron even more | 12:05 |
| noonedeadpunk | Ah, yes, with small amount of namespaces it's perfectly fine | 12:05 |
| kleini | ouch, that is too much failure, if 5 per mille fail in provisioning | 12:05 |
| noonedeadpunk | but when it goes like >400... | 12:05 |
| jrosser | I would hope that this sort of scalability trouble is addressed with OVN | 12:05 |
| noonedeadpunk | In ovn there's anouther trouble which is ovsdb | 12:06 |
| jrosser | but I think there are also other gotchas like not being able to spread the active routers between nodes | 12:06 |
| noonedeadpunk | as all records going to one db and at some scale it becomes quite problematic as well | 12:06 |
| noonedeadpunk | or, let's say, too resource consuming | 12:07 |
| noonedeadpunk | but it's far beyond our scale at least... | 12:10 |
| noonedeadpunk | ok, thanks all, that was quite good chat. Call me lazy, but I think I will leave lxb on controllers... | 12:13 |
| kleini | me too! | 12:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Allow to customize networking configuration https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/868490 | 14:43 |
| noonedeadpunk | jrosser: I'd love to get your opinion on ^ | 14:43 |
| *** dviroel is now known as dviroel|lunch | 15:28 | |
| jamesdenton | noonedeadpunk is there a good way to leverage vars from os_neutron in os_octavia playbooks? i am not sure the group_vars approach is the right one | 16:12 |
| noonedeadpunk | jamesdenton: no, not really. group_vars also won't help unless it's all | 16:13 |
| noonedeadpunk | So basically only way is to define them somewhere in user_* and then they will be available everywhere | 16:13 |
| jamesdenton | hmm, which i don't think we want to do here since a lot of this is dynamic. There's an octavia-driver-agent for the octavia-ovn-provider implementation, and in octavia.conf there's expected to be an [ovn] section with the nb/db connection info. Maybe for the service i'll just bring in ml2_conf.ini as a config file and forgo the octavia.conf duplication | 16:15 |
| jamesdenton | seems like a PITA to try and make that go | 16:15 |
| jamesdenton | our role separation makes that a little difficult, i think | 16:15 |
| jamesdenton | but bringing in ml2_conf might present its own challenges/conflict | 16:16 |
| noonedeadpunk | I guess octavia will also need certificates being copied there? | 16:18 |
| jamesdenton | TBD | 16:19 |
| jamesdenton | TBH it's a bit more involved than i expected. Should've devstack'd it first | 16:19 |
| noonedeadpunk | So, what we did with ceilometer, for example, is we pasted other roles defaults and used vars from these roles | 16:19 |
| noonedeadpunk | So in case anything will be overriden - it will reflect ceilometer as well | 16:20 |
| jamesdenton | oh, interesting. But i guess you have to maintain parity between those defaults, but prob not too bad | 16:20 |
| noonedeadpunk | like that https://opendev.org/openstack/openstack-ansible-os_ceilometer/src/branch/master/defaults/main.yml#L118 | 16:20 |
| noonedeadpunk | well, we're not changing defaults too often... | 16:20 |
| jamesdenton | ok, smart. right | 16:21 |
| noonedeadpunk | I'm not sure what defaults we do have for ovn though. As I don't think we have things configurable | 16:21 |
| noonedeadpunk | it mostly overrides I guess | 16:22 |
| jamesdenton | yeah | 16:22 |
| noonedeadpunk | but you can create smth like ovn here https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all | 16:22 |
| noonedeadpunk | and that will be affecting both neutron and octavia (and everything actually) | 16:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-systemd_networkd master: Allow to provide multiple VLANs https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/868500 | 16:27 |
| jamesdenton | ok, i'll play around and come up with something you can pick apart :) | 16:33 |
| jamesdenton | thanks for the tips | 16:39 |
| *** dviroel|lunch is now known as dviroel | 16:41 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Update repositiories for mirroring https://review.opendev.org/c/openstack/openstack-ansible/+/868506 | 18:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add example on how to provision LXC bridges with OSA https://review.opendev.org/c/openstack/openstack-ansible/+/868507 | 18:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Add example on how to provision LXC bridges with OSA https://review.opendev.org/c/openstack/openstack-ansible/+/868507 | 18:23 |
| *** rgunasekaran_ is now known as rgunasekaran | 18:50 | |
| *** dviroel is now known as dviroel|out | 19:28 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!