| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Install distro_packages in pre-main https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/889934 | 00:04 |
|---|---|---|
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893797 | 00:10 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/888729 | 00:31 |
| opendevreview | Merged openstack/openstack-ansible-os_aodh master: Fix linters and metadata https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/888471 | 00:37 |
| jrosser | good morning | 07:39 |
| jrosser | can we merge this without updating ansible first? https://review.opendev.org/c/openstack/openstack-ansible/+/892373 | 07:43 |
| jrosser | or should i make a patch to separately bump the plugins repo SHA | 07:43 |
| noonedeadpunk | o/ | 07:44 |
| noonedeadpunk | jrosser: we can't as is, so yeah, we need a separate patch to just update plugins | 07:45 |
| jrosser | ok | 07:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Fix container bridge name for octavia https://review.opendev.org/c/openstack/openstack-ansible/+/893767 | 07:45 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Fix container bridge name for octavia https://review.opendev.org/c/openstack/openstack-ansible/+/893768 | 07:45 |
| noonedeadpunk | we actually just need to fix adjutant to merge the patch in topic | 07:46 |
| noonedeadpunk | ans core bump itself | 07:46 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Bump SHA for openstack-ansible-plugins collection https://review.opendev.org/c/openstack/openstack-ansible/+/893835 | 07:48 |
| noonedeadpunk | hm, I wonder why we still have diskfull on rocky upgrade... I hoped it to be fixed with ovs upgrade that landed on zed | 07:48 |
| jrosser | for adjutant we were needing some upgrade job fixes as i recall? | 07:48 |
| noonedeadpunk | Yeah, and we can do that now | 07:48 |
| noonedeadpunk | as fix for 2023.1 landed now | 07:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Update Adjutant and Neutron SHAs https://review.opendev.org/c/openstack/openstack-ansible/+/893837 | 07:56 |
| noonedeadpunk | ^ | 07:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible-core to 2.15.3 and ansible-lint https://review.opendev.org/c/openstack/openstack-ansible/+/892371 | 07:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible collection versions https://review.opendev.org/c/openstack/openstack-ansible/+/892373 | 08:01 |
| noonedeadpunk | hm.... somehow it's still ovs 2.17 that's installed on zed.... | 08:15 |
| noonedeadpunk | but in aio I get 3.1 | 09:42 |
| noonedeadpunk | hm | 09:42 |
| noonedeadpunk | are we upgrading from yoga then..... | 09:43 |
| noonedeadpunk | that should be a separate job though | 09:43 |
| jrosser | looks like we are missing some calls to `log_instance_info` | 09:46 |
| jrosser | there are a few through the job but unhelpfully not one at the end | 09:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Fix typo for vpnaas_custom_config distribution https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/893856 | 10:06 |
| farbod | Hello, Can somebody help me in deployment? | 10:43 |
| farbod | I will be very thankful | 10:47 |
| jrosser | farbod: just ask :) | 10:48 |
| farbod | OK :) | 10:48 |
| farbod | i am stuck at openstack-ansible setup-openstack.yml step. | 10:49 |
| farbod | here is the error: | 10:49 |
| farbod | TASK [openstack.osa.mq_setup : Add RPC RabbitMQ user] **************************************************************************************************************** | 10:49 |
| farbod | fatal: [infra1_glance_container-9054d3f8 -> infra1_rabbit_mq_container-ace2411f(172.29.236.151)]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} | 10:49 |
| jrosser | please use paste.opendev.org | 10:49 |
| jrosser | for anything large | 10:49 |
| farbod | 🆗️ | 10:50 |
| farbod | I searched a lot and didn't find any answer | 10:50 |
| farbod | I changed no_log: true in the playbooks/rabbitmq-install.yml but nothing changed | 10:51 |
| jrosser | no_log: true will make there be no log | 10:52 |
| jrosser | and the actual task failing is in the openstack-ansible collection, not in the playbook | 10:52 |
| farbod | i changed it to false. but still there is no log | 10:52 |
| farbod | where can i find that collection | 10:53 |
| jrosser | you can see that becasue the task is openstack.osa.mq_setup | 10:53 |
| jrosser | farbod: can i ask which release you are running? | 10:53 |
| noonedeadpunk | farbod: what version? | 10:53 |
| noonedeadpunk | yeah :) | 10:53 |
| jrosser | :) | 10:53 |
| farbod | i am following documentation for deploying and using git clone -b master https://opendev.org/openstack/openstack-ansible /opt/openstack-ansible for deployig on debian 11 with two nodes | 10:55 |
| noonedeadpunk | aha, master | 10:55 |
| noonedeadpunk | we have a bug on master :) | 10:55 |
| jrosser | ok, so 'master' is the development branch | 10:55 |
| jrosser | this is where we/you could do work for the next stable release | 10:55 |
| noonedeadpunk | this patch is needed there https://review.opendev.org/c/openstack/openstack-ansible/+/893835 | 10:55 |
| farbod | ohom i get it | 10:56 |
| noonedeadpunk | I would suggest to use stable/2023.1 | 10:56 |
| jrosser | if you want to instead test for a production deployment then one of the stable branches is probably what you want | 10:56 |
| jrosser | ^ like that | 10:56 |
| farbod | where to pull that stable version? | 10:56 |
| jrosser | they are different branches in the same repo | 10:56 |
| farbod | there is nothing in openstack-ansible deployment versioning in documenation | 10:56 |
| farbod | oh i found the branch | 10:57 |
| jrosser | the documentation is also branched rather than one document for all branches | 10:57 |
| jrosser | https://docs.openstack.org/openstack-ansible/latest/user/aio/quickstart.html | 10:57 |
| jrosser | ^ latest really refers to master branch there | 10:57 |
| farbod | i see | 10:57 |
| jrosser | this would be the most recent stable branch https://docs.openstack.org/openstack-ansible/2023.1/user/aio/quickstart.html | 10:57 |
| farbod | thanks | 10:57 |
| farbod | let me check it | 10:58 |
| jrosser | we are about to make a significant bugfix release on 2023.1 so if you run into any difficulty please ask | 10:58 |
| farbod | Thanks | 10:59 |
| jrosser | noonedeadpunk: more fun with member role https://github.com/ceph/ceph-ansible/blob/main/profiles/rgw-keystone-v3#L16 | 11:04 |
| jrosser | not a problem for OSA deployed ceph i think, but anyone who takes the ceph-ansible examples and uses them will have trouble, it seems thats case sensitive | 11:04 |
| noonedeadpunk | yeah, true | 11:15 |
| noonedeadpunk | worth pushing a PR for that | 11:15 |
| jrosser | i have memory of admin1 maybe getting difficulty with something like this | 11:17 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Do not add all computes as OVN gateways https://review.opendev.org/c/openstack/openstack-ansible/+/893547 | 11:25 |
| farbod | so by changing the branch my last problem solved! Thanks a lot. | 11:59 |
| farbod | but now there is another problem | 12:00 |
| farbod | https://paste.opendev.org/show/balXyfhEWLNXp3kbk9C3/ | 12:00 |
| farbod | take a look at this | 12:00 |
| farbod | my first node interfaces config: https://paste.opendev.org/show/bSmIAu2x5ynAldyT9R45/ | 12:08 |
| farbod | second node interfaces config: https://paste.opendev.org/show/bu6XRfsGmVbHDCt2gR1D/ | 12:08 |
| farbod | user config yml file: https://paste.opendev.org/show/bVbYmvqpzPDmjU5VvDBv/ | 12:08 |
| noonedeadpunk | jrosser: regarding mariadbcheck@ - eventually from what I got from systemd docs - they recommend not to use `accept` for newly designed applications | 12:12 |
| noonedeadpunk | so if that would be a proper daemon that does not require accept - we could remove template | 12:12 |
| noonedeadpunk | it's matter of refactoring and using dummy flask or aiohttp or smth like that... | 12:13 |
| noonedeadpunk | so that's by far best alternative... | 12:13 |
| noonedeadpunk | farbod: huh, I _really_ saw that 3 times in last 2 weeks... We should finally update docs to reflect OVN setup... | 12:14 |
| farbod | is the problem with br_vlan? | 12:14 |
| noonedeadpunk | farbod: I think main issue is with openstack_user_config | 12:14 |
| noonedeadpunk | the config you've made is valid for OVS/LXB drivers, but not for OVN | 12:15 |
| farbod | i dont understand the role of this br_vlan bridge :) | 12:15 |
| noonedeadpunk | And there're 2 ways around that: 1. Add override to use OVS 2. Fix config to be applicable for OVN | 12:15 |
| farbod | i am kinda noob in these things :) can you explain more? | 12:15 |
| noonedeadpunk | For the first thing - you need to add this to user_variables: https://paste.openstack.org/show/bEJxwE5J3z3SZ4YPnO6T/ | 12:17 |
| noonedeadpunk | for the second it needs slightly more effort | 12:17 |
| noonedeadpunk | farbod: so neutron does have quite some network dirvers, that are conceptually different | 12:17 |
| noonedeadpunk | most popular ones are OVS and OVN | 12:18 |
| farbod | i have these two servers connected together with Hetzner vswitches. So VLAN IDs like 4020 are for that. What should i fo in addition to these configuration? | 12:20 |
| noonedeadpunk | we have switched default to OVN lately, but didn't update docs to reflect that yet | 12:20 |
| noonedeadpunk | So I would suggest checking what you actually want. If you don't mind any option - you can stick with default OVN and I can guide you through configuration for that | 12:21 |
| farbod | yes it doesnt matter for me to what it should be | 12:21 |
| farbod | i am trying to learn and work with Openstack | 12:21 |
| noonedeadpunk | ok, then I'd suggest OVN. While we have limited experience here, "it's future" | 12:24 |
| farbod | ok | 12:25 |
| noonedeadpunk | Give me couple of mins to adjust your config | 12:25 |
| farbod | Thanks a lot. | 12:25 |
| *** dviroel_ is now known as dviroel | 12:27 | |
| noonedeadpunk | smth like that might work https://paste.openstack.org/show/b2mRNxVFs2X9Pvh6nKBT/ | 12:28 |
| farbod | Thank you | 12:29 |
| farbod | but my question is what is br_vlan network? | 12:29 |
| farbod | and why we should use it? | 12:29 |
| noonedeadpunk | you can skip it if it's not needed in your usecase | 12:31 |
| noonedeadpunk | But generally it's to allow having multiple external networks | 12:32 |
| farbod | On the other hand, is there a need to change the user variables file? | 12:32 |
| noonedeadpunk | and you can also skip flat network from other side and always use jsut vlans | 12:32 |
| noonedeadpunk | So your current openstack_user_config is valid only for linuxbridges | 12:32 |
| noonedeadpunk | if you want to have OVS or OVN instead - it must be changed | 12:32 |
| farbod | Let me test it | 12:33 |
| noonedeadpunk | if you're fine with linuxbridges - then you should add an override to use them instead of OVN that is default | 12:33 |
| farbod | i don't understand. Now with your config what shoyld user variables be? | 12:34 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Retry applying OVN connection settings https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/893667 | 12:34 |
| noonedeadpunk | Though, changes to openstack_user_config would require to re-generate (or better wipe) inventory, which is kind of - start from scratch approach, since containers should be also deleted when inventory is fully wiped | 12:34 |
| noonedeadpunk | with my adjustments it should work for OVN | 12:34 |
| noonedeadpunk | with variavbles I've pasted here https://paste.openstack.org/show/bEJxwE5J3z3SZ4YPnO6T/ - for linuxbridges and ovs | 12:35 |
| noonedeadpunk | as I said - there're multiple things you can do | 12:35 |
| farbod | Sorry for taking your time | 12:36 |
| farbod | but | 12:36 |
| farbod | Imagine i have a subnet assigned to one of the hosts | 12:37 |
| farbod | how can i set this IPsfor Openstack instances? | 12:37 |
| farbod | i mean a public subnet | 12:37 |
| opendevreview | Merged openstack/openstack-ansible stable/2023.1: Fix container bridge name for octavia https://review.opendev.org/c/openstack/openstack-ansible/+/893767 | 12:41 |
| noonedeadpunk | farbod: so. you need to create a network in neutron. When creating a network - you can supply if it's flat network or vlan. And tell that it is external network | 12:43 |
| noonedeadpunk | If it is vlan - you will be able to supply tag id | 12:43 |
| noonedeadpunk | if it's flat - then interface will be taken and added to bridge | 12:43 |
| farbod | I get it. Thanks! | 12:43 |
| noonedeadpunk | for vlan neutron will create an interface with required tag id and add it to the bridge as well | 12:44 |
| noonedeadpunk | so, you can't have same interface defined for vlan and flat network, jsut in case | 12:44 |
| noonedeadpunk | as if flat is part of the bridge, neutron will fail to create a vlan interface | 12:45 |
| farbod | aha | 12:45 |
| farbod | also you said i have to delete inventory and containers | 12:45 |
| farbod | how to do that | 12:45 |
| farbod | i deleted the inventory .json file in /etc/openstack_deploy | 12:46 |
| farbod | and delete containers with openstack-ansible lxc-containers-destroy.yml | 12:46 |
| farbod | but i encounter errors in setup hosts step | 12:46 |
| noonedeadpunk | um | 12:46 |
| noonedeadpunk | you first delete containers, then delete inventory.... | 12:47 |
| noonedeadpunk | if you've already dropped containers.... | 12:47 |
| noonedeadpunk | then maybe try smth like that `for cont in $(lxc-ls -1); do lxc-stop -n ${cont}; lxc-destroy -n ${cont}; done` | 12:48 |
| opendevreview | Merged openstack/openstack-ansible-os_adjutant stable/2023.1: Use proper galera port in configuration https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/892474 | 12:48 |
| farbod | 👍️ | 12:48 |
| noonedeadpunk | also - before running playbooks again - can you share new inventory, or better output of /opt/openstack-ansible/scripts/inventory-manage -G | 12:49 |
| farbod | yes, wait | 12:49 |
| opendevreview | Merged openstack/openstack-ansible-os_adjutant stable/2023.1: Stop reffering _member_ role https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/892099 | 12:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Update Adjutant and Neutron SHAs https://review.opendev.org/c/openstack/openstack-ansible/+/893837 | 12:55 |
| farbod | noonedeadpunk: I didn't find the file | 12:59 |
| noonedeadpunk | what file? | 12:59 |
| farbod | this /opt/openstack-ansible/scripts/inventory-manage it says it doesnt exist | 13:00 |
| noonedeadpunk | ah, it;s inventory-manage.py | 13:00 |
| farbod | https://paste.openstack.org/show/bMqFGo8u7htfTk79Kxss/ | 13:01 |
| noonedeadpunk | um... output of execution with flag `-G` | 13:02 |
| noonedeadpunk | `/opt/openstack-ansible/scripts/inventory-manage.py -G` | 13:02 |
| farbod | https://paste.openstack.org/show/bN0tP6uEYCtORa6EAWk2/ | 13:03 |
| noonedeadpunk | do you have `network-gateway_hosts` in openstack_user_config? | 13:04 |
| farbod | network-gateway_hosts: | 13:05 |
| farbod | compute1: | 13:05 |
| farbod | ip: 172.29.236.12 | 13:05 |
| noonedeadpunk | Just in case - I was reffering to this doc for OVN configuration: https://docs.openstack.org/openstack-ansible-os_neutron/latest/app-ovn.html | 13:05 |
| farbod | yes | 13:05 |
| noonedeadpunk | ok, can you try to run /opt/openstack-ansible/inventory/dynamic_inventory.py and the check output of inventory-manage.py -G again? | 13:05 |
| farbod | yes | 13:06 |
| noonedeadpunk | As I don't see compute there at all for some reason | 13:06 |
| opendevreview | Merged openstack/openstack-ansible stable/zed: Fix container bridge name for octavia https://review.opendev.org/c/openstack/openstack-ansible/+/893768 | 13:09 |
| farbod | This time only this error: TASK [os_neutron : Setup Network Provider Bridges] ******************************************************************************************************************* | 13:18 |
| farbod | fatal: [compute1]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: list object has no element 1\n\nThe error appears to be in '/etc/ansible/roles/os_neutron/tasks/providers/setup_ovs_ovn.yml': line 55, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- name: Setup Network Provider Bridges\n ^ here\n"} | 13:18 |
| jamesdenton | hi farbod did you happen to share your openstack_user_config.yml and user_variables.yml already? | 13:20 |
| farbod | i changed it. let me share it again | 13:20 |
| farbod | use config: https://paste.opendev.org/show/bHpUizqytme2CVxNv62q/ | 13:20 |
| jamesdenton | and the other?> | 13:21 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_nova stable/2023.1: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893773 | 13:21 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_nova stable/zed: Config has changed for pci passthrough. https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/893774 | 13:21 |
| farbod | user variable: https://paste.opendev.org/show/bP8FTleM8igiUuXzuOVw/ | 13:21 |
| jamesdenton | ok | 13:21 |
| jamesdenton | thanks | 13:21 |
| jamesdenton | so, in openstack_user_config.yml you are missing the neutron provider bridge definition. That section of the config is a little tricky, as 'provider_network' is a misnomer.. but essentially, you want to use that spot to add what will become "br-ex" or "br-provider" for OVS | 13:22 |
| jamesdenton | one sec | 13:22 |
| farbod | also my network configs is up there | 13:23 |
| jamesdenton | ok, check this out: https://paste.opendev.org/show/bjw3b5ncP6dbhj34ltJU/ | 13:26 |
| jamesdenton | i added a section and added comments | 13:26 |
| jamesdenton | you can change as values to fit your environment | 13:26 |
| jamesdenton | *change the values | 13:26 |
| farbod | can you explain it more? | 13:27 |
| farbod | did you check my network configs? | 13:27 |
| jamesdenton | i';m not sure :D | 13:28 |
| jamesdenton | one sec | 13:28 |
| noonedeadpunk | jamesdenton: but I guess it fails on br-vxlan? | 13:28 |
| noonedeadpunk | as it can't fail on smth that is not defined | 13:28 |
| jamesdenton | i wouldn't expect br-vxlan to be eligible | 13:29 |
| jamesdenton | need to look at that task | 13:29 |
| noonedeadpunk | it's failing here basically farbod: container_interface should be a valid interface | 13:30 |
| jamesdenton | farbod you are hampered by the use of a single interface on each node | 13:30 |
| farbod | unfortunately yes. i only have one interface | 13:30 |
| noonedeadpunk | farbod: I would add `host_bind_override` to br-vxlan network, that would be a proper interface on the host | 13:31 |
| jamesdenton | that's only for linuxbridge | 13:31 |
| jamesdenton | and container_interface isn't needed without lxc | 13:31 |
| noonedeadpunk | jamesdenton: we have _exact_ same query on our ML from yesterday fwiw | 13:31 |
| jrosser | this is all virtualised, right? `farbod> i have these two servers connected together with Hetzner vswitches` | 13:32 |
| farbod | servers are baremetal | 13:32 |
| farbod | but connections are on vswtiches | 13:32 |
| jrosser | but the environment limits you to one interface? | 13:32 |
| farbod | yes | 13:32 |
| noonedeadpunk | jamesdenton: I tried t look into our code yestarday, and bridge_mapping (where it fails) https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/providers/setup_ovs_ovn.yml#L64-L74 | 13:33 |
| jamesdenton | looking at that now, too | 13:33 |
| noonedeadpunk | was produced by net['network']['net_name'] and net['network']['container_bridge'] | 13:33 |
| jamesdenton | neutron_provider_networks.network_mappings is defined | 13:33 |
| jrosser | right so thats another factor to take into account when designing this, as probably some stuff is required that would not otherwise be for "real" servers/switches | 13:33 |
| noonedeadpunk | OR net['network']['host_bind_override'] | 13:33 |
| noonedeadpunk | farbod: and can you have vlans? | 13:34 |
| farbod | vswitches are on vlans right now | 13:34 |
| farbod | https://paste.opendev.org/show/bSmIAu2x5ynAldyT9R45/ | 13:35 |
| farbod | look at this infra node network config | 13:35 |
| jamesdenton | you can try to add the bits i suggested, just remove the network_interface line | 13:35 |
| noonedeadpunk | yeah, so you just need to have couple of them - one for mgmt network, another for public and preferably for internal tenant networks (vxlans) | 13:35 |
| jamesdenton | i think that might get it working, just need to look at the logic to confirm | 13:35 |
| farbod | jamesdenton: sorry i didnt understand:) what changes? | 13:36 |
| jamesdenton | https://paste.opendev.org/show/bjw3b5ncP6dbhj34ltJU/ | 13:36 |
| jamesdenton | Line 22 | 13:37 |
| jamesdenton | just remove lines 24 and 26 | 13:37 |
| farbod | like this? https://paste.opendev.org/show/bY40y4SUzN6xI9JIhTgd/ | 13:38 |
| jamesdenton | yes. might wanna remove the comments, too | 13:39 |
| farbod | ok | 13:39 |
| farbod | but whats the purpose of this? | 13:40 |
| farbod | it's not in documentation | 13:40 |
| farbod | i am kinda noob sorry :) | 13:40 |
| noonedeadpunk | jamesdenton: if you have a minute, can you kindly check this ML as well:) as apparently I've missed br-ex there as well: https://lists.openstack.org/pipermail/openstack-discuss/2023-September/034956.html | 13:40 |
| jamesdenton | i absolutely will | 13:40 |
| jamesdenton | farbod it's possible the documentation needs to be updated | 13:40 |
| jamesdenton | or the logic needs to not assume a provider network bridge is defined | 13:40 |
| farbod | ok let me test it | 13:41 |
| farbod | for another deployment i need to delete containers and inventory only? | 13:42 |
| jamesdenton | you should just re-run os-neutron-install.yml | 13:42 |
| farbod | no need to change network configs? | 13:43 |
| jamesdenton | i am testing on my side, too | 13:44 |
| jamesdenton | no other changes, the goal is just to get the playbook to finish | 13:44 |
| jamesdenton | but there will be other changes ultimately needed | 13:44 |
| jamesdenton | ok, i as able to replicate | 13:45 |
| farbod | ? | 13:45 |
| jamesdenton | ok, there's a logic difference between ovs and ovn. noonedeadpunk https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/ovs_config.yml#L27 vs https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_ovn.yml#L74 | 13:48 |
| jamesdenton | farbod you can proceed with that change | 13:48 |
| noonedeadpunk | ugh | 13:48 |
| farbod | OK i am testing | 13:48 |
| noonedeadpunk | jamesdenton: checking length sounds reasonable | 13:48 |
| jamesdenton | i will push a change in a sec | 13:49 |
| noonedeadpunk | I can answer a ML if you want :) | 13:49 |
| noonedeadpunk | (but you're the knowledgable one so appriciated if you have a minute to do that) | 13:50 |
| jamesdenton | sure, i don't mind | 13:50 |
| farbod | Oh boy | 13:51 |
| farbod | it's a huge project | 13:51 |
| farbod | how you guys | 13:51 |
| jamesdenton | lots of moving pieces, but with time and experience comes knowledge,. it's not so bad :) | 13:51 |
| farbod | How do you guys handle it? | 13:51 |
| farbod | yes I became interested in it | 13:52 |
| jamesdenton | well, this is how you do it. | 13:52 |
| jamesdenton | install it. break it. fix it. do it again | 13:52 |
| farbod | yes | 13:52 |
| farbod | I would love to reach your level so that I can at least contribute to the project | 13:53 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_neutron master: Check length of network_mappings https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/893924 | 13:54 |
| jamesdenton | farbod the change you made will help get the playbook to finish, but it will not result in a completely functional setup. You might be able to create VMs and attach the vxlan networks, but without a provider network you can't use floating IPs | 13:55 |
| farbod | you mean i need a NIC to be able to assign public IPs? | 13:57 |
| jamesdenton | you would need a VLAN that has a subnet that's useable for external floating IPs | 14:00 |
| jamesdenton | public or private | 14:00 |
| jamesdenton | in a lab, that network might be 192.168.1.0/24 and routable from within your home/work, and the vm network might be 172.16.0.0/24 that isn't. The router assigns a 1:1 NAT from the floating IP network -> tenant network | 14:01 |
| jamesdenton | there are some fundamental openstack concepts that need to be looked at before diving into the actual deployment processes | 14:02 |
| farbod | So if i add another VLAN with provided Vswitch like enp8s0.4040 i can assign IPs from that network? | 14:02 |
| jamesdenton | you could, potentially. | 14:02 |
| farbod | How configs would be? | 14:03 |
| jrosser | jamesdenton: whilst you're around i could do with some advice on AIO+octavia with OVN | 14:05 |
| jamesdenton | yeeeeah, hadn't thought about that one | 14:06 |
| jrosser | or rather AIO(lxc)+octavia | 14:06 |
| jamesdenton | what are we doing now for OVS? | 14:06 |
| jrosser | well i'm not totally sure :) | 14:06 |
| jamesdenton | lol | 14:06 |
| jrosser | yeah | 14:06 |
| jamesdenton | but it's working? maybe? | 14:06 |
| jrosser | deal with these other things is you're in the middle of it ^^^^ | 14:07 |
| jamesdenton | is this the need to make sure the lbaas mgmt network is reachable? | 14:07 |
| jrosser | i think so | 14:07 |
| jrosser | there is also this https://review.opendev.org/c/openstack/openstack-ansible/+/893315 | 14:07 |
| jrosser | i guess i'm just uncertain atm if the LXC AIO works for octavia | 14:08 |
| jrosser | as all we test is metal | 14:08 |
| jamesdenton | are there actually functional tests? | 14:09 |
| jrosser | i think so https://github.com/openstack/openstack-ansible/blob/master/tests/roles/bootstrap-host/templates/user_variables_octavia.yml.j2#L14 | 14:09 |
| jrosser | though i am running a very complicated AIO config just now and we know that the role tempest tests trample all over each other | 14:10 |
| jrosser | so i think that the alphabetically last user_variables* will be the one which has tests that actually run | 14:10 |
| jrosser | but regardless | 14:11 |
| jamesdenton | ok, i'll prob need to roll an AIO/OVS w/ LXC and Metal to compare what works and doesn't | 14:11 |
| jrosser | i think i just try to undertstand how br-lbass is supposed to be with OVN | 14:11 |
| jamesdenton | and try to get OVN going, too. | 14:11 |
| jamesdenton | IIRC there was an IP on a provider bridge for lbaas that allowed the octavia worker to hit those VMs | 14:11 |
| jrosser | as it has to be hooked to the containers but also a neutron flat network via OVN | 14:12 |
| jamesdenton | right, it's a little convoluted. I think it will end up working the same way, ultimately, as OVS | 14:12 |
| jrosser | hmm i see | 14:12 |
| jamesdenton | i need to refresh my memory, been a very long time | 14:14 |
| jrosser | indeed | 14:14 |
| jamesdenton | but is the aio_lxc_octavia scenario enough? | 14:14 |
| jrosser | i beleive so yes | 14:14 |
| jrosser | i am still totally with my head in linuxbridge unfortunately | 14:14 |
| jrosser | so still trying to grok all the OVS/OVN parts | 14:14 |
| jamesdenton | understandable | 14:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/2023.1: Update Adjutant and Neutron SHAs https://review.opendev.org/c/openstack/openstack-ansible/+/893837 | 14:21 |
| farbod | You guys saved my life | 14:24 |
| farbod | it finally finished | 14:24 |
| farbod | and i accessed the dashboard | 14:25 |
| farbod | Thanks a lot | 14:25 |
| jamesdenton | woot | 14:25 |
| jamesdenton | you found a bug, so thank you | 14:25 |
| farbod | Actually, I was surprised by all the support | 14:27 |
| farbod | you guys are perfect | 14:28 |
| jrosser | farbod: the networking really is one of the most complicated bits to understand | 14:28 |
| noonedeadpunk | ++ | 14:28 |
| farbod | yeah | 14:28 |
| noonedeadpunk | especially for the long run | 14:29 |
| jrosser | becasue there are so many options and possible use cases, and everyone has different requirements | 14:29 |
| jrosser | and all of the possiblities in openstack we try to expose through openstack-ansible | 14:29 |
| farbod | so what that br-ex bridge did exactly? | 14:29 |
| jrosser | but that can make the config appear a bit abstract in order to support all these things | 14:29 |
| farbod | what if i wanted to use a a network with IPs on it for assigning them to instances? | 14:30 |
| jrosser | farbod: you mean some already existing network that you want to connect to your openstack? | 14:32 |
| jamesdenton | the provider bridge (br-ex) exposes neutron resources to the physical network. For the self-service model, a tenant creates a tenant network (vxlan), creates a VM and attaches to the tenant network. The tenant creates a vRouter, attaches the tenant network on one side (LAN) and connects router to the provider network to the (WAN) side | 14:32 |
| farbod | is this br-ex same as br-vlan? | 14:34 |
| jamesdenton | not quite? it's complicated :D | 14:34 |
| jrosser | farbod: ^ this is another great thing to think about "For the self-service model...." <- if you are building a cloud for many users and projects who all create virtual networks / routers / etc themselves this is great, and you design for it | 14:34 |
| jamesdenton | for openstack-ansible, there are a set of LINUX bridges used to connect LXC containers to the physical network and other hosts. But if you're not using LXC, then br-vlan and other linux bridges like it are less important | 14:35 |
| jrosser | on the other hand if you are doing something like telco perhaps where network performance is key and there is no multitenancy, you might make very different choices | 14:35 |
| jamesdenton | +1 | 14:35 |
| jamesdenton | we don't really call out those differentiators very well in the docs | 14:35 |
| farbod | aha | 14:36 |
| farbod | understand | 14:36 |
| jrosser | probably good to consider openstack a bit of a toolbox | 14:36 |
| jrosser | so some learning the parts then assembling what you need to meet a specific use case well is a good aproach | 14:37 |
| jrosser | or take some advice from people with similar goals | 14:37 |
| farbod | so imagine i have a enp8s0.4040 along side other VLANS. How to connect it to the br-ex to use the IPs on that network? | 14:37 |
| jamesdenton | good question. | 14:38 |
| jamesdenton | Neutron can handle VLAN tagging for you. When you create a provider network is gets a VLAN ID. When br-ex contains enp8s0, OVS/OVN can tag traffic as 4040 for that provider network, and 4041 for another, etc | 14:38 |
| jamesdenton | if br-ex contains enp8s0.4040 directly, then you are limited to only vlan 4040 and neutron won't tag. It would then be a FLAT provider network vs a VLAN provider network | 14:39 |
| farbod | OK | 14:39 |
| jamesdenton | doable, but less flexible | 14:39 |
| farbod | and how to set it in user conf .yml file? | 14:39 |
| jamesdenton | farbod https://paste.opendev.org/show/bpaUpXY1t64vx1TSzQZk/ | 14:44 |
| jamesdenton | i recommend looking around for openstack networking essentials to get an idea of what it looks like | 14:44 |
| farbod | Yes i surly do | 14:45 |
| farbod | Now as a test how to run an instance? | 14:46 |
| jrosser | jamesdenton: https://paste.opendev.org/show/bmAtcVVQ7aCiN1LcPHGv/ | 14:46 |
| jamesdenton | jrosser i'm guessing br-lbaas exists as a linux bridge already? | 14:46 |
| jrosser | yes it does, that created by the boostrap-aio things | 14:46 |
| jamesdenton | and also an ovs bridge | 14:46 |
| jrosser | indeed | 14:47 |
| jamesdenton | which might be why it was called br-octavia | 14:47 |
| jamesdenton | for ovs | 14:47 |
| jrosser | argh | 14:47 |
| jrosser | ok well that blew up in a different way | 14:47 |
| admin1 | don't osa workaround was to add br-lbaas as a vtep to br-vlan ? | 14:47 |
| jamesdenton | there is some veth shenanigans iirc | 14:47 |
| admin1 | and initialized via rc.local | 14:47 |
| jrosser | jamesdenton: oh well my patch might be totally bogus fix then | 14:47 |
| admin1 | i still use those .. in absense of a better way | 14:47 |
| jamesdenton | jrosser which scenario was failing? | 14:48 |
| jrosser | something like aio_lxc_octavia broke locally for me | 14:48 |
| jamesdenton | admin1 in prod make the lbaas mgmt network routable, but thats trickier in aio | 14:48 |
| jrosser | therefore the comment in my patch that we don't test the LXC scenario | 14:48 |
| jamesdenton | ok, and thats LXC+OVN now, right? | 14:49 |
| jrosser | it is | 14:49 |
| jamesdenton | kk | 14:49 |
| jamesdenton | i will look at that today | 14:49 |
| jrosser | awesome thanks | 14:49 |
| admin1 | i do have lxc + ovn + with dual lb providers .. octavia and ovn .. both work | 14:49 |
| admin1 | ovn is all manual setup via cli only | 14:49 |
| admin1 | ovn lb | 14:49 |
| admin1 | just below my desk is signed book from jamesdenton :D | 14:50 |
| jamesdenton | as a foot rest, i hope | 14:50 |
| admin1 | you need to do a new book covering ovb for the next summit jamesdenton .. | 14:50 |
| jamesdenton | keeps the desk from wobbling | 14:50 |
| farbod | Sorry, Another question. There is nothing to show in the dashboard as images or flavors for creating an instance. even there is not an active network | 14:50 |
| admin1 | farbod , that is by default | 14:51 |
| farbod | So how to test it out? | 14:51 |
| jamesdenton | farbod thats because you need to upload images, create flavor and network | 14:51 |
| admin1 | you need to do all those yourself, as those differ per providers | 14:51 |
| jrosser | the tempest role would load up cirros and make some tiny flavors? | 14:51 |
| jamesdenton | i recommend the all-in-one | 14:51 |
| jrosser | but in general this is the point where something external is expected to set stuff up | 14:52 |
| admin1 | farbod https://docs.openstack.org/install-guide/launch-instance.html | 14:52 |
| jrosser | which can be as simple as you clicking around horizon | 14:52 |
| jrosser | or as complicated as some bunch of further automation | 14:52 |
| farbod | Aha Thanks | 14:52 |
| jrosser | a whole load of things also might not be possible through horizon | 14:52 |
| jrosser | so it's worth getting familiar with the command line tools that get installed into the "utility" host | 14:53 |
| admin1 | farbod , and more -> https://docs.openstack.org/glance/pike/admin/index.html | 14:54 |
| farbod | 👍️ | 14:54 |
| admin1 | sorry .. ignore the pike, use the latest version you have installed | 14:54 |
| admin1 | https://docs.openstack.org/operations-guide/ | 14:55 |
| jrosser | jamesdenton: from memory the octavia lxc fails to start because it cannot find attach to br-octavia | 14:57 |
| jrosser | something like that | 14:57 |
| admin1 | jamesdenton, jrosser, are you trying to fix/setup br-octavia without veth and in an automated way ? | 14:58 |
| jrosser | admin1: i want to the AIO to work out of the box for octavia+OVN | 14:58 |
| jrosser | with LXC | 14:58 |
| admin1 | aha .. that means 1 more dedicated network just for octavia ? | 14:59 |
| jrosser | well, i understand how it used to be for linuxbridge | 15:00 |
| jrosser | but not how it should be for OVN | 15:00 |
| farbod | can i upload images from URL? | 15:01 |
| noonedeadpunk | You can with interoperable import feature | 15:02 |
| noonedeadpunk | there's a web-download method: https://docs.openstack.org/glance/latest/admin/interoperable-image-import.html#image-import-methods | 15:03 |
| noonedeadpunk | though you'd need to define some overrides for that | 15:03 |
| noonedeadpunk | You can define `glance_glance_image_import_conf_location` as absolute path on localhost with config for interoperable import | 15:04 |
| noonedeadpunk | also likely you'd need to set `glance_use_uwsgi: false` | 15:04 |
| farbod | how to change the configs to be able to download it from dashboard? | 15:05 |
| johnsom | jrosser Not sure if this is helpful or not, but here is the neutron devstack setup for the lb-mgmt-subnet on OVS/OVN: https://github.com/openstack/neutron/blob/master/devstack/lib/octavia | 15:05 |
| johnsom | Maybe that will give some hints/ideas | 15:06 |
| admin1 | jrosser, this still works , even for ovn -> https://www.openstackfaq.com/openstack-octavia/ | 15:06 |
| admin1 | i am still using this same method for ovs as well as ovn | 15:07 |
| jamesdenton | thanks | 15:07 |
| jamesdenton | ultimately might end up doing the same thing, not sure what's busted, yet | 15:07 |
| farbod | how to use this openstack command for management? | 15:08 |
| noonedeadpunk | there's `--import` flag for openstack image create command | 15:09 |
| farbod | yes i found it | 15:09 |
| noonedeadpunk | But I'm really not sure if all options are possible | 15:10 |
| farbod | but how to use openstack command | 15:10 |
| farbod | where to use it | 15:10 |
| farbod | ? | 15:10 |
| noonedeadpunk | as IIRC you;d need to supply an url... | 15:10 |
| noonedeadpunk | we pre-install all clients inside utility container | 15:10 |
| noonedeadpunk | so you need to lxc-attach -n <utility_name> | 15:10 |
| admin1 | farbod, in order ... create flavors .. create images .. .. with this 2, u should be able to launch vms that acn talk to each other ( with correct security group applied) .. for vms -> internet, you need to add provider network that can route | 15:10 |
| noonedeadpunk | then `source /root/openrc` | 15:10 |
| noonedeadpunk | and feel free to use openstack command | 15:10 |
| farbod | which utility container has this connection to cluster? | 15:11 |
| farbod | oh oh | 15:11 |
| farbod | i found it | 15:11 |
| farbod | sorry :) | 15:11 |
| admin1 | neutron_lbaas_octavia: true -- also needs to be set .. without this, ovn will be detected | 15:11 |
| admin1 | and used | 15:11 |
| jamesdenton | johnsom https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L475 looks like this will be helpful, too. thanks | 15:11 |
| admin1 | dual providers for octavia -> https://gist.github.com/a1git/725599d5b08994766a5a5bab25ad43da ( but not yet in osa yet ) .. would love to do it in osa also | 15:21 |
| jrosser | admin1 why can’t you do that in osa? | 15:22 |
| jamesdenton | because this needs to be addressed: https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/868462 | 15:25 |
| jamesdenton | (my fault) | 15:25 |
| farbod | Thank you guys for today. I finally deployed a test cluster with the help of you. I really appreciate that. Thanks a lot and Goodbye. | 15:27 |
| jamesdenton | good luck! | 15:27 |
| admin1 | do you plan to address it soon jamesdenton ? | 15:29 |
| jamesdenton | uhhhhh | 15:29 |
| jamesdenton | i will take a fresh look at it | 15:30 |
| admin1 | thanks jamesdenton .. | 15:30 |
| jrosser | oh hmm yes that patch has some unusal choices for variable names | 15:40 |
| jamesdenton | i think i was trying to leverage existing neutron-generated certs | 15:41 |
| jamesdenton | for reasons i can't remember | 15:41 |
| jrosser | i wonder if it has to talk directly to OVN cluster | 15:42 |
| jamesdenton | oh yeah, octavia needed the ovn certs/keys in its config | 15:42 |
| jrosser | imho that needs to be all made *much* clearer what is going on | 15:44 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_neutron master: Fix l3 agent group determination for vpnaas https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/893938 | 15:44 |
| jrosser | as this is an interesting example, a thing that is not part of OVN as such but needs to interact with it via SSL | 15:44 |
| jamesdenton | +1 certainly can be made better. have to context switch a bit to get back there | 15:45 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Ensure tempest include and exclude lists all use unique names https://review.opendev.org/c/openstack/openstack-ansible/+/893968 | 16:30 |
| admin1 | is trove working good with osa ? | 16:30 |
| jamesdenton | let us know :D | 16:32 |
| admin1 | :) | 16:32 |
| admin1 | ok | 16:32 |
| jamesdenton | jrosser do you recall offhand where the lxc configuration files live? | 16:34 |
| jrosser | i think its /var/lib/lxc/*/config | 16:35 |
| jamesdenton | thanks | 16:36 |
| jamesdenton | so i don't think i'm crazy... https://paste.opendev.org/show/bFIzyhZTi6BEjLaBDpL4/ | 20:50 |
| Moha | To mitigate security risks on the provider network should we put each customer to a PVLAN? If yes, how this feature can be used with OVN/Openflow? | 20:54 |
| Moha | jamesdenton: ^ | 20:54 |
| jamesdenton | tenant networks can be of vlan or geneve type; either provides segmentation between workloads/tenants unless those are "shared" networks | 20:55 |
| jamesdenton | meaning... the default network segmentation behavior is 1:1 network:tenant, which is probably as secure as its going to get | 20:56 |
| jamesdenton | Moha i think PVLAN might be doable with OVS but am not aware of any Neutron integration. You might be able to simulate the behavior w/ sec groups, depending on what you're trying to allow/disallow | 21:02 |
| Moha | Some ref: https://access.redhat.com/solutions/4726741 | 21:06 |
| Moha | https://wiki.openstack.org/wiki/Isolated-network | 21:07 |
| jamesdenton | "At no point can we assume that tenants will do the right thing with regards to security groups." <- that's the issue :D | 21:07 |
| jamesdenton | and they're not wrong | 21:07 |
| Moha | In the current OVN configuration, by a simple scanning, all of the MAC addresses that exist in the provider network are visible from any instance that resides on that provider network! | 21:09 |
| jamesdenton | noonedeadpunk re: glance - https://paste.opendev.org/show/bvJxtyalBK2pKi1T21u5/ - vhost is being added as [{'name': '/glance', 'state': 'absent'}, {'name': 'glance', 'state': 'present'}] | 21:10 |
| jamesdenton | Moha yes, that is the nature of Layer 2, unfortunately | 21:11 |
| jamesdenton | you might consider routing NFS traffic vs using a shared network | 21:11 |
| jrosser | jamesdenton: you need a newer SHA for openstack-ansible-plugins | 21:31 |
| jamesdenton | ahhh | 21:32 |
| jamesdenton | https://opendev.org/openstack/openstack-ansible-plugins/commit/9f13a58e2b8596ae43b11dd1b112be8522868c90 | 21:32 |
| jamesdenton | lol | 21:32 |
| jamesdenton | "Allow to manage more the one vhost with mq_setup" | 21:32 |
| jamesdenton | thanks :D | 21:32 |
| jrosser | theres a bit of a banana skin that in CI it uses master of that collection | 21:39 |
| jrosser | but pins to the SHA in ansible-collection-requirements.yml for a local run | 21:39 |
| jrosser | hence https://review.opendev.org/c/openstack/openstack-ansible/+/893835 | 21:40 |
| jamesdenton | thanks jrosser. need to re-run setup-hosts now or bootstrap-ansible? | 21:43 |
| jrosser | which release do you try? | 21:44 |
| jrosser | becasue it would be great if you could test drive this https://review.opendev.org/c/openstack/openstack-ansible/+/893230 | 21:44 |
| jamesdenton | just wanna make sure i get that collection updated | 21:44 |
| jrosser | sure | 21:45 |
| jrosser | bootstrap-ansible does a ton of stuff | 21:45 |
| jamesdenton | oh, since you're up... | 21:45 |
| jrosser | 893230 makes it so you can `openstack-ansible scripts\get-ansible-collection-requirements.yml` | 21:46 |
| jamesdenton | setup-hosts is failing because it wants to connect the octavia lxc container to the "provider" bridge, which is an OVS bridge, which doesn't exist yet because OVS isn't installed until os-neutron-install | 21:46 |
| jamesdenton | so, i installed OVS and created the bridge and now that is happy | 21:46 |
| jrosser | and that will just sort the collections without messing with anything else | 21:46 |
| jamesdenton | and trying to get setup-openstack to finish to test tempest | 21:46 |
| jrosser | right so we have a bit of a catch-22 to think about then | 21:47 |
| jamesdenton | ack 893230 | 21:47 |
| jamesdenton | yes, we do need to think about that (if this does indeed work) | 21:47 |
| jrosser | dunno if that points back to making a 2nd bridge and wiring them together | 21:48 |
| jrosser | +/- eth14 | 21:49 |
| jamesdenton | it's bridges all the way down | 21:50 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!