| jrosser | good morning | 10:06 |
|---|---|---|
| jrosser | noonedeadpunk: can we do a zed sha bump / release soon? | 10:07 |
| admin1 | hi guys .. what do you use to limit vm bandwidth ? does rx/tx work for kvm as well ? | 10:35 |
| admin1 | sorry .. i meant do you use flavor parameters .. like vif_* or neutron qos ? | 10:37 |
| moha7 | To have `vim` install ed on all containers, how should it be don via all_containers group in user_variables.yml? | 11:45 |
| admin1 | moha7, i do install vim , curl and wget as well, but i do it via adhoc ansible command | 11:47 |
| jrosser | moha7: admin1 see this, only relevant at the point that the container image is created https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/defaults/main.yml#L188 | 11:51 |
| jrosser | moha7: admin1 and then if you want something installed independantly of making the container image on all hosts, use this https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/defaults/main.yml#L142 | 11:52 |
| moha7 | jrosser: Great | 11:54 |
| jrosser | you can also install things only on the hosts, not containers with this https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/defaults/main.yml#L148 | 11:55 |
| moha7 | what about 22 | 11:56 |
| moha7 | I want them to be installed within containers | 11:57 |
| jrosser | 22? | 11:57 |
| moha7 | L22 | 11:57 |
| jrosser | https://github.com/openstack/openstack-ansible-openstack_hosts/blob/master/defaults/main.yml#L22 ? | 11:57 |
| moha7 | https://github.com/openstack/openstack-ansible-lxc_hosts/blob/master/defaults/main.yml#L22 | 11:57 |
| jrosser | well that does what it says :) install packages to the host | 11:58 |
| jrosser | the lxc_hosts role is responsible for setting up the host, and buiding the container image | 11:58 |
| jrosser | it does not start containers | 11:58 |
| moha7 | right | 11:58 |
| jrosser | so you have the choice of installing things into the container image at the point that it is created | 11:58 |
| jrosser | or you can install things once the containers have been created using the openstack_hosts role which is run against everything | 11:59 |
| jrosser | moha7: does this give you the options you need? | 12:00 |
| moha7 | openstack_hosts --> points to the containers after they're created, right? | 12:01 |
| moha7 | jrosser: yes, it gives | 12:02 |
| jrosser | moha7: openstack_hosts runs against everything, hosts and containers | 12:17 |
| jrosser | and you have variables in that role for "everything" or "hosts only" | 12:17 |
| moha7 | In both the documentation for provider: https://docs.openstack.org/neutron/latest/admin/ovn/refarch/provider-networks.html and also in `/opt/openstack-ansible//etc/openstack_deploy/openstack_user_config.yml.example`, it's said to have both vlan and flat types with same value for container_bridge: `container_bridge: "br-vlan"`, both mapped to same interface | 12:19 |
| moha7 | Error in neutron_server container --> in rsyslog : | 12:22 |
| moha7 | ERROR neutron.plugins.ml2.managers raise ValueError(_("Value %(value)s in mapping: '%(mapping)s' "#0122023-01-16 15:41:07.821 3130 ERROR neutron.plugins.ml2.managers ValueError: Value br-vlan in mapping: 'flat:br-vlan' not unique#0122023-01-16 15:41:07.821 3130 ERROR neutron.plugins.ml2.managers | 12:22 |
| jrosser | moha7: i am not sure what you are asking? | 12:23 |
| jrosser | when LXC are created, connect `container_bridge` on the host to `container_interface` in the LXC | 12:25 |
| jrosser | thats not really to do with the neutron config? | 12:25 |
| jrosser | unfortunately `provider_networks` has some fields used to define how the container networking as arranged, and some fields are relevant to the neutron config | 12:26 |
| jrosser | moha7: did you also compare with the section in the AIO config file for OVN? https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio.j2#L178-L192 | 12:29 |
| jrosser | i think also we have given advice that for a production deployment it is probably better to not have a flat network at all | 12:29 |
| moha7 | yes; but I couldn't finally solved the provider issue; then i decided to test it for flat as it ls simpler | 12:31 |
| moha7 | In document, it says: `host_bind_override: "ens2f0"` | 12:31 |
| moha7 | But, james said me to replace it with: `network_interface: "enp6s21"` | 12:33 |
| jrosser | that document you refer to is for a very specific example given here https://docs.openstack.org/openstack-ansible/latest/user/prod/provnet_groups.html | 12:34 |
| moha7 | L178-L192: Ah, it's `using group_binds: - neutron_ovn_gateway` while I'm using neutron_ovn_controller | 12:34 |
| jrosser | i am not sure that will fix your duplicate mapping | 12:37 |
| moha7 | I had another mistake in openstack_user_config.yml, setting `network-gateway_hosts:` to controllers, while it should be on Computes | 12:56 |
| moha7 | it made to have all hosts, both infras and computes, listed for gateways in the inventory.js | 12:58 |
| moha7 | json* | 12:58 |
| jrosser | moha7: the idea is that you can choose if you want the gateways to be on the computes, controllers or both depending on how you define that group | 13:02 |
| moha7 | `network-gateway_hosts: *controller_hosts` and `network-northd_hosts: *controller_hosts` and `group_binds: - neutron_ovn_gateway #(for provider network)` -----> results in inventory.json: https://ibb.co/x58ksRk, jrosser | 13:15 |
| moha7 | Is it correct to have all hosts as ovn_gataway? | 13:18 |
| jrosser | moha7: it depends, what do you want? | 13:18 |
| moha7 | I want a working provider network (: | 13:20 |
| jrosser | well like i say with OVN you get to choose these things | 13:21 |
| moha7 | would you please take a look at my openstack_user_config.yml: http://ix.io/4lqn to see if there's aomething wrong there | 13:22 |
| jrosser | have you yet fixed the neutron error with duplicated mappings? | 13:22 |
| moha7 | no, I'm going to remove the flat network and deploy it again | 13:24 |
| jrosser | i don't even see br-vlan there which is from your previous error :/ | 13:24 |
| moha7 | I renamed it just now to br-ex | 13:25 |
| moha7 | brvlan = br-ex | 13:25 |
| jrosser | ok well please take this the right way but it feels like we are changing so much continually it is very hard to help | 13:25 |
| moha7 | my user_variables.yml: http://ix.io/4lqp | 13:25 |
| jrosser | if you have a specific error from neutron then you should fix that before moving on | 13:26 |
| jrosser | it's really really had to understand this from a distance with so much change | 13:26 |
| moha7 | I see, I'll go read more logs | 13:27 |
| jrosser | just a side note that the utility container should have a complete openstack CLI setup for you | 13:27 |
| jrosser | there is no need to do your own lxc_cache_extra_distro_packages: python3-openstackclient | 13:27 |
| jrosser | that will install it from an apt package which is totally not what you want | 13:27 |
| jrosser | moha7: if you have a specific error, like the duplicate you have before then lets look at it one step at a time | 13:28 |
| jrosser | it might be we need to ask jamesdenton again for some advice | 13:29 |
| moha7 | sure | 13:31 |
| mgariepy | hello | 13:31 |
| jrosser | hello | 13:33 |
| mgariepy | how are things going ? | 13:37 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow git servers for openstack services and tempest to be overridden https://review.opendev.org/c/openstack/openstack-ansible/+/869748 | 13:40 |
| jrosser | mgariepy: ok i think - wish i understood more OVN really! | 13:41 |
| mgariepy | what don't you understand about it ? | 13:41 |
| mgariepy | input packet >> ovn magic >> output packet | 13:41 |
| jrosser | ^that bit :) | 13:42 |
| jrosser | i guess we have a lot of people apparently struggling | 13:42 |
| mgariepy | the physical binding is pretty much the same as ovs | 13:42 |
| jrosser | and hard to say if its structural errors in OSA or docs should be better, or openstack_user_config is just too complicated | 13:42 |
| jamesdenton | good morning | 13:43 |
| mgariepy | hey good morning jamesdenton | 13:43 |
| admin1 | morning | 13:43 |
| jamesdenton | moha7 when you say your provider network isn't working, can you elaborate on that a bit? the network exists but no traffic? | 13:43 |
| mgariepy | i want to add some docs on how to find differents component and how to map them in the ovn DB and neutron db | 13:44 |
| jamesdenton | good call | 13:44 |
| admin1 | moha7, do you plan to use both flat and br-vlan over the same network ? | 13:45 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow git servers for openstack services and tempest to be overridden https://review.opendev.org/c/openstack/openstack-ansible/+/869748 | 13:45 |
| mgariepy | not that it's that complicated but there is pretty much a dict of uuid:names for eache component. once we get the idea of how the commands are structured it's pretty clean too. | 13:46 |
| jamesdenton | moha7 the openstack_user_config.yml looks pretty good, but i would just say that because you've defined br-ex twice (once for vlan, once for flat) you may want to remove network_interface from one of them. | 13:50 |
| mgariepy | this is the task that is causing the duplicate : https://zuul.opendev.org/t/openstack/build/869e5843628a4a2f9a4318d253af3e77/log/job-output.txt#17366-17370 | 13:54 |
| mgariepy | https://github.com/openstack/openstack-ansible-os_neutron/blob/master/tasks/providers/setup_ovs_ovn.yml#L55-L77 | 13:54 |
| jamesdenton | yeah, i can't recall if when it goes to process the second bridge w/ the same name, and connect a port that already is connected, if it will complain or not | 13:56 |
| jamesdenton | but state is "present" so maybe it skips? | 13:56 |
| jamesdenton | ERROR neutron.plugins.ml2.managers raise ValueError(_("Value %(value)s in mapping: '%(mapping)s' "#0122023-01-16 15:41:07.821 3130 ERROR neutron.plugins.ml2.managers ValueError: Value br-vlan in mapping: 'flat:br-vlan' not unique#0122023-01-16 15:41:07.821 3130 ERROR neutron.plugins.ml2.managers | 13:57 |
| jamesdenton | ahh ok, i missed that. | 13:57 |
| jrosser | thats from the neutron config file isnt it | 13:57 |
| jamesdenton | so, the issue with that is OVN no longer allows two mappings w/ the same bridge | 13:58 |
| jamesdenton | the fix, IIRC, is to rename the "flat" network to vlan (to match the other) but keep type as flat. I think. i can lab it out, but it will be a while before i can get to it | 13:58 |
| moha7 | jamesdenton: everything works well except the access from the outside; My case: virtualization infra: ProxMox with a bunch of subnets under one vlan (ID: 3647); Each Ubuntu machine has 4 interface (for example enp6s18 for br-mgmt as a bridge routed to the a subnet under vlan3647, netplan: http://ix.io/4lqw, Machine interfaces: https://ibb.co/3c2m5nd); I can create external network successfully and when intruduced to a | 13:58 |
| moha7 | router, the router takes a random IP from the external subnet range. This router gateway is oingable from the insiade of instances (then SNATing works); but that gateway IP is not available from the outside, for example from the upstream router wher the vlan3647 has ben defined) | 13:58 |
| jamesdenton | or, eliminate flat altogether | 13:58 |
| jamesdenton | kk - i have a call right now but will take a look at this when i';m done, nmaybe an hour or less | 13:59 |
| jamesdenton | moha7 what kind of network is the external provider network? | 14:02 |
| jamesdenton | and did you use vlan or flat? | 14:02 |
| moha7 | "everything works well except the access from the outside" for lab1 where there'e no flat network | 14:03 |
| moha7 | "Error: Value br-vlan in mapping: 'flat:br-vlan' not unique" ---> on lab2 where I add a flat network too | 14:03 |
| moha7 | on lab1? only vlan | 14:03 |
| jamesdenton | lab1 and lab2 are different machine names? or different lab environments? | 14:04 |
| admin1 | moha7, remove the flat network | 14:09 |
| admin1 | and then tcpdump -ni any -e vlan XYZ to see if you can see the tagged vlan packets | 14:09 |
| admin1 | that way, you can see where its seen | 14:10 |
| admin1 | maybe its blocked in the proxmox host | 14:10 |
| moha7 | jamesdenton: two different labs | 14:10 |
| moha7 | two env | 14:10 |
| jamesdenton | cool, and this http://ix.io/4lqn corresponds to which? | 14:11 |
| moha7 | where I get this error: "Error: Value br-vlan in mapping: 'flat:br-vlan' not unique" | 14:14 |
| jamesdenton | yes | 14:14 |
| jamesdenton | to fix that, try changing 'net_name: "flat"' to 'net_name: "vlan"' | 14:14 |
| jamesdenton | so, you'll have two with net_name vlan, but one whose type is vlan adn the other flat | 14:15 |
| jamesdenton | the playbooks should merge them | 14:15 |
| jamesdenton | and then on the flat one, also removenetwork_interface: "enp6s21" | 14:15 |
| jamesdenton | *remove | 14:15 |
| jrosser | ^ this was one of the most confusing things for me in the early days, where name/type use the same strings | 14:15 |
| moha7 | then I use `net_name: "provider"` for better understanding | 14:15 |
| moha7 | for both of them | 14:16 |
| jamesdenton | ok that's fine | 14:16 |
| *** dviroel is now known as dviroel|lunch | 15:03 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_nova master: Support configuration of resource providers with config files https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/870559 | 15:17 |
| *** dviroel|lunch is now known as dviroel | 15:28 | |
| jamesdenton | moha7 any luck? | 15:51 |
| moha7 | I let him do his job and left the house. I will give you the result right here in a few hours | 15:59 |
| moha7 | Deployment (from scratch) on 5 nodes with local Ubuntu repo and SSD disk takes about 5 hours | 16:00 |
| jrosser | it shouldnt be necessary to redo everything | 16:01 |
| moha7 | 30min for setup-hosts | 16:02 |
| moha7 | 1h30m for setup-infra | 16:02 |
| moha7 | 3h for setup-openstack | 16:02 |
| moha7 | jamesdenton: ^ | 16:02 |
| jamesdenton | 1:30, woof | 16:02 |
| jamesdenton | so, if it's not prod, i tend to disable ansible hardening, as that takes a bit | 16:03 |
| jrosser | for just config changes re-running just the neutron playbook should suffice | 16:03 |
| jamesdenton | ^^ | 16:03 |
| jrosser | maybe there is some stickyness in old OVS config to delete by hand, but thats not my expertise | 16:04 |
| moha7 | jrosser: yeah, I asked you yesterday (`--tags neutron-config`); But I wanted to be sure everything would be well | 16:04 |
| jamesdenton | well, i wouldn't use the tag but just running os-neutron-install would be ok | 16:04 |
| jrosser | moha7: what will you do with your production deployment? :) | 16:04 |
| jrosser | this is practice | 16:04 |
| moha7 | ((((= | 16:04 |
| moha7 | you're right | 16:05 |
| jrosser | the worst place to end up is being too worried to ever do maintainance | 16:05 |
| jrosser | and also that openstack-ansible is not really magical in any way | 16:06 |
| jrosser | it just installs some stuff, writes some config files and manages some services | 16:06 |
| jrosser | it's totally possible to debug whats going on, maybe edit some files by hand to get things working | 16:06 |
| jrosser | then figure out what changes need to be made to your variables to end up with that working config | 16:07 |
| moha7 | jamesdenton: hardening's been disabled in this way: http://ix.io/4lqp | 16:19 |
| admin1 | i have a test lab today to setup ovn :) | 16:20 |
| jamesdenton | moha7 good deal | 16:38 |
| admin1 | moha7, what is the rational behind having this everywhere ? python3-openstackclient | 17:00 |
| jrosser | admin1: i already mentioned that this was not necessary | 17:03 |
| jrosser | and also perhaps breaking things | 17:03 |
| moha7 | admin1: When you are in a test lab, it's a facility to have it any where; For example T-shooting nova on nova container and you can use `openstack hypervisor list` at the same place; Bu I removed it from the variables file as jrosser mentioned it's a deb package going to work alongside with source-based services. | 17:05 |
| jrosser | please don't install it like that | 17:05 |
| jrosser | just keep a window/tab open in a utility container | 17:06 |
| moha7 | I didn't; removed. | 17:06 |
| moha7 | +1 | 17:06 |
| jrosser | there is an openrc file written to the utility container | 17:07 |
| jrosser | you should be able to source that and use the CLI | 17:07 |
| admin1 | there is a utility container just for that purpose | 17:13 |
| admin1 | with admin creds already setup | 17:14 |
| admin1 | yeah else you are going to break your containers when openstack might pull its own stuff and the source needs its own stuff | 17:14 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow git servers for openstack services and tempest to be overridden https://review.opendev.org/c/openstack/openstack-ansible/+/869748 | 17:59 |
| moha7 | In documents: "2. Check the integrity of your YAML files. (Note: To check your YAML syntax online, you can use the YAML Lint program.)" | 19:42 |
| moha7 | What does 'integrity' mean here? | 19:42 |
| moha7 | Ref: https://docs.openstack.org/project-deploy-guide/openstack-ansible/zed/run-playbooks.html | 19:43 |
| jamesdenton | integrity==format here | 19:45 |
| jamesdenton | syntax | 19:46 |
| moha7 | `nohup bash -c "time openstack-ansible /opt/openstack-ansible/playbooks/setup-hosts.yml" &` ---> ~28min (https://ibb.co/mSQ9zJM) | 19:52 |
| moha7 | `setup-infrastrucure.yml` --> This time: ~41m (https://ibb.co/4YvskCN) | 19:52 |
| moha7 | `setup-openstack.yml` --> in progress, but probably takes a bit longer because of this retried task: https://ibb.co/dgp00b2 | 19:52 |
| moha7 | Got exhausted; This is the last time I'll run it from scratch ((: | 19:54 |
| moha7 | Netx time, I just run the required playbooks. | 19:54 |
| jamesdenton | k | 20:03 |
| jrosser | moha7: that task should not fail | 20:08 |
| jrosser | this is probably why you have some slowness | 20:09 |
| jrosser | moha7: can you please try `git clone https://opendev.org/openstack/keystone.git` from one of your hosts | 20:11 |
| jrosser | i am interested to see the speed / throughput and how long that takes | 20:11 |
| moha7 | Oops! Started from 5MB but decreased to 100KB very fast! | 20:20 |
| moha7 | 100KB/s | 20:20 |
| moha7 | So, I need to talk with the network team | 20:21 |
| moha7 | jrosser: ^ | 20:21 |
| jrosser | moha7: i think you have connectivity trouble to opendev.org | 20:23 |
| jrosser | i had the same last week with just 50kbits through 100G+ uplink | 20:23 |
| jrosser | it was clear that when our transit went via zayo it was terrible | 20:23 |
| jrosser | other hosts i had which went via cogent were fine | 20:24 |
| jrosser | moha7: i also have a work-in-progress patch which would let you swing all the repos over to github easily https://review.opendev.org/c/openstack/openstack-ansible/+/869748 | 20:29 |
| moha7 | Yeah, there's probably something with the my path to the git repo as I get the stable speed of ~5MB/s for downloading `wget https://releases.ubuntu.com/22.04.1/ubuntu-22.04.1-live-server-amd64.iso`; We have some proxies. I'll ask network guys to put my management VLAN into a VPN tunnel. | 20:39 |
| jrosser | moha7: i think this might help with your deployment time - some of the openstack service repos are large | 20:40 |
| *** dviroel is now known as dviroel|out | 23:25 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!