| moha7 | Hi | 06:24 |
|---|---|---|
| noonedeadpunk | o/ | 08:52 |
| jrosser | morning | 09:06 |
| * noonedeadpunk trying to check https://bugs.launchpad.net/openstack-ansible/+bug/2002897 | 09:12 | |
| noonedeadpunk | I'm not sure if it was discussed yestarday though | 09:12 |
| jrosser | no it wasnt | 09:14 |
| jrosser | i wonder what version that way | 09:14 |
| jrosser | also very interesting ML thread about vgpu | 09:15 |
| noonedeadpunk | that basically stopped me from proposing bump for Z as sounds quite critical | 09:15 |
| noonedeadpunk | oh yes | 09:15 |
| noonedeadpunk | I was o_O that you can have different vgpu types on one card | 09:15 |
| jrosser | it is very interesting to see possibility that cyborg handles MIG | 09:15 |
| noonedeadpunk | As that was not possible like 6 month ago | 09:15 |
| jrosser | yeah, thats mig + sriov right though? | 09:15 |
| jrosser | as they list the pcie addresses for each type | 09:16 |
| noonedeadpunk | Well I don't have mig on mine... | 09:16 |
| noonedeadpunk | But vGPU is quite the same | 09:16 |
| noonedeadpunk | We still list PCI | 09:16 |
| jrosser | i did look quickly yesterday at cyborg and its probably not to bad to make a role | 09:16 |
| noonedeadpunk | The problem is, that nvidia driver says you can't create any more mdev on any other type once 1 vgpu was created of specific one | 09:17 |
| jrosser | unfortunately no time at all in my team to investigate or test it :( | 09:17 |
| noonedeadpunk | Well, I had a look at cyborg year ago or so and they've struggled to have proper vgpu support back then. And overall I missed the poit why to use it as usecases are mostly covered with nova solely as Sean has written | 09:17 |
| jrosser | though having said all this the biggest requirement i get from users right now is "whole A100/80G" | 09:18 |
| noonedeadpunk | Well. We have request for whole T4 | 09:18 |
| noonedeadpunk | Still. | 09:18 |
| noonedeadpunk | So if saying to have A100 - we likely can split in 3 | 09:19 |
| noonedeadpunk | But we have A10 | 09:19 |
| jrosser | anyway - to bug 2002897 | 09:19 |
| jrosser | it doesnt say what release does it? | 09:19 |
| noonedeadpunk | Nope, it does not. I just assumed it's Z | 09:20 |
| jrosser | oh well | 09:20 |
| jrosser | `type_drivers = geneve,vlan,flat` | 09:20 |
| jrosser | `tenant_network_types = vxlan,flat,vlan` | 09:20 |
| jrosser | that looks suspect | 09:20 |
| noonedeadpunk | huh, yes | 09:21 |
| jrosser | i think i have a recent AIO, lets compare | 09:21 |
| noonedeadpunk | To be frank - Ive never checked horizon on Z for OVN even in AIO | 09:21 |
| jrosser | ok well in my AIO i have `type_drivers` matching `tenant_network_types` | 09:22 |
| jrosser | where do we define `neutron_provider_networks` | 09:26 |
| noonedeadpunk | I think that's the issue https://opendev.org/openstack/openstack-ansible-os_horizon/src/branch/master/templates/horizon_local_settings.py.j2#L343 | 09:26 |
| noonedeadpunk | And that doesn't look configurable or overridable. Well. We have some sort of override, but I guess they will need to define whole OPENSTACK_NEUTRON_NETWORK | 09:27 |
| noonedeadpunk | neutron_provider_networks are now only in neutron role defaults I believe | 09:28 |
| jrosser | oh well hold on | 09:28 |
| jrosser | regardless of horizon, neutron server is down | 09:29 |
| noonedeadpunk | well, I assume it's down specifically because of having vxlan there | 09:29 |
| noonedeadpunk | It's less of concern now, as I think bug itself quite valid | 09:30 |
| noonedeadpunk | Even if neutron was configured properly | 09:30 |
| jrosser | yeah, we just need to be clear that theres two things going on | 09:31 |
| jrosser | also ovn does support vxlan | 09:34 |
| noonedeadpunk | it does? | 09:35 |
| jrosser | yes as it has to interoperate with physical TOR switches | 09:35 |
| noonedeadpunk | um... I'm confused then | 09:36 |
| jrosser | https://bugzilla.redhat.com/show_bug.cgi?id=1881704 | 09:36 |
| noonedeadpunk | as everyone I talked about migration from OVS were marking migration to geneve as main pain point | 09:36 |
| jrosser | so we need to add `horizon_supported_neutron_provider_types` to os_horizon role? | 09:38 |
| noonedeadpunk | yeah, smth like that | 09:38 |
| jrosser | there is also the special value '*' which enables them all | 09:38 |
| noonedeadpunk | Now I'm really confused why we made all that complexity with vxlan/geneve | 09:39 |
| jrosser | i'm not sure | 09:40 |
| jrosser | geneve adds flow specific information to the headers rather than just source/destination | 09:40 |
| jrosser | but then none of that seems to be surfaced/leveraged from a user POV so i also am kind of confused by it | 09:41 |
| jrosser | on another topic, do you have any idea how to do this? https://paste.opendev.org/show/b8oNDxVrtGZ3a0ieYr7T/ | 09:41 |
| noonedeadpunk | I was quite sure that vxlan simply not implemented | 09:42 |
| jrosser | i try to implement something to disassemble a url, then rebuild it according to some spec supplied in a string | 09:42 |
| jrosser | but the answer is always the spec string itself rather than being evaluated as a template | 09:43 |
| noonedeadpunk | Um, you've forgotten {{ }}? | 09:43 |
| jrosser | well no, becasue i want to supply the spec as a var | 09:44 |
| jrosser | to make it changeable | 09:44 |
| noonedeadpunk | ah | 09:44 |
| jrosser | its totally fine if i add the {{ }} :) | 09:44 |
| noonedeadpunk | and format doesn't work for you for some reason? | 09:49 |
| noonedeadpunk | https://jinja.palletsprojects.com/en/3.1.x/templates/#jinja-filters.format | 09:49 |
| noonedeadpunk | I'm not sure if you can have there named arguments though | 09:50 |
| jrosser | hmm let me try that | 09:54 |
| noonedeadpunk | jrosser: https://paste.opendev.org/show/br8DLxuBlhWteNLO2iMd/ | 09:58 |
| jrosser | noonedeadpunk: thats excellent | 10:06 |
| moha7 | The 3rd step deployed in 7 hours! | 10:20 |
| moha7 | Issue: metadata injection into the instances does not worked well in this new deployed env. Instances got IP and could ping their router hands, but the hostname did not set on it! Ctrl+F for 169.254.169.254 in http://ix.io/4ltx to see the error. | 10:21 |
| moha7 | jrosser: ERROR ovsdbapp.backend.ovs_idl.connection [-] non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>: ValueError: non-zero flags not allowed in calls to send() on <class 'eventlet.green.ssl.GreenSSLSocket'>023-01-17 10:34:26.475 2290 ERROR ovsdbapp.backend.ovs_idl.connection Traceback (most recent call last) | 10:28 |
| moha7 | Solution: I copied SSLs (these items: https://paste.opendev.org/show/bXxUevJGtWu023mfZmNE/) from ` /etc/neutron/plugins/ml2/ml2_conf.ini` and inserted to ` /etc/neutron/neutron_ovn_metadata_agent.ini`. Now the instance gets metadata after some retrying on checking http://169.254.169.254/2009-04-04/instance-id | 10:28 |
| jrosser | moha7: if i remember this is what you experienced a couple of weeks ago? | 10:28 |
| moha7 | jamesdenton: I created two types of external networks: Flat and VLAN; And then I created two routers each in one of these external networks. For the router with a hand in the flat-type external network, everything works well and I can ping the gateway of the router from the outside of OpenStack (Thanks for your hints). But the gateway of the other router that is in the VLAN-type external network is not accessible from | 10:31 |
| moha7 | outside! | 10:31 |
| admin1 | moha7, you can also use configdrive | 10:31 |
| admin1 | so if metadata does not work, configdrive supplies those details | 10:32 |
| moha7 | (Note: OpenStack has been installed within a VM hosted on the ProxMox. By tcpdump on the ProxMox I see packets of OpenStack instances in the flat-type network, but nothing for the other one. That means there's something with OpenStack itself.) | 10:32 |
| admin1 | especially useful when you have workload that does not connect to outside world | 10:32 |
| moha7 | jrosser: I think so, I have ad lots of issues during these days. | 10:33 |
| moha7 | had* | 10:33 |
| noonedeadpunk | hm, there's no options for ssl in neutron docs https://docs.openstack.org/networking-ovn/latest/configuration/networking_ovn_metadata_agent.html | 10:35 |
| noonedeadpunk | oh, well, that page is super old | 10:36 |
| jrosser | i was asking last time this happened if it was a service restart | 10:36 |
| jrosser | which would be indistinguishable from changing config and restarting a service | 10:36 |
| moha7 | admin1: you mean like this: https://serverfault.com/a/1114711/968071 | 10:37 |
| admin1 | yes | 10:37 |
| moha7 | `configdrive` as a support? | 10:37 |
| admin1 | but you can pass this as nova override | 10:37 |
| admin1 | during deployment | 10:38 |
| jrosser | noonedeadpunk: thoughts on this as an approach? https://paste.opendev.org/show/bQRTtLCc4WgWWhBp0nep/ | 10:40 |
| jrosser | idea is to let you reconstruct the collection URL to point at a mirror | 10:40 |
| jrosser | for example `export OSA_COLLECTIONS_GIT_LOCATION="https://mirror.example.com/{hostname}{path}"` | 10:43 |
| noonedeadpunk | well. then it would make sense to add auth as well... | 10:43 |
| moha7 | jrosser: The issue happens during creating the instance and also if you restart the instance, it takes around 30 minutes to pass and comes up. | 10:44 |
| jrosser | i think netloc would add username/password from original URL | 10:44 |
| noonedeadpunk | ah, might be | 10:44 |
| jrosser | ah no it wont | 10:44 |
| jrosser | need to add username/password as well i think | 10:45 |
| jrosser | moha7: i think that you need to debug this with someone like jamesdenton who has great expertise with neutron | 10:45 |
| moha7 | +1 | 10:46 |
| noonedeadpunk | well, except names of variables I think it's quite good and flexible | 10:48 |
| noonedeadpunk | as I'd use user_collections_git_location_schema or smth instead | 10:49 |
| jrosser | sure | 10:49 |
| jrosser | it was really hard to work out how to do this | 10:49 |
| jrosser | as there are no collections installed and nowhere to write vars files | 10:49 |
| noonedeadpunk | Well, it took quite some time to realize that as well | 10:49 |
| noonedeadpunk | s/realize/read/ | 10:49 |
| jrosser | i am also not sure if the URL from user-collection-requirements should be rewritten | 10:49 |
| jrosser | my feeling is that they should be left alone as that is user input | 10:50 |
| noonedeadpunk | yup, agree here | 10:50 |
| jrosser | but interested in your thoughts on that too | 10:50 |
| jrosser | right - so the code needs adjusting to only modify non-user URLs | 10:50 |
| noonedeadpunk | as it might be annoying if user provided urls got overwritten/corrupted in an unexpected way | 10:51 |
| jrosser | yes | 10:51 |
| noonedeadpunk | but, I'm not sure what can get wrong with such default template. Except, if you change it for general you might not it to be applied for user provided ones | 10:52 |
| noonedeadpunk | So some separation is still needed regardless | 10:52 |
| *** dviroel|out is now known as dviroel | 11:44 | |
| amarao | Can we bump `ansible-role-requirements.yml` for stable/zed? I'm waiting for circular dependencies fix... | 11:50 |
| noonedeadpunk | amarao: well, you can override role if required in /etc/openstack_deploy/user-role-requirements.yml | 11:53 |
| noonedeadpunk | Im checking one another bug right now that seems to affect Zed | 11:54 |
| amarao | I know, I've tested fix this way. Now I'm waiting it get it to stable. | 11:54 |
| noonedeadpunk | and during bumps we update all roles right before creating new tag | 11:54 |
| noonedeadpunk | so it will be part of 26.1.0 | 11:55 |
| amarao | ack, thanks. | 11:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Allow to override supported_provider_types https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870801 | 12:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Allow to override supported_provider_types https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870801 | 12:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: [doc] Add LXB scenario documentation https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/870805 | 13:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Add Glance tempest plugin repo to testing SHA pins list https://review.opendev.org/c/openstack/openstack-ansible/+/870777 | 13:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Add Glance tempest plugin repo to testing SHA pins list https://review.opendev.org/c/openstack/openstack-ansible/+/870777 | 13:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Add Glance tempest plugin repo to testing SHA pins list https://review.opendev.org/c/openstack/openstack-ansible/+/870778 | 13:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add container_ip option for metal hosts https://review.opendev.org/c/openstack/openstack-ansible/+/870113 | 14:02 |
| mgariepy | jamesdenton, i'm looking at your mnaiov2 code. | 14:03 |
| mgariepy | using existing images would be nice addition | 14:04 |
| mgariepy | same for flavor | 14:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Bump OpenStack-Ansible Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/870810 | 14:13 |
| opendevreview | chandan kumar proposed openstack/openstack-ansible-os_tempest master: Add support for whitebox-neutron-tempest-plugin https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/870812 | 14:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: [doc] Add LXB scenario documentation https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/870805 | 14:17 |
| jamesdenton | thanks mgagne | 14:21 |
| jamesdenton | sorry, mgariepy | 14:21 |
| jamesdenton | hi moha7 - we might be missing metadata config for ssl, i will look at that today | 14:23 |
| jamesdenton | and you're saying that your flat external provider network works, but a vlan-tagged provider network doesn't? | 14:23 |
| noonedeadpunk | fwiw I tried to look at ovn metadata code and I haven't seen good half of options we have in it's config ;( | 14:24 |
| noonedeadpunk | And indeed we might have some race condition on service restart as jrosser mentioned | 14:24 |
| jrosser | is this stuff we run via uwsgi? | 14:25 |
| noonedeadpunk | um, no, I think that uwsgi for neutron-api don't work if ovn is used | 14:25 |
| noonedeadpunk | so we default uwsgi to disable for ovn | 14:25 |
| jrosser | ah ok becasue we still have the bug there where only one of the many neutron.conf... is passed to the uwsgi service | 14:26 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/defaults/main.yml#L189 | 14:26 |
| noonedeadpunk | um, haven't someone pushed a fix for that? | 14:26 |
| jrosser | i don't think so | 14:26 |
| noonedeadpunk | It was genericswith? | 14:26 |
| jrosser | yes thats where we found it | 14:26 |
| jamesdenton | jrosser i had a fix for that, i thought | 14:26 |
| jrosser | afaik we just moved that config to neutron.conf :( | 14:27 |
| jamesdenton | might need to hit eavesdrop | 14:27 |
| jrosser | via override | 14:27 |
| jrosser | i did do some looking at uwsgi how to pass multiple config files | 14:27 |
| noonedeadpunk | yeah, jamesdenton totally found the fix | 14:27 |
| noonedeadpunk | or well, the way how to handle through uwsgi | 14:28 |
| noonedeadpunk | or maybe it was you... | 14:28 |
| jrosser | perhaps :) | 14:28 |
| noonedeadpunk | https://bugs.launchpad.net/openstack-ansible/+bug/1987405 | 14:28 |
| noonedeadpunk | So there's solution I guess, but no patch | 14:28 |
| jamesdenton | whoops :D | 14:29 |
| jrosser | right so multiple use of --config-file | 14:29 |
| jrosser | looks like one of those things where we need to distinguish between string and list | 14:29 |
| jrosser | and iterate if it's a list | 14:30 |
| mgariepy | jamesdenton, have you seen this project ? https://github.com/ComputeCanada/magic_castle | 14:33 |
| jamesdenton | i have not | 14:34 |
| jamesdenton | but i love duplicating efforts :D | 14:34 |
| mgariepy | it's not the same thing :D | 14:35 |
| mgariepy | it's more an hpc training ground | 14:36 |
| mgariepy | but the use of TF is quite nice there. | 14:36 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add facility to rewrite source URLs for ansible collections during bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/870820 | 14:36 |
| jrosser | noonedeadpunk: ^ this is pretty much for your isolated deploy - i don't need this on my deploy hosts | 14:37 |
| jamesdenton | it is helpful, i'm pretty new to terraform | 14:37 |
| mgariepy | ho. also. booting from volume would be nice too :d | 14:37 |
| noonedeadpunk | jrosser: much appreciated! I was just gonna check out these patches for bump script | 14:38 |
| jrosser | could do with a sanity check on that last one, i've tried it in an AIO | 14:38 |
| jamesdenton | yes, i thought about that. i'm implementing small ceph VMs with small OSDs | 14:39 |
| jrosser | need to look next at ansible-role-requirements and what to do with that | 14:39 |
| mgariepy | should i create issue on your project? | 14:39 |
| noonedeadpunk | well... I assume I'm going to have another problem with OSA_COLLECTIONS_GIT_SCHEMA. As I assume user.rc (https://opendev.org/openstack/openstack-ansible/src/commit/e315e2e32738aed2b0a59ef8099280dee698607b/scripts/openstack-ansible.sh#L58) is not going to work during bootstrap | 14:44 |
| noonedeadpunk | or it will... | 14:44 |
| noonedeadpunk | nah, we run directly /opt/ansible-runtime/bin/ansible-playbook | 14:45 |
| noonedeadpunk | well, I will take care of that anyway | 14:46 |
| jamesdenton | mgariepy yes, that would be helpful. thank you. | 14:57 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Jan 17 15:00:18 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| jrosser | hello | 15:00 |
| damiandabrowski | hi! | 15:00 |
| jamesdenton | o/, sortof | 15:00 |
| noonedeadpunk | #topic office hours | 15:02 |
| mgariepy | hello o/ | 15:02 |
| noonedeadpunk | I was about to do bump for Z, but then https://bugs.launchpad.net/openstack-ansible/+bug/2002897 raised | 15:03 |
| noonedeadpunk | Fix for it has been already pushed, so given some reviews on https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870801 it should be fast after all | 15:04 |
| noonedeadpunk | Then for OVN driver for Octavia, we're still missing fixing SHA for the plugin. jamesdenton do you want to push the change? I can do it if needed | 15:05 |
| jamesdenton | if you can that would be great, i'm tied up a bit | 15:05 |
| admin1 | i am trying to recreate a setup for upgrade test .. its on focal and tag 24.4.2 .. having trouble with rabbitmq-server install rabbitmq-server : Depends: erlang-base (< 1:25.0) but 1:25.2-1 is to be installed or erlang-base-hipe (< 1:25.0) but it is not going to be installed or esl-erlang (< 1:25.0) but it is not installable | 15:05 |
| noonedeadpunk | #link https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/868462 | 15:05 |
| NeilHanlon | o/ | 15:06 |
| admin1 | oh .. we are in a meeting .. | 15:06 |
| admin1 | \o | 15:06 |
| noonedeadpunk | admin1: https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/868107 | 15:06 |
| admin1 | thank you noonedeadpunk | 15:07 |
| noonedeadpunk | jamesdenton: ok, will do. | 15:07 |
| jamesdenton | it will also help me understand what you mean about the sha :) | 15:07 |
| noonedeadpunk | btw, I'm going to join you NeilHanlon on FOSDEM :) | 15:07 |
| mgariepy | cool | 15:08 |
| NeilHanlon | :gasp: | 15:08 |
| admin1 | i have 50/50 chances of coming in fosdem | 15:08 |
| noonedeadpunk | jamesdenton: I mean adding octavia_ovn_octavia_provider_git_install_branch to https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/defaults/repo_packages/openstack_services.yml | 15:09 |
| noonedeadpunk | And yes, we also need to decide a scope we want for AA | 15:09 |
| jamesdenton | ahhh yes yes, thanks | 15:10 |
| noonedeadpunk | I've checked etherpad lately, and we already merged most of things | 15:11 |
| noonedeadpunk | #link https://etherpad.opendev.org/p/osa-antelope-ptg | 15:11 |
| jamesdenton | woot | 15:11 |
| noonedeadpunk | We have huge topic about internal TLS at very least | 15:12 |
| noonedeadpunk | Also finilizing skyline is smth we can totally do | 15:13 |
| noonedeadpunk | oh, and we also do have huge question about modular libvirt | 15:14 |
| noonedeadpunk | And things regaridng CI/functional testing, but that's not blocking releasing at least | 15:15 |
| noonedeadpunk | As of functional changes I'd try to minimize them as Zed was quite stressfull | 15:15 |
| noonedeadpunk | We always saying about releasing earlier, but never managed to do so :( | 15:16 |
| noonedeadpunk | It's mostly me to blame though, I guess | 15:16 |
| jrosser | is internal TLS realistic for AA? | 15:16 |
| noonedeadpunk | damiandabrowski: ? | 15:16 |
| jrosser | like huge and we have not really started | 15:16 |
| jamesdenton | noonedeadpunk you're a beast, thank you | 15:17 |
| jrosser | i do wonder how much we need to step back and just let the OVN stuff settle | 15:17 |
| damiandabrowski | jrosser: yes totally | 15:17 |
| damiandabrowski | and i to have started | 15:17 |
| damiandabrowski | do* | 15:17 |
| jamesdenton | maybe a big docs push for AA? | 15:17 |
| noonedeadpunk | yes, this is good ^ | 15:18 |
| noonedeadpunk | Let me start new etherpad maybe? | 15:18 |
| jamesdenton | sure | 15:19 |
| damiandabrowski | right now i'm testing if it's worth to configure haproxy services separately in os_ roles rather then preconfigure all of them with haproxy_server | 15:19 |
| damiandabrowski | today i deployed all base openstack services like this, but i need to improve few things before I push a change | 15:20 |
| noonedeadpunk | #link https://etherpad.opendev.org/p/osa-bobcat-ptg | 15:21 |
| noonedeadpunk | decided to start PTG etherpad for tracking these things | 15:21 |
| noonedeadpunk | yes, totally jrosser I want also to calm things down with OVN a bit, and maybe do docs and bug fixes mostly | 15:23 |
| jrosser | it is frustrating not to be able to resolve moha7 troubles too | 15:23 |
| jrosser | its not possible to determine if it is environment trouble or actual bugs | 15:23 |
| noonedeadpunk | but as internal tls was started quite a while ago and presumably damiandabrowski will have some time for it, and given there won't be anything really breaking I'd say we can see how this will go | 15:24 |
| noonedeadpunk | yeah, true. But I'd say with OVN it's in general hard to tell/debug some things. We also had some step back in using journals for logging | 15:25 |
| noonedeadpunk | As example - ovn and gluster that do just local logs | 15:25 |
| jamesdenton | jrosser i think a small combination, but likely due to lack of clarity in the docs | 15:26 |
| jamesdenton | i will do a deploy now and test the SSL metadata issue mentioned | 15:26 |
| jrosser | it is also hard translating bugs/trouble in multinode onto an AIO | 15:26 |
| noonedeadpunk | I'd say what worth testing is somehow test ordering issues when config is changed | 15:27 |
| jamesdenton | agreed. Hoping to solve that with this MNAIOv2 thing i'm working on, but it's dependent on deploying VMs on OpenStack vs KVM. | 15:27 |
| jrosser | or at the end of a fresh deploy, i think thats a really good question | 15:27 |
| jrosser | if just letting the playbooks run to success is it working/broken | 15:27 |
| noonedeadpunk | I assume it's working, otherwise tempest won't succeed... | 15:28 |
| noonedeadpunk | Hm, does octavia leverage metadata in any way? | 15:28 |
| *** dviroel is now known as dviroel|lunch | 15:28 | |
| noonedeadpunk | as for cirros it's not really required... | 15:28 |
| jamesdenton | i don't know - the agent is http based i think | 15:29 |
| jamesdenton | no ssh key required afaik | 15:29 |
| jrosser | there is possibility of a debug key - not sure how that gets in | 15:29 |
| jamesdenton | metadata, but i don't think we test that | 15:29 |
| noonedeadpunk | I assume through metadata, but that could be config drive as well | 15:29 |
| noonedeadpunk | yeah, so that's good to test... | 15:30 |
| NeilHanlon | jamesdenton: random question, any OSV resources you recommend for deep dives/learning? | 15:33 |
| jamesdenton | OVS? Hmm, not offhand | 15:38 |
| jamesdenton | happy to look | 15:38 |
| mgariepy | i think the best way to learn is to have issue that you want to fix .. :D | 15:38 |
| NeilHanlon | hehe.. yeah, that's been my learning path so far but I was looking for a more unified/top down approach to the architecture, I guess. sometimes it leaves me scratching my head | 15:45 |
| NeilHanlon | and yeah jamesdenton: OVS.. oops :) | 15:45 |
| NeilHanlon | this helped.. a little :P https://www.kernel.org/doc/html/latest/networking/openvswitch.html | 15:46 |
| jamesdenton | i have sustained surface level knowledge, then i go down a rabbit hole when necessary and promptly forget | 15:46 |
| mgariepy | i want to add some doc on how to find specific stuff in the ovn/ovs db | 15:46 |
| NeilHanlon | well at least it sounds like it's not just me 😂 | 15:47 |
| jamesdenton | there's just too much to know! | 15:47 |
| NeilHanlon | good company, anyways :) | 15:47 |
| mgariepy | https://blog.russellbryant.net/2016/11/11/ovn-logical-flows-and-ovn-trace/ | 15:48 |
| jamesdenton | :) | 15:48 |
| jamesdenton | 6 years ago... is a lifetime :D | 15:48 |
| mgariepy | well the base is kinda the same anyway ;p | 15:48 |
| mgariepy | and also doc is always outdated ;p haha | 15:48 |
| NeilHanlon | problem? buy a <company> license and our experts will fix it for you! | 15:49 |
| jamesdenton | juju what? | 15:49 |
| mgariepy | yeah.. pay us and we will figure it out to fix it.. | 15:49 |
| mgariepy | that's how it works. | 15:49 |
| NeilHanlon | anyways, sorry for the diversion lol | 15:52 |
| johnsom | Octavia defaults to using config drive to get the initial configuration data on an amphora boot. It can use the metadata service, but that was unstable back in the day we developed this (since fixed), so we stuck to config drive. | 15:54 |
| noonedeadpunk | thanks johnsom, that explains why we don't see failure in our tests if we have broken metadata :) | 15:55 |
| jamesdenton | nice | 15:59 |
| opendevreview | James Denton proposed openstack/openstack-ansible master: Add Octavia OVN Provider repo requirements https://review.opendev.org/c/openstack/openstack-ansible/+/870834 | 16:00 |
| noonedeadpunk | #endmeeting | 16:02 |
| opendevmeet | Meeting ended Tue Jan 17 16:02:35 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-01-17-15.00.html | 16:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-01-17-15.00.txt | 16:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-01-17-15.00.log.html | 16:02 |
| NeilHanlon | ty noonedeadpunk ! | 16:07 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add facility to rewrite source URLs for ansible collections during bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/870820 | 16:12 |
| jrosser | hmm do we ever take tempest plugins from local zuul repos - i don't see them in required-projects | 16:17 |
| jrosser | i think we only deal with playbooks/defaults/repo_packages/openstack_services.yml in the past | 16:19 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add tempest and tempest plugins to required jobs for source deploys https://review.opendev.org/c/openstack/openstack-ansible/+/870839 | 16:26 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow git servers for openstack services and tempest to be overridden https://review.opendev.org/c/openstack/openstack-ansible/+/869748 | 16:28 |
| noonedeadpunk | jrosser: I had patch for that | 16:31 |
| jrosser | oh cool | 16:31 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/834289 | 16:32 |
| noonedeadpunk | as you might see I made some mistake somewhere | 16:32 |
| *** dviroel|lunch is now known as dviroel | 16:35 | |
| jrosser | oh and i forgot about the ubuntu distro needing tempest source | 16:39 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add tempest and tempest plugins to required jobs for source deploys https://review.opendev.org/c/openstack/openstack-ansible/+/870839 | 16:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Allow git servers for openstack services and tempest to be overridden https://review.opendev.org/c/openstack/openstack-ansible/+/869748 | 16:42 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add variable to control no_log in service_setup role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/869604 | 16:45 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add variable to control no_log in mq_setup role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/869602 | 16:46 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Add variable to control no_log in service_setup role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/869604 | 16:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Fix usage of _oslodb_setup_nolog https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/870842 | 16:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Add variable to control no_log in service_setup role https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/869604 | 16:50 |
| noonedeadpunk | Well, I tried to leverage loop_control labels as contrary to no_log but without much result | 16:51 |
| noonedeadpunk | I had hopes that it's somehow possible to do register and some logic for no_log depending task is failing or not, but register happens after no_log being evaluated | 16:52 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Fix no_log variable templating in db_setup role. https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/870843 | 16:53 |
| jrosser | noonedeadpunk: ^ i completely expected that to behave like a when: .... i.e the "{{ }}" not required | 16:54 |
| noonedeadpunk | yeah, I was under same impression before tested | 16:54 |
| jrosser | yeah i just tested locally and it totally doesnt work or warn or error | 16:54 |
| jrosser | which is unfortunate | 16:55 |
| jrosser | even wierder it's not just evaluating the string to be 'True' and still not logging | 16:55 |
| jrosser | it just logs regardless | 16:56 |
| noonedeadpunk | oh, ok. I for some reason decided it's not logging regardless.... | 16:56 |
| noonedeadpunk | but I added extra stuff there, that wasn't working either, so maybe that's why | 16:57 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add facility to rewrite source URLs for ansible collections during bootstrap https://review.opendev.org/c/openstack/openstack-ansible/+/870820 | 17:38 |
| mgariepy | for the ovn-meta it seems like it needs restart. | 17:39 |
| mgariepy | i'll have lunch and look at it a bit more. | 17:42 |
| jamesdenton | well, i just did an AIO and the service is pulling in /etc/neutron/plugins/ml2/ml2_conf.ini, which has the SSL bits | 17:47 |
| noonedeadpunk | jrosser: I think we should place openstack_services_opendev_server/openstack_services_github_server elsewhere, like group_vars or smth like that | 17:48 |
| mgariepy | yes but the ovn-meta service is tracebacking. | 17:50 |
| noonedeadpunk | as even in bump script it's quite messy now. As you need to read yaml file, then get variables from it, then render original non-parsed "string", and then parse renderred one as yaml again | 17:50 |
| noonedeadpunk | or well, hardcode | 17:50 |
| mgariepy | i think it's racing because of the ordering. | 17:51 |
| noonedeadpunk | And not sure there's any need in splitting openstack_services_opendev_server from openstack_testing_opendev_server | 17:52 |
| noonedeadpunk | or well, then we're missing github services for gnocchi and consoles | 17:52 |
| jamesdenton | hmm, no trace here. | 17:52 |
| mgariepy | my aio was tracing :/ | 17:53 |
| mgariepy | did you run a vm ? | 17:53 |
| jamesdenton | but no working metadata, either :D | 17:53 |
| jamesdenton | yes | 17:53 |
| jamesdenton | did master, not zed. wasn't thinking about it | 17:54 |
| jrosser | jamesdenton: i had a dig around in my AIO too | 17:55 |
| jrosser | the only thing i could see that looked suspicious was https://paste.opendev.org/show/bi8cKjYlKNS8TyoT4Dq4/ | 17:55 |
| mgariepy | i'm on zed. | 17:55 |
| jrosser | search that ^ for "died" | 17:55 |
| jamesdenton | ahh yes, died. i see that too | 17:56 |
| jamesdenton | so, i see traffic from instance->metadata namespace, but [S] then [R] | 18:02 |
| jamesdenton | i think it may be another permissions issue, one sec | 18:04 |
| jrosser | i was trying to check that i could see the ovn haproxy listening socket in the metadata namespace | 18:07 |
| jrosser | thats not obvious how to do | 18:07 |
| jamesdenton | do you see this in your logs? Proxy listener started. | 18:09 |
| jamesdenton | Jan 17 18:08:49 aio1 haproxy-metadata-proxy-142e1e29-b98b-481c-beba-a73d99ea54b0[242356]: Proxy listener started. | 18:09 |
| jrosser | which log do you check there? | 18:09 |
| jamesdenton | the neutron-ovn-metadata-agent journal | 18:10 |
| mgariepy | https://paste.openstack.org/show/bqbgN34FD1P5NHjjZdRB/ | 18:10 |
| jamesdenton | tcp LISTEN 0 1024 169.254.169.254:80 0.0.0.0:* users:(("haproxy",pi | 18:10 |
| jrosser | no, i don't think i do | 18:11 |
| jrosser | was that just netstat on the host? or in the ns? | 18:11 |
| mgariepy | in the ns probably | 18:11 |
| mgariepy | like in my paste | 18:11 |
| jamesdenton | ok, i think a couple of things: 1. the permissions on /var/lib/neutron/ovn-metadata-proxy/ need to be neutron:neutron, and the hapropxy pid needs to be killed, ovnmeta namespace deleted, then ovn metadata agent restarted | 18:12 |
| jrosser | right, so i don't see the socket open there | 18:12 |
| jamesdenton | https://paste.opendev.org/show/bhshXaNHzOGIfSpp4ASL/ | 18:13 |
| jrosser | theres also a `metadata_proxy` socket which is root:root | 18:13 |
| jamesdenton | yeah, but i think that's because: https://github.com/openstack/openstack-ansible-os_neutron/blob/master/vars/main.yml#L496-L497 | 18:14 |
| jamesdenton | and if we change the dir perms to neutron:neutron maybe that root can be avoided | 18:14 |
| jrosser | metadata agent journal is full of privsep stuff so hopefully none of this needs to be root | 18:15 |
| jrosser | more suspicious stuff https://paste.opendev.org/show/bDy27zbLeBJTPH16zp43/ | 18:17 |
| jrosser | two metadata agents? | 18:18 |
| mgariepy | hmm probably need to go.. :/ | 18:20 |
| mgariepy | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c9b/852243/2/check/openstack-ansible-deploy-aio_lxc-ubuntu-jammy/c9b06a3/logs/host/neutron-metadata-agent.service.journal-12-39-53.log.txt | 18:20 |
| mgariepy | https://github.com/openstack/openstack-ansible-os_neutron/blob/master/vars/main.yml#L288 probably need to filter for ovn. | 18:25 |
| jrosser | also the dhcp agent? | 18:31 |
| jrosser | i also have to head off | 18:31 |
| mgariepy | i was more speaking of the service.. lol not me.. | 18:32 |
| mgariepy | haha | 18:32 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_neutron master: Disable dhcp-agent and metadata-agent for OVN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/870854 | 18:45 |
| mgariepy | jamesdenton, do you patch the perms issue ? | 18:46 |
| mgariepy | or you want me to push it ? | 18:46 |
| prometheanfire | anyone use the ovn driver for octavia in OSA? | 18:50 |
| mgariepy | prometheanfire, https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/868462 | 18:52 |
| jamesdenton | i haven't, but feel free. would be curious to see how CI does | 18:52 |
| prometheanfire | ya, looks like a reasonable patch | 18:54 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_neutron master: Fix user for neutron-ovn-metadata-agent https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/870857 | 18:55 |
| jamesdenton | i do think it will default to neutron if not called out, but this is a good test | 18:56 |
| jamesdenton | thank you | 18:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Allow git servers for openstack services and tempest to be overridden https://review.opendev.org/c/openstack/openstack-ansible/+/869748 | 18:58 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_neutron master: Fix user for neutron-ovn-metadata-agent https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/870857 | 18:58 |
| noonedeadpunk | I have list of comments to that path, more more cosmetical ones rather then functional | 19:09 |
| noonedeadpunk | To octavia/ovn I mean | 19:09 |
| mgariepy | for the ownership of the folder do we need to try to fix it in zed for deployment ? | 19:10 |
| jrosser | mgariepy: i think 870854 only disables for new deployments, rather than existing ones | 19:10 |
| jrosser | becasue of https://github.com/openstack/openstack-ansible-os_neutron/blob/d4cbd2d7adcf4bba95a885c87d2c6e4dc9c1b012/vars/main.yml#L340-L341 | 19:11 |
| noonedeadpunk | just in case I've patched bump script to work with 869748 | 19:11 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon master: Allow to override supported_provider_types https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870801 | 19:11 |
| mgariepy | so we add a release note then ? | 19:12 |
| mgariepy | :/ | 19:12 |
| mgariepy | for zed i mean. | 19:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon stable/zed: Allow to override supported_provider_types https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870779 | 19:14 |
| noonedeadpunk | what is that and why we have it then... https://github.com/openstack/openstack-ansible-os_neutron/blob/d4cbd2d7adcf4bba95a885c87d2c6e4dc9c1b012/vars/main.yml#L488-L493 | 19:17 |
| moha7 | Sorry for bumping the group; I'm resending my question as I was disconnected for hours. | 19:18 |
| moha7 | jamesdenton: Hi james; To give you a feedback on deploying both flat and vlan networks: | 19:19 |
| noonedeadpunk | to be frank I don't have good solution, except quite complicated one | 19:19 |
| moha7 | In the new deployed env, I created two types of external networks: Flat and VLAN; And then I created two routers each in one of these external networks; For the router with a hand in the flat-type external network, everything works well and I can ping the gateway of the router from the outside of OpenStack. But the gateway of the other router that is in the VLAN-type external network is not accessible from outside! | 19:20 |
| moha7 | Note: OpenStack has been installed within a VM hosted on the ProxMox. By tcpdump-ing from the Prox, I see packets of OpenStack instances in the flat-type network, but nothing for the other one | 19:21 |
| noonedeadpunk | As we could set `enabled: false masked:true` based on the ovn/non-ovn but then services will be created, but left disabled | 19:21 |
| noonedeadpunk | While we don't really want to create them for new deployments at all | 19:21 |
| mgariepy | where does the packet for the vlan on is ending ? | 19:21 |
| noonedeadpunk | Or create some "upgrade/transitional" task that will check for services and disable/mask dhcp/metadata ones | 19:22 |
| moha7 | `ip netnes exec <ovn-metadata-somesring> ping upstream gateway` leads to nothing | 19:22 |
| noonedeadpunk | I kind of like idea of such task tbh. But I wonder if for that we would need to gather systemd_service facts wich are huge | 19:22 |
| moha7 | (`ip netns` on the computes) | 19:22 |
| noonedeadpunk | release note... meh, dunno | 19:23 |
| mgariepy | lol | 19:23 |
| noonedeadpunk | I will try to take a look tomorrow how complicated it would be | 19:23 |
| jrosser | moha7: do you know of your proxmox supports vlan tagged traffic on the VM interfaces? | 19:25 |
| mgariepy | if you tcpdump on the interface do you see something in the compute vm ? | 19:25 |
| prometheanfire | iirc a bit over a decade ago it supported passing through trunks | 19:26 |
| mgariepy | ok thanks noonedeadpunk | 19:27 |
| moha7 | mgariepy: subnet: 172.17.238.0/24; upstream router gateway: 172.17.238.254 that is set on the TOR router, the hop after ProxMox; The ip that OpenStack router has takes randomly is 172.17.238.176; the other hand is an internal network connected to instances. I can ping 172.17.238.176 within instances, but not 172.17.238.254; from the outside, 172.17.238.176 is not pingable too. | 19:27 |
| mgariepy | ok | 19:27 |
| mgariepy | in the computes that is hosting the router. if you tcp dump the network interface do you see something passing ? | 19:28 |
| moha7 | let me check | 19:30 |
| moha7 | jrosser: I'm not sure. going to search to see if there' similar issues with Prox | 19:30 |
| moha7 | Should the "Segmentation ID" be the same as VLAN ID in the PrX and TOR router? <-- https://ibb.co/SBgP4pk | 19:40 |
| jrosser | yes it should | 19:46 |
| jrosser | but as far as i can see you will need to make config in proxmox so that whatever your vm connects to is vlan aware | 19:46 |
| jrosser | and permit that vlan | 19:46 |
| moha7 | I think Trunking has a role here, doesn't it? https://docs.openstack.org/neutron/zed/admin/config-trunking.html | 19:48 |
| jrosser | no | 19:48 |
| moha7 | considering te Prox, I'm looking in some forums posts | 19:48 |
| moha7 | the* | 19:49 |
| jrosser | unless you can see arp or other chatter from your physical router interface on the inside of your VM, not much point moving forward | 19:50 |
| jrosser | the neutron trunking docs is about creating trunk interfaces to connect to instances | 19:50 |
| *** dviroel_ is now known as dviroel|afk | 19:57 | |
| jamesdenton | hi moha7 | 19:59 |
| moha7 | Hi | 19:59 |
| moha7 | By the config you gave me, now there's no error in neutron logs and the flat network is working well. | 20:00 |
| moha7 | I had also an issue with the metadata as below: | 20:01 |
| jamesdenton | let me review your notes | 20:01 |
| moha7 | > metadata injection into the instances does not work well. Instances gets IP and can ping their router hands, but the hostname does not set on it! Ctrl+F for 169.254.169.254 in http://ix.io/4ltx to see the error. | 20:02 |
| jamesdenton | yes, we have a patch we are testing right now for that | 20:03 |
| moha7 | but solved as I described in above comments | 20:03 |
| jamesdenton | ok | 20:05 |
| jamesdenton | so, in this lab, the infra and computes are all VMs? | 20:05 |
| moha7 | yes | 20:05 |
| jamesdenton | for ESX, if you were doing trunking, the port group would be VLAN 4095 (which translates to trunk, essentially) | 20:05 |
| jamesdenton | and then you can tag traffic in your VM and if the switchport was configured to allow it, it will | 20:05 |
| moha7 | All VMs with these interfaces: http://ix.io/4lwO | 20:06 |
| jamesdenton | yes, and enp6s21 is used for your OVS bridge, right? | 20:06 |
| jamesdenton | on the proxmox side, that enp6s21 interface would be tied to some network or port definition, i would think? | 20:07 |
| moha7 | The free one, enp6s21, is set for the provider networks (both flat and vlan) in the openstack_user_config.yml | 20:07 |
| jamesdenton | yes, and untagged traffic is working OK and tagged is not, so i suspect something needs to be tweaked in proxmox to support tagged | 20:08 |
| jamesdenton | on the proxmox side, are there bridge configurations? | 20:08 |
| jamesdenton | vmbr0, vmbrX, etc? | 20:09 |
| moha7 | i proxmox, the interface is on separate vlan, definde as vmbr0, wit the vlan ID 3677 | 20:09 |
| moha7 | in* | 20:09 |
| moha7 | Yes, they are all bridge | 20:09 |
| jamesdenton | can you show the config? | 20:10 |
| jrosser | are there any bridge-vlan-aware and bridge-vids set? | 20:10 |
| jamesdenton | ^^ | 20:10 |
| jamesdenton | if you are using eth0.3677 in vmbr0, then flat will work and vlan won't, essentially | 20:11 |
| jamesdenton | but the config will help see what what looks ike | 20:11 |
| moha7 | https://ibb.co/wYY5pTh --> vmbr4 is bunch of subnets all on one vlan, set on those other interfaces, enp6s19-20 | 20:12 |
| jamesdenton | any chance you can show us what the CLI is showing? Is it in /etc/network/interfaces maybe? | 20:12 |
| moha7 | Sure; w8 plz | 20:13 |
| jamesdenton | sure brb | 20:15 |
| moha7 | O_o | 20:33 |
| moha7 | A mistake by my colleague <- http://ix.io/4lwZ who has defined all subnets under vmbr0 the same! | 20:34 |
| moha7 | vmbr4* | 20:34 |
| moha7 | jamesdenton: Then, it's proxmox network misconfiguration :'( | 20:35 |
| jamesdenton | looking | 20:36 |
| jamesdenton | so, i would expect vmbr0 to contain ens3f1 (no tag), and have "bridge-vlan-aware yes" set | 20:37 |
| jamesdenton | then, it ought to pass vlan tags (i think, anyway) | 20:37 |
| jamesdenton | your flat network will continue to work *IF* 3677 is the native vlan on the switchport | 20:38 |
| jamesdenton | but in this case, i expect it is not | 20:38 |
| jrosser | somehow this feels more complicated than it needs to be | 20:39 |
| jrosser | are there really multiple interfaces in the vm hooked down to the same bridge? | 20:40 |
| moha7 | Oops, but vmbr0 has been correct. vmbr0 <--> enp6s21 | 20:40 |
| jamesdenton | yes, but the interface connected to that bridge, ens3f1.3677, is a vlan-tagged interface (by the underlying host) | 20:41 |
| jamesdenton | so Neutron is adding a VLAN tag on top of that | 20:41 |
| moha7 | Umm, seems there's no `bridge-vlan-aware yes` there! | 20:41 |
| jamesdenton | ens3f1.3677 being tagged already is sort of transparent to neutron | 20:41 |
| jamesdenton | i think will need to remove ens3f1.3677 and make it ens3f1, and possible add bridge-vlan-aware | 20:41 |
| jamesdenton | with a flat network, neutron is not tagging. As neutron passes untagged traffic thru enp6s21, it hits vmbr0, where a VLAN 3677 is applied on the way out ens3f1 | 20:42 |
| jamesdenton | if you want neutron tagging to work, neutron needs to be able to tag on enp6s21, and it hit vmbr0->ens3f1 (intact and not tagged again) | 20:43 |
| moha7 | `3677` is native vlan on TOR switch. Indeed, 3677 is created by the network team introduced to us. | 20:43 |
| jrosser | moha7: is that how it has to be? | 20:44 |
| jamesdenton | are you sure it's the native vlan? | 20:44 |
| jamesdenton | either way, if you only have vlan 3677 then you will only be able to support a flat provider network OR a single VLAN network (Segment ID 3677). But if the switchport is configured as an access port in 3677, then you're limited to flat. | 20:45 |
| jrosser | it kind of has to be a trunk with ens3f1.3677 existing | 20:46 |
| moha7 | If I understand 'native' in the right way, 3677 is VLAN on the switch that is controlled by the network guys; Actually we don't have more information how they are handling it; Maybe I need to talk to that team. | 20:46 |
| jamesdenton | ens3f1 looks like a trunk carrying 3677,3674 based on your config anyway | 20:46 |
| moha7 | Yes, it's trunk as I remember some arguments with network administrator. | 20:47 |
| jamesdenton | ok | 20:47 |
| jamesdenton | i speak in Cisco/Arista config, so apologies if 'access' port doesn't make sense | 20:48 |
| jamesdenton | bottom line is, with this configuration in place, you're likely stuck with a flat provider network | 20:48 |
| moha7 | let me review your comments to see how I should edit that file; there's lots of info now here (: | 20:49 |
| jamesdenton | which is fine -- tenant traffic would traverse geneve tunnel. What isn't clear is which interface that rides on | 20:49 |
| jrosser | ^ though if you can change the proxmox host config, you can do vlan too? | 20:49 |
| jrosser | moha7: i think maybe taking some time to think about what your physical deployment might look like, away from the proxmox stuff would help | 20:50 |
| jrosser | if you already won the battle to get a trunk port with vlans from your network team, that is good | 20:51 |
| moha7 | +1 | 20:51 |
| jrosser | things seem to be getting confused with the virtualisation setup, which will all go away with physical hosts | 20:51 |
| jamesdenton | +2 | 20:51 |
| jrosser | then see how you would map what you actually want from the real deployment back to proxmox to make a realistic lab | 20:52 |
| jrosser | you are probably in a much better position to do that now having spent some time with the deployment | 20:52 |
| moha7 | jrosser: but I also should deliver the deployment task as it's taking 3 sprints long; For the physical network topic, I really need to speak with the whole team, plus joining network administrators too. | 20:53 |
| moha7 | Now, I think my task is going to be moved to the done basket. | 20:53 |
| jrosser | imho getting a grip on the network requirements is one of the largest thing to wrap the brain around for an openstack deployment | 20:54 |
| jamesdenton | mgariepy As neutron, neutron-ovn-metadata agent can't open socket: Exception: Could not retrieve schema from unix:/var/run/openvswitch/db.sock | 20:54 |
| mgariepy | ha | 20:59 |
| mgariepy | sad | 20:59 |
| jamesdenton | turns out i mentioned that here, whoops: https://github.com/openstack/openstack-ansible-os_neutron/commit/d4cbd2d7adcf4bba95a885c87d2c6e4dc9c1b012 | 21:00 |
| jrosser | maybe stupid question but do we run ovs-vswitchd with too much priviledge too? | 21:01 |
| jamesdenton | yes? | 21:02 |
| jamesdenton | but we install from package, right? | 21:02 |
| jamesdenton | meaning, i don't think we go out of our way there | 21:03 |
| jrosser | man page suggests there is some priv dropping possible with --user user:group | 21:04 |
| mgariepy | on rhel it runs as openvswitch | 21:04 |
| mgariepy | ubuntu ship as root. | 21:05 |
| jamesdenton | well, maybe we can drop to neutron:neutron and see what breaks? | 21:05 |
| mgariepy | all the ssl stuff will breajk | 21:05 |
| jamesdenton | meh | 21:05 |
| jamesdenton | sounds like a mgariepy problem | 21:06 |
| mgariepy | lol | 21:06 |
| mgariepy | https://github.com/search?q=repo%3Aopenstack/openstack-ansible-os_neutron%20neutron_ovn_system_user_name&type=code | 21:07 |
| mgariepy | what a mess. | 21:07 |
| mgariepy | lol | 21:07 |
| jamesdenton | freedom to choose the tool used to stab yourself in the eye | 21:08 |
| jamesdenton | Linux™ | 21:08 |
| mgariepy | i wonder how a migration from root to neutron or openvswitch. | 21:08 |
| jamesdenton | openvswitch user doesn't exist in Ubuntu, FWIW | 21:09 |
| mgariepy | yeah i know | 21:10 |
| jrosser | changing the cert ownership to root:neutron shouldnt hurt should it? | 21:19 |
| jrosser | thats proabably how it should be anyway, rw by root, r by neutron | 21:19 |
| jamesdenton | https://developers.redhat.com/blog/2018/03/23/non-root-open-vswitch-rhel | 21:27 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon stable/zed: Allow to override supported_provider_types https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/870779 | 23:52 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!