| hamburgler2 | Is anyone actively using trove? - Struggling with the guest-agent config at the moment and the trove channel is pretty dead, I can dump logs to swift without issue, but looks like there is some kind of bug in the guest agent image venv - /opt/guest-agent/backup/main.py when that calls the create backup function in datastore/service.py, looks like the command being executed is malformed/out of order. | 05:22 |
|---|---|---|
| hamburgler2 | Error in https://paste.openstack.org/show/byFcSFmIpjfED8XKt0ad/ - guest agent conf also in there too. Has anyone come across this before? Database instances work fine also, just backups do not. | 05:22 |
| gokhan__ | is there a way to override upper constraints in osa? I am trying to install neutron fwaas on antelope, but it doesn't create pzmq wheel because it doesn't have python 3.10 whl. I need to override pyzmq version. | 06:59 |
| noonedeadpunk | gokhan__: you can override the url to your own "fork" of it or just random url | 09:05 |
| noonedeadpunk | but not specific occurency | 09:05 |
| noonedeadpunk | hamburgler2: well, I used it a while ago, but wanted to get it running in next month in a some "pet" project | 09:05 |
| noonedeadpunk | hamburgler2: and you do have a swift api endpoint for the region I assume... | 09:22 |
| noonedeadpunk | but yeah, it seems an issue in settings, agree | 09:22 |
| noonedeadpunk | oh my, that's a mess /o\ https://opendev.org/openstack/trove/src/branch/master/trove/guestagent/datastore/service.py#L485-L493 | 09:26 |
| noonedeadpunk | why in the world it opens self with popen... | 09:26 |
| noonedeadpunk | ah, because it runs that inside the docker container..... | 09:27 |
| noonedeadpunk | question is - what is main.py.... | 09:29 |
| noonedeadpunk | and eventually, argument is passed from code regardless https://opendev.org/openstack/trove/src/branch/master/trove/guestagent/datastore/service.py#L467 | 09:30 |
| noonedeadpunk | hamburgler2: ensure you have this commit in your image: https://opendev.org/openstack/trove/commit/e998b6886602575127ebe613e56cee3a5a01c6c6 | 09:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use container setup role from plugins repo https://review.opendev.org/c/openstack/openstack-ansible/+/905004 | 09:40 |
| gokhan__ | noonedeadpunk, thanks. For distribution upgrade, it seems we also need to migrate volumes to other hosts if we deploy cinder volumes to containers. | 09:57 |
| gokhan__ | noonedeadpunk, an other question for installing a service from different version, we need to override git package urls and requirement url? do wee need any settings extra? | 09:59 |
| noonedeadpunk | gokhan__: um, you're using non-ceph? | 09:59 |
| gokhan__ | no I am using ceph | 10:00 |
| noonedeadpunk | so.. why you need to migrate volumes to other hosts? you've disabled active/active setup? | 10:00 |
| gokhan__ | I am using default settings. I am not aware of active/active setup. How can we check this | 10:02 |
| noonedeadpunk | I can't recall _for sure_, but there should be `cluster` setting in cinder.conf, and then for each volume in database there's a field `cluster_name` or smth, that should match | 10:03 |
| noonedeadpunk | also, for "proper" active/active setup you'd need to have some coordination, like zookeeper or etcd | 10:04 |
| noonedeadpunk | as otherwise you can catch some race conditions | 10:04 |
| noonedeadpunk | but we worked without it for a while without anything too obvious too frequent | 10:05 |
| gokhan__ | cluster is ceph in cinder.conf | 10:05 |
| gokhan__ | there is no coordination url setting in cinder | 10:06 |
| gokhan__ | cinder.conf | 10:06 |
| noonedeadpunk | well, so proably active/active can have race conditions, but it shouldn't require to migrate volumes afaik | 10:27 |
| gokhan__ | ok thanks, I need to set cinder coordination group | 10:40 |
| noonedeadpunk | well, if you add zookeeper to inventory - it would be added on it's own | 10:41 |
| noonedeadpunk | but if you have etcd running somewhere - you can point to it manually | 10:41 |
| gokhan__ | noonedeadpunk, we have already zookeeper and we will add it. | 10:50 |
| noonedeadpunk | huh | 10:51 |
| noonedeadpunk | if it's spawned with osa - it should be jsut added by role I guess... | 10:51 |
| jrosser | yeah there should be no need to manually add that | 10:52 |
| noonedeadpunk | ugh, crap, seems rdo has borked ovn installation for rocky.... | 10:59 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/913582?tab=change-view-tab-header-zuul-results-summary | 10:59 |
| gokhan__ | noonedeadpunk, we have our own zookeeper role. if ı set cinder-coordination group to our zookeer group, it can work | 11:02 |
| noonedeadpunk | yeah, sure | 11:03 |
| gokhan__ | noonedeadpunk, I want to also ask for using magnum with bobcat version. we will override magnum github settings and change magnum role to bobcat. do we need another setting ? I will also test cluster api driver. | 11:13 |
| noonedeadpunk | Probably not? except different fedora-coreos image and k8s/etcd versions for templates.... | 11:15 |
| gokhan__ | noonedeadpunk, I am not sure about requirement git url. it seems we need to change also requirement git url in magnum role | 11:17 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Create an openrc for nb/sb clients https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/913582 | 11:17 |
| noonedeadpunk | oh, yes, you probavbly need :) | 11:18 |
| noonedeadpunk | it's `magnum_upper_constraints_url` | 11:18 |
| noonedeadpunk | basically you can set that to `magnum_upper_constraints_url: https://releases.openstack.org/constraints/upper/18ef0785c4d95c0b7a144c2f9b3ca6a97df20e52` | 11:19 |
| gokhan__ | thanks noonedeadpunk :) | 11:22 |
| jrosser | gokhan__: you will have some challenges to run my "complete" magnum cluster_api patches on an older OSA | 11:37 |
| gokhan__ | jrosser, I am planning to test on antelope if it is possible | 11:38 |
| jrosser | if you want OSA to deploy the control plane k8s cluster for you automatically then there are lots of patches needed on Antelope, will be less on Bobcat and hopefully none on C | 11:38 |
| jrosser | the testing is all currently in the context of what will be the C release | 11:38 |
| gokhan__ | jrosser, if I can install management kubernetes cluster on lxc containers, I can handle other challenges. | 11:40 |
| jrosser | you can - my patches do that | 11:41 |
| gokhan__ | jrosser, ok thanks ı will test it next week. | 11:42 |
| jrosser | what i am saying is that you will need to significantly patch a whole bunch of other stuff in OSA to make that managment k8s cluster on the LXC deploy properly | 11:43 |
| gokhan__ | thanks jrosser I got it :) I will look at the patches. | 11:44 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Mar 19 15:00:39 2024 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic roll call | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| NeilHanlon | o/ | 15:00 |
| damiandabrowski | hi! | 15:01 |
| jrosser | o/ hello | 15:01 |
| noonedeadpunk | #topic office hours | 15:02 |
| noonedeadpunk | so one thing that raised today, is that Rocky seems to reliably failing OVN installation | 15:02 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/b48c718f0b794b18b313eb4a513b0cac | 15:03 |
| noonedeadpunk | "nothing provides ovn23.09 needed by rdo-ovn-2:23.09-2.el9s.noarch from rdo-deps" | 15:03 |
| noonedeadpunk | with that it feels like CentOS is passing somehow? | 15:03 |
| jrosser | does this mean we are blocked on everything? | 15:03 |
| noonedeadpunk | I think so, yes | 15:03 |
| NeilHanlon | ack. I see c9s had a build of this. https://cbs.centos.org/koji/packageinfo?packageID=11329 | 15:03 |
| NeilHanlon | I will build, test, and release. | 15:04 |
| NeilHanlon | to get around for now we could pin to rdn-ovn-23.06, if that exists | 15:04 |
| noonedeadpunk | yeah, I think they don't drop things from rdo repos | 15:05 |
| noonedeadpunk | so should be possible | 15:05 |
| noonedeadpunk | would be good to do for rocky specifically and leave centos with latest to catch up if smth goes off | 15:05 |
| NeilHanlon | yep. i will submit a Change to do that | 15:05 |
| noonedeadpunk | ok, awesome | 15:06 |
| noonedeadpunk | it seems that most projects have branched 2024.1 | 15:06 |
| noonedeadpunk | So probably we should switch to tracking it? | 15:06 |
| jrosser | it would be a great time to look through our backlog and draw up a list of whats left to merge | 15:07 |
| noonedeadpunk | yes, totally | 15:08 |
| jrosser | theres the ovn bgp stuff and also a few changes from jimmy | 15:08 |
| noonedeadpunk | most contraversal thing - quorum queues as default | 15:08 |
| noonedeadpunk | also skyline potentially | 15:08 |
| jrosser | yeah certainly | 15:09 |
| noonedeadpunk | (I failed with replacing nginx actually) | 15:09 |
| jrosser | right - i remember looking at it and thinking it was straightforward | 15:09 |
| jrosser | and basically deciding i didnt really understand the nginx setup at all | 15:09 |
| noonedeadpunk | somehow it does smth quite different from what maps are doing | 15:09 |
| noonedeadpunk | and not saying about hardcoded pathes in static failes... | 15:10 |
| noonedeadpunk | so to make it work on same ports as horizon - it can't be really in subdirectory | 15:10 |
| noonedeadpunk | which potentially would just break nice urls | 15:11 |
| noonedeadpunk | there's also https://review.opendev.org/q/topic:%22osa-eom%22 | 15:12 |
| noonedeadpunk | which seems to be failing due to error in zuul? | 15:12 |
| noonedeadpunk | I haven't look into that either :( | 15:13 |
| jrosser | do those branches still exist? | 15:16 |
| noonedeadpunk | I think so? | 15:16 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/stable/xena | 15:16 |
| jrosser | oh yes ok | 15:17 |
| noonedeadpunk | https://review.opendev.org/c/openstack/releases/+/910414 | 15:18 |
| noonedeadpunk | but I'm not sure if branches are in zuul actually.... | 15:18 |
| noonedeadpunk | they should be I assume | 15:18 |
| jrosser | does it not even run those? | 15:19 |
| noonedeadpunk | actually I haven't checked on that | 15:20 |
| noonedeadpunk | let's try some recheck and see | 15:20 |
| noonedeadpunk | so, it appears for a second | 15:21 |
| noonedeadpunk | and that's it | 15:21 |
| noonedeadpunk | so I think some config issue | 15:21 |
| noonedeadpunk | yeah https://zuul.opendev.org/t/openstack/config-errors?project=openstack%2Fopenstack-ansible&skip=0 | 15:22 |
| noonedeadpunk | buster | 15:22 |
| noonedeadpunk | I think you was proposing smth? | 15:22 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/910192 | 15:23 |
| jrosser | yeah i have a bunch https://review.opendev.org/q/topic:%22osa/zuul-errors%22 | 15:23 |
| jrosser | but tbh i really really would like these just to get force merged where possible | 15:24 |
| jrosser | and they are about to become pointless when when the branches are renamed, so it's just /o\ and i kind of wonder why to put effort in | 15:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Remove use of undefined ceph distro job zuul template https://review.opendev.org/c/openstack/openstack-ansible/+/910192 | 15:26 |
| noonedeadpunk | aha, ok, I now recall the discission | 15:27 |
| NeilHanlon | i've got ovn23.09 in the oven for rocky/rebuilds | 15:27 |
| jrosser | ah so there is a ton more to do for debian-buster and centos-7 | 15:27 |
| noonedeadpunk | wow, that's fast | 15:27 |
| NeilHanlon | #link https://cbs.centos.org/koji/taskinfo?taskID=3877769 | 15:27 |
| jrosser | but also there is branches that need deleting like pike and stuff | 15:27 |
| NeilHanlon | trying to figure out where/what requires 'rdo-openvswitch' which is what we need to pin back down | 15:28 |
| noonedeadpunk | Yeah, I think pike and rest was slightly different track | 15:28 |
| noonedeadpunk | NeilHanlon: well, if it's already tested, how long it might take to get released? | 15:28 |
| noonedeadpunk | as if it's like 24h or smth - might be easier to just wait? | 15:29 |
| NeilHanlon | I think like, tomorrow | 15:29 |
| NeilHanlon | yeah | 15:29 |
| NeilHanlon | and, i've created a ticket at work to implement automation, or at least automatic tickets ... for this | 15:29 |
| noonedeadpunk | I think we can live with borked gates until then | 15:29 |
| jrosser | i can do some more work on the zuul errors patches | 15:30 |
| jrosser | for buster and centos-7 | 15:30 |
| noonedeadpunk | centos-7 feels still be present.... | 15:30 |
| noonedeadpunk | I see that now job is in zul | 15:31 |
| NeilHanlon | fyi CentOS 7 is going to be EoL on June 30th | 15:31 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Switch SHAs to EOM https://review.opendev.org/c/openstack/openstack-ansible/+/913414 | 15:31 |
| jrosser | lots of `The nodeset "centos-7" was not found.` | 15:31 |
| noonedeadpunk | well, I was checking https://zuul.opendev.org/t/openstack/config-errors?project=openstack%2Fopenstack-ansible&skip=0 | 15:31 |
| jrosser | oh well.... | 15:32 |
| jrosser | if only it was possible to wildcard on that page | 15:32 |
| jrosser | becasue the errors are all over the repos | 15:32 |
| noonedeadpunk | ah, well | 15:34 |
| jrosser | like https://zuul.opendev.org/t/openstack/config-errors?project=openstack%2Fopenstack-ansible-ops&skip=0 | 15:35 |
| *** f0o_ is now known as f0o | 15:35 | |
| noonedeadpunk | oh, actually, another thing to merge is https://review.opendev.org/q/topic:%22osa/apt_key%22 | 15:38 |
| noonedeadpunk | I clean forgot about that :( | 15:38 |
| noonedeadpunk | so sounds like we have quite some outstanding topics right now | 15:39 |
| jrosser | yeah, perhaps we need an etherpad for things to do before release? | 15:39 |
| jrosser | everyone busy++ right now so might be helpful | 15:40 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Remove use of undefined ceph distro job zuul template https://review.opendev.org/c/openstack/openstack-ansible/+/910255 | 15:42 |
| noonedeadpunk | I'd suggest using PTG etherepad | 15:42 |
| noonedeadpunk | https://etherpad.opendev.org/p/osa-dalmatian-ptg ? | 15:43 |
| jrosser | sure - the current work section is what we want | 15:43 |
| noonedeadpunk | btw, are there any updates for availability for the ptg week? | 15:44 |
| jrosser | i pretty much cant make it | 15:45 |
| noonedeadpunk | ok, I see | 15:45 |
| noonedeadpunk | I was thinking to more-or-less moving what we didn't managed to work on during this cycle | 15:48 |
| noonedeadpunk | So not to scope anything too breaking I assume | 15:48 |
| opendevreview | Merged openstack/ansible-hardening master: reno: Update master for unmaintained/xena https://review.opendev.org/c/openstack/ansible-hardening/+/913136 | 15:55 |
| noonedeadpunk | #endmeeting | 15:59 |
| opendevmeet | Meeting ended Tue Mar 19 15:59:31 2024 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-03-19-15.00.html | 15:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-03-19-15.00.txt | 15:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2024/openstack_ansible_meeting.2024-03-19-15.00.log.html | 15:59 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: reno: Update master for unmaintained/victoria https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/913016 | 16:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Switch SHAs to EOM https://review.opendev.org/c/openstack/openstack-ansible/+/913413 | 16:00 |
| noonedeadpunk | I think we might need to squash these 2 things to have a chance to pass bootstrap.... | 16:01 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/xena: Remove use of undefined ceph distro job zuul template https://review.opendev.org/c/openstack/openstack-ansible/+/910255 | 16:03 |
| jrosser | grrr .vscode files :( | 16:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Switch SHAs to EOM https://review.opendev.org/c/openstack/openstack-ansible/+/913413 | 16:10 |
| spatel | noonedeadpunk sorry I won't able to join meeting | 16:37 |
| spatel | dealing with production issue :( | 16:38 |
| spatel | I have a question, if someone delete cinder volume then where to find the logs of that.. I did search but didn't find anywhere in logs of cinder | 16:38 |
| noonedeadpunk | spatel: no worries, it just ended and was quite productive I guess. We will catch up again about the progress in 2 weeks (on APril 2) | 16:42 |
| spatel | +1 please let me know if you need to test or something. soon I am going to deploy freezer on one of my lab to test all the function and will start reporting bug. | 17:03 |
| ThiagoCMC | So, Ceph Ansible `stable-8.0` is working with Ubuntu 22.04 + UCA Bobcat to deploy Ceph Reef! | 17:06 |
| ThiagoCMC | =P | 17:07 |
| gebz | A simple null delayed me 3 days :'D | 17:08 |
| gebz | It was true whoever said that null was a billion dollar mistake | 17:08 |
| gebz | @noonedeadpunk thank you man, it run smooth :D | 17:08 |
| gebz | Now for the moment of truth.. Fingers crossed | 17:09 |
| noonedeadpunk | gebz: oh, sweet it worked out | 17:20 |
| noonedeadpunk | spatel_: yeah, actually I'm pretty sure that current master is broken badly there due to sqlalchemy | 17:20 |
| noonedeadpunk | but I had plans for my pet thing to try out freezer and some effort into it like in couple of month | 17:21 |
| NeilHanlon | noonedeadpunk: ovn23.09 is out at https://mirror.stream.centos.org/SIGs/9/nfv/x86_64/openvswitch-common/Packages/o/ -- I think that means CI should start working soon.. | 17:47 |
| noonedeadpunk | it was waaaaaay faster then 24 h :D | 17:47 |
| NeilHanlon | :D | 17:48 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: [doc] Expand documentation on OVN useful commands https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/913588 | 18:17 |
| *** jamesdenton_ is now known as jamesdenton | 18:41 | |
| ThiagoCMC | jrosser! I'm using Ceph Ansible `stable-8.0` branch to deploy native Ceph Reef on Ubuntu 24.04! No Docker/Podman required! | 18:48 |
| ThiagoCMC | Party time! | 18:48 |
| hamburgler2 | noonedeadpunk: that commit is present, Swift api endpoint for region is up and functional, can dump trove logs to swift container and visible on dashboard through horizon or via accessing swift cli, just seems to be when the guest image goes to create the backup using backup container image, the command when constructed, the positional arguments are out of order in some way | 19:26 |
| noonedeadpunk | hm, that exact patch should have solved that.... | 19:26 |
| noonedeadpunk | or it broke in a different way :D | 19:27 |
| noonedeadpunk | but imo, that line should be enough to get arg respected: https://opendev.org/openstack/trove/src/commit/e998b6886602575127ebe613e56cee3a5a01c6c6/backup/main.py#L55 | 19:28 |
| noonedeadpunk | it could be some other place though | 19:30 |
| noonedeadpunk | but I don't see anything obvious... | 19:31 |
| f0o | just as I was gonna drop out for the day I noticed that with this OVN setup my VMs with vxlan that OVS/OVN leaks all connected subnets (br-mgmt for instance) into the OVS | 19:41 |
| f0o | I'm hoping this is just some silly VRF fuckup on my end | 19:42 |
| f0o | VRF fuckup on my end | 19:52 |
| f0o | man that got me worried really fast | 19:53 |
| f0o | I know OVS isn't VRF aware which makes it a bit tedious to work with | 19:53 |
| jrosser | f0o: do you have some unusual setup there? | 19:55 |
| f0o | possibly | 20:01 |
| f0o | Openstack -> OVS -> TopOfRackRouter -> BGP -> Internet | 20:01 |
| f0o | usually I just segregate things with VRFs and call it a day | 20:02 |
| jrosser | you mean on the TOR? | 20:02 |
| jrosser | so not an issue with OSA getting mgmt network mixed up into OVS? | 20:02 |
| f0o | but here for some reason the router is actually routing 10.0.3.1 (which is lxcbr0) and also 10.20.0.0/22 which is br-mgmt and a bunch of other things | 20:02 |
| f0o | well it's hard to say because I dont have a vanilla negative test here | 20:03 |
| f0o | I think OVS will be just as happy to route br-mgmt on your setup as it is on mine | 20:03 |
| f0o | because I can nuke the VRF table leakage to prevent internet access but that does not prevent access to 10.0.3.1 nor br-mgmt range | 20:04 |
| jrosser | from where? I’m unsure if you mean from tenant vm or elsewhere | 20:04 |
| f0o | inside a tenant-vm with geneve network | 20:04 |
| jrosser | really? | 20:05 |
| f0o | that vm can happily connect to br-mgmt, br-vxlan, lxcbr0 | 20:05 |
| f0o | gets routed from the hypervisor through geneve_sys_6081 interface on the Gateway node and then out lxcbr0/br-mgmt/... | 20:05 |
| jrosser | this is reproducible in an all-in-one? | 20:06 |
| jrosser | noonedeadpunk: ^^ | 20:06 |
| f0o | I will need to do that tomorrow when my brain is fresh | 20:06 |
| f0o | I just noticed it because this vm happily resolved dns against 10.0.3.1 which made me very suspicious | 20:06 |
| f0o | and I was just about to sign off and call this done... | 20:06 |
| f0o | from what I can say is that OVS does everything right - packet goes out of the libvrt tap interface, gets pushed into the vxlan/geneve overlay and delivered to the openstack-router on the gateway node. | 20:07 |
| f0o | Now on the gateway node that packet gets full access too all connected networks of the gateway node it seems | 20:08 |
| f0o | and because OVS is not VRF aware, you cant limit it by just stuffing it into a VRF. | 20:08 |
| f0o | So you likely have to resort to iptables to forbid forwarding based on interfaces | 20:09 |
| f0o | which seems more like a bandaid than a stable fix | 20:09 |
| f0o | but packet-logic spoken it makes sense why it is how it is | 20:09 |
| f0o | in linuxbridge you didnt had this issue because you flushed the packet into an interface but OVS doesnt do that, the flows governed by northd do all the routing (S/DNAT) and then it's handed over to the kernel | 20:10 |
| f0o | and the kernel knows where things are | 20:10 |
| f0o | sorry for wall of text, just had to write it down before the fever hits me and I get brainmsuh again | 20:11 |
| jrosser | no it’s fine, sounds like something that shouldn’t be able to happen really | 20:14 |
| f0o | I will set up some wiretaps tomorrow and get some packet tracing from the hypervisors to the routers and see where what when touches it to get more observations into this | 20:24 |
| f0o | OVS is hot and new, so I have no real clue how it works. maybe there's just one silly setting that's forgotten or some other gotcha | 20:25 |
| f0o | I just did a low-effort fix by adding a blackhole route for 10.0.0.0/8 on the VRF that my br-ext is attached to. That did solve it because it matches early | 20:30 |
| f0o | the routing looks like: br-ext[vlan123 slaved to vrf OVS] -> default next-hop-vrf IBGP -> IBGP has full bgp tables with next-hop-vrf/s onto the next router/s. | 20:32 |
| f0o | so any packet that's exiting vlan123 would be pushed to IBGP with next-hop-self where it would find it's destination directly or return with DestinationUnknown error. 10.0.0.0/8 is in the MGMT vrf, far away and not leaked into anything | 20:33 |
| f0o | so vlan123 should not be pushing packets into that vrf whatsoever. I tripple checked the routing tables just now for the ranges of br-mgmt etc, they only exist in MGMT. | 20:34 |
| f0o | I'm dropping out now, will do more tests tomorrow | 20:35 |
| int33h | Hi anyone had any issues with ssh controlmaster multiplexing issues when trying to do a openstack ansible deployment | 22:05 |
| int33h | i havnt found anyway to disable multiplexing of the ssh sessiosn but i cant seem to find any way | 22:06 |
| jrosser | int33h: best to share whatever error you get in a paste, if you can | 22:07 |
| jrosser | but we don't get really much/any trouble with ssh stuff at all | 22:08 |
| int33h | jrosser: right now im not getting any error it just stands still, https://pastebin.com/CRn5cWbR | 22:10 |
| int33h | did a pastebin where it is atm | 22:10 |
| int33h | I tried the ssh command manualy and it works if i remove the controlpath | 22:10 |
| int33h | Its one of these weeks where everything keep not working in the most unexpected ways :P | 22:11 |
| int33h | last thing i expected was hanging ssh sessions :P | 22:14 |
| jrosser | maybe turning up the sshd logging on the target and look for issues there | 22:18 |
| jrosser | everything i read suggests this is somehow related to the ssh setup on the target | 22:18 |
| int33h | hmmm strange its not even trying to connect | 22:23 |
| int33h | yea with controlpath its not even trying to connect , as soon as i try without the controlpath it works fine with controlpath still there, there isnt even a line in the sshd with debug3 | 22:25 |
| int33h | yes increasing loglevel on ssh, it gets stuck on debug1: auto-mux: Trying existing master | 22:27 |
| int33h | Is it possible to disable all multiplexing | 22:37 |
| int33h | I found how to disable it , seems like i get a broken pipe now , maybe the more proper error | 22:54 |
| int33h | Think i found the real issue now, since i got a proper error mesage now. I have a pair of pfsense firewalls infront of the managment interface, i changed the config to relect to a diffrent interface not goign thtough the firewall | 23:06 |
| int33h | Now its chugging away | 23:06 |
| int33h | Figuring out why pfsense kills the ssh sessions il do tomorrow, time to sleep | 23:07 |
| int33h | jrosser: thanks for the idea about sshd logging , it took me in the right direction | 23:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!