| *** dviroel|rover|dinner is now known as dviroel|rover | 00:33 | |
| *** dviroel|rover is now known as dviroel|out | 00:36 | |
| *** ysandeep is now known as ysandeep|afk | 06:42 | |
| -opendevstatus- NOTICE: review.opendev.org (Gerrit) is currently down, we are working to restore service as soon as possible | 07:30 | |
| *** ysandeep|afk is now known as ysandeep | 07:51 | |
| derekokeeffe85 | Morning all, ran into an issue you guys helped me with already but for the life of me I can't remember what I did to fix it. https://paste.openstack.org/show/bEzHCNsWXPjByvlYIKsf/ I was sure it was log into the container and umount /var/www/repo but that's not working for me now | 09:26 |
|---|---|---|
| noonedeadpunk | I'm not sure about unmount. It should be mounted as otherwise you will get other repo containers broken | 09:33 |
| jrosser | derekokeeffe85: you are getting connection refused from the loadbalancer? | 09:48 |
| jrosser | derekokeeffe85: if you don't even get a 5xx code back then that suggest some fundamental networking problem | 09:49 |
| jrosser | try the same URL with curl to double check | 09:49 |
| derekokeeffe85 | Maybe the umount was for a different issue so noonedeadpunk. Ok a networking error jrosser, the curl doesn't work either | 10:13 |
| jrosser | the error code is important | 10:14 |
| jrosser | connection refused from haproxy means one thing | 10:14 |
| jrosser | connection but 5xx code means something else | 10:14 |
| *** dviroel|out is now known as dviroel|rover | 11:28 | |
| *** ysandeep is now known as ysandeep|afk | 11:36 | |
| derekokeeffe85 | jrosser thanks. Had the wrong config on an interface :( | 12:12 |
| *** ysandeep|afk is now known as ysandeep | 12:15 | |
| *** ysandeep is now known as ysandeep|dinner | 14:22 | |
| -opendevstatus- NOTICE: review.opendev.org (Gerrit) is back online | 14:25 | |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Nov 1 15:00:18 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | o/ | 15:00 |
| jrosser | hello | 15:00 |
| mgariepy | hello | 15:00 |
| noonedeadpunk | yay, gerrit finally back -jsut in time :D | 15:01 |
| noonedeadpunk | with regards time change I guess it makes sense to raise question if current meeting time is ok for you? | 15:02 |
| jrosser | this is ok for me | 15:02 |
| noonedeadpunk | Or should I make a poll and pick up new one ? | 15:02 |
| noonedeadpunk | ok, if everyone is fine with 15UTC - let's leave it as is | 15:04 |
| mgariepy | works for me | 15:04 |
| noonedeadpunk | If not - let me know and we will arrange a poll | 15:04 |
| noonedeadpunk | #topic office hours | 15:05 |
| noonedeadpunk | So I have good progress on zookeeper role | 15:05 |
| noonedeadpunk | I still feel quite confused about how to have 2 repos for same purpose under opendev umbrella... | 15:05 |
| noonedeadpunk | I tired to fit in what's already there and it's quite far from beeing usable by us I'd say | 15:06 |
| jrosser | there is more than one deployment project so maybe we don't worry too much | 15:06 |
| noonedeadpunk | yeah.... | 15:07 |
| noonedeadpunk | I planned to push patches for repo creation today after the meeting | 15:07 |
| noonedeadpunk | And also create skyline repo with that | 15:07 |
| noonedeadpunk | We have quite a few patches for review | 15:08 |
| *** dviroel|rover is now known as dviroel|rover|lunch | 15:09 | |
| * noonedeadpunk checking PTG etherpad | 15:10 | |
| mgariepy | i'm still working on ovn ssl stuff, i'm getting closer (i think) | 15:11 |
| noonedeadpunk | I think we have quite good progress for things we want for Z | 15:12 |
| jrosser | i am doing another ironic deployment so hopefully we find any last bits there too | 15:13 |
| noonedeadpunk | damiandabrowski is not around today and I'm not sure about conclusions what we should do with glance. As we've landed (or about to land) changes that will disable show_multiple_locations by default | 15:13 |
| noonedeadpunk | and if we still need to run 2 api servers just for show_directl_url as it was considered as lower risk from what I got | 15:14 |
| jrosser | yes i think that was the result | 15:16 |
| jrosser | just need a tidy way to do it | 15:16 |
| noonedeadpunk | Yeah, will try to check that once done with zookeeper | 15:19 |
| noonedeadpunk | oh, btw, one frustrating thing has happened during PTG. Regarding u-c and our way of filtering | 15:21 |
| noonedeadpunk | So basically I was told that u-c as of today should be never used for stable branches as basically we deploy outdated software - requirements team does not manage security issues in packages that are in u-c | 15:22 |
| jrosser | thats always been the case i think | 15:22 |
| jamesdenton | hey, sorry i'm latye | 15:23 |
| noonedeadpunk | And when I asked how then openstack should be installed, as it will hardly pass without u-c they said that system packages is the only way to do that | 15:23 |
| noonedeadpunk | because we all trust distributions to maintain python bindings... | 15:23 |
| noonedeadpunk | On top - filtering projects (like neutron or ceilometer) should be considered as a bug and fixed from our side | 15:24 |
| jrosser | but - you cant? | 15:24 |
| jrosser | like pip blows up? | 15:24 |
| noonedeadpunk | While having these project in u-c is okey. | 15:24 |
| noonedeadpunk | So what they proposed - install just using u-c, and then update package on top of installed one | 15:25 |
| jrosser | i thought neutron is only there becasue work on neutron-lib is not complete | 15:25 |
| jrosser | thats the actual bug | 15:25 |
| noonedeadpunk | ie - do installation 2 times separately, first install all requirements with constraints and then install whatever needed | 15:25 |
| noonedeadpunk | There was quite harsh argue for this topic | 15:25 |
| noonedeadpunk | And because they didn't want to change anything it all ended up in - u-c for CI only, never use on prod, use system packaged | 15:26 |
| noonedeadpunk | which got me very frustrated and confused | 15:27 |
| jrosser | i can imagine | 15:27 |
| jrosser | anyone from kolla perspective there? | 15:27 |
| noonedeadpunk | by that time I guess not. But infra folks were on releases team side | 15:27 |
| jrosser | no comment | 15:27 |
| noonedeadpunk | Well, when things a bit calmed down I got suggestion to install requirements+u-c and then package from source independently | 15:29 |
| noonedeadpunk | while this can work - there's one tricky thing (at least) - if package is older then from u-c and we're building wheels - it can still be troublesome | 15:30 |
| * noonedeadpunk got frustrated again after raising this topic.... | 15:31 | |
| jrosser | perhaps it's worth talking to mgoddard or someone from kolla as the problem i guess is identical for them? | 15:32 |
| noonedeadpunk | They install things jsut from pip though | 15:33 |
| noonedeadpunk | We quite recently got issue when running cinder-api deployed at beginning on Xena, but we added a bunch of cinder-volume from top of stable/xena (due to some bugfixes in code) and they were ignoring detach commands - nova was detaching, but cinder-volumes just ignored that. Until we had to upgrade cinder-api to same version | 15:34 |
| noonedeadpunk | So I wonder if I will be suggested to enable unattended-upgrades as well to cover that issue with system packages... | 15:35 |
| noonedeadpunk | anyway | 15:35 |
| noonedeadpunk | well, I've checked devstack and it's also filtering the same way we do. But argument was - devstack is CI only, so we can do nasty things there while osa/kolla should not do that | 15:36 |
| jamesdenton | wouldn't you want prod to mirror what you're doing in testing? <insert kermit meme> | 15:37 |
| noonedeadpunk | yeah, sure, you're right. | 15:37 |
| noonedeadpunk | btw, we can drop that filtering for tempest, as tempest in not in u-c for a while now | 15:39 |
| noonedeadpunk | eventually, neutron also is not for Zed as of today. But not sure if it's intended or not | 15:39 |
| *** ysandeep|dinner is now known as ysandeep | 15:40 | |
| noonedeadpunk | so if it's only ceilometer that left.... ugh | 15:40 |
| noonedeadpunk | fix bug by dropping telemetry roles ? :D | 15:40 |
| noonedeadpunk | anyway | 15:40 |
| jamesdenton | :D | 15:41 |
| noonedeadpunk | I don't have anything else on agenda | 15:41 |
| jamesdenton | i hope to revisit the default ml2 plugin drama today or tomorrow | 15:41 |
| noonedeadpunk | aha, yes, good | 15:42 |
| mgariepy | i hope being able to fix the ovn ssl stuff today :/ | 15:42 |
| noonedeadpunk | My personal opinion is that we should provide some default.... | 15:42 |
| noonedeadpunk | (and we still do in neutron role) | 15:42 |
| jamesdenton | the haproxy templates need to be adjusted to account for neutron_plugin_type not being global, which in one case is for calico but i thought about adding an 'is defined' check | 15:42 |
| jamesdenton | yes, a default is fine. just trying to avoid stepping on the toes of lxb default deployments | 15:43 |
| noonedeadpunk | yeah, let's add this check and will clean up later with calico itself | 15:43 |
| jamesdenton | that works | 15:43 |
| noonedeadpunk | I tried to take a look on calico yestarday actually and get done quite fast | 15:43 |
| noonedeadpunk | I think current issue at very least is that etcdgw driver can't work with modern etcd properly | 15:44 |
| noonedeadpunk | as it tries to check URLs for health that are not valid with latest stable etcd | 15:44 |
| jamesdenton | I found this, not sure how relevant it is today (from 2020): https://github.com/projectcalico/calico/issues/3015#issuecomment-573094997 | 15:45 |
| noonedeadpunk | And I didn't want to dig much deeper there | 15:45 |
| jamesdenton | " In particular, we used to advise core_plugin = ml2 and configuring Calico as an ML2 mechanism driver, but now we prefer core_plugin = calico" | 15:45 |
| noonedeadpunk | So `calico-dhcp-agent.service` fails now with not able to connect to etcd | 15:46 |
| noonedeadpunk | Not sure if it's related to ml2 overall... | 15:46 |
| noonedeadpunk | And we don't install neutron-metadata-agent... But yes, we still have core_plugin=ml2 for this scenario | 15:48 |
| noonedeadpunk | But I'm not sure how it will solve that calico services rely on https://opendev.org/openstack/etcd3gw/commits/branch/master/etcd3gw/client.py that seems not to work with modern etcd | 15:49 |
| noonedeadpunk | They even don't override DEFAULT_API_PATH.. Also I kind of hardly understand how it's supposed to be overriden when its set as constant... | 15:50 |
| noonedeadpunk | anyway | 15:50 |
| noonedeadpunk | I gave up | 15:50 |
| jamesdenton | well, to be fair, even the upstream docs stop at 18.04 | 15:51 |
| jamesdenton | so who knows how well it's been maintained | 15:51 |
| jamesdenton | i don't know how much it will be missed if it were deprecated and removed | 15:52 |
| noonedeadpunk | yeah, no idea. I know logan- was using it, but I haven't seen him for a while now... | 15:53 |
| noonedeadpunk | or heard | 15:53 |
| noonedeadpunk | so no idea | 15:54 |
| jrosser | i think that we should remove it | 15:57 |
| jrosser | there is a big mess with calico metadata service and internal SSL as well that there is no good solution to | 15:58 |
| jrosser | it uses an iptables rule to forward traffic to the metadata service, rather than haproxy like neutron would normally do | 15:58 |
| jamesdenton | the iptables method was the old way, even for neutron | 15:59 |
| jrosser | so there is no opportunity to resolve (metadata http request) -> (internal endpoint being https) | 15:59 |
| noonedeadpunk | #endmeeting | 16:02 |
| opendevmeet | Meeting ended Tue Nov 1 16:02:00 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-11-01-15.00.html | 16:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-11-01-15.00.txt | 16:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-11-01-15.00.log.html | 16:02 |
| noonedeadpunk | yeah, I agree with that. Except someone really wants to step in and maintain that | 16:02 |
| jamesdenton | These diagrams will need to be updated/expanded to support the LXB, OVS/OVN deployments: https://docs.openstack.org/openstack-ansible/latest/reference/architecture/container-networking.html. but i can try to highlight OVN as the prferred way forward | 16:03 |
| noonedeadpunk | we should communicate though this decision first | 16:03 |
| noonedeadpunk | yeah.... I think we actually should also document/draw on how to use ovs bridges for lxc connectivity... | 16:03 |
| jamesdenton | uhhhh yeah, i've been avoiding that | 16:04 |
| mgariepy | lol | 16:04 |
| noonedeadpunk | Doesn't that much easier for things like octavia? | 16:04 |
| noonedeadpunk | But I think I will take that part later on | 16:04 |
| noonedeadpunk | Hopefully we will have some deployment soon where will try this out | 16:05 |
| jamesdenton | i don't know, really. Our Octavia lbaas-mgmt network is routed, so it's not really a problem | 16:05 |
| jamesdenton | i noticed for OSP, that their lbaas-mgmt network is vxlan and there's some cute plumbing done to make that accessible from the control plane | 16:05 |
| noonedeadpunk | is it also routed between computes? octavia uses unicast for vrrp? | 16:05 |
| johnsom | Yes, unicast only | 16:06 |
| jamesdenton | routed, as in a vlan provider network that hangs off a firewall/router somewhee | 16:06 |
| noonedeadpunk | jamesdenton: and you have single or active/passive setup? | 16:09 |
| jamesdenton | the amphora? | 16:09 |
| noonedeadpunk | yeah | 16:09 |
| jamesdenton | active/passive i believe | 16:09 |
| noonedeadpunk | johnsom: yeah, unicast indeed makes more sense as there're 2 amphoras tops | 16:09 |
| noonedeadpunk | hm... why I thought that routed setup won't work then... | 16:10 |
| johnsom | Yeah, it was a design point to allow the lb-mgmt-net to be fully routed. | 16:10 |
| noonedeadpunk | maybe I thought it's multicast... | 16:10 |
| johnsom | Multicast was also to unreliable in some of the ML2's when we started. | 16:10 |
| jamesdenton | i think for OSA CI it was problematic, which resulted in some convoluted bridging and veths | 16:11 |
| johnsom | Yeah, that old design in OSA was... interesting. | 16:11 |
| jamesdenton | that was a case of test -> prod | 16:11 |
| *** dviroel|rover|lunch is now known as dviroel|rover | 16:12 | |
| noonedeadpunk | ah, you also run things on bare metal... I wonder what can go wrong with containers and fully routed setup... | 16:12 |
| noonedeadpunk | As if switch our keepalived to unicast as well - everything should jsut work I guess.... | 16:12 |
| johnsom | It simplifies your keepalived instance IDs too. grin | 16:13 |
| jamesdenton | yes, we do baremetal now, but i'm not sure it matters, since the octavia worker has egress networking (even if it's SNAT'd), the routing just needs to be there | 16:14 |
| jamesdenton | *octavia worker in LXC | 16:14 |
| noonedeadpunk | johnsom: I just not sure how unicast is good when having 3 or 5 keepalived servers | 16:15 |
| noonedeadpunk | as then it likely it makes less sense | 16:16 |
| johnsom | It's one packet, per instance, per heartbeat interval. I guess it just depends on how fast you want convergence. We don't do sub-second failover configs usually. | 16:18 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_ironic master: Allow ironic bmaas network gateway and dns servers to be undefined https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/863160 | 16:42 |
| *** ysandeep is now known as ysandeep|out | 17:07 | |
| mgariepy | jamesdenton, do you think it would be better to deploy northd before the compute nodes ? | 17:42 |
| noonedeadpunk | jrosser: do you have zuul internally? if not - can you kindly drop zuul.d directory from https://github.com/jrosser/openstack-ansible-os_skyline/blob/master/zuul.d/ as it will fail to get cloned - https://zuul.opendev.org/t/openstack/build/e2085d34f3dd447a94d58855fb36609e | 17:47 |
| noonedeadpunk | or well, I can push PR I guess as well | 17:49 |
| jrosser | ah yes I probably just search/replaced one of the existing roles to make that | 17:52 |
| jrosser | make a PR and I can merge it when I get home | 17:52 |
| jrosser | noonedeadpunk: it’s merged | 18:21 |
| jamesdenton | mgariepy i didn't really consider the order. are you seeing issues? | 18:41 |
| mgariepy | if you have a lot of computes it might take quite a while for the controller to get up. | 18:48 |
| mgariepy | and we usualy setup the control plane before the compute service | 18:49 |
| jamesdenton | any approach in mind? | 18:50 |
| mgariepy | not 100% sure | 18:52 |
| mgariepy | not 100% sure it would cause issue tbh | 18:53 |
| mgariepy | hmm `ansible neutron_ovn_northd -m command -a "ovs-appctl -t /var/run/ovn/ovnnb_db.ctl cluster/status OVN_Northbound"` | 19:07 |
| mgariepy | jamesdenton, what does it give you ? | 19:07 |
| jamesdenton | let's see | 19:09 |
| mgariepy | the bind is on port 6643 ? | 19:10 |
| jamesdenton | https://paste.opendev.org/show/baK0RyP1TjrrfqQyuEbU/ | 19:10 |
| jamesdenton | yes, 6643 | 19:10 |
| mgariepy | something is listening on 6641 also ? | 19:11 |
| jamesdenton | ovsdb-server? | 19:11 |
| mgariepy | what are the args ? | 19:13 |
| mgariepy | https://paste.openstack.org/show/b2zWK0WMxIrzdwqyd1cF/ | 19:13 |
| mgariepy | mine doesn't listen to the port anymore :( lol | 19:13 |
| mgariepy | what a mess | 19:13 |
| jamesdenton | you borked it good | 19:14 |
| jamesdenton | https://paste.opendev.org/show/b95vmTwjtmVyKCxWX3IC/ | 19:14 |
| mgariepy | you know when a app has 200 options with defaults that are not compatible. | 19:15 |
| mgariepy | it does create some edgecase lol | 19:15 |
| jamesdenton | :D | 19:15 |
| jamesdenton | i don't trust my /etc/default/ovn-central looks correct | 19:15 |
| mgariepy | ` /usr/share/ovn/scripts/ovn-ctl --help` | 19:16 |
| mgariepy | i wonder if i need to set `--db-nb-sync-from-proto` | 19:17 |
| mgariepy | https://paste.openstack.org/show/bRb0pyBP48HTYttPNkcB/ | 19:17 |
| mgariepy | ha. nop ovn-sbctl set-connection pssl:6642 | 19:26 |
| *** dviroel|rover is now known as dviroel|rover|bbl | 21:32 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!