| noonedeadpunk | admin1: so if you're setting VIP as /24 or smth - that might/will change your controller outbound IP to VIP | 10:08 |
|---|---|---|
| noonedeadpunk | As then a new route will be added with the same netmask through on the interface that might have default route | 10:08 |
| jrosser | keepalived docs are quite sparse about this | 10:35 |
| jrosser | we should add a comment in ours about why it’s a cidr because it’s confusing | 10:36 |
| admin1 | noonedeadpunk, that was exactly what i was trying to explain to Mohaa7 as there are certain use cases that you need to use a /subnet in the end | 10:42 |
| noonedeadpunk | yeah, right, but you can easily have issues if having strict firewalling for example | 10:45 |
| admin1 | we can put in the docs like we recommend /32 for most use cases .. you can also use something else , then you already know what you are doing :D | 10:47 |
| noonedeadpunk | yeah, exactly - you should know what you're doing when adding not /32 | 10:55 |
| harunalbayrak | hey all, i am trying to install openstack (stable/yoga) with openstack-ansible but i have just got an error in os-keystone-install.yml. i have pasted the error to (https://paste.openstack.org/show/boWvnxZCZegIr0pfyiGt/). Can you help me | 11:26 |
| admin1 | harunalbayrak, are you also using an ip in the br-mgmt range as your VIP ? | 11:37 |
| admin1 | as your external VIP ? | 11:37 |
| admin1 | paste your variables and config as well | 11:39 |
| admin1 | i was able to get magnum work and deploy k8s .. only thing not working now is that when i try to expose a service using LB, the LB is in pending create state and never completed .. | 11:55 |
| harunalbayrak | my config & variables are here: (https://paste.openstack.org/show/b01LiH4eszDtCiTcq8Qw/) | 12:20 |
| harunalbayrak | i am not using an ip in the br-mgmt range as my external vip address | 12:22 |
| admin1 | you can lxc-exec to the galera container and do mysql ENTER .. to check if mysql is running.. after that, lxc-exec to util and mysql ENTER to check if VIP is working .. if you cannot from the util container, that means something in the VIP is wrong .. either bad ip, or ip not added, or haproxy not making 3306 accessible | 12:39 |
| harunalbayrak | root@infra0-utility-container-08f00222:~# mysql | 12:46 |
| harunalbayrak | ERROR 2013 (HY000): Lost connection to server at 'handshake: reading initial communication packet', system error: 11 | 12:46 |
| harunalbayrak | admin1: i can do mysql in galera container but i cannot do mysql in utility container. the error message is above | 12:46 |
| noonedeadpunk | harunalbayrak: have you checked that haproxy backends are happy? | 13:00 |
| noonedeadpunk | like `echo 'show stat' | nc -U /run/haproxy.stat | grep galera` | 13:00 |
| harunalbayrak | yes, i have checked galera containers it seems down all galera-back (https://paste.openstack.org/show/bIyG3l0vX62fv1LimENB/) but galera containers are running | 13:02 |
| harunalbayrak | this is the haproxy log: https://paste.openstack.org/show/bzXcKlxfxOa4Yz21PeOS/ | 13:02 |
| noonedeadpunk | I think that can be actually one of the side effects of having keepalived address as not /32 | 14:10 |
| *** mathlin is now known as masken | 14:21 | |
| admin1 | noonedeadpunk right .. without a /32, you have to add that ip in the mariadbcheck socket | 14:48 |
| admin1 | as the outgoing ip might change | 14:48 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:00 |
| opendevmeet | Meeting started Tue Feb 21 15:00:40 2023 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:00 |
| noonedeadpunk | #topic rollcall | 15:00 |
| noonedeadpunk | \o/ | 15:00 |
| damiandabrowski | hi! | 15:00 |
| noonedeadpunk | #topic bug triage | 15:06 |
| noonedeadpunk | We have a couple of new bugs here | 15:06 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2007296 | 15:07 |
| noonedeadpunk | Basically idea/proposal here was to create folder under inventory/group_vars for each group we have basically, and move playbooks/defaults/repo_packages there | 15:08 |
| noonedeadpunk | but some naming convention for files should be present, so that bump script could find them and update | 15:09 |
| noonedeadpunk | This will also affect haproxy thing I beleive, as instead group_vars file a directory worth to be used | 15:09 |
| noonedeadpunk | any thoughts on that? | 15:09 |
| damiandabrowski | IMO it's ok, we should leverage group_vars more often. That's also what i did for separated haproxy service config | 15:10 |
| noonedeadpunk | I'd say it would be a bit more tough to find version that's being used, as file location will depend on group | 15:12 |
| noonedeadpunk | But not sure it matters much to be frank | 15:13 |
| noonedeadpunk | Ok, next one | 15:15 |
| noonedeadpunk | #link https://bugs.launchpad.net/openstack-ansible/+bug/2007849 | 15:15 |
| noonedeadpunk | I don't have anything to say here... I wasn't really digging deep into code of our linear implementation | 15:15 |
| noonedeadpunk | But it looks like it's not even required after all? | 15:16 |
| damiandabrowski | i also didn't dig deeper into this, but https://review.opendev.org/c/openstack/openstack-ansible/+/874482 looks good without it | 15:18 |
| noonedeadpunk | It's hard to say also if there's any benefit in execution speed... At the moment it looks like load on nodepool workers is still high, so we have long executions overall | 15:18 |
| damiandabrowski | there was a timeout for ceph scenario but it happens very often nowadays so i believe it's not relevant | 15:18 |
| noonedeadpunk | nah, it's not. | 15:18 |
| NeilHanlon | o/ sorry am late :) | 15:19 |
| noonedeadpunk | I was trying to roughly compare time spent on LXC jobs of this patch and others | 15:19 |
| noonedeadpunk | no worries Neil! | 15:19 |
| damiandabrowski | hi Neil! | 15:20 |
| damiandabrowski | yeah, i'm not sure how to compare performance looking at zuul becuse i believe it may strongly depend on a servers' provider | 15:21 |
| noonedeadpunk | I think worth trying to calculate execution time on some more predictable AIO deployment | 15:21 |
| damiandabrowski | maybe i should do some tests locally and compare results | 15:21 |
| noonedeadpunk | and see if there's any benefit from custom strategy | 15:21 |
| noonedeadpunk | yeah, would be great | 15:21 |
| damiandabrowski | ok, i'll do that | 15:21 |
| noonedeadpunk | #topic office hours | 15:22 |
| noonedeadpunk | So haproxy role was updated after last review. I still haven't reviewed it as last 2 days were quite tough internally | 15:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Accept both HTTP and HTTPS also for external VIP during upgrade https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/864785 | 15:23 |
| damiandabrowski | no worries, there is also neutron and glance PKI/TLS support waiting for reviews | 15:23 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/873654 | 15:23 |
| damiandabrowski | https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/821011 | 15:23 |
| damiandabrowski | currently I'm working on TLS support for nova but it's a bit complicated due to already existing TLS support for consoles | 15:24 |
| noonedeadpunk | That's the topic for review | 15:25 |
| noonedeadpunk | #link https://review.opendev.org/q/topic:separated-haproxy-service-config+status:open | 15:25 |
| noonedeadpunk | #link https://review.opendev.org/q/topic:tls-backend+status:open | 15:26 |
| noonedeadpunk | damiandabrowski: it's not only consoles but also libvirt | 15:26 |
| noonedeadpunk | as we do encrypt live migrations and libvirt makes cert auth | 15:26 |
| damiandabrowski | yeah..theoretically speaking we can share the same certs for API, libvirt and console if all of them reside on the same host, right? | 15:28 |
| noonedeadpunk | well. I think consoles do reside on APIs, but they can use different interface iirc. | 15:29 |
| damiandabrowski | i believe in most cases the do reside on the same host, that's why I'm thinking of sharing the same cert | 15:30 |
| damiandabrowski | they* | 15:30 |
| NeilHanlon | I've made some progress on cloud-init v22.2+ for RHEL 9 and friends.. hoping in the next week or so | 15:31 |
| NeilHanlon | cc jrosser | 15:31 |
| noonedeadpunk | And I think we still haven't backported curl hassle to stable branches | 15:31 |
| noonedeadpunk | Also zuul result is quite confusing here: https://review.opendev.org/c/openstack/openstack-ansible/+/873289 | 15:33 |
| noonedeadpunk | But we still need reviews on dependant patch - maybe it will make zuul happier... | 15:33 |
| noonedeadpunk | Eventually - we need plenty of reviews. Since Andrew is not around, damiandabrowski can you take a round of reviews on current patches? | 15:34 |
| damiandabrowski | yeah, ofc | 15:34 |
| noonedeadpunk | Another thing I was going to discuss. I started looking at quorum queues for rabbit as a replacement of our HA queues that are going to be removed from rabbit 4 | 15:35 |
| noonedeadpunk | And the thing is, that exchange must be removed in order to create quorum queues, since as of today exchange is not durable while it should be for quorum | 15:35 |
| noonedeadpunk | And removing exchange is quite a hussle, as then you need to stop all services at the same time using this exchange and have a user with broad permissions | 15:36 |
| noonedeadpunk | So what I was thinking - maybe we can create a new "clean" vhost, for example without leading `/` (it's sooooo confusing to be frank to have that `/`) and make vhost name conditional depending on usage of quorum queues or not | 15:37 |
| noonedeadpunk | This way it should be possible to switch back and forth as well without stopping service for a really long time | 15:38 |
| noonedeadpunk | But yes, service will be desynced until role is finished anyway, as members will be configured with different vhosts | 15:39 |
| noonedeadpunk | The thing is that easiest way I found to drop exchange is along with vhost.... | 15:40 |
| noonedeadpunk | As I failed to drop exchange using rabbitmqadmin with administrator user... | 15:40 |
| damiandabrowski | i'm not a rabbitmq expert but looks good at first glance. I believe you know what to do :D | 15:41 |
| noonedeadpunk | I hope I do lol | 15:41 |
| noonedeadpunk | Will know soon :D | 15:42 |
| mnaser | you're not a rabbitmq expert if you think you're a rabbitmq expert | 15:42 |
| noonedeadpunk | ^ soooo true | 15:42 |
| mnaser | so you're on the right track damiandabrowski :) | 15:42 |
| damiandabrowski | haha :D | 15:42 |
| noonedeadpunk | So that's kind of it from my side | 15:45 |
| damiandabrowski | btw. don't you think we have quite many intermittent gating failures/timeouts these days? | 15:46 |
| damiandabrowski | for ex. I had to trigger recheck 5 times for https://review.opendev.org/c/openstack/openstack-ansible/+/871189 | 15:46 |
| noonedeadpunk | damiandabrowski: regarding time outs - it's known issue that affects literally every project as of today | 15:47 |
| noonedeadpunk | My thinking is that it's related to high load on providers we're using for CI, or our CI is a noisy neighbour for itself | 15:48 |
| noonedeadpunk | and afaik some quite big provider stopped donating infra for our CI, so load on others has increased | 15:49 |
| damiandabrowski | ahhh okok, makes sense | 15:50 |
| noonedeadpunk | #endmeeting | 16:00 |
| opendevmeet | Meeting ended Tue Feb 21 16:00:46 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-02-21-15.00.html | 16:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-02-21-15.00.txt | 16:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2023/openstack_ansible_meeting.2023-02-21-15.00.log.html | 16:00 |
| jrosser | I think consoles can be also in ironic compute containers | 16:08 |
| jrosser | that’s currently broken today with wss:/ vs ws:/ for ironic consoles | 16:09 |
| jrosser | tbh the setup is odd and it might make more sense to have console services only in nova container to cover both nova and ironic | 16:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Update ironic documentation https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/867547 | 16:12 |
| jrosser | oh also I’m not really around at all this week…… | 16:29 |
| noonedeadpunk | ++ | 16:33 |
| noonedeadpunk | I hope everything is fine and you're just taking time off | 16:34 |
| noonedeadpunk | Uhg, have you seen that? https://github.com/unbit/uwsgi/commit/5838086dd4490b8a55ff58fc0bf0f108caa4e079 | 16:48 |
| mnaser | doesnt really leave much choice for deployment platforms | 17:29 |
| mnaser | mod_wsgi is maintained by one person only | 17:30 |
| *** lowercase_ is now known as lowercase | 18:39 | |
| lowercase | hey guys. | 18:40 |
| lowercase | yoga release - nova-manage api_sync is failing because a table and column are not being removed... because they don't exist. | 18:40 |
| lowercase | (1091, \"Can't DROP COLUMN `vm_state`; check that it exists\")", "[SQL: ALTER TABLE build_requests DROP COLUMN vm_state]", | 18:40 |
| lowercase | I created a fake one. created a new table with a column with garbage information. Well, 2 columns because the alter command doesn't remove the last column. The drop command is used to remove the last column | 18:40 |
| lowercase | and it still didn't remove the vm_state column from build_requests. | 18:40 |
| lowercase | any ideas? | 18:40 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_neutron master: Fix typo in ansible_facts['pkg_mgr'] https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/874687 | 18:55 |
| opendevreview | Merged openstack/openstack-ansible master: Update documentation for LXC/metal and LXB/OVS/OVN https://review.opendev.org/c/openstack/openstack-ansible/+/867577 | 19:02 |
| Mohaa7 | noonedeadpunk: I deployed OSA into a multi-node environment, this time without `network_hosts`, and I was not encountered with the error we discussed yesterday. (FYI) | 19:28 |
| opendevreview | Merged openstack/openstack-ansible master: [doc] Update Ubuntu 22.04 support status https://review.opendev.org/c/openstack/openstack-ansible/+/873091 | 19:47 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Update hatop to latest release, 0.8.2 https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/872262 | 19:53 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Use let's encrypt standalone flag only for http-01 https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/873633 | 20:11 |
| Mohaa7 | I heard about Foreman and am reading about it. Its connections with OpenStack seem outdated. Can it create any value for OpenStack? | 20:15 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/zed: [doc] Update Ubuntu 22.04 support status https://review.opendev.org/c/openstack/openstack-ansible/+/874618 | 20:15 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Install openvswitch repo for RDO scenario https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/873368 | 20:16 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-haproxy_server stable/zed: Use let's encrypt standalone flag only for http-01 https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/874619 | 20:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-haproxy_server stable/yoga: Use let's encrypt standalone flag only for http-01 https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/874620 | 20:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-haproxy_server stable/xena: Use let's encrypt standalone flag only for http-01 https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/874621 | 20:17 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Add a variable to allow extra raw config to be applied to all frontends https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/873745 | 20:18 |
| opendevreview | Merged openstack/ansible-role-systemd_mount master: Change default mode of mount files to 644 https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/873250 | 20:24 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Test multiple keystone containers for os_keystone tests https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/843714 | 20:58 |
| opendevreview | Merged openstack/openstack-ansible master: Allow git servers for openstack services and tempest to be overridden https://review.opendev.org/c/openstack/openstack-ansible/+/869748 | 21:42 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Allow maridbcheck socket to FreeBind https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/873334 | 22:06 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server master: Move selinux fix to haproxy_post_install.yml https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/873703 | 23:12 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!