| opendevreview | Merged openstack/openstack-ansible master: Revert "Skip haproxy with setup-infrastructure for upgrades" https://review.opendev.org/c/openstack/openstack-ansible/+/880091 | 16:51 |
|---|---|---|
| psymin | I assume the most common target OS is Ubuntu 22 server? If not, which one is the most common? | 18:17 |
| jrosser | psymin: we test on ubuntu, debian, rocky linux | 18:41 |
| jrosser | psymin: you can look at the CI jobs we run here http://bit.ly/osa-review-board-v4_1 | 18:42 |
| jrosser | pick any of the patches to the openstack-ansible repo like this one https://review.opendev.org/c/openstack/openstack-ansible/+/881030 | 18:42 |
| jrosser | actually this is a better example https://review.opendev.org/c/openstack/openstack-ansible/+/880091 | 18:43 |
| jrosser | then click "zuul summary" and you can look at the logs and collected files for any of those jobs on various operating systems | 18:43 |
| jrosser | it is probably true that "most people" are deploying on ubuntu | 18:44 |
| psymin | thanks :) | 18:46 |
| jrosser | psymin: did you fix your error "No package centos-release-gluster9 available"? | 19:20 |
| jrosser | my guess would be you were doing a deployment that expected rocky9 (master branch maybe?) on a rocky8 system | 19:21 |
| psymin | jrosser, I did not. Targets were rocky 9 servers. | 19:21 |
| psymin | I'm installing ubuntu 22.04.2 server alongside these on separate disks to test a similar deployment on a more common OS | 19:21 |
| jrosser | well if it does not behave as documented thats a bug, and we should investigate | 19:22 |
| jrosser | the testing is done on "cloud images" exactly like you'd boot in a cloud provider rather than installations with the OS installer | 19:22 |
| jrosser | so small differences can creep in | 19:22 |
| jrosser | the glusterfs part of the repos for RH-like OS should have been enabled with this https://github.com/openstack/openstack-ansible-plugins/blob/master/roles/glusterfs/vars/redhat.yml#L17 | 19:24 |
| psymin | I'm open to helping. If the ubuntu 22.04.2 targets work with less friction I might use it instead of rocky. But I could help after that by deploying openstack-ansible to some VMs on an ubuntu 22 openstack ansible deployment? | 19:24 |
| jrosser | well, i mean more that we test the glusterfs stuff today on rocky9, and it has to pass tests to merge every single patch we make | 19:25 |
| jrosser | so if there is something different to the test jobs "in the wild" then thats worth knowing about | 19:26 |
| psymin | I have a log of the output somewhere. The target servers are all booted ubuntu 22 at the moment, so I couldn't test any changes for a bit. | 19:26 |
| psymin | Which info would be useful at the moment? | 19:27 |
| jrosser | seeing the log when it fails is useful, at least the failing task and the few before, perhaps putting them at paste.opendev.org and linking them here | 19:28 |
| jrosser | and certainly during EU working hours you'll probablt get some help to debug it here too | 19:29 |
| psymin | https://paste.opendev.org/show/bnOVBZ5O7zbCyZq5tl1Y/ | 19:30 |
| psymin | rocky 9 deploy and target hosts | 19:30 |
| psymin | I dug briefly into this https://docs.rockylinux.org/guides/file_sharing/glusterfs/ | 19:30 |
| psymin | added the CentOS-Gluster-9.repo repo under the "installation" section, tried sudo dnf install centos-release-gluster9 with no luck. Mentioned it to the rocky linux channel as well. | 19:32 |
| NeilHanlon | psymin: i'm spinning up my multi-node lab to double what you're doing.. gimme an hour or so | 19:44 |
| psymin | NeilHanlon, my config is almost identitcal to yours, lvm, iscsi, but different private networks, and some kludging for mtus | 19:51 |
| psymin | for ubuntu 22, am I going to be shooting myself in the foot by installing network-manager to get nmcli? (to make the bridge setup experience more similar to rocky) | 19:52 |
| psymin | another minor issue I saw when deploying to rocky was this regarding sysctl, but rebooting the targets and trying again got past it https://paste.opendev.org/show/bexbc72iQkT1OrajJSWD/ | 19:55 |
| psymin | rocky 9.1 ansible zed | 19:56 |
| jrosser | psymin: it would be pretty unusual to use network-manager on ubuntu | 20:45 |
| psymin | I learned the netplan way :) it was simple enough | 20:45 |
| psymin | ubuntu 22.04.2 server, openstack Zed, issue with setup-infrastructure.yml during "TASK [python_venv_build : Upgrade pip/setuptools/wheel to the versions we want] ***" Digging a bit. https://paste.opendev.org/show/b5glJhy1dGxXRCMV2ygE/ | 21:44 |
| psymin | changed mtu in container to 1464 and re-ran, that time it worked | 22:01 |
| jrosser | why do you need these odd mtu? | 22:02 |
| jrosser | sounds like PPPoE | 22:04 |
| psymin | I'm sure it is a misconfiguration somewhere on a network element I don't have access to change at the moment. | 22:07 |
| jrosser | if you have unwise ICMP filtering them PMTUD can get broken and lead to this sort of thing | 22:07 |
| psymin | tomorrow I'll have it looked at | 22:07 |
| psymin | setup-openstack.yml https://paste.opendev.org/show/bfBQbeYLmYez49fJ9sNd/ retrying | 22:08 |
| jrosser | 3 way tcp handshake completes with tiny packets but first payload > MTU will break | 22:09 |
| * psymin nods. | 22:09 | |
| jrosser | should check that the galera backend is up according to haproxy (the db connection is via the loadbalancer) | 22:11 |
| jrosser | there’s a healthcheck run by the LB against the db host over http | 22:11 |
| jrosser | and a range of allowed IP for that which might need adjusting | 22:12 |
| * jrosser sleep time | 22:12 | |
| psymin | thank you :) | 22:13 |
| psymin | I wonder where I could change the variable no_log to false to get more info | 22:18 |
| psymin | looks like the galena container is connected properly to haproxy and I can query it (with a permissions error, properly since I dont have permissions) on the haproxy IP. | 22:20 |
| psymin | I changed that no_log variable to false: "A MySQL module is required: for Python 2.7 either PyMySQL, or MySQL-python, or for Python 3.X mysqlclient or PyMySQL. Consider setting ansible_python_interpreter to use the intended Python version." | 22:36 |
| psymin | sounds like I need to manually do something like pip install mysqlclient on one of these containers? | 22:37 |
| psymin | Odd I think setting this "ansible_python_interpreter: "/usr/bin/python3"" might have done it. in /etc/ansible/ansible_collections/openstack/osa/roles/db_setup/tasks/main.yml | 22:54 |
| psymin | making progresss, whew :) https://paste.opendev.org/show/bXyDEtCmnmpUpwsRBejt/ | 23:05 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!