| Elnaz | jrosser: I'll check those patches asap. I accidentally removed the `openstack_inventory.json` file! dns name and IPs redesigned. I'm deploying again from the beginning. | 07:46 |
|---|---|---|
| Elnaz | And every time I have a new conectivity issue: I have mirrored the Ubuntu-Jammy repositories. But the Task `Build the base image using a CLI tool` has a hardcoded link to https://archive.ubuntu.com | 07:46 |
| Elnaz | How can I override this weird variable in the variables file: | 07:47 |
| Elnaz | `lxc_ubuntu_mirror: "{{ (ansible_facts['architecture'] == 'x86_64') | ternary('http://archive.ubuntu.com/ubuntu', 'http://ports.ubuntu.com/ubuntu-ports') }}"` | 07:47 |
| jrosser | Elnaz: a tar file backup is auto generated in /etc/openstack_deploy in case you mess up the inventory | 08:02 |
| Elnaz | [Crying] I didn't noticed! | 08:10 |
| jrosser | when you get to a real deployment it is also very useful to make that whole directory a git repo | 08:11 |
| jrosser | then you can version control your config | 08:11 |
| Elnaz | Cool idea | 08:15 |
| jrosser | Elnaz: so as far as overriding `lxc_ubuntu_mirror`goes - do you know enough ansible to be familiar with roles? | 08:22 |
| tr | hey guys, iam trying to setup openstack-ansible. somehow the playbook (/etc/ansible/ansible_collections/openstack/osa/roles/db_setup/tasks/main.yml) is failing: "username": "keystone"}]}, "msg": "unable to connect to database, check login_user and login_password are correct or /root/.my.cnf has the credentials. Exception message: (2013, 'Lost connection to MySQL server during query')" | 08:40 |
| tr | jrosser: gave me hint, thats its about the networking - but i might need another hint on how to debug this, since iam somehow stuck at the moment | 08:40 |
| jrosser | tr: can yuo describe your deployment a bit? | 08:43 |
| noonedeadpunk | mornings | 08:48 |
| noonedeadpunk | tr: I think that can also be haproxy marking galera backends as DOWN | 08:48 |
| noonedeadpunk | due to whitelisting not correct IPs that haproxy uses to talk to backends | 08:49 |
| admin1 | noonedeadpunk, is it an idea to whitelist all found ips in the controller ? | 13:00 |
| noonedeadpunk | We don't gather all found ips as facts to start with | 13:01 |
| admin1 | we can also gather that | 13:01 |
| admin1 | because i also face this issue and i see more people facing this same issue | 13:01 |
| noonedeadpunk | that does increase facts stanza dramatically, that results in ansible performnce degradation | 13:02 |
| admin1 | but its only on the controllers | 13:02 |
| noonedeadpunk | I'm more inclinded jsut to allow accessing this for everyone on mgmt network | 13:03 |
| noonedeadpunk | It's super easy to do actually | 13:03 |
| admin1 | i think an easy way is to put this as ## and the var in user_variables with an explaination | 13:03 |
| noonedeadpunk | And controllers have most amount of interfaces, as they have LXC as well | 13:03 |
| admin1 | by default | 13:03 |
| admin1 | i can attempt this :D | 13:04 |
| noonedeadpunk | we have variable `container_cidr` that's available for each host | 13:06 |
| noonedeadpunk | So it's basically `galera_monitoring_allowed_source: "{{ container_cidr }} 127.0.0.1" | 13:08 |
| tr___ | jrosser: its a virtualzed env, because of lack of hardware, based on proxmox. four nodes (deploy, infra1, compute1, storage1). all playbooks run fine, except setup-infrastrucure | 13:13 |
| tr___ | (/etc/ansible/ansible_collections/openstack/osa/roles/db_setup/tasks/main.yml) - using default openstack_user_config.yml. hosts can reach each other, also containers are reachable by ips | 13:13 |
| jrosser | tr___: can you reach one container from inside another, i.e eth1 to eth1, and also eth1 to the internal VIP | 13:32 |
| jrosser | tr___: as noonedeadpunk says it is important to first check that haproxy thinks that the database is ok | 13:33 |
| jrosser | tr___: because connections to the database are routed via the loadbalancer | 13:34 |
| noonedeadpunk | tr___: `echo "show stat" | nc -U /run/haproxy.stat | grep galera` from your haproxy host | 13:39 |
| jrosser | noonedeadpunk: this is the add compute node failure https://bugs.launchpad.net/openstack-ansible/+bug/2009834 | 13:42 |
| noonedeadpunk | Yeah, I've already seen that... | 13:44 |
| jrosser | i am not really sure what it means to be making variables `nova_all_software_versions nova_all_software_deployed nova_all_software_updated` for the case of adding one compute node | 13:44 |
| jrosser | becasue i'm not sure if the intention of those vars is to refer really to nova_all, or control plane nova, or what tbh | 13:45 |
| jrosser | eventually `nova_all_software_updated`is used to restart nova_console and nova_conductor groups | 13:47 |
| noonedeadpunk | Well. We execute service restart and online data migrations based on these | 13:47 |
| jrosser | but perhaps that needs to be also interlocked with the compute nodes also being all upgraded? | 13:47 |
| noonedeadpunk | I don't think we need to run migrations on adding compute | 13:48 |
| noonedeadpunk | We'd might want to run compute discovery for computes being added though | 13:48 |
| jrosser | right - bit distinguishing that case in the playbook might be difficult | 13:48 |
| noonedeadpunk | Yeah like `nova_all_software_updated` depends on that | 13:48 |
| noonedeadpunk | ah, yes, disregard it's sae variable) | 13:49 |
| noonedeadpunk | But yes, I assume we want to do migrations only when are fully done with upgrade | 13:49 |
| jrosser | becasue those are generated across `nova_all` it feels like the code wants the control plane and computes to be all at the new version before restart | 13:49 |
| noonedeadpunk | TO be frank I was looking at this part couple of times but didn't dare to touch | 13:50 |
| jrosser | hah yes | 13:50 |
| noonedeadpunk | Hm, would be actually interesting to check if that also might be related to our strategy... | 13:51 |
| noonedeadpunk | I have actually couple of computes to play with, but I have to finish some pre-requisitives first | 13:52 |
| jrosser | we could either add instructions to do an ansible nov_all -m setup ..... before adding the compute node | 14:03 |
| jrosser | but thats kind of sad because i think this will all break if any compute node is down/inaccessible at the point you want to add a new one | 14:03 |
| jrosser | alternatively there could be a new variable to specifically disable console/conductor restart which skips these troublesome sections when adding new computes | 14:04 |
| noonedeadpunk | this all sucks in one way or another | 14:30 |
| jrosser | maybe this needs to be it's own playbook https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-nova-install.yml#L23-L27 | 14:32 |
| jrosser | then we can deal with computes on their own | 14:32 |
| jrosser | as adding a compute really need not have anything to do with conductor/console restarts | 14:32 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Bump OpenStack-Ansible Zed https://review.opendev.org/c/openstack/openstack-ansible/+/876028 | 14:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/yoga: Bump OpenStack-Ansible Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/876982 | 14:53 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Update ansible openstack collection to 2.0.0 https://review.opendev.org/c/openstack/openstack-ansible/+/872421 | 16:03 |
| opendevreview | Merged openstack/openstack-ansible stable/zed: Fix comment typo in nova install playbook https://review.opendev.org/c/openstack/openstack-ansible/+/876678 | 16:55 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Update ansible openstack collection to 2.0.0 https://review.opendev.org/c/openstack/openstack-ansible/+/872421 | 16:59 |
| -opendevstatus- NOTICE: Yesterday's change to Gerrit configs to use submit-requirements had a boolean logic bug. This has now been corrected and any changes that did not merge as a result can be rechecked. We have reenqueued the changes we identified as being affected. | 17:23 | |
| noonedeadpunk | ^ this looks waay better now | 17:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/victoria: Bump erlang versions https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/877033 | 18:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Backport openstack_testing from Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/877014 | 18:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Backport openstack_testing from Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/877014 | 18:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Backport openstack_testing from Yoga https://review.opendev.org/c/openstack/openstack-ansible/+/876434 | 18:11 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Do not use 'always' tag in inappropriate places https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/875971 | 21:54 |
| Elnaz | jrosser: > https://review.opendev.org/c/openstack/openstack-ansible-ops/+/876855 | 21:57 |
| Elnaz | Error: https://paste.ubuntu.ir/elhcl | 21:57 |
| Elnaz | downloading the file https://artifacts.elastic.co/downloads/elasticsearch-plugins/ingest-attachment/ingest-attachment-linux-x86_64-7.17.9.zip in the browser, I get this warning: `Your client does not have permission to get URL /downloads/elasticsearch-plugins/ingest-attachment/ingest-attachment-linux-x86_64-7.17.9.zip from this server.` | 21:58 |
| jrosser | Elnaz: you are getting 403 there now | 22:01 |
| jrosser | so thats "Forbidden" rather than the 404 you had before | 22:01 |
| Elnaz | Can you download that file? | 22:02 |
| jrosser | i can `curl -O https://artifacts.elastic.co/downloads/elasticsearch-plugins/ingest-attachment/ingest-attachment-7.17.9.zip` | 22:02 |
| jrosser | then you can also look in the test job logs for the patch i made https://zuul.opendev.org/t/openstack/build/6dec562216a648d09d91b820fe8f5d81/log/job-output.txt#972 | 22:03 |
| Elnaz | Where can I see the user_variables.yml configuration of this test job? | 22:08 |
| jrosser | it is not a deployment of openstack-ansible, just the ELK stack on its own | 22:09 |
| jrosser | Elnaz: but really - you do not need this plugin - see the documentation for what it does https://www.elastic.co/guide/en/elasticsearch/plugins/7.17/ingest-attachment.html | 22:10 |
| jrosser | so i think the best thing to do is override this https://github.com/openstack/openstack-ansible-ops/blob/master/elk_metrics_7x/roles/elasticsearch/defaults/main.yml#L34-L35 | 22:10 |
| jrosser | set `elastic_plugins: []` and then it will just skip this step | 22:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-rabbitmq_server stable/victoria: Bump erlang versions https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/877033 | 22:11 |
| Elnaz | "The Linux world has never "evolved" a central place to configure proxy servers, so there simply is no way to enforce the usage of a specific proxy server short of redirecting all network traffic in the firewall, https://superuser.com/a/608273" | 22:13 |
| jrosser | well my test lab is behind a proxy, so i know pretty much about this :) | 22:14 |
| jrosser | but just disable the plugin! | 22:14 |
| noonedeadpunk | Let's unblock Xena by landing https://review.opendev.org/c/openstack/openstack-ansible/+/876434 :) | 22:16 |
| Elnaz | I have set up an squid proxy server that has chained to another server in Canada through vmess protocol. Setting `http(s)_proxy` in /etc/environment does not work most of the times! | 22:16 |
| jrosser | java has it's own mechanisms too | 22:17 |
| jrosser | and ELK is java | 22:17 |
| jrosser | noonedeadpunk: as its a backport i guess you can also vote :) | 22:18 |
| noonedeadpunk | yeah, I was thinking about exactly the same ) | 22:18 |
| noonedeadpunk | W and V are way more messy... | 22:19 |
| noonedeadpunk | But it's also about tempest actually, plus tests repo | 22:19 |
| jrosser | elk roles support proxy anyway https://github.com/openstack/openstack-ansible-ops/blob/master/elk_metrics_7x/roles/elasticsearch/tasks/elasticsearch_plugins.yml#L29-L34 | 22:25 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!