| jrosser | rally somehow broken again https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/887058?tab=change-view-tab-header-zuul-results-summary | 07:23 |
|---|---|---|
| jrosser | on zed | 07:23 |
| opendevreview | Christian Rohmann proposed openstack/openstack-ansible-galera_server stable/2023.1: Add optional compression to mariabackup https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/887139 | 07:44 |
| opendevreview | Christian Rohmann proposed openstack/openstack-ansible-galera_server stable/zed: Add optional compression to mariabackup https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/887143 | 07:58 |
| opendevreview | Christian Rohmann proposed openstack/openstack-ansible-galera_server stable/yoga: Add optional compression to mariabackup https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/887144 | 07:59 |
| hamidlotfi_ | https://www.irccloud.com/pastebin/v4CTx1Ae/ | 08:23 |
| hamidlotfi_ | https://www.irccloud.com/pastebin/2U56pMV0/ | 08:23 |
| hamidlotfi_ | Hi there, | 08:24 |
| hamidlotfi_ | I have this error in nova install on compute node: | 08:24 |
| hamidlotfi_ | https://www.irccloud.com/pastebin/5aR8x4tM/ | 08:24 |
| admin1 | hamidlotfi_, new install ? | 08:28 |
| hamidlotfi_ | yes in the new install | 08:34 |
| noonedeadpunk | mornings | 08:45 |
| admin1 | you can rm -rf /etc/nova and retry the playbook with -v or -vvv to check if its having issues writing those files in the first place | 08:45 |
| admin1 | \o | 08:46 |
| hamidlotfi_ | let me check | 08:46 |
| noonedeadpunk | jrosser: that rally thing looks exactly like gnocchi with setuptools | 08:46 |
| noonedeadpunk | I have some hopes that this can work https://review.opendev.org/c/openstack/openstack-ansible/+/886731 | 08:47 |
| noonedeadpunk | Maybe worth backporting to Z now and adding as depends-on to see if it helps? | 08:47 |
| anskiy | noonedeadpunk: that won't fix gnocchi | 08:48 |
| noonedeadpunk | nah, it won't | 08:48 |
| noonedeadpunk | but might fix rally :) | 08:48 |
| noonedeadpunk | as rally doesn't have pyproject.toml | 08:48 |
| anskiy | about rabbitmq stuff: yesterday I was able to reproduce the problem in AIO a couple of times: it was `nova-conductor`, that didn't properly restore connection to RabbitMQ, and every new server ended being stuck in `BUILD` state. | 08:50 |
| anskiy | today I can't reproduce it anymore for some reason :( | 08:50 |
| jrosser | hamidlotfi_: are you able to reproduce that error in an AIO? | 09:29 |
| hamidlotfi_ | remove the directory /openstack/venv/nova.... and run again, the nova error is solved | 09:36 |
| noonedeadpunk | anskiy: yeah, there're totally some race conditions present in some services | 09:37 |
| hamidlotfi_ | So far I have not used AIO . | 09:37 |
| hamidlotfi_ | I think my problem was in downloading or compiling the package, which was solved by deleting the directory /openstack/venv/nova... | 09:40 |
| noonedeadpunk | I guess that's a good thing :D | 09:40 |
| noonedeadpunk | yeah, that sounds about right | 09:40 |
| noonedeadpunk | I wonder though if providing -e venv_rebuild=true would help or not.. I guess it won't as failure is before this flag would drop /openstack/venv/nova for you | 09:41 |
| jrosser | i wonder if there was an earlier failure which then results in this | 09:41 |
| jrosser | because afaik that api-paste stuff should be pretty reliable | 09:42 |
| noonedeadpunk | yeah, as it's not a common issue for sure... Unless, you did smth like `rm -rf /etc/nova/*` | 09:42 |
| noonedeadpunk | since /etc/nova/ is a symlink to /openstack/venv/nova-* | 09:42 |
| jrosser | we never ever see that being trouble in CI jobs for example | 09:42 |
| hamidlotfi_ | Is it possible to put this parameter in the user_variables.yml file? | 09:42 |
| noonedeadpunk | You should not have that there :) | 09:43 |
| hamidlotfi_ | Oh, thanks for reminding me of this important point ;) | 09:45 |
| kleini | How is it possible to force python_venv_build to completely build up the wheels newly including dependencies on the repo-container? I would have needed that for rolling back the dnspython library. I ended up with deleting the whole component on the repo-container. | 09:45 |
| jrosser | kleini: how did you roll the version back? | 09:54 |
| jrosser | ^ of dnspython | 09:54 |
| kleini | I changed it in upper-constraints file in /etc/openstack on the deploy host and then deployed only the os-designate component. | 09:55 |
| kleini | As I was not able to get that into the constraints file of the designate component on the repo-container without deleting the whole designate component there. | 09:56 |
| jrosser | you would have needed to re-run the repo server playbook, unfortunately | 09:57 |
| kleini | I found the configuration variable to define a different upper-constraints just for the designate component: designate_upper_constraints_url | 09:58 |
| jrosser | this is ugly as it's actually the repo playbook stages the constraints file from the deploy host to the web server on the repo host | 09:58 |
| kleini | jrosser, that makes sense. then I would have to use designate_upper_constraints_url to get different upper constraints only for designate | 09:58 |
| jrosser | yes, the idea is that you can have as many constraints files as you need staged on the repo server | 09:59 |
| noonedeadpunk | I think you can just place one to repo container directly? | 09:59 |
| jrosser | ^ this also, manually | 09:59 |
| jrosser | it really would be nice to make this better | 10:00 |
| kleini | the fix is already in review. this is currently enough for me. I will fork designate repository and test that fix. | 10:00 |
| kleini | so for me, I have a possible solution. was just curious, how I would have to do that "correctly". | 10:01 |
| jrosser | kleini: which bit correctly - how to use a fork? | 10:47 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible stable/zed: Pin version of setuptools https://review.opendev.org/c/openstack/openstack-ansible/+/887157 | 10:59 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_cinder stable/zed: Use v3 service type in keystone_authtoken config https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/887058 | 11:00 |
| noonedeadpunk | jrosser: probably that won't help rally (but dunno): https://opendev.org/openstack/rally-openstack/src/branch/master/upper-constraints.txt#L163 | 12:03 |
| halali | folk, dose this failure https://paste.openstack.org/show/bAlVbYyUWYLxgwDuvCYW/ on utility playbook using 27.0.0 tag because of the pip version of rally/pip 23.1.2 is different than the other pip version 22.3.1 on the other components or it something else | 12:12 |
| noonedeadpunk | nah, it's smth else, which I'm not sure how to overcome right now | 12:17 |
| noonedeadpunk | That sounds totally related to setuptools for me... But at the same time dropping down setuptools doesn't help much, as I believe some project does have pyproject.toml, so constraints are just ignored | 12:17 |
| opendevreview | Merged openstack/openstack-ansible master: Pin version of setuptools https://review.opendev.org/c/openstack/openstack-ansible/+/886731 | 13:30 |
| anskiy | OVN users, FYI: https://www.mail-archive.com/ovs-discuss@openvswitch.org/msg07431.html. Was debuggin the same problem. | 13:45 |
| jrosser | anskiy: do we need a patch to set that timout? | 13:54 |
| anskiy | I'm trying to figure proper way to do this | 13:56 |
| anskiy | kolla does set those values tho: https://review.opendev.org/c/openstack/kolla-ansible/+/839501. At least, they set `ovn_db_inactivity_probe` to non-default value of 60s | 13:58 |
| anskiy | not sure about the other values they touch | 13:59 |
| mgariepy | https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/ovn-db/tasks/bootstrap-db.yml | 14:28 |
| mgariepy | the seems to only set the inactivity probe | 14:28 |
| mgariepy | and some probe here : https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/ovn-controller/tasks/setup-ovs.yml | 14:38 |
| anskiy | yeah, and that's strange, as they introduce a couple more variables | 14:38 |
| mgariepy | how big is your deployment ? | 14:38 |
| anskiy | there're actually numerous options for probe timeouts on different parts of OVN+OVS daemons | 14:39 |
| anskiy | small for now, I think: about 1k SGs, and almost the same amount of instances | 14:45 |
| mgariepy | ok | 14:51 |
| opendevreview | Danila Balagansky proposed openstack/openstack-ansible-os_neutron master: Configure ONV NB and SB DB Connection probes https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/887192 | 14:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!