| *** aussielunix_ is now known as aussielunix | 00:56 | |
| *** aussielunix is now known as aussielunix_ | 04:51 | |
| *** aussielunix_ is now known as aussielunix | 04:51 | |
| *** frenzyfriday|PTO is now known as frenzyfriday | 05:20 | |
| noonedeadpunk | OutBackDingo_: sounds like you don't have lsync running properly or indeed you just missed repo_setup.yml | 08:06 |
|---|---|---|
| *** odyssey4me|away is now known as odyssey4me | 09:00 | |
| *** odyssey4me is now known as odyssey4me|away | 09:00 | |
| *** odyssey4me|away is now known as odyssey4me | 09:02 | |
| *** odyssey4me is now known as odyssey4me|away | 09:25 | |
| *** odyssey4me|away is now known as odyssey4me | 09:25 | |
| jrosser | morning | 10:00 |
| noonedeadpunk | o/ | 10:01 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement https://review.opendev.org/c/openstack/openstack-ansible-tests/+/833596 | 10:09 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/831530 | 10:11 |
| noonedeadpunk | jrosser: I believe we need netcommon there as for utils there's no redirect iirc so ipaddr without full path was failing as well | 10:11 |
| jrosser | oh so i need netcommon in the same patch? | 10:12 |
| noonedeadpunk | yeah, I guess so. Not 100% sure, but ~90% | 10:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement https://review.opendev.org/c/openstack/openstack-ansible-tests/+/833596 | 10:13 |
| jrosser | theres a couple things needing collections installing with openstack-ansible-tests which is failing | 10:14 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone stable/xena: add oauth support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/833552 | 10:23 |
| *** arxcruz is now known as arxcruz|lunch | 11:04 | |
| jrosser | so this fails on not having the uwsgi role https://review.opendev.org/c/openstack/openstack-ansible-tests/+/833596 | 11:06 |
| jrosser | what did we break there | 11:06 |
| *** dviroel is now known as dviroel|ruck | 11:14 | |
| noonedeadpunk | ansible.netcommon should be 2.5.1 I guess, but not sure it was reason | 11:15 |
| noonedeadpunk | oh, well, uwsgi not here https://opendev.org/openstack/openstack-ansible-tests/src/branch/master/test-ansible-role-requirements.yml | 11:16 |
| noonedeadpunk | And I guess it's result of switching keystone to uwsgi | 11:17 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_keystone/commit/cb7eaa7ce305328a8677bcbe6a34f7577b77b041 | 11:17 |
| noonedeadpunk | jrosser: ^ | 11:17 |
| noonedeadpunk | I think we need to patch everything in one commit anyway... | 11:18 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement https://review.opendev.org/c/openstack/openstack-ansible-tests/+/833596 | 11:33 |
| *** arxcruz|lunch is now known as arxcruz | 12:04 | |
| *** odyssey4me is now known as odyssey4me|away | 12:18 | |
| *** odyssey4me|away is now known as odyssey4me | 12:18 | |
| *** odyssey4me is now known as odyssey4me|away | 12:33 | |
| jrosser | i wonder if they change how the ipaddr filter works | 12:43 |
| jrosser | systemd-networkd.service /etc/systemd/network/8-osa_testing-br-lbaas.network:5: Invalid address '['172.29.232.100/22']', ignoring assignment: Invalid argument | 12:43 |
| jrosser | from here https://zuul.opendev.org/t/openstack/build/501de017a698406c8c952edc93b9add4/log/logs/host/systemd-networkd.service.journal-17-40-50.log.txt | 12:43 |
| *** odyssey4me|away is now known as odyssey4me | 12:45 | |
| jrosser | hmm https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_501/831603/1/check/openstack-ansible-deploy-aio_metal-ubuntu-focal/501de01/logs/etc/host/systemd/network/8-osa_testing-br-lbaas.network.txt | 12:45 |
| *** odyssey4me is now known as odyssey4me|away | 12:46 | |
| *** odyssey4me|away is now known as odyssey4me | 12:50 | |
| *** odyssey4me is now known as odyssey4me|away | 12:50 | |
| mgariepy | arf it returns a list of ips instead of an ip.. | 12:50 |
| jrosser | indeed, but kind of not previously | 12:54 |
| jrosser | which is wierd | 12:55 |
| mgariepy | https://github.com/ansible-collections/ansible.utils/blob/main/plugins/filter/ipaddr.py | 12:58 |
| mgariepy | https://github.com/ansible-collections/ansible.netcommon/commit/db4920ebf6bae6476ff8829e2cf475f19f83a990#diff-0121234d8f291cc643e532655eb9bf3d94e0377b723e5e867177baca8311a752L148-L153 | 13:00 |
| *** odyssey4me|away is now known as odyssey4me | 13:00 | |
| mgariepy | from here : https://github.com/ansible-collections/ansible.netcommon/commit/db4920ebf6bae6476ff8829e2cf475f19f83a990#diff-0121234d8f291cc643e532655eb9bf3d94e0377b723e5e867177baca8311a752L509 | 13:00 |
| jrosser | well i guess i more mean that the implementation in netcommon didnt used to return a list? | 13:21 |
| mgariepy | yep. it was returning a string | 13:26 |
| mgariepy | netcomm was returning a string and utils do return lists. | 13:27 |
| jrosser | thats unfortunate | 13:33 |
| jrosser | probably means we have some circular dependancies then | 13:33 |
| jrosser | as [0] on a string is not the same as [0] on a list | 13:33 |
| mgariepy | there are some QA missing on ansible .. | 13:35 |
| noonedeadpunk | jrosser: there was a bug for utility claiming on same issue | 13:37 |
| jrosser | i was not spotting anything here https://github.com/ansible-collections/ansible.utils/issues | 13:37 |
| noonedeadpunk | https://github.com/ansible-collections/ansible.utils/issues/151 | 13:37 |
| jrosser | ahha yes just found that | 13:38 |
| noonedeadpunk | they have 2.5.2 now | 13:38 |
| jrosser | https://github.com/ansible-collections/ansible.utils/issues/148 | 13:38 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement https://review.opendev.org/c/openstack/openstack-ansible-tests/+/833596 | 13:39 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add ansible.utils collectoin requirement https://review.opendev.org/c/openstack/openstack-ansible/+/831525 | 13:48 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Update netcommon collection https://review.opendev.org/c/openstack/openstack-ansible/+/833117 | 13:49 |
| NeilHanlon | noonedeadpunk: would some love for the openstack-ansible-ops graylog role be welcome? Seems it's in a bit of a bad state right now w.r.t. archived upstream dependencies and old versions | 13:58 |
| noonedeadpunk | NeilHanlon: they will for sure, yes | 13:59 |
| NeilHanlon | 👍 | 14:00 |
| noonedeadpunk | love for everything is always welcome :p | 14:00 |
| NeilHanlon | :D | 14:00 |
| NeilHanlon | i have a terrible habit of finding more work for me to do hehe | 14:00 |
| spatel | good habit though NeilHanlon | 14:00 |
| NeilHanlon | very true, I'm never not busy lol | 14:01 |
| spatel | noonedeadpunk is xena 24.0.1 is ready to rollout in prod, hope no lingering issue :) | 14:02 |
| noonedeadpunk | mmmm, I guess we could have... | 14:03 |
| spatel | then i would like to hold on | 14:03 |
| noonedeadpunk | hm, and why 24.1.0 not released yet | 14:03 |
| spatel | hmm | 14:04 |
| noonedeadpunk | I'm also now seeing issues regarding decreased db pooling I believe in our biggest region | 14:05 |
| noonedeadpunk | and https://review.opendev.org/q/topic:bug%252F1961603 is also related | 14:05 |
| spatel | what do you mean decreased db pool? | 14:05 |
| noonedeadpunk | spatel: mean this https://review.opendev.org/q/topic:db-pooling | 14:32 |
| spatel | oh! i thought this patch is part of 24.0.0 | 14:34 |
| spatel | you are saying in order to upgrade xena we have to increate db pool size ? | 14:35 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-lxc_container_create master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/833118 | 14:36 |
| noonedeadpunk | spatel: yep, that is part of 24.0.0 but seems it's causing us issues | 14:39 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-lxc_hosts master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/833119 | 14:39 |
| noonedeadpunk | well, I'm not 100% sure it's it, but the only thing to suspect atm and testing actively | 14:40 |
| noonedeadpunk | the problem is that upon connection_recycle_time even non-idle connections will be recycled, and with conjuction decreasing of pool of connections we hit interminent failures for some mysql requests | 14:41 |
| spatel | Nothing wrong to increate pooling limit correct? | 14:42 |
| jrosser | andrewbonney: ^ you seen this? | 14:42 |
| andrewbonney | I don't believe so | 14:43 |
| jrosser | this somehow feels like a bug if active connections are recycled | 14:43 |
| andrewbonney | Does it prompt any tracebacks or similar? I could have a search if so | 14:44 |
| noonedeadpunk | I also pushed https://review.opendev.org/c/openstack/oslo.db/+/833623 as this could be likely other way around | 14:45 |
| noonedeadpunk | Related doc: https://docs.sqlalchemy.org/en/14/core/pooling.html#disconnect-handling-pessimistic | 14:45 |
| noonedeadpunk | andrewbonney: I have sample traceback | 14:46 |
| noonedeadpunk | https://paste.citynetwork.se/ozipediyegirimo.rb | 14:46 |
| noonedeadpunk | damn :) | 14:46 |
| noonedeadpunk | https://paste.openstack.org/show/bcyfeUiNi3HTIxHKrZos/ | 14:47 |
| andrewbonney | Ok, I can't see any sign of that here | 14:50 |
| MrClayPole | While testing the an OSA upgrade from train (20.2.6) to victoria (20.4.1 ) following the guide https://docs.openstack.org/openstack-ansible/victoria/admin/upgrades/major-upgrades.html, it fails with following error on all containers during the "openstack-ansible setup-hosts.yml --limit '!galera_all:!rabbitmq_all'" with the error "(item={'key': 'lxcbr0_address', 'value': {'bridge': 'lxcbr0', 'interface': 'eth0', 'type': | 15:00 |
| MrClayPole | 'veth'}}) => {"ansible_loop_var": "item", "changed": false, "item": {"key": "lxcbr0_address", "value": {"bridge": "lxcbr0", "interface": "eth0", "type": "veth"}}, "msg": "AnsibleError: template error while templating string: no filter named 'ipaddr'." I've tried a manual "pip install ipaddr" on one of the containers but this doesn't resolve the issue. | 15:00 |
| MrClayPole | This is during this part of the playbook https://opendev.org/openstack/openstack-ansible-lxc_container_create/src/branch/stable/victoria/tasks/lxc_container_network_new.yml#L21 | 15:00 |
| jrosser | MrClayPole: probably this https://review.opendev.org/c/openstack/openstack-ansible/+/831853 | 15:03 |
| jrosser | ansible have made breaking changes unfortunately | 15:03 |
| MrClayPole | Thanks for the pointer. I'll get installed and see if that helps. | 15:04 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-lxc_hosts master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/833119 | 15:10 |
| *** dviroel|ruck is now known as dviroel|ruck|lunch | 15:36 | |
| noonedeadpunk | we actually see that only in one region out of 3 upgraded | 15:44 |
| noonedeadpunk | so can be unrelated but I can hardly imagine any other reason tbh | 15:45 |
| *** melwitt_ is now known as melwitt | 16:14 | |
| admin1 | is there a way to override polling.yaml of ceilometer .. or add more more stuff via osa ? | 16:18 |
| admin1 | add more items in the meters: in polling.yaml | 16:18 |
| noonedeadpunk | I bet there was | 16:19 |
| noonedeadpunk | iirc you could even provide your own one | 16:19 |
| noonedeadpunk | `ceilometer_polling_default_file_path` - for your own file or ceilometer_polling_yaml_overrides for overrides | 16:20 |
| admin1 | thanks .. | 16:21 |
| admin1 | got it | 16:21 |
| *** dviroel|ruck|lunch is now known as dviroel|ruck | 16:22 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-tests master: Add ansible.utils collection requirement https://review.opendev.org/c/openstack/openstack-ansible-tests/+/833596 | 16:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Change location of ipaddr filter https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/831530 | 16:28 |
| noonedeadpunk | spatel: 24.1.0 released jsut in case | 16:29 |
| spatel | +1 thanks i will push out to my dev openstack first and validate | 16:30 |
| MrClayPole | Thanks jrosser, that patch fixed my issue. | 16:46 |
| *** arxcruz is now known as arxcruz|off | 17:54 | |
| spatel | Any idea what is wrong here? I deployed k8s cluster and when trying to obtain creds/config getting this error - https://paste.opendev.org/show/bhkwVmwwerPHDM46Ymgv/ | 18:43 |
| spatel | Nevermind upgrade of magnumclient resolved issue | 18:58 |
| opendevreview | Merged openstack/openstack-ansible master: Add ansible.utils collectoin requirement https://review.opendev.org/c/openstack/openstack-ansible/+/831525 | 19:01 |
| *** dviroel|ruck is now known as dviroel|ruck|afjk | 20:50 | |
| *** dviroel|ruck|afjk is now known as dviroel|ruck|afk | 20:50 | |
| opendevreview | Merged openstack/ansible-role-systemd_networkd master: Change location of ipaddr filter https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/831603 | 22:46 |
| *** dviroel|ruck|afk is now known as dviroel|ruck | 23:49 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!