| noonedeadpunk | hm, what's wrong with git-core.... https://zuul.opendev.org/t/openstack/build/cec068efae7c4a3991162fa659cdb1e7/log/job-output.txt#13859 is that some another weird dependency resolver issue brought in by apt module changes... | 08:22 |
|---|---|---|
| jrosser_ | do we need to backport this https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/834183 (re ML posts) | 08:24 |
| noonedeadpunk | That is great question | 08:25 |
| noonedeadpunk | I haven't checked when we introduced to use overrides_only | 08:26 |
| noonedeadpunk | wait | 08:26 |
| noonedeadpunk | I provided wrong patch in ML | 08:26 |
| noonedeadpunk | damn | 08:27 |
| noonedeadpunk | or it's correct one.... | 08:28 |
| kleini | Is it somehow possible to change configuration for keepalived in haproxy_server role? I have issues sending VRRP multicast packages if the network interface is a Linux bridge. sendmsg in keepalived gets EPERM. unicast VRRP works. | 08:29 |
| noonedeadpunk | kleini: anskiy has switched to unicast with overrides. He also pushed several patches to improve way of defining these overrides. | 08:30 |
| noonedeadpunk | so yes, in short that is possible | 08:30 |
| noonedeadpunk | jrosser_: I think that's right patch at the end :D As we place thing to /etc/systemd which indeed applies overrides. So I guess we can backport it relatively safely | 08:31 |
| jrosser_ | yeah it's definatly a bug - i remember spatel having trouble with this | 08:31 |
| jrosser_ | but you are right it needs the right features in systemd_service role to match | 08:32 |
| noonedeadpunk | I guess ansible 2.13 still breaks installation of meta packages... | 08:33 |
| jrosser_ | where it tries to work out the versions? | 08:34 |
| noonedeadpunk | yeah | 08:34 |
| noonedeadpunk | but I fail to understand why it does work in some containers but not in others | 08:35 |
| jrosser_ | feels like eventually they have to re-implement the whole of apt somehow | 08:35 |
| jrosser_ | very odd | 08:35 |
| noonedeadpunk | they said they still do that to identify changed state... | 08:36 |
| noonedeadpunk | so prefer fixing own dependency resolution thing | 08:36 |
| noonedeadpunk | but maybe this failure is because we don't update cache... | 08:37 |
| noonedeadpunk | "attempts": 5, "cache_update_time": 1664887751, "cache_updated": false, "changed": false, "msg": "'/usr/bin/apt-get -y -o \"Dpkg::Options::=--force-confdef\" -o \"Dpkg::Options::=--force-confold\" install 'cmake=3.18.4-2+deb11u1' 'python3-pkgconfig=1.5.1-3' 'python3-setuptools=59.6.0-1.2~bpo11+1' 'git-core=1:2.34.1-1~bpo11+1' 'libkmod-dev=28-1' 'libpq-dev=13.8-0+deb11u1' 'libxslt1-dev=1.1.34-4+deb11u1'' failed: E: Version '1:2.34.1- | 08:38 |
| noonedeadpunk | 1~bpo11+1' for 'git-core' was not found | 08:38 |
| jrosser_ | noonedeadpunk: probably becasue of this https://github.com/openstack/openstack-ansible-os_cinder/blob/d120669a7d8f14af3ab1ba1ee3b438ac88225358/vars/debian.yml#L35 | 08:38 |
| jrosser_ | i expect we can drop that no problem | 08:38 |
| noonedeadpunk | but we do have git-core in more roles | 08:38 |
| noonedeadpunk | oh, I see difference though | 08:39 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-os_placement/src/branch/master/vars/redhat.yml#L17 | 08:39 |
| kleini | MariaDB is configured with 200 max connections. my deployment has total 15 nodes. haproxy seems to send all connections to the first instance and it runs constantly into max connections making then the clustercheck fail. is it correct, that first instance needs to cover all connections? | 08:40 |
| noonedeadpunk | kleini: with haproxy there's no other way. With more proper balancer (L7 instead of L4) it would be possible to send read connections to other backends | 08:42 |
| noonedeadpunk | but maria still needs to recieve writes only on 1 backend | 08:42 |
| noonedeadpunk | jrosser_: so I'm quite confused how placement passed and cinder not | 08:42 |
| noonedeadpunk | ah lol | 08:43 |
| noonedeadpunk | I've opened redhat for placement | 08:44 |
| * noonedeadpunk needs some extra coffee | 08:44 | |
| noonedeadpunk | Debian is indeed does smth weird with git-core | 08:46 |
| noonedeadpunk | https://paste.openstack.org/show/bbkAID5tL50dCygQfTLX/ | 08:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cinder master: Replace git-core with git for debian https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/860349 | 08:51 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_ironic master: Replace git-core with git for debian https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/860350 | 08:53 |
| noonedeadpunk | zigo: you might be interested in this behaviour as well (how ansible-core 2.13 tries to install git-core specifically on Debian) | 08:54 |
| zigo | noonedeadpunk: The git-core package doesn't exist since buster... | 08:55 |
| noonedeadpunk | well... there's smth transitioning present? | 08:56 |
| zigo | It's a virtual package (ie: a package that other packages have a "Provides:" only). | 08:56 |
| zigo | "git" Provides: it... | 08:56 |
| noonedeadpunk | so basically nobody supposed to use `apt install git-core`? | 08:57 |
| zigo | Correct. | 08:57 |
| noonedeadpunk | ok, thanks | 08:57 |
| zigo | Same in Ubuntu btw. | 08:57 |
| noonedeadpunk | I wonder how ansible does find (and tries to install) 'git-core=1:2.34.1-1~bpo11+1' when it's said to isntall git-core | 08:58 |
| noonedeadpunk | for some reason ubuntu does not fail same way... | 08:58 |
| zigo | No idea. I'm not an ansible user, I think Ansible is a way too slow, and I don't like yaml everywhere. I'm currently working on packaging https://github.com/purpleidea/mgmt in Debian to see if it's doing the job better than puppet/salt/ansible... | 08:59 |
| zigo | I use Ansible for small tasks only. | 08:59 |
| noonedeadpunk | oh, ok, thanks for input anyway! | 09:00 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_cloudkitty master: Replace git-core with git for debian https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/860353 | 09:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Replace git-core with git for debian https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/860354 | 09:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_mistral master: Replace git-core with git for debian https://review.opendev.org/c/openstack/openstack-ansible-os_mistral/+/860355 | 09:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Replace git-core with git for debian https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/860356 | 09:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_swift master: Replace git-core with git for debian https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/860357 | 09:05 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_tempest master: Replace git-core with git for debian https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/860358 | 09:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump ansible-core version to 2.13.4 https://review.opendev.org/c/openstack/openstack-ansible/+/857506 | 09:08 |
| kleini | Do we have somewhere a sizing, how much database connections are necessary at maximum? Upgrade to Ubuntu 20.04 on infra nodes seems to increase dramatically the amount of necessary database connections. Only half of OS services are up now and the double amount of database connections are used. | 09:41 |
| noonedeadpunk | kleini: I don't think we do. But I'd say set smth like 2000 | 09:47 |
| noonedeadpunk | or 1500 | 09:47 |
| kleini | oh, found now: "computed value which is (100 x vCPUs) with a cap of 1600", which would mean 1200 connections on my primary infra node. but this is calculated now to 200. here is the issue! | 10:04 |
| opendevreview | Merged openstack/ansible-hardening master: Remove sebang from get_users https://review.opendev.org/c/openstack/ansible-hardening/+/860276 | 10:17 |
| kleini | https://paste.opendev.org/show/bAdKVbg2kPAZcqNLx5O6/ <- here is the output of the calcuation for Galera max connections. It does something weird on Ubuntu 20.04. | 10:45 |
| jrosser_ | kleini: which branch/release are you using? | 10:51 |
| damiandabrowski | it would be calculated to 200 if `hostvars[galera_node]['ansible_facts']['processor_vcpus']` is not set | 10:51 |
| kleini | 23.4.1 | 10:51 |
| damiandabrowski | i wonder if this fact was available for all galera nodes when you were running galera playbook | 10:52 |
| jrosser_ | if the facts are more than 24h old then they are timed-out | 10:52 |
| kleini | https://paste.opendev.org/show/bq4OxVNGb04kevk62BmX/ | 10:55 |
| kleini | yes, sorry, only 2 infra nodes and even number of Galera instances. | 10:55 |
| jrosser_ | but if those facts were not gathered in the galera playbook they would not have been there | 10:56 |
| jrosser_ | oh well.... not sure actually | 10:57 |
| kleini | I have in staging one infra node with 18.04 and one with 20.04 and I can compare directly. ansible_facts vcpus is correct there. must be something with the calculation in the jinja2 template: my.cnf.j2 | 10:58 |
| kleini | but I have no idea, how to debug a jjnja2 template on the fly... | 10:58 |
| jrosser_ | you can just template more stuff out into the resulting file, in comments or something | 11:00 |
| jrosser_ | perhaps insert a fail: task into the tasks right after the templating and write out whatever you need | 11:03 |
| damiandabrowski | when it comes to debugging, you can start with this: https://paste.openstack.org/raw/biln0fya9AlvMooYCwZf/ | 11:03 |
| damiandabrowski | unfortunately i wasn't able to finish it as I need to leave now :/ good luck | 11:04 |
| jrosser_ | ^ nice | 11:05 |
| jrosser_ | kleini: https://paste.opendev.org/show/bg5ONlfDiwfaElFhtIRJ/ | 11:12 |
| kleini | that works and provides good values | 11:15 |
| jrosser_ | hopefully thats helpful to work out what is going on | 11:17 |
| kleini | I can not reproduce the issue in production any more. In staging processor_vcpus is unknown. It gets more and more strange... | 11:19 |
| kleini | *becomes* | 11:19 |
| jrosser_ | take a look in the cached facts in /etc/openstack_deploy/ansible_facts/..... | 11:21 |
| jrosser_ | see if the vcpu data is there | 11:21 |
| jrosser_ | the task "gather additional hardware facts" in the playbook i pasted is the one that should make that be populated | 11:21 |
| opendevreview | Merged openstack/openstack-ansible-galera_server master: Use policy_rc_d attribute instead of copy https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/859272 | 11:22 |
| jrosser_ | i had `"The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'processor_vcpus'` until i added that extra facts gathering step in | 11:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Bump mariadb version to 10.6.10 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/858555 | 11:26 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Use policy_rc_d attribute instead of copy https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/859273 | 11:27 |
| kleini | I have ansible_processor_vcpus in cached facts but not processor_vcpus. And if that is missing that default(2) seems to be used. | 11:28 |
| jrosser_ | it should be that ansible_processor_vcpus is equivalent to ['ansible_facts']['processor_vcpus'] | 11:29 |
| jrosser_ | perhaps print the both of those as debug tasks | 11:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Allow do disable SSL only for stats frontend https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/858939 | 11:29 |
| kleini | https://paste.opendev.org/show/bDcZIadUhradI8knd2QG/ <- do you see a difference? I am searching for the typo | 11:34 |
| opendevreview | Merged openstack/openstack-ansible-os_trove master: Change networks_info output structure https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/855075 | 11:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server master: Validate haproxy conf after assemble https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/860402 | 11:37 |
| jrosser_ | well the only thing that comes to mind is https://docs.ansible.com/ansible/latest/reference_appendices/config.html#inject-facts-as-vars | 11:38 |
| jrosser_ | but we don't mess with that | 11:38 |
| jrosser_ | kleini: your paste is two different deployments even though the container names are the same? | 11:40 |
| noonedeadpunk | jrosser_: btw I guess we never disabled INJECT_FACTS_AS_VARS? | 11:44 |
| jrosser_ | no - as there were a bunch of things in external repos, which actually we might have got rid of now | 11:45 |
| jrosser_ | i think sshd role being one of them | 11:45 |
| noonedeadpunk | btw in older releases running with tags could also result in not gathering facts | 11:45 |
| noonedeadpunk | and keepalived another one, but I know it was also fixed | 11:46 |
| noonedeadpunk | maybe ceph-ansible.... | 11:46 |
| jrosser_ | kleini: ^ this is a great point - can you check that you have this patch? https://github.com/openstack/openstack-ansible/commit/2492d82da6fe3070528f6155bb918e45d0501809 | 11:46 |
| jrosser_ | but if no tags are used it should be OK? | 11:47 |
| noonedeadpunk | yes, without tags it should be fine | 11:47 |
| anskiy | there was a patch for the calculation some time ago that, I think, jrosser_ did | 11:48 |
| anskiy | where it wrongly tried to do operation on string, and afair, it was merged in W?.. | 11:48 |
| jrosser_ | noonedeadpunk: here for disabling fact vars injections https://review.opendev.org/c/openstack/openstack-ansible/+/778396 | 11:50 |
| jrosser_ | maybe we are close now to being able to disable that | 11:50 |
| anskiy | https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/820058 this one | 11:50 |
| noonedeadpunk | we should try rebasing 778396 and see I guess | 11:51 |
| jrosser_ | i think thats fairly obvious in the debug output as you get the cpu count number repeated 100 times | 11:52 |
| anskiy | this thing was fixed here: https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/820058, and I think it was merged in X?.. Whilst kleini runs 23.something | 11:54 |
| anskiy | kleini: here is an example of unicast things for keepalive: https://paste.opendev.org/show/bgimTlo8VKwEqNvG02ad/, if you aren't running master :) This thing goes into `user_variables.yaml`. | 11:58 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Disable fact variables https://review.opendev.org/c/openstack/openstack-ansible/+/778396 | 11:59 |
| opendevreview | Merged openstack/openstack-ansible master: Fix cinder-volume AIO ceph scenario https://review.opendev.org/c/openstack/openstack-ansible/+/859263 | 12:01 |
| opendevreview | Merged openstack/openstack-ansible master: Extend NFS scenario to support cinder https://review.opendev.org/c/openstack/openstack-ansible/+/859336 | 12:01 |
| opendevreview | Merged openstack/openstack-ansible master: Run test_minimum_basic_scenario test for cinder https://review.opendev.org/c/openstack/openstack-ansible/+/859339 | 12:01 |
| opendevreview | Merged openstack/openstack-ansible master: Replace usage of which with command https://review.opendev.org/c/openstack/openstack-ansible/+/859550 | 12:01 |
| opendevreview | Merged openstack/openstack-ansible master: Require usage of service_token_roles https://review.opendev.org/c/openstack/openstack-ansible/+/856437 | 12:30 |
| opendevreview | Merged openstack/openstack-ansible master: Do not spawn ironic-api on metal https://review.opendev.org/c/openstack/openstack-ansible/+/858877 | 12:30 |
| kleini | https://paste.opendev.org/show/bDcZIadUhradI8knd2QG/ <- this is the same deployment. feels like heisenberg's uncertainty | 12:31 |
| opendevreview | Merged openstack/openstack-ansible-os_keystone master: Improve way of cache backend selection https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/859054 | 12:44 |
| kleini | thanks, anskiy | 12:51 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-plugins master: Use ansible_facts[] instead of injected fact vars https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/860424 | 14:18 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Disable fact variables https://review.opendev.org/c/openstack/openstack-ansible/+/778396 | 14:22 |
| anskiy | noonedeadpunk: is there some kind of ETA for Zed release? Trying to figure out if I could finish the changes until then. | 14:49 |
| noonedeadpunk | in OSA or in general? | 14:53 |
| noonedeadpunk | as in general it's already a feature freeze | 14:53 |
| noonedeadpunk | in osa - we have 2 month after general release | 14:53 |
| noonedeadpunk | so release date more depends on our definition of "done" | 14:54 |
| anskiy | in OSA. Okay, got it, thank you. | 14:54 |
| noonedeadpunk | the earier we release is better, but historically we don't release early | 14:55 |
| jrosser_ | anskiy: you should add stuff to here then it won't get forgotton https://etherpad.opendev.org/p/osa-antelope-ptg | 15:04 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Disable fact variables https://review.opendev.org/c/openstack/openstack-ansible/+/778396 | 15:07 |
| anskiy | jrosser_: they're a bit too specific and not that big, but here you go :) | 15:16 |
| jrosser_ | perhaps i misunderstand - it sounded like you were working on patches you wanted in the Z release? | 15:17 |
| anskiy | yeah, I want these to be available in Zed release, that means I need to finish them until the cut happens, I suppose | 15:17 |
| anskiy | but they look kinda like normal patches, not topics, like the other points in the list | 15:19 |
| anskiy | that's why the question if it's okay to add them | 15:24 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Disable fact variables https://review.opendev.org/c/openstack/openstack-ansible/+/778396 | 15:25 |
| NeilHanlon | anskiy: can you share an example? | 15:40 |
| NeilHanlon | see also: https://docs.openstack.org/openstack-ansible/latest/reference/releases.html#when-does-a-patch-to-an-osa-role-get-into-a-release | 15:43 |
| anskiy | NeilHanlon: I've already added them to etherpad... Thank you for the link with explanations. | 15:45 |
| jrosser_ | anskiy: regarding OVN there is still a bunch of work to do on TLS | 15:47 |
| anskiy | jrosser_: which, I assume, just porting spatel's doc on it (https://satishdotpatel.github.io/ovn-ssl-setup-with-openstack/) to Ansible :) | 15:52 |
| jrosser_ | anskiy: well almost the opposite really | 15:53 |
| jrosser_ | OSA has it's own CA which should be used rather than whatever randomness OVN self-generates | 15:53 |
| anskiy | well, except that bit, yeah, but everything else looks fine (never used OVN with SSL) | 15:54 |
| anskiy | can't promise anything tho :( I still have some work to do with upstreaming my SDK patch, and creating another one for Nova. Hopefully, by the end of october I will see if I would have some spare time to work on this too | 15:59 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Disable fact variables https://review.opendev.org/c/openstack/openstack-ansible/+/778396 | 16:05 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Add nfs and ceph jobs to templates https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/859337 | 17:08 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Use ansible_facts[] rather than ansible_ variables https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/860480 | 17:13 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Disable fact variables https://review.opendev.org/c/openstack/openstack-ansible/+/778396 | 17:13 |
| *** dviroel is now known as dviroel|afk | 21:17 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!