| opendevreview | Merged openstack/openstack-ansible-tests master: Add config_template to collections https://review.opendev.org/c/openstack/openstack-ansible-tests/+/819897 | 09:08 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Add config_tempalte to collections https://review.opendev.org/c/openstack/openstack-ansible/+/819814 | 09:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Update ceph-ansible release to pacific https://review.opendev.org/c/openstack/openstack-ansible/+/820008 | 09:40 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-ceph_client master: Update ceph clients release to pacific https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/820009 | 09:41 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/819802 | 09:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/819861 | 09:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Define manila_osapi_share_workers for CI https://review.opendev.org/c/openstack/openstack-ansible/+/820010 | 09:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_manila master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_manila/+/819861 | 09:55 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_zun master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/819810 | 09:56 |
| admin1 | this tends to come up from time to time on new installs .. when setup-infra is ran the 2nd time ( because something like rabbit or something else other than galera failed ) -- galera_server : Fail if galera_cluster_name doesnt match provided value => The galera_cluster_name variable does not match what is set in mysql. Check your | 11:05 |
| admin1 | galera_cluster_name setting in your user_*.yml files in | 11:05 |
| admin1 | the name is same in all -- wsrep_cluster_name = "openstack_galera_cluster" | 11:12 |
| noonedeadpunk | I saw that happening somewhere, but didn't have time to debug that... | 11:35 |
| noonedeadpunk | but eventually it;s worth checking output of `mysql --silent --skip-column-names --connect-timeout=10 -e 'SHOW VARIABLES LIKE "wsrep_cluster_name";'` | 11:36 |
| noonedeadpunk | and actually debug _cluster_name somewhere here https://opendev.org/openstack/openstack-ansible-galera_server/src/branch/master/tasks/galera_server_cluster_state.yml#L39 to see how ansible see it | 11:37 |
| jamesdenton | admin1 I ran into this issue recently, and it seemed to be related to mysql/mariadb not running on the given node. it was not able to perform the query | 11:58 |
| noonedeadpunk | oh. so we should be smater I believe... | 12:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Use fast SSH cipher in the gate https://review.opendev.org/c/openstack/openstack-ansible/+/658473 | 12:06 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server stable/wallaby: Fix mysql_upgrade possible race condition https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/819842 | 12:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server stable/victoria: Fix mysql_upgrade possible race condition https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/819843 | 12:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Revert UEFI firmware workaround https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/793007 | 12:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Exclude ironic_compute hosts from PKI tasks https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/819661 | 12:14 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Use fast SSH cipher in the gate https://review.opendev.org/c/openstack/openstack-ansible/+/658473 | 12:27 |
| opendevreview | Merged openstack/ansible-role-systemd_service master: Use slice name for lock/run by default https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/819298 | 12:29 |
| opendevreview | Merged openstack/openstack-ansible master: Globally define systemd_lock_dir https://review.opendev.org/c/openstack/openstack-ansible/+/819300 | 13:33 |
| jamesdenton | noonedeadpunk do you know if the OSA masakari implementation is complete? | 14:24 |
| noonedeadpunk | jamesdenton: there's is a thing I never come back to and it's related to pacemaker-remote deployment which is required by masakari-hostmonitoer | 14:25 |
| noonedeadpunk | so by default pacemaker will be installed on all computes wich has severe limitation on the amount of computes in cluster | 14:26 |
| noonedeadpunk | and pacemaker-remote should be used instead | 14:26 |
| noonedeadpunk | but masakari itself is really fine - I used that in some of my deploymens and looking forward to add it to current ones next year | 14:27 |
| jamesdenton | ok - i must be missing something in this test since i don't see the monitors being installed | 14:27 |
| jamesdenton | i was hoping to document the process a little bit | 14:27 |
| jamesdenton | are your process and instance monitors working then? only hostmonitor thats not functional? | 14:29 |
| noonedeadpunk | so I had everything functional, but I had deployment on not big clusters which don't care about compute numbers in cluster | 14:30 |
| noonedeadpunk | (it's limitation of 64 computes or smth like that iirc) | 14:30 |
| noonedeadpunk | but eventually monitors should be just deployed.... | 14:31 |
| noonedeadpunk | But, iirc you need to define `masakari-monitor_hosts` eqyal to compute_hosts | 14:31 |
| noonedeadpunk | which is not default env.d behaviour | 14:31 |
| jamesdenton | yep, ok. i'm a dummy. i have monitoring_hosts not monitor_hosts | 14:34 |
| jamesdenton | not sure how i munged that one | 14:34 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Remove references to unsupported operating systems https://review.opendev.org/c/openstack/openstack-ansible/+/782557 | 14:51 |
| jrosser | need reviews on https://review.opendev.org/c/openstack/openstack-ansible/+/819814 | 14:56 |
| andrewbonney | done - thought I had already so must have missed it | 14:57 |
| spatel | noonedeadpunk that rabbitMQ patch works but look like we need little more improvement to adjust policy per vhost. (Example: if i want non-HA only for /nova and rest of world use HA) | 15:16 |
| noonedeadpunk | But I think there's a way for that? | 15:23 |
| noonedeadpunk | or, damn, we don't | 15:23 |
| noonedeadpunk | yeah, we need to fix that probably.... | 15:24 |
| noonedeadpunk | it's matter of patching all the roles though | 15:24 |
| spatel | do you have idea then i would say go ahead and implement. (just trying to make sure we don't step on each other feet) :) | 15:33 |
| noonedeadpunk | for example we can add like nova_oslomsg_rpc_policies: [] and nova_oslomsg_notify_policies: [] somehwere https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/defaults/main.yml#L128 | 15:36 |
| noonedeadpunk | and then pass that in vars https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tasks/main.yml#L80 | 15:36 |
| noonedeadpunk | the only thing I'm a bit concerned is that we merge these lists... which is probably not best idea tbh https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tasks/mq_setup.yml#L43 | 15:37 |
| opendevreview | Merged openstack/openstack-ansible-rabbitmq_server master: Fix tags usage https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/819914 | 15:37 |
| noonedeadpunk | as I'd see defined as nova_oslomsg_rpc_policies: "{{ oslomsg_rpc_policies | default([]) }}" | 15:37 |
| spatel | i like that idea, each role pass own HA policy to RabbitMQ and if value is empty then rabbitMQ set default to all HA | 15:37 |
| spatel | I am planning to deploy this kind of custom policy - https://paste.opendev.org/show/811383/ | 15:38 |
| spatel | "ha-mode":"exactly" for compute and non-HA for neutron because neutron has 5x queues compare to other components and in most case neutron is culprit | 15:39 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/819304 | 15:44 |
| spatel | i would say let first fix this for /nova, /heat , /cinder and /neutron roles only because others are not resource hungry | 15:44 |
| noonedeadpunk | so I'd say that as a result we probably should also patch https://opendev.org/openstack/openstack-ansible-os_nova/src/branch/master/tasks/mq_setup.yml#L43 and iterate only against "{{ _oslomsg_rpc_policies | default([]) }}" and indeed implement variables in defaults as nova_oslomsg_rpc_policies: "{{ oslomsg_rpc_policies | default([]) }}" | 15:48 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: Refactor galera_use_ssl behaviour https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/810258 | 15:50 |
| opendevreview | Merged openstack/openstack-ansible-os_ironic master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/819791 | 15:53 |
| spatel | noonedeadpunk go ahead and i will test out that patch | 15:54 |
| noonedeadpunk | Tbh I'd prefer doing that after releasing | 15:56 |
| noonedeadpunk | We have small time left and that is not _that_ critical to aim to merge before + a big amount of work must be done | 15:57 |
| spatel | sure no rush | 16:01 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/819795 | 16:02 |
| opendevreview | Merged openstack/openstack-ansible-os_swift master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/819805 | 16:05 |
| opendevreview | Merged openstack/openstack-ansible-os_murano stable/wallaby: Fix murano role https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/818734 | 16:40 |
| *** sshnaidm is now known as sshnaidm|afk | 16:46 | |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Revert UEFI firmware workaround https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/793007 | 16:50 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Exclude ironic_compute hosts from PKI tasks https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/819661 | 16:50 |
| opendevreview | Merged openstack/openstack-ansible-os_murano stable/wallaby: Updated from OpenStack Ansible Tests https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/798617 | 16:57 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add security headers to HAProxy Horizon service https://review.opendev.org/c/openstack/openstack-ansible/+/818533 | 16:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Define manila_osapi_share_workers for CI https://review.opendev.org/c/openstack/openstack-ansible/+/820010 | 16:58 |
| opendevreview | Merged openstack/openstack-ansible master: Add config_tempalte to collections https://review.opendev.org/c/openstack/openstack-ansible/+/819814 | 17:24 |
| *** sshnaidm|afk is now known as sshnaidm | 17:29 | |
| *** gmann is now known as gmann_afk | 17:34 | |
| jonher | i think i found a bug, galera max_connections is not calculated correctly due to this logic https://opendev.org/openstack/openstack-ansible-galera_server/src/branch/master/templates/mysql_defaults.j2#L3-L9 looking at galera_all -> galera -> lxc containers. those do not have facts for processor_xxxxx | 17:48 |
| jonher | if vcpus = 0 the logic will fall back to the default 200 max_connections | 17:48 |
| jrosser | there was a looooong discussion about this several times | 17:49 |
| jrosser | and i think it's interesting to know why the fact is missing | 17:50 |
| jrosser | the facts cache on the controller is only valid for 24hours, and if whatever playbook you ran has not gathered the facts then there is a bug | 17:50 |
| jonher | hmm yeah i see when i search back that you pasted https://paste.opendev.org/show/810346/ | 17:50 |
| jrosser | oh blimey i don't know if we addressed that, which is different again | 17:51 |
| jrosser | it needed an (int) somewhere? | 17:51 |
| jrosser | jonher: do you have a link to that in the irc logs? | 17:54 |
| jonher | https://meetings.opendev.org/irclogs/%23openstack-ansible/%23openstack-ansible.2021-11-02.log.html#t2021-11-02T16:39:53 is the point in time where you sent that link, i haven't read thru that yet | 17:56 |
| mgariepy | yeah we did test a bit of that.. | 18:00 |
| mgariepy | jonher, did you limit the run to a group ? | 18:04 |
| mgariepy | when i do run playbook i often just do `ansible all -m setup` just to be on the safe-ish side. | 18:05 |
| jonher | we observed this in a new deployment simply running the regular setup-hosts, setup-infrastructure and it still being set to 200 while we expected max | 18:05 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-galera_server master: Ensure max_connections is calculated correctly with integers https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/820058 | 18:05 |
| jrosser | jonher: ^ that will address what i saw in my paste, but I think it's a different issue | 18:05 |
| jrosser | jonher: and there was no delay between setup-hosts/infrastructure? | 18:06 |
| jrosser | there is a place where this vcpu fact is collected, if I could find it | 18:07 |
| jonher | yes there were close to eachother and can be replicated in aio as well | 18:08 |
| jrosser | oh well thats good, theres a chance to understand whats happening | 18:08 |
| mgariepy | is the time shifting on deployment ? | 18:08 |
| jrosser | you can look in the facts cache in the deploy host /etc/openstack_deploy/.... | 18:09 |
| jrosser | and first check if the fact is ever gathered | 18:09 |
| jrosser | and also if the timestamp on the fact cache file is too old | 18:09 |
| mgariepy | like you run setup-host, it does install chrony time sync >24 hours later. then when you run. setup-infrastructure it does think it's expired? | 18:10 |
| jonher | not that i recall, but i'm gonna start with trying to rerun the galera playbook, check the facts for the lxc container and see if it has the processor_xx facts or if only a subset of the facts are there | 18:11 |
| jrosser | this could totally be a ug tbh | 18:12 |
| jrosser | *bug | 18:12 |
| jrosser | look at this https://github.com/openstack/openstack-ansible/commit/f89d87c4b5f2aaf08dd783d0402766a2663a026d | 18:12 |
| jrosser | that doesnt seem to touch the galera playbook, which seems suspicious | 18:13 |
| jonher | indeed no processor_xx after rerunning galera playbook | 18:14 |
| jrosser | can you drop something like this https://github.com/openstack/openstack-ansible/blob/master/playbooks/os-glance-install.yml#L33-L37 into the galera playbook? | 18:15 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_zun master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_zun/+/819877 | 18:16 |
| jonher | yes already did, that recreates the container and sets the max_connections to 400 so that works | 18:17 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Ensure hardware facts are gathered for galera hosts. https://review.opendev.org/c/openstack/openstack-ansible/+/820059 | 18:22 |
| jonher | ^ "pre_tasks:" twice | 18:22 |
| jrosser | argh | 18:23 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Ensure hardware facts are gathered for galera hosts. https://review.opendev.org/c/openstack/openstack-ansible/+/820059 | 18:24 |
| jonher | thanks for helping me confirm and track down this :) | 18:28 |
| opendevreview | Merged openstack/openstack-ansible-galera_server stable/wallaby: Fix mysql_upgrade possible race condition https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/819842 | 18:38 |
| *** gmann_afk is now known as gmann | 18:53 | |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/819868 | 19:08 |
| opendevreview | Merged openstack/openstack-ansible-os_cloudkitty master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_cloudkitty/+/819850 | 19:16 |
| opendevreview | Merged openstack/openstack-ansible-os_blazar master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_blazar/+/819828 | 19:18 |
| opendevreview | Merged openstack/ansible-role-systemd_mount master: Use config_template as a collection https://review.opendev.org/c/openstack/ansible-role-systemd_mount/+/819822 | 19:23 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/819827 | 19:23 |
| opendevreview | Merged openstack/openstack-ansible-os_cinder master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_cinder/+/819851 | 19:24 |
| opendevreview | Merged openstack/openstack-ansible-os_barbican master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_barbican/+/819829 | 19:27 |
| opendevreview | Merged openstack/openstack-ansible-os_heat master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/819857 | 19:28 |
| opendevreview | Merged openstack/openstack-ansible-os_gnocchi master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_gnocchi/+/819855 | 19:28 |
| opendevreview | Merged openstack/openstack-ansible-os_swift master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_swift/+/819873 | 19:28 |
| opendevreview | Merged openstack/openstack-ansible-os_sahara master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_sahara/+/819872 | 19:29 |
| opendevreview | Merged openstack/openstack-ansible-os_magnum master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/819865 | 19:30 |
| opendevreview | Merged openstack/ansible-role-uwsgi master: Use config_template as a collection https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/819820 | 19:31 |
| opendevreview | Merged openstack/openstack-ansible-os_rally master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_rally/+/819870 | 19:31 |
| opendevreview | Merged openstack/openstack-ansible-os_placement master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_placement/+/819869 | 19:32 |
| opendevreview | Merged openstack/openstack-ansible master: Remove provider_networks from neutron playbook https://review.opendev.org/c/openstack/openstack-ansible/+/817914 | 19:32 |
| opendevreview | Merged openstack/openstack-ansible-os_masakari master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/819862 | 19:35 |
| opendevreview | Merged openstack/openstack-ansible-os_senlin master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_senlin/+/819871 | 19:39 |
| opendevreview | Merged openstack/openstack-ansible-os_trove master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/819875 | 19:40 |
| opendevreview | Merged openstack/openstack-ansible-os_designate master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/819853 | 19:40 |
| opendevreview | Merged openstack/openstack-ansible-os_ceilometer master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_ceilometer/+/819878 | 19:45 |
| opendevreview | Merged openstack/openstack-ansible-os_adjutant master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_adjutant/+/819826 | 19:45 |
| opendevreview | Merged openstack/openstack-ansible-os_tacker master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_tacker/+/819874 | 19:45 |
| opendevreview | Merged openstack/openstack-ansible-os_nova master: Refactor definition of lock path https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/819800 | 19:51 |
| opendevreview | Merged openstack/openstack-ansible-os_aodh master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_aodh/+/819852 | 19:53 |
| opendevreview | Merged openstack/openstack-ansible-os_murano master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/819864 | 20:16 |
| opendevreview | Merged openstack/openstack-ansible-os_horizon master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/819858 | 20:17 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/819866 | 20:18 |
| opendevreview | Merged openstack/openstack-ansible-ceph_client master: Use config_template as a collection https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/819824 | 20:26 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_masakari master: Fix libvirt service name https://review.opendev.org/c/openstack/openstack-ansible-os_masakari/+/820065 | 20:27 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible-os_nova master: Database connection pooling improvements https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/819426 | 21:05 |
| opendevreview | Damian Dąbrowski proposed openstack/openstack-ansible master: Define default variables for database connection pooling https://review.opendev.org/c/openstack/openstack-ansible/+/819424 | 21:18 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server stable/wallaby: Fix inconsistency in haproxy_frontend_raw naming https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/818744 | 21:35 |
| opendevreview | Merged openstack/openstack-ansible-haproxy_server stable/victoria: Fix inconsistency in haproxy_frontend_raw naming https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/818745 | 22:37 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!