| admin1 | what would give this error ? oslo_messaging.rpc.server libvirt.libvirtError: internal error: process exited while connecting to monitor: 2023-05-01T07:15:06.275384Z qemu-system-x86_64: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/qemu,endpoint=server,verify-peer=on: Our own certificate /etc/pki/qemu/server-cert.pem failed validation | 07:53 |
|---|---|---|
| admin1 | against /etc/pki/qemu/ca-cert.pem: The certificate hasn't got a known issuer | 07:53 |
| noonedeadpunk | admin1: I'd say smth went wrong with CA/PKI stuff. For example, Root CA got rotated somehow | 08:36 |
| admin1 | i see a new tag 26.1.1 out .. so following the minor upgrade in the hope that it will fix this error also | 08:50 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Fix usage of {% raw %} tags and Jinja2 headers https://review.opendev.org/c/openstack/ansible-config_template/+/881887 | 09:18 |
| admin1 | noonedeadpunk, the nova playbooks have run, but i still have the same error qemu-system-x86_64: -object tls-creds-x509,id=vnc-tls-creds0,dir=/etc/pki/qemu,endpoint=server,verify-peer=on: Our own certificate /etc/pki/qemu/server-cert.pem failed validation against /etc/pki/qemu/ca-cert.pem: The certificate hasn't got a known issue | 10:00 |
| admin1 | pasting variables .. . | 10:02 |
| noonedeadpunk | admin1: as I said, I think this is related to the changed Root CA. It's being deployed with openstack_hosts role | 10:02 |
| admin1 | i did ran openstack-ansible setup-hosts.yml -e package_state=latest | 10:03 |
| noonedeadpunk | what's the output of `openssl verify -verbose -CAfile /etc/pki/qemu/ca-cert.pem /etc/pki/qemu/server-cert.pem`? | 10:05 |
| admin1 | CN = hostname, error 20 at 0 depth lookup: unable to get local issuer certificate , error /etc/pki/qemu/server-cert.pem: verification failed | 10:09 |
| noonedeadpunk | and what `openssl x509 -in /etc/pki/qemu/server-cert.pem -noout -text` tells then? | 10:19 |
| noonedeadpunk | As for example, in AIO or any prod compute I have `Subject: CN = aio1` | 10:20 |
| noonedeadpunk | Or maybe it's rather `Issuer` CN rather then certificate CN | 10:20 |
| admin1 | Issuer: C = GB, ST = England, CN = Example Corp Openstack Infrastructure Intermediate CA | 10:23 |
| admin1 | ==- this is diff from what i have set in the variables | 10:23 |
| admin1 | is it possible to force recreate and upgrade/update all the certs in the platform ? | 10:26 |
| admin1 | noonedeadpunk https://gist.githubusercontent.com/a1git/f1f31917d64722638b010e9bcf3d4055/raw/3d8b180cd0f17dd0c6dc1ab180e3e769e15bb521/gistfile1.txt | 10:31 |
| noonedeadpunk | and what's in `openssl x509 -in /etc/pki/qemu/ca-cert.pem -noout -text` then? | 10:32 |
| noonedeadpunk | admin1: but yes, you can regen certs quite easily | 10:33 |
| noonedeadpunk | for that you can supply `-e nova_pki_regen_cert=true` to os-nova-install.yml playbook | 10:33 |
| admin1 | noonedeadpunk - with the last output added: https://gist.githubusercontent.com/a1git/f1f31917d64722638b010e9bcf3d4055/raw/3ed5bed19ad8c2a4a1ad9c97a5e9989ffc334a38/gistfile1.txt | 10:43 |
| admin1 | this one says NCloud which is correct | 10:43 |
| noonedeadpunk | Yeah, so root/intermediate was changed | 10:44 |
| noonedeadpunk | Thus, you indeed need to re-generate certificates | 10:44 |
| noonedeadpunk | Though I'd assume more services should be affected, like rabbit or galera | 10:44 |
| admin1 | can i pass nova_pki_regen_cert=true to setup-hosts and setup-infra ? | 10:45 |
| admin1 | i want to run the regen and fix it globally | 10:45 |
| admin1 | so that in case some services are not mixed, it will fix it also | 10:45 |
| noonedeadpunk | um, no | 10:55 |
| noonedeadpunk | but you can use `pki_regen_cert=true` there. Likely | 10:56 |
| noonedeadpunk | extra-vars has highest prescedence, so it should work | 10:57 |
| admin1 | from time to time, i see this . Lost connection to MySQL server during query .. max connetions is set to 6000 ... total connections on active 17xx .. on the others 2-3 | 15:22 |
| noonedeadpunk | yeah, as haproxy makes only 1 backend as active one | 15:42 |
| noonedeadpunk | though it's weird you see such error - it should not happen. | 15:43 |
| noonedeadpunk | Do you see anything in mariadb logs? | 15:43 |
| mgariepy | or haproxy. | 17:03 |
| noonedeadpunk | yeah, flapping VIP could be a reason | 17:23 |
| mgariepy | or interface reset or something else. | 17:28 |
| admin1 | is it possible to have logs only on error ? | 17:44 |
| admin1 | so instead of normal logs for services, where it logs all lines all the time, every request, maybe a possiblity to reduce it even further and have it log only on errror | 17:44 |
| noonedeadpunk | sure, you can do that. but I think there's no variable in osa to allign log verbosity, so only overrides can be leverage for that as of today | 17:58 |
| opendevreview | Merged openstack/openstack-ansible-repo_server master: Add TLS support to repo_server backends https://review.opendev.org/c/openstack/openstack-ansible-repo_server/+/876429 | 21:03 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Revert "Publish openstack.config_template on galaxy" https://review.opendev.org/c/openstack/ansible-config_template/+/881668 | 21:39 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Bump collection version in metafile https://review.opendev.org/c/openstack/ansible-config_template/+/881929 | 21:40 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Bump collection version in metafile https://review.opendev.org/c/openstack/ansible-config_template/+/881929 | 21:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!