| opendevreview | James Denton proposed openstack/openstack-ansible-os_neutron master: Update metering agent to use interface_driver alias https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/799911 | 03:09 |
|---|---|---|
| noonedeadpunk | mgariepy: I'm not sure you can set notify on role include tbh | 08:38 |
| noonedeadpunk | I think that daemon reload should have happened here https://opendev.org/openstack/ansible-role-systemd_service/src/branch/master/tasks/main.yml#L137-L141 but it feels like we don't satisfy condition here | 08:41 |
| mgariepy | noonedeadpunk, https://paste.openstack.org/show/811124/ | 13:24 |
| mgariepy | it wasn't triggered for some reason. | 13:25 |
| mgariepy | https://opendev.org/openstack/ansible-role-systemd_service/src/branch/master/tasks/main.yml#L131-L132 | 13:26 |
| noonedeadpunk | systemd service changed is handler that doesn't do service reload | 13:26 |
| noonedeadpunk | I guess we need to adjust this condition to make it work https://opendev.org/openstack/ansible-role-systemd_service/src/branch/master/tasks/main.yml#L141 | 13:27 |
| mgariepy | i'll fill the mariadb bug and take a look after | 13:41 |
| mgariepy | https://jira.mariadb.org/browse/MDEV-27068 | 13:53 |
| mgariepy | good news is i can reproduce it reliably on my cloud | 13:55 |
| mgariepy | 4 out of 5 runs hangs so it's kinda nice :D | 13:55 |
| noonedeadpunk | well, it depends on what should be considered as nice hehe | 15:19 |
| noonedeadpunk | but yeah, good that it's reproducale at least | 15:21 |
| mgariepy | lol indeed, but well the point was that it's not only something that is reproducible one time out of 100 .. ;p | 15:24 |
| spatel | noonedeadpunk how do i regenerate certificate for each compute nodes? | 15:55 |
| spatel | i am testing OVN SSL and i want it to generate cert for each compute node | 15:56 |
| spatel | one of my stein cloud throwing this error when trying to run os_neutron playbook on compute nodes - https://paste.opendev.org/show/811140/ | 16:14 |
| spatel | raise SSLError(e, request=request)\nSSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:618) | 16:14 |
| spatel | how do i find how which package and repo causing this issue.. i know its older but trying to fix it | 16:14 |
| mgariepy | do you have the latest ca-cert ? | 16:22 |
| mgariepy | for the letsencrypt stuff ? | 16:22 |
| spatel | are you talking about generating SSL for OVN or my second issue related os_neutron playbook ? | 16:23 |
| mgariepy | second one. | 16:24 |
| spatel | hmm how do i check latest ca-cert? this is centos7 | 16:25 |
| spatel | mgariepy - check this out - https://paste.opendev.org/show/811141/ | 16:27 |
| spatel | same command working on other compute nodes pip2 list but not working on bad compute nodes.. | 16:27 |
| spatel | 120 is the one having issue.. may be i should re-install and try.. | 16:28 |
| mgariepy | all the same pkg / update and so on on the os ? | 16:28 |
| spatel | yes i did match /etc/redhat-release file version and kernel version also | 16:29 |
| spatel | even yum repos are same | 16:29 |
| spatel | oh wait. check this out - https://paste.opendev.org/show/811142/ | 16:31 |
| mgariepy | what is the url that it fails on ? | 16:31 |
| spatel | different version of pip2 | 16:31 |
| mgariepy | hmm weird | 16:32 |
| spatel | yep.. how that happened.. | 16:32 |
| spatel | i can see pip=19.1 in /openstack/venvs/neutron-19.0.0.0rc3.dev6/global-constraints.txt file.. | 16:33 |
| spatel | let me wipe out /openstack/venv/neutron* and re-run playbook | 16:33 |
| spatel | same.. pip 9.0.1 from /openstack/venvs/neutron-19.0.0.0rc3.dev6/lib/python2.7/site-packages (python 2.7) | 16:36 |
| spatel | why its installing 9.0.1 default | 16:36 |
| mgariepy | what is in your pip.conf ? | 16:38 |
| spatel | i don't have pip.conf file anywhere in system | 16:43 |
| spatel | so must be using default options | 16:44 |
| spatel | when i did this it upgraded version - https://paste.opendev.org/show/811143/ | 16:44 |
| jrosser | spatel: https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/814559 | 16:48 |
| spatel | ohhh | 16:50 |
| jrosser | spatel: also if you are testing OVN + SSL against master then you should look at how os_nova is now set up for TLS | 16:50 |
| jrosser | yes, the version of pip/virtualenv/blah on centos 7 that you get with yum won't get an update for its internal bundled version of the ca certs in the python certifi package | 16:51 |
| jrosser | you have to hack up pip.conf to look at the system trust store instead | 16:51 |
| spatel | jrosser thank you so let me try this patch and see | 16:51 |
| jrosser | this assumes that you have used yum to update ca-certificates or whatever it's called | 16:52 |
| spatel | for POC i created /etc/pip.conf file with cert=/etc/pki/tls/certs/ca-bundle.crt and re-running playbook to see | 16:58 |
| spatel | its still installing pip 9.0.1 and throwing SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed | 17:00 |
| spatel | i think i should apply that patch and try.. | 17:01 |
| jrosser | well you should try to use curl or something against whatever the URL is | 17:04 |
| jrosser | check that the system can understand the certificate at all | 17:04 |
| jrosser | it will still install pip 9.0.1 regardless of the patch | 17:04 |
| spatel | jrosser so far no SSL error during curl - https://paste.opendev.org/show/811144/ | 17:08 |
| spatel | how do i find source URL of package which pip installing from | 17:28 |
| spatel | curl https://pypi.org/project/pip/ | 17:39 |
| spatel | works without issue that means SSL not a issue.. | 17:40 |
| spatel | jrosser holy cow! check this out - https://paste.opendev.org/show/811147/ | 17:46 |
| spatel | problem is git.openstack.org ?? | 17:46 |
| jrosser | have you updated the CA certs on your host with yum? | 18:01 |
| spatel | i didn't do anything with ca-cert | 18:02 |
| spatel | i have this version installed - ca-certificates-2019.2.32-76.el7_7.noarch | 18:02 |
| jrosser | from before | 18:02 |
| jrosser | jrosser> this assumes that you have used yum to update ca-certificates or whatever it's called | 18:02 |
| jrosser | that is your issue | 18:02 |
| spatel | hmm how do i undo ? | 18:02 |
| jrosser | that there are new Root CA since 2019, and your host does not have them | 18:02 |
| jrosser | you need to get a newer ca-certificates | 18:02 |
| spatel | let me upgrade and see | 18:03 |
| jrosser | yum <blah> <blah> etc | 18:03 |
| spatel | hmm how do i undo ? | 18:03 |
| spatel | sorry | 18:03 |
| jrosser | undo what? | 18:03 |
| spatel | it was my keyborad up/down arrow :) | 18:04 |
| spatel | yum upgrade ca-certificates | 18:04 |
| spatel | it works.. now i am not seeing SSL error | 18:04 |
| spatel | let me re-run playbook and see | 18:04 |
| jrosser | ok, so with that modification to /etc/pip.conf it should now make pip use that, rather than the ancient one it ships with | 18:04 |
| spatel | let me re-run playbook and see | 18:05 |
| spatel | jrosser now i am seeing different error so look like pip SSL issue has been resolved. Thank you for the help :) | 18:12 |
| jrosser | no worries :) | 18:13 |
| spatel | let me deal with this new issue.. its related to neutron-sriov-nic-agent | 18:13 |
| spatel | https://paste.opendev.org/show/811148/ | 18:14 |
| spatel | looks very odd one but let me debug | 18:14 |
| spatel | hmm missing neutron-keepalived-state-change file | 18:24 |
| spatel | after wipe out everything in /openstack dir it fixed all my issue :) | 18:30 |
| kleini | I regularly stumble over https://github.com/openvswitch/ovs-issues/issues/175 Do you know about any other solutions to fix this except upgrading to glibc from newer ubuntu? | 18:32 |
| mgariepy | spatel, 19.0.0 ? | 18:47 |
| mgariepy | do you track stable/stein at least? | 18:47 |
| spatel | i am planning migrate this to newer cloud using ubunut | 18:48 |
| spatel | I am in process to buy some more servers and create ubuntu based infra and start migrating centos to ubunut | 18:48 |
| spatel | centos7 is dead soon so i don't want to spend any time on it to fix it | 18:49 |
| mgariepy | ok | 18:58 |
| mgariepy | but still there are a few commits between 19.0.0 and stable/stein | 18:59 |
| mgariepy | ;) | 18:59 |
| spatel | i have 900 vms running on this cloud and worried if i try to upgrade and something go wrong :) | 19:01 |
| spatel | i am thinking i can just migrate them one by one to new infra | 19:02 |
| spatel | by the way my rabbitMQ issue has been resolved since i bump but rpc worker count from 1 to 8 for neutron-server | 19:03 |
| spatel | its been 1 week and not a single crash | 19:04 |
| damiandabrowski[m] | hey guys, regarding to ca-certificates I suggest upgrading it everywhere, especially on repo-container as i hit this issue there during T->V upgrade lately ;) | 19:06 |
| damiandabrowski[m] | root@control03-repo-container-29249829:~# git clone -q https://opendev.org/openstack/keystone /tmp/pip-wheel-in1q8u3t/keystone | 19:06 |
| damiandabrowski[m] | fatal: unable to access 'https://opendev.org/openstack/keystone/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: none | 19:06 |
| kleini | was hit by that issue, too. I am looking for a way to newly create the ubuntu-bionic-amd64 base image or to upgrade packages in LXC containers. | 19:07 |
| damiandabrowski[m] | how about: `ansible -m package -a "name=ca-certificates state=latest update_cache=yes only_upgrade=yes" all` ? | 19:09 |
| kleini | yes, that is a good idea | 19:10 |
| spatel | why don't we give this task to OSA to upgrade ca before any work | 19:11 |
| spatel | ca-upgrade never going to break anything as far as i know | 19:11 |
| damiandabrowski[m] | it may be out of scope for OSA, but definitely worth to talk about it ;) | 19:18 |
| damiandabrowski[m] | maybe it's worth adding this package to `_openstack_host_distro_packages` | 19:18 |
| spatel | +1 | 19:28 |
| spatel | who is going to Berlin, Germany for 2022 OpenInfra summit ? | 19:57 |
| damiandabrowski[m] | i'll most likely be there, glad it's super close to Poland :D | 21:54 |
| admin1 | i will be going :) | 23:15 |
| admin1 | met a lot of guys here in berlin 2018 and in barcelona 2016s | 23:19 |
| admin1 | was in line to be the 1st to get signed book by jamesdenton | 23:20 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!