| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/wallaby: Fix inconsistency in haproxy_frontend_raw naming https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/818744 | 08:43 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/victoria: Fix inconsistency in haproxy_frontend_raw naming https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/818745 | 08:43 |
| opendevreview | Lukas Piwowarski proposed openstack/openstack-ansible-os_tempest master: Update URL for documentation of python-tempestconf https://review.opendev.org/c/openstack/openstack-ansible-os_tempest/+/818869 | 09:22 |
| opendevreview | Andrew Bonney proposed openstack/openstack-ansible-os_octavia master: keypair: copy key to deploy host rather than setup host https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/816997 | 10:49 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts stable/wallaby: Add CentOS 8.5 support https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818577 | 12:05 |
| admin1 | is it safe to do a major version upgrade from 20.x tag to 22.x tag or 23.x tag ? | 12:23 |
| noonedeadpunk | well I did upgrade from Train to Victoria in all our regions | 12:38 |
| noonedeadpunk | never tried to 23 though | 12:38 |
| noonedeadpunk | but you should keep in mind that in fact some services (like nova) never tested this path. And starting from W or X explicitly block jumping through releases | 12:39 |
| noonedeadpunk | it was really fine for T->V though | 12:39 |
| opendevreview | Merged openstack/openstack-ansible-os_murano master: [reno] Stop publishing release notes https://review.opendev.org/c/openstack/openstack-ansible-os_murano/+/772039 | 12:43 |
| *** sshnaidm|afk is now known as sshnaidm | 13:24 | |
| opendevreview | Merged openstack/openstack-ansible master: Increase Apache thread limit for keystone https://review.opendev.org/c/openstack/openstack-ansible/+/818674 | 13:32 |
| admin1 | trying to upgrade from 20.x to 21.2.9 and getting "fatal: unable to access 'https://opendev.org/op | 14:43 |
| admin1 | enstack/keystone/': server certificate verification failed. CAfile: /etc/ssl/certs/ca-certificates.crt CRLfile: n | 14:43 |
| admin1 | one\nERROR: Command errored out with exit status " -- | 14:43 |
| admin1 | oops sorry | 14:43 |
| admin1 | was meant to be in 1 line | 14:43 |
| admin1 | is this a known issue .. anything to do with the letsencrypt cert / | 14:43 |
| admin1 | failing on TASK [python_venv_build : Build wheels for the packages to be installed into the venv] | 14:44 |
| noonedeadpunk | `ansible -m package -a "name=ca-certificates state=latest update_cache=yes only_upgrade=yes" all` | 14:44 |
| jrosser | looks like we totally need to add a task for this and backport it | 14:45 |
| admin1 | what exactly is the reason of this error ? some major ca cert expiry ? | 14:46 |
| noonedeadpunk | yep, let's ecnrypt root ca | 14:46 |
| jrosser | https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ | 14:46 |
| admin1 | thanks noonedeadpunk jrosser .. | 14:48 |
| noonedeadpunk | jrosser: I'd say it's worth adding to openstack_hosts, but we have weird package_state for that https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/all.yml#L41 | 14:53 |
| jrosser | well thats odd | 14:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts master: Always upgrade ca-certificates https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/818937 | 15:03 |
| noonedeadpunk | doh clean forgot | 15:04 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:04 |
| opendevmeet | Meeting started Tue Nov 23 15:04:48 2021 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:04 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:04 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:04 |
| noonedeadpunk | #topic rollcall | 15:04 |
| damiandabrowski[m] | hey! | 15:05 |
| mgariepy | hey half here. \o/ | 15:06 |
| noonedeadpunk | \o/ | 15:08 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Do not upgrade packages without upgrades https://review.opendev.org/c/openstack/openstack-ansible/+/818939 | 15:09 |
| noonedeadpunk | #topic office hours | 15:09 |
| noonedeadpunk | I've started writing proxysql role and today it become obvious for me that we won't be able to land it in time for release | 15:10 |
| noonedeadpunk | So I'd say that we do it without proxysql this time unfortunatelly ( | 15:10 |
| noonedeadpunk | As it would involve a lot of changes including db_setup across all roles | 15:11 |
| noonedeadpunk | which we won't be able to test properly during left 2 weeks (or so) | 15:11 |
| noonedeadpunk | and eventually I think we're staying on galera 10.5 as well | 15:11 |
| noonedeadpunk | I don't see how we can easily workaround mysql_upgrade | 15:12 |
| noonedeadpunk | the only thing I guess we can merge would be https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 | 15:12 |
| mgariepy | the issue is creating a small wave tho. | 15:13 |
| mgariepy | for proxysql maybe we cloud backport it in beta when it's done ? | 15:13 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 15:15 |
| noonedeadpunk | well, I will continue working on it and let's see where we will end up | 15:15 |
| noonedeadpunk | as I expect pretty big hardly backportable changes... | 15:16 |
| noonedeadpunk | as eventually we would need to manage mysql users differently | 15:16 |
| noonedeadpunk | but let's see | 15:16 |
| noonedeadpunk | maybe we will be able to backport it in beta before 24.1.0 | 15:17 |
| mgariepy | ping me when you have some patches up for review | 15:17 |
| noonedeadpunk | sure | 15:17 |
| noonedeadpunk | regarding keystone patch - for upgrade to pass we need to land https://review.opendev.org/c/openstack/openstack-ansible/+/818733 | 15:18 |
| noonedeadpunk | and I guess after that I will suggest roles freeze patch for milestone release | 15:18 |
| noonedeadpunk | also we have these 2 wrongly backported patches https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818485 and https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818486 | 15:20 |
| noonedeadpunk | which make symlink from non-existing file .... | 15:20 |
| admin1 | bootstrap, hosts, infra all went ok .. but then on setup openstack, i get python_venv_build : Slurp up the constraints file for later re-deployment .. file not found: /var/www/repo/os-releases/21.2.9/keystone-21.2.9-constraints.txt | 15:21 |
| noonedeadpunk | oh, also Ususri goes to EM jsut in case | 15:21 |
| noonedeadpunk | so we will have last release soon | 15:21 |
| noonedeadpunk | and that would be it | 15:21 |
| noonedeadpunk | admin1: this file should have been created with previous task named `Build constraints file for installation purposes` | 15:23 |
| noonedeadpunk | https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L180 | 15:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Set default for octavia_barbican_enabled https://review.opendev.org/c/openstack/openstack-ansible/+/769179 | 15:24 |
| jrosser | apologies i'm in another meeting | 15:30 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update nariadb to 10.5.13 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/818946 | 15:33 |
| mgariepy | have you seen any other timeout on galera 10.6.5 ? | 15:35 |
| noonedeadpunk | nope, I guess I haven't | 15:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.5.13 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/818946 | 15:36 |
| noonedeadpunk | I've jsut placed processlist to see if we really hit mariadb-upgrade | 15:36 |
| noonedeadpunk | as I have some feeling, that it might be unrelated to us running upgrade.... | 15:37 |
| mgariepy | the logs didn't show up the upgrade was over.. | 15:37 |
| mgariepy | yeah hard to tell. | 15:37 |
| noonedeadpunk | as eventually, now we don't run upgrade but process still stuck | 15:38 |
| noonedeadpunk | huh, but maybe we perform restart or smth like that... | 15:38 |
| noonedeadpunk | before upgrade is finished | 15:38 |
| mgariepy | the upgrade was launche by the debian-start no ? | 15:38 |
| noonedeadpunk | yes | 15:38 |
| noonedeadpunk | but I mean that in https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384/6/tasks/galera_server_setup.yml we already have `when: ansible_facts['os_family'] | lower == 'redhat'` | 15:39 |
| mgariepy | well it's sad that the file created on install if no db is prensent need upgrades.. | 15:39 |
| noonedeadpunk | maybe indeed place our version of debian-start? | 15:40 |
| noonedeadpunk | or just ensure with lineinfile that affecting us line is not present... | 15:40 |
| mgariepy | well is the order fixed ? | 15:41 |
| mgariepy | as i recall it the variable were not in the correct order. | 15:42 |
| noonedeadpunk | yeah, it was | 15:42 |
| noonedeadpunk | which still prevents us from easily overriding command in defaults | 15:42 |
| noonedeadpunk | but returning back to your suggestion - we can mess up with debian-start directly | 15:43 |
| mgariepy | maybe we wait a bit to see how maria will fix it ? | 15:43 |
| noonedeadpunk | yeah, that's probably the wisest choice given | 15:43 |
| mgariepy | the issue is still generating traffic and more ppl are adding to it. | 15:44 |
| noonedeadpunk | oh, there're updates I haven't read actually | 15:48 |
| mgariepy | i do read all of them :) | 15:49 |
| mgariepy | at least i try. | 15:49 |
| noonedeadpunk | `It appears regularly in new or modified tests which are initially written optimistically, assuming it would work, but then we add all sorts of "wait till there is no mysql_upgrade process running [anymore]", and that does the trick.` \ | 15:54 |
| noonedeadpunk | lol | 15:54 |
| noonedeadpunk | mgariepy: have `SELECT GET_LOCK("mariadb-upgrade");` ever worked for you? | 15:57 |
| mgariepy | i havent checked that | 15:57 |
| noonedeadpunk | I just have `ERROR 1582 (42000): Incorrect parameter count in the call to native function 'GET_LOCK'` | 15:57 |
| noonedeadpunk | well, now I was able to get lock while mariadb-upgrade was running | 16:00 |
| mgariepy | get_lock needs anther param .. but well. | 16:00 |
| noonedeadpunk | ah, yes... | 16:02 |
| noonedeadpunk | I misread that I guess | 16:02 |
| noonedeadpunk | #endmeeting | 16:03 |
| opendevmeet | Meeting ended Tue Nov 23 16:03:00 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:03 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-11-23-15.04.html | 16:03 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-11-23-15.04.txt | 16:03 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2021/openstack_ansible_meeting.2021-11-23-15.04.log.html | 16:03 |
| noonedeadpunk | hm, maybe we can check for /var/lib/mysql/mysql_upgrade_info indeed.... | 16:04 |
| noonedeadpunk | it's created only once mysql_upgrade is fully finished | 16:05 |
| mgariepy | so we need to check until the content is the version we are installing ? | 16:06 |
| mgariepy | # cat /var/lib/mysql/mysql_upgrade_info | 16:07 |
| mgariepy | 10.3.25-MariaDB | 16:07 |
| noonedeadpunk | perfectly - yes | 16:07 |
| mgariepy | since if it's an upgrade it will exist. | 16:07 |
| noonedeadpunk | it's not that hard to parse though | 16:07 |
| mgariepy | indeed. | 16:08 |
| mgariepy | how is it reacting on a cluster ? | 16:08 |
| mgariepy | you only need to run it on 1 node ? | 16:09 |
| noonedeadpunk | yep, you need it only once | 16:09 |
| noonedeadpunk | oh, well | 16:09 |
| mgariepy | how are we going to block access to the cluster ? | 16:09 |
| noonedeadpunk | but we already do with disabling haproxy backend? | 16:09 |
| mgariepy | don't we only disable one ? | 16:10 |
| mgariepy | we only write on one but it's happy to receive writes to the other nodes.. | 16:10 |
| mgariepy | git cloning on a 56kbs modem is no fun. :/ | 16:11 |
| noonedeadpunk | yeah, I dunno | 16:11 |
| noonedeadpunk | oh, wait_for module can even search regexp in provided file! | 16:12 |
| mgariepy | from what i did catch while talking with monty you should take your DB offline while performing the mariadb_upgrade | 16:13 |
| noonedeadpunk | which is close to impossible (or at least unrealistic) to do by end users | 16:13 |
| mgariepy | it depends. | 16:13 |
| mgariepy | how long does it take ? | 16:14 |
| mgariepy | lol | 16:14 |
| noonedeadpunk | so eventually what they told about upgrade should be part of server startup is valid | 16:14 |
| noonedeadpunk | I know close to nobody whould would startup server os some reserve socket after update to perform mariadb_upgrade | 16:15 |
| noonedeadpunk | and tbh I just discovered that it's must thing to do | 16:15 |
| mgariepy | we did had a lock when the upgrade process was running. | 16:15 |
| mgariepy | but it was for creating users. | 16:16 |
| mgariepy | which might be a bit different. | 16:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 16:21 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 16:22 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_magnum master: Drop config_template from role https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/818999 | 17:29 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_trove master: Drop config_template from role https://review.opendev.org/c/openstack/openstack-ansible-os_trove/+/819000 | 17:30 |
| opendevreview | Merged openstack/openstack-ansible stable/wallaby: Increase Apache thread limit for keystone https://review.opendev.org/c/openstack/openstack-ansible/+/818733 | 17:52 |
| admin1 | strange .. horizon fails with : In template /openstack/venvs/horizon-22.3.3/lib/python3.6/site-packages/horizon/templates/horizon/common/_form_field.html, error at line 32 too many values to unpack (expected 2) .. line32 is {{ field|add_bootstrap_class }} | 17:59 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-config_template master: Move config_template to plugins/action https://review.opendev.org/c/openstack/ansible-config_template/+/819010 | 18:37 |
| noonedeadpunk | I've messed up with config_template | 18:46 |
| noonedeadpunk | I bet I was testing it but well... Collection didn't work previous way ( | 18:46 |
| noonedeadpunk | I think I was using plugin from ceph-ansible while testing ( | 18:46 |
| noonedeadpunk | would be great if we could review it | 18:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Update functional test requirements url https://review.opendev.org/c/openstack/openstack-ansible/+/819015 | 18:54 |
| noonedeadpunk | thankfully that plugin way was not broken so... | 18:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 18:57 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 19:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 19:17 |
| noonedeadpunk | from the mariadb bug comments it feels they will only implement smth to 10.7 tbh... | 19:28 |
| noonedeadpunk | but it's already rc1, so might be even 10.8 or smth... | 19:28 |
| mgariepy | yeah | 19:28 |
| mgariepy | maybe | 19:29 |
| spatel | In rabbitMQ management GUI queue tab is empty, do i need to give permission to monitoring to have access of queues? | 19:37 |
| spatel | nevermind, created special user with full_access permission | 19:46 |
| jamesdenton_alt | spatel what are you using to benchmark your sriov and dpdk deployments? | 20:12 |
| *** jamesdenton_alt is now known as jamesdenton | 20:12 | |
| spatel | i had poor performance with dpdk (very poor) | 20:13 |
| spatel | i am not using dpdk in prod | 20:13 |
| spatel | only using sriov | 20:13 |
| jamesdenton | ok | 20:13 |
| spatel | I am testing dpdk in lab currently but without OVS | 20:13 |
| jamesdenton | do you have any sort of benchmark process besides iperf? are you using trex? | 20:13 |
| spatel | I am planning to use l2fwd and pktgen | 20:14 |
| spatel | both DPDK application and good for doing DPDK perf test | 20:14 |
| spatel | run l2fwd inside VM which will do packet forwarding and pktgen outside somewhere to pump packet | 20:15 |
| spatel | In past i used Trex | 20:15 |
| spatel | jamesdenton here is the demo of my lab when i did that last year - https://asciinema.org/a/376367 | 20:16 |
| jamesdenton | very good, thank you | 20:17 |
| spatel | as you can see i hit 1.4Mpps with SRIOV | 20:18 |
| jamesdenton | yes | 20:18 |
| jamesdenton | let me run that same test and see what i get in this ASAP2 env. brb | 20:18 |
| spatel | This is DPDK test result with Trex - https://asciinema.org/a/376378 | 20:19 |
| spatel | Trex is very advance and it can use complex scenario http payload or many more thing. (i really like Trex but downside of it is complexity to setup) | 20:23 |
| spatel | Pktgen is very simple which can just generate packets in high speed but doesn't support complex scenario like Trex | 20:24 |
| jamesdenton | do you happen to have your trex_cfg files? | 20:25 |
| spatel | no but i would like to blog that at sometime. | 20:26 |
| spatel | I have some useful link which i used that to setup my lab - https://the.vpn.center/2017/01/up-and-rawring-with-trex-ciscos-open.html | 20:27 |
| jamesdenton | oh yeah i remember that one | 20:28 |
| spatel | other one is here - https://sudonull.com/post/79789-Stroking-a-lizard-or-network-load-testing-with-cisco-trex | 20:29 |
| spatel | jamesdenton Please Please Please disable port security on openstack vm :) | 20:30 |
| jamesdenton | oh yeah, it is | 20:30 |
| spatel | i wasted 2 days to figure that our :) | 20:30 |
| spatel | Trex installation comes with lots of scenario so all you need to take one and just modify and run it - https://github.com/cisco-system-traffic-generator/trex-core/blob/master/scripts/stl/udp_1pkt_simple.py | 20:34 |
| spatel | Please blog it :) which i totally missed during my setup | 20:35 |
| jamesdenton | would love to blog it - a lot of whats out there is outdated. i just need to make sure it's right and my numbers are legit | 20:36 |
| spatel | you run your test and give me your config and i will try same test to see numbers are same or not | 20:40 |
| spatel | I am going to try l2fwd with pktgen | 20:40 |
| jamesdenton | i'll get some stuff together tomorow | 20:47 |
| spatel | cool! | 20:47 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!