| noonedeadpunk | I wonder if we try to restart mariadb now _before_ upgrade is finished | 07:59 |
|---|---|---|
| noonedeadpunk | Feels like we need to find the way to check for upgrade to finish with debian-start.... | 08:00 |
| *** akahat|rover is now known as akahat|lunch | 08:36 | |
| *** akahat|lunch is now known as akahat|rover | 09:59 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Add CentOS 8.5 support https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818532 | 10:46 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add security headers to HAProxy Horizon service https://review.opendev.org/c/openstack/openstack-ansible/+/818533 | 10:50 |
| rohit02 | hi tem,deploying OSA victoria on centos 8 facing issue in ansiblehost: FAILED - RETRYING: Ensure that the LXC cache has been prepared (58 retries left).what could be the reason to be failed at this tasks | 11:24 |
| noonedeadpunk | rohit02: I believe you use centos, not centos stream? | 11:52 |
| rohit02 | noonedeadpunk:yes centos 8 | 11:58 |
| noonedeadpunk | you need this https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818532 | 11:59 |
| noonedeadpunk | eventually centos doesn't reliable differentiated between stream and non-stream so we did some assumptions that are broken now with 8.5 release | 11:59 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add security headers to HAProxy Horizon service https://review.opendev.org/c/openstack/openstack-ansible/+/818533 | 12:01 |
| rohit02 | noonedeadpunk: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818532 not getting this part could plzz send me the steps | 12:02 |
| noonedeadpunk | cd /etc/ansible/roles/lxc_hosts/vars; ln -s centos-8.3.yml centos-8.5.yml | 12:03 |
| rohit02 | noonedeadpunk: centos-8.3.yml not present does centos-8.4 works for me for the ln command | 12:15 |
| rohit02 | noonedeadpunk: and that file[centos-8.4.yml] is empty | 12:17 |
| noonedeadpunk | hm... I'm looking at the repo, and it feels like it should have been a symlink that is broken.... | 12:18 |
| noonedeadpunk | what if you try to drop it? | 12:18 |
| noonedeadpunk | what releaser are you running btw? | 12:18 |
| noonedeadpunk | * release | 12:18 |
| rohit02 | noonedeadpunk: [root@controller ~]# cat /etc/redhat-release | 12:20 |
| rohit02 | CentOS Linux release 8.5.2111 | 12:20 |
| noonedeadpunk | osa release, sorry :) | 12:20 |
| rohit02 | noonedeadpunk: any alternative solution to proceed further | 12:22 |
| noonedeadpunk | are you running victoria or ussuri? | 12:22 |
| rohit02 | victoria | 12:22 |
| noonedeadpunk | hm... | 12:24 |
| noonedeadpunk | and what actually error you get after all are failed? | 12:24 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/ussuri: Revert "Add CentOS 8.4 support" https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818485 | 12:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/victoria: Revert "Add CentOS 8.4 support" https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818486 | 12:25 |
| noonedeadpunk | damn. I cherry-picked wrong thing I beleive | 12:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/victoria: Add CentOS 8.4 support https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/818487 | 12:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/ussuri: Add CentOS 8.4 support https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/818488 | 12:28 |
| noonedeadpunk | rohit02: can you pls try out https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/818487 ? | 12:29 |
| rohit02 | noonedeadpunk: yes will try and let you know......thanx for your help | 12:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-galera_server master: Update mariadb to 10.6.5 https://review.opendev.org/c/openstack/openstack-ansible-galera_server/+/817384 | 12:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Drop Nginx webserver support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 | 13:04 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Drop Nginx webserver support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 | 13:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Drop Nginx webserver support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 | 13:13 |
| mgariepy | noonedeadpunk, <monty__> mysql_upgrade should normally only be done when one updates the MariaDB package. As part of the process MariadB server should probably be used with another socket and then restarted with the new socket. That would solve the problem as no one would be able to connect until upgrade is done | 13:32 |
| mgariepy | i'm at around 200 reinstallation of mariadb without any issue on my testbed. | 13:32 |
| mgariepy | https://paste.opendev.org/show/811193/ | 13:38 |
| noonedeadpunk | well, Michael is great actually... | 13:56 |
| noonedeadpunk | he's always super helpful and solves stuff :) | 13:57 |
| mgariepy | yep :D | 13:57 |
| noonedeadpunk | I was kind of surprised first time that he's still hanging around and do stuff on his own | 13:58 |
| noonedeadpunk | in the meanwhile - upgrade does create tables, so we can check if they do exists and assume that migration is done if they are... | 13:59 |
| noonedeadpunk | I know it's not great way to do that... | 13:59 |
| mgariepy | or we divert the debian-start not to run upgradE? | 14:00 |
| mgariepy | hmm. | 14:00 |
| mgariepy | it's annoying, for greenfield deploy it doesnt really matter. but if you upgarde a prod system you might end with a DB that is dead locked. | 14:01 |
| noonedeadpunk | so you mean to set in /etc/default/mariadb empty MYUPGRADE? | 14:01 |
| noonedeadpunk | then upgrade jsut won't run? | 14:01 |
| mgariepy | yeah it would be run by the task later in the playnook | 14:02 |
| mgariepy | playbook. | 14:02 |
| noonedeadpunk | and eventually we already ship defaults file... | 14:02 |
| noonedeadpunk | looking on https://paste.opendev.org/show/811194/ you can't really make it empty... but can provide smth meaningless | 14:05 |
| mgariepy | otherwise we can edit debian-start not to call it. | 14:08 |
| noonedeadpunk | nah, it's more nasty hack I guess :) | 14:08 |
| mgariepy | can we dpkg-divert the file in /etc ? | 14:08 |
| mgariepy | we divert the file, restart the db, upgrade it then revert-back the file ? | 14:08 |
| mgariepy | or we can also wait to see how the debian pkg will be fixed | 14:09 |
| mgariepy | unless your really want to ship mariadb 10.6.5 with X ? | 14:10 |
| mgariepy | i'll tell infra to release the vm now. | 14:11 |
| mgariepy | unless you think i should keep it a bit longer? | 14:11 |
| noonedeadpunk | nah, I think you can release now... | 14:12 |
| noonedeadpunk | Eventually for several minutes I thought that you can just define `MYUPGRADE` in /etc/default/mariadb but just realized that it won't be respected | 14:13 |
| noonedeadpunk | let's at least upgrade rabbitmq) https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/817380 | 14:14 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add security headers to HAProxy Horizon service https://review.opendev.org/c/openstack/openstack-ansible/+/818533 | 14:14 |
| noonedeadpunk | and we also do need https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818532 | 14:15 |
| mgariepy | noonedeadpunk, i guess for the debian var issue you will need to create a new issue. | 14:23 |
| noonedeadpunk | wonder wtf is going here https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_9f9/817390/5/check/openstack-ansible-deploy-aio_metal-debian-bullseye/9f977d0/logs/ara-report/results/1201.html | 14:30 |
| noonedeadpunk | o_O [Fri Nov 19 13:59:04.059426 2021] [mpm_event:error] [pid 65708:tid 139901775236416] AH03490: scoreboard is full, not at MaxRequestWorkers.Increase ServerLimit. | 14:30 |
| noonedeadpunk | feels like we have weird defaults set for apache | 14:48 |
| noonedeadpunk | in terms of ThreadsPerChild/MaxRequestWorkers/ThreadLimit | 14:48 |
| rohit02 | noonedeadpunk: the fix you have provided for centos 8.4 support not works for me....centos-8.yml is there but its empty | 14:52 |
| noonedeadpunk | it's supposed to be redhat-8, not centos-8 | 14:52 |
| noonedeadpunk | and it should not be empty | 14:53 |
| rohit02 | and redhat-8.yml file is not there | 14:53 |
| noonedeadpunk | are we talking about openstack_hosts role now? | 14:53 |
| rohit02 | my deployment failed at https://paste.opendev.org/show/811198/ | 14:55 |
| rohit02 | redhat-8.yml file should be in this directory cd /etc/ansible/roles/lxc_hosts/vars right? | 14:57 |
| noonedeadpunk | no | 14:57 |
| noonedeadpunk | please check https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/818487 | 14:57 |
| rohit02 | noonedeadpunk: have u chk this https://paste.opendev.org/show/811198/ | 15:00 |
| opendevreview | Merged openstack/ansible-role-python_venv_build stable/train: Revert "Set centos-7 jobs to non voting" https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/817221 | 15:01 |
| noonedeadpunk | it doesn't say anything about failure reason | 15:01 |
| noonedeadpunk | image build log should be stored in /var/log/lxc-cache-prep-commands.log | 15:02 |
| rohit02 | fix already in stable/victoria branch so i dont need to add fix locally right? | 15:04 |
| rohit02 | so i can rebuild the complete openstack | 15:04 |
| noonedeadpunk | I'm not sure since I don't see what error do you hit | 15:10 |
| noonedeadpunk | Because reall error are mentioned only in log file | 15:10 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Drop Nginx webserver support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 | 15:52 |
| noonedeadpunk | rohit02: also, I guess you know that CentOS 8 will enter EOL on 1st Januray 2022? | 15:54 |
| opendevreview | Merged openstack/openstack-ansible-lxc_hosts master: Add CentOS 8.5 support https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818532 | 15:56 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_keystone master: Drop Nginx webserver support https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/817390 | 15:58 |
| rohit02 | noonedeadpunk: yes but for our product testing will need that.....what i observed now in lxc-cache-prep-commands log dnf command itself in hang state for a while | 16:01 |
| rohit02 | [root@controller log]# tail -f lxc-cache-prep-commands.log | 16:01 |
| rohit02 | (195/197): zlib-1.2.11-17.el8.x86_64.rpm 2.8 MB/s | 102 kB 00:00 | 16:01 |
| rohit02 | (196/197): yum-4.7.0-4.el8.noarch.rpm 2.3 MB/s | 205 kB 00:00 | 16:01 |
| rohit02 | (197/197): util-linux-2.32.1-28.el8.x86_64.rpm 6.4 MB/s | 2.5 MB 00:00 | 16:01 |
| rohit02 | -------------------------------------------------------------------------------- | 16:01 |
| rohit02 | Total 12 MB/s | 116 MB 00:09 | 16:01 |
| rohit02 | Running transaction check | 16:01 |
| rohit02 | Transaction check succeeded. | 16:01 |
| rohit02 | Running transaction test | 16:02 |
| rohit02 | Transaction test succeeded. | 16:02 |
| rohit02 | Running transaction | 16:02 |
| noonedeadpunk | hm | 16:10 |
| noonedeadpunk | what if you jsut increase timeout? | 16:10 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-lxc_hosts/src/branch/master/defaults/main.yml#L190-L194 | 16:11 |
| noonedeadpunk | maybe you have jsut slow machine or smth like that... | 16:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts stable/wallaby: Add CentOS 8.5 support https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/818577 | 16:15 |
| noonedeadpunk | mgariepy: oh, that _is_ interesting https://zuul.opendev.org/t/openstack/build/15e624950e2743cd95f2a6e460605b18/logs | 16:19 |
| noonedeadpunk | if you look at output, it seems that `/var/lib/mysql/sys/waits_global_by_latency.frm` is present as `Wait for MySQL Upgrade to finish` passes as OK | 16:19 |
| noonedeadpunk | BUT | 16:19 |
| noonedeadpunk | in https://zuul.opendev.org/t/openstack/build/15e624950e2743cd95f2a6e460605b18/log/logs/openstack/aio1_galera_container-3c8aeb78/mariadb.service.journal-15-47-19.log.txt this is not reflected... | 16:20 |
| noonedeadpunk | but I haven't checked actually if this table present even without that upgrade, damn.... | 16:20 |
| noonedeadpunk | so disregard... | 16:20 |
| mgariepy | lol | 16:21 |
| mgariepy | the mysql_upgrade should be completed tho. | 16:21 |
| mgariepy | brb . need to reboot my laptop. | 16:24 |
| mgariepy | noonedeadpunk, could we have a anto-hold on timeout on this job ? | 16:41 |
| mgariepy | or.. can we add mysql show processlist to the logs before we exit ? | 16:42 |
| spatel | jamesdenton around? | 17:11 |
| jamesdenton | yessssss | 17:11 |
| spatel | I am setting up SSL for OVN and having some issue, my issue is NB trying to connect to SB on same server using SSL | 17:15 |
| spatel | i believe that should be Unix socket correct? on same box we don't need them to use SSL to communicate | 17:16 |
| spatel | does OSA default use TCP port for NB/SB communication? | 17:16 |
| jamesdenton | i believe tcp is used, yes | 17:18 |
| spatel | hmm.. | 17:18 |
| jamesdenton | but it's overridable if you want to try a new value | 17:18 |
| jamesdenton | neutron_ovn_sb_connection i think | 17:18 |
| spatel | lets say i want to change it to unix socket right now just test my SSL issue.. what is the command? | 17:19 |
| spatel | https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/providers/setup_ovs_ovn.yml#L29 | 17:21 |
| jamesdenton | i don't know what the socket file name would be | 17:21 |
| spatel | oh that that config is for ovn-controller | 17:21 |
| jrosser | isnt that a list? | 17:22 |
| jamesdenton | you could try unix:/var/run/ovn/ovnsb_db.sock but i'm not sure that is the preferred approach | 17:23 |
| jamesdenton | and yeah, neutron_ovn_sb_connection is a list | 17:24 |
| jrosser | spatel: what actual issue do you have? seems unusal to treat the local connection specially when it's a cluster anyway | 17:24 |
| jamesdenton | though these docs do mention a socket: https://docs.openstack.org/neutron/latest/configuration/ovn.html | 17:24 |
| spatel | jamesdenton jrosser as soon as i turn on SSL for OVN i started seeing error in nb logs that not able to communicate with sb - https://paste.opendev.org/show/811202/ | 17:26 |
| jamesdenton | how did you enable it? | 17:26 |
| spatel | https://github.com/ovn-org/ovn-kubernetes/blob/master/docs/INSTALL.SSL.md | 17:26 |
| spatel | currently i am following OVN doc for PKI just to understand process. later we will deal with osa/pki | 17:27 |
| jamesdenton | 6641:10.62.7.252: bind: Address already in use | 17:27 |
| jamesdenton | so maybe try a different port? or clear out the existing entires | 17:27 |
| jamesdenton | *entries | 17:27 |
| spatel | hmm | 17:28 |
| jrosser | also "received JSON-RPC data on SSL channel" <- one end talks not-ssl, the other expects ssl | 17:28 |
| spatel | let me first fix port and then see.. | 17:31 |
| jamesdenton | jrosser have you looked at ASAP^2 recently? | 18:13 |
| jamesdenton | (and is it still a thing) | 18:13 |
| jrosser | not for a long time | 18:13 |
| jrosser | the things we need off the mellanox hardware currently only work for SRIOV | 18:14 |
| jrosser | you can't combine them with e-switch | 18:14 |
| jamesdenton | i see | 18:14 |
| jrosser | focus for us is currently on bluefield | 18:15 |
| jrosser | i think that asap^2 is still certainly a thing on the datasheets for cx-6 | 18:16 |
| jamesdenton | i found some recent OSP 16.1 docs that mention it, too | 18:17 |
| jamesdenton | wondering if ovs firewall (security groups) is supposed or not. It wasn't at one time | 18:17 |
| jrosser | might be worth revisiting because seems the nova/neutron understanding of smartnic in general seems really much better than when we talked about this before | 18:19 |
| jamesdenton | indeed | 18:19 |
| jamesdenton | given spatel's foray into sriov/dpdk, could be worthwhile for him, too | 18:19 |
| jrosser | have you seen these too? https://review.opendev.org/q/topic:%22integration-with-off-path-network-backends%22+(status:open%20OR%20status:merged) | 18:19 |
| jrosser | thats for where the NIC is it's own linux system running effectively OVS in between the SFP and PCIe slot | 18:20 |
| jamesdenton | *mind blown* | 18:20 |
| jrosser | so ironic++ for example | 18:20 |
| jrosser | there was also a talk at nvidia GTC last week where some university had implemented per-tenant-network crypto on a bluefield smartnic | 18:22 |
| jamesdenton | time to dust off the lab | 18:24 |
| spatel | jamesdenton sure i can take that :) | 18:39 |
| spatel | jamesdenton jrosser as soon as i remove OVN clustering and just use single node everything works with SSL (i believe i need to debug cluster SSL part) | 18:40 |
| jamesdenton | good to know | 18:41 |
| jrosser | are you sure the connection string is specifying ssl for each cluster member? | 18:41 |
| spatel | first i thought cluster doesn't need SSL communication because they have totally different ports to communicate 6643 and 6644. but somehow it doesn't like when i turn on SSL. (at this point i don't have much data but i will keep poking to find a workaround) | 18:43 |
| spatel | jamesdenton i am talking to one of INTEL developer to find issue related DPDK performance. He claimed that it should work and give you good boost in performance even with virtio. | 18:46 |
| spatel | I have zoomed with him and he was able to generate 7million packet using dpdk on single CPU core. now next step is to try with OVS switch and see where is the bottleneck | 18:47 |
| jamesdenton | oh, awesome | 18:49 |
| spatel | but truly speaking i don't like DPDK, lots of piece and hard to troubleshoot :( | 18:50 |
| spatel | jamesdenton i tried this but getting that error not sure from where its coming - https://paste.opendev.org/show/811203/ | 20:49 |
| spatel | https://github.com/openshift/ovn-kubernetes/blob/master/docs/OVN-NORTHD.SSL.md | 20:51 |
| spatel | jamesdenton damn it figure it out :) | 21:05 |
| spatel | trust me OVN document is all over the place and not very clear... | 21:05 |
| spatel | Let me put this on my blog, next i am going to see how we can fit this lego pieces together using OSA/PKI | 21:08 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!