| damiandabrowski[m] | most of the people are on their way home from openinfra summit, but normally yes, we are active here every day ;) | 06:36 |
|---|---|---|
| admin1 | hi | 07:12 |
| Mouaa | hi | 07:14 |
| admin1 | quick checking .. if anyone knows when/how limits added on existing flavors are activated | 07:15 |
| jrosser_ | admin1: I’m not sure what you mean - do you have an example? | 07:16 |
| admin1 | found my answer here: https://docs.openstack.org/nova/xena/admin/flavors.html .. adding extra specs like limits need a cold migration | 07:19 |
| Mouaa | Hi. When upgrading from ubuntu 18.04 to 20.04 under ussuri, I noticed a lot of problems with the keealived check scripts (vrrp_script haproxy_check_script and vrrp_script pingable_check_script). Script pingable_check_script now returning 2 and VRRP_Script(haproxy_check_script) failed (exited with status 1). | 07:51 |
| Mouaa | The only related bug found is https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1806004 related to "script_user root" option in the keepalived config, présent in our config. When I run the scripts manually as root in a shell, I don't reproduce these errors. | 07:51 |
| Mouaa | The only solution I found was to modify these scripts, script "/bin/kill -0 `cat /var/run/haproxy.pid`" changed to script "/usr/bin/killall -0 haproxy" and remove 1>&2 redirect from vrrp_script pingable_check_script. | 07:52 |
| Mouaa | Has anyone encountered a similar problem? I prefer to ask before raising a bug... | 07:52 |
| *** ysandeep|out is now known as ysandeep | 08:05 | |
| andrewbonney | Mouaa: do you override the keepalived_scripts variable at all? This sounds familiar and I can see changes such as https://github.com/openstack/openstack-ansible/commit/f91b88078a71525608664df6b4559cd7908117b2 which are important when moving from keepalived v1.x to v2.x | 08:31 |
| Mouaa | @andrewbonney This problem appeared during the OS upgrade of the nodes, not while redeploying therefore by replaying the ussuri playbooks, so my fix was directly made in the keepalived conf on the controllers, not actually by overriding the keepalived vars. Indeed, OS upgrades have moved from Keepalived v1.3.9 to v2.0.19. | 08:58 |
| andrewbonney | Had you set the keepalived_scripts variable previously though? So that it differs from the variable settings provided in the Ussuri OSA release? | 08:59 |
| *** dviroel_ is now known as dviroel | 11:30 | |
| Mouaa | @andrewbonney, it's seems you're right, I asked the initial committer why he overridden the behavior of check_scripts | 11:39 |
| mgariepy | https://github.com/openstack/openstack-ansible/commit/d6ca5af79ec10d1a6c2cd98b27ca7e755abfc9b5 | 12:23 |
| mgariepy | Mouaa, andrewbonney i think it's missing for U ? | 12:23 |
| mgariepy | or not.. | 12:24 |
| mgariepy | are you on the latest Ussuri ? | 12:24 |
| mgariepy | Mouaa, ^^ | 12:25 |
| Mouaa | This platfrom are in place from stein version, actually git on "ussuri-em" for the migration from train to ussuri, actually i'm upgrading the ubuntu OS. Install method : distro | 12:32 |
| mgariepy | what does the global_def in keepalived.conf file looks like ? | 12:34 |
| Mouaa | Thanks for the help ! global_defs { enable_script_security script_user root } | 12:36 |
| mgariepy | `which kill` on the controller? | 12:41 |
| mgariepy | i thing the path is not quite correct.. | 12:41 |
| mgariepy | hmm. both should exist :/ | 12:42 |
| mgariepy | can you try to add the full path for cat in the script ? | 12:45 |
| mgariepy | doing killall if there are other haproxy process on the server might cause some other issues. | 12:47 |
| Mouaa | @mgariepy, Good idea, I will try to put full path for cat | 12:49 |
| Mouaa | I think to reproduce https://github.com/openstack/openstack-ansible/commit/f91b88078a71525608664df6b4559cd7908117b2 manually on the controllers (without playbook), after upgrading the OS and LXC containers, when I execute the deployments to upgrade from ussuri to victoria, everything will fit in order... | 12:49 |
| Mouaa | full path for cat didn't change nothing : Script `haproxy_check_script` now returning 1 | 12:52 |
| mgariepy | :/ that's weird. | 12:52 |
| mgariepy | how did you do the upgrade ? os upgrade on one controller? | 12:53 |
| mgariepy | then rerun the playbook for it? | 12:53 |
| mgariepy | does /var/run/haproxy.pid exist ? | 12:55 |
| Mouaa | I upgraded ubuntu 18.04 to 20.04 sequentially (1 by 1) each controller, looking at the status of each service (keepalived, haproxy, ...) between each and restarting each service if out of sync | 12:59 |
| mgariepy | can you run the haproxy playbook ? | 13:00 |
| Mouaa | I haven't rerun the playbooks yet, I was thinking of upgrading all the nodes, destroying the LXCs and then replaying the playbooks | 13:01 |
| mgariepy | usually i do controllers one by one. and redeploy services on the reinstalled node. | 13:02 |
| *** ysandeep is now known as ysandeep|break | 13:02 | |
| Mouaa | OK, i'll rerun the haproxy-install.yml for testing | 13:04 |
| mgariepy | something like that : https://etherpad.opendev.org/p/osa-newton-xenial-upgrade or https://etherpad.opendev.org/p/osa-rocky-bionic-upgrade | 13:04 |
| Mouaa | The only doc for upgrade reference is https://docs.openstack.org/openstack-ansible/victoria/admin/upgrades/distribution-upgrades.html, so not adapted for me because distro install method, so i'm trying on the DEV (nested openstack in openstack PF) | 13:07 |
| mgariepy | why did you go with distro install, out of curiosity ? | 13:08 |
| Mouaa | no idea, I get the project set up in 2018, I don't have all the info... | 13:10 |
| Mouaa | @mgariepy I spent a lot of time making our dev PF functional (Openstack DEV nested on opensatck PROD: vxlan in vxlan, physical mtu 1500, tenant network and instance ports in DEV to 1400). I took some time to configure everything to have something functional thing. And since the keepalived heartbeats use vxlan, I directed my research on that | 13:27 |
| mgariepy | yeah it's kinda a mess when you need to mess with mtu and such. | 13:28 |
| *** ysandeep|break is now known as ysandeep | 13:44 | |
| goldenfri | Hi, is anyone around that could some advice? I've been running a rocky install of OSA for years, but recently something happened and the openstack-ansible command no longer works, I get an Unexpected Exception Error | 14:21 |
| goldenfri | TypeError: expected string or buffer | 14:22 |
| jrosser_ | goldenfri: you can copy the error to paste.opendev.org perhaps? | 14:52 |
| goldenfri | here is is, thanks! https://paste.opendev.org/show/bFmFQarBe89z8tQ76DEL/ | 14:55 |
| *** dviroel is now known as dviroel|lunch | 15:02 | |
| *** ysandeep is now known as ysandeep|out | 15:21 | |
| jrosser_ | goldenfri: i'm not sure really - it looks like something invalid in your inventory | 15:22 |
| jrosser_ | i think there is a tar archive kept in /etc/openstack_deploy of the inventory history so you could check if anything has changed there | 15:22 |
| jrosser_ | or hopefully you have that directory in git to track things :) | 15:23 |
| jrosser_ | also you could try to see if `ansible-inventory` will parse the inventory | 15:23 |
| goldenfri | Thanks @jrosser_ that at least give me some place to start looking | 15:28 |
| Mouaa | mgariepy FYI, yours advices correct my keepalived/haproxy problems | 15:29 |
| mgariepy | cool | 15:32 |
| *** dviroel|lunch is now known as dviroel | 16:13 | |
| *** dviroel is now known as dviroel|afk | 19:20 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!