| kevko | I was trying upgrade from zed to antelope as we are going to do it this week and I found that rabbitmq nodes are restarted right after configure phase and then one by one ...this is bad , isn't it ? Log in openstack-discuss ...is there some hidden reason or what ? | 06:46 |
|---|---|---|
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Add configuration for NTP server in Ironic Inspector DHCP server https://review.opendev.org/c/openstack/kolla-ansible/+/930170 | 06:47 |
| SvenKieske | kevko: how exactly did you run the command to upgrade? so direct upgrade from zed to antelope with k-a antelope code? or when exactly does the error occur? | 07:05 |
| SvenKieske | kevko: restart behaviour was changed in https://review.opendev.org/c/openstack/kolla-ansible/+/886485 wondering if we ever tested that :D | 07:10 |
| SvenKieske | that being said, you are the first to complain | 07:10 |
| Vii | Welcome RMQ 4.x ;) https://github.com/rabbitmq/rabbitmq-server/releases/tag/v4.0.2 | 07:14 |
| kevko | SvenKieske kolla-ansible upgrade -t rabbitmq --diff | 07:17 |
| kevko | SvenKieske: sometimes I have a feeling that I am the one who is testing 𤣠| 07:18 |
| SvenKieske | I mean batch size is 33%, just looked up the definition on ansible docs, seems it should work as advertised. | 07:19 |
| ayhanilhan_ | Hi, I have 5 controllers and I want to temporarily remove and add these controllers for OS upgrade. After the "multinode stop controller-01" command, can I proceed with the "multinode deploy --limit controller-01" command instead of the "multinode deploy --limit control" command? I don't want Rabbitmq and MariaDB restarts on the other controllers. | 07:21 |
| kevko | SvenKieske: But this doesnât make sense at all. First, the entire RabbitMQ cluster is restarted... and then it's restarted one by one? Whatâs the point of a 33% serial if RabbitMQ could already be broken after the previous step (in the step where the handlers are triggered and all containers are restarted at once)? | 07:25 |
| SvenKieske | did you check the code? afaik this just triggers the main rmq handler when checking the container dimensions etc. | 07:26 |
| kevko | SvenKieske: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_44c/930170/1/check/kolla-ansible-ubuntu-upgrade/44cc5e9/primary/logs/ansible/upgrade | 07:28 |
| kevko | SvenKieske: ^^ this is regular upgrade also in CI ... | 07:29 |
| opendevreview | Roman KrÄek proposed openstack/kolla master: Change copy-cacerts behaviour https://review.opendev.org/c/openstack/kolla/+/926882 | 07:36 |
| opendevreview | Roman KrÄek proposed openstack/kolla-ansible master: Don't notify handlers during copy-cert https://review.opendev.org/c/openstack/kolla-ansible/+/924651 | 07:40 |
| kevko | SvenKieske: so ? | 07:43 |
| SvenKieske | I'm not sure what you expect, actually. The log you linked is from a single node installation. sure it reboots "all" services, even with serial: 33% as is documented. I guess you are expecting it to not restart the container twice but that is what we do in the code? so maybe we should fix that, if possible. | 07:46 |
| kevko | SvenKieske: but I've sent a multinode to email :) | 07:47 |
| SvenKieske | but now you are comparing multinode with single node...which..just isn't a good comparison? why don't you look at a multinode job? | 07:47 |
| kevko | SvenKieske: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_2ea/929771/19/check/kolla-ansible-ubuntu-upgrade-ovn/2eab932/primary/logs/ansible/upgrade << this is multinode upgrade | 07:49 |
| kevko | SvenKieske: i can see this | 07:49 |
| kevko | NOTIFIED HANDLER rabbitmq : Restart rabbitmq container for primary | 07:49 |
| kevko | NOTIFIED HANDLER rabbitmq : Restart rabbitmq container for secondary1 | 07:49 |
| kevko | NOTIFIED HANDLER rabbitmq : Restart rabbitmq container for secondary2 | 07:49 |
| kevko | SvenKieske: so instead of restarting one by one (during the upgrade) we restarts all at once ...and then we are going to restart one by one :D ...it doesn't make a sense ... if check-containers which are calling handlers are removed from upgrade phase ...yeah ..maybe it should work | 07:52 |
| kevko | but this sounds for me like a bug | 07:52 |
| SvenKieske | that might very well be the case. | 07:53 |
| kevko | :D | 07:55 |
| kevko | SvenKieske: Personally, I'm quite afraid of restarting rabbitmq nodes all at once. Last time we did that, we ran into trouble and couldn't get rabbitmq up and running for 20 minutes because it broke during the restart. In the end, we figured out how to force start it, but it was nerve-wracking. That's why I'm testing it again now. In CI, there | 07:58 |
| kevko | isn't data like on a cloud under load... there it doesn't break. | 07:58 |
| kevko | aaaa, this is completly broken :'( | 08:05 |
| kevko | SvenKieske: and what about https://review.opendev.org/c/openstack/kolla-ansible/+/929848/8 << this ? And of course whole stack .. i am done with it btw . | 08:07 |
| mnasiadka | morning | 08:14 |
| kevko | morning | 08:14 |
| mnasiadka | kevko: I don't believe we restart all at once, can you check docker engine logs for time of restart? | 08:14 |
| kevko | mnasiadka: i will do it right now .. | 08:15 |
| kevko | mnasiadka: w8 a minute | 08:15 |
| mnasiadka | looking in Ansible output it's not restarting all at the same time, it's going through some playbook one host at a time, setting it to maint mode, restarting and then checking if it's up | 08:17 |
| kevko | and before ? | 08:17 |
| kevko | hmm, let me finish my test | 08:18 |
| kevko | hmm, it's probably work | 08:43 |
| ayhanilhan_ | Hi, I have 5 controllers and I want to temporarily remove and add these controllers for OS upgrade. After the "multinode stop controller-01" command, can I proceed with the "multinode deploy --limit controller-01" command instead of the "multinode deploy --limit control" command? I don't want Rabbitmq and MariaDB restarts on the other controllers. | 09:42 |
| kevko | ayhanilhan_: why you just don't stop the services on that node ...upgrade your os ..and reboot ? | 09:43 |
| ayhanilhan_ | @kevko I want to do a fresh install. Additionaly, i need to find out if I can proceed in this way not only when I upgrade the OS but also when I completely failure or replace the physical host? | 09:49 |
| kevko | ayhanilhan_: If i am correct, kolla-ansible doesn't support removal of host ..so if you want to remove ..you can remove from inventory so kolla-ansible will not touch the server ..but be careful ..because then all generated configuration will be without that host ... (this can work btw ...) ...but then you also need to remove services from | 09:59 |
| kevko | openstack keystone catalog for example .... and then you can redeploy | 09:59 |
| kevko | ayhanilhan_: you probably do the best if you replicate your workflow on some testing env | 10:01 |
| opendevreview | Merged openstack/kolla master: Switch to Dynamic Python Version Detection https://review.opendev.org/c/openstack/kolla/+/926744 | 10:01 |
| kevko | mnasiadka: can u check this revert ? https://review.opendev.org/c/openstack/kolla/+/930065 < i wanted to do it correctly but its' not working as I thought on proxysql side ... tested | 10:04 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Add support for Ubuntu Noble Numbat (24.04) LTS https://review.opendev.org/c/openstack/kayobe/+/930026 | 10:19 |
| ayhanilhan_ | @kevko kolla-ansible support removal of host. https://docs.openstack.org/kolla-ansible/latest/user/adding-and-removing-hosts.html . After stopping the services on the controller to be deleted as described in the document, when we run the âmultinode deploy --limit controlâ command, it restarts services such as mariadb and rabbitmq in other controllers. I want this not to happen. | 10:32 |
| ayhanilhan_ | I will not run âmultinode deploy --limit controlâ command after âmultinode stop --limit controller-01â command in the controller I want to remove. I will do Controller OS upgrade. Then I will do bootstrap-servers, image pull operations. When adding the controller, I want to use âmultinode deploy --limit controller-01â command instead of âmultinode deploy --limit controlâ. In this way, the services in | 10:32 |
| ayhanilhan_ | other controllers will not be restarted. We will try in our test cluster. I would like to get your suggestions. | 10:32 |
| goozuu_ | Hi, I have a qustion, how should be implemented Designate in Kolla? Is possible to have Dedidacated Designated as a service for each project ? How VMS are able to do lookups? Is it possible to expose port to internal private IP in Network ? Thanks | 10:48 |
| kevko | ayhanilhan_: you can limit to specified hosts as --limit ctlr0,ctrl1 <- without removed hosts | 11:07 |
| chembervint | hi. could please somebody proceed backports for my patch? https://review.opendev.org/c/openstack/kolla-ansible/+/929745 | 11:24 |
| kevko | SvenKieske: second vote please ? https://review.opendev.org/c/openstack/kolla-ansible/+/929848 << | 11:36 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Add support for Ubuntu Noble Numbat (24.04) LTS https://review.opendev.org/c/openstack/kayobe/+/930026 | 11:38 |
| ravlew | Hello, second vote please: https://review.opendev.org/c/openstack/kolla/+/926884 | 11:39 |
| SvenKieske | kevko: done | 11:48 |
| kevko | Thanks | 11:49 |
| SvenKieske | ravlew: can you add a short reno to that? | 11:49 |
| stan | is there any plans to integrate Cisco aci directly with kolla? | 11:49 |
| kevko | stan: do you have ACI for testing :) ? | 11:53 |
| kevko | SvenKieske: wondering if you have a time for some other reviews from me :) ? | 11:54 |
| ravlew | SvenKieske: I was under the impression that it is not needed if a bug is linked to the change. Is that no longer valid? | 11:54 |
| kevko | ravlew: Closes-Bug in commit message works with launchpad ...so you can see automatically updates for that bug in review..... release note is needed for documentation itself ...it's then rendered as release note ...so when the package is releases ..you can nicely see what was fixed ... | 11:55 |
| ravlew | I see, I will add it then. Thanks for clarifying kevko :). | 11:56 |
| SvenKieske | yeah, that would just be nice for end users to inform them that this now works as expected, e.g. you hit that bug in your version and you scan renos to see if it's already fixed. also: https://docs.openstack.org/kolla/latest/contributor/release-notes.html | 11:56 |
| SvenKieske | not everybody directly git greps :) | 11:57 |
| kevko | ravlew: no problemo | 11:57 |
| kevko | SvenKieske: haha, git is only one place i really trust :D | 11:57 |
| ravlew | Thanks SvenKieske | 11:57 |
| SvenKieske | thank you :) | 11:57 |
| kevko | SvenKieske: even if i want to check the renos i am going to generate it from a source in git :D | 11:57 |
| kevko | SvenKieske: question above ? ^^ :) | 11:58 |
| SvenKieske | uh, which one? | 11:58 |
| SvenKieske | a reviews? this week is still rather short on time, unfortunately. | 11:59 |
| SvenKieske | and I was lazy during the weekend | 11:59 |
| kevko | https://review.opendev.org/c/openstack/kolla-ansible/+/929771 << this chain i think it's pretty well prepared | 11:59 |
| kevko | SvenKieske: this is also step towards notifiers and restart and state of container configs as overal > https://review.opendev.org/c/openstack/kolla/+/915440 << also prepared very well i think ... with unit tests ..and also with docstrings you requested before | 12:00 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla master: Add missing iso packages to ironic-conductor https://review.opendev.org/c/openstack/kolla/+/926884 | 12:01 |
| kevko | SvenKieske: this is quite huge ... https://review.opendev.org/c/openstack/kolla/+/829295 << but most of that huge change is just same command in dockerfile ...logic is one function and one script | 12:01 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla master: Add missing iso packages to ironic-conductor https://review.opendev.org/c/openstack/kolla/+/926884 | 12:07 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Add support for Ubuntu Noble Numbat (24.04) LTS https://review.opendev.org/c/openstack/kayobe/+/930026 | 12:29 |
| mnasiadka | kevko: regarding the revert it looks good - but did you raise a bug in proxysql? Maybe we're doing something wrong? | 12:36 |
| kevko | mnasiadka: "No, we're not doing anything wrongâunfortunately, this is just a consequence of how ProxySQL is developed. They recognize only the internal database configuration. However, that's not okay for us, which is why I implemented it this way in the pastâand it really works reliably (I just forgot that I had already tried it once). This is | 12:49 |
| kevko | also confirmed by the official documentationâhttps://proxysql.com/documentation/configuration-file/ -> IMPORTANT: The configuration file isnât recommended for complex configurations; the database file should be used for that purpose. Please refer to the following section for more details | 12:49 |
| kevko | mnasiadka: BUT | 12:49 |
| kevko | wait a minute | 12:49 |
| kevko | mnasiadka: maybe you are right | 12:50 |
| kevko | mnasiadka: now i am reading section about https://proxysql.com/documentation/configuring-proxysql/#initialstartup | 12:50 |
| kevko | mnasiadka: the flag --initial | 12:50 |
| mnasiadka | It seems the designed way is to update the database using sqlite client (or admin web interface) | 12:52 |
| kevko | mnasiadka: Yes, that's true, you can configure ProxySQL on-the-fly ... you can experimentally change the configuration while it's running ... but in our use case, you want to restart the container and have the configuration that you set there with k-a. | 12:54 |
| kevko | but initial will work same i think ..wait a minute | 12:54 |
| kevko | mnasiadka: btw, tested recovery of backed up database ...it's not working ...but not because of PERCONA_SCHEMA vs mysql ...because of version of mysql redo log ... | 12:54 |
| mnasiadka | seems configuration file has no input validation, but I guess we just need to live with that | 12:55 |
| kevko | mnasiadka: i needed to start temporary container on max 10.7 mariadb ...then restore of backup was working ... | 12:55 |
| kevko | mnasiadka: then i need to bypass somehow galera ...safe to bootstrap and this stuff | 12:55 |
| kevko | mnasiadka: then i need to upgrade container ...then i deployed the higher version of mariadb | 12:56 |
| kevko | but it was pain as f... , this raises the question of whether it's really okay to use a binary backup instead of mysqldump for example | 12:57 |
| kevko | TLDR ...problem was with backup or better said mariadb version the backup was created on .... not the stuff you asked to test ... | 12:58 |
| opendevreview | Merged openstack/kolla-ansible stable/2024.1: Fix os-brick multipathd del map fails for SAN multipath devices https://review.opendev.org/c/openstack/kolla-ansible/+/929745 | 12:59 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.1: Fix os-brick multipathd del map fails for SAN multipath devices https://review.opendev.org/c/openstack/kolla-ansible/+/929747 | 12:59 |
| opendevreview | Merged openstack/kolla-ansible stable/2023.2: Fix os-brick multipathd del map fails for SAN multipath devices https://review.opendev.org/c/openstack/kolla-ansible/+/929746 | 13:00 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: CI: Generate Octavia certificates when required https://review.opendev.org/c/openstack/kayobe/+/930196 | 13:01 |
| kevko | mnasiadka: yep, initial working as expected | 13:03 |
| kevko | root@controller0:~# ls -la /var/lib/docker/volumes/proxysql/_data/ | 13:03 |
| kevko | total 640 | 13:03 |
| kevko | drwxr-xr-x 2 42487 42487 4096 Sep 23 13:02 . | 13:03 |
| kevko | drwx-----x 3 root root 4096 Sep 23 12:54 .. | 13:03 |
| kevko | -rw-r--r-- 1 42487 42487 1082 Sep 23 12:54 proxysql-ca.pem | 13:03 |
| kevko | -rw-r--r-- 1 42487 42487 1086 Sep 23 12:54 proxysql-cert.pem | 13:03 |
| kevko | -rw-r--r-- 1 42487 42487 1675 Sep 23 12:54 proxysql-key.pem | 13:03 |
| kevko | -rw-r--r-- 1 42487 42487 225280 Sep 23 13:02 proxysql.db | 13:03 |
| kevko | -rw-r--r-- 1 42487 42487 225280 Sep 23 12:54 proxysql.db.bak | 13:03 |
| kevko | -rw-r--r-- 1 42487 42487 184320 Sep 23 13:02 proxysql_stats.db | 13:03 |
| kevko | bak created | 13:03 |
| kevko | mnasiadka: so i will drop the revert and commit the patch with --initial flag ... (it works the same as removal of DB ...but moreover it's createing the bak file) | 13:07 |
| stan | SvenKieske Yes, we have Cisco aci in our environment. my network guys are very keen to get aci working with Openstack. | 13:14 |
| SvenKieske | stan: patches are generally welcome, but it's difficult from a CI perspective, there is the option to run some external CI but I don't personally know the requirements for that. so in short: you would need to do quite some investment for that yourself. | 13:15 |
| opendevreview | Merged openstack/kolla-ansible master: Fix IPv6 address format in ProxySQL configuration https://review.opendev.org/c/openstack/kolla-ansible/+/929848 | 13:15 |
| SvenKieske | and some maintainers are not that interested in maintaining proprietary stuff, this is also strongly correlated with the fact that it's hard or impossible to test. | 13:16 |
| opendevreview | Jakub Darmach proposed openstack/kayobe master: Add support for Ubuntu Noble Numbat (24.04) LTS https://review.opendev.org/c/openstack/kayobe/+/930026 | 13:35 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Improvement of ProxySQL Monitoring Configuration https://review.opendev.org/c/openstack/kolla-ansible/+/930077 | 13:39 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Switch mariadb's loadbalancer from HAProxy to ProxySQL https://review.opendev.org/c/openstack/kolla-ansible/+/913724 | 13:39 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Change from PERCONA_SCHEMA to mysql for mariabackup https://review.opendev.org/c/openstack/kolla-ansible/+/922729 | 13:39 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix mariadb backup unable to create history table https://review.opendev.org/c/openstack/kolla-ansible/+/929827 | 13:39 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: [CI] Add mariadb backup into mariadb scenario https://review.opendev.org/c/openstack/kolla-ansible/+/929771 | 13:39 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Fix ProxySQL startup mode to use --initial https://review.opendev.org/c/openstack/kolla-ansible/+/930199 | 13:39 |
| opendevreview | Michal Arbet proposed openstack/kolla master: Add mechanism for patching files in containers https://review.opendev.org/c/openstack/kolla/+/829295 | 13:59 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [CI] Test kolla patch https://review.opendev.org/c/openstack/kolla/+/929446 | 13:59 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Remove api-paste config files for masakari, cyborg https://review.opendev.org/c/openstack/kolla-ansible/+/911087 | 14:04 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: WIP: kolla: Add support for cross-arch builds https://review.opendev.org/c/openstack/kayobe/+/930204 | 14:17 |
| ravlew | Status green, release note added SvenKieske : https://review.opendev.org/c/openstack/kolla/+/926884 | 14:23 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: Set extra kernel parameters for managed inspection https://review.opendev.org/c/openstack/kolla-ansible/+/930206 | 14:26 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: Set extra kernel parameters for managed inspection https://review.opendev.org/c/openstack/kolla-ansible/+/930206 | 14:26 |
| opendevreview | Merged openstack/kolla master: Add missing iso packages to ironic-conductor https://review.opendev.org/c/openstack/kolla/+/926884 | 16:26 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: WIP: kolla: Add support for cross-arch builds https://review.opendev.org/c/openstack/kayobe/+/930204 | 16:29 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: WIP: kolla: Add support for cross-arch builds https://review.opendev.org/c/openstack/kayobe/+/930204 | 16:33 |
| opendevreview | Michal Nasiadka proposed openstack/kayobe master: kolla-build: Add support for cross-arch builds https://review.opendev.org/c/openstack/kayobe/+/930204 | 16:33 |
| opendevreview | Pierre Riteau proposed openstack/kolla-ansible master: Set extra kernel parameters for managed inspection https://review.opendev.org/c/openstack/kolla-ansible/+/930206 | 16:39 |
| stan | SvenKieske Makes sense. Thank you | 16:54 |
| opendevreview | Roman KrÄek proposed openstack/kolla-ansible master: Don't notify handlers during copy-cert https://review.opendev.org/c/openstack/kolla-ansible/+/924651 | 16:55 |
| SvenKieske | stan: you're welcome! and of course there's always the possibility to pay someone to build your desired feature for you, as another option :) | 17:06 |
| JH | Hi all, I'm having some trouble logging into skyline. When I use the keystone admin password I get the username or password error. However I can log in with no issues using the same credentials for Horizon. | 18:45 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: Standardize python version paths https://review.opendev.org/c/openstack/kolla-ansible/+/926745 | 19:39 |
| opendevreview | Michal Arbet proposed openstack/kolla master: [DNM] Add APACHE_EXECUTABLE into http_setup script https://review.opendev.org/c/openstack/kolla/+/930239 | 20:04 |
| Core3861 | JH From my experience, skyline breaks easily whenever you reconfigure anything related to keystone and requires restart to properly work. | 22:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!