| opendevreview | Thomas Goirand proposed openstack/puppet-magnum master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-magnum/+/944828 | 08:23 |
|---|---|---|
| zigo | tkajinam: Is there more puppet modules I should patch to add the feature? I believe I have them all, no? | 08:24 |
| tkajinam | there are still a few more modules such as cloudkitty, trove, mistral and so on | 08:27 |
| tkajinam | if you grep oslo::messaging then you may find the full list | 08:28 |
| zigo | Ok will do it. | 09:17 |
| opendevreview | Merged openstack/puppet-nova master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-nova/+/944785 | 09:34 |
| opendevreview | Thomas Goirand proposed openstack/puppet-cloudkitty master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-cloudkitty/+/944846 | 09:40 |
| opendevreview | Thomas Goirand proposed openstack/puppet-mistral master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-mistral/+/944847 | 09:47 |
| opendevreview | Thomas Goirand proposed openstack/puppet-trove master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-trove/+/944850 | 09:53 |
| zigo | tkajinam: There you go, cloudkitty, mistral and trove. | 09:53 |
| tkajinam | zigo, I wasn't quite specific but there are still number of modules left | 09:55 |
| tkajinam | like watcher, vitrage etc | 09:55 |
| tkajinam | as I said please grep the code to find any module using oslo::messaging::.* | 09:55 |
| zigo | Yeah, but didn't have time to do it ... yet ! :) | 09:55 |
| tkajinam | if you don't want to cover all then just leave these | 09:55 |
| tkajinam | I can take care of these when I get time | 09:55 |
| zigo | I do carefully read what you write, since you're always right ... :) | 09:56 |
| zigo | It wont take long, I'll do. | 09:56 |
| tkajinam | thx | 09:56 |
| tkajinam | you had to read the next sentence, not that one, carefully :-P | 09:56 |
| tkajinam | number of modules is still painful some time as you know, but that's how the life in this project has been, as you also know | 09:57 |
| zigo | Ironic missing too. | 09:57 |
| zigo | Then I think that should be it. | 09:57 |
| tkajinam | yes !! | 09:57 |
| zigo | Aka: watcher, vitrage, trove and Ironic. | 09:58 |
| tkajinam | you might also have to check puppet-swift | 09:58 |
| zigo | I doesn't use Rabbit, does it? | 09:58 |
| tkajinam | swift::proxy::ceilometer | 09:58 |
| tkajinam | it does | 09:58 |
| tkajinam | I mean it does not use rabbitmq internally but it does use it for notification to ceilometer | 09:59 |
| zigo | That's in ceilometermiddleware, and I don't think it has these options. | 10:00 |
| zigo | The notification bus is always the same... | 10:00 |
| zigo | BTW, not sure if you used it, but it doesn't scale. | 10:00 |
| zigo | We used to have it in production, and it just destroyed our billing system, with so many messages in the queue. | 10:01 |
| zigo | We ended up just disabling it, and made bandwidth "free" on our swiftproxies. | 10:01 |
| tkajinam | zigo, it didn't expose full oslo.messaging options which I fixed by adding a separate config file | 10:03 |
| zigo | IMO, it would need a bit of caching, somehow, so that it could gather traffic over the last 10 seconds, and send a way fewer messages. | 10:03 |
| zigo | Oh... | 10:03 |
| tkajinam | it was needed to configure ssl options for rabbitmq | 10:04 |
| tkajinam | I guess it has an option to send metrics asynchonously and I wonder if it can help | 10:04 |
| zigo | What would be that config file ? | 10:04 |
| zigo | Oh, does it? | 10:04 |
| tkajinam | zigo, https://review.opendev.org/c/openstack/ceilometermiddleware/+/904328 | 10:04 |
| tkajinam | zigo, https://review.opendev.org/c/openstack/puppet-swift/+/916622 | 10:05 |
| zigo | Thanks. | 10:05 |
| tkajinam | I've decided to create /etc/swift/ceilometer.conf in puppet-swift but I guess it's not shipped by any of the packages currently | 10:05 |
| tkajinam | even the one of rdo does not ship it | 10:05 |
| tkajinam | the file path is configurable and actually needs to be configured to use the file | 10:06 |
| zigo | What namespaces would be there? Only oslo.messaging ? | 10:06 |
| tkajinam | yeah | 10:06 |
| zigo | Issue also is that swift is still not using oslo.config ... | 10:06 |
| tkajinam | just oslo.messaging | 10:07 |
| tkajinam | that's what I had to add a separate file | 10:07 |
| zigo | Yeah. | 10:07 |
| tkajinam | which is loaded by oslo.config within ceilometermiddleware | 10:07 |
| tkajinam | instead of exposing every single option in swift.conf | 10:07 |
| zigo | I wish Swift people accepted switching to many oslo.X stuff. | 10:07 |
| tkajinam | zigo, the nonblocking_notify option is what I was referring to > async | 10:07 |
| zigo | Oh ok. | 10:07 |
| tkajinam | well that would be nice but as long as anyone is interested in paying effort for | 10:07 |
| tkajinam | you know | 10:07 |
| zigo | Yeah, but I did send some patches to swift, and got rejected because they didn't have oslo libs as dependency and said it should be discussed first. | 10:08 |
| zigo | One annoying bit I had: we tried switching to json logs, so it is easier to push to Elasticsearch. | 10:08 |
| zigo | Swift doesn't support it because it doesn't use oslo.log. | 10:08 |
| tkajinam | maybe you need to update their log formatter | 10:09 |
| zigo | Well, the easier way would be to teach Swift how to use oslo.log ! :) | 10:10 |
| tkajinam | I personally dislike json logs after being asked to read it to debug complicated nova neutron logs ... | 10:10 |
| zigo | You're not supposed to read it in the files, but through ES. | 10:10 |
| tkajinam | I know that's friendly to elastic search or other search tool but that's a hell for human reading it to debug something | 10:10 |
| zigo | I agree it's barely readable. | 10:10 |
| zigo | Yeah. | 10:10 |
| tkajinam | I don't know if I "can" read logs through es for debugging to be very honest. | 10:11 |
| zigo | It'd be nice to have both, in fact. | 10:11 |
| tkajinam | yeah | 10:11 |
| tkajinam | I ended up writing a tool to convert json to bare string logs | 10:11 |
| tkajinam | for my mental safety | 10:11 |
| tkajinam | json logs are sometimes useful but I remember some problematic cases where it tries to load full object from db | 10:14 |
| tkajinam | as I requires to load full attribute to be dumped | 10:14 |
| tkajinam | there were some deadlock case caused by logging trying to load db while db access requires logging in connection. | 10:15 |
| tkajinam | I mean I'm sure you may find "interesting" things. | 10:15 |
| zigo | BTW, if you use debug=True and use_json=True with neutron-bgp-agent, it just freezes ... :) | 10:19 |
| zigo | (I of course opened a bug for that...) | 10:19 |
| tkajinam | I suspect that is the dead lock problem I was talking about | 10:21 |
| tkajinam | IIRC I saw deadlock in cinder with debug=True and use_json=True | 10:21 |
| tkajinam | and disable one of these resolves the problem | 10:21 |
| zigo | So that would be a generic issue in oslo.log? Or is this project specific? | 10:22 |
| tkajinam | you may have to dump GMR (if neutron-bgp-agent supports it) to see where the threads get stuck | 10:22 |
| zigo | GMR ? | 10:23 |
| zigo | What's that? | 10:23 |
| tkajinam | https://docs.openstack.org/oslo.reports/latest/reference/api/oslo_reports.html#module-oslo_reports.guru_meditation_report | 10:24 |
| zigo | Oh... :) | 10:24 |
| opendevreview | Thomas Goirand proposed openstack/puppet-vitrage master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-vitrage/+/944856 | 10:44 |
| opendevreview | Thomas Goirand proposed openstack/puppet-ironic master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-ironic/+/944857 | 10:49 |
| opendevreview | Thomas Goirand proposed openstack/puppet-swift master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-swift/+/944859 | 11:09 |
| opendevreview | Thomas Goirand proposed openstack/puppet-magnum master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-magnum/+/944828 | 11:50 |
| zigo | tkajinam: Just checked, we already have the [filter:ceilometer]/nonblocking_notify=True option in our proxy-server.conf, and that's not enough: with the huge traffic from customers, this still kills our billing system with aprox 5k messages per second. | 14:18 |
| zigo | There's no "let's take this 10 swift queries and make a single notification message" kind of feature. | 14:18 |
| opendevreview | Merged openstack/puppet-cloudkitty master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-cloudkitty/+/944846 | 14:32 |
| opendevreview | Merged openstack/puppet-nova master: Rename nova::compute::pci::paththrough https://review.opendev.org/c/openstack/puppet-nova/+/944615 | 14:58 |
| opendevreview | Merged openstack/puppet-mistral master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-mistral/+/944847 | 15:01 |
| steveames | Hello... with the Epoxy release coming up, I was wondering if the puppet modules would be updated to support Ubuntu 24.04? I just glanced at the nova metadata.json file and it's still saying 22.04 | 15:13 |
| opendevreview | Merged openstack/puppet-nova master: Stop accepting encoded JSON string https://review.opendev.org/c/openstack/puppet-nova/+/944614 | 15:27 |
| opendevreview | Merged openstack/puppet-magnum master: Add a rabbit_transient_queues_ttl and amqp_auto_delete params https://review.opendev.org/c/openstack/puppet-magnum/+/944828 | 15:34 |
| tkajinam | steveames, I hoped I could complete migration but the whole work is blocked by too old python-openstackclient package in UCA Dalmatian | 15:36 |
| tkajinam | https://bugs.launchpad.net/bugs/2097764 | 15:37 |
| tkajinam | zigo, if the problem is caused by the whole amount of notification then I agree the nonblocking feature is not really useful | 15:39 |
| tkajinam | a problem is that it's not really easy to 'batch' the notification and we eventually have to send individual messages per request ... | 15:40 |
| tkajinam | steveames, the current module may work in Ubuntu 22.04 but we can't run full tests due to that buggy openstackclient in Dalmatian. Once it is released in UCA Dalmatian we can switch the master job to 24.04 + Dalmatian to prepare for 2024.2 Epoxy coming | 15:41 |
| tkajinam | though honestly I'm unsure if they are really willing to help us... seeing sort of late response to their "non-prioritized" things | 15:41 |
| tkajinam | I mean 24.04 + Epoxy, not 2024.2 Epoxy | 15:42 |
| zigo | tkajinam: Are you talking about the fact that Dalmatian openstackclient says something like "no server like <UUID>" but it still spawns it? | 15:42 |
| zigo | I'm currently experiencing that ... | 15:43 |
| tkajinam | zigo, no I'm talking about a few problems with keystone-related command caused by internal switch to sdk | 15:43 |
| zigo | Ah ok. | 15:43 |
| tkajinam | another annoying point of ubuntu deployment is that they do not provide beta packages following master (like ones we heavily use from RDO) and they usually provide packages even some time after GA | 15:45 |
| tkajinam | so there is no real way for us to test ubuntu deployment with the upcoming release | 15:45 |
| steveames | tkajinam: sadness. Thanks for the explanation. I was hoping to update everything to Ubuntu24.04 but it looks like I'll hold off. | 15:56 |
| zigo | steveames: Feel free to switch to Debian ! :) | 16:04 |
| zigo | Oh, btw, I'm late with the RC1 ... :/ | 16:04 |
| opendevreview | Merged openstack/puppet-neutron master: Add support for [DEFAULT] pagination_max_limit https://review.opendev.org/c/openstack/puppet-neutron/+/944709 | 16:40 |
| opendevreview | Merged openstack/puppet-neutron master: Deprecate useless rootwrap class https://review.opendev.org/c/openstack/puppet-neutron/+/944692 | 18:23 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!