| *** tosky has quit IRC | 00:30 | |
| *** eandersson has joined #openstack-oslo | 01:35 | |
| *** eandersson has quit IRC | 01:42 | |
| *** eandersson has joined #openstack-oslo | 01:42 | |
| openstackgerrit | Wenping Song proposed openstack/oslo.utils master: Drop lower-constraints https://review.opendev.org/c/openstack/oslo.utils/+/772630 | 02:16 |
|---|---|---|
| *** zzzeek has quit IRC | 02:47 | |
| *** zzzeek has joined #openstack-oslo | 02:50 | |
| *** SpamapS has quit IRC | 02:59 | |
| *** gouthamr has quit IRC | 02:59 | |
| *** bcafarel has quit IRC | 03:00 | |
| *** SpamapS has joined #openstack-oslo | 03:00 | |
| *** gouthamr has joined #openstack-oslo | 03:00 | |
| *** gouthamr has quit IRC | 03:00 | |
| *** gouthamr has joined #openstack-oslo | 03:01 | |
| *** gouthamr has quit IRC | 03:01 | |
| *** gouthamr has joined #openstack-oslo | 03:01 | |
| *** gouthamr has quit IRC | 03:01 | |
| *** gouthamr has joined #openstack-oslo | 03:02 | |
| *** gouthamr has quit IRC | 03:02 | |
| *** gouthamr has joined #openstack-oslo | 03:03 | |
| *** gouthamr has quit IRC | 03:03 | |
| *** gouthamr has joined #openstack-oslo | 03:04 | |
| *** gouthamr has quit IRC | 03:04 | |
| *** gouthamr has joined #openstack-oslo | 03:04 | |
| *** gouthamr has quit IRC | 03:04 | |
| *** gouthamr has joined #openstack-oslo | 03:05 | |
| *** gouthamr has quit IRC | 03:05 | |
| *** gouthamr has joined #openstack-oslo | 03:06 | |
| *** gouthamr has quit IRC | 03:06 | |
| *** gouthamr has joined #openstack-oslo | 03:07 | |
| *** gouthamr has quit IRC | 03:07 | |
| *** gouthamr has joined #openstack-oslo | 03:11 | |
| *** gouthamr has quit IRC | 03:11 | |
| *** gouthamr has joined #openstack-oslo | 03:17 | |
| *** gouthamr has quit IRC | 03:17 | |
| *** gouthamr has joined #openstack-oslo | 03:22 | |
| *** gouthamr has quit IRC | 03:22 | |
| *** bcafarel has joined #openstack-oslo | 03:25 | |
| *** gouthamr has joined #openstack-oslo | 03:28 | |
| *** gouthamr has quit IRC | 03:28 | |
| *** gouthamr has joined #openstack-oslo | 03:32 | |
| *** gouthamr has quit IRC | 03:32 | |
| *** rcernin has quit IRC | 03:36 | |
| *** gouthamr has joined #openstack-oslo | 03:38 | |
| *** gouthamr has quit IRC | 03:38 | |
| *** gouthamr has joined #openstack-oslo | 03:43 | |
| *** gouthamr has quit IRC | 03:43 | |
| *** gouthamr has joined #openstack-oslo | 03:49 | |
| *** gouthamr has quit IRC | 03:49 | |
| openstackgerrit | Wander Way proposed openstack/osprofiler master: Uncap PrettyTable https://review.opendev.org/c/openstack/osprofiler/+/776308 | 03:50 |
| *** gouthamr has joined #openstack-oslo | 03:53 | |
| *** gouthamr has quit IRC | 03:53 | |
| *** gouthamr has joined #openstack-oslo | 03:59 | |
| *** gouthamr has quit IRC | 03:59 | |
| *** rcernin has joined #openstack-oslo | 04:16 | |
| *** gouthamr has joined #openstack-oslo | 04:25 | |
| *** gouthamr has quit IRC | 04:26 | |
| *** gouthamr has joined #openstack-oslo | 05:00 | |
| *** gouthamr has quit IRC | 05:00 | |
| *** gouthamr has joined #openstack-oslo | 05:26 | |
| *** gouthamr has quit IRC | 05:26 | |
| *** gouthamr has joined #openstack-oslo | 06:00 | |
| *** eandersson has quit IRC | 06:53 | |
| *** eandersson has joined #openstack-oslo | 06:54 | |
| *** eandersson has quit IRC | 06:56 | |
| *** eandersson has joined #openstack-oslo | 06:56 | |
| *** lpetrut has joined #openstack-oslo | 07:06 | |
| *** ralonsoh has joined #openstack-oslo | 07:21 | |
| *** brinzhang has quit IRC | 07:45 | |
| *** brinzhang has joined #openstack-oslo | 07:45 | |
| *** rcernin has quit IRC | 07:58 | |
| *** rcernin has joined #openstack-oslo | 08:02 | |
| *** rcernin has quit IRC | 08:07 | |
| *** rpittau|afk is now known as rpittau | 08:14 | |
| *** sboyron has joined #openstack-oslo | 08:24 | |
| *** tosky has joined #openstack-oslo | 08:45 | |
| *** rcernin has joined #openstack-oslo | 08:49 | |
| *** rcernin has quit IRC | 08:53 | |
| *** brinzhang has quit IRC | 09:00 | |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.messaging stable/train: Fix type of direct_mandatory_flag opt https://review.opendev.org/c/openstack/oslo.messaging/+/776416 | 10:34 |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.messaging stable/train: Cancel consumer if queue down https://review.opendev.org/c/openstack/oslo.messaging/+/747892 | 10:36 |
| hberaud | stephenfin: o/ around? | 10:40 |
| stephenfin | yup | 10:40 |
| hberaud | I've a question for you | 10:40 |
| stephenfin | shoot | 10:41 |
| hberaud | stephenfin: Do you think we need to backport this deprecation to stable branches? => https://review.opendev.org/c/openstack/oslo.messaging/+/764776 | 10:41 |
| hberaud | my goal is to backport https://review.opendev.org/c/openstack/oslo.messaging/+/771232 | 10:41 |
| hberaud | but I'm not sure we want to backport the deprecation first | 10:42 |
| hberaud | What do you think about that? | 10:42 |
| hberaud | I think it could be weird to deprecate this on stable branches too | 10:43 |
| stephenfin | Hmm, that's a tricky one. It's technically a change in behaviour, but I guess the previous behaviour was incorrect | 10:43 |
| hberaud | but on the other hand we don't give date and indeed it will be removed in next series | 10:43 |
| stephenfin | I'd be okay with backporting it, but I'd like to see what e.g. bnemec thinks too to be honest | 10:44 |
| hberaud | yes sure | 10:44 |
| hberaud | stephenfin, bnemec: I'll just prepare the backport for victoria (of the both patches) and we will discuss directly on them, let me know what do you think about that | 10:45 |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.messaging stable/victoria: Deprecate the mandatory flag https://review.opendev.org/c/openstack/oslo.messaging/+/776417 | 10:47 |
| openstackgerrit | Hervé Beraud proposed openstack/oslo.messaging stable/victoria: Correctly handle missing RabbitMQ queues https://review.opendev.org/c/openstack/oslo.messaging/+/776418 | 10:47 |
| hberaud | stephenfin, bnemec ^ | 10:47 |
| stephenfin | ack | 10:47 |
| hberaud | no rush | 10:47 |
| openstackgerrit | Daniel Bengtsson proposed openstack/pbr master: Dropping lower constraints testing https://review.opendev.org/c/openstack/pbr/+/767724 | 10:57 |
| jrosser | hi! i've been putting a bunch of effort into improving the CI reliability for openstack-ansible, one of the occasional failures we get is this between keystone<>memcached like this http://paste.openstack.org/show/802485/ | 12:54 |
| jrosser | not sure where to start on this, if we've made bad config on OSA or if it's a real bug somewhere | 12:54 |
| hberaud | jrosser: Which version of pymemcache is in use? | 13:10 |
| hberaud | jrosser: not sure if it's related but that could be a track to follow https://review.opendev.org/c/openstack/ironic-inspector/+/772714 | 13:10 |
| jrosser | hberaud: unless something has gone horribly wrong pymemcache will be according to upper-constraints | 13:11 |
| hberaud | jrosser: we recently did few back and forth with pymemcache => https://review.opendev.org/c/openstack/requirements/+/772713 https://review.opendev.org/c/openstack/requirements/+/772684 | 13:11 |
| hberaud | ok | 13:11 |
| hberaud | jrosser: anyway pymemcache doesn't appear in your stacktrace | 13:12 |
| jrosser | this is what we install https://github.com/openstack/openstack-ansible-os_keystone/blob/master/defaults/main.yml#L489-L499 | 13:13 |
| hberaud | apparently you use from python-memcached through the oslo.cache.memcache_pool_backend | 13:13 |
| hberaud | ah | 13:13 |
| jrosser | install is one thing, use maybe another | 13:14 |
| hberaud | Can take a look to the number of active connections on the memcache server side? | 13:14 |
| hberaud | jrosser: could be related => https://bugs.launchpad.net/oslo.cache/+bug/1883659 https://bugs.launchpad.net/oslo.cache/+bug/1888394 | 13:15 |
| openstack | Launchpad bug 1883659 in oslo.cache "keystonemiddleware connections to memcached from neutron-server grow beyond configured values" [Undecided,Confirmed] | 13:15 |
| openstack | Launchpad bug 1888394 in oslo.cache "Oslo.cache exponencially raising up connection to memcached" [Undecided,In progress] | 13:15 |
| jrosser | it's hard to say how many connections there were, but i have the memcached log from one of these https://zuul.opendev.org/t/openstack/build/ae58b54011b345fdb9210abd6a0541e3/log/logs/host/memcached.service.journal-18-20-09.log.txt | 13:17 |
| jrosser | unfortunatley our logging collects a dump of listening sockets, not active connections | 13:18 |
| jrosser | i did see the chat the other day about keystone itself not using keystonemiddleware though..... | 13:19 |
| * jrosser lunch, bbl | 13:19 | |
| tosky | hberaud: did you plan to backport https://review.opendev.org/c/openstack/oslo.versionedobjects/+/767748 to victoria, ussuri, (train?) | 13:20 |
| tosky | or can I do it? | 13:20 |
| tosky | I'm just trying to merge https://review.opendev.org/c/openstack/oslo.versionedobjects/+/746506 :) (which requires https://review.opendev.org/c/openstack/oslo.versionedobjects/+/776295 ) | 13:21 |
| hberaud | tosky: This is on my todo list but feel free to start to backport them :) | 13:21 |
| hberaud | however... | 13:21 |
| * tosky is scared by the however | 13:22 | |
| hberaud | lol | 13:22 |
| hberaud | yeah some elements are missing | 13:22 |
| hberaud | the tox.ini updates | 13:22 |
| hberaud | and the removal of L-C.txt | 13:22 |
| hberaud | so to start to backport these patches we need first to submit a new patch to master to append these elements and then start to backport these 2 patches as a single one patch | 13:23 |
| hberaud | tosky: ^ | 13:23 |
| tosky | uh, I see | 13:23 |
| hberaud | let me give you an example | 13:23 |
| hberaud | those are the initial patches => https://review.opendev.org/c/openstack/oslo.messaging/+/767736 https://review.opendev.org/c/openstack/oslo.messaging/+/774231 | 13:24 |
| hberaud | tosky: and this how we backported them => https://review.opendev.org/c/openstack/oslo.messaging/+/772654 | 13:25 |
| openstackgerrit | Luigi Toscano proposed openstack/oslo.versionedobjects master: Dropping lower constraints references https://review.opendev.org/c/openstack/oslo.versionedobjects/+/776432 | 13:26 |
| hberaud | tosky: the previous one was for victoria and this one is for ussuri https://review.opendev.org/c/openstack/oslo.messaging/+/772655 | 13:26 |
| tosky | like this ^^? | 13:26 |
| hberaud | exactly | 13:26 |
| tosky | just in time before my 2.5 hours of meetings \o/ /o\ | 13:27 |
| hberaud | :) | 13:27 |
| hberaud | I'll fast approve this one | 13:27 |
| tosky | thanks! | 13:27 |
| hberaud | tosky: thanks to YOU ;) | 13:29 |
| hberaud | jrosser: $(grep memcache.log -e "client connection" | wc -l) = 173 connections in ~30 minutes that doesn't seems so much terrible | 13:35 |
| jrosser | yes, it didnt look excessive, and theres also not really anything suspicious in the memcached log either | 13:46 |
| *** iurygregory has quit IRC | 13:47 | |
| jrosser | is there anything we should be doing python-memcache <> pymemcache | 13:47 |
| *** iurygregory has joined #openstack-oslo | 13:50 | |
| stephenfin | jrosser: I'm pretty sure I was looking at a patch for this recently | 14:31 |
| stephenfin | one sec | 14:31 |
| stephenfin | jrosser: Does this look useful? https://review.opendev.org/c/openstack/keystonemiddleware/+/773939 | 14:32 |
| stephenfin | Apologies, in no less than two meetings rn so I haven't had a chance to read all the scrollback | 14:33 |
| jrosser | stephenfin: possibly - though i think what i am seeing is between the keystone flask app and memcached - does that use keystoneclient? | 14:33 |
| jrosser | sorry keystonemiddleware i mean | 14:36 |
| *** tosky has quit IRC | 14:51 | |
| *** tosky_ has joined #openstack-oslo | 14:51 | |
| *** tosky_ is now known as tosky | 14:51 | |
| stephenfin | jrosser: Sorry, meetings--. I'm not sure. I'm reading the scrollback to get context now | 14:54 |
| stephenfin | jrosser: Ah, no, doesn't seem like it. That patch I linked solves an issue where the amount of memcache pools grows without bound | 14:58 |
| stephenfin | at least until memcache eventually crashes and burns | 14:58 |
| stephenfin | which based on the discussion you had with hberaud doesn't sound like an issue you're seeing | 14:59 |
| jrosser | is there anything obvious we're doing wrong in the keystone cache config here https://github.com/openstack/openstack-ansible-os_keystone/blob/master/templates/keystone.conf.j2#L48-L59 | 14:59 |
| hberaud | stephenfin: At first glance nothing shocks me | 15:02 |
| hberaud | sorry | 15:02 |
| hberaud | jrosser: ^ | 15:03 |
| hberaud | can you try to add? `memcache_use_advanced_pool = True` | 15:03 |
| hberaud | jrosser: ^ | 15:03 |
| jrosser | sure we can try that, in the [cache] section? | 15:03 |
| hberaud | jrosser: hm no I think in `[keystone_authtoken]` | 15:04 |
| jrosser | even for keystone itself? | 15:05 |
| hberaud | no in neutron.conf | 15:05 |
| stephenfin | jrosser: Nothing jumps out at me, but I'm afraid my keystone-fu is rudimentary at best | 15:06 |
| stephenfin | The only thing I've noted is that vexxhost appear to be using a backend from dogpile rather than oslo_config, but I do not know why or if it matters /o\ https://opendev.org/vexxhost/openstack-operator/src/branch/master/openstack_operator/templates/keystone/secret-config.yml.j2 | 15:06 |
| hberaud | stephenfin: it's due to => https://opendev.org/openstack/oslo.cache/src/branch/master/oslo_cache/backends/memcache_pool.py#L20 | 15:08 |
| hberaud | stephenfin: oslo.cache's PooledMemcachedBackend inherit from dogpile | 15:08 |
| hberaud | stephenfin: I suppose that you meant oslo.cache and not oslo.config, isn't? | 15:10 |
| stephenfin | I did, yes /o\ | 15:10 |
| hberaud | np | 15:10 |
| hberaud | :) | 15:10 |
| *** dave-mccowan has joined #openstack-oslo | 15:11 | |
| hberaud | jrosser: I think that you should update this config file => https://github.com/openstack/openstack-ansible-os_neutron/blob/master/templates/neutron.conf.j2#L178 | 15:12 |
| hberaud | I can't find trace of `memcache_use_advanced_pool` into this template | 15:13 |
| jrosser | do you think that the error i showed is related to neutron, or just a general case for how we should configure keystone_authtoken for all the services? | 15:14 |
| hberaud | jrosser: AFAIK only neutron was impacted... | 15:16 |
| hberaud | but maybe I miss info | 15:17 |
| jrosser | looks like kolla set that for neutron | 15:18 |
| jrosser | and the puppet stuff seems to put it many places | 15:19 |
| jrosser | thanks for the tip, i can try that easily | 15:19 |
| hberaud | let me know if that unlock your path | 15:20 |
| jrosser | it doesnt fail very often, but often enough it's made my list of broken things since the call went on the ML about taming CI usage | 15:21 |
| openstackgerrit | Ben Nemec proposed openstack/oslo.config master: Bump minimum version for PyYAML to 5.1 https://review.opendev.org/c/openstack/oslo.config/+/776481 | 15:47 |
| *** lpetrut has quit IRC | 16:00 | |
| openstackgerrit | Merged openstack/oslo.versionedobjects master: Dropping lower constraints references https://review.opendev.org/c/openstack/oslo.versionedobjects/+/776432 | 16:29 |
| *** rpittau is now known as rpittau|afk | 17:47 | |
| openstackgerrit | Ben Nemec proposed openstack/oslo.config master: Bump minimum version for PyYAML to 5.1 https://review.opendev.org/c/openstack/oslo.config/+/776481 | 18:05 |
| *** ralonsoh has quit IRC | 18:15 | |
| stephenfin | bnemec: https://review.opendev.org/c/openstack/oslo.config/+/771102 | 18:20 |
| stephenfin | for the PyYAML issue | 18:21 |
| bnemec | Ah, nice. Thanks! | 18:22 |
| openstackgerrit | Ben Nemec proposed openstack/oslo.config master: Bump minimum version for PyYAML to 5.1 https://review.opendev.org/c/openstack/oslo.config/+/776481 | 18:24 |
| openstackgerrit | Merged openstack/oslo.config master: remove lower-constraints.txt https://review.opendev.org/c/openstack/oslo.config/+/771102 | 19:34 |
| openstackgerrit | Merged openstack/oslo.context master: Switch to collections.abc.MutableMapping https://review.opendev.org/c/openstack/oslo.context/+/773116 | 21:04 |
| *** mgagne has quit IRC | 21:08 | |
| *** dougwig has quit IRC | 21:08 | |
| *** dougwig has joined #openstack-oslo | 21:09 | |
| *** sboyron has quit IRC | 21:10 | |
| openstackgerrit | Merged openstack/oslo.config master: Bump minimum version for PyYAML to 5.1 https://review.opendev.org/c/openstack/oslo.config/+/776481 | 22:09 |
| *** rcernin has joined #openstack-oslo | 23:03 | |
| *** rcernin has quit IRC | 23:11 | |
| *** rcernin has joined #openstack-oslo | 23:12 | |
| *** tosky has quit IRC | 23:31 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!