| *** achanda has joined #openstack-oslo | 00:07 | |
| *** achanda has quit IRC | 00:12 | |
| *** subscope has quit IRC | 00:18 | |
| openstackgerrit | Merged openstack/oslo-incubator: Add tox env for listing unreleased changes https://review.openstack.org/162634 | 00:57 |
|---|---|---|
| *** kiseok7_ has joined #openstack-oslo | 01:11 | |
| *** liusheng has quit IRC | 01:13 | |
| *** alexpilotti has quit IRC | 01:19 | |
| *** harlowja_at_home has joined #openstack-oslo | 01:21 | |
| *** salv-orlando has quit IRC | 01:22 | |
| *** YorikSar has quit IRC | 01:23 | |
| *** mtanino has joined #openstack-oslo | 01:25 | |
| *** amotoki has joined #openstack-oslo | 01:38 | |
| openstackgerrit | Joshua Harlow proposed openstack/taskflow: Just use the class name instead of TYPE constant https://review.openstack.org/164554 | 01:41 |
| openstackgerrit | Merged openstack/oslo.messaging: Fix coverage report generation https://review.openstack.org/163967 | 01:46 |
| openstackgerrit | Merged openstack/oslo.messaging: Fix the bug redis do not delete the expired keys https://review.openstack.org/152471 | 01:48 |
| *** harlowja_at_home has quit IRC | 01:50 | |
| *** bknudson has quit IRC | 01:53 | |
| *** dims_ has quit IRC | 02:13 | |
| *** salv-orlando has joined #openstack-oslo | 02:22 | |
| *** salv-orlando has quit IRC | 02:25 | |
| *** salv-orlando has joined #openstack-oslo | 02:29 | |
| *** salv-orlando has quit IRC | 02:38 | |
| *** hogepodge has joined #openstack-oslo | 02:43 | |
| *** takedakn has joined #openstack-oslo | 02:49 | |
| *** jungleboyj_ has joined #openstack-oslo | 02:53 | |
| *** takedakn has quit IRC | 02:57 | |
| openstackgerrit | Merged openstack/taskflow: Attempt to extract traceback from exception https://review.openstack.org/164519 | 03:00 |
| *** sputnik13 has joined #openstack-oslo | 03:04 | |
| *** kiseok7_ has quit IRC | 03:05 | |
| *** kiseok7_ has joined #openstack-oslo | 03:06 | |
| *** salv-orlando has joined #openstack-oslo | 03:12 | |
| *** dimsum__ has joined #openstack-oslo | 03:14 | |
| *** salv-orlando has quit IRC | 03:16 | |
| *** stevemar has joined #openstack-oslo | 03:19 | |
| *** dimsum__ has quit IRC | 03:19 | |
| *** salv-orlando has joined #openstack-oslo | 03:26 | |
| *** amotoki has quit IRC | 03:28 | |
| *** ChuckC has joined #openstack-oslo | 03:28 | |
| *** salv-orlando has quit IRC | 03:29 | |
| *** salv-orlando has joined #openstack-oslo | 03:32 | |
| *** salv-orlando has quit IRC | 03:32 | |
| *** takedakn has joined #openstack-oslo | 03:35 | |
| *** salv-orlando has joined #openstack-oslo | 03:36 | |
| *** isviridov has quit IRC | 03:38 | |
| *** salv-orlando has quit IRC | 03:38 | |
| *** salv-orlando has joined #openstack-oslo | 03:41 | |
| *** salv-orlando has quit IRC | 03:45 | |
| *** takedakn has quit IRC | 03:48 | |
| *** salv-orlando has joined #openstack-oslo | 03:50 | |
| *** salv-orlando has quit IRC | 03:57 | |
| *** achanda has joined #openstack-oslo | 04:10 | |
| *** salv-orlando has joined #openstack-oslo | 04:19 | |
| *** salv-orlando has quit IRC | 04:24 | |
| *** salv-orlando has joined #openstack-oslo | 04:27 | |
| *** salv-orlando has quit IRC | 04:28 | |
| *** mtanino is now known as mtanino_away | 04:40 | |
| lan | Anyone help review this patch: https://review.openstack.org/#/c/163316/ , this patch fix the problem of "the [DEFAULT]amqp_durable_queues option can't work in config file", thanks! | 05:33 |
| *** stevemar has quit IRC | 05:55 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/oslo.db: Imported Translations from Transifex https://review.openstack.org/164579 | 06:01 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/oslo.utils: Imported Translations from Transifex https://review.openstack.org/160649 | 06:11 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/oslo.messaging: Imported Translations from Transifex https://review.openstack.org/164404 | 06:11 |
| *** achanda has quit IRC | 06:15 | |
| *** salv-orlando has joined #openstack-oslo | 06:17 | |
| *** amotoki has joined #openstack-oslo | 06:21 | |
| *** salv-orlando has quit IRC | 06:23 | |
| *** salv-orlando has joined #openstack-oslo | 06:34 | |
| *** achanda has joined #openstack-oslo | 06:42 | |
| *** amotoki has quit IRC | 06:43 | |
| *** amotoki has joined #openstack-oslo | 06:44 | |
| *** sreshetn1 has joined #openstack-oslo | 06:46 | |
| *** amotoki has quit IRC | 06:48 | |
| *** salv-orlando has quit IRC | 06:50 | |
| *** salv-orlando has joined #openstack-oslo | 06:57 | |
| *** achanda has quit IRC | 07:12 | |
| *** inc0 has joined #openstack-oslo | 07:13 | |
| *** pcaruana has quit IRC | 07:14 | |
| *** mtanino_away has quit IRC | 07:28 | |
| *** e0ne has joined #openstack-oslo | 07:34 | |
| *** andreykurilin_ has joined #openstack-oslo | 07:37 | |
| *** salv-orlando has quit IRC | 07:37 | |
| *** e0ne has quit IRC | 07:39 | |
| *** sreshetn1 has quit IRC | 07:40 | |
| *** amotoki has joined #openstack-oslo | 07:44 | |
| *** liusheng has joined #openstack-oslo | 07:44 | |
| *** andreykurilin_ has quit IRC | 07:56 | |
| *** ihrachyshka has joined #openstack-oslo | 07:58 | |
| *** ihrachyshka has quit IRC | 08:14 | |
| *** dulek has joined #openstack-oslo | 08:16 | |
| *** pblaho has joined #openstack-oslo | 08:19 | |
| *** shardy has joined #openstack-oslo | 08:31 | |
| *** ndipanov has joined #openstack-oslo | 08:33 | |
| *** ihrachyshka has joined #openstack-oslo | 08:35 | |
| *** ajo has joined #openstack-oslo | 08:36 | |
| *** salv-orlando has joined #openstack-oslo | 08:38 | |
| *** dtantsur has joined #openstack-oslo | 08:40 | |
| openstackgerrit | Mehdi Abaakouk proposed openstack/oslo.messaging: rabbit: heartbeat implementation https://review.openstack.org/146047 | 08:41 |
| *** salv-orl_ has joined #openstack-oslo | 08:50 | |
| *** jd__ has quit IRC | 08:50 | |
| *** jd__` has joined #openstack-oslo | 08:50 | |
| *** jd__` is now known as jd__ | 08:51 | |
| *** salv-orlando has quit IRC | 08:51 | |
| *** salv-orl_ has quit IRC | 08:57 | |
| *** ihrachyshka has quit IRC | 08:59 | |
| *** salv-orlando has joined #openstack-oslo | 09:02 | |
| *** YorikSar has joined #openstack-oslo | 09:03 | |
| *** salv-orlando has quit IRC | 09:05 | |
| *** sreshetn1 has joined #openstack-oslo | 09:15 | |
| *** salv-orlando has joined #openstack-oslo | 09:22 | |
| *** salv-orlando has quit IRC | 09:25 | |
| *** salv-orlando has joined #openstack-oslo | 09:34 | |
| *** salv-orlando has quit IRC | 09:37 | |
| *** salv-orlando has joined #openstack-oslo | 09:38 | |
| *** salv-orlando has quit IRC | 09:43 | |
| *** salv-orlando has joined #openstack-oslo | 09:44 | |
| *** dimsum__ has joined #openstack-oslo | 09:46 | |
| *** cdent has joined #openstack-oslo | 09:52 | |
| *** salv-orlando has quit IRC | 09:56 | |
| *** enykeev has joined #openstack-oslo | 09:56 | |
| *** salv-orl_ has joined #openstack-oslo | 09:58 | |
| *** sabari has quit IRC | 10:00 | |
| *** sabari has joined #openstack-oslo | 10:01 | |
| *** salv-orl_ has quit IRC | 10:08 | |
| *** salv-orlando has joined #openstack-oslo | 10:13 | |
| *** salv-orlando has quit IRC | 10:17 | |
| *** salv-orlando has joined #openstack-oslo | 10:22 | |
| *** salv-orlando has quit IRC | 10:29 | |
| *** salv-orlando has joined #openstack-oslo | 10:33 | |
| openstackgerrit | Mehdi Abaakouk proposed openstack/tooz: fix concurrency issue with file lock https://review.openstack.org/164642 | 10:34 |
| sileht | jd__, I got a concurrency bug in tooz: a | 10:34 |
| sileht | https://review.openstack.org/#/c/164642/ | 10:34 |
| sileht | jd__, the driver use a bool to tracking locking inside a process, do we consider GIL is sufficient in case of locking is done is multiple threads ? | 10:36 |
| *** salv-orlando has quit IRC | 10:36 | |
| jd__ | sileht: I guess it's not enough | 10:36 |
| jd__ | the GIL is an implementation detail, not a feature | 10:37 |
| sileht | jd__, I will fix that too, so | 10:37 |
| *** SridharGaddam has joined #openstack-oslo | 10:37 | |
| haypo | sileht: most cpython bytecodes are atomic, but some simple python instructions are compiled to multiple bytecodes | 10:44 |
| haypo | sileht: so it depends on your exact code | 10:44 |
| haypo | sileht, jd__ : (hi) | 10:44 |
| *** dtantsur is now known as dtantsur|bbl | 10:49 | |
| *** salv-orlando has joined #openstack-oslo | 10:51 | |
| *** exploreshaifali has joined #openstack-oslo | 10:51 | |
| *** salv-orlando has quit IRC | 10:56 | |
| *** dims_ has joined #openstack-oslo | 11:02 | |
| *** dimsum__ has quit IRC | 11:04 | |
| *** dims_ has quit IRC | 11:07 | |
| *** dimsum__ has joined #openstack-oslo | 11:13 | |
| *** takedakn has joined #openstack-oslo | 11:14 | |
| *** exploreshaifali has quit IRC | 11:14 | |
| openstackgerrit | Abhijeet Malawade proposed openstack/tooz: fix mysql driver url parsing https://review.openstack.org/159835 | 11:15 |
| *** takedakn has quit IRC | 11:18 | |
| *** takedakn has joined #openstack-oslo | 11:18 | |
| *** rushiagr_away is now known as rushiagr | 11:39 | |
| *** tsufiev_ is now known as tsufiev | 11:42 | |
| *** jaypipes has joined #openstack-oslo | 11:51 | |
| *** _amrith_ is now known as amrith | 11:51 | |
| *** rushiagr is now known as rushiagr_away | 11:55 | |
| *** dtantsur|bbl is now known as dtantsur | 11:57 | |
| *** rushiagr_away is now known as rushiagr | 12:01 | |
| *** kgiusti has joined #openstack-oslo | 12:02 | |
| *** salv-orlando has joined #openstack-oslo | 12:07 | |
| *** sreshetn1 has quit IRC | 12:11 | |
| *** salv-orlando has quit IRC | 12:14 | |
| *** salv-orlando has joined #openstack-oslo | 12:21 | |
| *** salv-orlando has quit IRC | 12:22 | |
| *** salv-orlando has joined #openstack-oslo | 12:22 | |
| *** sreshetn1 has joined #openstack-oslo | 12:25 | |
| *** alexpilotti has joined #openstack-oslo | 12:27 | |
| *** dimsum__ has quit IRC | 12:32 | |
| *** dimsum__ has joined #openstack-oslo | 12:32 | |
| *** subscope has joined #openstack-oslo | 12:37 | |
| *** openstackgerrit has quit IRC | 12:50 | |
| *** openstackgerrit has joined #openstack-oslo | 12:50 | |
| openstackgerrit | Abhijeet Malawade proposed openstack/tooz: fix mysql driver url parsing https://review.openstack.org/159835 | 13:17 |
| jd__ | now = timeutils.parse_isotime(timeutils.strtime()) | 13:19 |
| *** amrith is now known as _amrith_ | 13:25 | |
| *** amotoki has quit IRC | 13:26 | |
| *** bknudson has joined #openstack-oslo | 13:30 | |
| *** ChuckC has quit IRC | 13:33 | |
| *** dimsum__ is now known as dims | 13:33 | |
| dims | bknudson: hi | 13:35 |
| bknudson | dims: what's up? | 13:35 |
| dims | bknudson: is there some code that looks at the incoming http headers and builds up an oslo.context's RequestContext? | 13:35 |
| dims | bknudson: we have this code in Nova, https://github.com/openstack/nova/blob/master/nova/api/auth.py#L119 | 13:36 |
| bknudson | dims: I don't think I've ever looking into that... | 13:36 |
| *** jungleboyj_ has quit IRC | 13:37 | |
| bknudson | dims: so that's getting the fields from the auth_token middleware | 13:37 |
| *** mriedem has joined #openstack-oslo | 13:37 | |
| bknudson | auth_token middleware sets a bunch of fields in the request (X_USER_ID, etc.) | 13:37 |
| dims | y https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py | 13:37 |
| bknudson | right, it's all these guys https://github.com/openstack/keystonemiddleware/blob/master/keystonemiddleware/auth_token/__init__.py#L86 | 13:38 |
| bknudson | if all that nova code is take the auth_token variables and make an oslo RequestContext then no reason that couldn't be shared everywhere | 13:39 |
| bknudson | or, if you want auth_token to make a RequestContext we could do that too. | 13:39 |
| dims | nova code does not pick up domain etc, so wanted to see what else i need to pick up from request headers | 13:39 |
| dims | bknudson: latter would be great | 13:39 |
| dims | "auth_token to make a RequestContext" | 13:39 |
| dims | now that we have an oslo.context standardized | 13:40 |
| *** exploreshaifali has joined #openstack-oslo | 13:41 | |
| dims | once we have that, it will be easy for nova components to query/support domain concepts | 13:41 |
| bknudson | dims: is this something you've got time to work on? | 13:42 |
| bknudson | I could get it done sometime. | 13:42 |
| bknudson | (or, ask jamielennox since he spends a lot of time in auth_token) | 13:43 |
| *** jaosorior has joined #openstack-oslo | 13:43 | |
| dulek | Hi, wanted to ask about lockutils.remove_external_lock_file | 13:44 |
| dulek | The docstring says "Remove an external lock file when it's not used anymore" | 13:44 |
| dims | bknudson: will ping jamielennox later, i was chasing an old bug and realized we still have that hole to fill. thanks! | 13:44 |
| dulek | But I'm able to use this method even if lock is acquired. | 13:45 |
| dulek | Am I missing something here? | 13:45 |
| dims | dulek: you are only supposed to use it when you know the lock is not used. sorry the wording may not have been clear. | 13:46 |
| *** stpierre has joined #openstack-oslo | 13:47 | |
| dulek | dims: Ah, okay... | 13:47 |
| dims | dulek: meant for cleaning up | 13:47 |
| dulek | dims: Any idea how I can delete the lock only if it isn't acquired? | 13:48 |
| dims | dulek: i don't think that scenario has come up before... | 13:48 |
| dulek | dims: I'm trying to write clean up procedure for Cinder, but we have locks working between processes so I would need to be sure that lock isn't held by anyone. | 13:49 |
| dims | dulek: bnemec and dhellmann should be around shortly | 13:50 |
| dulek | dims: Okay, I'll repeat the question then. Thanks! | 13:51 |
| dims | thanks dulek | 13:51 |
| *** _amrith_ is now known as amrith | 13:52 | |
| bnemec | dulek: There isn't really a way to do that. You have to know on your end that the lock is not being used anymore. | 14:00 |
| bnemec | dulek: TBH, I would not worry about cleaning up the files unless it's causing problems. | 14:00 |
| dulek | bnemec: Well, cinder is creating a lockfile per every delete volume request, so this can create a lot of them. | 14:04 |
| dulek | bnemec: There's trylock in _FileLock but I wonder if this is working correctly... | 14:04 |
| bnemec | dulek: Sure, but a modern filesystem can handle a _lot_ of empty files. | 14:04 |
| bnemec | dulek: You could try grabbing the lock and then deleting the file, but I seem to recall we were concerned that might cause issues if anyone else was waiting on the lock. | 14:05 |
| dulek | bnemec: Uhm, I can imagine that situation. | 14:06 |
| dulek | bnemec: Thanks for help anyway! | 14:06 |
| *** takedakn has quit IRC | 14:09 | |
| *** inc0 has quit IRC | 14:11 | |
| *** ChuckC has joined #openstack-oslo | 14:15 | |
| *** jecarey has joined #openstack-oslo | 14:21 | |
| *** jungleboyj_ has joined #openstack-oslo | 14:24 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 14:25 | |
| *** jungleboyj_ is now known as jungleboyj | 14:26 | |
| *** amotoki has joined #openstack-oslo | 14:26 | |
| *** amotoki has quit IRC | 14:32 | |
| *** zz_jgrimm- is now known as jgrimm | 14:35 | |
| *** stevemar has joined #openstack-oslo | 14:40 | |
| *** ChuckC has quit IRC | 14:43 | |
| *** sreshetn1 has quit IRC | 14:46 | |
| *** sreshetn1 has joined #openstack-oslo | 14:52 | |
| *** gordc has joined #openstack-oslo | 14:59 | |
| *** zzzeek has joined #openstack-oslo | 15:01 | |
| *** rushiagr is now known as rushiagr_away | 15:02 | |
| *** prad has joined #openstack-oslo | 15:07 | |
| *** david-lyle_afk is now known as david-lyle | 15:15 | |
| *** achanda has joined #openstack-oslo | 15:17 | |
| *** tsekiyama has joined #openstack-oslo | 15:22 | |
| *** achanda has quit IRC | 15:25 | |
| *** rushiagr_away is now known as rushiagr | 15:30 | |
| *** roaet has joined #openstack-oslo | 15:37 | |
| roaet | anyone have any tips for migrating away from the ArgsAlreadyParsedError: arguments already parsed: cannot register CLI option | 15:38 |
| roaet | I believe that's oslo.config | 15:39 |
| roaet | It'd be pretty sweet if that exception would throw out which CLI option failed to load. | 15:39 |
| *** achanda has joined #openstack-oslo | 15:48 | |
| dhellmann | roaet: "migrating away from"? | 15:49 |
| roaet | dhellmann: nevermind I need to just fix a lot of things | 15:51 |
| roaet | dhellmann: I didn't realize how big of a change that was. it broke quite a bit of things. | 15:51 |
| dhellmann | roaet: ok | 15:52 |
| *** exploreshaifali has quit IRC | 15:52 | |
| *** rushiagr is now known as rushiagr_away | 15:53 | |
| *** harlowja_at_home has joined #openstack-oslo | 15:56 | |
| harlowja_at_home | hmmm, is there a meeting today | 16:01 |
| bnemec | dhellmann: ^ | 16:01 |
| *** rushiagr_away is now known as rushiagr | 16:02 | |
| *** achanda has quit IRC | 16:04 | |
| *** dulek has quit IRC | 16:11 | |
| *** openstackgerrit has quit IRC | 16:11 | |
| *** openstackgerrit has joined #openstack-oslo | 16:11 | |
| *** amotoki has joined #openstack-oslo | 16:16 | |
| *** amotoki has quit IRC | 16:21 | |
| toabctl | zzzeek: you added recently (6ccea346) testresources and testscenarios to requirements.txt in oslo.db . shouldn't that be in test-requirements.txt ? | 16:24 |
| zzzeek | toabctl: no because those packages are part of the oslo_db/sqlalchemy/test_utils package which is used by the test suites of other libraries | 16:25 |
| zzzeek | toabctl: hence those other libs have oslo.db in their test requirements (but really they’d have oslo.db in their main requirements) | 16:25 |
| zzzeek | there was some controversy over whether those packages should also be required to add testresources/testscenarios to their own test-requirements.txt or if oslo.db should be spun out into oslo.db.test | 16:26 |
| zzzeek | i think there may be a plan for oslo.db.test but not for kilo | 16:26 |
| toabctl | zzzeek: I guess you mean oslo_db/sqlalchemy/test_base.py ... | 16:27 |
| zzzeek | toabctl: yes | 16:27 |
| zzzeek | dhellmann: I know we’re in meeting, but for later, ^^^ | 16:29 |
| dims | zzzeek: sileht: dhellmann: please bless https://review.openstack.org/#/c/164767/ | 16:31 |
| toabctl | zzzeek: hm. the main problem seems that oslo_db/sqlalchemy/provision.py uses | 16:31 |
| dims | backport zzzeek's commit to stable/kilo | 16:31 |
| toabctl | testresources. | 16:31 |
| *** salv-orlando has quit IRC | 16:31 | |
| zzzeek | toabctl: yeah, this whole series of fixes took months of back and forth on the list to get committed | 16:31 |
| toabctl | zzzeek: doesn't that mean that I need to install testresources in a production env? | 16:32 |
| zzzeek | toabctl: so its awesome that people have an issue after all that discussion is done ! :) | 16:32 |
| zzzeek | toabctl: yes | 16:32 |
| zzzeek | toabctl: feel fere to raise on the list, whatever | 16:32 |
| toabctl | zzzeek: ok. I guess the discussion was in the gerrit review? | 16:32 |
| zzzeek | toabctl: it was everywehre. i had a whole solution without any testresources/testsceanrios ready to go, lifeless -1’ed it, etc. | 16:33 |
| toabctl | :) | 16:33 |
| zzzeek | toabctl: the key issue is that all the “help me with tests” stuff in oslo.db should be in a new package | 16:33 |
| toabctl | zzzeek: I haven't followed the discussions. I just recognized this fact while I was packaging the new version for openSUSE. | 16:33 |
| zzzeek | toabctl: so, im not sure if smoeone has proposed taht already, or waht | 16:33 |
| dims | zzzeek: toabctl: i started down that path - https://review.openstack.org/#/c/162276/ | 16:35 |
| dims | and then it turned into the discussion of how best to specify optional dependencies and test fixture dependencies... | 16:36 |
| zzzeek | dims: i think test_base.py should be moved out entirely | 16:36 |
| openstackgerrit | Joshua Harlow proposed openstack/oslo.utils: Provide common `fetch_current_thread_functor` function https://review.openstack.org/164771 | 16:41 |
| *** exploreshaifali has joined #openstack-oslo | 16:44 | |
| *** cdent has quit IRC | 16:52 | |
| *** openstackgerrit has quit IRC | 16:54 | |
| *** openstackgerrit has joined #openstack-oslo | 16:54 | |
| *** mtanino has joined #openstack-oslo | 16:59 | |
| *** harlowja_at_home has quit IRC | 17:00 | |
| *** yamahata has quit IRC | 17:00 | |
| *** dtantsur is now known as dtantsur|afk | 17:02 | |
| *** salv-orlando has joined #openstack-oslo | 17:07 | |
| *** zzzeek has quit IRC | 17:12 | |
| *** amrith is now known as _amrith_ | 17:15 | |
| *** harlowja has joined #openstack-oslo | 17:28 | |
| *** klindgren has joined #openstack-oslo | 17:35 | |
| *** rushiagr is now known as rushiagr_away | 17:36 | |
| *** _amrith_ is now known as amrith | 17:38 | |
| *** saikrishna has joined #openstack-oslo | 17:38 | |
| *** yamahata has joined #openstack-oslo | 17:38 | |
| *** jungleboyj has quit IRC | 17:46 | |
| *** jungleboyj has joined #openstack-oslo | 17:49 | |
| *** achanda has joined #openstack-oslo | 17:50 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 17:51 | |
| *** achanda_ has joined #openstack-oslo | 17:55 | |
| *** salv-orl_ has joined #openstack-oslo | 17:57 | |
| *** achanda has quit IRC | 17:58 | |
| *** yamahata has quit IRC | 18:00 | |
| *** yamahata has joined #openstack-oslo | 18:00 | |
| *** salv-orlando has quit IRC | 18:01 | |
| klindgren | I am doing testing on https://review.openstack.org/#/c/146047/29 and in gerneal this makes things *much* better for dealing with failovers. Though in failover testing I am running into a single problem | 18:02 |
| klindgren | I am seeing errors during _publish | 18:02 |
| klindgren | Occassionally. I have two rabbitmq servers behind haproxy. I will set the rabbitmq server that I want to drain connections from in maintenace mode in haproxy | 18:03 |
| klindgren | then restart rabbitmq on that host | 18:03 |
| klindgren | most of the connections will switch over to the second host in my test setup ~330 active rabbit connections | 18:04 |
| klindgren | ~320 will move over and ~10 stay stuck to the old rabbit server | 18:04 |
| klindgren | when I look at these connections on the hosts that have them open they are in a CLOSED_WAIT state | 18:05 |
| klindgren | Errors are logged like the following: AMQP server on <vip>:5671 is unrechable: connection already closed. Trying again in 1 seconds | 18:06 |
| klindgren | https://gist.github.com/krislindgren/2d6527d876ae10346862 | 18:09 |
| *** amrith is now known as _amrith_ | 18:10 | |
| klindgren | Eventually the connection gets closed and the errors stop happening | 18:10 |
| klindgren | Which is when I see: https://gist.github.com/krislindgren/2d6527d876ae10346862#file-gistfile1-txt | 18:12 |
| *** _amrith_ is now known as amrith | 18:13 | |
| dims | klindgren: does that ssl error stop after some time? | 18:16 |
| *** rushiagr_away is now known as rushiagr | 18:17 | |
| harlowja | sileht ^ | 18:17 |
| klindgren | yea - the ssl error seems to be the error that happens when it finally realized the connection was closed and did something different - after that ssl error - their were no more errors | 18:17 |
| dims | ok, so if we mute the stack trace, we are good then? | 18:18 |
| dims | print something similar to the "connection already closed" in the logs instead | 18:19 |
| *** sreshetn1 has quit IRC | 18:19 | |
| klindgren | I dont think so. The issue is that for some period of time something (everything that I saw was a call to conductor in nova) gets stuck trying to send a message using a closed channel - which eventually after many many minutes recovers | 18:20 |
| dims | is it easy to recreate this klindgren? | 18:21 |
| *** jungleboyj has quit IRC | 18:21 | |
| *** jungleboyj has joined #openstack-oslo | 18:22 | |
| klindgren | I think so - let me do a few more failovers just to make sure. It happened twice during the two failover tests I have done | 18:23 |
| *** saikrishna has quit IRC | 18:25 | |
| *** jecarey has quit IRC | 18:25 | |
| dims | klindgren: could there be something sticky in your haproxy settings? | 18:27 |
| dims | "~10 stay stuck to the old rabbit server" | 18:27 |
| dims | klindgren: will you be able to share snippets? | 18:31 |
| klindgren | yea - I can share whatever | 18:32 |
| klindgren | checking the haproxy stuff - but I dont think anything would specifically send stuff there if the backend was set to maintenace mode | 18:33 |
| *** jungleboyj has quit IRC | 18:38 | |
| *** jungleboyj has joined #openstack-oslo | 18:39 | |
| harlowja | haypo i merged some of your reasons for why eventlet isn't good into https://review.openstack.org/#/c/164035/ + a few more of my own releated to wsgi servers/containers... if u get some time check it out, lifeless also, or any others that wish to read some stuff, lol | 18:39 |
| *** sreshetn1 has joined #openstack-oslo | 18:40 | |
| dhellmann | dims: +2a | 18:45 |
| *** jecarey has joined #openstack-oslo | 18:47 | |
| openstackgerrit | Merged openstack/oslo.utils: Imported Translations from Transifex https://review.openstack.org/160649 | 18:50 |
| *** amrith is now known as _amrith_ | 18:53 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 19:04 | |
| *** jecarey has quit IRC | 19:08 | |
| *** sputnik13 has quit IRC | 19:14 | |
| *** jecarey has joined #openstack-oslo | 19:18 | |
| *** _amrith_ is now known as amrith | 19:24 | |
| openstackgerrit | Joshua Harlow proposed openstack/oslo-incubator: Recommend users of service.py use `launch_service_cls` https://review.openstack.org/164836 | 19:35 |
| harlowja | dhellmann ^ might just do the trick, now if people will just use it :-P | 19:36 |
| klindgren | dims, https://gist.github.com/krislindgren/9a66dd02e0619d6cdbcc | 19:37 |
| *** mriedem has quit IRC | 19:39 | |
| dhellmann | harlowja: I like the general idea, but I'll have to give it a closer review. An example in the docstring for that module would be useful. | 19:39 |
| harlowja | sureee | 19:40 |
| *** rushiagr is now known as rushiagr_away | 19:42 | |
| *** mriedem has joined #openstack-oslo | 19:47 | |
| lifeless | harlowja: ack aye | 19:48 |
| *** alexpilotti has quit IRC | 19:50 | |
| *** amotoki has joined #openstack-oslo | 19:53 | |
| *** salv-orl_ has quit IRC | 19:55 | |
| *** sputnik13 has joined #openstack-oslo | 19:56 | |
| *** amotoki has quit IRC | 19:58 | |
| *** ajo has quit IRC | 19:58 | |
| harlowja | dhellmann ok; example @ http://paste.ubuntu.com/10611676/ | 19:58 |
| harlowja | although guess i could include that whole thing in the docstring, lol | 19:58 |
| *** ajo has joined #openstack-oslo | 19:58 | |
| dhellmann | harlowja: yeah, the docstring is a good place for it now but we can move it to a separate doc file when we graduate the module (we don't have good external docs for a lot of the incubator code) | 20:03 |
| *** sreshetn1 has quit IRC | 20:06 | |
| *** rushiagr_away is now known as rushiagr | 20:07 | |
| *** tsufiev is now known as tsufiev_ | 20:10 | |
| *** sreshetn1 has joined #openstack-oslo | 20:13 | |
| *** ndipanov has quit IRC | 20:16 | |
| *** alexpilotti has joined #openstack-oslo | 20:19 | |
| openstackgerrit | Doug Hellmann proposed openstack/oslo-specs: Change oslo.log to keep other libraries quiet in the logs https://review.openstack.org/164851 | 20:19 |
| *** jungleboyj has quit IRC | 20:21 | |
| *** jungleboyj has joined #openstack-oslo | 20:22 | |
| dims | klindgren: thanks looking | 20:23 |
| *** kgiusti has left #openstack-oslo | 20:23 | |
| *** jungleboyj has quit IRC | 20:27 | |
| *** rushiagr is now known as rushiagr_away | 20:27 | |
| *** jungleboyj has joined #openstack-oslo | 20:28 | |
| *** jungleboyj has quit IRC | 20:33 | |
| *** jungleboyj has joined #openstack-oslo | 20:33 | |
| *** ajo has quit IRC | 20:34 | |
| *** salv-orlando has joined #openstack-oslo | 20:35 | |
| dims | klindgren: the line numbers in the stack trace don't match up to the line numbers in the commit (https://review.openstack.org/#/c/146047/29/oslo_messaging/_drivers/impl_rabbit.py,cm) can you please look? | 20:37 |
| *** e0ne has joined #openstack-oslo | 20:39 | |
| *** e0ne has quit IRC | 20:49 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/oslo.versionedobjects: Updated from global requirements https://review.openstack.org/160918 | 20:50 |
| *** andreykurilin_ has joined #openstack-oslo | 20:50 | |
| dims | klindgren: please review this information as well :) http://john.eckersberg.com/improving-ha-failures-with-tcp-timeouts.html | 20:53 |
| dhellmann | dims, rpodolyaka, zzzeek: the backport in oslo.db landed so I'll cut 1.7.1 from the stable/kilo branch tomorrow | 20:54 |
| dims | klindgren: could explain some of what you see | 20:54 |
| dims | dhellmann: sounds good, and rev the g-r update too? | 20:54 |
| dhellmann | dims: no, as a bug fix we don't need to bump the minimum again | 20:55 |
| dims | ah ok | 20:55 |
| dhellmann | we only need to do that to ensure that new features are present | 20:55 |
| dhellmann | if a deployer has an issue fixed in a later release, we can just tell them to update the lib, but if they get the wrong version on a fresh install so something doesn't work at all it looks bad | 20:55 |
| *** jaypipes has quit IRC | 20:56 | |
| *** jaypipes has joined #openstack-oslo | 20:57 | |
| *** harlowja is now known as harlowja_away | 20:58 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 21:00 | |
| openstackgerrit | Doug Hellmann proposed openstack/oslo-specs: Change oslo.log to keep other libraries quiet in the logs https://review.openstack.org/164851 | 21:06 |
| *** jgrimm is now known as zz_jgrimm | 21:07 | |
| *** amotoki has joined #openstack-oslo | 21:09 | |
| *** andreykurilin_ has quit IRC | 21:09 | |
| *** sreshetn1 has quit IRC | 21:14 | |
| klindgren | dims - I have seen htis before... but https://gist.githubusercontent.com/krislindgren/9a66dd02e0619d6cdbcc/raw/286e961a2e3680a2363838dda32a1b0b1dbcd6b1/impl_rabbit.py is the code from the server | 21:16 |
| bknudson | dhellmann: stevedore question -- what do you think about https://github.com/openstack/neutron/blob/master/neutron/manager.py#L130-L143 ? | 21:17 |
| bknudson | and https://bugs.launchpad.net/python-keystoneclient/+bug/1425345 | 21:17 |
| openstack | Launchpad bug 1425345 in python-keystoneclient "Can't load auth_plugin by full-class-name of plugin class" [Wishlist,In progress] - Assigned to Yuki Nishiwaki (uckey-1067) | 21:17 |
| bknudson | feature for stevedore? | 21:17 |
| klindgren | and https://gist.github.com/krislindgren/9a66dd02e0619d6cdbcc#file-impl_rabbit-errors-to-commit is a translation of the lines from that file to the line numbers in the error | 21:18 |
| dhellmann | bknudson: I don't understand what the first function is doing | 21:18 |
| dhellmann | oh, it's taking a specifier for a class name? | 21:18 |
| bknudson | dhellmann: if stevedore doesn't load it then it imports the mod. | 21:18 |
| dhellmann | bknudson: that's not needed. A plugin can be registered with that name pointing to itself and pkg_resources will do the load | 21:19 |
| klindgren | dims - I know about ha failures with tcp timeouts. But I thought the point of this commit to was to enable heartbeats so that the connection can see that it has not seen or replied with a heartbeat after x amount of seconds and will be able to detect a broken channel and take a corrective action? | 21:19 |
| dhellmann | bknudson: when we first moved to oslo.messaging that's how we handled backwards compatibility for the old strings that pointed to the rpc modules that lived in nova, for instance | 21:20 |
| klindgren | It seems that while it mostly does this - their does end up being an issue with services that get "stuck" on a connection that is closed from server but the client is still keeping it open/trying to use it. This should be caught by the heart beating and fixed no? | 21:20 |
| bknudson | dhellmann: it's not a feature you want to see in stevedore? | 21:20 |
| bknudson | or a pattern you want to see copy-pasted all over? | 21:21 |
| *** sreshetn1 has joined #openstack-oslo | 21:21 | |
| dhellmann | bknudson: no, it can be handled without any code at all - just register the class specifier as the name of a plugin | 21:21 |
| dims | klindgren: you are one patchset behind :) | 21:21 |
| dhellmann | bknudson: for example, set up a plugin like: mypackage.mymodule.myclass = mypackage.mymodule:myclass | 21:23 |
| dims | klindgren: can you please rev-up to changeset 29 and try the same exact steps and let me know? | 21:23 |
| bknudson | dhellmann: can you do that in code, or do I have to edit a file? | 21:23 |
| bknudson | this is the part I'm always confused about. | 21:23 |
| *** jungleboyj has quit IRC | 21:23 | |
| dhellmann | bknudson: the package that installs the plugin should have ^^ in its setup.cfg in the appropriate namespace | 21:23 |
| klindgren | dism - ok - checking.... I am pretty sure at thats changed is only enabling heartbeating if everythign supports it - otherwise it appears to be the same | 21:24 |
| bknudson | so in this case the package is python-keystoneclient, and the plugin is one that the user just has on their system. | 21:24 |
| bknudson | we don't know about it ahead of time. | 21:24 |
| *** YorikSar has quit IRC | 21:25 | |
| dhellmann | bknudson: so it's the responsibility of the plugin to keystoneclient to set this up, then | 21:25 |
| bknudson | is setup.cfg a config file, like in etc/? | 21:25 |
| bknudson | ok, as long as the plugin can do it that works. | 21:26 |
| dhellmann | bknudson: no it's part of the packaging files for the thing that's being installed | 21:26 |
| bknudson | wonders how a plugin sets it up. | 21:26 |
| dhellmann | bknudson: http://git.openstack.org/cgit/openstack/oslo.messaging/tree/setup.cfg#n32 | 21:27 |
| bknudson | right, so say I'm implementing a oslo.messaging driver out of tree... | 21:27 |
| bknudson | how do I get myself in http://git.openstack.org/cgit/openstack/oslo.messaging/tree/setup.cfg#n32 ? | 21:28 |
| bknudson | for example, wouldn't it be nice to get the "fake" driver out of setup.cfg? | 21:28 |
| dhellmann | you don't have to -- the namespaces for plugins are global across all installed packages, so you can declare in your own packaging metadata that you have a plugin that is part of that namespace | 21:28 |
| bknudson | ahhh | 21:28 |
| dhellmann | the fact that we use library/module names as namespaces is a little confusing | 21:28 |
| dhellmann | the two things are actually completely separate | 21:29 |
| bknudson | so I just have an [entry_points] with mydriver = mydrvier:MyDriver in mydriver's setup.cfg | 21:29 |
| dhellmann | bknudson: https://etherpad.openstack.org/p/GHG6Kl8hCD | 21:30 |
| * dhellmann should really blog about entry points | 21:30 | |
| dhellmann | bknudson: make more sense? | 21:32 |
| bknudson | dhellmann: yes that makes total sense now. | 21:32 |
| *** stpierre has quit IRC | 21:32 | |
| bknudson | I was mostly confused about how you could register your plugins from a different setup.cfg. | 21:32 |
| *** mriedem is now known as mriedem_away | 21:33 | |
| dhellmann | yeah, that's not obvious if you don't understand that the namespaces are global | 21:33 |
| dhellmann | I need to add an example to the stevedore docs showing how to create a plugin out of tree | 21:33 |
| bknudson | that would be great. | 21:34 |
| dhellmann | it's on my todo list, so maybe it'll happen before the summit | 21:35 |
| dhellmann | sdague: you may be interested in this logging-related spec: https://review.openstack.org/164851 | 21:35 |
| bknudson | actually, we've got an example: https://github.com/openstack/python-openstackclient/blob/master/setup.cfg#L30 | 21:35 |
| dhellmann | sdague: I think we talked about that in paris | 21:35 |
| dhellmann | bknudson: yeah, that's it | 21:36 |
| *** cdent has joined #openstack-oslo | 21:38 | |
| *** sreshetn1 has quit IRC | 21:44 | |
| *** sigmavirus24_awa is now known as sigmavirus24 | 21:47 | |
| *** jamielennox has joined #openstack-oslo | 21:50 | |
| *** zzzeek has joined #openstack-oslo | 21:50 | |
| *** prad has quit IRC | 21:53 | |
| *** cdent has quit IRC | 21:58 | |
| *** harlowja_away is now known as harlowja | 22:11 | |
| klindgren | dims, Still happens | 22:12 |
| *** gordc has quit IRC | 22:12 | |
| klindgren | dims, https://gist.github.com/krislindgren/6a05e91263801de94ca1 | 22:13 |
| *** stevemar has quit IRC | 22:13 | |
| klindgren | I included both the failures when I restarted rabbitmq and then what happened 4 minutes later when nova-compute tried to publish something to conductor | 22:13 |
| *** sputnik13 has quit IRC | 22:21 | |
| *** bknudson has quit IRC | 22:22 | |
| openstackgerrit | Joshua Harlow proposed openstack/oslo-incubator: Recommend users of service.py use `launch_service_class` https://review.openstack.org/164836 | 22:22 |
| openstackgerrit | Joshua Harlow proposed openstack/oslo-incubator: Recommend users of service.py use `launch_service_class` https://review.openstack.org/164836 | 22:24 |
| openstackgerrit | Joshua Harlow proposed openstack/oslo.utils: Provide common `fetch_current_thread_functor` function https://review.openstack.org/164771 | 22:33 |
| *** salv-orl_ has joined #openstack-oslo | 22:37 | |
| *** salv-orlando has quit IRC | 22:39 | |
| dims | klindgren: one of the stack traces changed slightly | 22:42 |
| dims | back shortly | 22:43 |
| *** dims has quit IRC | 22:43 | |
| *** sigmavirus24 is now known as sigmavirus24_awa | 22:43 | |
| openstackgerrit | Joshua Harlow proposed openstack/oslo-incubator: Recommend users of service.py use `launch_service_class` https://review.openstack.org/164836 | 22:46 |
| *** dims has joined #openstack-oslo | 22:46 | |
| *** dims has quit IRC | 22:47 | |
| *** dims has joined #openstack-oslo | 22:47 | |
| *** tsekiyam_ has joined #openstack-oslo | 22:52 | |
| *** YorikSar has joined #openstack-oslo | 22:53 | |
| *** tsekiyama has quit IRC | 22:55 | |
| *** tsekiyam_ has quit IRC | 22:57 | |
| *** amotoki has quit IRC | 23:01 | |
| *** jecarey has quit IRC | 23:05 | |
| *** ChuckC has joined #openstack-oslo | 23:09 | |
| lifeless | dhellmann: did you see my thing about requirements-py3 being harmful ? | 23:11 |
| *** jaosorior has quit IRC | 23:12 | |
| *** dims has quit IRC | 23:18 | |
| *** dims has joined #openstack-oslo | 23:18 | |
| *** andreykurilin_ has joined #openstack-oslo | 23:20 | |
| harlowja | lifeless then i guess u are ok with https://review.openstack.org/#/c/164523/ :-P | 23:20 |
| *** salv-orl_ has quit IRC | 23:23 | |
| lifeless | yes | 23:24 |
| harlowja | ;) | 23:25 |
| harlowja | thought so, ha | 23:25 |
| lifeless | can't have universal wheels otherwise | 23:28 |
| harlowja | ya | 23:31 |
| *** salv-orlando has joined #openstack-oslo | 23:31 | |
| *** salv-orlando has quit IRC | 23:35 | |
| *** jaypipes has quit IRC | 23:41 | |
| *** andreykurilin_ has quit IRC | 23:48 | |
| dims | klindgren: still around? | 23:50 |
| klindgren | dims, I am | 23:50 |
| *** sputnik13 has joined #openstack-oslo | 23:51 | |
| dims | am scratching my head trying to figure out who logs this traceback at the TRACE level - https://gist.github.com/krislindgren/6a05e91263801de94ca1#file-fail-after-reconnect-L40 | 23:51 |
| dims | klindgren: i can see where we print the line just before it... | 23:52 |
| *** bknudson has joined #openstack-oslo | 23:55 | |
| lifeless | harlowja: dhellmann: mailed the list for visibility | 23:56 |
| harlowja | k | 23:56 |
| harlowja | lifeless thx | 23:57 |
| *** alexpilotti has quit IRC | 23:57 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!