| *** gus has joined #openstack-oslo | 00:21 | |
| gus | zzzeek: (If you're around) - I was wondering if you'd seen https://review.openstack.org/#/c/104111/ | 00:22 |
|---|---|---|
| gus | I wrote a test that tries to reproduce the transaction/eventlet deadlocks that neutron has been suffering from. | 00:23 |
| gus | it fails on sqlite (since you can't open an sqlite database twice), and fails on mysqldb (due to the eventlet deadlock). | 00:24 |
| gus | fyi, it passes with mysqlconnector - and presumably others but I haven't done an audit. | 00:24 |
| gus | I appreciate the desire to support mysqldb - I fear the only way to do so is to take a python/eventlet lock before entering a mysqldb transaction. This will be expensive, since we'll be locking the entire database (or tables?) whereas a more knowledgeable database transaction would only lock on the relevant row range. But if we limit it to only mysqldb, that won't penalise other drivers. | 00:29 |
| gus | In particular, the neutron schedule-actions-for-db-transaction-closed bp is inadequate, since it only deals with *known* cases of overlapping transactions. | 00:30 |
| gus | You could still get (eg) two simultaneous user requests to modify the same row and you would still deadlock (aiui). | 00:31 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Add proxy heartbeat support https://review.openstack.org/104397 | 00:50 |
| zzzeek | gus: looking | 00:56 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Add proxy heartbeat support https://review.openstack.org/104397 | 00:57 |
| zzzeek | gus: last night was talking to someone and we established that overlapping transactions w/ eventlet is not really feasible unless a eventlet-compat driver were used | 00:59 |
| zzzeek | gus: anytime MySQL locks a row, we can deadlock against eventlret | 00:59 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Add proxy heartbeat support https://review.openstack.org/104397 | 00:59 |
| zzzeek | gus: disturbing here is that you’re getting a lock w/o using REPEATABLE READ or SELECT FOR UPDATE | 00:59 |
| gus | yeah, I'm doing an INSERT INTO with the same value. I'm no DB expert, but I assume it still needs to lock between those two conflicting updates. | 01:01 |
| harlowja | zzzeek do u think its worthwhile for openstack to seriously consider pymysql (or other) | 01:02 |
| harlowja | it starts to feel that we should seriously start asking that question | 01:02 |
| zzzeek | harlowja: i just commented there, why cant we fork / enhance mysql-connector or mysqldb | 01:03 |
| zzzeek | we dont need to be pure python to do this | 01:03 |
| zzzeek | gus: i htink the reason it deadlocks is because thread A locks, does something to defer to thread B, B hits the lock and blocks, deadlock, because eventlet + mysqldb doesnt know that “lock” means “defer" | 01:03 |
| zzzeek | its such a stupid issue | 01:04 |
| zzzeek | its entirely inapprorpiate how openstack does this right now | 01:04 |
| harlowja | zzzeek i know https://review.openstack.org/#/c/73437/ was tried (getting pymysql into openstack requirements) | 01:05 |
| zzzeek | yeah | 01:05 |
| gus | zzzeek: what are your thoughts on my suggestion above? I really do think the only way out is to 1) Say mysqldb is unsupported and use some other mysql driver, or 2) use one of the hooks to take a python/eventlet lock before mysqldb tries to block on the DB itself. | 01:07 |
| zzzeek | the problem with #1 is that the only compatible drivers are pure python so are way slower in all other ways | 01:08 |
| gus | oh ok, I don't have any experience really with the alternatives. | 01:08 |
| zzzeek | #2 is a bandaid for now, lock eventlets with transactions | 01:08 |
| gus | I know mysqlconnector works. | 01:09 |
| zzzeek | gus: it does ?! | 01:09 |
| gus | #2 is going to be real expensive (for mysqldb), since we'd have to lock the entire db/table | 01:09 |
| zzzeek | gus: its written in C, isnt it? | 01:09 |
| gus | it passes my simple test, yes. | 01:09 |
| zzzeek | gus: im +100 on mysqlconnector because its mysql’s official driver, however, need to understand why it “works” | 01:09 |
| gus | yes, but it does something more clever with eventlets.. | 01:09 |
| zzzeek | because eventlet needs to patch the socket | 01:10 |
| gus | let me dig up the stuff I was reading yesterday ... | 01:10 |
| gus | oh right, mysqlconnector is pure python | 01:12 |
| harlowja | ya, interseting that mysqlconnector is that much faster when its also pure python | 01:13 |
| * harlowja just skimming over the mysqlconnector code and it doesn't seem like anything special | 01:14 | |
| gus | are there speed benchmarks somewhere? I'm similarly surprised there's such a difference. | 01:15 |
| gus | aiui, all these are basically just copying bytes to/from a mysql TCP socket. | 01:15 |
| harlowja | * http://pydoc.net/Python/mysql-connector-python/1.1.6/ | 01:15 |
| harlowja | ^ just the code (without requiring a download) | 01:16 |
| zzzeek | gus + harlowja : just added https://wiki.openstack.org/wiki/OpenStack_and_SQLAlchemy#MySQLdb_.2B_eventlet_.3D_sad | 01:18 |
| harlowja | nice | 01:19 |
| harlowja | +2 for just solving, even if we take a hit with 'its slower', i like the path of correctness first then optimize | 01:20 |
| zzzeek | yup | 01:21 |
| zzzeek | e.g. pypy :) | 01:21 |
| harlowja | someday that will happen, ha | 01:22 |
| harlowja | mysqldb will have bigger issues with py3 + asyncio anyway | 01:22 |
| harlowja | even if they fix the current problem | 01:22 |
| zzzeek | i hate hacky things | 01:23 |
| harlowja | u won't get disagreement from me, ha | 01:23 |
| gus | zzzeek: +1 from me too. The bp/schedule-actions-for-db-transaction-closed route seemed to be piling on significant complexity, without actually fixing the issue in all cases. | 01:24 |
| zzzeek | this stuff woudl run great if eventlet were allowed to do its job, otherwise its totally dumb to be using async libraries | 01:24 |
| zzzeek | i dont see much in openstack that is CPU bound | 01:24 |
| zzzeek | on the python side | 01:24 |
| harlowja | checksumming every now and then | 01:24 |
| gus | (even with eventlet, you still can't use more than one core which is sad in this day and age) | 01:24 |
| harlowja | checksumming and string concat, thats pretty much the CPU bound things, lol | 01:25 |
| gus | well, and latency. | 01:25 |
| gus | with eventlet, you need to wait for your turn on the one-true cpu. | 01:25 |
| harlowja | ya, i've always considered eventlet like a temptress | 01:26 |
| harlowja | oh so sexxyyyy | 01:26 |
| harlowja | ahhhh | 01:26 |
| harlowja | and then thats how u get here, lol | 01:26 |
| harlowja | and then u have to figure out how to get rid of all those dieases and stuff after the fact | 01:27 |
| openstackgerrit | Angus Lees proposed a change to openstack/oslo-incubator: Add DBDuplicateEntry detection for mysqlconnector driver https://review.openstack.org/104405 | 01:28 |
| harlowja | and then u come and use taskflow, and it saves u | 01:28 |
| harlowja | hahaha | 01:28 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/oslo.messaging: Add a thread + futures executor based executor https://review.openstack.org/70914 | 01:42 |
| openstackgerrit | Angus Lees proposed a change to openstack/oslo-incubator: Allow tox tests with complex OS_TEST_DBAPI_CONNECTION URLs https://review.openstack.org/104410 | 01:46 |
| *** zzzeek has quit IRC | 02:07 | |
| openstackgerrit | Angus Lees proposed a change to openstack/oslo-incubator: Don't drop pre-existing database before tests https://review.openstack.org/104419 | 02:27 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Add proxy heartbeat support https://review.openstack.org/104397 | 02:52 |
| openstackgerrit | Angus Lees proposed a change to openstack/oslo.db: Add DBDuplicateEntry detection for mysqlconnector driver https://review.openstack.org/104425 | 02:57 |
| openstackgerrit | Angus Lees proposed a change to openstack/oslo.db: Allow tox tests with complex OS_TEST_DBAPI_CONNECTION URLs https://review.openstack.org/104428 | 03:06 |
| *** SridharG has joined #openstack-oslo | 03:10 | |
| *** praneshp has quit IRC | 03:12 | |
| openstackgerrit | Angus Lees proposed a change to openstack/oslo.db: Don't drop pre-existing database before tests https://review.openstack.org/104430 | 03:15 |
| openstackgerrit | Angus Lees proposed a change to openstack/oslo-incubator: Test that concurrent sqlalchemy transactions don't block https://review.openstack.org/104111 | 03:25 |
| gus | (sorry for the autospam - I'm moving all those changes to oslo.db) | 03:25 |
| *** dims has quit IRC | 03:32 | |
| *** mriedem has left #openstack-oslo | 03:34 | |
| *** mriedem has quit IRC | 03:34 | |
| harlowja | we love the autospam :-P | 03:38 |
| openstackgerrit | A change was merged to openstack/taskflow: Add a tree type https://review.openstack.org/97325 | 03:40 |
| openstackgerrit | A change was merged to openstack/taskflow: Create a top level time type https://review.openstack.org/101681 | 03:41 |
| *** zhiyan_ is now known as zhiyan | 03:47 | |
| *** zzzeek has joined #openstack-oslo | 03:54 | |
| openstackgerrit | Angus Lees proposed a change to openstack/oslo.db: Test that concurrent sqlalchemy transactions don't block https://review.openstack.org/104436 | 03:55 |
| *** dstanek is now known as dstanek_zzz | 03:57 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Increase the logging of the WBE response/send activities https://review.openstack.org/102959 | 03:57 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Tidy up the WBE cache (now WBE types) module https://review.openstack.org/103245 | 03:57 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Improve WBE testing coverage https://review.openstack.org/101717 | 04:11 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Use a common message dispatcher https://review.openstack.org/101094 | 04:11 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Allow handlers to provide validation callables https://review.openstack.org/101679 | 04:11 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: WBE notification message validation https://review.openstack.org/103657 | 04:17 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: WBE response message validation https://review.openstack.org/103690 | 04:17 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: WBE request message validation https://review.openstack.org/103695 | 04:17 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Add basic WBE validation sanity tests https://review.openstack.org/104016 | 04:17 |
| *** ajc_ has joined #openstack-oslo | 04:17 | |
| harlowja | oh the autorebase spam fun | 04:17 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Add proxy heartbeat support https://review.openstack.org/104397 | 04:43 |
| openstackgerrit | Angus Lees proposed a change to openstack/oslo.db: Support OS_TEST_DBAPI_ADMIN_CONNECTION override https://review.openstack.org/104447 | 04:48 |
| *** ihrachyshka has joined #openstack-oslo | 05:26 | |
| *** zzzeek has quit IRC | 05:30 | |
| *** ihrachyshka has quit IRC | 05:39 | |
| *** praneshp has joined #openstack-oslo | 05:41 | |
| *** ihrachyshka has joined #openstack-oslo | 05:45 | |
| *** harlowja is now known as harlowja_away | 05:47 | |
| *** ihrachyshka has quit IRC | 05:56 | |
| openstackgerrit | Marc Koderer proposed a change to openstack/oslotest: Add known issue about time.time mocking https://review.openstack.org/95411 | 06:22 |
| *** sreshetn1ak has quit IRC | 06:34 | |
| *** ildikov has joined #openstack-oslo | 06:37 | |
| *** tkelsey has joined #openstack-oslo | 06:38 | |
| *** ihrachyshka has joined #openstack-oslo | 07:19 | |
| *** flaper87|afk is now known as flaper87 | 07:24 | |
| *** i159 has joined #openstack-oslo | 07:25 | |
| *** markmc has joined #openstack-oslo | 07:27 | |
| *** liusheng has joined #openstack-oslo | 07:30 | |
| liusheng | Hi folks, could you please review https://review.openstack.org/#/c/103506/ ? | 07:31 |
| *** viktors|afk is now known as viktors | 07:32 | |
| *** ildikov_ has joined #openstack-oslo | 07:42 | |
| *** ildikov has quit IRC | 07:45 | |
| *** nacim has joined #openstack-oslo | 07:58 | |
| *** stannie has joined #openstack-oslo | 07:59 | |
| openstackgerrit | liusheng proposed a change to openstack/oslo.config: Fix the return of ConfigOpts's __call__ method https://review.openstack.org/103506 | 08:13 |
| openstackgerrit | Alex Xu proposed a change to openstack/oslo-specs: Support policy configuration directories https://review.openstack.org/104157 | 08:16 |
| openstackgerrit | Thomas Herve proposed a change to openstack/oslo-incubator: Allow test_service to run in isolation https://review.openstack.org/104474 | 08:19 |
| ihrachyshka | gus: hey | 08:20 |
| *** pblaho has joined #openstack-oslo | 08:20 | |
| openstackgerrit | Thomas Herve proposed a change to openstack/oslo-incubator: Allow test_lockutils to run in isolation https://review.openstack.org/104475 | 08:24 |
| *** praneshp has quit IRC | 08:34 | |
| openstackgerrit | A change was merged to openstack/oslo-incubator: Allow test_service to run in isolation https://review.openstack.org/104474 | 08:38 |
| *** mrda is now known as mrda-away | 08:39 | |
| *** enykeev has joined #openstack-oslo | 08:43 | |
| *** jaosorior has joined #openstack-oslo | 08:55 | |
| *** ajo has quit IRC | 09:28 | |
| *** Alexei_9871 has joined #openstack-oslo | 09:34 | |
| *** ajo has joined #openstack-oslo | 09:42 | |
| openstackgerrit | garyk proposed a change to openstack/oslo.vmware: Add support for using extensions https://review.openstack.org/100911 | 10:03 |
| *** dstanek_zzz is now known as dstanek | 10:14 | |
| *** pblaho has quit IRC | 10:21 | |
| *** pcm_ has joined #openstack-oslo | 10:56 | |
| *** AAzza_afk is now known as AAzza | 11:20 | |
| *** tcammann_ has quit IRC | 11:26 | |
| *** dims has joined #openstack-oslo | 12:04 | |
| *** pblaho has joined #openstack-oslo | 12:13 | |
| *** yamahata has joined #openstack-oslo | 12:16 | |
| *** amotoki has joined #openstack-oslo | 12:19 | |
| openstackgerrit | Radoslav Gerganov proposed a change to openstack/oslo.vmware: Log additional details of suds faults https://review.openstack.org/104534 | 12:24 |
| *** AAzza is now known as AAzza_afk | 12:27 | |
| *** ajc_ has quit IRC | 12:30 | |
| *** lbragstad has quit IRC | 12:40 | |
| *** jaosorior has quit IRC | 12:42 | |
| *** markmcclain has joined #openstack-oslo | 12:42 | |
| *** mtreinish has quit IRC | 12:45 | |
| *** dims_ has joined #openstack-oslo | 12:58 | |
| openstackgerrit | Ilya Pekelny proposed a change to openstack/oslo.db: Add a base test case for DB schema comparison https://review.openstack.org/93398 | 13:02 |
| *** dims has quit IRC | 13:02 | |
| ttx | dhellmann: I +1ed the oslo.rootwrap spec we discussed Tuesday (https://review.openstack.org/#/c/94613/) | 13:05 |
| *** sreshetnyak has joined #openstack-oslo | 13:11 | |
| *** oomichi has quit IRC | 13:14 | |
| *** alexpilotti has joined #openstack-oslo | 13:20 | |
| *** mriedem has joined #openstack-oslo | 13:20 | |
| *** jecarey has quit IRC | 13:26 | |
| *** zzzeek has joined #openstack-oslo | 13:37 | |
| *** zzzeek has quit IRC | 13:42 | |
| *** zzzeek has joined #openstack-oslo | 13:43 | |
| *** mriedem has quit IRC | 13:54 | |
| dhellmann | ttx: thanks | 13:54 |
| *** mriedem has joined #openstack-oslo | 13:56 | |
| *** arnaud__ has joined #openstack-oslo | 13:57 | |
| openstackgerrit | Mark McLoughlin proposed a change to openstack/oslo.config: Add cfgfilter.ConfigFilter https://review.openstack.org/97233 | 14:07 |
| *** alexpilotti_ has joined #openstack-oslo | 14:08 | |
| *** alexpilotti has quit IRC | 14:09 | |
| *** alexpilotti_ is now known as alexpilotti | 14:09 | |
| dhellmann | bknudson: have a sec to talk about oslo.i18n? | 14:14 |
| bknudson | dhellmann: sure | 14:14 |
| dhellmann | I spent some more time thinking about adoption, and I have some other alternatives. | 14:14 |
| bknudson | I posted my change -- https://review.openstack.org/#/c/104400/ | 14:15 |
| dhellmann | Yesterday we talked about adding an i18n module to the incubator to cover the incubated modules' use of the library, and then copying that into the projects | 14:15 |
| dhellmann | oh, nice, I'll look at that, too | 14:15 |
| bknudson | monkey-patched oslo-incubator _, _LI, etc. | 14:15 |
| dhellmann | hrm | 14:15 |
| bknudson | probably not ideal | 14:15 |
| dhellmann | I realized last night that it's not technically a problem to have more than one translation function. | 14:15 |
| bknudson | https://review.openstack.org/#/c/104400/1/keystone/i18n.py | 14:15 |
| dhellmann | So if we have two versions of _() defined using the same domain they will just pull messages from the same catalog | 14:16 |
| *** dims_ has quit IRC | 14:16 | |
| bknudson | y, but then we have to initialize both libs | 14:17 |
| dhellmann | so that means we *could* wait to do anything with the other modules in the incubator until they are moved into a library, at which point the new library would need its own integration module with marker functions configured to use the library's domain | 14:17 |
| dhellmann | there's minimal overhead in that initialization, though, right? | 14:17 |
| bknudson | dhellmann: y, but it's annoying and complicated already | 14:18 |
| bknudson | especially when you have to enable_lazy in 2 places | 14:18 |
| bknudson | this is https://review.openstack.org/#/c/104400/1/bin/keystone-all | 14:18 |
| dhellmann | true, the 2 calls to enable_lazy would be error-prone | 14:18 |
| bknudson | also, that code is copied in several places in keystone | 14:19 |
| dhellmann | right | 14:19 |
| dhellmann | what about putting it in the integration module? | 14:19 |
| dhellmann | I guess then we're starting to build up a longer list of things that integration module has to do the same way in every project, at which point it's something we would want in the incubator | 14:19 |
| bknudson | I think it would work to put it in the integration module | 14:19 |
| bknudson | this is where things get kind of "magical", because we have to set up lazy initialization on import | 14:20 |
| dhellmann | you can actually toggle it on and off | 14:21 |
| dhellmann | you wouldn't want to, but the value is checked at runtime for each message | 14:21 |
| bknudson | enable_lazy has to happen before any _() and keystone has some _() that happen on import | 14:21 |
| dhellmann | ok | 14:21 |
| bknudson | http://git.openstack.org/cgit/openstack/keystone/tree/keystone/exception.py#n79 | 14:22 |
| bknudson | it's in some exception class-level strings | 14:22 |
| bknudson | although maybe I'm wrong about that | 14:22 |
| dhellmann | no, you're right, those are evaluated at import time | 14:22 |
| dhellmann | you know, I was thinking of adding a wrapper function to the integration module but you could just put both calls to enable_lazy right there without a wrapper so they are called on import | 14:23 |
| *** alexpilotti has quit IRC | 14:23 | |
| dhellmann | that assumes keystone wants all of its daemons to have lazy translation on or off, of course | 14:24 |
| bknudson | for some reason we're setting enable_lazy in keystone-manage which doesn't make sense | 14:24 |
| bknudson | I think I should get rid of that | 14:24 |
| dhellmann | maybe someone added it to all of the main programs, without considering whether it was useful in the management tool? | 14:25 |
| bknudson | y, it was probably me | 14:25 |
| bknudson | but it's still used in 1) keystone-all, 2) keystone wsgi launcher, 3) tests | 14:25 |
| bknudson | keystone.tests.__init__ | 14:25 |
| bknudson | I can look into moving it into keystone.i18n | 14:26 |
| bknudson | I've got to admit I kind of prefer the monkey patching anyways | 14:26 |
| bknudson | it's in one place so will be easy to remove | 14:26 |
| bknudson | and it's not like those symbols are going to change. | 14:26 |
| dhellmann | true, they won't | 14:27 |
| dhellmann | as long as the import order is always correct, monkey patching should be safe | 14:27 |
| dhellmann | I'll review that patch. If monkeypatching works that's an alternative, but if someone doesn't like that I think just calling enable_lazy when the integration module is imported will achieve the same thing | 14:28 |
| dhellmann | it's good to have options :-) | 14:28 |
| *** tsekiyama has joined #openstack-oslo | 14:29 | |
| *** alexpilotti has joined #openstack-oslo | 14:42 | |
| *** ihrachyshka has quit IRC | 14:43 | |
| viktors | zzzeek: around? | 14:49 |
| zzzeek | hey | 14:49 |
| viktors | zzzeek: afaik, recently you and rpodolyaka discussed alembic migrations support in oslo.db | 14:49 |
| viktors | and you told, that it’s would be nice to have common `env.py` file to reuse it in projects | 14:50 |
| zzzeek | most of what ive seen so far is this emphasis on testing migrations ….yeah | 14:50 |
| zzzeek | viktors: yaeh i dont think env.py should be stated in any other project except a stub: “from oslo.db.migrations import env; env()” | 14:50 |
| zzzeek | oslo.db should be in control of every aspect of how a project sets up DB environment. its a framework | 14:51 |
| viktors | zzzeek: ok, so my question is - do you plan to add this file to oslo.db? Or we can you it? | 14:52 |
| zzzeek | this is a big issue which means we have to go through every project using alembic and figure out what they need, then figure out what a common env.py looks like for them | 14:52 |
| zzzeek | currently im looking into oslo.db execption rewriting and baked queries | 14:53 |
| viktors | there is 4 projects in OS with use alembic at the monent | 14:53 |
| zzzeek | ok | 14:53 |
| zzzeek | viktors: seems like we need a blueprint to do this dont we | 14:53 |
| viktors | ironic, barbican, neutron and sahara | 14:53 |
| * SergeyLukjanov lurking | 14:53 | |
| viktors | SergeyLukjanov: https://github.com/openstack/sahara/blob/master/sahara/db/migration/alembic_migrations/env.py | 14:54 |
| SergeyLukjanov | viktors, yeah, I know :) | 14:54 |
| viktors | zzzeek: I'm not really sure about blueprint.. | 14:55 |
| zzzeek | viktors: my understanding is that anythign that changes things for anyone needs a blueprint :) | 14:55 |
| zzzeek | why are we making this change, what is the rationale, who is impacted, etc. etc. | 14:55 |
| zzzeek | but youve been here longer than me :) | 14:55 |
| *** AAzza_afk is now known as AAzza | 14:56 | |
| zzzeek | so you’d make the env.py for oslo.db | 14:57 |
| viktors | zzzeek: if we will file a specification, it should be more general, IMO | 14:57 |
| zzzeek | then submit reviews to all of ironic, baribican, neutraon and sahara | 14:57 |
| zzzeek | all of which replace their env.py with “oslo.db env" | 14:57 |
| zzzeek | do all those proejcts have oslo.db as a dependency already ? | 14:57 |
| viktors | zzzeek: their `env.py` are different, unfortunately | 14:58 |
| viktors | zzzeek: only neutron | 14:58 |
| *** pcm_ has quit IRC | 14:58 | |
| *** pcm_ has joined #openstack-oslo | 14:59 | |
| *** james_li has joined #openstack-oslo | 15:00 | |
| zzzeek | well depends on differences. if they all ultimatately call into run env in a similar way, in a transaction, etc. that part at least can be in oslo | 15:02 |
| zzzeek | because, waht we need for this specifcally is a way to inject our own DB connection stuff into the environment. so oslo should have a vote in the process before alembic environment is actually called in env.py | 15:02 |
| dhellmann | keep in mind the requirements of the projects not yet using alembic, too | 15:02 |
| dhellmann | keystone and ceilometer may have multiple databases, for example | 15:02 |
| dhellmann | multiple dbs with different schemas, that is | 15:03 |
| *** nacim has quit IRC | 15:03 | |
| zzzeek | dhellmann: yeah if they have migrations that run on multple dbs using a restructured env.py that is more complicated….but im still not getting my head around the whole use case all of this seems to be oriented towrads, that of unit testing migrations | 15:04 |
| *** dims_ has joined #openstack-oslo | 15:04 | |
| dhellmann | zzzeek: I know we test the migrations. I'm not sure of the details, I just noticed you guys talking about looking at existing projects for requirements. | 15:04 |
| zzzeek | because im not understanding things like “sql=True” in https://review.openstack.org/#/c/99965/13/oslo/db/sqlalchemy/test_migrations.py | 15:05 |
| zzzeek | viktors: well like i said if you just want to bypass env.py i can add that hook to alembic | 15:05 |
| zzzeek | viktors: but i think oslo.db should have control over the patterns used, probjects sohuldn’t make an env.py in isolation | 15:06 |
| zzzeek | basically anytime a project is writing boilerplate in their app that is similar / the same to other apps, and it has to do with DBs, it should be in oslo.db instaed, and they should only have very high level declarations | 15:07 |
| *** dims_ has quit IRC | 15:08 | |
| *** markmcclain has quit IRC | 15:10 | |
| i159 | zzzeek: about "sql=True". I don't know Alembic good enough, so it's just the way how it started to work. You welcome with suggestions. | 15:10 |
| viktors | zzzeek: I agree with you. But I don't know the best practices of alembic usage :) | 15:10 |
| zzzeek | i159: take out echo=True :) | 15:10 |
| zzzeek | i159: these fixtures arent tested then, is that right? dont we test the fixtures also ? | 15:11 |
| zzzeek | viktors: OK the purpose of env.py is: 1. determine what database(s) we need to connect to; 2. determine which MetaData or MetaData’s we’d be autogenerating from; 3. call the Alembic Environment API with those two details. that’s really it | 15:11 |
| i159 | zzzeek: Are you talking about 99965 patch's mixin? | 15:12 |
| zzzeek | viktors: so we can make an oslo.db specfic API that simplifies that and handles the transactional scope and any kind of modifications to what DB is being conected to | 15:12 |
| zzzeek | i159: im looking at https://review.openstack.org/#/c/99965/13/oslo/db/sqlalchemy/test_migrations.py | 15:12 |
| i159 | zzzeek: I have tested this mixin on Ironic | 15:13 |
| zzzeek | i159: the migrations there will be dumped into self.buf and not emitted to the DB, unless im missing something | 15:13 |
| zzzeek | because of sql=True | 15:13 |
| zzzeek | is that not waht’s being observed? | 15:13 |
| rpodolyaka | >>> i159: the migrations there will be dumped into self.buf and not emitted to the DB, unless im missing something - true :) | 15:14 |
| rpodolyaka | but that's not what we want | 15:15 |
| rpodolyaka | i159: ^ | 15:15 |
| rpodolyaka | as we *want* to issue those queries against the database | 15:15 |
| rpodolyaka | to actually test the migration sctips | 15:15 |
| viktors | rpodolyaka: what do you think about zzzeek's proposal as for env.py ? | 15:15 |
| rpodolyaka | viktors: ++ | 15:16 |
| rpodolyaka | viktors: are you willing to volunteer? :) | 15:16 |
| viktors | rpodolyaka: rather looking for volunteer :) | 15:17 |
| *** pblaho has quit IRC | 15:17 | |
| zzzeek | all three of you have been submitting tons of migration / test stuff | 15:17 |
| zzzeek | throw more coal on teh fire ! :) | 15:17 |
| rpodolyaka | heh | 15:18 |
| viktors | :) | 15:18 |
| zzzeek | i cant even follow it all | 15:18 |
| viktors | zzzeek: feel free to ask :) | 15:18 |
| zzzeek | the subclassing and the test_XYZ everywhere, then it comes in as a dozen different reviews | 15:18 |
| zzzeek | viktors: Id need to just focus on it for a few days | 15:18 |
| i159 | zzzeek: and suggest=) | 15:18 |
| zzzeek | i say, get it to work first | 15:18 |
| rpodolyaka | it's probably not a good sign meaning the code is more complicated then it really should be | 15:19 |
| zzzeek | rpodolyaka: ive had trouble following test code everywhere, in all the proejcts ive looked | 15:19 |
| zzzeek | rpodolyaka: its not an oslo.db issue specifically | 15:19 |
| viktors | zzzeek: oslo.db proposal is to generalize code for work with database and make it better :) | 15:20 |
| zzzeek | viktors: yup | 15:20 |
| zzzeek | viktors: i wasnt focusing on tests/migrations first because it is the more thorny issue :) | 15:21 |
| viktors | zzzeek: so we can improve some feature (pretty migration testing, for example) in oslo.db and then re-use in in projects | 15:21 |
| zzzeek | viktors: so the things ive seen so far, look fine | 15:21 |
| zzzeek | they are improvements | 15:21 |
| zzzeek | here’s a vanilla env.py https://bitbucket.org/zzzeek/alembic/src/f38eaad4a80d7e3d893c3044162971971ae009bf/alembic/templates/generic/env.py?at=master | 15:23 |
| zzzeek | so you can see it 1. figures out what MetaData we might want to autogenrate from, if any, as its optional in any case 2. makes a decision based on —sql mode or not, doesnt seem like openstack really uses “offline” mode, 3. figures out where the DB is, connects, starts a transaction and calls run_migrations() | 15:24 |
| zzzeek | an oslo.db vesion of this would probably provide a function that does what run_migrations_online() does | 15:25 |
| zzzeek | so that it can control how the API is ultimately called | 15:25 |
| zzzeek | e.g. a facade | 15:25 |
| zzzeek | :) | 15:25 |
| dhellmann | zzzeek: if we could allow deployers to use offline mode and generate the SQL to run their own migrations, that might be good. I know there are some extremely large shops that do things like figure out how to hand-migrate their systems to avoid huge downtime blocking on updates to very large tables. | 15:27 |
| dhellmann | even if they don't run them by hand, knowing what the migration is going to do can be helpful for capacity planning | 15:28 |
| markmc | flaper87, viktors, http://lists.openstack.org/pipermail/openstack-dev/2014-July/039291.html | 15:28 |
| rpodolyaka | unfortunately, some projects do crazy stuff in migrations scripts that requires us to use online migrations :( | 15:28 |
| zzzeek | dhellmann: I woudl agree, but at the same time it seems like some projects want to pursue extremely complex and interactive migration models, see https://review.openstack.org/#/c/102545/ | 15:28 |
| flaper87 | markmc++ | 15:28 |
| markmc | flaper87, viktors, ^^ summary of the asyncio/oslo.messaging stuff | 15:28 |
| zzzeek | dhellmann: a very elaborate system like that owuld be very difficult to reconclie with generation of —sql scripts | 15:29 |
| viktors | markmc: are you looking for Victor Stinner? ) | 15:29 |
| markmc | viktors, sorry, yes | 15:29 |
| dhellmann | markmc: I think you mean haypo | 15:29 |
| markmc | viktors, he's sitting beside me :) | 15:29 |
| viktors | markmc: np | 15:29 |
| dhellmann | heh | 15:29 |
| *** haypo has joined #openstack-oslo | 15:30 | |
| haypo | hey | 15:30 |
| markmc | haypo, http://lists.openstack.org/pipermail/openstack-dev/2014-July/039291.html :) | 15:31 |
| dhellmann | zzzeek: that sounds like an alternative approach to solving the problem of upgrading large live installations. My suggestion was just to expose information about what the migrations are going to do so deployers can look at it if they want to. | 15:31 |
| openstackgerrit | Ilya Pekelny proposed a change to openstack/oslo.db: Opportunistic migration tests https://review.openstack.org/93424 | 15:32 |
| openstackgerrit | Ilya Pekelny proposed a change to openstack/oslo.db: Implementation Alembic as migration engine https://review.openstack.org/99965 | 15:32 |
| zzzeek | dhellmann: this is straightforward if everyone is on the same system. | 15:32 |
| *** praneshp has joined #openstack-oslo | 15:33 | |
| i159 | zzzeek: should we discuss now sumething else about oslo.db? | 15:36 |
| zzzeek | i159: is https://review.openstack.org/#/c/99965/14/migration/env.py specific to testing migrations only? not actualyl running them ? | 15:37 |
| dhellmann | zzzeek: "same system" meaning "using the same approach"? | 15:37 |
| zzzeek | dhellmann: yes, some common API for migrations that knows how to delliver SQL scripts in a consistent way | 15:37 |
| dhellmann | zzzeek: ok, I think that's the goal | 15:38 |
| zzzeek | dhellmann: “offline” mode is difficult though when a system migrates to multple database connections in one go | 15:38 |
| i159 | zzzeek: I have run it on Ironic, but it is not works yet, due to engine issue. | 15:38 |
| dhellmann | zzzeek: I don't know if that's something we do in keystone or ceilometer | 15:38 |
| *** AAzza has left #openstack-oslo | 15:39 | |
| zzzeek | i159: my q is, is this only for testing, and if so, why isnt the word “test” mentioned in the filename, the path, anywhere, its not clear that this file has anyhing to do with testing, other than that there is a hardcoded DB url in it, which I’ve come to assume is a “magic” value that is used in openstack projects | 15:39 |
| zzzeek | another thing that confuses me, why tests/sqlalchemy/test_migrations.py and then oslo/db/sqlalchemy/test_migrations.py, is one a unit test file and the other a library for testing, or are htese both accessed directly by a test collector? the convention test_XYZ.py indicates a file that is intended to be consuemd directly by a test collector. so then why is a test_XYZ file outside of tests? | 15:42 |
| zzzeek | i159: since you asked, these are things that make the code hard to follow | 15:42 |
| zzzeek | dhellmann: the work would be to take a survey of all migration environemnts, gather requirements so that an API that meets all of them can be constructed | 15:44 |
| zzzeek | dhellmann: one that relieves those projects of any kind of repetitive boilerplate or decision making as to how alembic should be used | 15:45 |
| i159 | zzzeek: hardcoded DB url is just a plug (gag). env.py named as original one. This patch is deep in progress, its purpose at the moment is to uncover potential problems. When common approach will be clear, this patch will become good and clean as possible. | 15:45 |
| dhellmann | zzzeek: yeah, that will be a large project and should have lots of publicity so we get input from devs and deployers | 15:45 |
| *** arnaud__ has quit IRC | 15:45 | |
| zzzeek | i159: that’s great ! i was just pointing out why it confuses me :) | 15:45 |
| i159 | zzzeek: I see =) Thanks you! Unfortunately I should go. Feel free to review. See you! | 15:46 |
| zzzeek | later | 15:46 |
| i159 | yep | 15:47 |
| *** i159 has quit IRC | 15:47 | |
| zzzeek | dhellmann: is there a reason we dont/cant use relative imports ? | 15:48 |
| bknudson | dhellmann: what do you mean keystone can have multiple dbs with different schemas? | 15:54 |
| *** ihrachyshka has joined #openstack-oslo | 15:56 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Add proxy heartbeat support https://review.openstack.org/104397 | 15:56 |
| dhellmann | zzzeek: that's a project style decision, to keep things consistent, but I don't know the history | 15:57 |
| dhellmann | bknudson: aren't the different backends potentially in different databases? or are say identity and tokens both in the same db instance if they're both using the sql driver? | 15:57 |
| dhellmann | bknudson: I may just have misunderstood something... | 15:57 |
| zzzeek | dhellmann: so if i wnat to nix these ugly “from oslo.db.sqlalchemy” into just “from .” …. ? :) http://git.openstack.org/cgit/openstack/oslo.db/tree/oslo/db/sqlalchemy/test_base.py#n24 | 15:58 |
| dhellmann | zzzeek: no, there's a project-wide style guide that says to use absolute imports | 15:59 |
| *** ildikov has joined #openstack-oslo | 15:59 | |
| zzzeek | for oslo you mean | 15:59 |
| dhellmann | zzzeek: for openstack | 15:59 |
| Alexei_9871 | quick and stupid question about oslo.messaging: is it possible to do and RPC call/cast on fanout topic? | 15:59 |
| zzzeek | dhellmann: oh. when you said “ a project style” somehow that sounded like “per project” | 15:59 |
| dhellmann | zzzeek: H304 "Do not make relative imports" - http://git.openstack.org/cgit/openstack-dev/hacking/tree/HACKING.rst | 16:00 |
| *** mtreinish has joined #openstack-oslo | 16:00 | |
| zzzeek | dhellmann: smash the system ! | 16:00 |
| dhellmann | zzzeek: everyone has their least favorite hacking rule :-) | 16:00 |
| bknudson | dhellmann: they all have to be in the same database if they're in a database. | 16:00 |
| zzzeek | the first rule of hacking rules is we do not talk about hacking rules | 16:01 |
| bknudson | dhellmann: you can have an LDAP backend, but then it isn't using the sql db | 16:01 |
| dhellmann | bknudson: yay! I misunderstood and it turns out things are easier than expected instead of harder! | 16:01 |
| dhellmann | zzzeek: I was wrong about the keystone db stuff ^^ | 16:01 |
| *** ildikov_ has quit IRC | 16:02 | |
| zzzeek | dhellmann: my take on migrations right now is: 1. wait for ilya, roman, victor to get all their patches in and get dust to settle 2. spend a week understanding the whole issue better 3. propose anotehr round of improvemnts/enhancements as needed | 16:02 |
| zzzeek | dhellmann: they are moving too fast with things, if their stuff works then that is OK for now, getting a unified migration + testing + migration testing API will be iterative | 16:02 |
| bknudson | with the change for domain-specific backends there may be support for different db connection settings... but even then they'd have to be at the latest level. | 16:02 |
| dhellmann | zzzeek: that sounds like a good approach -- I expect it to take a fair amount of time to understand how all of the projects are actually using the db and to identify things we can/should do differently, so don't feel rushed | 16:02 |
| zzzeek | dhellmann: ive tried to poke into it and not felt like i was getting the whole picture | 16:03 |
| dhellmann | zzzeek, viktors, rpodolyaka: I haven't looked lately, but one thing we're going to need is lots of documentation on usage in the oslo.db doc tree | 16:03 |
| *** ildikov has quit IRC | 16:03 | |
| dhellmann | it may already be there, as I said I haven't looked recently | 16:03 |
| dhellmann | time for lunch, bbiab | 16:04 |
| *** mtreinish has quit IRC | 16:04 | |
| *** dhellmann is now known as dhellmann_ | 16:04 | |
| *** mtreinish has joined #openstack-oslo | 16:04 | |
| *** dims_ has joined #openstack-oslo | 16:05 | |
| rpodolyaka | dhellmann: yeah, this is definitely the area we should improve in | 16:05 |
| *** dims_ has quit IRC | 16:09 | |
| *** praneshp has quit IRC | 16:11 | |
| *** flaper87 is now known as flaper87|afk | 16:12 | |
| *** tkelsey has quit IRC | 16:18 | |
| *** dims_ has joined #openstack-oslo | 16:21 | |
| *** pcm_ has quit IRC | 16:26 | |
| *** pcm_ has joined #openstack-oslo | 16:26 | |
| *** zhiyan is now known as zhiyan_ | 16:29 | |
| *** markmc has quit IRC | 16:31 | |
| *** dhellmann_ is now known as dhellmann | 16:44 | |
| *** stannie has quit IRC | 16:47 | |
| *** arnaud has joined #openstack-oslo | 16:52 | |
| *** Alexei_9871 has quit IRC | 16:52 | |
| *** ihrachyshka has quit IRC | 17:12 | |
| *** praneshp has joined #openstack-oslo | 17:13 | |
| *** tmcpeak has joined #openstack-oslo | 17:15 | |
| * bnemec watches as the oslo.messaging/asyncio thread goes right over his head :-) | 17:15 | |
| *** tmcpeak has left #openstack-oslo | 17:15 | |
| *** bnemec is now known as bnemec_appt | 17:15 | |
| dhellmann | bnemec_appt: I'm glad to have experts on the case. :-) | 17:20 |
| *** pcm_ has quit IRC | 17:30 | |
| *** pcm_ has joined #openstack-oslo | 17:31 | |
| dims_ | bnemec_appt, totally agree! | 17:37 |
| *** praneshp_ has joined #openstack-oslo | 17:38 | |
| *** praneshp has quit IRC | 17:42 | |
| *** praneshp_ is now known as praneshp | 17:42 | |
| *** arnaud has quit IRC | 17:52 | |
| *** arnaud has joined #openstack-oslo | 18:02 | |
| *** arnaud has quit IRC | 18:02 | |
| *** pcm_ has quit IRC | 18:06 | |
| *** pcm_ has joined #openstack-oslo | 18:07 | |
| *** liusheng has quit IRC | 18:10 | |
| *** liusheng has joined #openstack-oslo | 18:11 | |
| boris-42 | dhellmann hi | 18:12 |
| boris-42 | dhellmann I just finished work on version 0.2.0 https://github.com/stackforge/osprofiler | 18:16 |
| boris-42 | dhellmann and I update README.rst | 18:16 |
| boris-42 | dhellmann now it's much cleaner and simpler for understanding | 18:16 |
| boris-42 | dhellmann I am going to test that everything still works after all these changes in real life (e.g. deploy openstack with patches and oslo.messaging changes) | 18:17 |
| boris-42 | dhellmann and if everything is ok, I'll cut version 0.2.0 | 18:17 |
| dhellmann | boris-42: sounds good | 18:19 |
| boris-42 | dhellmann so now NO API changes | 18:19 |
| boris-42 | =) | 18:19 |
| dhellmann | never say never... | 18:20 |
| boris-42 | dhellmann I hope there won't be version 0.3.x at all=) | 18:20 |
| dhellmann | boris-42: 1.0? | 18:21 |
| boris-42 | dhellmann maybe is better then 0.2 | 18:21 |
| dhellmann | only if you're sure you're ready | 18:21 |
| boris-42 | dhellmann let's keep 0.2 I am sure | 18:21 |
| boris-42 | dhellmann but not sure that community is sure=) | 18:21 |
| boris-42 | dhellmann mostly I am worried about oslo.messaging stuff | 18:22 |
| boris-42 | dhellmann and it's last one fundamental that is required | 18:23 |
| *** arnaud__ has joined #openstack-oslo | 18:27 | |
| *** arnaud__ has quit IRC | 18:40 | |
| *** james_li_ has joined #openstack-oslo | 19:01 | |
| *** james_li has quit IRC | 19:04 | |
| *** james_li_ is now known as james_li | 19:04 | |
| *** arnaud__ has joined #openstack-oslo | 19:07 | |
| openstackgerrit | A change was merged to openstack/oslo-specs: Graduate oslo.concurrency https://review.openstack.org/97296 | 19:08 |
| *** dims_ has quit IRC | 19:09 | |
| *** AAzza_afk has joined #openstack-oslo | 19:17 | |
| *** AAzza_afk is now known as AAzza | 19:17 | |
| *** arnaud__ has quit IRC | 19:18 | |
| boris-42 | dhellmann btw I have some question, about translations | 19:19 |
| boris-42 | dhellmann why do we keep them in project trees? | 19:19 |
| boris-42 | dhellmann e.g. nova repo now is really large | 19:19 |
| *** SridharG has quit IRC | 19:19 | |
| boris-42 | And I think half of size are translations | 19:19 |
| dhellmann | boris-42: where else would you put them? | 19:20 |
| boris-42 | dhellmann "nova-translations" | 19:20 |
| boris-42 | dhellmann like we have *-specs | 19:20 |
| dhellmann | boris-42: the inputs for the catalog file come from the nova source code | 19:24 |
| openstackgerrit | A change was merged to openstack/oslo-specs: Update adoption notes about Message objects https://review.openstack.org/104234 | 19:25 |
| boris-42 | dhellmann yep agree | 19:25 |
| boris-42 | dhellmann but does it mean that we should keep it together? | 19:26 |
| *** AAzza is now known as AAzza_afk | 19:27 | |
| dhellmann | boris-42: I don't think this is the most productive thing either of us could be doing right now. :-) | 19:30 |
| boris-42 | dhellmann yep sure, I am not going to it know=) | 19:30 |
| dhellmann | boris-42: we would have to change a lot about how our packaging system works to move those files, for what I see as little benefit | 19:31 |
| *** AAzza_afk is now known as AAzza | 19:38 | |
| boris-42 | dhellmann okay agree with that | 19:40 |
| boris-42 | =) | 19:40 |
| boris-42 | dhellmann thanks for review of spec | 19:40 |
| *** ajo|mac has joined #openstack-oslo | 19:40 | |
| *** pblaho has joined #openstack-oslo | 19:49 | |
| *** jecarey has joined #openstack-oslo | 19:52 | |
| *** dims_ has joined #openstack-oslo | 19:53 | |
| *** AAzza is now known as AAzza_afk | 19:57 | |
| *** ajo|mac has quit IRC | 19:59 | |
| *** arnaud has joined #openstack-oslo | 20:03 | |
| *** pblaho has quit IRC | 20:08 | |
| *** pblaho has joined #openstack-oslo | 20:08 | |
| *** pblaho has quit IRC | 20:21 | |
| *** pcm_ has quit IRC | 20:21 | |
| *** james_li_ has joined #openstack-oslo | 20:23 | |
| *** james_li has quit IRC | 20:26 | |
| *** james_li_ is now known as james_li | 20:26 | |
| *** bnemec_appt is now known as bnemec | 20:27 | |
| jecarey | dhellmann: There's a bug in i18n in oslo-incubator that's fixed in the library, but I think it needs to be brought back. | 20:33 |
| jecarey | The bug is that the _make_translation_func returns a method generated by checking lazy instead of a method that checks USE_LAZY at runtime. | 20:34 |
| openstackgerrit | A change was merged to openstack/oslo-incubator: Allow test_lockutils to run in isolation https://review.openstack.org/104475 | 20:44 |
| *** ihrachyshka has joined #openstack-oslo | 20:44 | |
| *** ildikov has joined #openstack-oslo | 20:53 | |
| *** zzzeek has quit IRC | 21:09 | |
| *** zzzeek has joined #openstack-oslo | 21:12 | |
| boris-42 | dhellmann tested new version | 21:15 |
| boris-42 | dhellmann it works | 21:15 |
| *** zzzeek has quit IRC | 21:15 | |
| *** zzzeek has joined #openstack-oslo | 21:20 | |
| boris-42 | dhellmann Ok I bumped it https://review.openstack.org/#/c/104691/1 | 21:30 |
| *** james_li has quit IRC | 21:35 | |
| *** james_li has joined #openstack-oslo | 21:38 | |
| *** ajo|mac has joined #openstack-oslo | 21:43 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Move attribute dictionary to a dictionary type https://review.openstack.org/104695 | 21:46 |
| *** ajo|mac has quit IRC | 21:48 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Move the stopwatch tests to test_types https://review.openstack.org/104698 | 21:52 |
| *** ajo|mac has joined #openstack-oslo | 21:52 | |
| *** mrda-away is now known as mrda | 21:53 | |
| openstackgerrit | Boris Pavlovic proposed a change to openstack/oslo.messaging: Enable cross service profiling https://review.openstack.org/103271 | 22:06 |
| openstackgerrit | pritesh proposed a change to openstack-dev/hacking: Add hacking check for editor configuration in source files https://review.openstack.org/101969 | 22:06 |
| *** dims__ has joined #openstack-oslo | 22:13 | |
| *** dims_ has quit IRC | 22:15 | |
| openstackgerrit | pritesh proposed a change to openstack-dev/hacking: Add hacking check for editor configuration in source files https://review.openstack.org/101969 | 22:19 |
| *** zzzeek has quit IRC | 22:20 | |
| *** yamahata has quit IRC | 22:41 | |
| *** harlowja_at_home has joined #openstack-oslo | 22:42 | |
| *** ajo has quit IRC | 22:48 | |
| *** ihrachyshka has quit IRC | 22:51 | |
| *** ildikov has quit IRC | 23:10 | |
| *** gmurphy has quit IRC | 23:13 | |
| *** gmurphy has joined #openstack-oslo | 23:15 | |
| *** alexpilotti has quit IRC | 23:19 | |
| *** james_li has quit IRC | 23:22 | |
| *** zzzeek has joined #openstack-oslo | 23:30 | |
| *** mtreinish has quit IRC | 23:31 | |
| *** tsekiyama has quit IRC | 23:32 | |
| *** mtreinish has joined #openstack-oslo | 23:39 | |
| *** oomichi has joined #openstack-oslo | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!