| zzzeek | where is the assumption that it’s utf8 | 00:00 |
|---|---|---|
| zzzeek | im poking around looking for other charsets | 00:00 |
| zzzeek | taht are accepted by “charset” and are unicode capable | 00:00 |
| gus | .. and my main fear (and point I'm hoping to get across) is that all this is covered up by your current test environment which does indeed make everything default to utf8. | 00:00 |
| zzzeek | so far I found utf8mb4 | 00:00 |
| gus | oh, I think mysql is capable of utf8mb4 (the default is 3-byte), ucs2, utf16, etc. | 00:00 |
| gus | It can't use ucs2 for the connection encoding apparently - I assume because of the embedded nulls. | 00:01 |
| zzzeek | yes the charset param is rejecting ucs2 and utf16 | 00:01 |
| zzzeek | gus: where is the utf-8 coming from on the SQLAlchemy side when we create an ENUM ? | 00:01 |
| gus | from that test case? | 00:02 |
| zzzeek | es | 00:02 |
| zzzeek | yes | 00:02 |
| gus | the test case uses unicode literals in python ... | 00:02 |
| gus | they need to get encoded to something on the wire. | 00:02 |
| gus | mysqlconnector seems to issue a 'set names utf8' call on connect. | 00:03 |
| zzzeek | well there are two choices for where they can be encoded | 00:03 |
| zzzeek | right | 00:03 |
| gus | so I assume it knows it's going to use utf8. | 00:03 |
| gus | I don't think(?) mysqldb issues that. | 00:03 |
| zzzeek | gus: not unless you send it “charset" | 00:03 |
| gus | zzzeek: aha. | 00:03 |
| gus | so it doesn't detect or specify a particular encoding>? | 00:03 |
| zzzeek | in the case of MySQLdb, we curerntly dont assume MySQLdb can accept u’’asdf’ for a statement | 00:04 |
| gus | I guess it makes sense for that to be a client-specified option. | 00:04 |
| zzzeek | so in that case, we are encoding to utf-8 currently, based on the default encoding of DefaultDialect. | 00:04 |
| zzzeek | gus: however, MySQLdb is pretty slick these days, and I can actually change that boolean to be True | 00:04 |
| zzzeek | and it workos | 00:04 |
| zzzeek | works | 00:04 |
| zzzeek | e.g. if you do this: | 00:04 |
| zzzeek | diff --git a/lib/sqlalchemy/dialects/mysql/mysqldb.py b/lib/sqlalchemy/dialects/mysql/mysqldb.py | 00:05 |
| zzzeek | index 8ee367a..30802f2 100644 | 00:05 |
| zzzeek | --- a/lib/sqlalchemy/dialects/mysql/mysqldb.py | 00:05 |
| zzzeek | +++ b/lib/sqlalchemy/dialects/mysql/mysqldb.py | 00:05 |
| zzzeek | @@ -75,7 +75,7 @@ class MySQLIdentifierPreparer_mysqldb(MySQLIdentifierPreparer): | 00:05 |
| zzzeek | 00:05 | |
| zzzeek | class MySQLDialect_mysqldb(MySQLDialect): | 00:05 |
| zzzeek | driver = 'mysqldb' | 00:05 |
| zzzeek | - supports_unicode_statements = False | 00:05 |
| zzzeek | + supports_unicode_statements = True | 00:05 |
| zzzeek | then SQLAlchemy is doing zero with encoding | 00:05 |
| gus | so what does it mean when this test_unicode_enum is passing? | 00:05 |
| zzzeek | PDB’ing into it, I can see this, being sent directly to mysqldb’s cursor: | 00:06 |
| zzzeek | >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB set_trace (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> | 00:06 |
| zzzeek | > /Users/classic/dev/sqlalchemy/lib/sqlalchemy/engine/default.py(439)do_execute() | 00:06 |
| zzzeek | -> cursor.execute(statement, parameters) | 00:06 |
| zzzeek | (Pdb) statement | 00:06 |
| zzzeek | u"\nCREATE TABLE `table` (\n\tid INTEGER NOT NULL AUTO_INCREMENT, \n\tvalue ENUM('r\xe9veill\xe9','dr\xf4le','S\u2019il'), \n\tvalue2 ENUM('r\xe9veill\xe9','dr\xf4le','S\u2019il'), \n\tPRIMARY KEY (id)\n)\n\n" | 00:06 |
| zzzeek | that is a u'' | 00:06 |
| gus | yeah that's what I was seeing too. | 00:06 |
| gus | What's with the mix of \x bytes and \u codepoint? | 00:06 |
| zzzeek | it means that my database here is set to utf-8 on the server side | 00:06 |
| zzzeek | um, repr() stuff. I dont know why its doing that | 00:07 |
| gus | anyway, that seems to work fine on the create side. | 00:07 |
| zzzeek | maybe its the py3k compat stuff not working correctly | 00:07 |
| gus | On the reflection side, it seems to misinterpret those bytes. | 00:07 |
| zzzeek | gus: is your only point that you want to add a “charset” to the test ? | 00:08 |
| gus | my real point, is that the test shouldn't be able to pass without a charset :P | 00:08 |
| zzzeek | gus: ehhhhh….seems weird | 00:08 |
| zzzeek | gus: if i use sqlite, there’s no charset. it all magically works | 00:09 |
| zzzeek | gus: mysql has more awkard unicode settings, but that is all configuational | 00:09 |
| zzzeek | gus: im not sure that SQLAlchemy has to work aroudn configurational errors | 00:09 |
| gus | ok, so you only want to support mysql that is configured to default to utf8? | 00:10 |
| zzzeek | gus: im not actualyl sure how to make the test always fail without a charset. if the charset on the db being tested happens to be utf-8, itll pass | 00:10 |
| gus | that seems perfectly fine, but I think we're very close to not needing to require that. | 00:10 |
| zzzeek | gus: not at all? how is that implied ? | 00:10 |
| zzzeek | still really not sure what your proposal is. | 00:11 |
| gus | so if you had a database that wasn't capable of storing unicode unless it was explicitly marked in some way. | 00:11 |
| zzzeek | most of them are like that | 00:11 |
| zzzeek | if you put “ASCII” encoding on your PG database, none of the unicode tests will work. | 00:12 |
| gus | then that test_unicode_enum test shouldn't pass, because there's nothing in the Enum() (for mysql) that indicates that the contents are unicode. | 00:12 |
| gus | all we need to do (aiui) is to add a 'charset utf8' or a 'UNICODE' to that column type, and it would work correctly regardless of the rest of the server/db/table encoding. | 00:13 |
| zzzeek | gus: OK but this is not an application test, this isn’t an openstack test. this is just testing that SQLAlchemy sends DDL that can be understood | 00:13 |
| gus | ... and we need to ensure the connection encoding is unicode-capable. | 00:13 |
| zzzeek | gus: yikes really, what if there isnt a connection ? | 00:13 |
| gus | zzzeek: oh, I meant for mysql | 00:13 |
| gus | where it has separate encoding options for all these things. | 00:14 |
| zzzeek | gus: i want to generate a —sql script with a unicode enum in it. what do i check ? | 00:14 |
| *** tsekiyam_ has joined #openstack-oslo | 00:14 | |
| gus | you need to specify/assume an encoding for that. | 00:14 |
| gus | and probably preceed the output with a 'set names <encoding>'. | 00:14 |
| zzzeek | gus: DDL-level encodings are somethign the user needs to set. SQLAlchemy isnt in the business of second guessing what you tell it to do. an environemtn taht knows its using all utf-8 shouldnt need these surprise markers at the indvidual type level | 00:15 |
| gus | ah ok, I think this is exactly where we're skewing off into different conclusions. | 00:16 |
| zzzeek | gus: the mysql.ENUM() type outputs just that: “ENUM" | 00:16 |
| zzzeek | gus: if you want it to say “utf8” in it, you have to say, mysql.ENUM(charset=‘utf8') | 00:16 |
| zzzeek | explicit | 00:16 |
| zzzeek | vs. implicit | 00:16 |
| zzzeek | refuse the temptation to guess | 00:17 |
| gus | I'm assuming the environment isn't necessarily set to utf8 (since that isn't the "usual" default with mysql) | 00:17 |
| zzzeek | the DDL emitted by mysql.ENUM is not related to the charset on the connection within SQLAlchemy, those are two different things | 00:17 |
| *** tsekiyama has quit IRC | 00:17 | |
| zzzeek | gus: if we really wanted mysql.ENUM to guarantee that “charset” is hardcoded into it, the test woudl be, self.assert_compile(mysql.ENUM(‘a’, ‘b’, ‘c’), ‘…check that utf8 is present’) | 00:18 |
| *** tsekiyam_ has quit IRC | 00:18 | |
| gus | ok, I think I'm finally understanding :P | 00:19 |
| zzzeek | its important that SQLAlchemy tries, as best as is possible (and its very hard), to be consistent and predictable. if it spits out exactly what its told, and doesnt try to figure out that this DDL happens to be headed for a connection that is implied that its utf-8 or isnt, that is less surprising than something that’s trying to magically figure everything out | 00:19 |
| zzzeek | gus: i can kind of see what youre going for, thoguh | 00:20 |
| zzzeek | gus: havent considered that conneciton encoding should have a say in DDL compilation. it is architecturally feasible, just very different from how things have worked | 00:21 |
| gus | ok, so I think we agree that Unicode/UnicodeText should specify 'UNICODE' in some way? | 00:21 |
| zzzeek | gus: yes are you the one who set up https://github.com/zzzeek/sqlalchemy/pull/102 ? | 00:21 |
| gus | .. and I think mysqldb should issue a 'set names <encoding>' call just to be sure the connection encoding and it's understanding of the encoding agree. | 00:21 |
| gus | zzzeek: yes, that's me. | 00:21 |
| zzzeek | gus: yes i agree with that PR because the unicode type is explicit, “this is the datatype that knows how to accept unicode" | 00:22 |
| zzzeek | gus: its almost like we’re looking for UnicodeEnum | 00:22 |
| zzzeek | gus: woudl that help ? | 00:22 |
| gus | yes. although I guess a similar problem pops up for unicode column/table names. | 00:22 |
| zzzeek | gus: that is a whole ball of wax | 00:22 |
| gus | I agree. (iow, it doesn't seem worth addressing enums and leaving column names out) | 00:23 |
| zzzeek | gus: SQLA is in the business of sending strings, it assumes by default that you have the two sides ready to talk to each other and encoding things are out of its purview, unless you are explciit | 00:23 |
| zzzeek | gus: column / table names, you have to have the DB encoding set up | 00:23 |
| gus | yep. | 00:23 |
| zzzeek | gus: total PITA on some backends | 00:23 |
| gus | ok, so I should assume that the DB, connection and sqla ?charset= are correctly configured, and that if they aren't, it isn't sqla's problem to fix. | 00:25 |
| gus | got it. :) | 00:25 |
| *** alexpilotti has joined #openstack-oslo | 00:25 | |
| gus | (thanks for your patience) | 00:25 |
| zzzeek | good convo! | 00:26 |
| gus | I only started looking at all this following the neutron issue a week or two ago. So I'm stumbling into all sorts of mysql/sqla/charset details that are completely new to me :/ | 00:27 |
| zzzeek | OK. well, its been this way for much longer than openstack has been around :) | 00:29 |
| zzzeek | it raelly tries to be a tool that does exactly what you tell it. its very much meant for ppl that know exactly what they want to see on the DB, they jsut want to automate it | 00:30 |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Make greenexecutor not keep greenthreads active https://review.openstack.org/105922 | 00:38 |
| *** jaosorior has quit IRC | 00:42 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Move attribute dictionary to a dicts type https://review.openstack.org/104695 | 01:00 |
| *** zzzeek has quit IRC | 01:19 | |
| *** praneshp has quit IRC | 01:36 | |
| *** praneshp has joined #openstack-oslo | 01:41 | |
| *** praneshp has quit IRC | 01:42 | |
| *** praneshp has joined #openstack-oslo | 01:44 | |
| *** praneshp has quit IRC | 01:51 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Avoid naming time type module the same as a builtin https://review.openstack.org/105938 | 01:58 |
| openstackgerrit | Arnaud Legendre proposed a change to openstack/oslo.vmware: Store PBM wsdl in the oslo.vmware git repository https://review.openstack.org/102409 | 02:00 |
| *** arnaud has quit IRC | 02:03 | |
| *** SridharG has joined #openstack-oslo | 02:05 | |
| *** praneshp has joined #openstack-oslo | 02:06 | |
| *** zhiyan_ is now known as zhiyan | 02:07 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Avoid naming time type module the same as a builtin https://review.openstack.org/105938 | 02:08 |
| *** alexpilotti has quit IRC | 02:08 | |
| openstackgerrit | jizhilong proposed a change to openstack-dev/hacking: Add check: unnecessary list comprehension in set() https://review.openstack.org/105942 | 02:10 |
| *** praneshp has quit IRC | 02:22 | |
| *** SridharG has quit IRC | 02:57 | |
| *** HenryG has joined #openstack-oslo | 03:03 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/oslo-specs: Add a redis backed job/jobboard https://review.openstack.org/105298 | 03:07 |
| *** SridharG has joined #openstack-oslo | 03:18 | |
| *** dims_ has quit IRC | 03:22 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Make greenexecutor not keep greenthreads active https://review.openstack.org/105922 | 03:22 |
| *** dims_ has joined #openstack-oslo | 03:23 | |
| *** SridharG has quit IRC | 03:26 | |
| openstackgerrit | jizhilong proposed a change to openstack-dev/hacking: Add check: unnecessary list comprehension in set() https://review.openstack.org/105942 | 03:38 |
| *** arnaud has joined #openstack-oslo | 03:46 | |
| *** dims_ has quit IRC | 03:57 | |
| *** ildikov has quit IRC | 04:09 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Rework wbe example to use in-memory transport and threads https://review.openstack.org/105963 | 04:18 |
| *** mtreinish has quit IRC | 04:19 | |
| *** mtreinish has joined #openstack-oslo | 04:20 | |
| *** jecarey has joined #openstack-oslo | 04:29 | |
| *** YorikSar has quit IRC | 04:39 | |
| *** YorikSar has joined #openstack-oslo | 04:41 | |
| *** dims_ has joined #openstack-oslo | 04:53 | |
| *** dims_ has quit IRC | 05:02 | |
| *** jecarey has quit IRC | 05:14 | |
| *** SridharG has joined #openstack-oslo | 05:20 | |
| *** jecarey has joined #openstack-oslo | 05:20 | |
| *** dstanek_zzz is now known as dstanek | 05:31 | |
| *** k4n0 has joined #openstack-oslo | 05:35 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Rework wbe example to be simpler to use and understand https://review.openstack.org/105963 | 05:35 |
| openstackgerrit | jizhilong proposed a change to openstack-dev/hacking: Add check: unnecessary list comprehension in set() https://review.openstack.org/105942 | 05:35 |
| *** praneshp has joined #openstack-oslo | 05:36 | |
| openstackgerrit | Joshua Harlow proposed a change to openstack/taskflow: Rework wbe example to be simpler to use and understand https://review.openstack.org/105963 | 05:36 |
| *** Alexei_987 has quit IRC | 06:06 | |
| *** YorikSar has quit IRC | 06:08 | |
| *** AAzza_afk is now known as AAzza | 06:11 | |
| *** mkoderer has joined #openstack-oslo | 06:26 | |
| *** rmcall has quit IRC | 06:44 | |
| *** AAzza is now known as AAzza_afk | 06:46 | |
| *** harlowja is now known as harlowja_away | 06:47 | |
| *** AAzza_afk is now known as AAzza | 07:02 | |
| *** dstanek is now known as dstanek_zzz | 07:02 | |
| *** viktors|afk has quit IRC | 07:10 | |
| *** praneshp_ has joined #openstack-oslo | 07:31 | |
| *** tkelsey has joined #openstack-oslo | 07:32 | |
| *** praneshp has quit IRC | 07:34 | |
| *** praneshp_ is now known as praneshp | 07:34 | |
| *** ujjain has joined #openstack-oslo | 07:38 | |
| *** praneshp has quit IRC | 07:47 | |
| *** dstanek_zzz is now known as dstanek | 07:53 | |
| *** dstanek is now known as dstanek_zzz | 08:03 | |
| *** viktors has joined #openstack-oslo | 08:05 | |
| *** nacim has joined #openstack-oslo | 08:07 | |
| *** YorikSar has joined #openstack-oslo | 08:08 | |
| *** ihrachyshka has joined #openstack-oslo | 08:17 | |
| *** ihrachyshka has quit IRC | 08:24 | |
| *** ihrachyshka has joined #openstack-oslo | 08:25 | |
| *** AAzza is now known as AAzza_afk | 08:29 | |
| *** stannie has joined #openstack-oslo | 08:37 | |
| *** Alexei_987 has joined #openstack-oslo | 08:52 | |
| *** ildikov has joined #openstack-oslo | 09:04 | |
| *** zhiyan is now known as zhiyan_ | 09:04 | |
| *** zhiyan_ is now known as zhiyan | 09:27 | |
| *** dstanek_zzz is now known as dstanek | 09:52 | |
| viktors | haypo: around? | 09:58 |
| *** alexpilotti has joined #openstack-oslo | 10:00 | |
| *** dstanek is now known as dstanek_zzz | 10:02 | |
| *** alexpilotti has quit IRC | 10:06 | |
| *** pblaho has joined #openstack-oslo | 10:07 | |
| *** jd__ has quit IRC | 10:09 | |
| *** haypo has quit IRC | 10:10 | |
| *** haypo has joined #openstack-oslo | 10:12 | |
| *** haypo has joined #openstack-oslo | 10:12 | |
| *** jd__ has joined #openstack-oslo | 10:13 | |
| *** dstanek_zzz is now known as dstanek | 11:00 | |
| *** ihrachyshka has quit IRC | 11:01 | |
| *** pcm_ has joined #openstack-oslo | 11:12 | |
| *** ihrachyshka has joined #openstack-oslo | 11:15 | |
| *** k4n0 has quit IRC | 11:26 | |
| *** yamahata has quit IRC | 11:38 | |
| *** yamahata has joined #openstack-oslo | 11:41 | |
| *** pblaho has quit IRC | 11:43 | |
| *** openstackgerrit has quit IRC | 11:47 | |
| *** mrda is now known as mrda-away | 11:50 | |
| *** mriedem has joined #openstack-oslo | 11:50 | |
| *** pcm_ has quit IRC | 11:56 | |
| *** dstanek is now known as dstanek_zzz | 12:10 | |
| haypo | viktors: hello | 12:10 |
| viktors | haypo: can I ask you a quick question about porting to python3 ? | 12:11 |
| viktors | haypo: what is the best replacement for socket._fileobject in python3.3 ? | 12:16 |
| *** pcm_ has joined #openstack-oslo | 12:23 | |
| haypo | viktors: i'm looking at the code | 12:25 |
| *** dims has joined #openstack-oslo | 12:25 | |
| haypo | viktors: you can get a file-like object using socket.makefile() | 12:25 |
| *** markmc has joined #openstack-oslo | 12:26 | |
| haypo | viktors: or do you mean that you want to instanciate directly a socket._fileobject object, without a socket object? | 12:26 |
| viktors | haypo: just looking into eventlet code :) | 12:29 |
| *** i159 has joined #openstack-oslo | 12:29 | |
| haypo | viktors: are you trying eventlet on python3? | 12:30 |
| viktors | haypo: a bit | 12:30 |
| haypo | therve: i know that you played with eventlet and python3 with cyril, did you send some patches upstream? | 12:30 |
| viktors | haypo: they are working on porting | 12:30 |
| therve | haypo, I sent one patch yeah | 12:31 |
| haypo | viktors: according to cyril & therve, eventlet port to python3 is incomplete | 12:31 |
| therve | haypo, https://github.com/eventlet/eventlet/pull/99 | 12:31 |
| haypo | therve: cool! | 12:31 |
| *** tkelsey has quit IRC | 12:31 | |
| viktors | haypo: yes I know it ) | 12:31 |
| haypo | therve: oh!? temoto closed your pull request 2 hours ago, why? | 12:32 |
| viktors | haypo: but we can help tham with it | 12:32 |
| haypo | viktors: sure | 12:32 |
| therve | Hum, no idea | 12:32 |
| haypo | viktors: ok, i see in greenio.py that makefile() creates a _fileobject directly | 12:33 |
| haypo | viktors: IMO you should copy/paste the makefile() method of Lib/socket.py of Python 3.5 | 12:33 |
| haypo | viktors: it uses a new SocketIO object and then standard buffered objects (BufferedRWPair, BufferedReader, BufferedWriter) and TextIOWrapper | 12:34 |
| haypo | viktors: http://hg.python.org/cpython/file/d1f89eb9ea1e/Lib/socket.py#l196 | 12:34 |
| viktors | haypo: thanks, let me look at it | 12:35 |
| therve | haypo, I think he deleted the parent branch | 12:35 |
| viktors | haypo: I have a doubt as for this line - https://github.com/eventlet/eventlet/blob/6c4823c80575899e98afcb12f84dcf4d54e277cd/eventlet/greenio.py#L422 | 12:36 |
| viktors | haypo: oh, sorry, I got it | 12:36 |
| viktors | haypo: ... or not :) | 12:36 |
| haypo | viktors: I think _fileobject was replaced with SocketIO | 12:36 |
| haypo | viktors: but SocketIO design is different if I remember correctly, you have to override different methods | 12:37 |
| viktors | haypo: yes, you right | 12:37 |
| haypo | viktors: hum, GreenPipe is for files, not for sockets. i'm wrong | 12:38 |
| viktors | haypo: I see custom implementation in some projects or using socket.SocketIO | 12:38 |
| *** dhellmann is now known as dhellmann_ | 12:38 | |
| *** Kiall has joined #openstack-oslo | 12:39 | |
| Kiall | dhellmann / viktors - Q re oslo.db. We had an issue a few days ago with a fairly unusual Percona Cluster quirk, it's likely something that needs to be fixed in oslo.db - but I'm not sure what the best approach is. Have a minute and I'll explain? :) | 12:40 |
| viktors | Kiall: shoot :) | 12:40 |
| Kiall | So - Percona cluster (ab)uses MySQL error codes rather than inventing new ones where it needs to communicate something non-standard to the client. We hit an issue where we ended up connected to a cluster member that had been partitioned from the rest of the cluster, but was in the process of rejoining. | 12:41 |
| Kiall | During this time - The joining member will happily accept connections, but all queries will be rejected with: ERROR 1047 (08S01): Unknown command | 12:42 |
| Kiall | This should probably be treated as a disconnect - and in the pre-oslo.db days i would have added 1047 to the list here: https://github.com/openstack/designate/blob/stable/icehouse/designate/sqlalchemy/session.py#L91 | 12:43 |
| Kiall | oslo.db now defers to SQLAlchemy for this list, rather than having it's own. which makes that fairly difficult. | 12:43 |
| Kiall | sorry - brb in a couple of mins and I can explain the fallout that causes SQLA, have a short meeting with ttx | 12:44 |
| viktors | Kiall: in real, I don't like the idea - "to catch ``Unknown command`` error and suppose, that mysql server has gone away in this case" | 12:47 |
| viktors | Kiall: I think we should ask zzzeek about this issue. He is the author of SQLA, so I trust him in such questions :) | 12:48 |
| Alexei_987 | markmc: Hi | 12:49 |
| Alexei_987 | markmc: could you please check this patch? https://review.openstack.org/#/c/104983/ | 12:49 |
| * dims says rechecks are driving me nuts | 12:54 | |
| *** HenryG has quit IRC | 13:01 | |
| *** zzzeek has joined #openstack-oslo | 13:03 | |
| *** dstanek_zzz is now known as dstanek | 13:04 | |
| *** eezhova has quit IRC | 13:08 | |
| haypo | viktors: do you still need my help for the _fileobjet thing? | 13:09 |
| viktors | haypo: it's would be nice | 13:12 |
| haypo | viktors: so, what is your question? :) | 13:12 |
| viktors | haypo: is there a simple replacement for it in python3 ? =) | 13:14 |
| haypo | viktors: no :) | 13:15 |
| haypo | viktors: python3 I/O has been rewritten from scratch for good reasons :) | 13:15 |
| viktors | haypo: maybe do you know, what is the best practice of porting such things to py3 ? | 13:16 |
| *** HenryG has joined #openstack-oslo | 13:19 | |
| HenryG | rpodolyaka: ping | 13:19 |
| Kiall | viktors: back - so yea, I do agree. Percona abusing a rare error code to mean something totally different sucks. | 13:19 |
| *** zzzeek has quit IRC | 13:20 | |
| rpodolyaka | HenryG: pong | 13:20 |
| Kiall | viktors: But - there's lots of people running OpenStack on Percona - which (to me anyway) means we need to find a way to make it work | 13:20 |
| Kiall | I'll talk to zzzeek when he comes on to see if this is something that should be fixed at the SQLA side | 13:20 |
| HenryG | rpodolyaka: Hi. I am rebasing a patch with db changes in neutron and hitting a problem with oslo.db in unit tests | 13:21 |
| rpodolyaka | HenryG: hmm, interesting | 13:21 |
| rpodolyaka | HenryG: can you show me the code? | 13:21 |
| HenryG | rpodolyaka: just a minute while I paste the error ... | 13:21 |
| HenryG | rpodolyaka: http://paste.openstack.org/show/85942/ | 13:24 |
| viktors | Kiall: some time ago I heard about the similar issue with Galera cluster. And folks solved it in a similar way. | 13:24 |
| * rpodolyaka looks | 13:24 | |
| HenryG | rpodolyaka: something to do with get_engine() I believe | 13:25 |
| viktors | Kiall: so lets ask also zzzeek opinion ) | 13:25 |
| HenryG | rpodolyaka: wait, that is not the error I was seeing yesterday | 13:27 |
| Kiall | viktors: well Percona Cluster is basically galera, just Percona's distribution of it | 13:30 |
| rpodolyaka | HenryG: that's really weird, as cfg is imported in that file | 13:30 |
| Kiall | and yea - happy to discuss with zzzeek | 13:30 |
| HenryG | rpodolyaka: sorry, I had changes there for debugging, here is the real error that I am hitting: http://paste.openstack.org/show/85943/ | 13:31 |
| *** jecarey has quit IRC | 13:33 | |
| rpodolyaka | HenryG: hmm, any chance you can upload a draft/WIP? | 13:33 |
| rpodolyaka | HenryG: or at least give the pdb -> args output in that frame :) | 13:34 |
| rpodolyaka | HenryG: looks like something is wrong with sql_connection | 13:34 |
| HenryG | rpodolyaka: pdb args --> http://paste.openstack.org/show/85944/ | 13:36 |
| rpodolyaka | HenryG: oops, sorry, I wasn't clear, I really meant /home/henry/Dev/neutron/.tox/py27/local/lib/python2.7/site-packages/oslo/db/sqlalchemy/session.py +650, right before the exception is raised | 13:38 |
| HenryG | rpodolyaka: it's unit test, so config should be from neutron/tests/etc/neutron.conf.test | 13:38 |
| rpodolyaka | wow, I didn't know we had that | 13:39 |
| HenryG | rpodolyaka: is it wrong for oslo.db? | 13:39 |
| rpodolyaka | HenryG: 'sqlite://' must be ok - this is something we use widely | 13:40 |
| * rpodolyaka checks | 13:41 | |
| HenryG | rpodolyaka: I think the error has something to do with mocking. When I run the test individually it works fine. When I run all unit tests (sequentially with nose, in this case) then it breaks when it hits this test case. | 13:41 |
| rpodolyaka | HenryG: probably. I'd start from checking create_engine() arguments | 13:43 |
| rpodolyaka | HenryG: seems like sql_connection is somehow not the string we expect | 13:43 |
| HenryG | rpodolyaka: thanks, I will do that | 13:45 |
| rpodolyaka | HenryG: np | 13:45 |
| *** tsufiev has quit IRC | 13:48 | |
| *** dteselkin has quit IRC | 13:48 | |
| *** dstanek is now known as dstanek_zzz | 13:52 | |
| haypo | therve: I asked "Why did you close the pull request?", temoto replied "I think it closed automatically as I deleted py3-greenio branch. Your work will not be lost." | 13:55 |
| *** zhiyan is now known as zhiyan_ | 13:56 | |
| *** tcammann has joined #openstack-oslo | 13:56 | |
| therve | haypo, Yeah that's what I thought | 13:56 |
| therve | haypo, What we should really ask is why they kept fixing the code without fixing the tests :/ | 13:57 |
| haypo | therve: just do it (ask) | 13:57 |
| haypo | ;) | 13:57 |
| therve | I won't like the answer | 13:58 |
| haypo | therve :) | 13:59 |
| *** markmcclain has joined #openstack-oslo | 14:02 | |
| *** lbragstad_ has joined #openstack-oslo | 14:03 | |
| *** lbragstad_ is now known as lbragstad | 14:03 | |
| *** ildikov has quit IRC | 14:06 | |
| *** dstanek_zzz is now known as dstanek | 14:08 | |
| *** jaosorior has joined #openstack-oslo | 14:09 | |
| *** ildikov has joined #openstack-oslo | 14:24 | |
| *** tcammann has quit IRC | 14:28 | |
| *** tcammann has joined #openstack-oslo | 14:29 | |
| *** bknudson has joined #openstack-oslo | 14:29 | |
| *** AAzza_afk is now known as AAzza | 14:30 | |
| *** dstanek is now known as dstanek_zzz | 14:34 | |
| *** morganfainberg_Z is now known as morganfainberg | 14:36 | |
| *** Alexei_987 has quit IRC | 14:37 | |
| *** arnaud has quit IRC | 14:41 | |
| *** dteselkin has joined #openstack-oslo | 14:44 | |
| *** arnaud has joined #openstack-oslo | 14:44 | |
| *** tsekiyama has joined #openstack-oslo | 14:44 | |
| *** mriedem has quit IRC | 14:45 | |
| *** tsufiev has joined #openstack-oslo | 14:47 | |
| *** mriedem has joined #openstack-oslo | 14:48 | |
| *** nacim has quit IRC | 14:54 | |
| *** dstanek_zzz is now known as dstanek | 15:08 | |
| *** dstanek is now known as dstanek_zzz | 15:09 | |
| *** viktors has quit IRC | 15:10 | |
| *** bknudson has left #openstack-oslo | 15:12 | |
| *** morganfainberg is now known as morganfainberg_Z | 15:19 | |
| *** zzzeek has joined #openstack-oslo | 15:19 | |
| *** morganfainberg_Z is now known as morganfainberg | 15:20 | |
| *** ihrachyshka has quit IRC | 15:21 | |
| *** SridharG has quit IRC | 15:30 | |
| *** lbragstad has quit IRC | 15:30 | |
| *** openstackgerrit has joined #openstack-oslo | 15:32 | |
| *** beagles has joined #openstack-oslo | 15:39 | |
| *** ihrachyshka has joined #openstack-oslo | 15:59 | |
| *** bknudson1 has joined #openstack-oslo | 15:59 | |
| *** dstanek_zzz is now known as dstanek | 16:04 | |
| *** bknudson1 has quit IRC | 16:06 | |
| *** lbragstad_ has joined #openstack-oslo | 16:16 | |
| *** lbragstad_ is now known as lbragstad | 16:16 | |
| *** bknudson has joined #openstack-oslo | 16:21 | |
| *** bknudson has left #openstack-oslo | 16:23 | |
| *** markmcclain has quit IRC | 16:25 | |
| *** arnaud has quit IRC | 16:32 | |
| *** arnaud has joined #openstack-oslo | 16:33 | |
| openstackgerrit | Ilya Pekelny proposed a change to openstack/oslo.db: Opportunistic migration tests https://review.openstack.org/93424 | 16:40 |
| openstackgerrit | Ilya Pekelny proposed a change to openstack/oslo.db: Implementation Alembic as migration engine https://review.openstack.org/99965 | 16:40 |
| *** devananda has joined #openstack-oslo | 16:42 | |
| devananda | anyone on the oslo.db team got a minute to chat about the sqlalchemy tests? I'm probably missing something obvious... | 16:42 |
| *** i159 has quit IRC | 16:43 | |
| devananda | but I don't see where the SessionErrorWrapperTestCase is inserting a test row into the table with which to actually test locking | 16:43 |
| devananda | or any of the errors it asserts are raised | 16:43 |
| zzzeek | devananda: looking | 16:44 |
| *** tkelsey has joined #openstack-oslo | 16:44 | |
| devananda | eg, test_flush_wrapper | 16:45 |
| zzzeek | devananda: I have a patch that removes all those tests :) but currently, where you see tbl.save(_session), that is inserting the row | 16:46 |
| devananda | it calls tbl.update({'foo': 10}) twice | 16:46 |
| devananda | but i dont see any insert | 16:46 |
| zzzeek | its the tbl.save call, that calls into a routine oslo.db has that does: session.add(obj); session.flush() | 16:46 |
| devananda | ah | 16:46 |
| zzzeek | devananda: which is also somethign i want to change :) | 16:46 |
| devananda | so these tests are relying on update-or-insert by a secondary key? | 16:48 |
| zzzeek | um | 16:48 |
| zzzeek | these are testing an insert/update that is on a col with a UNIQUE cosntraint, not the PK yes | 16:49 |
| devananda | yep | 16:49 |
| zzzeek | because they want to see the unique constraint violation error | 16:49 |
| devananda | the test code obfuscates whether they are inserting or updating -- as someone familiar with databases but not this abstraction layer, it wasn't obvious what was going on | 16:50 |
| devananda | thanks for clarifying :) | 16:50 |
| *** flaper87 is now known as flaper87|afk | 16:51 | |
| zzzeek | devananda: there are tests that are much more consistent in my patch here: https://review.openstack.org/#/c/105891/ | 16:53 |
| zzzeek | devananda: these dont use the ORM as I am moving the error handling to a more fundamental level | 16:53 |
| zzzeek | devananda: well the ones that do a real integration of a row does use the ORM :) but in a more direct way | 16:54 |
| devananda | zzzeek: so these tests are making assumptions that the storage engine is transactional, eg. InnoDB | 16:57 |
| devananda | zzzeek: is that asserted somewhere? | 16:57 |
| devananda | zzzeek: or rather, this test is making that assumption: https://review.openstack.org/#/c/104436/1/tests/sqlalchemy/test_sqlalchemy.py | 16:57 |
| zzzeek | devananda: right now these tests are all hardwired to SQLite which is transactional | 16:57 |
| devananda | eh? | 16:58 |
| zzzeek | devananda: if/when we have tests against MySQL that require transacvtional, we can lift some methods that I use in SQLAlchemy tests to ensure innodb | 16:58 |
| zzzeek | devananda: the oslo.db tests are fixed to sqlite in most cases at the moment | 16:58 |
| * devananda points at the mailing list thread discussing running this test against msyqldb to look for races in eventlet | 16:58 | |
| zzzeek | devananda: OK it appears in the current code you can set OS_TEST_DBAPI_CONNECTION to a mysql URL to override. | 17:00 |
| zzzeek | devananda: however i dont know that that env is set anywhere regularly | 17:00 |
| openstackgerrit | garyk proposed a change to openstack/oslo.vmware: Add support for using extensions https://review.openstack.org/100911 | 17:00 |
| zzzeek | devananda: this is yet *another* system I want to improve/make more consistent: https://bugs.launchpad.net/oslo/+bug/1339206 | 17:01 |
| zzzeek | devananda: however at the moment i think a lot of these tests wont run against any old backend and they heavily skew towards sqlite behavior atm | 17:01 |
| devananda | that's ... less than ideal | 17:02 |
| zzzeek | devananda: what is | 17:02 |
| devananda | not testing against mysql | 17:02 |
| zzzeek | devananda: current state of the tests? sure | 17:02 |
| zzzeek | devananda: hence my bug report | 17:02 |
| zzzeek | devananda: however, note that these tests specifically are only testing an internal mechanism, not that particular databases raise the error in this case | 17:03 |
| zzzeek | devananda: they are only testing that, if SQLAlchemy raises an exception of type “X”, that it is transformed into “Y" | 17:03 |
| zzzeek | devananda: its not testing that a MySQL InnoDB table knows how to emit a unique csontraint violation | 17:03 |
| devananda | look at the one I linked -- it's relying on a specific behavior of MVCC-capable databases | 17:03 |
| devananda | and teh interaction between concurrent transactions affecting the same row | 17:03 |
| zzzeek | devananda: that code is not merged | 17:04 |
| zzzeek | devananda: -1 it | 17:04 |
| devananda | except I *want* that to be tested :) | 17:05 |
| zzzeek | devananda: OK so what do you propose | 17:05 |
| devananda | test it against mysql | 17:05 |
| devananda | since that's where teh bug manifests, and the most frequently used db backend for openstack projects | 17:05 |
| zzzeek | devananda: seems like a good idea, this should for the moment be under a MySQL opportunistic test framework | 17:06 |
| zzzeek | devananda: see test_base.MySQLOpportunisticTestCase | 17:06 |
| * devananda looks | 17:06 | |
| *** lbragstad has quit IRC | 17:09 | |
| ihrachyshka | zzzeek: well it seems that setting raise_on_warnings to False in oslo.db does not help much, because it's alembic that is used for migration, I guess without even getting into oslo.db | 17:12 |
| zzzeek | ihrachyshka: the alembic config should be using oslo’s create_engine() | 17:13 |
| *** SridharG has joined #openstack-oslo | 17:13 | |
| zzzeek | ihrachyshka: however, i’ve no idae what it deos at the moment | 17:13 |
| *** harlowja_away is now known as harlowja | 17:13 | |
| zzzeek | ihrachyshka: a lot of migration thigns are changing right now i am waitiing for them to settle, my priority is converging all these db-connection / migrations/ test running into a single unified system | 17:14 |
| ihrachyshka | zzzeek: https://github.com/openstack/neutron/blob/master/neutron/db/migration/alembic_migrations/env.py#L77 | 17:14 |
| zzzeek | because it is all over the place right now | 17:14 |
| ihrachyshka | I guess it's not oslo.db, is it the reason why? | 17:14 |
| zzzeek | ihrachyshka: yes that should be changed IMO | 17:14 |
| ihrachyshka | zzzeek: roger :) | 17:15 |
| zzzeek | ihrachyshka: there’s the concept that all oslo.db projects shoudl have an almsot empty env.py file and that oslo.db should provide it | 17:15 |
| zzzeek | ihrachyshka: too many things are happening too fast so i am just trying to slowly work over oslo.db one file at a time | 17:16 |
| zzzeek | ihrachyshka: got my first big change in, now am waiting the requisite 1-2 weeks for it to hopefully be accepted :) then the next change, etc. | 17:16 |
| zzzeek | :) | 17:16 |
| zzzeek | ihrachyshka: going to take a long time | 17:16 |
| *** dstanek is now known as dstanek_zzz | 17:19 | |
| *** adrian_otto has joined #openstack-oslo | 17:21 | |
| *** praneshp has joined #openstack-oslo | 17:26 | |
| *** dhellmann_ is now known as dhellmann | 17:27 | |
| *** JayF has joined #openstack-oslo | 17:27 | |
| JayF | Is there a way in oslo.processutils to specify the working directory I want the command executed in? | 17:28 |
| *** harlowja has quit IRC | 17:30 | |
| *** harlowja has joined #openstack-oslo | 17:30 | |
| *** adrian_otto has left #openstack-oslo | 17:32 | |
| JayF | It appears subprocess.Popen() supports it in some python versions, but that support isn't in oslo :( | 17:33 |
| *** markmcclain has joined #openstack-oslo | 17:33 | |
| ihrachyshka | zzzeek: I think you're better working on multiple changes in parallel :) reviews take loooong this side of earth :) | 17:35 |
| dhellmann | JayF: it's likely you're just the first person to need that ability. You could add it to the code in oslo-incubator. | 17:37 |
| *** arnaud has quit IRC | 17:38 | |
| zzzeek | ihrachyshka: i will be doing that :) | 17:39 |
| *** dstanek_zzz is now known as dstanek | 17:40 | |
| zzzeek | dhellmann: not sure if i did the right ettiquete here - two folks posted reviews that add new features to the exception-rewriting system of oslo.db ; however, I’d prefer that my rework of the exception rewriting system be merged first, and they build on top of that. so I -1’ed them, “I’d prefer you dont merge this”. is that uncool or is that valid? | 17:40 |
| dhellmann | zzzeek: if you explained the case, it should be ok, since you'll save them some work in the long run | 17:41 |
| zzzeek | dhellmann: OK | 17:41 |
| zzzeek | dhellmann: yes i poiinted them to the newer idea. but im not sure of timing, if their issue is urgent, and mine is like, “oh mike’s big idae we need to take some time on that”, etc | 17:41 |
| zzzeek | nobody has made any comments on my blueprint :) | 17:42 |
| dhellmann | zzzeek: that'll be up to you guys to work out :-) | 17:42 |
| dhellmann | zzzeek: it's been a busy week, so I haven't had a chance to look at it yet | 17:42 |
| dhellmann | I hope to have that rectified soon | 17:43 |
| zzzeek | dhellmann: OK, wasn’t calling you out specifically :) | 17:43 |
| dhellmann | sure, just filling you in :-) | 17:43 |
| *** AAzza is now known as AAzza_afk | 17:45 | |
| *** zzzeek has quit IRC | 17:46 | |
| *** Alexei_987 has joined #openstack-oslo | 17:50 | |
| *** zzzeek has joined #openstack-oslo | 17:52 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Remove mirror testing from the integration script https://review.openstack.org/106125 | 17:53 |
| *** dstanek is now known as dstanek_zzz | 17:55 | |
| Alexei_987 | markmc: Hi | 17:56 |
| *** dstanek_zzz is now known as dstanek | 17:57 | |
| *** noelbk has joined #openstack-oslo | 18:07 | |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Remove mirror testing from the integration script https://review.openstack.org/106125 | 18:08 |
| noelbk | markmc: I'm working through a problem in oslo.messaging where RPC is timing out after a rabbit reconnect. Would you mind looking at https://bugs.launchpad.net/oslo/+bug/1338732? I'm trying to understand how impl_rabbit._consume is supposed to work | 18:09 |
| *** SridharG has quit IRC | 18:11 | |
| Alexei_987 | noelbk: I can explain how it's supposed to work, cause I've already spent a lot of time on it | 18:11 |
| noelbk | Alexei_987: thanks. what is self.do_consume for? | 18:12 |
| Alexei_987 | noelbk: Most likely it's timing out cause reply queue is destroyed when connection is lost | 18:12 |
| Alexei_987 | noelbk: IMHO you should not dive deep in cosume logic. Main idea is that you have several consumers that consume from different queues. Each time a message is received a callback is called and the callback is AMQPListener.__call__ method | 18:13 |
| noelbk | The publisher successfullly publishes the message. Do you think the reply_queue gets autodeleted if the publisher closes the connection? | 18:14 |
| Alexei_987 | noelbk: sure.. if queue is not marked durable it's destroyed when client disconnects | 18:14 |
| Alexei_987 | noelbk: wait.. I'll check what we have in the code | 18:14 |
| noelbk | I did try with auto_delete=False everywhere, but same result | 18:16 |
| Alexei_987 | options = {'durable': False, | 18:16 |
| Alexei_987 | queue is not durable for replies | 18:16 |
| Alexei_987 | you should check ReplyWaiter | 18:16 |
| Alexei_987 | noelbk: it creates a direct consumer but queue is not durable | 18:16 |
| Alexei_987 | so reply is lost after a reconnect | 18:16 |
| Alexei_987 | we could fix that by changing the logic how replies are processed | 18:17 |
| noelbk | ah, so what's the difference between auto_delete and durable? | 18:17 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Remove mirror testing from the integration script https://review.openstack.org/106125 | 18:17 |
| Alexei_987 | autodelete is exchange option not queue option | 18:18 |
| Alexei_987 | so exchange is deleted when there are no queues on it | 18:18 |
| Alexei_987 | however IMHO we should change our rpc driver so we would use single queue per process not per rpc call | 18:19 |
| *** tkelsey has quit IRC | 18:19 | |
| noelbk | That would make sense. I'll set durable everywhere and see if that solves my timeout first | 18:20 |
| Alexei_987 | noelbk: cool let me know the results | 18:20 |
| Alexei_987 | dhellmann: could you please review my patch? https://review.openstack.org/#/c/104983/ it has nice specs and diagrams | 18:22 |
| dhellmann | Alexei_987: I'll put that on my review list, but it may be a while before I get to it. The oslo.messaging cores will have better insight, so I'm likely to lean on them for a detailed review | 18:24 |
| *** ihrachyshka has quit IRC | 18:24 | |
| Alexei_987 | dhellmann: I'm trying to ask markmc to review it but cannot catch him online | 18:26 |
| Alexei_987 | dhellmann: thanks anyway | 18:26 |
| dhellmann | Alexei_987: give him time to get back to it | 18:26 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Remove mirror testing from the integration script https://review.openstack.org/106125 | 18:29 |
| openstackgerrit | Doug Hellmann proposed a change to openstack-dev/pbr: Remove mirror testing from the integration script https://review.openstack.org/106125 | 18:30 |
| *** arnaud has joined #openstack-oslo | 18:31 | |
| *** bknudson has joined #openstack-oslo | 18:33 | |
| *** bknudson has quit IRC | 18:37 | |
| *** lbragstad_ has joined #openstack-oslo | 18:37 | |
| *** mkoderer has quit IRC | 18:52 | |
| *** ihrachyshka has joined #openstack-oslo | 18:55 | |
| *** bknudson has joined #openstack-oslo | 18:56 | |
| *** ihrachyshka has quit IRC | 18:57 | |
| *** lbragstad_ is now known as lbragstad | 19:16 | |
| *** lbragstad has quit IRC | 19:18 | |
| *** lbragstad has joined #openstack-oslo | 19:20 | |
| *** flaper87|afk is now known as flaper87 | 19:30 | |
| *** pcm_ has quit IRC | 19:31 | |
| *** arnaud has quit IRC | 19:34 | |
| *** stannie has quit IRC | 19:47 | |
| *** zhiyan_ is now known as zhiyan | 19:52 | |
| *** arnaud has joined #openstack-oslo | 19:57 | |
| *** stannie has joined #openstack-oslo | 20:01 | |
| *** bknudson has quit IRC | 20:04 | |
| *** bknudson has joined #openstack-oslo | 20:06 | |
| *** nacim has joined #openstack-oslo | 20:12 | |
| *** bknudson has quit IRC | 20:17 | |
| *** nacim has quit IRC | 20:19 | |
| *** bknudson has joined #openstack-oslo | 20:26 | |
| *** pcm_ has joined #openstack-oslo | 20:27 | |
| noelbk | Alexei_987: That was it. setting all the queues to durable removed the timed out errors. | 20:28 |
| Alexei_987 | noelbk: nice to hear that but still it's not a production fix since you'll need to delete queues explicitly | 20:29 |
| Alexei_987 | noelbk: will you update bug description? | 20:29 |
| noelbk | Exactly. We like non-durable queues because they're faster and we don't have to explicitly clean them up. | 20:29 |
| Alexei_987 | I hope we'll be able to schedule major refactoring of messaging lib for next cycle | 20:30 |
| Alexei_987 | I have lot's of stuff that should be improved | 20:30 |
| Alexei_987 | in terms of performance code structure and error handling | 20:30 |
| *** bknudson has quit IRC | 20:30 | |
| openstackgerrit | Michael Bayer proposed a change to openstack/oslo.db: Check for mysql_sql_mode is not None in create_engine() https://review.openstack.org/106163 | 20:31 |
| noelbk | excellent. I'll look at setting amqp_durable_queues=true in our configs without hacking oslo.messaging | 20:32 |
| noelbk | We were looking at switching to zeromq instead. Any experience with that, or should we wait for the next refactor of impl_rabbit? | 20:33 |
| Alexei_987 | noelbk: would not recomment switching to zeromq any time soon | 20:33 |
| Alexei_987 | noelbk: it's the same bad as rabbit and almost not working | 20:34 |
| Alexei_987 | noelbk: I hope to land some patches soon greatly improving it | 20:34 |
| Alexei_987 | noelbk: but I fear it won't be ready until the end of the cycle | 20:34 |
| Alexei_987 | noelbk: so right now my opinion is that zeromq is a bad choice | 20:35 |
| noelbk | OK, that sounds good to me. Anything I can do to help with rabbit? | 20:35 |
| Alexei_987 | noelbk: not sure... we should ask our core team for this | 20:35 |
| Alexei_987 | noelbk: would be cool if you would join next meeting to ask the same | 20:35 |
| noelbk | When's the meeting? | 20:36 |
| openstackgerrit | Yuriy Taraday proposed a change to openstack/oslo.rootwrap: Add an option to run rootwrap as a daemon https://review.openstack.org/81798 | 20:36 |
| Alexei_987 | https://wiki.openstack.org/wiki/Meetings#Oslo_Team_meeting | 20:36 |
| Alexei_987 | noelbk: one of the important stuff yet uncovered is unittests for messaging | 20:37 |
| Alexei_987 | noelbk: I think we should start creating some kind of functionality tests for API classes and run them with different configuration of underling drivers | 20:38 |
| Alexei_987 | noelbk: we have some tests but still they could be improved greatly | 20:38 |
| Alexei_987 | noelbk: and such functionality tests would help us to land refactorings | 20:39 |
| noelbk | Yes, I tried running the oslo.messaging unit tests on my cluster and got about 35% failure after installing all dependencies | 20:39 |
| noelbk | How do you run functional tests? | 20:39 |
| *** pcm_ has quit IRC | 20:40 | |
| Alexei_987 | noelbk: that's one of the problems. right now we have tests with "fake" driver | 20:40 |
| *** pcm_ has joined #openstack-oslo | 20:40 | |
| Alexei_987 | noelbk: for functional tests we'll have to create new jenkins jobs that have real brokers installed | 20:41 |
| Alexei_987 | tox won't help us here since it cannot install brokers | 20:41 |
| Alexei_987 | noelbk: so I think a good start would be to run some tests on a devstack VM | 20:41 |
| Alexei_987 | noelbk: since it already has messaging broker and we can use it in jenkins | 20:41 |
| noelbk | Is there a way to get the brokers connect args into the oslo tests now? If not... environment variables? | 20:42 |
| Alexei_987 | noelbk: I guess we should discuss it on the meeting tomorrow | 20:43 |
| Alexei_987 | I will try to allocate some resources on functional tests too since I'm very interested in this topic | 20:44 |
| noelbk | Cool. In my calendar for 0900 PST | 20:45 |
| Alexei_987 | noelbk: ok see you tomorrow then. Will go get some sleep | 20:46 |
| Alexei_987 | noelbk: have a nice day | 20:46 |
| *** bknudson has joined #openstack-oslo | 20:47 | |
| *** bknudson has left #openstack-oslo | 20:47 | |
| *** dims has quit IRC | 20:49 | |
| haypo | coro_repr = '%s() done at %s:%s' % (coro_name, filename, lineno) | 20:49 |
| haypo | oops, sorry | 20:50 |
| *** dims has joined #openstack-oslo | 20:50 | |
| *** flaper87 is now known as flaper87|afk | 20:57 | |
| noelbk | Alexei_987: thanks a lot for your help! | 20:58 |
| *** harlowja is now known as harlowja_away | 21:01 | |
| *** harlowja_away is now known as harlowja | 21:08 | |
| openstackgerrit | Ben Nemec proposed a change to openstack-dev/hacking: Fix import check interaction with namespace modules https://review.openstack.org/106169 | 21:12 |
| openstackgerrit | Ben Nemec proposed a change to openstack-dev/hacking: Relax H305 and H307 checks https://review.openstack.org/106170 | 21:12 |
| openstackgerrit | Ben Nemec proposed a change to openstack-dev/hacking: Relax H305 and H307 checks https://review.openstack.org/106170 | 21:25 |
| openstackgerrit | Ben Nemec proposed a change to openstack-dev/hacking: Fix import check interaction with namespace modules https://review.openstack.org/106169 | 21:25 |
| *** zhiyan is now known as zhiyan_ | 21:26 | |
| boris-42 | zzzeek around? | 21:31 |
| zzzeek | sure | 21:31 |
| boris-42 | zzzeek one more crazy query for you | 21:32 |
| boris-42 | zzzeek http://paste.openstack.org/ | 21:32 |
| boris-42 | zzzeek it's glance image-list query | 21:32 |
| zzzeek | try again | 21:32 |
| zzzeek | boris-42: try again | 21:33 |
| boris-42 | ahah=) | 21:33 |
| boris-42 | zzzeek http://paste.openstack.org/show/86005/ | 21:33 |
| boris-42 | zzzeek sorry for that | 21:33 |
| boris-42 | zzzeek I made glance almost work | 21:34 |
| zzzeek | so wahts tthe Q | 21:34 |
| *** mrda-away is now known as mrda | 21:35 | |
| zzzeek | boris-42: this is a very inefficient query, tahts about all i can say | 21:35 |
| boris-42 | zzzeek yep | 21:35 |
| zzzeek | boris-42: it probably is written wrong on the SQLAlhcemy side, it has some hallmarks of some common mistakes | 21:36 |
| boris-42 | zzzeek lemme show you the code | 21:36 |
| zzzeek | boris-42: am i fixing this ? what are we doing | 21:36 |
| zzzeek | boris-42: thought you were only profiling | 21:36 |
| boris-42 | zzzeek ?) | 21:36 |
| boris-42 | zzzeek I don't know if you would like to make your wiki even better=) | 21:37 |
| boris-42 | zzzeek I can just point you to some painful places in openstack | 21:37 |
| boris-42 | =) | 21:37 |
| boris-42 | zzzeek so this is the issue https://github.com/openstack/glance/blob/master/glance/db/sqlalchemy/api.py#L485 =) | 21:37 |
| zzzeek | boris-42: OK can you just places these all somewhere, it basically will be alist of things I can look at at some point | 21:38 |
| boris-42 | zzzeek where?) | 21:38 |
| boris-42 | zzzeek to wiki? | 21:38 |
| boris-42 | zzzeek I mean I am going to start fixing everything | 21:38 |
| boris-42 | zzzeek when I get profiler out of box | 21:38 |
| boris-42 | so I won't spend billions of time to get such info | 21:38 |
| zzzeek | boris-42: well we just need to track these queries somewhere | 21:39 |
| zzzeek | boris-42: actually ive formatted it some more, its not as bad as I thought | 21:39 |
| zzzeek | boris-42: it seemed like it had some cartesian products but that is not the case | 21:39 |
| zzzeek | boris-42: http://paste.openstack.org/show/86007/ | 21:41 |
| boris-42 | zzzeek probably I should integrate some formatting for sql requests somewhere | 21:41 |
| zzzeek | boris-42: we can even just put launchpad bugs up, and tag them “slowquery” | 21:41 |
| *** dims_ has joined #openstack-oslo | 21:41 | |
| boris-42 | zzzeek there is one guy in glance | 21:41 |
| boris-42 | zzzeek that is tuning thig | 21:41 |
| boris-42 | this | 21:41 |
| boris-42 | zzzeek probably he can try to tune this as well | 21:41 |
| zzzeek | ok | 21:42 |
| zzzeek | the UNION can be UNION ALL so there is no comparison going on in there | 21:42 |
| zzzeek | but also MySQL sucks at subqueries | 21:42 |
| zzzeek | there seem to be eager loads tacked on afterwards, Id take those out | 21:43 |
| *** haypo has left #openstack-oslo | 21:43 | |
| zzzeek | anyway, going for a walk... | 21:43 |
| *** dims has quit IRC | 21:43 | |
| *** stannie has quit IRC | 21:44 | |
| boris-42 | zzzeek see you | 21:44 |
| *** JayF has left #openstack-oslo | 21:55 | |
| *** lbragstad has quit IRC | 22:12 | |
| *** lbragstad has joined #openstack-oslo | 22:13 | |
| openstackgerrit | A change was merged to openstack/oslo.config: Changes imports order to pass H305, enables check https://review.openstack.org/99639 | 22:17 |
| *** oomichi has joined #openstack-oslo | 22:24 | |
| *** markmc has quit IRC | 22:32 | |
| *** yamahata has quit IRC | 22:32 | |
| *** ujjain has quit IRC | 22:32 | |
| *** bnemec has quit IRC | 22:32 | |
| *** hyakuhei has quit IRC | 22:32 | |
| *** dstanek has quit IRC | 22:32 | |
| *** therve has quit IRC | 22:32 | |
| *** SergeyLukjanov has quit IRC | 22:32 | |
| *** dmitryme has quit IRC | 22:32 | |
| *** tsekiyama has quit IRC | 22:32 | |
| *** krotscheck has quit IRC | 22:32 | |
| *** dims_ has quit IRC | 22:32 | |
| *** rpodolyaka has quit IRC | 22:33 | |
| *** ajo has quit IRC | 22:33 | |
| *** sreshetnyak has quit IRC | 22:33 | |
| *** mriedem has quit IRC | 22:33 | |
| *** jd__ has quit IRC | 22:33 | |
| *** beagles has quit IRC | 22:33 | |
| *** devananda has quit IRC | 22:33 | |
| *** SlickNik has quit IRC | 22:33 | |
| *** Kiall has quit IRC | 22:33 | |
| *** Alexei_987 has quit IRC | 22:33 | |
| *** anteaya has quit IRC | 22:33 | |
| *** jogo has quit IRC | 22:33 | |
| *** yamahata__ has quit IRC | 22:33 | |
| *** arnaud has quit IRC | 22:33 | |
| *** zzzeek has quit IRC | 22:33 | |
| *** dteselkin has quit IRC | 22:33 | |
| *** noelbk has quit IRC | 22:33 | |
| *** jraim has quit IRC | 22:33 | |
| *** zhiyan_ has quit IRC | 22:33 | |
| *** harlowja has quit IRC | 22:33 | |
| *** openstackgerrit has quit IRC | 22:33 | |
| *** tcammann has quit IRC | 22:33 | |
| *** mtreinish has quit IRC | 22:33 | |
| *** jokke_ has quit IRC | 22:33 | |
| *** ekarlso has quit IRC | 22:33 | |
| *** gmurphy has quit IRC | 22:33 | |
| *** morganfainberg has quit IRC | 22:33 | |
| *** boris-42 has quit IRC | 22:33 | |
| *** oomichi has quit IRC | 22:33 | |
| *** lbragstad has quit IRC | 22:33 | |
| *** markmcclain has quit IRC | 22:33 | |
| *** hartsocks has quit IRC | 22:33 | |
| *** wendar has quit IRC | 22:33 | |
| *** jroll has quit IRC | 22:33 | |
| *** AAzza_afk has quit IRC | 22:33 | |
| *** mrda has quit IRC | 22:33 | |
| *** sileht has quit IRC | 22:33 | |
| *** toabctl has quit IRC | 22:33 | |
| *** dhellmann has quit IRC | 22:33 | |
| *** JoshNang has quit IRC | 22:33 | |
| *** praneshp has quit IRC | 22:33 | |
| *** tsufiev has quit IRC | 22:33 | |
| *** YorikSar has quit IRC | 22:33 | |
| *** GheRivero has quit IRC | 22:33 | |
| *** pcm_ has quit IRC | 22:33 | |
| *** ildikov has quit IRC | 22:33 | |
| *** HenryG has quit IRC | 22:33 | |
| *** mgagne has quit IRC | 22:33 | |
| *** ttx has quit IRC | 22:33 | |
| *** jaosorior has quit IRC | 22:33 | |
| *** russellb has quit IRC | 22:33 | |
| *** creiht has quit IRC | 22:33 | |
| *** flaper87|afk has quit IRC | 22:33 | |
| *** ruhe has quit IRC | 22:33 | |
| *** gpocentek has quit IRC | 22:33 | |
| *** bnemec has joined #openstack-oslo | 22:36 | |
| *** SergeyLukjanov has joined #openstack-oslo | 22:36 | |
| *** ujjain has joined #openstack-oslo | 22:36 | |
| *** dmitryme has joined #openstack-oslo | 22:36 | |
| *** markmc has joined #openstack-oslo | 22:36 | |
| *** yamahata has joined #openstack-oslo | 22:36 | |
| *** hyakuhei has joined #openstack-oslo | 22:36 | |
| *** therve has joined #openstack-oslo | 22:36 | |
| *** oomichi has joined #openstack-oslo | 22:36 | |
| *** lbragstad has joined #openstack-oslo | 22:36 | |
| *** dims_ has joined #openstack-oslo | 22:36 | |
| *** pcm_ has joined #openstack-oslo | 22:36 | |
| *** arnaud has joined #openstack-oslo | 22:36 | |
| *** noelbk has joined #openstack-oslo | 22:36 | |
| *** zzzeek has joined #openstack-oslo | 22:36 | |
| *** Alexei_987 has joined #openstack-oslo | 22:36 | |
| *** markmcclain has joined #openstack-oslo | 22:36 | |
| *** harlowja has joined #openstack-oslo | 22:36 | |
| *** praneshp has joined #openstack-oslo | 22:36 | |
| *** devananda has joined #openstack-oslo | 22:36 | |
| *** beagles has joined #openstack-oslo | 22:36 | |
| *** openstackgerrit has joined #openstack-oslo | 22:36 | |
| *** mriedem has joined #openstack-oslo | 22:36 | |
| *** tsufiev has joined #openstack-oslo | 22:36 | |
| *** tsekiyama has joined #openstack-oslo | 22:36 | |
| *** dteselkin has joined #openstack-oslo | 22:36 | |
| *** tcammann has joined #openstack-oslo | 22:36 | |
| *** ildikov has joined #openstack-oslo | 22:36 | |
| *** jaosorior has joined #openstack-oslo | 22:36 | |
| *** HenryG has joined #openstack-oslo | 22:36 | |
| *** Kiall has joined #openstack-oslo | 22:36 | |
| *** jd__ has joined #openstack-oslo | 22:36 | |
| *** YorikSar has joined #openstack-oslo | 22:36 | |
| *** mtreinish has joined #openstack-oslo | 22:36 | |
| *** boris-42 has joined #openstack-oslo | 22:36 | |
| *** rpodolyaka has joined #openstack-oslo | 22:36 | |
| *** anteaya has joined #openstack-oslo | 22:36 | |
| *** jogo has joined #openstack-oslo | 22:36 | |
| *** krotscheck has joined #openstack-oslo | 22:36 | |
| *** jokke_ has joined #openstack-oslo | 22:36 | |
| *** GheRivero has joined #openstack-oslo | 22:36 | |
| *** yamahata__ has joined #openstack-oslo | 22:36 | |
| *** ajo has joined #openstack-oslo | 22:36 | |
| *** hartsocks has joined #openstack-oslo | 22:36 | |
| *** mgagne has joined #openstack-oslo | 22:36 | |
| *** jraim has joined #openstack-oslo | 22:36 | |
| *** russellb has joined #openstack-oslo | 22:36 | |
| *** wendar has joined #openstack-oslo | 22:36 | |
| *** sreshetnyak has joined #openstack-oslo | 22:36 | |
| *** ekarlso has joined #openstack-oslo | 22:36 | |
| *** jroll has joined #openstack-oslo | 22:36 | |
| *** gmurphy has joined #openstack-oslo | 22:36 | |
| *** AAzza_afk has joined #openstack-oslo | 22:36 | |
| *** zhiyan_ has joined #openstack-oslo | 22:36 | |
| *** SlickNik has joined #openstack-oslo | 22:36 | |
| *** mrda has joined #openstack-oslo | 22:36 | |
| *** sileht has joined #openstack-oslo | 22:36 | |
| *** ttx has joined #openstack-oslo | 22:36 | |
| *** toabctl has joined #openstack-oslo | 22:36 | |
| *** creiht has joined #openstack-oslo | 22:36 | |
| *** flaper87|afk has joined #openstack-oslo | 22:36 | |
| *** JoshNang has joined #openstack-oslo | 22:36 | |
| *** morganfainberg has joined #openstack-oslo | 22:36 | |
| *** dhellmann has joined #openstack-oslo | 22:36 | |
| *** gpocentek has joined #openstack-oslo | 22:36 | |
| *** ruhe has joined #openstack-oslo | 22:36 | |
| openstackgerrit | A change was merged to openstack/oslo.config: generator: tweak how MultiStrOpt defaults are handled https://review.openstack.org/105671 | 22:38 |
| *** sreshetnyak has quit IRC | 22:38 | |
| *** sreshetnyak has joined #openstack-oslo | 22:38 | |
| *** bknudson has joined #openstack-oslo | 22:39 | |
| *** devananda has left #openstack-oslo | 22:59 | |
| *** pcm_ has quit IRC | 23:01 | |
| *** harlowja has quit IRC | 23:05 | |
| *** harlowja has joined #openstack-oslo | 23:05 | |
| openstackgerrit | Arnaud Legendre proposed a change to openstack/oslo.vmware: Store PBM wsdl in the oslo.vmware git repository https://review.openstack.org/102409 | 23:06 |
| *** bknudson has quit IRC | 23:09 | |
| *** lbragstad has quit IRC | 23:14 | |
| *** HenryG has quit IRC | 23:14 | |
| *** morganfainberg is now known as morganfainberg_Z | 23:22 | |
| *** markmcclain has quit IRC | 23:23 | |
| *** pcm_ has joined #openstack-oslo | 23:24 | |
| *** pcm_ has quit IRC | 23:27 | |
| *** pcm_ has joined #openstack-oslo | 23:27 | |
| *** pcm_ has quit IRC | 23:40 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/oslo.db: Updated from global requirements https://review.openstack.org/105173 | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!