Tuesday, 2014-10-21

*** zigo has quit IRC00:07
*** rmoe has quit IRC00:07
*** jmcbride has joined #openstack-dns00:12
*** zigo has joined #openstack-dns00:13
*** rmoe has joined #openstack-dns00:24
*** EricGonczer_ has joined #openstack-dns01:41
*** tryggvil has quit IRC01:42
*** richm has quit IRC01:53
*** nosnos has joined #openstack-dns02:02
*** EricGonczer_ has quit IRC02:46
*** nosnos has quit IRC03:26
*** nosnos has joined #openstack-dns03:27
*** nosnos has quit IRC03:32
*** nosnos has joined #openstack-dns04:16
*** k4n0 has joined #openstack-dns05:50
*** ryanpetrello has quit IRC05:55
*** ryanpetrello has joined #openstack-dns05:55
*** amcrn has quit IRC07:20
*** bauruine_ has quit IRC07:46
*** jordanP has joined #openstack-dns07:46
*** openstackgerrit has quit IRC08:34
*** openstackgerrit has joined #openstack-dns08:34
*** ChanServ sets mode: +v openstackgerrit08:34
*** viktors has joined #openstack-dns09:07
*** mugsie has quit IRC09:27
*** mugsie has joined #openstack-dns09:29
*** tryggvil has joined #openstack-dns09:30
*** Wiscontiy has joined #openstack-dns09:42
Kiallrjrjr: I'm personally happy to merge any Pools code that doesn't break existing code - even if the code isn't "usable" without more code.. e.g. your code shouldn't have to wait in the review queue too long :)09:51
rjrjrkiall: okay.  give me about 1 hour.  i have the pool manager database worked out.  just going through the code now fixing it up.09:52
Kiallrjrjr: cool :) I have a few hours later set aside to do reviews09:54
KiallAlso .. Why are you awake? lol09:54
Kiall3am!09:54
*** Wiscontiy has quit IRC09:58
*** nosnos has quit IRC10:02
*** nosnos has joined #openstack-dns10:03
*** nosnos has quit IRC10:08
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add Server object validations  https://review.openstack.org/12962510:08
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Implement a DesignateObject Registry  https://review.openstack.org/12959910:08
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add basic validation functionality to Designate Objects  https://review.openstack.org/12784610:08
rjrjri have my day job and this is my night job. 8^)10:12
openstackgerritRon Rickard proposed a change to openstack/designate: Pool Manager Service Changes  https://review.openstack.org/12830010:32
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add Server object validations  https://review.openstack.org/12962510:34
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add basic validation functionality to Designate Objects  https://review.openstack.org/12784610:34
Kiallrjrjr: Q - PoolManagerBackend and Bind9PoolManagerBackend etc feel a little "weird" to me, is the intent to always keep both kinds of backends in place, or will PoolManagerBackend's become the only kind of backends?10:40
rjrjrI think these become the only backends once pool manager is working.10:44
rjrjrso, PoolManagerBackend becomes Backend and Bind9PoolManagerBackend becomes Bind9Backend.10:46
KiallOkay - I'll have a proper look later today, that was the only thing that immediately jumped out at me at a glance :)10:46
rjrjri saw we did something similar with PowerDNSMDNSBackend.10:47
rjrjrexcept I needed another base class to encapsulate registering the configuration options.10:48
KiallKinda :) PowerDNSMDNSBackend extends the standard Backend class, while this adds a whole new hierarchy that doesn't pass through the standard Backend class :)10:49
KiallMaybe Backend -> PoolBackend -> Bind9PoolBackend?10:49
Kiall(reduces the boilerplate duplication and still offers a place for the shared config logic..)10:50
KiallAnyway .. As I said, I haven't done a proper review .. Just a quick glance, that might cause more pain  ;)10:50
rjrjri didn't want a bunch of empty methods. 8^)  90% of the methods in Backend are not needed going forward.10:50
KiallYep - PoolBackend would stub them out, so the "real" backends don't need to implement them :)10:51
KiallAnyway.. brb!10:51
rjrjri can do Backend -> PoolBackend -> Bind9PoolBackend11:00
rjrjrzuul/gerrit/jenkins is taking its good old time again.  i want to make sure this passes testing so i can go to bed. lol11:01
Kiallhah.. you can always run them locally :) `tox -r -epy27,pep8` would give you a clean run the same as the gate11:07
ekarlsowhat's the diff between the backend > poolbackend ?11:09
*** alokj has joined #openstack-dns11:10
rjrjri did flake8 and tox -e py27 -r.  i'll run pep8 too and head to bed.11:12
Kiallflake8 and pep8 are the same really.. 2 names for the same check11:13
Kiallpep8 is there because the gate used to do the actual pep8 tool, and it stuck when people moved to flake811:14
*** mwagner_lap has quit IRC11:14
ekarlsorjrjr: is poolbackend a newer kind of thing ?11:15
rjrjri've had my tests pass locally and then they fail on the review.  mostly this happens with the import stuff at the top of the files.11:15
rjrjrekarlso: poolbackend is backend stripped of 90% of the methods we don't need anymore (tsigkeys, recordset, record, server CRUD).  additionally, i use it to get the backend configurations for pool manager to instantiate the server (currently pool_server) objects.11:17
ekarlsoah ok11:18
Kiallrjrjr: usually that's caused by having out of date requirements, tox `-r` causes it to reinstall all deps, which helps avoid it :)11:18
rjrjri had this problem just last week and i always run a tox -e py27 -r and flake8 right before i push the code for review.11:19
KiallOh.. Weird..11:19
rjrjri'm just lucky i guess.  lol11:20
rjrjrthe exact problem was i had the different imports mushed together without blank lines.  it passed locally.  it failed on review.11:21
rjrjri don't remember the exact file, but it was from oslo.XXX followed by a from designate.XXX without a blank line between or something like that.11:22
ekarlsohttps://bugs.launchpad.net/designate/+bug/1383670 < I added that if anyone has opinions : )11:25
uvirtbotLaunchpad bug 1383670 in designate "Migrate to Alembic for Migrations" [Wishlist,New]11:25
rjrjrwell, the tox tests passed, so i'm going to take the chance and go to bed.  i'll be back on in 5-6 hours.11:26
*** k4n0 has quit IRC11:37
*** k4n0 has joined #openstack-dns11:38
*** k4n0 has quit IRC11:46
*** k4n0 has joined #openstack-dns11:46
*** ttrumm has joined #openstack-dns11:48
*** EricGonczer_ has joined #openstack-dns12:03
*** mwagner_lap has joined #openstack-dns12:11
*** EricGonczer_ has quit IRC12:12
*** paul_glass has joined #openstack-dns12:14
*** k4n0 has quit IRC12:28
*** ttrumm has quit IRC12:30
*** richm has joined #openstack-dns13:06
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Implement a DesignateObject Registry  https://review.openstack.org/12959913:09
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add basic validation functionality to DesignateObjects  https://review.openstack.org/12784613:09
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Support Nested/Recursive Object Validations  https://review.openstack.org/12989513:09
KiallI love when changes end up easier than you expect :D https://review.openstack.org/#/c/129895/ is a perfect example of one of those things I thought would be a PITA ;)13:11
*** EricGonczer_ has joined #openstack-dns13:26
*** jmcbride has quit IRC13:37
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add Server object validations  https://review.openstack.org/12962513:47
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add Domain object validations  https://review.openstack.org/12990913:47
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add Server object validations  https://review.openstack.org/12962513:48
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add Domain object validations  https://review.openstack.org/12990913:48
*** paul_glass has quit IRC13:49
*** paul_glass has joined #openstack-dns13:59
*** rmoe has quit IRC14:19
*** rmoe has joined #openstack-dns14:24
*** jmcbride has joined #openstack-dns14:28
*** timsim has joined #openstack-dns14:31
*** alokj has quit IRC14:41
*** rmoe has quit IRC14:41
*** rmoe has joined #openstack-dns14:43
*** vinod1 has joined #openstack-dns14:49
*** jmcbride has quit IRC14:59
*** jmcbride has joined #openstack-dns15:00
*** paul_glass has quit IRC15:03
openstackgerritA change was merged to openstack/designate: Reduce SQLAlchemy loglevel during tests  https://review.openstack.org/12827815:03
*** vinod1 has quit IRC15:04
*** EricGonc_ has joined #openstack-dns15:06
*** EricGonczer_ has quit IRC15:06
*** timsim has quit IRC15:09
*** timsim has joined #openstack-dns15:12
*** vinod1 has joined #openstack-dns15:12
*** vinod1 has quit IRC15:16
*** jmcbride has quit IRC15:17
*** jmcbride1 has joined #openstack-dns15:18
*** vinod1 has joined #openstack-dns15:21
*** vinod1 has quit IRC15:41
*** vinod1 has joined #openstack-dns15:44
*** rmoe has quit IRC15:47
*** paul_glass has joined #openstack-dns15:56
*** viktors is now known as viktors|afk15:57
*** tryggvil has quit IRC15:58
KiallHumm.. The 4 pools patches weigh in at about 4k lines of code.. I only managed to set aside an hour to review them, that's not enough -_-16:02
*** tryggvil has joined #openstack-dns16:03
*** rmoe has joined #openstack-dns16:09
Kiallbetsy: about?16:11
betsyyep16:12
KiallHeya - Looking at your Server Pools Storage change..16:13
betsygot a question?16:13
KiallI'm still reviewing, but I think we might want to extend that review to actually remove the old "servers" stuff all together16:14
betsyok. I was gonna do that in a separate patch16:14
betsyTrying to keep them smaller. The code work needs to be done, too, before all the old server stuff is removed16:15
KiallFor me anyway, Ideally, a single migration handles the new tables, data move, and deletion of the old tables..16:15
betsyok16:15
KiallYea...16:15
betsyI can do that16:15
KiallAnyway - Still reviewing :)16:15
betsyOh here’s what I ran into with that...16:15
KiallWas just curious if you had though it it.. and it seems you have ;)16:15
betsyTo make the change to the domain tables, I need the pool_id from the pool table16:16
KiallAh.. My draft comment has a suggestion for that16:16
betsyOh cool.16:16
Kialllet me publish that comment real quick16:16
betsyYeah. I was thinking it needed to be separate patches because of that16:16
KiallDone..16:17
KiallThe suggestion basically boils down to pre-choosing the default pool's UUID16:17
KiallThat allows you to set the well-known UUID as the default value of the "default_pool_id" config option, and the migration can read from the config to get the ID it uses, allowing end users to "choose" a different one if they wish.. Similar to what Keystone a while back for a similar migration16:18
betsyOk. Can I read the config file from this to get the default pool name?16:19
KiallYep - You should just be able to import the CONF object like any other place16:19
betsySo, take all of the create the default pool out of central/service.py16:20
KiallI hadn't got that far :D16:20
betsyok16:20
KiallBut, pre-creating it let's us move stuff (e.g. servers) around and update every domain with pool_id = <UUID> during the migration16:21
Kiallrather than slowly as stuff get's updated over time (which could be years :D)16:21
betsyYeah, I was planning on doing it in two separate migrations becuase of the pool_id, but this solves that problem, as long as I can read the conf file to get the default pool name. The spec just has it being created in the central service16:23
KiallReally? I missed that, I would have said something about that :D16:24
betsyWell, unless I misread the spec. :)16:24
Kiall(specifically, updating all N thousand domains pool_id's to the just-created pool's ID really needs to be done in the migration)16:24
betsyRight. I was just going to have it as a separate migration. After this one was in place, and the defaut pool was created, then the subsequent migration would read that pool_id and update all the domains’ pool_ids16:26
Kiall(In case you can't tell - I'm trying to see how we can get stuff merged without duplicating efforts or breaking things between merges!)16:26
betsyBut if we can do it in one fell swoop, I’m fine with that16:26
*** jordanP has quit IRC16:26
betsyWell, I don’t think this patch would break anything because there’s no code change16:26
Kialland - your review kinds sits at at the core - so likely needs to go in first ;)16:26
betsyMore work needs to be done after this16:27
betsyBut if you want it done all in one patch, I’m fine with that too16:27
KiallYea - I *personally* think it's better in this case to handle removing servers in the same review, as the pools code you have in place seems to provide everything necessary to make that happen, and we get the advantage of an "atomic" 1 migration to remove the old and add the new :)16:28
betsyCool. I make those changes16:30
*** OpenStack_ has joined #openstack-dns16:31
KiallSome more comments up BTW - But looks great, the sooner we get this merged, the sooner others can build on it :)16:31
*** vinod1 has quit IRC16:31
*** OpenStack_ has left #openstack-dns16:32
Kiallbetsy: actually, looking at the servers code that would have to be removed, I'm not sure about removing servers stuff straight away.. More to that than I expected!16:39
betsyYeah16:40
KiallBut... I think the other comments around creating the default pool etc still apply?16:40
betsySure. I’ll go ahead and do that and then do the server changes in a separate patch16:40
betsyDoes that work?16:40
betsySo all the migration stuff is done at once16:40
betsyWell, except for dropping the server table, which would be done later16:41
KiallYep.. We'll probably still need another migration to move the servers data, but yea, makes sense16:41
betsyok16:41
Kialltimsim: about? Looking at your DB indexing patch, I think we can reduce the # of index's created by removing some of the "duplicates" e.g.:16:45
Kiall['zone_id_tenant', zones_table.c.id, zones_table.c.tenant_id],16:45
Kiall['zone_id_tenant_deleted', zones_table.c.id, zones_table.c.tenant_id,16:45
Kiall zones_table.c.deleted],16:45
Kiallthe second index there covers off the first index, so the first index won't improve read performance, but it will reduce write performance16:46
KiallAn index will be chosen based on the # of matches to the left hand side, e.g. a query with `WHERE id = 1 AND tenant_id = 2` can be satisfied by either of those indexes, and a query on `WHERE id = 1 AND deleted = 0` won't be satisfied by either index16:48
Kiall(though, ID is bad example ;) Since it will produce exactly 1 hit anyway :P)16:48
timsimKiall: Yeah I didn't want to present those as a legit answer. I did a barbaric analysis of some queries and blindly created those. There are certainly some duplicates there, and some that I'm missing I'm sure.16:49
Kiall:D16:49
timsimI'm trying to find time to do a more in depth query analysis and go over some of the stuff with one of our DBAs so I can get a more definitive set of indices to create16:50
KiallYea, DB indexing can be a bit of a black art sometimes.. lol16:51
Kiallhttp://www.percona.com/doc/percona-toolkit/2.1/pt-duplicate-key-checker.html <-- Likely a good way to remove "duplicates" quicky16:51
timsimI have almost no experience with it so I was just kind of flailing around in the dark.16:51
timsimNice, I'll definitely use that.16:52
Kiallpercona's toolkit is GREAT.. If you set the MySQL slow query log to 0 seconds, then point pt-query-digest at it, you can get some really useful pointers16:53
KiallOh.. there's a new PT tool I didn't know of.. http://www.percona.com/doc/percona-toolkit/2.2/pt-index-usage.html16:55
Kiall"This tool connects to a MySQL database server, reads through a query log, and uses EXPLAIN to ask MySQL how it will use each query. When it is finished, it prints out a report on indexes that the queries didn’t use."16:55
timsimThat's good advice, I'll do that. I was going to turn on query logging and do some tests and see what queries single API requests/MiniDNS ops execute.16:55
timsimThat looks useful.16:55
KiallYea, everytime I go looking for something MySQL related.. Percona has the answer ready to use ;)16:56
timsimI hadn't thought to look. I'll be looking at all those tools.16:57
*** mwagner_lap has quit IRC16:58
*** paul_glass has quit IRC17:07
*** zigo has quit IRC17:08
*** paul_glass has joined #openstack-dns17:08
*** zigo has joined #openstack-dns17:08
*** paul_glass has quit IRC17:08
*** vinod1 has joined #openstack-dns17:11
*** skyler_ has joined #openstack-dns17:12
*** paul_glass has joined #openstack-dns17:13
*** paul_glass has quit IRC17:14
vinod1Kiall, while you are up reviewing code - could you also review https://review.openstack.org/#/c/125868/17:15
vinod1I updated the code so that it works both with the current setup and going forward with the pool managers17:16
vinod1Also the object code has reached a point (for me) where a document would be helpful to understand what is going on :-)17:17
*** jmcbride1 has quit IRC17:23
*** jmcbride has joined #openstack-dns17:26
*** ryanpetrello has quit IRC17:27
*** amcrn has joined #openstack-dns17:27
Kiallvinod1: yea, I thought the same! There's still more to go to actually make use of it and replace the existing validation throughout17:29
KiallOn the plus side, it's uncovering dodgy code and tests already ;)17:30
*** jmcbride has quit IRC17:31
*** ryanpetrello has joined #openstack-dns17:31
KiallI'll try write something up this evening, for now though, time to head home before the rain starts!17:32
*** zigo has quit IRC17:41
*** zigo has joined #openstack-dns17:41
*** mwagner_lap has joined #openstack-dns17:42
rjrjrwhy did the code I submitted 7 hours ago never get reviewed by gated?17:44
rjrjrreviewed = tested17:46
rjrjrgated = zuul/jenkins17:50
*** jmcbride has joined #openstack-dns17:54
vinod1rjrjr: Looks like it just got done - Oct 21, 2014 12:53 PM17:56
vinod1the other designate patchsets too are sitting there for the last 5 hrs17:56
rjrjryeah, i ran a 'recheck no bug'.  glad that worked!17:58
rjrjris there a URL for the zuul console to look at the status of the checks?  we run these apps inhouse and i can look at the status to see what zuul is working on.17:59
vinod1http://status.openstack.org/zuul/18:01
vinod1You can set a filter to designate18:01
vinod1All the waiting tests are currently waiting for the devstack tests to complete18:02
*** jmcbride has quit IRC18:16
*** paul_glass has joined #openstack-dns18:24
*** timsim has quit IRC18:32
*** timsim has joined #openstack-dns18:36
*** jmcbride has joined #openstack-dns18:44
*** jmcbride has quit IRC18:46
*** jmcbride has joined #openstack-dns18:46
*** paul_glass has quit IRC19:05
*** paul_glass has joined #openstack-dns19:09
Kiallrjrjr: looks like openstack infra had some sort of failure earlier, queues have been backed up for nearly 24 hours19:28
Kiallwith 800 VMs doing testing.. lol19:28
KiallAnd I dislike when our internal zuul doesn't start on a change immediately ;)19:28
*** amcrn has quit IRC19:54
*** jmcbride has quit IRC20:00
*** jmcbride has joined #openstack-dns20:01
*** vinod1 has quit IRC20:02
*** jmcbride has quit IRC20:05
*** paul_glass has quit IRC20:06
*** RaginBaj_ has quit IRC20:12
*** tryggvil has quit IRC20:28
*** drawesome has joined #openstack-dns20:29
*** jmcbride has joined #openstack-dns20:29
*** paul_glass has joined #openstack-dns20:44
*** paul_glass1 has joined #openstack-dns20:49
*** paul_glass has quit IRC20:50
*** EricGonc_ has quit IRC21:08
*** timsim has quit IRC21:13
*** paul_glass1 has quit IRC21:38
*** tryggvil has joined #openstack-dns21:40
*** amcrn has joined #openstack-dns21:54
*** vinod1 has joined #openstack-dns22:01
*** vinod1 has quit IRC22:11
*** EricGonczer_ has joined #openstack-dns22:22
*** vinod1 has joined #openstack-dns22:22
*** EricGonczer_ has quit IRC22:29
*** jmcbride has quit IRC22:37
*** vinod1 has quit IRC22:40
*** bauruine has quit IRC23:07
*** bauruine has joined #openstack-dns23:08
*** skyler_ has quit IRC23:16
*** EricGonczer_ has joined #openstack-dns23:44
*** EricGonczer_ has quit IRC23:49
*** rmoe has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!