Wednesday, 2015-01-07

« Tuesday, 2015-01-06 Index Thursday, 2015-01-08 »
*** barra204 has joined #openstack-dns00:05
*** ryanpetrello has quit IRC00:12
*** ryanpetrello has joined #openstack-dns00:22
*** barra204 has quit IRC00:37
*** ryanpetrello has quit IRC00:37
*** Stanley00 has joined #openstack-dns00:59
*** rmoe has quit IRC01:04
*** rmoe has joined #openstack-dns01:21
*** nosnos has joined #openstack-dns01:27
*** stanzgy has joined #openstack-dns01:35
*** ryanpetrello has joined #openstack-dns01:48
*** ryanpetrello has quit IRC01:58
*** nkinder has joined #openstack-dns02:14
*** nosnos has quit IRC03:20
*** vinod1 has joined #openstack-dns03:23
*** richm has quit IRC03:23
*** vinod1 has quit IRC03:23
*** chlong has quit IRC03:41
*** chlong has joined #openstack-dns03:42
*** chlong has quit IRC03:42
*** chlong has joined #openstack-dns03:44
*** chlong has quit IRC03:48
*** chlong has joined #openstack-dns03:48
*** GonZoPT has quit IRC03:58
*** mwagner_lap has joined #openstack-dns04:01
*** GonZo2K has joined #openstack-dns04:02
*** nosnos has joined #openstack-dns04:15
*** nihilifer has joined #openstack-dns05:35
openstackgerritRon Rickard proposed openstack/designate: Domain Creation Includes Zone Transfer For Proper Status  https://review.openstack.org/14534605:49
openstackgerritRon Rickard proposed openstack/designate: Separate Periodic Recovery from Periodic Sync  https://review.openstack.org/14540906:00
*** GonZo2K has quit IRC06:14
*** GonZo2K has joined #openstack-dns06:28
openstackgerritRon Rickard proposed openstack/designate: Pool Manager Throws An Error When No Servers are Defined  https://review.openstack.org/14541806:43
*** GonZo2K has quit IRC07:14
*** GonZo2K has joined #openstack-dns07:29
*** GonZo2K has quit IRC07:48
*** GonZo2K has joined #openstack-dns07:48
*** GonZo2K has quit IRC07:49
*** chlong has quit IRC08:07
*** GonZo2K has joined #openstack-dns08:27
*** nosnos has quit IRC09:00
*** GonZo2K has quit IRC09:01
*** nosnos has joined #openstack-dns09:01
*** jordanP has joined #openstack-dns09:05
*** jordanP has quit IRC09:05
*** jordanP has joined #openstack-dns09:05
*** nosnos has quit IRC09:08
*** nosnos has joined #openstack-dns09:10
*** eandersson has quit IRC09:48
*** eandersson has joined #openstack-dns09:49
*** boris-42 has quit IRC09:53
*** Stanley00 has quit IRC10:21
eanderssonMorning!10:28
eanderssonKiall: I am testing out your latest commit now, but seeing new errors. :D10:28
eanderssonoh nvm I missed something10:29
eanderssonOk def better now :D10:34
eanderssonIt managed to create 7 out of 8.10:34
eanderssonThis one is odd. I see the traceback in the sink logs, but not in central logs.10:36
eanderssonIt looks like it failed at some point, does a rollback and tries to delete the record (but as it does not exists, raises a RecordNotFound exception).10:37
*** chlong has joined #openstack-dns10:39
*** stanzgy has quit IRC10:56
eanderssonKiall: I changed the second return to use deepcopy as well and now it almost works.10:58
eanderssonI can use all 10 workers without error.10:58
eanderssonHonestly I think that it is just hitting the max retry limit now.10:59
*** untriaged-bot has joined #openstack-dns11:02
untriaged-botUntriaged bugs so far:11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140326711:02
uvirtbotLaunchpad bug 1403267 in designate "create_domain should handle status asynchronously" [Undecided,In progress]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140439511:02
uvirtbotLaunchpad bug 1404395 in designate "Pool manager attempts to periodically sync *all* zones" [Undecided,In progress]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140641411:02
uvirtbotLaunchpad bug 1406414 in designate "Delete zone fails to propagate to all (Bind) nameservers in a pool depending on threshold_percentage" [Undecided,New]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140820011:02
uvirtbotLaunchpad bug 1408200 in designate "Pool Manager Needs Better Logging" [Undecided,New]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140359111:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/128944411:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140820111:02
uvirtbotLaunchpad bug 1403591 in designate "A ZeroDivisionError is Thrown Without Servers" [Undecided,In progress]11:02
uvirtbotLaunchpad bug 1289444 in designate "Designate with postgres backend is having issues" [Undecided,New]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140452911:02
uvirtbotLaunchpad bug 1408201 in designate "Pool Manager Needs Unit Tests" [Undecided,New]11:02
uvirtbotLaunchpad bug 1404529 in designate "DynECT is called twice when any domain action happens." [Undecided,Confirmed]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140820211:02
uvirtbotLaunchpad bug 1408202 in designate "Statuses in Pool Manager Cache Are Persistent" [Undecided,New]11:02
*** untriaged-bot has quit IRC11:02
*** boris-42 has joined #openstack-dns11:06
*** chlong has quit IRC11:08
eanderssonOk so I was hitting the max retry limit.11:09
eanderssonKiall: Modifing like 95 (result = f(self, *args, **kwargs)) to be a deepcopy as well and it works a 100%11:10
eandersson*line11:10
*** mwagner_lap has quit IRC11:54
*** nihilifer has quit IRC12:17
*** nihilifer has joined #openstack-dns12:18
*** nosnos has quit IRC12:43
*** ryanpetrello has joined #openstack-dns12:44
*** EricGonczer_ has joined #openstack-dns13:21
*** EricGonczer_ has quit IRC13:31
*** nihilifer has quit IRC13:31
*** betsy has joined #openstack-dns13:45
*** kbyrne has quit IRC13:50
*** betsy has quit IRC13:51
Kialleandersson: yea, I think my latest patchset does that14:12
Kiallbut it breaks lots of the tests (and possibly code?) - as they rely on the mutate in place that existed before the copy :/14:13
eanderssonActually, I had to modify the latest patch.14:14
eanderssonhttps://review.openstack.org/#/c/134524/6/designate/central/service.py14:14
eanderssonLine 95 had to be a deepcopy as well for it to work.14:14
*** mwagner_lap has joined #openstack-dns14:14
*** richm has joined #openstack-dns14:14
eanderssonNot sure why that would be needed though.14:15
eanderssonI increased the retry limit on my patches, although unless you have 10 workers (like in my dev environment) it shouldn't be needed.14:17
KiallYea - the deepcopy is what I have in the latest code, but it causes other issues that need fixing :/14:18
Kiallyea - the retry count will be a config option once I'm done, just need to fix the casualties of adding the deepcopy soon14:21
*** nkinder has quit IRC14:25
openstackgerritKiall Mac Innes proposed openstack/designate: Retry transactions on database deadlocks  https://review.openstack.org/13452414:42
Kialleandersson: ^ is closer.. Still not 100% though...14:42
eanderssonOh yea that is the same as I am running with at the moment.14:43
eanderssonbtw  _retry_on_deadlock is no longer needed right?14:43
KiallYea, the rety lib I was using originally doesn't give me a way to do the deepcopy :'(14:44
KiallSo - Had to rewrite ..14:44
openstackgerritKiall Mac Innes proposed openstack/designate: Retry transactions on database deadlocks  https://review.openstack.org/13452414:45
Kiallpep8 fixed.. Hopefully passes all tests etc :)14:45
Kiallah - I never removed that function -_-14:46
KiallHumm.. Looking at it, i should probably keep it.. the current code will retry on any exception - which isn't right14:47
eanderssonSo happy to see that I can create a huge number of VM's now without any errors. :D14:50
*** EricGonczer_ has joined #openstack-dns14:50
*** kbyrne has joined #openstack-dns14:52
openstackgerritKiall Mac Innes proposed openstack/designate: Retry transactions on database deadlocks  https://review.openstack.org/13452414:57
Kialleandersson: :)14:57
KiallIn theory ... I'm done -_-14:57
*** bauruine has quit IRC14:58
eanderssonWell it is sure looking good. :D14:58
eanderssonPreviously I broke something just setting up, but nothing I have been able to throw at it now has generated a single error message.14:59
Kiall:D14:59
openstackgerritKiall Mac Innes proposed openstack/designate: Retry transactions on database deadlocks  https://review.openstack.org/13452415:00
KiallCan you try the absolute latest patchset, it'll log "Deadlock detected. Retrying..." again when things go bad..15:00
KiallYou'll be able to see then how often the TX had to be retried..15:00
eanderssonhmmm15:01
eanderssonif RETRY_STATE.retries >= retries:15:01
KiallYep?15:01
eanderssonI don't see the declaration for that15:01
KiallL73 :)15:01
Kiall            if not hasattr(RETRY_STATE, 'held'):15:02
Kiall                # Create the state vars if necessary15:02
Kiall                RETRY_STATE.held = False15:02
Kiall                RETRY_STATE.retries = 015:02
eanderssonYea the RETRY_STATE15:02
eanderssonbut not retries15:02
Kiall(we just check for "held", we know they are always init'd together)15:02
eandersson*RETRY_STATE.retries, but not retries15:02
KiallThat comes from L6515:02
eanderssonoh lol15:03
eanderssonI was only looking at the wrapper15:03
eandersson:D15:03
*** timsim has joined #openstack-dns15:03
*** betsy has joined #openstack-dns15:04
eanderssonIf f(...) raises an exception, will result still be declared? :D15:05
eanderssonI assume it will, but not a 100 on how Python handles this.15:05
*** bauruine has joined #openstack-dns15:07
KiallNo, there won't be a result var if f() raises15:09
Kiallbut, it'll also never get as far as `return result` if it raises - so it's OK15:10
eanderssonI was more worried about return result triggering a NameError exception. =]15:10
eanderssonah kk15:10
eanderssonah yea it doesnt break15:10
eanderssonneed some more coffee :p15:10
Kiallyep :)15:10
KiallIf you have a backport to Juno, could you submit it when you get a chance? :)15:12
eanderssonFor sure. They are fixing my internet tomorrow.15:14
eanderssonI ran into a new issue now though :p15:14
eanderssonChecking for typos on my side.15:15
eanderssonTypeError: outer() takes exactly 1 argument (6 given)15:15
eanderssonIt's being thrown from https://github.com/openstack/designate/blob/master/designate/central/service.py#L13215:15
*** nkinder has joined #openstack-dns15:15
eanderssonoh nvm15:16
eanderssonI missed one change.15:16
Kiall:)15:16
eandersson@retry(cb=_retry_on_deadlock)15:16
eandersson:D15:16
*** ryanpetrello_ has joined #openstack-dns15:17
eanderssonok all good15:18
*** ryanpetrello has quit IRC15:19
*** ryanpetrello_ is now known as ryanpetrello15:19
*** vinod1 has joined #openstack-dns15:21
openstackgerritKiall Mac Innes proposed openstack/designate: Retry transactions on database deadlocks  https://review.openstack.org/13452415:39
*** EricGonc_ has joined #openstack-dns15:50
*** EricGonczer_ has quit IRC15:53
*** vinod1 has quit IRC16:28
*** betsy has quit IRC16:28
*** betsy has joined #openstack-dns16:30
eanderssonKiall: One thing I noticed with the new code is that when something fails, I don't see the traceback in the logs.16:31
eanderssonIn the Sink I can clearly see the error message, but not in Central16:31
Kialleandersson: really? I can't think of a reason why it won;'t be logged after it's hit the max # of retries?16:33
eanderssonAh actually16:33
eanderssonI think it is succesful16:33
eanderssonbut still sends the traceback to Sink16:33
eanderssonNot sure lol16:34
eanderssonI'll have to go in and check if all DNS entries are there.16:34
Kialllol - you might be confused, I can't see how the code could change that behaviour :/16:34
eanderssonI usually am :p16:34
Kiall:D16:34
eanderssonbut I did something like this on both servers16:35
eanderssongrep RecordNotFound designate-central.log16:35
eanderssonor grep "raise exc_notfound()" designate-central.log16:35
eanderssonand nothing16:35
eanderssonbut with sink I get16:35
eandersson2015-01-07 16:24:58.997 316 TRACE oslo.messaging.notify.dispatcher     raise exc_notfound()16:35
*** vinod1 has joined #openstack-dns16:36
KiallNotFound exceptions won't be logged on the "server" side of a RPC call16:36
eanderssonah16:36
eanderssonah yea confirmed that in the lab :p16:37
eanderssonI didn't test with multiple instances in the lab, only multiple workers.16:38
eanderssonGonna try to increase the workers and see what happens16:39
eanderssonOk, so apparently I didn't test your latest push with _retry_on_deadlock implemented.16:56
eanderssonIf I always retry, regardless of exception it works.16:56
KiallHumm16:56
KiallCould it be a backport bug? I think the unit tests I added show it definatly retires :/16:57
eanderssonIt could be, but I see retries with the old code as well.16:57
eandersson2015-01-07 16:24:58.613 9182 WARNING designate.central.service [req-44c5bfaa-c620-42ab-83c3-668eb144b377 - - - - -] Deadlock detected. Retrying...16:58
eanderssonThis is before I changed it to catch all.16:58
KiallThat looks like it's working?16:58
eanderssonYep. Actually. I see it erroring out when actually retrying.16:59
*** pk_ has joined #openstack-dns16:59
* Kiall is confused!16:59
eanderssonSo for some reason it catches it, and then fails (probably with the same issue)?16:59
eanderssonFile "/usr/local/designate/local/lib/python2.7/site-packages/designate/central/service.py", line 96, in wrapper17:00
Kiallgot a full set of logs etc?17:00
eandersson**copy.deepcopy(kwargs))17:00
eanderssonSure. It's a bit of a mess as it is from the sink17:00
KiallYea, I should be able to follow it :)17:00
timsimMeeting time?17:01
mugsieIRC Meet?17:01
timsim:P17:01
mugsie:)17:01
mugsiegreat minds ;)17:01
eanderssonhttp://paste.openstack.org/show/pjcWeazOaLu0OERYnKqT/17:01
Kiall;)17:01
eanderssonThis is when creating a record, but I see similar errors when deleting.17:02
eandersson2015-01-07 16:53:12.673 12569 WARNING designate.central.service [req-4508b697-13af-4432-829e-63a60d05f977 - - - - -] Deadlock detected. Retrying...17:02
*** untriaged-bot has joined #openstack-dns17:02
untriaged-botUntriaged bugs so far:17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140326717:02
uvirtbotLaunchpad bug 1403267 in designate "create_domain should handle status asynchronously" [Undecided,In progress]17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140641417:02
uvirtbotLaunchpad bug 1406414 in designate "Delete zone fails to propagate to all (Bind) nameservers in a pool depending on threshold_percentage" [Undecided,New]17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140452917:02
uvirtbotLaunchpad bug 1404529 in designate "DynECT is called twice when any domain action happens." [Undecided,Confirmed]17:02
eanderssonPretty sure that is the one.17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140359117:02
uvirtbotLaunchpad bug 1403591 in designate "A ZeroDivisionError is Thrown Without Servers" [Undecided,In progress]17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/128944417:02
uvirtbotLaunchpad bug 1289444 in designate "Designate with postgres backend is having issues" [Undecided,New]17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140820117:02
uvirtbotLaunchpad bug 1408201 in designate "Pool Manager Needs Unit Tests" [Undecided,New]17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140820017:02
uvirtbotLaunchpad bug 1408200 in designate "Pool Manager Needs Better Logging" [Undecided,New]17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/140820217:02
uvirtbotLaunchpad bug 1408202 in designate "Statuses in Pool Manager Cache Are Persistent" [Undecided,New]17:02
*** untriaged-bot has quit IRC17:02
eanderssonbrb got a meeting :D17:02
*** penick has joined #openstack-dns17:14
*** rmoe has quit IRC17:20
*** jordanP has quit IRC17:24
*** penick has quit IRC17:26
*** penick has joined #openstack-dns17:30
*** rmoe has joined #openstack-dns17:34
*** pk_ has quit IRC17:57
*** pk_ has joined #openstack-dns18:04
*** pk_ has quit IRC18:04
*** pk_ has joined #openstack-dns18:04
*** stanzgy has joined #openstack-dns18:07
*** stanzgy has quit IRC18:13
*** mwagner_lap has quit IRC18:19
*** pk_ has quit IRC18:20
*** vinod1 has quit IRC19:11
*** ryanpetrello_ has joined #openstack-dns19:16
*** mwagner_lap has joined #openstack-dns19:17
*** ryanpetrello has quit IRC19:18
*** ryanpetrello_ is now known as ryanpetrello19:18
*** GonZo2K has joined #openstack-dns19:32
*** GonZo2K has quit IRC19:53
*** GonZo2K has joined #openstack-dns19:54
openstackgerritRon Rickard proposed openstack/designate: Pool Manager Throws An Error When No Servers are Defined  https://review.openstack.org/14541819:55
*** ryanpetrello_ has joined #openstack-dns20:08
*** ryanpetrello has quit IRC20:08
*** ryanpetrello_ is now known as ryanpetrello20:08
*** nkinder has quit IRC20:20
*** vinod1 has joined #openstack-dns20:22
openstackgerritTim Simmons proposed openstack/designate: WIP - Experimental Agent Pool Manager Backend  https://review.openstack.org/14451020:43
openstackgerritTim Simmons proposed openstack/designate: Agent - Basic Service  https://review.openstack.org/14081520:43
openstackgerritTim Simmons proposed openstack/designate: Agent - Add AXFR Capability  https://review.openstack.org/14081720:43
openstackgerritTim Simmons proposed openstack/designate: Agent - Add basic NOTIFY support  https://review.openstack.org/14081620:43
openstackgerritTim Simmons proposed openstack/designate: Agent - Add Backend Capabilities  https://review.openstack.org/14081920:43
openstackgerritTim Simmons proposed openstack/designate: Agent - Add Support for Receiving Private CLASS/RRDATA Messages  https://review.openstack.org/14081820:43
timsimSorry guise20:44
ahuyou do realise you'll end up authoring DNSSEC non-empty terminal wildcard synthesis for NXDOMAINs, right? ;-)20:55
timsimahu: https://i.imgur.com/2RMOh.jpg20:57
ahu;-)20:59
ahuwell, that is what lies at the end of the path of adding AXFR, notify, & private class RRDATA messages21:00
ahua full blown nameserver21:00
timsimWe are DEFINITELY not planning to do that :P But I could see how it could happen :P21:02
*** openstackgerrit has quit IRC21:05
*** openstackgerrit has joined #openstack-dns21:05
*** ChanServ sets mode: +v openstackgerrit21:05
openstackgerritRon Rickard proposed openstack/designate: Pool Manager Throws An Error When No Servers are Defined  https://review.openstack.org/14541821:07
ahuno one has succeeded in writing a 'mini nameserver'21:08
ahubut again let me reiterate, powerdns is more than willing to bend to a lot your needs21:08
*** stanzgy has joined #openstack-dns21:09
timsimNoted.21:09
*** stanzgy has quit IRC21:14
*** nkinder has joined #openstack-dns21:43
*** chlong has joined #openstack-dns21:57
*** chlong has quit IRC22:03
*** vinod1 has quit IRC22:26
*** betsy has quit IRC22:33
*** EricGonc_ has quit IRC22:34
*** EricGonczer_ has joined #openstack-dns22:34
*** EricGonczer_ has quit IRC22:40
*** timsim has quit IRC22:51
rjrjrtimsim/vinod: just wanted to let you know i am looking into removing the status from pool manager when they are no longer needed.23:16
rjrjri agree with what you both are saying.23:17
*** chlong has joined #openstack-dns23:22
*** GonZo2K has quit IRC23:23
*** vinod1 has joined #openstack-dns23:56
*** penick has quit IRC23:56
« Tuesday, 2015-01-06 Index Thursday, 2015-01-08 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!