Tuesday, 2014-08-12

« Monday, 2014-08-11 Index Wednesday, 2014-08-13 »
*** nkinder has quit IRC00:10
*** EricGonczer_ has quit IRC00:20
*** agrebennikov has quit IRC00:43
*** dtx00ff has quit IRC00:52
*** dtx00ff has joined #openstack-dns00:53
*** dtx00ff has quit IRC01:04
*** richm has left #openstack-dns01:27
*** rossk has quit IRC01:29
*** msisk has joined #openstack-dns01:40
*** msisk has quit IRC01:41
*** msisk has joined #openstack-dns01:41
*** nosnos has joined #openstack-dns01:43
*** nosnos has quit IRC01:45
*** nosnos has joined #openstack-dns01:45
*** nosnos has quit IRC01:46
*** nosnos has joined #openstack-dns01:47
*** nkinder has joined #openstack-dns01:51
*** vinod1 has joined #openstack-dns02:27
*** vinod1 has quit IRC02:35
*** msisk has quit IRC03:07
*** nosnos has quit IRC03:22
*** crc32 has quit IRC03:41
*** nosnos has joined #openstack-dns04:25
*** k4n0 has joined #openstack-dns05:37
*** agrebennikov has joined #openstack-dns06:34
openstackgerritRon Rickard proposed a change to openstack/designate-specs: Server Pools - Storage  https://review.openstack.org/11344706:36
openstackgerritRon Rickard proposed a change to openstack/designate-specs: Server Pools - Storage  https://review.openstack.org/11344706:47
openstackgerritRon Rickard proposed a change to openstack/designate-specs: Server Pools - Storage  https://review.openstack.org/11344707:20
*** agrebennikov has quit IRC07:22
*** sballe has joined #openstack-dns07:27
*** Krast has joined #openstack-dns08:06
openstackgerritRon Rickard proposed a change to openstack/designate-specs: Server Pools - Manager  https://review.openstack.org/11346208:29
*** Krast has quit IRC08:40
*** Krast has joined #openstack-dns08:40
*** serverascode has quit IRC09:47
*** simonmcc_ has quit IRC09:48
*** serverascode has joined #openstack-dns09:48
*** CaptTofu_ has quit IRC09:49
*** simonmcc_ has joined #openstack-dns09:53
*** serverascode has quit IRC09:53
*** serverascode has joined #openstack-dns09:59
*** CaptTofu_ has joined #openstack-dns10:00
*** sballe has quit IRC10:46
*** Krast has quit IRC10:50
*** RaginBajin has joined #openstack-dns11:25
*** k4n0 has quit IRC12:53
*** EricGonczer_ has joined #openstack-dns13:01
*** nkinder has quit IRC13:11
*** nosnos has quit IRC13:13
*** nkinder has joined #openstack-dns13:56
*** vinod1 has joined #openstack-dns14:00
*** agrebennikov has joined #openstack-dns14:10
*** richm has joined #openstack-dns14:10
openstackgerritGraham Hayes proposed a change to openstack/designate-specs: Added olsosphinx theme, and enable incubation option for specs output  https://review.openstack.org/11353514:20
*** eankutse has joined #openstack-dns14:39
*** vinod1 has quit IRC14:43
*** zain has joined #openstack-dns14:43
*** timsim has joined #openstack-dns14:43
*** zain has quit IRC14:46
*** zain has joined #openstack-dns14:47
*** RaginBajin has quit IRC14:48
*** vinod1 has joined #openstack-dns14:55
*** vinod1 has quit IRC14:59
*** timsim_ has joined #openstack-dns15:01
*** timsim has quit IRC15:02
*** merlinguy has joined #openstack-dns15:07
merlinguycan anyone point me to documentation on how to write a backend agent similar to bind9?15:08
*** vinod1 has joined #openstack-dns15:08
mugsiemerlinguy: it is the same as writing a backend in general15:26
*** zain1 has joined #openstack-dns15:26
timsim_merlinguy: I'm not sure there is a specific document about how to do it (I could be totally wrong there) but essentially you need to implement what's described here (the base one): https://github.com/openstack/designate/blob/master/designate/backend/base.py15:26
mugsieyou just load the new backend in the agent15:26
*** agrebennikov has quit IRC15:26
mugsiebut - FYI backends are in the process of changing dramatically15:27
mugsieand the agent is .... in an interesting state15:27
*** zain1 has quit IRC15:27
timsim_Here's one that's pretty straightforward: (https://github.com/openstack/designate/blob/master/designate/backend/impl_nsd4slave.py)15:27
*** zain1 has joined #openstack-dns15:27
mugsiewhat are you trying to do?15:27
merlinguyI am picking up code from someone else and the new agent isn't being found when designate-agent starts.15:28
*** zain has quit IRC15:28
merlinguyI assumed that all I needed to do was create a new python file in designate/backend and then point to i in designate.conf15:29
merlinguyc/i/it15:29
mugsiemerlinguy: yup, but it has to go here: https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L11215:30
mugsieand this https://github.com/openstack/designate/blob/master/etc/designate/designate.conf.sample#L49 should be set to rpc15:31
merlinguyYeah, got that in there.  Thanks.15:32
merlinguyMight need to dump what I was given and just start from scratch.15:32
merlinguyPerhaps there is a syntax error in the code I inherited.15:32
mugsiemerlinguy: has https://github.com/openstack/designate/blob/master/setup.cfg#L66 been update as well?15:33
mugsiepointing to the new backend code?15:33
merlinguyI have been using someone else's imply_ file15:34
merlinguyLet me check the setup.cfg.  I think I already caught that.15:34
mugsiemerlinguy: if it is something that can be pasted on paste.openstack.org, I can take a look15:35
mugsieif not, do you have a stack trace?15:36
merlinguyyep15:36
merlinguy2014-08-12 15:36:14.694 28255 CRITICAL designate [-] RuntimeError: No 'designate.backend' driver found, looking for 'ro'15:36
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate Traceback (most recent call last):15:36
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate   File "/usr/local/bin/designate-agent", line 10, in <module>15:36
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate     sys.exit(main())15:36
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate   File "/home/stack/designate/designate/cmd/agent.py", line 35, in main15:36
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate     binary='designate-agent')15:36
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate   File "/home/stack/designate/designate/service.py", line 92, in create15:36
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate     endpoints=endpoints)15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate   File "/home/stack/designate/designate/agent/service.py", line 37, in __init__15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate     central_service=central_api)15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate   File "/home/stack/designate/designate/backend/__init__.py", line 25, in get_backend15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate     cls = Backend.get_driver(backend_driver)15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate   File "/home/stack/designate/designate/plugin.py", line 71, in get_driver15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate     mgr = driver.DriverManager(cls.__plugin_ns__, name)15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate   File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 43, in __init__15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate     verify_requirements=verify_requirements,15:37
richmplease use paste15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate   File "/usr/local/lib/python2.7/dist-packages/stevedore/named.py", line 56, in __init__15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate     self._init_plugins(extensions)15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate   File "/usr/local/lib/python2.7/dist-packages/stevedore/driver.py", line 91, in _init_plugins15:37
merlinguy2014-08-12 15:36:14.694 28255 TRACE designate     (se15:37
merlinguyYeah, I already picked up on the setup.cfg.15:37
richmnext time, please use paste.openstack.org15:37
merlinguysorry, new to irc, not sure what you mean by paste.openstack.org15:38
mugsiemerlinguy: if you can paste the full trace on paste.openstack.org , and give me the link, it looks like you missed on of the important lines ;)15:38
mugsiehttp://paste.openstack.org15:39
timsim_Copy and paste your stack trace there and then copy the link into IRC ;)15:39
merlinguylike this?15:39
merlinguyPaste #9392315:39
merlinguyor15:39
merlinguyhttp://paste.openstack.org/show/93923/15:39
timsim_Yes, that.15:39
mugsiethats the one :)15:39
merlinguyThanks, much appreciated.15:40
richmthanks15:40
*** RaginBajin has joined #openstack-dns15:40
mugsietry re running python setup.py develop in the agent venv again...15:41
merlinguydoes setup.cfg need to get run or does something other than designate-agent need to get restarted?15:41
merlinguy:-)15:41
merlinguytoo fast for me.15:41
mugsie:)15:41
merlinguyAhhhhh! thanks that did it.15:44
merlinguyYou guys rock!15:44
mugsie:D15:44
mugsiehere to help15:44
merlinguyIf you're ever in Ft. Collins, the beers on me.15:44
mugsieI will remember that :) - but I think you may be safe for a while15:46
merlinguyThank again, mugsie, timsim, and richm15:48
*** merlinguy has left #openstack-dns16:10
*** vinod1 has quit IRC16:30
*** spiffxp_ has left #openstack-dns16:31
*** agrebennikov has joined #openstack-dns16:33
*** zain1 has quit IRC17:01
openstackgerritA change was merged to openstack/designate: API v1 for creating domains ignore some attributes now  https://review.openstack.org/10177917:19
*** rmoe has quit IRC17:23
*** rossk has joined #openstack-dns17:39
*** vinod1 has joined #openstack-dns17:43
*** dtx00ff has joined #openstack-dns18:02
*** dtx00ff has quit IRC18:04
*** dtx00ff has joined #openstack-dns18:05
openstackgerritA change was merged to openstack/designate: Reimplement Filtering by Record Data  https://review.openstack.org/10911918:15
*** zain has joined #openstack-dns18:19
*** rmoe has joined #openstack-dns18:23
*** rmoe has quit IRC18:24
*** rmoe has joined #openstack-dns18:29
*** crc32 has joined #openstack-dns18:32
*** crc32 has quit IRC18:33
*** crc32 has joined #openstack-dns18:34
*** agrebennikov has quit IRC18:36
*** vinod1 has quit IRC18:41
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Add two more hacking checks  https://review.openstack.org/11361018:44
*** agrebennikov has joined #openstack-dns18:48
*** vinod1 has joined #openstack-dns18:51
*** vinod1 has quit IRC19:00
*** timsim_ has quit IRC19:01
*** timsim has joined #openstack-dns19:01
*** agrebennikov has quit IRC19:02
*** agrebennikov_ has joined #openstack-dns19:02
*** dtx00ff has quit IRC19:20
*** zain has left #openstack-dns19:25
timsimHey Kiall mugsie I've put an item on tomorrows agenda that I wanted to get out there a little early so you had an idea of what I'm talking about. Here it is summarized: https://wiki.openstack.org/wiki/Designate/MdnsScalability19:30
KiallHeya19:33
*** vinod1 has joined #openstack-dns19:34
Kialltimsim: just finished reading.. Question "A (very) large DNS deployment can have DNS masters that handle upwards of 2000 refresh requests per second"19:35
KiallSo, with a "normal" architecture, you have a small set of masters (Usually just a handful - 1-5), the load of all slaves is concentrated on these small set of server19:36
Kialls19:36
timsimKiall: right. So in our situation we would have many MiniDNS instances, right?19:37
KiallIt sounds like the wiki page is making the assumption that all mDNS instances would receive the same query load, rather than a subset of the query load as you scale them out to have however many you want.19:37
Kiall(The DB issue remains, regardless of how many mDNS you have..)19:38
timsimWell there can be a lot of MiniDNS instances, but they're all reading from the same database?19:38
timsimYes, lol.19:38
Kiall:)19:38
timsimThe MiniDNS load was kind of a secondary concern, I didn't really make that clear. But if we can reduce the load so there doesn't have to be a hundred of them, that's secondarily good, I'd think.19:38
mugsiei just read it there... afaik - option 2 can't be done, and there is no 'master' bind server anymore19:39
KiallSo .. I think we have lots of options for making things more efficient without resorting to MongoDB/Cassandra etc19:39
mugsieMiniDNS is the 'master' bind server as far as the slaves are concerned19:39
Kiallmugsie: Option 2 feels like exactly what we had discussed for AD Integrated zones with Microsoft DNS.19:39
mugsieand mongo will get you shot in OpenStack right now19:39
timsimYeah I don't actually want to do a NoSQL thing but it is an option.19:40
mugsieKiall: yeah - and it is a hack and a half to allow for AD not giving up control, or using AFXR19:40
*** dtx00ff has joined #openstack-dns19:40
mugsieand on reflection, I forsee .... issues ..... with that approch19:40
KiallAnyway... timsim, I expect much of the load is is Nx public facing DNS servers querying the masters for the same zone (1 master, 5 public DNS servers, 1 zone change = 5 SOA queries, 5 AXFRs)19:40
KiallWe could in theory reduce that to 1x with some caching19:41
mugsiefor the refresh requests we can look at caching things like serail as well, can we not19:41
Kiall(e.g. a distributed cache like memcache)19:41
timsimWe could.19:41
timsimI think in most cases, the current architecture would be fine.19:42
mugsieyou would want to really trust you cache poisoning code though ;)19:42
mugsieyour*19:42
Kiallcache invalidation you mean? ;)19:42
mugsieshhhh19:42
mugsieits late19:42
mugsieand I am still jet lagged19:42
timsimBut even if you cut the requests by 80%, having five different places send all those requests to one master database probably woudln't work.19:42
mugsiei think replicating read only slaves is the best bet19:43
mugsieand as they are read only, there should not be a big issue19:43
mugsiebut ^ is based on gut feeling19:43
timsimFor a large deployment, having all refresh requests of many slaves handled by an actual master DNS server is probably best.19:43
mugsiebind doesnt allow it to be a master and a slave concurently19:44
mugsieufortunatly19:44
timsimRight. So that's the "agent" thing we came up wtih.19:44
mugsieso, we would have to have bind using the old backend style...19:44
timsimSort of.19:44
Kiallmugsie: no, it would be the style we discussed for AD integrated MS DNS zones..19:44
timsimReally the agent would be dancing with MiniDNS, and possibly creating/deleting zones.19:45
mugsiei dont think there will be a massive issue with bind, to force us to do that, personally....19:45
Kialltimsim: anyway, as you can probably tell, we've done some thinking along the lines of your Option 2 - But for different reasons.19:45
Kialland both Option 1 (NoSQL) and Option 2 (Agent of sorts) are very doable without needing to change anything we've already discussed (I think?)19:47
timsimKiall: That's the idea. Or, the hope at least.19:47
mugsieexcept the agent is back ;)19:47
timsimIt'd be completely optional though.19:47
Kialltimsim: let's s/agent/proxy/ or someting19:47
Kiall"agent" seems to be a bad word these days ;)19:47
mugsieI think we can have an option 3, keeping the current setup, and have a cache of serial numbers for the current zones that miniDNS reads... I would be interested in getting a test running for it though19:48
Kialleffectively, timsim is suggesting taking bind and turning it into a caching proxy - just where we have to pre-fill the cache, since it aint actually a a proxy19:48
mugsieyeah, but we would be as effective doing that as a non miniDNS interaction, I think19:49
mugsiejust having it go straight out... not sure how that would work though19:49
mugsieit effectivly replaces the AMQP connection with a DNS query19:50
timsimThe cache was the first thing we (internally) talked about. But the issue is again that we'd be putting this cache in a bunch of different places, and it would have to talk to miniDNS, which was going to change the contract (a contract that I really liked)19:50
mugsieyou don't think you can replicate a MySQL DB?19:50
mugsie(vs Casandra etc)19:51
timsimI don't know anything :)19:51
mugsie:)19:51
timsimBut our Ops folks really really don't think so.19:51
timsimAnd keep it in sync and relatively pain free from Dallas to Sydney to Hong Kong and everywhere in between.19:51
*** vinod1 has quit IRC19:51
Kialltimsim: Yea - All valid concerns, and something we should keep in mind.. Once we get betsy's patch for expose NS/SOA fixed up landed, and mDNS updated to use the SOA correctly in AXFRs, we could setup a scale test of all this :)19:53
mugsie+119:53
KiallIs betsy on holiday BTW? Haven't seen her online since I got back19:54
timsimYep, until...Friday I think.19:54
KiallAh.. Ok.. If people can comment on https://review.openstack.org/#/c/112063/ then today (either agreeing or disagreeing with me!) I'll try and make the changes tomorrow so we can land it19:56
Kiall(that's betsy's change)19:56
timsimAbsolutely. The reason I'm bring this up is I've got people breathing down my neck trying to make sure we get the architecture right for our use case. I think we have plenty of time to figure it out, but our folks are really intent on making sure we have a real solid architecture for this massive deployment we're planning on.19:56
Kialltimsim: totally understand... Much better to get it right from the start rather than hack it up later19:57
timsimBut the hope is that we can continue work without really worrying about it, as long as we've got this stuff in mind so that we can plug something killer in later.19:58
timsimSo I think we're great right now.19:58
timsimBut I just wanted to get that out there in the community, as we've been having lots of discussion internally :)19:59
Kialltimsim: what would be really interesting to know is the query profile you guys see against your masters so we know where to focus optimization's...20:00
Kialldon't necessarily need raw #'s ... I can see there being issues releasing those, but %'s of things like ..  % SOA vs AXFR queries, avg AXFR response size, query duplication (e.g. what could have we cached?) etc20:01
KiallWith HP using Akamai, we loose out on seeing these kinda numbers.20:02
*** vinod1 has joined #openstack-dns20:02
timsimKiall: I'll try and get some of those stats.20:02
timsimThe only thing I know for sure is that, when you have a large number of zones. It's easy to have a few thousand SOA queries a second. I don't know how many of those at any given time actually turn into zone transfers.20:08
KiallYea, that's kinda what I expected.. And also (possibly) an easy win for caching20:08
*** agrebennikov_ has quit IRC20:10
*** agrebennikov_ has joined #openstack-dns20:10
timsimProbably. I figured, for this use case, which is exclusively a bind-centric issue, it'd be better to not have to change the way Designate handles all the other backends or add a dependency on a cache (which I thought wasn't a great idea for Openstack-ness)20:10
Kiallcaching isn't non-OpenStack, it's just there are few things in OpenStack that can really take advantage of it.. The only real area that uses heaving caching right now (that I know of) is Swift...20:11
openstackgerritA change was merged to openstack/designate: Updated from global requirements  https://review.openstack.org/11009720:12
KiallAlso - I've personally always assumed we're going to need to do lots of caching in mDNS to make it scale :)20:14
timsimOk. It'd probably be something to add to MiniDNS then.20:14
timsimAh.20:14
Kialljust, as graham said, haven't figured out how invalidation might occur :)20:14
timsimThe buzzword that my folks keep throwing around is "resilience," (they're thinking of all kinds of horrible scenarios that I'd rather not think about) and I *think* that was where they weren't wild about the cache.20:18
Kiallmugsie: re https://review.openstack.org/#/c/108422/220:19
KiallWe should also remove the other policy.init lines now that they are in the baseclass e.g. https://github.com/openstack/designate/blob/master/designate/central/service.py#L8320:19
Kialltimsim: Personally I see resilience and caching as things that go hand in hand.. Scale requires caching (at some value of "scale") otherwise the system will fall over.. :)20:22
Kiall(even with NoSQL :P)20:22
KiallAnyway.. Will talk more tomorrow :)20:23
timsimFair enough. I'm currently in a meeting talking about this, two places at once.20:25
Kiall:D20:25
*** crc32 has quit IRC20:38
*** vinod1 has quit IRC20:41
*** RaginBajin has quit IRC20:43
openstackgerritGraham Hayes proposed a change to openstack/designate: Added policy checking for all all_tenent=True contexts  https://review.openstack.org/10842220:50
mugsieKiall: ^20:51
*** vinod1 has joined #openstack-dns20:59
KiallOne day, VirtualBox will switch to DMKS and not make me recompile the damn kernel modules every other week -_-21:02
Kiallmugsie: looking21:03
*** msisk has joined #openstack-dns21:05
*** vinod1 has quit IRC21:17
*** nkinder has quit IRC21:28
*** msisk has quit IRC21:30
*** msisk has joined #openstack-dns21:32
*** crc32 has joined #openstack-dns21:36
*** timsim has quit IRC21:39
*** eankutse has quit IRC21:46
*** vinod1 has joined #openstack-dns21:54
*** vinod1 has quit IRC21:55
*** vinod1 has joined #openstack-dns21:55
*** msisk has quit IRC22:01
*** msisk has joined #openstack-dns22:05
*** msisk_ has joined #openstack-dns22:07
*** msisk has quit IRC22:09
*** dtx00ff has quit IRC22:29
*** EricGonczer_ has quit IRC22:36
*** msisk_ has quit IRC22:48
*** vinod1 has quit IRC22:54
*** vinod1 has joined #openstack-dns23:07
*** vinod1 has quit IRC23:07
*** vinod1 has joined #openstack-dns23:59
« Monday, 2014-08-11 Index Wednesday, 2014-08-13 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!