Wednesday, 2015-01-28

« Tuesday, 2015-01-27 Index Thursday, 2015-01-29 »
*** jmcbride has joined #openstack-dns00:22
*** crc32 has joined #openstack-dns00:24
*** nkinder has joined #openstack-dns00:46
openstackgerritRon Rickard proposed openstack/designate: WIP: Modifications for Resilient Pool Manager Cache  https://review.openstack.org/14942800:56
*** rmoe has quit IRC01:09
*** penick has quit IRC01:26
*** rmoe has joined #openstack-dns01:30
*** stanzgy has joined #openstack-dns01:35
*** jmcbride has quit IRC01:50
*** jmcbride has joined #openstack-dns02:31
*** richm has quit IRC03:09
*** crc32 has quit IRC03:16
*** MentalRay has joined #openstack-dns03:48
*** MentalRay_ has joined #openstack-dns03:48
*** jmcbride has quit IRC04:01
*** stanzgy has quit IRC04:11
*** stanzgy has joined #openstack-dns04:13
*** MentalRay_ has quit IRC04:45
*** MentalRay has quit IRC04:45
*** MentalRay_ has joined #openstack-dns05:24
*** MentalRay has joined #openstack-dns05:24
*** MentalRay has quit IRC05:27
*** MentalRay_ has quit IRC05:27
*** nihilifer has joined #openstack-dns06:10
openstackgerritKiall Mac Innes proposed openstack/designate: Allow Periodic Sync to be restricted to a subset of zones  https://review.openstack.org/14838606:47
*** rmoe has quit IRC08:11
*** rmoe has joined #openstack-dns08:15
*** MentalRay has joined #openstack-dns08:23
*** MentalRay_ has joined #openstack-dns08:23
*** jordanP has joined #openstack-dns09:09
openstackgerritEndre Karlson proposed openstack/designate: Remove dead server code in storage and add tests  https://review.openstack.org/14768709:15
*** mariusv_ has quit IRC09:41
*** mariusv has joined #openstack-dns09:44
*** mariusv has joined #openstack-dns09:44
*** Kodokuu has joined #openstack-dns09:56
KodokuuHi, i have issue with designate-sink. I launch 9 Vm and i  just 4 records in my domains. In sink log 5 errors :  oslo.messaging.notify.dispatcher [-] Exception during message handling09:59
Kodokuumy pastebin central log : http://pastebin.com/AKpJpsB410:06
*** kodokuu_ has joined #openstack-dns10:09
*** Kodokuu has quit IRC10:11
kodokuu_ekarlso hi10:25
ekarlsokodokuu_: hey10:26
kodokuu_ekarlso i have a little issue with my DB10:29
kodokuu_ekarlso When i launch >5 VM on openstack, designate have 1205 Lock wait timeout exceeded; try restarting transaction10:31
kodokuu_ekarlso and in my processlist i see designate process in sleep.10:31
ekarlsoKiall: got a clue ?10:35
*** MentalRay has quit IRC10:39
*** MentalRay_ has quit IRC10:39
kodokuu_ekarlso I think one request forget to commit10:42
kodokuu_http://pastebin.com/VhzHHi2Y10:42
kodokuu_all request stay sleep10:42
Kiallkodokuu_:  eanderson hit this issue a little while back, he backported a change to fix this exact issue up - but I don't think it was pushed up10:45
KiallHe usually pops online later in the day, I'll see if he can push it up then...10:45
kodokuu_Kiall ok great, i wait eanderson :)10:46
*** openstackgerrit has quit IRC10:50
*** openstackgerrit has joined #openstack-dns10:50
*** ChanServ sets mode: +v openstackgerrit10:50
*** stanzgy has quit IRC10:56
*** untriaged-bot has joined #openstack-dns11:03
untriaged-botUntriaged bugs so far:11:03
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141243111:03
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141302411:03
untriaged-bothttps://bugs.launchpad.net/designate/+bug/128944411:03
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141380611:03
*** untriaged-bot has quit IRC11:03
*** kodokuu_ has quit IRC11:26
*** MentalRay has joined #openstack-dns12:02
*** MentalRay_ has joined #openstack-dns12:05
*** Kodokuu has joined #openstack-dns12:16
*** timbyr_ has quit IRC12:17
*** MentalRay_ has quit IRC12:19
*** MentalRay has quit IRC12:19
*** timbyr_ has joined #openstack-dns12:20
*** MentalRay has joined #openstack-dns12:20
*** MentalRay_ has joined #openstack-dns12:20
*** Kodokuu has quit IRC12:41
*** redixin has joined #openstack-dns12:58
*** richm has joined #openstack-dns14:15
*** nihilifer has quit IRC14:29
*** jmcbride has joined #openstack-dns14:39
*** jmcbride has quit IRC14:45
*** jmcbride has joined #openstack-dns14:45
*** jmcbride1 has joined #openstack-dns15:02
*** jmcbride1 has quit IRC15:03
*** jmcbride1 has joined #openstack-dns15:03
*** jmcbride has quit IRC15:04
*** timsim has joined #openstack-dns15:09
*** kodokuu has joined #openstack-dns15:33
rjrjranyone on?15:41
KiallKinda.. Eating lunch then spending the rest of the day till our weekly meet on calls ;)15:43
openstackgerritRon Rickard proposed openstack/designate: WIP: Modifications for Resilient Pool Manager Cache  https://review.openstack.org/14942815:44
rjrjrokay.  just wanted to let everyone know i ran into a roadblock.  trying to figure out how to get status for create and delete.  polling for SOA isn't doing it.15:45
rjrjrthis is where we get the status when the cache doesn't have the status.15:46
KiallHumm.. So - Ignoring PM for a minute, what methods of checking do we have for Bind / PowerDNS etc?15:49
rjrjri'm thinking it can be in the poll for SOA code, but I want to be able to differentiate between a failure because of no domain versus a failure because the backend is down, etc.15:52
rjrjrright now, mdns responds with the same error regardless of why there was a failure by design.15:52
rjrjrtrying to work through that now.  just wanted everyone to know.15:53
Kiallrjrjr: humm..15:58
KiallYea, makes sense.. We can probbaly have mDNS give move info15:59
rjrjrone other thing, i am passing around a 'force_check' argument now.  this is so we know when in the code to force the check of the status when it does not exist in the cache and when not too.  i had to pass this to mdns too so it can pass it back.16:03
rjrjri wouldn't mind other eyes letting me know if there is a better way to do that.  i have the current code checked in so others can look.16:03
rjrjrso, 'force_check' is False when a domain is created, deleted, or updated.  'force_check' is True when periodic_sync/periodic_recovery is run.  it is what determines if the status should be determined regardless of cache.16:04
*** jmcbride1 has quit IRC16:08
*** jmcbride has joined #openstack-dns16:08
*** jmcbride has joined #openstack-dns16:09
kodokuuKiall no news from eandersson ? :p16:19
KiallNot yet - haven't seen him drop on IRC today...16:19
kodokuuok, so it's for tomorrow16:20
kodokuuKiall just one question. now I use "Stable Juno." Do you think that master branch can be used in production?16:21
KiallNo, it's going through lots of changes right now - enough that I wouldn't say to use it in production yet :)16:27
*** jmcbride has quit IRC16:33
*** paul_glass has joined #openstack-dns16:35
timsimrjrjr: I would think mdns' poll_for_serial would need to change to return something different when it gets an NXDOMAIN and a timeout. That way you could do the right thing.16:35
*** jmcbride has joined #openstack-dns16:37
*** penick has joined #openstack-dns16:39
*** penick_ has joined #openstack-dns16:42
*** penick has quit IRC16:44
*** penick_ is now known as penick16:44
rjrjrtimsim: yeah.  i'm now trying to find out why 'designate pool-manager-cache sync' is not working.   trying to get to that code.16:46
*** vinod has joined #openstack-dns16:53
*** rjrjr has quit IRC16:55
*** rjrjr has joined #openstack-dns16:55
*** jmcbride has quit IRC16:57
*** untriaged-bot has joined #openstack-dns17:03
untriaged-botUntriaged bugs so far:17:03
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141243117:03
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141302417:03
untriaged-bothttps://bugs.launchpad.net/designate/+bug/128944417:03
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141380617:03
untriaged-bothttps://bugs.launchpad.net/python-designateclient/+bug/141556017:03
*** untriaged-bot has quit IRC17:03
*** jmcbride has joined #openstack-dns17:07
*** jmcbride has quit IRC17:07
*** jmcbride has joined #openstack-dns17:07
*** jmcbride has quit IRC17:16
*** jmcbride has joined #openstack-dns17:16
*** jmcbride has quit IRC17:18
*** jmcbride has joined #openstack-dns17:18
*** rmoe has quit IRC17:21
*** kodokuu has quit IRC17:24
*** rmoe has joined #openstack-dns17:37
*** jmcbride has quit IRC17:59
*** jordanP has quit IRC18:00
*** openstackgerrit has quit IRC18:14
*** openstackgerrit has joined #openstack-dns18:14
*** ChanServ sets mode: +v openstackgerrit18:14
openstackgerritVinod Mangalpally proposed openstack/designate: Add a new API for abandoning a zone from storage  https://review.openstack.org/14877919:21
openstackgerritVinod Mangalpally proposed openstack/designate: Add a new API for abandoning a zone from storage  https://review.openstack.org/14877919:24
*** jmcbride has joined #openstack-dns19:29
openstackgerritVinod Mangalpally proposed openstack/designate: Add a new API for abandoning a zone from storage  https://review.openstack.org/14877919:39
*** paul_glass has quit IRC20:03
*** paul_glass has joined #openstack-dns20:05
*** jmcbride has quit IRC20:06
*** jmcbride has joined #openstack-dns20:10
*** jmcbride has quit IRC20:15
*** jmcbride has joined #openstack-dns20:15
*** rjrjr_ has joined #openstack-dns21:03
*** rjrjr_ has joined #openstack-dns21:03
*** jmcbride has quit IRC21:04
*** vinod has quit IRC21:15
*** jmcbride has joined #openstack-dns21:31
*** vinod1 has joined #openstack-dns21:34
rjrjr_timsim: for some reason, mdns is not getting NXDOMAIN, but NOERROR21:34
timsimEven when the domain doesn't exist?21:35
timsimNOERROR comes when everything is fine.21:35
rjrjr_and this is for a domain that i get a NXDOMAIN error with a dig 'dig @localhost example.test. soa21:35
timsimwat21:35
rjrjr_it makes no sense to me.21:35
timsimIs the make_and_send_dns_message catching it or something?21:36
rjrjr_;; ->>HEADER<<- opcode: QUERY, status: NXDOMAIN, id: 953821:36
rjrjr_in the dig21:36
rjrjr_rcode NOERROR21:36
rjrjr_in mdns21:36
rjrjr_nearest i can tell, mdns is just spitting out the response it gets.21:37
timsimWell...the domain has to be there in some way then.21:37
rjrjr_originally, i was testing with 'example.com.' and noticed that is a real domain and got a real response with dig.  so i moved to using example.test. but i get the same thing, except dig is showing the response correctly.21:37
timsimTry poo.com or something like that.21:38
rjrjr_it's not.21:38
rjrjr_most of the .com's i tried are real.  poo.com is a real domain.21:39
rjrjr_not kidding.21:39
rjrjr_so i decided to use the .test TLD since it is the recommended TLD for testing.21:39
timsimWell yeah. I own it! (jk, I wish) But if MiniDNS is pointed at a DNS server to tango with that doesn't have poo you should be good?21:39
timsimUnless it's recursing.21:39
rjrjr_i'm recursing.21:39
rjrjr_trust me, example.test. does not exist in any way on my server and it cannot resolve it.21:40
timsimWell then it'll probably go and get the domain that exists IRL. You don't want to do that.21:40
rjrjr_strange, the last two mdns calls are now showing21:41
rjrjr_rcode NXDOMAIN21:41
rjrjr_hmmmm....21:41
timsimJust randomly started working? lol21:41
rjrjr_well, after a few minutes have transpired.21:42
rjrjr_that IS strange.21:42
rjrjr_no nscd or anything running locally that would cache the domain.21:42
* timsim shrugs21:43
rjrjr_i created the domain using designate.  removed the domain by removing the files and cycled bind9.  the first few attempts afterwards showed NOERROR, not NXDOMAIN.21:43
rjrjr_i'll have to play here.21:43
rjrjr_never mind.21:43
rjrjr_i wonder if the DNS library we are using has a cache?21:44
rjrjr_i believe it's java counterpart does...21:44
*** stanzgy has joined #openstack-dns21:45
timsimNah. Bind must have still had it in memory somehow. My goto for cleaning out Bind9 is "rm 3bf305731dd26307.nzf && touch 3bf305731dd26307.nzf && chown bind:bind 3bf305731dd26307.nzf && service bind9 restart" crude but effective21:45
rjrjr_i did rm *.nzf slave* and 'service bind9 restart'21:46
rjrjr_i'll play.  let you know what i find.21:46
rjrjr_rcode NOERROR21:51
rjrjr_it is still doing it.  tried a different domain, test.test.  there is definitely something strange going on here.21:51
timsimProbably not MiniDNS' fault though?21:51
rjrjr_server was cleaned with 'rm /var/cache/bind/*nzf /var/cache/bind/slave*' and 'service bind9 restart'.21:52
rjrjr_nothing in the DNS logs makes me believe test.test. exists at all.21:52
rjrjr_but mdns is giving me an rcode NOERROR21:52
*** jmcbride has joined #openstack-dns21:52
rjrjr_i'll bet in a few sync attempts it will show NXDOMAIN like before too.21:54
timsimPut "recursion no;" in your bind9 conf and see. I'm pretty convinced it's not mdns doing something weird.21:54
rjrjr_timsim: i turned off recursive.  same issue.21:55
rjrjr_got NOERROR again.21:56
timsimI don't know, maybe it's staying in memory for some reason. Caching the bad result?21:56
rjrjr_28-Jan-2015 14:54:10.531 notice: client 127.0.0.1#33884: received notify for zone 'test.test': not authoritative21:57
rjrjr_the NOTIFY is showing it does not exist.21:57
rjrjr_28-Jan-2015 14:47:05.783 info: client 127.0.0.1#59932: received notify for zone 'test.test'21:57
rjrjr_when it did exist, that is what i got.21:57
*** stanzgy has quit IRC22:00
timsimSo it was in there somewhere I suppose.22:00
rjrjr_DNS python does have a cache.22:00
timsimIt looks like it has a cache you can use if you're using it's stub resolver, but we're just sending DNS messages.22:05
rjrjr_auth-nxdomain no;    # conform to RFC103522:10
rjrjr_interesting...22:10
rjrjr_that is set in the BIND9 configuration.  it is being set via devstack scripts.22:11
rjrjr_i wonder if that is the issue?22:11
rjrjr_it took 15 minutes, but I am now seeing NXDOMAIN in mdns.22:11
timsimIf auth-nxdomain is 'yes' allows the server to answer authoritatively (the AA bit is set) when returning NXDOMAIN (domain does not exist) answers, if 'no' (the default) the server will not answer authoritatively.22:12
rjrjr_yes.  just trying to figure out why this is happening.  it makes no sense.22:13
*** vinod1 has quit IRC22:13
rjrjr_no sssd or nscd running.22:13
rjrjr_are there any other DNS caching services i should be looking for?22:13
timsimMaybe paste your bind config?22:14
rjrjr_http://paste.openstack.org/show/163383/22:16
timsimI guess try flipping auth-nxdomain.22:17
rjrjr_yeah, trying that now.  just a few minutes.22:17
rjrjr_it is strange though host, dig, nslookup all are correct, but mdns is not.22:18
rjrjr_if it was the DNS server, i wouldn't think this was the case.22:18
timsimMdns wants an authoritative answer...I know that. Other than that I don't really know. It's probably just doing a different query than those other tools are. you could try with the query dnspython is making.22:20
rjrjr_okay, saw NXDOMAIN on the first attempt.22:26
rjrjr_trying again with this setting change.22:26
rjrjr_do we know why this is being set in devstack via the Designate bind9 plugin?22:27
rjrjr_this is = auth-nxdomain no;22:27
timsimApparently to conform to RFC 1035. "When processing queries with QCLASS=*, or some other QCLASS which22:28
timsimmatches multiple classes, the response should never be authoritative22:28
timsimunless the server can guarantee that the response covers all classes." is all I can see22:28
rjrjr_i'm also seeing the AA flag which is expected22:28
*** jmcbride has quit IRC22:30
*** jmcbride has joined #openstack-dns22:31
rjrjr_i'll go out on a limb here and say we don't need auth-nxdomain no; set.  i'll change this to auth-nxdomain yes;22:31
timsimCool22:32
*** MentalRay has quit IRC22:32
*** MentalRay_ has quit IRC22:32
rjrjr_still broken.  that is not the issue.22:36
rjrjr_first attempt via MDNS shows NOERROR again. <sigh>22:36
timsimI'd just try and move on if I were you.22:38
rjrjr_well, if someone has time to help me diagnose this issue that would be great.22:38
rjrjr_current code is checked it.  to setup the situation, create a domain via designate.22:39
rjrjr_the remove the domain from the DNS server.  i'm using BIND.22:39
rjrjr_when periodic_sync runs, you'll see the issue.22:39
rjrjr_after some time, we get the correct NXDOMAIN response.22:40
*** jmcbride has quit IRC22:40
*** rjrjr_ has quit IRC22:54
timsimI see something even more fun: http://paste.openstack.org/show/163389/22:57
*** jmcbride has joined #openstack-dns23:02
timsimAlright that was weird and went away23:03
timsimI get http://paste.openstack.org/show/163390/23:03
timsimRefused instead of NXDOMAIN23:03
*** openstackgerrit has quit IRC23:06
*** openstackgerrit has joined #openstack-dns23:06
*** ChanServ sets mode: +v openstackgerrit23:06
timsimInternet makes me think REFUSED is default in bind > 9.7 instead of NXDOMAIN.23:13
timsimBut either way rjrjr, I seem to get the correct result.23:16
*** paul_glass has quit IRC23:18
*** jmcbride has quit IRC23:24
*** timsim has quit IRC23:30
« Tuesday, 2015-01-27 Index Thursday, 2015-01-29 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!