Thursday, 2015-04-09

*** chlong has joined #openstack-dns00:25
*** MentalRay_ has joined #openstack-dns00:40
*** MentalRay has quit IRC00:43
*** km has quit IRC00:53
*** km has joined #openstack-dns00:53
*** kei_yama has quit IRC00:53
*** kei_yama_ has joined #openstack-dns00:53
*** crc32 has quit IRC01:14
*** stanzgy has joined #openstack-dns01:35
*** ericpeterson has joined #openstack-dns01:40
*** ericpeterson has quit IRC01:45
*** MentalRay_ has quit IRC01:45
*** jmcbride has quit IRC02:17
*** kei_yama_ has quit IRC02:29
*** km_ has joined #openstack-dns02:31
*** kei_yama has joined #openstack-dns02:32
*** km has quit IRC02:33
*** andrewbogott_afk is now known as andrewbogott02:46
*** richm has quit IRC02:52
*** EricGonczer_ has joined #openstack-dns03:03
*** EricGonczer_ has quit IRC03:19
*** andrewbogott is now known as andrewbogott_afk03:58
*** MentalRay has joined #openstack-dns04:23
*** raginbajin has quit IRC04:32
*** ericpeterson has joined #openstack-dns04:34
*** raginbajin has joined #openstack-dns04:35
*** ericpeterson has quit IRC04:37
*** ericpeterson has joined #openstack-dns04:38
*** ericpeterson has quit IRC04:52
*** chlong has quit IRC05:09
*** ToMiles has quit IRC05:22
*** ToMiles has joined #openstack-dns05:24
*** ericpeterson has joined #openstack-dns05:52
*** ericpeterson has quit IRC05:57
openstackgerritOpenStack Proposal Bot proposed openstack/designate: Imported Translations from Transifex  https://review.openstack.org/17193907:03
*** jordanP has joined #openstack-dns07:07
*** MentalRay has quit IRC07:30
*** ToMiles has quit IRC08:39
*** boris-42 has joined #openstack-dns09:16
*** ericpeterson has joined #openstack-dns09:25
*** ericpeterson has quit IRC09:29
*** untriaged-bot has joined #openstack-dns10:01
untriaged-botUntriaged bugs so far:10:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/143769910:01
openstackLaunchpad bug 1437699 in Designate "mDNS should Handle Bad File Descriptor" [Medium,New] - Assigned to Ron Rickard (rjrjr)10:01
*** untriaged-bot has quit IRC10:01
*** eandersson has joined #openstack-dns10:14
eandersson[17:33] <kodoku> ok so I can't use pdns on rhel 7, so go use bind10:15
eanderssonAny specific reason why pdns wont run on rhel7?10:15
ahufrom what we hear it works pretty well10:16
eanderssonI got a bit scared there for a moment :P10:24
ekarlsoi'd just use nsd4 rly...10:24
ekarlsofast and easy to setup10:24
ekarlso:P10:24
nihiliferhello10:34
nihiliferI have one question about this revert https://github.com/openstack/designate/commit/37c3952d5aa02948e050785e11957b2da72ccd0b10:34
nihiliferI see that support of sink notification handlers in devstack caused CI failures10:35
nihiliferis there any plan to revisit this support?10:35
nihiliferit would be nice to have a possibility to test sink without manual config ;)10:36
Kiallnihilifer: I actually have a review up to ad dit10:37
Kialladd it*10:37
KiallStill some quirks tho10:38
Kiallhttps://review.openstack.org/#/c/170678/10:38
nihiliferKiall: ok, that's great, thanks for information10:40
openstackgerritGraham Hayes proposed openstack/designate: Restrict editing of managed records to policy based ACL  https://review.openstack.org/17162110:40
openstackgerritMerged openstack/designate: Imported Translations from Transifex  https://review.openstack.org/17193910:49
*** stanzgy has quit IRC10:58
*** redixin has joined #openstack-dns10:59
*** ericpeterson has joined #openstack-dns11:11
*** ericpeterson has quit IRC11:15
*** ericpeterson has joined #openstack-dns11:54
*** kei_yama has quit IRC12:03
*** ericpeterson has quit IRC12:18
*** km_ has quit IRC12:31
*** kodoku has joined #openstack-dns12:55
kodokuHi Kiall ekarlson eandersson, I have a big question.12:56
kodokuI have juno designate on Rhel 7 and I use pdns12:56
kodokubut pdns on rhel 7 use new schema for database*12:57
kodokuso How I can use pdns and juno designate on rhel 7 ?12:58
*** ericpeterson has joined #openstack-dns13:03
Kiallkodoku: any idea what the schema difference is?13:03
KiallDesignate adds an extra column or two to track things, which is why we have migrations for the pDNS database, but, if there's something missing, we can likely add it. From memory, pDNS added a "disabled" column it needs recently, but I'm not familar with any other recent changes.. and I'm guessing rhel7 doesn't included the latest pDNS with that column.13:05
Kiallahu: Q actually, any reason pDNS doesn't provide DB migrations of some kind?13:05
Kiallbrb13:06
*** ericpeterson has quit IRC13:07
ahuhttps://doc.powerdns.com/md/authoritative/upgrading/ is what we provide13:07
mugsieah13:09
mugsiekodoku: if you run the designate migration, and then run the sql from https://doc.powerdns.com/md/authoritative/upgrading/ it should work13:09
*** mwagner_lap has quit IRC13:12
*** ericpeterson has joined #openstack-dns13:15
KiallRight.. Our pDNS migrations are probably WAYY off them.. Heh.13:17
KiallI've not noticed the SQL upgrade scripts before!13:17
ahuyou are not the first to miss them, so perhaps we should improve there13:18
ahuone thing we'd love to have is a schema version in the database13:18
KiallYea, we keep that in the DB.. and you just "designate-manage database sync", and finds the current version, and runs each of the upgrade scripts13:20
KiallFar from ideal, but makes it simple.. Upgraded designate? Run `designate-manage database sync` and your DB will be updated, no matter what version it's currently on.13:21
KiallAnyway - Gotta run, back in an hour.13:21
*** richm has joined #openstack-dns13:23
*** ericpeterson has quit IRC13:26
*** ericpeterson has joined #openstack-dns13:27
*** ericpeterson has quit IRC13:35
*** ericpeterson has joined #openstack-dns13:36
*** ericpeterson has quit IRC13:36
*** kodoku_ has joined #openstack-dns13:38
kodoku_Kiall re13:39
kodoku_So yes inded I have error when I start pdns like "disabled" column13:39
kodoku_even if I exec manage powerdns sync13:39
*** kodoku has quit IRC13:40
*** johnbelamaric has joined #openstack-dns13:40
*** jmcbride has joined #openstack-dns13:42
*** ericpeterson has joined #openstack-dns13:42
kodoku_mugsie thx, how you launch designate migration ?13:47
mugsiethe "designate-manage powerdns sync" does it13:48
kodoku_ok13:49
mugsieso, now that you ran that, run the sql snippets from the page ahu linked13:49
kodoku_ok13:49
mugsiehttps://doc.powerdns.com/md/authoritative/upgrading/13:49
kodoku_yes13:49
kodoku_on this page many script13:49
mugsieand it should add in the columns13:49
mugsieyeah - ahu would have a much better idea of which one to run13:49
mugsieahu: any input on ^ ?13:49
kodoku_ok13:50
*** ericpeterson has quit IRC13:50
kodoku_because I see create table and I think designate create table when migration13:50
*** openstackgerrit has quit IRC13:53
kodoku_ahu If I'am disconnect, write the answer. I'll see irc logs ;)13:53
*** openstackgerrit has joined #openstack-dns13:53
*** ChanServ sets mode: +v openstackgerrit13:53
*** EricGonczer_ has joined #openstack-dns13:56
*** mwagner_lap has joined #openstack-dns13:58
kodoku_mugsie Is it possible to refresh powerdns database if I already have domain and records in designate database and I recreate pdns database13:58
kodoku_because after powerdns sync, I start pdns and If I restart "powerdns sync" ==> Trace: ValueError: Tables "cryptokeys,domainmetadata,domains,records,supermasters,tsigkeys" have non utf8 collation14:00
kodoku_and on powerdns logs ==> Unable to launch gmysql connection: Unable to connect to database: Unknown MySQL server host14:01
*** vinod1 has joined #openstack-dns14:05
*** paul_glass has joined #openstack-dns14:10
*** jmcbride has quit IRC14:30
*** andrewbogott_afk is now known as andrewbogott14:30
vinod1kiall: re: https://review.openstack.org/#/c/171621/14:31
vinod1it looks like a zone create is not done correctly14:32
vinod1the pool manager creates the zone but the mdns does not get the correct serial number from the server even after several retries14:33
vinod1i wonder if the zone was not created correctly but no error was still indicated and the pool manager thinks the zone is created when in reality it might not have been created14:34
*** csoukup has joined #openstack-dns14:35
mugsievinod1: it looks like some AFXRs can fail sometimes14:36
mugsieand we dont have a high enough timeouts / fast enough refreshs in the tests14:36
vinod1the tests wait for 20 seconds and we do multiple retries in mdns over the course of the 20 seconds14:38
mugsievinod1: we dont it seems :(14:39
mugsieonly one notify is sent14:39
*** MentalRay has joined #openstack-dns14:46
*** andrewbogott is now known as andrewbogott_afk14:55
*** andrewbogott_afk is now known as andrewbogott14:56
*** jmcbride has joined #openstack-dns14:59
kodoku_ping  Kilo version of designate works with juno openstack version ?14:59
mugsiekodoku_: it should do OK15:01
kodoku_ok :)15:04
kodoku_because I see commit fix my issue15:04
kodoku_so I'll try with kilo15:04
*** jmcbride has quit IRC15:05
*** andrewbogott is now known as andrewbogott_afk15:09
*** andrewbogott_afk is now known as andrewbogott15:09
*** andrewbogott is now known as andrewbogott_afk15:10
*** Kiall has quit IRC15:10
*** andrewbogott_afk is now known as andrewbogott15:10
*** Kiall has joined #openstack-dns15:12
eanderssonkodoku_: Upgrading to Kilo for the same reason.15:14
Kiallvinod1: It's this - https://deepthought.isc.org/article/AA-01213/0/What-causes-refresh%3A-failure-trying-master-...%3A-operation-canceled-error-messages.html15:14
Kialla non-determinitic kernel bug.. cuz.. yea, that sounds like fun to fix -_-15:15
KiallWe can work around it by lowering the periodic-sync interval in the gate15:15
KiallThat way, a second notify will be sent within the period and hopefully the kernel doesn't splode twice15:15
Kiall(the bind logs from here gave me the clue .. http://logs.openstack.org/21/171621/5/gate/gate-designate-dsvm-bind9/4bca735/logs/syslog.txt.gz  )15:16
eanderssonKilo is looking great so far though15:17
eanderssonA lot of progress since Juno.15:18
openstackgerritVinod Mangalpally proposed openstack/designate: Update error status on a NO_DOMAIN error  https://review.openstack.org/17211115:19
vinod1kiall:  ^^^ How about this fix for the gate failures?15:19
Kiallvinod1: Based only on the commit messagem, I think that will result in all nearly zones going to ERROR briefly with e.g. PowerDNS which takes a few seconds before it notices the new zone, and 100% of zones with backends like Akamai and Dyn15:20
*** andrewbogott is now known as andrewbogott_afk15:21
vinod1hmm. if a zone is not created for some reason, but pool manager thinks it is (i.e. the backend does not report any error), we would not recover from that error15:24
openstackgerritThierry Carrez proposed openstack/designate: Open Liberty development  https://review.openstack.org/17211615:26
Kiallvinod1: Humm, PowerDNS/Akamai/Dyn etc - I don't believe that can happen, does rncd addzone sometimes "suceed" but actually fail?15:26
KiallOkay - Thierry wants us to hold RC1 until we make the call on APIv2 stable or not.. So we'll need to make that call ASAP15:31
vinod1i haven't seen that failure but wondering of that scenario15:31
KiallI'll push a review to make the change, and we +/- it on gerrit..15:31
Kiallvinod1: we could have the periodic_sync attempt to correct zones where the serial comes back as None from 1 or more nameservers?15:32
KiallI guess the issue is that you might have 2 targets, and 6 nameservers which slave from both targets... If one target fails, all nameservers will get data.15:33
Kiallyou can't "see into" the target in many cases15:33
vinod1that is a possibility - we can come back to this in liberty15:36
openstackgerritMerged openstack/designate: Restrict editing of managed records to policy based ACL  https://review.openstack.org/17162115:36
Kiallvinod1: yea, agreed :)15:36
*** ericpeterson has joined #openstack-dns15:38
*** kodoku_ has quit IRC15:38
*** ericpeterson has quit IRC15:38
openstackgerritKiall Mac Innes proposed openstack/designate: Add stable/kilo placeholder migrations  https://review.openstack.org/17212415:41
Kiallvinod1 / timsim / mugsie: ^ can be approved, it;'s dependant on the Open Liberty development review, so won't merge until we open for L15:41
openstackgerritKiall Mac Innes proposed openstack/designate: DevStack: Reduce periodic intervals in the gate  https://review.openstack.org/17213115:55
Kiallvinod1 / mugsie: In thory, ^ get's us past  https://deepthought.isc.org/article/AA-01213/0/What-causes-refresh%3A-failure-trying-master-...%3A-operation-canceled-error-messages.html15:56
*** andrewbogott_afk is now known as andrewbogott15:59
*** ericpeterson has joined #openstack-dns16:00
Kiall(It's also untested ;))16:00
*** ericpeterson has quit IRC16:01
*** untriaged-bot has joined #openstack-dns16:01
untriaged-botUntriaged bugs so far:16:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144222116:01
openstackLaunchpad bug 1442221 in Designate "Pool manager does not update domain create errors correctly" [Undecided,In progress] - Assigned to Vinod Mangalpally (vinod-mang)16:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/144214116:01
openstackLaunchpad bug 1442141 in Designate "designate-api Throws Error on stop()" [Undecided,New]16:01
untriaged-bothttps://bugs.launchpad.net/designate/+bug/143769916:01
openstackLaunchpad bug 1437699 in Designate "mDNS should Handle Bad File Descriptor" [Medium,New] - Assigned to Ron Rickard (rjrjr)16:01
*** untriaged-bot has quit IRC16:01
*** johnbelamaric has quit IRC16:07
*** johnbelamaric has joined #openstack-dns16:08
*** johnbelamaric has quit IRC16:09
paul_glassI'm trying out importing some large-ish zone files into designate... I imported a zone with 601 records and get a 504 connection timed out from the API, although it looks like all of the records get to the nameserver16:15
*** redixin has quit IRC16:19
Kiallpaul_glass: From what I remember, the zone import code doesn't work amazingly with bigger zone files - I can't remember the specific tho, let me check16:20
mugsieyeah, ron had to bump up timeouts all over the place to get it to work16:20
KiallActually, that code was reworked a few weeks or a month or something back..16:20
KiallI thought that would have sorted it :/16:20
mugsieit still does the "read records, do central.create(), read next rcords afaik"16:21
Kiallvinod1: "Kiall, how about adding some spare migrations into pool manager cache too?" .. lol yes, and pdns.. heh, nice catch ;)16:21
*** ericpeterson has joined #openstack-dns16:22
Kiallmugsie: I don't think it does anymore.. checking116:22
Kiallpretty sure ekarlso sorted that during a refactor for Seconday zone support.16:22
KiallYea, looks like he did..16:23
Kiallzone = self.central_api.create_domain(context, domain) <-- That's the only call in the import zone controller16:23
Kiallpaul_glass: so, we're doing daft things there too -_- We end up issuing lots of singular inserts, rather than a bulk instert.16:26
KiallThe whole thing is wrapped in a single transaction, so we get all or nothing, but it's likely hitting a delay somewhere (or the DB is busy etc) and it's hitting the RPC timeout, which will trigger a 504 once hit.16:27
paul_glassKiall: is that rpc timeout controlled by `rpc_response_timeout` in designate.conf16:29
KiallCorrect, it us16:29
Kiallis*16:29
KiallAlso - If you have something else in front of Designate - e.g. haproxy, it can trigger a 504 too with it's own set of timeouts..16:29
Kiall(Just in case your hitting that, rather than anything else)16:30
paul_glassokay. yeah, I was seeing a MessagingTimeout exception in designate logs and set rpc_response_timeout = 600 and I that exception seems to have gone away.16:30
KiallIs this a all-in-one VM or similar? I'm wondering how ~600 inserts could take so long?16:30
paul_glassit's not particularly powerful... the api/central/rabbit/mysql are all on the same server.16:31
KiallMy dev env is down right now.. Will build it back up and do some experimentation16:32
paul_glassoh, I do get back some html with the generic "connection timed out" message in the 504 response body.16:32
paul_glasswhich looks like it's not from designate.16:32
KiallAh, if it's not a Designate style JSON error, chances are it's something timing out before we do?16:33
paul_glassbut I'm just hitting the api directly, no weirdo proxies. except being on the internal rax network.16:33
KiallAt HP Cloud, we set out RPC timeout 10 seconds less than our LB timeouts, ensuring it's always Designate that decides something took too long.16:33
Kiallpaul_glass: http proxy?16:33
Kiallbrb16:34
paul_glassno proxy in this env. I'll try doing the import zone from the same box to see if that 504 goes away.16:35
KiallHow are you doing the API call? curl?16:41
paul_glasspython requests16:41
KiallThat likely has a timeout setting somewhere too, it could be generated the 504 "response"16:42
paul_glassthat thought crossed my mind.16:42
*** csoukup has quit IRC16:42
KiallSeems their docs don't mention the default, but it seems it's not "wait forever"16:44
Kiallso it's gotta be set to something ..16:44
KiallYou can try adding requests.post(http://bla, timeout=None) or whatever to the call, and see if/when it times out.. can't seem to find the default value16:47
Kiallpaul_glass: also .. you tempest tests have .. umm.. identified a kernel bug that affects us. LOL16:47
paul_glassthis is good and bad.16:47
paul_glasslol16:47
KiallYea, just .. Heh16:47
Kiallhttps://deepthought.isc.org/article/AA-01213/0/What-causes-refresh%3A-failure-trying-master-...%3A-operation-canceled-error-messages.html has the details ;)16:48
KiallNot sure if it's only BIND it hurts, but they claim it's an as yet unknown issue somewhere in the netfilter kernel moduke16:48
Kiallmodule*16:48
openstackgerritKiall Mac Innes proposed openstack/designate: DevStack: Reduce periodic intervals in the gate  https://review.openstack.org/17213116:51
openstackgerritKiall Mac Innes proposed openstack/designate: DevStack: Reduce periodic intervals in the gate  https://review.openstack.org/17213116:51
Kiallif pep8 gate fails me for the line length in that URL, I'm gonna shoot it. That's a URL, I can't split it and keep it clickable ;)16:52
paul_glassdoes running tox -e pep8 locally check the commit message?16:53
KiallAfter you commit yea, but I was lazy and edited it online via Gerrit ;)16:54
paul_glassthat 504 seems to go away when running the zone import request from the same box as the api.16:57
paul_glassit takes 87 seconds for the 202 response to get back though.16:57
paul_glassnot as much of a problem on a more powerful environment.17:00
*** openstack has quit IRC17:13
*** openstack has joined #openstack-dns17:13
*** ChanServ sets mode: +v openstack17:13
*** jordanP has quit IRC17:24
*** boris-42 has quit IRC17:28
*** andrewbogott is now known as andrewbogott_afk17:28
*** MentalRay has quit IRC17:32
*** MentalRay has joined #openstack-dns17:37
*** andrewbogott_afk is now known as andrewbogott17:47
*** MentalRay has quit IRC18:07
Kiallpaul_glass: so what is the env? a VM on RAX or?18:12
KiallStill seems way too slow, but I've been in meetings so haven't got my dev env back up18:13
paul_glassit's a 2 GB rackspace cloud server and a few other smaller servers. api/central/rabbit/mysql are on one node. mdns, pool manager, nameservers are each on their own nodes.18:15
paul_glassit's severely underpowered compared to what we'll have in production.18:16
Kiallmugsie / timsim / vinod1 - https://review.openstack.org/#/c/172131/ hit the "operation cancelled" bind issue again, and passed with the changes.. Should be good :)18:31
Kiallpaul_glass: Cool, I'll test it out tomorrow as well.. It still feels too long, even if underpowered!18:31
timsimKernel bug....wat18:32
paul_glassit's slower than I was hoping for... but if designate isn't batching the record(set) creates for all the records in the zone file, that's 600 or so sql inserts18:33
paul_glasson pretty weak mysql instance.18:33
Kialltimsim: yea, I know.18:34
vinod1kiall: do you want https://review.openstack.org/#/c/172131/ to be kilo rc1?18:34
Kiallvinod1: yea, the tempst tests gates fail randomly due to the kernel bug.. So it should land in the stable branch (which will cut from master)18:35
Kiall(Actually, I'm not sure I explained that very well in the commit message.. heh)18:35
vinod1+A'ed18:37
Kiallcool18:41
*** MentalRay has joined #openstack-dns18:43
openstackgerritKiall Mac Innes proposed openstack/designate: Add stable/kilo placeholder migrations  https://review.openstack.org/17212418:53
*** eandersson has quit IRC18:54
Kiallvinod1: you still had some concerns re calling APIv2 stable, I plan to put that first on the HP DNS team meetup next week.. So we'll spend a few hours Monday reviewing, it'd be excellent to close that out and cut rc1 monday if you can review before they too :)18:59
*** ericpete_ has joined #openstack-dns19:00
*** ericpeterson has quit IRC19:03
openstackgerritMerged openstack/designate: DevStack: Reduce periodic intervals in the gate  https://review.openstack.org/17213119:11
*** MentalRay has quit IRC19:13
*** MentalRay has joined #openstack-dns19:23
*** ericpete_ has quit IRC19:37
*** ericpeterson has joined #openstack-dns19:37
*** mwagner_lap has quit IRC20:22
*** johnbelamaric has joined #openstack-dns20:51
*** johnbelamaric has quit IRC20:55
*** johnbelamaric has joined #openstack-dns20:55
*** crc32 has joined #openstack-dns21:01
*** mwagner_lap has joined #openstack-dns21:04
*** ericpeterson has quit IRC21:08
*** ericpeterson has joined #openstack-dns21:14
*** vinod1 has quit IRC21:31
*** bauruine has quit IRC21:34
*** bauruine has joined #openstack-dns21:38
*** ericpeterson has quit IRC21:42
*** EricGonczer_ has quit IRC21:51
*** paul_glass has quit IRC22:07
*** MentalRay has quit IRC22:28
*** ericpeterson has joined #openstack-dns22:32
*** ericpeterson has quit IRC22:32
*** ericpeterson has joined #openstack-dns22:32
*** paul_glass has joined #openstack-dns22:59
*** EricGonczer_ has joined #openstack-dns23:00
*** EricGonczer_ has quit IRC23:05
*** EricGonczer_ has joined #openstack-dns23:06
*** EricGonczer_ has quit IRC23:21
*** EricGonczer_ has joined #openstack-dns23:29
*** ericpeterson has quit IRC23:35
*** kei_yama has joined #openstack-dns23:36
*** crc32 has quit IRC23:38
*** andrewbogott is now known as andrewbogott_afk23:52
*** ericpeterson has joined #openstack-dns23:56

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!