Wednesday, 2015-03-11

*** GonZo2K has quit IRC00:06
*** MentalRay has joined #openstack-dns00:07
*** andrewbogott is now known as andrewbogott_afk00:18
*** bandarji has joined #openstack-dns00:20
*** rmoe has quit IRC00:21
*** rmoe has joined #openstack-dns00:40
*** ianbrown has joined #openstack-dns00:49
*** crc32 has joined #openstack-dns01:20
*** crc32 has quit IRC01:20
*** crc32 has joined #openstack-dns01:22
*** jmcbride has quit IRC01:27
*** MentalRay has quit IRC01:39
*** MentalRay has joined #openstack-dns01:41
*** stanzgy has joined #openstack-dns01:41
*** stanzgy_ has joined #openstack-dns01:41
*** pk_ has quit IRC01:58
*** EricGonczer_ has joined #openstack-dns02:06
*** MentalRay has quit IRC02:14
*** jmcbride has joined #openstack-dns02:24
*** EricGonczer_ has quit IRC02:28
*** bandarji has quit IRC02:34
openstackgerritstanzgy proposed openstack/designate: Introduce Guru Meditation Reports into Designate  https://review.openstack.org/16327203:00
openstackgerritstanzgy proposed openstack/designate: Introduce Guru Meditation Reports into Designate  https://review.openstack.org/16327203:08
*** pk_ has joined #openstack-dns03:09
openstackgerritstanzgy proposed openstack/designate: Introduce Guru Meditation Reports into Designate  https://review.openstack.org/16292803:10
*** pk_ has quit IRC03:13
*** ianbrown has quit IRC03:13
*** jmcbride has quit IRC03:16
*** ianbrown has joined #openstack-dns03:21
*** EricGonczer_ has joined #openstack-dns03:22
*** EricGonczer_ has quit IRC03:28
*** richm has quit IRC04:38
*** crc32 has quit IRC04:39
*** ianbrown has quit IRC04:45
*** ianbrown has joined #openstack-dns04:51
*** ianbrown has quit IRC06:29
*** chlong has quit IRC08:01
*** chlong has joined #openstack-dns10:23
*** stanzgy_ has quit IRC10:55
*** stanzgy has quit IRC10:55
openstackgerritMerged openstack/designate-specs: Add guru-meditation-reports spec  https://review.openstack.org/16255711:01
*** untriaged-bot has joined #openstack-dns11:02
untriaged-botUntriaged bugs so far:11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142566811:02
openstackLaunchpad bug 1425668 in Designate "Poor error message when using same database for designate and the pool manager cache" [Undecided,New]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/143045511:02
openstackLaunchpad bug 1430455 in Designate "sync-record is failing with 500 error" [Undecided,New]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141633711:02
openstackLaunchpad bug 1416337 in Designate "Designate server create with concurrent request is not listing all servers even after successful creation." [Undecided,Incomplete]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142742511:02
openstackLaunchpad bug 1427425 in Designate "Zones remain in ERROR indefinitely with the noop pool manager cache" [Undecided,In progress] - Assigned to Vinod Mangalpally (vinod-mang)11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142743311:02
openstackLaunchpad bug 1427433 in Designate "Pool Manager Recovery Code Needs to Update Status in Central" [Undecided,In progress] - Assigned to Vinod Mangalpally (vinod-mang)11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142741111:02
openstackLaunchpad bug 1427411 in Designate "Recordset updates remain PENDING until periodic sync" [Undecided,New]11:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142511711:02
openstackLaunchpad bug 1425117 in Designate "Designate does not work with postgres" [High,New]11:02
*** untriaged-bot has quit IRC11:02
*** EricGonczer_ has joined #openstack-dns11:47
*** EricGonczer_ has quit IRC11:49
*** EricGonczer_ has joined #openstack-dns12:06
*** richm has joined #openstack-dns12:15
*** mariusv has joined #openstack-dns12:58
*** mariusv has quit IRC12:58
*** mariusv has joined #openstack-dns12:58
*** mariusv has quit IRC13:01
*** mariusv has joined #openstack-dns13:02
*** mariusv has quit IRC13:02
*** mariusv has joined #openstack-dns13:02
*** mariusv has quit IRC13:02
*** mariusv has joined #openstack-dns13:02
*** mariusv has quit IRC13:02
*** mariusv has joined #openstack-dns13:02
openstackgerritKiall Mac Innes proposed openstack/designate: Enable Tempest Functional Tests  https://review.openstack.org/16314213:03
*** EricGonczer_ has quit IRC13:20
*** jmcbride has joined #openstack-dns13:46
*** jmcbride has quit IRC13:57
*** jmcbride has joined #openstack-dns13:57
*** MentalRay has joined #openstack-dns14:00
*** MentalRay has quit IRC14:01
*** MentalRay has joined #openstack-dns14:05
*** vinod1 has joined #openstack-dns14:22
openstackgerritKiall Mac Innes proposed openstack/designate: Enable Tempest Functional Tests  https://review.openstack.org/16314214:29
openstackgerritGraham Hayes proposed openstack/designate: Fix DesignateObjects ListObjectMixin interactions  https://review.openstack.org/16346114:47
*** jmcbride has quit IRC14:53
*** paul_glass has joined #openstack-dns14:58
*** EricGonczer_ has joined #openstack-dns14:59
*** jmcbride has joined #openstack-dns15:05
openstackgerritGraham Hayes proposed openstack/designate: Fix DesignateObjects ListObjectMixin interactions  https://review.openstack.org/16346115:05
openstackgerritVinod Mangalpally proposed openstack/designate: Call central's update_status correctly  https://review.openstack.org/16275415:15
*** EricGonczer_ has quit IRC15:25
*** jschwarz has joined #openstack-dns15:31
jschwarzhi guys :)15:32
jschwarzI followed this guide here: http://designate.readthedocs.org/en/latest/devstack.html to install designate using devstack15:34
jschwarzbut I'm having a hard time getting the pdns process connected to the instances15:35
jschwarzany ideas what I'm doing wrong? :<15:35
*** EricGonczer_ has joined #openstack-dns15:39
Kialljschwarz: Humm - DevStack should just setup everything for you?15:40
KiallWhat specifically is failing?15:40
KiallOh.. That's out of date.15:41
jschwarzKiall, well it looks to be working (designate-api, designate-central are working)15:41
Kiallthe localrc specifically - here's one I use daily - https://github.com/openstack/designate/blob/master/contrib/vagrant/localrc15:41
jschwarzSo I created a new domain name and figured I'd be able to ping it from an instance15:41
KiallKilo has added a new service or two, which aren't listed in the docs localrc file15:42
jschwarzahh I see15:42
jschwarzalright, I'll try to restack and see then15:42
jschwarzother than that it's supposed to work out of the box, right?15:42
KiallYep, I'll update the docs..15:42
openstackgerritGraham Hayes proposed openstack/designate: Added Schemas to DesignateObjects  https://review.openstack.org/16313115:43
KiallYea, it should setup everything for you.. Next to that localrc is  Vagrantfile BTW .. If you know Vgarant, that can be easier :)15:43
jschwarzKiall, thanks :)15:44
*** einarr has quit IRC15:44
jschwarzKiall, also wanted to thank for accepting my patch the other day on the devstack/install.sh script15:44
jschwarzwas expecting a few review cycles at least :)15:45
KiallHah, no problem :)15:46
*** einarr has joined #openstack-dns15:47
KiallOh .. Also .. http://designate.readthedocs.org/en/latest/devstack.html <-- Very very old! I keep meaning to figure out how to delete those15:49
Kiallhttp://docs.openstack.org/developer/designate/ <-- Where the docs live these days15:49
jschwarz:O15:49
jschwarzgoogle doesn't point there AT ALL15:49
timsimIdeally we could redirect that somehow...15:50
openstackgerritKiall Mac Innes proposed openstack/designate: Update localrc docs sample  https://review.openstack.org/16349915:53
jschwarzwe could probably edit that site to redirect to the correct page15:53
Kialljschwarz: good idea.. I can point it at a temp repo and add a "We've moved" to every page..15:54
jschwarz:)15:54
jschwarzKiall, right so I've restacked and got 4 designate processes16:00
jschwarzKiall, then created a new domain (john.com.) and a new A record (nova1.john.com.), but no ping to nova1.john.com16:01
jschwarzwhat am I doing wrong?16:01
KiallWell, a quick way to validate things are up and running is to run ./exercises/designate.sh from within the DevStack folder - it'll make sure things are all up + running ...16:02
KiallAlso - If your just pining it - then unless you own john.com, and have it delegated to Designate it won't resolve :)16:03
Kiall`host nova1.john.com. 127.0.0.1` will bypass the the fact that you haven't delegated the domain to designate, and just force a lookup locally16:03
jschwarzKiall, ended successfully16:03
Kialljust pinging*16:03
jschwarzalright. where do I put that string?16:04
jschwarz(obviously I don't own john.com :< :<)16:04
KiallOh, no - it's a command you can run the lookup the DNS records for a given name, against a specific server16:04
jschwarzKiall, right. so I should run that against mdns?16:06
jschwarzactually powerdns is installed on my vm16:06
KiallAgainst pdns usually16:06
*** EricGonczer_ has quit IRC16:10
*** einarr has quit IRC16:13
*** d34dh0r53 has quit IRC16:13
*** timsim has quit IRC16:14
*** jroll has quit IRC16:14
*** zigo_ has quit IRC16:14
*** zigo_ has joined #openstack-dns16:14
*** timsim has joined #openstack-dns16:15
*** d34dh0r53 has joined #openstack-dns16:15
jschwarzright, thanks a lot for the help :)16:15
jschwarzwill try to get by from here :)16:15
*** jroll has joined #openstack-dns16:15
*** jroll has quit IRC16:16
*** jroll has joined #openstack-dns16:16
*** einarr has joined #openstack-dns16:16
*** rmoe has quit IRC16:17
*** EricGonczer_ has joined #openstack-dns16:21
*** jschwarz has quit IRC16:29
Kiallvinod1 / timsim / rjrjr etc - DST happens, meeting is 30 mins rather than 1hr 30mins16:31
timsimYeah we were talking about it internal, it's actually an hour later for us than it was.16:32
* timsim doesn't understand time16:32
*** andrewbogott_afk is now known as andrewbogott16:35
*** jschwarz has joined #openstack-dns16:36
*** rmoe has joined #openstack-dns16:36
*** jmcbride has quit IRC16:39
mugsietimsim: its this (somewhat) linear thing16:41
timsimWhy doesn't everyone just go on America time? That would be much easier and better. I mean, I think everyone agrees we do everything right ;)16:42
elarson+1 :)16:44
*** jmcbride has joined #openstack-dns16:44
Kialltimsim: which America time? PT? ET? CT? MT? ;)16:45
mugsiehey, I already live on ET16:45
timsimCST IS THE ONLY ONE THAT MATTERS ANYWAY http://media.giphy.com/media/pYfEywOAolwnm/giphy.gif16:45
mugsiethe last time I flew west, I slept in the first morning;)16:45
mugsietimsim: I now have a headache ;)16:46
Kialltimsim: Next time I see you, you're getting a punch ;)16:46
KiallI bet you voted in the "Liberty" name too? ;)16:46
mugsiethat could be the logo for the next OpenStack release16:46
timsim:P yeah I actually can't get the image to go away in my client16:46
mugsiecli based clients ftw16:47
timsimI was all for "Lizard" actually, I thought that would have been hilarious.16:47
KiallI wanted Lemon ;)16:47
mugsieits ironic the Liberty Design summit is being held... in Canada16:47
KiallOr Lemming ;)16:47
timsimhaha16:48
*** rjrjr_ has joined #openstack-dns16:51
*** jmcbride has quit IRC16:52
*** jmcbride has joined #openstack-dns16:52
rjrjr_i won't be able to attend the weekly meeting today.16:53
*** openstack has joined #openstack-dns16:54
*** ChanServ sets mode: +v openstack16:54
jschwarzthanks16:55
*** MentalRay has quit IRC16:56
Kiallrjrjr: BTW, re your icehouse review.  I left a link in there to a commit made in Juno.. It might be the full fix..16:58
mugsierjrjr: cool - have anything to flag? or you just busy with the internal deploy still?16:58
*** untriaged-bot has joined #openstack-dns17:02
untriaged-botUntriaged bugs so far:17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142566817:02
openstackLaunchpad bug 1425668 in Designate "Poor error message when using same database for designate and the pool manager cache" [Undecided,New]17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/141633717:02
openstackLaunchpad bug 1416337 in Designate "Designate server create with concurrent request is not listing all servers even after successful creation." [Undecided,Incomplete]17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142742517:02
openstackLaunchpad bug 1427425 in Designate "Zones remain in ERROR indefinitely with the noop pool manager cache" [Undecided,In progress] - Assigned to Vinod Mangalpally (vinod-mang)17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142743317:02
openstackLaunchpad bug 1427433 in Designate "Pool Manager Recovery Code Needs to Update Status in Central" [Undecided,In progress] - Assigned to Vinod Mangalpally (vinod-mang)17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142741117:02
openstackLaunchpad bug 1427411 in Designate "Recordset updates remain PENDING until periodic sync" [Undecided,New]17:02
untriaged-bothttps://bugs.launchpad.net/designate/+bug/142511717:02
openstackLaunchpad bug 1425117 in Designate "Designate does not work with postgres" [High,New]17:02
*** untriaged-bot has quit IRC17:02
*** mwagner_lap has quit IRC17:03
*** jordanP has joined #openstack-dns17:16
*** chlong has quit IRC17:21
*** MentalRay has joined #openstack-dns17:24
*** jschwarz has quit IRC17:43
*** mwagner_lap has joined #openstack-dns17:49
Kialloo.. that tempest stuff is workign in the gate :) Just need to get the results logged somewhere more useful ;)18:03
Kiall( this get's it turned on.. https://review.openstack.org/#/c/163142/ )18:03
*** rjrjr_ has joined #openstack-dns18:04
*** pk_ has joined #openstack-dns18:07
ekarlsoKiall: what was the api change ?18:10
Kiallit's not up yet.. was agreed today in the meet18:11
rjrjr_Kiall: we are seeing intermittent timeouts when doing things like create records, list records, etc.18:12
rjrjr_we are on Icehouse.18:12
rjrjr_the timeout is between the API and Central.18:12
rjrjr_is the RPC call using MQ?18:13
Kiallrjrjr: With a large # of records in the zone?18:13
*** kbyrne has quit IRC18:13
rjrjr_we have about 25,000 record in Designate right now, spread across approximately 100 zones.18:13
rjrjr_biggest zone has about 14,000 records.18:13
rjrjr_not too terribly large.18:13
KiallOkay, and does it happen most often to that zone?18:13
rjrjr_no.  when the issue occurs, it appears to affect all zones.18:14
*** kbyrne has joined #openstack-dns18:14
rjrjr_for example, i tried to create a PTR record today in a zone with about 50 records.18:14
rjrjr_and got a timeout.18:14
KiallThat sounds like either central is either overloaded / locked up - or the DB is taking a while to respond..  i'm guessing you've checked the DB is all good already?18:16
rjrjr_are the RPC calls using the MQ?  could the MQ be bogged down with non Designate traffic?18:16
Kiallyea, they go over the queue - probably rabbitmq or qpid - whichever you have deployed18:17
rjrjr_the DB is responding.  in fact, while the issue is occurring, we have been going into the DB to see what we have.18:17
rjrjr_rabbitMQ18:17
KiallOne thing to check, grab the request-id from a timed out request (you should see that in the API logs), then grep central's logs for that request id.. I'd start by comparing the timings on both sides.. e.g. the API waited X seconds, but central took X+10 seconds to return..  Or the message never arrived at any of the central's, which indicates it sat in the queue for longer than the timeout + 10sec  (I think +10 sec anyway)18:18
rjrjr_i'm thinking we are seeing the later.18:18
rjrjr_okay, that gives me some information to look further.18:18
rjrjr_generally, would it be better for us to move to Juno versus Icehouse?18:19
KiallDo you have any access to the rmq mgmt UI? it's real time rather than historic - but will show spikes etc18:19
rjrjr_i've already run into issues in Icehouse that look like they were fixed in Juno.  (like the 'sync-record' issue i opened a bug on.)18:19
KiallI'm not sure I clearly remember all the Icehouse  -> Juno changes, so if it's the queue being bogged down, it might not help..18:20
rjrjr_okay.  thanks.  back to firefighting...18:20
KiallThe queue, is it a shared thing or dedicated to Designate?18:21
KiallIf it's shared - and it's the cause - I'd be expecting other services to be hurting too18:21
rjrjr_rabbitMQ is shared.  sink is involved.18:21
rjrjr_with us using Designate Sink, not sure we can not use a shared rabbitMQ.18:22
KiallAh yes.. If other services aren't hurting, I'd expect it to be designate rather than the queue18:22
rjrjr_Kiall: i appreciate the pointers.  i have more digging to do.18:22
KiallAre your seeing DB deadlocks in the central logs etc? That was an issue eanderson pointed out in icehouse/juno, where central could get itself into a twist18:23
rjrjr_i do see a lot of 'lock wait' issues, yes.  looking into that right now.18:27
rjrjr_Lots of these: \'Lock wait timeout exceeded; try restarting transaction\'18:28
rjrjr_were there fixes for this in Kilo?18:29
openstackgerritEndre Karlson proposed openstack/designate: Support secondary zones  https://review.openstack.org/13368218:34
openstackgerritEndre Karlson proposed openstack/designate: MDNS part of Secondary zones  https://review.openstack.org/13368318:35
Kiallrjrjr: ooh, yep.. that's going to be the one.18:36
KiallI *think* you actually fixed that one in Kilo as part of one of your pools reviews..18:36
Kiallhttps://bugs.launchpad.net/designate/+bug/1392762 - there's a relativly small temporary fix we added in there18:39
openstackLaunchpad bug 1392762 in Designate juno "Concurrent requests can cause designate-central to lock up" [High,Triaged]18:39
Kiallhttps://git.openstack.org/cgit/openstack/designate/commit/?id=768ee18830c5eba554344117509e2caa9980ee3418:39
Kialland then, this did fix properly.. https://git.openstack.org/cgit/openstack/designate/commit/?id=4e75f5b54c149bfd1aeea139bd8108c3031d049c18:40
Kiallthough .. we still have the temp fix in the codebase.. whoops.18:40
*** andrewbogott is now known as andrewbogott_afk18:45
*** andrewbogott_afk is now known as andrewbogott19:01
*** paul_glass has quit IRC19:02
*** rmoe has quit IRC19:06
*** jordanP has quit IRC19:08
*** paul_glass has joined #openstack-dns19:10
*** jmcbride has quit IRC19:10
*** rmoe has joined #openstack-dns19:11
*** jmcbride has joined #openstack-dns19:12
openstackgerritEndre Karlson proposed openstack/designate: Support secondary zones  https://review.openstack.org/13368219:17
*** MentalRay_ has joined #openstack-dns19:22
ekarlsoelarson: what you mean with your commebt on >https://review.openstack.org/#/c/133682/35/designate/storage/impl_sqlalchemy/__init__.py?19:23
*** MentalRay has quit IRC19:26
*** jmcbride has quit IRC19:36
*** jmcbride has joined #openstack-dns19:39
*** pk__ has joined #openstack-dns20:07
*** pk_ has quit IRC20:07
*** MentalRay_ has quit IRC20:10
*** MentalRay_ has joined #openstack-dns20:13
*** vinod1 has quit IRC20:16
*** vinod1 has joined #openstack-dns20:17
*** mwagner_lap has quit IRC20:30
ekarlsoelarson: u up ? :p20:47
elarsonekarlso: indeed I am!20:47
* elarson has had a flaky connection to his bouncer today20:48
*** paul_glass has quit IRC21:01
*** paul_glass has joined #openstack-dns21:02
*** andrewbogott is now known as andrewbogott_afk21:04
*** andrewbogott_afk is now known as andrewbogott21:24
ekarlsoelarson: / vinod1 care to review secondary stuff ?21:29
ekarlsoabout to upload new ps ..21:29
elarsonekarlso: happy to21:29
* elarson hopes his feedback has been helpful 21:29
*** boris-42 has quit IRC21:32
*** jmcbride has quit IRC21:34
*** penick has joined #openstack-dns21:34
*** vinod1 has quit IRC21:46
*** pk_ has joined #openstack-dns21:48
*** pk__ has quit IRC21:48
*** EricGonczer_ has quit IRC21:48
openstackgerritEndre Karlson proposed openstack/designate: MDNS part of Secondary zones  https://review.openstack.org/13368321:58
openstackgerritEndre Karlson proposed openstack/designate: Support secondary zones  https://review.openstack.org/13368221:58
*** paul_glass has quit IRC21:59
elarsonekarlso: I'll take a look at that review tomorrow.22:01
elarsonI'm sure you were on the edge of your seat wondering, so now you know :)22:02
ekarlsoah :p22:02
*** csoukup has joined #openstack-dns22:09
*** bandarji has joined #openstack-dns22:11
*** penick has quit IRC22:14
andrewbogottzigo: have a moment to help me with package building?22:15
andrewbogottmy attempts bomb out at 'setup.py': [Errno 2] No such file or directory which makes me think i’m making some very naive mistake22:16
*** rjrjr_ has quit IRC22:17
*** pk_ has quit IRC22:32
*** tundrajimmy has joined #openstack-dns22:39
*** penick has joined #openstack-dns22:56
*** penick has quit IRC22:56
*** EricGonczer_ has joined #openstack-dns23:00
*** jmcbride has joined #openstack-dns23:02
*** jmcbride has quit IRC23:03
*** pk_ has joined #openstack-dns23:06
*** pk_ has quit IRC23:06
*** pk_ has joined #openstack-dns23:07
*** pk_ has quit IRC23:07
*** pk_ has joined #openstack-dns23:07
*** EricGonczer_ has quit IRC23:20
*** EricGonczer_ has joined #openstack-dns23:39
*** MentalRay_ has quit IRC23:39
*** chlong has joined #openstack-dns23:42
*** penick has joined #openstack-dns23:48
*** penick has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!