Monday, 2015-12-07

*** tom_simpson has joined #openstack-dns00:08
*** chlong has quit IRC00:13
*** ducttape_ has quit IRC00:25
*** chlong has joined #openstack-dns00:27
*** jet-rongl has joined #openstack-dns00:29
*** ducttape_ has joined #openstack-dns00:34
*** ducttape_ has quit IRC00:41
*** tom_simpson has quit IRC00:42
*** chlong has quit IRC01:01
*** tom_simpson has joined #openstack-dns01:04
*** ducttape_ has joined #openstack-dns01:08
*** ducttape_ has quit IRC01:17
*** stanzgy has joined #openstack-dns01:37
*** tom_simpson has quit IRC01:38
*** ducttape_ has joined #openstack-dns01:52
*** ducttape_ has quit IRC01:52
*** cvstealth has quit IRC01:53
*** cvstealth has joined #openstack-dns01:53
*** EricGonczer_ has joined #openstack-dns01:55
*** ducttape_ has joined #openstack-dns01:56
*** cvstealth has quit IRC02:00
*** cvstealth has joined #openstack-dns02:04
*** cvstealth has quit IRC02:10
*** ducttape_ has quit IRC02:11
*** EricGonczer_ has quit IRC02:12
*** ducttape_ has joined #openstack-dns02:23
*** tom_simpson has joined #openstack-dns02:29
*** ducttape_ has quit IRC02:32
*** chlong has joined #openstack-dns02:36
*** sonuk has joined #openstack-dns02:51
*** ducttape_ has joined #openstack-dns02:53
*** cvstealth has joined #openstack-dns03:02
*** cvstealth has quit IRC03:07
*** cvstealth has joined #openstack-dns03:14
*** cvstealth has quit IRC03:18
*** ducttape_ has quit IRC03:20
*** bpokorny has joined #openstack-dns03:23
*** ducttape_ has joined #openstack-dns03:55
*** ianbrown has joined #openstack-dns04:00
*** ducttape_ has quit IRC04:10
*** ianbrown has quit IRC04:15
*** ianbrown_ has joined #openstack-dns04:15
*** cvstealth has joined #openstack-dns04:25
*** ianbrown_ is now known as ianbrown04:28
*** ducttape_ has joined #openstack-dns04:33
*** ianbrown has quit IRC04:42
*** ianbrown has joined #openstack-dns04:42
*** bzhao0825 has quit IRC04:43
*** tom_simpson has quit IRC04:45
*** chlong has quit IRC05:05
*** tom_simpson has joined #openstack-dns05:06
*** ducttape_ has quit IRC05:10
openstackgerritKATO Tomoyuki proposed openstack/designate: Use git.openstack.org as Git repository  https://review.openstack.org/25401305:19
*** chlong has joined #openstack-dns05:22
*** ducttape_ has joined #openstack-dns05:23
*** ducttape_ has quit IRC05:28
*** ianbrown has quit IRC05:29
*** ianbrown has joined #openstack-dns05:29
openstackgerritsonu proposed openstack/designate-specs: Pass environment variables of proxy to tox  https://review.openstack.org/25401705:41
*** rudrajit has quit IRC05:42
*** tom_simpson has quit IRC05:43
*** rudrajit has joined #openstack-dns05:53
*** tom_simpson has joined #openstack-dns05:57
*** bpokorny has quit IRC06:01
*** stanzgy has quit IRC06:01
*** jmcbride has quit IRC06:04
*** ducttape_ has joined #openstack-dns06:24
*** ianbrown has quit IRC06:26
*** ianbrown has joined #openstack-dns06:26
*** stanzgy has joined #openstack-dns06:29
*** ianbrown has quit IRC06:30
*** ducttape_ has quit IRC06:30
*** ianbrown has joined #openstack-dns06:30
*** ianbrown has quit IRC06:31
*** ianbrown has joined #openstack-dns06:32
*** boris-42_ has quit IRC06:33
*** tom_simpson has quit IRC06:45
*** openstackgerrit_ has joined #openstack-dns06:51
*** rudrajit has quit IRC07:22
*** ducttape_ has joined #openstack-dns07:27
*** chlong has quit IRC07:29
*** ducttape_ has quit IRC07:31
*** ianbrown_ has joined #openstack-dns07:55
*** ianbrown has quit IRC07:58
*** nyechiel_ has joined #openstack-dns08:03
openstackgerritsonu proposed openstack/designate: Removes MANIFEST.in as it is not needed explicitely by PBR  https://review.openstack.org/25405708:05
openstackgerritsonu proposed openstack/python-designateclient: Removes MANIFEST.in as it is not needed explicitely by PBR  https://review.openstack.org/25405908:10
*** shakamunyi has quit IRC08:14
*** ianbrown_ has quit IRC08:19
*** ianbrown_ has joined #openstack-dns08:23
openstackgerritsonu proposed openstack/designate-dashboard: Drop MANIFEST.in as it is not needed explicitely by PBR  https://review.openstack.org/25406708:25
*** ducttape_ has joined #openstack-dns08:27
openstackgerritsonu proposed openstack/designate-specs: Removes MANIFEST.in as it is not needed explicitely by PBR  https://review.openstack.org/25406808:30
*** ducttape_ has quit IRC08:32
*** ianbrown_ has quit IRC08:51
*** jordanP has joined #openstack-dns08:53
*** nyechiel_ has quit IRC08:54
*** nyechiel_ has joined #openstack-dns08:55
openstackgerritEndre Karlson proposed openstack/python-designateclient: Indicate py34 support  https://review.openstack.org/25408008:55
*** ianbrown has joined #openstack-dns08:56
*** sonuk has quit IRC09:06
*** nyechiel has joined #openstack-dns09:13
*** nyechiel_ has quit IRC09:13
openstackgerritEndre Karlson proposed openstack/designate: Move config to avoid import issues with eventlet  https://review.openstack.org/23698709:18
*** nyechiel has quit IRC09:19
openstackgerritEndre Karlson proposed openstack/designate: Support running Designate API in Apache 2  https://review.openstack.org/23623609:25
*** ducttape_ has joined #openstack-dns09:28
*** ducttape_ has quit IRC09:32
*** ianbrown has quit IRC09:40
*** ianbrown has joined #openstack-dns09:45
*** nyechiel has joined #openstack-dns09:57
*** ianbrown has quit IRC10:13
*** ianbrown has joined #openstack-dns10:13
*** en_austin has joined #openstack-dns10:21
en_austinhi all, anybody here? :)10:22
*** sonuk has joined #openstack-dns10:23
*** openstackgerrit_ has quit IRC10:23
*** ducttape_ has joined #openstack-dns10:29
*** ducttape_ has quit IRC10:33
openstackgerritFederico Ceratto proposed openstack/designate: Add delayed NOTIFY  https://review.openstack.org/24758810:44
*** stanzgy has quit IRC10:48
*** sonuk has quit IRC11:01
ekarlsoen_austin: ya11:01
*** sonuk has joined #openstack-dns11:08
en_austinekarlso: hi :)11:22
en_austinI need an advice...11:22
en_austinTwice for last week I've encountered a strange issue with Designate. It just hanging with backend communication (or whatever, I don't know) until restart.11:23
en_austine.g API receives a queries, but they are not actually provisioned. Zones is pending in PENDING status (not even ERROR).11:23
en_austinAfter restart of all Designate services it continues running as usual without any problems.11:24
en_austinWhat it can be?11:24
*** ducttape_ has joined #openstack-dns11:30
*** Guest20179 has quit IRC11:34
*** ducttape_ has quit IRC11:34
*** noqa_v_q1ovnie is now known as noqa_v_qoovnie11:48
*** timsim has quit IRC11:56
*** timsim has joined #openstack-dns12:00
*** jet-rongl has quit IRC12:09
*** nyechiel has quit IRC12:12
*** kei_yama has quit IRC12:21
*** chlong has joined #openstack-dns12:30
*** ducttape_ has joined #openstack-dns12:30
Kiallen_austin: interesting, any idea which service restart kicked it?12:33
KiallOne thing I've seen, is oslo.messaging - the openstack RPC lib - doesn't really like network blips when combined with RabbitMQ ... Things can end up listening to dead air until they are restarted, with no way for the service to know  it's busted.12:34
KiallWe saw this issue all the time, when we had some netutron issues and had designate running in-cloud.12:34
Kiall(Theres a multiple year old bug on this one ;))12:35
Kiall(bug against oslo.m, not us ;))12:35
*** ducttape_ has quit IRC12:35
Kialland - logs etc would be useful otherwise12:35
*** ducttape_ has joined #openstack-dns12:45
*** ducttape_ has quit IRC13:01
*** EricGonczer_ has joined #openstack-dns13:06
*** EricGonczer_ has quit IRC13:09
openstackgerritFederico Ceratto proposed openstack/designate: Emit notification on update_status() on a zone  https://review.openstack.org/25417613:10
en_austinI'll try to fetch logs belongs to this.13:19
en_austinKiall: for example - here is a stuck domain deletion query http://paste.openstack.org/show/B1TAJVf06w9L7Doju8E8/13:21
en_austinAwh. Why paste.openstack has cut my logs...13:22
Kialllol13:22
KiallI was just thinking there's not much of relavance ;)13:22
en_austinit pasted only 186 lines13:23
en_austinwhile I've expected 249 :)13:23
en_austinwell, part two http://paste.openstack.org/show/ZUoR01J6R9bYy3oCARti/13:23
en_austinand about oslo.m - RabbitMQ and Designate services are on the same box, talking by 127.0.0.1 interface... so probably it's not a problem here13:25
Kiallcan you check the 2x nameserver logs at around 2015-12-07 08:20:03.06513:28
en_austinyep, will do it now13:28
KiallLooking to see if they logged a NOTIFY received for for int.zone13:28
en_austin07-Dec-2015 08:20:04.384 zone aqa.int.zone/IN: notify from 10.28.0.16#34271: refresh in progress, refresh check queued13:29
en_austinonly that13:29
en_austin(and same lines with different timestamps)13:29
KiallOkay, that seems about right - after that line, do you see when the next AXFR complete log entry for that zone is?13:30
KiallActually, That log is for aqa.int.zone vs int.zone13:30
en_austini've just cut aqa.* from logs i've posted above13:31
en_austinit's ok here13:31
KiallAh :)13:31
en_austin07-Dec-2015 09:59:50.081 zone aqa.int.zone/IN: transferred serial 144947157513:31
en_austinafter couple hundreds of "refresh in progress..."13:31
en_austin(that's when I've manually restarted a Designate)13:31
KiallHumm, K.. Can you grep the mdns logs, loking for any stacktraces (they should match a grep for TRACE) before 2015-12-07 08:20:03.06513:32
KiallI'm wondering if we have an unhandled exception somewhere, which is shutting down the listener thread13:33
en_austinno any traces - already done that :(13:33
en_austinthe only exception was only when i've restarted D13:33
en_austincentral.log.6:Exception TypeError: "'NoneType' object is not callable" in <bound method _SocketDuckForFd.__del__ of _SocketDuckForFd:5> ignored13:33
en_austin(and same in zone-manager.log)13:34
*** sonuk has quit IRC13:34
KiallOo, you can paste some of the context around that log entry?13:36
en_austinSure13:37
en_austinhttp://paste.openstack.org/show/481016/13:38
KiallFrom ^ it looks like you're running the servers with workers = 4 - are you doing that for all of them?13:41
en_austinwhat "that"?13:41
Kiallthat log :)13:41
en_austin"for all of them" - all workers? yea13:42
en_austinthat's a single log for all of them :)13:42
en_austinwell, will afk for half an hour13:42
KiallK, I wonder if wokers > 1 is related, it's certainly the less tested deployment pattern.. How often does this happen? It seems like we're failing to log something of importance for this issue, it may need some debugging while it's in a failed state to learn more :(13:44
openstackgerritFederico Ceratto proposed openstack/designate: Emit notification on update_status() on a zone  https://review.openstack.org/25417613:45
Kiallbrb13:48
*** nyechiel has joined #openstack-dns13:51
*** boris-42_ has joined #openstack-dns13:52
Kiallback13:54
*** ducttape_ has joined #openstack-dns13:55
*** rsyed_away is now known as rsyed14:01
*** ducttape_ has quit IRC14:14
*** EricGonczer_ has joined #openstack-dns14:15
*** ducttape_ has joined #openstack-dns14:16
*** ducttape_ has quit IRC14:18
*** raginbajin has quit IRC14:44
*** fawadkhaliq has joined #openstack-dns14:44
*** raginbajin has joined #openstack-dns14:46
*** mlavalle has joined #openstack-dns14:49
openstackgerritEric Larson proposed openstack/designate: Update database and add ALIAS record support  https://review.openstack.org/24719314:55
openstackgerritEric Larson proposed openstack/designate: Add ALIAS record flattening zone manager task  https://review.openstack.org/24719214:55
openstackgerritEric Larson proposed openstack/designate: Comment out ALIAS records in zone export  https://review.openstack.org/24719514:55
openstackgerritEric Larson proposed openstack/designate: Add ALIAS support in central and zone manager  https://review.openstack.org/24719414:55
openstackgerritEric Larson proposed openstack/designate: Add ALIAS records to designate objects  https://review.openstack.org/24719114:55
*** mwagner_lap has quit IRC14:57
*** ducttape_ has joined #openstack-dns14:57
*** jmcbride has joined #openstack-dns15:02
en_austinKiall: so...15:06
en_austinyou think that reducing of workers can solve this problem?15:07
en_austin"How often" - twice from end of November till now15:07
en_austinwhen it'll fail next time - I'll create a snapshot of VM and contact you:)15:08
en_austinthe main problem is - the production fails, not dev instance - so I must restart it asap15:08
*** fawadkhaliq has quit IRC15:51
*** pglass has joined #openstack-dns15:53
*** nyechiel has quit IRC15:54
rsyedhere's a low priority, general question: what's the value in recordsets (as opposed to working directly with records on a zone?)  any quick canonical example that would demonstrate their value?15:55
*** mwagner_lap has joined #openstack-dns16:06
mugsiersyed: in the RFCs records belong to a recordset. we tried to keep to the RFCs as much as possible16:09
timsim(The underlying question is "why you no recordsets/<id>/records)16:09
mugsietimsim: I blame you16:10
timsimI AGREED WITH YOU!!!!16:10
rsyedmugsie: he was taking the blame in person too16:10
openstackgerritFederico Ceratto proposed openstack/designate: Emit notification on update_status() on a zone  https://review.openstack.org/25417616:10
timsimI mean I still should have fought it harder. But you know, intern taking 18 hours of classes had bigger problems :P16:11
*** james_li has joined #openstack-dns16:13
timsimmugsie: We could still do that now though right? It'd just be an /v2.1 or something?16:13
timsimOur UI folks are probably going to flip out when they see that.16:13
mugsietimsim: yeaj16:13
mugsieor just add 2 ways16:14
rsyedmugsie: yeah, the addition of the second way shouldn't mandate a version bump should it?16:14
mugsiehave a /recordsewts/<id>/records16:14
mugsieand keep the other way16:14
timsimAh yeah. That's what I meant.16:15
mugsiersyed: nope, as long as the old interaction works we are ++16:15
timsimWell let's do that.16:15
mugsietimsim: spec please :)16:15
* mugsie ducks16:15
*** james_li has quit IRC16:16
mugsiehttp://replygif.net/i/1507.gif16:16
timsimadd it to https://review.openstack.org/#/c/182016/6/specs/liberty/api-v2-structured-data-format.rst16:19
timsim:P16:19
*** fawadkhaliq has joined #openstack-dns16:24
*** bpokorny has joined #openstack-dns16:30
*** EricGonc_ has joined #openstack-dns16:31
*** EricGonczer_ has quit IRC16:35
*** jordanP has quit IRC16:45
*** james_li has joined #openstack-dns16:47
mlavallemugsie, timsim, Kiall: that new way to configure Designate in Devstack rocks! Just drop one line in local.conf and you are up an running! :-)16:47
*** en_austin has quit IRC16:50
mugsiemlavalle: yeah, well done to ekarlso16:52
rsyedmlavalle is this what you're referring to?  http://docs.openstack.org/developer/designate/devstack.html16:52
mugsiersyed: nope - we need to update the docs16:52
mugsiethere is a new devstack plugin16:52
rsyedhttps://github.com/openstack/designate/tree/master/devstack ?16:53
mugsieyeah16:53
mlavallersyed: this last one is what I was referring to16:53
rsyedthanks!  might give it a shot myself, looks simple16:53
*** rudrajit has joined #openstack-dns17:33
*** rudrajit has quit IRC17:34
*** rudrajit has joined #openstack-dns17:35
*** cliles has quit IRC17:37
*** fawadkhaliq has quit IRC17:52
*** fawadkhaliq has joined #openstack-dns18:02
*** cliles has joined #openstack-dns18:06
*** rsyed is now known as rsyed_away18:25
*** penick has joined #openstack-dns18:27
*** rudrajit_ has joined #openstack-dns18:38
*** rudrajit has quit IRC18:41
*** james_li has quit IRC18:45
*** pglass has quit IRC18:47
*** rsyed_away is now known as rsyed19:03
*** rudrajit has joined #openstack-dns19:14
*** rudrajit_ has quit IRC19:17
*** james_li has joined #openstack-dns19:27
*** pglass has joined #openstack-dns19:32
*** fawadkhaliq has quit IRC19:54
*** jasonsb has quit IRC19:55
*** EricGonc_ has quit IRC20:22
*** james_li has quit IRC20:27
*** jasonsb has joined #openstack-dns20:37
*** pglass has quit IRC20:50
rsyedzone_primary_or_admin should be satisfied when using the noauth context (as all users are admins) right?  i can't seem to create a recordset (PTR specifically) because the policy check fails for "create_recordset" in central20:54
elarsonrsyed: noauth should allow you to create a PTR recordset20:57
rsyedhttp://paste.openstack.org/show/JBETvcmjwNb6bkKFCscn/20:59
*** johnbelamaric has joined #openstack-dns21:00
*** mwagner_lap has quit IRC21:04
*** jasonsb has quit IRC21:07
*** mlavalle has quit IRC21:10
*** mlavalle has joined #openstack-dns21:10
*** ducttape_ has quit IRC21:10
*** jasonsb has joined #openstack-dns21:11
rsyedtimsim helped me figure it out, it was the big rename...a templated version of our policy.json still had domain references instead of zone.21:13
mugsieoh21:15
mugsiecrap21:15
mugsiethats not going to be good21:15
timsimIt's fixed in master, but yeah, folks' policy.jsons are going to have to change.21:15
mugsiegah21:16
timsimMan whose idea was this big rename....21:16
timsim:troll:21:16
elarson:troll2:21:16
rsyedyeah master was fine...it was just our deployment had a template we have to get updated.  why is that a concern?  tell everyone to just deal with it!21:16
timsimPeople will be very offended.21:17
mugsiersyed: if only it was that easy21:18
mugsiehttp://replygif.net/i/1509.gif <- openstack users when you break shit from under them21:19
* mugsie must remember to tell zigo about this21:19
*** bradjones has quit IRC21:20
rsyedthis is a unique case, but i think it's unfortunate that the policy executor wasn't providing too much feedback.  think there's value in providing additional debug detail there?  i'm not sure how often debugging policies would need to happen?21:21
elarsonwe could probably havea deprecation warning and support the old names for the M release?21:21
timsimrsyed: you might find something helpful if you went spelunking in configs for oslo policy21:22
*** bradjones has joined #openstack-dns21:22
*** bradjones has quit IRC21:22
*** bradjones has joined #openstack-dns21:22
*** pglass has joined #openstack-dns21:23
rsyedtimsim: ah thanks for the tip21:23
mugsiei think elarson is probably right - just have to figure out how to do it21:23
*** rudrajit_ has joined #openstack-dns21:29
*** sonuk has joined #openstack-dns21:30
*** mariusv has joined #openstack-dns21:30
*** mariusv has quit IRC21:30
*** mariusv has joined #openstack-dns21:30
*** bpokorny has quit IRC21:31
*** bpokorny has joined #openstack-dns21:32
*** rudrajit has quit IRC21:32
*** EricGonczer_ has joined #openstack-dns21:34
*** ianbrown has quit IRC21:39
*** ianbrown has joined #openstack-dns21:39
*** jasonsb has quit IRC21:39
*** jasonsb has joined #openstack-dns21:40
*** mwagner_lap has joined #openstack-dns21:43
timsimCan we start an etherpad for mid-cycle topics, I've got like 5 from today alone. http://i.imgur.com/t0XHtgJ.gif21:49
mugsiehttp://replygif.net/i/1524.gif21:53
timsimHope you don't run into an unhandled exception anywhere in handling any of the ERROR'd zones in Designate: https://github.com/openstack/designate/blob/master/designate/pool_manager/service.py#L171-L202 otherwise you won't recover.21:59
mugsiegit blame22:02
mugsieand who do we have22:02
mugsieoh. me and Kiall22:02
*** rudrajit has joined #openstack-dns22:02
timsimwomp womp22:02
pglassbut I think kiall was just indenting that for a different change22:03
mugsiehttp://replygif.net/i/923.gif22:03
timsimyeah, looks like it22:03
mugsiepglass: dont give him an out22:03
mugsie:D22:03
timsimSorry Kiall https://github.com/openstack/designate/commit/3386d9539ca02e4a9142cb4cf03b990f64387ded#diff-d4dc7341c5de88be06bd4ca2cf8c76c5R18622:04
*** rudraji__ has joined #openstack-dns22:05
*** rudrajit_ has quit IRC22:05
*** rudrajit_ has joined #openstack-dns22:06
*** rudrajit has quit IRC22:09
timsimI'm going to go ahead a file a bug for that. Although I'm not sure what the proper fix is.22:09
timsimWell, besides https://media.giphy.com/media/as9jjco4BcAkE/giphy.gif22:09
*** rudraji__ has quit IRC22:10
*** james_li has joined #openstack-dns22:18
*** mariusv_ has joined #openstack-dns22:29
*** mariusv_ has quit IRC22:32
timsimmugsie: https://etherpad.openstack.org/p/designate_2016_winter_midcycle_meetup22:34
*** rudrajit has joined #openstack-dns22:38
*** rudrajit_ has quit IRC22:41
*** mariusv has quit IRC22:45
*** rsyed is now known as rsyed_away22:48
*** ducttape_ has joined #openstack-dns22:59
*** jmcbride has quit IRC23:06
*** johnbelamaric has quit IRC23:06
*** km has joined #openstack-dns23:08
*** km is now known as Guest8410123:08
*** ianbrown has quit IRC23:12
*** ianbrown has joined #openstack-dns23:12
*** sonuk has quit IRC23:18
*** james_li has quit IRC23:20
*** boris-42_ has quit IRC23:23
*** kei_yama has joined #openstack-dns23:34
*** ianbrown has quit IRC23:41
*** ianbrown has joined #openstack-dns23:42
*** boris-42_ has joined #openstack-dns23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!