Thursday, 2014-07-17

« Wednesday, 2014-07-16 Index Friday, 2014-07-18 »
*** rmoe has quit IRC00:10
*** rmoe has joined #openstack-dns00:24
*** jaycaz has quit IRC00:40
*** jaycaz has joined #openstack-dns00:42
*** eankutse has joined #openstack-dns01:01
*** vinod1 has joined #openstack-dns01:05
*** eankutse has quit IRC01:09
*** vinod1 has quit IRC01:11
*** eankutse has joined #openstack-dns01:30
*** eankutse has quit IRC01:30
*** eankutse has joined #openstack-dns01:30
*** eankutse1 has joined #openstack-dns01:39
*** eankutse has quit IRC01:41
*** eankutse1 has quit IRC01:42
*** eankutse has joined #openstack-dns01:43
*** eankutse has quit IRC01:47
*** nosnos has joined #openstack-dns01:49
*** jaycaz has quit IRC01:56
*** richm has quit IRC01:59
*** darshan104 has quit IRC02:05
*** nosnos has quit IRC03:06
*** vinod1 has joined #openstack-dns03:20
*** vinod1 has quit IRC03:29
*** nosnos has joined #openstack-dns03:33
*** crc32 has quit IRC03:41
*** msisk has quit IRC03:42
*** rjrjr has quit IRC03:46
*** darshan104 has joined #openstack-dns04:42
*** k4n0 has joined #openstack-dns06:22
*** sballe_ has joined #openstack-dns06:28
*** sballe has quit IRC06:34
*** rektide_ has joined #openstack-dns06:34
*** rektide has quit IRC06:34
*** k4n0 has quit IRC06:34
*** k4n0 has joined #openstack-dns06:37
*** darshan104 has quit IRC08:23
*** darshan104 has joined #openstack-dns08:23
*** darshan104 has quit IRC08:28
*** Dafna has quit IRC09:43
*** nosnos has quit IRC09:54
*** nosnos has joined #openstack-dns10:01
*** nosnos has quit IRC10:10
*** nosnos has joined #openstack-dns10:12
*** nosnos has quit IRC10:38
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Ensure each greenthread uses it's own Session instance  https://review.openstack.org/10764410:53
*** Dafna has joined #openstack-dns11:00
*** Dafna has quit IRC11:29
*** Dafna has joined #openstack-dns11:43
*** mwagner_lap has quit IRC12:17
*** richm has joined #openstack-dns12:46
*** k4n0 has quit IRC13:01
*** vinod1 has joined #openstack-dns13:04
*** vinod1 has quit IRC13:07
KiallFYI .. https://review.openstack.org/#/c/107688/113:22
KiallGerrit ACL changes13:22
*** vinod1 has joined #openstack-dns13:26
*** vinod1 has quit IRC13:27
mugsieFYI - links for designate on docs.openstack.org https://review.openstack.org/#/c/107691/13:33
*** jaycaz has joined #openstack-dns13:35
*** eankutse has joined #openstack-dns13:36
*** eankutse has quit IRC13:38
*** eankutse has joined #openstack-dns13:38
*** jaycaz has quit IRC13:41
*** msisk has joined #openstack-dns13:48
*** mwagner_lap has joined #openstack-dns13:53
*** msisk has quit IRC14:10
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Attach records to RecordSet objects  https://review.openstack.org/10588714:15
*** timsim has joined #openstack-dns14:15
*** crc32 has joined #openstack-dns14:18
*** msisk has joined #openstack-dns14:18
*** betsy has quit IRC14:20
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Attach records to RecordSet objects  https://review.openstack.org/10588714:39
*** crc32_ has joined #openstack-dns14:53
*** eankutse1 has joined #openstack-dns14:56
*** eankutse1 has quit IRC14:56
*** eankutse1 has joined #openstack-dns14:56
*** eankutse has quit IRC14:57
*** jaycaz has joined #openstack-dns14:58
*** jaycaz has quit IRC14:58
*** crc32 has quit IRC15:02
*** rektide_ has quit IRC15:02
*** Dafna has quit IRC15:02
*** rektide has joined #openstack-dns15:03
*** Dafna has joined #openstack-dns15:07
*** eankutse1 has quit IRC15:15
*** eankutse has joined #openstack-dns15:16
*** jaycaz has joined #openstack-dns15:17
*** crc32_ has quit IRC15:20
*** betsy has joined #openstack-dns15:20
*** eankutse1 has joined #openstack-dns15:22
*** msisk has quit IRC15:24
*** msisk_ has joined #openstack-dns15:24
*** jaycaz1 has joined #openstack-dns15:25
*** eankutse has quit IRC15:25
*** jaycaz has quit IRC15:26
*** timsim has quit IRC15:26
*** vinod1 has joined #openstack-dns15:27
*** rmoe has quit IRC15:29
*** eankutse has joined #openstack-dns15:31
*** eankutse1 has quit IRC15:31
*** eankutse1 has joined #openstack-dns15:47
*** eankutse has quit IRC15:47
*** eankutse1 has quit IRC15:47
*** eankutse has joined #openstack-dns15:47
*** eankutse has quit IRC15:48
*** eankutse has joined #openstack-dns15:48
*** rmoe has joined #openstack-dns15:49
*** eankutse1 has joined #openstack-dns15:54
*** eankutse has quit IRC15:55
*** eankutse1 has quit IRC15:56
*** eankutse has joined #openstack-dns15:56
openstackgerritA change was merged to openstack/designate: Attach records to RecordSet objects  https://review.openstack.org/10588716:00
*** eankutse1 has joined #openstack-dns16:03
*** eankutse has quit IRC16:05
*** eankutse2 has joined #openstack-dns16:06
*** eankutse1 has quit IRC16:07
*** eankutse2 has quit IRC16:07
*** eankutse1 has joined #openstack-dns16:08
Kiallwoo - 1 step closer :)16:10
vinod1kiall: regarding your change - https://review.openstack.org/#/c/107644/16:11
Kiallvinod1: heya16:12
vinod1did you hit some issue with this?16:12
Kiallkinda ;)16:12
vinod1did youget a chance to look through at other places in the code where we might need thread locals?16:13
KiallI don't think they are any other places we need them, and if there are, then those are bugs which should be fixed without the threadlocal. Ideally this wouldn't use it either, but avoiding it would be a large change to how our Storage layer works and interacts with the rest of the system :(16:15
Kiall(But - Since our storage API is the way it is - it's OK in this case to use them, it's actually even how SQLA recommends you do it..)16:16
vinod1Thanks Kiall for the explanations16:17
Kiallnp16:17
*** crc32 has joined #openstack-dns16:22
*** eankutse has joined #openstack-dns16:23
*** eankutse1 has quit IRC16:24
*** eankutse has quit IRC16:25
*** jaycaz1 has quit IRC16:25
*** eankutse has joined #openstack-dns16:25
*** jaycaz has joined #openstack-dns16:26
*** jaycaz1 has joined #openstack-dns16:28
*** jaycaz has quit IRC16:31
*** eankutse has quit IRC16:34
*** JensOfSweden has quit IRC16:34
*** eankutse has joined #openstack-dns16:34
openstackgerritA change was merged to openstack/designate: Ensure each greenthread uses it's own Session instance  https://review.openstack.org/10764416:37
*** eankutse has quit IRC16:39
*** eankutse has joined #openstack-dns16:40
*** rossk has joined #openstack-dns16:43
*** eankutse1 has joined #openstack-dns16:53
*** eankutse has quit IRC16:56
*** eankutse1 has quit IRC16:57
*** eankutse has joined #openstack-dns16:57
jaycaz1Hey, has anyone ran into issues with oslo.messaging when they try to send api requests from multiple processes?17:11
jaycaz1That's what I'm running into.  vinod mentioned that it's related to this bug? https://launchpad.net/mistral/+bug/133770517:11
uvirtbotLaunchpad bug 1337705 in mistral "Exception during message handling: _oslo_messaging_localcontext" [Undecided,New]17:11
jaycaz1The situation is that I have a python script to create n zones, one at a time, and I am trying to run that simultaneously on multiple processes17:12
vinod1kiall:/ekarlso: Jordan ran into this issue when trying to test out zone creates17:12
vinod1Do you know if there is already a fix for this is oslo.messaging?17:12
*** eankutse1 has joined #openstack-dns17:15
*** Dafna has quit IRC17:16
*** eankutse has quit IRC17:18
*** msisk_ has quit IRC17:34
*** msisk has joined #openstack-dns17:35
*** eankutse1 has quit IRC17:36
*** eankutse has joined #openstack-dns17:36
*** msisk has quit IRC17:41
*** msisk has joined #openstack-dns17:41
vinod1kiall: i had a question on mdns18:06
vinod1the payload received on tcp seems different from that received on udp.  the tcp one seems to have about 4 extra bytes in the beginning.18:17
vinod1but i could not see why that should be18:17
vinod1wondering if you could spot something in the code there18:17
Kiallvinod1: could there be 4 extra bytes going over the wire? (e.g. have you used wireshark or something to look at the packets on-the-wire?)18:25
vinod1no not yet18:29
vinod1i used dig to send a tcp request to mdns18:29
vinod1with udp there is no issue, but tcp has something extra before the payload18:30
*** darshan104 has joined #openstack-dns18:40
*** JensOfSweden has joined #openstack-dns18:53
Kiallvinod1: humm - if you strip the leading 4 bytes of a TCP payload before passing to the handler, does it work?18:57
Kialljaycaz1: for some reason, I think that might be an out of date dependency. No clue why I'm thinking that though18:57
ekarlsogood q vinod1 ;)18:58
Kialljaycaz1: related bug - https://bugs.launchpad.net/oslo.messaging/+bug/128887818:58
uvirtbotLaunchpad bug 1288878 in ceilometer "localcontext store is not monkeypatched by eventlet" [Undecided,Fix released]18:58
ekarlsoor what Kiall said :D18:58
*** timsim has joined #openstack-dns18:59
Kialljaycaz1: Is this a script something that talks to the REST API, or RabbitMQ BTW?19:00
jaycaz1Kiall: it's through the REST API19:01
KiallHumm - I wonder if we've introduced an issue recently (or if oslo.messaging has..)19:01
ekarlsojaycaz1: u care to share your script ?19:02
jaycaz1sure thing!19:03
jaycaz1you can clone it here: https://github.com/jaycaz/designate-scripts19:03
KiallJust standing up a fresh copy of designate.. Let's see if I can reproduce19:04
KiallHumm.. Should have turned keystone off before starting :(19:06
ekarlso:P19:07
KiallOOOHHH.. jaycaz1 .. I may have an idea.19:08
ekarlsoKiall: ?19:08
Kialllooking for the commit19:09
Kialljaycaz1: if you try revision 0a2406739a588a52c8555bcd7ca8695323cfa0f1 of designate, does it work?19:09
jaycaz1Kiall: I'll try that and get back to you!19:10
Kiallactually, you can scratch that19:12
Kiallhttps://github.com/openstack/designate/blob/master/designate/__init__.py19:12
Kiallimports oslo.messaging, and it does it before https://github.com/openstack/designate/blob/master/designate/cmd/__init__.py issues the eventlet.monkey_patch(os=False)19:13
KiallLooks like ekarlso broke it ;)19:13
ekarlsoPFFFT19:13
Kiallhttps://github.com/openstack/designate/commit/0d3a4427d9c9111cd0229d1b05d8517d9a2945b6 <-- That's the first commit that is going to fail ;)19:14
ekarlso:D19:14
* Kiall is VERY glad we haven't deployed that at HP Cloud yet19:14
ekarlsowhy does it need to patch before o.m ?19:14
Kiallbecause otherwise oslo-messaging get's + holds an unpatched reference to the threading module19:15
*** timsim has quit IRC19:16
*** timsim has joined #openstack-dns19:17
ekarlso:)19:18
openstackgerritKiall Mac Innes proposed a change to openstack/designate: Ensure eventlet monkey patching happens as early as possible  https://review.openstack.org/10780719:21
Kialljaycaz1: ^ should sort it19:21
Kiallalso filed bug 1288878 for it19:21
uvirtbotLaunchpad bug 1288878 in ceilometer "localcontext store is not monkeypatched by eventlet" [Undecided,Fix released] https://launchpad.net/bugs/128887819:21
Kiallehh19:21
KiallI mean 134352619:21
jaycaz1Kiall: great, thanks! I'll give it a shot!19:21
Kiallbug 134352619:21
uvirtbotLaunchpad bug 1343526 in designate "Eventlet patching is occurring too late during service startup" [Critical,In progress] https://launchpad.net/bugs/134352619:21
Kiallthere we go :)19:21
jaycaz1Kiall: success!19:23
KiallCool :D19:23
* Kiall is very glad that was found before we released with it -_-19:24
*** JensOfSweden has quit IRC19:30
*** JensOfSweden has joined #openstack-dns19:32
*** vinod1 has quit IRC19:38
*** JensOfSweden has quit IRC19:50
timsimKiall: We're seeing an error "Backend: Unknown backend failure: DBError(u'Session is already flushing',)" Is that related to the bug you filed this morning?19:52
Kialltimsim: not that I know of, any more info to hand?19:56
jaycaz1it happens when we are sending requests from two processes at once19:56
timsimWe're just playing around, but yeah that ^ We're not pushing the requests through a web server or anything, and it's quite a lot of requests.19:57
jaycaz1when the requests are slowed down (i.e. 1 per second) there seem to be no issues19:58
timsimhttp://paste.openstack.org/show/87010/ <- stacktrace19:58
KiallJust managed to reproduce with jaycaz1's change19:58
Kialljaycaz1's script&19:58
Kialljaycaz1's script*19:58
*** JensOfSweden has joined #openstack-dns20:01
KiallWithout my fix from earlier today - the SQLA Session one - It's a different trace:20:02
Kiall2014-07-17 20:01:36.474 TRACE oslo.messaging.rpc.dispatcher InvalidRequestError: This Session's transaction has been rolled back by a nested rollback() call.  To begin a new transaction, issue Session.rollback() first.20:02
Kiall^ is what we saw that prompted the fix in the first place20:02
Kiallhttps://bugs.launchpad.net/ceilometer/+bug/1183996 seems similar, and their fix was basically what we did to the session today.20:04
uvirtbotLaunchpad bug 1183996 in ceilometer "session flushing error" [Undecided,Fix released]20:04
Kialltimsim: can you file a bug? Looking into it now20:06
KiallIt looks related, but not 100% sure yet20:07
*** JensOfSweden has quit IRC20:07
timsimSure.20:09
timsimbug 134356120:12
uvirtbotLaunchpad bug 1343561 in designate "DB Error ('Sesssion is already flushing')" [Undecided,New] https://launchpad.net/bugs/134356120:12
Kialltimsim: Just in time :) Got it20:14
KiallExact same issue, different place. Can't believe I didn't realize this when vinod actually asked me earlier -_-20:14
Kiall10 copies of jaycaz1 script going creating 1000 zones each, no stack traces :)20:14
timsimNice.20:15
jaycaz1awesome!20:15
* Kiall schedules another deploy for tomorrow :(20:16
*** vinod1 has joined #openstack-dns20:16
openstackgerritKiall Mac Innes proposed a change to openstack/designate: PowerDNS: Ensure each greenthread uses it's own Session instance  https://review.openstack.org/10781720:18
Kiallvinod1: ^ you were totally right earlier, and even with your question, I didn't realize it ;)20:18
*** msisk has quit IRC20:22
*** JensOfSweden has joined #openstack-dns20:24
*** msisk has joined #openstack-dns20:24
vinod1today seems to be the day to identify issues20:25
Kiall:)20:27
Kialljaycaz1: is that working for you too?20:28
jaycaz1we're making a branch with that change and the earlier change combined20:32
jaycaz1once we do that, we'll let you know20:32
timsimKiall: It's working great20:34
Kiallcool.. and on that note... I'm going to get out of here before you find something else!20:34
openstackgerritGraham Hayes proposed a change to openstack/designate: Added functionality to allow for zone ownership transfers  https://review.openstack.org/10782220:38
* mugsie runs20:38
timsimmugsie: oooh interesting.20:39
jaycaz1Kiall: in case you're still here, it all works! thanks for all your help!20:40
*** darshan104 has quit IRC20:49
*** darshan104 has joined #openstack-dns20:56
openstackgerritBetsy Luzader proposed a change to openstack/designate: Modify Recordset API  https://review.openstack.org/10289220:57
*** jaycaz1 has quit IRC21:07
*** timsim has quit IRC21:12
*** mwagner_lap has quit IRC21:43
openstackgerritA change was merged to openstack/designate: PowerDNS: Ensure each greenthread uses it's own Session instance  https://review.openstack.org/10781722:32
*** mwagner_lap has joined #openstack-dns22:35
*** betsy has quit IRC22:38
openstackgerritA change was merged to openstack/designate: Ensure eventlet monkey patching happens as early as possible  https://review.openstack.org/10780722:39
*** darshan104 has quit IRC22:41
*** mwagner_lap has quit IRC22:47
*** JensOfSweden has quit IRC22:51
*** ekarlso has quit IRC22:51
*** mugsie has quit IRC22:51
*** retr0h has quit IRC22:51
*** shufflebot has quit IRC22:51
*** uvirtbot has quit IRC22:52
*** vinod1 has quit IRC22:53
*** uvirtbot has joined #openstack-dns22:53
*** JensOfSweden has joined #openstack-dns22:53
*** ekarlso has joined #openstack-dns22:53
*** mugsie has joined #openstack-dns22:53
*** retr0h has joined #openstack-dns22:53
*** shufflebot has joined #openstack-dns22:53
*** mikal has quit IRC22:54
*** mikal has joined #openstack-dns22:54
*** soren has quit IRC22:54
*** zigo has quit IRC22:54
*** mgagne has quit IRC22:54
*** simonmcc has quit IRC22:54
*** baffle has quit IRC22:55
*** bauruine has quit IRC22:55
*** soren has joined #openstack-dns22:55
*** zigo has joined #openstack-dns22:55
*** mgagne has joined #openstack-dns22:55
*** simonmcc has joined #openstack-dns22:55
*** mikal has left #openstack-dns22:55
*** mikal has joined #openstack-dns22:55
*** baffle has joined #openstack-dns22:56
*** bauruine has joined #openstack-dns22:56
*** darshan104 has joined #openstack-dns23:11
*** darshan104 has quit IRC23:16
*** darshan104 has joined #openstack-dns23:21
*** eankutse1 has joined #openstack-dns23:51
*** eankutse has quit IRC23:52
« Wednesday, 2014-07-16 Index Friday, 2014-07-18 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!