| openstackgerrit | Ron Rickard proposed openstack/designate: Pool Manager https://review.openstack.org/131656 | 00:13 |
|---|---|---|
| Kiall | lol.. I was just starting devstack with the previous patchset -_- | 00:14 |
| rjrjr | all the bugs I am aware of have been fixed (retry_interval in notify_zone_changed; added delay, records are deleted properly; NS/SOA records have proper action,status,serial) | 00:15 |
| rjrjr | i'll monitor Gerrit and this chat periodically this weekend. | 00:17 |
| Kiall | K.. Started devstack back up with the latest version of the patchset | 00:18 |
| rjrjr | mugsie, isn't ireland small enough that you can walk the length of the country in a day? 8^) (kidding) | 00:23 |
| Kiall | boom - devstack exercices make it splode ;) | 00:23 |
| Kiall | http://paste.openstack.org/show/139158/ | 00:23 |
| Kiall | looks like it was a simple fix .. | 00:28 |
| rjrjr | i'll fix it, but it's strange it works for me. | 00:28 |
| rjrjr | give me one sec. | 00:28 |
| Kiall | http://paste.openstack.org/show/139162/ | 00:29 |
| Kiall | ^ diff to fix it | 00:29 |
| Kiall | mimics what you did in the update_record_status method | 00:29 |
| *** rmoe has quit IRC | 00:31 | |
| openstackgerrit | Ron Rickard proposed openstack/designate: Pool Manager https://review.openstack.org/131656 | 00:32 |
| rjrjr | i've been running this code on devstack myself. regardless, fix is in. | 00:32 |
| Kiall | rjrjr: I just ran the exercises/designate.sh script and spotted it :) | 00:32 |
| rjrjr | what is that? | 00:33 |
| Kiall | It just creates/deletes some domains and records etc | 00:33 |
| Kiall | i.e. quick sanity test for correct behaviour | 00:33 |
| rjrjr | hmmm.... i'll have to try that. | 00:34 |
| Kiall | It seems to be picking up a few issues, others could be issues in the script, not 100% sure yet | 00:35 |
| rjrjr | i was unaware of the script. i've been running ./stack.sh and testing manually. | 00:35 |
| Kiall | Oh .. Heh :) | 00:35 |
| Kiall | https://github.com/openstack/designate/blob/master/contrib/devstack/exercises/designate.sh | 00:36 |
| Kiall | It just makes a bunch of API calls via the client, does DNS lookups to be sure the action happend eyc | 00:36 |
| Kiall | etc | 00:36 |
| rjrjr | i'm starting up devstack now and will run the script. | 00:37 |
| Kiall | Humm.. Trying to trace where the 127.0.0.1:53 is coming from in this.. | 00:40 |
| Kiall | 2014-11-27 00:33:25.617 WARNING designate.mdns.notify [req-6d4f0d20-3080-44ce-bd95-d1854b5bac7b 9452ab847bda4498bc2ba9320dd3e543 38fc10befe4042399ea1fe9e89d95d3a] Got Timeout while trying to send 'SOA' for 'test-1.com.' to '127.0.0.1:53'. Timeout='30' seconds. Retry='3' | 00:40 |
| Kiall | (My devstack will have that on 192.168.x.xx:5354 | 00:41 |
| rjrjr | exercises/designate.sh completed with "SUCCESS" | 00:41 |
| Kiall | Yea, I've got weirdness that needs an extra setting configure, those seem to be my remaining fails :) | 00:42 |
| rjrjr | i'll post my localrc. just a second. | 00:42 |
| rjrjr | http://paste.openstack.org/show/139173/ | 00:43 |
| rjrjr | i believe your issue might be the HOST_IP setting missing in localrc | 00:43 |
| rjrjr | unfortunately, bind is not listening on 127.0.0.1:53 with the current configuration. | 00:44 |
| Kiall | In theory it should work without that.. (or if it doesn't, it's a bug in our devstack stuff :)) | 00:45 |
| rjrjr | it has never worked for me. | 00:45 |
| rjrjr | 8^( | 00:45 |
| rjrjr | i'll add it to my list of things to look at/fix. | 00:45 |
| *** rmoe has joined #openstack-dns | 00:46 | |
| rjrjr | i always use the BIND backend driver. it might work with the other backend drivers. | 00:46 |
| Kiall | with this: http://paste.openstack.org/show/139174/ | 00:46 |
| Kiall | I'm using the bind9 pool driver :) | 00:46 |
| Kiall | What options are deciding the destinations to send NOTIFY etc's to> | 00:47 |
| Kiall | to?* | 00:47 |
| Kiall | I'm thinking that's defaulting to 127.0.0.1:53 or something, somehwere | 00:47 |
| rjrjr | port/host in server specific configurtion section. | 00:47 |
| rjrjr | kiall, run a 'netstat -an'. is your host listening on port 127.0.0.1:53? | 00:48 |
| rjrjr | the devstack doesn't setup BIND to listen on that port. | 00:48 |
| rjrjr | sorry, host/port. | 00:48 |
| rjrjr | trying again: devstack doesn't setup BIND to listen on that host/port. | 00:48 |
| rjrjr | hence why i always set HOST_IP in my localrc. | 00:49 |
| Kiall | It's not, hence trying to figure out where the 127.0.0.1 is coming from :) | 00:49 |
| Kiall | http://paste.openstack.org/show/139175/ | 00:49 |
| Kiall | (In the gate, we're unable to have BIND etc on *:53) | 00:49 |
| rjrjr | gotcha. no 127.0.0.1 in designate.conf at all? | 00:50 |
| Kiall | Nope, But I think I see it | 00:50 |
| Kiall | backend/base.py now has: | 00:50 |
| Kiall | server_opts.append(cfg.StrOpt('host', default='127.0.0.1', | 00:50 |
| Kiall | help='Server Host')) | 00:50 |
| Kiall | server_opts.append(cfg.IntOpt('port', default=53, help='Server Port')) | 00:50 |
| rjrjr | correct, but your designate.conf will override that. | 00:51 |
| Kiall | devstack didn't do that for me it seems | 00:51 |
| Kiall | or.. Humm | 00:51 |
| Kiall | [backend:bind9_pool:f26e0b32-736f-4f0a-831b-039a415c481e] | 00:51 |
| Kiall | port = 53 | 00:51 |
| Kiall | host = 192.168.27.100 | 00:51 |
| * Kiall is confused.. ;) | 00:53 | |
| rjrjr | grep 127.0.0.1 from *pool-manager.log. do you have any logs for that IP in pool manager? | 00:57 |
| Kiall | Yep - Think I have it (again): | 00:58 |
| Kiall | [service:pool_manager] | 00:58 |
| Kiall | pool_id = 794ccc2c-d751-44fe-b57f-8894c9f5c842 | 00:58 |
| Kiall | vs | 00:58 |
| Kiall | [backend:bind9_pool:f26e0b32-736f-4f0a-831b-039a415c481e] | 00:59 |
| Kiall | port = 53 | 00:59 |
| Kiall | host = 192.168.27.100 | 00:59 |
| Kiall | the UUID's don't line up.. and I can't find anything in the DB with f26e0b32-736f-4f0a-831b-039a415c481e | 00:59 |
| rjrjr | pool_id is not the same a server_id. | 00:59 |
| rjrjr | the f26e... is a server ID. | 00:59 |
| Kiall | Ah, But, doesn't exist in the database? | 00:59 |
| rjrjr | no, servers are only in the config file per our mid-cycle meeting in Seattle. | 01:00 |
| rjrjr | kiall 'grep 127.0.0.1 *pool-manager.log' and pasteit. | 01:00 |
| Kiall | It's about 50k lines matching 127.0.0.1 ;) | 01:01 |
| rjrjr | ouch. | 01:01 |
| rjrjr | i'm just looking for the startup configuration information. | 01:01 |
| rjrjr | the part where it prints what it found in the config file. | 01:01 |
| rjrjr | like so: http://paste.openstack.org/show/139185/ | 01:03 |
| Kiall | http://paste.openstack.org/show/idvexynqqxUK7Z369BG7/ | 01:07 |
| rjrjr | backend:bind9_pool:6a5032b6-2d96-43ee-b25b-7d784e2bf3b2.host = 127.0.0.1 | 01:08 |
| rjrjr | from your paste. | 01:09 |
| Kiall | Yea, I see that .. but.. | 01:09 |
| Kiall | vagrant@vagrant:~/devstack$ grep -r '6a5032b6-2d96-43ee-b25b-7d784e2bf3b2' /etc/designate/ | 01:09 |
| Kiall | vagrant@vagrant:~/devstack$ | 01:09 |
| Kiall | lol .. | 01:09 |
| rjrjr | maybe the designate.conf is somewhere else? | 01:09 |
| Kiall | /opt/stack/designate/etc/designate/designate.conf.sample:[backend:bind9_pool:6a5032b6-2d96-43ee-b25b-7d784e2bf3b2] | 01:09 |
| Kiall | /opt/stack/designate/etc/designate/designate.conf:[backend:bind9_pool:6a5032b6-2d96-43ee-b25b-7d784e2bf3b2] | 01:09 |
| Kiall | -_- | 01:10 |
| rjrjr | which one is being used? | 01:10 |
| Kiall | It seems both were loaded :) | 01:10 |
| rjrjr | strange. | 01:11 |
| rjrjr | i don't have a /opt/stack/etc/designate/designate.conf | 01:11 |
| Kiall | Well, we do support multiple config files... | 01:11 |
| *** ekarlso- has quit IRC | 01:12 | |
| Kiall | (I've removed it BTW, rerunning) | 01:12 |
| *** yfujioka has joined #openstack-dns | 01:12 | |
| rjrjr | was it created during the devstack process? | 01:12 |
| Kiall | Nah, from testing outside of devstack | 01:12 |
| rjrjr | okay. | 01:13 |
| *** ekarlso- has joined #openstack-dns | 01:13 | |
| Kiall | New error though .. http://paste.openstack.org/show/b1cAKrMjJZ3W9tYop3cU/ could be something dud leftover in my env causing it though | 01:13 |
| *** EricGonczer_ has joined #openstack-dns | 01:14 | |
| rjrjr | now you are making me feel self conscious. | 01:15 |
| Kiall | Yea, It looks like there might be a race happening.. it delete's a record, then deletes the zone, then queries DNS for it (and finds it's still present) | 01:16 |
| *** vinod has joined #openstack-dns | 01:16 | |
| Kiall | the delete record is still in-process, meanwhile the delete zone is actioned in storage and sent to pool mgr.. | 01:16 |
| Kiall | I think | 01:17 |
| Kiall | I'm betting my slow VM is highlighting it ;) | 01:17 |
| rjrjr | you mean fast VM. mine is pretty slow. | 01:17 |
| Kiall | It's a VM - There ALL slow.. ;) | 01:17 |
| Kiall | (Never tell my employer I said that.. lol) | 01:18 |
| rjrjr | i'm still not following. | 01:18 |
| rjrjr | the deletes only occur in update_status. | 01:18 |
| Kiall | Oh, then.. Humm | 01:19 |
| rjrjr | granted, i haven't tried any rapid fire testing, but i'm adding that to my list. | 01:20 |
| Kiall | add the "rally" service to your devstack, then `rally task start /opt/stack/designate/rally-jobs/designate-designate.yaml` | 01:20 |
| Kiall | It'll pound it ;) | 01:21 |
| Kiall | (Also note that - It fails even without pool manager right now due to some SQL lock contention.. But it'll likey find other new stuff aswell) | 01:21 |
| rjrjr | i'm concerned. so, exercises/designate.sh is failing for you. | 01:22 |
| rjrjr | i've tried it 3 times and it passed every time. | 01:22 |
| Kiall | I'm just rerunning devstack again, without the extra config file.. Will try once it's one | 01:22 |
| Kiall | once it's done* | 01:23 |
| rjrjr | okay. let me know. | 01:23 |
| rjrjr | thinking about it, i can see adding more checks in the update_status if the domain is delete before a status comes in to update it. | 01:23 |
| rjrjr | sorry, been up over 40 hours now. | 01:24 |
| Kiall | Ouch. Go to bed. | 01:24 |
| Kiall | (And - Yea, it's 01:25 here.. Still in the office -_-) | 01:24 |
| rjrjr | what i meant was, i can see a situation where a problem occurs where the users deletes a record, then deletes a domain. the delete domain update_status fires first, then the update_status for the record. | 01:25 |
| rjrjr | i can add a check in the code for that. | 01:25 |
| rjrjr | only execute the status update for a domain if the domain exists ('if domain:') | 01:26 |
| rjrjr | i'll have to play some more. | 01:26 |
| rjrjr | i want to make sure your exercises/designate.sh works before i take off though. 8^) | 01:27 |
| Kiall | Or, use the "show_deleted = True" property on the context object to load it anyway.. If it makes sense to update it agaian | 01:28 |
| Kiall | I'm not sure I'll be around that long ;) | 01:28 |
| rjrjr | it doesn't. | 01:28 |
| *** jmcbride1 has quit IRC | 01:28 | |
| rjrjr | i thought you were restarting devstack? | 01:28 |
| Kiall | exercise is running again | 01:29 |
| rjrjr | <fingers crossed> | 01:29 |
| Kiall | <toes crossed> | 01:29 |
| rjrjr | <eyes crossed> | 01:30 |
| Kiall | Failing much earlier now, but now logs as to why | 01:31 |
| * Kiall is likely causing the issues, too tied! | 01:32 | |
| Kiall | I'll take another look in the morning and see what's up :) | 01:32 |
| rjrjr | okay. get some sleep and we'll see where we are at in the morning. i'll login around 10:00AM MST and check to see if you made progress. | 01:33 |
| *** ryanpetrello has joined #openstack-dns | 01:33 | |
| rjrjr | sorry you are having issues. 8^( | 01:33 |
| *** EricGonczer_ has quit IRC | 01:33 | |
| rjrjr | i'll look to fix the BIND9 127.0.0.1/53 issue this weekend. | 01:33 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: TEST: Switch Gate to PoolManager + Bind9 https://review.openstack.org/137496 | 01:34 |
| Kiall | rjrjr: Let's see if the gate agrees with your or my tests ;) | 01:35 |
| rjrjr | okay. | 01:35 |
| rjrjr | ummmm... HOST_IP? | 01:36 |
| *** vinod has joined #openstack-dns | 01:37 | |
| vinod | I don't think the gate starts the pool manager or changes the backends appropriately for pool manager to kick in | 01:38 |
| Kiall | rjrjr: we can't, every job in the gate is ran on a new VM with a new IP | 01:38 |
| Kiall | vinod: It should, with this change: https://review.openstack.org/137496 | 01:38 |
| rjrjr | ignore that. shouldn't be required (i'm not seeing it in any of the designate devstack scripts.) | 01:38 |
| Kiall | vinod: I added another patch on top of rjrjr's to enable pool manager instead of powerdns | 01:38 |
| vinod | Ahh. Okay | 01:39 |
| vinod | How about changing the central's backend to pool_manager_proxy? Is that the default? | 01:40 |
| Kiall | I think it'll work anyway.. The gate is filled with confusing series of bash scripts spread between Jenkins jobs, devstack, designate and the devstack-gate repos.. heh .. hard to tell sometimes ;) | 01:40 |
| Kiall | our devstack code has: | 01:40 |
| Kiall | elif is_service_enabled designate-pool-manager; then | 01:40 |
| Kiall | iniset $DESIGNATE_CONF service:central backend_driver pool_manager_proxy | 01:40 |
| Kiall | Job started up.. https://jenkins03.openstack.org/job/gate-designate-devstack-dsvm/196/consoleFull | 01:41 |
| vinod | i just saw that | 01:41 |
| * Kiall twiddles his thumbs while the job runs... | 01:48 | |
| rjrjr | 8 minutes and counting... | 01:49 |
| Kiall | It went boom - but likely for a whole other reason | 01:56 |
| Kiall | I see it.. | 01:56 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: TEST: Switch Gate to PoolManager + Bind9 https://review.openstack.org/137496 | 01:56 |
| Kiall | new run @ https://jenkins07.openstack.org/job/gate-designate-devstack-dsvm/226/consoleFull | 01:58 |
| Kiall | Will check the results when I get home ;) | 01:58 |
| rjrjr | okay. i'll keep watch. talk to you in the afternoon. | 01:59 |
| Kiall | rjrjr: rally is running against it too in another VM BTW .. Logs for that will end up @ http://logs.openstack.org/96/137496/2/check/gate-rally-dsvm-designate-designate/f94ba0e/ | 02:00 |
| Kiall | Cya :) | 02:03 |
| *** nosnos has joined #openstack-dns | 02:11 | |
| *** ryanpetrello has quit IRC | 02:12 | |
| openstackgerrit | Merged openstack/python-designateclient: Refactor KS session creation and support CA certs https://review.openstack.org/137479 | 02:28 |
| Kiall | rjrjr: so the error from the latest run is what I was seeing on my last devstack run | 02:45 |
| Kiall | ah, I see some warnings in the logs: | 02:46 |
| Kiall | 2014-11-27 02:09:39.528 29316 WARNING designate.mdns.notify [req-026e6ef0-ac89-4f17-889f-fe5ee42fe0b6 fef3a392256e42e5af9e0feb31fa1a3e 0f89081244dc4ed88ea6c5f21a8b830a - - -] Failed to get expected response while trying to send 'NOTIFY' for 'exercise-0cdcf9a4.com.' to '127.0.0.1:53'. | 02:46 |
| Kiall | 127.0.0.1 came back to haunt us | 02:47 |
| Kiall | or, maybe it's the :53 .. | 02:49 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: TEST: Switch Gate to PoolManager + Bind9 https://review.openstack.org/137496 | 02:50 |
| Kiall | I've included a devstack change in there that fixes a hardcoded port 53 to use the right variable.. we'll see if that sorts it | 02:50 |
| *** GonZoPT has quit IRC | 02:53 | |
| *** vinod has quit IRC | 03:10 | |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: TEST: Switch Gate to PoolManager + Bind9 https://review.openstack.org/137496 | 03:22 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Update gate_hook to support multiple drivers https://review.openstack.org/137505 | 03:32 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: Update gate_hook to support multiple drivers https://review.openstack.org/137505 | 03:53 |
| *** nosnos has quit IRC | 04:00 | |
| rjrjr | so, not working because it is trying to dig @127.0.0.1 correct? | 04:25 |
| rjrjr | and BIND9 is not listening on 127.0.0.1. | 04:26 |
| rjrjr | i can see it is failing to create the domain using rndc on 127.0.0.1:953. the bind9 driver (sans pool manager) will fail with the same problem. i'll look at this issue on Friday. | 04:37 |
| rjrjr | just need to get BIND9 listening on 127.0.0.1. | 04:38 |
| *** nosnos has joined #openstack-dns | 04:49 | |
| *** jmcbride has joined #openstack-dns | 05:08 | |
| *** jmcbride has quit IRC | 05:39 | |
| *** ryanpetrello has joined #openstack-dns | 05:45 | |
| *** vipul has quit IRC | 06:02 | |
| *** vipul has joined #openstack-dns | 06:04 | |
| *** ryanpetrello has quit IRC | 06:11 | |
| *** k4n0 has joined #openstack-dns | 06:26 | |
| *** nihilifer has joined #openstack-dns | 06:35 | |
| *** ekarlso- has quit IRC | 06:54 | |
| *** ekarlso- has joined #openstack-dns | 06:54 | |
| ekarlso- | rjrjr: morning :D | 06:54 |
| ekarlso- | betsy: you around mam ? :) | 07:35 |
| * ekarlso- hope he used the right title :) | 07:35 | |
| * ekarlso- dislikes migration 039 atm that betsy added to add SOA / NS records for domains :'( | 07:38 | |
| ekarlso- | taking like light years to complete on a sample of the HP production db | 07:38 |
| *** ToMiles has quit IRC | 08:20 | |
| * ekarlso- realized that he forgot to purge old domains :P | 08:40 | |
| *** nosnos has quit IRC | 08:49 | |
| *** jordanP has joined #openstack-dns | 08:51 | |
| *** nosnos has joined #openstack-dns | 09:39 | |
| *** rediskin has joined #openstack-dns | 11:24 | |
| *** jmcbride has joined #openstack-dns | 11:30 | |
| *** jmcbride has quit IRC | 11:53 | |
| *** ryanpetrello has joined #openstack-dns | 11:54 | |
| *** ryanpetrello has quit IRC | 12:06 | |
| *** k4n0 has quit IRC | 12:26 | |
| *** nosnos has quit IRC | 13:01 | |
| *** yfujioka has quit IRC | 13:10 | |
| *** betsy has quit IRC | 13:22 | |
| *** GonZoPT has joined #openstack-dns | 13:27 | |
| *** GonZoPT has quit IRC | 13:33 | |
| Kiall | rjrjr: there are env vars for all the IPs to use, we need to make sure those are all used in the right places etc etc | 13:44 |
| Kiall | (and ports) | 13:45 |
| Kiall | e.g. we can't assume or force it to be on 127.0.0.1 or port 53/953 | 13:45 |
| Kiall | rjrjr: also, https://review.openstack.org/#/c/137505/ and https://review.openstack.org/#/c/137506/ | 13:48 |
| Kiall | Adding gate jobs to run BIND9+Pool (non-voting to start) | 13:48 |
| *** rjrjr has quit IRC | 15:27 | |
| *** nihilifer has quit IRC | 15:52 | |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: TEST: Switch Gate to PoolManager + Bind9 https://review.openstack.org/137496 | 16:34 |
| openstackgerrit | Kiall Mac Innes proposed openstack/designate: TEST: Switch Gate to PoolManager + Bind9 https://review.openstack.org/137496 | 16:37 |
| *** nihilifer has joined #openstack-dns | 16:50 | |
| *** jordanP has quit IRC | 18:12 | |
| *** mugsie has quit IRC | 18:40 | |
| *** mugsie has joined #openstack-dns | 18:41 | |
| *** nihilifer has quit IRC | 19:09 | |
| *** GonZo2K has joined #openstack-dns | 20:16 | |
| *** EricGonczer_ has joined #openstack-dns | 20:23 | |
| *** EricGonczer_ has quit IRC | 20:23 | |
| *** rediskin has left #openstack-dns | 23:26 | |
| *** jmcbride has joined #openstack-dns | 23:39 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!