| trungnv | arbaindu, did you solve your issue? | 00:53 |
|---|---|---|
| trungnv | arbaindu, BIND backend which you need to provide permission for them | 00:54 |
| trungnv | Stderr: u"rndc: 'addzone' failed: permission denied\n -------------- this is issue. | 00:54 |
| trungnv | arbaindu, pls perform these command: # chown bind:bind /etc/rndc.key and # chmod 644 /etc/rndc.key | 00:55 |
| arbaindu | trungnv: i fixed that issue, permissioned were wrong on /var/named | 00:56 |
| arbaindu | chown named /var/named fixed it | 00:56 |
| trungnv | yep. sure | 00:56 |
| arbaindu | but i still see issue with zones/records remainging in PENDING state for a long time, until i finally see worker displaying that WARNING above | 00:57 |
| trungnv | depend on your config for BIND then you take permission for them where they located. | 00:57 |
| arbaindu | i am eventually able to verify DNS behavior - created a test record on a test zone (pointing to some remote VM IP of mine), and on same host as i'm running bind, set /etc/resolv.conf to 127.0.0.1 | 00:58 |
| arbaindu | was able to ping and resolve it via the test.testdomain.net | 00:58 |
| trungnv | perhaps you have a lot of record on DB then worker will call mdns to check them. | 00:58 |
| arbaindu | but it took a long time for zone and record to actually become ACTIVE | 00:58 |
| trungnv | how many zone record in SQL? | 00:59 |
| arbaindu | just 1 | 00:59 |
| trungnv | Let's check with new zone? | 01:01 |
| arbaindu | I have API/central/producer on one host (controller), same as DB. | 01:01 |
| arbaindu | On another host, I have worker and mdns running. I'm running bind9/named here, on localhost (127.0.0.1) | 01:02 |
| arbaindu | just for testing purposes for now | 01:02 |
| trungnv | why did you setup all designate service on 1 Node and remain Node for BIND? | 01:03 |
| trungnv | why didn't you setup all designate service on 1 Node and remain Node for BIND? | 01:03 |
| arbaindu | eventually, only 1 host or some hosts may have access to backend DNS server | 01:03 |
| arbaindu | eg, for example a network node | 01:03 |
| arbaindu | mdns and worker need the host to have access to this subnet, right? | 01:04 |
| arbaindu | the rabbit connections between the services are working - openstack dns service list lists all of em as UP | 01:06 |
| trungnv | only 1 rabbitmq-server. is right? | 01:08 |
| arbaindu | in a neutron environment, we might have a network node that hosts SNAT for dvr, dhcp agent, legacy routers, etc... this host would have access to DNS backend | 01:08 |
| arbaindu | yea | 01:08 |
| arbaindu | here is pools.yaml: | 01:08 |
| arbaindu | where i defined a testdomain, and specified nameserver as 127.0.0.1 (since that's where bind is located on, for worker/mdns to connect to) | 01:09 |
| arbaindu | https://gist.github.com/xagent003/5874e4cc82be46fbc9c42b0e5173a76f | 01:09 |
| arbaindu | No zones currently, I just created 1: | 01:10 |
| arbaindu | openstack zone create --email dnsmaster@testdomain.net testdomain123.net. | 01:10 |
| trungnv | I saw that you setup BIND and mdns both on the same VM. right? | 01:11 |
| trungnv | masters: - host: 127.0.0.1 -port: 5354 | 01:12 |
| trungnv | this config to mdns read BIND in local. | 01:12 |
| trungnv | if you separate them then you must change this config. | 01:13 |
| arbaindu | correct - both services are running on same hypervisor (which is a VM) | 01:14 |
| trungnv | and on BIND, are you accept 53 port to LISTEN on this? | 01:14 |
| arbaindu | for now i am just attemtping to validate designate behavior | 01:14 |
| arbaindu | the bind is working as mentioned. i created a zone/record and it *eventually* gets populated, and i am able to resolve my hostname | 01:15 |
| arbaindu | for example from that same host, i set /etc/resolv.conf nameserver to 127.0.0.1 | 01:15 |
| arbaindu | then from that host i did ping www.testdomain123.net | 01:15 |
| arbaindu | for which i added a record pointing to my laptop's IP | 01:15 |
| trungnv | yep. | 01:16 |
| arbaindu | the issue is worker/mdns is taking a long time to actually add these records | 01:16 |
| arbaindu | i am tailing worker.log, and i see nothing | 01:16 |
| arbaindu | I only see following period logs in worker: | 01:17 |
| arbaindu | 2017-07-05 21:15:13.100 16262 DEBUG designate.plugin [-] Looking for driver sqlalchemy in designate.storage get_driver /usr/lib/python2.7/site-packages/designate/plugin.py:115 2017-07-05 21:15:13.102 16262 DEBUG designate.plugin [-] Loaded plugin storage:sqlalchemy __init__ /usr/lib/python2.7/site-packages/designate/plugin.py:39 2017-07-05 21:15:13.206 16262 DEBUG designate.worker.processing [req-d59833cb-1614-453d-ab60-be1ac1e | 01:17 |
| arbaindu | only after a long time, i finally seee the WARNING about PENDING for 455 seconds, then i see the DNS queries / rdnc commands being executed | 01:17 |
| trungnv | on the worker log still nothing when DNS query done? | 01:19 |
| arbaindu | no, it takes a long time for it to be done: | 01:20 |
| arbaindu | worker does nothing. then I see WARNING Found 1 zones PENDING for more than 455 seconds | 01:20 |
| arbaindu | and immediately after i see the Attempting CREATE zone testdomain123.net. | 01:20 |
| arbaindu | and Executing RNDC call | 01:20 |
| arbaindu | but worker is idle for nearly 10 minutes before that | 01:20 |
| arbaindu | https://gist.github.com/xagent003/f03230d788dd66d6f7c9ce42d859ee45 | 01:22 |
| arbaindu | worker logs ^^ | 01:22 |
| arbaindu | trungnv: notice that the actual attempts by worker to CREATE the zone, and issue the rndc calls took place at 21:19:13 | 01:23 |
| arbaindu | i created the zone at time 21:10:09 | 01:23 |
| trungnv | arbaindu, please try with new zone to check issue again? because if you separate designate services on both Node then they also have some issues with no_proxy, host name, iptable etc | 01:23 |
| arbaindu | so, it took about 9 minutes for zone to actually get populated in bind | 01:24 |
| arbaindu | creating a record now: openstack recordset create --records '10.4.165.22' --type A testdomain123.net. www | 01:24 |
| arbaindu | at time 21:24:35 | 01:25 |
| trungnv | yep | 01:25 |
| arbaindu | trungnv: i tried with new zone, this was abbout the 5th zone i created/deleted | 01:25 |
| arbaindu | worker logs are now just idle, but recordset still PENDING | 01:26 |
| trungnv | pls check log central? | 01:27 |
| arbaindu | here are logs from central: https://gist.github.com/xagent003/d096e9b1d1e1b61f4b766b6598387b55 | 01:27 |
| arbaindu | it indicates "Queueing notification for dns.recordset.create" and "Emitting dns.recordset.create notification " | 01:27 |
| arbaindu | i did notice this: | 01:28 |
| arbaindu | DEBUG designate.pool_manager.rpcapi [req-03f13197-4e9b-4d7c-8215-99d267261cea 5f280602772643e6a8a25900575ab11d c613941e66b247afac342cb5a11d3aca - - -] Calling designate.pool_manager.update_zone() over RPC wrapped /usr/lib/python2.7/site-packages/designate/loggingutils.py:24 | 01:28 |
| arbaindu | given that i have enabled service:worker, and i have producer service running as well, why is there a log from pool_manager? | 01:28 |
| arbaindu | or is it just a misnomer? | 01:28 |
| trungnv | sure. if you enable worker then pool_manger going to down. | 01:30 |
| arbaindu | correct - then why that log about pool_manager.update_zone()? | 01:30 |
| arbaindu | I was wondering if that had something to do with why my worker service is taking so long to actually create or update the zone/record | 01:30 |
| trungnv | pls show designate serivces at the moment? what are currently services? | 01:31 |
| trungnv | api - central - worker -producer -agent - mdns. right? | 01:31 |
| arbaindu | on controller node: api, central, producer | 01:31 |
| arbaindu | on a hypervisor/network node: worker, mdns | 01:31 |
| arbaindu | in /etc/designate/designate.conf (on both controller and hypervisor nodes), in the service:worker section, I have set enabled = True and notify = True | 01:32 |
| arbaindu | openstack dns service list displays all services - api, central, producer, worker, mdns - as UP | 01:32 |
| trungnv | on controller node pls check via ps -aux | grep designate. | 01:33 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/designate-dashboard master: Updated from global requirements https://review.openstack.org/480782 | 01:33 |
| trungnv | if you can, let's stop pool-manager and zone-manager then check with new zone. | 01:34 |
| arbaindu | https://gist.github.com/xagent003/d710d4808ba7755be1ad8f66c5025898 | 01:34 |
| arbaindu | pool-manager and zone-manager services are not running. i never enabled them, nor started the services, nor are they running | 01:35 |
| trungnv | did you try with service designate-pool-manager stop? | 01:36 |
| trungnv | I am not sure why pool-manger log in here. | 01:36 |
| arbaindu | can you explain exactly what the workflow is when i create a zone? | 01:39 |
| trungnv | yep. | 01:39 |
| arbaindu | i am a little unclear on what the producer does, or how the worker is supposed to get this information to issue the rndc calls to actually CREATE it | 01:40 |
| arbaindu | also when i create a recordset | 01:40 |
| trungnv | with ocata version, api - central - worker - backend driver (agent) -calling to mdns. | 01:40 |
| arbaindu | i understand what goes on between worker/mdns/the actual backend DNS via the notify and DNS transfers | 01:41 |
| arbaindu | but not the central-producer-worker communiucation | 01:41 |
| arbaindu | fyi we are using Newton | 01:41 |
| arbaindu | for example i create a record - is central supposed to send a RPC call to worker to create this in backend? Or does producer send RPC call to worker? | 01:44 |
| arbaindu | Or is it the worker that's supposed to initiate this via scanning the DB or doing some type of sync/periodic update? the docs mention for worker, "Is a generic task runner, that runs both zone create / update and deletes, and periodic tasks, from designate-producer" | 01:46 |
| trungnv | central call to worker to worker connect to backend via backend driver then worker will call mdns. | 01:47 |
| trungnv | I am not sure why you can setup designate-worker on newton package. | 01:48 |
| trungnv | Version '1:2.0.0-0ubuntu1' for 'designate-worker' was not found | 01:48 |
| *** cuongnv has joined #openstack-dns | 01:49 | |
| arbaindu | where are you seeing that was not found? | 01:52 |
| arbaindu | i am able to setup designate-worker. the services is running fine on the host | 01:53 |
| arbaindu | as mentioned, it eventually does add the zone or record. it just doesn't do it triggered by central, but only after the "WARNING designate.worker.tasks.zone [-] Found 1 zones PENDING for more than 455 seconds" | 01:54 |
| arbaindu | so the "central call to worker" seems to not be working | 01:54 |
| arbaindu | What does the producer do in this case? | 01:54 |
| trungnv | yep. your behavior is right. | 01:54 |
| trungnv | which version, 1:2 is mitaka version. from 1:3 we have worker and producer on it. | 01:57 |
| trungnv | central and worker have issue with when they communicate difference host. | 01:59 |
| trungnv | Let's check no_proxy, network on them. | 02:00 |
| openstackgerrit | Ngo Quoc Cuong proposed openstack/designate master: Remove log translations https://review.openstack.org/447825 | 02:31 |
| arbaindu | what's no_proxy and network? | 02:40 |
| arbaindu | trungnv: where are these set? | 02:40 |
| arbaindu | we havent seen any problems via RPC (using rabbitmq) and other controller-side services talking to host-side services (such as neutron-server talking to ovs-agent or l3-agent) | 02:42 |
| *** cuongnv has left #openstack-dns | 02:44 | |
| *** renmak_ has joined #openstack-dns | 02:47 | |
| *** renmak__ has joined #openstack-dns | 02:47 | |
| *** arbaindu has quit IRC | 03:06 | |
| *** Krenair has quit IRC | 05:24 | |
| *** Krenair has joined #openstack-dns | 05:31 | |
| *** therve has quit IRC | 05:44 | |
| *** therve has joined #openstack-dns | 05:46 | |
| *** baffle has quit IRC | 06:03 | |
| *** baffle has joined #openstack-dns | 06:03 | |
| *** abalutoiu has joined #openstack-dns | 07:02 | |
| *** KeithMnemonic2 has joined #openstack-dns | 07:12 | |
| *** abalutoiu_ has joined #openstack-dns | 07:12 | |
| *** KeithMnemonic1 has quit IRC | 07:15 | |
| *** abalutoiu has quit IRC | 07:16 | |
| openstackgerrit | Ngo Quoc Cuong proposed openstack/designate master: Remove log translations https://review.openstack.org/447825 | 07:31 |
| *** egonzalez has joined #openstack-dns | 07:50 | |
| *** renmak__ has quit IRC | 07:54 | |
| *** renmak_ has quit IRC | 07:54 | |
| *** MarkBaker has joined #openstack-dns | 08:03 | |
| *** cvstealt1 has joined #openstack-dns | 08:17 | |
| *** cvstealth has quit IRC | 08:18 | |
| *** abalutoiu_ is now known as abalutoiu | 08:26 | |
| *** abalutoiu has quit IRC | 09:13 | |
| *** abalutoiu has joined #openstack-dns | 09:41 | |
| *** chlong_ has joined #openstack-dns | 12:37 | |
| *** catintheroof has joined #openstack-dns | 13:25 | |
| *** catintheroof has quit IRC | 13:36 | |
| *** catintheroof has joined #openstack-dns | 13:37 | |
| *** renmak__ has joined #openstack-dns | 13:41 | |
| *** renmak_ has joined #openstack-dns | 13:41 | |
| *** MarkBaker has quit IRC | 14:35 | |
| *** catintheroof has quit IRC | 14:58 | |
| *** renmak__ has quit IRC | 14:59 | |
| *** renmak_ has quit IRC | 14:59 | |
| *** catintheroof has joined #openstack-dns | 15:02 | |
| *** renmak_ has joined #openstack-dns | 15:02 | |
| *** renmak__ has joined #openstack-dns | 15:02 | |
| *** MarkBaker has joined #openstack-dns | 15:08 | |
| *** renmak__ has quit IRC | 15:13 | |
| *** renmak_ has quit IRC | 15:13 | |
| *** renmak_ has joined #openstack-dns | 15:14 | |
| *** renmak__ has joined #openstack-dns | 15:14 | |
| *** renmak__ has quit IRC | 15:33 | |
| *** renmak_ has quit IRC | 15:33 | |
| *** renmak_ has joined #openstack-dns | 15:37 | |
| *** renmak__ has joined #openstack-dns | 15:37 | |
| *** MarkBaker has quit IRC | 15:52 | |
| *** egonzalez has quit IRC | 16:05 | |
| *** MarkBaker has joined #openstack-dns | 16:06 | |
| *** abalutoiu has quit IRC | 17:21 | |
| *** hieulq has quit IRC | 17:33 | |
| *** daidv has quit IRC | 17:33 | |
| *** trungnv has quit IRC | 17:33 | |
| *** daidv has joined #openstack-dns | 17:48 | |
| *** hieulq has joined #openstack-dns | 17:48 | |
| *** trungnv has joined #openstack-dns | 17:48 | |
| *** MarkBaker has quit IRC | 17:54 | |
| *** MarkBaker has joined #openstack-dns | 18:07 | |
| *** abalutoiu has joined #openstack-dns | 18:12 | |
| *** hieulq has quit IRC | 18:51 | |
| *** daidv has quit IRC | 18:51 | |
| *** trungnv has quit IRC | 18:51 | |
| *** cliles_ has quit IRC | 18:52 | |
| *** cliles has joined #openstack-dns | 18:52 | |
| *** daidv has joined #openstack-dns | 19:06 | |
| *** hieulq has joined #openstack-dns | 19:06 | |
| *** trungnv has joined #openstack-dns | 19:07 | |
| *** abalutoiu has quit IRC | 19:11 | |
| *** pcaruana has quit IRC | 19:12 | |
| *** renmak__ has quit IRC | 19:13 | |
| *** renmak_ has quit IRC | 19:13 | |
| *** KeithMnemonic2 has quit IRC | 19:35 | |
| *** KeithMnemonic has joined #openstack-dns | 19:36 | |
| *** renmak_ has joined #openstack-dns | 19:45 | |
| *** renmak__ has joined #openstack-dns | 19:45 | |
| *** daidv has quit IRC | 20:21 | |
| *** hieulq has quit IRC | 20:21 | |
| *** trungnv has quit IRC | 20:23 | |
| *** daidv has joined #openstack-dns | 20:36 | |
| *** trungnv has joined #openstack-dns | 20:36 | |
| *** hieulq has joined #openstack-dns | 20:37 | |
| *** renmak__ has quit IRC | 20:41 | |
| *** renmak_ has quit IRC | 20:41 | |
| *** renmak_ has joined #openstack-dns | 21:48 | |
| *** renmak__ has joined #openstack-dns | 21:48 | |
| *** hieulq has quit IRC | 21:53 | |
| *** daidv has quit IRC | 21:53 | |
| *** trungnv has quit IRC | 21:53 | |
| *** catintheroof has quit IRC | 21:55 | |
| *** trungnv has joined #openstack-dns | 22:08 | |
| *** daidv has joined #openstack-dns | 22:08 | |
| *** hieulq has joined #openstack-dns | 22:08 | |
| *** abalutoiu has joined #openstack-dns | 22:44 | |
| *** kbyrne has quit IRC | 23:14 | |
| *** daidv has quit IRC | 23:24 | |
| *** hieulq has quit IRC | 23:24 | |
| *** trungnv has quit IRC | 23:24 | |
| *** trungnv has joined #openstack-dns | 23:40 | |
| *** hieulq has joined #openstack-dns | 23:40 | |
| *** daidv has joined #openstack-dns | 23:40 | |
| -openstackstatus- NOTICE: nb03.openstack.org has been cleaned up and rebooted, and should return to building rotation | 23:43 | |
| *** renmak__ has quit IRC | 23:54 | |
| *** renmak_ has quit IRC | 23:54 | |
| *** renmak_ has joined #openstack-dns | 23:55 | |
| *** renmak__ has joined #openstack-dns | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!