| *** masber has joined #openstack-dns | 00:37 | |
| *** trungnv_ has joined #openstack-dns | 01:19 | |
| *** masber has quit IRC | 01:20 | |
| *** trungnv_ has quit IRC | 01:21 | |
| *** trungnv_ has joined #openstack-dns | 01:40 | |
| *** masber has joined #openstack-dns | 01:51 | |
| *** trungnv_ has quit IRC | 01:53 | |
| trungnv | abaindur, Is this Bug? Could you tell exactly issues which affected to you? | 02:05 |
|---|---|---|
| abaindur | trungnv: thats what im trying to determine. is it a possible race condition? | 02:06 |
| abaindur | it seems to only occur when adding 2nd or 3rd recordset to a zone, and it only happens every now and then | 02:07 |
| abaindur | (we are only adding around 3-4 records per zone in our testing) | 02:07 |
| abaindur | As mentioned, https://github.com/openstack/designate/blob/master/designate/central/service.py#1050 | 02:07 |
| abaindur | zone.action and zone.status are set to UPDATE/PENDING there. | 02:08 |
| trungnv | the worker service will determine action and status of zone. | 02:08 |
| trungnv | please see following link https://github.com/openstack/designate/blob/master/designate/worker/tasks/zone.py#L466 | 02:08 |
| abaindur | but when increment serial/self.storage.update_zone returns at end of that function, we see the returned zone has action NONE and status ACTIVE | 02:08 |
| abaindur | trungnv: this is before worker. This is in central, when it is about to send msg to worker. worker error's out because central sends update_zone RPC with action = NONE | 02:09 |
| abaindur | see here, https://github.com/openstack/designate/blob/master/designate/storage/impl_sqlalchemy/__init__.py | 02:10 |
| abaindur | go to "def update_zone" | 02:10 |
| abaindur | At beginnning of function, we even see LOG.debug("Updating zone %s", zone) | 02:10 |
| abaindur | the zone has correct values (since this was sent in from previous function in central i pointed out) | 02:11 |
| abaindur | then it looks like it does some sqlalchemy DB operations. the updated_zone returned at the end has wrong action and status | 02:11 |
| abaindur | Shouldn't the storage.update_zone set it to UPDATE/PENDING, as that's what central calls it with? | 02:15 |
| trungnv | abaindur, Could you confirm zone.action and zone.status at this line https://github.com/openstack/designate/blob/master/designate/central/service.py#L1063? | 02:21 |
| trungnv | And, some logs about this "worker error's out because central sends update_zone RPC with action = NONE". | 02:26 |
| *** abaindur has quit IRC | 02:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/designate master: Updated from global requirements https://review.openstack.org/497023 | 05:48 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/designate-dashboard master: Updated from global requirements https://review.openstack.org/497024 | 05:48 |
| *** pcaruana has joined #openstack-dns | 06:21 | |
| *** tpatzig_ has joined #openstack-dns | 06:37 | |
| *** tpatzig_ has quit IRC | 06:39 | |
| *** MarkBaker has joined #openstack-dns | 07:12 | |
| *** MarkBaker has quit IRC | 07:45 | |
| *** egonzalez has joined #openstack-dns | 07:56 | |
| *** MarkBaker has joined #openstack-dns | 08:36 | |
| *** MarkBaker has quit IRC | 08:56 | |
| *** MarkBaker has joined #openstack-dns | 09:03 | |
| frickler | timsim: my issue with designate-manage yesterday was caused by https://bugs.launchpad.net/designate/+bug/1673403, are you still working on that one? | 09:11 |
| openstack | Launchpad bug 1673403 in Designate "transport_url not being used while doing pool update" [High,Confirmed] - Assigned to Tim Simmons (timsim) | 09:11 |
| *** abalutoiu has joined #openstack-dns | 10:16 | |
| *** abalutoiu has quit IRC | 10:21 | |
| *** egonzalez has quit IRC | 10:30 | |
| *** abalutoiu has joined #openstack-dns | 10:41 | |
| *** egonzalez has joined #openstack-dns | 10:51 | |
| *** egonzalez has quit IRC | 11:07 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/designate-dashboard master: Imported Translations from Zanata https://review.openstack.org/497200 | 11:09 |
| *** egonzalez has joined #openstack-dns | 11:18 | |
| *** abalutoiu has quit IRC | 11:38 | |
| *** abalutoiu has joined #openstack-dns | 11:40 | |
| *** abalutoiu has quit IRC | 12:01 | |
| *** abalutoiu has joined #openstack-dns | 12:05 | |
| *** MarkBaker has quit IRC | 12:13 | |
| *** abalutoiu has quit IRC | 12:14 | |
| *** MarkBaker has joined #openstack-dns | 12:14 | |
| *** brandor5 has joined #openstack-dns | 12:24 | |
| brandor5 | Hey guys, I was wondering if anyone has a working HA deployment of designate using tripleo? | 12:24 |
| *** abalutoiu has joined #openstack-dns | 12:31 | |
| egonzalez | brandor5, with tripleo nope, might be worth asking in tripleo's channel, #tripleo | 12:37 |
| brandor5 | egonzalex: ok thanks. I'm working my way through the channels :) appreciate the response | 12:38 |
| brandor5 | bleh sorry, egonzalez | 12:41 |
| *** catintheroof has joined #openstack-dns | 13:02 | |
| *** brandor5 has left #openstack-dns | 13:08 | |
| openstackgerrit | Anusree A proposed openstack/designate-dashboard master: AngularJS unittest with Jasmine and Karma https://review.openstack.org/441700 | 13:19 |
| openstackgerrit | Merged openstack/designate-dashboard master: Imported Translations from Zanata https://review.openstack.org/497200 | 13:19 |
| openstackgerrit | Anusree A proposed openstack/designate-dashboard master: AngularJS unittest with Jasmine and Karma https://review.openstack.org/441700 | 13:26 |
| openstackgerrit | Merged openstack/designate-dashboard master: Updated from global requirements https://review.openstack.org/497024 | 13:27 |
| *** abalutoiu has quit IRC | 13:40 | |
| *** nkinder has joined #openstack-dns | 13:40 | |
| *** trungnv_ has joined #openstack-dns | 14:59 | |
| *** abalutoiu has joined #openstack-dns | 15:06 | |
| *** abalutoiu has quit IRC | 15:14 | |
| *** trungnv_ has quit IRC | 15:25 | |
| *** pcaruana has quit IRC | 15:57 | |
| *** ianychoi has joined #openstack-dns | 16:00 | |
| *** egonzalez has quit IRC | 16:01 | |
| *** openstackgerrit has quit IRC | 16:04 | |
| *** abalutoiu has joined #openstack-dns | 16:06 | |
| *** abalutoiu_ has joined #openstack-dns | 16:08 | |
| *** kbyrne has quit IRC | 16:09 | |
| *** kbyrne has joined #openstack-dns | 16:10 | |
| *** abalutoiu has quit IRC | 16:12 | |
| *** MarkBaker has quit IRC | 17:07 | |
| *** MarkBaker has joined #openstack-dns | 17:07 | |
| *** MarkBaker has quit IRC | 17:20 | |
| *** catinthe_ has joined #openstack-dns | 17:44 | |
| *** catintheroof has quit IRC | 17:45 | |
| *** egonzalez has joined #openstack-dns | 18:31 | |
| *** masber has quit IRC | 19:00 | |
| *** abalutoiu_ has quit IRC | 19:28 | |
| *** egonzalez has quit IRC | 19:41 | |
| *** abalutoiu_ has joined #openstack-dns | 20:26 | |
| *** catinthe_ has quit IRC | 20:53 | |
| *** catintheroof has joined #openstack-dns | 20:54 | |
| *** catintheroof has quit IRC | 20:58 | |
| *** abalutoiu_ has quit IRC | 21:09 | |
| *** MarkBaker has joined #openstack-dns | 21:48 | |
| *** catintheroof has joined #openstack-dns | 21:59 | |
| *** MarkBaker has quit IRC | 22:04 | |
| *** abaindur has joined #openstack-dns | 22:49 | |
| abaindur | trungnv: Yes, as I mentioned, the zone that is returned at the bottom of that function has action=NONE, status=ACTIVE | 22:50 |
| abaindur | https://github.com/openstack/designate/blob/stable/newton/designate/central/service.py#L1065 | 22:50 |
| abaindur | even though at the top of that function, before storage.update_zone is called, zone.action and zone.status are explicitly set to UPDATE and PENDING | 22:51 |
| abaindur | so somehow the sqlalchemy storage driver is not updating zone to UPDATE/PENDING | 22:51 |
| abaindur | the returned zone there is what central passing into worker via update_zone RPC. but worker then errors out because action is NONE | 22:52 |
| abaindur | I suspect some kind of race condition here, as it only happens some of the time | 22:52 |
| abaindur | The 1st recorset was created fine. After which zone moves from UPDATE/PENDING to NONE/ACTIVE. But it seems to be 2nd or 3rd recordset in a zone which faces this issue | 22:53 |
| abaindur | Here we see this debug: https://github.com/openstack/designate/blob/stable/newton/designate/storage/impl_sqlalchemy/__init__.py#L309 | 22:55 |
| abaindur | which also indicates the storage's update_zone was called with UPDATE/PENDING. but final updated_zone returned there has NONE/ACTIVE. so issue lies 100% in there | 22:55 |
| *** catintheroof has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!