Tuesday, 2021-03-16

*** ianychoi_ has joined #openstack-dns01:03
*** ianychoi has quit IRC01:12
*** michchap has quit IRC01:12
*** hamalq has quit IRC01:23
*** kd has joined #openstack-dns03:03
*** k-s-dean has quit IRC03:04
*** k-s-dean has joined #openstack-dns03:06
*** kd has quit IRC03:08
*** k-s-dean has quit IRC03:18
openstackgerritMerged openstack/designate-tempest-plugin master: Testing "Lists all recordsets owned by a project in Designate" API  https://review.opendev.org/c/openstack/designate-tempest-plugin/+/77823205:52
*** icey_ has quit IRC07:00
*** icey has joined #openstack-dns07:06
fricklerthis is an invalid configuration "< k-s-dean> extension_drivers = port_security,dns,dns_domain_ports"07:11
fricklerdns_domain_ports includes the dns extension, specifying both leads to internal errors07:12
*** k-s-dean has joined #openstack-dns08:28
*** zigo has joined #openstack-dns08:29
*** k-s-dean has quit IRC09:03
*** k-s-dean has joined #openstack-dns09:03
openstackgerritArkady Shtempler proposed openstack/designate-tempest-plugin master: New API test - test_list_service_statuses  https://review.opendev.org/c/openstack/designate-tempest-plugin/+/78056711:35
*** michchap has joined #openstack-dns14:22
*** jobewan has joined #openstack-dns14:22
openstackgerritArkady Shtempler proposed openstack/designate-tempest-plugin master: Testing "API test for Get Project Limits"  https://review.opendev.org/c/openstack/designate-tempest-plugin/+/78087114:56
*** hamalq has joined #openstack-dns16:48
hamalqhi can anyone give +1 on https://review.opendev.org/c/openstack/designate/+/755379/16:49
*** lbragstad has quit IRC18:07
k-s-deanjohnsom, im about if your available18:15
k-s-deanif not no worries18:16
johnsomk-s-dean Hi, so still having trouble?18:16
k-s-deanYeah, I asked the neutron guys today and they said create a LP ticket. haha18:17
k-s-deanso far I've just disabled fixedIP in the designate conf so that the entries aren't lodged in the external DNS.18:17
johnsomOk, so my plan here is to have a discussion to work through a debug scenario. Is that ok? It may take a bit of time18:18
k-s-deanThats fine with me.18:19
johnsomNice. Did you happen to try the proposed solution from Jens?18:19
k-s-deanI may have missed that18:20
johnsomHis proposal was to not set both dns and dns_domain_ports in the extension_drivers setting for neutron.18:21
k-s-deanso dont set either ?18:21
johnsomhttps://www.irccloud.com/pastebin/E2YmgymG/18:21
k-s-deanof them*18:21
johnsomI think he was recommending to just set dns_domain_ports and not dns as well18:22
k-s-deanright ok.18:22
johnsomThere is probably a bug in there that we should fix so it doesn't cause internal errors, or at least makes them obvious. But that is a side issue.18:23
k-s-deanhold on. I re-deployed last night after reading the neutron docs thoroughly18:23
johnsomOk, no porblem.18:23
k-s-deanright my config is like so at the moment18:24
k-s-dean[ml2]18:24
k-s-deantype_drivers = flat,vlan,vxlan18:24
k-s-deantenant_network_types = vxlan18:24
k-s-deanmechanism_drivers = linuxbridge,l2population18:24
k-s-deanextension_drivers = port_security,dns_domain_ports18:24
johnsomOk, that looks good.18:25
k-s-deanok, let me unset the modifications I made to the designate config.18:25
johnsomOk18:25
*** lbragstad has joined #openstack-dns18:27
k-s-deanQuick question I currently have an LDAP backend configured. it shouldn't matter which domain I create the dns domain in should it ?18:27
k-s-deane.g. i can create in either the default or my ldap backend18:27
johnsomIs the LDAP for keystone or a Designate backend driver?18:29
k-s-deankeystone, its simply just a read only copy of my LDAP directory.18:30
k-s-deanall the service endpoints are registered in the default domain18:31
k-s-deananyway it should be fine. I've seen the VMs in my ldap domain being registered in DNS18:31
k-s-deanso I'll create the zone in the default domain18:32
k-s-deanright kolla reconfiguring, with the zone I've just created18:32
johnsomOk, let's collect some basic information. If any of the data is confidential, change it or PM me. Just be sure to change it consistently.18:34
johnsomOtherwise, I will get confused. lol18:34
johnsomDo you have the VM booted in nova  you are wanting to attach the floating IP to?18:35
k-s-deanNo problem, time for some regex18:36
k-s-deanNot yet, but I can create one18:36
johnsomLet's do that.18:36
k-s-deankollas almost done reconfiguring18:36
k-s-deantwo seconds18:36
johnsomNo worries, I am going to go fill my water18:36
johnsomback18:39
k-s-deanAwesome.18:39
k-s-deanright.18:39
johnsomSo, once that instance is booted, the first thing I would like to see is "openstack network show <instance net>"18:40
k-s-deanI've set the dns-domain on the public network.18:40
johnsomFollowed by the instance subnet18:40
k-s-deantwo seconds let me get paste.18:40
johnsomGreat. paste.openstack.org is handy for this18:41
k-s-deanhttp://paste.openstack.org/show/803628/18:41
k-s-deansubnet18:42
k-s-deanhttp://paste.openstack.org/show/803629/18:42
k-s-deanok now I'm going to go and boot an instance.18:42
johnsomOk18:42
johnsomLooks good so far18:43
johnsomAre you planning to use "public" for the floating IP as well?18:43
k-s-deanyes public is the floating IP network18:43
k-s-deanwould you like to see the private networks18:43
johnsomWhich network will have the instance port on it?18:44
johnsomIf private, yes please18:44
k-s-deanok so on instance boot the private network vxlan network will be the initial network18:45
johnsomOk, no problem18:45
k-s-deanhttp://paste.openstack.org/show/803630/18:45
k-s-deanthats is DNS before instance boot18:45
johnsomperfect18:46
k-s-dean http://paste.openstack.org/show/803631/18:47
k-s-deanprivate network18:47
k-s-deannext I'll boot the instance.18:47
johnsomOk, I see one issue. The private network does not have a dns_domain set18:48
k-s-deanok. didn't think that would be an issue. thought it needed to be set on the public.18:49
k-s-deanbugger. I'm being called for my dinner.18:50
johnsomWell, I think we may need to create the floating IP differently in that case.18:50
johnsomOk, do we need to stop for your dinner?18:51
johnsomMy tomorrow morning is busy again, until 10am my time. I could ping you tomorrow if you would like18:51
k-s-deanyeah unfortunately, I'll be about 30 minutes. but before I go.18:52
johnsomWednesday is my meeting day. lol18:52
k-s-deanthe instance has now booted.18:52
johnsomOk, a show on that would be good as well18:52
johnsomI can save my place. lol18:52
k-s-deanhttp://paste.openstack.org/show/803632/18:53
k-s-deanthere is the paste with the VXLAN IPs in DNS18:53
k-s-deanbefore I've attached a floating IP18:53
johnsomOk18:54
k-s-deanI'll ping you again once Ive had dinner my folks are looking at me like what you doing.18:54
k-s-deanif you available great if not no worries.18:54
johnsomOk, no problem18:54
johnsomI will be around18:54
k-s-deanjohnsom, Back19:23
johnsomk-s-dean Hi19:24
k-s-deanyou still available  :)19:25
johnsomYes19:25
k-s-deanawesome. Ok, some from here do you want me to attach the floating IP19:25
johnsomSo, I am a bit surprised at your last zone show. The network for that instance does not have a dns_domain set, but you have three addresses assigned.19:26
k-s-deanYeah, thats what has been confusing me.19:26
k-s-deanI'm not expecting those entries to be added at all.19:26
johnsomThat is a bit unexpected. Are you running the Designate Sink process?19:27
k-s-deanyes.19:27
johnsomAnd you are on which version of Openstack ?19:27
k-s-deanvictoria.19:27
k-s-deanbuilt from source.19:27
k-s-deandeployed with kolla.19:27
johnsomHmm, ok. Let's keep going, but I will note that as unexpected.  It may be because of the sink, which isn't used for nova/neutron integration anymore.19:27
k-s-deanohhh. ok19:28
johnsomYeah, I have zero experience with Kolla, so I'm not going to be much help there.19:28
johnsomCan you provide a "openstack server show"?19:28
k-s-deanyeah thats fine. I'm pretty versed with ansible. and i know how kolla works19:28
k-s-deansure19:28
johnsomYeah, kolla is just one of the deployment tools I haven't ever needed to use.19:29
k-s-deanhttp://paste.openstack.org/show/803635/19:29
johnsomOk, that looks fine. Now the neutron port "openstack port list --device_id cfceceb3-3b84-41e7-a071-093305b364cd" and "openstack port show <port id>" I only need the port show output19:32
k-s-deanhttp://paste.openstack.org/show/803636/19:35
johnsomThanks19:36
johnsomso, this is a total mystery: test-instance.operations.os.example.com.19:36
k-s-deanI can show you where thats coming from19:36
johnsomOk, I am interested19:36
k-s-deanhttp://paste.openstack.org/show/803637/19:37
k-s-deanbtw the I have modified the notification topics. the default is notifications_designate19:38
johnsomOh, so you do have sink configured.19:38
k-s-deanyes kolla is configuring it.19:38
k-s-deanwhich service is now responsible for DNS then ?19:39
johnsomSo, we may have a conflict situation going on, where sink has already created some records and when the floating comes along it's not able to create19:39
johnsomneutron calls directly to the designate API now19:39
k-s-deanahhhh, ok. now that makes more sense.19:39
johnsomLet's continue, then we can try a few things and see if we can identify the issue.19:40
k-s-deanok cool.19:40
k-s-deanwhat would you like me to do next ?19:40
johnsomMy crystal ball says the problem is the network/port doesn't have a domain. But let's see19:40
johnsomOk, please create the floating IP and associate it to the instance port. Then do a show on the floating IP and the zone.  All of those can be in the same paste.19:41
k-s-deanhttp://paste.openstack.org/show/803638/19:46
johnsomYep, ok, so we have reproduced the issue you are seeing.19:48
k-s-deanthe conflict is deffiently real19:48
hamalqsorry for interrupting if the neutron does not create DNS records it could be because of this https://github.com/openstack/neutron/blob/e9a75a379ea423f8ee452015888dc954d0decb08/neutron/plugins/ml2/extensions/dns_integration.py#L37419:48
hamalqspecially this line https://github.com/openstack/neutron/blob/e9a75a379ea423f8ee452015888dc954d0decb08/neutron/plugins/ml2/extensions/dns_integration.py#L38319:49
johnsomhamalq Please feel free to add ideas!19:49
k-s-deanI'm see a bunch of duplicate entry errors in designate-sink19:49
hamalqjohnsom: thanks, also no need to use sink when u enable the neutron external dns plugin19:52
johnsomThere are not any segments on the VM instance network or the subnet. I think this function is passing as the IP address record gets added, it's just the name that doesn't get added.19:52
johnsomhamalq Yeah, I mentioned that earlier as well19:52
johnsomI think it is either that the network/port doesn't have a domain or the sink being enabled is conflicting19:53
johnsomk-s-dean Let's try this.19:54
k-s-deanso, from what your saying here is that, one disable designate-sink its no longer required. two assign dns-domain to both private and public network19:54
k-s-deango on.19:54
johnsomDe-associate the floating IP from the port. Then, on the port for the instance, configure the dns_domain setting. Then re-associate the floating IP19:55
k-s-deansorry, private IP port19:58
johnsomYes19:58
johnsome915ca08-baa5-4879-8ee3-b28888446fb019:58
k-s-deanok but the recordsets are still in DNS. might get a duplicate name clash here. setting re-associating now20:00
k-s-deanno difference.20:01
johnsomWell, you can have multiple A records for a name. (DNS round robin) That would be a different problem, but would show that the floating IP is getting a name20:01
k-s-deanhttp://paste.openstack.org/show/803640/20:03
k-s-deanhttp://paste.openstack.org/show/803641/20:04
k-s-deansecond one is floatin IP20:04
johnsomYou shared the show zone for example.com, but put os.armourcomms.com in the dns_domain field. Did the os.armourcomms.com zone get the record?20:05
k-s-deanshit20:05
k-s-deanthat is example.com20:05
johnsomOk20:06
hamalqneutron will not allow the ip duplication20:06
johnsomhamalq The float should have a different IP than the instance port, so that should be fine20:07
johnsomI was kind of expecting: test-instance.os.example.com. 192.168.230.237 and a test-instance.os.example.com. 10.30.0.106 in the "example.com" domain20:07
johnsom192.168.230.237 is the instance port IP, 10.30.0.106 is the floating IP20:08
johnsomThe floating IP didn't "inherit" the instance port DNS names20:09
johnsomOk, I am pretty sure this is part of the problem:20:11
johnsomhttp://paste.openstack.org/show/803637/20:11
johnsomThe floating IP config there is limited to the IP20:11
johnsomSince it's using the sink, maybe what you need is to add another line there:20:12
k-s-deanif i comment out the nova:fixed section the private IPs dont get put in DNS.20:12
johnsomformatv4 = %(hostname)s.%(zone)s20:12
k-s-deanunder neutron:floating IP yes20:12
k-s-deanbecause that produces and error.20:13
johnsomYeah, that is probably a kolla issue where if it is still setting up sink, it's probably not configuring the connection for neutron to call designate20:13
k-s-deanI've already tried that20:13
johnsomlol, ok20:13
k-s-deanhold on let me see if I can find the particular LP that i came across20:13
johnsomI am super rusty on the sink setup20:13
k-s-deanhttps://bugs.launchpad.net/designate/+bug/177292520:14
openstackLaunchpad bug 1772925 in Designate "Error in sink formatv4 neutron_floatingip handler" [Undecided,New]20:14
johnsomYeah, ok, I was afraid of that.20:15
johnsomOk, so the path forward is going to be: Undo the kolla sink setup, configure neutron for the direct to designate path.20:15
k-s-deanok20:16
johnsomSadly, you will need to do that outside of the kolla managed config20:16
k-s-deanso this should be fairly simple then. Delete the sink container20:16
johnsomProbably. That would be a question for the kolla channel20:16
johnsomWell, just stop it for now20:16
k-s-deandone20:17
johnsomYou will then need to follow the instructions here: https://docs.openstack.org/neutron/victoria/admin/config-dns-int-ext-serv.html20:17
k-s-deanok thats done as well.20:18
k-s-deanI have that particular section in my neutron.conf20:18
k-s-deanand external_dns_driver is set to designate20:19
k-s-deanI assume from here it would just be a case of setting the dns-domain on the private network20:19
johnsomHmm, well, in theory, but if this was already set I would have expected that last test to work.20:20
k-s-deanone thing that I can confirm works. In horizon20:21
k-s-deanif i set the dns_domain and dns_name on creation of the floating IP the DNS name does register in DNS20:21
johnsomYeah, you can set the directly in the floating IP on the CLI as well.20:22
johnsomI didn't think that would solve your problem though.20:22
k-s-deanit doesn't20:22
k-s-deanlet me try our theory out and see what happens20:22
k-s-deanwithout sink the DNS entries are not being removed, but thats expected.20:24
k-s-deanwhat would be the best service to look at to see the api calls from neurton ?20:24
k-s-deanneutron*20:24
johnsomCan you provide "openstack extension list --network -c Alias"20:25
johnsomThe calls from neutron will be in your API log. I'm not sure how kolla deploys it, it could be in the apache logs, or a designate specific api log20:25
k-s-deanhttp://paste.openstack.org/show/803642/20:26
hamalqtry delete all the records u have in the zone then create/delete a service20:27
hamalqi mean a server20:27
johnsomYeah, I agree or create another instance with a different name20:28
k-s-deanyeah nothing being registered. I created a new server test-instance220:28
k-s-deanwhich would have a different dns name so should be fine20:28
hamalqthe ip also matters20:28
k-s-deanI created a brand new floating IP also20:29
k-s-deanI can also confirm that the fixed IP is different20:29
hamalqthen it should be https://github.com/openstack/neutron/blob/e9a75a379ea423f8ee452015888dc954d0decb08/neutron/plugins/ml2/extensions/dns_integration.py#L383 is returning false20:30
k-s-deanI find that pretty odd that the flat network returns false20:31
k-s-deanthe docs do refrence a vlan provider network.20:31
k-s-deanI can create that if needed.20:31
hamalqhttps://github.com/openstack/neutron/blob/e9a75a379ea423f8ee452015888dc954d0decb08/neutron/plugins/ml2/extensions/dns_integration.py#L37020:31
k-s-deanbrb need to take a leak20:32
k-s-deanback20:33
k-s-deanwhich component handles DNS registration now then ?20:34
k-s-deanif sink is deprecated is it neutron its self20:34
johnsomyes20:34
johnsomhttps://docs.openstack.org/neutron/victoria/admin/config-dns-int-ext-serv.html#configuring-openstack-networking-for-integration-with-an-external-dns-service20:34
johnsomThis gives neutron the credentials to do so.20:34
johnsomI would check the designate API logs to make sure neutron is successful at logging in to post the update.20:35
k-s-deanI can see it checking for the zone20:40
k-s-deanSo no post requests only get requests20:42
k-s-deansame as before create a new vm with new name, new floating IP20:42
k-s-deanI would expect a post request to designate20:43
johnsomYes, me too20:43
k-s-deangot an intresting log here.20:45
k-s-deanhttp://paste.openstack.org/show/803643/20:47
k-s-dean.320:47
johnsomDoes the zone show two records?20:49
johnsomOr none20:50
k-s-deannone20:50
k-s-deandid you scroll down on the paste ?20:50
k-s-deanI left to much of a gap there.20:50
johnsomlol, missed it20:50
k-s-deanneutron.db.dns_db neutron_lib.exceptions.dns.DNSDomainNotFound: Domain os.example.com. not found in the external DNS service20:50
k-s-deanthis20:50
k-s-deannow why would that be.20:51
k-s-deanit exists. so neutron should see it.20:51
k-s-deanI'm wondering do I need to specify the ID of the zone in the neutron conf20:51
k-s-deandns_domain =20:52
johnsomThat would just be the default.20:53
k-s-deanobviously mine is set to dns_domain = example.com.20:53
johnsomCould it be that the zone you created for os.example.com is not visible to the account you have in neutron in the [designate] section?20:54
johnsomI.e. is the zone owned by a different project?20:54
k-s-deanyeah... it is.20:54
k-s-deanok let me trying creating the zone in that domain and in that particular project then20:55
johnsomHmm, but the zone should be owned by the same project as the port.20:55
johnsomreally20:55
johnsomIt should impersonate. Let me look in the code.20:55
johnsomCan you do an "openstack zone show" for the os.example.com zone?20:59
k-s-deanhttp://paste.openstack.org/show/803644/21:00
johnsomYeah, ok, that is the problem21:01
johnsomThe zone is owned by 78293d3b575c43ed81c517fbb751abf4 and the VM is project ID 306d564b6a6e42cc92060361aa87f7fd21:01
k-s-deanok.21:02
johnsomYou can't insert records into someone else's zone via creating a port.21:02
k-s-deanso it has to be in the same project21:02
k-s-dean and domain21:02
k-s-deanor is it just project. It will impersonate the domain ?21:02
johnsomThe comparison should be a the project ID level. The concept of a credential domain is in keystone and maps down to the project ID21:03
k-s-deanok.21:04
johnsomSo, if the user creating the VM, also creates a zone in designate, they should be able to use it.21:04
k-s-deanmakes sense21:05
k-s-deanheyhey....21:08
k-s-deanwe have lift off.21:08
johnsomYay21:08
k-s-deanthank you so much21:08
johnsomSorry for the long about path to an answer21:08
k-s-deanno problem.21:08
johnsomBut at least you have an answer21:08
k-s-deanI do you've made me a very happy man21:08
k-s-deanso basically. bye bye desingate-sink21:09
johnsomCool. Happy computing. I am off to do a ton of reviews I have backlogged21:09
k-s-deanThanks so much dude.21:09
johnsomYeah, it had a bunch of limitations, so the decision was made to just go direct and not use the message queues21:09
* johnsom disappears back into the dungeon where he is locked with launchpad and gerrit21:10
k-s-deanAwesome. well you have no idea how thankfull I am you helped me out.21:11
k-s-deangood luck with your code reviews :)21:11
hamalqjohnsom: can u check this too it was approved before but small changes asked from frickler all done https://review.opendev.org/c/openstack/designate-tempest-plugin/+/755876, https://review.opendev.org/c/openstack/designate/+/755379/21:12
johnsomIt's on my list but behind a bunch of high priority stuff (for internal company needs). I thought you had the reviews you need on those, don't you have two +2?21:13
hamalqjohnsom: frickler asked for some changes so i did them so i need approval again21:14
hamalqhttps://review.opendev.org/c/openstack/designate/+/748285/, https://review.opendev.org/c/openstack/designate/+/754226/ those two dpend on it but they are approved21:15
johnsomOk, hopefully some of the other cores can get to those sooner than I can.21:16
hamalqthanks21:16
*** k-s-dean has quit IRC22:17
*** lbragstad_ has joined #openstack-dns22:29
*** lbragstad has quit IRC22:35
*** k-s-dean has joined #openstack-dns23:18

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!