| bar_ | so... there isn't really a way to test your amphora API in devstack deployment, or is there? | 00:00 |
|---|---|---|
| *** longstaff has quit IRC | 00:00 | |
| rm_work | the amp API? | 00:01 |
| rm_work | yeah that works | 00:01 |
| rm_work | you can devstack it | 00:01 |
| rm_work | that's one of the pieces that was originally *in* my WIP driver patch, but I pulled out and made sure worked upstream | 00:01 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Amphora API Failover call https://review.openstack.org/525778 | 00:02 |
| rm_work | fixed it a little | 00:02 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add unit tests for neutron utils, add model/util for floating_ip https://review.openstack.org/525353 | 00:02 |
| rm_work | and rebases | 00:02 |
| *** sshank has quit IRC | 00:05 | |
| bar_ | rm_work, can amphora and its load balancer reside in different projects? | 00:05 |
| johnsom | bar_ They do | 00:05 |
| rm_work | yeah they always do :) | 00:06 |
| johnsom | The amphora is under the octavia service account project and the load balancer is under the tenant | 00:06 |
| rm_work | the /octavia/ endpoint is admin | 00:06 |
| johnsom | Well, if you run everything under "admin".... | 00:06 |
| rm_work | it's not intended for end-users | 00:06 |
| bar_ | so if we failover amphora, don't we need to check authorize against both the amphora.project_id and amphora.lb.project_id? | 00:06 |
| rm_work | it's the same | 00:06 |
| rm_work | or rather | 00:07 |
| rm_work | project-id of the amphora is kinda meaningless | 00:07 |
| rm_work | do we even have that field? | 00:07 |
| rm_work | you could loosely say the amphora belongs to the LB's project but the compute instance that runs it doesn't | 00:07 |
| johnsom | no | 00:07 |
| rm_work | and again, this call is "admin only" | 00:07 |
| rm_work | users don't go to anything under /v2.0/octavia/ | 00:08 |
| rm_work | that's where we put our "admin API" functions | 00:08 |
| rm_work | johnsom: feel free to review that patch, in your copious free time :P:P | 00:08 |
| johnsom | Funny man | 00:08 |
| johnsom | The project_id thing does look odd, but the key line is: https://review.openstack.org/#/c/525778/7/octavia/policies/amphora.py:35 | 00:10 |
| rm_work | errr | 00:10 |
| rm_work | wut | 00:11 |
| johnsom | opps, :35 doesn't work there | 00:11 |
| rm_work | weird lol | 00:11 |
| johnsom | https://review.openstack.org/#/c/525778/7/octavia/policies/amphora.py Line 35 | 00:11 |
| rm_work | augh | 00:12 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Amphora API Failover call https://review.openstack.org/525778 | 00:13 |
| rm_work | self.assertFalse wasn't autocompleting so i did assertEquals(False | 00:14 |
| rm_work | apparently i was right at first even though it doesn't autocomplete | 00:14 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Add unit tests for neutron utils, add model/util for floating_ip https://review.openstack.org/525353 | 00:15 |
| rm_work | patch chains >_< so fun | 00:15 |
| rm_work | the biggest reason i'd like to merge the first two ;P | 00:15 |
| bar_ | same with _get_resource(). 5 patch-sets, 3 of them to resolve conflicts. | 00:19 |
| bar_ | and already rebased ontop of it to resolve more. | 00:19 |
| *** fnaval has joined #openstack-lbaas | 00:24 | |
| *** yuanying_ has joined #openstack-lbaas | 00:25 | |
| *** yuanying has quit IRC | 00:27 | |
| bar_ | rm_work, load balancer that go into ERROR state make me sad. Would you mind considering a more conservative approach where the load balancer is to ERROR only if the Amphora is already in ERROR state? | 00:29 |
| rm_work | if the Amp goes to ERROR, the LB has to | 00:31 |
| rm_work | there's not really a way around that AFAIK | 00:31 |
| rm_work | the purpose is to block updates | 00:31 |
| rm_work | because if you leave the LB mutable, it will be unable to provision changes | 00:31 |
| rm_work | because the amp is in ERROR | 00:31 |
| bar_ | sure, but currently any exception would result in load balancer go into error, correct? | 00:32 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 00:32 |
| rm_work | any exception on the amp flows would, yes | 00:32 |
| *** armax has joined #openstack-lbaas | 00:33 | |
| bar_ | Would you say that for any exception that might arise during amphora failover, it is guaranteed that the amphora will be in state of ERROR? | 00:33 |
| rm_work | yes | 00:34 |
| bar_ | ah | 00:34 |
| rm_work | if we can't failover the amp ... SOMETHING is wrong | 00:34 |
| rm_work | and we can't trust the amp anymore to be working | 00:34 |
| bar_ | perhaps we didn't even get to execute the flow... | 00:35 |
| bar_ | something has failed before we began | 00:35 |
| *** Swami has quit IRC | 00:35 | |
| rm_work | still we don't know what's going on because we can't do anything with the amp... | 00:39 |
| rm_work | though if something fails before the flows | 00:39 |
| *** ssmith has quit IRC | 00:39 | |
| rm_work | the LB will actually be stuck in PENDING_UPDATE I think | 00:39 |
| rm_work | it won't be in ERROR technically :P | 00:39 |
| dayou | @johnsom, I heard you were able to do a test with slaac and it worked, where cloud-init wrote it as a static config. | 00:39 |
| dayou | Could you post your solution so I can take a look? | 00:39 |
| johnsom | dayou Give me 20 minutes | 00:41 |
| *** harlowja has quit IRC | 00:45 | |
| rm_work | it's interesting that we don't stamp the topology on the LB object until AFTER it's created | 00:52 |
| rm_work | shouldn't we stamp that on there on the initial create of the DB object? | 00:52 |
| rm_work | we KNOW then... | 00:52 |
| rm_work | for stuff that goes to ERROR during a create, it ends up with NULL topology in the DB | 00:52 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 00:53 | |
| johnsom | rm_work It is that way to deal with spares pool amps | 00:54 |
| rm_work | it is? O_o | 00:54 |
| johnsom | There is a chicken/egg timing thing. I think I put a comment in the code about it | 00:54 |
| *** AlexeyAbashkin has quit IRC | 00:57 | |
| johnsom | https://www.irccloud.com/pastebin/OJA4RPRu/ | 00:58 |
| johnsom | dayou ^^^ that is how I set it up. The amp comes up fine. You of course have to change your o-hm0 devstack config or however you have your lb-mgmt-net configured to have access to that network | 00:59 |
| *** longstaff has joined #openstack-lbaas | 00:59 | |
| *** yuanying_ has quit IRC | 01:00 | |
| *** longstaff has quit IRC | 01:00 | |
| dayou | Is it v6 only or v4 + v6? | 01:00 |
| dayou | I know v4 + v6 should work since it first gets the v4 address from dhcp and then v6 slacc. | 01:01 |
| *** yuanying has joined #openstack-lbaas | 01:09 | |
| rm_work | uuuuugh ok this is dumb | 01:30 |
| johnsom | dayou It is IPv6 only | 01:31 |
| rm_work | if we delete an LB, and mark its amp deleted, and we delete the amp-health record for them ... and then a new health record comes in ... and then expires... we try to failover the amp even though it was deleted. which actually brings the loadbalancer back to active. | 01:31 |
| johnsom | rm_work ummm, if it's deleted how is it sending another heartbeat? Or is this, "see above delays" | 01:32 |
| rm_work | ^^ | 01:33 |
| rm_work | well actually | 01:33 |
| rm_work | it's that AND somehow right now deletes aren't working always | 01:33 |
| rm_work | trying to figure out why on that last one | 01:33 |
| rm_work | like, nova says it deleted it to the worker.... buuut it's still active | 01:33 |
| rm_work | i've got some bug somewhere | 01:33 |
| rm_work | maybe around what auth it's using? | 01:34 |
| rm_work | need to dig through logs | 01:34 |
| dayou | johnsom, this seems has nothing to do with cloud init | 01:34 |
| johnsom | Ok, I am about to disappear for the evening. Just had a fun 30 minutes on the phone playing IT for my mother, so... | 01:34 |
| rm_work | >_< | 01:35 |
| johnsom | dayou Correct, cloud-init "does the right thing" in this scenario | 01:35 |
| johnsom | It just works | 01:35 |
| dayou | What's the cloud init generated config file ? | 01:35 |
| dayou | I am curious to take a look at that also | 01:36 |
| johnsom | Umm, I deleted that VM after chatting with Jason. It's a standard interfaces.d file. It does have a static address because nova/neutron puts one in for it (since it knows the prefix and MAC for the NIC) | 01:37 |
| dayou | auto lo | 01:38 |
| dayou | iface lo inet loopback | 01:38 |
| dayou | auto eth0 | 01:38 |
| dayou | iface eth0 inet manual | 01:38 |
| dayou | auto eth0 | 01:38 |
| dayou | iface eth0 inet6 auto | 01:38 |
| johnsom | dayou This was the cloud-init metadata file I shared with Jason: | 01:38 |
| johnsom | https://www.irccloud.com/pastebin/H3zWM4js/ | 01:38 |
| johnsom | dayou No, that is an invalid interfaces file | 01:38 |
| dayou | johnsom, seems you are using ovs | 01:41 |
| dayou | I don | 01:41 |
| dayou | 't know whether that makes a difference | 01:41 |
| johnsom | rm_work I did find that IPv6 VIPs aren't working right however. Sigh. ifup -a throws an error that the secondary IP is already up. I need to look into that | 01:41 |
| dayou | Since we are not using ovs at all | 01:41 |
| johnsom | dayou Yeah, that doesn't matter here really | 01:41 |
| rm_work | johnsom: i thought dlundquist was looking into that forever ago | 01:41 |
| rm_work | i remember specifically that -a didn't work | 01:41 |
| rm_work | and he had some workaround | 01:42 |
| johnsom | rm_work Well, it *was* working fine | 01:42 |
| dayou | If I can get the interfaces file I can be more sure about it | 01:42 |
| johnsom | rm_work I fixed that CentOS doesn't have -a, that was a different issue | 01:42 |
| rm_work | hmm k | 01:42 |
| rm_work | yeah i thought it had to do with local scoping or something | 01:42 |
| rm_work | like, we were adding an address that was kinda a dupe? or something | 01:43 |
| johnsom | dayou Ugh, umm, ok, I will run those instructions and grab it. 10-20 minutes. | 01:43 |
| rm_work | anyway, dlundquist might know | 01:43 |
| dayou | johnsom, thanks | 01:43 |
| johnsom | dayou My wife can give you the stare.... grin | 01:43 |
| rm_work | though he may not want to touch this stuff with a 10 foot pole :P | 01:43 |
| dayou | johnsom, what's the stare? | 01:43 |
| johnsom | rm_work Oh, link-local addresses, whole different animal, there is a patch up about that . | 01:43 |
| dayou | johnsom, I'd rather have some steak | 01:43 |
| johnsom | dayou stare of why aren't you cooking dinner with me stare | 01:44 |
| dayou | johnsom, because I get up at 4 and watched your meeting silently and pushed changed and go back to irc to get a stare | 01:44 |
| dayou | after a snap | 01:45 |
| dayou | :-C | 01:45 |
| rm_work | augh | 01:46 |
| rm_work | johnsom: ok so you have to go but | 01:46 |
| johnsom | dayou https://wiki.openstack.org/wiki/Octavia/Meeting_Minutes#2017-12-13_Weekly_meeting: | 01:46 |
| rm_work | LATER.... | 01:46 |
| rm_work | something happened... | 01:46 |
| dayou | what happened? | 01:46 |
| dayou | Also, what is IT, kind of phone game? | 01:46 |
| dayou | It's the first time I heard that | 01:46 |
| johnsom | "Information Technology" (IT) also known as technical support | 01:47 |
| johnsom | Helping her with Wifi issues | 01:47 |
| dayou | Ah, kind of joke, I thought it was a game | 01:48 |
| rm_work | unknown keyword 'external-check' in 'global' section | 01:48 |
| rm_work | when trying to upload listener config | 01:48 |
| rm_work | on the amp agent | 01:48 |
| johnsom | You are missing the patch for RHEL in the amp image | 01:48 |
| rm_work | eugh | 01:49 |
| johnsom | https://docs.openstack.org/releasenotes/octavia/unreleased.html#upgrade-notes | 01:49 |
| rm_work | so i need a new amp image | 01:49 |
| rm_work | gross | 01:49 |
| johnsom | Type PING? | 01:49 |
| rm_work | no HM | 01:49 |
| johnsom | Umm, that is unexpected | 01:49 |
| rm_work | :/ | 01:49 |
| rm_work | just on listener create | 01:49 |
| rm_work | and ... interestingly, it is marked as ACTIVE (like, it worked? or something) | 01:50 |
| johnsom | Oh, that is a bug isn't it | 01:50 |
| rm_work | yeah so it looks like it fails but doesn't revert the flow / mark as ERROR | 01:50 |
| rm_work | for some reason | 01:50 |
| johnsom | rm_work This line: https://review.openstack.org/#/c/518632/11/octavia/common/jinja/haproxy/templates/base.j2 | 01:50 |
| johnsom | The other has a HM PING wrapper | 01:50 |
| johnsom | https://review.openstack.org/#/c/518632/11/octavia/common/jinja/haproxy/templates/macros.j2 | 01:52 |
| rm_work | soooo | 01:52 |
| johnsom | So, yeah, my bad on that. It was planned to be only need to roll the amp if HM of PING | 01:52 |
| rm_work | :/ | 01:52 |
| rm_work | so there's the other bug you uncovered somehow | 01:53 |
| rm_work | which is... i have no idea how, but | 01:53 |
| rm_work | when it fails in this way | 01:53 |
| rm_work | it doesn't send the LB to ERROR | 01:53 |
| rm_work | it sends it to ACTIVE | 01:54 |
| rm_work | is that a bug? | 01:54 |
| rm_work | it doesn't create the listener and just sends it back to ACTIVE? | 01:54 |
| johnsom | Well, the jinja stuff succeeds, but the restart/start should have raised the failure | 01:54 |
| rm_work | so now i've got an ACTIVE LB | 01:54 |
| rm_work | without the listener i just created | 01:54 |
| rm_work | (tried to create) | 01:55 |
| rm_work | that is ... confusing | 01:55 |
| rm_work | k i'll let you go, this is sucky but i can look at it tomorrow, it's just in Staging | 01:55 |
| johnsom | dayou | 01:57 |
| johnsom | https://www.irccloud.com/pastebin/yjfwHgc8/ | 01:57 |
| rm_work | oh nm the Listener *does get created in the DB* | 01:57 |
| rm_work | but the LB goes back to ACTIVE somehow | 01:57 |
| rm_work | instead of ERROR | 01:57 |
| rm_work | when haproxy fails to start | 01:57 |
| rm_work | or ... i need to get some logs i think, this is weird | 01:57 |
| johnsom | Yeah, it's the restart/reload/start code that should have raised an error | 01:57 |
| rm_work | hmmm but i did get | 01:58 |
| rm_work | Flow 'octavia-create-listener_flow' (d972404d-b881-42fb-abdd-4672dbac4159) transitioned into state 'REVERTED' from state 'RUNNING' | 01:58 |
| rm_work | but ... that put it in ACTIVE status | 01:58 |
| rm_work | even though the listener object is now created | 01:58 |
| johnsom | Say what? | 01:58 |
| rm_work | which guarantees a failover because the listener count is wrong | 01:58 |
| rm_work | this is what i'm seeing | 01:58 |
| rm_work | i can do it again from scratch to prove it | 01:59 |
| rm_work | one sec | 01:59 |
| johnsom | Oh, so listener is in error, but LB is active? | 01:59 |
| johnsom | That is normal | 01:59 |
| rm_work | hmm k | 01:59 |
| rm_work | interestingly in the client we don't return prov-status for Listeners? O_o | 02:00 |
| johnsom | That allows the "delete listener" to try again thing | 02:00 |
| rm_work | err, we don't display it | 02:00 |
| rm_work | johnsom: ok well that's great and all but | 02:00 |
| johnsom | We don't? | 02:00 |
| rm_work | we do not seem to, no | 02:00 |
| rm_work | anyway | 02:00 |
| rm_work | in most deployments, what is the health failover time? a minute or two? | 02:00 |
| rm_work | so if the user doesn't delete that Listener... | 02:00 |
| johnsom | rm_work https://github.com/openstack/python-octaviaclient/blob/master/octaviaclient/osc/v2/constants.py#L66 | 02:01 |
| johnsom | It's in the client | 02:01 |
| rm_work | we will do a failover | 02:01 |
| rm_work | and continue to do so | 02:01 |
| rm_work | until they fix it | 02:01 |
| rm_work | maybe i need to update | 02:01 |
| johnsom | Well, it will only try a failover once, then should leave the LB in error | 02:01 |
| rm_work | ah true | 02:01 |
| rm_work | but ... then the user can't delete the listener anymore | 02:01 |
| rm_work | so they're stuck | 02:01 |
| johnsom | Yeah, cascade delete works | 02:02 |
| rm_work | so if a Listener create fails, it starts a timer for them for like ... 1-2 minutes | 02:02 |
| rm_work | they need to delete the Listener or the LB is doomed | 02:02 |
| rm_work | ah i'm looking at a `listener list` not a show | 02:02 |
| johnsom | oh, yeah, that is a shorter list of columns. Otherwise the output is unreadable | 02:02 |
| rm_work | yeah i guess so but status seems important <_< | 02:03 |
| rm_work | anyway | 02:03 |
| rm_work | this seems like a shitty workflow | 02:03 |
| rm_work | "create listener fails, put LB to ACTIVE and Listener to ERROR so the user can fix it" *2 minutes later* "health failover happens, LB is immutable and dead" | 02:04 |
| johnsom | So if the listener is error it should *not* attempt a failover recovery? | 02:04 |
| rm_work | i mean | 02:05 |
| rm_work | yeah maybe it shouldn't count that toward listener count | 02:05 |
| johnsom | LB can still be deleted with cascade | 02:05 |
| rm_work | when deciding whether to store health data | 02:05 |
| johnsom | I mean, this is a coding error on our part, not exactly a "something that randomly happens". | 02:06 |
| johnsom | I think I would rather have it try a failover for cases where the host ran out of disk or something like that | 02:06 |
| rm_work | hmm k | 02:07 |
| rm_work | just seems lame | 02:07 |
| rm_work | anyway i guess it's fine so long as failures don't happen very often | 02:07 |
| rm_work | just need to fix this to not explode | 02:07 |
| *** jniesz has quit IRC | 02:07 | |
| rm_work | i guess I can roll a new image but testing it is difficult in my new env <_< | 02:07 |
| johnsom | Just saying, but happy to start a brain storming etherpad and talk through scenarios to see if we can come to a better conclusion | 02:07 |
| rm_work | this ping patch is T_T | 02:07 |
| rm_work | i would rather literally remove PING as an option | 02:07 |
| rm_work | than have to deal with this patch, lol | 02:08 |
| johnsom | Well, there is that. Sadly we have had it as part of the API forever | 02:08 |
| rm_work | yeah | 02:08 |
| johnsom | never worked for the reference drivers | 02:08 |
| johnsom | You are welcome to propose a patch to fix my sins | 02:09 |
| rm_work | i'll think on it | 02:09 |
| *** bar_ has quit IRC | 02:09 | |
| rm_work | i don't really understand what is happening here yet | 02:09 |
| johnsom | The other option is to load a real version of haproxy up and move on | 02:09 |
| rm_work | like why it's trying to pass that option to my amp | 02:09 |
| johnsom | haproxy doesn't support PING, so I had to add an external monitor command to do the ping. That line enables external commands, the lower line defines the command, etc. | 02:10 |
| johnsom | On versions of haproxy that weren't from our high school years, that command does not fail. | 02:11 |
| johnsom | grin | 02:11 |
| rm_work | hmm but | 02:12 |
| rm_work | why is that line there | 02:12 |
| rm_work | in the case of centos amps | 02:12 |
| rm_work | shouldn't it depend on the amp type | 02:12 |
| rm_work | if we know centos amps don't have that ability | 02:12 |
| johnsom | Because this: https://review.openstack.org/#/c/518632/11/octavia/amphorae/backends/agent/api_server/haproxy_compatibility.py | 02:13 |
| rm_work | blagh | 02:13 |
| rm_work | ok so yeah i just need to roll a new amp | 02:13 |
| rm_work | >_> | 02:13 |
| johnsom | didn't run and strip it from the config because your amphora-agent doesn't have that code, so it doesn't look for ancient haproxy versions and remove it | 02:13 |
| rm_work | but ummm | 02:13 |
| rm_work | this means existing amps can't create listeners | 02:13 |
| rm_work | so i have to recycle my whole fleet | 02:13 |
| rm_work | srsly | 02:13 |
| johnsom | On CentOS, that is the bug. | 02:13 |
| rm_work | ugh. | 02:14 |
| johnsom | We probably just need to fix it differently | 02:14 |
| rm_work | yes prolly | 02:14 |
| rm_work | i'll think on it | 02:14 |
| johnsom | I have been explaining what it is, not an idea of how to do it different/better | 02:14 |
| rm_work | yeah k | 02:14 |
| johnsom | Maybe query the amp details call before rendering the config to send down? | 02:15 |
| rm_work | sooo what i'm going to do is patch this out locally <_< | 02:15 |
| rm_work | because I need this working NOW | 02:15 |
| johnsom | It's evil either way really | 02:15 |
| johnsom | Yeah, just rip it out of the templates | 02:15 |
| *** bar_ has joined #openstack-lbaas | 02:16 | |
| *** threestrands_ has joined #openstack-lbaas | 02:22 | |
| *** threestrands_ has quit IRC | 02:22 | |
| *** threestrands_ has joined #openstack-lbaas | 02:22 | |
| *** threestrands has quit IRC | 02:22 | |
| *** threestrands_ has quit IRC | 02:22 | |
| *** threestrands_ has joined #openstack-lbaas | 02:23 | |
| *** threestrands_ has quit IRC | 02:23 | |
| *** threestrands_ has joined #openstack-lbaas | 02:23 | |
| rm_work | k that works | 02:23 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 02:23 |
| *** numans has quit IRC | 02:29 | |
| *** numans has joined #openstack-lbaas | 02:32 | |
| *** threestrands_ has quit IRC | 02:32 | |
| *** bar_ has quit IRC | 02:32 | |
| *** yuanying has quit IRC | 02:32 | |
| *** amotoki has quit IRC | 02:32 | |
| *** openstackstatus has quit IRC | 02:32 | |
| *** dmellado has quit IRC | 02:32 | |
| openstackgerrit | Bar RH proposed openstack/octavia master: [WIP] Remove reliance on NeutronException message field https://review.openstack.org/527839 | 02:34 |
| *** threestrands_ has joined #openstack-lbaas | 02:36 | |
| *** yuanying has joined #openstack-lbaas | 02:36 | |
| *** amotoki has joined #openstack-lbaas | 02:36 | |
| *** openstackstatus has joined #openstack-lbaas | 02:36 | |
| *** dmellado has joined #openstack-lbaas | 02:36 | |
| *** barjavel.freenode.net sets mode: +v openstackstatus | 02:36 | |
| openstackgerrit | Hengqing Hu proposed openstack/octavia-dashboard master: Handle singular and plural when deleting resources https://review.openstack.org/527840 | 02:55 |
| *** tongl has quit IRC | 03:07 | |
| *** tongl has joined #openstack-lbaas | 03:09 | |
| *** armax has quit IRC | 03:21 | |
| *** annp has joined #openstack-lbaas | 03:31 | |
| *** armax has joined #openstack-lbaas | 03:55 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 04:07 |
| *** tongl has quit IRC | 04:09 | |
| *** armax has quit IRC | 04:12 | |
| *** armax has joined #openstack-lbaas | 04:13 | |
| *** armax has quit IRC | 04:13 | |
| *** armax has joined #openstack-lbaas | 04:14 | |
| *** armax has quit IRC | 04:14 | |
| *** armax has joined #openstack-lbaas | 04:14 | |
| *** armax has quit IRC | 04:15 | |
| *** yamamoto has joined #openstack-lbaas | 04:18 | |
| openstackgerrit | Santhosh Fernandes proposed openstack/octavia master: [WIP] ACTIVE-ACTIVE create distributor flow https://review.openstack.org/527784 | 04:31 |
| *** annp has quit IRC | 04:40 | |
| *** annp has joined #openstack-lbaas | 04:40 | |
| *** links has joined #openstack-lbaas | 05:35 | |
| *** sapd has quit IRC | 05:38 | |
| *** sapd_ has joined #openstack-lbaas | 05:38 | |
| *** eandersson has quit IRC | 05:47 | |
| *** sticker has quit IRC | 05:49 | |
| *** links has quit IRC | 06:13 | |
| openstackgerrit | Andreas Jaeger proposed openstack/octavia-dashboard master: WIP: Use standard nodejs4 tests https://review.openstack.org/527861 | 06:17 |
| openstackgerrit | Andreas Jaeger proposed openstack/neutron-lbaas-dashboard master: WIP: Use standard nodejs4 tests https://review.openstack.org/527863 | 06:19 |
| *** links has joined #openstack-lbaas | 06:21 | |
| *** links has quit IRC | 06:21 | |
| *** annp has quit IRC | 06:41 | |
| *** annp has joined #openstack-lbaas | 06:41 | |
| *** openstackgerrit has quit IRC | 06:47 | |
| *** threestrands_ has quit IRC | 06:53 | |
| *** sapd__ has joined #openstack-lbaas | 07:07 | |
| *** sapd_ has quit IRC | 07:07 | |
| *** annp has quit IRC | 07:14 | |
| *** annp has joined #openstack-lbaas | 07:14 | |
| *** Alex_Staf has joined #openstack-lbaas | 07:27 | |
| *** openstackgerrit has joined #openstack-lbaas | 07:28 | |
| openstackgerrit | Andreas Jaeger proposed openstack/octavia-dashboard master: WIP: Use standard nodejs4 tests https://review.openstack.org/527861 | 07:28 |
| *** gcheresh has joined #openstack-lbaas | 07:32 | |
| dayou | @johnsom, I just remembered with v6 it can get slacc address, but the problem is the boot tooks up to 3 minutes | 07:35 |
| dayou | because cloud init is trying to get an ip address with dhcp | 07:36 |
| dayou | That's the reason I disabled cloud init networking for slaac totally | 07:36 |
| *** AlexeyAbashkin has joined #openstack-lbaas | 08:09 | |
| *** reedip has quit IRC | 08:15 | |
| *** rcernin has quit IRC | 08:15 | |
| *** reedip has joined #openstack-lbaas | 08:18 | |
| *** tesseract has joined #openstack-lbaas | 08:28 | |
| *** links has joined #openstack-lbaas | 08:52 | |
| *** pckizer_ has joined #openstack-lbaas | 08:52 | |
| *** pckizer has quit IRC | 08:54 | |
| *** links has quit IRC | 08:57 | |
| *** Alex_Staf has quit IRC | 08:59 | |
| nmagnezi | rm_work, o/ | 09:07 |
| *** links has joined #openstack-lbaas | 09:10 | |
| openstackgerrit | Nguyen Van Trung proposed openstack/neutron-lbaas master: Avoid tox_install.sh for constraints support https://review.openstack.org/527901 | 09:12 |
| *** Alex_Staf has joined #openstack-lbaas | 09:16 | |
| *** sanfern has quit IRC | 09:17 | |
| *** eN_Guruprasad_Rn has joined #openstack-lbaas | 09:17 | |
| *** rcernin has joined #openstack-lbaas | 09:23 | |
| *** dokua has joined #openstack-lbaas | 09:27 | |
| rm_work | dayou: yeah see that sounds like my issue again -- the dhcp process happens BEFORE cloud-init, so you have to disable it on the image first, THEN cloud-init can take care of the real config after | 09:31 |
| rm_work | nmagnezi: o/ unfortunately about to head to bed T_T | 09:32 |
| nmagnezi | rm_work, understandable.. | 09:32 |
| nmagnezi | :) | 09:32 |
| openstackgerrit | Santhosh Fernandes proposed openstack/octavia master: Adding exabgp-speaker element to amphora image https://review.openstack.org/490164 | 09:32 |
| rm_work | nmagnezi: maybe i can catch you in the morning for me <_< | 09:35 |
| nmagnezi | rm_work, maybe :-) | 09:35 |
| nmagnezi | rm_work, no worries. we'll sync eventually.. | 09:36 |
| dayou | rm_work: yes, so many issues and headaches | 10:00 |
| dayou | rm_work: needs to gets some good sleep tonight and then continue solving this one by one | 10:01 |
| *** annp has quit IRC | 10:07 | |
| *** salmankhan has joined #openstack-lbaas | 10:14 | |
| dayou | rm_work: in my case, for slacc,cloud init is not needed at all, all interface should get slaas with router announcement or other means of layer two address assignment facility | 10:19 |
| *** sapd__ has quit IRC | 10:37 | |
| *** links has quit IRC | 10:43 | |
| *** reedip has quit IRC | 10:44 | |
| *** sapd__ has joined #openstack-lbaas | 10:51 | |
| *** Alex_Staf has quit IRC | 10:55 | |
| *** eN_Guruprasad_Rn has quit IRC | 10:55 | |
| *** Alex_Staf has joined #openstack-lbaas | 10:55 | |
| *** links has joined #openstack-lbaas | 10:57 | |
| *** reedip has joined #openstack-lbaas | 10:59 | |
| *** AlexeyAbashkin has quit IRC | 11:10 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 11:11 | |
| *** yamamoto has quit IRC | 11:12 | |
| *** eN_Guruprasad_Rn has joined #openstack-lbaas | 11:27 | |
| *** dokua has quit IRC | 11:29 | |
| *** eN_Guruprasad_Rn has quit IRC | 11:31 | |
| *** bar_ has joined #openstack-lbaas | 11:49 | |
| openstackgerrit | Hengqing Hu proposed openstack/neutron-lbaas-dashboard master: WIP: Use standard nodejs4 tests https://review.openstack.org/527863 | 11:52 |
| *** yamamoto has joined #openstack-lbaas | 11:53 | |
| *** yamamoto has quit IRC | 11:54 | |
| *** yamamoto has joined #openstack-lbaas | 11:55 | |
| *** sapd__ has quit IRC | 12:11 | |
| *** sapd__ has joined #openstack-lbaas | 12:12 | |
| openstackgerrit | Hengqing Hu proposed openstack/neutron-lbaas-dashboard master: WIP: Use standard nodejs4 tests https://review.openstack.org/527863 | 12:15 |
| *** salmankhan has quit IRC | 12:19 | |
| *** yamamoto has quit IRC | 12:19 | |
| *** salmankhan has joined #openstack-lbaas | 12:25 | |
| *** eN_Guruprasad_Rn has joined #openstack-lbaas | 12:39 | |
| openstackgerrit | Hengqing Hu proposed openstack/neutron-lbaas-dashboard master: WIP: Use standard nodejs4 tests https://review.openstack.org/527863 | 12:45 |
| *** reedip has quit IRC | 12:54 | |
| *** fnaval has quit IRC | 12:55 | |
| *** yamamoto has joined #openstack-lbaas | 13:07 | |
| *** reedip has joined #openstack-lbaas | 13:08 | |
| openstackgerrit | Bar RH proposed openstack/octavia master: Specify management IP addresses per amphora https://review.openstack.org/505158 | 13:17 |
| openstackgerrit | Santhosh Fernandes proposed openstack/octavia master: Adding exabgp-speaker element to amphora image https://review.openstack.org/490164 | 13:21 |
| *** fnaval has joined #openstack-lbaas | 13:35 | |
| *** yamamoto has quit IRC | 13:41 | |
| *** yamamoto has joined #openstack-lbaas | 13:50 | |
| *** yamamoto has quit IRC | 13:54 | |
| *** rcernin has quit IRC | 13:59 | |
| *** atoth has joined #openstack-lbaas | 14:00 | |
| *** dosaboy has quit IRC | 14:04 | |
| *** fnaval has quit IRC | 14:06 | |
| *** dosaboy has joined #openstack-lbaas | 14:10 | |
| *** salmankhan has quit IRC | 14:11 | |
| *** yamamoto has joined #openstack-lbaas | 14:14 | |
| *** salmankhan has joined #openstack-lbaas | 14:16 | |
| *** yamamoto has quit IRC | 14:19 | |
| *** sapd__ has quit IRC | 14:41 | |
| *** eN_Guruprasad_Rn has quit IRC | 14:50 | |
| *** fnaval has joined #openstack-lbaas | 14:51 | |
| *** fnaval has quit IRC | 14:56 | |
| *** fnaval has joined #openstack-lbaas | 14:59 | |
| *** fnaval has quit IRC | 15:03 | |
| *** fnaval has joined #openstack-lbaas | 15:07 | |
| *** fnaval_ has joined #openstack-lbaas | 15:08 | |
| *** fnaval has quit IRC | 15:11 | |
| *** fnaval has joined #openstack-lbaas | 15:12 | |
| *** fnaval_ has quit IRC | 15:14 | |
| *** amotoki has quit IRC | 15:22 | |
| *** Alex_Staf has quit IRC | 15:26 | |
| *** links has quit IRC | 15:26 | |
| *** armax has joined #openstack-lbaas | 15:47 | |
| *** Alex_Staf has joined #openstack-lbaas | 15:57 | |
| *** gcheresh has quit IRC | 15:57 | |
| openstackgerrit | Bar RH proposed openstack/octavia master: Remove reliance on NeutronException message field https://review.openstack.org/527839 | 16:04 |
| *** sanfern has joined #openstack-lbaas | 16:34 | |
| *** Alex_Staf has quit IRC | 16:37 | |
| johnsom | dayou That is not how it works for me. Please send the /etc/network/interfaces, /etc/network/interfaces.d/*, /etc/netns/amphora-haproxy/network/interfaces, /etc/netns/amphora-haproxy/network/interfaces.d/* , /var/log/cloud-init*, your config drive network_data.json file | 16:43 |
| sanfern | hi johnsom , | 16:43 |
| sanfern | do u have any mechanism to have /usr/local/lib/python2.7/dist-packages/exabgp/ across all the versions and distributions | 16:45 |
| *** yamamoto has joined #openstack-lbaas | 16:46 | |
| sanfern | ref - https://review.openstack.org/#/c/491016/13 line 36 | 16:46 |
| johnsom | sanfern Hi, no, we install into the path appropriate for the python version the amphora image is created with. | 16:46 |
| johnsom | So installation into the correct python environment is automatic. The paths to those files varies with the distribution vendors. | 16:47 |
| sanfern | we need to refer to env path variable instead of absolute path | 16:47 |
| johnsom | Well, yeah, that varies by package. For example octavia installs the the binaries or links to the binaries into the path, so you can just run them | 16:48 |
| johnsom | That happens here: https://github.com/openstack/octavia/blob/master/setup.cfg#L64 | 16:50 |
| *** slaweq has joined #openstack-lbaas | 16:51 | |
| *** slaweq has left #openstack-lbaas | 16:51 | |
| *** tongl has joined #openstack-lbaas | 16:51 | |
| sanfern | should I define here the path for exabgp | 16:53 |
| johnsom | Not in our code, no, it should be in the exabgp package if this is a runnable | 16:53 |
| johnsom | That file defines runnables in our package | 16:53 |
| *** yamamoto has quit IRC | 16:54 | |
| johnsom | http://python-packaging.readthedocs.io/en/latest/command-line-scripts.html#the-console-scripts-entry-point | 16:56 |
| *** bar_ has quit IRC | 16:56 | |
| sanfern | ok | 16:57 |
| johnsom | You might check if other openstack projects are using that command and how they are handling it. I just hope those other projects are py2.7/py35 safe | 16:58 |
| johnsom | It looks to me like maybe exabgp is missing its setuptools config file to setup these tools. | 16:59 |
| johnsom | dayou I will also note, you have to have cloud-init, without it you will not have a proper instance configuration or interfaces configured to connect to the network you pass into the instance. | 17:02 |
| sanfern | ok | 17:06 |
| sanfern | do we need to submit the fix for exabgp ? | 17:07 |
| johnsom | That might be the best approach to solving the problem. | 17:07 |
| *** sri_ has quit IRC | 17:07 | |
| *** sri_ has joined #openstack-lbaas | 17:21 | |
| *** AlexeyAbashkin has quit IRC | 17:23 | |
| sanfern | https://github.com/Exa-Networks/exabgp/pull/735 is this a one | 17:28 |
| johnsom | Not quiet, but that is someone solving a similar problem for an RPM package | 17:31 |
| sanfern | ok | 17:32 |
| johnsom | It doesn't look like the RPM includes that command either.... | 17:35 |
| numans | Request to have a look at this patch - https://review.openstack.org/#/c/510921/ | 17:38 |
| johnsom | namans Thanks | 17:38 |
| johnsom | numans ugh, my typing this morning | 17:39 |
| numans | johnsom, thanks. no issues :) | 17:39 |
| *** logan- has quit IRC | 17:44 | |
| *** logan- has joined #openstack-lbaas | 17:45 | |
| *** tesseract has quit IRC | 17:53 | |
| *** sanfern has quit IRC | 17:57 | |
| *** sanfern has joined #openstack-lbaas | 17:57 | |
| sanfern | johnsom, https://github.com/Exa-Networks/exabgp/issues/757 | 17:57 |
| sanfern | johnsom, finally gates were through for https://review.openstack.org/#/c/524722/ | 17:58 |
| johnsom | Cool, yeah, you might want to fill in a bit more information about the need (i.e. the installation of the CLI tools, etc.) | 17:58 |
| *** Alex_Staf has joined #openstack-lbaas | 17:58 | |
| sanfern | need your help to fix those repo unit tests | 17:58 |
| johnsom | sanfern Ok, I will take a look | 18:00 |
| sanfern | Thank you, I will be out of town next two weeks | 18:02 |
| sanfern | Jacky and jneiz will continue | 18:03 |
| *** Swami has joined #openstack-lbaas | 18:03 | |
| *** yamamoto has joined #openstack-lbaas | 18:05 | |
| johnsom | sanfern Yeah, you already have a comment on that bug that it needs more info | 18:05 |
| johnsom | I can't edit it, so can't help you out there | 18:05 |
| sanfern | there is already a issue - https://github.com/Exa-Networks/exabgp/issues/725 | 18:05 |
| sanfern | yeah :( | 18:07 |
| sanfern | johnsom, Anything other than cli tools, i am updating the issue | 18:15 |
| johnsom | I think that is the biggest issue you are facing there | 18:15 |
| rm_work | dayou: what amp OS are you using? | 18:15 |
| rm_work | johnsom: maybe you don't see this issue because the networks you test with support DHCP or something? | 18:16 |
| rm_work | because my network (and dayou's it seems) have no DHCP server | 18:16 |
| rm_work | so the OS will sit and wait and wait and wait until timeout | 18:16 |
| rm_work | BEFORE it gets to cloud-init | 18:16 |
| rm_work | and it is difficult for me to log in prior to the networking coming up to get the logs | 18:16 |
| rm_work | err, rather, to get the network files | 18:16 |
| rm_work | and once it DOES time out after 3-5 minutes, cloud-init runs and overwrites them | 18:17 |
| rm_work | maybe it's just a CentOS boot services ordering problem? | 18:17 |
| rm_work | like, cloud-init on ubuntu runs before the networking starts, but on centos it's after? | 18:17 |
| johnsom | No, I'm asking for those file after the host is up. I'm just trying to figure out what is going wrong there, I think it is the network dayou is plugging in. Those files will help me see where/when he is seeing the issue. The test I am running is a network/subnet with NO dhcp and NO IPv4 subnets | 18:19 |
| rm_work | ok i mean | 18:19 |
| rm_work | looking at the files that are there AFTER cloud-init on my cloud... they look correct | 18:19 |
| rm_work | but cloud-init runs AFTER dhcp runs | 18:19 |
| rm_work | so .... | 18:19 |
| rm_work | it doesn't really matter what cloud-init puts | 18:20 |
| johnsom | The question of timing very well may be a redhat thing, I am testing with Ubuntu. It could be that on CentOS network manager is running and running first where Ubuntu the systemd is ordered correctly. Again, just asking for logs here. | 18:20 |
| rm_work | k, well, getting logs from those runs is prohibitive | 18:20 |
| johnsom | I really matters to me to see the outcomes, those files and logs. It will tell me ordering and actions taken. | 18:20 |
| *** sanfern has quit IRC | 18:21 | |
| johnsom | I understand you drop SSH so it's hard to get the logs, but he can. | 18:22 |
| rm_work | it's more like, i just can't test images right now | 18:22 |
| *** SumitNaiksatam has joined #openstack-lbaas | 18:22 | |
| rm_work | because i had to move my staging env to point to my prod cloud as a backend | 18:22 |
| rm_work | which means i'm on prod glance | 18:22 |
| rm_work | hmmm well... actually i guess i could adjust my image build automation to use a different tag for staging uploads | 18:23 |
| rm_work | and build an image without that patch and with a root password | 18:23 |
| rm_work | what it really boils down to is i have two days until i'm on vacation and i have a big task to finish so I don't really have time to do that at the moment >_> | 18:24 |
| *** reedip has quit IRC | 18:29 | |
| *** harlowja has joined #openstack-lbaas | 18:29 | |
| *** reedip has joined #openstack-lbaas | 18:41 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 18:47 | |
| *** Alex_Staf has quit IRC | 18:49 | |
| *** AlexeyAbashkin has quit IRC | 18:52 | |
| *** salmankhan has quit IRC | 19:08 | |
| *** openstack has joined #openstack-lbaas | 20:33 | |
| *** ChanServ sets mode: +o openstack | 20:33 | |
| *** AlexeyAbashkin has joined #openstack-lbaas | 20:46 | |
| *** AlexeyAbashkin has quit IRC | 20:51 | |
| *** tongl has quit IRC | 20:52 | |
| *** tongl has joined #openstack-lbaas | 20:53 | |
| *** salmankhan has joined #openstack-lbaas | 21:00 | |
| *** AJaeger has joined #openstack-lbaas | 21:04 | |
| *** salmankhan has quit IRC | 21:04 | |
| openstackgerrit | Andreas Jaeger proposed openstack/neutron-lbaas-dashboard master: Use standard nodejs4 tests https://review.openstack.org/527863 | 21:06 |
| openstackgerrit | Andreas Jaeger proposed openstack/octavia-dashboard master: Use standard nodejs4 tests https://review.openstack.org/527861 | 21:06 |
| *** yamamoto has joined #openstack-lbaas | 21:07 | |
| AJaeger | johnsom: I converted the jobs to native Zuul v3, your previous change is reverted ^ | 21:08 |
| johnsom | AJaeger Cool, thanks for working on that. Glad to have more central jobs | 21:09 |
| *** openstack has joined #openstack-lbaas | 21:13 | |
| *** ChanServ sets mode: +o openstack | 21:13 | |
| AJaeger | johnsom: once the dependent job is in, I'll ping for merge to not have both old and new jobs run - feel free to review already here and the dependent change as well. | 21:15 |
| *** yamamoto has quit IRC | 21:15 | |
| johnsom | Yep, will do, saw the fix just go in | 21:16 |
| *** threestrands has joined #openstack-lbaas | 21:31 | |
| AJaeger | ok, dependency merged - cores, please review and merge my two changes above ^ | 21:45 |
| AJaeger | johnsom: thanks! | 21:45 |
| AJaeger | btw. no need to merge *now* - just wanted to join and ping in case of questions before I sign off for the day | 21:46 |
| *** sticker has joined #openstack-lbaas | 21:46 | |
| *** sshank has joined #openstack-lbaas | 21:48 | |
| *** yamamoto has joined #openstack-lbaas | 21:51 | |
| *** rcernin has joined #openstack-lbaas | 21:58 | |
| *** eandersson has joined #openstack-lbaas | 22:03 | |
| johnsom | Yeah, I think we are good | 22:07 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s) https://review.openstack.org/435612 | 22:17 |
| *** bar_ has joined #openstack-lbaas | 22:21 | |
| *** sshank has quit IRC | 22:35 | |
| *** tongl has quit IRC | 22:36 | |
| *** sshank has joined #openstack-lbaas | 22:50 | |
| rm_work | hmm, amphora-api bug ... trying to show an amp by ID when it's a spare doesn't work, because it tries to do the validate-auth based on the project_id of the amp's LB which it doesn't have | 23:07 |
| *** threestrands_ has joined #openstack-lbaas | 23:10 | |
| bar_ | rm_work, Even in SINGLE mode, am I correct? | 23:12 |
| *** threestrands has quit IRC | 23:13 | |
| rm_work | yeah it won't matter | 23:14 |
| johnsom | Use the context project_id | 23:22 |
| johnsom | Should be fine | 23:22 |
| johnsom | Maybe put a comment as to why that one is different | 23:22 |
| bar_ | Is there work left to do on that basic tempest test to get it in? (https://review.openstack.org/#/c/492311/) | 23:27 |
| bar_ | (If so, than what? rm_work ?) | 23:28 |
| *** harlowja has quit IRC | 23:32 | |
| rm_work | johnsom: yeah not sure why i didn't do that before, did it on the get_all >_> | 23:35 |
| rm_work | prolly copy/paste fail | 23:35 |
| *** harlowja has joined #openstack-lbaas | 23:39 | |
| *** harlowja has quit IRC | 23:40 | |
| *** harlowja has joined #openstack-lbaas | 23:42 | |
| *** tongl has joined #openstack-lbaas | 23:50 | |
| *** bar_ has quit IRC | 23:50 | |
| *** harlowja has quit IRC | 23:50 | |
| *** bar_ has joined #openstack-lbaas | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!