| *** abaindur has joined #openstack-lbaas | 01:06 | |
| *** abaindur has quit IRC | 01:09 | |
| *** abaindur has joined #openstack-lbaas | 01:09 | |
| *** cbrumm_ has joined #openstack-lbaas | 01:10 | |
| *** cbrumm has quit IRC | 01:12 | |
| *** kklimonda has quit IRC | 01:12 | |
| *** kklimonda has joined #openstack-lbaas | 01:12 | |
| *** hongbin has joined #openstack-lbaas | 01:41 | |
| *** hongbin has quit IRC | 03:00 | |
| openstackgerrit | Merged openstack/octavia master: Fix VIP plugging on CentOS-based amphorae https://review.openstack.org/649282 | 03:12 |
|---|---|---|
| openstackgerrit | Merged openstack/octavia stable/queens: Add missing import octavia/opts.py https://review.openstack.org/637398 | 03:12 |
| openstackgerrit | Merged openstack/octavia master: Fix diskimage-create tox, add ``build`` and ``test`` targets https://review.openstack.org/632948 | 03:12 |
| *** psachin has joined #openstack-lbaas | 03:16 | |
| openstackgerrit | Merged openstack/octavia stable/queens: Ensure pool object contains the listener_id if passed https://review.openstack.org/637402 | 03:16 |
| openstackgerrit | Merged openstack/octavia stable/rocky: Fix possible state machine hole in failover https://review.openstack.org/637399 | 03:16 |
| openstackgerrit | Merged openstack/octavia stable/rocky: Add missing import octavia/opts.py https://review.openstack.org/637397 | 03:32 |
| openstackgerrit | Merged openstack/octavia stable/queens: Slightly reorder member flows https://review.openstack.org/647381 | 03:36 |
| *** ramishra has joined #openstack-lbaas | 04:17 | |
| johnsom | cgoncalves: when you are back online (likely before me) feel free to start the stable releases if we have what we want merged. It looks like we got a lot of it today. | 04:45 |
| *** vishalmanchanda has joined #openstack-lbaas | 05:39 | |
| *** lemko has joined #openstack-lbaas | 05:42 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/stein: Fix VIP plugging on CentOS-based amphorae https://review.openstack.org/649503 | 05:49 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/rocky: Fix VIP plugging on CentOS-based amphorae https://review.openstack.org/649504 | 05:49 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/queens: Fix VIP plugging on CentOS-based amphorae https://review.openstack.org/649505 | 05:49 |
| cgoncalves | johnsom, we need these ^ backports merged first. I wonder if we should also release Stein RC2, otherwise we'd be releasing Stein GA with broken spare pool | 05:50 |
| cgoncalves | final RCs are due up to Friday | 05:53 |
| openstackgerrit | Adit Sarfaty proposed openstack/octavia master: Fix catching driver exceptions https://review.openstack.org/648853 | 05:58 |
| cgoncalves | oops! s/spare pool/vip plug on centos/ | 06:02 |
| *** ccamposr has joined #openstack-lbaas | 06:13 | |
| openstackgerrit | Kobi Samoray proposed openstack/octavia master: Fix catching driver exceptions https://review.openstack.org/648853 | 06:25 |
| *** gcheresh has joined #openstack-lbaas | 06:25 | |
| *** pcaruana has joined #openstack-lbaas | 06:36 | |
| *** pcaruana has quit IRC | 06:38 | |
| *** pcaruana has joined #openstack-lbaas | 06:38 | |
| *** rpittau|afk is now known as rpittau | 06:50 | |
| *** luksky has joined #openstack-lbaas | 06:50 | |
| openstackgerrit | Merged openstack/octavia-dashboard master: Drop nodejs4 jobs https://review.openstack.org/649380 | 07:30 |
| *** velizarx has joined #openstack-lbaas | 07:31 | |
| openstackgerrit | pengyuesheng proposed openstack/neutron-lbaas-dashboard master: When using update_member_list method, need to pass pool_id https://review.openstack.org/649519 | 07:31 |
| *** velizarx has quit IRC | 07:32 | |
| *** velizarx has joined #openstack-lbaas | 07:36 | |
| *** abaindur has quit IRC | 07:56 | |
| *** ramishra has quit IRC | 08:07 | |
| *** velizarx has quit IRC | 08:11 | |
| dulek | cgoncalves: Hi! Thanks for yesterday's debugging, with ltomasbo's suggestions we start to think that Amp is getting into L2 mode for some reason, even though we don't provide subnet_id when creating a member. | 08:13 |
| dulek | cgoncalves: I checked my older env, and there the amp has a default route to a router connecting our two subnets. | 08:13 |
| dulek | cgoncalves: Any idea here? Do you even have some distinction between amp's L2 and L3 modes internally? | 08:13 |
| ltomasbo | dulek, if memory works, the net namespace inside the amphora was different | 08:18 |
| ltomasbo | as it had 2 nics instead of one | 08:18 |
| ltomasbo | one connected to the VIP subnet, and one to the member subnet | 08:18 |
| dulek | ltomasbo: http://paste.openstack.org/show/748772/ | 08:19 |
| dulek | ltomasbo: That's my "old" netns setup., | 08:19 |
| dulek | ltomasbo: But 10.1.0.168 is service subnet, I think. | 08:19 |
| *** ramishra has joined #openstack-lbaas | 08:19 | |
| ltomasbo | yes, that one is right | 08:21 |
| ltomasbo | dulek, ^^ | 08:21 |
| ltomasbo | that was working, right? | 08:21 |
| dulek | ltomasbo: Yup. | 08:21 |
| ltomasbo | and that means L3 | 08:21 |
| dulek | ltomasbo: Yep. I think something is forcing L2 mode on amp. | 08:22 |
| ltomasbo | dulek, and do you have the pastebin for the non-working env? | 08:23 |
| dulek | ltomasbo: Not really, I think I'll restack without my fix again. | 08:24 |
| ltomasbo | dulek, ok! it will be great to configure is an L2 vs L3 setting problem | 08:28 |
| dulek | ltomasbo: I'm running my patch with experimental queue, we have an L2 job there. | 08:29 |
| dulek | ltomasbo: Most likely it'll pass and we'll know that something is forcing L2 mode for us. | 08:29 |
| *** jiteka1 has quit IRC | 08:30 | |
| *** lemko has quit IRC | 08:31 | |
| *** velizarx has joined #openstack-lbaas | 08:37 | |
| *** velizarx has quit IRC | 08:40 | |
| *** velizarx has joined #openstack-lbaas | 08:42 | |
| *** luksky has quit IRC | 08:46 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia master: Fix spare amphora check and creation https://review.openstack.org/649381 | 08:53 |
| *** salmankhan has joined #openstack-lbaas | 08:56 | |
| *** ccamposr has quit IRC | 09:03 | |
| *** ccamposr has joined #openstack-lbaas | 09:04 | |
| *** celebdor has joined #openstack-lbaas | 09:07 | |
| *** luksky has joined #openstack-lbaas | 09:18 | |
| dulek | ltomasbo: That's the broken one: http://paste.openstack.org/show/748776/ | 09:19 |
| cgoncalves | dulek, ltomasbo: I'm not following what's the behavior you're suggesting that changed in the amp | 09:39 |
| dulek | cgoncalves: So this is how amphora-haproxy namespace is networking in older version: http://paste.openstack.org/show/748772/ | 09:40 |
| dulek | cgoncalves: And you saw how it looks like now. | 09:40 |
| dulek | cgoncalves: Also in case of the older one, I have a default route set. | 09:40 |
| cgoncalves | in your environment of yesterday, you had two networks each with its own subnet. both subnets, though, had the same range but again were on different networks | 09:41 |
| dulek | cgoncalves: I'm pretty sure 10.0.0.64/26 and 10.0.0.128/26 are totally separated, if that what you mean. | 09:42 |
| cgoncalves | you created the member without passing in a subnet id, so the amp assumed it was on the same network hence not plugging a new interface (eth2) and configuring it | 09:42 |
| dulek | cgoncalves: Anyway I only had two networks when I spawned member with subnet_id. | 09:42 |
| cgoncalves | uhm, right. ok. different subnet ranges. still my point stands | 09:42 |
| dulek | cgoncalves: Yes, I agree. I just want to know what change made our plugin not working and establish what's the expected behavior, so we can code it correctly. | 09:43 |
| dulek | It's totally possible that we were abusing some bug/assumption that got fixed recently. | 09:43 |
| dulek | But to decide I need to know. | 09:44 |
| dulek | cgoncalves: ltomasbo was mentioning L3 and L2 modes of amphora. Is that a thing on Octavia's side or had Kuryr just made it up from its internal stuff? | 09:45 |
| *** yamamoto has quit IRC | 09:58 | |
| *** yamamoto has joined #openstack-lbaas | 10:01 | |
| *** yamamoto has quit IRC | 10:03 | |
| *** ltomasbo has quit IRC | 10:12 | |
| *** yamamoto has joined #openstack-lbaas | 10:22 | |
| cgoncalves | dulek, sorry, what is L3 and L2 modes of amphora? I know what L3 and L2 is but I don't get in the context of amphora | 10:22 |
| dulek | cgoncalves: Well, I assume this means - there's no such thing on Octavia's side and we made it up. :D | 10:23 |
| dulek | cgoncalves: Based on octaviamember_mode | 10:23 |
| cgoncalves | right | 10:24 |
| dulek | cgoncalves: Based on [octavia_defaults]member_mode we were adding or not the subnet_id when creating a member. | 10:24 |
| cgoncalves | only thing you have to consider is you have to specify subnet on member create otherwise amp will assume vip network | 10:24 |
| dulek | cgoncalves: So since the weekend not adding it stopped to work. I see first hit of that issue on Sunday. | 10:25 |
| *** salmankhan has quit IRC | 10:26 | |
| dulek | cgoncalves: Yeah, I assume that we did that L2-L3 distinction for a reason, but I don't know it and ltomasbo seems to be lunching. | 10:26 |
| *** salmankhan has joined #openstack-lbaas | 10:26 | |
| dulek | I'll try to dig the code to understand why we have that in the first place… | 10:26 |
| rm_work | dulek: johnsom did most of that refactoring I believe (some stuff did change in the networking code, some around the switch to bionic, some for bug fixes) so he might be the best one to ask. but, I would say that what you're describing does seem like expected behavior, you must have been abusing a bug before unknowingly, and the solution is really just to pass in the member subnet_id always :) | 10:32 |
| dulek | rm_work: Can you point to the johnsom commit? I haven't noticed anything relevant landing since Friday. | 10:33 |
| * dulek looks again. | 10:33 | |
| rm_work | nothing SUPER recently | 10:33 |
| rm_work | the stuff I'm talking about was across the whole last cycle | 10:33 |
| rm_work | but, we did just cut releases semi-recently for stein? what version are you running exactly? I think I missed that in the scrollback | 10:34 |
| dulek | rm_work: Okay, well, that thing broke our gates on Friday. | 10:34 |
| dulek | rm_work: It's on master gates so whatever's latest + upper-constraints for most libs. | 10:34 |
| *** Dinesh_Bhor has quit IRC | 10:34 | |
| rm_work | so you know for sure it was a change that happened like ... this weekend-ish? | 10:34 |
| dulek | rm_work: I'll try to find latest build that worked. | 10:35 |
| rm_work | IE, you are deploying/testing somewhat constantly and you have a pass and a fail in short succession? | 10:35 |
| rm_work | yeah that'd be good, would like a time window | 10:35 |
| rm_work | cgoncalves: is https://review.openstack.org/#/q/95a872fcd905c0f7c4f2b4cf63e93fa9770d13c1 one of those ones that's going to have to merge queens->rocky->stein because of grenade? or is something wonky happening on our stable gate tests again | 10:36 |
| rm_work | i think it's the latter <_< | 10:37 |
| rm_work | oh rofl you already did rechecks on those and i didn't see them heh | 10:37 |
| *** ltomasbo has joined #openstack-lbaas | 10:39 | |
| openstackgerrit | Adit Sarfaty proposed openstack/octavia master: Fix catching driver exceptions https://review.openstack.org/648853 | 10:42 |
| dulek | rm_work: Okay, so it seems like it's between Friday 16:00 (probably UTC? Not sure what logstash.openstack.org uses) and first failed run is on Sunday, 12:00. | 10:46 |
| rm_work | Hmm | 10:53 |
| rm_work | Really not that much would have changed on our side | 10:54 |
| rm_work | Maybe it's something funny like you were getting lucky sort ordering and some lib changed the way it sorts | 10:54 |
| * rm_work shrugs | 10:55 | |
| *** ramishra has quit IRC | 10:56 | |
| dulek | Yeah, sounds possible. I'm testing without one Neutron commit, let's see… | 10:59 |
| cgoncalves | rm_work, no, can be merged in any order. the grenade issue was fixed already | 11:04 |
| *** ramishra has joined #openstack-lbaas | 11:04 | |
| rm_work | K | 11:08 |
| *** yamamoto has quit IRC | 11:15 | |
| dulek | cgoncalves, rm_work: Do I need to set anything to test with CentOS amphora, or just switch the image we use (in the gates we download the nightly tarball)? | 11:17 |
| cgoncalves | OCTAVIA_AMP_BASE_OS=centos | 11:18 |
| cgoncalves | OCTAVIA_AMP_DISTRIBUTION_RELEASE_ID=7 | 11:18 |
| cgoncalves | OCTAVIA_AMP_IMAGE_SIZE=3 | 11:18 |
| cgoncalves | if you want to use nightly centos, http://tarballs.openstack.org/octavia/test-images/test-only-amphora-x64-haproxy-centos-7.qcow2 | 11:18 |
| *** yamamoto has joined #openstack-lbaas | 11:20 | |
| dulek | cgoncalves: Ah damn nightly won't have your fix as it was built around 6 AM and fix was merged after 7… | 11:21 |
| rm_work | T_T | 11:37 |
| rm_work | Can we trigger that? lol | 11:37 |
| *** rcernin has quit IRC | 11:38 | |
| rm_work | But yeah, whatever image you use, there's nothing you need to configure anymore on the control plane side. They all just work. | 11:50 |
| rm_work | That wasn't the case in... Maybe Pike? But we fixed it | 11:51 |
| *** celebdor has quit IRC | 12:06 | |
| *** trown|outtypewww is now known as trown | 12:07 | |
| *** ramishra has quit IRC | 12:10 | |
| *** ramishra has joined #openstack-lbaas | 12:12 | |
| dulek | rm_work, cgoncalves: Oh well, centos works. | 12:14 |
| dulek | I now think about cloud-init update in latest ubuntu-minimal. | 12:14 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: Fix setting of VIP QoS policy https://review.openstack.org/645817 | 13:00 |
| *** boden has joined #openstack-lbaas | 13:04 | |
| *** lemko has joined #openstack-lbaas | 13:10 | |
| rm_work | Ok, but you really should still include the subnet id for members :) | 13:22 |
| *** ricolin has joined #openstack-lbaas | 13:30 | |
| *** Vorrtex has joined #openstack-lbaas | 13:40 | |
| *** oanson has quit IRC | 13:55 | |
| *** fnaval has joined #openstack-lbaas | 13:58 | |
| *** ramishra has quit IRC | 14:16 | |
| johnsom | We don’t have a L2 or L3 mode for the amps. It is always L3 and up. | 14:19 |
| johnsom | Your paste shows only the VIP network is plugged. Maybe you had s route between those networks before? Routes and gateways are both given to us by neutron, we just honor what was given to us. Host routes, gateways, etc. | 14:22 |
| *** yamamoto has quit IRC | 14:26 | |
| *** yamamoto has joined #openstack-lbaas | 14:26 | |
| *** yamamoto has quit IRC | 14:26 | |
| *** yamamoto has joined #openstack-lbaas | 14:27 | |
| *** yamamoto has quit IRC | 14:27 | |
| *** yamamoto has joined #openstack-lbaas | 14:28 | |
| *** ramishra has joined #openstack-lbaas | 14:32 | |
| *** luksky has quit IRC | 14:47 | |
| *** celebdor has joined #openstack-lbaas | 14:52 | |
| *** ramishra has quit IRC | 14:57 | |
| *** gcheresh has quit IRC | 14:57 | |
| cgoncalves | johnsom, do you agree to release stein RC2 and hold release of maintenance versions until https://review.openstack.org/#/c/649381/ and https://review.openstack.org/#/q/I56947e0d2bb207b59b0b3928efc96546d6410f43 are all merged? | 14:58 |
| johnsom | It seems like we should try to get those two in rc2. | 15:00 |
| cgoncalves | +1 | 15:00 |
| *** celebdor has quit IRC | 15:06 | |
| *** ccamposr has quit IRC | 15:17 | |
| *** gcheresh has joined #openstack-lbaas | 15:24 | |
| *** fnaval has quit IRC | 15:31 | |
| *** fnaval_ has joined #openstack-lbaas | 15:31 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia master: Fix invalid query selector with list_ports https://review.openstack.org/649382 | 15:34 |
| *** goldyfruit has joined #openstack-lbaas | 15:37 | |
| *** gcheresh has quit IRC | 15:44 | |
| cgoncalves | scenario: admin restarts all controllers at same time. when nodes are up again, LBs are failed over by HM if reboot is longer than heartbeat_timeout. | 15:47 |
| cgoncalves | should the periodic health check be started for the first time only after heartbeat_timeout? | 15:47 |
| cgoncalves | currently running immediately -- https://github.com/openstack/octavia/blob/372ff99a030e6b33dad11a35cb9d5c4058805c53/octavia/cmd/health_manager.py#L63 | 15:48 |
| cgoncalves | hmm, there's L76 | 15:48 |
| johnsom | It sleeps a whole heartbeat timeout interval on startup. | 15:49 |
| johnsom | 60 seconds by default. | 15:49 |
| cgoncalves | right, that's L76. | 15:50 |
| cgoncalves | might not be good enough in some deployments, it seems. network might not be 100% up again. | 15:51 |
| *** rpittau is now known as rpittau|afk | 15:57 | |
| *** velizarx has quit IRC | 16:00 | |
| dmellado | dulek: o/ | 16:02 |
| dmellado | hey johnsom cgoncalves | 16:02 |
| dulek | johnsom: Sorry, missed your answers. Doesn't the fact that CentOS amp works fine with our setup and only newest Ubuntu isn't suggests that it's something else? | 16:02 |
| dmellado | the sick and restless me comes back, beware if you have a baby | 16:03 |
| * dmellado reading the backlog | 16:03 | |
| dmellado | larsks: o/ | 16:03 |
| dmellado | dulek: told me that the issue could be related to cloud-init | 16:03 |
| dmellado | and I was wondering if you could have a look as well | 16:03 |
| larsks | dmellado: howdy! | 16:03 |
| dmellado | larsks: hey! just coming back from paternity leave and it's been tough so far xD | 16:04 |
| dmellado | dulek could you summarize the issue to larsks, pls? | 16:04 |
| larsks | dmellado: hope you had a fun paternity leave, though! | 16:04 |
| dmellado | larsks yeah, I'll send you some pics of the kid! | 16:04 |
| dulek | larsks: Newest Ubuntu amphora doesn't have a default route pointing to the router inside amphora-haproxy netns. | 16:05 |
| cgoncalves | dulek, you were running ubuntu amp yesterday. we root-caused it to be the SG of the member port not allowing ingress | 16:05 |
| dulek | larsks: That's why we have issues in kuryr-kubernetes where we depend on routing from the LB's. | 16:05 |
| larsks | dulek: I'm not sure why dmellado pointed you at me. dmellado?? | 16:05 |
| *** velizarx has joined #openstack-lbaas | 16:05 | |
| dmellado | larsks: there comes the hint | 16:06 |
| dmellado | the only difference we notices in between ubuntu amphora and centos amphora | 16:06 |
| dmellado | was the cloud init version | 16:06 |
| larsks | Ah, I see. I assume ubuntu is ahead of centos by a chunk. | 16:06 |
| dulek | dmellado: It's not the only one, but the only one that is suspicious to me. | 16:06 |
| dmellado | dulek: yep, let's call it the main one | 16:07 |
| larsks | And I know there has been a lot of churn in cloud-init's handling of network configuration. | 16:07 |
| dulek | cgoncalves: Yes, that was a valid workaround. The SG problem was there because now we needed to open traffic from the pod subnet (as by providing --subnet-id when creating the member the traffic comes from pod subnet). | 16:07 |
| larsks | But I haven't followed it closely for a while. | 16:07 |
| dulek | cgoncalves: Previously it came from service subnet as it was routed through default route. | 16:07 |
| dmellado | larsks: seems like ubuntu amphora is missing the default route | 16:07 |
| larsks | Is the amphora using dhcp to configure interfaces? Or is it getting a static config generated by cloud-init from network metadata? | 16:08 |
| dmellado | cgoncalves: dulek ^^ | 16:08 |
| cgoncalves | neither the amphora-haproxy netns nor its interfaces are configured by cloud-init | 16:08 |
| larsks | cgoncalves: so, ruling out cloud-init as an issue? | 16:08 |
| dulek | cgoncalves: Ah, that's interesting, so I probably wasn't right with cloud-init upgrade. | 16:08 |
| cgoncalves | larsks, IIRC the problem yes | 16:08 |
| johnsom | Can you check that the neutron subnet had a gateway? | 16:09 |
| johnsom | Yeah, that is true, only the lb-mgmt-net is configured via cloud-init | 16:09 |
| cgoncalves | the VIP interface (eth1) in centos is configured via DHCP it seems | 16:10 |
| *** velizarx has quit IRC | 16:10 | |
| johnsom | dmellado Congratulations BTW | 16:10 |
| dmellado | johnsom: thanks! | 16:10 |
| *** trown is now known as trown|lunch | 16:10 | |
| cgoncalves | so it might be getting the gateway from there | 16:10 |
| cgoncalves | https://github.com/openstack/octavia/blob/fa6e02cc8d8191c042625a352ce56e72adecce6f/octavia/amphorae/backends/agent/api_server/templates/rh_plug_vip_ethX.conf.j2#L50 | 16:11 |
| johnsom | Is it always DHCP or is it dependent on what neutron gives us? I know for Ubuntu, we decide based on what the port and subnet has from neutron. If they give us the details, we honor them, otherwise it's DHCP | 16:11 |
| dulek | johnsom: Okay, that's a very solid clue to check this in the morning. At the moment my setup has CentOS amp which works as previously. | 16:13 |
| dulek | But I can easily tell that the subnet we create loadbalancer in *has* a gateway_ip. | 16:14 |
| dulek | And it's exactly the one we should have there. ;) | 16:14 |
| johnsom | Ok, yeah, so in a one-armed amphora (where the members were not provided a subnet-id, inside the netns you should have one interface and a default route set to the gateway neutron shows for the subnet | 16:15 |
| dmellado | dulek: ok, I'll spin up a ubuntu based one so we can do A-B | 16:15 |
| dulek | johnsom: That's definitely not what I see: http://paste.openstack.org/show/748776/ | 16:15 |
| dulek | johnsom: Wait, might that "unable to resolve host amphora-6a609b93-6f92-4057-893b-8ba9fd2da0cb" be related? | 16:16 |
| dulek | Yeah, seems like I don't get this on CentOS… | 16:17 |
| johnsom | dulek No, we disable DNS | 16:17 |
| johnsom | So, wait a second, that amp isn't up. Is that the backup amphora in an active/standby pair? | 16:17 |
| dulek | johnsom: No, there's no pairing configured. | 16:18 |
| dulek | Unless the default changed recently? | 16:18 |
| johnsom | It's VIP interface doesn't have the second IP address it should have, so it's either not up yet, something else bad happened, or .... | 16:18 |
| johnsom | VIPs should always have two IPs in our current configuration. | 16:19 |
| dulek | johnsom: I have two IP's on my env with CentOS AMP. | 16:19 |
| dulek | As I said - I don't have the broken one at the moment to poke. I can fire up one tomorrow. | 16:20 |
| dulek | johnsom: What data should I gather? | 16:20 |
| johnsom | can you pastebin cat /etc/netns/amphora-haproxy/network/interfaces.d/eth1.cfg? | 16:20 |
| johnsom | Ok, that ubuntu pastebin, something is not right there. | 16:20 |
| dulek | johnsom: I'll ping you tomorrow with that file, okay? I'll also make sure to have the env ready in case you have more questions. | 16:21 |
| johnsom | I would collect the interface config file (ignore the eth1 file, it's the eth1.cfg that is used), an ip a, ip link, ip r, ip rule all from inside the netns | 16:22 |
| johnsom | Ok, sure. | 16:22 |
| johnsom | cgoncalves Want to chat about the network outage issue? | 16:23 |
| cgoncalves | johnsom, sure. I don't have logs, though, but isn't the first case reporting this behavior | 16:24 |
| johnsom | Yeah, so I'm just thinking out loud here.... | 16:25 |
| johnsom | Currently, if the HM is shutdown, on startup it waits the full HM timeout period before starting to do failovers. This is 60 seconds by default, which would give the amps six tries to report in. | 16:26 |
| cgoncalves | right | 16:26 |
| johnsom | If neutron is still hosed up and the lb-mgmt-net is still down, we currently don't have a way to know that. | 16:27 |
| johnsom | I think the original thoughts were that if it is down, that is a valid failure situation and we should be attempting to recover. | 16:28 |
| johnsom | However, currently we only have one lb-mgmt-net, so it's just cycling on it's own. | 16:28 |
| cgoncalves | believe it or not, that was my suspicious. I asked beagles a few minutes ago if we could have a dependency between them | 16:28 |
| *** cbrumm_ has quit IRC | 16:29 | |
| johnsom | Well, I would also ask, "should we have a dependency with neutron?" | 16:29 |
| dmellado | thanks johnsom | 16:29 |
| johnsom | I have considered having a dead mans hand, which attempts a connection to the amphora-agent API before a failover. | 16:30 |
| *** yamamoto has quit IRC | 16:30 | |
| johnsom | However I'm not sure that is a good idea either, as if the heartbeat isn't coming through we aren't getting status, stats, etc. | 16:31 |
| cgoncalves | agreed | 16:32 |
| johnsom | I'm guessing in your case, the failover ended up failing as neutron or nova was still down? I.e. why is a failover a problem? | 16:33 |
| johnsom | Other than just churn, which we have rate limiting for already if that is a problem. | 16:33 |
| cgoncalves | I wanted to confirm that from the logs | 16:33 |
| *** psachin has quit IRC | 16:34 | |
| johnsom | We could try pinging the neutron gateway as a test of the lb-mgmt-net, but that wouldn't catch net splits. | 16:35 |
| johnsom | I have seen the net split problem where someone messed up an MLAG and *one* rack is only partially functional. | 16:36 |
| cgoncalves | checking neutron agent status could be another option but I fear that would be too intrusive | 16:36 |
| johnsom | Yeah, and how would we know what the *right* number of happy agents are? | 16:36 |
| cgoncalves | and still doesn't guarantee flows are installed | 16:37 |
| johnsom | Right | 16:37 |
| johnsom | I also don't want to do in-band testing by connecting to the VIP, etc. as that means some controller is on tenant networks. | 16:37 |
| cgoncalves | let's have octavia agents deployed in all compute nodes and attached to lb-mgmt-net! kidding | 16:38 |
| johnsom | The good news I have had is if they are in act/stdby and this happens, the LB is still up and passing traffic even though it's in ERROR | 16:38 |
| johnsom | Lol, yeah, isn't that neutron's job? lol | 16:38 |
| johnsom | We can go down the path the appliances do, require a dedicated cable between them | 16:39 |
| johnsom | I remember the cisco one was a DB-9 proprietary serial cable. lol | 16:40 |
| cgoncalves | well, you know, someone once upon a time added a data plane status extension | 16:40 |
| cgoncalves | https://developer.openstack.org/api-ref/network/v2/?expanded=show-network-details-detail#data-plane-status-extension | 16:40 |
| johnsom | https://usercontent.irccloud-cdn.com/file/UT4G3KuG/image.png | 16:42 |
| cgoncalves | lol | 16:42 |
| johnsom | That was the actual cable... | 16:42 |
| cgoncalves | doesn't help much, the cable is broken | 16:42 |
| * johnsom The room thinks, "whatever old timer" | 16:43 | |
| johnsom | So that extension, I'm sure everyone is delploying it right? | 16:43 |
| cgoncalves | sure...... | 16:44 |
| johnsom | My devstack, HM port shows "| data_plane_status | None " | 16:44 |
| johnsom | So, yeah, it's there, but useful???? | 16:45 |
| cgoncalves | it would be VERY useful if it was in fact being used, which is not the case | 16:45 |
| johnsom | Not sure I would trust it anyway.... net splits, are likely to impact it too | 16:46 |
| cgoncalves | I know one large deployment that uses it. the folks that manage that cloud were the ones that asked me to implement it ;) | 16:47 |
| johnsom | So it monitors OVS or something else and populates that? | 16:48 |
| cgoncalves | the field is set by an 3rd party (monitoring) tool | 16:49 |
| johnsom | Right, I was just curious what it was monitoring | 16:49 |
| cgoncalves | the extension is just a parameter exposed via the API | 16:49 |
| johnsom | Right | 16:50 |
| cgoncalves | https://specs.openstack.org/openstack/neutron-specs/specs/backlog/ocata/port-data-plane-status.html | 16:50 |
| *** cbrumm_ has joined #openstack-lbaas | 16:50 | |
| *** cbrumm_ has quit IRC | 16:52 | |
| johnsom | So maybe we should open an RFE to add code that looks at that? maybe? | 16:53 |
| johnsom | Still not sure what our action would be other than trigger a failover | 16:53 |
| cgoncalves | failover is the right thing to do if we're sure lb-mgmt-net is up | 16:55 |
| johnsom | Yeah, like at least some part of it | 16:55 |
| *** luksky has joined #openstack-lbaas | 16:55 | |
| cgoncalves | maybe neutron could set dataplane_status to DOWN if a given neutron L2 agent that installs flows for a given network is down | 16:57 |
| cgoncalves | I don't like that, though. it would be a very ML2/OVS specific way | 16:58 |
| *** cbrumm has joined #openstack-lbaas | 17:00 | |
| *** trown|lunch is now known as trown | 17:00 | |
| johnsom | BTW, I have started work on adding unset to the client. I just need to figure out the tags situation and add a unit test. Then I can post an LB unset patch. The rest should be easy after that. | 17:03 |
| cgoncalves | to unset a VIP policy ID one can do --vip-policy-id None. why would we need to add unset? | 17:05 |
| johnsom | Customers expect it and don't read the comments | 17:06 |
| cgoncalves | so, "--vip-policy-id unset" or "--no-vip-policy-id"? | 17:06 |
| johnsom | https://www.irccloud.com/pastebin/0waalZhZ/ | 17:07 |
| cgoncalves | ah! | 17:07 |
| johnsom | Lol, need to fix the comment | 17:07 |
| cgoncalves | ok, makes total sense | 17:07 |
| *** jniesz has joined #openstack-lbaas | 17:11 | |
| cgoncalves | would it work not to fail over amps unless at least one heartbeat from any amp is received by the HM on start up? | 17:14 |
| *** yamamoto has joined #openstack-lbaas | 17:22 | |
| *** Vorrtex has quit IRC | 17:23 | |
| *** yamamoto has quit IRC | 17:32 | |
| *** yamamoto has joined #openstack-lbaas | 17:33 | |
| *** Vorrtex has joined #openstack-lbaas | 17:37 | |
| colin- | we have thoughts about this behavior that i just haven't captured yet johnsom cgoncalves | 17:46 |
| colin- | it's a high priority to articulate them for this community | 17:47 |
| colin- | have a deep concern over conditions that could lead to duplicate MAC addresses, and any HM/failover outcomes that might lead to it | 17:47 |
| cgoncalves | colin-, I'd love to hear more from you :) | 17:48 |
| colin- | yeah i will get a story in that describes the scenario as specifically as possible and why the outcome is undesirable | 17:48 |
| colin- | it's been a tricky one to pin down | 17:48 |
| colin- | >johnsom> BTW, I have started work on adding unset to the client. <--- cool! | 17:49 |
| *** velizarx has joined #openstack-lbaas | 17:50 | |
| *** yamamoto has quit IRC | 18:00 | |
| johnsom | cgoncalves I have +2'd the stuff we are targeting for RC2 | 18:07 |
| johnsom | Actually I think there is a problem with the spares fix | 18:14 |
| johnsom | Yeah, ok, I confirmed that the current fix does not work. | 18:26 |
| johnsom | cgoncalves Is Greg still around or should I take a pass at this patch? | 18:26 |
| johnsom | We kind of need to get that in today | 18:26 |
| *** yamamoto has joined #openstack-lbaas | 18:37 | |
| *** ricolin has quit IRC | 18:38 | |
| cgoncalves | johnsom, most likely offline. go for it as it is urgent | 18:45 |
| johnsom | Ok | 18:45 |
| *** yamamoto has quit IRC | 18:45 | |
| openstackgerrit | Merged openstack/octavia stable/queens: Fix VIP plugging on CentOS-based amphorae https://review.openstack.org/649505 | 18:47 |
| xgerman | So much scrollback today... | 19:02 |
| cgoncalves | tl;dr: Earth keeps spinning | 19:03 |
| *** salmankhan has quit IRC | 19:08 | |
| *** boden has quit IRC | 19:25 | |
| *** boden has joined #openstack-lbaas | 19:32 | |
| *** abaindur has joined #openstack-lbaas | 19:36 | |
| *** abaindur has quit IRC | 19:39 | |
| *** abaindur has joined #openstack-lbaas | 19:39 | |
| *** abaindur has quit IRC | 19:42 | |
| *** abaindur has joined #openstack-lbaas | 19:42 | |
| johnsom | #startmeeting Octavia | 19:59 |
| openstack | Meeting started Wed Apr 3 19:59:14 2019 UTC and is due to finish in 60 minutes. The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:59 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:59 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 19:59 | |
| openstack | The meeting name has been set to 'octavia' | 19:59 |
| cgoncalves | hello | 19:59 |
| johnsom | Hi folks | 19:59 |
| johnsom | poke rm_work our future PTL | 19:59 |
| johnsom | #topic Announcements | 20:00 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 20:00 | |
| johnsom | This is the final RC week. I think we need an RC2 for octavia, so we will try to get those fixes in today and hopefully do the RC2 today as well. | 20:00 |
| johnsom | We are also close to doing some stable branch releases. | 20:01 |
| johnsom | Other than that, I don't have any other announcements this week. Anyone else? | 20:01 |
| johnsom | #topic Proposal to change meeting time (cgoncalves) | 20:02 |
| *** openstack changes topic to "Proposal to change meeting time (cgoncalves) (Meeting topic: Octavia)" | 20:02 | |
| johnsom | cgoncalves Do you want to talk to this? | 20:02 |
| cgoncalves | sure, thanks | 20:02 |
| cgoncalves | so currently our weekly meetings are at 1pm PST, which means 10 pm CEST and 11 pm IST (Israel) | 20:03 |
| johnsom | 2000 UTC | 20:04 |
| cgoncalves | in Asia, it is in the middle of the night | 20:04 |
| cgoncalves | I was wondering if we could have our meetings earlier to be more friendly to folks in EMEA and Asia | 20:04 |
| cgoncalves | thanks for the correction | 20:05 |
| johnsom | Yes as long as we get quorum with the change. | 20:05 |
| cgoncalves | agreed | 20:05 |
| johnsom | (I was just adding to the conversation, grin) | 20:05 |
| xgerman | sure, what time are you proposing? | 20:05 |
| johnsom | So how this works, community process wise: | 20:06 |
| xgerman | also we should make sure rm_work is available | 20:06 |
| johnsom | 1. we propose some times/days | 20:06 |
| cgoncalves | right | 20:06 |
| johnsom | 2. I will create a doodle for those times/days | 20:06 |
| johnsom | 3. We e-mail the openstack list with the details and the doodle. | 20:06 |
| xgerman | s/I/rm_work/g | 20:06 |
| johnsom | 4. We let that soak a week, then if we have quorum for a new time, I will go update all the places that need updating and we have a new time. | 20:07 |
| cgoncalves | sounds good to me | 20:08 |
| johnsom | Questions/comments on the process? | 20:08 |
| xgerman | +1 (other than we should have rm_work own more of the process) | 20:08 |
| xgerman | I think it will stretch into when he takes over | 20:09 |
| johnsom | I would hope that rm_work would participate in the proposals. | 20:09 |
| * johnsom wonders how many times we can ping him.... grin | 20:09 | |
| cgoncalves | perhaps current meeting time is also not much convenient for rm_work | 20:09 |
| xgerman | yeah, who knows which time zone he lives by nowerdays | 20:09 |
| johnsom | So I will start, 1600 UTC is a nice time for me | 20:09 |
| cgoncalves | true. it's 5 am in Japan | 20:10 |
| xgerman | since for the people here the time works we are the wrong ones to ask to begin with | 20:12 |
| cgoncalves | let's throw more time options in to doodle. say 1500 UTC | 20:13 |
| johnsom | +1 | 20:13 |
| cgoncalves | let's also make sure we include the current meeting time | 20:13 |
| johnsom | Ok, fair point | 20:14 |
| xgerman | +1 | 20:14 |
| johnsom | Do we have any particular day that we should propose or are no-go for folks? | 20:14 |
| xgerman | Let’s stick with Wednesday - Friday/Monday are funny in a lot of time zones | 20:15 |
| cgoncalves | Fridays are no go for Israelis | 20:15 |
| johnsom | I am guessing Friday, Saturday, Sunday are bad | 20:15 |
| johnsom | Yeah, so Tue-Wed-Thur | 20:15 |
| cgoncalves | Tue-Thu | 20:15 |
| xgerman | +1 | 20:15 |
| johnsom | Ok, any other proposed times for the doodle? | 20:16 |
| cgoncalves | cool. thank you, all! | 20:16 |
| cgoncalves | johnsom, there's an option in doodle that allows anyone to add new rows (= times), no? | 20:17 |
| cgoncalves | s/rows/columns/ | 20:17 |
| johnsom | Ok. I will get the process going. | 20:17 |
| xgerman | =1 | 20:17 |
| johnsom | yes I think so, you think we should leave it open? | 20:17 |
| xgerman | but really rm_work... | 20:17 |
| cgoncalves | why not | 20:17 |
| * johnsom notes it's been a year or two since I did this | 20:17 | |
| johnsom | Ok, will do | 20:18 |
| cgoncalves | thanks | 20:18 |
| johnsom | It just means folks need to check back to it in case new times are added | 20:18 |
| johnsom | #topic Brief progress reports / bugs needing review | 20:19 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 20:19 | |
| johnsom | I have worked on removing the last references to oslosphinx which is broken with sphinx 2, deprecated, and won't be fixed. | 20:20 |
| johnsom | For the most part we have already done that, but there were two references we missed. You should not see any major changes in the docs/release notes | 20:20 |
| johnsom | I helped figure out a solution to our grenade issue. | 20:21 |
| johnsom | Lots of reviews, etc. | 20:21 |
| cgoncalves | johnsom, thank you for your help troubleshooting and proposing a fix to the grenade issue. really appreciate! | 20:22 |
| johnsom | Currently I'm working on adding the "unset" option to our openstack client. This will make it more clear for users of how to clear settings. | 20:22 |
| johnsom | I'm going to go through the main options first, then come back and do tags. | 20:22 |
| xgerman | I send back my laptop, did a week vacation… slowly getting my 2008 Mac into 2019’s software | 20:23 |
| johnsom | I need to move a module out of neutron in OSC up to osc-lib so we can share the tags code. | 20:23 |
| johnsom | Yeah, I am also running on "alternate" hardware now. Seems to be working ok though. | 20:23 |
| johnsom | I also fixed a security related issue in the OSA role. | 20:24 |
| johnsom | #link https://review.openstack.org/648744 | 20:24 |
| johnsom | xgerman You might want to do a quick review on that | 20:24 |
| xgerman | on it | 20:25 |
| johnsom | So, that is my plan for the next few days, work on unset and then tags for the client. | 20:25 |
| johnsom | Also, I will be travelling and not available much Sun-Wed. | 20:26 |
| johnsom | Just as a heads up | 20:26 |
| cgoncalves | there is a patch in master and stein that broke spare pools. Change https://review.openstack.org/#/c/649381/ will fix it. we need to backport it to stein and release Stein RC2 this week | 20:26 |
| johnsom | Yeah, I am going to take a stab at that after lunch. | 20:26 |
| cgoncalves | it would be nice if we could merge a tempest test for spare pool to prevent regressions like this in the future | 20:27 |
| cgoncalves | #link https://review.openstack.org/#/c/634988/ | 20:27 |
| johnsom | I think we just need to do another migration and we can fix it that way | 20:27 |
| cgoncalves | migration? as in DB migration? | 20:28 |
| johnsom | Yep. Good stuff. I had previously +2'd, will circle back | 20:28 |
| cgoncalves | patch set I take | 20:28 |
| johnsom | Yeah | 20:28 |
| johnsom | Any other updates this week? | 20:28 |
| cgoncalves | I also addressed reviews on https://review.openstack.org/#/c/645817/ | 20:28 |
| cgoncalves | I have a customer waiting for it | 20:29 |
| johnsom | +2, you addressed my only issue with it. | 20:29 |
| xgerman | looking | 20:30 |
| cgoncalves | thank you, thank you! | 20:30 |
| xgerman | need to see what happened to my patches... | 20:30 |
| johnsom | #topic Open Discussion | 20:31 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 20:31 | |
| johnsom | Ok, other topics this week? | 20:31 |
| cgoncalves | some folks were discussing here on the channel earlier today about an issue where health manager would trigger failover | 20:32 |
| cgoncalves | while the network was still being configured, i.e. flows, etc | 20:32 |
| xgerman | yeah, it can do that ;-) | 20:32 |
| xgerman | how do we know that the network is configures | 20:32 |
| cgoncalves | no resolution yet | 20:32 |
| xgerman | ? | 20:32 |
| cgoncalves | precisely, that's the question | 20:33 |
| johnsom | Really? The HM honors the lock on the objects, so it should not be able to start a failover if another controller owns the resouce | 20:33 |
| johnsom | Oh, you mean the neutron networking.... | 20:33 |
| johnsom | Right | 20:33 |
| xgerman | I was thinking neutron was pulling sh*t | 20:33 |
| cgoncalves | yes | 20:33 |
| cgoncalves | would it work not to fail over amps unless at least one heartbeat from any amp is received by the HM on start up? | 20:34 |
| xgerman | that goes back to should we go i to ERROR and tell the operator neutron is broken or keep retrying | 20:34 |
| xgerman | see [1] http://blog.eichberger.de/posts/yolo_cloud/ | 20:34 |
| cgoncalves | we don't know if neutron is "broken". all we know is the HM hasn't received heartbeat within the heartbeat_timeout (60 seconds) | 20:35 |
| johnsom | I suspect the issue is around net splits where some hosts and racks are working, but others are not, so any heartbeat would likely have the same issue | 20:35 |
| xgerman | cgoncalves: we had it not fail over stuff when there was no heartbeat which caused other problems | 20:35 |
| xgerman | (mainly amp doesn’t come up right and we never know) | 20:35 |
| cgoncalves | xgerman, we still do not failover on newly created LBs | 20:35 |
| xgerman | I thought we fixed that a while back... | 20:36 |
| johnsom | I thought someone was looking at that again and proposed a fix as well. Not positive though. | 20:36 |
| cgoncalves | my understanding was that it was a feature/desired behavior, not a bug | 20:36 |
| johnsom | It's a trickier problem than it seems on the surface | 20:36 |
| *** gcheresh has joined #openstack-lbaas | 20:37 | |
| xgerman | yeah, it’s the wait or trow hands-up in the air | 20:37 |
| cgoncalves | just wanted to bring this up in case anyone had some thoughts | 20:38 |
| cgoncalves | this is affecting some customers on this side | 20:38 |
| xgerman | how? OSP 13 is not HA... | 20:39 |
| cgoncalves | I heard other people also facing same issue | 20:39 |
| johnsom | Personally, if the amp isn't talking to us, it seems like it is the right answer to fail it over. The question is what to do after that, specifically if the neutron outage causes the failover to not be successful. Right now we fail "safe", in that it's marked ERROR, but the secondary amp is still passing traffic. | 20:39 |
| johnsom | lol, ouch | 20:39 |
| cgoncalves | having a HA tempest job would play a long way in having it supported in OSP | 20:40 |
| johnsom | We could have a periodic that looks for ERROR LBs with an amp in ERROR and attempts an amp failover again. We would just need to figure out the right back off and make super sure we don't bother the other functional amp. | 20:41 |
| johnsom | I think we have some work to do on the failover flows in general actually. | 20:42 |
| xgerman | mmh, I would leave that for the PTG… | 20:42 |
| cgoncalves | I don't like that, to be honest. we would be killing an amphora that is actually up by failing over not once but twice | 20:42 |
| johnsom | Yes, good topic for the PTG | 20:42 |
| xgerman | yep , agree on failover fpws | 20:42 |
| xgerman | was beating that drum for a ear now | 20:42 |
| johnsom | Added a topic to the PTG etherpad | 20:43 |
| johnsom | #link https://etherpad.openstack.org/p/octavia-train-ptg | 20:44 |
| cgoncalves | is there a reason you guys see not to like my proposed approach? | 20:44 |
| *** rcernin has joined #openstack-lbaas | 20:45 | |
| johnsom | The one heartbeat one or did I miss something? | 20:45 |
| cgoncalves | "would it work not to fail over amps unless at least one heartbeat from any amp is received by the HM on start up?" | 20:45 |
| cgoncalves | let me know if I'm not being clear | 20:46 |
| johnsom | I responded: "I suspect the issue is around net splits where some hosts and racks are working, but others are not, so any heartbeat would likely have the same issue" | 20:46 |
| cgoncalves | ah, sorry. I read that but didn't read it as a reply to my message | 20:46 |
| johnsom | We could also try to set some threshold, say if more than x amps are "down", pause failovers | 20:47 |
| *** Vorrtex has quit IRC | 20:47 | |
| johnsom | I guess now that we can mutate the config this could be more feasible now. It would allow the operator to have a knob to turn. | 20:47 |
| cgoncalves | it would be expected that sometime "soon" the network would be back again so HMs would start receiving heartbeats | 20:48 |
| johnsom | Well, there is always the "rack got zapped by a evil mastermind's laser" scenario where it will not come back | 20:49 |
| johnsom | Or the scenario I saw once, where the host was just powered off for a day | 20:50 |
| *** abaindur has quit IRC | 20:50 | |
| cgoncalves | hmm... | 20:51 |
| johnsom | That one was nice, it led to a bunch of zombie instances showing up again | 20:51 |
| cgoncalves | health manager kills them now, thanks to xgerman patch | 20:52 |
| johnsom | Stuff to think about. I captured a few on the etherpad, please add more! | 20:52 |
| johnsom | Yeah, that is some of the background on that patch | 20:52 |
| *** trown is now known as trown|outtypewww | 20:52 | |
| johnsom | We have about 5 minutes, were there other topics we needed to discuss? | 20:53 |
| johnsom | Ok, just wanted to check. Sometimes we run out of time before we discuss everything. | 20:55 |
| johnsom | Oh, BTW, the devstack patch still hasn't merged, so the barbican job is still going to fail. | 20:55 |
| cgoncalves | which devstack patch? | 20:56 |
| johnsom | #link https://review.openstack.org/648951 | 20:56 |
| cgoncalves | thanks | 20:58 |
| johnsom | Ok, sounds like we are wrapping up. Thanks folks! Have a great week. | 20:58 |
| johnsom | #endmeeting | 20:58 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Train PTG etherpad: https://etherpad.openstack.org/p/octavia-train-ptg" | 20:58 | |
| openstack | Meeting ended Wed Apr 3 20:58:30 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:58 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-04-03-19.59.html | 20:58 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-04-03-19.59.txt | 20:58 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-04-03-19.59.log.html | 20:58 |
| *** gcheresh has quit IRC | 21:04 | |
| *** pcaruana has quit IRC | 21:06 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Fix spare amphora check and creation https://review.openstack.org/649381 | 21:14 |
| cgoncalves | a db migration after all | 21:14 |
| cgoncalves | makes sense, I guess | 21:15 |
| johnsom | Yeah, I think that is the best option to not end up with a race. | 21:15 |
| *** abaindur has joined #openstack-lbaas | 21:18 | |
| *** boden has quit IRC | 21:39 | |
| *** rcernin has quit IRC | 21:42 | |
| *** velizarx has quit IRC | 21:43 | |
| *** goldyfruit has quit IRC | 21:46 | |
| *** fnaval_ has quit IRC | 22:26 | |
| openstackgerrit | Merged openstack/octavia master: Fix setting of VIP QoS policy https://review.openstack.org/645817 | 22:37 |
| openstackgerrit | Michael Johnson proposed openstack/python-octaviaclient master: Adds "unset" action to the loadbalancer command https://review.openstack.org/649758 | 22:40 |
| *** yamamoto has joined #openstack-lbaas | 22:43 | |
| *** abaindur has quit IRC | 22:44 | |
| *** yamamoto has quit IRC | 22:47 | |
| rm_work | augh for some reason this meeting isn't on my calendar anymore | 22:49 |
| johnsom | Well, the time might be changing.... | 22:49 |
| johnsom | We pinged you a few times | 22:49 |
| rm_work | still getting through scrollback | 22:51 |
| rm_work | "cgoncalves:would it work not to fail over amps unless at least one heartbeat from any amp is received by the HM on start up?" actually sounds like a decently easy switch to flip too | 22:51 |
| *** fnaval has joined #openstack-lbaas | 22:53 | |
| rm_work | yeah | 22:57 |
| rm_work | i schedule my life around my calendar meeting times | 22:57 |
| rm_work | if it's not on my calendar, i probably won't make it | 22:57 |
| rm_work | lol | 22:57 |
| rm_work | just fixing it now | 22:58 |
| *** yamamoto has joined #openstack-lbaas | 23:00 | |
| *** abaindur has joined #openstack-lbaas | 23:07 | |
| johnsom | Doodle poll: https://doodle.com/poll/9sxbzfhwirqiyqe8 | 23:11 |
| rm_work | the times constantly change on me as i travel <_< | 23:12 |
| johnsom | E-mail sent | 23:26 |
| *** abaindur has quit IRC | 23:48 | |
| *** abaindur has joined #openstack-lbaas | 23:49 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia stable/stein: Fix spare amphora check and creation https://review.openstack.org/649766 | 23:50 |
| johnsom | Cores: We would like to get these two fixes into Stein: https://review.openstack.org/#/c/649503/ and https://review.openstack.org/#/c/649766/ | 23:50 |
| johnsom | We need those landed by tomorrow | 23:51 |
| *** abaindur has quit IRC | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
