| *** livelace has joined #openstack-lbaas | 00:02 | |
| *** livelace has quit IRC | 00:12 | |
| *** yamamoto has joined #openstack-lbaas | 00:31 | |
| *** yamamoto has quit IRC | 01:25 | |
| *** sapd1 has joined #openstack-lbaas | 01:48 | |
| *** wuchunyang has joined #openstack-lbaas | 03:04 | |
| *** sapd1 has quit IRC | 03:14 | |
| *** sapd1_x has joined #openstack-lbaas | 03:14 | |
| *** wuchunyang has quit IRC | 04:01 | |
| *** gcheresh_ has joined #openstack-lbaas | 04:49 | |
| *** wuchunyang has joined #openstack-lbaas | 04:56 | |
| *** vishalmanchanda has joined #openstack-lbaas | 05:29 | |
| *** wuchunyang has quit IRC | 06:10 | |
| *** wuchunyang has joined #openstack-lbaas | 06:28 | |
| *** wuchunyang has quit IRC | 06:32 | |
| *** wuchunyang has joined #openstack-lbaas | 06:34 | |
| *** wuchunyang has quit IRC | 06:41 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 06:58 | |
| *** ccamposr__ has joined #openstack-lbaas | 07:18 | |
| *** ccamposr has quit IRC | 07:20 | |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Migrate grenade job to native Zuul v3 https://review.opendev.org/726741 | 07:33 |
|---|---|---|
| openstackgerrit | Carlos Goncalves proposed openstack/octavia stable/ussuri: Fix healthmanager not update amphora health when LB disable https://review.opendev.org/726742 | 07:33 |
| *** rpittau|afk is now known as rpittau | 07:36 | |
| *** wuchunyang has joined #openstack-lbaas | 07:50 | |
| *** eandersson7 has joined #openstack-lbaas | 08:07 | |
| *** wuchunyang has quit IRC | 08:08 | |
| openstackgerrit | Merged openstack/octavia stable/rocky: Validate resource access when creating loadbalancer or member https://review.opendev.org/722159 | 08:12 |
| *** mloza has quit IRC | 08:15 | |
| *** eandersson has quit IRC | 08:15 | |
| *** eandersson7 is now known as eandersson | 08:15 | |
| *** salmankhan has joined #openstack-lbaas | 08:35 | |
| *** salmankhan has quit IRC | 08:42 | |
| *** salmankhan has joined #openstack-lbaas | 08:42 | |
| *** livelace has joined #openstack-lbaas | 08:44 | |
| *** kevinz has quit IRC | 09:07 | |
| *** kevinz has joined #openstack-lbaas | 09:07 | |
| archiephan | some one help me with this issue https://storyboard.openstack.org/#!/story/2007635 . cc cgoncalves | 09:11 |
| *** wuchunyang has joined #openstack-lbaas | 10:04 | |
| *** wuchunyang has quit IRC | 10:10 | |
| *** happyhemant has joined #openstack-lbaas | 10:15 | |
| *** rpittau is now known as rpittau|bbl | 10:15 | |
| *** yamamoto has joined #openstack-lbaas | 10:26 | |
| *** wuchunyang has joined #openstack-lbaas | 10:45 | |
| *** yamamoto has quit IRC | 10:47 | |
| *** yamamoto has joined #openstack-lbaas | 10:54 | |
| *** wuchunyang has quit IRC | 10:55 | |
| *** wuchunyang has joined #openstack-lbaas | 10:55 | |
| *** yamamoto has quit IRC | 11:01 | |
| *** yamamoto has joined #openstack-lbaas | 11:22 | |
| *** yamamoto has quit IRC | 11:27 | |
| lxkong | johnsom: could you please help review https://review.opendev.org/#/c/726042/? | 11:41 |
| *** servagem has joined #openstack-lbaas | 11:43 | |
| *** yamamoto has joined #openstack-lbaas | 11:44 | |
| *** yamamoto has quit IRC | 11:51 | |
| *** rpittau|bbl is now known as rpittau | 11:58 | |
| *** yamamoto has joined #openstack-lbaas | 12:09 | |
| *** wuchunyang has quit IRC | 12:10 | |
| *** yamamoto has quit IRC | 12:10 | |
| *** tkajinam has quit IRC | 12:31 | |
| *** wuchunyang has joined #openstack-lbaas | 12:50 | |
| *** sapd1_x has quit IRC | 12:59 | |
| *** livelace has quit IRC | 13:04 | |
| *** livelace has joined #openstack-lbaas | 13:04 | |
| *** TrevorV has joined #openstack-lbaas | 13:25 | |
| *** zzzeek has quit IRC | 13:25 | |
| *** zzzeek has joined #openstack-lbaas | 13:26 | |
| *** zzzeek has quit IRC | 13:26 | |
| *** zzzeek has joined #openstack-lbaas | 13:27 | |
| *** livelace has quit IRC | 14:04 | |
| *** sapd1_x has joined #openstack-lbaas | 14:40 | |
| *** wuchunya_ has joined #openstack-lbaas | 14:45 | |
| *** livelace has joined #openstack-lbaas | 15:00 | |
| -openstackstatus- NOTICE: Our CI mirrors in OVH BHS1 and GRA1 regions were offline between 12:55 and 14:35 UTC, any failures there due to unreachable mirrors can safely be rechecked | 15:08 | |
| *** riuzen has joined #openstack-lbaas | 15:17 | |
| *** gcheresh_ has quit IRC | 15:21 | |
| *** armax has joined #openstack-lbaas | 15:22 | |
| *** sapd1_x has quit IRC | 15:37 | |
| *** livelace has quit IRC | 15:41 | |
| *** livelace has joined #openstack-lbaas | 15:42 | |
| *** sapd1_x has joined #openstack-lbaas | 15:52 | |
| *** sapd1_x has quit IRC | 15:56 | |
| dulek | I'm having some Internal Server Error from Amphoras on https://e6332a3d5132173ad6a0-16bb8c3ba389774898cb84788f30927d.ssl.cf2.rackcdn.com/726241/3/check/kuryr-kubernetes-tempest/74891a5/controller/logs/screen-o-cw.txt. | 15:59 |
| dulek | Seems like OVN gate is fine, so it's probably amp. | 16:00 |
| dulek | We're using the testing build, maybe that got broken? | 16:00 |
| dulek | We use bionic from here: https://tarballs.opendev.org/openstack/octavia/test-images/ | 16:02 |
| *** wuchunya_ has quit IRC | 16:04 | |
| johnsom | dulek Yeah, we don't use those so maybe there is an issue in one of those. The build job was broken for a while recently. | 16:04 |
| dulek | johnsom: Though it seems it got built today, dates seem to fit. Had you merged some change breaking compatibility recently? | 16:05 |
| johnsom | It appears haproxy took too long to start according to systemd.... Hmm, I have not seen that before. Is it a very large load balancer or overloaded host? | 16:05 |
| dulek | johnsom: It might be overloaded, it's gate. :D | 16:05 |
| dulek | johnsom: Okay, so maybe recheck will help. BTW where do I see the Amp logs? | 16:05 |
| johnsom | This is the root error: Job for haproxy-bf44a17b-1b65-4057-9b39-4499eeb5a729.service failed because a timeout was exceeded.\nSee "systemctl status haproxy-bf44a17b-1b65-4057-9b39-4499eeb5a729.service | 16:06 |
| johnsom | Well, that error is right in the cw log, but if log offloading is enabled there will be a "octavia-amphora-log.txt" in the root of the log files | 16:08 |
| dulek | johnsom: Okay, thanks! Let's see if rechecks will help, if not I'll dig this tomorrow. | 16:09 |
| *** rpittau is now known as rpittau|afk | 16:09 | |
| *** ianychoi_ is now known as ianychoi | 16:09 | |
| johnsom | dulek Yeah, let me know. If it happens again I can make some time and pull down that image. | 16:10 |
| dulek | johnsom: Sure, thanks! | 16:10 |
| *** sapd1_x has joined #openstack-lbaas | 16:13 | |
| *** maciejjozefczyk_ has joined #openstack-lbaas | 17:06 | |
| *** maciejjozefczyk has quit IRC | 17:09 | |
| *** maciejjozefczyk has joined #openstack-lbaas | 17:10 | |
| *** maciejjozefczyk_ has quit IRC | 17:14 | |
| *** ccamposr has joined #openstack-lbaas | 17:16 | |
| *** ccamposr__ has quit IRC | 17:18 | |
| *** sapd1_x has quit IRC | 17:22 | |
| *** riuzen has quit IRC | 17:47 | |
| *** gcheresh_ has joined #openstack-lbaas | 17:47 | |
| *** salmankhan has quit IRC | 17:57 | |
| *** happyhemant has quit IRC | 18:05 | |
| *** gcheresh_ has quit IRC | 18:30 | |
| *** gcheresh_ has joined #openstack-lbaas | 18:46 | |
| *** maciejjozefczyk has quit IRC | 18:54 | |
| *** gcheresh_ has quit IRC | 20:20 | |
| johnsom | rm_work This is an interesting one: https://storyboard.openstack.org/#!/story/2007635. Ghost amp with a duplicate lb-mgmt-net address. | 21:17 |
| rm_work | yeah i bet it's a cloud misconfig | 21:18 |
| johnsom | rotation should be easy to fix, have it verify like it should, but the case of a duplicate lb-mgmt-ip is interesting. | 21:18 |
| rm_work | I have seen duplicate addresses | 21:18 |
| rm_work | it's always because we had our IP allocation pools messed up somehow | 21:18 |
| johnsom | Well, I can see it happening if they have an amp record in the DB in ERROR, but the amp VM was deleted/lost in nova. | 21:19 |
| rm_work | i also bet it's actually legit a duped IP and it's actually trying to refresh the correct amp but the connection is going to the wrong one | 21:19 |
| rm_work | ahh hmm yeah i see, true | 21:19 |
| rm_work | re-allocated | 21:19 |
| johnsom | Yeah | 21:19 |
| rm_work | which yes, the issue is really the dupe IP | 21:20 |
| rm_work | and that the amp is dead | 21:20 |
| rm_work | but it's trying to cert-refresh | 21:20 |
| rm_work | so yeah, maybe just ignore ERROR amps for cert refresh like they're saying? | 21:21 |
| johnsom | Yeah. Maybe we should have some kind of duplicate IP check in the rest client. Log and pick the one not in ERROR/DELETED | 21:21 |
| rm_work | ehh | 21:21 |
| rm_work | i think that is overkill | 21:21 |
| rm_work | other systems should eventually clean up the ERROR amp | 21:21 |
| rm_work | probably if we had your failover patch landed it would have removed that one | 21:22 |
| rm_work | the fact that an amp record EXISTS for an amp that is literally not there, is the bug | 21:22 |
| johnsom | Yeah, if it failed over | 21:22 |
| rm_work | i bet it did? | 21:22 |
| rm_work | either that or they don't have healthmanager set up right | 21:22 |
| johnsom | yeah, it should have I guess. | 21:22 |
| rm_work | IMO the solution is to land the failover reflow :D | 21:23 |
| johnsom | lol, sounds good to me | 21:23 |
| johnsom | Sadly I have to stop work on the v2 version of failover, at least for now. I have an internal paperwork issue that got bumped in priority today. | 21:23 |
| johnsom | But that shouldn't stop v1 going in | 21:24 |
| rm_work | i'm +2 | 21:25 |
| rm_work | as of just now | 21:25 |
| *** servagem has quit IRC | 21:25 | |
| *** livelace has quit IRC | 21:48 | |
| *** wuchunyang has quit IRC | 21:54 | |
| *** KeithMnemonic has joined #openstack-lbaas | 22:14 | |
| *** TrevorV has quit IRC | 22:31 | |
| *** tkajinam has joined #openstack-lbaas | 22:55 | |
| openstackgerrit | Brian Haley proposed openstack/octavia master: Fix E741 pep8 errors https://review.opendev.org/726972 | 23:00 |
| haleyb | johnsom, rm_work: ^^^ that is probably needed to fix the gate, some library revved i guess, broke the ovn provider an hour ago | 23:01 |
| haleyb | feel free to edit as i'm off | 23:01 |
| johnsom | Ok, thanks! | 23:02 |
| rm_work | Kk will look | 23:02 |
| haleyb | flake8 3.8.0 most likely | 23:03 |
| haleyb | yes, released 2 hours ago | 23:04 |
| *** kevinz has quit IRC | 23:52 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!