| *** yamamoto has joined #openstack-lbaas | 00:34 | |
| johnsom | My trip report is already nine pages long and I'm only to Friday. I guess this is a careful what you ask for moment.... | 01:28 |
|---|---|---|
| *** yamamoto has quit IRC | 01:33 | |
| *** logan- has quit IRC | 01:33 | |
| *** logan- has joined #openstack-lbaas | 01:37 | |
| *** Swami has quit IRC | 01:44 | |
| *** hongbin has joined #openstack-lbaas | 02:19 | |
| *** rcernin has quit IRC | 02:43 | |
| rm_work | :P | 03:09 |
| rm_work | i wanna read that, lol | 03:09 |
| *** ramishra has joined #openstack-lbaas | 03:10 | |
| *** yamamoto has joined #openstack-lbaas | 03:20 | |
| *** rcernin has joined #openstack-lbaas | 03:29 | |
| *** ivve has quit IRC | 04:17 | |
| *** ricolin has joined #openstack-lbaas | 04:26 | |
| *** hongbin has quit IRC | 04:33 | |
| *** AlexStaf has joined #openstack-lbaas | 04:36 | |
| cgoncalves | johnsom, 9 pages? you know it's supposed to be a report, not a novel, right? :) | 04:53 |
| johnsom | It ended up 11 pages. I have to fix the formatting though. Darn gmail. | 04:55 |
| *** gcheresh has joined #openstack-lbaas | 05:01 | |
| *** ivve has joined #openstack-lbaas | 05:20 | |
| *** ricolin has quit IRC | 05:24 | |
| *** ccamposr has joined #openstack-lbaas | 05:35 | |
| *** HW-Peter has quit IRC | 05:41 | |
| *** vishalmanchanda has joined #openstack-lbaas | 06:14 | |
| *** henriqueof has joined #openstack-lbaas | 06:18 | |
| *** pcaruana has joined #openstack-lbaas | 06:21 | |
| *** gthiemonge has joined #openstack-lbaas | 06:31 | |
| openstackgerrit | sahid proposed openstack/octavia master: db: add missing primary key in spares_pool table https://review.opendev.org/656459 | 06:42 |
| *** rpittau|afk is now known as rpittau | 06:47 | |
| *** yboaron_ has joined #openstack-lbaas | 07:05 | |
| *** ivve has quit IRC | 07:25 | |
| *** tesseract has joined #openstack-lbaas | 07:26 | |
| *** rcernin has quit IRC | 07:27 | |
| *** ivve has joined #openstack-lbaas | 07:33 | |
| *** ramishra_ has joined #openstack-lbaas | 07:39 | |
| *** ramishra has quit IRC | 07:42 | |
| *** ivve has quit IRC | 07:44 | |
| *** ivve has joined #openstack-lbaas | 07:59 | |
| *** trident has quit IRC | 08:01 | |
| *** trident has joined #openstack-lbaas | 08:02 | |
| openstackgerrit | Merged openstack/octavia stable/stein: Fix IPv6 in Active/Standby topology on CentOS https://review.opendev.org/656300 | 08:30 |
| *** pcaruana has quit IRC | 10:19 | |
| *** nmagnezi has joined #openstack-lbaas | 10:30 | |
| *** pcaruana has joined #openstack-lbaas | 10:55 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Jobboard based controller https://review.opendev.org/647406 | 10:56 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] DB objects to dicts https://review.opendev.org/657569 | 10:56 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Jobboard based controller https://review.opendev.org/647406 | 10:59 |
| *** mkuf_ has joined #openstack-lbaas | 11:13 | |
| *** rpittau has quit IRC | 11:14 | |
| *** mkuf has quit IRC | 11:16 | |
| *** rpittau has joined #openstack-lbaas | 11:17 | |
| *** yamamoto has quit IRC | 11:18 | |
| *** mkuf_ has quit IRC | 11:48 | |
| *** yamamoto has joined #openstack-lbaas | 11:49 | |
| *** yamamoto has quit IRC | 11:53 | |
| *** yamamoto has joined #openstack-lbaas | 11:53 | |
| *** yamamoto has quit IRC | 11:54 | |
| *** yamamoto has joined #openstack-lbaas | 11:54 | |
| *** yamamoto has quit IRC | 11:54 | |
| *** yamamoto has joined #openstack-lbaas | 11:56 | |
| *** boden has joined #openstack-lbaas | 11:57 | |
| *** mkuf_ has joined #openstack-lbaas | 11:57 | |
| *** mkuf has joined #openstack-lbaas | 12:00 | |
| *** mkuf_ has quit IRC | 12:03 | |
| *** osmanlicilegi has joined #openstack-lbaas | 12:10 | |
| *** mkuf has quit IRC | 12:42 | |
| *** yamamoto has quit IRC | 12:57 | |
| *** yamamoto has joined #openstack-lbaas | 12:58 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Jobboard based controller https://review.opendev.org/647406 | 13:02 |
| *** yamamoto has quit IRC | 13:03 | |
| *** mkuf has joined #openstack-lbaas | 13:29 | |
| *** yamamoto has joined #openstack-lbaas | 13:32 | |
| *** yamamoto has quit IRC | 13:37 | |
| *** yboaron_ has quit IRC | 13:58 | |
| *** Vorrtex has joined #openstack-lbaas | 14:05 | |
| *** pcaruana has quit IRC | 14:19 | |
| *** fnaval has joined #openstack-lbaas | 14:37 | |
| *** pcaruana has joined #openstack-lbaas | 14:38 | |
| openstackgerrit | Merged openstack/octavia-tempest-plugin master: Drop requirement on octavia-lib https://review.opendev.org/656845 | 14:43 |
| *** KeithMnemonic has quit IRC | 15:00 | |
| *** pcaruana has quit IRC | 15:16 | |
| *** ccamposr has quit IRC | 15:17 | |
| *** ivve has quit IRC | 15:29 | |
| xgerman | yeah, would love to see that 9 page trip report, too — will probably be like I was in the room :-) | 15:34 |
| johnsom | xgerman lol, you have seen my previous trip reports..... | 15:34 |
| xgerman | none of them were 9 pages though :-) | 15:35 |
| johnsom | Pretty close I'm sure | 15:35 |
| xgerman | I would have said maybe 2-3 pages… but 9 is a new record :-) | 15:36 |
| johnsom | It was a busy summit! | 15:36 |
| johnsom | Lots of familiar faces too | 15:36 |
| xgerman | yeah, I saw mestery made an appearance :-_ | 15:37 |
| johnsom | Yeah, didn't catch up with him sadly. Sean Collins is back, chatted with him for a bit. Carl Baldwin also stopped by the neutron dinner. | 15:40 |
| *** sapd1_x has joined #openstack-lbaas | 15:45 | |
| xgerman | Yeah, I saw Carl in the pictures — Sean is surprising. He was IBM?! | 15:48 |
| johnsom | I think he is at Comcast now | 15:49 |
| xgerman | ha, that’s where he started | 15:50 |
| *** rpittau is now known as rpittau|afk | 15:51 | |
| *** KeithMnemonic has joined #openstack-lbaas | 16:05 | |
| *** vishalmanchanda has quit IRC | 16:28 | |
| *** ivve has joined #openstack-lbaas | 16:40 | |
| *** gcheresh has quit IRC | 16:42 | |
| openstackgerrit | Margarita Shakhova proposed openstack/octavia master: Support create amphora instance from volume based. https://review.opendev.org/570505 | 16:51 |
| *** sapd1_x has quit IRC | 16:58 | |
| *** odb has quit IRC | 18:01 | |
| johnsom | FYI, I have posted our "vote on backports" change discussed at the PTG: https://review.opendev.org/#/c/657657 | 18:06 |
| cgoncalves | thanks! | 18:08 |
| johnsom | Blah, they force alpha sort order on the list. ah well. | 18:20 |
| *** tesseract has quit IRC | 19:10 | |
| *** yamamoto has joined #openstack-lbaas | 19:13 | |
| *** yamamoto has quit IRC | 19:47 | |
| *** Vorrtex has quit IRC | 20:05 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: WIP: Failover stop threshold https://review.opendev.org/656811 | 21:01 |
| johnsom | FYI, adding the stories from our PTG etherpad, so might update a few commit messages.... | 21:02 |
| colin- | someone in the group mentioned a patch when we described the mass-failover events at PTG, after we discussed it more it became less clear whether there is an upstream patch that directly addresses what we described or not. currently deployed master where https://github.com/openstack/octavia/commit/ad7e627185c62e4fd0f0b29228bf202f9578a858 was merged | 21:14 |
| colin- | if anything stands out to anybody in memory, would welcome a reminder on what that was in reference to exactly | 21:14 |
| *** boden has quit IRC | 21:16 | |
| johnsom | colin- It was this one: https://review.opendev.org/#/c/600876/ | 21:17 |
| colin- | thanks johnsom, seems that merged back in september so we should be running it | 21:18 |
| colin- | will validate | 21:18 |
| johnsom | Worth a double check. | 21:18 |
| colin- | [root@computer ~]# docker exec -it octavia-health-manager cat /usr/local/openstack/lib/python2.7/site-packages/octavia/controller/healthmanager/health_manager.py | grep db_exc.DBConnectionError | 21:28 |
| colin- | except db_exc.DBConnectionError: | 21:28 |
| colin- | definitely have that one | 21:28 |
| colin- | will see about logs | 21:28 |
| johnsom | Ok, that is as much story board 500's as I can take today. I think I captured most of the stories from the etherpad | 21:59 |
| colin- | so far it's almost exclusively this, repeating for most resources: | 22:05 |
| colin- | 2019-04-27T12:27:11.547428+00:00 computer-name octavia-health-manager WARNING octavia.controller.healthmanager.health_drivers.update_db [-] Amphora da58d20f-d34d-B33f-8f94-75d3deea70a2 health message was processed too slowly: 51.6245429516s! The system may be overloaded or otherwise malfunctioning. This heartbeat has been ignored and no update was made to the amphora health entry. THIS IS NOT GOOD.#033[00m | 22:05 |
| colin- | 2019-04-27T12:40:02.060680+00:00 computer-name octavia-health-manager WARNING octavia.controller.healthmanager.health_drivers.update_db [-] The amphora da58d20f-d34d-B33f-8f94-75d3deea70a2 with IP 172.479.821.431 is missing from the DB, so it cannot be automatically deleted (the compute_id is unknown). An operator must manually delete it from the compute service.#033[00m | 22:05 |
| colin- | looking for anything outside of those flows that stands out | 22:06 |
| johnsom | So that first one points to DB performance issues as opposed to an outage. It is saying a transaction with the database that normally takes 0.006 seconds took 51.6 to complete. | 22:08 |
| colin- | in this case a connection limit was coming into play, preventing some new connections from being created but allowing those that the select 1 db health check exception uses to remain healhty | 22:09 |
| colin- | which, i don't believe https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/health_manager.py#L113 accounts for unless i'm mistaken | 22:10 |
| johnsom | Yeah, so this seems like a new/different failure vector with the DB. I mean it did eventually complete, just after 51 seconds, so we threw it out as too old. | 22:11 |
| colin- | yeah, for sure it's going to be hard to predit every possible problem with the db | 22:12 |
| johnsom | This is a zombie: The amphora da58d20f-d34d-B33f-8f94-75d3deea70a2 with IP 172.479.821.431 is missing from the DB, so it cannot be automatically deleted | 22:12 |
| johnsom | An amp we previously deleted but as come back from the dead. | 22:13 |
| colin- | deleted from octavia's perspective | 22:13 |
| colin- | * | 22:13 |
| colin- | clearly not deleted from openstack's perspective | 22:13 |
| colin- | but yeah i wasn't confused about what that was showing | 22:13 |
| johnsom | Right, we successfully deleted it. But somehow it came back. | 22:13 |
| colin- | it wasn't deleted, the instance never was destroyed by nova | 22:13 |
| johnsom | This is the whole nova stuck in deleting thing most likely | 22:14 |
| colin- | so the line may have been gone from the amphora table but the instance wasn't deleted | 22:14 |
| colin- | we should be very clear about that to avoid any confusion imho | 22:14 |
| johnsom | Yeah, we are hosed on this as we were told it was successfully deleted..... | 22:14 |
| colin- | do you expect that any of the other processes would have logged anything telling? am continuing to browse hm for now | 22:15 |
| colin- | going to look for the delete request in nova/neutron to see if it entered their flows, i guess? | 22:16 |
| johnsom | Yeah, those zombies usually have failures in the nova logs | 22:16 |
| colin- | ok | 22:16 |
| *** ccstone has joined #openstack-lbaas | 22:20 | |
| lxkong | johnsom: could you please review the patch https://review.opendev.org/#/c/657462/ which fixes the issue we talked about yesterday? | 22:22 |
| johnsom | lxkong Yes, I saw it. I just have a big backlog of work from the PTG last week. | 22:24 |
| lxkong | johnsom: ah ok, i just wait for it to be merged and backport, so i could carry on the upgrade work | 22:26 |
| johnsom | I understand, will try to get to it soon | 22:26 |
| lxkong | thanks | 22:26 |
| *** fnaval has quit IRC | 22:51 | |
| *** threestrands has joined #openstack-lbaas | 23:00 | |
| *** threestrands has quit IRC | 23:00 | |
| *** rcernin has joined #openstack-lbaas | 23:06 | |
| *** yamamoto has joined #openstack-lbaas | 23:17 | |
| *** openstackstatus has joined #openstack-lbaas | 23:24 | |
| *** ChanServ sets mode: +v openstackstatus | 23:24 | |
| -openstackstatus- NOTICE: If your jobs failed due to connectivity issues to opendev.org they can be rechecked now. Services have been restored at that domain. | 23:26 | |
| *** fnaval has joined #openstack-lbaas | 23:36 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!