| *** zzzeek has quit IRC | 00:01 | |
| *** zzzeek has joined #openstack-lbaas | 00:02 | |
| *** ianychoi__ has quit IRC | 00:37 | |
| *** ianychoi__ has joined #openstack-lbaas | 00:37 | |
| *** zzzeek has quit IRC | 01:33 | |
| *** zzzeek has joined #openstack-lbaas | 01:34 | |
| *** zzzeek has quit IRC | 01:45 | |
| *** zzzeek has joined #openstack-lbaas | 01:48 | |
| *** zzzeek has quit IRC | 01:57 | |
| *** zzzeek has joined #openstack-lbaas | 01:59 | |
| *** dasp has quit IRC | 02:04 | |
| *** haleyb has quit IRC | 02:18 | |
| *** dasp has joined #openstack-lbaas | 02:32 | |
| *** rcernin has quit IRC | 03:24 | |
| *** rcernin has joined #openstack-lbaas | 03:27 | |
| *** zzzeek has quit IRC | 04:08 | |
| *** zzzeek has joined #openstack-lbaas | 04:09 | |
| *** vishalmanchanda has joined #openstack-lbaas | 05:38 | |
| *** mchlumsky has quit IRC | 07:33 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Fix copy output in two-node jobs https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/752936 | 07:37 |
|---|---|---|
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add new scenario test to create LB in specific AZ https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/695349 | 07:37 |
| *** gcheresh has joined #openstack-lbaas | 07:38 | |
| *** gcheresh has quit IRC | 07:39 | |
| *** gcheresh has joined #openstack-lbaas | 07:40 | |
| *** xgerman has quit IRC | 07:44 | |
| *** rpittau|afk is now known as rpittau | 07:57 | |
| *** rcernin has quit IRC | 07:57 | |
| *** vesper11 has joined #openstack-lbaas | 07:58 | |
| *** rcernin has joined #openstack-lbaas | 08:14 | |
| *** ccamposr__ has joined #openstack-lbaas | 08:18 | |
| *** rcernin has quit IRC | 08:19 | |
| *** ccamposr has quit IRC | 08:21 | |
| *** rcernin has joined #openstack-lbaas | 08:26 | |
| *** vesper11 has quit IRC | 08:30 | |
| *** rcernin has quit IRC | 08:32 | |
| *** rcernin has joined #openstack-lbaas | 08:44 | |
| *** rcernin has quit IRC | 08:50 | |
| *** rcernin has joined #openstack-lbaas | 09:01 | |
| *** rcernin has quit IRC | 09:06 | |
| *** rcernin has joined #openstack-lbaas | 09:27 | |
| *** rcernin has quit IRC | 09:43 | |
| *** yamamoto has quit IRC | 09:49 | |
| *** yamamoto has joined #openstack-lbaas | 09:49 | |
| *** yamamoto has quit IRC | 09:50 | |
| *** yamamoto has joined #openstack-lbaas | 10:07 | |
| *** yamamoto has quit IRC | 10:11 | |
| *** rcernin has joined #openstack-lbaas | 10:19 | |
| *** rcernin has quit IRC | 10:21 | |
| openstackgerrit | Gregory Thiemonge proposed openstack/octavia-tempest-plugin master: Add new scenario test to create LB in specific AZ https://review.opendev.org/c/openstack/octavia-tempest-plugin/+/695349 | 12:00 |
| *** yamamoto has joined #openstack-lbaas | 12:19 | |
| *** yamamoto has quit IRC | 12:30 | |
| *** yamamoto has joined #openstack-lbaas | 12:35 | |
| *** dulek has quit IRC | 12:43 | |
| *** ccamposr__ has quit IRC | 12:56 | |
| *** ccamposr__ has joined #openstack-lbaas | 12:56 | |
| openstackgerrit | Carlos Gonçalves proposed openstack/octavia stable/train: WIP: fix amphora image build https://review.opendev.org/c/openstack/octavia/+/774239 | 12:57 |
| *** devfaz has quit IRC | 13:15 | |
| *** yamamoto has quit IRC | 13:42 | |
| *** tkajinam has quit IRC | 13:50 | |
| *** yamamoto has joined #openstack-lbaas | 14:37 | |
| *** mchlumsky has joined #openstack-lbaas | 14:43 | |
| *** yamamoto has quit IRC | 14:53 | |
| *** rpittau is now known as rpittau|afk | 15:17 | |
| *** gcheresh has quit IRC | 15:22 | |
| *** devfaz has joined #openstack-lbaas | 15:39 | |
| *** ccamposr has joined #openstack-lbaas | 16:36 | |
| *** ccamposr__ has quit IRC | 16:36 | |
| *** xgerman has joined #openstack-lbaas | 16:47 | |
| *** yamamoto has joined #openstack-lbaas | 16:50 | |
| *** dulek has joined #openstack-lbaas | 16:59 | |
| *** yamamoto has quit IRC | 17:04 | |
| *** gcheresh has joined #openstack-lbaas | 17:32 | |
| *** gcheresh has quit IRC | 17:45 | |
| *** gthiemonge has quit IRC | 18:39 | |
| *** gthiemonge has joined #openstack-lbaas | 18:40 | |
| *** yamamoto has joined #openstack-lbaas | 19:01 | |
| *** yamamoto has quit IRC | 19:06 | |
| *** gcheresh has joined #openstack-lbaas | 19:32 | |
| *** gcheresh has quit IRC | 20:25 | |
| *** vishalmanchanda has quit IRC | 20:58 | |
| *** yamamoto has joined #openstack-lbaas | 21:02 | |
| *** yamamoto has quit IRC | 21:08 | |
| *** fnpanic has joined #openstack-lbaas | 21:24 | |
| fnpanic | Hi | 21:24 |
| fnpanic | we are using openstack ansible train release with octavia. | 21:24 |
| johnsom | Hello, welcome. | 21:25 |
| fnpanic | we see very often when we create a magnum cluster with LBs that it fails because of db deadlock | 21:25 |
| fnpanic | i searched the bugtracker and found some work with a db retry wrapper but we have most of these patches | 21:25 |
| johnsom | You should not be seeing db deadlock in the Octavia DB | 21:26 |
| johnsom | We have stress tests that create resources at crazy high rates to test for that. | 21:27 |
| fnpanic | any idea? our galera cluster seems to be fast enough. the race condition only happens when creating two lbs very fast like magnum does | 21:27 |
| fnpanic | for the etcd and api lb | 21:27 |
| fnpanic | ok so it must be something in our enviroment or something with OSA? | 21:28 |
| johnsom | I know we have run ~2000 simultaneous builds without issue, so... Can you provide the logs for this? | 21:28 |
| fnpanic | sure | 21:28 |
| johnsom | Well, I have seen this with galera mis configured before, where the query locks waiting for a secondary to sync. Or misconfigured load balancing in front of the DB where some requests get sent to the secondary by mistake. | 21:29 |
| fnpanic | oh this is a good point. | 21:30 |
| fnpanic | i will look at the lb quickly while i am triggering the race condition to have logs | 21:31 |
| johnsom | Ok, you can use paste.openstack.org for sharing logs. | 21:31 |
| fnpanic | triggered it already. heat shows this message | 21:31 |
| fnpanic | Resource CREATE failed: OctaviaClientException: resources.etcd_lb.resources.loadbalancer: Project busy. Unable to lock the project. Please try again. (HTTP 503) (Request-ID: req-f037396a-ee81-4966-85c4-0f369e2f076b) | 21:31 |
| johnsom | Ah, that is normal. That is quota management. Your automation tool should just retry. | 21:33 |
| johnsom | There are a number of cases where the API will return messages that the API is busy for a given object. Such as when a load balancer is created, it will be in a PENDING_CREATE state and will be immutable. The client just needs to retry until the object is mutable again. | 21:35 |
| johnsom | So, that HEAT template needs to be fixed to handle retry HTTP status codes. | 21:36 |
| fnpanic | are you sure? | 21:36 |
| fnpanic | hold a sec i paste the logs | 21:36 |
| johnsom | Ok | 21:36 |
| johnsom | It could be some other issue that bubbles up to the project busy message, you are correct. | 21:37 |
| *** TMM has quit IRC | 21:37 | |
| *** TMM has joined #openstack-lbaas | 21:37 | |
| fnpanic | http://paste.openstack.org/show/802410/ | 21:38 |
| fnpanic | btw just forgot octavia is really great! :-) | 21:39 |
| johnsom | Ha, thanks. | 21:39 |
| fnpanic | it works very well | 21:39 |
| johnsom | Looking | 21:40 |
| fnpanic | i will check the lb config | 21:41 |
| johnsom | Yeah, this is the quota lock. It's one of the places that requires a hard lock on the project ID to ensure the quota is maintained consistently. | 21:42 |
| fnpanic | haproxy has stick store-request src set for the backend | 21:42 |
| fnpanic | so the heat template needs a wait condition? | 21:43 |
| johnsom | Yeah, it should handle 503 and 409 as retriable status codes. To quote the HTTP specification: "The implication is that this is a temporary condition which will be alleviated after some delay." | 21:44 |
| fnpanic | ok so it is a "bug" in the magnum template or heat in general? | 21:48 |
| johnsom | Yeah, that I don't know. I really haven't worked much with either of them. | 21:48 |
| fnpanic | ok :-) | 21:48 |
| fnpanic | thanks for helping! | 21:51 |
| johnsom | Sure, any time | 21:51 |
| fnpanic | i will take a look and see what can be done | 21:51 |
| *** yamamoto has joined #openstack-lbaas | 22:15 | |
| *** dasp_ has joined #openstack-lbaas | 22:18 | |
| *** dasp has quit IRC | 22:20 | |
| *** ccamposr has quit IRC | 22:51 | |
| *** ccamposr has joined #openstack-lbaas | 22:51 | |
| *** yamamoto has quit IRC | 23:12 | |
| *** yamamoto has joined #openstack-lbaas | 23:13 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!