| openstackgerrit | Merged openstack/octavia stable/queens: Fix allocate_and_associate DB deadlock https://review.opendev.org/665558 | 00:00 |
|---|---|---|
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: Add octavia-v2-dsvm-scenario-ipv6 to check queue https://review.opendev.org/594078 | 00:42 |
| *** mithilarun has joined #openstack-lbaas | 00:51 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: ACTIVE-ACTIVE: Initial distributor data model https://review.opendev.org/528850 | 00:54 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: ACTIVE-ACTIVE: Amphora driver updates https://review.opendev.org/529191 | 01:25 |
| rm_work | johnsom: ok, rebased your two patches ^^ | 01:38 |
| rm_work | and fixed some random stuff while i was there <_< | 01:38 |
| rm_work | not much tho | 01:38 |
| johnsom | Yeah, it was WIP for sure | 01:38 |
| rm_work | do i need https://review.opendev.org/#/c/537842/12 also? | 01:39 |
| johnsom | No | 01:40 |
| rm_work | or is that only for the walmart L3 method | 01:40 |
| johnsom | That is only L3 | 01:40 |
| rm_work | i'm doing this tag: https://review.opendev.org/#/q/topic:active-active-with-distributor+(status:open+OR+status:merged) | 01:41 |
| rm_work | to clarify what i am working on | 01:41 |
| rm_work | since these chains are all intermixed | 01:41 |
| johnsom | Good idea | 01:41 |
| johnsom | rm_work Do you remember how to add tables to this: https://github.com/openstack/octavia/blob/master/octavia/tests/functional/db/base.py#L39 | 01:42 |
| johnsom | Such that they are built in sqlite? | 01:42 |
| johnsom | I know I have run into this before with the brandon sqlalchemy voodoo | 01:42 |
| rm_work | uhh yeah hold on I JUST looked at this I think | 01:42 |
| johnsom | It's not creating the vip table in sqlite aparently | 01:42 |
| rm_work | hmmmm | 01:43 |
| rm_work | actually those should just happen | 01:43 |
| rm_work | i was thinking of _seed_lookup_tables | 01:43 |
| rm_work | but that's for data | 01:43 |
| rm_work | I think/ | 01:43 |
| rm_work | ? | 01:43 |
| johnsom | https://www.irccloud.com/pastebin/rTHHAelu/ | 01:43 |
| johnsom | Yeah, that is for data. | 01:44 |
| rm_work | hmm soooo | 01:44 |
| rm_work | try running just that test again | 01:44 |
| rm_work | i ran into a weird thing where it said the load_balancer table was missing | 01:44 |
| rm_work | in the middle of a test suite run, for only one test | 01:44 |
| rm_work | i think SQLite is not threadsafe so much? | 01:44 |
| rm_work | so maybe the transition between test classes it's being rebuilt or something? | 01:44 |
| johnsom | Yeah, something strange is happening, we have repo tests that use the VIP table..... | 01:45 |
| rm_work | all the tables should exist | 01:45 |
| rm_work | yes | 01:45 |
| rm_work | my advice is "try again" | 01:45 |
| johnsom | ok, so I'm doing something else wrong. | 01:45 |
| johnsom | Problem for another day. Catch you later | 01:46 |
| rm_work | kk, later :D | 01:46 |
| *** mithilarun has quit IRC | 01:47 | |
| *** mithilarun has joined #openstack-lbaas | 01:47 | |
| *** mithilarun has quit IRC | 01:52 | |
| *** hongbin has joined #openstack-lbaas | 01:58 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Allow multiple VIPs per LB https://review.opendev.org/660239 | 02:13 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: WIP: fix plugging member subnets on existing networks https://review.opendev.org/665402 | 02:18 |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 02:21 |
| *** ricolin has joined #openstack-lbaas | 02:55 | |
| *** psachin has joined #openstack-lbaas | 03:32 | |
| *** psachin has quit IRC | 04:01 | |
| *** ramishra has joined #openstack-lbaas | 04:03 | |
| *** psachin has joined #openstack-lbaas | 04:03 | |
| *** hongbin has quit IRC | 04:25 | |
| *** gcheresh has joined #openstack-lbaas | 04:49 | |
| *** vishalmanchanda has joined #openstack-lbaas | 04:55 | |
| *** gcheresh has quit IRC | 05:02 | |
| *** gcheresh has joined #openstack-lbaas | 05:20 | |
| *** yamamoto_ has joined #openstack-lbaas | 05:24 | |
| *** yamamoto has quit IRC | 05:27 | |
| *** irclogbot_2 has quit IRC | 05:30 | |
| *** irclogbot_0 has joined #openstack-lbaas | 05:31 | |
| *** fnaval has quit IRC | 05:39 | |
| *** ricolin has quit IRC | 06:18 | |
| *** ricolin has joined #openstack-lbaas | 06:19 | |
| *** gthiemon1e has quit IRC | 06:27 | |
| *** threestrands has joined #openstack-lbaas | 06:27 | |
| *** gthiemonge has joined #openstack-lbaas | 06:27 | |
| *** gcheresh has quit IRC | 06:37 | |
| *** rcernin has quit IRC | 06:51 | |
| *** ccamposr has joined #openstack-lbaas | 06:53 | |
| *** gcheresh has joined #openstack-lbaas | 07:01 | |
| *** gcheresh has quit IRC | 07:02 | |
| *** rpittau|afk is now known as rpittau | 07:02 | |
| *** gcheresh has joined #openstack-lbaas | 07:03 | |
| *** luksky has joined #openstack-lbaas | 07:06 | |
| *** ivve has joined #openstack-lbaas | 07:07 | |
| *** luksky has quit IRC | 07:11 | |
| *** tesseract has joined #openstack-lbaas | 07:20 | |
| *** luksky has joined #openstack-lbaas | 07:23 | |
| *** yamamoto_ has quit IRC | 07:54 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 07:57 |
| *** trident has quit IRC | 07:57 | |
| *** threestrands has quit IRC | 07:59 | |
| *** trident has joined #openstack-lbaas | 08:01 | |
| *** yamamoto has joined #openstack-lbaas | 08:18 | |
| *** pcaruana has quit IRC | 08:27 | |
| *** yboaron_ has joined #openstack-lbaas | 08:27 | |
| *** yamamoto has quit IRC | 08:29 | |
| *** yamamoto has joined #openstack-lbaas | 08:32 | |
| *** yamamoto has quit IRC | 08:32 | |
| *** pcaruana has joined #openstack-lbaas | 08:45 | |
| *** lemko has joined #openstack-lbaas | 09:02 | |
| *** luksky has quit IRC | 09:24 | |
| *** yamamoto has joined #openstack-lbaas | 09:48 | |
| *** yamamoto has quit IRC | 09:57 | |
| *** gcheresh_ has joined #openstack-lbaas | 09:57 | |
| *** gcheresh has quit IRC | 09:57 | |
| *** luksky has joined #openstack-lbaas | 10:07 | |
| *** rcernin has joined #openstack-lbaas | 10:18 | |
| *** yamamoto has joined #openstack-lbaas | 10:29 | |
| *** ccamposr__ has joined #openstack-lbaas | 10:38 | |
| *** ccamposr has quit IRC | 10:40 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: Transition l7policy flows to dicts https://review.opendev.org/665977 | 10:48 |
| *** gcheresh has joined #openstack-lbaas | 11:33 | |
| *** gcheresh_ has quit IRC | 11:33 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia-tempest-plugin master: WIP: Test Additional VIPs https://review.opendev.org/664462 | 11:38 |
| *** yamamoto has quit IRC | 11:57 | |
| *** rcernin has quit IRC | 12:15 | |
| *** goldyfruit has quit IRC | 12:25 | |
| *** yamamoto has joined #openstack-lbaas | 12:41 | |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Convert pool flows to use dicts https://review.opendev.org/665381 | 12:44 |
| openstackgerrit | Elod Illes proposed openstack/neutron-lbaas stable/stein: neutron-lbaas haproxy agent prevent vif unplug when failover occurs https://review.opendev.org/666280 | 12:54 |
| luksky | hello, I have following error during creation of loadbalancer: 2019-06-19 15:07:30.827 2525 DEBUG neutronclient.v2_0.client [req-c749b070-7fea-4e13-a8ed-f6e941ea0f63 - 2037dd6132e348749e830b89af87df7d - default default] Error message: {"message": "The resource could not be found.<br /><br />\nExtension with alias qos does not exist\n\n", "code": "404 Not Found", "title": "Not Found"} _handle_fault_response /usr/local/lib/python2.7/dist-packages/neu | 13:10 |
| luksky | tronclient/v2_0/client.py:259 | 13:10 |
| luksky | what does it mean ? | 13:10 |
| cgoncalves | luksky, it is not an error message. Octavia checks if QoS is enabled in Neutron and based on that makes decisions on what to configure or skip | 13:15 |
| luksky | ok, so nothing to worry about ? | 13:16 |
| cgoncalves | in your case, QoS is not enabled. actions like setting a QoS policy on the VIP will not work and Octavia will tell you so | 13:16 |
| cgoncalves | yes | 13:16 |
| luksky | ok, thx | 13:16 |
| openstackgerrit | Ann Taraday proposed openstack/octavia master: [WIP] Convert pool flows to use dicts https://review.opendev.org/665381 | 13:22 |
| *** goldyfruit has joined #openstack-lbaas | 13:26 | |
| *** lemko has quit IRC | 13:32 | |
| luksky | one more question: | 13:50 |
| luksky | 2019-06-19 15:49:37.991 1666 DEBUG octavia.controller.worker.amphora_rate_limit [-] Available build slots 0 has_build_slot /usr/local/lib/python2.7/dist-packages/octavia/controller/worker/amphora_rate_limit.py:52 | 13:50 |
| luksky | I'm trying to build LB, but nothing happen | 13:50 |
| luksky | get only this line: 2019-06-19 15:49:21.026 1663 INFO octavia.api.v2.controllers.load_balancer [req-ee512b75-d1ec-4f16-a7cd-e7c277062d3e - 2037dd6132e348749e830b89af87df7d - default default] Sending created Load Balancer 9518ef5c-c078-4373-b048-cbf4aff4d01f to the handler | 13:51 |
| luksky | and nothing more is happening | 13:51 |
| johnsom | luksky That is a debug message that has no impact on the build. It's just saying it got a build slot from the rate limiting which probably means you have rate limiting disabled | 13:51 |
| johnsom | luksky Is your controller worker process running? | 13:52 |
| luksky | yes | 13:52 |
| luksky | [root@octavia-01 ~]# systemctl status octavia-worker.service | 13:52 |
| luksky | * octavia-worker.service - OpenStack Octavia Worker service | 13:52 |
| luksky | Loaded: loaded (/lib/systemd/system/octavia-worker.service; enabled; vendor preset: enabled) | 13:52 |
| luksky | Active: active (running) since Wed 2019-06-19 15:47:53 CEST; 4min 26s ago | 13:52 |
| luksky | Main PID: 1679 (octavia-worker:) | 13:52 |
| luksky | CGroup: /system.slice/octavia-worker.service | 13:52 |
| luksky | |-1679 octavia-worker: master process [/usr/local/bin/octavia-worker --config-file /etc/octavia/octavia.conf --log-file /var/log/octavia/worker.log | 13:52 |
| luksky | |-1696 octavia-worker: ConsumerService worker(0) | 13:52 |
| luksky | `-1698 octavia-worker: ConsumerService worker(1) | 13:52 |
| johnsom | Check the log from that process, it is the one that will get that message from the API and run it | 13:52 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1698]: 2019-06-19 15:47:58.720 1698 DEBUG cotyledon.oslo_config_glue [-] neutron.ca_certificates_file = None log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2585 | 13:53 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1698]: 2019-06-19 15:47:58.725 1698 DEBUG cotyledon.oslo_config_glue [-] neutron.endpoint = None log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2585 | 13:53 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1698]: 2019-06-19 15:47:58.728 1698 DEBUG cotyledon.oslo_config_glue [-] neutron.endpoint_type = internalURL log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2585 | 13:53 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1696]: 2019-06-19 15:47:58.728 1696 INFO octavia.controller.queue.consumer [-] Starting consumer... | 13:53 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1698]: 2019-06-19 15:47:58.730 1698 DEBUG cotyledon.oslo_config_glue [-] neutron.insecure = False log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2585 | 13:53 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1698]: 2019-06-19 15:47:58.741 1698 DEBUG cotyledon.oslo_config_glue [-] neutron.region_name = None log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2585 | 13:53 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1698]: 2019-06-19 15:47:58.741 1698 DEBUG cotyledon.oslo_config_glue [-] neutron.service_name = neutron log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2585 | 13:53 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1698]: 2019-06-19 15:47:58.742 1698 DEBUG cotyledon.oslo_config_glue [-] ******************************************************************************** log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2587 | 13:53 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1698]: 2019-06-19 15:47:58.749 1698 DEBUG cotyledon._service [-] Run service ConsumerService(1) [1698] wait_forever /usr/local/lib/python2.7/dist-packages/cotyledon/_service.py:241 | 13:53 |
| luksky | Jun 19 15:47:58 octavia-01 octavia-worker[1698]: 2019-06-19 15:47:58.760 1698 INFO octavia.controller.queue.consumer [-] Starting consumer... | 13:53 |
| luksky | 2019-06-19 15:47:58.760 1698 INFO octavia.controller.queue.consumer [-] Starting consumer... | 13:53 |
| luksky | this last line | 13:53 |
| luksky | and nothing more | 13:53 |
| johnsom | Ok, so your rabbit queue (oslo messaging) is not working. The controller-worker didn't get the message from the API process. I sould check you transport and queue configuration in both of the two processes. | 13:54 |
| luksky | ok, will check | 13:54 |
| *** yboaron_ has quit IRC | 14:22 | |
| *** ivve has quit IRC | 14:23 | |
| *** fnaval has joined #openstack-lbaas | 14:37 | |
| *** gcheresh has quit IRC | 14:46 | |
| *** luksky has quit IRC | 14:50 | |
| zigo | Hi there! We current have so far been running Octavia in SINGLE mode, and want to migrate our load balancers to ACTIVE_STANDBY. These load balancers are in production, unfortunately, but I guess some short down time is acceptable. What's the procedure ? Will I have to re-create my load balancers? | 14:54 |
| johnsom | zigo The official answer is you have to re-create them as we have no tested procedure for migrating them, nor an RFE for adding a method to do so. | 14:55 |
| johnsom | In other words, you are on your own on that and let us know if you figure out how to make it work. grin | 14:56 |
| zigo | johnsom: Is it fine to reconfigure Octavia to ACTIVE_STANDBY, then delete the load balancer, and recreate them? | 14:56 |
| zigo | I wont attempt migration ... | 14:58 |
| zigo | johnsom: It's a shame the switch is on octavia.conf, and not as an attribute for balancers. | 14:58 |
| johnsom | zigo Yes, that should be fine. If you have Stein deployed you can create flavors and have a mix in your deployment | 14:58 |
| zigo | johnsom: We're running Rocky. | 14:58 |
| johnsom | zigo Yeah, sorry, we added that feature in Stein. | 14:59 |
| zigo | Because my colleagues fear too much running Buster in production until it's released on the 6th of July, and I've packaged Stein only for Buster... | 14:59 |
| zigo | johnsom: Packaging for both Stretch and Buster would have been too much work ... :P | 15:00 |
| *** Vorrtex has joined #openstack-lbaas | 15:12 | |
| *** yamamoto has quit IRC | 15:15 | |
| openstackgerrit | jacky06 proposed openstack/octavia-tempest-plugin master: Sync Sphinx requirement https://review.opendev.org/666316 | 15:26 |
| rm_work | #startmeeting Octavia | 16:00 |
| openstack | Meeting started Wed Jun 19 16:00:02 2019 UTC and is due to finish in 60 minutes. The chair is rm_work. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Octavia)" | 16:00 | |
| openstack | The meeting name has been set to 'octavia' | 16:00 |
| cgoncalves | o/ | 16:00 |
| rm_work | o/ | 16:00 |
| gthiemonge | o/ | 16:00 |
| johnsom | o/ | 16:00 |
| *** ataraday_ has joined #openstack-lbaas | 16:00 | |
| rm_work | #topic Announcements | 16:01 |
| *** openstack changes topic to "Announcements (Meeting topic: Octavia)" | 16:01 | |
| rm_work | So, the TC has decided to stop tracking the health of project teams | 16:02 |
| rm_work | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007085.html | 16:02 |
| rm_work | So... I guess no more worrying about that | 16:02 |
| johnsom | Yeah, probably for the best. It was a strange process anyway. | 16:02 |
| cgoncalves | this is what it looked like before for Octavia | 16:02 |
| cgoncalves | #link https://wiki.openstack.org/w/index.php?title=OpenStack_health_tracker&direction=prev&oldid=170660#Octavia | 16:02 |
| johnsom | Yeah, the paraphrasing was, interesting | 16:03 |
| johnsom | I think they only did that once anyway. | 16:03 |
| rm_work | Any other announcements? | 16:05 |
| johnsom | Shanghai call for papers is open and end in less than a month. | 16:05 |
| johnsom | deadline: July 2, 2019 at 11:59pm PT | 16:06 |
| rm_work | Ah yes. I'm not sure what we want to submit this time. | 16:06 |
| johnsom | #link http://cfp.openstack.org/?_ga=2.124047753.2032053596.1560783728-1706076231.1557509450 | 16:06 |
| rm_work | I'm aiming to go -- who else is? | 16:06 |
| cgoncalves | dulek and I submitted a proposal today on kuryr and octavia | 16:07 |
| johnsom | I am planning to not attend in person, but could be available virtual. | 16:07 |
| rm_work | ok, so cgoncalves you will be there?] | 16:07 |
| johnsom | Happy to help folks with slides too | 16:07 |
| cgoncalves | rm_work, I have no idea | 16:08 |
| rm_work | Then we could maybe do one of the normal presentations... if we think it's useful | 16:08 |
| rm_work | ah, hmmm | 16:08 |
| rm_work | well, ok | 16:08 |
| cgoncalves | having a session helps but does not guarantee a lottery ticket | 16:08 |
| johnsom | It would be great if we can do the project update at least. | 16:08 |
| rm_work | heh | 16:08 |
| rm_work | yeah but that isn't part of the CFP | 16:08 |
| rm_work | I am planning to do that | 16:09 |
| johnsom | Correct, you as PTL should get an e-mail about the project update sessions | 16:09 |
| rm_work | so maybe we're good enough with just that and onboarding, which they said will be part of the PTG side this time | 16:09 |
| rm_work | alright, is that it for announcements? | 16:10 |
| *** rpittau is now known as rpittau|afk | 16:10 | |
| *** ramishra has quit IRC | 16:11 | |
| rm_work | ok | 16:11 |
| rm_work | #topic Brief progress reports / bugs needing review | 16:11 |
| *** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)" | 16:11 | |
| ataraday_ | I started a couple of new transition to dicts changes, and review needed for #link https://review.opendev.org/#/c/662791/ and #link https://review.opendev.org/#/c/659538/ | 16:12 |
| johnsom | Log offloading is done and merged. I still want to see if I can get creative with a tempest test for that. | 16:12 |
| rm_work | I've been working on multiple things recently, the biggest of which is MultiVIP support, which could use reviews: https://review.opendev.org/#/c/660239/ | 16:12 |
| rm_work | #link https://review.opendev.org/#/c/660239/ | 16:12 |
| johnsom | Currently I'm working on some octavia-lib enhancements and a functional test for the driver-agent. | 16:12 |
| gthiemonge | I have some changes related to UDP LB that need reviews: https://review.opendev.org/#/q/status:open+project:openstack/octavia+branch:master+topic:udp_states | 16:13 |
| rm_work | Also working on some changes to the member subnet plugging calculations/handling, to resolve issues plugging additional subnets on the same network | 16:13 |
| rm_work | #link https://review.opendev.org/#/c/665402/ | 16:13 |
| johnsom | I have also put up some patches removing references to neutron-lbaas from the neutron and neutron-lib repos. | 16:14 |
| cgoncalves | I resumed work on the VIP ACL RFE side but progresses slowly. the octavia-lib patch is ready for review | 16:15 |
| johnsom | And finally, I did a PoC switching ubutnu over to the -kvm kernel for the image buids. It saves ~200MB in space for the image by removing a bunch of kernel modules we don't need. I have some cleanup to do in DIB for that, but look for that soon. | 16:15 |
| cgoncalves | #link https://review.opendev.org/#/q/topic:vip-acl | 16:15 |
| johnsom | So comparing the old kernel to the new, size: 605397504 vs 393244160 according to glance | 16:16 |
| cgoncalves | the active-standby tempest scenario patch merged. I have to go now enable the jobs also in octavia | 16:16 |
| johnsom | -rw-r--r-- 1 stack stack 376M Jun 17 18:59 amphora-x64-haproxy.qcow2.kvm | 16:16 |
| johnsom | -rw-r--r-- 1 stack stack 578M Jun 11 17:18 amphora-x64-haproxy.qcow2.orig | 16:16 |
| rm_work | noice, now just need to make centos not huge :D | 16:17 |
| * johnsom hold my coffee | 16:17 | |
| rm_work | Ok, cool, lots of work going on | 16:18 |
| cgoncalves | ah, I also propose an octavia-tempest-plugin tag release: https://review.opendev.org/#/c/666037/ | 16:18 |
| rm_work | Ah, before Open Discussion, I think gthiemonge did have a topic? Guess it wasn't added to the agenda page | 16:19 |
| rm_work | gthiemonge: i forgot what exactly it was, hopefully you remember :D | 16:19 |
| gthiemonge | oh yes, we were talking about UDB LB that mixes IPv4 and IPv6 | 16:20 |
| gthiemonge | UDP | 16:20 |
| gthiemonge | currently, we can create a such LB with members, but keepalived keeps crashing because it doesn't support mixing IPv4/IPV6 | 16:20 |
| johnsom | Ah, yeah, so LVS doesn't/didn't support mixing the VIP and member protocol versions. | 16:20 |
| gthiemonge | so we want to find a good way to handle this | 16:21 |
| johnsom | I thought there was a check in the API that blocked it.... Maybe that was missed. | 16:21 |
| rm_work | Yeah... so... do we try to validate the members that are added? | 16:21 |
| rm_work | Is that the right approach? | 16:21 |
| rm_work | and how does that work with multivip? if you have both ipv4 and ipv6... do you allow both kinds of members, but only add the ones that match the address-family for each individual vip? | 16:22 |
| rm_work | that could be confusing | 16:22 |
| rm_work | like if you add three members, 2x IPv4 and 1x IPv6, the IPv4 VIP would balance between two of them, and the IPv6 VIP would go directly to one | 16:23 |
| rm_work | seems like it's very non-intuitive | 16:24 |
| johnsom | There might be a way to make it work. Someone should spend some quality time with the keepalived bug list on github and see if there is a fix or workaround. | 16:24 |
| rm_work | well, I don't know if it is literally possible to route UDP cross-family | 16:24 |
| johnsom | Sure, it's just the IP wrapper that needs NAT really | 16:25 |
| rm_work | can LVS do the necessary packet work? | 16:25 |
| johnsom | I don't know. Like I said, you may be able to work around it with some iptables NAT rules. | 16:26 |
| rm_work | hmm | 16:26 |
| rm_work | k, need some help probably from someone who understands the low level networking aspects of this better than I do :) | 16:27 |
| johnsom | I know at the time of the UDP work it was identified as a problem, so we did a release note about it. But I don't think it was investigated at all | 16:27 |
| johnsom | I'm seeing some comments that this was fixed in the kernel. So, maybe needs a re-test or test for that matter. | 16:28 |
| johnsom | #link https://github.com/acassen/keepalived/issues/876 | 16:28 |
| rm_work | k | 16:28 |
| rm_work | yep, seems like it should work | 16:31 |
| gthiemonge | I will test it | 16:31 |
| rm_work | so, ok. just need to fix that. I wonder if moving to a new enough keepalived will be difficult | 16:31 |
| rm_work | or kernel... is cent7 still on 2.6.x? | 16:31 |
| johnsom | Thank you gthiemonge. We should be able to solve it one way or another. | 16:32 |
| rm_work | or did they get to 3.x yet | 16:32 |
| johnsom | Pretty sure it's 3.x | 16:33 |
| johnsom | Let's check the log offload.... grin | 16:33 |
| cgoncalves | fully loaded with feature backports, I must add | 16:33 |
| rm_work | need at least 3.18 | 16:34 |
| johnsom | Linux version 3.10.0-957.21.2.el7.x86_64 | 16:34 |
| rm_work | also i was joking but i guess maybe it still is that old, rofl | 16:34 |
| * rm_work dies | 16:34 | |
| johnsom | Yeah, but it's hard to say if that feature was backported | 16:34 |
| johnsom | #link http://logs.openstack.org/29/665029/3/check/octavia-v2-dsvm-py2-scenario-centos-7/735918f/controller/logs/octavia-amphora_log.txt.gz#_Jun_19_00_28_26 | 16:34 |
| rm_work | rofl ok, so yes, actually ancient | 16:35 |
| cgoncalves | if not, we could try to check with the kernel team if it's possible | 16:35 |
| rm_work | can we immediately drop cent7 support once cent8 is ready? >_> | 16:35 |
| cgoncalves | yes IMO | 16:35 |
| rm_work | i guess we can check for HAVE_DECL_IPVS_DEST_ATTR_ADDR_FAMILY | 16:35 |
| rm_work | in keepalived/check/libipvs.c | 16:36 |
| cgoncalves | our commercial Stein-based product will be fully on RHEL 8 | 16:36 |
| rm_work | err or is it in the kernel's configure.ac | 16:36 |
| rm_work | well anyway yeah, we can check | 16:36 |
| rm_work | ubuntu should be on 4.x so no issues right | 16:37 |
| rm_work | and keepalived version 1.4.5+ probably | 16:37 |
| johnsom | Linux version 4.4.0-151-generic | 16:37 |
| rm_work | hmm no | 16:37 |
| rm_work | bionic even still has 1.3.9 | 16:37 |
| rm_work | disco as 2.0 .... | 16:38 |
| rm_work | even cosmic is still 1.3.9 | 16:38 |
| rm_work | that's problematic | 16:38 |
| rm_work | i wonder if it can be backported | 16:38 |
| johnsom | keepalived amd64 1:1.3.9-1ubuntu0.18.04.2 | 16:39 |
| johnsom | Again, hard to say on what they pulled back, etc. | 16:39 |
| rm_work | yeah gross | 16:39 |
| rm_work | i mean | 16:39 |
| johnsom | We are just too cutting edge... lol | 16:40 |
| rm_work | i don't think they would have backported a ton of features from 1.4.x right? wouldn't they just... RUN 1.4.x in that case? | 16:40 |
| rm_work | anywho, we can work this out | 16:40 |
| rm_work | #topic Open Discussion | 16:41 |
| *** openstack changes topic to "Open Discussion (Meeting topic: Octavia)" | 16:41 | |
| johnsom | I gave up trying to guess that stuff a long time ago | 16:41 |
| johnsom | Just a quick qeustion. For the driver-agent functional tests, I need to create real files on the filesystem. I'm currently generating unique files in /tmp for that. Is that the right approach? | 16:43 |
| colin- | are their contents meaningful? | 16:43 |
| johnsom | I need to open the Unix domain sockets and create a DB file for sqlite. | 16:43 |
| johnsom | The test has a cleanup hook to remove them | 16:43 |
| johnsom | Basically they are the live driver agent sockets the tests will use. We will be firing up a driver-agent, without the full devstack for the functional tests. | 16:44 |
| cgoncalves | I think that is fine | 16:45 |
| johnsom | They are all uuid'd so they won't conflict with others running the same tests, etc. | 16:45 |
| colin- | i would do it in /tmp, too | 16:46 |
| johnsom | Cool, I thought so, just thought I would ask. I should have something posted for review today on that. | 16:46 |
| *** ricolin has quit IRC | 16:48 | |
| colin- | do you folks think there is anything in the 2.0 release of haproxy that we should especially be looking forward to? most of what i'm interested in is outside the scope of octavia (prometheus scraping, for example) | 16:49 |
| johnsom | I'm excited about the HTTP/2 work personally. | 16:50 |
| rm_work | fyi looks like newer keepalived is needed for other reasons in bionic: https://bugs.launchpad.net/ubuntu/+source/keepalived/+bug/1819074 | 16:50 |
| openstack | Launchpad bug 1819074 in systemd (Ubuntu) "Keepalived < 2.0.x in Ubuntu 18.04 LTS not compatible with systemd-networkd" [Undecided,Confirmed] | 16:50 |
| rm_work | so... it could happen | 16:50 |
| colin- | good point johnsom more streaming and fewer conn brokering is always a good thing | 16:51 |
| johnsom | Yeah, they have backported newer versions of haproxy for us in the past. Just need to request it on launchpad and reference the version in a newer release. | 16:51 |
| rm_work | yes, 2.0 looks quite good actually | 16:51 |
| rm_work | also the dataplane api will be interesting | 16:51 |
| rm_work | I made this "question" but not sure if it should just be a bug: https://answers.launchpad.net/ubuntu/+source/keepalived/+question/681490 | 16:52 |
| johnsom | colin- The challenge we have is that the distros won't have 2.0 for a while, so it's a decision if we want to do things that require custom built images... | 16:52 |
| johnsom | rm_work In the past I have just opened a backport request bug. | 16:52 |
| rm_work | 2.1 will bring UDP loadbalancing to haproxy :D | 16:52 |
| colin- | ah, i see yeah | 16:52 |
| johnsom | I think they have a process | 16:52 |
| rm_work | but yeah, a couple new protocols will be nice, http2 especially | 16:54 |
| johnsom | We had a topic about that at the last PTG. Even 1.9 brings some really useful stuff like fully functional threading. | 16:54 |
| rm_work | i wonder if we could just maintain an octavia packages repo <_< | 16:54 |
| rm_work | for haproxy and keepalived | 16:55 |
| rm_work | lol | 16:55 |
| johnsom | And the kernel... Oh, wait. | 16:55 |
| johnsom | I think we just need to come up with a strategy of how we handle such things and document it. | 16:56 |
| rm_work | i mean it'd be possible with periodics | 16:56 |
| rm_work | we could build them in-gate and upload them to the openstack artifact store | 16:56 |
| rm_work | or our own PPA | 16:56 |
| johnsom | For example, if someone asks for HTTP/2 via the API but the amp we get doesn't have a compatible version of HAproxy. | 16:56 |
| johnsom | We can detect that, but do we just ERROR out the object? Ignore the setting and fall back to HTTP 1.1, etc. | 16:57 |
| rm_work | probably safe-fallback strategies | 16:57 |
| rm_work | or the admin configures which protocols are available | 16:58 |
| rm_work | based on the amp version they run | 16:58 |
| cgoncalves | ERROR out at API with a suggestion to an alternative (fall back to HTTP 1.1) | 16:58 |
| johnsom | How would they know? | 16:58 |
| rm_work | cgoncalves: API can't know | 16:58 |
| rm_work | unless admin configures it | 16:58 |
| cgoncalves | ah, right | 16:58 |
| johnsom | Right, things to think about. We can continue the discussion next week. | 16:59 |
| rm_work | yep, good meeting everyone | 16:59 |
| cgoncalves | +1 | 16:59 |
| rm_work | #endmeeting | 16:59 |
| *** openstack changes topic to "Discussions for OpenStack Octavia | Train PTG etherpad: https://etherpad.openstack.org/p/octavia-train-ptg" | 16:59 | |
| openstack | Meeting ended Wed Jun 19 16:59:31 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-06-19-16.00.html | 16:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-06-19-16.00.txt | 16:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/octavia/2019/octavia.2019-06-19-16.00.log.html | 16:59 |
| rm_work | Oh, I forgot to mention I'm about to be working on Active-Active | 16:59 |
| rm_work | ah well | 17:00 |
| colin- | \o/ | 17:00 |
| colin- | i'll rejoice | 17:00 |
| johnsom | Flavors might be a way to solve this too. | 17:00 |
| emccormick | Can anyone think of a reason why octavia-api would report SubnetNotFound when it definitely exists? | 17:00 |
| colin- | does it behave that way with both the UUID and name? | 17:01 |
| emccormick | it's actually managing to translate name to id and then reporting the ID as not found. | 17:01 |
| emccormick | it does | 17:01 |
| emccormick | and I can see Neutron Server returning a 200 to the query at least | 17:01 |
| rm_work | colin-: if you want to mention the multi-vip stuff to the senlin folks... would maybe be useful to have senlin make LBs that can do ipv4 and ipv6 because right now it can't | 17:01 |
| johnsom | Is the subnet visible to the right account/project? | 17:01 |
| *** trident has quit IRC | 17:02 | |
| colin- | will do, good idea | 17:02 |
| emccormick | I'm creating the loadbalancer with a --project the same as the one the subnet is in... | 17:02 |
| emccormick | let me double check user permissions there | 17:02 |
| *** trident has joined #openstack-lbaas | 17:04 | |
| emccormick | The user creating the loadbalancer is admin, load-balancer-admin, and _member_ in the project | 17:05 |
| rm_work | i'm not sure if the user project is relevant in this case | 17:05 |
| rm_work | octavia uses its own service account for all of these calls | 17:06 |
| emccormick | and as that user I can query all the properties of the network. In fact this user created the networks to begin with | 17:06 |
| rm_work | we only forward a user-token for barbican interactions specifically | 17:06 |
| emccormick | well those at least used to work. I have existing loadbalancers elsewhere | 17:06 |
| rm_work | check if the octavia user has neutron admin | 17:06 |
| johnsom | rm_work No, users can't create VIPs on other projects. | 17:06 |
| rm_work | i believe it should as that's necessary to do a lot of the port attachments tho? | 17:07 |
| rm_work | since we plug ports on other projects' networks | 17:07 |
| johnsom | Meaning you can't create a VIP on another project's subnet. Neutron is not helpful with this as it returns not found even when it's a permission denied | 17:07 |
| johnsom | rm_work We can, but we don't allow users to do it. That would be *bad* | 17:08 |
| rm_work | right | 17:08 |
| rm_work | err actually, do we check subnet ownership? | 17:08 |
| emccormick | hmm. So which role am I looking for where? | 17:08 |
| rm_work | emccormick: are you doing this with the Openstack CLI? I assume so if you're seeing name->id translation | 17:09 |
| emccormick | yeah CLI | 17:09 |
| johnsom | emccormick Sorry for the tangent. If the user you are creating the load balancer with can do a subnet show with that ID this is something else. | 17:09 |
| emccormick | I ran the create in debug | 17:09 |
| rm_work | so that lookup is happening with your user, but the actual operation on the API side is with the octavia user | 17:09 |
| emccormick | I can do a subnet show with name or ID | 17:09 |
| johnsom | It could be that it's the lb-mgmt-net configured subnet it's not finding too | 17:09 |
| emccormick | if I feed the name to Octavia it gives me back the correct ID and says it's not found which is kinda nutty | 17:10 |
| emccormick | since it had to find it to get the name | 17:10 |
| emccormick | hmm | 17:10 |
| rm_work | octavia isn't really doing the translation, the cli is | 17:10 |
| johnsom | Right. My guess is the lb-mgmt-net configuration is bad | 17:10 |
| rm_work | the cli does that stuff pre-octavia and then just calls octavia with the translated IDs | 17:10 |
| emccormick | ah ok. That makes sense | 17:10 |
| rm_work | johnsom: it would make sense to me that the octavia user might not be able to see a private subnet? | 17:11 |
| rm_work | is this for the vip subnet or a member add? | 17:11 |
| emccormick | The ID it feeds me back as not found is the vip subnet | 17:12 |
| rm_work | hmm ok | 17:12 |
| rm_work | johnsom: so we do checking on the ownership of the vip subnet matching the project of the LB? | 17:12 |
| emccormick | openstack --debug loadbalancer create --project foo --name foo-wp-lb --vip-subnet-id foo-subnet1 | 17:12 |
| rm_work | that makes sense but didn't remember if we actually did it | 17:12 |
| johnsom | Yeah, if you could paste.openstack.org the error from running the command via --debug would be helpful | 17:12 |
| emccormick | that's what I'm doing | 17:13 |
| emccormick | sure thing | 17:13 |
| johnsom | Out of curiosity, why are you passing a project on the command line? Are you trying to create the LB on a different project than the current user? | 17:14 |
| emccormick | http://paste.openstack.org/show/753197/ | 17:16 |
| emccormick | johnson Yeah I'm just on as my generic admin user. The load balancer is in a client's project | 17:17 |
| emccormick | FWIW I have working load balancers previously created | 17:17 |
| emccormick | going to try and make another in a project that already has one and see what happens | 17:17 |
| johnsom | Yeah, I guess the next thing to check is the neutron svc log to see why it's rejecting it. | 17:19 |
| emccormick | should the request ID to neutron be the one at the bottom of that pastebin? | 17:20 |
| *** tesseract has quit IRC | 17:21 | |
| johnsom | I'm not 100% sure. I think we have some bugs around the request IDs. Here I think it will be in neutron, but not 100% sure. | 17:22 |
| johnsom | You can always look for the subnet ID too though. | 17:22 |
| emccormick | nm, I think I see the neutron one farther up. My kingdom for global request IDs to get finished. | 17:23 |
| johnsom | Yeah, we do need to get to fixing that | 17:23 |
| *** ccamposr has joined #openstack-lbaas | 17:23 | |
| *** ccamposr__ has quit IRC | 17:26 | |
| *** ccamposr__ has joined #openstack-lbaas | 17:29 | |
| *** ccamposr has quit IRC | 17:32 | |
| *** ataraday_ has quit IRC | 17:32 | |
| emccormick | hrmph. Maybe have an answer. Let's see | 17:34 |
| emccormick | I had been working on upgrading to Rocky but not done yet. I used this box to run a test deploy also and it's got octaviaclient 1.6.1 | 17:35 |
| emccormick | perhaps the rocky client is unhappy talking to queens Octavia ;) | 17:35 |
| *** tesseract has joined #openstack-lbaas | 17:36 | |
| johnsom | No, I would really be surprised if there is some bug there. LB create really hasn't changed for what you are doing. | 17:37 |
| emccormick | hrm | 17:38 |
| emccormick | So from what I see, neutron server is returning a full list of subnets | 17:38 |
| emccormick | the next thing I see is Octavia complaining that the subnet doesn't exist. I can see it in the list returned from Neutron though | 17:39 |
| johnsom | That list is the CLI list, the call we are looking for is a show/get not a list. | 17:44 |
| johnsom | This is the call on the Octavia side: https://github.com/openstack/octavia/blob/stable/queens/octavia/network/drivers/neutron/base.py#L175 | 17:45 |
| *** psachin has quit IRC | 17:48 | |
| emccormick | hrm. OK so I found Octavia querying the subnet explicitly. The only thing on the Neutron side is a 200 success message and a message about it blocking the "shared" attribute due to policy. | 18:08 |
| emccormick | doesn't seem like that should break it though should it? | 18:08 |
| emccormick | Attributes excluded by policy engine: [u'shared'] _exclude_attributes_by_policy /var/lib/kolla/venv/lib/python2.7/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py:256 | 18:09 |
| johnsom | Ok, yeah, that is the issue. The project requesting the subnet doesn't have permission from neutron. | 18:09 |
| emccormick | ah hah | 18:09 |
| johnsom | It's probably a subnet with the "shared" flag set, but "shared" subnets are disabled via RBAC in neutron? | 18:09 |
| emccormick | hmm. it shouldn't be set at all. Lemme see | 18:10 |
| *** pcaruana has quit IRC | 18:11 | |
| johnsom | I sure hate that they translate the 503's to 404's | 18:12 |
| emccormick | so Shared would be an attribute of the parent network. Shared = False on both that I've tried with. | 18:13 |
| openstackgerrit | Merged openstack/octavia stable/stein: Fix allocate_and_associate DB deadlock https://review.opendev.org/665556 | 18:14 |
| johnsom | You are getting into the mysteries of the neutron RBAC that I may not be able to help with. It is either the user project or the one being passed in that doesn't have permission to show/get that subnet. I just don't know why on the neutron side. | 18:16 |
| emccormick | OK will keep digging and report back later. Thanks for the guidance. | 18:21 |
| *** luksky has joined #openstack-lbaas | 18:30 | |
| rm_work | ahahaha, found my pycharm issue | 18:42 |
| rm_work | https://github.com/testing-cabal/testtools/blob/master/testtools/testcase.py#L31 | 18:42 |
| rm_work | dynamic import of unittest lib makes newer versions of pycharm incapable of detecting that it actually is of type unittest | 18:43 |
| rm_work | so none of the automatic testing helpers work | 18:43 |
| rm_work | on any of our tests T_T | 18:43 |
| *** tesseract has quit IRC | 18:49 | |
| *** ivve has joined #openstack-lbaas | 18:50 | |
| *** ccamposr has joined #openstack-lbaas | 18:57 | |
| *** ccamposr__ has quit IRC | 19:00 | |
| *** gcheresh has joined #openstack-lbaas | 19:14 | |
| *** ivve has quit IRC | 19:16 | |
| *** Vorrtex has quit IRC | 20:10 | |
| *** gcheresh has quit IRC | 20:22 | |
| *** mithilarun has joined #openstack-lbaas | 20:29 | |
| *** mithilarun has quit IRC | 20:30 | |
| *** mithilarun has joined #openstack-lbaas | 20:30 | |
| *** mithilarun has quit IRC | 20:55 | |
| *** mithilarun has joined #openstack-lbaas | 21:07 | |
| *** mithilarun has quit IRC | 21:16 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Specify the linux-image-kvm kernel for ubuntu https://review.opendev.org/665861 | 21:23 |
| johnsom | Ok, that should be good to go. Just need to confirm with the gate jobs. | 21:25 |
| *** fnaval has quit IRC | 21:47 | |
| cgoncalves | whelp! I see a tab in L140 | 21:48 |
| *** yamamoto has joined #openstack-lbaas | 21:51 | |
| *** goldyfruit has quit IRC | 21:54 | |
| *** yamamoto has quit IRC | 21:56 | |
| johnsom | Blah | 21:56 |
| *** mkuf has quit IRC | 22:01 | |
| *** mkuf has joined #openstack-lbaas | 22:02 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Specify the linux-image-kvm kernel for ubuntu https://review.opendev.org/665861 | 22:02 |
| *** vishalmanchanda has quit IRC | 22:03 | |
| *** devfaz has quit IRC | 22:03 | |
| *** devfaz has joined #openstack-lbaas | 22:03 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Specify the linux-image-kvm kernel for ubuntu https://review.opendev.org/665861 | 22:10 |
| johnsom | FYI, the rax infra mirror is broken again, expect jobs to fail. | 22:16 |
| *** luksky has quit IRC | 22:39 | |
| *** mithilarun has joined #openstack-lbaas | 22:49 | |
| *** mithilarun has quit IRC | 22:51 | |
| *** ccamposr__ has joined #openstack-lbaas | 23:04 | |
| *** ccamposr has quit IRC | 23:07 | |
| *** mithilarun has joined #openstack-lbaas | 23:08 | |
| *** mithilarun has quit IRC | 23:14 | |
| *** mithilarun has joined #openstack-lbaas | 23:15 | |
| *** rcernin has joined #openstack-lbaas | 23:16 | |
| *** mithilarun has quit IRC | 23:40 | |
| *** yamamoto has joined #openstack-lbaas | 23:53 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!