| johnsom | rm_work Seems like our barbican client stuff should look more like these: https://github.com/openstack/octavia/blob/master/octavia/common/clients.py | 00:17 |
|---|---|---|
| *** hongbin has joined #openstack-lbaas | 00:22 | |
| rm_work | probably, yes | 00:35 |
| rm_work | should be a quick patch? | 00:36 |
| johnsom | Yeah, probably should just put a story in for it. I won't have cycles for a bit | 01:11 |
| bzhao__ | johnsom: Hi, michael. I had arrived my "war" office. ;-). I will test in centos to see what happened in it. Is there any other thing I missed? | 01:20 |
| bzhao__ | PlugVip during LB creating hit 500 in centos case.. | 01:34 |
| bzhao__ | Seem osutils need to work on. | 01:35 |
| johnsom | bzhao__: that is great! It would be good to have centos working | 01:56 |
| johnsom | I am making dinner now, but may be on later | 01:56 |
| johnsom | To | 01:56 |
| johnsom | Two patches have a +2 | 01:56 |
| bzhao__ | johnsom: I saw, thanks to your very huge help.. Many thanks. | 01:57 |
| bzhao__ | johnsom: I will continue the work today for make sure ubuntu and centos work. But may take some time, as my env is not very performance. ;-) | 01:58 |
| bzhao__ | johnsom: Maybe patch 2 still need to change as lack review. But I must to make it work as wish at least. ;-) | 02:00 |
| *** abaindur has quit IRC | 02:10 | |
| *** LutzB has quit IRC | 02:59 | |
| *** yamamoto has quit IRC | 03:20 | |
| bzhao__ | Is there any good choices to debug agent code easily? Now I have to login the vm and configure the necessary DNS, make the network is OK, then download some tools and debug. It seems too complex. ;-). Once a step is wrong , the amp VM gone... | 03:46 |
| *** hongbin has quit IRC | 03:46 | |
| *** ramishra has joined #openstack-lbaas | 03:47 | |
| *** yamamoto has joined #openstack-lbaas | 03:55 | |
| *** yamamoto has quit IRC | 04:09 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: UDP for [2] https://review.openstack.org/529651 | 04:20 |
| *** yamamoto has joined #openstack-lbaas | 04:40 | |
| *** yamamoto has quit IRC | 04:47 | |
| *** yamamoto has joined #openstack-lbaas | 04:55 | |
| *** yamamoto has quit IRC | 05:05 | |
| *** yamamoto has joined #openstack-lbaas | 05:06 | |
| *** yamamoto has quit IRC | 05:06 | |
| bzhao__ | Let's see if it can work as wish based on new centos CI(<== what a good thing ;P) . My test env is too slow for running big flavor amp centos instance. | 05:06 |
| bzhao__ | johnsom: I build out the centos image, there is no nc cmd in it.. Seem need to do that. | 05:20 |
| johnsom | Ok, add it to the agent element | 05:21 |
| johnsom | May need the pkg map | 05:22 |
| bzhao__ | yeah. Thanks michael. | 05:24 |
| bzhao__ | The centos CI pass, ;-) | 06:07 |
| bzhao__ | Woo, another reason why the data plane does not work, seem the same script not success running on different OS. | 06:19 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/c3amNV2m/ | 06:20 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/hluQ6YRb/ | 06:21 |
| *** abaindur has joined #openstack-lbaas | 06:35 | |
| *** phuoc has joined #openstack-lbaas | 06:40 | |
| *** phuoc_ has quit IRC | 06:40 | |
| *** rcernin has quit IRC | 07:03 | |
| *** velizarx has joined #openstack-lbaas | 07:04 | |
| *** kobis1 has joined #openstack-lbaas | 07:10 | |
| *** ptoohill has quit IRC | 07:36 | |
| *** celebdor has joined #openstack-lbaas | 07:38 | |
| *** yamamoto has joined #openstack-lbaas | 07:41 | |
| *** devfaz has quit IRC | 07:45 | |
| *** strigazi has quit IRC | 07:45 | |
| *** eandersson has quit IRC | 07:46 | |
| *** abaindur has quit IRC | 07:48 | |
| *** velizarx has quit IRC | 07:59 | |
| *** velizarx has joined #openstack-lbaas | 08:04 | |
| bzhao__ | I change the script, it must be OK this time | 08:22 |
| bzhao__ | But the data plane still not work well. kernel configuration is the same with ubuntu case, also iptables . Packets arrive the amp but not route to realserver. | 08:25 |
| cgoncalves | bzhao__, hi. thanks for the good work on udp support! | 08:35 |
| cgoncalves | bzhao__, I am starting today looking at why LB creation fails on centos based amphorae. have you ever tried on centos or only ubuntu? | 08:35 |
| bzhao__ | cgoncalves: All thanks to you for introducing that amazing centos CI. ;-) | 08:36 |
| bzhao__ | cgoncalves: I'm trying on centos now. | 08:36 |
| bzhao__ | LB creation success in my test, for udp, just create is OK, it can not use, so I think I could figure out why the data plane not work as wish. | 08:38 |
| cgoncalves | bzhao__, thanks but credit for centos job is due to someone else :) | 08:42 |
| celebdor | cgoncalves: udp load balancing is being added to octavia? | 08:42 |
| cgoncalves | bzhao__, ok. I pulled latest patch set from last night. I will try again now | 08:42 |
| cgoncalves | celebdor, yes | 08:42 |
| celebdor | cool! | 08:42 |
| bzhao__ | cgoncalves: ;-) | 08:43 |
| cgoncalves | celebdor, take it with a grain of salt, though! it's not stable yet nor have been extensively tested especially on centos | 08:43 |
| cgoncalves | also no scenario tests yet | 08:43 |
| bzhao__ | Yeah. ;-( | 08:44 |
| celebdor | cgoncalves: this Rocky or S? | 08:45 |
| cgoncalves | celebdor, rocky (as in Rocky Balboa) | 08:46 |
| celebdor | :-) | 08:46 |
| cgoncalves | bzhao__, oh! I see that octavia-v1-dsvm-scenario-kvm-centos.7 is green now with your latest patch set in https://review.openstack.org/#/c/529651/ | 08:48 |
| cgoncalves | bzhao__, awesome! | 08:48 |
| openstackgerrit | Carlos Goncalves proposed openstack/octavia master: UDP for [3][5][6] https://review.openstack.org/539391 | 08:49 |
| bzhao__ | cgoncalves: Yeah, but just like you said, we lack so many fullstack tests.. | 08:49 |
| cgoncalves | bzhao__, yes but at least now it means that we can continue using centos amps for TCP/HTTP traffic | 08:50 |
| cgoncalves | before it was failing at LB creation so really broken | 08:50 |
| bzhao__ | cgoncalves: Yeah, you have done a great work. ;-) | 08:50 |
| celebdor | cgoncalves: did the work to make the centos image based on minimal yum install instead of an image happen? | 08:53 |
| *** ktibi has joined #openstack-lbaas | 08:54 | |
| cgoncalves | celebdor, no. we had an internal chat yesterday about it. it's a bit low prio for us at this moment | 08:57 |
| celebdor | ok | 08:58 |
| celebdor | cgoncalves: do you have documentation on all the tools that need to be there? | 08:58 |
| celebdor | if you do, I could possibly make a really small image like I did for the kuryr/demo container | 08:58 |
| celebdor | where everything is statically compiled over busybox | 08:58 |
| cgoncalves | celebdor, we use DIB for building amp images so that is element-enabled. create your own base OS element and replace centos/ubuntu element with it | 09:00 |
| *** pcaruana has joined #openstack-lbaas | 09:00 | |
| cgoncalves | celebdor, also there's centos-minimal but no rhel-minimal | 09:00 |
| *** salmankhan has joined #openstack-lbaas | 09:03 | |
| rm_work | bzhao__: I use https://github.com/ZephrFish/static-tools | 09:05 |
| rm_work | you can curl / wget them from the centos amps | 09:05 |
| rm_work | makes things a little easier without having to rebuild an image, good for debugging | 09:05 |
| rm_work | OH I see, but you need nc to actually be in *all* the amps, lol | 09:18 |
| rm_work | not for debugging, but for the UDP HM stuff to work :P | 09:18 |
| bzhao__ | rm_work: Thanks very much, that's very help to me. ;-). | 09:34 |
| bzhao__ | rm_work: I'm hanging on centos, the whole same configuration with ubuntu, but the udp not work.. And also I yum a nc cmd, the nc is also different ubuntu, some options not support.. | 09:35 |
| *** kobis1 has quit IRC | 09:51 | |
| cgoncalves | bzhao__, it works! | 09:59 |
| cgoncalves | test logs in https://review.openstack.org/#/c/539391/ | 09:59 |
| *** yamamoto has quit IRC | 10:02 | |
| bzhao__ | cgoncalves: Thanks, ;-) maybe something wrong in my test. I can success to create any UDP associated resources, but I can not connect to the behind realserver with iperf + virtual IP.. Let me check the log you show me carefully . | 10:04 |
| cgoncalves | bzhao__, I only tested creation of UDP listener. I have not yet tested connecting to members | 10:05 |
| bzhao__ | cgoncalves: As I found the keepalived in my centos env seem in trouble. | 10:05 |
| *** kobis1 has joined #openstack-lbaas | 10:06 | |
| *** kobis1 has quit IRC | 10:06 | |
| bbbbzhao_ | https://www.irccloud.com/pastebin/5oXTuJsg/ | 10:07 |
| bbbbzhao_ | It ignore all option what I want. | 10:07 |
| bbbbzhao_ | https://www.irccloud.com/pastebin/lSa9U7AA/ | 10:08 |
| bbbbzhao_ | But I can show them with ipvsadm.. So strange, I suspect that hit some issues during build image which I didn't realize.. | 10:09 |
| cgoncalves | I still need to dig in the code and how things are being set up | 10:16 |
| cgoncalves | right now I am an ordinary user testing it :) | 10:17 |
| rm_work | probably the version of nc in centos is from like 1999 | 10:21 |
| cgoncalves | -_- | 10:21 |
| rm_work | so it would be missing a lot of the newer options | 10:21 |
| rm_work | sorry cgoncalves, i just can't help it | 10:22 |
| rm_work | seriously tho, i get that centos is *stable*, but AT WHAT COST?! | 10:23 |
| rm_work | https://www.youtube.com/watch?v=x9Ag_aTTuK8 | 10:23 |
| rm_work | it's just "not for me" but I am constantly forced to use it anyway T_T | 10:24 |
| rm_work | if i weren't forced to use it constantly, i would just acknowledge that there's someone out there that appreciates the stability/feature trade-off, and move on | 10:25 |
| rm_work | but AUGH | 10:25 |
| rm_work | it actively degrades my quality of life | 10:25 |
| rm_work | because I'm *stuck with it* | 10:25 |
| *** yamamoto has joined #openstack-lbaas | 10:26 | |
| rm_work | </rant> | 10:26 |
| cgoncalves | RHEL/CentOS 8 shouldn't be that far away | 10:50 |
| cgoncalves | and it will be python 3 only | 10:50 |
| *** velizarx has quit IRC | 10:52 | |
| *** velizarx has joined #openstack-lbaas | 10:53 | |
| *** kobis1 has joined #openstack-lbaas | 11:00 | |
| *** yamamoto has quit IRC | 11:00 | |
| *** wolsen has quit IRC | 11:04 | |
| *** amitry_ has quit IRC | 11:04 | |
| *** yamamoto has joined #openstack-lbaas | 11:05 | |
| *** ramishra has quit IRC | 11:43 | |
| *** ramishra has joined #openstack-lbaas | 11:58 | |
| *** amuller has joined #openstack-lbaas | 12:13 | |
| bzhao__ | johnsom: I test on centos, and for now I just found the udp things can work with lvs only, but not work with keepalived if using keepalived to configure lvs. I'm not sure if my keepalived is not work well(even though I just configure virtual server and realserver with weight in its config file, it still can not work). Also I found a vrrp script bug, not post yet(if fortunatelly, I find the reason of keepalived not | 12:17 |
| bzhao__ | work with lvs, I will post it)..Today just fix that we can create udp staff on centos, but maybe can not connect the backend realserver accually. | 12:17 |
| *** velizarx has quit IRC | 12:48 | |
| *** kobis1 has quit IRC | 12:53 | |
| *** velizarx has joined #openstack-lbaas | 12:55 | |
| openstackgerrit | zhouchangxun proposed openstack/octavia master: Fix update pool without session_persistence failed https://review.openstack.org/588242 | 13:17 |
| openstackgerrit | zhouchangxun proposed openstack/octavia master: Fix update pool without session_persistence failed https://review.openstack.org/588242 | 13:22 |
| *** velizarx has quit IRC | 13:41 | |
| *** velizarx has joined #openstack-lbaas | 13:42 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Delete zombie amphora when detected https://review.openstack.org/587505 | 13:55 |
| *** celebdor has quit IRC | 14:12 | |
| *** celebdor has joined #openstack-lbaas | 14:15 | |
| *** hongbin_ has joined #openstack-lbaas | 14:16 | |
| *** celebdor1 has joined #openstack-lbaas | 14:33 | |
| *** celebdor has quit IRC | 14:36 | |
| bbbbzhao_ | It may hit image build error, I install a lower version keepalived, it can works with lvs. | 14:51 |
| *** celebdor1 is now known as celebdor | 14:52 | |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: UDP for [2] https://review.openstack.org/529651 | 14:54 |
| johnsom | bbbbzhao_ Hi, starting my day. Where are we at? Can we freeze patch 2 and only do follow up patches? | 15:00 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: UDP for [3][5][6] https://review.openstack.org/539391 | 15:01 |
| openstackgerrit | ZhaoBo proposed openstack/octavia master: Followup patch for UDP support https://review.openstack.org/587690 | 15:01 |
| bbbbzhao_ | johnsom: Hi, Michael. Ok, sorry for rebase... Yeah, I will follow you direction. | 15:04 |
| johnsom | Ok, thanks. We need to stablize to get them merged | 15:04 |
| bbbbzhao_ | johnsom: OK. I will continue to work based on the end of the patch list.. | 15:05 |
| johnsom | bbbbzhao_ So my understanding is the only open issue is centos image. Keepalived is not working properly with centos? | 15:06 |
| bbbbzhao_ | And I just test again. As I usually rebuild the image to test the new code, but maybe hit the build issue. The keepalived which build into image can not work with lvs on centos, but I reinstall a lower version keepalived and re-do the previous thing, it can work now. So the data plane is OK, it must be related with my local test env. | 15:08 |
| johnsom | Ok, we can look at that. I don't think that would stop a merge, just a bug to follow up on | 15:09 |
| bbbbzhao_ | Yeah. | 15:10 |
| bbbbzhao_ | I wrote on my little notes for concerned . ;-) | 15:11 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Clarify that the driver support library is interim https://review.openstack.org/588320 | 15:35 |
| johnsom | Sigh, people.... | 15:35 |
| johnsom | Trying to label Octavia as a library project.... | 15:35 |
| celebdor | johnsom: why? | 16:09 |
| johnsom | Yeah, exactly. | 16:09 |
| *** eandersson has joined #openstack-lbaas | 16:10 | |
| *** velizarx has quit IRC | 16:25 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Followup patch for UDP support https://review.openstack.org/587690 | 16:31 |
| *** ramishra has quit IRC | 16:32 | |
| *** yamamoto has quit IRC | 16:37 | |
| *** yamamoto has joined #openstack-lbaas | 16:40 | |
| *** ktibi has quit IRC | 16:42 | |
| *** yamamoto has quit IRC | 16:55 | |
| *** yamamoto has joined #openstack-lbaas | 17:19 | |
| *** salmankhan has quit IRC | 17:23 | |
| *** yamamoto has quit IRC | 17:34 | |
| *** yamamoto has joined #openstack-lbaas | 17:35 | |
| *** yamamoto has quit IRC | 17:35 | |
| *** yamamoto has joined #openstack-lbaas | 17:35 | |
| *** yamamoto has quit IRC | 17:35 | |
| *** yamamoto has joined #openstack-lbaas | 18:01 | |
| *** yamamoto has quit IRC | 18:05 | |
| colby_ | Is there a recommended specs for the amphora instances (vcpu/ram/disk)? | 19:07 |
| xgerman_ | depends what you like to do TLS termination or not | 19:07 |
| colby_ | Im creating the flavor for it. I was going to do 1vcpu/1G RAM/5GB disk. | 19:07 |
| colby_ | I was thinking of doing it eventually | 19:08 |
| *** abaindur has joined #openstack-lbaas | 19:08 | |
| colby_ | that will require barbican wich we do not have yet | 19:08 |
| xgerman_ | and more CPU | 19:08 |
| xgerman_ | RAM | 19:08 |
| xgerman_ | so for non TLS that sounds good - you can probably save on disk but GB are cheap | 19:08 |
| colby_ | what would you suggest for TLS offload? | 19:09 |
| johnsom | Yeah, so right now, 1vcpu is good, 1GB ram is good if you are not doing a lot of SSL termination connections, and disk is up to you on how much logging you expect or if you disable it. | 19:09 |
| johnsom | Again really depends on the number of concurrent connections you expect, but I would start with 2GB for a moderate load of TLS terminations. | 19:10 |
| xgerman_ | maybe another CPU… | 19:10 |
| colby_ | ok thanks | 19:11 |
| colby_ | Barbican is the preferred option with TLS right? Which in itself presents the problem of needing an HSM or dogtag setup (to be done correctly). | 19:13 |
| johnsom | Right, barbican or castellan. then a store behind it. | 19:14 |
| xgerman_ | we also support vault + barbican can run with mysql — but I haven’t been down that rabbit hole in a while | 19:14 |
| johnsom | vault with castellan. I don't know if the vault stuff for barbican merged or not | 19:15 |
| colin- | are there any successful examples of provisioning amphorae as containers instead of as VMs? | 19:15 |
| johnsom | lol, no, not yet. We have continually run into bugs in the container stacks that has slowed us down. | 19:16 |
| colby_ | is there an advantage of castellan over barbican (from the docs barbican can be its backend so Im confused) | 19:17 |
| colby_ | Ive looked into barbican, to integrate our magnum setup but have not looked at castellan | 19:18 |
| *** kobis1 has joined #openstack-lbaas | 19:26 | |
| *** kobis1 has quit IRC | 19:27 | |
| *** kobis1 has joined #openstack-lbaas | 19:27 | |
| *** kobis1 has quit IRC | 19:31 | |
| *** kobis1 has joined #openstack-lbaas | 19:32 | |
| *** kobis1 has quit IRC | 19:32 | |
| *** salmankhan has joined #openstack-lbaas | 19:35 | |
| *** salmankhan has quit IRC | 19:39 | |
| abaindur | hi johnsom: question about the config options if we don't have barbican or anchor set up. in order to just get things validated on an internal dev setup | 19:51 |
| abaindur | https://docs.openstack.org/octavia/queens/configuration/configref.html#certificates | 19:51 |
| abaindur | what should cert_manager be set to? Says the default is barbican_cert_manager | 19:51 |
| xgerman_ | yeah, keep the default and not use TLS termination on the LB | 19:52 |
| johnsom | colby_ It is confusing to a number of us. If it helps: https://governance.openstack.org/tc/reference/base-services.html | 19:52 |
| abaindur | and any other options we need to set that differ from defaults | 19:52 |
| johnsom | There are a number of timeouts that are set super high for dev work, but for a test setup, the defaults should work pretty well for you. | 19:53 |
| abaindur | yea, we dont plan to use TLS termination - for now, until we figure out how to use barbican - and we are using the pre-generated certs from the create_certificates.sh script | 19:53 |
| abaindur | if we use something else, what does cert_generator do? | 19:54 |
| xgerman_ | ok, yeah, other than the nova flavor, management net, etc. you should be fine with defaults | 19:54 |
| abaindur | so cert_manager only comes into play if we want the LBs to do TLS termination? | 19:55 |
| abaindur | and what anbout the cert_generator ? | 19:55 |
| johnsom | So cert_generator is what is used to create the TLS certificates we issue to the amphora service VMs. | 19:55 |
| abaindur | the ones i've created using the bin/create_certificates.sh right? | 19:56 |
| johnsom | Our command/control protocol uses two-way TLS authentication. Those certs are automatically managed by the control plane processes | 19:56 |
| xgerman_ | it uses the certs create by the script to make individual ones for the service vms | 19:56 |
| johnsom | This is how we configure the certs for our devstack test gates: https://github.com/openstack/octavia/blob/master/devstack/plugin.sh#L295-L305 | 19:56 |
| abaindur | what are the other options for cert_genarator? like in a production env? would it still be local? | 19:57 |
| xgerman_ | yes | 19:57 |
| johnsom | The anchor project was the alternative, but I think that project died. | 19:57 |
| xgerman_ | +1 | 19:57 |
| johnsom | I was a front end for HSM generators, etc. | 19:58 |
| abaindur | ahhh so anchor is no longer even viable | 19:58 |
| abaindur | that simplifies things. was thing it was recommended to use anchor | 19:59 |
| johnsom | Correct, to my knowledge it is not being developed any longer | 19:59 |
| *** amuller has quit IRC | 19:59 | |
| johnsom | You could ask in the #openstack-barbican channel if there are plans for other certificate generation options. | 20:00 |
| johnsom | That is the channel most of the certificate folks for OpenStack hang out | 20:00 |
| abaindur | thanks | 20:01 |
| abaindur | when would we want to set cert_manager to local_cert_manager ? | 20:02 |
| xgerman_ | it’s the default so will be set all the time | 20:02 |
| abaindur | the cfg guide says barbican_cert_manager is the default | 20:03 |
| abaindur | hence why i was aking above about leaving that as athe default | 20:03 |
| xgerman_ | ah, yes, sorry, confused with cert_generator — ok, if you don;t so TLS just leave it with barbican. Local is only for tests | 20:04 |
| johnsom | abaindur Really never. It is just there for tests. Local cert manager allows you to define a filesystem location for a cert to pull into the loadbalancer. | 20:05 |
| abaindur | got it. so it will still work for TLS termination, but its not recommended in any production use | 20:05 |
| johnsom | Well, all of your listeners would have the same cert... It's really just a test framework | 20:05 |
| abaindur | thanks , i thibk that clears it up for now! :) | 20:06 |
| rm_work | abaindur: there is an option in the api section to just disable the ability to create tls listeners | 20:51 |
| rm_work | you will want to set that, and then it won't matter what you have configured for cert_manager | 20:51 |
| rm_work | `allow_tls_terminated_listeners` | 20:52 |
| rm_work | set tha to False | 20:52 |
| openstackgerrit | Merged openstack/octavia master: UDP jinja template https://review.openstack.org/525420 | 21:49 |
| rm_work | johnsom: on https://review.openstack.org/#/c/525420/45/octavia/common/jinja/lvs/templates/base.j2 | 22:04 |
| rm_work | i agree that we didn't need to hold that patch up for it, but | 22:05 |
| rm_work | I am a little concerned if we're using that comment for *actual code* | 22:05 |
| rm_work | also -- it does look like the listener-id is in the filename | 22:05 |
| rm_work | so I'm not sure why we need to have it there | 22:05 |
| rm_work | but yeah, adding an additional line for LBID works fine | 22:05 |
| johnsom | Yeah, I already added that in my followup patch. | 22:05 |
| rm_work | like, i'm fine with having both -- more comments are great | 22:06 |
| rm_work | but .... seriously, it's being used by code for a mapping? what? | 22:06 |
| rm_work | THAT is a concern for me | 22:06 |
| rm_work | lol | 22:06 |
| johnsom | Yeah, haven't got that far yet. I'm half way through the beast of patch #2 | 22:06 |
| rm_work | yeah i started on it but got pulled off for k8s issues | 22:06 |
| johnsom | I think it is for figuring out the stats | 22:06 |
| rm_work | I don't think we should be using a *comment* for that | 22:07 |
| rm_work | <_< | 22:07 |
| johnsom | As the lb stats are all in the kernel and don't have a way to get our IDs associated. | 22:07 |
| rm_work | just saying | 22:07 |
| johnsom | I don't disagree | 22:07 |
| johnsom | Just haven't got that far in my review yet. | 22:07 |
| rm_work | kk | 22:07 |
| johnsom | I mean it's not THAT bad since it's a config file and you would just be storing the same mapping in another flat file somewhere likely. So, saving like 5 bytes.... lol | 22:08 |
| cgoncalves | bzhao__, does this configuration seem reasonable/expected? http://paste.openstack.org/show/727188/ | 22:08 |
| cgoncalves | I'd have expected to find some pool config | 22:09 |
| johnsom | That looks right to me, but I can tell you for sure in a minute | 22:10 |
| johnsom | Yeah, with keepalived the listener and the pool are one "thing" | 22:10 |
| cgoncalves | good | 22:11 |
| johnsom | Let me configure up a UDP on my stack and we can compare. | 22:12 |
| cgoncalves | there's something fishy still. LB provisioning_status keeps on PENDING_UPDATE and failing over amphora | 22:13 |
| johnsom | one of those check scripts might be bombing. Act/Stdby? | 22:14 |
| cgoncalves | standalone | 22:14 |
| cgoncalves | I see bzhao__ uploaded a new PS touching check scripts a couple of hours ago | 22:15 |
| cgoncalves | I count 20 amp failovers attempts since 6 hours ago xD | 22:16 |
| johnsom | Ouch | 22:17 |
| cgoncalves | actually more since o-hk comes after and cleanup DB, right? | 22:17 |
| cgoncalves | virsh id count is at 169 :D | 22:17 |
| johnsom | depends on your settings, it's a week by default | 22:17 |
| cgoncalves | hmm and I have disable_revert = True | 22:18 |
| cgoncalves | but I guess that that doesnt prevent o-hm from triggering failover | 22:18 |
| johnsom | https://www.irccloud.com/pastebin/8b8kDvp9/ | 22:18 |
| johnsom | My keepalived config (with latest and greatest patch chain) | 22:19 |
| cgoncalves | IPv6 \o/ | 22:19 |
| johnsom | cgoncalves FYI, I had a UDP amp up for hours yesterday, so either new issue or centos issue | 22:19 |
| cgoncalves | latter is just impossible xD | 22:20 |
| johnsom | cgoncalves Yeah, about that.... I think we have a bug in our ACt/Stdby when using V6 | 22:20 |
| johnsom | centos, could by y2k issue in a app | 22:21 |
| johnsom | grin | 22:21 |
| cgoncalves | at least I can assert with no margin of doubts that standalone amp failover works fine on centos (controller and amp) | 22:24 |
| cgoncalves | virsh count: 169 -> 172 | 22:25 |
| johnsom | Good, that was a bug earlier I caught. Failover would fail | 22:25 |
| johnsom | CHF testing? Continuous hours of failover | 22:25 |
| cgoncalves | ha! :) | 22:28 |
| cgoncalves | this is what I keep seeing: http://paste.openstack.org/show/727189/ | 22:28 |
| johnsom | Hmm, yeah, so the heartbeat isn't saying a listener is "present" when the DB says there should be one. | 22:30 |
| johnsom | FYI, failover is not a revert, so the config setting will not stop this. | 22:30 |
| cgoncalves | ok | 22:31 |
| *** rcernin has joined #openstack-lbaas | 22:32 | |
| johnsom | Yeah, this does not look right, but I don't know why mine isn't failing. | 22:33 |
| johnsom | I'm in patch 2 octavia/amphorae/backends/health_daemon/health_daemon.py | 22:33 |
| johnsom | It looks like if it can't get stats it doesn't send a udp listener message | 22:33 |
| cgoncalves | agreed | 22:36 |
| johnsom | Is your listener member less? Did you disable something? | 22:37 |
| cgoncalves | pool has 1 member | 22:38 |
| johnsom | Yeah, mine too and it's not failing over | 22:39 |
| *** hongbin_ has quit IRC | 22:39 | |
| cgoncalves | johnsom, sorry to ask again. what was the config option to prevent amp failover? | 22:55 |
| johnsom | Stop the health monitor process | 22:55 |
| johnsom | We only have a config for the reverts, not the failovers | 22:56 |
| cgoncalves | ah, ok | 22:56 |
| rm_work | or just mark the amp you want to debug "busy" in the health table | 23:01 |
| * rm_work shrugs | 23:01 | |
| rm_work | that is what I always do | 23:01 |
| rm_work | i wish i was better at RMQ debugging.... | 23:17 |
| rm_work | getting a lot of shit from oslo about rmq | 23:17 |
| rm_work | http://paste.openstack.org/show/727191/ | 23:17 |
| johnsom | No good | 23:18 |
| rm_work | missing heartbeats... but only from one pod | 23:18 |
| rm_work | the other seems fine | 23:18 |
| rm_work | my guess is networking? :/ | 23:18 |
| rm_work | just seems odd | 23:18 |
| johnsom | Ugh, ok, burned on reviewing this beast. will have to pick up tomorrow. | 23:23 |
| johnsom | I see a flow was re-ordered which makes me very concerned | 23:23 |
| rm_work | hmm yeah | 23:28 |
| rm_work | flow ordering is pretty sensitive <_< | 23:28 |
| rm_work | i will try to look tomorrow too | 23:28 |
| *** bcafarel has quit IRC | 23:30 | |
| bbbbzhao_ | Sorry all, just awake. Let me check what I need to explain. | 23:30 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Followup patch for UDP support https://review.openstack.org/587690 | 23:32 |
| bbbbzhao_ | cgoncalves > | 23:34 |
| bbbbzhao_ | cgoncalves Yeah, as keepalived lvs can not mapping the pool like haproxy, just a virthal server -> some realserver from a pool. | 23:35 |
| bbbbzhao_ | cgoncalves The config file is expected. | 23:36 |
| cgoncalves | bbbbzhao_, yeah, johnsom confirmed that. thank you | 23:42 |
| cgoncalves | bbbbzhao_, the problem I'm having now is http://paste.openstack.org/show/727189/ | 23:42 |
| bbbbzhao_ | johnsom: Does that udp amp failover/revert flow is in trouble? But I tried the failover, it works.. I hold the healthmonitor process not down for 1 night, it did what I want, recover the amp which I delete.. | 23:43 |
| cgoncalves | I'm debugging and I see that is_udp_listener_running returns False | 23:43 |
| cgoncalves | because need_check_listener_ids is empty | 23:43 |
| cgoncalves | I mean, need_check_listener_ids is empty because is_udp_listener_running returns False | 23:44 |
| cgoncalves | i have one listener. get_udp_listeners returns one uuid | 23:44 |
| cgoncalves | I need to go now. I'll continue tomorrow | 23:45 |
| bbbbzhao_ | cgoncalves: Ok, thank you very much for testing. | 23:45 |
| bbbbzhao_ | cgoncalves: I will leave some comments in irc for explain the possible issue. | 23:46 |
| cgoncalves | thank you! | 23:47 |
| cgoncalves | btw this is my LB: http://paste.openstack.org/show/727192/ | 23:47 |
| *** bcafarel has joined #openstack-lbaas | 23:51 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!