| *** sshank has quit IRC | 00:04 | |
| rm_work | johnsom: errr | 00:05 |
|---|---|---|
| rm_work | johnsom: getting reports that member weight of "0" breaks stuff | 00:06 |
| rm_work | we have 'MIN_WEIGHT = 0' | 00:06 |
| johnsom | Yeah, 0 means drain the member, i.e. don't allow new connections to the backend server, but finish existing connections | 00:06 |
| rm_work | kk | 00:07 |
| rm_work | also -- admin_state_down = true -> members report ERROR? | 00:08 |
| rm_work | i am not sure why that is the case | 00:09 |
| rm_work | need to dig | 00:09 |
| johnsom | http://cbonte.github.io/haproxy-dconv/1.6/configuration.html#5.2-weight | 00:09 |
| rm_work | k well | 00:10 |
| rm_work | what seems to be happening is | 00:10 |
| rm_work | it goes to "ERROR" | 00:10 |
| rm_work | when you set it to weight 0 | 00:10 |
| *** ssmith has joined #openstack-lbaas | 00:10 | |
| johnsom | Hmmm, yeah, ERROR could be due to the config being removed from haproxy, so health message won't acknowledge it. | 00:10 |
| rm_work | hmm | 00:11 |
| rm_work | yeah | 00:11 |
| rm_work | it shows up that way a lot | 00:11 |
| rm_work | usually when it shouldn't | 00:11 |
| johnsom | I wonder if we aren't handling the stats message from haproxy right for zero: | 00:11 |
| johnsom | Note that a zero | 00:11 |
| johnsom | weight is reported on the stats page as "DRAIN" since it has the same | 00:11 |
| johnsom | effect on the server | 00:11 |
| rm_work | hmm | 00:11 |
| rm_work | so it reports DRAIN back to Octavia? | 00:11 |
| rm_work | in our health message? | 00:12 |
| rm_work | or what | 00:12 |
| johnsom | It may be telling us it's "DRAIN" and we may be taking that as not ONLINE | 00:12 |
| rm_work | yeah probably | 00:12 |
| rm_work | != ONLINE -> ERROR? | 00:12 |
| openstack | rm_work: Error: "=" is not a valid command. | 00:12 |
| johnsom | No, it's in the haproxy stats gather | 00:12 |
| rm_work | hmm | 00:12 |
| rm_work | err yeah that's from the health messaging right? | 00:12 |
| johnsom | https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/utils/haproxy_query.py#L101 | 00:13 |
| johnsom | In here somewhere | 00:13 |
| johnsom | Thing is, we need to check what haproxy is ACTUALLY outputting for the status when weight is zero | 00:14 |
| johnsom | The docs just say ... | 00:15 |
| johnsom | I have one up, just a sec | 00:15 |
| rm_work | yeah i think we need to add "DRAIN" to possible node states | 00:17 |
| rm_work | dunno how much that breaks n-lbaas | 00:18 |
| johnsom | Well, we can just make it online | 00:18 |
| johnsom | That seems right to me | 00:18 |
| johnsom | Well, first test, with no health monitor, weight 0 still reports NO_CHECK | 00:23 |
| rm_work | with HM | 00:24 |
| rm_work | do with HM | 00:24 |
| johnsom | Yeah, working on it | 00:24 |
| johnsom | Nope, with a healthy health check it reports UP | 00:25 |
| rm_work | erm | 00:25 |
| rm_work | hmmmmmm | 00:25 |
| johnsom | Let me look at my member | 00:25 |
| rm_work | with weight 0? | 00:25 |
| * johnsom that sounds bad... | 00:25 | |
| rm_work | or with admin_state_down=True? | 00:25 |
| rm_work | lol | 00:25 |
| johnsom | weight 0, admin up | 00:26 |
| rm_work | do admin state down too | 00:26 |
| johnsom | https://www.irccloud.com/pastebin/X5oji1rw/ | 00:26 |
| rm_work | hmmm | 00:26 |
| rm_work | wait a min and look again? | 00:26 |
| rm_work | still doing that? | 00:27 |
| rm_work | oh and admin-down | 00:27 |
| johnsom | Ok, waiting | 00:27 |
| rm_work | if yours exhibits NEITHER of the issues i'm seeing | 00:27 |
| rm_work | then wtf | 00:27 |
| rm_work | gonna need to debug | 00:27 |
| johnsom | Ok, so I think weight 0 is working correctly. | 00:28 |
| johnsom | However, note, weight 0 and an unhealthy member will report down | 00:28 |
| johnsom | or error or whatever we called it | 00:28 |
| rm_work | hmmm | 00:28 |
| johnsom | Ok, still good, so going to admin down | 00:29 |
| rm_work | k... | 00:29 |
| rm_work | ah hmmmmm | 00:30 |
| rm_work | i wonder | 00:30 |
| johnsom | https://www.irccloud.com/pastebin/QJBeODlm/ | 00:30 |
| johnsom | But I want the heartbeats to have some more time... | 00:30 |
| rm_work | yeah | 00:30 |
| rm_work | hmmmmmmmmmm | 00:30 |
| rm_work | this is super weird | 00:30 |
| rm_work | so we get ERROR for both | 00:31 |
| rm_work | but also what we see is | 00:31 |
| rm_work | immediately after any member change | 00:31 |
| rm_work | every member flips to ERROR for a second | 00:31 |
| rm_work | until the next healthcheck | 00:31 |
| rm_work | O_o | 00:31 |
| johnsom | Yeah, not sure I understand that one | 00:31 |
| rm_work | or, after one healthcheck and not after the next | 00:31 |
| rm_work | you don't see that ever? | 00:32 |
| rm_work | i need to dig in to that... | 00:32 |
| johnsom | I haven't watched the logs and stuff to truely see, but just observationally, no, haven't seen it | 00:32 |
| johnsom | https://www.irccloud.com/pastebin/0W9zPoqt/ | 00:32 |
| rm_work | fff | 00:35 |
| johnsom | Yeah, still happy. I have nothing for you on this one | 00:36 |
| *** ipsecguy has quit IRC | 00:36 | |
| *** ipsecguy has joined #openstack-lbaas | 00:37 | |
| belharar | johnsom, hey | 00:41 |
| johnsom | cycling admin state up / down seems to be fine | 00:41 |
| johnsom | belharar Hi | 00:41 |
| johnsom | same with the weight | 00:43 |
| belharar | johnsom, I'm trying to invoke pdb during creation of amphora from nova_driver, but it keeps failing. e.g. it raises bpbquit, but it's not consistent across the build() func | 00:44 |
| belharar | johnsom, do you have any idea how can resolve it? | 00:44 |
| johnsom | belharar Hmm, not familiar with that issue but I know pdb has had issues with some of the OpenStack code. | 00:46 |
| johnsom | belharar I would make sure to run the o-cw process in standalone mode and not under uWSGI | 00:46 |
| belharar | johnsom, can you point me to where I could check that? (I'm using devstack) | 00:49 |
| johnsom | belharar "ps -ef | grep octavia-uwsgi.ini" if it's there you are running under uwsgi which will make pdb very hard. | 00:50 |
| belharar | johnsom, You're correct... | 00:51 |
| johnsom | Oh, wait, I am confusing this. | 00:51 |
| johnsom | Ignore that | 00:51 |
| johnsom | I was thinking API process, you are debugging the controller worker. | 00:52 |
| johnsom | It's been a long day | 00:52 |
| belharar | :-) | 00:52 |
| rm_work | yeah i was gonna say... | 00:53 |
| rm_work | anyway, yes, long day | 00:53 |
| rm_work | i wonder why we get very different results for our HMs :/ | 00:53 |
| rm_work | need to dig in a bit | 00:53 |
| johnsom | belharar With CW, a couple of thoughts. Change the workers to 1 and max_workers to 1 in octavia.conf, then "systemctl restart devstack@o-cw" then make sure you attach to the process listed as "ConsumerService" in ps -ef | 00:55 |
| johnsom | That will probably help you get what you want by minimizing the threads, etc. | 00:55 |
| johnsom | Yeah, I have been up 12 hours to make an early meeting, plus the pike-3 rush | 00:56 |
| *** JudeC has quit IRC | 01:02 | |
| rm_work | <_< | 01:08 |
| *** armax has joined #openstack-lbaas | 01:16 | |
| belharar | johnsom, I use 'screen' in my devstack setup. How can I verify I have only one active octavia controller? | 01:18 |
| *** harlowja has quit IRC | 01:19 | |
| rm_work | umm johnsom | 01:21 |
| johnsom | So instead of systemctl just join the screen session, crtl-c, up arrow, return to restart the process. Then in ps -ef there should only be one consumerservice | 01:22 |
| rm_work | https://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/update_db.py#L170-L173 | 01:24 |
| rm_work | johnsom: ^^ | 01:24 |
| rm_work | i don't know how yours is working | 01:24 |
| rm_work | oh if it gets neither of those it is "no change" | 01:25 |
| rm_work | i see | 01:25 |
| rm_work | so it comes back to "why do i get ERRORs for all members immediately after any change | 01:26 |
| rm_work | " | 01:26 |
| *** yamamoto_ has quit IRC | 01:31 | |
| *** gongysh has joined #openstack-lbaas | 01:35 | |
| belharar | johnsom, It appears to be pretty stubborn. I continue to see two of those CS workers, plus another 'master process'. | 01:35 |
| *** yamamoto has joined #openstack-lbaas | 01:35 | |
| belharar | johnsom, was i correct to modify it in /opt/stack/octavia/etc/octavia.conf? | 01:35 |
| johnsom | No, /etc/octavia/octavia.conf | 01:36 |
| johnsom | That one was the sample file from git | 01:36 |
| *** ssmith has quit IRC | 01:37 | |
| belharar | johnsom, thx | 01:37 |
| rm_work | johnsom: MAINT <-- is it that? | 01:38 |
| rm_work | trying to figure out what it actually is showing | 01:38 |
| johnsom | We don't use that status | 01:38 |
| rm_work | is that the draining status? | 01:38 |
| rm_work | what does it print for weight=0 ? | 01:39 |
| rm_work | i'm talking about HAProxy | 01:39 |
| johnsom | No, from what I see on a running haproxy, there is no DRAIN status | 01:39 |
| johnsom | It prints UP | 01:39 |
| rm_work | erm | 01:40 |
| rm_work | hmm so if the node is failing a healthcheck, and is "OFFLINE" | 01:40 |
| rm_work | weight 0/1 and state UP/DOWN all show as "OFFLINE" | 01:41 |
| rm_work | nm i lied | 01:41 |
| rm_work | eventually it goes to ERROR here | 01:41 |
| rm_work | wtf | 01:41 |
| rm_work | it should be "DOWN" | 01:41 |
| rm_work | err, OFFLINE | 01:41 |
| rm_work | no? | 01:41 |
| rm_work | bleh spinning a new stack | 01:42 |
| johnsom | Yeah, admin down should go to offline, but I don't think it is | 01:45 |
| johnsom | https://www.irccloud.com/pastebin/nKAHj8RJ/ | 01:45 |
| johnsom | There is a dump from HAPRoxy BTW. 18875c67-7818-4121-a58d-73c6f2e32f3e is admin up, weight 0 | 01:46 |
| *** gongysh has quit IRC | 01:47 | |
| *** sanfern has joined #openstack-lbaas | 01:49 | |
| johnsom | We should probably refactor that code to use the "disabled" keyword and the MAINT state for member admin down. Though pool admin down should also mark them OFFLINE, so maybe just fixing the controller logic is the right answer for both. | 01:50 |
| johnsom | https://bugs.launchpad.net/octavia/+bug/1706828 | 01:56 |
| openstack | Launchpad bug 1706828 in octavia "admin_state_up == false, member still ONLINE" [Critical,Triaged] - Assigned to Michael Johnson (johnsom) | 01:56 |
| johnsom | OFFLINE for pools and member is just broken | 01:56 |
| johnsom | The LB actually does what you want, but operating_status isn't reflecting it like you would want it to | 01:57 |
| rm_work | hmm | 02:02 |
| johnsom | rm_work Take if if you want it, I won't get to it until Tuesday at the soonest. | 02:03 |
| rm_work | yeah i dunno if i will either | 02:04 |
| rm_work | my env does something different anyway' | 02:04 |
| rm_work | lol i just noticed vip_Address has a capital letter, in the client T_T | 02:07 |
| johnsom | Hmm, I think there is a bug open that it doesn't work, maybe that is why... I don't know if it was open for api or cli | 02:08 |
| johnsom | https://bugs.launchpad.net/octavia/+bug/1695331 | 02:11 |
| openstack | Launchpad bug 1695331 in octavia "Octavia v2 API Requesting a VIP address is not working" [Critical,In progress] - Assigned to Pradeep Kumar Singh (pradeep-singh-u) | 02:11 |
| rm_work | no i think that's very different | 02:17 |
| rm_work | this is just the display formatting | 02:17 |
| johnsom | A | 02:17 |
| johnsom | Ah | 02:17 |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Update release notes for work done in Pike https://review.openstack.org/487666 | 02:23 |
| rm_work | johnsom: ummmmmmmmmmmmm | 02:27 |
| rm_work | OK I know why it's different | 02:27 |
| rm_work | http://paste.openstack.org/show/srtiQPptyYj6r4SDYi9c/ | 02:27 |
| rm_work | notice: DRAIN | 02:27 |
| johnsom | What version of HAPROXY are you running? | 02:28 |
| rm_work | 1.5.? | 02:28 |
| rm_work | HA-Proxy version 1.5.18 2016/05/10 | 02:28 |
| johnsom | ii haproxy 1.6.3-1ubuntu0.1 | 02:29 |
| rm_work | this is centos | 02:29 |
| johnsom | ChangeLog: 2016/11/20 : 1.6.10 - BUG/MEDIUM: srv-state: properly restore the DRAIN state | 02:30 |
| johnsom | So, it looks like they broke DRAIN for a while..... | 02:31 |
| rm_work | and we coded against that? | 02:31 |
| johnsom | It's now fixed, but the ubuntu version doesn't have the fix yet. | 02:31 |
| rm_work | ok well | 02:32 |
| johnsom | It had to have been broken for a while, like sometime into 1.5 too | 02:32 |
| rm_work | we can put in the fix in our code | 02:32 |
| rm_work | but yeah i was expecting DRAIN | 02:32 |
| rm_work | now i feel less confused | 02:32 |
| rm_work | yours is just broken :P | 02:32 |
| rm_work | (in the same way as our code) | 02:32 |
| johnsom | Yeah, we coded to what we actually saw. The docs, had some grey areas here | 02:33 |
| rm_work | lol | 02:33 |
| rm_work | so >_> | 02:33 |
| rm_work | this is sucky | 02:33 |
| johnsom | Yep | 02:33 |
| rm_work | WTB ubuntu fixing it | 02:33 |
| rm_work | so you want it to be "UP"? | 02:34 |
| rm_work | or can I add a state for members "DRAINING"? | 02:34 |
| rm_work | i kinda want to do this: | 02:34 |
| *** sanfern has quit IRC | 02:35 | |
| johnsom | If you add operating status DRAINING you need to mask it when we send the event to lbaas and update lbaas | 02:35 |
| rm_work | ugh yeah i guess so, fff | 02:36 |
| rm_work | k one sec | 02:36 |
| johnsom | I am ok with adding draining | 02:36 |
| rm_work | lol my computer is lagging so bad it still hasn't finished sending the review | 02:36 |
| johnsom | Ugh, we have to figure this out too: http://logs.openstack.org/37/487137/1/gate/gate-neutron-lbaasv2-dsvm-scenario-ubuntu-xenial/b1ecb54/logs/screen-o-cw.txt.gz#_Jul_27_00_29_40_396143 | 02:38 |
| johnsom | No suitable network interface found | 02:38 |
| johnsom | Strange that I only see it on nlbaas jobs | 02:39 |
| rm_work | DRAIN would be UP? | 02:40 |
| johnsom | For nlbaas yes, but the more I think about it, we just have to make nlbaas know DRAIN as it polls the API.... | 02:41 |
| openstackgerrit | Merged openstack/octavia master: Fix url_path valid check https://review.openstack.org/479220 | 02:41 |
| rm_work | so I do DRAIN->DRAINING | 02:42 |
| rm_work | see this: | 02:42 |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Properly handle DRAIN state from HAProxy health messages https://review.openstack.org/487671 | 02:42 |
| rm_work | i have had this ready to commit for a bit | 02:42 |
| rm_work | was just trying to validate 100% I was right | 02:42 |
| *** JudeC has joined #openstack-lbaas | 02:43 | |
| rm_work | but still | 02:43 |
| rm_work | what status does it flip to RIGHT AFTER a member change | 02:44 |
| rm_work | that causes it to go to error too | 02:44 |
| johnsom | A few comments... | 02:46 |
| rm_work | lolk | 02:48 |
| rm_work | that was a quick first draft | 02:48 |
| rm_work | ugh theres a db migration | 02:48 |
| rm_work | bleugh | 02:48 |
| rm_work | i forgot we put states there | 02:49 |
| johnsom | WTF now: http://logs.openstack.org/66/487666/1/check/gate-octavia-python27-ubuntu-xenial/d95227d/console.html#_2017-07-27_02_27_31_560833 | 02:49 |
| rm_work | the second comment for a release note was a nice touch, even if probably accidental | 02:49 |
| johnsom | That is just a release notes patch | 02:49 |
| rm_work | did you ask in barbican? | 02:49 |
| johnsom | No, this JUST happened | 02:50 |
| rm_work | wtf | 02:50 |
| rm_work | https://github.com/openstack/python-barbicanclient/commit/4a3c7cd23a22cb9f8fd376e3de3a6f4f6c69da0d | 02:51 |
| rm_work | barbicanclient released probably just nowish | 02:51 |
| rm_work | oh | 02:51 |
| rm_work | or g-r | 02:51 |
| rm_work | upper-constraints? | 02:51 |
| rm_work | gatefix time | 02:52 |
| johnsom | https://review.openstack.org/#/c/487549/ | 02:52 |
| johnsom | Yep, g-r 7:13 | 02:52 |
| rm_work | anyway just need to add .v1 | 02:52 |
| rm_work | you or me | 02:52 |
| rm_work | or .... | 02:52 |
| * rm_work looks around the room | 02:53 | |
| johnsom | You, I don't know what you are talking about.... | 02:53 |
| rm_work | i linked it | 02:53 |
| rm_work | https://github.com/openstack/python-barbicanclient/commit/4a3c7cd23a22cb9f8fd376e3de3a6f4f6c69da0d | 02:53 |
| rm_work | this | 02:53 |
| johnsom | That is a pretty nasty breaking change. Are they not stable? | 02:53 |
| * rm_work shrugs | 02:54 | |
| rm_work | asked | 02:56 |
| rm_work | anyway, fixing it, sec | 02:56 |
| johnsom | You will have more patience than I right now.... | 02:58 |
| rm_work | bly maybe | 03:00 |
| rm_work | barely maybe | 03:00 |
| rm_work | my compute is so f'd | 03:00 |
| rm_work | it's agin so bd it's missing ltters | 03:00 |
| rm_work | *lagging | 03:00 |
| johnsom | Broke heat too | 03:00 |
| *** JudeC has quit IRC | 03:01 | |
| johnsom | rm_work I need to sign off, but if you have fixes for this gate breakage, ping me and I will get back on to review.... | 03:10 |
| rm_work | k... i will fix | 03:12 |
| *** belharar has quit IRC | 03:21 | |
| *** aojea has joined #openstack-lbaas | 03:23 | |
| rm_work | reverting the upper-constraints change | 03:27 |
| rm_work | so we can be back in business while they fix things | 03:27 |
| *** aojea has quit IRC | 03:28 | |
| *** harlowja has joined #openstack-lbaas | 03:31 | |
| *** sanfern has joined #openstack-lbaas | 03:50 | |
| *** gongysh has joined #openstack-lbaas | 04:32 | |
| *** links has joined #openstack-lbaas | 04:33 | |
| reedip | johnsom : hi | 04:34 |
| reedip | johnsom : where did you see the error in https://review.openstack.org/#/c/458308/ | 04:34 |
| johnsom | I loaded up the patch and attempted to do an update to add a qos_policy_id | 04:37 |
| johnsom | I also confirmed that this does not happen on master | 04:37 |
| johnsom | reedip In the morning I need to feature freeze Pike | 04:41 |
| reedip | johnsom : okay , so the deadline is another 12 hours, I guess | 04:41 |
| johnsom | Sadly | 04:42 |
| reedip | hmm , got it | 04:42 |
| reedip | how did you attempt to update the qos_policy_id though | 04:42 |
| johnsom | I did it via curl direct to the API | 04:42 |
| johnsom | reedip Something like: curl -X PUT -H "Content-Type: application/json" -H "X-Auth-Token: <token>" -d '{"loadbalancer": {"vip_qos_policy_id": "<uuid>"}' http://198.51.100.10:9876/v2.0/lbaas/loadbalancers/8b6fc468-07d5-4d8b-a0b9-695060e72c31 | 04:45 |
| johnsom | After I fixed the other bug with the port json body | 04:46 |
| reedip | johnsom : ok, means we are also lacking some sort of test case | 04:46 |
| johnsom | Yep | 04:46 |
| johnsom | I'm not sure the cause of the ip_address issue, I haven't had time to dig in. We've been reviewing like 28 patches | 04:47 |
| reedip | wow ... sorry I was busy with other stuff , didnt get time to jump back in | 04:48 |
| *** harlowja has quit IRC | 04:52 | |
| *** gongysh has quit IRC | 05:05 | |
| *** gongysh has joined #openstack-lbaas | 05:10 | |
| *** gongysh has quit IRC | 05:13 | |
| openstackgerrit | Michael Johnson proposed openstack/octavia master: Spec detailing Octavia service flavors support https://review.openstack.org/392485 | 05:16 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements https://review.openstack.org/487474 | 05:27 |
| *** gcheresh_ has joined #openstack-lbaas | 05:29 | |
| *** armax has quit IRC | 05:30 | |
| *** aojea has joined #openstack-lbaas | 05:31 | |
| *** armax has joined #openstack-lbaas | 05:31 | |
| *** armax has quit IRC | 05:31 | |
| *** armax has joined #openstack-lbaas | 05:32 | |
| *** armax has quit IRC | 05:32 | |
| *** armax has joined #openstack-lbaas | 05:33 | |
| *** armax has quit IRC | 05:33 | |
| *** kobis has joined #openstack-lbaas | 05:52 | |
| *** yuanying has joined #openstack-lbaas | 05:57 | |
| *** Alex_Staf has joined #openstack-lbaas | 06:04 | |
| *** kobis has quit IRC | 06:11 | |
| openstackgerrit | Xing Zhang proposed openstack/octavia master: Fix haproxy_check_script for delete listener https://review.openstack.org/485254 | 06:30 |
| *** kobis has joined #openstack-lbaas | 06:47 | |
| *** aojea_ has joined #openstack-lbaas | 07:08 | |
| *** aojea has quit IRC | 07:12 | |
| *** rcernin has joined #openstack-lbaas | 07:13 | |
| *** Alex_Staf has quit IRC | 07:31 | |
| rm_work | reedip: you MIGHT have a little extra time if we can sell that we CAN | 07:39 |
| rm_work | *CAN'T merge anything today because of gate breakage | 07:39 |
| rm_work | stuff got a little wonky because of a barbican client change | 07:40 |
| reedip | hehe ... got it rm_work :) | 07:40 |
| rm_work | eugh | 08:25 |
| *** dayou1 has joined #openstack-lbaas | 08:25 | |
| rm_work | i'm seeing another recent fail? but confused as to why it's only now failing | 08:25 |
| rm_work | looks like pyroute2 has some stuff that's not OSX compatible? but it's never failed before now | 08:25 |
| rm_work | grrr | 08:26 |
| rm_work | can't runpy27/py35 tests | 08:27 |
| *** dayou has quit IRC | 08:27 | |
| *** gongysh has joined #openstack-lbaas | 08:36 | |
| *** nmagnezi_ has joined #openstack-lbaas | 08:39 | |
| *** nmagnezi has quit IRC | 08:46 | |
| *** nmagnezi_ is now known as nmagnezi | 08:46 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Properly handle DRAIN state from HAProxy health messages https://review.openstack.org/487671 | 08:55 |
| *** Alex_Staf has joined #openstack-lbaas | 08:59 | |
| openstackgerrit | Adam Harwell proposed openstack/octavia master: Properly handle DRAIN state from HAProxy health messages https://review.openstack.org/487671 | 09:02 |
| rm_work | johnsom: there, hope you're happier with that :P though i need to see if it passes unit tests, because i can't run them locally right now :/ | 09:03 |
| *** redondo-mk has quit IRC | 09:59 | |
| *** yamamoto has quit IRC | 10:00 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron-lbaas-dashboard master: Imported Translations from Zanata https://review.openstack.org/487800 | 10:05 |
| *** yamamoto has joined #openstack-lbaas | 10:31 | |
| *** sanfern has quit IRC | 10:52 | |
| *** sanfern has joined #openstack-lbaas | 11:59 | |
| *** catintheroof has joined #openstack-lbaas | 12:11 | |
| *** yamamoto has quit IRC | 12:14 | |
| *** atoth has joined #openstack-lbaas | 12:15 | |
| *** gongysh has quit IRC | 12:24 | |
| *** links has quit IRC | 12:52 | |
| *** yamamoto has joined #openstack-lbaas | 13:14 | |
| *** cristicalin has joined #openstack-lbaas | 13:15 | |
| *** yamamoto has quit IRC | 13:21 | |
| *** leitan has joined #openstack-lbaas | 13:28 | |
| *** kobis has quit IRC | 13:49 | |
| *** ssmith has joined #openstack-lbaas | 13:50 | |
| *** cpusmith has joined #openstack-lbaas | 13:51 | |
| *** ssmith has quit IRC | 13:55 | |
| *** cristicalin has quit IRC | 13:56 | |
| *** kobis has joined #openstack-lbaas | 14:23 | |
| *** armax has joined #openstack-lbaas | 14:39 | |
| *** mnaser has joined #openstack-lbaas | 14:42 | |
| *** gcheresh_ has quit IRC | 14:56 | |
| *** Alex_Staf has quit IRC | 14:57 | |
| *** sanfern has quit IRC | 14:58 | |
| *** sanfern has joined #openstack-lbaas | 14:58 | |
| openstackgerrit | Merged openstack/octavia master: Updated from global requirements https://review.openstack.org/487474 | 14:59 |
| *** kobis has quit IRC | 14:59 | |
| *** rcernin has quit IRC | 15:03 | |
| *** links has joined #openstack-lbaas | 15:04 | |
| *** links has quit IRC | 15:07 | |
| *** kobis has joined #openstack-lbaas | 16:04 | |
| *** fnaval has joined #openstack-lbaas | 16:18 | |
| *** kobis has quit IRC | 16:23 | |
| *** JudeC has joined #openstack-lbaas | 16:28 | |
| *** bzhao has joined #openstack-lbaas | 16:30 | |
| *** bbzhao has quit IRC | 16:33 | |
| *** kobis has joined #openstack-lbaas | 16:43 | |
| *** kobis has quit IRC | 16:44 | |
| *** sshank has joined #openstack-lbaas | 16:45 | |
| *** tongl has joined #openstack-lbaas | 16:58 | |
| openstackgerrit | Merged openstack/neutron-lbaas-dashboard master: Make it work with devstack https://review.openstack.org/480823 | 17:01 |
| openstackgerrit | Merged openstack/neutron-lbaas-dashboard master: Add loading and error status to detail pages https://review.openstack.org/481968 | 17:05 |
| openstackgerrit | Merged openstack/neutron-lbaas-dashboard master: Imported Translations from Zanata https://review.openstack.org/487800 | 17:05 |
| *** harlowja has joined #openstack-lbaas | 17:09 | |
| *** rcernin has joined #openstack-lbaas | 17:16 | |
| *** JudeC has quit IRC | 17:33 | |
| openstackgerrit | Merged openstack/neutron-lbaas master: consume load_class_by_alias_or_classname from neutron-lib https://review.openstack.org/487137 | 17:34 |
| *** SumitNaiksatam has joined #openstack-lbaas | 17:38 | |
| rm_work | johnsom: fix is up | 17:44 |
| rm_work | i guess we need to test it against a centos image | 17:44 |
| johnsom | Which? | 17:44 |
| rm_work | https://review.openstack.org/#/c/487671/ | 17:44 |
| rm_work | i think i got all the bits | 17:44 |
| rm_work | tests, release note, docs | 17:44 |
| johnsom | Ok, I will have a look. This is kind of a bug, so I'm going to work to get Pike-3 out and then take a look | 17:45 |
| openstackgerrit | Merged openstack/octavia-dashboard master: Add loading and error status to detail pages https://review.openstack.org/481954 | 17:45 |
| openstackgerrit | Merged openstack/neutron-lbaas master: use synchronized decorator from neutron-lib https://review.openstack.org/487161 | 17:49 |
| johnsom | rm_work https://review.openstack.org/#/c/487666/ | 17:55 |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Ignore 404 amphora error when deleting resources https://review.openstack.org/487232 | 17:55 |
| *** SumitNaiksatam has quit IRC | 17:56 | |
| *** SumitNaiksatam has joined #openstack-lbaas | 17:56 | |
| openstackgerrit | Merged openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/486325 | 18:05 |
| rm_work | xgerman_: commented on your last one | 18:10 |
| *** kobis has joined #openstack-lbaas | 18:15 | |
| *** kobis has quit IRC | 18:27 | |
| tomtomtom | @johnsom, @rm_work - I was able to get the octavia load balancing working after some post configuration checks and configuration changes. Here is a list of stuff I did: 1) Place the VM (web server) network into the list of plugged_interfaces for IP namespace in amphora instance (for health checks) 2) Remove the interface from /etc/networks/interfaces.d 3) Add controller ip port list controllers, for our install 10.80.20.14 | 18:29 |
| tomtomtom | etc (in amphora instance /etc/octavia/amphora*.conf file) 4) Update the octavia listener table with the barbican tls cert url 5) ensure the cert gets copied to the amphora instance in /var/lib/octavia/certs directory (if directory is empty, it's not been done) 6) if cert does not exist, run openstack loadbalancer listener disable/enable to "force" it to update 7) delete the default route in the network namespace and add the | 18:30 |
| tomtomtom | 8) if member is still offline ensure that monitor port for member is 80 and monitor ip address is the same as the member ip address in the member | 18:30 |
| *** kobis has joined #openstack-lbaas | 18:31 | |
| johnsom | tomtomtom Hmmm, let me try to pull this apart. | 18:36 |
| johnsom | 3 - This is a configuration setting in octavia.conf : https://docs.openstack.org/octavia/latest/configuration/configref.html#health_manager.controller_ip_port_list | 18:36 |
| johnsom | If that is set, it is automatically set in the amphora.conf in the amp | 18:36 |
| johnsom | 1 & 2 - Theses should automatically be done when the member is added to the pool | 18:37 |
| johnsom | 4 - This can be done from the API or CLI | 18:37 |
| *** kobis has quit IRC | 18:38 | |
| johnsom | 5 - happens when 4 is done via the API or CLI or dashboard | 18:38 |
| johnsom | 6 - same, happens when 4 | 18:38 |
| tomtomtom | 1 & 2 right, but for some reason never got done "automatically" | 18:39 |
| johnsom | 7 - default route in the network namespace is by default the VIP network default route. For members, if the subnet specified when the member was created has host routes, those are honored | 18:39 |
| tomtomtom | 3 - just saw that, hmm... I remember setting it, weird. | 18:39 |
| tomtomtom | 4 - Tried the updated openstack cli and direct "curl" http call and it still does not get updated in the DB. | 18:40 |
| tomtomtom | 5 - yeah it happens automatically in most cases but it's also taken the restart as mentioned. | 18:40 |
| johnsom | What OS are you using for your amphora image? | 18:41 |
| tomtomtom | 7 - I believe you, but when I switched the other over to the namespace it picked up that default route instead. | 18:41 |
| tomtomtom | Using ubuntu for amphora image. | 18:42 |
| johnsom | Yeah, if a host route is not specified on the network, it will pick up the other default rout | 18:42 |
| johnsom | So, on 4, can we get your octavia API log to look at? | 18:43 |
| *** armax has quit IRC | 18:49 | |
| openstackgerrit | Merged openstack/octavia master: Update release notes for work done in Pike https://review.openstack.org/487666 | 18:54 |
| *** cpusmith_ has joined #openstack-lbaas | 19:02 | |
| *** cpusmith has quit IRC | 19:06 | |
| *** SumitNaiksatam has quit IRC | 19:07 | |
| mnaser | hi everyone! i was hoping if i can get a +W on this -- it's useful for us to be able to do zero configuration deployments with puppet (notably, not having to manually put in a uuid), we use it in newton but i backported to ocata as it's just as useful there too -- https://review.openstack.org/#/c/487879/ and https://review.openstack.org/#/c/487883/ | 19:13 |
| *** gcheresh_ has joined #openstack-lbaas | 19:16 | |
| *** cpusmith_ has quit IRC | 19:19 | |
| *** cpusmith_ has joined #openstack-lbaas | 19:19 | |
| *** cpusmith has joined #openstack-lbaas | 19:20 | |
| *** cpusmith_ has quit IRC | 19:24 | |
| *** sshank has quit IRC | 19:36 | |
| *** gcheresh_ has quit IRC | 19:43 | |
| openstackgerrit | Merged openstack/neutron-lbaas master: Remove some LBaaSv1 references https://review.openstack.org/486716 | 19:43 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/488169 | 19:50 |
| *** gcheresh_ has joined #openstack-lbaas | 19:51 | |
| xgerman_ | rm_work questions about the proxy? | 19:51 |
| xgerman_ | we replace the normal plugin with one which proxies EVERY request to Octavia — so 3rd party would need to be in Octavia | 19:52 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements https://review.openstack.org/488170 | 19:52 |
| *** JudeC has joined #openstack-lbaas | 19:52 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/python-octaviaclient master: Updated from global requirements https://review.openstack.org/488171 | 19:57 |
| rm_work | xgerman_: that was not really my question :P | 20:02 |
| rm_work | i'll get to it | 20:04 |
| *** KeithMnemonic1 has joined #openstack-lbaas | 20:05 | |
| rm_work | xgerman_: this needs a review tho: https://review.openstack.org/#/c/486859/ | 20:05 |
| xgerman_ | ok | 20:06 |
| *** KeithMnemonic2 has quit IRC | 20:08 | |
| *** sshank has joined #openstack-lbaas | 20:14 | |
| johnsom | Ugh, was just about to do the release then this g-r came in swapping the one that came in this morning. | 20:15 |
| johnsom | Can I get one of you to hit these two so if they pass they go straight in? | 20:15 |
| johnsom | https://review.openstack.org/#/c/488170 | 20:15 |
| rm_work | lol sure | 20:15 |
| johnsom | https://review.openstack.org/#/c/488169 | 20:16 |
| rm_work | got them | 20:16 |
| johnsom | Thanks! | 20:16 |
| johnsom | After those I think we are ready to go | 20:16 |
| *** gcheresh_ has quit IRC | 20:34 | |
| *** KeithMnemonic2 has joined #openstack-lbaas | 20:38 | |
| openstackgerrit | German Eichberger proposed openstack/octavia master: Ignore 404 amphora error when deleting resources https://review.openstack.org/487232 | 20:40 |
| *** KeithMnemonic1 has quit IRC | 20:42 | |
| *** sshank has quit IRC | 20:50 | |
| *** sshank has joined #openstack-lbaas | 20:52 | |
| *** yamamoto_ has joined #openstack-lbaas | 21:13 | |
| *** yamamoto_ has quit IRC | 21:20 | |
| *** gongysh has joined #openstack-lbaas | 21:36 | |
| *** gongysh has quit IRC | 21:36 | |
| *** aojea_ has quit IRC | 21:40 | |
| *** aojea has joined #openstack-lbaas | 21:41 | |
| *** aojea has quit IRC | 21:45 | |
| *** armax has joined #openstack-lbaas | 21:53 | |
| *** aojea has joined #openstack-lbaas | 21:54 | |
| *** tongl has quit IRC | 21:56 | |
| *** aojea has quit IRC | 21:59 | |
| *** yamamoto has joined #openstack-lbaas | 22:04 | |
| openstackgerrit | Merged openstack/neutron-lbaas master: Updated from global requirements https://review.openstack.org/488169 | 22:06 |
| johnsom | There is one.... Zuul is being super slow on the octavia one, worried it's going to fail. Which frankly if it's going to fail I hope it hurries up.... | 22:12 |
| *** aojea has joined #openstack-lbaas | 22:14 | |
| *** fnaval has quit IRC | 22:16 | |
| *** armax has quit IRC | 22:17 | |
| *** armax has joined #openstack-lbaas | 22:18 | |
| *** cpusmith has quit IRC | 22:18 | |
| *** aojea has quit IRC | 22:19 | |
| *** rcernin has quit IRC | 22:26 | |
| *** catintheroof has quit IRC | 22:27 | |
| johnsom | 2 hr 36 min gate time. Geez, I thought we had timeout limits... | 22:29 |
| *** KeithMnemonic1 has joined #openstack-lbaas | 22:38 | |
| *** openstack has joined #openstack-lbaas | 22:46 | |
| *** KeithMnemonic2 has joined #openstack-lbaas | 22:46 | |
| *** jidar has joined #openstack-lbaas | 22:47 | |
| *** oomichi has joined #openstack-lbaas | 22:47 | |
| *** m-greene has joined #openstack-lbaas | 22:48 | |
| *** zw has joined #openstack-lbaas | 22:48 | |
| *** KeithMnemonic1 has quit IRC | 22:49 | |
| *** leitan has quit IRC | 23:23 | |
| *** leitan has joined #openstack-lbaas | 23:35 | |
| openstackgerrit | Merged openstack/octavia master: Updated from global requirements https://review.openstack.org/488170 | 23:42 |
| *** kbyrne has quit IRC | 23:47 | |
| *** kbyrne has joined #openstack-lbaas | 23:48 | |
| *** catintheroof has joined #openstack-lbaas | 23:52 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!