Wednesday, 2017-05-31

« Tuesday, 2017-05-30 Index Thursday, 2017-06-01 »
*** sshank has quit IRC00:05
*** leitan has quit IRC00:23
*** leitan has joined #openstack-lbaas00:24
*** sanfern has joined #openstack-lbaas00:31
*** cpuga is now known as cpuga_00:38
*** cpuga_ is now known as cpuga00:38
*** cpuga has quit IRC01:04
*** cpuga has joined #openstack-lbaas01:07
openstackgerritMichael Johnson proposed openstack/octavia master: Add v2 pool API section  https://review.openstack.org/45827201:15
openstackgerritJude Cross proposed openstack/octavia master: Add filtering and field selection to API  https://review.openstack.org/46927501:26
*** ianychoi_ is now known as ianychoi01:35
*** JudeC has quit IRC01:39
*** sanfern has quit IRC01:59
*** sanfern has joined #openstack-lbaas02:43
*** JudeC has joined #openstack-lbaas02:54
*** cody-somerville has quit IRC03:33
*** cody-somerville has joined #openstack-lbaas03:34
*** gans has joined #openstack-lbaas03:35
openstackgerritZhaoBo proposed openstack/octavia master: Extend api to accept qos_policy_id  https://review.openstack.org/45830803:38
*** cody-somerville has quit IRC03:42
*** links has joined #openstack-lbaas03:45
*** JudeC has quit IRC04:06
openstackgerritDeepak Mourya proposed openstack/octavia master: Stop using legacy facade  https://review.openstack.org/46932304:06
*** yamamoto has joined #openstack-lbaas04:13
*** aojea has joined #openstack-lbaas04:18
*** leitan has quit IRC04:20
*** aojea has quit IRC04:22
*** cody-somerville has joined #openstack-lbaas04:23
*** cody-somerville has quit IRC04:23
*** cody-somerville has joined #openstack-lbaas04:23
*** JudeC has joined #openstack-lbaas04:45
*** cody-somerville has quit IRC04:53
*** cody-somerville has joined #openstack-lbaas04:54
*** cody-somerville has quit IRC05:05
*** cody-somerville has joined #openstack-lbaas05:09
*** cody-somerville has quit IRC05:09
*** cody-somerville has joined #openstack-lbaas05:09
*** cody-somerville has quit IRC05:18
*** cody-somerville has joined #openstack-lbaas05:19
*** armax has quit IRC05:42
*** sanfern has quit IRC05:43
*** gans819 has joined #openstack-lbaas05:43
*** gans has quit IRC05:43
*** sanfern has joined #openstack-lbaas05:44
*** gans819 has quit IRC05:46
nmagnezirm_work, johnsom o/05:57
*** rcernin has joined #openstack-lbaas06:10
openstackgerritJude Cross proposed openstack/octavia master: Add filtering and field selection to API  https://review.openstack.org/46927506:12
*** KeithMnemonic1 has joined #openstack-lbaas06:16
openstackgerritZhaoBo proposed openstack/octavia master: Extend api to accept qos_policy_id  https://review.openstack.org/45830806:17
*** KeithMnemonic has quit IRC06:18
*** yamamoto has quit IRC06:30
*** JudeC has quit IRC06:36
*** atoth has quit IRC06:41
*** atoth has joined #openstack-lbaas06:42
*** basilAB has quit IRC06:42
rm_worknmagnezi: o/06:43
*** basilAB has joined #openstack-lbaas06:44
*** gcheresh has joined #openstack-lbaas06:51
*** tesseract has joined #openstack-lbaas06:53
*** sanfern has quit IRC06:56
*** sanfern has joined #openstack-lbaas06:57
*** Dave has quit IRC07:02
*** kbyrne has quit IRC07:05
*** kbyrne has joined #openstack-lbaas07:06
*** Dave has joined #openstack-lbaas07:07
*** yamamoto has joined #openstack-lbaas07:13
*** pcaruana has joined #openstack-lbaas07:26
*** aojea has joined #openstack-lbaas07:26
*** cpuga has quit IRC07:40
*** kbyrne has quit IRC08:14
*** gcheresh has quit IRC08:15
*** kbyrne has joined #openstack-lbaas08:17
*** blogan_ has joined #openstack-lbaas08:52
*** bbzhao has joined #openstack-lbaas08:52
*** dasanind_ has joined #openstack-lbaas08:54
*** strigazi_ has joined #openstack-lbaas08:56
*** oomichi_ has joined #openstack-lbaas08:59
*** diltram_ has joined #openstack-lbaas09:00
*** links has quit IRC09:01
*** blogan has quit IRC09:01
*** bzhao has quit IRC09:01
*** eandersson has quit IRC09:01
*** dasanind has quit IRC09:01
*** diltram has quit IRC09:01
*** mhayden has quit IRC09:01
*** oomichi has quit IRC09:01
*** strigazi has quit IRC09:01
*** oomichi_ is now known as oomichi09:01
*** dasanind_ is now known as dasanind09:01
*** beagles has quit IRC09:03
*** b3nt_pin has joined #openstack-lbaas09:06
*** b3nt_pin is now known as Guest4632509:06
*** links has joined #openstack-lbaas09:07
*** mhayden has joined #openstack-lbaas09:07
*** eandersson has joined #openstack-lbaas09:08
*** cpuga has joined #openstack-lbaas09:10
*** cpuga has quit IRC09:14
rm_workOK cool, already fixed and rolled out 0.4.15 of pyroute2, very fast09:36
rm_workjohnsom: ^^09:36
*** gcheresh has joined #openstack-lbaas10:08
*** sanfern has quit IRC10:15
*** links has quit IRC10:16
*** sanfern has joined #openstack-lbaas10:17
*** gcheresh has quit IRC10:23
*** links has joined #openstack-lbaas10:36
*** atoth has quit IRC10:38
*** sanfern has quit IRC10:51
*** yamamoto has quit IRC11:02
*** kbyrne has quit IRC11:23
*** atoth has joined #openstack-lbaas11:39
*** Guest46325 is now known as beagles11:41
*** yamamoto has joined #openstack-lbaas11:43
*** chlong has quit IRC12:03
*** kbyrne has joined #openstack-lbaas12:08
*** links has quit IRC12:11
*** sanfern has joined #openstack-lbaas12:17
*** gans has joined #openstack-lbaas12:39
gansi am bale to create LBs with new image generated via diskimage-create utility after applying this change https://review.openstack.org/#/c/469426/12:52
*** leitan has joined #openstack-lbaas13:08
*** gans has quit IRC13:08
*** gans has joined #openstack-lbaas13:10
*** strigazi_ is now known as strigazi13:15
*** gans has quit IRC13:15
*** gans_ has joined #openstack-lbaas13:15
*** gans_ has left #openstack-lbaas13:15
*** gans has joined #openstack-lbaas13:17
*** fnaval has joined #openstack-lbaas13:20
*** fnaval has quit IRC13:27
*** fnaval has joined #openstack-lbaas13:28
*** chlong has joined #openstack-lbaas13:30
*** gans_ has joined #openstack-lbaas13:43
*** chlong has quit IRC13:43
*** gans_ has quit IRC13:44
*** chlong has joined #openstack-lbaas13:46
openstackgerritOpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements  https://review.openstack.org/46747313:51
*** cpuga has joined #openstack-lbaas14:06
*** cpuga has quit IRC14:11
*** gans has quit IRC14:17
*** armax has joined #openstack-lbaas14:25
*** fnaval has quit IRC14:31
*** cpuga has joined #openstack-lbaas14:41
*** cpuga_ has joined #openstack-lbaas14:42
*** cpuga has quit IRC14:45
*** fnaval has joined #openstack-lbaas14:46
*** fnaval has quit IRC14:46
*** fnaval has joined #openstack-lbaas14:47
*** csomerville has joined #openstack-lbaas14:49
*** cody-somerville has quit IRC14:52
*** JudeC has joined #openstack-lbaas15:29
*** fnaval has quit IRC15:33
*** csomerville has quit IRC15:34
*** blogan_ is now known as blogan15:34
*** cody-somerville has joined #openstack-lbaas15:35
*** cody-somerville has quit IRC15:35
*** cody-somerville has joined #openstack-lbaas15:35
*** csomerville has joined #openstack-lbaas15:35
*** gcheresh has joined #openstack-lbaas15:37
*** cody-somerville has quit IRC15:39
*** gans has joined #openstack-lbaas15:56
*** aojea has quit IRC15:59
*** rcernin has quit IRC16:07
*** tesseract has quit IRC16:08
*** gans has quit IRC16:22
*** gcheresh has quit IRC16:33
*** gcheresh has joined #openstack-lbaas16:38
openstackgerritOpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements  https://review.openstack.org/46747316:42
*** rcernin has joined #openstack-lbaas16:46
*** pcaruana has quit IRC16:50
*** armax has quit IRC16:51
*** fnaval has joined #openstack-lbaas16:54
*** fnaval has quit IRC16:54
*** fnaval has joined #openstack-lbaas16:55
rm_workjohnsom: morning17:18
rm_workthe gates are killing me17:18
rm_worki want to merge like 8 things17:18
johnsomYeah, that 404 issue is really super annoying17:19
rm_work0.4.15 is out but do we still need to wait for this g-r update with !=0.4.14 to land? >_> i don't understand17:19
johnsomYeah, I think it is caching getting us17:20
*** aojea has joined #openstack-lbaas17:29
*** ipsecguy_ has quit IRC17:30
*** armax has joined #openstack-lbaas17:30
*** ipsecguy has joined #openstack-lbaas17:30
*** gcheresh has quit IRC17:38
*** aojea has quit IRC17:50
*** aojea has joined #openstack-lbaas17:51
*** atoth has quit IRC17:53
*** belharar has joined #openstack-lbaas17:54
*** aojea has quit IRC17:55
*** belharar has quit IRC17:57
*** aojea has joined #openstack-lbaas17:57
*** aojea has quit IRC18:03
*** chlong has quit IRC18:05
*** sanfern has quit IRC18:12
*** reedip_ has joined #openstack-lbaas18:17
*** aojea has joined #openstack-lbaas18:18
rm_workgaaaah and my internet keeps disconnecting today18:19
*** chlong has joined #openstack-lbaas18:22
*** aojea has quit IRC18:22
*** reedip_ has quit IRC19:02
rm_workjohnsom: working on removing flask *entirely* from the amp (replacing the request/response stuff with webob equiv)19:15
rm_workhoping maybe it'd help with 404s? :/19:16
rm_workdunno19:16
rm_workgrasping at straws there19:16
johnsomSeems like a long shot19:16
rm_workyeah but it needed to be done anyway19:17
johnsomI am looking at the HM flows patch and starting to look at the pile of OpenStack email19:17
rm_worklol k19:17
rm_worki'll be back in a bit19:18
rm_worklunching19:18
nmagnezio/19:25
nmagnezijohnsom, meeting in 30min, right? :-)19:25
johnsomYes, 35 I think...  grin19:25
*** aojea has joined #openstack-lbaas19:28
*** gcheresh has joined #openstack-lbaas19:31
*** gcheresh has quit IRC19:32
nmagnezijohnsom, ha.. no one grinned here for few weeks.. welcome back :-)19:33
*** yamamoto has quit IRC19:33
johnsomAhh, you all missed me...  Ha19:33
rm_workahh crap that meeting is at 1PST?19:34
rm_workwhelp19:34
rm_worki'll try to get on mobile :/19:35
rm_workaugh i also have a conflicting meeting for the second half19:35
rm_workugh19:35
*** csomerville has quit IRC19:35
*** csomerville has joined #openstack-lbaas19:36
*** gcheresh has joined #openstack-lbaas19:38
*** gcheresh has quit IRC19:42
*** pcaruana has joined #openstack-lbaas19:45
*** cpuga_ is now known as cpuga19:57
johnsomOctavia meeting starting soon on #openstack-meeting-alt19:58
*** gcheresh has joined #openstack-lbaas19:59
*** pcaruana has quit IRC20:02
*** rm_mobile has joined #openstack-lbaas20:03
*** rcernin has quit IRC20:21
*** rcernin has joined #openstack-lbaas20:22
*** yamamoto has joined #openstack-lbaas20:33
*** rm_mobile| has joined #openstack-lbaas20:42
*** rcernin has quit IRC20:42
*** yamamoto has quit IRC20:43
*** rm_mobile has quit IRC20:45
*** chlong has quit IRC20:53
*** JudeC has quit IRC20:53
openstackgerritMerged openstack/octavia master: Updated from global requirements  https://review.openstack.org/46747320:58
*** sshank has joined #openstack-lbaas21:00
johnsomI agree, it would be great to reinvestigate the nproc stuff.  It makes sense to some degree for the flavors and scaling.21:01
rm_workomgfinally21:01
johnsomSigh, yeah21:02
*** jniesz has joined #openstack-lbaas21:03
jnieszyes for the reloads we would want to support leveraging the -x option in haproxy 1.821:04
jnieszwhich you pass the socket as the arg21:04
*** chlong has joined #openstack-lbaas21:05
johnsomRight21:05
jnieszin addition to the -x would be a way to add in the haproxy systemctl script to dump the server state to a file21:06
jnieszthis way when it reloads the backends instantly come up21:07
johnsomSo this doesn't just sync over the sockets like a normal stick table sync?21:07
*** gcheresh has quit IRC21:08
jnieszthis is for the old and new process to handover the file descriptor to the socket21:08
jnieszhttps://www.haproxy.com/blog/truly-seamless-reloads-with-haproxy-no-more-hacks/21:09
johnsomOh, ok, you are thinking of the partial open sockets.  got it.  I was thinking about the peer sync and session persistence data.21:10
johnsomThe issue they are addressing with 1.8 is only an impact for new connections during the transition.  Partially open connections.  Otherwise, existing flows and session persistent new connections continue to function during reloads21:11
jnieszcorrect21:11
*** pcaruana has joined #openstack-lbaas21:12
jnieszthe k8 usecase is the one that comes to mind most21:12
jnieszif we are updating the member pool constantly21:12
jnieszas containers spin up/down21:12
jnieszalso for code upgrades21:13
*** aojea has quit IRC21:13
johnsomRight, that is the case that makes the rejected connections an issue21:13
johnsomIf you are running in that kind of environment, I would spin a custom image.  Probably using the 1.8 code.21:14
johnsomUpstream we tend to stick to what the vendors ship.  So, it would be a while before we would have 1.8 native upstream.  Though you could probably make a good case to get the vendors to do a backport of 1.8.  We did that for trusty and haproxy.21:16
*** sshank has quit IRC21:16
jnieszyea, we are making new image now to run 1.7.521:18
jnieszthe distro version is to far behind for us21:18
jnieszalready have people beating on my door for http221:18
johnsomYeah, I hear you on that21:19
jnieszalso for active/active, I updated the spec https://review.openstack.org/#/c/453005/21:20
jnieszI told Santosh to work on the lb management changes21:20
jnieszso I probably will pull out the lb-mgmt part out of the active/active spec21:20
johnsomOk, yeah, I chatted with him some last night about the lb-mgmt multiple networks21:21
jnieszyea, I am going to remove that part from the active/active spec21:21
*** harlowja has quit IRC21:21
jnieszthe pool we need because of the size of our regions21:21
jnieszwe only get a /22 at most from network team21:21
johnsomWe knew we needed to add a pool of networks at some point.  Mostly to alleviate # of ports per network issues.21:22
jnieszand we will have over 1k vips21:22
jnieszin a single region21:22
johnsomIf I was leading your deployment, I would make the management network v621:22
johnsomIt's private, it doesn't need to be publicly routable21:23
jnieszin the newer regions we can leverage ipv6 for that21:23
jnieszagree 100%21:23
jnieszare there any limitations for ipv6 and lb-mgmt?21:23
johnsomBut still, there have been concerns about too many neutron ports on a single 'network"21:23
jnieszyea, we still would want the pool even with ipv621:23
johnsomNo, it works fine.  There is a proposed patch to make that standard, but it needed some work21:24
*** cpuga_ has joined #openstack-lbaas21:24
johnsomThere is a config change you need to do if you want to use link-local v6 addresses for mgmt, but that is all.21:25
johnsomhttps://github.com/openstack/octavia/blob/master/etc/octavia.conf#L15121:26
jnieszthanks.  That is something I will try out in th elab21:26
johnsomThis is the proposed patch: https://review.openstack.org/#/c/391204/21:27
*** pcaruana has quit IRC21:29
*** csomerville has quit IRC21:30
jnieszThat is all I had for now.  When you get a chance if you provide feedback on the active/active spec that would be great. I completed the data model and rest api section21:30
johnsomCool, yeah I will have a look.  Still working through the backlog21:31
jnieszyea, I know you probabl have a lot of stuff to catch up with since you just got back21:31
openstackgerritMerged openstack/octavia master: Remove _LI, _LW, _LE, _LC from i18n  https://review.openstack.org/46709721:34
*** sshank has joined #openstack-lbaas21:39
*** cpuga has quit IRC21:39
*** cpuga has joined #openstack-lbaas21:40
nmagnezijohnsom, rm_work, my connection was dropped so I didn't get the chance to ask this in the open discussion: in the summit talk recordings ppl asked about lbaasv2 haproxy namespace migration to Octavia and the replay was the in the future some tools will be provided for this or ppl can simply migrate manually between neutron-lbaas and Octavia by  temporarily have two keystone endpoints (one for each) in their cloud. the question is21:44
nmagnezi about the tools: is there any concrete planning for this?21:44
*** cpuga has quit IRC21:44
jnieszdoes anybody know the reasoning behind the default values for net.ipv4.tcp_rmem  and net.ipv4.tcp_wmem?21:46
jnieszhttps://github.com/openstack/octavia/blob/master/elements/haproxy-octavia-ubuntu/post-install.d/20-haproxy-tune-kernel21:47
*** cpuga_ has quit IRC21:47
*** csomerville has joined #openstack-lbaas21:47
johnsomjniesz We got a number of those from haproxy technologies a long time ago.21:47
jnieszah ok.  Was wondering if it was still relevant21:48
jnieszthe max numbers seemed to be a bit small21:48
jnieszlower than os defaults21:48
johnsomYeah, that is the question.  Things have changed and those might need to be updated21:49
jnieszespecially with 10Gb21:49
jniesz10Gb+21:50
jnieszI can see the logic for the min default21:50
jnieszas you don't want to waste a lot of memory21:51
jnieszi think it would mainly be an issue for large files21:51
*** jniesz has quit IRC21:56
*** JudeC has joined #openstack-lbaas21:56
*** csomerville has quit IRC21:57
johnsomnmagnezi We need to have migration tools, but no one has started on them yet.22:03
johnsomThe old neutron-lbaas keystone endpoint is just the neutron endpoint, so that isn't going away, but we are adding the octavia one22:04
*** harlowja has joined #openstack-lbaas22:06
*** leitan has quit IRC22:06
*** cpuga has joined #openstack-lbaas22:07
*** leitan has joined #openstack-lbaas22:07
*** leitan has quit IRC22:12
*** sshank has quit IRC22:14
rm_workjohnsom: hmm this is weird, just started up 6 health monitor processes (on 6 different VMs), then put all my amps (16 of them) into failover state (updated the last_update to 0), and it appears that the same one HM process put ALL of them into busy and is doing them kindof sequentially???22:15
rm_workthe other 5 processes are seeing nothing, and just spinning polling for work22:15
rm_worknot seeing why that'd be the case22:15
rm_work(in the config i set failover threads to 2)22:16
johnsomI don't think there is code that would set the last_update to 0, only if it never received a heartbeat22:16
rm_worki did that myself in the DB22:16
rm_worktechnically there were no health messages ever from them, so i added them all to the table with last-update 0 so they'd be in a failover state22:17
rm_workbut anyway, point being that ONE health-manager process picked up *everything*22:17
johnsomOh, follow now.  Hmmm, It might claim them all and then do the failovers two at a time.  Trying to remember.  Plus that build throttling code might have changed this since I last looked at it.22:18
johnsomYeah, on it's cycle, one HM could claim them all.22:19
johnsomhttps://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/health_manager.py#L4722:19
*** cpuga has quit IRC22:22
*** cpuga has joined #openstack-lbaas22:29
rm_workugh, we should fix that22:33
rm_work"we" probably means me I guess <_<22:34
rm_worki have a bunch of stuff in line though T_T22:34
rm_workhmmm also seeing a weird thing apparently22:35
johnsomYeah, me too.  L7policy, L7rules, and quotas for example.  It's whoever gets there first22:35
rm_workupdating one member and ALL members briefly flip to ERROR for at least one health message cycle it seems22:35
johnsomA simple way would be to just add a config setting for max at a time interval22:35
rm_workhow about don't grab more per interval than you have threads for...22:36
johnsomThere was a bug about that once, but I could never see why or repro22:36
rm_workyeah this seems repro-able22:37
johnsomWe just need to think of the case where this is only one hm process.  Though that would be rare I suspect22:37
rm_workactually in a way that is safer than many22:37
rm_workif it goes down, no failovers22:37
rm_workversus...22:38
rm_workif you have 10 and 5 go down, could failover everything22:38
rm_workwhich is easy if a network segment falls off22:38
johnsomYeah, I don't like that case where hm claiming the world but failing and not finishing....22:38
*** cpuga has quit IRC22:41
*** rm_mobile| has quit IRC22:43
*** sshank has joined #openstack-lbaas22:49
rm_workxgerman: you around?22:51
rm_workxgerman: need some reviews / +As :P22:51
rm_workhttps://review.openstack.org/46581322:52
rm_workhttps://review.openstack.org/46779822:52
rm_workhttps://review.openstack.org/46740722:52
rm_workhttps://review.openstack.org/46518322:52
rm_workhttps://review.openstack.org/46778022:52
*** fnaval has quit IRC22:57
*** chlong has quit IRC23:00
johnsomhttps://review.openstack.org/45827223:04
johnsomhttps://review.openstack.org/46405023:04
johnsomhttps://review.openstack.org/46409523:04
rm_workjohnsom: hey so i had a really funky idea23:09
johnsomJust one?23:09
rm_workwhat if we had a param like ... "sync=true" that made the API not return until the status went back to ACTIVE/ERROR?23:10
rm_workso it sends to the handler... then polls internally23:10
johnsomI bet you don't even have to poll, but um, why?23:10
rm_workerr well, not so much poll but23:11
johnsomGenerally you run into timeout issues, etc.23:11
rm_workcontinue checking the db until it's active23:11
rm_workyeah sometimes23:11
rm_workinternally people are asking for it23:11
rm_workand i honestly don't think it'd be difficult to do23:11
johnsomI have a famous story about a dev team that used the curl library without reading the docs.  It closes a connection by default after a fixed time period.  Downloading or not.  They blamed my underlying network.  I told management they need to Read the Fine Manual before slapping together libraries....23:13
johnsomUm, I'm fine with it as long as it doesn't become nasty ugly code....23:14
rm_workmost operations (besides LB Create) are like 1-3s23:14
rm_workyeah I am thinking I can do it pretty elegantly23:14
rm_worklike maybe literally have it be a decorator23:14
johnsomrm_work sync=true seems a bit generic (could have other meanings), what about synchronous=true?  It's going to be like a code to code thing mostly I suspect23:17
rm_workyeah23:19
rm_workjohnsom: lol did i really do it right the first(ish) time? :P23:28
johnsomIt looked good from what I saw....23:28
*** armax has quit IRC23:41
*** armax has joined #openstack-lbaas23:42
*** cpuga has joined #openstack-lbaas23:48
openstackgerritMerged openstack/octavia master: Devstack plugin should mark the HM ovs port for cleanup skip  https://review.openstack.org/46052823:52
rm_workjohnsom: you looking at the client stuff?23:56
johnsomNot yet, still looking at OpenStack email stuffs23:57
rm_worklol k23:57
rm_workxgerman: no xgerman yet?23:57
johnsomWorrying about the "community goals" and proposals for Queens23:57
rm_workugh yeah23:57
johnsomOye.  They may be the only work for the cycle...  Ha23:57
*** cpuga has quit IRC23:59
« Tuesday, 2017-05-30 Index Thursday, 2017-06-01 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!