Friday, 2017-06-16

« Thursday, 2017-06-15 Index Saturday, 2017-06-17 »
rm_workjohnsom: https://review.openstack.org/#/c/474790/ passed00:12
rm_workrechecking for failures... <_<00:12
rm_workxgerman_: around?00:12
johnsomYeah, just on the fence about the lack of functional test00:12
rm_workah well, I was going to ask you to +A something but it probably won't pass anyway00:12
rm_workjohnsom: <_< there's a unit test00:12
rm_workwhat would a functional test look like exactly00:12
rm_workdo we HAVE functionals for the amp agent?00:12
johnsomYes, you modified it00:13
johnsomoctavia/tests/functional/amphorae/backend/agent/api_server/test_server.py00:13
johnsomI am just thinking a functional test where it returns a few 404s non-json then success00:14
rm_workblegh00:15
rm_workalright well one sec00:15
rm_workI have another ... interesting one for you in a sec :P00:15
rm_workas soon as pep8 passes00:15
rm_workmy continued quest to make this thing actually *deployable* for a wide variety of operators with weird restrictions...00:16
openstackgerritAdam Harwell proposed openstack/octavia master: Allow operators to tune VIP creation parameters  https://review.openstack.org/47481300:16
rm_workok let me look at a functional test00:16
rm_workwhat is the actual request we see this on? plug_vip?00:17
rm_worki can do it on like... anything though really00:19
johnsomhttp://logs.openstack.org/74/474674/1/check/gate-neutron-lbaasv2-dsvm-py3x-scenario-ubuntu-xenial-nv/6e89fc5/logs/screen-o-cw.txt.gz#_Jun_15_17_23_20_51689500:19
openstackgerritMerged openstack/octavia master: Updated from global requirements  https://review.openstack.org/47467400:20
johnsomFinally, something...  ha00:20
rm_workso yes, plug-vip00:20
johnsomUgh, I am worried that my scheme may be crazy in the eyes of others and I have to change a lot to make the tests work with this.  Do you have a minute for a quick glance and some RBAC rules?00:22
rm_worki know almost nothing about RBAC in Openstack, but I can look00:22
johnsomhttp://paste.openstack.org/show/612777/00:22
johnsomYeah, this is concept not code00:23
rm_workthough i was trying to figure out a functional test for this thing that SOMEONE wanted00:23
rm_work<_<00:23
johnsomSo basically you have to be a member of *one* of the load-balancer roles to get access to anything.00:25
johnsomI will fix up devstack plugin.sh to have that in the two accounts.  Of course operators can override this with a policy.json.00:26
*** cpuga has joined #openstack-lbaas00:27
xgerman_rm_work wassup?00:29
rm_workxgerman_: nm was gonna have you review and hopefully +A something but ... our gates are so broke00:29
rm_workit doesn't matter00:29
johnsomxgerman_ Since you are here, quick glance at this: http://paste.openstack.org/show/612777/00:30
rm_workoh00:30
rm_workhttps://review.openstack.org/#/c/474800/00:30
rm_workit did get +100:30
rm_workso yeah00:30
rm_work+A plz xgerman_00:30
xgerman_k - people use the Batman light Symbol in slack to get my attention00:30
xgerman_You got it...00:32
johnsomhttps://usercontent.irccloud-cdn.com/file/VYBoTPa9/image.png00:32
xgerman_Yeah.00:33
rm_workthat seems right00:33
johnsomOk, not too hard core with requiring users to have a role?00:34
rm_workjohnsom: so absolutely 100% requiring a functional test for this?00:34
johnsomIf xgerman_ gives you a pass I will let it go for now00:34
xgerman_Ha00:34
rm_worklol00:34
*** blogan has quit IRC00:38
*** amotoki is now known as amotoki_away00:48
openstackgerritMerged openstack/octavia master: Also remove amphora_health entry during house_keeping amps  https://review.openstack.org/47431800:50
*** amotoki_away is now known as amotoki00:52
rm_workok i'm not sure how to do this functional test00:57
rm_workso for now i'm gonna just keep rechecking this to see if it fails ever00:57
rm_workand kinda assume eventually that it works00:57
rm_workbut i'll be afk for a while00:57
rm_workjohnsom: does 'admin_or_owner' not also need 'role:load-balancer_admin'?00:58
johnsomYeah, that one needs to be deleted00:59
johnsomIt was the old way01:00
*** armax_ has joined #openstack-lbaas01:40
*** armax has quit IRC01:41
*** armax_ is now known as armax01:41
*** cpuga has quit IRC01:54
*** chlong has quit IRC03:12
*** gans has joined #openstack-lbaas03:37
*** gans819 has joined #openstack-lbaas03:47
*** gans has quit IRC03:49
*** armax has quit IRC03:52
*** cpuga has joined #openstack-lbaas04:15
*** cpuga has quit IRC04:20
*** gans819 has quit IRC04:20
*** gans has joined #openstack-lbaas04:21
*** cody-somerville has joined #openstack-lbaas05:10
*** csomerville has quit IRC05:10
*** cody-somerville has quit IRC05:10
*** cody-somerville has joined #openstack-lbaas05:11
*** gans has quit IRC05:33
*** krypto has joined #openstack-lbaas05:57
*** krypto has joined #openstack-lbaas05:57
*** cpuga has joined #openstack-lbaas06:17
*** rcernin has joined #openstack-lbaas06:19
*** cpuga has quit IRC06:21
*** rcernin has quit IRC06:24
*** krypto has quit IRC06:31
*** krypto has joined #openstack-lbaas06:31
*** rcernin has joined #openstack-lbaas06:39
*** pcaruana has joined #openstack-lbaas06:43
*** sticker has quit IRC06:51
*** sanfern has quit IRC06:51
*** sanfern has joined #openstack-lbaas06:54
openstackgerritThomas Herve proposed openstack/neutron-lbaas master: Send haproxy stat command as bytes  https://review.openstack.org/47486807:01
korean101johnsom: can you have some hints for me about o-hm <--> amp?07:04
korean101https://docs.openstack.org/developer/octavia/ocata/guides/dev-quick-start.html#load-balancer-network-configuration07:04
korean101Finally, you need to add routing or interfaces to this network such that the Octavia controller (which will be described below) is able to communicate with hosts on this network. This also implies you should have some idea where you’re going to run the Octavia controller components.07:04
korean101johnsom: i just copy and paste devstck/plugin.sh phase (sudo ovs-vsctl -- --may-exist add-port br-int o-hm0)07:05
korean101johnsom: i wanna deploy three octavia separated node (not network node)07:05
korean101johnsom: but i don't know routing to network node's br-int07:06
korean101some clues? or docs?07:06
*** tesseract has joined #openstack-lbaas07:09
rm_workkorean101: you just need to have that network routable, nothing special07:23
rm_workany routable network will do07:23
rm_workjohnsom: for the record I am PRETTY sure https://review.openstack.org/#/c/474790/ works07:25
rm_workit might be a *liiiiittle* early07:25
rm_workbut07:25
rm_workI'm full of confidence :P07:25
korean101rm_work: static route?07:39
korean101rm_work: controller-001 (eth1) --> network-001 (o-hm0) ?07:39
korean101rm_work: oh!!! just static route works07:45
korean101rm_work: but only o-hm0 not amps...07:45
openstackgerritJude Cross proposed openstack/octavia master: Add statistics to V2 API  https://review.openstack.org/47488508:05
*** JudeC has joined #openstack-lbaas08:07
*** JudeC has quit IRC08:14
*** cpuga has joined #openstack-lbaas08:18
*** rcernin has quit IRC08:21
*** rcernin has joined #openstack-lbaas08:22
*** cpuga has quit IRC08:22
*** sanfern has quit IRC08:23
*** sanfern has joined #openstack-lbaas08:23
*** krypto has quit IRC08:30
*** krypto has joined #openstack-lbaas08:31
*** korean101 has quit IRC08:34
*** krypto has quit IRC08:34
*** krypto has joined #openstack-lbaas08:35
*** korean101 has joined #openstack-lbaas08:48
*** krypto has quit IRC09:15
*** krypto has joined #openstack-lbaas09:15
*** cpuga has joined #openstack-lbaas10:20
*** cpuga has quit IRC10:24
*** krypto has quit IRC11:08
*** krypto has joined #openstack-lbaas11:08
*** krypto has quit IRC11:08
*** krypto has joined #openstack-lbaas11:08
*** krypto has quit IRC11:28
*** krypto has joined #openstack-lbaas11:28
*** Guest9296 has joined #openstack-lbaas12:07
*** Guest9296 has quit IRC12:12
*** krypto has quit IRC12:18
*** cpuga has joined #openstack-lbaas12:21
*** cpuga has quit IRC12:25
*** catintheroof has joined #openstack-lbaas12:54
*** krypto has joined #openstack-lbaas12:55
*** KeithMnemonic has joined #openstack-lbaas13:04
*** atoth has quit IRC13:18
*** atoth has joined #openstack-lbaas13:19
*** yamamoto_ has joined #openstack-lbaas13:41
*** yamamoto has quit IRC13:44
*** reedip has quit IRC13:46
*** reedip has joined #openstack-lbaas13:47
*** cpuga has joined #openstack-lbaas13:47
*** krypto has quit IRC13:48
*** cpuga has quit IRC14:05
*** cpuga has joined #openstack-lbaas14:05
*** leitan has joined #openstack-lbaas14:08
*** chlong has joined #openstack-lbaas14:14
xgerman_o/14:20
*** leitan has quit IRC14:25
*** leitan has joined #openstack-lbaas14:26
*** leitan has quit IRC14:30
*** fnaval has joined #openstack-lbaas14:59
*** krypto has joined #openstack-lbaas15:00
*** rcernin has quit IRC15:15
*** pcaruana has quit IRC15:20
*** fnaval has quit IRC15:25
*** kobis has joined #openstack-lbaas15:28
*** fnaval has joined #openstack-lbaas15:31
*** fnaval has quit IRC15:36
*** yamamoto_ has quit IRC15:39
*** yamamoto has joined #openstack-lbaas15:40
*** yamamoto has quit IRC15:44
*** armax has joined #openstack-lbaas15:57
*** kobis has quit IRC16:00
openstackgerritJason Niesz proposed openstack/octavia master: blueprint: l3-active-active  https://review.openstack.org/45300516:01
*** yamamoto has joined #openstack-lbaas16:13
*** csomerville has joined #openstack-lbaas16:21
*** cody-somerville has quit IRC16:24
johnsomI think this will fix the mkfs DIB failures:16:31
johnsomhttps://review.openstack.org/#/c/475022/16:31
johnsomIt looks like trove was also seeing these16:31
*** yamamoto has quit IRC16:38
*** amotoki is now known as amotoki_away16:47
sanfernHi,16:52
johnsomo/16:53
sanfern2017-06-16 15:22:28.996 3196 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ACTIVE in DB for load balancer id: 072322ad-e534-4784-ad3f-8a2c6f6f985016:53
sanfernwhen I look into the LB details - operating_status    | OFFLINE  and provisioning_status | ERROR16:54
*** sshank has joined #openstack-lbaas16:54
sanfernAny clue ?16:55
johnsomI would check the o-cw and o-hm logs.  Those are the two processes likely to transition an LB into ERROR state.16:56
johnsomFor provisioning status16:56
sanfern2017-06-16 15:18:38.200 3196 INFO octavia.controller.worker.tasks.database_tasks [-] Mark ALLOCATED in DB for amphora: 85325fe8-d5fb-4171-9674-0381e5515dae with compute id 1bdd935e-cdcd-4160-8548-8e4c51da4297 for load balancer: 072322ad-e534-4784-ad3f-8a2c6f6f985016:58
sanfern| 1bdd935e-cdcd-4160-8548-8e4c51da4297 | amphora-85325fe8-d5fb-4171-9674-0381e5515dae | 12d2570589d344f2b7eaf9bcefaf5b70 | ACTIVE | -          | Running     |16:58
johnsomYeah, the ERROR transition is going to be somewhere after 2017-06-16 15:22:28.996 3196 when it went ACTIVE16:59
*** krypto has quit IRC16:59
johnsomSince it went ACTIVE the sequence has to be ACTIVE->{maybe a PENDING_*}->ERROR17:00
*** blogan has joined #openstack-lbaas17:00
sanfernjohnsom, ok, if port_state is down will that cause for marking ERROR17:09
johnsomNo, that would impact operational status.  ERROR mean some form of command/control failed to complete as expected and the controllers could not automatically recover.  Like a nova call failed, or neutron call failed to execute.17:10
johnsomOne of the worker or health manager logs will have the ERROR event in it17:19
openstackgerritAdam Harwell proposed openstack/octavia master: Agent: swap flask responses to webob, handle 404 retries better  https://review.openstack.org/47479017:26
rm_workok lets see this thing in action17:26
*** SumitNaiksatam_ has joined #openstack-lbaas17:30
*** SumitNaiksatam has quit IRC17:32
*** SumitNaiksatam_ is now known as SumitNaiksatam17:32
*** catintheroof has quit IRC17:37
*** yamamoto has joined #openstack-lbaas17:39
*** yamamoto has quit IRC17:43
*** sshank has quit IRC17:54
*** sshank has joined #openstack-lbaas17:56
*** ipsecguy has joined #openstack-lbaas18:02
*** ipsecguy_ has quit IRC18:03
*** pcaruana has joined #openstack-lbaas18:04
*** JudeC has joined #openstack-lbaas18:05
*** armax has quit IRC18:05
*** sshank has quit IRC18:33
*** yamamoto has joined #openstack-lbaas18:40
*** pcaruana has quit IRC18:41
*** armax has joined #openstack-lbaas18:42
*** yamamoto has quit IRC18:44
*** SumitNaiksatam_ has joined #openstack-lbaas18:55
*** SumitNaiksatam has quit IRC18:57
*** SumitNaiksatam_ is now known as SumitNaiksatam18:57
johnsomrm_work No hits that round that I see.  Feel free to push your change19:05
rm_workyeah was just waiting for it to land19:06
rm_workso we get it logged19:06
rm_workotherwise the run goes *poof*19:06
johnsomOk19:06
rm_workit's kinda excruciating tho19:06
rm_worki really hate running the neutron gates19:07
johnsomWell, in this case it's a few more tries at hitting the issue19:08
johnsomrm_work Ok, there you go19:09
rm_workk19:12
openstackgerritAdam Harwell proposed openstack/octavia master: Agent: swap flask responses to webob, handle 404 retries better  https://review.openstack.org/47479019:13
rm_workdid we get any hits on the warning?19:13
rm_worki can't find any :/19:19
rm_workis there a logstash i can search i wonder19:19
johnsomI didn't see any either.  logstash is at http://logstash.openstack.org but it usually runs a pretty big backlog, so they may not show up until later tonight19:24
johnsomHmmm, the logstash graph looks pretty good today, maybe it's been fixed.  Or there just aren't as many jobs running anymore....19:25
johnsomI guess your mileage may vary....19:25
rm_worknot much running checks today yet19:26
rm_worki'll keep waiting for it to show up in a run19:26
rm_workwe did hit the mkfs thing...19:28
johnsomYeah, I have a patch up for it.19:28
rm_workyeah19:28
johnsomhttps://review.openstack.org/#/c/475022/19:29
rm_workyeah19:29
rm_worki've been watching it19:29
johnsomIt hit infra DNS issues the first round.19:29
rm_workthought about depends-on but i wanted to be able to merge first if necessary19:30
johnsomI love elastic recheck, yeah, we know it's broken, so recheck....19:30
johnsomExactly19:30
rm_workyeah why wouldn't it just .... run that one check again19:30
rm_workthat'd be actually useful19:30
rm_workTOO usefull O_o19:30
johnsomHa19:30
*** sanfern has quit IRC19:36
*** armax has quit IRC19:36
rm_worklol we hit mkfs again19:39
rm_workso i guess we are guaranteed another test run anyway :P19:39
*** yamamoto has joined #openstack-lbaas19:41
*** yamamoto has quit IRC19:45
*** tesseract has quit IRC19:53
johnsomI wish log highlighting worked for our logs.  I put in a change for that once, but then they changed things again.19:53
rm_work:/20:11
rm_workbrb breakfast20:11
xgerman_fish?20:14
*** sshank has joined #openstack-lbaas20:20
johnsomUgh, this mkfs thing is getting annoying.  Almost worse than the 404 issue.20:59
*** dayou has quit IRC21:14
*** dayou has joined #openstack-lbaas21:15
*** chlong has quit IRC21:30
openstackgerritJude Cross proposed openstack/octavia master: Add statistics to V2 API  https://review.openstack.org/47488521:31
*** yamamoto has joined #openstack-lbaas21:42
openstackgerritAdam Harwell proposed openstack/octavia master: Allow operators to tune VIP creation parameters  https://review.openstack.org/47481321:45
rm_workjohnsom: addressed21:45
*** yamamoto has quit IRC21:47
*** cpuga has quit IRC21:47
*** sshank has quit IRC21:47
*** cpuga has joined #openstack-lbaas21:48
*** cpuga has quit IRC21:54
rm_workjohnsom: seriously, not seeing the 404 thing on any of that run either >_<21:54
johnsomYeah, I know, I have been watching too.21:56
johnsomI wonder if it's because it is Friday and the servers are not busy....21:56
*** sshank has joined #openstack-lbaas21:57
rm_work>_<21:59
rm_workquick, kick a ton of rechecks on EVERYTHING22:00
rm_workyeah now it's mostly mkfs hitting us22:06
rm_workjohnsom: yeah what DID happen to the log highlighting...22:09
*** bzhao has quit IRC22:09
rm_workit used to work22:09
*** bzhao has joined #openstack-lbaas22:10
johnsomThat is the famous OpenStack quote these days22:31
*** yamamoto has joined #openstack-lbaas22:43
*** yamamoto has quit IRC22:49
*** sshank has quit IRC22:50
*** sshank has joined #openstack-lbaas22:55
johnsomWell, I don't know.  Maybe the root issue with the 404 got fixed yesterday.22:59
johnsomHa22:59
*** cpuga has joined #openstack-lbaas22:59
*** csomerville has quit IRC23:12
*** sshank has quit IRC23:14
openstackgerritMichael Johnson proposed openstack/octavia master: Add RBAC enforcement to Octavia v2 API  https://review.openstack.org/47287223:15
*** armax has joined #openstack-lbaas23:16
johnsomWell, that should be load balancer working under the new scheme.  Restacking to make sure the demo account can create LBs.23:18
*** pck_ has joined #openstack-lbaas23:20
*** kbyrne_ has joined #openstack-lbaas23:25
*** logan_ has joined #openstack-lbaas23:25
*** mjblack- has joined #openstack-lbaas23:25
*** kbyrne has quit IRC23:26
*** logan- has quit IRC23:26
*** pck has quit IRC23:26
*** mjblack has quit IRC23:26
*** bradjones has quit IRC23:26
*** kbyrne_ is now known as kbyrne23:26
*** mjblack- is now known as mjblack23:26
*** raginbajin has quit IRC23:27
*** logan_ is now known as logan-23:29
*** bradjones has joined #openstack-lbaas23:31
*** bradjones has quit IRC23:32
*** bradjones has joined #openstack-lbaas23:32
openstackgerritMichael Johnson proposed openstack/octavia master: Add RBAC enforcement to Octavia v2 API  https://review.openstack.org/47287223:32
*** raginbajin has joined #openstack-lbaas23:34
johnsomIan workflow-ed the mkfs fix, so should merge and not be a problem soon23:35
*** yamamoto has joined #openstack-lbaas23:45
*** yamamoto has quit IRC23:50
« Thursday, 2017-06-15 Index Saturday, 2017-06-17 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!