Monday, 2018-04-16

« Sunday, 2018-04-15 Index Tuesday, 2018-04-17 »
*** slaweq has joined #openstack-lbaas00:39
*** dims has quit IRC00:42
*** slaweq has quit IRC00:44
*** tzumainn has joined #openstack-lbaas02:30
*** tzumainn has quit IRC02:58
xgerman_mnaser: johnsom os_octavia does not deploy the legacy policy by default:03:19
xgerman_https://www.irccloud.com/pastebin/ylGLi8fk/03:19
*** sanfern has joined #openstack-lbaas04:24
*** slaweq has joined #openstack-lbaas04:41
*** slaweq has quit IRC04:45
openstackgerritGanpat Agarwal proposed openstack/octavia master: Active-Active: ExaBGP amphora L3 distributor driver  https://review.openstack.org/53784205:42
*** sanfern has quit IRC06:50
*** slaweq has joined #openstack-lbaas06:51
*** rcernin has quit IRC06:56
*** velizarx has joined #openstack-lbaas06:56
*** pcaruana has joined #openstack-lbaas07:13
*** velizarx has quit IRC07:20
*** tesseract has joined #openstack-lbaas07:21
*** tesseract has quit IRC07:33
*** tesseract has joined #openstack-lbaas07:35
*** velizarx has joined #openstack-lbaas07:40
*** threestrands has quit IRC07:47
*** Alex_Staf has joined #openstack-lbaas07:49
*** AlexeyAbashkin has joined #openstack-lbaas07:58
openstackgerritJacky Hu proposed openstack/octavia-dashboard master: Allow detail pages to auto refresh upon action  https://review.openstack.org/56145808:14
openstackgerritJacky Hu proposed openstack/octavia-dashboard master: Allow detail pages to auto refresh upon action  https://review.openstack.org/56145808:18
openstackgerritJacky Hu proposed openstack/octavia-dashboard master: Allow detail pages to auto refresh upon action  https://review.openstack.org/56145808:35
*** gans has joined #openstack-lbaas08:39
openstackgerritGanpat Agarwal proposed openstack/octavia master: Active-Active: ExaBGP amphora L3 distributor driver  https://review.openstack.org/53784208:42
nmagnezidayou, thanks for the answer :) I'm re-stacking so will review it later today08:54
dayounmagnezi, thanks :)08:57
nmagnezidayou, np. let me know when you think the auto refresh patch is ready08:58
cgoncalvesdayou, thanks a zillion for the auto-refresh!08:59
dayounmagnezi, https://review.openstack.org/#/c/561457/, you needs some patching on horizon side08:59
dayoucgoncalves, my pleasure :)09:00
*** yboaron has joined #openstack-lbaas09:08
*** ipsecguy has quit IRC09:13
*** ipsecguy has joined #openstack-lbaas09:14
*** gans has quit IRC09:23
*** gans has joined #openstack-lbaas09:25
*** salmankhan has joined #openstack-lbaas09:44
*** gans has quit IRC09:48
*** salmankhan has quit IRC10:36
*** salmankhan has joined #openstack-lbaas10:37
*** salmankhan has quit IRC10:44
*** salmankhan has joined #openstack-lbaas10:47
*** salmankhan has quit IRC10:54
*** salmankhan has joined #openstack-lbaas10:54
*** salmankhan has quit IRC11:02
openstackgerritJacky Hu proposed openstack/octavia-dashboard master: Allow detail pages to auto refresh upon action  https://review.openstack.org/56145811:04
*** salmankhan has joined #openstack-lbaas11:10
*** salmankhan has quit IRC11:23
*** atoth has joined #openstack-lbaas11:59
openstackgerritJacky Hu proposed openstack/octavia master: Allow default pool without a name on single create  https://review.openstack.org/56160012:27
dayounmagnezi, auto refresh patch should be ready for review12:29
*** dims has joined #openstack-lbaas12:30
*** yamamoto has quit IRC12:38
mnaserxgerman_: awesome so i'm using the newer one :)12:47
*** oanson has quit IRC12:49
*** irenab has quit IRC12:49
*** irenab has joined #openstack-lbaas12:57
*** oanson has joined #openstack-lbaas12:57
*** ArchiFleKs has quit IRC13:00
*** velizarx has quit IRC13:01
*** phuoc_ has joined #openstack-lbaas13:01
*** yamamoto has joined #openstack-lbaas13:01
*** tzumainn has joined #openstack-lbaas13:05
*** velizarx has joined #openstack-lbaas13:20
*** KeithMnemonic has joined #openstack-lbaas13:22
gokhanhi folks, I installed octavia with openstack ansible . when I try to create loadbalancer ı am getting amphora not found instance couldn't be found error. This is error log:  http://paste.openstack.org/show/719298/ . How can ı solve this problem what is your opinions??13:33
*** velizarx has quit IRC13:36
*** velizarx has joined #openstack-lbaas13:43
*** yboaron has quit IRC14:39
*** mugsie has quit IRC14:53
*** mugsie has joined #openstack-lbaas14:53
*** mugsie has quit IRC14:53
*** mugsie has joined #openstack-lbaas14:53
openstackgerritJacky Hu proposed openstack/octavia-dashboard master: Allow detail pages to auto refresh upon action  https://review.openstack.org/56145815:03
*** pcaruana has quit IRC15:09
*** Alex_Staf has quit IRC15:10
*** velizarx has quit IRC15:14
xgerman_gokhan: the kog before that error should have it tryign to reach the amphora agent until it gives up… when I looked at your network config I didn’t see the veth pair between br-vlan and the octavia conatiner. Wonde rif you cna post an outout of `brctl show`15:42
xgerman_johnsom: rm_work my periodic job for osa borks on the diskimage build (need to replace that part with the download eventually) http://logs.openstack.org/periodic/git.openstack.org/openstack/openstack-ansible/master/openstack-ansible-deploy-translations-ubuntu-xenial/a8d4a5e/job-output.txt.gz#_2018-04-16_06_15_38_96506215:44
xgerman_anything we know about DOB breakage?15:45
xgerman_DIB15:45
johnsomYeah, DIB and devstack were both impacted by the pip 10 release over the weekend16:13
xgerman_k16:20
*** yamamoto has quit IRC16:41
*** velizarx has joined #openstack-lbaas16:45
*** yamamoto has joined #openstack-lbaas16:46
*** yamamoto has quit IRC16:51
*** samccann has joined #openstack-lbaas16:53
*** samccann has quit IRC16:53
*** samccann has joined #openstack-lbaas16:53
*** winircuser-713 has joined #openstack-lbaas16:56
*** winircuser-713 has left #openstack-lbaas16:58
*** yamamoto has joined #openstack-lbaas17:01
*** gokhan27 has joined #openstack-lbaas17:01
*** yamamoto has quit IRC17:06
*** yamamoto has joined #openstack-lbaas17:11
*** yamamoto has quit IRC17:11
gokhan27hi xgerman_  again, I created veth pairs. this is my last network config: http://paste.openstack.org/show/719312/ and this is my openstack_user_config.yml file for provider networks: http://paste.openstack.org/show/719313/17:14
*** velizarx has quit IRC17:15
gokhan27do you think which I may miss ?17:15
*** AlexeyAbashkin has quit IRC17:17
xgerman_you have the `brctl show` outout?17:17
xgerman_also some `ip a ` on the octavia cintainer is helpul17:18
gokhan27xgerman_: and this output of brctl Show: http://paste.openstack.org/show/719315/17:18
xgerman_ok, that looks good17:18
gokhan27xgerman_: and output of ip a in Octavia container: http://paste.openstack.org/show/719316/17:20
gokhan27xgerman_: ı see that Octavia management network is created by service Project not admin Project, is that be problem ?17:21
xgerman_no, shouldn;t be17:21
xgerman_that all looks fine17:21
xgerman_johnsom: and I had out fair share of network debugging :-)17:22
xgerman_there are two possible next steps:17:23
xgerman_1) enable some more advanced debug settings in Octavia OR17:23
xgerman_2) boot up a vm on the octavia management net and ping it from an octavia container (after the security groups are not blocking ICMP)17:23
gokhan27xgerman_: I made Octavia network share and public and created an instance with amphora agent image. I can reach this instance and ping it from Octavia container17:25
xgerman_ok, that’s good ;-)17:25
gokhan27xgerman_: when run 'openstack loadbalancer create lb --vip-subnet Octavia-subnet '17:26
gokhan27instance is created but it is shutdown :(17:26
xgerman_yes, if it can’t reach it’s REST API it will do that17:26
xgerman_mmh, we usually use not the Octavia management net for the VIP net but we have people for whom that works…17:28
gokhan27xgerman_: and ampora instances are taking 2 ips from neutron17:29
gokhan27may be it can not create communicate with neutron correctly17:30
xgerman_yep, one for the menagement net and one for the VIP net17:30
gokhan27xgerman_: hımm can ı use any private subnet for VIP ?17:30
xgerman_yes17:31
gokhan27xgerman_: sorry ı am new to Octavia and ı am asking a lot of questions :(17:31
gokhan27xgerman_: ok ı am trying now with private subnet17:31
xgerman_sounds good17:32
gokhan27xgerman_: after ı source my demo Project and try create loadbalancer , ı am getting http 403 error. ı can not reach Octavia ? What is the reason of this ?17:37
johnsomgokhan27 https://developer.openstack.org/api-ref/load-balancer/v2/index.html#faults17:38
johnsom403 means the Role Based Access Control (RBAC) is not allowing you to do that.17:38
johnsomgokhan27 https://docs.openstack.org/octavia/latest/configuration/policy.html17:39
johnsomIf you want the simpler RBAC config (owner or admin) copy https://github.com/openstack/octavia/blob/master/etc/policy/admin_or_owner-policy.json into your /etc/octavia directory as policy.json and restart the API processes17:40
gokhan27johnsom: ok but openstack-ansible-os_octavia does not install policy.yaml with default. I will add it now.17:41
johnsomgokhan27: right, it OSA runs with the default advanced RBAC enabled which does not require a policy file17:43
xgerman_yep, but we will create the roles ;-)17:43
gokhan27xgerman_: I really thank you for Octavia role :) it is good work. I have problems because ı do not have enough information about Octavia :(17:47
xgerman_no worries - make sure to submit a patch updating the docs so the next perosn has an easier time17:48
gokhan27xgerman_: yep if ı achieve create loadbalancers, I will submit a patch. where do I need define policy.yaml file in Octavia.conf?17:53
xgerman_https://github.com/openstack/openstack-ansible-os_octavia/blob/master/defaults/main.yml#L3317:54
xgerman_or you can add the loadbalancer_member role to the user you are using17:54
*** yamamoto has joined #openstack-lbaas17:57
gokhan27xgerman_: ok thanks17:59
*** yamamoto has quit IRC18:02
*** sanfern has joined #openstack-lbaas18:10
*** tesseract has quit IRC18:36
*** harlowja has joined #openstack-lbaas18:40
*** AlexeyAbashkin has joined #openstack-lbaas18:44
*** atoth has quit IRC18:46
*** AlexeyAbashkin has quit IRC18:49
*** AlexeyAbashkin has joined #openstack-lbaas18:49
gokhan27xgerman_: ı am getting errors on health manager http://paste.openstack.org/show/719320/. this is weird18:50
xgerman_yep, so if we recive haleth message from an amphora we don;t have in our DB we will log things like that — so this is ‘ok’ and the system will work/ignore them — but question is why are there amphoras which are not accounted for?18:53
xgerman_(we just added that so maybe need some “nicer” error message)18:53
gokhan27xgerman_: ı can see three amphora on db. I have HA installation and have 3 controllers. may be beacouse of that18:57
*** yamamoto has joined #openstack-lbaas18:58
xgerman_yeha, ther emigth be a decrepancy between what’s in the DN and what’s in nova list18:58
*** AlexeyAbashkin has quit IRC18:59
gokhan27xgerman_: ok if ı can ignore them, it is not problem :)19:00
xgerman_well, if it’s not too many it’s fine — if it gets out of hand it will chew up resources (we have like mnore then a 100 once and that was a drag)19:01
openstackgerritAdam Harwell proposed openstack/neutron-lbaas master: WIP: Test l7 proxy to octavia  https://review.openstack.org/56104919:01
*** yamamoto has quit IRC19:02
*** openstackgerrit has quit IRC19:05
rm_workxgerman_ / gokhan27 err, i don't think you should see *tracebacks*. That's wonky19:07
rm_workand, if you have amps that aren't in your system, that's generally a bad sign, as it would be a symptom of something broken19:08
xgerman_rm_work: yep, we clearly need to improve those error messages19:09
rm_workerr but why is that happening19:11
rm_workeugh19:11
xgerman_1) If you start amphora by hand19:12
xgerman_2) I have seen amps related  to  failing over an ERROR LB19:13
xgerman_or maybe not19:13
xgerman_3) If we send a nova dleete and nova doesn’t do it19:14
xgerman_(2) is speculation since that might also be part of (3)19:14
rm_workyeah 3 is not great if that's happening19:18
rm_worksomething is busted in your env and it's only going to get worse19:18
rm_worki had another "big hammer" change for that19:18
rm_workbut dunno what approach we want to take19:19
gokhan27xgerman_: I create load balancer with private subnet and it is now active :)19:20
xgerman_yeah, you are good19:20
xgerman_rm_work: I am on the fence of hammering #3 - on the one hand people should run an HA nova and if they do planned maintenance stop Octavia - on the other hand we should be able to handle short API outages19:21
johnsomI have a patch in flight to fix that traceback error.  Now how you got an amp in that state is a different question19:24
xgerman_he said earlier he started one amp by hand19:25
johnsomMust have configured it too19:26
johnsomhttps://review.openstack.org/#/c/561369/19:26
gokhan27xgerman_: I started more than one :)19:29
*** sanfern has quit IRC19:31
*** imacdonn has quit IRC19:35
*** imacdonn has joined #openstack-lbaas19:35
rm_workxgerman_: i mean, api outages is one thing, but nova consistently saying "deleted successfully!" and then leaving servers around, is another issue altogether19:56
rm_workbrb19:56
rm_workjohnsom: oh, do we have a plan for PIP? is that "not our problem" since it's primarily a DIB issue? is someone working it?19:56
johnsomWell, I think devstack is getting back into good shape, there are a few patches there, mine included.19:57
johnsomI have been poking at DIB this morning.19:57
johnsomIt's a bit complicated as cloud-init has a depends-on for the ubuntu pyyaml package.19:58
johnsomSo, still working on that one19:58
*** yamamoto has joined #openstack-lbaas19:58
*** yamamoto has quit IRC20:04
*** gokhan27 has quit IRC20:22
*** AlexeyAbashkin has joined #openstack-lbaas20:41
*** kbyrne has quit IRC20:42
*** kbyrne has joined #openstack-lbaas20:44
*** cgoncalves has quit IRC20:45
*** AlexeyAbashkin has quit IRC20:45
*** harlowja has quit IRC20:55
*** yamamoto has joined #openstack-lbaas21:00
*** yamamoto has quit IRC21:06
rm_workjohnsom: ok, i'm focusing internally until gate issues are resolved then, as it seems it's handled, and i can't do much upstream21:16
rm_workunless there's stuff that needs eyes21:16
* rm_work checks reviews etherpad21:16
johnsomYeah, everything is stuck at the moment, plus I'm getting pulled internal too21:17
*** samccann has quit IRC21:17
rm_workk21:18
nmagnezijohnsom, rm_work, yikes. and I thought that this breakage only occurs in my node and I should just deploy a clean one21:20
johnsomNo, it's the pip version 10 issue21:21
johnsomIt doesn't like distribution packages of python modules21:21
nmagnezi:<21:21
nmagnezican't we force it back to pip 9.x until it gets fixed? where is that bump coming from?21:22
johnsomYeah, that is what they did for devstack.21:22
johnsomI'm poking at DIB to see if there is another fix or not21:22
nmagnezianything I can do to help?21:23
nmagnezijohnsom, ?21:26
johnsomWell, I don't want to keep you up all night....21:26
nmagnezihaha21:26
nmagneziit's morning somewhere in the world I guess.. :)21:27
johnsomIf you are highly motivated: somewhere in here: https://github.com/openstack/diskimage-builder/tree/master/diskimage_builder/elements/pip-and-virtualenv21:27
johnsomWe need to pin the version of pip like they did for devstack https://review.openstack.org/#/c/561526/1/tools/cap-pip.txt21:28
johnsomThis is the script https://github.com/openstack-dev/devstack/blob/master/tools/install_pip.sh21:28
johnsomdevstack uses.21:28
johnsomSomeone could propose that patch while I keep playing with the elements to see if I can work around the problem package.21:29
nmagneziwill try to find my way around there21:30
nmagnezibrb21:30
*** openstackgerrit has joined #openstack-lbaas21:34
openstackgerritAdam Harwell proposed openstack/octavia master: When SG delete fails on vip deallocate, try harder  https://review.openstack.org/54926321:34
*** kong is now known as lxkong21:45
*** rcernin has joined #openstack-lbaas21:51
nmagnezijohnsom, https://review.openstack.org/#/c/561740/ (I still need to test it)21:53
johnsomCool21:54
nmagnezijohnsom,I'm building an image locally, will report soon21:58
*** yamamoto has joined #openstack-lbaas22:02
nmagnezijohnsom, still not working :<22:03
nmagnezi2018-04-16 22:02:22.513 |   Found existing installation: PyYAML 3.1122:03
nmagnezi2018-04-16 22:02:23.173 | Cannot uninstall 'PyYAML'. It is a distutils installed project and thus we cannot accurately determine which files belong to it which would lead to only a partial uninstall.22:03
nmagnezijohnsom, another fix is on the way https://review.openstack.org/#/c/561479/22:08
*** yamamoto has quit IRC22:08
*** slaweq has quit IRC22:16
*** slaweq has joined #openstack-lbaas22:16
nmagnezijohnsom, the log does not explain why it does not work22:18
nmagnezibut maybe it's just worth waiting for https://review.openstack.org/#/c/561479/ ?22:18
johnsomnmagnezi Ian claims his fix will solve this.  I didn't understand that from looking at the patch, but that is cool22:19
* nmagnezi crossing fingers for Ian22:19
johnsomI'm not sure how he is doing it.... I'm running into nasty cross-package issues22:19
nmagnezimaybe worth commenting this in the patch22:19
*** slaweq has quit IRC22:21
johnsomnmagnezi http://logs.openstack.org/40/561740/1/check/dib-dsvm-functests-python2-ubuntu-trusty/e35bcb7/job-output.txt.gz#_2018-04-16_22_17_33_59299822:27
*** yamamoto has joined #openstack-lbaas23:04
*** yamamoto has quit IRC23:09
*** ianychoi has quit IRC23:18
*** rm_mobile has joined #openstack-lbaas23:20
rm_mobile#txgaming23:20
rm_mobileErrrrrr23:20
rm_mobileThat was not where I meant to type that23:20
rm_worklol23:22
xgerman_it’s almost rm_work and rm_mobile are different people23:22
rm_workalmost :P23:22
*** rm_mobile has quit IRC23:22
johnsomWe only worry if they start talking to each other23:23
rm_workmulti-sleeving is punishable by Real Death23:23
rm_workI wouldn't dare23:23
rm_work(referencing Altered Carbon, for those who haven't seen it yet)23:25
*** ianychoi has joined #openstack-lbaas23:25
rm_workwait.. wut23:29
rm_workhow did... wut23:29
*** tzumainn has left #openstack-lbaas23:30
rm_workah nm23:31
rm_workjohnsom: err, this didn't set correctly: https://review.openstack.org/#/c/561049/7/zuul.d/jobs.yaml@7223:35
rm_workhttp://logs.openstack.org/49/561049/7/check/neutron-lbaasv2-dsvm-api-l7octavia/381bff3/logs/devstacklog.txt.gz#_2018-04-16_19_24_26_90823:36
rm_workdid I do it wrong?23:36
johnsomLooks fine there, let me check where you use it23:38
rm_workhttps://review.openstack.org/#/c/561049/7/devstack/plugin.sh@15923:46
rm_workthat was the log-line i linked23:46
rm_workit was just blank23:46
rm_workumm23:48
rm_workit didn't get set in the localrc23:48
rm_workhttp://logs.openstack.org/49/561049/7/check/neutron-lbaasv2-dsvm-api-l7octavia/381bff3/logs/local.conf.txt.gz23:48
rm_worknothing about PROXY23:49
rm_workso obviously I set it wrong23:49
*** ianychoi has quit IRC23:55
« Sunday, 2018-04-15 Index Tuesday, 2018-04-17 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!