Monday, 2017-05-29

*** amotoki_away is now known as amotoki01:03
*** yamamoto has joined #openstack-lbaas02:16
*** sanfern has joined #openstack-lbaas02:48
*** fnaval has joined #openstack-lbaas03:29
*** links has joined #openstack-lbaas03:35
*** links has quit IRC03:43
*** links has joined #openstack-lbaas03:45
*** aojea has joined #openstack-lbaas04:26
*** Dinesh_Bhor has joined #openstack-lbaas04:29
*** aojea has quit IRC04:30
*** krypto has joined #openstack-lbaas04:35
*** krypto has quit IRC04:35
*** krypto has joined #openstack-lbaas04:35
*** armax has quit IRC04:47
*** bzhao has quit IRC05:00
*** bzhao has joined #openstack-lbaas05:02
*** yamamoto has quit IRC05:02
*** gcheresh has joined #openstack-lbaas05:26
*** krypto has quit IRC05:29
*** yamamoto has joined #openstack-lbaas05:45
*** fnaval has quit IRC06:07
*** rcernin has joined #openstack-lbaas06:33
*** aojea has joined #openstack-lbaas07:20
*** aojea has quit IRC07:20
*** aojea has joined #openstack-lbaas07:20
*** tesseract has joined #openstack-lbaas07:33
*** krypto has joined #openstack-lbaas07:53
*** krypto has quit IRC07:58
*** krypto has joined #openstack-lbaas08:00
*** fnaval has joined #openstack-lbaas08:08
*** fnaval has quit IRC08:13
nmagnezi rm_work o/08:14
*** gans has joined #openstack-lbaas08:14
gansanyone facing issues with amphora image created via disk-image-create utility08:16
gansfor me it is giving error "Internal Server Error" while creating LB. I am using stable/newton08:17
nmagnezigans, hi there, where do you see the "Internal Server Error" ?08:18
nmagnezigans, also, which image are you using, Ubuntu?08:18
gansin octavia-worker log08:18
ganswhile plugging VIP in the amphora, it is making the REST call, but failing with code 50008:18
gansyeah, ubuntu-xenial08:19
nmagnezigans, i suggest you'll login the the amphora instance before you try to plug the vip and tail the amphora-agent log (/var/log/amphora-agent.log IIRC) to get more details08:19
gansit is deleting the VM once LB goes in ERROR state08:20
ganshow can i defer that08:20
nmagnezigans, i can't recall at the moment, i know some cores here know a way but I don't at the moment. anyhow, it won't matter since you'll tail the log *before* it is deleted so it will get printer to your terminal08:21
gans2017-05-29 08:20:34.186 13099 WARNING octavia.controller.worker.controller_worker [-] Task 'octavia.controller.worker.tasks.amphora_driver_tasks.AmphoraePostVIPPlug' (7e9fa09c-5047-46f6-b8be-b2c978e5c95c) transitioned into state 'FAILURE' from state 'RUNNING'08:22
gansthanks, i will try that08:22
nmagnezigans, yup. that log snippet does not say much :-)08:22
nmagnezigans, np08:22
gansi was just trying to point out the flow in which it is failing. Interesting thing, i am able to create LB with images generated on Saturday.08:23
gansHere is the log from amphora-agent08:45
ganshttp://paste.openstack.org/show/610830/08:45
nmagnezigans, strange.. i don't see anything going on there08:46
nmagnezigans, did you attach a vip?08:47
nmagnezigans, maybe try to create a listener?08:47
gansLB is going in error state and VM is getting deleted.08:48
gansi am not manually adding the VIP, it does automatically as part of loadbalancer-create08:48
nmagnezigans, yeah i know, but nothing happens in that log so I was wondering if that part works for you now08:49
nmagnezinevermind08:49
gansthis time i tried stable/ocata branch08:49
nmagnezihmm...08:49
*** belharar has joined #openstack-lbaas08:49
nmagnezigans, maybe you need to set a pdb in the worker to allow yourself time to login and also set a pdb in the agent for in the vip plug part so you get the exception08:51
nmagnezigans, if you know how, I can point you to the relevant parts in the code08:51
nmagnezigans, for the amphora vm you'll need to shutdown the agent from systemd and start it in a bash prompt so pdb will work08:52
gansnmagnezi, i know how08:52
nmagnezigans, that's what i would do to debug this08:52
nmagnezigans, ack08:52
nmagnezigans, shall I point you in stable/newton?08:53
gansi had done pdf for haproxy config on agent08:53
ganspdb*08:53
gansnow i am on stable/ocata08:53
nmagnezigans, just tell me which version you prefer and I'll provide the code entry points for you to place pdb :)08:54
gansstable/ocata please, sorry for confusion08:54
nmagnezigans, for the controller worker i'd place a pdb right after the request to nova: https://github.com/openstack/octavia/blob/stable/ocata/octavia/compute/drivers/nova_driver.py#L14508:56
nmagnezigans, at the amphora agent side, you can place a pdb here: https://github.com/openstack/octavia/blob/stable/ocata/octavia/amphorae/backends/agent/api_server/server.py#L14608:56
gansokay08:56
gansthanks, will try that now08:57
nmagnezigans, so place a pdb, restart the worker process and create a loadbalancer08:57
nmagnezigans, when the instance booted, login and place the breakpoint in the agent08:57
nmagnezigans, start the agent yourself (ps -ef | grep amphora-agent to get the command line for it), but don't forget to 'systemctl stop amphora-agent.service' first08:58
gansawesome, doing it now08:58
nmagnezigans, one last thing before you start the agent08:59
nmagnezigans, enable debug mode in /etc/octavia/amphora-agent.conf08:59
nmagnezican't hurt08:59
gansokay08:59
gansit will never :-)08:59
*** krypto has quit IRC09:01
ganshttp://paste.openstack.org/show/610833/09:12
gansamphora interface failed to come up09:12
gansnmagnezi, any clues09:14
*** krypto has joined #openstack-lbaas09:34
*** krypto has quit IRC09:34
*** krypto has joined #openstack-lbaas09:34
*** fnaval has joined #openstack-lbaas10:08
*** fnaval has quit IRC10:14
xgermanhttps://www.irccloud.com/pastebin/cy9h6pPO/10:17
nmagnezigans, check xgerman's url, he pointed you to the problem10:24
nmagnezixgerman, hi btw o/ :D10:24
gansxgerman, nmagnezi, i will try that now10:25
*** sanfern has quit IRC10:47
*** yamamoto has quit IRC10:55
gansi put breakpoint here : https://github.com/openstack/octavia/blob/stable/ocata/octavia/amphorae/backends/agent/api_server/osutils.py#L18611:03
ganson pdf prompt, fetched the command and executed on separate window, got same error11:03
gansubuntu@amphora-65f4000d-c620-48b0-a0b9-dd5f7814b831:~$ sudo su11:03
ganssudo: unable to resolve host amphora-65f4000d-c620-48b0-a0b9-dd5f7814b83111:03
gansroot@amphora-65f4000d-c620-48b0-a0b9-dd5f7814b831:/home/ubuntu# ip netns exec amphora-haproxy ifup eth111:03
gansCannot find device "eth1"11:03
gansFailed to bring up eth1.11:03
*** gans has quit IRC11:05
*** aojea has quit IRC11:08
*** circ-user-TeTGj has joined #openstack-lbaas11:13
*** openstackgerrit has joined #openstack-lbaas11:23
openstackgerritNir Magnezi proposed openstack/octavia master: Devstack plugin should mark the HM ovs port for cleanup skip  https://review.openstack.org/46052811:23
nmagnezixgerman, around?11:35
xgermanmy train has Wifi…11:35
xgermanfor a few more minutes ;-)11:35
nmagnezixgerman, lol :-)11:35
xgermanyeah, will be back in the states Friday night…11:36
nmagnezixgerman, something strange is going on. i set up a new devstack nodes (on Ubuntu, after it fails to create loadbalancers in Centos based devstack -> this is what I try to debug). anyhow it fails to create a loadbalancer for me11:36
nmagnezixgerman, now i get the same internal server error as gans11:36
nmagnezixgerman, *but*, the worker service refuses to stop at the breakpoint I set11:37
nmagnezixgerman, not sure why11:37
xgermanmmh, if eth1 is missing neutron either didn’t plug the port or it is under a different name11:37
nmagnezixgerman, http://paste.openstack.org/show/610850/11:37
nmagnezixgerman, i didn't even got the the point that I'm able to create an instance and debug it11:38
nmagnezixgerman, a Ubuntu image for the amphora btw11:38
xgermanah11:39
nmagnezixgerman, i just rebased one of my patches to see if anything comes up in zuul, maybe something is broken..11:39
xgermanyeah, it looks like the queue RPC part has issues…11:39
nmagneziyay :<11:40
xgermanso they use rabbit and there are rabbit CLI commands but I expect that maybe that new systemd stuff is causing trouble11:41
nmagnezixgerman, i'm still using screens11:41
xgermanok11:41
nmagnezixgerman, not sure if that's an issue or not11:41
nmagnezias if devstack is not bad enough..11:42
xgermanneither — so the command gets send ok from the controller?11:42
xgermancheck the controller logs…11:42
nmagnezixgerman, from what i go so far (and it's hard to debug when my breakpoints won't work) it spawns a nova instance but it fails not much afterwards and deletes that instance immediately11:43
nmagneziit looks like it just won't wait for the instance to boot11:44
*** yamamoto has joined #openstack-lbaas11:44
*** yamamoto_ has joined #openstack-lbaas11:44
nmagnezixgerman, maybe something changed in the image creation and broke the agent?11:47
*** yamamoto has quit IRC11:48
xgermanOk. Usually in debug logging it will have the nova error11:49
nmagnezixgerman, checking.. i managed to connect the the instance11:50
nmagnezixgerman, now the worker tries to communicate with the agent, which is how it should be but didn't happen up until now11:51
*** gans has joined #openstack-lbaas11:51
nmagnezixgerman, yup. I got the same issue as gans. looking into this.11:52
gansAwesome..So I am not alone11:52
nmagnezixgerman, i think it happens in the gates as well http://logs.openstack.org/28/460528/5/check/gate-octavia-v1-dsvm-py3x-scenario-ubuntu-xenial/db044e8/logs/screen-o-cw.txt.gz#_May_29_11_49_39_28915211:52
*** chandankumar is now known as chkumar|hackersp11:52
nmagnezigans, indeed11:53
nmagnezirm_work, ^11:53
*** links has quit IRC11:57
*** gans has quit IRC11:57
*** gans has joined #openstack-lbaas12:00
xgermanOk. Let's see how good the Swiss cell data is...12:02
nmagnezixgerman, ha.. i'm looking at this as well. will update you if/when i find anything12:03
xgermanOk. The gate can't connect to the amp12:04
xgermanSo it shuts down12:05
nmagnezixgerman, something is wrong with the agent12:06
nmagnezixgerman, until my instance boot i'm looking for recent merges in that area12:07
nmagnezixgerman, but it's possible that changes in diskimage-builder break us as well12:07
xgermanOk. There is a setting to prevent the vm from being deleted12:07
nmagneziplease share :-)12:08
nmagnezibtw it's worth documenting this somewhere12:08
nmagnezijohnsom once showed me how, but i forgot12:08
xgermanSame!12:08
*** gans has quit IRC12:09
xgermanOk. AFK for a while...12:10
nmagnezixgerman, ack12:10
openstackgerritEvgeny Fedoruk proposed openstack/neutron-lbaas master: Adding configuration flag for skipping SSL certificate verification  https://review.openstack.org/46886612:10
*** gans has joined #openstack-lbaas12:12
*** pcaruana has joined #openstack-lbaas12:20
*** gans has quit IRC12:20
*** gans has joined #openstack-lbaas12:22
*** gans has quit IRC12:23
*** yamamoto_ has quit IRC12:37
*** yamamoto has joined #openstack-lbaas12:37
*** belharar has quit IRC12:43
*** belharar has joined #openstack-lbaas12:45
*** aojea has joined #openstack-lbaas12:52
*** sanfern has joined #openstack-lbaas12:54
*** belharar has quit IRC13:01
*** gans has joined #openstack-lbaas13:04
*** gans_ has joined #openstack-lbaas13:08
*** gans has quit IRC13:10
*** gans_ has quit IRC13:10
*** gans has joined #openstack-lbaas13:10
nmagnezixgerman, i suspect the interface naming in the image was changed13:11
nmagnezixgerman, and i think we have eth1 hardcoded for the namespace13:11
nmagnezixgerman, so when it tries to ifup eth1 if just fails to locate it13:11
xgermanLikely - though we compare mac addresses--13:13
*** gans has quit IRC13:13
xgermanNot sure if that logic does something..13:13
*** gans has joined #openstack-lbaas13:14
nmagnezixgerman,13:14
nmagnezixgerman, http://paste.openstack.org/show/610876/13:14
nmagnezixgerman, and btw noticed another interesting thing13:15
nmagnezixgerman, we used to run the agent in python3 using the dib-python executable13:15
nmagnezixgerman, but now it just uses python213:15
nmagnezivery strange..13:15
xgermanYeah, that is weird as well...13:16
xgermanhttps://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/plug.py13:17
xgermanIt searches the name by the mac13:18
xgermanSo maybe an old version?13:18
nmagneziold version of what?13:18
nmagnezixgerman, we still have references to a hardcoded eth1 https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/plug.py#L17813:19
nmagnezixgerman, i need to go AFK, will reconnect a bit later to continue working on this13:19
*** gans has quit IRC13:21
xgermanYeah, that's bad. But we "name" it in the interface file... but who knows...13:21
johnsomWe assign those interface names when we add them to the namespace, so it is fine.13:40
johnsomTo stop reverts, edit common/taskflow_base and add the taskflow engine option alway_resolve =False (something like that).13:41
johnsomBack in town and working tomorrow13:42
openstackgerritEvgeny Fedoruk proposed openstack/neutron-lbaas master: Adding configuration flag for skipping SSL certificate verification  https://review.openstack.org/46886613:53
*** cpuga has joined #openstack-lbaas14:01
*** cpuga_ has joined #openstack-lbaas14:02
*** cpuga has quit IRC14:05
*** fnaval has joined #openstack-lbaas14:10
*** krypto has quit IRC14:10
*** krypto has joined #openstack-lbaas14:11
*** krypto has quit IRC14:11
*** krypto has joined #openstack-lbaas14:11
*** fnaval has quit IRC14:15
*** cpuga_ has quit IRC14:22
*** leitan has joined #openstack-lbaas14:28
leitanHi guys i was looking at manual rescheduling of lbaas to another agent on Mitaka, i know that automatic rescheduling has been implemented on Ocata when a down agent is detected, but is there any sort of manual evacuation on Mitaka ?14:29
leitanmaybe rm_work xgerman ?14:30
*** cpuga has joined #openstack-lbaas14:35
*** gcheresh has quit IRC14:39
*** catintheroof has joined #openstack-lbaas14:47
*** catinthe_ has joined #openstack-lbaas14:48
*** catintheroof has quit IRC14:52
*** krypto has quit IRC15:10
*** belharar has joined #openstack-lbaas15:16
*** armax has joined #openstack-lbaas15:22
*** cpuga has quit IRC15:28
*** cpuga has joined #openstack-lbaas15:29
*** cpuga has quit IRC15:34
*** chkumar|hackersp is now known as chandankumar15:36
*** krypto has joined #openstack-lbaas15:37
*** belharar has quit IRC15:48
*** JudeC has joined #openstack-lbaas15:50
*** tesseract has quit IRC15:53
*** rcernin has quit IRC16:01
*** aojea has quit IRC16:01
*** aojea has joined #openstack-lbaas16:02
*** aojea has quit IRC16:06
*** fnaval has joined #openstack-lbaas16:11
*** fnaval has quit IRC16:15
*** pcaruana has quit IRC16:17
*** gans has joined #openstack-lbaas16:21
*** JudeC has quit IRC16:21
*** fnaval has joined #openstack-lbaas16:22
*** cpuga has joined #openstack-lbaas16:29
*** armax has quit IRC16:36
*** cpuga has quit IRC16:37
*** cpuga has joined #openstack-lbaas16:38
*** armax has joined #openstack-lbaas16:41
*** fnaval_ has joined #openstack-lbaas16:42
*** cpuga has quit IRC16:43
*** fnaval has quit IRC16:45
*** pcaruana has joined #openstack-lbaas16:45
*** armax has quit IRC16:49
*** gans has quit IRC16:50
*** yamamoto has quit IRC17:06
leitansomebody ? rm_work xgerman17:12
*** krypto has quit IRC17:12
*** aojea has joined #openstack-lbaas17:14
*** armax has joined #openstack-lbaas17:14
*** aojea has quit IRC17:18
rm_workleitan: it has been a weekend / holiday in the IS17:22
rm_work*US17:22
rm_workand xgerman / johnsom were on vacation17:22
rm_worktoday is also a holiday17:23
rm_workerr, today is the holiday17:23
leitanooo17:52
leitanim sorry17:52
leitanbtw, made octavia work, just today17:52
leitanthe create_certificates.sh was messing with the client.pem at the end17:53
leitanrm_work: thanks, if you have it in mind, what i asked in the morning, let me know, if dont ill keep searching / wait17:54
rm_workleitan: hmm, curious what it was doing17:57
rm_workleitan: if there's a bug in that script (it is old and a little wonky, would love a ... better version of it) submit a bug report or patch! :)17:57
rm_workleitan: i really don't know about rescheduling in the agents, or really ANYTHING about neutron-lbaas unfortunately17:58
rm_workI'm focused almost 100% on octavia17:58
leitanrm_work: i imagine, thanks anyway, ill submit a bug report right away17:59
leitani need to get into the oepsntack dev circuit so i can push patches17:59
*** fnaval has joined #openstack-lbaas17:59
rm_workheh yeah there is a guide...18:00
rm_workumm18:00
rm_workhttps://docs.openstack.org/infra/manual/developers.html18:01
rm_workI think that18:01
rm_workmake sure you follow it exactly18:01
rm_workthere are some places where I skipped a step before and things don't work right T_T18:02
*** fnaval_ has quit IRC18:03
leitanrm_work: will do thanks !18:03
*** yamamoto has joined #openstack-lbaas18:06
*** armax has quit IRC18:14
*** yamamoto has quit IRC18:19
*** fnaval_ has joined #openstack-lbaas18:45
*** fnaval has quit IRC18:49
*** aojea has joined #openstack-lbaas18:52
*** gcheresh has joined #openstack-lbaas19:23
*** fnaval has joined #openstack-lbaas19:29
*** fnaval_ has quit IRC19:33
leitanrm_work: one quick question about octavia security groups, im managing them on the tenant service since i  used this credentials to the amphora part, but when a user creates a listener, it will update the secgroups or the admin need to change the secgroup generated for the LB ?19:50
*** fnaval_ has joined #openstack-lbaas19:53
*** fnaval has quit IRC19:57
johnsomThey are automatically updated19:59
leitanjohnsom: lovely thanks !  and ssh only listens on the mgmt interface of the mgmt network on the image correct ?20:11
*** pcaruana has quit IRC20:24
*** fnaval_ has quit IRC20:29
johnsomCorrect, and only if you specify a keypair in the config20:30
leitanjohnsom: roger that, thanks !"20:35
*** fnaval has joined #openstack-lbaas20:37
*** fnaval has quit IRC20:43
*** gcheresh has quit IRC20:44
nmagnezijohnsom, o/20:49
*** aojea has quit IRC20:58
*** aojea has joined #openstack-lbaas20:59
*** JudeC has joined #openstack-lbaas20:59
*** JudeC has quit IRC21:00
*** aojea has quit IRC21:03
nmagnezileitan, re: your question about auto-reschedule21:03
nmagnezileitan, to the best of my knowledge there was never a manual evacuation21:03
nmagnezileitan, i recall seeing a patch for such a thing on either ocata or pike but i don't think it was eventually merged21:04
leitannmagnezi: hi ! i know theres a feature (i guess since ocata) to auto-reschedule balancers from dead agents to alive ones, cause i saw the rewiew of the commit, but i wanted to know if previous to this, was a manual res-cheduling version21:04
leitanor if prior to acata if agent dies, all lb dies with hime21:05
leitanhim21:05
nmagnezileitan, to the best of my knowledge (i coded the auto-reschedule patch), prior to stable/ocata if the agent dies the loadbalancers will not get to other agents21:06
nmagnezileitan, that being said21:06
nmagnezileitan, keep in mind that the actual loadbalancing it done in the haproxy process21:06
nmagnezileitan, meaning that if the agent died, as long as the haproxy process is up loadbalancing will work21:06
nmagnezileitan, you will not be able to modify that loadbalancer with new configs21:07
nmagnezias long at that agent is dead21:07
leitannmagnezi: yes thats totally clear, i was thinking about a dead server that hosts the LB more than the agent itself21:07
leitanthinking about backporting that to mitaka, in the meantime i play with octavia21:07
nmagnezileitan, so yeah.. that is not a recoverable scenario in mitaka (at least until you revive the server)21:08
leitanto offer an ""HA"" to the users in the meantime21:08
leitannmagnezi: so great, that totally answer my question, thanks nmagnezi21:08
nmagnezileitan, well, please keep in mind that an official backport to Mitaka (or even newton) will not get accepted as it is against the openstack backports policy21:09
nmagnezileitan, obviously, on your machines you can modify the code to your liking but i just wanted you to know this21:09
leitannmagnezi: yes, it will be an "internal" backport21:09
nmagnezileitan, keep in mind that in order for this to work as an internal backport for mitaka you'll need to backport some things in Neutron as well21:10
nmagnezileitan, i was actually writing a  blog post about it, but didn't get the chance to finalize and publish it at the moment21:11
leitannmagnezi: i didnt analized the complexity of the backport yet, im planning to do that tomorrow to measure the effort of doing that or launching octavia into production sooner21:11
nmagnezileitan, anyhow, Octavia does work with HA in stable/mitaka :21:11
leitannmagnezi: yes, works ACTIVE_STANDBY, im currently testing that setup, it took me a while to get it working, but i got my first ACTIVE load balancer today :)21:12
nmagnezileitan, well done!21:12
leitannmagnezi: haha thanks, reading the ansible-playbooks was the best choice i made to finally understand everything21:14
nmagnezileitan, if there are any gaps you were able to fill by doing so, that's great. it would be awesome if you could file a bug against octavia with specific things missing from the docs so we can improve21:18
leitannmagnezi: yes, i filled a lot already, and also an RFE to "simplify" the mgmt network part21:19
*** catinthe_ has quit IRC21:20
*** leitan has quit IRC21:24
*** leitan has joined #openstack-lbaas21:25
*** circ-user-TeTGj has quit IRC21:33
openstackgerritNir Magnezi proposed openstack/octavia master: The VIP NIC should not be hard-coded to eth1  https://review.openstack.org/46896321:52
openstackgerritNir Magnezi proposed openstack/octavia master: The VIP NIC should not be hard-coded to eth1  https://review.openstack.org/46896322:49
openstackgerritNir Magnezi proposed openstack/octavia master: The VIP NIC should not be hard-coded to eth1  https://review.openstack.org/46896323:20

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!