Friday, 2017-06-30

*** amotoki is now known as amotoki_away00:17
*** amotoki_away is now known as amotoki00:18
*** sshank has quit IRC00:19
*** amotoki is now known as amotoki_away00:27
*** sanfern has quit IRC00:34
*** sanfern has joined #openstack-lbaas00:36
*** junbo has quit IRC00:45
rm_workjohnsom: AH so00:48
rm_workjohnsom: if a failover fails, the LB is still ACTIVE, then you try to make a change, the change will fail due to the Amp being unreachable, then your LB is stuck in ERROR00:49
rm_workretry the failover, it works, LB is back up, but still in ERROR00:49
*** junbo has joined #openstack-lbaas00:49
johnsomUmmm, that doesn't sound like the upstream logic at all00:50
johnsomIf failover fails it should always go to ERROR from the capstone task00:51
rm_workerm00:56
rm_worki'm not seeing that00:56
rm_worklike failover BROKE in the middle of the run00:56
rm_workwith an exception00:56
rm_workthere00:57
rm_work*there's: AmphoraToErrorOnRevertTask00:57
rm_workbut00:57
rm_workI see nothing that would send the LB to error00:57
rm_worknor anything that would send it back to ACTIVE once a failover completes00:57
rm_work(the same behavior could probably be replicated just by trying to do an action DURING a failover)00:58
rm_workjohnsom: can you show me where the failover flow would make the *LB* immutable?00:58
johnsomOh, you are right!  it is not acting on the LB... Oye01:03
rm_workT_T01:03
johnsomSoB01:04
johnsomThat thing has been through a few iterations.01:04
rm_workI'm going to find ALL the bugs. In like a week. because production01:05
rm_work<_<01:05
* rm_work cries01:05
johnsomSo, we need to somehow detect if this is a spares or in-use and lock the LB01:05
rm_workwe know this01:05
rm_work# If this is an unallocated amp (spares pool), we're done01:05
rm_work        if status != constants.AMPHORA_ALLOCATED:01:05
rm_workI'm already futzing with my version of the failover01:06
rm_workI can try to fix them both01:06
rm_workjust, what do we do01:06
rm_workmark the LB ... PENDING_UPDATE?01:06
rm_workand then back01:06
rm_work?01:06
rm_workdo we need to lock it?01:06
johnsomYeah, just like an PUT call01:06
johnsomYeah, I think we should definitely lock it01:06
rm_worklike, actually do a lock attempt?01:07
rm_worki mean, lock yes, but *run the lock-get code*01:07
rm_workI think no01:07
rm_workbecause we need to just ... override whatever01:07
rm_workbut then we'll set it to ACTIVE at the end?01:07
rm_workdo we need to save the existing status?01:07
rm_workIDK01:07
johnsomSo, we need to basically call this:01:09
johnsom        self._test_lb_and_listener_statuses(01:09
johnsom            context.session, lb_id=db_pool.load_balancer_id,01:09
johnsom            listener_ids=self._get_affected_listener_ids(db_pool))01:09
johnsomWe are only acting on provisioning status, operating is measured and not a factor here.01:09
rm_workright but01:09
johnsomThen octavia.controller.worker.tasks.database_tasks.MarkLBAndListenersActiveInDB01:09
rm_workbut01:09
rm_workwhat if the amp crashes DURING an update01:10
rm_workthe LB is in PENDING_UPDATE01:10
rm_workwe send a new config01:10
rm_worksomething about that crashes the amp agent01:10
rm_workamp goes down01:10
rm_workwe try to lock01:10
rm_workfail01:10
johnsomWait, we are talking failover right?01:10
rm_workyes01:10
johnsomAny user action is locked with test_lb_and_listener_statuses01:10
rm_workyes01:11
rm_workbut i'm saying01:11
johnsomSo, no other controller or API can touch it.01:11
rm_workthat also CHECKS immutability01:11
rm_workwhat if it's ALREADY locked01:11
rm_workwhen we need to failover01:11
johnsomgotcha01:11
rm_workso, OK -- we ignore the check, and just straight up set the status01:11
rm_workso, at the end, we reset to ACTIVE?01:12
rm_workwhat happens to the status it was in?01:12
rm_workwhat if we failover during a PENDING_DELETE? (is that possible?)01:12
johnsomI think we should wait for the locking flow to release the lock.  Hmmm01:12
johnsomNo, delete should set HM busy to block failovers01:12
johnsomYeah, LB delete marks HM busy01:13
johnsomCan't start a failover01:13
johnsomThe locking flow *should* fail and go to ERROR01:13
johnsomRight?01:14
rm_workerr01:15
rm_workso a failover IMO is #1 priority01:15
johnsomI mean ignoring the lock is *bad* IMO.  It means two controller processes could be *TRYING* to act on the same entity01:15
rm_workany operation is invalid (besides DELETE?) if we have no amp01:15
rm_worki think the first thing we do is just SET the status of the LB, period01:16
rm_workthough it'd be nice to know what the status WAS01:16
johnsomYeah, I don't think so.  If it was ERROR the LB is in unkown state land and should stay in ERROR.  The user then has the choice of calling up the operator or deleting and re-creating01:17
rm_workif something is stuck in PENDING_UPDATE, it could still be serving traffic, and if we just don't failover because it's in that state, we have downtime01:17
rm_workhmm k01:17
rm_workbut ERROR is actually fine01:17
rm_worksince it's not immutable :P01:17
rm_workso we'd get a lock01:17
johnsomWe should never get "stuck"  That is fail on our part01:17
johnsomYeah, but failing over ERROR is just going farther into "unknown state" land01:18
rm_workok but01:18
rm_workthat means we should... check the state first?01:18
johnsomERROR is the "Oh Sh1t" state01:18
rm_workright but01:18
rm_worktest_lb_and_listener_statuses01:18
rm_workwill pass and give us a lock01:18
rm_workright?01:18
rm_workon ERROR01:18
johnsomNo, I don't think so01:18
rm_workERROR isn't immutable?01:19
rm_workI thought01:19
johnsomI am using that as an example, I don't think we can just use that code, but it should be similar01:19
rm_workhmmmmmmm01:20
rm_workok01:20
rm_workwe need to map this out I think01:20
rm_worki don't know whether "spec" style writeup or just ... code is better01:21
rm_workto start the conversation01:21
johnsomhttps://github.com/openstack/octavia/blob/master/octavia/db/repositories.py#L64101:21
johnsomYeah, in the back of my mind a state diagram would be nice, but, see above, hate graphviz01:22
rm_workMUTABLE_STATES includes ERROR, right?01:22
johnsomSo, delete is special case and bypassed ERROR01:22
rm_workalso: that was a weird way to do that01:22
johnsomYeah, agreed that code is ummm01:23
rm_workand the docstring is wrong01:23
rm_workactively wrong01:23
rm_worklol01:23
*** amotoki_away is now known as amotoki01:23
rm_workoh hmm01:24
rm_workI see01:24
johnsomSo, it's date night here.  I'm already getting an eye.01:24
rm_workERROR *is* immutable except for a delete01:24
rm_workalright01:24
rm_worklater :P01:24
rm_worki was wrong about ERROR immutability01:24
rm_workso fine01:24
rm_workalright01:24
johnsomYeah, it would be bad to allow update on ERROR01:24
johnsomI will try to check in later and see if you need to bounce something off mee01:26
rm_worki have it fixed kinda in my flow, going to port that to the main flow and put it up and call it a day01:41
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)  https://review.openstack.org/43561201:43
openstackgerritAdam Harwell proposed openstack/octavia master: Failover should cause LB to be Immutable  https://review.openstack.org/47910901:56
rm_work^^ there01:56
*** sanfern has quit IRC02:25
*** csomerville has quit IRC02:33
*** csomerville has joined #openstack-lbaas02:33
*** cody-somerville has joined #openstack-lbaas02:35
*** csomerville has quit IRC02:38
*** aojea has joined #openstack-lbaas02:50
*** aojea has quit IRC02:54
*** cody-somerville has quit IRC02:58
*** yamamoto has joined #openstack-lbaas03:05
*** kobis has joined #openstack-lbaas03:19
*** gans has joined #openstack-lbaas03:25
*** armax has joined #openstack-lbaas03:31
*** kobis has quit IRC03:35
*** sanfern has joined #openstack-lbaas03:38
*** armax has quit IRC03:48
openstackgerritSanthosh Fernandes proposed openstack/octavia master: This patch will emit provisioning status to be sync with neutron db  https://review.openstack.org/47838503:49
*** gans has quit IRC04:06
*** diltram has quit IRC04:18
*** diltram has joined #openstack-lbaas04:23
*** yuanying has quit IRC04:24
*** yuanying has joined #openstack-lbaas04:24
*** diltram has quit IRC04:37
*** diltram has joined #openstack-lbaas04:40
*** aojea has joined #openstack-lbaas05:27
*** yuanying_ has joined #openstack-lbaas05:32
*** yuanying has quit IRC05:34
*** aojea has quit IRC05:42
*** rcernin has joined #openstack-lbaas05:45
*** aojea has joined #openstack-lbaas06:00
*** aojea has quit IRC06:01
*** links has quit IRC06:07
*** links has joined #openstack-lbaas06:21
*** diltram has quit IRC06:22
*** diltram has joined #openstack-lbaas06:29
*** sticker has quit IRC06:31
*** diltram has quit IRC06:35
*** diltram has joined #openstack-lbaas06:42
*** pcaruana has joined #openstack-lbaas06:45
*** tesseract has joined #openstack-lbaas07:14
*** kobis has joined #openstack-lbaas07:17
*** aojea has joined #openstack-lbaas07:25
*** kobis has quit IRC07:30
*** gtrxcb has quit IRC07:38
openstackgerritcheng proposed openstack/octavia master: Fix url_path valid check  https://review.openstack.org/47922009:17
*** yamamoto has quit IRC09:35
*** yamamoto has joined #openstack-lbaas09:36
*** yamamoto has quit IRC09:38
*** yamamoto has joined #openstack-lbaas09:42
*** yamamoto has quit IRC09:42
*** links has quit IRC09:44
*** yamamoto has joined #openstack-lbaas09:46
*** yamamoto has quit IRC09:51
*** links has joined #openstack-lbaas10:01
*** yamamoto has joined #openstack-lbaas10:10
*** yamamoto has quit IRC10:12
*** yamamoto has joined #openstack-lbaas10:13
*** yamamoto has quit IRC10:16
*** yamamoto has joined #openstack-lbaas10:16
*** yamamoto has quit IRC10:18
*** diltram has quit IRC10:33
*** diltram has joined #openstack-lbaas10:37
*** diltram has quit IRC10:42
*** diltram has joined #openstack-lbaas10:47
*** yamamoto has joined #openstack-lbaas10:52
*** diltram has quit IRC10:55
*** sanfern has quit IRC10:57
*** diltram has joined #openstack-lbaas11:02
*** amotoki is now known as amotoki_away11:11
*** rtjure has joined #openstack-lbaas11:32
*** yamamoto has quit IRC11:34
*** diltram has quit IRC11:35
*** yamamoto has joined #openstack-lbaas11:36
*** diltram has joined #openstack-lbaas11:46
*** diltram has quit IRC11:51
*** diltram has joined #openstack-lbaas11:56
*** yamamoto has quit IRC11:58
*** diltram has quit IRC12:06
*** diltram has joined #openstack-lbaas12:13
*** amotoki_away is now known as amotoki12:19
*** yamamoto has joined #openstack-lbaas12:26
*** yamamoto has quit IRC12:55
*** yamamoto has joined #openstack-lbaas12:56
*** yamamoto has quit IRC13:01
*** diltram has quit IRC13:04
*** diltram has joined #openstack-lbaas13:08
*** links has quit IRC13:20
*** sanfern has joined #openstack-lbaas13:39
*** yamamoto has joined #openstack-lbaas13:58
*** reedip_ has joined #openstack-lbaas13:59
*** amotoki is now known as amotoki_away14:04
*** sanfern has quit IRC14:08
*** sanfern has joined #openstack-lbaas14:08
*** yamamoto has quit IRC14:10
johnsomo/14:12
johnsomLooks like devstack was broken last night (not our code).  I will kick off a recheck to see if it is fixed now14:13
johnsomhttps://www.irccloud.com/pastebin/ZdNrTlRN/14:14
*** reedip_ has quit IRC14:18
sanfernjohnsom, yes my patch failed :(14:21
johnsomYeah, a whole bunch of them did14:22
*** aojea has quit IRC14:34
xgerman_o/14:53
*** rcernin has quit IRC15:11
johnsomIt looks like devstack is fixed15:11
*** aojea has joined #openstack-lbaas15:47
xgerman_are healthmonitors top level nowerdays as well?15:49
*** aojea has quit IRC15:51
*** diltram has quit IRC15:53
johnsomWe have a handy API reference for that: https://developer.openstack.org/api-ref/load-balancer/v2/index.html#health-monitor15:53
johnsomGrin15:53
johnsomObject wise they are still under pools, but the API supports the concept of them being independent15:54
johnsomIt is actually a wonky-ness in the LBaaSv2 API if you ask me15:55
*** diltram has joined #openstack-lbaas15:59
*** mdavidson has quit IRC16:03
*** cody-somerville has joined #openstack-lbaas16:03
*** cody-somerville has joined #openstack-lbaas16:03
*** csomerville has joined #openstack-lbaas16:07
*** cody-somerville has quit IRC16:10
*** tesseract has quit IRC16:11
openstackgerritQingchuanHao proposed openstack/neutron-lbaas master: Fix object's dict variable can be changed unexpectedly  https://review.openstack.org/47936416:23
openstackgerritQingchuanHao proposed openstack/neutron-lbaas master: Fix object's dict variable can be changed unexpectedly  https://review.openstack.org/47936416:24
rm_workjohnsom: i've started to actually be ok with things being top-level16:57
rm_workwhich makes me feel dumb previously16:57
rm_workbut i think maybe it was not explained well16:57
rm_worklike if i get a unique ID for an object16:59
rm_workwhy should i have to track a bunch of other IDs16:59
rm_workjust to look it up16:59
rm_worki know what i want16:59
rm_workit's the pool+member thing we dealt with in the client17:00
rm_work"why do i suddenly need a poolID to look up a member I have a guid for"17:00
johnsomYeah, my comment is we are not consistent with it, which is a bit annoying17:01
*** aojea has joined #openstack-lbaas17:04
*** sshank has joined #openstack-lbaas17:14
xgerman_healthmonitor on top makes sense since our idea was to share them17:15
xgerman_same with pools17:15
xgerman_listeners17:15
xgerman_on the same token we could do members - what a rabbit hole17:15
rm_worki was always in favor of17:15
rm_workeverything is both17:15
rm_workit's not like it's hard17:15
johnsomUgh, this new theme for the docs is making me....  Grumpy17:17
*** reedip_ has joined #openstack-lbaas17:35
rm_workjohnsom: oh, it's not good?17:40
rm_worki glanced at it in the patch for it and it looked a little better I thought17:41
johnsomIt has worts IMO.  Like wanted a Contents on each page (I have already disabled this) and then sub-section titles being bold where the section titles are not.  Some of the font and layout is odd too17:42
rm_workjohnsom: uhhh did this test not get updated with the rest of the code, but everything still passed? lolol https://review.openstack.org/#/c/478859/17:42
johnsomWe will see what I come up with17:42
*** pcaruana has quit IRC17:42
johnsomEvidently17:43
rm_workugh i thought that test-and-set would raise17:45
*** sanfern has quit IRC18:03
openstackgerritAdam Harwell proposed openstack/octavia master: Failover should cause LB to be Immutable  https://review.openstack.org/47910918:26
rm_workjohnsom: ^^ ok there you go, that is what I meant to have happen18:26
johnsomOk18:26
*** reedip_ has quit IRC18:29
*** tomtomtom has joined #openstack-lbaas18:55
tomtomtomhi, anyone know what the "handler" would be from the octavia-api.log in this message?  2017-06-30 18:49:58.483 19126 INFO octavia.api.v1.controllers.load_balancer [req-770b6ae0-44da-4c60-b0a5-3766f2c22ea4 - 8826fa87d1544309bc4f36e650e37eb2 - default default] Sending created Load Balancer 34e270c0-d483-46a9-9079-beae82ee99d2 to the handler18:55
tomtomtomIt's configured in openstack newton with neutron18:56
tomtomtomthanks in advance for any help18:56
johnsomtomtomtom It is the handler that will take action on the request.  It is set in the configuration file: api_handler = queue_producer18:57
johnsomqueue_producer is a bad name for octavia handler18:57
johnsomIt puts the request on an oslo messaging queue which one of the octavia controller worker processes will pick up.18:58
johnsomThis will be changing, likely in queens as we add support for other providers (vendor drivers).18:58
tomtomtomok so somewhere in the creation of a load balancer I end up getting an error, would the next logical place to look be the octavia-worker log?18:58
johnsomYes, o-cw log18:59
tomtomtomthank you19:00
johnsomSure, come back if you need help understanding the error19:00
*** aojea has quit IRC19:09
tomtomtomYeah it's a failed to create compute instance, and it looks like it's complaining about this specifically: 2017-06-30 18:50:00.919 250 ERROR octavia.compute.drivers.nova_driver [-] Error building nova virtual machine.: GlanceNoTaggedImages: No Glance images are tagged with octavia-amphora-image tag.19:13
tomtomtomSo I probably need a glance image with that tag it can use to build the compute instance.19:13
johnsomYes, you need an amphora image setup19:14
tomtomtomOk I will check the docs on that then.  Thank you for your help.19:14
johnsomhttps://docs.openstack.org/developer/octavia/guides/dev-quick-start.html#create-amphora-image19:14
johnsomdiskimage-create -s 219:15
tomtomtomawesome! thanks19:18
*** m-greene- has joined #openstack-lbaas19:21
*** m-greene has quit IRC19:22
openstackgerritCarlos Puga proposed openstack/octavia master: Spec detailing Octavia service flavors support  https://review.openstack.org/39248519:55
*** aojea has joined #openstack-lbaas20:01
*** cpuga has joined #openstack-lbaas20:01
*** blogan has quit IRC21:05
rm_workxgerman_: https://review.openstack.org/#/c/478693/21:23
rm_workjohnsom: once that merges you can make a good release right?21:23
xgerman_k21:23
johnsomrm_work yeah, what about that other patch in there?21:23
johnsomhttps://review.openstack.org/#/c/478724/21:24
johnsomOh, I guess it's just tests so21:24
rm_workthough that LGTM too21:26
johnsomI down from over 100 errors in our docs and docstrings to <20...  So getting close.21:27
openstackgerritAdam Harwell proposed openstack/octavia master: Allow PUT to /pools/<id>/members to batch update members  https://review.openstack.org/47703421:28
rm_workjohnsom: \o/ yay21:28
rm_workeventually I would like to get people to look at that21:28
rm_workideally before feature freeze21:28
rm_workbut not a SUPER hurry21:28
rm_workobviously we've got some bugs that are higher priority :P21:28
johnsomBonus here is we will be doing syntax checking on docstrings going forward21:30
*** aojea has quit IRC21:31
*** cpuga has quit IRC21:32
*** diltram has quit IRC21:38
*** diltram has joined #openstack-lbaas21:43
*** gtrxcb has joined #openstack-lbaas21:45
rm_worki thought we already did21:53
rm_workT_T21:53
johnsomI can assure you, my morning has demonstrated that we are not21:53
rm_worklol.22:02
openstackgerritMichael Johnson proposed openstack/octavia master: Update Octavia docs for documentation migration  https://review.openstack.org/47894722:03
openstackgerritMichael Johnson proposed openstack/octavia master: Update Octavia docs for documentation migration  https://review.openstack.org/47894722:12
johnsomOk this is odd:22:18
johnsomhttp://logs.openstack.org/93/478693/1/gate/gate-python-octaviaclient-pep8-ubuntu-xenial/642b8c6/console.html#_2017-06-30_21_27_01_00219022:18
openstackgerritMichael Johnson proposed openstack/octavia master: Update Octavia docs for documentation migration  https://review.openstack.org/47894722:20
*** sshank has quit IRC22:21
openstackgerritMichael Johnson proposed openstack/octavia master: Update Octavia docs for documentation migration  https://review.openstack.org/47894722:27
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Remove rogue calls to str.upper in l7rules  https://review.openstack.org/47869322:29
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Remove rogue calls to str.upper in l7rules  https://review.openstack.org/47869322:57
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Remove rogue calls to str.upper in l7rules  https://review.openstack.org/47869323:09
openstackgerritMichael Johnson proposed openstack/octavia master: Update Octavia docs for documentation migration  https://review.openstack.org/47894723:12
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Remove rogue calls to str.upper in l7rules  https://review.openstack.org/47869323:14
johnsomNot my day23:14
*** armax has joined #openstack-lbaas23:22
*** armax has quit IRC23:41
rm_workT_T23:45
rm_workjohnsom: figure most of it out?23:45
johnsomYes23:46
johnsomThere is trickery for dealing with upper constraints.  So I added it to your patch23:46
rm_workah23:47
rm_worklol23:47
rm_workthat makes23:47
rm_workok23:47
rm_worki am not sure i get it but ok23:47
rm_work:P23:47
johnsomYeah23:47
rm_workneed xgerman_ on https://review.openstack.org/47869323:48
rm_worki'm heading out a little early23:48
rm_workbut PM me if you need any reviews tonight/weekend23:48
johnsomOk, I think I am good at the moment.  Would like feedback on the new docs layout when you are bored and have a few: http://docs-draft.openstack.org/47/478947/7/check/gate-octavia-docs-ubuntu-xenial/606bb7e//doc/build/html/23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!