Friday, 2016-02-05

*** longstaff has left #openstack-lbaas00:02
johnsomsbalukoff ping00:04
*** SumitNaiksatam has quit IRC00:04
sbalukoffjohnsom: Pong00:06
sbalukoffjohnsom: Er... did you want to chat?00:10
johnsomYeah, sorry, looked away for a minute00:11
johnsomI looked at the flow order situation.00:11
sbalukoffYeeees?00:11
johnsomI see why you did what you did.  However, I'm not sure it is the right answer.00:12
johnsomI see that Sherif put in that database lookup last summer which invalidates all of that model code.00:12
johnsomI'm wondering if the right answer isn't to remove his database lookup and fix whatever the model issue was he was hitting00:12
johnsomWhat really puzzled me was this had been fully tested (thus the issue with the member identified)00:13
sbalukoffPossibly. Does that need to be done before shared pools merges, though?  Because things are definitely not working correctly right now without the order fixed.00:13
sbalukoff(Even without the shared pools patch)00:13
johnsomSo I was pretty puzzled by the re-order00:13
rm_worki'm generally a fan of fixing the core problem and removing workarounds :)00:14
sbalukoffRight, so what's in place is a stop-gap. I get that. Just how much technical debt do I need to solve before this patch can merge?00:15
sbalukoff(It's actually getting really annoying eh...)00:15
johnsomWell, from my perspective the changes you made increased the technical debt as roll back is dirty now00:15
johnsomThat is my hangout00:15
johnsomhangup00:15
sbalukoffDoesn't the roll back mark the objects as the 'error' state anyway?00:16
johnsomI would +2 without the reordered flows.00:16
sbalukoff...except it doesn't work.00:16
sbalukoffSeriously, the stop-gap is better!00:16
sbalukoffRight now, in master, this is broken.00:16
johnsomWe can create and delete on master, so it's not totally broken00:17
sbalukoffYou can  change a pool, or health monitor, or whatever... and watch how nothing changes on the amphora.00:18
sbalukoffUntil you do it again.00:18
sbalukoffThat is broken.00:18
sbalukoffThe re-order means that if you change something, it changes on the amphora...00:18
sbalukoffAnd if something goes wrong with that change... well, aren't you already going to end up in an error state?00:19
sbalukoffRegardless?00:19
sbalukoffThe problem with "fixing" these flows is that we really don't have a good way to test them right now without scenario testing.00:21
johnsomIt's not about the error state.  It's about the database state for those amphora.  The config on the amphora no longer matches the database config.  It limits what we can do in the reverts.00:21
sbalukoff...00:21
sbalukoffYou realize that the example I gave means that's already happening right now, during normal operation?00:22
johnsomEither way this is more work.  If you want to leave the re-order then the model manipulation should be removed00:22
*** jwarendt has joined #openstack-lbaas00:23
sbalukoffHonestly, I would like to have better reverts...  but I also don't think we've thought through how most of these reverts should work anyway.00:23
johnsomPersonally I would flip them back, it's as broken as master, open a bug or update the one there and independent of this patch fix it.00:23
johnsomI agree with you on that00:24
johnsomThese changes aren't really related to shared pools00:24
sbalukoffThey're were only related in the fact that scenario testing is broken, and how are people supposed to test things like updating a member of a shared pool updates all its listeners without it?00:25
sbalukoffI'd prefer to merge this as is, and file a bug to revisit the revert scenario here and come up with a better solution.00:25
sbalukoffIn my mind, it's not acceptable for things to break like this during *normal* operation.00:26
johnsomAre you saying this won't make it through the gate without those flow re-orders?00:26
sbalukoffWhen you run into problems talking to an amphora or whatever... it's more acceptable for things to break. :/00:26
sbalukoffNo, it will make it through the gate-- look at the tempest tests that are running right now!00:26
johnsomBecause I am pretty sure patches have been getting in all day.00:26
johnsomRight.00:26
sbalukoffBut our neutron-lbaas tempest tests are apparently not very thorough if they're not catching this. :/00:27
sbalukoff(That is to say, neutron-lbaas's database and octavia's database get updated when, say, a change to a health monitor happens...  but the amphora does not, presently.)00:27
johnsomThat is true too.00:27
johnsomYeah, with that hack in listerupdate it is causing a lot of problems.  I see that.00:28
sbalukoffYep.00:28
sbalukoffSo...  I'll revert the change in task order if you really want me to, to get this patch merged... but then fixing this needs to be a high-priority item.00:29
sbalukoff(Again, I think it's personally better for the 'revert' case to be somewhat broken right now, instead of the normal workflow case.)00:30
johnsomIt's just the re-order is wrong and not related to this patch.00:32
rm_workyeah I'm torn00:33
johnsomSame here.00:33
rm_workbut tend to want to side with johnsom which i think is actually beneficial to you in a way sbalukoff -- undo the reordering in your patch, and we can fix it as a clear separate CR00:33
johnsomBasically the fix, will come back and flip them back along with addressing whatever the other hack issue was00:34
rm_workand you can stop cleaning up all of our technical debt as a prereq00:34
sbalukoff:)00:34
rm_workwhich IS definitely unfair00:34
*** jwarendt has quit IRC00:34
sbalukoffOk... I'll roll things back. Do you want me to put the re-order stop-gap into another patch for now, or are we not going to do that at all? (ie. leave this broken in master?)00:34
*** jwarendt has joined #openstack-lbaas00:35
sbalukoffAgain, this stuff is really hard to fix without a good way to test it.00:35
johnsomLet's leave master broken, but I will start another patch and try to figure this out.00:35
rm_workwe SHOULD do it00:35
sbalukoffjohnsom: Sounds good.00:35
rm_workbut it doesn't have to be done by you right now00:35
*** doug-fis_ has joined #openstack-lbaas00:36
johnsomWell, there is the old elbow grease manual testing which is how I did it originally00:36
sbalukoffOk, give me 15 minutes or so and I'll have those re-orderings rolled back.00:36
rm_workyeah i have been pushing internally for dedicating resources to getting scenario tests actually running on octavia (re, franklin basically)00:36
rm_workhopefully we'll get there00:36
sbalukoffjohnsom: Hah! That's how I figured out a re-order would fix things for the normal workflow.00:36
rm_workyeah having to manual test EVERY CHANGE before we workflow is annoying00:36
sbalukoffrm_work: Yeah, I have high hopes for what he produces.00:36
johnsomsbalukoff Thanks.  That was my last concern, so we should be good to go00:36
rm_workand slows velocity a ton because people don't want to +1 without testing >_<00:37
rm_workbecause we HAVE to test before +A00:37
rm_work(i meant +A not +1)00:37
sbalukoffjohnsom: I appreciate your efforts on this, eh!00:37
sbalukoffrm_work: Agreed!00:37
johnsomYeah, we put Min on starting scenario tests for Octavia, but we all got confused on the framework to use; tempest, tempest-lib, rally etc.  So we stopped work on it00:37
rm_workso many tests *almost* through zuul00:37
sbalukoffjohnsom: And I really do appreciate you voicing your concerns and not (always) letting me push you in directions that are not the best for this project. ;)00:38
sbalukoffWell...00:38
rm_workgoing to be afk a bit -- but will look at what you have tonight sbalukoff and even run some tests if it's got a +200:38
johnsomThanks for being open to a constructive conversation00:38
rm_workmaybe can get this finished out before the end of the week00:38
sbalukoffSo, I know we've hashed out this question a bazillion times... but, given it seems that using tempest means we have to copy-paste a crap ton of code that will eventually turn into cruft we all hate, does rally have a better story here?00:39
johnsomOh please don't make my head explode right now....00:39
rm_workI don't think so -- whichever way you go, you need to write models for the stuff you're testing, at least as far as i can comprehend00:39
sbalukoffjohnsom: I'll go ahead and open a bug for these re-ordering / revert problems we have right now, as I've gotten a pretty good look at them in the last couple of weeks.00:40
rm_workat least per the testing patterns i've done in the past00:40
johnsomCopying code in is really, really, really bad.  I will say that.00:40
sbalukoffUnless you're already on it.00:40
rm_workyeah we need to ditch all the "managed copy-paste" tempest stuff00:40
*** doug-fish has quit IRC00:40
rm_workhonestly I am not sure why we can't just import from tempest00:40
johnsomsbalukoff Go ahead.  You can assign it to me as well, I will work on it.00:40
rm_workinstead of copying it into tree00:40
sbalukoffrm_work: I know fnaval is working on that for us. I'm hoping he's successful in minimizing the cruft.00:40
rm_workunless it needs to CHANGE, and then it's not really just a copy/paste at that point00:40
johnsomI think that is what cedrick is working on for neutron-lbaas00:41
johnsomhttps://review.openstack.org/#/c/273817/00:41
rm_workwell the eventstreamer breakage fix will be merged momentarily, which is good00:41
sbalukoffjohnsom: I appreciate that! (I'm under a lot of pressure to get this L7 code to a state where y'all can start reviewing *all* of it, so not having to fix that myself is a very welcome thing for me right now!)00:41
rm_workand tests for your active/passive fix are at least out of queue00:42
rm_worklol and your revert fix has 3 +200:43
sbalukoffOk, gonna ignore chat for a bit to get this next patchset done...00:43
rm_workjohnsom: is that still a relevant change?00:43
rm_workI assume so00:43
rm_workbut i'm not sure how to effectively test it00:43
johnsomYes, it depends on the failover flow patch, so has been sitting for a while00:43
rm_worktempted to just +A and roll the dice, but that feels a little bit irresponsible >_>00:43
johnsomIf it maked you feel better, it has been +A'd before00:44
rm_workah wasn't actually sure of the order00:44
rm_worklol00:44
rm_workit ... kinda does :P00:44
openstackgerritMerged openstack/octavia: Fix event_queue entry in setup.cfg from EventStreamer break  https://review.openstack.org/27645100:44
rm_workjohnsom: you know you want it: https://review.openstack.org/#/c/275990/00:50
rm_workand also https://review.openstack.org/#/c/276368/ :P00:50
rm_workblogan: ^^00:50
johnsomrm_work I was just waiting for the gates so I could look in the logs00:50
rm_workjohnsom: testing out your new patch00:51
rm_workjohnsom: you'll find some stuff markedly absent :)00:51
rm_workit was o-cw00:51
rm_workin the latest CR it is SO clean :)00:51
johnsomYep, I am all too familiar with those log messages (thus the two bugs I opened)00:51
rm_workhttp://logs.openstack.org/68/276368/3/check/gate-neutron-lbaasv2-dsvm-scenario/06c7076/logs/screen-o-cw.txt.gz00:51
rm_workthe only other thing is the new key stuff we added in the first one...00:52
rm_workwhich just makes it more officially valid00:52
johnsomOh, there was a separate bug for the context issue.00:52
rm_workthough all that will be meaningless when we switch to Anchor tho00:53
johnsomThis was SSLContext: https://bugs.launchpad.net/octavia/+bug/149441700:53
openstackLaunchpad bug 1494417 in octavia "REST driver - A true SSLContext object is not available" [Low,In progress] - Assigned to Adam Harwell (adam-harwell)00:53
rm_workyes00:53
rm_workit's marked00:53
johnsomThis was subjectAlt: https://bugs.launchpad.net/octavia/+bug/149442000:53
openstackLaunchpad bug 1494420 in octavia "REST Driver TLS - Certificate has no `subjectAltName" [Low,Confirmed] - Assigned to Carlos Garza (carlos-garza)00:53
rm_workyeah00:53
rm_workthey're both on their respective CRs00:53
johnsomNope, both have the same bug number on them00:54
rm_workerr00:54
johnsomOh, wait, wrong windows00:54
rm_work:P00:54
johnsomStrange, yeah, ok, you commented the same, it didn't mark it in launchpad00:55
rm_workyeah <_<00:55
rm_workno clue why not00:55
johnsomI have had that happen to me a few times too00:55
johnsomI think on yours, gerrit ignores the stuff below the change-id00:56
rm_workcrap I shouldn't have let xgerman's anchor plugin through yet00:56
rm_workhe didn't actually fully implement the API for it00:56
rm_workso it won't work :/00:56
rm_workjohnsom: i was wondering if that was the issue00:57
rm_workis it worth fixing the commit message? :/00:57
rm_worki think I'll just update the bug00:57
johnsomYeah, just the bug.  You don't want to wait for zuul again00:58
rm_workexactly T_T00:58
rm_workaight, AFK for a bit00:59
rm_workback later00:59
rm_workoh wait00:59
rm_workwas gonna finish this test00:59
rm_workwhich will let me +2 you01:00
openstackgerritStephen Balukoff proposed openstack/octavia: Shared pools support  https://review.openstack.org/25636901:00
sbalukoffjohnsom, rmwork ^^^^ task order should be back to master in that.01:01
rm_workk01:01
rm_worki will look at that later tonight01:01
*** chlong has joined #openstack-lbaas01:01
*** ducttape_ has joined #openstack-lbaas01:01
sbalukoffI think it was the health_monitor, listener, pool and member flows which are the ones affected. Please let me know if you find any others that need attention.01:01
* sbalukoff goes off to file a bug.01:01
sbalukoffThanks, rm_work!01:02
*** bana_k has quit IRC01:02
johnsomThanks!01:02
sbalukoffNo problem.01:02
rm_worknp sbalukoff01:02
rm_workcool johnsom, WFM now01:02
rm_work+2'd01:03
rm_worknp sbalukoff, thank me again when I actually +2 it :P01:03
johnsomAwesome01:03
rm_workright now all I have to give is empty promises and broken dreams01:03
sbalukoffrm_work: That's my breakfast every morning.01:04
rm_workHearty. Hopefully plenty of fiber.01:04
rm_workbbl01:05
bloganrm_work: does taht review, the ssl context one, just install that lib?01:05
bloganand is it in global reqs?01:05
rm_workblogan: two of the three are in global reqs01:05
rm_workand yes that is all it has to do01:05
rm_workbecause pyopenssl injects itself into urllib if the right stuff exists01:05
bloganokay01:05
rm_worktake a look at the scenario test o-cw logs01:05
blogani did01:05
rm_workyeah01:06
bloganthey need to have debug turned on01:06
bloganbut yeah its nice they dont have that SSLContext shit in there01:06
rm_worksoooo i don't know if global-reqs will kick it back tho for the third req01:06
rm_workbut i say we try to merge it and find out :P01:06
bloganit'll fail devstack if its in global reqs though bc of the sync requirements does i thought01:07
rm_workmaybe?01:07
rm_workin which case, woot01:07
rm_workseems to be ok01:07
rm_workblogan: for context: http://stackoverflow.com/a/29202163/465571701:08
bloganexcellent01:08
johnsomYeah, I was surprised to see that debug wasn't on for the gate jobs...01:09
rm_workthat's a one-liner iniset right?01:09
bloganwonder if that'll work with my local clients01:09
rm_workshouldn't have any problems01:09
rm_workwe already pulled in pyopenssl, and the other two are minor libs01:09
bloganwelp since i stack locally, i'll get it soon enough :)01:09
rm_workheh01:09
johnsomYeah, if there is a package in there that isn't in global req's it will get kicked out01:10
rm_workwe'll see01:10
rm_workah but that one depends on https://review.openstack.org/#/c/275990/ so need that to +A too01:10
rm_workand now, for serious, I am going afk for a while :P01:11
openstackgerritMichael Johnson proposed openstack/octavia: Updates the failover flow for active/standby  https://review.openstack.org/25372401:15
johnsomDarn, ptoohill had a good point, I should have updated the taskflow version in requirements.txt.  It worked fine on my fresh build, but might bite others.01:15
ptoohillit bit me, but rm_work brought up global requirements, im not sure what the right way is :P01:18
*** Aish has quit IRC01:20
johnsomYeah, I saw the update where they added it to the upper constraints file, but I didn't notice that the main global requirements is still at 1.2501:22
johnsomOk, catch you all tomorrow01:24
*** ducttape_ has quit IRC01:24
sbalukoffjohnsom:  There's your bug, eh: https://bugs.launchpad.net/octavia/+bug/154212001:27
openstackLaunchpad bug 1542120 in octavia "Changes made via API do not immediately get pushed to amphorae" [Undecided,New]01:27
*** chlong has quit IRC01:34
*** chlong has joined #openstack-lbaas01:35
*** piet has joined #openstack-lbaas01:38
*** allan_h has joined #openstack-lbaas01:38
*** _cjones_ has quit IRC01:40
*** allan_h has quit IRC01:40
*** raginbajin has quit IRC01:42
*** bharathm has joined #openstack-lbaas01:44
*** raginbajin has joined #openstack-lbaas01:46
*** bharathm has quit IRC01:48
*** bharathm has joined #openstack-lbaas01:49
*** armax has joined #openstack-lbaas01:51
*** doug-fis_ has quit IRC01:51
*** doug-fish has joined #openstack-lbaas01:52
*** doug-fis_ has joined #openstack-lbaas01:53
*** Aish has joined #openstack-lbaas01:53
*** doug-fish has quit IRC01:57
*** madhu_ak_ has quit IRC01:57
*** ducttape_ has joined #openstack-lbaas02:03
*** Aish has quit IRC02:08
*** davidlenwell has quit IRC02:18
*** davidlenwell has joined #openstack-lbaas02:19
*** yamamoto_ has joined #openstack-lbaas02:21
*** davidlenwell has quit IRC02:23
*** davidlenwell has joined #openstack-lbaas02:27
*** piet has quit IRC02:27
rm_workdependent patch has +A but no love for https://review.openstack.org/#/c/275990/ ? lol02:49
*** Aish has joined #openstack-lbaas03:00
rm_workjohnsom: you around?03:01
rm_workor sbalukoff or ptoohill03:01
ptoohillrm_work: I'm here03:02
rm_workis there any documentation for why taskflow 1.26 is needed? was there a bug or something I can reference? we need to bump it in global-requirements, just figuring out a good commit message, they like completeness03:02
ptoohillThe depends on03:02
ptoohillThat bug was merged into 1.26, this patch cannot function without it03:02
rm_workI mean, I know that we need it03:03
rm_workbut03:03
rm_workI would like some sort of written reference that SHOWS why we need it03:03
ptoohillhttps://review.openstack.org/#/c/246051/03:03
rm_workthanks03:03
ptoohillurm03:03
rm_workthat should work03:03
rm_workat least it's solid03:03
*** doug-fis_ has quit IRC03:03
*** doug-fish has joined #openstack-lbaas03:04
ptoohillWell there's no description there, but this was in the DependsOn in the patch in question. We need it because it fixes a bug with decider graphs/flows03:04
rm_workyeah03:04
ptoohilli dont' quite remember specifics off hand03:04
rm_workhttps://review.openstack.org/27654903:04
ptoohillworks for me03:05
rm_workI have done this before so I KNOW they will ask questions like this03:05
ptoohillThank you03:05
rm_workand it's a lot faster if we just give them the info up front03:05
ptoohill'because we want it.'03:05
rm_workapparently johnsom's patch *will* fail to merge if the deps don't match global-req *exactly*03:06
rm_workso we need this in first03:06
ptoohillah, k03:06
rm_workand I guess I need to get them to add ndg-httpsclient too03:06
rm_workwhich will be a lot less simple than a bump by 0.103:06
ptoohill:(03:07
*** doug-fish has quit IRC03:08
*** Aish has left #openstack-lbaas03:13
*** yuanying_ has joined #openstack-lbaas03:18
*** yuanying has quit IRC03:21
*** doug-fish has joined #openstack-lbaas03:24
*** neelashah has joined #openstack-lbaas03:27
*** doug-fish has quit IRC03:28
*** piet has joined #openstack-lbaas03:33
*** ducttape_ has quit IRC03:35
*** ducttape_ has joined #openstack-lbaas03:37
xgermanrm_work what did I forget with Anchor — it worked fine in my devstack?03:39
bloganlov given03:39
*** yuanying has joined #openstack-lbaas03:42
*** fnaval has quit IRC03:44
*** yuanying_ has quit IRC03:44
*** piet has quit IRC03:45
rm_workxgerman: i am not sure HOW it worked -- you are completely missing implementation for one of the methods in the interface03:45
rm_workunless it wasn't actually used somehow?03:45
rm_workbut otherwise I'm not sure how the key and CSR were being generated03:46
xgermanoh, the CSR will be generated by the code in the local generator03:46
rm_workerrrr03:46
rm_workso, the way it works03:47
xgermanso I inherit from the local generator and just do the signing with Anchor03:47
rm_workAH03:47
rm_workok I see03:47
rm_workyou inherit03:47
rm_workthat's ... lulzy03:47
rm_workI guess it works03:47
rm_workI was wondering how that would ever have worked for you03:48
rm_work(that is how)03:48
rm_workalso: https://review.openstack.org/27655403:48
rm_workand https://review.openstack.org/#/c/276549/03:48
rm_workblogan / xgerman03:48
rm_workthe second should be a no-brainer03:48
rm_workthe first I need to see what people's thoughts are03:48
johnsomrm_work the taskflow bug was: https://bugs.launchpad.net/taskflow/+bug/147946603:48
openstackLaunchpad bug 1479466 in octavia "The needs to treat subflows as standalone entity" [High,Fix released]03:48
rm_workI assume there will be a general sense of distrust about adding another completely new lib03:49
rm_workthanks johnsom03:49
xgermanrm_work I give it slim chances — I think I hacked the https context somehow that it didn’t but should work03:49
xgermanalso added jharlow to the task flow one03:51
johnsomStill don't understand why the gate would block it, the global requirements have >= and upper constraint == 1.2603:52
rm_workyeah well03:52
rm_workme either03:52
rm_workbut03:52
rm_work[20:59:39]  <clarkb>rm_work: has to match exactly03:52
openstackgerritBrandon Logan proposed openstack/neutron-lbaas: WIP - Genericize octavia event handler  https://review.openstack.org/27655703:53
rm_workreally, you're doing that already?03:53
rm_workI was going to wait for someone to ask at least03:53
rm_worklol03:53
rm_workI guess it looks like that was a pretty simple fix03:55
bloganrm_work: i had started on it weeks ago and just hadn't pushed the code up03:58
bloganso not sure exactly if it works right now, pretty sure it needs tests updated03:59
*** localloop127 has joined #openstack-lbaas04:01
rm_workwoo, hynek fixed pem04:01
rm_workso now we could theoretically use it04:01
rm_workand remove the manual string parsing we do for cert loading/splitting04:02
bloganyou mean the low level stuff?04:04
* blogan ducks04:04
*** jwarendt has quit IRC04:04
crc32|zncwow04:05
crc32|zncsplitting a string is considered too low level.04:06
bloganshhh04:06
crc32|zncwhat library do you plane to use to split PEMs this time.04:06
*** yuanying has quit IRC04:06
bloganhook, line, sinker04:06
crc32|zncfor real. The last pemsplitter didn't even have a working setup.py04:07
rm_work"pem"04:07
*** yuanying has joined #openstack-lbaas04:07
blogani like perm splitters04:08
crc32|zncand when that pem splitter breaks don't forget to baby sit a patch and submit a fix to the pem splitter library frame work.04:08
*** fnaval has joined #openstack-lbaas04:08
*** ducttape_ has quit IRC04:08
*** neelashah has quit IRC04:18
xgermandidn’t realize rm_work was feeling that strongly about my workaround: https://review.openstack.org/#/c/276368/04:18
rm_worklol04:20
rm_workI would like to do things the right way generally :P04:20
rm_workbut also, that spam is super annoying T_T04:20
rm_workI guess we could ignore all warnings from urllib <_<04:21
rm_workbut, we should just do it the right way04:21
xgermanyeah, agreed04:21
xgermanalso in other new Michael + I will be in San Antonio in May04:23
xgerman5/15-18 - in case you need to skip town :-)04:23
bloganfor what?04:25
* blogan hears the air raid sirens already04:25
xgermansome HP conference04:25
xgermanthey decided San Antonio would be nice to fly a 100 HP people to04:28
rm_work:P04:28
rm_workwell, I'm always down for BBQ04:28
bloganmore bbq!04:28
rm_workW^_^04:28
xgermanyeah, so are we :-)04:29
bloganat least you're coming in the spring when its not so hot04:40
bloganoh wait, its texas04:40
bloganit'll be 10004:40
*** links has joined #openstack-lbaas04:43
sbalukoffHeh!04:50
openstackgerritMerged openstack/octavia: Add SubjectAltName and KeyUsage for localcertgen  https://review.openstack.org/27599004:53
openstackgerritMerged openstack/octavia: Fix for ssl context wrapper spam  https://review.openstack.org/27636804:57
openstackgerritStephen Balukoff proposed openstack/octavia: Add L7 database structures  https://review.openstack.org/26543004:59
*** localloop127 has quit IRC05:05
*** crc32 has joined #openstack-lbaas05:05
openstackgerritStephen Balukoff proposed openstack/octavia: Update repos for L7 objects / methods  https://review.openstack.org/26552905:08
*** amotoki has joined #openstack-lbaas05:11
*** numan_ has quit IRC05:12
*** piet has joined #openstack-lbaas05:32
sbalukoffBlah...  probably one of the more annoying factors you have to watch for when maintaining a long chain of giant patches are to find those areas where the auto-merge process that happens when you rebase fails to do the right thing. :P  Especially if the error introduced by this is subtle and wouldn't be caught by an existing test. :P05:37
openstackgerritStephen Balukoff proposed openstack/octavia: Add L7 database structures  https://review.openstack.org/26543005:37
openstackgerritStephen Balukoff proposed openstack/octavia: Add L7 database structures  https://review.openstack.org/26543005:38
openstackgerritStephen Balukoff proposed openstack/octavia: Update repos for L7 policy / methods  https://review.openstack.org/26552905:44
*** numan_ has joined #openstack-lbaas05:47
rm_workwtf how did that second match merge without ndg-httpsclient in global-reqs >_>05:53
rm_workweeeeird05:53
rm_workanywho, I think this is good to go, someone else had +2'd in the past right? https://review.openstack.org/#/c/253724/05:55
rm_workah, xgerman had05:55
*** woodster_ has quit IRC05:56
xgermanOk, +2 +a05:57
*** piet has quit IRC05:59
*** prabampm has joined #openstack-lbaas06:00
rm_workyeah now i get to test sbalukoff's stuff06:01
sbalukoffYay?06:01
sbalukoff;)06:01
sbalukoffThank you for doing that, rm_work!06:01
rm_workhttps://review.openstack.org/#/c/256369/ is first?06:01
sbalukoffCorrect.06:02
sbalukoffThe others in the chain you needn't worry about tonight, eh.06:02
rm_workoh man i have to test this stuff manually with curl don't I >_>06:04
sbalukoffrm_work: yeah. :/06:05
sbalukoffrm_work: well....06:05
sbalukoffactually, you could also put the neutron-lbaas shared pools patch and cli patch on your devstack.06:06
sbalukoffThat works too.06:06
rm_workheh06:06
rm_workmight do that06:06
rm_workwell06:06
sbalukoffThey're listed at the top here: https://etherpad.openstack.org/p/lbaas-l7-todo-list06:06
rm_workeh, already started stacking06:06
rm_worki'll figure out the curl stuff06:06
sbalukoffalso keep in mind our discussion about task ordering . At this point, when you ask for a pool to get updated... do it twice.06:07
sbalukoffBecause of the bug in master. :P06:07
rm_worklol06:07
rm_workright T_T06:07
*** yuanying_ has joined #openstack-lbaas06:08
sbalukoffrm_work:  Maybe helpful: https://gist.github.com/anonymous/f59d96918f3103ad565106:09
*** yamamoto_ has quit IRC06:09
xgermanWe will have Min check it out on06:09
xgermanOur end...06:09
sbalukoffObviously, replace UUIDs as necessary.06:09
rm_workso, with big patchsets, I am sometimes a fan of "just merge it, and hotfix the issues" <_<06:10
rm_workif it looks like it should be working 95%06:10
rm_workwe're still a ways off our feature freeze, and even then have even longer for bugfixes...06:10
xgermanDon't try to kill Johnson06:11
rm_workheh06:11
sbalukoffWell...  I feel the same way so long as people have given it a good run through some manual scenarios.06:11
rm_workyeah06:11
sbalukoffI think this patch has seen quite a bit of that. :)06:11
rm_worki'd like to see basic stuff working06:11
sbalukoffYep.06:11
rm_workbut i'm not going to try to hit every edge case manually06:11
*** yuanying has quit IRC06:11
sbalukoffRight.06:12
rm_workand yes, that gist is exactly what i was considering asking you for :P06:12
sbalukoffI look forward to being able to do actual scenario tests.06:12
sbalukoffAgainst Octavia directly.06:12
sbalukoffI'm pretty jazzed about the work fnaval is doing. :)06:13
fnavali hope i can get it done by tomorrow (today, it seems)06:13
*** yuanying_ has quit IRC06:13
sbalukoffWow!06:13
fnavalor at least something working06:13
sbalukoffI was actually expecting it to take at least another week. XD06:14
*** yuanying has joined #openstack-lbaas06:14
sbalukoffGiven how much you've had to sift through with the tempest code stuff.06:14
fnavalah could be if i run into more issues06:14
fnavalthere's a lot of layers there06:14
*** yuanying has quit IRC06:15
rm_worksometimes I kinda just wish all arguments were keyword arguments (or at least treated that way, since python lets you do that)06:18
rm_workmakes things super clear when you don't know the signature of a function and aren't in an IDE06:19
rm_worklol, just noticed "Copyright 2016 Blue Box, an IBM Company"06:23
rm_workis that actually legally in the name now? :P06:24
sbalukoffYep.06:28
rm_workdo they not realize that is REALLY WEIRD06:28
*** yuanying has joined #openstack-lbaas06:28
sbalukoffThough, I was just transferred to the Open Source Networking Group within IBM. So, I don't really know whether I'm officially a Blue Box employee anymore. Nor do I have any idea what the copyright line should look like. So... I'mma go with this until someone tells me differently.06:29
sbalukoffHaha! Yeah...06:29
sbalukoffI do recall like 2 weeks after the merger they told us to update all our e-mail signatures with that, as well as to mention that as the official company name whenever asked.06:30
*** reedip is now known as reedip_away06:30
sbalukoffAnd yes, as part of the acquisition, they did rename the company to that.06:30
rm_workI am sometimes tempted to put something like "Copyright 2016 Santa Claus"06:30
sbalukoff(I remember reading the paperwork and grinning when I saw.)06:30
sbalukoffEh...06:30
rm_workbecause I don't think these copyright notices actually matter, because we give up actual ownership when we contribute due to the licensing of the project as a whole and the contributor agreement, AFAIK06:31
rm_workbut IANAL <_<06:31
sbalukoffIt's all apache2 licensed code anyway, and we've got git history which will show what code was written by who. I think the copyright line doesn't matter that much.06:31
rm_work^^ yes06:31
sbalukoffYou should have seen the hand-wringing that happened when I asked the IBM laywers to look over the legal paperwork that transfers the Octavia trademark to the OpenStack Foundation. Seriously, like 2 weeks spent on the wording of one sentence (in this 5-sentence document) that didn't make any substantial difference.06:33
sbalukoffSpeaking of which.. I haven't heard back from the OSF people on this in a while... I should check in with them.06:33
rm_workand to think I was considering law school...06:33
sbalukoff(Though, given the speed of the USPTO, it'll probably transfer sometime in late 2036)06:33
sbalukoffHaha!06:34
sbalukoffrm_work: At least computers usually make sense and do what you instruct them to do. ;)06:34
rm_work(might explain why I am so pedantic)06:34
rm_workurg, you're doing de-duplication here which is great but it's STILL duplicated (across files) in some cases06:35
rm_workpain06:35
rm_worklike _get_db_listener and some of these test_and_set methods06:36
openstackgerritMerged openstack/octavia: Updates the failover flow for active/standby  https://review.openstack.org/25372406:36
rm_worknot sure a better way to do that tho06:36
*** crc32 has quit IRC06:36
rm_workyeah so many dupes of _test_lb_and_listener_statuses06:40
*** yamamoto_ has joined #openstack-lbaas06:44
openstackgerritKen'ichi Ohmichi proposed openstack/neutron-lbaas: Use tempest-lib's floating_ips_client  https://review.openstack.org/27515806:44
sbalukoffrm_work: Those methods are all slightly different...06:45
sbalukoffrm_work: They're specific to the controller they're in. :P06:45
rm_workurgh06:45
sbalukoffrm_work: But yes... this probably *could* be combined, in a helper function.06:45
sbalukoffMaybe a later optimization?06:45
rm_workyeah that's fine06:45
sbalukoffDang, I should get some dinner...06:46
sbalukoffHTTP_HEADER_REGEX = "^[a-zA-Z0-9\\!\\#\\$\\%\\&'\\*\\+\\-\\.\\^\\_\\`\\|\\~]+$"06:46
sbalukoffThe more you know, eh.06:46
sbalukoffProbably don't need that many back-slashes... let me try killing some...06:47
rm_workurgh i don't know if i can finish reviewing this tonight, my eyes are starting to blur06:48
rm_workmight have to finish tomorrow morning06:48
rm_workon the plus side, your review is obviating my need to take sleep aids <_<06:49
sbalukoffHAHA06:49
sbalukoffSleep well, man.06:49
rm_worki've got this devstack instance set up though and waiting to make a shared pool...06:49
rm_worki wonder if i can do it06:49
rm_workcurl http://localhost:9876/v1/loadbalancers06:54
rm_work^^ is returning nothing06:54
rm_workI feel like i remember this issue06:54
rm_workI need to send auth maybe?06:54
rm_workbut i feel like there isn't actually auth?06:54
sbalukoffThere isn't auth.06:54
rm_workso....06:54
sbalukoffIf you don't have any loadbalancers...06:55
rm_workthere is one06:55
sbalukoffHmmm...06:55
rm_workit shows up with "neutron lbaas-loadbalancer-list"06:55
rm_workbecause i created the LB itself with neutron06:55
sbalukoffWhat's your octavia DB look like?06:55
rm_workand I see the VMs octavia made06:55
sbalukoffYeah...06:55
sbalukoffYou should be getting something.06:55
rm_worktable shows one LB06:56
rm_workand that it has a damned underscore in load_balancer06:56
rm_work*shakes fist*06:56
rm_worki think the LB is not on my project06:56
rm_workneed to remember how to send that with my request06:57
*** prabampm has quit IRC06:57
rm_workgot it06:57
*** bharathm has quit IRC06:57
rm_workyeah had to pass X-Project-Id06:57
sbalukoffAah.06:58
rm_workis it supposed to list the pool in the loadbalancer json?06:59
rm_workor not06:59
rm_worki guess not07:00
*** bharathm has joined #openstack-lbaas07:03
*** links has quit IRC07:04
rm_workwhelp I think I might just +2 this07:05
rm_workI got through most of the code and didn't see anything hideous07:05
rm_workand i managed to get a pool to be on two listeners07:05
rm_worksooooo i'm pretty sold07:05
rm_workOH GOD IT'S IN MERGE CONFLICT07:07
rm_workT_T07:07
rm_worksbalukoff: whyyyyy07:07
sbalukoffDammit...07:07
rm_worki guess active/standby07:07
sbalukoffProbably the failover flow stuff.07:07
rm_workhopefully it's something dumb07:07
sbalukoffI'll see if I can fix it real quick.07:08
rm_workkk cause i'm good on this07:09
rm_workverified, added the same pool to two listeners07:09
rm_workseems legit07:09
rm_workhopefully I don't regret a +2 in the morning07:09
sbalukoffHeh!07:11
rm_worklol like EVERYTHING is in conflict now07:13
sbalukoffYeah, this is annoying.07:13
rm_workoh yeah it's all yours07:13
rm_workthat's why07:13
*** prabampm has joined #openstack-lbaas07:14
*** links has joined #openstack-lbaas07:14
sbalukoffOk, let me run tox locally on this real quick...07:15
rm_workI like how blogan just dropped this: https://review.openstack.org/#/c/274993/07:16
rm_workinteresting that this doesn't seem to be kicking off: https://review.openstack.org/#/c/259550/07:18
openstackgerritStephen Balukoff proposed openstack/octavia: Shared pools support  https://review.openstack.org/25636907:19
sbalukoffOk, hopefully that works.07:19
sbalukoffYay for rebasing against another huge patch. :P07:20
* rm_work +2s and crashes07:21
sbalukoffSleep well!07:21
*** kobis has joined #openstack-lbaas07:36
openstackgerritStephen Balukoff proposed openstack/octavia: Add L7 database structures  https://review.openstack.org/26543007:38
openstackgerritStephen Balukoff proposed openstack/octavia: Update repos for L7 policy / methods  https://review.openstack.org/26552907:46
*** paco20151113 has joined #openstack-lbaas08:09
*** prabampm1 has joined #openstack-lbaas08:11
*** prabampm has quit IRC08:12
*** doug-fish has joined #openstack-lbaas08:16
*** doug-fish has quit IRC08:21
openstackgerritBrandon Logan proposed openstack/octavia: Default api_handler to queue_producer  https://review.openstack.org/27499308:24
bloganthere rm_work08:24
*** doug-fish has joined #openstack-lbaas08:29
*** doug-fish has quit IRC08:34
sbalukoffHeh!08:41
sbalukoffblogan: Question for you if you're around, eh.08:42
sbalukoffRegarding the validations on l7 rules:  I'm not sure how to do a 'validation layer' per se--  there are a couple places where we may end up wanting to use some types of string validations and what not at different layers within the code base. (eg. at the API, and also at the repo layers)08:43
sbalukoffSo for now, I've created a bunch of helper functions in octavia/common/validations.py which do things like sanity checking on specific string formats, and even l7 rule internal consistency checking.08:44
sbalukoffThe idea here is that even at the repo layer you should never be able to instantiate an invalid l7rule object.08:45
sbalukoffIf you've got a better idea about how to do this, or where it should go in the code tree, I'm all ears, eh!08:45
sbalukoff(I'll try to get something committed on this branch tonight so you can see what I mean...)08:46
*** chlong has quit IRC08:52
*** armax has quit IRC09:01
*** openstackgerrit has quit IRC09:17
*** openstackgerrit has joined #openstack-lbaas09:18
*** prabampm has joined #openstack-lbaas09:25
*** prabampm1 has quit IRC09:28
openstackgerritStephen Balukoff proposed openstack/octavia: WIP: Update repos for L7 rules / validations  https://review.openstack.org/27664309:30
openstackgerritStephen Balukoff proposed openstack/octavia: WIP: Update repos for L7 rules / validations  https://review.openstack.org/27664309:32
openstackgerritStephen Balukoff proposed openstack/octavia: WIP: Update repos for L7 rules / validations  https://review.openstack.org/27664309:33
*** kobis has quit IRC09:56
*** yamamoto_ has quit IRC10:15
*** yamamoto_ has joined #openstack-lbaas10:17
*** kobis has joined #openstack-lbaas10:52
*** kobis has quit IRC11:00
*** yamamoto_ has quit IRC11:18
*** rtheis has joined #openstack-lbaas11:27
openstackgerritStephen Balukoff proposed openstack/octavia: WIP: Update repos for L7 rules / validations  https://review.openstack.org/27664311:43
*** doug-fish has joined #openstack-lbaas11:49
*** doug-fish has quit IRC12:02
*** yamamoto has joined #openstack-lbaas12:19
*** yamamoto has quit IRC12:25
*** paco20151113 has quit IRC12:27
*** rtheis has quit IRC12:41
*** links has quit IRC12:43
*** ducttape_ has joined #openstack-lbaas12:58
*** kobis has joined #openstack-lbaas13:01
*** prabampm1 has joined #openstack-lbaas13:05
*** prabampm has quit IRC13:06
*** ducttape_ has quit IRC13:16
*** rtheis has joined #openstack-lbaas13:28
*** localloop127 has joined #openstack-lbaas13:35
*** woodster_ has joined #openstack-lbaas13:42
*** doug-fish has joined #openstack-lbaas13:49
*** neelashah has joined #openstack-lbaas13:54
*** piet has joined #openstack-lbaas14:19
*** yamamoto has joined #openstack-lbaas14:24
*** doug-fish has quit IRC14:29
*** doug-fish has joined #openstack-lbaas14:30
*** Bjoern_ has joined #openstack-lbaas14:35
*** amotoki has quit IRC14:37
*** longstaff has joined #openstack-lbaas14:50
*** ducttape_ has joined #openstack-lbaas14:58
*** yamamoto has quit IRC15:01
*** doug-fis_ has joined #openstack-lbaas15:03
*** doug-fish has quit IRC15:04
*** doug-fish has joined #openstack-lbaas15:05
*** doug-fish has quit IRC15:05
*** doug-fish has joined #openstack-lbaas15:05
*** doug-fi__ has joined #openstack-lbaas15:07
*** doug-fis_ has quit IRC15:07
*** doug-fish has quit IRC15:10
*** doug-fi__ has quit IRC15:11
*** amotoki has joined #openstack-lbaas15:14
*** doug-fish has joined #openstack-lbaas15:22
*** yamamoto has joined #openstack-lbaas15:29
*** yamamoto has quit IRC15:31
*** yamamoto has joined #openstack-lbaas15:32
*** numan_ has quit IRC15:36
*** yamamoto has quit IRC15:37
*** amotoki has quit IRC15:40
*** doug-fish has quit IRC15:42
*** piet has quit IRC15:42
*** yamamoto has joined #openstack-lbaas15:42
*** piet has joined #openstack-lbaas15:43
*** yamamoto has quit IRC15:44
*** doug-fish has joined #openstack-lbaas15:53
*** amotoki has joined #openstack-lbaas15:54
*** mgarza has joined #openstack-lbaas15:55
*** allan_h has joined #openstack-lbaas15:56
pai15Hello, nubie here. Trying to get lbaas v2 working with devstack. Once I create loadbalancer, it stays in PENDINZG_CREATE state. Hence can not configure listener and all. Any pointers?15:58
*** logan_ has joined #openstack-lbaas16:00
*** logan_ is now known as Guest7865416:01
*** logan- has quit IRC16:01
*** Guest78654 is now known as logan-16:02
rm_workknew it! https://review.openstack.org/#/c/276711/16:03
rm_workjohnsom: ^^16:03
*** alhu_ has joined #openstack-lbaas16:11
*** allan_h has quit IRC16:14
rm_worksbalukoff / xgerman / blogan / johnsom do you remember when we are deprecating haproxyNamespaceDriver for v2?16:21
*** armax has joined #openstack-lbaas16:21
xgermanI don’t think we are deprecating it16:22
rm_worki thought we were16:22
rm_workbut maybe not?16:22
xgermanthe plan was to eventually have it in it’s own repo like a 3rd party driver for interested parties to mainteain16:22
xgermanem_work also https://review.openstack.org/27655416:23
xgermanrm_work16:23
rm_workyeah i know16:23
rm_workgoing to try to figure out exactly what's up16:24
xgermanyeah, in  our tests it connected to expired certs — there should be an Octavia bug for that16:24
xgermanfor the namespace blogan/dougwig should have the latests16:24
*** prometheanfire has joined #openstack-lbaas16:25
*** ducttape_ has quit IRC16:25
rm_workok yeah i'll take a look16:26
*** johnsom_ has joined #openstack-lbaas16:28
*** alhu_ has quit IRC16:32
*** _cjones_ has joined #openstack-lbaas16:34
*** piet has quit IRC16:36
*** _cjones_ has quit IRC16:37
*** _cjones_ has joined #openstack-lbaas16:37
*** piet has joined #openstack-lbaas16:37
xgermanrm_work thanks for digging that up16:41
*** yamamoto has joined #openstack-lbaas16:45
*** ducttape_ has joined #openstack-lbaas16:49
*** yamamoto has quit IRC16:51
*** fnaval has quit IRC16:57
*** fnaval has joined #openstack-lbaas16:57
*** prabampm1 has quit IRC17:01
bloganrm_work: don't think we are17:06
bloganrm_work: don't think it was ever planned17:06
rm_workkk17:08
rm_worki heard talk about that I thought since we were saying we didn't want to maintain two drivers if octavia was up and running as the default17:08
rm_workbut i guess it's ok?17:08
blogani think there might be some people that may want to maintain it17:14
*** Aish has joined #openstack-lbaas17:21
johnsom_Which driver?  IRC history missing17:26
rm_workhaproxynamespace17:26
johnsom_Oh, yeah, probably should hang around17:27
rm_workprometheanfire: ^^ there you go17:27
rm_workprometheanfire: nothing being deprecated that would force the use of octavia for lbaas yet17:28
rm_workor even in the near future17:28
prometheanfirethanks17:29
prometheanfireI'll still hang around, want to know about the management network changes for octavia, using admin level stuff for it is notsogood :D17:30
*** alhu_ has joined #openstack-lbaas17:34
*** jwarendt has joined #openstack-lbaas17:39
*** fnaval has quit IRC17:42
*** amotoki has quit IRC17:45
*** johnsom_ has quit IRC17:56
*** numan_ has joined #openstack-lbaas18:03
*** madhu_ak_ has joined #openstack-lbaas18:03
rm_workxgerman: can't seem to repro the expired certificate being accepted18:04
rm_workmaybe it's not a real issue and we are good without the thing18:05
xgermanmmh, well, it happened a long time ago so might be fixed18:05
rm_workBREACH attack is worrying, but18:05
*** fnaval has joined #openstack-lbaas18:06
rm_workwe may not fit the full criteria for it18:10
rm_work Be served from a server that uses HTTP-level compression18:10
rm_work  Reflect user-input in HTTP response bodies18:10
rm_work  Reflect a secret (such as a CSRF token) in HTTP response bodies18:10
rm_worksince these are all server-side things, I'm not sure the *client* can possibly be related?18:11
*** Aish has quit IRC18:14
rm_workxgerman: what actually runs the flask app for the agent on the amp?18:17
xgermanwerkzeug18:17
rm_workah I found it18:17
rm_workcmd/agent.py18:17
xgermanyep18:18
*** ducttape_ has quit IRC18:21
*** piet has quit IRC18:26
*** bana_k has joined #openstack-lbaas18:26
*** Aish has joined #openstack-lbaas18:27
*** _cjones_ has quit IRC18:27
*** drjones has joined #openstack-lbaas18:27
dougwigwe'll likely eject it if/when we eject vendors, like neutron.18:27
*** longstaff has quit IRC18:28
openstackgerritMerged openstack/octavia: Default api_handler to queue_producer  https://review.openstack.org/27499318:29
sbalukoffI've got to commute into the office (big Blue Box party thingy happening today), but in the meantime:  johnsom: https://review.openstack.org/#/c/256369/   One little conflict made me have to rebase in such a way that I lost your +2 on that. I'd appreciate it if you could have another look. (or blogan or dougwig or xgerman or blallau)18:31
johnsomYep, watching it18:32
sbalukoffThanks!18:32
johnsomCongrats BTW18:32
sbalukoffOk, BBIAB18:32
sbalukoffAnd yes-- thanks on that front, too!18:32
xgermansbalukoff you had a party for your core promotion scheduled? How confident?18:32
rm_work:P18:32
sbalukoffxgerman: I like to plan ahead. XD18:32
xgermanCongrats as well18:32
johnsomHahaha, wow, say it isn't so18:32
sbalukoffNo-- this is the annual company party.18:32
johnsomWe'll have to revoke immediately18:32
johnsomgrin18:32
sbalukoffWe usually skip having it in December because holidays/vacation.18:33
sbalukoffXD18:33
doug-fishannual company party?! who do you work for?!18:33
sbalukoffdoug-fish:  Blue Box, an IBM Company18:33
doug-fish:-)18:33
rm_worklol18:33
rm_workit is seriously impossible to NOT laugh when anyone says that full company name18:33
rm_workT_T18:34
doug-fishwhen you work for IBM, every day is a party.18:34
sbalukoffAt least, nobody's told me differently yet. Techincally I was transferred to the Open Source Networking Group within IBM about a week ago. But the paperwork hasn't gone through on that yet, so I'm going to pretend I still belong at that party. XD18:34
johnsomEspecially when the person asking works for the same company18:34
sbalukoffHAHA!18:34
doug-fishKind of a mix between a search party and a rescue party. But still.18:34
*** ducttape_ has joined #openstack-lbaas18:34
sbalukoffdoug-fish: HAHAHA!18:34
sbalukoffAt least it isn't a Donner party.18:35
sbalukoffOk, I've gotta run. Be online a little later, eh!18:35
doug-fish*waves*18:35
*** longstaff has joined #openstack-lbaas18:38
*** sbalukoff has quit IRC18:39
*** longstaff has quit IRC18:42
*** longstaff has joined #openstack-lbaas18:45
*** longstaff has quit IRC18:45
*** longstaff has joined #openstack-lbaas18:46
*** localloop127 has quit IRC18:53
*** localloop127 has joined #openstack-lbaas18:56
*** localloop127 has quit IRC19:18
*** madhu_ak_ is now known as madhu_ak19:19
openstackgerritBrandon Logan proposed openstack/octavia: Remove out-dated README file  https://review.openstack.org/27688019:19
*** ducttape_ has quit IRC19:21
*** localloop127 has joined #openstack-lbaas19:21
*** ducttape_ has joined #openstack-lbaas19:36
*** sbalukoff has joined #openstack-lbaas19:47
bloganjohnsom: couple of questions19:47
johnsomOk19:47
bloganjohnsom: how to enable the debug messages on teh scenario tests?19:47
blogantyring to track down one of them intermittent issues19:47
johnsomset it in the gate hook19:47
bloganand 2) is it the agent log viewable on the job logs?19:48
blogans/is it/is/19:48
johnsomno, we don't harvest the agent log from the amps19:48
blogandamn, i think we should find a way to do that bc it'll make these much easier to figure out19:48
bloganhttp://logs.openstack.org/37/275337/5/check/gate-neutron-lbaasv2-dsvm-scenario/688c19e/console.html#_2016-02-04_10_22_46_22119:49
bloganlike this one, basically is saying that after a pool got deleted, the lb was still load balancing to the members in that pool19:49
sbalukoffMan, this place is a mad-house today.19:49
bloganbut the pool delete logs all the way to the o-cw say that the delete happened correctly19:50
bloganso at that point, insepecting the haproxy config would be beneficial19:50
blogansbalukoff: got your message early this morning or last night, can't remember about the validation layer19:50
sbalukoffblogan: Yay!19:50
johnsomHmm, timing maybe.19:50
blogansbalukoff: sounded like what you said was fine to me, but could you go over it again?19:51
bloganjohnsom: perhaps timing, the agent is synchronous right?19:51
sbalukoffblogan: The delete problem you are seeing is probably related to this bug: https://bugs.launchpad.net/octavia/+bug/154212019:51
openstackLaunchpad bug 1542120 in octavia "Changes made via API do not immediately get pushed to amphorae" [Critical,New] - Assigned to Michael Johnson (johnsom)19:51
blogansbalukoff: ah19:52
sbalukoffblogan: Certainly! I put validations for l7rules here: https://review.openstack.org/#/c/276643/4/octavia/common/validations.py19:52
sbalukoff(It's easier to show you what I'm doing now that I have something committed.)19:53
johnsomCould be, but they are intermittent so, odd19:53
sbalukoffjohnsom: Yeah, the intermittent part of that problem throws me off as well. :/19:53
sbalukoffer... intermittent nature19:53
johnsomYeah, I just got a similar hit:19:53
johnsomhttp://logs.openstack.org/50/259550/7/check/gate-neutron-lbaasv2-dsvm-scenario/d48b77d/console.html#_2016-02-05_19_50_43_25519:54
bloganseems to be happening more frequent, even without the session persistence bug19:54
bloganthen again the session persistence failures could be the same issue19:54
sbalukoffblogan: regarding the validation "layer"--  I wasn't sure how to do it as a layer, since I could see this begin done at the API level as well as the repo level. Plus, these kinds of validations are context-independent, so I threw them in a new file in octavia/common19:54
johnsomYeah, it is definitely showing up more all of a sudden.19:55
blogansbalukoff: yeah makes sense, works for me, just really wanted the validation logic separate from the control logic, and i figured we'd need them to be reusable19:56
sbalukoffblogan: Ok! Sounds good. Glad I wasn't totally off-base in doing it that way. :)19:56
johnsomThe hosts seem to be slower now too, which makes me think it is a timing thing.  Look at how long it takes for the initial API connect.19:57
bloganwe'll also need to add in some validation pieces that go and validate against neutron and nova, which will be quite hairy bc the api will have to load a network and/or compute driver for that19:57
sbalukoffblogan: Awesome...  well, tempest tests are coming, right?19:58
blogandamn rackspace VMs!19:58
sbalukoffHaha!19:58
blogansbalukoff: yes, should be, right fnaval? :)19:58
johnsomActually I think mine was a bluebox host19:58
sbalukoffjohnsom: Sorry!19:59
fnavalchecking for context.. 1 sec19:59
sbalukofffnaval: We love you!19:59
johnsomNo worries, probably still faster than the old HP hosts19:59
fnavalyea working on it right now20:00
fnavaltempest tests that is.  thanks sbalukoff!20:00
*** doug-fis_ has joined #openstack-lbaas20:05
johnsomhttps://bugs.launchpad.net/neutron/+bug/154246720:06
openstackLaunchpad bug 1542467 in neutron "LBAAS Intermittent gate failure TestHealthMonitorBasic.test_health_monitor_basic" [Undecided,New]20:06
johnsomI logged that one.20:06
johnsomwill take a look after grabbing lunch20:06
blogantest_listener_basic is happening as wel20:07
bloganprobably every scenario test lol20:07
bloganat some point20:07
johnsomI got this error: socket.error: [Errno 104] Connection reset by peer20:08
johnsomWhich seems different, but if you think related you can add on20:08
*** doug-fish has quit IRC20:08
bloganyeah i'm willing to bet its all related20:08
bloganand could be the stale data being pushed up thing20:08
johnsomMmmm, not so sure.  This error is after the test, on shutdown20:09
*** alhu_ has quit IRC20:09
johnsomMaybe20:09
*** doug-fis_ has quit IRC20:10
openstackgerritBrandon Logan proposed openstack/neutron-lbaas: Enable debug logging in octavia scenario tests  https://review.openstack.org/27690020:14
bloganah shit20:15
blogandont' want to use the noop drivers20:15
blogani mean ah poop20:15
sbalukoffDammit, did you swear?20:16
blogansomeone must have changed their nick to mine20:16
sbalukoffClearly.20:16
openstackgerritBrandon Logan proposed openstack/neutron-lbaas: Enable debug logging in octavia scenario tests  https://review.openstack.org/27690020:19
blogani don't know if id id that right but we shall see by  the tests20:19
openstackgerritmin wang proposed openstack/octavia: Implements: blueprint anti-affinity server group  https://review.openstack.org/27234420:20
*** localloop127 has quit IRC20:20
sbalukoffLook good to me. :/20:21
*** numan_ has quit IRC20:21
sbalukoffOh handy-- we'll not need to merge that for you to get your results. :)  Or are you planning on merging that anyway so we can see failures on other tests?20:22
bloganmerging anyway, every other job has the debug logs20:23
bloganand they're quite useful when troubleshooting20:23
blogannext thing we should do is find out some way to get the agent logs uploaded, which i'm not even sure if its possible, well possible in the sense that we don't hack it up20:24
rm_workagent logs from the amp?20:26
rm_workI don't even...20:26
rm_workwe'd need to use paramiko to grab them or something before we delete each LB? or ... add a log retrieval endpoint to the agent?20:26
rm_workand have the deletion flow grab the logs before shutting down the amp?20:26
rm_workthat might work20:27
rm_workis there actually a shutdown flow that lets the driver define shutdown tasks on deletion?20:27
bloganyeah something like that20:27
blogani don't think so20:27
rm_workbecause that would work -- let the driver run some tasks it wants before spinning down the amps, llke a cleanup function20:29
rm_workand that could grab the logs and store them locally20:29
rm_workgetting the agent logs is not a small effort, I would guess :/20:30
xgermanscp ?20:32
sbalukoffxgerman: That's crazy talk, man.20:32
rm_workyeah that is what i meant with paramiko20:32
rm_worki mean we need to do it in code before deletion of the LB20:32
xgermanthen dump them into the worker logs20:32
rm_worki would keep them separate probably20:32
xgermanand only if verbose/debug/ssh-key enabled20:32
rm_workie, paramiko for scp20:33
xgermanyep20:35
rm_workblogan: i think you missed a thing on https://review.openstack.org/#/c/276900/220:35
sbalukoffrm_work: You're right. Need append, not overwrite.20:39
rm_workwere we going to try to merge https://review.openstack.org/#/c/256369/27 today?20:40
* sbalukoff looks hopeful20:40
sbalukoffIt would totally make my Friday. :)20:40
bloganthats why you bash experts look at it :)20:41
openstackgerritBrandon Logan proposed openstack/neutron-lbaas: Enable debug logging in octavia scenario tests  https://review.openstack.org/27690020:41
bloganwell if assume scp is an option, then we're assuming the amp can be ssh'ed into which may not be true20:43
openstackgerritStephen Balukoff proposed openstack/octavia: Update repos for L7 policy / methods  https://review.openstack.org/26552920:43
ptoohillI added option to rest driver stuff to disable ssh as it was still open20:43
ptoohillThat's been in there for a bit though, I think20:43
ptoohillso assuming deployer deploys with right options20:44
*** rtheis has quit IRC20:44
ptoohillunless scp is only option20:45
sbalukoffI wouldn't be against adding an amphora API endpoint to grab logs, eh.20:46
ptoohillI like that idea20:47
rm_workyeah this ^^20:47
sbalukoffThat seems like it could be useful in a production environment as well.20:47
ptoohillall the options are optional20:48
sbalukoffHaha20:49
sbalukoffOk... It sounds like the big Blue Box meeting, followed by a party, will happen in about 1.5 hours. I'll probably be AFK for the rest of the day following that. So if you need anything from me (especially if it involves a particular shared pools patch ;), please let me know before then!20:50
*** localloop127 has joined #openstack-lbaas20:51
ptoohillHave fun!20:54
ptoohillIs gate still a day long?20:54
ptoohilldoesnt look too bad20:54
rm_workptoohill: any patches you really want people to review that are ready to go?20:58
rm_workptoohill: haven't heard you really selling any of your CRs recently :)20:58
ptoohillI've been kinda of waiting for other things20:59
ptoohillMy stuff wont be used for a little while, but do need it in soon. I was hoping to get other important things in21:00
ptoohillWe need this: https://review.openstack.org/#/c/275558/21:00
johnsomThanks ptoohill for the selfless act...  I have also be trying to prioritize stuff.21:00
ptoohillOctavia queue is <trump voice> HUGE21:01
ptoohilljohnsom: :)21:01
ptoohillLots going on21:01
rm_workok, taking a look at that one21:01
ptoohillOh wait, he's a second place loser now, now I really aint voting for him21:01
ptoohill:P21:01
johnsomYep, which is awesome!  We are making progress though21:01
ptoohillIndeed21:02
rm_workman, i do not understand jinja as well as i thought apparently21:06
rm_workwhat is the hyphen for in {%-21:06
ptoohillto remove white space21:11
ptoohillI could set it globally, but we dont want that, we need certain white space21:11
ptoohilljinja can be quirky21:11
ptoohillrm_work:21:11
ptoohillhttp://jinja.pocoo.org/docs/dev/templates/ Under 'Whitespace Control'21:13
rm_workah ok21:13
ptoohillThere's some limitations regarding formatting that makes things interesting sometimes21:13
ptoohillthis being a case because of yaml, and the filter doesnt work. Maybe at some point I could write a custom filter, but..21:13
rm_workonly one nit21:15
rm_workneed to run it tho21:15
ptoohillNeed to set use_upstart = False and it works with devstack21:16
ptoohiller21:16
ptoohillwait21:17
ptoohillcrossing lines21:17
ptoohilluser_data_config_drive = True21:17
ptoohillin controller_worker21:17
ptoohillFeel free to rip it apart, ill get it updated21:18
ptoohillDOH, yes i need to disable that ><21:20
openstackgerritPhillip Toohill proposed openstack/octavia: Allow user-data on amphora creation  https://review.openstack.org/27555821:28
*** bana_k has quit IRC21:29
*** localloop127 has quit IRC21:43
*** allan_h has joined #openstack-lbaas21:52
*** bana_k has joined #openstack-lbaas22:12
*** bank_ has joined #openstack-lbaas22:12
openstackgerritStephen Balukoff proposed openstack/octavia: Update repos for L7 policy / methods  https://review.openstack.org/26552922:12
*** longstaff has quit IRC22:19
*** bank_ has quit IRC22:27
*** bana_k has quit IRC22:27
openstackgerritStephen Balukoff proposed openstack/octavia: Update repos for L7 rules / validations  https://review.openstack.org/27664322:28
rm_worksbalukoff: i thought you were at a party22:29
*** bana_k has joined #openstack-lbaas22:40
*** bank_ has joined #openstack-lbaas22:41
*** ducttape_ has quit IRC22:41
fnavalhow do I make changes to this doc: http://docs.openstack.org/developer/octavia/api/octaviaapi.html#list-members22:43
fnavalor propose changes22:44
johnsomMin knows.  Let me see if I can ping her22:45
fnavalcool. thanks johnsom. i think it might be syncing down a particular docs tree?22:46
*** kobis has quit IRC22:46
*** minwang2 has joined #openstack-lbaas22:46
johnsomYeah, she knows that voodoo22:47
fnavalcool thanks johnsom22:47
minwang2fnaval, can you repeat the question again22:47
fnavalhi minwang2: I want to make a proposed change to the octavia docs located at http://docs.openstack.org/developer/octavia/api/octaviaapi.html22:47
fnavalwhat do I have to checkout for that?22:48
minwang2https://github.com/openstack/octavia/blob/master/doc/source/api/octaviaapi.rst22:48
fnavalah ok thanks!22:49
fnavali thought it lived in a separate repo.22:49
minwang2nope, you can edit it directly in that file22:49
openstackgerritFranklin Naval proposed openstack/octavia: Octavia:  API docs Create Member url fix  https://review.openstack.org/27694422:56
fnavalcool thanks minwang2 - please take a look at that if it's a valid fix22:56
minwang2fnaval, np, let me take a look22:57
fnavaljust a 1 line change22:57
fnavalthanks22:57
minwang2btw, is there a ticket linked with this patch, why do we need to change it22:57
fnavalno ticket. where do i file that22:58
fnavaloh i got a +2 from brandon.  i think it might not need it then?22:59
minwang2the code itself is pretty simple, but it would be a good habbit to link a ticket or write some description in the commit msg, in that way, for people who has no background about this patch, they can easier to review :)23:02
openstackgerritFranklin Naval proposed openstack/octavia: Octavia:  API docs Create Member url fix  https://review.openstack.org/27694423:02
fnavalah ok.  in launch pad?23:03
fnavalbah, i just realized that the url gets fixed in the shared pools patch but anyway, this fixes that url before that lands23:03
openstackgerritMerged openstack/octavia: Remove out-dated README file  https://review.openstack.org/27688023:05
openstackgerritFranklin Naval proposed openstack/octavia: Octavia:  API docs Create Member url fix  https://review.openstack.org/27694423:07
openstackgerritMichael Johnson proposed openstack/octavia: Fixes Octavia flows that are not reverting  https://review.openstack.org/25955023:07
fnavalminwang2: thanks, yes that makes sense. plus, it's good to have some metrics on opened bugs/fixed bugs23:07
minwang2i just gave +1, i like this way better23:08
openstackgerritFranklin Naval proposed openstack/octavia: Octavia:  API docs Create Member url fix  https://review.openstack.org/27694423:08
johnsomfnaval FYI: http://docs.openstack.org/infra/manual/developers.html#working-on-bugs23:09
johnsomIt should be Closes-Bug: #1234523:09
openstackbug 12345 in isdnutils (Ubuntu) "isdn does not work, fritz avm (pnp?)" [Medium,Fix released] https://launchpad.net/bugs/12345 - Assigned to Matthias Klose (doko)23:09
johnsomOpps, I guess my example # worked23:10
fnavalthanks johnsom: reading and bookmarking it23:10
johnsomI always have to refer back to this page to get the formatting right23:10
*** nmagnezi_ has joined #openstack-lbaas23:10
fnavalthe documentation on that same page shows to add the line "Fixes Bug: 1234"  http://docs.openstack.org/infra/manual/developers.html#peer-review23:10
openstackbug 1234 in Launchpad itself "Gina is an unmaintainable mess of command line options, environment variables and shell scripts" [Medium,Fix released] https://launchpad.net/bugs/1234 - Assigned to Daniel Henrique Debonzi (debonzi)23:10
fnavalmight need to fix that too23:10
openstackgerritFranklin Naval proposed openstack/octavia: Octavia:  API docs Create Member url fix  https://review.openstack.org/27694423:11
johnsomfnaval I'm not trying to nit-pick you, but I want you to know how to do it right so you get stackalytics credit.23:16
johnsomCloses-Bug: #154252823:17
openstackbug 1542528 in octavia "Fix URL for Create Member" [Undecided,Fix committed] https://launchpad.net/bugs/1542528 - Assigned to Franklin Naval (franknaval)23:17
fnavaljohnsom: ah I didn't think of it as that actually23:17
johnsomNot a big deal, but could be nice if you guys look at those #s23:17
fnavalno worries on the nit-pick23:18
fnavalim totally (re)learning this23:18
fnavallol23:18
johnsomWhen it's right the commit message will pop into the bug itself23:18
fnavalah23:18
fnavalforgot to add the '3"23:19
fnaval'#'23:19
johnsomYeah23:19
openstackgerritFranklin Naval proposed openstack/octavia: Octavia:  API docs Create Member url fix  https://review.openstack.org/27694423:20
johnsomThe docs are so un-helpful on that23:20
fnavalyeah, I'm trying to find where that doc repo lives so i can propose changes23:20
johnsombrandon's patch here is a good example: https://review.openstack.org/#/c/260291/23:21
fnavalah ok good one23:23
fnavalso, the OpenStack Infra user will automatically update the bug in the comments with any related changes if the commit message is correct?23:24
*** nmagnezi_ has quit IRC23:25
johnsomYep23:29
fnavalah ok thanks johnsom23:29
johnsomIt tracks the lifecycle of the bug and patch and will update the bug as it progresses23:29
*** mgarza has quit IRC23:37
*** neelashah has quit IRC23:38
*** chlong has joined #openstack-lbaas23:38
openstackgerritMerged openstack/neutron-lbaas: Implementing EventStreamer reciever for octavia  https://review.openstack.org/24147423:44
*** ducttape_ has joined #openstack-lbaas23:58
*** Bjoern_ has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!