Monday, 2016-02-08

*** amotoki has joined #openstack-lbaas00:16
*** allan_h has quit IRC00:22
*** chlong has joined #openstack-lbaas00:26
*** allan_h has joined #openstack-lbaas00:28
*** chlong has quit IRC01:03
*** yamamoto_ has joined #openstack-lbaas01:48
*** chlong has joined #openstack-lbaas02:06
*** yamamoto_ has quit IRC02:07
*** yamamoto_ has joined #openstack-lbaas02:08
*** yamamoto_ has quit IRC02:10
*** yamamoto_ has joined #openstack-lbaas02:10
*** yamamoto_ has quit IRC02:10
*** yamamoto_ has joined #openstack-lbaas02:58
*** reedip_away is now known as reedip03:03
*** amotoki has quit IRC04:13
*** links has joined #openstack-lbaas04:14
*** amotoki has joined #openstack-lbaas04:27
*** jidar_ has joined #openstack-lbaas04:38
*** jidar has quit IRC04:38
*** jidar_ is now known as jidar04:38
*** amotoki has quit IRC04:39
*** amotoki has joined #openstack-lbaas04:40
*** ducttape_ has joined #openstack-lbaas04:43
*** neelashah has joined #openstack-lbaas04:47
*** neelashah has quit IRC04:55
*** amotoki has quit IRC04:56
*** amotoki has joined #openstack-lbaas05:11
*** ducttape_ has quit IRC05:18
*** prabampm has joined #openstack-lbaas05:32
*** amotoki_ has joined #openstack-lbaas06:08
*** amotoki has quit IRC06:11
openstackgerritAishwarya Thangappa proposed openstack/octavia: Adding "region and endpoint_type" parameters to barbican_acl.py  https://review.openstack.org/27286106:33
*** numans has joined #openstack-lbaas06:34
*** allan_h has quit IRC06:39
*** chlong has quit IRC07:03
*** nmagnezi has joined #openstack-lbaas07:49
openstackgerritMerged openstack/neutron-lbaas: Enable debug logging in octavia scenario tests  https://review.openstack.org/27690008:07
*** rcernin has joined #openstack-lbaas08:09
*** chlong has joined #openstack-lbaas08:48
reedipblogan: ping?08:54
*** amotoki_ has quit IRC08:55
*** jschwarz has joined #openstack-lbaas09:00
*** openstackgerrit has quit IRC09:02
*** openstackgerrit has joined #openstack-lbaas09:03
*** amotoki has joined #openstack-lbaas09:14
*** reedip is now known as outofmemory09:24
*** yuanying is now known as yuanying_influen09:25
*** yuanying_influen is now known as yuanying_flu09:25
*** amotoki has quit IRC09:32
*** Alex_Stef has joined #openstack-lbaas09:46
*** ebagdasa has quit IRC10:10
*** jschwarz is now known as jschwarz|lunch11:11
*** rcernin has quit IRC11:19
*** rcernin has joined #openstack-lbaas11:22
*** rcernin has quit IRC11:25
*** rcernin has joined #openstack-lbaas11:26
*** yamamoto_ has quit IRC11:27
*** prabampm has quit IRC11:37
*** prabampm has joined #openstack-lbaas11:39
*** prabampm1 has joined #openstack-lbaas11:51
*** prabampm has quit IRC11:51
*** numans has quit IRC12:03
*** amotoki has joined #openstack-lbaas12:13
*** yamamoto has joined #openstack-lbaas12:21
*** rtheis has joined #openstack-lbaas12:30
*** amotoki has quit IRC12:35
*** yamamoto has quit IRC12:42
*** mdavidson has joined #openstack-lbaas12:43
*** yamamoto has joined #openstack-lbaas12:43
*** yamamoto has quit IRC12:48
*** amotoki has joined #openstack-lbaas12:58
*** yamamoto has joined #openstack-lbaas12:58
*** ducttape_ has joined #openstack-lbaas13:13
*** amotoki has quit IRC13:15
*** links has quit IRC13:20
*** localloop127 has joined #openstack-lbaas13:27
*** yamamoto has quit IRC13:27
*** ducttape_ has quit IRC13:29
*** Alex_Stef has quit IRC13:35
*** yamamoto has joined #openstack-lbaas13:58
*** neelashah has joined #openstack-lbaas14:00
*** yamamoto has quit IRC14:28
*** Aish has joined #openstack-lbaas14:49
*** ducttape_ has joined #openstack-lbaas14:51
*** jschwarz|lunch is now known as jschwarz14:58
*** TrevorV has joined #openstack-lbaas15:00
*** nmagnezi has quit IRC15:02
*** piet has joined #openstack-lbaas15:03
openstackgerritTrevor Vardeman proposed openstack/octavia: Amphora create will add a 'fake' heartbeat on creation  https://review.openstack.org/27151215:10
*** ajmiller has joined #openstack-lbaas15:14
*** Bjoern has joined #openstack-lbaas15:17
*** amotoki has joined #openstack-lbaas15:24
*** rcernin has quit IRC15:42
*** doug-fish has joined #openstack-lbaas15:42
*** amotoki has quit IRC15:46
*** links has joined #openstack-lbaas15:49
*** links has quit IRC15:57
*** ajmiller_ has joined #openstack-lbaas16:05
*** ajmiller has quit IRC16:09
*** _ducttape_ has joined #openstack-lbaas16:19
*** armax has joined #openstack-lbaas16:20
*** ducttape_ has quit IRC16:21
*** sbalukoff has quit IRC16:24
*** numans has joined #openstack-lbaas16:32
*** allan_h has joined #openstack-lbaas16:35
openstackgerritTrevor Vardeman proposed openstack/octavia: Amphora create will add a 'fake' heartbeat on creation  https://review.openstack.org/27151216:37
*** _ducttape_ has quit IRC16:40
*** ducttape_ has joined #openstack-lbaas16:41
*** armax has quit IRC16:46
*** jschwarz has quit IRC16:47
*** fnaval has joined #openstack-lbaas17:03
*** madhu_ak has joined #openstack-lbaas17:07
*** Aish has quit IRC17:18
*** yamamoto has joined #openstack-lbaas17:21
*** yamamoto has quit IRC17:22
openstackgerritAdam Harwell proposed openstack/octavia: Revert "Fix for ssl context wrapper spam", add warning filter  https://review.openstack.org/27750617:33
*** Aish has joined #openstack-lbaas17:34
*** yamamoto has joined #openstack-lbaas17:43
*** armax has joined #openstack-lbaas17:56
xgermanrm_work I had to rewrite that endpoint filter logic three times to appease blogan17:58
rm_worklol17:59
rm_workpoor xgerman17:59
xgermanyeah, we need to streamline simple changes...18:00
xgermanguess time to fork octavia so I can add an endpoint type18:01
rm_workxgerman: is "publicURL" the correct endpoint_type for devstack?18:03
xgermanyes18:03
rm_workk18:03
rm_workthen it is wrong in https://review.openstack.org/#/c/272861/18:03
rm_workwanted to check before I left a -1 there18:03
xgermanand it’s the default in the keystone client if you don’t specify one18:04
rm_workYeah I thought yours was correct18:04
xgermanin our cloud we need to use “internal” but people also have “admin"18:04
rm_workbut before I told Aish to fix it I wanted to make sure18:04
xgermanyou can also say “public"18:04
rm_workah18:04
xgermanboth work18:04
rm_workwell18:04
rm_workI'd like it to be consistent18:04
rm_workso, sorry Aish T_T18:05
xgermanyeah, agreed on that one18:05
rm_workconfusing for deployers elsewise18:05
xgermanif they will even find those settings since they are now per service and not for all18:05
rm_workheh18:06
rm_workwell, it matters18:06
rm_workthough for barbican it technically doesn't matter, until BarbicanCertGenerator works18:06
rm_workbecause the client doesn't use those values for retrieval18:06
xgermanit does not?18:06
rm_workit uses the HREF for the container18:06
rm_workignoring the service catalog values18:06
xgermanI thought Barbican can’t do certs but if it could we could use the generator18:06
rm_workright18:07
rm_workbut ... I honestly don't know if I ever see that happening18:07
rm_workI think Anchor will be the way forward18:07
xgermanyep18:07
*** armax has quit IRC18:07
rm_workI am tempted to remove the dead code there18:07
rm_workdelete the LocalCertManager and BarbicanCertGenerator classes18:08
*** doug-fish has quit IRC18:08
xgermanok18:08
rm_workmight do that now18:08
xgermanyeah, you have my +218:08
*** doug-fish has joined #openstack-lbaas18:09
*** armax has joined #openstack-lbaas18:12
*** doug-fish has quit IRC18:13
rm_workwho else runs on OSX?18:18
rm_workI'm getting a couple of tox failures on py27/py34 that look like false positives and only happen to me on my OSX box18:18
*** yamamoto has quit IRC18:26
xgermanI filed a bug18:26
xgermanrm_work18:26
rm_workfor whichwhat18:27
xgermanfor the OS X error18:28
*** armax has quit IRC18:31
xgermancan’t find it somebody must have closed it18:32
rm_workah18:34
rm_workoctavia.tests.functional.amphorae.backend.agent.api_server.test_server.ServerTestCase.test_get_interface and octavia.tests.functional.amphorae.backend.agent.api_server.test_server_sysvinit.ServerTestCase.test_get_interface18:34
rm_workuhh18:34
johnsomI don't remember an OS/X bugs in the octavia bug list.  Maybe it was filed in a different project?18:36
*** prometheanfire has left #openstack-lbaas18:41
rm_workumm18:41
rm_workaddress_format = netifaces.AF_INET618:41
rm_workon OSX this is "30"18:41
xgermanyeah, I thought if filed18:43
rm_workon linux this is 1018:43
rm_workwtf18:43
johnsomThe constants are different????18:43
rm_workyes18:43
johnsomEvil18:43
rm_workso need to use the constants in the test18:43
rm_workit's fixable18:43
rm_worki'll do it now18:43
xgermancool18:43
xgermanok, I guess that interface patch attracts all the nit-pickers in the universe18:43
johnsomThat is as bad as debian not taking eth1.cfg in interfaces.d but ubuntu requires .cfg18:44
*** armax has joined #openstack-lbaas18:44
openstackgerritAdam Harwell proposed openstack/octavia: Remove dead code around certificate handling  https://review.openstack.org/27753218:46
xgermanrm_work +218:46
rm_workwtf is "17"?18:47
rm_workfor type18:47
rm_workah, AF_ROUTE18:47
*** piet has quit IRC18:47
rm_workok, easy peasyf ix18:48
TrevorVrm_work https://review.openstack.org/#/c/27151218:48
rm_workkk18:48
TrevorVxgerman johnsom https://review.openstack.org/#/c/27151218:48
rm_workwill look momentarily18:48
TrevorVPreez!18:48
rm_workthat session persistence bug is really ... persistent18:48
rm_workis session persistence actually *broken* or does that intermittent bug just hit 90% of the time <_<18:48
TrevorVI noticed that.18:48
TrevorVI have zero idea what's even wrong... ha ha18:49
johnsomNot sure.  I'm also tracking the ovh hosts seem to be failing out our scenario test where the rax hosts aren't failing.  May be related, may not18:50
johnsomWell, TrevorV's patch was on rax and failed too.18:51
rm_workfiled https://bugs.launchpad.net/octavia/+bug/1543248 xgerman18:51
openstackLaunchpad bug 1543248 in octavia "OSX has tox test failures due to netifaces inconsistency" [Undecided,New]18:51
TrevorVOh god please someone tackle that bug, please...18:52
openstackgerritAdam Harwell proposed openstack/octavia: Use netifaces constants correctly in testing  https://review.openstack.org/27753318:52
rm_work^^ resolved18:52
johnsomHmmm, there is still a problem with you patch TrevorV and a failover bug, but I'm really puzzled by that, as it was fixed in the patch that merged18:53
TrevorVjohnsom wait what?18:53
*** bank_ has joined #openstack-lbaas18:53
*** bana_k has joined #openstack-lbaas18:53
rm_workas soon as https://review.openstack.org/#/c/277533/ gets through Zuul I will be bugging you guys for a quick merge :P18:53
xgerman+218:54
johnsomTrevorV it still tried to failover an amp.  It failed because of some failover flow issue that puzzles me (looking at that now).18:54
TrevorVOh.  Really?  Maybe a different topology issue?18:55
johnsomIt claims there is no "amphora" object in the flow for markallocatedindb, which is strange as the steps before that get the amphora18:56
rm_workO_o18:57
TrevorVSo the flow is complaining, not a task specifically failing?18:58
johnsomright18:58
TrevorVOh.  I find that more interesting.  Weird.  Would it throw that error if "amphora=None"?18:59
johnsomWhat the heck, the line above it, in the flow also requires 'amphora' but that didn't errorr19:00
TrevorVHah... are you sure you don't have different code running than what you're looking at?19:01
rm_workxgerman / johnsom can you take a look at my comments on https://review.openstack.org/#/c/271476/8 and tell me if i'm wrong?19:03
johnsomYep, I'm looking at master vs. the error you got19:03
*** neelashah has quit IRC19:03
*** _ducttape_ has joined #openstack-lbaas19:08
*** ducttape_ has quit IRC19:11
*** doug-fish has joined #openstack-lbaas19:16
*** sbalukoff has joined #openstack-lbaas19:16
*** fnaval has quit IRC19:16
*** doug-fish has quit IRC19:21
*** _ducttape_ has quit IRC19:22
*** neelashah has joined #openstack-lbaas19:22
*** ducttape_ has joined #openstack-lbaas19:22
*** doug-fish has joined #openstack-lbaas19:22
*** armax has quit IRC19:26
openstackgerritBrandon Logan proposed openstack/octavia: Fix / and /v1 API uris  https://review.openstack.org/27754419:34
*** armax has joined #openstack-lbaas19:34
*** armax has quit IRC19:36
rm_workblogan / xgerman / johnsom / anyone: have you see this failure before? http://logs.openstack.org/06/277506/1/check/gate-neutron-lbaasv2-dsvm-scenario/33327b1/console.html#_2016-02-08_19_26_57_62219:37
bloganrm_work: yes19:37
bloganrm_work: one of them intermittent ones we're not sure about right now19:37
bloganrm_work: could you put a launchpad bug in for this one? we should start keeping track of this one as well, we already have one for the session persistence one and i think there's another one that doesn't have a bug for it that causes the listener test to break19:38
blogani have a feeling they're all related though19:39
bloganrm_work: ^ the patch i just pushed up, it was all staticmethods fault!19:39
rm_worklol19:41
rm_workstaticmethods don't kill people, people kill people19:41
*** piet has joined #openstack-lbaas19:42
*** fnaval has joined #openstack-lbaas19:42
rm_worklooks super ugly but: https://bugs.launchpad.net/octavia/+bug/154326419:48
openstackLaunchpad bug 1543264 in octavia "intermittent scenario failure in test_health_monitor_basic" [Undecided,New]19:48
rm_workblogan: ^^19:48
sbalukoffWoot! My Monday's going great-- got my first -1 of the week in! ;)19:49
rm_workheh19:50
sbalukoffblogan: Seriously though, good catch on that. I never would have guessed. (Also, I do seriously think we should change it to "v1", then I'm good with it.)19:50
rm_workYeah switched to -1 :P19:50
rm_workSince it isn't JUST me being pedantic19:50
sbalukoffIt's both of us being pedantic.19:51
rm_workYes, I'm ok with being in a pedantic mob19:51
rm_workbut it's lonely being a solitary pedant19:51
*** neelashah1 has joined #openstack-lbaas19:51
sbalukoffBBIAB, gotta grab some lunch before this stupid audit today. (Thankfully the last of these I'll have to do in a long while.)19:51
*** neelashah has quit IRC19:52
bloganrm_work: thats good enough, be sure to recheck bug 1543264 when you need a recheck bc of this19:52
openstackbug 1543264 in octavia "intermittent scenario failure in test_health_monitor_basic" [Undecided,New] https://launchpad.net/bugs/154326419:52
rm_workk19:52
rm_workI mean, technically NEVER need a recheck for scenario failures <_<19:52
blogansbalukoff: johnsom hunted me down for it, he threatened me if i didnt help figure it out19:52
rm_workblogan: https://review.openstack.org/#/c/277506/19:53
blogansbalukoff: and what do you mean change it to "v1"? ohhh its V!19:53
bloganV119:53
bloganno its lowercase, at least the uri is19:53
rm_work<_<19:53
blogansbalukoff: you'll have to clear it up for me when youg et back19:54
rm_workblogan: lowercase the V1 to v119:54
rm_workthat's it :P19:54
bloganbut it is lowercase in the uri19:54
rm_workis .lower() being run on it?19:54
bloganoh he put comments19:55
blogannvm19:55
bloganthats just the body19:55
bloganthat gets returned19:55
rm_worklol19:55
bloganit was just a "ill put something here"19:55
rm_workis it even used19:55
bloganha no19:55
rm_worklol19:55
rm_workI *wasn't* going to -119:55
rm_workas I said19:55
rm_workbut19:55
bloganthat calls has been broken for probably 6 mos to a year, 500, and no one has really noticed19:55
rm_workmob mentality19:55
rm_workI noticed!19:55
rm_workand complained19:55
bloganprove it!19:56
rm_workand didn't bother to fix it because who cares <_<19:56
* blogan watches rm_work spend all day going through the eavesdop logs for the past year19:56
rm_worklol19:56
rm_workI was about to start grepping19:56
rm_workdecided I don't care enough19:56
bloganperhaps if someone in particualr challenged you to it you would19:57
rm_work<_<19:57
rm_workI wonder if I can get Korean food delivered....19:57
*** neelashah1 has quit IRC19:58
bloganair mail?19:58
rm_workit's the Lunar New Year today, and thus is https://en.wikipedia.org/wiki/Korean_New_Year :)19:59
johnsomFYI, I also put this one in: https://bugs.launchpad.net/neutron/+bug/154246719:59
openstackLaunchpad bug 1542467 in neutron "LBAAS Intermittent gate failure TestHealthMonitorBasic.test_health_monitor_basic" [Critical,Opinion]19:59
blogani figured the moon would have the same new year as earth, the thing it orbits19:59
rm_workwas going to get some Tteokguk20:00
bloganjohnsom: ah so thats a duplicate of what rm_work put in20:00
*** numans has quit IRC20:00
rm_workah k so it did exist20:00
bloganor the other way around20:00
rm_workah20:00
blogansurprised launchpad didn't tell you there was one that looked similar20:00
bloganor did you ignore that suggestin?20:00
rm_workwell it searches Octavia20:00
rm_worknot Neutron20:00
rm_workalso no, didn't see anything20:00
bloganah20:00
bloganyeah didnt see it was in neutron and octavia respectively20:01
rm_workwill dupe mine20:01
rm_workdone20:01
johnsomWell, the gate is technically a neutron-lbaas gate/test so I put it over there20:01
rm_workyeah20:02
rm_worki agree it makes sense20:02
*** woodster_ has joined #openstack-lbaas20:02
rm_worknow it will show up in both places because mine is in octavia and marked as a dupe of yours20:02
rm_workblogan: comments on https://review.openstack.org/#/c/271476/ ?20:03
*** neelashah has joined #openstack-lbaas20:03
johnsomI didn't threaten anyone BTW.  Just asked for help as it was not obvious to me what was wrong.20:05
bloganrm_work: good catch on that, though i think the original intention of the get_client methods were nto to eb a factory but to be a singleton, so perhaps that needs to be changed20:05
johnsomIt became relevant for API endpoint monitoring20:05
bloganjohnsom: no witnesses, i felt my life was in danger20:05
TrevorVjohnsom20:05
bloganjohnsom: cornered me in a dark corner of irc20:05
rm_workblogan: you think a different patch is fine?20:06
TrevorVjohnsom did you set the new environment variable on my patch?20:06
openstackgerritAishwarya Thangappa proposed openstack/octavia: Adding "region and endpoint_type" parameters to barbican_acl.py  https://review.openstack.org/27286120:06
rm_workI am just worried it'll be forgotten20:06
bloganrm_work: yeah basically bc it does what is needed for the original intention of the clients.py code20:06
rm_workT_T20:06
rm_workso if the first thing to initialize it happens to want an admin client...20:07
rm_workEVERYTHING gets an admin client20:07
bloganyep20:07
rm_workgood game20:07
blogani know20:07
bloganso bug report?20:07
rm_workGuess so20:07
johnsomblogan Nobody is buying it.  Especially when I'm how many miles away????  2,25920:07
bloganjohnsom: haven't you seen the movie hackers? they were able to sink a oil tanker from that distance!20:08
johnsomSigh20:08
blogani'm sure you could hack into my chair and make it roll down the escalator20:08
johnsomI'll just blind you by making your IRC client blink20:09
johnsomYeah....20:09
* blogan keeps a history of that threat20:09
johnsomTrevorV Environment variable???  I was looking at the gate job logs20:10
TrevorVjohnsom lemme PM you20:10
johnsomblogan Want me to appease the spoil sports and change the text to v1?  At least for now, or are you feeling ambitious and going to do the endpoint thing?20:12
rm_workhttps://bugs.launchpad.net/octavia/+bug/154326920:12
openstackLaunchpad bug 1543269 in octavia "Nova and Neutron client factories can return incorrect clients on second call" [Undecided,New]20:12
*** armax has joined #openstack-lbaas20:13
bloganjohnsom: i could claim that V1 -> v1 is outside the scope of that review :)20:15
bloganbc it is20:15
bloganis it worth that battle? nope20:15
johnsomYeah, I was mostly thinking about a way to clear the -1's20:15
johnsomgrin20:15
rm_workheh20:16
rm_workyeah20:16
openstackgerritBrandon Logan proposed openstack/octavia: Fix / and /v1 API uris  https://review.openstack.org/27754420:16
sbalukoffDelicous scope...20:16
bloganjohnsom, rm_work: the endpoint thing is going to require some discussion20:16
johnsomAgreed20:17
bloganmain question, is it abd to instantiate a new client every time? or does it need to be smart enough to keep track of the options as rm_work suggested20:17
blogans/abd/bad20:17
rm_workit isn't that hard to track them20:17
rm_workI could probably do that and submit it20:17
*** armax has quit IRC20:17
rm_workin like an hour (90% of which would be testing)20:18
bloganbut is it worth that effort if its not even a big deal to instantiate a new one everytime?20:18
*** armax has joined #openstack-lbaas20:18
bloganjohnsom: is that what you meant by the endpoint thing? i assume dyou were talking about what came out of that review20:19
rm_workuhh, well20:20
rm_workit has to *hit keystone* every time, right?20:20
rm_workto get the URL20:20
rm_workso that's a request to keystone every time we use a client20:20
rm_workwhen we could just as easily *not*20:20
rm_workunless that stuff is cached at a different level/20:21
bloganthats what the session does though, if you instantiate the client with the session then it shouldn't hit keystone everytime20:21
bloganbut honeslty i dont know that for fact, just what i've assumed20:21
rm_workhmm20:21
rm_workwe can try just removing cacheing20:21
rm_workcan't have cache invalidation if you don't cache20:21
rm_workproblem solved20:21
johnsomblogan I was talking about your comment when I asked about the V1->v1 thing20:21
rm_workwhelp http://www.mysanantonio.com/news/local/article/San-Antonio-fire-crews-respond-to-fire-at-parking-6815580.php20:22
johnsomSo, since it's +A'd, nevermind20:22
bloganjohnsom: ahh so different, my best guess right now is just to return a body with the /loadbalancers uri20:22
bloganlol20:22
bloganah the rivercenter parking lot!20:22
blogangarage20:22
rm_workheh20:23
rm_work"Aly Zamora said she was driving past the mall when the fire began and initially thought it may have been a terrorist attack and was worried."20:23
rm_work>_>20:23
sbalukoff...20:23
sbalukoffWow20:23
sbalukoffWell, everyone knows that 99% of fires in the U.S. are started by terrorists.20:23
rm_workI mean, they may as well just clasify Arson as Terrorism, right? >_>20:24
blogananytime i see a dead animal in the road i immediately think terrorist20:24
rm_work(not that this was arson)20:24
johnsomI'm sure they already have...20:24
bloganwell they can already just say you're a terrorist and kill you without any courts involved20:25
bloganor some secret court is involved20:26
rm_workis this an episode of Twilight Zone or something where it pans out and they ask "and who are the REAL terrorists?"20:27
bloganand then they show ... aliens?20:27
*** armax has quit IRC20:30
rm_workI was thinking "the government", but, could be aliens :P20:30
sbalukoffWhy not both?20:32
rm_workhttps://www.youtube.com/watch?v=jEUJfEBNgmM20:33
johnsomI gave a pass at the e-mail to the openstack-dev octavia list from Major Hayden about security stuff.20:35
johnsomFeel free to join in the fun20:35
rm_workblogan: https://review.openstack.org/#/c/277533/20:35
rm_workoh I guess it's waiting on zuul20:36
rm_work<_<20:36
rm_workso close to done20:36
rm_workalso: https://review.openstack.org/#/c/277506/20:37
rm_worktried to keep the warning squelch as local in scope as possible20:38
bloganthat looks like something i wrote :)20:38
bloganthough i'm not sure20:39
rm_workok now https://review.openstack.org/#/c/277533/ passed20:40
*** ducttape_ has quit IRC20:41
*** madhu_ak has quit IRC20:43
bloganscenario tests failed20:43
blogansession persistence20:43
johnsomYeah.  I looked at that and +A'd it anyway20:45
openstackgerritGerman Eichberger proposed openstack/octavia: Adds a parameter to specify endpoint type  https://review.openstack.org/27147620:48
xgermanhope it pleases the crowd...20:51
bloganxgerman: is this the first time you've done a patch in openstack?20:52
bloganxgerman: you seem surprised by the many opinions :)20:52
xgermanwell, I am not doing pools… I was thinking maybe 4 reviews… but 8 or 9 is getting silly20:53
rm_workyeah I really hope we aren't expected to recheck for the nonvoting scenario tests every time they fail due to the several outstanding intermittent bugs :P20:53
bloganrm_work: i look to see what the failure is, but if it sone of those intermitten failures i don't request to wait for it ot pass20:54
rm_workspeaking of pools/L7, isn't this one ready? https://review.openstack.org/#/c/265430/20:54
johnsomYeah, I dig into the failures because that gate has caught a number of valid issues too20:55
rm_workyep20:55
*** ducttape_ has joined #openstack-lbaas20:56
openstackgerritDoug Wiegley proposed openstack/neutron-lbaas: WIP - do not merge, negative CI test  https://review.openstack.org/27755720:56
*** jwarendt has joined #openstack-lbaas20:58
*** madhu_ak has joined #openstack-lbaas20:58
*** armax has joined #openstack-lbaas20:59
*** crc32 has joined #openstack-lbaas21:04
openstackgerritMerged openstack/octavia: Remove dead code around certificate handling  https://review.openstack.org/27753221:07
openstackgerritMerged openstack/octavia: Use netifaces constants correctly in testing  https://review.openstack.org/27753321:12
*** bank_ has quit IRC21:13
*** bana_k has quit IRC21:13
*** Bjoern has quit IRC21:20
openstackgerritMerged openstack/neutron-lbaas: Updated from global requirements  https://review.openstack.org/27723921:22
openstackgerritAdam Harwell proposed openstack/octavia: Do not use __builtin__ in python3  https://review.openstack.org/26251621:24
rm_worktaking that over since no one wants to own it21:24
johnsomrm_work thanks!21:28
rm_workI *love* having time to focus on octavia21:28
rm_workbeen cooped up with internal stuff for a while21:28
johnsomyeah, I am trying to balance21:29
rm_workand I am OCD about having too many open bugs / CRs so I am trying to get some of this older stuff cleaned up...21:29
rm_workTrevorV: did you manage to replicate johnsom's problem on https://review.openstack.org/#/c/271512/21:30
TrevorVNot so far, but scenario tests are still running21:31
TrevorVIt got past the test that WAS failing though...21:31
*** rooney_ has joined #openstack-lbaas21:32
TrevorVjohnsom I don't understand... the tests are passing on my devstack instance in our Rackspace cloud.21:34
johnsomI don't understand that gate failure either.  Maybe we should just recheck?21:34
TrevorVMaybe... Lemme finish my scenario test run first just in case21:35
johnsomI really worry about the timing dimension to this patch21:35
TrevorVYou mean the amount of time to wait before a failover occurs, or with "how fast we merge it"?21:35
TrevorVI'm lost21:35
rm_workyeah, that is why suggested a long timeout value for default21:36
rm_workwhich SHOULD be plenty, though maybe we need even longer?21:36
rm_workare there actually failover tests that run though in scenario?21:36
johnsomTimeouts21:36
johnsomNo, the scenario should not trigger a failover.  That is really an Octavia concept.21:37
rm_workyeah, that is what I thought21:37
johnsomIt would go in Octavia scenario tests21:37
rm_workso where are you talking about?21:37
rm_workwhat failed?21:37
johnsomWith this patch, it triggered a failover in the scenario gate21:37
TrevorVThe problem isn't that the test is triggering a failover, its that a failover is happening during the testing21:37
TrevorVThe distinction is the "unexpectedness" of the failover.21:38
TrevorVIt SHOULDN'T be failing over anything during the testing.21:38
rm_workah rofl21:42
TrevorVAlright, so I guess a failover did occur on my scenario test stuffs21:44
TrevorVLemme check logs21:44
TrevorVWait, nope.21:44
TrevorVIdk why21:44
TrevorVI don't understand... its only failing during the cleanups...21:46
rm_workblogan / johnsom: is http://logs.openstack.org/06/277506/1/check/gate-neutron-lbaasv2-dsvm-scenario/856b8fd/console.html#_2016-02-08_21_36_46_298 also expected?21:46
rm_worktest_listener_basic21:46
bloganrm_work: yep21:47
bloganwell not expected but a known failure21:47
johnsomExpected?  No.  Happening, sadly yes21:47
rm_workthat doesn't look like the session persistence one21:47
rm_workor the HM one21:47
rm_work>_>21:47
bloganits not21:47
rm_workhow many of these are there21:48
rm_workjesus21:48
blogan321:48
rm_work>_<21:48
bloganmaybe 421:48
rm_workok I am going to start looking at one or more of these21:48
johnsomYeah, there are three or four of these similar failures21:48
bloganthey're definitely happening much more frequently21:48
rm_workwho is working on any of them?21:48
rm_workanyone?21:48
johnsom+121:48
blogansbalukoff did for the session persistence one, but couldn't reproduce21:48
johnsomI am assigned one, but my time is hit and miss21:48
johnsomTake it if you want it21:49
rm_workhmm21:49
rm_workwhich one do we see the most21:49
johnsomsession persistence21:49
johnsomIMHO21:49
TrevorVI just... I don't even right now...21:50
TrevorVjohnsom it failed in the cleanup of the test_session_persistence test, right after deleting all the security groups...21:51
TrevorVJust like in the log in the job, just an entirely different test...21:51
openstackgerritMerged openstack/octavia: Fixes Octavia Amphora RHEL based images  https://review.openstack.org/27290521:52
openstackgerritAishwarya Thangappa proposed openstack/octavia: Adding "region and endpoint_type" parameters to barbican_acl.py  https://review.openstack.org/27286121:53
*** bana_k has joined #openstack-lbaas21:59
*** bank_ has joined #openstack-lbaas22:00
*** rtheis has quit IRC22:07
rm_workam I just BAD at searching these bugs? can't seem to find the session persistence failure bug22:12
*** localloop127 has quit IRC22:15
bloganrm_work: neutron22:17
bloganrm_work: check it out in neutron's launchpad22:17
rm_workyeah i am in neutron's launchpad22:18
rm_worksearching for "lbaas session" and nothing22:18
rm_workwell, nothing relevant22:19
rm_workhttps://bugs.launchpad.net/neutron?field.searchtext=lbaas+session&search=Search&field.status%3Alist=NEW&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.assignee=&field.bug_reporter=&field.omit_dupes=on&field.has_patch=&field.has_no_package=22:19
rm_workthat's one of like 10 queries i tried22:19
rm_worklbaas gate; lbaas intermittent22:19
bloganrm_work: if i find it in 30 seconds will murder you22:19
rm_work"lbaas gate" is funny because it doesn't even bring up the bug I have open in the other window22:19
bloganhttps://bugs.launchpad.net/octavia/+bug/154167522:20
openstackLaunchpad bug 1541675 in octavia "intermittent session persistence scenario test failure" [Critical,New]22:20
bloganrm_work: ^22:20
bloganit was in octavia22:20
rm_workwhich is titled "LBAAS Intermittent gate failure TestHealthMonitorBasic.test_health_monitor_basic"22:20
rm_workrofl22:20
rm_workawesome22:20
bloganguess we'll have to standardize on where bugs should be reported22:20
rm_worki think search is broken anyway22:20
* blogan murders rm_work22:20
rm_work"LBAAS Intermittent gate failure TestHealthMonitorBasic.test_health_monitor_basic" isn't found by "lbaas gate" or "lbaas intermittent"22:20
blogani couldn't be a liar22:20
bloganyeah that search sucks22:21
rm_workAH it's because it's an "opinion" what22:22
rm_workand that's filtered off by default in search22:22
rm_work>_<22:22
bloganwhat?22:22
rm_workso I am missing a ton of bugs possibly when i search22:22
rm_workhttps://bugs.launchpad.net/neutron/+bugs?field.searchtext=lbaas+gate&orderby=-importance&search=Search&field.status%3Alist=NEW&field.status%3Alist=OPINION&field.status%3Alist=CONFIRMED&field.status%3Alist=TRIAGED&field.status%3Alist=INPROGRESS&field.status%3Alist=FIXCOMMITTED&field.status%3Alist=INCOMPLETE_WITH_RESPONSE&field.status%3Alist=INCOMPLETE_WITHOUT_RESPONSE&assignee_option=any&field.assignee=&field.bug_reporter=&field.bug_c22:22
rm_workommenter=&field.subscriber=&field.structural_subscriber=&field.tag=&field.tags_combinator=ANY&field.has_cve.used=&field.omit_dupes.used=&field.omit_dupes=on&field.affects_me.used=&field.has_patch.used=&field.has_branches.used=&field.has_branches=on&field.has_no_branches.used=&field.has_no_branches=on&field.has_blueprints.used=&field.has_blueprints=on&field.has_no_blueprints.used=&field.has_no_blueprints=on22:22
rm_workoh god22:22
blogannew bugs are opinions?22:22
rm_workhttps://bugs.launchpad.net/neutron/+bug/154246722:22
openstackLaunchpad bug 1542467 in neutron "LBAAS Intermittent gate failure TestHealthMonitorBasic.test_health_monitor_basic" [Critical,Opinion]22:22
rm_workthis one is22:22
rm_workmarked that way by Nate22:22
rm_work... can i file that as a bug? :P22:23
bloganahh22:23
bloganNate is the bug deputy22:23
blogani asked him to join our channel22:24
johnsomYeah, that was really strange.  Not sure why that was mark opinion22:24
*** bana_k has left #openstack-lbaas22:25
openstackgerritMerged openstack/octavia: Fix / and /v1 API uris  https://review.openstack.org/27754422:25
xgermanthanks blogan!!22:28
rm_workthis is an interesting bug22:28
rm_workfirst of the app cookie session persistence requests goes to server1, everything else to server2 -- is it possible haproxy takes a moment after being configured to use cookie based persistence to actually reload? like, our agent returns OK before it's actually reloaded the config fully?22:31
johnsomrm_work Oh!  That is probably the bug stephen reported and I assigned to myself.22:32
johnsomDarn!22:32
johnsomSherif added https://github.com/openstack/octavia/blob/master/octavia/controller/worker/tasks/amphora_driver_tasks.py#L5822:32
johnsomWhich is wrong and broken22:32
sbalukoffYep, that sounds like the bug I was seeing...22:32
rm_workhmm22:33
johnsomIt causes the first listener update to have the wrong data22:33
rm_workso this is an ACTUAL bug22:33
rm_workyeah22:33
rm_workthat would make sense I think for what i'm seeing?22:33
rm_workit's one update behind the whole time?22:33
johnsomThe strange thing is that has been there since last summer, so why is it hitting us now???22:33
johnsomYeah, one update behind22:34
johnsomhttps://bugs.launchpad.net/octavia/+bug/154212022:34
openstackLaunchpad bug 1542120 in octavia "Changes made via API do not immediately get pushed to amphorae" [Critical,New] - Assigned to Michael Johnson (johnsom)22:34
rm_workLOL22:34
rm_workthat thing >_<22:35
rm_workdidn't sbalukoff essentially have the fix already22:35
sbalukoffIt's a work-around.22:35
rm_workjust was going to move it out to another patchset?22:35
rm_workah22:35
johnsomFixed a different way.22:35
sbalukoffThe work-around is detailed in my bug report.22:35
johnsomSo, the fix is to remove that database line from the code22:35
sbalukoffThough johnsom is right in that it's probably not the *right* solution to the problem.22:35
johnsomProblem is I'm not sure if the issue Sherif saw is still an issue or not.  Either way it should be fixed a better way22:36
sbalukoffjohnsom: And then revisit the old bug and try to figure out why we needed it in the first place?22:36
johnsomYep22:36
johnsomI suspect it is a sqlalchemy model invalidation bug22:36
sbalukoffFun!22:37
johnsommodel/cache whatever they call it in sqlalchemy22:37
rm_workAH, i see, yeah22:37
*** neelashah has quit IRC22:37
rm_workalright22:37
sbalukoffWell, if the bug is still there (and unlikely to go away, since SQLAlchemy is... obtuse), there are other work-arounds we can try. it'd be nice not to have to use a work-around, though.22:38
rm_workwhat's more surprising is that this ever DOESN'T happen22:38
sbalukoffrm_work: You and me both.22:38
sbalukoffrm_work: That tells me the current tempest tests are... well, not thorough.22:38
johnsomYeah, me three22:38
rm_work<_<22:38
sbalukoffwhich is one of the reason's I'm looking forward to fnaval's patch.22:39
sbalukoff:)22:39
rm_workyeah, very tired of cringing every time I +A anything22:39
rm_workbecause the tests passing often means nothing because either it isn't tested or the test is wrong or imcomplete <_<22:39
rm_work*incomplete22:39
*** doug-fish has quit IRC22:41
johnsomSo, if you want I can push a patch that takes that line out.  We just need to put some test cycles behind it to find any other issues that pop out.22:42
johnsomThat line *is* wrong22:42
rm_workyeah22:42
rm_workI'd say we go that route22:42
rm_workI am decently comfortable looking into sqlalchemy issues22:43
rm_workmoreso than futzing with flows22:43
johnsomYeah, I know those flows worked prior to that change22:43
*** doug-fish has joined #openstack-lbaas22:44
bloganeven when octavia gets its own tests, we're still going to be under teh 2 hour restrictin22:44
bloganso at that point should the db have the correct information and sqlalchemy has just cached it?22:45
bloganjohnsom, rm_work, sbalukoff: ^^?22:45
johnsomblogan You have confused me22:45
rm_workI will need to see what exactly is happening22:46
rm_workif it still is22:46
bloganjohnsom: in the line you linked above, in the amphora_driver_tasks22:46
*** doug-fis_ has joined #openstack-lbaas22:46
bloganyall mentioned it could be a sqlalchemy issue22:46
rm_workblogan: no at that point the DB has the WRONG info22:46
rm_workit doesn't have the info from the update that is being made22:46
rm_workso22:46
rm_work>_>22:46
bloganrm_work: the actual db has the wrong informatin or sqlalchemy is returning hte wrong information as if its being cached?22:46
rm_workthe db actually has the wrong info22:47
rm_workit still has the old info22:47
rm_workDB is updated from the request in the next flow22:47
bloganah okay bc we went the hacky route of not updating the db until it got finished so we could roll back to a known state22:47
rm_workyes22:47
rm_workjohnsom: it's literally just ... removing that line, eh? you want to propose it?22:48
johnsomWorking on it right onw22:48
*** doug-fish has quit IRC22:48
bloganif that bug that is in the comments comes up, we can probably do a session.refresh22:49
rm_workyeah i have a bunch of thoughts too22:49
rm_workbut it all depends on what actually is the problem or if it even has a problem22:49
bloganyou're the problem22:50
bloganand the solution!22:50
rm_workyour FACE is the problem22:50
bloganbut mostly the problem22:50
*** crc32 has quit IRC22:50
bloganmy FACADE is the problem22:50
bloganbehind the facade is a face that is the solution22:50
*** doug-fis_ has quit IRC22:50
rm_workhttps://www.youtube.com/watch?v=x9kJ6IJnBz022:50
*** crc32 has joined #openstack-lbaas22:51
johnsomOk, something strange is going on.  When I git commit on a clean checked out directory I get a commit message and "modifed" files that aren't modified in the repo22:54
johnsomdirectory22:54
rm_work<_<22:54
johnsomI have never seen this before22:55
rm_workuhh22:55
rm_workwhat steps did you do that i can repro22:55
rm_workand what is the output of git status22:55
johnsomOh, nevermind, I know what my issue is.  Doing 12 things at once like an idiot22:56
rm_workheh22:56
rm_workyou mean doing 12 things at once like someone who is a natural multitasker? :P22:56
rm_worki honestly bet all of these intermittent issues are related to this same thing22:57
rm_workold data22:57
*** doug-fish has joined #openstack-lbaas22:59
*** allan_h has quit IRC22:59
openstackgerritMichael Johnson proposed openstack/octavia: Reverts a workaround that is impacting amp updates  https://review.openstack.org/27760223:00
rm_workalright, will look forward to seeing results from that -- and will start building my own env with that patch now23:01
johnsomI think there is a delete member patch that probably needs to be reverted and the model update code applied as well.23:02
*** TrevorV has quit IRC23:03
openstackgerritMerged openstack/octavia: Adds a parameter to specify endpoint type  https://review.openstack.org/27147623:03
rm_workjohnsom: ah yeah i think i remember that23:03
*** doug-fish has quit IRC23:03
rm_workthey did the flow-flip thing, right?23:03
rm_workduring the midcycle to fix that bug23:04
johnsomYeah23:04
*** doug-fish has joined #openstack-lbaas23:04
rm_worki'll try to figure this out23:04
johnsomI have it open, if this works out I will submit that too23:04
rm_workand then we can try to include that in this patch too23:04
rm_workyeah23:04
johnsomIt's this one: https://bugs.launchpad.net/octavia/+bug/151451023:04
openstackLaunchpad bug 1514510 in octavia "Delete of last member does not remove it from haproxy config" [Critical,Fix released] - Assigned to Eran Raichstein (eranra)23:04
johnsomIt looks like the model update code was missing originally23:04
rm_workyep, i remember helping look at that23:04
rm_workshould just be the flow order reversal, since it uses listenerupdate as well23:05
johnsomYeah, but the model update code will need to be added to delete the member from the model23:06
rm_workah23:06
johnsomLike this: https://github.com/openstack/octavia/blob/master/octavia/controller/worker/flows/pool_flows.py#L4523:07
rm_workthat's already there, isn't it?23:07
rm_workline 56 here: https://review.openstack.org/#/c/260605/5/octavia/controller/worker/flows/member_flows.py23:08
johnsomOh, it is there23:08
johnsomOk23:08
rm_work... in like half an hour I'll have an environment :P23:09
rm_workblogan: https://review.openstack.org/#/c/277506/23:10
rm_worksbalukoff: ^^ you're a +2 now :)23:11
sbalukoffHeh!23:11
sbalukoff(Sorry, I'm being pulled in 5 directions at once at the moment.)23:12
rm_worklol no worries if you're busy just ignore me23:12
*** doug-fish has quit IRC23:14
*** doug-fish has joined #openstack-lbaas23:14
*** doug-fish has quit IRC23:15
*** doug-fish has joined #openstack-lbaas23:15
sbalukoffrm_work: Just had a look and +A'ed. Just out of curiosity though: was there a problem with pyasn1?23:17
rm_workit isn't used23:17
rm_workit's just required as part of the security thing23:17
rm_workIF we use ndg-httpsclient23:17
rm_workit's ... that whole thing is super weird with the auto-injecting23:17
rm_workI'm hoping we can get back on the global-reqs checking soon <_<23:19
sbalukoffAah.23:19
rm_workI think that was the only one besides the taskflow version23:19
xgermantask flow got +A I thoiught23:19
rm_workjust now23:19
rm_workwaiting for it to merge still23:19
rm_workhttps://review.openstack.org/#/c/276549/23:19
xgermanyep23:20
rm_workwell, hours ago23:20
rm_workbut still in the pipe23:20
xgermanthe gozers of this world23:20
johnsomYeah, I poked them in the oslo meeting this morning23:20
rm_worklol23:20
rm_workthanks for the +As guys :P23:20
*** ajmiller_ has quit IRC23:20
bloganrm_work: you asked for it23:20
rm_workjohnsom: still time for you to +A that23:21
johnsomHa!23:21
johnsomI would do no such vote stacking23:21
johnsomI have to be a role model23:22
johnsomgrin23:22
rm_worklol23:22
rm_worksuch integrity23:22
* blogan starts the smear compaign against johnsom23:23
blogansuperPAC23:23
johnsomI will laugh if it fails in the merge gate23:23
bloganwhat skeletons does johnsom have in his closet23:23
*** doug-fish has quit IRC23:23
johnsomPlenty I'm sure23:23
bloganmy closet is full of pre-skeletons...23:24
bloganand my crawlspaces23:25
johnsomMmmm, I bet it smells lovely23:25
*** doug-fish has joined #openstack-lbaas23:26
*** doug-fish has quit IRC23:27
*** doug-fish has joined #openstack-lbaas23:28
*** ajmiller has joined #openstack-lbaas23:29
rm_workwaaat https://review.openstack.org/#/c/276549/23:35
*** piet has quit IRC23:35
johnsomGreat, everyone's gates are broken....23:36
sbalukoffSmells like Monday, eh!23:37
*** doug-fish has quit IRC23:38
rm_workguess a recheck should fix it according to clarkb, giving that a shot23:38
*** piet has joined #openstack-lbaas23:38
*** doug-fish has joined #openstack-lbaas23:40
*** piet has quit IRC23:43
*** doug-fish has quit IRC23:46
johnsomsbalukoff Let me help you make it even more of a Monday:23:50
johnsomhttps://bugs.launchpad.net/octavia/+bug/154335623:50
openstackLaunchpad bug 1543356 in octavia "Errors found for octavia api----get listener stats" [Undecided,New]23:50
johnsomhttps://bugs.launchpad.net/octavia/+bug/154336623:50
openstackLaunchpad bug 1543366 in octavia "Errors found for octavia api----create health monitor on deprecated endpoint Edit" [Undecided,New]23:50
johnsomhttps://bugs.launchpad.net/octavia/+bug/154336423:50
openstackLaunchpad bug 1543364 in octavia "Errors found for octavia api----create health monitor on new endpoint" [Undecided,New]23:50
johnsomRelated to shared pools patch testing23:50
*** doug-fish has joined #openstack-lbaas23:50
*** doug-fish has quit IRC23:50
*** doug-fish has joined #openstack-lbaas23:50
*** ducttape_ has quit IRC23:53
openstackgerritMerged openstack/octavia: Revert "Fix for ssl context wrapper spam", add warning filter  https://review.openstack.org/27750623:53
*** doug-fish has quit IRC23:54
*** rooney_ has quit IRC23:56
rm_workok i need to be off for a while23:58
rm_workcatch most of you tomorrow and maybe some of you tonight :P23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!