Friday, 2018-03-23

« Thursday, 2018-03-22 Index Saturday, 2018-03-24 »
openstackgerritmelissaml proposed openstack/octavia master: fix a typo in documentation  https://review.openstack.org/55554000:26
*** harlowja_ has quit IRC01:13
openstackgerritOpenStack Proposal Bot proposed openstack/neutron-lbaas master: Updated from global requirements  https://review.openstack.org/55557401:22
openstackgerritOpenStack Proposal Bot proposed openstack/octavia master: Updated from global requirements  https://review.openstack.org/55557701:26
*** sapd has quit IRC01:34
openstackgerritOpenStack Proposal Bot proposed openstack/python-octaviaclient master: Updated from global requirements  https://review.openstack.org/55559101:50
*** sapd has joined #openstack-lbaas02:01
*** AlexeyAbashkin has joined #openstack-lbaas03:15
*** AlexeyAbashkin has quit IRC03:20
*** harlowja has joined #openstack-lbaas04:19
*** links has joined #openstack-lbaas04:26
*** sapd_ has joined #openstack-lbaas04:51
*** sapd has quit IRC04:51
*** kobis has joined #openstack-lbaas05:10
*** harlowja has quit IRC05:12
*** imacdonn has quit IRC05:15
*** AlexeyAbashkin has joined #openstack-lbaas05:15
*** imacdonn has joined #openstack-lbaas05:15
*** kobis has quit IRC05:17
*** AlexeyAbashkin has quit IRC05:19
*** kobis has joined #openstack-lbaas05:20
*** kobis has quit IRC05:28
*** AlexeyAbashkin has joined #openstack-lbaas06:17
*** AlexeyAbashkin has quit IRC06:22
*** velizarx has joined #openstack-lbaas06:48
*** aojea has joined #openstack-lbaas06:52
*** ssmith has joined #openstack-lbaas07:16
*** pcaruana has joined #openstack-lbaas07:21
*** ssmith has quit IRC07:21
*** aojea has quit IRC07:28
*** velizarx has quit IRC07:30
*** velizarx has joined #openstack-lbaas07:42
*** KeithMnemonic1 has quit IRC07:46
*** KeithMnemonic1 has joined #openstack-lbaas07:46
*** ianychoi has quit IRC08:00
*** ianychoi has joined #openstack-lbaas08:00
*** AlexeyAbashkin has joined #openstack-lbaas08:02
*** huseyin has joined #openstack-lbaas08:20
*** tesseract has joined #openstack-lbaas08:20
*** rcernin has quit IRC09:20
*** yamamoto has quit IRC10:05
*** yamamoto has joined #openstack-lbaas10:06
*** rcernin has joined #openstack-lbaas10:08
*** yamamoto has quit IRC10:11
*** salmankhan has joined #openstack-lbaas10:15
*** salmankhan1 has joined #openstack-lbaas10:20
*** salmankhan has quit IRC10:22
*** salmankhan1 is now known as salmankhan10:22
*** fnaval has quit IRC10:38
*** annp has quit IRC11:06
*** yamamoto has joined #openstack-lbaas11:07
*** huseyin has quit IRC11:08
*** yamamoto has quit IRC11:13
*** pcaruana has quit IRC11:27
*** salmankhan has quit IRC11:36
*** salmankhan has joined #openstack-lbaas11:45
*** yamamoto has joined #openstack-lbaas11:56
*** yamamoto has quit IRC11:56
*** yamamoto has joined #openstack-lbaas12:10
*** yamamoto has quit IRC12:25
*** velizarx has quit IRC12:28
*** pcaruana has joined #openstack-lbaas12:30
*** rcernin has quit IRC12:30
*** voelzmo has joined #openstack-lbaas12:31
*** voelzmo has quit IRC12:47
andreykurilin_rm_work: hi! can you help me with an issue related to neutron lbaas_v2 extension ?13:04
*** yamamoto has joined #openstack-lbaas13:08
*** yamamoto has quit IRC13:08
*** yamamoto has joined #openstack-lbaas13:08
*** yamamoto has quit IRC13:08
*** yamamoto has joined #openstack-lbaas13:15
*** ssmith has joined #openstack-lbaas13:16
*** cpusmith has joined #openstack-lbaas13:17
*** yamamoto has quit IRC13:19
*** ssmith has quit IRC13:21
*** yamamoto has joined #openstack-lbaas13:30
*** yamamoto has quit IRC13:35
*** toker_ has quit IRC13:39
*** yamamoto has joined #openstack-lbaas13:45
*** yamamoto has quit IRC13:50
*** yamamoto has joined #openstack-lbaas13:51
openstackgerritMonty Taylor proposed openstack/octavia master: Rename python-openstacksdk to openstacksdk  https://review.openstack.org/55578513:52
*** yamamoto has quit IRC13:56
*** links has quit IRC13:58
rm_workandreykurilin_: mayhaps14:04
rm_workandreykurilin_: where are you using n-lbaas?14:04
andreykurilin_in gates14:04
rm_workah14:04
andreykurilin_here is a configuration of a job - https://review.openstack.org/#/c/551942/19/.zuul.d/rally-task-octavia.yaml14:04
andreykurilin_it is quite simple14:05
andreykurilin_but it doesn't work14:05
andreykurilin_lol14:05
andreykurilin_http://logs.openstack.org/42/551942/19/check/rally-task-octavia/df8b8a6/results/report.html.gz#/NeutronLoadbalancerV2.create_and_list_loadbalancers/failures14:05
andreykurilin_`neutronclient.create_loadbalancer` fails with strange error14:05
rm_workhmm14:06
andreykurilin_and I do not know what to check in the logs - http://logs.openstack.org/42/551942/19/check/rally-task-octavia/df8b8a6/controller/logs/14:06
andreykurilin_firstly, I thought that some external services affects the installetion (there were bgvpn and other extenstions). But cleaning up the gate doesn't help14:07
rm_workneed to look at logs for o-api and o-cw14:07
rm_workcan we see those?14:08
andreykurilin_I do not see them14:08
rm_workhmm14:08
andreykurilin_But service catalog says that load-balancer service is available14:08
andreykurilin_http://logs.openstack.org/42/551942/19/check/rally-task-octavia/df8b8a6/job-output.txt.gz#_2018-03-23_12_22_29_96872014:08
rm_workyeah no screen-o-api on the controller node logs :(14:09
rm_workwhere is octavia running?14:09
andreykurilin_rm_work: should I list something else in devstack_services section here - https://review.openstack.org/#/c/551942/19/.zuul.d/rally-task-octavia.yaml ?14:09
rm_workhmmm no i don't think so14:09
rm_workwe need to track down the octavia logs14:10
rm_workor it's really impossible to say what's happening14:10
andreykurilin_I can try to add o-api: true and o-cw: true instead of octavia:true14:10
rm_workno that should not be necessary14:10
rm_workour devstack plugin takes care of it14:10
rm_workbut -- why are you building against neutron-lbaas? why not just octavia directly14:11
andreykurilin_it looks like not14:11
rm_workneutron-lbaas is deprecated14:11
andreykurilin_yes, but we have a scenario  for lbaas and it would be nice to check that it is not broken14:11
rm_workhmmm k14:11
rm_worki mean it would be nice to have rally for octavia directly :P14:12
rm_workbuilding more stuff for n-lbaas at this point is a little ... sad :/14:12
rm_worki need to afk for a few, will be back in ~3014:13
andreykurilin_rm_work: we do not plan to extend a number of plugins for lbaas ;) and we just merged an init patch for direct octavia plugins14:13
rm_workthat's good!14:13
*** irenab has quit IRC14:18
*** irenab has joined #openstack-lbaas14:19
*** fnaval has joined #openstack-lbaas14:48
-openstackstatus- NOTICE: zuul.o.o has been restarted to pick up latest code base and clear memory usage. Both check / gate queues were saved, be sure to check your patches and recheck when needed.14:51
*** yamamoto has joined #openstack-lbaas14:52
*** yamamoto has quit IRC14:57
rm_workso i know these are just going to fail again, but i'm rechecking them just so we have more examples of the multinode failing15:26
rm_workxgerman_: so, at LEAST the first three in this chain will be useful for you: https://review.openstack.org/#/c/555471/1 https://review.openstack.org/#/c/555472/1 https://review.openstack.org/#/c/555473/115:28
rm_workeven if we can't merge the rest15:28
rm_workbut I'm hoping we can figure out an exception for that15:28
rm_worki don't want to have to rewrite it without futurist15:28
johnsomUmm, why are we doing this?  This is pretty heavy duty backporting????15:30
rm_workthe HMs in pike are just *broken*15:30
rm_workI am honestly not sure how they are working for anyone15:30
rm_worki thought "ok, MAYBE it's just me" but German is seeing the same stuff15:31
johnsomNo, he is seeing different things, in a  highly hacked environment. He isn't even running current stable/pike15:31
johnsomHis context switch counts are all 015:31
johnsomHe also had a 3.3GB log file that was tanking IO15:32
rm_worki had a 64gb logfile15:32
rm_workit doesn't matter THAT much15:32
rm_workunless he's on like15:32
rm_workdieing 5400rpm drives15:32
johnsomI had to wait while it attempted to open the file with less15:32
rm_worki mean, sure, if you're opening the whole file15:33
rm_workbut the writing appends pretty seamlessly15:33
rm_worksame thing when i am looking at mine15:33
johnsomI just don't see any of the signs you had and a bunch of signs that this environment has been messed with badly15:34
rm_workwell, he can pull down this chain and test with it15:34
rm_workand see15:34
rm_workbut i don't see how the HM can work even acceptably in Pike15:34
johnsomSo, I don't want to jump to backporting huge amounts of code that verge on against policy if we can't show they are needed15:34
rm_workwell, he can test them in his env and show15:35
johnsomYeah, I think #1 is get it on an actual stable/pike release15:35
johnsomWe also need to merge this: https://review.openstack.org/#/c/554063/15:35
rm_workthe first three there shouldn't hurt though15:36
rm_workok15:36
johnsomI just hate to see you do a whole bunch of work for something I don't think is related. I think we should look at this closer before we burn a bunch of time15:36
*** ftersin has quit IRC15:40
*** links has joined #openstack-lbaas15:40
rm_workyeah15:42
*** links has quit IRC15:46
-openstackstatus- NOTICE: Gerrit will be temporarily unreachable as we restart it to complete the rename of some projects.15:49
rm_workand great, there goes gerrit15:52
rm_workright as I do a big review chain lol15:53
*** yamamoto has joined #openstack-lbaas15:53
*** yamamoto has quit IRC15:59
cgoncalvesFYI, containerizing neutron-lbaas API in triple-o https://review.openstack.org/#/c/555011/16:06
rm_workcgoncalves: s/containerizing/removing/ ?16:12
xgerman_rm_work: I am with you. They are going to give that to real customers soon and I hate to have to deal with those issues pressed for time. Also the only reason I am not on stable/pike is that I needed bug fixes which haven’t been released yet16:12
rm_workxgerman_: which ones are you running? can you not backport them too?16:13
johnsomIt looks like he is on a hacked 1.0.0 of pike16:13
cgoncalvesrm_work: haha nop16:14
*** AlexeyAbashkin has quit IRC16:14
johnsomcgoncalves Nice16:14
rm_workyeah but he must have a list of patches he is applying?16:14
rm_workcgoncalves: you may be doing it wrong :(16:14
xgerman_well, I run off a commit sha on stable/pike16:14
johnsomNone so far16:14
johnsomJust code changes16:14
xgerman_just not a released version16:14
rm_workerr16:14
rm_workif you run off a commit sha on stable pike16:14
xgerman_also I only hacked one of my 3 hms - the other ones ae still stock16:14
cgoncalvesrm_work: what would be the fun of doing things right?! ;)16:14
rm_workyou are either ... BEHIND stable/pike... or not on stable pike16:14
rm_workxgerman_: can you apply that series of patches to one of your HMs16:15
rm_workjust do the cherry-pick from each one16:15
xgerman_aure16:15
johnsomrm_work German says the compute IDs on the amps in this environment are wrong. Have you ever seen anything like that?16:15
rm_workno16:15
xgerman_well, they did some live migration16:15
johnsomThat should not change the compute ID though.....16:16
*** harlowja has joined #openstack-lbaas16:16
xgerman_yeah, it’s not like we tested that a lot ;-)16:17
rm_worki would mark that down as a "wtf" and just move on and see if it happens again16:18
johnsomI just don't think we should be panicing here. Not until we dig deeper and understand why this environment is behaving differently than all of the others.16:18
rm_workjohnsom: i don't care how hacked the environment is ... the old HM stuff was inherently flawed16:18
xgerman_ok, they are gearing up to give that to real customers with SLAs — so fun times ahead16:18
rm_workat LEAST he should have that first patch16:19
rm_workthe one that changes the queries from like ... hundreds, to 10s16:19
johnsomPersonally, my first step would be to add the timing code and see if it matters.16:19
rm_workand the third one16:19
rm_workwhich does the filtering when it gets too far behind, and keeps it from cascading16:20
rm_workthe middle one is just to prevent about a billion merge conflict lines16:20
rm_workcan you agree that those first three would be helpful?16:20
rm_workand should be ripe candidates for backport?16:20
rm_workI think we may need to backport the HM update threading to queens...16:21
johnsomI have commented on all three, still  reviewing the scope of #316:21
rm_worki fixed them16:21
rm_workrefresh16:21
rm_workthough gerrit is being dumb so the related-changes tab won't work :(16:22
johnsomIf this lab setup changes the compute ID when it does a live migration, that is a show stopper for them.  Like major not-in-compliance with upstream breakage.  How would an end user even track/find their instances?16:24
johnsomYeah, gerrit is really unhappy. The search page still shows old votes, but the patch details page is updated. so....16:25
xgerman_so I compared the compute-ids in the octavia DB with the ones I get from openstack server list and they line up16:25
rm_workT_T16:26
rm_workso, they didn't change16:26
rm_workgood16:26
rm_workthat would be *weird*16:26
johnsomrm_work The timeouts patch looks pretty good.  Do you want me to just edit that patch with proposed wording?16:27
johnsomOr comment on it16:27
rm_workcomment or edit, either way is fine16:28
rm_workbut, we need to maybe add more timeouts?16:28
rm_workthere was a longer list in the story16:28
johnsomOk, I see there are other bugs there, so maybe I will comment some proposals16:28
rm_worki only got the ones from the etherpad16:28
rm_workyeah also like16:28
rm_workthe max and defaults16:28
rm_worki was expecting some comments on that16:29
rm_worki just picked a random time (one year)16:29
johnsomHa, yeah, didn't look at the constants yet.  Umm, a year might be a bit long....16:29
johnsomOh, for max, yeah... that is probably fine16:30
xgerman_rm_work: However, I am getting messages from amps which have a compute-id in the DB which isn’t in openstack server list16:30
xgerman_now if we would log something else I can work with like IP16:31
rm_workerr16:32
rm_workso16:32
rm_worki've seen a couple of things16:32
johnsomrm_work which other timeouts are you considering? I'm fine with starting here and adding as needed, but we can talk about others too.16:32
rm_workone: amps where nova says it deletes it, and tells octavia it deleted it, and then it didn't actually go away so keeps heartbeating16:32
rm_workso we have a heartbeat coming from an amp that SHOULD be dead and we no longer have a record of16:32
johnsomYeah, he has that too16:33
rm_workyeah i have puzzled on that one16:33
rm_worki am not sure what we can do16:33
rm_workwhen nova straight up lies to us16:33
xgerman_why would I be unique16:33
xgerman_yep16:33
xgerman_I will have my support people look into that16:34
rm_worki have considered actually having the heartbeat code detect when that happens and VERY loudly complain16:34
rm_workor possibly even remediate (delete the server again) lol16:34
xgerman_sweet16:34
xgerman_I tried deleting but nova says “what are yiu talking about nothing by that id”16:34
rm_workwell the ID you get is the amp-id not the compute id16:35
rm_workso you need to look at what IP it's coming from16:35
rm_workwhich we DO log16:35
xgerman_I know how to do a selct on our DB to get to compute-id16:35
xgerman_https://www.irccloud.com/pastebin/ACYn1k2p/16:36
xgerman_^^ this needs the IP16:36
rm_workwell the problem is that the amp record is gone16:36
rm_workprobably16:36
rm_workso you need the IP so you can look up the server-id in nova by the IP16:36
xgerman_nope, I have the amp record16:36
xgerman_it’s DELETED16:37
rm_workah16:37
rm_workyeah16:37
rm_workinteresting that yours stayed around...16:37
*** bbzhao has quit IRC16:39
*** bbzhao has joined #openstack-lbaas16:40
cgoncalvesFYI (2): at last, python-cryptography and pyOpenSSL RDO packages updated. octavia.spec with bumped versions verified by CI16:45
rm_workyay16:52
rm_workTHAT is good news ;)16:52
*** Swami has joined #openstack-lbaas16:54
*** yamamoto has joined #openstack-lbaas16:55
rm_workhmmm though the timeout patch did NOT work well in zuul16:59
rm_workwtf did i do lol16:59
*** yamamoto has quit IRC17:00
rm_worki'm not seeing why this would have died :/17:10
*** pcaruana has quit IRC17:23
johnsomIt looked like something puked when talking to the amp agent17:33
rm_workyeah every time it tries to do a listener config to the amp17:46
rm_workwhich is ... weird17:46
rm_worki mean, I DID change that code17:47
rm_workbut it's passing every unit, and i can't see how it would fail17:47
rm_workoh17:47
rm_workunless i put something in the wrong place and haproxy is actually failing a check17:47
johnsomThere should be a plaque for that saying "it's passing every unit, and i can't see how it would fail"17:48
johnsomWe can just hand it to each other when we say that....17:48
*** harlowja has quit IRC17:50
rm_worklol17:51
rm_worknah haproxy check on the config is showing it's fine17:51
rm_workdid a test render and checked it with haproxy locally17:51
*** aojea has joined #openstack-lbaas17:51
*** AlexeyAbashkin has joined #openstack-lbaas17:53
rm_workgonna devstack it17:53
rm_workbut it'll take a while T_T17:53
*** ipsecguy has quit IRC17:55
*** yamamoto has joined #openstack-lbaas17:56
*** AlexeyAbashkin has quit IRC17:57
*** AlexeyAbashkin has joined #openstack-lbaas17:58
rm_workAHHHH got it17:59
rm_workALSO18:00
rm_workinteresting tidbit18:00
rm_workif the config we send to the amp fails18:00
rm_workwe mark the listener and LB as ACTIVE again?18:00
*** ipsecguy has joined #openstack-lbaas18:00
rm_workoh nm18:01
rm_workour client doesn't list the statuses on objects :(18:01
rm_workon a list18:01
*** ipsecguy has quit IRC18:01
*** yamamoto has quit IRC18:02
*** AlexeyAbashkin has quit IRC18:03
rm_workjohnsom: ah18:04
rm_workjohnsom: so18:04
rm_workthere's a couple things here i could fix18:04
rm_workfor existing LBs, after the migration, their timeouts in the DB are null18:04
rm_workand haproxy renderer isn't correctly substituting defaults for those because i removed that bit because it assumed there couldn't not be a default18:05
rm_workbut i forgot about that case18:05
rm_workSO18:05
rm_workI can fix that so it will default18:05
rm_workbut should I ALSO have the migration set the defaults?18:05
rm_workI think so18:05
johnsomWell, it's best to have the jinja handle the null situation as you could hit a scenario where the DB migration is done before the controllers and an LB could be created in between18:06
*** aojea has quit IRC18:09
rm_workright i can do that18:09
rm_worki am saying, both18:09
johnsom+118:09
rm_workk18:09
rm_workpep8 checks running and then i'll review the fix18:09
rm_worki took it out of jinja because i wanted to use constants for the defaults... just means i have to put it in the renderer code18:10
rm_worki added a unit test18:10
johnsomI gave some docs ideas18:10
rm_workk18:10
*** ipsecguy has joined #openstack-lbaas18:11
rm_workyeah thats kinda what i was thinking but i thought for someone who isn't an HAProxy or LB expert, that would not be very useful lol18:11
rm_worklike... what does that mean18:11
*** aojea has joined #openstack-lbaas18:12
johnsomWe can always put a more descriptive block above.18:12
rm_workyeah18:12
rm_worki was thinking on that too18:12
rm_workthough would we have to copy/paste it between CREATE/UPDATE?18:12
rm_workgotta update the reno note as well18:13
johnsomYeah, maybe. I can't remember what I did on that for the other sections.18:13
rm_workok, well, review up18:16
openstackgerritAdam Harwell proposed openstack/octavia master: Expose timeout options  https://review.openstack.org/55545418:16
rm_workfixed the issue18:16
rm_workverified fixed in my devstack18:18
rm_workzuul should be happier this time18:19
rm_worki would be happy to add at least one or two more timeouts though if we need them to close that story18:19
rm_workwhat do you think we need to *close* that?18:19
johnsomWell, I think we have the minimum that people are asking for.  The others are icing and start to lean towards haproxy specific18:20
rm_workyeah ok18:26
rm_workso you're ok with the story closing from this?18:26
rm_workblah, my devstack has neutron-openvsw spinning at 100% CPU for some reason >_>18:28
*** aojea has quit IRC18:30
*** harlowja has joined #openstack-lbaas18:33
rm_workuggggh and we need to figure out multinode18:35
rm_workhave you pinged anyone else yet?18:35
rm_workI might ping some folks today18:36
johnsomI have not yet18:37
*** harlowja has quit IRC18:38
rm_workk18:39
johnsomI wanted to figure out the failover part first18:40
rm_workk18:42
rm_worki feel like it's probably related18:47
rm_workespecially since it IS technically intermittent18:47
johnsomWell, yeah, I think we have two issues.  1. why is it failing over. 2. why does node 2 not have o-hm0 access18:47
rm_worki would guess there's some correlation ;P18:54
rm_worklike, perhaps the health monitoring is so close that without both, it times out18:55
rm_worki would try to fix #2 first18:55
*** yamamoto has joined #openstack-lbaas18:58
*** salmankhan has quit IRC18:59
*** yamamoto has quit IRC19:02
*** harlowja has joined #openstack-lbaas19:09
*** AlexeyAbashkin has joined #openstack-lbaas19:14
*** AlexeyAbashkin has quit IRC19:19
*** openstackgerrit has quit IRC19:34
*** yamamoto has joined #openstack-lbaas19:58
*** yamamoto has quit IRC20:04
*** tesseract has quit IRC20:09
*** AlexeyAbashkin has joined #openstack-lbaas20:14
*** AlexeyAbashkin has quit IRC20:18
*** openstackgerrit has joined #openstack-lbaas20:34
openstackgerritGerman Eichberger proposed openstack/neutron-lbaas master: Fix proxy extension for neutron RBAC  https://review.openstack.org/55400420:34
*** cpusmith_ has joined #openstack-lbaas20:50
*** cpusmith has quit IRC20:53
*** cpusmith_ has quit IRC20:55
*** yamamoto has joined #openstack-lbaas21:00
*** yamamoto has quit IRC21:06
openstackgerritDoug Hellmann proposed openstack/neutron-lbaas master: add lower-constraints job  https://review.openstack.org/55594321:28
openstackgerritDoug Hellmann proposed openstack/neutron-lbaas-dashboard master: add lower-constraints job  https://review.openstack.org/55594421:28
openstackgerritAdam Harwell proposed openstack/octavia master: Expose timeout options  https://review.openstack.org/55545421:59
*** yamamoto has joined #openstack-lbaas22:02
*** yamamoto has quit IRC22:08
*** fnaval has quit IRC22:18
*** KeithMnemonic1 has quit IRC22:24
johnsomFYI, my neutron-openvswitch-agent on my devstack is also spinning 100% CPU22:46
johnsomrestart of that doesn't seem to help either22:48
*** aojea has joined #openstack-lbaas22:49
*** yamamoto has joined #openstack-lbaas23:04
*** yamamoto has quit IRC23:10
*** aojea has quit IRC23:54
*** Swami has quit IRC23:59
« Thursday, 2018-03-22 Index Saturday, 2018-03-24 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!