Tuesday, 2018-05-08

« Monday, 2018-05-07 Index Wednesday, 2018-05-09 »
*** longkb has joined #openstack-lbaas00:23
*** longkb1 has joined #openstack-lbaas00:53
*** longkb has quit IRC00:55
*** longkb1 has quit IRC01:13
*** longkb has joined #openstack-lbaas01:13
*** sapd has joined #openstack-lbaas01:40
*** longkb1 has joined #openstack-lbaas02:14
*** longkb has quit IRC02:17
*** yamamoto has joined #openstack-lbaas02:17
*** xingzhang has joined #openstack-lbaas02:17
*** yamamoto has quit IRC02:22
*** phuoc_ has joined #openstack-lbaas02:35
*** longkb has joined #openstack-lbaas02:38
*** phuoc has quit IRC02:38
*** longkb1 has quit IRC02:39
*** yamamoto has joined #openstack-lbaas03:04
*** yamamoto has quit IRC03:31
*** sapd has quit IRC03:48
*** longkb has quit IRC04:01
*** longkb has joined #openstack-lbaas04:01
*** fnaval has quit IRC04:11
*** xingzhang has quit IRC04:13
*** xingzhang has joined #openstack-lbaas04:14
*** links has joined #openstack-lbaas04:38
*** yamamoto has joined #openstack-lbaas05:06
*** xingzhang has quit IRC05:16
*** xingzhang has joined #openstack-lbaas05:16
*** yboaron has joined #openstack-lbaas05:51
*** pcaruana has joined #openstack-lbaas06:01
*** xingzhang has quit IRC06:05
*** xingzhang has joined #openstack-lbaas06:05
*** xingzhang has quit IRC06:15
*** xingzhang has joined #openstack-lbaas06:15
*** annp has joined #openstack-lbaas06:23
*** rcernin has quit IRC07:08
*** sapd has joined #openstack-lbaas07:11
*** salmankhan has joined #openstack-lbaas07:24
*** xingzhang has left #openstack-lbaas07:26
openstackgerritOpenStack Proposal Bot proposed openstack/octavia-dashboard master: Imported Translations from Zanata  https://review.openstack.org/56680307:43
*** nmagnezi has quit IRC07:46
*** nmagnezi has joined #openstack-lbaas07:47
*** yamamoto has quit IRC07:56
*** yamamoto has joined #openstack-lbaas08:03
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create api+scenario tests for pools  https://review.openstack.org/56564008:12
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create api+scenario tests for members  https://review.openstack.org/56619908:12
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create basic traffic balancing scenario test  https://review.openstack.org/56670008:12
*** yamamoto_ has joined #openstack-lbaas08:40
*** zigo_ has quit IRC08:40
*** yamamoto has quit IRC08:41
*** salmankhan has quit IRC08:43
*** salmankhan has joined #openstack-lbaas08:43
*** zigo has joined #openstack-lbaas08:45
*** nmagnezi_ has joined #openstack-lbaas08:53
*** nmagnezi has quit IRC08:54
*** yamamoto_ has quit IRC08:56
*** yamamoto has joined #openstack-lbaas09:08
*** fnaval has joined #openstack-lbaas09:16
*** fnaval has quit IRC09:20
*** salmankhan has quit IRC09:46
*** salmankhan has joined #openstack-lbaas09:46
*** salmankhan has quit IRC09:51
*** salmankhan has joined #openstack-lbaas09:51
*** salmankhan has quit IRC09:58
*** salmankhan has joined #openstack-lbaas10:02
*** leitan has joined #openstack-lbaas10:06
*** salmankhan has quit IRC10:12
*** yamamoto has quit IRC10:15
*** leitan has quit IRC10:17
*** yamamoto has joined #openstack-lbaas10:20
*** salmankhan has joined #openstack-lbaas10:34
*** rcernin has joined #openstack-lbaas10:35
*** salmankhan has quit IRC10:45
*** yamamoto has quit IRC10:48
*** salmankhan has joined #openstack-lbaas11:01
*** longkb has quit IRC11:10
*** yamamoto has joined #openstack-lbaas11:37
*** annp has quit IRC11:53
*** samccann has joined #openstack-lbaas11:53
*** yboaron has quit IRC12:00
*** atoth has joined #openstack-lbaas12:01
devfazhi, whats the best way to re(create) an backup amphora if they got removed from amphora-table?12:40
*** openstackgerrit has quit IRC12:49
*** openstackgerrit has joined #openstack-lbaas12:51
openstackgerritJan Zerebecki proposed openstack/neutron-lbaas master: Get providers directly from ORM to make startup take half as long  https://review.openstack.org/56654312:51
*** yamamoto has quit IRC13:16
*** yamamoto has joined #openstack-lbaas13:16
*** yboaron has joined #openstack-lbaas13:20
*** amotoki has quit IRC13:51
*** gans has joined #openstack-lbaas13:52
*** yamamoto has quit IRC14:16
*** yamamoto has joined #openstack-lbaas14:19
xgerman_rm_work: before I change https://review.openstack.org/#/c/566726/ back to periodic (remove the job form the gate please notice the +114:22
openstackgerritGerman Eichberger proposed openstack/octavia master: Fix periodic job  https://review.openstack.org/56672614:26
*** fnaval has joined #openstack-lbaas14:35
*** yboaron has quit IRC14:37
*** gans has quit IRC14:39
*** rcernin has quit IRC14:47
openstackgerritGerman Eichberger proposed openstack/octavia master: Add a config to surpress amphora logging  https://review.openstack.org/56674114:49
*** sapd1 has joined #openstack-lbaas14:52
*** salmankhan has quit IRC15:08
*** links has quit IRC15:08
*** salmankhan has joined #openstack-lbaas15:10
*** dayou_ has joined #openstack-lbaas15:39
*** harlowja has joined #openstack-lbaas16:17
*** yamamoto has quit IRC16:21
*** dayou_ has quit IRC16:33
openstackgerritMichael Johnson proposed openstack/octavia master: Implement provider drivers - Members  https://review.openstack.org/56693916:33
*** yamamoto has joined #openstack-lbaas16:39
*** sshank has joined #openstack-lbaas16:49
*** sapd1 has quit IRC16:53
*** yamamoto has quit IRC17:10
*** harlowja has quit IRC17:31
*** salmankhan has quit IRC17:38
rm_workxgerman_: what did you want me to look at here?17:58
xgerman_just making sure that you saw the job pass in the check queue to give you enough confidence for a +217:58
rm_workah, did we actually remove it from periodic? we actually have to add it back17:59
rm_work?17:59
rm_workoh, no, just the fix, got it17:59
xgerman_yeah, this will make periodic work again18:00
rm_worki see the success on the ubuntu one18:00
rm_worki wish you'd done both of them18:00
rm_worksince the centos version sometimes has its own issues18:00
xgerman_yep, only did one of them since it’s the same DIB18:00
rm_worksame DIB but totally different packages in the OS18:00
xgerman_the modification was so the DIB would run…18:01
rm_workso it often runs into distinct issues from the ubuntu version :(18:01
rm_workk18:01
rm_worki guess we'll see18:01
rm_workit fixes *something*, so at worst we call it a fix for the ubuntu periodic18:01
rm_workso fine by me18:01
xgerman_yeah,  I only addressed “DIB installatin fails”18:01
xgerman_k18:01
rm_workjust saying, i wish you had done both. in the future it would be great to see both as it's not that much more work to temporarily enable them18:02
rm_work+2'd18:02
xgerman_k, understood - will do so in the future18:02
xgerman_thx18:02
rm_workcgoncalves: stestr migration takes two patches?18:03
rm_workT_T weird. but k, i see it18:03
rm_workcgoncalves: so how much of the grenade is this actually doing now? https://review.openstack.org/#/c/549654/18:06
rm_worki see it's passing, is it actually done or close?18:06
*** yamamoto has joined #openstack-lbaas18:11
openstackgerritMichael Johnson proposed openstack/octavia master: Implement provider drivers - Members  https://review.openstack.org/56693918:13
*** yamamoto has quit IRC18:21
johnsomxgerman_ I tested the "no log" in an amp, this gives us what we want.18:23
xgerman_ok18:23
xgerman_“no log” on the gloabl section?18:23
johnsomYeah, I commented on the patch with the config change required18:23
xgerman_thx18:23
johnsomrm_work I -1'd the member tempest test since it didn't cover batch updates. I sure would be nice if we had a test for that where I can point my driver patch at it.....  I manually tested, but that doesn't mean I didn't do something else dumb.18:24
*** harlowja has joined #openstack-lbaas18:25
rm_workk18:25
rm_workthat should be doable18:25
rm_worki'm still trying to fix the listener stuff tho18:25
rm_workapparently i got the statuses wrong? kinda?18:25
rm_workbut not entirely?18:25
johnsomOk, no problem. I'm moving on to health monitors now18:25
rm_workit's frustrating18:25
rm_workyeah i need to get to that too18:26
rm_workbut i can't get these to run properly :/18:26
johnsomI have HM and the two L7's left. Then a bunch of unit test work and the API18:26
rm_workcode claims listeners should go ONLINE so long as they don't have too many connections, as you said18:26
johnsomClean up stuff18:26
rm_workbut that isn't happening for me18:26
rm_work"Listener 0648d26b-43d6-4827-b37a-4c00ac38902f reported status of DOWN"18:27
rm_workwut18:27
rm_workthat's what my HM says18:27
*** devfaz has quit IRC18:28
johnsomAdmin down?18:28
rm_workummm18:28
rm_workso that comes from update_db.py18:28
rm_workif it doesn't get either OPEN or FULL18:28
johnsomOh, hmm18:29
rm_workit got "DOWN" from Haproxy18:29
rm_workthat's literally passed through from what HAProxy says18:29
rm_workso haproxy *has it*18:29
rm_workand it's admin_state_up = True18:29
johnsomRight18:30
*** strigazi has quit IRC18:30
rm_workso is this normal?18:30
rm_workwill HAProxy always report DOWN without a working backend?18:30
*** strigazi has joined #openstack-lbaas18:31
*** devfaz has joined #openstack-lbaas18:31
rm_workis that what you see?18:32
johnsomGive me a minute to build a fresh LB18:32
johnsomoperating_status          | ONLINE18:34
rm_workwith JUST a listener?18:34
johnsomJust an LB and a listener18:34
rm_workand that listener went to ONLINE18:34
johnsomyes18:34
rm_worknot the LB, the listener18:34
rm_work....18:34
rm_workwtg18:34
rm_work*WTF18:34
johnsomhttps://www.irccloud.com/pastebin/fhaId7e1/18:35
rm_workso is this a difference between haproxy versions?18:35
johnsomThe return from the listener create is OFFLINE, but it switches18:35
rm_workright18:35
rm_workand it switches because of the status update right? in the Healthmanager process?18:35
johnsomI would be surprised if that was the case.18:36
rm_workwhat switches it to ONLINE?18:36
rm_worki am fairly positive it's update_db18:36
johnsomI assume it was health manager, there is nothing else that sets operating status beyond the initial create18:36
rm_workhttps://github.com/openstack/octavia/blob/master/octavia/controller/healthmanager/health_drivers/update_db.py#L170-L19218:36
johnsomLet me turn debug on and create again18:36
rm_workso, it must be getting OPEN as the status from HAProxy18:37
rm_workright?18:37
rm_workLooking at my HM logs, I'm seeing:18:37
rm_work"Listener 0648d26b-43d6-4827-b37a-4c00ac38902f reported status of DOWN"18:37
rm_workwhich means `listener.get('status') == constants.DOWN`18:37
rm_workwhich means HAProxy is reporting DOWN18:38
rm_workthe haproxy config is there on the amp18:39
*** Swami has joined #openstack-lbaas18:39
johnsomMay 08 11:40:10 devstackpy27-2 octavia-health-manager[92262]: DEBUG octavia.controller.healthmanager.health_drivers.update_db [-] listener ff1cb44f-229f-4c13-87d9-81fbd4cd4fde status has changed from OFFLINE to ONLINE. Updating db and sending event. {{(pid=92279) _update_status_and_emit_event /opt/stack/octavia/octavia/controller/healthmanager/health_drivers/update_db.py:64}}18:40
rm_workright18:40
rm_workso my HAProxy is reporting DOWN18:40
rm_workwhile yours is reporting OPEN18:40
rm_workfor the same config?18:40
johnsomWant me to go pull the status socket?18:40
rm_workyes18:41
rm_workand remind me how to do that too :P18:41
rm_worksince there's no nc18:41
johnsomhttps://www.irccloud.com/pastebin/toO1BqSB/18:42
johnsomsocat is in the ubuntu image, not sure on the centos stuff18:43
rm_workerm18:43
rm_workok18:43
rm_worki get all the headers18:43
rm_workand no listener18:43
johnsomPositive it's not admin state down?18:44
johnsomPaste the haproxy.cfg18:44
*** Swami has quit IRC18:45
johnsomhttps://www.irccloud.com/pastebin/xz4LCjCC/18:45
rm_workhttp://paste.openstack.org/show/720610/18:45
johnsomYeah, you have no frontend config.  Something is admin_state_down18:45
rm_workhttp://paste.openstack.org/show/720611/18:46
johnsomWhat about the LB?18:46
rm_workomgwtf rofl18:47
rm_workyeah it is >_<18:47
rm_workFML18:47
johnsomHere is your sign....18:47
johnsomOk, off to make lunch, back in a few18:47
rm_worki thought it wouldn't even make the listener config18:47
johnsomfrontend is the listener18:48
johnsomand the LB combined, but if the LB is shut down, the whole thing is shutdown/removed.....18:48
*** openstackgerrit has quit IRC18:49
johnsomIt's still there and running though so it doesn't appear failed, because it's not failed, just disabled.18:49
*** openstackgerrit has joined #openstack-lbaas18:49
rm_workall of the LBs were down18:49
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create api+scenario tests for pools  https://review.openstack.org/56564018:49
rm_workbecause of a dumb copy/paste I think18:49
rm_workfixed in pools patch18:49
rm_workwe'll see what else explodes now18:50
rm_workalso brb lunch18:50
openstackgerritGerman Eichberger proposed openstack/octavia master: Add a config to surpress amphora logging  https://review.openstack.org/56674119:02
*** Swami has joined #openstack-lbaas19:10
*** Swami_ has joined #openstack-lbaas19:10
*** sshank has quit IRC19:17
*** pcaruana has quit IRC20:21
openstackgerritMichael Johnson proposed openstack/octavia master: Implement provider drivers - Health Monitor  https://review.openstack.org/56703720:27
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create api+scenario tests for pools  https://review.openstack.org/56564020:43
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create api+scenario tests for members  https://review.openstack.org/56619920:47
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create basic traffic balancing scenario test  https://review.openstack.org/56670020:47
rm_workok might have it now21:00
rm_workjohnsom: any clue why we decided to not let people update the `type` on a HM? like from HTTP to HTTPS21:02
rm_workis it because a pool's type can't change either?21:02
johnsomBecause you are essentially creating a new HM anyway21:02
rm_workok but like21:03
rm_workis there a REAL reason?21:03
rm_worki mean, why NOT allow it21:03
johnsomI don't know off my head right now.  Too bogged down in this driver stuff21:03
johnsomI certainly would become a validation nightmare I.e. HTTP to TCP21:04
rm_workkk21:04
rm_worki figured it out21:05
rm_workit is because the pool protocol also isn't changable21:05
rm_workso like... why would the HM need to change protocol21:05
*** ianychoi has quit IRC21:15
openstackgerritMichael Johnson proposed openstack/octavia master: Implement provider drivers - L7 Policy  https://review.openstack.org/56705921:29
*** samccann has quit IRC21:43
cgoncalvesrm_work, grenade should be good for reviewing :)21:53
johnsomYou skipped my "please run with debug" comment....21:53
cgoncalvesjohnsom, well I found out what was wrong with it :P21:53
cgoncalvesI'd like to revisit it sometime again because it was failing on some octavia tempest tests post-upgrade21:54
cgoncalveswhen comparing against other projects, they do not run tempest after upgrade though21:55
johnsomCould be that devstack issue (not ours) if it was just v121:56
cgoncalvesI was also hoping that the grenade zuulv3 native job would have been completed already21:56
cgoncalvesjohnsom, not following21:56
johnsomThere was an OpenStack wide issue that broke our v1 tempest gates21:57
cgoncalveshttp://logs.openstack.org/54/549654/28/check/octavia-grenade/b0a72ab/job-output.txt.gz#_2018-05-06_00_35_30_21041021:58
johnsomAh, nope, that is a bug Adam just fixed today in that test21:59
johnsomI think21:59
rm_workerr22:00
rm_workis that running our tempest tests? in which case, yes22:00
johnsomoctavia_tempest_plugin.tests.api.v2.test_listener.ListenerAPITest.test_listener_create22:00
rm_workyeah22:01
rm_workneed to merge pools22:01
rm_workI think i got it this last time22:01
rm_workthere are some comment fixes I need to make22:01
rm_workbut i'm following up with them in the next patch <_<22:01
cgoncalvesthis? https://review.openstack.org/#/c/565640/11/octavia_tempest_plugin/tests/api/v2/test_listener.py22:02
rm_workyep thats it22:04
johnsomcgoncalves FYI, the base driver functionality will be done today. I have some cleanup stuff to do in a follow on patch, but I'm working on the last object (L7rules) now22:04
rm_workbut i have one more fix i think22:04
cgoncalvesok. it would be worth re-enabling tempest post-upgrade on the grenade patch22:04
cgoncalvesjohnsom, THANK YOU!22:05
johnsomI konw you were anxious for it22:05
cgoncalvesI'm still swamped with internal tasks :(22:05
rm_workdoes it run the APIs as well as the scenarios?22:05
rm_workbecause...22:05
rm_workthe scenario tests SHOULD have been passing as-is, I think22:06
rm_work... i think22:06
rm_workit was "incorrect", but should have been passing22:06
rm_workthe API tests were the ones broken if you weren't noop22:06
johnsomcgoncalves Well, if you sudden have a bunch of time there is still flavors that has a bunch of work left.22:07
cgoncalvesrm_work, only 4 API tests failed: octavia_tempest_plugin.tests.api.v2.test_listener.ListenerAPITest.test_listener_{create,list,show,update}22:08
rm_workok yeah22:08
rm_workso the API tests were broken, yes22:08
rm_workyou're running those in a prod gate?22:08
rm_workhow is that working22:08
rm_worki mean, timing-wise22:08
rm_workcan we run that many tests without a timeout?22:08
cgoncalvesjohnsom, haha nice one :)22:09
cgoncalvesrm_work, prod gate? it's all upstream22:09
rm_workyeah i mean22:09
rm_worknot noop22:09
cgoncalvesbut it timed out after 7800 seconds22:09
johnsomrm_work What is up here? https://review.openstack.org/#/c/566199/22:15
rm_workworking on it22:22
rm_worki need to fix a listener test in the pool patch above it still22:22
rm_work(yes that sentence sounds dumb)22:22
rm_workbut i am in the middle of the batch_update test on members, so i'm going to submit that then go back22:22
johnsomOk, cool. Yeah, staring at code all day will get your mind warped. I know mine is a bit.22:23
rm_workwith these tests it'd be nice if we can merge them once i get them passing22:23
rm_workso we have a base22:24
rm_workand i can fix bugs as we find them22:24
johnsomYes. I'm trying to link my patches to your tests to give them a go.22:24
rm_work(though still I would like somewhat full reviews, to minimize the number of things that slip through... so ... meh)22:24
johnsomSame with the driver, it's a long chain we should try to start merging sooner than later22:24
rm_workyeah22:25
*** rcernin has joined #openstack-lbaas22:25
rm_workideally we'd have the tests in place first though T_T22:25
johnsomTrue22:25
openstackgerritMichael Johnson proposed openstack/octavia master: Implement provider drivers - L7 Rules  https://review.openstack.org/56707322:43
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create api+scenario tests for members  https://review.openstack.org/56619922:53
rm_workOK22:53
rm_workthat includes batch member update22:53
rm_worknow i need to fix the listeners bug22:53
johnsomAh, bummer, I hoped to delete the whole handler thing, but forgot v1 still uses it22:56
rm_workAUGH I SWEAR I HAVE FIXED THIS EXACT THING LIKE 3 TIMES BUT I CAN'T FIND ANY PROOF23:04
rm_workjohnsom: can't we just drop v1? :P23:05
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create api+scenario tests for pools  https://review.openstack.org/56564023:06
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create api+scenario tests for members  https://review.openstack.org/56619923:08
openstackgerritAdam Harwell proposed openstack/octavia-tempest-plugin master: Create basic traffic balancing scenario test  https://review.openstack.org/56670023:08
openstackgerritMichael Johnson proposed openstack/octavia master: Implement provider drivers - Members  https://review.openstack.org/56693923:38
rm_workretesting with batch?23:38
johnsomrebasing yes23:38
rm_workalso, I appreciate you didn't even touch my obvious troll about v123:39
openstackgerritMichael Johnson proposed openstack/octavia master: Implement provider drivers - Health Monitor  https://review.openstack.org/56703723:39
rm_worki was waiting for "we can't do that, because neutron-lbaas still uses it"23:39
johnsomI would if I could, in a heartbeat.  But hey, this is why we are working on what we are now....23:39
rm_workyep23:39
openstackgerritMichael Johnson proposed openstack/octavia master: Implement provider drivers - L7 Policy  https://review.openstack.org/56705923:40
openstackgerritMichael Johnson proposed openstack/octavia master: Implement provider drivers - L7 Rules  https://review.openstack.org/56707323:40
rm_workI mean technically, neutron-lbaas doesn't need v1 anymore23:40
rm_workwith a way to successfully proxy over23:40
rm_workwith v223:40
rm_workso we could already drop it <_<23:40
johnsomNot sure what I changed in pools and didn't rebase the chain, but hey, done now23:40
johnsomOctavia is starting to look like a top level project.  Chain 11 patches long in zuul23:41
johnsomI was up early to host a meeting, I think I might sign off early today. I have the last patch started, fixing/adding some unit tests, release notes, add the providers API, etc.  I will finish it up tomorrow.23:43
rm_workyes23:43
rm_workkk23:43
rm_worki'll get to HMs if I can, but not sure it'll happen today :(23:44
johnsomYeah, no worries23:44
« Monday, 2018-05-07 Index Wednesday, 2018-05-09 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!