Monday, 2018-10-01

« Sunday, 2018-09-30 Index Tuesday, 2018-10-02 »
*** JudeCross has joined #openstack-lbaas01:20
*** JudeCross has quit IRC01:24
*** kiennt26 has joined #openstack-lbaas01:27
*** ducnc has joined #openstack-lbaas02:10
*** yamamoto has joined #openstack-lbaas02:45
*** JudeCross has joined #openstack-lbaas03:21
*** kiennt26 has quit IRC03:22
*** JudeCross has quit IRC03:26
openstackgerritJacky Hu proposed openstack/octavia-tempest-plugin master: Raise build_timeout from 60 to 300  https://review.openstack.org/60674103:43
*** pcaruana has joined #openstack-lbaas04:06
*** JudeCross has joined #openstack-lbaas04:06
openstackgerritJacky Hu proposed openstack/octavia master: Make disk image buildable for fedora  https://review.openstack.org/60641704:10
*** pcaruana has quit IRC04:23
openstackgerritJacky Hu proposed openstack/octavia master: Make disk image buildable for fedora  https://review.openstack.org/60641704:33
*** ramishra has joined #openstack-lbaas04:47
*** pcaruana has joined #openstack-lbaas05:51
*** sapd1 has quit IRC07:26
*** Emine has joined #openstack-lbaas07:29
*** velizarx has joined #openstack-lbaas07:35
*** velizarx has quit IRC07:45
*** zigo has joined #openstack-lbaas07:46
*** velizarx has joined #openstack-lbaas07:52
*** abaindur has quit IRC08:04
*** ducnc has quit IRC08:08
*** celebdor has joined #openstack-lbaas08:09
*** velizarx has quit IRC08:12
*** velizarx has joined #openstack-lbaas08:15
*** sapd1 has joined #openstack-lbaas08:17
*** JudeCross has quit IRC08:20
*** yamamoto has quit IRC08:57
*** yamamoto has joined #openstack-lbaas08:58
*** yamamoto has quit IRC08:58
*** yamamoto has joined #openstack-lbaas08:59
*** yamamoto has quit IRC09:03
*** sapd1_ has joined #openstack-lbaas09:06
*** sapd1 has quit IRC09:06
openstackgerritVadim Ponomarev proposed openstack/octavia master: Fix auto setup Barbican's ACL in the legacy driver.  https://review.openstack.org/60691809:19
openstackgerritVadim Ponomarev proposed openstack/octavia master: Fix auto setup Barbican's ACL in the legacy driver.  https://review.openstack.org/60691809:20
*** salmankhan has joined #openstack-lbaas09:28
*** yamamoto has joined #openstack-lbaas09:57
*** yamamoto has quit IRC10:28
*** yamamoto has joined #openstack-lbaas10:29
*** yamamoto has quit IRC10:42
*** salmankhan1 has joined #openstack-lbaas10:46
*** salmankhan has quit IRC10:48
*** salmankhan1 is now known as salmankhan10:48
*** abaindur has joined #openstack-lbaas11:06
*** yamamoto has joined #openstack-lbaas11:18
*** savvas has joined #openstack-lbaas12:19
savvasGM everyone12:19
*** salmankhan1 has joined #openstack-lbaas12:30
*** salmankhan has quit IRC12:34
*** salmankhan1 is now known as salmankhan12:34
*** Emine has quit IRC12:38
*** ramishra has quit IRC12:46
*** velizarx has quit IRC12:47
*** Emine has joined #openstack-lbaas12:52
*** Emine has quit IRC12:59
*** yamamoto has quit IRC13:01
*** ccamposr__ has joined #openstack-lbaas13:01
*** yamamoto has joined #openstack-lbaas13:01
*** celebdor has quit IRC13:03
*** yamamoto has quit IRC13:17
*** Emine has joined #openstack-lbaas13:17
*** velizarx has joined #openstack-lbaas13:21
*** celebdor has joined #openstack-lbaas13:25
*** yamamoto has joined #openstack-lbaas14:00
xgerman_o/14:12
cgoncalvesxgerman_, https://review.openstack.org/#/c/605264/14:35
cgoncalvesonce ^ merges, I'd like to propose a rocky maintenance release14:36
cgoncalvesby my count, that would be the 7th bug fix for rocky14:36
*** velizarx has quit IRC15:05
xgerman_k15:06
*** velizarx has joined #openstack-lbaas15:08
*** yamamoto has quit IRC15:17
*** yamamoto has joined #openstack-lbaas15:18
*** yamamoto has quit IRC15:19
*** yamamoto has joined #openstack-lbaas15:19
*** yamamoto has quit IRC15:19
*** yamamoto has joined #openstack-lbaas15:23
*** yamamoto has quit IRC15:23
*** pcaruana has quit IRC15:30
*** ivve has joined #openstack-lbaas15:33
savvasHi guys, any thoughts on how I can troubleshoot this? http://paste.openstack.org/show/731180/15:52
xgerman_this can mean many things — run octavia with debug true in the config…15:52
johnsomsavvas That says that nova failed to start the service VM15:53
savvasDebug is on xgerman_ , this shows up right after SSL keys get installed and between reverting state15:53
johnsomWe timed out waiting for nova to mark the instance ACTIVE15:54
xgerman_mmh, when I switch on debug I can see the nova calls15:54
savvasI should be able to catch what's happening in my Nova logs than15:54
xgerman_yep15:55
johnsomYeah, it looks like an older version of Octavia.  I hope you are not using virtual-box....15:55
savvasnop, running OpenStack Ansible on 3-node cluster15:55
savvasQueens stable release15:55
johnsomHmm, ok, yeah, not sure why nova isn't starting in a timely way. That exception is pretty clear "Waiting for compute to go active timeout."15:57
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Add v2 two-node scenario test  https://review.openstack.org/60516315:59
savvasThink I caught it in the nova log now15:59
savvashttp://paste.openstack.org/show/731183/15:59
savvaschecking neutron now15:59
*** velizarx has quit IRC15:59
*** velizarx has joined #openstack-lbaas16:00
savvasI'll circle back in a bit, need to step out for a little, thanks guys16:01
savvashttp://paste.openstack.org/show/731184/ this is where it stops, sounds to me like I may have made an error setting up the network for Octavia16:02
johnsomYeah, check the boot network setting in the controller_worker section16:02
*** aojea has joined #openstack-lbaas16:03
savvasye it does seem to take the right network16:04
*** aojea has quit IRC16:15
*** savvas has quit IRC16:17
*** velizarx has quit IRC16:31
*** velizarx has joined #openstack-lbaas16:35
johnsomcores: Eyes on this patch would be good as it appears the centos 7 gate is broken without it: https://review.openstack.org/#/c/605894/16:41
johnsomWhich is blocking things from merging16:41
johnsomThanks German for already reviewing!16:41
xgerman_:-)16:42
openstackgerritMerged openstack/octavia stable/rocky: Fix health manager performance regression  https://review.openstack.org/60526416:45
*** evgenyf has joined #openstack-lbaas16:46
cgoncalvesstable/rocky 3.0.1: https://review.openstack.org/60700416:48
*** ccamposr__ has quit IRC16:51
openstackgerritCarlos Goncalves proposed openstack/octavia master: Delete zombie amphorae when detected  https://review.openstack.org/58750516:53
johnsomI had just started reading that, but got distracted....16:53
*** aojea has joined #openstack-lbaas16:55
*** pcaruana has joined #openstack-lbaas16:56
*** velizarx has quit IRC16:59
*** KeithMnemonic has joined #openstack-lbaas17:19
*** savvas has joined #openstack-lbaas17:19
*** yamamoto has joined #openstack-lbaas17:24
savvasjohnsom: can the network be a flat network?17:25
johnsomSure17:25
savvasAlright, well I am not sure what to look for at this point, it says port binding failed17:25
johnsomMaybe ask in the openstack-neutron channel?17:26
savvasGood point, the problem limits itself to the amphora instances though, my other interfaces and instances seem to be fine. I'll ask around , thanks17:26
*** sapd1 has joined #openstack-lbaas17:30
*** JudeCross has joined #openstack-lbaas17:31
*** salmankhan has quit IRC17:31
*** salmankhan has joined #openstack-lbaas17:31
*** salmankhan has quit IRC17:36
openstackgerritCarlos Goncalves proposed openstack/octavia master: Delete zombie amphorae when detected  https://review.openstack.org/58750517:48
rm_workjohnsom: i'm not sure why that would affect the centos gate17:55
rm_worki thought about it, but17:56
rm_workit should only have affected mismatches17:56
rm_work*version mismatches17:56
rm_workwas trying to figure out what the centos issue was but i came to the conclusion that it must be an upstream package/server issue that would hopefully resolve itself17:56
rm_workbut i wasn't 100% sure17:57
johnsomIf the amp has 1.5 haproxy in it the cfg verify is going to fail with the http-reuse line17:57
rm_workright, but it shouldn't ever in that gate17:57
rm_workcurrent centos amps have 1.817:57
*** blake has joined #openstack-lbaas17:57
rm_workif that was actually being tested by that gate, it would have caught it when we first tried to merge the patch that broke it17:58
johnsomI wonder if that is the case as the centos gates started dying right after that merged.17:58
johnsomI checked, my patch landed before the centos gate was there17:58
sapd1johnsom:  Could you review my patch https://review.openstack.org/#/c/601086/?18:01
rm_workah hmmm18:02
rm_workdoesn't make sense tho18:02
rm_workhow would it use such an old amp that we have 1.5?18:02
johnsomrm_work I don't see haproxy18 anywhere in the devstack log, I don't think it's installing it18:02
rm_workerrr18:02
rm_workit's just part of the DIB build18:02
rm_worklike18:02
rm_workhow would it NOT install it?18:02
rm_workunless it is accidentally pinned on a VERY old version for the amps?18:02
johnsomI see it calling the "cat" to install the repo, I just don't see a 1.8 haproxy install unless I'm totally missing it.18:07
rm_workthat would be problematic in its own respect18:09
rm_workbecause it absolutely should be18:09
rm_workso THAT would also be a bug18:09
rm_workO_o18:09
johnsomOh, the run I was looking at failed with a bad mirror18:10
rm_workyes18:10
rm_workthat was my conclusion, some server issue was causing package stuff to fail for centos18:11
cgoncalvesxgerman_, you made it! you're officially a zombie hunter :)18:11
xgerman_yeah!!!18:11
rm_work:P18:12
xgerman_I knew when I let you fix my silly mistakes it will all happen18:12
rm_worklol18:12
johnsomrm_work Ok, so I see the "Gate" job finished, and has 1.8, looking at why it failed18:12
johnsomYeah, failed inside the amp18:13
johnsomhttp://logs.openstack.org/24/604924/1/gate/octavia-v2-dsvm-scenario-centos-7/4db1414/controller/logs/screen-o-cw.txt.gz?level=ERROR#_Sep_26_15_48_56_94890918:13
rm_workhm18:14
johnsomIt's bombing out in octavia-create-l7policy-flow18:16
johnsomsapd1 Doesn't look like you needed me18:17
sapd1^^18:17
sapd1I think my patch for octavia-client need review as well?18:20
sapd1https://review.openstack.org/#/c/605914/118:20
openstackgerritsapd proposed openstack/python-octaviaclient master: Support REDIRECT_PREFIX for openstack client  https://review.openstack.org/60591418:31
*** aojea has quit IRC18:32
*** aojea has joined #openstack-lbaas18:32
*** sapd1 has quit IRC18:38
*** abaindur has quit IRC18:40
*** abaindur has joined #openstack-lbaas18:40
*** abaindur has quit IRC18:41
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: DNM: Testing bionic nodes  https://review.openstack.org/60053918:41
*** abaindur has joined #openstack-lbaas18:41
*** savvas has quit IRC18:48
*** blake has quit IRC19:04
openstackgerritMerged openstack/octavia master: Fix an upgrade issue for CentOS 7 amphora  https://review.openstack.org/60589419:08
rm_workjohnsom: so i don't THINK it was related still to that ^^19:21
rm_workbut that did PASS19:21
rm_workso19:21
rm_workO_o19:21
rm_worklet's see if any of the others do?19:21
rm_workugh one failed on a v1 scenario? :/19:22
openstackgerritMerged openstack/octavia master: Separate the thread pool for health and stats update  https://review.openstack.org/58158519:41
rm_workjohnsom: so err.... if someone does an update to a listener and it fails, the listener goes to error... and then our workflow is: the user has to delete the listener and recreate it?19:43
rm_workyou can't *update* an ERROR listener, right?19:43
openstackgerritCarlos Goncalves proposed openstack/octavia stable/rocky: Separate the thread pool for health and stats update  https://review.openstack.org/60703319:44
openstackgerritCarlos Goncalves proposed openstack/octavia stable/queens: Separate the thread pool for health and stats update  https://review.openstack.org/60703419:44
johnsomrm_work Right, it is delete/recreate at the moment19:45
rm_work:(19:45
rm_workthat sucks when you've spent a bunch of effort setting up L7 rules and stuff on it19:45
rm_workand then you do one update to like, tweak something, and it fails19:45
rm_worklol19:45
rm_worki think I may just trigger a failover to re-issue configs, and reset them to ACTIVE in the DB <_<19:46
rm_workthis is kinda why I wanted a "SYNC" API call19:46
rm_workfor admins19:46
johnsomYou have the power!19:47
rm_workwell19:48
rm_worki was GOING to19:48
rm_workbut everyone said they didn't want that19:48
*** fnaval has joined #openstack-lbaas20:14
rm_workjohnsom: so i haven't been able to deploy that fix yet from Friday... still seeing an abnormal number of LBs going to ERROR20:37
rm_workin my testing20:37
rm_worknot sure if just unlucky or related20:37
johnsomHmm, this is the haproxy fix?20:38
johnsomversion fix?20:38
rm_workyeah20:41
rm_workugh20:41
rm_workjohnsom: did you lie to me20:41
rm_workCalledProcessError: Command '['rpm', '-qi', 'haproxy']' returned non-zero exit status 120:41
rm_workseeing that in the amp agent log20:41
rm_worklooking into the code20:42
rm_workmaybe the old amps were actually broken20:42
rm_workif you ran a status on them <_<20:42
rm_worki may have not updated that command in the same patch that updated the version to haproxy18 <_<20:42
johnsomIt could be something else is broken20:43
rm_work...20:43
rm_worki mean, it is very clear20:43
rm_workthe status call runs20:43
rm_workand it breaks in the amp20:43
*** pcaruana has quit IRC20:43
rm_workbecause it's looking up "haproxy"20:44
rm_worknot "haproxy18"20:44
johnsomI was going off this that they have it handled: https://github.com/openstack/octavia/blob/master/octavia/amphorae/backends/agent/api_server/osutils.py#L52320:44
rm_workmaybe at the end of rocky20:44
rm_workbut not when my amps were built20:44
rm_workthat class doesn't even exist in the version of the amp agent here, lol20:46
rm_workyep20:46
rm_workcarlos fixed it in https://github.com/openstack/octavia/commit/1c4004c156684340406659535534abde7c6ad0e520:46
rm_workwhich is fine for most people (because they built amps for a release)20:47
rm_workbut i build amps constantly20:47
rm_workand mine were after the change to haproxy18 but before that20:47
rm_workso basically, this is a "me" problem, and I'm pretty f'd20:47
rm_workI need to failover everything old20:47
rm_workthat's just how it has to be20:47
*** ivve has quit IRC21:14
cgoncalvesjohnsom, re: https://review.openstack.org/#/c/606142/ I thought about adding a release note, too, so I could do so sure. my question to you is if you don't agree with the warning msg for better visibility21:15
cgoncalveshttp://logs.openstack.org/42/606142/1/check/openstack-tox-docs/d2b1279/html/contributor/guides/dev-quick-start.html21:15
johnsomI just think it's out of place in the quick-start guide given it is release specific, but I guess that is only on the queens branch so...21:16
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Add v2 two-node scenario test  https://review.openstack.org/60516321:28
openstackgerritCarlos Goncalves proposed openstack/octavia stable/queens: Add note to lower constraints for Jinja and pyOpenSSL  https://review.openstack.org/60614221:29
*** aojea has quit IRC21:41
*** fnaval has quit IRC21:47
cgoncalvesjohnsom, re: https://review.openstack.org/#/c/605163/ do you want 2 controller nodes or 1x controller+compute and 1x compute?21:55
cgoncalvesasking because "controller2" seems to be a compute node only21:56
cgoncalveswhy can't you use openstack-two-node from http://git.openstack.org/cgit/openstack-dev/devstack/tree/.zuul.yaml#n61 instead?21:56
cgoncalvesit is xenial21:57
openstackgerritAdam Harwell proposed openstack/octavia master: DNM: two dumb downstream things to fix, IGNORE ME  https://review.openstack.org/59398621:58
*** yamamoto has quit IRC22:00
*** yamamoto has joined #openstack-lbaas22:01
rm_workjohnsom: figured out a solution to my problem -- for now I am setting it so that function always returns 1.5 instead of making the call (as that should not impact anything else?) in my environment, until i can failover everything onto new amps22:03
*** yamamoto has quit IRC22:03
*** savvas_ has joined #openstack-lbaas22:05
openstackgerritAdam Harwell proposed openstack/octavia master: DNM: 3 dumb downstream things to fix, IGNORE ME  https://review.openstack.org/59398622:06
johnsomYeah, that would work22:09
openstackgerritAdam Harwell proposed openstack/octavia master: Experimental multi-az support  https://review.openstack.org/55896222:10
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: AZ Evacuation resource  https://review.openstack.org/55987322:10
*** yamamoto has joined #openstack-lbaas22:11
openstackgerritAdam Harwell proposed openstack/octavia master: WIP: Floating IP Network Driver (spans L3s)  https://review.openstack.org/43561222:14
savvas_johnsom: I managed to get around my networking problem. Changed my playbook a bit and recreated the lbaas interface, that did the trick. My instances boot now, but they terminate right a way22:17
johnsomNice22:18
savvas_http://paste.openstack.org/show/731205/ this is what I grab from the logs. Nova doesn't spit out any errors until it starts terminating the instance. Right before the qemu errors I catch this:22:18
savvas_http://paste.openstack.org/show/731207/ about the flavor,but when I check the Octavia config and the flavor list, it seems to match ids22:19
savvas_Any thoughts?22:19
openstackgerritAdam Harwell proposed openstack/octavia master: DNM: two dumb downstream things to fix, IGNORE ME  https://review.openstack.org/59398622:20
johnsomsavvas_ I have not seen that before.22:23
savvas_Great, leave it to me to find the good ones huh ;p22:24
johnsomTake a look at your qemu and libvirt logs22:24
johnsomYeah, you are definitively winning today.22:24
savvas_http://paste.openstack.org/show/731208/ just this22:28
johnsomsavvas_ You are looking for a qemu log like this one: http://logs.openstack.org/64/605264/1/check/octavia-v2-dsvm-scenario/7aa9e70/controller/logs/libvirt/qemu/instance-0000000a_log.txt.gz22:31
savvas_Ye browsed through those, lots of debug but no errors22:32
savvas_I may have an idea though, I see the default image that gets pulled is qcow222:32
savvas_going to build a custom image now22:33
*** threestrands has joined #openstack-lbaas22:41
*** celebdor has quit IRC22:44
*** rcernin has joined #openstack-lbaas22:49
savvas_Fixed johnsom22:56
johnsomOh good. Bad image somehow?22:58
savvas_it would be good to have a contingency in the playbook for os-octavia that checks whether or not qcow2 is supported in someone's environment22:58
savvas_well in my case I should've just paid better attention, kept going with the test image which is qcow2, but my environment runs on ceph storage22:58
johnsomYou don't support qcow2?  What do you use?22:58
rm_workRAW like RAX? :P23:00
johnsompublic cloud. Private uses qcow223:01
rm_workheh23:01
*** yamamoto_ has joined #openstack-lbaas23:05
*** yamamoto has quit IRC23:05
*** yamamoto_ has quit IRC23:08
*** yamamoto has joined #openstack-lbaas23:08
johnsomOh this is going to be a fun bug to fix: Keepalived[1259]: pid 4324 exited due to segmentation fault (SIGSEGV).23:08
rm_workO_o23:12
rm_workugh johnsom my failures from earlier testing were because one of our hypervisors was missing some vlan trunking23:26
rm_workso it just had no net, so nothing was coming up <_<23:26
rm_workand stuff kept hitting it23:27
rm_workthe patches all look good now23:27
johnsomlol, ok23:27
rm_workso throwing the new stuff into prod with my temp-fix, failing everything over today/tomorrow23:28
rm_workand then getting rid of that hack23:28
rm_worki hope no one else runs into this <_<23:28
rm_workonly people who used centos images generated mid-cycle23:28
« Sunday, 2018-09-30 Index Tuesday, 2018-10-02 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!