Tuesday, 2019-05-14

*** Vorrtex has quit IRC00:34
openstackgerritMerged openstack/octavia master: Remove v1 API and associated code  https://review.opendev.org/65670701:14
sapd1Please review boot amphora from volume patch. :)01:20
openstackgerritqian wang proposed openstack/octavia master: Modify README.rst and tox.ini Replace openstack.org URLs with opendev.org URLs  https://review.opendev.org/65894401:37
*** hongbin has joined #openstack-lbaas01:49
openstackgerritMerged openstack/octavia stable/stein: Bandit test exclusions syntax change  https://review.opendev.org/65876301:51
openstackgerritMerged openstack/octavia stable/rocky: Bandit test exclusions syntax change  https://review.opendev.org/65876401:51
openstackgerritMerged openstack/octavia master: Replace git.openstack.org URLs with opendev.org URLs  https://review.opendev.org/65504201:58
*** sapd1_x has joined #openstack-lbaas03:46
*** ricolin has joined #openstack-lbaas03:58
*** ivve has quit IRC04:10
*** yamamoto has joined #openstack-lbaas04:20
*** pcaruana|afk| has joined #openstack-lbaas04:25
*** hongbin has quit IRC04:29
*** pcaruana|afk| has quit IRC04:35
*** yamamoto has quit IRC05:11
*** ivve has joined #openstack-lbaas05:13
*** yamamoto has joined #openstack-lbaas05:13
openstackgerritmelissaml proposed openstack/octavia master: Rename review.openstack.org to review.opendev.org  https://review.opendev.org/65856505:29
*** vishalmanchanda has joined #openstack-lbaas05:35
*** ccamposr has quit IRC06:03
*** ccamposr has joined #openstack-lbaas06:03
*** gthiemonge has joined #openstack-lbaas06:31
cgoncalveswhat a wonderful morning! I wake up to see neutron-lbaas is no more :)06:38
*** gcheresh has joined #openstack-lbaas06:40
*** yboaron has joined #openstack-lbaas06:58
*** yboaron_ has joined #openstack-lbaas07:00
*** yboaron has quit IRC07:03
*** tesseract has joined #openstack-lbaas07:05
*** rcernin has quit IRC07:06
*** pcaruana has joined #openstack-lbaas07:12
cgoncalvesrm_work, you've earned the award of top neutron-lbaas contributor in lines of code category \o/07:20
cgoncalveshttps://www.stackalytics.com/?release=all&module=neutron-lbaas&metric=loc07:20
rm_workRofl07:22
rm_workGood cycle to get ptl guess 💪07:23
*** rpittau|afk is now known as rpittau07:32
*** yboaron_ has quit IRC07:35
*** yboaron_ has joined #openstack-lbaas07:35
*** ataraday has quit IRC07:45
*** yboaron_ has quit IRC08:05
*** luksky has joined #openstack-lbaas08:24
lukskyhi, I have octavia version 4.0.0 - octavia-health-manager eats all available memory, did You observe such behavior ?08:26
*** ataraday has joined #openstack-lbaas08:30
*** happyhemant has joined #openstack-lbaas08:31
*** yboaron_ has joined #openstack-lbaas08:36
rm_worki always forget what the version numbers are...08:41
rm_workis that rocky or stein?08:41
*** yboaron_ has quit IRC08:42
rm_workthere are some changes coming for optimizations for the HM, but i don't think it should ever be quite so bad08:42
rm_workluksky: firstly, what is "all memory"? is this like a 4GB VM, or a 128GB baremetal server?08:42
*** yboaron_ has joined #openstack-lbaas08:42
luksky4GB VM :)08:50
lukskyand there is only one LB created, with two members in pool08:50
lukskyit is taken from pip install, I assume it is stable stein08:51
zigoHi there!09:08
zigoI finally have Octavia working on my Debian setup.09:08
zigoI spent hours, days, weeks, etc. to find out what the final issue is.09:09
zigoFinally, I had this wrong directive in octavia.conf:09:09
zigo[certificates]/ca_certificates_file instead of ca_certificates09:09
zigoSo, octavia generates ca_certificates_file, but it doesn't seem like that what's needed.09:09
zigoIMO, that's a bug.09:09
zigoNow, I'm finalizing creating the Debian amphora...09:10
*** ricolin has quit IRC09:14
*** yamamoto has quit IRC09:35
*** yamamoto has joined #openstack-lbaas09:40
lukskyrm_work: did You encounter such behavior ?09:51
*** yamamoto has quit IRC09:58
rm_workluksky: it's possible that it could take that much, yes10:02
rm_workit does run a lot of threads by default10:02
rm_workyou could try reducing that (it is a config value) and see if it has a noticable effect on memory usage10:02
lukskyok, but on 2.0.1 it takes 'only' about 800-900 MB10:02
*** yamamoto has joined #openstack-lbaas10:02
lukskyok, will try thx10:02
*** yamamoto has quit IRC10:02
rm_workzigo: hmm, i'll look at that... but the config is generated using oslo directly from our config.py I think, so not sure how it'd be wrong10:03
*** yamamoto has joined #openstack-lbaas10:03
zigorm_work: Exactly, and that's what made me so confused...10:03
rm_workhmm, yeah, it should be ca_certificates_file10:03
rm_workwe don't reference just `ca_certificates` anywhere10:04
*** yamamoto has quit IRC10:07
zigorm_work: Well, using ca_certificates_file -> didn't work10:07
zigoUsing ca_certificates -> worked10:07
rm_workif that's the case, then removing it entirely should also work10:07
zigoWith _file, I had SSL_VERIFICATION_FAILED in octavia-worker.10:07
rm_workah yes10:08
rm_workthen removing it entirely would work10:08
rm_workall you've done is change it to an invalid variable name10:08
rm_workso it's not read10:08
zigoIn my PoC, I've set both variables (with puppet).10:08
rm_workand the default value (nothing) is working better than whatever you had configured :)10:08
rm_workjust delete that line entirely10:08
zigorm_work: BTW, I now have my Debian based image working.10:09
zigo:)10:09
zigorm_work: Here's the script: https://salsa.debian.org/openstack-team/debian/openstack-debian-images/tree/debian/stein/contrib/octavia10:09
zigoIt's a way simpler than the DIB thing...10:10
rm_workthe DIB thing is not exactly complicated... and that won't track upstream element changes10:10
zigorm_work: I hope to make it as an official Debian image at http://cdimage.debian.org/cdimage/openstack10:10
rm_workso that's a little concerning10:10
rm_workyou could do all of this by just slightly modifying the elements that exist to support debian10:11
zigorm_work: I just maintain an image regarding what's released in Debian, so that's OK.10:11
openstackgerritHanxiang Gao proposed openstack/octavia master: Blacklist bandit 1.6.0  https://review.opendev.org/65902010:11
zigorm_work: I don't want to use DIB, it's using PIP, and it's very ugly, I very much prefer using packages only.10:11
rm_workit doesn't have to use pip10:11
openstackgerritHanxiang Gao proposed openstack/octavia stable/rocky: Blacklist bandit 1.6.0  https://review.opendev.org/65902110:11
rm_workyou can just have it install debs with apt10:12
openstackgerritHanxiang Gao proposed openstack/octavia stable/stein: Blacklist bandit 1.6.0  https://review.opendev.org/65902210:12
rm_workthat's how RHEL works with it10:12
zigoI am using the Debian's "octavia-agent" package that I wrote.10:12
rm_workso why not have DIB install that?10:12
rm_workI'm not sure why you are convinced DIB only works with pup10:13
rm_work*pip10:13
zigorm_work: Because I don't like the way DIB works.10:13
zigoIt also would be a lot of work to get it convince Debian is ok. Everywhere, it only knows about Ubuntu...10:14
zigoExample from octavia/elements/haproxy-octavia/pre-install.d/01-backports:10:14
zigoif [ "$DISTRO_NAME" == "ubuntu" ] && [ "$DIB_RELEASE" == "trusty" ]; then10:14
zigo    echo deb http://archive.ubuntu.com/ubuntu trusty-backports main restricted universe multiverse > /etc/apt/sources.list.d/backports.list10:14
zigoelif [ "$DISTRO_NAME" == "centos7" ]; then10:14
zigo    cat > /etc/yum.repos.d/CentOS-PaaS.repo <<EOF10:14
rm_workDIB knows about debian as well10:14
rm_workwe just never set it up in octavia10:14
rm_workhttps://docs.openstack.org/diskimage-builder/1.25.2/elements/debian-minimal/README.html10:14
zigoResult: it's going to attempt installing a CentOS repository in my Debian image...10:15
rm_workno10:15
rm_workjust add the debian piece of that10:15
rm_workthat is not "DIB", that's just our usage of it10:15
zigoYeah, that's what I'm saying.10:15
zigoExactly.10:15
zigoAnd that's all over the place.10:15
rm_workso make it work for debian?10:15
rm_workthere's only a few elements that would need changes10:15
rm_workand most of them are simply additions to pkg-map10:16
zigoDIB is *not* the official Debian tool to create images.10:16
rm_worksure10:16
rm_workit's also not the official Ubuntu or RHEL way to create images10:16
zigoAnd my goal is targetting the official Debian images.10:16
rm_workbut it IS the official Octavia way to make images10:17
zigoDoes Canonical and Red Hat publish such an image?10:17
rm_workRed Hat does now10:17
zigoAh ok.10:17
zigo:)10:17
rm_workif you deviate from the officially supported method, it may work for you10:17
rm_workbut you're going to be on the hook to maintain it and track other upstream changes10:17
rm_workand I just don't understand why you'd want to deal with that10:18
zigoSure, will do !10:18
zigoIt's a simple diff -r -N in the elements folder ...10:18
zigorm_work: Because we already have openstack-debian-images hooked into the CD image creation, it'd be a lot more work to integrate DIB.10:19
rm_workI mean yeah, some work > no work I suppose10:19
zigoFor example, we have a list of source packages used to build the image, as soon as there's a security issue a new image is created.10:19
rm_workbut if we can get DIB to build debian images (which would honestly be fairly trivial) all you have to do is run a script and an image pops out the other side :D10:20
rm_workDIB uses officially published debian base images to build on10:20
rm_workso it'd already do that too?10:20
zigoAlso, we've automated the build of the source package, resulting in files like this one: http://cdimage.debian.org/cdimage/openstack/current/debian-9.9.0-openstack-amd64-source.tar.gz10:20
*** yamamoto has joined #openstack-lbaas10:20
rm_workand you can specify exactly which one you want to use with an env var10:20
rm_workif you don't want it to use the default base10:21
zigo(ie: a compilation of source packages used to build the image, as we want to ship source code too...)10:21
rm_workright, source for an amphora image would be `source_for_base.tar.gz + octavia_source`, which AFAIU should already be packaged and shipped?10:22
rm_worksince the amp agent source would be included in the same source tree as the octavia service package10:22
zigorm_work: BTW, I was wondering: is it ok to use a Stein amphora with a Rocky Octavia?10:23
rm_workyes, the amphora are always compatible in both directions10:23
zigoTo me, it looks like working, but I probably don't know enough...10:24
zigoOh, nice !10:24
rm_workIE, old amp works with new control-plane, and new amp works with old control-plane10:24
*** yamamoto has quit IRC10:24
*** yamamoto has joined #openstack-lbaas10:24
rm_workwe are very careful to maintain this compatibility, since both situations will happen *guaranteed* in a real deployment10:24
zigorm_work: How much of that is truth? How many releases? 1 or 2 ?10:24
rm_workAFAIK, all releases10:24
zigo:)10:24
zigoNice.10:24
rm_workI don't know that we've ever broken compatibility10:25
rm_workexcept possibly *within* a cycle, so not a release10:25
zigoGot ya.10:26
zigorm_work: Thanks a lot for your input.10:26
rm_workyeah, we appreciate the effort of getting stuff packaged up and released in debian10:27
rm_worki just wish i could convince you to let us help you get DIB working -- I can definitely assist with that if you decide you want to try, and I think we could probably get it pretty much done in a day or so10:28
rm_workthere's really not much that would need to be added10:28
rm_workbut I guess as long as something's published :)10:28
*** yamamoto has quit IRC11:03
*** lemko has joined #openstack-lbaas11:07
lukskyrm_work: decreasing number of threads from 50 to 20 solved issue - now I have on one controller 2300 MB, on second and third around 1700 MB usage (the same configuration)11:13
lukskyrm_work: what are best settings for threads ?11:14
lukskyI have one of LB stucked in PENDING_CREATE state11:42
lukskyis there any chance to delete it by openstack command ?11:42
luksky# openstack loadbalancer delete 1e255670-c442-4310-b7ed-aa901dc49f4011:42
lukskyInvalid state PENDING_CREATE of loadbalancer resource 1e255670-c442-4310-b7ed-aa901dc49f40 (HTTP 409) (Request-ID: req-0645e720-ebdb-4d5b-82e5-ac991aa33d57)11:42
*** yamamoto has joined #openstack-lbaas11:50
*** mkuf has joined #openstack-lbaas11:55
*** mkuf_ has quit IRC11:59
*** ccamposr has quit IRC12:02
*** yamamoto has quit IRC12:04
*** luksky has quit IRC12:10
*** gcheresh has quit IRC12:12
*** ccamposr has joined #openstack-lbaas12:18
*** ccamposr has quit IRC12:23
*** ccamposr has joined #openstack-lbaas12:25
*** boden has joined #openstack-lbaas12:26
*** mkuf_ has joined #openstack-lbaas12:29
*** mkuf has quit IRC12:32
*** yamamoto has joined #openstack-lbaas12:33
*** gcheresh has joined #openstack-lbaas12:50
*** luksky has joined #openstack-lbaas12:52
lukskyor only in database ?12:52
cgoncalvesluksky, only in the database12:53
openstackgerritAnn Taraday proposed openstack/octavia master: [WIP] Jobboard based controller  https://review.opendev.org/64740612:56
lukskyok, thx12:56
*** AlexStaf has quit IRC13:04
johnsomluksky: make sure the “streamer” settings are all disabled in the config. It is likely that rabbit code is eating your memory. (We just deleted it from master FYI)13:12
*** dmellado has quit IRC13:24
*** dmellado has joined #openstack-lbaas13:24
*** yamamoto has quit IRC13:28
*** yboaron_ has quit IRC14:04
*** lemko has quit IRC14:08
*** Vorrtex has joined #openstack-lbaas14:09
*** rpittau is now known as rpittau|afk14:15
openstackgerritMerged openstack/octavia-lib master: Do not install README.rst and LICENSE  https://review.opendev.org/64487414:20
*** yamamoto has joined #openstack-lbaas14:23
lukskyjohnsom: I commented out one option: #event_streamer_driver = queue_event_streamer14:28
lukskyYou were writing about that option ?14:29
johnsomluksky There are two places.  Give me a minute and I can give you references.14:41
lukskyjohnsom: ok, thx14:43
johnsomThat is the first one. It should be commented out or "noop_event_streamer". sync_provisioning_status should be commented out or false.14:43
johnsomI have seen events backup in the messaging system if those are enabled and there is no queue or receiver.14:44
*** boden has quit IRC14:51
johnsomFYI, Octavia helm charts blueprint: https://blueprints.launchpad.net/openstack-helm/+spec/openstack-helm-octavia15:16
*** fnaval has joined #openstack-lbaas15:27
*** fnaval has quit IRC15:27
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add a scenario test for spare pool  https://review.opendev.org/63498815:31
*** fnaval has joined #openstack-lbaas15:44
openstackgerritCarlos Goncalves proposed openstack/octavia master: Add Python 3.7 support  https://review.opendev.org/63523615:46
*** openstackgerrit has quit IRC15:54
*** openstackgerrit has joined #openstack-lbaas16:20
openstackgerritCarlos Goncalves proposed openstack/octavia-tempest-plugin master: Add a scenario test for spare pool  https://review.opendev.org/63498816:20
*** yamamoto has quit IRC16:24
*** yamamoto has joined #openstack-lbaas16:24
*** yamamoto has quit IRC16:30
*** yamamoto has joined #openstack-lbaas16:30
*** vishalmanchanda has quit IRC16:33
*** yamamoto has quit IRC16:36
*** yamamoto has joined #openstack-lbaas16:37
*** yamamoto has quit IRC16:37
*** yamamoto has joined #openstack-lbaas16:38
*** mlavalle has joined #openstack-lbaas16:40
*** yamamoto has quit IRC16:43
mlavalleHi. In my employers internal deployment (Rocky) I have to members behind a loadbalancer. I want to use the LB VIP to ssh to either of the two members. I can ssh directly to the members, so I know my security group rules are correct. Howver, when ssh'ing through the VIP I get: ssh_exchange_identification: Connection closed by remote host16:44
mlavalletwo members^^^16:44
mlavallehas this bahavior been seen upstream?16:45
openstackgerritCarlos Goncalves proposed openstack/octavia master: Cap to sphinx 2.0.0 on Python 2.7  https://review.opendev.org/65912316:49
cgoncalves^ as seen in http://logs.openstack.org/36/635236/16/check/requirements-check/37afa24/job-output.txt.gz#_2019-05-14_15_53_47_98070816:49
johnsommlavalle Hmmm, let me take a look at this.  I know for ssh you want to adjust the default listener timeouts as ssh is a longer-running protocol than HTTP. However this error does not sound like that issue.  It might be that you have ssh reverse lookups enabled in PAM or something like that.16:49
*** ccamposr has quit IRC16:51
*** luksky has quit IRC16:54
johnsommlavalle You might also have an issue where you are not using session persistence and the host ID is different between the two backends.  This would mean your "known_hosts" file stores the ID for the first member under the VIP address, then if you connect again the LB connects to host 2 and the  known_hosts key no longer matches.16:55
openstackgerritCarlos Goncalves proposed openstack/octavia master: Cap to sphinx 2.0.0 on Python 2.7  https://review.opendev.org/65912316:57
johnsommlavalle I would also check for hosts.allow  hosts.deny files on the SSH servers and make sure the LB from IPs are in there.16:59
*** boden has joined #openstack-lbaas17:00
mlavallejohnsom: thanks. s for your second comment, that would be in the case of a second connection, right?17:00
johnsommlavalle Right. You can "reset it" with ssh-keygen -R <VIP IP>17:01
johnsommlavalle I think you can test if the timeouts are too low with "ssh -o ServerAliveInterval=5 $HOST" which forces keepalive packets through the tunnel. Either way I know the defaults are too low for ssh sessions, they are tuned for HTTP traffic.17:07
mlavallejohnsom: ok, I'll take a look. I am also building an allinone devstack with master code to test the same scenario, to rule out something in our deployemnt17:09
openstackgerritCarlos Goncalves proposed openstack/octavia master: Add Python 3.7 support  https://review.opendev.org/63523617:11
openstackgerritMichael Johnson proposed openstack/octavia master: Add provider feature support matrix  https://review.opendev.org/65197417:11
johnsomI guess we are out of the post-PTG hibernation... lol17:11
cgoncalvesyeah! those rooms will not be missed https://media.giphy.com/media/euyLJkWydUslG/giphy.gif17:14
openstackgerritMerged openstack/octavia-lib stable/stein: Update UPPER_CONSTRAINTS_FILE for stable/stein  https://review.opendev.org/64215517:43
xgermancgoncalves: was there a train?18:15
*** luksky has joined #openstack-lbaas18:15
johnsomThe PTG rooms could be used for detention cells...  Gray, no windows, random loud fan sounds....18:16
xgermanthe fan was drainign out the train sound...18:19
*** ivve has quit IRC18:29
*** vishalmanchanda has joined #openstack-lbaas18:34
*** gcheresh has quit IRC18:44
*** goldyfruit has joined #openstack-lbaas18:49
*** pcaruana has quit IRC19:19
cgoncalvesmaster and stable/queens gate fixes: https://review.opendev.org/#/c/659123/, https://review.opendev.org/#/c/658765/19:33
*** yamamoto has joined #openstack-lbaas19:35
*** yamamoto has quit IRC19:40
*** gcheresh has joined #openstack-lbaas19:55
rm_workReady for review also:  https://review.opendev.org/#/c/657756/20:01
rm_workand https://review.opendev.org/#/c/657901/20:02
*** tesseract has quit IRC20:06
*** gcheresh has quit IRC20:07
*** Vorrtex has quit IRC20:15
rm_worksapd1: https://review.opendev.org/#/c/570505/ finally reviewed :)20:16
johnsomIs that passing the gate?20:16
rm_worki haven't looked for your new gate yet20:16
rm_workbut there's a ton of stuff that would be nice to get cleaned up before i even get that far20:17
johnsomThe init file needing a license was a pylint thing20:17
johnsomI think we switched most if not all over to have license in it20:17
johnsomIt was a lame pylint barf that was easier to just put a license in those files20:19
rm_workyeah i just noticed that and re-commented20:19
rm_workshould have checked before i posted those20:19
rm_workthe main thing is simple enough, which is that we don't need to use two different retry libs20:20
*** ivve has joined #openstack-lbaas20:36
*** vishalmanchanda has quit IRC20:44
openstackgerritAdam Harwell proposed openstack/octavia master: Force amp-agent communication to TLSv1.2  https://review.opendev.org/65790121:03
*** goldyfruit has quit IRC21:08
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Adds "unset" action to the pool command  https://review.opendev.org/65916521:10
*** gthiemonge has quit IRC21:13
*** gthiemonge has joined #openstack-lbaas21:13
*** boden has quit IRC21:23
*** gthiemonge has quit IRC21:41
*** gthiemonge has joined #openstack-lbaas21:42
openstackgerritMichael Johnson proposed openstack/python-octaviaclient master: Adds "unset" action to the pool command  https://review.opendev.org/65916521:45
*** ivve has quit IRC21:50
*** rcernin has joined #openstack-lbaas22:00
*** luksky has quit IRC22:01
*** fnaval has quit IRC22:01
*** goldyfruit has joined #openstack-lbaas22:02
*** yamamoto has joined #openstack-lbaas22:40
*** yamamoto has quit IRC22:45
*** happyhemant has quit IRC23:01
rm_workmugsie: hey https://review.opendev.org/#/q/owner:flux.adam%2540gmail.com+status:open+project:openstack/designate23:16
*** yamamoto has joined #openstack-lbaas23:26
openstackgerritMerged openstack/octavia master: Cap to sphinx 2.0.0 on Python 2.7  https://review.opendev.org/65912323:32
johnsomrm_work ignoring you in #openstack-dns?23:32
rm_worki don't have that channel :D23:33
rm_worki'm already in like 40 channels and he's here lol23:33
johnsomHe is in #openstack-gslb too.... lol23:33
*** mlavalle has quit IRC23:34
*** irclogbot_2 has quit IRC23:45
*** irclogbot_3 has joined #openstack-lbaas23:48

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!