Wednesday, 2018-07-11

« Tuesday, 2018-07-10 Index Thursday, 2018-07-12 »
openstackgerritMerged openstack/octavia master: Adding support for the octavia listener X-Forwarded-Proto header insertion.  https://review.openstack.org/57322700:26
*** longkb has joined #openstack-lbaas00:31
*** yamamoto has joined #openstack-lbaas00:40
*** yamamoto has quit IRC00:45
*** hongbin has joined #openstack-lbaas00:59
*** KeithMnemonic1 has quit IRC01:25
*** yamamoto has joined #openstack-lbaas01:42
*** yamamoto has quit IRC01:47
*** yamamoto has joined #openstack-lbaas01:51
openstackgerritJacky Hu proposed openstack/octavia master: Bring up lo when plugging interfaces  https://review.openstack.org/55086102:09
*** yamamoto has quit IRC02:10
*** yamamoto has joined #openstack-lbaas02:15
openstackgerritMichael Johnson proposed openstack/octavia master: Cleanup Octavia create VIP ports on LB delete  https://review.openstack.org/58116802:18
*** ramishra has joined #openstack-lbaas03:39
*** hongbin has quit IRC03:40
bzhao__johnsom: ping, hi michael,  https://review.openstack.org/#/c/529651/  could you please have a quick look if you are free? :) Thank you03:47
johnsombzhao__ I don't have much time, is there something specific you would like me to look at?03:50
openstackgerritMerged openstack/octavia master: Improve resource quota response message  https://review.openstack.org/57831803:52
*** ramishra has quit IRC03:59
bzhao__johnsom: sorry.  Thanks for leaving comments. So the ipv6 support we still  need to enable some kernel configuration. Let me google how to do.04:09
johnsomok, should be ip6tables too04:10
bzhao__yeah.04:10
bzhao__johnsom: errrr, the backend file resolver function need to change.. As I wrote based on ipv4...04:19
*** yboaron has joined #openstack-lbaas04:33
*** yboaron has quit IRC04:38
*** ramishra has joined #openstack-lbaas04:51
*** links has joined #openstack-lbaas05:03
*** yboaron has joined #openstack-lbaas05:34
*** velizarx has joined #openstack-lbaas06:15
*** kobis has joined #openstack-lbaas06:33
openstackgerritTatsuma Matsuki proposed openstack/octavia master: Separate the thread pool for health and stats update.  https://review.openstack.org/58158506:34
*** ispp has joined #openstack-lbaas06:39
*** kobis has quit IRC06:39
*** korean101 has joined #openstack-lbaas06:46
*** velizarx has quit IRC06:46
*** korean101 has left #openstack-lbaas06:46
*** kobis has joined #openstack-lbaas07:06
*** tesseract has joined #openstack-lbaas07:12
*** peereb has joined #openstack-lbaas07:17
*** velizarx has joined #openstack-lbaas07:18
*** nmanos has joined #openstack-lbaas07:18
*** ispp has quit IRC07:21
bzhao__is there anyone test lb with ipv6?　I'm afraid that whether it works. I debug and found the plug vip not work as wish.  Anyone hit the same issue?07:22
*** ispp has joined #openstack-lbaas07:25
*** rcernin has quit IRC08:03
*** rraja has joined #openstack-lbaas08:17
*** ktibi has joined #openstack-lbaas08:23
*** ivve has quit IRC08:37
bzhao__play lvs with iptables, if ipv4, we can set net.ipv4.vs.conntrack  , but  where is the one for ipv609:02
bzhao__I failed to google it for a while..09:03
bzhao__Any body knows?09:03
openstackgerritTatsuma Matsuki proposed openstack/octavia master: Separate the thread pool for health and stats update.  https://review.openstack.org/58158509:11
*** kobis has quit IRC09:38
*** yboaron has quit IRC09:38
*** issp has joined #openstack-lbaas09:38
*** ianychoi_ has joined #openstack-lbaas09:38
*** yboaron has joined #openstack-lbaas09:38
*** ispp has quit IRC09:41
*** ianychoi has quit IRC09:41
openstackgerritAllen proposed openstack/octavia master: Use openstack client command to replace nova client  https://review.openstack.org/58164609:45
openstackgerritAllen proposed openstack/octavia master: Add the missing markup for the hyperlink title  https://review.openstack.org/58165009:54
*** kobis has joined #openstack-lbaas09:57
bzhao__johnsom: errr,  I found enable net.ipv4.vs.conntrack, the ipv6 traffic also use that configuration....When I open it, it works as ipv4..10:00
bzhao__johnsom: OK, I will extend the udp for ipv6 support10:00
*** ianychoi_ has quit IRC10:05
openstackgerrityanpuqing proposed openstack/python-octaviaclient master: Add some filter options to load balancer list command  https://review.openstack.org/58032210:07
openstackgerritAllen proposed openstack/octavia-dashboard master: Add Apple OS X ".DS_Store" to ".gitignore" file  https://review.openstack.org/58165310:15
*** ianychoi has joined #openstack-lbaas10:30
*** longkb has quit IRC10:34
*** yamamoto has quit IRC11:01
*** atoth has quit IRC11:09
openstackgerritTatsuma Matsuki proposed openstack/octavia master: Separate the thread pool for health and stats update.  https://review.openstack.org/58158511:09
*** velizarx has quit IRC11:22
*** velizarx has joined #openstack-lbaas11:23
openstackgerrityanpuqing proposed openstack/python-octaviaclient master: Add some filter options to load balancer list command  https://review.openstack.org/58032211:24
*** ktibi has quit IRC11:25
*** ramishra has quit IRC11:32
*** ramishra has joined #openstack-lbaas11:44
*** yamamoto has joined #openstack-lbaas12:02
crazikHello,12:07
craziknew day, new questions ;)12:07
crazikI have an amphorae with 3 subnets connected in ACTIVE/STANDBY layout12:08
crazikwhen I login to the instance, I see only 1 IP address12:09
crazikwhat is the correct path to find the others?12:09
bzhao__please see the network namespace "amphora-haproxy".12:11
*** amuller has joined #openstack-lbaas12:12
cgoncalvescrazik, you're probably looking at the global namespace, hence seeing lb-mgmt network. "sudo ip netns exec amphora-haproxy ip a" to see all others12:12
crazikhm, no, I am looking inside amphora VM12:12
cgoncalvescrazik, yes, run that inside amphora VM12:12
*** atoth has joined #openstack-lbaas12:12
crazikoh12:13
crazikI didn't expected thtat12:13
crazikcgoncalves: thank you!12:14
*** yamamoto has quit IRC12:15
cgoncalvesyou're welcome12:16
crazikok, next question: I use some private address class for lb-mgmt-net, how to prevent users from using the same one?12:16
crazikfor example, let say i have 10.0.0.0/16, and LB user have the same12:17
*** ktibi has joined #openstack-lbaas12:17
cgoncalvescrazik, they would not conflict. lb-mgmt-net runs on global namespace and the rest on the amphora-haproxy net namespace12:21
crazikgood, great to hear that!12:21
crazikthank you again :)12:21
*** rraja has quit IRC12:32
*** rraja has joined #openstack-lbaas12:39
*** yamamoto has joined #openstack-lbaas12:50
*** openstack has joined #openstack-lbaas13:06
*** ChanServ sets mode: +o openstack13:06
*** KeithMnemonic has joined #openstack-lbaas13:18
*** nmanos has left #openstack-lbaas13:21
*** fnaval has joined #openstack-lbaas13:31
*** ramishra has quit IRC13:38
*** kobis has quit IRC13:38
*** kobis has joined #openstack-lbaas14:47
*** kobis has quit IRC14:52
*** ivve has joined #openstack-lbaas14:59
*** kobis has joined #openstack-lbaas15:05
*** links has quit IRC15:16
*** velizarx has quit IRC15:23
*** yamamoto has quit IRC15:33
johnsomThanks folks for helping with questions!15:36
openstackgerritMichael Johnson proposed openstack/octavia master: Cleanup Octavia create VIP ports on LB delete  https://review.openstack.org/58116815:38
*** sapd1 has joined #openstack-lbaas15:42
*** peereb has quit IRC15:46
kevkoguys, could you advice me why i have operating status offline for LB created ?15:47
cgoncalveskevko, what is the provisioning status you see?15:50
kevkocgoncalves: activ15:50
xgerman_mmh, if the health messages are not working it’s in OFFLINE15:53
cgoncalveskevko, ok. it could be that the health manager is not receiving health packets from the amphora15:53
cgoncalveskevko, make sure that the amphora is sending them (be default udp:5555 on eth0) and the health manager interface (by default o-hm0) is receiving them15:54
*** issp has quit IRC15:55
*** tesseract has quit IRC16:01
kevkocgoncalves: let me see16:12
kevkocheck16:12
kevkowhen tcpdumping  port 5555 i can see packets from amphora every 10 sec16:14
kevkocgoncalves: but maybe this is a issue , isn't it ?16:15
kevkocgoncalves: WARNING octavia.amphorae.backends.health_daemon.status_message [-] calculated hmac: c8cee43ce7580aa4fe006ade988d442001e85e1e7bf4692275d0fa1944342486 not equal to msg hmac: 6233303834366436323861333330323131616430653163383966376561363230 dropping packet16:15
cgoncalveskevko, correct. packets are being dropped16:16
kevkocgoncalves: before i will investigate it .. do you have some advice for me ? :)16:16
johnsomkevko Operating status is the observed status. Is the member server down or not reachable (security group)?16:16
johnsomkevko I think you have a mis-matched image.  We made a change (which I think we need to re-think) that changes how the hmac is handled.  I'm guessing the controller version is older than the image version.16:18
kevkojohnsom: so, firstly i should regenerate image ...16:19
johnsomhttps://review.openstack.org/#/c/571333/16:19
johnsomThat is the series of patches16:19
johnsomkevko Well, no, your image is new, it's the controller that seems to be an older version16:20
kevkojohnsom: ok.. let me try it16:20
kevkojohnsom: can i only ask why so old fix from 2016 is not in queens ? i mean released version ..not branch16:29
kevkojohnsom: just trying to understand the process16:29
johnsomWhich old fix?16:29
kevkojohnsom: oh, sorry , my fault ... i was looking on other monitor :D16:30
kevkojohnsom: sorry :D16:30
*** yamamoto has joined #openstack-lbaas16:33
*** kobis has quit IRC16:37
*** velizarx has joined #openstack-lbaas16:37
*** rraja has quit IRC16:38
*** yamamoto has quit IRC16:39
sapd1kevko: : I have some WARNING log with you.17:04
*** sapd1 has quit IRC17:04
kevkosapd: what ?17:04
*** velizarx has quit IRC17:13
*** yamamoto has joined #openstack-lbaas17:35
*** yamamoto has quit IRC17:41
kevkojohnsom: you were correct ...status is now working as expected and have online status in dashboard18:13
cgoncalveskevko, interesting case you have: control plane services version older than amphora (e.g. services on Queens while amphora based on Rocky)18:28
cgoncalvesjohnsom, I think this warrants an upgrade note18:29
kevkocgoncalves: i have one more issue ..18:30
johnsomcgoncalves It might warrant a revert18:30
kevkocgoncalves: but now trying to find issue myself firstly :D18:30
*** yamamoto has joined #openstack-lbaas18:37
*** yamamoto has quit IRC18:43
kevkoguys ..can i recheck review even if it is abandoned ?18:53
cgoncalveskevko, not sure it will trigger CI. try, there's no harm anyway18:57
*** harlowja has joined #openstack-lbaas19:00
openstackgerritGerman Eichberger proposed openstack/octavia master: Add a config to surpress amphora logging  https://review.openstack.org/56674119:07
*** atoth has quit IRC19:14
*** amuller has quit IRC19:27
*** ktibi has quit IRC19:34
*** yamamoto has joined #openstack-lbaas19:40
*** yamamoto has quit IRC19:45
johnsom#startmeeting Octavia20:00
openstackMeeting started Wed Jul 11 20:00:04 2018 UTC and is due to finish in 60 minutes.  The chair is johnsom. Information about MeetBot at http://wiki.debian.org/MeetBot.20:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.20:00
*** openstack changes topic to " (Meeting topic: Octavia)"20:00
openstackThe meeting name has been set to 'octavia'20:00
johnsomHi folks!20:00
cgoncalveshi20:00
johnsomI always worry that I start the meeting in the right channel... lol20:00
xgerman_o/20:00
johnsom#topic Announcements20:00
*** openstack changes topic to "Announcements (Meeting topic: Octavia)"20:00
johnsomSame story, new week, we have a priority bug list for Rocky20:01
johnsom#link https://etherpad.openstack.org/p/octavia-priority-reviews20:01
johnsomWe have made a lot of progress!20:01
johnsomBut more work to do....20:01
johnsomAlso a reminder, python-octaviaclient needs it's final Rocky release by the 26th20:02
johnsomAnd finally: Berlin summit presentation deadline is July 18th20:02
johnsomSo if you want to submit a talk for Berlin, you have a week to do it.20:03
johnsomAny other announcements today?20:03
xgerman_if you need soccer scores…20:03
johnsomSigh, I am so buried with work right now I haven't even had lunch yet let alone see who is playing...20:04
johnsom#topic Brief progress reports / bugs needing review20:04
*** openstack changes topic to "Brief progress reports / bugs needing review (Meeting topic: Octavia)"20:04
johnsomI have been working on a bunch of stuff.20:05
johnsomFixed the connection limit issue, added a port cleanup that was missing from the provider driver setup, testing UDP patches,20:06
johnsomCurrently I am focused on internal stuff, but will be starting work on an Active/Standby tempest test.20:06
johnsomI started looking at the flavors patch and getting the API reference parts in, but there is more work to do there. I'm not sure it will land for Rocky20:07
johnsomAny other updates today?20:08
xgerman_I have been putting some cycles into privsep but not sure if I will finish it in Rocky20:08
cgoncalvesno much from my side. backporting bug fixes to stable branches and some other integration and CI work in TripleO land20:09
johnsomOh, and I wrote a test gate for the migration tool. That was way more work than it's worth, but...  So many Ansible bugs/issues20:09
johnsomThanks for staying on top of the backports.20:09
johnsomI would like to see this merge so we can start backporting it: https://review.openstack.org/57734420:10
cgoncalveswe have to apologize for the asking for a test gate. we probably underestimated the effort required20:11
johnsomOh no worries. it's done now. maybe the code will be useful in the future if we have a need for a "special" gate test.20:12
cgoncalveshow can we easily reproduce the issue?20:12
johnsomI'm just grumpy about it because it took ~50 patches and a week to make a test run....20:13
cgoncalvesdeleting the two amphorae instances would do?20:13
johnsomYes, just nova delete both amps at the same time.20:13
johnsomwithout the patch the LB will go down in a ball of flames20:13
cgoncalvesok. I'll try20:14
johnsomwith the patch, it will "do the right thing" and repair the LB and both amps20:14
johnsomIt's a nasty bug that I know people have hit, that is why I want to make sure we get it backported20:15
johnsom#topic Versioning patch20:15
*** openstack changes topic to "Versioning patch (Meeting topic: Octavia)"20:15
johnsom#link https://review.openstack.org/55946020:15
johnsomI have put up another proposal for versioning our API and communicating that.  Please have a look and comment on this approach20:16
johnsomAs discussed before, I switched the endpoint to be /v2 with an alias for the old /v2.0 path20:16
johnsomAny other discussion on that or just review comments?20:17
johnsom#topic HMAC hexdigest patch20:18
*** openstack changes topic to "HMAC hexdigest patch (Meeting topic: Octavia)"20:18
johnsom#link https://review.openstack.org/#/c/571333/20:18
johnsomIt has come to my attention that we have a compatibility issue with this change20:18
johnsomIt has also be backported, so even more sad face20:19
xgerman_but not released? silver lining?20:19
johnsomWe have backward compatibility if the control plane is updated, but the amp is an old version.20:19
johnsomBut, if the amp is new and the control plane is old, the amps get ignored and failed over.20:20
johnsomI see a few options here:20:20
johnsom1. revert the patches20:20
xgerman_mmh, I am ok with just documenting - this is an odd case (why would you not update amps with control plane)20:21
johnsom2. Add a release note warning the deployer the order things need updating20:21
johnsom3. Throw up our hands and run for the hills20:21
* johnsom thinks 3 sounds good about now20:22
xgerman_CRO:ENG 2:120:22
johnsomYeah, it is an odd case. I might be ok with adding release notes if you all think that is an ok upgrade requirement20:22
cgoncalves3 sounds exhausting. climbing hills is not easy, but the price we may have to pay20:22
xgerman_I am fine with (2)20:23
xgerman_#vote ?20:23
johnsomDo we have enough people for that? grin Seems quiet today20:24
cgoncalveswhat would be an use case where the operator would upload new amp and keep same control plane version?20:24
cgoncalvesI can think of 1 or 2 maybe20:24
johnsomAt least our handy upgrade guide points out the right procedure20:25
johnsom#link https://docs.openstack.org/octavia/latest/admin/guides/upgrade.html#cold-upgrade20:25
johnsomYeah, I think it's really people that are grabbing the nightly build image and dropping it into a cloud that doesn't have the updated control plane20:26
johnsomThe issue will likely go away once we do a stable branch release that includes it20:26
cgoncalvesI'd need to think a bit more. maybe it's not something one would face in a normal day 2 using tripleo at least20:27
johnsomOk, I'm leaning towards #2 and adding a release note20:28
cgoncalves+120:28
johnsomOk, I will do that.20:28
johnsom#topic Open Discussion20:28
*** openstack changes topic to "Open Discussion (Meeting topic: Octavia)"20:28
cgoncalvesthanks20:28
johnsomThat is all I had on the agenda for today. Any other items?20:28
johnsomOk, thanks folks!  I'm going to go make lunch...20:29
johnsom#endmeeting20:29
*** openstack changes topic to "Discussion of OpenStack Load Balancing (Octavia) | https://etherpad.openstack.org/p/octavia-priority-reviews"20:29
openstackMeeting ended Wed Jul 11 20:29:54 2018 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)20:29
openstackMinutes:        http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-07-11-20.00.html20:29
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-07-11-20.00.txt20:29
openstackLog:            http://eavesdrop.openstack.org/meetings/octavia/2018/octavia.2018-07-11-20.00.log.html20:29
cgoncalvesso a unstack.sh, clean.sh followed by pip uninstall all, rm -rf /opt/stack ~/.cache/{dib,image-create} didn't help resolving the issue of yesterday (build of ubuntu amp image) :/20:40
*** yamamoto has joined #openstack-lbaas20:41
johnsomHmnmm (BTW, you don't need unstack if you are running clean.sh)20:42
johnsomcgoncalves Is there something odd with the permissions on /tmp?20:46
johnsomhttps://talk.plesk.com/threads/problem-with-apt-ubuntu-16-04.341306/20:46
johnsomOr whatever location you "TEMP" is20:47
*** yamamoto has quit IRC20:47
johnsomhttps://bugs.debian.org/cgi-bin/bugreport.cgi?bug=80794820:48
openstackDebian bug 807948 in apt "apt: 'update' fails with 'Couldn't create tempfiles for splitting up' InRelease files" [Important,Open]20:48
johnsomhttps://github.com/googledatalab/datalab/issues/126320:49
johnsomRunning with Docker by chance? ^^^20:50
cgoncalvesjohnsom, also found that bug report. isn't the image built in a chroot or something? if so /tmp would not be my localhost:/tmp20:54
johnsomcgoncalves It is, but there are phases that are outside the chroot: https://docs.openstack.org/diskimage-builder/latest/developer/developing_elements.html#phase-subdirectories20:55
cgoncalvesand my /tmp is 777 so...20:55
johnsomYou are failing in a root.d phase, which is outside the chroot20:55
openstackgerritGerman Eichberger proposed openstack/octavia master: [WIP] Switch amphora agent to use privsep  https://review.openstack.org/54929521:03
openstackgerritCarlos Goncalves proposed openstack/octavia master: Correct naming for quota resources  https://review.openstack.org/55967221:31
*** yamamoto has joined #openstack-lbaas21:43
*** yamamoto has quit IRC21:49
*** rcernin has joined #openstack-lbaas22:15
*** yboaron has quit IRC22:25
*** fnaval has quit IRC22:29
openstackgerritMichael Johnson proposed openstack/octavia master: Add release note for HMAC python3 fix  https://review.openstack.org/58190222:37
openstackgerritCarlos Goncalves proposed openstack/octavia master: Add release note for HMAC python3 fix  https://review.openstack.org/58190222:45
*** yamamoto has joined #openstack-lbaas22:45
johnsomlol22:45
cgoncalves"release the beast!" :)22:46
*** yamamoto has quit IRC22:51
*** harlowja has quit IRC23:05
*** KeithMnemonic has quit IRC23:07
*** yamamoto has joined #openstack-lbaas23:47
*** yamamoto has quit IRC23:52
« Tuesday, 2018-07-10 Index Thursday, 2018-07-12 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!