Monday, 2019-04-01

*** sapd1_x has joined #openstack-lbaas01:03
*** yamamoto has joined #openstack-lbaas02:02
*** hongbin has joined #openstack-lbaas02:05
*** ricolin has joined #openstack-lbaas02:07
*** sapd1_x has quit IRC02:15
*** irclogbot_0 has quit IRC02:19
*** yamamoto has quit IRC03:04
*** psachin has joined #openstack-lbaas03:17
*** ramishra has joined #openstack-lbaas03:24
*** mkuf has joined #openstack-lbaas03:34
*** yamamoto has joined #openstack-lbaas03:35
*** hongbin has quit IRC04:17
*** psachin has quit IRC05:09
*** psachin has joined #openstack-lbaas05:15
*** yamamoto has quit IRC05:16
*** yamamoto has joined #openstack-lbaas05:17
*** ivve has joined #openstack-lbaas05:22
*** ianychoi has quit IRC05:23
*** ianychoi has joined #openstack-lbaas05:24
*** gthiemonge has quit IRC05:36
*** psachin has quit IRC05:57
*** psachin has joined #openstack-lbaas06:06
*** ccamposr has joined #openstack-lbaas06:24
*** sapd1_x has joined #openstack-lbaas06:36
*** sapd1_x has quit IRC06:41
*** gcheresh has joined #openstack-lbaas06:50
*** rcernin has quit IRC06:58
*** pcaruana has joined #openstack-lbaas06:58
*** pcaruana has quit IRC07:02
*** pcaruana has joined #openstack-lbaas07:02
*** psachin has quit IRC07:08
*** psachin has joined #openstack-lbaas07:16
*** luksky has joined #openstack-lbaas07:23
*** rpittau|afk is now known as rpittau07:24
*** celebdor has joined #openstack-lbaas07:30
*** gthiemonge has joined #openstack-lbaas07:49
*** Dinesh_Bhor has quit IRC07:59
*** Dinesh_Bhor has joined #openstack-lbaas08:03
*** happyhemant has joined #openstack-lbaas08:04
openstackgerritCarlos Goncalves proposed openstack/octavia stable/rocky: Fix the loss of access to barbican secrets  https://review.openstack.org/63902008:37
*** gthiemonge has quit IRC08:39
*** gthiemonge has joined #openstack-lbaas08:49
*** ccamposr has quit IRC09:01
*** ccamposr has joined #openstack-lbaas09:01
*** ccamposr has quit IRC09:04
*** ccamposr has joined #openstack-lbaas09:05
*** velizarx has joined #openstack-lbaas09:09
*** velizarx has quit IRC09:11
*** salmankhan has joined #openstack-lbaas09:26
*** velizarx has joined #openstack-lbaas09:28
*** sapd1_x has joined #openstack-lbaas09:32
*** ramishra has quit IRC09:40
*** ramishra has joined #openstack-lbaas09:43
*** ramishra_ has joined #openstack-lbaas09:48
*** ramishra has quit IRC09:50
*** yamamoto has quit IRC09:59
*** ramishra has joined #openstack-lbaas10:07
*** ramishra_ has quit IRC10:10
*** celebdor has quit IRC10:12
*** luksky has quit IRC10:17
*** yamamoto has joined #openstack-lbaas10:27
*** sapd1_x has quit IRC10:28
*** yamamoto has quit IRC10:32
*** ivve has quit IRC10:35
*** ramishra has quit IRC10:38
*** ramishra has joined #openstack-lbaas10:40
*** yamamoto has joined #openstack-lbaas10:42
*** yamamoto has quit IRC10:46
*** yamamoto has joined #openstack-lbaas10:49
*** gcheresh_ has joined #openstack-lbaas11:05
*** gcheresh has quit IRC11:05
*** gcheresh_ has quit IRC11:10
*** gcheresh has joined #openstack-lbaas11:10
*** ivve has joined #openstack-lbaas11:25
*** luksky has joined #openstack-lbaas11:26
openstackgerritMerged openstack/octavia stable/queens: Fix LB failover when in ERROR  https://review.openstack.org/64374011:35
openstackgerritMerged openstack/octavia stable/rocky: Fix the loss of access to barbican secrets  https://review.openstack.org/63902011:51
*** boden has joined #openstack-lbaas11:53
openstackgerritMerged openstack/octavia stable/rocky: Replace openstack.org git:// URLs with https://  https://review.openstack.org/64670012:06
*** trown|outtypewww is now known as trown12:12
*** yamamoto has quit IRC12:17
*** rcernin has joined #openstack-lbaas12:19
*** yamamoto has joined #openstack-lbaas12:19
*** ramishra has quit IRC12:25
*** celebdor has joined #openstack-lbaas12:25
*** ramishra has joined #openstack-lbaas12:30
*** gthiemonge has quit IRC12:32
*** gthiemonge has joined #openstack-lbaas12:34
*** yamamoto has quit IRC12:47
*** ricolin has quit IRC12:50
*** gthiemonge has quit IRC12:52
*** gthiemonge has joined #openstack-lbaas12:54
*** gthiemonge has quit IRC13:02
*** gthiemonge has joined #openstack-lbaas13:04
*** gthiemonge has quit IRC13:13
*** yamamoto has joined #openstack-lbaas13:14
*** goldyfruit has joined #openstack-lbaas13:19
*** irclogbot_3 has joined #openstack-lbaas13:26
*** rcernin has quit IRC13:29
*** gthiemonge has joined #openstack-lbaas13:31
*** ramishra has quit IRC13:41
*** ramishra has joined #openstack-lbaas13:56
*** salmankhan has quit IRC14:02
cgoncalvescores, backport patches pending review: https://review.openstack.org/#/q/(branch:stable/rocky+OR+branch:stable/queens+)+AND+(project:openstack/octavia+OR+project:openstack/octavia-dashboard+OR+project:openstack/python-octaviaclient+OR+project:openstack/octavia-tempest-plugin)+AND+status:open+AND+NOT+label:Code-Review%253C0+AND+NOT+label:Verified%253C%253D0+AND+NOT+label:Workflow%253C014:03
cgoncalvesgoal is to do maintenance releases once we merge them all14:04
*** fnaval has joined #openstack-lbaas14:08
*** gcheresh has quit IRC14:11
cgoncalvesdayou_, thanks!14:16
openstackgerritMerged openstack/octavia-dashboard stable/stein: Add nodejs10 jobs on bionic.  https://review.openstack.org/64882914:16
dayou_cgoncalves: my pleasure!14:17
openstackgerritGregory Thiemonge proposed openstack/neutron-lbaas stable/queens: Fix proxy extension for neutron RBAC  https://review.openstack.org/64904814:30
*** gcheresh has joined #openstack-lbaas14:33
openstackgerritGregory Thiemonge proposed openstack/neutron-lbaas stable/rocky: Fix proxy extension for neutron RBAC  https://review.openstack.org/64904914:33
dulekcgoncalves: Hey! Was Octavia's master broken recently? We see failures on Kuryr-Kubernetes jobs and tracked this to an LB not passing traffic.14:34
cgoncalvesdulek, no14:35
cgoncalveswhat is the problem you're seeing?14:35
cgoncalvesI mean, traffic not passing but what is the scenario?14:35
cgoncalvesany errors or uncommon warnings?14:36
*** gcheresh has quit IRC14:37
dulekcgoncalves: Nothing really. I haven't reproduced that locally, so don't want to ask for help yet. ;)14:37
cgoncalvesuh, so independent! :P14:37
dulekcgoncalves: It just might be our fault. ;)14:40
*** ivve has quit IRC15:06
*** Vorrtex has joined #openstack-lbaas15:09
*** luksky has quit IRC15:23
*** sapd1_x has joined #openstack-lbaas15:25
*** gcheresh has joined #openstack-lbaas15:26
*** velizarx has quit IRC15:26
openstackgerritMargarita Shakhova proposed openstack/octavia master: Support create amphora instance from volume based.  https://review.openstack.org/57050515:33
openstackgerritMargarita Shakhova proposed openstack/octavia master: Support create amphora instance from volume based.  https://review.openstack.org/57050515:34
*** ccamposr has quit IRC15:44
*** velizarx has joined #openstack-lbaas15:48
*** gcheresh has quit IRC15:59
VorrtexHey johnsom !  I hope you remember me, Trevor V from Rackspace back in the day!  Forgive me for getting straight to the point, but I was hoping I could bother you to direct me at someone with experience concerning the diskimage-create.sh script for octavia!16:05
johnsomVorrtex: hey Trevor, long time no chat. I wrote that, so you came to the right place.16:08
VorrtexI definitely didn't check the annotations in pycharm or anything.16:10
*** velizarx has quit IRC16:10
Vorrtexjohnsom so, I'm trying to write an addition to it for building ppc64le images as amphora images, but having trouble finding the right spot to add "where to find the base image"16:11
johnsomLol16:11
*** rpittau is now known as rpittau|afk16:13
johnsomWell, diskimage-create is a wrapper on diskimage-builder. First step is to make sure DIB supports it. If so, you should be able to update the architecture options in my script to include it16:13
VorrtexSo the immediate problem I'm facing is when it looks up a cloud image for centos on ppc64le, which I have locally on the machine I'm trying to build the amphora on, but it searches online for it, and it doesn't exist there.16:16
johnsomVorrtex https://docs.openstack.org/diskimage-builder/latest/elements/centos7/README.html16:17
cgoncalvesyou want to set DIB_LOCAL_IMAGE16:18
johnsomWell, the readme says DIB_CLOUD_IMAGES, but yes, you need to export one of those with the path to the image file.16:18
*** gthiemonge has quit IRC16:18
johnsomSeems as though that was written for ppc64le... lol16:18
VorrtexSo... Um... Yeah... thanks... hah16:18
* Vorrtex is bashful because he probably should have found this himself.16:19
cgoncalvesI wouldn't have met you so... ;)16:20
johnsomNo worries, OpenStack is sprawling. It helps to get pointers.16:21
*** velizarx has joined #openstack-lbaas16:23
xgermanVorrtex: hi, glad to find you back jere16:24
Vorrtexxgerman hey hey!  How are ya?16:25
xgermanpretty well -16:25
xgermanwhat about you?16:26
VorrtexI'm not doin too shabby, honestly.  I could always be better, but nothing to complain about.16:28
xgermannice!16:29
VorrtexYessir!  I see octavia has been kept up, which is awesome.  Glad to have been a part of it ha ha16:30
*** sapd1_x has quit IRC16:30
xgermanyep, it’s going strong… and we now have the wise leadership of rm_work our new PTL16:31
*** velizarx has quit IRC16:39
Vorrtexrm_work is the PTL?  Impressive!  He didn't mention anything except saying that you xgerman were/are PTL at some point... I don't recall exactly when he said that though ha ha.16:40
*** velizarx has joined #openstack-lbaas16:40
xgermanwell, I ws it for a cycle or two, then johnsom for many years, and now Adam16:40
*** ramishra has quit IRC16:42
*** velizarx has quit IRC16:43
openstackgerritMerged openstack/octavia stable/rocky: Fix initialization of Barbican client  https://review.openstack.org/64885716:43
*** trown is now known as trown|lunch16:49
openstackgerritMerged openstack/octavia stable/stein: Fix initialization of Barbican client  https://review.openstack.org/64885617:01
*** velizarx has joined #openstack-lbaas17:03
*** velizarx has quit IRC17:05
*** velizarx has joined #openstack-lbaas17:17
colin-having connection issues through a UDP LB, have a tcpdump examining UDP traffic in the amphora-haproxy namespace and can see slight changes in the traffic when the timeouts/disconnects occur but am not really sure what's contributing to them. when i connect directly to the backend i don't get the same behavior17:28
colin-any tips for checking out what keepalivedlvs might be struggling with, if anything?17:29
*** zigo has quit IRC17:37
johnsomWhat UDP protocol are you putting through it?17:41
colin-proto UDP (17)17:42
johnsomQuote of the Day?17:42
colin-in this particular example it's a docker container running the game factorio that i'm connecting to17:42
*** yamamoto has quit IRC17:44
johnsomAh, I was asking about what was on top of UDP. So it's something proprietary. ok.17:44
colin-i examined the config it generates for keepalived and it _looks_ appropriate but i'm not super familiar with what i'm reading to know if anything i might want is absent17:44
johnsomIt's been a little while since I looked at the UDP stuff. Let me refresh my mind17:46
colin-no rush, just sounding out my troubleshooting :) thx17:46
johnsomSo the stream starts, but aborts at some point?17:47
colin-the only correlation i have right now is the length drops on the UDP traffic going over the amp when the client stops responding17:47
colin-the behavior is the client rubber banding when moving and occassionally getting popups about connectivity issues that always self-resolve in a second or two17:47
colin-when i bypass the amp these are not evident17:48
johnsomDo you have persistence enabled?17:49
colin-remind me which object that is set on? listener?17:49
johnsompool17:50
colin-duh, concerns backends17:50
colin-session_persistence | None17:50
colin-so maybe set that?17:51
*** velizarx_ has joined #openstack-lbaas17:51
johnsomOnly if you need a client to go back to the same backend server17:52
colin-well in this case there is only one pod17:52
*** velizarx has quit IRC17:52
colin-it is a kubernetes cluster though with kube-proxy relaying connections between hosts, but i know we typically wave our hands at that stuff17:52
johnsomYeah, the whole k8s pat stuff adds a whole additional layer of potentially broken.17:53
colin-that's one possibility17:53
colin-so far it seems fairly happy with the LB object and its corresponding services and endpoints17:53
johnsomSo, on the amp side, check the syslog and dmesg. The UDP flows are all handled in the kernel.17:54
johnsomTo dig into what LVS is doing, you can use the ipvsadm tool17:54
colin-so with the UDP LB being a relatively new endeavor for us and this being the first time i've tested it with anything that runs at all 7 layers i wanted to be able to speak about what it's doing intelligently17:54
colin-ok perfect17:54
johnsomThe other thing to check is that you haven't set your max connections to a level too low for your needs17:55
*** gthiemonge has joined #openstack-lbaas17:55
johnsomYeah, the UDP is pretty straight forward, it's a simple packet in, packet out with NAT.17:56
openstackgerritMargarita Shakhova proposed openstack/octavia master: Support create amphora instance from volume based.  https://review.openstack.org/57050517:58
*** trown|lunch is now known as trown18:04
*** velizarx has joined #openstack-lbaas18:05
*** velizarx_ has quit IRC18:07
*** yamamoto has joined #openstack-lbaas18:19
*** yamamoto has quit IRC18:26
openstackgerritMichael Johnson proposed openstack/neutron-lbaas master: Remove the references to oslosphinx  https://review.openstack.org/64916018:56
colin-i didn't expect it to, but session persistence had no discernable impact. haven't found anything in dmesg or journal suggesting any problem from LVS'18:58
colin-s perspective18:58
colin-looks healthy and i see the rr rule on the amps as i'd expect18:58
colin-going to examine timing next though i thought it would ignore most timeout values18:59
xgermanYeah, I would jump to tcpdump pretty soon to see what’s up :-)19:13
*** velizarx has quit IRC19:19
*** velizarx has joined #openstack-lbaas19:28
*** velizarx has quit IRC19:29
*** luksky has joined #openstack-lbaas19:35
colin-ip netns exec amphora-haproxy tcpdump -n udp dst port <my port>, so i grabbed the contents of this while i experienced a disconnect but don't see anything that really stands out. no errors or anything obviously unhealthy in wireshark20:23
colin-any suggestions for changes to my capture to get more info?20:23
johnsomNot really. UDP is pretty straight forward, it doesn't have a session or anything.20:25
colin-how can i set session_persistence on a pool back to null :)?20:26
johnsomYeah, this is an area we need to work on. If you update it with null or none it might take it.20:26
johnsomWe have some bugs with the "unset" features20:26
colin-loadbalancer pool set --session-persistence type=null20:27
colin-like this?20:27
johnsomyes, I think so20:27
*** gthiemonge has quit IRC20:28
*** jlaffaye has quit IRC20:32
*** jlaffaye has joined #openstack-lbaas20:37
Vorrtexjohnsom I'm getting a "curl: (3) <url> malformed" error when trying to fetch the base image... I have this set:  declare -x DIB_CLOUD_IMAGES="/root/CentOS-7-ppc64le-GenericCloud-201606-r1.qcow2.xz"20:39
VorrtexSo I'm not sure what's wrong.20:39
johnsomI think the docs said it's a URL, so file://20:39
openstackgerritMerged openstack/octavia stable/rocky: Fix prefix for vip_ipv6  https://review.openstack.org/63310520:41
*** trown is now known as trown|outtypewww20:42
Vorrtexjohnsom alright, will try that next, but the example didn't show that: https://docs.openstack.org/diskimage-builder/latest/elements/centos7/README.html20:44
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Fix octavia-tempest-plugin releasenotes theme  https://review.openstack.org/64918520:46
johnsomVorrtex you probably want to export and not use declare. It might not survive the layers of bash scripts20:47
rm_workcolin-: i appreciate that your test setup is using Factorio tho :D20:47
Vorrtexha ha ha, it is exported, just I did "export | grep qcow" to grab exactly what was set when I was posting my info above.20:47
rm_workheya Vorrtex o/20:47
Vorrtexrm_work how goes it, PTL?  :P20:48
rm_worknot quite yet, only after johnsom gives it up at the summit :D20:48
johnsomYou can have it now....  grin20:49
rm_worknooooope20:49
johnsomVorrtex So, https://github.com/openstack/diskimage-builder/blob/master/diskimage_builder/elements/centos7/root.d/10-centos7-cloud-image#L18 This implies it is a URL and you need to make sure you are also setting the ARCH20:49
johnsomCan't say I didn't try....20:50
Vorrtexarch is set already, johnsom , just working on getting this base image sorted.. Now I'm having the fun issue here:20:50
Vorrtexcurl: (37) Couldn't open file /root/CentOS-7-ppc64le-GenericCloud-201606-r1.qcow2.xz/CentOS-7-ppc64le-GenericCloud.qcow2.xz20:51
Vorrtexlol....20:51
VorrtexSo its taking the image I've made, and appending the image its auto-completing.20:51
rm_workuhh, is it maybe supposed to be a path to a directory of images? O_o not sure, never used it20:52
VorrtexI'm just gonna change its name to what it expects, and remove its name from the export, see where that get sme.20:52
Vorrtexgets me***20:52
rm_workbut i would have assumed you'd specify the image explicitly20:52
Vorrtexassumptions are fun, aren't they rm_work ;)20:53
VorrtexI make assumptions all day... most of the wrong, but hey, at least I assumed.20:53
Vorrtexmost of them***20:53
colin-somethign to be said for a well maintained docker hub project rm_work :) plus their client runs on everything21:00
*** ivve has joined #openstack-lbaas21:02
*** abaindur has quit IRC21:09
*** ivve has quit IRC21:09
*** abaindur has joined #openstack-lbaas21:09
openstackgerritMichael Johnson proposed openstack/octavia-tempest-plugin master: Fix octavia-tempest-plugin releasenotes theme  https://review.openstack.org/64918521:14
*** pcaruana has quit IRC21:17
*** boden has quit IRC21:45
*** velizarx has joined #openstack-lbaas22:06
VorrtexHey johnsom do you know if you can replace an existing image with a new one in docker.io?22:33
Vorrtexor maybe rm_work you might know? ^^?22:33
VorrtexAs in "docker push registry/image:tag", then update that image, and then push it *again* with the same command?22:35
Vorrtexwhere registry is specifically docker.io22:35
*** fnaval has quit IRC22:40
*** rcernin has joined #openstack-lbaas22:40
johnsomI think they are all unique, so no, I don't think you can just overwrite.  That said, I rarely use docker hub so could completely be work.  Wrong channel to ask about docker... grin22:47
johnsoma quick google says they may have changed that in later releases, so.... Who knows22:49
Vorrtexjohnsom yeah, I'm not really able to confirm or deny if you can just push a new image to the same name:tag in docker.io.22:56
VorrtexOh well, thanks for looking anyway.  I'll keep digging around.22:56
johnsomFyi, a devstack patch broke the tempest barbican gate, they have a proposed patch: https://review.openstack.org/64895123:01
johnsomThough apparently they also have troubles with compute instances failing, so zuul -1'd it.23:02
*** goldyfruit has quit IRC23:17
colin-interesting, bumping the timeout_member_connect and similar values had a (limited, but) positive effect on my service23:19
*** fnaval has joined #openstack-lbaas23:35
johnsomHmm, I don't think that setting is used for UDP....23:37
colin-i can't see why it would be respected for a udp listener but i swear i noticed a positive correlation23:39
colin-maybe a mistake on my part23:39
johnsomI'm looking just to make sure. My initial code scan says no, but I'm going to double check.23:39
johnsomI think we have a story open to implement some of those for UDP23:39
johnsomYeah, that is not used for UDP.  UDP doesn't have sessions, so doesn't "connect"23:42
colin-my thinking too, i wonder what i observed then23:45
colin-will need to try to get more.. something.. out of my packet capture23:46
*** Vorrtex has quit IRC23:46
johnsomIf you want a second set of eyes I am happy to look at a pcap file23:46
colin-thanks, i may take you up on that. going to compare one from a direct connection vs octavia and look for differences, may solicit some feedback at that stage23:48
colin-thanks for the input so far23:48
johnsomNP23:48
*** goldyfruit has joined #openstack-lbaas23:54
*** openstackgerrit has quit IRC23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!