Thursday, 2016-10-13

jascott1ok I was going with 'local registry is required for multinode setup'00:00
sdakeok multinode is not ready for prime time00:00
jascott1sounds good will try it00:00
sdakejascott1 have you deployed AIO yet using minikube?00:00
jascott1I did the kk8s aio yes00:00
*** liyifeng has quit IRC00:01
jascott1my k8s cluster seems to be working pretty well so far.. can "do stuff", hit the dashboard etc00:01
*** HyperJohnGraham_ has quit IRC00:01
sdakejascott1 ok - nwo is not a super good time to try multinode00:01
sdakejascott1 until the docs are in good enough shape that sbezverk_ and kfox1111 and rhallisey feel comfortable walking people through them00:02
sdakewe worked on that with hyperjohngraham for about 6 hours with no positive outcome00:02
sdakeits a docs issue entirely00:02
sdakeand i dont know how to do multinode deploy of that deliverable00:02
jascott1ok good to know. maybe I can help out w it later00:02
jascott1just knowing non TLS registries are supposed to work is helpful00:03
sdakejascott1 that said, ping sbezverk_ or kfox1111 or rhallisey and see what shape the docs are in00:03
sdakejascott1 tls/nontls both work00:03
jascott1I will probably keep playing w it on the side but do another AIO to get make some progress00:03
sdakeif configured properly00:03
sbezverk_jascott1: sdake with John's setup the issue was a misconfiguration. atm when we were installing kube, he had selinux turned on and it screwed up configuration of networking00:03
sdakejascott1 i'd try my recommendation of commenting out the registry00:03
sdakesbezverk_ cool thanks for getting to the root cause of that00:04
jascott1sdake ok will do00:04
jascott1just a thought but if your home is your 'workplace' then you might need to just get a keggorator like all the hip SF workplaces ;)00:05
sdakesbezverk_ did john get it going00:05
sbezverk_sdake: do not know. I was busy with fencing pod last few days, did not have opportunity to follow up with him..00:06
sdakeok time to tag00:09
sdakegoing to do one lsaat build and deploy00:09
sdakewish me well - bbiaf00:09
*** sdake_ has joined #openstack-kolla00:11
*** sdake has quit IRC00:14
sdake_fan tastic one of my machines is not operational00:15
sdake_bjolo ping00:17
jascott1sdake commenting out registry got me far. I also had to change image version in globals.yml. Now I am seeing sysctl errors but will work on that00:24
sdake_jascott1 cool00:25
sdake_jascott1 what is happening there is your using the docker registry00:25
sdake_it maybe that 12.0.1 or whateverversion you linked earlier doesn't work with that --insecure-registry flag00:25
sdake_docker was talking of removing that00:26
sdake_maybe they finally did so00:26
jascott1hmm these errors are 'setting sysctl values' like this error 'cannot stat /proc/sys/net/ipv4/tcp_max_sync_backlog: No such file or directory'00:28
*** tonanhngo has quit IRC00:32
*** yingjun has joined #openstack-kolla00:33
sdake_why on earth is xorg installed in kolla containers00:33
* sdake_ shakes head00:33
sdake_jascott1 your on an ubuntu host?00:33
jascott1yes00:33
jascott11400:33
sdake_jascott1 did you set the distro and install type in globals.yml?00:33
jascott1no sir00:33
* jascott1 looks at feet00:33
sdake_try that00:33
jascott1thanks00:33
sdake_the defaults are centos bianry00:33
sdake_you probably want ubuntu source00:33
*** tonanhngo has joined #openstack-kolla00:34
*** duonghq has joined #openstack-kolla00:37
duonghqmorning00:37
duonghqit's true autumn here (quite late than the past, but it come finally)00:38
*** tonanhngo has quit IRC00:39
jascott1woot. progress. made it all the way to 'haproxy : Waiting for virtual IP to appear'00:40
*** mtaylor22 has quit IRC00:40
duonghqjascott1: I'm sorry but which bug do you mention?00:41
jascott1duonghq sdake and other nice ppls in here have been helping me get a multinode kk8s going00:41
jascott1im running kolla-ansible deploy but having issues00:42
duonghqjascott1: you are using kolla-ansible or k8s?00:42
jascott1both00:42
jascott1I prob dont get your question00:43
duonghqI'm not sure you got trouble with which project00:43
jascott1I have k8s cluster and am trying to deploy kolla-kubernetes on it00:43
*** HyperJohnGraham_ has joined #openstack-kolla00:43
jascott1now that I am not using local registry I have gotten all the way to the install step 'haproxy : Waiting for virtual IP to appear'00:44
jascott1im not sure what IP that should be set to00:44
duonghqlet me check that,00:45
duonghqmoment00:45
jascott1thanks so much00:45
jascott1its in globals.yml but I just have it set to the IP of the host the deploy is running from00:46
duonghqdo you have your globals.yml and inventory in some pastebin?00:47
jascott1one sec00:47
jascott1http://paste.openstack.org/show/585524/00:48
jascott1^globals.yml00:48
jascott1http://paste.openstack.org/show/585525/00:50
jascott1^ inventory00:50
jascott1inventory is stock except for the three IPs I have used at the top00:50
*** tonanhngo has joined #openstack-kolla00:51
jascott1I will take a look at the role00:51
duonghqcan you "docker ps"00:52
duonghqguess that haproxy container and/or keepalived container got trouble00:52
*** tonanhngo_ has joined #openstack-kolla00:53
jascott1ya on the ps, will look at the containers too thx00:53
*** mtaylor22 has joined #openstack-kolla00:55
*** tonanhngo has quit IRC00:56
*** tonanhngo_ has quit IRC00:57
jascott1duonghq that ansible task is just waits for a response on <vip> <database_port>00:58
duonghqjascott1, right00:59
duonghqit use database port as signature for VIP00:59
*** phuongnh has joined #openstack-kolla00:59
jascott1hmm something up with the node it tried to land it on i think01:00
duonghqjascott1, what does keepalived log tell?01:08
*** eaguilar has quit IRC01:12
*** MarMat_ has quit IRC01:18
*** ssurana has quit IRC01:21
jascott1duonghq haproxy has error [ALERT] 285/173647 (9) : sendmsg logger #1 failed: Connection refused (errno=111)01:22
jascott1checking config01:22
jascott1oh01:24
duonghqiirc there is at least a bug about container cannot send log to heka01:24
jascott1i have wrong IP01:25
duonghqoh01:26
*** tonanhngo has joined #openstack-kolla01:26
jascott1unfortunately my kubectl command is not working atm01:26
jascott1but cluster is still trucking... so sayeth the dashboard01:27
sdake_Jeffrey4l__ you alive yet01:27
sdake_Jeffrey4l__ i know its been a rough slog01:27
sdake_curious if you have seen TASK [heat : Running Heat bootstrap container] *********************************01:27
sdake_fatal: [minime-01 -> minime-01]: FAILED! => {"changed": true, "failed": true, "msg": "Container exited with non-zero return code"}01:27
sdake_to retry, use: --limit @/home/sdake/kfinal/kolla/ansible/site.retry01:27
*** yingjun has quit IRC01:28
sdake_if someone could take a look at this: https://bugs.launchpad.net/kolla/+bug/163290501:32
openstackLaunchpad bug 1632905 in kolla "heat bootstrap failed" [Critical,Triaged]01:32
sdake_most appreciated01:32
*** yingjun has joined #openstack-kolla01:33
*** liyifeng has joined #openstack-kolla01:35
jascott1sdake user error? 'Conflict occurred attempting to store role - Duplicate Entry (HTTP 409) '01:37
sdake_huh01:37
jascott1from that issue01:37
jascott1you just linked01:37
sdake_oh yo uthink its user error01:37
sdake_i disagree01:37
sdake_i am pretty sure i know how to deploy kolla-ansible ;-)01:38
jascott1i  meant more like dupe line in config or something01:38
sdake_the config comes from defaults01:38
Pavoseems like a issue with the http service to me01:40
Pavolike a hickup01:40
*** zhubingbing has joined #openstack-kolla01:42
jascott1there are duplice 'auth_uri' entries in my heat.conf but I guess thats legit? I didnt know we could do that in one section01:49
*** Mech422 has quit IRC01:49
sdake_jascott1 auth_uri and auth_url01:49
sdake_two different things01:50
sdake_the bootstrrap rpoblem is about duplicata roles I think01:50
jascott1ya01:50
jascott1wow I am blind01:51
sdake_i can't see too well either jascott101:51
jascott113"mbp01:51
sdake_occupational hazard ;)01:51
jascott1no doubt01:52
jascott1treating myself to 12pt fontsize. livin the dream01:52
jascott1wow life is much better now. I think that was from some tmux foolery trying to get a happy medium with the resolutions01:53
jascott1maybe heat_stack_owner_role and heat_stack_user_role set to same name?01:58
jascott1sake maybe check keystone log?02:01
jascott1^sdake_02:01
*** phuongnh has quit IRC02:02
*** phuongnh has joined #openstack-kolla02:02
*** Mech422 has joined #openstack-kolla02:03
*** dave-mccowan has joined #openstack-kolla02:09
*** zhurong has joined #openstack-kolla02:19
sdake_jascott1 i am busy tagging at presnet02:21
sdake_rather testing if tagging is even possible02:22
sdake_any cores around - critical issue blocking rc202:26
* sdake_ wants to beat erlang with a shovel02:27
jascott1its erlang so you might need a list of shovels :)02:29
*** MarMat_ has joined #openstack-kolla02:33
*** huhaoran has joined #openstack-kolla02:43
*** mtaylor22 has quit IRC02:47
*** mtaylor22 has joined #openstack-kolla02:47
*** yuanying_ has joined #openstack-kolla02:49
*** yuanying has quit IRC02:49
sdake_Jeffrey4l__ https://bugs.launchpad.net/kolla/+bug/156477302:52
openstackLaunchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [Critical,Confirmed]02:52
sdake_this is blocking release of rc202:53
*** huikang has joined #openstack-kolla02:53
sdake_anyone else out there available to help - help is appreciatedo n this one ;)02:53
*** MarMat_ has quit IRC02:59
*** yuanying_ has quit IRC02:59
*** MarMat_ has joined #openstack-kolla03:00
*** liyifeng has quit IRC03:03
*** yuanying has joined #openstack-kolla03:06
*** yuanying has quit IRC03:08
*** eaguilar has joined #openstack-kolla03:09
sdake_MarMat_ around?03:10
*** dave-mccowan has quit IRC03:10
MarMat_sdake whatsup03:10
sdake_https://bugs.launchpad.net/kolla/+bug/156477303:10
openstackLaunchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [Critical,Confirmed]03:10
sdake_blocking release of rc203:10
sdake_can you have a look at the last comment and provide any thoughts03:10
sdake_i'm really at a loss as to how to proceed03:11
sdake_so any technical advice anyone can offer is appreciated03:11
sdake_other then we are not tagging somethign doa03:11
MarMat_sdake sure, looking into it03:11
*** huikang has quit IRC03:13
sdake_i think that shell script is busted03:17
sdake_i  think it needss to ignore sigusr2 as well03:17
MarMat_sdake is that last issue came with rabbitmq update? Any idea when that was updated?03:20
sdake_it came when we itnrroduced rabbitmq from the cloud sig03:20
sdake_this was working for me multinode in the past03:21
sdake_i have no idea why clusterer is throwing a sigusr203:21
*** phuongnh has quit IRC03:23
*** phuongnh has joined #openstack-kolla03:24
*** absubram has quit IRC03:27
MarMat_sdake what node is impacted in your multinode setup?03:29
*** huhaoran has quit IRC03:30
*** huhaoran has joined #openstack-kolla03:30
*** huikang has joined #openstack-kolla03:31
sdake_Reconfigure ................: kill -SIGUSR2 `cat sneakers.pid`03:34
*** g3ek has quit IRC03:34
*** haplo37 has quit IRC03:35
sdake_file:///Users/sdake/Downloads/Mastering%20RabbitMQ%20-%202015.pdf03:35
sdake_looks like sigusr2 should reconfigure rabbitmq03:35
*** haplo37 has joined #openstack-kolla03:36
sdake_rather https://bitbucket.org/xurongzhong/python-chinese-library/downloads/Mastering%20RabbitMQ%20-%202015.pdf03:36
*** g3ek has joined #openstack-kolla03:36
*** absubram has joined #openstack-kolla03:38
*** jidar has quit IRC03:39
*** coolsvap has quit IRC03:40
sdake_well need to eat bbiaf03:43
sdake_MarMat_ if you see anyone else on - ping em03:43
Jeffrey4l__sdake_, sup03:43
sdake_https://bitbucket.org/xurongzhong/python-chinese-library/downloads/Mastering%20RabbitMQ%20-%202015.pdf03:43
sdake_https://bugs.launchpad.net/kolla/+bug/156477303:44
openstackLaunchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [Critical,Confirmed]03:44
* Jeffrey4l__ is checking the comments.03:44
sdake_lst comment03:44
sdake_only needs to be read03:45
sdake_last03:45
Jeffrey4l__roger03:45
Jeffrey4l__sdake_, i haven't got the same error. i will re-build my image and try to re-deploy.03:46
sdake_with 3 nodes?03:46
MarMat_sdake good time to start learning rabbitmq internals :-) but first would like to see that on my systems... not seen it so far. What to do to reproduce?03:46
Jeffrey4l__yes.03:46
sdake_Jeffrey4l__ i built like a couple hours ago03:47
sdake_but yousee the sigusr203:47
sdake_i'm not sure if thats just a normal thing or not03:47
MarMat_sdake ok, i'll try to rebuild freshly03:47
sdake_i think that means "reconfgiure" to rabbitmq03:47
*** yuanying has joined #openstack-kolla03:47
sdake_but the shell script for rabbitmq-server is wrong and not reconfiguring it03:48
sdake_MarMat_ how many nodes do you have?03:48
MarMat_normaly deploying on 2, can have also 303:48
Jeffrey4l__i think it should be OK. dumb-init should handle this properly.03:48
sdake_read the hsell script03:48
sdake_it has signal handlers in it03:48
sdake_sigusr2 is definately cuaght by it03:49
sdake_its not ignored as it should be03:49
*** huikang has quit IRC03:49
sdake_but i'm just guessing based upon that book above03:49
*** coolsvap has joined #openstack-kolla03:52
*** huikang has joined #openstack-kolla03:53
MarMat_sdake you have 2 control nodes? Or what is your node setup?03:53
sdake_Jeffrey4l__ you can log into my boxe if ou like03:54
Jeffrey4l__ok.03:54
Jeffrey4l__which node?03:54
sdake_MarMat_ 3 control03:54
sdake_Jeffrey4l__ logint ot103 but i think the continaer donw is on 101 or 10203:54
sdake_not sure if its helpful03:54
sdake_i deubged it in the log a bit03:55
Jeffrey4l__192.168.1.103 minime-03  this ?03:55
sdake_but wouldn't hurt fory ou to take a seocnd look03:55
sdake_right03:55
sdake_minime-02 had own container03:55
sdake_which seemed to crater my deployment03:56
Jeffrey4l__got. i am check the script.03:56
*** neilus has joined #openstack-kolla04:00
*** mewald has joined #openstack-kolla04:01
*** eaguilar has quit IRC04:02
Jeffrey4l__sdake_, fyi find the commit message about this https://github.com/rabbitmq/rabbitmq-server/commit/e56bec64777c277833552303caa2d0c09b70283104:02
sdake_Jeffrey4l__ that may be same file04:04
sdake_but I dont thin that is theissue04:04
sdake_there is some trapstuff in that shell script04:04
*** salv-orlando has joined #openstack-kolla04:05
Jeffrey4l__sdake_, the trap existed long time ago04:05
Jeffrey4l__https://github.com/rabbitmq/rabbitmq-server/blame/master/scripts/rabbitmq-server04:05
Jeffrey4l__https://github.com/rabbitmq/rabbitmq-server/blame/master/scripts/rabbitmq-server#L27704:05
*** neilus has quit IRC04:06
*** mewald has quit IRC04:06
sdake_2o204:06
sdake_wow04:06
sdake_haven't seen that before04:06
sdake_i use gitblame on cli04:06
Jeffrey4l__github is cool ;)04:08
Jeffrey4l__so this bug can be re-produced in your server? if re-deploy the whole kolla?04:08
sdake_i dont know i havn't tried04:09
sdake_feel free to try - i really need to eat04:09
*** salv-orlando has quit IRC04:09
Jeffrey4l__ok. i will try re-deploy04:10
Jeffrey4l__now?04:10
sdake_sure04:11
sdake_so what is wierd is hat script exits with sigusr204:11
sdake_but i think we should be ignoring it04:11
sdake_i just dont understand how to ignore it04:11
sdake_i think this ia a 3.6 problem in rabbitmq upstream04:12
Jeffrey4l__ok.04:12
Jeffrey4l__since we use dumb-init now. i can we can add some environment variable to avoid that logical.04:13
Jeffrey4l__i can/i think.04:13
openstackgerritZeyu Zhu proposed openstack/kolla: Visit target nodes without password.  https://review.openstack.org/36852404:16
*** mewald has joined #openstack-kolla04:16
*** ssurana1 has joined #openstack-kolla04:19
*** jidar has joined #openstack-kolla04:19
*** zhubingbing has quit IRC04:20
*** jidar has left #openstack-kolla04:20
*** mewald has quit IRC04:21
*** ssurana1 has quit IRC04:24
*** mewald has joined #openstack-kolla04:31
*** unicell has joined #openstack-kolla04:32
*** zhurong has quit IRC04:32
*** unicell has quit IRC04:33
*** mewald has quit IRC04:36
*** unicell has joined #openstack-kolla04:37
Jeffrey4l__sdake_, you are using old image on node2 http://pix.toile-libre.org/upload/original/1476333859.png04:44
Jeffrey4l__sdake_, rabbitmq works find now. ( i didn't re-build the image, just pull latest images in all nodes )04:49
*** mewald has joined #openstack-kolla04:51
*** mewald has quit IRC05:00
*** salv-orlando has joined #openstack-kolla05:00
*** yuanying has quit IRC05:00
*** jmccarthy1 has left #openstack-kolla05:06
*** huhaoran has quit IRC05:08
*** huhaoran has joined #openstack-kolla05:08
*** huikang has quit IRC05:19
*** mewald has joined #openstack-kolla05:21
*** msimonin has joined #openstack-kolla05:24
*** msimonin has quit IRC05:24
*** haplo37_ has quit IRC05:37
*** huikang has joined #openstack-kolla05:38
*** haplo37_ has joined #openstack-kolla05:40
*** neilus has joined #openstack-kolla05:42
*** huikang has quit IRC05:43
*** neilus has quit IRC05:47
*** HyperJohnGraham_ has quit IRC05:50
*** HyperJohnGraham_ has joined #openstack-kolla05:54
MarMat_sdake_ freshly build using most recent kolla, deployed to 3 control nodes, rabbitmq seems to be ok06:03
*** zhurong has joined #openstack-kolla06:04
*** mtaylor22 has quit IRC06:05
Jeffrey4l__MarMat_, i re-deployed openstack on sdake_ 's node. it works. the issue is node2 is not the latest images.06:06
MarMat_Jeffrey4l__ yep, saw your message before06:07
*** zhubingbing has joined #openstack-kolla06:08
*** yuanying has joined #openstack-kolla06:10
*** MarMat_ has quit IRC06:15
duonghqMarMat_ me too06:19
duonghq3 control nodes is run properly06:19
*** yuanying has quit IRC06:23
*** msimonin1 has joined #openstack-kolla06:23
duonghqsdake_ which service do you enable for storage node?06:23
*** mnasiadka has joined #openstack-kolla06:25
*** MarMat_ has joined #openstack-kolla06:26
*** yuanying has joined #openstack-kolla06:29
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix Stale namespace removal issue  https://review.openstack.org/38555906:31
*** tovin07 has joined #openstack-kolla06:33
*** yuanying_ has joined #openstack-kolla06:37
*** egonzalez90 has joined #openstack-kolla06:38
*** yuanying has quit IRC06:39
*** mnasiadka has quit IRC06:40
*** mnasiadka has joined #openstack-kolla06:40
*** mnasiadka has quit IRC06:40
*** mewald has quit IRC06:41
*** mnasiadka has joined #openstack-kolla06:41
*** mewald has joined #openstack-kolla06:50
*** mkoderer has joined #openstack-kolla06:59
*** msimonin1 has quit IRC06:59
*** MarMat_ has quit IRC06:59
*** absubram has quit IRC07:03
*** absubram has joined #openstack-kolla07:03
*** b_bezak has joined #openstack-kolla07:04
*** ankush has joined #openstack-kolla07:04
*** zhubingbing has quit IRC07:05
*** zhubingbing has joined #openstack-kolla07:08
*** shardy has joined #openstack-kolla07:09
*** matrohon has joined #openstack-kolla07:12
berendthave we not yet tagged rc2?07:13
berendtshouldn't this happen yesterday night (my night)?07:14
*** mewald has quit IRC07:22
duonghqwhat is real reconfigure task mean? for apply new configuration file to existing service, or include change the node role? (i.e add/remove compute node)07:29
*** steve-noyes has joined #openstack-kolla07:30
*** mewald has joined #openstack-kolla07:30
*** g3ek has quit IRC07:31
openstackgerritzhongshengping proposed openstack/kolla: Deprecate api_paste_config option  https://review.openstack.org/38580407:32
*** haplo37 has quit IRC07:34
*** yuanying_ has quit IRC07:36
*** berendt has quit IRC07:36
*** haplo37 has joined #openstack-kolla07:36
openstackgerritzhongshengping proposed openstack/kolla: Deprecate auth_plugin option  https://review.openstack.org/38580507:36
*** yuanying has joined #openstack-kolla07:36
*** msimonin has joined #openstack-kolla07:39
*** Serlex has joined #openstack-kolla07:39
*** g3ek has joined #openstack-kolla07:40
*** rmart04 has joined #openstack-kolla07:48
*** ankush has quit IRC07:59
*** psanchez has joined #openstack-kolla08:06
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix Stale namespace removal issue  https://review.openstack.org/38555908:06
*** janem has joined #openstack-kolla08:10
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix Stale namespace removal issue  https://review.openstack.org/38555908:12
*** mewald has quit IRC08:13
*** kproskurin has joined #openstack-kolla08:24
*** neilus has joined #openstack-kolla08:24
*** mewald has joined #openstack-kolla08:25
*** neilus has quit IRC08:29
*** b_bezak has quit IRC08:33
*** b_bezak has joined #openstack-kolla08:33
*** mewald has quit IRC08:34
openstackgerritzhongshengping proposed openstack/kolla: Fix api_paste_config option  https://review.openstack.org/38580408:35
*** athomas has joined #openstack-kolla08:38
*** salv-orl_ has joined #openstack-kolla08:44
*** salv-orlando has quit IRC08:47
*** tonanhngo has quit IRC08:48
*** ankush has joined #openstack-kolla08:54
*** yuanying has quit IRC08:57
*** berendt has joined #openstack-kolla09:01
*** zhurong has quit IRC09:06
*** zhurong has joined #openstack-kolla09:08
*** ChanServ has quit IRC09:14
*** egonzalez90 has quit IRC09:16
*** psanchez has quit IRC09:16
*** hyakuhei- has quit IRC09:17
*** wirehead_ has quit IRC09:17
*** berendt has quit IRC09:17
*** Mech422 has quit IRC09:17
*** portdirect has quit IRC09:17
*** ajafo has quit IRC09:17
*** salv-orl_ has quit IRC09:17
*** janem has quit IRC09:17
*** rmart04 has quit IRC09:17
*** msimonin has quit IRC09:17
*** steve-noyes has quit IRC09:17
*** duonghq has quit IRC09:17
*** sdake_ has quit IRC09:17
*** hieulq__ has quit IRC09:17
*** pbourke has quit IRC09:17
*** openstackgerrit has quit IRC09:17
*** ntpttr has quit IRC09:17
*** jascott1 has quit IRC09:17
*** cinerama has quit IRC09:17
*** b_bezak has quit IRC09:17
*** mkoderer has quit IRC09:17
*** tovin07 has quit IRC09:17
*** huhaoran has quit IRC09:17
*** macsz_ has quit IRC09:17
*** zigo has quit IRC09:17
*** rbbratta has quit IRC09:17
*** NachoDuck has quit IRC09:17
*** amaged__ has quit IRC09:17
*** skramaja has quit IRC09:17
*** ediardo has quit IRC09:17
*** kbyrne has quit IRC09:17
*** ccesario has quit IRC09:17
*** manjeets_ has quit IRC09:17
*** robcresswell has quit IRC09:17
*** harlowja has quit IRC09:17
*** strigazi has quit IRC09:17
*** zhenguo has quit IRC09:17
*** DuncanT has quit IRC09:17
*** dmsimard has quit IRC09:17
*** dasm has quit IRC09:17
*** Kimmo__ has quit IRC09:17
*** matrohon has quit IRC09:17
*** zhubingbing has quit IRC09:17
*** coolsvap has quit IRC09:17
*** irtermit- has quit IRC09:17
*** kklimonda has quit IRC09:17
*** karlamrhein has quit IRC09:17
*** Pavo has quit IRC09:17
*** qwang has quit IRC09:17
*** flaper87 has quit IRC09:17
*** afranc has quit IRC09:17
*** cmceniry_ has quit IRC09:17
*** dotplus has quit IRC09:17
*** quasar has quit IRC09:17
*** elemoine_ has quit IRC09:17
*** nick-ma has quit IRC09:17
*** serverascode has quit IRC09:17
*** ankush has quit IRC09:17
*** Serlex has quit IRC09:17
*** shardy has quit IRC09:17
*** HyperJohnGraham_ has quit IRC09:17
*** qbjolof1 has quit IRC09:17
*** jgriffith has quit IRC09:17
*** Guest66666 has quit IRC09:17
*** esmiurium has quit IRC09:17
*** peterjenkins_ has quit IRC09:17
*** EmilienM has quit IRC09:17
*** cu5 has quit IRC09:17
*** wznoinsk has quit IRC09:17
*** NobodyCam has quit IRC09:17
*** denaitre has quit IRC09:17
*** mgkwill has quit IRC09:17
*** SaMnCo has quit IRC09:17
*** kong has quit IRC09:17
*** nihilifer has quit IRC09:17
*** dims has quit IRC09:17
*** harbie has quit IRC09:17
*** harmw has quit IRC09:17
*** cj has quit IRC09:17
*** smekel_ has quit IRC09:17
*** jroll has quit IRC09:17
*** tonyb has quit IRC09:17
*** cloudnull has quit IRC09:17
*** absubram has quit IRC09:20
*** g3ek has quit IRC09:20
*** zeus has quit IRC09:21
*** zeus has joined #openstack-kolla09:21
*** g3ek has joined #openstack-kolla09:22
*** yuanying has joined #openstack-kolla09:22
*** timss has quit IRC09:22
*** igordcard has quit IRC09:22
*** igordcard has joined #openstack-kolla09:22
*** salv-orlando has joined #openstack-kolla09:30
*** karlamrhein has joined #openstack-kolla09:32
*** yingjun has quit IRC09:34
*** yingjun has joined #openstack-kolla09:34
*** salv-orlando has quit IRC09:35
*** yingjun has quit IRC09:39
*** janem has joined #openstack-kolla09:42
*** janem has quit IRC09:49
*** yuanying has quit IRC09:49
*** matrohon has joined #openstack-kolla10:06
*** ajafo has joined #openstack-kolla10:06
*** Mech422 has joined #openstack-kolla10:06
*** portdirect has joined #openstack-kolla10:06
*** serverascode has joined #openstack-kolla10:07
*** DuncanT has joined #openstack-kolla10:07
*** amaged__ has joined #openstack-kolla10:07
*** timss has joined #openstack-kolla10:07
*** Serlex has joined #openstack-kolla10:07
*** shardy has joined #openstack-kolla10:07
*** Guest66666 has joined #openstack-kolla10:07
*** qbjolof1 has joined #openstack-kolla10:07
*** jgriffith has joined #openstack-kolla10:07
*** EmilienM has joined #openstack-kolla10:07
*** esmiurium has joined #openstack-kolla10:07
*** peterjenkins_ has joined #openstack-kolla10:07
*** wznoinsk has joined #openstack-kolla10:07
*** cu5 has joined #openstack-kolla10:07
*** NobodyCam has joined #openstack-kolla10:07
*** denaitre has joined #openstack-kolla10:07
*** mgkwill has joined #openstack-kolla10:07
*** SaMnCo has joined #openstack-kolla10:07
*** harbie has joined #openstack-kolla10:07
*** nihilifer has joined #openstack-kolla10:07
*** harmw has joined #openstack-kolla10:07
*** dims has joined #openstack-kolla10:07
*** cj has joined #openstack-kolla10:07
*** smekel_ has joined #openstack-kolla10:07
*** jroll has joined #openstack-kolla10:07
*** tonyb has joined #openstack-kolla10:07
*** cloudnull has joined #openstack-kolla10:07
*** psanchez has joined #openstack-kolla10:09
*** salv-orlando has joined #openstack-kolla10:09
*** ankush has joined #openstack-kolla10:09
*** kong has joined #openstack-kolla10:09
*** zhubingbing has joined #openstack-kolla10:09
*** coolsvap has joined #openstack-kolla10:09
*** irtermit- has joined #openstack-kolla10:09
*** kklimonda has joined #openstack-kolla10:09
*** Pavo has joined #openstack-kolla10:09
*** qwang has joined #openstack-kolla10:09
*** flaper87 has joined #openstack-kolla10:09
*** afranc has joined #openstack-kolla10:09
*** cmceniry_ has joined #openstack-kolla10:09
*** dotplus has joined #openstack-kolla10:09
*** quasar has joined #openstack-kolla10:09
*** elemoine_ has joined #openstack-kolla10:09
*** nick-ma has joined #openstack-kolla10:09
*** zhurong has quit IRC10:10
*** hyakuhei- has joined #openstack-kolla10:10
*** wirehead_ has joined #openstack-kolla10:10
*** hyakuhei- has quit IRC10:10
*** wirehead_ has quit IRC10:10
*** salv-orlando has quit IRC10:10
*** ankush has quit IRC10:10
*** kong has quit IRC10:10
*** zhubingbing has quit IRC10:10
*** coolsvap has quit IRC10:10
*** irtermit- has quit IRC10:10
*** kklimonda has quit IRC10:10
*** Pavo has quit IRC10:10
*** qwang has quit IRC10:10
*** flaper87 has quit IRC10:10
*** afranc has quit IRC10:10
*** cmceniry_ has quit IRC10:10
*** dotplus has quit IRC10:10
*** quasar has quit IRC10:10
*** elemoine_ has quit IRC10:10
*** nick-ma has quit IRC10:10
*** psanchez has quit IRC10:10
*** serverascode has quit IRC10:10
*** DuncanT has quit IRC10:10
*** amaged__ has quit IRC10:10
*** timss has quit IRC10:10
*** Serlex has quit IRC10:10
*** shardy has quit IRC10:10
*** qbjolof1 has quit IRC10:10
*** Guest66666 has quit IRC10:10
*** jgriffith has quit IRC10:10
*** esmiurium has quit IRC10:10
*** peterjenkins_ has quit IRC10:10
*** EmilienM has quit IRC10:10
*** cu5 has quit IRC10:10
*** wznoinsk has quit IRC10:10
*** NobodyCam has quit IRC10:10
*** denaitre has quit IRC10:10
*** mgkwill has quit IRC10:10
*** SaMnCo has quit IRC10:10
*** harmw has quit IRC10:10
*** nihilifer has quit IRC10:10
*** harbie has quit IRC10:10
*** dims has quit IRC10:10
*** cj has quit IRC10:10
*** smekel_ has quit IRC10:10
*** jroll has quit IRC10:10
*** tonyb has quit IRC10:10
*** cloudnull has quit IRC10:10
*** Mech422 has quit IRC10:10
*** portdirect has quit IRC10:10
*** matrohon has quit IRC10:10
*** ajafo has quit IRC10:10
*** psanchez has joined #openstack-kolla10:26
*** matrohon has joined #openstack-kolla10:26
*** ajafo has joined #openstack-kolla10:26
*** serverascode has joined #openstack-kolla10:26
*** amaged__ has joined #openstack-kolla10:26
*** timss has joined #openstack-kolla10:26
*** Serlex has joined #openstack-kolla10:26
*** shardy has joined #openstack-kolla10:26
*** Guest66666 has joined #openstack-kolla10:26
*** qbjolof1 has joined #openstack-kolla10:26
*** jgriffith has joined #openstack-kolla10:26
*** EmilienM has joined #openstack-kolla10:26
*** esmiurium has joined #openstack-kolla10:26
*** peterjenkins_ has joined #openstack-kolla10:26
*** wznoinsk has joined #openstack-kolla10:26
*** cu5 has joined #openstack-kolla10:26
*** NobodyCam has joined #openstack-kolla10:26
*** denaitre has joined #openstack-kolla10:26
*** mgkwill has joined #openstack-kolla10:26
*** SaMnCo has joined #openstack-kolla10:26
*** harbie has joined #openstack-kolla10:26
*** nihilifer has joined #openstack-kolla10:26
*** harmw has joined #openstack-kolla10:26
*** dims has joined #openstack-kolla10:26
*** cj has joined #openstack-kolla10:26
*** smekel_ has joined #openstack-kolla10:26
*** jroll has joined #openstack-kolla10:26
*** tonyb has joined #openstack-kolla10:26
*** cloudnull has joined #openstack-kolla10:26
*** hyakuhei- has joined #openstack-kolla10:27
*** wirehead_ has joined #openstack-kolla10:27
*** Mech422 has joined #openstack-kolla10:27
*** portdirect has joined #openstack-kolla10:27
*** domi007 has joined #openstack-kolla10:27
*** domi007 has left #openstack-kolla10:27
*** elemoine_ has joined #openstack-kolla10:27
*** DuncanT has joined #openstack-kolla10:27
*** salv-orlando has joined #openstack-kolla10:27
*** ankush has joined #openstack-kolla10:27
*** zhubingbing has joined #openstack-kolla10:27
*** coolsvap has joined #openstack-kolla10:27
*** irtermit- has joined #openstack-kolla10:27
*** kklimonda has joined #openstack-kolla10:27
*** Pavo has joined #openstack-kolla10:27
*** qwang has joined #openstack-kolla10:27
*** flaper87 has joined #openstack-kolla10:27
*** afranc has joined #openstack-kolla10:27
*** cmceniry_ has joined #openstack-kolla10:27
*** dotplus has joined #openstack-kolla10:27
*** quasar has joined #openstack-kolla10:27
*** nick-ma has joined #openstack-kolla10:27
*** janem has joined #openstack-kolla10:28
*** rmart04 has joined #openstack-kolla10:28
*** msimonin has joined #openstack-kolla10:28
*** steve-noyes has joined #openstack-kolla10:28
*** sdake_ has joined #openstack-kolla10:28
*** hieulq__ has joined #openstack-kolla10:28
*** pbourke has joined #openstack-kolla10:28
*** openstackgerrit has joined #openstack-kolla10:28
*** ntpttr has joined #openstack-kolla10:28
*** jascott1 has joined #openstack-kolla10:28
*** cinerama has joined #openstack-kolla10:28
*** kong has joined #openstack-kolla10:28
*** ediardo has joined #openstack-kolla10:28
*** gfidente has joined #openstack-kolla10:28
*** b_bezak has joined #openstack-kolla10:28
*** mkoderer has joined #openstack-kolla10:28
*** macsz_ has joined #openstack-kolla10:28
*** zigo has joined #openstack-kolla10:28
*** rbbratta has joined #openstack-kolla10:28
*** NachoDuck has joined #openstack-kolla10:28
*** skramaja has joined #openstack-kolla10:28
*** kbyrne has joined #openstack-kolla10:28
*** ccesario has joined #openstack-kolla10:28
*** manjeets_ has joined #openstack-kolla10:28
*** robcresswell has joined #openstack-kolla10:28
*** strigazi has joined #openstack-kolla10:28
*** zhenguo has joined #openstack-kolla10:28
*** dmsimard has joined #openstack-kolla10:28
*** dasm has joined #openstack-kolla10:28
*** Kimmo__ has joined #openstack-kolla10:28
*** DuncanT has quit IRC10:28
*** NobodyCam has quit IRC10:29
*** serverascode has quit IRC10:29
*** ChanServ has joined #openstack-kolla10:30
*** tepper.freenode.net sets mode: +o ChanServ10:30
*** elemoine_ has quit IRC10:31
*** coolsvap has quit IRC10:32
*** rmart04 has left #openstack-kolla10:33
*** DuncanT has joined #openstack-kolla10:33
*** rmart04 has joined #openstack-kolla10:33
*** sdake_ has quit IRC10:35
*** karlamrhein has quit IRC10:35
*** karlamrhein has joined #openstack-kolla10:35
openstackgerritMark Duggan proposed openstack/kolla: Parameterising iscsi_helper and setting default value  https://review.openstack.org/38555310:35
*** amaged__ has quit IRC10:35
*** amaged__ has joined #openstack-kolla10:35
*** ankush has quit IRC10:36
*** ankush has joined #openstack-kolla10:36
*** DuncanT has quit IRC10:37
*** DuncanT has joined #openstack-kolla10:37
*** NobodyCam has joined #openstack-kolla10:38
*** elemoine_ has joined #openstack-kolla10:40
*** zeus is now known as Guest1082510:42
*** coolsvap has joined #openstack-kolla10:45
*** yuanying has joined #openstack-kolla10:45
openstackgerritWaldemar Znoinski proposed openstack/kolla: set basic kolla structures for iscsid  https://review.openstack.org/38194210:49
*** salv-orlando has quit IRC10:51
*** serverascode has joined #openstack-kolla10:51
openstackgerritMark Duggan proposed openstack/kolla: Parameterising iscsi_helper and setting default value  https://review.openstack.org/38555310:51
*** mewald has joined #openstack-kolla10:52
*** awidders_ has quit IRC10:59
*** salv-orlando has joined #openstack-kolla11:03
*** sdake has joined #openstack-kolla11:03
sdakemorning peeps11:04
sdakeJeffrey4l__ any luck looking at that problem?11:04
sdakeJeffrey4l__ I think I feel asleep at the keyboard11:04
*** awebber_ has joined #openstack-kolla11:06
sdakeanyone worked on https://bugs.launchpad.net/kolla/+bug/156477311:07
openstackLaunchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [Critical,Confirmed]11:07
sdakeblocking tag of rc211:07
sdakepbourke ^^11:08
sdakecoolsvap ^^11:08
pbourkesdake: will look now11:08
sdakejust last comment11:08
openstackgerritMerged openstack/kolla: Change keystone.log's permit to keystone:keystone  https://review.openstack.org/38414611:08
*** sdake_ has joined #openstack-kolla11:11
*** sdake has quit IRC11:13
sdake_pbourke i guess problem on my end http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2016-10-13.log.html#t2016-10-13T04:44:5111:15
sdake_although I'd expect deploy to you know, actually deploy new images11:15
*** tonanhngo has joined #openstack-kolla11:15
pbourkesdake_: yeah it doesn't :/11:16
*** tonanhngo has quit IRC11:16
pbourkesdake_: you have to explicitly pull or remove them11:16
pbourkesdake_: unsure why that decision was made11:16
pbourkesdake_: so does it work for you now?11:16
sdake_not sure - need  to test the deploy11:17
pbourkeok11:17
sdake_likei i said i feel asleep att he keyboard11:17
awebber_sdake i have tested external ceph with cinder however it does not deploy. there is a different approach between external ceph and normal ceph deployment. a 'merge config' function is used, only in the external ceph.yml, that fails. i had to revert to a simple copy approach, forgoing merging config approach, so that the deployment was successful. cinder with external ceph now works.11:18
Jeffrey4l__sdake_, you are using old image on node2 http://pix.toile-libre.org/upload/original/1476333859.png11:20
Jeffrey4l__sdake_, rabbitmq works find now. ( i didn't re-build the image, just pull latest images in all nodes )11:20
sdake_Jeffrey4l__ any idea why deploy doesn't deploy latest image11:22
sdake_behavior seems inconsistent to me11:22
sdake_deploy fresh environet -> pulls images11:22
sdake_deploy on existing environemnt -> dont pull fresh images11:22
Jeffrey4l__sdake_, you didn't pull the latest image. and docker-engine doesn't add insecure-registry 103:400011:22
sdake_--insecure-registery 103:4000 wasn't set on 102?11:23
Jeffrey4l__sdake_, yes. i added it today.11:23
sdake_was it set at all?11:23
Jeffrey4l__sorry?11:24
Jeffrey4l__pbourke, please review this https://review.openstack.org/37876211:27
Jeffrey4l__add this https://review.openstack.org/38459811:27
*** sdake has joined #openstack-kolla11:27
pbourkeJeffrey4l__: I think we need a bug id if we're to merge for Newton11:27
*** haplo37_ has quit IRC11:27
pbourkeJeffrey4l__: Im not seeing any high profile breakages without this11:28
Jeffrey4l__pbourke, ok. np. i will file a bug.11:28
*** sdake_ has quit IRC11:28
Jeffrey4l__pbourke, it may cause issue. and when you changed the timezone, the time in the container will be jump from one time to another ( he doesn't know the timezone info ) like from 5am to 8am11:29
*** domi007_ has joined #openstack-kolla11:29
pbourkeok11:30
*** haplo37_ has joined #openstack-kolla11:30
domi007_hi all, I'd like to get some help on an issue I'm facing: I'm trying to deploy using kolla to a coreos host, and while trying to deploy neutron-dhcp-agent the following error pops up in the log:11:30
domi007_level=error msg="Handler for POST /v1.21/containers/create returned error: invalid mode for volumes-from: shared"11:31
*** mliima has joined #openstack-kolla11:31
pbourkedomi007_: you need to set the mountflags on your docker daemon11:31
Jeffrey4l__domi007_, check this doc http://docs.openstack.org/developer/kolla/quickstart.html#install-dependencies11:31
pbourkedomi007_: check the quickstart docs for this11:31
domi007_pbourke: already set them11:31
Jeffrey4l__dmsimard, docker version?11:32
Jeffrey4l__sorry pinged wrong person. domi007_ ^^11:32
domi007_pbourke: MountFlags=1048576 just checked via systemctl11:32
pbourkeit needs to be MountFlags=shared11:32
domi007_Jeffrey4l__: 1.9.111:32
mliimamorning11:32
Jeffrey4l__domi007_, we need 1.10 at min11:33
domi007_Jeffrey4l__: ah okay, let me try to update it11:33
domi007_and see if it work11:33
domi007_thank you11:33
Jeffrey4l__cool.11:33
pbourkeJeffrey4l__: dont suppose you've ever tried setting a difference value for network_interface on one host?11:33
Jeffrey4l__pbourke, yes. i tried with success.11:34
Jeffrey4l__but you need remove the network_interface from globals.yml and set it in inventory file.11:34
pbourkeJeffrey4l__: ahhhhh!11:34
pbourkeJeffrey4l__: so obvious! thank you11:34
pbourkewasted hours on this o_011:35
Jeffrey4l__;) np.11:35
openstackgerritpawnesh kumar proposed openstack/kolla: Use upper constraints for all jobs in tox.ini  https://review.openstack.org/38422111:35
Jeffrey4l__this is how ansible works. globals.yml overwrite all values.11:35
openstackgerritJeffrey Zhang proposed openstack/kolla: Mount host localtime for containers  https://review.openstack.org/37876211:36
*** salv-orlando has quit IRC11:36
pbourkeI know just somehow forgot11:36
pbourkeglobals.yml suck11:36
Jeffrey4l__pbourke, re localtime  add a bug id ^^11:36
pbourkethere are better override mechanisms11:36
pbourkeJeffrey4l__: cool thanks +211:36
Jeffrey4l__ya.11:36
Jeffrey4l__thanks.11:36
*** tonanhngo has joined #openstack-kolla11:39
*** tonanhngo has quit IRC11:40
sdakeJeffrey4l__ what is your thinking on why deploy doesn't pull if images exist?11:42
sdakei dont think that makes a whole lot of sense11:43
*** zhubingbing has quit IRC11:43
sdakealso I have tagged at this hash: 40e768ec2a370dc010be773af37e2ce417adda8011:43
sdakeanything aftre thta needs a backport11:43
*** ppalacios has joined #openstack-kolla11:44
openstackgerritPraveen N proposed openstack/kolla: [ovs] bridge_mappings new line fix  https://review.openstack.org/38592411:48
*** athomas_ has joined #openstack-kolla11:53
*** rmart04_ has joined #openstack-kolla11:55
*** flaper87 has quit IRC11:56
*** athomas has quit IRC11:56
*** rmart04 has quit IRC11:57
*** rmart04_ is now known as rmart0411:57
*** dwalsh_ has joined #openstack-kolla12:02
*** phuongnh has quit IRC12:03
*** sdake_ has joined #openstack-kolla12:05
*** tonanhngo has joined #openstack-kolla12:06
*** sdake has quit IRC12:06
*** tonanhngo has quit IRC12:06
sdake_here is rc3: https://launchpad.net/kolla/+milestone/newton-rc312:11
openstackgerritPraveen N proposed openstack/kolla: Fix template for multiple physical networks  https://review.openstack.org/37827612:11
*** flaper87 has joined #openstack-kolla12:12
*** flaper87 has quit IRC12:12
*** flaper87 has joined #openstack-kolla12:12
*** williamcaban has joined #openstack-kolla12:14
sbezverk_pbourke: ping12:15
pbourkesbezverk_: hi12:15
sbezverk_pbourke: good morning, or no afternoon for you :-) I replied to your comment12:16
pbourkethanks :) will look now12:16
sbezverk_about drop root12:16
pbourkecan you remind me the link12:16
sbezverk_sure12:16
*** JoseMello has joined #openstack-kolla12:16
sbezverk_https://review.openstack.org/#/c/384988/312:16
pbourkeceph will be dropping root very soon12:17
pbourkebut it seems you've looked at it so it's good for now12:17
*** rhallisey has joined #openstack-kolla12:17
sbezverk_pbourke: thanks once it is done on kolla side we will do it too and re-use your sudoers file12:17
pbourkesounds good!12:18
sbezverk_second question is about fluentd repo location.12:18
sbezverk_I mean since kolla does not use it do you still wants it to be in the base?12:18
sbezverk_https://review.openstack.org/#/c/375615/1312:19
sbezverk_pbourke: personnaly I do not care, I was just concern to hit more resistance from kolla cores ;-)12:20
pbourkesbezverk_: did you see inc0 also did a k8s toolbox?12:20
wznoinsksdake_: would we make https://bugs.launchpad.net/kolla/+bug/1632850 targetting both milestones or it's worth a short discussion with the rest of the guys?12:21
openstackLaunchpad bug 1632850 in kolla "lack of basic kolla structures for iscsid " [High,In progress] - Assigned to Waldemar Znoinski (wznoinsk)12:21
sbezverk_really?? when ?? he did not say anything .. if it is the case then we need only one..12:21
pbourkeI think his patch went up first actually but yours seems ready to go so I dont think he'll mind12:21
pbourkeI asked him to check yours anyway12:21
sbezverk_pbourke: I will ping him later for PS12:22
*** srwilkers has joined #openstack-kolla12:33
*** tonanhngo has joined #openstack-kolla12:35
*** tonanhngo has quit IRC12:36
*** egonzalez90 has joined #openstack-kolla12:38
sbezverk_pbourke: sorry but what about fluentd repo, do you still want me to move to base?12:42
bjolo-afkgood afternoon12:52
*** prithiv has joined #openstack-kolla12:58
*** zhubingbing has joined #openstack-kolla12:58
*** lamt has joined #openstack-kolla12:59
*** vks100 has joined #openstack-kolla13:00
bjolo-afksdake_, around? can i please challenge your verdict on this one :) https://bugs.launchpad.net/kolla/+bug/162695813:01
openstackLaunchpad bug 1626958 in kolla "neutron_server not working when lbaas enabled" [Medium,Confirmed] - Assigned to Swapnil Kulkarni (coolsvap)13:01
*** bjolo-afk is now known as bjolo13:01
sdake_fell asleep at thek keyboard again groan13:02
sdake_bjolo to be brutally frank, I didn't take a hard look at medium bugs for rc313:02
sdake_only high and critical13:02
sdake_bjolo so if something got missed its probably my fault13:02
sdake_i did inspect all high and critical bugs for about 8 hours oro so13:02
sdake_to make sure they were triaged propely13:02
sdake_and to determine what made rc313:02
sdake_that said, could have mde mistakes there13:02
sdake_we work at this as a community bjolo no verdict13:02
sdake_bjolo so if there is something you need for rc3 and its critical lets see if we can get it solved13:03
bjolo:) im new here so i might choose the wrong words sometimes13:03
sdake_bjolo its good we know english is not native for you13:04
sdake_bjolo i even choosse wrong words and I am english native speaker13:04
bjoloi just feel that enable lbaas breaking neutron completely is higher than medium. lbaas is the foundation to run cloud native applications so it is a very common feature. about to say standard to run on a cloud13:04
sdake_ok so looking at bug13:04
sdake_bjolo give me a moment to process the bug13:04
vks100Hi, running into some problems with kolla/magnum13:05
vks100status_reason       | Failed to create trustee or trust for Cluster: 34f9066e-c3f4-438f-bfb3-29cff772174c13:05
mnasiadkavks100: stable/mitaka or master?13:06
vks100 pip show kolla --- Metadata-Version: 2.0 Name: kolla Version: 2.0.2 Summary: Kolla OpenStack Deployment Home-page: http://www.openstack.org/ Author: OpenStack Author-email: openstack-dev@lists.openstack.org Installer: pip License: UNKNOWN Location: /usr/lib/python2.7/site-packages Requires: pbr, graphviz, Jinja2, six, oslo.config, pycrypto, gitdb, beautifulsoup4, GitPython, docker-py, setuptools Classifiers:   Environment13:06
*** awebber_ has quit IRC13:06
vks100stable/mitaka..13:06
sdake_bjolo i think course of action here is to document how to enable lbaas/vpnaas13:06
sdake_bjolo as for including fwaas an dvpnaas directly in the contianer themselves, I am not sure that makes alot of sense13:07
sdake_bjolo I'd have to think about that for 5 minutes after waking up :)13:07
sdake_bjolo these are plugins -and neutron is going to more plugins not less13:08
sdake_bjolo what we want to avoid is having 60 plugins installed by default because they are needed for some specific use cse13:08
sdake_bjolo thats why image customization exists13:08
mnasiadkavks100: I remember stable/mitaka did not had barbican enabled and there were a lot of problems with creating trust/trustees - but wait for some expert advice from the developers ;)13:08
sdake_bjolo the policy decision about inclluding by dedfault vpnaas and fwaas - i think we need to have that discussion13:08
bjolosdake_, i can agree with that reasoning to some extent, but here we are talking about kolla supported services enabled in globals.yml13:09
sdake_bjolo yes i jus tthought of that condition13:09
sdake_bjolo again awake for 5 minutes :)13:10
bjolo:)13:10
bjolotake your time and have a coffee first :)13:10
sdake_mexican coke13:10
bjoloor that13:10
sdake_i dont drink coffee13:10
sdake_often atleast13:10
zhubingbinghello guys13:13
*** salv-orlando has joined #openstack-kolla13:14
*** lamt has quit IRC13:14
*** jheroux has joined #openstack-kolla13:14
mnasiadkavks100: https://bugs.launchpad.net/kolla/+bug/1551992 - this may come in handy, but I see it was only merged in master13:14
openstackLaunchpad bug 1551992 in kolla mitaka "Magnum failed to create trustee" [Critical,Confirmed] - Assigned to Vikram Hosakote (vhosakot)13:14
vks100mnasiadka thank you. I will try that.13:16
mnasiadkavks100: maybe just wait a week or more for stable/newton kolla release13:17
*** awebber_ has joined #openstack-kolla13:17
sdake_bjolo theoretically, high/critical bugs in ocata will be backported to 3.0.113:18
sdake_bjolo its too late to give the proper thought to that problem for 3.0.013:18
sdake_bjolo as it affects more then vpnaas and fwaas, your asking a policy be made13:18
sdake_and pbourke is requesting that policy decision be punted to post newton13:18
sdake_bjolo fwiw I agree the policy decision should be punted to post newton, and agree with you bjolo that the tricky part is that we have enables for these things yet they dont *enable*13:19
bjolook i respect that and i know how to build my own images to get around it13:20
*** haplo37 has quit IRC13:20
sdake_bjolo we do have an easy way to customize the images13:20
sdake_bjolo i'm not sure if that is goodd enough for this csaes13:20
sdake_bjolo i think wht we need to make the policy decision is the full list of things affected now and into the future13:21
*** g3ek has quit IRC13:21
*** srwilkers has quit IRC13:21
bjolobut as you said, they are official kolla services.13:21
sdake_bjolo right13:21
*** haplo37 has joined #openstack-kolla13:21
sdake_bjolo so is cinder13:21
bjoloi just fear that new users to kolla will do a pip install and deploy with the official images on docker hub and things dont work13:22
sdake_cinder has 50+ plugins13:22
sdake_do we install all 50 plugins for cinder13:22
bjoloyes, but we dont have 50 cinder plugins in globals.yml13:22
sdake_or do we install only a subset13:22
*** g3ek has joined #openstack-kolla13:22
sdake_bjolo right - i understand we dont list those plugisn out13:22
sdake_bjolo maybe where we went wrong is having an enable_vpnanss or enable_fwwas in the first place13:22
*** jtriley has joined #openstack-kolla13:22
bjolokollas contract to users ought to be that all that is enabled in globals.yml should wokr13:22
sdake_or maybe they aren't actually plugins13:23
sdake_bjolo yup tend to agree13:23
*** portdirect has quit IRC13:23
*** portdirect_ has joined #openstack-kolla13:23
bjolosdake_, agree as well. remove vpn and lbaas from globals.yml13:23
*** portdirect_ is now known as portdirect13:23
sdake_bjolo - i'm not sure what that would break or fix :)13:23
bjoloif it is in there, it should work with the official binary images on docker hub13:24
sdake_bjolo ok so thats what it would fix13:24
sdake_i'm not sure what it would break13:24
* sdake_ never wanted to ship stuff on docker hub13:24
* sdake_ sniffles13:24
wznoinsksdake_: could you look at my last comment on https://bugs.launchpad.net/kolla/+bug/163284013:25
openstackLaunchpad bug 1632840 in kolla "ldlinux.c32 File not found in ironic_pxe" [High,In progress] - Assigned to Waldemar Znoinski (wznoinsk)13:25
bjoloits about being 100% clear on what to expect as a user. Only ship what works. things that are halv broken tend to tick people off13:26
bjoloeither support and fix vpn and lbaas all the way, including the official docker images or remove it from globals.yml13:26
sdake_bjolo ya i get that13:27
*** inc0 has joined #openstack-kolla13:27
*** ayoung has joined #openstack-kolla13:28
openstackgerritDavanum Srinivas (dims) proposed openstack/kolla: Update reno for stable/newton  https://review.openstack.org/38598513:28
sdake_bjolo i guess we could improve on setting expectations aroudn what works and what doesn't - we have a summit session on fixing that problem in the future13:28
sdake_bjolo although that session doesn't specficially include this issue13:29
inc0good morning13:29
sdake_bjolo here is the deal - if someone fixes it prior to 18th, I dont particularly care if it goes in - as long as its not considered a policy deicision13:29
inc0dims, I see you're in process of creating stable branch? great!13:29
dimsinc0 : done, please cross check13:30
inc0cool:) thanks!13:30
sbezverk_sdake_: remember last night one guy had problem with https/http access to registry? I know see this problem with build on the lastest master13:30
bjolocan we have a policy that everything in globals.yml is supported and if it does not work it is critical? :)13:30
sbezverk_what was the solution for him?13:30
*** williamcaban has quit IRC13:31
*** fguillot has joined #openstack-kolla13:31
*** tonanhngo has joined #openstack-kolla13:31
inc0bjolo, so you can override every variable in whole ansible in globals;)13:32
*** tonanhngo has quit IRC13:33
bjoloinc0,  yes i know and love that part of kolla but :)13:33
bjolothis is more about the contract to the users of what should be guaranteed and tested to work13:34
inc0bjolo, what are we talking about anyway? what's not working?13:34
bjoloas it is right now, we have kolla services in globals that we know are broke13:35
inc0bjolo, we'll always have sth like this13:35
bjolohttps://bugs.launchpad.net/kolla/+bug/162695813:35
openstackLaunchpad bug 1626958 in kolla "neutron_server not working when lbaas enabled" [High,Confirmed] - Assigned to Swapnil Kulkarni (coolsvap)13:35
inc0however what we need to do is to make it clear what is well tested and what isn't13:36
inc0we probably have biggest portfolio of services deployable by Kolla13:36
sdake_probably/do13:36
inc0out of all deployment tools13:36
bjoloyes13:37
bjolodont get me wrong. kolla fricking rocks!13:37
inc0it's impossible to keep everything on same level of testing13:37
sdake_bjolo gets that inc013:37
sdake_this isn't a testing issue13:37
sdake_this is a policy issue13:37
bjolospent all day pushing it internally in my company :)13:37
sdake_the issue is13:37
inc0bjolo, I hear ya, what I was thinking is to make it crystal clear somewhere that "this service doesnt have CI"13:37
inc0like a matrix13:37
sdake_1) if a service can be enabled in globals.yml, it should work out of the box without customizations in the docker images."13:38
inc0so operators will know what to expect and new contributors will know what needs fixing13:38
inc0so ofc, if it doesn't work ootb - that is a bug13:39
sdake_inc0 bjolo isn't really talking about that13:39
inc0and it needs fixing13:39
sdake_right - thats why its not marked invalid13:39
sdake_bjolo was more concerned with the fact that its not marked critical I think :)13:40
*** prithiv has quit IRC13:40
sbezverk_sdake: I check kolla log which is working for me and it does not have commit commit e8e1ad70893496f8956559775391500149a1205d13:40
bjolosdake_, yes pushed to ocata-113:40
sbezverk_sdake: I think that might bethe PS broke build.by13:41
bjolomy view is simply since it is a official kolla service, it should be critical and fixed13:41
sdake_bjolo anyone in the drivers group can bring stuff into rc313:41
sdake_bjolo are you in the drivers group?13:41
bjolodont think so13:41
sdake_launchpad id pls13:41
bjolobjolo13:41
bjolobjorn-lofdahl13:41
sdake_christ13:42
sdake_bjolo your lauchpad id isn't bjolo?13:42
bjoloyes it is13:42
*** dave-mccowan has joined #openstack-kolla13:42
sdake_bjolo can ou please change the maintainer of kolla to kolla-drivers13:43
bjolosorry nope13:43
bjolobjorn-lofdahl13:43
sdake_i fat fingered because i'm beat13:43
bjolothats my launchpad ID13:43
*** kproskurin has quit IRC13:43
bjolosdake_, how do i do that?13:44
inc0bjolo, basically you are allowed now to change priorities and everything of bugs13:44
inc0if you feel something is critical, just change it to critical13:44
sdake_bjolo ok - first lets fix my mistake13:44
sdake_bjolo open up https://launchpad.net/kolla13:45
sdake_see where it says "maintainer: bjolo"13:45
sdake_click the yllow button and set it to "kolla-drivers" pls13:45
sdake_bjolo - the drivers team is our extended team of bug triagers13:46
sdake_bjolo the triage process is described in the documentation13:46
sdake_bjolo you can help as much or as little as you want :)13:46
coolsvapsdake_: bjolo when I tested it last week I think, I did not get the error trace as mentioned in the defect13:46
bjolocoolsvap, lets take a look at that. i can reproduce consistently13:47
sdake_bjolo are you able to chane the maintainer13:47
sdake_bjolo need maintainer chnaged asap :)13:47
bjolomaintainer says bjolo13:47
bjolodriver says kolla-drivers13:47
sdake_ya do you see a yellow button next to maintainer13:48
sdake_next to bjolo13:48
*** athomas_ has quit IRC13:48
bjolook done13:48
sdake_thanks13:48
*** mgoddard has joined #openstack-kolla13:49
bjolocoolsvap, how did you do your test?13:49
*** Marx314 has joined #openstack-kolla13:49
bjoloAIO?13:49
bjoloi do multinode, ubuntu/centos source from kolla master13:50
bjolofails every time13:50
openstackgerritMerged openstack/kolla: Fix rabbitmq upgrade permission issue  https://review.openstack.org/38459813:50
openstackgerritMerged openstack/kolla: Fix keystone fernet file exchange via ssh  https://review.openstack.org/36941813:50
bjoloafk for a while. time to pick up the little devils :)13:52
*** MarMat has joined #openstack-kolla13:53
*** srwilkers has joined #openstack-kolla13:53
sdake_ok folks -> we need action for 1 week on these bugs: https://launchpad.net/kolla/+milestone/newton-rc313:55
*** huikang has joined #openstack-kolla13:57
*** huikang has quit IRC13:57
*** huikang has joined #openstack-kolla13:57
*** vks100 has quit IRC13:58
*** huikang has quit IRC13:58
*** huikang has joined #openstack-kolla13:59
*** papacz has joined #openstack-kolla14:00
*** huikang has quit IRC14:03
*** papacz has quit IRC14:05
*** papacz has joined #openstack-kolla14:05
openstackgerritMerged openstack/kolla: Mount host localtime for containers  https://review.openstack.org/37876214:06
openstackgerritMerged openstack/kolla: set basic kolla structures for iscsid  https://review.openstack.org/38194214:06
openstackgerritMerged openstack/kolla: Update reno for stable/newton  https://review.openstack.org/38598514:08
*** sdake_ has quit IRC14:09
*** sdake has joined #openstack-kolla14:10
*** yingjun has joined #openstack-kolla14:11
*** mewald has quit IRC14:12
*** ankush has quit IRC14:12
*** mnasiadka has quit IRC14:14
*** absubram has joined #openstack-kolla14:14
*** huikang has joined #openstack-kolla14:14
*** tonanhngo has joined #openstack-kolla14:14
*** tonanhngo has quit IRC14:15
*** liyifeng has joined #openstack-kolla14:17
*** ChanServ sets mode: +o sdake14:17
*** openstackgerrit has quit IRC14:18
*** openstackgerrit has joined #openstack-kolla14:18
*** sdake changes topic to "FINAL NEWTON DEADLINE - OCT 18, 2016; please work on these bugs https://launchpad.net/kolla/+milestone/newton-rc3; Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA); Summit schedule ht"14:19
*** ChanServ sets mode: -o sdake14:19
inc0sbezverk_, you fencing works?:)14:23
sdakeinc0 have a look at comment #9 https://bugs.launchpad.net/kolla/+bug/156477314:23
openstackLaunchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [Critical,Confirmed]14:23
inc0hmm14:24
inc0I know why:)14:24
sdakegood lets hear it14:24
*** mewald has joined #openstack-kolla14:24
sdakethis bug may be fixed in rc2 if the pull mechanics are logical14:24
sdakethis is wht all the erlang rigamarole was about14:25
inc0hmm..we had this var14:25
inc0docker pull strategy or something which was either always or missing14:25
sdakei remember14:25
inc0can't see it now tho14:25
*** yingjun has quit IRC14:26
*** yingjun has joined #openstack-kolla14:26
inc0I guess we need to force upgrade tasks to pull always14:26
inc0no...we can't14:26
*** mark-casey1 has joined #openstack-kolla14:27
inc0because if somebody did pre-pull we need to make sure to use it14:27
sdakeif someone pre-pulls that is the same thing as deploy pullling again (or ugprade)14:27
sdakepull operation is cheap14:27
sdakeif the eimage is already there14:28
inc0it's not14:28
inc0well pull image will do docker pull14:28
sdakeif someone prepulls image is already there no?14:28
inc0well I guess it might be cheap14:28
inc0as docker will detect that layers are ok14:28
sdakeright14:29
inc0but I'd be careful with doing this in code14:29
*** tonanhngo has joined #openstack-kolla14:29
sdakelets sort out why the pull strategy was removed14:29
inc0I guess because we added pull task14:29
sdakethats a piss poor reason14:30
inc0and it does make it kinda redundant14:30
inc0still, nothing we can do about it right now14:30
sdakedeploy doesn't work14:30
sdakeif old images are on system14:30
inc0what we can do is to ensure that upgrade docs are correct14:30
sdakeDEPLOY not upgrade14:30
inc0although that wont matter because we tag14:30
sdakegod i'd hope upgrade always pulls14:31
*** yingjun has quit IRC14:31
inc0upgrade should not always pull, but on the other hand if you do actual upgrade there won't be images:3.0.0 in system14:31
inc0technically upgrade should be always between tags14:31
inc0so this is only development env issue14:32
inc0tbh14:32
inc0because this will happen only if you run 3.0.0 and you redeploy 3.0.0 with different tags14:32
inc0different code*14:32
inc0which is master branch14:32
*** papacz has quit IRC14:32
sdakei see14:32
inc0and for development you should do kolla-ansible destroy with --remove-images14:33
sdakecan you document that then14:33
inc0or whatever the flag is called14:33
*** papacz has joined #openstack-kolla14:33
sdakethat is solution to a different bug i think14:33
inc0are we positive that that's the root cause of this bug?14:33
sdakebut my comment in #8 was more about why deploy behaves the way it does14:33
sdakeinc0 the root cause of THIS bug is the erlang and rabbitmq changes we went through14:34
sdakethere are about 4 or 5 erlang rabbitmq backtraces14:34
sdakemy hope is by moving forward, we have fixed all those crashes14:34
inc0ok, so destroying images and redeploying should be all good14:34
*** yingjun has joined #openstack-kolla14:34
sdakeyup that should work14:34
sdakei wasn't aware a --remove-images flag was added14:34
sdakeso didn't remove images obviously :)14:35
inc0pbourke, ping14:36
pbourkeinc0: hi14:36
inc0https://bugs.launchpad.net/kolla/+bug/156477314:36
openstackLaunchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [Critical,Confirmed]14:36
inc0look at my comment14:36
inc0if that's the case, I'd bump priority of bug to medium if not even close it alltogether14:37
*** HyperJohnGraham_ has joined #openstack-kolla14:38
*** lamt has joined #openstack-kolla14:39
pbourkeinc0: the problem with this bug is every time rabbit fails for someone it's getting bundled in there14:39
mewaldsdake: You marked https://bugs.launchpad.net/kolla/+bug/1632811 as no longer affects stable/mitaka. The bug is actually valid for mitaka, too, or did I get it wrong what that means on launchpad?14:39
openstackLaunchpad bug 1632811 in kolla "Keystone token table filling up" [Critical,In progress] - Assigned to Mathias Ewald (mewald)14:39
pbourkeinc0: and because rabbit is hard to debug its very hard to tell if they're all the same issue14:39
sdakemewald possible i made  a mistake - feel free to correct14:39
pbourkeinc0: if its working for us Id just close it14:39
inc0well we do check it every single time in gate;)14:40
*** yingjun has quit IRC14:40
sdakesee comment #1114:40
pbourkeinc0: well yeah, not upgrade though14:40
*** yingjun has joined #openstack-kolla14:40
pbourkethough this is not an upgrade bug14:40
sdakeupgraade includes pull.yml14:40
sdakehttps://bugs.launchpad.net/kolla/+bug/1564773 - comment #1114:41
openstackLaunchpad bug 1564773 in kolla mitaka "Continously restarting rabbitmq container for CentOS" [Critical,Confirmed]14:41
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod  https://review.openstack.org/38392214:41
*** yingjun has quit IRC14:41
*** yingjun has joined #openstack-kolla14:41
pbourkesdake: id rather just close it out14:42
sdakepbourke i want to collect any rabbitmq crashes14:42
sdakeand it needs to be backported14:42
pbourkethere's no bug there14:42
pbourkecrashes can be reported as a new bug14:42
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod  https://review.openstack.org/38392214:43
sdakeok fiar enough - there is a bug here pbourke14:43
pbourkewhat is it?14:43
*** yingjun_ has joined #openstack-kolla14:43
sdakepbourke the changess to move to rabbitmq 3.6 in centos and ubuntu fixed them14:43
sdakethere are multiple bugs in this report14:43
*** salv-orl_ has joined #openstack-kolla14:43
sdakethat set of changes related to rabbitmq 3.6 needs backporting14:43
pbourkeceph jewel is so flaky for me14:44
pbourkei have to wonder if our configs are still good14:45
*** yingjun has quit IRC14:46
*** yingjun_ is now known as yingjun14:46
*** salv-orlando has quit IRC14:46
sdakepbourke sounds frieghtening14:47
sdakepbourke multinode?14:47
pbourkeyeah14:47
pbourkemore frustrating14:47
sdakepbourke how mnay nodes14:47
pbourkeI put ceph in the rabbitmq category of difficulty to debug14:48
pbourke3 mons, 3 osds. The issue is the mons will not talk to each other14:48
sdakeis that missing in the rc3 list?14:48
*** HyperJohnGraham_ has quit IRC14:49
pbourkei haven't logged it because I dont know if its a kolla bug or just my setup14:49
pbourkeseems to be ok for others14:50
sdakepbourke i wouldn't ssume that unless you have verbal confirmation14:50
sdakepbourke although AIO works in the kube gate14:50
sdakeyou are doing multinode14:50
pbourkeinc0: Jeffrey4l__: sean-k-mooneyAFK: ceph working ok for you guys?14:51
inc0pbourke, yeah14:51
sdakemultinode?14:51
inc0always14:51
pbourkei dont know what Im doing wrong :/14:51
inc03 mons 2 osds14:51
*** sean-k-mooneyAFK is now known as sean-k-mooney14:51
sdakepbourke i noticed ole isn't using jewel some time ago14:51
sean-k-mooneypbourke: i will be setting up a ceph cluster i a while14:51
sdakea couple days back14:51
sean-k-mooneypbourke: are you still haveing the same ceph mon issue i was haveing a few weeks ago where the mons timeout?14:53
sdakepbourke nm i see that problem has been fixed14:53
pbourkesean-k-mooney: yeah14:53
*** huikang_ has joined #openstack-kolla14:54
sean-k-mooneypbourke: ok as i said above i will be deploy ceph multinode soon. im just building the images again. ill see if i hit it again or not14:54
pbourkesean-k-mooney: thanks a mill14:54
inc0I'll do full upgrade ceph included in a few14:55
pbourkein summary last time I tried the master build it was happening, did not see it in our internal builds. Now Im seeing it there too which leads me to think something is flaky14:55
openstackgerritWaldemar Znoinski proposed openstack/kolla: copy iscsid logrotate config  https://review.openstack.org/38602714:55
inc0pbourke, and osd's can't access mons?14:55
sdakepbourke the last time you tried master perhaps you were using hammer version of ceph...14:55
pbourkesdake: maybe but dont think so. I'll try confirm14:56
pbourkeinc0: when the mons dont form a quorum 'ceph status' just hangs14:56
*** huikang has quit IRC14:56
pbourkeinc0: playbook doesn't notice this until it tries to start the osds though14:56
wznoinskinc0: sdake my bad, I missed one line, here's fix https://review.openstack.org/#/c/386027/ , could you look?14:57
sdake0a50cb88 docker/base/Dockerfile.j2           (Michal (inc0) Jastrzebski   2016-09-06 15:52:24 +0000 129)         http://mirror.centos.org/centos-7/7/extras/x86_64/Packages/centos-release-ceph-jewel-1.0-1.el7.centos.noarch.rpm \14:57
sdakepbourke does that 9/6 date line up with when problems started occuring?14:57
inc0sdake, yeah...I did ceph upgrade:P14:58
sdakeinc0 - was pointing out date not blamee :)14:58
pbourkemaybe but it worked initially14:58
pbourkesdake: dont worry it honestly may just be a problem on my end14:58
pbourkewhat sucks it the lack of useful debugging info from ceph but c'est la vie14:59
inc0well, or oracle linux for whatever reason14:59
*** sdake has quit IRC15:00
*** srwilkers has quit IRC15:02
sean-k-mooneypbourke: my only concern there is i hit the same issue as you15:02
pbourkesean-k-mooney: i think you said you had 6 mons which would make sense though15:03
pbourkesean-k-mooney: has to be an odd number15:03
sean-k-mooneypbourke: you would think that but it works with 2 and 415:03
inc0well we never confirmed that even number is an issue15:03
inc0maybe it's just mon scallability problem15:04
sean-k-mooneypbourke: in my case though i stopped haveing the isue when i went to 3 but it could be a timeing ting. eg it take longer to have a quourm with more mons15:04
sean-k-mooneyif ceph could not work with an even number of mons the if you lost a single mon everything would break15:04
*** sdake has joined #openstack-kolla15:05
*** egonzalez90 has quit IRC15:05
kfox1111sean-k-mooney: it works until it doesn't.15:05
sean-k-mooneykfox1111: you tested 4 mons on your setup correct and that worked for you?15:05
kfox11114 will work, so long as only one dies at a time. if it ever goes split brained evenly badness ensues.15:06
inc0sean-k-mooney, I wouldn't be surprised if ceph builds cluster witn n**2 complexity15:06
kfox1111so, not a good idea.15:06
inc0as in every mon talks to every mon15:06
kfox1111everything must be in sync,15:07
inc0so scallability can quicky becomes an issue15:07
kfox1111so there isn't much benifit in not having every mon talk to every other.15:07
sean-k-mooneyperhps if we started the mons serially it may resolve the qurom issue?15:07
*** zhugx has joined #openstack-kolla15:07
inc0yeah I agree, what I'm saying is that is plausable that 6 mons simply is too much15:07
sdakesean-k-mooney we tested 120 mons...15:07
kfox1111we've run 7. worlked fine.15:08
sdakesean-k-mooney oh nah, that was osds15:08
inc0120 osds sdake15:08
kfox1111it just has to be odd for safety.15:08
sean-k-mooneysdake: i think only 3 mons were used in osic15:08
sdakeyes15:08
sdakebeen up all ngiht sorting out this tagging15:08
pbourkei think bringing them up serially is well worth trying15:08
sdakenot at top of game ;)15:08
kfox1111bootstrapping or restarting?15:09
*** jistr has joined #openstack-kolla15:09
kfox1111bootstrapping, its a good idea to do them all serial.15:09
kfox1111restarting should be fine in parallel if you have an odd number.15:09
kfox1111otherwise you risk splitbraining your cluster.15:09
sdakehere is the bug: https://bugs.launchpad.net/kolla/+bug/162923715:11
openstackLaunchpad bug 1629237 in kolla "Ceph monitors not responding (Error connecting to cluster: TimedOut)" [Critical,Incomplete] - Assigned to Paul Bourke (pauldbourke)15:11
kfox1111ah. yeah, I'm guessing it is probably splitbraining itself.15:12
kfox1111serial bootstrapping should help, as is using a proper number of mons.15:12
sdaketerrible that software designed with ha built lin doesn't use virtual synchrony15:13
sdakethere would be no split brain if true replication were used properly15:13
kfox1111I haven't seen a ha cluster that handles an even number of nodes well. not really a thing.15:13
kfox1111and bootstrapping is a special case.15:13
sean-k-mooneyone modifciation to ceph i would like to do for ocata is preset the lables so that we can restore them when we do a destroy. it would make debuging ceph a lot simpler.15:14
kfox1111it happens so infrequently that they probably didn't put much effort into ensuring it properly assembles itself in parallel.15:14
sbezverk_kfox1111: hey!! any news for me?15:14
*** vks100 has joined #openstack-kolla15:14
sdakekfox1111 they do exist - but require virtual synchrony15:14
kfox1111sbezverk_: been really swamped :/ had a crazy outage thrust on me.15:15
kfox1111they forced us to turn on cadf across every service in the ha cloud. :/15:15
*** vhosakot has joined #openstack-kolla15:15
kfox1111sdake: yeah but still not seen a virtual synchrony system handle failure well in a non even number of node situation.15:16
sdakenote, I asked the reporter of this bug: https://bugs.launchpad.net/kolla/+bug/161733415:16
openstackLaunchpad bug 1617334 in kolla "reconfigure action fails on [neutron | Restart the neutron_openvswitch_agent container] " [Critical,Incomplete]15:16
sdaketo visit us in irc so we could have a debug session of his system15:17
sdakeif you come acorss him, pleasse help him :)15:17
sdakeso we can either close this bug out as invalid or fixx the problem15:17
sbezverk_kfox1111: no prob..15:17
sdakemost likely user error15:17
*** salv-orl_ has quit IRC15:17
*** ssurana has joined #openstack-kolla15:18
sdakethat whole deploy doesn't pull is a new can of worm we wil lbe dealing with constantly15:18
* sdake groans15:18
sdakestill dealing with pbr nonsense15:18
sdakeespecially since kolla-kubernetes tells folks to pip install kolla....15:18
sdakeconfusing :)15:18
kfox1111hmm... is  pip install bad?15:19
sdakekfox1111 yes15:20
sdakekfox1111 for developers15:20
kfox1111but for ops?15:20
sdakekfox1111 that wnat to build master images - it doesn't work correctly15:20
sdakefor ops its the right approach15:20
kfox1111ah. k.15:21
sdakekfox1111 reference kolla docs15:21
*** huikang_ has quit IRC15:21
sdakethere are two workflows15:21
*** zhugx has quit IRC15:22
*** huikang has joined #openstack-kolla15:22
*** b_bezak has quit IRC15:23
kfox1111ah. so its the git tagging thing?15:23
inc0god our ceph deployment became better since mitaka15:26
openstackgerritMerged openstack/kolla: Cleanup container script improvement  https://review.openstack.org/38586015:30
openstackgerritMerged openstack/kolla: copy ldlinux.c32 to /tftpboot in ironic_pxe  https://review.openstack.org/38561215:31
sdakekfox1111 sort of15:32
sdakekfox1111 if you read the docs it explains the issue in detail15:32
*** rmart04 has quit IRC15:36
openstackgerritMerged openstack/kolla: Move loc tool into tools directory  https://review.openstack.org/37949715:37
*** fguillot has quit IRC15:38
openstackgerritMerged openstack/kolla: Update the location of Glance Registry HAProxy checks  https://review.openstack.org/37818315:38
*** irtermit- is now known as irtermite15:39
*** tonanhngo has quit IRC15:41
sdakecoolsvap ping re requirements15:47
sdakecoolsvap will stable/newton get an automatic update from the global requirements proposal bot?15:47
openstackgerritcaoyuan proposed openstack/kolla: Add watcher checking to port_checks.yml  https://review.openstack.org/37823315:47
openstackgerritWaldemar Znoinski proposed openstack/kolla: copy iscsid logrotate config  https://review.openstack.org/38602715:52
*** papacz has quit IRC15:55
openstackgerritMathias proposed openstack/kolla: Use memcache for keystone token persistence  https://review.openstack.org/38558615:56
*** huikang has quit IRC16:00
sdakeinc0 was that problem that was discussed on irc with the inventory file fixed16:03
sdakeinc0 or was a bug filed16:03
sdakebjolo i believe you may have raised the issue16:04
sbezverk_kfox1111: when have you tried to deploy kube last time?16:07
sbezverk_I see some issues with keystone bootstarpping16:07
*** mark-casey1 has quit IRC16:08
*** papacz has joined #openstack-kolla16:08
sdakecan someone take on confirming this bug: https://bugs.launchpad.net/bugs/163290516:09
openstackLaunchpad bug 1632905 in kolla "heat bootstrap failed" [Critical,Triaged]16:09
sdakei reproduced it back to back last night16:09
*** vhosakot has quit IRC16:09
*** vhosakot_ has joined #openstack-kolla16:09
sdakebut i am the reporter, it needs soemone else to reproduce it16:09
sdaketo confirm it16:10
*** domi007_ has quit IRC16:10
*** papacz has quit IRC16:13
sdakebjolo you about16:13
sdakebjolo preparing to reproduce this https://bugs.launchpad.net/kolla/+bug/162564816:14
openstackLaunchpad bug 1625648 in kolla "horizon login fails with TLS enabled" [Critical,Triaged]16:14
*** vhosakot has joined #openstack-kolla16:14
*** vhosakot_ has quit IRC16:15
kfox1111sbezverk_: the last gate test I did. :)16:15
kfox1111what are you seeing?16:16
*** awebber_ has quit IRC16:16
openstackgerritMerged openstack/kolla: Remove do_reconfigure.yml file for all roles  https://review.openstack.org/37587816:20
*** steve-noyes has quit IRC16:20
*** rmart04 has joined #openstack-kolla16:22
pbourkesdake: fyi ceph master for oraclelinux is up to date (jewel)16:22
pbourkesdake: going to try deploy now16:22
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Allow service_checks to run independently of kolla-ansible  https://review.openstack.org/38116116:23
sdakepbourke yes i know - i showed you the git blame for the change16:25
*** huikang has joined #openstack-kolla16:25
sdakewhich ws 9/9 iirc16:25
pbourkesdake: oh yeah just thought maybe there an issue with ol16:25
sdakeor soething like that16:25
*** rmart04_ has joined #openstack-kolla16:25
openstackgerritMerged openstack/kolla: Fix a few typos in documentation  https://review.openstack.org/38026616:25
sdakeso my theory is you were using hammer16:25
sdakean that was working for you16:25
sdakenow your using jewel16:25
sdakeand that isn't16:25
sdakenot sure if that is correct or not16:25
pbourkelets see16:27
sdakewow gliffy just totally changed around their sharing options16:27
*** rmart04 has quit IRC16:27
*** rmart04_ is now known as rmart0416:27
sdaketo share you ahve to buy the top package16:27
sdakewhic his 49$ a month16:28
sdakerediculous16:28
sdaketoo good to be true - probably is16:29
* sdake sadfaces16:29
sdakegoogle just aquire gliffy already16:29
*** harlowja_at_home has joined #openstack-kolla16:30
pbourkeok ceph master came up first go!16:31
pbourkeso something wrong with our internal build16:31
pbourkeforks ftl16:31
wznoinskinc0: mind looking again at https://review.openstack.org/#/c/386027/ ?16:32
*** tonanhngo has joined #openstack-kolla16:32
*** msimonin has quit IRC16:33
*** ajpower has quit IRC16:34
*** inc0_ has joined #openstack-kolla16:38
openstackgerritMerged openstack/kolla: copy iscsid logrotate config  https://review.openstack.org/38602716:38
*** inc0 has quit IRC16:40
*** sdake_ has joined #openstack-kolla16:42
openstackgerritMerged openstack/kolla: add ceilometer congress to precheck.yml  https://review.openstack.org/37815016:43
*** rmart04_ has joined #openstack-kolla16:43
openstackgerritMerged openstack/kolla: Readability Improvements to Advanced Config Doc  https://review.openstack.org/34710216:43
*** fguillot has joined #openstack-kolla16:45
*** sdake has quit IRC16:46
*** rmart04 has quit IRC16:46
*** rmart04_ is now known as rmart0416:46
*** huikang has quit IRC16:47
openstackgerritMerged openstack/kolla: Updated from global requirements  https://review.openstack.org/38512716:49
openstackgerritMerged openstack/kolla: Hyperlink is expired  https://review.openstack.org/38524616:50
wznoinskwhat are would be chances to push 3 changes (iscsi+ironic) back to newton? so people can actually use ironic with kolla in newton?16:51
openstackgerritMerged openstack/kolla: Modify QSG to install dependencies needed to build code with pip  https://review.openstack.org/38407316:51
*** rmart04 has quit IRC16:52
wznoinskI guess if I find enough cores to back up the idea it's virtually possible?16:53
pbourkebjolo: just wondering if in your TLS horizon bug, is it a self signed cert?16:54
*** dwalsh_ has quit IRC16:56
*** huikang has joined #openstack-kolla16:57
*** dwalsh has joined #openstack-kolla16:58
openstackgerritMerged openstack/kolla: Fix wrong mongodb address variable in ceilomter bootstrap  https://review.openstack.org/38553316:59
*** hyakuhei- has quit IRC17:00
*** hyakuhei- has joined #openstack-kolla17:00
*** hyakuhei- has quit IRC17:00
*** hyakuhei- has joined #openstack-kolla17:00
*** hyakuhei- is now known as hyakuhei17:00
sdake_well gliffy was nice while it lasted17:00
*** sdake_ is now known as sdake17:01
sdakei just fired off one of two emails to them17:01
openstackgerritMerged openstack/kolla: Fix template for multiple physical networks  https://review.openstack.org/37827617:01
sdakefirst = complaint17:01
sdakesecond = closing account17:01
openstackgerritMerged openstack/kolla: precheck fails due to unknown cinder_volume_group  https://review.openstack.org/38587317:01
*** srwilkers has joined #openstack-kolla17:02
sdakerbergeron ping17:03
*** msimonin has joined #openstack-kolla17:03
*** msimonin has quit IRC17:05
*** msimonin has joined #openstack-kolla17:05
*** ajpower has joined #openstack-kolla17:05
*** dwalsh has quit IRC17:06
*** msimonin has quit IRC17:06
*** dwalsh has joined #openstack-kolla17:06
*** vks100 has quit IRC17:06
openstackgerritMartin Matyas proposed openstack/kolla: Make iscsi config.yml consistent with start.yml  https://review.openstack.org/38191417:12
*** lamt has quit IRC17:18
*** huikang has quit IRC17:18
*** mgoddard has quit IRC17:20
*** unicell has quit IRC17:24
*** neilus has joined #openstack-kolla17:29
*** huikang has joined #openstack-kolla17:33
*** neilus has quit IRC17:34
*** dwalsh has quit IRC17:34
*** dwalsh has joined #openstack-kolla17:36
openstackgerritMathias proposed openstack/kolla: Set memcache connection limit to 5000  https://review.openstack.org/38610517:39
*** msimonin has joined #openstack-kolla17:40
*** mkoderer has quit IRC17:42
*** dwalsh has quit IRC17:44
*** dwalsh has joined #openstack-kolla17:44
*** unicell has joined #openstack-kolla17:47
*** yingjun has quit IRC17:48
openstackgerritMathias proposed openstack/kolla: Monitor docker with telegraf  https://review.openstack.org/38611017:51
*** rhallisey has quit IRC17:51
*** huikang has quit IRC17:56
*** mgoddard has joined #openstack-kolla17:59
*** rhallisey has joined #openstack-kolla18:00
openstackgerritMathias proposed openstack/kolla: Monitor mariadb with telegraf  https://review.openstack.org/38611918:02
*** huikang has joined #openstack-kolla18:11
*** huikang has quit IRC18:12
*** huikang has joined #openstack-kolla18:12
mewaldIs performance tuning related stuff wanted in the code?18:16
*** lamt has joined #openstack-kolla18:17
mewaldWe've made some experience lately regarding mariadb in this context which I would share if wanted18:17
*** huikang has quit IRC18:17
*** sdake_ has joined #openstack-kolla18:19
*** harlowja_still_a has joined #openstack-kolla18:20
*** sdake has quit IRC18:20
*** Serlex has quit IRC18:21
inc0_mewald, depends on change I guess18:23
inc0_although I'd rather keep conf changes to minimum18:23
*** harlowja_at_home has quit IRC18:23
openstackgerritMathias proposed openstack/kolla: Mariadb performance tuning  https://review.openstack.org/38613018:24
mewaldinc0_: I just submitted patch. Let's see what people say about it. All I can say is: we had a lot of api timeouts and that stuff solved it and gave us better insight into maridb performance18:24
inc0_mewald, will that break upgrade?18:25
inc0_I mean if mariadb was running without these confs, will it keep running?18:25
mewaldinc0_: I don't think I understand you. Mariadb never crashed for us but responded very very slowly so that api requests timed out18:26
*** shardy has quit IRC18:26
inc0_mewald, what I mean, if someone had running mitaka18:27
inc0_and add these confs18:27
inc0_their data is safe right?18:27
mewaldI am not a mariadb specialist but I see nothing bad about increasing buffers :)18:28
inc0_yeah I don't suppose that is the case too18:28
inc0_just wanted to be sure18:28
openstackgerritMathias proposed openstack/kolla: Mariadb performance tuning  https://review.openstack.org/38613018:28
mewaldas I said: I am not 100% sure and I dont want to be charged guilty when something breaks :D But I've talked to some mariadb ops who suggested this stuff18:28
*** harlowja_still_a has quit IRC18:30
mewaldinc0_: I also noticed that query_cache_size should be set to 0 when running Galera but the default is 104857618:31
*** dwalsh has quit IRC18:31
SamYaplemewald: nah the default is disabled when using galera18:33
SamYapleotherwise youd be seeing some deadlocks18:33
mewaldSamYaple: Ah does it do that automatically? fine then :)18:33
mewaldso that line is not necessary, will remove it from the patch18:33
openstackgerritMathias proposed openstack/kolla: Mariadb performance tuning  https://review.openstack.org/38613018:35
SamYaplemewald: its really strange that you want to enable the logging of every query....18:35
SamYapleor if you dont the last 4 options are not needed18:35
SamYaplei dont think general-log takes params and the fact that you have it there may be enabling it18:35
mewaldSamYaple: it have us great insight into what's happening from our dashboards. does it draw a lot of performance you think?18:36
mewalds/have/gave/18:37
SamYaplemewald: it shouldnt be default is my point18:37
SamYapleits not on default for a reason18:37
mewaldok makes sense, if people want it they can use ini merge to get it in18:38
mewaldwannt get rid of performance_schema = on as well?18:38
mewaldas I am on it I can do it right away :)18:38
openstackgerritMathias proposed openstack/kolla: Mariadb performance tuning  https://review.openstack.org/38613018:41
SamYaplemewald: i posted a reply to PS318:42
SamYaplethe first 4 options are good, the rest really shouldn't be touched18:43
SamYapleor in the patchset that is18:43
SamYaplethe only really questionable one is skip-name-resolve18:43
SamYapletechnically you can point to that being a bad environment configuration, but if lots of people have problems with that then maybe it should be the default for kolla18:44
SamYaplei dont think thats the case though18:44
openstackgerritJustin Scott proposed openstack/kolla-kubernetes: Add workflow for cinder  https://review.openstack.org/38614418:44
*** rhallisey has quit IRC18:45
SamYapleinc0_: mewald: do note, that on AIO type systems that share compute, you are giving 40% of the ram to mariadb with this configuration. so maybe think through if that needs to be a default, or make it more intelgent18:46
SamYaplelike 40% or 8GB, which ever is lower18:46
*** rhallisey has joined #openstack-kolla18:46
SamYapleini can still override it (I have set these settings since the beginning with Kolla if that matters)18:47
*** salv-orlando has joined #openstack-kolla18:47
*** michauds has joined #openstack-kolla18:49
openstackgerritMathias proposed openstack/kolla: Mariadb performance tuning  https://review.openstack.org/38613018:49
mewaldSamYaple: AIO is indeed something I didn't think about18:50
openstackgerritMathias proposed openstack/kolla: Mariadb performance tuning  https://review.openstack.org/38613018:57
mewaldSamYaple: Please check it that works for you18:57
mewaldif18:57
openstackgerritMathias proposed openstack/kolla: Monitor mariadb with telegraf  https://review.openstack.org/38611918:58
SamYaplemewald: reviewed. lgtm18:59
*** sdake has joined #openstack-kolla19:01
mewaldcool!19:01
*** dwalsh has joined #openstack-kolla19:02
*** sdake_ has quit IRC19:04
*** dwalsh has quit IRC19:08
*** dwalsh has joined #openstack-kolla19:09
openstackgerritMathias proposed openstack/kolla: Use memcache for keystone token persistence  https://review.openstack.org/38558619:10
janemIm getting a timeout for haproxy - waiting for vip .. aio 2.0.2 ubuntu host.. prechecks are fine.19:10
janemdeploying, that is..19:11
mewaldjanem: which task exactly? can you paste the output somewhere?19:11
*** pbourke has quit IRC19:11
*** pbourke has joined #openstack-kolla19:11
janemmewald: yeah, just waiting for it to timeout again :)19:12
janembut the task is TASK: [haproxy | Waiting for virtual IP to appear]19:13
mewaldjanem: it's all-in-one, right? Which environment are you using? VirtualBox?19:13
*** rhallisey has quit IRC19:13
janemUbuntu - Metal19:13
janemyes aio19:14
mewaldis the keepalived container running?19:14
mewaldin both cases: see any errors?19:15
janemmewald: http://pastebin.com/WRqEAakJ19:15
janemmewald: keepalived is running19:15
janemat least now after failed deploy..19:16
mewald"ip addr show" shows the VIP?19:16
janemmewald: no, actually not19:17
janemmewald: shouldnt that have been taken care of during the deploy?19:18
mewaldyeah when keepalived starts that should be done19:19
janemhmz19:19
mewaldcheck your keepalived config and output when running "docker start -a keepalived"19:19
janemmewald: http://pastebin.com/QvL8MtaU19:21
*** ssurana has quit IRC19:21
*** papacz has joined #openstack-kolla19:24
mewaldjanem: never seen that before19:25
janemme neighter ..searching..19:25
mewaldme too ^^19:26
mewaldcan you give me your keepalived.conf and output if "ip addr show"?19:27
janemmewald: http://pastebin.com/TM7wG3LL19:29
inc0_janem, do you have other keepalived runinng in same network?19:30
janeminc0_: yes19:31
inc0_like for example another kolla?19:31
inc0_that's the issue.19:31
inc0_you need to change this variable in your globals.yml:19:31
inc0_https://github.com/openstack/kolla/blob/master/etc/kolla/globals.yml#L9119:31
inc0_you need to put a different number there19:32
inc0_each keepalived cluster has to have unique router id19:32
*** papacz has quit IRC19:32
mewaldhaha nice inc0_! I would never have thought of this :D19:33
inc0_mewald, I've seen that one before19:33
janeminc0_: ahh.. makes sense..19:33
janeminc0_: thanks, trying again now.19:33
mliimaguys, I need some reviews here or just +Workflow. https://review.openstack.org/#/c/382054/19:33
inc0_here you go mliima19:34
janeminc0_: mewald shabammm! works. Thanks a million.19:34
janem:)19:34
inc0_glad to hear that19:34
mliimathanks inc0_ \o/19:35
inc0_people ask about that from time to time here19:36
inc0_if you see "bogus packet" something something in keepalived logs19:36
inc0_that's probably it19:36
openstackgerritMerged openstack/kolla: Add HNAS as Manila backend  https://review.openstack.org/38205419:37
*** narasimha_SV_ has joined #openstack-kolla19:40
narasimha_SV_http://paste.openstack.org/show/585650/ anyone working on ironic in kolla is getting this issue ?19:40
janeminc0_: hopefully I will be able to help the next guy/gal ;)19:43
inc0_janem, that's how it works, one time you need help another time you help somebody;)19:44
janeminc0_: yup!19:45
*** goldyfruit has joined #openstack-kolla19:48
goldyfruithaplo37 https://blueprints.launchpad.net/kolla/+spec/heka-deprecation19:50
*** jtriley has quit IRC19:57
goldyfruityop EmilienM20:00
goldyfruit<- Gaetan20:00
EmilienMgoldyfruit: what's up?20:00
goldyfruitEmilienM: Start ny new job at Omruco20:01
goldyfruitOrmuco20:01
goldyfruitwhat about you ?20:01
*** janem has quit IRC20:02
bjolopbourke, about the TLS cert. i have tried both selfsigned and a real one. same result20:13
*** ppalacios has quit IRC20:15
*** rhallisey has joined #openstack-kolla20:16
*** openstackgerrit has quit IRC20:19
*** openstackgerrit has joined #openstack-kolla20:19
*** unicell has quit IRC20:20
*** peterjenkins_ has quit IRC20:20
*** peterjenkins_ has joined #openstack-kolla20:23
*** unicell has joined #openstack-kolla20:23
*** unicell1 has joined #openstack-kolla20:24
*** unicell has quit IRC20:24
*** janem has joined #openstack-kolla20:32
*** salv-orlando has quit IRC20:36
*** narasimha_SV_ has quit IRC20:40
*** mliima has quit IRC20:41
*** mark-casey1 has joined #openstack-kolla20:41
*** sdake has quit IRC20:44
*** ssurana has joined #openstack-kolla20:47
*** salv-orlando has joined #openstack-kolla20:48
*** gfidente has quit IRC20:51
*** huikang has joined #openstack-kolla20:52
*** mewald has quit IRC20:56
*** sdake has joined #openstack-kolla20:59
*** mewald has joined #openstack-kolla20:59
openstackgerritJustin Scott proposed openstack/kolla-kubernetes: Ansible workflow for Neutron  https://review.openstack.org/38622621:00
*** janem has quit IRC21:01
*** ayoung has quit IRC21:02
*** srwilkers has left #openstack-kolla21:02
*** dave-mccowan has quit IRC21:04
*** dwalsh has quit IRC21:08
*** steve-noyes has joined #openstack-kolla21:08
*** papacz has joined #openstack-kolla21:13
*** absubram has quit IRC21:18
*** steve-noyes has quit IRC21:23
*** fguillot has quit IRC21:29
*** michauds has quit IRC21:30
*** jheroux has quit IRC21:34
*** Jeffrey4l__ has quit IRC21:35
*** Jeffrey4l_ has joined #openstack-kolla21:35
*** huikang has quit IRC21:38
*** huikang has joined #openstack-kolla21:38
*** Pavo has quit IRC21:40
*** JoseMello has quit IRC21:41
*** huikang has quit IRC21:42
*** Pavo has joined #openstack-kolla21:44
openstackgerritJustin Scott proposed openstack/kolla-kubernetes: Add workflow for cinder  https://review.openstack.org/38614421:46
*** Pavo has quit IRC21:51
*** Pavo has joined #openstack-kolla21:56
*** ssurana has quit IRC21:57
openstackgerritJustin Scott proposed openstack/kolla-kubernetes: Ansible workflow for Neutron  https://review.openstack.org/38622622:01
*** tonanhngo has quit IRC22:04
*** papacz has quit IRC22:08
*** tonanhngo has joined #openstack-kolla22:09
*** huikang has joined #openstack-kolla22:12
*** tonanhngo has quit IRC22:14
*** tonanhngo has joined #openstack-kolla22:15
*** huikang has quit IRC22:17
*** mgoddard has quit IRC22:18
*** inc0_ has quit IRC22:19
*** tonanhngo has quit IRC22:19
*** Pavo has quit IRC22:22
*** rhallisey has quit IRC22:24
*** ayoung has joined #openstack-kolla22:25
*** ssurana has joined #openstack-kolla22:26
*** Pavo has joined #openstack-kolla22:28
*** dwalsh has joined #openstack-kolla22:29
*** salv-orlando has quit IRC22:30
*** ntpttr has quit IRC22:31
*** ntpttr has joined #openstack-kolla22:31
*** Pavo has quit IRC22:33
*** mewald has left #openstack-kolla22:34
*** quasar has quit IRC22:35
*** msimonin has quit IRC22:40
*** Pavo has joined #openstack-kolla22:41
*** quasar has joined #openstack-kolla22:42
jascott1https://blog.buoyant.io/2016/10/04/a-service-mesh-for-kubernetes-part-i-top-line-service-metrics/22:46
*** tonanhngo has joined #openstack-kolla22:59
*** dwalsh has quit IRC23:00
kfox1111jascott1: https://xkcd.com/927/23:00
*** lamt has quit IRC23:00
kfox1111:)23:00
kfox1111jascott1: looks interesting. but also seems to have a lot of overlap with what k8s provides itself.23:02
jascott1haha thats a great one on the short list. yeah idk either but seems to be generating a lot of buzz today Hightower et all23:03
*** ayoung has quit IRC23:03
kfox1111the gui bits look sweet.23:03
jascott1the premise is interesting, abstracting the retries and 'communication' management23:04
kfox1111yeah.23:04
jascott1also impressive if you really can just run 3 apply commands and get all that goodness23:04
kfox1111that may speak more to the power of k8s then linkd... but, yeah. :)23:05
kfox1111linkerd I mean.23:05
kfox1111the argument feels much like the argument that has been made around most middleware. leave it to us, so you don't have to figure out the hard bits.23:06
kfox1111which is a good arguement.23:06
jascott1:)23:07
kfox1111but has spawned thousands of 'standards' :)23:07
jascott1yeah I havent even learned all the existing network overlay stuff let alone abstracting pieces of it away!23:08
* kfox1111 nods23:08
kfox1111I think its going to change quite a bit too once k8s gets multi l2 network support, ala neutron.23:09
jascott1cool23:09
*** Pavo has quit IRC23:13
*** asalkeld has joined #openstack-kolla23:13
*** Pavo has joined #openstack-kolla23:14
*** igordcard has quit IRC23:14
*** igordcard has joined #openstack-kolla23:16
*** awiddersheim has joined #openstack-kolla23:18
*** mtaylor22 has joined #openstack-kolla23:19
mtaylor22morning.23:19
jascott1o/23:19
awiddersheimhaving some trouble with RC223:21
awiddersheimtrying to do a new setup and things fail on keepalived and waiting for the VIP23:21
*** v1k0d3n has joined #openstack-kolla23:22
jascott1awiddersheim I have the same problem but i dont fully know what I am doing23:22
awiddersheimbut the containers aren't making any logs for me to trouble shoot23:22
jascott1as I understand it, its waiting for the mariadb ip and port to become available23:23
jascott1I havent figured out how you know what that host is tho23:23
awiddersheimyeah23:23
v1k0d3nhey sdake who was it that we were on a webex a couple of weeks ago that was helping with multi-kube install of kolla-kube? i can't remember the name.23:24
awiddersheimdoesn't even seem like it made it far enough to stand that container up23:24
*** vhosakot has quit IRC23:24
*** mtaylor22 has quit IRC23:27
*** frzn_ has joined #openstack-kolla23:27
*** frzn_ is now known as mtaylor2223:27
kfox1111v1k0d3n: might have been sbezverk_ or me.23:27
*** Pavo has quit IRC23:28
*** Pavo has joined #openstack-kolla23:28
*** srwilkers_ has joined #openstack-kolla23:28
*** lamt has joined #openstack-kolla23:28
*** srwilkers_ has left #openstack-kolla23:29
v1k0d3nso wanted to get your thoughts on using this repo as a base for developing multi-node kube-kolla...23:30
v1k0d3nhttps://github.com/att-comdev/halcyon-kubernetes23:30
v1k0d3nkfox1111: it was you :)23:30
v1k0d3nnow i remember! lol23:30
v1k0d3ni reached out to sebgoa about merging efforts with his repo too...https://github.com/skippbox/kubeadm-centos23:31
*** neilus has joined #openstack-kolla23:32
*** neilus has quit IRC23:36
v1k0d3nkfox1111: thoughts...good/bad? something that could be useful for kolla-kube perhaps?23:37
kfox1111looking...23:37
kfox1111docs talk a lot about vagrent. is that just for testing?23:39
kfox1111seems to only support ubuntu at the moment, and for some reason installs docker-compose23:40
kfox1111in general though, looks quite promicing.23:40
kfox1111the docs make it seem like they really intend it for vagrent setup.23:42
v1k0d3nso here's my actual goal with it.23:42
v1k0d3nbreak away from vagrant. vagrant will be on it's own...and submodule for the playbooks.23:43
* kfox1111 nods23:43
kfox1111shouldn't be too hard by the look of it.23:43
kfox1111though my ansible's pretty green at this point, so take that with a grain of salt. :)23:43
v1k0d3nso they can be maintained separately. however, vagrant is multi-provider. so with the config.rb...you can use the --provider=openstack and deploy directly to openstack for development.23:44
v1k0d3ni will be adding aws either tomorrow or early next week, to deploy directly to aws with some config options.23:44
kfox1111yeah... Ive tried vagrent pariodically against kvm... never really got it to work well. :/23:44
*** ayoung has joined #openstack-kolla23:44
kfox1111kind of gave up on vagrent for a while.23:44
v1k0d3nkfox1111: no worries...it just works out of the box (or should anyway).23:45
v1k0d3ninterested if you have issues.23:45
kfox1111cool. I might have to give it another shot then. :)23:45
v1k0d3nif you do, let me know...i'll correct them.23:45
kfox1111I just REALLY didn't want to have to install virtualbox.23:45
v1k0d3ni think it would have a good place to start for kolla-kube multi-node development.23:46
v1k0d3nlol23:46
v1k0d3nyeah, i can understand that. but with vbox extensions...it just works so well.23:46
v1k0d3n^^ hating that i like any product from...*coughs*23:46
kfox1111I talked with the infra folks and they can't do vm creation in vm without doing non accelerated.23:46
kfox1111v1k0d3n: until openstack has a virtualbox driver, I can't run openstack and virtualbox on the same laptop.23:47
kfox1111so multi-node deployment with vagrent won't work in the gate.23:47
kfox1111but I did get some info out of them how to get two vm's launched at the same time and get ssh between them.23:47
kfox1111so that should be enough to just ssh in and run the kubeadm join.23:48
v1k0d3nwhat is your setup again? just curious.23:48
kfox1111the rest should deploy with daemonsets, so should just work I think.23:48
v1k0d3nyeah daemonsets is good stuff.23:49
kfox1111for what? my personal dev system, what I t est with, production system, etc?23:49
*** sdake_ has joined #openstack-kolla23:51
v1k0d3nif you'd use something like this i mean.23:52
v1k0d3ni mean, we could just use the playbooks for setting up the kubeadm portion on existing systems. pretty straight forward.23:52
*** sdake has quit IRC23:52
kfox1111yeah. I think that would be useful.23:53
v1k0d3nor terraform could be used, vagrant can be used....whatever. i do most of my development in openstack projects...23:53
v1k0d3nso i use this vagrant up in an openstack project. but then there are times i need to do local development, local hypervisor. so i use vbox.23:53
kfox1111my goal right now is to get the gate up and going to the point, where that's the easiest way to test. just submit a review, and it will be tested. :)23:53
v1k0d3nbut playbooks at least stay the same...portable.23:53
* kfox1111 nods23:54
v1k0d3ngotcha...makes sense.23:54
kfox1111kola-kubernetes doesn't have any k8s deployment stuff.  not sure it really makes sense to include it directly, as there are so many ways.23:54
kfox1111but pointing at an ansible based deployment tool sounds good to me.23:55
v1k0d3nbtw, i already reached out to sebastian to merge these https://github.com/skippbox/kubeadm-centos23:55
kfox1111"If you don't aready have a k8s and want something quick that will just work, go here"23:55
v1k0d3nso it would cover both ubuntu and centos deployments of kubeadm23:55
v1k0d3n^^ exactly...that's the scenario i'm talking about.23:56
kfox1111yeah. that would be cool.23:56
*** cj has quit IRC23:56
v1k0d3nfor instance...my guys wanted to start contributing to kolla-kube...but they were new to everything. so they needed to find a way to deploy kubernetes.23:56
kfox1111right. :)23:57
v1k0d3nminikube is great...but not the same issues as a multi-node.23:57
kfox1111yeah. totally.23:57
kfox1111to be fair, most of that was the focus only on single node for now.23:57
kfox1111:)23:57
kfox1111that is changing, as single node is now working pretty well, and has gate checks now.23:57
kfox1111so I'm moving on to multinode next.23:57
kfox1111and others are working on the ansible worflow so that should get much easier too.23:58
kfox1111the multinode stuff really hinges on the ceph stuff getting ironed out, and that wasn't working until a few days ago.23:58
kfox1111that code merged though, so shouldn't regress.23:58
kfox1111it just needs a bit more polish and hooking up to the workflow now.23:59
kfox1111but mariadb backed by ceph, and laungching a vm with image stored in ceph and cinder volume attachment/movement where the volume is in ceph all work.23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!