Wednesday, 2017-04-05

rwellumsdake: cherry pick is still wrong imo (in the doc)00:04
sbezverkkfox1111: I have an idea, what if for multinode we finish master with all running and only then join second node00:04
spsuryasdake: hi...00:04
sbezverkwe see single node works 100% of cases00:04
*** rstarmer has joined #openstack-kolla00:05
kolla-slack<kfox1111> The idea of multinode was to test out split controller/compute.00:05
kolla-slack<kfox1111> Not sure what good a single node but later with 2 really buys?00:06
spsuryasbezverk:   nice          but i just got into this        [preflight] Some fatal errors occurred:00:07
spsurya/proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 100:07
spsurya/var/lib/etcd is not empty00:07
kolla-slack<kfox1111> I was going to suggest the opposite before 1.6, to remove more of the single node tests.00:07
spsuryasbezverk: did you guys faced this ?00:07
sbezverkkfox1111: no I meant only the way to isntall it00:08
sbezverkright now when we install, we install both nodes before networking00:08
sbezverkI run one node + networking and then add second node00:09
sbezverkI think it will be more reliable at the gate00:09
sbezverkspsurya: make sure you run the latest kube buts00:09
sbezverkbits :)00:09
spsuryasbezverk: 1.6.1                Package ebtables-2.0.10-15.el7.x86_64 already installed and latest version00:11
spsuryaPackage kubeadm-1.6.1-0.x86_64 already installed and latest version00:11
spsuryaPackage kubectl-1.6.1-0.x86_64 already installed and latest version00:11
spsuryaPackage kubelet-1.6.1-0.x86_64 already installed and latest version00:11
spsuryaPackage kubernetes-cni-0.5.1-0.x86_64 already installed and latest version00:11
spsuryaNothing to do00:11
sbezverkspsurya: not sure then..00:12
kfox1111sbezverk: still don't follow. are you talking about just k8s?00:12
sbezverkhave you tried to set it /proc/sys/net/bridge/bridge-nf-call-iptables ?00:12
kfox1111like, ensure calico is up before doing the second node?00:12
sbezverkkfox1111: exactly00:12
sbezverkI tested it in mu local lab and it works00:13
spsuryasbezverk: usually we don't set00:13
kfox1111sbezverk: ah.... couldn't hurt. and would remove one variable.00:13
spsuryabut i can give a try after change00:13
sbezverkI thinking since our goal is not to test kubeadm installation process00:13
sbezverkwe could try to optimize it a bit00:13
kfox1111works for me.00:14
spsuryasbezverk: roger00:14
kfox1111weird.... http://logs.openstack.org/77/453277/1/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/b28ddb2/logs/pods/kube-system-kube-dns-3913472980-1k2f7.txt00:16
kfox1111so... it failed to contact kube-apiserver...00:18
kfox1111looks like it started on the slave system..00:20
sdakerwellum was OTP00:21
sdakerwellum back - looking00:21
sdakesup spur00:21
sdakesup spsurya00:21
kfox1111interesting....00:21
sdakespsurya what can i do for ya00:21
*** yingjun has joined #openstack-kolla00:21
*** yingjun has quit IRC00:22
spsuryasdake: looking into this00:22
*** qiliang27 has quit IRC00:22
openstackgerritSteven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation  https://review.openstack.org/44735600:24
sdakekfox1111 sbezverk can you fellas ack that above review and approve it so we can get a rendering on docs.oo00:24
sdakekfox1111 i will continue to work on it from this point and solidify it00:24
sbezverksdake: I have not tested it, can you ask folks who tested it at least +1 it if they are happy?00:25
sdakesbezverk its    hard to tet without a rendering on docs.oo00:25
sdakethere is a warning at tthe top that indicates it may not work perfectly00:25
kfox1111well, I do plan on kind of reusing the gate to ensure we cache a tested k8s.00:25
sdakesbezverk i will commit to having it in good shape by friday before i travel00:26
sdakeas in people will have tested it completely and all the kinks worked out00:26
sdakei've got wed-fri to do that, and only one dentist appointment in the way :)00:26
sdakesbezverk fwiw I have tested it ,and it wfm00:27
sdakeand also eanlyin got it working but is afk atm00:27
rwellumsdake it's -1 for now because that cherry-pick is wrong imo00:27
sdakerwellum see latets review- that is fixed00:28
sdakerwellum thanks for finding that btw, not sure why that snuck back in00:28
sdakethings keep "sneaking back in' because its not merged00:28
sdakeif i can merge it i have smoething to work from in git diff wise00:28
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338400:29
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Disable placement for v3 gate.  https://review.openstack.org/45327700:31
rwellumsdake: looks better thanks. I can +1 - even though I haven't got it working fully myself.00:31
sdakerwellum if your up to working on it today - we can tackle i t together via webex00:31
sdakerwellum it might help if i see if your doing something different then I do00:31
*** ksumit has quit IRC00:32
rwellumsdake: currently assisting Daughter with Earth Environment Science Honors lab :) But I can try at 9pm EST?00:33
sdakeso 30 minutes?00:33
sdakei'm around all night00:33
rwellumyes that works00:34
*** yingjun has joined #openstack-kolla00:34
rwellumI'll ping you.00:34
*** yuanying has joined #openstack-kolla00:34
*** lucasxu has joined #openstack-kolla00:36
*** lucasxu has quit IRC00:40
kfox1111sdake: a few things on the review, then I'm good as per our previous conversation.00:45
sdakekfox1111 got it00:45
*** xiaoruiguo has joined #openstack-kolla00:46
*** zhurong has joined #openstack-kolla00:47
*** jascott1_ has joined #openstack-kolla00:47
*** hrw has quit IRC00:48
*** tovin07_ has joined #openstack-kolla00:49
japestinhosdake I was asked a username by gerrit when exec git review -x 439740, I've never used gerrit before yet :(00:50
sdakejapestinho you ahve to have your ssh environment setup00:51
sdakejapestinho and gerrit needs to be setup00:51
sdakean alternatie is to go find the review and use the cherrypick feature00:51
sdakeleave a comment and I will add a note for those that don't have gerrit available.00:51
sdake(in the rview leave a comment)00:52
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add Glance Swift backend support  https://review.openstack.org/45205900:53
sdakekfox1111 thanks for the review, i'm struggling (with my lack of networking experience) to know exactly what needs to be done for the broad direction you have given.  could you narrow it down a bit :)00:53
mnaseraw man00:53
mnasernova_compute on stable/newton + ceph => broke00:53
mnaser:-(00:53
mnasercontainer runs under user nova00:54
mnasertries to delete /etc/ceph on start00:54
mnaserthe file /etc/ceph/rbdmap cannot be modified by the nova user00:54
*** hrw has joined #openstack-kolla00:54
*** jascott1_ has quit IRC00:56
*** qiliang27 has joined #openstack-kolla00:57
mnaserthe file /etc/ceph/rbdmap cannot be modified by the nova user00:57
*** eanylin has joined #openstack-kolla00:58
japestinhosdake my comment still in draft, how to commit it?00:58
japestinhohttps://www.irccloud.com/pastebin/hTckYXZ0/comment%20on%2044%20patch%20set00:58
sbezverkkfox1111: it looks like that little thing stabilized multinode gate jobs.00:59
rwellumjapestinho: on the main review page, there's a reply button at the top.00:59
spsuryamnaser: not sure but seems like some permission issue00:59
rwellumpress that, answer -1,0 or +100:59
mnaserspsurya it is.. but it looks like the whole codebase is broken00:59
mnaserbecause i fail to see how any ceph deployment would have successfully completed.01:00
sdakejapestinho click reply01:00
*** qiliang28 has joined #openstack-kolla01:00
sdakelooks like rwellum answered :)01:00
*** hrw has quit IRC01:00
*** qiliang27 has quit IRC01:01
*** qiliang28 is now known as qiliang2701:01
*** hrw has joined #openstack-kolla01:02
spsuryamnaser: hmmm01:02
mnaserspsurya strangely enough cinder volume worked?01:02
japestinhorwellum thanks but I forgot give answer -1 or +1 , my bad :(01:03
*** Pavo has joined #openstack-kolla01:03
rwellumjapestinho: just edit one of your comments, do the reply again.01:04
rwellumsdake: what is the tox command to render the doc correctly please?01:04
sdakerwellum tox -e docs01:05
sdakecd doc/build01:05
sdaketar -cvzf html.tar.gz html01:05
sdakescp the html.tar.gz to your local machine01:05
sdakethen untar them01:05
sdakethen open in chrome or hte like01:06
spsuryamnaser: nice01:06
spsuryaso issue resolved at a level01:06
mnaserim so confused as to why one worked and one didnt?01:07
japestinhorwellum sorry but actually what's the meaning -1/+1 thing? just check begore I reply again01:07
mnaserdocker run -it --rm kolla/centos-binary-cinder-volume:3.0.3 rm -rfv /etc/ceph01:08
rwellum+1 you're happy, -1 unhappy :)01:08
rwellumprobably sdake can answer that better01:08
*** duonghq has joined #openstack-kolla01:09
*** iceyao has joined #openstack-kolla01:10
rwellumsdake: i got many pages of angry red output from that tox command and no doc/build01:11
*** jascott1_ has joined #openstack-kolla01:11
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338401:12
spsuryajapestinho: +1 means you have reviewed the patch and  agreed that it is fine from your view. -1 means you have some concern with the changes in patch set that you will tell via firing the comment on that patch.01:12
sbezverkrwellum: deployment guide is for single node only or for multinode as well?01:13
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939501:13
spsuryaand that would be incorporated by the committer01:13
rwellumjapestinho: recommend you try going through https://docs.openstack.org/infra/manual/developers.html - will answer a lot of questions about how to set this up, environment etc.01:14
spsuryaif he finds that the review commnet is right01:14
spsuryahe will incorparate your comment01:14
rwellumsbezverk: originally sdake was talking about 2 bare-metal nodes. That's been removed since then. Personally I've been following it as a single node deployment.01:16
mnaserspsurya trying to emulate what kolla-config does... docker run -it --rm kolla/centos-binary-cinder-volume:3.0.3 rm -rfv /etc/ceph01:17
mnaserrm: cannot remove '/etc/ceph/rbdmap': Permission denied01:17
sbezverkrwellum: got it, thanks.01:17
sbezverkif it goes back to multinode, then we need to change a bit the way nodes join the cluster..01:18
openstackgerritjimmygc proposed openstack/kolla master: Add vSphere Support  https://review.openstack.org/44939501:20
spsuryamnaser: may be egonzalez can help better01:21
mnaseri was hoping to do a test deployemnts today01:21
spsuryayes01:23
spsuryanice01:23
mnaseri wonder how cinder-volume worked before01:24
rwellumsbezverk: yes - the 'joins' become very important...01:25
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add Glance Swift backend support  https://review.openstack.org/45205901:27
spsuryamnaser: kolla cinder log will certainly help in that01:28
mnaserINFO:__main__:Removing existing destination: /etc/ceph/01:28
mnaserINFO:__main__:Creating dest parent directory: /etc/ceph01:28
mnaserwhich means that kolla-config was able to successfully do this01:28
*** hrw has quit IRC01:29
spsuryaright01:31
*** hrw has joined #openstack-kolla01:31
sdakerwellum the red is normal01:37
*** shashank_t_ has quit IRC01:37
sdakerwellum it sould have spit out a buidl dire01:37
sdakesbezverk its AIO at present - may be expanded to multinode later01:37
mnaserhttps://bugs.launchpad.net/kolla-kubernetes/+bug/165919801:38
openstackLaunchpad bug 1659198 in kolla-kubernetes "nova-compute can not run when use ceph" [Undecided,Confirmed]01:38
mnaserlooks like its present there too01:38
sdakegoal is tjust to get *something* that documents deployment01:38
sdakeas currently the devenv is busted and deployment doesn't work01:38
sdakeanywhere01:38
mnasersdake are you doing any ceph-related deployemnt in k8s right now?01:38
sdakerwellum did you miss this part of the guide: Load the kubedm credentials into the system::01:39
sdake    sudo cp /etc/kubernetes/admin.conf $HOME/01:39
sdake    sudo chown $(id -u):$(id -g) $HOME/admin.conf01:39
sdake    export KUBECONFIG=$HOME/admin.conf01:39
sdakerwellum with that (and with the insecure port thing left as is) i can connect - the --insecure-port=0 is intentional as kubeadm 1.6.1 added that dmin.conf feature01:39
sdakekubectl reads env variable KUBAECONFIGA01:40
sdakerwellum i fixed one typo and replced it with another 9firewalld:)01:42
sdakeold and tired i am01:42
rwellumsdake: the issue might be I am running as sudo su -01:44
rwellum$HOME is /root etc01:44
rwellumnot sure01:44
eanylinsbezverk: Is this needed?01:46
eanylinkollakube res create secret nova-libvirt01:46
eanylinsdake: I saw that typo, was able to put a comment on it :)01:46
*** cuongnv has joined #openstack-kolla01:48
sdakerwellum run as a normal user01:49
sdakerwellum no need ot run as root01:49
sdakethat is what the sudos are for :)01:50
sdakesudo su - is definately wrong - try sudo -i01:50
sdakeif you want to run as root01:50
sdakei'd highly recommend running as a normal user though and using the sudo commands as they are specified01:50
sdakesome commands don't work at all with sudo01:50
sdakefor example I ran the adminopenrc creation as sudo and it craters the openstack deployment01:50
sdakecan some brilliant networking dude explain to me how to pick a service network01:51
sdakekfox1111 seemed to indicate this is easy01:51
sdakeI guess it is for someone that understands networking in detial01:51
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Add VMware DataStore support to glance  https://review.openstack.org/45217601:52
*** zhurong has quit IRC01:52
sdakei have no idea how to pick that service addr, and as a result have no idea how to change the document to match01:52
rwellumsdake: ack - I had reasons - running in a vm is slightly different but will try01:52
sdakerwellum just do sudo -i01:53
sdakeadduser rwellum01:54
sdakethen exit01:54
sdakethen su - rwellum01:54
rwellumsdake: ack01:54
sdakedont forget to add  rwellum to the wheel group :)01:54
sdakeand enable the sudoers file01:55
sdakerwellum out of curiosity what is the rationale for running as sudo su -?01:56
*** masber has quit IRC01:57
rwellumsdake: you had some commands with sudo and some not, wasn't consistent. So instead of adding sudo everywhere I just ran as root.01:57
*** zhurong has joined #openstack-kolla01:57
rwellumI commented on a few01:57
sdakerwellum right - the commands are now completely consisten01:57
sdakerwellum everything should work as a non-root user01:58
sdakeif they dont they should be fixed :)01:58
sdakerending latest docs and running through guide locally now01:58
spsuryasdake: need to clean the directory `/var/lib/etcd` too in `kubeadm reset`01:59
spsuryagot into this                    [centos@ip-172-31-13-146 kollaK8s]$ sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/1602:00
spsurya[kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters.02:00
spsurya[init] Using Kubernetes version: v1.6.002:00
spsurya[init] Using Authorization mode: RBAC02:00
spsurya[preflight] Running pre-flight checks02:00
spsurya[preflight] Some fatal errors occurred:02:00
spsurya/var/lib/etcd is not empty02:00
spsurya[preflight] If you know what you are doing, you can skip pre-flight checks with `--skip-preflight-checks`02:00
*** eaguilar has quit IRC02:00
spsuryaI re run the command after cleaning02:00
sdakespsurya did you clean up with kubeadm reset first?02:00
spsuryathen it worked02:00
spsuryasdake: yes02:01
sdakekubeadm init may not be rerun after its run02:01
sdakea reset step is needed first02:01
sdakekubeadm reset02:01
spsuryasdake: that what i did02:01
rwellumsdake: I am a getting this for the last few hours - seen this before? https://www.irccloud.com/pastebin/PRLXWXqy/02:01
sdakeya - i dont think kubeadm reeset fuly cleans up02:01
sdakespsurya which versiosn of kubeadm do you ahve?02:02
spsuryasdake: 1.6.102:02
sdakerwellum gah - no idea - let me try a fresh install02:02
eanylinhttps://kubernetes.io/docs/getting-started-guides/kubeadm/#tear-down02:02
eanylinspsurya: There are some extra steps02:02
eanylinit seems, for the new version02:03
*** unicell has quit IRC02:03
rwellumsdake: work-around is to set the gpg key to 002:03
sdakespsurya ther eis always tools/cleanup-k8s.sh :)02:03
spsuryasdake: i did in between :) i mean before getting kolla-k8s repo i.e before step 2 and step 302:06
sdakespsurya there is another step02:06
sdakesudo yum uninstall -y docker ebtables kubeadm kubectl kubelet kubernetes-cni02:06
sdakekolla-k8s.sh needs that sudo yum install02:06
*** Pavo has quit IRC02:06
openstackgerritZeyu Zhu proposed openstack/kolla-kubernetes master: Uodate the version of cirros  https://review.openstack.org/45343202:07
sdakecleanup-k8s.sh that is02:07
rwellumsdake: running as 'me'  I can confirm that the workaround mentioned does not seem to be working  - I cannot connect02:08
sdakerwellum let me run through the directions, I am certian i confirmed it02:08
spsuryasdake: eanylin this is what i tested http://paste.openstack.org/show/605450/02:08
spsuryaand how that worked02:09
eanylinah, ok, remove everything under /var/lib/etcd/*02:09
spsuryaeanylin: yes, I removed that then got working for me02:10
eanylinspsurya: Ok02:11
spsuryathat is why i was saying that kubeadm reset should clean `/var/lib/etcd/*` too02:11
eanylinspsurya: Ok, that will probably be cleaner then02:13
spsuryaeanylin: yeh02:13
eanylinspsurya: Where you able to get the cluster up since yesterday or?02:13
japestinhothanks so much rwellum and sbezverk for the informations02:13
spsuryaeanylin:  yesterday i tried with 1.5 one02:14
spsuryaand cluster was up02:14
eanylinspsurya: Ok, thats good02:15
spsuryabut till now not with 1.6+02:15
eanylinspsurya: I got busy with work today, didnt really follow the chatroom02:15
spsuryahope i will get that i.e with 1.6+02:15
spsuryaeanylin: i think you got that with 1.6.1 yesterday IIRC02:16
spsuryaright ?02:16
eanylinspsurya: I think it should work. With the latest code from sbezverk, I think there is zero restart02:16
eanylinspsurya: Yeah, http://paste.openstack.org/show/605451/02:17
eanylinso this time round the number of restart is 002:17
spsuryaeanylin: can you please run once `rpm -qa | grep kub1` and paste the output02:18
spsurya`rpm -qa | grep kube`02:18
spsurya*02:19
eanylinhttp://paste.openstack.org/show/605452/02:19
sdakeok need to eat02:19
sdakebbi10 mins02:19
spsuryaeanylin: nice02:21
sdakespsurya i have deployed 1.6.1 and deployed openstack on top and created a vm - the vm creation fails02:21
eanylinsdake: I got the VM up this time round02:21
spsuryaeanylin: what about openstack02:21
spsurya?02:21
sdakeeanylin what is missing from the docs tog et a vm operational02:21
spsuryaeanylin: great02:21
sdakeeanylin i'm crious if it is the /16 subnet specified (vs using the defaults)02:22
eanylinspsuray: http://paste.openstack.org/show/605453/02:22
eanylinsdake: I think I am still having problems though. This time round its with floating IP.02:23
eanylinI am trying different thing, I am using single bridge now02:23
eanylinand its causing some issues02:23
sdakeeanylin what were you doing prior to single bridge?02:23
eanylinsdake: I was using 2 bridges02:23
eanylin1 for OAM/Mgmt02:23
eanylin1 for Neutron02:23
sdakeeanylin you setup a bridge ?02:23
eanylinsdake: Yeah02:24
sdakethat wasn't in the paste ;)02:24
spsuryaeanylin: please save the history and write the comment as per even little problem, you face02:24
eanylinsdake: That is true02:24
spsuryataht would be helpful02:24
sdakesteps 1 and 2 work well now02:24
eanylinspsurya: Yeah, I can do that02:24
sdakestep 3 - needs some love02:24
eanylinmy issue really is, I cannot SSH or ping my cirros VM02:24
sdakeand apparently someone needs to expalin how to fulfill kfox's request02:24
eanylinI still cannot figure out why...02:24
spsuryasdake: seems like we are good to go with merge02:25
eanylinso am trying different things02:25
eanylinsdake: I have something like this on my VM02:26
eanylinhttp://paste.openstack.org/show/605454/02:26
eanylinIt was a config when I used for 1.6.002:26
eanylinbr0 is mgmt02:26
eanylinbr1 is for neutron02:26
eanylinsdake: This time round I did it with just 1 bridge, probably not a good idea, it messed up the floating IP it seems, maybe my config has problem02:27
eanylinsdake: The gate is doing this, I followed the gate02:29
eanylinkubeadm init --skip-preflight-checks --service-cidr 172.16.128.0/2402:30
eanylindo we need /16 ?02:30
sdakenope we apparently need /2402:32
sdakehowever 172.16.128.0 is I believe a gate specific cidr02:32
sdakeI am nto really sure as noboy will tell me :)02:32
eanylinI see02:32
eanylinso seems like --service-cidr (default ‘10.96.0.0/12’)02:33
eanylinthey have it as quite big too02:33
eanylinlike the default values02:33
sdakei hae no idea what a service cidr does02:33
sdakeeanylin check this out:02:33
sdakehttps://review.openstack.org/#/c/447356/02:34
sdakekevin added the comment:02:34
sdakeLine 103:02:35
sdakeI disagree with the assertion it should work on virtually any system. Doesn't work at any of the sites I have k8s deployed.02:35
sdakeI'd rather see this changed to an instruction stating how to pick a range that will work for their site if these defaults don't work, and what needs to change in the instructions for fixing it.02:35
sdakeno idea what to do with that comment02:35
sdakehow does one pick a range?02:35
sdakekfox1111 ^^02:35
sdakeI have asked this over and over, to no avail02:36
sdakeseems like a good goal - however- without approprirate domain knoweldge I am unable to solve the problem02:36
sdakeI dont even know what a servicie-cidr is02:36
sdakeupstream docs are parse02:36
eanylinsdake: Ok. I usually stick to default. But yeah, I agree that there is better approach to just taking default values02:36
sdakeeanylin well the review apparently isnt' merging until this is fixed02:37
sdakeadn I am leaving friday02:37
eanylinsdake: If its single VM, it seems to be working ok02:37
sdakeso - we need to sort this out02:37
eanylinoh man, where to?02:37
sdakeupstream servant leadership training02:37
eanylinsdake: What if we use similar range as the gate? I used the exact same one and it worked fine.02:37
sdakeok - you suggest using a range of the gate02:37
eanylinsdake: Hmm, not sure what is that02:37
sdake*why*02:37
*** zhurong has quit IRC02:38
sdakeeanylin basically 4 days of training02:38
eanylinsdake: I think as long as no IP conflicts, it should be ok, that seems to be what people are saying in forum02:38
sdakeno IP conflicts *where*02:38
sdakewhat is a serice cidr02:38
sdakelets start by answering that question02:38
eanylinsdake: I see, 4 days training02:39
sdakeclearly ips shoudln't have conflicts02:39
eanylinsdake: Right, since we dont have those IPs on our VM02:39
eanylinand we are just doing AIO02:39
eanylinsdake: I probably told you that it works for multi node too, I have compute running on its own and all control on the other VM02:39
eanylinsdake: Using similar settings as Gate02:40
sdakeeanylin you ahve not told me that -however that is great news :)02:40
eanylinsdake: I didnt?02:40
eanylinsdake: Ok, my bad02:40
sdakecopying the gate without understanding hwy is not useful :)02:40
eanylinsdake: But I probably tested with 1.6.002:40
sdaketo answering kevin's response02:40
sdake"tell them how to select a range"02:40
sdakehow means wehave to have knowledge02:40
rwellumsdake - i think if you go /24 etc then there's some defaults in kubernetes/manifests that need prodding. Because they are /1602:40
sdakewe dont have knoweldge02:40
sdakerwellum ack there02:40
eanylinsdake: That is correct, its good to understand the logic02:40
rwellumsdake: >>> https://www.irccloud.com/pastebin/MtFehwOM/02:40
sdakerwellum I can sort the /16->/2402:41
sdakei just dont understnad what a service-cidr is02:41
eanylinrwellum: That is true, which is why the gate made changes02:41
eanylin--service-cidr (default ‘10.96.0.0/12’)02:42
eanylinYou can use the --service-cidr flag to override the subnet Kubernetes uses to assign pods IP addresses. If you do, you will also need to update the /etc/systemd/system/kubelet.service.d/10-kubeadm.conf file to reflect this change else DNS will not function correctly.02:42
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Fix ceilometer meters for swift service  https://review.openstack.org/45237302:42
eanylinso there is a default value, in order to not use default, we need to make changes to 10-kubeadm.conf02:42
sdakeSERVICE_IP_RANGE=10.3.0.0/2402:42
sdakeThe CIDR network to use for service cluster VIPs (Virtual IPs). Each service will be assigned a cluster IP out of this range. This must not overlap with any IP ranges assigned to the POD_NETWORK, or other existing network infrastructure. Routing to these VIPs is handled by a local kube-proxy service to each host, and are not required to be routable between hosts.02:42
sdakeeanylin read the review - it has changed drastically02:43
eanylinsdake: Ok. I havent read the latest.02:43
sdakeeanylin - oh well that epalins it :)02:43
sdakeeanylin its dramatically improved02:43
sdakeeanylin based upon your paste last night i went thorugh and actually figured out how to deploy the whole hting02:44
sdake;)02:44
sdakeeanylin++ for the paste02:44
rwellumsdake: probably not a good time, but I am semi-convinced when I add the kubeadm credentials workaround, that's when my nslookup kubernetes fails....02:44
eanylinsdake: Thats cool :)02:44
rwellumTried 3x with and without02:44
eanylinrwellum: Wasnt sure if you saw my session logs from yesterday night02:46
sdakerwellum which credentials workaround02:46
eanylinshould have worked ok02:46
sdakethe 8080 port change?02:46
rwellumsdake: yes02:46
sdakeright - so kubernetes fails nslooup with --insecure-port=808002:46
rwellumeanylin: that's what sdake has in his review02:46
sdakekubeadm spits out a list of operations (3 of them) to do to get the creds up02:46
eanylinthe 8080 port change is handled by02:46
rwellumDoesn't work fo rme and seems to break nslookup02:46
eanylinmkdir -p ~/.kube02:46
eanylin    sudo cp /etc/kubernetes/admin.conf ~/.kube/config02:46
eanylin    sudo chown $(id -u):$(id -g) ~/.kube/config02:46
eanylinif you do these 3 steps after kubeadm init02:47
eanylinit will work without sed02:47
eanylinI just tested02:47
eanylinrwellum: Ok02:47
sdakerwellum what si your underlying virt provider02:47
rwellumsdake: ambiguous qn or I'm really tired sorry02:48
sdakerwellum libvirt, virtualbox, etc?02:48
rwellumlibvirt running in centos 7.x vm02:49
eanylinactually thats a valid question, I been testing on KVM/libvirt02:49
rwellumsdake:     ./get_helm.sh - that requires sudo priv to run.02:49
sdakerwellum your running libvirt within the vm?02:49
rwellumsdake: yeah - host-passthrough, nested etc02:49
eanylinrwellum: Why do we need libvirt on that VM? It will cause problems later when you try and install the libvirt container02:51
sdakeagreed we are missing a step disable libvirt02:52
sdakethats a big problem02:52
eanylinsdake: Yah, but ideally I think we should build a CentOS from minimal ISO02:52
eanylinit will not have all these stuff02:52
eanylinand its very clean02:52
rwellumI wonder if this is the root of many of my issues bringing up openstack - even once I have a stable k8s02:53
spsuryayes agree with eanylin we should have clean one at the first place02:53
rwellumSo eanylin - i should not do nested virt either?02:53
eanylinrwellum: I have a base CentOS that is build from minimal ISO, so it doesnt have libvirt, bridge-utils etc02:54
eanylinso you install and build on top of it whenever you need new stuff02:54
eanylinI think that will help to eliminate conflicts that can happen02:54
eanylinI once had OVS on my VM02:54
eanylinand it messed up the OVS in kolla-ansible02:54
rwellumRight but usually when you build a vm you have to decide if you do a host-passthrough operation02:54
eanylinrwellum: Ok02:55
eanylinI usually use virt-inst02:55
eanylinor virt-manager02:56
rwellumSame - but in the CPU options - you can select host-passthrough - otherwise if you use you VM to create more VM's they won't work. So just curious if this is what you do or not. I agree about the minimum ISO part.02:57
eanylinAm looking at the option I selected02:58
eanylinI did Hypervisor Default02:58
eanylinand use qemu02:58
eanylinwhen I create Cirros VM02:58
rwellumOk wow - :)02:59
eanylinyeah02:59
rwellumsdake: I'l try tomorrow from a minimum ISO VM - sorry if this slows you down right now.02:59
eanylinrwellum: I have 2 bridges in my cases, virsh net03:00
eanylin1 for mgmt, 1 for neutron03:00
eanylinthat worked pretty well in most cases03:00
eanylinI still have things left to sort out but I think it should at least bring you further03:01
rwellumeanylin: I guess if you get a chance, virsh dumpxml <domain> - and paste it in irc. If networking fails at that level would be good to compare.03:03
eanylinrwellum: Sure03:03
spsuryaeanylin: see this http://paste.openstack.org/show/605456/03:05
eanylinhttp://paste.openstack.org/show/605457/03:05
eanylinrwellum: My host is Ubuntu, VM is CentOS btw03:05
eanylinspsurya: Interesting that you all have problems with nslookup03:06
spsuryaeanylin: yeah03:06
spsuryaeanylin: but here i am not with livbvirt kind of thing ;)03:07
eanylinspsurya: Hmm, you think you followed all the steps so far in my session logs?03:07
eanylinspsurya: Maybe its easier to follow sesison logs than looking at guide03:08
eanylinspsurya: Thats true :)03:08
rwellumty eanylin that'll help a lot I think - but it's 11pm here so will try again tomorrow.03:08
eanylinrwellum: Sure, its late in EST03:08
spsuryaeanylin: yeah, that is very necessary just to avoid repeated query ;)03:10
eanylinspsurya: I forgot that link actually, haha :D03:10
sdakerwellum night dude :)03:11
spsuryaeanylin: np03:11
sdakewe got 3 days left to get this sorted out03:11
sdakeas I am going out of town03:11
sdakerwellum lets sync up in the morning when iwake up and webex?03:12
spsuryasdake: roger03:12
sdakeand i want this merged03:12
sdakethen other peole can incrementally imrpove it03:12
spsuryarwellum: night03:12
*** zhurong has joined #openstack-kolla03:13
spsuryasdake: +103:13
eanylinspsurya: I scrolled up and looked at history, http://paste.openstack.org/show/605332/03:13
spsuryafor merege03:13
eanylinspsurya: Give that a try?03:13
*** MarginHu has joined #openstack-kolla03:14
eanylinsdake: Did I tell you I was able to create a LVM Cinder Volume too?03:14
sdakeeanylin you did but you didnt 'expad as to how03:14
eanylinsdake: Ok, so you need to run a script that is used in the gate to create your vg for cinder-volumes03:15
eanylinkolla-kubernetes/tests/bin/setup_gate_loopback_lvm.sh03:15
rwellumsdake: will catch you in the morning.03:16
eanylinthis is similar to workaround we did in kolla-ansible I think03:16
rwellumeanylin: did you create that vm with virt-inst? And if so - can you share the command line please?03:16
eanylinrwellum: I might have done it from virt-manager a while back, I saved a based image03:17
eanylinso that I can keep re-using it03:17
rwellumeanylin: ok fair enough.03:17
rwellumSame :)03:17
rwellumg'night03:17
eanylinmainly because I was using ISO to install03:18
eanylinI want to see the GUI03:18
eanylinrwellum: Good night03:19
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338403:22
spsuryaeanylin: did you see the latest deploy guide patch03:23
spsurya?03:23
eanylinspsurya: Maybe not fully yet, I think there has been quite some changes throughout the day03:24
spsurya--insecure-port updation part is missing i guess03:24
spsuryaas i understand03:24
spsuryaeanylin: can you please check ?03:26
japestinhosdake rwellum finally I can do cherry-pick with thi command03:26
japestinhogit fetch git://git.openstack.org/openstack/kolla-kubernetes refs/changes/40/439740/21 && git cherry-pick FETCH_HEAD03:26
japestinhosorry I can't comment on review due to bad bot request from ubuntu one login03:26
eanylinspsurya: Actually that one is not needed if we do03:26
eanylin mkdir -p ~/.kube03:27
eanylin    sudo cp /etc/kubernetes/admin.conf ~/.kube/config03:27
eanylin    sudo chown $(id -u):$(id -g) ~/.kube/config03:27
*** bmace has quit IRC03:27
eanylinthe sed workaround is not the ideal one, as I understand03:27
spsuryaeanylin: that is the problem, i changed and that worked for me03:28
spsurya;)03:28
eanylinoh03:28
eanylinnice :)03:28
eanylinyeah, quite some changes today, to the guide03:28
openstackgerritZeyu Zhu proposed openstack/kolla-kubernetes master: Update the version of cirros  https://review.openstack.org/45343203:29
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338403:34
*** gkadam has joined #openstack-kolla03:45
*** masber has joined #openstack-kolla03:56
*** iceyao has quit IRC04:04
*** dave-mccowan has quit IRC04:05
*** zhurong has quit IRC04:12
*** iceyao has joined #openstack-kolla04:24
*** iceyao has quit IRC04:28
*** unicell has joined #openstack-kolla04:32
*** zhurong has joined #openstack-kolla04:35
openstackgerritMerged openstack/kolla-ansible master: Use versionless keystone endpoints  https://review.openstack.org/43902404:51
*** iceyao has joined #openstack-kolla04:51
*** lamt has joined #openstack-kolla05:03
*** jaosorior_away is now known as jaosorior05:11
*** skramaja has joined #openstack-kolla05:14
*** jascott1_ has quit IRC05:16
japestinhoI got this when running helm install05:16
japestinhohttps://www.irccloud.com/pastebin/g8XPLfmh/05:16
*** jascott1_ has joined #openstack-kolla05:17
japestinhosdake what did I miss the step?05:18
*** jascott1_ has quit IRC05:21
*** tonyb_ is now known as tonyb05:25
*** targon has joined #openstack-kolla05:36
*** lamt has quit IRC05:40
*** manheim has joined #openstack-kolla05:40
*** manheim has quit IRC05:45
*** rstarmer has quit IRC05:52
openstackgerritMartin André proposed openstack/kolla master: Add missing cleanup step for new apache images  https://review.openstack.org/45347005:59
openstackgerritMartin André proposed openstack/kolla master: Add missing cleanup step for heat images  https://review.openstack.org/45347105:59
*** yingjun has quit IRC06:06
sdakespiette whic hone06:07
*** pcaruana has joined #openstack-kolla06:08
sdakeeanylin whih sed workaround06:08
sdakethe susdo cp /etc/kubernetesadmin.conf is preferrable06:09
sdakejapestinho you didn't follow the guide - hence i think that is the problem06:09
sdakejapestinho the helm charts are not being acessed at the correct place06:10
openstackgerritMartin André proposed openstack/kolla master: Add missing cleanup step for heat images  https://review.openstack.org/45347106:11
openstackgerritMartin André proposed openstack/kolla master: Add missing cleanup step for new apache images  https://review.openstack.org/45347006:11
sdakebeoodnight folks06:15
sdakei      may be up for out 1 or  hoar06:15
sdakehour06:15
duonghqmandre, are you there?06:16
mandrehi duonghq06:17
duonghqabout the bp: https://blueprints.launchpad.net/kolla/+spec/apache-packages-for-apis06:17
duonghqI don't understand why we only add apache package to the images?06:17
*** caoyuan has joined #openstack-kolla06:18
mandrethat's because we leave it to the operator to chose whether they want to run apache or not for these services06:19
mandrewe don't make it the default06:19
duonghqmandre, this is done in service config files?06:20
mandreduonghq: exactly06:21
duonghqmandre, thanks06:21
mandreif you want to run over apache you'll provide config file for apache and run the apache service06:21
duonghqunderstood06:22
mandrebut we need the binary in the image of course, that's the scope of this bp06:22
duonghqit would be better if we can turn on/off the bundling ...06:23
mandreyou mean provide an image with apache and an image without?06:23
duonghqI mean we can enable/disable the apache in image by build options06:24
*** jrobinson has quit IRC06:25
mandrehmm, I guess we could, but it would have to be a per-image option06:27
mandreI'm not sure it's worth the effoer06:27
*** caowei has joined #openstack-kolla06:27
duonghqyup, guess that it doesn't06:28
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Telegraf: activate influxdb input plugin  https://review.openstack.org/45347806:35
masberhi, I am getting the following error when I deploy Ironic using Kolla http://paste.openstack.org/show/605464/06:43
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Fix ceph role set target_max_bytes for cach pools  https://review.openstack.org/45244406:47
*** athomas has joined #openstack-kolla07:02
*** iniazi has joined #openstack-kolla07:03
*** mgoddard has joined #openstack-kolla07:04
*** Jezekus has joined #openstack-kolla07:07
*** dasTor_ has joined #openstack-kolla07:10
*** dasTor has quit IRC07:11
*** caoyuan has quit IRC07:16
*** matrohon has joined #openstack-kolla07:16
*** shardy has joined #openstack-kolla07:22
*** satyar has joined #openstack-kolla07:23
*** asmaa has joined #openstack-kolla07:29
*** jascott1_ has joined #openstack-kolla07:34
*** Serlex has joined #openstack-kolla07:35
*** nathharp has joined #openstack-kolla07:36
*** britthouser8 has joined #openstack-kolla07:39
*** jascott1_ has quit IRC07:39
*** britthouser has quit IRC07:42
*** gkadam is now known as gkadam-afk07:43
*** jmccarthy has joined #openstack-kolla07:46
*** matrohon has quit IRC07:47
*** egonzalez has joined #openstack-kolla07:48
*** manheim has joined #openstack-kolla07:49
*** yingjun has joined #openstack-kolla07:51
*** bjolo has joined #openstack-kolla07:52
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Fix heat ec2 keystone auth  https://review.openstack.org/44495307:52
*** mgoddard has quit IRC07:58
*** jascott1 has quit IRC08:01
*** iceyao has quit IRC08:12
*** magicboiz has quit IRC08:13
*** magicboiz has joined #openstack-kolla08:13
*** iceyao has joined #openstack-kolla08:13
*** magicboiz has quit IRC08:16
*** magicboiz has joined #openstack-kolla08:16
*** magicboiz has quit IRC08:17
*** magicboiz has joined #openstack-kolla08:17
*** magicboiz has quit IRC08:20
hrwsdake: gigabyte has some with cavium cpu and bmc. up to 512-1024GB ram08:20
*** magicboiz has joined #openstack-kolla08:20
hrwsdake: no idea about price08:20
hrwmorning all08:20
*** jaosorior is now known as jaosorior_lunch08:31
*** mgoddard has joined #openstack-kolla08:32
*** bjolo has quit IRC08:51
*** bjolo has joined #openstack-kolla08:51
bjolomorning08:51
*** matrohon has joined #openstack-kolla08:51
japestinhomorning all08:53
japestinhoif I want add some more controller / compute nodes on existing kolla-ansible deployment which command should I use?08:55
japestinhokolla-ansible upgrade -i multinode or just repeat kolla-ansible deploy -i multinode again?08:55
manheimhello guys08:55
*** matrohon has quit IRC08:57
*** gfidente has joined #openstack-kolla08:59
*** gfidente has quit IRC08:59
*** gfidente has joined #openstack-kolla08:59
*** bjolo has quit IRC09:05
egonzalezjapestinho, use kolla-ansible deploy, upgrades will also do a db sync and replace older containers if a new image is found in the registry09:06
*** sambetts|afk is now known as sambetts09:09
*** blallau has joined #openstack-kolla09:12
japestinhoegonzalez if it same command for remove some nodes too? using this kolla-ansible deploy?09:26
egonzalezjapestinho, deploy only add and configure services09:26
*** iceyao has quit IRC09:28
japestinhoegonzalez did you mean kolla-ansible reconfigure (some --tags) for 'configure services'?09:35
egonzalezjapestinho, deploy and reconfigure in ocata are mostly the same in most services09:35
egonzalezjapestinho, if deploying API services i would not use --tags because some service have dependencies in other(add the node in HAproxy as example)09:36
egonzalezjapestinho, deploy is idempotent and should not restart nothing unless changed09:37
*** MarginHu has quit IRC09:37
*** MarginHu has joined #openstack-kolla09:37
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible master: Add graceful_timeout argument to kolla_docker  https://review.openstack.org/45305509:37
*** MarginHu has quit IRC09:38
*** MarginHu has joined #openstack-kolla09:38
*** MarginHu has quit IRC09:39
*** iceyao has joined #openstack-kolla09:43
japestinhoegonzalez okay thank you I'll try it now. If I want to add 2 more controller nodes with HA mode, what parameters must be enabled in globals.yml beside enable_haproxy, enable_neutron_lbaas, enable_neutron_agent_ha?09:49
egonzalezjapestinho, if the previous deployment already had those settings, just add the new nodes in the inventory and do kolla-ansible deploy09:51
egonzalezjapestinho, if not, enabling those setting may require a huge change in a lot of components, including compute nodes09:52
japestinhoegonzalez, so better I enable those parameters from the start if in future time I'll add some more controller nodes? thanks for the advice09:55
*** duonghq has quit IRC09:57
*** bjolo has joined #openstack-kolla09:58
*** tovin07_ has quit IRC10:04
*** satyar has quit IRC10:13
Deys1anyone has link to the document which states docker tags that is related to the openstack releases, i can't seem to remember where it was in the docs10:14
*** asmaa has quit IRC10:17
*** Jezekus has quit IRC10:18
*** asmaa has joined #openstack-kolla10:19
*** pcaruana|afk| has joined #openstack-kolla10:21
*** xiaoruiguo has quit IRC10:22
*** pcaruana|afk| has quit IRC10:23
*** pcaruana has quit IRC10:24
*** pcaruana has joined #openstack-kolla10:25
*** cuongnv has quit IRC10:25
*** zhurong has quit IRC10:30
*** jaosorior_lunch is now known as jaosorior10:31
*** yingjun has quit IRC10:34
*** pbourke has quit IRC10:39
*** pbourke has joined #openstack-kolla10:41
*** athomas has quit IRC10:41
*** haplo37 has quit IRC10:43
*** shardy is now known as shardy_lunch10:45
*** rmart04 has joined #openstack-kolla10:50
*** athomas has joined #openstack-kolla10:51
egonzalezsomeone have any guess why is python35 tests failing randomly on test_build_image?10:52
*** haplo37 has joined #openstack-kolla10:52
*** dave-mccowan has joined #openstack-kolla11:07
*** caowei has quit IRC11:11
*** iniazi_ has joined #openstack-kolla11:11
*** iniazi has quit IRC11:12
*** dasTor_ has quit IRC11:15
*** dasTor has joined #openstack-kolla11:15
spsuryaegonzalez: looked into but not getting much idea11:18
spsurya:(11:19
*** yingjun has joined #openstack-kolla11:19
openstackgerritMerged openstack/kolla-ansible master: Fix qos rules are not applied on ocata with ovs  https://review.openstack.org/45239211:20
openstackgerritMerged openstack/kolla-ansible master: Fix the "tox -e docs" failed  https://review.openstack.org/45260211:20
openstackgerritMerged openstack/kolla-ansible master: Telegraf: activate influxdb input plugin  https://review.openstack.org/45347811:21
*** cristicalin has joined #openstack-kolla11:22
*** ccesario has joined #openstack-kolla11:23
*** dixiaoli has joined #openstack-kolla11:27
*** dixiaoli has quit IRC11:27
*** dixiaoli has joined #openstack-kolla11:27
*** dixiaoli has quit IRC11:28
openstackgerritMerged openstack/kolla-ansible master: Fix Telegraf retention policy not found  https://review.openstack.org/45305811:32
*** rwallner has joined #openstack-kolla11:35
*** rwallner has quit IRC11:35
*** rwallner has joined #openstack-kolla11:35
*** rwallner has quit IRC11:35
*** rwallner has joined #openstack-kolla11:36
pbourkeegonzalez: I think those tests need some refactoring, there is global variables that are being manipulated and not properly reset11:41
pbourke- after each test11:41
pbourkemasber: you need to provide those files, please check the ironic documentation11:42
*** targon has quit IRC11:43
openstackgerritMerged openstack/kolla-ansible master: Remove min_l3_agents_per_router option  https://review.openstack.org/45115911:52
*** cristicalin has quit IRC11:52
egonzalezsayantani01, around?11:57
manheimfrom Dockerfiles, can I check against vars in global.yml? beased on if a specific service is enabled, I want to install a package in a dockerfile12:02
egonzalezmanheim, nope, kolla and kolla-ansible does not share globals, each one is independent of each other12:03
manheimi'm on newton branch12:03
manheimcan it be done for newton?12:04
egonzalezmanheim, if you are using a local build for your deployment, use package customization https://github.com/openstack/kolla/blob/master/doc/image-building.rst#dockerfile-customisation12:04
manheimthanks12:07
openstackgerritMerged openstack/kolla-ansible master: Remove Ubuntu 14.04 related code  https://review.openstack.org/44879212:26
*** Pavo has joined #openstack-kolla12:26
*** yingjun has quit IRC12:31
*** skramaja has quit IRC12:35
openstackgerritMerged openstack/kolla master: Fix oslo_debug_helper not working  https://review.openstack.org/44439812:38
*** lamt has joined #openstack-kolla12:41
*** matrohon has joined #openstack-kolla12:48
*** gkadam-afk has quit IRC12:49
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Add release note for versionless keystone endpoint  https://review.openstack.org/45362012:49
*** rwallner has quit IRC12:50
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Add release note for versionless keystone endpoint  https://review.openstack.org/45362012:50
*** rwallner has joined #openstack-kolla12:50
-openstackstatus- NOTICE: The Gerrit service on http://review.openstack.org is being restarted to address hung remote replication tasks, and should return to an operable state momentarily12:51
rwellumsdake: when you get in I have the minimum Centos VM running, my username, following the guide and I cannot get past the 8080 connect issue using the current eanylin method.12:53
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Add release note for versionless keystone endpoint  https://review.openstack.org/45362012:53
*** jtriley has quit IRC12:54
*** yingjun has joined #openstack-kolla12:56
*** iceyao has quit IRC13:03
*** goldyfruit has joined #openstack-kolla13:06
*** shardy_lunch is now known as shardy13:08
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338413:09
*** eanylin has quit IRC13:15
*** shashank_t_ has joined #openstack-kolla13:17
*** shashank_t_ has quit IRC13:22
*** mbruzek has joined #openstack-kolla13:22
*** jtriley has joined #openstack-kolla13:24
*** lrensing has joined #openstack-kolla13:25
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338413:25
*** mkoderer has joined #openstack-kolla13:28
*** eanylin has joined #openstack-kolla13:30
*** ipsecguy_ is now known as ipsecguy13:31
*** shashank_t_ has joined #openstack-kolla13:33
*** rmart04 has quit IRC13:33
*** rmart04 has joined #openstack-kolla13:35
eanylinrwellum: Morning, what was the issue you are facing?13:36
rwellumeanylin: this doesn't work for me https://www.irccloud.com/pastebin/sz8e4s2p/13:37
eanylinok13:38
eanylinrwellum: I did this13:39
eanylin mkdir -p ~/.kube13:39
eanylinsudo cp /etc/kubernetes/admin.conf ~/.kube/config13:39
eanylinsudo chown $(id -u):$(id -g) ~/.kube/config13:39
eanylintry this instead13:39
*** papacz has quit IRC13:39
rwellumCan you add this as a comment to the doc review please?13:39
rwellumI'll try next time around.13:39
eanylinrwellum: Ok, I will add it to comment13:40
rwellumty13:40
eanylinnp13:40
*** eaguilar has joined #openstack-kolla13:43
eanylinrwellum: Added as comments13:43
rwellumty eanylin13:44
*** rmart04 has quit IRC13:44
*** lamt has quit IRC13:45
eanylinrwellum: Welcome13:45
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338413:53
*** rmart04 has joined #openstack-kolla13:55
*** rmart04 has quit IRC13:57
*** ksumit has joined #openstack-kolla13:59
*** manheim has quit IRC14:00
*** rmart04 has joined #openstack-kolla14:01
*** iceyao has joined #openstack-kolla14:03
*** rwallner has quit IRC14:05
*** rmart04 has quit IRC14:06
*** jistr is now known as jistr|mtg14:07
*** shardy is now known as shardy_mtg14:07
*** iceyao has quit IRC14:08
*** jascott1 has joined #openstack-kolla14:10
*** bjolo has quit IRC14:14
*** bjolo has joined #openstack-kolla14:14
*** jascott1 has quit IRC14:15
*** lrensing has quit IRC14:18
*** rwallner has joined #openstack-kolla14:23
*** satyar has joined #openstack-kolla14:24
*** manheim has joined #openstack-kolla14:26
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Remove all Ubuntu 14.04 related documentation  https://review.openstack.org/45366814:28
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338414:28
*** ipsecguy_ has joined #openstack-kolla14:29
*** manheim has quit IRC14:30
*** ipsecguy has quit IRC14:31
*** lrensing has joined #openstack-kolla14:32
sdakehrw so morning14:33
sdakemorning14:33
sdakehrw the gigabye model - is that a mainboard or do they make a full box?14:33
sdakein the past gigabyte only made motherboards - but admittdly its been a long time since i've looked14:34
hrwsdake: server. half width14:34
hrwsdake: http://b2b.gigabyte.com/Rack-Server/ARM-SoC - ignore X-Gene ones14:35
*** iceyao has joined #openstack-kolla14:37
sdakehrw not knowing anything about arm cpus14:39
sdakewhich one woudl  i want to do real dev work with14:39
hrwsdake: ThunderX is the only cpu on that list of boards14:40
sdakethis one looks pretty good minus the qsfp ports: http://b2b.gigabyte.com/Rack-Server/R150-T60-rev-110#14:40
sdakehrw just woke up14:40
hrwsdake: you just select 1cpu or 2cpu, density etc14:40
sdakehrw sorry if I ask dubm qs :)14:40
hrwsdake: there are no such thing as 'dumb question'14:40
sdakei dont see a pricetag14:41
sdakehrw questions i could naswer myself14:41
sdakeare dumb questions :)14:41
sdakebut toherwise i agree with your statement :)14:41
hrwsdake: you have to find reseller as they are non consumer boards14:41
sdakehrw ok i'll check around thanks14:41
sdakeyou said 600 bucks in your video14:41
sdakeis that bottom of the barrel14:41
hrwsdake: ThunderX cpu has huge amount of ethernet speed. 1GbE is only port for BMC14:42
sdakewhat pricetag would I be looking at for a dual core with 64gb ram (or enough slots for 64gb of ram)14:42
hrwsdake: 600$ is SoftIron 1000. no BMC14:42
sdakeok - well ened a bmc14:42
sdakeso thats not gonna work :)14:42
hrwsdake: thunderx allow to have 512GB per cpu14:42
sdakecool14:43
sdakei know everything about intel designs14:43
sdakein terms of what to get14:43
sdakeare there "gotchas"14:43
sdake(this I can't find out on my own)14:43
*** goldyfruit has quit IRC14:43
sdakelike 5409 processor doesn't support virt for example14:43
hrwsdake: no idea - never used thunderx systems14:43
hrwsdake: on aarch64 virtualization is mandatory ;d14:43
sdakeis tunderx a next gen processor?14:44
hrwsdake: especially on aarch64 servers14:44
sdakewhat about io virt?14:44
hrwdefine 'next gen'14:44
sdakelatest generation cpu14:44
sdakeyou said you dont use them14:44
sdakecurious why?14:44
sdakemy sepculatino was you just didn't have them yet14:44
hrwsdake: $$$ to get those to home14:44
sdakeright - speculation was correct :)14:44
hrwsdake: there are some thunderx systems at RH and at Linaro - I just do not need to have access to them14:45
sdakehrw but will they work?14:45
hrwyes14:45
sdakeor should I choose a different processor type14:45
sdakemy end game is to be able to pxeboot via ironic arm server14:45
hrwsdake: there is no big choice when it comes to mass market aarch64 servers14:45
sdakerather to an arm server14:46
sdakei want to pxeboot an arm server using ironic14:46
sdakethere - should be more clear :)14:46
hrwsdake: it has standard ipmi14:47
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338414:47
sdakeare all the nics LOM?14:48
sdakeits annoying they dont have rj45 copper lan ports14:49
sdakeqfp is super expensive14:49
sdakei recognize qfp goes 25k meters14:49
sdakeIDC, I need something for 3 meters :)14:49
*** Manheim has joined #openstack-kolla14:53
hrwsdake: LOM?14:54
sdakecool quad channel emory design14:54
sdakelan on motherboard14:54
hrwyes they are14:54
hrwsdake: it is ARM SoC. everything is in SoC.14:54
sdakeinteresting14:55
hrwyou have SoC with cpu, storage controllers, network controllers etc.14:55
sdakehttps://www.servethehome.com/gigabyte-r120-t30-overview-first-cavium-thunderx-system/14:55
sdake48 cores out of the box14:55
hrwoutside chips are mostly PHYs and BMC14:55
sdakecan't finda resller14:55
sdakelooks like a beefy cpu14:55
sdakevery copetitive to intc's xeon line14:56
hrwsdake: https://www.scan.co.uk/products/gigabyte-r120-p30-single-socket-1u-rackmount-armv8-24ghz-pcie-30-8x-ddr3-slots-ecc-unbuffered-2x-10g14:56
sdakenotso sure about the server market in general competitiveness :)14:56
sdake1300 elbows14:56
sdakethtas not terrible14:56
sdakei guess i'll tell my wife since she requested the purchase14:56
sdakeneed to find a us distributor unfortunately14:56
sdakebut she can do that14:56
sdakeif she really wants one14:57
*** eaguilar has quit IRC15:03
sdakerwellum cool so centos starts up with tehe guide?15:03
rwellumsdake: the 8080 work-around is not correct, I asked eanylin to add a review comment and he did. I am adding charts as we speak.15:06
rwellumsdake:  https://www.irccloud.com/pastebin/WKjpI80a/15:06
rwellumSo far much better.15:07
sdakeok - so the kubeadm instructions are wrong15:09
sdakeits not so much a workaround as the design of kubeadm+kubectl15:09
sdakesound slike kubectl is busted15:09
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: gnocchi-base: add some devel packages for non-x86  https://review.openstack.org/45370015:09
hrwuf. CI do not complain on any of my patches15:14
*** iniazi_ has quit IRC15:16
*** iniazi_ has joined #openstack-kolla15:16
inc0good morning15:17
hrwhi inc015:20
*** lamt has joined #openstack-kolla15:21
*** shardy_mtg is now known as shardy15:21
*** Serlex has quit IRC15:22
kfox1111morning.15:30
*** goldyfruit has joined #openstack-kolla15:30
sbezverkkfox1111: morning.. I had interesting observation15:31
*** iceyao has quit IRC15:31
rwellumsdake: speechless! https://www.irccloud.com/pastebin/DfqGIT39/15:31
kfox1111sbezverk: yeah?15:32
sbezverkkfox1111: when we start canal setup, none of control plane pods were actually in running state, but we still pushed canal setup. kube eventually sorts this out, but lots of racing happens15:32
kfox1111rwellum: nice. :)15:32
rwellumkfox1111: it's a first for me :)15:33
kfox1111sbezverk: hmm.. so long as kube-apiserver is running, it shouldn't matter how long it takes to get the scheduler/etc up.15:33
sbezverkkfox1111: I put together ps to prevent this condition.15:33
kfox1111sbezverk: but, yeah. if we wanted more consitstent behavior, making it easier to debug, we shoudl put in more waits.15:33
*** chas has joined #openstack-kolla15:33
kfox1111rwellum: congrats! :)15:33
sbezverkkfox1111: it has not15:33
sbezverkkfox1111: http://logs.openstack.org/84/453384/9/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/223eb4e/console.html#_2017-04-05_15_00_00_54196215:34
sbezverkyou will see how control plane were coming up15:34
kfox1111sbezverk: really? hmm...15:34
sbezverkrwellum: COngrats,15:34
kfox1111I thought we already had a wait for that in setup_kubernetes.sh15:34
rwellumCheers sbezverk !15:34
sbezverkhopefully many more in future ;)15:34
sbezverkkfox1111: https://review.openstack.org/#/c/453384/915:35
kfox1111sbezverk: yeah. setup_kubernetes.sh ~line 113.15:35
*** chas has quit IRC15:35
*** chas has joined #openstack-kolla15:36
sbezverkkfox1111: well I have proof that it was not working at that time15:36
sbezverkand it could not15:36
sbezverkbecuase it would always be failing as kube-dns were in pending until networking is done15:37
*** duonghq has joined #openstack-kolla15:37
duonghqevening guys15:37
kfox1111sbezverk: I think thats something we need to dig deeper into. if thats failing, other things could be too.15:38
sdakeeqlooka pewrry eoxkin susw :)15:38
sbezverkkfox1111: Now before pushing canal, I make sure all essential pods are running before doing canal15:38
sdakerwellum looks pretty rockin dude :)15:38
kfox1111all its doing is trying to get a successfull kube get pods, so in order for that to work, apiserver must be up.15:38
kfox1111that means etcd/apiserver must be responsive. and the rest can come up too.15:38
sbezverkkfox1111: I do not know but I clearly see apiserver in pending in the log I sent you15:39
sdakerwellum are ll necessary steps required in the deployment guide15:39
sdakeI haven't looked at the review this mroning - had a late start (7am)15:39
sbezverkkfox1111: and it is up time only 7 seconds15:39
kfox1111in fact, it must be true for the kube update that goes in a few lines after to go in....15:39
kfox11112017-04-05 15:00:00.398944 | clusterrolebinding "cluster-admin" replaced15:40
kfox1111so apiserver/etc had to be up...15:40
kfox1111then it failed?15:40
*** chas has quit IRC15:40
sbezverkkfox1111: 2017-04-05 15:01:14.143039 | Pod name: kube-apiserver-centos-7-2-node-osic-cloud1-s3500-8284205 status: Pending15:40
sbezverkit is hard to argue ;)15:40
*** jistr|mtg is now known as jistr15:40
sdakerwellum were you able to start a vm from that kubernetes + kolla deployment?15:41
*** gardlt has joined #openstack-kolla15:41
*** yingjun has quit IRC15:41
kfox1111sbezverk: it happened right after the node was untanted.15:41
kfox1111maybe that is related?15:41
*** sayantan_ has joined #openstack-kolla15:41
kfox1111it looks like it maybe decided to move the etcd/apiserver to the slave node at that point?15:41
sbezverkkfox1111: I would be very surprised to see untained killing apiserver ;)15:41
sdakekfox1111 in the review -- you said to make recommendations how ot pick a service network15:41
sdakekfox1111 I have no idea how to make such recommendations15:41
sdakeI blindly picked an rfc1918 address15:42
sbezverkkfox1111: I can see the same happens in a signle node as well15:42
sdakekfox1111 which is probably not optimal - can you help me through how to document such a recommendation?15:42
kfox1111sbezverk: look at the log though. it was stable as of setup_kubernetes, and then failed afterwards.15:42
kfox1111but untaint talked to the api server, and then yoru script says it broke.15:42
sbezverkkfox1111: http://logs.openstack.org/84/453384/9/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-iscsi-nv/e507816/console.html#_2017-04-05_14_56_57_91731615:43
kfox1111sdake: yes. the user picing a non overlapping range from rfc1918 from their organizaion is important.15:43
kfox1111sdake: basically, if your organization is a personal one, pick whatever you want. if its a buisness, ask your network admin.15:43
sdakethanks - so "non-overlapping" is the key15:43
kfox1111from the rfc1918 space I mean.15:43
kfox1111right.15:43
sdakeand  10.3.0.0/16 would have overlaped15:44
sdake(in your environment)15:44
sdakepod cidr range doesn't matter then?15:44
kfox1111hmm... let me try and rephrase that a bit....15:44
kfox1111there are two requirements.15:44
kfox1111the service and pod address spaces cant overlap with each other.15:44
kfox1111and, the address spaces cant be already allocated by your organization.15:44
rwellumsdake: pretty much followed the guide, with a couple of exceptions I am studying now and will feed back if necessary on the review15:44
sbezverkkfox1111: http://logs.openstack.org/84/453384/9/check/gate-kolla-kubernetes-deploy-centos-binary-2-iscsi-nv/ec5377c/console.html#_2017-04-05_15_00_08_72248715:45
kfox1111at my site, the addresses picked are already allocated by my org.15:45
sdakekfox1111 right15:45
sdakemakes sense15:45
kfox1111the same is true of the openstack-infra org.15:45
sdakedidn't knwo the openstack part15:45
kfox1111those adress spaces overlap with the org.15:45
sdakemust they always be rfc1918 addresses?15:45
kfox1111not strictly, no.15:46
kfox1111any address space will do.15:46
kfox1111BUT,15:46
kfox1111pod network is very very wasteful of ip's.15:46
sdakeright - so pod network should be rfc191815:46
kfox1111the service network could, but isn't routable anyway, so not relaly much benifit there either.15:46
sdakebecause it consumes a /24 per host15:46
*** ksumit has quit IRC15:46
openstackgerritMerged openstack/kolla-ansible master: Change heat cfn service description  https://review.openstack.org/44645315:46
kfox1111sdake: right.15:47
sdakeok - the service network - is thta the network used to connect into the atual services?15:47
kfox1111whether its allocated the ips or not.15:47
sdakekfox1111 thanks btw, this is very helpful15:47
kfox1111kind: service <- those get service ip's.15:47
sbezverkkfox1111: all these showed that transition happened from pending to running after setup_kubernetes script has completed..15:47
kfox1111service ip's come from the service ip space.15:48
kfox1111to complicate matters, in the service spec, they call them "clusterIP"s too.15:48
kfox1111sbezverk: I dont disagree. but I'm saying, there's something wrong at a deeper level we're not understanding I think.15:48
sdakewhat ip address range is used to communicate with the actual pods from the outside?15:48
kfox1111sbezverk: the control plane shoudl have been stable after setup_kubernetes.sh finished.15:49
kfox1111if its not, we have bigger issues.15:49
kfox1111sdake: there's kind of an east/west north/south response involved there.15:49
kfox1111different paths use different ips'.15:50
mnasermeeting in 10 fyi15:50
kfox1111well... hold on. let me parse your sentence again...15:50
kfox1111sdake: maybe an example. are you asking, say,15:50
sdakekfox1111 let me clarify - when I connect to keystone externally - how do i do that15:50
sdakewhat ip is used15:50
kfox1111how does a keystoen user-list from outside k8s get to the pod?15:50
sbezverkkfox1111: it is not ready after that, with the script I added it becomes stable and canal gets deployed15:50
sdakekfox1111 yup15:51
kfox1111sbezverk: I'm saying it is stable, but then shortly after becomes unstable. I wana know why. I'm guessing its the taint, as thats the last time in your log it is stable?15:51
kfox1111sdake: ah.15:51
inc0meeting in T-9min15:51
kfox1111sdake: so, thats a bit up to the admin of k8s. there are multiple ways of setting that up. I can give you some reference ideas though.15:52
sbezverkkfox1111: sorry I do not get the last part, I mean your question..15:52
sdakekfox1111 please15:52
kfox1111sdake: like, how we are doing it in the gate, or how I am doing it at one of my sites.15:52
sdakekfox1111 your site15:52
sdakekfox1111 or generalize it :)15:52
sdakekfox1111 i know i may be running after i just learned to crawl15:52
sdakehowever, Id like to have multiple nodes and hae that work15:53
sdakekfox1111 this doesn't necessarily have to go in the docs - more for my own seeding of my brain - for later15:53
kfox1111sbezverk: I think the system is stable, then going unstable, and then your checking for the instability. better would be to understand why its going unstable and fix it. then we don't need a double check.15:53
sdake(sometiem in the next few months this will click)15:53
kfox1111sdake: the easiest way to do multinode is what we're doing in the gate. has no ha though.15:54
kfox1111like I'm doing at my site.15:54
kfox1111so maybe we cover that first?15:54
sdakekfox1111 sure15:54
sdakekfox1111 we have a meeting - hoepfully we can continue that after our team meeting15:55
kfox1111k. so, there are multiple types of kubernetes services.15:55
kfox1111k.15:55
sbezverkkfox1111: hmm fixing kubernetes is not really our goal, as long as we have a way to run it with stability should sufice, no?15:56
*** vhosakot has joined #openstack-kolla15:57
*** bjolo has quit IRC15:57
*** bjolo_ has joined #openstack-kolla15:57
kfox1111sbezverk: I'm not saying its our bug to fix. or that its even a bug. but we should understand what is going on.15:57
kfox1111if its the taint destabilizing things, maybe we're untainting wrong. or the untaint belongs in the setup_kubernetes.sh before the kube-api stability check.15:57
kfox1111sbezverk: https://review.openstack.org/#/c/453277/ seems to have fixed the v3 gate.15:58
*** skramaja has joined #openstack-kolla15:59
sbezverkkfox1111: perfect15:59
*** skramaja has quit IRC15:59
sdakeinc0 i updated the wiki for the meeting just now15:59
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add external rabbitmq  https://review.openstack.org/45372415:59
sdakeinc0 in case you had it open - plz reload :)15:59
sbezverkkfox1111: I think the confusion comes from the fact that kubectl does not immediately fail on commands but retries which creates impression that it is working, when in fact it is still trying..16:00
kfox1111sbezverk: but get pods worked. and untaint worked. it wasn't just waiting. it successfully talked to kube-apiserver and etcd.16:00
kfox1111and kubectl update ClusterRoleBinding16:01
*** jascott1 has joined #openstack-kolla16:02
*** jascott1- has joined #openstack-kolla16:02
*** shardy is now known as shardy_afk16:02
kfox1111jascott1: congrats! :)16:02
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338416:02
*** eaguilar has joined #openstack-kolla16:02
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new rabbitmq instance for Murano  https://review.openstack.org/45372416:04
*** skramaja has joined #openstack-kolla16:04
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new rabbitmq instance for Murano  https://review.openstack.org/45372416:06
mnaserpbourke nice patch, this can probably be re-used for trove as well16:10
openstackgerritMerged openstack/kolla-kubernetes master: Move memcached config to kolla-k8s  https://review.openstack.org/45062016:10
pbourkemnaser: thanks, its working for me just needs one or two tweaks. original credit goes to bdaca who I believe has moved on from the project16:10
pbourkeegonzalez: I completely missed your original ML thread!16:11
*** rwallner has quit IRC16:11
mnaserpbourke gotcha, be nice to have trove support in there but if thats not something that you have the time to work on, i can pick up from the rest of your work16:11
*** chas has joined #openstack-kolla16:11
pbourkemnaser: sure, lets get it polished and merged for murano first, then you can copy/paste for trove16:11
mnaserpbourke cool :)16:12
egonzalezpbourke, yep trove is also affected, don't know if someother projects are using that kind of architecture16:12
pbourkeI think bdaca's original idea was to have one external rabbit for all these kind of projects16:12
pbourkemaybe that would be better than a murano_rabbit, trove_rabbit, etc16:12
openstackgerritMerged openstack/kolla-kubernetes master: Move cinder config to kolla-k8s  https://review.openstack.org/45055616:13
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338416:13
*** sayantan_ has quit IRC16:13
*** sayantan_ has joined #openstack-kolla16:14
pbourkeegonzalez: "Hell no, this is not an option" - ha16:14
mnaserpbourke egonzalez yeah when i saw that option listed in the relm of possibilities16:15
mnasermy heart sank till i saw the cons16:15
mnaserhaha16:15
pbourkelol16:15
*** chas has quit IRC16:15
*** gardlt has quit IRC16:16
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Make rabbitmq log dir configurable  https://review.openstack.org/45373516:18
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new rabbitmq instance for Murano  https://review.openstack.org/45372416:20
*** blallau has quit IRC16:20
*** chas has joined #openstack-kolla16:21
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new rabbitmq instance for Murano  https://review.openstack.org/45373816:22
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Make rabbitmq log dir configurable  https://review.openstack.org/45373516:22
sbezverkkfox1111: see this output, it proves that no containers are running when we exist from setup_kubernetes, specifically line 1316:24
sbezverkhttp://paste.openstack.org/show/605534/16:24
kfox1111sbezverk: thats not how I interprite that output.16:25
kfox1111it says: k8s was up enough to respond to get pods,16:25
kfox1111to update the clusterrolebinding, and untaint a node.16:26
sbezverkkfox1111: why pods then are not shown?16:26
kfox1111where?16:26
*** chas has quit IRC16:26
kfox1111oh. by get pods?16:26
sbezverkplease check latest PS16:26
kfox1111in setup_kubernetes.sh?16:26
sbezverkyes16:26
sbezverkit supposed to oputput something16:27
kfox1111kubectl get pods -n kube-system > /dev/null16:27
kfox1111no its not. :)16:27
sbezverkkfox1111: man, please check my latest PS16:27
kfox1111ok. sec.16:27
sbezverkthese logs are from there16:27
kfox1111ok. so you added a second get pods...16:28
kfox1111looking at the logs....16:28
kfox1111oh. I may know....16:29
kfox1111etcd/kube-apiserver/scheduler/controller-manager are static pods.16:29
pbourkemnaser: egonzalez: I think im going to refactor this to one "external rabbitmq" murano/trove/whatever can use. shout it you think we need one rabbit per service16:29
kfox1111they get dummy entries added to the apiserver by kubelet.16:29
kfox1111kubelet may not have added them yet.16:29
kfox1111but that doesn't mean they aren't there/working.16:29
*** matrohon has quit IRC16:30
sbezverkkfox1111: I do not see why you insist on using check within setup_kubernetes16:30
kfox1111the check in setup_kubernetes checks to see if k8s is working.16:30
mnaserpbourke i think that's the easiest way to start things off i guess16:30
kfox1111not if its reporting the shadow pods.16:30
mnaserrabbitmq per service feels excessive16:30
sbezverkdoing it in the script is more relaible as ALL important containers are checked before the networking gets pushed16:30
pbourkemnaser: it would be easier not to refactor ;)16:30
sbezverkkfox1111: which shadow pods ?!?!?16:31
mnaseri dont know, there's pros and cons to each one pbourke :(16:31
kfox1111sbezverk: not aposed to doing your check. but was trying to understand why it was failing when cli was working.16:31
pbourke:/16:31
kfox1111the answer is shadow pods.16:31
*** shardy_afk is now known as shardy16:31
kfox1111the shadow pods for etc/kube-apiserver/scheduler/controller-manager.16:31
egonzalezpbourke, I think we can start with one rabbit instance shared between end-user services and then add support to use one instance per service16:32
pbourkeegonzalez: that support is in my current patch16:33
sbezverkkfox1111: here is my logic..  when we install cluster we never see race conditions as we see at the gate16:33
pbourkeegonzalez: im just not sure its a good idea to have lots and lots of rabbit clusters16:33
pbourkeseems like overkill16:33
sbezverkto prevent it, we need to control the process better..16:33
kfox1111sbezverk: hand rolled is slower then the gate. the races tend to not be issues then.16:33
kfox1111fullly orchestrated is usually shows more races, unless you throw in a tun of sleeps to slow it down to what a human normally does.16:33
kfox1111the other issue is the gate is made up of many differing machine types.16:34
kfox1111sometimes issues there creap in.16:34
sbezverkkfox1111: exactly, so makeing sure that cluster is completely up without shadow pods or other crap, will gives us better base for next steps..16:34
kfox1111sbezverk: no, I'm saying, your check is checking for shadow pods, not if the cluster is up.16:35
kfox1111its not checking for what you think its checking for.16:35
mnaserpbourke i can imagine multiple instances causing complications indeed16:35
mnaserin kolla meeting, i can discuss further in a ibt :X16:35
pbourkearg I totally forgot about the meeting16:35
sbezverkkfox1111: well, one thing for sure, once I added this check multinode stopped failing on networking installation..16:35
sbezverkit seems more reliable now16:36
kfox1111sbezverk: its very likely that it takes a minute or two for the shadow pods to show.16:36
kfox1111which means its similar to adding a sleep 60 at that point.16:36
kfox1111which gives things more time to settle.16:36
kfox1111hiding a race condition.16:36
kfox1111I totally buy that would make things seem more stable most of the time.16:37
sbezverkkfox1111: right it is a delay, but it is controlled delay16:37
sbezverkalso even when you install the cluster you do use kubectl get pods to see if pods are there16:37
sbezverkand if you do not see them, you think something is wrong16:37
kfox1111yeah. but the problem is, thats not quite true in static pod case. threw me until  Itried to understand the issue.16:38
kfox1111I'm ok merging in your ps now that I understand the issue. though I think it probably should merge into setup_kubernetes.sh.16:38
*** iceyao has joined #openstack-kolla16:39
sbezverkkfox1111: it is already look very complex, this way looks much cleaner and better to understand for people looking at the code16:39
sbezverkkfox1111: please find 2nd core to ack v3 ps16:40
kfox1111the code should be broken up into tasks I think. setup_kubernetes is about ensureing a working k8s comes out the end of the script.16:40
sbezverkkfox1111: we might get all green gate soon16:40
kfox1111k. will do.16:41
*** ksumit has joined #openstack-kolla16:43
*** sayantan_ has quit IRC16:43
*** sayantan_ has joined #openstack-kolla16:43
*** iceyao has quit IRC16:43
sbezverkkfox1111: when I think about shadow pods you mentioned, at one point normal pods takes over of shadow pods, so I suspect if something happening at the same time, it could potentially get impacted, resulting instability we observe.16:45
*** hieulq_ has joined #openstack-kolla16:49
kfox1111there shouldn't be any deployments/daemonsets for those services.16:53
kfox1111I think they should always be shadows.16:53
kfox1111if it was bootkube, I'd say that oculd be a problem.16:53
sbezverkkfox1111: but we can clearly see kube-apiserver going from pending to running state16:55
kfox1111hmm... true.16:55
kfox1111but I'm not aware of any machinery that would ever cause that.16:55
sbezverkit means in pending state your shadow container serves api requests16:55
kfox1111well... let me check the log again.16:56
kfox1111is it actually pending?16:56
*** rwallner has joined #openstack-kolla16:56
sbezverkkfox1111: http://logs.openstack.org/84/453384/11/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/610f58b/console.html#_2017-04-05_16_20_55_81702716:57
kfox1111sdake: https://review.openstack.org/#/c/453277/ please16:57
sbezverkit is pending and then 5 seconds later becomes running16:57
kfox1111sbezverk: weird.16:57
*** rwallner has quit IRC16:57
kfox1111I wouldn't htink a shadow pod would enter pending state.16:57
*** rwallner has joined #openstack-kolla16:57
kfox1111unless the controller manager does sanity checks....16:58
kfox1111maybe thats it...16:58
kfox1111it could be the readyness check in the static pod blocking kube-proxy for the kubernetes.default.svc...16:58
*** nathharp has quit IRC16:58
kfox1111may be a change of behavior in 1.6...16:58
inc0kfox1111: have you seen mgoddard project?16:59
inc0since you both work on scientific openstacks16:59
kfox1111inc0: not sure. got a link?16:59
inc0probably something interesting:)16:59
rwellumsdake: from the DG - is this expected to work? "openstack server add floating ip demo1 $(openstack floating ip create public1)"16:59
inc0kfox1111: hold on17:00
kfox1111inc0: https://review.openstack.org/#/c/453277/ please17:00
sdakerwellum i tried it17:00
sdakeand it does not work17:00
sdakerwellum someone suggested that in an earlier review17:00
sdakeand i either c&p fubared it17:00
kfox1111rwellum: yeah, I think you need to parse out the id.17:00
inc0https://github.com/stackhpc/kayobe17:00
sdakeor the person that added it was wrong :)17:00
sdakerwellum after init-runonce runs - it shows you how to create a vm17:01
kfox1111inc0: ah. very interesting. :)17:01
rwellumyeah kfox1111 otherwise it's a big old table...17:01
inc0ok bbiaf, going to office17:01
*** jascott1 has quit IRC17:01
*** jascott1- is now known as jascott117:01
kfox1111inc0: thats what I want to do, but with kolla-kubernetes. :)17:01
inc0we should combine efforts then17:01
kfox1111inc0: +117:01
sdakeeanylin can you confirm you did get 4.0.0 images running and booting a vm with source17:01
sdakewith centos-source imgaes?17:01
rwellumBTW I have to say, I've attended a few kolla meetings and this by far is my favorite statement anyone has every written: "have to go - starts raining and I am at the park now".17:02
*** rstarmer has joined #openstack-kolla17:02
inc0how about we'll comandeer one of sciengific wg meetings to talk about "how to make good scientific openstack deployment tool"?17:02
*** mgoddard has quit IRC17:02
kfox1111sbezverk: so... if that theory is true, then we actually need to ensure the shadow pods enter ready state and your patch set is not just a sleep, but critical.17:03
*** shardy has quit IRC17:03
inc0anyway, bbiaf17:03
*** egonzalez has quit IRC17:03
kfox1111inc0: sure17:03
kfox1111sbezverk: my long term goal of setup_kubernetes is to have something that the docs can say "if you don't want to deal with k8s deployment, just call setup_kubernetes.sh"17:04
*** skramaja has quit IRC17:04
*** saneax has quit IRC17:04
sbezverkkfox1111: kubernetes is smart enough to sort things out but not always. When I saw in "messages" lots of retries for different things at the gate and not in my test bed, it made me thinking..17:04
kfox1111so having the check in setup_kubernetes makes it easier to use for that use case.17:05
sbezverkkfox1111: we use wait_for_something already in the script17:05
kfox1111sbezverk: yeah.17:05
sbezverkso it does not break already existing concept17:05
kfox1111sbezverk: maybe just move the script callout from setup_get to setup_kubernetes in the master section?17:05
kfox1111setup_gate I mean17:06
*** duonghq has quit IRC17:06
mnaserpbourke as a middle ground could we do external rabbitmq with vhost per service?17:06
sbezverkI can replace already existing check in setup_kuebernets17:06
sbezverkmaybe it will be less confusing17:07
kfox1111k.17:07
kfox1111works for me.17:07
openstackgerritMerged openstack/kolla-kubernetes master: Disable placement for v3 gate.  https://review.openstack.org/45327717:08
*** jaosorior is now known as jaosorior_away17:08
*** athomas has quit IRC17:10
rwellumkfox: with the ip address I got closer https://www.irccloud.com/pastebin/XimbyWzA/17:10
*** saneax has joined #openstack-kolla17:11
kfox1111neutron net-list?17:11
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338417:11
*** unicell has quit IRC17:12
openstackgerritQin Wang (qwang) proposed openstack/kolla-kubernetes master: [WIP] Condition and tags in requirements.yaml for keystone service  https://review.openstack.org/44446117:12
rwellumkfox1111 https://www.irccloud.com/pastebin/jpEQHFbD/17:12
kfox1111rwellum: neutron net-show demo-net17:13
kfox1111and one for public117:13
rwellumkfox1111 https://www.irccloud.com/pastebin/lFfdNlQb/17:14
kfox1111weird. k....17:15
kfox1111anything in the neutron-server logs?17:15
sdakerwellum rendering new docs now - and running through quickstart17:17
*** lrensing has quit IRC17:18
rwellumkfox1111 https://www.irccloud.com/pastebin/ZURYpFAx/17:19
*** ccesario has quit IRC17:19
kfox1111rwellum: kubectl exec -it neutron-server-2216068945-1gjk1 -n kolla -c main vi /var/log/kolla/neutron/*17:20
kfox1111and see if there is anything interesting there?17:20
rwellumsdake: I think theres a couple of things you need to know before you run through another deployment - that helped me anyway. Not in your guide.17:20
sdakerwellum its our guide - i am just an editor :)17:20
sdakeplease do tell and I'll rev hte guide  first with notes and then render and deploy17:21
sdakerwellum can you leave notes?17:21
*** athomas has joined #openstack-kolla17:22
rwellumsdake - I have added several today - just give me a minute to add a couple more17:23
rwellumJust don't want you to waste your time.17:23
rwellumkfox1111: that's a huge log - anything I should look for?17:23
kfox1111anything that looks like a stacktrace?17:24
sbezverkkfox1111: moved to setup_kubernetes, please check if you are ok.. from what I see it is etcd takes the longest time to come up..17:24
rwellumkfox1111: not yet but 7k lines of logs17:26
rwellumsdake: I left some notes yesterday in reviews and I think you missed them.17:26
inc0back17:27
*** hieulq__ has joined #openstack-kolla17:30
kfox1111sbezverk: there's an issue with the change.17:31
kfox1111you moved setup_canal up to before the second node gets added.17:31
kfox1111but then the tests to ensure canal is stable on the second node get dropped, as they are in setup_canal...17:31
*** gfidente is now known as gfidente|afk17:32
kfox1111I think test_kube_dns needs to be removed from setup_canal then, and move to where setup_canal was before.17:32
kfox1111and right in front, do:17:33
kfox1111$DIR/tools/pull_containers.sh kube-system17:33
kfox1111$DIR/tools/wait_for_pods.sh kube-system17:33
*** sambetts is now known as sambetts|afk17:33
sdakerwellum ok - editing is hard with the hundreds of comments on the doc so far17:34
sdakerwellum if you could update with the missed notes the latest review - i'll make suer to incorporate them as notes for now17:34
sdakeand then render them17:34
sbezverkkfox1111: that is what I want17:34
*** hieulq_ has quit IRC17:34
sdakeandwhen i get a workflow that wfm, I'll clean it up17:34
sdakeand then hopefully we can merge it17:34
sbezverkI tested several times and setting up everything on a master including networking and then adding second node look way more stable17:35
kfox1111sbezverk: yeah. but with those changes, it should be more stable yet.17:35
kfox1111cause now, its not testing that dns on the second node ever works.17:35
kfox1111which we have seen break in the past.17:35
sbezverkkfox1111: I do not want to take any chances.. when you build kube cluster you cannot join all your nodes and only then setup networking17:36
kfox1111sbezverk: I'm ok with yoru canges, provided that the testing that was done before remains.17:36
kfox1111thats the onlyh issue I have.17:37
kfox1111the change as proposed just drops some validity testing that was happening before.17:37
rwellumsdake: :) doing that now.17:39
*** unicell has joined #openstack-kolla17:41
*** rstarmer has quit IRC17:42
*** ccesario has joined #openstack-kolla17:42
*** bjolo_ has quit IRC17:46
*** tonanhngo has joined #openstack-kolla17:47
*** tonanhngo_ has joined #openstack-kolla17:50
*** tonanhngo has quit IRC17:51
*** tonanhngo_ has quit IRC17:54
*** vhosakot_ has joined #openstack-kolla17:55
kfox1111wow... hadn't seen this issue in a long time: http://logs.openstack.org/84/453384/12/check/gate-kolla-kubernetes-deploy-centos-binary-2-iscsi-nv/4bb7cfd/console.html17:55
sdakerwellum thanks dude :)17:55
*** tonanhngo has joined #openstack-kolla17:55
kfox1111guess docker still has the bug.17:55
*** vhosakot has quit IRC17:56
*** hieulq__ has quit IRC17:56
rwellumsdake: done. I think I have some important ones here.17:58
*** Pavo has quit IRC18:01
*** rwallner has quit IRC18:03
*** rwallner has joined #openstack-kolla18:03
*** rwallner has quit IRC18:04
*** rwallner has joined #openstack-kolla18:04
*** krtaylor has quit IRC18:07
*** manheim_ has joined #openstack-kolla18:07
rwellumsdake: I think I made 1 mistake with current deployment. I used wrong external_vip address. I've added a comment in the review about how that's misleading in the guide.18:07
*** manheim_ has quit IRC18:11
sdakerwallner ok - so guide ready to go for a rev and then test - or need more commentary?18:14
*** jascott1_ has joined #openstack-kolla18:15
*** lucasxu has joined #openstack-kolla18:18
sdakesbezverk is the gate functional now after your work?18:19
kfox1111sdake: thats the last ps I know of to fix issues.18:21
kfox1111doesn't mean there isn't more, but should be stable I think.18:21
sdakecool18:21
sdakei'll recheck cinder18:21
rwellumsdake: yes ready18:22
sdakeif that works - i'll resubit the entire patch stream to trigger the gate again18:22
sdakerwellum ok - processing the new rev now18:22
rwellumsdake: rwellum != rwallner18:22
sdakerwellum re this comment: I think we need better defaults in the suggested cloud.yaml file. The external_vip should be a 192 address agreed. The others should be 10.x18:23
sdakerwellum i know, its a tab cmopletion thing that screws up - sorry :)18:23
sdakerwellum 10.0.0.1?18:23
sdakeor some other value18:23
sdakei dont know what to make these values18:23
*** lrensing has joined #openstack-kolla18:23
rwellumsdake: agreed tricky because unique to each environment. Thing is you want it to be a local address, and not the same as the other 192.x addresses - that's gotten me a few times especially as 2 lines down we reference the local interface name.18:24
rwellumHow about 10.X.X.X ?18:25
rwellumReally make it clear it's unique18:25
sdakerwellum not being difficutl here - I just want to understand what you understand :)18:25
sdakerwellum sbezverk told me to set dns_name to my hosts ip address18:26
*** ntpttr_ has quit IRC18:26
rwellumThat's different to the external_vip - not sure your point?18:27
sdakerwellum i have no point really - i want to explore what ou understand about these values18:28
sdakebecause  Idon't know what the IPs shoudl be set to in cloud.yaml18:28
sdakeif I could map them to my environment correctly, then I could explain how to map them to others18:28
*** rstarmer has joined #openstack-kolla18:29
sdake(i.e. make the instructions generic)18:29
kfox1111networking's always the hard part.18:29
sdakeit is possible this is why my vms don't come up18:29
sdakekfox1111 as my wife says - my company didn't build its empire because networking is easy :)18:30
sdakerather my employer :)18:30
kfox1111yup.18:30
sdakerwellum you likely ahve deep domain expertise in networking - i dont18:30
rwellumI'll share my cloud.yaml18:30
sdakerwellum cool18:30
sdakei have written a protocol18:30
rwellumhttps://www.irccloud.com/pastebin/jHbeikHE/18:30
rwellumI am an original author of EIGRP :)18:31
sdakehowever, that protocol was for lans :)18:31
sdakecool - not sure what that is - the one I wrote was Totem18:31
sdakerwellum quiz time18:31
sdakewhat is this ip address in your network:18:31
sdake           external_vip: "192.168.122.251"18:31
sdakejust some random ip ?18:32
sdake(that is unused?)18:32
*** rwallner has quit IRC18:32
rwellumsdake - no it's the ip addr of my external interface on eth1 https://www.irccloud.com/pastebin/hJ3fHeTx/18:32
sdakeeth1 is your neutron network or your management network?18:33
*** harlowja has quit IRC18:33
rwellumNeutron - although I don't like that name.18:33
rwellumexternal is better18:33
sdakethanks18:33
rwellumimo18:33
sdakeso that is p1p218:33
sdakein my environment18:33
rwellumyes as I recall18:33
rwellumNotice the tunnel_interface right underneath is not going to be p1p218:34
kfox1111one thing we do need that we dont have yet is some reference architecture documents.18:34
kfox1111this kind of thing would be easier with that.18:35
sdakeand this address is what:              dns_name: "10.240.43.213"18:35
sdakekfox1111 right - i'd like to just get some basic "how do you get it breathing" instructions first18:36
sdakekfox1111 rather then make sea salt with a "what do all the config optoins mean" :)18:36
*** chas has joined #openstack-kolla18:36
*** sayanta__ has joined #openstack-kolla18:36
kfox1111sdake: yeah. just saying, sometimes a picutre is worth a thousand words. maybe a ref doc with a picutre with nics/ips on it would really help.18:36
rwellumI set that to my management ip address sdake18:36
sdakeok my management ip is 192.168.7.10118:37
sdakeso that should work with the sed operation18:37
sdakerwellum and this:                 - 10.240.43.213: cinder-volumes18:37
sdakesame story?18:37
rwellumwhat is your external address?18:37
rwellumYes.18:37
sdakeexternal address ?  p1p2 doesn't have an Ip address associated with it18:37
sdakeneutron does not work if an ip address is associated with it18:38
sdakekfox1111 picture would be helpful - maybe after I understand what to draw I can draw one :)18:39
rwellumThis is why your cloud.yaml is confusing to me then. That's why you have external and internal ip addresses the same - because you only have one ip address.18:39
rwellumSo what are you going to set external_vip to?18:39
sdakei dont know what external_vip does18:39
rwellumeanylin: and I both have two bridges and two addresses.18:39
sdakebridges - how did you get bridges ? create them?18:40
*** sayantan_ has quit IRC18:40
*** ntpttr_ has joined #openstack-kolla18:40
*** chas has quit IRC18:40
*** rwallner has joined #openstack-kolla18:41
*** rwallner has quit IRC18:41
*** rwallner has joined #openstack-kolla18:41
rwellumWe're both running on VM's. Actually eanylin shared his XML last night and he has two virsh virtual bridges which is very common with VM's right? In my case I have one NIC br0, and one I NATed - which is just using virsh virtual bridges.18:42
sdakethis may explain why eanlyn can't access his neutron interface - because he has an ip address assigne to it18:42
rwellumYeah but if that's the case, we're still stuck on what the heck external_vip is.18:43
rwellumThe cloud.yaml clearly wants two different ip addresses.18:43
sdakekfox1111 can yo udefine external_vip for us plz :)18:43
sdakehere is what we have in the review:18:44
sdake  set 'ext_interface_name': name of the interface or bridge which will be used by neutron's provider interface.  To determine the value of this parameter,18:44
sdake   run ``grep neutron_network /etc/kolla/globals.yml``.18:44
sdake   set 'ext_bridge_name': name of the bridge you want neutron to use as an external bridge.  By default it should be br-ex.18:44
sdake   set 'tunnel_interface': interface name used for connectivity between nodes in kubernetes cluster, in most of cases it matches the name of the kubernetes host management interface.  To determine this, ``grep network_interface /etc/kolla/globals.yml``.18:44
sdakeand this:    set 'external_vip': your external ip address18:45
kolla-slack<kfox1111> Yes, but give me 15 min. Its complicated18:45
sdakekfox1111 roger - yo ugot it :)18:45
sdakerwellum i'll udpate the ret of the doc whlie we wait on kfox1111 - this is crucial information and i think closes the loop on the current open questions in hte deployment guide18:46
sdakein other news, my wife wants me to order an arm 64 rackmount server18:46
sdakeso hrw I guess I will have hardware to work wit h:)18:47
*** ksumit has quit IRC18:47
sdakekfox1111 in 15 mins if you could answer - is there a QFP to RJ45 connector18:47
sdakeall these arm servers come with QFP outputs18:48
sdakeand my switch is rj4518:48
sdakerwellum if you knwo as well - that would be cool :)18:48
kolla-slack<kfox1111> Qsfp you mean?18:48
kolla-slack<kfox1111> And what speed?18:48
sdakekfox1111 thats 40gig, there is a 10gig qfp18:48
sdakeor sfp18:48
sdakemaybe i got the term wrong18:49
kolla-slack<kfox1111> Sfp is gig. Spf+ 10g. Qsfp 4018:49
sdakei'd really like to not hae to pay 1k/port for a switch18:49
sdakei've found a nexus 9k on provantage that is 48 port for 13k18:50
sdakewhich is too much for me to bite off for personal use18:50
kolla-slack<kfox1111> Which do you have?18:50
sdakei struggled to rationalize 100$/port ;)18:50
kolla-slack<kfox1111> Probably sfp gigabit?18:50
sdakei hae a netgear prosafe 10gig rj45 switch18:50
kolla-slack<kfox1111> The spec for sfp allows enough ppwer for a sfptorj45 adapter at gigabit.18:51
kolla-slack<kfox1111> Sfp+ doesnt. :/18:51
sdakesadface18:51
kolla-slack<kfox1111> Yeah. :(18:51
*** krtaylor has joined #openstack-kolla18:52
sdakethanks for answering that question kfox111118:52
sdakekfox1111 i've been looking for an answer to that for days18:52
kolla-slack<kfox1111> Someone really dropped the ball on that one. Np18:52
sdakeas I build out my datacenter^Hhacked together lab18:52
*** iceyao has joined #openstack-kolla18:52
kolla-slack<kfox1111> Had to fight that one alot myself.18:52
sdakethe stupid copper qfp+ connectors are 300 bucks a crack too iirc18:53
sdakeredonkulous18:53
*** jascott1_ has quit IRC18:55
rwellumsdake: I wonder if eanylin issue was that in globals.yaml he did not specify: neutron_plugin_agent: "openvswitch"18:57
*** iceyao has quit IRC18:57
kfox1111back.18:58
kfox1111ok... so...18:58
*** nathharp has joined #openstack-kolla18:58
kfox1111kubernetes services.18:58
sdakerwellum I followed the guide *exactly*18:58
kfox1111its kind of a load balancer.18:59
sdakeI don't know if eanlyin does something special18:59
rwellumI added that as a comment sdake18:59
rwellumI feel it's needed18:59
rwellumfeel=tested18:59
kfox1111by default the type is ClusterIp.18:59
sdakerwellum openvswitch should be the default18:59
rwellumsdake: ok18:59
sdakerwellum no reason to override it18:59
kfox1111and it grabs an unallocated one from the service ip range if the user hasn't specified.18:59
rwellumack18:59
sdakerwellum unless it didn't work without it - which would indicate a problem elsewhere18:59
kfox1111another important part is kube-proxy.19:00
kfox1111it basically reads all svc's from k8s, and writes iptables rules that19:00
kfox1111convert all traffic going to the svc's cluster ip to go one of the pods ip's bound to the svc.19:00
kfox1111so kube-proxy is basically the component that implements the load balancer funcionality.19:01
kfox1111but its a distributed load balancer. all nodes have to have it.19:01
sdakeit being kube-proxy?19:01
kfox1111yeah.19:02
sdakeok makes sense19:02
*** rstarmer has quit IRC19:02
kfox1111so.19:02
kfox1111the svc ips never actually exist in any real way.19:02
kfox1111traffic destined for the ip get rewritten by iptables, but19:03
kfox1111the ip isn't assigned to any interface like normally happens.19:03
kfox1111so. then comes external ip's.19:04
kfox1111so, k8s services have the option to use an external ip instead of a cluster ip.19:04
kfox1111the behavior is identicle.19:04
kfox1111all kube-proxies write iptables rules on their local machiens to map the external ip to flow traffic to the service's pods.19:05
kfox1111But,19:05
kfox1111if you make that ip a real ip on one of the hosts, it will work external to k8s.19:05
kfox1111and iptables will still route the traffic to the pods just like it was coming from within k8s.19:06
sdakekool19:07
sdakecool19:07
sdakeso - we have this:19:07
sdake         keystone:19:07
sdake           all:19:07
sdake             admin_port_external: "true"19:07
sdakethe reason for that is otherwise nobody can do any admin activites with keystone19:08
sdakein that case, we should set            external_vip: "192.168.7.101"19:08
sdaketo our mangement ip?19:08
kfox1111with it set to false, the kubernetes service gets no external ip's.19:08
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Revert "Fix Fluentd warn on dnsmasq.log file parsing"  https://review.openstack.org/45383719:08
kfox1111and the cli then only works from within the k8s cluster.19:09
sdakeright - people want to use it externally19:09
kfox1111if you make it an external port, its bound to the external ip and can be used externally to k8s.19:09
kfox1111yeah.19:09
kfox1111you can set it to whatever network you want. juts needs to be set to an ip on a mchine running kube-proxy and the sdn.19:10
kfox1111for ha, that can be a vip managed by keepalived that floats between machines.19:10
kfox1111(or pacemaker, or whatever your failover tool of choice)19:10
sdakekfox1111 small sips at a time )19:10
sdakejust tyring to get AIO rolling19:11
kfox1111k19:11
sdakeso really the cloud.yaml in the review is correct as is (with the sed operation)19:11
sdakefor AIO?19:11
rwellumsdake: seems like it as per kfox111119:12
rwellumMy main issue is that I saw the 192.x address in your example and assumed it was an external interface19:13
kfox1111just glancing over it, seems ok.19:13
rwellumSo I used my external interface instead of mgmt19:13
sdakerwellum ya that is bound to cause problems19:13
sdakerwellum i'd also recommend not ssigning an ip addres to your external neutron interface19:13
kfox1111sdake: maybe the docs should use variables and @VARNAME@ substituations.19:14
kfox1111then its set explicitly all at once at the top.19:14
kfox1111would probably be less confusing.19:14
kfox1111external_vip=xxxxxx19:14
kfox1111sed -i "s/@external_vip@/$external_vip/" ....19:14
kfox1111etc.19:14
sdakekfox1111 without an explination of what external_vip is (as you ahve provided) that would be more confusing :)19:15
sdakekfox1111 baby steps = AIO guide19:15
kfox1111sdake: less error prone then.19:15
sdakeif peopel can get deployment running AIO they can evaluate kolla-kubernetes as it stands today19:15
sdakeif they can't get deployment running becasue they have the same questions about all these variables I have, then they will crater and may not eval at all19:16
*** lrensing has quit IRC19:16
kfox1111sdake: but if they cant run it at their site becausee they cant find the 72 places ips are scatterd about the docs and set them right, the same issue will happen.19:17
kfox1111just my 2cents.19:17
sdakekfox1111 agreed - this doc needs further work19:18
sdakekfox1111 i'd liek to mostly rm -rf docs/*19:18
sdakeand start over19:18
sdakethis is that :)19:18
sdakesince the docs are mostly bitrot19:18
sdakenote i said mostly - some things should stay19:18
sdakewe started over the code with helm but never started ove the docs19:19
sdakedoing that now19:19
kfox1111I've already stated my beliefs around the docs. not going to do so again.19:19
*** ccesario has quit IRC19:19
sdakekfox1111 i guess I dont recall you having strong beliefs about them19:20
sdakewould you mind repeating - i've got like 20 things going on related to docs efforts and have trouble maintaining context19:20
kfox1111copy paste nightmare today. not tested.19:20
kfox1111sorry. not gated.19:21
sdakeoh right19:21
sdakei do recall this19:21
sdakecopy and paste is good - lets people get going19:21
kfox1111shell scripts are good.19:21
sdakewe can simplify that into a contianer implementation19:21
kfox1111they can copy paste if they care. they can ignore if they don't.19:21
kfox1111and its gateable.19:21
*** nathharp has quit IRC19:21
kfox1111copy paste from text, bad.19:21
*** bswartz has quit IRC19:22
sdakewe can't pbulish shell scripts on docs.oo19:22
sdakethat is a key key thing we should be doing19:23
sdakeis pubishng docs on docs.oo19:23
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338419:23
sdakerather then say "go cut and paste this shelll script"19:23
sdakebut I understand your position19:23
sdakeok rwellum - you have enough to get going now?19:24
rwellumYeah 'ish19:24
rwellumThat 8080 fix still doesn't work all the way. eanylin second attempt works better but fails during secret gen.19:24
sbezverkkfox1111: updated PS, let see what happen19:25
sbezverkrwellum: are you talking about aipserver?19:25
rwellumStill not sure if I need two working bridges or not19:25
sdakerwellum you shouldnl't need any bridge i think19:25
sdakejust two interfaces in your vm19:25
sdakeone interface is your management interface, the other being your neutron interface (without an IP)19:25
rwellumsbezverk: the work-around to the 'cannot connect' to kuebadm credentials issue19:27
*** bmace has joined #openstack-kolla19:28
sbezverkrwellum: do you mind to share yot setup in webex? I have never seen failure after following steps I provided..19:30
rwellumsbezverk: actually I got it up and working this morning, just some confusion over the external_vip - but I think what I had orginally was fine based on sdake and kfox111119:31
*** lrensing has joined #openstack-kolla19:32
rwellumSo I am rebuilding back to that19:32
rwellumI had an issue launching a vm - the final step if you like - might need help there19:32
sdakerwellum i also had an issue there19:32
sdakethere is some report that the placemenet api source container is busted19:32
sbezverksdake: 3.0.0 binary19:33
sbezverkis busted for sure19:33
sdakesbezverk i know that19:33
sbezverk4.0.0 source works19:33
sdakesbezverk there is no placement -api for 3.0.0 images19:34
sdakeand we dont backport features19:34
*** ksumit has joined #openstack-kolla19:34
kfox1111https://review.openstack.org/#/c/452850/ is good for a second.19:34
*** rwallner has quit IRC19:35
*** rwallner has joined #openstack-kolla19:36
*** nathharp has joined #openstack-kolla19:37
*** manheim_ has joined #openstack-kolla19:37
*** rwallner has quit IRC19:42
sbezverkrwellum: when you get to that point, please ping me19:42
openstackgerritMerged openstack/kolla-kubernetes master: Remove services/cinder  https://review.openstack.org/45285019:44
*** nathharp has quit IRC19:44
*** rwallner_ has joined #openstack-kolla19:46
*** oanson has quit IRC19:47
*** rwallner_ has quit IRC19:51
rwellumAck sbezverk19:52
*** nathharp has joined #openstack-kolla19:53
*** oanson has joined #openstack-kolla19:55
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Placement support in nova service chart.  https://review.openstack.org/45384619:56
*** nathharp has quit IRC19:56
kfox1111sbezverk: please have a look at that review. still needs some deps added, but should be a way to move forward without dropping 2/3 support.19:56
*** harlowja has joined #openstack-kolla19:58
kfox1111harlowja: hi. :)19:58
harlowjaoh hi19:58
* harlowja noticied my irc client wasn't on, lol19:58
kfox1111:)19:58
harlowjakfox1111 whats up19:58
kfox1111not much. still plugging along. u?19:58
harlowjasame19:58
kfox1111inc0: sdake: https://review.openstack.org/#/c/453384/ please.20:01
*** pcaruana has quit IRC20:01
sbezverkkfox1111: all green :-)20:01
kfox1111yup. :)20:01
sdakekfox1111 30 mins meeting then will review if inc0 hasn't got to it20:01
*** magicboiz has quit IRC20:01
kfox1111hopefully taht will get us back to the stability we had before 1.6.20:01
sbezverkkfox1111: yep.. about the placement, looks good, just a small request about var name..20:02
inc0done kfox111120:02
kfox1111inc0: thx.20:03
kfox1111sbezverk: fixed. :)20:04
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: WIP: Placement support in nova service chart.  https://review.openstack.org/45384620:04
sbezverkkfox1111: nice, thanks, now even I can understand it ;)20:05
kfox1111helm 2.3 is imminent.20:05
kfox1111in the last 24 hours, like dozens of issues have closed. down to 5 open now.20:05
*** manheim_ has quit IRC20:06
sbezverkkfox1111: I hope going to helm 2.3 will not be as distructive as going to kube 1.6 ;)20:06
kfox1111+120:06
kfox1111https://review.openstack.org/#/c/446274/'s good for reviews I think.20:07
openstackgerritMerged openstack/kolla-kubernetes master: Optimizing multinode job at the gate  https://review.openstack.org/45338420:09
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Fernet Token Support  https://review.openstack.org/44627420:12
*** magicboiz has joined #openstack-kolla20:14
*** satyar has quit IRC20:20
rwellumsbezverk: I'm at the point where runonce has run successfully and I can create a demo vm, floating ip etc20:35
*** eaguilar has quit IRC20:35
sbezverkrwellum: nice20:36
kfox1111fasinating: https://review.openstack.org/#/c/451525/20:36
kfox1111rwellum: congrats. :)20:36
rwellumthanks20:36
kfox1111I wonder if that revew could be tweaked to push out fluentd formatted messages directly.20:36
rwellumsbezverk - the script recommends: proceed? https://www.irccloud.com/pastebin/oOVn3BEV/20:37
kfox1111though... I guess that ps along with the kubernetes systemd driver and some kind of fluentd shipper would do the job too.20:37
sbezverkrwellum: net id used in the command line is yours right?20:38
sbezverkkfox1111: easy. fluentd can get and accumulate all that info from different sources20:38
rwellumsbezverk: it's what 'init-runonce' spat out at the end - assuming it's correct?20:38
kfox1111oh, nice. they finally released fluent-bit. 0.11.20:38
kfox1111sbezverk: fluentd directly is a bit complicated as its needs log rolling and some other bits.20:39
kfox1111if the logger in python itself could send out directly, it would simplify things a lot.20:39
kfox1111enabling proper log watching for rotation broke our site's neutron for a while. :/20:39
sbezverkkfox1111: I see20:40
kfox1111I really really really want to see a log shipper build in log rotation and log pruning all into one solution.20:40
kfox1111rather then the mismatched seperate tools we have today. :/20:40
sbezverkrwellum: I have not used that script, but try it20:40
sbezverkkfox1111: what is log shipper?20:41
sbezverkI mean I have not heard that name before20:41
kfox1111fluentd, fluent-bit, filebeat, logstashd, etc are all shippers.20:41
kfox1111they read the logs from some logfile and ship them somewhere else.20:41
kfox1111fluentd and logstash are also processors.20:42
kfox1111they can manipulate the logs before shipping them elsewhere.20:42
rwellumsbezverk:  https://www.irccloud.com/pastebin/tFhFZnPH/20:43
sbezverkkfox1111: ah ok, so it was just a generic name, I thought there is a product with this name20:43
kfox1111sbezverk: ah. sorry, yeah. generic term for all the similar tools.20:43
sbezverkrwellum: untill you see error, so far so good ;)20:43
rwellumsbezverk, kfox1111 so the guide states: https://www.irccloud.com/pastebin/YWV8Ehsz/20:44
rwellumDoesn't seem at all correct to me.20:44
kfox1111rwellum: yeah, thats wrong.20:45
sbezverkrwellum: hm looks ok20:45
rwellum:)20:45
sbezverkkfox1111: why?20:45
kfox1111the optput of openstack floating ip create public1 is not only a uuid.20:45
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/45101820:45
sbezverkit creates floating ip, return it is id and then adds it to vm??20:45
rwellumsbezverk: it returns a whole table of info20:46
kfox1111sbezverk: rwellum: see ttests/bin/basic_tests.sh"20:46
kfox1111sbezverk: rwellum: see tests/bin/basic_tests.sh20:46
sbezverkkfox1111: right right, it is missing awk20:46
kfox1111FIP=$(openstack floating ip create external -f value -c floating_ip_address)20:46
sbezverkor -c -f in command line20:47
rwellum    openstack server add floating ip demo1 $(openstack floating ip create public1 -f value -c floating_ip_address)20:48
kfox1111demo1 or public1?20:49
sbezverkit should be number net20:49
sbezverkin create floating ip command20:49
rwellum'number net'? sbezverk20:50
sbezverkrwellum: uuid or name will work20:50
sbezverkin your case public1 is good20:51
kfox1111rwellum: oh. sorry. I didn't see the nesting.20:51
kfox1111that should work.20:51
rwellumhttps://www.irccloud.com/pastebin/5TQTx205/20:52
rwellumI think I'm confused....20:52
*** rwsu has quit IRC20:53
kfox1111oh.20:53
kfox1111whats the name of the vm?20:53
kfox1111oh....20:54
kfox1111nova show demo120:54
kfox1111I'm guessing its not scheduled yet.20:54
kfox1111cant add a fip to a vm that hasn't got an ip allocated to it.20:55
rwellumhttps://www.irccloud.com/pastebin/3XLPtmFN/20:55
*** vhosakot_ has quit IRC20:55
rwellumvm state is error20:55
kfox1111see tests/bin/basic_tests.sh for some of the stuff we do in the gate. it has pauses in there to deal with letting things start up before proceeding on.20:55
kfox1111yeah. that would be the problem.20:55
kfox1111that implies the scheduler's not workign...20:56
kfox1111is this a 2, a 3 or a 4.x deployment?20:56
kfox1111(4.x  is rather lightly tested at the moment)20:56
rwellumSorry kfox1111  - just following the guide - I'm not sure.20:56
kfox1111docker ps20:56
kfox1111and see what image versions its using.20:56
rwellumHmm says docker not running20:57
kfox1111hmm... sudo?20:57
rwellumhttps://www.irccloud.com/pastebin/jivZzLH3/20:58
kfox1111wow... its using sha's now... how not helpful. :/21:00
sbezverkrwellum: can you get nova scheduler and conductor logs?21:00
*** jrobinson has joined #openstack-kolla21:00
*** jtriley has quit IRC21:00
kfox1111kubectl get pods nova-compute-zjx8f -o yaml | grep image21:01
kfox1111kubectl get pods nova-compute-zjx8f -n kolla -o yaml | grep image21:01
rwellumsudo kubectl log nova-scheduler-0 -n kolla - like that sbezverk21:02
rwellumsbezverk https://www.irccloud.com/pastebin/dZbqkePp/21:03
sbezverkrwellum: nope kubectl exec nova-scheduler-0  -n kolla -- cat /var/log/kolla/nova/nova-scheduler.log21:03
sbezverkand same thing for conductor21:03
*** athomas has quit IRC21:03
sdakerwellum re secret generation failing21:05
sdakerwellum that isn't totally surprising - the secret generator python code is probably not exporting the envrionment to kubectl21:06
sbezverksdake: right it expects to have current namespace set to kolla21:07
sdakesbezverk nah - what i mean is kubectl needs KUBE_CONFIG env variable set21:07
sdakehen failed later on secret generation:21:08
sdakeCommand 'kubectl create secret generic congress-database-password  --from-literal=password=SM35oqzKO7J6uKodeCLCsz5dBbaMr3O6Au2nPgsz --namespace=kolla' returned non-zero exit status 121:08
sdakeThe connection to the server localhost:8080 was refused - did you specify the right host or port?21:08
sdakeCommand 'kubectl create secret generic panko-database-password  --from-literal=password=2OfzleXR4sABcP9LH8GaKHpj0DQctXxfhbHVbY9h --namespace=kolla' returned non-zero exit status 121:08
sdakeThen I did:21:08
sdakesudo sed -i 's/insecure-port=0/insecure-port=8080/' /etc/kubernetes/manifests/kube-apiserver.yaml21:08
sdakeAnd secret generation worked again.21:08
sbezverksdake: it is so much easier to use $HOME/.kube/config21:08
*** mgoddard has joined #openstack-kolla21:08
kfox1111sdake: what sbezverk said. :)21:08
sdakethis is what we have in our docs: Load the kubedm credentials into the system::21:09
sdake    sudo cp /etc/kubernetes/admin.conf $HOME/21:09
sdake    sudo chown $(id -u):$(id -g) $HOME/admin.conf21:09
sdake    export KUBECONFIG=$HOME/admin.conf21:09
sdakeare you saying that sudo cp should be to $HOME/.kube/config?21:09
kfox1111yeah.21:09
rwellumhttp://paste.openstack.org/show/605565/ and http://paste.openstack.org/show/605567/ - sbezverk21:10
rwellumsdake: we've already made that comment in the review21:10
kfox1111sdake: the dev env, what version?21:10
kfox1111sorry, not dev env. your doc.21:10
kfox11112, 3 or 4?21:10
sdakehttps://review.openstack.org/#/c/447356/21:11
sdakeversion 4 of images21:11
kfox1111did you do the cell stuff?21:11
sdakeindeed21:11
sdakeit is done last21:11
rwellumhttps://www.irccloud.com/pastebin/z2JPvX7N/kfox111121:11
kfox1111sdake: have you gotten al the way through sshing into a workign vm?21:12
sdakekfox1111 nope - waiting on everyont to finish barfing on the doc before i rev it21:13
sdake;)21:13
sdakeand then test it21:13
sdakeits pretty close21:13
sdakelast time i started a vm it entered FAILED state21:13
kfox1111ah. so testing has gotten up to service starting. not vm's.21:13
kfox1111rwellum: ok. so where your at is the farthest anyone's gotten.21:13
sdakeeanlin has got to the point he can use novanovnc to ping / curl out of vms21:13
sdakerwellum you can ssh into the vm?21:13
kfox1111sdake: its not scheduling for him.21:14
kfox1111hence the wquestions about cells.21:14
sdakethe edocs are pretty clear , you have to wait for nova copute to enter running state21:14
sdakeegonzales verified it21:14
kfox1111yeah, we really need a job to add more nodes after its been stood up though.21:15
kfox1111so, that may be the problem though.21:15
kfox1111not sujre how to tell if a nova-compute is bound to a cell.21:15
kfox1111thats all new stuff to me.21:16
sdakeegonzalz seemd to understnd it in great detail21:16
sbezverkkfox1111: it seems by default all VM belongs to cell121:16
kfox1111sbezverk: yes, but only when added to the cell.21:17
kfox1111which only happens if:21:17
kfox11111. it exists before adding the cell1, or21:17
kfox11112. running a tool manually afterwards.21:17
sbezverkright21:17
kfox1111so rwellum might have a nova-compute in limbo.21:17
sbezverkI think it is only applicable to 1st cell21:17
sbezverkyou can add other cells later (I have not tried)21:17
sbezverkand allocate compute resource to new cells21:18
kfox1111yeah. though never figured out how to say what cell a compute should lland in.21:18
kfox1111but I'm starting to really struggle with nova's decidion to need to go to cells.21:19
kfox1111I have hit the wall myself with nova. much past 100 computes, it starts running into issues.21:19
kfox1111but kubernetes does 5000 compute nodes at scale.21:19
kfox1111with one 'cell'.21:19
inc0kfox1111: one word - rabbitmq21:20
kfox1111inc0: yeah. and mariadb.21:20
*** rcherrueau has joined #openstack-kolla21:20
kfox1111and maybe python.21:20
inc0and mariadb21:20
inc0python nto so much21:20
kfox1111and lack of gating on performance.21:20
inc0non-optimal python - maybe21:20
rwellumkfox1111: how to tell if nova-compute is in limbo? Looks good from k8s pov21:21
inc0you rarely end on mem/cpu performance21:21
inc0it's io that kills you21:21
kfox1111even still, nova should be able to do within one order of magitude, http://blog.kubernetes.io/2017/03/scalability-updates-in-kubernetes-1.6.html21:21
inc0yeah but it would need to be rewritten21:21
kfox1111rwellum: no clue. :/ never done ocata yet. :/21:21
rwellumFair enough.21:22
kfox1111inc0: yes. but its less effort then them splitting everything out to cells I think.21:22
inc0depends, if you want to remain upgradable, then no21:22
inc0if you want to start fresh and re-thing certain ideas21:22
inc0without need of backwards compatibility21:23
inc0stuff could look totally differently21:23
kfox1111inc0: they should be able to dump mysql state into etcd or something.21:23
*** rwallner has joined #openstack-kolla21:23
kfox1111there should be a migration path.21:23
kfox1111they have to do that with cells anyway.21:23
inc0I don't think that mysql is necessary a problem here21:23
*** goldyfruit has quit IRC21:23
rwellumHave to head home, will be back this evening and will leave my cluster in this current state if anyone wants to jump on a take a look etc.21:23
kfox1111I'm pretty sure its probably 50/50 between mariadb and rabbit.21:23
inc0I mean mysql can handle millions of records21:23
inc0if used correctly21:23
kfox1111inc0: latency.21:24
inc0etcd will have latency too21:24
*** mgoddard has quit IRC21:24
kfox1111yeah, but key value is so much simpler to implement then acid compliant relational db's.21:24
kfox1111thats gotta effect latency at some point.21:24
sdakerwellum i have a dentist appointment from 3-5pm21:25
sdakerwellum should be back tonight21:25
inc0yes, but I thin problem at hand is more in sheer number of microservices that need to take part in spawning a vm21:25
sbezverkrwellum: did you manage to get logs?21:25
inc0api->conductor->scheduler->conductor->compute21:25
inc0and add http calls to neutron/cinder in between21:25
inc0blocking http21:26
inc0and keystone...21:26
sdakekfox1111 etcd is not all that scalable with HA enabled nor performanct21:26
inc0every piece asks keystone for permission21:26
sdakeand its a key value store not a ACID database21:26
kfox1111inc0: that too.21:26
kfox1111sdake: scalable enough for 5000 nodes and 150,000 pods. good enough for me. :)21:27
kfox1111thats way less management though then trying to manage the same number of nodes / vms then with nova.21:27
inc0but again, it's design of k8s not databases they use21:27
kfox1111you'd have to have dozens of cells I'm afraid.21:27
*** rwallner has quit IRC21:27
inc0but agree, I wouldnt use relational db myself today for software like that21:28
kfox1111inc0: yeah. thats certainly part of it.21:28
inc0lessons learned;)21:28
inc0google knows how to write scallable software21:28
inc0we know that too...today21:29
sdakei analyzed the perfomrnace ofa 123 node cluster running21:29
inc0after all the lessons;)21:29
sdake40% cpu utilization consumed by the i40 driver21:29
*** Manheim has quit IRC21:29
inc0sometimes I'd love to just start fresh21:30
kfox1111I think thats one of the things that potentially makes openstack great.21:30
kfox1111its greatest streangth is its standard api's.21:30
inc0not to solve this one21:30
kfox1111those don't strictly need to continue to use the same code base to be functional.21:30
inc0but you need to follow same API21:31
inc0and that's issue21:31
kfox1111nova api could retarget onto k8s directly and use k8s's scheduler.21:31
kfox1111no one would be the wiser.21:31
inc0k8s scheduler isn't much better21:31
kfox1111and would scale much larger, emdediately.21:31
inc0is it scheduler that dies first tho?21:32
inc0and not really it couldn't as scheduler communicates via amqp21:32
kfox1111nova's falls over at about 300 nodes.21:32
inc0and it's rabbit to die21:32
*** manheim has joined #openstack-kolla21:32
inc0also cells is bad idea imho because well...they don't work with neutron well21:33
kfox1111potato patato? if it takes a architectural rewrite to get the scedular decoupled from rabbit to make it scale, it may just be easier to target k8s's directly and not need to provide one at all.21:33
*** signed8bit has joined #openstack-kolla21:33
inc0and since quotas was that much of a pain to synchronize...imagine cells21:33
kfox1111that too. neutron needs cells too before its really viable.21:33
kfox1111unless neutron can scale and nova can't, so cells are the only way.21:34
kfox1111yeah. :/21:34
inc0that part I have no info...in fact we asked this exact question in PTG21:34
kfox1111I still see nova's statistics table get out of sync with reality. :/21:34
kfox1111buggy. :/21:34
inc0another thing that k8s did correctly is that they built on top of existing sdn controller like calico21:39
inc0instead of writing neutron21:39
inc0but that's because this thing existed;)21:39
sbezverkinc0: I thought k8s was way before any calico ;)21:39
inc0I'm pretty sure calico is older21:40
sbezverkinc0: it was running on flannel21:40
kfox1111calico is older I think.21:40
sbezverkinc0: if you count 10 years while k8s was under google umbrella, I doubt ;)21:40
kfox1111sbezverk: k8s isn't that old. borg is.21:41
inc0well, borg != k8s21:41
inc0and I'm pretty damn sure it's not even close21:41
inc0they might've used some ideas21:41
kfox1111k8s is as nice as it is, because it doesn't have a lot of legacy stuff from borg. :)21:41
kfox1111labels for example are way better in k8s then in borg I hear.21:41
kfox1111k8s is borg 3.x from what I gather.21:42
kfox1111borg was first. there was a second attempt that failed, and k8s is third.21:42
sbezverkkfox1111: reminds me Agent Smith monolog in Matrix ;)21:42
kfox11113.x releases are often much better. :)21:43
kfox11111.x you think you know what your doing but really have no clue.21:43
kfox11112.x you know what you did wrong and think you know how to fix it. then realize there was a bit more you didn't understand.21:43
kfox1111by 3, you usually have it figured out.21:43
sbezverkkfox1111: :)21:43
inc0if we would rewrite openstack from scrach today we'd end up with great project that nobody will use21:44
sbezverkmental note, start versioning from 3.X ;)21:44
*** manheim has quit IRC21:44
kfox1111sbezverk: some folks to that... operators don't look highly on that kind of behavior. ;)21:44
inc0kolla-k8s is 2.0 in this nomenclature21:44
inc0kolla-ansible is kolla 3.0;)21:44
inc01 - k8s, 2 - compose, 3 - ansible21:44
kfox11112 or 4.21:45
kfox1111depending on how you look at it.21:45
kfox1111yeah.21:45
inc0but I don't think kolla-k8s is evolution of kolla-ansible21:45
inc0as we're not trying to fix kolla-ansible problems with kolla-k8s21:45
kfox1111I kind of disagree with that. ;) but I'm biased, so you can ignore me.21:46
inc0I refuse to say that ansible is problem because it's ansible:P21:46
kfox1111config's something that I think is kind of broken in kolla-ansible and being fixed in kolla-kubernetes.21:46
inc0how so?21:46
*** fooliouno has joined #openstack-kolla21:47
kfox1111kolla-ansible is fairly rigid around architectural choices.21:47
kfox1111like, one mariadb or one rabbit.21:47
inc0well, technically not really21:47
sbezverkkfox1111: another thing is fixed inventory21:47
inc0I mean you can go crazy if you want21:47
kfox1111I'd like a config that lets me do one rabbit per opentack service, with a second rabbit setup for metering.21:47
fooliounokfox1111: Is it theoretically possible to run ODL on kolla-k8s with Flannel as an overlay network?21:48
kfox1111and when I say want, I really mean, need. :)21:48
inc0for i in service; kolla-ansible deploy -t mariadb -i dedicated_inventory;21:48
fooliounoOr does ODL need to replace Flannel21:48
kfox1111fooliouno: theoretically, yeah.21:48
inc0and override mariadb_address in globals21:48
sbezverkfooliouno: kfox1111: is there ODL cni plugin?21:48
kfox1111inc0: overriding it for each service how?21:48
inc0ahh well, good point21:49
inc0but that's easy fix21:49
inc0literally one commit21:49
fooliounosbezverk: Not that I am aware of. I am trying to bring up an ODL container on kolla-k8s and test it.21:49
sbezverkdo not remember seeing it21:49
inc0just nobody wrote it21:49
*** manheim has joined #openstack-kolla21:49
inc0add nova_mariadb = {{ mariadb_address }}21:50
kfox1111inc0: yeah. I'm trying to fix that kind of thing out of the box in kolla-kubernetes.21:50
inc0and that's it21:50
kfox1111not insurmountable at all on the kolla-ansible side.21:50
fooliounosbezverk: I waned to see if ODL can co-exist with Flannel before I go about messing with it.21:50
inc0yeah, cool, waht I'm saying this isn't fixing ansible21:50
sbezverkfooliouno: do you want to use it just for kube cluster or for openstack network plugin?21:50
kfox1111but jsut stuff we've learned from the kolla-ansible side that are fixing in implementing kolla-kubernetes.21:50
inc0and it's not design issue with kolla-ansible21:50
inc0it's just commit nobody made21:50
kfox1111yeah.21:50
inc0if you want this, really quick to implement21:51
inc0again, nobody did that because nobody wanted it;)21:51
fooliounosbezverk: I want to use ODL for openstack network only21:52
inc0imho kolla-ansible is really flexible as it is today21:52
inc0kolla-k8s will need some time to arrive to this kind of flexibility21:52
kfox1111inc0: does it have an easy way to support multiple, different host aggregate computes?21:52
inc0to me these are 2 different modes of deployment, neither better than other21:52
kfox1111configured seperately?21:53
inc0yes21:53
inc0in inventory you can override variables21:53
sbezverkfooliouno: Oh, ok. then you will need to do some legwork, especially in configuration area as we generate config only for ovs/ml221:53
kfox1111inc0: hmm... ok.21:53
kfox1111so, thats pretemplated values that then render into the config?21:54
inc0for example you can have host1 enable_openvswitch=false21:54
inc0config, playbook logic, all that21:54
fooliounosbezverk: Willing to do that if its feasible. Any pointers on where to start looking.21:54
inc0good example would be to have different interface name per hosts21:54
kfox1111inc0: cool.21:54
sbezverkinc0: when you know you setup, yes, but let's say you add new nodes and want to move some roles around, how easy it would be?21:55
inc0what you do is specify neutron_external_interface=eth0 in inventoryu in same line as host declaration21:55
sbezverkwith k8s it is really really easy21:55
inc0(there are other ways to specify too)21:55
inc0one caveat - you don't define it in globals then because what is in globals takes precedence21:55
kfox1111inc0: interesting. I didn't really think of using inventory that way.21:55
mnaserwe use it that way ^21:56
kfox1111thats good and bad I guess. can you define vars at a group level,21:56
mnaserto do very questionable things21:56
kfox1111and tag them onto nodes?21:56
mnaserlike enable_ceph for rgw only21:56
inc0yeah kfox111121:56
kfox1111cool. :)21:56
inc0sbezverk: add node is easy21:56
inc0add it to inventory and run deploy21:56
kfox1111then its much closer then I thought.21:56
mnaserits useful for brownfield things21:56
inc0we don't really use "host roles" per se in ansible21:56
inc0you can specify this all the way down to services runing21:57
sbezverkinc0: ok I see21:57
inc0wanna run api on copute node for reasons? add this compute node to nova-api group and run deploy21:57
*** mbruzek has quit IRC21:58
inc0kfox1111: I dare you to give me use case and I'll either show you how to do it or how to fix ansible to do it;)21:58
inc0like mariadb separated thing, agree not easy now, simple fix tho21:59
kfox1111inc0: I may take you up on that some day. :)21:59
inc0I'm really happy about kolla-ansible tbh21:59
inc0with all issues we had with ansible gplv3, it's treating us really well all in all21:59
*** shashank_t_ has quit IRC22:00
*** shashank_t_ has joined #openstack-kolla22:00
inc0one thing we have problems with today due to ansible is rolling upgrade22:00
inc0as it's not possible in ansible as of today to specify "I want to do this task at same time and this task in rolling"22:01
inc0but still possible if you do --forks 122:01
inc0then it will do stuff on one host at the time22:01
inc0not ideal22:01
*** magicboiz has quit IRC22:02
*** magicboiz has joined #openstack-kolla22:02
kfox1111ah.22:03
mnaserinc0 there is ia hack22:04
*** shashank_t_ has quit IRC22:04
mnaseri rememeber using it22:05
mnaserone second22:05
*** harlowja has quit IRC22:05
mnaserinc0 https://groups.google.com/d/msg/ansible-project/rBcWzXjt-Xc/CDW2FdlqRh0J22:09
*** jrobinson has quit IRC22:12
*** fooliouno has quit IRC22:15
*** lucasxu has quit IRC22:15
*** jrobinson has joined #openstack-kolla22:18
*** manheim has quit IRC22:25
*** gfidente|afk has quit IRC22:26
kfox1111bummer... incubator/elasticserach is not only bit roten, but too big for minikube testing.22:28
kfox1111:/22:28
*** rwsu has joined #openstack-kolla22:32
*** chas has joined #openstack-kolla22:36
*** chas has quit IRC22:41
*** rcherrueau has quit IRC22:41
*** krtaylor has quit IRC22:45
*** jascott1_ has joined #openstack-kolla22:54
inc0so I ahve funny problem with one of vms22:56
inc0vim, top and dmesg hangs, what's common between these 3?22:56
*** signed8b_ has joined #openstack-kolla22:56
*** signed8b_ has quit IRC22:57
*** chas has joined #openstack-kolla22:57
*** jascott1_ has quit IRC22:58
ksumitThe manila-data image is broken. I tried building it from source on both centos and ubuntu, but it fails to build and deploy.22:58
*** signed8bit has quit IRC23:00
inc0ksumit: fails how?23:00
ksumitFollowing the quick start guide, I get the following message after the 'kolla-build' command:23:01
*** chas has quit IRC23:01
ksumitINFO:kolla.image.build:===========================23:01
ksumitINFO:kolla.image.build:Images that failed to build23:01
ksumitINFO:kolla.image.build:===========================23:01
ksumitERROR:kolla.image.build:manila-data Failed with status: error23:01
inc0ksumit: check logs above23:02
inc0there will be error somewhere23:02
inc0you can try to run kolla-build manila-data23:02
inc0to build only this image23:02
inc0afk, meeting23:03
ksumitFor Centos at least, it seems to be related to https://bugzilla.redhat.com/show_bug.cgi?id=142696323:04
openstackbugzilla.redhat.com bug 1426963 in openstack-manila "missing manlia-data rpm when using Kolla build image" [Unspecified,Closed: notabug] - Assigned to tbarron23:04
*** lrensing has quit IRC23:04
ksumit"The command '/bin/sh -c echo 'binary not yet available for centos'     && /bin/false' returned a non-zero code: 1"23:06
*** lrensing has joined #openstack-kolla23:07
*** lamt has quit IRC23:23
*** lrensing has quit IRC23:23
*** lamt has joined #openstack-kolla23:24
*** lamt has quit IRC23:24
*** harlowja has joined #openstack-kolla23:25
*** lrensing has joined #openstack-kolla23:25
*** jtriley has joined #openstack-kolla23:29
*** rwellum has quit IRC23:31
*** chas has joined #openstack-kolla23:33
*** chas has quit IRC23:38
*** bswartz has joined #openstack-kolla23:39
*** lrensing has quit IRC23:45
*** lrensing has joined #openstack-kolla23:45
*** Pavo has joined #openstack-kolla23:48
sdakesbezverk server has arrived23:49
sdakeit sure is big23:49
sdakemy wife said "protecting shareholder value I see"23:49
kfox1111interesting... rkt is in minikube...23:49
*** krtaylor has joined #openstack-kolla23:51
sdakesup kfox111123:53
*** lrensing has quit IRC23:57
sbezverksdake: so you enjoy the noise ;)23:58
sdakesbezverk i havent truend it on yet23:58
sdakehoepfullly its quiet23:58
sdakeif its not I'm sending it back :)23:58
sbezverkdream on ;)23:58
sdakei got a model with two power supplies23:58
sdakethat will probably be the loudest part of the server23:58
*** alop has joined #openstack-kolla23:59
*** yingjun has joined #openstack-kolla23:59
*** lrensing has joined #openstack-kolla23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!