| rwellum | sdake: cherry pick is still wrong imo (in the doc) | 00:04 |
|---|---|---|
| sbezverk | kfox1111: I have an idea, what if for multinode we finish master with all running and only then join second node | 00:04 |
| spsurya | sdake: hi... | 00:04 |
| sbezverk | we see single node works 100% of cases | 00:04 |
| *** rstarmer has joined #openstack-kolla | 00:05 | |
| kolla-slack | <kfox1111> The idea of multinode was to test out split controller/compute. | 00:05 |
| kolla-slack | <kfox1111> Not sure what good a single node but later with 2 really buys? | 00:06 |
| spsurya | sbezverk: nice but i just got into this [preflight] Some fatal errors occurred: | 00:07 |
| spsurya | /proc/sys/net/bridge/bridge-nf-call-iptables contents are not set to 1 | 00:07 |
| spsurya | /var/lib/etcd is not empty | 00:07 |
| kolla-slack | <kfox1111> I was going to suggest the opposite before 1.6, to remove more of the single node tests. | 00:07 |
| spsurya | sbezverk: did you guys faced this ? | 00:07 |
| sbezverk | kfox1111: no I meant only the way to isntall it | 00:08 |
| sbezverk | right now when we install, we install both nodes before networking | 00:08 |
| sbezverk | I run one node + networking and then add second node | 00:09 |
| sbezverk | I think it will be more reliable at the gate | 00:09 |
| sbezverk | spsurya: make sure you run the latest kube buts | 00:09 |
| sbezverk | bits :) | 00:09 |
| spsurya | sbezverk: 1.6.1 Package ebtables-2.0.10-15.el7.x86_64 already installed and latest version | 00:11 |
| spsurya | Package kubeadm-1.6.1-0.x86_64 already installed and latest version | 00:11 |
| spsurya | Package kubectl-1.6.1-0.x86_64 already installed and latest version | 00:11 |
| spsurya | Package kubelet-1.6.1-0.x86_64 already installed and latest version | 00:11 |
| spsurya | Package kubernetes-cni-0.5.1-0.x86_64 already installed and latest version | 00:11 |
| spsurya | Nothing to do | 00:11 |
| sbezverk | spsurya: not sure then.. | 00:12 |
| kfox1111 | sbezverk: still don't follow. are you talking about just k8s? | 00:12 |
| sbezverk | have you tried to set it /proc/sys/net/bridge/bridge-nf-call-iptables ? | 00:12 |
| kfox1111 | like, ensure calico is up before doing the second node? | 00:12 |
| sbezverk | kfox1111: exactly | 00:12 |
| sbezverk | I tested it in mu local lab and it works | 00:13 |
| spsurya | sbezverk: usually we don't set | 00:13 |
| kfox1111 | sbezverk: ah.... couldn't hurt. and would remove one variable. | 00:13 |
| spsurya | but i can give a try after change | 00:13 |
| sbezverk | I thinking since our goal is not to test kubeadm installation process | 00:13 |
| sbezverk | we could try to optimize it a bit | 00:13 |
| kfox1111 | works for me. | 00:14 |
| spsurya | sbezverk: roger | 00:14 |
| kfox1111 | weird.... http://logs.openstack.org/77/453277/1/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/b28ddb2/logs/pods/kube-system-kube-dns-3913472980-1k2f7.txt | 00:16 |
| kfox1111 | so... it failed to contact kube-apiserver... | 00:18 |
| kfox1111 | looks like it started on the slave system.. | 00:20 |
| sdake | rwellum was OTP | 00:21 |
| sdake | rwellum back - looking | 00:21 |
| sdake | sup spur | 00:21 |
| sdake | sup spsurya | 00:21 |
| kfox1111 | interesting.... | 00:21 |
| sdake | spsurya what can i do for ya | 00:21 |
| *** yingjun has joined #openstack-kolla | 00:21 | |
| *** yingjun has quit IRC | 00:22 | |
| spsurya | sdake: looking into this | 00:22 |
| *** qiliang27 has quit IRC | 00:22 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 00:24 |
| sdake | kfox1111 sbezverk can you fellas ack that above review and approve it so we can get a rendering on docs.oo | 00:24 |
| sdake | kfox1111 i will continue to work on it from this point and solidify it | 00:24 |
| sbezverk | sdake: I have not tested it, can you ask folks who tested it at least +1 it if they are happy? | 00:25 |
| sdake | sbezverk its hard to tet without a rendering on docs.oo | 00:25 |
| sdake | there is a warning at tthe top that indicates it may not work perfectly | 00:25 |
| kfox1111 | well, I do plan on kind of reusing the gate to ensure we cache a tested k8s. | 00:25 |
| sdake | sbezverk i will commit to having it in good shape by friday before i travel | 00:26 |
| sdake | as in people will have tested it completely and all the kinks worked out | 00:26 |
| sdake | i've got wed-fri to do that, and only one dentist appointment in the way :) | 00:26 |
| sdake | sbezverk fwiw I have tested it ,and it wfm | 00:27 |
| sdake | and also eanlyin got it working but is afk atm | 00:27 |
| rwellum | sdake it's -1 for now because that cherry-pick is wrong imo | 00:27 |
| sdake | rwellum see latets review- that is fixed | 00:28 |
| sdake | rwellum thanks for finding that btw, not sure why that snuck back in | 00:28 |
| sdake | things keep "sneaking back in' because its not merged | 00:28 |
| sdake | if i can merge it i have smoething to work from in git diff wise | 00:28 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 00:29 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Disable placement for v3 gate. https://review.openstack.org/453277 | 00:31 |
| rwellum | sdake: looks better thanks. I can +1 - even though I haven't got it working fully myself. | 00:31 |
| sdake | rwellum if your up to working on it today - we can tackle i t together via webex | 00:31 |
| sdake | rwellum it might help if i see if your doing something different then I do | 00:31 |
| *** ksumit has quit IRC | 00:32 | |
| rwellum | sdake: currently assisting Daughter with Earth Environment Science Honors lab :) But I can try at 9pm EST? | 00:33 |
| sdake | so 30 minutes? | 00:33 |
| sdake | i'm around all night | 00:33 |
| rwellum | yes that works | 00:34 |
| *** yingjun has joined #openstack-kolla | 00:34 | |
| rwellum | I'll ping you. | 00:34 |
| *** yuanying has joined #openstack-kolla | 00:34 | |
| *** lucasxu has joined #openstack-kolla | 00:36 | |
| *** lucasxu has quit IRC | 00:40 | |
| kfox1111 | sdake: a few things on the review, then I'm good as per our previous conversation. | 00:45 |
| sdake | kfox1111 got it | 00:45 |
| *** xiaoruiguo has joined #openstack-kolla | 00:46 | |
| *** zhurong has joined #openstack-kolla | 00:47 | |
| *** jascott1_ has joined #openstack-kolla | 00:47 | |
| *** hrw has quit IRC | 00:48 | |
| *** tovin07_ has joined #openstack-kolla | 00:49 | |
| japestinho | sdake I was asked a username by gerrit when exec git review -x 439740, I've never used gerrit before yet :( | 00:50 |
| sdake | japestinho you ahve to have your ssh environment setup | 00:51 |
| sdake | japestinho and gerrit needs to be setup | 00:51 |
| sdake | an alternatie is to go find the review and use the cherrypick feature | 00:51 |
| sdake | leave a comment and I will add a note for those that don't have gerrit available. | 00:51 |
| sdake | (in the rview leave a comment) | 00:52 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add Glance Swift backend support https://review.openstack.org/452059 | 00:53 |
| sdake | kfox1111 thanks for the review, i'm struggling (with my lack of networking experience) to know exactly what needs to be done for the broad direction you have given. could you narrow it down a bit :) | 00:53 |
| mnaser | aw man | 00:53 |
| mnaser | nova_compute on stable/newton + ceph => broke | 00:53 |
| mnaser | :-( | 00:53 |
| mnaser | container runs under user nova | 00:54 |
| mnaser | tries to delete /etc/ceph on start | 00:54 |
| mnaser | the file /etc/ceph/rbdmap cannot be modified by the nova user | 00:54 |
| *** hrw has joined #openstack-kolla | 00:54 | |
| *** jascott1_ has quit IRC | 00:56 | |
| *** qiliang27 has joined #openstack-kolla | 00:57 | |
| mnaser | the file /etc/ceph/rbdmap cannot be modified by the nova user | 00:57 |
| *** eanylin has joined #openstack-kolla | 00:58 | |
| japestinho | sdake my comment still in draft, how to commit it? | 00:58 |
| japestinho | https://www.irccloud.com/pastebin/hTckYXZ0/comment%20on%2044%20patch%20set | 00:58 |
| sbezverk | kfox1111: it looks like that little thing stabilized multinode gate jobs. | 00:59 |
| rwellum | japestinho: on the main review page, there's a reply button at the top. | 00:59 |
| spsurya | mnaser: not sure but seems like some permission issue | 00:59 |
| rwellum | press that, answer -1,0 or +1 | 00:59 |
| mnaser | spsurya it is.. but it looks like the whole codebase is broken | 00:59 |
| mnaser | because i fail to see how any ceph deployment would have successfully completed. | 01:00 |
| sdake | japestinho click reply | 01:00 |
| *** qiliang28 has joined #openstack-kolla | 01:00 | |
| sdake | looks like rwellum answered :) | 01:00 |
| *** hrw has quit IRC | 01:00 | |
| *** qiliang27 has quit IRC | 01:01 | |
| *** qiliang28 is now known as qiliang27 | 01:01 | |
| *** hrw has joined #openstack-kolla | 01:02 | |
| spsurya | mnaser: hmmm | 01:02 |
| mnaser | spsurya strangely enough cinder volume worked? | 01:02 |
| japestinho | rwellum thanks but I forgot give answer -1 or +1 , my bad :( | 01:03 |
| *** Pavo has joined #openstack-kolla | 01:03 | |
| rwellum | japestinho: just edit one of your comments, do the reply again. | 01:04 |
| rwellum | sdake: what is the tox command to render the doc correctly please? | 01:04 |
| sdake | rwellum tox -e docs | 01:05 |
| sdake | cd doc/build | 01:05 |
| sdake | tar -cvzf html.tar.gz html | 01:05 |
| sdake | scp the html.tar.gz to your local machine | 01:05 |
| sdake | then untar them | 01:05 |
| sdake | then open in chrome or hte like | 01:06 |
| spsurya | mnaser: nice | 01:06 |
| spsurya | so issue resolved at a level | 01:06 |
| mnaser | im so confused as to why one worked and one didnt? | 01:07 |
| japestinho | rwellum sorry but actually what's the meaning -1/+1 thing? just check begore I reply again | 01:07 |
| mnaser | docker run -it --rm kolla/centos-binary-cinder-volume:3.0.3 rm -rfv /etc/ceph | 01:08 |
| rwellum | +1 you're happy, -1 unhappy :) | 01:08 |
| rwellum | probably sdake can answer that better | 01:08 |
| *** duonghq has joined #openstack-kolla | 01:09 | |
| *** iceyao has joined #openstack-kolla | 01:10 | |
| rwellum | sdake: i got many pages of angry red output from that tox command and no doc/build | 01:11 |
| *** jascott1_ has joined #openstack-kolla | 01:11 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 01:12 |
| spsurya | japestinho: +1 means you have reviewed the patch and agreed that it is fine from your view. -1 means you have some concern with the changes in patch set that you will tell via firing the comment on that patch. | 01:12 |
| sbezverk | rwellum: deployment guide is for single node only or for multinode as well? | 01:13 |
| openstackgerrit | jimmygc proposed openstack/kolla master: Add vSphere Support https://review.openstack.org/449395 | 01:13 |
| spsurya | and that would be incorporated by the committer | 01:13 |
| rwellum | japestinho: recommend you try going through https://docs.openstack.org/infra/manual/developers.html - will answer a lot of questions about how to set this up, environment etc. | 01:14 |
| spsurya | if he finds that the review commnet is right | 01:14 |
| spsurya | he will incorparate your comment | 01:14 |
| rwellum | sbezverk: originally sdake was talking about 2 bare-metal nodes. That's been removed since then. Personally I've been following it as a single node deployment. | 01:16 |
| mnaser | spsurya trying to emulate what kolla-config does... docker run -it --rm kolla/centos-binary-cinder-volume:3.0.3 rm -rfv /etc/ceph | 01:17 |
| mnaser | rm: cannot remove '/etc/ceph/rbdmap': Permission denied | 01:17 |
| sbezverk | rwellum: got it, thanks. | 01:17 |
| sbezverk | if it goes back to multinode, then we need to change a bit the way nodes join the cluster.. | 01:18 |
| openstackgerrit | jimmygc proposed openstack/kolla master: Add vSphere Support https://review.openstack.org/449395 | 01:20 |
| spsurya | mnaser: may be egonzalez can help better | 01:21 |
| mnaser | i was hoping to do a test deployemnts today | 01:21 |
| spsurya | yes | 01:23 |
| spsurya | nice | 01:23 |
| mnaser | i wonder how cinder-volume worked before | 01:24 |
| rwellum | sbezverk: yes - the 'joins' become very important... | 01:25 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add Glance Swift backend support https://review.openstack.org/452059 | 01:27 |
| spsurya | mnaser: kolla cinder log will certainly help in that | 01:28 |
| mnaser | INFO:__main__:Removing existing destination: /etc/ceph/ | 01:28 |
| mnaser | INFO:__main__:Creating dest parent directory: /etc/ceph | 01:28 |
| mnaser | which means that kolla-config was able to successfully do this | 01:28 |
| *** hrw has quit IRC | 01:29 | |
| spsurya | right | 01:31 |
| *** hrw has joined #openstack-kolla | 01:31 | |
| sdake | rwellum the red is normal | 01:37 |
| *** shashank_t_ has quit IRC | 01:37 | |
| sdake | rwellum it sould have spit out a buidl dire | 01:37 |
| sdake | sbezverk its AIO at present - may be expanded to multinode later | 01:37 |
| mnaser | https://bugs.launchpad.net/kolla-kubernetes/+bug/1659198 | 01:38 |
| openstack | Launchpad bug 1659198 in kolla-kubernetes "nova-compute can not run when use ceph" [Undecided,Confirmed] | 01:38 |
| mnaser | looks like its present there too | 01:38 |
| sdake | goal is tjust to get *something* that documents deployment | 01:38 |
| sdake | as currently the devenv is busted and deployment doesn't work | 01:38 |
| sdake | anywhere | 01:38 |
| mnaser | sdake are you doing any ceph-related deployemnt in k8s right now? | 01:38 |
| sdake | rwellum did you miss this part of the guide: Load the kubedm credentials into the system:: | 01:39 |
| sdake | sudo cp /etc/kubernetes/admin.conf $HOME/ | 01:39 |
| sdake | sudo chown $(id -u):$(id -g) $HOME/admin.conf | 01:39 |
| sdake | export KUBECONFIG=$HOME/admin.conf | 01:39 |
| sdake | rwellum with that (and with the insecure port thing left as is) i can connect - the --insecure-port=0 is intentional as kubeadm 1.6.1 added that dmin.conf feature | 01:39 |
| sdake | kubectl reads env variable KUBAECONFIGA | 01:40 |
| sdake | rwellum i fixed one typo and replced it with another 9firewalld:) | 01:42 |
| sdake | old and tired i am | 01:42 |
| rwellum | sdake: the issue might be I am running as sudo su - | 01:44 |
| rwellum | $HOME is /root etc | 01:44 |
| rwellum | not sure | 01:44 |
| eanylin | sbezverk: Is this needed? | 01:46 |
| eanylin | kollakube res create secret nova-libvirt | 01:46 |
| eanylin | sdake: I saw that typo, was able to put a comment on it :) | 01:46 |
| *** cuongnv has joined #openstack-kolla | 01:48 | |
| sdake | rwellum run as a normal user | 01:49 |
| sdake | rwellum no need ot run as root | 01:49 |
| sdake | that is what the sudos are for :) | 01:50 |
| sdake | sudo su - is definately wrong - try sudo -i | 01:50 |
| sdake | if you want to run as root | 01:50 |
| sdake | i'd highly recommend running as a normal user though and using the sudo commands as they are specified | 01:50 |
| sdake | some commands don't work at all with sudo | 01:50 |
| sdake | for example I ran the adminopenrc creation as sudo and it craters the openstack deployment | 01:50 |
| sdake | can some brilliant networking dude explain to me how to pick a service network | 01:51 |
| sdake | kfox1111 seemed to indicate this is easy | 01:51 |
| sdake | I guess it is for someone that understands networking in detial | 01:51 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add VMware DataStore support to glance https://review.openstack.org/452176 | 01:52 |
| *** zhurong has quit IRC | 01:52 | |
| sdake | i have no idea how to pick that service addr, and as a result have no idea how to change the document to match | 01:52 |
| rwellum | sdake: ack - I had reasons - running in a vm is slightly different but will try | 01:52 |
| sdake | rwellum just do sudo -i | 01:53 |
| sdake | adduser rwellum | 01:54 |
| sdake | then exit | 01:54 |
| sdake | then su - rwellum | 01:54 |
| rwellum | sdake: ack | 01:54 |
| sdake | dont forget to add rwellum to the wheel group :) | 01:54 |
| sdake | and enable the sudoers file | 01:55 |
| sdake | rwellum out of curiosity what is the rationale for running as sudo su -? | 01:56 |
| *** masber has quit IRC | 01:57 | |
| rwellum | sdake: you had some commands with sudo and some not, wasn't consistent. So instead of adding sudo everywhere I just ran as root. | 01:57 |
| *** zhurong has joined #openstack-kolla | 01:57 | |
| rwellum | I commented on a few | 01:57 |
| sdake | rwellum right - the commands are now completely consisten | 01:57 |
| sdake | rwellum everything should work as a non-root user | 01:58 |
| sdake | if they dont they should be fixed :) | 01:58 |
| sdake | rending latest docs and running through guide locally now | 01:58 |
| spsurya | sdake: need to clean the directory `/var/lib/etcd` too in `kubeadm reset` | 01:59 |
| spsurya | got into this [centos@ip-172-31-13-146 kollaK8s]$ sudo kubeadm init --pod-network-cidr=10.1.0.0/16 --service-cidr=10.3.0.0/16 | 02:00 |
| spsurya | [kubeadm] WARNING: kubeadm is in beta, please do not use it for production clusters. | 02:00 |
| spsurya | [init] Using Kubernetes version: v1.6.0 | 02:00 |
| spsurya | [init] Using Authorization mode: RBAC | 02:00 |
| spsurya | [preflight] Running pre-flight checks | 02:00 |
| spsurya | [preflight] Some fatal errors occurred: | 02:00 |
| spsurya | /var/lib/etcd is not empty | 02:00 |
| spsurya | [preflight] If you know what you are doing, you can skip pre-flight checks with `--skip-preflight-checks` | 02:00 |
| *** eaguilar has quit IRC | 02:00 | |
| spsurya | I re run the command after cleaning | 02:00 |
| sdake | spsurya did you clean up with kubeadm reset first? | 02:00 |
| spsurya | then it worked | 02:00 |
| spsurya | sdake: yes | 02:01 |
| sdake | kubeadm init may not be rerun after its run | 02:01 |
| sdake | a reset step is needed first | 02:01 |
| sdake | kubeadm reset | 02:01 |
| spsurya | sdake: that what i did | 02:01 |
| rwellum | sdake: I am a getting this for the last few hours - seen this before? https://www.irccloud.com/pastebin/PRLXWXqy/ | 02:01 |
| sdake | ya - i dont think kubeadm reeset fuly cleans up | 02:01 |
| sdake | spsurya which versiosn of kubeadm do you ahve? | 02:02 |
| spsurya | sdake: 1.6.1 | 02:02 |
| sdake | rwellum gah - no idea - let me try a fresh install | 02:02 |
| eanylin | https://kubernetes.io/docs/getting-started-guides/kubeadm/#tear-down | 02:02 |
| eanylin | spsurya: There are some extra steps | 02:02 |
| eanylin | it seems, for the new version | 02:03 |
| *** unicell has quit IRC | 02:03 | |
| rwellum | sdake: work-around is to set the gpg key to 0 | 02:03 |
| sdake | spsurya ther eis always tools/cleanup-k8s.sh :) | 02:03 |
| spsurya | sdake: i did in between :) i mean before getting kolla-k8s repo i.e before step 2 and step 3 | 02:06 |
| sdake | spsurya there is another step | 02:06 |
| sdake | sudo yum uninstall -y docker ebtables kubeadm kubectl kubelet kubernetes-cni | 02:06 |
| sdake | kolla-k8s.sh needs that sudo yum install | 02:06 |
| *** Pavo has quit IRC | 02:06 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-kubernetes master: Uodate the version of cirros https://review.openstack.org/453432 | 02:07 |
| sdake | cleanup-k8s.sh that is | 02:07 |
| rwellum | sdake: running as 'me' I can confirm that the workaround mentioned does not seem to be working - I cannot connect | 02:08 |
| sdake | rwellum let me run through the directions, I am certian i confirmed it | 02:08 |
| spsurya | sdake: eanylin this is what i tested http://paste.openstack.org/show/605450/ | 02:08 |
| spsurya | and how that worked | 02:09 |
| eanylin | ah, ok, remove everything under /var/lib/etcd/* | 02:09 |
| spsurya | eanylin: yes, I removed that then got working for me | 02:10 |
| eanylin | spsurya: Ok | 02:11 |
| spsurya | that is why i was saying that kubeadm reset should clean `/var/lib/etcd/*` too | 02:11 |
| eanylin | spsurya: Ok, that will probably be cleaner then | 02:13 |
| spsurya | eanylin: yeh | 02:13 |
| eanylin | spsurya: Where you able to get the cluster up since yesterday or? | 02:13 |
| japestinho | thanks so much rwellum and sbezverk for the informations | 02:13 |
| spsurya | eanylin: yesterday i tried with 1.5 one | 02:14 |
| spsurya | and cluster was up | 02:14 |
| eanylin | spsurya: Ok, thats good | 02:15 |
| spsurya | but till now not with 1.6+ | 02:15 |
| eanylin | spsurya: I got busy with work today, didnt really follow the chatroom | 02:15 |
| spsurya | hope i will get that i.e with 1.6+ | 02:15 |
| spsurya | eanylin: i think you got that with 1.6.1 yesterday IIRC | 02:16 |
| spsurya | right ? | 02:16 |
| eanylin | spsurya: I think it should work. With the latest code from sbezverk, I think there is zero restart | 02:16 |
| eanylin | spsurya: Yeah, http://paste.openstack.org/show/605451/ | 02:17 |
| eanylin | so this time round the number of restart is 0 | 02:17 |
| spsurya | eanylin: can you please run once `rpm -qa | grep kub1` and paste the output | 02:18 |
| spsurya | `rpm -qa | grep kube` | 02:18 |
| spsurya | * | 02:19 |
| eanylin | http://paste.openstack.org/show/605452/ | 02:19 |
| sdake | ok need to eat | 02:19 |
| sdake | bbi10 mins | 02:19 |
| spsurya | eanylin: nice | 02:21 |
| sdake | spsurya i have deployed 1.6.1 and deployed openstack on top and created a vm - the vm creation fails | 02:21 |
| eanylin | sdake: I got the VM up this time round | 02:21 |
| spsurya | eanylin: what about openstack | 02:21 |
| spsurya | ? | 02:21 |
| sdake | eanylin what is missing from the docs tog et a vm operational | 02:21 |
| spsurya | eanylin: great | 02:21 |
| sdake | eanylin i'm crious if it is the /16 subnet specified (vs using the defaults) | 02:22 |
| eanylin | spsuray: http://paste.openstack.org/show/605453/ | 02:22 |
| eanylin | sdake: I think I am still having problems though. This time round its with floating IP. | 02:23 |
| eanylin | I am trying different thing, I am using single bridge now | 02:23 |
| eanylin | and its causing some issues | 02:23 |
| sdake | eanylin what were you doing prior to single bridge? | 02:23 |
| eanylin | sdake: I was using 2 bridges | 02:23 |
| eanylin | 1 for OAM/Mgmt | 02:23 |
| eanylin | 1 for Neutron | 02:23 |
| sdake | eanylin you setup a bridge ? | 02:23 |
| eanylin | sdake: Yeah | 02:24 |
| sdake | that wasn't in the paste ;) | 02:24 |
| spsurya | eanylin: please save the history and write the comment as per even little problem, you face | 02:24 |
| eanylin | sdake: That is true | 02:24 |
| spsurya | taht would be helpful | 02:24 |
| sdake | steps 1 and 2 work well now | 02:24 |
| eanylin | spsurya: Yeah, I can do that | 02:24 |
| sdake | step 3 - needs some love | 02:24 |
| eanylin | my issue really is, I cannot SSH or ping my cirros VM | 02:24 |
| sdake | and apparently someone needs to expalin how to fulfill kfox's request | 02:24 |
| eanylin | I still cannot figure out why... | 02:24 |
| spsurya | sdake: seems like we are good to go with merge | 02:25 |
| eanylin | so am trying different things | 02:25 |
| eanylin | sdake: I have something like this on my VM | 02:26 |
| eanylin | http://paste.openstack.org/show/605454/ | 02:26 |
| eanylin | It was a config when I used for 1.6.0 | 02:26 |
| eanylin | br0 is mgmt | 02:26 |
| eanylin | br1 is for neutron | 02:26 |
| eanylin | sdake: This time round I did it with just 1 bridge, probably not a good idea, it messed up the floating IP it seems, maybe my config has problem | 02:27 |
| eanylin | sdake: The gate is doing this, I followed the gate | 02:29 |
| eanylin | kubeadm init --skip-preflight-checks --service-cidr 172.16.128.0/24 | 02:30 |
| eanylin | do we need /16 ? | 02:30 |
| sdake | nope we apparently need /24 | 02:32 |
| sdake | however 172.16.128.0 is I believe a gate specific cidr | 02:32 |
| sdake | I am nto really sure as noboy will tell me :) | 02:32 |
| eanylin | I see | 02:32 |
| eanylin | so seems like --service-cidr (default ‘10.96.0.0/12’) | 02:33 |
| eanylin | they have it as quite big too | 02:33 |
| eanylin | like the default values | 02:33 |
| sdake | i hae no idea what a service cidr does | 02:33 |
| sdake | eanylin check this out: | 02:33 |
| sdake | https://review.openstack.org/#/c/447356/ | 02:34 |
| sdake | kevin added the comment: | 02:34 |
| sdake | Line 103: | 02:35 |
| sdake | I disagree with the assertion it should work on virtually any system. Doesn't work at any of the sites I have k8s deployed. | 02:35 |
| sdake | I'd rather see this changed to an instruction stating how to pick a range that will work for their site if these defaults don't work, and what needs to change in the instructions for fixing it. | 02:35 |
| sdake | no idea what to do with that comment | 02:35 |
| sdake | how does one pick a range? | 02:35 |
| sdake | kfox1111 ^^ | 02:35 |
| sdake | I have asked this over and over, to no avail | 02:36 |
| sdake | seems like a good goal - however- without approprirate domain knoweldge I am unable to solve the problem | 02:36 |
| sdake | I dont even know what a servicie-cidr is | 02:36 |
| sdake | upstream docs are parse | 02:36 |
| eanylin | sdake: Ok. I usually stick to default. But yeah, I agree that there is better approach to just taking default values | 02:36 |
| sdake | eanylin well the review apparently isnt' merging until this is fixed | 02:37 |
| sdake | adn I am leaving friday | 02:37 |
| eanylin | sdake: If its single VM, it seems to be working ok | 02:37 |
| sdake | so - we need to sort this out | 02:37 |
| eanylin | oh man, where to? | 02:37 |
| sdake | upstream servant leadership training | 02:37 |
| eanylin | sdake: What if we use similar range as the gate? I used the exact same one and it worked fine. | 02:37 |
| sdake | ok - you suggest using a range of the gate | 02:37 |
| eanylin | sdake: Hmm, not sure what is that | 02:37 |
| sdake | *why* | 02:37 |
| *** zhurong has quit IRC | 02:38 | |
| sdake | eanylin basically 4 days of training | 02:38 |
| eanylin | sdake: I think as long as no IP conflicts, it should be ok, that seems to be what people are saying in forum | 02:38 |
| sdake | no IP conflicts *where* | 02:38 |
| sdake | what is a serice cidr | 02:38 |
| sdake | lets start by answering that question | 02:38 |
| eanylin | sdake: I see, 4 days training | 02:39 |
| sdake | clearly ips shoudln't have conflicts | 02:39 |
| eanylin | sdake: Right, since we dont have those IPs on our VM | 02:39 |
| eanylin | and we are just doing AIO | 02:39 |
| eanylin | sdake: I probably told you that it works for multi node too, I have compute running on its own and all control on the other VM | 02:39 |
| eanylin | sdake: Using similar settings as Gate | 02:40 |
| sdake | eanylin you ahve not told me that -however that is great news :) | 02:40 |
| eanylin | sdake: I didnt? | 02:40 |
| eanylin | sdake: Ok, my bad | 02:40 |
| sdake | copying the gate without understanding hwy is not useful :) | 02:40 |
| eanylin | sdake: But I probably tested with 1.6.0 | 02:40 |
| sdake | to answering kevin's response | 02:40 |
| sdake | "tell them how to select a range" | 02:40 |
| sdake | how means wehave to have knowledge | 02:40 |
| rwellum | sdake - i think if you go /24 etc then there's some defaults in kubernetes/manifests that need prodding. Because they are /16 | 02:40 |
| sdake | we dont have knoweldge | 02:40 |
| sdake | rwellum ack there | 02:40 |
| eanylin | sdake: That is correct, its good to understand the logic | 02:40 |
| rwellum | sdake: >>> https://www.irccloud.com/pastebin/MtFehwOM/ | 02:40 |
| sdake | rwellum I can sort the /16->/24 | 02:41 |
| sdake | i just dont understnad what a service-cidr is | 02:41 |
| eanylin | rwellum: That is true, which is why the gate made changes | 02:41 |
| eanylin | --service-cidr (default ‘10.96.0.0/12’) | 02:42 |
| eanylin | You can use the --service-cidr flag to override the subnet Kubernetes uses to assign pods IP addresses. If you do, you will also need to update the /etc/systemd/system/kubelet.service.d/10-kubeadm.conf file to reflect this change else DNS will not function correctly. | 02:42 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix ceilometer meters for swift service https://review.openstack.org/452373 | 02:42 |
| eanylin | so there is a default value, in order to not use default, we need to make changes to 10-kubeadm.conf | 02:42 |
| sdake | SERVICE_IP_RANGE=10.3.0.0/24 | 02:42 |
| sdake | The CIDR network to use for service cluster VIPs (Virtual IPs). Each service will be assigned a cluster IP out of this range. This must not overlap with any IP ranges assigned to the POD_NETWORK, or other existing network infrastructure. Routing to these VIPs is handled by a local kube-proxy service to each host, and are not required to be routable between hosts. | 02:42 |
| sdake | eanylin read the review - it has changed drastically | 02:43 |
| eanylin | sdake: Ok. I havent read the latest. | 02:43 |
| sdake | eanylin - oh well that epalins it :) | 02:43 |
| sdake | eanylin its dramatically improved | 02:43 |
| sdake | eanylin based upon your paste last night i went thorugh and actually figured out how to deploy the whole hting | 02:44 |
| sdake | ;) | 02:44 |
| sdake | eanylin++ for the paste | 02:44 |
| rwellum | sdake: probably not a good time, but I am semi-convinced when I add the kubeadm credentials workaround, that's when my nslookup kubernetes fails.... | 02:44 |
| eanylin | sdake: Thats cool :) | 02:44 |
| rwellum | Tried 3x with and without | 02:44 |
| eanylin | rwellum: Wasnt sure if you saw my session logs from yesterday night | 02:46 |
| sdake | rwellum which credentials workaround | 02:46 |
| eanylin | should have worked ok | 02:46 |
| sdake | the 8080 port change? | 02:46 |
| rwellum | sdake: yes | 02:46 |
| sdake | right - so kubernetes fails nslooup with --insecure-port=8080 | 02:46 |
| rwellum | eanylin: that's what sdake has in his review | 02:46 |
| sdake | kubeadm spits out a list of operations (3 of them) to do to get the creds up | 02:46 |
| eanylin | the 8080 port change is handled by | 02:46 |
| rwellum | Doesn't work fo rme and seems to break nslookup | 02:46 |
| eanylin | mkdir -p ~/.kube | 02:46 |
| eanylin | sudo cp /etc/kubernetes/admin.conf ~/.kube/config | 02:46 |
| eanylin | sudo chown $(id -u):$(id -g) ~/.kube/config | 02:46 |
| eanylin | if you do these 3 steps after kubeadm init | 02:47 |
| eanylin | it will work without sed | 02:47 |
| eanylin | I just tested | 02:47 |
| eanylin | rwellum: Ok | 02:47 |
| sdake | rwellum what si your underlying virt provider | 02:47 |
| rwellum | sdake: ambiguous qn or I'm really tired sorry | 02:48 |
| sdake | rwellum libvirt, virtualbox, etc? | 02:48 |
| rwellum | libvirt running in centos 7.x vm | 02:49 |
| eanylin | actually thats a valid question, I been testing on KVM/libvirt | 02:49 |
| rwellum | sdake: ./get_helm.sh - that requires sudo priv to run. | 02:49 |
| sdake | rwellum your running libvirt within the vm? | 02:49 |
| rwellum | sdake: yeah - host-passthrough, nested etc | 02:49 |
| eanylin | rwellum: Why do we need libvirt on that VM? It will cause problems later when you try and install the libvirt container | 02:51 |
| sdake | agreed we are missing a step disable libvirt | 02:52 |
| sdake | thats a big problem | 02:52 |
| eanylin | sdake: Yah, but ideally I think we should build a CentOS from minimal ISO | 02:52 |
| eanylin | it will not have all these stuff | 02:52 |
| eanylin | and its very clean | 02:52 |
| rwellum | I wonder if this is the root of many of my issues bringing up openstack - even once I have a stable k8s | 02:53 |
| spsurya | yes agree with eanylin we should have clean one at the first place | 02:53 |
| rwellum | So eanylin - i should not do nested virt either? | 02:53 |
| eanylin | rwellum: I have a base CentOS that is build from minimal ISO, so it doesnt have libvirt, bridge-utils etc | 02:54 |
| eanylin | so you install and build on top of it whenever you need new stuff | 02:54 |
| eanylin | I think that will help to eliminate conflicts that can happen | 02:54 |
| eanylin | I once had OVS on my VM | 02:54 |
| eanylin | and it messed up the OVS in kolla-ansible | 02:54 |
| rwellum | Right but usually when you build a vm you have to decide if you do a host-passthrough operation | 02:54 |
| eanylin | rwellum: Ok | 02:55 |
| eanylin | I usually use virt-inst | 02:55 |
| eanylin | or virt-manager | 02:56 |
| rwellum | Same - but in the CPU options - you can select host-passthrough - otherwise if you use you VM to create more VM's they won't work. So just curious if this is what you do or not. I agree about the minimum ISO part. | 02:57 |
| eanylin | Am looking at the option I selected | 02:58 |
| eanylin | I did Hypervisor Default | 02:58 |
| eanylin | and use qemu | 02:58 |
| eanylin | when I create Cirros VM | 02:58 |
| rwellum | Ok wow - :) | 02:59 |
| eanylin | yeah | 02:59 |
| rwellum | sdake: I'l try tomorrow from a minimum ISO VM - sorry if this slows you down right now. | 02:59 |
| eanylin | rwellum: I have 2 bridges in my cases, virsh net | 03:00 |
| eanylin | 1 for mgmt, 1 for neutron | 03:00 |
| eanylin | that worked pretty well in most cases | 03:00 |
| eanylin | I still have things left to sort out but I think it should at least bring you further | 03:01 |
| rwellum | eanylin: I guess if you get a chance, virsh dumpxml <domain> - and paste it in irc. If networking fails at that level would be good to compare. | 03:03 |
| eanylin | rwellum: Sure | 03:03 |
| spsurya | eanylin: see this http://paste.openstack.org/show/605456/ | 03:05 |
| eanylin | http://paste.openstack.org/show/605457/ | 03:05 |
| eanylin | rwellum: My host is Ubuntu, VM is CentOS btw | 03:05 |
| eanylin | spsurya: Interesting that you all have problems with nslookup | 03:06 |
| spsurya | eanylin: yeah | 03:06 |
| spsurya | eanylin: but here i am not with livbvirt kind of thing ;) | 03:07 |
| eanylin | spsurya: Hmm, you think you followed all the steps so far in my session logs? | 03:07 |
| eanylin | spsurya: Maybe its easier to follow sesison logs than looking at guide | 03:08 |
| eanylin | spsurya: Thats true :) | 03:08 |
| rwellum | ty eanylin that'll help a lot I think - but it's 11pm here so will try again tomorrow. | 03:08 |
| eanylin | rwellum: Sure, its late in EST | 03:08 |
| spsurya | eanylin: yeah, that is very necessary just to avoid repeated query ;) | 03:10 |
| eanylin | spsurya: I forgot that link actually, haha :D | 03:10 |
| sdake | rwellum night dude :) | 03:11 |
| spsurya | eanylin: np | 03:11 |
| sdake | we got 3 days left to get this sorted out | 03:11 |
| sdake | as I am going out of town | 03:11 |
| sdake | rwellum lets sync up in the morning when iwake up and webex? | 03:12 |
| spsurya | sdake: roger | 03:12 |
| sdake | and i want this merged | 03:12 |
| sdake | then other peole can incrementally imrpove it | 03:12 |
| spsurya | rwellum: night | 03:12 |
| *** zhurong has joined #openstack-kolla | 03:13 | |
| spsurya | sdake: +1 | 03:13 |
| eanylin | spsurya: I scrolled up and looked at history, http://paste.openstack.org/show/605332/ | 03:13 |
| spsurya | for merege | 03:13 |
| eanylin | spsurya: Give that a try? | 03:13 |
| *** MarginHu has joined #openstack-kolla | 03:14 | |
| eanylin | sdake: Did I tell you I was able to create a LVM Cinder Volume too? | 03:14 |
| sdake | eanylin you did but you didnt 'expad as to how | 03:14 |
| eanylin | sdake: Ok, so you need to run a script that is used in the gate to create your vg for cinder-volumes | 03:15 |
| eanylin | kolla-kubernetes/tests/bin/setup_gate_loopback_lvm.sh | 03:15 |
| rwellum | sdake: will catch you in the morning. | 03:16 |
| eanylin | this is similar to workaround we did in kolla-ansible I think | 03:16 |
| rwellum | eanylin: did you create that vm with virt-inst? And if so - can you share the command line please? | 03:16 |
| eanylin | rwellum: I might have done it from virt-manager a while back, I saved a based image | 03:17 |
| eanylin | so that I can keep re-using it | 03:17 |
| rwellum | eanylin: ok fair enough. | 03:17 |
| rwellum | Same :) | 03:17 |
| rwellum | g'night | 03:17 |
| eanylin | mainly because I was using ISO to install | 03:18 |
| eanylin | I want to see the GUI | 03:18 |
| eanylin | rwellum: Good night | 03:19 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 03:22 |
| spsurya | eanylin: did you see the latest deploy guide patch | 03:23 |
| spsurya | ? | 03:23 |
| eanylin | spsurya: Maybe not fully yet, I think there has been quite some changes throughout the day | 03:24 |
| spsurya | --insecure-port updation part is missing i guess | 03:24 |
| spsurya | as i understand | 03:24 |
| spsurya | eanylin: can you please check ? | 03:26 |
| japestinho | sdake rwellum finally I can do cherry-pick with thi command | 03:26 |
| japestinho | git fetch git://git.openstack.org/openstack/kolla-kubernetes refs/changes/40/439740/21 && git cherry-pick FETCH_HEAD | 03:26 |
| japestinho | sorry I can't comment on review due to bad bot request from ubuntu one login | 03:26 |
| eanylin | spsurya: Actually that one is not needed if we do | 03:26 |
| eanylin | mkdir -p ~/.kube | 03:27 |
| eanylin | sudo cp /etc/kubernetes/admin.conf ~/.kube/config | 03:27 |
| eanylin | sudo chown $(id -u):$(id -g) ~/.kube/config | 03:27 |
| *** bmace has quit IRC | 03:27 | |
| eanylin | the sed workaround is not the ideal one, as I understand | 03:27 |
| spsurya | eanylin: that is the problem, i changed and that worked for me | 03:28 |
| spsurya | ;) | 03:28 |
| eanylin | oh | 03:28 |
| eanylin | nice :) | 03:28 |
| eanylin | yeah, quite some changes today, to the guide | 03:28 |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-kubernetes master: Update the version of cirros https://review.openstack.org/453432 | 03:29 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 03:34 |
| *** gkadam has joined #openstack-kolla | 03:45 | |
| *** masber has joined #openstack-kolla | 03:56 | |
| *** iceyao has quit IRC | 04:04 | |
| *** dave-mccowan has quit IRC | 04:05 | |
| *** zhurong has quit IRC | 04:12 | |
| *** iceyao has joined #openstack-kolla | 04:24 | |
| *** iceyao has quit IRC | 04:28 | |
| *** unicell has joined #openstack-kolla | 04:32 | |
| *** zhurong has joined #openstack-kolla | 04:35 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Use versionless keystone endpoints https://review.openstack.org/439024 | 04:51 |
| *** iceyao has joined #openstack-kolla | 04:51 | |
| *** lamt has joined #openstack-kolla | 05:03 | |
| *** jaosorior_away is now known as jaosorior | 05:11 | |
| *** skramaja has joined #openstack-kolla | 05:14 | |
| *** jascott1_ has quit IRC | 05:16 | |
| japestinho | I got this when running helm install | 05:16 |
| japestinho | https://www.irccloud.com/pastebin/g8XPLfmh/ | 05:16 |
| *** jascott1_ has joined #openstack-kolla | 05:17 | |
| japestinho | sdake what did I miss the step? | 05:18 |
| *** jascott1_ has quit IRC | 05:21 | |
| *** tonyb_ is now known as tonyb | 05:25 | |
| *** targon has joined #openstack-kolla | 05:36 | |
| *** lamt has quit IRC | 05:40 | |
| *** manheim has joined #openstack-kolla | 05:40 | |
| *** manheim has quit IRC | 05:45 | |
| *** rstarmer has quit IRC | 05:52 | |
| openstackgerrit | Martin André proposed openstack/kolla master: Add missing cleanup step for new apache images https://review.openstack.org/453470 | 05:59 |
| openstackgerrit | Martin André proposed openstack/kolla master: Add missing cleanup step for heat images https://review.openstack.org/453471 | 05:59 |
| *** yingjun has quit IRC | 06:06 | |
| sdake | spiette whic hone | 06:07 |
| *** pcaruana has joined #openstack-kolla | 06:08 | |
| sdake | eanylin whih sed workaround | 06:08 |
| sdake | the susdo cp /etc/kubernetesadmin.conf is preferrable | 06:09 |
| sdake | japestinho you didn't follow the guide - hence i think that is the problem | 06:09 |
| sdake | japestinho the helm charts are not being acessed at the correct place | 06:10 |
| openstackgerrit | Martin André proposed openstack/kolla master: Add missing cleanup step for heat images https://review.openstack.org/453471 | 06:11 |
| openstackgerrit | Martin André proposed openstack/kolla master: Add missing cleanup step for new apache images https://review.openstack.org/453470 | 06:11 |
| sdake | beoodnight folks | 06:15 |
| sdake | i may be up for out 1 or hoar | 06:15 |
| sdake | hour | 06:15 |
| duonghq | mandre, are you there? | 06:16 |
| mandre | hi duonghq | 06:17 |
| duonghq | about the bp: https://blueprints.launchpad.net/kolla/+spec/apache-packages-for-apis | 06:17 |
| duonghq | I don't understand why we only add apache package to the images? | 06:17 |
| *** caoyuan has joined #openstack-kolla | 06:18 | |
| mandre | that's because we leave it to the operator to chose whether they want to run apache or not for these services | 06:19 |
| mandre | we don't make it the default | 06:19 |
| duonghq | mandre, this is done in service config files? | 06:20 |
| mandre | duonghq: exactly | 06:21 |
| duonghq | mandre, thanks | 06:21 |
| mandre | if you want to run over apache you'll provide config file for apache and run the apache service | 06:21 |
| duonghq | understood | 06:22 |
| mandre | but we need the binary in the image of course, that's the scope of this bp | 06:22 |
| duonghq | it would be better if we can turn on/off the bundling ... | 06:23 |
| mandre | you mean provide an image with apache and an image without? | 06:23 |
| duonghq | I mean we can enable/disable the apache in image by build options | 06:24 |
| *** jrobinson has quit IRC | 06:25 | |
| mandre | hmm, I guess we could, but it would have to be a per-image option | 06:27 |
| mandre | I'm not sure it's worth the effoer | 06:27 |
| *** caowei has joined #openstack-kolla | 06:27 | |
| duonghq | yup, guess that it doesn't | 06:28 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Telegraf: activate influxdb input plugin https://review.openstack.org/453478 | 06:35 |
| masber | hi, I am getting the following error when I deploy Ironic using Kolla http://paste.openstack.org/show/605464/ | 06:43 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix ceph role set target_max_bytes for cach pools https://review.openstack.org/452444 | 06:47 |
| *** athomas has joined #openstack-kolla | 07:02 | |
| *** iniazi has joined #openstack-kolla | 07:03 | |
| *** mgoddard has joined #openstack-kolla | 07:04 | |
| *** Jezekus has joined #openstack-kolla | 07:07 | |
| *** dasTor_ has joined #openstack-kolla | 07:10 | |
| *** dasTor has quit IRC | 07:11 | |
| *** caoyuan has quit IRC | 07:16 | |
| *** matrohon has joined #openstack-kolla | 07:16 | |
| *** shardy has joined #openstack-kolla | 07:22 | |
| *** satyar has joined #openstack-kolla | 07:23 | |
| *** asmaa has joined #openstack-kolla | 07:29 | |
| *** jascott1_ has joined #openstack-kolla | 07:34 | |
| *** Serlex has joined #openstack-kolla | 07:35 | |
| *** nathharp has joined #openstack-kolla | 07:36 | |
| *** britthouser8 has joined #openstack-kolla | 07:39 | |
| *** jascott1_ has quit IRC | 07:39 | |
| *** britthouser has quit IRC | 07:42 | |
| *** gkadam is now known as gkadam-afk | 07:43 | |
| *** jmccarthy has joined #openstack-kolla | 07:46 | |
| *** matrohon has quit IRC | 07:47 | |
| *** egonzalez has joined #openstack-kolla | 07:48 | |
| *** manheim has joined #openstack-kolla | 07:49 | |
| *** yingjun has joined #openstack-kolla | 07:51 | |
| *** bjolo has joined #openstack-kolla | 07:52 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix heat ec2 keystone auth https://review.openstack.org/444953 | 07:52 |
| *** mgoddard has quit IRC | 07:58 | |
| *** jascott1 has quit IRC | 08:01 | |
| *** iceyao has quit IRC | 08:12 | |
| *** magicboiz has quit IRC | 08:13 | |
| *** magicboiz has joined #openstack-kolla | 08:13 | |
| *** iceyao has joined #openstack-kolla | 08:13 | |
| *** magicboiz has quit IRC | 08:16 | |
| *** magicboiz has joined #openstack-kolla | 08:16 | |
| *** magicboiz has quit IRC | 08:17 | |
| *** magicboiz has joined #openstack-kolla | 08:17 | |
| *** magicboiz has quit IRC | 08:20 | |
| hrw | sdake: gigabyte has some with cavium cpu and bmc. up to 512-1024GB ram | 08:20 |
| *** magicboiz has joined #openstack-kolla | 08:20 | |
| hrw | sdake: no idea about price | 08:20 |
| hrw | morning all | 08:20 |
| *** jaosorior is now known as jaosorior_lunch | 08:31 | |
| *** mgoddard has joined #openstack-kolla | 08:32 | |
| *** bjolo has quit IRC | 08:51 | |
| *** bjolo has joined #openstack-kolla | 08:51 | |
| bjolo | morning | 08:51 |
| *** matrohon has joined #openstack-kolla | 08:51 | |
| japestinho | morning all | 08:53 |
| japestinho | if I want add some more controller / compute nodes on existing kolla-ansible deployment which command should I use? | 08:55 |
| japestinho | kolla-ansible upgrade -i multinode or just repeat kolla-ansible deploy -i multinode again? | 08:55 |
| manheim | hello guys | 08:55 |
| *** matrohon has quit IRC | 08:57 | |
| *** gfidente has joined #openstack-kolla | 08:59 | |
| *** gfidente has quit IRC | 08:59 | |
| *** gfidente has joined #openstack-kolla | 08:59 | |
| *** bjolo has quit IRC | 09:05 | |
| egonzalez | japestinho, use kolla-ansible deploy, upgrades will also do a db sync and replace older containers if a new image is found in the registry | 09:06 |
| *** sambetts|afk is now known as sambetts | 09:09 | |
| *** blallau has joined #openstack-kolla | 09:12 | |
| japestinho | egonzalez if it same command for remove some nodes too? using this kolla-ansible deploy? | 09:26 |
| egonzalez | japestinho, deploy only add and configure services | 09:26 |
| *** iceyao has quit IRC | 09:28 | |
| japestinho | egonzalez did you mean kolla-ansible reconfigure (some --tags) for 'configure services'? | 09:35 |
| egonzalez | japestinho, deploy and reconfigure in ocata are mostly the same in most services | 09:35 |
| egonzalez | japestinho, if deploying API services i would not use --tags because some service have dependencies in other(add the node in HAproxy as example) | 09:36 |
| egonzalez | japestinho, deploy is idempotent and should not restart nothing unless changed | 09:37 |
| *** MarginHu has quit IRC | 09:37 | |
| *** MarginHu has joined #openstack-kolla | 09:37 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Add graceful_timeout argument to kolla_docker https://review.openstack.org/453055 | 09:37 |
| *** MarginHu has quit IRC | 09:38 | |
| *** MarginHu has joined #openstack-kolla | 09:38 | |
| *** MarginHu has quit IRC | 09:39 | |
| *** iceyao has joined #openstack-kolla | 09:43 | |
| japestinho | egonzalez okay thank you I'll try it now. If I want to add 2 more controller nodes with HA mode, what parameters must be enabled in globals.yml beside enable_haproxy, enable_neutron_lbaas, enable_neutron_agent_ha? | 09:49 |
| egonzalez | japestinho, if the previous deployment already had those settings, just add the new nodes in the inventory and do kolla-ansible deploy | 09:51 |
| egonzalez | japestinho, if not, enabling those setting may require a huge change in a lot of components, including compute nodes | 09:52 |
| japestinho | egonzalez, so better I enable those parameters from the start if in future time I'll add some more controller nodes? thanks for the advice | 09:55 |
| *** duonghq has quit IRC | 09:57 | |
| *** bjolo has joined #openstack-kolla | 09:58 | |
| *** tovin07_ has quit IRC | 10:04 | |
| *** satyar has quit IRC | 10:13 | |
| Deys1 | anyone has link to the document which states docker tags that is related to the openstack releases, i can't seem to remember where it was in the docs | 10:14 |
| *** asmaa has quit IRC | 10:17 | |
| *** Jezekus has quit IRC | 10:18 | |
| *** asmaa has joined #openstack-kolla | 10:19 | |
| *** pcaruana|afk| has joined #openstack-kolla | 10:21 | |
| *** xiaoruiguo has quit IRC | 10:22 | |
| *** pcaruana|afk| has quit IRC | 10:23 | |
| *** pcaruana has quit IRC | 10:24 | |
| *** pcaruana has joined #openstack-kolla | 10:25 | |
| *** cuongnv has quit IRC | 10:25 | |
| *** zhurong has quit IRC | 10:30 | |
| *** jaosorior_lunch is now known as jaosorior | 10:31 | |
| *** yingjun has quit IRC | 10:34 | |
| *** pbourke has quit IRC | 10:39 | |
| *** pbourke has joined #openstack-kolla | 10:41 | |
| *** athomas has quit IRC | 10:41 | |
| *** haplo37 has quit IRC | 10:43 | |
| *** shardy is now known as shardy_lunch | 10:45 | |
| *** rmart04 has joined #openstack-kolla | 10:50 | |
| *** athomas has joined #openstack-kolla | 10:51 | |
| egonzalez | someone have any guess why is python35 tests failing randomly on test_build_image? | 10:52 |
| *** haplo37 has joined #openstack-kolla | 10:52 | |
| *** dave-mccowan has joined #openstack-kolla | 11:07 | |
| *** caowei has quit IRC | 11:11 | |
| *** iniazi_ has joined #openstack-kolla | 11:11 | |
| *** iniazi has quit IRC | 11:12 | |
| *** dasTor_ has quit IRC | 11:15 | |
| *** dasTor has joined #openstack-kolla | 11:15 | |
| spsurya | egonzalez: looked into but not getting much idea | 11:18 |
| spsurya | :( | 11:19 |
| *** yingjun has joined #openstack-kolla | 11:19 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix qos rules are not applied on ocata with ovs https://review.openstack.org/452392 | 11:20 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix the "tox -e docs" failed https://review.openstack.org/452602 | 11:20 |
| openstackgerrit | Merged openstack/kolla-ansible master: Telegraf: activate influxdb input plugin https://review.openstack.org/453478 | 11:21 |
| *** cristicalin has joined #openstack-kolla | 11:22 | |
| *** ccesario has joined #openstack-kolla | 11:23 | |
| *** dixiaoli has joined #openstack-kolla | 11:27 | |
| *** dixiaoli has quit IRC | 11:27 | |
| *** dixiaoli has joined #openstack-kolla | 11:27 | |
| *** dixiaoli has quit IRC | 11:28 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix Telegraf retention policy not found https://review.openstack.org/453058 | 11:32 |
| *** rwallner has joined #openstack-kolla | 11:35 | |
| *** rwallner has quit IRC | 11:35 | |
| *** rwallner has joined #openstack-kolla | 11:35 | |
| *** rwallner has quit IRC | 11:35 | |
| *** rwallner has joined #openstack-kolla | 11:36 | |
| pbourke | egonzalez: I think those tests need some refactoring, there is global variables that are being manipulated and not properly reset | 11:41 |
| pbourke | - after each test | 11:41 |
| pbourke | masber: you need to provide those files, please check the ironic documentation | 11:42 |
| *** targon has quit IRC | 11:43 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove min_l3_agents_per_router option https://review.openstack.org/451159 | 11:52 |
| *** cristicalin has quit IRC | 11:52 | |
| egonzalez | sayantani01, around? | 11:57 |
| manheim | from Dockerfiles, can I check against vars in global.yml? beased on if a specific service is enabled, I want to install a package in a dockerfile | 12:02 |
| egonzalez | manheim, nope, kolla and kolla-ansible does not share globals, each one is independent of each other | 12:03 |
| manheim | i'm on newton branch | 12:03 |
| manheim | can it be done for newton? | 12:04 |
| egonzalez | manheim, if you are using a local build for your deployment, use package customization https://github.com/openstack/kolla/blob/master/doc/image-building.rst#dockerfile-customisation | 12:04 |
| manheim | thanks | 12:07 |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove Ubuntu 14.04 related code https://review.openstack.org/448792 | 12:26 |
| *** Pavo has joined #openstack-kolla | 12:26 | |
| *** yingjun has quit IRC | 12:31 | |
| *** skramaja has quit IRC | 12:35 | |
| openstackgerrit | Merged openstack/kolla master: Fix oslo_debug_helper not working https://review.openstack.org/444398 | 12:38 |
| *** lamt has joined #openstack-kolla | 12:41 | |
| *** matrohon has joined #openstack-kolla | 12:48 | |
| *** gkadam-afk has quit IRC | 12:49 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add release note for versionless keystone endpoint https://review.openstack.org/453620 | 12:49 |
| *** rwallner has quit IRC | 12:50 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add release note for versionless keystone endpoint https://review.openstack.org/453620 | 12:50 |
| *** rwallner has joined #openstack-kolla | 12:50 | |
| -openstackstatus- NOTICE: The Gerrit service on http://review.openstack.org is being restarted to address hung remote replication tasks, and should return to an operable state momentarily | 12:51 | |
| rwellum | sdake: when you get in I have the minimum Centos VM running, my username, following the guide and I cannot get past the 8080 connect issue using the current eanylin method. | 12:53 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add release note for versionless keystone endpoint https://review.openstack.org/453620 | 12:53 |
| *** jtriley has quit IRC | 12:54 | |
| *** yingjun has joined #openstack-kolla | 12:56 | |
| *** iceyao has quit IRC | 13:03 | |
| *** goldyfruit has joined #openstack-kolla | 13:06 | |
| *** shardy_lunch is now known as shardy | 13:08 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 13:09 |
| *** eanylin has quit IRC | 13:15 | |
| *** shashank_t_ has joined #openstack-kolla | 13:17 | |
| *** shashank_t_ has quit IRC | 13:22 | |
| *** mbruzek has joined #openstack-kolla | 13:22 | |
| *** jtriley has joined #openstack-kolla | 13:24 | |
| *** lrensing has joined #openstack-kolla | 13:25 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 13:25 |
| *** mkoderer has joined #openstack-kolla | 13:28 | |
| *** eanylin has joined #openstack-kolla | 13:30 | |
| *** ipsecguy_ is now known as ipsecguy | 13:31 | |
| *** shashank_t_ has joined #openstack-kolla | 13:33 | |
| *** rmart04 has quit IRC | 13:33 | |
| *** rmart04 has joined #openstack-kolla | 13:35 | |
| eanylin | rwellum: Morning, what was the issue you are facing? | 13:36 |
| rwellum | eanylin: this doesn't work for me https://www.irccloud.com/pastebin/sz8e4s2p/ | 13:37 |
| eanylin | ok | 13:38 |
| eanylin | rwellum: I did this | 13:39 |
| eanylin | mkdir -p ~/.kube | 13:39 |
| eanylin | sudo cp /etc/kubernetes/admin.conf ~/.kube/config | 13:39 |
| eanylin | sudo chown $(id -u):$(id -g) ~/.kube/config | 13:39 |
| eanylin | try this instead | 13:39 |
| *** papacz has quit IRC | 13:39 | |
| rwellum | Can you add this as a comment to the doc review please? | 13:39 |
| rwellum | I'll try next time around. | 13:39 |
| eanylin | rwellum: Ok, I will add it to comment | 13:40 |
| rwellum | ty | 13:40 |
| eanylin | np | 13:40 |
| *** eaguilar has joined #openstack-kolla | 13:43 | |
| eanylin | rwellum: Added as comments | 13:43 |
| rwellum | ty eanylin | 13:44 |
| *** rmart04 has quit IRC | 13:44 | |
| *** lamt has quit IRC | 13:45 | |
| eanylin | rwellum: Welcome | 13:45 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 13:53 |
| *** rmart04 has joined #openstack-kolla | 13:55 | |
| *** rmart04 has quit IRC | 13:57 | |
| *** ksumit has joined #openstack-kolla | 13:59 | |
| *** manheim has quit IRC | 14:00 | |
| *** rmart04 has joined #openstack-kolla | 14:01 | |
| *** iceyao has joined #openstack-kolla | 14:03 | |
| *** rwallner has quit IRC | 14:05 | |
| *** rmart04 has quit IRC | 14:06 | |
| *** jistr is now known as jistr|mtg | 14:07 | |
| *** shardy is now known as shardy_mtg | 14:07 | |
| *** iceyao has quit IRC | 14:08 | |
| *** jascott1 has joined #openstack-kolla | 14:10 | |
| *** bjolo has quit IRC | 14:14 | |
| *** bjolo has joined #openstack-kolla | 14:14 | |
| *** jascott1 has quit IRC | 14:15 | |
| *** lrensing has quit IRC | 14:18 | |
| *** rwallner has joined #openstack-kolla | 14:23 | |
| *** satyar has joined #openstack-kolla | 14:24 | |
| *** manheim has joined #openstack-kolla | 14:26 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Remove all Ubuntu 14.04 related documentation https://review.openstack.org/453668 | 14:28 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 14:28 |
| *** ipsecguy_ has joined #openstack-kolla | 14:29 | |
| *** manheim has quit IRC | 14:30 | |
| *** ipsecguy has quit IRC | 14:31 | |
| *** lrensing has joined #openstack-kolla | 14:32 | |
| sdake | hrw so morning | 14:33 |
| sdake | morning | 14:33 |
| sdake | hrw the gigabye model - is that a mainboard or do they make a full box? | 14:33 |
| sdake | in the past gigabyte only made motherboards - but admittdly its been a long time since i've looked | 14:34 |
| hrw | sdake: server. half width | 14:34 |
| hrw | sdake: http://b2b.gigabyte.com/Rack-Server/ARM-SoC - ignore X-Gene ones | 14:35 |
| *** iceyao has joined #openstack-kolla | 14:37 | |
| sdake | hrw not knowing anything about arm cpus | 14:39 |
| sdake | which one woudl i want to do real dev work with | 14:39 |
| hrw | sdake: ThunderX is the only cpu on that list of boards | 14:40 |
| sdake | this one looks pretty good minus the qsfp ports: http://b2b.gigabyte.com/Rack-Server/R150-T60-rev-110# | 14:40 |
| sdake | hrw just woke up | 14:40 |
| hrw | sdake: you just select 1cpu or 2cpu, density etc | 14:40 |
| sdake | hrw sorry if I ask dubm qs :) | 14:40 |
| hrw | sdake: there are no such thing as 'dumb question' | 14:40 |
| sdake | i dont see a pricetag | 14:41 |
| sdake | hrw questions i could naswer myself | 14:41 |
| sdake | are dumb questions :) | 14:41 |
| sdake | but toherwise i agree with your statement :) | 14:41 |
| hrw | sdake: you have to find reseller as they are non consumer boards | 14:41 |
| sdake | hrw ok i'll check around thanks | 14:41 |
| sdake | you said 600 bucks in your video | 14:41 |
| sdake | is that bottom of the barrel | 14:41 |
| hrw | sdake: ThunderX cpu has huge amount of ethernet speed. 1GbE is only port for BMC | 14:42 |
| sdake | what pricetag would I be looking at for a dual core with 64gb ram (or enough slots for 64gb of ram) | 14:42 |
| hrw | sdake: 600$ is SoftIron 1000. no BMC | 14:42 |
| sdake | ok - well ened a bmc | 14:42 |
| sdake | so thats not gonna work :) | 14:42 |
| hrw | sdake: thunderx allow to have 512GB per cpu | 14:42 |
| sdake | cool | 14:43 |
| sdake | i know everything about intel designs | 14:43 |
| sdake | in terms of what to get | 14:43 |
| sdake | are there "gotchas" | 14:43 |
| sdake | (this I can't find out on my own) | 14:43 |
| *** goldyfruit has quit IRC | 14:43 | |
| sdake | like 5409 processor doesn't support virt for example | 14:43 |
| hrw | sdake: no idea - never used thunderx systems | 14:43 |
| hrw | sdake: on aarch64 virtualization is mandatory ;d | 14:43 |
| sdake | is tunderx a next gen processor? | 14:44 |
| hrw | sdake: especially on aarch64 servers | 14:44 |
| sdake | what about io virt? | 14:44 |
| hrw | define 'next gen' | 14:44 |
| sdake | latest generation cpu | 14:44 |
| sdake | you said you dont use them | 14:44 |
| sdake | curious why? | 14:44 |
| sdake | my sepculatino was you just didn't have them yet | 14:44 |
| hrw | sdake: $$$ to get those to home | 14:44 |
| sdake | right - speculation was correct :) | 14:44 |
| hrw | sdake: there are some thunderx systems at RH and at Linaro - I just do not need to have access to them | 14:45 |
| sdake | hrw but will they work? | 14:45 |
| hrw | yes | 14:45 |
| sdake | or should I choose a different processor type | 14:45 |
| sdake | my end game is to be able to pxeboot via ironic arm server | 14:45 |
| hrw | sdake: there is no big choice when it comes to mass market aarch64 servers | 14:45 |
| sdake | rather to an arm server | 14:46 |
| sdake | i want to pxeboot an arm server using ironic | 14:46 |
| sdake | there - should be more clear :) | 14:46 |
| hrw | sdake: it has standard ipmi | 14:47 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 14:47 |
| sdake | are all the nics LOM? | 14:48 |
| sdake | its annoying they dont have rj45 copper lan ports | 14:49 |
| sdake | qfp is super expensive | 14:49 |
| sdake | i recognize qfp goes 25k meters | 14:49 |
| sdake | IDC, I need something for 3 meters :) | 14:49 |
| *** Manheim has joined #openstack-kolla | 14:53 | |
| hrw | sdake: LOM? | 14:54 |
| sdake | cool quad channel emory design | 14:54 |
| sdake | lan on motherboard | 14:54 |
| hrw | yes they are | 14:54 |
| hrw | sdake: it is ARM SoC. everything is in SoC. | 14:54 |
| sdake | interesting | 14:55 |
| hrw | you have SoC with cpu, storage controllers, network controllers etc. | 14:55 |
| sdake | https://www.servethehome.com/gigabyte-r120-t30-overview-first-cavium-thunderx-system/ | 14:55 |
| sdake | 48 cores out of the box | 14:55 |
| hrw | outside chips are mostly PHYs and BMC | 14:55 |
| sdake | can't finda resller | 14:55 |
| sdake | looks like a beefy cpu | 14:55 |
| sdake | very copetitive to intc's xeon line | 14:56 |
| hrw | sdake: https://www.scan.co.uk/products/gigabyte-r120-p30-single-socket-1u-rackmount-armv8-24ghz-pcie-30-8x-ddr3-slots-ecc-unbuffered-2x-10g | 14:56 |
| sdake | notso sure about the server market in general competitiveness :) | 14:56 |
| sdake | 1300 elbows | 14:56 |
| sdake | thtas not terrible | 14:56 |
| sdake | i guess i'll tell my wife since she requested the purchase | 14:56 |
| sdake | need to find a us distributor unfortunately | 14:56 |
| sdake | but she can do that | 14:56 |
| sdake | if she really wants one | 14:57 |
| *** eaguilar has quit IRC | 15:03 | |
| sdake | rwellum cool so centos starts up with tehe guide? | 15:03 |
| rwellum | sdake: the 8080 work-around is not correct, I asked eanylin to add a review comment and he did. I am adding charts as we speak. | 15:06 |
| rwellum | sdake: https://www.irccloud.com/pastebin/WKjpI80a/ | 15:06 |
| rwellum | So far much better. | 15:07 |
| sdake | ok - so the kubeadm instructions are wrong | 15:09 |
| sdake | its not so much a workaround as the design of kubeadm+kubectl | 15:09 |
| sdake | sound slike kubectl is busted | 15:09 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: gnocchi-base: add some devel packages for non-x86 https://review.openstack.org/453700 | 15:09 |
| hrw | uf. CI do not complain on any of my patches | 15:14 |
| *** iniazi_ has quit IRC | 15:16 | |
| *** iniazi_ has joined #openstack-kolla | 15:16 | |
| inc0 | good morning | 15:17 |
| hrw | hi inc0 | 15:20 |
| *** lamt has joined #openstack-kolla | 15:21 | |
| *** shardy_mtg is now known as shardy | 15:21 | |
| *** Serlex has quit IRC | 15:22 | |
| kfox1111 | morning. | 15:30 |
| *** goldyfruit has joined #openstack-kolla | 15:30 | |
| sbezverk | kfox1111: morning.. I had interesting observation | 15:31 |
| *** iceyao has quit IRC | 15:31 | |
| rwellum | sdake: speechless! https://www.irccloud.com/pastebin/DfqGIT39/ | 15:31 |
| kfox1111 | sbezverk: yeah? | 15:32 |
| sbezverk | kfox1111: when we start canal setup, none of control plane pods were actually in running state, but we still pushed canal setup. kube eventually sorts this out, but lots of racing happens | 15:32 |
| kfox1111 | rwellum: nice. :) | 15:32 |
| rwellum | kfox1111: it's a first for me :) | 15:33 |
| kfox1111 | sbezverk: hmm.. so long as kube-apiserver is running, it shouldn't matter how long it takes to get the scheduler/etc up. | 15:33 |
| sbezverk | kfox1111: I put together ps to prevent this condition. | 15:33 |
| kfox1111 | sbezverk: but, yeah. if we wanted more consitstent behavior, making it easier to debug, we shoudl put in more waits. | 15:33 |
| *** chas has joined #openstack-kolla | 15:33 | |
| kfox1111 | rwellum: congrats! :) | 15:33 |
| sbezverk | kfox1111: it has not | 15:33 |
| sbezverk | kfox1111: http://logs.openstack.org/84/453384/9/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/223eb4e/console.html#_2017-04-05_15_00_00_541962 | 15:34 |
| sbezverk | you will see how control plane were coming up | 15:34 |
| kfox1111 | sbezverk: really? hmm... | 15:34 |
| sbezverk | rwellum: COngrats, | 15:34 |
| kfox1111 | I thought we already had a wait for that in setup_kubernetes.sh | 15:34 |
| rwellum | Cheers sbezverk ! | 15:34 |
| sbezverk | hopefully many more in future ;) | 15:34 |
| sbezverk | kfox1111: https://review.openstack.org/#/c/453384/9 | 15:35 |
| kfox1111 | sbezverk: yeah. setup_kubernetes.sh ~line 113. | 15:35 |
| *** chas has quit IRC | 15:35 | |
| *** chas has joined #openstack-kolla | 15:36 | |
| sbezverk | kfox1111: well I have proof that it was not working at that time | 15:36 |
| sbezverk | and it could not | 15:36 |
| sbezverk | becuase it would always be failing as kube-dns were in pending until networking is done | 15:37 |
| *** duonghq has joined #openstack-kolla | 15:37 | |
| duonghq | evening guys | 15:37 |
| kfox1111 | sbezverk: I think thats something we need to dig deeper into. if thats failing, other things could be too. | 15:38 |
| sdake | eqlooka pewrry eoxkin susw :) | 15:38 |
| sbezverk | kfox1111: Now before pushing canal, I make sure all essential pods are running before doing canal | 15:38 |
| sdake | rwellum looks pretty rockin dude :) | 15:38 |
| kfox1111 | all its doing is trying to get a successfull kube get pods, so in order for that to work, apiserver must be up. | 15:38 |
| kfox1111 | that means etcd/apiserver must be responsive. and the rest can come up too. | 15:38 |
| sbezverk | kfox1111: I do not know but I clearly see apiserver in pending in the log I sent you | 15:39 |
| sdake | rwellum are ll necessary steps required in the deployment guide | 15:39 |
| sdake | I haven't looked at the review this mroning - had a late start (7am) | 15:39 |
| sbezverk | kfox1111: and it is up time only 7 seconds | 15:39 |
| kfox1111 | in fact, it must be true for the kube update that goes in a few lines after to go in.... | 15:39 |
| kfox1111 | 2017-04-05 15:00:00.398944 | clusterrolebinding "cluster-admin" replaced | 15:40 |
| kfox1111 | so apiserver/etc had to be up... | 15:40 |
| kfox1111 | then it failed? | 15:40 |
| *** chas has quit IRC | 15:40 | |
| sbezverk | kfox1111: 2017-04-05 15:01:14.143039 | Pod name: kube-apiserver-centos-7-2-node-osic-cloud1-s3500-8284205 status: Pending | 15:40 |
| sbezverk | it is hard to argue ;) | 15:40 |
| *** jistr|mtg is now known as jistr | 15:40 | |
| sdake | rwellum were you able to start a vm from that kubernetes + kolla deployment? | 15:41 |
| *** gardlt has joined #openstack-kolla | 15:41 | |
| *** yingjun has quit IRC | 15:41 | |
| kfox1111 | sbezverk: it happened right after the node was untanted. | 15:41 |
| kfox1111 | maybe that is related? | 15:41 |
| *** sayantan_ has joined #openstack-kolla | 15:41 | |
| kfox1111 | it looks like it maybe decided to move the etcd/apiserver to the slave node at that point? | 15:41 |
| sbezverk | kfox1111: I would be very surprised to see untained killing apiserver ;) | 15:41 |
| sdake | kfox1111 in the review -- you said to make recommendations how ot pick a service network | 15:41 |
| sdake | kfox1111 I have no idea how to make such recommendations | 15:41 |
| sdake | I blindly picked an rfc1918 address | 15:42 |
| sbezverk | kfox1111: I can see the same happens in a signle node as well | 15:42 |
| sdake | kfox1111 which is probably not optimal - can you help me through how to document such a recommendation? | 15:42 |
| kfox1111 | sbezverk: look at the log though. it was stable as of setup_kubernetes, and then failed afterwards. | 15:42 |
| kfox1111 | but untaint talked to the api server, and then yoru script says it broke. | 15:42 |
| sbezverk | kfox1111: http://logs.openstack.org/84/453384/9/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-iscsi-nv/e507816/console.html#_2017-04-05_14_56_57_917316 | 15:43 |
| kfox1111 | sdake: yes. the user picing a non overlapping range from rfc1918 from their organizaion is important. | 15:43 |
| kfox1111 | sdake: basically, if your organization is a personal one, pick whatever you want. if its a buisness, ask your network admin. | 15:43 |
| sdake | thanks - so "non-overlapping" is the key | 15:43 |
| kfox1111 | from the rfc1918 space I mean. | 15:43 |
| kfox1111 | right. | 15:43 |
| sdake | and 10.3.0.0/16 would have overlaped | 15:44 |
| sdake | (in your environment) | 15:44 |
| sdake | pod cidr range doesn't matter then? | 15:44 |
| kfox1111 | hmm... let me try and rephrase that a bit.... | 15:44 |
| kfox1111 | there are two requirements. | 15:44 |
| kfox1111 | the service and pod address spaces cant overlap with each other. | 15:44 |
| kfox1111 | and, the address spaces cant be already allocated by your organization. | 15:44 |
| rwellum | sdake: pretty much followed the guide, with a couple of exceptions I am studying now and will feed back if necessary on the review | 15:44 |
| sbezverk | kfox1111: http://logs.openstack.org/84/453384/9/check/gate-kolla-kubernetes-deploy-centos-binary-2-iscsi-nv/ec5377c/console.html#_2017-04-05_15_00_08_722487 | 15:45 |
| kfox1111 | at my site, the addresses picked are already allocated by my org. | 15:45 |
| sdake | kfox1111 right | 15:45 |
| sdake | makes sense | 15:45 |
| kfox1111 | the same is true of the openstack-infra org. | 15:45 |
| sdake | didn't knwo the openstack part | 15:45 |
| kfox1111 | those adress spaces overlap with the org. | 15:45 |
| sdake | must they always be rfc1918 addresses? | 15:45 |
| kfox1111 | not strictly, no. | 15:46 |
| kfox1111 | any address space will do. | 15:46 |
| kfox1111 | BUT, | 15:46 |
| kfox1111 | pod network is very very wasteful of ip's. | 15:46 |
| sdake | right - so pod network should be rfc1918 | 15:46 |
| kfox1111 | the service network could, but isn't routable anyway, so not relaly much benifit there either. | 15:46 |
| sdake | because it consumes a /24 per host | 15:46 |
| *** ksumit has quit IRC | 15:46 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Change heat cfn service description https://review.openstack.org/446453 | 15:46 |
| kfox1111 | sdake: right. | 15:47 |
| sdake | ok - the service network - is thta the network used to connect into the atual services? | 15:47 |
| kfox1111 | whether its allocated the ips or not. | 15:47 |
| sdake | kfox1111 thanks btw, this is very helpful | 15:47 |
| kfox1111 | kind: service <- those get service ip's. | 15:47 |
| sbezverk | kfox1111: all these showed that transition happened from pending to running after setup_kubernetes script has completed.. | 15:47 |
| kfox1111 | service ip's come from the service ip space. | 15:48 |
| kfox1111 | to complicate matters, in the service spec, they call them "clusterIP"s too. | 15:48 |
| kfox1111 | sbezverk: I dont disagree. but I'm saying, there's something wrong at a deeper level we're not understanding I think. | 15:48 |
| sdake | what ip address range is used to communicate with the actual pods from the outside? | 15:48 |
| kfox1111 | sbezverk: the control plane shoudl have been stable after setup_kubernetes.sh finished. | 15:49 |
| kfox1111 | if its not, we have bigger issues. | 15:49 |
| kfox1111 | sdake: there's kind of an east/west north/south response involved there. | 15:49 |
| kfox1111 | different paths use different ips'. | 15:50 |
| mnaser | meeting in 10 fyi | 15:50 |
| kfox1111 | well... hold on. let me parse your sentence again... | 15:50 |
| kfox1111 | sdake: maybe an example. are you asking, say, | 15:50 |
| sdake | kfox1111 let me clarify - when I connect to keystone externally - how do i do that | 15:50 |
| sdake | what ip is used | 15:50 |
| kfox1111 | how does a keystoen user-list from outside k8s get to the pod? | 15:50 |
| sbezverk | kfox1111: it is not ready after that, with the script I added it becomes stable and canal gets deployed | 15:50 |
| sdake | kfox1111 yup | 15:51 |
| kfox1111 | sbezverk: I'm saying it is stable, but then shortly after becomes unstable. I wana know why. I'm guessing its the taint, as thats the last time in your log it is stable? | 15:51 |
| kfox1111 | sdake: ah. | 15:51 |
| inc0 | meeting in T-9min | 15:51 |
| kfox1111 | sdake: so, thats a bit up to the admin of k8s. there are multiple ways of setting that up. I can give you some reference ideas though. | 15:52 |
| sbezverk | kfox1111: sorry I do not get the last part, I mean your question.. | 15:52 |
| sdake | kfox1111 please | 15:52 |
| kfox1111 | sdake: like, how we are doing it in the gate, or how I am doing it at one of my sites. | 15:52 |
| sdake | kfox1111 your site | 15:52 |
| sdake | kfox1111 or generalize it :) | 15:52 |
| sdake | kfox1111 i know i may be running after i just learned to crawl | 15:52 |
| sdake | however, Id like to have multiple nodes and hae that work | 15:53 |
| sdake | kfox1111 this doesn't necessarily have to go in the docs - more for my own seeding of my brain - for later | 15:53 |
| kfox1111 | sbezverk: I think the system is stable, then going unstable, and then your checking for the instability. better would be to understand why its going unstable and fix it. then we don't need a double check. | 15:53 |
| sdake | (sometiem in the next few months this will click) | 15:53 |
| kfox1111 | sdake: the easiest way to do multinode is what we're doing in the gate. has no ha though. | 15:54 |
| kfox1111 | like I'm doing at my site. | 15:54 |
| kfox1111 | so maybe we cover that first? | 15:54 |
| sdake | kfox1111 sure | 15:54 |
| sdake | kfox1111 we have a meeting - hoepfully we can continue that after our team meeting | 15:55 |
| kfox1111 | k. so, there are multiple types of kubernetes services. | 15:55 |
| kfox1111 | k. | 15:55 |
| sbezverk | kfox1111: hmm fixing kubernetes is not really our goal, as long as we have a way to run it with stability should sufice, no? | 15:56 |
| *** vhosakot has joined #openstack-kolla | 15:57 | |
| *** bjolo has quit IRC | 15:57 | |
| *** bjolo_ has joined #openstack-kolla | 15:57 | |
| kfox1111 | sbezverk: I'm not saying its our bug to fix. or that its even a bug. but we should understand what is going on. | 15:57 |
| kfox1111 | if its the taint destabilizing things, maybe we're untainting wrong. or the untaint belongs in the setup_kubernetes.sh before the kube-api stability check. | 15:57 |
| kfox1111 | sbezverk: https://review.openstack.org/#/c/453277/ seems to have fixed the v3 gate. | 15:58 |
| *** skramaja has joined #openstack-kolla | 15:59 | |
| sbezverk | kfox1111: perfect | 15:59 |
| *** skramaja has quit IRC | 15:59 | |
| sdake | inc0 i updated the wiki for the meeting just now | 15:59 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add external rabbitmq https://review.openstack.org/453724 | 15:59 |
| sdake | inc0 in case you had it open - plz reload :) | 15:59 |
| sbezverk | kfox1111: I think the confusion comes from the fact that kubectl does not immediately fail on commands but retries which creates impression that it is working, when in fact it is still trying.. | 16:00 |
| kfox1111 | sbezverk: but get pods worked. and untaint worked. it wasn't just waiting. it successfully talked to kube-apiserver and etcd. | 16:00 |
| kfox1111 | and kubectl update ClusterRoleBinding | 16:01 |
| *** jascott1 has joined #openstack-kolla | 16:02 | |
| *** jascott1- has joined #openstack-kolla | 16:02 | |
| *** shardy is now known as shardy_afk | 16:02 | |
| kfox1111 | jascott1: congrats! :) | 16:02 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 16:02 |
| *** eaguilar has joined #openstack-kolla | 16:02 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new rabbitmq instance for Murano https://review.openstack.org/453724 | 16:04 |
| *** skramaja has joined #openstack-kolla | 16:04 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new rabbitmq instance for Murano https://review.openstack.org/453724 | 16:06 |
| mnaser | pbourke nice patch, this can probably be re-used for trove as well | 16:10 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Move memcached config to kolla-k8s https://review.openstack.org/450620 | 16:10 |
| pbourke | mnaser: thanks, its working for me just needs one or two tweaks. original credit goes to bdaca who I believe has moved on from the project | 16:10 |
| pbourke | egonzalez: I completely missed your original ML thread! | 16:11 |
| *** rwallner has quit IRC | 16:11 | |
| mnaser | pbourke gotcha, be nice to have trove support in there but if thats not something that you have the time to work on, i can pick up from the rest of your work | 16:11 |
| *** chas has joined #openstack-kolla | 16:11 | |
| pbourke | mnaser: sure, lets get it polished and merged for murano first, then you can copy/paste for trove | 16:11 |
| mnaser | pbourke cool :) | 16:12 |
| egonzalez | pbourke, yep trove is also affected, don't know if someother projects are using that kind of architecture | 16:12 |
| pbourke | I think bdaca's original idea was to have one external rabbit for all these kind of projects | 16:12 |
| pbourke | maybe that would be better than a murano_rabbit, trove_rabbit, etc | 16:12 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Move cinder config to kolla-k8s https://review.openstack.org/450556 | 16:13 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 16:13 |
| *** sayantan_ has quit IRC | 16:13 | |
| *** sayantan_ has joined #openstack-kolla | 16:14 | |
| pbourke | egonzalez: "Hell no, this is not an option" - ha | 16:14 |
| mnaser | pbourke egonzalez yeah when i saw that option listed in the relm of possibilities | 16:15 |
| mnaser | my heart sank till i saw the cons | 16:15 |
| mnaser | haha | 16:15 |
| pbourke | lol | 16:15 |
| *** chas has quit IRC | 16:15 | |
| *** gardlt has quit IRC | 16:16 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Make rabbitmq log dir configurable https://review.openstack.org/453735 | 16:18 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new rabbitmq instance for Murano https://review.openstack.org/453724 | 16:20 |
| *** blallau has quit IRC | 16:20 | |
| *** chas has joined #openstack-kolla | 16:21 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new rabbitmq instance for Murano https://review.openstack.org/453738 | 16:22 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Make rabbitmq log dir configurable https://review.openstack.org/453735 | 16:22 |
| sbezverk | kfox1111: see this output, it proves that no containers are running when we exist from setup_kubernetes, specifically line 13 | 16:24 |
| sbezverk | http://paste.openstack.org/show/605534/ | 16:24 |
| kfox1111 | sbezverk: thats not how I interprite that output. | 16:25 |
| kfox1111 | it says: k8s was up enough to respond to get pods, | 16:25 |
| kfox1111 | to update the clusterrolebinding, and untaint a node. | 16:26 |
| sbezverk | kfox1111: why pods then are not shown? | 16:26 |
| kfox1111 | where? | 16:26 |
| *** chas has quit IRC | 16:26 | |
| kfox1111 | oh. by get pods? | 16:26 |
| sbezverk | please check latest PS | 16:26 |
| kfox1111 | in setup_kubernetes.sh? | 16:26 |
| sbezverk | yes | 16:26 |
| sbezverk | it supposed to oputput something | 16:27 |
| kfox1111 | kubectl get pods -n kube-system > /dev/null | 16:27 |
| kfox1111 | no its not. :) | 16:27 |
| sbezverk | kfox1111: man, please check my latest PS | 16:27 |
| kfox1111 | ok. sec. | 16:27 |
| sbezverk | these logs are from there | 16:27 |
| kfox1111 | ok. so you added a second get pods... | 16:28 |
| kfox1111 | looking at the logs.... | 16:28 |
| kfox1111 | oh. I may know.... | 16:29 |
| kfox1111 | etcd/kube-apiserver/scheduler/controller-manager are static pods. | 16:29 |
| pbourke | mnaser: egonzalez: I think im going to refactor this to one "external rabbitmq" murano/trove/whatever can use. shout it you think we need one rabbit per service | 16:29 |
| kfox1111 | they get dummy entries added to the apiserver by kubelet. | 16:29 |
| kfox1111 | kubelet may not have added them yet. | 16:29 |
| kfox1111 | but that doesn't mean they aren't there/working. | 16:29 |
| *** matrohon has quit IRC | 16:30 | |
| sbezverk | kfox1111: I do not see why you insist on using check within setup_kubernetes | 16:30 |
| kfox1111 | the check in setup_kubernetes checks to see if k8s is working. | 16:30 |
| mnaser | pbourke i think that's the easiest way to start things off i guess | 16:30 |
| kfox1111 | not if its reporting the shadow pods. | 16:30 |
| mnaser | rabbitmq per service feels excessive | 16:30 |
| sbezverk | doing it in the script is more relaible as ALL important containers are checked before the networking gets pushed | 16:30 |
| pbourke | mnaser: it would be easier not to refactor ;) | 16:30 |
| sbezverk | kfox1111: which shadow pods ?!?!? | 16:31 |
| mnaser | i dont know, there's pros and cons to each one pbourke :( | 16:31 |
| kfox1111 | sbezverk: not aposed to doing your check. but was trying to understand why it was failing when cli was working. | 16:31 |
| pbourke | :/ | 16:31 |
| kfox1111 | the answer is shadow pods. | 16:31 |
| *** shardy_afk is now known as shardy | 16:31 | |
| kfox1111 | the shadow pods for etc/kube-apiserver/scheduler/controller-manager. | 16:31 |
| egonzalez | pbourke, I think we can start with one rabbit instance shared between end-user services and then add support to use one instance per service | 16:32 |
| pbourke | egonzalez: that support is in my current patch | 16:33 |
| sbezverk | kfox1111: here is my logic.. when we install cluster we never see race conditions as we see at the gate | 16:33 |
| pbourke | egonzalez: im just not sure its a good idea to have lots and lots of rabbit clusters | 16:33 |
| pbourke | seems like overkill | 16:33 |
| sbezverk | to prevent it, we need to control the process better.. | 16:33 |
| kfox1111 | sbezverk: hand rolled is slower then the gate. the races tend to not be issues then. | 16:33 |
| kfox1111 | fullly orchestrated is usually shows more races, unless you throw in a tun of sleeps to slow it down to what a human normally does. | 16:33 |
| kfox1111 | the other issue is the gate is made up of many differing machine types. | 16:34 |
| kfox1111 | sometimes issues there creap in. | 16:34 |
| sbezverk | kfox1111: exactly, so makeing sure that cluster is completely up without shadow pods or other crap, will gives us better base for next steps.. | 16:34 |
| kfox1111 | sbezverk: no, I'm saying, your check is checking for shadow pods, not if the cluster is up. | 16:35 |
| kfox1111 | its not checking for what you think its checking for. | 16:35 |
| mnaser | pbourke i can imagine multiple instances causing complications indeed | 16:35 |
| mnaser | in kolla meeting, i can discuss further in a ibt :X | 16:35 |
| pbourke | arg I totally forgot about the meeting | 16:35 |
| sbezverk | kfox1111: well, one thing for sure, once I added this check multinode stopped failing on networking installation.. | 16:35 |
| sbezverk | it seems more reliable now | 16:36 |
| kfox1111 | sbezverk: its very likely that it takes a minute or two for the shadow pods to show. | 16:36 |
| kfox1111 | which means its similar to adding a sleep 60 at that point. | 16:36 |
| kfox1111 | which gives things more time to settle. | 16:36 |
| kfox1111 | hiding a race condition. | 16:36 |
| kfox1111 | I totally buy that would make things seem more stable most of the time. | 16:37 |
| sbezverk | kfox1111: right it is a delay, but it is controlled delay | 16:37 |
| sbezverk | also even when you install the cluster you do use kubectl get pods to see if pods are there | 16:37 |
| sbezverk | and if you do not see them, you think something is wrong | 16:37 |
| kfox1111 | yeah. but the problem is, thats not quite true in static pod case. threw me until Itried to understand the issue. | 16:38 |
| kfox1111 | I'm ok merging in your ps now that I understand the issue. though I think it probably should merge into setup_kubernetes.sh. | 16:38 |
| *** iceyao has joined #openstack-kolla | 16:39 | |
| sbezverk | kfox1111: it is already look very complex, this way looks much cleaner and better to understand for people looking at the code | 16:39 |
| sbezverk | kfox1111: please find 2nd core to ack v3 ps | 16:40 |
| kfox1111 | the code should be broken up into tasks I think. setup_kubernetes is about ensureing a working k8s comes out the end of the script. | 16:40 |
| sbezverk | kfox1111: we might get all green gate soon | 16:40 |
| kfox1111 | k. will do. | 16:41 |
| *** ksumit has joined #openstack-kolla | 16:43 | |
| *** sayantan_ has quit IRC | 16:43 | |
| *** sayantan_ has joined #openstack-kolla | 16:43 | |
| *** iceyao has quit IRC | 16:43 | |
| sbezverk | kfox1111: when I think about shadow pods you mentioned, at one point normal pods takes over of shadow pods, so I suspect if something happening at the same time, it could potentially get impacted, resulting instability we observe. | 16:45 |
| *** hieulq_ has joined #openstack-kolla | 16:49 | |
| kfox1111 | there shouldn't be any deployments/daemonsets for those services. | 16:53 |
| kfox1111 | I think they should always be shadows. | 16:53 |
| kfox1111 | if it was bootkube, I'd say that oculd be a problem. | 16:53 |
| sbezverk | kfox1111: but we can clearly see kube-apiserver going from pending to running state | 16:55 |
| kfox1111 | hmm... true. | 16:55 |
| kfox1111 | but I'm not aware of any machinery that would ever cause that. | 16:55 |
| sbezverk | it means in pending state your shadow container serves api requests | 16:55 |
| kfox1111 | well... let me check the log again. | 16:56 |
| kfox1111 | is it actually pending? | 16:56 |
| *** rwallner has joined #openstack-kolla | 16:56 | |
| sbezverk | kfox1111: http://logs.openstack.org/84/453384/11/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/610f58b/console.html#_2017-04-05_16_20_55_817027 | 16:57 |
| kfox1111 | sdake: https://review.openstack.org/#/c/453277/ please | 16:57 |
| sbezverk | it is pending and then 5 seconds later becomes running | 16:57 |
| kfox1111 | sbezverk: weird. | 16:57 |
| *** rwallner has quit IRC | 16:57 | |
| kfox1111 | I wouldn't htink a shadow pod would enter pending state. | 16:57 |
| *** rwallner has joined #openstack-kolla | 16:57 | |
| kfox1111 | unless the controller manager does sanity checks.... | 16:58 |
| kfox1111 | maybe thats it... | 16:58 |
| kfox1111 | it could be the readyness check in the static pod blocking kube-proxy for the kubernetes.default.svc... | 16:58 |
| *** nathharp has quit IRC | 16:58 | |
| kfox1111 | may be a change of behavior in 1.6... | 16:58 |
| inc0 | kfox1111: have you seen mgoddard project? | 16:59 |
| inc0 | since you both work on scientific openstacks | 16:59 |
| kfox1111 | inc0: not sure. got a link? | 16:59 |
| inc0 | probably something interesting:) | 16:59 |
| rwellum | sdake: from the DG - is this expected to work? "openstack server add floating ip demo1 $(openstack floating ip create public1)" | 16:59 |
| inc0 | kfox1111: hold on | 17:00 |
| kfox1111 | inc0: https://review.openstack.org/#/c/453277/ please | 17:00 |
| sdake | rwellum i tried it | 17:00 |
| sdake | and it does not work | 17:00 |
| sdake | rwellum someone suggested that in an earlier review | 17:00 |
| sdake | and i either c&p fubared it | 17:00 |
| kfox1111 | rwellum: yeah, I think you need to parse out the id. | 17:00 |
| inc0 | https://github.com/stackhpc/kayobe | 17:00 |
| sdake | or the person that added it was wrong :) | 17:00 |
| sdake | rwellum after init-runonce runs - it shows you how to create a vm | 17:01 |
| kfox1111 | inc0: ah. very interesting. :) | 17:01 |
| rwellum | yeah kfox1111 otherwise it's a big old table... | 17:01 |
| inc0 | ok bbiaf, going to office | 17:01 |
| *** jascott1 has quit IRC | 17:01 | |
| *** jascott1- is now known as jascott1 | 17:01 | |
| kfox1111 | inc0: thats what I want to do, but with kolla-kubernetes. :) | 17:01 |
| inc0 | we should combine efforts then | 17:01 |
| kfox1111 | inc0: +1 | 17:01 |
| sdake | eanylin can you confirm you did get 4.0.0 images running and booting a vm with source | 17:01 |
| sdake | with centos-source imgaes? | 17:01 |
| rwellum | BTW I have to say, I've attended a few kolla meetings and this by far is my favorite statement anyone has every written: "have to go - starts raining and I am at the park now". | 17:02 |
| *** rstarmer has joined #openstack-kolla | 17:02 | |
| inc0 | how about we'll comandeer one of sciengific wg meetings to talk about "how to make good scientific openstack deployment tool"? | 17:02 |
| *** mgoddard has quit IRC | 17:02 | |
| kfox1111 | sbezverk: so... if that theory is true, then we actually need to ensure the shadow pods enter ready state and your patch set is not just a sleep, but critical. | 17:03 |
| *** shardy has quit IRC | 17:03 | |
| inc0 | anyway, bbiaf | 17:03 |
| *** egonzalez has quit IRC | 17:03 | |
| kfox1111 | inc0: sure | 17:03 |
| kfox1111 | sbezverk: my long term goal of setup_kubernetes is to have something that the docs can say "if you don't want to deal with k8s deployment, just call setup_kubernetes.sh" | 17:04 |
| *** skramaja has quit IRC | 17:04 | |
| *** saneax has quit IRC | 17:04 | |
| sbezverk | kfox1111: kubernetes is smart enough to sort things out but not always. When I saw in "messages" lots of retries for different things at the gate and not in my test bed, it made me thinking.. | 17:04 |
| kfox1111 | so having the check in setup_kubernetes makes it easier to use for that use case. | 17:05 |
| sbezverk | kfox1111: we use wait_for_something already in the script | 17:05 |
| kfox1111 | sbezverk: yeah. | 17:05 |
| sbezverk | so it does not break already existing concept | 17:05 |
| kfox1111 | sbezverk: maybe just move the script callout from setup_get to setup_kubernetes in the master section? | 17:05 |
| kfox1111 | setup_gate I mean | 17:06 |
| *** duonghq has quit IRC | 17:06 | |
| mnaser | pbourke as a middle ground could we do external rabbitmq with vhost per service? | 17:06 |
| sbezverk | I can replace already existing check in setup_kuebernets | 17:06 |
| sbezverk | maybe it will be less confusing | 17:07 |
| kfox1111 | k. | 17:07 |
| kfox1111 | works for me. | 17:07 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Disable placement for v3 gate. https://review.openstack.org/453277 | 17:08 |
| *** jaosorior is now known as jaosorior_away | 17:08 | |
| *** athomas has quit IRC | 17:10 | |
| rwellum | kfox: with the ip address I got closer https://www.irccloud.com/pastebin/XimbyWzA/ | 17:10 |
| *** saneax has joined #openstack-kolla | 17:11 | |
| kfox1111 | neutron net-list? | 17:11 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 17:11 |
| *** unicell has quit IRC | 17:12 | |
| openstackgerrit | Qin Wang (qwang) proposed openstack/kolla-kubernetes master: [WIP] Condition and tags in requirements.yaml for keystone service https://review.openstack.org/444461 | 17:12 |
| rwellum | kfox1111 https://www.irccloud.com/pastebin/jpEQHFbD/ | 17:12 |
| kfox1111 | rwellum: neutron net-show demo-net | 17:13 |
| kfox1111 | and one for public1 | 17:13 |
| rwellum | kfox1111 https://www.irccloud.com/pastebin/lFfdNlQb/ | 17:14 |
| kfox1111 | weird. k.... | 17:15 |
| kfox1111 | anything in the neutron-server logs? | 17:15 |
| sdake | rwellum rendering new docs now - and running through quickstart | 17:17 |
| *** lrensing has quit IRC | 17:18 | |
| rwellum | kfox1111 https://www.irccloud.com/pastebin/ZURYpFAx/ | 17:19 |
| *** ccesario has quit IRC | 17:19 | |
| kfox1111 | rwellum: kubectl exec -it neutron-server-2216068945-1gjk1 -n kolla -c main vi /var/log/kolla/neutron/* | 17:20 |
| kfox1111 | and see if there is anything interesting there? | 17:20 |
| rwellum | sdake: I think theres a couple of things you need to know before you run through another deployment - that helped me anyway. Not in your guide. | 17:20 |
| sdake | rwellum its our guide - i am just an editor :) | 17:20 |
| sdake | please do tell and I'll rev hte guide first with notes and then render and deploy | 17:21 |
| sdake | rwellum can you leave notes? | 17:21 |
| *** athomas has joined #openstack-kolla | 17:22 | |
| rwellum | sdake - I have added several today - just give me a minute to add a couple more | 17:23 |
| rwellum | Just don't want you to waste your time. | 17:23 |
| rwellum | kfox1111: that's a huge log - anything I should look for? | 17:23 |
| kfox1111 | anything that looks like a stacktrace? | 17:24 |
| sbezverk | kfox1111: moved to setup_kubernetes, please check if you are ok.. from what I see it is etcd takes the longest time to come up.. | 17:24 |
| rwellum | kfox1111: not yet but 7k lines of logs | 17:26 |
| rwellum | sdake: I left some notes yesterday in reviews and I think you missed them. | 17:26 |
| inc0 | back | 17:27 |
| *** hieulq__ has joined #openstack-kolla | 17:30 | |
| kfox1111 | sbezverk: there's an issue with the change. | 17:31 |
| kfox1111 | you moved setup_canal up to before the second node gets added. | 17:31 |
| kfox1111 | but then the tests to ensure canal is stable on the second node get dropped, as they are in setup_canal... | 17:31 |
| *** gfidente is now known as gfidente|afk | 17:32 | |
| kfox1111 | I think test_kube_dns needs to be removed from setup_canal then, and move to where setup_canal was before. | 17:32 |
| kfox1111 | and right in front, do: | 17:33 |
| kfox1111 | $DIR/tools/pull_containers.sh kube-system | 17:33 |
| kfox1111 | $DIR/tools/wait_for_pods.sh kube-system | 17:33 |
| *** sambetts is now known as sambetts|afk | 17:33 | |
| sdake | rwellum ok - editing is hard with the hundreds of comments on the doc so far | 17:34 |
| sdake | rwellum if you could update with the missed notes the latest review - i'll make suer to incorporate them as notes for now | 17:34 |
| sdake | and then render them | 17:34 |
| sbezverk | kfox1111: that is what I want | 17:34 |
| *** hieulq_ has quit IRC | 17:34 | |
| sdake | andwhen i get a workflow that wfm, I'll clean it up | 17:34 |
| sdake | and then hopefully we can merge it | 17:34 |
| sbezverk | I tested several times and setting up everything on a master including networking and then adding second node look way more stable | 17:35 |
| kfox1111 | sbezverk: yeah. but with those changes, it should be more stable yet. | 17:35 |
| kfox1111 | cause now, its not testing that dns on the second node ever works. | 17:35 |
| kfox1111 | which we have seen break in the past. | 17:35 |
| sbezverk | kfox1111: I do not want to take any chances.. when you build kube cluster you cannot join all your nodes and only then setup networking | 17:36 |
| kfox1111 | sbezverk: I'm ok with yoru canges, provided that the testing that was done before remains. | 17:36 |
| kfox1111 | thats the onlyh issue I have. | 17:37 |
| kfox1111 | the change as proposed just drops some validity testing that was happening before. | 17:37 |
| rwellum | sdake: :) doing that now. | 17:39 |
| *** unicell has joined #openstack-kolla | 17:41 | |
| *** rstarmer has quit IRC | 17:42 | |
| *** ccesario has joined #openstack-kolla | 17:42 | |
| *** bjolo_ has quit IRC | 17:46 | |
| *** tonanhngo has joined #openstack-kolla | 17:47 | |
| *** tonanhngo_ has joined #openstack-kolla | 17:50 | |
| *** tonanhngo has quit IRC | 17:51 | |
| *** tonanhngo_ has quit IRC | 17:54 | |
| *** vhosakot_ has joined #openstack-kolla | 17:55 | |
| kfox1111 | wow... hadn't seen this issue in a long time: http://logs.openstack.org/84/453384/12/check/gate-kolla-kubernetes-deploy-centos-binary-2-iscsi-nv/4bb7cfd/console.html | 17:55 |
| sdake | rwellum thanks dude :) | 17:55 |
| *** tonanhngo has joined #openstack-kolla | 17:55 | |
| kfox1111 | guess docker still has the bug. | 17:55 |
| *** vhosakot has quit IRC | 17:56 | |
| *** hieulq__ has quit IRC | 17:56 | |
| rwellum | sdake: done. I think I have some important ones here. | 17:58 |
| *** Pavo has quit IRC | 18:01 | |
| *** rwallner has quit IRC | 18:03 | |
| *** rwallner has joined #openstack-kolla | 18:03 | |
| *** rwallner has quit IRC | 18:04 | |
| *** rwallner has joined #openstack-kolla | 18:04 | |
| *** krtaylor has quit IRC | 18:07 | |
| *** manheim_ has joined #openstack-kolla | 18:07 | |
| rwellum | sdake: I think I made 1 mistake with current deployment. I used wrong external_vip address. I've added a comment in the review about how that's misleading in the guide. | 18:07 |
| *** manheim_ has quit IRC | 18:11 | |
| sdake | rwallner ok - so guide ready to go for a rev and then test - or need more commentary? | 18:14 |
| *** jascott1_ has joined #openstack-kolla | 18:15 | |
| *** lucasxu has joined #openstack-kolla | 18:18 | |
| sdake | sbezverk is the gate functional now after your work? | 18:19 |
| kfox1111 | sdake: thats the last ps I know of to fix issues. | 18:21 |
| kfox1111 | doesn't mean there isn't more, but should be stable I think. | 18:21 |
| sdake | cool | 18:21 |
| sdake | i'll recheck cinder | 18:21 |
| rwellum | sdake: yes ready | 18:22 |
| sdake | if that works - i'll resubit the entire patch stream to trigger the gate again | 18:22 |
| sdake | rwellum ok - processing the new rev now | 18:22 |
| rwellum | sdake: rwellum != rwallner | 18:22 |
| sdake | rwellum re this comment: I think we need better defaults in the suggested cloud.yaml file. The external_vip should be a 192 address agreed. The others should be 10.x | 18:23 |
| sdake | rwellum i know, its a tab cmopletion thing that screws up - sorry :) | 18:23 |
| sdake | rwellum 10.0.0.1? | 18:23 |
| sdake | or some other value | 18:23 |
| sdake | i dont know what to make these values | 18:23 |
| *** lrensing has joined #openstack-kolla | 18:23 | |
| rwellum | sdake: agreed tricky because unique to each environment. Thing is you want it to be a local address, and not the same as the other 192.x addresses - that's gotten me a few times especially as 2 lines down we reference the local interface name. | 18:24 |
| rwellum | How about 10.X.X.X ? | 18:25 |
| rwellum | Really make it clear it's unique | 18:25 |
| sdake | rwellum not being difficutl here - I just want to understand what you understand :) | 18:25 |
| sdake | rwellum sbezverk told me to set dns_name to my hosts ip address | 18:26 |
| *** ntpttr_ has quit IRC | 18:26 | |
| rwellum | That's different to the external_vip - not sure your point? | 18:27 |
| sdake | rwellum i have no point really - i want to explore what ou understand about these values | 18:28 |
| sdake | because Idon't know what the IPs shoudl be set to in cloud.yaml | 18:28 |
| sdake | if I could map them to my environment correctly, then I could explain how to map them to others | 18:28 |
| *** rstarmer has joined #openstack-kolla | 18:29 | |
| sdake | (i.e. make the instructions generic) | 18:29 |
| kfox1111 | networking's always the hard part. | 18:29 |
| sdake | it is possible this is why my vms don't come up | 18:29 |
| sdake | kfox1111 as my wife says - my company didn't build its empire because networking is easy :) | 18:30 |
| sdake | rather my employer :) | 18:30 |
| kfox1111 | yup. | 18:30 |
| sdake | rwellum you likely ahve deep domain expertise in networking - i dont | 18:30 |
| rwellum | I'll share my cloud.yaml | 18:30 |
| sdake | rwellum cool | 18:30 |
| sdake | i have written a protocol | 18:30 |
| rwellum | https://www.irccloud.com/pastebin/jHbeikHE/ | 18:30 |
| rwellum | I am an original author of EIGRP :) | 18:31 |
| sdake | however, that protocol was for lans :) | 18:31 |
| sdake | cool - not sure what that is - the one I wrote was Totem | 18:31 |
| sdake | rwellum quiz time | 18:31 |
| sdake | what is this ip address in your network: | 18:31 |
| sdake | external_vip: "192.168.122.251" | 18:31 |
| sdake | just some random ip ? | 18:32 |
| sdake | (that is unused?) | 18:32 |
| *** rwallner has quit IRC | 18:32 | |
| rwellum | sdake - no it's the ip addr of my external interface on eth1 https://www.irccloud.com/pastebin/hJ3fHeTx/ | 18:32 |
| sdake | eth1 is your neutron network or your management network? | 18:33 |
| *** harlowja has quit IRC | 18:33 | |
| rwellum | Neutron - although I don't like that name. | 18:33 |
| rwellum | external is better | 18:33 |
| sdake | thanks | 18:33 |
| rwellum | imo | 18:33 |
| sdake | so that is p1p2 | 18:33 |
| sdake | in my environment | 18:33 |
| rwellum | yes as I recall | 18:33 |
| rwellum | Notice the tunnel_interface right underneath is not going to be p1p2 | 18:34 |
| kfox1111 | one thing we do need that we dont have yet is some reference architecture documents. | 18:34 |
| kfox1111 | this kind of thing would be easier with that. | 18:35 |
| sdake | and this address is what: dns_name: "10.240.43.213" | 18:35 |
| sdake | kfox1111 right - i'd like to just get some basic "how do you get it breathing" instructions first | 18:36 |
| sdake | kfox1111 rather then make sea salt with a "what do all the config optoins mean" :) | 18:36 |
| *** chas has joined #openstack-kolla | 18:36 | |
| *** sayanta__ has joined #openstack-kolla | 18:36 | |
| kfox1111 | sdake: yeah. just saying, sometimes a picutre is worth a thousand words. maybe a ref doc with a picutre with nics/ips on it would really help. | 18:36 |
| rwellum | I set that to my management ip address sdake | 18:36 |
| sdake | ok my management ip is 192.168.7.101 | 18:37 |
| sdake | so that should work with the sed operation | 18:37 |
| sdake | rwellum and this: - 10.240.43.213: cinder-volumes | 18:37 |
| sdake | same story? | 18:37 |
| rwellum | what is your external address? | 18:37 |
| rwellum | Yes. | 18:37 |
| sdake | external address ? p1p2 doesn't have an Ip address associated with it | 18:37 |
| sdake | neutron does not work if an ip address is associated with it | 18:38 |
| sdake | kfox1111 picture would be helpful - maybe after I understand what to draw I can draw one :) | 18:39 |
| rwellum | This is why your cloud.yaml is confusing to me then. That's why you have external and internal ip addresses the same - because you only have one ip address. | 18:39 |
| rwellum | So what are you going to set external_vip to? | 18:39 |
| sdake | i dont know what external_vip does | 18:39 |
| rwellum | eanylin: and I both have two bridges and two addresses. | 18:39 |
| sdake | bridges - how did you get bridges ? create them? | 18:40 |
| *** sayantan_ has quit IRC | 18:40 | |
| *** ntpttr_ has joined #openstack-kolla | 18:40 | |
| *** chas has quit IRC | 18:40 | |
| *** rwallner has joined #openstack-kolla | 18:41 | |
| *** rwallner has quit IRC | 18:41 | |
| *** rwallner has joined #openstack-kolla | 18:41 | |
| rwellum | We're both running on VM's. Actually eanylin shared his XML last night and he has two virsh virtual bridges which is very common with VM's right? In my case I have one NIC br0, and one I NATed - which is just using virsh virtual bridges. | 18:42 |
| sdake | this may explain why eanlyn can't access his neutron interface - because he has an ip address assigne to it | 18:42 |
| rwellum | Yeah but if that's the case, we're still stuck on what the heck external_vip is. | 18:43 |
| rwellum | The cloud.yaml clearly wants two different ip addresses. | 18:43 |
| sdake | kfox1111 can yo udefine external_vip for us plz :) | 18:43 |
| sdake | here is what we have in the review: | 18:44 |
| sdake | set 'ext_interface_name': name of the interface or bridge which will be used by neutron's provider interface. To determine the value of this parameter, | 18:44 |
| sdake | run ``grep neutron_network /etc/kolla/globals.yml``. | 18:44 |
| sdake | set 'ext_bridge_name': name of the bridge you want neutron to use as an external bridge. By default it should be br-ex. | 18:44 |
| sdake | set 'tunnel_interface': interface name used for connectivity between nodes in kubernetes cluster, in most of cases it matches the name of the kubernetes host management interface. To determine this, ``grep network_interface /etc/kolla/globals.yml``. | 18:44 |
| sdake | and this: set 'external_vip': your external ip address | 18:45 |
| kolla-slack | <kfox1111> Yes, but give me 15 min. Its complicated | 18:45 |
| sdake | kfox1111 roger - yo ugot it :) | 18:45 |
| sdake | rwellum i'll udpate the ret of the doc whlie we wait on kfox1111 - this is crucial information and i think closes the loop on the current open questions in hte deployment guide | 18:46 |
| sdake | in other news, my wife wants me to order an arm 64 rackmount server | 18:46 |
| sdake | so hrw I guess I will have hardware to work wit h:) | 18:47 |
| *** ksumit has quit IRC | 18:47 | |
| sdake | kfox1111 in 15 mins if you could answer - is there a QFP to RJ45 connector | 18:47 |
| sdake | all these arm servers come with QFP outputs | 18:48 |
| sdake | and my switch is rj45 | 18:48 |
| sdake | rwellum if you knwo as well - that would be cool :) | 18:48 |
| kolla-slack | <kfox1111> Qsfp you mean? | 18:48 |
| kolla-slack | <kfox1111> And what speed? | 18:48 |
| sdake | kfox1111 thats 40gig, there is a 10gig qfp | 18:48 |
| sdake | or sfp | 18:48 |
| sdake | maybe i got the term wrong | 18:49 |
| kolla-slack | <kfox1111> Sfp is gig. Spf+ 10g. Qsfp 40 | 18:49 |
| sdake | i'd really like to not hae to pay 1k/port for a switch | 18:49 |
| sdake | i've found a nexus 9k on provantage that is 48 port for 13k | 18:50 |
| sdake | which is too much for me to bite off for personal use | 18:50 |
| kolla-slack | <kfox1111> Which do you have? | 18:50 |
| sdake | i struggled to rationalize 100$/port ;) | 18:50 |
| kolla-slack | <kfox1111> Probably sfp gigabit? | 18:50 |
| sdake | i hae a netgear prosafe 10gig rj45 switch | 18:50 |
| kolla-slack | <kfox1111> The spec for sfp allows enough ppwer for a sfptorj45 adapter at gigabit. | 18:51 |
| kolla-slack | <kfox1111> Sfp+ doesnt. :/ | 18:51 |
| sdake | sadface | 18:51 |
| kolla-slack | <kfox1111> Yeah. :( | 18:51 |
| *** krtaylor has joined #openstack-kolla | 18:52 | |
| sdake | thanks for answering that question kfox1111 | 18:52 |
| sdake | kfox1111 i've been looking for an answer to that for days | 18:52 |
| kolla-slack | <kfox1111> Someone really dropped the ball on that one. Np | 18:52 |
| sdake | as I build out my datacenter^Hhacked together lab | 18:52 |
| *** iceyao has joined #openstack-kolla | 18:52 | |
| kolla-slack | <kfox1111> Had to fight that one alot myself. | 18:52 |
| sdake | the stupid copper qfp+ connectors are 300 bucks a crack too iirc | 18:53 |
| sdake | redonkulous | 18:53 |
| *** jascott1_ has quit IRC | 18:55 | |
| rwellum | sdake: I wonder if eanylin issue was that in globals.yaml he did not specify: neutron_plugin_agent: "openvswitch" | 18:57 |
| *** iceyao has quit IRC | 18:57 | |
| kfox1111 | back. | 18:58 |
| kfox1111 | ok... so... | 18:58 |
| *** nathharp has joined #openstack-kolla | 18:58 | |
| kfox1111 | kubernetes services. | 18:58 |
| sdake | rwellum I followed the guide *exactly* | 18:58 |
| kfox1111 | its kind of a load balancer. | 18:59 |
| sdake | I don't know if eanlyin does something special | 18:59 |
| rwellum | I added that as a comment sdake | 18:59 |
| rwellum | I feel it's needed | 18:59 |
| rwellum | feel=tested | 18:59 |
| kfox1111 | by default the type is ClusterIp. | 18:59 |
| sdake | rwellum openvswitch should be the default | 18:59 |
| rwellum | sdake: ok | 18:59 |
| sdake | rwellum no reason to override it | 18:59 |
| kfox1111 | and it grabs an unallocated one from the service ip range if the user hasn't specified. | 18:59 |
| rwellum | ack | 18:59 |
| sdake | rwellum unless it didn't work without it - which would indicate a problem elsewhere | 18:59 |
| kfox1111 | another important part is kube-proxy. | 19:00 |
| kfox1111 | it basically reads all svc's from k8s, and writes iptables rules that | 19:00 |
| kfox1111 | convert all traffic going to the svc's cluster ip to go one of the pods ip's bound to the svc. | 19:00 |
| kfox1111 | so kube-proxy is basically the component that implements the load balancer funcionality. | 19:01 |
| kfox1111 | but its a distributed load balancer. all nodes have to have it. | 19:01 |
| sdake | it being kube-proxy? | 19:01 |
| kfox1111 | yeah. | 19:02 |
| sdake | ok makes sense | 19:02 |
| *** rstarmer has quit IRC | 19:02 | |
| kfox1111 | so. | 19:02 |
| kfox1111 | the svc ips never actually exist in any real way. | 19:02 |
| kfox1111 | traffic destined for the ip get rewritten by iptables, but | 19:03 |
| kfox1111 | the ip isn't assigned to any interface like normally happens. | 19:03 |
| kfox1111 | so. then comes external ip's. | 19:04 |
| kfox1111 | so, k8s services have the option to use an external ip instead of a cluster ip. | 19:04 |
| kfox1111 | the behavior is identicle. | 19:04 |
| kfox1111 | all kube-proxies write iptables rules on their local machiens to map the external ip to flow traffic to the service's pods. | 19:05 |
| kfox1111 | But, | 19:05 |
| kfox1111 | if you make that ip a real ip on one of the hosts, it will work external to k8s. | 19:05 |
| kfox1111 | and iptables will still route the traffic to the pods just like it was coming from within k8s. | 19:06 |
| sdake | kool | 19:07 |
| sdake | cool | 19:07 |
| sdake | so - we have this: | 19:07 |
| sdake | keystone: | 19:07 |
| sdake | all: | 19:07 |
| sdake | admin_port_external: "true" | 19:07 |
| sdake | the reason for that is otherwise nobody can do any admin activites with keystone | 19:08 |
| sdake | in that case, we should set external_vip: "192.168.7.101" | 19:08 |
| sdake | to our mangement ip? | 19:08 |
| kfox1111 | with it set to false, the kubernetes service gets no external ip's. | 19:08 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Revert "Fix Fluentd warn on dnsmasq.log file parsing" https://review.openstack.org/453837 | 19:08 |
| kfox1111 | and the cli then only works from within the k8s cluster. | 19:09 |
| sdake | right - people want to use it externally | 19:09 |
| kfox1111 | if you make it an external port, its bound to the external ip and can be used externally to k8s. | 19:09 |
| kfox1111 | yeah. | 19:09 |
| kfox1111 | you can set it to whatever network you want. juts needs to be set to an ip on a mchine running kube-proxy and the sdn. | 19:10 |
| kfox1111 | for ha, that can be a vip managed by keepalived that floats between machines. | 19:10 |
| kfox1111 | (or pacemaker, or whatever your failover tool of choice) | 19:10 |
| sdake | kfox1111 small sips at a time ) | 19:10 |
| sdake | just tyring to get AIO rolling | 19:11 |
| kfox1111 | k | 19:11 |
| sdake | so really the cloud.yaml in the review is correct as is (with the sed operation) | 19:11 |
| sdake | for AIO? | 19:11 |
| rwellum | sdake: seems like it as per kfox1111 | 19:12 |
| rwellum | My main issue is that I saw the 192.x address in your example and assumed it was an external interface | 19:13 |
| kfox1111 | just glancing over it, seems ok. | 19:13 |
| rwellum | So I used my external interface instead of mgmt | 19:13 |
| sdake | rwellum ya that is bound to cause problems | 19:13 |
| sdake | rwellum i'd also recommend not ssigning an ip addres to your external neutron interface | 19:13 |
| kfox1111 | sdake: maybe the docs should use variables and @VARNAME@ substituations. | 19:14 |
| kfox1111 | then its set explicitly all at once at the top. | 19:14 |
| kfox1111 | would probably be less confusing. | 19:14 |
| kfox1111 | external_vip=xxxxxx | 19:14 |
| kfox1111 | sed -i "s/@external_vip@/$external_vip/" .... | 19:14 |
| kfox1111 | etc. | 19:14 |
| sdake | kfox1111 without an explination of what external_vip is (as you ahve provided) that would be more confusing :) | 19:15 |
| sdake | kfox1111 baby steps = AIO guide | 19:15 |
| kfox1111 | sdake: less error prone then. | 19:15 |
| sdake | if peopel can get deployment running AIO they can evaluate kolla-kubernetes as it stands today | 19:15 |
| sdake | if they can't get deployment running becasue they have the same questions about all these variables I have, then they will crater and may not eval at all | 19:16 |
| *** lrensing has quit IRC | 19:16 | |
| kfox1111 | sdake: but if they cant run it at their site becausee they cant find the 72 places ips are scatterd about the docs and set them right, the same issue will happen. | 19:17 |
| kfox1111 | just my 2cents. | 19:17 |
| sdake | kfox1111 agreed - this doc needs further work | 19:18 |
| sdake | kfox1111 i'd liek to mostly rm -rf docs/* | 19:18 |
| sdake | and start over | 19:18 |
| sdake | this is that :) | 19:18 |
| sdake | since the docs are mostly bitrot | 19:18 |
| sdake | note i said mostly - some things should stay | 19:18 |
| sdake | we started over the code with helm but never started ove the docs | 19:19 |
| sdake | doing that now | 19:19 |
| kfox1111 | I've already stated my beliefs around the docs. not going to do so again. | 19:19 |
| *** ccesario has quit IRC | 19:19 | |
| sdake | kfox1111 i guess I dont recall you having strong beliefs about them | 19:20 |
| sdake | would you mind repeating - i've got like 20 things going on related to docs efforts and have trouble maintaining context | 19:20 |
| kfox1111 | copy paste nightmare today. not tested. | 19:20 |
| kfox1111 | sorry. not gated. | 19:21 |
| sdake | oh right | 19:21 |
| sdake | i do recall this | 19:21 |
| sdake | copy and paste is good - lets people get going | 19:21 |
| kfox1111 | shell scripts are good. | 19:21 |
| sdake | we can simplify that into a contianer implementation | 19:21 |
| kfox1111 | they can copy paste if they care. they can ignore if they don't. | 19:21 |
| kfox1111 | and its gateable. | 19:21 |
| *** nathharp has quit IRC | 19:21 | |
| kfox1111 | copy paste from text, bad. | 19:21 |
| *** bswartz has quit IRC | 19:22 | |
| sdake | we can't pbulish shell scripts on docs.oo | 19:22 |
| sdake | that is a key key thing we should be doing | 19:23 |
| sdake | is pubishng docs on docs.oo | 19:23 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 19:23 |
| sdake | rather then say "go cut and paste this shelll script" | 19:23 |
| sdake | but I understand your position | 19:23 |
| sdake | ok rwellum - you have enough to get going now? | 19:24 |
| rwellum | Yeah 'ish | 19:24 |
| rwellum | That 8080 fix still doesn't work all the way. eanylin second attempt works better but fails during secret gen. | 19:24 |
| sbezverk | kfox1111: updated PS, let see what happen | 19:25 |
| sbezverk | rwellum: are you talking about aipserver? | 19:25 |
| rwellum | Still not sure if I need two working bridges or not | 19:25 |
| sdake | rwellum you shouldnl't need any bridge i think | 19:25 |
| sdake | just two interfaces in your vm | 19:25 |
| sdake | one interface is your management interface, the other being your neutron interface (without an IP) | 19:25 |
| rwellum | sbezverk: the work-around to the 'cannot connect' to kuebadm credentials issue | 19:27 |
| *** bmace has joined #openstack-kolla | 19:28 | |
| sbezverk | rwellum: do you mind to share yot setup in webex? I have never seen failure after following steps I provided.. | 19:30 |
| rwellum | sbezverk: actually I got it up and working this morning, just some confusion over the external_vip - but I think what I had orginally was fine based on sdake and kfox1111 | 19:31 |
| *** lrensing has joined #openstack-kolla | 19:32 | |
| rwellum | So I am rebuilding back to that | 19:32 |
| rwellum | I had an issue launching a vm - the final step if you like - might need help there | 19:32 |
| sdake | rwellum i also had an issue there | 19:32 |
| sdake | there is some report that the placemenet api source container is busted | 19:32 |
| sbezverk | sdake: 3.0.0 binary | 19:33 |
| sbezverk | is busted for sure | 19:33 |
| sdake | sbezverk i know that | 19:33 |
| sbezverk | 4.0.0 source works | 19:33 |
| sdake | sbezverk there is no placement -api for 3.0.0 images | 19:34 |
| sdake | and we dont backport features | 19:34 |
| *** ksumit has joined #openstack-kolla | 19:34 | |
| kfox1111 | https://review.openstack.org/#/c/452850/ is good for a second. | 19:34 |
| *** rwallner has quit IRC | 19:35 | |
| *** rwallner has joined #openstack-kolla | 19:36 | |
| *** nathharp has joined #openstack-kolla | 19:37 | |
| *** manheim_ has joined #openstack-kolla | 19:37 | |
| *** rwallner has quit IRC | 19:42 | |
| sbezverk | rwellum: when you get to that point, please ping me | 19:42 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Remove services/cinder https://review.openstack.org/452850 | 19:44 |
| *** nathharp has quit IRC | 19:44 | |
| *** rwallner_ has joined #openstack-kolla | 19:46 | |
| *** oanson has quit IRC | 19:47 | |
| *** rwallner_ has quit IRC | 19:51 | |
| rwellum | Ack sbezverk | 19:52 |
| *** nathharp has joined #openstack-kolla | 19:53 | |
| *** oanson has joined #openstack-kolla | 19:55 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Placement support in nova service chart. https://review.openstack.org/453846 | 19:56 |
| *** nathharp has quit IRC | 19:56 | |
| kfox1111 | sbezverk: please have a look at that review. still needs some deps added, but should be a way to move forward without dropping 2/3 support. | 19:56 |
| *** harlowja has joined #openstack-kolla | 19:58 | |
| kfox1111 | harlowja: hi. :) | 19:58 |
| harlowja | oh hi | 19:58 |
| * harlowja noticied my irc client wasn't on, lol | 19:58 | |
| kfox1111 | :) | 19:58 |
| harlowja | kfox1111 whats up | 19:58 |
| kfox1111 | not much. still plugging along. u? | 19:58 |
| harlowja | same | 19:58 |
| kfox1111 | inc0: sdake: https://review.openstack.org/#/c/453384/ please. | 20:01 |
| *** pcaruana has quit IRC | 20:01 | |
| sbezverk | kfox1111: all green :-) | 20:01 |
| kfox1111 | yup. :) | 20:01 |
| sdake | kfox1111 30 mins meeting then will review if inc0 hasn't got to it | 20:01 |
| *** magicboiz has quit IRC | 20:01 | |
| kfox1111 | hopefully taht will get us back to the stability we had before 1.6. | 20:01 |
| sbezverk | kfox1111: yep.. about the placement, looks good, just a small request about var name.. | 20:02 |
| inc0 | done kfox1111 | 20:02 |
| kfox1111 | inc0: thx. | 20:03 |
| kfox1111 | sbezverk: fixed. :) | 20:04 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Placement support in nova service chart. https://review.openstack.org/453846 | 20:04 |
| sbezverk | kfox1111: nice, thanks, now even I can understand it ;) | 20:05 |
| kfox1111 | helm 2.3 is imminent. | 20:05 |
| kfox1111 | in the last 24 hours, like dozens of issues have closed. down to 5 open now. | 20:05 |
| *** manheim_ has quit IRC | 20:06 | |
| sbezverk | kfox1111: I hope going to helm 2.3 will not be as distructive as going to kube 1.6 ;) | 20:06 |
| kfox1111 | +1 | 20:06 |
| kfox1111 | https://review.openstack.org/#/c/446274/'s good for reviews I think. | 20:07 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Optimizing multinode job at the gate https://review.openstack.org/453384 | 20:09 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Fernet Token Support https://review.openstack.org/446274 | 20:12 |
| *** magicboiz has joined #openstack-kolla | 20:14 | |
| *** satyar has quit IRC | 20:20 | |
| rwellum | sbezverk: I'm at the point where runonce has run successfully and I can create a demo vm, floating ip etc | 20:35 |
| *** eaguilar has quit IRC | 20:35 | |
| sbezverk | rwellum: nice | 20:36 |
| kfox1111 | fasinating: https://review.openstack.org/#/c/451525/ | 20:36 |
| kfox1111 | rwellum: congrats. :) | 20:36 |
| rwellum | thanks | 20:36 |
| kfox1111 | I wonder if that revew could be tweaked to push out fluentd formatted messages directly. | 20:36 |
| rwellum | sbezverk - the script recommends: proceed? https://www.irccloud.com/pastebin/oOVn3BEV/ | 20:37 |
| kfox1111 | though... I guess that ps along with the kubernetes systemd driver and some kind of fluentd shipper would do the job too. | 20:37 |
| sbezverk | rwellum: net id used in the command line is yours right? | 20:38 |
| sbezverk | kfox1111: easy. fluentd can get and accumulate all that info from different sources | 20:38 |
| rwellum | sbezverk: it's what 'init-runonce' spat out at the end - assuming it's correct? | 20:38 |
| kfox1111 | oh, nice. they finally released fluent-bit. 0.11. | 20:38 |
| kfox1111 | sbezverk: fluentd directly is a bit complicated as its needs log rolling and some other bits. | 20:39 |
| kfox1111 | if the logger in python itself could send out directly, it would simplify things a lot. | 20:39 |
| kfox1111 | enabling proper log watching for rotation broke our site's neutron for a while. :/ | 20:39 |
| sbezverk | kfox1111: I see | 20:40 |
| kfox1111 | I really really really want to see a log shipper build in log rotation and log pruning all into one solution. | 20:40 |
| kfox1111 | rather then the mismatched seperate tools we have today. :/ | 20:40 |
| sbezverk | rwellum: I have not used that script, but try it | 20:40 |
| sbezverk | kfox1111: what is log shipper? | 20:41 |
| sbezverk | I mean I have not heard that name before | 20:41 |
| kfox1111 | fluentd, fluent-bit, filebeat, logstashd, etc are all shippers. | 20:41 |
| kfox1111 | they read the logs from some logfile and ship them somewhere else. | 20:41 |
| kfox1111 | fluentd and logstash are also processors. | 20:42 |
| kfox1111 | they can manipulate the logs before shipping them elsewhere. | 20:42 |
| rwellum | sbezverk: https://www.irccloud.com/pastebin/tFhFZnPH/ | 20:43 |
| sbezverk | kfox1111: ah ok, so it was just a generic name, I thought there is a product with this name | 20:43 |
| kfox1111 | sbezverk: ah. sorry, yeah. generic term for all the similar tools. | 20:43 |
| sbezverk | rwellum: untill you see error, so far so good ;) | 20:43 |
| rwellum | sbezverk, kfox1111 so the guide states: https://www.irccloud.com/pastebin/YWV8Ehsz/ | 20:44 |
| rwellum | Doesn't seem at all correct to me. | 20:44 |
| kfox1111 | rwellum: yeah, thats wrong. | 20:45 |
| sbezverk | rwellum: hm looks ok | 20:45 |
| rwellum | :) | 20:45 |
| sbezverk | kfox1111: why? | 20:45 |
| kfox1111 | the optput of openstack floating ip create public1 is not only a uuid. | 20:45 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/451018 | 20:45 |
| sbezverk | it creates floating ip, return it is id and then adds it to vm?? | 20:45 |
| rwellum | sbezverk: it returns a whole table of info | 20:46 |
| kfox1111 | sbezverk: rwellum: see ttests/bin/basic_tests.sh" | 20:46 |
| kfox1111 | sbezverk: rwellum: see tests/bin/basic_tests.sh | 20:46 |
| sbezverk | kfox1111: right right, it is missing awk | 20:46 |
| kfox1111 | FIP=$(openstack floating ip create external -f value -c floating_ip_address) | 20:46 |
| sbezverk | or -c -f in command line | 20:47 |
| rwellum | openstack server add floating ip demo1 $(openstack floating ip create public1 -f value -c floating_ip_address) | 20:48 |
| kfox1111 | demo1 or public1? | 20:49 |
| sbezverk | it should be number net | 20:49 |
| sbezverk | in create floating ip command | 20:49 |
| rwellum | 'number net'? sbezverk | 20:50 |
| sbezverk | rwellum: uuid or name will work | 20:50 |
| sbezverk | in your case public1 is good | 20:51 |
| kfox1111 | rwellum: oh. sorry. I didn't see the nesting. | 20:51 |
| kfox1111 | that should work. | 20:51 |
| rwellum | https://www.irccloud.com/pastebin/5TQTx205/ | 20:52 |
| rwellum | I think I'm confused.... | 20:52 |
| *** rwsu has quit IRC | 20:53 | |
| kfox1111 | oh. | 20:53 |
| kfox1111 | whats the name of the vm? | 20:53 |
| kfox1111 | oh.... | 20:54 |
| kfox1111 | nova show demo1 | 20:54 |
| kfox1111 | I'm guessing its not scheduled yet. | 20:54 |
| kfox1111 | cant add a fip to a vm that hasn't got an ip allocated to it. | 20:55 |
| rwellum | https://www.irccloud.com/pastebin/3XLPtmFN/ | 20:55 |
| *** vhosakot_ has quit IRC | 20:55 | |
| rwellum | vm state is error | 20:55 |
| kfox1111 | see tests/bin/basic_tests.sh for some of the stuff we do in the gate. it has pauses in there to deal with letting things start up before proceeding on. | 20:55 |
| kfox1111 | yeah. that would be the problem. | 20:55 |
| kfox1111 | that implies the scheduler's not workign... | 20:56 |
| kfox1111 | is this a 2, a 3 or a 4.x deployment? | 20:56 |
| kfox1111 | (4.x is rather lightly tested at the moment) | 20:56 |
| rwellum | Sorry kfox1111 - just following the guide - I'm not sure. | 20:56 |
| kfox1111 | docker ps | 20:56 |
| kfox1111 | and see what image versions its using. | 20:56 |
| rwellum | Hmm says docker not running | 20:57 |
| kfox1111 | hmm... sudo? | 20:57 |
| rwellum | https://www.irccloud.com/pastebin/jivZzLH3/ | 20:58 |
| kfox1111 | wow... its using sha's now... how not helpful. :/ | 21:00 |
| sbezverk | rwellum: can you get nova scheduler and conductor logs? | 21:00 |
| *** jrobinson has joined #openstack-kolla | 21:00 | |
| *** jtriley has quit IRC | 21:00 | |
| kfox1111 | kubectl get pods nova-compute-zjx8f -o yaml | grep image | 21:01 |
| kfox1111 | kubectl get pods nova-compute-zjx8f -n kolla -o yaml | grep image | 21:01 |
| rwellum | sudo kubectl log nova-scheduler-0 -n kolla - like that sbezverk | 21:02 |
| rwellum | sbezverk https://www.irccloud.com/pastebin/dZbqkePp/ | 21:03 |
| sbezverk | rwellum: nope kubectl exec nova-scheduler-0 -n kolla -- cat /var/log/kolla/nova/nova-scheduler.log | 21:03 |
| sbezverk | and same thing for conductor | 21:03 |
| *** athomas has quit IRC | 21:03 | |
| sdake | rwellum re secret generation failing | 21:05 |
| sdake | rwellum that isn't totally surprising - the secret generator python code is probably not exporting the envrionment to kubectl | 21:06 |
| sbezverk | sdake: right it expects to have current namespace set to kolla | 21:07 |
| sdake | sbezverk nah - what i mean is kubectl needs KUBE_CONFIG env variable set | 21:07 |
| sdake | hen failed later on secret generation: | 21:08 |
| sdake | Command 'kubectl create secret generic congress-database-password --from-literal=password=SM35oqzKO7J6uKodeCLCsz5dBbaMr3O6Au2nPgsz --namespace=kolla' returned non-zero exit status 1 | 21:08 |
| sdake | The connection to the server localhost:8080 was refused - did you specify the right host or port? | 21:08 |
| sdake | Command 'kubectl create secret generic panko-database-password --from-literal=password=2OfzleXR4sABcP9LH8GaKHpj0DQctXxfhbHVbY9h --namespace=kolla' returned non-zero exit status 1 | 21:08 |
| sdake | Then I did: | 21:08 |
| sdake | sudo sed -i 's/insecure-port=0/insecure-port=8080/' /etc/kubernetes/manifests/kube-apiserver.yaml | 21:08 |
| sdake | And secret generation worked again. | 21:08 |
| sbezverk | sdake: it is so much easier to use $HOME/.kube/config | 21:08 |
| *** mgoddard has joined #openstack-kolla | 21:08 | |
| kfox1111 | sdake: what sbezverk said. :) | 21:08 |
| sdake | this is what we have in our docs: Load the kubedm credentials into the system:: | 21:09 |
| sdake | sudo cp /etc/kubernetes/admin.conf $HOME/ | 21:09 |
| sdake | sudo chown $(id -u):$(id -g) $HOME/admin.conf | 21:09 |
| sdake | export KUBECONFIG=$HOME/admin.conf | 21:09 |
| sdake | are you saying that sudo cp should be to $HOME/.kube/config? | 21:09 |
| kfox1111 | yeah. | 21:09 |
| rwellum | http://paste.openstack.org/show/605565/ and http://paste.openstack.org/show/605567/ - sbezverk | 21:10 |
| rwellum | sdake: we've already made that comment in the review | 21:10 |
| kfox1111 | sdake: the dev env, what version? | 21:10 |
| kfox1111 | sorry, not dev env. your doc. | 21:10 |
| kfox1111 | 2, 3 or 4? | 21:10 |
| sdake | https://review.openstack.org/#/c/447356/ | 21:11 |
| sdake | version 4 of images | 21:11 |
| kfox1111 | did you do the cell stuff? | 21:11 |
| sdake | indeed | 21:11 |
| sdake | it is done last | 21:11 |
| rwellum | https://www.irccloud.com/pastebin/z2JPvX7N/kfox1111 | 21:11 |
| kfox1111 | sdake: have you gotten al the way through sshing into a workign vm? | 21:12 |
| sdake | kfox1111 nope - waiting on everyont to finish barfing on the doc before i rev it | 21:13 |
| sdake | ;) | 21:13 |
| sdake | and then test it | 21:13 |
| sdake | its pretty close | 21:13 |
| sdake | last time i started a vm it entered FAILED state | 21:13 |
| kfox1111 | ah. so testing has gotten up to service starting. not vm's. | 21:13 |
| kfox1111 | rwellum: ok. so where your at is the farthest anyone's gotten. | 21:13 |
| sdake | eanlin has got to the point he can use novanovnc to ping / curl out of vms | 21:13 |
| sdake | rwellum you can ssh into the vm? | 21:13 |
| kfox1111 | sdake: its not scheduling for him. | 21:14 |
| kfox1111 | hence the wquestions about cells. | 21:14 |
| sdake | the edocs are pretty clear , you have to wait for nova copute to enter running state | 21:14 |
| sdake | egonzales verified it | 21:14 |
| kfox1111 | yeah, we really need a job to add more nodes after its been stood up though. | 21:15 |
| kfox1111 | so, that may be the problem though. | 21:15 |
| kfox1111 | not sujre how to tell if a nova-compute is bound to a cell. | 21:15 |
| kfox1111 | thats all new stuff to me. | 21:16 |
| sdake | egonzalz seemd to understnd it in great detail | 21:16 |
| sbezverk | kfox1111: it seems by default all VM belongs to cell1 | 21:16 |
| kfox1111 | sbezverk: yes, but only when added to the cell. | 21:17 |
| kfox1111 | which only happens if: | 21:17 |
| kfox1111 | 1. it exists before adding the cell1, or | 21:17 |
| kfox1111 | 2. running a tool manually afterwards. | 21:17 |
| sbezverk | right | 21:17 |
| kfox1111 | so rwellum might have a nova-compute in limbo. | 21:17 |
| sbezverk | I think it is only applicable to 1st cell | 21:17 |
| sbezverk | you can add other cells later (I have not tried) | 21:17 |
| sbezverk | and allocate compute resource to new cells | 21:18 |
| kfox1111 | yeah. though never figured out how to say what cell a compute should lland in. | 21:18 |
| kfox1111 | but I'm starting to really struggle with nova's decidion to need to go to cells. | 21:19 |
| kfox1111 | I have hit the wall myself with nova. much past 100 computes, it starts running into issues. | 21:19 |
| kfox1111 | but kubernetes does 5000 compute nodes at scale. | 21:19 |
| kfox1111 | with one 'cell'. | 21:19 |
| inc0 | kfox1111: one word - rabbitmq | 21:20 |
| kfox1111 | inc0: yeah. and mariadb. | 21:20 |
| *** rcherrueau has joined #openstack-kolla | 21:20 | |
| kfox1111 | and maybe python. | 21:20 |
| inc0 | and mariadb | 21:20 |
| inc0 | python nto so much | 21:20 |
| kfox1111 | and lack of gating on performance. | 21:20 |
| inc0 | non-optimal python - maybe | 21:20 |
| rwellum | kfox1111: how to tell if nova-compute is in limbo? Looks good from k8s pov | 21:21 |
| inc0 | you rarely end on mem/cpu performance | 21:21 |
| inc0 | it's io that kills you | 21:21 |
| kfox1111 | even still, nova should be able to do within one order of magitude, http://blog.kubernetes.io/2017/03/scalability-updates-in-kubernetes-1.6.html | 21:21 |
| inc0 | yeah but it would need to be rewritten | 21:21 |
| kfox1111 | rwellum: no clue. :/ never done ocata yet. :/ | 21:21 |
| rwellum | Fair enough. | 21:22 |
| kfox1111 | inc0: yes. but its less effort then them splitting everything out to cells I think. | 21:22 |
| inc0 | depends, if you want to remain upgradable, then no | 21:22 |
| inc0 | if you want to start fresh and re-thing certain ideas | 21:22 |
| inc0 | without need of backwards compatibility | 21:23 |
| inc0 | stuff could look totally differently | 21:23 |
| kfox1111 | inc0: they should be able to dump mysql state into etcd or something. | 21:23 |
| *** rwallner has joined #openstack-kolla | 21:23 | |
| kfox1111 | there should be a migration path. | 21:23 |
| kfox1111 | they have to do that with cells anyway. | 21:23 |
| inc0 | I don't think that mysql is necessary a problem here | 21:23 |
| *** goldyfruit has quit IRC | 21:23 | |
| rwellum | Have to head home, will be back this evening and will leave my cluster in this current state if anyone wants to jump on a take a look etc. | 21:23 |
| kfox1111 | I'm pretty sure its probably 50/50 between mariadb and rabbit. | 21:23 |
| inc0 | I mean mysql can handle millions of records | 21:23 |
| inc0 | if used correctly | 21:23 |
| kfox1111 | inc0: latency. | 21:24 |
| inc0 | etcd will have latency too | 21:24 |
| *** mgoddard has quit IRC | 21:24 | |
| kfox1111 | yeah, but key value is so much simpler to implement then acid compliant relational db's. | 21:24 |
| kfox1111 | thats gotta effect latency at some point. | 21:24 |
| sdake | rwellum i have a dentist appointment from 3-5pm | 21:25 |
| sdake | rwellum should be back tonight | 21:25 |
| inc0 | yes, but I thin problem at hand is more in sheer number of microservices that need to take part in spawning a vm | 21:25 |
| sbezverk | rwellum: did you manage to get logs? | 21:25 |
| inc0 | api->conductor->scheduler->conductor->compute | 21:25 |
| inc0 | and add http calls to neutron/cinder in between | 21:25 |
| inc0 | blocking http | 21:26 |
| inc0 | and keystone... | 21:26 |
| sdake | kfox1111 etcd is not all that scalable with HA enabled nor performanct | 21:26 |
| inc0 | every piece asks keystone for permission | 21:26 |
| sdake | and its a key value store not a ACID database | 21:26 |
| kfox1111 | inc0: that too. | 21:26 |
| kfox1111 | sdake: scalable enough for 5000 nodes and 150,000 pods. good enough for me. :) | 21:27 |
| kfox1111 | thats way less management though then trying to manage the same number of nodes / vms then with nova. | 21:27 |
| inc0 | but again, it's design of k8s not databases they use | 21:27 |
| kfox1111 | you'd have to have dozens of cells I'm afraid. | 21:27 |
| *** rwallner has quit IRC | 21:27 | |
| inc0 | but agree, I wouldnt use relational db myself today for software like that | 21:28 |
| kfox1111 | inc0: yeah. thats certainly part of it. | 21:28 |
| inc0 | lessons learned;) | 21:28 |
| inc0 | google knows how to write scallable software | 21:28 |
| inc0 | we know that too...today | 21:29 |
| sdake | i analyzed the perfomrnace ofa 123 node cluster running | 21:29 |
| inc0 | after all the lessons;) | 21:29 |
| sdake | 40% cpu utilization consumed by the i40 driver | 21:29 |
| *** Manheim has quit IRC | 21:29 | |
| inc0 | sometimes I'd love to just start fresh | 21:30 |
| kfox1111 | I think thats one of the things that potentially makes openstack great. | 21:30 |
| kfox1111 | its greatest streangth is its standard api's. | 21:30 |
| inc0 | not to solve this one | 21:30 |
| kfox1111 | those don't strictly need to continue to use the same code base to be functional. | 21:30 |
| inc0 | but you need to follow same API | 21:31 |
| inc0 | and that's issue | 21:31 |
| kfox1111 | nova api could retarget onto k8s directly and use k8s's scheduler. | 21:31 |
| kfox1111 | no one would be the wiser. | 21:31 |
| inc0 | k8s scheduler isn't much better | 21:31 |
| kfox1111 | and would scale much larger, emdediately. | 21:31 |
| inc0 | is it scheduler that dies first tho? | 21:32 |
| inc0 | and not really it couldn't as scheduler communicates via amqp | 21:32 |
| kfox1111 | nova's falls over at about 300 nodes. | 21:32 |
| inc0 | and it's rabbit to die | 21:32 |
| *** manheim has joined #openstack-kolla | 21:32 | |
| inc0 | also cells is bad idea imho because well...they don't work with neutron well | 21:33 |
| kfox1111 | potato patato? if it takes a architectural rewrite to get the scedular decoupled from rabbit to make it scale, it may just be easier to target k8s's directly and not need to provide one at all. | 21:33 |
| *** signed8bit has joined #openstack-kolla | 21:33 | |
| inc0 | and since quotas was that much of a pain to synchronize...imagine cells | 21:33 |
| kfox1111 | that too. neutron needs cells too before its really viable. | 21:33 |
| kfox1111 | unless neutron can scale and nova can't, so cells are the only way. | 21:34 |
| kfox1111 | yeah. :/ | 21:34 |
| inc0 | that part I have no info...in fact we asked this exact question in PTG | 21:34 |
| kfox1111 | I still see nova's statistics table get out of sync with reality. :/ | 21:34 |
| kfox1111 | buggy. :/ | 21:34 |
| inc0 | another thing that k8s did correctly is that they built on top of existing sdn controller like calico | 21:39 |
| inc0 | instead of writing neutron | 21:39 |
| inc0 | but that's because this thing existed;) | 21:39 |
| sbezverk | inc0: I thought k8s was way before any calico ;) | 21:39 |
| inc0 | I'm pretty sure calico is older | 21:40 |
| sbezverk | inc0: it was running on flannel | 21:40 |
| kfox1111 | calico is older I think. | 21:40 |
| sbezverk | inc0: if you count 10 years while k8s was under google umbrella, I doubt ;) | 21:40 |
| kfox1111 | sbezverk: k8s isn't that old. borg is. | 21:41 |
| inc0 | well, borg != k8s | 21:41 |
| inc0 | and I'm pretty damn sure it's not even close | 21:41 |
| inc0 | they might've used some ideas | 21:41 |
| kfox1111 | k8s is as nice as it is, because it doesn't have a lot of legacy stuff from borg. :) | 21:41 |
| kfox1111 | labels for example are way better in k8s then in borg I hear. | 21:41 |
| kfox1111 | k8s is borg 3.x from what I gather. | 21:42 |
| kfox1111 | borg was first. there was a second attempt that failed, and k8s is third. | 21:42 |
| sbezverk | kfox1111: reminds me Agent Smith monolog in Matrix ;) | 21:42 |
| kfox1111 | 3.x releases are often much better. :) | 21:43 |
| kfox1111 | 1.x you think you know what your doing but really have no clue. | 21:43 |
| kfox1111 | 2.x you know what you did wrong and think you know how to fix it. then realize there was a bit more you didn't understand. | 21:43 |
| kfox1111 | by 3, you usually have it figured out. | 21:43 |
| sbezverk | kfox1111: :) | 21:43 |
| inc0 | if we would rewrite openstack from scrach today we'd end up with great project that nobody will use | 21:44 |
| sbezverk | mental note, start versioning from 3.X ;) | 21:44 |
| *** manheim has quit IRC | 21:44 | |
| kfox1111 | sbezverk: some folks to that... operators don't look highly on that kind of behavior. ;) | 21:44 |
| inc0 | kolla-k8s is 2.0 in this nomenclature | 21:44 |
| inc0 | kolla-ansible is kolla 3.0;) | 21:44 |
| inc0 | 1 - k8s, 2 - compose, 3 - ansible | 21:44 |
| kfox1111 | 2 or 4. | 21:45 |
| kfox1111 | depending on how you look at it. | 21:45 |
| kfox1111 | yeah. | 21:45 |
| inc0 | but I don't think kolla-k8s is evolution of kolla-ansible | 21:45 |
| inc0 | as we're not trying to fix kolla-ansible problems with kolla-k8s | 21:45 |
| kfox1111 | I kind of disagree with that. ;) but I'm biased, so you can ignore me. | 21:46 |
| inc0 | I refuse to say that ansible is problem because it's ansible:P | 21:46 |
| kfox1111 | config's something that I think is kind of broken in kolla-ansible and being fixed in kolla-kubernetes. | 21:46 |
| inc0 | how so? | 21:46 |
| *** fooliouno has joined #openstack-kolla | 21:47 | |
| kfox1111 | kolla-ansible is fairly rigid around architectural choices. | 21:47 |
| kfox1111 | like, one mariadb or one rabbit. | 21:47 |
| inc0 | well, technically not really | 21:47 |
| sbezverk | kfox1111: another thing is fixed inventory | 21:47 |
| inc0 | I mean you can go crazy if you want | 21:47 |
| kfox1111 | I'd like a config that lets me do one rabbit per opentack service, with a second rabbit setup for metering. | 21:47 |
| fooliouno | kfox1111: Is it theoretically possible to run ODL on kolla-k8s with Flannel as an overlay network? | 21:48 |
| kfox1111 | and when I say want, I really mean, need. :) | 21:48 |
| inc0 | for i in service; kolla-ansible deploy -t mariadb -i dedicated_inventory; | 21:48 |
| fooliouno | Or does ODL need to replace Flannel | 21:48 |
| kfox1111 | fooliouno: theoretically, yeah. | 21:48 |
| inc0 | and override mariadb_address in globals | 21:48 |
| sbezverk | fooliouno: kfox1111: is there ODL cni plugin? | 21:48 |
| kfox1111 | inc0: overriding it for each service how? | 21:48 |
| inc0 | ahh well, good point | 21:49 |
| inc0 | but that's easy fix | 21:49 |
| inc0 | literally one commit | 21:49 |
| fooliouno | sbezverk: Not that I am aware of. I am trying to bring up an ODL container on kolla-k8s and test it. | 21:49 |
| sbezverk | do not remember seeing it | 21:49 |
| inc0 | just nobody wrote it | 21:49 |
| *** manheim has joined #openstack-kolla | 21:49 | |
| inc0 | add nova_mariadb = {{ mariadb_address }} | 21:50 |
| kfox1111 | inc0: yeah. I'm trying to fix that kind of thing out of the box in kolla-kubernetes. | 21:50 |
| inc0 | and that's it | 21:50 |
| kfox1111 | not insurmountable at all on the kolla-ansible side. | 21:50 |
| fooliouno | sbezverk: I waned to see if ODL can co-exist with Flannel before I go about messing with it. | 21:50 |
| inc0 | yeah, cool, waht I'm saying this isn't fixing ansible | 21:50 |
| sbezverk | fooliouno: do you want to use it just for kube cluster or for openstack network plugin? | 21:50 |
| kfox1111 | but jsut stuff we've learned from the kolla-ansible side that are fixing in implementing kolla-kubernetes. | 21:50 |
| inc0 | and it's not design issue with kolla-ansible | 21:50 |
| inc0 | it's just commit nobody made | 21:50 |
| kfox1111 | yeah. | 21:50 |
| inc0 | if you want this, really quick to implement | 21:51 |
| inc0 | again, nobody did that because nobody wanted it;) | 21:51 |
| fooliouno | sbezverk: I want to use ODL for openstack network only | 21:52 |
| inc0 | imho kolla-ansible is really flexible as it is today | 21:52 |
| inc0 | kolla-k8s will need some time to arrive to this kind of flexibility | 21:52 |
| kfox1111 | inc0: does it have an easy way to support multiple, different host aggregate computes? | 21:52 |
| inc0 | to me these are 2 different modes of deployment, neither better than other | 21:52 |
| kfox1111 | configured seperately? | 21:53 |
| inc0 | yes | 21:53 |
| inc0 | in inventory you can override variables | 21:53 |
| sbezverk | fooliouno: Oh, ok. then you will need to do some legwork, especially in configuration area as we generate config only for ovs/ml2 | 21:53 |
| kfox1111 | inc0: hmm... ok. | 21:53 |
| kfox1111 | so, thats pretemplated values that then render into the config? | 21:54 |
| inc0 | for example you can have host1 enable_openvswitch=false | 21:54 |
| inc0 | config, playbook logic, all that | 21:54 |
| fooliouno | sbezverk: Willing to do that if its feasible. Any pointers on where to start looking. | 21:54 |
| inc0 | good example would be to have different interface name per hosts | 21:54 |
| kfox1111 | inc0: cool. | 21:54 |
| sbezverk | inc0: when you know you setup, yes, but let's say you add new nodes and want to move some roles around, how easy it would be? | 21:55 |
| inc0 | what you do is specify neutron_external_interface=eth0 in inventoryu in same line as host declaration | 21:55 |
| sbezverk | with k8s it is really really easy | 21:55 |
| inc0 | (there are other ways to specify too) | 21:55 |
| inc0 | one caveat - you don't define it in globals then because what is in globals takes precedence | 21:55 |
| kfox1111 | inc0: interesting. I didn't really think of using inventory that way. | 21:55 |
| mnaser | we use it that way ^ | 21:56 |
| kfox1111 | thats good and bad I guess. can you define vars at a group level, | 21:56 |
| mnaser | to do very questionable things | 21:56 |
| kfox1111 | and tag them onto nodes? | 21:56 |
| mnaser | like enable_ceph for rgw only | 21:56 |
| inc0 | yeah kfox1111 | 21:56 |
| kfox1111 | cool. :) | 21:56 |
| inc0 | sbezverk: add node is easy | 21:56 |
| inc0 | add it to inventory and run deploy | 21:56 |
| kfox1111 | then its much closer then I thought. | 21:56 |
| mnaser | its useful for brownfield things | 21:56 |
| inc0 | we don't really use "host roles" per se in ansible | 21:56 |
| inc0 | you can specify this all the way down to services runing | 21:57 |
| sbezverk | inc0: ok I see | 21:57 |
| inc0 | wanna run api on copute node for reasons? add this compute node to nova-api group and run deploy | 21:57 |
| *** mbruzek has quit IRC | 21:58 | |
| inc0 | kfox1111: I dare you to give me use case and I'll either show you how to do it or how to fix ansible to do it;) | 21:58 |
| inc0 | like mariadb separated thing, agree not easy now, simple fix tho | 21:59 |
| kfox1111 | inc0: I may take you up on that some day. :) | 21:59 |
| inc0 | I'm really happy about kolla-ansible tbh | 21:59 |
| inc0 | with all issues we had with ansible gplv3, it's treating us really well all in all | 21:59 |
| *** shashank_t_ has quit IRC | 22:00 | |
| *** shashank_t_ has joined #openstack-kolla | 22:00 | |
| inc0 | one thing we have problems with today due to ansible is rolling upgrade | 22:00 |
| inc0 | as it's not possible in ansible as of today to specify "I want to do this task at same time and this task in rolling" | 22:01 |
| inc0 | but still possible if you do --forks 1 | 22:01 |
| inc0 | then it will do stuff on one host at the time | 22:01 |
| inc0 | not ideal | 22:01 |
| *** magicboiz has quit IRC | 22:02 | |
| *** magicboiz has joined #openstack-kolla | 22:02 | |
| kfox1111 | ah. | 22:03 |
| mnaser | inc0 there is ia hack | 22:04 |
| *** shashank_t_ has quit IRC | 22:04 | |
| mnaser | i rememeber using it | 22:05 |
| mnaser | one second | 22:05 |
| *** harlowja has quit IRC | 22:05 | |
| mnaser | inc0 https://groups.google.com/d/msg/ansible-project/rBcWzXjt-Xc/CDW2FdlqRh0J | 22:09 |
| *** jrobinson has quit IRC | 22:12 | |
| *** fooliouno has quit IRC | 22:15 | |
| *** lucasxu has quit IRC | 22:15 | |
| *** jrobinson has joined #openstack-kolla | 22:18 | |
| *** manheim has quit IRC | 22:25 | |
| *** gfidente|afk has quit IRC | 22:26 | |
| kfox1111 | bummer... incubator/elasticserach is not only bit roten, but too big for minikube testing. | 22:28 |
| kfox1111 | :/ | 22:28 |
| *** rwsu has joined #openstack-kolla | 22:32 | |
| *** chas has joined #openstack-kolla | 22:36 | |
| *** chas has quit IRC | 22:41 | |
| *** rcherrueau has quit IRC | 22:41 | |
| *** krtaylor has quit IRC | 22:45 | |
| *** jascott1_ has joined #openstack-kolla | 22:54 | |
| inc0 | so I ahve funny problem with one of vms | 22:56 |
| inc0 | vim, top and dmesg hangs, what's common between these 3? | 22:56 |
| *** signed8b_ has joined #openstack-kolla | 22:56 | |
| *** signed8b_ has quit IRC | 22:57 | |
| *** chas has joined #openstack-kolla | 22:57 | |
| *** jascott1_ has quit IRC | 22:58 | |
| ksumit | The manila-data image is broken. I tried building it from source on both centos and ubuntu, but it fails to build and deploy. | 22:58 |
| *** signed8bit has quit IRC | 23:00 | |
| inc0 | ksumit: fails how? | 23:00 |
| ksumit | Following the quick start guide, I get the following message after the 'kolla-build' command: | 23:01 |
| *** chas has quit IRC | 23:01 | |
| ksumit | INFO:kolla.image.build:=========================== | 23:01 |
| ksumit | INFO:kolla.image.build:Images that failed to build | 23:01 |
| ksumit | INFO:kolla.image.build:=========================== | 23:01 |
| ksumit | ERROR:kolla.image.build:manila-data Failed with status: error | 23:01 |
| inc0 | ksumit: check logs above | 23:02 |
| inc0 | there will be error somewhere | 23:02 |
| inc0 | you can try to run kolla-build manila-data | 23:02 |
| inc0 | to build only this image | 23:02 |
| inc0 | afk, meeting | 23:03 |
| ksumit | For Centos at least, it seems to be related to https://bugzilla.redhat.com/show_bug.cgi?id=1426963 | 23:04 |
| openstack | bugzilla.redhat.com bug 1426963 in openstack-manila "missing manlia-data rpm when using Kolla build image" [Unspecified,Closed: notabug] - Assigned to tbarron | 23:04 |
| *** lrensing has quit IRC | 23:04 | |
| ksumit | "The command '/bin/sh -c echo 'binary not yet available for centos' && /bin/false' returned a non-zero code: 1" | 23:06 |
| *** lrensing has joined #openstack-kolla | 23:07 | |
| *** lamt has quit IRC | 23:23 | |
| *** lrensing has quit IRC | 23:23 | |
| *** lamt has joined #openstack-kolla | 23:24 | |
| *** lamt has quit IRC | 23:24 | |
| *** harlowja has joined #openstack-kolla | 23:25 | |
| *** lrensing has joined #openstack-kolla | 23:25 | |
| *** jtriley has joined #openstack-kolla | 23:29 | |
| *** rwellum has quit IRC | 23:31 | |
| *** chas has joined #openstack-kolla | 23:33 | |
| *** chas has quit IRC | 23:38 | |
| *** bswartz has joined #openstack-kolla | 23:39 | |
| *** lrensing has quit IRC | 23:45 | |
| *** lrensing has joined #openstack-kolla | 23:45 | |
| *** Pavo has joined #openstack-kolla | 23:48 | |
| sdake | sbezverk server has arrived | 23:49 |
| sdake | it sure is big | 23:49 |
| sdake | my wife said "protecting shareholder value I see" | 23:49 |
| kfox1111 | interesting... rkt is in minikube... | 23:49 |
| *** krtaylor has joined #openstack-kolla | 23:51 | |
| sdake | sup kfox1111 | 23:53 |
| *** lrensing has quit IRC | 23:57 | |
| sbezverk | sdake: so you enjoy the noise ;) | 23:58 |
| sdake | sbezverk i havent truend it on yet | 23:58 |
| sdake | hoepfullly its quiet | 23:58 |
| sdake | if its not I'm sending it back :) | 23:58 |
| sbezverk | dream on ;) | 23:58 |
| sdake | i got a model with two power supplies | 23:58 |
| sdake | that will probably be the loudest part of the server | 23:58 |
| *** alop has joined #openstack-kolla | 23:59 | |
| *** yingjun has joined #openstack-kolla | 23:59 | |
| *** lrensing has joined #openstack-kolla | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!