| *** salv-orlando has quit IRC | 00:05 | |
| *** vhosakot has quit IRC | 00:07 | |
| *** david-lyle has joined #openstack-kolla | 00:08 | |
| *** vhosakot has joined #openstack-kolla | 00:08 | |
| *** yingjun has joined #openstack-kolla | 00:09 | |
| *** jmccarthy has joined #openstack-kolla | 00:10 | |
| *** gfhellma has joined #openstack-kolla | 00:12 | |
| *** david-lyle has quit IRC | 00:15 | |
| *** jrobinson has joined #openstack-kolla | 00:16 | |
| *** lamt has quit IRC | 00:18 | |
| *** gfhellma has quit IRC | 00:20 | |
| *** haplo37 has quit IRC | 00:20 | |
| *** saneax is now known as saneax-_-|AFK | 00:22 | |
| *** goldyfruit has quit IRC | 00:24 | |
| *** dave-mccowan has joined #openstack-kolla | 00:25 | |
| *** haplo37 has joined #openstack-kolla | 00:29 | |
| *** jtriley has joined #openstack-kolla | 00:30 | |
| inc0 | ehh...ceilometer | 00:31 |
|---|---|---|
| inc0 | dunno jemcevoy | 00:31 |
| inc0 | that's out of my comfort zone | 00:31 |
| *** duonghq has joined #openstack-kolla | 00:33 | |
| duonghq | morning guys | 00:33 |
| inc0 | up already duonghq?:) | 00:33 |
| inc0 | hey | 00:33 |
| duonghq | ya | 00:33 |
| duonghq | just arrive my office | 00:34 |
| duonghq | thank you for start voting | 00:34 |
| inc0 | so far so good:) | 00:34 |
| inc0 | you've earned it | 00:34 |
| duonghq | :) last night I cannot help much, have we cleared critical bug? | 00:35 |
| inc0 | yeah | 00:36 |
| inc0 | we already tagged rc | 00:36 |
| inc0 | all of critical stuff is merged and fixed | 00:36 |
| inc0 | as far as I know | 00:36 |
| duonghq | nice | 00:36 |
| duonghq | inc0, as development of Docker can lead to something not backward compatible, I think we should have recommend max version of docker in qsg? | 00:39 |
| inc0 | duonghq: we did that in liberty when we were pinned | 00:40 |
| inc0 | so far I haven't seen anything that would lock us out of 17.03 | 00:40 |
| inc0 | besides docker-py change I think, but fix for it is already proposed | 00:40 |
| inc0 | anyway, if we hit a wall then yes, we pin max version until we teach kolla_docker module to use it | 00:41 |
| inc0 | that being said, docker-ee worries me | 00:41 |
| duonghq | sure, maybe it's all about docker-py's trouble | 00:41 |
| duonghq | yup, it's becoming many other FOSS project way | 00:42 |
| duonghq | *following | 00:42 |
| inc0 | yeah, but I'm worried that they'll lock out features from community edition | 00:43 |
| inc0 | they have nasty habit of doing stuff like that even before they introduced ee | 00:44 |
| inc0 | btw duonghq https://review.openstack.org/#/c/438813/ | 00:44 |
| inc0 | this is change I was talking about | 00:45 |
| duonghq | roger | 00:45 |
| *** manheim has quit IRC | 00:45 | |
| duonghq | what is your opinion about rkt? | 00:45 |
| *** zhurong has joined #openstack-kolla | 00:46 | |
| duonghq | inc0, this is my bug with 17.03: http://paste.openstack.org/show/601928/ | 00:47 |
| duonghq | when I downgrade to 1.12.x, it's fine | 00:48 |
| duonghq | not sure it's about docker or docker-py | 00:48 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 00:49 |
| inc0 | duonghq: so I have this on my todo list: check if we can run our containers on rkt | 00:52 |
| inc0 | I'd lie if I'd say I have lots of experience with it | 00:52 |
| *** david-lyle has joined #openstack-kolla | 00:54 | |
| *** MarginHu has joined #openstack-kolla | 00:54 | |
| *** david-lyle has quit IRC | 00:54 | |
| *** Jeffrey4l_ has quit IRC | 01:00 | |
| duonghq | inc0, but you have one hand on rkt? | 01:00 |
| *** schwicht has joined #openstack-kolla | 01:01 | |
| *** dave-mccowan has quit IRC | 01:02 | |
| *** ssurana has joined #openstack-kolla | 01:03 | |
| *** cuongnv has joined #openstack-kolla | 01:06 | |
| *** daidv has joined #openstack-kolla | 01:06 | |
| hogepodge | inc0: if I want to get rid of ironic-inspector in my all-in-one config, how do I do that? | 01:08 |
| duonghq | bpattewar, ping | 01:09 |
| hogepodge | inc0: it wants to start up dnsmasq, and I do not want that running, nor do I need server discovery and inspection. | 01:09 |
| duonghq | bpattewar_, ping | 01:10 |
| bpattewar | duonghq hi | 01:10 |
| *** rwsu has quit IRC | 01:11 | |
| *** zhubingbing_ has joined #openstack-kolla | 01:11 | |
| *** alop has quit IRC | 01:16 | |
| *** MasterOfBugs has quit IRC | 01:22 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 01:25 |
| *** unicell has quit IRC | 01:34 | |
| *** deadnull has quit IRC | 01:35 | |
| *** kbaegis has joined #openstack-kolla | 01:37 | |
| *** salv-orlando has joined #openstack-kolla | 01:37 | |
| *** salv-orlando has quit IRC | 01:41 | |
| *** kbaegis has quit IRC | 01:45 | |
| *** kbaegis has joined #openstack-kolla | 01:45 | |
| duonghq | kbaegis, so, currently you are using ubuntu for the cluster? | 01:45 |
| *** adrian_otto has quit IRC | 01:46 | |
| *** dave-mcc_ has joined #openstack-kolla | 01:50 | |
| *** david-lyle has joined #openstack-kolla | 01:52 | |
| *** bpattewar has quit IRC | 01:54 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot https://review.openstack.org/443300 | 01:55 |
| *** bpattewar_ has quit IRC | 01:58 | |
| *** tovin07_ has joined #openstack-kolla | 02:06 | |
| *** haplo37 has quit IRC | 02:12 | |
| *** trinaths1 has left #openstack-kolla | 02:17 | |
| *** haplo37 has joined #openstack-kolla | 02:21 | |
| *** erlon has quit IRC | 02:25 | |
| *** japestinho has joined #openstack-kolla | 02:28 | |
| *** salv-orlando has joined #openstack-kolla | 02:38 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 02:39 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible master: Modify the hosts of the post-deploy.yml playbook https://review.openstack.org/436822 | 02:41 |
| *** salv-orlando has quit IRC | 02:42 | |
| *** dave-mcc_ has quit IRC | 02:54 | |
| *** vhosakot has quit IRC | 03:00 | |
| *** daidv has quit IRC | 03:04 | |
| *** daidv has joined #openstack-kolla | 03:05 | |
| kbaegis | Did https://bugs.launchpad.net/kolla/+bug/1631072 end up actually getting fixed? | 03:06 |
| openstack | Launchpad bug 1631072 in kolla "iscsid container: mkdir /sys/kernel/config: operation not permitted" [High,Confirmed] | 03:06 |
| *** gmann has joined #openstack-kolla | 03:09 | |
| *** sudipto has joined #openstack-kolla | 03:13 | |
| *** sudipto_ has joined #openstack-kolla | 03:13 | |
| *** zhubingbing_ has quit IRC | 03:17 | |
| *** zhubingbing_ has joined #openstack-kolla | 03:18 | |
| *** rwsu has joined #openstack-kolla | 03:23 | |
| *** ssurana has quit IRC | 03:26 | |
| *** daidv has quit IRC | 03:29 | |
| *** sudipto has quit IRC | 03:37 | |
| *** sudipto_ has quit IRC | 03:37 | |
| *** salv-orlando has joined #openstack-kolla | 03:39 | |
| *** salv-orlando has quit IRC | 03:43 | |
| *** sayantan_ has quit IRC | 03:58 | |
| *** g3ek has quit IRC | 03:58 | |
| *** sayantan_ has joined #openstack-kolla | 03:59 | |
| *** schwicht has quit IRC | 04:02 | |
| *** g3ek has joined #openstack-kolla | 04:08 | |
| *** l4yerffeJ__ has joined #openstack-kolla | 04:21 | |
| *** l4yerffeJ_ has quit IRC | 04:21 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 04:26 | |
| *** l4yerffeJ__ has quit IRC | 04:26 | |
| *** mdnadeem has joined #openstack-kolla | 04:28 | |
| *** salv-orlando has joined #openstack-kolla | 04:39 | |
| *** salv-orlando has quit IRC | 04:44 | |
| *** denaitre has quit IRC | 04:47 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 04:47 |
| *** rwsu has quit IRC | 04:49 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 04:59 |
| *** l4yerffeJ_ has quit IRC | 04:59 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 04:59 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot https://review.openstack.org/443300 | 05:01 |
| *** jrobinson has quit IRC | 05:03 | |
| *** fooliouno has joined #openstack-kolla | 05:06 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 05:07 |
| *** l4yerffeJ__ has joined #openstack-kolla | 05:07 | |
| *** l4yerffeJ_ has quit IRC | 05:08 | |
| *** Serlex has joined #openstack-kolla | 05:12 | |
| *** unicell has joined #openstack-kolla | 05:14 | |
| *** Serlex has quit IRC | 05:15 | |
| *** ssurana has joined #openstack-kolla | 05:16 | |
| masber | hi | 05:17 |
| masber | I can't find tools/start-registry on my kolla installation | 05:17 |
| masber | I am using kolla 3.0.2 | 05:17 |
| *** unicell1 has joined #openstack-kolla | 05:18 | |
| masber | I would like to install the docker registry | 05:18 |
| *** unicell has quit IRC | 05:19 | |
| *** Serlex has joined #openstack-kolla | 05:22 | |
| *** sudipto has joined #openstack-kolla | 05:25 | |
| *** skramaja has joined #openstack-kolla | 05:25 | |
| *** jrobinson has joined #openstack-kolla | 05:31 | |
| *** salv-orlando has joined #openstack-kolla | 05:40 | |
| MarginHu | masber,docker run -d -p 4000:5000 --restart=always --name registry registry:2 | 05:44 |
| *** salv-orlando has quit IRC | 05:45 | |
| *** trinaths has joined #openstack-kolla | 05:58 | |
| *** saneax-_-|AFK is now known as saneax | 06:03 | |
| MarginHu | hi, when I want to update dnsmasq.conf of ironic service , I found it couldn't. | 06:08 |
| *** satyar has joined #openstack-kolla | 06:22 | |
| MarginHu | I got it, yeah! | 06:22 |
| *** jbadiapa has quit IRC | 06:23 | |
| *** trinaths1 has joined #openstack-kolla | 06:29 | |
| *** trinaths has quit IRC | 06:29 | |
| *** rwsu has joined #openstack-kolla | 06:31 | |
| *** salv-orlando has joined #openstack-kolla | 06:33 | |
| *** ssurana has quit IRC | 06:42 | |
| *** ssurana has joined #openstack-kolla | 06:45 | |
| *** rabel has joined #openstack-kolla | 06:51 | |
| *** bmace has quit IRC | 06:53 | |
| *** bmace has joined #openstack-kolla | 06:53 | |
| *** jrobinson has quit IRC | 06:54 | |
| fooliouno | When the command "kolla-kubernetes bootstrap mariadb" is run, is the mariadb-bootstrap pod expected to terminate after completion or should it remain up and running. | 07:04 |
| *** ssurana has quit IRC | 07:06 | |
| *** sayantan_ has quit IRC | 07:09 | |
| *** sayantan_ has joined #openstack-kolla | 07:09 | |
| duonghq | fooliouno, it's job, so it's expected to finish | 07:13 |
| fooliouno | Thanks duonghq! Another quick question. My keystone-create-db pod keeps restarting due to the following error. Any pointers on what could be wrong. | 07:16 |
| fooliouno | 01 | 07:16 |
| fooliouno | "msg": "unable to find /var/lib/ansible/.my.cnf. Exception message: (2003, 'Can\\'t connect to MySQL server on \\'mariadb\\' (110 \"Connection timed out\")')" | 07:16 |
| duonghq | can you check is there any mariadb running in the cluster? | 07:17 |
| fooliouno | I verified that the secrets are generated correctly from passwords.yml for keystone | 07:17 |
| duonghq | (helm ls) | 07:17 |
| fooliouno | I listed all pods and dont see any for mariadb. Hmmm .. | 07:18 |
| duonghq | can you put log of mariadb bootstrap in to some pastebin? | 07:19 |
| fooliouno | duonghq: http://pastebin.com/WPM0JFw7 | 07:21 |
| *** salv-orl_ has joined #openstack-kolla | 07:22 | |
| duonghq | hmm, and mariadb still not running? | 07:23 |
| fooliouno | yes .. no sign of it | 07:23 |
| *** salv-orlando has quit IRC | 07:25 | |
| fooliouno | Let me clarify. There are no mariadb pods, but there is a mariadb service running | 07:27 |
| *** rabel has quit IRC | 07:37 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Add ironic-ui in horizon binary install https://review.openstack.org/443453 | 07:41 |
| *** mnasiadka has joined #openstack-kolla | 07:44 | |
| *** rabel has joined #openstack-kolla | 07:55 | |
| MarginHu | hi, question , I need to rebuild horizon image , but how to tag horizon image version ? | 08:03 |
| MarginHu | how to tag horizon image version instead of whole project version ? | 08:03 |
| *** matrohon has joined #openstack-kolla | 08:04 | |
| *** shardy has joined #openstack-kolla | 08:10 | |
| MarginHu | Jeffrey4l | 08:10 |
| *** mgoddard has joined #openstack-kolla | 08:11 | |
| zhubingbing_ | sup sdake | 08:11 |
| zhubingbing_ | https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP | 08:11 |
| zhubingbing_ | i have success deploy kolla-k8s | 08:12 |
| zhubingbing_ | i have test pass wth document | 08:12 |
| openstackgerrit | Martin André proposed openstack/kolla master: Add heat-api-cloudwatch image https://review.openstack.org/443077 | 08:13 |
| *** l4yerffeJ_ has joined #openstack-kolla | 08:17 | |
| *** l4yerffeJ__ has quit IRC | 08:17 | |
| *** jbadiapa has joined #openstack-kolla | 08:24 | |
| *** mgoddard has quit IRC | 08:28 | |
| *** gfidente has joined #openstack-kolla | 08:36 | |
| openstackgerrit | Tuan Luong-Anh proposed openstack/kolla master: Indicating the location tests directory in oslo_debug_helper https://review.openstack.org/443488 | 08:37 |
| openstackgerrit | Tuan Luong-Anh proposed openstack/kolla-kubernetes master: Indicating the location tests directory in oslo_debug_helper https://review.openstack.org/443490 | 08:40 |
| *** mgoddard has joined #openstack-kolla | 08:40 | |
| *** egonzalez has joined #openstack-kolla | 08:41 | |
| *** nathharp has joined #openstack-kolla | 08:45 | |
| egonzalez | duonghq, topology tab was due a bug on horizon https://bugs.launchpad.net/horizon/+bug/1671084 | 08:47 |
| openstack | Launchpad bug 1671084 in OpenStack Dashboard (Horizon) "heat topology tab not present in horizon" [Critical,Fix released] - Assigned to Richard Jones (r1chardj0n3s) | 08:47 |
| duonghq | egonzalez, roger | 08:48 |
| *** nhlfr has quit IRC | 08:53 | |
| *** nhlfr[m] is now known as nhlfr | 08:55 | |
| *** aolwas has joined #openstack-kolla | 08:57 | |
| *** cuongnv has quit IRC | 09:03 | |
| *** athomas has joined #openstack-kolla | 09:08 | |
| *** manheim has joined #openstack-kolla | 09:12 | |
| *** manheim has quit IRC | 09:20 | |
| *** magicboiz has quit IRC | 09:21 | |
| *** Mannheim has joined #openstack-kolla | 09:22 | |
| *** magicboiz has joined #openstack-kolla | 09:22 | |
| *** cuongnv has joined #openstack-kolla | 09:26 | |
| *** magicboiz has quit IRC | 09:28 | |
| nathharp | Morning | 09:28 |
| *** g3ek has quit IRC | 09:28 | |
| *** haplo37 has quit IRC | 09:28 | |
| nathharp | can anyone comment on the below from the Ocata Nova release notes: | 09:29 |
| nathharp | When making connections to Ceph-backed volumes via the Libvirt driver, the auth values (rbd_user, rbd_secret_uuid) are now pulled from the backing cinder.conf rather than nova.conf. The nova.conf values are only used if set and the cinder.conf values are not set, but this fallback support is considered accidental and will be removed in the Nova 16.0.0 Pike release. See the Ceph documentation for configuring Cinder | 09:29 |
| nathharp | RBD auth. | 09:29 |
| nathharp | 4.0.0 tarball deployment onto existing Ceph - cinder works, Glance works, Nova gives Ceph uath errors pointing at the UUID | 09:31 |
| Jeffrey4l_ | nathharp, this is fixed in yesterday. | 09:31 |
| Jeffrey4l_ | try the latest code. | 09:31 |
| nathharp | aha | 09:31 |
| nathharp | Jeffrey4l_ thanks for the quick response | 09:32 |
| Jeffrey4l_ | np ;) | 09:32 |
| *** yingjun has quit IRC | 09:37 | |
| *** haplo37 has joined #openstack-kolla | 09:38 | |
| *** g3ek has joined #openstack-kolla | 09:38 | |
| *** hrw has quit IRC | 09:43 | |
| *** hrw has joined #openstack-kolla | 09:45 | |
| egonzalez | sayantani01, are you the one working on the deployment docs? | 09:53 |
| *** duonghq has quit IRC | 09:58 | |
| *** MarginHu has quit IRC | 10:00 | |
| *** MarginHu has joined #openstack-kolla | 10:00 | |
| *** MarginHu has quit IRC | 10:02 | |
| *** cuongnv has quit IRC | 10:05 | |
| *** sayantan_ has quit IRC | 10:09 | |
| *** tovin07 has quit IRC | 10:11 | |
| *** tovin07_ has quit IRC | 10:12 | |
| *** magicboiz has joined #openstack-kolla | 10:18 | |
| *** manheim has joined #openstack-kolla | 10:18 | |
| *** deadnull has joined #openstack-kolla | 10:19 | |
| *** manheim has quit IRC | 10:19 | |
| *** manheim has joined #openstack-kolla | 10:20 | |
| *** manheim has quit IRC | 10:22 | |
| *** pbourke has quit IRC | 10:23 | |
| *** satyar has quit IRC | 10:23 | |
| *** manheim has joined #openstack-kolla | 10:24 | |
| *** pbourke has joined #openstack-kolla | 10:25 | |
| *** sambetts|afk is now known as sambetts | 10:35 | |
| *** zhubingbing_ has quit IRC | 10:36 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Enable sanity checks from kolla-ansible https://review.openstack.org/437919 | 10:41 |
| sdake | sbezverk rwellum is blocked here: https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP | 10:46 |
| *** sudipto has quit IRC | 10:51 | |
| *** salv-orl_ has quit IRC | 11:00 | |
| *** salv-orlando has joined #openstack-kolla | 11:01 | |
| *** ccesario__ has quit IRC | 11:01 | |
| *** salv-orlando has quit IRC | 11:01 | |
| *** salv-orlando has joined #openstack-kolla | 11:02 | |
| *** Murali has joined #openstack-kolla | 11:11 | |
| *** athomas has quit IRC | 11:13 | |
| *** furlongm has quit IRC | 11:13 | |
| *** mnasiadka has quit IRC | 11:13 | |
| *** manheim has quit IRC | 11:14 | |
| *** athomas has joined #openstack-kolla | 11:20 | |
| *** JoseMello has joined #openstack-kolla | 11:21 | |
| *** mnasiadka has joined #openstack-kolla | 11:23 | |
| *** manheim has joined #openstack-kolla | 11:27 | |
| *** mnasiadka has quit IRC | 11:27 | |
| *** mnasiadka has joined #openstack-kolla | 11:28 | |
| *** mnasiadk1 has joined #openstack-kolla | 11:28 | |
| *** sean-k-mooney has quit IRC | 11:32 | |
| *** mnasiadk1 has quit IRC | 11:36 | |
| *** mnasiadka has quit IRC | 11:37 | |
| *** rmart04 has joined #openstack-kolla | 11:39 | |
| *** magicboiz has quit IRC | 11:45 | |
| *** mnasiadk1 has joined #openstack-kolla | 11:46 | |
| *** mnasiadka has joined #openstack-kolla | 11:46 | |
| *** athomas has quit IRC | 11:51 | |
| *** rmart04 has quit IRC | 11:51 | |
| *** rmart04 has joined #openstack-kolla | 11:52 | |
| *** rmart04 has quit IRC | 11:57 | |
| *** mnasiadk1 has quit IRC | 11:59 | |
| *** mnasiadka has quit IRC | 11:59 | |
| *** athomas has joined #openstack-kolla | 12:01 | |
| kbaegis | Morning all! Anyone up? | 12:02 |
| kbaegis | https://bugs.launchpad.net/kolla/+bug/1671447 | 12:02 |
| openstack | Launchpad bug 1671447 in kolla "iscsi container fails to start, "no space left on device"" [Undecided,New] | 12:02 |
| *** rwellum_ has joined #openstack-kolla | 12:04 | |
| *** mnasiadka has joined #openstack-kolla | 12:05 | |
| *** mnasiadk1 has joined #openstack-kolla | 12:05 | |
| *** zhurong has quit IRC | 12:08 | |
| des_consolado | kbaegis: have you tried rebooting? :p | 12:08 |
| des_consolado | tbh I've been getting this error here and there | 12:08 |
| des_consolado | with random stuff... | 12:08 |
| des_consolado | it's quite scary tbh, makes me not want to do kolla in production | 12:08 |
| des_consolado | I've heard some of the others saying about using other storage drivers | 12:08 |
| des_consolado | my manager isn't keen on the idea of other storage drivers though due to their supposed immaturity | 12:09 |
| *** mnasiadka has quit IRC | 12:10 | |
| *** mnasiadk1 is now known as mnasiadka | 12:10 | |
| des_consolado | https://github.com/openstack/kolla-ansible/blob/master/doc/production-architecture-guide.rst#storage-driver it's suggested here to use 'btrfs or aufs' | 12:10 |
| des_consolado | well I've been testing with aufs for a couple monthsish now and I've been having these dodgy errors a lot... maybe I should try btrfs | 12:10 |
| kbaegis | Thanks! | 12:13 |
| kbaegis | Yeah, I had more luck with btrfs on gentoo | 12:14 |
| *** rwallner has joined #openstack-kolla | 12:19 | |
| *** g3ek has quit IRC | 12:21 | |
| kbaegis | des_consolado: Is there a handy way to migrate? Docker is currently pulling from my root drive (ext4) | 12:26 |
| kbaegis | I absolutely hate that Ubuntu still uses ext | 12:26 |
| kbaegis | by default | 12:26 |
| *** sudipto has joined #openstack-kolla | 12:29 | |
| *** g3ek has joined #openstack-kolla | 12:30 | |
| *** furlongm has joined #openstack-kolla | 12:31 | |
| *** shardy is now known as shardy_lunch | 12:32 | |
| *** salv-orlando has quit IRC | 12:38 | |
| *** rwallner has quit IRC | 12:45 | |
| nathharp | Jeffrey4l_ - I grabbed the latest code as suggested | 12:45 |
| *** shardy_lunch has quit IRC | 12:45 | |
| *** shardy_lunch has joined #openstack-kolla | 12:45 | |
| *** Jezekus has joined #openstack-kolla | 12:46 | |
| nathharp | now attempting to start nova-compute on my controller? Failing all 20 tries and bombs out | 12:46 |
| Serlex | Why am I getting " 'module' object has no attribute 'set_defaults'" when I upgrade a container? | 12:47 |
| Jezekus | Hi all, may I ask about getting an error stating Project ID "someID" is not a valid Project. (HTTP 400). I am getting such error after clean deployment on single node | 12:47 |
| *** fguillot has joined #openstack-kolla | 12:49 | |
| *** schwicht has joined #openstack-kolla | 12:53 | |
| *** schwicht has quit IRC | 12:54 | |
| *** dave-mccowan has joined #openstack-kolla | 12:56 | |
| *** rwallner has joined #openstack-kolla | 12:57 | |
| egonzalez | Jezekus, is that error when changing nova quotas? | 12:58 |
| *** trinaths1 has left #openstack-kolla | 12:58 | |
| Jezekus | egonzalez: yes | 12:59 |
| egonzalez | Jezekus, is due a recent change in nova (master code not stable) | 12:59 |
| egonzalez | Jezekus, keystone endpoint must be set to versionless http://VIP:35357/ without v3 at the end | 13:00 |
| egonzalez | the fix is in progress, for now update the endpoint | 13:00 |
| egonzalez | openstack endpoint set <interface> --url <http://VIP:PORT/> <endpoint_id> | 13:01 |
| *** fooliouno has quit IRC | 13:03 | |
| *** jtriley has quit IRC | 13:04 | |
| egonzalez | Serlex, which task fail, which container | 13:06 |
| *** Jezekus has quit IRC | 13:07 | |
| *** sean-k-mooney has joined #openstack-kolla | 13:09 | |
| rwallner | hey channel, wondering if anything has hit an issue using bond interfaces when deploying. Im getting errors like these (item=ceph-osd) => {"failed": true, "item": "ceph-osd", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute u'ansible_bond0.###' ### is an actual bond number | 13:16 |
| rwallner | nvm ! this seems to be a config issue | 13:18 |
| rwallner | sorry for tha spam :-| | 13:18 |
| egonzalez | rwallner, is ansible -m setup -i inventory all seeing such bond interface? | 13:20 |
| rwallner | yes, its a bug in our baremetal deployment / globals | 13:21 |
| rwallner | literally just using the wrong interface | 13:21 |
| rwallner | so ansible / kolla is doing the right thing | 13:21 |
| *** schwicht has joined #openstack-kolla | 13:25 | |
| *** shardy_lunch is now known as shardy | 13:27 | |
| *** salv-orlando has joined #openstack-kolla | 13:28 | |
| *** jtriley has joined #openstack-kolla | 13:29 | |
| *** schwicht has quit IRC | 13:30 | |
| *** schwicht_at_work has joined #openstack-kolla | 13:33 | |
| *** ccesario__ has joined #openstack-kolla | 13:37 | |
| rwellum_ | Hi - general question: I'm running k8s on a Centos VM, prior to setting up kola. If I suspend and resume my VM, or make a snapshot and later revert. My VM comes back and shoots up to 100% CPU, eventually backing off, but I never recover the VM, can't log back in to see what's going on etc. Anyone seen this before? | 13:37 |
| *** goldyfruit has joined #openstack-kolla | 13:45 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot https://review.openstack.org/443300 | 13:47 |
| *** schwicht_at_work has quit IRC | 13:51 | |
| *** zhubingbing has joined #openstack-kolla | 13:53 | |
| egonzalez | actually keystone endpoint does not have description, would be OK add description as one of the register tasks? | 13:54 |
| egonzalez | rather service than endpoint | 13:54 |
| *** lamt has joined #openstack-kolla | 13:58 | |
| *** rhallisey has joined #openstack-kolla | 14:04 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla master: Adding iscsi bits to ironic-conductor https://review.openstack.org/443646 | 14:04 |
| *** yingjun has joined #openstack-kolla | 14:05 | |
| *** schwicht has joined #openstack-kolla | 14:07 | |
| *** srwilkers has joined #openstack-kolla | 14:14 | |
| *** yingjun has quit IRC | 14:16 | |
| openstackgerrit | Merged openstack/kolla-ansible master: ironic_pxe bootstrap should run on all hosts https://review.openstack.org/437189 | 14:25 |
| *** lamt has quit IRC | 14:27 | |
| *** lamt has joined #openstack-kolla | 14:31 | |
| *** satyar has joined #openstack-kolla | 14:34 | |
| *** saneax is now known as saneax-_-|AFK | 14:35 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Use versionless keystone endpoints https://review.openstack.org/439024 | 14:35 |
| *** caowei has joined #openstack-kolla | 14:37 | |
| *** Jeffrey4l_ has quit IRC | 14:39 | |
| *** strigazi has quit IRC | 14:44 | |
| *** manheim has quit IRC | 14:45 | |
| *** adrian_otto has joined #openstack-kolla | 14:45 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 14:51 | |
| *** lamt has quit IRC | 14:53 | |
| *** vcn has joined #openstack-kolla | 14:55 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot https://review.openstack.org/443300 | 14:55 |
| *** Serlex has quit IRC | 14:58 | |
| *** yingjun has joined #openstack-kolla | 14:59 | |
| *** manheim has joined #openstack-kolla | 15:00 | |
| *** vcn has quit IRC | 15:00 | |
| openstackgerrit | Merged openstack/kolla master: Add heat-api-cloudwatch image https://review.openstack.org/443077 | 15:00 |
| *** mnasiadka has quit IRC | 15:02 | |
| *** caowei has quit IRC | 15:03 | |
| *** strigazi has joined #openstack-kolla | 15:04 | |
| egonzalez | Jeffrey4l_, is this missing in Ocata? https://review.openstack.org/#/c/437914/1 | 15:08 |
| *** lucker has joined #openstack-kolla | 15:08 | |
| Jeffrey4l_ | egonzalez, it is not critical. just one warning message during build. So it is OK. | 15:09 |
| Jeffrey4l_ | but will fix it later. | 15:09 |
| egonzalez | roger | 15:09 |
| *** lamt has joined #openstack-kolla | 15:09 | |
| *** skramaja has quit IRC | 15:14 | |
| *** salv-orlando has quit IRC | 15:14 | |
| *** salv-orlando has joined #openstack-kolla | 15:15 | |
| egonzalez | Jeffrey4l_, can you remove -2 https://review.openstack.org/#/c/416369/ ? | 15:16 |
| *** unicell has joined #openstack-kolla | 15:16 | |
| NachoDuck | Is there a way to do a reconfigure on nova with running instances? As the restart of nova_libvirt fails | 15:16 |
| egonzalez | NachoDuck, when or how fails? | 15:17 |
| egonzalez | kolla version and docker version? | 15:17 |
| *** unicell1 has quit IRC | 15:17 | |
| *** zhubingbing has quit IRC | 15:18 | |
| NachoDuck | egonzalez, Kolla 3.0.0 I think and Docker 1.12.3 | 15:19 |
| kbaegis | des_consolado: Switched to btrfs. Had to reinstall ubuntu :) | 15:19 |
| egonzalez | NachoDuck, can you ensure about kolla version? pip freeze | grep kolla | 15:20 |
| NachoDuck | egonzalez: kolla==3.0.0 | 15:20 |
| egonzalez | NachoDuck, can you share error trace? | 15:21 |
| openstackgerrit | Merged openstack/kolla stable/newton: Fix ironic inspector https://review.openstack.org/439913 | 15:22 |
| openstackgerrit | Merged openstack/kolla stable/newton: Fix binary ironic-inspector rootwrap configuration https://review.openstack.org/439882 | 15:22 |
| NachoDuck | egonzalez: http://paste.openstack.org/show/Icmq72noy1TANLWDpNrV/ | 15:23 |
| nathharp | Jeffrey4l_ think I have found a typo in kolla-ansible/ansible/roles/nova/tasks/external-ceph.yml - line 17 - ‘state’ should be ‘stat’ | 15:31 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot https://review.openstack.org/443300 | 15:31 |
| egonzalez | nathharp, yep | 15:32 |
| *** adrian_otto has quit IRC | 15:32 | |
| egonzalez | nathharp, this have to be backported to ocata, can you open a bug? | 15:32 |
| sbezverk | srwilkers: ping | 15:33 |
| sbezverk | srwilkers: appreciate if you could review this: https://review.openstack.org/443375 | 15:33 |
| *** zhubingbing has joined #openstack-kolla | 15:34 | |
| nathharp | egonzalez - will do | 15:37 |
| egonzalez | nathharp, thanks | 15:37 |
| des_consolado | kbaegis: yeah D: I need to try it too! just had to reboot again about 15 minutes ago lmao, not even running anything on this crap, it's just a deployment with no actual compute being used or anything!! | 15:37 |
| des_consolado | just seems that redeploying and redeploying and redeploying really screws AUFS :\ | 15:38 |
| des_consolado | even done a lot of docker system prune | 15:38 |
| kbaegis | des_consolado: AUFS sucks I've decided | 15:38 |
| kbaegis | Yeah, even the cleanup-image container doesn't work | 15:38 |
| des_consolado | does kubernetes have an opinion about which storage driver the underlying docker daemons ought to be using? | 15:39 |
| kbaegis | des_consolado: Excellent question. | 15:40 |
| kbaegis | des_consolado: raw RBD looks to be an option: https://kubernetes.io/docs/getting-started-guides/ubuntu/storage/ | 15:41 |
| egonzalez | NachoDuck, are you using qemu or kvm? | 15:41 |
| kbaegis | des_consolado: Looks to be the equivalent of drive-mapping in docker on top of ceph RBD | 15:41 |
| kbaegis | Pretty smart honestly | 15:42 |
| *** mdnadeem has quit IRC | 15:42 | |
| NachoDuck | egonzalez: kvm I believe | 15:42 |
| *** jbadiapa has quit IRC | 15:42 | |
| *** sayantan_ has joined #openstack-kolla | 15:42 | |
| des_consolado | kbaegis: where did you find that? links plz :D | 15:42 |
| kbaegis | des_consolado: ^^ ->>https://kubernetes.io/docs/getting-started-guides/ubuntu/storage/ and some synthesis of information in my head ;) | 15:43 |
| kbaegis | I might be misunderstanding | 15:43 |
| kbaegis | There's this too http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html | 15:44 |
| kbaegis | Basically, rbd is a distributed block device. Think millions of 512k objects. Advantage is that if you move a pod, you don't have to xfer block across a storage network | 15:46 |
| sdake | rwellum_ YMMV with virtualm achines | 15:47 |
| sdake | zhubingbing ping | 15:48 |
| sdake | zhubingbing yous aid you went through the etherpad instructions howeveryo u didn't add a tested and date for each item? | 15:48 |
| *** zhubingbing has quit IRC | 15:48 | |
| *** zhubingbing has joined #openstack-kolla | 15:48 | |
| rwellum_ | sdake, I spoke to others at the PTG who were doing it fine, and in fact that was a factor in explaining their environment, they got it working once, snapshot their VM's and then always just cloned etc when needed. So perhaps I am just being unlucky. | 15:49 |
| egonzalez | NachoDuck, I don't know why is complying with "Is a directory: u'/var/lib/kolla/config_files/secrets'" | 15:53 |
| *** jtriley_ has joined #openstack-kolla | 15:55 | |
| des_consolado | kbaegis: https://coreos.com/os/docs/latest/mounting-storage.html#creating-and-mounting-a-btrfs-volume-file "Container Linux 561.0.0 and later are installed with ext4 + overlayfs to provide a layered filesystem for the root partition. Installations from prior to this, use btrfs for this functionality." | 15:56 |
| egonzalez | nathharp, are you going to push a change for the bug? | 15:57 |
| des_consolado | not kubernetes strictly speaking but coreos container linux os defaults changed from btrfs to ext4+overlayfs interestingly | 15:57 |
| *** jtriley has quit IRC | 15:58 | |
| *** kbaegis has quit IRC | 15:58 | |
| NachoDuck | egonzalez: I see, it appears the containers restarted on one of the other nodes, so might not be a Kolla problem. The container stopped but it won't start now as it can't unmount the container volume | 15:59 |
| *** rabel has quit IRC | 16:00 | |
| *** jtriley_ has quit IRC | 16:01 | |
| *** manheim has quit IRC | 16:01 | |
| kfox1111 | morning. | 16:02 |
| *** kbaegis has joined #openstack-kolla | 16:02 | |
| kbaegis | des_consolado: Huh. Interesting | 16:04 |
| *** manheim has joined #openstack-kolla | 16:05 | |
| openstackgerrit | Merged openstack/kolla master: Allow full customisation of kube image binaries https://review.openstack.org/436897 | 16:07 |
| openstackgerrit | Thomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service. https://review.openstack.org/443732 | 16:07 |
| des_consolado | ^^^ first thing I've pushed for review... it works in so far as the image works and then running with a kolla-ansible deployment with the changes I've made to kolla-ansible... well yes the services all run and there are no errors :D:D | 16:08 |
| des_consolado | I'm sure I'm missing things though, I probably need to write tests etc? | 16:09 |
| des_consolado | I was thinking of even pushing this as WIP because I doubt you will want to merge that right off the bat. I'm not sure. Feedback appreciated. | 16:09 |
| *** jtriley has joined #openstack-kolla | 16:09 | |
| sbezverk | kfox1111: good morning.. could you please review this ps? https://review.openstack.org/443375 | 16:09 |
| kfox1111 | sure. sec | 16:10 |
| des_consolado | actually I don't think there's anything wrong with it and it's just an image I guess... | 16:13 |
| openstackgerrit | Merged openstack/kolla-ansible master: Set empty openstack_release option https://review.openstack.org/443291 | 16:15 |
| openstackgerrit | Merged openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/443353 | 16:18 |
| *** zhubingbing has quit IRC | 16:18 | |
| egonzalez | des_consolado, added a couple of comments. Just nits and code style | 16:19 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot https://review.openstack.org/443300 | 16:24 |
| openstackgerrit | Thomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service. https://review.openstack.org/443732 | 16:25 |
| kbaegis | des_consolado: btw- Tap as a service? | 16:25 |
| kbaegis | Does kolla do that? | 16:25 |
| des_consolado | egonzalez: I made the correcting changes and just did a commit --amend and then a git review again, can see it's now https://review.openstack.org/443732 - hope this is the correct workflow | 16:26 |
| *** rabel has joined #openstack-kolla | 16:26 | |
| des_consolado | kbaegis: no idea what that even is sorry :D | 16:26 |
| des_consolado | oh, pcapping... | 16:26 |
| kbaegis | des_consolado: Oh, the thing I meant to direct to you: just had btrfs crash on ubuntu 16.04 on the root drive during a kolla-build | 16:27 |
| des_consolado | oh dear, lol D: | 16:27 |
| egonzalez | des_consolado, yep thats the workflow, use --amend to push the the same change id and not replicate commits | 16:27 |
| des_consolado | ah feels so good to be losing my openstack contribution virginity finally | 16:28 |
| des_consolado | egonzalez: oh shit, the author is wrong D: | 16:30 |
| des_consolado | I just tried to change it and do a commit --amend also but it rejected it saying there were no changes :( | 16:31 |
| *** adrian_otto has joined #openstack-kolla | 16:31 | |
| egonzalez | des_consolado, if the commit was one release before you had a free summit ticket, but this does not happen anymore :( | 16:31 |
| egonzalez | des_consolado, hrm, dunno if author can be changed :/ | 16:32 |
| openstackgerrit | Thomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service. https://review.openstack.org/443732 | 16:32 |
| des_consolado | ah it is fixed | 16:33 |
| des_consolado | had to do git commit --amend --author "blah blah blah" | 16:34 |
| egonzalez | something new :) | 16:34 |
| kfox1111 | sbezverk: reviewed. | 16:36 |
| kfox1111 | it looks good, if thats the right path to go down. | 16:36 |
| sbezverk | kfox1111: just replied.. this change was mostly to address growing list in nova service-list with nova-compute-ironic down state | 16:38 |
| sbezverk | since it registers itself with its host name everytime. | 16:38 |
| kfox1111 | oh.. cause the daemonset is not net=host... | 16:38 |
| sbezverk | yep | 16:38 |
| kfox1111 | still. not sure whats the best aproach. net=host or statefulset. | 16:39 |
| sbezverk | kfox1111: ironic people told me that in ocata, they do not ask nova/neutron to bind a port | 16:39 |
| sbezverk | so the issue is resolved by more smart networking.. | 16:39 |
| kbaegis | ooc | 16:40 |
| kfox1111 | yeah, but we still need a matching ovsagent anyway? | 16:40 |
| kbaegis | Has anyone tried anything crazy like putting mariadb on a ramdisk? | 16:40 |
| sbezverk | kfox1111: nope they do not need it at that poijt | 16:41 |
| kfox1111 | oh. so the ironic ml2 driver made it in for ocata? | 16:41 |
| sbezverk | kfox1111: they done it a bit differenely, they let ironic port to be in not bound state | 16:42 |
| kfox1111 | ah. | 16:42 |
| sbezverk | kfox1111: since ironic completely controls it | 16:42 |
| sbezverk | it does not matter | 16:42 |
| kfox1111 | ok. then +1 to statefulset. | 16:42 |
| sbezverk | kfox1111: :) | 16:42 |
| kfox1111 | :) | 16:43 |
| kfox1111 | when is 4.0.x containers gettign released (if not already?) | 16:43 |
| *** sudipto has quit IRC | 16:44 | |
| sbezverk | kfox1111: do not know.. I have been trying to clean up ironic discovered issues and address them.. | 16:44 |
| kfox1111 | yeah. | 16:44 |
| kfox1111 | if 4.x solves the issue cleanly by not needint the ovs agent, I think we should probably just push forward and require 4.x for that? | 16:44 |
| sbezverk | kfox1111: for sure.. even ironic people recommends using ocata at minimum.. | 16:45 |
| kfox1111 | k. | 16:45 |
| kfox1111 | then I think the only issue is that one ws char, then +2. :) | 16:46 |
| openstackgerrit | Thomas O'Neill proposed openstack/kolla-ansible master: Add neutron-bgp-dragent playbooks and sensible defaults. https://review.openstack.org/443747 | 16:46 |
| kfox1111 | and I think we need to start a document that says what is the minimum supported version of things. | 16:46 |
| kfox1111 | like, lvm doesn't work until 3.x. ironic doesn't work until 4.x, etc. | 16:46 |
| sbezverk | kfox1111: damn ws, I need an editor which tracks them for me ;) | 16:47 |
| des_consolado | how do we know what's going on with these gates? | 16:47 |
| kfox1111 | des_consolado: which gates? | 16:47 |
| des_consolado | kfox1111: I'm not sure, I'm Thomas O'Neill, I just submitted those 2 changes for kolla and kolla-ansible for the neutron-bgp-dragent stuff | 16:47 |
| egonzalez | des_consolado, once gates finish you can see results in gerrit | 16:47 |
| des_consolado | I think they have to go through some gates now or something? | 16:47 |
| egonzalez | des_consolado, if want to see the status check here http://status.openstack.org/zuul/ add gerrit id in the search field | 16:48 |
| kfox1111 | des_consolado: ah. ok. I'm not very familior with those gates. | 16:48 |
| des_consolado | egonzalez: cool, how long's it usually take? | 16:48 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 16:49 |
| egonzalez | depends if zuul is busy or not, often between 30~40m | 16:49 |
| kfox1111 | there's one more ws one more line down it looks like. :/ | 16:49 |
| des_consolado | oh there was 1 failure... better look at that.. | 16:49 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 16:51 |
| egonzalez | des_consolado, seems unrelated to your change. keystone bootstrap failed | 16:51 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm 2.2 and localpath https://review.openstack.org/436946 | 16:55 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix typo in nova external ceph https://review.openstack.org/443753 | 16:56 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm 2.2 and localpath https://review.openstack.org/436946 | 16:57 |
| *** rabel has quit IRC | 16:58 | |
| *** lucasxu has joined #openstack-kolla | 16:58 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm 2.2 and localpath https://review.openstack.org/436946 | 16:58 |
| *** rhallisey has quit IRC | 17:02 | |
| *** gfhellma has joined #openstack-kolla | 17:03 | |
| sbezverk | kfox1111: done, and here is another one for missing pxelinux.0 file https://review.openstack.org/443300 | 17:03 |
| *** egonzalez has quit IRC | 17:03 | |
| sbezverk | kfox1111: this one is important as it requires to copy this file everytime :( | 17:03 |
| nathharp | Jeffrey4l_ has your external_ceph fix hard coded ceph client names? | 17:08 |
| *** lucasxu has quit IRC | 17:08 | |
| *** lucker has quit IRC | 17:09 | |
| *** hieulq has quit IRC | 17:12 | |
| *** rhallisey has joined #openstack-kolla | 17:15 | |
| *** hieulq has joined #openstack-kolla | 17:15 | |
| *** yingjun has quit IRC | 17:16 | |
| *** matrohon has quit IRC | 17:17 | |
| *** lucasxu has joined #openstack-kolla | 17:17 | |
| *** lucasxu has quit IRC | 17:17 | |
| *** l4yerffeJ_ has quit IRC | 17:17 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 17:18 | |
| *** shardy has quit IRC | 17:21 | |
| des_consolado | what's the difference between verifying and reviewing | 17:24 |
| *** manheim has quit IRC | 17:25 | |
| inc0 | good morning | 17:25 |
| des_consolado | hi | 17:25 |
| jascott1 | verifying is installing, running tests etc. Reviewing is reviewing code. | 17:26 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot https://review.openstack.org/443300 | 17:26 |
| openstackgerrit | Merged openstack/kolla master: Updated from global requirements https://review.openstack.org/443352 | 17:26 |
| *** unicell has quit IRC | 17:29 | |
| *** athomas has quit IRC | 17:30 | |
| *** jtriley has quit IRC | 17:30 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Enable sanity checks from kolla-ansible https://review.openstack.org/437919 | 17:31 |
| des_consolado | I feel like I should probably write some kind of tests for the neutron-bgp-dragent kolla-ansible stuff... | 17:32 |
| openstackgerrit | Merged openstack/kolla-ansible master: Enable run nova-compute-ironic with nova-compute on the same node https://review.openstack.org/443138 | 17:33 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix the incompatible of docker-py 2.0.0 https://review.openstack.org/438813 | 17:34 |
| *** nathharp has quit IRC | 17:39 | |
| des_consolado | ah thanks inc0, I didn't know about raising unavailability on repo binaries, yeah neutron-dynamic-routing isn't available as an rpm | 17:40 |
| *** jrich523 has quit IRC | 17:40 | |
| *** rhallisey has quit IRC | 17:40 | |
| sbezverk | inc0: are ocata images available? | 17:41 |
| *** jrich523 has joined #openstack-kolla | 17:41 | |
| *** ssurana has joined #openstack-kolla | 17:41 | |
| des_consolado | wtf??? http://paste.openstack.org/show/602102/ | 17:42 |
| des_consolado | my repo just murdered itself ?! :'( | 17:42 |
| inc0 | des_consolado: np, we get this from time to time | 17:43 |
| inc0 | otherwise change looks really good:) | 17:43 |
| inc0 | sbezverk: not yet | 17:43 |
| des_consolado | oh I just had to do cd .. | 17:43 |
| *** sayantan_ has quit IRC | 17:43 | |
| openstackgerrit | Thomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service. https://review.openstack.org/443732 | 17:44 |
| *** sayantan_ has joined #openstack-kolla | 17:44 | |
| inc0 | gerrit takes some time to get used to | 17:44 |
| inc0 | I ususally reset --hard origin/master and then checkout change from review.openstack.org | 17:44 |
| inc0 | you have small "download" button on top right corner | 17:45 |
| openstackgerrit | Merged openstack/kolla-ansible master: Remove cell setup and db creation nova upgrade https://review.openstack.org/443290 | 17:45 |
| des_consolado | oh ic, ty | 17:46 |
| inc0 | this way you don't need to track your local fs | 17:46 |
| inc0 | handy when you do more than one change at the time | 17:46 |
| *** mattmceuen has joined #openstack-kolla | 17:48 | |
| inc0 | des_consolado: question, does dragent has anything to do with calico? | 17:49 |
| des_consolado | inc0: I don't think so, not that I know of. It's just the BGP speaker. It uses the Ryu driver. | 17:50 |
| des_consolado | Isn't calico an overlay networking driver or something... | 17:50 |
| *** vhosakot has joined #openstack-kolla | 17:50 | |
| *** fooliouno has joined #openstack-kolla | 17:50 | |
| inc0 | well Calico uses bgp for vm networking | 17:51 |
| des_consolado | yeah am just reading this http://docs.projectcalico.org/v2.0/getting-started/openstack/ - looks like it does alotalotalot with the networking... heh | 17:51 |
| inc0 | so I guess this works differently | 17:51 |
| inc0 | yeah it pretty much defines whole networking for you | 17:52 |
| *** eaguilar has quit IRC | 17:52 | |
| inc0 | which is cool, I asked because both uses bgp and I was curious if these are connected in any way | 17:52 |
| *** eaguilar_ has joined #openstack-kolla | 17:52 | |
| des_consolado | looks like Calico uses BIRD | 17:52 |
| *** gfidente has quit IRC | 17:53 | |
| inc0 | I think it can also use quagga | 17:54 |
| inc0 | not sure tho, I'm not networking guy | 17:55 |
| *** rhallisey has joined #openstack-kolla | 17:55 | |
| inc0 | nope, I was wrong, no quagga | 17:57 |
| des_consolado | inc0: how do you usually run this reno command? I see that it's in .tox/genconfig/bin/reno | 17:57 |
| des_consolado | I guess can you also just run python -m reno ? | 17:57 |
| des_consolado | ah it's documented here I guess https://docs.openstack.org/developer/reno/ | 17:58 |
| inc0 | no, do pip install -r test_requitements.txt | 17:58 |
| *** lucasxu has joined #openstack-kolla | 17:58 | |
| inc0 | and then just reno new dragent_support (or sth like that) | 17:58 |
| vhosakot | v1k0d3n: portdirect: saw these errors twice today. docker and kubelet failed to start. have you seen this error, is it something intermittent? - http://paste.openstack.org/show/602105/ | 17:58 |
| *** eaguilar has joined #openstack-kolla | 17:59 | |
| vhosakot | digging logs. | 17:59 |
| *** eaguilar_ has quit IRC | 18:00 | |
| des_consolado | inc0: also re: the naming, I think that's an issue with the plugin itself - https://github.com/openstack/neutron-dynamic-routing/blob/master/setup.cfg#L35 - see how the package is called neutron-dynamic-routing, but the agent that runs is called neutron-bgp-dragent | 18:00 |
| des_consolado | I gather these guys in the future plan to implement other dynamic routing protocols/agents, like maybe for ospf or something | 18:01 |
| inc0 | yeah...it's..unfortunate naming | 18:01 |
| inc0 | well, that being said, one driver to rule all the routing is bad design too | 18:01 |
| inc0 | especially if it's only driven by name... | 18:02 |
| *** pbourke has quit IRC | 18:03 | |
| *** adrian_otto has quit IRC | 18:05 | |
| *** Murali has quit IRC | 18:06 | |
| openstackgerrit | Thomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service. https://review.openstack.org/443732 | 18:07 |
| openstackgerrit | Thomas O'Neill proposed openstack/kolla-ansible master: Add neutron-bgp-dragent playbooks and sensible defaults. https://review.openstack.org/443747 | 18:08 |
| des_consolado | I gotta go home anyway back tomorrow thanks everyone | 18:11 |
| inc0 | cya des_consolado, thanks | 18:11 |
| vhosakot | v1k0d3n: portdirect: I see each vagrant VM using more than 200% CPU on my host. My host has 16 CPUs, will dig more - http://paste.openstack.org/show/602108/ | 18:15 |
| *** unicell has joined #openstack-kolla | 18:15 | |
| vhosakot | anybody see each vagrant VM use more than 200% CPU on the host when deploying kolla-k8s per https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html? | 18:15 |
| sbezverk | vhosakot: I think rwellum_ reported similar issue | 18:16 |
| vhosakot | sbezverk: hmm, thanks, I think somebody else also saw it, I dont remember who it was here in IRC when I asked the question about 200% CPU usage. | 18:17 |
| vhosakot | rwellum_: how did you resolve the issue? what is the RAM and CPU of your vagrant host? I think we need to add the steps to resolve this issue in the docs in https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html as it looks like a day-1 blocker for a new developer :) | 18:18 |
| vhosakot | I asked the same question about 200% CPU usage 2 weeks ago, digging IRC logs to see who replied and if there was a solution mentioned. | 18:19 |
| vhosakot | sorry I missed yesterday's kolla meeting as I had an appt at the same time. | 18:20 |
| *** zioproto has quit IRC | 18:21 | |
| vhosakot | sbezverk: how much RAM and CPU do you have on your vagrant host? also, are you using libvirt vagrant driver to standup your devenv for kolla-k8s per https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html ? | 18:24 |
| vhosakot | I have 16 CPUs and 32G RAM on my vagrant host with libvirt driver, thought it was enough to boot the four vagrant VMs for kolla-k8s. Any info about host resources to standup vagrant dev env for kolla-k8s would be very helpful :) | 18:30 |
| qwang | kfox1111: ping | 18:31 |
| vhosakot | I'm asking in the #vagrant IRC channel :) | 18:32 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm 2.2 and localpath https://review.openstack.org/436946 | 18:39 |
| kfox1111 | qwang: hi | 18:40 |
| openstackgerrit | Merged openstack/kolla master: Add ironic-ui in horizon binary install https://review.openstack.org/443453 | 18:40 |
| rwellum_ | vhosakot, have not solved it unfortunately. And I've been blocked on various day 1 issues with kolla for over a month... | 18:42 |
| sbezverk | kfox1111: please when you have a chance two PS are ready | 18:42 |
| kfox1111 | the statefulset and pxe image? | 18:42 |
| qwang | kfox1111: I want to use the new conditions/tag feature from helm in the service charts. One idea is to add all dependencies including those common ones (mariadb/rabbitmq/etc) in each of the services and use tag to enable/disable for installation | 18:42 |
| kfox1111 | both are +2'ed. | 18:42 |
| rwellum_ | vhosakot, 32G Ram and 6 CPU's - so less cpu's than you. | 18:42 |
| kfox1111 | qwang: +1. I was thinking the same thing. | 18:43 |
| qwang | kfox1111: another idea, as jascott1 pointed out, use condition/tags for some optional parts, like some ovs components for neutron | 18:43 |
| kfox1111 | +1 to that too. :) | 18:43 |
| rwellum_ | vhosakot, which SDN are you using? Weave etc? | 18:44 |
| vhosakot | rwellum_: hmm, thanks for the reply, are you deploying kolla-kubernetes per the docs in https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html and blocked due to vagrant vm using 200% CPU? | 18:44 |
| qwang | kfox1111: I wonder what's your original desire for it. Any other use case you'd like to have? | 18:44 |
| vhosakot | rwellum_: yes, weave, which comes OTB, do you think that can cause the issue? | 18:44 |
| kfox1111 | qwang: you hit the two I was thinking of. | 18:45 |
| kfox1111 | I can't off hand think of a 3rd use case. | 18:45 |
| rwellum_ | vhosakot, I've tried both that guide and also the bare metal route - which I am helping to write. Have not come to a conclusion with either. | 18:45 |
| vhosakot | rwellum_: well, bare metal deploy does now have vagrant right? so, the bare metal issue you see are non-vagrant related? | 18:46 |
| rwellum_ | vhosakot, weave seems to me to be the most unstable, Try: | 18:46 |
| rwellum_ | vhosakot, | 18:46 |
| rwellum_ | vhosakot, sed -i 's/weave/canal/g' halcyon-kubernetes/kube-deploy/group_vars/all.yml | 18:46 |
| qwang | kfox1111: thank you very much. You help me sort it out. I'll start to work on them | 18:47 |
| kfox1111 | qwang: cool. thanks. :) | 18:47 |
| rwellum_ | No vagrant with bare metal vhosakot my issues there are unrelated - mainly networking. Documented here: https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP | 18:47 |
| *** rwallner has quit IRC | 18:47 | |
| kfox1111 | qwang: I'm not sure just how usable it will be without configmap stuff being done though. we need a way to point at the service level db's/things. | 18:47 |
| kfox1111 | though that work can be done in parallel. | 18:47 |
| rwellum_ | vhosakot, but eventually canal died too, so I don't think it's related to the vagrant issue - just something to try | 18:48 |
| *** papacz has quit IRC | 18:48 | |
| rwellum_ | vhosakot, note that sdake mentioned using within a VM is not a consistent path - although I see plenty doing it successfully. | 18:49 |
| vhosakot | rwellum_: cool, thanks for the info.. I'll try canal and will keep you posted. I might tray flannel as well, good luck with bare metal deploy, I'd like to help with it once I'm able to standup my devenv | 18:49 |
| vhosakot | I know sdake and I both are _not_ vagrant fans... standing up within VMs :) | 18:50 |
| rwellum_ | vhosakot, should also mention I wasn't seeing issues with Vagrant until recently - last couple days it seems like. | 18:50 |
| vhosakot | rwellum_: sdake: once I'm able to standup my devenv, I'll try to contribute to the bare metal deploy you guys are working on. | 18:51 |
| rwellum_ | vhosakot, I've just followed the dev guide on a baremetal server, so no VM - and so far no performance issues. | 18:51 |
| vhosakot | rwellum_: the high CPU usage could be vagrant issue too. which version are you using? I'm using 1.8.1 per the docs in https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html. | 18:52 |
| *** sambetts is now known as sambetts|afk | 18:52 | |
| vhosakot | rwellum_: cool, good you have eliminated VM/vagrant issues in bare-metal deploy.. nice. | 18:52 |
| robcresswell | Hey, question, why does kolla's horizon binary bake in all the plugins? | 18:52 |
| rwellum_ | vhosakot, yeah completely following the guide like you. Also the guide doesn't mention installing docker which you'll have to do. | 18:53 |
| rwellum_ | vhosakot, this: http://paste.openstack.org/show/602110/ | 18:53 |
| *** Jezekus has joined #openstack-kolla | 18:54 | |
| vhosakot | rwellum_: does the task "install kubeadm and required prereqs" in https://github.com/att-comdev/halcyon-kubernetes/blob/master/kube-deploy/roles/deploy-kube/tasks/centos.yml#L35 install docker? | 18:55 |
| *** adrian_otto has joined #openstack-kolla | 18:55 | |
| vhosakot | rwellum_: why do we need to do those steps manually? I see halcyon-kubernetes already does it through these tasks in Ansible's stdout - http://paste.openstack.org/show/602111/ | 18:56 |
| vhosakot | rwellum_: it was all working great around 3 weeks ago, and I was able to standup the dev env with weave.. don't know the issue today. | 18:57 |
| rwellum_ | vhosakot, for me docker does not get installed, i asked the dock author | 19:00 |
| vhosakot | rwellum_: interesting, for me, docker is properly installed on all the vagrant vms through the tasks in http://paste.openstack.org/show/602111/. do you see those tasks running for you? | 19:01 |
| rwellum_ | vhosakot, when I try with a vm again I'll let you know. But i def. did not have docker daemon running | 19:02 |
| vhosakot | rwellum_: cool, thanks. I'll use canal/flannel now... thanks for your help! | 19:02 |
| vhosakot | sbezverk: kfox1111: do you use canal to standup kolla-kubernetes dev env? | 19:02 |
| *** Jezekus has quit IRC | 19:02 | |
| *** jtriley has joined #openstack-kolla | 19:03 | |
| *** eaguilar has quit IRC | 19:03 | |
| rwellum_ | vhosakot, last time I worked with sbezverk he was using canal | 19:04 |
| rwellum_ | vhosakot, in fact he told me too as it was more stable | 19:04 |
| *** eaguilar has joined #openstack-kolla | 19:05 | |
| vhosakot | rwellum_: cool cool, thanks... BTW, I see this error in the libvirt logs of each vagrant VM that uses 200% CPU on the host - http://paste.openstack.org/show/602112/ | 19:05 |
| *** rhallisey has quit IRC | 19:07 | |
| rwellum_ | vhosakot, plenty on google on that one... | 19:08 |
| sbezverk | vhosakot: yep canal seems to work for me well | 19:08 |
| vhosakot | rwellum_: yup, googling now :) | 19:08 |
| vhosakot | sbezverk: thanks, how much RAM and CPU do you have on your vagrant host? | 19:08 |
| rwellum_ | sbezverk, running kubectl on bare-metal is giving me permission errors. Do you run normally as sudo? | 19:11 |
| sbezverk | vhosakot: no vagrant, bare metal vm | 19:12 |
| sbezverk | rwellum_: should not be | 19:13 |
| vhosakot | sbezverk: cool.. \m/ \m/ for bare-metal :) | 19:13 |
| sbezverk | kubectl creates .kube folder for the config | 19:13 |
| *** david-lyle has quit IRC | 19:14 | |
| vhosakot | sbezverk: how many bare-metal VMs do you have? I think I can create few too, are there docs to deploy on bare-metal VMs? | 19:15 |
| vhosakot | rwellum_ told me he is adding the bare-metal docs.. is any doc available for me to start bare-metal deploy? | 19:15 |
| sbezverk | vhosakot: I use two VMs one where I run kube cluster and one for development | 19:16 |
| vhosakot | sbezverk: cool, like one master, one worker node? did you dpeloy using the Ansible playbooks in https://github.com/att-comdev/halcyon-kubernetes? | 19:18 |
| inc0 | robcresswell: well, mostly because we don't know what plutins you might want | 19:19 |
| sbezverk | vhosakot: no just master, second vm is just linux ws with development tools | 19:19 |
| inc0 | on the bright note, if that's a problem it's super easy to make it install just what you want | 19:19 |
| vhosakot | sbezverk: so, you deploy kolla-kubernetes in a single-vm then? | 19:20 |
| *** rhallisey has joined #openstack-kolla | 19:20 | |
| sbezverk | vhosakot: I deployed everything manually it took longer but seeing how much trouble you guys hit, I am starting to this it was not that bad ;) | 19:20 |
| vhosakot | sbezverk: yes, I am not a fan of vagrant :) and debugging vagrant issues instead of kolla is not something I enjoy ;) | 19:21 |
| vhosakot | sbezverk: have you added docs for the way you deployed so the community can use it? | 19:21 |
| rwellum_ | vhosakot, that's the etherpad I pointed you too earlier | 19:21 |
| vhosakot | rwellum_: cool | 19:21 |
| rwellum_ | vhosakot, https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP | 19:21 |
| rwellum_ | vhosakot, but... I still think it's very complicated and there are lots of unknowns. | 19:22 |
| *** salv-orl_ has joined #openstack-kolla | 19:22 | |
| *** satyar has quit IRC | 19:23 | |
| rwellum_ | So inc0 - I'm not whining :) -but in the last few days there have been at least 3 maybe 4 people on this IRC including me who are struggling to get going with Kolla. I know part of this is the fact the k8s is much newer than ansible. But I feel like maybe some time needs to be spent on this to get this part stable? | 19:23 |
| vhosakot | rwellum_: right.. I understand the docs are still WIP now.. cool | 19:23 |
| robcresswell | inc0: Right, but wouldn't they be installed as other binaries? I'm not very familiar with Kolla, just curious | 19:24 |
| *** sayantan_ has quit IRC | 19:24 | |
| vhosakot | inc0: I agree too and +1 rwellum_ comment above ^^^ :) | 19:24 |
| inc0 | rwellum_: couldn't agree more, issue is we had release yesterday (of kolla and kolla-ansible) so we really had fires to put out in our stable deliverables | 19:24 |
| *** sayantan_ has joined #openstack-kolla | 19:24 | |
| inc0 | now it's a bit quieter and we can focus more on k8s | 19:24 |
| inc0 | you just caught us in release-fever time | 19:25 |
| *** salv-orlando has quit IRC | 19:25 | |
| vhosakot | sbezverk: rwellum_: that etherpad does now have the steps to deploy, does it... since sbezverk has got it working in a single-vm, he could help me thru webex may be if he is available sometime. | 19:25 |
| inc0 | robcresswell: what do you mean by "other binaries"? | 19:25 |
| rwellum_ | inc0, ack | 19:26 |
| rwellum_ | vhosakot, what is your question? | 19:26 |
| inc0 | issue is, kolla clearly separates build phase from deployment phase | 19:26 |
| inc0 | which means decisions you make on deployment can't affect build | 19:26 |
| *** MasterOfBugs has joined #openstack-kolla | 19:26 | |
| vhosakot | rwellum_: the steps to deploy kolla-kubernetes on a bare-metal VM without using vagrant :) | 19:26 |
| rwellum_ | vhosakot, I've webexed with sbezverk a couple of times and it's incredibly helpful but it's not going to scale is it? | 19:26 |
| kfox1111 | sbezverk: portdirect: srwilkers: can we get this through quickly? https://review.openstack.org/#/c/436946/ | 19:26 |
| inc0 | and since you can add/remove projects really easily, and that's deployment phase | 19:26 |
| kfox1111 | pretty trivial, but touches a lot of things. | 19:27 |
| kfox1111 | inc0: you too. :) | 19:27 |
| inc0 | when you for example want to deploy new project to existing openstack | 19:27 |
| rwellum_ | vhosakot, that doc is WYSIWYG | 19:27 |
| inc0 | you don't want to replace horizon container too | 19:27 |
| vhosakot | rwellum_: yes, 1:1 webex with sbezverk is not scalable :) alteast, I can help add the docs with you and sbezverk | 19:27 |
| robcresswell | inc0: I was just curious, with patches lik https://review.openstack.org/#/c/443453, why all the ui plugins are included in an install; if they are installed and have issues, they could well break the entire Horizon install; this might just be because I dont understand Kolla, tho | 19:27 |
| inc0 | robcresswell: well, if they're installed but not enabled (enable would happen on deploy phase) that should be fine | 19:28 |
| inc0 | you just have few mbs of dormant code on filesystem otherwise | 19:28 |
| vhosakot | sbezverk: I'll read https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP and start deploying on my bare-metal VM, might ask your webex help, I'll pay back by adding nice docs for the community :) | 19:28 |
| rwellum_ | vhosakot, I've tried on bare-metal, on a VM, with both 'deployment' and 'development' - where deployment is really for me just running through the steps manually. | 19:28 |
| inc0 | if there are issues - we'll deal with issues | 19:28 |
| inc0 | yay local path in action | 19:29 |
| robcresswell | inc0: Cool, okay | 19:30 |
| inc0 | robcresswell: https://docs.openstack.org/developer/kolla/image-building.html#plugin-functionality | 19:30 |
| vhosakot | rwellum_: I might also deploy the playbooks in https://github.com/att-comdev/halcyon-kubernetes on my bare-metal VMs, I'll keep you posted. | 19:31 |
| robcresswell | inc0: Thanks | 19:31 |
| inc0 | or rather https://docs.openstack.org/developer/kolla/image-building.html#dockerfile-customisation | 19:31 |
| inc0 | this is guide which will show you how to modify stuff we install in containers | 19:31 |
| *** rwallner has joined #openstack-kolla | 19:31 | |
| kfox1111 | inc0: thanks for the review. | 19:32 |
| inc0 | yeah | 19:32 |
| *** rwellum_ has quit IRC | 19:32 | |
| sbezverk | inc0: if you have time I need two PS to review, kfox1111 has already reviewed them.. | 19:32 |
| inc0 | I'll do it in a few Serguey, I really want to get back to k8s installation | 19:32 |
| kfox1111 | sbezverk: I think if mine goes in before yours, it should just work? | 19:33 |
| sbezverk | https://review.openstack.org/443300 and https://review.openstack.org/443375 | 19:33 |
| kfox1111 | if your goes in first, I think a rebase will be needed. | 19:33 |
| *** rwellum has joined #openstack-kolla | 19:33 | |
| inc0 | we need this frackin' guide;) to rwellum's point | 19:33 |
| sbezverk | kfox1111: np I can go after yours | 19:33 |
| inc0 | and I really want ansible playbook to deploy k8s that just works:( | 19:33 |
| rwellum | inc0, +1 | 19:34 |
| rwellum | inc0, vagrant adds another layer | 19:34 |
| inc0 | ok, I'll ask again here maybe | 19:34 |
| inc0 | I don't want vagrant | 19:34 |
| inc0 | I have openstack installation running | 19:34 |
| inc0 | I'll spawn 5 vms there | 19:34 |
| inc0 | clean xenial | 19:34 |
| inc0 | I have ceph running on the side | 19:34 |
| inc0 | I easy way to install working k8s connected to this ceph | 19:35 |
| inc0 | ideas? | 19:35 |
| inc0 | kargo doesn't work for me, it seems half-baked | 19:35 |
| inc0 | I need to test more halcyon plays | 19:35 |
| rwellum | vhosakot, ack to trying on bare metal | 19:36 |
| vhosakot | rwellum: cool | 19:36 |
| kfox1111 | inc0: if you don't want it to be production grade, kubeadm is really nice. | 19:36 |
| inc0 | kfox1111: how to orchiestrate kubeadm? | 19:36 |
| kfox1111 | inc0: someone in meat space. :) | 19:37 |
| inc0 | yay. | 19:37 |
| rwellum | kfox1111, inc0 the current etherpad design uses kubeadm - you probably know that. | 19:37 |
| kfox1111 | its not a hard/timeconsuming thing. | 19:37 |
| *** salv-orl_ has quit IRC | 19:37 | |
| sbezverk | kfox1111: if you specify some parameters for kubeadm, I do not know why you cannot call it even production greade | 19:37 |
| inc0 | kfox1111: you have any docs/quickstart handy? | 19:37 |
| sbezverk | inc0: I was using kuberenetes kubeadm lage | 19:38 |
| sbezverk | page | 19:38 |
| *** gfhellma has quit IRC | 19:38 | |
| sbezverk | and it worked for me with slight modifications | 19:38 |
| *** salv-orlando has joined #openstack-kolla | 19:38 | |
| inc0 | right | 19:38 |
| kfox1111 | inc0: https://kubernetes.io/docs/getting-started-guides/kubeadm/ | 19:38 |
| inc0 | I'll probably write quick playbook for it | 19:38 |
| inc0 | yet another | 19:38 |
| inc0 | you deploy with calico? | 19:39 |
| inc0 | or which networking overlay? | 19:39 |
| kfox1111 | cannal | 19:39 |
| kfox1111 | which is calico+flannel | 19:39 |
| sbezverk | +1 for canal | 19:39 |
| vhosakot | rwellum: sbezverk, so how (ehterpad vs other ways) we deploy kolla-kubernetes does not matter, right? does it? I might also try the k8s official docs to deploy multi-node on centos - https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/ | 19:39 |
| kfox1111 | I'd like to get canal's plugin added to helm. | 19:39 |
| kfox1111 | I have helm running in a static pod on one of my systems now. works great. :) | 19:40 |
| sbezverk | vhosakot: use kubeadm installation | 19:40 |
| kfox1111 | so should be able to use helm to bootstrap most of k8s. | 19:40 |
| sbezverk | the link kfox1111 pasted earlier | 19:40 |
| vhosakot | sbezverk: ok, thanks. will do. | 19:40 |
| inc0 | ok, I'll get to it after lunch | 19:41 |
| sbezverk | vhosakot: there is just one thing with kubeadm it uses bunch of default ranges | 19:41 |
| srwilkers | o/ | 19:41 |
| sbezverk | I suggest to check those ranges before you run it | 19:41 |
| inc0 | hey srwilkers, hows life? | 19:41 |
| srwilkers | inc0: always exciting. how about you? | 19:41 |
| inc0 | I'm hungry:( | 19:42 |
| vhosakot | sbezverk: cool, which range? the cidr range? | 19:42 |
| sbezverk | here is command line I used: | 19:42 |
| sbezverk | kubeadm init --pod-network-cidr 10.57.0.0/19 --service-cidr 10.57.32.0/19 --service-dns-domain openstack.kolla. | 19:42 |
| vhosakot | sbezverk: I'll try https://kubernetes.io/docs/getting-started-guides/kubeadm/. | 19:42 |
| vhosakot | cool | 19:42 |
| inc0 | ehh...can you add whole subnet to neutron address-pairs? | 19:43 |
| vhosakot | sbezverk: one more question, did you deploy manually or run the Ansibke playbooks in https://github.com/att-comdev/halcyon-kubernetes on your cluster VM? | 19:43 |
| sbezverk | vhosakot: everything was done manaully | 19:43 |
| vhosakot | sbezverk: perfect, thank your all! :) | 19:44 |
| sbezverk | inc0: was it a question to me? | 19:44 |
| inc0 | no, it was question to anyone who knows neutron;) | 19:44 |
| vhosakot | sbezverk: I'll get a working kubeadm deploy on my CentOS VM and ping you guys later to help with contribution and testing, thanks guys! :) | 19:44 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Helm 2.2 and localpath https://review.openstack.org/436946 | 19:45 |
| *** rabel has joined #openstack-kolla | 19:46 | |
| kfox1111 | inc0: I don't understand the question. | 19:46 |
| kfox1111 | which neutron address-pairs are you talking about? | 19:46 |
| inc0 | kfox1111: let say I have vm with 2 ports in subnets 192.168.1.0/24 and 2.0/24 | 19:46 |
| inc0 | kfox1111: of nodes on top of which I'll install k8s | 19:46 |
| kfox1111 | k | 19:47 |
| vhosakot | inc0: don't think so. you can however add multiple addresses like "--allowed-address-pairs type=dict list=true ip_address=1.1.1.2,ip_address=1.1.1.3,ip_address=1.1.1.4" | 19:47 |
| inc0 | because I assume that if I'll add pod network cidr 10.57.0.0/24, neutron will block that | 19:47 |
| *** sayanta__ has joined #openstack-kolla | 19:47 | |
| inc0 | oh joy.. | 19:47 |
| kfox1111 | inc0: if you use canal, it tunnels the traffic through vxlans. | 19:47 |
| kfox1111 | so you only need the one subnet. | 19:47 |
| sbezverk | kfox1111: with local-path ps, any changes required to templates? | 19:48 |
| rwellum | vhosakot, so far running the dev environment on bare metal has been MUCH healthier than on a VM... | 19:48 |
| vhosakot | yes, neutron's knowledge of what subnet is used comes thru the CIDR when the subnet is created... allowed address pair is per-address. | 19:48 |
| inc0 | ok, I can use regular ports and subnets on it and canal will create vxlans? that'll work | 19:48 |
| kfox1111 | sbezverk: no. it just adds the requriements.yaml. | 19:48 |
| inc0 | (note...change mtu) | 19:48 |
| rwellum | vhosakot, have you enabled host-passthrough? | 19:48 |
| vhosakot | inc0: you can add multiple subnets (and cidr) for the same network though. | 19:48 |
| inc0 | vhosakot: yeah but it will probably block all besides port-associated ip anyway | 19:48 |
| inc0 | kfox1111: in my case, kubeadm+canal will work then right? | 19:49 |
| vhosakot | inc0: yes, that's is default security/anti-spoofing thing that neutron does OTB. | 19:49 |
| kfox1111 | inc0: I'm saying, I think you only need one neutron subnet. | 19:49 |
| inc0 | cool | 19:49 |
| kfox1111 | and needs a security group to allow the vxlan traffic. | 19:49 |
| inc0 | I have pretty lax security groups;) | 19:49 |
| inc0 | devenv | 19:49 |
| kfox1111 | k. :) | 19:50 |
| inc0 | accept all;) | 19:50 |
| *** sayantan_ has quit IRC | 19:50 | |
| inc0 | ok, maybe that was part of problem | 19:50 |
| inc0 | I tried to use calico | 19:50 |
| inc0 | so probably nto tunneled | 19:50 |
| kfox1111 | yeah. calico does routing, which is blocked by neutron by dfeault. | 19:50 |
| vhosakot | rwellum: are you talking about cpu_mode=host-passthrough in nova.conf? | 19:50 |
| kfox1111 | you need the port security extention and disable it on the ports otherwise. | 19:51 |
| vhosakot | kfox1111: I've been playing with calico which comes OTB in kubernetes kargo (https://github.com/kubernetes-incubator/kargo).. yes, neutron drops all packets involved in the BGP routing... options would be allow BDP ranges thru allowed-address-pairs, or diable port-security as inc0 mentions. | 19:52 |
| *** adrian_otto1 has joined #openstack-kolla | 19:52 | |
| *** david-lyle has joined #openstack-kolla | 19:53 | |
| vhosakot | inc0: operators might not like to disable port-security as it is risky. | 19:53 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot https://review.openstack.org/443300 | 19:53 |
| rwellum | vhosakot, no at a more basic level, in your VM definition you need host-passthrough otherwise the nested virtualization won't work. | 19:53 |
| kfox1111 | vhosakot: or option 3, use canal. :) | 19:53 |
| inc0 | vhosakot: operators can figure it out;) | 19:54 |
| inc0 | canal seems like fun | 19:54 |
| inc0 | trying now | 19:54 |
| rwellum | vhosakot, I've had many issues with this no realizing that I reverted my VM and lost that setting | 19:54 |
| *** adrian_otto has quit IRC | 19:56 | |
| vhosakot | rwellum: ah, you're talking about enabling host-passthrough in the host VM to enable nested virtualization (VM-in-VM for kolla-k8s)? | 19:57 |
| vhosakot | rwellum: let me check me host virt configs. | 19:57 |
| kfox1111 | sbezverk: sorry. nova-compute-ironic failed to merge. I thought it would be ok. | 19:58 |
| vhosakot | rwellum: where is that libvirt setting? Is it cpu mode='host-passthrough' in /usr/share/libvirt/cpu_map.xml? If so, I don't have it. | 20:00 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 20:01 |
| sbezverk | kfox1111: srwilkers: had to rebase one PS, please re-ack it when you have a moment.. | 20:02 |
| kfox1111 | sbezverk: done. | 20:02 |
| sbezverk | kfox1111: np, it was an easy fix | 20:02 |
| vhosakot | rwellum: let me know where that libvirt setting (host-passthrough) is? I usually check "grep -i vmx /proc/cpuinfo" and "lsmod | grep kvm" if my host can do virtualization and create VMs using a hypervisor. | 20:04 |
| *** manheim has joined #openstack-kolla | 20:04 | |
| rwellum | vhosakot, in my XML definition I make sure I have: <cpu mode='host-passthrough'> | 20:05 |
| vhosakot | rwellum: ah, I don't have it.. should I restart libvirt after adding it? | 20:06 |
| *** salv-orlando has quit IRC | 20:08 | |
| *** manheim has quit IRC | 20:08 | |
| rwellum | vhosakot, I just suspended and resumed my VM - but I assume there's a less invasive way of doing this... | 20:09 |
| *** vhosakot_ has joined #openstack-kolla | 20:09 | |
| vhosakot_ | the summit sched is up guys - https://www.openstack.org/summit/boston-2017/summit-schedule#day=2017-05-08 | 20:09 |
| *** kbaegis has quit IRC | 20:11 | |
| *** gfhellma has joined #openstack-kolla | 20:11 | |
| *** vhosakot has quit IRC | 20:11 | |
| rwellum | vhosakot_, did you see note on VM above - looks like you timed out for a few? | 20:12 |
| vhosakot_ | rwellum: yep, my wifi dropped, saw your message about suspending and resuming the VM.. cool.. I'll add it, thanks for the info! | 20:17 |
| rwellum | vhosakot_, ack | 20:17 |
| vhosakot_ | rwellum: cool! | 20:17 |
| rwellum | sbezverk, I got past the docker issues on bare-metal! But now my pods are not coming up. | 20:20 |
| *** mgoddard has quit IRC | 20:29 | |
| *** dave-mccowan has quit IRC | 20:30 | |
| *** kbaegis has joined #openstack-kolla | 20:32 | |
| kbaegis | Hi all. So I see that you're supposed to provide your own netboot kernel for ironic? | 20:33 |
| rwellum | vhosakot_, btw - not a bad work-flow is to bring up k8's via that k8s official doc. Then when sane, jump into the dev doc post vagrant step. | 20:33 |
| kbaegis | '"Unable to find '/etc/kolla/config/ironic/ironic-agent.kernel' in expected paths'? | 20:33 |
| kbaegis | Any documentation on the kernel generation? | 20:33 |
| kbaegis | oh, I use the diskimage-builder project? | 20:34 |
| kbaegis | coolbeanw | 20:34 |
| vhosakot_ | rwellum: yes, sbezverk told we need to deploy kubeadm for kolla-k8s. so, I''ll follow the k8s official docs to deploy kubeadm on CentOS (https://kubernetes.io/docs/getting-started-guides/kubeadm/), and then use the post-vagrant steps to continue dev, very good point. | 20:35 |
| kbaegis | Okay, no space left on device keeps showing up for cinder | 20:38 |
| kbaegis | cinder-volume | 20:38 |
| *** ccesario__ has quit IRC | 20:38 | |
| kbaegis | switched from aufs to btrfs. Is ubuntu-source broken for stable/ocata? | 20:39 |
| rwellum | vhosakot_, I got to the last stage of the dev guide doing that, but had docker issues. And then this week those performance issues. But will be good to see how far you get :) | 20:41 |
| vhosakot_ | rwellum: yep, I'll keep you and the team posted! :) | 20:41 |
| kbaegis | https://bugs.launchpad.net/kolla/+bug/1671447 updated | 20:44 |
| openstack | Launchpad bug 1671447 in kolla "iscsi container fails to start, "no space left on device"" [Undecided,New] | 20:44 |
| kbaegis | Can someone please take a look? | 20:45 |
| *** dave-mccowan has joined #openstack-kolla | 20:45 | |
| *** dave-mcc_ has joined #openstack-kolla | 20:47 | |
| *** Jezekus has joined #openstack-kolla | 20:48 | |
| sdake | portdirect ure talk got accepted ;) | 20:49 |
| *** dave-mccowan has quit IRC | 20:50 | |
| vhosakot_ | rwellum: sbezverk: I convinced the vagrant devs in hte vagrant IRC channel to implement a new command "vagrant top" (similar to "docker top") to check the CPU and memory usage of the vagrant VMs. we can use "vagrant top" to see CPU/memory stats on the vagrant host instead of ssh'ing into each vagrant vm and running the "top" command. I'll submit the proposal today. | 20:53 |
| *** nathharp has joined #openstack-kolla | 20:53 | |
| *** m1dev has joined #openstack-kolla | 20:54 | |
| rwellum | vhosakot_, :) | 20:55 |
| *** JoseMello has quit IRC | 20:55 | |
| *** jrobinson has joined #openstack-kolla | 20:57 | |
| *** rwallner has quit IRC | 21:00 | |
| *** gfhellma has quit IRC | 21:03 | |
| *** rwallner has joined #openstack-kolla | 21:03 | |
| inc0 | ok guys, halcyon-ansible is no bueno for me;) | 21:04 |
| inc0 | I now have 3 master nodes of k8s;) separately | 21:04 |
| vhosakot_ | inc0: how did you deploy k8s without halcyon-ansible? use https://kubernetes.io/docs/getting-started-guides/kubeadm/ ? | 21:06 |
| *** Jezekus has quit IRC | 21:07 | |
| inc0 | vhosakot_: I used it | 21:07 |
| *** gfhellma has joined #openstack-kolla | 21:07 | |
| inc0 | and it deployed 3 separate clusters;) | 21:07 |
| inc0 | so I'll need to rework it probably | 21:07 |
| inc0 | with kubeadm it's going to be easy | 21:07 |
| *** rwallner has quit IRC | 21:08 | |
| inc0 | that or I'll just create simple playbook to just wrap kubeadm | 21:08 |
| inc0 | and make it part of kolla-ansible mwhahaha | 21:09 |
| *** salv-orlando has joined #openstack-kolla | 21:09 | |
| *** zioproto has joined #openstack-kolla | 21:09 | |
| vhosakot_ | inc0: haha, three clusters.. there is sample inventory in https://github.com/att-comdev/halcyon-kubernetes/blob/master/sample-inventory that deploys 1 master 3 workers. | 21:11 |
| inc0 | so I wanted proper ha | 21:11 |
| inc0 | right? | 21:11 |
| inc0 | ;) | 21:11 |
| inc0 | but I can deploy 1 controller I guess | 21:12 |
| vhosakot_ | multi-cluster, multi-master _proper_ HA inc0, yes :) | 21:12 |
| vhosakot_ | inc0: 1 controller meaning 1 worker? | 21:12 |
| inc0 | I have 3 aios | 21:12 |
| inc0 | but I think I know what I did wrong | 21:13 |
| inc0 | in halcyon master != controller | 21:13 |
| inc0 | 1 master and 3 controllers should be fine | 21:13 |
| inc0 | let's try that | 21:13 |
| sdake | inc0 sbezverk, portdirect, and srwilkers's talk was accepted | 21:13 |
| inc0 | yay | 21:13 |
| sdake | inc0 also we apparently ahve a 4 person project update session that we got roped into :) | 21:13 |
| sdake | are you to blame for that | 21:13 |
| inc0 | not sure about 4 person, but I submitted it yes | 21:13 |
| sdake | cool | 21:14 |
| inc0 | results are published? | 21:14 |
| sdake | also the panel was accepted | 21:14 |
| sdake | yes it was accepted | 21:14 |
| sdake | yes they are published | 21:14 |
| jascott1 | ah that is "good" news | 21:14 |
| inc0 | link plz | 21:14 |
| sdake | sup jascott1 ;-) | 21:14 |
| jascott1 | whasup! | 21:14 |
| vhosakot_ | inc0: I think the master as the controller as it runs the kube-controller-manager process. | 21:16 |
| inc0 | yeah that might be | 21:16 |
| vhosakot_ | sdake, inc0 sbezverk, portdirect, and srwilkers, congrats to all... I'll supply kolla stickers to attendees lol ;) | 21:18 |
| inc0 | vhosakot_: we have plenty, and I hope we'll get more | 21:19 |
| vhosakot_ | inc0: nice | 21:19 |
| inc0 | kfox1111: I guess you'll be presenting project update with us;) | 21:19 |
| sdake | gotta roll | 21:19 |
| srwilkers | i came home from atlanta with quite a few ;) | 21:19 |
| *** jtriley has quit IRC | 21:19 | |
| inc0 | I like that we have session - workshop - made by people I don't know:) | 21:20 |
| inc0 | qwang, jascott1 panel was also accepted | 21:21 |
| rwellum | Those stickers were hard to get hold of at the PTG - the table was a swarm.. | 21:22 |
| *** nathharp has quit IRC | 21:22 | |
| rwellum | I didn't deserve one though as I can't get kolla up... | 21:23 |
| kbaegis | Guys, I can't get cinder-volume to start | 21:23 |
| kbaegis | Theres space on disk, yet I get a 'no space left on device' warning | 21:23 |
| kbaegis | I followed the storage doc | 21:24 |
| kbaegis | Namely https://docs.openstack.org/developer/kolla-ansible/cinder-guide.html#for-ubuntu-and-lvm2-iscsi | 21:24 |
| sbezverk | kfox1111: have you noticed gate is all red | 21:25 |
| sbezverk | even python27 is red | 21:25 |
| kfox1111 | sbezverk: no. have a link? | 21:25 |
| srwilkers | yeah, saw that :( | 21:27 |
| sbezverk | kfox1111: https://review.openstack.org/#/c/443375/ | 21:27 |
| sbezverk | kfox1111: there is any reference in my ps to nova-compute-ironic-daemonset, but helm by some reason still tries to render it | 21:28 |
| kfox1111 | sbezverk: oh. | 21:28 |
| kfox1111 | I think maybe we have a 2.0 reference somewhere still somhow? | 21:29 |
| sbezverk | nope, just checked helm is 2.2.2 | 21:30 |
| kfox1111 | hmm... | 21:30 |
| *** eaguilar has quit IRC | 21:30 | |
| sbezverk | kfox1111: what I do not get is if in my ps there is no any reference to old name, where it takes it?!?!? | 21:30 |
| kfox1111 | yeah. shouldnt... | 21:31 |
| kfox1111 | oh... | 21:31 |
| *** manheim has joined #openstack-kolla | 21:31 | |
| kfox1111 | I think the ps needs to rename the requirements.yaml file over too. | 21:31 |
| *** rabel has quit IRC | 21:31 | |
| manheim | hello | 21:32 |
| manheim | i have a question about setting a proxy in cotainers | 21:32 |
| manheim | i saw the documentation on how to do that | 21:32 |
| manheim | can I set that only once somewhere and the containers will inherit that proxy? | 21:32 |
| sbezverk | kfox1111: it looks like there is no requirements.yaml at all in the folder | 21:33 |
| manheim | or do i have to each container and set it in the dockerfile lets say | 21:33 |
| *** l4yerffeJ has joined #openstack-kolla | 21:34 | |
| *** l4yerffeJ_ has quit IRC | 21:34 | |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 21:35 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 21:36 |
| *** dave-mcc_ has quit IRC | 21:36 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 21:38 |
| *** krtaylor has quit IRC | 21:42 | |
| *** rhallisey has quit IRC | 21:45 | |
| *** dave-mccowan has joined #openstack-kolla | 21:46 | |
| *** markmcclain has joined #openstack-kolla | 21:49 | |
| *** MarginHu has joined #openstack-kolla | 21:51 | |
| *** jtriley has joined #openstack-kolla | 21:52 | |
| *** vhosakot_ has quit IRC | 21:56 | |
| *** vhosakot has joined #openstack-kolla | 21:56 | |
| kbaegis | So no one else has seen that? The "no space left on device"? inc0 sdake? | 21:58 |
| *** fguillot has quit IRC | 21:59 | |
| inc0 | kbaegis: frankly I didn't test iscsi - I always deploy with ceph | 21:59 |
| inc0 | sbezverk: can you help here? | 22:00 |
| sbezverk | inc0: yep | 22:01 |
| MarginHu | I use iscsi | 22:01 |
| sdake | sup peeps | 22:01 |
| MarginHu | kbaegis , I didn't experience your issue. | 22:01 |
| sbezverk | kbaegis: can you provide a bot more context to the problem? | 22:02 |
| sbezverk | kbaegis: nvm please check volume group | 22:03 |
| sbezverk | kbaegis: it should be big enough to hold the volume you try to create | 22:03 |
| kbaegis | sbezverk: It's failing on the container start. There's 12+tb of unallocated space | 22:05 |
| kbaegis | aufs and btrfs both failing | 22:05 |
| kbaegis | I might try ceph, but I really need the iops for a demo so ceph concerns me (AIO) | 22:06 |
| inc0 | kbaegis: ceph is easier in our case than iscsi;) | 22:06 |
| kbaegis | all detailed here https://bugs.launchpad.net/kolla/+bug/1671447 | 22:06 |
| openstack | Launchpad bug 1671447 in kolla "iscsi container fails to start, "no space left on device"" [Undecided,New] | 22:06 |
| inc0 | as counter-intuitive as it sounds | 22:06 |
| kbaegis | inc0: Makes sense. ceph is awesome if you actually have multiple nodes | 22:06 |
| kbaegis | ooc, is there.a config option to set the default storage pool replication factor? | 22:07 |
| kbaegis | In case I do want ceph+AIO? | 22:07 |
| inc0 | kbaegis: https://docs.openstack.org/developer/kolla-ansible/ceph-guide.html#configuration <- you have tip how to do ai | 22:07 |
| inc0 | o | 22:07 |
| kbaegis | Ahh, advanced config | 22:08 |
| inc0 | "advanced" | 22:08 |
| *** vhosakot_ has joined #openstack-kolla | 22:08 | |
| kbaegis | makes sense. Trying a redeploy after a reboot, but I'll give ceph a go after this and update the bug | 22:08 |
| kbaegis | inc0: Thanks for the tip | 22:08 |
| inc0 | np | 22:08 |
| inc0 | only caveat is that ceph wants dedicated block device | 22:09 |
| inc0 | I know it can be done with loopback, but never done it myself | 22:09 |
| sbezverk | kbaegis: can you paste vgs | 22:09 |
| kbaegis | inc0: I'm not using my cinder drives for anything yet | 22:09 |
| kbaegis | sbezverk: Thats in there | 22:09 |
| inc0 | well if you use ceph for them | 22:10 |
| inc0 | that's actually more prod-ready than iscsi | 22:10 |
| sbezverk | kbaegis: sorry in where? | 22:10 |
| inc0 | (other than it's aio...) | 22:10 |
| *** manheim has quit IRC | 22:10 | |
| kbaegis | sbezverk: In the bug | 22:10 |
| kbaegis | https://hastebin.com/bocarucile.scala | 22:11 |
| kbaegis | Just renamed cinder-hdd-volumes to cinder-volumes and mirrored change in globals | 22:11 |
| kbaegis | But that's the same | 22:11 |
| *** vhosakot has quit IRC | 22:11 | |
| *** fooliouno has quit IRC | 22:12 | |
| sbezverk | kbaegis: can you run vgs | 22:12 |
| sbezverk | what you pasted is not what I asked.. | 22:12 |
| sbezverk | also cinder service-list | 22:12 |
| sbezverk | and cinder.conf would be helpfull | 22:12 |
| kbaegis | https://hastebin.com/rinuvibumi.css | 22:13 |
| kbaegis | cinder volumes doesn't start | 22:13 |
| inc0 | yay kube multinode deployed | 22:13 |
| kbaegis | inc0: Congrats | 22:14 |
| inc0 | now I need to figure out how to connect ceph to it | 22:14 |
| inc0 | kfox1111: little help here?:) | 22:14 |
| kfox1111 | reading back. | 22:16 |
| kfox1111 | ah. | 22:16 |
| kfox1111 | sec... | 22:16 |
| kfox1111 | yeah. I have time now. | 22:16 |
| inc0 | cool, so my statys | 22:16 |
| inc0 | I have k8s | 22:16 |
| *** rwallner has joined #openstack-kolla | 22:16 | |
| inc0 | and I have ceph | 22:16 |
| inc0 | and they don't know each other | 22:17 |
| kfox1111 | and ceph is accessable to k8s? (just double checking. :) | 22:17 |
| kfox1111 | (connectivity) | 22:17 |
| sbezverk | kfox1111: have you seen this error? http://logs.openstack.org/75/443375/8/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/abbfc8a/console.html#_2017-03-09_22_16_06_671439 | 22:17 |
| inc0 | yeah, same network | 22:17 |
| kfox1111 | k :) | 22:17 |
| kfox1111 | sbezverk: looking | 22:17 |
| inc0 | where do I put user/keyring stuff? | 22:17 |
| sbezverk | kfox1111: I see it on 80% of jobs | 22:17 |
| kfox1111 | sbezverk: thats new. looks like they forgot to sign a new package it seems. | 22:17 |
| kfox1111 | sbezverk: we really need to mirror our k8s packages somewhere to ahve a stable version. :/ | 22:18 |
| inc0 | btw zuul v3 will be ansible based, we can leverage halcyon playbook in gates | 22:18 |
| kfox1111 | sbezverk: you could try droping the gpg check on the repo. | 22:18 |
| kfox1111 | inc0: interesting... | 22:19 |
| inc0 | indeed, I'll wait for it to create kolla-ansible multinode gates | 22:19 |
| sbezverk | kfox1111: what is strange that for some jobs it works, I guess there is an inconsistency between mirrors | 22:20 |
| kfox1111 | sbezverk: likely. I've seen that happen a bit with their repo. | 22:20 |
| kfox1111 | inc0: so, the gate uses this: tools/setup_gate_ceph.sh | 22:21 |
| *** rwallner has quit IRC | 22:21 | |
| kfox1111 | inc0: we probably should start at line 111. | 22:21 |
| kfox1111 | though probaly need a step or two before that for your case. | 22:21 |
| kfox1111 | so, the build_test_ceph.sh script builds a test script and uploads a configmap/secret to bind to ceph. | 22:22 |
| kfox1111 | we need to build those by hand I think. | 22:22 |
| inc0 | oh insteresing, so we run ceph client in container | 22:22 |
| inc0 | that's nice | 22:22 |
| kfox1111 | yup. :) | 22:22 |
| kfox1111 | ceph isn't installed on the host at all. | 22:23 |
| kfox1111 | even for the k8s support. :) | 22:23 |
| inc0 | right | 22:23 |
| masber | hi | 22:23 |
| inc0 | cool I'll dig through it | 22:23 |
| kfox1111 | so, if we look in tests/bin/build_test_ceph.sh | 22:23 |
| masber | does kolla-ansible has the docker registry integrated? | 22:23 |
| kfox1111 | we have: | 22:23 |
| inc0 | masber: integrated - not really, but it's really one liner | 22:23 |
| masber | I can't find the tools/start-registry | 22:23 |
| kfox1111 | kubectl create configmap ceph-conf --namespace=kolla --from-file=ceph.conf=/etc/ceph/ceph.conf | 22:23 |
| masber | im following this doc https://docs.openstack.org/developer/kolla-ansible/multinode.html | 22:24 |
| kfox1111 | so the first part of the instructions should be, get a copy of ceph.conf from the ceph cluster, to your machine with kubectl, | 22:24 |
| kfox1111 | and run: | 22:24 |
| kfox1111 | kubectl create configmap ceph-conf --namespace=kolla --from-file=ceph.conf=<your cluster ceph.conf> | 22:24 |
| inc0 | masber: it's in kolla, not kolla-ansible | 22:24 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 22:24 |
| inc0 | geenrally - build == kolla, deploy == kolla-ansible | 22:25 |
| inc0 | and keyring goes to secrets right? | 22:25 |
| kfox1111 | inc0: yeah. thats next. we need accounts into secrets. | 22:26 |
| masber | doesn't looks like it is in /usr/share/kolla/tools | 22:26 |
| kfox1111 | those come from tools/setup-ceph-secrets.sh | 22:26 |
| kfox1111 | I think all we need is ceph.client.admin.keyring; | 22:26 |
| inc0 | right | 22:26 |
| inc0 | I'll play around | 22:27 |
| inc0 | first - get helm working | 22:27 |
| masber | this is how I installed kolla pip install kolla==3.0.2 | 22:27 |
| kfox1111 | so, like: | 22:27 |
| inc0 | masber: I'm not sure if tools/start-registry was present in newton | 22:27 |
| inc0 | as again...one line | 22:27 |
| inc0 | yup, not present in newton | 22:27 |
| deadnull | Is CoreOS/rkt going to (or already) a supported route to take? | 22:28 |
| masber | yeah it is not present | 22:28 |
| masber | so I will have to install the registry by myself | 22:28 |
| inc0 | deadnull: well, that's a good question | 22:28 |
| *** adrian_otto1 has quit IRC | 22:28 | |
| deadnull | inc0 :) | 22:28 |
| masber | but actually kolla can deploy without a registry am I right? | 22:28 |
| kfox1111 | well... | 22:28 |
| inc0 | doing r&d how much it would take to run kolla on rkt is on my list to do | 22:28 |
| kfox1111 | I think we have to figure this step out a bit. | 22:28 |
| inc0 | masber: yeah | 22:28 |
| masber | kk | 22:28 |
| deadnull | inc0 I am trying to free up some cycles, I might be interested in helping you out | 22:29 |
| inc0 | I think our images should work | 22:29 |
| inc0 | kolla-ansible lacks support for rkt | 22:29 |
| deadnull | right, that seems like it would be where the lionshare of work is | 22:29 |
| inc0 | but that's doable | 22:29 |
| masber | ok i will check that | 22:29 |
| inc0 | not necessarly | 22:29 |
| inc0 | we have kolla-docker ansible module | 22:29 |
| kfox1111 | inc0: do you remember the output of FETCH_CEPH_KEYS? | 22:29 |
| inc0 | if we rename it to kolla-runtime (or sth) and provide equivalent commands to kolla_docker | 22:30 |
| inc0 | kfox1111: no:() | 22:30 |
| inc0 | it might actually be pretty easy | 22:30 |
| inc0 | or very hard | 22:30 |
| inc0 | depends on how much rkt differs from docker in our use cases | 22:30 |
| kfox1111 | looks like it walks /etc/ceph/*, and just passes the whole file out. | 22:30 |
| *** adrian_otto has joined #openstack-kolla | 22:30 | |
| inc0 | one thing tho, if our images aren't OCI-compatible, I would like them to be OCI-compatible | 22:31 |
| *** schwicht has quit IRC | 22:31 | |
| inc0 | yeha I just copy files around | 22:31 |
| *** lucasxu has quit IRC | 22:31 | |
| kfox1111 | inc0: so, tools/setup-ceph-secrets.sh just fetches the output, and unencodes them. | 22:31 |
| inc0 | so I'll dig through our setup gate stuff | 22:31 |
| kfox1111 | so looks like they are in full keytab format. | 22:31 |
| inc0 | one thing I learned - I need to setup helm and kolla repos first | 22:31 |
| *** jtriley has quit IRC | 22:31 | |
| inc0 | so I'll focus on that | 22:32 |
| kbaegis | Sooo | 22:32 |
| kfox1111 | so, should be able to copy the admin key somewhere that kubectl has access, then: | 22:32 |
| kbaegis | Renaming it cinder-volumes worked | 22:32 |
| deadnull | inc0 cool sounds good. will you be at the Summit in Boston? | 22:32 |
| inc0 | I think so | 22:32 |
| kbaegis | So the configuration value for cinder_volume_group isn't properly implemented | 22:32 |
| inc0 | I hope to have some answers by then | 22:32 |
| kfox1111 | kubectl create secret generic ceph.client.admin.keyring --namespace=kolla --from-file=data=<your admin keyring> | 22:33 |
| kbaegis | Can I mark a bug a defect, or does someone else have to do that? | 22:33 |
| deadnull | cool | 22:33 |
| inc0 | if our images are oci-compatible, kolla-kubernetes should just work | 22:33 |
| *** goldyfruit has quit IRC | 22:33 | |
| *** manheim has joined #openstack-kolla | 22:33 | |
| *** MarginHu has quit IRC | 22:34 | |
| kfox1111 | inc0: what version of ceph you got? | 22:35 |
| inc0 | jewel | 22:36 |
| kfox1111 | k. | 22:36 |
| kfox1111 | do you have anything in it you care about? :) | 22:36 |
| masber | another thing | 22:37 |
| masber | putting this --> http://pastebin.com/7epXvP1a on my /etc/systemd/system/docker.service.d/kolla.conf file, breaks docker | 22:37 |
| masber | i am using centos7 and docker Docker version 1.12.6, build 78d1802 | 22:37 |
| kfox1111 | inc0: ok, I think this might do the trick, but do double check what its running if you care about your data: | 22:38 |
| kfox1111 | http://paste.openstack.org/show/smlkVK54cUxk3IjBmBup/ | 22:38 |
| *** gfhellma has quit IRC | 22:40 | |
| *** gfhellma_ has joined #openstack-kolla | 22:40 | |
| sbezverk | kfox1111: disabling gpg check helped.. but I lost again your +2, could you ack this ps? https://review.openstack.org/443375 | 22:40 |
| sbezverk | kfox1111: when the gate finishes I will ask inc0 for a final bullet ;) | 22:40 |
| *** MarginHu has joined #openstack-kolla | 22:40 | |
| kfox1111 | sbezverk: done. | 22:40 |
| sbezverk | kfox1111: our talk got accepted :) | 22:41 |
| kfox1111 | sbezverk: sweet. :) | 22:41 |
| sbezverk | kfox1111: now is the hard part put together this talk ;) | 22:42 |
| kfox1111 | inc0: hows that look? | 22:42 |
| kfox1111 | sbezverk: indeed. :) | 22:42 |
| *** adrian_otto has quit IRC | 22:49 | |
| *** jtriley has joined #openstack-kolla | 22:50 | |
| sdake | sbezverk you will rock it i'm sure :) | 22:53 |
| *** jtriley has quit IRC | 22:55 | |
| sbezverk | sdake: you were right, received email 10 minutes ago, how you knew about it before that? | 22:56 |
| sdake | someone else got an email and i looked on the website and saw it | 22:57 |
| sbezverk | sdake: cool, I thought you might have a secret channel for board members ;) | 22:59 |
| *** sayanta__ has quit IRC | 23:00 | |
| *** schwicht has joined #openstack-kolla | 23:03 | |
| rwellum | sbezverk, which talk is this? | 23:03 |
| *** jrobinson_ has joined #openstack-kolla | 23:05 | |
| *** MarginHu has quit IRC | 23:05 | |
| *** MarginHu has joined #openstack-kolla | 23:05 | |
| sbezverk | rwellum: we submitted kolla-kube talk for this summit | 23:05 |
| *** jrobinson has quit IRC | 23:07 | |
| rwellum | I see it now: https://www.openstack.org/summit/boston-2017/summit-schedule/events/17731 | 23:10 |
| rwellum | Helps if you look at the full schedule... | 23:10 |
| *** jrobinson_ has quit IRC | 23:18 | |
| *** jrobinson_ has joined #openstack-kolla | 23:18 | |
| manheim | i have a question about setting a proxy, saw the documentation | 23:19 |
| manheim | can I set that only once somewhere and the containers will inherit that proxy? | 23:19 |
| *** jrobinson_ is now known as jrobinson | 23:19 | |
| manheim | or do i have to each container and set it in the dockerfile lets say | 23:19 |
| *** zioproto has quit IRC | 23:21 | |
| sdake | sbezverk wednesday 9am slot is decent | 23:22 |
| sdake | can't ccomplain ;) | 23:22 |
| sdake | sbezverk there is an unoffical channel for board members however it is far from secret | 23:22 |
| sdake | i found out about it via eavesdropping ;) | 23:23 |
| sdake | the talks being accepted that is | 23:23 |
| sbezverk | sdake: the review I asked you earlier, would be great help to merge it ;) | 23:24 |
| sdake | you know your on to something when 50 people cop your shit | 23:24 |
| *** MarginHu has quit IRC | 23:24 | |
| sdake | sbezverk gotta bio - will do after | 23:24 |
| sdake | after i pay my bill and slam this drink | 23:24 |
| kbaegis | Umm. So when I run mount, I get about 50000 lines of shm on . . . | 23:24 |
| *** MarginHu has joined #openstack-kolla | 23:24 | |
| kbaegis | Anyone think this is related to the "No space left on device" issue? | 23:25 |
| *** MarginHu has joined #openstack-kolla | 23:25 | |
| *** saneax-_-|AFK is now known as saneax | 23:25 | |
| kbaegis | lxd was still installed on ubuntu 16 | 23:26 |
| kbaegis | How do I fix/recover? | 23:26 |
| *** rwellum has quit IRC | 23:27 | |
| *** schwicht has quit IRC | 23:27 | |
| *** MarginHu has quit IRC | 23:28 | |
| sdake | sbezverk acked due | 23:30 |
| sdake | dude | 23:30 |
| inc0 | kfox1111: rignt now I'm getting deployment itself streamlined | 23:30 |
| inc0 | stuff like insecure address and such | 23:30 |
| sbezverk | sdake: thanks a lot | 23:31 |
| *** schwicht has joined #openstack-kolla | 23:32 | |
| *** schwicht has quit IRC | 23:33 | |
| kfox1111 | inc0: k. | 23:34 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset https://review.openstack.org/443375 | 23:34 |
| kfox1111 | inc0: well, what part of deployment are you talking about? | 23:37 |
| spsurya | morning all | 23:37 |
| inc0 | kfox1111: so I my deploy node which will have kubectl and generally won't change | 23:38 |
| inc0 | I want to be able to access it without redoing ssl cers all the time | 23:38 |
| inc0 | which means changing apiserver manifest to allow insecure on 0.0.0.0 | 23:38 |
| inc0 | trying to figure out how to do it in halcyon in a way that it will be set every time | 23:39 |
| kfox1111 | ok. | 23:39 |
| kfox1111 | this is one of the better times to discuss ceph though. | 23:39 |
| inc0 | you know how to specify configs to apiserver like that for kubeadm? | 23:39 |
| kfox1111 | not off hand. | 23:40 |
| inc0 | ok good point | 23:40 |
| inc0 | I'll manually do it | 23:40 |
| inc0 | hold on | 23:40 |
| sbezverk | inc0: what do you want to know abount api server config? | 23:40 |
| sbezverk | there is a place in etc/kubernetes/manifests/ | 23:41 |
| inc0 | sbezverk: I'm trying to figure out how to teach kubeadm to deploy k8s with listen insecure 0.0.0.0 | 23:41 |
| inc0 | yeah, but how kubeadm generates that is my question | 23:41 |
| sbezverk | inc0: you cannot | 23:41 |
| inc0 | right | 23:41 |
| sbezverk | you need to change manifest | 23:41 |
| sbezverk | at /etc/kubernetes/manifests | 23:42 |
| sbezverk | let me give you example of config | 23:42 |
| kbaegis | Restart nova-libvirt container is failing "no space left on device" | 23:42 |
| *** dave-mccowan has quit IRC | 23:43 | |
| kbaegis | Made it past cinder at least | 23:43 |
| sbezverk | inc0: see linbe 7 http://paste.openstack.org/show/602126/ | 23:43 |
| sbezverk | inc0: all you need is to add that line | 23:44 |
| sbezverk | kube automatically detect change and restart api server | 23:44 |
| openstackgerrit | Abel Lopez proposed openstack/kolla-kubernetes master: Test for kube config https://review.openstack.org/444037 | 23:44 |
| *** alop has joined #openstack-kolla | 23:44 | |
| inc0 | sbezverk: done that | 23:45 |
| inc0 | now dealing with tiller restarting | 23:45 |
| kfox1111 | inc0: kubectl describe pod tiller-xxx --namespace kube-system | 23:46 |
| kfox1111 | did canal load ok? | 23:46 |
| inc0 | yeah I know, not helpful for now | 23:46 |
| inc0 | how to check that? | 23:46 |
| kfox1111 | kubectl get pods --all-namespaces | 23:47 |
| kfox1111 | in the describe output, does it say why its failing? | 23:47 |
| inc0 | all the pods besides tiller and dashboard are running | 23:47 |
| kfox1111 | if it says stuff about cni, its calico. | 23:47 |
| kfox1111 | tiller and dashboard will run on the slave nodes I think. | 23:48 |
| kfox1111 | so it could be networking related. | 23:48 |
| inc0 | liveness probes gets connection refused | 23:48 |
| inc0 | so something network | 23:49 |
| kfox1111 | do an 'ip a show dev flannel.1' on the master, | 23:49 |
| kfox1111 | log into a slave, and ping it. | 23:49 |
| inc0 | so pinging container works | 23:51 |
| kbaegis | here's nova-compute failing to start | 23:51 |
| kbaegis | https://hastebin.com/aruliquvum.sql | 23:51 |
| inc0 | from master | 23:52 |
| masber | sorry another question | 23:52 |
| masber | in regards kolla 3.0.2 | 23:52 |
| masber | I have my kolla vm I am using for deployment | 23:53 |
| masber | but I want to install openstack control plane and nona on a different VM | 23:53 |
| *** MasterOfBugs has quit IRC | 23:53 | |
| masber | so I changed the all-in-one inventory | 23:53 |
| *** MasterOfBugs has joined #openstack-kolla | 23:53 | |
| kbaegis | there's no log made whatsoever. Docker doesn't start the container | 23:53 |
| sbezverk | inc0: when you installed canal plugin have you changed ip address for its etcd? | 23:54 |
| masber | and changed the network, control, compute and monitoring from localhost to my other VM's IP | 23:54 |
| masber | after that, do i need to run kolla-ansible -i all-in-one bootstrap-servers? | 23:54 |
| inc0 | sbezverk: no | 23:55 |
| sbezverk | inc0: when you downloaded canal.yaml there are a couple of places where you need to change addresses they use by default | 23:56 |
| *** vhosakot_ has quit IRC | 23:56 | |
| inc0 | sbezverk: I used halcyon | 23:57 |
| inc0 | but I think I see what's wrong | 23:57 |
| inc0 | tiller has strange kube address | 23:57 |
| *** jrobinson has quit IRC | 23:58 | |
| inc0 | it seems I used cluster network that's different from my api interface has | 23:58 |
| inc0 | let me try to rebuild/redeploy | 23:58 |
| sbezverk | inc0: ah I see that I do not know how it works with halcyon | 23:59 |
| *** schwicht has joined #openstack-kolla | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!