Thursday, 2017-03-09

*** salv-orlando has quit IRC00:05
*** vhosakot has quit IRC00:07
*** david-lyle has joined #openstack-kolla00:08
*** vhosakot has joined #openstack-kolla00:08
*** yingjun has joined #openstack-kolla00:09
*** jmccarthy has joined #openstack-kolla00:10
*** gfhellma has joined #openstack-kolla00:12
*** david-lyle has quit IRC00:15
*** jrobinson has joined #openstack-kolla00:16
*** lamt has quit IRC00:18
*** gfhellma has quit IRC00:20
*** haplo37 has quit IRC00:20
*** saneax is now known as saneax-_-|AFK00:22
*** goldyfruit has quit IRC00:24
*** dave-mccowan has joined #openstack-kolla00:25
*** haplo37 has joined #openstack-kolla00:29
*** jtriley has joined #openstack-kolla00:30
inc0ehh...ceilometer00:31
inc0dunno jemcevoy00:31
inc0that's out of my comfort zone00:31
*** duonghq has joined #openstack-kolla00:33
duonghqmorning guys00:33
inc0up already duonghq?:)00:33
inc0hey00:33
duonghqya00:33
duonghqjust arrive my office00:34
duonghqthank you for start voting00:34
inc0so far so good:)00:34
inc0you've earned it00:34
duonghq:) last night I cannot help much, have we cleared critical bug?00:35
inc0yeah00:36
inc0we already tagged rc00:36
inc0all of critical stuff is merged and fixed00:36
inc0as far as I know00:36
duonghqnice00:36
duonghqinc0, as development of Docker can lead to something not backward compatible, I think we should have recommend max version of docker in qsg?00:39
inc0duonghq: we did that in liberty when we were pinned00:40
inc0so far I haven't seen anything that would lock us out of 17.0300:40
inc0besides docker-py change I think, but fix for it is already proposed00:40
inc0anyway, if we hit a wall then yes, we pin max version until we teach kolla_docker module to use it00:41
inc0that being said, docker-ee worries me00:41
duonghqsure, maybe it's all about docker-py's trouble00:41
duonghqyup, it's becoming many other FOSS project way00:42
duonghq*following00:42
inc0yeah, but I'm worried that they'll lock out features from community edition00:43
inc0they have nasty habit of doing stuff like that even before they introduced ee00:44
inc0btw duonghq https://review.openstack.org/#/c/438813/00:44
inc0this is change I was talking about00:45
duonghqroger00:45
*** manheim has quit IRC00:45
duonghqwhat is your opinion about rkt?00:45
*** zhurong has joined #openstack-kolla00:46
duonghqinc0, this is my bug with 17.03: http://paste.openstack.org/show/601928/00:47
duonghqwhen I downgrade to 1.12.x, it's fine00:48
duonghqnot sure it's about docker or docker-py00:48
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155000:49
inc0duonghq: so I have this on my todo list: check if we can run our containers on rkt00:52
inc0I'd lie if I'd say I have lots of experience with it00:52
*** david-lyle has joined #openstack-kolla00:54
*** MarginHu has joined #openstack-kolla00:54
*** david-lyle has quit IRC00:54
*** Jeffrey4l_ has quit IRC01:00
duonghqinc0, but you have one hand on rkt?01:00
*** schwicht has joined #openstack-kolla01:01
*** dave-mccowan has quit IRC01:02
*** ssurana has joined #openstack-kolla01:03
*** cuongnv has joined #openstack-kolla01:06
*** daidv has joined #openstack-kolla01:06
hogepodgeinc0: if I want to get rid of ironic-inspector in my all-in-one config, how do I do that?01:08
duonghqbpattewar, ping01:09
hogepodgeinc0: it wants to start up dnsmasq, and I do not want that running, nor do I need server discovery and inspection.01:09
duonghqbpattewar_,  ping01:10
bpattewarduonghq hi01:10
*** rwsu has quit IRC01:11
*** zhubingbing_ has joined #openstack-kolla01:11
*** alop has quit IRC01:16
*** MasterOfBugs has quit IRC01:22
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337501:25
*** unicell has quit IRC01:34
*** deadnull has quit IRC01:35
*** kbaegis has joined #openstack-kolla01:37
*** salv-orlando has joined #openstack-kolla01:37
*** salv-orlando has quit IRC01:41
*** kbaegis has quit IRC01:45
*** kbaegis has joined #openstack-kolla01:45
duonghqkbaegis, so, currently you are using ubuntu for the cluster?01:45
*** adrian_otto has quit IRC01:46
*** dave-mcc_ has joined #openstack-kolla01:50
*** david-lyle has joined #openstack-kolla01:52
*** bpattewar has quit IRC01:54
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot  https://review.openstack.org/44330001:55
*** bpattewar_ has quit IRC01:58
*** tovin07_ has joined #openstack-kolla02:06
*** haplo37 has quit IRC02:12
*** trinaths1 has left #openstack-kolla02:17
*** haplo37 has joined #openstack-kolla02:21
*** erlon has quit IRC02:25
*** japestinho has joined #openstack-kolla02:28
*** salv-orlando has joined #openstack-kolla02:38
*** Jeffrey4l_ has joined #openstack-kolla02:39
openstackgerritZeyu Zhu proposed openstack/kolla-ansible master: Modify the hosts of the post-deploy.yml playbook  https://review.openstack.org/43682202:41
*** salv-orlando has quit IRC02:42
*** dave-mcc_ has quit IRC02:54
*** vhosakot has quit IRC03:00
*** daidv has quit IRC03:04
*** daidv has joined #openstack-kolla03:05
kbaegisDid https://bugs.launchpad.net/kolla/+bug/1631072 end up actually getting fixed?03:06
openstackLaunchpad bug 1631072 in kolla "iscsid container: mkdir /sys/kernel/config: operation not permitted" [High,Confirmed]03:06
*** gmann has joined #openstack-kolla03:09
*** sudipto has joined #openstack-kolla03:13
*** sudipto_ has joined #openstack-kolla03:13
*** zhubingbing_ has quit IRC03:17
*** zhubingbing_ has joined #openstack-kolla03:18
*** rwsu has joined #openstack-kolla03:23
*** ssurana has quit IRC03:26
*** daidv has quit IRC03:29
*** sudipto has quit IRC03:37
*** sudipto_ has quit IRC03:37
*** salv-orlando has joined #openstack-kolla03:39
*** salv-orlando has quit IRC03:43
*** sayantan_ has quit IRC03:58
*** g3ek has quit IRC03:58
*** sayantan_ has joined #openstack-kolla03:59
*** schwicht has quit IRC04:02
*** g3ek has joined #openstack-kolla04:08
*** l4yerffeJ__ has joined #openstack-kolla04:21
*** l4yerffeJ_ has quit IRC04:21
*** l4yerffeJ_ has joined #openstack-kolla04:26
*** l4yerffeJ__ has quit IRC04:26
*** mdnadeem has joined #openstack-kolla04:28
*** salv-orlando has joined #openstack-kolla04:39
*** salv-orlando has quit IRC04:44
*** denaitre has quit IRC04:47
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337504:47
*** rwsu has quit IRC04:49
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate  https://review.openstack.org/44155004:59
*** l4yerffeJ_ has quit IRC04:59
*** l4yerffeJ_ has joined #openstack-kolla04:59
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot  https://review.openstack.org/44330005:01
*** jrobinson has quit IRC05:03
*** fooliouno has joined #openstack-kolla05:06
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337505:07
*** l4yerffeJ__ has joined #openstack-kolla05:07
*** l4yerffeJ_ has quit IRC05:08
*** Serlex has joined #openstack-kolla05:12
*** unicell has joined #openstack-kolla05:14
*** Serlex has quit IRC05:15
*** ssurana has joined #openstack-kolla05:16
masberhi05:17
masberI can't find tools/start-registry on my kolla installation05:17
masberI am using kolla 3.0.205:17
*** unicell1 has joined #openstack-kolla05:18
masberI would like to install the docker registry05:18
*** unicell has quit IRC05:19
*** Serlex has joined #openstack-kolla05:22
*** sudipto has joined #openstack-kolla05:25
*** skramaja has joined #openstack-kolla05:25
*** jrobinson has joined #openstack-kolla05:31
*** salv-orlando has joined #openstack-kolla05:40
MarginHumasber,docker run -d -p 4000:5000 --restart=always --name registry registry:205:44
*** salv-orlando has quit IRC05:45
*** trinaths has joined #openstack-kolla05:58
*** saneax-_-|AFK is now known as saneax06:03
MarginHuhi, when I want to update dnsmasq.conf of ironic service , I found it couldn't.06:08
*** satyar has joined #openstack-kolla06:22
MarginHuI got it, yeah!06:22
*** jbadiapa has quit IRC06:23
*** trinaths1 has joined #openstack-kolla06:29
*** trinaths has quit IRC06:29
*** rwsu has joined #openstack-kolla06:31
*** salv-orlando has joined #openstack-kolla06:33
*** ssurana has quit IRC06:42
*** ssurana has joined #openstack-kolla06:45
*** rabel has joined #openstack-kolla06:51
*** bmace has quit IRC06:53
*** bmace has joined #openstack-kolla06:53
*** jrobinson has quit IRC06:54
fooliounoWhen the command "kolla-kubernetes bootstrap mariadb" is run, is the mariadb-bootstrap pod expected to terminate after completion or should it remain up and running.07:04
*** ssurana has quit IRC07:06
*** sayantan_ has quit IRC07:09
*** sayantan_ has joined #openstack-kolla07:09
duonghqfooliouno, it's job, so it's expected to finish07:13
fooliounoThanks duonghq! Another quick question. My keystone-create-db pod keeps restarting due to the following error. Any pointers on what could be wrong.07:16
fooliouno0107:16
fooliouno"msg": "unable to find /var/lib/ansible/.my.cnf. Exception message: (2003, 'Can\\'t connect to MySQL server on \\'mariadb\\' (110 \"Connection timed out\")')"07:16
duonghqcan you check is there any mariadb running in the cluster?07:17
fooliounoI verified that the secrets are generated correctly from passwords.yml for keystone07:17
duonghq(helm ls)07:17
fooliounoI listed all pods and dont see any for mariadb. Hmmm ..07:18
duonghqcan you put log of mariadb bootstrap in to some pastebin?07:19
fooliounoduonghq: http://pastebin.com/WPM0JFw707:21
*** salv-orl_ has joined #openstack-kolla07:22
duonghqhmm, and mariadb still not running?07:23
fooliounoyes .. no sign of it07:23
*** salv-orlando has quit IRC07:25
fooliounoLet me clarify. There are no mariadb pods, but there is a mariadb service running07:27
*** rabel has quit IRC07:37
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Add ironic-ui in horizon binary install  https://review.openstack.org/44345307:41
*** mnasiadka has joined #openstack-kolla07:44
*** rabel has joined #openstack-kolla07:55
MarginHuhi, question , I need to rebuild horizon image , but how to tag horizon image version ?08:03
MarginHu how to tag horizon image version  instead of whole project version ?08:03
*** matrohon has joined #openstack-kolla08:04
*** shardy has joined #openstack-kolla08:10
MarginHuJeffrey4l08:10
*** mgoddard has joined #openstack-kolla08:11
zhubingbing_sup sdake08:11
zhubingbing_https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP08:11
zhubingbing_i have success deploy kolla-k8s08:12
zhubingbing_i have test pass wth document08:12
openstackgerritMartin AndrĂ© proposed openstack/kolla master: Add heat-api-cloudwatch image  https://review.openstack.org/44307708:13
*** l4yerffeJ_ has joined #openstack-kolla08:17
*** l4yerffeJ__ has quit IRC08:17
*** jbadiapa has joined #openstack-kolla08:24
*** mgoddard has quit IRC08:28
*** gfidente has joined #openstack-kolla08:36
openstackgerritTuan Luong-Anh proposed openstack/kolla master: Indicating the location tests directory in oslo_debug_helper  https://review.openstack.org/44348808:37
openstackgerritTuan Luong-Anh proposed openstack/kolla-kubernetes master: Indicating the location tests directory in oslo_debug_helper  https://review.openstack.org/44349008:40
*** mgoddard has joined #openstack-kolla08:40
*** egonzalez has joined #openstack-kolla08:41
*** nathharp has joined #openstack-kolla08:45
egonzalezduonghq, topology tab was due a bug on horizon https://bugs.launchpad.net/horizon/+bug/167108408:47
openstackLaunchpad bug 1671084 in OpenStack Dashboard (Horizon) "heat topology tab not present in horizon" [Critical,Fix released] - Assigned to Richard Jones (r1chardj0n3s)08:47
duonghqegonzalez, roger08:48
*** nhlfr has quit IRC08:53
*** nhlfr[m] is now known as nhlfr08:55
*** aolwas has joined #openstack-kolla08:57
*** cuongnv has quit IRC09:03
*** athomas has joined #openstack-kolla09:08
*** manheim has joined #openstack-kolla09:12
*** manheim has quit IRC09:20
*** magicboiz has quit IRC09:21
*** Mannheim has joined #openstack-kolla09:22
*** magicboiz has joined #openstack-kolla09:22
*** cuongnv has joined #openstack-kolla09:26
*** magicboiz has quit IRC09:28
nathharpMorning09:28
*** g3ek has quit IRC09:28
*** haplo37 has quit IRC09:28
nathharpcan anyone comment on the below from the Ocata Nova release notes:09:29
nathharpWhen making connections to Ceph-backed volumes via the Libvirt driver, the auth values (rbd_user, rbd_secret_uuid) are now pulled from the backing cinder.conf rather than nova.conf. The nova.conf values are only used if set and the cinder.conf values are not set, but this fallback support is considered accidental and will be removed in the Nova 16.0.0 Pike release. See the Ceph documentation for configuring Cinder09:29
nathharpRBD auth.09:29
nathharp4.0.0 tarball deployment onto existing Ceph - cinder works, Glance works, Nova gives Ceph uath errors pointing at the UUID09:31
Jeffrey4l_nathharp, this is fixed in yesterday.09:31
Jeffrey4l_try the latest code.09:31
nathharpaha09:31
nathharpJeffrey4l_ thanks for the quick response09:32
Jeffrey4l_np ;)09:32
*** yingjun has quit IRC09:37
*** haplo37 has joined #openstack-kolla09:38
*** g3ek has joined #openstack-kolla09:38
*** hrw has quit IRC09:43
*** hrw has joined #openstack-kolla09:45
egonzalezsayantani01, are you the one working on the deployment docs?09:53
*** duonghq has quit IRC09:58
*** MarginHu has quit IRC10:00
*** MarginHu has joined #openstack-kolla10:00
*** MarginHu has quit IRC10:02
*** cuongnv has quit IRC10:05
*** sayantan_ has quit IRC10:09
*** tovin07 has quit IRC10:11
*** tovin07_ has quit IRC10:12
*** magicboiz has joined #openstack-kolla10:18
*** manheim has joined #openstack-kolla10:18
*** deadnull has joined #openstack-kolla10:19
*** manheim has quit IRC10:19
*** manheim has joined #openstack-kolla10:20
*** manheim has quit IRC10:22
*** pbourke has quit IRC10:23
*** satyar has quit IRC10:23
*** manheim has joined #openstack-kolla10:24
*** pbourke has joined #openstack-kolla10:25
*** sambetts|afk is now known as sambetts10:35
*** zhubingbing_ has quit IRC10:36
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Enable sanity checks from kolla-ansible  https://review.openstack.org/43791910:41
sdakesbezverk rwellum is blocked here: https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP10:46
*** sudipto has quit IRC10:51
*** salv-orl_ has quit IRC11:00
*** salv-orlando has joined #openstack-kolla11:01
*** ccesario__ has quit IRC11:01
*** salv-orlando has quit IRC11:01
*** salv-orlando has joined #openstack-kolla11:02
*** Murali has joined #openstack-kolla11:11
*** athomas has quit IRC11:13
*** furlongm has quit IRC11:13
*** mnasiadka has quit IRC11:13
*** manheim has quit IRC11:14
*** athomas has joined #openstack-kolla11:20
*** JoseMello has joined #openstack-kolla11:21
*** mnasiadka has joined #openstack-kolla11:23
*** manheim has joined #openstack-kolla11:27
*** mnasiadka has quit IRC11:27
*** mnasiadka has joined #openstack-kolla11:28
*** mnasiadk1 has joined #openstack-kolla11:28
*** sean-k-mooney has quit IRC11:32
*** mnasiadk1 has quit IRC11:36
*** mnasiadka has quit IRC11:37
*** rmart04 has joined #openstack-kolla11:39
*** magicboiz has quit IRC11:45
*** mnasiadk1 has joined #openstack-kolla11:46
*** mnasiadka has joined #openstack-kolla11:46
*** athomas has quit IRC11:51
*** rmart04 has quit IRC11:51
*** rmart04 has joined #openstack-kolla11:52
*** rmart04 has quit IRC11:57
*** mnasiadk1 has quit IRC11:59
*** mnasiadka has quit IRC11:59
*** athomas has joined #openstack-kolla12:01
kbaegisMorning all! Anyone up?12:02
kbaegishttps://bugs.launchpad.net/kolla/+bug/167144712:02
openstackLaunchpad bug 1671447 in kolla "iscsi container fails to start, "no space left on device"" [Undecided,New]12:02
*** rwellum_ has joined #openstack-kolla12:04
*** mnasiadka has joined #openstack-kolla12:05
*** mnasiadk1 has joined #openstack-kolla12:05
*** zhurong has quit IRC12:08
des_consoladokbaegis: have you tried rebooting? :p12:08
des_consoladotbh I've been getting this error here and there12:08
des_consoladowith random stuff...12:08
des_consoladoit's quite scary tbh, makes me not want to do kolla in production12:08
des_consoladoI've heard some of the others saying about using other storage drivers12:08
des_consoladomy manager isn't keen on the idea of other storage drivers though due to their supposed immaturity12:09
*** mnasiadka has quit IRC12:10
*** mnasiadk1 is now known as mnasiadka12:10
des_consoladohttps://github.com/openstack/kolla-ansible/blob/master/doc/production-architecture-guide.rst#storage-driver it's suggested here to use 'btrfs or aufs'12:10
des_consoladowell I've been testing with aufs for a couple monthsish now and I've been having these dodgy errors a lot... maybe I should try btrfs12:10
kbaegisThanks!12:13
kbaegisYeah, I had more luck with btrfs on gentoo12:14
*** rwallner has joined #openstack-kolla12:19
*** g3ek has quit IRC12:21
kbaegisdes_consolado: Is there a handy way to migrate? Docker is currently pulling from my root drive (ext4)12:26
kbaegisI absolutely hate that Ubuntu still uses ext12:26
kbaegisby default12:26
*** sudipto has joined #openstack-kolla12:29
*** g3ek has joined #openstack-kolla12:30
*** furlongm has joined #openstack-kolla12:31
*** shardy is now known as shardy_lunch12:32
*** salv-orlando has quit IRC12:38
*** rwallner has quit IRC12:45
nathharpJeffrey4l_ - I grabbed the latest code as suggested12:45
*** shardy_lunch has quit IRC12:45
*** shardy_lunch has joined #openstack-kolla12:45
*** Jezekus has joined #openstack-kolla12:46
nathharpnow attempting to start nova-compute on my controller?  Failing all 20 tries and bombs out12:46
SerlexWhy am I getting " 'module' object has no attribute 'set_defaults'" when I upgrade a container?12:47
JezekusHi all, may I ask about getting an error stating Project ID "someID" is not a valid Project. (HTTP 400). I am getting such error after clean deployment on single node12:47
*** fguillot has joined #openstack-kolla12:49
*** schwicht has joined #openstack-kolla12:53
*** schwicht has quit IRC12:54
*** dave-mccowan has joined #openstack-kolla12:56
*** rwallner has joined #openstack-kolla12:57
egonzalezJezekus, is that error when changing nova quotas?12:58
*** trinaths1 has left #openstack-kolla12:58
Jezekusegonzalez: yes12:59
egonzalezJezekus, is due a recent change in nova (master code not stable)12:59
egonzalezJezekus, keystone endpoint must be set to versionless http://VIP:35357/ without v3 at the end13:00
egonzalezthe fix is in progress, for now update the endpoint13:00
egonzalezopenstack endpoint set <interface> --url <http://VIP:PORT/> <endpoint_id>13:01
*** fooliouno has quit IRC13:03
*** jtriley has quit IRC13:04
egonzalezSerlex, which task fail, which container13:06
*** Jezekus has quit IRC13:07
*** sean-k-mooney has joined #openstack-kolla13:09
rwallnerhey channel, wondering if anything has hit an issue using bond interfaces when deploying. Im getting errors like these  (item=ceph-osd) => {"failed": true, "item": "ceph-osd", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute u'ansible_bond0.###' ### is an actual bond number13:16
rwallnernvm ! this seems to be a config issue13:18
rwallnersorry for tha spam :-|13:18
egonzalezrwallner, is ansible -m setup -i inventory all seeing such bond interface?13:20
rwallneryes, its a bug in our baremetal deployment / globals13:21
rwallnerliterally just using the wrong interface13:21
rwallnerso ansible / kolla is doing the right thing13:21
*** schwicht has joined #openstack-kolla13:25
*** shardy_lunch is now known as shardy13:27
*** salv-orlando has joined #openstack-kolla13:28
*** jtriley has joined #openstack-kolla13:29
*** schwicht has quit IRC13:30
*** schwicht_at_work has joined #openstack-kolla13:33
*** ccesario__ has joined #openstack-kolla13:37
rwellum_Hi - general question: I'm running k8s on a Centos VM, prior to setting up kola. If I suspend and resume my VM, or make a snapshot and later revert. My VM comes back and shoots up to 100% CPU, eventually backing off, but I never recover the VM, can't log back in to see what's going on etc. Anyone seen this before?13:37
*** goldyfruit has joined #openstack-kolla13:45
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot  https://review.openstack.org/44330013:47
*** schwicht_at_work has quit IRC13:51
*** zhubingbing has joined #openstack-kolla13:53
egonzalezactually keystone endpoint does not have description, would be OK add description as one of the register tasks?13:54
egonzalezrather service than endpoint13:54
*** lamt has joined #openstack-kolla13:58
*** rhallisey has joined #openstack-kolla14:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla master: Adding iscsi bits to ironic-conductor  https://review.openstack.org/44364614:04
*** yingjun has joined #openstack-kolla14:05
*** schwicht has joined #openstack-kolla14:07
*** srwilkers has joined #openstack-kolla14:14
*** yingjun has quit IRC14:16
openstackgerritMerged openstack/kolla-ansible master: ironic_pxe bootstrap should run on all hosts  https://review.openstack.org/43718914:25
*** lamt has quit IRC14:27
*** lamt has joined #openstack-kolla14:31
*** satyar has joined #openstack-kolla14:34
*** saneax is now known as saneax-_-|AFK14:35
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Use versionless keystone endpoints  https://review.openstack.org/43902414:35
*** caowei has joined #openstack-kolla14:37
*** Jeffrey4l_ has quit IRC14:39
*** strigazi has quit IRC14:44
*** manheim has quit IRC14:45
*** adrian_otto has joined #openstack-kolla14:45
*** Jeffrey4l_ has joined #openstack-kolla14:51
*** lamt has quit IRC14:53
*** vcn has joined #openstack-kolla14:55
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot  https://review.openstack.org/44330014:55
*** Serlex has quit IRC14:58
*** yingjun has joined #openstack-kolla14:59
*** manheim has joined #openstack-kolla15:00
*** vcn has quit IRC15:00
openstackgerritMerged openstack/kolla master: Add heat-api-cloudwatch image  https://review.openstack.org/44307715:00
*** mnasiadka has quit IRC15:02
*** caowei has quit IRC15:03
*** strigazi has joined #openstack-kolla15:04
egonzalezJeffrey4l_, is this missing in Ocata? https://review.openstack.org/#/c/437914/115:08
*** lucker has joined #openstack-kolla15:08
Jeffrey4l_egonzalez, it is not critical. just one warning message during build. So it is OK.15:09
Jeffrey4l_but will fix it later.15:09
egonzalezroger15:09
*** lamt has joined #openstack-kolla15:09
*** skramaja has quit IRC15:14
*** salv-orlando has quit IRC15:14
*** salv-orlando has joined #openstack-kolla15:15
egonzalezJeffrey4l_, can you remove -2 https://review.openstack.org/#/c/416369/ ?15:16
*** unicell has joined #openstack-kolla15:16
NachoDuckIs there a way to do a reconfigure on nova with running instances? As the restart of nova_libvirt fails15:16
egonzalezNachoDuck, when or how fails?15:17
egonzalezkolla version and docker version?15:17
*** unicell1 has quit IRC15:17
*** zhubingbing has quit IRC15:18
NachoDuckegonzalez, Kolla 3.0.0 I think and Docker 1.12.315:19
kbaegisdes_consolado: Switched to btrfs. Had to reinstall ubuntu :)15:19
egonzalezNachoDuck, can you ensure about kolla version? pip freeze | grep kolla15:20
NachoDuckegonzalez: kolla==3.0.015:20
egonzalezNachoDuck, can you share error trace?15:21
openstackgerritMerged openstack/kolla stable/newton: Fix ironic inspector  https://review.openstack.org/43991315:22
openstackgerritMerged openstack/kolla stable/newton: Fix binary ironic-inspector rootwrap configuration  https://review.openstack.org/43988215:22
NachoDuckegonzalez: http://paste.openstack.org/show/Icmq72noy1TANLWDpNrV/15:23
nathharpJeffrey4l_ think I have found a typo in kolla-ansible/ansible/roles/nova/tasks/external-ceph.yml - line 17 - ‘state’ should be ‘stat’15:31
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot  https://review.openstack.org/44330015:31
egonzaleznathharp, yep15:32
*** adrian_otto has quit IRC15:32
egonzaleznathharp, this have to be backported to ocata, can you open a bug?15:32
sbezverksrwilkers: ping15:33
sbezverksrwilkers: appreciate if you could review this: https://review.openstack.org/44337515:33
*** zhubingbing has joined #openstack-kolla15:34
nathharpegonzalez - will do15:37
egonzaleznathharp, thanks15:37
des_consoladokbaegis: yeah D: I need to try it too! just had to reboot again about 15 minutes ago lmao, not even running anything on this crap, it's just a deployment with no actual compute being used or anything!!15:37
des_consoladojust seems that redeploying and redeploying and redeploying really screws AUFS :\15:38
des_consoladoeven done a lot of docker system prune15:38
kbaegisdes_consolado: AUFS sucks I've decided15:38
kbaegisYeah, even the cleanup-image container doesn't work15:38
des_consoladodoes kubernetes have an opinion about which storage driver the underlying docker daemons ought to be using?15:39
kbaegisdes_consolado: Excellent question.15:40
kbaegisdes_consolado: raw RBD looks to be an option: https://kubernetes.io/docs/getting-started-guides/ubuntu/storage/15:41
egonzalezNachoDuck, are you using qemu or kvm?15:41
kbaegisdes_consolado: Looks to be the equivalent of drive-mapping in docker on top of ceph RBD15:41
kbaegisPretty smart honestly15:42
*** mdnadeem has quit IRC15:42
NachoDuckegonzalez: kvm I believe15:42
*** jbadiapa has quit IRC15:42
*** sayantan_ has joined #openstack-kolla15:42
des_consoladokbaegis: where did you find that? links plz :D15:42
kbaegisdes_consolado: ^^ ->>https://kubernetes.io/docs/getting-started-guides/ubuntu/storage/ and some synthesis of information in my head ;)15:43
kbaegisI might be misunderstanding15:43
kbaegisThere's this too http://blog.kubernetes.io/2016/10/dynamic-provisioning-and-storage-in-kubernetes.html15:44
kbaegisBasically, rbd is a distributed block device. Think millions of 512k objects. Advantage is that if you move a pod, you don't have to xfer block across a storage network15:46
sdakerwellum_ YMMV with virtualm achines15:47
sdakezhubingbing ping15:48
sdakezhubingbing yous aid you went through the etherpad instructions howeveryo u didn't add a  tested and date for each item?15:48
*** zhubingbing has quit IRC15:48
*** zhubingbing has joined #openstack-kolla15:48
rwellum_sdake, I spoke to others at the PTG who were doing it fine, and in fact that was a factor in explaining their environment, they got it working once, snapshot their VM's and then always just cloned etc when needed. So perhaps I am just being unlucky.15:49
egonzalezNachoDuck, I don't know why is complying  with "Is a directory: u'/var/lib/kolla/config_files/secrets'"15:53
*** jtriley_ has joined #openstack-kolla15:55
des_consoladokbaegis: https://coreos.com/os/docs/latest/mounting-storage.html#creating-and-mounting-a-btrfs-volume-file "Container Linux 561.0.0 and later are installed with ext4 + overlayfs to provide a layered filesystem for the root partition. Installations from prior to this, use btrfs for this functionality."15:56
egonzaleznathharp, are you going to push a change for the bug?15:57
des_consoladonot kubernetes strictly speaking but coreos container linux os defaults changed from btrfs to ext4+overlayfs interestingly15:57
*** jtriley has quit IRC15:58
*** kbaegis has quit IRC15:58
NachoDuckegonzalez: I see, it appears the containers restarted on one of the other nodes, so might not be a Kolla problem. The container stopped but it won't start now as it can't unmount the container volume15:59
*** rabel has quit IRC16:00
*** jtriley_ has quit IRC16:01
*** manheim has quit IRC16:01
kfox1111morning.16:02
*** kbaegis has joined #openstack-kolla16:02
kbaegisdes_consolado: Huh. Interesting16:04
*** manheim has joined #openstack-kolla16:05
openstackgerritMerged openstack/kolla master: Allow full customisation of kube image binaries  https://review.openstack.org/43689716:07
openstackgerritThomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service.  https://review.openstack.org/44373216:07
des_consolado^^^ first thing I've pushed for review... it works in so far as the image works and then running with a kolla-ansible deployment with the changes I've made to kolla-ansible... well yes the services all run and there are no errors :D:D16:08
des_consoladoI'm sure I'm missing things though, I probably need to write tests etc?16:09
des_consoladoI was thinking of even pushing this as WIP because I doubt you will want to merge that right off the bat. I'm not sure. Feedback appreciated.16:09
*** jtriley has joined #openstack-kolla16:09
sbezverkkfox1111: good morning.. could you please review this ps?  https://review.openstack.org/44337516:09
kfox1111sure. sec16:10
des_consoladoactually I don't think there's anything wrong with it and it's just an image I guess...16:13
openstackgerritMerged openstack/kolla-ansible master: Set empty openstack_release option  https://review.openstack.org/44329116:15
openstackgerritMerged openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/44335316:18
*** zhubingbing has quit IRC16:18
egonzalezdes_consolado, added a couple of comments. Just nits and code style16:19
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot  https://review.openstack.org/44330016:24
openstackgerritThomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service.  https://review.openstack.org/44373216:25
kbaegisdes_consolado: btw- Tap as a service?16:25
kbaegisDoes kolla do that?16:25
des_consoladoegonzalez: I made the correcting changes and just did a commit --amend and then a git review again, can see it's now https://review.openstack.org/443732 - hope this is the correct workflow16:26
*** rabel has joined #openstack-kolla16:26
des_consoladokbaegis: no idea what that even is sorry :D16:26
des_consoladooh, pcapping...16:26
kbaegisdes_consolado: Oh, the thing I meant to direct to you: just had btrfs crash on ubuntu 16.04 on the root drive during a kolla-build16:27
des_consoladooh dear, lol D:16:27
egonzalezdes_consolado, yep thats the workflow, use --amend to push the the same change id and not replicate commits16:27
des_consoladoah feels so good to be losing my openstack contribution virginity finally16:28
des_consoladoegonzalez: oh shit, the author is wrong D:16:30
des_consoladoI just tried to change it and do a commit --amend also but it rejected it saying there were no changes :(16:31
*** adrian_otto has joined #openstack-kolla16:31
egonzalezdes_consolado, if the commit was one release before you had a free summit ticket, but this does not happen anymore :(16:31
egonzalezdes_consolado, hrm, dunno if author can be changed :/16:32
openstackgerritThomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service.  https://review.openstack.org/44373216:32
des_consoladoah it is fixed16:33
des_consoladohad to do git commit --amend --author "blah blah blah"16:34
egonzalezsomething new :)16:34
kfox1111sbezverk: reviewed.16:36
kfox1111it looks good, if thats the right path to go down.16:36
sbezverkkfox1111: just replied.. this change was mostly to address growing list in nova service-list with nova-compute-ironic down state16:38
sbezverksince it registers itself with its host name everytime.16:38
kfox1111oh.. cause the daemonset is not net=host...16:38
sbezverkyep16:38
kfox1111still. not sure whats the best aproach. net=host or statefulset.16:39
sbezverkkfox1111: ironic people told me that in ocata, they do not ask nova/neutron to bind a port16:39
sbezverkso the issue is resolved by more smart networking..16:39
kbaegisooc16:40
kfox1111yeah, but we still need a matching ovsagent anyway?16:40
kbaegisHas anyone tried anything crazy like putting mariadb on a ramdisk?16:40
sbezverkkfox1111: nope they do not need it at that poijt16:41
kfox1111oh. so the ironic ml2 driver made it in for ocata?16:41
sbezverkkfox1111: they done it a bit differenely, they let ironic port to be in not bound state16:42
kfox1111ah.16:42
sbezverkkfox1111: since ironic completely controls it16:42
sbezverkit does not matter16:42
kfox1111ok. then +1 to statefulset.16:42
sbezverkkfox1111: :)16:42
kfox1111:)16:43
kfox1111when is 4.0.x containers gettign released (if not already?)16:43
*** sudipto has quit IRC16:44
sbezverkkfox1111: do not know.. I have been trying to clean up ironic discovered issues and address them..16:44
kfox1111yeah.16:44
kfox1111if 4.x solves the issue cleanly by not needint the ovs agent, I think we should probably just push forward and require 4.x for that?16:44
sbezverkkfox1111: for sure.. even ironic people recommends using ocata at minimum..16:45
kfox1111k.16:45
kfox1111then I think the only issue is that one ws char, then +2. :)16:46
openstackgerritThomas O'Neill proposed openstack/kolla-ansible master: Add neutron-bgp-dragent playbooks and sensible defaults.  https://review.openstack.org/44374716:46
kfox1111and I think we need to start a document that says what is the minimum supported version of things.16:46
kfox1111like, lvm doesn't work until 3.x. ironic doesn't work until 4.x, etc.16:46
sbezverkkfox1111: damn ws, I need an editor which tracks them for me ;)16:47
des_consoladohow do we know what's going on with these gates?16:47
kfox1111des_consolado: which gates?16:47
des_consoladokfox1111: I'm not sure, I'm Thomas O'Neill, I just submitted those 2 changes for kolla and kolla-ansible for the neutron-bgp-dragent stuff16:47
egonzalezdes_consolado, once gates finish you can see results in gerrit16:47
des_consoladoI think they have to go through some gates now or something?16:47
egonzalezdes_consolado, if want to see the status check here http://status.openstack.org/zuul/ add gerrit id in the search field16:48
kfox1111des_consolado: ah. ok. I'm not very familior with those gates.16:48
des_consoladoegonzalez: cool, how long's it usually take?16:48
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337516:49
egonzalezdepends if zuul is busy or not, often between 30~40m16:49
kfox1111there's one more ws one more line down it looks like. :/16:49
des_consoladooh there was 1 failure... better look at that..16:49
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337516:51
egonzalezdes_consolado, seems unrelated to your change. keystone bootstrap failed16:51
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Helm 2.2 and localpath  https://review.openstack.org/43694616:55
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Fix typo in nova external ceph  https://review.openstack.org/44375316:56
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Helm 2.2 and localpath  https://review.openstack.org/43694616:57
*** rabel has quit IRC16:58
*** lucasxu has joined #openstack-kolla16:58
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Helm 2.2 and localpath  https://review.openstack.org/43694616:58
*** rhallisey has quit IRC17:02
*** gfhellma has joined #openstack-kolla17:03
sbezverkkfox1111: done, and here is another one for missing pxelinux.0 file https://review.openstack.org/44330017:03
*** egonzalez has quit IRC17:03
sbezverk kfox1111: this one is important as it requires to copy this file everytime :(17:03
nathharpJeffrey4l_ has your external_ceph fix hard coded ceph client names?17:08
*** lucasxu has quit IRC17:08
*** lucker has quit IRC17:09
*** hieulq has quit IRC17:12
*** rhallisey has joined #openstack-kolla17:15
*** hieulq has joined #openstack-kolla17:15
*** yingjun has quit IRC17:16
*** matrohon has quit IRC17:17
*** lucasxu has joined #openstack-kolla17:17
*** lucasxu has quit IRC17:17
*** l4yerffeJ_ has quit IRC17:17
*** l4yerffeJ_ has joined #openstack-kolla17:18
*** shardy has quit IRC17:21
des_consoladowhat's the difference between verifying and reviewing17:24
*** manheim has quit IRC17:25
inc0good morning17:25
des_consoladohi17:25
jascott1verifying is installing, running tests etc. Reviewing is reviewing code.17:26
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot  https://review.openstack.org/44330017:26
openstackgerritMerged openstack/kolla master: Updated from global requirements  https://review.openstack.org/44335217:26
*** unicell has quit IRC17:29
*** athomas has quit IRC17:30
*** jtriley has quit IRC17:30
openstackgerritMerged openstack/kolla-ansible master: Enable sanity checks from kolla-ansible  https://review.openstack.org/43791917:31
des_consoladoI feel like I should probably write some kind of tests for the neutron-bgp-dragent kolla-ansible stuff...17:32
openstackgerritMerged openstack/kolla-ansible master: Enable run nova-compute-ironic with nova-compute on the same node  https://review.openstack.org/44313817:33
openstackgerritMerged openstack/kolla-ansible master: Fix the incompatible of docker-py 2.0.0  https://review.openstack.org/43881317:34
*** nathharp has quit IRC17:39
des_consoladoah thanks inc0, I didn't know about raising unavailability on repo binaries, yeah neutron-dynamic-routing isn't available as an rpm17:40
*** jrich523 has quit IRC17:40
*** rhallisey has quit IRC17:40
sbezverkinc0: are ocata images available?17:41
*** jrich523 has joined #openstack-kolla17:41
*** ssurana has joined #openstack-kolla17:41
des_consoladowtf??? http://paste.openstack.org/show/602102/17:42
des_consoladomy repo just murdered itself ?! :'(17:42
inc0des_consolado: np, we get this from time to time17:43
inc0otherwise change looks really good:)17:43
inc0sbezverk: not yet17:43
des_consoladooh I just had to do cd ..17:43
*** sayantan_ has quit IRC17:43
openstackgerritThomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service.  https://review.openstack.org/44373217:44
*** sayantan_ has joined #openstack-kolla17:44
inc0gerrit takes some time to get used to17:44
inc0I ususally reset --hard origin/master and then checkout change from review.openstack.org17:44
inc0you have small "download" button on top right corner17:45
openstackgerritMerged openstack/kolla-ansible master: Remove cell setup and db creation nova upgrade  https://review.openstack.org/44329017:45
des_consoladooh ic, ty17:46
inc0this way you don't need to track your local fs17:46
inc0handy when you do more than one change at the time17:46
*** mattmceuen has joined #openstack-kolla17:48
inc0des_consolado: question, does dragent has anything to do with calico?17:49
des_consoladoinc0: I don't think so, not that I know of. It's just the BGP speaker. It uses the Ryu driver.17:50
des_consoladoIsn't calico an overlay networking driver or something...17:50
*** vhosakot has joined #openstack-kolla17:50
*** fooliouno has joined #openstack-kolla17:50
inc0well Calico uses bgp for vm networking17:51
des_consoladoyeah am just reading this http://docs.projectcalico.org/v2.0/getting-started/openstack/ - looks like it does alotalotalot with the networking... heh17:51
inc0so I guess this works differently17:51
inc0yeah it pretty much defines whole networking for you17:52
*** eaguilar has quit IRC17:52
inc0which is cool, I asked because both uses bgp and I was curious if these are connected in any way17:52
*** eaguilar_ has joined #openstack-kolla17:52
des_consoladolooks like Calico uses BIRD17:52
*** gfidente has quit IRC17:53
inc0I think it can also use quagga17:54
inc0not sure tho, I'm not networking guy17:55
*** rhallisey has joined #openstack-kolla17:55
inc0nope, I was wrong, no quagga17:57
des_consoladoinc0: how do you usually run this reno command? I see that it's in .tox/genconfig/bin/reno17:57
des_consoladoI guess can you also just run python -m reno ?17:57
des_consoladoah it's documented here I guess https://docs.openstack.org/developer/reno/17:58
inc0no, do pip install -r test_requitements.txt17:58
*** lucasxu has joined #openstack-kolla17:58
inc0and then just reno new dragent_support (or sth like that)17:58
vhosakotv1k0d3n: portdirect: saw these errors twice today. docker and kubelet failed to start.  have you seen this error, is it something intermittent? - http://paste.openstack.org/show/602105/17:58
*** eaguilar has joined #openstack-kolla17:59
vhosakotdigging logs.17:59
*** eaguilar_ has quit IRC18:00
des_consoladoinc0: also re: the naming, I think that's an issue with the plugin itself - https://github.com/openstack/neutron-dynamic-routing/blob/master/setup.cfg#L35 - see how the package is called neutron-dynamic-routing, but the agent that runs is called neutron-bgp-dragent18:00
des_consoladoI gather these guys in the future plan to implement other dynamic routing protocols/agents, like maybe for ospf or something18:01
inc0yeah...it's..unfortunate naming18:01
inc0well, that being said, one driver to rule all the routing is bad design too18:01
inc0especially if it's only driven by name...18:02
*** pbourke has quit IRC18:03
*** adrian_otto has quit IRC18:05
*** Murali has quit IRC18:06
openstackgerritThomas O'Neill proposed openstack/kolla master: Add neutron-bgp-dragent service.  https://review.openstack.org/44373218:07
openstackgerritThomas O'Neill proposed openstack/kolla-ansible master: Add neutron-bgp-dragent playbooks and sensible defaults.  https://review.openstack.org/44374718:08
des_consoladoI gotta go home anyway back tomorrow thanks everyone18:11
inc0cya des_consolado, thanks18:11
vhosakotv1k0d3n: portdirect: I see each vagrant VM using more than 200% CPU on my host.  My host has 16 CPUs, will dig more - http://paste.openstack.org/show/602108/18:15
*** unicell has joined #openstack-kolla18:15
vhosakotanybody see each vagrant VM use more than 200% CPU on the host when deploying kolla-k8s per https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html?18:15
sbezverkvhosakot: I think rwellum_ reported similar issue18:16
vhosakotsbezverk: hmm, thanks, I think somebody else also saw it, I dont remember who it was here in IRC when I asked the question about 200% CPU usage.18:17
vhosakotrwellum_: how did you resolve the issue? what is the RAM and CPU of your vagrant host?  I think we need to add the steps to resolve this issue in the docs in https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html as it looks like a day-1 blocker for a new developer :)18:18
vhosakotI asked the same question about 200% CPU usage 2 weeks ago, digging IRC logs to see who replied and if there was a solution mentioned.18:19
vhosakotsorry I missed yesterday's kolla meeting as I had an appt at the same time.18:20
*** zioproto has quit IRC18:21
vhosakotsbezverk: how much RAM and CPU do you have on your vagrant host?  also, are you using libvirt vagrant driver to standup your devenv for kolla-k8s per https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html ?18:24
vhosakotI have 16 CPUs and 32G RAM on my vagrant host with libvirt driver, thought it was enough to boot the four vagrant VMs for kolla-k8s. Any info about host resources to standup vagrant dev env for kolla-k8s would be very helpful :)18:30
qwangkfox1111: ping18:31
vhosakotI'm asking in the #vagrant IRC channel :)18:32
openstackgerritKevin Fox proposed openstack/kolla-kubernetes master: Helm 2.2 and localpath  https://review.openstack.org/43694618:39
kfox1111qwang: hi18:40
openstackgerritMerged openstack/kolla master: Add ironic-ui in horizon binary install  https://review.openstack.org/44345318:40
rwellum_vhosakot, have not solved it unfortunately. And I've been blocked on various day 1 issues with kolla for over a month...18:42
sbezverkkfox1111: please when you have a chance two PS are ready18:42
kfox1111the statefulset and pxe image?18:42
qwangkfox1111: I want to use the new conditions/tag feature from helm in the service charts. One idea is to add all dependencies including those common ones (mariadb/rabbitmq/etc) in each of the services and use tag to enable/disable for installation18:42
kfox1111both are +2'ed.18:42
rwellum_vhosakot, 32G Ram and 6 CPU's - so less cpu's than you.18:42
kfox1111qwang: +1. I was thinking the same thing.18:43
qwangkfox1111: another idea, as jascott1 pointed out, use condition/tags for some optional parts, like some ovs components for neutron18:43
kfox1111+1 to that too. :)18:43
rwellum_vhosakot, which SDN are you using? Weave etc?18:44
vhosakotrwellum_: hmm, thanks for the reply, are you deploying kolla-kubernetes per the docs in https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html and blocked due to vagrant vm using 200% CPU?18:44
qwangkfox1111: I wonder what's your original desire for it. Any other use case you'd like to have?18:44
vhosakotrwellum_: yes, weave, which comes OTB, do you think that can cause the issue?18:44
kfox1111qwang: you hit the two I was thinking of.18:45
kfox1111I can't off hand think of a 3rd use case.18:45
rwellum_vhosakot, I've tried both that guide and also the bare metal route - which I am helping to write. Have not come to a conclusion with either.18:45
vhosakotrwellum_: well, bare metal deploy does now have vagrant right?  so, the bare metal issue you see are non-vagrant related?18:46
rwellum_vhosakot, weave seems to me to be the most unstable, Try:18:46
rwellum_vhosakot,18:46
rwellum_vhosakot, sed -i 's/weave/canal/g' halcyon-kubernetes/kube-deploy/group_vars/all.yml18:46
qwangkfox1111: thank you very much. You help me sort it out. I'll start to work on them18:47
kfox1111qwang: cool. thanks. :)18:47
rwellum_No vagrant with bare metal vhosakot my issues there are unrelated - mainly networking. Documented here: https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP18:47
*** rwallner has quit IRC18:47
kfox1111qwang: I'm not sure just how usable it will be without configmap stuff being done though. we need a way to point at the service level db's/things.18:47
kfox1111though that work can be done in parallel.18:47
rwellum_vhosakot, but eventually canal died too, so I don't think it's related to the vagrant issue - just something to try18:48
*** papacz has quit IRC18:48
rwellum_vhosakot, note that sdake mentioned using within a VM is not a consistent path - although I see plenty doing it successfully.18:49
vhosakotrwellum_: cool, thanks for the info.. I'll try canal and will keep you posted.  I might tray flannel as well, good luck with bare metal deploy, I'd like to help with it once I'm able to standup my devenv18:49
vhosakotI know sdake and I both are _not_ vagrant fans... standing up within VMs :)18:50
rwellum_vhosakot, should also mention I wasn't seeing issues with Vagrant until recently - last couple days it seems like.18:50
vhosakotrwellum_: sdake: once I'm able to standup my devenv, I'll try to contribute to the bare metal deploy you guys are working on.18:51
rwellum_vhosakot, I've just followed the dev guide on a baremetal server, so no VM - and so far no performance issues.18:51
vhosakotrwellum_: the high CPU usage could be vagrant issue too. which version are you using? I'm using 1.8.1 per the docs in https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html.18:52
*** sambetts is now known as sambetts|afk18:52
vhosakotrwellum_: cool, good you have eliminated VM/vagrant issues in bare-metal deploy.. nice.18:52
robcresswellHey, question, why does kolla's horizon binary bake in all the plugins?18:52
rwellum_vhosakot, yeah completely following the guide like you. Also the guide doesn't mention installing docker which you'll have to do.18:53
rwellum_vhosakot, this: http://paste.openstack.org/show/602110/18:53
*** Jezekus has joined #openstack-kolla18:54
vhosakotrwellum_: does the task "install kubeadm and required prereqs" in https://github.com/att-comdev/halcyon-kubernetes/blob/master/kube-deploy/roles/deploy-kube/tasks/centos.yml#L35 install docker?18:55
*** adrian_otto has joined #openstack-kolla18:55
vhosakotrwellum_: why do we need to do those steps manually?  I see halcyon-kubernetes already does it through these tasks in Ansible's stdout - http://paste.openstack.org/show/602111/18:56
vhosakotrwellum_: it was all working great around 3 weeks ago, and I was able to standup the dev env with weave.. don't know the issue today.18:57
rwellum_vhosakot, for me docker does not get installed, i asked the dock author19:00
vhosakotrwellum_: interesting, for me, docker is properly installed on all the vagrant vms through the tasks in http://paste.openstack.org/show/602111/.  do you see those tasks running for you?19:01
rwellum_vhosakot, when I try with a vm again I'll let you know. But i def. did not have docker daemon running19:02
vhosakotrwellum_: cool, thanks. I'll use canal/flannel now... thanks for your help!19:02
vhosakotsbezverk: kfox1111: do you use canal to standup kolla-kubernetes dev env?19:02
*** Jezekus has quit IRC19:02
*** jtriley has joined #openstack-kolla19:03
*** eaguilar has quit IRC19:03
rwellum_vhosakot, last time I worked with sbezverk he was using canal19:04
rwellum_vhosakot, in fact he told me too as it was more stable19:04
*** eaguilar has joined #openstack-kolla19:05
vhosakotrwellum_: cool cool, thanks... BTW, I see this error in the libvirt logs of each vagrant VM that uses 200% CPU on the host - http://paste.openstack.org/show/602112/19:05
*** rhallisey has quit IRC19:07
rwellum_vhosakot, plenty on google on that one...19:08
sbezverkvhosakot: yep canal seems to work for me well19:08
vhosakotrwellum_: yup, googling now :)19:08
vhosakotsbezverk: thanks, how much RAM and CPU do you have on your vagrant host?19:08
rwellum_sbezverk, running kubectl on bare-metal is giving me permission errors. Do you run normally as sudo?19:11
sbezverkvhosakot: no vagrant, bare metal vm19:12
sbezverkrwellum_: should not be19:13
vhosakotsbezverk:  cool.. \m/  \m/  for bare-metal :)19:13
sbezverkkubectl creates .kube folder for the config19:13
*** david-lyle has quit IRC19:14
vhosakotsbezverk: how many bare-metal VMs do you have?  I think I can create few too, are there docs to deploy on bare-metal VMs?19:15
vhosakotrwellum_ told me he is adding the bare-metal docs.. is any doc available for me to start bare-metal deploy?19:15
sbezverkvhosakot: I use two VMs one where I run kube cluster and one for development19:16
vhosakotsbezverk: cool, like one master, one worker node?  did you dpeloy using the Ansible playbooks in https://github.com/att-comdev/halcyon-kubernetes?19:18
inc0robcresswell: well, mostly because we don't know what plutins you might want19:19
sbezverkvhosakot: no just master, second vm is just linux ws with development tools19:19
inc0on the bright note, if that's a problem it's super easy to make it install just what you want19:19
vhosakotsbezverk: so, you deploy kolla-kubernetes in a single-vm then?19:20
*** rhallisey has joined #openstack-kolla19:20
sbezverkvhosakot: I deployed everything manually it took longer but seeing how much trouble you guys hit, I am starting to this it was not that bad ;)19:20
vhosakotsbezverk: yes, I am not a fan of vagrant :)  and debugging vagrant issues instead of kolla is not something I enjoy ;)19:21
vhosakotsbezverk: have you added docs for the way you deployed so the community can use it?19:21
rwellum_vhosakot, that's the etherpad I pointed you too earlier19:21
vhosakotrwellum_: cool19:21
rwellum_vhosakot, https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP19:21
rwellum_vhosakot, but... I still think it's very complicated and there are lots of unknowns.19:22
*** salv-orl_ has joined #openstack-kolla19:22
*** satyar has quit IRC19:23
rwellum_So inc0 - I'm not whining :) -but in the last few days there have been at least 3 maybe 4 people on this IRC including me who are struggling to get going with Kolla. I know part of this is the fact the k8s is much newer than ansible. But I feel like maybe some time needs to be spent on this to get this part stable?19:23
vhosakotrwellum_: right.. I understand the docs are still WIP now.. cool19:23
robcresswellinc0: Right, but wouldn't they be installed as other binaries? I'm not very familiar with Kolla, just curious19:24
*** sayantan_ has quit IRC19:24
vhosakotinc0: I agree too and +1 rwellum_ comment above ^^^ :)19:24
inc0rwellum_: couldn't agree more, issue is we had release yesterday (of kolla and kolla-ansible) so we really had fires to put out in our stable deliverables19:24
*** sayantan_ has joined #openstack-kolla19:24
inc0now it's a bit quieter and we can focus more on k8s19:24
inc0you just caught us in release-fever time19:25
*** salv-orlando has quit IRC19:25
vhosakotsbezverk: rwellum_: that etherpad does now have the steps to deploy, does it... since sbezverk has got it working in a single-vm, he could help me thru webex may be if he is available sometime.19:25
inc0robcresswell: what do you mean by "other binaries"?19:25
rwellum_inc0, ack19:26
rwellum_vhosakot, what is your question?19:26
inc0issue is, kolla clearly separates build phase from deployment phase19:26
inc0which means decisions you make on deployment can't affect build19:26
*** MasterOfBugs has joined #openstack-kolla19:26
vhosakotrwellum_:  the steps to deploy kolla-kubernetes on a bare-metal VM without using vagrant :)19:26
rwellum_vhosakot, I've webexed with sbezverk a couple of times and it's incredibly helpful but it's not going to scale is it?19:26
kfox1111sbezverk: portdirect: srwilkers: can we get this through quickly? https://review.openstack.org/#/c/436946/19:26
inc0and since you can add/remove projects really easily, and that's deployment phase19:26
kfox1111pretty trivial, but touches a lot of things.19:27
kfox1111inc0: you too. :)19:27
inc0when you for example want to deploy new project to existing openstack19:27
rwellum_vhosakot, that doc is WYSIWYG19:27
inc0you don't want to replace horizon container too19:27
vhosakotrwellum_: yes, 1:1 webex with sbezverk is not scalable :)  alteast, I can help add the docs with you and sbezverk19:27
robcresswellinc0: I was just curious, with patches lik https://review.openstack.org/#/c/443453, why all the ui plugins are included in an install; if they are installed and have issues, they could well break the entire Horizon install; this might just be because I dont understand Kolla, tho19:27
inc0robcresswell: well, if they're installed but not enabled (enable would happen on deploy phase) that should be fine19:28
inc0you just have few mbs of dormant code on filesystem otherwise19:28
vhosakotsbezverk: I'll read https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP and start deploying on my bare-metal VM, might ask your webex help, I'll pay back by adding nice docs for the community :)19:28
rwellum_vhosakot, I've tried on bare-metal, on a VM, with both 'deployment' and 'development' - where deployment is really for me just running through the steps manually.19:28
inc0if there are issues - we'll deal with issues19:28
inc0yay local path in action19:29
robcresswellinc0: Cool, okay19:30
inc0robcresswell: https://docs.openstack.org/developer/kolla/image-building.html#plugin-functionality19:30
vhosakotrwellum_: I might also deploy the playbooks in https://github.com/att-comdev/halcyon-kubernetes on my bare-metal VMs, I'll keep you posted.19:31
robcresswellinc0: Thanks19:31
inc0or rather https://docs.openstack.org/developer/kolla/image-building.html#dockerfile-customisation19:31
inc0this is guide which will show you how to modify stuff we install in containers19:31
*** rwallner has joined #openstack-kolla19:31
kfox1111inc0: thanks for the review.19:32
inc0yeah19:32
*** rwellum_ has quit IRC19:32
sbezverkinc0: if you have time I need two PS to review, kfox1111 has already reviewed them..19:32
inc0I'll do it in a few Serguey, I really want to get back to k8s installation19:32
kfox1111sbezverk: I think if mine goes in before yours, it should just work?19:33
sbezverkhttps://review.openstack.org/443300 and  https://review.openstack.org/44337519:33
kfox1111if your goes in first, I think a rebase will be needed.19:33
*** rwellum has joined #openstack-kolla19:33
inc0we need this frackin' guide;) to rwellum's point19:33
sbezverkkfox1111: np I can go after yours19:33
inc0and I really want ansible playbook to deploy k8s that just works:(19:33
rwelluminc0, +119:34
rwelluminc0, vagrant adds another layer19:34
inc0ok, I'll ask again here maybe19:34
inc0I don't want vagrant19:34
inc0I have openstack installation running19:34
inc0I'll spawn 5 vms there19:34
inc0clean xenial19:34
inc0I have ceph running on the side19:34
inc0I easy way to install working k8s connected to this ceph19:35
inc0ideas?19:35
inc0kargo doesn't work for me, it seems half-baked19:35
inc0I need to test more halcyon plays19:35
rwellumvhosakot, ack to trying on bare metal19:36
vhosakotrwellum: cool19:36
kfox1111inc0: if you don't want it to be production grade, kubeadm is really nice.19:36
inc0kfox1111: how to orchiestrate kubeadm?19:36
kfox1111inc0: someone in meat space. :)19:37
inc0yay.19:37
rwellumkfox1111, inc0 the current etherpad design uses kubeadm - you probably know that.19:37
kfox1111its not a hard/timeconsuming thing.19:37
*** salv-orl_ has quit IRC19:37
sbezverkkfox1111: if you specify some parameters for kubeadm, I do not know why you cannot call it even production greade19:37
inc0kfox1111: you have any docs/quickstart handy?19:37
sbezverkinc0: I was using kuberenetes kubeadm lage19:38
sbezverkpage19:38
*** gfhellma has quit IRC19:38
sbezverkand it worked for me with slight modifications19:38
*** salv-orlando has joined #openstack-kolla19:38
inc0right19:38
kfox1111inc0: https://kubernetes.io/docs/getting-started-guides/kubeadm/19:38
inc0I'll probably write quick playbook for it19:38
inc0yet another19:38
inc0you deploy with calico?19:39
inc0or which networking overlay?19:39
kfox1111cannal19:39
kfox1111which is calico+flannel19:39
sbezverk+1 for canal19:39
vhosakotrwellum: sbezverk, so how (ehterpad vs other ways) we deploy kolla-kubernetes does not matter, right? does it?  I might also try the k8s official docs to deploy multi-node on centos - https://kubernetes.io/docs/getting-started-guides/centos/centos_manual_config/19:39
kfox1111I'd like to get canal's plugin added to helm.19:39
kfox1111I have helm running in a static pod on one of my systems now. works great. :)19:40
sbezverkvhosakot: use kubeadm installation19:40
kfox1111so should be able to use helm to bootstrap most of k8s.19:40
sbezverkthe link kfox1111 pasted earlier19:40
vhosakotsbezverk: ok, thanks. will do.19:40
inc0ok, I'll get to it after lunch19:41
sbezverkvhosakot: there is just one thing with kubeadm it uses bunch of default ranges19:41
srwilkerso/19:41
sbezverkI suggest to check those ranges before you run it19:41
inc0hey srwilkers, hows life?19:41
srwilkersinc0: always exciting.  how about you?19:41
inc0I'm hungry:(19:42
vhosakotsbezverk: cool, which range? the cidr range?19:42
sbezverkhere is command line I used:19:42
sbezverkkubeadm init --pod-network-cidr 10.57.0.0/19  --service-cidr 10.57.32.0/19 --service-dns-domain openstack.kolla.19:42
vhosakotsbezverk:  I'll try https://kubernetes.io/docs/getting-started-guides/kubeadm/.19:42
vhosakotcool19:42
inc0ehh...can you add whole subnet to neutron address-pairs?19:43
vhosakotsbezverk: one more question, did you deploy manually or run the Ansibke playbooks in https://github.com/att-comdev/halcyon-kubernetes on your cluster VM?19:43
sbezverkvhosakot: everything was done manaully19:43
vhosakotsbezverk: perfect, thank your all! :)19:44
sbezverkinc0: was it a question to me?19:44
inc0no, it was question to anyone who knows neutron;)19:44
vhosakotsbezverk: I'll get a working kubeadm deploy on my CentOS VM and ping you guys later to help with contribution and testing, thanks guys! :)19:44
openstackgerritMerged openstack/kolla-kubernetes master: Helm 2.2 and localpath  https://review.openstack.org/43694619:45
*** rabel has joined #openstack-kolla19:46
kfox1111inc0: I don't understand the question.19:46
kfox1111which neutron address-pairs are you talking about?19:46
inc0kfox1111: let say I have vm with 2 ports in subnets 192.168.1.0/24 and 2.0/2419:46
inc0kfox1111: of nodes on top of which I'll install k8s19:46
kfox1111k19:47
vhosakotinc0: don't think so.  you can however add multiple addresses like "--allowed-address-pairs type=dict list=true ip_address=1.1.1.2,ip_address=1.1.1.3,ip_address=1.1.1.4"19:47
inc0because I assume that if I'll add pod network cidr 10.57.0.0/24, neutron will block that19:47
*** sayanta__ has joined #openstack-kolla19:47
inc0oh joy..19:47
kfox1111inc0: if you use canal, it tunnels the traffic through vxlans.19:47
kfox1111so you only need the one subnet.19:47
sbezverkkfox1111: with local-path ps, any changes required to templates?19:48
rwellumvhosakot, so far running the dev environment on bare metal has been MUCH healthier than on a VM...19:48
vhosakotyes, neutron's knowledge of what subnet is used comes thru the CIDR when the subnet is created... allowed address pair is per-address.19:48
inc0ok, I can use regular ports and subnets on it and canal will create vxlans? that'll work19:48
kfox1111sbezverk: no. it just adds the requriements.yaml.19:48
inc0(note...change mtu)19:48
rwellumvhosakot, have you enabled host-passthrough?19:48
vhosakotinc0: you can add multiple subnets (and cidr) for the same network though.19:48
inc0vhosakot: yeah but it will probably block all besides port-associated ip anyway19:48
inc0kfox1111: in my case, kubeadm+canal will work then right?19:49
vhosakotinc0: yes, that's is default security/anti-spoofing thing that neutron does OTB.19:49
kfox1111inc0: I'm saying, I think you only need one neutron subnet.19:49
inc0cool19:49
kfox1111and needs a security group to allow the vxlan traffic.19:49
inc0I have pretty lax security groups;)19:49
inc0devenv19:49
kfox1111k. :)19:50
inc0accept all;)19:50
*** sayantan_ has quit IRC19:50
inc0ok, maybe that was part of problem19:50
inc0I tried to use calico19:50
inc0so probably nto tunneled19:50
kfox1111yeah. calico does routing, which is blocked by neutron by dfeault.19:50
vhosakotrwellum: are you talking about cpu_mode=host-passthrough in nova.conf?19:50
kfox1111you need the port security extention and disable it on the ports otherwise.19:51
vhosakotkfox1111: I've been playing with calico which comes OTB in kubernetes kargo (https://github.com/kubernetes-incubator/kargo).. yes, neutron drops all packets involved in the BGP routing... options would be allow BDP ranges thru allowed-address-pairs, or diable port-security as inc0 mentions.19:52
*** adrian_otto1 has joined #openstack-kolla19:52
*** david-lyle has joined #openstack-kolla19:53
vhosakotinc0: operators might not like to disable port-security as it is risky.19:53
openstackgerritMerged openstack/kolla-kubernetes master: Copy pxelinux.0 file to tftpboot  https://review.openstack.org/44330019:53
rwellumvhosakot, no at a more basic level, in your VM definition you need host-passthrough otherwise the nested virtualization won't work.19:53
kfox1111vhosakot: or option 3, use canal. :)19:53
inc0vhosakot: operators can figure it out;)19:54
inc0canal seems like fun19:54
inc0trying now19:54
rwellumvhosakot, I've had many issues with this no realizing that I reverted my VM and lost that setting19:54
*** adrian_otto has quit IRC19:56
vhosakotrwellum: ah, you're talking about enabling host-passthrough in the host VM to enable nested virtualization (VM-in-VM for kolla-k8s)?19:57
vhosakotrwellum: let me check me host virt configs.19:57
kfox1111sbezverk: sorry. nova-compute-ironic failed to merge. I thought it would be ok.19:58
vhosakotrwellum: where is that libvirt setting?  Is it    cpu mode='host-passthrough'   in /usr/share/libvirt/cpu_map.xml?  If so, I don't have it.20:00
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337520:01
sbezverkkfox1111: srwilkers: had to rebase one PS, please re-ack it when you have a moment..20:02
kfox1111sbezverk: done.20:02
sbezverkkfox1111: np, it was an easy fix20:02
vhosakotrwellum: let me know where that libvirt setting (host-passthrough) is?  I usually check "grep -i vmx /proc/cpuinfo" and "lsmod | grep kvm" if my host can do virtualization and create VMs using a hypervisor.20:04
*** manheim has joined #openstack-kolla20:04
rwellumvhosakot, in my XML definition I make sure I have: <cpu mode='host-passthrough'>20:05
vhosakotrwellum: ah, I don't have it.. should I restart libvirt after adding it?20:06
*** salv-orlando has quit IRC20:08
*** manheim has quit IRC20:08
rwellumvhosakot, I just suspended and resumed my VM - but I assume there's a less invasive way of doing this...20:09
*** vhosakot_ has joined #openstack-kolla20:09
vhosakot_the summit sched is up guys - https://www.openstack.org/summit/boston-2017/summit-schedule#day=2017-05-0820:09
*** kbaegis has quit IRC20:11
*** gfhellma has joined #openstack-kolla20:11
*** vhosakot has quit IRC20:11
rwellumvhosakot_, did you see note on VM above - looks like you timed out for a few?20:12
vhosakot_rwellum: yep, my wifi dropped, saw your message about suspending and resuming the VM.. cool.. I'll add it, thanks for the info!20:17
rwellumvhosakot_, ack20:17
vhosakot_rwellum: cool!20:17
rwellumsbezverk, I got past the docker issues on bare-metal! But now my pods are not coming up.20:20
*** mgoddard has quit IRC20:29
*** dave-mccowan has quit IRC20:30
*** kbaegis has joined #openstack-kolla20:32
kbaegisHi all. So I see that you're supposed to provide your own netboot kernel for ironic?20:33
rwellumvhosakot_, btw - not a bad work-flow is to bring up k8's via that k8s official doc. Then when sane, jump into the dev doc post vagrant step.20:33
kbaegis'"Unable to find '/etc/kolla/config/ironic/ironic-agent.kernel' in expected paths'?20:33
kbaegisAny documentation on the kernel generation?20:33
kbaegisoh, I use the diskimage-builder project?20:34
kbaegiscoolbeanw20:34
vhosakot_rwellum: yes, sbezverk told we need to deploy kubeadm for kolla-k8s.  so, I''ll follow the k8s official docs to deploy kubeadm on CentOS (https://kubernetes.io/docs/getting-started-guides/kubeadm/), and then use the post-vagrant steps to continue dev, very good point.20:35
kbaegisOkay, no space left on device keeps showing up for cinder20:38
kbaegiscinder-volume20:38
*** ccesario__ has quit IRC20:38
kbaegisswitched from aufs to btrfs. Is ubuntu-source broken for stable/ocata?20:39
rwellumvhosakot_, I got to the last stage of the dev guide doing that, but had docker issues. And then this week those performance issues. But will be good to see how far you get :)20:41
vhosakot_rwellum: yep, I'll keep you and the team posted! :)20:41
kbaegishttps://bugs.launchpad.net/kolla/+bug/1671447 updated20:44
openstackLaunchpad bug 1671447 in kolla "iscsi container fails to start, "no space left on device"" [Undecided,New]20:44
kbaegisCan someone please take a look?20:45
*** dave-mccowan has joined #openstack-kolla20:45
*** dave-mcc_ has joined #openstack-kolla20:47
*** Jezekus has joined #openstack-kolla20:48
sdakeportdirect ure talk got accepted ;)20:49
*** dave-mccowan has quit IRC20:50
vhosakot_rwellum: sbezverk: I convinced the vagrant devs in hte vagrant IRC channel to implement a new command "vagrant top" (similar to "docker top") to check the CPU and memory usage of the vagrant VMs.  we can use "vagrant top" to see CPU/memory stats on the vagrant host instead of ssh'ing into each vagrant vm and running the "top" command. I'll submit the proposal today.20:53
*** nathharp has joined #openstack-kolla20:53
*** m1dev has joined #openstack-kolla20:54
rwellumvhosakot_, :)20:55
*** JoseMello has quit IRC20:55
*** jrobinson has joined #openstack-kolla20:57
*** rwallner has quit IRC21:00
*** gfhellma has quit IRC21:03
*** rwallner has joined #openstack-kolla21:03
inc0ok guys, halcyon-ansible is no bueno for me;)21:04
inc0I now have 3 master nodes of k8s;) separately21:04
vhosakot_inc0: how did you deploy k8s without halcyon-ansible? use https://kubernetes.io/docs/getting-started-guides/kubeadm/ ?21:06
*** Jezekus has quit IRC21:07
inc0vhosakot_: I used it21:07
*** gfhellma has joined #openstack-kolla21:07
inc0and it deployed 3 separate clusters;)21:07
inc0so I'll need to rework it probably21:07
inc0with kubeadm it's going to be easy21:07
*** rwallner has quit IRC21:08
inc0that or I'll just create simple playbook to just wrap kubeadm21:08
inc0and make it part of kolla-ansible mwhahaha21:09
*** salv-orlando has joined #openstack-kolla21:09
*** zioproto has joined #openstack-kolla21:09
vhosakot_inc0: haha, three clusters.. there is sample inventory in https://github.com/att-comdev/halcyon-kubernetes/blob/master/sample-inventory that deploys 1 master 3 workers.21:11
inc0so I wanted proper ha21:11
inc0right?21:11
inc0;)21:11
inc0but I can deploy 1 controller I guess21:12
vhosakot_multi-cluster, multi-master _proper_ HA inc0, yes :)21:12
vhosakot_inc0: 1 controller meaning 1 worker?21:12
inc0I have 3 aios21:12
inc0but I think I know what I did wrong21:13
inc0in halcyon master != controller21:13
inc01 master and 3 controllers should be fine21:13
inc0let's try that21:13
sdakeinc0 sbezverk, portdirect, and srwilkers's talk was accepted21:13
inc0yay21:13
sdakeinc0 also we apparently ahve a 4 person project update session that we got roped into :)21:13
sdakeare you to blame for that21:13
inc0not sure about 4 person, but I submitted it yes21:13
sdakecool21:14
inc0results are published?21:14
sdakealso the panel was accepted21:14
sdakeyes it was accepted21:14
sdakeyes they are published21:14
jascott1ah that is "good" news21:14
inc0link plz21:14
sdakesup jascott1  ;-)21:14
jascott1whasup!21:14
vhosakot_inc0: I think the master as the controller as it runs the kube-controller-manager process.21:16
inc0yeah that might be21:16
vhosakot_sdake, inc0 sbezverk, portdirect, and srwilkers, congrats to all... I'll supply kolla stickers to attendees lol ;)21:18
inc0vhosakot_: we have plenty, and I hope we'll get more21:19
vhosakot_inc0: nice21:19
inc0kfox1111: I guess you'll be presenting project update with us;)21:19
sdakegotta roll21:19
srwilkersi came home from atlanta with quite a few ;)21:19
*** jtriley has quit IRC21:19
inc0I like that we have session - workshop - made by people I don't know:)21:20
inc0qwang, jascott1 panel was also accepted21:21
rwellumThose stickers were hard to get hold of at the PTG - the table was a swarm..21:22
*** nathharp has quit IRC21:22
rwellumI didn't deserve one though as I can't get kolla up...21:23
kbaegisGuys, I can't get cinder-volume to start21:23
kbaegisTheres space on disk, yet I get a 'no space left on device' warning21:23
kbaegisI followed the storage doc21:24
kbaegisNamely https://docs.openstack.org/developer/kolla-ansible/cinder-guide.html#for-ubuntu-and-lvm2-iscsi21:24
sbezverkkfox1111: have you noticed gate is all red21:25
sbezverkeven python27 is red21:25
kfox1111sbezverk: no. have a link?21:25
srwilkersyeah, saw that :(21:27
sbezverkkfox1111: https://review.openstack.org/#/c/443375/21:27
sbezverkkfox1111: there is any reference in my ps to nova-compute-ironic-daemonset, but helm by some reason still tries to render it21:28
kfox1111sbezverk: oh.21:28
kfox1111I think maybe we have a 2.0 reference somewhere still somhow?21:29
sbezverknope, just checked helm is 2.2.221:30
kfox1111hmm...21:30
*** eaguilar has quit IRC21:30
sbezverkkfox1111: what I do not get is if in my ps there is no any reference to old name, where it takes it?!?!?21:30
kfox1111yeah. shouldnt...21:31
kfox1111oh...21:31
*** manheim has joined #openstack-kolla21:31
kfox1111I think the ps needs to rename the requirements.yaml file over too.21:31
*** rabel has quit IRC21:31
manheimhello21:32
manheimi have a question about setting a proxy in cotainers21:32
manheimi saw the documentation on how to do that21:32
manheimcan I set that only once somewhere and the containers will inherit that proxy?21:32
sbezverkkfox1111: it looks like there is no requirements.yaml at all in the folder21:33
manheimor do i have to each container and set it in the dockerfile lets say21:33
*** l4yerffeJ has joined #openstack-kolla21:34
*** l4yerffeJ_ has quit IRC21:34
*** Jeffrey4l_ has quit IRC21:35
*** Jeffrey4l_ has joined #openstack-kolla21:35
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337521:36
*** dave-mcc_ has quit IRC21:36
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337521:38
*** krtaylor has quit IRC21:42
*** rhallisey has quit IRC21:45
*** dave-mccowan has joined #openstack-kolla21:46
*** markmcclain has joined #openstack-kolla21:49
*** MarginHu has joined #openstack-kolla21:51
*** jtriley has joined #openstack-kolla21:52
*** vhosakot_ has quit IRC21:56
*** vhosakot has joined #openstack-kolla21:56
kbaegisSo no one else has seen that? The "no space left on device"? inc0 sdake?21:58
*** fguillot has quit IRC21:59
inc0kbaegis: frankly I didn't test iscsi - I always deploy with ceph21:59
inc0sbezverk: can you help here?22:00
sbezverkinc0: yep22:01
MarginHuI use iscsi22:01
sdakesup peeps22:01
MarginHukbaegis , I didn't experience your issue.22:01
sbezverkkbaegis: can you provide a bot more context to the problem?22:02
sbezverkkbaegis: nvm please check volume group22:03
sbezverkkbaegis: it should be big enough to hold the volume you try to create22:03
kbaegissbezverk: It's failing on the container start. There's 12+tb of unallocated space22:05
kbaegisaufs and btrfs both failing22:05
kbaegisI might try ceph, but I really need the iops for a demo so ceph concerns me (AIO)22:06
inc0kbaegis: ceph is easier in our case than iscsi;)22:06
kbaegisall detailed here https://bugs.launchpad.net/kolla/+bug/167144722:06
openstackLaunchpad bug 1671447 in kolla "iscsi container fails to start, "no space left on device"" [Undecided,New]22:06
inc0as counter-intuitive as it sounds22:06
kbaegisinc0: Makes sense. ceph is awesome if you actually have multiple nodes22:06
kbaegisooc, is there.a config option to set the default storage pool replication factor?22:07
kbaegisIn case I do want ceph+AIO?22:07
inc0kbaegis: https://docs.openstack.org/developer/kolla-ansible/ceph-guide.html#configuration <- you have tip how to do ai22:07
inc0o22:07
kbaegisAhh, advanced config22:08
inc0"advanced"22:08
*** vhosakot_ has joined #openstack-kolla22:08
kbaegismakes sense. Trying a redeploy after a reboot, but I'll give ceph a go after this and update the bug22:08
kbaegisinc0: Thanks for the tip22:08
inc0np22:08
inc0only caveat is that ceph wants dedicated block device22:09
inc0I know it can be done with loopback, but never done it myself22:09
sbezverkkbaegis: can you paste vgs22:09
kbaegisinc0: I'm not using my cinder drives for anything yet22:09
kbaegissbezverk: Thats in there22:09
inc0well if you use ceph for them22:10
inc0that's actually more prod-ready than iscsi22:10
sbezverkkbaegis: sorry in where?22:10
inc0(other than it's aio...)22:10
*** manheim has quit IRC22:10
kbaegissbezverk: In the bug22:10
kbaegishttps://hastebin.com/bocarucile.scala22:11
kbaegisJust renamed cinder-hdd-volumes to cinder-volumes and mirrored change in globals22:11
kbaegisBut that's the same22:11
*** vhosakot has quit IRC22:11
*** fooliouno has quit IRC22:12
sbezverkkbaegis: can you run vgs22:12
sbezverkwhat you pasted is not what I asked..22:12
sbezverkalso cinder service-list22:12
sbezverkand cinder.conf would be helpfull22:12
kbaegishttps://hastebin.com/rinuvibumi.css22:13
kbaegiscinder volumes doesn't start22:13
inc0yay kube multinode deployed22:13
kbaegisinc0: Congrats22:14
inc0now I need to figure out how to connect ceph to it22:14
inc0kfox1111: little help here?:)22:14
kfox1111reading back.22:16
kfox1111ah.22:16
kfox1111sec...22:16
kfox1111yeah. I have time now.22:16
inc0cool, so my statys22:16
inc0I have k8s22:16
*** rwallner has joined #openstack-kolla22:16
inc0and I have ceph22:16
inc0and they don't know each other22:17
kfox1111and ceph is accessable to k8s? (just double checking. :)22:17
kfox1111(connectivity)22:17
sbezverkkfox1111: have you seen this error? http://logs.openstack.org/75/443375/8/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/abbfc8a/console.html#_2017-03-09_22_16_06_67143922:17
inc0yeah, same network22:17
kfox1111k :)22:17
kfox1111sbezverk: looking22:17
inc0where do I put user/keyring stuff?22:17
sbezverkkfox1111: I see it on 80% of jobs22:17
kfox1111sbezverk: thats new. looks like they forgot to sign a new package it seems.22:17
kfox1111sbezverk: we really need to mirror our k8s packages somewhere to ahve a stable version. :/22:18
inc0btw zuul v3 will be ansible based, we can leverage halcyon playbook in gates22:18
kfox1111sbezverk: you could try droping the gpg check on the repo.22:18
kfox1111inc0: interesting...22:19
inc0indeed, I'll wait for it to create kolla-ansible multinode gates22:19
sbezverkkfox1111: what is strange that for some jobs it works, I guess there is an inconsistency between mirrors22:20
kfox1111sbezverk: likely. I've seen that happen a bit with their repo.22:20
kfox1111inc0: so, the gate uses this: tools/setup_gate_ceph.sh22:21
*** rwallner has quit IRC22:21
kfox1111inc0: we probably should start at line 111.22:21
kfox1111though probaly need a step or two before that for your case.22:21
kfox1111so, the build_test_ceph.sh script builds a test script and uploads a configmap/secret to bind to ceph.22:22
kfox1111we need to build those by hand I think.22:22
inc0oh insteresing, so we run ceph client in container22:22
inc0that's nice22:22
kfox1111yup. :)22:22
kfox1111ceph isn't installed on the host at all.22:23
kfox1111even for the k8s support. :)22:23
inc0right22:23
masberhi22:23
inc0cool I'll dig through it22:23
kfox1111so, if we look in tests/bin/build_test_ceph.sh22:23
masberdoes kolla-ansible has the docker registry integrated?22:23
kfox1111we have:22:23
inc0masber: integrated - not really, but it's really one liner22:23
masberI can't find the tools/start-registry22:23
kfox1111kubectl create configmap ceph-conf --namespace=kolla --from-file=ceph.conf=/etc/ceph/ceph.conf22:23
masberim following this doc https://docs.openstack.org/developer/kolla-ansible/multinode.html22:24
kfox1111so the first part of the instructions should be, get a copy of ceph.conf from the ceph cluster, to your machine with kubectl,22:24
kfox1111and run:22:24
kfox1111kubectl create configmap ceph-conf --namespace=kolla --from-file=ceph.conf=<your cluster ceph.conf>22:24
inc0masber: it's in kolla, not kolla-ansible22:24
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337522:24
inc0geenrally - build == kolla, deploy == kolla-ansible22:25
inc0and keyring goes to secrets right?22:25
kfox1111inc0: yeah. thats next. we need accounts into secrets.22:26
masberdoesn't looks like it is in /usr/share/kolla/tools22:26
kfox1111those come from tools/setup-ceph-secrets.sh22:26
kfox1111I think all we need is ceph.client.admin.keyring;22:26
inc0right22:26
inc0I'll play around22:27
inc0first - get helm working22:27
masberthis is how I installed kolla pip install kolla==3.0.222:27
kfox1111so, like:22:27
inc0masber: I'm not sure if tools/start-registry was present in newton22:27
inc0as again...one line22:27
inc0yup, not present in newton22:27
deadnullIs CoreOS/rkt going to (or already) a supported route to take?22:28
masberyeah it is not present22:28
masberso I will have to install the registry by myself22:28
inc0deadnull: well, that's a good question22:28
*** adrian_otto1 has quit IRC22:28
deadnullinc0 :)22:28
masberbut actually kolla can deploy without a registry am I right?22:28
kfox1111well...22:28
inc0doing r&d how much it would take to run kolla on rkt is on my list to do22:28
kfox1111I think we have to figure this step out a bit.22:28
inc0masber: yeah22:28
masberkk22:28
deadnullinc0 I am trying to free up some cycles, I might be interested in helping you out22:29
inc0I think our images should work22:29
inc0kolla-ansible lacks support for rkt22:29
deadnullright, that seems like it would be where the lionshare of work is22:29
inc0but that's doable22:29
masberok i will check that22:29
inc0not necessarly22:29
inc0we have kolla-docker ansible module22:29
kfox1111inc0: do you remember the output of FETCH_CEPH_KEYS?22:29
inc0if we rename it to kolla-runtime (or sth) and provide equivalent commands to kolla_docker22:30
inc0kfox1111: no:()22:30
inc0it might actually be pretty easy22:30
inc0or very hard22:30
inc0depends on how much rkt differs from docker in our use cases22:30
kfox1111looks like it walks /etc/ceph/*, and just passes the whole file out.22:30
*** adrian_otto has joined #openstack-kolla22:30
inc0one thing tho, if our images aren't OCI-compatible, I would like them to be OCI-compatible22:31
*** schwicht has quit IRC22:31
inc0yeha I just copy files around22:31
*** lucasxu has quit IRC22:31
kfox1111inc0: so, tools/setup-ceph-secrets.sh just fetches the output, and unencodes them.22:31
inc0so I'll dig through our setup gate stuff22:31
kfox1111so looks like they are in full keytab format.22:31
inc0one thing I learned - I need to setup helm and kolla repos first22:31
*** jtriley has quit IRC22:31
inc0so I'll focus on that22:32
kbaegisSooo22:32
kfox1111so, should be able to copy the admin key somewhere that kubectl has access, then:22:32
kbaegisRenaming it cinder-volumes worked22:32
deadnullinc0 cool sounds good. will you be at the Summit in Boston?22:32
inc0I think so22:32
kbaegisSo the configuration value for cinder_volume_group isn't properly implemented22:32
inc0I hope to have some answers by then22:32
kfox1111kubectl create secret generic ceph.client.admin.keyring --namespace=kolla --from-file=data=<your admin keyring>22:33
kbaegisCan I mark a bug a defect, or does someone else have to do that?22:33
deadnullcool22:33
inc0if our images are oci-compatible, kolla-kubernetes should just work22:33
*** goldyfruit has quit IRC22:33
*** manheim has joined #openstack-kolla22:33
*** MarginHu has quit IRC22:34
kfox1111inc0: what version of ceph you got?22:35
inc0jewel22:36
kfox1111k.22:36
kfox1111do you have anything in it you care about? :)22:36
masberanother thing22:37
masberputting this --> http://pastebin.com/7epXvP1a on my /etc/systemd/system/docker.service.d/kolla.conf file, breaks docker22:37
masberi am using centos7 and docker Docker version 1.12.6, build 78d180222:37
kfox1111inc0: ok, I think this might do the trick, but do double check what its running if you care about your data:22:38
kfox1111http://paste.openstack.org/show/smlkVK54cUxk3IjBmBup/22:38
*** gfhellma has quit IRC22:40
*** gfhellma_ has joined #openstack-kolla22:40
sbezverkkfox1111: disabling gpg check helped.. but I lost again your +2, could you ack this ps? https://review.openstack.org/44337522:40
sbezverkkfox1111: when the gate finishes I will ask inc0 for a final bullet ;)22:40
*** MarginHu has joined #openstack-kolla22:40
kfox1111sbezverk: done.22:40
sbezverkkfox1111: our talk got accepted :)22:41
kfox1111sbezverk: sweet. :)22:41
sbezverkkfox1111: now is the hard part put together this talk ;)22:42
kfox1111inc0: hows that look?22:42
kfox1111sbezverk: indeed. :)22:42
*** adrian_otto has quit IRC22:49
*** jtriley has joined #openstack-kolla22:50
sdakesbezverk you will rock it i'm sure :)22:53
*** jtriley has quit IRC22:55
sbezverksdake: you were right, received email 10 minutes ago, how you knew about it before that?22:56
sdakesomeone else got an email and i looked on the website and saw it22:57
sbezverksdake: cool, I thought you might have a secret channel for board members ;)22:59
*** sayanta__ has quit IRC23:00
*** schwicht has joined #openstack-kolla23:03
rwellumsbezverk, which talk is this?23:03
*** jrobinson_ has joined #openstack-kolla23:05
*** MarginHu has quit IRC23:05
*** MarginHu has joined #openstack-kolla23:05
sbezverkrwellum: we submitted kolla-kube talk for this summit23:05
*** jrobinson has quit IRC23:07
rwellumI see it now: https://www.openstack.org/summit/boston-2017/summit-schedule/events/1773123:10
rwellumHelps if you look at the full schedule...23:10
*** jrobinson_ has quit IRC23:18
*** jrobinson_ has joined #openstack-kolla23:18
manheimi have a question about setting a proxy, saw the documentation23:19
manheimcan I set that only once somewhere and the containers will inherit that proxy?23:19
*** jrobinson_ is now known as jrobinson23:19
manheimor do i have to each container and set it in the dockerfile lets say23:19
*** zioproto has quit IRC23:21
sdakesbezverk wednesday 9am slot is decent23:22
sdakecan't ccomplain ;)23:22
sdakesbezverk there is an unoffical channel for board members however it is far from secret23:22
sdakei found out about it via eavesdropping ;)23:23
sdakethe talks being accepted that is23:23
sbezverksdake: the review I asked you earlier, would be great help to merge it ;)23:24
sdakeyou know your on to something when 50 people cop your shit23:24
*** MarginHu has quit IRC23:24
sdakesbezverk gotta bio - will do after23:24
sdakeafter i pay my bill and slam this drink23:24
kbaegisUmm. So when I run mount, I get about 50000 lines of shm on . . .23:24
*** MarginHu has joined #openstack-kolla23:24
kbaegisAnyone think this is related to the "No space left on device" issue?23:25
*** MarginHu has joined #openstack-kolla23:25
*** saneax-_-|AFK is now known as saneax23:25
kbaegislxd was still installed on ubuntu 1623:26
kbaegisHow do I fix/recover?23:26
*** rwellum has quit IRC23:27
*** schwicht has quit IRC23:27
*** MarginHu has quit IRC23:28
sdakesbezverk acked due23:30
sdakedude23:30
inc0kfox1111: rignt now I'm getting deployment itself streamlined23:30
inc0stuff like insecure address and such23:30
sbezverksdake: thanks a lot23:31
*** schwicht has joined #openstack-kolla23:32
*** schwicht has quit IRC23:33
kfox1111inc0: k.23:34
openstackgerritMerged openstack/kolla-kubernetes master: Swithcing nova-compute-ironic to use statefulset  https://review.openstack.org/44337523:34
kfox1111inc0: well, what part of deployment are you talking about?23:37
spsuryamorning all23:37
inc0kfox1111: so I my deploy node which will have kubectl and generally won't change23:38
inc0I want to be able to access it without redoing ssl cers all the time23:38
inc0which means changing apiserver manifest to allow insecure on 0.0.0.023:38
inc0trying to figure out how to do it in halcyon in a way that it will be set every time23:39
kfox1111ok.23:39
kfox1111this is one of the better times to discuss ceph though.23:39
inc0you know how to specify configs to apiserver like that for kubeadm?23:39
kfox1111not off hand.23:40
inc0ok good point23:40
inc0I'll manually do it23:40
inc0hold on23:40
sbezverkinc0: what do you want to know abount api server config?23:40
sbezverkthere is a place in etc/kubernetes/manifests/23:41
inc0sbezverk: I'm trying to figure out how to teach kubeadm to deploy k8s with listen insecure 0.0.0.023:41
inc0yeah, but how kubeadm generates that is my question23:41
sbezverkinc0: you cannot23:41
inc0right23:41
sbezverkyou need to change manifest23:41
sbezverkat /etc/kubernetes/manifests23:42
sbezverklet me give you example of config23:42
kbaegisRestart nova-libvirt container is failing "no space left on device"23:42
*** dave-mccowan has quit IRC23:43
kbaegisMade it past cinder at least23:43
sbezverkinc0: see linbe 7 http://paste.openstack.org/show/602126/23:43
sbezverkinc0: all you need is to add that line23:44
sbezverkkube automatically detect change and restart api server23:44
openstackgerritAbel Lopez proposed openstack/kolla-kubernetes master: Test for kube config  https://review.openstack.org/44403723:44
*** alop has joined #openstack-kolla23:44
inc0sbezverk: done that23:45
inc0now dealing with tiller restarting23:45
kfox1111inc0: kubectl describe pod tiller-xxx --namespace kube-system23:46
kfox1111did canal load ok?23:46
inc0yeah I know, not helpful for now23:46
inc0how to check that?23:46
kfox1111kubectl get pods --all-namespaces23:47
kfox1111in the describe output, does it say why its failing?23:47
inc0all the pods besides tiller and dashboard are running23:47
kfox1111if it says stuff about cni, its calico.23:47
kfox1111tiller and dashboard will run on the slave nodes I think.23:48
kfox1111so it could be networking related.23:48
inc0liveness probes gets connection refused23:48
inc0so something network23:49
kfox1111do an 'ip a show dev flannel.1' on the master,23:49
kfox1111log into a slave, and ping it.23:49
inc0so pinging container works23:51
kbaegishere's nova-compute failing to start23:51
kbaegishttps://hastebin.com/aruliquvum.sql23:51
inc0from master23:52
masbersorry another question23:52
masberin regards kolla 3.0.223:52
masberI have my kolla vm I am using for deployment23:53
masberbut I want to install openstack control plane and nona on a different VM23:53
*** MasterOfBugs has quit IRC23:53
masberso I changed the all-in-one inventory23:53
*** MasterOfBugs has joined #openstack-kolla23:53
kbaegisthere's no log made whatsoever. Docker doesn't start the container23:53
sbezverkinc0: when you installed canal plugin have you changed ip address for its etcd?23:54
masberand changed the network, control, compute and monitoring from localhost to my other VM's IP23:54
masberafter that, do i need to run kolla-ansible -i all-in-one bootstrap-servers?23:54
inc0sbezverk: no23:55
sbezverkinc0: when you downloaded canal.yaml there are a couple of places where you need to change addresses they use by default23:56
*** vhosakot_ has quit IRC23:56
inc0sbezverk: I used halcyon23:57
inc0but I think I see what's wrong23:57
inc0tiller has strange kube address23:57
*** jrobinson has quit IRC23:58
inc0it seems I used cluster network that's different from my api interface has23:58
inc0let me try to rebuild/redeploy23:58
sbezverkinc0: ah I see that I do not know how it works with halcyon23:59
*** schwicht has joined #openstack-kolla23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!