| *** thumpba has quit IRC | 00:04 | |
| *** thumpba has joined #openstack-kolla | 00:12 | |
| *** salv-orl_ has quit IRC | 00:13 | |
| *** neophy has joined #openstack-kolla | 00:16 | |
| *** rcarrillocruz has joined #openstack-kolla | 00:19 | |
| *** kklimonda has joined #openstack-kolla | 00:19 | |
| *** diogogmt has joined #openstack-kolla | 00:19 | |
| *** Jeffrey4l has joined #openstack-kolla | 00:26 | |
| *** weiyu has joined #openstack-kolla | 00:27 | |
| *** thumpba has quit IRC | 00:31 | |
| *** thumpba has joined #openstack-kolla | 00:34 | |
| *** vhosakot has quit IRC | 00:35 | |
| *** poul7777777 has left #openstack-kolla | 00:38 | |
| *** diogogmt has quit IRC | 00:53 | |
| *** mbound has joined #openstack-kolla | 00:55 | |
| *** thumpba has quit IRC | 00:55 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: [WIP] Pin the base distro release version https://review.openstack.org/319992 | 00:55 |
|---|---|---|
| *** diogogmt has joined #openstack-kolla | 00:56 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the rabbitmq crash on CentOS binary and source deploy https://review.openstack.org/319926 | 00:56 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Changes Glance bootstrap to use kolla-toolbox https://review.openstack.org/320022 | 00:57 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Configure the rabbitmq ha policy in config file https://review.openstack.org/319995 | 00:57 |
| *** mbound has quit IRC | 00:59 | |
| *** wznoinsk has joined #openstack-kolla | 00:59 | |
| *** weiyu has quit IRC | 01:00 | |
| *** kollausr has quit IRC | 01:01 | |
| *** brad[] has quit IRC | 01:03 | |
| *** brad[] has joined #openstack-kolla | 01:03 | |
| *** weiyu_ has joined #openstack-kolla | 01:04 | |
| *** diogogmt has quit IRC | 01:18 | |
| *** daneyon_ has joined #openstack-kolla | 01:23 | |
| *** daneyon has quit IRC | 01:28 | |
| *** vhosakot has joined #openstack-kolla | 01:42 | |
| *** tfukushima has joined #openstack-kolla | 01:43 | |
| *** ccesario_ has joined #openstack-kolla | 01:47 | |
| *** daneyon has joined #openstack-kolla | 01:50 | |
| *** rhallisey has quit IRC | 01:54 | |
| *** daneyon has quit IRC | 01:55 | |
| *** vhosakot has quit IRC | 02:04 | |
| *** vhosakot has joined #openstack-kolla | 02:04 | |
| *** neophy has quit IRC | 02:07 | |
| *** neophy has joined #openstack-kolla | 02:13 | |
| *** unicell1 has quit IRC | 02:14 | |
| *** sacharya has joined #openstack-kolla | 02:19 | |
| *** salv-orlando has joined #openstack-kolla | 02:38 | |
| *** salv-orlando has quit IRC | 02:43 | |
| *** diogogmt has joined #openstack-kolla | 02:49 | |
| openstackgerrit | Merged openstack/kolla: Change ansible image to kolla_toolbox https://review.openstack.org/320088 | 03:01 |
| *** vhosakot has quit IRC | 03:20 | |
| *** sdake_ has quit IRC | 03:29 | |
| *** unicell has joined #openstack-kolla | 03:35 | |
| *** dave-mcc_ has quit IRC | 03:38 | |
| *** daneyon has joined #openstack-kolla | 03:39 | |
| *** daneyon has quit IRC | 03:44 | |
| *** salv-orlando has joined #openstack-kolla | 04:08 | |
| *** ccesario___ has joined #openstack-kolla | 04:10 | |
| *** ccesario_ has quit IRC | 04:10 | |
| *** thumpba has joined #openstack-kolla | 04:10 | |
| *** salv-orlando has quit IRC | 04:17 | |
| *** sdake has joined #openstack-kolla | 04:17 | |
| *** sdake has quit IRC | 04:18 | |
| *** absubram has joined #openstack-kolla | 04:31 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Manila driver session is needed only in manila-share https://review.openstack.org/311232 | 04:31 |
| *** absubram_ has joined #openstack-kolla | 04:32 | |
| *** absubram has quit IRC | 04:36 | |
| *** absubram_ is now known as absubram | 04:36 | |
| *** tfukushima has quit IRC | 04:43 | |
| *** ntpttr has quit IRC | 05:15 | |
| *** ntpttr has joined #openstack-kolla | 05:16 | |
| *** tfukushima has joined #openstack-kolla | 05:20 | |
| *** sacharya_ has joined #openstack-kolla | 05:21 | |
| *** tfukushima has quit IRC | 05:21 | |
| openstackgerrit | Merged openstack/kolla: Generate configuration using ansible for any project https://review.openstack.org/318356 | 05:21 |
| *** sacharya has quit IRC | 05:23 | |
| *** sacharya has joined #openstack-kolla | 05:26 | |
| *** sacharya_ has quit IRC | 05:28 | |
| *** thumpba has quit IRC | 05:47 | |
| *** Mr_Broken has quit IRC | 06:09 | |
| *** tfukushima has joined #openstack-kolla | 06:10 | |
| *** sdake has joined #openstack-kolla | 06:11 | |
| *** ediardo has quit IRC | 06:14 | |
| *** elemoine_ has quit IRC | 06:14 | |
| *** NachoDuck has quit IRC | 06:14 | |
| *** serverascode has quit IRC | 06:15 | |
| *** DuncanT has quit IRC | 06:15 | |
| *** mgkwill has quit IRC | 06:15 | |
| *** ediardo has joined #openstack-kolla | 06:16 | |
| *** NobodyCam has quit IRC | 06:16 | |
| *** serverascode has joined #openstack-kolla | 06:17 | |
| *** mgkwill has joined #openstack-kolla | 06:18 | |
| *** DuncanT has joined #openstack-kolla | 06:18 | |
| *** NobodyCam has joined #openstack-kolla | 06:19 | |
| *** coolsvap has quit IRC | 06:21 | |
| *** macsz has quit IRC | 06:21 | |
| *** elemoine_ has joined #openstack-kolla | 06:22 | |
| *** serverascode has quit IRC | 06:22 | |
| *** daneyon has joined #openstack-kolla | 06:22 | |
| *** macsz has joined #openstack-kolla | 06:23 | |
| *** sdake has quit IRC | 06:25 | |
| *** coolsvap has joined #openstack-kolla | 06:26 | |
| *** daneyon has quit IRC | 06:27 | |
| *** harlowja has quit IRC | 06:28 | |
| *** NachoDuck has joined #openstack-kolla | 06:28 | |
| *** cmceniry has quit IRC | 06:29 | |
| *** serverascode has joined #openstack-kolla | 06:31 | |
| *** cmceniry has joined #openstack-kolla | 06:34 | |
| *** sdake has joined #openstack-kolla | 06:37 | |
| *** Serlex has joined #openstack-kolla | 06:37 | |
| *** salv-orlando has joined #openstack-kolla | 06:37 | |
| *** neophy has quit IRC | 06:40 | |
| *** fragatin_ has joined #openstack-kolla | 06:45 | |
| *** fragatina has quit IRC | 06:47 | |
| *** fragatin_ has quit IRC | 06:50 | |
| *** unicell has quit IRC | 06:50 | |
| *** unicell has joined #openstack-kolla | 06:51 | |
| *** archjun has joined #openstack-kolla | 06:52 | |
| *** sdake has quit IRC | 06:53 | |
| *** fragatina has joined #openstack-kolla | 06:57 | |
| *** archjun has quit IRC | 07:00 | |
| *** archjun has joined #openstack-kolla | 07:01 | |
| *** fragatina has quit IRC | 07:02 | |
| *** Jeffrey4l has quit IRC | 07:06 | |
| *** fragatina has joined #openstack-kolla | 07:07 | |
| mdnadeem | Jeffrey4l, coolsvap , i have deploying liberty openstack, however nova-novncproxy container keeps on restarting state , because of 0.0.0.0:6080 address already in use | 07:08 |
| *** fragatina has quit IRC | 07:08 | |
| coolsvap | mdnadeem, which service is using the port? | 07:08 |
| mdnadeem | Any idea , how to correct it, here is the logs: http://paste.openstack.org/show/498496/ | 07:08 |
| *** fragatina has joined #openstack-kolla | 07:09 | |
| mdnadeem | coolsvap, 27122/haproxy service is listen on it | 07:09 |
| *** archjun has quit IRC | 07:10 | |
| kjelly | mdnadeem: is the ip is your host ip ? 192.168.121.55 | 07:11 |
| *** athomas has joined #openstack-kolla | 07:12 | |
| *** sacharya has quit IRC | 07:13 | |
| mdnadeem | kjelly, no this is the spare ip in my network , that i have set "kolla_internal_vip_address: " in global.yml file | 07:14 |
| *** chopmann has joined #openstack-kolla | 07:19 | |
| *** mikelk has joined #openstack-kolla | 07:26 | |
| *** rmart04 has joined #openstack-kolla | 07:40 | |
| openstackgerrit | Liyi Meng proposed openstack/kolla-kubernetes: Implement neutron pods and service https://review.openstack.org/319293 | 07:43 |
| *** fragatina has quit IRC | 08:01 | |
| *** mbound has joined #openstack-kolla | 08:04 | |
| *** mbound has quit IRC | 08:05 | |
| *** mbound has joined #openstack-kolla | 08:06 | |
| *** berendt has joined #openstack-kolla | 08:08 | |
| *** shardy has joined #openstack-kolla | 08:12 | |
| *** sacharya has joined #openstack-kolla | 08:13 | |
| *** berendt has quit IRC | 08:13 | |
| *** dmk0202 has joined #openstack-kolla | 08:16 | |
| *** sacharya has quit IRC | 08:18 | |
| *** berendt has joined #openstack-kolla | 08:19 | |
| openstackgerrit | Liyi Meng proposed openstack/kolla-kubernetes: Implement swift pods and serivces https://review.openstack.org/319985 | 08:20 |
| *** salv-orlando has quit IRC | 08:25 | |
| openstackgerrit | Liyi Meng proposed openstack/kolla-kubernetes: Implement neutron pods and service https://review.openstack.org/319293 | 08:26 |
| openstackgerrit | Liyi Meng proposed openstack/kolla-kubernetes: Implement swift pods and serivces https://review.openstack.org/319985 | 08:29 |
| openstackgerrit | Liyi Meng proposed openstack/kolla: Replace underscore sign with dash in neutron config filename https://review.openstack.org/320159 | 08:36 |
| *** ankush has joined #openstack-kolla | 08:40 | |
| *** ankush has quit IRC | 08:46 | |
| *** pbourke_ has quit IRC | 08:47 | |
| *** pbourke_ has joined #openstack-kolla | 08:47 | |
| *** athomas has quit IRC | 08:53 | |
| *** wznoinsk has quit IRC | 08:59 | |
| *** wznoinsk has joined #openstack-kolla | 08:59 | |
| *** athomas has joined #openstack-kolla | 09:03 | |
| *** gfidente has joined #openstack-kolla | 09:09 | |
| *** SiRiuS__ has joined #openstack-kolla | 09:14 | |
| *** sdake has joined #openstack-kolla | 09:20 | |
| openstackgerrit | Liyi Meng proposed openstack/kolla-kubernetes: Implment swift bootstrap job https://review.openstack.org/320316 | 09:20 |
| *** zigo_ is now known as zigo | 09:28 | |
| *** prithiv has joined #openstack-kolla | 09:55 | |
| *** PyroMani has joined #openstack-kolla | 09:57 | |
| *** daneyon has joined #openstack-kolla | 09:58 | |
| *** sdake has quit IRC | 10:01 | |
| *** daneyon has quit IRC | 10:03 | |
| *** sacharya has joined #openstack-kolla | 10:15 | |
| *** sacharya has quit IRC | 10:19 | |
| *** prithiv has quit IRC | 10:23 | |
| *** prithiv has joined #openstack-kolla | 10:23 | |
| *** ccesario___ has quit IRC | 10:29 | |
| *** rhallisey has joined #openstack-kolla | 10:35 | |
| *** Jeffrey4l has joined #openstack-kolla | 10:45 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Cleanup Advanced Configuration doc https://review.openstack.org/319592 | 10:47 |
| *** prithiv has quit IRC | 10:53 | |
| *** prithiv has joined #openstack-kolla | 10:55 | |
| *** mdnadeem_ has joined #openstack-kolla | 10:56 | |
| *** prithiv has quit IRC | 10:57 | |
| *** mdnadeem has quit IRC | 11:00 | |
| *** weiyu_ has quit IRC | 11:03 | |
| *** dcwangmit01 has quit IRC | 11:04 | |
| *** liyi has quit IRC | 11:05 | |
| *** mlima has joined #openstack-kolla | 11:06 | |
| mlima | morning guys | 11:06 |
| rhallisey | hey | 11:06 |
| mlima | rhallisey, i did some updates in docs https://review.openstack.org/#/c/319899/ | 11:11 |
| patchbot | mlima: patch 319899 - kolla - Update Documentation | 11:11 |
| rhallisey | sweet | 11:11 |
| mlima | but you did too haha https://review.openstack.org/#/c/319592/3 | 11:11 |
| patchbot | mlima: patch 319592 - kolla - Cleanup Advanced Configuration doc | 11:11 |
| *** salv-orlando has joined #openstack-kolla | 11:15 | |
| mlima | i'll remove advanced configuration, and submmit my ps again. ok? | 11:15 |
| mlima | rhallisey, ^ | 11:15 |
| rhallisey | mlima, ya sure | 11:16 |
| *** brad[] has quit IRC | 11:16 | |
| *** sdake has joined #openstack-kolla | 11:17 | |
| rhallisey | mlima, you have +2 power so you can approve | 11:19 |
| rhallisey | if you think it's ready | 11:19 |
| *** salv-orlando has quit IRC | 11:19 | |
| mlima | not yet rhallisey | 11:19 |
| rhallisey | gotcha | 11:19 |
| mlima | i've only +1 and -1 | 11:22 |
| mlima | rhallisey, | 11:22 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Update Documentation https://review.openstack.org/319899 | 11:23 |
| *** brad[] has joined #openstack-kolla | 11:24 | |
| *** tfukushima has quit IRC | 11:28 | |
| rhallisey | I think you get added today mlima | 11:31 |
| mlima | hmm, no problem rhallisey | 11:32 |
| *** ccesario___ has joined #openstack-kolla | 11:39 | |
| *** salv-orlando has joined #openstack-kolla | 11:46 | |
| *** sdake has quit IRC | 11:49 | |
| openstackgerrit | Francesco Vollero proposed openstack/kolla: Fixed typos in Ansible files https://review.openstack.org/320382 | 11:50 |
| *** ccesario___ has quit IRC | 11:51 | |
| openstackgerrit | Francesco Vollero proposed openstack/kolla: Fixed typos in Ansible files https://review.openstack.org/320386 | 11:51 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Bump the ansible to 2 https://review.openstack.org/317421 | 11:53 |
| pbourke_ | anyone know for our ceph implementation, what was the reason behind using a label on the disks as opposed to just passing a list of disks to be provisioned | 11:55 |
| *** brad[] has quit IRC | 11:56 | |
| *** brad[] has joined #openstack-kolla | 11:57 | |
| *** prithiv has joined #openstack-kolla | 11:57 | |
| *** prithiv has quit IRC | 12:04 | |
| *** ccesario___ has joined #openstack-kolla | 12:05 | |
| *** weiyu_ has joined #openstack-kolla | 12:06 | |
| *** tfukushima has joined #openstack-kolla | 12:19 | |
| *** ayoung has quit IRC | 12:19 | |
| sbezverk | rhallisey: could you take another look at these two: https://review.openstack.org/#/c/316411/3 and https://review.openstack.org/#/c/320022/6 | 12:21 |
| patchbot | sbezverk: patch 316411 - kolla-kubernetes - Add Horizon's kubernetes objects | 12:21 |
| patchbot | sbezverk: patch 320022 - kolla-kubernetes - Changes Glance bootstrap to use kolla-toolbox | 12:21 |
| openstackgerrit | Liyi Meng proposed openstack/kolla: Replace underscore sign with dash in neutron config filename https://review.openstack.org/320159 | 12:26 |
| rhallisey | sbezverk, sure | 12:29 |
| *** liyi has joined #openstack-kolla | 12:33 | |
| *** weiyu_ has quit IRC | 12:34 | |
| *** salv-orlando has quit IRC | 12:35 | |
| *** SiRiuS__ has quit IRC | 12:40 | |
| *** weiyu_ has joined #openstack-kolla | 12:47 | |
| *** mgoddard__ has quit IRC | 12:50 | |
| *** diogogmt has quit IRC | 12:50 | |
| *** mgoddard__ has joined #openstack-kolla | 12:51 | |
| *** diogogmt has joined #openstack-kolla | 12:53 | |
| *** weiyu_ has quit IRC | 12:58 | |
| *** alyson_ has joined #openstack-kolla | 13:07 | |
| *** SiRiuS__ has joined #openstack-kolla | 13:07 | |
| *** diogogmt has quit IRC | 13:07 | |
| *** sdake has joined #openstack-kolla | 13:09 | |
| *** salv-orlando has joined #openstack-kolla | 13:11 | |
| rhallisey | liyi, maybe we could add a add something at the end of bootstrap in the kolla containers so that kube can determine by a probe that bootstrapping has completed | 13:12 |
| * rhallisey checks readiness probe | 13:12 | |
| *** sdake has quit IRC | 13:13 | |
| liyi | K8s is unlikey to have a way to check readiness of a container/pod. It just make sure if the process that run container is running. if it is in dead loop or in serivce, kube unlikely know about that. | 13:15 |
| rhallisey | liyi, but we can define a health check to be anything | 13:16 |
| liyi | the startup script could print readyness indication into fs | 13:16 |
| rhallisey | http://kubernetes.io/docs/user-guide/liveness/ | 13:16 |
| rhallisey | that is just a script | 13:17 |
| rhallisey | yes it could print that the db sync is complete | 13:17 |
| *** vdo has joined #openstack-kolla | 13:17 | |
| liyi | exactly, that is the way I though it can be done as well | 13:18 |
| rhallisey | so then the pods would need to figure our if db sync is done by reading the same location | 13:18 |
| liyi | so we need to define a common script for doing so, like the start.sh in kolla at the moment. Do you know how kolla perform health check now? | 13:19 |
| liyi | ping on tcp sockets? | 13:19 |
| rhallisey | haproxy does it | 13:19 |
| rhallisey | just checks the endpoints | 13:19 |
| liyi | Ok, then it is kind of ping on tcp sockets | 13:20 |
| liyi | i don't think it is a bad idea to add something like k8s is doing. it will improve the granularity of liveness checking | 13:21 |
| rhallisey | https://github.com/openstack/kolla/blob/master/ansible/roles/mariadb/tasks/lookup_cluster.yml | 13:21 |
| liyi | I think that could be done inside container, if not mariadb container itself, kolla_toolbox container could do that. | 13:23 |
| *** dave-mccowan has joined #openstack-kolla | 13:24 | |
| rhallisey | we did something for COPY_ONCE where if a file existed, then don't copy configs | 13:24 |
| rhallisey | I can't find that example | 13:24 |
| *** ayoung has joined #openstack-kolla | 13:27 | |
| rhallisey | liyi, I think it would require a shared mount on the host | 13:27 |
| rhallisey | if we save a file that maintains database info | 13:28 |
| rhallisey | liyi, that won't work for multinode without shared storage | 13:33 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the gate lying https://review.openstack.org/320458 | 13:36 |
| liyi | Ryan, not sure if i am following. do you mean shared mount between host and container to check readiness is not working? | 13:36 |
| rhallisey | liyi, there needs to be for the mariadb pods to determine that the mariadb bootstrap in in progress | 13:37 |
| rhallisey | s/needs to be/needs to be a way/ | 13:37 |
| *** inc0 has joined #openstack-kolla | 13:39 | |
| liyi | hmm, i am not quite sure about how mariadb cluster bootstrap work. | 13:40 |
| inc0 | liyi, nobody is;) hey everyone btw | 13:41 |
| rhallisey | liyi, https://github.com/openstack/kolla/blob/master/docker/mariadb/extend_start.sh#L37 | 13:41 |
| liyi | dose it have seed nodes concept? | 13:41 |
| rhallisey | liyi, upon completion, we need a way to let the mariabd pod know it can run | 13:41 |
| rhallisey | inc0, hey :) | 13:42 |
| sbezverk | rhallisey: we can store key/value in etcd and check | 13:42 |
| rhallisey | yes | 13:42 |
| rhallisey | that's what I was thinking too | 13:42 |
| inc0 | hey | 13:42 |
| rhallisey | because we can't use shared storage on the host adn this is what etcd is good at | 13:42 |
| inc0 | but aren't we running single maria? | 13:43 |
| inc0 | non-clustered? | 13:43 |
| rhallisey | yes single maria | 13:43 |
| sbezverk | rhallisey: very easy to store and retrieve | 13:43 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Pin the base distro release version https://review.openstack.org/319992 | 13:43 |
| sbezverk | we just need to add logic to chck for it | 13:43 |
| * rhallisey is reading the raft consneus algorithm :) | 13:44 | |
| rhallisey | sbezverk, yes. We can pass it around as an env var | 13:44 |
| *** ssurana has joined #openstack-kolla | 13:45 | |
| rhallisey | inc0, so the idea here is that we need a way for bootstrapping to run uninteruppted | 13:45 |
| rhallisey | since mariadb pods keep killing the bootstrap | 13:45 |
| inc0 | ahh...yeah. the bootstrap | 13:46 |
| *** chopmann has quit IRC | 13:46 | |
| liyi | a bit lost, are we talking a cluster or a single node? if it is single node, then no problem regarding multi-nodes, right? | 13:46 |
| inc0 | liyi, with kubernetes you can run single node mariadb | 13:47 |
| *** m3m0 has quit IRC | 13:47 | |
| inc0 | and for HA you depend on k8s to restart containers if something goes wrong | 13:47 |
| rhallisey | liyi, in the mariadb exmaple we're talking single node | 13:47 |
| rhallisey | but this problem extends to all bootstrapping | 13:47 |
| rhallisey | we don't want services starting before they are bootstrapping | 13:48 |
| liyi | OK, it is single node, seprating bootstrap from run service could be helpful? | 13:48 |
| rhallisey | bootstrapped* | 13:48 |
| inc0 | rhallisey, well, why? | 13:48 |
| rhallisey | liyi, it already is that way | 13:48 |
| inc0 | so bootstrap is running | 13:48 |
| inc0 | you start service, it fails | 13:48 |
| inc0 | k8s restart it | 13:48 |
| rhallisey | inc0, but it doesn't always fail | 13:48 |
| inc0 | bootstrap finishes, and service stop failing | 13:48 |
| inc0 | maybe add healthcheck then? | 13:49 |
| liyi | Ryan, but you still have probme? | 13:49 |
| rhallisey | inc0, this is the healthcheck | 13:49 |
| *** mgoddard__ has quit IRC | 13:49 | |
| *** m3m0 has joined #openstack-kolla | 13:49 | |
| *** chopmann has joined #openstack-kolla | 13:49 | |
| inc0 | ahh...ok | 13:49 |
| inc0 | ;) | 13:49 |
| rhallisey | make sure db sync is done | 13:49 |
| rhallisey | this also would apply to upgrades | 13:49 |
| d_code_ | morning all… I’m trying to get PCI passthrough to work with nova… I feel like I’m really close… I think something is mismatched on translating the pci_alias from the flavor to the list of devices… how could I patch nova (in my dev env) to dump the data I’m interested in? Can I just modify the container and restart it somehow? | 13:49 |
| rhallisey | we can start the services back up again until *some actions* take place | 13:50 |
| rhallisey | s/can/cant | 13:50 |
| rhallisey | inc0, unfortunately health checks are not advanced enought to be cluster away | 13:51 |
| liyi | what if we have a startup script pulling a ENV from etcd like system to get it synced? | 13:51 |
| rhallisey | aware | 13:51 |
| *** chopmann_ has joined #openstack-kolla | 13:51 | |
| rhallisey | inc0, check etcd to see if bootstrap_mariadb=DONE would give that container the check it needs to run :) | 13:52 |
| rhallisey | hehe we're making dependencies in kube :) | 13:52 |
| inc0 | well, truth be told you're asking for trouble with this | 13:52 |
| inc0 | so what happends if node with bootstrap running dies? | 13:53 |
| inc0 | who will release the lock? | 13:53 |
| rhallisey | kube would reschedule another bootstrapping job | 13:53 |
| liyi | Ryan, so the option might be kolla-k8s rebuild image with overriting startup scripts | 13:53 |
| *** mgoddard_ has joined #openstack-kolla | 13:53 | |
| rhallisey | since there would be 0 bootstrapping mariadb jobs running and etcd would show it's not complete | 13:53 |
| inc0 | why rebuild? | 13:53 |
| *** chopmann has quit IRC | 13:54 | |
| inc0 | you shouldn't rebuild at any point really | 13:54 |
| inc0 | hmm | 13:54 |
| liyi | Inc0, we want to change the startup behaviour of container | 13:54 |
| inc0 | how? | 13:54 |
| rhallisey | liyi, that is another option, but how would you make the cluster away a bootstapping task is completed? | 13:55 |
| rhallisey | docker ps -a | grep bootstrapp? | 13:55 |
| rhallisey | inside the contiainer? | 13:55 |
| inc0 | so can you make k8s schedule pods when number_of_bootstraps=0 and BOOTSTRAPPED=false? | 13:55 |
| liyi | Inc0, if we have k8s startup script for container, we re-build the container image with overrite it entry.sh | 13:55 |
| sbezverk | rhallisey confused, if we use etcd, it is a cluster and its key/value already synced across all nodes | 13:56 |
| inc0 | well, I'd love to have co-usable containers | 13:56 |
| inc0 | but we had extended logic inside container before with mesos | 13:56 |
| rhallisey | inc0, I think so. I'm not sure of the kube etcd interaction | 13:56 |
| inc0 | so if you could do that | 13:56 |
| rhallisey | sbezverk, yes that's what we want | 13:56 |
| liyi | Ryan, while(bootstrap=flase); sleep(10);readbootstrap() | 13:57 |
| inc0 | you can start bootstraping by simply managing these variables | 13:57 |
| sbezverk | rhallisey: skydns uses this approach | 13:57 |
| inc0 | you want to start mariadb? you add variable BOOTSTRAPPED_MARIA=false | 13:57 |
| sbezverk | they store dns DB in a local instance of etcd | 13:57 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Pin the base distro release version https://review.openstack.org/319992 | 13:57 |
| inc0 | k8s takes over and mariadb container, inside will change BOOTSTRAPPED_MARIA=true when it finishes | 13:57 |
| inc0 | and mariadb container healthcheck will check this variable | 13:58 |
| inc0 | if not true- die painfully | 13:58 |
| rhallisey | liyi, where does readbootstrap() go? | 13:58 |
| inc0 | see where I'm getting at | 13:58 |
| inc0 | ? | 13:58 |
| rhallisey | inc0, yes | 13:59 |
| liyi | readbootstrapFromEctd() | 13:59 |
| liyi | the idea is to read the Env periodically until it is true | 13:59 |
| rhallisey | liyi, so we're saying something very similar | 13:59 |
| liyi | aha, that is nice. | 14:00 |
| liyi | i have problem in catching up in IRC | 14:00 |
| inc0 | well I wouldn't sleep | 14:00 |
| inc0 | I'd kill container | 14:00 |
| rhallisey | inc0, what other issues do you think we can have | 14:01 |
| liyi | inc0 let k8s restart it? | 14:01 |
| rhallisey | inc0, liyi, I think we would just exit(1) | 14:01 |
| inc0 | liyi, correct | 14:01 |
| rhallisey | let kube restart it | 14:01 |
| inc0 | rhallisey, let's prototype this maria | 14:01 |
| inc0 | and go from there | 14:01 |
| rhallisey | ya I agree | 14:01 |
| liyi | That is even smarter :) | 14:01 |
| inc0 | if it works, we can do something like that for everything else | 14:02 |
| rhallisey | if we track the bootstrap state in etcd it should translate into upgrades | 14:02 |
| inc0 | yeah | 14:02 |
| rhallisey | this would fix some of the upgrade issues I mentioned on the mailing list | 14:02 |
| liyi | that is right ! | 14:02 |
| inc0 | really it would be as simple as changing bootstrapped var to false | 14:03 |
| rhallisey | or be insight into how we could fix it | 14:03 |
| rhallisey | ya exactly | 14:03 |
| inc0 | k8s will do the rest | 14:03 |
| rhallisey | inc0, have do made a service yet for kube? | 14:04 |
| inc0 | nope | 14:04 |
| inc0 | about to spawn vms to deploy k8s | 14:04 |
| inc0 | what's best way to deploy k8s? | 14:05 |
| rhallisey | ceph is open | 14:05 |
| rhallisey | :D | 14:05 |
| rhallisey | I use hypercube | 14:05 |
| rhallisey | kubernetes in containers | 14:05 |
| inc0 | I don't think we should deploy ceph with kube | 14:05 |
| liyi | I have it with hypercube on Cent7 | 14:05 |
| rhallisey | inc0, using hypercude is super easy to get up and running | 14:05 |
| inc0 | so even mons aren't really good idea | 14:05 |
| rhallisey | https://github.com/openstack/kolla-kubernetes/blob/master/doc/source/kubernetes-all-in-one.rst | 14:06 |
| inc0 | well, you could | 14:06 |
| inc0 | but I'd keep ceph out of it really | 14:06 |
| inc0 | and make kube use ceph if anything | 14:06 |
| rhallisey | ya | 14:07 |
| rhallisey | there's some further discussion to be had around that for sure | 14:07 |
| rhallisey | once we transition to multinode | 14:07 |
| *** dwalsh has joined #openstack-kolla | 14:08 | |
| *** rhallisey has left #openstack-kolla | 14:08 | |
| inc0 | I'll deploy multinode;) | 14:09 |
| *** thumpba has joined #openstack-kolla | 14:09 | |
| *** thumpba has quit IRC | 14:09 | |
| *** rhallisey has joined #openstack-kolla | 14:10 | |
| *** thumpba has joined #openstack-kolla | 14:11 | |
| sbezverk | inc0 you are late, I already have multinode ;) but after playing with aio, I like it better.. | 14:12 |
| liyi | inc0, in case you want to have hypekube with mutlinode on rh7/CentOS7, this might help https://github.com/kubernetes/kubernetes/compare/master...liyimeng:master | 14:12 |
| *** absubram has quit IRC | 14:13 | |
| inc0 | thanks guys! | 14:14 |
| *** diogogmt has joined #openstack-kolla | 14:14 | |
| inc0 | I'm running butnu | 14:14 |
| *** 64MAAATDJ has joined #openstack-kolla | 14:14 | |
| *** 64MAAATDJ has quit IRC | 14:14 | |
| *** thumpba has quit IRC | 14:14 | |
| *** diogogmt has quit IRC | 14:23 | |
| *** sacharya has joined #openstack-kolla | 14:24 | |
| *** sacharya has quit IRC | 14:24 | |
| sean-k-mooney | rhallisey: i have not been following the kolla-kubernetese work as closely as i should. is the aio setup functional | 14:25 |
| rhallisey | sean-k-mooney, yes | 14:25 |
| rhallisey | follow that doc I linked above | 14:26 |
| rhallisey | https://github.com/openstack/kolla-kubernetes/blob/master/doc/source/kubernetes-all-in-one.rst | 14:26 |
| sean-k-mooney | yes reading it now | 14:26 |
| rhallisey | its like 4 steps | 14:26 |
| sean-k-mooney | cool i might try to deploy it soon | 14:26 |
| rhallisey | cool | 14:27 |
| liyi | Ryan, I am off for today now. If you want me to help in some task, e.g. splitting cli, pls let me know. | 14:28 |
| *** logan- has quit IRC | 14:29 | |
| rhallisey | liyi, sounds good. See ya later! | 14:29 |
| *** jmccarthy has joined #openstack-kolla | 14:29 | |
| sean-k-mooney | rhallisey: is the aio setup currently using nova-networks? i do not see neutron in the services folder | 14:30 |
| rhallisey | sean-k-mooney, the services aren't fully functionally yet | 14:30 |
| *** logan- has joined #openstack-kolla | 14:30 | |
| rhallisey | neutron is up in the queue | 14:30 |
| sean-k-mooney | ok cool just trying to understand what the aio setup currently deploys before i try it out | 14:31 |
| rhallisey | sean-k-mooney, I'll need to add a bunch more in the docs to describe that. We're still working through some of the bootstrapping kinks | 14:32 |
| rhallisey | but some of the services do get up and runnig | 14:32 |
| sean-k-mooney | ok cool i know this is just forming so dont expect it to all work right way :) | 14:33 |
| sean-k-mooney | do ye recommend any particalar os combination ubuntu source vs centos binary 14.04 vs 16.04 | 14:34 |
| *** ssurana has quit IRC | 14:35 | |
| rhallisey | :) | 14:35 |
| rhallisey | I've only tried on centos thus far | 14:35 |
| rhallisey | inc0, is trying on ubuntu as we speak | 14:35 |
| rhallisey | sean-k-mooney, kolla-kubernetes uses kolla for config generation and building images | 14:36 |
| sean-k-mooney | i have hit issues building and deploying both recently with ansible so i am going to set up a clean env tomorow. | 14:38 |
| *** ssurana has joined #openstack-kolla | 14:38 | |
| sean-k-mooney | i think part of the problem was the node i built the container on was centos 7 as were the contaienr but the nodes i deployed too were ubuntu 16.04 | 14:38 |
| rhallisey | sean-k-mooney, ya you want to be careful with mixing the base os. It doesn't work in all cases | 14:40 |
| sean-k-mooney | does ubuntu 16.04 work with kolla yet? i might revert to 14.04 | 14:42 |
| *** wznoinsk has quit IRC | 14:42 | |
| *** mgoddard_ has quit IRC | 14:42 | |
| *** ccesario___ has quit IRC | 14:42 | |
| *** athomas has quit IRC | 14:42 | |
| sean-k-mooney | i could install centos but i want to use btrfs as the storage driver and im not sure how well brtfs is supported in the centos kernel | 14:43 |
| openstackgerrit | Merged openstack/kolla: Make configurable the location where config files are merged https://review.openstack.org/318648 | 14:44 |
| *** fragatina has joined #openstack-kolla | 14:45 | |
| *** fragatina has quit IRC | 14:45 | |
| coolsvap | sean-k-mooney, you need a device with btrfs and mount it on /var/lib/docker | 14:46 |
| coolsvap | thats what i do | 14:46 |
| *** fragatina has joined #openstack-kolla | 14:46 | |
| sean-k-mooney | coolsvap: ya thats what i do on my home ceph/nas and it works fine there but i replaced the 3.10 centos kernel with an upstream 4.4 kenel as i think btrfs was still beta in 3.10 | 14:47 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add title to index page of developer documentation https://review.openstack.org/320499 | 14:48 |
| *** sdake has joined #openstack-kolla | 14:49 | |
| sdake | morning | 14:50 |
| *** dwalsh has quit IRC | 14:50 | |
| inc0 | sean-k-mooney, afair we merged ansible 2 yesterday | 14:50 |
| inc0 | do you have ansible issues? | 14:50 |
| inc0 | and btrfs works well with buntu 16+ | 14:51 |
| inc0 | (or 15.10, thats what I'm running) | 14:51 |
| *** Lyncos has left #openstack-kolla | 14:52 | |
| *** mlima has quit IRC | 14:54 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Specify registry volume location https://review.openstack.org/319585 | 14:55 |
| openstackgerrit | Merged openstack/kolla: Fixed typos in Ansible files https://review.openstack.org/320382 | 15:00 |
| sean-k-mooney | inc0: i was hitting some issue with trying deploy multi node with mariadb | 15:00 |
| openstackgerrit | Merged openstack/kolla: Fixed typos in Ansible files https://review.openstack.org/320386 | 15:00 |
| inc0 | sean-k-mooney, "some issue"?;) | 15:00 |
| *** absubram has joined #openstack-kolla | 15:02 | |
| sean-k-mooney | i do not have access to the system currently | 15:02 |
| inc0 | uhh | 15:03 |
| sean-k-mooney | basically the deploy failed either trying to start the mariadb or creating the hecka user | 15:03 |
| inc0 | is it fresh master? | 15:03 |
| sean-k-mooney | it was but it was a centos source build deploying to a 16.04 host which is why i want to try this again using the same host and guest os | 15:04 |
| *** mgoddard has joined #openstack-kolla | 15:04 | |
| *** inc0 has quit IRC | 15:05 | |
| sean-k-mooney | i was useing centos for the bifrost stuff but had ubuntu installed on all my other nodes so i didnt get around to rebuilding the images. | 15:05 |
| *** inc0 has joined #openstack-kolla | 15:06 | |
| inc0 | sean-k-mooney, we don't support 16.04 inside containers yet | 15:06 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Update ubuntu dockerfiles for formatting https://review.openstack.org/317238 | 15:06 |
| sean-k-mooney | inc0: good to know. i saw your patch that chages latest to 14.04 | 15:07 |
| inc0 | yeah | 15:07 |
| inc0 | we're working on 16.04 | 15:07 |
| inc0 | but still stuff to do | 15:07 |
| *** jmccarthy has left #openstack-kolla | 15:08 | |
| sean-k-mooney | so the best way to deploy kolla currently would be ubuntu source on a 14.04 host or centos source on a centos 7.2 host correct | 15:08 |
| inc0 | so I deploy with 15.10 on host and 14.04 inside container | 15:09 |
| inc0 | didnt test 16.04 on host | 15:10 |
| inc0 | but should work | 15:10 |
| *** diogogmt has joined #openstack-kolla | 15:10 | |
| sean-k-mooney | that should work fine. the main problem with mixing centos and ubutue is the fact that selinux and apparmor dont like each other | 15:11 |
| *** Kennan has quit IRC | 15:11 | |
| *** vhosakot has joined #openstack-kolla | 15:13 | |
| *** jmccarthy has joined #openstack-kolla | 15:14 | |
| *** Kennan has joined #openstack-kolla | 15:14 | |
| sean-k-mooney | i also hit this bug in the centos source build https://bugs.launchpad.net/kolla/+bug/1577273 | 15:14 |
| openstack | Launchpad bug 1577273 in kolla "Failure of creating haproxy mysql user" [Critical,New] - Assigned to Serguei Bezverkhi (sbezverk) | 15:14 |
| *** jrist has quit IRC | 15:15 | |
| *** jtriley has joined #openstack-kolla | 15:15 | |
| sean-k-mooney | the workaround got me past the intial error but the [mariadb | Creating haproxy mysql user] task was the one that was still failing | 15:16 |
| *** Junhongl_ has joined #openstack-kolla | 15:16 | |
| *** ssurana has quit IRC | 15:18 | |
| *** Junhongl_ has left #openstack-kolla | 15:20 | |
| *** ssurana has joined #openstack-kolla | 15:24 | |
| *** mlima has joined #openstack-kolla | 15:28 | |
| inc0 | sbezverk, which vars I need to change to deploy multinode k8s with hypercube? | 15:29 |
| *** jtriley has quit IRC | 15:30 | |
| *** salv-orlando has quit IRC | 15:34 | |
| *** daneyon has joined #openstack-kolla | 15:36 | |
| *** mlima is now known as mliima | 15:37 | |
| *** dmk0202 has quit IRC | 15:38 | |
| *** mliima has left #openstack-kolla | 15:38 | |
| *** dmk0202 has joined #openstack-kolla | 15:38 | |
| *** daneyon_ has quit IRC | 15:39 | |
| *** ssurana has quit IRC | 15:39 | |
| *** mliima has joined #openstack-kolla | 15:40 | |
| *** egonzalez has joined #openstack-kolla | 15:40 | |
| *** ccesario___ has joined #openstack-kolla | 15:40 | |
| *** athomas has joined #openstack-kolla | 15:40 | |
| *** Junhongl_ has joined #openstack-kolla | 15:41 | |
| *** salv-orlando has joined #openstack-kolla | 15:41 | |
| *** fragatina has quit IRC | 15:43 | |
| *** Junhongl_ has quit IRC | 15:44 | |
| *** Junhongl_ has joined #openstack-kolla | 15:44 | |
| *** wznoinsk has joined #openstack-kolla | 15:45 | |
| *** tfukushima has quit IRC | 15:46 | |
| *** jrist has joined #openstack-kolla | 15:46 | |
| *** mikelk has quit IRC | 15:46 | |
| *** Junhongl_ has quit IRC | 15:47 | |
| *** jrist has quit IRC | 15:47 | |
| *** jrist has joined #openstack-kolla | 15:48 | |
| *** Junhongl has joined #openstack-kolla | 15:48 | |
| *** fragatina has joined #openstack-kolla | 15:49 | |
| *** ssurana has joined #openstack-kolla | 15:49 | |
| sbezverk | inc0 I deployed multinode manually.. | 15:51 |
| *** Junhongl has quit IRC | 15:51 | |
| *** ssurana has quit IRC | 15:52 | |
| *** diogogmt has quit IRC | 15:54 | |
| *** ssurana has joined #openstack-kolla | 15:56 | |
| *** jrist has quit IRC | 15:57 | |
| *** wznoinsk has quit IRC | 15:57 | |
| *** mgoddard_ has joined #openstack-kolla | 15:58 | |
| inc0 | kk | 15:58 |
| inc0 | thanks | 15:58 |
| *** diogogmt has joined #openstack-kolla | 15:59 | |
| *** jtriley has joined #openstack-kolla | 16:00 | |
| *** mgoddard has quit IRC | 16:01 | |
| *** fragatina has quit IRC | 16:02 | |
| *** inc0 has quit IRC | 16:04 | |
| *** jrist has joined #openstack-kolla | 16:06 | |
| *** wznoinsk has joined #openstack-kolla | 16:06 | |
| *** inc0 has joined #openstack-kolla | 16:06 | |
| *** jmccarthy has left #openstack-kolla | 16:09 | |
| *** ssurana has quit IRC | 16:16 | |
| *** chopmann_ has quit IRC | 16:16 | |
| *** sacharya has joined #openstack-kolla | 16:18 | |
| sdake | sean-k-mooney re that haproxy failing that is likely an envrionment problem | 16:21 |
| *** ssurana has joined #openstack-kolla | 16:22 | |
| sdake | does kolla work with ansible 2.0 now Jeffrey4l inc0 ? | 16:22 |
| inc0 | yeah | 16:23 |
| sdake | does it work with 1.9.4? | 16:23 |
| inc0 | nope | 16:23 |
| Jeffrey4l | sdake, yes. patch is here https://review.openstack.org/317421 | 16:23 |
| sdake | our docs are wrong | 16:23 |
| Jeffrey4l | the patch is not merged. | 16:23 |
| inc0 | hmm | 16:23 |
| Jeffrey4l | after it is merged, i will update the doc. | 16:24 |
| inc0 | I thought I +2 it yesterday | 16:24 |
| *** dmk0202 has quit IRC | 16:24 | |
| Jeffrey4l | i pushed a new one. Mauricio found a bug. | 16:24 |
| *** unicell has quit IRC | 16:24 | |
| sdake | did the centos gat esomehow magically fix itself? | 16:24 |
| inc0 | ok, well, I have memory of a goldfish apperantly | 16:25 |
| sdake | inc0 wait until your 42! | 16:25 |
| sdake | i've got a million things to remember | 16:25 |
| sdake | and can still remember 7 +/- 2 things just like the rest of humanity | 16:25 |
| sdake | Jeffrey4l did you fix the centos gate, and if so what was the resolution? | 16:26 |
| Jeffrey4l | sdake, just pushed a workaround and got merged. | 16:26 |
| Jeffrey4l | sdake, https://review.openstack.org/315860 | 16:26 |
| Jeffrey4l | here. | 16:26 |
| *** jtriley has quit IRC | 16:27 | |
| Jeffrey4l | it is a hostname issue as I told. But i have no idea why it is not changed on the CentOS. | 16:27 |
| sean-k-mooney | sdake: untill https://bugs.launchpad.net/kolla/+bug/1577273 is fixed the centos source build will be broken | 16:27 |
| openstack | Launchpad bug 1577273 in kolla "Failure of creating haproxy mysql user" [Critical,New] - Assigned to Serguei Bezverkhi (sbezverk) | 16:27 |
| sdake | sean-k-mooney i see, that is our typical playbook where a failure happens because of environmental problems | 16:30 |
| sdake | in this case, it looks like something is being installed in the container that shouldn't be (see last comment) | 16:30 |
| sdake | playbook/task | 16:32 |
| sdake | sbezverk you want me to take that one | 16:32 |
| sdake | i have extensive understanding of the setuptools issue | 16:33 |
| sdake | sbezverk your solution is not viable | 16:33 |
| sbezverk | sdake sure thing | 16:33 |
| *** sdake_ has joined #openstack-kolla | 16:35 | |
| *** d_code_ has quit IRC | 16:37 | |
| sdake_ | rhallisey ping | 16:38 |
| rhallisey | wassup | 16:38 |
| *** sdake has quit IRC | 16:38 | |
| sdake_ | re custom repos/files/etc | 16:38 |
| sdake_ | here is what i'd like | 16:38 |
| sdake_ | and i'm willing to do the work since i think your super busy with kolla-kubernetes | 16:38 |
| sdake_ | i'd like per-container repositories | 16:38 |
| sdake_ | so different containers could have different dependencies | 16:39 |
| rhallisey | sure. It's on my to do list, but would love some help | 16:40 |
| rhallisey | so the case you're outlining are you moving for example the mariadb repo into the mariadb container? | 16:40 |
| *** rmart04 has quit IRC | 16:40 | |
| rhallisey | there would still be a base repo, but I guess you override it with custom-<service>.repo | 16:41 |
| sdake_ | i am not sure how to do the actual work | 16:42 |
| sdake_ | i was thinking something with buildconf | 16:42 |
| sdake_ | but yup that is the general idea | 16:42 |
| rhallisey | the main reason I was kinda stuck was because I wasn't sure what to do next | 16:42 |
| rhallisey | I"m not sure either | 16:42 |
| sdake_ | so defaults woudl be base contains everything we need | 16:43 |
| sdake_ | if any overrides are found, base no longer becomes the base for that repo/file/git | 16:43 |
| rhallisey | well we don't install any service specific packages in the base. I think if you say enabled delorean in the base and had a custom maraidb repo install that package, it would install the correct package in the mariadb contianer | 16:44 |
| sdake_ | i think operators will want to build from one set of docker files with different rpm dependencies | 16:45 |
| rhallisey | so I think you can still layer off the base unless the base repo has a newer version of the pacakge | 16:45 |
| sdake_ | to run mitaka side by side with liberty for example | 16:45 |
| sdake_ | is buildconf read by oslo? | 16:46 |
| rhallisey | so that case you would use rdo-liberty with a custom repo. Then rebuild with rdo-mitaka and a custom repo | 16:47 |
| sdake_ | yes i knwo the rebuild part works | 16:47 |
| sdake_ | but i think people will want to integrate in their ci/cd pipelines | 16:47 |
| sdake_ | and not build each container indivdiually | 16:47 |
| sdake_ | sean-k-mooney are you buildilng from source or from rpm? | 16:49 |
| rhallisey | cool, that could work | 16:50 |
| rhallisey | better than stuffing build.cong | 16:50 |
| rhallisey | we could use a few includes | 16:51 |
| rhallisey | should do the trick | 16:51 |
| *** harlowja has joined #openstack-kolla | 16:51 | |
| *** egonzalez90 has joined #openstack-kolla | 16:53 | |
| *** mbound has quit IRC | 16:54 | |
| *** egonzalez has quit IRC | 16:54 | |
| *** egonzalez has joined #openstack-kolla | 16:54 | |
| *** egonzalez90 has quit IRC | 16:54 | |
| *** dwalsh has joined #openstack-kolla | 16:57 | |
| *** egonzalez has quit IRC | 17:00 | |
| sdake_ | rhallisey i was thinking stuffing build.conf initially ;) | 17:00 |
| rhallisey | I think there were comments of fear of that getting out of had | 17:00 |
| rhallisey | hand | 17:00 |
| sdake_ | from pbourke? | 17:00 |
| sdake_ | if it did get out of hand, it would be in the individual operators environment | 17:01 |
| sdake_ | we would't actuallly ship a config like that | 17:01 |
| sdake_ | we would ship a config that builds everythign in the top level base container | 17:01 |
| *** ccesario___ has quit IRC | 17:03 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Adds Neutron LBaaS Support https://review.openstack.org/318322 | 17:03 |
| sean-k-mooney | sdake: sorry was talking to someone at my desk. i was building from source | 17:04 |
| rhallisey | sdake_, what we could do is have a section in each service dockerfile that copies in the .repo | 17:05 |
| rhallisey | we put an if around it that say if custom_mariadb_repo_enabled then copy it in | 17:05 |
| rhallisey | docker expects the file to be in the same directory as the Dockerfile for that container | 17:05 |
| rhallisey | so it will grab it there | 17:06 |
| rhallisey | mariadb.repo | 17:06 |
| rhallisey | <service.repo> | 17:06 |
| sean-k-mooney | rhallisey: rather then an if per serivce would it be better to use a dictionay and check if there is an entry for this service copy in the file | 17:07 |
| sean-k-mooney | so custom_repo={ 'mariadb':'mariadb.repo' } | 17:09 |
| rhallisey | sean-k-mooney, the if per service would live in each service's Dockerfile | 17:09 |
| rhallisey | sean-k-mooney, you could do that | 17:09 |
| rhallisey | it would probably be done in the base container then | 17:09 |
| rhallisey | you could also just check if there are any .repo files in the same directory | 17:10 |
| rhallisey | and add all of them | 17:10 |
| sean-k-mooney | you could have somting like this custom_repo=[{ 'mariadb':{ 'repofile':'mariadb.repo', 'keyfile':'key' }] } | 17:12 |
| rhallisey | I think that's similar to what I have now.. let me see | 17:12 |
| sean-k-mooney | so that you you can specify a list of the custom repos for each service and the corresponding key | 17:12 |
| rhallisey | https://review.openstack.org/#/c/314614/4/kolla/common/config.py | 17:13 |
| patchbot | rhallisey: patch 314614 - kolla - Allow the base container to be more customizable | 17:13 |
| sean-k-mooney | actully that should be custom_repo={ 'mariadb-base':[{ 'repofile':'mariadb.repo', 'keyfile':'key' }] } | 17:13 |
| sean-k-mooney | cool ill take a look now | 17:14 |
| inc0 | so, rhallisey an idea | 17:15 |
| rhallisey | hit me | 17:15 |
| inc0 | we make mariadb_bootstrap pod | 17:15 |
| inc0 | replication controller for this with replicas = 0 | 17:15 |
| inc0 | then when we want to bootstrap stuff | 17:16 |
| inc0 | we change it to 1 | 17:16 |
| inc0 | and bootstrap script inside container at the end will change it back to 0 | 17:16 |
| inc0 | then if we want to re-run bootstrap, we just scale it back to 1 | 17:16 |
| rhallisey | ok so that mean kube dynamically monitors etcd variable BOOTSTRAP | 17:16 |
| sdake_ | rhallisey that doesn't work - because I am thinking about a case of nova wantss liberty repos, heat wants mitaka repos | 17:16 |
| rhallisey | and flips it | 17:16 |
| inc0 | no, no etcd involved, instead of variable in etcd we'll use number of replicas | 17:17 |
| inc0 | I mean we can add etcd at the end so actual mariadb container will fail if bootstrapped=false | 17:17 |
| inc0 | so total deployment will look like that | 17:18 |
| rhallisey | inc0, the way we bootstrap mariadb at least as AIO it wouldn't work | 17:18 |
| rhallisey | because maraidb processes would kill eachother | 17:18 |
| inc0 | hold on | 17:18 |
| inc0 | total bootstrap logic would look like that | 17:18 |
| inc0 | we have 2 pods - mariadb and mariadb_bootstrap | 17:18 |
| inc0 | mariadb container will have additional logic in start script -> | 17:19 |
| rhallisey | sdake_, to which solution. The enable service.repo one or the giant list in the base container? | 17:19 |
| inc0 | if variable MARIADB_BOOTSTRAPPED in etcd == false => exit 0 | 17:19 |
| sdake_ | both | 17:19 |
| inc0 | so it will be in restarting loop | 17:19 |
| rhallisey | inc0, ok so we change the way we bootstrap in kolla | 17:19 |
| rhallisey | slightly | 17:19 |
| inc0 | yeah we have to | 17:19 |
| inc0 | I think | 17:20 |
| rhallisey | ya we do | 17:20 |
| rhallisey | either way we would | 17:20 |
| *** unicell has joined #openstack-kolla | 17:20 | |
| *** unicell has quit IRC | 17:20 | |
| sdake_ | i wish all the bootstrapping tasks happened first... | 17:20 |
| *** unicell has joined #openstack-kolla | 17:20 | |
| inc0 | so bootstrap_maria pod will have replication controller with replicas: 0 | 17:20 |
| rhallisey | sdake_, we're throwing all the eggs at the wall here :) | 17:20 |
| inc0 | and when we want to start deploying mariadb, we change it to 1 | 17:20 |
| sdake_ | when you throw eggs at walls they stick ;) | 17:21 |
| inc0 | then inside this container we have logic that after successful bootstrap change it back to 0 and add MARIADB_BOOTSTRAPPED=true | 17:21 |
| rhallisey | well the expression is we'll see who doesn't crack | 17:21 |
| rhallisey | :) | 17:21 |
| rhallisey | but they do stick | 17:21 |
| rhallisey | hehe | 17:21 |
| inc0 | then original mariadb gets a green light and have it all nice and bootstrapped | 17:21 |
| inc0 | how does that sound to you guys | 17:22 |
| inc0 | ? | 17:22 |
| rhallisey | inc0, how would we signal to change it back to 0 | 17:22 |
| *** dwalsh has quit IRC | 17:22 | |
| rhallisey | etcd? | 17:22 |
| *** mark-casey has joined #openstack-kolla | 17:22 | |
| inc0 | rhallisey, kubectl | 17:22 |
| *** ayoung has quit IRC | 17:22 | |
| *** jtriley has joined #openstack-kolla | 17:22 | |
| inc0 | or however API works for k8s | 17:23 |
| rhallisey | how would you automate that? | 17:23 |
| rhallisey | ansible task? | 17:23 |
| inc0 | no | 17:23 |
| *** ayoung has joined #openstack-kolla | 17:23 | |
| sbezverk | inc0 we are not using replication controller at this point | 17:23 |
| inc0 | so our bootstrap inside container is a shell script right? | 17:23 |
| inc0 | sbezverk, at this point;) | 17:23 |
| rhallisey | are you saying this gets done within the container? | 17:24 |
| inc0 | we can add additional curl at the end | 17:24 |
| inc0 | yeah | 17:24 |
| rhallisey | eh | 17:24 |
| sdake_ | curl to what | 17:24 |
| sbezverk | how about cur info from etcd | 17:24 |
| inc0 | container will manage k8s | 17:24 |
| sdake_ | curl isn't viable to etcd, but etcdtcl is | 17:24 |
| rhallisey | rainbows and unicorns? | 17:24 |
| sbezverk | sdake_ curl -fs -X PUT "http://${host}:${port}/v2/keys/_test" | 17:24 |
| rhallisey | inc0, :D I had to say it | 17:25 |
| inc0 | :P | 17:25 |
| sbezverk | we can store bootstrap completion in etcd per service | 17:25 |
| rhallisey | I"m jk | 17:25 |
| inc0 | sbezverk, yeah that's the plan | 17:25 |
| *** athomas has quit IRC | 17:25 | |
| sdake_ | sbezverk how does that work with tls? | 17:25 |
| inc0 | and also manage it running/not running by replication controllers | 17:25 |
| inc0 | sdake_, as well as k8s/etcd works with tls | 17:26 |
| inc0 | which I hope it does | 17:26 |
| sbezverk | sdake_ since it is initiated from within container I do not think tls is requiored | 17:26 |
| inc0 | well it will be all private network | 17:26 |
| inc0 | and we don't use tls on management anyway | 17:26 |
| sdake_ | there is only one endpoint for etcd, it can be tls enabled or disabled | 17:26 |
| sbezverk | inc0 exactly only k8s cluster components have visibility for this traffic | 17:27 |
| inc0 | well, we can hide it behind haproxy | 17:27 |
| inc0 | and then have 2 | 17:27 |
| inc0 | endpoints | 17:27 |
| sdake_ | win | 17:27 |
| inc0 | what do you guys think about it? | 17:28 |
| inc0 | we need to have discussion how do we do startup scripts depending on deployment method | 17:28 |
| inc0 | we had this in set configs, but that's more | 17:29 |
| rhallisey | the container managing kubernetes thing might erk some the wrong way | 17:29 |
| inc0 | rhallisey, container managing k8s will be mariadb_bootstrap container itself | 17:29 |
| rhallisey | it's really a container managing itself | 17:29 |
| inc0 | it will literally commit suicide | 17:29 |
| inc0 | yup | 17:29 |
| rhallisey | smart containers? | 17:29 |
| rhallisey | can we coin that term | 17:30 |
| inc0 | not that smart, just saying "I'm done, kill me now" | 17:30 |
| rhallisey | self-managed containers | 17:30 |
| openstackgerrit | Steven Dake proposed openstack/kolla: change keyfile to key https://review.openstack.org/320589 | 17:30 |
| *** jtriley has quit IRC | 17:30 | |
| inc0 | and really this logic can be generic, any bootstrap will work with this | 17:31 |
| openstackgerrit | Steven Dake proposed openstack/kolla: change keyfile to key https://review.openstack.org/320589 | 17:31 |
| rhallisey | inc0, I think it's good dude. This give us native bootstrapping and upgrades | 17:31 |
| *** athomas has joined #openstack-kolla | 17:31 | |
| inc0 | and upgrades will either change no of replicas to 1 for rolling-upgradable services | 17:31 |
| rhallisey | it also gives a sense of control | 17:31 |
| inc0 | or change BOOTSTRAP_KEYSTONE=false->keystone replucas to 0->keystone_bootstrap_replicas=1->keystone_replicas=2 | 17:32 |
| inc0 | really in one script | 17:32 |
| inc0 | it will stand up on it's own after bootstrap is done | 17:32 |
| inc0 | for these non-rolling upgradable | 17:33 |
| rhallisey | I think it's idempotent too | 17:33 |
| inc0 | it is | 17:33 |
| rhallisey | say docker dies between setting those vars | 17:33 |
| rhallisey | replication controller is still 1 | 17:33 |
| inc0 | but we need to figure out how to tell containers that they're managed by k8s now | 17:33 |
| rhallisey | so kube will run it again and fix it | 17:33 |
| inc0 | and run this k8s logic instead of what we have with ansible | 17:33 |
| inc0 | yup | 17:33 |
| sbezverk | rhallisey replica is per pod, so even if you have 1 replica all containers in the pod will start, I do nto see how playing with replica number helps here? | 17:34 |
| inc0 | the only time it fails and kinda breaks stuff if it fails between change of no of replicas and setting up BOOTSTRAPPED var | 17:34 |
| rhallisey | sbezverk, the bootstrap pod is manageed by the replication controller | 17:35 |
| inc0 | which is super improbable and if we change BOOTSTRAPPEd before changing of replicas, at worst we;ll run bootstrap twice | 17:35 |
| rhallisey | there is a mariadb_bootstrap pod and a mariadb pod | 17:35 |
| rhallisey | sbezverk, 2 pods | 17:35 |
| inc0 | which shouldnt be a problem as it's idemptent on its own | 17:35 |
| sbezverk | ah, I see it is new :-) | 17:35 |
| sbezverk | how abount shared volumes between pods? | 17:35 |
| *** ayoung has quit IRC | 17:35 | |
| inc0 | hmm, good point | 17:35 |
| inc0 | I hope we can do this | 17:36 |
| *** ayoung has joined #openstack-kolla | 17:36 | |
| sbezverk | when we had everything in one pod we could share configmaps, volumes etc | 17:36 |
| rhallisey | sbezverk, what would need to be shared? | 17:36 |
| inc0 | rhallisey, mariadb data files for one | 17:36 |
| rhallisey | we could use labels | 17:36 |
| rhallisey | assign them to the same hosts | 17:36 |
| rhallisey | as a group | 17:36 |
| inc0 | for openstack services it's not that bad as bootstrap works with maria | 17:36 |
| sbezverk | rhallisey I am just saying, right now we have bunch of shared staff between bootstrap and actual service contianer | 17:36 |
| inc0 | sbezverk, not that much really | 17:37 |
| sbezverk | if it is not needed even better | 17:37 |
| inc0 | for openstack services | 17:37 |
| inc0 | just same database | 17:37 |
| inc0 | but it's not about volumes | 17:37 |
| rhallisey | let's group the pods by labels | 17:37 |
| rhallisey | so woudl mariadb float? | 17:37 |
| rhallisey | or would it be anchored? | 17:37 |
| inc0 | it has to float | 17:38 |
| rhallisey | ok so floating services require outside shared storage | 17:38 |
| inc0 | otherwise we need more than 1 replica of it and we don't want to have galera cluster with k8s underneath | 17:38 |
| inc0 | yeah, true, ceph works well | 17:38 |
| rhallisey | unless their data is copies amoung all nodes | 17:38 |
| rhallisey | for floating multinode we need shared storage | 17:39 |
| rhallisey | we could also do anchored multinode | 17:39 |
| rhallisey | idk something to think about | 17:39 |
| rhallisey | could support both | 17:39 |
| rhallisey | with clever templating | 17:40 |
| inc0 | well anchored multinode is ansible | 17:40 |
| inc0 | we already have it | 17:40 |
| rhallisey | that's true | 17:40 |
| inc0 | no value for k8s whatsoever then | 17:40 |
| rhallisey | additional HA | 17:40 |
| inc0 | well not really | 17:40 |
| mark-casey | anchored multinode? | 17:40 |
| inc0 | so HA model for k8s is having one mariadb container | 17:41 |
| inc0 | and if node dies k8s restarts it somewhere else | 17:41 |
| inc0 | if you have anchored, it can't restart it anywhere else | 17:41 |
| inc0 | so you need more than 1, so 3 nodes | 17:41 |
| inc0 | and galera cluster | 17:41 |
| inc0 | which is what we do with ansible and it's painful | 17:41 |
| inc0 | k8s takes away this pain, but it needs to float | 17:42 |
| rhallisey | well you could have 10 nodes labeled controller | 17:42 |
| rhallisey | it could anchor to any of those 10 | 17:42 |
| inc0 | then it floats between these 10 | 17:42 |
| rhallisey | vs any in the cluster | 17:42 |
| rhallisey | ya that's anchored | 17:42 |
| *** Jeffrey4l has quit IRC | 17:43 | |
| rhallisey | vs literlly go anywhere | 17:43 |
| inc0 | that's a design decision really | 17:43 |
| inc0 | we can run k8s just on controller nodes | 17:43 |
| rhallisey | ok well I guess you can't really call that anchored because it does float | 17:43 |
| rhallisey | selective floating | 17:43 |
| sdake_ | kolla craters on this line TASK: [keystone | Creating admin project, user, role, service, and endpoint] *** | 17:43 |
| sdake_ | master | 17:43 |
| mark-casey | it could absolutely restart anywhere in the cluster. that is the whole point of state transfer in galera. the data dir gets wiped out | 17:43 |
| sdake_ | source centos | 17:43 |
| sdake_ | anyone seen that? | 17:44 |
| rhallisey | sdake_, nope | 17:44 |
| rhallisey | could be a 2.0 issue | 17:44 |
| sdake_ | i dont have ansible 2.0 | 17:44 |
| rhallisey | upgrade to it? I thought <2.0 doesn't work anymore ? | 17:44 |
| rhallisey | could be wrong | 17:44 |
| sdake_ | you are wrong | 17:44 |
| rhallisey | idk I haven't tried it | 17:45 |
| sdake_ | jeffrey4l's patch hasnt hit the repo yet | 17:45 |
| rhallisey | oh I see | 17:45 |
| inc0 | rhallisey, yeah, I +2+A the change but got race condition in getty and change still sits;) | 17:45 |
| rhallisey | :( | 17:45 |
| dmsimard | sdake_: Hi o/ | 17:47 |
| sdake_ | sup dmsimard | 17:47 |
| rhallisey | inc0, so 2 pods then? | 17:48 |
| *** ccesario___ has joined #openstack-kolla | 17:48 | |
| inc0 | worth to try | 17:48 |
| dmsimard | sdake_: Last weekend I posted about a project I eventually want to send to /openstack, RDO (and by extension, OpenStack) will be the primary driver behind it, I thought you might be interested: https://dmsimard.com/2016/05/21/ara-an-idea-to-store-browse-and-troubleshoot-ansible-playbook-runs/ | 17:49 |
| rhallisey | Ok. I"ll do nova while you do mariadb | 17:49 |
| dmsimard | I posted about it on openstack-infra as well: http://lists.openstack.org/pipermail/openstack-infra/2016-May/004257.html | 17:49 |
| inc0 | rhallisey, let's start with 1 and check out if it even make sense | 17:49 |
| inc0 | we need to figure out how to change kolla_start.sh depeding on deployment method | 17:49 |
| *** esimone has quit IRC | 17:49 | |
| inc0 | we need to change containers with this approach unfortunately | 17:50 |
| rhallisey | inc0, how can we split up the task? | 17:50 |
| inc0 | changing the contianer will be separate review | 17:50 |
| inc0 | but we need to discuss how do we do it | 17:50 |
| rhallisey | hangout then? | 17:50 |
| inc0 | hold on | 17:51 |
| rhallisey | dmsimard, you could add a doc to kolla with that | 17:51 |
| mark-casey | inc0: rhallisey: was just trying to read back chat, what problem was there with changing bootstrap tasks so that you could run it 9 times in a row and not hurt anything? sort of like ansible's 'creates=' flag on the shell module | 17:52 |
| rhallisey | dmsimard, it's a pretty visualization of playbooks | 17:52 |
| rhallisey | mark-casey, I'm sure I understand what you mean. The issue arose from being able to run a bootstrap job and signal when it's done | 17:54 |
| *** dcwangmit01 has joined #openstack-kolla | 17:54 | |
| *** mbound has joined #openstack-kolla | 17:54 | |
| rhallisey | mark-casey, inc0 has a clever way to do it by manipulating the # of replications of a pod | 17:55 |
| *** Jeffrey4l has joined #openstack-kolla | 17:55 | |
| dmsimard | rhallisey: *nods*, there is some polishing and improvements to do first .. and I also want the project to be under the openstack namespace before poking (too much) parties that may be interested | 17:56 |
| mark-casey | rhallisey: oh. ok. that just clicked a little on how that could work. I thought it was two different pods coordinating. Still I wonder whether it would be easier/better to make every single task in a bootstrap idempotent. (i.e.: "stat file_created_by_db_bootstrap >/dev/null || bootstrap_db" ) | 17:56 |
| rhallisey | dmsimard, neat! | 17:57 |
| dmsimard | rhallisey: I could even do an ara web ui container :p | 17:57 |
| rhallisey | hehe :) | 17:57 |
| rhallisey | mark-casey, It is two different pods. One pod waits_for_bootstrap_done and the other run the bootstrap | 17:58 |
| * rhallisey needs to add this to the ML thread & the spec | 17:58 | |
| rhallisey | if it works :) | 17:59 |
| *** mbound has quit IRC | 17:59 | |
| inc0 | mark-casey, we have better places to put this kind of lock | 18:05 |
| inc0 | namely ETCD | 18:05 |
| inc0 | which we want to use in my idea;) | 18:05 |
| *** mgoddard_ has quit IRC | 18:06 | |
| *** daneyon_ has joined #openstack-kolla | 18:06 | |
| *** athomas has quit IRC | 18:07 | |
| *** mbound has joined #openstack-kolla | 18:08 | |
| *** daneyon_ has quit IRC | 18:10 | |
| sdake_ | need help with this bug | 18:11 |
| sdake_ | http://paste.fedoraproject.org/370466/46411345/ -> http://paste.fedoraproject.org/370466/46411345 | 18:11 |
| *** sdake_ is now known as sdake | 18:11 | |
| sdake | any ideas? | 18:11 |
| rhallisey | wonder what file it's failing to decode | 18:12 |
| *** wznoinsk_ has joined #openstack-kolla | 18:14 | |
| sdake | note this is with the 2.0.0 tag | 18:15 |
| *** jrist has quit IRC | 18:15 | |
| *** jrist has joined #openstack-kolla | 18:16 | |
| *** wznoinsk has quit IRC | 18:16 | |
| sean-k-mooney | inc0: just reading back through the last 2 hours of chat. | 18:16 |
| inc0 | sean-k-mooney, it's about kubernetes | 18:17 |
| *** dwalsh has joined #openstack-kolla | 18:17 | |
| sean-k-mooney | inc0: so in your case would we create teh inital mariadb pod with a replica size of 0 and only increase it to 1 after the maria_bootstrap pod had finished | 18:17 |
| inc0 | no | 18:18 |
| inc0 | mariadb pod will always be 1 | 18:18 |
| sean-k-mooney | sorry yes because its hosting the db your are bootstraping | 18:18 |
| inc0 | but mariadb container will exit 0 if MARIADB_BOOTSTRAPPED==false | 18:18 |
| inc0 | no, bootstrapping maria itself is different | 18:19 |
| inc0 | (really not sure if we need to, but in case we need, let's asume we bootstrap it) | 18:19 |
| inc0 | mariadb_bootsrap pod will have logic to bootstrap maria itself - like create files and such | 18:19 |
| inc0 | and when it's done it will change MARIADB_BOOTSTRAPPED to True | 18:20 |
| inc0 | will change mariadb_bootstrap replicas to 0 | 18:20 |
| inc0 | (kill itself) | 18:20 |
| inc0 | and exit 0 | 18:20 |
| sean-k-mooney | oh ok so in this context what was the mariadb_bootstrap pode actully bootstraping. that was what was confusing me | 18:20 |
| sdake | File "/usr/local/bin/kolla_keystone_bootstrap", line 21, in <module>\n from keystoneclient import exceptions\nImportError: No module named keystoneclient', 'rc': 1, 'invocation': {'module_name': u'command', 'module_complex_args' | 18:20 |
| sean-k-mooney | inc0: yep that makes sense | 18:20 |
| *** salv-orlando has quit IRC | 18:21 | |
| inc0 | same logic can be replicated for any other service | 18:21 |
| inc0 | then if you want to upgrade, just put bootstrap replicas back to 1 | 18:21 |
| sean-k-mooney | inc0: would you put the replicas back to 1 or create a new pod with the new image and put its replica set to 1 | 18:22 |
| *** fragatina has joined #openstack-kolla | 18:22 | |
| inc0 | you don't need to create new pod to change images | 18:23 |
| inc0 | you just tag new image with old tag | 18:23 |
| inc0 | I mean mariadb_bootstrap:latest | 18:23 |
| *** thumpba has joined #openstack-kolla | 18:24 | |
| rhallisey | inc0, do you think we should track the recpla count in etcd? | 18:24 |
| sean-k-mooney | hum ok but if i was upgrading nova for example i may want to have different tag e.g. mitaka and newton rather then replace my existing tags | 18:25 |
| inc0 | nah, it's alredy being tracked there | 18:25 |
| inc0 | by k8s itself | 18:25 |
| inc0 | in this case yeah, you need new pod | 18:25 |
| rhallisey | inc0, ya that's better, could get some crazy corner cases if we did that | 18:25 |
| inc0 | yeah, one source of truth ftw | 18:25 |
| mark-casey | inc0: I just liked the idea that if the bootstrap process itself was repeatable without harm, it would work the same in ansible and k8s and not have to touch k8s' etcd, which i'd seen advised against | 18:26 |
| *** jtriley has joined #openstack-kolla | 18:26 | |
| *** ccesario___ has quit IRC | 18:27 | |
| *** ccesario___ has joined #openstack-kolla | 18:27 | |
| sean-k-mooney | mark-casey: well other then time is there any reason not to make the bootstrap logic idempotent as well | 18:29 |
| *** d_code has quit IRC | 18:30 | |
| sdake | http://paste.fedoraproject.org/370474/14637146/ -> http://paste.fedoraproject.org/370474/14637146 | 18:30 |
| sdake | i rebuilt 2.0.0 | 18:30 |
| sdake | and get this result | 18:30 |
| sdake | so essentially - every version of kolla is broken | 18:31 |
| *** d_code has joined #openstack-kolla | 18:31 | |
| *** thumpba has quit IRC | 18:31 | |
| inc0 | mark-casey, it should be idempotent | 18:32 |
| inc0 | anyway | 18:32 |
| inc0 | but you don't want mariadb to run while it's being bootstrapped | 18:32 |
| mark-casey | sean-k-mooney: what he said :D | 18:32 |
| sean-k-mooney | :) | 18:33 |
| mark-casey | inc0: that makes sense. that is a different issue :) | 18:33 |
| inc0 | that's the one we're trying to solve;) | 18:33 |
| *** jtriley has quit IRC | 18:33 | |
| sean-k-mooney | sdake is this an issue with the json decoder or what it is parsing | 18:34 |
| sdake | yo can we have a timeout from design to focus on the fact that all of kolla in all branches are broken | 18:34 |
| *** dwalsh has quit IRC | 18:34 | |
| sdake | sean-k-mooney it looks like keystone client is busted in some way | 18:34 |
| inc0 | hmm | 18:34 |
| inc0 | it was working yesterday | 18:34 |
| inc0 | hold on, deploying | 18:34 |
| sean-k-mooney | when you say keystone client to you mean the client liberary or commandline client | 18:35 |
| *** mbound has quit IRC | 18:35 | |
| sean-k-mooney | you posted another error message a few minutes ago related to keystone correct | 18:36 |
| sdake | the version i tested and tagged with 2.0.0 from source has the same version of python-keystoneclient | 18:37 |
| sdake | can someone else test this and make sure its not environmental? | 18:37 |
| *** ccesario___ has quit IRC | 18:37 | |
| inc0 | sdake, building containers now | 18:38 |
| *** jtriley has joined #openstack-kolla | 18:38 | |
| sean-k-mooney | sdake was the 'No module named keystoneclient' error you hit on File "/usr/local/bin/kolla_keystone_bootstrap", line 21, also related to this build | 18:41 |
| sdake | yup | 18:41 |
| sdake | oh no that was a build from master | 18:41 |
| sean-k-mooney | oh ok | 18:41 |
| d_code | so, I’ve been beating my head against OpenStack for a couple days now relating to PCI passthrough… dropping the relevant config into /etc/kolla/config/nova.conf seems to work appropriately, the compute nodes register the pci device, but the scheduler filter removes all servers when it comes time to comparing the pci capabilities against the alias of the flavor. Has anyone done any of this even outside of Kolla? | 18:43 |
| *** Avi82 has joined #openstack-kolla | 18:44 | |
| sdake | d_code i'm sure someone hs done it, but probably not in kolla land | 18:44 |
| sdake | d_code most folks are doing basic stuff with kolla at this point (mostly evaluations going on) | 18:44 |
| vhosakot | d_code: yes, I have successfully used and tested SR-IOV (PCI passthru) with kolla internally in my company | 18:44 |
| inc0 | d_code, sounds like nova issue, I can hardly imagine how kolla can affect that if cnodes sees pci devices | 18:44 |
| d_code | SamYaple mentioned that he had done it a few weeks ago…but he doesn’t seem to be around | 18:44 |
| *** ccesario___ has joined #openstack-kolla | 18:45 | |
| d_code | inc0: I don’t think it’s a kolla issue, specifically. I just happen to be using it and #openstack is quite idle | 18:45 |
| vhosakot | d_code: neutron needs the hardware plugin for that card on which SR-IOV neutorn ports are created | 18:45 |
| sean-k-mooney | d_code: i have done sow work with pci passthrough but its been a while | 18:45 |
| inc0 | d_code, try #openstack-nova | 18:45 |
| d_code | so…I want to pass through the PF though, not using SR-IOV | 18:45 |
| vhosakot | ah, SR-IOV needs the PF to get outside the server | 18:46 |
| d_code | vhosakot: http://paste.fedoraproject.org/370480/14641156 | 18:47 |
| d_code | that’s /etc/kolla/config/nova.conf | 18:47 |
| sean-k-mooney | d_code: in addtion the the pci_whitelist and adding the alais request in the flavor have you also enabled the pci passthrough filter | 18:47 |
| sean-k-mooney | thats a yes :) | 18:48 |
| sean-k-mooney | so your are trying to pass throuhg a niantic | 18:48 |
| sean-k-mooney | what version of openstack? | 18:48 |
| d_code | mitaka | 18:49 |
| d_code | I want to pass through an entire nic | 18:49 |
| d_code | or…one of the two ports, whatever it does…need the physical function | 18:49 |
| d_code | then, `openstack flavor show m1.sensor` returns http://paste.fedoraproject.org/370481/14641157 | 18:49 |
| *** ccesario___ has quit IRC | 18:50 | |
| d_code | I feel like the alias isn’t getting through | 18:50 |
| *** ccesario___ has joined #openstack-kolla | 18:50 | |
| sean-k-mooney | do you have a pasted of the nova scheduler log with the error | 18:50 |
| d_code | b/c nova-compute on each of my 3 nodes has 2 pci devices that match | 18:50 |
| d_code | sure...sec | 18:50 |
| sean-k-mooney | im wondering what filter actully eliminated the nodes. | 18:50 |
| d_code | pcipassthrough | 18:51 |
| d_code | http://paste.fedoraproject.org/370482/14641159 | 18:52 |
| sean-k-mooney | i assume if you check the pci_devices table in the nova db they are listed and available? also in mitaka the sriov nic agent needs to be deployed on the compute node | 18:52 |
| d_code | is there a way to increase verbosity of logging? | 18:53 |
| d_code | sean-k-mooney: I haven’t done anything with a nic agent | 18:53 |
| sean-k-mooney | kolla does not run nova in debug mode by default | 18:53 |
| d_code | they are in the nova pci_devices table | 18:53 |
| d_code | neutron_metadata_agent, neutron_l3_agent, neutron_dhcp_agent, neutron_openvswitch_agent <— those are the containers running on my compute host …well..it’s an all in one | 18:55 |
| d_code | two others that are compute only | 18:55 |
| sean-k-mooney | so prior to mitaka intel nics did not use or require the sriov nic agent but as of mitaka it is required by nova. if its not deployed it may casue the failure but without an error message to back that up its just a guess | 18:55 |
| vhosakot | d_code: the following in nova.conf works fine for me for SR-IOV --> http://paste.openstack.org/show/498736/ | 18:56 |
| inc0 | sdake, ubuntu source worked | 18:56 |
| d_code | vhosakot: but you hvae some sort of vlan in neutron config tying to VF’s, right? | 18:57 |
| sdake | inc0 that doesn't mean centos isn't broken | 18:57 |
| inc0 | well, I'm just saying ubuntu isn't broken, someone should test centos | 18:58 |
| sean-k-mooney | d_code: vhosakot both of your nova config look correct | 18:58 |
| d_code | sean-k-mooney: to increase logging of scheduler, do I need to rebuild the container? | 18:58 |
| sean-k-mooney | d_code: vhosakot also has the "physical_network":"phys_prov" set so that it will work with neutron too | 18:58 |
| sean-k-mooney | d_code: no you just need to change the value in the nova.conf and restart the container if you deployed with copy_always | 18:59 |
| *** ravig has joined #openstack-kolla | 18:59 | |
| vhosakot | yes, the neutron must know the VLAN | 18:59 |
| vhosakot | yes, "physical_network":"phys_prov" is needed for provider networks | 18:59 |
| d_code | vhosakot: k | 19:00 |
| d_code | sean-k-mooney: I’ll give that a whirl | 19:01 |
| sean-k-mooney | sdake: if i get a chance tonight i will install a centos node and try and build/deploy 2.0.0 and see if it works for me just as an other sanatiy check | 19:02 |
| vhosakot | d_code: cool | 19:02 |
| sean-k-mooney | sdake: was that a binary or source build that failed? | 19:02 |
| sdake | it was source but i just rebuilt everything and it seems to deploy | 19:11 |
| sdake | notsure which branch i'm on | 19:11 |
| sdake | waiting for deploy to finish | 19:11 |
| d_code | claims “doesn't have the required PCI devices “ for all compute hosts | 19:12 |
| d_code | that would be a reason….. except for that it does have them :/ | 19:12 |
| *** ravig has quit IRC | 19:12 | |
| sean-k-mooney | ok cool im just going to head home catch people later. | 19:13 |
| d_code | later sean-k-mooney | 19:17 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Render the common role in the CLI https://review.openstack.org/320097 | 19:18 |
| *** sdake has quit IRC | 19:21 | |
| *** sdake has joined #openstack-kolla | 19:21 | |
| *** jtriley has quit IRC | 19:27 | |
| *** sdake has quit IRC | 19:30 | |
| *** sdake has joined #openstack-kolla | 19:31 | |
| *** ravig has joined #openstack-kolla | 19:31 | |
| sdake | sbezverk can you ink again plz | 19:31 |
| *** rmart04 has joined #openstack-kolla | 19:32 | |
| *** diogogmt has quit IRC | 19:32 | |
| *** ravig has quit IRC | 19:32 | |
| *** gfidente has quit IRC | 19:34 | |
| *** Jeffrey4l has quit IRC | 19:34 | |
| *** ravig has joined #openstack-kolla | 19:34 | |
| sdake | is there a way to specify a regex of just "base" to build.py? | 19:36 |
| inc0 | build.py base | 19:36 |
| sdake | so only base is built not everything with base in it? | 19:36 |
| inc0 | should | 19:37 |
| sdake | should what? | 19:38 |
| inc0 | should build just base images | 19:38 |
| inc0 | and it's deps | 19:38 |
| sdake | i want to build "base" | 19:38 |
| sdake | and that is it | 19:38 |
| sdake | nothign else | 19:38 |
| inc0 | just use this command | 19:38 |
| *** rmart04 has quit IRC | 19:40 | |
| *** rmart04 has joined #openstack-kolla | 19:41 | |
| *** salv-orlando has joined #openstack-kolla | 19:42 | |
| *** rmart04 has quit IRC | 19:43 | |
| *** mliima has quit IRC | 19:44 | |
| *** Jeffrey4l has joined #openstack-kolla | 19:46 | |
| sdake | inc0 which command | 19:46 |
| inc0 | build.py base | 19:48 |
| sdake | inc0 that builds all base images | 19:52 |
| sdake | I onlly want to build the one base image | 19:52 |
| inc0 | which one? | 19:52 |
| sdake | lets take the example differently | 19:52 |
| inc0 | it will build image and all it's dependencies | 19:52 |
| sdake | no it builds all base | 19:52 |
| sdake | try it for yourself | 19:52 |
| sdake | it will buid lnova-base for exxample | 19:53 |
| inc0 | yes | 19:53 |
| inc0 | it will | 19:53 |
| inc0 | it will build everything with base in it | 19:53 |
| sdake | yes its \*base\* | 19:53 |
| sdake | that is the regex | 19:53 |
| inc0 | yup | 19:53 |
| sdake | base as a regex = "base" | 19:53 |
| *** dmk0202 has joined #openstack-kolla | 19:54 | |
| sdake | sbezverk ping - master works for me | 19:55 |
| sdake | i'm not sure what happened with my last build | 19:55 |
| sdake | sbezverk lets get to the bottom of your problem with nova | 19:55 |
| sdake | sbezverk do you have time for webex to discuss? | 19:55 |
| sdake | or alternatively - can you tell me which version of python-novaclient is on your system | 19:56 |
| sdake | i have 3.4.0 | 19:56 |
| *** ccesario___ has quit IRC | 20:00 | |
| *** ccesario___ has joined #openstack-kolla | 20:00 | |
| *** Avi82 has quit IRC | 20:01 | |
| *** ravig has quit IRC | 20:01 | |
| *** sdake_ has joined #openstack-kolla | 20:02 | |
| *** ravig has joined #openstack-kolla | 20:04 | |
| *** sdake has quit IRC | 20:05 | |
| *** shardy has quit IRC | 20:06 | |
| sbezverk | sdake I had 3.3.0 and it did not work, after I updated it to 3.4.0 it sarted working. | 20:13 |
| *** jtriley has joined #openstack-kolla | 20:21 | |
| *** ayoung has quit IRC | 20:21 | |
| sdake_ | nice! | 20:24 |
| sdake_ | ya the dependencies and versions are a mess, no? :) | 20:24 |
| *** jtriley has quit IRC | 20:32 | |
| sbezverk | inc0 ping | 20:34 |
| sdake_ | sbezverk does the data plane thing work? | 20:35 |
| sbezverk | :-( no | 20:35 |
| sdake_ | workflow -1 your patch | 20:35 |
| sbezverk | everything looks nice and clean | 20:36 |
| sdake_ | i dont want to prematurely merge it | 20:36 |
| sdake_ | sbezverk do you need config files from devstack with lbaas setup in it? | 20:36 |
| inc0 | sbezverk, here | 20:36 |
| sbezverk | sdake_ I got the best possible source for lbaas configuration as I used my red hat contact | 20:37 |
| sbezverk | I am confident from lbaas/opensatck perspective I have everything right | 20:37 |
| sdake_ | so upstream is broken? | 20:38 |
| sdake_ | or user error? | 20:38 |
| sbezverk | sdake_ what I am not sure is kolla | 20:38 |
| sdake_ | or network config problem | 20:38 |
| sbezverk | in a sense I thought I could use haproxy container | 20:38 |
| sbezverk | lbaas required haproxy | 20:38 |
| sdake_ | lets keep haproxy that kolla uses separate from lbaass' haproxy | 20:39 |
| sdake_ | are you saying the haproxies are conflicting? | 20:39 |
| sbezverk | ok then I need to install haproxy on my server where I run kolla? | 20:39 |
| sdake_ | no, but it probably needs to be installed in the container | 20:39 |
| sbezverk | so I would have 1 haproxy in a container for kolla and one just for lbaas?? | 20:39 |
| sbezverk | I think it is more architecture related issue than just config. | 20:40 |
| sbezverk | sdake_ I have a idea | 20:42 |
| sbezverk | sdake_ haproxy running container has in /run haproxy.pid what if I create a named volume and share it between haproxy and neutron lbaas containers: | 20:43 |
| sbezverk | ? | 20:43 |
| *** rmart04 has joined #openstack-kolla | 20:45 | |
| *** rmart04 has quit IRC | 20:45 | |
| *** ravig has quit IRC | 20:47 | |
| *** ravig has joined #openstack-kolla | 20:49 | |
| sdake_ | haproxy.pid doesn't help | 20:49 |
| *** ravig has quit IRC | 20:52 | |
| *** ravig has joined #openstack-kolla | 20:53 | |
| *** ccesario___ has quit IRC | 20:54 | |
| *** ravig has quit IRC | 20:55 | |
| sdake_ | sbezverk lets not confuse our haproxy used by kolla with the one used by lbaas | 20:55 |
| sdake_ | i think that would be a recipe for disaster | 20:55 |
| *** ravig has joined #openstack-kolla | 20:55 | |
| *** mbound has joined #openstack-kolla | 20:55 | |
| *** fragatina has quit IRC | 20:57 | |
| *** ccesario___ has joined #openstack-kolla | 20:57 | |
| *** mliima has joined #openstack-kolla | 20:57 | |
| mliima | sdake_, ping | 20:57 |
| sdake_ | mliima wound me sir | 20:58 |
| *** ravig has quit IRC | 20:58 | |
| *** mbound has quit IRC | 21:00 | |
| sdake_ | mlima whats up | 21:00 |
| mliima | well, you said to me for ping you | 21:00 |
| *** wirehead_ has joined #openstack-kolla | 21:01 | |
| sdake_ | right | 21:01 |
| *** ravig has joined #openstack-kolla | 21:01 | |
| sdake_ | wound me = reference to shoot = which means speak in english :) | 21:01 |
| sdake_ | can you open up a review | 21:01 |
| *** ravig has quit IRC | 21:01 | |
| sdake_ | any one will do | 21:01 |
| sdake_ | you need to be logged in | 21:01 |
| *** ravig has joined #openstack-kolla | 21:01 | |
| sdake_ | i want to show you how gerrrit behaves now for you | 21:01 |
| mliima | I had to change the nick, someone registred mlima :( | 21:02 |
| *** wirehead_ is now known as wirehead | 21:02 | |
| sdake_ | change which nick | 21:02 |
| mliima | irc | 21:02 |
| sdake_ | bummer | 21:02 |
| *** wirehead is now known as wirehead_ | 21:03 | |
| mliima | now i have 2 i | 21:03 |
| sdake_ | ok | 21:03 |
| sdake_ | so basically the way it works | 21:03 |
| sdake_ | the first core reviewer comes along | 21:03 |
| sdake_ | these people are called "first to review" | 21:03 |
| sdake_ | it can also be non-core reviewerfs first to review | 21:03 |
| sdake_ | if your first to review, your expected to see the review through until completion | 21:03 |
| sdake_ | i sometimes vote +1 if I'm in favor of the idea, but the implementation isn't quite right but I don't have suggestions on hwo to fix it | 21:04 |
| sdake_ | a -1 vote means "fix x y z" | 21:04 |
| sdake_ | a -2 vote essentially kills a review, meaning it will never go into the code base | 21:04 |
| sdake_ | we reserve -2 votes for serious situations | 21:04 |
| sdake_ | atleasst i do | 21:05 |
| sdake_ | everyone behaves differently as you will find out ;) | 21:05 |
| sdake_ | the final vote you can case is a +2 | 21:05 |
| mliima | i understood :) | 21:05 |
| *** ayoung has joined #openstack-kolla | 21:05 | |
| sdake_ | the second person to the review has a special obligation to +w the patch | 21:05 |
| sdake_ | if they also +2 it | 21:05 |
| sdake_ | one thing I do on every review, is look at the bug id | 21:05 |
| sdake_ | if its high or critical, I target it to mitaka and liberty | 21:06 |
| *** ravig has quit IRC | 21:06 | |
| sdake_ | I am not superhuman | 21:06 |
| sdake_ | so I can't review everything, and would appreciate help in targeting bugs for backporting | 21:06 |
| sdake_ | to backport, a bug *must* have a bug id | 21:06 |
| mliima | i'm very happy and excited to be part of the team. | 21:06 |
| sdake_ | this is why trivialfix is limited use | 21:06 |
| sdake_ | so dont +2 and +w if there isn't already a +2 | 21:06 |
| sdake_ | there is no obligation to manually test something that you didn't persoanlly author | 21:07 |
| sdake_ | sometimes I pull down patches | 21:07 |
| sdake_ | sometimes I don't | 21:07 |
| sdake_ | really up to you | 21:07 |
| sdake_ | welcome to the core reviewer team for kolla, its a pleassure to have you :) | 21:07 |
| mliima | the pleassure is mine | 21:08 |
| *** ccesario___ has quit IRC | 21:08 | |
| *** ccesario___ has joined #openstack-kolla | 21:08 | |
| vhosakot | good work mliima! :) | 21:08 |
| sdake_ | look we both mistyped pleasure ;) | 21:12 |
| sdake_ | lol | 21:12 |
| mliima | many fellings sdake_ | 21:13 |
| wirehead_ | The only thing more fun than endless piles of patches to review is directly shocking the basal ganglia. | 21:14 |
| sdake_ | wirehead_ ++ | 21:14 |
| sdake_ | wirehead_ someone has to do it :) | 21:15 |
| sdake_ | the work doesn't do itself :) | 21:15 |
| *** callahanca has quit IRC | 21:16 | |
| openstackgerrit | Liyi Meng proposed openstack/kolla-kubernetes: Break deployment procedure into two steps https://review.openstack.org/320668 | 21:17 |
| *** haplo37_ has joined #openstack-kolla | 21:19 | |
| *** callahanca has joined #openstack-kolla | 21:20 | |
| *** fragatina has joined #openstack-kolla | 21:20 | |
| d_code | scheduler output: http://paste.fedoraproject.org/370482/14641159/ | 21:21 |
| d_code | bah…wrong channel | 21:21 |
| d_code | sorry | 21:21 |
| *** haplo37_ has quit IRC | 21:24 | |
| sbezverk | sdake_ well, I think it is bigger disaster to run to haproxies :-) | 21:25 |
| sdake_ | you can probably use the same haproxy container just a different copy | 21:26 |
| sdake_ | thin containers = do one job | 21:26 |
| sdake_ | haproxy container does the job of proxying connectivity from the external to the internal networks authentiating along the way | 21:26 |
| sdake_ | i want to have good separation of concerns here | 21:26 |
| sdake_ | but my question is on the data plane stuff, were there any errors in the logs? | 21:27 |
| sbezverk | sdake_ yes, agree but that job is acting as haproxy, nobody said only one application can use thin container.. | 21:27 |
| *** ccesario___ has quit IRC | 21:27 | |
| sdake_ | i think it ould be a security concern | 21:28 |
| sbezverk | sdake_ there is no errors in logs, what happened is haproxy bring up VIP for lbaas, if haproxy is not running hence VIP is not active | 21:28 |
| sdake_ | i'd just rather not even go there with it | 21:28 |
| sdake_ | is haproxy meant to run wherever lbaas is running? | 21:28 |
| sbezverk | yes in the config you can see it uses haproxy driver to communicate with it | 21:29 |
| sdake_ | is lbaas setup to run on the network nodes then? | 21:30 |
| sdake_ | note haproxy probably has some other files besides haproxy.pid shared | 21:30 |
| sdake_ | within the /run directory | 21:31 |
| sdake_ | I'd recommend making a named volume just for haproxy+lbaas | 21:31 |
| sbezverk | sdake_ I check haproxy container and I do not see anything suspicious | 21:31 |
| sdake_ | sbezverk could you expand ;) | 21:31 |
| sbezverk | ok but haproxy(kolla) or haproxy(additional container) ? | 21:32 |
| sdake_ | there has got to be a haproxy socket in /run | 21:32 |
| *** ccesario___ has joined #openstack-kolla | 21:32 | |
| sdake_ | please make a separate instance for lbaas | 21:32 |
| sbezverk | sdake_ can you access links now? | 21:32 |
| sdake_ | my main concern is security | 21:32 |
| *** dmk0202 has quit IRC | 21:32 | |
| sdake_ | i can - reboot happened 1 hr ago :) | 21:32 |
| sbezverk | sdake_ http://paste.openstack.org/show/498751/ | 21:33 |
| sdake_ | ls -l /run | 21:33 |
| sdake_ | wierd | 21:33 |
| sdake_ | i wonder how the client communicates with the server | 21:33 |
| sbezverk | sdake_ normally they do not | 21:34 |
| sbezverk | haproxy we use is manually configured | 21:34 |
| sbezverk | so it does what we told it to do | 21:34 |
| *** sdake has joined #openstack-kolla | 21:34 | |
| sbezverk | I suspect lbaas uses some sort of api for dynamic reconfiguration of haproxy | 21:34 |
| sdake | back | 21:35 |
| sdake | yes but to do that there is typically a socket | 21:35 |
| sbezverk | because one of lbaas config items is haproxy group id | 21:35 |
| sdake | /var/run/haproxy.sock | 21:36 |
| sbezverk | sdake I tried to get more details from lbaas developers, but man these guys ignores you like you do not exist :-) | 21:36 |
| sdake | http://haproxy.tech-notes.net/9-2-unix-socket-commands/ | 21:36 |
| sdake | haha | 21:36 |
| *** mliima has quit IRC | 21:36 | |
| sdake | put yourself in their shoes | 21:36 |
| sdake | the neutron devs and nova devs have people harassing them nonstop for features bugs etc | 21:37 |
| *** diogogmt has joined #openstack-kolla | 21:37 | |
| sdake | they dont need any more adoption, they need more third party support | 21:37 |
| sdake | kolla on the other hand needs adoption, that is why we spend so much time on support here ;) | 21:37 |
| *** ravig has joined #openstack-kolla | 21:37 | |
| *** dmk0202 has joined #openstack-kolla | 21:37 | |
| sbezverk | "/var/lib/kolla/haproxy/haproxy.sock" | 21:38 |
| *** sdake_ has quit IRC | 21:38 | |
| sdake | disable frontend <frontend> | 21:38 |
| sdake | we need this for upgrade | 21:39 |
| sbezverk | sdake that socket seems to be for stats: stats socket /var/lib/kolla/haproxy/haproxy.sock | 21:40 |
| sdake | you may have to read the lbaas python code to figure out how it communicates with haproxy | 21:40 |
| *** Serlex has quit IRC | 21:46 | |
| sbezverk | sdake yeah it seems it comminucates via sock I found some references in code | 21:57 |
| sdake | sounds like you need a haproxy ontainer started in the lbaas role | 21:58 |
| sbezverk | sdake it look too complicated, nested roles?? | 22:00 |
| sbezverk | sdake I need to do PoC, I will install haproxy locally, add lbaas container hostmount for haproxy internals and see if it works | 22:02 |
| *** dmk0202 has quit IRC | 22:09 | |
| *** sdake has quit IRC | 22:09 | |
| *** sdake has joined #openstack-kolla | 22:19 | |
| sdake | sbezverk soundsgood | 22:19 |
| sdake | sbezverk actually a host mount there wont work, because I assume your running lbaas on the network nodes where it belongs | 22:20 |
| sdake | and haproxy has a host mount too | 22:21 |
| sbezverk | sdake nope haproxy does not have host mount | 22:24 |
| sdake | oh that makes sene then | 22:24 |
| sbezverk | it keeps its stuff inside wihtout exposing it | 22:24 |
| *** rhallisey has quit IRC | 22:27 | |
| *** thumpba has joined #openstack-kolla | 22:31 | |
| sbezverk | sdake these guys turned out to be much smarter than I thought :-) they do not use haproxy in daemon mode | 22:47 |
| sbezverk | sdake they use it in foreground mode and start it per name space, really cool | 22:47 |
| sdake | cool | 22:48 |
| sdake | that doesn't answer the Q why the dataplane is busted | 22:48 |
| sbezverk | :-) give a bit of time I think I know what is wrong | 22:49 |
| *** stvnoyes has quit IRC | 22:55 | |
| *** stvnoyes has joined #openstack-kolla | 22:55 | |
| *** absubram has quit IRC | 22:58 | |
| *** openstackgerrit has quit IRC | 23:03 | |
| *** openstackgerrit has joined #openstack-kolla | 23:03 | |
| *** vhosakot has quit IRC | 23:10 | |
| *** ravig has quit IRC | 23:14 | |
| *** ravig has joined #openstack-kolla | 23:21 | |
| *** ravig has quit IRC | 23:22 | |
| *** 32NAA9JNR has joined #openstack-kolla | 23:22 | |
| *** thumpba has quit IRC | 23:24 | |
| *** ravig has joined #openstack-kolla | 23:27 | |
| *** ravig has quit IRC | 23:34 | |
| *** ravig has joined #openstack-kolla | 23:34 | |
| *** 32NAA9JNR has quit IRC | 23:39 | |
| *** weiyu_ has joined #openstack-kolla | 23:44 | |
| *** SiRiuS__ has quit IRC | 23:48 | |
| *** inc0 has quit IRC | 23:49 | |
| *** ccesario___ has quit IRC | 23:50 | |
| *** weiyu_ has quit IRC | 23:53 | |
| *** jtriley has joined #openstack-kolla | 23:54 | |
| *** ravig has quit IRC | 23:54 | |
| *** sdake has quit IRC | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!