| *** oikiki has quit IRC | 00:01 | |
| *** rcernin has quit IRC | 00:09 | |
| *** rcernin_ has joined #openstack-containers | 00:09 | |
| *** marst has quit IRC | 00:35 | |
| *** rcernin_ has quit IRC | 00:50 | |
| *** hieulq has quit IRC | 00:53 | |
| *** hieulq has joined #openstack-containers | 00:54 | |
| *** oikiki has joined #openstack-containers | 01:01 | |
| *** kiennt26 has joined #openstack-containers | 01:03 | |
| *** yuanying has quit IRC | 01:14 | |
| *** openstackgerrit has joined #openstack-containers | 01:22 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/magnum master: Updated from global requirements https://review.openstack.org/528406 | 01:22 |
|---|---|---|
| *** vishwanathj has joined #openstack-containers | 01:30 | |
| *** linkmark has quit IRC | 01:33 | |
| *** oikiki has quit IRC | 01:33 | |
| *** dardelean_ has joined #openstack-containers | 01:49 | |
| *** dardelean_ has quit IRC | 01:54 | |
| *** dardelean_ has joined #openstack-containers | 02:01 | |
| *** penick has quit IRC | 02:07 | |
| *** rcernin has joined #openstack-containers | 02:11 | |
| *** AlexeyAbashkin has joined #openstack-containers | 02:38 | |
| *** AlexeyAbashkin has quit IRC | 02:43 | |
| *** ramishra has joined #openstack-containers | 02:43 | |
| *** ramishra has quit IRC | 03:15 | |
| *** ramishra has joined #openstack-containers | 03:43 | |
| *** dardelean_ has quit IRC | 03:49 | |
| *** dave-mccowan has quit IRC | 03:51 | |
| *** dpawar has joined #openstack-containers | 03:52 | |
| *** flwang1 has quit IRC | 04:17 | |
| *** fragatina has quit IRC | 04:20 | |
| *** fragatina has joined #openstack-containers | 04:20 | |
| *** janonymous has joined #openstack-containers | 04:22 | |
| *** ricolin has joined #openstack-containers | 04:31 | |
| *** janki has joined #openstack-containers | 04:43 | |
| *** ykarel has joined #openstack-containers | 04:44 | |
| *** kiennt26 has quit IRC | 04:54 | |
| *** hieulq has quit IRC | 04:54 | |
| *** kiennt26 has joined #openstack-containers | 04:55 | |
| *** hieulq has joined #openstack-containers | 04:55 | |
| *** yamamoto has joined #openstack-containers | 04:57 | |
| *** chhavi has joined #openstack-containers | 05:03 | |
| *** yamamoto has quit IRC | 05:07 | |
| *** chhavi has quit IRC | 05:07 | |
| *** akhil_jain has quit IRC | 05:07 | |
| *** yamamoto has joined #openstack-containers | 05:08 | |
| *** janki has quit IRC | 05:09 | |
| *** janki has joined #openstack-containers | 05:10 | |
| *** yamamoto has quit IRC | 05:27 | |
| *** yamamoto has joined #openstack-containers | 05:30 | |
| *** penick has joined #openstack-containers | 05:32 | |
| *** yamamoto has quit IRC | 05:36 | |
| *** penick_ has joined #openstack-containers | 05:36 | |
| *** penick has quit IRC | 05:36 | |
| *** chhavi has joined #openstack-containers | 05:39 | |
| *** chhavi has quit IRC | 05:45 | |
| *** chhavi has joined #openstack-containers | 05:46 | |
| *** penick_ has quit IRC | 06:12 | |
| *** kiennt26 has quit IRC | 06:22 | |
| *** kiennt26 has joined #openstack-containers | 06:22 | |
| *** janki has quit IRC | 06:22 | |
| *** yamamoto has joined #openstack-containers | 06:35 | |
| *** yamamoto has quit IRC | 06:41 | |
| *** yolanda has joined #openstack-containers | 06:47 | |
| *** dardelean_ has joined #openstack-containers | 06:49 | |
| *** dardelean_ has quit IRC | 06:54 | |
| *** armaan has quit IRC | 06:55 | |
| *** armaan has joined #openstack-containers | 06:55 | |
| *** mjura has joined #openstack-containers | 06:59 | |
| *** jchhatbar has joined #openstack-containers | 07:00 | |
| *** dpawar has quit IRC | 07:06 | |
| *** dsariel has quit IRC | 07:17 | |
| *** dpawar has joined #openstack-containers | 07:23 | |
| *** magicboiz has quit IRC | 07:25 | |
| *** rcernin has quit IRC | 07:31 | |
| *** dpawar has quit IRC | 07:35 | |
| *** dpawar has joined #openstack-containers | 07:35 | |
| *** yamamoto has joined #openstack-containers | 07:45 | |
| *** AlexeyAbashkin has joined #openstack-containers | 07:52 | |
| *** rcernin has joined #openstack-containers | 08:02 | |
| *** hieulq has quit IRC | 08:10 | |
| *** hieulq has joined #openstack-containers | 08:11 | |
| *** yolanda__ has joined #openstack-containers | 08:34 | |
| gokhan | hi ykarel , are you there ? I apply https://review.openstack.org/#/c/525662/ and https://review.openstack.org/#/c/447687/ then I get user data too large error | 08:34 |
| gokhan | ykarel, http://paste.openstack.org/show/629285/ | 08:35 |
| *** yolanda has quit IRC | 08:36 | |
| openstackgerrit | Ricardo Rocha proposed openstack/magnum master: [kubernetes] add ingress controller https://review.openstack.org/528756 | 08:45 |
| *** mdnadeem has joined #openstack-containers | 08:47 | |
| armaan | hello folks, i just installed magnum in ocata setup ans i am getting this error in master "kubelet[1950]: Error: unknown flag: --config" which stopped Kubernetes Kubelet Server, Any idea what could be the reason? | 08:49 |
| armaan | morning folks, I installed magnum in both Ocata and Pike and in both versions Kubernetes services die in master nodes for some reason. | 08:50 |
| openstackgerrit | Ricardo Rocha proposed openstack/magnum master: [kubernetes] add ingress controller https://review.openstack.org/528756 | 08:50 |
| armaan | Should i file a bug for this error? | 08:51 |
| *** magicboiz has joined #openstack-containers | 09:00 | |
| *** magicboiz has quit IRC | 09:04 | |
| *** magicboiz has joined #openstack-containers | 09:05 | |
| armaan | gokhan: hi, did you use OSA to deploy magnum? | 09:12 |
| *** flwang1 has joined #openstack-containers | 09:31 | |
| *** armaan has quit IRC | 09:33 | |
| *** armaan has joined #openstack-containers | 09:34 | |
| flwang1 | strigazi: around? | 09:40 |
| strigazi | flwang1 hi | 09:43 |
| flwang1 | strigazi: sorry, are you in holiday? | 09:43 |
| strigazi | flwang1: no, I'm at the office | 09:44 |
| flwang1 | strigazi: cool, have a moment? | 09:44 |
| flwang1 | need your comments on the features i'm working on | 09:44 |
| strigazi | flwang1: I'm looking in the affinity patch | 09:45 |
| *** armaan has quit IRC | 09:45 | |
| *** armaan has joined #openstack-containers | 09:46 | |
| strigazi | flwang1: Are there cases that setting anti-affinity might not work? | 09:46 |
| flwang1 | strigazi: yes, for example, if there are only 3 host available, and user want to create a cluster with 4 nodes, it will fail | 09:47 |
| flwang1 | actually, my question is if we should leave the option to end user, or just hardcode it | 09:48 |
| strigazi | flwang1 or a conf in magnum.conf | 09:49 |
| flwang1 | or in the middle, make it configurable with magnum.conf | 09:49 |
| flwang1 | yes | 09:49 |
| strigazi | flwang1 What works for you? | 09:49 |
| flwang1 | for the first step, i like to add a config in magnum.conf | 09:49 |
| strigazi | flwang1 If we pass "" to heat, will it work? | 09:50 |
| flwang1 | and see if we can get any keen from ops before adding a new argument | 09:50 |
| flwang1 | IIRC, the default policy for server group is anti-affinity | 09:51 |
| flwang1 | but i'm not sure if heat can support an empty policy for server group | 09:51 |
| flwang1 | ricolin: ^ | 09:51 |
| strigazi | flwang1 btw, adding a label field doesn't change the api | 09:51 |
| strigazi | but I prefer an option in magnum conf | 09:52 |
| flwang1 | strigazi: i saw that, but I'm not sure if it's a 'label' | 09:52 |
| flwang1 | a label means it's a metadata/attr of coe cluster, but for this case, it's not | 09:52 |
| flwang1 | personally, i think we can start with a baby step and see how things going | 09:53 |
| flwang1 | if we do have users want to have an argument, we can easily add it later | 09:54 |
| *** mgoddard has joined #openstack-containers | 09:54 | |
| strigazi | flwang1: sure, I agree. We just need to decide which default won't complicate things for new magnum deployments | 09:54 |
| flwang1 | strigazi: yes, that's why i use soft-anti-affinity | 09:55 |
| strigazi | flwang1: ok, so soft-anti-affinity works in all cases? | 09:57 |
| strigazi | flwang1: if so, have a look in this patch: to see how to pass a parameter from magnum.conf to heat https://review.openstack.org/#/c/525662/ | 09:59 |
| flwang1 | https://specs.openstack.org/openstack/nova-specs/specs/kilo/approved/soft-affinity-for-server-group.html | 09:59 |
| flwang1 | strigazi: cool, thanks for sharing | 10:00 |
| *** kiennt26 has quit IRC | 10:02 | |
| strigazi | flwang1 for monitoring, we need a kind of policy on when to mark the cluster unhealthy and which nodes are down | 10:02 |
| *** hishh has joined #openstack-containers | 10:02 | |
| flwang1 | strigazi: i'm thinking another way after some investigation | 10:02 |
| flwang1 | or i would say another additional info | 10:03 |
| flwang1 | because i failed to figure out the policy how to define if the cluster is healthy based on the number of 'failed' nodes | 10:04 |
| flwang1 | for example, if there are 2 minions nodes are unhealthy, but all master nodes are good, can we say the cluster is unhealthy? | 10:05 |
| flwang1 | it would be nice if we can be consistent with how k8s 'calculate' this | 10:05 |
| strigazi | flwang1 even if a single node is down, the cluster is not perfectly healthy | 10:06 |
| strigazi | flwang1: we can say in the status field how many and which nodes are down | 10:07 |
| flwang1 | strigazi: yep | 10:08 |
| flwang1 | that's what i'm trying to do, instead of telling if the cluster is healthy or not | 10:09 |
| flwang1 | just tell them how many nodes are down and the total | 10:09 |
| *** salmankhan has joined #openstack-containers | 10:09 | |
| flwang1 | does that make sense? | 10:09 |
| strigazi | flwang1 what do you mean instead? Isn't it the same? | 10:09 |
| flwang1 | initially, i just want to put HEALTHY and UNHEALTHY, but now i'm thinking if we can just say 1 down / 10 total, or something like that | 10:11 |
| strigazi | flwang1 was actually thinking both | 10:12 |
| strigazi | or only unhealthy actually | 10:12 |
| strigazi | so when you do cluster list, you will see that a cluster is UNHEALTHY | 10:12 |
| flwang1 | you mean 'adding' a field when showing a cluster? | 10:13 |
| flwang1 | i will +1 for that idea | 10:13 |
| flwang1 | and we can even add the component status as well | 10:13 |
| flwang1 | like scheduler, etcd, controller-manager, etc | 10:14 |
| strigazi | flwang1 right now we have status and and status_reasomn | 10:15 |
| flwang1 | strigazi: i see. but unfortunately, it's useless after created the cluster | 10:17 |
| flwang1 | strigazi: i think the discussion related to how to position magnum | 10:17 |
| strigazi | flwang1: if everything is ok, we don't change anything. If something is down or 'problematic' we change the status_reason to says what is the problem | 10:17 |
| flwang1 | if something is wrong, will the 'status' be updated? | 10:18 |
| strigazi | flwang1 We can have a new status which would be UNHEALTHY (henve the upper case letters) | 10:19 |
| flwang1 | strigazi: we should if we go for that idea, because current status is useless IMHO | 10:20 |
| flwang1 | that's 'status' position mangum like a deployment tool, not a service | 10:20 |
| gokhan | hi armaan, yes I used OSA pike | 10:21 |
| strigazi | flwang1 good point | 10:21 |
| armaan | gokhan: Is it working for you? | 10:21 |
| flwang1 | if magnum is providing a coe management service, magnum need to monitor the cluster status, reflect it to end user and even auto healing | 10:21 |
| *** fragatin_ has joined #openstack-containers | 10:22 | |
| gokhan | armaan, yep it is working but I made lots of changes | 10:22 |
| *** fragatina has quit IRC | 10:22 | |
| strigazi | flwang1 yes, that is the goal henve ths blueprint: https://blueprints.launchpad.net/magnum/+spec/cluster-healing | 10:22 |
| flwang1 | strigazi: if magnum want to do auto healing, that means magnum is providing a service | 10:23 |
| flwang1 | that's the thing i'm happy to see | 10:23 |
| strigazi | gokhan can you share your changes summarized somewhere? | 10:23 |
| armaan | gokhan: Is it possible that you could share the changes with me. I have been trying to make it work for few days now? | 10:23 |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack/magnum master: Update kubernetes dashboard to v1.8.1 https://review.openstack.org/507465 | 10:25 |
| gokhan | strigazi, ok I can share but before that for certificates we must apply this patches https://review.openstack.org/#/c/525662/ and https://review.openstack.org/#/c/447687/ then I get user data too large error. | 10:27 |
| gokhan | strigazi, if I don't apply these patches I need move my CA manual | 10:28 |
| gokhan | armaan, are you using heat wih ssl ? | 10:29 |
| strigazi | gokhan Ok, in master I fixed the problem with user_data with this patch: https://review.openstack.org/#/c/468816/ | 10:29 |
| armaan | gokhan: ssl termination happens at the haproxy | 10:29 |
| *** dardelean_ has joined #openstack-containers | 10:30 | |
| gokhan | armaan, ok I get it . it is like my environment. did you use also self signed-cert? | 10:30 |
| gokhan | strigazi, ok it means I also need this patch :) can it be possible to cherry pick thise patches for pike ? | 10:31 |
| strigazi | we will | 10:31 |
| strigazi | we will cherry-pick | 10:31 |
| armaan | gokhan: I think so, my user_variables.yml have this https://pastebin.com/jKtcg4JP | 10:32 |
| flwang1 | strigazi: i have a question about the certs in magnum | 10:32 |
| armaan | gokhan: Master spawns up but Kubernetes API never get started and minion also never spawns up | 10:33 |
| flwang1 | strigazi: i got a problem, when create cluster, there is not certs under /etc/kuberneters/certs | 10:34 |
| flwang1 | which cause etcd can't start | 10:34 |
| flwang1 | strigazi: any idea? or how can i debug it? | 10:34 |
| strigazi | Every one has the same problem with apis behind tls :( | 10:34 |
| strigazi | This is the first patch we start to carry | 10:35 |
| strigazi | flwang1 check in /var/log/cloud-init-output.log | 10:35 |
| strigazi | if part004 failed it means that the node failed to get the ca from magnum | 10:36 |
| gokhan | strigazi, ok thanks and last question where I can place openstack_cafile in magnum.conf ? | 10:36 |
| gokhan | armaan, ok your environment is like me. is it pike or ocata ? | 10:37 |
| strigazi | in [drivers] openstack_ca_file | 10:37 |
| gokhan | armaan, on ocata you can not create kubernates cluster, because it uses kubernates previos version and we can not define ca-file on it. so you must use pike or upgrade kubernates on heat templates | 10:38 |
| armaan | gokhan: Pike | 10:38 |
| gokhan | strigazi, ok thanks. | 10:39 |
| armaan | gokhan: Oh, you mean for Ocata we need to change the magnum heat templates | 10:40 |
| flwang1 | strigazi: cool, thanks a lot | 10:40 |
| gokhan | armaan, it is great :) Firstly you need your auth url on heat templates. Because for auth url it uses keystone public endpoint and magnum can not work with versionless keystone, magnum need keystone v3 | 10:41 |
| *** syedarmani has joined #openstack-containers | 10:41 | |
| gokhan | armaan, can you ssh master node ? | 10:41 |
| *** jchhatbar is now known as janki | 10:41 | |
| armaan | gokhan: Yes, i can ssh into master node | 10:41 |
| strigazi | the above problem is solved in magnum pike | 10:41 |
| gokhan | strigazi, may be bug is about openstack ansible I am not sure. | 10:42 |
| strigazi | it is in ansible | 10:43 |
| gokhan | armaan, can you share your /var/log/cloud-init-output.log? | 10:44 |
| gokhan | armaan, and share output of openstack endpoint list | grep keystone | 10:44 |
| gokhan | armaan, follow this http://eavesdrop.openstack.org/irclogs/%23openstack-containers/%23openstack-containers.2017-12-13.log.html | 10:46 |
| armaan | gokhan: cloud-init https://pastebin.com/eFN3zZJS | 10:46 |
| armaan | gokhan: keystone endpoint example https://pastebin.com/2nzuj3ag | 10:49 |
| gokhan | armaan, look for AUTH_URL in /etc/sysconfig/heat-params ? it it same with https://tky1.cloud.com:5000 ? | 10:52 |
| gokhan | armaan, you need change it with https://tky1.cloud.com:5000/v3 | 10:52 |
| armaan | gokhan: AUTH_URL="https://tky1.cloud.com:5000/v3/ | 10:53 |
| armaan | gokhan: do i need to change my keystone endpoint and add /v3 | 10:54 |
| strigazi | armaan: gokhan or add redirection | 10:54 |
| *** yamamoto has quit IRC | 10:55 | |
| *** yamamoto has joined #openstack-containers | 10:55 | |
| *** yamamoto has quit IRC | 10:56 | |
| gokhan | armaan, no you don't need. your problem is about /var/lib/cloud/instance/scripts/part-005. this is different from me. | 10:56 |
| *** yamamoto has joined #openstack-containers | 10:56 | |
| gokhan | armaan, it seem tky1.citycloud.com is unreacheable on openstack instance | 10:56 |
| *** fragatina has joined #openstack-containers | 10:57 | |
| *** fragatin_ has quit IRC | 10:57 | |
| gokhan | armaan, for looking problems /var/log/cloud-init-output.log is great. strigazi I haven't enough information about /var/lib/cloud/instance/scripts/part-005. can you help armaan ? | 11:00 |
| armaan | gokhan: I can ping the public endpoint of keystone but curl does not work | 11:04 |
| armaan | nevermind, i can curl as well | 11:05 |
| gokhan | armaan, "curl -v https://tky1.cloud.com:5000/v3/" result is ok ? | 11:06 |
| gokhan | armaan, can you share your /var/lib/cloud/instance/scripts/part-005 ? | 11:08 |
| gokhan | armaan, you need -k option in curl lines | 11:09 |
| *** yamamoto has quit IRC | 11:09 | |
| *** hishh has quit IRC | 11:11 | |
| armaan | gokhan: Yep, curl works fine. Here is the /var/lib/cloud/instance/scripts/part-005 file https://pastebin.com/yh3sttyxhttps://pastebin.com/yh3sttyx | 11:12 |
| *** yamamoto has joined #openstack-containers | 11:13 | |
| gokhan | armaan, can you run /var/lib/cloud/instance/scripts/part-005 this file | 11:17 |
| *** yamamoto has quit IRC | 11:18 | |
| armaan | gokhan: Failed to connect to tky1.cloud.com port 9511: No route to host | 11:20 |
| *** AlexeyAbashkin has quit IRC | 11:22 | |
| armaan | gokhan: I have to leave office now. thanks a lot for your help here. I will let you know if could debug my issue :) | 11:28 |
| *** armaan has quit IRC | 11:28 | |
| *** armaan has joined #openstack-containers | 11:28 | |
| gokhan | armaan, your tky1.cloud.com is unrecheable I can't find reason of it. may be you need consult magnum cores. if you want, I can share my openstack.user.config.yml and user_variables .yml for OSA | 11:30 |
| flwang1 | strigazi: here is the log i got from cloud init http://paste.openstack.org/show/629298/ | 11:30 |
| flwang1 | part005 failed | 11:30 |
| *** armaan has quit IRC | 11:33 | |
| *** armaan has joined #openstack-containers | 11:33 | |
| flwang1 | strigazi: i'm using pike, so i think it's matching the result https://github.com/openstack/magnum/blob/stable/pike/magnum/drivers/k8s_fedora_atomic_v1/templates/kubemaster.yaml#L474 | 11:34 |
| armaan | gokhan: awesome, could you please share the user_variables.yml? i think that will be helpful for me :) | 11:34 |
| gokhan | armaan, my environment is HA multinode with ceph | 11:34 |
| armaan | gokhan: I have the same environment, HA multinode with ceph | 11:35 |
| gokhan | armaan, this is openstack user config yml file http://paste.openstack.org/show/629295/ | 11:35 |
| gokhan | armaan, and this is user_variables.yml file http://paste.openstack.org/show/629299/ | 11:37 |
| *** AlexeyAbashkin has joined #openstack-containers | 11:38 | |
| *** magicboiz has quit IRC | 11:42 | |
| *** magicboiz has joined #openstack-containers | 11:44 | |
| *** armaan has quit IRC | 11:45 | |
| *** yolanda__ is now known as yolanda | 11:50 | |
| *** dardelean_ has quit IRC | 11:59 | |
| flwang1 | rochapor1o: ping | 12:01 |
| strigazi | flwang1 Check if you can contact the magnum keystone and heat api from the node, there is either a problem with finding a route or about certs | 12:05 |
| flwang1 | /var/lib/cloud/instance/scripts/part-003: line 5: [: 15c5c48e-ba4e-41c1-9ee9-b0ff0f38a592_SIZE: integer expression expected Cloud-init v. 0.7.9 running 'modules:final' at Tue, 19 Dec 2017 02:39:25 +0000. Up 550.14 seconds. 2017-12-19 02:39:39,264 - util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-005 [1] | 12:05 |
| flwang1 | 15c5c48e-ba4e-41c1-9ee9-b0ff0f38a592_SIZE looks weird | 12:06 |
| *** yamamoto has joined #openstack-containers | 12:06 | |
| flwang1 | https://github.com/openstack/magnum/blob/master/magnum/drivers/common/templates/kubernetes/fragments/configure-etcd.sh#L5 | 12:06 |
| strigazi | flwang1 did you pass --docker-volume-size? | 12:06 |
| flwang1 | yes, with 5GB | 12:06 |
| strigazi | flwang1 but you didn't pass etcd_volume_size | 12:07 |
| strigazi | flwang1 openstack stack show <your stack> | grep etcd_volume_size | 12:08 |
| flwang1 | strigazi: where to pass in the etcd_volume_size? | 12:08 |
| strigazi | https://docs.openstack.org/magnum/pike/user/index.html#etcd-volume-size | 12:09 |
| flwang1 | ubuntu@feilong-dev-conf0:~$ openstack stack show k8scluster-ssxntk7cu4m6 | grep etcd_volume_size | 12:09 |
| flwang1 | | | etcd_volume_size: '0' | 12:09 |
| flwang1 | so when setting the docker-volume-size, i have to set the etcd_volume_size as well? | 12:10 |
| strigazi | No, they are optional | 12:10 |
| strigazi | you don't have to pass anything | 12:10 |
| *** yamamoto has quit IRC | 12:10 | |
| flwang1 | ok, so what's the problem? | 12:10 |
| strigazi | grep ETCD_VOLUME /etc/sysconfig/heat-params | 12:10 |
| flwang1 | [root@k8scluster-ssxntk7cu4m6-master-0 fedora]# grep ETCD_VOLUME /etc/sysconfig/heat-params | 12:11 |
| flwang1 | ETCD_VOLUME="15c5c48e-ba4e-41c1-9ee9-b0ff0f38a592" | 12:11 |
| flwang1 | ETCD_VOLUME_SIZE="15c5c48e-ba4e-41c1-9ee9-b0ff0f38a592_SIZE" | 12:11 |
| strigazi | flwang1 oh no, there is the problem | 12:12 |
| flwang1 | is it because i'm using a grandpa heat? | 12:13 |
| strigazi | this is a small bug, but it is not related to your problem | 12:13 |
| flwang1 | does that mean i can skip it? | 12:14 |
| strigazi | flwang1 give me a sec to check | 12:14 |
| flwang1 | ok | 12:14 |
| *** salmankhan has quit IRC | 12:15 | |
| strigazi | flwang1: yes it is a problem of grandpa heat but we can address. | 12:15 |
| strigazi | how ever | 12:15 |
| strigazi | however, the problem that you have is different | 12:15 |
| flwang1 | strigazi: do you know what's the problem i'm facing ;) | 12:16 |
| strigazi | flwang1: yes, you can not connect to the openstack api from the nodes. | 12:17 |
| flwang1 | strigazi: really? | 12:17 |
| flwang1 | let me double check | 12:17 |
| strigazi | flwang1: set -x && /var/lib/cloud/instance/scripts/part-005 | 12:18 |
| *** salmankhan has joined #openstack-containers | 12:18 | |
| flwang1 | i think you're right | 12:20 |
| flwang1 | we may need another change for the network in our CI | 12:21 |
| flwang1 | strigazi: thank you very much. but i don't really understand why the connection issue will make the etcd fail | 12:23 |
| *** dpawar has quit IRC | 12:24 | |
| strigazi | flwang1 the node generates a csr request and asks the magnum API to sign it's key | 12:24 |
| strigazi | then the certs are use to secure etcd and the kubernetes api | 12:25 |
| strigazi | then the certs are used to secure etcd and the kubernetes api | 12:25 |
| strigazi | Check what part-005 does | 12:25 |
| flwang1 | ah, i see. that makes much sense | 12:25 |
| flwang1 | thank a lot | 12:25 |
| strigazi | it gets the CA from magnum and then asks magnum to sign it | 12:25 |
| strigazi | you are welcome | 12:26 |
| flwang1 | btw, what's the difference between make-cert.sh and make-cert-client.sh ? | 12:26 |
| *** dardelean_ has joined #openstack-containers | 12:27 | |
| strigazi | make-cert.sh is executed on the master node, client on the worker nodes | 12:27 |
| strigazi | after rbac the two of them are becoming more different | 12:27 |
| flwang1 | got | 12:28 |
| flwang1 | thanks | 12:28 |
| *** dpawar has joined #openstack-containers | 12:31 | |
| *** yamamoto has joined #openstack-containers | 12:38 | |
| *** yamamoto has quit IRC | 12:39 | |
| *** janki has quit IRC | 13:11 | |
| *** janki has joined #openstack-containers | 13:11 | |
| *** rcernin has quit IRC | 13:20 | |
| *** dpawar has quit IRC | 13:25 | |
| *** dsariel has joined #openstack-containers | 13:30 | |
| *** armaan has joined #openstack-containers | 13:34 | |
| *** dardelean_ has quit IRC | 13:35 | |
| *** dardelean_ has joined #openstack-containers | 13:36 | |
| gokhan | ping strigazi , | 13:37 |
| strigazi | gokhan: hi | 13:38 |
| gokhan | strigazi, ı applied three patch for certs but ı got error on notify heat | 13:38 |
| gokhan | [fedora@test-zx3ewgo7w5lg-master-0 ~]$ /usr/local/bin/wc-notify | 13:38 |
| gokhan | #!/bin/bash -v | 13:38 |
| gokhan | until curl -sf "http://127.0.0.1:8080/healthz"; do | 13:38 |
| gokhan | echo "Waiting for Kubernetes API..." | 13:38 |
| gokhan | sleep 5 | 13:38 |
| gokhan | done | 13:38 |
| gokhan | okcurl -i -X POST -H 'X-Auth-Token: gAAAAABaOQ9PcCrsmsEzmHsB9VT5O-YYRIMD-6Q77w1Yqg8UyKK7juGbMo9_oAHmgLsFSuo2LL6sxRIWqbID3otmhVsg9ui8TQyoenGAWYUBOxDY2mRLAAjg64pKALnED9nq5lHm5JSnzWVBLibINrmN8z4IUV2wqXT6lxBxnaCBZ6dGSYY18S8' -H 'Content-Type: application/json' -H 'Accept: application/json' https://safircloud.b3lab.org:8004/v1/a32d6d8183d6416687c8a5bfcb9b9b85/stacks/test-zx3ewgo7w5lg-kube_masters-sg27lkigtb2k-0-gfyxrzsksv4u/b329ac5a-248d-4edf-966d-f | 13:38 |
| gokhan | 09730167130/resources/master_wait_handle/signal True -k --data-binary '{"status": "SUCCESS"}' | 13:39 |
| gokhan | HTTP/1.1 200 OK | 13:39 |
| gokhan | Content-Type: application/json | 13:39 |
| gokhan | Content-Length: 4 | 13:39 |
| gokhan | x-openstack-request-id: req-93957cef-3211-402e-9374-d54d6833fb80 | 13:39 |
| gokhan | nullcurl: (6) Could not resolve host: True | 13:39 |
| strigazi | gokhan you should use paste.openstack.org :) | 13:39 |
| strigazi | what is the problem? us returned 200, no? | 13:39 |
| strigazi | what is the problem? it returned 200, no? | 13:40 |
| *** yamamoto has joined #openstack-containers | 13:40 | |
| *** dardelean_ has quit IRC | 13:40 | |
| gokhan | strigazi, yep it is 200 but wc-notify.service is failed | 13:40 |
| gokhan | strigazi, is this not problem ? | 13:41 |
| strigazi | gokhan it says inactive dead or failed? | 13:42 |
| strigazi | gokhan: journalctl -u wc-notify.service --no-pager | 13:43 |
| gokhan | strigazi, it is failed http://paste.openstack.org/show/629316/ | 13:44 |
| *** armaan has quit IRC | 13:45 | |
| openstackgerrit | Spyros Trigazis (strigazi) proposed openstack/magnum master: Update kubernetes dashboard to v1.8.1 https://review.openstack.org/507465 | 13:45 |
| *** dave-mccowan has joined #openstack-containers | 13:45 | |
| strigazi | gokhan I don't think it is a problem, it sent the signal and got 200. I don't know why system says failed | 13:47 |
| strigazi | gokhan I don't think it is a problem, it sent the signal and got 200. I don't know why systemd says failed | 13:47 |
| *** yamamoto has quit IRC | 13:48 | |
| *** ykarel has quit IRC | 13:48 | |
| *** dsariel has quit IRC | 13:53 | |
| gokhan | strigazi, because of failure of wc-notify.service master node stucks | 13:56 |
| strigazi | gokhan what do you mean stucks? | 13:57 |
| strigazi | gokhan what is the flavor of the vm? | 13:57 |
| gokhan | strigazi, it is about 50 minutes, and can not spawn minion nodes | 13:57 |
| strigazi | openstack stack resource list -n 2 <stack_id> | grep -v COMPLETE | 13:58 |
| gokhan | strigazi, it is medium, 2 vcpu, 4GB RAM | 13:58 |
| strigazi | disk? | 13:59 |
| gokhan | strigazi, http://paste.openstack.org/show/629318/ | 14:01 |
| gokhan | strigazi, disk 40 GB | 14:02 |
| gokhan | strigazi, I have to go now I will be back later | 14:06 |
| *** dardelean_ has joined #openstack-containers | 14:06 | |
| *** ramishra has quit IRC | 14:47 | |
| *** jmlowe has joined #openstack-containers | 14:51 | |
| *** dardelean_ has quit IRC | 14:54 | |
| *** marst has joined #openstack-containers | 15:10 | |
| *** kiennt26 has joined #openstack-containers | 15:11 | |
| *** dardelean_ has joined #openstack-containers | 15:11 | |
| *** livelace has joined #openstack-containers | 15:23 | |
| *** kiennt26 has quit IRC | 15:27 | |
| *** chhavi has quit IRC | 15:30 | |
| strigazi | Hello everyone, the magnum meeting will start in 25' in #openstack-meeting-alt | 15:35 |
| *** slunkad_ has quit IRC | 15:36 | |
| *** ykarel has joined #openstack-containers | 15:37 | |
| *** slunkad has joined #openstack-containers | 15:39 | |
| *** slunkad has quit IRC | 15:43 | |
| *** dardelean_ has quit IRC | 15:51 | |
| *** oikiki has joined #openstack-containers | 15:55 | |
| *** slunkad has joined #openstack-containers | 15:59 | |
| *** dardelean_ has joined #openstack-containers | 16:00 | |
| strigazi | slunkad: are you there? | 16:03 |
| openstackgerrit | Kirsten G. proposed openstack/magnum master: Add enable_pull_coe_data configuration parameter https://review.openstack.org/529098 | 16:03 |
| slunkad | slunkad: yes | 16:03 |
| *** mjura has quit IRC | 16:03 | |
| slunkad | strigazi: sorry forgot about the meeting | 16:05 |
| *** livelace has quit IRC | 16:06 | |
| *** dardelean_ has quit IRC | 16:06 | |
| *** dardelean_ has joined #openstack-containers | 16:06 | |
| *** dardelean_ has quit IRC | 16:07 | |
| *** ykarel has quit IRC | 16:11 | |
| *** jmlowe has quit IRC | 16:13 | |
| *** dardelean_ has joined #openstack-containers | 16:14 | |
| *** ricolin has quit IRC | 16:19 | |
| strigazi | oikiki: hi | 16:29 |
| oikiki | strigazi: hi! | 16:29 |
| strigazi | Did you manage to deploy devstack with barbican? | 16:29 |
| oikiki | I just got in from a redeye and am now utc-5 | 16:30 |
| oikiki | i submitted a patch to #1 | 16:30 |
| *** janki has quit IRC | 16:31 | |
| oikiki | i had some issues getting devstack running again | 16:32 |
| oikiki | but im going to work on getting barbican running today | 16:32 |
| oikiki | after i do my cherry pick | 16:32 |
| strigazi | oikiki: ok, do you need any help? | 16:33 |
| oikiki | i dont think so im going to give it a try after i get a bit of sleep | 16:33 |
| strigazi | oikiki: ok cool | 16:34 |
| oikiki | could you explain why the cherry pick? | 16:34 |
| strigazi | oikiki: magnum deployments may use stable/pike or stable/ocata, so we need to make the patch available for those releases | 16:35 |
| strigazi | oikiki: now, your patch is in the master branch | 16:35 |
| strigazi | oikiki: and it will be available with the next release in February | 16:35 |
| strigazi | that would be stable/queens | 16:36 |
| oikiki | strigazi: ah i understand now | 16:36 |
| strigazi | ok | 16:36 |
| strigazi | oikiki: do you prefer a different time to sync? | 16:36 |
| oikiki | no im ok | 16:36 |
| strigazi | ok | 16:36 |
| *** armaan has joined #openstack-containers | 16:37 | |
| strigazi | oikiki: If you need anything, send me an email | 16:37 |
| oikiki | so my commit was in master, we will make it avail in stable/pike and eventually it be a part of the next release stable/queens | 16:37 |
| strigazi | oikiki it will be available in both releases | 16:37 |
| strigazi | yes | 16:37 |
| oikiki | strigazi: and I just cherry pick in gerrit | 16:38 |
| oikiki | neat! | 16:38 |
| strigazi | oikiki: if the patch applies cleanly you can use just gerrit | 16:38 |
| oikiki | strigazi: so when i cherry pick in gerrit i will see any conflicts there to fix? | 16:39 |
| strigazi | oikiki if there are conflicts, gerrit will say it can't do the cherry-pick | 16:40 |
| strigazi | in this case | 16:40 |
| strigazi | you need to clone tha magnum repo | 16:40 |
| strigazi | checkout stable/pike | 16:40 |
| strigazi | branch on stable/pike | 16:40 |
| strigazi | git checkout -b <a-branch-name> | 16:41 |
| strigazi | and then you can do git review -x 447687 or git fetch https://git.openstack.org/openstack/magnum refs/changes/87/447687/38 && git cherry-pick FETCH_HEAD | 16:42 |
| *** penick has joined #openstack-containers | 16:42 | |
| strigazi | which will cherry-pick the patch | 16:42 |
| strigazi | then you do git status and git will tell you how to solve the conflicts | 16:42 |
| strigazi | makes sense? | 16:42 |
| oikiki | gotcha! | 16:42 |
| oikiki | i'll give it a try thank you! | 16:44 |
| strigazi | oikiki you are welcome | 16:44 |
| *** salmankhan has quit IRC | 16:46 | |
| oikiki | strigazi: also I am utc-5 this week and next week (if you are around, I know it's the holidays) | 16:46 |
| strigazi | This is east coast in the US right? | 16:48 |
| oikiki | yep | 16:48 |
| strigazi | Are you going to be online tmr morning? | 16:49 |
| oikiki | yep | 16:49 |
| oikiki | :) | 16:49 |
| strigazi | cool, I'll be online as well | 16:49 |
| oikiki | great! | 16:49 |
| armaan | gokhan: We had to add a rule in iptables for port 9511 and then curl started working and both master and minion came up. In the master node all services are up except for dashboard http://paste.openstack.org/show/629329/ | 16:52 |
| armaan | gokhan: Do we have to open any port to make dashboard work as well | 16:52 |
| *** basvanveen has joined #openstack-containers | 16:53 | |
| armaan | strigazi: Hello! | 16:53 |
| strigazi | armaan hi | 16:54 |
| armaan | strigazi: I was wondering if you have any suggestion about this: http://paste.openstack.org/show/629329/ | 16:54 |
| *** basvanve_ has joined #openstack-containers | 16:54 | |
| strigazi | armaan kube-system-namespace. is not a problem | 16:55 |
| strigazi | armaan: kubernetes creates it by itself | 16:55 |
| strigazi | armaan: you are in pike right? | 16:55 |
| armaan | strigazi: ok and what do you think about the dashboard? | 16:55 |
| armaan | strigazi: Yep, Pike. | 16:56 |
| strigazi | kube 1.7 ? | 16:56 |
| armaan | strigazi: Kubernetes v1.6.7 | 16:56 |
| strigazi | in pike? how is this possible? | 16:57 |
| strigazi | armaan: in pike we pull kuberentes in containers | 16:57 |
| *** basvanveen has quit IRC | 16:57 | |
| strigazi | and the default is 1.7.4 | 16:57 |
| *** salmankhan has joined #openstack-containers | 16:58 | |
| strigazi | something didn't run, anyway, you can add the dashboard manually or patch magnum to use dashboard 1.6.3 | 16:59 |
| *** basvanve_ has quit IRC | 16:59 | |
| armaan | strigazi: I am using this tag of OSA https://github.com/openstack/openstack-ansible/tree/16.0.5 | 17:00 |
| armaan | which is Pike. | 17:00 |
| armaan | magnum 4301 4198 0 Dec18 ? 00:00:13 /openstack/venvs/magnum-16.0.5/bin/python | 17:00 |
| armaan | this is from within the magnum container | 17:00 |
| strigazi | ok | 17:00 |
| strigazi | armaan: on the master node what is in /var/log/cloud-init-output.log ? | 17:01 |
| *** AlexeyAbashkin has quit IRC | 17:01 | |
| armaan | strigazi: http://paste.openstack.org/show/629333/ | 17:02 |
| strigazi | line 130 says apiserver v1.7.4 | 17:03 |
| strigazi | output of kubectl version | 17:04 |
| armaan | # kubelet --version | 17:04 |
| armaan | Kubernetes v1.6.7 | 17:04 |
| strigazi | armaan these are the binaries installed in the qcow | 17:04 |
| strigazi | they are not used | 17:04 |
| strigazi | doL | 17:04 |
| strigazi | do: | 17:04 |
| strigazi | atomic containers list | 17:05 |
| armaan | ok | 17:05 |
| armaan | strigazi: atomic container list http://paste.openstack.org/show/629334/ | 17:05 |
| strigazi | --no-trunc | 17:05 |
| armaan | strigazi: http://paste.openstack.org/show/629335/ | 17:06 |
| strigazi | see 1.7.4 :) | 17:06 |
| strigazi | that is good | 17:07 |
| armaan | strigazi: cool :) | 17:07 |
| armaan | now i know how to find out the binary in use | 17:07 |
| *** dardelean_ has quit IRC | 17:08 | |
| armaan | back to dashboard, any ideas on how to troubleshoot it? | 17:08 |
| strigazi | I don't have a fix to deploy the right dashboard automatically, I'm working on it, What you can do, in a runnig cluster, is, delete the existing dashboard and deploy kuberenetes dashboard 1.6.3 | 17:09 |
| armaan | strigazi: do you happen to know any document or article on how to do that? | 17:10 |
| strigazi | with kubectl create -f https://raw.githubusercontent.com/kubernetes/dashboard/v1.6.3/src/deploy/kubernetes-dashboard-no-rbac.yaml | 17:10 |
| strigazi | armaan to delete the old on check | 17:11 |
| strigazi | kubectl -n kube-system get deployment | 17:11 |
| strigazi | kubectl -n kube-system get cm | 17:11 |
| strigazi | kubectl -n kube-system get svc | 17:11 |
| strigazi | and delete everything about the old dashboard | 17:11 |
| armaan | strigazi: thanks a million for these commands. Could you please recommend any book which can help me learn these things? | 17:12 |
| armaan | strigazi: http://paste.openstack.org/show/629337/ | 17:14 |
| armaan | strigazi: kubectl delete kubernetes-dashboard? | 17:15 |
| strigazi | try the kunbernetes tutorials and I think there is a free MOOC from the linux foundation. https://kubernetes.io/docs/tutorials/ | 17:15 |
| strigazi | kubectl --namespace delete deployment kubernetes-dashboard | 17:16 |
| strigazi | kubectl --namespace delete service kubernetes-dashboard | 17:16 |
| strigazi | kubectl --namespace delete serviceaccount kubernetes-dashboard | 17:16 |
| strigazi | kubectl --namespace delete configmap kubernetes-dashboard | 17:16 |
| armaan | strigazi: awesome, thanks. there is a edx course as well. nice! | 17:16 |
| strigazi | kubectl --namespace get <a kube object> # with this you can check what is there to delete | 17:17 |
| strigazi | autocomplete work very well | 17:17 |
| strigazi | kubectl completion bash > kube-complete.sh && source kube-complete.sh | 17:18 |
| strigazi | kubectl get namespace # see your namespaces | 17:18 |
| strigazi | kubectl --namepace kube-system get <tab> <tab> :) | 17:18 |
| *** dsariel has joined #openstack-containers | 17:22 | |
| armaan | strigazi: :) :) http://paste.openstack.org/show/629340/ | 17:22 |
| strigazi | armaan better don't be root | 17:23 |
| strigazi | the fedora user has access to the correct kubectl version | 17:23 |
| strigazi | do whereis kubectl | 17:24 |
| armaan | roger | 17:24 |
| armaan | strigazi: As fedora user: http://paste.openstack.org/show/629341/ | 17:25 |
| strigazi | here is the kubectl you need /var/usrlocal/bin/kubectl | 17:27 |
| armaan | strigazi: sorry for being a noob here. I am new to K8. | 17:28 |
| strigazi | Magnum provides a cluster-config commands to configure your client and talk to kube from your laptop | 17:28 |
| strigazi | you can do openstack coe cluser config <cluster> | 17:29 |
| strigazi | magnum will set your credentials and you can talk to kube from somewhere that you can reach the master node | 17:29 |
| strigazi | https://docs.openstack.org/magnum/latest/install/launch-instance.html#provision-a-kubernetes-cluster-and-create-a-deployment | 17:30 |
| armaan | strigazi: thanks for sharing the link. I tried to install the kubectl but something is strange here http://paste.openstack.org/show/629347/ | 17:34 |
| strigazi | chmox +x kubectl && ./kubectl version | 17:35 |
| armaan | damn :/ | 17:36 |
| armaan | stupid mistake | 17:36 |
| armaan | :( | 17:36 |
| armaan | strigazi: I am wondering what runs on port 8080? http://paste.openstack.org/show/629348/ | 17:41 |
| armaan | strigazi: In our prod environment we run swift on 8080 | 17:41 |
| armaan | ohh /usr/bin/kube-controller-manager --logtostderr=true --v=0 --master=http://127.0.0.1:8080 | 17:44 |
| strigazi | armaan only on 127.0.0.1 | 17:44 |
| strigazi | armaan: I have to go | 17:45 |
| strigazi | see you tmr | 17:45 |
| * strigazi is away | 17:45 | |
| armaan | strigazi: thanks a million for your help | 17:45 |
| armaan | strigazi: I appreciate it a lot! | 17:45 |
| *** mdnadeem has quit IRC | 17:52 | |
| *** janonymous has quit IRC | 18:02 | |
| *** dsariel has quit IRC | 18:09 | |
| *** harlowja has joined #openstack-containers | 18:15 | |
| *** dsariel has joined #openstack-containers | 18:17 | |
| *** AlexeyAbashkin has joined #openstack-containers | 18:22 | |
| *** armaan has quit IRC | 18:24 | |
| *** AlexeyAbashkin has quit IRC | 18:27 | |
| *** dsariel has quit IRC | 18:27 | |
| *** AlexeyAbashkin has joined #openstack-containers | 18:46 | |
| *** armaan has joined #openstack-containers | 18:57 | |
| *** AlexeyAbashkin has quit IRC | 18:58 | |
| *** basvanveen has joined #openstack-containers | 19:06 | |
| *** dardelean_ has joined #openstack-containers | 19:08 | |
| *** basvanveen has quit IRC | 19:11 | |
| *** ricolin has joined #openstack-containers | 19:11 | |
| *** flwang1 has quit IRC | 19:15 | |
| *** ricolin_ has joined #openstack-containers | 19:38 | |
| *** basvanveen has joined #openstack-containers | 19:38 | |
| *** ricolin has quit IRC | 19:40 | |
| *** openstack has joined #openstack-containers | 19:43 | |
| *** ChanServ sets mode: +o openstack | 19:43 | |
| *** basvanveen has quit IRC | 19:43 | |
| *** dardelean_ has joined #openstack-containers | 19:46 | |
| *** dardelean_ has quit IRC | 19:51 | |
| *** itlinux has quit IRC | 19:52 | |
| *** itlinux has joined #openstack-containers | 19:52 | |
| *** ricolin_ has quit IRC | 19:53 | |
| *** salmankhan has quit IRC | 19:53 | |
| *** fragatina has quit IRC | 19:56 | |
| *** dardelean_ has joined #openstack-containers | 20:20 | |
| *** syedarmani has quit IRC | 20:22 | |
| *** dardelean_ has quit IRC | 20:24 | |
| *** armaan has quit IRC | 20:27 | |
| *** jmlowe has joined #openstack-containers | 20:28 | |
| *** syedarmani has joined #openstack-containers | 20:34 | |
| *** AlexeyAbashkin has joined #openstack-containers | 20:37 | |
| *** armaan has joined #openstack-containers | 20:38 | |
| *** AlexeyAbashkin has quit IRC | 20:41 | |
| *** flwang1 has joined #openstack-containers | 20:44 | |
| oikiki | I think strigazi is away. Is anyone able to answer a quick question re: cherry picking for an Openstack intern? | 20:50 |
| *** penick has quit IRC | 20:53 | |
| oikiki | I am cherry picking my commit that added verify_ca to stable/pike | 20:59 |
| oikiki | I added 1 line but the cherry pick is picking up another change where someone added master_flavor prior to my commit to master | 20:59 |
| oikiki | http://paste.openstack.org/show/629373 | 20:59 |
| *** oikiki has quit IRC | 20:59 | |
| *** oikiki has joined #openstack-containers | 21:00 | |
| *** armaan has quit IRC | 21:16 | |
| *** armaan_ has joined #openstack-containers | 21:17 | |
| *** penick has joined #openstack-containers | 21:20 | |
| *** armaan_ has quit IRC | 21:28 | |
| *** armaan has joined #openstack-containers | 21:29 | |
| *** salmankhan has joined #openstack-containers | 21:30 | |
| *** salmankhan has quit IRC | 21:34 | |
| *** dardelean_ has joined #openstack-containers | 21:35 | |
| *** dardelean_ has quit IRC | 21:40 | |
| openstackgerrit | Kirsten G. proposed openstack/magnum stable/pike: Add verify_ca configuration parameter https://review.openstack.org/529166 | 21:46 |
| *** armaan has quit IRC | 21:48 | |
| *** armaan has joined #openstack-containers | 21:48 | |
| armaan | cd /var/lib/cloud/ | 21:54 |
| *** armaan has quit IRC | 22:05 | |
| *** armaan has joined #openstack-containers | 22:06 | |
| *** pcichy has joined #openstack-containers | 22:08 | |
| *** pcichy has quit IRC | 22:09 | |
| *** pcichy has joined #openstack-containers | 22:09 | |
| *** penick has quit IRC | 22:19 | |
| *** dardelean_ has joined #openstack-containers | 22:19 | |
| *** oikiki has quit IRC | 22:21 | |
| *** dardelean_ has quit IRC | 22:23 | |
| *** flwang1 has quit IRC | 22:24 | |
| *** penick has joined #openstack-containers | 22:25 | |
| *** rcernin has joined #openstack-containers | 22:28 | |
| *** dardelean_ has joined #openstack-containers | 22:29 | |
| *** pcichy has quit IRC | 22:38 | |
| *** penick has quit IRC | 22:40 | |
| *** marst has quit IRC | 22:41 | |
| *** penick has joined #openstack-containers | 22:43 | |
| *** oikiki has joined #openstack-containers | 22:57 | |
| *** flwang1 has joined #openstack-containers | 23:02 | |
| *** flwang1 has quit IRC | 23:08 | |
| *** flwang1 has joined #openstack-containers | 23:14 | |
| *** dardelean_ has quit IRC | 23:19 | |
| *** oikiki has quit IRC | 23:21 | |
| *** itlinux has quit IRC | 23:33 | |
| *** syedarmani has quit IRC | 23:37 | |
| *** flwang1 has quit IRC | 23:37 | |
| *** penick has quit IRC | 23:39 | |
| *** syedarmani has joined #openstack-containers | 23:39 | |
| *** syedarmani has joined #openstack-containers | 23:40 | |
| *** flwang1 has joined #openstack-containers | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!