| *** crushil has joined #openstack-kolla | 00:00 | |
| *** crushil has quit IRC | 00:03 | |
| *** crushil has joined #openstack-kolla | 00:03 | |
| *** crushil has quit IRC | 00:06 | |
| *** crushil has joined #openstack-kolla | 00:07 | |
| *** crushil has quit IRC | 00:10 | |
| *** crushil has joined #openstack-kolla | 00:10 | |
| *** crushil has quit IRC | 00:13 | |
| *** dave-mccowan has joined #openstack-kolla | 00:13 | |
| *** crushil has joined #openstack-kolla | 00:13 | |
| *** crushil has quit IRC | 00:17 | |
| *** crushil has joined #openstack-kolla | 00:17 | |
| *** crushil has quit IRC | 00:20 | |
| *** crushil has joined #openstack-kolla | 00:20 | |
| *** crushil has quit IRC | 00:23 | |
| *** crushil has joined #openstack-kolla | 00:24 | |
| *** crushil has quit IRC | 00:27 | |
| *** crushil has joined #openstack-kolla | 00:27 | |
| *** crushil has quit IRC | 00:30 | |
| *** crushil has joined #openstack-kolla | 00:31 | |
| *** crushil has quit IRC | 00:34 | |
| *** crushil has joined #openstack-kolla | 00:34 | |
| *** crushil has quit IRC | 00:37 | |
| *** crushil has joined #openstack-kolla | 00:38 | |
| *** crushil has quit IRC | 00:41 | |
| *** crushil has joined #openstack-kolla | 00:41 | |
| *** schwicht has joined #openstack-kolla | 00:42 | |
| *** cu5 has quit IRC | 00:44 | |
| *** crushil has quit IRC | 00:44 | |
| *** cu5 has joined #openstack-kolla | 00:44 | |
| *** crushil has joined #openstack-kolla | 00:44 | |
| *** duonghq has joined #openstack-kolla | 00:50 | |
| duonghq | morning guys | 00:50 |
|---|---|---|
| *** crushil has quit IRC | 00:51 | |
| openstackgerrit | pangliye proposed openstack/kolla-ansible master: update configuration of telegraf "config_files" https://review.openstack.org/446888 | 00:51 |
| openstackgerrit | pangliye proposed openstack/kolla-ansible master: Update configuration of Telegraf retention_policy https://review.openstack.org/447179 | 00:53 |
| sdake | morning duonghq | 00:53 |
| duonghq | hi sdake | 00:54 |
| sdake | duonghq finally back in the office now - \o/ | 00:54 |
| *** tovin07_ has joined #openstack-kolla | 00:55 | |
| *** eaguilar has quit IRC | 00:57 | |
| *** schwicht has quit IRC | 00:58 | |
| duonghq | sdake, glad to here that, | 00:58 |
| *** qiliang27 has joined #openstack-kolla | 01:09 | |
| *** cuongnv has joined #openstack-kolla | 01:12 | |
| sdake | duonghq just read the logs from the droproot discussion | 01:19 |
| sdake | duonghq it appeared like youwere going to ask me a question | 01:19 |
| duonghq | sdake, yup, I asked you one day later | 01:19 |
| sdake | cool | 01:19 |
| sdake | so allset then? | 01:19 |
| duonghq | yes, after that, we decided to close the bug: drop root for crontab | 01:20 |
| *** eaguilar has joined #openstack-kolla | 01:23 | |
| *** jrobinson_ has joined #openstack-kolla | 01:28 | |
| *** jrobinson_ has quit IRC | 01:29 | |
| *** jrobinson_ has joined #openstack-kolla | 01:30 | |
| *** jrobinson has quit IRC | 01:31 | |
| *** jrobinson_ is now known as jrobinson | 01:31 | |
| *** zhubingbing_ has joined #openstack-kolla | 01:31 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Update the "virt_driver" to "virt_type" https://review.openstack.org/447336 | 01:35 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Update the "virt_driver" to "virt_type" https://review.openstack.org/447336 | 01:36 |
| sdake | duonghq https://review.openstack.org/#/c/411043/3 | 01:41 |
| duonghq | sdake, I will abandon it, seem that nobody care about this atm :) | 01:44 |
| sdake | cares about which | 01:44 |
| sdake | use cases? | 01:44 |
| sdake | one problem we have is a lack of review capacity | 01:45 |
| duonghq | the way which I proposed: use temporary review for use cases contribution | 01:45 |
| sdake | duonghq if you want to do that - that works put a [DNM} in the front of the message header to keep in queue | 01:45 |
| sdake | that tells reviewers that it is not ready for merge | 01:45 |
| sdake | an alternative approach is to set workflow -1 | 01:45 |
| duonghq | the problem is: I'm not sure if anybody like that way to tracking use cases | 01:46 |
| duonghq | how do you think? | 01:46 |
| sdake | tracking use cases is best done in blueprints | 01:48 |
| duonghq | yes | 01:48 |
| zhubingbing_ | agree | 01:56 |
| sdake | zhubingbing_ how are you this fine evening | 01:58 |
| * sdake got a fitbit tracker and has been averaging 7 hours of sleep a night | 01:59 | |
| sdake | surprising to me, as I thought I was averaging 6 :) | 01:59 |
| zhubingbing_ | ;) | 01:59 |
| sdake | zhubingbing_ i see your review related to fluentd was abandoned and unabandoned | 02:00 |
| zhubingbing_ | yes | 02:00 |
| sdake | zhubingbing_ could you explain? | 02:00 |
| sdake | misclick or planning to submit a new review? | 02:00 |
| zhubingbing_ | Fluentd should be every pod should have， | 02:02 |
| zhubingbing_ | i don't sure my patch is right | 02:03 |
| zhubingbing_ | I need to look at the code again and think about it | 02:03 |
| zhubingbing_ | ；） | 02:03 |
| zhubingbing_ | My English is so bad | 02:04 |
| sdake | every kubernetes deployment object should have a fluentd process, or every microchart? | 02:14 |
| zhubingbing_ | yes | 02:15 |
| sdake | yes to which - both? :) | 02:15 |
| zhubingbing_ | every kubernetes deployment object should have a fluentd process | 02:15 |
| zhubingbing_ | pod | 02:15 |
| *** daidv has joined #openstack-kolla | 02:17 | |
| zhubingbing_ | https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_lib.yaml | 02:17 |
| zhubingbing_ | sdake | 02:17 |
| zhubingbing_ | same as this | 02:17 |
| sdake | do you mean this? https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_lib.yaml#L30 | 02:18 |
| zhubingbing_ | yes | 02:19 |
| sdake | anything that uses the common_containers macro? | 02:19 |
| zhubingbing_ | kfox1111 tell me we should same as this to do | 02:20 |
| sdake | zhubingbing_ have a irc log handy? | 02:21 |
| zhubingbing_ | i finding | 02:21 |
| duonghq | zhubingbing_, which is tag of your mariadb in k8s cluster? | 02:23 |
| zhubingbing_ | http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-03-15.log.html#t2017-03-15T13:50:00 | 02:25 |
| zhubingbing_ | 4.0.0 ？ | 02:25 |
| zhubingbing_ | sdake link http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-03-15.log.html#t2017-03-15T13:50:00 | 02:25 |
| duonghq | hmm | 02:26 |
| zhubingbing_ | duonghq i don't know tag what 's mean ~~ | 02:27 |
| duonghq | docker images | 02:27 |
| duonghq | the tag column | 02:27 |
| zhubingbing_ | i know your problems | 02:27 |
| zhubingbing_ | ；） i don‘t running mariadb in k8s | 02:28 |
| zhubingbing_ | you | 02:28 |
| duonghq | I use helm to deploy mariadb | 02:28 |
| duonghq | you don't? | 02:28 |
| zhubingbing_ | you don‘t running mariadb in k8s ， | 02:28 |
| zhubingbing_ | i use helm | 02:28 |
| zhubingbing_ | do you deploy success？ | 02:28 |
| duonghq | no, mariadb stuck | 02:29 |
| zhubingbing_ | i | 02:29 |
| zhubingbing_ | i know your problems | 02:29 |
| zhubingbing_ | -) | 02:29 |
| duonghq | ya, still stuck | 02:30 |
| duonghq | the k8s cluster deployed by kargo is very stable | 02:30 |
| sdake | thanks zhubingbing_ | 02:30 |
| zhubingbing_ | I can help you solve this problem duonghq | 02:31 |
| duonghq | the log of bootstrap: http://paste.openstack.org/show/603355/ | 02:31 |
| zhubingbing_ | yes | 02:31 |
| zhubingbing_ | i know this problem | 02:31 |
| *** japestinho has joined #openstack-kolla | 02:32 | |
| zhubingbing_ | first you should remove /var/lib/kolla/volumes/mariadb | 02:33 |
| zhubingbing_ | rm -rf /var/lib/kolla/volumes/mariadb/* | 02:34 |
| zhubingbing_ | next helm delete --purge mariadb | 02:34 |
| zhubingbing_ | finally helm install again | 02:34 |
| sdake | zhubingbing_ a better workflow for that solution si to helm delete --purge mariadb and then rm -rf the volumes | 02:35 |
| sdake | (I think your workflow is out of order to start over) | 02:35 |
| zhubingbing_ | yes | 02:35 |
| zhubingbing_ | ;) | 02:35 |
| zhubingbing_ | sdake it is right | 02:35 |
| zhubingbing_ | sdake say right | 02:36 |
| duonghq | sdake, zhubingbing_ thanks, I already did it last Friday, no luck, but today, it's fine | 02:36 |
| duonghq | don't know why | 02:36 |
| sdake | duonghq what have people been using to deploy kubernetes this last two weeks instead of halcyon which apparently has some problems? | 02:37 |
| sdake | duonghq kargo or kubeadm? | 02:37 |
| zhubingbing_ | it’s mariadb a bug | 02:37 |
| duonghq | sdake, I'm using kargo | 02:37 |
| sdake | does kargo do vms? | 02:37 |
| sdake | duonghq or you setting up on bare metal? | 02:37 |
| duonghq | I created 3 vms (kvm), | 02:38 |
| duonghq | and point kargo to those | 02:38 |
| duonghq | kargo take the duty | 02:38 |
| sdake | kubeadm can do the same thing | 02:38 |
| sdake | did you try out kubeadm? | 02:38 |
| duonghq | I tried it last year, I slapped me many time, this year I haven't | 02:39 |
| zhubingbing_ | sdake i use kubeadm | 02:39 |
| zhubingbing_ | ;) | 02:39 |
| zhubingbing_ | doughq it‘s mariadb bug，not k8s | 02:39 |
| duonghq | zhubingbing_, can you explain? | 02:39 |
| zhubingbing_ | yeah | 02:40 |
| sdake | we have been working on a workfow for deployment in production in https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP | 02:40 |
| duonghq | sdake, I follow this workflow | 02:40 |
| duonghq | and stuck with mariadb | 02:40 |
| sdake | duonghq nice | 02:40 |
| sdake | duonghq i think we can maek the first part (whether to use kubeadm or kargo or bring your own kubernetes deployment tool) up to the operator | 02:41 |
| sdake | duonghq how well does the workflow work for you? | 02:41 |
| sdake | duonghq it sounds like "not well" :( | 02:41 |
| zhubingbing_ | if mariadb has been created before, mariadb.sock file if not deleted, will retain the previous data. | 02:41 |
| duonghq | I got trouble with mariadb from kolla-ansible, so I think mariadb don't like me, if I can get over this, maybe everything will be fine | 02:41 |
| zhubingbing_ | so mysql-safe will need you create new root password | 02:41 |
| sdake | duonghq trouble with mariadb from kolla-ansible how? | 02:42 |
| duonghq | iirc, I got stuck with Galera, WSREP... it was a bug | 02:43 |
| duonghq | but it's fine now | 02:43 |
| duonghq | zhubingbing_, got it | 02:43 |
| sdake | ah - so mariadb was broken but is no longer | 02:43 |
| sdake | duonghq so in theory those docs are good to convert to rst? | 02:43 |
| duonghq | so I'll add solving procedure to the guide | 02:43 |
| sdake | as is the docs are hard to understand - i'd like to get them into rst as soon as possible | 02:43 |
| zhubingbing_ | ;) | 02:43 |
| sdake | and I have all monday to do so | 02:43 |
| sdake | nobody followed the directions and added 'TESTED" with a date however | 02:44 |
| sdake | zhubingbing_ did those instructoins work for you? | 02:44 |
| duonghq | hmm, I think we can convert to rst, and add some progress later, | 02:45 |
| sdake | kargo has some wierd dns thing which may make things not work properly | 02:45 |
| sdake | kargo needs some workaround in the resolver workaround | 02:46 |
| sdake | we got workarounds for ourworkarounds ;) | 02:46 |
| duonghq | so, I should move back to kubeadm or find workaround for kargo, seem tricky | 02:46 |
| duonghq | one more think, we use different tag (which mean different version) for different microservice? | 02:47 |
| duonghq | *thing | 02:47 |
| zhubingbing_ | How to build k8s cluster, I think we can refer to K8s official website ；） | 02:48 |
| duonghq | zhubingbing_, can you give me link? | 02:49 |
| zhubingbing_ | ok | 02:49 |
| duonghq | last time I tried, no luck for me | 02:50 |
| zhubingbing_ | https://kubernetes.io/docs/ | 02:50 |
| zhubingbing_ | kargo and kubedm | 02:50 |
| duonghq | okay | 02:51 |
| duonghq | so we should note that we recommend that way | 02:51 |
| zhubingbing_ | i thinks sow | 02:52 |
| zhubingbing_ | so | 02:52 |
| duonghq | now, I got this error: Entrypoint WARNING: 2017/03/20 02:54:03 entrypoint.go:81: Resolving dependency for mariadb-init-element failed: Get https://10.233.0.1:443/apis/batch/v1/namespaces/kolla/jobs/mariadb-init-element: dial tcp 10.233.0.1:443: getsockopt: no route to host | 02:53 |
| duonghq | 02:53 | |
| zhubingbing_ | i am learning how to note ；） | 02:53 |
| zhubingbing_ | dns？ | 02:54 |
| duonghq | yup, dns again | 02:54 |
| zhubingbing_ | i suggest dns ip is setting the machine ip | 02:56 |
| openstackgerrit | Gao Zexu proposed openstack/kolla-ansible master: Add a missing blank space https://review.openstack.org/447348 | 02:58 |
| duonghq | zhubingbing_, you mean the configmap? | 03:08 |
| *** Margin has joined #openstack-kolla | 03:10 | |
| zhubingbing_ | cloud.yaml | 03:14 |
| sdake | hey folks looks like the etherpad was edited on March 9th and critical data was delete | 03:19 |
| sdake | d | 03:19 |
| *** dave-mccowan has quit IRC | 03:20 | |
| zhubingbing_ | roger | 03:21 |
| sdake | i'll get an rst pulled down from pre-9th and check the changes after the 9th for this same sort of behavior | 03:21 |
| sdake | no wonder everyone is struggling to get the docs deployed | 03:22 |
| *** zhurong has joined #openstack-kolla | 03:25 | |
| zhubingbing_ | sdake when do we make a deployment command | 03:27 |
| sdake | zhubingbing_ you mean like kolla-kubernetes deploy? | 03:28 |
| zhubingbing_ | same as kolla-ansible deploy | 03:28 |
| zhubingbing_ | yes | 03:28 |
| zhubingbing_ | yes | 03:28 |
| sdake | we are building up the stack | 03:28 |
| sdake | the next step of implementation is the external config maps | 03:28 |
| zhubingbing_ | are we planning？ | 03:30 |
| zhubingbing_ | ok | 03:31 |
| duonghq | zhubingbing_, which section in cloud.yaml? | 03:34 |
| sdake | zhubingbing_ lets get a manual deploy working first ;) | 03:41 |
| zhubingbing_ | sorry dough | 03:42 |
| zhubingbing_ | i was not here | 03:42 |
| zhubingbing_ | sdake，https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP i thinks it is good | 03:42 |
| zhubingbing_ | but we can optimize this file | 03:43 |
| sdake | zhubingbing_ there was a bit of vandalism - working on removing that | 03:47 |
| sdake | and then its going in gerrit where it can't be vandalized | 03:48 |
| zhubingbing_ | ok | 03:48 |
| zhubingbing_ | so we should write a new deploy gudie | 03:49 |
| zhubingbing_ | sdake https://review.openstack.org/#/c/439740/ | 03:50 |
| sdake | zhubingbing_ that patch needs to be made to pass the gate | 03:50 |
| sdake | it is correct, however it fails on the gate | 03:50 |
| sdake | zhubingbing_ if you want to do that instead of fluentd, that would have a larger immediate impact | 03:51 |
| zhubingbing_ | this patch we should merge it ；） | 03:51 |
| sdake | I've been traveling for 3 weeks | 03:51 |
| *** eaguilar has quit IRC | 03:51 | |
| sdake | 2 weeks | 03:51 |
| duonghq | sdake, why we still use mariadb 2.0.2 in k8s? | 03:51 |
| sdake | and the week before was very busy haven't had time to rework the gate for this particular patch | 03:51 |
| zhubingbing_ | i help | 03:51 |
| sdake | duonghq no clue | 03:51 |
| zhubingbing_ | ；） | 03:51 |
| zhubingbing_ | duonghq i use 4.0.0 | 03:52 |
| zhubingbing_ | ;) | 03:52 |
| sdake | duonghq from what I gather, kfox1111 seems to indicate it makes migration easier for existing deployments | 03:52 |
| sdake | although kolla-kubernetes in my mind will be a mostly greenfield deployment so the docs should indicate as much | 03:52 |
| duonghq | ya, | 03:52 |
| duonghq | agreed with sdake | 03:52 |
| duonghq | zhubingbing_, ok | 03:52 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 03:52 |
| *** Margin has quit IRC | 03:53 | |
| zhubingbing_ | duonghq sdake we should finish Deployment Guide Documentation in this week | 03:59 |
| zhubingbing_ | ;) | 03:59 |
| sdake | zhubingbing_ YES | 03:59 |
| duonghq | yup | 04:00 |
| *** jrobinson has quit IRC | 04:00 | |
| zhubingbing_ | ok | 04:00 |
| *** jrobinson has joined #openstack-kolla | 04:05 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 04:10 |
| sdake | https://review.openstack.org/#/c/447356/1..2 | 04:11 |
| sdake | you should see the changes between 1 and 2 - they are glorious | 04:11 |
| sdake | the interim changes that is.. | 04:12 |
| duonghq | ya, got the mariadb up | 04:15 |
| *** zhurong has quit IRC | 04:16 | |
| zhubingbing_ | ；0 | 04:17 |
| zhubingbing_ | ;) | 04:17 |
| zhubingbing_ | luncheon | 04:17 |
| duonghq | seem that we use many image from 2.0.0 | 04:17 |
| zhubingbing_ | eating | 04:17 |
| duonghq | 2.0.2 | 04:17 |
| zhubingbing_ | ;) | 04:17 |
| duonghq | any idea | 04:17 |
| duonghq | rabbitmq .... | 04:17 |
| zhubingbing_ | default is 3.0.2 | 04:17 |
| duonghq | zhubingbing_, you need setting default to 3.0.2 manually or it's in code? | 04:18 |
| zhubingbing_ | in code | 04:18 |
| duonghq | zhubingbing_, latest code is 2.0.2 | 04:20 |
| duonghq | I just try building cloud.yaml | 04:20 |
| duonghq | it put 2.0.2 in image_tag | 04:20 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 04:21 |
| sdake | you can put 4.0.0 image tag in cloud.yaml | 04:22 |
| sdake | although I don't know that anyone has tested 4.0.0 - I'd go with 3.0.2 | 04:22 |
| zhubingbing_ | https://github.com/openstack/kolla-kubernetes/blob/master/helm/all_values.yaml | 04:22 |
| zhubingbing_ | i thinks 4.0.0 is better 3.0.2 | 04:22 |
| zhubingbing_ | ;) | 04:22 |
| zhubingbing_ | yes | 04:25 |
| duonghq | sdake, zhubingbing_ should we discuss with Kevin to bump the default version to 3.0.2? | 04:26 |
| sdake | he seems resistent | 04:26 |
| sdake | although we can put whatever we like in the deployment guide - and the answer there is 4.0.0 | 04:26 |
| zhubingbing_ | 2.0.2 is too old i think | 04:27 |
| zhubingbing_ | yes | 04:27 |
| sdake | although lets get 3.0.2 deploy docs published - I know these instructions work and there is video evidence indicating as much :) | 04:27 |
| duonghq | also, in https://etherpad.openstack.org/p/kolla-create-configmaps we miss many configmap *-logging | 04:27 |
| sdake | duonghq no clue why that is | 04:27 |
| duonghq | something like this: 3m1m9{kubelet master}WarningFailedMountMountVolume.SetUp failed for volume "kubernetes.io/configmap/163e6fe6-0d25-11e7-a419-52540066ae24-logging-config" (spec.Name: "logging-config") pod "163e6fe6-0d25-11e7-a419-52540066ae24" (UID: "163e6fe6-0d25-11e7-a419-52540066ae24") with: configmaps "rabbitmq-logging" not found | 04:27 |
| sdake | duonghq let me de-vandalize the deployment guide and I'll be right with you | 04:28 |
| duonghq | roger | 04:28 |
| zhubingbing_ | duonghq | 04:29 |
| duonghq | zhubingbing_, here | 04:29 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Fix the remove mariadb volume failed https://review.openstack.org/447359 | 04:29 |
| zhubingbing_ | you should kollakube res create configmap mariadb keystone horizon rabbitmq memcached nova-api nova-conductor nova-scheduler glance-api-haproxy glance-registry-haproxy glance-api glance-registry neutron-server neutron-dhcp-agent neutron-l3-agent neutron-metadata-agent neutron-openvswitch-agent openvswitch-db-server openvswitch-vswitchd nova-libvirt nova-compute nova-consoleauth nova-novncproxy nova-novncproxy-hapro | 04:29 |
| zhubingbing_ | xy neutron-server-haproxy nova-api-haproxy cinder-api cinder-api-haproxy cinder-backup cinder-scheduler cinder-volume iscsid tgtd keepalived | 04:29 |
| zhubingbing_ | http://paste.openstack.org/show/603360/ | 04:30 |
| duonghq | zhubingbing_, already do this | 04:30 |
| zhubingbing_ | you should create configmap before deploy | 04:30 |
| duonghq | but rabbitmq-logging does not in the list | 04:30 |
| duonghq | zhubingbing_, the rabbitmq statefulset is: http://paste.openstack.org/show/603361/ | 04:31 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 04:32 |
| zhubingbing_ | [root@kube1 root]# aa get pods | 04:32 |
| zhubingbing_ | NAME READY STATUS RESTARTS AGE | 04:32 |
| zhubingbing_ | mariadb-0 1/1 Running 0 3d | 04:32 |
| zhubingbing_ | rabbitmq-0 1/1 Running 0 1h | 04:32 |
| zhubingbing_ | ;) | 04:32 |
| * duonghq get 2 helm service up and running | 04:33 | |
| duonghq | with 2 more configmaps | 04:33 |
| duonghq | mariadb-logging and rabbitmq-logging | 04:33 |
| zhubingbing_ | i know ;) | 04:33 |
| *** daidv has quit IRC | 04:35 | |
| duonghq | situation becomes worst, it requires memcached-logging but none is defined | 04:39 |
| sdake | wonder where the logging stuff came from :) | 04:39 |
| sdake | the etherpad for create-configmaps was not modified - although an unnamed author added a lowercase l on the 13th | 04:40 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Fix the remove mariadb volume failed https://review.openstack.org/447359 | 04:40 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Deployment Guide Documentation https://review.openstack.org/447356 | 04:40 |
| sdake | duonghq https://review.openstack.org/#/c/447356/4..5/doc/source/deployment-guide.rst | 04:41 |
| duonghq | sdake, nice | 04:42 |
| zhubingbing_ | nice | 04:43 |
| duonghq | hmm, zhubingbing_ where is logging section in cloud.yaml | 04:43 |
| zhubingbing_ | i testing | 04:44 |
| zhubingbing_ | wait | 04:44 |
| zhubingbing_ | ；） | 04:44 |
| duonghq | openstack_logging_debug: "True" | 04:45 |
| duonghq | does it relate? | 04:45 |
| zhubingbing_ | no | 04:45 |
| zhubingbing_ | i don't about logging section | 04:46 |
| zhubingbing_ | i have see the code | 04:46 |
| zhubingbing_ | - name: logging | 04:47 |
| zhubingbing_ | image: "duonghq:40000/kolla/centos-binary-fluentd:4.0.0" | 04:47 |
| zhubingbing_ | imagePullPolicy: "IfNotPresent" | 04:47 |
| zhubingbing_ | volumeMounts: | 04:47 |
| zhubingbing_ | - mountPath: /var/lib/kolla/config_files | 04:47 |
| zhubingbing_ | name: logging-config | 04:47 |
| zhubingbing_ | - mountPath: /var/log/kolla/ | 04:47 |
| zhubingbing_ | name: kolla-logs | 04:47 |
| zhubingbing_ | - mountPath: /etc/localtime | 04:47 |
| zhubingbing_ | name: host-etc-localtime | 04:47 |
| zhubingbing_ | readOnly: true | 04:47 |
| zhubingbing_ | env: | 04:47 |
| zhubingbing_ | - name: KOLLA_CONFIG_STRATEGY | 04:47 |
| zhubingbing_ | value: COPY_ONCE | 04:47 |
| zhubingbing_ | volumes: | 04:47 |
| zhubingbing_ | - name: host-etc-localtime | 04:47 |
| zhubingbing_ | hostPath: | 04:47 |
| zhubingbing_ | path: /etc/localtime | 04:47 |
| zhubingbing_ | - name: kolla-logs | 04:47 |
| zhubingbing_ | emptyDir: {} | 04:47 |
| zhubingbing_ | - name: logging-config | 04:47 |
| zhubingbing_ | configMap: | 04:47 |
| zhubingbing_ | name: rabbitmq-logging | 04:47 |
| zhubingbing_ | - name: pod-main-config | 04:48 |
| zhubingbing_ | emptyDir: {} | 04:48 |
| zhubingbing_ | - name: rabbitmq-config | 04:48 |
| zhubingbing_ | configMap: | 04:48 |
| zhubingbing_ | name: rabbitmq | 04:48 |
| zhubingbing_ | - name: rabbitmq-persistent-storage | 04:48 |
| zhubingbing_ | persistentVolumeClaim: | 04:48 |
| zhubingbing_ | claimName: rabbitmq | 04:48 |
| zhubingbing_ | sorry | 04:48 |
| zhubingbing_ | ;) | 04:48 |
| zhubingbing_ | http://paste.openstack.org/show/603363/ | 04:48 |
| zhubingbing_ | https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_lib.yaml#L30 | 04:48 |
| duonghq | can I disable it, due to something trigger configmap dependency to *-logging configmap | 04:50 |
| *** rwsu has joined #openstack-kolla | 04:51 | |
| zhubingbing_ | I think there should be a switch | 04:51 |
| zhubingbing_ | ；） | 04:51 |
| duonghq | yup | 04:52 |
| *** unicell1 has joined #openstack-kolla | 04:53 | |
| *** unicell has quit IRC | 04:54 | |
| zhubingbing_ | doughq i don‘t see about logging config | 04:58 |
| sdake | zhubingbing_ --:( | 04:58 |
| sdake | duonghq arey ou using cloud.yaml? | 04:58 |
| *** zhurong has joined #openstack-kolla | 04:59 | |
| zhubingbing_ | ; | 04:59 |
| zhubingbing_ | https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_lib.yaml#L29 | 04:59 |
| zhubingbing_ | this | 04:59 |
| duonghq | sdake, yes | 04:59 |
| duonghq | zhubingbing_, will test | 04:59 |
| sdake | duonghq which etherpad did you us efor the reference | 04:59 |
| zhubingbing_ | https://github.com/openstack/kolla-kubernetes/blob/master/helm/kolla-common/templates/_common_lib.yaml#L61 | 05:00 |
| duonghq | sdake, your bp | 05:00 |
| duonghq | one was converted to review | 05:00 |
| zhubingbing_ | http://paste.openstack.org/show/603364/ | 05:02 |
| zhubingbing_ | duonghq you can look my | 05:02 |
| zhubingbing_ | my testing | 05:02 |
| *** saneax-_-|AFK is now known as saneax | 05:07 | |
| sdake | zhubingbing_ is it accurate to say you ahve a complete deployment operational with those instructions? | 05:07 |
| sdake | zhubingbing_ and also accurate to say duonghq does not | 05:07 |
| sdake | and the delta is related to the logging? | 05:07 |
| *** saneax is now known as saneax-_-|AFK | 05:08 | |
| zhubingbing_ | I'm not sure where is logging on the switch | 05:08 |
| zhubingbing_ | and i don‘t find logging configmap | 05:09 |
| duonghq | hmm, ok | 05:10 |
| sdake | zhubingbing_ do you ahve an operational AIO/ | 05:10 |
| duonghq | I'll find that one | 05:10 |
| zhubingbing_ | yes | 05:10 |
| sdake | zhubingbing_ can you paste your globals.yaml? | 05:10 |
| zhubingbing_ | i deploy | 05:10 |
| zhubingbing_ | ok | 05:10 |
| sdake | duonghq can you compare your globals.yaml with zhubingbing_ 's? | 05:11 |
| duonghq | zhubingbing_, can you post your globals.yaml | 05:12 |
| zhubingbing_ | wait | 05:12 |
| zhubingbing_ | http://paste.openstack.org/show/603366/ | 05:13 |
| sdake | zhubingbing_ how did you come up with that globals.yaml? | 05:14 |
| zhubingbing_ | it's my globals.yaml | 05:14 |
| *** zhurong has quit IRC | 05:14 | |
| duonghq | min, I need go out for awhile | 05:14 |
| zhubingbing_ | ok | 05:15 |
| duonghq | Will try your global.conf | 05:15 |
| duonghq | thanks | 05:15 |
| zhubingbing_ | http://paste.openstack.org/show/603367/ | 05:16 |
| zhubingbing_ | http://paste.openstack.org/show/603368/ | 05:17 |
| zhubingbing_ | my cloud.yaml | 05:17 |
| duonghq | zhubingbing_, no haproxy? | 05:18 |
| sdake | zhubingbing_ how did you generate cloud.yaml? | 05:18 |
| zhubingbing_ | no haproxy | 05:18 |
| *** skramaja has joined #openstack-kolla | 05:19 | |
| duonghq | sorry guys, I need taking rest for a few mins, | 05:19 |
| sdake | me too | 05:19 |
| duonghq | get back to you later | 05:19 |
| sdake | enjoy :) | 05:19 |
| sdake | its about 10pm here | 05:19 |
| zhubingbing_ | in your bp | 05:19 |
| zhubingbing_ | ；） | 05:19 |
| zhubingbing_ | kube_logger: false | 05:19 |
| zhubingbing_ | i think should set it | 05:20 |
| *** saneax-_-|AFK is now known as saneax | 05:20 | |
| zhubingbing_ | ;) i continue test log | 05:20 |
| zhubingbing_ | ;) | 05:20 |
| sdake | duonghq rwellum updated the etherpad on the 13th with which appears to be a more correct cloud.yaml file | 05:21 |
| sdake | or a less correct cloud.yaml | 05:21 |
| sdake | not sure which :) | 05:21 |
| duonghq | sdake, hmm, ok | 05:22 |
| sdake | it includes the following snippet: | 05:22 |
| sdake | (forgive spam): mariadb: | 05:22 |
| sdake | all: | 05:22 |
| sdake | database_user: root | 05:22 |
| sdake | image_tag: 3.0.3 | 05:22 |
| sdake | node_port: 3306 | 05:22 |
| sdake | node_port_enabled: false | 05:22 |
| sdake | port: 3306 | 05:22 |
| sdake | port_external: false | 05:22 |
| sdake | kube_logger: false | 05:22 |
| sdake | the version I posted lacked a kube_logger: false setting for mariadb | 05:23 |
| sdake | it is possible that from march 1st to march 13th there was some regression in the code base which caused kube_logger to break | 05:23 |
| sdake | (for mariadb) | 05:23 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Delete the deprecated option from the group "DEFAULT" https://review.openstack.org/447368 | 05:23 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Delete the deprecated option from the group "DEFAULT" https://review.openstack.org/447368 | 05:25 |
| duonghq | sdake, I got it error both for mariadb, rabbitmq, memcached | 05:26 |
| duonghq | Will see what kube_logger has done for my installation | 05:27 |
| duonghq | cya | 05:27 |
| sdake | duonghq when you return - try changing helm/all_values.yaml kube_logger:false | 05:29 |
| sdake | duonghq you will need to go through the prebuild process again | 05:30 |
| zhubingbing_ | yes | 05:33 |
| zhubingbing_ | duonghq are you ok ? | 05:37 |
| *** Margin has joined #openstack-kolla | 05:52 | |
| *** caowei has joined #openstack-kolla | 05:58 | |
| *** jaosorior has joined #openstack-kolla | 06:04 | |
| duonghq | zhubingbing_, sdake seem that it's ok | 06:22 |
| duonghq | thanks | 06:22 |
| zhubingbing_ | yeah | 06:22 |
| zhubingbing_ | i reading about kube_logger | 06:23 |
| zhubingbing_ | code | 06:23 |
| duonghq | yup, it's kube_logger, not sure why it's here before your fluentd | 06:24 |
| zhubingbing_ | i thinks it's have some problems about kube_logger | 06:25 |
| zhubingbing_ | i have testing | 06:25 |
| jaosorior | are kolla images uploaded somewhere to dockerhub? is it possible for me to just do docker run with a specific image and just try out a service? | 06:27 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix Telegraf startup config files issue https://review.openstack.org/447382 | 06:29 |
| zhubingbing_ | duonghq do you know {{- include .extraConfigmapConfig . | indent 12 }} | 06:30 |
| zhubingbing_ | what's mean with "indent 12" | 06:30 |
| duonghq | it include the config map, and push 12 spaces before each line of code | 06:35 |
| duonghq | zhubingbing_, | 06:35 |
| duonghq | jaosorior, no | 06:35 |
| zhubingbing_ | ok | 06:36 |
| *** masuberu has joined #openstack-kolla | 06:37 | |
| duonghq | zhubingbing_, can you can post-deploy with your global.yaml | 06:42 |
| jaosorior | duonghq: thanks. I'll proceed to build images then. | 06:42 |
| zhubingbing_ | hmm yes | 06:42 |
| duonghq | zhubingbing_, only http://paste.openstack.org/show/603366/ ? | 06:44 |
| *** schwicht has joined #openstack-kolla | 06:46 | |
| zhubingbing_ | http://paste.openstack.org/show/603367/ | 06:48 |
| zhubingbing_ | duonghq | 06:48 |
| duonghq | okay | 06:51 |
| duonghq | seem that we donot have teardown job jet | 06:51 |
| duonghq | yet | 06:51 |
| *** masuberu has quit IRC | 06:52 | |
| zhubingbing_ | yes | 06:52 |
| *** jaosorior has quit IRC | 06:53 | |
| *** jaosorior has joined #openstack-kolla | 06:55 | |
| *** jrobinson has quit IRC | 07:00 | |
| zhubingbing_ | duonghq | 07:08 |
| zhubingbing_ | [root@kube1 ~]# aa get pods | 07:08 |
| zhubingbing_ | NAME READY STATUS RESTARTS AGE | 07:08 |
| zhubingbing_ | mariadb-0 1/1 Running 0 3d | 07:08 |
| zhubingbing_ | rabbitmq-0 2/2 Running 0 1m | 07:08 |
| zhubingbing_ | [root@kube1 ~]# docker ps | grep flu | 07:08 |
| zhubingbing_ | 0bd5f9fbc80b 192.168.121.64:4000/kolla/centos-source-fluentd:4.0.0 "kolla_start" About a minute ago Up About a minute k8s_logging.8446b10c_rabbitmq-0_kolla_ac0f5db2-0d3b-11e7-b389-525400b7c6af_864ad213 | 07:08 |
| zhubingbing_ | ;) | 07:08 |
| zhubingbing_ | I know how to play logging in kolla-k8s | 07:09 |
| duonghq | hmm, do you need fluentd? | 07:12 |
| duonghq | ah, understand | 07:12 |
| duonghq | you get log directory from the logging container | 07:12 |
| duonghq | *directly | 07:12 |
| zhubingbing_ | yes | 07:13 |
| zhubingbing_ | But I think this design seems a bit of a problem | 07:14 |
| zhubingbing_ | ；） | 07:14 |
| *** jaosorior has quit IRC | 07:14 | |
| *** jaosorior has joined #openstack-kolla | 07:15 | |
| *** jaosorior has quit IRC | 07:16 | |
| *** masuberu has joined #openstack-kolla | 07:18 | |
| *** jaosorior has joined #openstack-kolla | 07:19 | |
| *** satyar has joined #openstack-kolla | 07:20 | |
| duonghq | zhubingbing_, can you provide some more detail | 07:23 |
| duonghq | about problem of design? | 07:23 |
| zhubingbing_ | wait a few min | 07:23 |
| satyar | Hi zhubingbing_ | 07:24 |
| satyar | hi Jeffrey4l__ | 07:24 |
| Jeffrey4l__ | sup satyar | 07:28 |
| satyar | How are you Jeffrey4l__ | 07:28 |
| Jeffrey4l__ | not bad except busying ;) | 07:28 |
| satyar | ha ha ha | 07:29 |
| satyar | want to check if you have tried freezer | 07:29 |
| satyar | openstack freezer ;) | 07:29 |
| *** Margin has quit IRC | 07:30 | |
| *** Margin has joined #openstack-kolla | 07:30 | |
| *** Jeffrey4l__ has quit IRC | 07:33 | |
| *** zhubingbing_ has quit IRC | 07:33 | |
| *** zhubingbing_ has joined #openstack-kolla | 07:34 | |
| *** MarginHu has joined #openstack-kolla | 07:34 | |
| *** Margin has quit IRC | 07:35 | |
| *** manheim has joined #openstack-kolla | 07:35 | |
| *** Jeffrey4l__ has joined #openstack-kolla | 07:36 | |
| *** Jezekus has joined #openstack-kolla | 07:38 | |
| *** MarginHu has quit IRC | 07:43 | |
| *** maniram477 has joined #openstack-kolla | 07:44 | |
| *** zhugaoxiao has quit IRC | 07:57 | |
| *** zhugaoxiao has joined #openstack-kolla | 07:58 | |
| *** matrohon has joined #openstack-kolla | 08:00 | |
| *** shardy has joined #openstack-kolla | 08:04 | |
| *** shardy has joined #openstack-kolla | 08:07 | |
| *** maniram477 has quit IRC | 08:09 | |
| *** pcaruana has joined #openstack-kolla | 08:14 | |
| *** Serlex has joined #openstack-kolla | 08:15 | |
| *** Masber_080 has joined #openstack-kolla | 08:28 | |
| *** masuberu has quit IRC | 08:29 | |
| *** matrohon has quit IRC | 08:32 | |
| *** mpansky has joined #openstack-kolla | 08:36 | |
| *** mnasiadka has joined #openstack-kolla | 08:38 | |
| *** hrw has quit IRC | 08:38 | |
| *** egonzalez has joined #openstack-kolla | 08:40 | |
| *** rmart04 has joined #openstack-kolla | 08:54 | |
| *** athomas has joined #openstack-kolla | 08:57 | |
| hawi | hi. i have question regarding error "Error: Unable to get quota info." after installation. is it possible to fix it somehow or what is actually missing? | 08:59 |
| hawi | some API endpoint is wrong? | 08:59 |
| egonzalez | hawi, which quota? | 09:00 |
| *** mpansky has quit IRC | 09:00 | |
| *** mpansky has joined #openstack-kolla | 09:00 | |
| hawi | i'm getting it from Horizon UI -> Admin -> System -> Defaults | 09:00 |
| *** duke__ has joined #openstack-kolla | 09:01 | |
| *** openstackgerrit has quit IRC | 09:02 | |
| egonzalez | hawi, what version are you using? | 09:02 |
| *** hrw has joined #openstack-kolla | 09:04 | |
| *** schwicht has joined #openstack-kolla | 09:07 | |
| *** mpansky has quit IRC | 09:07 | |
| *** mpansky has joined #openstack-kolla | 09:08 | |
| hawi | ocata on ubuntu 16.04.2. i have no idea how to check kolla/kolla-ansible version, but when downloaded, i took stable/ocata branches | 09:08 |
| *** schwicht has quit IRC | 09:11 | |
| *** Masber_080 has quit IRC | 09:12 | |
| egonzalez | hawi, can you check if quotas are working? to see if the issue is with horizon or general to quotas | 09:12 |
| egonzalez | from CLI ^^ openstack quota show | 09:13 |
| hawi | Project ID 5973eefd76fb4227b3a44273208fee4d is not a valid project. (HTTP 400) (Request-ID: req-92044052-5732-4872-b637-9c4f064e2ee8) | 09:16 |
| hawi | for admin tenant | 09:16 |
| egonzalez | hawi, i suspect you are using master instead of ocata https://bugs.launchpad.net/kolla-ansible/+bug/1668663 | 09:18 |
| openstack | Launchpad bug 1668663 in kolla-ansible "nova quota update command error" [Undecided,In progress] - Assigned to Eduardo Gonzalez (egonzalez90) | 09:18 |
| egonzalez | this bug only affect to master(pike) | 09:19 |
| *** mpansky has quit IRC | 09:24 | |
| hawi | in somehow i was able to install it without any errors...i'll check it, what i actually have | 09:24 |
| *** mpansky has joined #openstack-kolla | 09:25 | |
| hawi | thanks | 09:25 |
| *** masuberu has joined #openstack-kolla | 09:28 | |
| hawi | egonzalez: i think i need to reinstall everything | 09:37 |
| thomas_oneill | morning all | 09:40 |
| egonzalez | hawi, if using ocata, better install kolla-ansible with pip install kolla-ansible==4.0.0 | 09:41 |
| thomas_oneill | I'm trying to fix up the bifrost-base image with it's mysql-server/mariadb-server hack, there is this in bifrost https://github.com/openstack/bifrost/tree/master/playbooks/roles/bifrost-ironic-install/defaults | 09:41 |
| egonzalez | hawi, this way ensure are using correct version, unless you need to use development branch | 09:41 |
| thomas_oneill | when I use the kolla-build.conf with the type = local and the location pointing to the location of the git repo on the machine I'm working on, it doesn't work | 09:41 |
| thomas_oneill | like I change all these defaults to mariadb-server instead of mysql-server in the bifrost repo on my local machine, I set up the kolla-build.conf to point at this local path with this repo with the changes, and then I run the image and go into the container with bash and the bifrost that's in there is the same old bifrost with the usual stuff from the 3.0.1 tarball and not my changes | 09:42 |
| thomas_oneill | is there an easier way to debug this without just debugging the actual image/build.py line by line lol or does anybody have any suggestions | 09:43 |
| *** blallau has joined #openstack-kolla | 09:44 | |
| *** masuberu has quit IRC | 09:44 | |
| *** duonghq has quit IRC | 09:53 | |
| *** mgoddard has joined #openstack-kolla | 09:53 | |
| *** athomas has quit IRC | 10:02 | |
| *** cuongnv has quit IRC | 10:04 | |
| *** pbourke has quit IRC | 10:07 | |
| *** pbourke has joined #openstack-kolla | 10:07 | |
| *** jmccarthy has joined #openstack-kolla | 10:09 | |
| *** athomas has joined #openstack-kolla | 10:09 | |
| *** manheim has quit IRC | 10:10 | |
| *** tovin07_ has quit IRC | 10:16 | |
| thomas_oneill | also could this be workflowed if nobody has any issues: https://review.openstack.org/#/c/443747/ | 10:29 |
| *** satyar has quit IRC | 10:30 | |
| thomas_oneill | egonzalez: Jeffrey4l__ zhubingbing_ ^^ sorry this has been hanging for many days nowe | 10:31 |
| Daviey_ | Hi, please could i get a second review on https://review.openstack.org/#/c/444771/ .. it has been sad idle for a bit, and pretty urgent | 10:34 |
| Daviey_ | thomas_oneill: How confident are you the "when" stanza is correct? https://review.openstack.org/#/c/443747/7/ansible/roles/neutron/handlers/main.yml | 10:35 |
| thomas_oneill | Daviey_: 56% confident. Why? | 10:36 |
| Daviey_ | hah | 10:38 |
| Daviey_ | Jus curious... it jumped out at me as something to check | 10:38 |
| Daviey_ | thomas_oneill: Also, where is neutron_bgp_router_id defined? | 10:38 |
| thomas_oneill | group_vars | 10:38 |
| Daviey_ | thomas_oneill: oh yes, missed that. Thanks | 10:39 |
| thomas_oneill | I did look at your change too, I have no understanding/experience of heka though, so cannot give review :p | 10:43 |
| Daviey_ | thomas_oneill: +W'd | 10:43 |
| thomas_oneill | oh ty :) | 10:43 |
| *** openstackgerrit has joined #openstack-kolla | 10:47 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Add neutron-bgp-dragent playbooks and sensible defaults. https://review.openstack.org/443747 | 10:47 |
| *** masuberu has joined #openstack-kolla | 10:48 | |
| *** mpansky has quit IRC | 10:52 | |
| *** matrohon has joined #openstack-kolla | 10:54 | |
| hawi | egonzalez: still the same issue | 10:57 |
| hawi | used pip install to have 4.0.0 version for kolla-ansible and reinstalled everything | 10:57 |
| egonzalez | hawi, images were rebuilt ? | 10:58 |
| *** masuberu has quit IRC | 11:00 | |
| hawi | egonzalez: you mean, also kolla-build to build new images in registry? | 11:00 |
| egonzalez | hawi, yep, the issue is in nova codebase, if the images used are master the bug still present | 11:01 |
| hawi | ok, will rebuild them also completely. i'm pretty sure that i used not master kolla to build them, but i'll rebuild them as well. | 11:03 |
| *** schwicht has joined #openstack-kolla | 11:04 | |
| hawi | egonzalez: how to ensure i'm building them for 4.0.0? | 11:04 |
| egonzalez | pip install kolla==4.0.0 | 11:05 |
| hawi | and the kolla-build -t source -b ubuntu --registry ip:port --push ? | 11:09 |
| *** mpansky has joined #openstack-kolla | 11:10 | |
| *** rwallner has joined #openstack-kolla | 11:10 | |
| *** manheim has joined #openstack-kolla | 11:11 | |
| egonzalez | hawi, yep | 11:12 |
| *** matrohon has quit IRC | 11:12 | |
| egonzalez | hawi, docker images are in docker hub, don't need to rebuild them, just push from there | 11:14 |
| *** rwellum has joined #openstack-kolla | 11:15 | |
| *** zhurong has joined #openstack-kolla | 11:17 | |
| *** matrohon has joined #openstack-kolla | 11:18 | |
| hawi | egonzalez: i had knowledge that for multinode i need to have my own registry. | 11:23 |
| egonzalez | hawi, if use public docker registry don't need your own | 11:24 |
| *** masuberu has joined #openstack-kolla | 11:26 | |
| *** mnasiadka has quit IRC | 11:32 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 11:34 | |
| *** Jeffrey4l__ has quit IRC | 11:36 | |
| *** zhurong has quit IRC | 11:37 | |
| openstackgerrit | Yu Chaochao proposed openstack/kolla master: Remove py34 settings from gating https://review.openstack.org/447247 | 11:56 |
| *** manheim has quit IRC | 11:59 | |
| *** manheim has joined #openstack-kolla | 11:59 | |
| sdake | morning folks | 12:03 |
| *** schwicht has quit IRC | 12:06 | |
| spsurya | morning sdake | 12:08 |
| *** dave-mccowan has joined #openstack-kolla | 12:08 | |
| *** nea1 has quit IRC | 12:09 | |
| *** masuberu has quit IRC | 12:11 | |
| *** rhallisey has joined #openstack-kolla | 12:11 | |
| jaosorior | mandre: any idea why those two are both trying to bind to 443? are those tests running with SSL? | 12:13 |
| *** mnasiadka has joined #openstack-kolla | 12:14 | |
| jaosorior | mandre: or, how can we reproduce gate-kolla-dsvm-deploy-centos-binary-centos-7-nv to log into the containers and see why something is trying to use SSL there :/ I can see why dashboard would be trying, but not why nova-placement API. | 12:14 |
| *** Jezekus has quit IRC | 12:19 | |
| *** schwicht has joined #openstack-kolla | 12:25 | |
| hawi | egonzalez: thanks, everything is fine now! | 12:27 |
| *** matrohon has quit IRC | 12:28 | |
| *** schwicht has quit IRC | 12:30 | |
| *** matrohon has joined #openstack-kolla | 12:31 | |
| mnasiadka | Is it only me, or horizon in ubuntu/binary build (stable/ocata) is missing proper path to images? | 12:35 |
| *** schwicht has joined #openstack-kolla | 12:36 | |
| *** schwicht has quit IRC | 12:37 | |
| openstackgerrit | Yu Chaochao proposed openstack/kolla master: Remove py34 settings from gating https://review.openstack.org/447247 | 12:38 |
| *** schwicht_at_work has joined #openstack-kolla | 12:38 | |
| mnasiadka | Yeah, it does | 12:42 |
| *** shardy is now known as shardy_lunch | 12:44 | |
| mnaser | morning everyone | 12:48 |
| spsurya | morning mnaser | 12:48 |
| spsurya | inc0: hi... | 12:49 |
| spsurya | it would be great if yo review this https://review.openstack.org/#/c/425446/ | 12:50 |
| mandre | jaosorior: I haven't looked into it why it tries to bind to port 443 yet | 12:50 |
| spsurya | as gate is green now | 12:50 |
| mandre | jaosorior: the CI job simply does a tools/deploy_aio.sh from kolla-ansible with the centos binary image it just built | 12:50 |
| hawi | i have generic question regarding Kolla. What should i do, when i change globals, in order to apply changes? is it kolla-ansible reconfigure or just deploy again, i guess upgrade is more for replacing images with newer ones? | 12:52 |
| *** schwicht_at_work has quit IRC | 12:52 | |
| mnaser | hawi i think reconfigure => deploy in later verisons of kolla-ansible | 12:53 |
| *** satyar has joined #openstack-kolla | 12:54 | |
| *** nea1 has joined #openstack-kolla | 12:56 | |
| *** srwilkers has joined #openstack-kolla | 12:59 | |
| *** zhubingbing_ has quit IRC | 12:59 | |
| *** mpansky has quit IRC | 13:00 | |
| pbourke | Jeffrey4l_: egonzalez: sdake: can one or more of you please revisit https://review.openstack.org/#/c/400187/ | 13:00 |
| pbourke | it's badly needed | 13:00 |
| *** mpansky has joined #openstack-kolla | 13:00 | |
| pbourke | reasons it makes sense are outlined in comments | 13:01 |
| Jeffrey4l_ | pbourke, i change my thought. use multi glance cause almost the same issue. i think vip is more better. and operators should sync ( or use share storage ) | 13:03 |
| hawi | mnaser: so, if i add enable_neutron_lbaas: "yes", then i need to do kolla-ansible -i inventory deploy, yes? | 13:04 |
| sdake | pbourke changed my vote - apologies for not getting back to that review | 13:04 |
| *** lamt has joined #openstack-kolla | 13:04 | |
| mnaser | that should do the job hawi | 13:04 |
| mnaser | granted i never tested it personally | 13:04 |
| sdake | pbourke i have a hard time actively keeping up with the amount of email i have passing through my inbox, and review email is tough to process | 13:05 |
| hawi | mnaser: i started with reconfigure right now (it sounded more like), but if it fails, then i'll go with deploy :) | 13:05 |
| sdake | pbourke feel free to ping if you need a specific review - thanks :) | 13:05 |
| openstackgerrit | Merged openstack/kolla-ansible master: Use kolla_internal_vip_address for glance_api servers https://review.openstack.org/400187 | 13:06 |
| hawi | mnaser: reconfigure did the job | 13:07 |
| *** lamt has quit IRC | 13:08 | |
| sdake | Jeffrey4l_ i saw in wednesday's meeting notes, there is activity underway to get images built in infra | 13:09 |
| sdake | Jeffrey4l_ i spent over 1 year working on that particular problem with little to no success | 13:09 |
| sdake | Jeffrey4l_ curious if you know the current status of that solution? | 13:10 |
| Jeffrey4l_ | no solution right now. | 13:10 |
| sdake | mnaser had mentioned he could talk to infra about how to make it so | 13:11 |
| Jeffrey4l_ | still need find a way to save hub.docker.c credential. | 13:11 |
| sdake | in the meantime, I'd suggest an automated workaround based upon portdirect 's repo and a volunteer effort | 13:11 |
| mnaser | sdake let me share a discussion i had with fungi from infra | 13:11 |
| sdake | Jeffrey4l_ yup the credentials and the build times are the two things that stopped my progress | 13:11 |
| sdake | mnaser thanks :) | 13:11 |
| Jeffrey4l_ | yep. i saw their talks. | 13:11 |
| *** goldyfruit has joined #openstack-kolla | 13:11 | |
| sdake | mnaser apparently I was asleep :) | 13:11 |
| mnaser | sdake Jeffrey4l_ http://eavesdrop.openstack.org/irclogs/%23openstack-infra/%23openstack-infra.2017-03-18.log.html#t2017-03-18T13:29:04 | 13:12 |
| *** lamt has joined #openstack-kolla | 13:12 | |
| Jeffrey4l_ | seems we need set up another slave progress, right? | 13:12 |
| mnaser | so we'd need a dedicated slave where credentials are store under root and the "job" file, ex: push image process happens under another user | 13:12 |
| Jeffrey4l_ | then which part is impossible now? like pypi case? | 13:13 |
| mnaser | so no one can craft a change which will let that password change | 13:13 |
| mnaser | oh i dont think its impossible, my suggestion was to have a push_images.sh which does the push after things are built | 13:13 |
| mnaser | and sudo access is locked down to that only | 13:14 |
| Jeffrey4l_ | before pushing the image, should download it from tarballs.o.o site? or build the image agains? | 13:14 |
| sdake | mnaser: for low-volume/low-data jobs which can run without root access and entirely from statically-defined scripts and preinstalled applications, yes we have some persistent slaves which can act as credential safes | 13:14 |
| sdake | credential safes | 13:15 |
| mnaser | yeah well we'd have a dedicated slave | 13:15 |
| mnaser | for docker stuff | 13:15 |
| mnaser | because of the volume and data involved | 13:15 |
| sdake | rockin | 13:15 |
| sdake | would the dedicated slave build the image - or pull it from tarballs.oo? | 13:15 |
| sdake | image/images | 13:16 |
| mnaser | the idea was: dedicated slave build image under "jenkins" user, sudo /usr/local/bin/push_images $TAG and that pushes it | 13:16 |
| mnaser | i mean the choice is up to us but i think its probably better to let it build it | 13:16 |
| *** lamt has quit IRC | 13:16 | |
| mnaser | rather than downloading 2.5gb and pushing it out again | 13:16 |
| Jeffrey4l_ | is it possible to add jenkins user to docker group? | 13:17 |
| mnaser | Jeffrey4l_ the idea is that this is a persistent slave that never gets deleted | 13:17 |
| sdake | the problem with that is that rebuilding does not actually always result in the same thing the gate tested | 13:17 |
| mnaser | so we configure it the way we want | 13:17 |
| mnaser | sdake thats a good point | 13:17 |
| sdake | mnaser a curl - untar - tar and push | 13:17 |
| mnaser | or we get a hosted registry backed by swift | 13:17 |
| sdake | that would be very easy to vet - much easier than infra going trough kolla-build.py :) | 13:17 |
| sdake | mnaser i'd like to keep things simple :) | 13:18 |
| mnaser | ideally we'd also ask for the slave to be hosted in the same region as tarballs.o. | 13:18 |
| sdake | right | 13:18 |
| Jeffrey4l_ | that will be great. | 13:18 |
| sdake | then the question becomes, how to release per tag? | 13:19 |
| mnaser | there is a release pipeline in zuul | 13:19 |
| sdake | spin in a script on a git clone to checkout the repo? | 13:19 |
| mnaser | which gets triggered when you tag | 13:19 |
| sdake | i know that | 13:19 |
| sdake | how does that interact with persistent slaves? | 13:19 |
| mnaser | id assume the persistent slave gets a job with all the ZUUL_* stuff | 13:20 |
| Jeffrey4l_ | current logical should works when we add the job into release pipeline. | 13:20 |
| *** lamt has joined #openstack-kolla | 13:20 | |
| mnaser | and we check if there is ZUUL_TAG and tag with it | 13:20 |
| mnaser | if not, we tag ZUUL_BRANCH | 13:20 |
| sdake | the persistent slave is always running - not run per job if I understand correctly | 13:20 |
| mnaser | yep, that's corret | 13:20 |
| sdake | is the idea tha tthe release pineline would ssh into the release slave? | 13:20 |
| mnaser | i think with some zuul magic they can say "always run this job on this slave" | 13:21 |
| mnaser | so the release pipeline job + gate/merge job always hits the same "uploading" slave | 13:21 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/446771 | 13:21 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/446772 | 13:21 |
| Jeffrey4l_ | release slave should be configured run at the end of the jobs. | 13:21 |
| *** m1dev has joined #openstack-kolla | 13:22 | |
| sdake | i think we need more information on the magic - to prepare a script for infra to review | 13:22 |
| sdake | also the persistent slaves need docker installed ;) | 13:22 |
| mnaser | yeah, im pretty sure they would get customized using system-config | 13:22 |
| Jeffrey4l_ | for this point, technically, is it possible to push a docker registry without docker engine? | 13:22 |
| *** eaguilar has joined #openstack-kolla | 13:23 | |
| mnaser | nope dont think so Jeffrey4l_ | 13:23 |
| sdake | it would be possible if you spent months writing a custom docker engine | 13:23 |
| Jeffrey4l_ | lol | 13:23 |
| sdake | so technically possible | 13:23 |
| sdake | feasible - not really :) | 13:23 |
| *** eaguilar has quit IRC | 13:24 | |
| mnaser | once people in the west coast start waking up today | 13:24 |
| Jeffrey4l_ | just a few docker registry api. not related docker engine, actually. | 13:24 |
| mnaser | ill try to poke some people if we can get ourself an initial machine | 13:24 |
| *** jtriley has joined #openstack-kolla | 13:24 | |
| mnaser | then we can start the process of getting docker installed on it with puppet using system-config, and writing out a very simple script to push whats in the registry | 13:25 |
| Jeffrey4l_ | mnaser, do u have any idea how to test the release pipeline? | 13:25 |
| mnaser | oh also zuulv3 apparently has in-repo secrets | 13:25 |
| *** lamt has quit IRC | 13:25 | |
| mnaser | so that would be a nice thing to have | 13:25 |
| Jeffrey4l_ | the issue is: when zuulv3 will be landed? | 13:25 |
| mnaser | good question | 13:25 |
| mnaser | /shrug | 13:25 |
| mnaser | Jeffrey4l_ is that necessary? i mean whatever is tagged should already have passed tests | 13:25 |
| mnaser | in other cool news we migrated our nova, neutron, cinder and glance control plane to kolla over the weekend | 13:26 |
| mnaser | so woo | 13:26 |
| Jeffrey4l_ | just wanna to make sure it works. and we need push tags registry tarballs to tarballs.o.o site. | 13:26 |
| Jeffrey4l_ | that's cool | 13:26 |
| Jeffrey4l_ | need add build/push image jobs into release pipeline. | 13:27 |
| portdirect | mornin' o/ | 13:27 |
| Jeffrey4l_ | morning | 13:28 |
| *** shardy_lunch is now known as shardy | 13:28 | |
| Jeffrey4l_ | hrm. checked the code. simple add the build/push images job into release pipeline should work. | 13:30 |
| hawi | question regarding neutron: i would like to add provider networks to ml2_conf.ini, for vlan and flat. what is the best place to put it? i have ceph.conf in /etc/kolla/config, should i put ml2_conf.ini there as well? | 13:33 |
| sdake | mnaser nice! | 13:34 |
| sdake | ok - re zuul v3 | 13:34 |
| sdake | my current speculation not based upon any factual data is as follows | 13:34 |
| *** erhudy has joined #openstack-kolla | 13:34 | |
| sdake | mordred had indicated in his last communication to the mailing list that he will provide more updates in the "next few months" | 13:34 |
| sdake | or something to that effect | 13:35 |
| sdake | Jeffrey4l_ the earliest I htink we can expect zuul v3 is June based upon the word few (3 or more) in that statement | 13:35 |
| Jeffrey4l_ | roger. | 13:36 |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla master: Added snap-telemetry container https://review.openstack.org/446164 | 13:36 |
| sdake | Jeffrey4l_ in speaking with the zuulv3 team, they have not disclosed their release timelines at all | 13:37 |
| sdake | Jeffrey4l_ I htink they dont want to commit and then miss - which is reasonable | 13:37 |
| sdake | we struggle ever ycycle to release on time :) | 13:38 |
| sdake | mnaser re your migration - how did you go about it? | 13:39 |
| sdake | mnaser a brownfield migration doc would be fantastic | 13:39 |
| sdake | mnaser atleast some tribal knowledge on how it was done would also be useful | 13:39 |
| mnaser | sdake: start up mariadb using kolla, start up rabbitmq using kolla, switch all data plan and old control plane to new rabbitmq, create db in new mariadb, migrate db, restart control plane services with db hosted on kolla, match configuration from old infra with kolla, deploy $service, now we have 2 control planes that are identical, test new control plane (we had the luxury of haproxy running before), when all is a-ok, turn off old control | 13:40 |
| mnaser | plane | 13:40 |
| sdake | mnaser note - I dont know how to write puppet code - nor do many people on the team | 13:40 |
| mnaser | sdake thats okay, i can help doing this. our infra ran extensively on puppet for quite sometime | 13:41 |
| egonzalez | hi guys, any idea about libvirtError: Activation of org.freedesktop.machine1 timed out | 13:41 |
| egonzalez | while creating instances | 13:41 |
| nea1 | Hi, where do I find the logs of the services? | 13:42 |
| egonzalez | it was working before | 13:42 |
| *** zhubingbing_ has joined #openstack-kolla | 13:42 | |
| sdake | egonzalez groan - never seen that one | 13:42 |
| sdake | egonzalez that would imply a communication error to systemd | 13:42 |
| sdake | egonzalez was libvirt possibly updated? | 13:42 |
| sdake | egonzalez note dbus is used to communicate to systemd from libvirt | 13:43 |
| sdake | egonzalez those are some components to look at - on your host or in the container | 13:43 |
| egonzalez | sdake, nope, no change | 13:43 |
| sdake | egonzalez probably something changed :) | 13:43 |
| sdake | egonzalez did you update your host at all? | 13:44 |
| sdake | mnaser your running mariadb in containers then? | 13:44 |
| mnaser | sdake yeah, its running as part of it all now | 13:45 |
| mnaser | as part of the kolla deployment | 13:45 |
| egonzalez | sdake, nope is in the same state as when was deployed | 13:45 |
| sdake | mnaser running haproxy? | 13:45 |
| mnaser | sdake yeah, we're still relying on our old haproxy (we used to run keepalived+haproxy) but we're soon going to switch to the kolla haproxy | 13:46 |
| egonzalez | sdake, dbus and machine8 services are in error state, but those services are static and activated from libivrt calls IIRC | 13:46 |
| mnaser | both are running at once | 13:46 |
| sdake | mnaser i mean is enable_haproxy: yes set? | 13:46 |
| mnaser | yep | 13:46 |
| sdake | nice that is almost a full migration | 13:46 |
| sdake | also the data plane? | 13:46 |
| pbourke | thanks for that sdake & Jeffrey4l_ :) | 13:47 |
| sdake | note in this instance i mean things like neutron | 13:47 |
| mnaser | sdake data plane is a wip, our next thing to tackle is migrating neutron control plane | 13:49 |
| mnaser | we purposefully kept the same host names so that we can shutdown the service on $old_controller_n and start it up on $new_controller_n | 13:49 |
| mnaser | and we dont have to move ports around, it'll just detect itself | 13:49 |
| sdake | egonzalez where are the logs stored now a adays for kolla-ansible for nea1 ? | 13:49 |
| *** rhallisey has quit IRC | 13:50 | |
| egonzalez | /var/lib/docker/volumes/kolla_logs as always ;) | 13:51 |
| zhubingbing_ | hey guys | 13:52 |
| *** crushil has joined #openstack-kolla | 13:52 | |
| zhubingbing_ | ；） | 13:53 |
| mnaser | o/ | 13:53 |
| sdake | sup zhubingbing_ | 13:53 |
| zhubingbing_ | sup sdake | 13:53 |
| sdake | zhubingbing_ any progress on the deployment gerrit doc? | 13:53 |
| sdake | zhubingbing_ i crashed last night | 13:53 |
| zhubingbing_ | i look it | 13:53 |
| zhubingbing_ | ；） | 13:54 |
| sdake | zhubingbing_ is it your end of day or beginning | 13:54 |
| sdake | I think its your end - although hard to tell :) | 13:54 |
| zhubingbing_ | i will beaginng to write | 13:54 |
| *** lamt has joined #openstack-kolla | 13:54 | |
| sdake | https://review.openstack.org/#/c/447356/ | 13:55 |
| zhubingbing_ | https://review.openstack.org/#/c/447356/ | 13:55 |
| sdake | yup :) | 13:55 |
| zhubingbing_ | i look it ；） | 13:55 |
| sdake | yes please review | 13:55 |
| sdake | eveyrone that contributed to the etherpad - please review - lets remove all the extra crap | 13:55 |
| zhubingbing_ | ok | 13:55 |
| sdake | qwang are you about? | 13:55 |
| sdake | rwellum are you about? | 13:56 |
| *** rhallisey has joined #openstack-kolla | 13:57 | |
| zhubingbing_ | sdake can i modify this patch ? | 13:57 |
| sdake | rwellum I determined why you were struggling so much with the deployment - the etherpad was vandalized interrupting eveyrone's work: https://review.openstack.org/#/c/447356/1..2 | 13:57 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Get correct tag name when job is running in release pipeline https://review.openstack.org/447507 | 13:58 |
| nea1 | sdake: egonzalez thanks | 13:58 |
| sdake | zhubingbing_ I think a better way of doing the work is to review it although if you need to include big chunks or want to convert to rst, that wfm | 13:58 |
| *** lamt has quit IRC | 13:58 | |
| zhubingbing_ | sure | 13:58 |
| zhubingbing_ | rouger | 13:58 |
| zhubingbing_ | roger | 13:58 |
| sdake | zhubingbing_ what did you want to do exactly | 13:58 |
| sdake | zhubingbing_ I am planning to spend the day on working on this doc - need some reviews from the ocntribs | 13:59 |
| sdake | zhubingbing_ how did you want to modify it? | 13:59 |
| *** japestinho has quit IRC | 14:00 | |
| nea1 | how do I at best see if the node is done with the reboot and all services are up | 14:00 |
| zhubingbing_ | I mean I'm afraid you do not have time to change | 14:00 |
| zhubingbing_ | ；） | 14:00 |
| sdake | zhubingbing_ iv'e committed to working on this today | 14:00 |
| zhubingbing_ | nice | 14:00 |
| sdake | zhubingbing_ lets take my commitments one day at a time this week - i've been on work travel for 2 weeks so need to catch up a bit | 14:00 |
| sdake | but feel I can commit atleast today to doing the rework on thedoc | 14:01 |
| *** lamt has joined #openstack-kolla | 14:01 | |
| zhubingbing_ | ；） | 14:01 |
| sdake | need reviews :) | 14:02 |
| *** jmccarthy has left #openstack-kolla | 14:02 | |
| *** jtriley_ has joined #openstack-kolla | 14:03 | |
| zhubingbing_ | doning | 14:03 |
| zhubingbing_ | sdake | 14:04 |
| zhubingbing_ | https://review.openstack.org/#/c/439740/ | 14:04 |
| zhubingbing_ | need fix it ;) | 14:04 |
| sdake | zhubingbing_ if you want to work on that - that would be great :) | 14:05 |
| sdake | zhubingbing_ python is your native language :) | 14:05 |
| *** jtriley has quit IRC | 14:05 | |
| sdake | zhubingbing_ and bash and ansible etc ;) | 14:05 |
| *** lamt has quit IRC | 14:05 | |
| pomac | egonzalez: recheck? ;) | 14:06 |
| sdake | sbezverk are you about? | 14:06 |
| egonzalez | pomac, about? | 14:07 |
| pomac | egonzalez: https://review.openstack.org/#/c/439468/ - your comment | 14:07 |
| zhubingbing_ | ;) | 14:07 |
| zhubingbing_ | ok | 14:07 |
| egonzalez | pomac, is to gates to run again to see if all is green | 14:08 |
| pomac | egonzalez: ah, oki | 14:08 |
| egonzalez | pomac, the same issue I was having hehe, thanks for it ;) | 14:08 |
| pomac | egonzalez: no problem ;) | 14:08 |
| sdake | zhubingbing_ do you want some guidance on https://review.openstack.org/#/c/439740/ - as in where to get started? | 14:09 |
| *** lamt has joined #openstack-kolla | 14:09 | |
| sdake | zhubingbing_ the review is there with no context on why the gate is busted | 14:10 |
| zhubingbing_ | i think it can;t find kolla-ansible patch | 14:11 |
| *** lamt has quit IRC | 14:12 | |
| sdake | zhubingbing_ right - the root of the problem is here: | 14:13 |
| zhubingbing_ | yes | 14:14 |
| mnaser | sdake (and anyone else), would you be interested to be listed as a volunteer under the dockerhub image publishing spec that im writing up for infra? | 14:14 |
| mnaser | inc0 perhaps? | 14:14 |
| *** skramaja has quit IRC | 14:14 | |
| *** schwicht has joined #openstack-kolla | 14:15 | |
| *** mpansky has quit IRC | 14:16 | |
| sdake | zhubingbing_ https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate_common.sh#L69-L71 | 14:16 |
| sdake | mnaser i'm not sure i would be all that valuable for that effort tbh - inc0 did most of the work on the getting the registry into tarballs.oo | 14:17 |
| zhubingbing_ | yes | 14:17 |
| zhubingbing_ | http://paste.openstack.org/show/601021/ | 14:17 |
| zhubingbing_ | ；） | 14:17 |
| sdake | mnaser kfox1111 also has deep context in tarballs.oo work | 14:17 |
| sdake | zhubingbing_ that needs to be changed to sudo pip install . | 14:18 |
| *** lamt has joined #openstack-kolla | 14:18 | |
| sdake | zhubingbing_ the symlinks in the gate cause that patch to fail | 14:18 |
| sdake | zhubingbing_ the solution is to install via pip - as our deployment docs recommend | 14:18 |
| sdake | mnaser i can provide credentials to the dockerhub account to those that need it | 14:18 |
| sdake | mnaser inc0 has credentials, however, we probably want a test credentials account | 14:19 |
| sdake | mnaser perhaps even a new org, as deleting images in docker hub is a huge pITA - requires clicking 200+ times | 14:19 |
| zhubingbing_ | sdake we should pip install kolla-ansbile/ kolla/ kolla-kubernetes/ | 14:19 |
| zhubingbing_ | don't use pip install . | 14:19 |
| sdake | mnaser the new org would be for testing - so we can abandon it easily | 14:19 |
| zhubingbing_ | ;) | 14:19 |
| nea1 | "Failed to start libvirt guest" http://termbin.com/2kjn anyone got an idea what could be wrong? | 14:20 |
| sdake | zhubingbing_ right! | 14:20 |
| sdake | zhubingbing_ i'll update the docs accordingly | 14:20 |
| zhubingbing_ | https://review.openstack.org/#/c/447356/5 | 14:20 |
| zhubingbing_ | i continue review it | 14:20 |
| *** lamt has quit IRC | 14:21 | |
| sdake | mnaser i can help guide a bit - but can't do any implementation work as I don't know how beyond the docker commands and creds and whatnot | 14:23 |
| *** lamt has joined #openstack-kolla | 14:24 | |
| *** lamt has quit IRC | 14:24 | |
| sdake | zhubingbing_ by the end of the week, i'd prefer to have our gate in sync with our deploy docs | 14:27 |
| zhubingbing_ | emm | 14:28 |
| zhubingbing_ | yes | 14:28 |
| sdake | zhubingbing_ this means merging https://review.openstack.org/#/c/439740/ which I am not sure if i can get to | 14:28 |
| sdake | zhubingbing_ asking someone to cherrypick a patch in a deployment doc is kind of - well not ideal :) | 14:28 |
| sdake | zhubingbing_ we can't merge https://review.openstack.org/#/c/439740/ if the gates are red | 14:29 |
| zhubingbing_ | so we should modify gate ? | 14:29 |
| sdake | zhubingbing_ yes - by removing symlinks and changing to: | 14:30 |
| sdake | [07:19:50] <zhubingbing_>sdake we should pip install kolla-ansbile/ kolla/ kolla-kubernetes/ | 14:30 |
| zhubingbing_ | hmm i try do it ；） | 14:30 |
| sdake | zhubingbing_ I suspect this will entail alot of rework of the gate scripts | 14:30 |
| zhubingbing_ | yes | 14:31 |
| sdake | finally ejected lastpass from my life | 14:33 |
| sdake | portdirect ^^ | 14:34 |
| mnaser | sdake inc0 anyone interested in dockerhub publishing of images -- please review kindly - https://review.openstack.org/#/c/447524/ | 14:34 |
| zhubingbing_ | kolla-k8s have supporting placement api in master | 14:35 |
| zhubingbing_ | https://github.com/openstack/kolla-kubernetes/commit/b89e1136aa082e47d29a23ee542a89dc7b703efa | 14:36 |
| spsurya | sdake: would be great, if you take a look on the drafted BP for kolla-k8s. https://blueprints.launchpad.net/kolla-kubernetes/+spec/stabilization-unittest-coverage-improvement | 14:36 |
| spsurya | sdake: provide your views on this and approve the same, if it feasible. | 14:37 |
| *** mpansky has joined #openstack-kolla | 14:38 | |
| sdake | mnaser reviewed | 14:39 |
| *** duke__ has quit IRC | 14:41 | |
| zhubingbing_ | sup kfox | 14:43 |
| sdake | spsurya needs an assignee to be approved | 14:43 |
| zhubingbing_ | sup kfox1111 | 14:43 |
| sdake | spsurya however will approve once there is an assignee :) | 14:43 |
| sdake | spsurya note i moved the work items to the correct location in launchpad | 14:43 |
| zhubingbing_ | sup sbezverrk | 14:44 |
| spsurya | sdake: thanks | 14:44 |
| sdake | sbezverk is the nick zhubingbing_ :) | 14:44 |
| zhubingbing_ | our kolla-k8s gate is deploy in 2.0.2 | 14:44 |
| zhubingbing_ | ；） | 14:44 |
| sdake | zhubingbing_ the gate actually deploys 2.0.2 and 3.0.2 | 14:44 |
| zhubingbing_ | ok | 14:45 |
| sdake | zhubingbing_ i'd leave that alone during the gate rework - will probably just confuse the patch work on https://review.openstack.org/#/c/439740/ | 14:45 |
| zhubingbing_ | so we install kolla-ansible ocata version ? | 14:45 |
| sdake | zhubingbing_ yup | 14:45 |
| *** saneax is now known as saneax-_-|AFK | 14:46 | |
| sdake | zhubingbing_ actually master | 14:46 |
| *** lamt has joined #openstack-kolla | 14:46 | |
| sdake | kolla-ansible is needed for genconfig | 14:46 |
| sdake | we want to install mater of kolla-ansible to get the latest fixes for generating configuration | 14:46 |
| sdake | zhubingbing_ I know this creates some chaos with 2.0.2 images | 14:46 |
| zhubingbing_ | so https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate_common.sh#L80L92 | 14:46 |
| zhubingbing_ | OK | 14:47 |
| sdake | zhubingbing_ after we finish this doc, we are ejecting kolla-ansible entirely from the code base | 14:47 |
| sdake | zhubingbing_ that is the next blueprint of essential nature | 14:47 |
| zhubingbing_ | yes | 14:48 |
| sdake | doc essential blueprint, external-configmaps essential blueprint | 14:48 |
| spsurya | sdake: After kolla-k8s dev env setup, i will assign it to myself and will start adding the missing one. | 14:48 |
| spsurya | sdake: thanks for placing it to right place | 14:48 |
| sdake | spsurya cool - you can assign yourself now and I can approve the blueprint then or ping anyone in the kolla-kubernetes-drivers group later once your ready to approve the blueprint | 14:49 |
| mnaser | sdake rewrite that paragraph | 14:49 |
| mnaser | rewrote* | 14:49 |
| *** jtriley_ has quit IRC | 14:51 | |
| sdake | mnaser LGTM | 14:52 |
| mnaser | cool, ill wait for a bit more info from folks here and then throw it to -infra | 14:52 |
| *** lamt has quit IRC | 14:54 | |
| *** lamt has joined #openstack-kolla | 14:56 | |
| *** lamt has quit IRC | 14:57 | |
| *** jtriley has joined #openstack-kolla | 14:59 | |
| jaosorior | is this the right place to ask kolla-ansible questions? | 15:01 |
| jaosorior | I had done a deployment in my local machine with just mariadb and keystone. Decided to do a kolla-ansible destroy; rebuilt my images, and now that I try again, I get the following error: https://paste.fedoraproject.org/paste/tp1-k5m4E8J6TMSlAxgNwV5M1UNdIGYhyRLivL9gydE= which is pretty strange, since I did build the kolla-toolbox container image :/ | 15:04 |
| *** lamt has joined #openstack-kolla | 15:04 | |
| *** mnasiadka has quit IRC | 15:05 | |
| mnaser | 2 easy ones missing a +w https://review.openstack.org/#/c/446701/ and https://review.openstack.org/#/c/447226/ | 15:05 |
| *** lamt has quit IRC | 15:06 | |
| spsurya | sdake: ok | 15:07 |
| sdake | jaosorior indeed it is the right place - wrong time possibly :) | 15:08 |
| zhubingbing_ | sup <egonzalez | 15:08 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible master: Fix can't find /usr/lib/libCryptoki2_64.so in barbican https://review.openstack.org/444561 | 15:08 |
| zhubingbing_ | sup egonzalez | 15:08 |
| egonzalez | zhubingbing_, hey | 15:08 |
| zhubingbing_ | https://review.openstack.org/444561 | 15:09 |
| sdake | jaosorior can you run docker images | fpaste | 15:09 |
| zhubingbing_ | ;) | 15:09 |
| zhubingbing_ | can u look it ? | 15:09 |
| egonzalez | jaosorior, also `docker volume ls` with no kolla image running | 15:09 |
| sdake | jaosorior and a rpm -qi docker-engine | fpaste | 15:10 |
| zhubingbing_ | i am teseting magnum | 15:10 |
| jaosorior | sdake: docker images -> https://paste.fedoraproject.org/paste/TyRm31dUplOO0uINxZwka15M1UNdIGYhyRLivL9gydE=/ | 15:11 |
| sdake | jaosorior image is built correctly | 15:12 |
| egonzalez | zhubingbing_, there is not any 32-byte base64 passwd generator at generate passwords script | 15:13 |
| zhubingbing_ | ;) | 15:13 |
| zhubingbing_ | so | 15:13 |
| jaosorior | sdake: docker engine info https://paste.fedoraproject.org/paste/t-6HSdsRjUT4zqaJ7T3HBl5M1UNdIGYhyRLivL9gydE= | 15:13 |
| zhubingbing_ | should i how to modify | 15:13 |
| zhubingbing_ | ;) | 15:13 |
| *** mnasiadka has joined #openstack-kolla | 15:13 | |
| egonzalez | zhubingbing_, by default will create a uuid passwd, barbican expect a 32byte base64, tho | 15:13 |
| jaosorior | egonzalez: docker volume ls -> https://paste.fedoraproject.org/paste/To7YaWXima9G83oOztJdgF5M1UNdIGYhyRLivL9gydE=/ | 15:15 |
| mnaser | pbourke https://review.openstack.org/#/c/447226/ goof'ed on the original change, this one is tested in prod :( | 15:15 |
| zhubingbing_ | hmm we don't add it kek = '{{ barbican_crypto_password }} | 15:15 |
| sdake | Daviey_ can you review https://review.openstack.org/#/c/447226/1 and validate it is compliant with the stable backports policy? | 15:15 |
| zhubingbing_ | right ? | 15:16 |
| *** saneax-_-|AFK has quit IRC | 15:16 | |
| egonzalez | jaosorior, hrm, i think the issue is with devicemapper and docker 1.13.6, for some reason when tries to restart the container devicemapper still blocking devices and fail to restart but removes the container | 15:17 |
| *** mnasiadka has quit IRC | 15:17 | |
| egonzalez | jaosorior, if try again with no kolla-toolbox container running should work fine | 15:17 |
| sdake | egonzalez you mean 1.12.6 in fedora? | 15:18 |
| jaosorior | egonzalez: right, so currently kolla-toolbox is not running. And yet I still get that error :/ | 15:18 |
| sdake | jaosorior there are lots o problems with devicemapper as a storage backend | 15:19 |
| sdake | jaosorior i'd recommend btrfs - this is somewhere in our documentation - moment let me find it | 15:19 |
| egonzalez | sdake, didnt check the version paste :( | 15:19 |
| mnaser | if you use devicemapper | 15:19 |
| mnaser | you have to set it up properly | 15:19 |
| mnaser | http://www.projectatomic.io/blog/2015/06/notes-on-fedora-centos-and-docker-storage-drivers/ | 15:20 |
| mnaser | xfs+overlayfs will cause problems (learned that one myself) | 15:20 |
| jaosorior | I was just using the defaults. If there is something I should be using instead, I can change it. | 15:20 |
| sdake | mnaser are you using lvm in your deployment? | 15:21 |
| mnaser | docker info should tell you which drive you're using | 15:21 |
| *** crushil has quit IRC | 15:21 | |
| *** lamt has joined #openstack-kolla | 15:22 | |
| mnaser | sdake https://docs.docker.com/engine/userguide/storagedriver/device-mapper-driver/#configure-docker-with-devicemapper > "Configure direct-lvm mode for production" | 15:23 |
| *** lamt has quit IRC | 15:23 | |
| nea1 | how do I disable keepalived / haproxy? (for the all-in-one setup) | 15:25 |
| mnaser | enable_haproxy: false | 15:25 |
| mnaser | in globals.yml | 15:25 |
| jaosorior | mnaser: thanks for the link; I'm attempting to use overlayfs instead | 15:26 |
| mnaser | jaosorior overlayfs was broken for me | 15:27 |
| mnaser | mariadb fails to install | 15:27 |
| mnaser | wait actually | 15:27 |
| mnaser | the problems were only happening in build time | 15:27 |
| mnaser | so maybe you'll be okay in deploy | 15:27 |
| jaosorior | oh | 15:27 |
| jaosorior | was attempting to rebuild the images | 15:27 |
| jaosorior | well, lets see how that goes | 15:27 |
| kfox1111 | morning. | 15:28 |
| *** manheim has quit IRC | 15:28 | |
| mnaser | o/ kfox1111 | 15:29 |
| *** mnasiadka has joined #openstack-kolla | 15:29 | |
| mnaser | kfox1111 love to hear your input on this - https://review.openstack.org/#/c/447524/ :) | 15:29 |
| kfox1111 | hey mnaser. | 15:32 |
| kfox1111 | sure. sec | 15:32 |
| nea1 | mnaser: strange, why isn't it in there (commented out like with all other things that are available?) | 15:33 |
| *** mnasiadka has quit IRC | 15:34 | |
| nea1 | but thanks :) | 15:34 |
| *** hrw has quit IRC | 15:35 | |
| *** lamt has joined #openstack-kolla | 15:35 | |
| kfox1111 | mnaser: looks good to me. | 15:35 |
| *** hrw has joined #openstack-kolla | 15:36 | |
| *** mnasiadka has joined #openstack-kolla | 15:37 | |
| jaosorior | mnaser: had the same issue with overlayfs | 15:41 |
| *** mnasiadka has quit IRC | 15:42 | |
| thomas_oneill | what's the process for a bug in kolla that ends up being a bug in another project? | 15:43 |
| thomas_oneill | well not even a 'bug', just something that needs changing... Like if I need to submit a PS to bifrost to change some stuff, can we still do Depends-On for stuff in other projects? | 15:43 |
| thomas_oneill | ah yeah surely we can because I could do that from kolla-ansible to kolla, so yeah, nevermind, ignore me | 15:43 |
| *** lamt has quit IRC | 15:44 | |
| *** jaosorior has quit IRC | 15:45 | |
| *** matrohon has quit IRC | 15:46 | |
| *** mpansky has quit IRC | 15:46 | |
| *** lamt has joined #openstack-kolla | 15:47 | |
| *** mnasiadka has joined #openstack-kolla | 15:47 | |
| *** lamt has quit IRC | 15:50 | |
| *** matrohon has joined #openstack-kolla | 15:50 | |
| *** mnasiadka has quit IRC | 15:52 | |
| *** lamt has joined #openstack-kolla | 15:52 | |
| *** blallau has quit IRC | 15:54 | |
| *** sayantani01 has joined #openstack-kolla | 15:55 | |
| *** mpansky has joined #openstack-kolla | 15:55 | |
| *** mnasiadka has joined #openstack-kolla | 15:57 | |
| *** jaosorior has joined #openstack-kolla | 15:59 | |
| *** mnasiadka has quit IRC | 16:01 | |
| *** iceyao has joined #openstack-kolla | 16:02 | |
| nea1 | mnaser: "TASK [haproxy : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node] ***" is failing, after I disabled haproxy in the globals.yml | 16:07 |
| nea1 | while running the prechecks | 16:07 |
| *** manheim has joined #openstack-kolla | 16:08 | |
| thomas_oneill | nea1: can you ping the address you've specified for those vars? | 16:09 |
| *** rmart04 has quit IRC | 16:09 | |
| jaosorior | sdake: what was the documentation for the btrfs configuration? Still haven't gotten my setup to work :/ | 16:10 |
| *** tonanhngo has joined #openstack-kolla | 16:10 | |
| thomas_oneill | oh sorry I re-read what you put | 16:10 |
| *** crushil has joined #openstack-kolla | 16:10 | |
| thomas_oneill | nea1: you're using the kolla-ansible binary, right? do a: which kolla-ansible | 16:10 |
| thomas_oneill | nea1: if you want to run the kolla-ansible using the playbooks in your working directory, you need to run tools/kolla-ansible | 16:11 |
| *** jmccarthy has joined #openstack-kolla | 16:12 | |
| *** iceyao has quit IRC | 16:12 | |
| thomas_oneill | nea1: but yeah you need to be aware of how you've installed these binaries and what code they are actually running | 16:12 |
| sdake | ja09:12:29] <sdake>jaosorior did you change your docker config to use direct-lvm? | 16:13 |
| sdake | [09:12:39] <sdake>jaosorior if not, i'd start there first | 16:13 |
| *** lamt has quit IRC | 16:13 | |
| jaosorior | sdake: I didn't. I went for the overlayfs option | 16:13 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible master: Fix can't find /usr/lib/libCryptoki2_64.so in barbican https://review.openstack.org/444561 | 16:14 |
| *** lamt has joined #openstack-kolla | 16:14 | |
| sdake | jaosorior note if you change the docker engine storage driver, you need to sudo systemctl stop docker; rm -rf /var/lib/docker; sudo systemctl start docker- note this will erase all of your built images and you will have to build again | 16:14 |
| jaosorior | sdake: I did | 16:15 |
| jaosorior | yep | 16:15 |
| jaosorior | removed everything | 16:15 |
| jaosorior | rebuilt my images | 16:15 |
| jaosorior | and still got the same error | 16:15 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible master: Fix can't find /usr/lib/libCryptoki2_64.so in barbican https://review.openstack.org/444561 | 16:16 |
| sdake | jaosorior run docker info | 16:16 |
| sdake | -> fpaste | 16:16 |
| jaosorior | sdake: Storage Driver: overlay | 16:16 |
| *** saneax-_-|AFK has joined #openstack-kolla | 16:17 | |
| sdake | did yo ustop docker prior to the rm -rf? | 16:17 |
| jaosorior | sdake: https://paste.fedoraproject.org/paste/DAkn3Oc3jUtXTSE1zMUu015M1UNdIGYhyRLivL9gydE=/ | 16:17 |
| jaosorior | sdake: yes | 16:17 |
| sdake | jaosorior note - fedora 25 is an unsupported host - kolla may or may not work - need more context however on your failure | 16:18 |
| sdake | jaosorior you were able to get through one deployment correct? | 16:19 |
| jaosorior | yep | 16:19 |
| *** lamt has quit IRC | 16:19 | |
| jaosorior | the first one | 16:19 |
| sdake | you destroyed? | 16:19 |
| jaosorior | I did | 16:19 |
| sdake | you deployed and that failed? | 16:19 |
| jaosorior | correct | 16:19 |
| jaosorior | If I may add, I did a destroy including the images | 16:19 |
| sdake | you changed the docker storage driver as per recommendations? | 16:19 |
| sdake | roger | 16:19 |
| sdake | at any point did you reboot your host | 16:20 |
| jaosorior | sdake: yep; used to be devicemapper, and I switched it to overlayfs | 16:20 |
| jaosorior | sdake: haven't | 16:20 |
| sdake | ok - i'd give that a spin as painful as that may be | 16:20 |
| sdake | then try deploy destroy deploy with overlayfs | 16:20 |
| sdake | i speculate that the kernel has some lost references on /var/lib/docker - devicemapper does that | 16:20 |
| sdake | when switching from devicemapper a reboot is always required | 16:21 |
| sdake | i'm surprised you were able to rm -rf /var/lib/docker at all without errors | 16:21 |
| jaosorior | I'll try that | 16:21 |
| jaosorior | thanks | 16:21 |
| hrw | kolla on f25... wfm | 16:23 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible master: Fix can't find /usr/lib/libCryptoki2_64.so in barbican https://review.openstack.org/444561 | 16:23 |
| *** pcaruana has quit IRC | 16:27 | |
| *** lamt has joined #openstack-kolla | 16:28 | |
| *** jrich523 has quit IRC | 16:29 | |
| nea1 | thomas_oneill: When running an All-In-One | 16:29 |
| nea1 | 28 # without haproxy and keepalived, this should be the first IP on your | 16:29 |
| nea1 | 29 # 'network_interface' as set in the Networking section below. | 16:29 |
| nea1 | so it was the IP from my first interface | 16:30 |
| *** rwellum has quit IRC | 16:30 | |
| thomas_oneill | nea1: yeah sorry I read what you put too fast and then corrected myself. It shouldn't have been running those checks if you disabled haproxy, so you must be running some installed binary that isn't looking at the playbooks/vars that you've amended. | 16:32 |
| thomas_oneill | nea1: if you pip install kolla-ansible or pip install . in the kolla-ansible directory, it will install all those playbooks and vars hard-coded into your python environment | 16:33 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 16:34 |
| openstackgerrit | Merged openstack/kolla stable/newton: Turn on SSH for Keystone for all distros. https://review.openstack.org/447226 | 16:35 |
| sdake | hrw good to hear :) | 16:37 |
| hrw | sdake: 1094 images built. centos, debian, ubuntu. binary and source | 16:37 |
| *** zhubingbing_ has quit IRC | 16:38 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Enable heat-api proxy header parsing https://review.openstack.org/439468 | 16:38 |
| *** lamt has quit IRC | 16:40 | |
| sdake | https://github.com/openstack/kolla - 42 releases | 16:41 |
| sdake | 3 meals a day for 2 weeks - thats the diet i'm on :) | 16:41 |
| *** lamt has joined #openstack-kolla | 16:41 | |
| hrw | ;) | 16:43 |
| hrw | btw - how to list tags on image present in repo? | 16:43 |
| *** lamt has quit IRC | 16:43 | |
| hrw | instead of 'docker pull --all imagename' and then looking at output | 16:43 |
| *** lamt has joined #openstack-kolla | 16:48 | |
| nea1 | thomas_oneill: I did pip install kolla-ansible | 16:48 |
| *** bmace has quit IRC | 16:48 | |
| *** MasterOfBugs has joined #openstack-kolla | 16:49 | |
| nea1 | but I now solved it differently, I created a bridge with I'm using as network device so I can have a nated network there | 16:49 |
| *** bmace has joined #openstack-kolla | 16:49 | |
| *** lamt has quit IRC | 16:49 | |
| *** jaosorior has quit IRC | 16:50 | |
| thomas_oneill | nea1: well that's great but it's still definitely worth being aware of what code you're actually running ;) | 16:51 |
| *** jaosorior has joined #openstack-kolla | 16:52 | |
| nea1 | thomas_oneill: well I'll have a look at it later, I'm kind of lacking time, and if it now builds then it'll be fine for not | 16:53 |
| *** Serlex has quit IRC | 16:55 | |
| *** manheim has quit IRC | 16:56 | |
| nea1 | ever tried using a bridge as network_interface which contains a "-" e.g. "br-mgmt"? | 16:57 |
| thomas_oneill | extremely unlikely that the - will make any difference | 16:58 |
| srwilkers | morning | 16:58 |
| thomas_oneill | the fact that a bridge is a different kind of network interface than a regular physical network interface is far more likely to be the problem with that nea1 | 16:58 |
| jascott1 | mornin' srwilkers | 16:58 |
| nea1 | ansible facts will have it as "ansible_br_mgmt" not "ansible_br-mgmt" https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L8 | 16:58 |
| thomas_oneill | okay but is it moaning about something? | 16:59 |
| nea1 | thomas_oneill: with the neutron network yes for sure, but is it trying to bridge network_interface as well? I think it only adds an IP to it (the vip IP) | 16:59 |
| *** unicell1 has quit IRC | 17:00 | |
| nea1 | thomas_oneill: yep, as it can't find it in the dict - ['ansible_' + api_interface | regex_replace('-', '_')] fixes it | 17:00 |
| *** egonzalez has quit IRC | 17:00 | |
| *** zhubingbing has joined #openstack-kolla | 17:00 | |
| srwilkers | sup jascott1 | 17:00 |
| thomas_oneill | oh I see | 17:01 |
| nea1 | but that will have to be done on more places than just there - I'll go over it later for now I'm moving to non "-" containing bridges | 17:01 |
| nea1 | I'll open a bug ticket with a patch or so tomorrow | 17:03 |
| zhubingbing | good night ;) guys | 17:06 |
| zhubingbing | bye | 17:06 |
| qwang | sdake: hi | 17:07 |
| sdake | qwang would you mind reviewing the deployment guide now in gerrit plz | 17:07 |
| sdake | qwang want to remove the cruft - will git review a new rev after you do a review | 17:08 |
| qwang | sdake: my pleasure | 17:08 |
| jaosorior | sdake: rebooting did the trick! thanks! | 17:11 |
| nea1 | I'll be gone too for today, thanks so far for the help | 17:11 |
| jaosorior | mnaser: also, it seems that overlayfs worked; was able to start the mariadb container. | 17:12 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible master: Fix can't find /usr/lib/libCryptoki2_64.so in barbican https://review.openstack.org/444561 | 17:12 |
| v1k0d3n | i think some folks may have asked portdirect about an updated a v1.5.2 kube-controller-manager for halcyon...that's been updated for anyone who's interested. | 17:13 |
| sdake | thanks qwang | 17:13 |
| v1k0d3n | morning/afternoon y'all :) | 17:14 |
| sdake | v1k0d3n i think we are on to 1.5.4 or 1.5.3 now - not sure ;) | 17:14 |
| v1k0d3n | if you guys run into issues, please let me know. | 17:14 |
| *** matrohon has quit IRC | 17:14 | |
| v1k0d3n | ok sdake...i think someone out there wanted a v1.5.2....so i guess this applies to them. | 17:15 |
| sdake | v1k0d3n halcyon needs this image iiuc - and we would like our docs to be in sync with our dev env and gate | 17:16 |
| *** eaguilar has joined #openstack-kolla | 17:16 | |
| mnaser | i had the issue during build time so that might be it jaosorior | 17:20 |
| v1k0d3n | sdake, not sure what you're asking. | 17:21 |
| sdake | v1k0d3n bbiaf | 17:21 |
| v1k0d3n | if you ever need an update to the image...just submit a PR; that'll keep things in sync. i'm in the process of creating a v1.5.4 image now as well (since you mentioned it). always willing to assist the kolla folks. 👍🏼 | 17:24 |
| srwilkers | for what it's worth, if anyone notices we need an updated image or finds any other issues with halcyon, it'd be great if they could file an issue against halcyon proper so we can address it in a timely manner. | 17:24 |
| srwilkers | what v1k0d3n said | 17:25 |
| v1k0d3n | aaaaand what srwilkers said too 😃 | 17:25 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 17:26 |
| v1k0d3n | we don't really use halcyon much anymore, so we're probably not going to find as many issues as you guys. | 17:26 |
| *** mpansky has quit IRC | 17:26 | |
| *** unicell has joined #openstack-kolla | 17:27 | |
| *** jaosorior has quit IRC | 17:35 | |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla-ansible master: [WIP] Added playbooks for snap deployment https://review.openstack.org/446165 | 17:36 |
| *** iceyao has joined #openstack-kolla | 17:36 | |
| *** lazyPwr is now known as lazyPower | 17:37 | |
| *** rwellum has joined #openstack-kolla | 17:37 | |
| *** lazyPower has quit IRC | 17:38 | |
| *** lazyPower has joined #openstack-kolla | 17:38 | |
| hrw | argh | 17:38 |
| hrw | fscking bifrost | 17:39 |
| inc0 | :D | 17:39 |
| inc0 | all the love | 17:39 |
| inc0 | kfox1111 sbezverk around | 17:39 |
| inc0 | ? | 17:39 |
| kfox1111 | inc0: hey | 17:42 |
| *** jmccarthy has quit IRC | 17:42 | |
| hrw | whenever fails fail for me I check for bifrost-base. like http://logs.openstack.org/40/430940/26/check/gate-kolla-dsvm-build-ubuntu-source-ubuntu-xenial/a33c9d8/console.html#_2017-03-20_16_53_12_325149 or http://logs.openstack.org/40/430940/26/check/gate-kolla-dsvm-build-centos-source-centos-7/2c6c11f/console.html#_2017-03-20_17_27_01_387087 | 17:42 |
| *** zhubingbing has quit IRC | 17:42 | |
| inc0 | kfox1111: correct me if I'm wrong, but these instructions are pre-helm right? https://review.openstack.org/#/c/447356/2 | 17:43 |
| sdake | v1k0d3n I was a bit cryptic earlier - had meeting - what i think kolla needs is that controller built for every version of kubernetes | 17:43 |
| *** masber has quit IRC | 17:43 | |
| sdake | v1k0d3n our gate builds its own controller for ceph - our docs reference version 1.5.2 (iirc) and we need the dev env = deploy guide = gate for versions | 17:44 |
| v1k0d3n | yeah ok. well, let me know if you need an updated controller-manager, or your welcome to submit a PR. | 17:45 |
| v1k0d3n | super easy :) | 17:45 |
| sdake | v1k0d3n sounds good | 17:45 |
| sdake | v1k0d3n i don't know yet - and dont want to waste your time with frivilous requests :) | 17:45 |
| v1k0d3n | well...right now, people are going to invididual teammembers on our side, rather than just submitting a PR or issue. | 17:46 |
| kfox1111 | inc0: the pointed at etherpad is https://etherpad.openstack.org/p/kolla-service-start | 17:46 |
| v1k0d3n | that actually takes more time. | 17:46 |
| *** masber has joined #openstack-kolla | 17:46 | |
| v1k0d3n | anyway...i gotta run. don't have a lot of time to chat. | 17:46 |
| v1k0d3n | later | 17:46 |
| sdake | v1k0d3n the workflow wasn't defined for building the controller - however we can disemminate that information to the various parties involved | 17:46 |
| inc0 | right, I missed that, I'll move this to change | 17:47 |
| sdake | inc0 your moving that etherpad where - to the review? | 17:49 |
| inc0 | yeah, all that stuff needs to land in review | 17:50 |
| inc0 | we can't link etherpads... | 17:50 |
| sdake | inc0 agreed i have a review in place here: https://review.openstack.org/#/c/447356/ - please review and i'll address | 17:50 |
| sdake | inc0 if folks provide their reviews in next 1 hour, i'l address all comments - spin the review - then spin the review again converted to RST | 17:51 |
| kfox1111 | gnocchi's trying to leave the union. fasinating.. | 17:53 |
| inc0 | now we need to find good name | 17:53 |
| inc0 | gnout? | 17:53 |
| kfox1111 | :) | 17:54 |
| kfox1111 | " 3+ | 17:55 |
| kfox1111 | node clusters with reliable network" | 17:55 |
| kfox1111 | what does that mean? | 17:55 |
| sdake | kfox1111 context? | 17:56 |
| kfox1111 | the thread "[openstack-dev] [kolla] rabbitmq cluster_partition_handling config in kolla-ansible" | 17:56 |
| kfox1111 | what is a "reliable network" though. | 17:56 |
| kfox1111 | not sure I've seen one yet. | 17:56 |
| inc0 | lol | 17:57 |
| * hrw -> checking where centos images failed | 17:57 | |
| kfox1111 | stuff fails. those who assume it won't get to keep the pieces when it breaks. | 17:57 |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla master: Added snap-telemetry container https://review.openstack.org/446164 | 17:58 |
| sdake | kfox1111 i think you just answered your own question or his :) | 17:58 |
| kfox1111 | yeah | 17:58 |
| *** duke__ has joined #openstack-kolla | 17:59 | |
| hrw | kfox1111: I know why it fails. have to check which images | 17:59 |
| sdake | kfox1111 it appears rabbitmq thinks reliable networks are a thing - when your opinion is they are not | 17:59 |
| kfox1111 | sdake: yeah. I know their opinion. and thats why so many ops have turned off clustering after a while of using it. | 18:00 |
| sdake | qwang would you mind adding a -1 to https://review.openstack.org/#/c/447356 so I can track which comments need attention? | 18:00 |
| kfox1111 | if you design a system off a flawed assumption, badness ensues. :/ | 18:00 |
| kfox1111 | every summit we go around asking folks their experience with rabbit clustering... hoping to hear a different answer. :/ | 18:01 |
| inc0 | I talked with Mirantis ops once and if they experienced net partition, they just wipe rabbit clean and restart it with message loss | 18:02 |
| qwang | sdake: sure. no reason to reject this kind of requests, I think. | 18:02 |
| kfox1111 | inc0: yeah. that works if you don't value the data. :/ | 18:03 |
| kfox1111 | for rpc, its generally fine. | 18:03 |
| sdake | qwang typically you would -1 vote if you see a problem with a review, so the submitter knows you object to the doc/code/etc | 18:03 |
| kfox1111 | if you really want to get event data out though, that can be a problem. | 18:03 |
| inc0 | agree | 18:03 |
| kfox1111 | for some of the stuf we do, we do care. :/ | 18:05 |
| *** masber has quit IRC | 18:05 | |
| *** ccesario has joined #openstack-kolla | 18:06 | |
| sdake | kfox1111 sbezverk can i get one review out of you int he next 30 minutes: https://review.openstack.org/#/c/447356/5/doc/source/deployment-guide.rst | 18:07 |
| *** mgoddard has quit IRC | 18:07 | |
| sdake | inc0 mnaser had a question for you earlier whether you wanted to participate in the creation of the docker push infrastructure blueprint | 18:07 |
| sdake | inc0: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-03-20.log.html#t2017-03-20T14:14:24 | 18:08 |
| *** athomas has quit IRC | 18:09 | |
| sdake | inc0 can you review more than lines 1-42 of the document, there are many other problems with the document as it stands | 18:09 |
| inc0 | sdake: I still work on it | 18:10 |
| inc0 | these are issues I see right now | 18:10 |
| sdake | inc0 roger - thought you were done | 18:10 |
| sdake | inc0 there are other links to etherpad which need migration as well | 18:10 |
| kfox1111 | sdake: why force override all the defaults? | 18:11 |
| sdake | kfox1111 ask the question in the review - I have no idea why its that way - perhaps another reviewer can answer, such as sbezverk :) | 18:11 |
| kfox1111 | its in an etherpad though. | 18:11 |
| sdake | kfox1111 what i'd like to know is how to generate cloud.yaml in the review | 18:11 |
| kfox1111 | I guess I could generally comment though... | 18:11 |
| sdake | kfox1111 you can comment on the line that links to the etherpad - that owuld be helpful | 18:11 |
| kfox1111 | sdake: shjouldh't have to generate it. onlyu specify the stuff that deviates from default. | 18:12 |
| kfox1111 | k | 18:12 |
| sdake | kfox1111 I have no idea what should deviate from defaults - that was the cloud.yaml sbezverk handed me in a paste in irc | 18:12 |
| kfox1111 | ah. ok. | 18:12 |
| sdake | kfox1111 this question is important to answer there is a TODO(sbezverk) I added there - however - i think he was traveling and didn't answer it - perhaps you could | 18:13 |
| sdake | end goal: merge this document prior to friday since it was first demoed on feb 15th, over a month ago ;) | 18:14 |
| kfox1111 | I can't identify further customizations unless I get time to actually try deploying. | 18:14 |
| kfox1111 | If its working though, its working? | 18:14 |
| *** duke__ has quit IRC | 18:17 | |
| *** adrian_otto has joined #openstack-kolla | 18:17 | |
| inc0 | kfox1111: qq - are you going to Boston? | 18:20 |
| kfox1111 | yeah. | 18:22 |
| inc0 | wanna join me on stage with kolla-k8s update?;) | 18:22 |
| inc0 | I ninja-added your name when I was asked for co-presenters, but that's totally fine to remove it :) | 18:23 |
| inc0 | just making sure | 18:23 |
| kfox1111 | maybe. | 18:23 |
| inc0 | think about it and let me know when you know plz | 18:23 |
| inc0 | I can run whole thing myself | 18:23 |
| inc0 | bbiaf, need some food | 18:25 |
| kfox1111 | k. will do. | 18:25 |
| *** crushil has quit IRC | 18:26 | |
| *** vhosakot has joined #openstack-kolla | 18:30 | |
| *** crushil has joined #openstack-kolla | 18:34 | |
| sdake | vhosakot can you please provide a review of https://review.openstack.org/#/c/447356 | 18:41 |
| vhosakot | sure, will do, thanks for the link sdake | 18:42 |
| sdake | thanks vhosakot | 18:42 |
| *** mgoddard has joined #openstack-kolla | 18:43 | |
| *** iceyao has quit IRC | 18:46 | |
| mnaser | https://review.openstack.org/#/c/447524/ inc0 id appreciate your thoughts on this and anyone else is welcome before i make another revision for infra folks | 18:46 |
| *** crushil has quit IRC | 18:50 | |
| *** david-lyle_ has joined #openstack-kolla | 18:52 | |
| *** david-lyle has quit IRC | 18:52 | |
| *** david-lyle_ is now known as david-lyle | 18:53 | |
| *** lamt has joined #openstack-kolla | 18:55 | |
| *** jemcevoy has joined #openstack-kolla | 18:55 | |
| *** eaguilar has quit IRC | 18:56 | |
| *** masber has joined #openstack-kolla | 18:58 | |
| *** lamt has quit IRC | 18:59 | |
| sbezverk | inc0: sorry for delayed reply, have backlog from last week absence. what can I do for you? | 19:01 |
| *** fooliouno has joined #openstack-kolla | 19:03 | |
| *** lamt has joined #openstack-kolla | 19:04 | |
| *** iceyao has joined #openstack-kolla | 19:04 | |
| *** crushil has joined #openstack-kolla | 19:05 | |
| *** lamt has quit IRC | 19:11 | |
| sbezverk | kfox1111: ping? | 19:13 |
| kfox1111 | ping | 19:13 |
| kfox1111 | whats up? :) | 19:13 |
| openstackgerrit | Graeme Seaton proposed openstack/kolla-ansible master: Allow alternative usernames for openstack in external-ceph https://review.openstack.org/447650 | 19:14 |
| sbezverk | kfox1111: have you every configured dhcp options for pools used by neutron dhcp server? | 19:14 |
| kfox1111 | hmm... | 19:15 |
| kfox1111 | globally, yes. but not for specific pools... | 19:15 |
| kfox1111 | there was an option that got merged to do overrides per port. | 19:15 |
| kfox1111 | and I think there was one in review to do it per subnet, but I lost track of that review, and I don't think it may have ever merged? | 19:15 |
| sbezverk | kfox1111: :( it seems there is a json command to configure it but I do not see any cli | 19:16 |
| kfox1111 | well, thats not horible. | 19:16 |
| kfox1111 | shoudl be able to wrap that up in a job? | 19:17 |
| *** masber has quit IRC | 19:17 | |
| sbezverk | kfox1111: yeah, but it is like a hack really | 19:20 |
| kfox1111 | agreed. but its either that, or wait until the openstack cli catches up with reality. which can be a while sometimes. : | 19:21 |
| kfox1111 | :/ | 19:21 |
| sbezverk | kfox1111: got it, thanks.. will try it then.. | 19:21 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 19:24 |
| *** lamt has joined #openstack-kolla | 19:24 | |
| openstackgerrit | Graeme Seaton proposed openstack/kolla-ansible master: Set fluentd output to elasticsearch use elasticsearch_address https://review.openstack.org/447654 | 19:30 |
| openstackgerrit | Graeme Seaton proposed openstack/kolla-ansible master: Set fluentd output to elasticsearch to use elasticsearch_address https://review.openstack.org/447654 | 19:32 |
| *** lamt has quit IRC | 19:33 | |
| *** adrian_otto has quit IRC | 19:34 | |
| *** mpansky has joined #openstack-kolla | 19:35 | |
| *** crushil has quit IRC | 19:37 | |
| *** crushil has joined #openstack-kolla | 19:41 | |
| *** iceyao has quit IRC | 19:41 | |
| *** m1dev has quit IRC | 19:42 | |
| Daviey_ | sdake: It doesn't change config files, affects a significant amount of users. Does expose a service that wasn't there before (but users would have EXPECTED it to be exposed)... Minimal regression potential, i'd be happy to accept it | 19:42 |
| *** krtaylor has quit IRC | 19:43 | |
| mnaser | Daviey_ and especially given that the feature would literally be broken without it | 19:47 |
| *** masber has joined #openstack-kolla | 19:50 | |
| *** lamt has joined #openstack-kolla | 20:00 | |
| *** rstarmer has joined #openstack-kolla | 20:01 | |
| *** rstarmer has quit IRC | 20:02 | |
| *** lamt has quit IRC | 20:04 | |
| *** satyar has quit IRC | 20:04 | |
| *** matrohon has joined #openstack-kolla | 20:12 | |
| *** masber has quit IRC | 20:13 | |
| *** dave-mcc_ has joined #openstack-kolla | 20:15 | |
| *** kencjohnston_ has joined #openstack-kolla | 20:15 | |
| *** pbourke_ has joined #openstack-kolla | 20:16 | |
| *** adrian_otto has joined #openstack-kolla | 20:17 | |
| *** zhenguo_ has joined #openstack-kolla | 20:18 | |
| *** esmiurium has joined #openstack-kolla | 20:20 | |
| *** srwilkers has quit IRC | 20:20 | |
| *** Aju has joined #openstack-kolla | 20:21 | |
| *** krtaylor has joined #openstack-kolla | 20:21 | |
| *** DuncanT_ has joined #openstack-kolla | 20:22 | |
| *** sdake_ has joined #openstack-kolla | 20:24 | |
| *** sdake_ has quit IRC | 20:24 | |
| *** sdake_ has joined #openstack-kolla | 20:24 | |
| *** spsurya__ has joined #openstack-kolla | 20:24 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 20:24 |
| *** jgriffith_ has joined #openstack-kolla | 20:27 | |
| sbezverk | kfox1111: ping | 20:29 |
| *** dave-mccowan has quit IRC | 20:29 | |
| *** pbourke has quit IRC | 20:29 | |
| *** shasha_t_ has quit IRC | 20:29 | |
| *** DuncanT has quit IRC | 20:29 | |
| *** aolwas has quit IRC | 20:29 | |
| *** spsurya has quit IRC | 20:29 | |
| *** sdake has quit IRC | 20:29 | |
| *** zhenguo has quit IRC | 20:29 | |
| *** kklimonda has quit IRC | 20:29 | |
| *** jgriffith has quit IRC | 20:29 | |
| *** alanmeadows has quit IRC | 20:29 | |
| *** afranc has quit IRC | 20:29 | |
| *** esmiurium_ has quit IRC | 20:29 | |
| *** kencjohnston has quit IRC | 20:29 | |
| *** spsurya__ is now known as spsurya | 20:29 | |
| kfox1111 | sbezverk: ping | 20:30 |
| *** zhenguo_ is now known as zhenguo | 20:30 | |
| *** jgriffith_ is now known as jgriffith | 20:30 | |
| *** srwilkers has joined #openstack-kolla | 20:31 | |
| *** shardy has quit IRC | 20:32 | |
| sbezverk | kfox1111: I am seeing strange thing, I have a linux bridge with ip address | 20:32 |
| *** DuncanT_ is now known as DuncanT | 20:32 | |
| kfox1111 | yeah? | 20:33 |
| sbezverk | kfox1111: I can ping it, no prob | 20:33 |
| sbezverk | kfox1111: but when another container arp for ip assigned to the bridge | 20:33 |
| *** kklimonda has joined #openstack-kolla | 20:33 | |
| *** dave-mcc_ is now known as dave-mccowan | 20:33 | |
| sbezverk | it does not reply to arp requests, I can see them with tcpdump right on that bridge | 20:33 |
| kfox1111 | iptables enabled? | 20:34 |
| kfox1111 | iptables by default affects linuxbridge packets, but not ovs bridge packets. | 20:34 |
| kfox1111 | stp being on might effect things too, if its a recently brought up port. | 20:34 |
| sbezverk | kfox1111: well it is basic kube cluster running openstack | 20:34 |
| sbezverk | and ironic | 20:34 |
| kfox1111 | in the gate? | 20:34 |
| sbezverk | container in question runs in net=host mode | 20:34 |
| kfox1111 | hmm.... | 20:35 |
| sbezverk | kfox1111: no, had to move from the gate, was close to impossible to troubleshoot at it reuired packet captures | 20:35 |
| kfox1111 | so, container net=host arps fine, container net != host on the same host fails? | 20:35 |
| sbezverk | kfox1111: I tcpdump in the container with net=host and I can see arp request packets | 20:36 |
| kfox1111 | yeah, I get that. this level of deep network inspection would be tricky that wya. | 20:36 |
| sbezverk | by some reason nothing replies to arp request, not even host which I think it shold | 20:36 |
| kfox1111 | ok... so, let me double check: | 20:37 |
| kfox1111 | you have a single host. | 20:37 |
| kfox1111 | on the host, you have a linux bridge. | 20:37 |
| kfox1111 | you put an ip address on that linux bridge. | 20:37 |
| sbezverk | kfox1111: yes | 20:38 |
| kfox1111 | you launch a container on the host, net=host, then try and ping the linux bridge's address. | 20:38 |
| sbezverk | kfox1111: not exactly | 20:38 |
| kfox1111 | and you can tcpdump the bridge and see the arp from the container, | 20:38 |
| kfox1111 | but not the response? | 20:38 |
| kfox1111 | ok. whats different? | 20:38 |
| sbezverk | kfox1111: I can ping from the container in net=host | 20:38 |
| sbezverk | but I have my "baremetal" vm whcih is also plugged into this bridge | 20:38 |
| *** Pavo has joined #openstack-kolla | 20:39 | |
| *** shardy has joined #openstack-kolla | 20:39 | |
| sbezverk | and it is my baremetal vm arping for that ip since it is ip of tftp server | 20:39 |
| sbezverk | to download pxelinux.0 | 20:39 |
| *** jtriley has quit IRC | 20:39 | |
| sbezverk | I see arp requests coming from that VM but nothing responds to them | 20:39 |
| sbezverk | kfox1111: http://paste.openstack.org/show/603495/ | 20:40 |
| sbezverk | .10 is ip address of tftp server .104 is ip address which neutron dhcp server gave to my baremetal VM | 20:41 |
| kfox1111 | oh, so... two boxes? or 1? | 20:42 |
| sbezverk | sorry .103 | 20:42 |
| kfox1111 | (vm's or otherwise) | 20:42 |
| sbezverk | one box | 20:42 |
| kfox1111 | two containers? | 20:42 |
| sbezverk | one physcial box may VMs | 20:42 |
| kfox1111 | oh. k. | 20:42 |
| sbezverk | one container and one vm | 20:42 |
| kfox1111 | hmm... | 20:42 |
| sbezverk | pxe is ironic container | 20:42 |
| sbezverk | vm-1 is baremetal vm | 20:42 |
| kfox1111 | k. | 20:42 |
| kfox1111 | what launched the vm? | 20:42 |
| sbezverk | I am simulating baremetal provisioning | 20:43 |
| sbezverk | vmbc | 20:43 |
| sbezverk | works like a charm | 20:43 |
| kfox1111 | so, libvirt? | 20:43 |
| sbezverk | virtualbmc | 20:43 |
| sbezverk | yes vbmc contacts libvirt | 20:43 |
| kfox1111 | virtualbmc -> libvirt -> kvm? | 20:43 |
| sbezverk | yep | 20:43 |
| kfox1111 | k. | 20:43 |
| kfox1111 | and the libvirt network backend is linuxbridge? | 20:43 |
| kfox1111 | in bridging mode? | 20:43 |
| sbezverk | I can see vm comes up and gets ip address from neutron | 20:43 |
| kfox1111 | ok. so dnsmasq is working. | 20:44 |
| kfox1111 | hmm... | 20:44 |
| sbezverk | vm is configured to plg its interface into a specific linuxbridge | 20:44 |
| kfox1111 | and thats in the container too? | 20:44 |
| sbezverk | yes | 20:44 |
| sbezverk | whole neutron runs in containers | 20:44 |
| kfox1111 | and that bridge has 172.21.0.10 on it? | 20:44 |
| *** manheim has joined #openstack-kolla | 20:44 | |
| sbezverk | yes | 20:44 |
| *** Pavo has quit IRC | 20:44 | |
| sbezverk | if you have time I can show you in webex | 20:45 |
| kfox1111 | centos/ubuntu host? | 20:45 |
| sbezverk | cenits | 20:45 |
| kfox1111 | um... yeah. | 20:45 |
| sbezverk | centos | 20:45 |
| kfox1111 | did you turn off iptables/firewalld? | 20:45 |
| sbezverk | firewalld is off otherwise kube would not work | 20:45 |
| kfox1111 | k. just double checking. | 20:45 |
| sbezverk | iptables I have not touched | 20:45 |
| kfox1111 | and the tcpdump you gave is from the bridge? | 20:45 |
| sbezverk | it is controlled by openstack | 20:45 |
| sbezverk | yes | 20:46 |
| kfox1111 | did the vm recently come up at that point, or was it running for more then a minute? | 20:46 |
| sbezverk | ironic contacts vbmc - it brings up VM, VM gets ip addresss from neutron dhcp server then it gets tftp server ip in dhcp option and then it tries to download pxelinux.o from tftp | 20:48 |
| sbezverk | so I do not control much, everything is done by ironic | 20:48 |
| kfox1111 | so if its getting the ip first, then its not STP on the bridge. | 20:49 |
| sbezverk | agree, it seems connectivity works | 20:49 |
| sbezverk | https://cisco.webex.com/join/sbezverk | 20:49 |
| kfox1111 | anything in /etc/sysconfig/iptables? | 20:49 |
| kfox1111 | k. | 20:49 |
| sbezverk | anybody who wants to join and see ironic in action are welcome | 20:49 |
| *** unicell1 has joined #openstack-kolla | 20:51 | |
| *** unicell has quit IRC | 20:52 | |
| openstackgerrit | Mohammed Naser proposed openstack/kolla master: Speed up builds by increasing threads for builds https://review.openstack.org/446701 | 20:56 |
| mnaser | i had to rebase this patch so i lost my +W .. any cores? ^ <3 | 20:56 |
| *** rhallisey has quit IRC | 21:00 | |
| *** Pavo has joined #openstack-kolla | 21:08 | |
| jemcevoy | SamYaple: you there | 21:10 |
| *** crushil has quit IRC | 21:12 | |
| *** Pavo has quit IRC | 21:16 | |
| *** hawi has quit IRC | 21:16 | |
| *** hawi has joined #openstack-kolla | 21:17 | |
| *** goldyfruit has quit IRC | 21:24 | |
| *** schwicht has quit IRC | 21:29 | |
| *** srwilkers has quit IRC | 21:30 | |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| *** manheim has quit IRC | 21:35 | |
| *** jrobinson has joined #openstack-kolla | 21:35 | |
| *** Pavo has joined #openstack-kolla | 21:37 | |
| *** shardy has quit IRC | 21:37 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 21:38 | |
| *** Pavo has quit IRC | 21:39 | |
| *** rwellum has quit IRC | 21:47 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix Telegraf startup config files issue https://review.openstack.org/447382 | 21:48 |
| *** matrohon has quit IRC | 21:49 | |
| mnaser | ugh | 21:51 |
| mnaser | bifrost broke gate again | 21:51 |
| *** vhosakot has quit IRC | 21:53 | |
| jemcevoy | inc0: Is there a way for me to run kolla deploy on just one host? maradb is dead on controller2 and I want to delete the mariadb container, image and volume and just deploy on controller2. | 21:53 |
| jemcevoy | I am running 3.0.2 in production/ | 21:54 |
| inc0 | jemcevoy: so in general re-running kolla deploy shouldn't touch anthing besides things that needs changing | 21:55 |
| inc0 | but when you run kolla-ansible deploy it will print out long ansible command | 21:55 |
| inc0 | as first line of output | 21:55 |
| inc0 | you can just use this long ansible command, they're equivalent | 21:55 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 21:55 |
| *** rwallner has quit IRC | 21:56 | |
| inc0 | and then you have all sorts of filtering/manipulations of ansible at your disposal | 21:56 |
| jemcevoy | and add --limit to the end right | 21:56 |
| inc0 | that should do it | 21:56 |
| jemcevoy | Great I can just scroll back to my last run over the weekend | 21:57 |
| *** schwicht has joined #openstack-kolla | 22:01 | |
| sdake_ | inc0 biforst is not building - the gates are jammed up | 22:02 |
| mnaser | sdake_ https://bugs.launchpad.net/kolla/+bug/1674483 | 22:02 |
| openstack | Launchpad bug 1674483 in kolla "Bifrost failing because of missing SUDO_USER" [Undecided,New] | 22:02 |
| sdake_ | inc0 as the gates are voting, no progress can be made | 22:02 |
| inc0 | what changed?:/ | 22:02 |
| *** fooliouno has quit IRC | 22:02 | |
| mnaser | im trying to figure it out if you want to help out | 22:02 |
| mnaser | ansible_env.SUDO_USER is .. unset for some reason | 22:02 |
| mnaser | https://github.com/openstack/bifrost/commit/36e32ecdc26035ecd853f830b30d42827662a68a | 22:03 |
| mnaser | this merged | 22:03 |
| mnaser | which is running that role which is failing | 22:03 |
| mnaser | i guess when this merged it broke things https://review.openstack.org/#/c/444820/ | 22:04 |
| mnaser | because that role is now running regardless? and i assume it didnt run before | 22:04 |
| inc0 | mnaser: can you link me failing review? | 22:04 |
| mnaser | (didnt verify this, im speculating) | 22:04 |
| mnaser | inc0 https://review.openstack.org/#/c/446701/ | 22:04 |
| mnaser | i put a bit of info in the bug i created | 22:05 |
| mnaser | im going to assume ansible_env.SUDO_USER is unset when running it locally | 22:05 |
| sdake_ | mnaser why would that cause the build to fail? | 22:06 |
| mnaser | sdake_ as part of bifrost installation it runs a set of ansible playbooks | 22:07 |
| *** sdake_ is now known as sdake | 22:07 | |
| sdake | i see | 22:07 |
| mnaser | i assume its meant to target a remote host so ansible_env.SUDO_USER is supposed to be set | 22:07 |
| mnaser | i think the problem is they should be using a lookup instead of referring to ansible_env directly | 22:07 |
| *** schwicht has quit IRC | 22:08 | |
| *** leseb has quit IRC | 22:09 | |
| *** dave-mccowan has quit IRC | 22:12 | |
| mnaser | ok i think i know how to solve this in bifrost land | 22:12 |
| mnaser | there is an existing default but the problem is the lookup | 22:13 |
| *** Pavo has joined #openstack-kolla | 22:13 | |
| *** leseb has joined #openstack-kolla | 22:15 | |
| *** mgoddard has quit IRC | 22:15 | |
| *** Pavo has quit IRC | 22:16 | |
| inc0 | mnaser: makes sense | 22:17 |
| * inc0 building locally | 22:18 | |
| mnaser | wrote a patch inc0 | 22:18 |
| mnaser | i am submitting to bifrost soon | 22:18 |
| *** alanmeadows has joined #openstack-kolla | 22:19 | |
| mnaser | inc0 sdake https://review.openstack.org/447713 | 22:20 |
| mnaser | im not sure if you can somehow cherry pick that and try building with it but i think itll fix things | 22:23 |
| *** Pavo has joined #openstack-kolla | 22:23 | |
| *** Pavo has quit IRC | 22:25 | |
| *** sayantani01 has quit IRC | 22:28 | |
| mnaser | spoke with ironic core folks | 22:31 |
| mnaser | they'll merge once it passes | 22:31 |
| mnaser | and hopefully we can recheck and all is good | 22:31 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/446772 | 22:31 |
| sdake | portdirect about? | 22:34 |
| *** Pavo has joined #openstack-kolla | 22:34 | |
| *** manheim has joined #openstack-kolla | 22:37 | |
| inc0 | kfox1111: sbezverk question - kollakube res create configmap mariadb ended up with error [Errno 2] No such file or directory: '/home/ubuntu/kolla/ansible/roles/neutron/defaults/main.yml' | 22:38 |
| sdake | inc0 you ahve to cherrypick a patch for that to work | 22:38 |
| inc0 | link plz | 22:38 |
| kfox1111 | weird. | 22:38 |
| inc0 | it is weird | 22:38 |
| sdake | https://review.openstack.org/#/c/447356/ | 22:39 |
| sdake | search for "CHERRYPICK" | 22:39 |
| inc0 | kfox1111: https://review.openstack.org/#/c/439740/ | 22:40 |
| inc0 | ok...different question | 22:41 |
| inc0 | why does it even need kolla-ansible? | 22:41 |
| sdake | genconfig | 22:41 |
| inc0 | I already done that | 22:41 |
| inc0 | I have configs generated | 22:41 |
| inc0 | I just want to create configmap | 22:41 |
| sdake | dunno code probably has the asnwer :) | 22:42 |
| inc0 | ok this is wrong | 22:42 |
| sdake | i thought you were asking why kolla-kubernetes needs kolla-ansible | 22:42 |
| inc0 | I know that one | 22:42 |
| inc0 | let me check what this tool even intends to do, because this should be pretty simple one liner to create configmap from existing file | 22:43 |
| kfox1111 | sbezverk: echo "0" > /proc/sys/net/bridge/bridge-nf-call-iptables | 22:44 |
| inc0 | how to turn off kernel module like a pro | 22:44 |
| *** krtaylor has quit IRC | 22:45 | |
| *** masber has joined #openstack-kolla | 22:45 | |
| sdake | this needs an ack: https://review.openstack.org/#/c/447217/ | 22:46 |
| *** Pavo has quit IRC | 22:47 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 22:57 |
| inc0 | kfox1111: how horrible would it be if I'd create playbook to template all the configmaps/secrets? | 22:59 |
| inc0 | because frankly all we need is something to read yaml and output kube resources | 22:59 |
| inc0 | then you do kubectl create -f /path/to/dir | 22:59 |
| kfox1111 | inc0: isn't that what kolla-ansible does today? :) | 23:00 |
| inc0 | kfox1111: for kolla-k8s | 23:00 |
| inc0 | and yes | 23:00 |
| inc0 | it's extremally easy playbook | 23:00 |
| inc0 | (already wrote piece of play for ceph secrets) | 23:00 |
| *** alanmeadows has quit IRC | 23:00 | |
| *** alanmeadows has joined #openstack-kolla | 23:00 | |
| inc0 | in fact, from my perspective it would be great if I could just submit piece of ansible to do all the prework for kolla-k8s | 23:01 |
| kfox1111 | ideally we'd just put the effort into getting it into helm. | 23:01 |
| kfox1111 | not much extra work I think. | 23:01 |
| sdake | kfox1111 tend to agree | 23:01 |
| sdake | external config maps ftw | 23:02 |
| kfox1111 | yeah | 23:02 |
| inc0 | I don't know how good helm is for reading local files and if golang have stuff like filters to base64 | 23:02 |
| kfox1111 | doesn't need to? | 23:02 |
| inc0 | for secrets yeah | 23:02 |
| kfox1111 | we just copy the kolla-ansible templates into the teplate dir, and tweak some stuff? | 23:02 |
| inc0 | ahh you mean create full configs in helm? | 23:03 |
| kfox1111 | yeah, secret would be the exception. though there isn't many of those. | 23:03 |
| kfox1111 | yeah. | 23:03 |
| kfox1111 | helm install neutron-configmap | 23:03 |
| *** rwallner has joined #openstack-kolla | 23:03 | |
| inc0 | well yes and no, it won't be that easy | 23:03 |
| inc0 | we'd need to copy all the values from ansible too | 23:04 |
| sdake | yup - that needs to be done | 23:04 |
| inc0 | yes I know | 23:04 |
| inc0 | but playbook I'm talking about is 1 day of work | 23:04 |
| inc0 | then creating secrets+configmaps will be ansible-playbook and kubectl | 23:05 |
| inc0 | 2 commands | 23:05 |
| inc0 | anyway, I'll do it anyway | 23:06 |
| inc0 | because we need more robust tooling for that | 23:06 |
| inc0 | and after that we can move configs to helm one by one | 23:06 |
| inc0 | it's not going to be *that* easy | 23:06 |
| inc0 | especially with overrides and shit | 23:06 |
| *** rwallner has quit IRC | 23:08 | |
| kfox1111 | why would that be easier then just using genconfig? | 23:10 |
| *** jrobinson has quit IRC | 23:11 | |
| inc0 | kfox1111: I'm using genconfig | 23:12 |
| inc0 | I'm talking about kollakube resource configmap create replacement | 23:12 |
| kfox1111 | oh. | 23:12 |
| inc0 | I don't see much value in this pseudotool anyway by now | 23:12 |
| inc0 | I'd rather have some simple playbook to take existing configs/password.yaml and output dir with rendered kube resources | 23:13 |
| inc0 | then you just kubectl whole dir | 23:13 |
| sdake | inc0 as long as you aren't married to the code inc0 - as when we move to helm its going byebye :) | 23:13 |
| inc0 | it's going to be single file ansible play | 23:14 |
| inc0 | no, I'll get rid of it happily | 23:14 |
| sdake | cool | 23:14 |
| inc0 | at least that will allow us to kickstart work easily and do one config at the time | 23:14 |
| kfox1111 | we need it for just a little bit of ceph, and the openvswitch-set-ip thingy. | 23:14 |
| kfox1111 | too. | 23:14 |
| inc0 | yeah, I have local cephy thing too | 23:15 |
| kfox1111 | those should be moved to helm for sure though. shouldn't be much work left. | 23:15 |
| inc0 | I'll publish it with same idea | 23:15 |
| kfox1111 | ceph's half way over already. | 23:15 |
| inc0 | what I'm afraid of is lack of overrides | 23:15 |
| sdake | overrides are manageable directly in the openstack services | 23:15 |
| inc0 | it's gonna to be bitch to arrive to our merge_config feature parity | 23:15 |
| sdake | inc0 i ahd same question of kfox1111 | 23:16 |
| *** mpansky has quit IRC | 23:16 | |
| sdake | inc0 read this log: | 23:16 |
| kfox1111 | is merge_config seperable from kolla-ansible? | 23:17 |
| kfox1111 | or is it ansible code? | 23:17 |
| inc0 | kfox1111: it's ansible module | 23:18 |
| inc0 | I mean we can rewrite it | 23:18 |
| inc0 | but doesn't make sense | 23:18 |
| inc0 | imho | 23:18 |
| kfox1111 | it could be used as an init container. | 23:18 |
| inc0 | as tbh kolla-ansible genconfig just works | 23:18 |
| inc0 | right...well | 23:18 |
| inc0 | it's going to be harder as per-node override | 23:19 |
| kfox1111 | why? | 23:19 |
| inc0 | ansible already gives us infrastructure to define it by hostname and such | 23:20 |
| sdake | well i can't find the log at this moment | 23:20 |
| inc0 | ofc, doable | 23:20 |
| sdake | however- the idea is to run nova config1 config2 config3 | 23:20 |
| sdake | that permits config overrides | 23:20 |
| inc0 | yeah we discussed that in pth | 23:20 |
| sdake | inc0 we were unable to determine why we didn't use this in kolla-ansible | 23:20 |
| inc0 | ptg | 23:20 |
| sdake | it was also discussed on irc recently | 23:20 |
| inc0 | because oslo.config started to support it relatively lately | 23:21 |
| sdake | the exchange on irc indicated its been a feature since essex. | 23:21 |
| inc0 | by the time we wrote merge_config that wasn't the case | 23:21 |
| inc0 | well | 23:21 |
| inc0 | then I'm mistaken | 23:21 |
| inc0 | anyway | 23:21 |
| sdake | i htink the reasosn we didn't use it is we didn't know about it as a feature | 23:21 |
| inc0 | possible | 23:21 |
| sdake | per node custom configs is easy in this model | 23:22 |
| kfox1111 | inc0: its supported it for many years. | 23:22 |
| sdake | I believe all the use cases are covered | 23:22 |
| inc0 | ok then, we just need to write the thing | 23:23 |
| kfox1111 | yup. | 23:23 |
| sdake | thing being helm external config maps? | 23:23 |
| inc0 | in the meantime I'll crack my quick playbook to get us rolling | 23:23 |
| mnaser | https://review.openstack.org/#/c/447713/ this merged | 23:27 |
| mnaser | so i think recheck | 23:27 |
| mnaser | should clear the gate up | 23:27 |
| kfox1111 | still think we shouldn't spend muhc time writing code that we shoudl get rid of asap. | 23:27 |
| kfox1111 | but, whatever. | 23:28 |
| kfox1111 | the problem isn't pip install kolla-kubernetes is hard. its that all pip/genconfig's a pain. | 23:28 |
| portdirect | sdake: ping | 23:29 |
| sdake | portdirect shoot | 23:29 |
| sdake | mnaser yup a recheck will do the job - no need to rebase | 23:30 |
| mnaser | donezo, hopefully it flows through | 23:30 |
| inc0 | kfox1111: yes and no - if it takes few hrs to automate this thing, I'll just do it | 23:30 |
| portdirect | sdake: just replying to your ping, sup dude? | 23:30 |
| *** eaguilar has joined #openstack-kolla | 23:31 | |
| sdake | portdirect oh - v1k0d3n indicated i should communicate with his team via PR's, so I've done that - however, here is the PR: https://github.com/att-comdev/halcyon-kubernetes/pull/56 | 23:31 |
| kfox1111 | inc0: plus more to redo the docs | 23:32 |
| inc0 | kfox1111: what docs?:P | 23:32 |
| kfox1111 | inc0: the docs people keep writing. ;) | 23:32 |
| inc0 | also, that will need redoing anyway | 23:32 |
| *** masber has quit IRC | 23:32 | |
| inc0 | so it's a quck fix to what I consider broken today | 23:32 |
| *** rhallisey has joined #openstack-kolla | 23:32 | |
| inc0 | keyword is quick | 23:32 |
| v1k0d3n | sdake done/merged | 23:33 |
| sdake | thanks v1k0d3n | 23:33 |
| v1k0d3n | the image was created earlier today already. | 23:33 |
| sdake | v1k0d3n cool thanks :) | 23:33 |
| v1k0d3n | anytime | 23:33 |
| kfox1111 | replacing python with ansible just doesn't seem to be a step in the right direction to me. but again, whatever. | 23:34 |
| sdake | kfox1111 agreed - we will just delete the ansible but inc0 said he wasn't married to it, so if inc0 wants to waste his time more power to him :) | 23:34 |
| portdirect | Not sure I follow but long as you guys are good. | 23:35 |
| sbezverk | gents kfox1111 and myself just brought up a baremetal VM using ironic running on kolla kubernetes | 23:35 |
| sbezverk | that was fraking awesome!!! | 23:35 |
| sdake | portdirect http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-03-20.log.html#t2017-03-20T17:46:21 | 23:35 |
| sbezverk | freaking I meant | 23:35 |
| portdirect | sbezverk: you got time this week to chat? | 23:35 |
| sdake | portdirect for more context | 23:35 |
| sbezverk | portdirect: absolutely | 23:35 |
| spsurya | morning all | 23:36 |
| *** vhosakot has joined #openstack-kolla | 23:36 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Fernet Token Support https://review.openstack.org/446274 | 23:38 |
| sdake | sup spsurya | 23:38 |
| portdirect | sbezverk: nice one, think wed is good for me? I can hit up srwilkers and check with him. | 23:38 |
| sdake | sbezverk rockin dude, was it baremetal in the gate? | 23:39 |
| sbezverk | portdirect: wednesday work for me too, prefer am if it s possible | 23:39 |
| sbezverk | sdake: not at the gate yet, to resolv connectivity issues I had to build it locally to debug | 23:40 |
| sdake | sbezverk ya its pretty sweet to see that first bare metal node boot up in ironic :) | 23:40 |
| sbezverk | sdake: especially ALL is running ont the single physcial server ;) | 23:41 |
| *** dave-mccowan has joined #openstack-kolla | 23:44 | |
| *** manheim has quit IRC | 23:44 | |
| sdake | sbezverk could you expand - i'm not sure I follow how you run kubernetes on the same server as the baremetal node that ironic boots | 23:45 |
| *** vhosakot has quit IRC | 23:45 | |
| kfox1111 | sdake: baremetal vm. | 23:45 |
| kfox1111 | simulated baremetal. | 23:46 |
| sdake | kfox1111 got it thanks :) | 23:46 |
| kfox1111 | :) | 23:46 |
| *** vhosakot has joined #openstack-kolla | 23:46 | |
| sdake | vhosakot this needs an ack: https://review.openstack.org/#/c/447217/ | 23:48 |
| vhosakot | sdake: done, although the ubuntu source xenial gate has an unrelated http://logs.openstack.org/17/447217/2/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/40fa69b/console.html#_2017-03-18_15_17_35_503927. | 23:50 |
| sdake | vhosakot yup iiuc that has been broken for ages | 23:50 |
| vhosakot | yep, I have seen that zuul error before. | 23:51 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Update to 1.5.4 in development environment documentation https://review.openstack.org/447731 | 23:52 |
| inc0 | kfox1111 sbezverk can I get kubectl get configmap mariadb -o yaml please? | 23:53 |
| inc0 | to confirm structure I'm generating | 23:53 |
| kfox1111 | inc0: config's are here: http://logs.openstack.org/74/446274/10/check/gate-kolla-kubernetes-deploy-centos-binary-2-external-ovs-nv/a7858c7/logs/configmaps.yaml | 23:56 |
| inc0 | thank you good sir | 23:57 |
| kfox1111 | almost there... just gotta fix rotate now... | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!