| *** lujinluo has joined #openstack-kolla | 00:04 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 00:06 |
|---|---|---|
| sdake_ | tonanhngo we dont release 3.0.0 images on dockerhub utnil 30.0 is releaed | 00:06 |
| sdake_ | tonanhngo if you want 3.0.0 images you have to build them yourself | 00:06 |
| *** zhurong has quit IRC | 00:06 | |
| tonanhngo | ah ok, got it | 00:06 |
| tonanhngo | sdake_: I am a bit confused about the variable kolla_internal_vip_address in the file /etc/kolla/globals.yml | 00:11 |
| inc0 | tonanhngo, read through this http://docs.openstack.org/developer/kolla/production-architecture-guide.html | 00:12 |
| tonanhngo | inc0: ok, thanks for the pointer! | 00:13 |
| inc0 | we need to make proper config reference | 00:13 |
| inc0 | internal vip address is basically an IP which keepalived will manage | 00:14 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding resolv.conf workaround to logging containers https://review.openstack.org/378021 | 00:18 |
| *** tonanhngo has quit IRC | 00:18 | |
| *** duonghq has joined #openstack-kolla | 00:23 | |
| *** huikang has quit IRC | 00:24 | |
| *** huikang has joined #openstack-kolla | 00:24 | |
| *** tonanhngo has joined #openstack-kolla | 00:25 | |
| duonghq | morning | 00:26 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Adding resolv.conf workaround to logging containers https://review.openstack.org/378021 | 00:29 |
| *** tonanhngo has quit IRC | 00:29 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fixing keystone logging related issue https://review.openstack.org/378076 | 00:30 |
| *** huikang has quit IRC | 00:30 | |
| *** huikang has joined #openstack-kolla | 00:35 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 00:37 |
| *** caowei has joined #openstack-kolla | 00:40 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Fixing keystone logging related issue https://review.openstack.org/378076 | 00:41 |
| *** sgordon has quit IRC | 00:42 | |
| *** sgordon has joined #openstack-kolla | 00:43 | |
| *** salv-orlando has joined #openstack-kolla | 00:48 | |
| *** salv-orlando has quit IRC | 00:52 | |
| *** schwicht has quit IRC | 00:52 | |
| *** neilus has joined #openstack-kolla | 00:59 | |
| *** neilus has quit IRC | 01:04 | |
| duonghq | inc0, sdake_: do we have today meeting agenda | 01:10 |
| *** senk_ has joined #openstack-kolla | 01:11 | |
| openstackgerrit | dupengfei proposed openstack/kolla: Fix keystone,database,memcache for k8s. https://review.openstack.org/372089 | 01:12 |
| inc0 | duonghq, rough | 01:13 |
| inc0 | we need to talk about 2 topics mostly - rc2 and summit sessions | 01:13 |
| inc0 | you want to put anything in? | 01:13 |
| duonghq | inc0: I have idea about dependency between openstack service | 01:14 |
| inc0 | what do you mean? | 01:15 |
| *** senk_ has quit IRC | 01:15 | |
| duonghq | e.g: Magnum depends on Heat so when Magnum image is built, Heat should be built too | 01:16 |
| duonghq | even if it's not specified | 01:16 |
| inc0 | we shouldnt do that | 01:16 |
| inc0 | magnum image doesnt depend on heat | 01:16 |
| duonghq | but if Heat image is not built, Magnum cannot run | 01:17 |
| *** gfhellma has quit IRC | 01:18 | |
| *** zhurong has joined #openstack-kolla | 01:18 | |
| openstackgerrit | dupengfei proposed openstack/kolla: Support shared path for glance images and nova instances disk https://review.openstack.org/372420 | 01:19 |
| inc0 | built != deployed | 01:20 |
| duonghq | we should also define "core set" of images, which must be built for kolla work, like heka, ... | 01:20 |
| *** zhubingbing has joined #openstack-kolla | 01:21 | |
| duonghq | inc0: agree, but if image isn't ready, how can we deploy? | 01:21 |
| inc0 | well, somebody may have it already deployed | 01:22 |
| duonghq | I'm supporting one multimedia company deploy Kolla | 01:24 |
| duonghq | they are new to Ansible, Kolla | 01:24 |
| duonghq | and do not want to built all image (currently we build every openstack service if not any is specified) | 01:24 |
| duonghq | so they have trouble to determine which image need to be built | 01:25 |
| duonghq | (use local registry) | 01:25 |
| *** lamt has joined #openstack-kolla | 01:27 | |
| inc0 | yeah, but when they figure it out they can prepare their own profile | 01:27 |
| inc0 | you can help them with it | 01:27 |
| duonghq | sure, but when other team need deploy Kolla, they must repeat the procedure | 01:28 |
| inc0 | you know how build profiles work right? | 01:30 |
| duonghq | sure | 01:30 |
| inc0 | so each deployer should determine their own set of images | 01:30 |
| duonghq | except core image like heka,.... | 01:30 |
| duonghq | the images set must be determined case-to-case | 01:31 |
| inc0 | hold on, in kolla-build.conf you have profiles config | 01:31 |
| *** lamt has quit IRC | 01:31 | |
| inc0 | -p default will build what is deployed by default (not changed anything in globals.yml) | 01:31 |
| inc0 | you can define your own profile by simple configuration | 01:31 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO_NOT_MERGE: Use better URL for China https://review.openstack.org/329060 | 01:32 |
| *** tonanhngo has joined #openstack-kolla | 01:33 | |
| duonghq | inc0: so users should figure out the dependency graph by their own? | 01:33 |
| duonghq | and create their profile? | 01:33 |
| inc0 | well, most of the time servies aren't dependant on each other | 01:34 |
| *** tonanhngo has quit IRC | 01:34 | |
| inc0 | heka and other stuff like that is part of default profile | 01:34 |
| inc0 | if you want to add magnum, you can always copy default and add magnum | 01:34 |
| inc0 | then you just build by kolla-build -p my-profile-name | 01:34 |
| duonghq | but when more and more PaaS join big tents, the dependency come in between PaaS project and IaaS | 01:36 |
| *** schwicht has joined #openstack-kolla | 01:37 | |
| inc0 | so there are other reasons why I don't want to have hard dependency between containers | 01:37 |
| inc0 | for example someone might want to have heat from mitaka but magnum from newton | 01:37 |
| *** haplo37_ has quit IRC | 01:38 | |
| inc0 | so 2 different builds | 01:38 |
| inc0 | I have to go:) play around build profiles | 01:40 |
| inc0 | this should answer your needs, it's a matter of finding the correct one | 01:41 |
| inc0 | we can always can add more defaults if you want | 01:41 |
| inc0 | like -p paas | 01:41 |
| duonghq | inc0: understood, but should we profile default dependency, and user can remove if they need? | 01:41 |
| duonghq | thank you inc0 for the dicussion | 01:41 |
| inc0 | by default we build everything | 01:41 |
| inc0 | so that's not a problem | 01:41 |
| inc0 | we have basic profile -p default | 01:41 |
| inc0 | we have couple other ones | 01:41 |
| inc0 | couple different ones* | 01:42 |
| inc0 | and it's super easy to add your own | 01:42 |
| duonghq | sure, I mean default dependency between some project? | 01:42 |
| inc0 | that is done with image deps | 01:42 |
| *** g3ek has quit IRC | 01:43 | |
| duonghq | roger, I will thinking about this and discuss with you later, when you have free time | 01:43 |
| inc0 | when you build kolla-build magnum, all of magnum images and their dependencies (magnum-base, openstack-base and base) will be built | 01:43 |
| inc0 | sure, I'll be back tomorrow my morning:) also Jeffrey will knwo everything about that | 01:43 |
| *** tonanhngo has joined #openstack-kolla | 01:43 | |
| inc0 | good night! thanks :) | 01:43 |
| inc0 | always good to have ideas | 01:44 |
| *** haplo37 has quit IRC | 01:44 | |
| *** tonanhngo has quit IRC | 01:44 | |
| *** inc0 has quit IRC | 01:44 | |
| duonghq | night inc0 | 01:44 |
| *** haplo37 has joined #openstack-kolla | 01:45 | |
| *** huikang has quit IRC | 01:45 | |
| *** g3ek has joined #openstack-kolla | 01:46 | |
| *** huikang has joined #openstack-kolla | 01:46 | |
| *** hieulq has joined #openstack-kolla | 01:47 | |
| *** huikang has quit IRC | 01:50 | |
| *** tonanhngo has joined #openstack-kolla | 01:54 | |
| *** tonanhngo has quit IRC | 01:55 | |
| *** daneyon has joined #openstack-kolla | 01:56 | |
| *** huikang has joined #openstack-kolla | 01:57 | |
| *** Jeffrey4l has joined #openstack-kolla | 01:58 | |
| *** phuongnh has joined #openstack-kolla | 02:00 | |
| *** daneyon has quit IRC | 02:01 | |
| openstackgerrit | dupengfei proposed openstack/kolla: Support shared path for glance images and nova instances disk https://review.openstack.org/372420 | 02:02 |
| *** tonanhngo has joined #openstack-kolla | 02:07 | |
| *** tonanhngo has quit IRC | 02:08 | |
| *** unicell has quit IRC | 02:08 | |
| MarMat_ | Jeffrey4l magnum trustee re-review please please? | 02:11 |
| Jeffrey4l | Got. will do. | 02:11 |
| MarMat_ | thanks | 02:11 |
| openstackgerrit | zhubingbing proposed openstack/kolla: add ceilometer congress to precheck.yml https://review.openstack.org/378150 | 02:12 |
| Jeffrey4l | np :0 | 02:12 |
| Jeffrey4l | :) | 02:12 |
| MarMat_ | have posted couple of new bug reports regarding magnum, not that severe as the trustee issue though | 02:13 |
| Jeffrey4l | MarMat_, did u file a bug about that? | 02:14 |
| MarMat_ | the trustee issue? There is an older one open and in progress https://bugs.launchpad.net/kolla/+bug/1551992 | 02:15 |
| openstack | Launchpad bug 1551992 in kolla "Magnum failed to create trustee" [Critical,In progress] - Assigned to Martin Matyáš (martinx-maty) | 02:15 |
| Jeffrey4l | about this "have posted couple of new bug reports regarding magnum" | 02:16 |
| MarMat_ | https://bugs.launchpad.net/kolla/+bug/1628335 | 02:16 |
| openstack | Launchpad bug 1628335 in kolla "magnum-conductor container failing behind a proxy" [Undecided,New] | 02:16 |
| MarMat_ | https://bugs.launchpad.net/kolla/+bug/1628353 | 02:16 |
| openstack | Launchpad bug 1628353 in kolla "heat failing when using trusted user" [Undecided,New] | 02:16 |
| Jeffrey4l | cool. will u try to fix that? | 02:17 |
| MarMat_ | those 2 are not covered in current PS and not intended so | 02:17 |
| MarMat_ | the heat yes, it's trivial | 02:17 |
| MarMat_ | the proxy thing needs some architectural decision | 02:18 |
| Jeffrey4l | MarMat_, re proxy: need we have a talk with other guys? | 02:18 |
| MarMat_ | Jeffrey4l initially started discussion with sdake, it yet needs discussion | 02:19 |
| Jeffrey4l | MarMat_, could u explain why upgrade shade to 1.5? should we backport it to mitaka? | 02:20 |
| Jeffrey4l | ok. it saw the commit message. | 02:21 |
| MarMat_ | Jeffrey4l yep | 02:21 |
| MarMat_ | Jeffrey4l need to leave, will read your comments in the morning :-) | 02:23 |
| Jeffrey4l | MarMat_, LGTM you may want add `co-author-by` in commit message, if u want. | 02:23 |
| Jeffrey4l | i will +2 | 02:23 |
| duonghq | MarMat_ so 2 trustee bug is same of not? sorry, losing sleep last night | 02:23 |
| *** awiddersheim has quit IRC | 02:23 | |
| MarMat_ | duonghq heat trustee issue is different and trivial to fix or workaround | 02:25 |
| duonghq | okay | 02:25 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 02:25 |
| *** jtriley has quit IRC | 02:28 | |
| *** awiddersheim has joined #openstack-kolla | 02:28 | |
| *** MarMat_ has quit IRC | 02:31 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: add congress api to haproxy https://review.openstack.org/378169 | 02:32 |
| *** huikang has quit IRC | 02:35 | |
| *** huikang has joined #openstack-kolla | 02:35 | |
| *** salv-orlando has joined #openstack-kolla | 02:36 | |
| *** salv-orlando has quit IRC | 02:40 | |
| *** huikang has quit IRC | 02:40 | |
| *** neilus has joined #openstack-kolla | 02:48 | |
| *** neilus has quit IRC | 02:52 | |
| *** zhubingbing has quit IRC | 02:53 | |
| *** huhaoran has joined #openstack-kolla | 02:55 | |
| *** zhubingbing has joined #openstack-kolla | 02:55 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Update the location of Glance resigtry checks https://review.openstack.org/378183 | 02:56 |
| *** mark-casey11 has quit IRC | 02:58 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Update the location of Glance registry checks https://review.openstack.org/378183 | 03:02 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 03:03 |
| *** david-lyle has quit IRC | 03:04 | |
| *** nihilifer has quit IRC | 03:06 | |
| *** jtriley has joined #openstack-kolla | 03:11 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Update the location of Glance Registry HAProxy checks https://review.openstack.org/378183 | 03:14 |
| *** dcwangmit01_ has joined #openstack-kolla | 03:16 | |
| *** sdake_ has quit IRC | 03:17 | |
| *** dcwangmit01 has quit IRC | 03:19 | |
| *** unicell has joined #openstack-kolla | 03:21 | |
| *** aswadr_ has joined #openstack-kolla | 03:21 | |
| *** MarMat_ has joined #openstack-kolla | 03:22 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: add etcd to precheck https://review.openstack.org/378210 | 03:31 |
| openstackgerrit | Martin Matyas proposed openstack/kolla: Fix Magnum trustee issues https://review.openstack.org/326163 | 03:34 |
| *** sdake has joined #openstack-kolla | 03:39 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: add congress api to haproxy https://review.openstack.org/378169 | 03:41 |
| *** daneyon has joined #openstack-kolla | 03:45 | |
| *** dave-mccowan has quit IRC | 03:47 | |
| *** daneyon has quit IRC | 03:50 | |
| *** sdake_ has joined #openstack-kolla | 03:50 | |
| *** sdake has quit IRC | 03:51 | |
| sdake_ | sup peeps | 03:51 |
| *** huikang has joined #openstack-kolla | 03:52 | |
| *** zhubingbing has quit IRC | 03:53 | |
| *** bjolo has joined #openstack-kolla | 03:55 | |
| *** tonanhngo has joined #openstack-kolla | 03:56 | |
| *** zhubingbing has joined #openstack-kolla | 03:57 | |
| *** g3ek has quit IRC | 03:58 | |
| *** haplo37 has quit IRC | 04:00 | |
| *** haplo37 has joined #openstack-kolla | 04:01 | |
| *** g3ek has joined #openstack-kolla | 04:01 | |
| duonghq | evening sdake_ | 04:03 |
| openstackgerrit | zhubingbing proposed openstack/kolla: add congress api to haproxy https://review.openstack.org/378169 | 04:03 |
| openstackgerrit | Martin Matyas proposed openstack/kolla: Fix wrong heat trustee configuration https://review.openstack.org/378221 | 04:04 |
| *** huikang has quit IRC | 04:07 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Introducing common-lib and common_volumes macro https://review.openstack.org/378223 | 04:08 |
| *** huikang has joined #openstack-kolla | 04:08 | |
| *** zhubingbing has quit IRC | 04:09 | |
| *** huikang_ has joined #openstack-kolla | 04:09 | |
| *** huikang has quit IRC | 04:13 | |
| *** huikang_ has quit IRC | 04:14 | |
| MarMat_ | sdake_ Jeffrey4l or anyone can we discuss the magnum proxy issue https://bugs.launchpad.net/kolla/+bug/1628335 | 04:21 |
| openstack | Launchpad bug 1628335 in kolla "magnum-conductor container failing behind a proxy" [Undecided,New] | 04:21 |
| *** jtriley has quit IRC | 04:24 | |
| openstackgerrit | Martin Matyas proposed openstack/kolla: Fix Magnum trustee issues https://review.openstack.org/326163 | 04:24 |
| sdake_ | MarMat_ shoot | 04:29 |
| sdake_ | duonghq sup | 04:29 |
| MarMat_ | sdake_ well, is the issue understood? | 04:30 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Add profile usage to kolla-build guide https://review.openstack.org/378230 | 04:30 |
| sdake_ | MarMat_ reading bug | 04:30 |
| sdake_ | MarMat_ give me moment to rpoces pls | 04:30 |
| duonghq | does mitaka support external journal drive? (ceph) | 04:30 |
| MarMat_ | sdake_ ok | 04:30 |
| duonghq | I mean kolla mitaka | 04:31 |
| sdake_ | duonghq my gut says no- i thin kwe added that feature in newton | 04:32 |
| sdake_ | duonghq check release notes webssite | 04:32 |
| duonghq | roger sdake_ | 04:33 |
| sdake_ | MarMat_ processed | 04:34 |
| sdake_ | MarMat_ i don't have any easy anwers for you | 04:34 |
| *** bdaca has joined #openstack-kolla | 04:34 | |
| sdake_ | MarMat_ it seems like conductor is only service that requires internets during runtime | 04:34 |
| bdaca | hello all | 04:35 |
| sdake_ | possibly also kuryr since it uses etcd | 04:35 |
| sdake_ | huikang may have some ideas here | 04:35 |
| sdake_ | hey bdaca | 04:35 |
| openstackgerrit | caoyuan proposed openstack/kolla: Add watcher checking to port_checks.yml https://review.openstack.org/378233 | 04:35 |
| *** neilus has joined #openstack-kolla | 04:36 | |
| MarMat_ | sdake_ proposal: configure proxy vars in global.yml and consume in start.yml tasks for container start where it's needed? | 04:36 |
| sdake_ | MarMat_ i see some problems with that | 04:36 |
| sdake_ | we like to keep the footprint of globals.yml small so that it is easy to understand | 04:36 |
| sdake_ | and those variables would need to be defined there | 04:37 |
| sdake_ | that said, the footprint of globals.yml has grown a bit over time for mandatory reasons and this seems like a mandatory reason | 04:37 |
| MarMat_ | sdake_ maybe it's a generic thing that can be potentially needed for more services | 04:37 |
| sdake_ | MarMat_ how about make a patch and bounce it off the review ueue | 04:37 |
| sdake_ | right i just said that above ;) | 04:38 |
| sdake_ | it affects kuryr as much as magnum | 04:38 |
| sdake_ | effects | 04:38 |
| sdake_ | whatever ;) | 04:38 |
| MarMat_ | sdake_ today I'm already exhausted from other magnum patchsets but will take a look on it tomorrow :-) | 04:39 |
| sdake_ | roger that | 04:39 |
| sdake_ | we are all tired ;) | 04:39 |
| sdake_ | we get some good r&r after summit | 04:39 |
| sdake_ | wil you be at summit btw MarMat_ ? | 04:40 |
| *** neilus has quit IRC | 04:41 | |
| sdake_ | MarMat_ re implementation details we have a slew of ansible experts on the team | 04:41 |
| sdake_ | MarMat_ i'm not a super ansible expert like others | 04:41 |
| sdake_ | MarMat_ might ru nthe implementation idea by Jeffrey4l | 04:41 |
| *** unicell has quit IRC | 04:41 | |
| sdake_ | or just write the patch and let the review process sort itself out | 04:42 |
| *** unicell has joined #openstack-kolla | 04:42 | |
| MarMat_ | nope, at time of summit I'll be on vacation moreover i'm not that experienced kolla guy, my first patchets are still waiting for a merge :-) | 04:43 |
| MarMat_ | sdake_ ^^^ | 04:43 |
| sdake_ | MarMat_ anyone is welomce at the summit sessions | 04:44 |
| sdake_ | butroger on the vacation | 04:44 |
| * sdake_ wtb a vacation :) | 04:44 | |
| sdake_ | so you will have an extra week of real r&R :) | 04:44 |
| *** unicell1 has joined #openstack-kolla | 04:48 | |
| *** unicell has quit IRC | 04:51 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Add config.yml and start.yml to upgrade.yml https://review.openstack.org/378237 | 04:52 |
| *** zhubingbing_ has joined #openstack-kolla | 04:53 | |
| *** unicell1 has quit IRC | 04:55 | |
| *** unicell has joined #openstack-kolla | 04:56 | |
| *** haplo37- has quit IRC | 05:00 | |
| sdake_ | if anyone sees pbourke today | 05:01 |
| sdake_ | let him know the schedule is in the topic - oct10th-13th = 2.5 weeks to fix the problems | 05:02 |
| sdake_ | Jeffrey4l or zhubingbing_ ^^ | 05:02 |
| zhubingbing_ | hi | 05:02 |
| zhubingbing_ | i am here | 05:02 |
| *** haplo37_ has joined #openstack-kolla | 05:02 | |
| sdake_ | can you pass along tht message | 05:02 |
| sdake_ | he said we have 1 week left earlier | 05:02 |
| sdake_ | then probably crashed for the day :) | 05:03 |
| zhubingbing_ | ok | 05:03 |
| zhubingbing_ | That's what we need to deal with. | 05:04 |
| zhubingbing_ | oct13th Time to deal with all the problems | 05:05 |
| Jeffrey4l | roger | 05:12 |
| sdake_ | sdake -> crashing night all | 05:13 |
| zhubingbing_ | crashing night all what‘s mean ？ －） | 05:16 |
| sdake_ | it means i'm going to bed :) | 05:17 |
| zhubingbing_ | ok | 05:17 |
| zhubingbing_ | good night | 05:17 |
| sdake_ | thanks | 05:17 |
| *** salv-orlando has joined #openstack-kolla | 05:18 | |
| *** salv-orlando has quit IRC | 05:22 | |
| *** sdake_ has quit IRC | 05:30 | |
| *** unicell has quit IRC | 05:32 | |
| *** unicell1 has joined #openstack-kolla | 05:32 | |
| *** msimonin has joined #openstack-kolla | 05:36 | |
| bjolo | morning | 05:37 |
| *** msimonin1 has joined #openstack-kolla | 05:43 | |
| *** msimonin has quit IRC | 05:46 | |
| *** unicell has joined #openstack-kolla | 05:58 | |
| *** unicell1 has quit IRC | 05:59 | |
| *** duonghq_ has joined #openstack-kolla | 06:01 | |
| *** duonghq has quit IRC | 06:04 | |
| *** egonzalez90 has joined #openstack-kolla | 06:10 | |
| *** msimonin1 has quit IRC | 06:12 | |
| egonzalez90 | morning | 06:13 |
| openstackgerrit | Merged openstack/kolla: Use merge_configs for gnocchi-api-paste.ini file https://review.openstack.org/375295 | 06:17 |
| openstackgerrit | Merged openstack/kolla: Rename the gnocchi api-paste filename https://review.openstack.org/376092 | 06:18 |
| *** neilus has joined #openstack-kolla | 06:24 | |
| *** daneyon has joined #openstack-kolla | 06:27 | |
| *** duonghq_ is now known as duonghq | 06:27 | |
| *** neilus has quit IRC | 06:29 | |
| *** daneyon has quit IRC | 06:32 | |
| *** huhaoran has quit IRC | 06:35 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Fix template for multiple physical networks https://review.openstack.org/378276 | 06:40 |
| *** tonanhngo has quit IRC | 06:40 | |
| openstackgerrit | Merged openstack/kolla: add congress api to haproxy https://review.openstack.org/378169 | 06:45 |
| *** mnasiadka has joined #openstack-kolla | 06:46 | |
| *** salv-orlando has joined #openstack-kolla | 06:50 | |
| *** zhubingbing_ has quit IRC | 06:52 | |
| *** tonanhngo has joined #openstack-kolla | 06:54 | |
| *** jrist has joined #openstack-kolla | 06:54 | |
| *** tonanhngo has quit IRC | 06:56 | |
| *** jrist has quit IRC | 07:04 | |
| *** gfidente has joined #openstack-kolla | 07:07 | |
| *** gfidente has joined #openstack-kolla | 07:07 | |
| *** huhaoran has joined #openstack-kolla | 07:09 | |
| *** matrohon has joined #openstack-kolla | 07:10 | |
| *** MarMat_ has quit IRC | 07:12 | |
| *** tonanhngo has joined #openstack-kolla | 07:13 | |
| *** tonanhngo has quit IRC | 07:14 | |
| *** msimonin has joined #openstack-kolla | 07:19 | |
| *** msimonin1 has joined #openstack-kolla | 07:23 | |
| *** msimonin has quit IRC | 07:24 | |
| *** b_bezak has joined #openstack-kolla | 07:29 | |
| *** zhubingbing_ has joined #openstack-kolla | 07:29 | |
| *** neilus has joined #openstack-kolla | 07:30 | |
| *** neilus has quit IRC | 07:35 | |
| *** mewald has joined #openstack-kolla | 07:42 | |
| *** salv-orl_ has joined #openstack-kolla | 07:43 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: add ceilometer congress to precheck.yml https://review.openstack.org/378150 | 07:44 |
| *** mewald has quit IRC | 07:45 | |
| *** mewald1 has joined #openstack-kolla | 07:45 | |
| *** salv-orlando has quit IRC | 07:46 | |
| *** strigazi_AFK is now known as strigazi | 07:49 | |
| *** Serlex has joined #openstack-kolla | 07:50 | |
| *** msimonin1 has quit IRC | 07:51 | |
| *** msimonin has joined #openstack-kolla | 07:51 | |
| *** duonghq has quit IRC | 07:54 | |
| *** zhubingbing_ has quit IRC | 07:56 | |
| *** zhubingbing_ has joined #openstack-kolla | 07:57 | |
| *** zhubingbing_ has quit IRC | 08:01 | |
| *** zhubingbing_ has joined #openstack-kolla | 08:01 | |
| *** duonghq has joined #openstack-kolla | 08:10 | |
| zhubingbing_ | hi | 08:11 |
| zhubingbing_ | guys | 08:11 |
| *** hfu has joined #openstack-kolla | 08:14 | |
| *** neilus has joined #openstack-kolla | 08:23 | |
| *** neilus has quit IRC | 08:23 | |
| *** tonanhngo has joined #openstack-kolla | 08:24 | |
| *** pbourke has joined #openstack-kolla | 08:26 | |
| *** tonanhngo has quit IRC | 08:26 | |
| *** rstarmer has joined #openstack-kolla | 08:30 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: add kuryr port in precheck.yml https://review.openstack.org/378375 | 08:32 |
| bjolo | morning zhubingbing_ | 08:33 |
| zhubingbing_ | hi | 08:33 |
| zhubingbing_ | moring | 08:33 |
| zhubingbing_ | morning | 08:33 |
| zhubingbing_ | -) | 08:33 |
| bjolo | :) | 08:35 |
| openstackgerrit | zhubingbing proposed openstack/kolla: add kuryr to precheck https://review.openstack.org/378375 | 08:35 |
| *** sdake has joined #openstack-kolla | 08:41 | |
| *** sdake has quit IRC | 08:42 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: add magnum and mistral to precheck https://review.openstack.org/378394 | 08:44 |
| *** tonanhngo has joined #openstack-kolla | 08:46 | |
| *** haplo37 has quit IRC | 08:47 | |
| *** tonanhngo has quit IRC | 08:47 | |
| *** openstackgerrit has quit IRC | 08:48 | |
| *** openstackgerrit has joined #openstack-kolla | 08:49 | |
| *** haplo37 has joined #openstack-kolla | 08:49 | |
| duonghq | anybody use kolla-build on centos with firewalld is turn off | 08:51 |
| duonghq | for build ceph image | 08:51 |
| bdaca | yes | 09:12 |
| bdaca | duonghq what problem do you have? | 09:12 |
| duonghq | bdaca: it got trouble in fetching some package | 09:13 |
| duonghq | *packages | 09:13 |
| duonghq | I'm not sure it's network or fw problem | 09:15 |
| duonghq | I'm supporting other person | 09:15 |
| pbourke | duonghq: it's built as part of the centos gate | 09:17 |
| pbourke | (also oraclelinux) | 09:17 |
| duonghq | pbourke: I'm sorry but I don't get your point | 09:18 |
| pbourke | duonghq: you're asking if anyone is using kolla-build on centos to build ceph | 09:18 |
| duonghq | ah, understood | 09:18 |
| pbourke | duonghq: Im saying jenkins is in infra | 09:18 |
| duonghq | sorry for laggy | 09:18 |
| pbourke | are they building behind a proxy? | 09:19 |
| pbourke | can they build other images | 09:19 |
| pbourke | ? | 09:19 |
| duonghq | no, they have direct internet access | 09:19 |
| duonghq | but they claim the connection is not good | 09:19 |
| duonghq | other images was built successful | 09:20 |
| duonghq | the jenkins in infra has fw off or on? | 09:20 |
| pbourke | that im not sure of | 09:22 |
| pbourke | if they can build other images it's unlikely to be a firewall issue | 09:22 |
| pbourke | if you can paste the build error we might be able to help more | 09:22 |
| duonghq | after they turn off the fw, everything is ok, | 09:23 |
| duonghq | and deploy success | 09:23 |
| duonghq | seems that I must reproduce it by myself | 09:24 |
| duonghq | they're turning in validation cluster | 09:25 |
| rstarmer | wacky, I'm suddenly seeing "firewall" issues on newly installed bare metal machines. services come up with "no route to host" issues. | 09:27 |
| rstarmer | but it's not clear if that's due to the way I'm building my systems. | 09:28 |
| *** tonanhngo has joined #openstack-kolla | 09:29 | |
| duonghq | rstarmer: centos 7? | 09:29 |
| *** tonanhngo has quit IRC | 09:30 | |
| rstarmer | yeah, new built hardware machines, deployed with Cobbler, PXE | 09:30 |
| rstarmer | well, actually they are RHEL 7.2 | 09:30 |
| rstarmer | but basically same difference | 09:30 |
| *** schwicht_at_work has joined #openstack-kolla | 09:31 | |
| *** schwicht has quit IRC | 09:32 | |
| duonghq | I didn't work with RHEL yet, but as I know, the difference is not negligible | 09:32 |
| *** berendt has joined #openstack-kolla | 09:39 | |
| rstarmer | anway, I'm seeing failures in mutli-node deployment where services can't connect to peers (Specifically mariadb) | 09:39 |
| rstarmer | trying firewall disable | 09:39 |
| *** aswadr_ has quit IRC | 09:42 | |
| *** hfu has quit IRC | 09:42 | |
| *** haplo37_ has quit IRC | 09:44 | |
| *** hfu has joined #openstack-kolla | 09:46 | |
| *** haplo37_ has joined #openstack-kolla | 09:47 | |
| *** hfu has quit IRC | 09:50 | |
| duonghq | pbourke: the bug https://bugs.launchpad.net/kolla/+bug/1621153 should be mark as invalid or leave as confirmed? | 09:51 |
| openstack | Launchpad bug 1621153 in kolla "Deprecate Debian" [Medium,Confirmed] - Assigned to Christian Berendt (berendt) | 09:51 |
| pbourke | duonghq: how about 'wont fix' ? | 09:52 |
| berendt | pbourke i changed it to opinion | 09:52 |
| berendt | and will reference the mailing list thread | 09:52 |
| berendt | we will revisit this at the end of the next cycle | 09:52 |
| pbourke | good with me | 09:53 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Deprecate Fedora based Docker images https://review.openstack.org/369184 | 09:53 |
| *** huhaoran has quit IRC | 09:53 | |
| duonghq | berendt, pbourke: thanks | 09:53 |
| berendt | done | 09:54 |
| duonghq | berendt: I think the deprecation should w-1 ultil newton is branch out? | 09:54 |
| berendt | regarding the fedora deprection, is the release note sufficient? | 09:54 |
| duonghq | due to deprecation policy, we can remove it in newton | 09:55 |
| berendt | really? i thought we have to keep it in newton | 09:55 |
| duonghq | *can not | 09:55 |
| duonghq | sorry, typo, I mean can not | 09:55 |
| berendt | ah :) | 09:55 |
| berendt | deprecation in newton, removal in ocata | 09:56 |
| *** matrohon has quit IRC | 09:56 | |
| duonghq | understood | 09:56 |
| zhubingbing_ | berendt Debian is not going to cancel it？ | 09:58 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Deprecate Fedora based Docker images https://review.openstack.org/369184 | 10:00 |
| berendt | zhubingbing_ what do you mean? | 10:00 |
| zhubingbing_ | sorry， i see wrong | 10:01 |
| zhubingbing_ | sorry | 10:01 |
| openstackgerrit | dupengfei proposed openstack/kolla: Support shared path for glance images and nova instances disk https://review.openstack.org/372420 | 10:01 |
| *** zhurong has quit IRC | 10:02 | |
| *** mewald1 has quit IRC | 10:06 | |
| *** msimonin has quit IRC | 10:08 | |
| *** lujinluo has quit IRC | 10:13 | |
| *** neilus has joined #openstack-kolla | 10:23 | |
| *** neilus has quit IRC | 10:27 | |
| *** mewald has joined #openstack-kolla | 10:28 | |
| *** skramaja has quit IRC | 10:31 | |
| *** duonghq has quit IRC | 10:31 | |
| *** skramaja has joined #openstack-kolla | 10:31 | |
| *** rstarmer has quit IRC | 10:39 | |
| *** awiddersheim has quit IRC | 10:39 | |
| *** salv-orlando has joined #openstack-kolla | 10:40 | |
| *** salv-orl_ has quit IRC | 10:40 | |
| openstackgerrit | prameswar proposed openstack/kolla: patch that will check hardware acceleration support, if not support then it will add entry virt_type=qemu in nova.conf https://review.openstack.org/378518 | 10:41 |
| *** zhubingbing_ has quit IRC | 10:43 | |
| *** awiddersheim has joined #openstack-kolla | 10:44 | |
| *** mewald1 has joined #openstack-kolla | 10:45 | |
| *** mewald has quit IRC | 10:47 | |
| openstackgerrit | Merged openstack/kolla: Add newline in grafana template https://review.openstack.org/376837 | 10:51 |
| *** rmart04 has joined #openstack-kolla | 10:56 | |
| *** phuongnh has quit IRC | 10:58 | |
| *** daneyon has joined #openstack-kolla | 10:58 | |
| *** schwicht has joined #openstack-kolla | 10:59 | |
| *** schwicht_at_work has quit IRC | 11:02 | |
| *** haplo37 has quit IRC | 11:02 | |
| *** daneyon has quit IRC | 11:03 | |
| *** haplo37 has joined #openstack-kolla | 11:08 | |
| *** awiddersheim has quit IRC | 11:15 | |
| *** dcwangmit01_ has quit IRC | 11:15 | |
| *** domi0077 has joined #openstack-kolla | 11:19 | |
| domi0077 | hi all, I've been experimenting with kolla in the past couple of days, got my first AIO installation done, but now I can't figure out one thing: if I change any of the config options in let's say /etc/kolla/nova-compute/nova.conf how can I apply these changes to the running container? | 11:20 |
| *** dcwangmit01 has joined #openstack-kolla | 11:20 | |
| berendt | domi0077 there is a reconfigure command in kolla-ansible | 11:21 |
| *** dwalsh has joined #openstack-kolla | 11:21 | |
| domi0077 | berendt: If I run that it changes my config changes back to the original values | 11:22 |
| domi0077 | I was under the impression I just need to edit the config files directly not the playbooks | 11:22 |
| berendt | domi0077 that's the way to do it, yes | 11:22 |
| berendt | domi0077 have you modified already deployed configuration files? | 11:23 |
| berendt | http://docs.openstack.org/developer/kolla/advanced-configuration.html#openstack-service-configuration-in-kolla | 11:23 |
| domi0077 | berendt: yes | 11:24 |
| berendt | you have to use overlay configuration files, do not modify already deployed files | 11:26 |
| domi0077 | could you point me to any documentation about this please? | 11:26 |
| berendt | have a look at the url i send 3 minutes ago... | 11:26 |
| domi0077 | oh I see, I missed the point where it's mentioned I need to create /etc/kolla/config hence my stupidity, sorry | 11:29 |
| mliima_ | morning guys | 11:30 |
| *** schwicht has quit IRC | 11:34 | |
| *** zhurong has joined #openstack-kolla | 11:41 | |
| *** tonanhngo has joined #openstack-kolla | 11:43 | |
| *** salv-orlando has quit IRC | 11:44 | |
| *** tonanhngo has quit IRC | 11:44 | |
| *** jrist has joined #openstack-kolla | 11:49 | |
| domi0077 | berendt: I'm sorry to bother again, but it doesn't work. Here are my steps: created /etc/kolla/config/nova-compute/nova.conf added a section and a change to it (DEFAULT and debug = True), then modified globals.yml to include the node_custom_config variable, then run kolla-ansible reconfigure...still no change to the config file | 11:54 |
| *** berendt has quit IRC | 11:54 | |
| *** msimonin has joined #openstack-kolla | 11:59 | |
| *** huhaoran has joined #openstack-kolla | 11:59 | |
| *** msimonin has quit IRC | 12:01 | |
| *** sdake has joined #openstack-kolla | 12:02 | |
| *** huhaoran has quit IRC | 12:04 | |
| *** haplo37_ has quit IRC | 12:05 | |
| *** b_bezak_ has joined #openstack-kolla | 12:06 | |
| *** haplo37_ has joined #openstack-kolla | 12:07 | |
| *** b_bezak_ has quit IRC | 12:09 | |
| *** b_bezak has quit IRC | 12:09 | |
| *** dave-mccowan has joined #openstack-kolla | 12:09 | |
| *** tonanhngo has joined #openstack-kolla | 12:12 | |
| sdake | morning peeps | 12:12 |
| sdake | pbourke did you get my messae | 12:12 |
| *** tonanhngo has quit IRC | 12:12 | |
| *** huhaoran has joined #openstack-kolla | 12:19 | |
| *** rhallisey has joined #openstack-kolla | 12:26 | |
| *** zhubingbing has joined #openstack-kolla | 12:28 | |
| *** salv-orlando has joined #openstack-kolla | 12:33 | |
| pbourke | sdake: i did thanks for that | 12:41 |
| sdake | pbourke hopefully that givessyou a bit more time :) | 12:43 |
| kfox1111 | sdake: do you have other things with docker in the gate? | 12:45 |
| kfox1111 | I think I'm htting the redhat docker starting from within cloud init boot ordering thing. | 12:45 |
| *** egonzalez90 has quit IRC | 12:46 | |
| *** domi0077 has quit IRC | 12:47 | |
| *** tonanhngo has joined #openstack-kolla | 12:53 | |
| *** tonanhngo has quit IRC | 12:55 | |
| *** david-lyle has joined #openstack-kolla | 12:57 | |
| *** egonzalez90 has joined #openstack-kolla | 12:58 | |
| sdake | uh | 12:59 |
| sdake | 6am | 12:59 |
| sdake | kfox1111 mind expanding while my brain catches up | 12:59 |
| sdake | do i have another example o docker gating? | 13:00 |
| *** rstarmer has joined #openstack-kolla | 13:00 | |
| *** zhubingbing has quit IRC | 13:01 | |
| sdake | kfox1111 docker starts when we tell it to start via setup_gate.sh | 13:02 |
| sdake | and its decendents | 13:02 |
| sdake | our centos images have very little on them preinstalled | 13:03 |
| *** rstarmer has quit IRC | 13:04 | |
| sdake | we should be using docker-engine, red hat's docker is sort of forky and has alot of problems for us | 13:04 |
| *** schwicht has joined #openstack-kolla | 13:05 | |
| sdake | e.g. red hat's docker rquires a text confirmation for each image push operation | 13:05 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 13:05 |
| sdake | YES, are you sure you want to push this image?" | 13:05 |
| sdake | I'm not sure who thought that was a good idea but its absolutely terrible | 13:05 |
| *** portdirect has joined #openstack-kolla | 13:07 | |
| *** bdaca has quit IRC | 13:07 | |
| *** lrensing has joined #openstack-kolla | 13:08 | |
| sdake | in generla i don't like confirmation questions | 13:08 |
| sdake | i typed the ommand, I want it to happen | 13:09 |
| sdake | but obviously the case of permanent destruction is different | 13:09 |
| sdake | sometimes for eample I wish yum update asked "are you really surey ou want to do that" | 13:09 |
| sdake | because fedora upgrades would often implode my system into a reintall :( | 13:09 |
| *** inc0 has joined #openstack-kolla | 13:16 | |
| *** berendt has joined #openstack-kolla | 13:17 | |
| *** berendt has quit IRC | 13:17 | |
| *** aswadr_ has joined #openstack-kolla | 13:18 | |
| inc0 | good morning | 13:18 |
| sdake | morning | 13:18 |
| sdake | asgood s it is | 13:19 |
| sdake | i feel hungover and i didn't have anything to drink | 13:19 |
| inc0 | sdake, you know Terry Pratchet? | 13:19 |
| inc0 | a writer? | 13:19 |
| sdake | terry? | 13:19 |
| sdake | do not | 13:19 |
| inc0 | ok, he came up with idea of reverse wine, wine which works back in time | 13:20 |
| sdake | loads of people i dot know | 13:20 |
| inc0 | first hangover then drinking | 13:20 |
| inc0 | you should drink or you can break time-space continuum | 13:20 |
| sdake | i feel not lik edrinking :) | 13:20 |
| sdake | must be coming down with a cold | 13:21 |
| sdake | wife was sick on her way out of town | 13:21 |
| *** salv-orlando has quit IRC | 13:21 | |
| sdake | as she abandoned me with the children for playtime | 13:21 |
| sdake | 3 weeks of travel fforher, followed by barcelona | 13:22 |
| *** jtriley has joined #openstack-kolla | 13:29 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 13:31 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Deprecate Fedora based Docker images https://review.openstack.org/369184 | 13:36 |
| *** athomas has joined #openstack-kolla | 13:38 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 13:43 |
| bjolo | sdake, inc0 can i get triage on this one please https://bugs.launchpad.net/kolla/+bug/1627717 | 13:43 |
| openstack | Launchpad bug 1627717 in kolla "mariadb_recovery fails and data loss" [Undecided,New] - Assigned to Jeffrey Zhang (jeffrey4l) | 13:43 |
| sdake | bjolo you can | 13:43 |
| sdake | bjolo done | 13:45 |
| bjolo | tnx | 13:45 |
| sdake | bjolo i agree with your assertions as wel | 13:45 |
| sdake | hopefully not too tricky to fix | 13:45 |
| bjolo | hope not | 13:46 |
| inc0 | bjolo, that's expected behaviour tho | 13:46 |
| inc0 | it's not a bug | 13:46 |
| bjolo | should not be. the playbook works given that it does recovery on the right node | 13:46 |
| inc0 | Galera works in a way, that when you save data, it waits till acks from majority of quorum | 13:47 |
| inc0 | to prevent net-partition caused conflicts in data | 13:47 |
| *** awiddersheim has joined #openstack-kolla | 13:47 | |
| inc0 | when you put down majority of quorum, minority (1 node in our case) effectively becomes RO | 13:47 |
| *** eaguilar has joined #openstack-kolla | 13:47 | |
| inc0 | and that's what is happening in your case | 13:47 |
| inc0 | if you repeat this excersize, but keep 2 nodes alive | 13:48 |
| inc0 | it should save the data | 13:48 |
| bjolo | inc0, i dont share that understanding. to my knowledge galera is strongly concistent | 13:48 |
| bjolo | it save the data even with only one node up | 13:48 |
| inc0 | it couldnt | 13:48 |
| bjolo | yes | 13:48 |
| inc0 | because if you lose connection to 2 other nodes | 13:48 |
| bjolo | at least in my testing | 13:49 |
| sdake | i cqan, but not consistently | 13:49 |
| inc0 | but they are alive and well | 13:49 |
| bjolo | shutdown two nodes, write to db, and read from db works | 13:49 |
| *** MarMat has joined #openstack-kolla | 13:49 | |
| sdake | right tht shuld work | 13:49 |
| inc0 | yes, but it doesn't land on disk | 13:49 |
| sdake | inc0 is talking about a theoretical problem | 13:49 |
| SamYaple | inc0: thats wrong | 13:49 |
| inc0 | try to do it fully manual | 13:49 |
| SamYaple | inc0: you are right for teh overall point, but it does land on disk | 13:49 |
| inc0 | SamYaple, how so? I thought commit requires ack from majority | 13:49 |
| *** MarMat has quit IRC | 13:50 | |
| SamYaple | nah its the only one up so its the majority | 13:50 |
| bjolo | there is no quroum in galara | 13:50 |
| *** MarMat has joined #openstack-kolla | 13:50 | |
| bjolo | ack is confirmed when all participating nodes are synced | 13:50 |
| openstackgerrit | prameswar proposed openstack/kolla: moved changes to precheck https://review.openstack.org/378518 | 13:50 |
| SamYaple | bjolo: is right about that point | 13:50 |
| inc0 | ofc there is quorum in galera | 13:51 |
| inc0 | that's why we need odd number of nodes | 13:51 |
| *** mnasiadka has quit IRC | 13:51 | |
| SamYaple | inc0: only for starting the cluster | 13:51 |
| SamYaple | its not like ceph quorum | 13:51 |
| SamYaple | where everything stops | 13:51 |
| SamYaple | when the net partition stops galera will attempt to merge (on a row level) all the dbs again | 13:52 |
| rhallisey | SamYaple, oh hi | 13:52 |
| inc0 | and use quorum majority only for conflicts? | 13:52 |
| SamYaple | inc0: nah it doesnt solve any conflicts for you | 13:52 |
| SamYaple | inc0: it jsut breaks | 13:52 |
| inc0 | how fun | 13:52 |
| SamYaple | inc0: its not the greatest | 13:53 |
| SamYaple | rhallisey: o/ | 13:53 |
| SamYaple | inc0: it does allow for high performance in most of all cases though | 13:53 |
| SamYaple | rhallisey: im just visting | 13:53 |
| bjolo | since we are on the topic of mariadb. whats the process for a new bp/feature request? https://blueprints.launchpad.net/kolla/+spec/database-backup-recovery | 13:53 |
| *** narasimha_SV_ has joined #openstack-kolla | 13:54 | |
| rhallisey | SamYaple, well, nice to see you back in town :) | 13:54 |
| narasimha_SV_ | http://paste.openstack.org/show/583301/ this is file for the server which it is getting generated for the server I am doing pxe booting http://paste.openstack.org/show/583302/ contents inside the id shown in above link am I missing anything here which is making my pxe boot stop after ramdisk installation http://paste.openstack.org/show/583303/ console output where it is getting stopped | 13:54 |
| narasimha_SV_ | the above links are the file inside ironic-pxe container | 13:54 |
| inc0 | ok...then I don't have idea why this bug happends | 13:55 |
| SamYaple | inc0: galera bug? | 13:57 |
| SamYaple | i should turn off my galera word notification | 13:57 |
| *** fguillot has joined #openstack-kolla | 13:57 | |
| SamYaple | oh https://bugs.launchpad.net/kolla/+bug/1627717 | 13:58 |
| openstack | Launchpad bug 1627717 in kolla "mariadb_recovery fails and data loss" [Critical,Confirmed] - Assigned to Jeffrey Zhang (jeffrey4l) | 13:58 |
| *** mewald1 has quit IRC | 13:58 | |
| SamYaple | yea bjolo there is a proper way to do recovery for a full stop | 13:58 |
| SamYaple | i bet thats not being done | 13:58 |
| inc0 | he does mariadb_recovery | 13:58 |
| *** mewald1 has joined #openstack-kolla | 13:58 | |
| SamYaple | yea that is only for lightout outage | 13:59 |
| SamYaple | if it was a proper shutdown that can lose data too | 13:59 |
| SamYaple | lightsout outage tehre is no way to know if it was a proper shutdown | 13:59 |
| bjolo | or an upgrade that fails | 13:59 |
| SamYaple | so you will probably lose data | 13:59 |
| SamYaple | so bjolo, there is a file called "grastate.dat" which tracks the clusters posistion | 14:00 |
| SamYaple | with a powerfailure, it will be -1 | 14:00 |
| SamYaple | with a proper shutdown it will be a large number | 14:00 |
| SamYaple | the largest number should be started first | 14:00 |
| SamYaple | thats not baked into the playbooks unless someone added it after me | 14:00 |
| *** mewald1 has quit IRC | 14:01 | |
| sdake | SamYaple it was not added after you | 14:01 |
| bjolo | doing upgrades with writes to db ongoing, tend to fail the upgrade since it does not seem to do it serial and wait for galara in sync before doing the next node | 14:01 |
| SamYaple | then it isn't there. i only wrote the lights out piece since the grastate.dat checking was complicated and that needed to be in mitaka(?) | 14:01 |
| sdake | SamYaple roger that - nice work there | 14:02 |
| inc0 | SamYaple, well, we didn't know about that;) | 14:02 |
| inc0 | mind sharing this in bug in question? | 14:02 |
| bjolo | i have a testvector trying to break the upgrade and lose data. If i can make it fail i will file a bug for that as well :) | 14:02 |
| inc0 | I'm pretty sure we won't be able to merge it in newton, but we can start working on it and backport it in O-1 | 14:03 |
| SamYaple | i believe it was mentioned. but i know for a fact the blueprint i worked on was called "lights-out" | 14:03 |
| inc0 | to mitaka and newton | 14:03 |
| sdake | ya its a defect bjolo - neeeds backport | 14:03 |
| sdake | not sure if we can hit it in newton or not - Jeffrey4l on it | 14:03 |
| bjolo | is it possible to get this in newton? https://blueprints.launchpad.net/kolla/+spec/database-backup-recovery | 14:03 |
| sdake | bjolo newton is closed to features | 14:04 |
| bjolo | ok | 14:04 |
| sdake | bjolo since that feature is vaporware atm, i'm not sure how you could get it either | 14:04 |
| bjolo | true | 14:04 |
| sdake | bjolo what people tpically do is carry a patchset of featurs they need for kolla | 14:05 |
| sdake | we can't backport features because of stable follows policy | 14:05 |
| SamYaple | inc0: kfox1111 http://paste.openstack.org/show/583306/ it was talked about at least once <---- | 14:05 |
| sdake | not optimal, but kolla is not a product- its a project | 14:05 |
| bjolo | i hear ya | 14:06 |
| inc0 | SamYaple, fair enough, just explain this in bug pretty please?;) | 14:06 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 14:06 |
| SamYaple | inc0: im just going to paste that conversation. just for you :) | 14:06 |
| inc0 | thank you;) | 14:07 |
| *** mewald has joined #openstack-kolla | 14:09 | |
| *** haplo37_ has quit IRC | 14:10 | |
| *** narasimha_SV_ has quit IRC | 14:10 | |
| SamYaple | inc0: alrighty updated with all my info that i know about the subject in the ticket. its not a solution, just the problem at hand | 14:12 |
| SamYaple | imma go back to lurking and turn off some of the words that ping me :) | 14:12 |
| sdake | inc0 say after the team meetin handoff today, i need toj et | 14:12 |
| sdake | inc0 internal meeting needing attention | 14:12 |
| *** haplo37_ has joined #openstack-kolla | 14:13 | |
| pbourke | galera | 14:13 |
| pbourke | ceph | 14:13 |
| *** tonanhngo has joined #openstack-kolla | 14:13 | |
| inc0 | sdake, your call;) meetings aren't obligatory for you any more | 14:13 |
| sdake | tonanhngo any chance you can get someone in the magnum community to respond on dane's thread? | 14:13 |
| sdake | they should be obligatory for all core reviewers if theey aren't sleeping ;) | 14:13 |
| inc0 | SamYaple, thanks, don't turn off words;) | 14:14 |
| sdake | but anyway have conflict ;) | 14:14 |
| *** tonanhngo has quit IRC | 14:14 | |
| sdake | some core reviewers always have a standing conflict - for e.g. daviey has a conflict with our timeslot | 14:14 |
| SamYaple | /kick pbourke | 14:15 |
| sdake | but our timeslot is prime, i have concerns moving git around will not get us more involvement and we will permanently lose our prime slot ;) | 14:15 |
| pbourke | rocket | 14:15 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 14:17 |
| *** salv-orlando has joined #openstack-kolla | 14:20 | |
| *** huhaoran has quit IRC | 14:24 | |
| *** zhubingbing has joined #openstack-kolla | 14:26 | |
| *** tonanhngo has joined #openstack-kolla | 14:27 | |
| tonanhngo | sdake: Hi Steve, yes we fixed the problem. He brought it up on the IRC meeting and we have revisited it a couple times. | 14:29 |
| lrensing | hello | 14:29 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 14:30 |
| sdake | tonanhngo cool thanks! | 14:34 |
| *** mdorman_ has left #openstack-kolla | 14:34 | |
| *** shardy has joined #openstack-kolla | 14:36 | |
| *** msimonin has joined #openstack-kolla | 14:38 | |
| *** caowei has quit IRC | 14:39 | |
| zhubingbing | hello guys | 14:46 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 14:46 |
| *** sean-k-mooney has joined #openstack-kolla | 14:55 | |
| sean-k-mooney | hi everyone | 14:58 |
| inc0 | hey sean-k-mooney | 14:58 |
| *** rmart04 has quit IRC | 14:58 | |
| *** eaguilar_ has joined #openstack-kolla | 14:59 | |
| sean-k-mooney | inc0: have you ever seen the the Bootstrapping ceph OSDs task fail with this output http://paste.openstack.org/show/583320/ | 14:59 |
| *** eaguila__ has joined #openstack-kolla | 14:59 | |
| inc0 | sean-k-mooney, you need to check docker logs ceph_osd on this node | 15:00 |
| *** eaguilar has quit IRC | 15:00 | |
| sean-k-mooney | ah thre is a stopped bootstrap_osd_0 continer | 15:01 |
| sean-k-mooney | INFO:__main__:Setting permissions for /etc/ceph/ceph.client.admin.keyring INFO:__main__:Writing out command to execute 2016-09-28 15:52:26.132543 7f2c66b7d700 0 monclient(hunting): authenticate timed out after 300 2016-09-28 15:52:26.132581 7f2c66b7d700 0 librados: client.admin authentication error (110) Connection timed out Error connecting to cluster: TimedOut | 15:01 |
| inc0 | check connection between osd node to mon nodes | 15:01 |
| *** zhurong has quit IRC | 15:01 | |
| inc0 | over mgmt network | 15:02 |
| inc0 | also check /etc/ceph/ceph.conf for addresses being used | 15:02 |
| sean-k-mooney | the mons and and osds are running on all nodes but cool ill look into why they cant connect. | 15:02 |
| sean-k-mooney | thanks | 15:03 |
| Jeffrey4l | hey guys | 15:03 |
| *** eaguilar_ has quit IRC | 15:04 | |
| pbourke | sean-k-mooney: i actually just saw that | 15:04 |
| sean-k-mooney | pbourke: i have been hitting this ceph issue for 2 weeks but i have also been on vaction. i hit it just before i left | 15:05 |
| sean-k-mooney | im back for a few hours today so trying to see if i can get this resolved so i can get a new kolla cluster runing. | 15:06 |
| *** caowei has joined #openstack-kolla | 15:07 | |
| pbourke | sean-k-mooney: im in the middle of trying to resolve it, will let you know if come up with anything | 15:07 |
| sean-k-mooney | cool ill do the same. i half wondering if could be a proxy issue in my enviroment but that may be a redhering | 15:08 |
| *** ayoung_ has joined #openstack-kolla | 15:09 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 15:09 |
| *** Serlex has quit IRC | 15:09 | |
| *** zhubingbing has quit IRC | 15:10 | |
| *** duonghq has joined #openstack-kolla | 15:11 | |
| *** ayoung_ is now known as ayoung | 15:11 | |
| duonghq | evening guys | 15:12 |
| *** egonzalez90 has quit IRC | 15:15 | |
| wirehead_ | good morning folks. :D | 15:16 |
| *** absubram has joined #openstack-kolla | 15:18 | |
| *** egonzalez90 has joined #openstack-kolla | 15:19 | |
| Daviey | sdake: For reference, i always look at the minutes.. | 15:19 |
| Daviey | (today i can make it tho) | 15:19 |
| *** caowei has quit IRC | 15:20 | |
| rhallisey | what's up wirehead_ :) | 15:23 |
| MarMat | Jeffrey4l Hey, how to go forward with https://review.openstack.org/#/c/326163/ to merge it? I guess it needs your +2 again in the latest PS (commit message changed), right? | 15:25 |
| *** eaguila__ has quit IRC | 15:26 | |
| Jeffrey4l | MarMat, done. nice work. :) | 15:27 |
| *** apuimedo is now known as apuimedo|away | 15:27 | |
| openstackgerrit | prameswar proposed openstack/kolla: precheck to check hardware acceleration support https://review.openstack.org/378518 | 15:27 |
| *** eaguilar has joined #openstack-kolla | 15:28 | |
| MarMat | Jeffrey4l thanks a lot for your support :-) What now to merge it? | 15:28 |
| Jeffrey4l | MarMat, I already +2. need another core to review it. | 15:29 |
| MarMat | sdhake Hi, will you suggest someone additional from core team to review the magnum trustee PS? | 15:30 |
| MarMat | sdake ^^^ | 15:31 |
| inc0 | MarMat, I already approved it | 15:31 |
| *** caowei has joined #openstack-kolla | 15:32 | |
| MarMat | inc0 great, thanks. Still someone has to hit the merge button, right? | 15:34 |
| inc0 | no, it's merging as soon as gates pass | 15:34 |
| MarMat | inc0 oh, cool then | 15:34 |
| openstackgerrit | Merged openstack/kolla: Fix Magnum trustee issues https://review.openstack.org/326163 | 15:34 |
| inc0 | lrensing you should talk to kfox1111, rhallisey or sbezverk about kolls-k8s issues:) | 15:35 |
| lrensing | will do | 15:35 |
| rhallisey | lrensing, hey | 15:35 |
| lrensing | hey | 15:36 |
| openstackgerrit | prameswar proposed openstack/kolla: precheck to check hardware acceleration support https://review.openstack.org/378518 | 15:36 |
| lrensing | i'm running the minikube guide kfox1111 made | 15:36 |
| MarMat | it seems to be merged now, thanks guys for great support with that. | 15:36 |
| lrensing | and it seems like creating pods is taking a long time | 15:36 |
| lrensing | i'm not the best with kubernetes so maybe there is just a misunderstanding | 15:37 |
| rhallisey | ok let's query them and see what the issue is | 15:37 |
| lrensing | when i watch the keystone pod being created, it says desired = 1, current 1, ready 0 | 15:37 |
| rhallisey | is that the bootstrap step or the deployment? | 15:37 |
| rhallisey | seem like the bootstrao | 15:37 |
| lrensing | hmm | 15:38 |
| rhallisey | kubectl get jobs | 15:38 |
| rhallisey | let's see if keystone bootstrap job worked | 15:38 |
| *** zhubingbing has joined #openstack-kolla | 15:39 | |
| duonghq | seem that he got same problem as me | 15:39 |
| duonghq | guess that configmap cannot mount | 15:39 |
| rhallisey | ok | 15:39 |
| *** lrensing has quit IRC | 15:39 | |
| rhallisey | are they created? | 15:39 |
| rhallisey | kubectl get configmaps | 15:39 |
| *** lrensing has joined #openstack-kolla | 15:39 | |
| *** eaguilar has quit IRC | 15:40 | |
| lrensing | keystone create db , endpoints, and manage-db all succeeded | 15:40 |
| rhallisey | lrensing, did the bootstrap job complete? | 15:40 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 15:41 |
| *** eaguilar has joined #openstack-kolla | 15:41 | |
| *** tonanhngo has quit IRC | 15:41 | |
| lrensing | yes | 15:41 |
| *** hrito has joined #openstack-kolla | 15:41 | |
| sbezverk | lrensing: kubectl describe pod {keystone pod name } --namespace=kolla | 15:42 |
| rhallisey | lrensing, ok which keystone pod is failing in particular | 15:42 |
| openstackgerrit | Merged openstack/kolla: Install the required packages for ceph-base image https://review.openstack.org/374783 | 15:43 |
| rhallisey | db endpoints pods passed, is it the service that failed? | 15:43 |
| rhallisey | sbezverk, I'm getting tired of add --namespace=kolla to the end ;/ | 15:43 |
| lrensing | http://paste.openstack.org/show/583338/ | 15:44 |
| lrensing | does this mean it worked? | 15:44 |
| sbezverk | rhallisey: you shoud be able to config kubectl to use namespace kolla by default | 15:44 |
| rhallisey | ya we'll need to doc that | 15:44 |
| *** tonanhngo has joined #openstack-kolla | 15:44 | |
| lrensing | im running this command | 15:45 |
| lrensing | kolla-kubernetes resource create pod keystone | 15:45 |
| sbezverk | lrensing: please paste kubectl get pods --namespace=kolla | 15:45 |
| openstackgerrit | Merged openstack/kolla: Change Mariadb default character set to utf8 https://review.openstack.org/371876 | 15:45 |
| openstackgerrit | Merged openstack/kolla: Turn off bindep https://review.openstack.org/367054 | 15:45 |
| sbezverk | then kubectl get rc --namespace=kolla | 15:46 |
| lrensing | http://paste.openstack.org/show/583340/ | 15:46 |
| sbezverk | then kubectl describe rc keystone --namespace=kolla | 15:46 |
| rhallisey | ok that looks good | 15:46 |
| sbezverk | lrensing: but your keystone pos is running?? | 15:47 |
| lrensing | http://paste.openstack.org/show/583341/ | 15:47 |
| rhallisey | lrensing, how long did it take to get the pods up? | 15:47 |
| rhallisey | you said earlier it took awhile | 15:47 |
| sbezverk | lrensing: not sure but all look good. RC shows that it created POD successfully | 15:48 |
| lrensing | it never told me that the pods were 'ready' | 15:48 |
| rhallisey | kfox1111, I see you added ceph into the minikube-quickstart | 15:48 |
| rhallisey | lrensing, I see. Ya to see them running you would look in kubectl | 15:49 |
| lrensing | when running watch kolla-kubernetes resource status pod keystone | 15:49 |
| rhallisey | hm | 15:49 |
| *** zhubingbing has quit IRC | 15:49 | |
| rhallisey | I think that should be watch kubectl get pods | 15:49 |
| rhallisey | lrensing, good observation | 15:50 |
| lrensing | yeah | 15:50 |
| sbezverk | lrensing: Replicas:1 current / 1 desired | 15:50 |
| sbezverk | Pods Status:1 Running / 0 Waiting / 0 Succeeded / 0 Failed | 15:50 |
| lrensing | maybe just a typo in the docs? | 15:50 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Mount host localtime for ceilometer https://review.openstack.org/378730 | 15:51 |
| lrensing | http://paste.openstack.org/show/583338/ is what mine looked like | 15:51 |
| *** zhubingbing has joined #openstack-kolla | 15:52 | |
| sbezverk | lrensing: Usually it gets in this state when some resources it needs are missing, after that you check with kubectl describe rc {name} | 15:52 |
| sbezverk | to get what exactly is missing | 15:52 |
| kfox1111 | hi all. | 15:53 |
| kfox1111 | rhallisey: yeah. | 15:53 |
| kfox1111 | rhallisey: got cinder/glance fully working, backed by ceph. | 15:54 |
| rhallisey | v nice! | 15:54 |
| kfox1111 | so the instructions test the whole compute kit now. vm launch, sdn attachment and block storage. | 15:54 |
| sdake | kfox1111 nice! now to turn those instrutions into a gate ;) | 15:55 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Use kubectl to watch for pod status https://review.openstack.org/378738 | 15:55 |
| kfox1111 | sdake: I'm part of the way there. ;) | 15:55 |
| rhallisey | kfox1111, btw | 15:55 |
| kfox1111 | we need workflow to fully automate it. but I've gotten k8s deploying so far, and staring to add genconfig. | 15:55 |
| rhallisey | kfox1111, https://review.openstack.org/#/c/330226/1 | 15:55 |
| rhallisey | that might help a little | 15:56 |
| rhallisey | did that back in june so it's a bit outdated | 15:56 |
| sdake | t-3 minutes for meeting | 15:56 |
| kfox1111 | rhallisey: ah. missed that. I'll have a look. I've got a good start going here too: https://review.openstack.org/#/c/378056 | 15:57 |
| rhallisey | nice | 15:57 |
| *** unicell1 has joined #openstack-kolla | 15:58 | |
| sbezverk | kfox1111: when you have time please check common volume ps | 15:58 |
| kfox1111 | sbezverk: k | 15:58 |
| inc0 | lrensing, when you'll get through first pains, if you'd like to work in kolla-k8s, kfox1111, rhallisey and sbezverk will be good people to talk to about work items:) | 15:58 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 16:00 |
| *** unicell has quit IRC | 16:00 | |
| rhallisey | lrensing, upstream meeting is now if you're curious | 16:01 |
| rhallisey | in #openstack-meeting-4 | 16:01 |
| *** berendt has joined #openstack-kolla | 16:02 | |
| *** srwilkers has joined #openstack-kolla | 16:03 | |
| *** lamt has joined #openstack-kolla | 16:03 | |
| sean-k-mooney | pbourke: i dont know if you have made any progress on the ceph issue but from inside the ceph_mon container i still cant execute basic ceph commands sucessfully so i think the issue is with the mons not the bootstrap container | 16:04 |
| pbourke | sean-k-mooney: yeah I think so too | 16:04 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 16:04 |
| sean-k-mooney | pbourke: the strang thing is i can connect to the mons with netcat so they are listening on the socket | 16:05 |
| awiddersheim | what does PS stand for? | 16:07 |
| pbourke | awiddersheim: patchset | 16:09 |
| awiddersheim | thanks | 16:10 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 16:12 |
| *** mdnadeem_home has joined #openstack-kolla | 16:15 | |
| *** msimonin has quit IRC | 16:20 | |
| *** zhubingbing has quit IRC | 16:21 | |
| *** strigazi is now known as strigazi_AFK | 16:21 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Mount host localtime for containers https://review.openstack.org/378762 | 16:21 |
| *** caowei has quit IRC | 16:22 | |
| *** shardy has quit IRC | 16:26 | |
| *** athomas has quit IRC | 16:28 | |
| *** jrist has quit IRC | 16:28 | |
| *** haplo37_ has quit IRC | 16:29 | |
| *** haplo37_ has joined #openstack-kolla | 16:31 | |
| *** jmccarthy1 has quit IRC | 16:37 | |
| *** jmccarthy has joined #openstack-kolla | 16:38 | |
| *** eaguilar has quit IRC | 16:39 | |
| *** sean-k-mooney has quit IRC | 16:44 | |
| bjolo | sdake, inc0 triage on this as well please https://bugs.launchpad.net/kolla/+bug/1627275 | 16:50 |
| openstack | Launchpad bug 1627275 in kolla "docker stop keepalived does not release VIPs" [Undecided,New] | 16:50 |
| *** gfhellma has joined #openstack-kolla | 16:50 | |
| *** unicell1 has quit IRC | 16:50 | |
| *** dwalsh has quit IRC | 16:52 | |
| Daviey | inc0: The performance monitoring stack isn't currently working... I raised a bug with the bad memcache config line.... but even with that fixed, telegraf pushing all the data is getting back a 500 error. | 17:00 |
| Daviey | So no data loaded | 17:00 |
| inc0 | damn:S | 17:00 |
| inc0 | ok, we need to file bugs for that | 17:01 |
| Daviey | Grafana isn't plugged into Influxdb OOTB, you need to config that by hand | 17:01 |
| duonghq | bjolo: iirc there is a bug like that | 17:01 |
| Daviey | and we have no default graphs setup | 17:01 |
| berendt | Daviey there are some more issues with Telegraf, i think it can not be used in an HA manner | 17:01 |
| berendt | (without buying licenses) | 17:01 |
| Daviey | berendt: what is HA in this context ? | 17:01 |
| inc0 | influx cannot, true | 17:01 |
| berendt | influxdb, not telegraf.. sorry | 17:01 |
| Daviey | ah | 17:02 |
| Daviey | Does that mean each control node is running a unique datasource? | 17:02 |
| bjolo | duonghq, ? | 17:02 |
| berendt | sbezverk regarding fluentd, i think the best way to proceed is to split the review into docker/ansible and to open/rename the referenced bp to fluentd-image-for-k8s | 17:03 |
| inc0 | https://blueprints.launchpad.net/kolla/+spec/heka-deprecation | 17:03 |
| berendt | sbezverk this way we can use the kill-heka bp for the ansible role when we decided to use fluentd for kolla as well | 17:03 |
| rhallisey | lrensing, ok so you have an issue with ovs | 17:03 |
| lrensing | so rhallisey i am running into an issue with the openvswitch-set-external-ip pod | 17:03 |
| duonghq | bjolo: got you point, it should be another bug | 17:03 |
| rhallisey | lrensing, ok let's see what you started | 17:03 |
| rhallisey | check the jobs | 17:03 |
| berendt | Daviey this means at the moment you have to deploy influxdb on a single node | 17:03 |
| lrensing | http://paste.openstack.org/show/583346/ | 17:04 |
| lrensing | so, it failed | 17:04 |
| Daviey | berendt: but kolla doesn't enforce that? (unlike mongodb?) | 17:04 |
| berendt | Daviey i hope the we require HA for each of our services | 17:05 |
| rhallisey | lrensing, k let's check the pods | 17:05 |
| duonghq | bjolo: can you fix that? | 17:05 |
| rhallisey | lrensing, and the containers | 17:05 |
| lrensing | http://paste.openstack.org/show/583347/ | 17:05 |
| berendt | Daviey and at the moment all our services are HA ready | 17:05 |
| Daviey | berendt: So, grafana supports elasticsearch... why arent we just using that instead of influx? | 17:05 |
| sbezverk | berendt: I will split that ps | 17:05 |
| berendt | Daviey i was not involved in this discussion, i think it was before my time in the kolla project | 17:06 |
| Daviey | yeah, i think only by a couple of weeks | 17:06 |
| berendt | sbezverk ok i think this way we can approve the fluentd image pretty fast | 17:06 |
| rhallisey | lrensing, ok let's check the logs of that container | 17:06 |
| berendt | Daviey but the whole monitoring stack in kolla needs a lot of love | 17:07 |
| sbezverk | lrensing: always run kubectl get pods --namesapce=kolla | 17:07 |
| sbezverk | lrensing: you will know whether it is crashing or what else might be happening | 17:07 |
| lrensing | yeah i did sbezverk | 17:07 |
| berendt | Daviey and i think it is not a big deal to replace TIG by something else, at the moment only the serivces itself are implemented | 17:07 |
| lrensing | crashloop | 17:07 |
| sbezverk | lrensing: that job is for bring up ip interface and assign address | 17:08 |
| duonghq | night all, see you tomorrow | 17:08 |
| rhallisey | lrensing, kubectl logs openvswitch-set-external-ip-6ln1j | 17:08 |
| *** duonghq has left #openstack-kolla | 17:08 | |
| sbezverk | could you confirm that ovsdb and vswitchd containers are up and running fir | 17:08 |
| *** hrito has quit IRC | 17:08 | |
| sbezverk | rhallisey do not forget --namesapce=kolla ;-0 | 17:09 |
| lrensing | hmm | 17:09 |
| lrensing | RTNETLINK answers: File exists | 17:09 |
| lrensing | did it succeed and i did it twice on accident? | 17:09 |
| *** papacz has joined #openstack-kolla | 17:09 | |
| lrensing | http://paste.openstack.org/show/583350/ | 17:10 |
| sbezverk | lrensing: could you please run: kubectl get pods --namespace=kolla | grep openv | 17:10 |
| rhallisey | sbezverk, ^ you've seen that issue before | 17:10 |
| kfox1111 | lrensing: I think that worked then. | 17:10 |
| lrensing | http://paste.openstack.org/show/583351/ | 17:10 |
| rhallisey | the pod is crashing | 17:10 |
| kfox1111 | hmmm... | 17:10 |
| *** papacz1 has quit IRC | 17:11 | |
| kfox1111 | on the host, do an 'ifconfig br-ex' and see if it has an ip and is up. | 17:11 |
| kfox1111 | we may need abit of logic to pass if its already configured. | 17:11 |
| sbezverk | kfox1111: it should not crashed | 17:12 |
| kfox1111 | I agree. but it may be throwing an error as the interface is already up. | 17:13 |
| sbezverk | lrensing: can you run: docker ps -a | grep openvswitch-set-external | 17:13 |
| sbezverk | then docker logs {container id} | 17:13 |
| kfox1111 | sbezverk: that error message looks like the one given by ip when a link is already up and asked to be brought up. | 17:14 |
| rhallisey | brb | 17:15 |
| kfox1111 | so it may be returning a fail code to the shell, and failing to run the job, but everythings ok. | 17:15 |
| *** mdnadeem_home has quit IRC | 17:16 | |
| *** dwalsh has joined #openstack-kolla | 17:16 | |
| lrensing | does the minikube guide install docker? | 17:17 |
| *** egonzalez90 has quit IRC | 17:18 | |
| kfox1111 | docker and all of k8s. | 17:18 |
| kfox1111 | minikube builds a vm with docker/k8s/k8s advanced services installed. | 17:18 |
| kfox1111 | the minikube instructions deploy on top of that. | 17:18 |
| *** gfhellma has quit IRC | 17:20 | |
| *** pbourke has quit IRC | 17:23 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: integrate gnocchi with ceilometer https://review.openstack.org/372737 | 17:23 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix error in Gnocchi service https://review.openstack.org/378927 | 17:23 |
| *** unicell has joined #openstack-kolla | 17:23 | |
| *** berendt has quit IRC | 17:27 | |
| *** apuimedo|away is now known as apuimedo | 17:34 | |
| *** tonanhngo has quit IRC | 17:35 | |
| *** lrensing has quit IRC | 17:38 | |
| *** aswadr_ has quit IRC | 17:42 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Introducing common-lib and common_volumes macro https://review.openstack.org/378223 | 17:44 |
| *** neilus has joined #openstack-kolla | 17:44 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Integrate gnocchi with ceph https://review.openstack.org/374937 | 17:44 |
| inc0 | ooo gnocci knows how to ceph? didn't know that | 17:47 |
| Jeffrey4l | gnocchi support file/swift and ceph now inc0 | 17:51 |
| *** lrensing has joined #openstack-kolla | 17:52 | |
| inc0 | cool! | 17:52 |
| inc0 | we need to make ceilometer support gnocci then:) | 17:52 |
| Jeffrey4l | inc0, this https://review.openstack.org/372737 | 17:53 |
| inc0 | hmm, I'm thinking if we should wait for ocata with these patches | 17:54 |
| inc0 | they're more features than bugs tbh | 17:55 |
| Jeffrey4l | inc0, l'd like to merge them. 1. they already there. just need some small change. 2. i think we are still before rc2 , we can add feature before rc2 | 17:56 |
| sdake | here is the deal on features and depednencies | 17:56 |
| inc0 | Jeffrey4l, no, no features in rc2 | 17:57 |
| sdake | we have always had an unwritten policy that if a dep needs to change to support a feature its nto a featuree | 17:57 |
| inc0 | only bugs | 17:57 |
| sdake | its in fact a bug - because what we have doesn't work | 17:57 |
| sdake | now i haven't looked at theseparticular patche | 17:57 |
| inc0 | so I'm on fence with this one | 17:57 |
| sdake | firehose day for me | 17:57 |
| inc0 | ceilometer works, with mongodb and mysql | 17:58 |
| sdake | but keep that guidancein mind | 17:58 |
| inc0 | gnocci works, with file and swift | 17:58 |
| inc0 | we can make gnocci work with ceph and ceilometer work with gnocci | 17:58 |
| inc0 | but lack of driver support is not a bug | 17:58 |
| Jeffrey4l | we scheduled to release rc2 on 15 sep. and before that day, we merged plenty of feature before that day. | 17:58 |
| inc0 | rc1 was sep15 | 17:58 |
| inc0 | rc2 is oct10 | 17:59 |
| inc0 | and rc1 was feature code freeze | 17:59 |
| *** eaguilar has joined #openstack-kolla | 17:59 | |
| sdake | ya this doessn't fit into the depndency argument in my mind | 17:59 |
| Jeffrey4l | ok you are right. | 18:00 |
| sdake | for example magnum requires updating shade | 18:00 |
| sdake | so we updateshade | 18:00 |
| sdake | thats not a feature | 18:00 |
| *** msimonin has joined #openstack-kolla | 18:00 | |
| sdake | thats a bug fix to make magnum work | 18:00 |
| inc0 | let's please bump these 2 to Ocata | 18:00 |
| sdake | black and white | 18:00 |
| *** msimonin has quit IRC | 18:01 | |
| Jeffrey4l | inc0, gnocchi only work with file, no swift now. And he do not support two gnocchi node two. ( file driver do not sync the data ) | 18:01 |
| *** msimonin has joined #openstack-kolla | 18:01 | |
| Jeffrey4l | so gnocchi is totally bust and can not be used | 18:01 |
| *** neilus has quit IRC | 18:01 | |
| *** msimonin has quit IRC | 18:01 | |
| *** eaguilar has quit IRC | 18:03 | |
| *** eaguilar has joined #openstack-kolla | 18:04 | |
| *** eaguilar_ has joined #openstack-kolla | 18:08 | |
| sdake | ok well thts a bug | 18:10 |
| sdake | two options | 18:10 |
| sdake | merge patch, or document gnocchi is bust | 18:10 |
| *** eaguilar has quit IRC | 18:11 | |
| rhallisey | phew. Just triages every bp and bug | 18:11 |
| *** daneyon has joined #openstack-kolla | 18:11 | |
| rhallisey | triaged* | 18:11 |
| * rhallisey get's lunch | 18:12 | |
| *** berendt has joined #openstack-kolla | 18:13 | |
| *** daneyon has quit IRC | 18:16 | |
| *** jheroux has joined #openstack-kolla | 18:18 | |
| *** portdirect has quit IRC | 18:20 | |
| *** bmace_ has quit IRC | 18:21 | |
| openstackgerrit | Martin Matyas proposed openstack/kolla: [WIP] Proxy settings for containers that need internet access https://review.openstack.org/378949 | 18:21 |
| *** bmace_ has joined #openstack-kolla | 18:21 | |
| MarMat | sdake proposal for the container proxy implementation we discussed yesterday https://review.openstack.org/#/c/378949/ | 18:22 |
| *** neilus has joined #openstack-kolla | 18:24 | |
| inc0 | hmm, I'm thinking how to make it more extendable | 18:25 |
| inc0 | so with kolla_docker we can specify environment, and I bet more than magnum would like to have this env in container | 18:25 |
| *** portdirect has joined #openstack-kolla | 18:27 | |
| *** Jeffrey4l has quit IRC | 18:28 | |
| *** tonanhngo has joined #openstack-kolla | 18:28 | |
| *** neilus has quit IRC | 18:29 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Introducing common-lib and common_volumes macro https://review.openstack.org/378223 | 18:30 |
| *** lrensing has quit IRC | 18:31 | |
| MarMat | inc0 that's tricky, maybe not all containers need proxy settings, maybe for some it's explicitly not wanted? In this case we can specify containers where we want to set the proxy | 18:32 |
| inc0 | yeah, but it can be important across the board | 18:33 |
| inc0 | however, bad news, it's going to Ocata...this if feature;) | 18:34 |
| *** salv-orlando has quit IRC | 18:34 | |
| MarMat | inc0 well, it depends, it can be also a bugfix https://bugs.launchpad.net/kolla/+bug/1628335 | 18:36 |
| openstack | Launchpad bug 1628335 in kolla "magnum-conductor container failing behind a proxy" [High,In progress] - Assigned to Martin Matyáš (martinx-maty) | 18:36 |
| inc0 | lol why magnum has to connect to internet? oO | 18:37 |
| inc0 | this will break in so many ways in so many scenerios... | 18:37 |
| MarMat | inc0 read the bug report | 18:37 |
| inc0 | we already have etcd container for kuryr...will have same effect? | 18:38 |
| MarMat | inc0 I guess so, this is how etcd works | 18:38 |
| inc0 | so one thing you can do | 18:38 |
| inc0 | for newton with container customization you can specify ENV variable per container | 18:39 |
| inc0 | and bake it into container | 18:39 |
| inc0 | so your customization file would look like: | 18:39 |
| MarMat | inc0 that's not fitting to my use case, I need images independent on deployment target network | 18:40 |
| inc0 | hmm, let me think for a moment | 18:41 |
| inc0 | so you want to reuse images on many different setups? | 18:41 |
| MarMat | inc0 yes, I need universal images. Similarly you cannot set IP setup or nw interface setup to images, with proxy settings it's similar. | 18:44 |
| inc0 | so issue I'm having with this patchset is that it will be magnum specific and config is not magnum specific | 18:44 |
| MarMat | inc0 it's not magnum specific, any container can use it if it needs internet access | 18:45 |
| MarMat | if kuryr needs it, 1 line can be added to kuryr kolla_docker | 18:46 |
| inc0 | the way you added is | 18:46 |
| *** sdake has quit IRC | 18:46 | |
| inc0 | so what you can do is | 18:46 |
| inc0 | https://github.com/openstack/kolla/blob/master/ansible/group_vars/all.yml#L88 | 18:46 |
| inc0 | override this in your globals.yml | 18:47 |
| inc0 | granted, all of containers will have this ENV | 18:47 |
| inc0 | but if you make your proxies right that should be ok | 18:47 |
| inc0 | you can override this section in your globals.yml and add new env vars | 18:48 |
| inc0 | just remember to re-declare all of existing stuff;) | 18:48 |
| inc0 | next step would be to allow to re-declare it on per-role basis | 18:49 |
| MarMat | inc0 yep, this can probably work now/for newton, but is that a good final solution of the problem? With respect of easy-to-set in globals.yml and having it for all containers | 18:50 |
| inc0 | no, not perfect solution | 18:50 |
| inc0 | however we need to figure out which will be perfect | 18:50 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Introducing common-lib and common_volumes macro https://review.openstack.org/378223 | 18:50 |
| inc0 | we can implement this mechanism in ocata | 18:50 |
| *** rhallisey has quit IRC | 18:50 | |
| inc0 | in the meantime, check if that works | 18:50 |
| MarMat | inc0 sounds good, I'll test that approach. | 18:51 |
| inc0 | I agree we need to have something like that, but I'd rather do it across the board | 18:51 |
| inc0 | sincce we don't know what else may need it later on | 18:51 |
| *** rhallisey has joined #openstack-kolla | 18:51 | |
| inc0 | bbiaf | 18:53 |
| *** inc0 has quit IRC | 18:53 | |
| *** mewald has quit IRC | 18:59 | |
| *** hogepodge has quit IRC | 19:01 | |
| *** eaguilar_ has quit IRC | 19:02 | |
| *** inc0 has joined #openstack-kolla | 19:04 | |
| *** eaguilar has joined #openstack-kolla | 19:06 | |
| *** huhaoran has joined #openstack-kolla | 19:06 | |
| *** berendt has quit IRC | 19:08 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 19:12 |
| *** berendt has joined #openstack-kolla | 19:12 | |
| *** papacz1 has joined #openstack-kolla | 19:13 | |
| *** papacz has quit IRC | 19:15 | |
| *** msimonin has joined #openstack-kolla | 19:19 | |
| *** rstarmer has joined #openstack-kolla | 19:20 | |
| *** rstarmer has quit IRC | 19:22 | |
| *** neilus has joined #openstack-kolla | 19:22 | |
| *** salv-orlando has joined #openstack-kolla | 19:22 | |
| *** neilus has quit IRC | 19:25 | |
| *** sdake has joined #openstack-kolla | 19:34 | |
| sdake | MarMat can you run tht by inc0, he has far more experience with proxy then me | 19:35 |
| sdake | MarMat i don't have a proxy to deal with so haven't any experienc there | 19:35 |
| MarMat | sdake sure, in progress | 19:35 |
| *** neilus has joined #openstack-kolla | 19:36 | |
| *** neilus has quit IRC | 19:37 | |
| sdake | We were able to deploy a sizeable OpenStack HA cluster in about half an hour. Thank you very much to you and the Kolla team. | 19:40 |
| sdake | private email - good job guys1 | 19:40 |
| rhallisey | sdake, nice | 19:40 |
| inc0 | :) brave new world in deployment;) | 19:41 |
| *** bjolo has quit IRC | 19:41 | |
| *** salv-orl_ has joined #openstack-kolla | 19:43 | |
| *** neilus has joined #openstack-kolla | 19:45 | |
| *** salv-orlando has quit IRC | 19:46 | |
| *** MarMat has quit IRC | 19:55 | |
| *** daneyon has joined #openstack-kolla | 19:59 | |
| *** berendt has quit IRC | 20:04 | |
| *** daneyon has quit IRC | 20:05 | |
| *** berendt has joined #openstack-kolla | 20:07 | |
| *** salv-orl_ has quit IRC | 20:07 | |
| *** salv-orlando has joined #openstack-kolla | 20:08 | |
| *** inc0 has quit IRC | 20:09 | |
| *** inc0 has joined #openstack-kolla | 20:11 | |
| *** mewald has joined #openstack-kolla | 20:16 | |
| *** huhaoran has quit IRC | 20:25 | |
| *** MarMat has joined #openstack-kolla | 20:33 | |
| *** lrensing has joined #openstack-kolla | 20:39 | |
| *** berendt has quit IRC | 20:42 | |
| *** ccesario has quit IRC | 20:43 | |
| *** eaguilar has quit IRC | 20:43 | |
| *** neilus has quit IRC | 20:54 | |
| *** Marx314 has joined #openstack-kolla | 21:00 | |
| *** eaguilar has joined #openstack-kolla | 21:02 | |
| *** jtriley has quit IRC | 21:06 | |
| *** neilus has joined #openstack-kolla | 21:06 | |
| *** awiddersheim has quit IRC | 21:06 | |
| sbezverk | inc0: ping | 21:07 |
| inc0 | here | 21:07 |
| inc0 | brb | 21:08 |
| *** neilus has quit IRC | 21:10 | |
| *** mewald1 has joined #openstack-kolla | 21:17 | |
| *** mewald has quit IRC | 21:20 | |
| *** eaguilar has quit IRC | 21:20 | |
| *** eaguilar has joined #openstack-kolla | 21:27 | |
| *** hogepodge has joined #openstack-kolla | 21:28 | |
| *** gfidente has quit IRC | 21:31 | |
| *** schwicht has quit IRC | 21:32 | |
| *** jheroux has quit IRC | 21:36 | |
| *** dwalsh has quit IRC | 21:41 | |
| *** absubram has quit IRC | 21:42 | |
| sbezverk | inc0: about fluentd, unfortunately separating fluentd docker and ansible will not work for us | 21:47 |
| inc0 | how so? | 21:47 |
| sbezverk | the reason being is when we render we need to render the image name | 21:48 |
| sbezverk | I had to add this to actually get it rendering correctly | 21:48 |
| sbezverk | http://paste.openstack.org/show/583391/ | 21:48 |
| *** daneyon has joined #openstack-kolla | 21:48 | |
| *** eaguilar has quit IRC | 21:48 | |
| sbezverk | inc0: since it is like heka and we do not have a separate resource for it, in case of kube it runs as a sidecar with main resource | 21:49 |
| inc0 | I don't understand | 21:49 |
| inc0 | what do you mean by render? | 21:50 |
| sbezverk | inc0: kolla-kubernetes renders jinja2 variables we use in templates | 21:50 |
| *** rhallisey has quit IRC | 21:51 | |
| sbezverk | it checks also kolla's yml files for some common variables like tag and bunch of others | 21:51 |
| sbezverk | when it renders it uses resource name when it looks for variable on kolla side | 21:51 |
| inc0 | ahh you mean we can't have 2 different logging architectures | 21:51 |
| inc0 | we don't want this anyway. | 21:51 |
| inc0 | so personally I think fluentd will be good way to go | 21:52 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Introducing common-lib and common_volumes macro https://review.openstack.org/378223 | 21:52 |
| inc0 | but I'd like to examine at least this filebeat | 21:52 |
| inc0 | however, regardless what we choose | 21:52 |
| inc0 | I would like both kollas to have same architecture | 21:52 |
| inc0 | and toolset | 21:52 |
| sbezverk | inc0: sure thing.. | 21:52 |
| inc0 | so when we fix one we fix another | 21:53 |
| inc0 | if filebeat turns out to be hands down better than fluentd | 21:53 |
| inc0 | then we'll move kolla-k8s to filebeat as well | 21:53 |
| *** daneyon has quit IRC | 21:53 | |
| sbezverk | ok, not a prob. but my prob is a little bit different. have you checked paste ? | 21:54 |
| inc0 | yes, you need these lines in asnible right? | 21:54 |
| sbezverk | yes | 21:54 |
| inc0 | so 2 things | 21:54 |
| inc0 | 1. yes in ocata | 21:54 |
| sbezverk | I cannot do without them :-( | 21:54 |
| sbezverk | but fine for ocata | 21:55 |
| *** schwicht has joined #openstack-kolla | 21:55 | |
| sbezverk | if you accept it | 21:55 |
| inc0 | 2. I'd love to decouple configs (and therefore variable defaults) from ansible and place them in kolla-core | 21:55 |
| inc0 | same as config files | 21:55 |
| inc0 | so you'll lose dependency on ansible and both kolla-ansible and kolla-k8s will consume these things | 21:55 |
| inc0 | but we need to examine how exactly that would work | 21:55 |
| sbezverk | when I had these variables in fluentd default it did not work | 21:55 |
| inc0 | we'll talk about it in summit ok? | 21:56 |
| sbezverk | ok sounds good | 21:56 |
| *** huikang has joined #openstack-kolla | 21:56 | |
| inc0 | we should decouple kolla-k8s from kolla-ansible totally | 21:57 |
| sbezverk | I hope we have kube related session | 21:57 |
| inc0 | 2 of them | 21:57 |
| sbezverk | super | 21:57 |
| inc0 | 2 working sessions goign to kolla-k8s | 21:57 |
| inc0 | talk with Ryan and Kevin on what you guys want to cover | 21:57 |
| *** schwicht has quit IRC | 21:57 | |
| sbezverk | do you have by chance etherpad with kolla design sessions handy by any chance? | 21:58 |
| *** lrensing has quit IRC | 21:58 | |
| inc0 | https://etherpad.openstack.org/p/kolla-o-summit-schedule | 21:58 |
| inc0 | this is current version of what I'm going to put as schedule | 21:58 |
| sbezverk | inc0: thank you | 21:58 |
| *** msimonin has quit IRC | 22:03 | |
| *** lamt has quit IRC | 22:09 | |
| *** salv-orlando has quit IRC | 22:10 | |
| *** huikang has quit IRC | 22:17 | |
| *** huikang has joined #openstack-kolla | 22:17 | |
| *** schwicht has joined #openstack-kolla | 22:17 | |
| *** huikang has quit IRC | 22:22 | |
| *** inc0 has quit IRC | 22:23 | |
| sdake | inc0 is this like a sorted list? | 22:23 |
| sdake | guess hes gone | 22:23 |
| sdake | oh well :) | 22:23 |
| *** mewald1 has quit IRC | 22:23 | |
| *** huikang has joined #openstack-kolla | 22:24 | |
| *** huikang has quit IRC | 22:24 | |
| *** huikang has joined #openstack-kolla | 22:25 | |
| *** huikang has quit IRC | 22:29 | |
| *** schwicht_at_work has joined #openstack-kolla | 22:30 | |
| *** mliima__ has joined #openstack-kolla | 22:31 | |
| *** schwicht has quit IRC | 22:33 | |
| *** tonanhngo has quit IRC | 22:33 | |
| *** mliima_ has quit IRC | 22:34 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP common_volume_mounts and common_containers macros https://review.openstack.org/379027 | 22:39 |
| *** awiddersheim has joined #openstack-kolla | 22:42 | |
| sdake | sbezverk note the actual schedule dates are not ye tavailale | 22:46 |
| sdake | sbezverk and i'm not sure all those sessions will make summit or not | 22:47 |
| sdake | sbezverk inc0 can speak to that later | 22:47 |
| sbezverk | sdake: got it will keep an eye on the schedule | 22:47 |
| sdake | sbezverk it changes right up to the last day before summit ;-) | 22:47 |
| *** salv-orlando has joined #openstack-kolla | 22:48 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate https://review.openstack.org/378056 | 22:49 |
| hogepodge | sdake: would you say that I can fix the multiple glance/cinder bug if I apply this patch to cinder and build from local source? https://review.openstack.org/#/c/306756/ | 22:51 |
| sdake | hogepodge the current state of that patch is broken imo | 22:52 |
| sdake | hogepodge patchset 1 worked | 22:52 |
| sdake | or whatever one sbezverk said hetested | 22:52 |
| sdake | current being latest revision | 22:52 |
| *** salv-orlando has quit IRC | 22:52 | |
| sdake | i think the cinder team has finallly relented and accepted that iscsi is how alot of vendors do disk io with cinder | 22:52 |
| sdake | and it needs to be a thing that works | 22:52 |
| sdake | jgriffith may know more | 22:53 |
| sdake | jgriffith he took over the work | 22:53 |
| hogepodge | sdake: broken how? | 22:54 |
| *** neilus has joined #openstack-kolla | 22:54 | |
| sdake | hogepodge it doesn't work broken | 22:54 |
| sdake | it doesn't solve the problem - the bug still exists with latest patch set | 22:54 |
| sdake | but in there is patchset that works | 22:54 |
| sdake | sbezverk says in the history patchset 2 works | 22:55 |
| *** sdake has quit IRC | 22:56 | |
| *** neilus has quit IRC | 23:00 | |
| *** kbyrne has quit IRC | 23:06 | |
| hogepodge | sdake: which is patchset 1 with an updated commit message | 23:08 |
| *** kbyrne has joined #openstack-kolla | 23:08 | |
| *** dwalsh has joined #openstack-kolla | 23:25 | |
| *** huikang has joined #openstack-kolla | 23:31 | |
| *** sdake has joined #openstack-kolla | 23:32 | |
| *** sdake_ has joined #openstack-kolla | 23:35 | |
| *** sdake has quit IRC | 23:38 | |
| *** dwalsh has quit IRC | 23:42 | |
| *** zhurong has joined #openstack-kolla | 23:46 | |
| *** sin_ has joined #openstack-kolla | 23:55 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!