| *** openstack has joined #openstack-kolla | 05:57 | |
| *** orwell.freenode.net sets mode: +ns | 05:57 | |
| *** orwell.freenode.net sets mode: -o openstack | 05:57 | |
| -orwell.freenode.net- *** Notice -- TS for #openstack-kolla changed from 1466056666 to 1410455203 | 05:57 | |
| *** orwell.freenode.net sets mode: +ct-s | 05:57 | |
| *** mbound has joined #openstack-kolla | 05:57 | |
| *** salv-orlando has joined #openstack-kolla | 05:57 | |
| *** coolsvap has joined #openstack-kolla | 05:57 | |
| *** MarMat has joined #openstack-kolla | 05:57 | |
| *** fragatin_ has joined #openstack-kolla | 05:57 | |
| *** david-lyle_ has joined #openstack-kolla | 05:57 | |
| *** tfukushima has joined #openstack-kolla | 05:57 | |
| *** ravig has joined #openstack-kolla | 05:57 | |
| *** yuanying has joined #openstack-kolla | 05:57 | |
| *** cu5 has joined #openstack-kolla | 05:57 | |
| *** diogogmt has joined #openstack-kolla | 05:57 | |
| *** zhiwei has joined #openstack-kolla | 05:57 | |
| *** Jeffrey4l has joined #openstack-kolla | 05:57 | |
| *** pumaranikar has joined #openstack-kolla | 05:57 | |
| *** ntpttr_ has joined #openstack-kolla | 05:57 | |
| *** papacz has joined #openstack-kolla | 05:57 | |
| *** dmsimard has joined #openstack-kolla | 05:57 | |
| *** kbyrne has joined #openstack-kolla | 05:57 | |
| *** cloudnull has joined #openstack-kolla | 05:57 | |
| *** jroll has joined #openstack-kolla | 05:57 | |
| *** intr1nsic has joined #openstack-kolla | 05:57 | |
| *** logan- has joined #openstack-kolla | 05:57 | |
| *** callahanca_ has joined #openstack-kolla | 05:57 | |
| *** robcresswell has joined #openstack-kolla | 05:57 | |
| *** slagle_ has joined #openstack-kolla | 05:57 | |
| *** Daviey_ has joined #openstack-kolla | 05:57 | |
| *** lukl_ has joined #openstack-kolla | 05:57 | |
| *** patchbot has joined #openstack-kolla | 05:57 | |
| *** openstackgerrit has joined #openstack-kolla | 05:57 | |
| *** harbie has joined #openstack-kolla | 05:57 | |
| *** mgkwill has joined #openstack-kolla | 05:57 | |
| *** NobodyCam has joined #openstack-kolla | 05:57 | |
| *** ccesario has joined #openstack-kolla | 05:57 | |
| *** stvnoyes has joined #openstack-kolla | 05:57 | |
| *** mgoddard has joined #openstack-kolla | 05:57 | |
| *** pbourke has joined #openstack-kolla | 05:57 | |
| *** cinerama has joined #openstack-kolla | 05:57 | |
| *** ayoung has joined #openstack-kolla | 05:57 | |
| *** timss has joined #openstack-kolla | 05:57 | |
| *** harlowja_ has joined #openstack-kolla | 05:57 | |
| *** The_Ball has joined #openstack-kolla | 05:57 | |
| *** Mech422 has joined #openstack-kolla | 05:57 | |
| *** jeh has joined #openstack-kolla | 05:57 | |
| *** wmiller has joined #openstack-kolla | 05:57 | |
| *** beekhof has joined #openstack-kolla | 05:57 | |
| *** cfarquhar has joined #openstack-kolla | 05:57 | |
| *** gmmaha has joined #openstack-kolla | 05:57 | |
| *** sgordon has joined #openstack-kolla | 05:57 | |
| *** sbezverk_ has joined #openstack-kolla | 05:57 | |
| *** Kennan has joined #openstack-kolla | 05:57 | |
| *** dcwangmit01 has joined #openstack-kolla | 05:57 | |
| *** zigo has joined #openstack-kolla | 05:57 | |
| *** elemoine_ has joined #openstack-kolla | 05:57 | |
| *** serverascode has joined #openstack-kolla | 05:57 | |
| *** DuncanT has joined #openstack-kolla | 05:57 | |
| *** peterjenkins has joined #openstack-kolla | 05:57 | |
| *** NachoDuck has joined #openstack-kolla | 05:57 | |
| *** akwasnie has joined #openstack-kolla | 05:57 | |
| *** clayton has joined #openstack-kolla | 05:57 | |
| *** tpeoples has joined #openstack-kolla | 05:57 | |
| *** lmiccini has joined #openstack-kolla | 05:57 | |
| *** ntpttr has joined #openstack-kolla | 05:57 | |
| *** macsz has joined #openstack-kolla | 05:57 | |
| *** smekel_ has joined #openstack-kolla | 05:57 | |
| *** akwasnie_ has joined #openstack-kolla | 05:57 | |
| *** ediardo has joined #openstack-kolla | 05:57 | |
| *** mdorman has joined #openstack-kolla | 05:57 | |
| *** jpeeler has joined #openstack-kolla | 05:57 | |
| *** mdnadeem_ has joined #openstack-kolla | 05:57 | |
| *** daneyon has joined #openstack-kolla | 05:57 | |
| *** jabari has joined #openstack-kolla | 05:57 | |
| *** dgonzalez has joined #openstack-kolla | 05:57 | |
| *** tuvenen has joined #openstack-kolla | 05:57 | |
| *** harmw_ has joined #openstack-kolla | 05:57 | |
| *** kong has joined #openstack-kolla | 05:57 | |
| *** vdo has joined #openstack-kolla | 05:57 | |
| *** m3m0 has joined #openstack-kolla | 05:57 | |
| *** mkoderer has joined #openstack-kolla | 05:57 | |
| *** dasm has joined #openstack-kolla | 05:57 | |
| *** bradjones has joined #openstack-kolla | 05:57 | |
| *** tonyb has joined #openstack-kolla | 05:57 | |
| *** elemoine has joined #openstack-kolla | 05:57 | |
| *** haplo37 has joined #openstack-kolla | 05:57 | |
| *** g3ek has joined #openstack-kolla | 05:57 | |
| *** wirehead_ has joined #openstack-kolla | 05:57 | |
| *** dims has joined #openstack-kolla | 05:57 | |
| *** akscram has joined #openstack-kolla | 05:57 | |
| *** nihilifer has joined #openstack-kolla | 05:57 | |
| *** anbehl has joined #openstack-kolla | 05:57 | |
| *** vincent_vdk has joined #openstack-kolla | 05:57 | |
| *** brad[] has joined #openstack-kolla | 05:57 | |
| *** wznoinsk has joined #openstack-kolla | 05:57 | |
| *** PyroMani has joined #openstack-kolla | 05:57 | |
| *** cmceniry has joined #openstack-kolla | 05:57 | |
| *** kklimonda has joined #openstack-kolla | 05:57 | |
| *** rcarrillocruz has joined #openstack-kolla | 05:57 | |
| *** apuimedo has joined #openstack-kolla | 05:57 | |
| *** jgriffith has joined #openstack-kolla | 05:57 | |
| *** Kimmo__ has joined #openstack-kolla | 05:57 | |
| *** rbergeron has joined #openstack-kolla | 05:57 | |
| *** zeus has joined #openstack-kolla | 05:57 | |
| *** jidar has joined #openstack-kolla | 05:57 | |
| *** mag009_ has joined #openstack-kolla | 05:57 | |
| *** mandre has joined #openstack-kolla | 05:57 | |
| *** afranc_ has joined #openstack-kolla | 05:57 | |
| *** EmilienM has joined #openstack-kolla | 05:57 | |
| *** ChanServ has joined #openstack-kolla | 05:57 | |
| *** ajafo has joined #openstack-kolla | 05:57 | |
| *** kwazar has joined #openstack-kolla | 05:57 | |
| *** orwell.freenode.net sets mode: +o ChanServ | 05:57 | |
| *** orwell.freenode.net changes topic to "Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA)" | 05:57 | |
| *** mbound has quit IRC | 05:59 | |
| *** z- has joined #openstack-kolla | 06:01 | |
| *** ravig has quit IRC | 06:07 | |
| *** neilus has joined #openstack-kolla | 06:19 | |
| *** daneyon_ has joined #openstack-kolla | 06:19 | |
| *** neilus has quit IRC | 06:23 | |
| *** daneyon_ has quit IRC | 06:24 | |
| *** b_bezak has joined #openstack-kolla | 06:35 | |
| *** b_bezak has joined #openstack-kolla | 06:36 | |
| *** coolsvap has quit IRC | 06:37 | |
| *** coolsvap has joined #openstack-kolla | 06:38 | |
| *** salv-orlando has quit IRC | 06:49 | |
| *** salv-orlando has joined #openstack-kolla | 06:50 | |
| *** salv-orl_ has joined #openstack-kolla | 06:55 | |
| *** salv-orlando has quit IRC | 06:57 | |
| *** neilus has joined #openstack-kolla | 07:00 | |
| *** mikelk has joined #openstack-kolla | 07:05 | |
| *** mewald has joined #openstack-kolla | 07:05 | |
| *** neilus has quit IRC | 07:05 | |
| mewald | How does Kolla allow to configure per domain backends in keystone? | 07:06 |
|---|---|---|
| *** coolsvap has quit IRC | 07:15 | |
| *** coolsvap has joined #openstack-kolla | 07:16 | |
| *** neilus has joined #openstack-kolla | 07:20 | |
| *** matrohon has joined #openstack-kolla | 07:20 | |
| *** neilus has quit IRC | 07:24 | |
| *** shardy has joined #openstack-kolla | 07:30 | |
| *** MarMat has quit IRC | 07:37 | |
| *** neilus has joined #openstack-kolla | 07:42 | |
| *** neilus has quit IRC | 07:47 | |
| *** salv-orl_ has quit IRC | 07:49 | |
| *** neilus has joined #openstack-kolla | 07:49 | |
| *** salv-orlando has joined #openstack-kolla | 07:49 | |
| *** neilus has quit IRC | 07:53 | |
| *** neilus has joined #openstack-kolla | 08:01 | |
| *** berendt has joined #openstack-kolla | 08:06 | |
| *** Serlex has joined #openstack-kolla | 08:06 | |
| *** coolsvap_ has joined #openstack-kolla | 08:07 | |
| *** athomas has joined #openstack-kolla | 08:07 | |
| *** coolsvap has quit IRC | 08:07 | |
| *** daneyon_ has joined #openstack-kolla | 08:07 | |
| *** daneyon_ has quit IRC | 08:12 | |
| *** b_bezak has quit IRC | 08:29 | |
| *** chopmann has joined #openstack-kolla | 08:32 | |
| *** chopmann is now known as chopmann_ | 08:32 | |
| *** chopmann_ has quit IRC | 08:33 | |
| *** chopmann_ has joined #openstack-kolla | 08:33 | |
| *** chopmann_ is now known as chopmann | 08:34 | |
| *** b_bezak has joined #openstack-kolla | 08:34 | |
| chopmann | is there a way to tell kolla to use "become" or "sudo" by default? | 08:35 |
| chopmann | im getting Permission denied: '/etc/kolla'", "path": "/etc/kolla//heka", "state": "absent"} | 08:35 |
| chopmann | while deploying | 08:36 |
| openstackgerrit | Han Chao proposed openstack/kolla: Format commands to the code block https://review.openstack.org/330371 | 08:39 |
| chopmann | ansible_become: yes in globals.yml does the trick | 08:40 |
| *** jamie_h has joined #openstack-kolla | 08:43 | |
| *** coolsvap_ is now known as coolsvap | 08:44 | |
| *** coolsvap has quit IRC | 08:45 | |
| *** coolsvap has joined #openstack-kolla | 08:45 | |
| *** salv-orlando has quit IRC | 08:55 | |
| *** salv-orlando has joined #openstack-kolla | 08:56 | |
| *** dmk0202 has joined #openstack-kolla | 09:02 | |
| mewald | How does Kolla allow to configure per domain backends in keystone? | 09:15 |
| *** jmccarthy has joined #openstack-kolla | 09:24 | |
| *** tloeb has joined #openstack-kolla | 09:34 | |
| *** daneyon_ has joined #openstack-kolla | 09:56 | |
| *** mbound has joined #openstack-kolla | 09:56 | |
| *** daneyon_ has quit IRC | 10:00 | |
| berendt | chopmann can you please add a note about this to the kolla documentation? | 10:01 |
| coolsvap | chopmann, its assumed to be root or equivalent | 10:02 |
| coolsvap | the documentation starts with it | 10:03 |
| coolsvap | Note Some commands below may require root permissions (e.g. pip, apt-get). | 10:03 |
| coolsvap | berendt, ^^ | 10:03 |
| berendt | coolsvap ok, i have not checked it | 10:03 |
| coolsvap | berendt np :) | 10:04 |
| coolsvap | just (log).info | 10:04 |
| *** jmccarthy has left #openstack-kolla | 10:07 | |
| openstackgerrit | Han Chao proposed openstack/kolla: Format commands to the code block https://review.openstack.org/330371 | 10:07 |
| *** jmccarthy1 has joined #openstack-kolla | 10:07 | |
| *** mgoddard_ has joined #openstack-kolla | 10:08 | |
| *** salv-orlando has quit IRC | 10:09 | |
| *** mgoddard has quit IRC | 10:11 | |
| *** pbourke has quit IRC | 10:15 | |
| *** pbourke has joined #openstack-kolla | 10:16 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Remove EC2 API related configurations and tasks https://review.openstack.org/330436 | 10:18 |
| berendt | Can somebody please review https://review.openstack.org/#/c/317124/1, is is annoying to manually create the _member_ role for horizon after a fresh deployment | 10:23 |
| patchbot | berendt: patch 317124 - kolla - add bootstrap_horizon ansible tasks and dockerfile | 10:23 |
| *** mikelk has quit IRC | 10:32 | |
| coolsvap | berendt, i agree with michal, i think it should be added in keystone bootstrap | 10:35 |
| coolsvap | some other service may need it in future | 10:36 |
| berendt | coolsvap that's fine for me, I will send an update for the review request to move it forward | 10:36 |
| *** papacz has quit IRC | 10:36 | |
| coolsvap | berendt, thanks! we can move it fast-forward | 10:37 |
| *** sdake has joined #openstack-kolla | 10:40 | |
| berendt | coolsvap does it makes sense to add it as an ansible task in the keystone role? | 10:42 |
| *** sdake_ has joined #openstack-kolla | 10:42 | |
| coolsvap | berendt, i think no, I think it will be required for kolla-kubernetes as well | 10:43 |
| *** janmalax has joined #openstack-kolla | 10:45 | |
| *** sdake has quit IRC | 10:45 | |
| *** tyrola has joined #openstack-kolla | 10:58 | |
| *** papacz has joined #openstack-kolla | 10:59 | |
| *** salv-orlando has joined #openstack-kolla | 11:07 | |
| *** tyrola has quit IRC | 11:09 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Make Neutron QoS service plugin configurable https://review.openstack.org/330470 | 11:09 |
| *** salv-orlando has quit IRC | 11:10 | |
| *** sdake_ has quit IRC | 11:10 | |
| *** salv-orlando has joined #openstack-kolla | 11:11 | |
| *** tfukushima has quit IRC | 11:17 | |
| *** mliima has joined #openstack-kolla | 11:27 | |
| *** stvnoyes has quit IRC | 11:38 | |
| *** daneyon_ has joined #openstack-kolla | 11:44 | |
| mewald | quit | 11:44 |
| *** mewald has quit IRC | 11:44 | |
| *** berendt has quit IRC | 11:45 | |
| *** salv-orlando has quit IRC | 11:45 | |
| *** salv-orlando has joined #openstack-kolla | 11:46 | |
| *** daneyon_ has quit IRC | 11:49 | |
| *** b_bezak has quit IRC | 11:58 | |
| openstackgerrit | Merged openstack/kolla: Add kolla-kubernetes bootstrap capability to mariadb https://review.openstack.org/320744 | 12:00 |
| openstackgerrit | Merged openstack/kolla: Add kolla-kubernetes bootstrap capability to Keystone https://review.openstack.org/321529 | 12:01 |
| openstackgerrit | Merged openstack/kolla: Format commands to the code block https://review.openstack.org/330371 | 12:01 |
| *** dwalsh has joined #openstack-kolla | 12:03 | |
| *** shardy has quit IRC | 12:04 | |
| *** sdake has joined #openstack-kolla | 12:05 | |
| Mech422 | Morning | 12:09 |
| *** coolsvap has quit IRC | 12:10 | |
| *** rhallisey has joined #openstack-kolla | 12:15 | |
| *** slagle_ is now known as slagle | 12:16 | |
| tloeb | good day! chopmann and me still trying to deploy openstack with ceph, now we encounter the problem that the Task "Bootstrapping Ceph OSDs" fails. The OSDs cant connect to the ceph-mon, and when we try to execute "ceph -s" in the ceph_mon container, we dont get any response. Any Ideas? | 12:18 |
| *** b_bezak has joined #openstack-kolla | 12:18 | |
| chopmann | on the osd we are getting 2016-06-16 14:14:33.692761 7f4c7595e700 0 librados: client.admin authentication error (110) Connection timed out | 12:19 |
| sdake | pbourke ping | 12:21 |
| tloeb | in the container ceph_mon "ceph --help" gives a response, only "ceph -s" and "ceph -w" doesnt work | 12:21 |
| sdake | dmsimard ping | 12:23 |
| dmsimard | o/ | 12:23 |
| sdake | hey | 12:23 |
| sdake | i looked over your data set | 12:24 |
| sdake | i noticed not many ubuntu failures | 12:24 |
| sdake | there is something uniqu about ubuntu compared to centos | 12:24 |
| sdake | we keep a local copy of the ubuntu universe mirrors in the infrasturcture | 12:24 |
| sdake | i also noticed lots of oraclelinux fialures | 12:24 |
| sdake | no idea what the trigger on those are | 12:24 |
| dmsimard | Same thing for CentOS btw | 12:24 |
| sdake | but that is a new gate and somewhat to be suspected | 12:24 |
| sdake | what is the same thing for centos | 12:25 |
| dmsimard | There is a local CentOS mirror in -infra | 12:25 |
| sdake | yes but kolla is not using it | 12:25 |
| dmsimard | Why not ? | 12:25 |
| sdake | the local mirror is a new thing - 1-2 weeks | 12:25 |
| sdake | bandwidth mostly | 12:25 |
| sdake | i did submit a patch a couple weeks ago to inspect it and havne't got back to it | 12:25 |
| chopmann | our hosts are RHEL 7.2 | 12:26 |
| sdake | anything that fails in 1 hour = timeout in the gate | 12:26 |
| sdake | dmsimard providing local mirrors will fix many of the gate issues | 12:28 |
| sdake | reverting ipv6 support in erlang would help too | 12:28 |
| dmsimard | sdake: I feel fixing your gate jobs' flapiness and making them voting should be a huge priority for the project | 12:30 |
| sdake | dmsimard i agree having reliable gates is important | 12:30 |
| dmsimard | I also learned that the integration tests don't span all of the supported projects in Kolla | 12:31 |
| sdake | dmsimard but i can' do it aloen | 12:31 |
| sdake | our integration tests are very inadequate | 12:31 |
| dmsimard | And I think that's a big deal | 12:31 |
| sdake | in all, we have about 25 test cases we want to run | 12:31 |
| sdake | atm we implement 1 test case | 12:31 |
| sdake | re integration tests, we are limited by gate memory | 12:33 |
| sdake | i've begged and pleaded for a 16gb node model but no go | 12:33 |
| dmsimard | sdake: this is the three integration tests puppet openstack runs with a very nice coverage matrix to stay within RAM constraints: https://github.com/openstack/puppet-openstack-integration/blob/master/README.md#description | 12:33 |
| *** stvnoyes has joined #openstack-kolla | 12:33 | |
| dmsimard | These tests run on Ubuntu and CentOS for every commit to every puppet module | 12:34 |
| dmsimard | https://review.openstack.org/#/c/329694/ | 12:35 |
| patchbot | dmsimard: patch 329694 - puppet-nova - Enable compute node micro-services (MERGED) | 12:35 |
| sdake | dmsimard i spent 3 hours training oflks on how the gate works | 12:36 |
| sdake | to teach the horse to drink so to speak ;) | 12:36 |
| sdake | draw your own conclusions | 12:36 |
| sdake | i thought the problem was nobody knew how our gates worked | 12:37 |
| sdake | and that indeed was one of the problems | 12:37 |
| sdake | but now that folks know HOW thegates work, i dont see any gate changes being submitted | 12:37 |
| *** athomas has quit IRC | 12:40 | |
| sdake | dmsimard fwiw before the erlang and bindep changes, our gates were near 100% working | 12:45 |
| sdake | (when code wasn't merged that was obviouslybroken) | 12:45 |
| sdake | i almost made them voting | 12:46 |
| sdake | but was waiting on the centos mirror | 12:46 |
| sdake | now centos mirror is availalbe, i think makingvoting gates is not going to happen until the gates are stable | 12:46 |
| *** athomas has joined #openstack-kolla | 12:46 | |
| sdake | many projects just run rehceck 5-10 times to get their patches past their voting gates | 12:46 |
| sdake | becaue that is how flakey rabbitmq is | 12:47 |
| *** berendt has joined #openstack-kolla | 12:49 | |
| *** mikelk has joined #openstack-kolla | 12:55 | |
| *** berendt has quit IRC | 12:57 | |
| *** berendt has joined #openstack-kolla | 12:58 | |
| *** ravig has joined #openstack-kolla | 13:02 | |
| pbourke | sdake: why is an infra centos mirror so important | 13:17 |
| pbourke | sdake: I dont understand | 13:17 |
| sdake | pbourke it keeps the builds being successful | 13:18 |
| sdake | becuasse the mirros timed out | 13:19 |
| *** jtriley has joined #openstack-kolla | 13:20 | |
| *** salv-orlando has quit IRC | 13:21 | |
| *** salv-orlando has joined #openstack-kolla | 13:21 | |
| *** v1k0d3n has joined #openstack-kolla | 13:25 | |
| *** v1k0d3n has quit IRC | 13:25 | |
| *** Guest48217 has joined #openstack-kolla | 13:26 | |
| *** ppowell has joined #openstack-kolla | 13:29 | |
| *** diogogmt has quit IRC | 13:29 | |
| *** diogogmt has joined #openstack-kolla | 13:31 | |
| *** daneyon_ has joined #openstack-kolla | 13:32 | |
| Mech422 | dmsimard: Since you appear to know so much about gates, will you be able to help sdake ? | 13:33 |
| Mech422 | btw - good morning everyone :-0 | 13:35 |
| dmsimard | Mech422: I can try to help if you have problems specific to RDO (I'm working full time on RDO) but I don't have the bandwidth to do much more | 13:35 |
| Mech422 | err... :-) | 13:35 |
| * Mech422 is ubuntu guy... | 13:35 | |
| Mech422 | for me, RDO is someplace you go to watch guys ride bulls | 13:35 |
| Mech422 | :-) | 13:35 |
| *** daneyon_ has quit IRC | 13:37 | |
| *** ppowell has quit IRC | 13:38 | |
| Mech422 | sdake: feeling better today ? | 13:39 |
| *** rmart04 has joined #openstack-kolla | 13:41 | |
| *** MarMat has joined #openstack-kolla | 13:42 | |
| *** inc0 has joined #openstack-kolla | 13:42 | |
| inc0 | good morning folks | 13:43 |
| inc0 | wassup? | 13:43 |
| Mech422 | Hey inc0 - Coffee time :-) | 13:43 |
| Mech422 | you ? | 13:44 |
| chopmann | Mech422: :-D | 13:44 |
| inc0 | I already have my lousy american tea | 13:44 |
| inc0 | actually its rooibos | 13:44 |
| inc0 | so not a tea technically | 13:44 |
| Mech422 | inc0: teas has become the new wine... | 13:45 |
| chopmann | inc0: i think its called an infusion | 13:45 |
| Mech422 | inc0: first wine snobs, then beer snobs, then coffee snobs, now tea snobs :-) | 13:45 |
| inc0 | Mech422, in Europe, where I come from, we drink a lot of tea;) | 13:46 |
| inc0 | in Poland I hardly drank any coffee | 13:46 |
| Mech422 | inc0: yeah - my parents were british...I know all bout the cuppa every 5 minutes :-P | 13:46 |
| *** ravig has quit IRC | 13:46 | |
| Mech422 | I think thats why I drik coffee - drank so much tea as a kid, I got sick of it | 13:47 |
| inc0 | lol | 13:47 |
| inc0 | I started to drink some coffee here too, one every week or so;) | 13:47 |
| Mech422 | inc0: European coffee a nice... its sorta like the japanese tea ceremony thing... | 13:48 |
| Mech422 | inc0: fancy lil cups, lots of ceremony...and you get CHOCOLATE! | 13:48 |
| inc0 | by European you mean Italian?;) | 13:48 |
| *** ravig has joined #openstack-kolla | 13:48 | |
| Mech422 | inc0: umm - my experience was mostly in Holland... | 13:48 |
| inc0 | have you ever tried turkish coffee? | 13:48 |
| inc0 | Turkey is in Europe too;) | 13:49 |
| Mech422 | inc0: I don't think so... | 13:49 |
| *** berendt has quit IRC | 13:49 | |
| Mech422 | LOL - is it? I thought europe kept trying to deny that ? | 13:49 |
| inc0 | Russia is in Europe to...partially | 13:49 |
| *** anbehl has quit IRC | 13:49 | |
| inc0 | more and more every year:D | 13:49 |
| inc0 | Turkish coffee is cool | 13:50 |
| inc0 | it's made differently than you're used to | 13:50 |
| Mech422 | inc0: oh? | 13:50 |
| inc0 | they have special jars | 13:50 |
| inc0 | they put coffe there, spices | 13:50 |
| inc0 | cold wated | 13:50 |
| *** dwalsh has quit IRC | 13:51 | |
| inc0 | and they boil the water with coffee in it 3 or 4 times | 13:51 |
| *** ppowell has joined #openstack-kolla | 13:51 | |
| Mech422 | oh - sounds interesting... | 13:51 |
| inc0 | really good | 13:51 |
| inc0 | originally, I think, they boiled it in sand | 13:51 |
| inc0 | ;) | 13:51 |
| Mech422 | inc0: I'm in the middle of BFE - so we have no ethnic resturants unless you count pizza :-P | 13:52 |
| inc0 | BFE? | 13:52 |
| Mech422 | Bum Fuck Egypt | 13:52 |
| Mech422 | slang for middle of nowhere ? | 13:52 |
| rhallisey | haha | 13:52 |
| inc0 | ok | 13:52 |
| inc0 | .. | 13:52 |
| inc0 | fair enough | 13:52 |
| Mech422 | err...yeah...I'm not allowed to speak in public :-P | 13:53 |
| inc0 | I'm sure Egiptians like s this | 13:53 |
| inc0 | Egyptians* | 13:53 |
| Mech422 | inc0: I'm sure they have similiar : BFA (Alaska) ? BFY (Yukon) ? | 13:53 |
| inc0 | don't worry Mech422, since I moved to US I became jaded to racism | 13:53 |
| inc0 | ;) | 13:53 |
| rhallisey | :/ | 13:54 |
| *** esharao__ has joined #openstack-kolla | 13:54 | |
| Mech422 | egyptians not a race - I'm a jingoist if anything :-P | 13:54 |
| chopmann | so folks, dont let this get derailed | 13:54 |
| rhallisey | hehe | 13:54 |
| chopmann | ;-) | 13:54 |
| inc0 | yeah, true | 13:54 |
| inc0 | nationalism is strong in US | 13:54 |
| inc0 | strongest I've ever seen | 13:55 |
| rhallisey | we're joking | 13:55 |
| inc0 | and I've been places | 13:55 |
| inc0 | ; | 13:55 |
| rhallisey | least I am | 13:55 |
| Mech422 | inc0: eh - I think it comes from being so 'isolated' - we only have 2 neighbors... | 13:55 |
| inc0 | well, I'm in TX so that's kinda different scale | 13:55 |
| chopmann | <--- german :D we tend to not joke about those things :P | 13:55 |
| *** dwalsh has joined #openstack-kolla | 13:55 | |
| Mech422 | inc0: eurpeans are used to switching countries like we switch states | 13:55 |
| inc0 | not really | 13:55 |
| inc0 | you switch much more | 13:55 |
| Mech422 | LOL | 13:56 |
| inc0 | language, culture and such | 13:56 |
| rhallisey | ya but it's in the same country | 13:56 |
| inc0 | it's same in US | 13:56 |
| inc0 | yeah | 13:56 |
| Mech422 | really? my family in britain is always bopping around 'the continent for holiday' | 13:56 |
| inc0 | that's what I mean | 13:56 |
| inc0 | holiday yeah, moving around, not so | 13:56 |
| inc0 | but that's true, you hardly see anything besides US | 13:56 |
| inc0 | well, you - general US population | 13:57 |
| Mech422 | inc0: its amazing how many people in us never get further then 200 miles from home | 13:57 |
| Mech422 | inc0: I grew up in the sticks (see - I didn't say BFE :-P ) - population 15,000 | 13:57 |
| Mech422 | inc0: and of my graduating class of 100, half never left the area | 13:58 |
| inc0 | yeah, that's strange to me tho | 13:58 |
| inc0 | you have enormous, beautiful and naturally diverse country | 13:58 |
| Mech422 | inc0: and we grew up in such a boring part of it... | 13:59 |
| inc0 | lol | 13:59 |
| Mech422 | you'd think people would want to see the rest | 13:59 |
| inc0 | I guess that partially explains this nationalism I've experienced | 13:59 |
| inc0 | it's easy to be nationalistic if you've never seen any other nation | 14:00 |
| Mech422 | inc0: nah - lately, its getting whipped to a bit of a fevor... | 14:00 |
| Mech422 | inc0: I'd be cool with isolationist - tired of 'policeman of the world' stuff - but its going to far lately | 14:00 |
| *** diogogmt has quit IRC | 14:01 | |
| inc0 | well, you don't need to be isolationist to not be a policeman of the world... | 14:01 |
| Mech422 | inc0: tell that to govt :-P | 14:01 |
| inc0 | just well...don't invade other countries | 14:01 |
| Mech422 | inc0: yeah - they we get yelled at for letting stuff happen re:serbs/croats | 14:01 |
| inc0 | evil things happends with or without you | 14:02 |
| Mech422 | inc0: tell that to govt :-P | 14:02 |
| chopmann | master & ceph == no deploy :-(, we switched to 2.0.1 and it's now deploying | 14:02 |
| Mech422 | anyway - my redneck butt doesn't really get to decided... | 14:02 |
| inc0 | chopmann, master is ansible 2 | 14:03 |
| inc0 | yeah, I know | 14:03 |
| chopmann | yeah, we knew that | 14:03 |
| openstackgerrit | Sergey Lukjanov proposed openstack/kolla: Proposal spec for universal containers https://review.openstack.org/330575 | 14:03 |
| inc0 | I don't blame you guys, don't get me wrong | 14:03 |
| chopmann | our config scripts (outside kolla) need 2.0.0> | 14:03 |
| inc0 | just an observation from recent culture shock | 14:03 |
| Mech422 | inc0: btw - I'm a yankee redneck - seperate breed from the texas rednecks and southern rednecks... | 14:03 |
| inc0 | chopmann, ansible 1 and 2 are not mutually compatible unfortunately | 14:03 |
| Mech422 | inc0: for 1 I don't play banjo, 2 I don't ride horses :-P | 14:04 |
| inc0 | lol | 14:04 |
| Mech422 | inc0: anyway, did you happen to look at augeas ? | 14:05 |
| inc0 | yeah | 14:05 |
| inc0 | and I have different idea:P | 14:05 |
| chopmann | inc0 that's why we wanted to deploy with master, we'll run our scripts with an venv for ansible 2.0.0 until master has the first rc | 14:05 |
| inc0 | so I'll run it with you | 14:06 |
| Mech422 | inc0: ahh...ok...I won't bother with lenses then.. | 14:06 |
| inc0 | so augeas looks cool | 14:06 |
| inc0 | but how about that: | 14:06 |
| inc0 | we decopule laying configs from starting containers | 14:06 |
| inc0 | so you can lay config without starting container and start contianer without laying configs | 14:06 |
| inc0 | that means, you lay configs across the cluster | 14:07 |
| Mech422 | inc0: oh - I thought thats how it was? lay down the configs, then start the containers ? | 14:07 |
| inc0 | it is | 14:07 |
| inc0 | but there is no breakpoint in between | 14:07 |
| Mech422 | ahh... yeah - I thought it was 2 steps | 14:07 |
| Mech422 | that makes a lot of sense - restarting a container doesn't really mean you want new configs.. | 14:07 |
| inc0 | 2 tasks, yeah | 14:07 |
| inc0 | but not 2 plays or anything | 14:08 |
| *** sacharya has joined #openstack-kolla | 14:08 | |
| Mech422 | you might be in the middle of updating configs or something when you need to restart | 14:08 |
| inc0 | well you don't update configs at all now really | 14:08 |
| inc0 | in a way, you do that, you run ansible and it will rollback to whatever it was before | 14:08 |
| inc0 | anyway, if we deploy with only configs or no-configs | 14:09 |
| Mech422 | inc0: 'no-configs' means 'use existing configs' right ? | 14:09 |
| inc0 | yeah | 14:09 |
| inc0 | ones in /var/lib/kolla | 14:09 |
| Mech422 | yeah - I like that idea... | 14:09 |
| inc0 | I just need to figure out how to make it work nicely with our playbook structure | 14:10 |
| Mech422 | so then, where does the non-ini files stuff and merging configs come in ? | 14:10 |
| inc0 | nowhere | 14:10 |
| inc0 | merge_ini keeps working | 14:10 |
| inc0 | non-ini, just ssh to a node and edit this in place | 14:10 |
| inc0 | we only have few non-ini services | 14:11 |
| Mech422 | hmm... | 14:11 |
| Mech422 | I already 'drive' kolla thru my own ansible playbooks - so that works for me... | 14:11 |
| inc0 | why? | 14:11 |
| inc0 | what was reason to create your own plays? | 14:11 |
| Mech422 | but if you use 'raw' kolla - it might be nice to have a central spot on deployment to keep your configs ? | 14:11 |
| Mech422 | inc0: I do bootstrapping, apply patches, etc etc | 14:12 |
| inc0 | well, yeah, it's deployment | 14:12 |
| inc0 | node | 14:12 |
| Mech422 | oh - so ssh to deployment node and edit them ? | 14:12 |
| inc0 | but if we enable this we can get functionality there quickly | 14:12 |
| inc0 | nah | 14:12 |
| inc0 | you need to ssh to nodes themselves | 14:12 |
| inc0 | as configs might differ node to node | 14:12 |
| Mech422 | inc0: I think thats gonna be hell for newbies... | 14:12 |
| Mech422 | inc0: at that point, your back to following the openstack install guide, except with containers | 14:13 |
| Mech422 | I think configuration and deployment should be 2 seperate steps... | 14:14 |
| Mech422 | its just how to organize configuration | 14:14 |
| *** sacharya_ has joined #openstack-kolla | 14:18 | |
| *** williamcaban has joined #openstack-kolla | 14:19 | |
| tloeb | Woop Woop chopman and me just deployed an openstack multinode with kolla 2.0.1. We had a few problems with kolla master branch and ceph, so we used 2.0.1 | 14:20 |
| *** Guest48217 has quit IRC | 14:20 | |
| *** sacharya has quit IRC | 14:21 | |
| *** zhiwei has quit IRC | 14:23 | |
| Mech422 | tloeb: Grats! | 14:23 |
| *** ravig has quit IRC | 14:31 | |
| *** sacharya_ has quit IRC | 14:32 | |
| *** sacharya has joined #openstack-kolla | 14:33 | |
| *** dwalsh has quit IRC | 14:33 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Remove the duplication of the inventory https://review.openstack.org/328891 | 14:33 |
| *** tloeb has quit IRC | 14:40 | |
| *** neilus has quit IRC | 14:41 | |
| *** tloeb has joined #openstack-kolla | 14:43 | |
| *** dwalsh has joined #openstack-kolla | 14:45 | |
| *** arizidon has joined #openstack-kolla | 14:46 | |
| *** b_bezak has quit IRC | 14:47 | |
| *** tyrola has joined #openstack-kolla | 14:49 | |
| *** chopmann has quit IRC | 14:53 | |
| *** chopmann has joined #openstack-kolla | 14:53 | |
| *** chopmann has quit IRC | 14:54 | |
| *** sdake has quit IRC | 14:54 | |
| *** mliima has quit IRC | 14:54 | |
| *** diogogmt has joined #openstack-kolla | 14:55 | |
| *** salv-orlando has quit IRC | 14:56 | |
| *** salv-orlando has joined #openstack-kolla | 14:57 | |
| openstackgerrit | Martin André proposed openstack/kolla: Properly fail precheck for internal VIP's network https://review.openstack.org/330628 | 14:58 |
| *** tloeb_ has joined #openstack-kolla | 14:59 | |
| *** sdake has joined #openstack-kolla | 14:59 | |
| *** salv-orlando has quit IRC | 15:00 | |
| *** salv-orlando has joined #openstack-kolla | 15:00 | |
| *** tloeb__ has joined #openstack-kolla | 15:00 | |
| *** tloeb has quit IRC | 15:00 | |
| *** david-lyle_ is now known as david-lyle | 15:00 | |
| *** tloeb_ has quit IRC | 15:00 | |
| *** ravig has joined #openstack-kolla | 15:04 | |
| *** tloeb__ has quit IRC | 15:04 | |
| *** wmiller has quit IRC | 15:04 | |
| *** wmiller has joined #openstack-kolla | 15:05 | |
| wirehead_ | So, can I get some Kolla core to look at https://review.openstack.org/#/c/327925/ ? It's blocking some other changes that are ready-to-merge. | 15:05 |
| patchbot | wirehead_: patch 327925 - kolla - Enable kolla k8s to override bind api bind address... | 15:05 |
| rhallisey | yo inc0 ^ | 15:06 |
| *** v1k0d3n has joined #openstack-kolla | 15:06 | |
| *** ppowell has quit IRC | 15:10 | |
| *** mliima has joined #openstack-kolla | 15:11 | |
| *** ravig has quit IRC | 15:13 | |
| *** coolsvap has joined #openstack-kolla | 15:16 | |
| *** daneyon_ has joined #openstack-kolla | 15:21 | |
| mandre | hey nihilifer, was there any specific reason why you abandoned https://review.openstack.org/#/c/306284/ beside "no time to work on it"? | 15:21 |
| patchbot | mandre: patch 306284 - kolla - Use MAC address for looking for DHCP leases for Va... | 15:21 |
| *** daneyon_ has quit IRC | 15:25 | |
| *** ppowell has joined #openstack-kolla | 15:28 | |
| *** sacharya_ has joined #openstack-kolla | 15:28 | |
| *** sacharya has quit IRC | 15:30 | |
| *** sacharya has joined #openstack-kolla | 15:30 | |
| *** jmccarthy1 has left #openstack-kolla | 15:31 | |
| *** fbarilla has joined #openstack-kolla | 15:32 | |
| nihilifer | mandre: this patch wasn't really fixing my issue in 100% | 15:32 |
| nihilifer | and i just planned to try to fix everything from scratch | 15:32 |
| nihilifer | but unfortunately, i didn't find time for that | 15:32 |
| nihilifer | it this commit helpful for you? | 15:33 |
| mandre | I happened to have empty hostnames in my lease file, and switching to mac addresses fixed the issue for me | 15:33 |
| *** sacharya_ has quit IRC | 15:33 | |
| mandre | so yeah, it's helpful :) | 15:33 |
| nihilifer | mandre: it worked on multinode as well? | 15:33 |
| *** openstackgerrit has quit IRC | 15:34 | |
| nihilifer | i recall that this patch fixed aio on centos for me | 15:34 |
| *** openstackgerrit has joined #openstack-kolla | 15:34 | |
| nihilifer | but multinode was still f*** up | 15:34 |
| mandre | I haven't tried multinode yet | 15:34 |
| mandre | will do | 15:34 |
| nihilifer | and lease file looked strange, despite the fact that all vm-s were created and libvirt saw them | 15:34 |
| *** mikelk has quit IRC | 15:35 | |
| nihilifer | finally, i gave up with multinode vagrant on centos and made the vm-s manually by virtmanager :v | 15:36 |
| mandre | eh... i'll try multinodes, we'll see if I have the same issue | 15:37 |
| *** rmart04 has quit IRC | 15:38 | |
| fbarilla | just to make sure I've a valid configuration to start with. Host: Ubuntu 16.04, docker 1.10, ansible 2.1 - Kolla from github:master - docker base image: Ubuntu 14.04. | 15:39 |
| *** sacharya has quit IRC | 15:44 | |
| *** sacharya has joined #openstack-kolla | 15:45 | |
| *** ravig has joined #openstack-kolla | 15:49 | |
| kklimonda | fbarilla: I'm using a similar setup, just ansible 2.0 instead, and it's working fine - I've seen some commits fixing 2.1 in master, so I think it may also be working now | 15:50 |
| Mech422 | I thought host was ubuntu 14.04 ? | 15:53 |
| Mech422 | kolla supports ubuntu 16.04 now ? | 15:54 |
| openstackgerrit | Md Nadeem proposed openstack/kolla: Added unit test for start container https://review.openstack.org/329847 | 15:54 |
| *** rahuls has joined #openstack-kolla | 15:54 | |
| *** belmoreira has joined #openstack-kolla | 15:56 | |
| kklimonda | Mech422: I haven't had any problems so far | 15:56 |
| *** mummer has joined #openstack-kolla | 15:56 | |
| *** z- has quit IRC | 15:57 | |
| *** z- has joined #openstack-kolla | 15:57 | |
| *** mark-casey has joined #openstack-kolla | 15:57 | |
| *** sacharya_ has joined #openstack-kolla | 15:57 | |
| kklimonda | Mech422: with 14.04 you needed a newer kernel and docker from upstream. I'm still using docker from upstream, and 4.4 kernel, so it's a pretty close setup. replacing upstart with systemd just made ubuntu look more like centos, so no issue with /run/netns mountpoint. | 16:00 |
| *** sacharya has quit IRC | 16:00 | |
| Mech422 | kklimonda: yeah - I run upstream kernel & docker | 16:01 |
| Mech422 | kklimonda: I'll have to setup a 16.04 in foreman to play with | 16:01 |
| Mech422 | kklimonda: I've been avoiding systemd :-P | 16:01 |
| kklimonda | it's a pretty common thing to do, but it seems it's here to stay, so you may as well dabble in it ;) | 16:03 |
| *** berendt has joined #openstack-kolla | 16:06 | |
| *** tyrola has quit IRC | 16:06 | |
| *** sacharya has joined #openstack-kolla | 16:07 | |
| *** ppowell has quit IRC | 16:08 | |
| *** sacharya_ has quit IRC | 16:10 | |
| *** mgoddard_ has quit IRC | 16:11 | |
| Mech422 | kklimonda: yeah - my sentiments as sell | 16:14 |
| Mech422 | s/sell/well/ | 16:15 |
| *** berendt has quit IRC | 16:16 | |
| *** cloudnau_ has joined #openstack-kolla | 16:18 | |
| *** mgoddard has joined #openstack-kolla | 16:18 | |
| *** ravig has quit IRC | 16:18 | |
| *** ravig has joined #openstack-kolla | 16:19 | |
| *** sdake has quit IRC | 16:20 | |
| *** berendt has joined #openstack-kolla | 16:23 | |
| *** sdake has joined #openstack-kolla | 16:28 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: GATE TESTING https://review.openstack.org/330192 | 16:29 |
| *** berendt has quit IRC | 16:31 | |
| *** huikang has joined #openstack-kolla | 16:31 | |
| *** matrohon has quit IRC | 16:33 | |
| *** david-lyle has quit IRC | 16:35 | |
| *** berendt has joined #openstack-kolla | 16:36 | |
| pbourke | Mech422: hey | 16:37 |
| pbourke | Mech422: i tried your patch, unfortuantely it doesn't work for my use case as I get almost no info out of udev | 16:38 |
| pbourke | Mech422: id still be happy to merge the patch though if it solves your issue | 16:39 |
| pbourke | need to find out what Jeffrey4l means by online drive | 16:41 |
| pbourke | does he mean mounted? | 16:41 |
| *** dmk0202 has quit IRC | 16:42 | |
| *** ravig has quit IRC | 16:45 | |
| sdake | inc0 ping | 16:46 |
| inc0 | I'm here | 16:46 |
| sdake | these folks want additive repos | 16:46 |
| sdake | not replacement | 16:46 |
| sdake | apparently a communication problem | 16:46 |
| inc0 | that's ok | 16:46 |
| rhallisey | what do you mean by additive vs replacement | 16:47 |
| Mech422 | pbourke: you still need udev with my patch? what part uses udev ? | 16:47 |
| rhallisey | add repos per service? | 16:47 |
| sdake | additive means add repos to a specific container and have it use those deps instead of the ones in the base image | 16:47 |
| Mech422 | pbourke: yeah - he doesn't like the idea of using drives where other partitions are in use... | 16:47 |
| pbourke | Mech422: line 74 | 16:47 |
| rhallisey | err | 16:47 |
| pbourke | ive had a terrible day trying to hack our ceph roles to not use udev | 16:48 |
| pbourke | :( | 16:48 |
| sdake | use case: magnum on mitaka, rest of liberty | 16:48 |
| sdake | (heat as well) | 16:48 |
| inc0 | sdake, no, what they want is github.com/openstack/kolla-heat-container | 16:48 |
| inc0 | separate repo | 16:48 |
| inc0 | for each Dockerfile/service | 16:48 |
| sdake | inc0 i am talking about my requirements | 16:49 |
| sdake | i havent read surgey's review yet | 16:49 |
| rhallisey | kolla-heat-api / kolla-heat-engine | 16:49 |
| inc0 | ahh, I thought Mirantis | 16:49 |
| Mech422 | pbourke: Hmm - yeah...We could prolly redo it to bypass udev... | 16:49 |
| sdake | but repo per container seems irrational to me | 16:49 |
| sdake | hard to miantain | 16:49 |
| sdake | i can just imagine releasing 20 packages at once | 16:49 |
| sdake | what a pain in the ass | 16:49 |
| pbourke | Mech422: i started but underestimated the amount of info we're pulling from udev | 16:49 |
| *** david-lyle has joined #openstack-kolla | 16:50 | |
| pbourke | i could be done just messy | 16:50 |
| Mech422 | pbourke: it shouldn't be that much - at least for the osd stuff all it needs is the data and journal partitions... | 16:50 |
| inc0 | sdake, so I'd say use tags | 16:50 |
| Mech422 | pbourke: does your use case need more then the osds ? | 16:50 |
| inc0 | you need separate base image for magnum in your case | 16:50 |
| rhallisey | sdake, we need to think about this though | 16:52 |
| Mech422 | pbourke: dam - that line is just me being lazy about getting the parent device name... | 16:52 |
| rhallisey | if this is what outside installers want, it's worth trying to address the features gained from all split out | 16:53 |
| rhallisey | PITA is a con | 16:53 |
| sdake | con? | 16:53 |
| rhallisey | but I think they know that | 16:53 |
| sdake | oh con | 16:53 |
| rhallisey | so I'm curious why they still want it | 16:53 |
| sdake | rhallisey i was having a different discussion with inc0 | 16:53 |
| sdake | on the multiple repos front, I haven't heard a *why* | 16:53 |
| rhallisey | ya I'm joining in | 16:54 |
| *** mbound has quit IRC | 16:54 | |
| rhallisey | I want to hear why also, but seems like releases? | 16:54 |
| inc0 | there is some explanation in spec but I don't really like it that much, if it's customizations, then I pointed them to work we're doing | 16:54 |
| rhallisey | customizations maybe? | 16:54 |
| sdake | rhallisey split repos vs differnet base images are two orthogonal things | 16:54 |
| inc0 | so ad separate magnum mitaka | 16:55 |
| rhallisey | are you sure they are arguing for different base images? | 16:55 |
| sdake | rhallisey no that is what i was talking about inc0 with though :) | 16:55 |
| inc0 | kolla-build --profile non-magnum-images | 16:55 |
| rhallisey | ok | 16:55 |
| sdake | rhallisey i wasn't tlaking about mirantis at all | 16:55 |
| rhallisey | ha | 16:55 |
| inc0 | kolla-build --profile magnum-images --template-override=<<change repos>> | 16:55 |
| inc0 | and add tag | 16:55 |
| inc0 | kolla-build --profile magnum-images --template-override=<<change repos>> --tag=magnum-mitaka | 16:56 |
| inc0 | then in globals.yml you specify magnum_image | 16:56 |
| *** arizidon has quit IRC | 16:56 | |
| *** berendt has quit IRC | 16:57 | |
| inc0 | https://github.com/openstack/kolla/blob/master/ansible/roles/magnum/defaults/main.yml#L16 | 16:58 |
| inc0 | so totally doable | 16:58 |
| inc0 | and by tailoring profiles you can do any combination of base/customization you want | 16:58 |
| Mech422 | pbourke: yeah - it looks like find_disks.py would basically need a re-write, and if we don't use udev it probably means a bunch of inc0-hates-scraping ... | 16:59 |
| pbourke | Mech422: :( | 16:59 |
| *** ppowell has joined #openstack-kolla | 16:59 | |
| sdake | 4 hour build times.. | 16:59 |
| * sdake groans | 16:59 | |
| inc0 | sdake, no... | 16:59 |
| inc0 | almost exactly the same build time as alternative | 16:59 |
| inc0 | just 2 calls | 16:59 |
| inc0 | hell, it can be faster | 16:59 |
| *** berendt has joined #openstack-kolla | 16:59 | |
| inc0 | if you run both as separate process | 16:59 |
| Mech422 | pbourke: hmm - we could create a dummy'd up version of find_disks for you though.. you could patch it in after kolla git checkout, before build ? | 17:00 |
| inc0 | no dependencies with each other | 17:00 |
| inc0 | so it will actually be quicker | 17:00 |
| *** belmoreira has quit IRC | 17:00 | |
| pbourke | Mech422: id stilll merge a screen scraping alternative | 17:00 |
| *** ravig has joined #openstack-kolla | 17:00 | |
| pbourke | *still | 17:00 |
| pbourke | Mech422: ugly is better than broken | 17:01 |
| sdake | screen scrape of what | 17:01 |
| sdake | inc0 i dont want a globals.yml full of custom image names | 17:01 |
| sdake | inc0 however I can build them all with the same tag from different profiles | 17:01 |
| inc0 | why? it's one conf... | 17:01 |
| inc0 | no, you can't | 17:01 |
| inc0 | you need 2 different base images | 17:01 |
| *** ravig has quit IRC | 17:01 | |
| inc0 | different. | 17:01 |
| Mech422 | sdake: output parsing to elimentate reliance on udev | 17:01 |
| Mech422 | pbourke: Hmm - what distro you using that udev is so broken on ? | 17:02 |
| Mech422 | pbourke: is it something common ? | 17:02 |
| sdake | inc0 consider a scenario whee every container has different deps, then each needs a diferent profile | 17:02 |
| pbourke | Mech422: oracle vm (based on rhel 6) | 17:02 |
| pbourke | Mech422: ive seen bad udev behaviour on newer distros such centos 7 also though | 17:03 |
| pbourke | it doesn't work reliably | 17:03 |
| Mech422 | pbourke: Hmm... I had suspicions udev might not be 100% reliable.. | 17:03 |
| *** arizidon has joined #openstack-kolla | 17:03 | |
| Mech422 | pbourke: is there anything in /proc we can use on those boxes ? | 17:04 |
| Mech422 | pbourke: udev only handles /sys right, /proc is direct from kernel ? | 17:04 |
| inc0 | pbourke, Mech422 so I'm not hateful towards screenscrapping, all I'm saying is we started with this and we decided we'd rather rewrite it to python than deal with all the edge cases | 17:05 |
| mark-casey | pbourke: Mech422: Is your issue at a high level based on getting up to date information back? I mean if you were to screen scrape the version that's working (sgdisk I think I heard) could you progress? | 17:05 |
| Mech422 | inc0: yeah - I know - I'm just blaming you, instead of saying "I'm lazy" :-P | 17:05 |
| *** mbound has joined #openstack-kolla | 17:05 | |
| Mech422 | inc0: soo hush, and let me weasel :-P | 17:05 |
| inc0 | don't forget, you want it replaced, you maintain it | 17:06 |
| Mech422 | mark-casey: umm - in general, yes - my problem wasn't as severe as pbourke's | 17:06 |
| Mech422 | inc0: LOL - thanks :-P | 17:06 |
| inc0 | how about we go to internets and try to find better python library? | 17:07 |
| kklimonda | sdake: wouldn't it actually make sense to sync tags with project releases, if containers were to be split? Although, I probably just don't like that current kolla tags version are not tied to software that they are deploying. | 17:07 |
| pbourke | have to run, catch you tomorrow Mech422 | 17:07 |
| inc0 | I refule to believe that there is no disk managment python lib | 17:07 |
| Mech422 | pbourke: ok... | 17:07 |
| inc0 | there is python lib to everything | 17:07 |
| sdake | later pbourke | 17:07 |
| Mech422 | inc0: its not that - its pbourke's udev is broken somehow | 17:07 |
| Mech422 | inc0: he basically can't get ANY info out of it | 17:07 |
| Mech422 | inc0: so anything based on udev is no-go for him | 17:08 |
| *** salv-orl_ has joined #openstack-kolla | 17:08 | |
| *** daneyon_ has joined #openstack-kolla | 17:09 | |
| mark-casey | Mech422: so this is just a shot in the dark, but you might see if there is some other "$thing" that looks at or reports the data you need. for example there are python libs for managing LVM and the one for pvcreate (I'm assuming) reports information about partitions. Maybe one for that, or mdadm, or iscsi, or etc. would have a python lib report back correctly | 17:09 |
| inc0 | Mech422, but there might be python lib that doens't use udev | 17:09 |
| Mech422 | inc0: might be... | 17:10 |
| inc0 | let's try to find it | 17:10 |
| inc0 | all I'm saying | 17:10 |
| Mech422 | inc0: I was busy weaseling...how did this fall on my plate :-P | 17:10 |
| inc0 | before writing something of our own | 17:10 |
| Mech422 | inc0: I guess europeans don't have a concept of 'weaseling' eh ? :-) | 17:11 |
| *** salv-orlando has quit IRC | 17:12 | |
| inc0 | oh trust me, we do | 17:12 |
| inc0 | hell, you're British heritage | 17:13 |
| inc0 | you should know all about it | 17:13 |
| inc0 | actually, I'd say it's in your blood:P | 17:13 |
| Mech422 | inc0: Ouch! | 17:13 |
| * Mech422 goes to throw some tea in a harbour... | 17:13 | |
| *** daneyon_ has quit IRC | 17:14 | |
| Mech422 | inc0: so, umm..I love pbourke like a brother - but how common is rhel 6 ? is that the current version? | 17:14 |
| Mech422 | inc0: i.e. would we be doing all this for something thats EOL ? | 17:14 |
| inc0 | Mech422, you're one not wanting udev | 17:16 |
| inc0 | if it doesn't work for Paul, let's look for alternatives | 17:16 |
| inc0 | I'm not religiously bound by having udev | 17:16 |
| Mech422 | inc0: Huh? | 17:16 |
| Mech422 | inc0: I'm happy with the patch I submitted a week ago... | 17:17 |
| inc0 | I just say that screen scrapping is far from robust/easy | 17:17 |
| Mech422 | inc0: its pbourke thats having problem because his udev is broken | 17:17 |
| inc0 | pbourke, is there alternative you think of? /proc? is there python lib to get things out of /proc? | 17:18 |
| Mech422 | inc0: Right - cuz his udev is broken | 17:18 |
| Mech422 | inc0: he logged - I asked him the same thing about 5 minutes ago | 17:18 |
| inc0 | well, let's figure it out with himn | 17:19 |
| Mech422 | thats what I was trying to do... | 17:19 |
| Mech422 | if we can just read /proc to get roughly the same info as we get from udev, that might work for him.. | 17:20 |
| Mech422 | we don't really need anything 'fancey' - just the list of drives, partitions and names | 17:20 |
| Mech422 | no geometery or anything | 17:20 |
| *** athomas has quit IRC | 17:22 | |
| *** berendt has quit IRC | 17:25 | |
| openstackgerrit | Merged openstack/kolla: Fix ownership of nova-novncproxy-source directory https://review.openstack.org/330275 | 17:26 |
| *** berendt has joined #openstack-kolla | 17:27 | |
| *** inc0 has quit IRC | 17:28 | |
| *** inc0 has joined #openstack-kolla | 17:29 | |
| *** ravig has joined #openstack-kolla | 17:29 | |
| *** dwalsh has quit IRC | 17:29 | |
| inc0 | rhallisey, so did you end up playing with ansible-bootstrappign-k8s idea | 17:30 |
| inc0 | ? | 17:30 |
| rhallisey | a little | 17:31 |
| rhallisey | may have mariadb working soon | 17:31 |
| rhallisey | had to skip writing the module | 17:31 |
| inc0 | cool | 17:31 |
| rhallisey | there is no kubernetes-py | 17:31 |
| inc0 | duh | 17:31 |
| rhallisey | well there is but not maintained by kube | 17:31 |
| inc0 | well there is REST api | 17:31 |
| rhallisey | I mean this all comes down to how much we want the container to do | 17:32 |
| rhallisey | will the containers need to check if they need to be bootstrapped or should an outside tool | 17:32 |
| rhallisey | I think kfox is right when he cautions about making the containers too smart | 17:33 |
| inc0 | well moving bootstrapping outside of containers, one Sergey described, might not be bad idea | 17:37 |
| inc0 | however we need to analyze it more | 17:37 |
| inc0 | actually, might not be super easy too | 17:37 |
| inc0 | if we consider networkign | 17:38 |
| inc0 | so for example ansible might not have access to openstack mgmt network | 17:38 |
| inc0 | therefore to mariadb | 17:38 |
| inc0 | so from deployment node we might not have access to bootstrap | 17:38 |
| inc0 | not to mention running mysql-admin stuff | 17:38 |
| rhallisey | I guess there's 2 ways to interpret that. 1) the way you mentioned 2) don't do everything in the container | 17:40 |
| *** ravig has quit IRC | 17:41 | |
| rhallisey | I was thinking more like #2 | 17:41 |
| *** zhiwei has joined #openstack-kolla | 17:41 | |
| *** berendt has quit IRC | 17:41 | |
| rhallisey | leave the hard workflow logic to the workflow engine | 17:41 |
| rhallisey | hmm | 17:42 |
| *** Mech422__ has joined #openstack-kolla | 17:42 | |
| rhallisey | inc0, so similar to the way ansible checks if the mariadb volume exists | 17:42 |
| rhallisey | we could ask ceph if it exists also | 17:42 |
| rhallisey | could also see if someone create a configmap 'mariabd_bootstrap' | 17:43 |
| rhallisey | and only let one person create that | 17:43 |
| *** berendt has joined #openstack-kolla | 17:43 | |
| *** zhiwei has quit IRC | 17:45 | |
| *** Mech422 has quit IRC | 17:45 | |
| wirehead_ | The more logic we move into sidecar containers or containers in general, the less we’re going to be dependent on the vagaricies of the user’s local system. | 17:46 |
| inc0 | wirehead_, that's the idea behind bootstrap containers | 17:47 |
| rhallisey | but the less control ops will have | 17:47 |
| inc0 | rhallisey, not really | 17:47 |
| rhallisey | debatable | 17:47 |
| inc0 | coreos and such are not meant to be logged it at all | 17:47 |
| inc0 | every tool should be running in container | 17:48 |
| inc0 | well logged in but not used as a normal OIS | 17:48 |
| inc0 | OS | 17:48 |
| wirehead_ | I think it requires a bit of change on the side of the tools we’re running, tho. | 17:49 |
| inc0 | what do you mean? | 17:49 |
| rhallisey | I think at the very least we need way more reliable checking | 17:50 |
| wirehead_ | Well, on the kube side, you can do a lot to a running cluster by running `kubectl exec mariadb-xxxxx -it — bash` or `kubectl exec mariadb-xxxxx mysql ….` | 17:50 |
| rhallisey | that's what I think is killing us here | 17:50 |
| wirehead_ | That doesn’t *quite* map as well outside of the kube model. | 17:50 |
| wirehead_ | Kolla-k8s is a bit weird because all of the tooling we’re using is designed for a very very different way of working and we’re shoehorning it in. | 17:51 |
| wirehead_ | By default, you just log kube-native apps to stdout. | 17:51 |
| wirehead_ | Running multiple processes? Sidecar containers. | 17:52 |
| rhallisey | so right now were using a job | 17:53 |
| rhallisey | it may not make sense | 17:53 |
| rhallisey | I like the method of throwing everything up there and health checking | 17:54 |
| wirehead_ | Yeah. | 17:54 |
| rhallisey | but is that really flexible? | 17:54 |
| wirehead_ | It gives me a bit of the heebie jeebies but the word on the street calls it the kubernetes way. | 17:55 |
| rhallisey | is openstack too complex of an application for this mehtod to work | 17:55 |
| wirehead_ | It’s also refactoring. | 17:55 |
| rhallisey | let's take nova | 17:55 |
| *** berendt has quit IRC | 17:56 | |
| rhallisey | start nova-api, scheduler, conductor | 17:56 |
| rhallisey | nova-api will bootstrap while the other containers start | 17:56 |
| *** berendt has joined #openstack-kolla | 17:56 | |
| *** vhosakot has joined #openstack-kolla | 17:57 | |
| rhallisey | instead of exit 0, we just use it as a blocking point | 17:57 |
| rhallisey | before nova-api starts | 17:57 |
| rhallisey | we also need to look at services as individual applications | 17:58 |
| rhallisey | they should have their own pods | 17:58 |
| Mech422__ | inc0: ok - not finding any libs so far - here's another example of parsing raw gpt table in python - just to give you an idea of what it takes: https://github.com/n0fate/raw/blob/master/gpt_parser.py | 17:58 |
| rhallisey | with a few exceptions | 17:58 |
| rhallisey | like nova-compute & libvirt | 17:58 |
| rhallisey | wirehead_, I'll add a few patches up for this | 18:00 |
| rhallisey | revisit this idea a little.. | 18:01 |
| inc0 | Mech422__, uhh.... | 18:01 |
| wirehead_ | And I was just looking through the list and feeling some victory that we were down to 4 Kolla patches. :/ | 18:01 |
| Mech422__ | inc0: thats why I didn't want to do it ourselves to begin with :-) | 18:01 |
| *** vhosakot has quit IRC | 18:01 | |
| rhallisey | wirehead_, :) | 18:01 |
| wirehead_ | I do tend to think that OpenStack is going to be on the high end of complexity that people have tried to make Kube-native. | 18:01 |
| Mech422__ | inc0: I'll keep looking for an existing lib - but if we really want to get rid of udev for this, I think we're gonna be looking at some scraping | 18:02 |
| *** vhosakot has joined #openstack-kolla | 18:02 | |
| Mech422__ | inc0: again, I am NOT advocating dropping udev - I don't know how much of a corner case pbourke stuff is... | 18:03 |
| Mech422__ | inc0: but if we want to support Oracle/RHEL 6 - it looks like udev has to go? | 18:03 |
| inc0 | :S | 18:04 |
| Mech422__ | since I'm not a RH guy - I have no idea how old RHEL 6 is anyway :-P | 18:05 |
| wirehead_ | Like, simultaneously, we’ve got the need to not make things harder on ourselves with kubernetes, while at the same time not diverging too much from kolla. | 18:06 |
| wirehead_ | RHEL 6 is like 3 Y! CEOs ago. | 18:06 |
| *** b_bezak has joined #openstack-kolla | 18:06 | |
| *** coolsvap has quit IRC | 18:06 | |
| Mech422__ | wirehead_: so, like 6 months ? :-P | 18:06 |
| *** dwalsh has joined #openstack-kolla | 18:06 | |
| Mech422__ | wirehead_: is it something worth supporting ? or is it EOL ? | 18:07 |
| *** coolsvap has joined #openstack-kolla | 18:07 | |
| Mech422__ | for find_disks.py all I need is a list of drives and partitions (can read that from /proc/partitions I think) | 18:08 |
| wirehead_ | RHEL 6 has just reached the end of “Production 1" | 18:08 |
| Mech422__ | and the partition label (I can scrape that from sgdisk) | 18:08 |
| Mech422__ | wirehead_: what does that mean ? | 18:08 |
| Mech422__ | how many productions do they have ? | 18:09 |
| wirehead_ | Apparently you can’t get full support, but they are still shipping security and other important fixes. | 18:09 |
| Mech422__ | so, anyway - redoing find_disks.py isn't a big deal - but i've no idea what other stuff pburke might have problems with | 18:10 |
| wirehead_ | So, RHEL 4 came out in 2010. | 18:10 |
| wirehead_ | Parallels Fedora 12. | 18:10 |
| wirehead_ | Err,, RHEL 6 came out in 2010. | 18:11 |
| Mech422__ | wirehead_: oh wow.. | 18:11 |
| *** coolsvap has quit IRC | 18:12 | |
| wirehead_ | But they’ve backported tons of stuff. | 18:12 |
| *** b_bezak has quit IRC | 18:12 | |
| kklimonda | yeah, it's a bit like asking if we should support precise-1 (I can't even remember what is the name anymore..) | 18:12 |
| wirehead_ | It’s like Ubuntu LTS but even longer. :P | 18:13 |
| Mech422__ | kklimonda: thats what I was trying to find out - I sorta think I'm better off just coming up with a 1 off patch for pbourke instead of redoing find_disk.py | 18:13 |
| wirehead_ | Like, the sort of long you’d get if you put Pinochio to the task of working at an adtech startup | 18:13 |
| Mech422__ | wirehead_: Hehe | 18:14 |
| kklimonda | it looks like docker isn't supported on rhel6 anymore? | 18:14 |
| Mech422__ | Anyway, I'm open to whatever...I don't know enough bout the support matrix to know who/how many this will effect | 18:15 |
| *** huikang has quit IRC | 18:15 | |
| sdake | is there a kolla build way to rebuild all children containers | 18:15 |
| sdake | for example, lets say i change nova-base | 18:15 |
| sdake | and want it to rebuild all nova containers as a result | 18:16 |
| *** huikang has joined #openstack-kolla | 18:16 | |
| kklimonda | I don't think so, I just do kolla-build "nova.*" and hope for the best | 18:16 |
| kklimonda | this works, but won't account for containers that don't share name with base | 18:16 |
| wirehead_ | So I’d somehow assumed in my head that there’s actually an official support matrix out of OpenStack for distros, at least marginally. | 18:18 |
| wirehead_ | But maybe that’s just a bit of a gentlebeing’s agreement in places. | 18:18 |
| kklimonda | I think it can be inferred from gates, but I haven't seen a document with distribution support matrix | 18:19 |
| *** huikang has quit IRC | 18:20 | |
| *** huikang has joined #openstack-kolla | 18:21 | |
| kklimonda | I know Canonical has a nice diagram with what OpenStack releases are supported, for the given distribution - but that is made by them I think | 18:21 |
| Mech422__ | err...well, if docker doesn't support RH EL 6... kinda leaves kolla in the cold doesn't it ? we need docker... | 18:21 |
| kklimonda | probably RH supports it for RHEL6? | 18:21 |
| Mech422__ | oh | 18:21 |
| Mech422__ | ok - so back to 'whatever you guys wanna do' | 18:21 |
| Mech422__ | :-P | 18:22 |
| kklimonda | wirehead_: https://wiki.ubuntu.com/ServerTeam/CloudArchive | 18:22 |
| kklimonda | but that's only for their binary packages - you can still build openstack from source... | 18:22 |
| *** berendt has quit IRC | 18:23 | |
| *** ravig has joined #openstack-kolla | 18:24 | |
| Mech422__ | inc0: the other option is to just move to the 'osd_...' vars in inventory file like we talked about | 18:25 |
| Mech422__ | inc0: that should work for pbourke and the fall back to my patch should handle everyone else ? | 18:26 |
| *** berendt has joined #openstack-kolla | 18:28 | |
| *** huikang has quit IRC | 18:30 | |
| *** ravig has quit IRC | 18:31 | |
| *** Mech422__ is now known as Mech422 | 18:32 | |
| *** sacharya has quit IRC | 18:32 | |
| *** sacharya_ has joined #openstack-kolla | 18:32 | |
| *** arizidon has quit IRC | 18:33 | |
| *** sacharya has joined #openstack-kolla | 18:33 | |
| *** sacharya_ has quit IRC | 18:37 | |
| openstackgerrit | Merged openstack/kolla: Properly fail precheck for internal VIP's network https://review.openstack.org/330628 | 18:40 |
| *** cloudnau_ has quit IRC | 18:40 | |
| inc0 | Mech422, we need to discuss more | 18:41 |
| inc0 | I'm ok with having somehow deterministic way to find disks as opposed to discovery | 18:41 |
| *** ravig has joined #openstack-kolla | 18:41 | |
| Mech422 | inc0: yeah - we hadn't resolved the bootstrap issue IIRC | 18:41 |
| inc0 | but I think it's solvable | 18:42 |
| Mech422 | inc0: well - if we want to be able to use file backing - discovery isn't gonna work... | 18:42 |
| *** mgoddard has quit IRC | 18:42 | |
| Mech422 | inc0: yeah - there's ways to solve it... could be as simple as a sentinel file somewhere | 18:43 |
| Mech422 | inc0: or maybe ask ceph | 18:43 |
| *** stvnoyes1 has joined #openstack-kolla | 18:43 | |
| Mech422 | inc0: I kinda like the idea of asking ceph, even if it means scraping - it would prevent 'out of sync' issues | 18:43 |
| *** huikang has joined #openstack-kolla | 18:44 | |
| *** arizidon has joined #openstack-kolla | 18:44 | |
| *** dmk0202 has joined #openstack-kolla | 18:44 | |
| inc0 | ceph has api | 18:44 |
| Mech422 | inc0: true | 18:45 |
| *** ravig has quit IRC | 18:45 | |
| Mech422 | inc0: only thing is, on replace drive - ceph will tell you 'ya I bootstrapped that already'...but you need to bootstrap the new drive | 18:45 |
| inc0 | basically we can look carefully what ceph-ansible does then | 18:45 |
| *** dmk0202 has quit IRC | 18:45 | |
| inc0 | this wheel is already invented | 18:45 |
| Mech422 | inc0: yeah | 18:46 |
| *** stvnoyes has quit IRC | 18:46 | |
| *** stvnoyes2 has joined #openstack-kolla | 18:46 | |
| inc0 | they use deterministic disk marking | 18:46 |
| inc0 | actually it would be interesting to merge their logic to kolla | 18:46 |
| inc0 | they'll be better ceph deployers than we can ever be | 18:46 |
| Mech422 | inc0: so is this where I should put cycles - working up support for the inventory var stuff ? | 18:46 |
| inc0 | can you add meeting agenda item? | 18:47 |
| Mech422 | inc0: lol - I'm noob...I don't think I can do anything :-P | 18:47 |
| inc0 | so we can get everyone involved | 18:47 |
| *** stvnoyes1 has quit IRC | 18:48 | |
| inc0 | you can do this;) | 18:48 |
| inc0 | just edit the wiki, anyone can do this | 18:48 |
| inc0 | anyway | 18:48 |
| inc0 | let's do that | 18:48 |
| Mech422 | I can? hmm - oh...I'll go look... | 18:48 |
| inc0 | let's look at ceph-ansible | 18:48 |
| Mech422 | inc0: ok | 18:48 |
| inc0 | and check out their method of dealing with that | 18:48 |
| inc0 | and get back to this discussion when we're wiser man | 18:48 |
| *** stvnoyes has joined #openstack-kolla | 18:49 | |
| Mech422 | inc0: I would love to use as much 'native ceph' tooling as possible... | 18:50 |
| inc0 | same here | 18:50 |
| Mech422 | inc0: cuts down on 'thats a non-standard install, go ask your vendor' if you ask questions | 18:50 |
| inc0 | issue is, they didn't deploy dockers before | 18:50 |
| inc0 | i think they might now tho | 18:50 |
| Mech422 | inc0: I'll look | 18:50 |
| inc0 | if they do, we can try to integrate that to kolla | 18:50 |
| *** stvnoyes2 has quit IRC | 18:51 | |
| inc0 | anyway, we should get another look at ceph ansible | 18:51 |
| inc0 | afk, meetings | 18:51 |
| *** berendt has quit IRC | 18:51 | |
| Mech422 | can anyone log into https://wiki.openstack.org/wiki/Meetings/Kolla with openid ? | 18:53 |
| Mech422 | I'm getting a blank page after sign-in | 18:54 |
| *** berendt has joined #openstack-kolla | 18:54 | |
| *** salv-orl_ has quit IRC | 18:54 | |
| *** salv-orlando has joined #openstack-kolla | 18:55 | |
| *** arizidon has quit IRC | 18:56 | |
| *** huikang has quit IRC | 18:58 | |
| *** ravig has joined #openstack-kolla | 18:58 | |
| *** ravig has quit IRC | 19:01 | |
| *** dwalsh has quit IRC | 19:03 | |
| *** dwalsh has joined #openstack-kolla | 19:04 | |
| *** huikang has joined #openstack-kolla | 19:04 | |
| *** arizidon has joined #openstack-kolla | 19:05 | |
| wirehead_ | Oh, rhallisey, did you want to chat about bootstrapping and stuff? | 19:10 |
| rhallisey | sure | 19:11 |
| rhallisey | let's say we drop the idea of jobs for a minute | 19:11 |
| wirehead_ | k | 19:12 |
| rhallisey | we would then need a) bootstrap side car container b) put the logic into extend_start | 19:12 |
| rhallisey | let's think about this service by service | 19:12 |
| rhallisey | starting with mariadb | 19:12 |
| wirehead_ | rhallisey: also, jobs are a corner case in some ways - you can fire off a job from inside a sidecar | 19:13 |
| *** sacharya has quit IRC | 19:13 | |
| wirehead_ | Actually. um. | 19:13 |
| *** sacharya has joined #openstack-kolla | 19:13 | |
| rhallisey | you can huh | 19:13 |
| wirehead_ | Yep. | 19:13 |
| rhallisey | we could keep the job in that case | 19:14 |
| rhallisey | is that kubernetes like though? | 19:14 |
| rhallisey | or is that hacky? | 19:14 |
| wirehead_ | However, if you are storing your mariadb /var/lib/mysql directory as a iSCSI device in a NAS or a via RBD and Ceph, you can only have it mounted on one container. | 19:16 |
| *** fbarilla has quit IRC | 19:16 | |
| *** ppowell has quit IRC | 19:16 | |
| *** ppowell has joined #openstack-kolla | 19:17 | |
| *** ravig has joined #openstack-kolla | 19:17 | |
| rhallisey | so then so side car for mariadb | 19:17 |
| wirehead_ | It’s not especially hacky to call into Kube via a container. | 19:17 |
| wirehead_ | Remember, Kube is explicitly a piece of a larger puzzle, not an OpenStack take-over-the-world effort. | 19:18 |
| rhallisey | in ansible a way to determine if we need to bootstrap is if the mariadb volume exists | 19:19 |
| wirehead_ | Yeah. Arguing a lot of sides of the argument here, but mariadb is very much a special case. | 19:20 |
| rhallisey | it is | 19:20 |
| rhallisey | we need a way to know if /var/lib/mysql exists | 19:21 |
| rhallisey | well | 19:21 |
| wirehead_ | Most of the rest of the services take care to store their data inside of MariaDB instead of doing it on their own. | 19:21 |
| rhallisey | if there is any info in it | 19:21 |
| *** ravig has quit IRC | 19:21 | |
| wirehead_ | I mean, there’s a patch to be made in the future to paramaterize the volume mount there. | 19:22 |
| wirehead_ | So, the way that bitnami’s chart (using the Helm tool for Kubernetes) bootstraps MariaDB is that they check every time at startup if everything’s set up. | 19:22 |
| *** ravig has joined #openstack-kolla | 19:22 | |
| wirehead_ | https://github.com/bitnami/charts/tree/master/mariadb | 19:23 |
| *** berendt has quit IRC | 19:23 | |
| rhallisey | ya even if we had ansible the same bootstrapping method wouldn't work | 19:27 |
| *** williamcaban has quit IRC | 19:27 | |
| *** SiRiuS__ has joined #openstack-kolla | 19:28 | |
| *** huikang has quit IRC | 19:28 | |
| rhallisey | maybe for now, AIO, we can check if /var/lib/mysql has any contents | 19:29 |
| rhallisey | we have to use shared storage for mutlinode, we can check on it there | 19:29 |
| *** ravig has quit IRC | 19:30 | |
| *** Jeffrey4l has quit IRC | 19:30 | |
| mark-casey | rhallisey: wirehead_: did the job/annotation plan hit roadblocks? | 19:31 |
| *** Jeffrey4l has joined #openstack-kolla | 19:31 | |
| rhallisey | mark-casey, well, it's not kubernetes like to do workflow | 19:31 |
| rhallisey | it | 19:31 |
| rhallisey | it | 19:32 |
| rhallisey | it's kubernetes like to throw everything up | 19:32 |
| rhallisey | and health check it | 19:32 |
| mark-casey | rhallisey: I thought I'd seen in their docs some opinions on it, that at least they recognize it as not always avoidable | 19:33 |
| wirehead_ | Yeah, I think for systems that use on-disk storage, doing the bootstrap at startup is kinda the best way to go. | 19:33 |
| rhallisey | there have been multiple attempts to bring dependencies to kubernetes, everytime is met with opposition | 19:33 |
| rhallisey | wirehead_, well for AIO it will work | 19:34 |
| rhallisey | multinode we will have to query ceph | 19:34 |
| wirehead_ | Well, you have to create the volume before mariadb can claim it. | 19:35 |
| *** huikang has joined #openstack-kolla | 19:35 | |
| rhallisey | won't it get created in the pod definition | 19:36 |
| *** ravig has joined #openstack-kolla | 19:36 | |
| wirehead_ | Not necessarily. The right way for 1.1-1.2 is a PersistantVolumeClaim. The 1.3 way is going to be to create a volumeClaimTemplate | 19:38 |
| wirehead_ | rhallisey: have you seen https://github.com/kubernetes/contrib/tree/master/pets/mysql/galera yet? | 19:38 |
| *** williamcaban has joined #openstack-kolla | 19:39 | |
| wirehead_ | Either way, I still think it’s OK to see if the mounted directory is empty on startup and, if so, do the bootstrap. | 19:39 |
| *** dmk0202 has joined #openstack-kolla | 19:40 | |
| wirehead_ | Alternatively, you can create the volume, run a job for boostrap, and then create the RC for mariadb which will fail to mount the volume until the job is complete. | 19:40 |
| rhallisey | I haven't | 19:43 |
| Mech422 | wirehead_: my boss sent this to me a while ago: http://vitess.io/ | 19:43 |
| Mech422 | haven't had a chance to really look at it, but it appears to be about running mysql in cloud/containers ? | 19:43 |
| *** mbound has quit IRC | 19:43 | |
| *** dmk0202 has quit IRC | 19:43 | |
| mark-casey | re: vitess https://youtu.be/E_xsKiuuXkY?t=17m14s | 19:44 |
| mark-casey | they had one of their guys at a coreos meetup | 19:44 |
| Mech422 | mark-casey: oh? did it look any good ? | 19:44 |
| *** ppowell has quit IRC | 19:45 | |
| *** ppowell has joined #openstack-kolla | 19:45 | |
| mark-casey | yes. I don't remember what all he was saying (to clarify I've just seen the video - wasn't there), but essentially it was "here are some things that we ran into with database in [I think it was for borg originally]" | 19:46 |
| *** dwalsh has quit IRC | 19:46 | |
| mark-casey | and the solutions in vitess sounded like a good fit. I need to re watch it. I don't remember anything about bootstrap though | 19:46 |
| *** dwalsh has joined #openstack-kolla | 19:47 | |
| *** williamcaban has quit IRC | 19:47 | |
| *** dwalsh has quit IRC | 19:47 | |
| *** ppowell has quit IRC | 19:47 | |
| *** dwalsh has joined #openstack-kolla | 19:48 | |
| Mech422 | mark-casey: gonna go watch video..brb | 19:48 |
| *** ppowell has joined #openstack-kolla | 19:48 | |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Be smarter about what to do when making a docker client https://review.openstack.org/330171 | 19:50 |
| mark-casey | rhallisey: wirehead_: so there have been several calls for external mysql in kolla. could that help in the k8s case if you can just assume it's up and not managed "in-band"? | 19:51 |
| rhallisey | wirehead_, ok so, Job will always bootstrap. Running the service will check if it should bootstrap or not | 19:51 |
| *** fragatin_ has quit IRC | 19:51 | |
| rhallisey | mark-casey, no we need to include a solution | 19:52 |
| *** daneyon_ has joined #openstack-kolla | 19:52 | |
| wirehead_ | mark-casey: not sure if that really helps. Ideally, Kolla-kubernetes should configure the right mysql DB for the Kubernetes world, even if that differs slightly from Kolla-ansible. | 19:52 |
| *** mliima has quit IRC | 19:52 | |
| rhallisey | wirehead_, I think that provides flexibility and allow for some ops to handle things in steps | 19:52 |
| wirehead_ | Either way, with Kubernetes service discovery, whatever is accessible by the right kube service will be used. Regardless of if we set it up or the user has provided their own way. | 19:53 |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Added some debugging operations to the MariaDB docs, wrapped lines. https://review.openstack.org/330762 | 19:54 |
| Mech422 | mark-casey: oh - vitess is youtube code ? | 19:55 |
| wirehead_ | I do think it’ll be different with e.g. Keystone, because there we’re setting up the database and running a provisioning script, all of which quite naturally fits inside of a job. | 19:55 |
| mark-casey | yea. started there and I think they're generalizing it and retargeting it to kube | 19:55 |
| Mech422 | mark-casey: yeah - I didn't know youtube was a mysql shop | 19:56 |
| *** daneyon_ has quit IRC | 19:56 | |
| *** huikang has quit IRC | 19:56 | |
| wirehead_ | Mostly for the sake of kicking off DB-schema-migration upgrades as well as provisioning, it kinda makes sense to have a sidecar container or boot script that triggers a job instead of driving that with the user. | 19:57 |
| *** zhiwei has joined #openstack-kolla | 19:59 | |
| *** salv-orlando has quit IRC | 20:00 | |
| *** salv-orlando has joined #openstack-kolla | 20:01 | |
| Mech422 | mark-casey: heh - forwarded that video to my boss - brown_points++ :-) | 20:01 |
| Mech422 | err..brownie_points++ even | 20:02 |
| Mech422 | mark-casey: thanks :-) | 20:02 |
| mark-casey | lol. sure thing | 20:02 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Remove code redundancy within service.py https://review.openstack.org/330765 | 20:02 |
| Mech422 | wirehead_: have you seen this stuff ? it seems like it would be really cool to have at least as an option ? | 20:02 |
| wirehead_ | Yeah, I’ve seen a few different options, mech422 | 20:03 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Consolidating path resolution and searching for program files https://review.openstack.org/329676 | 20:03 |
| *** zhiwei has quit IRC | 20:03 | |
| wirehead_ | There’s also Citrus’s patches against PostgreSQL, as well as a bunch of other options there. | 20:04 |
| wirehead_ | As well as the long-standing debate about a NoSQL database like Cassandra being used to drive OpenStack. | 20:04 |
| wirehead_ | I hadn’t actually had anyone point me to vitess, I think I ignored it last time. | 20:05 |
| Mech422 | wirehead_: definately worth a couple of minutes to watch that video mark-casey linked | 20:05 |
| wirehead_ | Yeah. | 20:05 |
| Mech422 | wirehead_: designed from the ground up for borg/k8 and in production use at youtube | 20:05 |
| Mech422 | wirehead_: thats more scale then I need :-D | 20:06 |
| wirehead_ | I’m deeply suspicious and have read too much aphyr, so I’d want to deeply dig into the stability guarantees. | 20:06 |
| mark-casey | wirehead_: your Keystone example sounds like the current topic. I guess a better question for me would be which attributes of the maria bootstrap aren't kosher for a job? | 20:06 |
| wirehead_ | mark-casey: Well, for one, most volume types are not able to work in shared-writable. | 20:06 |
| *** matrohon has joined #openstack-kolla | 20:07 | |
| wirehead_ | And Kubernetes doesn’t really want to handle mounting and unmounting drives on a running system, so you can’t do the trick where you mount things manually. | 20:07 |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Adding debugging documentation for Memcached https://review.openstack.org/330766 | 20:08 |
| *** fragatina has joined #openstack-kolla | 20:10 | |
| *** fragatina has quit IRC | 20:10 | |
| *** fragatina has joined #openstack-kolla | 20:11 | |
| wirehead_ | Mech422: Yeah, in the interests of simplicity, right now we’re doing the ‘easy’ option that’s largely free of problems with parallelism and distributed systems. A single Mariadb instance on (eventually) resilient storage. | 20:14 |
| wirehead_ | The problem is that if you yank the node that happens to have Mariadb schedueld on it, you are going to get a blip set to Kubernetes’s configured timeout before it calls a node dead. | 20:15 |
| mark-casey | and scaling | 20:16 |
| mark-casey | wirehead_: so would it be fair to say that this is difficult because mariadb is one of few things (or the only thing) that can't ask another service whether it has previously bootstrapped? | 20:17 |
| *** ayoung has quit IRC | 20:18 | |
| Mech422 | mark-casey: gah..I wish this guy would stop moving away from the microphone | 20:18 |
| *** openstackstatus has joined #openstack-kolla | 20:19 | |
| *** ChanServ sets mode: +v openstackstatus | 20:19 | |
| mark-casey | :) | 20:21 |
| *** inc0 has quit IRC | 20:23 | |
| *** dmk0202 has joined #openstack-kolla | 20:23 | |
| *** zhiwei has joined #openstack-kolla | 20:26 | |
| esharao__ | hi.. im running into issues Starting neutron-dhcp-agent container | 20:28 |
| esharao__ | Path /run/netns is mounted on /run but it is not a shared mount. | 20:28 |
| esharao__ | i did delete and redeploy as suggessted but its not working.. | 20:28 |
| Mech422 | esharao__: did you add the shared mount do your rc.local? | 20:28 |
| esharao__ | nope | 20:28 |
| wirehead_ | oh, rhallisey, can you take a peek at https://review.openstack.org/#/c/329676/ | 20:28 |
| patchbot | wirehead_: patch 329676 - kolla-kubernetes - Consolidating path resolution and searching for pr... | 20:28 |
| Mech422 | mount -oremount --make-shared /run | 20:28 |
| Mech422 | esharao__: I believe the docs mention it... | 20:29 |
| *** ravig has quit IRC | 20:29 | |
| Mech422 | esharao__: I just stuffed it in rc.local so it gets applied every boot | 20:29 |
| esharao__ | i think i missed that one.. i did do the other part to make it shared | 20:29 |
| Mech422 | other part ? | 20:29 |
| Mech422 | thats all I did | 20:29 |
| esharao__ | oh.. | 20:30 |
| esharao__ | mount --make-shared /run - i did it on the wrong server | 20:30 |
| esharao__ | thanks Mech422 | 20:30 |
| Mech422 | np | 20:30 |
| *** zhiwei has quit IRC | 20:31 | |
| esharao__ | i was trying the kubernates one on a diff vm and i did it there but not on the plain kolla one. | 20:31 |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Fix keystone bootstrap failure https://review.openstack.org/330777 | 20:32 |
| esharao__ | how is your trial of k8s going? | 20:32 |
| Mech422 | slow - too many things 'popping up' | 20:33 |
| esharao__ | :) | 20:33 |
| *** mbound has joined #openstack-kolla | 20:33 | |
| Mech422 | wirehead_: speaking of which - is ansible 2 ok for k8 ? the k8 contrib stuff says ansible 1.9 - but kolla is using 2 for master ? | 20:33 |
| wirehead_ | I’ve been testing with 2.1. | 20:34 |
| wirehead_ | Actually, that reminds me that I need to make a doc patch | 20:34 |
| Mech422 | cool - thanks | 20:34 |
| *** jtriley has quit IRC | 20:35 | |
| wirehead_ | ansible --version | 20:35 |
| wirehead_ | ansible 2.1.0.0 | 20:35 |
| rhallisey | wirehead_, sure | 20:38 |
| rhallisey | Mech422, it should be the same as kolla | 20:39 |
| rhallisey | so 2.0 or 2.1 | 20:39 |
| Mech422 | cool - I'm installing 2.1, so should be good - thanks | 20:40 |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Adding debugging documentation for Memcached https://review.openstack.org/330766 | 20:41 |
| *** daneyon has quit IRC | 20:42 | |
| *** huikang has joined #openstack-kolla | 20:50 | |
| *** b_bezak has joined #openstack-kolla | 20:52 | |
| *** arizidon has quit IRC | 20:53 | |
| *** sacharya has quit IRC | 20:54 | |
| *** b_bezak has quit IRC | 20:55 | |
| *** sacharya has joined #openstack-kolla | 20:55 | |
| *** sacharya_ has joined #openstack-kolla | 20:56 | |
| rhallisey | wirehead_, I'll add some comments tmr. I think I did find an issue though | 20:56 |
| *** jamie_h has quit IRC | 20:57 | |
| *** ppowell has quit IRC | 20:57 | |
| rhallisey | see ya! | 20:58 |
| *** rhallisey has quit IRC | 20:58 | |
| *** sacharya has quit IRC | 20:59 | |
| mark-casey | wirehead_: is this kind of what you were saying with letting the mount fail until the job is completed? http://kubernetes.io/docs/user-guide/jobs/#single-job-starts-controller-pod | 21:01 |
| wirehead_ | mark-casey: not quite. | 21:04 |
| *** Serlex has quit IRC | 21:06 | |
| *** sacharya has joined #openstack-kolla | 21:10 | |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Adding debugging documentation for Keystone https://review.openstack.org/330788 | 21:10 |
| *** sachary__ has joined #openstack-kolla | 21:11 | |
| *** ravig has joined #openstack-kolla | 21:11 | |
| *** sacharya_ has quit IRC | 21:13 | |
| *** ravig has quit IRC | 21:13 | |
| *** sacharya has quit IRC | 21:14 | |
| *** dwalsh has quit IRC | 21:15 | |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Adding debugging documentation for Keystone https://review.openstack.org/330788 | 21:16 |
| *** sachary__ has quit IRC | 21:20 | |
| *** sacharya has joined #openstack-kolla | 21:20 | |
| *** sacharya_ has joined #openstack-kolla | 21:21 | |
| *** arizidon has joined #openstack-kolla | 21:22 | |
| mark-casey | wirehead_: you mentioned most volume types not being shared-writable... but can you do multiple mounts of NAS/RBD volumes in read-only? | 21:23 |
| *** sacharya has quit IRC | 21:25 | |
| mark-casey | because then the bootstrap sidecar could mount /var/lib/mysql read-only, and another volume elsewhere read-write. If /var/lib/mysql is empty you just bind-mount the read-write volume over /var/lib/mysql inside the container and do bootstrap. Then the actual mariadb container just does the reverse: mount the actual datadir as read-write and the bootstrap volume read-only | 21:25 |
| wirehead_ | mark-casey: yah | 21:27 |
| Mech422 | my sister just called - she left her laptop with the IT guy at work to update office.. | 21:27 |
| Mech422 | got the machine back with win10, updated office, and ALL her files gone :-P | 21:27 |
| mark-casey | Mech422: ugh. stomach just dropped | 21:28 |
| Mech422 | Spent the next half hour trying to explain to her the linux isn't windows and I have no idea how to get her files back... | 21:28 |
| wirehead_ | Heh, so my mom is actually better at coaxing WIndows machines back to functionality than I Am. | 21:28 |
| Mech422 | mark-casey: yah - their 'IT guy' sounds like a winner... | 21:28 |
| wirehead_ | (Of course, she was a mainframe engineer back in the day) | 21:28 |
| Mech422 | wirehead_: oh nice - my mom used to think my name was 'help desk' :-P | 21:29 |
| mark-casey | Mech422: I can tell you that the first thing to do is turn the thing off | 21:29 |
| wirehead_ | Yeah. She’s got a large crop of very dedicated customers. | 21:30 |
| wirehead_ | And she’s patient about it. She’ll fix things instead of my usual burn-it-down-and-restore-from-backup approach. | 21:31 |
| mark-casey | IT man likely did a quick format with ntfs. tools like r-studio can recover data but not if new files overwrite :D | 21:31 |
| *** b_bezak has joined #openstack-kolla | 21:32 | |
| *** diogogmt has quit IRC | 21:34 | |
| *** mark-casey has quit IRC | 21:34 | |
| *** diogogmt has joined #openstack-kolla | 21:36 | |
| *** matrohon has quit IRC | 21:38 | |
| *** dmk0202 has quit IRC | 21:38 | |
| *** daneyon has joined #openstack-kolla | 21:40 | |
| *** b_bezak has quit IRC | 21:42 | |
| *** daneyon has quit IRC | 21:44 | |
| Mech422 | wirehead_: yeah - I have no patience for windows - thinks it smarter then you are | 21:45 |
| Mech422 | wirehead_: I want the machine to do as it's told, not argue with me | 21:45 |
| *** ravig has joined #openstack-kolla | 21:47 | |
| *** inc0 has joined #openstack-kolla | 21:51 | |
| *** huikang has quit IRC | 21:51 | |
| *** b_bezak has joined #openstack-kolla | 21:59 | |
| *** huikang has joined #openstack-kolla | 21:59 | |
| *** huikang has quit IRC | 22:04 | |
| *** dmk0202 has joined #openstack-kolla | 22:04 | |
| *** ayoung has joined #openstack-kolla | 22:07 | |
| *** b_bezak has quit IRC | 22:12 | |
| *** ravig has quit IRC | 22:14 | |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Fix docs for Ansible 2.1 support, remove some of the notes. https://review.openstack.org/330805 | 22:16 |
| *** inc0 has quit IRC | 22:21 | |
| *** vhosakot has quit IRC | 22:29 | |
| *** salv-orlando has quit IRC | 22:29 | |
| *** salv-orlando has joined #openstack-kolla | 22:30 | |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Create ConfigMaps with dashes instead of underscores https://review.openstack.org/330247 | 22:31 |
| *** salv-orlando has quit IRC | 22:34 | |
| *** salv-orlando has joined #openstack-kolla | 22:35 | |
| *** esharao__ has quit IRC | 22:38 | |
| *** dmk0202 has quit IRC | 22:39 | |
| *** ravig has joined #openstack-kolla | 22:41 | |
| *** arizidon has quit IRC | 22:42 | |
| *** zhiwei has joined #openstack-kolla | 22:45 | |
| *** zhiwei has quit IRC | 22:50 | |
| *** ravig has quit IRC | 22:53 | |
| *** ayoung has quit IRC | 23:07 | |
| openstackgerrit | Ken Wronkiewicz proposed openstack/kolla-kubernetes: Fix Horizon's configmap to work with underscores https://review.openstack.org/330816 | 23:07 |
| *** salv-orl_ has joined #openstack-kolla | 23:08 | |
| *** diogogmt has quit IRC | 23:10 | |
| *** salv-orlando has quit IRC | 23:11 | |
| *** daneyon has joined #openstack-kolla | 23:16 | |
| harlowja_ | oh ya, since i forgot, sdake the recommended filesystem for things kolla will be deploying to (so that docker images don't suck) was btrfs ? | 23:16 |
| sdake | harlowja_ i'd go with that and test it heavily ;) | 23:17 |
| harlowja_ | ya, mdorman you around, wondering how that will work out for godaddy, ha | 23:17 |
| sdake | use the -s option to docker | 23:17 |
| harlowja_ | didn't think (but not sure) if we use btrfs anywhere , lol | 23:17 |
| sdake | just /var/lib/docker | 23:17 |
| sdake | which version of centos are you on | 23:18 |
| * harlowja_ jumps on a test machine | 23:18 | |
| sdake | i have had really good success with overlayfs as a target for deployment | 23:18 |
| sdake | but building on overlayfs is no beuno | 23:18 |
| sdake | also red hat says overlayfswill eat chickens | 23:18 |
| sdake | in their 3.10 7.2 centos kernel | 23:19 |
| sdake | so ymmv | 23:19 |
| harlowja_ | ext3 with LVM members (hmmmm) , unsure if this test machine is the normal setup | 23:19 |
| harlowja_ | with g 3.10.0-327. | 23:19 |
| harlowja_ | * 3.10.0-327.4.5.el7.x86_64 | 23:19 |
| harlowja_ | seems like another machine is ext4 + lvm | 23:20 |
| harlowja_ | i'll divert which FS to mdorman :-P | 23:20 |
| mdorman | hey | 23:21 |
| harlowja_ | mdorman hey hey, i was remembering something something about filesystem type that best works with docker | 23:21 |
| harlowja_ | and how sdake was saying btrfs and then i was like hmmmm, uh | 23:21 |
| mdorman | yah all of our stuff is ext4 | 23:22 |
| mdorman | is it just bad performance b/c of the layering? | 23:22 |
| sdake | reliability because lvm device mapper sucks | 23:22 |
| sdake | aufs is an option on ubuntian systems | 23:22 |
| * harlowja_ wonders what other people using docker at godaddy are using :-/ | 23:23 | |
| sdake | overlayfs will run over ext4 | 23:23 |
| *** arafferty_ has joined #openstack-kolla | 23:23 | |
| sdake | and reliably from my year long testing | 23:23 |
| harlowja_ | but u said it eats my chickens | 23:23 |
| harlowja_ | lol | 23:23 |
| sdake | red hat says it eats chickens | 23:23 |
| sdake | i say it eats chickens if you docker build on it :) | 23:23 |
| harlowja_ | ah | 23:24 |
| sdake | i think the ovl red hat plugin is busted | 23:24 |
| mdorman | i am not sure if all of our stuff uses LVM or not. i thought most prod things were real physical partitions | 23:25 |
| mdorman | eh no i’m wrong, only /boot is a real partition | 23:25 |
| *** mummer has quit IRC | 23:26 | |
| sdake | what i mean is docker uses lvm | 23:26 |
| mdorman | so you’re saying btrfs is the recommendation? | 23:26 |
| sdake | by defualt | 23:26 |
| mdorman | oh | 23:26 |
| sdake | and lvm + docker is not good | 23:26 |
| harlowja_ | what a weird defaut | 23:26 |
| harlowja_ | lol | 23:26 |
| harlowja_ | *default | 23:26 |
| sdake | i hear they changed it recently | 23:26 |
| sdake | but not certain on that point | 23:26 |
| sdake | i'd go with btrfs + backups :) | 23:27 |
| sdake | if you really want to get wild throw ceph into the mix! | 23:27 |
| harlowja_ | mdorman has PTSD about that | 23:27 |
| wirehead_ | BeFS> | 23:27 |
| harlowja_ | :-/ | 23:28 |
| harlowja_ | lol | 23:28 |
| sdake | sadfacefs # | 23:28 |
| sdake | seriously though on centos btrfs is what i'd use for /var/lib/docker | 23:28 |
| sdake | and keep the rest on ext4 | 23:28 |
| sdake | then deciding on ceph is an orthogonal issue | 23:30 |
| sdake | depending on your risk tolerance | 23:30 |
| sdake | which I assume is low :) | 23:30 |
| sdake | great hting about ceph is its super easy to backup | 23:30 |
| sdake | bad thing about ceph is it has eaten chickens before | 23:30 |
| sdake | although so has btrfs | 23:30 |
| sdake | ceph + openstack is a great pair | 23:31 |
| sdake | really well integrated | 23:31 |
| mdorman | so can you tell me more why running it on ext4 is a bad idea? | 23:31 |
| sdake | sure | 23:31 |
| sdake | lvm + devicemapper locks containers disk space | 23:31 |
| sdake | then they become undeleteable | 23:31 |
| sdake | hence there is no way to upgrade without a reboot | 23:32 |
| sdake | its some kind of reference counting problem | 23:32 |
| sdake | the community pretty much gave up | 23:32 |
| sdake | and by community i mean docker and lvm community | 23:32 |
| sdake | once the reboot is done the locks are cleared | 23:33 |
| mdorman | i see. so because the underlying ext4 fs doesn’t have facilities for this layering stuff, docker has to do it using lvm | 23:33 |
| sdake | the containers can be deleted and replaced | 23:33 |
| mdorman | is that a corrrec statement? | 23:33 |
| sdake | roger that | 23:33 |
| mdorman | ok | 23:33 |
| sdake | btrfs overlayfs aufs all have layering built in | 23:33 |
| *** daneyon has quit IRC | 23:33 | |
| mdorman | well this should be fun | 23:33 |
| harlowja_ | lol | 23:33 |
| sdake | overlayfs is the cleanest implementation | 23:33 |
| sdake | but its clean because its new | 23:34 |
| sdake | and since its green its a bit buggy here and there | 23:34 |
| mdorman | right. | 23:34 |
| sdake | if you ere using master of kernel i'd recommend overlayfs over all else | 23:34 |
| sdake | or some hardened version of master | 23:34 |
| sdake | i.e. centos 8 :) | 23:34 |
| mdorman | k | 23:34 |
| harlowja_ | master of kernel = ? | 23:34 |
| harlowja_ | centos8 :-/ | 23:34 |
| harlowja_ | lol | 23:35 |
| harlowja_ | u a funny one | 23:35 |
| sdake | been one of those days | 23:35 |
| sdake | red hat has backported a bunch off overlayfs fixes into 3.10 kernel | 23:35 |
| sdake | but its a temporary (20 years in red hat) fork of the kernel and as such isn't as tidy as upstream | 23:36 |
| mdorman | yeah we’ll have to look into all this a little more. good to have the tips though for sure | 23:36 |
| sdake | cool happy to help | 23:36 |
| harlowja_ | sdake got a link to those patches? | 23:36 |
| harlowja_ | i can probably find them, but u might be quicker :-P | 23:37 |
| sdake | harlowja_ no i dont | 23:37 |
| sdake | harlowja_ just what i hear through grapevine | 23:37 |
| harlowja_ | ok, i find, ha | 23:37 |
| sdake | i think red hat is compressing their kernel patches now | 23:37 |
| harlowja_ | for real | 23:37 |
| harlowja_ | :( | 23:37 |
| sdake | rather squashing | 23:37 |
| harlowja_ | damn oracle damn | 23:37 |
| harlowja_ | damn | 23:37 |
| harlowja_ | bleep | 23:37 |
| sdake | so oracle | 23:37 |
| sdake | yup | 23:37 |
| sdake | oracle ;) | 23:37 |
| harlowja_ | ya, not cool | 23:37 |
| harlowja_ | stupid | 23:37 |
| sdake | funny thing is i hear oracle wrote an automated tool to de-squash the patches | 23:38 |
| harlowja_ | AI? | 23:38 |
| sdake | not sure | 23:38 |
| sdake | just wwhat i hear | 23:38 |
| sdake | no first hand knowledge | 23:38 |
| harlowja_ | wonder how that would be possible, since u sorta lose all the context | 23:38 |
| harlowja_ | anyways | 23:38 |
| harlowja_ | stupid | 23:38 |
| harlowja_ | ha | 23:38 |
| wirehead_ | I’m constantly amazed by the distro drama. | 23:38 |
| sdake | i think i could write such a tool given enough time nad money :) | 23:38 |
| harlowja_ | well can u locate the context from somewhere? | 23:39 |
| sdake | yup - from upstream | 23:39 |
| harlowja_ | i mean, u can split off the diff hunks into <something> | 23:39 |
| harlowja_ | ah | 23:39 |
| harlowja_ | ok, so the context is still somewhere | 23:39 |
| harlowja_ | then, the whole thing is really dumb | 23:39 |
| harlowja_ | lol | 23:39 |
| sdake | red hat wasn't wearing their red hats when they made that choice | 23:39 |
| harlowja_ | def | 23:39 |
| wirehead_ | I’ve long been on a quest to acquire a genuine Red Hat red fedora hat the hard way. As in, stealing it, preferebly with a story attached, instead of getting it through employment. | 23:40 |
| *** ravig has joined #openstack-kolla | 23:40 | |
| harlowja_ | https://oss.oracle.com/projects/RedPatch/ | 23:40 |
| harlowja_ | lol | 23:40 |
| harlowja_ | "What is RedPatch? | 23:40 |
| harlowja_ | The aim of the RedPatch project is to take all of the source code changes of the Red Hat Kernel from release to release and to break them up into smaller, stand-alone changes, using upstream commits whenever possible." | 23:40 |
| harlowja_ | sounds like that project | 23:41 |
| harlowja_ | ha | 23:41 |
| harlowja_ | stupid corporate wars | 23:41 |
| wirehead_ | Well, I worked with a bunch of ex-Cannonical folks and they had stories. | 23:42 |
| wirehead_ | In a sense, it’s kind of like Newer Better Now With Nitrogen Engine Cleansing Additives for gas. | 23:42 |
| wirehead_ | It’s the same gas, with some magical mystical ‘special sauce’ they try to sell you. | 23:43 |
| harlowja_ | lol | 23:43 |
| harlowja_ | true dat | 23:44 |
| *** ayoung has joined #openstack-kolla | 23:44 | |
| *** cloudnau_ has joined #openstack-kolla | 23:45 | |
| *** salv-orl_ has quit IRC | 23:47 | |
| *** salv-orlando has joined #openstack-kolla | 23:48 | |
| *** huikang has joined #openstack-kolla | 23:56 | |
| *** sdake has quit IRC | 23:57 | |
| *** mbound has quit IRC | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!