Tuesday, 2016-06-28

*** daneyon_ has joined #openstack-kolla00:14
*** sdake has quit IRC00:15
*** haplo37 has quit IRC00:15
*** g3ek has quit IRC00:16
*** haplo37 has joined #openstack-kolla00:16
*** diogogmt has quit IRC00:17
*** g3ek has joined #openstack-kolla00:17
*** daneyon_ has quit IRC00:18
*** dwalsh has joined #openstack-kolla00:42
*** aNupoisc has quit IRC00:42
*** zhiwei has joined #openstack-kolla00:53
*** Mr_Broken has quit IRC00:57
*** haplo37 has quit IRC00:58
*** aernhart has quit IRC01:06
*** haplo37 has joined #openstack-kolla01:07
*** Mr_Broken has joined #openstack-kolla01:07
*** Mr_Broken has quit IRC01:12
*** zhurong has joined #openstack-kolla01:12
*** Mr_Broken has joined #openstack-kolla01:14
*** zhurong has quit IRC01:16
*** jtriley has quit IRC01:18
*** dwalsh has quit IRC01:18
*** zhurong has joined #openstack-kolla01:19
*** yuanying has quit IRC01:22
*** williamcaban has joined #openstack-kolla01:25
*** yuanying has joined #openstack-kolla01:25
*** jtriley has joined #openstack-kolla01:29
*** banix has joined #openstack-kolla01:29
*** banix has quit IRC01:39
*** tfukushima has joined #openstack-kolla01:51
*** Mr_Broken has quit IRC01:54
*** Mr_Broken has joined #openstack-kolla01:55
*** sacharya has joined #openstack-kolla01:59
*** ssurana has joined #openstack-kolla02:01
openstackgerritYoshifumi Sumida proposed openstack/kolla: Heat - F21 images not available for download  https://review.openstack.org/33472902:02
*** daneyon_ has joined #openstack-kolla02:02
*** zhurong has quit IRC02:02
*** zhurong has joined #openstack-kolla02:03
openstackgerritYoshifumi Sumida proposed openstack/kolla: Heat - F21 images not available for download  https://review.openstack.org/33472902:06
*** Jeffrey4l__ has quit IRC02:07
*** daneyon_ has quit IRC02:07
*** jtriley has quit IRC02:08
*** jtriley has joined #openstack-kolla02:12
*** ssurana has quit IRC02:13
*** williamcaban has quit IRC02:16
*** jtriley has quit IRC02:17
*** fragatina has joined #openstack-kolla02:18
*** fragatina has quit IRC02:18
*** fragatina has joined #openstack-kolla02:19
*** sacharya_ has joined #openstack-kolla02:24
*** sacharya has quit IRC02:27
*** banix has joined #openstack-kolla02:28
*** mark-casey has joined #openstack-kolla02:34
*** beekhof has quit IRC02:37
*** g3ek has quit IRC02:37
*** g3ek has joined #openstack-kolla02:38
*** sdake has joined #openstack-kolla02:40
mark-caseysdake: ping02:41
*** beekhof has joined #openstack-kolla02:41
*** salv-orl_ has quit IRC02:41
*** jtriley has joined #openstack-kolla02:43
*** Mr_Broken has quit IRC02:44
*** yuanying has quit IRC02:50
*** klint has joined #openstack-kolla02:52
*** sdake has quit IRC02:56
*** Mr_Broken has joined #openstack-kolla03:02
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update kollaglue repo on docker to kolla  https://review.openstack.org/33311003:03
*** banix has quit IRC03:06
*** sdake has joined #openstack-kolla03:08
*** Mr_Broken has quit IRC03:10
*** Mr_Broken has joined #openstack-kolla03:12
*** haplo37 has quit IRC03:21
openstackgerritHui Kang proposed openstack/kolla: Add etcd Docker container  https://review.openstack.org/29845103:23
*** jtriley has quit IRC03:27
*** sdake has quit IRC03:27
*** haplo37 has joined #openstack-kolla03:29
*** yuanying has joined #openstack-kolla03:48
*** salv-orlando has joined #openstack-kolla03:48
*** salv-orlando has quit IRC03:57
*** yuanying has quit IRC04:00
*** yuanying has joined #openstack-kolla04:01
*** stvnoyes has quit IRC04:02
*** stvnoyes1 has joined #openstack-kolla04:03
*** y-sumida has joined #openstack-kolla04:05
*** y-sumida has left #openstack-kolla04:10
openstackgerritMd Nadeem proposed openstack/kolla: Fix Mariadb restarting issue  https://review.openstack.org/33375304:22
*** salv-orlando has joined #openstack-kolla04:23
*** Jeffrey4l has joined #openstack-kolla04:31
*** salv-orlando has quit IRC04:32
*** fragatina has quit IRC04:39
*** Mr_Broken has quit IRC04:42
*** daneyon_ has joined #openstack-kolla04:45
*** yuanying has quit IRC04:49
*** daneyon_ has quit IRC04:49
*** Mr_Broken has joined #openstack-kolla04:53
*** g3ek has quit IRC04:56
*** haplo37 has quit IRC04:56
*** salv-orlando has joined #openstack-kolla04:58
*** salv-orl_ has joined #openstack-kolla04:59
*** salv-orlando has quit IRC05:03
*** g3ek has joined #openstack-kolla05:03
*** haplo37 has joined #openstack-kolla05:04
*** salv-orl_ has quit IRC05:04
*** tfukushima has quit IRC05:13
*** ayoung has quit IRC05:14
*** hanchao has quit IRC05:16
*** mewald has joined #openstack-kolla05:18
*** fragatina has joined #openstack-kolla05:20
*** mewald has quit IRC05:24
*** JianqingJiang has joined #openstack-kolla05:27
*** hanchao has joined #openstack-kolla05:28
*** salv-orlando has joined #openstack-kolla05:33
*** Mr_Broken has quit IRC05:40
*** Mr_Broken has joined #openstack-kolla05:41
*** salv-orlando has quit IRC05:41
*** haplo37 has quit IRC05:44
*** g3ek has quit IRC05:45
*** g3ek- has joined #openstack-kolla05:47
*** g3ek- is now known as g3ek05:47
*** haplo37 has joined #openstack-kolla05:47
*** tfukushima has joined #openstack-kolla05:51
*** Mr_Broken has quit IRC05:56
*** Mr_Broken has joined #openstack-kolla05:57
mark-caseyanyone have a log or paste of running kolla-ansible upgrade? I'm trying to see some order of events and don't have a lab set up right now. Anything from the last few months would be fine06:02
*** yuanying has joined #openstack-kolla06:05
*** salv-orlando has joined #openstack-kolla06:05
*** yuanying has quit IRC06:11
*** yuanying has joined #openstack-kolla06:12
mark-caseyfound one, nevermind06:12
*** Mr_Broken has quit IRC06:14
*** JeroenBo has joined #openstack-kolla06:14
*** bootsha has joined #openstack-kolla06:21
*** y-sumida has joined #openstack-kolla06:23
*** daneyon_ has joined #openstack-kolla06:33
*** daneyon_ has quit IRC06:37
*** anbehl has quit IRC06:52
*** Mr_Broken has joined #openstack-kolla06:57
*** mewald has joined #openstack-kolla06:58
*** b_bezak has joined #openstack-kolla07:01
mewaldCan Kolla deploy Glance, Nova and Cinder with connection information to an existing Ceph cluster that wasn't deployed by Kolla?07:06
*** haplo37 has quit IRC07:14
*** g3ek has quit IRC07:14
*** g3ek has joined #openstack-kolla07:23
*** haplo37 has joined #openstack-kolla07:23
openstackgerritMerged openstack/kolla: Create ansible home directory in kolla-toolbox container  https://review.openstack.org/33423707:29
*** athomas has joined #openstack-kolla07:30
mdnadeemmark-casey, I have done some 2 week ago, here is ansible.log http://pastebin.com/ywRyhb5w07:31
mdnadeemmark-casey, Let me know if you need any other log07:31
mark-caseythanks :)07:31
openstackgerritMerged openstack/kolla: Add test for remove container, get state and get env  https://review.openstack.org/33155607:33
*** yuanying has quit IRC07:34
*** yuanying has joined #openstack-kolla07:34
*** yuanying has quit IRC07:35
*** belmoreira has joined #openstack-kolla07:35
*** yuanying has joined #openstack-kolla07:36
*** salv-orlando has quit IRC07:41
*** Serlex has joined #openstack-kolla07:50
*** diogogmt has joined #openstack-kolla07:53
*** shardy_afk is now known as shardy07:55
*** y-sumida has quit IRC07:57
*** kproskurin has joined #openstack-kolla07:59
mewaldwhat is nova-ssh container? looks like it's just a ssh service nothing else. Why?08:00
*** sacharya_ has quit IRC08:01
*** g3ek has quit IRC08:04
*** y-sumida has joined #openstack-kolla08:05
*** dmk0202 has joined #openstack-kolla08:06
*** haplo37 has quit IRC08:06
Jeffrey4lmewald, the nova migrate/resize/rebuild need the ssh service. nova copy the image to another host by using rsync08:08
*** g3ek has joined #openstack-kolla08:11
*** haplo37 has joined #openstack-kolla08:11
*** salv-orlando has joined #openstack-kolla08:13
*** gfidente has joined #openstack-kolla08:15
*** salv-orlando has quit IRC08:17
*** y-sumida has quit IRC08:20
*** Serlex has quit IRC08:21
*** daneyon_ has joined #openstack-kolla08:21
*** daneyon_ has quit IRC08:25
*** y-sumida has joined #openstack-kolla08:27
*** b_bezak has quit IRC08:29
*** b_bezak has joined #openstack-kolla08:32
*** b_bezak has quit IRC08:32
*** b_bezak has joined #openstack-kolla08:33
openstackgerritMerged openstack/kolla: Update kollaglue repo on docker to kolla  https://review.openstack.org/33311008:38
*** b_bezak_ has joined #openstack-kolla08:38
*** b_bezak has quit IRC08:38
*** mbound has joined #openstack-kolla08:42
*** bootsha has quit IRC08:43
*** salv-orlando has joined #openstack-kolla08:48
*** g3ek has quit IRC08:53
*** haplo37 has quit IRC08:53
*** haplo37 has joined #openstack-kolla08:54
*** g3ek has joined #openstack-kolla08:55
*** salv-orlando has quit IRC08:56
*** Serlex has joined #openstack-kolla08:58
*** salv-orlando has joined #openstack-kolla09:00
openstackgerritMerged openstack/kolla: Be smarter about what to do when making a docker client  https://review.openstack.org/33017109:10
*** bootsha has joined #openstack-kolla09:14
*** yuanying has quit IRC09:26
*** prithiv has joined #openstack-kolla09:28
*** mark-casey has quit IRC09:28
*** y-sumida has quit IRC09:32
*** dwalsh has joined #openstack-kolla09:34
*** bootsha has quit IRC09:38
*** prithiv has quit IRC09:40
*** david-lyle has quit IRC09:44
*** david-lyle has joined #openstack-kolla09:46
*** bootsha has joined #openstack-kolla09:49
*** hanchao has quit IRC09:58
*** zhurong has quit IRC10:01
*** JianqingJiang has quit IRC10:11
*** tfukushima has quit IRC10:27
*** dwalsh has quit IRC10:27
*** dwalsh has joined #openstack-kolla10:29
Mr_Broken@mewald: did you deploy kolla to a existing Ceph cluster ? or did you just use kolla-ansible reconfigure to fiddle with configs ?10:34
*** yuanying has joined #openstack-kolla10:47
openstackgerritMd Nadeem proposed openstack/kolla: Added unit test for image operation  https://review.openstack.org/33491710:54
pbourkemewald: not currently10:57
openstackgerritMd Nadeem proposed openstack/kolla: Added unit test for image operation  https://review.openstack.org/33491710:58
pbourkemewald: though you may be able to do it with a combo of augment files and customised container json files to pick up the keyring files10:58
Mr_Brokenpbourke: are you refering to deploying kolla with external ceph cluster ?10:58
Mr_BrokenI got the same task to do soon ;]10:59
*** salv-orl_ has joined #openstack-kolla10:59
mewaldMr_Broken pbourke: currently I did nothing, we are going to have to deploy OpenStack with an existing Fujitsu CD10000 (Ceph). So that's going to be a problem :D11:00
*** salv-orlando has quit IRC11:02
Mr_Brokenmewald: yeah i need to do similar thing - havent looked at the docs yet, but I assumed it will be a matter of changing some cinder configuration files ?11:09
*** anbehl has joined #openstack-kolla11:10
*** ppowell_ has joined #openstack-kolla11:16
*** tyrola has joined #openstack-kolla11:19
*** g3ek has quit IRC11:19
*** haplo37 has quit IRC11:20
*** fragatin_ has joined #openstack-kolla11:22
*** fragatina has quit IRC11:22
sean-k-mooneydeploying with an external ceph cluser would be a very nic enchancment11:24
sean-k-mooneyit is needed for kolla-kubernetes anyway so it would be nice to have in kolla-ansible11:24
*** haplo37 has joined #openstack-kolla11:25
*** g3ek has joined #openstack-kolla11:25
*** fragatin_ has quit IRC11:30
*** mliima has joined #openstack-kolla11:34
*** rhallisey has joined #openstack-kolla11:38
*** zhurong has joined #openstack-kolla11:42
*** fragatina has joined #openstack-kolla11:54
*** sdake has joined #openstack-kolla11:58
*** sdake_ has joined #openstack-kolla12:00
sdake_hey folks12:00
sdake_morning12:00
sdake_why oh wy do i wake up at 5am every day12:01
*** dwalsh has quit IRC12:01
*** dwalsh has joined #openstack-kolla12:01
*** neilus has quit IRC12:02
*** sdake has quit IRC12:03
*** neilus has joined #openstack-kolla12:03
*** neilus has quit IRC12:05
*** neilus has joined #openstack-kolla12:05
*** g3ek has quit IRC12:07
*** haplo37 has quit IRC12:08
openstackgerritPaul Bourke proposed openstack/kolla: Add an alternative disk provisioning method for Ceph  https://review.openstack.org/33497012:08
mewaldMr_Broken: Yeah that's not all unfortunately. Sure, some configuration needs to go into the ini file (e.g. glance-api.ini) but we need to place a ceph.conf and keyring in /etc/ceph/.12:08
mewaldAt the moment, that information is fetched executing "docker exec ceph auth get-or-create …" but with an existing Ceph Cluster it's not going to work like that12:09
*** haplo37 has joined #openstack-kolla12:13
*** g3ek has joined #openstack-kolla12:13
*** neilus1 has joined #openstack-kolla12:16
*** neilus1 has joined #openstack-kolla12:17
*** athomas has quit IRC12:18
*** neilus has quit IRC12:18
*** williamcaban has joined #openstack-kolla12:20
sdake_mewald might i inquire what your speaking about12:20
sdake_mewald are you talking about using external ceph?12:21
Mr_Brokensdake: ytes12:21
Mr_Broken*yes12:21
sdake_i'd love to see that work12:21
Mr_Brokenkolla deploy, and connecting to external ceph cluster12:21
*** diogogmt has quit IRC12:21
Mr_Brokenyeah, me 2 :]12:21
Mr_Brokensince i need to set it up next week :]12:21
sdake_i dodnt think its hard12:21
sdake_but i dont know for sure12:21
sdake_i dont have any idea how ceph works or i'd do the implementation already12:21
Mr_Brokenhopefully I will have some insight next week ;]12:22
*** neilus1 has quit IRC12:22
mewaldI will set up a lab today and begin working on what needs to be done12:22
sdake_Mr_Broken it will probablly require changes12:22
Mr_Brokenmy friend is seting up a ceph cluster right now, we will try to connect it to kolla as soon as its up12:22
mewaldI browsed the code a bit an I am pretty confident its not a lot12:22
*** diogogmt has joined #openstack-kolla12:22
sdake_mewald ya i think its probably asmall sizzed task12:23
*** athomas has joined #openstack-kolla12:23
Mr_Brokenmewald: any input appreciated :D12:23
sdake_what would help is if  someone that undersstood ceph did the work :)12:23
sdake_it would take me weeks to learn the ins and outs of ceph12:23
sdake_which Iguess I should do anyway12:23
*** berendt has joined #openstack-kolla12:23
mewaldsdake_: yeah that's why I am trying to do it. I know Ceph pretty well, BUT don't know Kolla very well yet :D12:24
sdake_mewald cool if your going ot take it on i can help you around with kolla12:24
sdake_is there an existing blueprint?12:25
mewaldwhere would I check?12:25
sdake_launchpad.net/kolla12:26
sdake_i'll take a look12:26
Mr_Broken@mewald: you said that using ceph auth (http://docs.ceph.com/docs/hammer/man/8/ceph-authtool/) is not working in docker ?12:26
Mr_Brokenor what is the problem there ?12:26
sdake_https://blueprints.launchpad.net/kolla/+spec/external-ceph12:26
mewaldyeah just found that too :)12:27
mewaldI need some time to be able to understand this Gerrit thingy :D I am not really a developer ^^12:27
sdake_looks like some progress on the blueprint12:28
mewaldYou gave it a down vote :D12:31
sdake_gave what a down vote12:31
sdake_a -1 vote only means more work needs to be done on the patch12:32
sdake_a -2 vote is a "down vote" or veto of the technical work12:32
mewaldah12:33
*** diogogmt has quit IRC12:34
sean-k-mooney-2 are also sometimes adminsitrative. e.g. we are past a code freeze or where a spec is required but not approve the code may be given a -2 untill the spec is approved.12:34
sean-k-mooneykolla tends not to use specs much though12:34
sdake_ya - at midcycle i think we are going to have a  40 minute discussion about specs and their future use12:35
sdake_kolla is becoming too hard to manage12:36
sdake_i tihnk its important to operate without specs for as long as possible12:36
sean-k-mooneywell as it grows specs become more needed for large changes but specs do slow down the development of smaller features12:36
sdake_yup12:37
sdake_tradeoff12:37
sean-k-mooneykolla currently has a nice blance where specs are only used for large or controveral features12:37
sdake_the problem we have as the core team is we have 87 blueprints12:37
mewaldsdake_: how do we proceed to make this work? What needs to be done with that Commit so it would pass and be merged?12:38
sean-k-mooneyactully 5 of them are mine but i tink that can be reduced to 312:38
sdake_mewald which commit ?12:38
sdake_sean-k-mooney if you have bps i should remove let me know their links12:38
sdake_sean-k-mooney all for tryign to get rid of work :)12:38
mewaldsdake_: the one that implements external ceph?12:39
mewald-?+.12:39
sdake_mewald link please?12:39
mewaldhttps://review.openstack.org/#/c/289462/12:39
patchbotmewald: patch 289462 - kolla (stable/liberty) - Modify the conditional strategy for cinder and gla... (ABANDONED)12:39
sean-k-mooneywell basically i think we can combine 2 of the smaller blueprints into the host prep playbook.12:39
sean-k-mooneysdake_: i will look at them later today and either close them or let you know the links to delete them12:40
sdake_sean-k-mooney cool thanks12:40
sdake_mewald that patch was not submitted against mater12:40
sdake_that is why it was -2ed12:40
mewaldwhat is mater? :D12:40
sdake_masater12:40
sdake_the tip of dev12:41
sdake_everything goes in master first12:41
sdake_then the core team handles backports12:41
*** matrohon has joined #openstack-kolla12:41
mewaldah ok12:42
*** neilus has joined #openstack-kolla12:42
*** bootsha has quit IRC12:43
mewaldso we can just commit it to master then? Or should I test it before that against my cluster?12:43
*** neilus has joined #openstack-kolla12:45
sdake_mewald it also needs a rebase12:47
sdake_mewald i dont know what htose two patches do12:47
sean-k-mooneysdake_: it allow you to deploy kolla with an external ceph cluster but still have cinder... use ceph12:48
sean-k-mooneyso basically it will remove the requirement for kolla to have deployed the ceph cluster before kolla can use it12:49
sdake_cool looks like the patch needs alot of work12:49
sdake_has alot of comments in it12:49
sean-k-mooneyits also currently abandoned https://review.openstack.org/#/c/28946212:50
sean-k-mooneythis will be very usefull to have though12:50
*** haplo37 has quit IRC12:50
Mr_Brokenquick question: what does kolla uses - priviledged or unpriviledged containers for ceph deployment ?12:50
sean-k-mooneyit will be needed for k8s12:50
*** rmart04 has joined #openstack-kolla12:51
sean-k-mooneyMr_Broken: i think privalaged for the OSDs12:51
sean-k-mooneyMr_Broken: not sure though12:51
Mr_Brokenkk thanks12:51
sean-k-mooneyMr_Broken: https://github.com/openstack/kolla/blob/master/ansible/roles/ceph/tasks/start_osds.yml#L4112:52
sean-k-mooneythe ceph mon and rados gateway are unprivalaged12:52
Mr_Brokenkk cool12:53
*** klint has quit IRC12:56
sdake_mewald - if you like I can rebase that patch to master - but it has a whole lot of comments that need addressing12:57
*** haplo37 has joined #openstack-kolla13:00
mewaldsdake_: all this is very new to me :) I just clone the Gerrit repository of Kolla and used "git fetch ssh://mewald@review.openstack.org:29418/openstack/kolla refs/changes/62/289462/1 && git checkout FETCH_HEAD" to get to the commit that includes one of the patches. How would I proceed?13:00
sdake_mewald did that fetch work properly?13:01
mewaldI said something about detached head - so Im not sure :D13:01
sdake_mewald if you have 15 mins to spend on webex, i can wlak you through how to rebase the patch to master13:01
sdake_its a little complicated since its also in merge conflict13:02
mewaldsure would be awesome!13:02
*** Mr_Broken has quit IRC13:02
sdake_but you will learn how to rebase13:02
sdake_do you have a mac or windows?13:02
mewaldmac13:02
sdake_have 15 misn now?13:02
mewaldyep13:02
sdake_http://cisco.webex.com/join/stdake13:03
sdake_anyone is welcome to join and watch in13:03
*** v1k0d3n has joined #openstack-kolla13:03
*** jtriley has joined #openstack-kolla13:03
sdake_ok i'm in13:04
mewaldworking on it :)13:04
sdake_ya - not sure what its like f your ssystem isn't etup13:05
sdake_we will need to screen share13:05
sdake_so have to connect via mac13:05
*** dwalsh has quit IRC13:07
*** zhurong has quit IRC13:08
*** diogogmt has joined #openstack-kolla13:15
pbourkeJeffrey4l: its not valid to dismiss bugs on LTS distros as "you need to upgrade"13:18
*** sdake_ has quit IRC13:23
*** banix has joined #openstack-kolla13:24
*** sdake has joined #openstack-kolla13:24
mewaldsdake: what happened? :D13:24
sdakelost power13:24
sdakesorry13:24
sdakeyou may have to rejoin13:24
mewaldok :D13:25
sdakelaptop went to sleep because not plugged in13:25
ccesariosdake, https://git.centos.org/blob/rpms!device-mapper-multipath.git/12917067990de3a976e7a1a5abfbe28891216c77/SOURCES!0106-RHBZ-1212590-dont-use-var.patch;jsessionid=glu0ds254mfg1v056w85c8q81#L18   need to be 'bindmounted' ?13:27
*** huikang has joined #openstack-kolla13:31
*** inc0 has joined #openstack-kolla13:31
inc0good morning13:32
huikangmonring13:32
sean-k-mooneyo/ inc013:32
sean-k-mooneyhi huikang  inc0  how are ye both13:33
inc0pretty well, how are things out there on islands?13:33
inc0I heard it's pretty tense now13:33
sean-k-mooneywith the brexit stuff?13:33
inc0yeah13:34
huikangor the Euro Cup13:34
sean-k-mooneyireland is still asessing what the impact will be13:34
sean-k-mooneywell of course the euros are more important :P13:35
inc0I've heard that Northern Ireland is thinking of separation from UK13:35
inc0yeah, Poland stronk in this euro \o/13:35
sean-k-mooneywell scotland are thingink of having a second reforendum on leaving the uk to remain in the eu13:36
sean-k-mooneyif they do its very unlikly that the north will not follow suite13:36
sean-k-mooneygrabing some tea13:37
sean-k-mooneybe back soon13:37
huikangWoW; this is real Democratic13:37
inc0well, technically united kingdom is not a country on it's own13:37
inc0Scotland, Ireland and such retain certain amount of independence13:38
huikangyup13:38
openstackgerritMathias proposed openstack/kolla: Add external ceph support for cinder and glance  https://review.openstack.org/31020813:40
*** haplo37 has quit IRC13:40
openstackgerritMathias proposed openstack/kolla: Add external ceph support for cinder and glance  https://review.openstack.org/31020813:45
*** banix has quit IRC13:45
*** daneyon_ has joined #openstack-kolla13:46
pbourkesean-k-mooney: you based in ireland??13:46
inc0a yeah guys, you can drink with each other13:46
openstackgerritMathias proposed openstack/kolla: Add external ceph support for cinder and glance  https://review.openstack.org/31020813:46
inc0my fault, I should introduce you to each other;)13:46
inc0Mathias, are you here?13:47
*** haplo37 has joined #openstack-kolla13:48
sbezverkwirehead_ bootstrapping keystone reports completed state 2/4 and log for last task shows a traceback, see here: http://paste.openstack.org/show/523777/13:50
*** Mr_Broken has joined #openstack-kolla13:50
*** daneyon_ has quit IRC13:50
sbezverkwirehead_ have you seen it?13:52
*** ayoung has joined #openstack-kolla13:52
*** rhallisey has quit IRC13:53
*** JeroenBo has quit IRC13:54
*** rhallisey has joined #openstack-kolla13:54
sdakei think pbourke wins the drinking contest13:54
sdakei was three sheets to the wind and pbourke had 3 or 4 beers on me13:55
pbourkelol13:55
sdakend he looked sober as normal13:55
pbourkethat's funny though didn't know there was a fellow irish man here13:55
pbourkemay have both been at openstack days ireland and didnt know13:55
sdakei wishthere was an openstack meetup in pheonix13:55
*** Mr_Broken has quit IRC13:57
openstackgerritPaul Bourke proposed openstack/kolla: Add an alternative disk provisioning method for Ceph  https://review.openstack.org/33504613:57
sdakenever having been to a meetup i wouldn't really know where to begin organizing one :)13:57
openstackgerritPaul Bourke proposed openstack/kolla: Add an alternative disk provisioning method for Ceph  https://review.openstack.org/33497013:58
sdakenew vim command mewald taught me to wrap lines - gq13:58
pbourkesdake: im assuming theres support there from the TC for organising them13:58
sdakepbourke ya i'm overloaded13:59
sdakemaybe next cycle13:59
pbourkeafaik Intel were the main organisers of the Irelan one13:59
pbourke*Ireland13:59
sdakehow many people were at the meetup?13:59
sdakeis it monthly or every 6 months or what?14:00
pbourkeI think these are annual14:00
pbourkeyou could easily do something smaller and less official14:00
sean-k-mooneypbourke: hi back from tea and yes im from ireland14:00
pbourkesean-k-mooney: hi :)14:01
pbourkesean-k-mooney: were you at the openstack day ireland?14:01
pbourkesdake: https://www.openstack.org/community/events/openstackdays14:01
sean-k-mooneypbourke: so the rest of my team were but no i did not attend14:02
pbourkeah14:02
sean-k-mooneypbourke: one of our team members was leaving so i wanted to send time with him before he did14:02
*** zhurong has joined #openstack-kolla14:04
*** Mr_Broken has joined #openstack-kolla14:06
sean-k-mooneymy manager was one of the people organising the openstack ireland day this year. next year it will likely still have 1 confernece track but also have some breakout sessions14:07
*** b_bezak has joined #openstack-kolla14:08
sean-k-mooneyby the way the kolla midcycle will be soon correct https://wiki.openstack.org/wiki/Sprints/KollaNewtonSprint14:09
Jeffrey4lpbourke, just curiosity that, the docker require kernel 3.10 minimum. is there really someone to install docker/kolla in RHEL 4?14:09
sean-k-mooneyRHEL6 is below 3.1014:10
*** banix has joined #openstack-kolla14:10
sean-k-mooneyrhel 7 uses 3.10 with backports form 4.x14:10
*** b_bezak_ has quit IRC14:11
Jeffrey4lsean-k-mooney, if so. at least, it is bad idea to install docker/kolla in rhel 6, right?14:11
Jeffrey4lpbourke, ^14:11
pbourkeoracle VM has a backported kernel14:13
pbourkeI assume similar is possible in RHEL14:13
sean-k-mooneyhow was the webex for the last mid cycle out of interest? i wont be able to attend in person my might try to join remotely14:13
sean-k-mooneyJeffrey4l: am docker 1.10 should be ok14:13
Jeffrey4lsean-k-mooney, what?14:13
pbourkethese distros are making use of docker specifically to allow newer software on older user space libs14:13
sean-k-mooneyjeffery docker 1.11.x was the first to use feature that actully needed 3.1014:14
pbourkeim not saying kolla has to 100% abide by older distros but should at least accept changes that helps them14:14
Jeffrey4lhow the docker 1.10 ? kolla need docker >= 1.10 actually.14:14
sdakesean-k-mooney yup midcycle is 12/1314:15
sdakesean-k-mooney will you be making it14:15
Jeffrey4lOK. backport kernel should work.14:15
Jeffrey4lanother question, what happen to gpt?14:15
sean-k-mooneysdake: ill try and join remotely14:15
Jeffrey4lwhat not use gpt without udev?( if udev is bust)14:16
pbourkeJeffrey4l: how do you parse the disk labels in that case14:16
pbourkeother than screen scraping parted or the like which requires a full rewrite of find_disks14:16
sean-k-mooneyJeffrey4l: so ya to summeries docker offically only support kernel 3.10+ but before the 1.11 release of docker docker did not use any feaures that prevented older kernels from working14:16
Jeffrey4lpbourke, the label is write in the disk. you can read the from the raw disk.14:17
Jeffrey4lpbourke, here is a example https://github.com/ceph/simplegpt/blob/master/simplegpt.py14:17
Jeffrey4lsean-k-mooney, ok. that's fine14:18
pbourkeJeffrey4l: I came across that code, its pretty low level though. Do we really want to be dealing with changes in GPT partition structures14:20
pbourkeJeffrey4l: I wouldnt be opposed to a rewrite of find_disks to use the above code but what I have works14:21
sean-k-mooneyit would be nice if gpt labels were not reuired14:21
Jeffrey4lpbourke, is there any tools can read the gpt table? like parse the gparted output. it should be more stable.14:22
sean-k-mooneynot sure if there is a cleaner way to do it generically14:22
pbourkesean-k-mooney: my patch does that14:23
mewaldsdake: do you have any link that explains the steps we took with Git and Gerrit? I would like to have a reference that helps me with this a bit :)14:23
sean-k-mooneypbourke: oh cool do you have a link14:23
sean-k-mooneyscrooling back to find it14:23
pbourkesean-k-mooney: yes Id appreciate a look https://review.openstack.org/#/c/334970/14:24
patchbotpbourke: patch 334970 - kolla - Add an alternative disk provisioning method for Ceph14:24
Jeffrey4lpbourke, msdos disk has some issue i think. how to handle that: how to handle the external journal?14:25
sean-k-mooneymsdos partition can have lable on primary partitions14:26
Jeffrey4lright.14:26
sean-k-mooneyyou can have up to 4 primary partition so that should be ok14:26
Jeffrey4lthat's the point.14:26
sean-k-mooneythe main issue with msdos partition tabels is i think the lables we use currently are too long14:26
sdakemewald http://docs.openstack.org/infra/manual/developers.html14:27
sdakeso we can't use msdos partitions14:27
sdakejust a limitation of the technology14:27
sdakethe world has moved to eufi and GPT anyway14:27
sean-k-mooneyi think the bootstrap lable works14:27
*** rhallisey has quit IRC14:27
sdakeUEFI that is14:28
sean-k-mooneybut if you want to use a cache partion with an external journal the current lable we use is too long for the msdos limit14:28
*** rhallisey has joined #openstack-kolla14:28
sdakeJeffrey4l i fixed the gates to build and test images correctly :)14:28
*** g3ek has quit IRC14:28
pbourkeAre there concerns with this proposed approach. If not, its solving a specific problem - there's many ways to skin a cat unfortunately I dont have time to write them14:29
Jeffrey4lsdake, cool14:29
sdakepbourke which approach14:29
sean-k-mooneysdake: ubuntu,centos and rhel still default to creating msdos partions on install14:29
*** haplo37 has quit IRC14:29
pbourkesdake: just discussing https://review.openstack.org/#/c/334970/14:29
patchbotpbourke: patch 334970 - kolla - Add an alternative disk provisioning method for Ceph14:29
sdakepbourke first question - what if different hardware has different disk identifiers14:31
*** Mr_Broken has quit IRC14:31
pbourkesdake: can you give an example?14:31
*** haplo37 has joined #openstack-kolla14:31
sdakewhich i think we cn expect to be the case14:31
sdakein my work lab14:31
sdakeI have 3 storage nodes, they have 20 spinning disks14:31
sdakemy compute nodes have 3-5 spinning disks14:31
*** g3ek has joined #openstack-kolla14:31
*** diogogmt has quit IRC14:32
sean-k-mooneywell the main issue with using /dev/sdb,/dev/sdc is that i can change between reboot correct?14:32
sdakesean-k-mooney its not supposed to change14:32
sean-k-mooneye.g. if you add or remove another disk14:32
sdakea better way would beto use disk labels or wwns tho14:32
pbourkewell it mounts them using UUIDs14:32
*** diogogmt has joined #openstack-kolla14:32
sean-k-mooneyok cool so udev maps the uuid to sdX14:32
sdakesean-k-mooney god i hope so by this point :)14:33
sdakepbourke it looks pretty good - i just worry about storage nodes with different storage setups14:33
sdakeit would be  nice if we could specify per node14:34
sdakeis that possible?14:34
pbourkesdake: yes, you can set it using ansible host vars / group vars14:34
sdakeper node?14:34
Jeffrey4lthe bad point is: you restrict to use /dev/sda1 as data, and /dev/sda2 as journal. and never use /dev/sda3+14:34
Jeffrey4lpbourke,14:34
pbourkeJeffrey4l: that's also the case with find_disks though?14:35
Jeffrey4lthe more use case is: use ssd as journal and sata as data disk.14:35
Jeffrey4lpbourke, no. find_disks is more power than that.14:35
sdakessds should also be used as cache tiers14:35
Jeffrey4lfind_disk can use external journal.14:35
sdakewell i'm glad someone is making ceph more usable in kolla14:35
*** Mr_Broken has joined #openstack-kolla14:35
sdakepbourke keep it up ;)14:36
Jeffrey4lin my env. I split one ssd disk into 4 partition, one partition for on sas disk.14:36
sdakeJeffrey4l the last time i used ceph it would only work on the entire disk14:36
sdakehas that changed?14:36
Jeffrey4lsdake, as far as i know, cache tiers is not good.14:36
Jeffrey4lsdake, no. it support this all the time.14:36
*** kproskurin has quit IRC14:37
sean-k-mooneyJeffrey4l: i do someting similar on my home ceph server. 6 partitions on one ssd. (os + 4 journals+ cache osd partition) + 4 spinning disks for data14:37
Jeffrey4lsean-k-mooney, yes. this is more common use case.14:37
sdakeJeffrey4l yes but does kolla operate in that way?14:38
sdakeJeffrey4l last i tried, you had to format the entire disk as a label14:38
Jeffrey4lsdake, there is a more hack way.14:38
sdakei.e. one disk = one partition14:38
sean-k-mooneyunfortunetly kolla didnt support that when i set it up so im looking forward to mewald's external ceph support14:38
Jeffrey4lsean-k-mooney, kolla support that, indeed.14:39
sean-k-mooneysdake: kolla now has partition support14:39
Jeffrey4llet me show u14:39
sdakesweet14:39
sdakesam said it was impossible ;)14:39
sdakecan someone document it?14:39
sean-k-mooneysdake: its documented in his blog but not in our ceph guide14:39
sdakeor tell me how and i'll document it14:39
sean-k-mooneysdake: we need to port it14:40
Jeffrey4lsdake, it is possible. but it lack of some doc.14:40
Jeffrey4lsam wrote a article about this.14:40
sdakesean-k-mooney did he have code changes14:40
sean-k-mooneysdake: no code chages its merged in master/mitaka14:40
sean-k-mooneysdake: https://yaple.net/2016/03/20/deploying-openstack-mitaka-with-kolla-docker-and-ansible/14:40
sean-k-mooneyis sams site down for others or is it just slow loading?14:41
sdakedown14:41
sean-k-mooneyoh it just loaded for me14:41
mewaldtakes ages14:41
sdakeoh slow :)14:42
Jeffrey4li opened it.14:42
mewaldbrowser is still "resolving host" :D14:42
Jeffrey4lit is this article.14:42
pbourketry the google cache14:42
sean-k-mooneyso ya you bassicaly add _J for the journal to the lables14:42
Jeffrey4lwrite.14:42
sean-k-mooney KOLLA_CEPH_OSD_BOOTSTRAP_1 first data partion  KOLLA_CEPH_OSD_BOOTSTRAP_1_J for journal partion14:42
sean-k-mooneyit also should work for cache partions14:43
Jeffrey4lcorrect. this require make the labels manually as expected.14:43
Jeffrey4lsean-k-mooney, no. in my env, i never used the cache partition.14:43
sdakeall that works in master?14:43
Jeffrey4lsdake, all the branches i think14:43
sdakesweet14:43
sean-k-mooneysdake: it should but i dont think we test it so its probably broken14:43
Jeffrey4l( we are using mitaka deploy liberty) so all14:43
Jeffrey4lsean-k-mooney, I use this. really.14:44
Jeffrey4lsean-k-mooney, in several env. it works very well.14:44
sean-k-mooneyJeffrey4l: well i was more commenting on the fact that if its not tested in openstack its usually broken14:44
sean-k-mooneywe should have a gate job that tests it14:45
pbourkeJeffrey4l: let me review the journal stuff and get back to you14:45
Jeffrey4lsean-k-mooney, yes. we should. but now, kolla even has not ceph gate.14:45
*** zhiwei has quit IRC14:45
sean-k-mooneyJeffrey4l: fair point :)14:45
sean-k-mooneyit would be nice to have the ablitiy to use files for osd in kolla to make gate testing of ceph simpler but we should be able to use loop back devices to test this in the gate14:46
sdakeloopback is what we use14:46
Jeffrey4lsdake, pbourke sean-k-mooney see my env to use the external journal. http://paste.openstack.org/show/523782/14:47
Jeffrey4li even split the ssd into multi data partition to improve the performance.14:47
sean-k-mooney20GB journals are quite big14:48
Jeffrey4lsean-k-mooney, what value are u using..14:48
sean-k-mooneythats what i was planning on using too by from talking to sam he said they could be smaller14:48
sdakepbourke might I ask what problem your tyring to solve - is that gpt isn't in use by distros?14:48
Jeffrey4lsdake, i think the root cause it udev rather than the gpt..14:49
pbourkesdake: the problem is udev is too old on some distros to work with kolla14:49
pbourkesdake: RHEL 6 for example14:49
sean-k-mooneyceph deply uses 5GB i think on a 2TB drive if you let it deploy ceph14:49
sdakewhich distros in particular14:49
*** thumpba has joined #openstack-kolla14:49
Jeffrey4lsean-k-mooney, hmm.. i am head that 5G is more small for that.14:49
sdakekolla is not meant for rhel6 fwiw :)14:49
sdakecan we have a quck improptu discussion about 16.0414:50
pbourkesdake: yes but there's no reason we can't do these distros14:50
sdakei really would like to see 16.04 implemented in newton14:50
sdakesince 14.04 is byebye14:50
sean-k-mooneyJeffrey4l: as far as i know there is no performance hit to larger journals so using 20GB journals should be fine14:50
pbourkesdake: RHEL 6 is LTS till 202014:50
sdakeanyone now the status of thi work?14:50
sdakepbourke yes I know, but rdo for example I think doesnt support rhel614:50
Jeffrey4lsdake, pbourke is talking about the host machine . not the docker image.14:50
sdakei understand but ideally they should match14:51
pbourkei diagree they should match14:51
sean-k-mooneysdake: well 14.04 is not eol for 2.5 years but it wont support openstack byond mitaka.14:51
pbourke*disagree14:51
sean-k-mooneysdake: i would like to see 16.04 though14:51
sdakewell its eol for openstack then :)14:51
sdakepbourke cn you expand14:52
*** tyrola has quit IRC14:52
sean-k-mooneyfor what its worth 14.04 docker images on a 16.04 host seams to work fine as it is but we have to update the quickstart somewhat14:52
*** tyrola has joined #openstack-kolla14:52
pbourkesdake: big selling point of containers is the whole "write once run anywhere" thing. Its not true for every case but there's no need to enforce the container base to match the host14:53
sean-k-mooneybasically the same systemd changes that you make on centos are need on the host for Ubuntu 16.04 hosts14:53
sean-k-mooneysome filepaths are different though14:53
sdakepbourke write once run anywhere is a fallcy14:53
sdakeeven docker inc has admitted as such14:53
sdakebut ya we dont have to enforce it14:53
pbourkesdake: i know there's issues with say centos containers on ubuntu14:54
sean-k-mooneyso you can have a different host os to container os but you need to carful on the host to configure selinux/apparmor so that it works correctly14:54
sdakeyou can run some ubuntu containers on centos as well :)14:54
pbourkebut saying you have to have centos 7 containers to run on centos 7 is bad14:54
Jeffrey4li think we need set up some rule. like does the kolla need support rhel6?14:54
Jeffrey4lif so, we need have gate to test this.14:54
sdakerhel6 isnt' in the upstream gates14:54
sdakeso we cant gate on rhel614:54
Jeffrey4lso i can say, we do not support rhel6 .14:55
pbourkei dont suggest we officially "support" these OSes14:55
Jeffrey4lbecause it is never tested.14:55
sdakei'm not sure how long 14.04 will be in the usptream gates either14:55
pbourkebut should not reject patches that help their case14:55
*** zhurong has quit IRC14:55
sdakepbourke ++14:55
sdakepbourke that said people do come looking to us for help14:56
sdakeand I don't have any centos6 setup14:56
sean-k-mooneyis kolla only intended to support the latest lts of any release. so rhel/centos 7 and ubuntu 16.04 would be what i gues we would support14:56
sean-k-mooneyfor what is worth i think infra where planning on swapping the upstream gate to 16.04 by mid july14:57
sean-k-mooneyi know they had the devstack gate job working at the summit14:57
Jeffrey4lsean-k-mooney, at least, we test the kolla on rhel/cento 7 ubuntu 14.04 now. we have no other os gate to ensure that.14:57
sean-k-mooneysdake: you were refering to the container os though right. not host os14:58
Jeffrey4lbut, at the udev bug issue point, add such fix make the kolla more complicated. :(14:59
sdakeactually rhel isn't in the gates either14:59
sean-k-mooneyso for newton you would like to see the ubuntu contaier base os move to ubuntu 16.0414:59
sdakehere is my take14:59
sdake:14:59
sdakevendors need some things like rhel6 implementation and 14.04 implemeentaitons, and we shouldn't try to stop that14:59
sdakethat said, we shoudl have a big fat warning in our documentation stating "these are the distros kolla are tested with, these are the ones known tow ork with - the ones that work with your on your own"15:00
sdake(in better english:)15:00
britthouserReading the scrollback regarding disk order re-arranging.15:00
britthouserWe've seen storage controllers re-arrange the disk orders15:00
sean-k-mooneysince cannonical will only be supporting Newton on 16.04/16.10 i think we do need to transition form 14.04 this cycle though.15:01
britthouserbut /dev/disk/by-path has always been consistent for us15:01
*** Mr_Broken has quit IRC15:01
pbourkei guess the issue is I've seen udev fail even on centos 715:02
pbourkesteve hindle also has a bug open related to this15:03
pbourkeso its not 100% related to older distros15:03
Jeffrey4lpbourke, i am using centos 7 + kolla + ceph. never hit any issue.15:03
pbourkei get that15:04
pbourkeJeffrey4l: are you running bare metal?15:04
Jeffrey4lpbourke, right. a prod env.15:05
pbourkesomeone mentioned the issue can arise in vms15:05
pbourke*VMs15:05
inc0Jeffrey4l, there can be lots of variations15:05
inc0fact that it never happened to you may mean that you're just lucky15:05
Jeffrey4lpbourke, vms.. really?15:05
pbourkeJeffrey4l: yes15:06
sdakebritthouser that is the wwn addressing right?15:06
Jeffrey4lso it must related to the kvm. because it can not provide a better *physical* env.15:06
britthouserno, sdake, /dev/disk/by-id is wwn address which is consistent also15:06
britthouserbut /dev/disk/by-path lets gives you consistent name which you can also narrow down by controller15:07
sdakeok, well i dont see objection to adding things to make kolla work with rhel615:07
britthousere.g. on UCS you can see that drives on this controller are flexflash, drives on another controller are the boot SSD, and drives on a different controller are attached to the RAID card.15:07
sdake(I mean technical objection)15:07
sdakeother then it complicates things15:08
*** Mr_Broken has joined #openstack-kolla15:08
Jeffrey4lbritthouser, kolla need label to record which disk the osd container need.15:08
*** sacharya has joined #openstack-kolla15:09
pbourkebritthouser: that's good info to know15:09
pbourkebritthouser: sounds like will need to use something other than /dev/xxx15:10
britthouserYeah I think a /dev/disk/by-(whatever) would be more consistent, but would require some user education15:10
*** sacharya_ has joined #openstack-kolla15:12
*** Mr_Broken has quit IRC15:12
britthouserlol...I see you already addd that info to the review pbourke. =)15:12
pbourkebritthouser: made sure to get it in there as is important!15:12
pbourkelast thing we want is to wipe the wrong disk15:12
britthouserI'm just slow this morning15:12
*** sacharya has quit IRC15:14
Jeffrey4lsean-k-mooney, here is how to calculate the journal size: journal size = (2 * (expected bandwidth throughput * filestore max sync interval))15:16
Jeffrey4lFor a well optimised 10GB network: 12,5GB = (2 * (1250 * 5))15:16
*** g3ek has quit IRC15:18
*** Mr_Broken has joined #openstack-kolla15:18
*** b_bezak has quit IRC15:19
*** haplo37 has quit IRC15:19
*** b_bezak has joined #openstack-kolla15:19
*** haplo37 has joined #openstack-kolla15:19
*** g3ek has joined #openstack-kolla15:20
*** b_bezak has quit IRC15:24
*** Mr_Broken has quit IRC15:27
*** Mr_Broken has joined #openstack-kolla15:30
*** belmoreira has quit IRC15:31
sdakecan someone that is a ceph rcoket scientist translate sam's blog post into the ceph user guide?15:33
*** Mr_Broken has quit IRC15:34
pbourkeimagine if all blog posts were docs15:37
*** y-sumida has joined #openstack-kolla15:37
pbourkei'll try if I get time15:38
pbourkeneed to see if my patch will work with what he proposes first15:38
sean-k-mooneyJeffrey4l: yes that sounds like what sam told me at the summit15:40
Jeffrey4li have 20G network. so 20G journal is find. sean-k-mooney :D15:40
*** dmk0202 has quit IRC15:42
sean-k-mooneyi still only have 1G at home and no ceph at work :(  yet :)15:43
*** pbourke has quit IRC15:43
*** pbourke has joined #openstack-kolla15:43
*** jmccarthy has joined #openstack-kolla15:44
*** rhallisey has quit IRC15:45
*** rhallisey has joined #openstack-kolla15:46
*** jmccarthy has left #openstack-kolla15:47
openstackgerritJoshua Harlow proposed openstack/kolla: Consistently use sys.exit  https://review.openstack.org/33469015:49
openstackgerritJoshua Harlow proposed openstack/kolla: Consistently use sys.exit  https://review.openstack.org/33469015:50
*** y-sumida has quit IRC15:53
*** v1k0d3n has quit IRC15:53
*** sdake_ has joined #openstack-kolla15:54
*** Jeffrey4l has quit IRC15:55
*** sdake has quit IRC15:57
*** haplo37 has quit IRC16:04
*** g3ek has quit IRC16:04
*** haplo37 has joined #openstack-kolla16:06
*** g3ek has joined #openstack-kolla16:07
*** rmart04 has quit IRC16:09
*** daneyon_ has joined #openstack-kolla16:28
*** vhosakot has joined #openstack-kolla16:32
*** ravig has joined #openstack-kolla16:32
*** daneyon_ has quit IRC16:33
*** salv-orl_ has quit IRC16:38
*** neilus has quit IRC16:40
*** dwalsh has joined #openstack-kolla16:47
*** g3ek has quit IRC16:52
*** haplo37 has quit IRC16:52
*** ravig has quit IRC16:53
*** ravig has joined #openstack-kolla16:54
*** haplo37 has joined #openstack-kolla16:55
*** g3ek has joined #openstack-kolla16:55
*** zhurong has joined #openstack-kolla16:56
*** Serlex has quit IRC16:56
*** matrohon has quit IRC17:04
*** dwalsh has quit IRC17:05
*** mbound has quit IRC17:17
*** salv-orlando has joined #openstack-kolla17:18
*** mewald has quit IRC17:18
*** ravig has quit IRC17:20
*** aNupoisc has joined #openstack-kolla17:24
*** vhosakot has quit IRC17:30
*** vhosakot has joined #openstack-kolla17:36
*** mark-casey has joined #openstack-kolla17:39
*** vhosakot has quit IRC17:39
*** haplo37 has quit IRC17:41
*** haplo37 has joined #openstack-kolla17:46
*** vhosakot has joined #openstack-kolla17:47
*** ravig has joined #openstack-kolla17:51
*** ravig has quit IRC17:55
*** fragatin_ has joined #openstack-kolla17:55
*** fragatina has quit IRC17:55
*** diogogmt has quit IRC17:58
*** banix has quit IRC17:59
*** diogogmt has joined #openstack-kolla18:00
*** zhurong has quit IRC18:01
*** ravig has joined #openstack-kolla18:01
*** ravig has quit IRC18:03
*** banix has joined #openstack-kolla18:03
*** ravig has joined #openstack-kolla18:03
*** huikang_ has joined #openstack-kolla18:06
*** salv-orlando has quit IRC18:13
*** diogogmt has quit IRC18:16
*** mbound has joined #openstack-kolla18:18
*** mewald has joined #openstack-kolla18:19
*** mewald has left #openstack-kolla18:22
*** mewald has joined #openstack-kolla18:23
*** mark-casey has quit IRC18:23
*** mbound has quit IRC18:24
*** mark-casey has joined #openstack-kolla18:24
mark-caseysdake_: ping18:28
*** diogogmt has joined #openstack-kolla18:28
sdake_mark-casey shoot18:28
mark-caseyWas reading eavesdrop yesterday and I wrote a blurb on what xtrabackup is doing for us; if that is still an issue. Pre-wrote it in an attempt at being succinct but not sure it worked. sorry for minor wall of text18:30
mark-caseyWhen a galera node fails it needs a state transfer from a healthy node **when it is restarting**. This is essentially just making a backup on a healthy node and sending it to the node that has failed and is recovering. The TL;DR is that without xtrabackup the "donor node" will become read-only for the duration of the state transfer. If two out of three nodes fail and then restart, the entire db will be unavailable until one of th18:30
mark-caseythe issue is in getting a consistent backup from a node while the cluster is running, as you obviously do not want to restore the failed node with tables from two (or more) different points in time. Xtrabackup uses InnoDB crash logs to get a consistent state w/out blocking writes, while switching to rsync for SST will block writes18:33
openstackgerritSteven Dake proposed openstack/kolla: Liberty is now on 1.1.2 prerelease  https://review.openstack.org/33463718:33
openstackgerritMerged openstack/kolla-kubernetes: Glance works without host network  https://review.openstack.org/33302218:33
sdake_mark-casey so if  Iprocessed that correctly we could offer both modes18:34
sdake_im glad to finally hear what why xtradbbackup is needed18:34
inc0sdake_, mark-casey that makes sense18:35
inc0but I'd say we need xtrabackup18:35
mark-casey:D yes - could offer both. you'd need the mariadb config template to switch the value of wsrep_sst_method and possibly wsrep_sst_donor_rejects_queries, and then tell the script that informs haproxy who is healthy to exclude donors18:35
inc0as having read-only master node effectively causes downtime18:35
inc0and having both modes may be hard to maintain18:36
mark-caseyyou only go read-only if you get to a single healthy node left, but are still -1 on HA magic for all failure scenarios18:36
sdake_mark-casey i worked on ha tech for 10+ years18:37
sdake_more magic is not necessarily a good thing :)18:37
inc0well, if healthy node goes read only it will have to sync itself later on too18:37
inc0sdake_, is it critical for you to get rid of it?18:38
*** ravig has quit IRC18:38
sdake_nah i just dont like deps18:38
mark-caseyinc0: IIRC background replication continues, but I personally still favor keeping read/write on donors18:38
inc0sdake_, I'd keep it in then. For the sake of maintainability18:39
inc0less logic == good18:39
sdake_ya i can deal with that on my own if  its as simple as changing a few flags18:39
inc0not really18:39
inc0it will change way DB servers will behave18:40
sdake_mark-casey can you send me an email to stdake@cisco.com with what you suggest i change to get it operational18:40
sdake_inc0 i mean for my internal stuff18:40
inc0sdake_, oh, ok18:40
mark-caseysdake_: sure18:40
inc0mark-casey, cc michaljastrzebski@intel.com please18:40
inc0michal.jastrzebski@intel.com18:41
mark-caseyinc0: will do18:41
inc0I should know my email address...18:41
inc0thanks18:41
*** athomas has quit IRC18:41
*** dwalsh has joined #openstack-kolla18:46
*** banix has quit IRC18:49
*** banix has joined #openstack-kolla18:55
*** huikang_ has quit IRC18:59
openstackgerritSteven Dake proposed openstack/kolla: Fix upgrades fail to upgrade  https://review.openstack.org/33464419:00
*** bootsha has joined #openstack-kolla19:01
*** salv-orlando has joined #openstack-kolla19:01
*** huikang_ has joined #openstack-kolla19:01
*** sdake has joined #openstack-kolla19:02
*** sdake_ has quit IRC19:03
*** huikang_ has quit IRC19:06
*** daneyon_ has joined #openstack-kolla19:10
*** daneyon_ has quit IRC19:15
*** sdake_ has joined #openstack-kolla19:16
*** tyrola_ has joined #openstack-kolla19:17
*** athomas has joined #openstack-kolla19:19
*** sdake has quit IRC19:20
*** tyrola has quit IRC19:21
*** tyrola_ has quit IRC19:21
*** mummer has joined #openstack-kolla19:25
*** dwalsh has quit IRC19:25
sdake_anyone have any idea what this means: GATHERING FACTS ***************************************************************19:26
sdake_failed: [minime-02] => {"cmd": "/usr/local/sbin -a", "failed": true, "rc": 13}19:26
sdake_msg: [Errno 13] Permission denied19:26
sean-k-mooneyit looks like a variable whas not defined19:28
sean-k-mooneye.g. it should /usr/local/sbin/somthing -a19:28
sdake_stable/lbierty multinode eploy19:28
sdake_aio works19:29
sdake_only change is to multinode19:29
*** aNupoisc has quit IRC19:30
sean-k-mooneyi assume that minime-02 is in you /etc/hosts or dns and the ansible user can ssh to it?19:30
*** ravig has joined #openstack-kolla19:30
sdake_yup and resolving via resolver library properly19:30
sdake_1 and 3 work19:31
sean-k-mooneythats odd :)19:31
sean-k-mooneysorry could not resist19:31
sdake_can ssh in as root to minime-0219:31
sdake_and the other nodes19:31
sdake_it is odd ;)19:31
sdake_i just caught that19:31
sdake_slow today :)19:31
*** ravig has quit IRC19:32
sdake_all machines have selinux disabled and sameetc hosts file19:32
*** ravig has joined #openstack-kolla19:33
sean-k-mooneyi have not seen that before. does the /etc/kolla directory exist. im guessing its failing right at the start in the common role?19:33
sdake_first task19:34
sdake_aio works though19:34
sdake_which is wier19:34
sdake_d19:34
sdake_do you know which task does that gathering facts thing19:34
sean-k-mooneyaio works in minime-0219:34
sdake_i have not  tried that19:34
sean-k-mooneyor on the deployment host19:34
sdake_on minime3 and 1 aio works19:34
sean-k-mooneymaybe run the precheck on minime2 if you havent19:34
*** ayoung has quit IRC19:35
mark-caseysdake_: pretty sure it's setup module - it is implicit if gather_facts: false is not set on a play19:35
mark-caseyhttp://docs.ansible.com/ansible/setup_module.html19:35
*** fragatin_ has quit IRC19:36
sdake_what binary is it trying to run19:36
sdake_how would I find that out?19:36
*** bootsha has quit IRC19:37
sdake_nm i'm a dummy19:37
sdake_i was trying t o run as non-root19:37
sdake_which i knkow works19:37
sdake_but requires some magic incantations which are undocumented19:37
sdake_inc0 re that upgrade problem19:38
inc0yeah?19:38
sdake_inc0 you said you didn't want the error to silenty as - is there a way to give a warning instead19:38
sdake_because as is what we have is busted19:38
inc0that's actually a good question19:38
inc0let me get back to you, I'll need to dig into ansible19:38
sdake_TASK: [mariadb | Copying over galera.cnf] *************************************19:39
sdake_fatal: [minime-03] => {'msg': "One or more undefined variables: 'dict object' has no attribute u'ansible_p1p1'", 'failed': True}19:39
sdake_fatal: [minime-03] => {'msg': 'One or more items failed.', 'failed': True, 'changed': False, 'results': [{'msg': "One or more undefined variables: 'dict object' has no attribute u'ansible_p1p1'", 'failed': True}]}19:39
sdake_any ieas?19:39
sdake_guess i should have pasted that19:39
sdake_sorry19:39
sdake_p1p1 = network interface19:40
sean-k-mooneysdake_: yes your network interface in your global.yml is wronge19:40
sean-k-mooneysdake_: or on that node its named something else19:40
sdake_unlikely19:41
sdake_all machines have p1p1 as their primary network 10gig interface19:41
*** rmart04 has joined #openstack-kolla19:41
sdake_all are up and running19:41
*** ravig has quit IRC19:42
sean-k-mooneyand i take it network_interface: is p1p1.19:42
*** ravig has joined #openstack-kolla19:42
sdake_network_interface: "p1p1"19:42
sean-k-mooneystrange i have only seen that fail if i set it wrong19:42
sdake_ya this is stable liberty19:42
sdake_i haven't seen it fail in a long long time like this19:43
sdake_docker engine configured properly on all nodes19:43
sean-k-mooneydose that interface have an ip?19:44
sean-k-mooneyon minime0319:44
sdake_yes as it should on all 3 nnodes19:44
*** rmart04 has quit IRC19:44
sdake_all 3 nodes have the correct ip adddress19:44
sean-k-mooneyok if i remember correctly that is what that variabel used as a key to retrive19:47
sean-k-mooneyhttps://github.com/openstack/kolla/blob/stable/liberty/ansible/roles/mariadb/templates/galera.cnf.j2#L319:47
*** mliima has quit IRC19:47
*** v1k0d3n has joined #openstack-kolla19:47
*** fragatina has joined #openstack-kolla19:48
sdake_nah it definately doesnt like soething about my minime-02 node19:49
sdake_There was a problem importing one of the Python modules19:50
sdake_required to run yum. The error leading to this problem was:19:50
sdake_   No module named yum19:50
sdake_groan19:50
sean-k-mooneywow where did you find that error?19:51
sdake_any python operation on minime-02...19:51
sdake_tared /usr/lib/ptyhon still doesn't find the yum module19:52
* sdake_ hates reinstalling19:52
sean-k-mooneyare you on centos/rhel?19:53
sdake_yup19:53
sdake_yum is in /usr/lib/python2.7/site-packages19:53
*** zhiwei has joined #openstack-kolla19:54
sean-k-mooneyok fedora 23 has a nasty packaging change where they made python depend on pip so devstack would remove python when it unistalled pip.19:54
sean-k-mooneyi think on centos yum depends on yum-fastmirros-plugin also19:55
sdake_i don ot run devstack on these machines19:55
sean-k-mooneyoh i know my point was i wonder i you unitsalled something that actidentally removed the python module.19:55
sdake_reinstalling yum does not fix it19:57
sdake_python-2.7.5-34.el7.x86_6419:57
*** gfidente has quit IRC19:57
sdake_python interepeter runs19:58
*** zhiwei has quit IRC19:58
*** haplo37 has quit IRC20:01
sean-k-mooneyif you do which python is it /usr/bin/python20:01
sdake_[sdake@minime-02 bin]$ pip show20:01
sdake_Traceback (most recent call last):20:01
sdake_  File "/usr/bin/pip", line 5, in <module>20:01
sdake_    from pkg_resources import load_entry_point20:01
sdake_ImportError: No module named pkg_resources20:01
sean-k-mooneygoogleing on line seams to point to a broke python instaces20:02
*** g3ek has quit IRC20:02
*** huikang_ has joined #openstack-kolla20:02
*** athomas has quit IRC20:02
*** g3ek has joined #openstack-kolla20:03
*** haplo37 has joined #openstack-kolla20:04
*** v1k0d3n has quit IRC20:06
*** huikang_ has quit IRC20:07
sdake_fixed it20:08
sdake_it is lucky the namespace change didn't break upgrades20:11
sdake_completely random20:12
sean-k-mooneyso what was the fix in the end? resinatll python? or someting else20:12
sdake_tarred /usr/lib and /usr/lib/64/python*20:14
sdake_rather /usr/lib/python*20:14
sdake_and untarred on busted machine20:14
sdake_TASK: [mariadb | Waiting for MariaDB service to be ready] *********************20:14
sdake_failed: [minime-02] => {"elapsed": 60, "failed": true}20:14
sdake_msg: Timeout when waiting for search string MariaDB in 192.168.1.102:330620:14
* sdake_ growls20:14
*** ppowell_ has quit IRC20:16
sdake_connection refused on 3306 yet mariadb running20:16
*** bootsha has joined #openstack-kolla20:17
sean-k-mooneyare you doing a fresh install or upgrade by the way20:19
sdake_fresh install atm20:19
sdake_but the machines have been installed before20:19
sdake_http://paste.fedoraproject.org/386044/45207146/ -> http://paste.fedoraproject.org/386044/4520714620:20
sdake_mariadb output20:20
sdake_data looks correct to me20:20
sdake_yet the fact remains i cannot connect to 10320:20
sdake_330620:20
sdake_flushing ip tables doesn't help20:22
mark-caseysdake_: WSREP: no nodes coming from prim view, prim not possible20:27
sdake_what does that mean20:28
mark-caseylikely all started nodes have that in their log. there is no quorum/all nodes think they are part of a partition20:28
sdake_i took all the log messages out o corosync, got tired of people asking me about them :)20:28
mark-caseythis is the case you'd run lights out for, but if I'm understanding you there shouldn't have been a need to20:29
mark-casey:D20:29
sdake_well whatever the case is,  a cleanup fixed it20:29
sdake_cleanup in mster pretty good20:29
sdake_kind of hacky implementation20:29
*** ayoung has joined #openstack-kolla20:29
sdake_i think we need a way to shut off cleanup20:31
openstackgerritDavid Wang proposed openstack/kolla-kubernetes: POC (do no merge): Kolla-Kubernetes multi-node persistence pattern  https://review.openstack.org/33521520:32
*** v1k0d3n has joined #openstack-kolla20:33
*** jtriley_ has joined #openstack-kolla20:35
*** v1k0d3n has quit IRC20:37
*** huikang_ has joined #openstack-kolla20:38
*** jtriley has quit IRC20:38
*** jtriley_ has quit IRC20:39
*** haplo37 has quit IRC20:44
*** thumpba has quit IRC20:45
*** aernhart has joined #openstack-kolla20:46
*** g3ek has quit IRC20:48
*** g3ek has joined #openstack-kolla20:50
*** aernhart has quit IRC20:52
*** haplo37 has joined #openstack-kolla20:54
*** fragatina has quit IRC20:54
*** fragatina has joined #openstack-kolla20:54
*** aNupoisc has joined #openstack-kolla20:55
*** ravig has quit IRC20:56
sdake_7 minutes to deploy 3 node ha cluster20:56
sdake_full control plane20:56
sdake_bet computeis probably 20 seconds or a less a node20:56
inc0little over 6 minutes for 10 node + ceph20:58
inc0you're slow sdake20:58
sdake_ssh did eem to block often20:58
*** huikang_ has quit IRC20:58
sdake_anyway no sense optimzing that anymore20:59
sdake_our big chunk of deploy tie will be bare metal itself20:59
*** ravig has joined #openstack-kolla20:59
*** sdake has joined #openstack-kolla21:01
sean-k-mooneyhopefully ironic will be up to the task21:02
*** ravig has quit IRC21:03
sean-k-mooneysdake_: one think i noticed about the cleanup script on master is if you put your invetory file in /etc/kolla it will happily delete it21:04
*** sdake_ has quit IRC21:04
sdakeya i know wierd etc deletion21:04
sdakeit also deletes certificates21:04
sdakei think cleanup is a bad bad misnomer21:04
sdakethat is getting renamed21:05
sdakeit shoudl be kolla-ansible remove-permanently21:05
sean-k-mooneyid does not delet the images or the gobal.yml/kolla-build.conf which is also strange21:06
*** ravig has joined #openstack-kolla21:08
*** dmk0202 has joined #openstack-kolla21:11
*** ravig has quit IRC21:12
*** bootsha has quit IRC21:13
*** rhallisey has quit IRC21:13
*** harlowja has quit IRC21:19
*** stvnoyes1 has quit IRC21:22
*** v1k0d3n has joined #openstack-kolla21:27
*** inc0 has quit IRC21:27
*** williamcaban has quit IRC21:28
*** haplo37 has quit IRC21:31
*** v1k0d3n has quit IRC21:34
*** vhosakot has quit IRC21:35
*** vhosakot has joined #openstack-kolla21:36
*** g3ek has quit IRC21:36
*** g3ek has joined #openstack-kolla21:37
*** banix has quit IRC21:39
*** sdake has quit IRC21:39
*** stvnoyes has joined #openstack-kolla21:39
*** haplo37 has joined #openstack-kolla21:40
*** dmk0202 has quit IRC21:42
*** daneyon_ has joined #openstack-kolla21:53
*** daneyon_ has quit IRC21:58
*** huikang_ has joined #openstack-kolla21:59
*** huikang_ has quit IRC22:04
*** huikang has quit IRC22:06
*** zhiwei has joined #openstack-kolla22:14
*** aNupoisc has quit IRC22:15
*** harlowja has joined #openstack-kolla22:16
*** papacz has quit IRC22:18
*** zhiwei has quit IRC22:18
*** dwalsh has joined #openstack-kolla22:21
*** harlowja has quit IRC22:23
*** g3ek has quit IRC22:24
*** haplo37 has quit IRC22:24
*** g3ek has joined #openstack-kolla22:29
*** mewald has quit IRC22:29
*** haplo37 has joined #openstack-kolla22:29
*** mbound_ has joined #openstack-kolla22:33
*** aNupoisc has joined #openstack-kolla22:35
*** mark-casey has quit IRC22:45
*** ravig has joined #openstack-kolla22:47
*** salv-orl_ has joined #openstack-kolla22:59
*** salv-orlando has quit IRC23:02
*** g3ek has quit IRC23:06
*** g3ek has joined #openstack-kolla23:11
*** harlowja has joined #openstack-kolla23:12
*** dcwangmit01_ has quit IRC23:17
*** dcwangmit01 has joined #openstack-kolla23:17
*** dwalsh has quit IRC23:38
*** mbound_ has quit IRC23:44
*** adn has joined #openstack-kolla23:44
*** aNupoisc has quit IRC23:45
*** aNupoisc has joined #openstack-kolla23:46
*** adn has quit IRC23:49
*** vhosakot has quit IRC23:55
harlowjaqq, why in rhel and such are all the repos turned on even if u have no desire to build said components?23:56
*** aNupoisc has quit IRC23:57
harlowjaie kibana, mariadb, elasticearch23:57
harlowjawhy aren't those just turned on if/when u are building  those components23:58
*** kbyrne has quit IRC23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!