Saturday, 2016-10-08

kfox1111_interesting....00:01
kfox1111_so it printed "Using" in the log.00:02
kfox1111_so it failed to IP=$(ip addr show dev {{ tunnel_interface }} | grep -G ''inet '' | awk ''{print $2}'' | sed ''s@/.*@@'');00:02
kfox1111_oh...00:02
kfox1111_I never set one.00:02
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104100:04
*** awiddersheim has quit IRC00:07
kfox1111_I guess that explains why I get random failures sometimes with the centos side too. if its landing on nodes with different nics.00:09
*** daneyon has joined #openstack-kolla00:12
*** zhubingbing_ has joined #openstack-kolla00:12
*** tonanhngo has joined #openstack-kolla00:13
*** tonanhngo has quit IRC00:14
*** tonanhngo has joined #openstack-kolla00:16
*** daneyon has quit IRC00:16
*** nick-ma has joined #openstack-kolla00:17
*** ayoung has quit IRC00:18
*** elemoine_ has joined #openstack-kolla00:19
*** ssurana has joined #openstack-kolla00:25
*** zhubingbing_ has quit IRC00:25
kfox1111_ok... so at least ubuntu and centos are equally failing now. :/00:30
kfox1111_still the same libvirt key missing error...00:30
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104100:39
*** dwalsh has joined #openstack-kolla00:53
*** zhubingbing has joined #openstack-kolla00:59
zhubingbinghello guys01:09
*** dwalsh has quit IRC01:13
*** zhurong has joined #openstack-kolla01:17
*** unicell has quit IRC01:21
*** haplo37_ has quit IRC01:28
*** haplo37_ has joined #openstack-kolla01:29
*** ssurana has quit IRC01:37
*** MarMat has quit IRC01:38
*** sdake has joined #openstack-kolla01:43
zhubingbinghello sdake01:46
sdakesup zhubingbing01:46
zhubingbinghttps://bugs.launchpad.net/kolla/+bug/162880601:47
openstackLaunchpad bug 1628806 in kolla "collectd requires sudoers entry" [Critical,Confirmed]01:47
zhubingbingcan u look it ?01:47
zhubingbingthis bug have reproduce01:47
zhubingbingso we should deal it  now?01:48
sdakeyes - critical - confirmed - looks easy to fix01:48
sdakeknock yourself out :)01:48
*** caowei has joined #openstack-kolla01:49
*** zhenguo has joined #openstack-kolla01:50
zhubingbingok01:51
zhubingbingi fix it01:51
sdakewhy is it critical01:58
sdakeseems "high" to me01:58
sdakesudo: no tty present and no askpass program specified01:59
sdakezhubingbing that isn't a sudoers file problem I think01:59
sdakeit looks more like a lack of tty on the launching of hte container01:59
sdakecould be wrong01:59
zhubingbingyes01:59
zhubingbingi have test02:00
zhubingbingi think so, sudo file problem02:00
*** bmace has quit IRC02:00
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104102:00
Pavogood evening02:01
Pavosdake guess what02:01
sdakesup pavo02:01
sdakesup dude02:01
Pavogonna be picking me up 2 HP servers Tuesday, 1 has 96 cpus, 512Gb RAM and 24TB storage and other is 96cpus, 512Gb RAM and 4.5Tb storage02:02
Pavogonna be using them to do some Kolla testing and Heat templating02:02
sdakecool02:03
sdakei have used hp gear, cisco gear, and dell gear02:03
sdakei prefer cisco ucs gear the best02:03
sdakenot becuase they employ me - but because it works well :)02:03
sdakemy dellgear is super noisy02:03
sdakethe hp gear seemed functional tho02:03
Pavowell it will also have a Cisco 4600 series router and Cisco 3950X switch02:04
sdakeya dont know much about cisco switches02:04
sdakeI don t thatis02:04
Pavonetworkign guru here02:04
Pavonetworking*02:05
Pavosigned up for Cisco's new CCNA Cyber OPS cert this weekend also02:05
Pavojust waiting on access02:05
sdakedo they just sell those hp servers at the store or something02:05
sdakeyou siad "pick up"02:05
sdakeseems like somethign wierd to stock :)02:06
Pavonah we got them as a demo from CritcalStack ( just got bought out from Capital One)02:06
sdakewhat is criticalstack02:06
Pavohttps://www.criticalstack.com02:07
sdakeif you need a contact for ucs gear, hit me up - i can get you to the right cat02:07
Pavowill do, might actuall want some demo for it sometime02:07
sdakedo you know a cat named chris claus02:08
Pavohmm, sound firmilar02:08
sdakehe founded internet security systems02:08
sdakehe would have founded it with me if i wasn't such a wimp02:09
sdakeinstead of being a multimllionare at 25, i'm still schlepping away writing code at 42 :(02:09
Pavoman I would kill to learn python, C, and Objective-C from someone that knows them very well02:10
Pavoteaching myself from reading books, watching youtube and other crap isn't helping at all02:11
sdakebooks are a good way to learn languages02:11
sdakethe basics02:11
sdaketo realy learn you have to wr ite code02:11
sdakedaily in that language02:11
Pavoyeah and they all are boring as hell02:11
Pavolol02:12
Pavoyeah what I have been doing is writting 1 function a day for things I want to try and build02:12
Pavowritting functions and everything else is pretty easy, but putting it all together is what is killing me02:13
Pavojust can not wrap my brain around it02:13
sdakepavo it tkaes 5 years of ddaily use to become an expert in language02:13
*** bmace has joined #openstack-kolla02:13
Pavoyeah I bet it does02:14
Pavoif I can get C down I think the others wouldn't be so bad02:14
Pavoso whats new02:20
Pavoany major changes or fixes?02:20
sdakeyes rdo is now based on newton instead of master02:23
Pavowow, huge jump02:23
Pavostable?02:23
sdakenotreally a huge jump02:23
Pavowell I was saying its a huge jump since master wasn't even stable yet02:24
Pavounless it is now02:24
Pavobeen a bit since I triyed master02:25
sdakenewton of kolla is not tagged02:25
sdakewe are just using newton upstream of openstack now02:25
Pavoah02:26
Pavoso... I am still working on getting all the repos and keys that are downloaded during image building for offline use02:26
PavoI got all of centos repos done, now working on all keys02:27
openstackgerritzhubingbing proposed openstack/kolla: fix collectd requires sudoers entry  https://review.openstack.org/38399702:28
Pavotrying to create a complete repos container that the build process will use to build images instead of going to the internet to download everything02:28
openstackgerritzhubingbing proposed openstack/kolla: fix collectd requires sudoers entry  https://review.openstack.org/38399702:30
*** Jeffrey4l has joined #openstack-kolla02:39
*** Jeffrey4l_ has joined #openstack-kolla02:43
sdakePavo fwiw I rarely write code these days02:43
sdakemostly yaml02:43
zhubingbingping Dave Walker02:44
sdakewhich i guess is code02:44
sdakesort of02:44
sdakezhubingbing his nick is Daviey02:44
zhubingbingok02:44
Pavoyeah I have been doing alot of ansible yaml files02:44
zhubingbing Daviey can u review https://review.openstack.org/38399702:44
Pavoand trying to make some Heat templetes02:44
sdakeya heat02:47
*** tonanhngo has quit IRC02:54
*** daneyon has joined #openstack-kolla02:55
zhubingbingsdake02:56
sdakesup02:56
zhubingbinghttps://review.openstack.org/38399702:56
zhubingbingplease review it02:56
zhubingbing-)02:56
sdakei did02:56
zhubingbingthank you02:57
*** MarMat has joined #openstack-kolla02:58
*** daneyon has quit IRC02:59
*** haplo37 has quit IRC03:03
*** jtriley has quit IRC03:03
*** tonanhngo has joined #openstack-kolla03:04
*** g3ek has quit IRC03:05
*** haplo37 has joined #openstack-kolla03:05
*** tonanhngo has quit IRC03:05
*** g3ek has joined #openstack-kolla03:06
*** tonanhngo has joined #openstack-kolla03:07
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104103:07
*** Pavo has quit IRC03:08
MarMatping sdake03:12
sdakesup marmat03:12
MarMatstuff sdake03:13
MarMatsdake concerning magnum03:13
sdakeeating shortly spit it out03:13
MarMatsdake from my pov it's gtg except of https://bugs.launchpad.net/kolla/+bug/163077003:13
openstackLaunchpad bug 1630770 in kolla "magnum deploy failing" [High,Incomplete] - Assigned to Martin Matyáš (martinx-maty)03:13
MarMatsdake how to handle that?03:13
*** Pavo has joined #openstack-kolla03:14
sdakelet me read the bug03:14
MarMatsdake need more info from reporter03:14
MarMatsdake not successful with reproduction03:14
sdakeMarMat fair enough - i've sked daviey to respond on the bug03:15
openstackgerritzhubingbing proposed openstack/kolla: fix collectd requires sudoers entry  https://review.openstack.org/38399703:17
MarMatsdake thanks let's see then03:17
sdakeMarMat other bugs towork on if our bored :)03:17
MarMatwill look around for some low hanging cookies03:19
MarMatsdake anyone working on barbican bugs? They seem to be unassigned.03:24
sdakeMarMat if unassigned feel free to work on them03:25
sdakeMarMat what we really need is a specific process followed related to bugs03:25
sdakeits in last weeks meeting notes03:25
sdakebut if you want to work on barbican bugs more power to ya ;)03:25
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104103:36
*** jtriley has joined #openstack-kolla03:37
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod  https://review.openstack.org/38392203:43
*** Jeffrey4l_ has quit IRC03:45
*** jtriley has quit IRC03:45
openstackgerritSteven Dake proposed openstack/kolla: Build source from Newton rather then Master  https://review.openstack.org/38400503:46
*** neilus has joined #openstack-kolla03:46
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod  https://review.openstack.org/38392203:49
*** neilus has quit IRC03:50
sdakeboy that patch was tedious03:53
*** sdake has quit IRC03:59
*** jtriley has joined #openstack-kolla04:03
*** Pavo has quit IRC04:08
*** Pavo has joined #openstack-kolla04:10
*** Pavo has quit IRC04:10
*** haplo37_ has quit IRC04:20
*** haplo37_ has joined #openstack-kolla04:22
*** salv-orlando has joined #openstack-kolla04:23
*** salv-orlando has quit IRC04:28
*** ajpower has quit IRC04:46
*** ajpower has joined #openstack-kolla04:48
*** unicell has joined #openstack-kolla04:48
*** sdake has joined #openstack-kolla04:53
sdakeJeffrey4l noticed some operation not permitted in various bugs i've seen04:55
sdakea pattern that is04:55
sdakere https://bugs.launchpad.net/kolla/+bug/163107204:55
openstackLaunchpad bug 1631072 in kolla "iscsid container: mkdir /sys/kernel/config: operation not permitted" [Undecided,New]04:55
sdakethey always involve /sys04:56
sdakeJeffrey4l need ack on https://review.openstack.org/#/c/384005/05:00
sdakeJeffrey4l so people are testing the corect things05:00
sdakeany other cores around - ditto :)05:00
*** salv-orlando has joined #openstack-kolla05:24
*** sdake_ has joined #openstack-kolla05:26
*** sdake has quit IRC05:27
*** salv-orlando has quit IRC05:29
*** neilus has joined #openstack-kolla05:34
*** daneyon has joined #openstack-kolla05:37
*** HyperJohnGraham_ has joined #openstack-kolla05:38
*** neilus has quit IRC05:39
*** daneyon has quit IRC05:41
Jeffrey4lroger.05:44
*** caowei has quit IRC05:49
Jeffrey4lsdake_, done. a nit.05:51
Jeffrey4lsdake_, can we merge this now? https://review.openstack.org/37273705:51
sdake_Jeffrey4l yes - but has gate failure05:52
sdake_can you recheck it05:52
Jeffrey4lOK>05:52
sdake_our policy is if two other core reviewers have reviewed it but a workflow remains unset05:52
sdake_you can +2 and workflow05:52
sdake_the reason we have the 2 +2s but no workflow is the gate job05:52
sdake_which likely just needs a recheck05:52
Jeffrey4lgot. i recheck the PS. and will +w when the gate is green05:53
*** g3ek has quit IRC05:59
*** haplo37 has quit IRC05:59
sdake_otavio around?06:01
sdake_re uid problem06:01
*** haplo37 has joined #openstack-kolla06:04
*** g3ek has joined #openstack-kolla06:05
*** caowei has joined #openstack-kolla06:07
sdake_Jeffrey4l have a look at https://bugs.launchpad.net/kolla/+bug/163150306:14
openstackLaunchpad bug 1631503 in kolla mitaka "inconsistent UID in rabbitmq results in inability to upgrade" [Critical,Confirmed] - Assigned to Steven Dake (sdake)06:14
sdake_Jeffrey4l before i spend a bunch of time fixing it06:14
sdake_lets have a quick chat about the right way to fix it06:15
Jeffrey4lsdake_, solution 1: force use the same uid in the Dockerfile     solution 2: chown the /var/lib/rabbitmq folder in extend_start.sh script.06:16
Jeffrey4lsdake_, what do u think?06:16
sdake_nah, /etc/passwd has the uid06:16
sdake_so 1 is the only solution06:16
sdake_oh ya, 2 might work06:16
sdake_this needs to be done for more then just rabbitmq06:17
Jeffrey4lthe optimal solution is use unified uid for all services.06:18
Jeffrey4lyep.06:18
sdake_well to unify them, we need to unify them with what was in 2.0.206:18
sdake_and needs to go in base image06:18
Jeffrey4lchown is bad, especially when the user owner and group owner is not the same for all the file in the folder. ( nova is a example )06:18
sdake_ya nova and neutron06:19
sdake_chown not ideal06:19
sdake_for rabbitmq eg we need to unify more then rabbitmq:06:19
Jeffrey4lyes.06:19
sdake_epmd:x:997:996:Erlang Port Mapper Daemon:/tmp:/sbin/nologin06:19
sdake_rabbitmq:x:996:995:RabbitMQ messaging server:/var/lib/rabbitmq:/sbin/nologin06:19
Jeffrey4lit is a little hard for us to unify the uid. we will break the upgrade stage if we use other uid06:20
sdake_well we needto use the uid in 2.0.206:21
sdake_for everything that accesses a volume06:21
sdake_andgid06:21
sdake_more tedious work06:21
* sdake_ groans06:21
Jeffrey4lsdake_, the root issue is: based on the install service order, the rabbitmq user id may changed06:22
sdake_Jeffrey4l not sure if thatis the root issue or not06:22
sdake_Jeffrey4l 2.0.2 has following uids:06:22
sdake_epmd:x:997:996:Erlang Port Mapper Daemon:/tmp:/sbin/nologin06:23
sdake_tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin06:23
sdake_geoclue:x:996:995:User for geoclue:/var/lib/geoclue:/sbin/nologin06:23
sdake_rabbitmq:x:995:994:RabbitMQ messaging server:/var/lib/rabbitmq:/bin/bash06:23
Jeffrey4lit is easy to prove. 1 sec06:23
sdake_based upon your assertion, I have no idea where the 995 UID is going in 2.0.206:23
sdake_oh i know06:24
sdake_epmd -> geoclue -> rabbitmq06:24
sdake_in 2.0.206:24
Jeffrey4lno one know :( if rdo or centos make make change for the package.06:24
sdake_in 3.0.0 epmd -> rabbitmq06:24
sdake_counts backwards from 100006:24
Jeffrey4lit is hard to predict.06:25
Jeffrey4lyes.06:25
sdake_we can hardcode to whatever was in 2.0.206:25
*** salv-orlando has joined #openstack-kolla06:25
sdake_we can probably just surgically fix rabbitmq for now06:25
sdake_and address it properly in ocata06:25
Jeffrey4lwithout kolla/docker container, the rabbitmq will not create user/group with new uid.06:25
Jeffrey4lagain.06:25
sdake_Jeffrey4l huh?06:26
sdake_uid/gid created by packaging or dockerfiles06:26
Jeffrey4lyes. but think of the rabbitmq upgrade on bare linux distro ( without docker container ), why it is not cause any issue.06:27
Jeffrey4lthe reason is:  the rabbitmq will not create user/group with new uid during upgrade, if the rabbitmq user already created.06:27
sdake_because the passwd file is loaded with the uid/gid already06:27
Jeffrey4lyes.06:27
*** salv-orlando has quit IRC06:30
Jeffrey4lwe need create the rabbitmq user at the beginning of rabbitmq/Dockerfile file. and centos and ubuntu may use different uid/gid.06:30
sdake_i was thinking in base06:30
sdake_but it probably needs conditional on centos vs ubuntu06:31
Jeffrey4lsdake_, no. it will break all the containers.06:31
sdake_groan06:31
Jeffrey4lin nova container: 996 may be nova user.06:31
sdake_ok well surgical fix for rabbitmq06:32
Jeffrey4lwe may need  rabbitmq:x:996 in rabbitmq container and  nova:x:996 in nova container.06:32
sdake_i got that - that is what the groan was about06:32
Jeffrey4lannoying issue :(06:32
sdake_i knew this would be a problem a year ago06:33
sdake_never had time to fix it :(06:33
sdake_perhaps ocata could have some type of migration as part of the upgrade to unify uid/gid across the system in base06:33
Jeffrey4lsdake_, yes. we need that.06:34
sdake_i dont like the idea of having uid/gid in each seprate file with conditionals on distros06:34
sdake_one file is better06:34
sdake_but too late to sort that out now06:34
Jeffrey4lya. create all user in base image with different uid/gid is better.06:35
sdake_just need something to migrate the volumes06:35
sdake_we dont have time for that now06:35
Jeffrey4lsdake_, we can add06:35
Jeffrey4ltypo.   yes. this should be fixed in O06:35
sdake_nor do I have the forttitude to take it on 4 days prior to release06:35
sdake_ok well that sounds like SEP for ocata ;)06:36
*** tonanhngo has quit IRC06:36
sdake_i'll fix up rabbitmq for newton06:36
Jeffrey4lcool06:37
*** tonanhngo has joined #openstack-kolla06:38
*** tonanhngo has quit IRC06:41
*** haplo37 has quit IRC06:48
*** g3ek has quit IRC06:49
*** haplo37 has joined #openstack-kolla06:50
*** g3ek has joined #openstack-kolla06:51
*** msimonin has joined #openstack-kolla06:51
*** MarMat has quit IRC06:52
openstackgerritSteven Dake proposed openstack/kolla: Build source from Newton rather then Master  https://review.openstack.org/38400506:53
*** jtriley has quit IRC06:55
*** tonanhngo has joined #openstack-kolla07:12
Jeffrey4labout the kuryr, i think we need use 0.1.0 tag   see here http://tarballs.openstack.org/kuryr/07:12
Jeffrey4lsdake_, ^07:13
*** tonanhngo has quit IRC07:13
sdake_Jeffrey4l there are two things there07:13
sdake_Jeffrey4l they broke out "lib"07:13
sdake_but there is also something else that is "not lib"07:14
sdake_they have not tagged the non-lib version07:14
Jeffrey4land the kuryr bust the gate in master now ( it add itself to kuryr-lib >= 0.1.0)07:14
openstackgerritzhubingbing proposed openstack/kolla: fix collectd  https://review.openstack.org/38399707:15
Jeffrey4lsdake_, check this https://github.com/openstack/kuryr/blob/0.1.0/setup.cfg#L2   the i think kuryr renamed its name to kuryr-lib.07:15
Jeffrey4lthere is only one package .07:15
Jeffrey4lre the  kuryr bust gate:  check the log http://logs.openstack.org/37/372737/10/check/gate-kolla-dsvm-build-centos-source-centos-7-nv/d447e04/console.html#_2016-10-08_06_15_15_01076307:15
Jeffrey4lneed your PS to merge ASAP ;)07:16
openstackgerritSteven Dake proposed openstack/kolla: [WIP] Fix upgrades for rabbitmq  https://review.openstack.org/38402907:18
openstackgerritprameswar proposed openstack/kolla:  apt instead of apt-get to install packages  in Ubuntu  https://review.openstack.org/37947707:19
zhubingbingsdake can u review it ?https://review.openstack.org/#/c/383576/07:21
*** neilus has joined #openstack-kolla07:23
openstackgerritSteven Dake proposed openstack/kolla: Build source from Newton rather then Master  https://review.openstack.org/38400507:25
*** salv-orlando has joined #openstack-kolla07:26
*** neilus has quit IRC07:27
sdake_zhubingbing on bug triage07:28
sdake_critical = everything = bust07:29
sdake_high = one component = bust07:29
zhubingbingthanks07:30
*** salv-orlando has quit IRC07:30
*** msimonin has quit IRC07:34
openstackgerritMerged openstack/kolla: add sahara log in heka  https://review.openstack.org/38357607:42
sdake_Jeffrey4l bit of a problem07:51
sdake_in ubuntu :07:51
sdake_systemd-bus-proxy:x:105:07:51
sdake_in 2.02-> -bash: rabbitmq:x:102:105:RabbitMQ: command not found07:51
Jeffrey4l... ubuntu xenial added systemd  :(07:51
sdake_right07:51
sdake_make it harder07:52
sdake_and ubuntu allocates at 100+07:52
sdake_instead of 1000-07:52
* Jeffrey4l groan 07:53
sdake_chown it is07:53
Jeffrey4lsdake_, so ubuntu should totally bust. not only the rabbitmq, other serivce should be affect, too.07:53
sdake_this totally craters upgrades if nova does wacky thing07:53
sdake_Jeffrey4l i have to admit i don't understand why - but its 1am and i'm out of steam07:54
sdake_a fresh deploy has no problems07:55
sdake_its upgrades that have problems07:55
Jeffrey4lsdake_, yes. and upgrade is totally busted.07:55
*** zhubingbing has quit IRC07:55
sdake_on debuntian?07:55
sdake_we have approximately 10 days to sort out a plan c07:55
sdake_if a surgical fix wont work07:55
Jeffrey4lsdake_, we need create the user with specified uid/gid. and then chown the named volume in extend_start script.07:56
sdake_you think upgrade on ubuntu is broken beyond just debuntian?07:56
sdake_ratherbeyond just rabbitmq on debuntain07:57
Jeffrey4lyes.07:57
sdake_my proposal -> remove ubuntu from system ;-)07:57
Jeffrey4lsystemd-bus-proxy:x:105: will affect the user id in all containers.07:58
Jeffrey4llol, let's start to remove it :)07:58
sdake_we only care about uids in containers with volume attachments07:58
*** salv-orlando has joined #openstack-kolla07:59
sdake_Jeffrey4l i was just kidding on the ubuntu thing07:59
Jeffrey4lsdake_, so, the fastest solution should be: change the uid/gid for all file in named volume.07:59
Jeffrey4lfor nova and neutron, we need more carefully. others are fine.08:00
sdake_heka?08:00
Jeffrey4lre remove ubuntu: i knew :)08:00
Jeffrey4l /var/lib/heka should be fine  /var/log/kolla/<service> should be well take care.08:01
Jeffrey4lsdake_, ^^08:03
*** msimonin has joined #openstack-kolla08:07
*** sdake_ has quit IRC08:11
openstackgerritJeffrey Zhang proposed openstack/kolla: Use mariadb.cnf for mariadb custom file name  https://review.openstack.org/38404308:15
openstackgerritzhubingbing proposed openstack/kolla: fix keystone log repeated collection in heka  https://review.openstack.org/38306608:16
*** daneyon has joined #openstack-kolla08:19
*** matrohon has joined #openstack-kolla08:22
*** sdake has joined #openstack-kolla08:23
*** daneyon has quit IRC08:23
sdakeJeffrey4l i think correct solution is combo of two things:08:28
sdakefind all the uids/gids used in volume mounts and give them persistent numbers08:28
sdakechown them to those uids08:29
sdakethe chown should be part of upgrade playbook i think08:29
*** liyifeng has joined #openstack-kolla08:29
Jeffrey4lthen how and when create users with the persistent numbers?08:30
Jeffrey4lsdake,08:30
sdakecreate uers with persistent uid/gid in base08:30
sdakeglobal mapping08:31
sdakethe advantage of this is it fixes itself once newton goes eol in 1 year08:31
Jeffrey4lso the first thing ( find all the uids/gids used in volume mounts and give them persistent numbers ) can be done now08:32
sdakewithout carrying the chown08:32
sdakewhat i proposed is proper way to fix it with least long term pain08:33
sdakelets talk about short term pain08:33
sdakethis upgrade playbook08:33
Jeffrey4l yes your solution is optimal.08:33
sdakehow would we do that?08:33
Jeffrey4lwe can implement this in O.08:33
Jeffrey4lhmm08:34
sdakeif we can do in newton easily we shouldl08:34
sdakethe mapping is a pain in the ass08:34
sdaketedious08:34
Jeffrey4lthe only solution i think is add chown all file in named volume in extend_start.sh file08:34
sdakedocker exec chown?08:35
sdakemount container with all named volumes, docker exec chown it08:35
sdakeor create a hack container08:35
sdakethat does this with extend_start08:35
Jeffrey4lexec should be fine. but we may need ` docker exec -u nova_api chown nova:nova -R /var/lib/nova`08:35
Jeffrey4lexec should be fine. but we may need ` docker exec -u root nova_api chown nova:nova -R /var/lib/nova`08:36
Jeffrey4li'd like to use extend_start solution.08:36
sdakei dont want that all over the containers08:36
sdakeour containers are going to become othe rpeoples dependencies08:36
Jeffrey4lso isn't extend_start more stable?08:37
sdakein one container specially designed for this chown that owuld be suitable to me08:38
sdakeimo we should leave extend_start alone across the codebase08:38
sdakecontainers are basically designed as good as they are going to get imo08:38
Jeffrey4lsdake, for example, /var/lib/nova, how could we know the nova user id in nova container now?08:38
Jeffrey4l we must run the nova container, then we can get the nova user uid/gui08:39
Jeffrey4lgid08:39
sdake2am - moment to think08:39
Jeffrey4lwe are not using unified uid/gid for all containers.08:40
sdakeat the moment no08:40
sdakefor newton i see no reason not to do so08:40
sdakeall - those with volume mounts08:41
Jeffrey4lso we can not the uid/gid without start/run each container.08:41
Jeffrey4lunified uid/gid should be added in newtone :(08:41
sdakeyou mean *all* containers?08:42
sdakethese are two separate problems08:42
sdakeone is the uid space08:42
sdakeone is how to get from 2.0.2 to 2.0.3 sanely08:42
sdakerather 2.0.3/3.0.008:43
*** salv-orl_ has joined #openstack-kolla08:43
Jeffrey4lso from 2.0.2 to 2.0.3 i think the only care about is rabbitmq, right?08:43
sdakethe third problem may or may not be something we care about which is pollution of extend_start in newton08:43
sdake2.0.2->2.0.3 seems to have no problems from my testing08:44
Jeffrey4lif no, that's be cool.08:44
sdakeif we backport rabbitmq fixes for rdo to 2.0.2 unclear08:44
sdakehowever we can solve that problem in the backport08:44
Jeffrey4lyes.08:45
Jeffrey4lso the big issue is upgrading from 2.0.3 to 3.0.008:45
sdakeonly issue at this point ;)08:45
*** salv-orlando has quit IRC08:46
Jeffrey4lthe long term solution should be : unified gid/uid08:46
Jeffrey4land short term solution is: ? ( no idea now )08:46
sdakethat should be short term solution08:46
sdakethe question is how to upgrade08:46
Jeffrey4lshould we post a mail to talk about this?08:47
sdakewe need to get cracking on an answer already08:47
sdakewe dont have time for ml on this one08:47
sdakethe prblem i have with extend_start across all the containers is that its permanent08:48
sdakethere is no way to undo it in the future08:48
sdaketest my assertion there in your brain make sure its correct08:48
Jeffrey4lyes. and it will be useful when we implement unified uid/gid08:48
sdakewe implement unified uid/gid now08:49
sdakebecausse ubuntu is totally broken without it08:49
Jeffrey4lunified uid/gid mean: create all the user in base, right?08:49
sdakeright and all groups as well08:49
Jeffrey4lyes. this is easy.08:50
Jeffrey4lthen we need change all files in named_volume to correct owner.08:51
Jeffrey4lhow about this: docker run -it -v nova:/var/lib/nvoa -v glance:/var/lib/glance -v ceph:/var/lib/ceph ... --rm centos-source-base:3.0.0  <change_named_volume_uid_gid.sh>08:53
Jeffrey4lsdake,08:53
sdakeyup that works08:53
sdakeno idea how to fix this one https://review.openstack.org/#/c/382470/08:54
Jeffrey4lafter newton branch is created, revert it in that branch?08:55
*** salv-orl_ has quit IRC08:55
*** salv-orlando has joined #openstack-kolla08:55
sdakeJeffrey4l jenkins is -1 on the patch08:56
sdakebecause it is checking against the global requirements file08:56
Jeffrey4loh, sorry. hmm08:57
Jeffrey4llet's revert it on newton branch after it is created.08:57
sdakethat may work08:58
Jeffrey4l+2 for this Build source from Newton rather then Master  https://review.openstack.org/#/c/384005/08:58
sdakehttps://github.com/openstack/requirements/branches08:59
sdakethere appears to be a newton branch08:59
sdakecan you add your comment to the review Jeffrey4l08:59
Jeffrey4lsdake, you need abandon your PS and push a new one to newton branch when it is created.09:00
sdakeyes - i keep stuff in queue as reminder of todo09:00
Jeffrey4lCool09:00
Jeffrey4li will -w for you PS with some comment.09:00
sdakeadd a note there so my brain doesn't forget ;)09:00
Jeffrey4lthere is a job for requirements check : gate-kolla-requirements, the branch name should be hardcode in its job.09:02
sdakeyup i understand09:03
*** dwalsh has joined #openstack-kolla09:13
*** tonanhngo has joined #openstack-kolla09:13
*** tonanhngo has quit IRC09:16
sdakewell i'm off to bed09:19
sdakelater09:19
*** sdake has quit IRC09:24
*** mewald has joined #openstack-kolla09:41
*** tonanhngo has joined #openstack-kolla09:49
*** salv-orlando has quit IRC09:50
*** tonanhngo has quit IRC09:51
*** zhurong has quit IRC10:00
*** neilus has joined #openstack-kolla10:05
*** neilus has quit IRC10:10
*** msimonin1 has joined #openstack-kolla10:11
*** msimonin has quit IRC10:14
*** tonanhngo has joined #openstack-kolla10:19
*** tonanhngo has quit IRC10:22
*** tonanhngo has joined #openstack-kolla10:48
*** dwalsh has quit IRC10:49
*** tonanhngo has quit IRC10:49
*** daneyon has joined #openstack-kolla11:01
*** daneyon has quit IRC11:07
*** tonanhngo has joined #openstack-kolla11:17
*** MarMat has joined #openstack-kolla11:18
*** tonanhngo has quit IRC11:18
*** kuntumtarm has joined #openstack-kolla11:29
openstackgerritprameswar proposed openstack/kolla: dependency packages added that need to be installed before ansible installation  https://review.openstack.org/38407311:32
*** tonanhngo has joined #openstack-kolla11:43
*** tonanhngo has quit IRC11:46
*** msimonin has joined #openstack-kolla12:01
*** zhurong has joined #openstack-kolla12:03
*** msimonin1 has quit IRC12:03
*** kuntumtarm has quit IRC12:12
*** zhurong has quit IRC12:30
*** haplo37_ has quit IRC12:41
*** haplo37_ has joined #openstack-kolla12:43
*** liyifeng has quit IRC12:45
*** mewald has quit IRC12:49
*** daneyon has joined #openstack-kolla12:50
*** daneyon has quit IRC12:54
*** dave-mccowan has joined #openstack-kolla12:57
*** sdake has joined #openstack-kolla13:19
*** sdake_ has joined #openstack-kolla13:22
*** sdake has quit IRC13:24
*** Pavo has joined #openstack-kolla13:30
*** sdake_ has quit IRC13:43
*** dwalsh has joined #openstack-kolla13:47
*** matrohon has quit IRC13:49
*** matrohon has joined #openstack-kolla13:50
*** tonanhngo has joined #openstack-kolla13:52
*** sdake has joined #openstack-kolla13:55
*** tonanhngo has quit IRC13:57
sdakeJeffrey4l are ou still about14:04
*** msimonin has quit IRC14:10
*** dave-mccowan has quit IRC14:20
*** msimonin has joined #openstack-kolla14:33
*** msimonin has quit IRC14:34
*** msimonin has joined #openstack-kolla14:36
*** sdake has quit IRC14:37
*** sdake has joined #openstack-kolla14:38
*** rhallisey has joined #openstack-kolla14:41
HyperJohnGraham_Morning all14:41
*** awiddersheim has joined #openstack-kolla14:42
HyperJohnGraham_sbezverk: i rebuilt docker with btrfs and now have a stable k8s canal running http://paste.openstack.org/show/584978/14:46
*** dwalsh has quit IRC14:49
otaviosdake: I reported the bug for RDO people; now awaiting for their feedback14:51
sdakeotavio cool14:51
sdakeotavio say this uid bug14:51
sdakeyou reported it from 2.0.2->2.0.314:51
sdakei am unable to reproduce that14:51
sdakealthough your report has opened up a can of worms for 2.0.2->3.0.0 that I can reproduce14:52
otaviosdake: our deploy was quite small; dunno if it may have change14:52
sdakeotavio how did you get 2.0.3 exactly?14:52
sdakesince it is unreleased ;)14:52
otaviosdake: bear in mind we are starting on this14:52
otaviosdake: we built it from kolla14:53
sdakeotavio roger dude - looks like your off to a good start ;)14:53
sdakeyou build from git you mean?14:53
sdakethe 2.0.3 images?14:53
otaviosdake: as we did for 3.014:53
otaviosdake: yes14:53
sdakewait a minute14:53
sdakethe bug report said 2.0.2->2.0.314:53
sdakeyou just said 3.014:53
sdakewhich versions we talking about :)14:53
otaviosdake: we did not use prebuilt images. We used kolla-build14:53
sdakeok14:54
sdakeso you pip pinstalled kolla- then built kolla 2.0.214:54
otaviosdake: we installed 3.0 only today so we were using 2.0.214:54
otaviosdake: yes14:54
sdakethen pip removed kolla and then build kolla 2.0.3 from stable/newton branch?14:54
otaviosdake: no; from stable/mitaka14:54
sdakesorry thats what i meant14:55
sdakedid you do the pip uninstall kolla part?14:55
otaviosdake: but we played a lot so we may have end building a mess14:55
otaviosdake: I think we did14:55
sdakeyou will neeed to do so14:55
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod  https://review.openstack.org/38392214:56
*** zhubingbing has joined #openstack-kolla14:58
zhubingbingsup sdake14:58
sdakehey zhubingbing14:59
zhubingbingyour area is moring right?15:00
otaviosdake: btw, how do you develop? do you install it every time for test?15:00
zhubingbingdevelop?15:01
otaviozhubingbing: git workflow15:01
zhubingbingi every time for test15:02
zhubingbinggit fetch path  pip install kolla15:02
zhubingbingtest it15:02
zhubingbing- )15:02
otaviozhubingbing: pip uninstall kolla; pip install . ?15:02
zhubingbingyeah15:02
otaviook15:02
zhubingbingpip uninall kolla/ ; pip install kolla/15:03
zhubingbingpip install .  have problems15:03
zhubingbingwho can help me15:05
zhubingbinghttps://review.openstack.org/#/c/383080/15:05
zhubingbinghelp review it15:05
zhubingbing-)  -)15:06
openstackgerritMerged openstack/kolla-kubernetes: Enable release notes translation  https://review.openstack.org/38322715:06
*** sdake_ has joined #openstack-kolla15:10
sdake_zhenguo its 8am here15:11
sdake_otavio what do yo umean?15:11
sdake_otavio I rebuild daily15:11
sdake_otavio rebuild the world that is15:11
sdake_then I develop from master15:11
sdake_or develop from stable/newton or whatever15:11
sdake_i rarely pip install unless testing upgrades15:11
*** AJaeger has left #openstack-kolla15:11
sdake_otavio you don't want to do pip install .15:12
sdake_otavio i run from git, and then run tools/build.py or tools/kolla-ansible deploy etc15:12
sdake_otavio it puts a little extra burden on the dev because of the PBR problem15:12
sdake_to know two sets of commands instead of one15:12
sdake_but really there are only two comands15:13
sdake_build and deploy15:13
sdake_zhubingbing its 8am here15:13
*** sdake has quit IRC15:13
sdake_any core reviewers around - this needs to go in stat -> https://review.openstack.org/#/c/384005/15:15
openstackgerritMerged openstack/kolla: fix can't collect horizon log in heka  https://review.openstack.org/38308015:16
sdake_21 critical bugs15:17
*** sdake_ is now known as sdake15:17
sdakewtb dev ;)15:17
zhubingbing-)15:17
sdakeotavio the problem is worse then you reported15:18
sdakeotavio all of ubuntu upgrades are broken15:18
sdakenot that i use or test ubuntu15:18
sdakeotavio because ubuntu added systemd15:18
zhubingbingi have test ubuntu15:19
openstackgerritRyan Hallisey proposed openstack/kolla: Allow cinder-volume to be configured to use nfs  https://review.openstack.org/27991315:19
zhubingbingbase system use ubuntu or  kolla use ubuntu?15:20
zhubingbinghttps://bugs.launchpad.net/kolla/+bug/1625705 sdake this bug you confirm ?15:21
openstackLaunchpad bug 1625705 in kolla "latest kolla/ubuntu-binary-heka:3.0.0 container keep on restarting " [Critical,Incomplete]15:21
*** salv-orlando has joined #openstack-kolla15:25
sdakezhubingbing was not able to confirm that15:28
sdakezhubingbing I left a note in the bug log speculating as to the root cause15:28
zhubingbingok15:28
zhubingbingroger15:29
*** salv-orlando has quit IRC15:29
zhubingbingi will try reproduce this bug15:29
sdakezhubingbing work on triaged bugs first plz15:31
sdakezhubingbing this cat has 8gb ram15:32
sdakehe has an OOM problem or OOD problem I am pretty sure15:32
zhubingbingok15:32
sdakeor eveyrone would be yelling about kolla not working15:32
zhubingbingright15:32
sdakewe have 3 separate people in that bug report that say they can't reproduce the problem15:32
*** daneyon has joined #openstack-kolla15:32
sdakeeither work on getting triaged -> some other state15:33
sdakeor fixing confirmed bugs of which there are plenty15:33
zhubingbingunderstand15:35
*** daneyon has quit IRC15:37
sdakezhubingbing whats the story with this review: https://review.openstack.org/#/c/375380/7/ansible/roles/gnocchi/templates/gnocchi-api.json.j215:44
*** eaguilar has joined #openstack-kolla15:44
sdakepaul asked a question and i have same question15:44
sdakewhy change that?15:44
sdakewhat specifically is broken in bug: https://bugs.launchpad.net/kolla/+bug/162695915:45
openstackLaunchpad bug 1626959 in kolla "change perm 06400 to 0600 in gnocchi-api.json.j2" [High,In progress] - Assigned to zhubingbing (zhubingbing)15:45
sdakeyou list a solution not a problem15:45
zhubingbingemm15:45
sdakezhubingbing ^^15:45
zhubingbingbecause  gnocchi api use http_mod_wsgi15:46
zhubingbingto start15:46
sdakecould you expand on what happens without this change15:46
sdakezhubingbing order should be 1) inprogress bugs your working on 2) triage->to any other state 3) high/critical bugs unassigned assign15:48
zhubingbing1) understand15:49
zhubingbing2) don‘t quite understand15:50
sdakezhubingbing i marked https://bugs.launchpad.net/kolla/+bug/1626959 incomplete15:50
openstackLaunchpad bug 1626959 in kolla "change perm 06400 to 0600 in gnocchi-api.json.j2" [High,Incomplete] - Assigned to zhubingbing (zhubingbing)15:50
zhubingbingany other status/?15:50
sdake2 we want to get bugs in triaged state to some other state such as confirmed invalid or incomplete15:51
zhubingbingok, understand15:51
sdakebut irst we want to get inprogress bugs we are working  on already fixed15:52
sdakeand finally we want  to work on high/critical bugs15:52
sdakesome of the cores may not  follow this pattern - this is fine - this pattern is more for new folks or people ocming up to speed15:52
zhubingbingI know, we should be organized to deal with these bug15:53
sdakezhubingbing re https://bugs.launchpad.net/kolla/+bug/163104115:57
openstackLaunchpad bug 1631041 in kolla "Keystone log repeated collection in heka" [Medium,In progress] - Assigned to zhubingbing (zhubingbing)15:57
sdakeheka-keystone.taml or whatever its called is the lua loggingfor keystone15:57
sdakeheka-openstack is the definition of all those things combined15:57
sdakeare you actually seeing keystone logged twice?15:57
kfox1111_down to this weird cinder attach issue.... so weird...15:58
kfox1111_http://logs.openstack.org/41/381041/94/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/ebd9451/logs/openstack/kolla-nova-libvirt-w20kj.txt15:58
zhubingbingheke monitor kolla_logs/keystone file15:59
*** awiddersheim has quit IRC16:01
sdakezhubingbing are the logs sent twice to log  files via heka?16:01
zhubingbingheka-keystone.taml have collect one times and heka-openstack collect keystone log one times16:02
sdakeright, but does that mean the log file is written with two lines?16:02
sdakeany cores around - the gate is jammed16:03
sdakeneed some help there16:03
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104116:03
zhubingbingand 2 files content are the same.16:03
rhalliseysdake, ya here16:04
rhalliseywhich patch16:04
sdakerhallisey moment16:05
sdakehttps://review.openstack.org/#/c/384005/16:05
zhubingbing[root@kolla-test-1 logstreamer]# diff keystone-apache-admin keystone_keystone-apache-admin-access16:05
zhubingbing[root@kolla-test-1 logstreamer]#16:05
sdakei see so writing to two files?16:06
sdakewith your patch, which file does it write to?16:06
openstackgerritMerged openstack/kolla: Build source from Newton rather then Master  https://review.openstack.org/38400516:09
zhubingbingwe have use heka-keystone.toml why also use heka-openstack16:09
zhubingbingI mean, keystone has been defined by heka for 2 times.16:10
zhubingbingheka-keystone.taml and heka-openstack16:10
*** dwalsh has joined #openstack-kolla16:12
*** tonanhngo has joined #openstack-kolla16:16
*** huikang has joined #openstack-kolla16:21
*** huikang has quit IRC16:22
hogepodgeI don't entirely understand how to do a custom configuration in kolla16:22
*** huikang has joined #openstack-kolla16:22
hogepodgesdake: say, for example, I want to enable http store for glance along with rdb, when and where do I update the config glance config files?16:23
sdakerhallisey can you do me a solid and read the irc logs from last night16:23
*** inc0 has joined #openstack-kolla16:23
inc0heyh16:23
inc0sdake,16:23
inc0wassup?16:24
sdakemoment i'll find link for both of you to read16:24
rhalliseyk16:24
sdakehogepodge moment pls16:24
sdakehogepodge there are others around that can answer your q - need to find link for inc0/rhallisey to read through16:24
*** salv-orlando has joined #openstack-kolla16:26
*** kuntumtarm has joined #openstack-kolla16:26
sdakehere is where it begins inc0 and rhallisey http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2016-10-08.log.html#t2016-10-08T06:14:4916:26
sdakethe solution is towards the end of our discussion16:26
sdakenote this affects EVERYTHING16:28
sdakethat uses a named volume16:28
sdakenot just rabbitmq16:28
inc0isn't that bug in docker?16:29
sdakehogepodge mkdir /etc/kolla/config16:29
sdakehogepodge put in ceph.conf your configuration stuff16:29
inc0like USER rabbitmq != USER rabbitmq if we redeploy16:29
sdakeinc0 nah - it depends on the order its built in16:29
sdakeand what packages install which dependencies that install users/groups16:30
openstackgerritJeffrey Zhang proposed openstack/kolla: Use mariadb.cnf for mariadb custom file name  https://review.openstack.org/38404316:30
openstackgerritJeffrey Zhang proposed openstack/kolla: integrate gnocchi with ceilometer  https://review.openstack.org/37273716:30
sdakehogepodge note i'm not sure ceph is customizable (if its ini file it is, if not it isn't)16:30
inc0so hold on, if it gets UID/GID during buildtime16:30
*** salv-orlando has quit IRC16:31
inc0hogepodge, what's the issue?16:31
rhalliseyI see16:31
sdakeinc0 he wants to customize ceph16:31
inc0customize how?16:31
*** tonanhngo has quit IRC16:32
sdakezhubingbing you didn't answer my question, the question was with your patch is there only one file, or is there two.  if only one file, what is it  called with your patch?16:32
inc0and yes, ceph.conf can be customized by merge_config16:32
inc0ok back to the rabbit issue16:32
hogepodgeinc0: if I enable rdb as the store for glance, it writes the glance conf files like this: http://paste.openstack.org/show/584984/16:33
sdakecool where there ya go hogepodge :)16:33
sdakehogepodge do you have enable_ceph: yes set?16:33
hogepodgeinc0: which is fine, except for normally glance has the line 'stores = rdb' configured to 'stores = file,http'16:33
*** tonanhngo has joined #openstack-kolla16:33
hogepodgesdake: inc0: so I'd like that line to read 'stores = rdb,http'16:33
inc0hogepodge, ok, easy enough, let me link you doc for it16:34
sdakehogepodge mind explaining what the http part does - perhaps that should go in our defaults16:34
sdakehogepodge nm - i misread your last statement16:34
hogepodgesdake: it lets glance pull images from remote locations. quite a few heat templates depend on it16:35
inc0hogepodge, https://github.com/openstack/kolla/blob/master/doc/advanced-configuration.rst#openstack-service-configuration-in-kolla16:35
sdakehogepodge can you file a bug for that problem please16:35
sdakei think that needs to go in our defaults16:35
inc0this method allows you to override any option in .ini confs16:35
inc0in this case you need override glance:)16:35
sdakehogepodge glance is overridable16:35
hogepodgeI find those docs confusing. If I'm doing multi-node, do I drop the conf on the system I run kolla-ansible from, or do I have to change the configs that are dropped on the target nodes?16:36
sdakerhallisey - so good with the solution - understand problem?16:36
inc0hogepodge, all in deployment node16:36
inc0I'll make a note to clarify that in docs16:36
sdakehogepodge yes - docs are #1, #2, #3 painpoints for all of our operators16:36
zhubingbingsdake Sorry, I may have made a mistake, the bug description is not accurate enough.16:36
sdakehogepodge you dont have to do any of that, kolla has single source of truth (the deployment node contains the single source)16:37
rhalliseysdake, ya I"m fine with the solution.  O is where we'll have to fix it everywhere16:37
inc0hogepodge, in general you shouldn't need to touch any node besides deployment16:37
sdakehogepodge you can poke around and hack stuff manually - that is what CONFIG_STRATEGY is about16:37
inc0ansible does all the rest16:37
sdakerhallisey i am pretty happy with the docker run option jeffrey and I came up with16:38
sdakedid you get that far into the discussion16:38
sdakerhallisey we need to fix it everywhere now unfortunately16:38
sdakebecause ubuntu upgrade is completely broken16:39
sdakebecause systemd is colliding with 2.0.2 container uids16:39
sdakeRTL :)16:39
hogepodgeI see, so I don't need the entire file, just the group and option I want changed, and kolla merges, right?16:40
sdakehogepodge you got it16:40
sdakehogepodge pretty sweet huh :)16:40
sdakeno turnaround on dev/test to add custom config16:40
sdakeno kolla making policy decisions for operators16:40
HyperJohnGraham_Hi All... whats the story with setup_gate.sh ? ( just did a git pull :)  I am rebuilding from scratch and it looks like there are lots of cool changes in the install process16:41
sdakeHyperJohnGraham_ - kfox1111_ has indicated there is a cinder blocker16:41
HyperJohnGraham_so i should start building a wall ??16:41
sdakepretty much ;)16:42
HyperJohnGraham_:){16:42
kfox1111_sbezverk: ceph osd blacklist add <--- this looks interesting too.16:42
hogepodgesdake: that's cool, thanks. I'll do a reconfigure16:42
rhalliseysdake, how is it that rabbit is the only problem now16:42
sdakerhallisey its the only reported problem16:42
sdakenobody clearly has tested ubuntu 2.0.2 to 3.0.016:43
sdakean upgrade thereof16:43
rhalliseyHyperJohnGraham, setup_gate script will be run by the gate.  It's not meant to be run locally16:43
rhalliseysdake, would a doc of the docker exec for this b ok?16:43
rhalliseyuntil its properly fixed in O16:44
sdakei expect ubuntu 2.0.2 to 3.0.0 would implode in bad ways16:44
sdakerhallisey no a doc wouldn't do the job16:44
rhalliseyor do you intend to add those into the upgrade playbook16:44
sdakerhallisey because the upgrade would stop and crater16:44
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104116:44
kfox1111_I wonder...16:44
sdakei intend to add a run command to the upgrade playbook16:44
*** zhubingbing has quit IRC16:44
sdakedocker exec rather16:44
sdakekeep scrolling rhallisey - read down more ;)16:44
rhalliseyI'm at the point you go to sleep16:45
sdakedocker run thatis16:45
sdakeright - so right above that jeffrey proposes a soltuion i had in my head16:45
sdakewhich is to mount the persistent volumes and chown them via a shell script16:45
sdakethen we can remove that crap in ocata16:45
sdakerather when newton goes eol16:45
inc0sdake, Jeffrey4l UID issue...you're saying that UID gets created on container build?16:46
inc0really?16:46
rhalliseyyes16:46
sdakeinc0 yes for packages installed via apt/yum16:46
Jeffrey4linc0, it is.16:46
rhalliseyinc0, docker fs works that way16:46
inc0so rabbimq package creates user?16:46
sdakeinc0 right16:46
inc0not USER clausule16:46
inc0?16:46
rhalliseylittle think rw layer on top16:46
rhalliseythin*16:47
sdakeas well as keepalived, etc16:47
inc0let's discuss one Kolla user then16:47
inc0because that seems wrong16:47
Jeffrey4lall packages ( including rpm/deb) may create user16:47
sdakeone kolla user is insecure16:47
sdakealso it wont work16:47
inc0seems like major docker issue to me16:48
sdakeinc0 no  shit16:48
inc0how about GID?16:48
rhalliseythis is kinds neat: docker run -it -v nova:/var/lib/nvoa -v glance:/var/lib/glance -v ceph:/var/lib/ceph ... --rm centos-source-base:3.0.016:48
sdakegid same story16:48
rhalliseythat replaces the underlying layers16:48
sdakerhallisey right16:48
rhalliseybut do the running contianers get it16:48
rhalliseydoubt it16:48
inc0can we create group for kolla just for perm and add users created to this group?16:48
rhalliseyI guess they woudl be restarted thouhg16:48
rhalliseyso they would16:48
sdakeinc0 that is massive rework16:49
rhalliseyfirst step of upgrade process16:49
inc0well, not necesarly, we could do this in kolla_start16:49
sdakehow do you end up chowning /etc/nova for example in kolla_start?16:50
sdakeor chmoding to 66016:50
inc0chmoding16:50
sdakewhich reduces security along the  way16:50
sdakewe have json to set permissions not kolla_start16:50
sdakethat would totally crater our abi16:50
inc0well, set_config file16:51
sdakeright - massive rework across all json files16:51
*** awiddersheim has joined #openstack-kolla16:51
sdakenot set_configs.py16:51
Jeffrey4l /etc/nova should be fine. But named volumes should be took care.16:51
rhalliseyawiddersheim, I updated that patch this morning16:51
rhalliseyI think it should be good to go16:52
awiddersheimsounds good16:52
sdakeJeffrey4l right named volumes16:52
sdakehow to chmod named volumes16:52
inc0let me read some internets about it16:52
Jeffrey4lchmod? chown?16:52
sdakethe options are docker run as jeffrey proposed for extend_start16:52
inc0other people had to have same issue16:52
sdakeinc0 i guess if they upgraded to master from 2.0.216:53
sdakeinc0 which i'm guessing hasn't happened16:53
Jeffrey4linc0, if he doesn't use docker/container, he haven't such issue.16:53
Jeffrey4lbecause the /etc/passwd file is not changed during upgrade.16:53
Jeffrey4li.e. rabbitmq user will not re-create with new uid/gid when upgrading rabbitmq16:54
sdakeinc0 this question is answered in the irc log16:54
sdakelast day to vote for tc election for stragglers ;)16:55
sdakeinc0 not many people use our pattern of dependent packages with two base images16:56
inc0so let's narrow down what's impacted16:56
sdakeinc0 anything with a named volume is impacted on ubuntu16:56
inc01. containers that installs apt package that creates users16:56
sdakeinc0 rabbitmq is impacted on centos16:57
inc02. containers that uses named volume16:57
sdakewe don't care about #1, it doesns't impact upgrade16:57
inc0we do, as when you build new containers they create user with potentially different gid16:57
sdakewe being i:)16:58
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104116:58
inc0that's the reason it is different and messes up perms16:58
sdakeinc0 right but this isn't a problem unless that uid is shared16:58
sdakeuids are only sahred on named volumes16:58
inc0it is kinda shared in perms16:58
inc0so we need to have it synced16:58
sdakewe definately need to have named volume uids/gids synced in base16:59
inc0base is rebuilt too16:59
inc0forcing uid/gid in containers like rabbitmq would work16:59
sdakeright - which means we need to chown the named volumes on upgrade via docker run16:59
inc0not necessarly, we can also just force uid during build time17:00
inc0to ensure that rabbitmq will get uid 950 no matter how many times you rebuild17:00
sdakeyou didn't read log or you would know this results in an entire 6 months of maintaining something that is not ideal17:00
sdakethat wont work17:00
sdakeuid in 2.0.2 = 19517:00
sdakeif uid in 3.0.0 is 95017:01
sdakeupgrade = crater17:01
sdakeuid in 3.0.0 must either be 950 or docker run must chown the named volumes during upgrade playbook17:01
sdakeand it must be in base17:01
sdakebecause uids are reused in different containers if not already in e/tc/passwd17:01
sdakethere is no global uid mapping17:02
sdakedocker may have added such a feature but it wont work for our use case17:02
*** janem has joined #openstack-kolla17:02
sdakesolving for rabbitmq is easy17:02
sdakesolving for ubuntu more difficult17:03
sdakebecause ubuntu has systemd17:03
sdakeand systemd uses the same uids as all kinds of stuff in the named volumes17:03
sdake(from 2.0.2)17:03
inc0chowning on docker run might not work as we run as underprivileged user17:04
inc0you can't change your uid on the fly17:04
sdakeinc0 if you read the log - Jeffrey4l proposed -u root17:04
sdakeR..T...L...17:05
*** eaguilar has quit IRC17:05
sdakewe burned up 3 hours brainstorming last night to solve this problem17:05
sdakeits captured in the log17:05
*** dwalsh has quit IRC17:05
sdakeif you have something further to add to the brainstorming - best wayt o catch up is spend 1 0 minutes reading the log17:05
sdake1 0/1017:05
sdakeupgrade is our greatest success, we can't crater it in 3.0.017:07
sdakerhallisey do you understand the issue and the solution?17:08
* rhallisey reads back17:09
sdakerhallisey the one in the log i posted17:09
rhalliseyyes17:09
rhalliseythe one you posted17:09
inc0will mariadb be affected too?17:09
*** g3ek has quit IRC17:09
sdakeon ubuntu - yup17:09
sdakeon centos - nope17:10
*** haplo37 has quit IRC17:10
*** tonanhngo has quit IRC17:10
inc0so issue with base is - we don't know uid/gid at build time17:10
sdake1. we assign it starting at 50017:11
Jeffrey4linc0, and uid/gid is not the same in 2.0.2 and 3.0.017:11
*** g3ek has joined #openstack-kolla17:11
sdake2. we run docker run with named volumes mounted -u root upgrade-uids.sh17:11
inc0so unless we force it in 3.0.0, same issue will appear in 4.0.017:11
sdakeupgra-uids.sh has all the chown  work in it17:12
sdakeyes the proposal is forcing it17:12
*** haplo37 has joined #openstack-kolla17:12
Jeffrey4lcorrect. it will also happen like rabbitmq case.17:12
Jeffrey4lduring upgrade from 2.0.2 to 2.0.317:12
inc0ok...yeah, that will work and will ensure uids in 3.0.017:12
inc0we can backport it to stable as far as I'm concerned17:12
inc0it's major issue17:12
sdakebecause #2 doesn't need to know the uids ahead of time we are golden17:13
sdakelets get master fixed first ;)17:13
inc0so we need to figure out full list of users created in containers..17:13
sdakeonly those with named volumes17:14
sdakeunless we want to do across code base17:14
sdakewhich wfm17:14
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH  https://review.openstack.org/32630717:14
inc0if we could, we make our life easier for N->O upgrade across the board17:14
sdakeagree17:14
sdakesince we dont have to map them ahead of time - only determine what the uids are17:14
sdakeit should be a little easier17:15
inc0we can map them ahead of time tho17:15
inc0in base17:15
sdakeright - that was the original proposal ;)17:15
inc0and then chown17:15
inc0that's good solution imho17:15
inc0we can chown in upgrade playbook17:15
sdakeyes - agree - so thats 3 people ;)17:15
sdakeenough +2s to work with :)17:16
sdakerhallisey when your cuaght up let us know what you think17:16
rhalliseyI'm caught up17:16
rhalliseysounds good to me17:16
inc0so first big thing -> get all users ever created across the board and assign gid to them17:17
inc0uid*17:17
sdakeand gid17:17
inc0put this perpared /etc/password file in base17:17
inc0and gid17:17
sdakewe can use useradd commands rather then prepared passwd file17:17
sdakepasswd file may change in base image on a rebuild17:18
inc0and in upgrade playbook if your volume uid/gid != your uid/gid when it should in config.json, chown17:18
sdakethe base one from centos:7 eg17:18
inc0yeah, useradd is better ofc17:18
Jeffrey4lwe need care about the user home when creating, useradd --user-group --create-home --home-dir /var/lib/nova nova17:18
sdakeJeffrey4l make it harder ;)17:19
rhalliseywe only need to fix one thing for N17:20
rhalliseyright?17:20
rhalliseyjust rabbit17:20
sdakerhallisey no all things17:20
sdakesee this review for the useradd logic and groupadd logic17:20
sdakei dislike it17:20
sdakebut i don't see any cli options to do it better17:20
sdakehttps://review.openstack.org/#/c/384029/17:20
rhalliseywhat happend to the docker run solution?17:21
sdakeJeffrey4l just think how much easier life would be with only cnetos17:21
sdakerhallisey ya i think that is best rather then jerking around with extend_start17:22
Pavoyes centos is the way to go :)17:22
rhalliseyyou could have the docker run solution as part of the upgrade playbook for N17:22
rhalliseyfirst step run for each service17:22
sdakerhallisey thats the plan17:22
sdakerhallisey i was thinking we would do it once17:22
sdakewith all the named volumes mounted against the chown_uids.sh script17:23
Jeffrey4llife is tough ;)17:23
Jeffrey4lcentos will also have such issue, if centos image add some thing which crated a new user.17:24
sdakeJeffrey4l you have no idea17:24
*** tonanhngo has joined #openstack-kolla17:24
sdakeJeffrey4l re review above, any ideas on better 1 liner for uid/gid addition?17:25
sdakenothing like a lasat minute fire drill to wake up the day :)17:26
*** salv-orlando has joined #openstack-kolla17:26
Jeffrey4lgroupadd -g 500 && useradd --user-group --create-home --home-dir /var/lib/nova --uid 500 --gid 500 nova17:27
Jeffrey4lseems need two lines.17:27
*** salv-orlando has quit IRC17:31
kfox1111_hmm... so its not the rbd mounts from the kernel mounts...17:31
sbezverkkfox1111_: hey, qq can in theory ceph volume has multiple locks?17:33
kfox1111_if its a read only volume.17:36
kfox1111_there can be only one writer lock, or potentially multiple readers.17:38
*** eaguilar has joined #openstack-kolla17:39
*** awidders_ has joined #openstack-kolla17:40
*** awiddersheim has quit IRC17:41
sdakelocking101 :)17:42
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104117:43
*** huikang has quit IRC17:44
sdakehogepodge ok bug triaged17:44
hogepodgesdake: thanks, it's a pretty simple fix. I'd be happy to send the patch up17:44
*** huikang has joined #openstack-kolla17:44
hogepodgeif you want to assign it to me17:44
sdakehogepodge all yours17:45
*** huikang has quit IRC17:45
*** huikang has joined #openstack-kolla17:46
sbezverkkfox1111_: I tested last night pod clening logic by killing compute nodes every few minutes and it works really well :-)17:46
sbezverkkfox1111_: I want to add ceph lock clearing, tired of doing it manually.17:46
kfox1111_awesome. :)17:47
*** huikang has quit IRC17:49
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104117:49
inc0bbiab I need to run some errands17:50
inc0will be back afternoon to fix some stuff:)17:50
*** inc0 has quit IRC17:50
sdakewe have 9 confirmed high/critical bugs to fix17:54
sdakeneed to fix those17:54
sdakepreferabbly before 12th17:54
sdakeand about 20 inprogress bugs17:55
sdakethose should be fixed too if there are patches in the queue17:55
sdakehave some errands to run -then get my workstation in working order17:59
sdakefi someone gets to that uid bug before me it won't hurt my feelings17:59
sdakeotherwise i'll have a patch up sunday17:59
sdakebbl17:59
sdakeworkstation/laptop17:59
*** sdake has quit IRC17:59
*** salv-orlando has joined #openstack-kolla18:00
*** huikang has joined #openstack-kolla18:05
*** dwalsh has joined #openstack-kolla18:05
*** eaguilar has quit IRC18:06
*** salv-orlando has quit IRC18:08
*** salv-orlando has joined #openstack-kolla18:08
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs  https://review.openstack.org/38104118:14
*** daneyon has joined #openstack-kolla18:15
*** mewald has joined #openstack-kolla18:15
*** msimonin has quit IRC18:16
*** daneyon has quit IRC18:19
*** mewald has quit IRC18:21
*** tonanhngo has quit IRC18:29
*** tonanhngo has joined #openstack-kolla18:31
*** salv-orlando has quit IRC18:45
*** mewald has joined #openstack-kolla18:56
*** mewald has quit IRC18:57
*** awidders_ has quit IRC18:58
*** dwalsh has quit IRC19:00
*** eaguilar has joined #openstack-kolla19:03
*** msimonin has joined #openstack-kolla19:03
*** pbourke has quit IRC19:09
*** pbourke has joined #openstack-kolla19:09
*** msimonin has quit IRC19:10
*** msimonin1 has joined #openstack-kolla19:10
*** tonanhngo has quit IRC19:19
*** sdake has joined #openstack-kolla19:20
sdakewhere doth zhubingbing go19:26
sdakeJeffrey4l you touched this last - how does this patch look to you https://review.openstack.org/#/c/382356/1/ansible/roles/horizon/templates/local_settings.j219:28
Jeffrey4lsdake, i think both public and intern should be fine in most case.19:31
*** dave-mccowan has joined #openstack-kolla19:32
*** msimonin1 has quit IRC19:32
sdakei dont know how that is used19:32
sdakewhy change it from public to internal?19:32
sdakeJeffrey4l ^19:33
Jeffrey4lsdake, i leave a comment to ask the author why this need to be changed.19:33
*** MarMat has quit IRC19:33
sdakewfm19:33
sdakei'm sure berendt will get to it19:33
*** jmccarthy has quit IRC19:33
*** jmccarthy has joined #openstack-kolla19:34
Jeffrey4lpublic should be more useful. if the horizon node can not touch the internal network. sdake19:34
Jeffrey4lso i do not think we need such a change now.19:35
*** msimonin has joined #openstack-kolla19:35
sdakei agree19:35
sdakenot sure why its needed19:35
*** haplo37 has quit IRC19:36
*** g3ek has quit IRC19:39
*** awiddersheim has joined #openstack-kolla19:39
*** g3ek has joined #openstack-kolla19:41
*** haplo37 has joined #openstack-kolla19:45
*** Pavo has quit IRC19:45
sdakerhallisey ping19:53
sdakerhallisey i gotta jet, but can you turn this into a bug report: https://ask.openstack.org/en/question/97154/why-start-heka-container-twice/19:53
rhalliseysure19:53
*** sdake has quit IRC19:54
openstackgerritRyan Hallisey proposed openstack/kolla: Remove unneeded Heka bootstrapping code  https://review.openstack.org/38412220:00
*** dave-mccowan has quit IRC20:00
*** neilus has joined #openstack-kolla20:15
*** sdake has joined #openstack-kolla20:17
*** HyperJohnGraham_ has quit IRC20:17
*** awiddersheim has quit IRC20:17
*** sdake has quit IRC20:20
*** msimonin has quit IRC20:24
*** msimonin has joined #openstack-kolla20:28
*** Serlex has joined #openstack-kolla20:30
*** msimonin has quit IRC20:47
*** kuntumtarm has quit IRC20:48
*** msimonin has joined #openstack-kolla20:49
*** Pavo has joined #openstack-kolla20:49
*** sdake has joined #openstack-kolla20:52
*** sdake has quit IRC21:05
*** dwalsh has joined #openstack-kolla21:14
*** msimonin1 has joined #openstack-kolla21:15
*** msimonin has quit IRC21:15
*** msimonin1 has left #openstack-kolla21:15
*** Pavo has quit IRC21:16
*** Pavo has joined #openstack-kolla21:17
*** sdake has joined #openstack-kolla21:22
sdakethanks rhallisey21:22
sdakeyou didn't have to fix the bug ust file it :)21:23
*** huikang has quit IRC21:24
*** Pavo has quit IRC21:25
*** sdake has quit IRC21:33
*** Pavo has joined #openstack-kolla21:34
*** mewald has joined #openstack-kolla21:37
*** Pavo has quit IRC21:46
*** salv-orlando has joined #openstack-kolla21:59
mewaldHi, I lost root access to my mariadb cluster. How do I recover that for a containerized mariadb?22:10
*** Serlex has quit IRC22:10
*** eaguilar has quit IRC22:15
*** msimonin has joined #openstack-kolla22:17
*** neilus has quit IRC22:18
*** eaguilar has joined #openstack-kolla22:23
*** dwalsh has quit IRC22:27
*** schwicht has joined #openstack-kolla22:33
*** dave-mccowan has joined #openstack-kolla22:35
*** Pavo has joined #openstack-kolla22:42
*** daneyon has joined #openstack-kolla22:46
*** eaguilar has quit IRC22:48
*** daneyon has quit IRC22:50
*** schwicht has joined #openstack-kolla22:58
*** msimonin has quit IRC23:07
*** neilus has joined #openstack-kolla23:09
*** neilus has quit IRC23:14
*** dave-mccowan has quit IRC23:24
*** eaguilar has joined #openstack-kolla23:26
*** salv-orlando has quit IRC23:26
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod  https://review.openstack.org/38392223:45
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod  https://review.openstack.org/38392223:47

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!