| kfox1111_ | interesting.... | 00:01 |
|---|---|---|
| kfox1111_ | so it printed "Using" in the log. | 00:02 |
| kfox1111_ | so it failed to IP=$(ip addr show dev {{ tunnel_interface }} | grep -G ''inet '' | awk ''{print $2}'' | sed ''s@/.*@@''); | 00:02 |
| kfox1111_ | oh... | 00:02 |
| kfox1111_ | I never set one. | 00:02 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 00:04 |
| *** awiddersheim has quit IRC | 00:07 | |
| kfox1111_ | I guess that explains why I get random failures sometimes with the centos side too. if its landing on nodes with different nics. | 00:09 |
| *** daneyon has joined #openstack-kolla | 00:12 | |
| *** zhubingbing_ has joined #openstack-kolla | 00:12 | |
| *** tonanhngo has joined #openstack-kolla | 00:13 | |
| *** tonanhngo has quit IRC | 00:14 | |
| *** tonanhngo has joined #openstack-kolla | 00:16 | |
| *** daneyon has quit IRC | 00:16 | |
| *** nick-ma has joined #openstack-kolla | 00:17 | |
| *** ayoung has quit IRC | 00:18 | |
| *** elemoine_ has joined #openstack-kolla | 00:19 | |
| *** ssurana has joined #openstack-kolla | 00:25 | |
| *** zhubingbing_ has quit IRC | 00:25 | |
| kfox1111_ | ok... so at least ubuntu and centos are equally failing now. :/ | 00:30 |
| kfox1111_ | still the same libvirt key missing error... | 00:30 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 00:39 |
| *** dwalsh has joined #openstack-kolla | 00:53 | |
| *** zhubingbing has joined #openstack-kolla | 00:59 | |
| zhubingbing | hello guys | 01:09 |
| *** dwalsh has quit IRC | 01:13 | |
| *** zhurong has joined #openstack-kolla | 01:17 | |
| *** unicell has quit IRC | 01:21 | |
| *** haplo37_ has quit IRC | 01:28 | |
| *** haplo37_ has joined #openstack-kolla | 01:29 | |
| *** ssurana has quit IRC | 01:37 | |
| *** MarMat has quit IRC | 01:38 | |
| *** sdake has joined #openstack-kolla | 01:43 | |
| zhubingbing | hello sdake | 01:46 |
| sdake | sup zhubingbing | 01:46 |
| zhubingbing | https://bugs.launchpad.net/kolla/+bug/1628806 | 01:47 |
| openstack | Launchpad bug 1628806 in kolla "collectd requires sudoers entry" [Critical,Confirmed] | 01:47 |
| zhubingbing | can u look it ? | 01:47 |
| zhubingbing | this bug have reproduce | 01:47 |
| zhubingbing | so we should deal it now? | 01:48 |
| sdake | yes - critical - confirmed - looks easy to fix | 01:48 |
| sdake | knock yourself out :) | 01:48 |
| *** caowei has joined #openstack-kolla | 01:49 | |
| *** zhenguo has joined #openstack-kolla | 01:50 | |
| zhubingbing | ok | 01:51 |
| zhubingbing | i fix it | 01:51 |
| sdake | why is it critical | 01:58 |
| sdake | seems "high" to me | 01:58 |
| sdake | sudo: no tty present and no askpass program specified | 01:59 |
| sdake | zhubingbing that isn't a sudoers file problem I think | 01:59 |
| sdake | it looks more like a lack of tty on the launching of hte container | 01:59 |
| sdake | could be wrong | 01:59 |
| zhubingbing | yes | 01:59 |
| zhubingbing | i have test | 02:00 |
| zhubingbing | i think so, sudo file problem | 02:00 |
| *** bmace has quit IRC | 02:00 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 02:00 |
| Pavo | good evening | 02:01 |
| Pavo | sdake guess what | 02:01 |
| sdake | sup pavo | 02:01 |
| sdake | sup dude | 02:01 |
| Pavo | gonna be picking me up 2 HP servers Tuesday, 1 has 96 cpus, 512Gb RAM and 24TB storage and other is 96cpus, 512Gb RAM and 4.5Tb storage | 02:02 |
| Pavo | gonna be using them to do some Kolla testing and Heat templating | 02:02 |
| sdake | cool | 02:03 |
| sdake | i have used hp gear, cisco gear, and dell gear | 02:03 |
| sdake | i prefer cisco ucs gear the best | 02:03 |
| sdake | not becuase they employ me - but because it works well :) | 02:03 |
| sdake | my dellgear is super noisy | 02:03 |
| sdake | the hp gear seemed functional tho | 02:03 |
| Pavo | well it will also have a Cisco 4600 series router and Cisco 3950X switch | 02:04 |
| sdake | ya dont know much about cisco switches | 02:04 |
| sdake | I don t thatis | 02:04 |
| Pavo | networkign guru here | 02:04 |
| Pavo | networking* | 02:05 |
| Pavo | signed up for Cisco's new CCNA Cyber OPS cert this weekend also | 02:05 |
| Pavo | just waiting on access | 02:05 |
| sdake | do they just sell those hp servers at the store or something | 02:05 |
| sdake | you siad "pick up" | 02:05 |
| sdake | seems like somethign wierd to stock :) | 02:06 |
| Pavo | nah we got them as a demo from CritcalStack ( just got bought out from Capital One) | 02:06 |
| sdake | what is criticalstack | 02:06 |
| Pavo | https://www.criticalstack.com | 02:07 |
| sdake | if you need a contact for ucs gear, hit me up - i can get you to the right cat | 02:07 |
| Pavo | will do, might actuall want some demo for it sometime | 02:07 |
| sdake | do you know a cat named chris claus | 02:08 |
| Pavo | hmm, sound firmilar | 02:08 |
| sdake | he founded internet security systems | 02:08 |
| sdake | he would have founded it with me if i wasn't such a wimp | 02:09 |
| sdake | instead of being a multimllionare at 25, i'm still schlepping away writing code at 42 :( | 02:09 |
| Pavo | man I would kill to learn python, C, and Objective-C from someone that knows them very well | 02:10 |
| Pavo | teaching myself from reading books, watching youtube and other crap isn't helping at all | 02:11 |
| sdake | books are a good way to learn languages | 02:11 |
| sdake | the basics | 02:11 |
| sdake | to realy learn you have to wr ite code | 02:11 |
| sdake | daily in that language | 02:11 |
| Pavo | yeah and they all are boring as hell | 02:11 |
| Pavo | lol | 02:12 |
| Pavo | yeah what I have been doing is writting 1 function a day for things I want to try and build | 02:12 |
| Pavo | writting functions and everything else is pretty easy, but putting it all together is what is killing me | 02:13 |
| Pavo | just can not wrap my brain around it | 02:13 |
| sdake | pavo it tkaes 5 years of ddaily use to become an expert in language | 02:13 |
| *** bmace has joined #openstack-kolla | 02:13 | |
| Pavo | yeah I bet it does | 02:14 |
| Pavo | if I can get C down I think the others wouldn't be so bad | 02:14 |
| Pavo | so whats new | 02:20 |
| Pavo | any major changes or fixes? | 02:20 |
| sdake | yes rdo is now based on newton instead of master | 02:23 |
| Pavo | wow, huge jump | 02:23 |
| Pavo | stable? | 02:23 |
| sdake | notreally a huge jump | 02:23 |
| Pavo | well I was saying its a huge jump since master wasn't even stable yet | 02:24 |
| Pavo | unless it is now | 02:24 |
| Pavo | been a bit since I triyed master | 02:25 |
| sdake | newton of kolla is not tagged | 02:25 |
| sdake | we are just using newton upstream of openstack now | 02:25 |
| Pavo | ah | 02:26 |
| Pavo | so... I am still working on getting all the repos and keys that are downloaded during image building for offline use | 02:26 |
| Pavo | I got all of centos repos done, now working on all keys | 02:27 |
| openstackgerrit | zhubingbing proposed openstack/kolla: fix collectd requires sudoers entry https://review.openstack.org/383997 | 02:28 |
| Pavo | trying to create a complete repos container that the build process will use to build images instead of going to the internet to download everything | 02:28 |
| openstackgerrit | zhubingbing proposed openstack/kolla: fix collectd requires sudoers entry https://review.openstack.org/383997 | 02:30 |
| *** Jeffrey4l has joined #openstack-kolla | 02:39 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 02:43 | |
| sdake | Pavo fwiw I rarely write code these days | 02:43 |
| sdake | mostly yaml | 02:43 |
| zhubingbing | ping Dave Walker | 02:44 |
| sdake | which i guess is code | 02:44 |
| sdake | sort of | 02:44 |
| sdake | zhubingbing his nick is Daviey | 02:44 |
| zhubingbing | ok | 02:44 |
| Pavo | yeah I have been doing alot of ansible yaml files | 02:44 |
| zhubingbing | Daviey can u review https://review.openstack.org/383997 | 02:44 |
| Pavo | and trying to make some Heat templetes | 02:44 |
| sdake | ya heat | 02:47 |
| *** tonanhngo has quit IRC | 02:54 | |
| *** daneyon has joined #openstack-kolla | 02:55 | |
| zhubingbing | sdake | 02:56 |
| sdake | sup | 02:56 |
| zhubingbing | https://review.openstack.org/383997 | 02:56 |
| zhubingbing | please review it | 02:56 |
| zhubingbing | －） | 02:56 |
| sdake | i did | 02:56 |
| zhubingbing | thank you | 02:57 |
| *** MarMat has joined #openstack-kolla | 02:58 | |
| *** daneyon has quit IRC | 02:59 | |
| *** haplo37 has quit IRC | 03:03 | |
| *** jtriley has quit IRC | 03:03 | |
| *** tonanhngo has joined #openstack-kolla | 03:04 | |
| *** g3ek has quit IRC | 03:05 | |
| *** haplo37 has joined #openstack-kolla | 03:05 | |
| *** tonanhngo has quit IRC | 03:05 | |
| *** g3ek has joined #openstack-kolla | 03:06 | |
| *** tonanhngo has joined #openstack-kolla | 03:07 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 03:07 |
| *** Pavo has quit IRC | 03:08 | |
| MarMat | ping sdake | 03:12 |
| sdake | sup marmat | 03:12 |
| MarMat | stuff sdake | 03:13 |
| MarMat | sdake concerning magnum | 03:13 |
| sdake | eating shortly spit it out | 03:13 |
| MarMat | sdake from my pov it's gtg except of https://bugs.launchpad.net/kolla/+bug/1630770 | 03:13 |
| openstack | Launchpad bug 1630770 in kolla "magnum deploy failing" [High,Incomplete] - Assigned to Martin Matyáš (martinx-maty) | 03:13 |
| MarMat | sdake how to handle that? | 03:13 |
| *** Pavo has joined #openstack-kolla | 03:14 | |
| sdake | let me read the bug | 03:14 |
| MarMat | sdake need more info from reporter | 03:14 |
| MarMat | sdake not successful with reproduction | 03:14 |
| sdake | MarMat fair enough - i've sked daviey to respond on the bug | 03:15 |
| openstackgerrit | zhubingbing proposed openstack/kolla: fix collectd requires sudoers entry https://review.openstack.org/383997 | 03:17 |
| MarMat | sdake thanks let's see then | 03:17 |
| sdake | MarMat other bugs towork on if our bored :) | 03:17 |
| MarMat | will look around for some low hanging cookies | 03:19 |
| MarMat | sdake anyone working on barbican bugs? They seem to be unassigned. | 03:24 |
| sdake | MarMat if unassigned feel free to work on them | 03:25 |
| sdake | MarMat what we really need is a specific process followed related to bugs | 03:25 |
| sdake | its in last weeks meeting notes | 03:25 |
| sdake | but if you want to work on barbican bugs more power to ya ;) | 03:25 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 03:36 |
| *** jtriley has joined #openstack-kolla | 03:37 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod https://review.openstack.org/383922 | 03:43 |
| *** Jeffrey4l_ has quit IRC | 03:45 | |
| *** jtriley has quit IRC | 03:45 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Build source from Newton rather then Master https://review.openstack.org/384005 | 03:46 |
| *** neilus has joined #openstack-kolla | 03:46 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod https://review.openstack.org/383922 | 03:49 |
| *** neilus has quit IRC | 03:50 | |
| sdake | boy that patch was tedious | 03:53 |
| *** sdake has quit IRC | 03:59 | |
| *** jtriley has joined #openstack-kolla | 04:03 | |
| *** Pavo has quit IRC | 04:08 | |
| *** Pavo has joined #openstack-kolla | 04:10 | |
| *** Pavo has quit IRC | 04:10 | |
| *** haplo37_ has quit IRC | 04:20 | |
| *** haplo37_ has joined #openstack-kolla | 04:22 | |
| *** salv-orlando has joined #openstack-kolla | 04:23 | |
| *** salv-orlando has quit IRC | 04:28 | |
| *** ajpower has quit IRC | 04:46 | |
| *** ajpower has joined #openstack-kolla | 04:48 | |
| *** unicell has joined #openstack-kolla | 04:48 | |
| *** sdake has joined #openstack-kolla | 04:53 | |
| sdake | Jeffrey4l noticed some operation not permitted in various bugs i've seen | 04:55 |
| sdake | a pattern that is | 04:55 |
| sdake | re https://bugs.launchpad.net/kolla/+bug/1631072 | 04:55 |
| openstack | Launchpad bug 1631072 in kolla "iscsid container: mkdir /sys/kernel/config: operation not permitted" [Undecided,New] | 04:55 |
| sdake | they always involve /sys | 04:56 |
| sdake | Jeffrey4l need ack on https://review.openstack.org/#/c/384005/ | 05:00 |
| sdake | Jeffrey4l so people are testing the corect things | 05:00 |
| sdake | any other cores around - ditto :) | 05:00 |
| *** salv-orlando has joined #openstack-kolla | 05:24 | |
| *** sdake_ has joined #openstack-kolla | 05:26 | |
| *** sdake has quit IRC | 05:27 | |
| *** salv-orlando has quit IRC | 05:29 | |
| *** neilus has joined #openstack-kolla | 05:34 | |
| *** daneyon has joined #openstack-kolla | 05:37 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 05:38 | |
| *** neilus has quit IRC | 05:39 | |
| *** daneyon has quit IRC | 05:41 | |
| Jeffrey4l | roger. | 05:44 |
| *** caowei has quit IRC | 05:49 | |
| Jeffrey4l | sdake_, done. a nit. | 05:51 |
| Jeffrey4l | sdake_, can we merge this now? https://review.openstack.org/372737 | 05:51 |
| sdake_ | Jeffrey4l yes - but has gate failure | 05:52 |
| sdake_ | can you recheck it | 05:52 |
| Jeffrey4l | OK> | 05:52 |
| sdake_ | our policy is if two other core reviewers have reviewed it but a workflow remains unset | 05:52 |
| sdake_ | you can +2 and workflow | 05:52 |
| sdake_ | the reason we have the 2 +2s but no workflow is the gate job | 05:52 |
| sdake_ | which likely just needs a recheck | 05:52 |
| Jeffrey4l | got. i recheck the PS. and will +w when the gate is green | 05:53 |
| *** g3ek has quit IRC | 05:59 | |
| *** haplo37 has quit IRC | 05:59 | |
| sdake_ | otavio around? | 06:01 |
| sdake_ | re uid problem | 06:01 |
| *** haplo37 has joined #openstack-kolla | 06:04 | |
| *** g3ek has joined #openstack-kolla | 06:05 | |
| *** caowei has joined #openstack-kolla | 06:07 | |
| sdake_ | Jeffrey4l have a look at https://bugs.launchpad.net/kolla/+bug/1631503 | 06:14 |
| openstack | Launchpad bug 1631503 in kolla mitaka "inconsistent UID in rabbitmq results in inability to upgrade" [Critical,Confirmed] - Assigned to Steven Dake (sdake) | 06:14 |
| sdake_ | Jeffrey4l before i spend a bunch of time fixing it | 06:14 |
| sdake_ | lets have a quick chat about the right way to fix it | 06:15 |
| Jeffrey4l | sdake_, solution 1: force use the same uid in the Dockerfile solution 2: chown the /var/lib/rabbitmq folder in extend_start.sh script. | 06:16 |
| Jeffrey4l | sdake_, what do u think? | 06:16 |
| sdake_ | nah, /etc/passwd has the uid | 06:16 |
| sdake_ | so 1 is the only solution | 06:16 |
| sdake_ | oh ya, 2 might work | 06:16 |
| sdake_ | this needs to be done for more then just rabbitmq | 06:17 |
| Jeffrey4l | the optimal solution is use unified uid for all services. | 06:18 |
| Jeffrey4l | yep. | 06:18 |
| sdake_ | well to unify them, we need to unify them with what was in 2.0.2 | 06:18 |
| sdake_ | and needs to go in base image | 06:18 |
| Jeffrey4l | chown is bad, especially when the user owner and group owner is not the same for all the file in the folder. ( nova is a example ) | 06:18 |
| sdake_ | ya nova and neutron | 06:19 |
| sdake_ | chown not ideal | 06:19 |
| sdake_ | for rabbitmq eg we need to unify more then rabbitmq: | 06:19 |
| Jeffrey4l | yes. | 06:19 |
| sdake_ | epmd:x:997:996:Erlang Port Mapper Daemon:/tmp:/sbin/nologin | 06:19 |
| sdake_ | rabbitmq:x:996:995:RabbitMQ messaging server:/var/lib/rabbitmq:/sbin/nologin | 06:19 |
| Jeffrey4l | it is a little hard for us to unify the uid. we will break the upgrade stage if we use other uid | 06:20 |
| sdake_ | well we needto use the uid in 2.0.2 | 06:21 |
| sdake_ | for everything that accesses a volume | 06:21 |
| sdake_ | andgid | 06:21 |
| sdake_ | more tedious work | 06:21 |
| * sdake_ groans | 06:21 | |
| Jeffrey4l | sdake_, the root issue is: based on the install service order, the rabbitmq user id may changed | 06:22 |
| sdake_ | Jeffrey4l not sure if thatis the root issue or not | 06:22 |
| sdake_ | Jeffrey4l 2.0.2 has following uids: | 06:22 |
| sdake_ | epmd:x:997:996:Erlang Port Mapper Daemon:/tmp:/sbin/nologin | 06:23 |
| sdake_ | tss:x:59:59:Account used by the trousers package to sandbox the tcsd daemon:/dev/null:/sbin/nologin | 06:23 |
| sdake_ | geoclue:x:996:995:User for geoclue:/var/lib/geoclue:/sbin/nologin | 06:23 |
| sdake_ | rabbitmq:x:995:994:RabbitMQ messaging server:/var/lib/rabbitmq:/bin/bash | 06:23 |
| Jeffrey4l | it is easy to prove. 1 sec | 06:23 |
| sdake_ | based upon your assertion, I have no idea where the 995 UID is going in 2.0.2 | 06:23 |
| sdake_ | oh i know | 06:24 |
| sdake_ | epmd -> geoclue -> rabbitmq | 06:24 |
| sdake_ | in 2.0.2 | 06:24 |
| Jeffrey4l | no one know :( if rdo or centos make make change for the package. | 06:24 |
| sdake_ | in 3.0.0 epmd -> rabbitmq | 06:24 |
| sdake_ | counts backwards from 1000 | 06:24 |
| Jeffrey4l | it is hard to predict. | 06:25 |
| Jeffrey4l | yes. | 06:25 |
| sdake_ | we can hardcode to whatever was in 2.0.2 | 06:25 |
| *** salv-orlando has joined #openstack-kolla | 06:25 | |
| sdake_ | we can probably just surgically fix rabbitmq for now | 06:25 |
| sdake_ | and address it properly in ocata | 06:25 |
| Jeffrey4l | without kolla/docker container, the rabbitmq will not create user/group with new uid. | 06:25 |
| Jeffrey4l | again. | 06:25 |
| sdake_ | Jeffrey4l huh? | 06:26 |
| sdake_ | uid/gid created by packaging or dockerfiles | 06:26 |
| Jeffrey4l | yes. but think of the rabbitmq upgrade on bare linux distro ( without docker container ), why it is not cause any issue. | 06:27 |
| Jeffrey4l | the reason is: the rabbitmq will not create user/group with new uid during upgrade, if the rabbitmq user already created. | 06:27 |
| sdake_ | because the passwd file is loaded with the uid/gid already | 06:27 |
| Jeffrey4l | yes. | 06:27 |
| *** salv-orlando has quit IRC | 06:30 | |
| Jeffrey4l | we need create the rabbitmq user at the beginning of rabbitmq/Dockerfile file. and centos and ubuntu may use different uid/gid. | 06:30 |
| sdake_ | i was thinking in base | 06:30 |
| sdake_ | but it probably needs conditional on centos vs ubuntu | 06:31 |
| Jeffrey4l | sdake_, no. it will break all the containers. | 06:31 |
| sdake_ | groan | 06:31 |
| Jeffrey4l | in nova container: 996 may be nova user. | 06:31 |
| sdake_ | ok well surgical fix for rabbitmq | 06:32 |
| Jeffrey4l | we may need rabbitmq:x:996 in rabbitmq container and nova:x:996 in nova container. | 06:32 |
| sdake_ | i got that - that is what the groan was about | 06:32 |
| Jeffrey4l | annoying issue :( | 06:32 |
| sdake_ | i knew this would be a problem a year ago | 06:33 |
| sdake_ | never had time to fix it :( | 06:33 |
| sdake_ | perhaps ocata could have some type of migration as part of the upgrade to unify uid/gid across the system in base | 06:33 |
| Jeffrey4l | sdake_, yes. we need that. | 06:34 |
| sdake_ | i dont like the idea of having uid/gid in each seprate file with conditionals on distros | 06:34 |
| sdake_ | one file is better | 06:34 |
| sdake_ | but too late to sort that out now | 06:34 |
| Jeffrey4l | ya. create all user in base image with different uid/gid is better. | 06:35 |
| sdake_ | just need something to migrate the volumes | 06:35 |
| sdake_ | we dont have time for that now | 06:35 |
| Jeffrey4l | sdake_, we can add | 06:35 |
| Jeffrey4l | typo. yes. this should be fixed in O | 06:35 |
| sdake_ | nor do I have the forttitude to take it on 4 days prior to release | 06:35 |
| sdake_ | ok well that sounds like SEP for ocata ;) | 06:36 |
| *** tonanhngo has quit IRC | 06:36 | |
| sdake_ | i'll fix up rabbitmq for newton | 06:36 |
| Jeffrey4l | cool | 06:37 |
| *** tonanhngo has joined #openstack-kolla | 06:38 | |
| *** tonanhngo has quit IRC | 06:41 | |
| *** haplo37 has quit IRC | 06:48 | |
| *** g3ek has quit IRC | 06:49 | |
| *** haplo37 has joined #openstack-kolla | 06:50 | |
| *** g3ek has joined #openstack-kolla | 06:51 | |
| *** msimonin has joined #openstack-kolla | 06:51 | |
| *** MarMat has quit IRC | 06:52 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Build source from Newton rather then Master https://review.openstack.org/384005 | 06:53 |
| *** jtriley has quit IRC | 06:55 | |
| *** tonanhngo has joined #openstack-kolla | 07:12 | |
| Jeffrey4l | about the kuryr, i think we need use 0.1.0 tag see here http://tarballs.openstack.org/kuryr/ | 07:12 |
| Jeffrey4l | sdake_, ^ | 07:13 |
| *** tonanhngo has quit IRC | 07:13 | |
| sdake_ | Jeffrey4l there are two things there | 07:13 |
| sdake_ | Jeffrey4l they broke out "lib" | 07:13 |
| sdake_ | but there is also something else that is "not lib" | 07:14 |
| sdake_ | they have not tagged the non-lib version | 07:14 |
| Jeffrey4l | and the kuryr bust the gate in master now ( it add itself to kuryr-lib >= 0.1.0) | 07:14 |
| openstackgerrit | zhubingbing proposed openstack/kolla: fix collectd https://review.openstack.org/383997 | 07:15 |
| Jeffrey4l | sdake_, check this https://github.com/openstack/kuryr/blob/0.1.0/setup.cfg#L2 the i think kuryr renamed its name to kuryr-lib. | 07:15 |
| Jeffrey4l | there is only one package . | 07:15 |
| Jeffrey4l | re the kuryr bust gate: check the log http://logs.openstack.org/37/372737/10/check/gate-kolla-dsvm-build-centos-source-centos-7-nv/d447e04/console.html#_2016-10-08_06_15_15_010763 | 07:15 |
| Jeffrey4l | need your PS to merge ASAP ;) | 07:16 |
| openstackgerrit | Steven Dake proposed openstack/kolla: [WIP] Fix upgrades for rabbitmq https://review.openstack.org/384029 | 07:18 |
| openstackgerrit | prameswar proposed openstack/kolla: apt instead of apt-get to install packages in Ubuntu https://review.openstack.org/379477 | 07:19 |
| zhubingbing | sdake can u review it ?https://review.openstack.org/#/c/383576/ | 07:21 |
| *** neilus has joined #openstack-kolla | 07:23 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Build source from Newton rather then Master https://review.openstack.org/384005 | 07:25 |
| *** salv-orlando has joined #openstack-kolla | 07:26 | |
| *** neilus has quit IRC | 07:27 | |
| sdake_ | zhubingbing on bug triage | 07:28 |
| sdake_ | critical = everything = bust | 07:29 |
| sdake_ | high = one component = bust | 07:29 |
| zhubingbing | thanks | 07:30 |
| *** salv-orlando has quit IRC | 07:30 | |
| *** msimonin has quit IRC | 07:34 | |
| openstackgerrit | Merged openstack/kolla: add sahara log in heka https://review.openstack.org/383576 | 07:42 |
| sdake_ | Jeffrey4l bit of a problem | 07:51 |
| sdake_ | in ubuntu : | 07:51 |
| sdake_ | systemd-bus-proxy:x:105: | 07:51 |
| sdake_ | in 2.02-> -bash: rabbitmq:x:102:105:RabbitMQ: command not found | 07:51 |
| Jeffrey4l | ... ubuntu xenial added systemd :( | 07:51 |
| sdake_ | right | 07:51 |
| sdake_ | make it harder | 07:52 |
| sdake_ | and ubuntu allocates at 100+ | 07:52 |
| sdake_ | instead of 1000- | 07:52 |
| * Jeffrey4l groan | 07:53 | |
| sdake_ | chown it is | 07:53 |
| Jeffrey4l | sdake_, so ubuntu should totally bust. not only the rabbitmq, other serivce should be affect, too. | 07:53 |
| sdake_ | this totally craters upgrades if nova does wacky thing | 07:53 |
| sdake_ | Jeffrey4l i have to admit i don't understand why - but its 1am and i'm out of steam | 07:54 |
| sdake_ | a fresh deploy has no problems | 07:55 |
| sdake_ | its upgrades that have problems | 07:55 |
| Jeffrey4l | sdake_, yes. and upgrade is totally busted. | 07:55 |
| *** zhubingbing has quit IRC | 07:55 | |
| sdake_ | on debuntian? | 07:55 |
| sdake_ | we have approximately 10 days to sort out a plan c | 07:55 |
| sdake_ | if a surgical fix wont work | 07:55 |
| Jeffrey4l | sdake_, we need create the user with specified uid/gid. and then chown the named volume in extend_start script. | 07:56 |
| sdake_ | you think upgrade on ubuntu is broken beyond just debuntian? | 07:56 |
| sdake_ | ratherbeyond just rabbitmq on debuntain | 07:57 |
| Jeffrey4l | yes. | 07:57 |
| sdake_ | my proposal -> remove ubuntu from system ;-) | 07:57 |
| Jeffrey4l | systemd-bus-proxy:x:105: will affect the user id in all containers. | 07:58 |
| Jeffrey4l | lol, let's start to remove it :) | 07:58 |
| sdake_ | we only care about uids in containers with volume attachments | 07:58 |
| *** salv-orlando has joined #openstack-kolla | 07:59 | |
| sdake_ | Jeffrey4l i was just kidding on the ubuntu thing | 07:59 |
| Jeffrey4l | sdake_, so, the fastest solution should be: change the uid/gid for all file in named volume. | 07:59 |
| Jeffrey4l | for nova and neutron, we need more carefully. others are fine. | 08:00 |
| sdake_ | heka? | 08:00 |
| Jeffrey4l | re remove ubuntu: i knew :) | 08:00 |
| Jeffrey4l | /var/lib/heka should be fine /var/log/kolla/<service> should be well take care. | 08:01 |
| Jeffrey4l | sdake_, ^^ | 08:03 |
| *** msimonin has joined #openstack-kolla | 08:07 | |
| *** sdake_ has quit IRC | 08:11 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use mariadb.cnf for mariadb custom file name https://review.openstack.org/384043 | 08:15 |
| openstackgerrit | zhubingbing proposed openstack/kolla: fix keystone log repeated collection in heka https://review.openstack.org/383066 | 08:16 |
| *** daneyon has joined #openstack-kolla | 08:19 | |
| *** matrohon has joined #openstack-kolla | 08:22 | |
| *** sdake has joined #openstack-kolla | 08:23 | |
| *** daneyon has quit IRC | 08:23 | |
| sdake | Jeffrey4l i think correct solution is combo of two things: | 08:28 |
| sdake | find all the uids/gids used in volume mounts and give them persistent numbers | 08:28 |
| sdake | chown them to those uids | 08:29 |
| sdake | the chown should be part of upgrade playbook i think | 08:29 |
| *** liyifeng has joined #openstack-kolla | 08:29 | |
| Jeffrey4l | then how and when create users with the persistent numbers? | 08:30 |
| Jeffrey4l | sdake, | 08:30 |
| sdake | create uers with persistent uid/gid in base | 08:30 |
| sdake | global mapping | 08:31 |
| sdake | the advantage of this is it fixes itself once newton goes eol in 1 year | 08:31 |
| Jeffrey4l | so the first thing ( find all the uids/gids used in volume mounts and give them persistent numbers ) can be done now | 08:32 |
| sdake | without carrying the chown | 08:32 |
| sdake | what i proposed is proper way to fix it with least long term pain | 08:33 |
| sdake | lets talk about short term pain | 08:33 |
| sdake | this upgrade playbook | 08:33 |
| Jeffrey4l | yes your solution is optimal. | 08:33 |
| sdake | how would we do that? | 08:33 |
| Jeffrey4l | we can implement this in O. | 08:33 |
| Jeffrey4l | hmm | 08:34 |
| sdake | if we can do in newton easily we shouldl | 08:34 |
| sdake | the mapping is a pain in the ass | 08:34 |
| sdake | tedious | 08:34 |
| Jeffrey4l | the only solution i think is add chown all file in named volume in extend_start.sh file | 08:34 |
| sdake | docker exec chown? | 08:35 |
| sdake | mount container with all named volumes, docker exec chown it | 08:35 |
| sdake | or create a hack container | 08:35 |
| sdake | that does this with extend_start | 08:35 |
| Jeffrey4l | exec should be fine. but we may need ` docker exec -u nova_api chown nova:nova -R /var/lib/nova` | 08:35 |
| Jeffrey4l | exec should be fine. but we may need ` docker exec -u root nova_api chown nova:nova -R /var/lib/nova` | 08:36 |
| Jeffrey4l | i'd like to use extend_start solution. | 08:36 |
| sdake | i dont want that all over the containers | 08:36 |
| sdake | our containers are going to become othe rpeoples dependencies | 08:36 |
| Jeffrey4l | so isn't extend_start more stable? | 08:37 |
| sdake | in one container specially designed for this chown that owuld be suitable to me | 08:38 |
| sdake | imo we should leave extend_start alone across the codebase | 08:38 |
| sdake | containers are basically designed as good as they are going to get imo | 08:38 |
| Jeffrey4l | sdake, for example, /var/lib/nova, how could we know the nova user id in nova container now? | 08:38 |
| Jeffrey4l | we must run the nova container, then we can get the nova user uid/gui | 08:39 |
| Jeffrey4l | gid | 08:39 |
| sdake | 2am - moment to think | 08:39 |
| Jeffrey4l | we are not using unified uid/gid for all containers. | 08:40 |
| sdake | at the moment no | 08:40 |
| sdake | for newton i see no reason not to do so | 08:40 |
| sdake | all - those with volume mounts | 08:41 |
| Jeffrey4l | so we can not the uid/gid without start/run each container. | 08:41 |
| Jeffrey4l | unified uid/gid should be added in newtone :( | 08:41 |
| sdake | you mean *all* containers? | 08:42 |
| sdake | these are two separate problems | 08:42 |
| sdake | one is the uid space | 08:42 |
| sdake | one is how to get from 2.0.2 to 2.0.3 sanely | 08:42 |
| sdake | rather 2.0.3/3.0.0 | 08:43 |
| *** salv-orl_ has joined #openstack-kolla | 08:43 | |
| Jeffrey4l | so from 2.0.2 to 2.0.3 i think the only care about is rabbitmq, right? | 08:43 |
| sdake | the third problem may or may not be something we care about which is pollution of extend_start in newton | 08:43 |
| sdake | 2.0.2->2.0.3 seems to have no problems from my testing | 08:44 |
| Jeffrey4l | if no, that's be cool. | 08:44 |
| sdake | if we backport rabbitmq fixes for rdo to 2.0.2 unclear | 08:44 |
| sdake | however we can solve that problem in the backport | 08:44 |
| Jeffrey4l | yes. | 08:45 |
| Jeffrey4l | so the big issue is upgrading from 2.0.3 to 3.0.0 | 08:45 |
| sdake | only issue at this point ;) | 08:45 |
| *** salv-orlando has quit IRC | 08:46 | |
| Jeffrey4l | the long term solution should be : unified gid/uid | 08:46 |
| Jeffrey4l | and short term solution is: ? ( no idea now ) | 08:46 |
| sdake | that should be short term solution | 08:46 |
| sdake | the question is how to upgrade | 08:46 |
| Jeffrey4l | should we post a mail to talk about this? | 08:47 |
| sdake | we need to get cracking on an answer already | 08:47 |
| sdake | we dont have time for ml on this one | 08:47 |
| sdake | the prblem i have with extend_start across all the containers is that its permanent | 08:48 |
| sdake | there is no way to undo it in the future | 08:48 |
| sdake | test my assertion there in your brain make sure its correct | 08:48 |
| Jeffrey4l | yes. and it will be useful when we implement unified uid/gid | 08:48 |
| sdake | we implement unified uid/gid now | 08:49 |
| sdake | becausse ubuntu is totally broken without it | 08:49 |
| Jeffrey4l | unified uid/gid mean: create all the user in base, right? | 08:49 |
| sdake | right and all groups as well | 08:49 |
| Jeffrey4l | yes. this is easy. | 08:50 |
| Jeffrey4l | then we need change all files in named_volume to correct owner. | 08:51 |
| Jeffrey4l | how about this: docker run -it -v nova:/var/lib/nvoa -v glance:/var/lib/glance -v ceph:/var/lib/ceph ... --rm centos-source-base:3.0.0 <change_named_volume_uid_gid.sh> | 08:53 |
| Jeffrey4l | sdake, | 08:53 |
| sdake | yup that works | 08:53 |
| sdake | no idea how to fix this one https://review.openstack.org/#/c/382470/ | 08:54 |
| Jeffrey4l | after newton branch is created, revert it in that branch? | 08:55 |
| *** salv-orl_ has quit IRC | 08:55 | |
| *** salv-orlando has joined #openstack-kolla | 08:55 | |
| sdake | Jeffrey4l jenkins is -1 on the patch | 08:56 |
| sdake | because it is checking against the global requirements file | 08:56 |
| Jeffrey4l | oh, sorry. hmm | 08:57 |
| Jeffrey4l | let's revert it on newton branch after it is created. | 08:57 |
| sdake | that may work | 08:58 |
| Jeffrey4l | +2 for this Build source from Newton rather then Master https://review.openstack.org/#/c/384005/ | 08:58 |
| sdake | https://github.com/openstack/requirements/branches | 08:59 |
| sdake | there appears to be a newton branch | 08:59 |
| sdake | can you add your comment to the review Jeffrey4l | 08:59 |
| Jeffrey4l | sdake, you need abandon your PS and push a new one to newton branch when it is created. | 09:00 |
| sdake | yes - i keep stuff in queue as reminder of todo | 09:00 |
| Jeffrey4l | Cool | 09:00 |
| Jeffrey4l | i will -w for you PS with some comment. | 09:00 |
| sdake | add a note there so my brain doesn't forget ;) | 09:00 |
| Jeffrey4l | there is a job for requirements check : gate-kolla-requirements, the branch name should be hardcode in its job. | 09:02 |
| sdake | yup i understand | 09:03 |
| *** dwalsh has joined #openstack-kolla | 09:13 | |
| *** tonanhngo has joined #openstack-kolla | 09:13 | |
| *** tonanhngo has quit IRC | 09:16 | |
| sdake | well i'm off to bed | 09:19 |
| sdake | later | 09:19 |
| *** sdake has quit IRC | 09:24 | |
| *** mewald has joined #openstack-kolla | 09:41 | |
| *** tonanhngo has joined #openstack-kolla | 09:49 | |
| *** salv-orlando has quit IRC | 09:50 | |
| *** tonanhngo has quit IRC | 09:51 | |
| *** zhurong has quit IRC | 10:00 | |
| *** neilus has joined #openstack-kolla | 10:05 | |
| *** neilus has quit IRC | 10:10 | |
| *** msimonin1 has joined #openstack-kolla | 10:11 | |
| *** msimonin has quit IRC | 10:14 | |
| *** tonanhngo has joined #openstack-kolla | 10:19 | |
| *** tonanhngo has quit IRC | 10:22 | |
| *** tonanhngo has joined #openstack-kolla | 10:48 | |
| *** dwalsh has quit IRC | 10:49 | |
| *** tonanhngo has quit IRC | 10:49 | |
| *** daneyon has joined #openstack-kolla | 11:01 | |
| *** daneyon has quit IRC | 11:07 | |
| *** tonanhngo has joined #openstack-kolla | 11:17 | |
| *** MarMat has joined #openstack-kolla | 11:18 | |
| *** tonanhngo has quit IRC | 11:18 | |
| *** kuntumtarm has joined #openstack-kolla | 11:29 | |
| openstackgerrit | prameswar proposed openstack/kolla: dependency packages added that need to be installed before ansible installation https://review.openstack.org/384073 | 11:32 |
| *** tonanhngo has joined #openstack-kolla | 11:43 | |
| *** tonanhngo has quit IRC | 11:46 | |
| *** msimonin has joined #openstack-kolla | 12:01 | |
| *** zhurong has joined #openstack-kolla | 12:03 | |
| *** msimonin1 has quit IRC | 12:03 | |
| *** kuntumtarm has quit IRC | 12:12 | |
| *** zhurong has quit IRC | 12:30 | |
| *** haplo37_ has quit IRC | 12:41 | |
| *** haplo37_ has joined #openstack-kolla | 12:43 | |
| *** liyifeng has quit IRC | 12:45 | |
| *** mewald has quit IRC | 12:49 | |
| *** daneyon has joined #openstack-kolla | 12:50 | |
| *** daneyon has quit IRC | 12:54 | |
| *** dave-mccowan has joined #openstack-kolla | 12:57 | |
| *** sdake has joined #openstack-kolla | 13:19 | |
| *** sdake_ has joined #openstack-kolla | 13:22 | |
| *** sdake has quit IRC | 13:24 | |
| *** Pavo has joined #openstack-kolla | 13:30 | |
| *** sdake_ has quit IRC | 13:43 | |
| *** dwalsh has joined #openstack-kolla | 13:47 | |
| *** matrohon has quit IRC | 13:49 | |
| *** matrohon has joined #openstack-kolla | 13:50 | |
| *** tonanhngo has joined #openstack-kolla | 13:52 | |
| *** sdake has joined #openstack-kolla | 13:55 | |
| *** tonanhngo has quit IRC | 13:57 | |
| sdake | Jeffrey4l are ou still about | 14:04 |
| *** msimonin has quit IRC | 14:10 | |
| *** dave-mccowan has quit IRC | 14:20 | |
| *** msimonin has joined #openstack-kolla | 14:33 | |
| *** msimonin has quit IRC | 14:34 | |
| *** msimonin has joined #openstack-kolla | 14:36 | |
| *** sdake has quit IRC | 14:37 | |
| *** sdake has joined #openstack-kolla | 14:38 | |
| *** rhallisey has joined #openstack-kolla | 14:41 | |
| HyperJohnGraham_ | Morning all | 14:41 |
| *** awiddersheim has joined #openstack-kolla | 14:42 | |
| HyperJohnGraham_ | sbezverk: i rebuilt docker with btrfs and now have a stable k8s canal running http://paste.openstack.org/show/584978/ | 14:46 |
| *** dwalsh has quit IRC | 14:49 | |
| otavio | sdake: I reported the bug for RDO people; now awaiting for their feedback | 14:51 |
| sdake | otavio cool | 14:51 |
| sdake | otavio say this uid bug | 14:51 |
| sdake | you reported it from 2.0.2->2.0.3 | 14:51 |
| sdake | i am unable to reproduce that | 14:51 |
| sdake | although your report has opened up a can of worms for 2.0.2->3.0.0 that I can reproduce | 14:52 |
| otavio | sdake: our deploy was quite small; dunno if it may have change | 14:52 |
| sdake | otavio how did you get 2.0.3 exactly? | 14:52 |
| sdake | since it is unreleased ;) | 14:52 |
| otavio | sdake: bear in mind we are starting on this | 14:52 |
| otavio | sdake: we built it from kolla | 14:53 |
| sdake | otavio roger dude - looks like your off to a good start ;) | 14:53 |
| sdake | you build from git you mean? | 14:53 |
| sdake | the 2.0.3 images? | 14:53 |
| otavio | sdake: as we did for 3.0 | 14:53 |
| otavio | sdake: yes | 14:53 |
| sdake | wait a minute | 14:53 |
| sdake | the bug report said 2.0.2->2.0.3 | 14:53 |
| sdake | you just said 3.0 | 14:53 |
| sdake | which versions we talking about :) | 14:53 |
| otavio | sdake: we did not use prebuilt images. We used kolla-build | 14:53 |
| sdake | ok | 14:54 |
| sdake | so you pip pinstalled kolla- then built kolla 2.0.2 | 14:54 |
| otavio | sdake: we installed 3.0 only today so we were using 2.0.2 | 14:54 |
| otavio | sdake: yes | 14:54 |
| sdake | then pip removed kolla and then build kolla 2.0.3 from stable/newton branch? | 14:54 |
| otavio | sdake: no; from stable/mitaka | 14:54 |
| sdake | sorry thats what i meant | 14:55 |
| sdake | did you do the pip uninstall kolla part? | 14:55 |
| otavio | sdake: but we played a lot so we may have end building a mess | 14:55 |
| otavio | sdake: I think we did | 14:55 |
| sdake | you will neeed to do so | 14:55 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod https://review.openstack.org/383922 | 14:56 |
| *** zhubingbing has joined #openstack-kolla | 14:58 | |
| zhubingbing | sup sdake | 14:58 |
| sdake | hey zhubingbing | 14:59 |
| zhubingbing | your area is moring right? | 15:00 |
| otavio | sdake: btw, how do you develop? do you install it every time for test? | 15:00 |
| zhubingbing | develop? | 15:01 |
| otavio | zhubingbing: git workflow | 15:01 |
| zhubingbing | i every time for test | 15:02 |
| zhubingbing | git fetch path pip install kolla | 15:02 |
| zhubingbing | test it | 15:02 |
| zhubingbing | - ) | 15:02 |
| otavio | zhubingbing: pip uninstall kolla; pip install . ? | 15:02 |
| zhubingbing | yeah | 15:02 |
| otavio | ok | 15:02 |
| zhubingbing | pip uninall kolla/ ; pip install kolla/ | 15:03 |
| zhubingbing | pip install . have problems | 15:03 |
| zhubingbing | who can help me | 15:05 |
| zhubingbing | https://review.openstack.org/#/c/383080/ | 15:05 |
| zhubingbing | help review it | 15:05 |
| zhubingbing | -) -) | 15:06 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Enable release notes translation https://review.openstack.org/383227 | 15:06 |
| *** sdake_ has joined #openstack-kolla | 15:10 | |
| sdake_ | zhenguo its 8am here | 15:11 |
| sdake_ | otavio what do yo umean? | 15:11 |
| sdake_ | otavio I rebuild daily | 15:11 |
| sdake_ | otavio rebuild the world that is | 15:11 |
| sdake_ | then I develop from master | 15:11 |
| sdake_ | or develop from stable/newton or whatever | 15:11 |
| sdake_ | i rarely pip install unless testing upgrades | 15:11 |
| *** AJaeger has left #openstack-kolla | 15:11 | |
| sdake_ | otavio you don't want to do pip install . | 15:12 |
| sdake_ | otavio i run from git, and then run tools/build.py or tools/kolla-ansible deploy etc | 15:12 |
| sdake_ | otavio it puts a little extra burden on the dev because of the PBR problem | 15:12 |
| sdake_ | to know two sets of commands instead of one | 15:12 |
| sdake_ | but really there are only two comands | 15:13 |
| sdake_ | build and deploy | 15:13 |
| sdake_ | zhubingbing its 8am here | 15:13 |
| *** sdake has quit IRC | 15:13 | |
| sdake_ | any core reviewers around - this needs to go in stat -> https://review.openstack.org/#/c/384005/ | 15:15 |
| openstackgerrit | Merged openstack/kolla: fix can't collect horizon log in heka https://review.openstack.org/383080 | 15:16 |
| sdake_ | 21 critical bugs | 15:17 |
| *** sdake_ is now known as sdake | 15:17 | |
| sdake | wtb dev ;) | 15:17 |
| zhubingbing | -) | 15:17 |
| sdake | otavio the problem is worse then you reported | 15:18 |
| sdake | otavio all of ubuntu upgrades are broken | 15:18 |
| sdake | not that i use or test ubuntu | 15:18 |
| sdake | otavio because ubuntu added systemd | 15:18 |
| zhubingbing | i have test ubuntu | 15:19 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Allow cinder-volume to be configured to use nfs https://review.openstack.org/279913 | 15:19 |
| zhubingbing | base system use ubuntu or kolla use ubuntu? | 15:20 |
| zhubingbing | https://bugs.launchpad.net/kolla/+bug/1625705 sdake this bug you confirm ? | 15:21 |
| openstack | Launchpad bug 1625705 in kolla "latest kolla/ubuntu-binary-heka:3.0.0 container keep on restarting " [Critical,Incomplete] | 15:21 |
| *** salv-orlando has joined #openstack-kolla | 15:25 | |
| sdake | zhubingbing was not able to confirm that | 15:28 |
| sdake | zhubingbing I left a note in the bug log speculating as to the root cause | 15:28 |
| zhubingbing | ok | 15:28 |
| zhubingbing | roger | 15:29 |
| *** salv-orlando has quit IRC | 15:29 | |
| zhubingbing | i will try reproduce this bug | 15:29 |
| sdake | zhubingbing work on triaged bugs first plz | 15:31 |
| sdake | zhubingbing this cat has 8gb ram | 15:32 |
| sdake | he has an OOM problem or OOD problem I am pretty sure | 15:32 |
| zhubingbing | ok | 15:32 |
| sdake | or eveyrone would be yelling about kolla not working | 15:32 |
| zhubingbing | right | 15:32 |
| sdake | we have 3 separate people in that bug report that say they can't reproduce the problem | 15:32 |
| *** daneyon has joined #openstack-kolla | 15:32 | |
| sdake | either work on getting triaged -> some other state | 15:33 |
| sdake | or fixing confirmed bugs of which there are plenty | 15:33 |
| zhubingbing | understand | 15:35 |
| *** daneyon has quit IRC | 15:37 | |
| sdake | zhubingbing whats the story with this review: https://review.openstack.org/#/c/375380/7/ansible/roles/gnocchi/templates/gnocchi-api.json.j2 | 15:44 |
| *** eaguilar has joined #openstack-kolla | 15:44 | |
| sdake | paul asked a question and i have same question | 15:44 |
| sdake | why change that? | 15:44 |
| sdake | what specifically is broken in bug: https://bugs.launchpad.net/kolla/+bug/1626959 | 15:45 |
| openstack | Launchpad bug 1626959 in kolla "change perm 06400 to 0600 in gnocchi-api.json.j2" [High,In progress] - Assigned to zhubingbing (zhubingbing) | 15:45 |
| sdake | you list a solution not a problem | 15:45 |
| zhubingbing | emm | 15:45 |
| sdake | zhubingbing ^^ | 15:45 |
| zhubingbing | because gnocchi api use http_mod_wsgi | 15:46 |
| zhubingbing | to start | 15:46 |
| sdake | could you expand on what happens without this change | 15:46 |
| sdake | zhubingbing order should be 1) inprogress bugs your working on 2) triage->to any other state 3) high/critical bugs unassigned assign | 15:48 |
| zhubingbing | 1） understand | 15:49 |
| zhubingbing | 2) don‘t quite understand | 15:50 |
| sdake | zhubingbing i marked https://bugs.launchpad.net/kolla/+bug/1626959 incomplete | 15:50 |
| openstack | Launchpad bug 1626959 in kolla "change perm 06400 to 0600 in gnocchi-api.json.j2" [High,Incomplete] - Assigned to zhubingbing (zhubingbing) | 15:50 |
| zhubingbing | any other status/? | 15:50 |
| sdake | 2 we want to get bugs in triaged state to some other state such as confirmed invalid or incomplete | 15:51 |
| zhubingbing | ok, understand | 15:51 |
| sdake | but irst we want to get inprogress bugs we are working on already fixed | 15:52 |
| sdake | and finally we want to work on high/critical bugs | 15:52 |
| sdake | some of the cores may not follow this pattern - this is fine - this pattern is more for new folks or people ocming up to speed | 15:52 |
| zhubingbing | I know, we should be organized to deal with these bug | 15:53 |
| sdake | zhubingbing re https://bugs.launchpad.net/kolla/+bug/1631041 | 15:57 |
| openstack | Launchpad bug 1631041 in kolla "Keystone log repeated collection in heka" [Medium,In progress] - Assigned to zhubingbing (zhubingbing) | 15:57 |
| sdake | heka-keystone.taml or whatever its called is the lua loggingfor keystone | 15:57 |
| sdake | heka-openstack is the definition of all those things combined | 15:57 |
| sdake | are you actually seeing keystone logged twice? | 15:57 |
| kfox1111_ | down to this weird cinder attach issue.... so weird... | 15:58 |
| kfox1111_ | http://logs.openstack.org/41/381041/94/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/ebd9451/logs/openstack/kolla-nova-libvirt-w20kj.txt | 15:58 |
| zhubingbing | heke monitor kolla_logs/keystone file | 15:59 |
| *** awiddersheim has quit IRC | 16:01 | |
| sdake | zhubingbing are the logs sent twice to log files via heka? | 16:01 |
| zhubingbing | heka-keystone.taml have collect one times and heka-openstack collect keystone log one times | 16:02 |
| sdake | right, but does that mean the log file is written with two lines? | 16:02 |
| sdake | any cores around - the gate is jammed | 16:03 |
| sdake | need some help there | 16:03 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 16:03 |
| zhubingbing | and 2 files content are the same. | 16:03 |
| rhallisey | sdake, ya here | 16:04 |
| rhallisey | which patch | 16:04 |
| sdake | rhallisey moment | 16:05 |
| sdake | https://review.openstack.org/#/c/384005/ | 16:05 |
| zhubingbing | [root@kolla-test-1 logstreamer]# diff keystone-apache-admin keystone_keystone-apache-admin-access | 16:05 |
| zhubingbing | [root@kolla-test-1 logstreamer]# | 16:05 |
| sdake | i see so writing to two files? | 16:06 |
| sdake | with your patch, which file does it write to? | 16:06 |
| openstackgerrit | Merged openstack/kolla: Build source from Newton rather then Master https://review.openstack.org/384005 | 16:09 |
| zhubingbing | we have use heka-keystone.toml why also use heka-openstack | 16:09 |
| zhubingbing | I mean, keystone has been defined by heka for 2 times. | 16:10 |
| zhubingbing | heka-keystone.taml and heka-openstack | 16:10 |
| *** dwalsh has joined #openstack-kolla | 16:12 | |
| *** tonanhngo has joined #openstack-kolla | 16:16 | |
| *** huikang has joined #openstack-kolla | 16:21 | |
| *** huikang has quit IRC | 16:22 | |
| hogepodge | I don't entirely understand how to do a custom configuration in kolla | 16:22 |
| *** huikang has joined #openstack-kolla | 16:22 | |
| hogepodge | sdake: say, for example, I want to enable http store for glance along with rdb, when and where do I update the config glance config files? | 16:23 |
| sdake | rhallisey can you do me a solid and read the irc logs from last night | 16:23 |
| *** inc0 has joined #openstack-kolla | 16:23 | |
| inc0 | heyh | 16:23 |
| inc0 | sdake, | 16:23 |
| inc0 | wassup? | 16:24 |
| sdake | moment i'll find link for both of you to read | 16:24 |
| rhallisey | k | 16:24 |
| sdake | hogepodge moment pls | 16:24 |
| sdake | hogepodge there are others around that can answer your q - need to find link for inc0/rhallisey to read through | 16:24 |
| *** salv-orlando has joined #openstack-kolla | 16:26 | |
| *** kuntumtarm has joined #openstack-kolla | 16:26 | |
| sdake | here is where it begins inc0 and rhallisey http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2016-10-08.log.html#t2016-10-08T06:14:49 | 16:26 |
| sdake | the solution is towards the end of our discussion | 16:26 |
| sdake | note this affects EVERYTHING | 16:28 |
| sdake | that uses a named volume | 16:28 |
| sdake | not just rabbitmq | 16:28 |
| inc0 | isn't that bug in docker? | 16:29 |
| sdake | hogepodge mkdir /etc/kolla/config | 16:29 |
| sdake | hogepodge put in ceph.conf your configuration stuff | 16:29 |
| inc0 | like USER rabbitmq != USER rabbitmq if we redeploy | 16:29 |
| sdake | inc0 nah - it depends on the order its built in | 16:29 |
| sdake | and what packages install which dependencies that install users/groups | 16:30 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use mariadb.cnf for mariadb custom file name https://review.openstack.org/384043 | 16:30 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: integrate gnocchi with ceilometer https://review.openstack.org/372737 | 16:30 |
| sdake | hogepodge note i'm not sure ceph is customizable (if its ini file it is, if not it isn't) | 16:30 |
| inc0 | so hold on, if it gets UID/GID during buildtime | 16:30 |
| *** salv-orlando has quit IRC | 16:31 | |
| inc0 | hogepodge, what's the issue? | 16:31 |
| rhallisey | I see | 16:31 |
| sdake | inc0 he wants to customize ceph | 16:31 |
| inc0 | customize how? | 16:31 |
| *** tonanhngo has quit IRC | 16:32 | |
| sdake | zhubingbing you didn't answer my question, the question was with your patch is there only one file, or is there two. if only one file, what is it called with your patch? | 16:32 |
| inc0 | and yes, ceph.conf can be customized by merge_config | 16:32 |
| inc0 | ok back to the rabbit issue | 16:32 |
| hogepodge | inc0: if I enable rdb as the store for glance, it writes the glance conf files like this: http://paste.openstack.org/show/584984/ | 16:33 |
| sdake | cool where there ya go hogepodge :) | 16:33 |
| sdake | hogepodge do you have enable_ceph: yes set? | 16:33 |
| hogepodge | inc0: which is fine, except for normally glance has the line 'stores = rdb' configured to 'stores = file,http' | 16:33 |
| *** tonanhngo has joined #openstack-kolla | 16:33 | |
| hogepodge | sdake: inc0: so I'd like that line to read 'stores = rdb,http' | 16:33 |
| inc0 | hogepodge, ok, easy enough, let me link you doc for it | 16:34 |
| sdake | hogepodge mind explaining what the http part does - perhaps that should go in our defaults | 16:34 |
| sdake | hogepodge nm - i misread your last statement | 16:34 |
| hogepodge | sdake: it lets glance pull images from remote locations. quite a few heat templates depend on it | 16:35 |
| inc0 | hogepodge, https://github.com/openstack/kolla/blob/master/doc/advanced-configuration.rst#openstack-service-configuration-in-kolla | 16:35 |
| sdake | hogepodge can you file a bug for that problem please | 16:35 |
| sdake | i think that needs to go in our defaults | 16:35 |
| inc0 | this method allows you to override any option in .ini confs | 16:35 |
| inc0 | in this case you need override glance:) | 16:35 |
| sdake | hogepodge glance is overridable | 16:35 |
| hogepodge | I find those docs confusing. If I'm doing multi-node, do I drop the conf on the system I run kolla-ansible from, or do I have to change the configs that are dropped on the target nodes? | 16:36 |
| sdake | rhallisey - so good with the solution - understand problem? | 16:36 |
| inc0 | hogepodge, all in deployment node | 16:36 |
| inc0 | I'll make a note to clarify that in docs | 16:36 |
| sdake | hogepodge yes - docs are #1, #2, #3 painpoints for all of our operators | 16:36 |
| zhubingbing | sdake Sorry, I may have made a mistake, the bug description is not accurate enough. | 16:36 |
| sdake | hogepodge you dont have to do any of that, kolla has single source of truth (the deployment node contains the single source) | 16:37 |
| rhallisey | sdake, ya I"m fine with the solution. O is where we'll have to fix it everywhere | 16:37 |
| inc0 | hogepodge, in general you shouldn't need to touch any node besides deployment | 16:37 |
| sdake | hogepodge you can poke around and hack stuff manually - that is what CONFIG_STRATEGY is about | 16:37 |
| inc0 | ansible does all the rest | 16:37 |
| sdake | rhallisey i am pretty happy with the docker run option jeffrey and I came up with | 16:38 |
| sdake | did you get that far into the discussion | 16:38 |
| sdake | rhallisey we need to fix it everywhere now unfortunately | 16:38 |
| sdake | because ubuntu upgrade is completely broken | 16:39 |
| sdake | because systemd is colliding with 2.0.2 container uids | 16:39 |
| sdake | RTL :) | 16:39 |
| hogepodge | I see, so I don't need the entire file, just the group and option I want changed, and kolla merges, right? | 16:40 |
| sdake | hogepodge you got it | 16:40 |
| sdake | hogepodge pretty sweet huh :) | 16:40 |
| sdake | no turnaround on dev/test to add custom config | 16:40 |
| sdake | no kolla making policy decisions for operators | 16:40 |
| HyperJohnGraham_ | Hi All... whats the story with setup_gate.sh ? ( just did a git pull :) I am rebuilding from scratch and it looks like there are lots of cool changes in the install process | 16:41 |
| sdake | HyperJohnGraham_ - kfox1111_ has indicated there is a cinder blocker | 16:41 |
| HyperJohnGraham_ | so i should start building a wall ?? | 16:41 |
| sdake | pretty much ;) | 16:42 |
| HyperJohnGraham_ | :){ | 16:42 |
| kfox1111_ | sbezverk: ceph osd blacklist add <--- this looks interesting too. | 16:42 |
| hogepodge | sdake: that's cool, thanks. I'll do a reconfigure | 16:42 |
| rhallisey | sdake, how is it that rabbit is the only problem now | 16:42 |
| sdake | rhallisey its the only reported problem | 16:42 |
| sdake | nobody clearly has tested ubuntu 2.0.2 to 3.0.0 | 16:43 |
| sdake | an upgrade thereof | 16:43 |
| rhallisey | HyperJohnGraham, setup_gate script will be run by the gate. It's not meant to be run locally | 16:43 |
| rhallisey | sdake, would a doc of the docker exec for this b ok? | 16:43 |
| rhallisey | until its properly fixed in O | 16:44 |
| sdake | i expect ubuntu 2.0.2 to 3.0.0 would implode in bad ways | 16:44 |
| sdake | rhallisey no a doc wouldn't do the job | 16:44 |
| rhallisey | or do you intend to add those into the upgrade playbook | 16:44 |
| sdake | rhallisey because the upgrade would stop and crater | 16:44 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 16:44 |
| kfox1111_ | I wonder... | 16:44 |
| sdake | i intend to add a run command to the upgrade playbook | 16:44 |
| *** zhubingbing has quit IRC | 16:44 | |
| sdake | docker exec rather | 16:44 |
| sdake | keep scrolling rhallisey - read down more ;) | 16:44 |
| rhallisey | I'm at the point you go to sleep | 16:45 |
| sdake | docker run thatis | 16:45 |
| sdake | right - so right above that jeffrey proposes a soltuion i had in my head | 16:45 |
| sdake | which is to mount the persistent volumes and chown them via a shell script | 16:45 |
| sdake | then we can remove that crap in ocata | 16:45 |
| sdake | rather when newton goes eol | 16:45 |
| inc0 | sdake, Jeffrey4l UID issue...you're saying that UID gets created on container build? | 16:46 |
| inc0 | really? | 16:46 |
| rhallisey | yes | 16:46 |
| sdake | inc0 yes for packages installed via apt/yum | 16:46 |
| Jeffrey4l | inc0, it is. | 16:46 |
| rhallisey | inc0, docker fs works that way | 16:46 |
| inc0 | so rabbimq package creates user? | 16:46 |
| sdake | inc0 right | 16:46 |
| inc0 | not USER clausule | 16:46 |
| inc0 | ? | 16:46 |
| rhallisey | little think rw layer on top | 16:46 |
| rhallisey | thin* | 16:47 |
| sdake | as well as keepalived, etc | 16:47 |
| inc0 | let's discuss one Kolla user then | 16:47 |
| inc0 | because that seems wrong | 16:47 |
| Jeffrey4l | all packages ( including rpm/deb) may create user | 16:47 |
| sdake | one kolla user is insecure | 16:47 |
| sdake | also it wont work | 16:47 |
| inc0 | seems like major docker issue to me | 16:48 |
| sdake | inc0 no shit | 16:48 |
| inc0 | how about GID? | 16:48 |
| rhallisey | this is kinds neat: docker run -it -v nova:/var/lib/nvoa -v glance:/var/lib/glance -v ceph:/var/lib/ceph ... --rm centos-source-base:3.0.0 | 16:48 |
| sdake | gid same story | 16:48 |
| rhallisey | that replaces the underlying layers | 16:48 |
| sdake | rhallisey right | 16:48 |
| rhallisey | but do the running contianers get it | 16:48 |
| rhallisey | doubt it | 16:48 |
| inc0 | can we create group for kolla just for perm and add users created to this group? | 16:48 |
| rhallisey | I guess they woudl be restarted thouhg | 16:48 |
| rhallisey | so they would | 16:48 |
| sdake | inc0 that is massive rework | 16:49 |
| rhallisey | first step of upgrade process | 16:49 |
| inc0 | well, not necesarly, we could do this in kolla_start | 16:49 |
| sdake | how do you end up chowning /etc/nova for example in kolla_start? | 16:50 |
| sdake | or chmoding to 660 | 16:50 |
| inc0 | chmoding | 16:50 |
| sdake | which reduces security along the way | 16:50 |
| sdake | we have json to set permissions not kolla_start | 16:50 |
| sdake | that would totally crater our abi | 16:50 |
| inc0 | well, set_config file | 16:51 |
| sdake | right - massive rework across all json files | 16:51 |
| *** awiddersheim has joined #openstack-kolla | 16:51 | |
| sdake | not set_configs.py | 16:51 |
| Jeffrey4l | /etc/nova should be fine. But named volumes should be took care. | 16:51 |
| rhallisey | awiddersheim, I updated that patch this morning | 16:51 |
| rhallisey | I think it should be good to go | 16:52 |
| awiddersheim | sounds good | 16:52 |
| sdake | Jeffrey4l right named volumes | 16:52 |
| sdake | how to chmod named volumes | 16:52 |
| inc0 | let me read some internets about it | 16:52 |
| Jeffrey4l | chmod? chown? | 16:52 |
| sdake | the options are docker run as jeffrey proposed for extend_start | 16:52 |
| inc0 | other people had to have same issue | 16:52 |
| sdake | inc0 i guess if they upgraded to master from 2.0.2 | 16:53 |
| sdake | inc0 which i'm guessing hasn't happened | 16:53 |
| Jeffrey4l | inc0, if he doesn't use docker/container, he haven't such issue. | 16:53 |
| Jeffrey4l | because the /etc/passwd file is not changed during upgrade. | 16:53 |
| Jeffrey4l | i.e. rabbitmq user will not re-create with new uid/gid when upgrading rabbitmq | 16:54 |
| sdake | inc0 this question is answered in the irc log | 16:54 |
| sdake | last day to vote for tc election for stragglers ;) | 16:55 |
| sdake | inc0 not many people use our pattern of dependent packages with two base images | 16:56 |
| inc0 | so let's narrow down what's impacted | 16:56 |
| sdake | inc0 anything with a named volume is impacted on ubuntu | 16:56 |
| inc0 | 1. containers that installs apt package that creates users | 16:56 |
| sdake | inc0 rabbitmq is impacted on centos | 16:57 |
| inc0 | 2. containers that uses named volume | 16:57 |
| sdake | we don't care about #1, it doesns't impact upgrade | 16:57 |
| inc0 | we do, as when you build new containers they create user with potentially different gid | 16:57 |
| sdake | we being i:) | 16:58 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 16:58 |
| inc0 | that's the reason it is different and messes up perms | 16:58 |
| sdake | inc0 right but this isn't a problem unless that uid is shared | 16:58 |
| sdake | uids are only sahred on named volumes | 16:58 |
| inc0 | it is kinda shared in perms | 16:58 |
| inc0 | so we need to have it synced | 16:58 |
| sdake | we definately need to have named volume uids/gids synced in base | 16:59 |
| inc0 | base is rebuilt too | 16:59 |
| inc0 | forcing uid/gid in containers like rabbitmq would work | 16:59 |
| sdake | right - which means we need to chown the named volumes on upgrade via docker run | 16:59 |
| inc0 | not necessarly, we can also just force uid during build time | 17:00 |
| inc0 | to ensure that rabbitmq will get uid 950 no matter how many times you rebuild | 17:00 |
| sdake | you didn't read log or you would know this results in an entire 6 months of maintaining something that is not ideal | 17:00 |
| sdake | that wont work | 17:00 |
| sdake | uid in 2.0.2 = 195 | 17:00 |
| sdake | if uid in 3.0.0 is 950 | 17:01 |
| sdake | upgrade = crater | 17:01 |
| sdake | uid in 3.0.0 must either be 950 or docker run must chown the named volumes during upgrade playbook | 17:01 |
| sdake | and it must be in base | 17:01 |
| sdake | because uids are reused in different containers if not already in e/tc/passwd | 17:01 |
| sdake | there is no global uid mapping | 17:02 |
| sdake | docker may have added such a feature but it wont work for our use case | 17:02 |
| *** janem has joined #openstack-kolla | 17:02 | |
| sdake | solving for rabbitmq is easy | 17:02 |
| sdake | solving for ubuntu more difficult | 17:03 |
| sdake | because ubuntu has systemd | 17:03 |
| sdake | and systemd uses the same uids as all kinds of stuff in the named volumes | 17:03 |
| sdake | (from 2.0.2) | 17:03 |
| inc0 | chowning on docker run might not work as we run as underprivileged user | 17:04 |
| inc0 | you can't change your uid on the fly | 17:04 |
| sdake | inc0 if you read the log - Jeffrey4l proposed -u root | 17:04 |
| sdake | R..T...L... | 17:05 |
| *** eaguilar has quit IRC | 17:05 | |
| sdake | we burned up 3 hours brainstorming last night to solve this problem | 17:05 |
| sdake | its captured in the log | 17:05 |
| *** dwalsh has quit IRC | 17:05 | |
| sdake | if you have something further to add to the brainstorming - best wayt o catch up is spend 1 0 minutes reading the log | 17:05 |
| sdake | 1 0/10 | 17:05 |
| sdake | upgrade is our greatest success, we can't crater it in 3.0.0 | 17:07 |
| sdake | rhallisey do you understand the issue and the solution? | 17:08 |
| * rhallisey reads back | 17:09 | |
| sdake | rhallisey the one in the log i posted | 17:09 |
| rhallisey | yes | 17:09 |
| rhallisey | the one you posted | 17:09 |
| inc0 | will mariadb be affected too? | 17:09 |
| *** g3ek has quit IRC | 17:09 | |
| sdake | on ubuntu - yup | 17:09 |
| sdake | on centos - nope | 17:10 |
| *** haplo37 has quit IRC | 17:10 | |
| *** tonanhngo has quit IRC | 17:10 | |
| inc0 | so issue with base is - we don't know uid/gid at build time | 17:10 |
| sdake | 1. we assign it starting at 500 | 17:11 |
| Jeffrey4l | inc0, and uid/gid is not the same in 2.0.2 and 3.0.0 | 17:11 |
| *** g3ek has joined #openstack-kolla | 17:11 | |
| sdake | 2. we run docker run with named volumes mounted -u root upgrade-uids.sh | 17:11 |
| inc0 | so unless we force it in 3.0.0, same issue will appear in 4.0.0 | 17:11 |
| sdake | upgra-uids.sh has all the chown work in it | 17:12 |
| sdake | yes the proposal is forcing it | 17:12 |
| *** haplo37 has joined #openstack-kolla | 17:12 | |
| Jeffrey4l | correct. it will also happen like rabbitmq case. | 17:12 |
| Jeffrey4l | during upgrade from 2.0.2 to 2.0.3 | 17:12 |
| inc0 | ok...yeah, that will work and will ensure uids in 3.0.0 | 17:12 |
| inc0 | we can backport it to stable as far as I'm concerned | 17:12 |
| inc0 | it's major issue | 17:12 |
| sdake | because #2 doesn't need to know the uids ahead of time we are golden | 17:13 |
| sdake | lets get master fixed first ;) | 17:13 |
| inc0 | so we need to figure out full list of users created in containers.. | 17:13 |
| sdake | only those with named volumes | 17:14 |
| sdake | unless we want to do across code base | 17:14 |
| sdake | which wfm | 17:14 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH https://review.openstack.org/326307 | 17:14 |
| inc0 | if we could, we make our life easier for N->O upgrade across the board | 17:14 |
| sdake | agree | 17:14 |
| sdake | since we dont have to map them ahead of time - only determine what the uids are | 17:14 |
| sdake | it should be a little easier | 17:15 |
| inc0 | we can map them ahead of time tho | 17:15 |
| inc0 | in base | 17:15 |
| sdake | right - that was the original proposal ;) | 17:15 |
| inc0 | and then chown | 17:15 |
| inc0 | that's good solution imho | 17:15 |
| inc0 | we can chown in upgrade playbook | 17:15 |
| sdake | yes - agree - so thats 3 people ;) | 17:15 |
| sdake | enough +2s to work with :) | 17:16 |
| sdake | rhallisey when your cuaght up let us know what you think | 17:16 |
| rhallisey | I'm caught up | 17:16 |
| rhallisey | sounds good to me | 17:16 |
| inc0 | so first big thing -> get all users ever created across the board and assign gid to them | 17:17 |
| inc0 | uid* | 17:17 |
| sdake | and gid | 17:17 |
| inc0 | put this perpared /etc/password file in base | 17:17 |
| inc0 | and gid | 17:17 |
| sdake | we can use useradd commands rather then prepared passwd file | 17:17 |
| sdake | passwd file may change in base image on a rebuild | 17:18 |
| inc0 | and in upgrade playbook if your volume uid/gid != your uid/gid when it should in config.json, chown | 17:18 |
| sdake | the base one from centos:7 eg | 17:18 |
| inc0 | yeah, useradd is better ofc | 17:18 |
| Jeffrey4l | we need care about the user home when creating, useradd --user-group --create-home --home-dir /var/lib/nova nova | 17:18 |
| sdake | Jeffrey4l make it harder ;) | 17:19 |
| rhallisey | we only need to fix one thing for N | 17:20 |
| rhallisey | right? | 17:20 |
| rhallisey | just rabbit | 17:20 |
| sdake | rhallisey no all things | 17:20 |
| sdake | see this review for the useradd logic and groupadd logic | 17:20 |
| sdake | i dislike it | 17:20 |
| sdake | but i don't see any cli options to do it better | 17:20 |
| sdake | https://review.openstack.org/#/c/384029/ | 17:20 |
| rhallisey | what happend to the docker run solution? | 17:21 |
| sdake | Jeffrey4l just think how much easier life would be with only cnetos | 17:21 |
| sdake | rhallisey ya i think that is best rather then jerking around with extend_start | 17:22 |
| Pavo | yes centos is the way to go :) | 17:22 |
| rhallisey | you could have the docker run solution as part of the upgrade playbook for N | 17:22 |
| rhallisey | first step run for each service | 17:22 |
| sdake | rhallisey thats the plan | 17:22 |
| sdake | rhallisey i was thinking we would do it once | 17:22 |
| sdake | with all the named volumes mounted against the chown_uids.sh script | 17:23 |
| Jeffrey4l | life is tough ;) | 17:23 |
| Jeffrey4l | centos will also have such issue, if centos image add some thing which crated a new user. | 17:24 |
| sdake | Jeffrey4l you have no idea | 17:24 |
| *** tonanhngo has joined #openstack-kolla | 17:24 | |
| sdake | Jeffrey4l re review above, any ideas on better 1 liner for uid/gid addition? | 17:25 |
| sdake | nothing like a lasat minute fire drill to wake up the day :) | 17:26 |
| *** salv-orlando has joined #openstack-kolla | 17:26 | |
| Jeffrey4l | groupadd -g 500 && useradd --user-group --create-home --home-dir /var/lib/nova --uid 500 --gid 500 nova | 17:27 |
| Jeffrey4l | seems need two lines. | 17:27 |
| *** salv-orlando has quit IRC | 17:31 | |
| kfox1111_ | hmm... so its not the rbd mounts from the kernel mounts... | 17:31 |
| sbezverk | kfox1111_: hey, qq can in theory ceph volume has multiple locks? | 17:33 |
| kfox1111_ | if its a read only volume. | 17:36 |
| kfox1111_ | there can be only one writer lock, or potentially multiple readers. | 17:38 |
| *** eaguilar has joined #openstack-kolla | 17:39 | |
| *** awidders_ has joined #openstack-kolla | 17:40 | |
| *** awiddersheim has quit IRC | 17:41 | |
| sdake | locking101 :) | 17:42 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 17:43 |
| *** huikang has quit IRC | 17:44 | |
| sdake | hogepodge ok bug triaged | 17:44 |
| hogepodge | sdake: thanks, it's a pretty simple fix. I'd be happy to send the patch up | 17:44 |
| *** huikang has joined #openstack-kolla | 17:44 | |
| hogepodge | if you want to assign it to me | 17:44 |
| sdake | hogepodge all yours | 17:45 |
| *** huikang has quit IRC | 17:45 | |
| *** huikang has joined #openstack-kolla | 17:46 | |
| sbezverk | kfox1111_: I tested last night pod clening logic by killing compute nodes every few minutes and it works really well :-) | 17:46 |
| sbezverk | kfox1111_: I want to add ceph lock clearing, tired of doing it manually. | 17:46 |
| kfox1111_ | awesome. :) | 17:47 |
| *** huikang has quit IRC | 17:49 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 17:49 |
| inc0 | bbiab I need to run some errands | 17:50 |
| inc0 | will be back afternoon to fix some stuff:) | 17:50 |
| *** inc0 has quit IRC | 17:50 | |
| sdake | we have 9 confirmed high/critical bugs to fix | 17:54 |
| sdake | need to fix those | 17:54 |
| sdake | preferabbly before 12th | 17:54 |
| sdake | and about 20 inprogress bugs | 17:55 |
| sdake | those should be fixed too if there are patches in the queue | 17:55 |
| sdake | have some errands to run -then get my workstation in working order | 17:59 |
| sdake | fi someone gets to that uid bug before me it won't hurt my feelings | 17:59 |
| sdake | otherwise i'll have a patch up sunday | 17:59 |
| sdake | bbl | 17:59 |
| sdake | workstation/laptop | 17:59 |
| *** sdake has quit IRC | 17:59 | |
| *** salv-orlando has joined #openstack-kolla | 18:00 | |
| *** huikang has joined #openstack-kolla | 18:05 | |
| *** dwalsh has joined #openstack-kolla | 18:05 | |
| *** eaguilar has quit IRC | 18:06 | |
| *** salv-orlando has quit IRC | 18:08 | |
| *** salv-orlando has joined #openstack-kolla | 18:08 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 18:14 |
| *** daneyon has joined #openstack-kolla | 18:15 | |
| *** mewald has joined #openstack-kolla | 18:15 | |
| *** msimonin has quit IRC | 18:16 | |
| *** daneyon has quit IRC | 18:19 | |
| *** mewald has quit IRC | 18:21 | |
| *** tonanhngo has quit IRC | 18:29 | |
| *** tonanhngo has joined #openstack-kolla | 18:31 | |
| *** salv-orlando has quit IRC | 18:45 | |
| *** mewald has joined #openstack-kolla | 18:56 | |
| *** mewald has quit IRC | 18:57 | |
| *** awidders_ has quit IRC | 18:58 | |
| *** dwalsh has quit IRC | 19:00 | |
| *** eaguilar has joined #openstack-kolla | 19:03 | |
| *** msimonin has joined #openstack-kolla | 19:03 | |
| *** pbourke has quit IRC | 19:09 | |
| *** pbourke has joined #openstack-kolla | 19:09 | |
| *** msimonin has quit IRC | 19:10 | |
| *** msimonin1 has joined #openstack-kolla | 19:10 | |
| *** tonanhngo has quit IRC | 19:19 | |
| *** sdake has joined #openstack-kolla | 19:20 | |
| sdake | where doth zhubingbing go | 19:26 |
| sdake | Jeffrey4l you touched this last - how does this patch look to you https://review.openstack.org/#/c/382356/1/ansible/roles/horizon/templates/local_settings.j2 | 19:28 |
| Jeffrey4l | sdake, i think both public and intern should be fine in most case. | 19:31 |
| *** dave-mccowan has joined #openstack-kolla | 19:32 | |
| *** msimonin1 has quit IRC | 19:32 | |
| sdake | i dont know how that is used | 19:32 |
| sdake | why change it from public to internal? | 19:32 |
| sdake | Jeffrey4l ^ | 19:33 |
| Jeffrey4l | sdake, i leave a comment to ask the author why this need to be changed. | 19:33 |
| *** MarMat has quit IRC | 19:33 | |
| sdake | wfm | 19:33 |
| sdake | i'm sure berendt will get to it | 19:33 |
| *** jmccarthy has quit IRC | 19:33 | |
| *** jmccarthy has joined #openstack-kolla | 19:34 | |
| Jeffrey4l | public should be more useful. if the horizon node can not touch the internal network. sdake | 19:34 |
| Jeffrey4l | so i do not think we need such a change now. | 19:35 |
| *** msimonin has joined #openstack-kolla | 19:35 | |
| sdake | i agree | 19:35 |
| sdake | not sure why its needed | 19:35 |
| *** haplo37 has quit IRC | 19:36 | |
| *** g3ek has quit IRC | 19:39 | |
| *** awiddersheim has joined #openstack-kolla | 19:39 | |
| *** g3ek has joined #openstack-kolla | 19:41 | |
| *** haplo37 has joined #openstack-kolla | 19:45 | |
| *** Pavo has quit IRC | 19:45 | |
| sdake | rhallisey ping | 19:53 |
| sdake | rhallisey i gotta jet, but can you turn this into a bug report: https://ask.openstack.org/en/question/97154/why-start-heka-container-twice/ | 19:53 |
| rhallisey | sure | 19:53 |
| *** sdake has quit IRC | 19:54 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla: Remove unneeded Heka bootstrapping code https://review.openstack.org/384122 | 20:00 |
| *** dave-mccowan has quit IRC | 20:00 | |
| *** neilus has joined #openstack-kolla | 20:15 | |
| *** sdake has joined #openstack-kolla | 20:17 | |
| *** HyperJohnGraham_ has quit IRC | 20:17 | |
| *** awiddersheim has quit IRC | 20:17 | |
| *** sdake has quit IRC | 20:20 | |
| *** msimonin has quit IRC | 20:24 | |
| *** msimonin has joined #openstack-kolla | 20:28 | |
| *** Serlex has joined #openstack-kolla | 20:30 | |
| *** msimonin has quit IRC | 20:47 | |
| *** kuntumtarm has quit IRC | 20:48 | |
| *** msimonin has joined #openstack-kolla | 20:49 | |
| *** Pavo has joined #openstack-kolla | 20:49 | |
| *** sdake has joined #openstack-kolla | 20:52 | |
| *** sdake has quit IRC | 21:05 | |
| *** dwalsh has joined #openstack-kolla | 21:14 | |
| *** msimonin1 has joined #openstack-kolla | 21:15 | |
| *** msimonin has quit IRC | 21:15 | |
| *** msimonin1 has left #openstack-kolla | 21:15 | |
| *** Pavo has quit IRC | 21:16 | |
| *** Pavo has joined #openstack-kolla | 21:17 | |
| *** sdake has joined #openstack-kolla | 21:22 | |
| sdake | thanks rhallisey | 21:22 |
| sdake | you didn't have to fix the bug ust file it :) | 21:23 |
| *** huikang has quit IRC | 21:24 | |
| *** Pavo has quit IRC | 21:25 | |
| *** sdake has quit IRC | 21:33 | |
| *** Pavo has joined #openstack-kolla | 21:34 | |
| *** mewald has joined #openstack-kolla | 21:37 | |
| *** Pavo has quit IRC | 21:46 | |
| *** salv-orlando has joined #openstack-kolla | 21:59 | |
| mewald | Hi, I lost root access to my mariadb cluster. How do I recover that for a containerized mariadb? | 22:10 |
| *** Serlex has quit IRC | 22:10 | |
| *** eaguilar has quit IRC | 22:15 | |
| *** msimonin has joined #openstack-kolla | 22:17 | |
| *** neilus has quit IRC | 22:18 | |
| *** eaguilar has joined #openstack-kolla | 22:23 | |
| *** dwalsh has quit IRC | 22:27 | |
| *** schwicht has joined #openstack-kolla | 22:33 | |
| *** dave-mccowan has joined #openstack-kolla | 22:35 | |
| *** Pavo has joined #openstack-kolla | 22:42 | |
| *** daneyon has joined #openstack-kolla | 22:46 | |
| *** eaguilar has quit IRC | 22:48 | |
| *** daneyon has quit IRC | 22:50 | |
| *** schwicht has joined #openstack-kolla | 22:58 | |
| *** msimonin has quit IRC | 23:07 | |
| *** neilus has joined #openstack-kolla | 23:09 | |
| *** neilus has quit IRC | 23:14 | |
| *** dave-mccowan has quit IRC | 23:24 | |
| *** eaguilar has joined #openstack-kolla | 23:26 | |
| *** salv-orlando has quit IRC | 23:26 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod https://review.openstack.org/383922 | 23:45 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP fencing pod https://review.openstack.org/383922 | 23:47 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!