| *** lamt has quit IRC | 00:00 | |
| sdake | if cinder runs again /run/ceph | 00:00 |
|---|---|---|
| Jeffrey4l | agree | 00:00 |
| sdake | that could be a problem | 00:00 |
| sdake | drwxrwx---. 1 ceph ceph 0 Oct 13 00:52 ceph | 00:00 |
| sdake | *really* need an strace of the crashing tool | 00:01 |
| sdake | i dont see why bootstrap would access /run in any regard | 00:01 |
| *** Pavo has joined #openstack-kolla | 00:01 | |
| Jeffrey4l | i am trying to deploy this locally. | 00:01 |
| sdake | Jeffrey4l roger - i'll get off your back ;) | 00:02 |
| *** hogepodge has quit IRC | 00:02 | |
| sdake | so dudes I bought the coolest thing today | 00:02 |
| sdake | http://www.apmex.com/product/102909/2016-australia-1-kilo-silver-koala-proof?utm_source=nextag&utm_medium=cpc&utm_campaign=shopping_engine | 00:03 |
| *** tonanhngo has quit IRC | 00:03 | |
| sdake | except i got a scremaing deal - 650 bucks! | 00:03 |
| Jeffrey4l | coin again... you love it so much. ;) | 00:04 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Modular gate https://review.openstack.org/386966 | 00:04 |
| sdake | its more like a brick then a coin | 00:05 |
| sdake | 1kilo = 2.2lbs | 00:05 |
| sdake | Jeffrey4l gotta have hobies | 00:05 |
| sdake | :) | 00:05 |
| HyperJohnGraham | pocket change ;) | 00:05 |
| kfox1111 | well, that's an interesting question to the legal list... | 00:06 |
| kfox1111 | "What are the options if someone active in the community decides to | 00:06 |
| kfox1111 | continue their work outside the foundation using the same project | 00:06 |
| kfox1111 | name?" | 00:06 |
| sdake | kfox1111 how would that happen | 00:06 |
| kfox1111 | not sure.... it was just asked though. | 00:07 |
| Jeffrey4l | like i create a repo named Linux ;) | 00:07 |
| sdake | heh :) | 00:07 |
| *** jtriley has quit IRC | 00:07 | |
| kfox1111 | I'm wondering if thats more of a "what if swift decided to leave the big tent" or that sort of thing. | 00:07 |
| *** hogepodge has joined #openstack-kolla | 00:08 | |
| kfox1111 | bootstrappping rabbit now... | 00:09 |
| kfox1111 | soon... | 00:09 |
| *** david-lyle has quit IRC | 00:09 | |
| *** janem has quit IRC | 00:10 | |
| Jeffrey4l | no idea about it. but i guess, in some case it is illegal when the foundation register the branch. | 00:10 |
| sdake | kfox1111 we dont have a polciy that limits our projec tto big tent projects | 00:12 |
| sdake | we use whatever makes sense | 00:12 |
| *** MarMat_ has quit IRC | 00:12 | |
| kfox1111 | sure. just wondering if thats a forshadowing of a big tent project leaving. | 00:13 |
| sdake | e.g. we deploy mariadb, docker, etc | 00:13 |
| sdake | al kinds of stuff | 00:13 |
| sdake | which is a forshadowing? | 00:13 |
| kfox1111 | the question to the legal list. | 00:13 |
| sdake | oh let me look :) | 00:13 |
| sdake | popcorn | 00:13 |
| Jeffrey4l | hmm. swift wanna leave big tent? | 00:13 |
| kfox1111 | look for " | 00:13 |
| kfox1111 | [legal-discuss] Trademark for a specific OpenStack project | 00:13 |
| kfox1111 | " | 00:13 |
| kfox1111 | not swift specifically. just a generic question. | 00:13 |
| sdake | Jeffrey4l nobody knows except swift :) | 00:13 |
| kfox1111 | I think they are none too happy about the go thing though. | 00:14 |
| sdake | my understanding is projects don't have trademarks | 00:14 |
| sdake | but i could be wrong | 00:14 |
| kfox1111 | but thats total specuation on my part. | 00:14 |
| sdake | no, they are not happy there is no speculation there | 00:14 |
| *** tonanhngo has joined #openstack-kolla | 00:14 | |
| kfox1111 | bootstrap passed with that one patch reverted. | 00:14 |
| kfox1111 | Jeffrey4l: the docker logs failed. | 00:15 |
| sdake | kfox1111 strace ;) | 00:15 |
| kfox1111 | oh.. because of no sudo. | 00:15 |
| Jeffrey4l | kfox1111, failed to run or? | 00:15 |
| sdake | kfox1111 strace will tell us which file its looking for | 00:15 |
| Jeffrey4l | kfox1111, add it in next patch. | 00:16 |
| kfox1111 | sdake: yeah. sudo's next. we know what patch it is for sure now though, and the rest of trunk seems clean. | 00:16 |
| sdake | kfox1111 would be nice if this is isolated to just cinder | 00:16 |
| kfox1111 | so if there have been other ones that merged with permissions, then its not that. | 00:16 |
| kfox1111 | sdake: seems like it is, since its just the one patch. did any other of the permissions ones merge? | 00:17 |
| sdake | kfox1111 a whole lot of em | 00:17 |
| kfox1111 | so then that part is probably in the clear... just something specific in the patch it doesn't like then. | 00:17 |
| kfox1111 | so shoudln't be too bad then. | 00:17 |
| sdake | that is one possible scenario | 00:17 |
| sdake | another scenario is some other patch (in ceph for e.g) triggers cinder to implode | 00:18 |
| Jeffrey4l | sdake, kfox1111 reproduced this locally. | 00:18 |
| Jeffrey4l | got the error message. | 00:18 |
| sdake | Jeffrey4l nice | 00:18 |
| Jeffrey4l | easy to fix. | 00:18 |
| sdake | error is what? | 00:18 |
| kfox1111 | the tricky bit now is to get it to strace with the workflow... | 00:18 |
| Jeffrey4l | invalid json | 00:18 |
| kfox1111 | oh. cool. :) | 00:18 |
| *** tonanhngo has quit IRC | 00:18 | |
| sdake | oh duh | 00:18 |
| kfox1111 | doh. | 00:18 |
| sdake | well - json looked correct to me | 00:19 |
| kfox1111 | that makes me sad. :( | 00:19 |
| Jeffrey4l | ya. me too. i am checking this carefully ;) | 00:19 |
| kfox1111 | we for sure need at least a json validator in the gate. :/ | 00:19 |
| *** hogepodge has quit IRC | 00:20 | |
| Jeffrey4l | kfox1111, try to collect all logs after container exist in k8s. then it will be easy to debug. | 00:20 |
| sdake | kfox1111 we have one | 00:20 |
| Jeffrey4l | but not for json.j2. | 00:20 |
| Jeffrey4l | json.js is a jinja template file first. | 00:20 |
| sdake | Jeffrey4l right | 00:20 |
| sdake | we should genconfig and jsonify it later | 00:20 |
| sdake | rather json gate after that | 00:21 |
| kfox1111 | Jeffrey4l: https://review.openstack.org/#/c/386966/21/tests/bin/gate_capture_logs.sh | 00:21 |
| *** hogepodge has joined #openstack-kolla | 00:21 | |
| kfox1111 | I've got logs up the wazoo! :) | 00:21 |
| kfox1111 | still seems like more and more are needed. :/ | 00:21 |
| kfox1111 | each run probably collects a few hundered megs of logs now. :/ | 00:21 |
| Jeffrey4l | wow. | 00:21 |
| *** david-lyle has joined #openstack-kolla | 00:21 | |
| Jeffrey4l | kolla ci catch: 1. docker logs 2. syslog /var/log 3. docker logs `journalctl -u docker ` 4. opentack logs /var/lib/docker/volumes/kolla_logs | 00:22 |
| kfox1111 | this gate job was horible to debug initially. :) | 00:22 |
| kfox1111 | yeah. I'm getting most of that now. plus a bunch more. | 00:22 |
| Jeffrey4l | lost a quota here: "owner: "cinder:cinder", :( | 00:23 |
| kfox1111 | the tricky part is to do it via k8s though, as direct docker won't work so easily when I go to multinode. | 00:23 |
| kfox1111 | so multinode should be relatively easy, since I did it with that in mind. | 00:23 |
| kfox1111 | Jeffrey4l: doh. that looks bad. | 00:23 |
| Jeffrey4l | yep. now kolla doesn't enable ci in gate. ;( | 00:24 |
| kfox1111 | we need to change that this cycle. :/ | 00:24 |
| Jeffrey4l | absolutely. more gates are really needed. | 00:25 |
| kfox1111 | I'm breaking up the kolla-kubernetes gate so that we can run it many different ways more easily | 00:25 |
| kfox1111 | was thinking we might hook tested kolla-kubernetes into the kolla gate. so stuff like this would be cought quickly. | 00:25 |
| *** schwicht has joined #openstack-kolla | 00:26 | |
| Jeffrey4l | sound cool. | 00:26 |
| kfox1111 | shoudl be pretty easy I think. just gota flip around a couple of repos. | 00:26 |
| *** sbezverk has joined #openstack-kolla | 00:26 | |
| kfox1111 | pull patch from kolla instead of kolla-kubernetes, but do all the same otherwise. | 00:26 |
| Jeffrey4l | it should not be that hard. | 00:26 |
| kfox1111 | yeah. | 00:27 |
| sdake | Jeffrey4l http://jsonlint.com/# | 00:27 |
| Jeffrey4l | add one of kolla job, which pull kolla-k8s and start the gate script. | 00:27 |
| kfox1111 | just want to get ceph+centos+cinder working first. its almost there. but odly broken. | 00:27 |
| kfox1111 | yeah. | 00:27 |
| Jeffrey4l | sdake, we should render the json file as jinja2 template, then validate it. | 00:27 |
| kfox1111 | Jeffrey4l: +1. | 00:28 |
| kfox1111 | Jeffrey4l: I'm doing that for all the kolla-kubernetes templates in the gate. has solved a bunch of issues. | 00:28 |
| Jeffrey4l | this is probable in kolla. which try to add this. | 00:28 |
| sdake | i've found it | 00:28 |
| sdake | "owner:" | 00:28 |
| *** frzn_ has joined #openstack-kolla | 00:28 | |
| sdake | that is wrong | 00:28 |
| sdake | it should be "onwer": | 00:28 |
| frzn_ | test | 00:28 |
| frzn_ | hi hello | 00:28 |
| *** frzn_ is now known as mtaylo22 | 00:28 | |
| kfox1111 | can probablyl just run a genconfig, then walk /etc/kolla and do a test on the json files. | 00:28 |
| *** mtaylo22 is now known as mtaylor22_ | 00:29 | |
| sdake | hello frz | 00:29 |
| mtaylor22_ | morning etc. | 00:29 |
| mtaylor22_ | :) | 00:29 |
| Jeffrey4l | kfox1111, yes. it is better. | 00:29 |
| sdake | kfox1111 emergeny averted, this is a simple bug in the json merged | 00:29 |
| sdake | i'll fix now | 00:29 |
| *** david-lyle has quit IRC | 00:29 | |
| *** ssurana has quit IRC | 00:30 | |
| kfox1111 | sdake: yup. :) thanks for letting me know to test. this would have been really anoying to let out the door. :) | 00:30 |
| *** mtaylor22 has quit IRC | 00:30 | |
| *** mtaylor22_ is now known as mtaylor22 | 00:30 | |
| *** david-lyle has joined #openstack-kolla | 00:31 | |
| sdake | kfox1111 yup - that is why we also manually test | 00:31 |
| * kfox1111 nods | 00:31 | |
| kfox1111 | I just got tired of manually testing. :) though probably spent a lot more effort, at least initially on gating... | 00:32 |
| kfox1111 | but I feel much more comfortable upgradoing my clouds if I know there's a solid gate there. | 00:32 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix for cinder not working with permissions changes https://review.openstack.org/387736 | 00:33 |
| Jeffrey4l | kfox1111, about this json validate, i will extend the exist script to support json.j2 file. | 00:33 |
| kfox1111 | cool. | 00:34 |
| *** david-lyle has quit IRC | 00:34 | |
| *** diogogmt has quit IRC | 00:34 | |
| sdake | kfox1111 can you cherrypick that patch into your test case | 00:35 |
| sdake | and validate it | 00:35 |
| *** david-lyle has joined #openstack-kolla | 00:35 | |
| sdake | kfox1111 right - gating has high upfront cost | 00:36 |
| sdake | but long term its the only way to roll | 00:36 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Testing trunk... https://review.openstack.org/387641 | 00:36 |
| *** haplo37_ has quit IRC | 00:36 | |
| Jeffrey4l | sdake, hmm should be Partial-Bug: #1631503 | 00:37 |
| openstack | bug 1631503 in kolla "inconsistent UID in named volume results in inability to upgrade" [Critical,In progress] https://launchpad.net/bugs/1631503 - Assigned to Steven Dake (sdake) | 00:37 |
| kfox1111 | sdake: done. https://review.openstack.org/#/c/387641/5 | 00:37 |
| kfox1111 | watch for the ubuntu-binary-ceph job. | 00:37 |
| kfox1111 | I've got to head out. will be back later. | 00:37 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix for cinder not working with permissions changes https://review.openstack.org/387736 | 00:38 |
| *** haplo37_ has joined #openstack-kolla | 00:39 | |
| sbezverk | kfox1111: how the latest version of fencing pod looking? | 00:40 |
| *** yingjun has joined #openstack-kolla | 00:41 | |
| sdake | Jeffrey4l if you could take a look at the same topics field for this bug: https://review.openstack.org/#/c/387603/1 | 00:41 |
| sdake | Jeffrey4l alot of them can be merged with one more +2 | 00:41 |
| Jeffrey4l | np. | 00:42 |
| * Jeffrey4l is reviewing all relate PS> | 00:42 | |
| sdake | Jeffrey4l any that can't need to be fixed | 00:42 |
| sdake | so that hwen our emea cr's wake up for the day they have some work to do :) | 00:42 |
| openstackgerrit | Merged openstack/kolla: Fix sahara upgrade permission issue https://review.openstack.org/387600 | 00:43 |
| sdake | i also don't know if any services were missed | 00:43 |
| Jeffrey4l | sdake, we track this here https://etherpad.openstack.org/p/kolla-bug-1631503 | 00:43 |
| sdake | Jeffrey4l yup - however its not tracking everything and not including backports | 00:44 |
| sdake | Jeffrey4l i thin ka git log should be able to tell us that all he files have been changed | 00:44 |
| Jeffrey4l | yep. need update. | 00:44 |
| sdake | find . -name \*json.j2 -exec git log {} \; | 00:44 |
| sdake | might work | 00:44 |
| sdake | lets get whats in the queue merged first | 00:44 |
| Jeffrey4l | ok. | 00:45 |
| sdake | Jeffrey4l i've got dinner in about 30mins-1hr for 20 or 30 mins | 00:46 |
| Jeffrey4l | roger. | 00:46 |
| *** tovin07 has joined #openstack-kolla | 00:46 | |
| *** david-lyle has quit IRC | 00:47 | |
| sdake | nay core reiewers out there - need a workflow on https://review.openstack.org/#/c/387736/ | 00:47 |
| *** david-lyle has joined #openstack-kolla | 00:48 | |
| *** tonanhngo has joined #openstack-kolla | 00:48 | |
| *** robotzzzz has joined #openstack-kolla | 00:48 | |
| *** robotzzzz has quit IRC | 00:49 | |
| *** tonanhngo has quit IRC | 00:49 | |
| Jeffrey4l | cinder is OK with above PS ^^ | 00:49 |
| Jeffrey4l | tested locally. | 00:49 |
| *** gfidente has quit IRC | 00:50 | |
| *** sdake_ has joined #openstack-kolla | 00:51 | |
| *** sdake_ has quit IRC | 00:51 | |
| *** sdake_ has joined #openstack-kolla | 00:51 | |
| *** david-lyle has quit IRC | 00:51 | |
| sdake_ | Jeffrey4l i'd say automate the json.j2 gate check later, nd focus on getting thos reviews done ;) | 00:52 |
| *** david-lyle has joined #openstack-kolla | 00:52 | |
| Jeffrey4l | agree. im reviewing the queue now. | 00:52 |
| sdake_ | Jeffrey4l stable/newton can be rubberstampped at this point - already passed master review | 00:52 |
| *** sdake has quit IRC | 00:53 | |
| sdake_ | although clearly reading json is not optimal - so we should validate when done | 00:53 |
| sdake_ | i stared a t that patch for hours before finding the " missing | 00:53 |
| openstackgerrit | Merged openstack/kolla: Fix telegraf upgrade permission issue https://review.openstack.org/387604 | 00:53 |
| sdake_ | and of course validate upgrade | 00:54 |
| Jeffrey4l | render the json.j2 file by using jinja2 first ( provide different variables ), then validate the json file. | 00:54 |
| sdake_ | ya i don twant to wait around for the gate to be automated for that work to be done | 00:54 |
| sdake_ | that could take days | 00:55 |
| sdake_ | especially since everyone is traveling shortly | 00:55 |
| sdake_ | Jeffrey4l which day do you depart for barcelona? | 00:55 |
| Jeffrey4l | 23 1:30 utc+8 | 00:56 |
| sdake_ | i leave 20th | 00:56 |
| sdake_ | at around 1600 utc | 00:56 |
| *** duonghq has joined #openstack-kolla | 00:57 | |
| *** tonanhngo has joined #openstack-kolla | 00:57 | |
| *** tonanhngo has quit IRC | 00:58 | |
| openstackgerrit | Merged openstack/kolla: Fix mongodb upgrade permission issue https://review.openstack.org/387615 | 00:58 |
| *** phuongnh has joined #openstack-kolla | 00:59 | |
| *** david-lyle has quit IRC | 00:59 | |
| Jeffrey4l | sdake_, got. see u then ( hmm, it is OK/correct to say this ;) ) | 01:00 |
| openstackgerrit | Merged openstack/kolla: Fix magnum upgrade permission issue https://review.openstack.org/387603 | 01:00 |
| sdake_ | huh? | 01:00 |
| *** david-lyle has joined #openstack-kolla | 01:00 | |
| sdake_ | i dont ike to rubberstamp backports, but its necessary at end of release | 01:01 |
| sdake_ | because if we find bug, we need that original backport in the repo | 01:01 |
| sdake_ | to be consistent with stable:follows-policy | 01:01 |
| sdake_ | as well as the fix to the backport | 01:01 |
| sdake_ | (consider cinder as an example) | 01:02 |
| Jeffrey4l | roger. | 01:02 |
| *** zhubingbing has joined #openstack-kolla | 01:03 | |
| sdake_ | sup zhenguo | 01:03 |
| sdake_ | zhubingbing | 01:03 |
| zhubingbing | hi | 01:03 |
| sdake_ | more zh* every day :) | 01:04 |
| zhubingbing | i'am here | 01:04 |
| Jeffrey4l | lol | 01:04 |
| sdake_ | zhubingbing - nice job on all the work related to upgrades | 01:04 |
| sdake_ | zhubingbing did you get them all - or do more remain | 01:04 |
| zhubingbing | ;) | 01:04 |
| zhubingbing | sure | 01:05 |
| sdake_ | sure you go tthem all or sure moer remain :) | 01:05 |
| *** tonanhngo has joined #openstack-kolla | 01:05 | |
| zhubingbing | yes | 01:05 |
| *** david-lyle has quit IRC | 01:05 | |
| sdake_ | yes what? :) | 01:05 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix wrong magnum volume path in json file https://review.openstack.org/387740 | 01:06 |
| *** tonanhngo has quit IRC | 01:06 | |
| zhubingbing | sure you go tthem all or sure moer remain :) yes | 01:06 |
| zhubingbing | ）） | 01:06 |
| *** david-lyle has joined #openstack-kolla | 01:06 | |
| Jeffrey4l | zhubingbing, update the etherpad if your patch is merged. https://etherpad.openstack.org/p/kolla-bug-1631503 | 01:07 |
| zhubingbing | ok | 01:07 |
| *** rhallisey has quit IRC | 01:07 | |
| zhubingbing | i fix it now | 01:07 |
| *** david-lyle has quit IRC | 01:07 | |
| zhubingbing | sdake I'll fix it all | 01:10 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 01:10 | |
| Jeffrey4l | brb | 01:10 |
| *** david-lyle has joined #openstack-kolla | 01:13 | |
| *** david-lyle has quit IRC | 01:14 | |
| *** david-lyle has joined #openstack-kolla | 01:15 | |
| *** tonanhngo has joined #openstack-kolla | 01:15 | |
| *** hogepodge has quit IRC | 01:16 | |
| *** tonanhngo has quit IRC | 01:16 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix influxdb upgrade permission issue https://review.openstack.org/387325 | 01:16 |
| Pavo | so this is my current setup here at home | 01:16 |
| Pavo | https://www.dropbox.com/s/r0bescg907id4l6/Screenshot%202016-10-17%2021.16.17.png?dl=0 | 01:16 |
| Pavo | what you guys think? | 01:16 |
| sdake_ | pavo what tool did you use to make that diagram? | 01:18 |
| Pavo | gliffy | 01:18 |
| sdake_ | thought so thanks | 01:18 |
| Pavo | https://www.gliffy.com/go/publish/11309187 | 01:18 |
| *** david-lyle has quit IRC | 01:19 | |
| sdake_ | note if you are a pyaing customer you can send them a nastygram and they will enable collaboration for your account | 01:19 |
| sdake_ | worked for me atleast ;-) | 01:19 |
| Pavo | just a free accoutn right now | 01:19 |
| Pavo | this better | 01:20 |
| Pavo | https://www.gliffy.com/go/publish/11309187 | 01:20 |
| *** david-lyle has joined #openstack-kolla | 01:20 | |
| *** david-lyle has quit IRC | 01:21 | |
| sdake_ | get me the anestaphine from the medlock! | 01:21 |
| Pavo | lol | 01:21 |
| *** david-lyle has joined #openstack-kolla | 01:23 | |
| *** tonanhngo has joined #openstack-kolla | 01:23 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix murano upgrade permission issue https://review.openstack.org/387743 | 01:24 |
| *** david-lyle has quit IRC | 01:24 | |
| *** david-lyle has joined #openstack-kolla | 01:25 | |
| *** pritesh has quit IRC | 01:27 | |
| *** david-lyle has quit IRC | 01:32 | |
| *** david-lyle has joined #openstack-kolla | 01:34 | |
| *** hogepodge has joined #openstack-kolla | 01:35 | |
| *** denaitre has quit IRC | 01:36 | |
| *** denaitre has joined #openstack-kolla | 01:37 | |
| *** david-lyle has quit IRC | 01:37 | |
| *** yee3796 has joined #openstack-kolla | 01:38 | |
| *** david-lyle has joined #openstack-kolla | 01:38 | |
| *** david-lyle has quit IRC | 01:38 | |
| *** yee379 has quit IRC | 01:39 | |
| *** NobodyCam has quit IRC | 01:42 | |
| *** NobodyCam has joined #openstack-kolla | 01:42 | |
| *** zhangyufei has joined #openstack-kolla | 01:43 | |
| *** david-lyle has joined #openstack-kolla | 01:44 | |
| *** david-lyle has quit IRC | 01:44 | |
| *** david-lyle has joined #openstack-kolla | 01:45 | |
| *** david-lyle has quit IRC | 01:46 | |
| *** david-lyle has joined #openstack-kolla | 01:46 | |
| *** david-lyle has quit IRC | 01:47 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add OS_CACERT in openrc when it is configured https://review.openstack.org/387744 | 01:50 |
| Jeffrey4l | kfox1111, have u ever tested swift? | 01:53 |
| Jeffrey4l | sdake_, re https://bugs.launchpad.net/kolla/+bug/1632811 leave a comment | 01:54 |
| openstack | Launchpad bug 1632811 in kolla "Keystone token table filling up" [Critical,In progress] - Assigned to Mathias Ewald (mewald) | 01:54 |
| Pavo | what are you guys using to hash passwords and stuff? MD5? | 01:55 |
| Pavo | hopefully atleast sha256 | 01:55 |
| sdake_ | pavo - password generation occurs in tools/genpwd.py | 01:55 |
| Pavo | yeah but which algorithm? | 01:55 |
| Jeffrey4l | Pavo, it is just random char | 01:56 |
| sdake_ | https://github.com/openstack/kolla/blob/master/kolla/cmd/genpwd.py#L25 | 01:56 |
| Jeffrey4l | the passwords in password.yaml file is raw. no encrypt . | 01:56 |
| sdake_ | pavo and: | 01:56 |
| *** david-lyle has joined #openstack-kolla | 01:56 | |
| sdake_ | https://github.com/openstack/kolla/blob/master/kolla/cmd/genpwd.py#L78 | 01:56 |
| Pavo | the reason I am asking is for FIPS 140-2 compliance | 01:57 |
| sdake_ | SystemRandom is a SPRNG that is built into python | 01:57 |
| Pavo | which is one of those CAT 1 requirement :( | 01:57 |
| sdake_ | it is based upon /dev/urandom | 01:57 |
| sdake_ | if you dont trust that type of SPRNG you can make your own secure passwords | 01:57 |
| Pavo | true | 01:57 |
| *** david-lyle has quit IRC | 01:57 | |
| sdake_ | as far as the ssh keys and whatnot, that is standard libssh stuff | 01:58 |
| sdake_ | rsa 4096 - should be secure to all but the most determined individuals (i.e. government agencies) | 01:58 |
| *** david-lyle has joined #openstack-kolla | 01:58 | |
| Pavo | yeah tell me about it | 01:58 |
| Pavo | :P | 01:59 |
| sdake_ | i am not sure if SystemRandom has gone through fips140 | 01:59 |
| sdake_ | same story for the Crypto library we use | 01:59 |
| sdake_ | I would think they would have | 01:59 |
| sdake_ | but don't know for sure | 01:59 |
| sdake_ | if you find out, please do let me know | 02:00 |
| sdake_ | note, you can always generate your own credentials using custom tools with more secure defaults | 02:00 |
| *** david-lyle has quit IRC | 02:00 | |
| sdake_ | I'm not sure how you would make it more secure | 02:00 |
| sdake_ | but maybe you have secret tricks ;-) | 02:00 |
| *** david-lyle has joined #openstack-kolla | 02:01 | |
| sdake_ | one advantage of not using our genpwd tool is your tool would be secure according to your requirements | 02:01 |
| sdake_ | a more secure approach would be to HMAC the SystemRandom output | 02:02 |
| *** david-lyle has quit IRC | 02:02 | |
| sdake_ | i'm not sure if that would be much better | 02:02 |
| sdake_ | the problem is SPRNG is not a S as one would like :( | 02:02 |
| Pavo | well we found out today that packstack uses MD5 for everything that it hashes | 02:02 |
| Pavo | which can not be found in any of the source code | 02:02 |
| *** david-lyle has joined #openstack-kolla | 02:02 | |
| sdake_ | pavo openstack itself may be using md5 - i am not sure | 02:03 |
| *** unicell has quit IRC | 02:03 | |
| Pavo | true | 02:03 |
| sdake_ | pavo but we dont use any md5 anywhere in kolla - it is insecure | 02:03 |
| *** david-lyle has quit IRC | 02:03 | |
| Pavo | good to know | 02:03 |
| sdake_ | another option would be to use a cipher on the systemrandom data | 02:03 |
| *** diogogmt has joined #openstack-kolla | 02:03 | |
| sdake_ | note the passwords.yml file (protected by filesystem permissions) can be encrypted via vault | 02:04 |
| *** david-lyle has joined #openstack-kolla | 02:04 | |
| *** david-lyle has quit IRC | 02:04 | |
| sdake_ | i am not sure if the systemrandom code uses a chaining block cipher or not | 02:05 |
| *** david-lyle has joined #openstack-kolla | 02:05 | |
| Pavo | sha256 at least is good enough | 02:05 |
| *** david-lyle has quit IRC | 02:05 | |
| sdake_ | tbh, I think there are backdoors planted in all encryption stuff | 02:05 |
| sdake_ | sha256 is secure from my POV | 02:05 |
| sdake_ | sha256 doesn't generate random data | 02:05 |
| sdake_ | unless combined with a cipher | 02:06 |
| *** schwicht has quit IRC | 02:06 | |
| sdake_ | i am not convinced rsa is secure | 02:06 |
| sdake_ | but if it is insecure, only a few people in teh world know how its busted | 02:06 |
| sdake_ | and at best, it would just shorten the search space | 02:06 |
| sdake_ | which again might be doable for a govt agency | 02:06 |
| sdake_ | but not for an individual cracker :) | 02:07 |
| *** david-lyle has joined #openstack-kolla | 02:07 | |
| *** mtaylor22 has quit IRC | 02:08 | |
| *** david-lyle has quit IRC | 02:08 | |
| *** mtaylor22 has joined #openstack-kolla | 02:09 | |
| sdake_ | Pavo on the plus side if there is any insecure parts of kolla - all patches must pass through 3 people's hands | 02:09 |
| sdake_ | and we trust our core reviewers implicitly | 02:09 |
| sdake_ | so it would take 3 people working in collusion to hide an exploit | 02:09 |
| sdake_ | and the exploit would likely be discovered by other core reviewers | 02:09 |
| sdake_ | pavo we also have a semi-policy of disallowing merges from 1 company by multiple poeple | 02:10 |
| *** david-lyle has joined #openstack-kolla | 02:10 | |
| sdake_ | this is why openstack in general is secure | 02:10 |
| sdake_ | not just kolla | 02:11 |
| sdake_ | there are security vulns found of course | 02:11 |
| sdake_ | none yet in kolla that i'm aware of | 02:11 |
| *** david-lyle has quit IRC | 02:11 | |
| sdake_ | but they are more like errors rather then intentional | 02:11 |
| sdake_ | not like when kernel.org got cracked and shit got inserted in kernel.org | 02:11 |
| *** david-lyle has joined #openstack-kolla | 02:12 | |
| sdake_ | that was like a 6 week reset | 02:12 |
| sdake_ | attacking openstack security wise woudl be very difficult - and possibly beyond government control - although there was that virus in iran ;) | 02:12 |
| sdake_ | Pavo any other q? | 02:13 |
| Pavo | not sof ar | 02:13 |
| Pavo | so far | 02:13 |
| sdake_ | ok - Jeffrey4l - so there are several -1s on reviews related to upgrade | 02:13 |
| sdake_ | Jeffrey4l can we split these up and fix em up | 02:13 |
| *** david-lyle has quit IRC | 02:14 | |
| Jeffrey4l | sdake_, two are -1 both mine. | 02:14 |
| Jeffrey4l | hmm. three, including nova. | 02:15 |
| sdake_ | Jeffrey4l i think nova is good to go | 02:15 |
| sdake_ | inc007 tested it | 02:16 |
| sdake_ | it does change the permissiosn of disk and the log file | 02:16 |
| sdake_ | but it seems toh ave no impact since libvirt runs as root | 02:16 |
| Jeffrey4l | OK. merge it. | 02:17 |
| *** david-lyle has joined #openstack-kolla | 02:17 | |
| *** david-lyle has quit IRC | 02:18 | |
| *** david-lyle has joined #openstack-kolla | 02:18 | |
| sdake_ | Jeffrey4l - ithink we merge everything we can then test from there | 02:19 |
| Jeffrey4l | so for ironic conductor. /tftpboot is used by pxe which runs as root, too. | 02:19 |
| Jeffrey4l | remove my -1 for ironic. | 02:19 |
| *** david-lyle has quit IRC | 02:19 | |
| Jeffrey4l | roger. | 02:19 |
| sdake_ | is ironic good to go though? | 02:19 |
| sdake_ | if /tftbpoot is owned by root shouldn't it stay so? | 02:19 |
| sdake_ | for security reasons | 02:19 |
| Jeffrey4l | sdake_, ironic will add files in tftpboot folder. | 02:20 |
| *** david-lyle has joined #openstack-kolla | 02:20 | |
| openstackgerrit | Merged openstack/kolla: Fix nova upgrade permission issue https://review.openstack.org/386709 | 02:21 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix nova upgrade permission issue https://review.openstack.org/387753 | 02:21 |
| *** david-lyle has quit IRC | 02:21 | |
| Jeffrey4l | i wanna to set up a test env for ironic to check. | 02:22 |
| sdake_ | btw i added newton to our lsit of irc announcements :) | 02:22 |
| sdake_ | Jeffrey4l wfm | 02:22 |
| Jeffrey4l | re announcements: what it means by that? meeting agenda? | 02:25 |
| Jeffrey4l | sdake_, ^^ | 02:25 |
| sdake_ | Jeffrey4l stable/newton changes notify our irc channel now Jeffrey4l | 02:26 |
| *** david-lyle has joined #openstack-kolla | 02:26 | |
| *** dave-mccowan has quit IRC | 02:26 | |
| Jeffrey4l | sound cool. what's kind of changes? each commit? or ? no idea about this. | 02:28 |
| Jeffrey4l | sdake_, | 02:28 |
| sdake_ | every review and commit | 02:28 |
| sdake_ | just like master and stable/mitaka /etc | 02:28 |
| Jeffrey4l | wow | 02:28 |
| *** yuanying has quit IRC | 02:29 | |
| Jeffrey4l | found it. configure by project-config/gerritbot/channels.yaml right? | 02:30 |
| sdake_ | yup | 02:30 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix neutron upgrade permission issue https://review.openstack.org/387212 | 02:32 |
| *** haplo37 has quit IRC | 02:37 | |
| *** g3ek has quit IRC | 02:37 | |
| *** haplo37 has joined #openstack-kolla | 02:39 | |
| *** hogepodge has quit IRC | 02:40 | |
| *** g3ek has joined #openstack-kolla | 02:40 | |
| Jeffrey4l | sdake_, https://review.openstack.org/#/c/387743/ | 02:41 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix mistral upgrade permission issue https://review.openstack.org/387760 | 02:43 |
| *** huhaoran has joined #openstack-kolla | 02:43 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix cinder upgrade permission issue https://review.openstack.org/387597 | 02:44 |
| Pavo | does the setup_RedHat.sh script take a new bare metal install of CentOS and make it ready for kolla deployment? | 02:47 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix heat upgrade permission issue https://review.openstack.org/387761 | 02:47 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix mistral upgrade permission issue https://review.openstack.org/387760 | 02:48 |
| openstackgerrit | Merged openstack/kolla: Fix murano upgrade permission issue https://review.openstack.org/387743 | 02:49 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix congress upgrade permission issue https://review.openstack.org/387762 | 02:50 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix influxdb upgrade permission issue https://review.openstack.org/387764 | 02:52 |
| sdake_ | well atleast our gate is working again Jeffrey4l :) | 02:54 |
| Pavo | sdake_ | 02:54 |
| Pavo | does the setup_RedHat.sh script take a new bare metal install of CentOS and make it ready for kolla deployment? | 02:54 |
| Jeffrey4l | ya. | 02:54 |
| openstackgerrit | Merged openstack/kolla: Fix influxdb upgrade permission issue https://review.openstack.org/387325 | 02:55 |
| Jeffrey4l | Pavo, no. it is using loop device for docker right now. | 02:55 |
| Pavo | but other than that everything else is done right? | 02:55 |
| Jeffrey4l | this script is only used for ci now. | 02:55 |
| Pavo | for all nodes | 02:55 |
| Pavo | ci? | 02:55 |
| Jeffrey4l | correct. | 02:55 |
| Pavo | whats ci | 02:56 |
| Jeffrey4l | Pavo, gate jobs? | 02:56 |
| Pavo | ah | 02:56 |
| Pavo | ok | 02:56 |
| Pavo | man there is still alot I need to learn | 02:56 |
| Jeffrey4l | this script is simple and only used for ci. please do not use it especially in produce env | 02:57 |
| sdake_ | Pavo there is a playbook tha twill setup centos | 02:57 |
| sdake_ | pavo i think its kolla-ansible bootstrap | 02:57 |
| Pavo | if you had a way to go train with a few companies of your choice to get more involved with openstack, docker and a few others which companies would you pick to get training from? | 02:57 |
| Pavo | sdake_ ok I'll look into it | 02:58 |
| Pavo | money no issue on the training btw | 02:58 |
| Pavo | which ones would you go to | 02:58 |
| *** inc0 has joined #openstack-kolla | 03:00 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Fix nova upgrade permission issue https://review.openstack.org/387753 | 03:00 |
| Pavo | sdake_ and Jeffrey4l which ones would you get training from? | 03:01 |
| Pavo | remember of your choice, money no issue | 03:01 |
| inc0 | trainings? where?:D | 03:01 |
| *** ssurana has joined #openstack-kolla | 03:02 | |
| sdake_ | pavo training? | 03:02 |
| sdake_ | OTJ | 03:02 |
| Pavo | training to get you more up to date with the whole openstack stuff and docker and anything else that would be beneficial | 03:02 |
| Jeffrey4l | no need training :/ . learn from code ;) | 03:02 |
| Pavo | yes companies you could go work with, Training with Industry | 03:03 |
| inc0 | what Jeffrey4l said, most of trainings will be outdated | 03:03 |
| inc0 | by the time you take them;) | 03:03 |
| sdake_ | sup inc0 | 03:03 |
| sdake_ | inc0 we need some acks on reviews | 03:03 |
| sdake_ | inc0 upgrades reviews in particular | 03:03 |
| sdake_ | everything else is good to go | 03:03 |
| Pavo | ok what coding languages, I am thinking python, yaml and json | 03:03 |
| inc0 | Pavo, so for kolla, start with ansible | 03:04 |
| Jeffrey4l | yaml and jons is not language. it is file format. | 03:04 |
| inc0 | not really coding language | 03:04 |
| Pavo | isn't that yaml? | 03:04 |
| Pavo | ah ok | 03:04 |
| *** hogepodge has joined #openstack-kolla | 03:04 | |
| Pavo | I see | 03:04 |
| sdake_ | probably lots of cats teach ansible at this point | 03:04 |
| sdake_ | i'd start with a book | 03:04 |
| Pavo | I think I am pretty good with ansible | 03:05 |
| inc0 | Pavo, you going to BCN? | 03:05 |
| Pavo | been playing with it for a bit now | 03:05 |
| inc0 | Docker is super easy to understand in principle | 03:05 |
| inc0 | if you're good at linux | 03:05 |
| *** portdirect has quit IRC | 03:05 | |
| Pavo | that what we use to apply STIGs to our stacks | 03:05 |
| Pavo | yeah getting better with Docker also | 03:05 |
| sdake_ | wot is a stig | 03:05 |
| Pavo | Security Technical Implementation Guides | 03:06 |
| inc0 | sdake_, some security procedure, Pavo already linked;) | 03:06 |
| Pavo | http://iase.disa.mil/stigs/Pages/index.aspx | 03:06 |
| sdake_ | cool | 03:06 |
| sdake_ | well i dont know anything about stigs | 03:06 |
| sdake_ | but we do intend to go through VMT at some point in the future ;-) | 03:06 |
| inc0 | Pavo, python won't be that much needed to running openstack | 03:06 |
| sdake_ | VMT = vulnerability management | 03:06 |
| Pavo | and what is BCN inc0 | 03:06 |
| inc0 | Barcelona - summit | 03:06 |
| sdake_ | pavo airport code for barcelona | 03:06 |
| Pavo | when is it? | 03:07 |
| inc0 | next week;) | 03:07 |
| sdake_ | next week | 03:07 |
| inc0 | so probably a bit too late;) | 03:07 |
| sdake_ | bjolo booked his trave lthis morning | 03:07 |
| inc0 | however, next big meeting will be in Atlanta in Febuary | 03:07 |
| Pavo | lol yeah WAY to late for that | 03:07 |
| Pavo | really | 03:07 |
| Pavo | thats only 2hours away from me | 03:07 |
| sdake_ | but i think he is in EU | 03:07 |
| Pavo | that I will probably be at | 03:07 |
| inc0 | if you'd want to meet us in person, make a trip:) | 03:07 |
| *** portdirect_ has joined #openstack-kolla | 03:08 | |
| Pavo | wait Atlanta, GA? | 03:08 |
| inc0 | cool, I'm not sure if it'll be free for everyone, but I'm pretty sure it will be free for commiters | 03:08 |
| sdake_ | i'm going to be on pto for the next 4 months | 03:08 |
| sdake_ | enjoy :) | 03:08 |
| *** portdirect_ is now known as portdirect | 03:08 | |
| inc0 | so if you make commit to kolla next couple of months, that should do it | 03:08 |
| sdake_ | people can attend ptg for free is my understanding | 03:08 |
| sdake_ | or for 25 bucks or something | 03:08 |
| Pavo | inc0 you meaning Atlanta, GA right | 03:08 |
| sdake_ | something reallly minimal | 03:08 |
| sdake_ | without having made a commit | 03:09 |
| inc0 | yeah | 03:09 |
| inc0 | big city in US | 03:09 |
| inc0 | that one | 03:09 |
| Pavo | ok cool | 03:09 |
| Pavo | yeah we are going to be at the Boston summit also | 03:09 |
| Pavo | next year | 03:09 |
| inc0 | cool | 03:09 |
| sdake_ | https://www.youtube.com/watch?v=cDm_ZHyYTrg | 03:09 |
| inc0 | probably we'll meet there too | 03:09 |
| Pavo | and I think military get into those summits for free | 03:09 |
| Pavo | but not sure | 03:10 |
| inc0 | Pavo everyone who makes commit to openstack project gets free entry | 03:10 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix grafana upgrade permission issue https://review.openstack.org/387768 | 03:10 |
| *** huhaoran has quit IRC | 03:10 | |
| inc0 | so if mil wont get for free, no problem, we'll figure a good patchset for you to make:) | 03:10 |
| Pavo | well thats easy lol, have done a few code reviews for the docs | 03:10 |
| inc0 | one step at the time, you'll commit stuff in no time | 03:11 |
| *** huhaoran has joined #openstack-kolla | 03:11 | |
| Pavo | been working on some SELINUX stuff with docker here lately, might commit that stuff since you guys not really using SELINUX | 03:12 |
| inc0 | if it helps with security, sure | 03:12 |
| *** eaguilar has joined #openstack-kolla | 03:15 | |
| sdake_ | wish i would have seen cash live before he passed | 03:15 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Ironic should be started before nova-ironic-compute https://review.openstack.org/387772 | 03:16 |
| openstackgerrit | Merged openstack/kolla: move sys.exit out of try finally block in set_config.py file https://review.openstack.org/387715 | 03:16 |
| openstackgerrit | Merged openstack/kolla: Fix common role upgrade permission issue https://review.openstack.org/387717 | 03:16 |
| openstackgerrit | Merged openstack/kolla: Fix for cinder not working with permissions changes https://review.openstack.org/387736 | 03:17 |
| *** ssurana has quit IRC | 03:18 | |
| inc0 | rubber-stamp all the things mwhahaha! | 03:18 |
| inc0 | sdake_, kolla started around kilo right? | 03:19 |
| *** coolsvap has quit IRC | 03:20 | |
| *** eaguilar has quit IRC | 03:20 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix grafana upgrade permission issue https://review.openstack.org/387774 | 03:20 |
| openstackgerrit | Merged openstack/kolla: Fix wrong magnum volume path in json file https://review.openstack.org/387740 | 03:20 |
| duonghq | Pavo: still there? | 03:23 |
| Pavo | yeah | 03:23 |
| *** david-lyle has quit IRC | 03:23 | |
| jascott1 | sdake_ me too (cash) | 03:23 |
| sdake_ | inc0 2 mo prior to kilo | 03:23 |
| sdake_ | about 3 years i htink | 03:23 |
| inc0 | https://www.youtube.com/watch?v=QfCOJLRk2D4 | 03:24 |
| *** david-lyle has joined #openstack-kolla | 03:24 | |
| Jeffrey4l | Launch at Sep 17, 2014 | 03:24 |
| Jeffrey4l | Join OpenStack Big Tent at Aug 5, 2015 | 03:24 |
| Jeffrey4l | First release at Liberty | 03:24 |
| *** david-lyle has quit IRC | 03:25 | |
| inc0 | yeah about right | 03:25 |
| sdake_ | no our first tag was actually juno jeffrey | 03:25 |
| duonghq | Pavo, https://bugs.launchpad.net/kolla/+bug/1621312 | 03:25 |
| openstack | Launchpad bug 1621312 in kolla "Remove duplicate jobs in gate setup script and bootstrap-servers" [High,Confirmed] - Assigned to Duong Ha-Quang (duonghq) | 03:25 |
| sdake_ | GIT LOG | 03:25 |
| duonghq | I reported this bug about duplicate login in setup gate and boostrap action | 03:25 |
| inc0 | I'll tell couple of words about history on summit;) | 03:25 |
| *** david-lyle has joined #openstack-kolla | 03:26 | |
| sdake_ | so 2 years then | 03:26 |
| sdake_ | Jeffrey4l sep 17 is from git log? | 03:27 |
| Pavo | is there a real reason to use bash scripts anymore? I mean ansible can do pretty much the same thing right? | 03:27 |
| openstackgerrit | Merged openstack/kolla: Fix cinder upgrade permission issue https://review.openstack.org/387597 | 03:27 |
| Jeffrey4l | sdake_, yep. | 03:27 |
| inc0 | Pavo, we use bash inside containers | 03:27 |
| Pavo | ah | 03:27 |
| duonghq | Pavo: nearly totally agreed with you | 03:27 |
| *** david-lyle has quit IRC | 03:27 | |
| inc0 | to limit deps | 03:27 |
| sdake_ | as well as python | 03:27 |
| Pavo | makes since | 03:27 |
| Jeffrey4l | Date: Wed Sep 17 08:01:48 2014 -0400 | 03:27 |
| Jeffrey4l | Initial commit | 03:27 |
| sdake_ | juno kilo liberty mitaka newton - so 2.5 years for me | 03:28 |
| *** david-lyle has joined #openstack-kolla | 03:28 | |
| duonghq | but the setup gate should use bootstrap action as much as possible | 03:28 |
| sdake_ | we had tags for all those versions | 03:28 |
| sdake_ | Jeffrey4l we floundered around for awhile until I got a gig at cisco and could properly lead kolla without PHBs breathing down my neck | 03:29 |
| *** david-lyle has quit IRC | 03:29 | |
| *** david-lyle has joined #openstack-kolla | 03:30 | |
| sdake_ | and then we had to convince the community to use ansible | 03:30 |
| sdake_ | not sure how we pulled that one off | 03:30 |
| sdake_ | started with 4 -2s and 3 +2s | 03:30 |
| inc0 | Sam pulled that off | 03:30 |
| sdake_ | he didn't pull that off inc0 | 03:30 |
| sdake_ | he wrote parts of the intial implementation | 03:31 |
| Jeffrey4l | what's the alternative if not ansible? | 03:31 |
| sdake_ | but getting the core team to buy into the idea | 03:31 |
| sdake_ | nothign of his doing | 03:31 |
| inc0 | he did, all the pocs and initial stuff were based on yaodu | 03:31 |
| sdake_ | inc0 obviously wasn't here for the vote on the spec ;) | 03:31 |
| inc0 | I wasn't core then but I was here | 03:31 |
| inc0 | I'm here since compose:) | 03:32 |
| Pavo | yeah we started off with icehouse | 03:32 |
| *** david-lyle has quit IRC | 03:33 | |
| *** david-lyle has joined #openstack-kolla | 03:34 | |
| *** david-lyle has quit IRC | 03:36 | |
| sdake_ | if we have anyone to owe to our total and utter success | 03:37 |
| sdake_ | it is oddesy4me | 03:37 |
| Pavo | so can someone explain to me your complete process from after commiting code and testing it, like what all is being used and how its get built and tore down? | 03:37 |
| sdake_ | who convinced me to write a manifesto | 03:37 |
| sdake_ | which our community did :) | 03:37 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix gnocchi upgrade permission issue https://review.openstack.org/387779 | 03:37 |
| sdake_ | pavo complictaed and not the right time to ask that quesiton | 03:38 |
| sdake_ | happy to answer next week after summit | 03:38 |
| sdake_ | it will take a couple hours to walk you through it | 03:39 |
| Pavo | lol yeah I figured it would be a pretty long answer | 03:39 |
| sdake_ | Jeffrey4l https://review.openstack.org/#/c/387779/1/ansible/roles/gnocchi/templates/gnocchi-metricd.json.j2 | 03:40 |
| sdake_ | Jeffrey4l does that look correct (gnocchi:kolla) | 03:40 |
| Pavo | so I noticed that kolla has elaticsearch connected to openstack, what is it collecting from, all the APIs? | 03:40 |
| sdake_ | pavo it collects all the logs that we collect - which are most of them via heka | 03:41 |
| sdake_ | heka loads logs into es, which can be visualized with kibana | 03:41 |
| Pavo | ok | 03:41 |
| Pavo | we use ELK and SO for that | 03:42 |
| Jeffrey4l | sdake_ gnocchi-api is started by using apache, which run as root, so we need change the /var/log/kolla/gnocchi first. | 03:43 |
| inc0 | Pavo, http://docs.openstack.org/infra/manual/developers.html | 03:43 |
| sdake_ | Jeffrey4l are the uids correct in that patch | 03:43 |
| sdake_ | i've never used gnocchi | 03:43 |
| Pavo | have you guys thought about switching apache to nginx? | 03:43 |
| openstackgerrit | Merged openstack/kolla: Fix congress upgrade permission issue https://review.openstack.org/387762 | 03:43 |
| Jeffrey4l | for ubuntu, it is not correct. ( systemd user ) | 03:44 |
| Pavo | inc0 that is prefect | 03:44 |
| Pavo | ty so much | 03:44 |
| Jeffrey4l | centos should be fine. | 03:44 |
| openstackgerrit | Merged openstack/kolla: Fix nova upgrade permission issue https://review.openstack.org/387753 | 03:44 |
| inc0 | yeah this piece of docs is well written | 03:44 |
| *** eaguilar has joined #openstack-kolla | 03:44 | |
| sdake_ | pavo no - nginx is not really how people recommend to deploy keystone/horizon | 03:45 |
| sdake_ | if that changes, i guess we could change it around | 03:45 |
| Pavo | is there a reason | 03:45 |
| inc0 | not sure why tho | 03:45 |
| inc0 | it's wsgi | 03:45 |
| sdake_ | pavo i dont know of any good reason | 03:45 |
| sdake_ | pavo then again i dont know of any good reason to use nginx over apache | 03:46 |
| Jeffrey4l | wanna know why use apache rather than nginx. ;) | 03:46 |
| Pavo | I would think nginx would be a better web service even though its not as mature as apache | 03:46 |
| Jeffrey4l | but apache is widely used in openstack. | 03:46 |
| *** david-lyle has joined #openstack-kolla | 03:46 | |
| inc0 | tbh it doesn't matter that much | 03:46 |
| Pavo | true | 03:46 |
| *** david-lyle has quit IRC | 03:46 | |
| inc0 | load is not a problem on this layer | 03:46 |
| sdake_ | i measured our cloud under overload at 60% cpu idle | 03:47 |
| inc0 | control plane performance is rarely an issue | 03:47 |
| sdake_ | 120ish nodes | 03:47 |
| inc0 | rabbitmq usually dies first | 03:47 |
| Pavo | I just think its easier to do backend proxies with nginx than it is with apache | 03:47 |
| sdake_ | ya that damn rabbitmq | 03:47 |
| sdake_ | pile of crap | 03:47 |
| Jeffrey4l | unless you are deploy a public cloud, with lots of user. | 03:47 |
| sdake_ | lets drink to erlang! | 03:47 |
| *** huhaoran has quit IRC | 03:47 | |
| inc0 | Jeffrey4l, still, usually it's rabbitmq that dies | 03:48 |
| *** huhaoran has joined #openstack-kolla | 03:48 | |
| inc0 | if you do multiregion with keystone | 03:48 |
| Jeffrey4l | inc0, ya. it is true | 03:48 |
| inc0 | then it starts to matter | 03:48 |
| sdake_ | multiregion with keystone = multile rabbitmqs | 03:48 |
| inc0 | but we're talking about multi-thousand setups | 03:48 |
| sdake_ | keystone doesn't use rabbitmq | 03:48 |
| inc0 | but uses apache | 03:48 |
| Pavo | is rabbitmq the only messaging service there is? | 03:48 |
| sdake_ | pavo there is a slew of em | 03:48 |
| *** eaguilar has quit IRC | 03:49 | |
| *** yuanying has joined #openstack-kolla | 03:49 | |
| inc0 | rabbitmq is best in terms of amqp | 03:49 |
| inc0 | amqp protocol is not ideal | 03:49 |
| sdake_ | infra uses zeromq iirc | 03:49 |
| sdake_ | blame carl for that one inc0 | 03:49 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix gnocchi upgrade permission issue https://review.openstack.org/387779 | 03:49 |
| Pavo | oh ok I understand now | 03:49 |
| Pavo | didn't think about amqp | 03:50 |
| *** david-lyle has joined #openstack-kolla | 03:50 | |
| inc0 | if you don't care that much about few vms failing to spawn you can go zmq | 03:50 |
| inc0 | without persistence it's way faster | 03:50 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix grafana upgrade permission issue https://review.openstack.org/387774 | 03:51 |
| inc0 | zmq creates kinda mesh so it's really strange thing to run | 03:51 |
| Jeffrey4l | is there any other defect about zmp? | 03:51 |
| *** david-lyle has quit IRC | 03:51 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix neutron upgrade permission issue https://review.openstack.org/387780 | 03:52 |
| inc0 | it's not as much defective as it is different from what you know | 03:52 |
| sdake_ | inc0 rabbitmq ca nbe setup in a mesh too | 03:52 |
| *** ssurana has joined #openstack-kolla | 03:53 | |
| *** diogogmt has quit IRC | 03:53 | |
| sdake_ | erlang weak link with rabbitmq | 03:53 |
| Jeffrey4l | i know it does not havecentral broker | 03:53 |
| *** david-lyle has joined #openstack-kolla | 03:53 | |
| sdake_ | why they didn't choose a proper language - fail | 03:53 |
| openstackgerrit | Merged openstack/kolla: Fix grafana upgrade permission issue https://review.openstack.org/387768 | 03:54 |
| *** david-lyle has quit IRC | 03:54 | |
| inc0 | anyway I'm going off | 03:54 |
| inc0 | tomorrow we have big people visiting and I need to be well awake | 03:54 |
| openstackgerrit | Merged openstack/kolla: Fix neutron upgrade permission issue https://review.openstack.org/387212 | 03:54 |
| inc0 | have a good night guys | 03:54 |
| Pavo | yeah I'm gonna head to bed myself got PT @0500 | 03:54 |
| Jeffrey4l | good night. | 03:54 |
| Pavo | have good one | 03:54 |
| sdake_ | nite pavo | 03:54 |
| Pavo | see ya tomorrow | 03:54 |
| *** inc0 has quit IRC | 03:54 | |
| Pavo | nite | 03:54 |
| Jeffrey4l | nite Pavo | 03:55 |
| *** david-lyle has joined #openstack-kolla | 03:55 | |
| *** david-lyle has quit IRC | 03:55 | |
| sdake_ | how many containers are needed to be converted for upgrade | 03:56 |
| sdake_ | are we closing in on done? | 03:56 |
| openstackgerrit | Merged openstack/kolla: Fix influxdb upgrade permission issue https://review.openstack.org/387764 | 03:57 |
| HyperJohnGraham_ | define done ;) | 03:57 |
| *** david-lyle has joined #openstack-kolla | 03:57 | |
| sdake_ | patches merged, upgrade from 2.0.2 to 3.0.0 works on ubuntu and centos | 03:57 |
| sdake_ | DOD ^^ | 03:57 |
| *** david-lyle has quit IRC | 03:57 | |
| openstackgerrit | Merged openstack/kolla: Fix heat upgrade permission issue https://review.openstack.org/387761 | 03:58 |
| HyperJohnGraham_ | so its safe for me to try a multinode install now ? | 03:58 |
| *** david-lyle has joined #openstack-kolla | 03:58 | |
| sdake_ | HyperJohnGraham_ - speaking of DOD for kolla repo not kolla-k8s repo | 03:59 |
| *** david-lyle has quit IRC | 03:59 | |
| HyperJohnGraham_ | or soon | 03:59 |
| HyperJohnGraham_ | gotcha | 03:59 |
| *** david-lyle has joined #openstack-kolla | 04:00 | |
| *** david-lyle has quit IRC | 04:01 | |
| sdake_ | 116 json filess | 04:01 |
| sdake_ | groan | 04:01 |
| *** david-lyle has joined #openstack-kolla | 04:01 | |
| *** david-lyle has quit IRC | 04:04 | |
| sdake_ | aodh needs work | 04:04 |
| sdake_ | barbican needs work | 04:04 |
| sdake_ | ceilometer needs work | 04:04 |
| *** david-lyle has joined #openstack-kolla | 04:04 | |
| sdake_ | ceph-mon needs work | 04:05 |
| sdake_ | all ceph needs work | 04:05 |
| *** david-lyle has quit IRC | 04:05 | |
| *** huhaoran has quit IRC | 04:05 | |
| sdake_ | hmm maybe my search is wrong | 04:05 |
| *** huhaoran has joined #openstack-kolla | 04:06 | |
| *** david-lyle has joined #openstack-kolla | 04:06 | |
| sdake_ | we have converted 57 of 116 json files | 04:07 |
| sdake_ | STDAKE-M-26CJ:kolla sdake$ find . -name \*json.j2 -exec git grep -c permission {} \; | wc -l | 04:07 |
| Jeffrey4l | sdake_, check the comments in ironic https://review.openstack.org/#/c/387180/1 | 04:08 |
| *** david-lyle has quit IRC | 04:08 | |
| Jeffrey4l | sdake_, lots of services are added during newton. i.e, upgrade won't affect. So no need fix the json file right now. sdake_ | 04:09 |
| sdake_ | Jeffrey4l ok | 04:09 |
| Jeffrey4l | sdake_, check the list here https://etherpad.openstack.org/p/kolla-bug-1631503 all essential service need be fixed are almost done. | 04:09 |
| sdake_ | Jeffrey4l nice | 04:09 |
| *** david-lyle has joined #openstack-kolla | 04:10 | |
| *** david-lyle has quit IRC | 04:11 | |
| sdake_ | 21 new projects | 04:11 |
| sdake_ | boy we move fast | 04:11 |
| *** haplo37_ has quit IRC | 04:11 | |
| sdake_ | we can fix the new projects in 3.0.1 | 04:12 |
| *** david-lyle has joined #openstack-kolla | 04:12 | |
| sdake_ | i think once existing projects are done, we can turn to testing for the next 15 hours :) | 04:12 |
| Jeffrey4l | ya. | 04:12 |
| sdake_ | zhubingbing you up for that | 04:13 |
| zhubingbing | i am here | 04:13 |
| *** david-lyle has quit IRC | 04:13 | |
| Jeffrey4l | btw, i want to use unified uid in O and then the permissions field can be removed. | 04:13 |
| *** haplo37_ has joined #openstack-kolla | 04:13 | |
| zhubingbing | i agree， i will go to test | 04:13 |
| *** david-lyle has joined #openstack-kolla | 04:14 | |
| *** david-lyle has quit IRC | 04:15 | |
| *** david-lyle has joined #openstack-kolla | 04:15 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix for cinder not working with permissions changes https://review.openstack.org/387785 | 04:16 |
| *** david-lyle has quit IRC | 04:16 | |
| *** david-lyle has joined #openstack-kolla | 04:17 | |
| sdake_ | Jeffrey4l can you ack https://review.openstack.org/#/c/387779/2 | 04:18 |
| *** david-lyle has quit IRC | 04:18 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix wrong magnum volume path in json file https://review.openstack.org/387786 | 04:18 |
| Jeffrey4l | done. | 04:18 |
| *** david-lyle has joined #openstack-kolla | 04:19 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix gnocchi upgrade permission issue https://review.openstack.org/387779 | 04:20 |
| *** david-lyle has quit IRC | 04:20 | |
| sdake_ | zhubingbing something is wrong with this cherrypick https://review.openstack.org/#/c/387779/ | 04:21 |
| *** david-lyle has joined #openstack-kolla | 04:21 | |
| sdake_ | zhubingbing it says "master" as branch - so shouldn't be a cherrypick | 04:21 |
| zhubingbing | ok | 04:21 |
| zhubingbing | understand | 04:22 |
| sdake_ | why was it cherrypicked? | 04:22 |
| Jeffrey4l | sdake_, last patch : only message is changed. | 04:23 |
| zhubingbing | i forget change barch | 04:23 |
| sdake_ | zhubingbing use the cherrypick button in gerrit | 04:23 |
| sdake_ | put in stable/newton | 04:23 |
| sdake_ | saves time | 04:23 |
| Jeffrey4l | zhubingbing, you handle this https://review.openstack.org/#/c/387779/ in wrong. | 04:23 |
| Jeffrey4l | it isn't be merged. | 04:24 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix gnocchi upgrade permission issue https://review.openstack.org/387788 | 04:24 |
| zhubingbing | em-) | 04:24 |
| Jeffrey4l | i will restore patch set 2 | 04:24 |
| Jeffrey4l | zhubingbing, check this https://review.openstack.org/#/q/If22b6cf9fa3ce9f1c18e4c9a71c637e96e7c2347,n,z | 04:24 |
| zhubingbing | ok | 04:24 |
| sdake_ | Jeffrey4l should there not be a recurse here: https://review.openstack.org/#/c/387788/1/ansible/roles/gnocchi/templates/gnocchi-api.json.j2 | 04:25 |
| *** eaguilar has joined #openstack-kolla | 04:25 | |
| Jeffrey4l | sdake_, no. api is run as root. so just change the folder permission | 04:25 |
| Jeffrey4l | some file in log folder is root, others are gnocchi. | 04:26 |
| Jeffrey4l | so recurse is wrong. | 04:27 |
| openstackgerrit | Merged openstack/kolla: Fix mistral upgrade permission issue https://review.openstack.org/387760 | 04:27 |
| *** ssurana has quit IRC | 04:29 | |
| Jeffrey4l | how to use patch 2 in gerrit? | 04:29 |
| sdake_ | Jeffrey4l it must be cherrypciekd and re-git reviewed | 04:30 |
| *** eaguilar has quit IRC | 04:30 | |
| sdake_ | i dont know of any other way to revert to an older ps | 04:30 |
| *** sdake_ has quit IRC | 04:30 | |
| *** unicell has joined #openstack-kolla | 04:30 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix gnocchi upgrade permission issue https://review.openstack.org/387779 | 04:31 |
| Jeffrey4l | ok. cherrypick works. | 04:31 |
| *** unicell has quit IRC | 04:32 | |
| *** sdake has joined #openstack-kolla | 04:33 | |
| sdake | Jeffrey4l when was our big tent applicationa gain? | 04:34 |
| sdake | you had the date earlier | 04:34 |
| *** unicell1 has joined #openstack-kolla | 04:35 | |
| Jeffrey4l | Join OpenStack Big Tent at Aug 5, 2015 ( this come from the approve PS ) sdake | 04:36 |
| sdake | Jeffrey4l thanks | 04:38 |
| *** huhaoran has quit IRC | 04:40 | |
| *** huhaoran has joined #openstack-kolla | 04:40 | |
| Jeffrey4l | does openstack have any policy ,rule or scheduler to release z stream? | 04:45 |
| Jeffrey4l | sdake, | 04:45 |
| *** unicell has joined #openstack-kolla | 04:45 | |
| sdake | Jeffrey4l our unofficial policy is every 45 days | 04:45 |
| sdake | although we havne't done that since 2.0.2 because of the crazyness around 3.0.0 release | 04:46 |
| sdake | we can release z streams whenever we like | 04:46 |
| Jeffrey4l | roger. | 04:46 |
| sdake | i'd ask that patches go in master first for the future of follows stable policy | 04:46 |
| *** unicell1 has quit IRC | 04:46 | |
| sdake | obviously if there is a security problem we will release a zstream immediately after fixing | 04:46 |
| *** asalkeld has quit IRC | 04:47 | |
| Jeffrey4l | got. | 04:47 |
| sdake | that said, i'd rather 3.0.0 work out of the box for all of our major features | 04:49 |
| sdake | including the core services | 04:49 |
| duonghq | what is zstream? | 04:49 |
| sdake | x.y.z | 04:49 |
| sdake | 2.0.2 | 04:49 |
| sdake | 2.0.3 would be a zstream | 04:49 |
| sdake | 2.0.4 woul dbe a new z stream | 04:49 |
| sdake | 3.0.0 is a x stream | 04:49 |
| duonghq | understood | 04:49 |
| *** yuanying has quit IRC | 04:54 | |
| *** zhangyufei has quit IRC | 04:55 | |
| sdake | we need those newton patches merged asap | 04:56 |
| sdake | any other core reviewers around? | 04:56 |
| sdake | egonzalez90 around? | 04:57 |
| sdake | mandre around? | 04:58 |
| *** unicell1 has joined #openstack-kolla | 04:58 | |
| sdake | pbourke_ around? | 04:58 |
| *** unicell has quit IRC | 04:59 | |
| sdake | Jeffrey4l if i fall asleep at keyboard take care to hound other core reviewers to review the last 6 or so patches | 04:59 |
| sdake | in the meantime lets begin testing | 05:00 |
| *** yuanying has joined #openstack-kolla | 05:00 | |
| *** yuanying has quit IRC | 05:02 | |
| *** zhangyufei has joined #openstack-kolla | 05:03 | |
| *** zhangyufei has quit IRC | 05:04 | |
| Jeffrey4l | ok | 05:07 |
| *** sdake_ has joined #openstack-kolla | 05:11 | |
| *** sdake has quit IRC | 05:14 | |
| openstackgerrit | Merged openstack/kolla: Fix gnocchi upgrade permission issue https://review.openstack.org/387779 | 05:16 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix gnocchi upgrade permission issue https://review.openstack.org/387788 | 05:18 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix aodh upgrade permission issue https://review.openstack.org/387793 | 05:26 |
| sdake_ | Jeffrey4l i am deploying 2.0.2 AIO | 05:33 |
| sdake_ | and receive following error: | 05:33 |
| sdake_ | http://paste.fedoraproject.org/454784/68868147/ -> http://paste.fedoraproject.org/454784/68868147 | 05:34 |
| sdake_ | i am on minime-03 | 05:34 |
| sdake_ | yet it is looking for minime-02 | 05:34 |
| * Jeffrey4l is checking | 05:35 | |
| Jeffrey4l | sdake_, which folder and user are u using? | 05:36 |
| *** schwicht has joined #openstack-kolla | 05:36 | |
| *** tonanhngo has quit IRC | 05:37 | |
| *** tonanhngo has joined #openstack-kolla | 05:38 | |
| sdake_ | sudo kolla-ansible deploy | 05:42 |
| sdake_ | running from ~/ | 05:42 |
| sdake_ | pip installed kolla | 05:42 |
| sdake_ | running as user sdake | 05:42 |
| Jeffrey4l | got. weird. | 05:42 |
| sdake_ | Jeffrey4l / was full, i am going to reboot | 05:44 |
| *** skramaja has quit IRC | 05:51 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix barbican upgrade permission issue https://review.openstack.org/387797 | 05:56 |
| *** skramaja has joined #openstack-kolla | 05:57 | |
| sdake_ | Jeffrey4l i think i had 1 vip being served by two machines | 06:01 |
| mandre | sdake: around now | 06:01 |
| mandre | hello :) | 06:01 |
| sdake_ | Jeffrey4l as i had minime-01 and minime-02 still running | 06:01 |
| sdake_ | mandre long time no tlak | 06:01 |
| *** yuanying has joined #openstack-kolla | 06:01 | |
| sdake_ | mandre need some changes acked if you ahve a moment | 06:01 |
| Jeffrey4l | ok | 06:01 |
| mandre | yeah, point me to them sdake_ | 06:02 |
| Jeffrey4l | mandre, https://review.openstack.org/#/q/topic:bug/1631503+is:open | 06:02 |
| sdake_ | yup thats the one :) | 06:02 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix cloudkitty upgrade permission issue https://review.openstack.org/387800 | 06:03 |
| mandre | thx Jeffrey4l | 06:03 |
| mandre | reading the bug report to get a bit of context | 06:04 |
| Jeffrey4l | thanks. | 06:04 |
| sdake_ | up that was it | 06:07 |
| sdake_ | up/yup | 06:07 |
| mandre | Jeffrey4l: is that the part that understands the new 'permissions' key? https://review.openstack.org/#/c/386412/1/docker/base/set_configs.py | 06:09 |
| mandre | or is there more to it? | 06:09 |
| Jeffrey4l | no more. that's it. | 06:10 |
| sdake_ | mandre we have spent 10s of hours bikeshedding the design | 06:10 |
| sdake_ | deadline is in about 12 hours to tag | 06:10 |
| mandre | cool | 06:10 |
| sdake_ | just trust that it works ;) | 06:10 |
| mandre | yep, it's good, just wanted to know if I wasn't missing anything | 06:11 |
| *** mnasiadka has joined #openstack-kolla | 06:12 | |
| mandre | do we still expect the perms to be set in the extend_start.sh? | 06:14 |
| Jeffrey4l | mandre, we did not talked this. But i prefer to remove permissions after we implemented unified uid/gid. | 06:15 |
| mnasiadka | morning | 06:15 |
| *** zhangyufei has joined #openstack-kolla | 06:16 | |
| mnasiadka | https://bugs.launchpad.net/kolla/+bug/1634107 - I think this bug is critical, if there are no volunteers to fix it - please assign it to me | 06:16 |
| openstack | Launchpad bug 1634107 in kolla "ubuntu builds fail on ceph packages" [Undecided,Confirmed] | 06:16 |
| mandre | Jeffrey4l: one more question :) | 06:16 |
| Jeffrey4l | mnasiadka, it is fixed. | 06:16 |
| Jeffrey4l | but with another bug. | 06:17 |
| sdake_ | mnasiadka i fixed that this mornign or jeffrey did | 06:17 |
| Jeffrey4l | let me find it. | 06:17 |
| mnasiadka | ok good, then mark it as duplicate | 06:17 |
| mandre | some commits appear to be for master, and some other to stable/newton | 06:17 |
| sdake_ | mandre right - stable/newton should be rubber stamps | 06:17 |
| sdake_ | if they are broken make note of it and fix in a followon patch | 06:17 |
| mandre | I'm surprised the ones for stable/newton are not cherry-picks | 06:17 |
| sdake_ | so we can run in tune with follows-stable policy | 06:17 |
| Jeffrey4l | mnasiadka, here https://bugs.launchpad.net/kolla/+bug/1633888 | 06:18 |
| openstack | Launchpad bug 1633888 in kolla newton "after ceph outage, ceph ubuntu totally broken" [Critical,Fix committed] - Assigned to Jeffrey Zhang (jeffrey4l) | 06:18 |
| sdake_ | mandre they are but gerrit didn't mark them as such | 06:18 |
| sdake_ | not sure why | 06:18 |
| mandre | oh, ok | 06:18 |
| Jeffrey4l | mandre, we use etherpad for tracking https://etherpad.openstack.org/p/kolla-bug-1631503 | 06:18 |
| sdake_ | maybe because the parent patch hadn't merged yetwhen the cherrypick was done | 06:18 |
| openstackgerrit | Merged openstack/kolla: Fix grafana upgrade permission issue https://review.openstack.org/387774 | 06:18 |
| mnasiadka | Jeffrey4l: let me rebuild and check | 06:18 |
| mandre | sdake_: that makes sense | 06:18 |
| *** huhaoran has quit IRC | 06:19 | |
| Jeffrey4l | fyi bug 1634107 marked as duplication now. | 06:19 |
| openstack | bug 1633888 in kolla newton "duplicate for #1634107 after ceph outage, ceph ubuntu totally broken" [Critical,Fix committed] https://launchpad.net/bugs/1633888 - Assigned to Jeffrey Zhang (jeffrey4l) | 06:19 |
| *** huhaoran has joined #openstack-kolla | 06:19 | |
| sdake_ | Jeffrey4l 2.0.2 works on my box - now time for upgrade ;) | 06:23 |
| Jeffrey4l | cool | 06:23 |
| sdake_ | heat works as well | 06:23 |
| sdake_ | yay :) | 06:23 |
| Jeffrey4l | sdake_, i think we can remove liberty and mitaka( not sure) now in https://bugs.launchpad.net/kolla/+bug/1631503 | 06:24 |
| openstack | Launchpad bug 1631503 in kolla "inconsistent UID in named volume results in inability to upgrade" [Critical,In progress] - Assigned to Steven Dake (sdake) | 06:24 |
| openstackgerrit | Merged openstack/kolla: Fix for cinder not working with permissions changes https://review.openstack.org/387785 | 06:25 |
| sdake_ | Jeffrey4l have a reload | 06:25 |
| openstackgerrit | Merged openstack/kolla: Fix wrong magnum volume path in json file https://review.openstack.org/387786 | 06:26 |
| openstackgerrit | Martin André proposed openstack/kolla: Fix syntax error in sahara permissions https://review.openstack.org/387806 | 06:28 |
| mandre | sdake_, Jeffrey4l: ^^ | 06:28 |
| mandre | this needs to be cherry-picked once it merges | 06:28 |
| Jeffrey4l | sdake_, hmm does rabbitmq issue on centos affect? since we upgrade rabbitmq in mitaka. | 06:28 |
| Jeffrey4l | mandre, got. thanks. | 06:29 |
| sdake_ | Jeffrey4l not sure 2.0.2 works - haven' ttried 3.0.0 | 06:29 |
| sdake_ | Jeffrey4l ack that real quick like pls :) | 06:29 |
| Jeffrey4l | done. | 06:29 |
| sdake_ | i'd like to lock down stable/newton now | 06:29 |
| sdake_ | or soonish | 06:29 |
| sdake_ | so i can get some sleep tonight :) | 06:29 |
| Jeffrey4l | but need wait the gate. :/ | 06:29 |
| *** hogepodge has quit IRC | 06:30 | |
| sdake_ | zhubingbing - can you sue a different bug id for other services | 06:30 |
| sdake_ | or can we just stop adding non-mitaka services | 06:31 |
| sdake_ | Jeffrey4l just insepct with eyeballs, our gate doesn't test sahara anyway | 06:31 |
| sdake_ | Jeffrey4l we need to do a final sanity test | 06:31 |
| sdake_ | which is what the next 8-12 hrs are about | 06:31 |
| Jeffrey4l | yep. | 06:32 |
| sdake_ | so ironic is wht remains? | 06:32 |
| Jeffrey4l | i will tell zhubingbing to stop push new PS for non-mitaka service in next 12 hours. | 06:32 |
| sdake_ | oh and neutron | 06:32 |
| sdake_ | Jeffrey4l i think we need to halt all new service reviews, or put them in a differnet bug id | 06:33 |
| sdake_ | its really hard to keep track of whats happening | 06:33 |
| sdake_ | mandre if you can ack https://review.openstack.org/#/c/387780/1 | 06:33 |
| Jeffrey4l | yes. how about -2 for already pushed before rc3 is tagged. | 06:33 |
| mandre | sdake_: one moment, looking at gnocchi atm | 06:34 |
| *** asalkeld has joined #openstack-kolla | 06:34 | |
| sdake_ | Jeffrey4l i think the problem there is communication of the -2ss | 06:34 |
| sdake_ | and we areclosing out the bug | 06:34 |
| sdake_ | non-mitaka services should really use a different bug id | 06:35 |
| sdake_ | we have already merged some and those can't be undone | 06:35 |
| sdake_ | but they can be categorized in a different bug id | 06:35 |
| Jeffrey4l | how categorized in a different bug id? | 06:35 |
| Jeffrey4l | the commit message changed not be updated too. | 06:36 |
| sdake_ | mitaka services = bug 1631503 | 06:36 |
| openstack | bug 1631503 in kolla ocata "inconsistent UID in named volume results in inability to upgrade" [Critical,In progress] https://launchpad.net/bugs/1631503 - Assigned to Steven Dake (sdake) | 06:36 |
| sdake_ | newton services = NEW bug | 06:36 |
| sdake_ | new bug may not be necessary anyway if we are unifyign the uid space | 06:36 |
| Jeffrey4l | yes. | 06:37 |
| Jeffrey4l | anyway, i'll -2 w/ some explain to stop the patch to merged now. | 06:37 |
| sdake_ | just request a different bug id | 06:37 |
| sdake_ | and put in the etherpad notes | 06:37 |
| openstackgerrit | Merged openstack/kolla: Fix gnocchi upgrade permission issue https://review.openstack.org/387788 | 06:37 |
| Jeffrey4l | ok. | 06:38 |
| sdake_ | last two patches needing reviews are: | 06:38 |
| sdake_ | https://review.openstack.org/#/c/387180/1 | 06:38 |
| sdake_ | https://review.openstack.org/#/c/387780/1 | 06:39 |
| sdake_ | egonzalez90 mandre Jeffrey4l :) ^^ | 06:40 |
| openstackgerrit | Martin André proposed openstack/kolla: Fix syntax error in sahara permissions https://review.openstack.org/387811 | 06:42 |
| mandre | hmm, a bit too quick, I thought it was merged in master already | 06:44 |
| openstackgerrit | Merged openstack/kolla: Fix neutron upgrade permission issue https://review.openstack.org/387780 | 06:44 |
| *** sdake_ is now known as sdake | 06:44 | |
| *** ChanServ sets mode: +o sdake | 06:44 | |
| Jeffrey4l | mandre, hmm. should be OK. since we won't update the master branch PS. ;) | 06:44 |
| *** mtaylor22 has quit IRC | 06:45 | |
| mnasiadka | by the way - has anybody thought about introducing ipxe in ironic containers? tftp is lousy and unreliable in production... | 06:45 |
| *** sdake changes topic to "Please TEST stable/newton until Oct 18th 23:45 UTC when rc3 will be tagged; RC3 is our final!; Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA)" | 06:45 | |
| Jeffrey4l | mnasiadka, no i think. we can implement this in Octava. | 06:47 |
| sdake | needs an ack: https://review.openstack.org/#/c/387180/1 | 06:48 |
| Jeffrey4l | mnasiadka, is ipxe widely supported? | 06:48 |
| Jeffrey4l | after it is merged. we still need backport it ^^ this is non-mitaka branch service, so it is not affect upgrade from mitaka to newton | 06:51 |
| Jeffrey4l | after it is merged. we still need backport it ^^ | 06:51 |
| mandre | Jeffrey4l: regarding https://review.openstack.org/#/c/387180/1, shouldn't we set the permissions for /tftpboot for the ironic-pxe container? | 06:51 |
| Jeffrey4l | ironic-pxe runs as root. | 06:52 |
| Jeffrey4l | so nothing to do in pxe container. | 06:52 |
| Jeffrey4l | mandre, ^ | 06:52 |
| *** tovin07 has quit IRC | 06:53 | |
| mandre | Jeffrey4l: I though we didn't have to do anything because /tftpboot is on a named volume and we already set the permission in ironic-conductor :) | 06:53 |
| Jeffrey4l | mandre, but only for bootstrap... hmm it should be a bug. these script should run all the times. | 06:54 |
| mnasiadka | Jeffrey4l: well, I'd like to use it - that's why I'm asking. I know there's not much work to be done :) | 06:54 |
| Jeffrey4l | bootstrap only run on one of node. | 06:55 |
| Jeffrey4l | so need: remove the KOLLA_BOOTSTRAP if condiction and run the script all the time. | 06:56 |
| Jeffrey4l | we can remove bootstrap_container in ansible, too. it will be useless. | 06:56 |
| Jeffrey4l | OK. i am trying to remove the permissions section in ironic-conductor. ironic-pxe will change the own during bootstrap. | 06:59 |
| zhubingbing | ping √ | 07:00 |
| zhubingbing | ping Mauricio Lima | 07:00 |
| openstackgerrit | Merged openstack/kolla: Fix ironic upgrade permission issue https://review.openstack.org/387180 | 07:00 |
| mandre | Jeffrey4l: it's a bit weird but I think we're good as long as we run ironic-conductor along with ironic-pxe | 07:00 |
| Jeffrey4l | mnasiadka, is it possible to run multi ironic-pxe and ironic-conductor? how the nodes get the correct url? loadbalance? | 07:00 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix ironic upgrade permission issue https://review.openstack.org/387822 | 07:01 |
| Jeffrey4l | hmm.. merged :/ | 07:01 |
| Jeffrey4l | mandre, i have to say it works but not optimal. | 07:01 |
| *** eaguilar has joined #openstack-kolla | 07:02 | |
| Jeffrey4l | zhubingbing, you need ping the irc id not the real name. like you should ping Jeffrey4l rather than Jeffrey Zhang | 07:02 |
| mandre | Jeffrey4l: it's a named volume, so every change made to /tftpboot in ironic-conductor also affect ironic-pxe | 07:02 |
| Jeffrey4l | yes. i known that. | 07:02 |
| sdake | the idea is to get something working not something perfect ;-) | 07:03 |
| mandre | Jeffrey4l: agreed, I would expect to have ironic-pxe set the perms | 07:03 |
| Jeffrey4l | OK. | 07:03 |
| Jeffrey4l | actually, pxe shouldn't touch the files created by ironic-conductor. that's why i put it into conductor. | 07:04 |
| *** msimonin has joined #openstack-kolla | 07:04 | |
| *** coolsvap has joined #openstack-kolla | 07:04 | |
| Jeffrey4l | ironic backport is the last one. lol | 07:05 |
| sdake | coolsvap need an ack on https://review.openstack.org/#/c/387822/1 | 07:05 |
| * coolsvap checks | 07:06 | |
| *** eaguilar has quit IRC | 07:06 | |
| mandre | are we good with the permissions patches? | 07:06 |
| sdake | mandre wont know until they merge and can test an upgrade | 07:07 |
| *** shardy has joined #openstack-kolla | 07:08 | |
| mandre | hey, I noticed gnocchi-api runs as root, is there a reason for it? | 07:08 |
| Jeffrey4l | mandre, apache run with wsgi | 07:08 |
| bjolo__ | morning | 07:08 |
| Jeffrey4l | bjolo__, sup | 07:08 |
| sdake | sup bjolo__ | 07:08 |
| *** msimonin1 has joined #openstack-kolla | 07:09 | |
| sdake | bjolo__ see topic :) | 07:09 |
| sdake | this is it | 07:09 |
| bjolo__ | got it | 07:09 |
| sdake | we are about 30 mins out from all patches being merged | 07:09 |
| bjolo__ | ok | 07:09 |
| mandre | Jeffrey4l: ack | 07:09 |
| Jeffrey4l | thanks. | 07:09 |
| bjolo__ | any particular os and install type that you want tested | 07:09 |
| sdake | we are all centos dudes | 07:09 |
| Jeffrey4l | bjolo__, Ubuntu upgrade test ;) | 07:09 |
| sdake | ubuntu would be good :) | 07:10 |
| bjolo__ | ok | 07:10 |
| sdake | from 2.0.2 to 3.0.0 | 07:10 |
| sdake | with stable/newton as your branch | 07:10 |
| bjolo__ | ubuntu binary 2.0.2 images from docker hub ok | 07:10 |
| bjolo__ | or should i build my own> | 07:10 |
| sdake | yup | 07:10 |
| sdake | use hub | 07:10 |
| bjolo__ | ok | 07:10 |
| *** msimonin has quit IRC | 07:10 | |
| sdake | take care to deploy the 2.0.2 images with ansible 1.9 and kolla 2.0.2 pip installed | 07:11 |
| sdake | then remove ansible1.9 | 07:11 |
| bjolo__ | hmm master branch build fails for centos?! | 07:11 |
| bjolo__ | :Cannot open: http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-3.noarch.rpm. | 07:11 |
| Jeffrey4l | network issue? i can open it. | 07:11 |
| sdake | bjolo__ that link works for me | 07:11 |
| sdake | bjolo__ git checkout stable/newton | 07:12 |
| bjolo__ | me too but build fails | 07:12 |
| bjolo__ | ? | 07:12 |
| bjolo__ | hmm let me ts this | 07:13 |
| sdake | bjolo__ mabye your out of disk | 07:14 |
| bjolo__ | ERROR:kolla.image.build.base:The command '/bin/sh -c yum -y install http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-3.noarch.rpm && yum clean all' returned a non-zero code: 1 | 07:14 |
| sdake | bjolo__ do you have kolla 2.0.2 pip installed? | 07:14 |
| *** matrohon has joined #openstack-kolla | 07:14 | |
| bjolo__ | plenty of disk | 07:15 |
| *** schwicht has quit IRC | 07:15 | |
| bjolo__ | im building master and stable/newton on two seperate nodes | 07:15 |
| sdake | ok | 07:15 |
| bjolo__ | same build script ive been using for weeks now | 07:15 |
| bjolo__ | centos source/binary and ubuntu source/binary | 07:15 |
| *** schwicht has joined #openstack-kolla | 07:15 | |
| sdake | stable/newton builds but master does not? | 07:15 |
| bjolo__ | checking | 07:16 |
| bjolo__ | i have it cron to build daily | 07:16 |
| *** clayton has quit IRC | 07:16 | |
| bjolo__ | need to go through the setup and logs here before concluding | 07:16 |
| sdake | if you haven't built recently percona had an outage earlier today | 07:17 |
| sdake | try --no-cache | 07:17 |
| sdake | stable/newton is building for me | 07:17 |
| *** rbbratta1 has quit IRC | 07:17 | |
| sdake | and its ubilt for the gate most of the day ;) | 07:17 |
| *** rbbratta1 has joined #openstack-kolla | 07:18 | |
| bjolo__ | ./build.py --registry dockreg.mydomain.net:4000 --push --nocache --threads 64 --base centos --type source --tag $TAG | 07:18 |
| sdake | threads 64 is a problem i tihnk | 07:18 |
| sdake | try the default | 07:18 |
| bjolo__ | ok no threads flag | 07:18 |
| sdake | --no-cache i think is the option not --nocache | 07:18 |
| *** clayton has joined #openstack-kolla | 07:19 | |
| bjolo__ | --no-cache Do not use the Docker cache when building | 07:19 |
| bjolo__ | --nocache The inverse of --cache | 07:19 |
| bjolo__ | there is both :) | 07:19 |
| sdake | wow - who thought that was a good idea | 07:19 |
| Jeffrey4l | me.. we removed --no-cache in newton. | 07:20 |
| openstackgerrit | Merged openstack/kolla: Fix syntax error in sahara permissions https://review.openstack.org/387806 | 07:20 |
| sdake | why did we emove --no-cache in newton? | 07:21 |
| Jeffrey4l | the oslo.config add --nocache automatically. so we have two options in newtone and will be one in O. | 07:21 |
| sdake | changing apis annoys people ;) | 07:21 |
| Jeffrey4l | i explained. ;) | 07:21 |
| Jeffrey4l | several days ago. | 07:21 |
| sdake | ok | 07:21 |
| sdake | well anyway | 07:21 |
| sdake | building images :) | 07:21 |
| bjolo__ | ok removed threads and will try build again | 07:22 |
| sdake | bjolo__ if that doesn't work let me know i've got another thing for you to try | 07:22 |
| *** shardy has quit IRC | 07:22 | |
| Jeffrey4l | at first. we have --no-cache only, after added oslo.conf, it add --nono-cache automically. so we have --cache --no-cache --nocache --nono-cache in after that. | 07:22 |
| *** tonanhngo_ has joined #openstack-kolla | 07:22 | |
| Jeffrey4l | in O we remove --no-cache and --nono-cache. | 07:22 |
| sdake | i see | 07:22 |
| Jeffrey4l | we in O only --cache and --nocache is valid ;0 | 07:23 |
| sdake | seems pretty trerible of oslo to do that :) | 07:23 |
| Jeffrey4l | ;)_ | 07:23 |
| *** tonanhngo has quit IRC | 07:24 | |
| openstackgerrit | Merged openstack/kolla: Fix syntax error in sahara permissions https://review.openstack.org/387811 | 07:28 |
| *** ssurana has joined #openstack-kolla | 07:29 | |
| sdake | Jeffrey4l have you tried fernet as of late | 07:31 |
| Jeffrey4l | hmm. is the patch merged? | 07:32 |
| * Jeffrey4l is finding fix. | 07:32 | |
| *** tonanhngo_ has quit IRC | 07:32 | |
| sdake | Jeffrey4l was merged in rc2 pretty sure | 07:33 |
| *** tonanhngo has joined #openstack-kolla | 07:33 | |
| *** b_bezak has joined #openstack-kolla | 07:33 | |
| *** tonanhngo has quit IRC | 07:33 | |
| Jeffrey4l | yes. | 07:34 |
| sdake | i am building 3.0.0 images atm - to test an upgrade of basic compute kit + heat | 07:34 |
| Jeffrey4l | cool. | 07:34 |
| Jeffrey4l | i am prepare my env. will test fernet later. | 07:34 |
| *** ssurana has quit IRC | 07:34 | |
| *** b_bezak_ has joined #openstack-kolla | 07:37 | |
| *** b_bezak_ has quit IRC | 07:38 | |
| *** b_bezak has quit IRC | 07:40 | |
| *** hogepodge has joined #openstack-kolla | 07:40 | |
| openstackgerrit | Merged openstack/kolla: Fix ironic upgrade permission issue https://review.openstack.org/387822 | 07:42 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO_NOT_MERGE: TEST NEWTON BRANCH https://review.openstack.org/386981 | 07:46 |
| sdake | we need a merge passwords tool | 07:47 |
| zhubingbing | passwords tool？ | 07:48 |
| Jeffrey4l | sdake, extend the generate_password.py file to load from etc/kolla/passwords and add it into /etc/kolla/password.py | 07:48 |
| sdake | somtehing that forward migrates a password | 07:48 |
| sdake | Jeffrey4l ya - work for ocata ;) | 07:48 |
| Jeffrey4l | how about Newton? | 07:49 |
| sdake | i guess if it can be done without breaking things and causing alot of chaos | 07:49 |
| sdake | no password migration is a serious problem | 07:49 |
| sdake | i'd rather focus on testing at this point | 07:50 |
| *** b_bezak has joined #openstack-kolla | 07:52 | |
| *** huhaoran has quit IRC | 07:55 | |
| *** huhaoran has joined #openstack-kolla | 07:56 | |
| *** haplo37_ has quit IRC | 07:56 | |
| sdake | ok AIO upgrade from 2.0.2 to 3.0.0 going now | 07:56 |
| Jeffrey4l | i am running tempest for deployment ;) | 07:58 |
| sdake | little late for that Ithink ;-) | 07:58 |
| *** haplo37_ has joined #openstack-kolla | 07:58 | |
| Jeffrey4l | should be finished in 1 hour i think.. | 07:59 |
| sdake | and that will give ou what kind of data? | 07:59 |
| Jeffrey4l | may found new critical bug. | 07:59 |
| Jeffrey4l | test result for all api test. | 07:59 |
| Jeffrey4l | sdake, http://paste.openstack.org/show/586123/ | 08:00 |
| sdake | http://paste.fedoraproject.org/454808/67776151/ -> http://paste.fedoraproject.org/454808/67776151 | 08:00 |
| sdake | Jeffrey4l upgrade with vm running on btrfs | 08:00 |
| sdake | Jeffrey4l yes i've run tempest prior | 08:00 |
| Jeffrey4l | btrfs failed for sure. no workaround ;) | 08:01 |
| Jeffrey4l | overlayfs is ok. | 08:01 |
| sdake | there is a workaround | 08:01 |
| Jeffrey4l | no idea about other docker storage driver. | 08:01 |
| sdake | i published a patch | 08:01 |
| sdake | eveyrone shit on it | 08:01 |
| sdake | and did some retry: 2 thing | 08:01 |
| sdake | liek that is supposed to work | 08:01 |
| Jeffrey4l | the container can not be removed manually even. | 08:01 |
| Jeffrey4l | try this manually. | 08:02 |
| Jeffrey4l | docker rm -f | 08:02 |
| sdake | Jeffrey4l i am doing another upgrade | 08:02 |
| sdake | i'd bet 1 USD it will work | 08:02 |
| sdake | just as it did before | 08:02 |
| Jeffrey4l | hmm, ok. i'd like to bet ;) | 08:03 |
| sdake | overlayfs doesn't work because you can't build on it | 08:03 |
| Jeffrey4l | we have retries in nova_libvirt container for sure now. | 08:03 |
| sdake | yes but that doesn't fix the problem! | 08:04 |
| sdake | look at that, upgrade works | 08:04 |
| sdake | you owe me 1 USD Jeffrey4l | 08:04 |
| sdake | i'll take it in the form of a patch that fixes this problem ;-) | 08:04 |
| Jeffrey4l | i used to use overlayfs, but it is full of bug :( | 08:04 |
| *** openstackgerrit has quit IRC | 08:04 | |
| sdake | yes overlyafs is unacceptable | 08:04 |
| sdake | btrfs is only answer | 08:04 |
| *** openstackgerrit has joined #openstack-kolla | 08:04 | |
| sdake | my patch worked | 08:04 |
| sdake | retries:2 does not work | 08:05 |
| Jeffrey4l | upgrade is ok now with retries:1 ? | 08:05 |
| Jeffrey4l | which patch? | 08:05 |
| *** matrohon has quit IRC | 08:05 | |
| sdake | jeffrey here is what I did | 08:05 |
| sdake | 2.0.2->deploy | 08:05 |
| sdake | init-runonce | 08:05 |
| sdake | launch with 1 vm | 08:05 |
| sdake | 3.0.0->upgrade | 08:05 |
| sdake | fails | 08:05 |
| sdake | (with backtrace above) | 08:05 |
| sdake | 3.0.0->upgrade | 08:05 |
| *** msimonin1 has quit IRC | 08:05 | |
| sdake | works: | 08:05 |
| *** msimonin has joined #openstack-kolla | 08:05 | |
| sdake | PLAY RECAP ********************************************************************* | 08:05 |
| sdake | localhost : ok=243 changed=48 unreachable=0 failed=0 | 08:05 |
| sdake | ' | 08:05 |
| Jeffrey4l | wow!! | 08:06 |
| sdake | yes | 08:06 |
| Jeffrey4l | why it failed for the first upgrade? | 08:06 |
| Jeffrey4l | retries does not work? | 08:06 |
| sdake | you can see backtrace | 08:06 |
| sdake | it would appear not | 08:07 |
| sdake | i thought you guys tested this stuff... | 08:07 |
| sdake | anyway upgrade works on centos | 08:07 |
| sdake | minus this problem | 08:07 |
| *** msimonin has quit IRC | 08:08 | |
| sdake | we need the docker module option to ignore errors | 08:08 |
| sdake | i think that is what i had before | 08:08 |
| Jeffrey4l | from the backtrace, retries doesn't work at all. i will try this later. | 08:08 |
| sdake | it needs to be fixed now if possible | 08:08 |
| Jeffrey4l | i own you 1 USD sdake ;) | 08:09 |
| sdake | you can use my box ifyou like | 08:09 |
| Jeffrey4l | cool. | 08:09 |
| sdake | the upgrade procedure is a little difficult | 08:09 |
| sdake | i just keep the passsword.yml file in place | 08:09 |
| *** msimonin has joined #openstack-kolla | 08:09 | |
| *** awebber has joined #openstack-kolla | 08:10 | |
| Jeffrey4l | upgrade exist service should be OK without update the password.yml file. | 08:10 |
| *** matrohon has joined #openstack-kolla | 08:10 | |
| Jeffrey4l | the same for global.yml file. | 08:10 |
| Jeffrey4l | can i destroy all the container now? | 08:10 |
| *** confisurya has quit IRC | 08:10 | |
| sdake | Jeffrey4l yup | 08:11 |
| sdake | Jeffrey4l to do the work i install ansible1.9 via yum, modify globals.yml to pull from docker hub, deploy 2.0.2 | 08:11 |
| sdake | then upgrade to ansible via yum | 08:11 |
| sdake | modify globals.yml to deploy 3.0.0 from docker hub | 08:12 |
| *** kproskurin has joined #openstack-kolla | 08:12 | |
| sdake | rather from my registry | 08:12 |
| Jeffrey4l | got. | 08:12 |
| sdake | that retries:2 can go too | 08:12 |
| sdake | since that is not doing anything useful | 08:12 |
| *** huhaoran has quit IRC | 08:12 | |
| *** huhaoran has joined #openstack-kolla | 08:13 | |
| sdake | my guess is a fatal ansible module call is fatal - retries are not done | 08:13 |
| sdake | see line 766 | 08:13 |
| Jeffrey4l | retries is design to handle module fatal i think. | 08:14 |
| sdake | well it aint working ;-) | 08:14 |
| * Jeffrey4l is running a small test for retries locally. | 08:15 | |
| sdake | everything that worked in mitaka seems to work in newton | 08:15 |
| openstackgerrit | zhubingbing proposed openstack/kolla: add panko dockerfile https://review.openstack.org/387232 | 08:18 |
| Jeffrey4l | test locally works. trying re-produce in your node. | 08:19 |
| sdake | Jeffrey4l did you launch a vm? | 08:22 |
| Jeffrey4l | the small test is a simple ansible task. not kolla. i am deploy 2.0.2 on your node. | 08:22 |
| Jeffrey4l | - command: aa | 08:23 |
| Jeffrey4l | register: a | 08:23 |
| Jeffrey4l | until: a|success | 08:23 |
| Jeffrey4l | retries: 1 | 08:23 |
| Jeffrey4l | delay: 1 | 08:23 |
| *** confisurya has joined #openstack-kolla | 08:23 | |
| Jeffrey4l | sdake, here is the test ^^ | 08:23 |
| sdake | Jeffrey4l that module may behave differently then kolla-ansible | 08:24 |
| sdake | https://bugs.launchpad.net/kolla/+bug/1634401 | 08:24 |
| openstack | Launchpad bug 1634401 in kolla ocata "upgrade from 2.0.2 to 3.0.0 with VM running on BTRFS STILL BROKEN" [Critical,Triaged] - Assigned to Jeffrey Zhang (jeffrey4l) | 08:24 |
| Jeffrey4l | may be. | 08:25 |
| sdake | the most frustrating part of this is I asked people if they tested it, and they claimed yes | 08:25 |
| sdake | yet clearly, it wasn't tested | 08:25 |
| sdake | and now, t-12 hrs to go to release and some rework required on kolla_docker module | 08:26 |
| Jeffrey4l | ok. will fix this. i need upgrade by using -vvv parameter to get more info. | 08:26 |
| sdake | sure - feel free to abuse machines as you see fit :) | 08:27 |
| sdake | I'd stick to AIO for now | 08:27 |
| sdake | i wonder when docker is giong to run out of money | 08:27 |
| Jeffrey4l | btw, tempest smoke test seems fine. | 08:27 |
| *** HyperJohnGraham_ has quit IRC | 08:28 | |
| sdake | yup kolla works well | 08:28 |
| sdake | minus the upgrade part | 08:28 |
| sdake | and possibly reconfigure | 08:28 |
| sdake | and the 21 new services we added ;-) | 08:28 |
| sdake | we need to give fernet a roll too | 08:28 |
| sdake | Jeffrey4l do you ahve tempest results from rc3? | 08:29 |
| sdake | Jeffrey4l if so, can you mail them to me plz | 08:29 |
| Jeffrey4l | ok. will post it to you. | 08:29 |
| sdake | I dont care to have all results, just end results | 08:29 |
| sdake | (# tests run/passed/etc) | 08:29 |
| Jeffrey4l | i only run smoke test. | 08:30 |
| Jeffrey4l | it takes long time. | 08:30 |
| sdake | ok dont sweat it then | 08:30 |
| Jeffrey4l | for full test result. | 08:32 |
| sdake | Jeffrey4l note i changed reproducer: https://bugs.launchpad.net/kolla/+bug/1634401 | 08:32 |
| openstack | Launchpad bug 1634401 in kolla ocata "upgrade from 2.0.2 to 3.0.0 with VM running on BTRFS STILL BROKEN" [Critical,Triaged] - Assigned to Jeffrey Zhang (jeffrey4l) | 08:32 |
| Jeffrey4l | roger./ | 08:32 |
| *** huhaoran has quit IRC | 08:33 | |
| *** huhaoran has joined #openstack-kolla | 08:33 | |
| sdake | bjolo__ ya - if yo ucould test the upgrade procedure on ubuntu it would be most appreciated :) | 08:34 |
| sdake | Jeffrey4l note i changed he reproucer slightly | 08:35 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 08:35 | |
| *** ChanServ sets mode: -o sdake | 08:36 | |
| Jeffrey4l | copy | 08:36 |
| sdake | Jeffrey4l note the 3.0.0 images in my registry are stable/newton images pulled about 30 or 40 mins ago | 08:37 |
| Jeffrey4l | ok. | 08:37 |
| sdake | docker is deleting the image - it just says it is unable to do so | 08:37 |
| sdake | or deleting the container rather | 08:38 |
| sdake | Jeffrey4l here was my fix in progress: https://review.openstack.org/#/c/334644/ | 08:43 |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Support Searchlight Docker container https://review.openstack.org/377145 | 08:43 |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Support searchlight ansible role https://review.openstack.org/387862 | 08:43 |
| *** athomas has joined #openstack-kolla | 08:48 | |
| sdake | note Jeffrey4l I *tested* this solution and it worked | 08:49 |
| Jeffrey4l | yep. issue reproduced. try to figure out why it doesn't work. | 08:50 |
| yingjun | sdake, Jeffrey4l, could you take a look at these patches: https://review.openstack.org/#/c/377145/, https://review.openstack.org/#/c/387862/, if you have time:) its’ the searchlight support related patches, thank you | 08:50 |
| sdake | yingjun see topic | 08:51 |
| sdake | yingjun totally wrong time to ask | 08:51 |
| Jeffrey4l | yingjun, will review it later :0 | 08:51 |
| yingjun | sdake: oops | 08:52 |
| sdake | Jeffrey4l my speculation is ansible doesn't know how to handle the exception from docker | 08:54 |
| sdake | Jeffrey4l so it thinks something went critically wrong | 08:54 |
| *** confisurya has quit IRC | 08:55 | |
| Jeffrey4l | sdake, no. kolla_docker catch all exception and just add the traceback into msg field. so you see the traceback. | 08:56 |
| sdake | i see | 08:56 |
| sdake | Jeffrey4l where is this code? | 08:57 |
| *** Liuqing has joined #openstack-kolla | 08:57 | |
| Jeffrey4l | sdake, https://github.com/openstack/kolla/blob/master/ansible/library/kolla_docker.py#L745 | 08:57 |
| Jeffrey4l | i suspend the change=true when failed. trying to make some test. | 08:57 |
| Jeffrey4l | suspend/doubt | 08:58 |
| *** eaguilar has joined #openstack-kolla | 08:58 | |
| sdake | Jeffrey4l https://github.com/openstack/kolla/blob/master/ansible/library/kolla_docker.py#L723 | 08:58 |
| sdake | the issue happens in start_container... | 08:58 |
| sdake | exception is not caught i think | 08:59 |
| sdake | nah thats not right | 08:59 |
| sdake | i dunno - the linkage to ansible seems not working to me | 09:00 |
| Jeffrey4l | sdake, trackback http://paste.openstack.org/show/586138/ | 09:00 |
| *** eaguilar has quit IRC | 09:02 | |
| sdake | Jeffrey4l start_contianer doesn't return true or false | 09:03 |
| sdake | so how can it know if the operation was successuful or not? | 09:03 |
| *** tonanhngo has joined #openstack-kolla | 09:03 | |
| *** openstackgerrit has quit IRC | 09:04 | |
| *** openstackgerrit has joined #openstack-kolla | 09:04 | |
| Jeffrey4l | if something wrong, docker-py ( self.dc.start) will raise some exception. | 09:04 |
| sdake | ok, and it raises exception at line 745 in kolla_docker | 09:05 |
| *** tonanhngo has quit IRC | 09:05 | |
| sdake | in that line is result | 09:05 |
| sdake | in line 743 is result rather | 09:05 |
| sdake | in line 745 is no result | 09:05 |
| sdake | just failed=True | 09:05 |
| Jeffrey4l | line #745 just test ansible, this script is failed. | 09:05 |
| Jeffrey4l | test/tell | 09:06 |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Support Searchlight Docker container https://review.openstack.org/377145 | 09:06 |
| *** rmart04 has joined #openstack-kolla | 09:06 | |
| sdake | Jeffrey4l | 09:07 |
| sdake | i'm not sure what the difference is between exit_json | 09:07 |
| sdake | and fail_json | 09:07 |
| sdake | but I think we want fail_json | 09:08 |
| sdake | module.fail_json(msg="unable to find %s. Exception message: %s" % (config_file, e)) | 09:08 |
| sdake | from mysql-db module | 09:08 |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Support searchlight ansible role https://review.openstack.org/387862 | 09:08 |
| Jeffrey4l | it is a shortcut function | 09:08 |
| *** Liuqing has quit IRC | 09:08 | |
| *** confisurya has joined #openstack-kolla | 09:09 | |
| Jeffrey4l | sdake, locally test works http://paste.openstack.org/show/586142/ | 09:09 |
| sdake | Jeffrey4l i don't see an exception being raised there | 09:10 |
| Jeffrey4l | we have try catch in kolla_docker. the msg just a long string in kolla_docker. | 09:10 |
| sdake | i understand | 09:11 |
| sdake | the backtrace you showed me earlier | 09:11 |
| sdake | http://paste.openstack.org/show/586138/ | 09:12 |
| sdake | look at line 3 | 09:12 |
| sdake | that is not 745 | 09:12 |
| sdake | that is 742 or so | 09:12 |
| sdake | the exception isn't being caught | 09:12 |
| Jeffrey4l | hmm. it works even i raise a exception without any catch. in main method locally | 09:15 |
| *** shardy has joined #openstack-kolla | 09:16 | |
| sdake | what works | 09:16 |
| sdake | retries:2? | 09:16 |
| Jeffrey4l | sdake, http://paste.openstack.org/show/586144/ | 09:17 |
| *** msimonin has quit IRC | 09:17 | |
| sdake | Jeffrey4l there is only one divide by zero error | 09:18 |
| *** duonghq has quit IRC | 09:18 | |
| sdake | with retries:2 shouldn't there be two divide by zero errors? | 09:18 |
| Jeffrey4l | retries:2 mean: try 3 times. and ansible only show the last error. | 09:19 |
| sdake | i think a counter in the output could validate that theory :) | 09:20 |
| sdake | i'd bet 1 USD that counter = 1 :) | 09:21 |
| sdake | can write counter to file and increment it each run | 09:22 |
| Jeffrey4l | which counter? retries? | 09:22 |
| sdake | oh nm | 09:23 |
| sdake | i see where it is retrying | 09:23 |
| sdake | why is it not retyring in the upgrade case? | 09:23 |
| *** tonanhngo has joined #openstack-kolla | 09:24 | |
| Jeffrey4l | i do not own you anything ;) | 09:24 |
| Jeffrey4l | ya. it is weird. | 09:24 |
| *** tonanhngo has quit IRC | 09:25 | |
| *** kproskurin has quit IRC | 09:25 | |
| sdake | possibly nested try/catch? | 09:26 |
| *** msimonin has joined #openstack-kolla | 09:26 | |
| sdake | your 1-10 has no try/except, yet ansible is running the module.exit_json | 09:26 |
| sdake | i think 745 is not working as expected | 09:27 |
| Jeffrey4l | hmm. | 09:30 |
| sdake | Jeffrey4l looking at other modules, there is no try/except logic in any of them | 09:30 |
| *** yingjun has quit IRC | 09:31 | |
| *** yingjun has joined #openstack-kolla | 09:31 | |
| Jeffrey4l | sdake, it is the same right now. http://paste.openstack.org/show/586147/ | 09:33 |
| sdake | between line 8 and 9 put a different exit_json | 09:34 |
| bjolo__ | sdake, centos source&binary build still fails on master. does anyone else have this issue? | 09:34 |
| sdake | perhaps with failed=False | 09:34 |
| sdake | bjolo__ no - centos source builds for me | 09:34 |
| bjolo__ | master branch? | 09:34 |
| sdake | bjolo stable/newton | 09:34 |
| *** gfidente has joined #openstack-kolla | 09:34 | |
| *** gfidente has joined #openstack-kolla | 09:34 | |
| sdake | git checkout stable/newton | 09:34 |
| bjolo__ | stable/newton works for me as well | 09:34 |
| bjolo__ | but not master | 09:35 |
| sdake | oh - you mean master is busted | 09:35 |
| sdake | ok - well don't immediately care about master ;) | 09:35 |
| sdake | will care more after 3.0.0 is tagged | 09:35 |
| sdake | 3.0.0 is tagged from stable/newton | 09:35 |
| bjolo__ | not me either but since it came up in my daily builds i had to dig into it | 09:35 |
| sdake | bjolo cool thanks for that | 09:36 |
| *** eaguilar has joined #openstack-kolla | 09:36 | |
| *** yingjun has quit IRC | 09:36 | |
| *** janem has joined #openstack-kolla | 09:36 | |
| *** haplo37 has quit IRC | 09:37 | |
| sdake | bjolo i'll build master while jeffrey is sorting out ansible madness | 09:37 |
| sdake | and take a look here | 09:37 |
| *** g3ek has quit IRC | 09:37 | |
| bjolo__ | ok deploying mitaka ubuntu binary and doing the upgrade to newton. multinode 3 controllers, 3 network, 16 compute | 09:38 |
| bjolo__ | host os centos 7 | 09:38 |
| sdake | that will mostly work i think | 09:38 |
| Jeffrey4l | bjolo__, what docker storage driver are u using? | 09:38 |
| sdake | make sure to follow correct procedure | 09:38 |
| *** g3ek has joined #openstack-kolla | 09:39 | |
| *** haplo37 has joined #openstack-kolla | 09:39 | |
| sdake | bjolo which image fails for oyu | 09:39 |
| *** eaguilar has quit IRC | 09:40 | |
| *** prithiv has joined #openstack-kolla | 09:40 | |
| *** janem has quit IRC | 09:41 | |
| sdake | bjolo on master try tools/build.py --type source | 09:43 |
| sdake | bjolo master deploys master of openstack (and builds master images) - stable/newton deploys newton version of openstack (and builds newton images) | 09:46 |
| *** wznoinsk_ has joined #openstack-kolla | 09:46 | |
| *** HyperJohnGraham_ has quit IRC | 09:46 | |
| *** wznoinsk has quit IRC | 09:47 | |
| *** wznoinsk_ is now known as wznoinsk | 09:47 | |
| *** tonanhngo has joined #openstack-kolla | 09:49 | |
| *** tonanhngo has quit IRC | 09:50 | |
| *** wznoinsk has quit IRC | 09:50 | |
| *** wznoinsk_ has joined #openstack-kolla | 09:50 | |
| sdake | bjolo__ ^^ | 09:50 |
| *** wznoinsk_ is now known as wznoinsk | 09:51 | |
| Jeffrey4l | sdake, found the root cause. | 09:53 |
| sdake | cool | 09:53 |
| sdake | hit me with it | 09:53 |
| Jeffrey4l | sdake, guess? | 09:53 |
| sdake | nested try/except? | 09:53 |
| Jeffrey4l | ansible bug on 2.1.0.0 :( | 09:53 |
| sdake | evidence? | 09:54 |
| Jeffrey4l | in ansible 2.1.0.0, reties: 1 means run once totally. in 2.1.2.0 , reties:1 means run twice totally. | 09:54 |
| sdake | this bug has been around since 2.0.0 | 09:54 |
| Jeffrey4l | my locally test result. | 09:54 |
| Jeffrey4l | no idea whether it is a regression bug | 09:54 |
| sdake | what is rretries set to now | 09:55 |
| Jeffrey4l | 1 | 09:55 |
| Jeffrey4l | set it too 2 should be OK. | 09:55 |
| sdake | can you set it to 2 and try an upgrade | 09:55 |
| Jeffrey4l | ya. np. | 09:55 |
| sdake | The above example run the shell module recursively till the module’s result has “all systems go” in its stdout or the task has been retried for 5 times with a delay of 10 seconds. The default value for “retries” is 3 and “delay” is 5. | 09:56 |
| sdake | http://docs.ansible.com/ansible/playbooks_loops.html | 09:56 |
| sdake | the correct interpreation of retries: 2 should be 2 attempts total | 09:57 |
| sdake | note the default values | 09:57 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add logrotate configuration for elasticsearch https://review.openstack.org/387630 | 09:57 |
| sdake | 2.1.2.0 looks like regression to me around retries | 09:57 |
| Jeffrey4l | sdake, i do not think kolla have retries: 3 in default. otherwise we won't need fix such issue. | 09:58 |
| sdake | Jeffrey4l right | 09:58 |
| sdake | you could be right on that part - don't knwo :) | 09:58 |
| Jeffrey4l | lol. when adding such this issue, i am sure, retries: 1 works. :9 | 09:58 |
| Jeffrey4l | ;( | 09:58 |
| sdake | 2 or 10 retries, doesn't matter, if 1 doesn't work lets bump it up | 09:58 |
| Jeffrey4l | roger. | 09:59 |
| sdake | retries:1 doesnt' even make any sense | 09:59 |
| sdake | acccording to the documentation | 10:00 |
| Jeffrey4l | anyway, if :2 works, we will bump the value. | 10:00 |
| sdake | ya its sloppy but whatever - lets get er done :) | 10:01 |
| *** pbourke_ is now known as pbourke | 10:05 | |
| *** berendt has joined #openstack-kolla | 10:09 | |
| berendt | our json file validation script is totally broken | 10:09 |
| *** zhubingbing has quit IRC | 10:09 | |
| *** zhubingbing has joined #openstack-kolla | 10:10 | |
| berendt | https://bugs.launchpad.net/kolla/+bug/1634447 | 10:10 |
| openstack | Launchpad bug 1634447 in kolla "JSON validation script is broken" [Undecided,New] | 10:10 |
| zhubingbing | ping e90 | 10:11 |
| zhubingbing | ~ | 10:11 |
| Jeffrey4l | berendt, i got a idea to extend the exist scrip to validate the jons.j2 file. ( need render as jinja2 template first ) | 10:12 |
| berendt | Jeffrey4l should i assign the bug to you? | 10:12 |
| Jeffrey4l | so would u mind assign that bug me. yes. | 10:12 |
| Jeffrey4l | thanks :) | 10:13 |
| berendt | Jeffrey4l i set priority to high, i think it is a no go to merge syntax issues | 10:13 |
| Jeffrey4l | yes. but it will not be merged into newton. | 10:14 |
| berendt | yes, but we should avoid it in the future | 10:14 |
| Jeffrey4l | agree. | 10:14 |
| berendt | i want to change the bandit job from non-voting to voting | 10:17 |
| Jeffrey4l | berendt, yep. change it. | 10:19 |
| berendt | https://review.openstack.org/387909 | 10:19 |
| Jeffrey4l | berendt, remove this line too - name: ^gate-kolla-tox-bandit-ubuntu-xenial$ | 10:20 |
| Jeffrey4l | berendt, ^^' | 10:21 |
| berendt | yes of course | 10:21 |
| berendt | done | 10:21 |
| Jeffrey4l | look good +1 | 10:22 |
| berendt | should we also enable gate-kolla-python35-nv ? | 10:22 |
| Jeffrey4l | berendt, i do not think so. ansible do not support python3 now. | 10:23 |
| berendt | gate-kolla-python34 is already enabled | 10:24 |
| berendt | this check is only relevant for our own tools | 10:24 |
| Jeffrey4l | hmm. OK. we may need add unittest for ansible module, then it may cause some issue. | 10:25 |
| berendt | and we do not have a releasenote check | 10:26 |
| Jeffrey4l | re python35-nv this non-voting comes from infra ci define, i.e. all python35-nv is non-voting. | 10:26 |
| berendt | ah ok | 10:26 |
| *** daneyon has quit IRC | 10:26 | |
| Jeffrey4l | i do not have strong opinion( right word?) on this. | 10:27 |
| *** daneyon has joined #openstack-kolla | 10:27 | |
| berendt | we should check everything that we deliver | 10:27 |
| berendt | we have to use releasenotes, we should check them | 10:27 |
| Jeffrey4l | sdake, good new. retries: 2 works. | 10:28 |
| Jeffrey4l | so i wanna bump it to 5 ;) | 10:28 |
| Jeffrey4l | we have to use releasenotes, we should check them <- what this mean? | 10:28 |
| Jeffrey4l | and yes. we should check everything we deliver. | 10:29 |
| Jeffrey4l | berendt, ^^ | 10:30 |
| *** zhangyufei has quit IRC | 10:30 | |
| *** berendt has quit IRC | 10:31 | |
| *** berendt has joined #openstack-kolla | 10:35 | |
| berendt | Jeffrey4l looks like we have release-note-jobs, at least they are listed in zuul layouts.yml | 10:35 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Bump retries in starting nova-libvirt to 5 https://review.openstack.org/387918 | 10:36 |
| *** rhallisey has joined #openstack-kolla | 10:37 | |
| *** prithiv has quit IRC | 10:40 | |
| Jeffrey4l | yep. we need enable releasenote release jobs berendt | 10:44 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Bump retries in starting nova-libvirt to 5 https://review.openstack.org/387918 | 10:45 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Remove /var/run/docker.sock volume from telegraf container https://review.openstack.org/387924 | 10:46 |
| *** phuongnh has quit IRC | 10:47 | |
| berendt | Jeffrey4l jobs are already there but only enabled when a release note was modified | 10:48 |
| Jeffrey4l | it need be added into check and gate field in zuul | 10:48 |
| openstackgerrit | prameswar proposed openstack/kolla: dependencies added to build code with pip in QSG https://review.openstack.org/387926 | 10:49 |
| Jeffrey4l | hmm. wait. | 10:49 |
| berendt | Jeffrey4l it is already working, i thought it is not working | 10:49 |
| berendt | Jeffrey4l check https://review.openstack.org/#/c/369184/6 for a sample | 10:49 |
| Jeffrey4l | yes. it works. i thought it didn't work, too. :) | 10:50 |
| Jeffrey4l | interesting. only enabled when releasenote changed. | 10:50 |
| *** prithiv has joined #openstack-kolla | 10:58 | |
| *** portdirect has quit IRC | 10:59 | |
| *** portdirect has joined #openstack-kolla | 11:00 | |
| *** awiddersheim has quit IRC | 11:00 | |
| *** aswadr_ has joined #openstack-kolla | 11:03 | |
| *** zhubingbing has quit IRC | 11:05 | |
| *** coolsvap has quit IRC | 11:06 | |
| openstackgerrit | Merged openstack/kolla: dependencies added to build code with pip in QSG https://review.openstack.org/387926 | 11:07 |
| *** haplo37_ has quit IRC | 11:10 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Remove inputs.filestat from telegraf configuration https://review.openstack.org/387935 | 11:10 |
| *** prithiv has quit IRC | 11:11 | |
| *** haplo37_ has joined #openstack-kolla | 11:12 | |
| *** berendt has quit IRC | 11:15 | |
| *** mliima has joined #openstack-kolla | 11:16 | |
| *** shardy is now known as shardy_lunch | 11:19 | |
| *** ppalacios has joined #openstack-kolla | 11:20 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Allow operators to use 'fallback mode' for Ceph disks https://review.openstack.org/384496 | 11:34 |
| *** ayoung has quit IRC | 11:34 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Allow operators to use 'fallback mode' for Ceph disks https://review.openstack.org/384496 | 11:37 |
| *** tonanhngo has joined #openstack-kolla | 11:45 | |
| *** tonanhngo has quit IRC | 11:46 | |
| *** neilus has joined #openstack-kolla | 11:48 | |
| *** janem has joined #openstack-kolla | 11:50 | |
| *** neilus has quit IRC | 11:52 | |
| *** srwilkers has joined #openstack-kolla | 12:03 | |
| *** zhubingbing_ has joined #openstack-kolla | 12:05 | |
| *** sdake has quit IRC | 12:06 | |
| *** dave-mccowan has joined #openstack-kolla | 12:10 | |
| *** portdirect has quit IRC | 12:20 | |
| *** portdirect has joined #openstack-kolla | 12:20 | |
| *** shardy_lunch is now known as shardy | 12:21 | |
| *** caowei has joined #openstack-kolla | 12:22 | |
| *** prithiv has joined #openstack-kolla | 12:28 | |
| *** schwicht has quit IRC | 12:33 | |
| openstackgerrit | prameswar proposed openstack/kolla: apt instead of apt-get to install packages in Ubuntu https://review.openstack.org/379477 | 12:34 |
| *** prithiv has quit IRC | 12:36 | |
| *** steve-noyes has joined #openstack-kolla | 12:41 | |
| *** prithiv has joined #openstack-kolla | 12:43 | |
| *** tonanhngo has joined #openstack-kolla | 12:44 | |
| *** tonanhngo has quit IRC | 12:44 | |
| *** rmart04 has quit IRC | 12:46 | |
| *** portdirect has quit IRC | 12:47 | |
| *** rmart04 has joined #openstack-kolla | 12:47 | |
| *** portdirect has joined #openstack-kolla | 12:48 | |
| *** hieulq_ has quit IRC | 12:52 | |
| *** hieulq_ has joined #openstack-kolla | 12:58 | |
| *** strigazi1AFK is now known as strigazi | 12:58 | |
| *** srwilkers has quit IRC | 12:58 | |
| *** srwilkers has joined #openstack-kolla | 12:59 | |
| *** ayoung has joined #openstack-kolla | 13:01 | |
| *** prithiv has quit IRC | 13:01 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Bump retries in starting nova-libvirt to 5 https://review.openstack.org/387918 | 13:05 |
| *** eaguilar has joined #openstack-kolla | 13:05 | |
| *** schwicht has joined #openstack-kolla | 13:09 | |
| *** Marx314 has joined #openstack-kolla | 13:11 | |
| *** huhaoran has quit IRC | 13:14 | |
| *** awiddersheim has joined #openstack-kolla | 13:17 | |
| *** rmart04_ has joined #openstack-kolla | 13:18 | |
| *** rmart04 has quit IRC | 13:19 | |
| *** rmart04_ is now known as rmart04 | 13:19 | |
| *** jrist has quit IRC | 13:22 | |
| openstackgerrit | Merged openstack/kolla: Adds Fluentd docker image required for kolla-kube https://review.openstack.org/375615 | 13:22 |
| *** jrist has joined #openstack-kolla | 13:22 | |
| *** jrist has quit IRC | 13:24 | |
| *** sean-k-mooney has quit IRC | 13:27 | |
| *** prithiv has joined #openstack-kolla | 13:28 | |
| *** jrist has joined #openstack-kolla | 13:30 | |
| *** confisurya has left #openstack-kolla | 13:31 | |
| *** rmart04 has quit IRC | 13:31 | |
| *** eaguilar_ has joined #openstack-kolla | 13:33 | |
| *** eaguilar has quit IRC | 13:34 | |
| *** LamT__ has joined #openstack-kolla | 13:34 | |
| *** sp_ has joined #openstack-kolla | 13:35 | |
| *** sdake has joined #openstack-kolla | 13:36 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add missing closing brace in grafana configuration file https://review.openstack.org/388023 | 13:36 |
| sdake | morning | 13:37 |
| sdake | hey Jeffrey4l | 13:37 |
| sdake | sorry - fell asleep at keyboard yet again | 13:37 |
| Jeffrey4l | sdake, morning. | 13:37 |
| sdake | where we at wrt relese | 13:37 |
| Jeffrey4l | you need some rest ;) | 13:37 |
| sdake | i can rest when im dead | 13:37 |
| sdake | where we at with the release | 13:37 |
| Jeffrey4l | :/ code joke | 13:37 |
| sdake | ya - i made a joke too :) | 13:38 |
| Jeffrey4l | code/cold | 13:38 |
| *** schwicht has quit IRC | 13:38 | |
| Jeffrey4l | https://review.openstack.org/387918 retries: 2 works i bump to 5 | 13:38 |
| sp_ | sdake: Jeffrey4l: | 13:39 |
| sp_ | this is sp_ | 13:39 |
| sdake | anything else found jeff? | 13:39 |
| sdake | bugwise | 13:39 |
| sdake | Jeffrey4l ^^ | 13:39 |
| Jeffrey4l | sp_, hi | 13:39 |
| sp_ | I am just new faulk to kolla | 13:39 |
| sdake | hey sp_ | 13:39 |
| sp_ | want to get some interaction you guys | 13:40 |
| Jeffrey4l | sdake, another bug fix for grafana https://review.openstack.org/388023 | 13:40 |
| *** zhurong has joined #openstack-kolla | 13:40 | |
| sdake | that tells me exctly zero people tested grafana | 13:40 |
| Jeffrey4l | no idea why mitaka branch broken in CI. it should be OK. https://review.openstack.org/299182 | 13:41 |
| Jeffrey4l | except me ;) | 13:41 |
| Jeffrey4l | i tested grafana + gnocchi, which is cool ;) | 13:41 |
| sdake | ok wee can deal wit htat fire later | 13:41 |
| sdake | grafana works then? | 13:41 |
| sdake | what about fernet? | 13:42 |
| sdake | what tests did you run Jeffrey4l | 13:42 |
| sdake | want ot get a feel for where the release is at | 13:42 |
| sdake | s in can we tag it now | 13:42 |
| Jeffrey4l | hmm. not tested fernet ;( | 13:43 |
| sdake | ca nyou tell me what you have tested | 13:43 |
| msimonin | Hello, I don't want to bother you with :https://bugs.launchpad.net/kolla/+bug/1617334 :) | 13:43 |
| openstack | Launchpad bug 1617334 in kolla "reconfigure action fails on [neutron | Restart the neutron_openvswitch_agent container] " [Critical,Incomplete] | 13:43 |
| sdake | msimonin was that you that reported that issue? | 13:43 |
| msimonin | yes | 13:44 |
| sdake | msimonin ok - we will get into it in a min if you are available | 13:44 |
| openstackgerrit | Merged openstack/kolla: Add missing closing brace in grafana configuration file https://review.openstack.org/388023 | 13:44 |
| sdake | 6am here - still waking up | 13:44 |
| msimonin | wow I see, take your time | 13:44 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add missing closing brace in grafana configuration file https://review.openstack.org/388026 | 13:44 |
| Jeffrey4l | newton deployment on multinode. with ceilometer, gnocchi, ceph, cinder | 13:44 |
| sdake | pbourke this needs an ack: https://review.openstack.org/#/c/387721/ | 13:45 |
| sdake | or mliima | 13:45 |
| sdake | or any other cores around | 13:45 |
| pbourke | done | 13:45 |
| Jeffrey4l | and grafana works with that patch. | 13:45 |
| sdake | oh thats mitaka | 13:45 |
| sdake | nm | 13:45 |
| sdake | Jeffrey4l nice! | 13:46 |
| sdake | wtf is with the gate | 13:46 |
| Jeffrey4l | sdake, that's what i said just now. | 13:46 |
| sdake | whisky tango foxtrot is letters of the day | 13:46 |
| sdake | Jeffrey4l oh i thought you were talking about newton not master | 13:46 |
| sp_ | sdake: want be like you, i also want same rhythm like to sleep on keyboard :) :) I have started working on this BP https://blueprints.launchpad.net/kolla/+spec/kolla-gui | 13:47 |
| Jeffrey4l | it should be some ci issue. | 13:47 |
| sdake | Jeffrey4l ok - lets put the ci on hold for the moment | 13:47 |
| openstackgerrit | Merged openstack/kolla: Fix stale namespace removal issue https://review.openstack.org/387721 | 13:48 |
| sdake | pbourke woudl yo ube kind enough to ack https://review.openstack.org/#/c/388026/1 | 13:48 |
| *** jtriley has joined #openstack-kolla | 13:48 | |
| sdake | sp_ nice - gui is killer feature - do you know how to do gui work? | 13:49 |
| sp_ | sdake : It would be finalise after Summit | 13:50 |
| sdake | sp_ i dont know what the hipsters use today for html5 stuff | 13:50 |
| sdake | angular | 13:50 |
| sdake | and there are like 10 other things | 13:50 |
| sp_ | sdake , and Yes we need separate GUI | 13:51 |
| openstackgerrit | Merged openstack/kolla: Add missing closing brace in grafana configuration file https://review.openstack.org/388026 | 13:51 |
| sp_ | sdake, It would be helpful, if you will share these 10 things | 13:51 |
| mliima | i like html5 :) | 13:53 |
| jmccarthy | https://hackernoon.com/how-it-feels-to-learn-javascript-in-2016-d3a717dd577f#.ihkg1th6t | 13:53 |
| *** schwicht has joined #openstack-kolla | 13:54 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO_NOT_MERGE: Use better URL for China https://review.openstack.org/329060 | 13:54 |
| sp_ | sdake: please provide your opinion | 13:57 |
| *** dwalsh has joined #openstack-kolla | 13:57 | |
| sdake | jmccarthy lol looks pretty hip | 13:59 |
| sdake | sp_ make it work | 13:59 |
| sdake | sp_ we will provide some sort of rest api that matches our kolla-ansible api | 14:00 |
| sdake | beyond that, I looked into web progrmaming in "2016" and -ETOOCOMPLICATED :) | 14:00 |
| *** rmart04 has joined #openstack-kolla | 14:01 | |
| sdake | i need a beer after drinking that article | 14:01 |
| sdake | and its only 7am! | 14:01 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Adding kubetoolbox image https://review.openstack.org/384988 | 14:02 |
| sdake | msimonin will be with you in a moment - brain still warming up | 14:03 |
| sdake | core reviewers - ANYONE have any problems that need to be pulled into the rc3 release before tagging | 14:03 |
| msimonin | sdake: ack | 14:03 |
| sdake | sp_ keep in mind i wrote web pages in html in 1992.. | 14:04 |
| jmccarthy | I like the part about 'web assembly' being a thing lol | 14:04 |
| sdake | using vi | 14:04 |
| -openstackstatus- NOTICE: We are away of pycparser failures in the gate and working to address the issue. | 14:05 | |
| sdake | ok cats, mirror being worked on | 14:06 |
| sdake | rather the gates | 14:06 |
| sdake | msimonin ok - lets look at your issue | 14:07 |
| sdake | msimonin first off, you are deploying 3.0.0rc2 ? | 14:07 |
| msimonin | sdake: At the time a raised the issue it was 2.022 | 14:07 |
| msimonin | 2.0.2* | 14:07 |
| openstackgerrit | Merged openstack/kolla: Bump retries in starting nova-libvirt to 5 https://review.openstack.org/387918 | 14:07 |
| msimonin | until now we are still using stable/mitaka branch | 14:08 |
| sdake | msimonin ok - does this problem exist in 3.0.0 at all? | 14:08 |
| sdake | msimonin fwiw the entire core reviewer team uses ovs and nobody runs into this problem | 14:08 |
| msimonin | sdake: to be honnest I haven't try | 14:08 |
| msimonin | tried* | 14:08 |
| sdake | ok so 2.0.2 is broken | 14:08 |
| sdake | how did you install it | 14:08 |
| sdake | from git, or from pip? | 14:09 |
| msimonin | from git | 14:09 |
| *** huhaoran has joined #openstack-kolla | 14:09 | |
| sdake | ok | 14:09 |
| sdake | about 6 weeks ago or so we changed the documentation | 14:09 |
| sdake | so explicitly tell people not to do that | 14:09 |
| msimonin | :) | 14:09 |
| sdake | i even sent a note to the mailing list | 14:09 |
| sdake | msimonin is this a production deploy you are working with or an eval? | 14:10 |
| msimonin | It's not production | 14:10 |
| *** huhaoran has quit IRC | 14:10 | |
| sdake | good | 14:10 |
| msimonin | We are evaluating different topologies / configurations | 14:11 |
| sdake | so we can alter it then? | 14:11 |
| msimonin | sure | 14:11 |
| sdake | when does the issue hapepn | 14:11 |
| sdake | Jeffrey4l if you dont have to turn into a pumpkin a quick check of fernet would be appreciated | 14:11 |
| Jeffrey4l | ok. test env broken. trying fix. | 14:12 |
| *** haplo37_ has quit IRC | 14:12 | |
| msimonin | just deploy "kolla-ansible deploy" and then launch "kolla-ansible reconfigure" | 14:12 |
| msimonin | the thing is I'm using fake drivers | 14:13 |
| msimonin | It's worth mentionning it | 14:13 |
| msimonin | except that there's nothing fancy about my deployment | 14:13 |
| *** tonanhngo has joined #openstack-kolla | 14:13 | |
| sdake | oh fake drivers | 14:14 |
| sdake | so fake drivers are for r&d work | 14:15 |
| sdake | not for actual deployment | 14:15 |
| *** haplo37_ has joined #openstack-kolla | 14:15 | |
| * sdake requests some anestaphine from the medlock | 14:15 | |
| sdake | this problem happens on reconfigure? | 14:15 |
| awebber | is this the right channel for the kolla-kubernetes project? | 14:16 |
| sdake | awebber roger | 14:16 |
| msimonin | yes we are using fake driver to evaluate Openstack control plane | 14:16 |
| awebber | sdake awesome | 14:16 |
| *** neilus has joined #openstack-kolla | 14:16 | |
| sbezverk | awebber: yes, what is up? | 14:16 |
| msimonin | sdake: yes, the problem appears on reconfigure | 14:16 |
| *** sdake has quit IRC | 14:16 | |
| awebber | sbezverk i am trying to deploy a successful kolla project to kubernetes | 14:16 |
| *** sdake_ has joined #openstack-kolla | 14:17 | |
| awebber | sbezverk i am following the documentation and building the config files all from within a container | 14:17 |
| sdake_ | msimonin so the problem occurs during reconfgiure not during deploy? | 14:17 |
| awebber | sbezverk it seems like when generating the config files some tasks try to call sysctl to probably setup forwarding | 14:17 |
| awebber | sbezverk this fails on coreos - "Read-only file system\nsysctl" | 14:18 |
| sdake_ | awebber groan | 14:18 |
| sbezverk | awebber: Have you set orchestration engine to KUBERNETES, before you started generating config files? | 14:19 |
| msimonin | sdake_: launch is perfectly fine, reconfigure is raising this error | 14:19 |
| awebber | sbezverk i have commented out those task and am continuing with "successfully generated configs" | 14:19 |
| sdake_ | msimonin what are you reconfgiuring exatly? | 14:19 |
| awebber | sbezverk yes | 14:19 |
| awebber | sbezverk its probably just a coreos issue, wouldnt show up on non read only files systems i guess | 14:20 |
| awebber | sbezverk im hoping i can continue :) | 14:20 |
| rhallisey | awebber, I think you'll need the kolla-tools containers | 14:20 |
| rhallisey | I think kfox builds configs in there | 14:20 |
| msimonin | sdake_: actually to reproduce this behaviour I don't need to reconfigure anything | 14:20 |
| *** neilus has quit IRC | 14:20 | |
| rhallisey | or you can switch to centos/ubuntu | 14:20 |
| sdake_ | msimonin does it happen without the fake driver? | 14:21 |
| sbezverk | awebber: we have not tested kolla-kube in coreos | 14:21 |
| msimonin | I don't think so | 14:21 |
| awebber | rhallisey i use the following container - http://pastebin.com/raw/B1C8z3YE | 14:21 |
| msimonin | sdake_: I don't think so | 14:21 |
| sdake_ | ok - hate to say we dont care | 14:21 |
| msimonin | sdake_: but I can try | 14:21 |
| awebber | sbezverk no problem, i am happy to be your tester :) | 14:21 |
| sdake_ | msimonin but in the big scheme of the 300 bugs or so open in kolla this one isn't real high priority | 14:21 |
| sdake_ | msimonin fake driver is cool and all but its meant for r&d | 14:22 |
| msimonin | sdake_: yes I can understand that :) | 14:22 |
| sdake_ | not for full workflow usage | 14:22 |
| awebber | sbezverk i dev on fedora but prefer to deploy within containers on the coreos cluster | 14:22 |
| sdake_ | msimonin if you come to me and say "libvirt doesn't work with reocnfigure" | 14:22 |
| sdake_ | then its #1 priority | 14:22 |
| sdake_ | or in the top tier | 14:22 |
| sbezverk | awebber: I get it, it is just I have never tried it, I am not sure about the end result ;-) | 14:23 |
| sdake_ | msimonin huikang is the cat to ask - he added the cool work around fake drivers | 14:23 |
| msimonin | sdake_: ack | 14:23 |
| sdake_ | my guess is it needs some reconfigure logic | 14:23 |
| msimonin | is he around ? | 14:23 |
| sdake_ | cool well i'm glad we got that settled | 14:23 |
| sdake_ | msimonin he is not - he is in us tz | 14:23 |
| awebber | sbezverk does the concept specifiying the network adapters for internal and external disappear with kolla-kubernetes? | 14:24 |
| sbezverk | awebber: no it does not especially for multinode installation | 14:25 |
| sbezverk | awebber: network or tunnel is used to get ip address for vxlan tunnels between compute nodes and network/controller nodes | 14:26 |
| msimonin | sdake_: thanks for the help, I guess I'll go find huikang | 14:26 |
| sdake_ | msimonin i asked him in the bug log to take a look | 14:26 |
| sdake_ | hopefully he will | 14:26 |
| sbezverk | awebber: if it is all in one then it does not matter much.. | 14:26 |
| msimonin | sdake_: ack | 14:27 |
| sdake_ | he will probably be in a few hours but i can't guarantee other peoples schedules i dont know :) | 14:27 |
| msimonin | :) | 14:27 |
| msimonin | so folks, I wish you a good release ! | 14:27 |
| sbezverk | awebber: what about images? are you planning to use from docker hub? | 14:27 |
| sdake_ | msimonin thanks! | 14:27 |
| sdake_ | msimonin thanks for spending time talking with us | 14:27 |
| awebber | sbezverk i am re-doing my successful multi-node kolla deployment with kolla-kubernetes - i'm finding it difficult to understanding networking configuration as an overlay is now in play | 14:27 |
| sdake_ | msimonin so we could get this off our plates (either fixed or categorized properly) | 14:27 |
| awebber | sbezverk i am using my own private registry as a result of kolla-build --registry --push | 14:28 |
| sdake_ | msimonin why using fake driver? | 14:28 |
| awebber | sbezverk i am hoping to reuse those containers | 14:28 |
| msimonin | sdake_: control plane evaluation | 14:28 |
| sbezverk | awebber: ok, when did you built those images? | 14:29 |
| sdake_ | msimonin cool i get the use case, but if you want a proper eval you ahve to bit the bullet nad get 3 machines (vms work with some pain) and test things out that way | 14:30 |
| sdake_ | msimonin note we have evaled kolla at 123 nodes (3 control 20 storage 100 compute0 and things went very well | 14:30 |
| sdake_ | msimonin there are lots of 100+ node evals either taking place or going into deployment with newton | 14:30 |
| msimonin | sdake_: we evaluate the control plane using 1000 computes | 14:30 |
| sdake_ | hrm | 14:30 |
| sdake_ | rabbitmq will probably implode there | 14:31 |
| sdake_ | you may have to wait for multiregion for that | 14:31 |
| sbezverk | awebber: the reason for my question is some minor changes were committed on kolla side to make kolla images a bit more kolla-kube friendly | 14:31 |
| msimonin | sdake_: it's actually tolerable | 14:31 |
| msimonin | sdake_: huge load under rally but it handle the load | 14:31 |
| msimonin | sdake_: next cycle we'll go multi region | 14:31 |
| sdake_ | would be interested to see an oprofile of your control nodes under that kind of load | 14:31 |
| sdake_ | along with an idle time measurement | 14:32 |
| msimonin | sdake_: actually the scheduler was more critical | 14:32 |
| sdake_ | msimonin any special settings needed that we are missing? | 14:33 |
| sdake_ | pbourke can i get an all go from you on tag or do you have further work needed | 14:33 |
| msimonin | sdake_: https://www.openstack.org/summit/barcelona-2016/summit-schedule/global-search?t=Chasing+1000+Nodes+Scale | 14:34 |
| sdake_ | akwasnie Jeffrey4l egonzalez90 mandre rhallisey | 14:35 |
| pbourke | sdake_: good from my end | 14:35 |
| sdake_ | Jeffrey4l how well does the perf stack work | 14:35 |
| sdake_ | anyone tested fernet recently? | 14:35 |
| sdake_ | like in last 1 mo | 14:35 |
| sdake_ | Jeffrey4l reconfigure works i assume? | 14:36 |
| sdake_ | Jeffrey4l or shall i validate that | 14:36 |
| Jeffrey4l | perf stack? ELK? | 14:36 |
| sdake_ | telegraph is the perf stck | 14:36 |
| sdake_ | grafana | 14:36 |
| sdake_ | etc | 14:36 |
| Jeffrey4l | only tested the grafana. no telegraph. | 14:36 |
| sdake_ | if you tested grafana you tested telegraph | 14:37 |
| Jeffrey4l | reconfigure should be fine, at lease for core service. | 14:37 |
| sdake_ | did it work | 14:37 |
| Jeffrey4l | yep. | 14:37 |
| sdake_ | Jeffrey4l nd upgrde with a vm now works? | 14:37 |
| Jeffrey4l | yep. | 14:38 |
| Jeffrey4l | tested | 14:38 |
| *** sdake_ has quit IRC | 14:39 | |
| *** michauds has joined #openstack-kolla | 14:40 | |
| awebber | sbezverk i am building from the master branch | 14:42 |
| awebber | sbezverk so i guess i have those commits unless they were within the last week | 14:42 |
| *** prithiv has quit IRC | 14:42 | |
| *** bmace has quit IRC | 14:44 | |
| *** zhubingbing_ has quit IRC | 14:45 | |
| *** sdake has joined #openstack-kolla | 14:46 | |
| *** awebber has quit IRC | 14:50 | |
| *** awebber has joined #openstack-kolla | 14:50 | |
| awebber | sbezverk when secret-generator.py create is run it produces the following error - No handlers could be found for logger "root" | 14:51 |
| sdake | msimonin - ok i hate to say it, but i think your use csae is going to have to wait until ocata if at all | 14:51 |
| sbezverk | awebber: master is good | 14:52 |
| sdake | msimonin it may be that reconfigure is not a good fit for the fake driver | 14:52 |
| sdake | ok 2 tests we need | 14:52 |
| sdake | fernet | 14:52 |
| *** HyperJohnGraham_ has joined #openstack-kolla | 14:52 | |
| sdake | enable all services and make sure they deploy | 14:52 |
| sdake | (not work but deploy) | 14:52 |
| sdake | and upgrade and reconfigure without crater | 14:52 |
| sbezverk | awebber: does it create secrets? kubectl get secrets --namespace=kolla | 14:52 |
| sdake | Jeffrey4l can ou take one of these and i'll take theother | 14:52 |
| awebber | sbezverk it doesnt get that far | 14:53 |
| Jeffrey4l | which one? | 14:53 |
| sdake | Jeffrey4l up to you | 14:53 |
| Jeffrey4l | sdake, i am testing fernet. | 14:53 |
| Jeffrey4l | right now. | 14:53 |
| awebber | sbezverk i am starting to think it might be better if i run this all in kolla-toolbox....? maybe it is trying to switch to the root user from ansible user | 14:53 |
| sdake | Jeffrey4l ok i'll test the other part AIO | 14:53 |
| Jeffrey4l | cool | 14:53 |
| sdake | Jeffrey4l i've got meetings this morning so should get to it around 11ish | 14:54 |
| awebber | sbezverk kolla_kubernetes/utils.py", line 126, in merge_configs_to_dict - No handlers could be found for logger "root" | 14:54 |
| Jeffrey4l | roger. | 14:54 |
| *** srwilkers has quit IRC | 14:55 | |
| sdake | ok - NO BACKPORTS TO STABLE/newton | 14:55 |
| sdake | unless it is a critical bug for rc3 and marked in the tracker as such | 14:55 |
| sdake | thanks :) | 14:55 |
| Jeffrey4l | hmm fernet fix need be backport. | 14:55 |
| Jeffrey4l | https://review.openstack.org/#/c/369418/ | 14:55 |
| sdake | Jeffrey4l can you sort it out | 14:56 |
| Jeffrey4l | it is not included in newton. | 14:56 |
| Jeffrey4l | yep. | 14:56 |
| awebber | sbezverk fails on KollaKubernetesResources.GetJinjaDict()[nsname] | 14:56 |
| sdake | i am witing on gate to tag | 14:56 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix keystone fernet file exchange via ssh https://review.openstack.org/388063 | 14:56 |
| Jeffrey4l | sdake, ^^ | 14:56 |
| awebber | awebber i have installed "full install" not "development"... | 14:56 |
| sdake | Jeffrey4l i assume tha twent through solid review - so I am just going to rubber stamp it | 14:57 |
| sdake | pbourke ^^ | 14:57 |
| *** bjolo has quit IRC | 14:57 | |
| sdake | Jeffrey4l if you can test it that would be grand | 14:57 |
| *** unicell has joined #openstack-kolla | 14:57 | |
| awebber | sbezverk ..and do a “Development Install” of kolla and kolla-kubernetes. This is absolutely required - perhaps thats my issue? | 14:57 |
| Jeffrey4l | sdake, i am testing it and found it is not backport. | 14:57 |
| Jeffrey4l | while add the fix and test again. | 14:57 |
| *** egonzalez90 has quit IRC | 14:57 | |
| *** unicell1 has quit IRC | 14:59 | |
| Jeffrey4l | while/will | 14:59 |
| sbezverk | awebber: yes it is required also you need to create kolla namespace | 14:59 |
| *** bjolo has joined #openstack-kolla | 14:59 | |
| sdake | Jeffrey4l one of your recent commits merged without a backport | 15:01 |
| sdake | and I can't find it | 15:01 |
| Jeffrey4l | hmm. did u remember what it is about? | 15:02 |
| sdake | it wa this morning | 15:02 |
| *** mnasiadka has quit IRC | 15:03 | |
| openstackgerrit | Merged openstack/kolla: ansible: swift: Fix swift-object-expirer restart loop https://review.openstack.org/386729 | 15:03 |
| sdake | retries:5? | 15:03 |
| Jeffrey4l | yep. | 15:03 |
| Jeffrey4l | i will push it. | 15:03 |
| sdake | its already in master | 15:04 |
| sdake | but not sure if its in stable/newton | 15:04 |
| *** eaguilar_ has quit IRC | 15:04 | |
| Jeffrey4l | it is merged in master and no backport in newton. | 15:04 |
| sdake | ok folks - t-7 hours to go! | 15:05 |
| sdake | Jeffrey4l right can you get the backportrolling | 15:05 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Bump retries in starting nova-libvirt to 5 https://review.openstack.org/388066 | 15:05 |
| Jeffrey4l | sdake, done ^^ | 15:05 |
| sdake | pbourke rubber stamp that one plz ^^ | 15:05 |
| *** piaccarino has joined #openstack-kolla | 15:05 | |
| sdake | folks I know the gates are busted | 15:05 |
| sdake | infra is working on it | 15:06 |
| sdake | until then lets proceed as planned | 15:06 |
| sdake | which is getting this stuff merged | 15:06 |
| sdake | everything went through ci in msater already | 15:06 |
| sdake | and newton and master have not diverged all that significantly yet | 15:06 |
| sdake | thanks pbourke | 15:06 |
| openstackgerrit | caoyuan proposed openstack/kolla: Fix aodh upgrade permission issue https://review.openstack.org/388067 | 15:06 |
| sdake | note - we are not backporting any more upgrade patches sunless upgrade is somehow broken | 15:07 |
| sdake | we have all the mitaka services | 15:07 |
| sdake | coveredi n backports already | 15:07 |
| sdake | cool so we know multinode ceph upgrade works | 15:08 |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla: Murano fails to deploy - cannot check core library https://review.openstack.org/388068 | 15:08 |
| sdake | we think reconfigure works - let me run that real quick | 15:08 |
| msimonin | sdake: It's reasonnable to wait Ocata indeed | 15:09 |
| sdake | msimonin i think newton is good to go - but for that bug ocata at earliest | 15:09 |
| sdake | we know TLS works | 15:10 |
| sdake | we know perf monitoring stack works - for a various definition of done :) | 15:10 |
| sdake | how about the other operations such as bootstrap/etc? | 15:11 |
| sdake | anyone know of problems with those | 15:11 |
| sdake | sean-k-mooney about? | 15:11 |
| *** spsingh has joined #openstack-kolla | 15:11 | |
| sdake | reconfigure works on the core services | 15:11 |
| sdake | what about mariadb recovery Jeffrey4l ? | 15:12 |
| *** MarMat has joined #openstack-kolla | 15:12 | |
| sdake | is that something to be covered in 3.0.1? | 15:12 |
| msimonin | sdake: do you have any link regarding the planned experiments for Newton ? (100+ node evals) | 15:12 |
| sdake | msimonin we ran on osic gear | 15:12 |
| sdake | moment let me find review | 15:12 |
| sdake | pbourke did the requirements thing ever get fixed for stable/newton | 15:13 |
| Jeffrey4l | my fix is merged in mitaka and newton. But it is not ideal. some edge case is not covered. | 15:13 |
| sdake | Jeffrey4l ok - better then what it was? | 15:13 |
| msimonin | sdake: I checked osic experiments using 123 nodes, but just wondering if you plan other experiment for the next cycle | 15:14 |
| Jeffrey4l | yep. one more case can be fixed. | 15:14 |
| sdake | msimonin nah - we have to submit proposals to osic, and they have to agree | 15:14 |
| sdake | msimonin the paperowrk requireements re duanting, we are working on sorting out paperwork requirements from our last experimentation on osic gear | 15:14 |
| sdake | msimonin plus I dont have another osic test in me rlated to kolla ;-) | 15:15 |
| openstackgerrit | caoyuan proposed openstack/kolla: Remove the unnecessary blank https://review.openstack.org/388071 | 15:15 |
| openstackgerrit | Merged openstack/kolla: Bump retries in starting nova-libvirt to 5 https://review.openstack.org/388066 | 15:16 |
| wznoinsk | sdake: hi, any chance to push the last bits of ironic/iscsid into newton ? | 15:17 |
| sdake | wznoinsk how many patches | 15:19 |
| sdake | wznoinsk and what does it touch | 15:19 |
| sdake | iscisd is lready implemented | 15:19 |
| wznoinsk | sdake: https://review.openstack.org/#/q/status:open+project:openstack/kolla+branch:stable/newton+owner:%22Waldemar+Znoinski+%253Cwaldemar.znoinski%2540intel.com%253E%22 | 15:20 |
| wznoinsk | iscsid is in kolla but for cinder, the above makes it working with ironic | 15:20 |
| kfox1111 | morning. | 15:21 |
| wznoinsk | sdake: and last (https://review.openstack.org/#/c/387684/) of 3 from the above is fixing last bit of ironic itself on xenial | 15:21 |
| *** eaguilar has joined #openstack-kolla | 15:22 | |
| sbezverk | kfox1111: morning | 15:22 |
| *** ssurana has joined #openstack-kolla | 15:22 | |
| wznoinsk | Jeffrey4l: could you have a look on those 3 pls ^ ? | 15:22 |
| * Jeffrey4l is checking | 15:23 | |
| sdake | wznoinsk make an etherpad | 15:23 |
| sdake | wznoinsk with your patches | 15:23 |
| sdake | wznoinsk get Jeffrey4l to sign off on them | 15:23 |
| sdake | Jeffrey4l if there is *ANY* risk to the release - -1 em for now | 15:23 |
| Jeffrey4l | roger. | 15:23 |
| *** skramaja has quit IRC | 15:24 | |
| wznoinsk | sdake: Jeffrey4l https://etherpad.openstack.org/p/kolla-newton-ironic-iscsid | 15:27 |
| Jeffrey4l | sdake, wznoinsk LGTM for these three PS. i think we can merged it. | 15:27 |
| Jeffrey4l | Patch Set 1: Code-Review-1 Workflow-1 | 15:27 |
| Jeffrey4l | this is non-mitaka branch service, so it is not affect upgrade from mitaka to newton. | 15:27 |
| Jeffrey4l | 1. use different bug id. | 15:27 |
| Jeffrey4l | 2. no need backport to newton. | 15:27 |
| Jeffrey4l | 2. -2 until rc3 is make | 15:27 |
| Jeffrey4l | https://review.openstack.org/#/q/status:open+project:openstack/kolla+branch:stable/newton+owner:%22Waldemar+Znoinski+%253Cwaldemar.znoinski%2540intel.com%253E%22 | 15:27 |
| *** srwilkers has joined #openstack-kolla | 15:27 | |
| Jeffrey4l | wznoinsk, roger. i have +2 for all of them. | 15:27 |
| wznoinsk | thanks | 15:27 |
| Jeffrey4l | it should be merged into newton. | 15:27 |
| wznoinsk | it would make me sleep better for sure :-) | 15:28 |
| *** prithiv has joined #openstack-kolla | 15:28 | |
| sdake | Jeffrey4l make sure the bugs are marked critical and in the rc3 tracker | 15:28 |
| Jeffrey4l | ok. | 15:28 |
| sdake | lets make a new etherpad | 15:29 |
| sdake | to track these last minute changes | 15:29 |
| sdake | if anyone breaks kolla - sdake ging to be severly unhappy :) | 15:29 |
| sdake | wznoinsk since all your patches are merging, you get to test ugprade/reconfgiure/deploy in place ;) | 15:29 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: Testing trunk... https://review.openstack.org/387641 | 15:31 |
| wznoinsk | never tried upgrade, that's from 2.0.0 to 3.0.0 when it's out? | 15:32 |
| Jeffrey4l | sdake, done. | 15:32 |
| Jeffrey4l | sdake, etherpad link? | 15:32 |
| Jeffrey4l | btw, keystone fernet works. | 15:33 |
| Jeffrey4l | with my patch https://review.openstack.org/388063 . | 15:33 |
| Jeffrey4l | but gate bust :/ | 15:33 |
| *** vhosakot has joined #openstack-kolla | 15:33 | |
| sdake | Jeffrey4l fernet works - good news! | 15:33 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO_NOT_MERGE: TEST NEWTON BRANCH https://review.openstack.org/386981 | 15:34 |
| openstackgerrit | Merged openstack/kolla: copy iscsid logrotate config https://review.openstack.org/387683 | 15:34 |
| sdake | ok folks - just spoke with infra | 15:36 |
| sdake | they knwo thegates are broken | 15:36 |
| sdake | they are working on it | 15:36 |
| sdake | it is not clear if it will be fixed by our tag time | 15:36 |
| sdake | SO | 15:36 |
| sdake | we need to totally lock down all further changes | 15:36 |
| sdake | and any reviews should be treated with a careful eye if they are not master backports where the master gate was successful | 15:37 |
| sdake | Jeffrey4l can you try deploy everything | 15:37 |
| sdake | Jeffrey4l and see whats broken wrt upgrade/reconfigure and stick in stubs if necessary | 15:37 |
| Jeffrey4l | sdake, sure | 15:37 |
| sdake | by stubs, I mean --- | 15:37 |
| sdake | - | 15:37 |
| mandre | ok | 15:38 |
| mandre | careful, stupid and totally unrelated question incoming | 15:38 |
| mandre | if the extend_start.sh scripts are normally run as unpriviledge users, how come they can create directories in /var/log/kolla? | 15:39 |
| sdake | all kolla-ansible API commands work for me with the core services plus central logging | 15:39 |
| otavio | sdake: Jeffrey4l: can my backport-fixes topic to be merged in mitaka? | 15:39 |
| sdake | mandre they are actually run as root ;-) | 15:39 |
| sdake | mandre via sudo | 15:39 |
| Jeffrey4l | what's it ? otavio | 15:39 |
| sdake | mandre moment | 15:39 |
| otavio | Jeffrey4l: the vagrant related fixes | 15:39 |
| otavio | Jeffrey4l: you did review it | 15:39 |
| mandre | sdake: ah, that might explain :) | 15:39 |
| Jeffrey4l | mandre, i know it. | 15:40 |
| openstackgerrit | Merged openstack/kolla: set basic kolla structures for iscsid https://review.openstack.org/387685 | 15:40 |
| openstackgerrit | Merged openstack/kolla: copy ldlinux.c32 to /tftpboot in ironic_pxe https://review.openstack.org/387684 | 15:40 |
| sdake | mandre https://github.com/openstack/kolla/blob/master/docker/base/start.sh#L14 | 15:40 |
| Jeffrey4l | /var/log/kolla is owner by :kolla group. so every user can create folders in it. ( every user is a member of kolla group ) | 15:40 |
| Jeffrey4l | mandre, sdake https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L286 | 15:41 |
| mandre | Jeffrey4l: what sets the group ownership of /var/log/kolla? | 15:41 |
| sdake | mandre actually that may not be the question you asked | 15:41 |
| Jeffrey4l | i added that recently. | 15:41 |
| mandre | sdake: actually, if you look at like 20, kolla_extend_start is run as normal user | 15:42 |
| openstackgerrit | Merged openstack/kolla: Fix keystone fernet file exchange via ssh https://review.openstack.org/388063 | 15:42 |
| sdake | mandre your right | 15:42 |
| sdake | mandre tools inside extend_start run as sudo however | 15:42 |
| sdake | (if needed) | 15:42 |
| sdake | nice - well it looks like 3.0.0 is golden from "core services" POV | 15:43 |
| *** SPSINGH__ has joined #openstack-kolla | 15:44 | |
| mandre | sdake: true, but not ensuring /var/log/kolla/foo dir exists which is run as normal user, I think Jeffrey4l is right about the kolla group but I can't find where it's set | 15:44 |
| sdake | pbourke do you recallt he name of that hip website that produces 1 page CVs | 15:44 |
| Jeffrey4l | mandre, it is added in the base image https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L286 | 15:44 |
| pbourke | sdake https://enhancv.com/ | 15:45 |
| sdake | pbourke thanks | 15:45 |
| *** piaccarino has quit IRC | 15:45 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO_NOT_MERGE: TEST NEWTON BRANCH https://review.openstack.org/386981 | 15:45 |
| *** piaccarino has joined #openstack-kolla | 15:46 | |
| *** vhosakot has quit IRC | 15:46 | |
| *** vhosakot has joined #openstack-kolla | 15:46 | |
| *** spsingh has quit IRC | 15:47 | |
| Jeffrey4l | otavio, i prefer not. we only backport critical issue. not feature. | 15:49 |
| *** zhurong has quit IRC | 15:52 | |
| sdake | we also backport high features | 15:55 |
| sdake | but not around release time | 15:55 |
| sdake | rather high bugs | 15:56 |
| sdake | we never backport features | 15:56 |
| *** msimonin has quit IRC | 15:56 | |
| Jeffrey4l | sdake, high feature means? | 15:56 |
| *** caowei has quit IRC | 15:56 | |
| sdake | Jeffrey4l it was a misspeak | 15:57 |
| sdake | i meant high bug | 15:57 |
| Jeffrey4l | roger. | 15:57 |
| sdake | 9am dude - almost awake ;) | 15:57 |
| Jeffrey4l | 12 pm here almost sleepy ;) | 15:58 |
| otavio | Jeffrey4l: I think vagrant is not a dev; it is there but broken | 15:58 |
| Jeffrey4l | otavio, you should think like this: vagrant only used by develop. and fewer develop will work on mitaka branch. so backport is almost useless ;) | 15:59 |
| *** ssurana has quit IRC | 16:01 | |
| *** sdake_ has joined #openstack-kolla | 16:03 | |
| mandre | awesome, thanks Jeffrey4l | 16:03 |
| *** rmart04 has left #openstack-kolla | 16:03 | |
| Jeffrey4l | np | 16:03 |
| *** sdake has quit IRC | 16:06 | |
| *** irtermit- is now known as irtermit | 16:06 | |
| *** irtermit is now known as irtermite | 16:06 | |
| vhosakot | yes, vagrant in master does not work for me either... I deploy kolla manually using the QSG in vm | 16:09 |
| -openstackstatus- NOTICE: pycparser 2.16 released to fix assertion error from today. | 16:11 | |
| *** b_bezak has quit IRC | 16:11 | |
| *** prithiv has quit IRC | 16:11 | |
| *** g3ek has quit IRC | 16:12 | |
| *** matrohon has quit IRC | 16:13 | |
| *** haplo37 has quit IRC | 16:13 | |
| *** confisurya has joined #openstack-kolla | 16:14 | |
| *** SPSINGH__ has quit IRC | 16:14 | |
| *** g3ek has joined #openstack-kolla | 16:15 | |
| *** haplo37 has joined #openstack-kolla | 16:15 | |
| Jeffrey4l | vhosakot, vagrant is out of maintain for long. because i didn't see any fix patch. | 16:20 |
| Jeffrey4l | recently. | 16:20 |
| vhosakot | Jeffrey4l: yep, I reviews some patches for vagrant.. but agreed, it is not under active dev/maintenance | 16:21 |
| *** awebber has quit IRC | 16:22 | |
| *** inc0 has joined #openstack-kolla | 16:23 | |
| inc0 | good morning | 16:23 |
| Jeffrey4l | morning | 16:23 |
| *** gfidente is now known as gfidente|afk | 16:28 | |
| mliima | morning | 16:29 |
| sbezverk | kfox1111: ping | 16:33 |
| kfox1111 | sbezverk: whats up? | 16:34 |
| sbezverk | kfox1111: hit wierd bug in nuetron | 16:34 |
| kfox1111 | whats it doing? | 16:34 |
| sbezverk | aplicable only to kube and on cluster reboot | 16:35 |
| sbezverk | https://bugs.launchpad.net/neutron/+bug/1634123 | 16:35 |
| openstack | Launchpad bug 1634123 in neutron "neutron openvswitch agent exits if unix:/var/run/openvswitch/db.sock is not yet created" [High,New] | 16:35 |
| *** manjeets has quit IRC | 16:35 | |
| sbezverk | I also propesd fix, curious what neutron people will say | 16:35 |
| *** manjeets_ is now known as manjeets | 16:36 | |
| kfox1111 | yeah. retrying is probably the best option. | 16:36 |
| kfox1111 | although, crashing would work too. k8s would clean it up and try again. | 16:36 |
| kfox1111 | a third option might be to just try and create the socket in the template. | 16:37 |
| sbezverk | kfox1111: I do not want to scare neutron with crashing option ;-) | 16:37 |
| kfox1111 | or a forth would we to look for the socket in the template before kolla start and bailing if it doesn't exist. | 16:37 |
| kfox1111 | hmm.. 4 might be the best option for us, for now, as it will work with mitaka/newton too. | 16:38 |
| sbezverk | kfox1111: I would prefer retrying as it is the simplest. | 16:38 |
| kfox1111 | [ -f /var/run/openvswitch/db.sock ] && exit -1 | 16:38 |
| kfox1111 | k8s would then retry. | 16:38 |
| kfox1111 | a little more complicated: | 16:38 |
| kfox1111 | while [ ! -f /var/run/openvswitch/db.sock ]; do sleep 1; done | 16:39 |
| sbezverk | kfox1111: I guess for newton, until retry gets merged, that could be a solution.. | 16:40 |
| bjolo__ | build errors stable/newton ubuntu binary mariadb container. anyone else getting this? | 16:40 |
| kfox1111 | sbezverk: I still want to support mitaka for a while too. | 16:41 |
| kfox1111 | that gives me a smooth transition to newton. | 16:41 |
| bjolo__ | http://paste.openstack.org/show/586239 | 16:41 |
| kfox1111 | upgrade non kolla-kubernetes cloud from mitaka to mitaka k8s. then in a seperate outage, upgrade from mitaka to newton. | 16:41 |
| sbezverk | kfox1111: another thing, if you could check latest fencing ps would be good.. I tested it with latest changes and it works well | 16:41 |
| *** ayoung has quit IRC | 16:42 | |
| kfox1111 | sbezverk: I comment on it some more on the channel yesterday. | 16:43 |
| kfox1111 | I really think it should only ever try and shoot nodes/pods that contain rbd locks. | 16:43 |
| kfox1111 | otherwise, noisy neighbor vm's can kill themselves and others on the same node potentially. | 16:44 |
| bjolo__ | ubuntu source also failing | 16:44 |
| kfox1111 | so the algorithm should be: look for potentially dead nodes. of those, look for ones that hold rbd locks. Of those: | 16:45 |
| kfox1111 | ipmitool power off | 16:45 |
| sbezverk | kfox1111: I missed that part will look into it | 16:45 |
| kfox1111 | remove rbd lock | 16:45 |
| kfox1111 | shoot the pods on it. | 16:46 |
| kfox1111 | that should effect the smallest amount of nodes. | 16:46 |
| kfox1111 | really just the nodes running mariadb and rabbit. | 16:46 |
| sbezverk | kfox1111: before you were saying a bit different, you want to power off first, then clear locks | 16:46 |
| kfox1111 | no. I've been consistent on the fencing. ipmi power off (fence) the node before the rbd lock is released, or else data corruption. | 16:47 |
| sbezverk | kfox1111: but I think shooting only nodes with locks is right approach | 16:47 |
| kfox1111 | kk | 16:48 |
| kfox1111 | I think some of the confusion was probably in that there are two parts to the rbd locks. | 16:49 |
| kfox1111 | looking for locks, and unlocking the locks. | 16:49 |
| kfox1111 | unlocking the locks must happen after the fencing of the node. | 16:49 |
| kfox1111 | but its safe to look for locks before. | 16:49 |
| kfox1111 | sdake_: trunk seems happy with kolla-kubernetes again. so that patch fixed it. | 16:52 |
| kfox1111 | I'll try and keep running tests against it. if you can let me know when we're done adding patches right before release though, I can run one more to be sure. | 16:52 |
| *** diogogmt has joined #openstack-kolla | 16:53 | |
| kfox1111 | bbiab | 16:54 |
| *** ssurana has joined #openstack-kolla | 16:57 | |
| otavio | Jeffrey4l: I disagree. Vagrant were used here as test environment before putting it at real hw | 17:11 |
| otavio | Jeffrey4l: so it is important to be kept running | 17:11 |
| otavio | Jeffrey4l: also the low usage can be seen as a low risk and people would benefit from the better user experience | 17:11 |
| Jeffrey4l | i doubt fewer people are using vargran. and vagrant is bust on master too. | 17:12 |
| kfox1111 | I havent gotten vagrent to ever work well with kvm, and I need kvm for my openstack stuff. I'd probably use vagrent more if I coudl figure that out. | 17:15 |
| Jeffrey4l | kfox1111, you can try nested kvm | 17:16 |
| kfox1111 | Jeffrey4l: yeah, in that way lies pain... :) | 17:16 |
| kfox1111 | I've just found it easier to skip vagrent because of that... so far it hasn't been too difficult. | 17:17 |
| *** unicell has quit IRC | 17:17 | |
| Jeffrey4l | yes. dependency is less for kolla. | 17:17 |
| *** confisurya has quit IRC | 17:18 | |
| inc0 | vagrant is bust because nobody cares for them | 17:23 |
| inc0 | in opensource, you fix what you feel is broken usually | 17:23 |
| otavio | inc0: we used it and sent the patches | 17:23 |
| inc0 | if somebody would want to take on fixing vagrant, we'll be happy to help and review | 17:24 |
| otavio | inc0: I dont care as now we moved to newton | 17:24 |
| inc0 | so I'm pretty sure that if we fix vagrant on master | 17:24 |
| inc0 | it should be pretty easy to deploy newton with it too | 17:24 |
| inc0 | even tho code itself will remain broken in tree | 17:25 |
| inc0 | hell, we might even backport this bug to newton if it's fixed | 17:25 |
| inc0 | not a big issue | 17:25 |
| inc0 | vagrant is in our tree -> it's broken -> it's bug -> possible to backport | 17:25 |
| inc0 | but we need fix first:) | 17:25 |
| *** tonanhngo has quit IRC | 17:26 | |
| *** tonanhngo has joined #openstack-kolla | 17:26 | |
| otavio | inc0: mitaka fixes are under review | 17:26 |
| inc0 | otavio, it has to be fixed in master | 17:27 |
| inc0 | and then backported to stable if anything | 17:27 |
| otavio | inc0: it is; the fixes I added for mitaka were backports | 17:27 |
| otavio | inc0: all them | 17:27 |
| inc0 | ok | 17:27 |
| inc0 | but you're saying that you're moving to newton right? | 17:27 |
| Jeffrey4l | if vagrant works in master, copy it to stable branch should works. | 17:29 |
| inc0 | but truth is, not our highest priority | 17:32 |
| *** kwazar has quit IRC | 17:33 | |
| *** jistr has quit IRC | 17:33 | |
| *** kklimonda has quit IRC | 17:33 | |
| *** irtermite has quit IRC | 17:33 | |
| *** nick-ma has quit IRC | 17:33 | |
| *** igordcard has quit IRC | 17:33 | |
| *** flaper87 has quit IRC | 17:33 | |
| *** DuncanT has quit IRC | 17:33 | |
| *** amaged__ has quit IRC | 17:33 | |
| *** qbjolof1 has quit IRC | 17:33 | |
| *** Guest66666 has quit IRC | 17:33 | |
| *** jgriffith has quit IRC | 17:33 | |
| *** esmiurium has quit IRC | 17:33 | |
| *** EmilienM has quit IRC | 17:33 | |
| *** cu5 has quit IRC | 17:33 | |
| *** mgkwill has quit IRC | 17:33 | |
| *** SaMnCo has quit IRC | 17:33 | |
| *** harmw has quit IRC | 17:33 | |
| *** nihilifer has quit IRC | 17:33 | |
| *** harbie has quit IRC | 17:33 | |
| *** dims has quit IRC | 17:33 | |
| *** smekel_ has quit IRC | 17:33 | |
| *** jroll has quit IRC | 17:33 | |
| *** tonyb has quit IRC | 17:33 | |
| *** cloudnull has quit IRC | 17:33 | |
| *** jgriffith_ has joined #openstack-kolla | 17:33 | |
| *** irtermite has joined #openstack-kolla | 17:33 | |
| *** dims has joined #openstack-kolla | 17:33 | |
| *** tonyb has joined #openstack-kolla | 17:33 | |
| *** igordcard has joined #openstack-kolla | 17:33 | |
| *** harbie has joined #openstack-kolla | 17:33 | |
| *** esmiurium has joined #openstack-kolla | 17:33 | |
| *** Guest66666 has joined #openstack-kolla | 17:33 | |
| *** qbjolof1 has joined #openstack-kolla | 17:33 | |
| *** kwazar has joined #openstack-kolla | 17:33 | |
| *** nihilifer has joined #openstack-kolla | 17:34 | |
| *** harbie has quit IRC | 17:34 | |
| *** harbie has joined #openstack-kolla | 17:34 | |
| *** cloudnull has joined #openstack-kolla | 17:34 | |
| *** janem has quit IRC | 17:34 | |
| *** cu5 has joined #openstack-kolla | 17:34 | |
| *** kklimonda has joined #openstack-kolla | 17:34 | |
| *** jistr has joined #openstack-kolla | 17:34 | |
| *** michauds has quit IRC | 17:35 | |
| *** jroll has joined #openstack-kolla | 17:35 | |
| *** EmilienM has joined #openstack-kolla | 17:36 | |
| *** steve-noyes has quit IRC | 17:37 | |
| *** athomas has quit IRC | 17:38 | |
| *** SaMnCo has joined #openstack-kolla | 17:38 | |
| *** nick-ma_ has joined #openstack-kolla | 17:40 | |
| *** haplo37_ has quit IRC | 17:41 | |
| Jeffrey4l | fyi, ci should be unblocked now. | 17:41 |
| *** DuncanT has joined #openstack-kolla | 17:41 | |
| *** tonanhngo has quit IRC | 17:42 | |
| *** mgkwill has joined #openstack-kolla | 17:42 | |
| *** amaged__ has joined #openstack-kolla | 17:43 | |
| *** haplo37_ has joined #openstack-kolla | 17:43 | |
| *** smekel_ has joined #openstack-kolla | 17:44 | |
| *** unicell has joined #openstack-kolla | 17:44 | |
| *** harmw has joined #openstack-kolla | 17:45 | |
| sbezverk | Jeffrey4l: ping | 17:46 |
| Jeffrey4l | sbezverk, pong | 17:46 |
| sbezverk | Jeffrey4l: listen this PS needed to be rebased it had all +2 already, and has not been merged only because of depenedcy | 17:47 |
| sbezverk | that is why it was cherrypicked | 17:47 |
| sdake_ | hey peeps | 17:47 |
| sbezverk | also we do not drop rott in kube | 17:47 |
| *** srwilkers has quit IRC | 17:47 | |
| sbezverk | you do not use this image in kolla | 17:47 |
| sbezverk | so what is uup with USER comment??? | 17:48 |
| *** srwilkers has joined #openstack-kolla | 17:48 | |
| Jeffrey4l | ok. | 17:48 |
| *** srwilkers has quit IRC | 17:48 | |
| Jeffrey4l | for the cherry-pick in commit. you are cherry pick patch set 3 into patch set 4. | 17:48 |
| Jeffrey4l | sbezverk, | 17:48 |
| Jeffrey4l | check the commit hash 78c981dc70e110c3d09039d14a5845cb46246399 | 17:49 |
| sdake_ | otavio - we are 6 hours from tag - not merging any non-critical issues related to the release | 17:49 |
| sbezverk | Jeffrey4l: main.yml modification is MUST because otherwise it does not get renedered properly | 17:49 |
| sdake_ | otavio if you really want that work done - we can backport it - but 3.0.0 is locked in unless there is some critical failure | 17:49 |
| sdake_ | thanks kfox1111 re kolla-kubernetes | 17:49 |
| sbezverk | Jeffrey4l: what does it change if hash do not match? | 17:50 |
| *** awebber has joined #openstack-kolla | 17:50 | |
| sdake_ | Jeffrey4l any critical failures on deploy everything? | 17:50 |
| otavio | sdake_: it is backports from 3.0.0 onto 2.0.3 | 17:50 |
| sdake_ | Jeffrey4l or did you get to that? | 17:51 |
| Jeffrey4l | sbezverk, still no idea why kolla ansible matter when you add a docker fie. | 17:51 |
| sdake_ | otavio 3.0.0 is working? | 17:51 |
| otavio | sdake_: it is backport, no new changes. | 17:51 |
| Jeffrey4l | sdake_ that? what? | 17:51 |
| otavio | oh my godness. Of course. All those changes ARE in 3.0.0. That's why the topic is called backport-fixes | 17:51 |
| *** neilus has joined #openstack-kolla | 17:52 | |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Allow cinder-volume to be configured to use NFS https://review.openstack.org/279913 | 17:52 |
| Jeffrey4l | the key point is: you are backporting a bunch of patch sets, some is bug fix ( without bug id, trivial fix ) , some are small nice-have feature. | 17:53 |
| otavio | sdake_: otherwise I would have sent them for master before asking for backport. This is how I did for the swift change as you remember | 17:53 |
| otavio | Jeffrey4l: in my pov all are bugfixes as vagrant were not working for us before | 17:53 |
| otavio | Jeffrey4l: but to be honest, I did it for the sake of contributing. We have our fork with those in so I dont care | 17:54 |
| otavio | Jeffrey4l: but as end user, I would hope for vagrant to be on a working stage | 17:54 |
| otavio | Jeffrey4l: or drop i | 17:54 |
| otavio | it | 17:54 |
| *** neilus has quit IRC | 17:56 | |
| *** srwilkers has joined #openstack-kolla | 17:57 | |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Allow cinder-volume to be configured to use NFS https://review.openstack.org/279913 | 17:57 |
| sdake_ | otavio ca nwe have this discussion tomorrow? | 17:58 |
| sdake_ | otavio i hate to push you off, but we are really working on getting 3.0.0 tagged | 17:58 |
| otavio | sdake_: sure; no problem | 17:58 |
| sdake_ | and today at 5pm pst is our deadline | 17:58 |
| otavio | sdake_: it is all fine. I just don't agree in ignoring end user | 17:59 |
| otavio | sdake_: as I said, my case is fine | 17:59 |
| Jeffrey4l | i do think it break the stable branch policy. some are feature, i think. | 17:59 |
| sdake_ | otavio lets chat more tomorrow | 17:59 |
| otavio | sdake_: so we talk later | 17:59 |
| *** srwilkers has quit IRC | 17:59 | |
| sdake_ | i have all afternoon and most of the morning open on wed :) | 17:59 |
| *** athomas has joined #openstack-kolla | 18:00 | |
| otavio | Jeffrey4l: If it does, and Vagrant is not seen as part of a key component which must be working, then drop it. In the end it seems to be non-supported | 18:00 |
| *** msimonin has joined #openstack-kolla | 18:01 | |
| otavio | Jeffrey4l: point me the one you think it is a feature and we discuss | 18:01 |
| otavio | Jeffrey4l: but focus on release now, we discuss tomorrow ;-) | 18:01 |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Allow cinder-volume to be configured to use NFS https://review.openstack.org/279913 | 18:02 |
| *** msimonin has quit IRC | 18:02 | |
| Jeffrey4l | sure | 18:02 |
| sdake_ | ok, so fernet - tested - core - tested - multinode - tested - ceph - tested - upgrades on centos - tested | 18:03 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Adding kubetoolbox image https://review.openstack.org/384988 | 18:04 |
| sdake_ | upgrades on ubuntu - not tested - full deploy of all services - not tested | 18:04 |
| Jeffrey4l | tesing: ceilomter gnocchi aodh logging ceph grafana cinder heat tempest horizon magnum murano deploy + reconfigure | 18:04 |
| Jeffrey4l | this almost a full deploy ^^ | 18:04 |
| sdake_ | Jeffrey4l i was thinking all the services at aonce | 18:04 |
| sdake_ | you know congress sahara etc | 18:04 |
| sdake_ | reconfigure of core - tested | 18:04 |
| sdake_ | all api commands - tested | 18:04 |
| sdake_ | where we haven't tested is ubuntu upgrades and full upgrade/reconfigure of all added services | 18:05 |
| sdake_ | i want to make sure upgrade and reconfigure dont' crater | 18:05 |
| sdake_ | i don't particuarly care if they work properly | 18:05 |
| sdake_ | just that they don't fail | 18:05 |
| sdake_ | inc0 can you test ubuntu upgrade with stable/newton? | 18:05 |
| sdake_ | i think bjolo__ was looking at this - not sure if he finished it or not | 18:05 |
| sdake_ | all that backport madness we went through yesterday will be wasted if upgrade in some way craters. | 18:07 |
| sdake_ | same story with reconfgiure | 18:07 |
| *** nihilifer has quit IRC | 18:08 | |
| Jeffrey4l | yes. really wanna get test result for ubuntu | 18:08 |
| Jeffrey4l | upgrade | 18:08 |
| sdake_ | https://etherpad.openstack.org/p/kolla-newton-final-testing | 18:09 |
| sdake_ | folks - please sign up for one of these and get er done in the next 1 hour | 18:09 |
| sdake_ | jeffreey is doing 2-4 | 18:10 |
| sdake_ | my computer is in the shop - so have no way to run a vm for ubuntu | 18:10 |
| sdake_ | TLS -tested | 18:10 |
| sdake_ | everything is looking really good | 18:10 |
| sdake_ | need ubuntu upgrade test | 18:11 |
| Jeffrey4l | Jeffrey4l, 2-4? 2-3 ;) | 18:11 |
| sdake_ | 1-3 ;) | 18:13 |
| sdake_ | i meant lines 2-4 in the etherpad | 18:13 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 18:13 |
| sbezverk | kfox1111: please check when you have time latest fencing. hopefully I captured the idea of the flow ;-) | 18:15 |
| sdake_ | ok - if no takers on the ubuntu tests, I am going to add a release note ubuntu is completely untested with upgrades and may crater your deployment | 18:15 |
| bjolo__ | sdake_, upgrade running now. | 18:15 |
| sdake_ | bjolo__ on ubuntu? | 18:15 |
| bjolo__ | should be done in a few | 18:15 |
| bjolo__ | yes ubuntu binary | 18:15 |
| Jeffrey4l | sound cool | 18:15 |
| bjolo__ | ouch failed on heat | 18:16 |
| sdake_ | have any kind of log? | 18:16 |
| Jeffrey4l | bjolo__, logs? | 18:16 |
| bjolo__ | http://paste.openstack.org/show/586252/ | 18:17 |
| sdake_ | bjolo__ did ubuntu deploy work? | 18:17 |
| bjolo__ | on mitaka? | 18:17 |
| bjolo__ | let me check one thing here first | 18:17 |
| Jeffrey4l | bjolo__, try `docker logs bootstrap_heat` | 18:17 |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Remove redundant Neutron Docker volumes https://review.openstack.org/388164 | 18:18 |
| sdake_ | bjolo__ oh nm your testing upgrades :) | 18:18 |
| bjolo__ | oh wait, heat deploy did not work on mitaka so i disabled that | 18:18 |
| bjolo__ | but it is enabled in my newton globals.yml. | 18:19 |
| bjolo__ | thats probably why the failure | 18:19 |
| Jeffrey4l | so no database in mariadb. | 18:19 |
| Jeffrey4l | during upgrade it won't create database. | 18:19 |
| sdake_ | bjolo__ why didn't heat deploy work on mitaka | 18:19 |
| bjolo__ | yes seems like | 18:20 |
| sdake_ | it works for me | 18:20 |
| bjolo__ | did not find the binary i believe | 18:20 |
| bjolo__ | using docker hub images | 18:20 |
| sdake_ | bjolo__ are you running hub 2.0.2 images? | 18:20 |
| bjolo__ | yes | 18:20 |
| sdake_ | ok, use source images pls | 18:20 |
| sdake_ | binary in 2.0.2 is afu | 18:20 |
| bjolo__ | ok | 18:20 |
| sdake_ | for ubuntu | 18:20 |
| Jeffrey4l | source is more stable in most case. | 18:21 |
| sdake_ | centos binary is generally stable as well | 18:21 |
| sdake_ | inc0 https://etherpad.openstack.org/p/kolla-newton-final-testing | 18:22 |
| bjolo__ | heka did fail tho | 18:22 |
| bjolo__ | http://paste.openstack.org/show/586255/ | 18:22 |
| bjolo__ | permission issue | 18:22 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 18:22 |
| sdake_ | groan | 18:22 |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Update globals.yml Docker registry example notes https://review.openstack.org/388167 | 18:22 |
| Jeffrey4l | hmm. heka read logs file, but lots file don't have correct permission right now. | 18:23 |
| Jeffrey4l | bjolo__, does the upgrade script done? | 18:23 |
| bjolo__ | ? | 18:24 |
| Jeffrey4l | does upgrade run successfully? | 18:24 |
| sdake_ | Jeffrey4l no it cratered on heka | 18:24 |
| Jeffrey4l | no. i do not think so. | 18:24 |
| Jeffrey4l | bjolo__, ? | 18:24 |
| bjolo__ | well it failed on heat right | 18:25 |
| bjolo__ | build mitaka ubuntu source now | 18:25 |
| sdake_ | bjolo__ enable_heat: "no" | 18:25 |
| sdake_ | bjolo__ use ubuntu source from dockerhub | 18:25 |
| Jeffrey4l | try docker restart heka bjolo__ | 18:25 |
| sdake_ | Jeffrey4l what exactly sets permissions on /var/log/kolla/haproxy? | 18:26 |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Add node_custom_config example to globals.yml https://review.openstack.org/388169 | 18:26 |
| Jeffrey4l | when heka is upgraded, the logs file in /var/log/kolla are not updated. so start heka success, and heka container failed later. | 18:27 |
| bjolo__ | restart heka did not work | 18:27 |
| Jeffrey4l | after all container is updated. heka should be fine. | 18:27 |
| Jeffrey4l | bjolo__, check the log again? | 18:27 |
| bjolo__ | disabled heat and running upgrade again | 18:27 |
| bjolo__ | same error in log | 18:27 |
| Jeffrey4l | ok. i am wrong . | 18:28 |
| Jeffrey4l | what's this file is? /var/log/kolla/haproxy/.hekad.perm_check443949710 | 18:28 |
| sdake_ | keepalive and haproxy need some json attention i think | 18:28 |
| Jeffrey4l | it is generated by heka in haproxy folder. | 18:29 |
| sdake_ | Jeffrey4l its obviously a way to see if heka can write to that folder ;) | 18:29 |
| Jeffrey4l | bjolo__, check the file in haproxy conainer /var/log/kolla/haproxy/.hekad.perm_check443949710 | 18:29 |
| bjolo__ | ok | 18:29 |
| sdake_ | there is no file | 18:29 |
| sdake_ | it couldn't be written - EPERM | 18:29 |
| sdake_ | bjolo docker exec heka ls -ld /var/log/kolla/haproxy | 18:30 |
| sdake_ | and same story with keepalived | 18:30 |
| bjolo__ | [root@eselde02u33 etc]# docker exec heka -ld /var/log/kolla/haproxy | 18:31 |
| bjolo__ | Error response from daemon: Container 281cb1fc7275430e701b57ca72ae1a370b48c460f53fe344b4fb48583a9f8d19 is restarting, wait until the container is running | 18:31 |
| bjolo__ | stuck in restarting so ... | 18:31 |
| sdake_ | ya | 18:31 |
| Jeffrey4l | bjolo__, try `ls -alh /var/lib/docker/volumes/kolla_logs/_data/haproxy/` | 18:31 |
| *** neilus has joined #openstack-kolla | 18:31 | |
| sdake_ | Jeffrey4l that won't give the container uid version of that file | 18:32 |
| sdake_ | haproxy is not writeable | 18:33 |
| sdake_ | heka tries to write to it to make sure it can | 18:33 |
| sdake_ | it can't | 18:33 |
| sdake_ | and exits | 18:33 |
| sdake_ | thats my analysis | 18:33 |
| Jeffrey4l | error opening file: open /var/log/kolla/haproxy/.hekad.perm_check443949710: permission denied | 18:34 |
| Jeffrey4l | i think he is trying to open that file. | 18:34 |
| sdake_ | open rw+ | 18:34 |
| sdake_ | = write | 18:34 |
| sdake_ | write on a permission denied on a new file = directory not writeable | 18:35 |
| sdake_ | i am tagging in 5 hours | 18:36 |
| mandre | Jeffrey4l: still around? | 18:36 |
| sdake_ | it will be with upgrades for ubuntu working or without | 18:36 |
| Jeffrey4l | mandre, yep. | 18:36 |
| sdake_ | if without - there will be a release note | 18:36 |
| mandre | Jeffrey4l: should we backport https://github.com/openstack/kolla/commit/73a39db326107212780fb3b4555b768269fc5c3f to mitaka? | 18:36 |
| sdake_ | i suggest if you care about ubuntu you get cracking on fixing it | 18:36 |
| sdake_ | i would love to - all my gear is busted | 18:37 |
| Jeffrey4l | mandre, hmm. but no one report any log error. | 18:37 |
| Jeffrey4l | when i fixed this in master, i really saw some error message, especially for new added service. | 18:38 |
| mandre | Jeffrey4l: well, I do :) | 18:38 |
| mandre | I get permission errors when testing mitaka images in my environment, and was wondering what sets the group perms for /var/log/kolla | 18:39 |
| Jeffrey4l | mandre, no one ;( | 18:39 |
| *** aswadr_ has quit IRC | 18:39 | |
| *** mliima has quit IRC | 18:39 | |
| Jeffrey4l | it may be root i think. | 18:39 |
| *** mliima has joined #openstack-kolla | 18:40 | |
| mandre | Jeffrey4l: right... I'll keep digging | 18:40 |
| sdake_ | Jeffrey4l rather then fixing with a backport and forcing people to upgrade to 2.0.3 - wouldn't it be easier to just use the permissions feature of set_config.py? | 18:40 |
| Jeffrey4l | no need. | 18:41 |
| *** mliima_ has joined #openstack-kolla | 18:41 | |
| Jeffrey4l | if container image has /var/log/kolla with :kolla owner, when mounting a named volume, container will change the own on named volume. | 18:42 |
| sdake_ | what about the keepalived directory? | 18:42 |
| Jeffrey4l | keepalived is create in extend_start.sh file in keepalived folder. | 18:43 |
| Jeffrey4l | folder/container. | 18:43 |
| openstackgerrit | Andreas Jaeger proposed openstack/kolla: Move bandit to pep8 as generic linting target https://review.openstack.org/388175 | 18:43 |
| Jeffrey4l | the owner will be the same with USER in keepalived. | 18:43 |
| mandre | Jeffrey4l: I assume the chmod is being taken care of by the heka container in mitaka, https://github.com/openstack/kolla/blob/stable/mitaka/docker/heka/heka_sudoers#L2 | 18:43 |
| mandre | https://github.com/openstack/kolla/blob/stable/mitaka/docker/heka/extend_start.sh#L12 | 18:44 |
| *** mliima has quit IRC | 18:44 | |
| Jeffrey4l | mandre, but the reality is if i mount kolla_logs at /var/log/kolla ( which is root:root ) in container, it will be change. | 18:44 |
| mandre | Jeffrey4l: yeah, that's the issue I'm facing, I'm not using the heka container | 18:45 |
| Jeffrey4l | heka container doesn't solve such issue. | 18:45 |
| Jeffrey4l | if it does, i will won't push that fix. | 18:45 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Remove inputs.filestat from telegraf configuration https://review.openstack.org/387935 | 18:45 |
| Jeffrey4l | sdake_, my test env bust again. ;( run reconfigure to nova. | 18:46 |
| *** flaper87 has joined #openstack-kolla | 18:46 | |
| *** flaper87 has quit IRC | 18:46 | |
| *** flaper87 has joined #openstack-kolla | 18:46 | |
| sdake_ | what do you mean bust again | 18:47 |
| *** pbourke has quit IRC | 18:47 | |
| Jeffrey4l | i am running 3 vms for test actually. | 18:47 |
| *** pbourke has joined #openstack-kolla | 18:47 | |
| sdake_ | vms suck use my gear | 18:47 |
| Jeffrey4l | and now here is the uptime: 02:47:42 up 18:45, 1 user, load average: 59.81, 54.59, 42.26 | 18:48 |
| mandre | Jeffrey4l: what I mean is, in mitaka we expect the heka container to setup proper perms for /var/log/kolla directory | 18:48 |
| mandre | Jeffrey4l: this is a wrong assumption that you fixed with your patch to master 3 weeks ago | 18:48 |
| Jeffrey4l | sdake_, so i will destroy all container in all your three nodes. is it OK? | 18:49 |
| inc0 | wirehead_, sbezverk I'll add 2 or so slides at the end of our talk ok? | 18:49 |
| inc0 | "community update" | 18:49 |
| inc0 | and "call to arms" | 18:49 |
| sdake_ | Jeffrey4l perfectly fine | 18:49 |
| Jeffrey4l | mandre, anyway. heka do not solve your issue. | 18:50 |
| sdake_ | inc0 - upgrade on ubuntu is broken | 18:50 |
| sdake_ | i am tagging in 5 hours | 18:50 |
| sdake_ | whether it remains broken or not | 18:50 |
| inc0 | how fun | 18:50 |
| inc0 | what's wrong? | 18:50 |
| sdake_ | inc0 need you to test and debug it | 18:50 |
| mandre | Jeffrey4l: isn't it the lack of heka that is creating my issue? | 18:50 |
| *** HyperJohnGraham_ has quit IRC | 18:50 | |
| sdake_ | inc0 nboody else here has ubuntu test beds | 18:50 |
| Jeffrey4l | sdake_, only one disk? it is impossible to test ceph ;( | 18:50 |
| sdake_ | Jeffrey4l sda is for ceph | 18:51 |
| sdake_ | Jeffrey4l /dev/nvme* is where the os is - don't mess with that | 18:51 |
| Jeffrey4l | mandre, yes. | 18:51 |
| Jeffrey4l | sdake_, got. | 18:51 |
| inc0 | sdake_, what's wrong with upgrade? do you have any info? | 18:52 |
| Jeffrey4l | mandre, even you add heka. the containers will mess the /var/log/kolla permission up. | 18:52 |
| sdake_ | bjolo ls -alh /var/lib/docker/volumes/kolla_logs/_data/haproxy/` | 18:52 |
| sdake_ | bjolo__ ^^ | 18:52 |
| mandre | Jeffrey4l: hmm, so how come our mitaka branch is not all busted? | 18:53 |
| sdake_ | inc0 info i have is in scrollbck | 18:53 |
| sdake_ | heka craters with keepalived and haproxy | 18:53 |
| sdake_ | permission denied | 18:53 |
| Jeffrey4l | mandre, no idea ;( have test this carefully. | 18:53 |
| bjolo__ | http://paste.openstack.org/show/586260/ | 18:53 |
| Jeffrey4l | have/haven't | 18:53 |
| Jeffrey4l | ok. sdake_ you r correct. | 18:54 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Fencing pod https://review.openstack.org/383922 | 18:54 |
| sdake_ | Jeffrey4l haproxy.log = 102 = systemd stuff | 18:54 |
| Jeffrey4l | what 120k is? | 18:54 |
| sdake_ | notice . is 102 as well | 18:54 |
| Jeffrey4l | bjolo__, is haproxy running? | 18:54 |
| bjolo__ | it was | 18:54 |
| bjolo__ | but not anymore | 18:55 |
| bjolo__ | destroyed and deploying mitaka again | 18:55 |
| Jeffrey4l | check the logs in haproxy container ` docker exec haproxy ls -alh /var/log/kolla/haproxy` | 18:55 |
| sdake_ | we need json for haproxy and keepalived | 18:55 |
| bjolo__ | ubuntu source 2.0.2 | 18:55 |
| Jeffrey4l | ok.. | 18:55 |
| mandre | Jeffrey4l: gotcha, thanks for your help, I'll look into it tomorrow | 18:55 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Use apt instead of apt-get to install packages in Ubuntu https://review.openstack.org/379477 | 18:55 |
| bjolo__ | upgrade completed the second time with heat disabled | 18:55 |
| Jeffrey4l | sdake_, yes. i am wrong for haproxy. it is not run as root. it run as haproxy user. | 18:56 |
| sdake_ | wht about keepalived - same story? | 18:56 |
| * Jeffrey4l is checking. | 18:57 | |
| Jeffrey4l | seems no. | 18:57 |
| sdake_ | seems no means what? :) | 18:57 |
| Jeffrey4l | keepalived is not the same story. | 18:58 |
| sdake_ | vhosakot ping | 18:58 |
| vhosakot | sdake_: pong | 18:58 |
| sdake_ | vhosakot do you have time now to dig into debug of keepalived and haproxy on ubuntu | 18:59 |
| sdake_ | sincey ou run ubuntu in your test beds | 18:59 |
| *** jmccarthy has quit IRC | 18:59 | |
| openstackgerrit | Merged openstack/kolla: Adding kubetoolbox image https://review.openstack.org/384988 | 19:00 |
| sdake_ | vhosakot i am tagging 3.0.0 in 5 hours exactly | 19:00 |
| vhosakot | sdake_: sure, I can debug it... is there a review in queue I need to test | 19:00 |
| *** tonanhngo has joined #openstack-kolla | 19:00 | |
| sdake_ | vhosakot ignore all other reviews for now | 19:00 |
| sdake_ | this is critical critical priority | 19:00 |
| *** jmccarthy has joined #openstack-kolla | 19:01 | |
| Jeffrey4l | keepalived's log is collected by syslog and is wrote to filesystem by heka. | 19:01 |
| vhosakot | Jeffrey4l: ah ok | 19:01 |
| Jeffrey4l | so heka should change the keepalived folder's permission to heka:kolla | 19:01 |
| vhosakot | sdake_: Jeffrey4l what exactly should I do? debug the issue? or , is there a a patch to test? | 19:01 |
| vhosakot | Jeffrey4l: ah ok | 19:02 |
| sdake_ | vhosakot first deploy 2.0.2 mitaka from docker hub | 19:02 |
| sdake_ | then upgrade to stable/newton | 19:02 |
| sdake_ | to do that pip install kolla | 19:02 |
| sdake_ | kolla-ansible deploy | 19:02 |
| Jeffrey4l | i can push a patch. then please test with it. | 19:02 |
| jascott1 | docs question: why does this call for setting up /etc/kolla before installing and copying over /etc/kolla? | 19:03 |
| jascott1 | http://docs.openstack.org/developer/kolla-kubernetes/host-setup.html#generate-config-file | 19:03 |
| sdake_ | pip uninstall kolla | 19:03 |
| vhosakot | sdake_: should I test it in the next 5 hours before tagging 3.0.0? | 19:03 |
| sdake_ | git clone http://github.com/openstack/kolla | 19:03 |
| sdake_ | vhosakot this needs to be fixed before tagging | 19:03 |
| sdake_ | the altnerative is I add a big ugly release note saying in all caps "UPGRADING ON UBUNTU WILL CRATER YOUR SYSTEM." | 19:04 |
| sdake_ | and you can be guranteed it will be added | 19:04 |
| sdake_ | tools/kolla-ansible upgrade (from your git repo) | 19:04 |
| *** AJaeger has joined #openstack-kolla | 19:04 | |
| sdake_ | after building stable/newton images of course | 19:04 |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Add node_custom_config example to globals.yml https://review.openstack.org/388169 | 19:05 |
| vhosakot | Jeffrey4l: what is the current permission of keepalived logs (other than heka:kolla) ? | 19:05 |
| sdake_ | vhosakot get your env setup - let Jeffrey4l work the problem for now | 19:05 |
| vhosakot | sdake_: ok, I will start now... need to create the VM... get the env setup.. I cleaned up 4 days ago. | 19:05 |
| Jeffrey4l | should be a uid/gid number after upgrade. | 19:05 |
| sdake_ | vhosakot shouldn't tkae more then 15 mins to deploy mitaka | 19:05 |
| openstackgerrit | Andrew Widdersheim proposed openstack/kolla: Update globals.yml Docker registry example notes https://review.openstack.org/388167 | 19:05 |
| sdake_ | my laptop has 8gb ram | 19:06 |
| sdake_ | i can't do ubuntu vm testing on it | 19:06 |
| *** AJaeger has left #openstack-kolla | 19:06 | |
| Jeffrey4l | sdake_, fyi, haproxy and keepalived are the same story now. their logs are both collected by haka syslog socket | 19:07 |
| sdake_ | so just a json fix then? | 19:07 |
| Jeffrey4l | yes. | 19:07 |
| sdake_ | cool can you make it happen - perhaps bjolo__ can test | 19:07 |
| Jeffrey4l | working on it. | 19:07 |
| bjolo__ | will do | 19:08 |
| sdake_ | bjolo__ recomend getting mitaka 2.0.2 back on your system in a clean state | 19:08 |
| Jeffrey4l | i am curiosity why the group is kolla rather then heka. | 19:08 |
| vhosakot | bjolo__: you hvae ubuntu 14.04 or 16 (xenial)? | 19:08 |
| bjolo__ | in progress already | 19:08 |
| bjolo__ | latest | 19:08 |
| bjolo__ | whatever that means | 19:08 |
| bjolo__ | 1404 mitaka 1604 newton | 19:08 |
| bjolo__ | host os is centos 7 | 19:08 |
| vhosakot | bjolo__: cool.. I have same.. 14.04 for mitaka and 16.04 newton.. cool | 19:09 |
| vhosakot | I'll ping once my env is ready | 19:09 |
| bjolo__ | need to ride home, back online in 1h | 19:09 |
| bjolo__ | mitaka deploy is running in the meantime | 19:10 |
| sdake_ | inc0 need your help here dude, your the only one with a working ubuntu system | 19:10 |
| sdake_ | we really dont want that big all caps release note | 19:11 |
| sdake_ | but we absolutely have to tag today | 19:11 |
| sdake_ | inc0 you have 5 days to work on slides | 19:11 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix wrong log permission for services collected through syslog https://review.openstack.org/388182 | 19:12 |
| piaccarino | Having keystone-fernet deploy issue with master - http://pastebin.com/S3UbEheQ | 19:12 |
| Jeffrey4l | sdake_, bjolo__ vhosakot here is the fix https://review.openstack.org/388182 | 19:13 |
| vhosakot | Jeffrey4l: ah ok, the needs the right recursive permission.. cool | 19:13 |
| sdake_ | Jeffrey4l what is the swift part about | 19:13 |
| vhosakot | I'll keep you guys posted... starting now | 19:13 |
| piaccarino | rhel7/mitaka in 3.0.0 containers | 19:13 |
| Jeffrey4l | sdake_, the same story with haproxy and keepalived. | 19:14 |
| sdake_ | piaccarino how do you ahve mitaka in 3.0.0 containers | 19:14 |
| sdake_ | master is msater of openstack | 19:14 |
| Jeffrey4l | it is collected through syslog, and wrote through heka. so should be ownned by heka. | 19:14 |
| sdake_ | piaccarino if you want to use something ,try out stable/newton | 19:14 |
| sdake_ | Jeffrey4l ack | 19:15 |
| sdake_ | vhosakot how long does it take you to get your env setup | 19:16 |
| sdake_ | it takes 2 hours for the patches to merge and be backported | 19:16 |
| vhosakot | sdake_: I am not creating the ubuntu 14 VM, then, need to deploy mitaka from scracth | 19:16 |
| sdake_ | so really we are working against a 3pm deadline | 19:16 |
| vhosakot | sdake_: it is 3:16 here in east coast :) | 19:17 |
| sdake_ | vhosakot note the instructions are changed | 19:17 |
| sdake_ | vhosakot 2200UTC | 19:17 |
| sdake_ | vhosakot since possibly the last time you deployed kolla | 19:17 |
| vhosakot | sdake_: yeah yeah.. I'm referring the least QSG | 19:17 |
| vhosakot | s/least/latest | 19:17 |
| inc0 | sbezverk, qq - we deploy HA on L3 agents right? | 19:18 |
| inc0 | in kolla-k8s? | 19:18 |
| inc0 | vrrp works well? | 19:18 |
| *** daneyon_ has joined #openstack-kolla | 19:20 | |
| *** goldyfruit has quit IRC | 19:23 | |
| kfox1111 | inc0: doing vvrp for the control ip. but haven't played much with nuetorn ha yet. | 19:23 |
| kfox1111 | inc0: ideally, we'd be enabling l3ha+dvr in neutron. | 19:24 |
| kfox1111 | I'm going to attempt it at some point, but haven't gotten there yet. | 19:24 |
| kfox1111 | we only use dvr currently on one of our production clouds. | 19:24 |
| *** daneyon has quit IRC | 19:24 | |
| kfox1111 | and use pacemaker to fail over the nn. | 19:24 |
| kfox1111 | looking at some of the summit stuff, the l3ha stuff shoudl fail over much quicker though, so we want to enable that too. | 19:25 |
| inc0 | kk | 19:25 |
| kfox1111 | at the time though, we only had the option of one or the other, and scalability was the more important of the two. | 19:25 |
| kfox1111 | so we went with dvr. | 19:25 |
| piaccarino | sdake_ Have a need for the node_custom_config and didn't want to fork so we worked rhel7/osp9 into 3.0.0 with a little help from some rdo packages. 3.0.0.0rc1 and forward have the same fernet issue. Going to try an all-in-one centos/newton to see if it is just me. | 19:28 |
| kfox1111 | sdake_: inc0 hows the patch queue looking? how close to release are we? | 19:28 |
| inc0 | 4.5 hrs | 19:29 |
| kfox1111 | k. | 19:29 |
| kfox1111 | I' just kicked off another gate run. | 19:29 |
| kfox1111 | last one passed ok. | 19:29 |
| sbezverk | inc0: if you refer to keepalived then it runs on controllers | 19:29 |
| sbezverk | other than that I am not aware of any HA | 19:30 |
| inc0 | well, neutron has it's own vrrp | 19:30 |
| inc0 | l3 agents | 19:30 |
| kfox1111 | inc0: yeah, when l3ha is configured. | 19:30 |
| kfox1111 | we're not configuring it. | 19:30 |
| inc0 | btw I also heard that shared /run/netns has issues with k8s..any ideas about that? | 19:30 |
| sdake_ | kfox1111 kolla-kubernetes has already been tagged | 19:30 |
| sbezverk | still waiting for fix | 19:30 |
| inc0 | and we don't have dvr yet right? | 19:31 |
| kfox1111 | inc0: yeah, I think that still is an issue. | 19:31 |
| sdake_ | inc0 we fixed that | 19:31 |
| kfox1111 | inc0: I've committed dvr support, but its not currently tested. | 19:31 |
| Jeffrey4l | use /run:/run:shared for all container will fix this issue. | 19:31 |
| kfox1111 | sdake_: ok. just testing kolla-kubernetes against kolla trunk. just wana make sure the release works. | 19:31 |
| sbezverk | kuberentes still do not support it, we are tracking bug on kube side | 19:32 |
| sbezverk | but it is going really really slow | 19:32 |
| rhallisey | kfox1111, ya it's already been tagged | 19:32 |
| kfox1111 | sbezverk: I'm needing it for cvmfs for another project too. if it doesn't land soon, I'm going to have to implement it myself anyway. | 19:32 |
| sbezverk | kfox1111: let's learn go ;-) | 19:33 |
| kfox1111 | rhallisey: ok. I think its kind of doa anyway, so doesn't really matter to me if we tag it or not. | 19:33 |
| sbezverk | unless you already know it | 19:33 |
| kfox1111 | sbezverk: been using go since 0.9 :) | 19:33 |
| kfox1111 | or should I say, :( | 19:33 |
| rhallisey | ya it doesn't matter | 19:33 |
| kfox1111 | its better now... its wasn't so good then. | 19:33 |
| sbezverk | kfox1111: cannot comment on go, have not used it, but I really like the quality of kube code so far.. | 19:34 |
| kfox1111 | sbezverk: yeah. go's pretty nice in a lot of ways. | 19:35 |
| kfox1111 | kind of an odd mix between c and python. | 19:35 |
| kfox1111 | but less of an identiy crisis then c++. | 19:35 |
| sbezverk | kfox1111: when you have time please check fencing pod latest version. I think it does now what we have discussed recently.. | 19:35 |
| kfox1111 | k | 19:36 |
| *** awebber has quit IRC | 19:36 | |
| *** mliima__ has joined #openstack-kolla | 19:36 | |
| *** mliima has joined #openstack-kolla | 19:39 | |
| *** goldyfruit has joined #openstack-kolla | 19:40 | |
| *** mliima_ has quit IRC | 19:40 | |
| *** mliima has quit IRC | 19:41 | |
| *** mliima__ has quit IRC | 19:41 | |
| sdake_ | bjolo__ inc0 vhosakot this is the reproducer i used: https://bugs.launchpad.net/kolla/+bug/1634401 | 19:44 |
| openstack | Launchpad bug 1634401 in kolla newton "upgrade from 2.0.2 to 3.0.0 with VM running on BTRFS STILL BROKEN" [Critical,Fix committed] - Assigned to Jeffrey Zhang (jeffrey4l) | 19:44 |
| sdake_ | that bug hs been fixed for centos | 19:44 |
| sdake_ | however the reproduction steps should produce the problem jeffrey4l reported | 19:45 |
| vhosakot | sdake_: ah ok, I'll read the steps | 19:45 |
| sdake_ | rather bjolo__ reported and Jeffrey4l fixed | 19:45 |
| sdake_ | we are oging to have more problems then just thi sone | 19:45 |
| sdake_ | need devs focused on this | 19:45 |
| sdake_ | or we cut ubuntu from the upgrade process | 19:46 |
| inc0 | or we cut btrfs from it | 19:46 |
| inc0 | I'm trying upgrade with overlay | 19:46 |
| sdake_ | inc0 - that PROBLEM HAS BEEN FIXED | 19:46 |
| sdake_ | dude, did you not read the scrollback | 19:47 |
| sdake_ | inc0 - bjoolo reported perm denied in heka container | 19:47 |
| *** haplo37 has quit IRC | 19:47 | |
| sdake_ | that bug however contains step by step instructions to upgrade | 19:47 |
| inc0 | ok, I have mitaka deployed now I need to get newton up, hold on | 19:47 |
| sdake_ | make sure to pull fresh stable/newton nad rebuild | 19:47 |
| sdake_ | over 30 patches merged last night | 19:47 |
| *** g3ek has quit IRC | 19:47 | |
| *** dwalsh has quit IRC | 19:47 | |
| inc0 | I reset every day | 19:48 |
| Jeffrey4l | + my patch ;) | 19:48 |
| Jeffrey4l | https://review.openstack.org/388182 | 19:48 |
| sdake_ | ya - run jeffrey's patch on top | 19:48 |
| *** daneyon has joined #openstack-kolla | 19:48 | |
| kfox1111 | sdake_: inc0: most recent test still passing. | 19:48 |
| inc0 | unfortunately my dev env is misbehaving lately | 19:48 |
| Jeffrey4l | run this on newton branch git fetch https://git.openstack.org/openstack/kolla refs/changes/82/388182/1 && git cherry-pick FETCH_HEAD | 19:49 |
| sdake_ | A I O | 19:49 |
| *** daneyon_ has quit IRC | 19:50 | |
| sdake_ | kfox1111 that is a good sign and works becausae you dont' use heka | 19:51 |
| * kfox1111 nods | 19:51 | |
| *** gfidente|afk is now known as gfidente | 19:52 | |
| sdake_ | Jeffrey4l aery ou trying deploy all on my gear? | 19:52 |
| Jeffrey4l | sdake_, yep. | 19:52 |
| sdake_ | cool | 19:52 |
| sdake_ | when you get done let me know | 19:52 |
| sdake_ | i am going to wipe one of the boxes and attempt to install ubuntu on it | 19:53 |
| Jeffrey4l | roger. need reboot node3. | 19:53 |
| Jeffrey4l | is it OK? | 19:53 |
| sdake_ | even though that is a huge pain for me | 19:53 |
| *** g3ek has joined #openstack-kolla | 19:53 | |
| sdake_ | reboot away | 19:53 |
| inc0 | so you don't need to install ubuntu | 19:53 |
| *** haplo37 has joined #openstack-kolla | 19:53 | |
| inc0 | you shouldn't | 19:53 |
| inc0 | but you can | 19:53 |
| sdake_ | inc0 are ou able to repair your dev env | 19:55 |
| inc0 | testing on it as it is | 19:55 |
| inc0 | it's fleaky but should work | 19:56 |
| inc0 | at least for issue you mention | 19:56 |
| *** awebber has joined #openstack-kolla | 19:56 | |
| inc0 | my vms are just super slow | 19:56 |
| inc0 | for some reason | 19:56 |
| inc0 | heka upgraded | 19:57 |
| sdake_ | `did you follow the procedure in https://bugs.launchpad.net/kolla/+bug/1634401 | 19:58 |
| openstack | Launchpad bug 1634401 in kolla newton "upgrade from 2.0.2 to 3.0.0 with VM running on BTRFS STILL BROKEN" [Critical,Fix committed] - Assigned to Jeffrey Zhang (jeffrey4l) | 19:58 |
| inc0 | no because I don't run btrfs, and you mentioned different issue | 19:59 |
| inc0 | also you linked it when I was well into testing | 19:59 |
| inc0 | so I'll rerun it later | 19:59 |
| inc0 | but for now upgrade is super slow for some reason | 19:59 |
| *** portdirect has quit IRC | 20:00 | |
| inc0 | Jeffrey4l, serial seems to be slowing things down a lot isn't i? | 20:00 |
| inc0 | I mean more than it should | 20:00 |
| Jeffrey4l | yes. | 20:00 |
| Jeffrey4l | it should so slowing. | 20:01 |
| Jeffrey4l | when you have 3 nodes. serial will run task one node by another. | 20:02 |
| jascott1 | When I try to run ./tools/setup-resolv-conf.sh I get an error about the service account not being created. Grepd in both repos but cant find where that is created. Anyone know where that is? | 20:02 |
| jascott1 | I think I must have missed a step | 20:04 |
| *** portdirect has joined #openstack-kolla | 20:05 | |
| inc0 | nope it's vm issue | 20:06 |
| *** dwalsh has joined #openstack-kolla | 20:07 | |
| *** dave-mccowan has quit IRC | 20:09 | |
| *** ssurana has quit IRC | 20:10 | |
| bjolo__ | back online | 20:13 |
| *** confisurya has joined #openstack-kolla | 20:14 | |
| bjolo__ | hmm mitaka heat deploy failed again. but never mind that today | 20:15 |
| *** ppalacios has quit IRC | 20:17 | |
| *** confisurya has quit IRC | 20:19 | |
| bjolo__ | Jeffrey4l, just so i got it correct. you want me to cherrypick this ps https://review.openstack.org/388182 | 20:26 |
| hogepodge | ok, so I have my installation up and running, almost everything is happy | 20:26 |
| Jeffrey4l | yes. | 20:26 |
| Jeffrey4l | run this on newton branch git fetch https://git.openstack.org/openstack/kolla refs/changes/82/388182/1 && git cherry-pick FETCH_HEAD | 20:26 |
| Jeffrey4l | then run upgrade. | 20:26 |
| bjolo__ | gotcha. no time for missunderstandings :) | 20:26 |
| hogepodge | I have one service that is misconfigured by default, as simplified example say [Default] service_id=name is set in the conf file | 20:27 |
| hogepodge | Using the merge conf method, I know I can add a configuration, say [Default] service_name=name | 20:27 |
| hogepodge | but how can I unset the bad variable? so service_id isn't part of the configuration any more? | 20:27 |
| Jeffrey4l | service_id = | 20:28 |
| hogepodge | that's easy. thanks :-D | 20:28 |
| hogepodge | fwiw, it's a bad configuration with Magnum | 20:28 |
| hogepodge | https://ask.openstack.org/en/question/94509/magnum-commands-return-error-not-authorized/ | 20:29 |
| *** jheroux has joined #openstack-kolla | 20:32 | |
| *** eaguilar has quit IRC | 20:46 | |
| *** jheroux has quit IRC | 20:51 | |
| *** ayoung has joined #openstack-kolla | 20:51 | |
| *** eaguilar has joined #openstack-kolla | 20:53 | |
| *** msimonin has joined #openstack-kolla | 20:58 | |
| inc0 | do we look at ubuntu not possible to build issue? | 20:58 |
| bjolo__ | Jeffrey4l, sdake_ seem like we have a winner. upgrade works for ubuntu :) | 20:58 |
| Jeffrey4l | woow bjolo__ | 20:58 |
| Pavo | evening gents | 20:59 |
| inc0 | Jeffrey4l, reviewed | 20:59 |
| Jeffrey4l | thanks. | 21:00 |
| bjolo__ | sdake_, inc0 Jeffrey4l anything else you want me to test out? | 21:00 |
| Jeffrey4l | need another cr https://review.openstack.org/388182 | 21:00 |
| inc0 | sdake_, ^ | 21:00 |
| Jeffrey4l | bjolo__, enable all kolla supported service on newton in ubuntu | 21:00 |
| bjolo__ | ok | 21:01 |
| Jeffrey4l | thanks. | 21:01 |
| Pavo | question | 21:01 |
| bjolo__ | we know that lbaas vpnaas wont work for source | 21:01 |
| Jeffrey4l | yes. disable them. | 21:01 |
| Pavo | how does provider networks with vlans work with kolla network design? | 21:01 |
| *** eaguilar has quit IRC | 21:02 | |
| inc0 | Pavo, so you'll need to write config override for neutron | 21:02 |
| Pavo | to include vlans correct | 21:02 |
| Pavo | in nova.conf | 21:03 |
| inc0 | neutrons api-paste.ini if my memory serves | 21:03 |
| inc0 | you need to specify provider | 21:03 |
| *** vhosakot has quit IRC | 21:06 | |
| *** awiddersheim has quit IRC | 21:07 | |
| sdake_ | hey cats sorry important meeting | 21:07 |
| sdake_ | so i see bjolo__ has victory | 21:07 |
| Jeffrey4l | sdake_, i enabled almost all service on node3 and all container works except for congress. | 21:08 |
| Pavo | currently with packstack we use CONFIG_NEUTRON_ML2_TENANT_NETWORK_TYPES=vlan | 21:08 |
| sdake_ | Jeffrey4l - do we need some backports? | 21:08 |
| sdake_ | Jeffrey4l ok - what part of congress doesn't work | 21:08 |
| Jeffrey4l | i am running reconfiguration now. | 21:08 |
| Jeffrey4l | all congress containers. | 21:08 |
| Jeffrey4l | seem wrong configuration. | 21:08 |
| sdake_ | ok - release note | 21:08 |
| Pavo | and we use CONFIG_NEUTRON_ML2_TYPE_DRIVERS=vxlan, local, flat, vlan | 21:09 |
| Jeffrey4l | re backport, of curse. | 21:09 |
| Pavo | so wouldn't those settings be in the ml2.ini | 21:09 |
| sdake_ | Jeffrey4l your patch that got bjolo working - is it merged? | 21:09 |
| Jeffrey4l | sdake_, https://review.openstack.org/388182 need +2 | 21:09 |
| Jeffrey4l | +w | 21:09 |
| *** jtriley has quit IRC | 21:10 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix wrong log permission for services collected through syslog https://review.openstack.org/388217 | 21:11 |
| dmsimard | Oh wow, who implemented ARA in Kolla? http://logs.openstack.org/82/388182/1/check/gate-kolla-dsvm-deploy-centos-binary-centos-7-nv/4f7a926/logs/playbook_reports/ | 21:11 |
| bjolo__ | Jeffrey4l, http://paste.openstack.org/show/586278/ | 21:11 |
| Jeffrey4l | cool. | 21:12 |
| Pavo | where in the docs can I find how to customize certain settings for the services? | 21:12 |
| Jeffrey4l | i didn't enable neutron_qos and agent_ha and dvr. | 21:12 |
| inc0 | Pavo, http://docs.openstack.org/developer/kolla/advanced-configuration.html#openstack-service-configuration-in-kolla | 21:12 |
| sbezverk | jascott1: service account get created when you install kube, afaik kolla-kubernetes is not creating it.. | 21:13 |
| Pavo | thanks inc0 | 21:13 |
| jascott1 | sbezverk thanks I will check it out | 21:14 |
| dmsimard | Jeffrey4l: Hey, thanks for https://review.openstack.org/#/c/368345/ ! Please let me know if you have any questions about ARA. | 21:15 |
| *** LamT__ has quit IRC | 21:15 | |
| Jeffrey4l | dmsimard, thanks for the cool tool ;) | 21:16 |
| *** LamT__ has joined #openstack-kolla | 21:16 | |
| *** wirehead_ has quit IRC | 21:16 | |
| *** dwalsh has quit IRC | 21:16 | |
| dmsimard | Jeffrey4l: I'm planning to attend the kolla CI work session, hopefully we can chat there :) | 21:17 |
| Jeffrey4l | nice. | 21:17 |
| sdake_ | Jeffrey4l did you try reconfigure on all services? | 21:17 |
| Jeffrey4l | sdake_, it is running | 21:17 |
| Pavo | ok so found the settings that needed to be customizes | 21:17 |
| *** bachp has quit IRC | 21:17 | |
| openstackgerrit | Merged openstack/kolla: Fix wrong log permission for services collected through syslog https://review.openstack.org/388182 | 21:18 |
| Pavo | its /etc/neutron/plugins/ml2/ml2_conf.ini | 21:18 |
| *** hyakuhei has quit IRC | 21:18 | |
| Jeffrey4l | btw, could ara can log the variable passed though -e parameter? it is hard to distinguish which playbook it is now. | 21:19 |
| Jeffrey4l | http://logs.openstack.org/82/388182/1/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/a4714de/logs/playbook_reports/ | 21:19 |
| Jeffrey4l | dmsimard, ^^ | 21:19 |
| *** hyakuhei has joined #openstack-kolla | 21:19 | |
| *** hyakuhei has quit IRC | 21:19 | |
| *** hyakuhei has joined #openstack-kolla | 21:19 | |
| *** hyakuhei has quit IRC | 21:19 | |
| *** hyakuhei has joined #openstack-kolla | 21:19 | |
| dmsimard | Jeffrey4l: -e, like --extra-vars you mean ? | 21:20 |
| *** sdake has joined #openstack-kolla | 21:20 | |
| Pavo | so if I put a custom /etc/neutron/plugins/ml2/ml2_conf.ini inside /etc/kolla/config/neutron/plugins/ml2 and rebuild neutron images it should use that custom config correct? | 21:20 |
| Jeffrey4l | yes. | 21:20 |
| Jeffrey4l | kolla is using -e for different jobs. | 21:20 |
| *** wirehead_ has joined #openstack-kolla | 21:20 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Fix wrong log permission for services collected through syslog https://review.openstack.org/388217 | 21:20 |
| Jeffrey4l | like -e action=pull and -e action=deploy | 21:20 |
| dmsimard | Jeffrey4l: I'd like to say yes but I need to double check if ara has access to that. | 21:20 |
| *** bachp has joined #openstack-kolla | 21:21 | |
| Pavo | inc0? | 21:21 |
| dmsimard | I'll go ahead and open up an issue about it so I don't forget. | 21:21 |
| inc0 | Pavo, no need to rebuild | 21:21 |
| Jeffrey4l | cool. thanks. | 21:21 |
| inc0 | deploy or reconfigure will work | 21:21 |
| Pavo | oh | 21:21 |
| Pavo | nice | 21:22 |
| *** sdake_ has quit IRC | 21:22 | |
| Pavo | so I take it that the docker container for neutron uses /etc/kolla/config/<< service name >>/ for persistent storage | 21:22 |
| inc0 | it's more complicated than that | 21:23 |
| *** MarMat has quit IRC | 21:23 | |
| inc0 | basicaally what happends is that container bind a directory from host | 21:23 |
| inc0 | default being /etc/kolla | 21:23 |
| Pavo | roger | 21:23 |
| dmsimard | Jeffrey4l: I think we don't do much around variables in general right now (extra vars, or from defaults) so that'd be an area for improvement. Extra vars are sort of easier because they're at the top of the variable hierarchy and are "immutable" but other variables are sort of dynamically compiled as they run (and can be different for each host). It'll require some thinking to do this right. | 21:24 |
| *** MarMat has joined #openstack-kolla | 21:24 | |
| inc0 | in this directory on host configs sits | 21:24 |
| inc0 | that kolla generates | 21:24 |
| dmsimard | Jeffrey4l: good feedback, thanks :) | 21:24 |
| inc0 | then when containers start it will look into it and copy contents to /etc/neutron | 21:24 |
| Pavo | on the host that is set in globals for that role right | 21:24 |
| inc0 | well globals, overrides, template | 21:24 |
| inc0 | pretty much all the stuff gets into config generation | 21:24 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Congress doesn't deploy release note https://review.openstack.org/388222 | 21:25 |
| sdake | Jeffrey4l inc0 ^^ | 21:25 |
| Pavo | I mean if I have set network role on computenode1 host it will use /etx/kolla/config directory on that host to put the service configs | 21:25 |
| sdake | note the reason i didn't use an url is i don't know how it will render | 21:26 |
| inc0 | sdake, do we really need to put this into release note? | 21:26 |
| sdake | unclear | 21:26 |
| sdake | it is only a high severity bug | 21:26 |
| inc0 | this is exactly why we need support matrix | 21:27 |
| inc0 | well if you want it there, it's going in. I'll ack it | 21:27 |
| sdake | we need lots of things | 21:27 |
| sdake | i dont know tbh | 21:27 |
| sdake | if it was critical then yes | 21:27 |
| sdake | we probably ahve a few services busted | 21:28 |
| sdake | i did release note that performance monitoring stack is subject tochange without notice ;) | 21:28 |
| sdake | lets wit until Jeffrey4l either passes out at his keyboard or gives us the a-ok on reconfigure | 21:29 |
| sdake | and then i'll tag it | 21:29 |
| sdake | looks like release note was acked | 21:29 |
| Jeffrey4l | fyi, bjolo__ is testing all services on ubuntu | 21:29 |
| sdake | so i guess we are going with it :) | 21:30 |
| *** neilus has quit IRC | 21:30 | |
| sdake | Jeffrey4l define testing all services | 21:30 |
| sdake | you mean testing that they deploy? | 21:30 |
| bjolo__ | doing the ubuntu source build atm | 21:30 |
| bjolo__ | sdake, yes | 21:31 |
| inc0 | anyone seen this? http://paste.openstack.org/show/586282/ | 21:31 |
| *** neilus has joined #openstack-kolla | 21:31 | |
| inc0 | my compute nodes behaves like that oO | 21:31 |
| sdake | inc0 i have seen that on my ucs gear that sbezverk is running in kubneretes | 21:31 |
| *** neilus has quit IRC | 21:32 | |
| sdake | inc0 run dmesg | 21:32 |
| sdake | it will give out he kerne loops | 21:32 |
| *** neilus has joined #openstack-kolla | 21:32 | |
| inc0 | I cant get to vm | 21:32 |
| sdake | your vm has 24 processors? | 21:32 |
| sdake | oh i guess thats 8 | 21:33 |
| sdake | been up far too long | 21:33 |
| Pavo | ok cool think I found all the files I need to customize | 21:33 |
| inc0 | host its on is about that | 21:33 |
| inc0 | fuck I need more predictible testbed | 21:34 |
| sdake | run centos? :-) | 21:34 |
| Jeffrey4l | sdake, wait. shouldn't the releasenote be merged into newton branch? | 21:34 |
| Jeffrey4l | do what i did. but on ubuntu. | 21:34 |
| sdake | Jeffrey4l yes it should | 21:34 |
| Pavo | btw sdake might see ya at cloudnativecon, asked today and was told its a good idea for us to have a few people there | 21:34 |
| *** daneyon has quit IRC | 21:34 | |
| sdake | nice | 21:34 |
| *** Jeffrey4l has quit IRC | 21:34 | |
| *** haplo37_ has quit IRC | 21:34 | |
| *** Jeffrey4l has joined #openstack-kolla | 21:35 | |
| Pavo | inc0 what days is that Atlanta conference, can you PM me all the details? | 21:35 |
| *** neilus has quit IRC | 21:35 | |
| Pavo | wanna have a few people come to that also | 21:35 |
| Jeffrey4l | route is rebooted at 5:30 am every day. what i miss? | 21:35 |
| Pavo | everything!!!! Jeffrey4l | 21:35 |
| Pavo | :P | 21:35 |
| inc0 | Pavo, The first Project Teams Gathering (PTG) will be held in Atlanta, Feb 20-24, 2017. | 21:36 |
| sdake | Jeffrey4l wiating for release note to hit the repo | 21:36 |
| sdake | and your last patch | 21:36 |
| sdake | and an ack from you and bjolo | 21:36 |
| Pavo | inc0 thanks | 21:36 |
| Jeffrey4l | sdake, wait. shouldn't the releasenote be merged into newton branch? | 21:36 |
| sdake | Jeffrey4l once release note hits the repo - it needs backport | 21:37 |
| Pavo | and this is specifically Kolla PTG correct? | 21:37 |
| sdake | pavo its all ptgs | 21:37 |
| sdake | but kolla willbe there | 21:37 |
| inc0 | it's all of openstack | 21:37 |
| Pavo | ah ok | 21:37 |
| *** haplo37_ has joined #openstack-kolla | 21:37 | |
| sdake | some big tent projects | 21:37 |
| sdake | not all | 21:37 |
| inc0 | well ok | 21:37 |
| Jeffrey4l | but is it will be remove in O if congress is OK? | 21:37 |
| sdake | Jeffrey4l we cn backport congress bug fixes | 21:37 |
| Jeffrey4l | reconfigure is done with success. wow | 21:38 |
| sdake | take awhile? | 21:38 |
| sdake | or surprised it worked :) | 21:38 |
| Jeffrey4l | yes. it works ;) | 21:39 |
| Jeffrey4l | i also made some change ( change openstack_logging_debug from False to True ) | 21:39 |
| sdake | ok ship it | 21:39 |
| sdake | idon tthin kwe want to change that default | 21:40 |
| Jeffrey4l | sdake, this is not merged https://review.openstack.org/388217 | 21:40 |
| sdake | Jeffrey4l i know - wiating on gates to merge them into repo | 21:40 |
| sdake | then will cherrypick | 21:41 |
| sdake | then another wait | 21:41 |
| sdake | then tag | 21:41 |
| Jeffrey4l | this is my patch for heka fix. | 21:41 |
| Jeffrey4l | it is cherry pick already. | 21:41 |
| sdake | oh cool | 21:41 |
| sdake | Jeffrey4l i've been up about 20 hours longer then you :) | 21:41 |
| sdake | i'm barely functional atm | 21:42 |
| Jeffrey4l | really no... i have been up around 22 hours ;) | 21:42 |
| sdake | i've been up since monday 5am | 21:42 |
| sdake | although i think i nodded off for a couple hours last night | 21:43 |
| Jeffrey4l | lol | 21:43 |
| sdake | death at the keyboard | 21:43 |
| Jeffrey4l | need get some sleep. night sdake | 21:45 |
| sdake | thanks Jeffrey4l | 21:45 |
| sdake | you rock as usual ;) | 21:45 |
| sdake | probably take some time for a couple days to recover before summit ;-) | 21:45 |
| Jeffrey4l | probably not. still have company stuff ;( | 21:46 |
| sdake | ok well 3.0.0 has wings | 21:46 |
| Jeffrey4l | anyway. 3.0.0 is released finally, stable and full of new feature ; ) | 21:47 |
| sdake | not released | 21:47 |
| sdake | we release rc3 today | 21:47 |
| sdake | then in a coupl edays the release team tags 3.0.0 from rc3 | 21:47 |
| *** portdirect has quit IRC | 21:48 | |
| Jeffrey4l | i know. but rc3 and 3.0 are almost the same ;) | 21:48 |
| sdake | should be identical | 21:48 |
| Jeffrey4l | yes. | 21:48 |
| inc0 | one more bug to fix tho | 21:49 |
| sdake | which one | 21:49 |
| inc0 | not being able to build ubuntu is bad | 21:49 |
| sdake | who can't build ubuntu | 21:49 |
| Jeffrey4l | inc0, logs?> | 21:49 |
| sdake | the gate builds it all the time | 21:49 |
| Pavo | meah ubuntu sucks anyways | 21:49 |
| Pavo | :P | 21:49 |
| sdake | master may not build | 21:49 |
| sdake | but stable/newton does | 21:49 |
| Jeffrey4l | master is OK too, now. | 21:49 |
| sdake | pavo++ | 21:49 |
| inc0 | hmm...mariadb failed to build | 21:50 |
| Pavo | lol | 21:50 |
| Jeffrey4l | inc0, log log | 21:50 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH https://review.openstack.org/326307 | 21:50 |
| bjolo__ | inc0, i had the same issue earlier today | 21:50 |
| sdake | bjolo__ how did you fix that btw | 21:50 |
| *** portdirect has joined #openstack-kolla | 21:50 | |
| Jeffrey4l | go to bed. bye guys ;) | 21:51 |
| Pavo | night Jeffrey4l | 21:52 |
| inc0 | http://paste.openstack.org/show/586285/ | 21:52 |
| inc0 | nite Jeffrey4l | 21:52 |
| sdake | INFO:kolla.image.build.mariadb:E: There were unauthenticated packages and -y was used without --allow-unauthenticated | 21:53 |
| bjolo__ | sdake, added --no-install-recommends to kolla/template/methods.py | 21:53 |
| bjolo__ | ah | 21:53 |
| bjolo__ | err | 21:54 |
| inc0 | yeah but thats really bad for security | 21:54 |
| bjolo__ | yes | 21:54 |
| bjolo__ | only as a workaround | 21:54 |
| bjolo__ | added the allow-unsigned-packages | 21:54 |
| inc0 | and gates work because we have mirrors | 21:55 |
| bjolo__ | or whatever that flag was | 21:55 |
| bjolo__ | its gone now | 21:55 |
| inc0 | I guess we need to wait till percona solves it | 21:55 |
| inc0 | yay for repos | 21:55 |
| sdake | can we special case percona | 21:55 |
| inc0 | I don't think so | 21:55 |
| sdake | install it first via run cmd? | 21:56 |
| inc0 | I don't want to even | 21:56 |
| sdake | and --allow-unahtenticated? | 21:56 |
| inc0 | unless you want to put security deep into... | 21:56 |
| inc0 | it's invitation for spoofing the packages | 21:56 |
| sdake | we are already pulling from percona.com | 21:56 |
| inc0 | for what we know someone could hack into pecrona.com | 21:56 |
| inc0 | and setup package with rootkit in it | 21:57 |
| sdake | right | 21:57 |
| sdake | i think correct answer here is to trust their packages as is | 21:57 |
| sdake | even though they may not be signed (is that what the problem is?) | 21:57 |
| inc0 | it apeared today | 21:57 |
| sdake | and fix in a z stream | 21:57 |
| inc0 | I hope they'll fix it soon | 21:57 |
| openstackgerrit | Merged openstack/kolla: Fix wrong log permission for services collected through syslog https://review.openstack.org/388217 | 21:58 |
| sdake | i think releasing something that is ubuntu doa is not good | 21:58 |
| sdake | worse then a hack on percona and spoofed unsigned packages | 21:59 |
| sdake | are we sure the packages are unsigned | 21:59 |
| sdake | maye they just changed their sign key | 21:59 |
| dmsimard | What do you guys install straight from percona repos ? | 22:00 |
| dmsimard | Do you use xtradb or something ? | 22:00 |
| dmsimard | The mariadb provided by Ubuntu or RDO is not sufficient ? | 22:01 |
| sdake | yes xtradb | 22:01 |
| sdake | whatever the hell that does | 22:01 |
| dmsimard | xtradb is the clustering bit iirc, sort of like galera | 22:01 |
| sdake | someone told me once | 22:01 |
| sdake | and i forgot | 22:01 |
| sdake | we already have galera | 22:01 |
| dmsimard | xtradb sounds like an odd, very opinionated choice :) | 22:02 |
| sdake | we are not removing percona at last minute | 22:02 |
| dmsimard | yeah, just saying. | 22:02 |
| sdake | although i want to remove it | 22:02 |
| sdake | i've wanted to for a year | 22:02 |
| sdake | but I dont know how | 22:02 |
| openstackgerrit | Merged openstack/kolla: Congress doesn't deploy release note https://review.openstack.org/388222 | 22:03 |
| sdake | http://www.openstack.org/analytics | 22:04 |
| sdake | http://www.openstack.org/analytics | 22:04 |
| sdake | http://www.openstack.org/analytics | 22:04 |
| sdake | woops | 22:04 |
| sdake | check that link out | 22:04 |
| jascott1 | sbezverk im getting this error trying to run tools/setup-resolv-conf.sh | 22:06 |
| jascott1 | service account kolla/default was not found, retry after the service account is created | 22:06 |
| *** bmace has joined #openstack-kolla | 22:10 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Congress doesn't deploy release note https://review.openstack.org/388237 | 22:11 |
| sdake | Jeffrey4l can you ack https://review.openstack.org/#/c/388237/ | 22:12 |
| sdake | inc0 can you ack https://review.openstack.org/#/c/388237/ | 22:12 |
| sdake | anyone else around such as rhallisey? | 22:12 |
| Pavo | how do I ack? | 22:13 |
| sdake | pavo you can't | 22:14 |
| Pavo | opps | 22:15 |
| Pavo | :) | 22:15 |
| Pavo | mybad I did a +1 | 22:16 |
| sdake | oh you can do that if you like | 22:16 |
| sdake | in0 - so whats the plan on ubuntu | 22:17 |
| sdake | inc0 | 22:17 |
| sdake | no plan? | 22:17 |
| sdake | inc0 maybe the keys in the bsae file are wrong? | 22:17 |
| sdake | I dont know anything about ubuntu | 22:17 |
| sdake | so kolla at 16% of interest level of deployments of eval | 22:18 |
| sdake | of 450 operators with openstack clouds | 22:18 |
| sdake | = winning ! :) | 22:18 |
| inc0 | keys got revoced | 22:19 |
| inc0 | revoked | 22:19 |
| sdake | frieghentning | 22:19 |
| inc0 | we need to get new one | 22:19 |
| sdake | how does one go about doing that | 22:19 |
| rhallisey | 22:20 | |
| rhallisey | 22:20 | |
| rhallisey | 22:20 | |
| sdake | let pavo submit the change | 22:20 |
| rhallisey | 22:20 | |
| sdake | oh there is rhallisey :) | 22:20 |
| rhallisey | 22:20 | |
| rhallisey | 22:20 | |
| rhallisey | yo | 22:20 |
| rhallisey | wo | 22:20 |
| inc0 | it has to be backported to both newton and mitaka | 22:20 |
| sdake | rhallisey rubber stamp https://review.openstack.org/#/c/388237/ | 22:20 |
| inc0 | or even liberty | 22:20 |
| sdake | ok well lets focus on master and newton for the moment | 22:20 |
| rhallisey | easy | 22:21 |
| rhallisey | sdake, did they release the interest already? | 22:21 |
| rhallisey | link? | 22:21 |
| sdake | http://www.openstack.org/analytics | 22:21 |
| sdake | we wer eat 7% in oct ;-) | 22:21 |
| sdake | i can live with 100% growth in one cycle ;-) | 22:22 |
| Pavo | thats actually pretty good compared to others | 22:22 |
| inc0 | yeah, wait for this summit | 22:22 |
| sdake | yup doubles every summit | 22:22 |
| bjolo__ | kolla is the best ops tool out there | 22:23 |
| sdake | thanks bjolo__ | 22:23 |
| sdake | its still imperfect but pretty close :) | 22:23 |
| Pavo | maybe it would be possible to add the ml2 settings to globals instead of using a custom ini file | 22:23 |
| bjolo__ | trust me, ive hacked around and tried most of them | 22:23 |
| bjolo__ | kolla is still a little rough and missing some stuff in my view, but overall and compared to the competition it rocks | 22:24 |
| sdake | bjolo__ roger that | 22:24 |
| *** dave-mccowan has joined #openstack-kolla | 22:25 | |
| sdake | missing docs for sure | 22:25 |
| bjolo__ | im working on about 5-6 blueprints for the summit | 22:25 |
| sdake | dave-mccowan http://www.openstack.org/analytics | 22:25 |
| bjolo__ | to add some operator features | 22:25 |
| sdake | inc0 working on this key thing or do I need to learn ubuntu :) | 22:26 |
| bjolo__ | ok ubuntu source deploy with all services enabled in progress | 22:26 |
| inc0 | sdake, working on it | 22:27 |
| sdake | inc0 cool thx :) | 22:27 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/387628 | 22:30 |
| sdake | that needs to merge as well | 22:32 |
| sdake | wondering when the bot would get around to our repo | 22:32 |
| inc0 | sdake, https://review.openstack.org/388205 | 22:33 |
| bjolo__ | keystone fernet failed | 22:34 |
| awebber | in terms of the kolla-kubernets multi-server guide, is there any information on "continuing" the installation - deploying neutron and nova etc.? | 22:34 |
| bjolo__ | http://paste.openstack.org/show/586291/ | 22:34 |
| awebber | i was successfull in deploying the minimal services | 22:34 |
| bjolo__ | sdake, ^^ | 22:34 |
| *** dave-mccowan has quit IRC | 22:34 | |
| *** msimonin has quit IRC | 22:35 | |
| hogepodge | sdake: Have you run into this problem, where a reconfigure runs this task on non-storage nodes and fails? https://github.com/openstack/kolla/blob/master/ansible/roles/ceph/tasks/start_osds.yml#L23 | 22:36 |
| sdake | bjolo__ no idea | 22:37 |
| sdake | kfox1111 - any chanceyou could naswer hogepodge 's q? | 22:37 |
| sdake | hogepodge i haven't seen it personally no - and hasn't been recorded | 22:37 |
| sdake | bjolo__ actually have an idea | 22:38 |
| sdake | bjolo__ when did you build images? | 22:38 |
| bjolo__ | just now | 22:38 |
| *** rhallisey has quit IRC | 22:39 | |
| sdake | git you git pull first? | 22:39 |
| sdake | git pull --rebase that is | 22:40 |
| sdake | bjolo__ a change merged today that introduced some fernet magic | 22:41 |
| sdake | which may get you going | 22:41 |
| kfox1111 | sdake: haven't played much with kolla ceph yet. just enough to get a single node cluster running in k8s using the kolla containers for testing. | 22:41 |
| sdake | kfox1111 yes but you understand the code | 22:41 |
| kfox1111 | barley/sort of. :) | 22:42 |
| kfox1111 | Once this outage is complete, I can try and look though. | 22:42 |
| bjolo__ | just checked. only diff since my build was the "Fix wrong log permission for services collected through syslog" | 22:42 |
| hogepodge | I found a workaround (I just added the node to the ceph cluster), but I'll file a bug. It's not a critical blocker for me. | 22:42 |
| kfox1111 | hogepodge: oh, good. | 22:43 |
| *** janem has joined #openstack-kolla | 22:43 | |
| bjolo__ | let me sanitize my env and try to deploy again | 22:43 |
| sdake | hogepodge thanks for the bug report - we have plenty of those ;) | 22:43 |
| sdake | bjolo__ sure you pushed the rebuild images? | 22:43 |
| bjolo__ | yes | 22:43 |
| bjolo__ | fresh tag for the latest build | 22:44 |
| sdake | bjolo__ do you have bedca5b35e8f9336182d5588d5650c1bb0682702 | 22:44 |
| sdake | in your repo | 22:44 |
| sdake | and run pip show kolla | 22:44 |
| bjolo__ | yes got it | 22:45 |
| sdake | bjolo__ - rather do you have 9cc3795b19e29d3400f09ec1708d0a889db4eaae | 22:46 |
| sdake | bjolo__ did you run git checkout stable/mitaka? | 22:46 |
| sdake | that fernet patch should be righton otp | 22:46 |
| bjolo__ | got it | 22:46 |
| sdake | got what? :) | 22:46 |
| sdake | sorry speak with people that say got it all day long -not sure what they got :) | 22:47 |
| *** gfidente has quit IRC | 22:47 | |
| sdake | ok guys if i stay at my keyboard any longer i am going to probably be dead soon | 22:47 |
| sdake | so I am taking a two hour powernap | 22:47 |
| sdake | please try to get the stuffs merged | 22:48 |
| bjolo__ | you want me to pip install kolla? | 22:48 |
| bjolo__ | ive been runing from git stable/newton | 22:49 |
| openstackgerrit | Merged openstack/kolla: Congress doesn't deploy release note https://review.openstack.org/388237 | 22:49 |
| bjolo__ | got the commit in my build git | 22:49 |
| sdake | bjolo__ run pip show kolla | 22:51 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Fix mariadb build issue https://review.openstack.org/388249 | 22:51 |
| bjolo__ | nothing since i dont have pip installed kolla | 22:52 |
| bjolo__ | is that what you wanted to check? | 22:52 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/387628 | 22:52 |
| bjolo__ | hold on | 22:53 |
| bjolo__ | could be my deploy git that was behind | 22:53 |
| * bjolo__ getting sleepy eyes | 22:53 | |
| bjolo__ | sdake, probably my fault. new deploy runing | 22:55 |
| sdake | bjolo__ i'm beat too if it makes you feel better :) | 22:57 |
| bjolo__ | tnx :) | 22:57 |
| inc0 | any core up? I need ack on ^ | 22:59 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Fix mariadb build issue https://review.openstack.org/388252 | 22:59 |
| sdake | inc0 try calling ryan | 22:59 |
| sdake | inc0 us coverage is pretty bad during this time of hour | 22:59 |
| sdake | we need another core reviewer in this tz | 22:59 |
| inc0 | and Jeffrey4l probably passed out | 23:00 |
| inc0 | sdake, can we push tag for tomorrow? | 23:00 |
| inc0 | unless we want unbuildable ubuntu | 23:00 |
| inc0 | or we just break rules and ack it with one +2 | 23:00 |
| sdake | no | 23:01 |
| sdake | we cna't push tag tomorrow | 23:01 |
| *** awiddersheim has joined #openstack-kolla | 23:01 | |
| sdake | inc0 do you have ryan's phone # | 23:01 |
| sdake | i've got it but my phone is out of juice | 23:01 |
| inc0 | yeah let me call him | 23:01 |
| *** eaguilar has joined #openstack-kolla | 23:02 | |
| bjolo__ | deploy has passed keystone fernet now :) | 23:03 |
| sdake | bjolo__ nice | 23:03 |
| sdake | so old images :) | 23:03 |
| bjolo__ | no | 23:03 |
| sdake | different problem? | 23:03 |
| bjolo__ | missed the latest commits in my git im doing the deploy from | 23:03 |
| inc0 | sdake, https://review.openstack.org/#/c/388252/ | 23:04 |
| inc0 | this one is for newton | 23:04 |
| inc0 | Ryan will ack them when he gets out of car:) | 23:04 |
| sdake | bjolo__ while your deploying - can you build with the first review mariadb | 23:04 |
| sdake | dont push just build | 23:05 |
| sdake | well i gotta say that was painful :) | 23:05 |
| sdake | we probably have a slew of bugs to fix for 3.0.1 | 23:05 |
| inc0 | charlie foxtrot | 23:05 |
| sdake | but overall everything seems to work | 23:05 |
| bjolo__ | ok | 23:05 |
| inc0 | good that mariadb broke today not tomorrow | 23:06 |
| sdake | bjolo__ make sure not to pull that last review | 23:06 |
| sdake | that will blow up your repo | 23:06 |
| sdake | pull the othe one for master inc0 posted | 23:06 |
| inc0 | sdake, https://review.openstack.org/#/c/388252/1 tag +W | 23:06 |
| kfox1111 | trunk passed another test. | 23:09 |
| kfox1111 | how far out from cutting the release? | 23:09 |
| inc0 | very close unless sdake got heart attach | 23:09 |
| sdake | kfox1111 waiting for gates on one more patch | 23:11 |
| sdake | did we get em all | 23:11 |
| *** mtaylor22 has joined #openstack-kolla | 23:11 | |
| sdake | been like 50+ patches merged since friday | 23:11 |
| kfox1111 | wow. | 23:11 |
| inc0 | sdake, I'm not sure how gates will behave due to mirrors | 23:12 |
| bjolo__ | nova_compute_ironic stuck restarting. | 23:12 |
| kfox1111 | k. let me know when the gate finishes, and I can do a quick test. | 23:12 |
| bjolo__ | http://paste.openstack.org/show/586293/ | 23:12 |
| bjolo__ | have not really deployed ironic before so could be on my end | 23:12 |
| sdake | bjolo__ ironic is busted imo | 23:12 |
| sdake | bjolo__ just ignore it | 23:12 |
| bjolo__ | ok | 23:13 |
| sdake | bjolo__ it is in various working shape | 23:13 |
| sdake | not what i'd like | 23:13 |
| sdake | as in "bulletproof" | 23:13 |
| bjolo__ | haha | 23:13 |
| bjolo__ | dont like it either | 23:13 |
| bjolo__ | as we discussed before. a new operator should be able to do pip install kolla, enable all services in globals, and deploy using the docker hub images | 23:14 |
| inc0 | bjolo__, should != ever going to happen | 23:15 |
| sdake | bjolo__ sounds like what you want is a product ;) | 23:15 |
| kfox1111 | bjolo__: really needs to happen. | 23:15 |
| inc0 | what I want is to have some sort of matrix that will show which projects are well tested and we vouch for them | 23:15 |
| inc0 | and which are not | 23:15 |
| sdake | bjolo if i was going to do that i would have bet all my marbles a year ago or more | 23:15 |
| kfox1111 | well, not the "enableing everything" part. | 23:15 |
| inc0 | there always be those which will not be tested as much | 23:15 |
| kfox1111 | but the hub part. | 23:15 |
| inc0 | hub will get all the images | 23:15 |
| inc0 | thats true | 23:16 |
| bjolo__ | we need more multinode testing | 23:16 |
| inc0 | that's hard in gates, but true | 23:16 |
| sdake | we need moar testing all over the place | 23:16 |
| kfox1111 | bjolo__: +1 | 23:16 |
| inc0 | we also need proper upgrade testing | 23:16 |
| bjolo__ | i dont mean that every container should be configured tip top | 23:16 |
| kfox1111 | I'm working on multinode kolla-kubernetes soon. | 23:16 |
| bjolo__ | but it should deploy at least | 23:16 |
| sdake | seems like nobrainer | 23:17 |
| bjolo__ | probably we need to support mixing binary and source containers as well since not all distros have packages for all services | 23:17 |
| * bjolo__ is writing a bp for that | 23:18 | |
| sdake | bjolo we can't reallly do that in the build system itself | 23:18 |
| sdake | it could be done in the deploy | 23:19 |
| sdake | but very difficult | 23:19 |
| *** dave-mccowan has joined #openstack-kolla | 23:19 | |
| bjolo__ | yes, the code for selecting what container to deploy can be overriden but it is not really practical | 23:19 |
| sdake | bjolo right | 23:20 |
| sdake | bjolo not practical | 23:20 |
| sdake | ok - guys - gotta get atleast a 2 hour powernap - i'll be back at 1:00 UTC :) | 23:20 |
| *** g3ek has quit IRC | 23:20 | |
| sdake | bbiaf | 23:20 |
| *** sdake has quit IRC | 23:20 | |
| inc0 | bjolo__, possible today | 23:20 |
| *** awebber has quit IRC | 23:20 | |
| kfox1111 | it would be nice to get the gate to the point where a user that wanted to test a certain combination of things to run the test themselves though. | 23:20 |
| kfox1111 | I'm working on getting the minikube stuff to that point. | 23:20 |
| *** haplo37 has quit IRC | 23:21 | |
| kfox1111 | so like a mitaka cloud, with a newton heat for example. | 23:21 |
| inc0 | possible today | 23:21 |
| kfox1111 | shoudl work fine, but you never know if you don't try. | 23:21 |
| inc0 | in multiple ways even:) | 23:21 |
| kfox1111 | being able to repeat the gate tests in that particular config would be awesome. :) | 23:21 |
| bjolo__ | inc0, yes but could be better imo. (could be me being a newbie as well ) | 23:22 |
| mtaylor22 | morning | 23:23 |
| inc0 | bjolo__, I'll be happy to show you these options...tomorrow;) | 23:23 |
| bjolo__ | yes please | 23:23 |
| bjolo__ | but tomorrow | 23:23 |
| jascott1 | good morning! | 23:23 |
| bjolo__ | is murano also busted in newton? | 23:23 |
| *** denaitre has quit IRC | 23:24 | |
| openstackgerrit | BorneMace proposed openstack/kolla: Added support for a stop playbook in kolla https://review.openstack.org/388254 | 23:25 |
| *** schwicht has quit IRC | 23:25 | |
| *** HyperJohnGraham has quit IRC | 23:26 | |
| *** haplo37 has joined #openstack-kolla | 23:26 | |
| bjolo__ | http://paste.openstack.org/show/586294/ | 23:27 |
| bjolo__ | inc0, | 23:27 |
| *** eaguilar has quit IRC | 23:27 | |
| *** g3ek has joined #openstack-kolla | 23:29 | |
| bjolo__ | is that murano that cant handle my TLS certificate? | 23:29 |
| *** eaguilar has joined #openstack-kolla | 23:30 | |
| *** diogogmt has quit IRC | 23:32 | |
| *** denaitre has joined #openstack-kolla | 23:41 | |
| *** inc0 has quit IRC | 23:47 | |
| openstackgerrit | Merged openstack/kolla: Fix mariadb build issue https://review.openstack.org/388252 | 23:49 |
| *** schwicht has joined #openstack-kolla | 23:49 | |
| *** jtriley has joined #openstack-kolla | 23:54 | |
| bjolo__ | https://bugs.launchpad.net/kolla/+bug/1634695 | 23:55 |
| openstack | Launchpad bug 1634695 in kolla "murano deploy fails when TLS is enabled" [Undecided,New] | 23:55 |
| bjolo__ | kfox1111, too many bugs caused by inter container dependencies | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
