| *** thumpba has quit IRC | 00:20 | |
| *** sdake has joined #kolla | 00:27 | |
| *** absubram has quit IRC | 00:31 | |
| *** jtriley has joined #kolla | 00:34 | |
| *** jtriley has quit IRC | 00:43 | |
| *** jtriley has joined #kolla | 00:47 | |
| *** jtriley has quit IRC | 00:56 | |
| *** JoseMello has quit IRC | 00:58 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Ubuntu custom apt repos https://review.openstack.org/275463 | 01:08 |
|---|---|---|
| *** jtriley has joined #kolla | 01:11 | |
| *** kjelly has quit IRC | 01:28 | |
| *** achanda has joined #kolla | 01:32 | |
| openstackgerrit | Martin André proposed openstack/kolla: Validate that all the containers are started https://review.openstack.org/275468 | 01:34 |
| SamYaple | mandre: is it just me, or have gates been more stable since file backend merge? | 01:37 |
| SamYaple | also mandre if you could review https://review.openstack.org/#/c/274928/ | 01:38 |
| *** weiyu has joined #kolla | 01:48 | |
| *** Marga_ has quit IRC | 01:56 | |
| *** jtriley has quit IRC | 02:00 | |
| *** sdake has quit IRC | 02:05 | |
| *** salv-orl_ has joined #kolla | 02:09 | |
| *** salv-orlando has quit IRC | 02:12 | |
| *** Marga_ has joined #kolla | 02:17 | |
| *** Marga_ has quit IRC | 02:22 | |
| *** Jeffrey4l has joined #kolla | 02:22 | |
| *** weiyu has quit IRC | 02:27 | |
| *** weiyu has joined #kolla | 02:32 | |
| *** Marga_ has joined #kolla | 02:34 | |
| *** weiyu has quit IRC | 02:36 | |
| *** phong_ has joined #kolla | 02:36 | |
| *** weiyu has joined #kolla | 02:36 | |
| *** Marga_ has quit IRC | 02:38 | |
| *** dims_ has quit IRC | 02:40 | |
| *** Jeffrey4l has quit IRC | 02:42 | |
| *** Marga_ has joined #kolla | 02:48 | |
| *** kjelly has joined #kolla | 02:50 | |
| *** Marga_ has quit IRC | 02:50 | |
| *** Marga_ has joined #kolla | 02:51 | |
| *** daneyon_ has joined #kolla | 02:51 | |
| *** daneyon has quit IRC | 02:54 | |
| *** sdake has joined #kolla | 03:06 | |
| sdake | evening folks | 03:06 |
| kjelly | hi sdake | 03:07 |
| sdake | hey kjelly | 03:07 |
| sdake | soooooo tired | 03:08 |
| SamYaple | go to bed sdake | 03:08 |
| sdake | note to self install tile flooring before moving into house next time | 03:08 |
| sdake | i've got atlast 4 hours of labor left to get my kids room into shape | 03:08 |
| sdake | 2 more weeks should be done | 03:09 |
| sdake | or 3 actually but 1 of them i am out of town | 03:09 |
| sdake | yay a labor vacation | 03:09 |
| *** jtriley has joined #kolla | 03:13 | |
| SamYaple | cool metadata working with thin neutron containers sdake | 03:17 |
| SamYaple | that was the final piece | 03:17 |
| sdake | hot | 03:17 |
| SamYaple | we can absolutely do this thin container thing now | 03:17 |
| sdake | now we just have to figure out how to ugprade it | 03:17 |
| SamYaple | easy peasy | 03:18 |
| SamYaple | neutron-agents is a hard down anyway | 03:18 |
| sdake | agreed | 03:18 |
| SamYaple | so now its just a hard down and removal | 03:18 |
| SamYaple | nothing to it | 03:18 |
| sdake | as long as once it starts up a gain the network comes back im good | 03:18 |
| SamYaple | there may be some dangling namespace issues and that would be the only hard part, but we can figure it out | 03:19 |
| SamYaple | just need to do some testing | 03:19 |
| *** yuanying_ has quit IRC | 03:20 | |
| SamYaple | ahhh man. with thin neutron containres its so much nicer | 03:21 |
| SamYaple | ive been testing with them for about a week now | 03:21 |
| SamYaple | much much cleaner | 03:21 |
| sdake | minor optimization but needed to be done | 03:22 |
| sdake | the whole architecture much more tidy now | 03:22 |
| SamYaple | yea. | 03:22 |
| sdake | kolla is highly consistent at this point | 03:22 |
| SamYaple | thin neutron containers patch finiliazes the consistency too | 03:23 |
| sdake | except upgrades ;) | 03:23 |
| SamYaple | i built in the drop-root patches | 03:23 |
| sdake | yes that is what i meant SamYaple | 03:23 |
| sdake | without neutron thin containers we were very inconsistent | 03:23 |
| SamYaple | well i wouldnt say vary | 03:23 |
| SamYaple | neutron is just a snowflake | 03:23 |
| sdake | somewhat | 03:23 |
| SamYaple | exclusing neutron we are perfect | 03:24 |
| SamYaple | (in consistency) | 03:24 |
| sdake | choice of words off, i'm a little beat | 03:24 |
| SamYaple | bad/good thing with metadata container | 03:24 |
| SamYaple | i have to share a socket with the l3-agent and im doing that through a named volume | 03:24 |
| *** yuanying has joined #kolla | 03:24 | |
| SamYaple | we can apply that to the other "need to share sockets" containers | 03:25 |
| sdake | thats awesome | 03:25 |
| sdake | there is a whole bunch of those iiirc | 03:25 |
| SamYaple | no | 03:25 |
| SamYaple | the named volumes solve the data container problem | 03:25 |
| SamYaple | specifically the "lose all data orphaned volumes" data contaienr issue | 03:25 |
| sdake | understood | 03:25 |
| sdake | named volumes are one of the 3 features i'm going to pimp out to the foundation press | 03:26 |
| sdake | actually four | 03:26 |
| sdake | docs, security, upgrades, named volumes | 03:27 |
| SamYaple | eh i dont know. thats kinda specific and technical, but not super useful other than "containers done right" | 03:27 |
| SamYaple | security we still suck at | 03:27 |
| SamYaple | we need selinux | 03:27 |
| sdake | yup | 03:27 |
| SamYaple | docker 1.10 has selinux fully supported i think | 03:27 |
| sdake | security better then it was | 03:27 |
| SamYaple | we need to tackle that | 03:27 |
| sdake | dockdr 1.5 had selinux fully supported | 03:27 |
| SamYaple | no it didn't | 03:27 |
| SamYaple | and certinaly not for all backends | 03:28 |
| sdake | nobody has written the rules for it yet | 03:28 |
| sdake | ryan said he would but its a multiperson big task job | 03:28 |
| SamYaple | what do you mean? | 03:28 |
| SamYaple | they ship rules for docker.... | 03:28 |
| sdake | that work with kolla? | 03:29 |
| sdake | kolla probably needs some special rules | 03:29 |
| SamYaple | oh no | 03:29 |
| SamYaple | thats a different thing | 03:29 |
| SamYaple | those filters are a pita | 03:29 |
| sdake | thats the part i' mtalking about | 03:29 |
| SamYaple | yea | 03:29 |
| SamYaple | it wouldn't have mattered though since its not until docker 1.10 selinux has the final major issues ironed out | 03:29 |
| SamYaple | and btrfs now works with selinux+docker | 03:30 |
| SamYaple | (with 1.10) | 03:30 |
| sdake | didn't prior? | 03:30 |
| SamYaple | nope | 03:30 |
| sdake | cool good information | 03:30 |
| SamYaple | btrfs+selinux == yes, +docker == no | 03:30 |
| sdake | is it still accruate btrfs = best build backend, overlayfs = best deploy backend? | 03:31 |
| sdake | boy i need my lab back in operation and right now | 03:32 |
| SamYaple | sdake: i think thats accurate | 03:32 |
| SamYaple | although i can now build on overlayfs with dev kernel and dev docker | 03:33 |
| sdake | interesting | 03:33 |
| sdake | dev docker and dev registry = 13 minute push in a vm | 03:33 |
| sdake | amazing difference | 03:33 |
| SamYaple | yup | 03:34 |
| sdake | we could possibly push per build now | 03:34 |
| SamYaple | thats about the same time as v1 | 03:34 |
| SamYaple | a bit faster now | 03:34 |
| SamYaple | we already do that sdake ;) | 03:34 |
| sdake | before i was at 30 min push on bare metal | 03:34 |
| SamYaple | oh you mean push to docker registry | 03:34 |
| sdake | yes | 03:34 |
| SamYaple | oh.... hmmm | 03:34 |
| SamYaple | infra will hate us | 03:34 |
| SamYaple | docker will hate us | 03:34 |
| SamYaple | unless we can do this in a gate only job..... | 03:35 |
| sdake | we could push our tagged releases | 03:35 |
| SamYaple | thats certainly doable | 03:35 |
| sdake | infra doesn't like gate only jobs | 03:35 |
| sdake | if there is a gate there has to be a check | 03:35 |
| SamYaple | well see | 03:35 |
| sdake | we would be #1 in storage on docker hub in like 3 weeks ;-) | 03:35 |
| SamYaple | we can possibly get a publish job | 03:35 |
| SamYaple | yes | 03:36 |
| SamYaple | anyway on tagged releases we absolutely can get taht job if the build+push is under an hour | 03:36 |
| sdake | yes publish job for tags | 03:36 |
| sdake | that would be so fantastic | 03:36 |
| sdake | pepole bitch non stop about building the containers | 03:36 |
| SamYaple | we absoluely could there | 03:37 |
| SamYaple | a per commit is actually reasonable too, but not a per patchset | 03:37 |
| SamYaple | we need a much better gate for any of this though | 03:37 |
| sdake | ya you can do per commit rather then per patchset | 03:37 |
| SamYaple | no you cant | 03:37 |
| SamYaple | not if the check job == gate job | 03:37 |
| SamYaple | if its a gate only job then yes | 03:37 |
| sdake | this is how docs publish jobs work | 03:38 |
| sdake | it can be done | 03:38 |
| SamYaple | but we need full build + deploy + functional testing and we need them voting first | 03:38 |
| sdake | ya voting gates would be nice | 03:38 |
| SamYaple | i think its a preq | 03:38 |
| sdake | i'd be happy with a voting eploy gate only | 03:38 |
| SamYaple | we cant be pushing broken containers | 03:38 |
| SamYaple | yea thats what i mean | 03:39 |
| SamYaple | were getting there | 03:39 |
| SamYaple | the all file backend gate has been very good | 03:39 |
| SamYaple | alot less failures | 03:39 |
| SamYaple | and failures now are _real_ failures | 03:39 |
| SamYaple | very few timeouts (though they still happen) | 03:39 |
| sdake | ahow did that nova prototype patch i put up work out for asalkeld | 03:39 |
| sdake | i haven't looked at my email for 5 days | 03:40 |
| asalkeld | sdake: haven't tried, super busy getting with a patch atm | 03:40 |
| sdake | asalkeld i need it tested before friday if posible | 03:40 |
| sdake | so i can work on it on the plane | 03:40 |
| asalkeld | see what i can do | 03:41 |
| SamYaple | sdake: i would recommend you get some cloud-servers to test it yourself | 03:41 |
| *** Allen_Gao has joined #kolla | 03:41 | |
| SamYaple | youll need to do lots of testing | 03:41 |
| sdake | i need bare metal for accurate testing | 03:42 |
| SamYaple | .... no you dont | 03:42 |
| sdake | anyway i just want to get it in the queue not approved | 03:42 |
| SamYaple | not for permissions | 03:42 |
| SamYaple | and not for accurate testing either | 03:42 |
| SamYaple | most people are not testing in baremetal actually | 03:42 |
| SamYaple | asalkeld may be. asalkeld are you testing on bare metal? | 03:43 |
| asalkeld | vagrant | 03:43 |
| sdake | i recognizze nobody tetssts bare metal | 03:43 |
| sdake | taht is why i must | 03:43 |
| SamYaple | i do bare metal | 03:43 |
| * asalkeld need better setup | 03:43 | |
| SamYaple | honestly there just isn't much difference | 03:43 |
| sdake | nobody/few | 03:43 |
| sdake | the networkign is different | 03:43 |
| SamYaple | we haven't had a bare metal-but-not-vm bug.... ever | 03:43 |
| sdake | we used to until we built for all distros ;-) | 03:44 |
| sdake | that fixed a whole slew of problems | 03:44 |
| sdake | sirius radio = 35$ a month for my household | 03:46 |
| sdake | i am 10$'ed to death | 03:46 |
| SamYaple | man sdake weve come a long way | 03:46 |
| SamYaple | remember symlink speghetti? | 03:46 |
| *** unicell1 has quit IRC | 03:46 | |
| sdake | i remember when the project had 2 people in the irc meetings ;) | 03:46 |
| SamYaple | that was two weeks ago | 03:46 |
| sdake | nah last two weeks there has been like 15+ people in the irc meetings | 03:47 |
| sdake | what i mean is kolla at one point was almost dead | 03:47 |
| sdake | until we moved to ansible | 03:47 |
| sdake | almost ahead of its time | 03:47 |
| sdake | good thing i dont give up :) | 03:48 |
| *** klint has joined #kolla | 03:48 | |
| sdake | actually not move o ansible | 03:48 |
| sdake | move away from kubernetes | 03:48 |
| sdake | thats what did it | 03:48 |
| sdake | single node deploy simplify saved the project | 03:49 |
| sdake | but yes i remember symlink madness | 03:49 |
| sdake | was worth a shot ;) | 03:49 |
| sdake | what we have now is cleaerly clearly superior | 03:49 |
| SamYaple | well project was dead before ansible | 03:50 |
| SamYaple | it had no path to multinode | 03:51 |
| sdake | it wassn't dead, dead = no interest | 03:51 |
| sdake | no path to multinode = not viable | 03:51 |
| sdake | at one point there was almost no interest | 03:51 |
| SamYaple | yea around that time | 03:51 |
| SamYaple | look at commit history | 03:52 |
| sdake | before your time i think :) | 03:52 |
| SamYaple | of course. i brought ansible :P | 03:52 |
| SamYaple | that was my first commit | 03:52 |
| SamYaple | ansible with a docker-compose module | 03:52 |
| SamYaple | man that was ugly | 03:52 |
| sdake | had to break that logjam | 03:52 |
| sdake | 5 -2 votes 2 +2 votes to 7 +2 votes | 03:52 |
| sdake | winning | 03:52 |
| sdake | while im patting myself on the back, that was my proudest momemt in openstack, ever | 03:53 |
| sdake | my director said "I dont know how you did it, jedi mind tricks i guess. " | 03:54 |
| *** tfukushima has joined #kolla | 03:54 | |
| sdake | "you want the ansible" | 03:54 |
| sdake | "you dont want the kubernetes" | 03:54 |
| sdake | "20000 republican credits will be fine" | 03:54 |
| sdake | I am dactarian, mind tricks no work on me, only money!! | 03:54 |
| SamYaple | you said no kubernetes though | 03:55 |
| sdake | sadly i hae not had a chance to see the new star wars | 03:55 |
| SamYaple | ansible didnt come along for several months after | 03:55 |
| sdake | i wanted to do both at the same time | 03:55 |
| SamYaple | lies! i have logs | 03:56 |
| sdake | but had to space em out | 03:56 |
| SamYaple | you didnt know about ansible | 03:56 |
| sdake | i knew about ansible of course | 03:56 |
| SamYaple | i only actually went to the #kolla channel after i heard you fully dropped kubernetes | 03:56 |
| SamYaple | every time i went to kolla before i had to stop because kubernetes was just the wrong path | 03:56 |
| sdake | agree | 03:57 |
| sdake | think back to what would have happened if i would have said "we are droppign kubernetes and going all in on ansible" | 03:57 |
| sdake | would you hae come on board? | 03:57 |
| SamYaple | based on the logs i have... that would have never happened | 03:57 |
| SamYaple | ansible wasnnt on your radar | 03:57 |
| sdake | it was on my radar, but the rest of the community radar it was not on | 03:57 |
| SamYaple | you have a funny memory dude | 03:57 |
| sdake | i dont tell you everything i'm thinking sam ;) | 03:58 |
| SamYaple | thats not what you said | 03:58 |
| SamYaple | you are contridiciiting what you _actually_ said | 03:58 |
| SamYaple | but im not going down this path | 03:58 |
| sdake | ya lets not fight i'm too tired | 03:58 |
| SamYaple | yup | 03:58 |
| sdake | and it serves no purpose | 03:58 |
| sdake | ;-) | 03:58 |
| sdake | anyway it all worked out for hte best | 03:59 |
| sdake | thats why I have an 80% hit rate on my projects ;) | 03:59 |
| sdake | ok folks well i got about 4 hours of labor to do then hitting the sack | 04:05 |
| sdake | i should be in tomorrow | 04:06 |
| sdake | my interview with the foundation got pushed to tomorrow afternoon | 04:06 |
| *** yuanying has quit IRC | 04:06 | |
| sdake | so i will be out for a few hours as a result of that | 04:06 |
| sdake | and mornings 7-9 are a disaster when robyn is out of town | 04:06 |
| sdake | enjoy :) | 04:06 |
| sdake | another 4 hour workday groan | 04:07 |
| *** yuanying has joined #kolla | 04:07 | |
| *** weiyu has quit IRC | 04:08 | |
| *** sdake has quit IRC | 04:10 | |
| *** tfukushima has quit IRC | 04:18 | |
| *** jtriley has quit IRC | 04:22 | |
| *** achanda has quit IRC | 04:28 | |
| *** achanda has joined #kolla | 04:43 | |
| *** jasonsb has joined #kolla | 04:53 | |
| *** tfukushima has joined #kolla | 05:02 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Convert Neutron to thin containers https://review.openstack.org/272715 | 05:15 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Correct issue with virtualenv rootwrap https://review.openstack.org/275504 | 05:15 |
| *** diga has joined #kolla | 05:19 | |
| *** weiyu has joined #kolla | 05:19 | |
| *** jasonsb has quit IRC | 05:21 | |
| *** weiyu has quit IRC | 05:23 | |
| *** weiyu has joined #kolla | 05:28 | |
| *** weiyu has quit IRC | 05:35 | |
| *** weiyu has joined #kolla | 05:37 | |
| *** ssurana has quit IRC | 05:47 | |
| *** Marga_ has quit IRC | 05:58 | |
| *** Jeffrey4l has joined #kolla | 06:02 | |
| *** jasonsb has joined #kolla | 06:04 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Convert Neutron to thin containers https://review.openstack.org/272715 | 06:05 |
| *** unicell has joined #kolla | 06:11 | |
| *** chandankumar has joined #kolla | 06:17 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Playbook for horizon upgrade https://review.openstack.org/274827 | 06:18 |
| *** diga has quit IRC | 06:19 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Convert Neutron to thin containers https://review.openstack.org/272715 | 06:21 |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Correct issue with virtualenv rootwrap https://review.openstack.org/275504 | 06:21 |
| *** chandankumar has quit IRC | 06:23 | |
| *** Marga_ has joined #kolla | 06:27 | |
| *** ssurana has joined #kolla | 06:34 | |
| *** chandankumar has joined #kolla | 06:36 | |
| *** phong_ has quit IRC | 06:38 | |
| *** dave-mccowan has quit IRC | 06:48 | |
| *** Allen_Gao has quit IRC | 06:51 | |
| *** Allen_Gao has joined #kolla | 07:08 | |
| *** diga has joined #kolla | 07:09 | |
| *** Allen_Gao has quit IRC | 07:20 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Cleanup named volumes https://review.openstack.org/275237 | 07:40 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers https://review.openstack.org/270213 | 07:40 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles https://review.openstack.org/270151 | 07:40 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use private network in default settings https://review.openstack.org/269545 | 07:40 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB https://review.openstack.org/270252 | 07:40 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml https://review.openstack.org/274560 | 07:40 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 07:40 |
| *** ppetit has joined #kolla | 07:44 | |
| *** ppetit has quit IRC | 07:45 | |
| *** Marga_ has quit IRC | 07:47 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Cleanup named volumes https://review.openstack.org/275237 | 08:04 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers https://review.openstack.org/270213 | 08:04 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles https://review.openstack.org/270151 | 08:04 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use private network in default settings https://review.openstack.org/269545 | 08:04 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB https://review.openstack.org/270252 | 08:04 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml https://review.openstack.org/274560 | 08:04 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 08:04 |
| *** chandankumar has quit IRC | 08:05 | |
| *** salv-orlando has joined #kolla | 08:10 | |
| *** dwalsh has joined #kolla | 08:11 | |
| *** salv-orl_ has quit IRC | 08:12 | |
| *** fgimenez has joined #kolla | 08:12 | |
| *** salv-orlando has quit IRC | 08:14 | |
| *** salv-orlando has joined #kolla | 08:14 | |
| *** chandankumar has joined #kolla | 08:18 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Cleanup named volumes https://review.openstack.org/275237 | 08:24 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers https://review.openstack.org/270213 | 08:24 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles https://review.openstack.org/270151 | 08:24 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use private network in default settings https://review.openstack.org/269545 | 08:24 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB https://review.openstack.org/270252 | 08:24 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml https://review.openstack.org/274560 | 08:24 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 08:24 |
| *** Jeffrey4l has quit IRC | 08:33 | |
| *** dwalsh has quit IRC | 08:38 | |
| *** mbound has joined #kolla | 08:42 | |
| *** weiyu has quit IRC | 08:43 | |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: [POC] Heka proof-of-concept tests https://review.openstack.org/269745 | 08:48 |
| *** jmccarthy has joined #kolla | 08:49 | |
| *** salv-orlando has quit IRC | 08:52 | |
| *** ppetit has joined #kolla | 08:55 | |
| *** athomas has joined #kolla | 09:09 | |
| *** achanda has quit IRC | 09:14 | |
| *** sdake has joined #kolla | 09:15 | |
| *** achanda has joined #kolla | 09:18 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Ensuring the openvswitch_db is ready before creating bridges https://review.openstack.org/275573 | 09:23 |
| *** ppetit has quit IRC | 09:24 | |
| *** achanda has quit IRC | 09:24 | |
| *** kproskurin has joined #kolla | 09:28 | |
| *** chandankumar has quit IRC | 09:31 | |
| *** ppetit has joined #kolla | 09:32 | |
| *** ppetit has quit IRC | 09:36 | |
| *** Allen_Gao has joined #kolla | 09:36 | |
| openstackgerrit | venkatamahesh proposed openstack/kolla: Fix some word spellings https://review.openstack.org/275582 | 09:38 |
| *** tzn has joined #kolla | 09:46 | |
| *** tfukushima has quit IRC | 09:51 | |
| *** tfukushima has joined #kolla | 09:51 | |
| *** akwasnie has joined #kolla | 09:53 | |
| *** macsz1 has quit IRC | 09:54 | |
| *** macsz has joined #kolla | 09:55 | |
| openstackgerrit | venkatamahesh proposed openstack/kolla: [Trivial Fix] Correct some word spellings https://review.openstack.org/275582 | 09:57 |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Support deploy without configs in tests https://review.openstack.org/275287 | 09:58 |
| openstackgerrit | Proskurin Kirill proposed openstack/kolla-mesos: Add memcached support https://review.openstack.org/266477 | 09:58 |
| openstackgerrit | venkatamahesh proposed openstack/kolla: Fix some words spellings https://review.openstack.org/275582 | 10:01 |
| *** gfidente has joined #kolla | 10:09 | |
| *** tfukushima has quit IRC | 10:09 | |
| openstackgerrit | Allen Gao proposed openstack/kolla: Fix config help message https://review.openstack.org/275603 | 10:12 |
| *** rhallisey has joined #kolla | 10:14 | |
| *** salv-orlando has joined #kolla | 10:16 | |
| *** salv-orlando has quit IRC | 10:21 | |
| ajafo | o/ | 10:25 |
| *** akwasnie has quit IRC | 10:36 | |
| *** dwalsh has joined #kolla | 10:37 | |
| *** akwasnie has joined #kolla | 10:41 | |
| *** pbourke_ has joined #kolla | 10:45 | |
| *** fgimenez has quit IRC | 10:47 | |
| *** fgimenez has joined #kolla | 10:48 | |
| *** fgimenez has quit IRC | 10:48 | |
| *** fgimenez has joined #kolla | 10:48 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: [WIP] Provide ovs-db socket patch directly https://review.openstack.org/275621 | 10:56 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla: [WIP] Provide ovs-db socket path directly https://review.openstack.org/275621 | 10:57 |
| *** chandankumar has joined #kolla | 10:57 | |
| ccesario | hello, does kolla support upgrades ? Kilo->Liberty by example... !? | 11:02 |
| nihilifer | ccesario: not really. it's going to support liberty->mitaka upgrade | 11:03 |
| *** alyson_ has joined #kolla | 11:03 | |
| ccesario | nihilifer, is there any document explain this process ? | 11:04 |
| nihilifer | ccesario: no. it's under development | 11:04 |
| nihilifer | we're working on that right now | 11:04 |
| nihilifer | but of course if we'll end the work, there will be a document explaining the process | 11:05 |
| ccesario | nihilifer, ehehe | 11:05 |
| ccesario | rigth.... thanks a lot by info..... | 11:06 |
| *** salv-orlando has joined #kolla | 11:07 | |
| *** gfidente has quit IRC | 11:26 | |
| *** diga has quit IRC | 11:26 | |
| *** mbound has quit IRC | 11:35 | |
| *** pbourke_ has quit IRC | 11:47 | |
| *** pbourke_ has joined #kolla | 11:49 | |
| *** chandankumar has quit IRC | 12:08 | |
| *** Oloremo has joined #kolla | 12:17 | |
| *** chandankumar has joined #kolla | 12:21 | |
| *** schwarzm has quit IRC | 12:25 | |
| *** mbound has joined #kolla | 12:28 | |
| *** kproskurin_ has joined #kolla | 12:29 | |
| *** rhallisey has quit IRC | 12:31 | |
| *** dims has joined #kolla | 12:32 | |
| *** kproskurin has quit IRC | 12:34 | |
| *** kproskurin_ is now known as kproskurin | 12:34 | |
| *** rhallisey has joined #kolla | 12:35 | |
| *** rhallisey has quit IRC | 12:35 | |
| *** rhallisey has joined #kolla | 12:36 | |
| *** chandankumar has quit IRC | 12:36 | |
| *** pbourke_ has quit IRC | 12:37 | |
| -openstackstatus- NOTICE: Infra running with lower capacity now, due to a temporary problem affecting one of our nodepool providers. Please expect some delays in your jobs. Apologies for any inconvenience caused. | 12:39 | |
| *** ppetit has joined #kolla | 12:41 | |
| *** pbourke_ has joined #kolla | 12:43 | |
| *** athomas has quit IRC | 12:48 | |
| *** mbound has quit IRC | 12:49 | |
| *** chandankumar has joined #kolla | 12:50 | |
| *** mbound has joined #kolla | 12:51 | |
| *** mbound has quit IRC | 12:53 | |
| *** akwasnie has quit IRC | 12:53 | |
| *** athomas has joined #kolla | 12:54 | |
| *** Jeffrey4l has joined #kolla | 12:55 | |
| *** gfidente has joined #kolla | 13:00 | |
| openstackgerrit | Merged openstack/kolla: Fix config help message https://review.openstack.org/275603 | 13:00 |
| *** openstackgerrit has quit IRC | 13:02 | |
| *** openstackgerrit has joined #kolla | 13:03 | |
| openstackgerrit | Merged openstack/kolla: Fix some words spellings https://review.openstack.org/275582 | 13:04 |
| *** dwalsh has quit IRC | 13:15 | |
| *** fgimenez has quit IRC | 13:15 | |
| *** fgimenez has joined #kolla | 13:19 | |
| *** fgimenez has quit IRC | 13:19 | |
| *** fgimenez has joined #kolla | 13:19 | |
| *** dratushnyy has joined #kolla | 13:23 | |
| dratushnyy | Hu guys | 13:23 |
| dratushnyy | *Hi | 13:25 |
| *** akwasnie has joined #kolla | 13:29 | |
| openstackgerrit | Merged openstack/kolla-mesos: Support deploy without configs in tests https://review.openstack.org/275287 | 13:30 |
| nihilifer | anyone else encountered problem with hanging up ovs-vsctl commands? ovs-db is running without any better logs than http://paste.openstack.org/show/485848/ and in the same time this happens http://paste.openstack.org/show/485849/ | 13:33 |
| nihilifer | in the same container | 13:33 |
| *** fthiagogv has quit IRC | 13:40 | |
| *** mbound has joined #kolla | 13:42 | |
| *** Jeffrey4l has quit IRC | 13:48 | |
| *** Jeffrey4l has joined #kolla | 13:49 | |
| *** dwalsh has joined #kolla | 13:55 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/kolla-mesos: Remove oslo.log https://review.openstack.org/275708 | 13:59 |
| *** Allen_Gao has quit IRC | 13:59 | |
| clayton | how far have you guys gotten with getting nova-compute to work with iscsi? I'm trying that now and getting weird libvirt errors with ptys. | 14:11 |
| clayton | I think because I'm mounting /dev as a volume mount, so that iscsiadm can see that the device mounted | 14:11 |
| SamYaple | clayton: ive had cinder+iscsi working in a fat container (multiprocess) I never put in the work to split it out and then i changed jobs | 14:15 |
| clayton | well, for now I'm leaving cinder outside the container, just trying to make nova-compute inside the container to be happy with it | 14:16 |
| *** thumpba has joined #kolla | 14:16 | |
| clayton | it appears that os-brick is using iscsiadm make sure the devices appear in /dev before telling libvirt to attach them | 14:16 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 14:16 |
| clayton | but -v /dev:/dev doesn't work because that breaks the pty creation (apparently?) | 14:16 |
| *** thumpba has quit IRC | 14:17 | |
| *** thumpba has joined #kolla | 14:17 | |
| *** Allen_Gao has joined #kolla | 14:17 | |
| clayton | looks like it might be looking just in /dev/disk/by-path, so maybe I can bind mount just that | 14:18 |
| openstackgerrit | Evgeny Sikachev proposed openstack/kolla-mesos: Remove oslo.log https://review.openstack.org/275708 | 14:18 |
| Jeffrey4l | https://review.openstack.org/#/c/275037/ Upgrade the ansible version to 2 | 14:19 |
| *** dave-mccowan has joined #kolla | 14:20 | |
| Jeffrey4l | SamYaple, Do u have any found about how to fix this? Or which ansible issue related to this. If not, I will continue to debug this, and try to find why. | 14:20 |
| *** mdnadeem1 has joined #kolla | 14:20 | |
| elemoine | it looks to me that nova-rootwrap and cinder-rootwrap are not used, although sudoers entries exist for them, so can nova_sudoers and cinder_sudoers be removed? | 14:20 |
| *** fgimenez has quit IRC | 14:24 | |
| *** akwasnie has quit IRC | 14:26 | |
| *** fgimenez has joined #kolla | 14:26 | |
| SamYaple | Jeffrey4l: yes i know the fix | 14:27 |
| SamYaple | but im trying to get it fixed in ansible rather than change all our playbooks | 14:27 |
| SamYaple | there is no rush to make the switch in the container | 14:27 |
| SamYaple | elemoine: no | 14:27 |
| SamYaple | elemoine: those files are used for a different purpose | 14:27 |
| Jeffrey4l | Got it. I will abandon mine. | 14:28 |
| *** klint has quit IRC | 14:28 | |
| SamYaple | clayton: /dev:/dev absolutely works, we do it all over the place in kolla | 14:28 |
| SamYaple | clayton: you are running into other issues i beilieve | 14:28 |
| elemoine | SamYaple, ok, interested in more details if possible (no rush though) | 14:28 |
| clayton | SamYaple: nod, that seems to be the case | 14:28 |
| SamYaple | elemoine: open up the sudoers file, it should be clear what its used for | 14:29 |
| clayton | SamYaple: it looks like nova-compute needs access to /dev/pty to create a serial console, and something about -v /dev:/dev is confusing it. | 14:29 |
| elemoine | that's the file: https://github.com/openstack/kolla/blob/master/docker/nova/nova-base/nova_sudoers | 14:29 |
| clayton | what's odd is that it works at all without /dev:/dev | 14:30 |
| elemoine | I just cannot find any code | 14:30 |
| elemoine | calling nova-rootwrap | 14:30 |
| elemoine | which has made me think this sudoers entry is useless | 14:30 |
| clayton | this is a corner of nova-compute that I've not looked into before. going to have to dig a bit | 14:31 |
| *** athomas has quit IRC | 14:32 | |
| *** Allen_Gao has quit IRC | 14:32 | |
| *** absubram has joined #kolla | 14:33 | |
| SamYaple | elemoine: its likely farmed out to oslo.rootwrap so the code may not say "rootwrap" in it, but it certainly uses it | 14:33 |
| *** jtriley has joined #kolla | 14:34 | |
| elemoine | o | 14:34 |
| elemoine | ok | 14:34 |
| *** akwasnie has joined #kolla | 14:34 | |
| elemoine | I'll try do dig further | 14:34 |
| *** absubram_ has joined #kolla | 14:34 | |
| *** chandankumar has quit IRC | 14:36 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Ubuntu custom apt repos https://review.openstack.org/275463 | 14:36 |
| dmsimard | SamYaple: ^ cool | 14:37 |
| dmsimard | I meant to do that but the Ubuntu binary patch hadn't merged yet :) | 14:37 |
| SamYaple | dmsimard: yea i was supposed to do that a while back and forgot | 14:37 |
| * elemoine did not know about olso.rootwrap | 14:37 | |
| SamYaple | elemoine: its oslo all the way down! | 14:37 |
| SamYaple | nihilifer: Jeffrey4l can I get some reviews on https://review.openstack.org/#/c/274928/ ? | 14:38 |
| *** absubram has quit IRC | 14:38 | |
| *** absubram_ is now known as absubram | 14:38 | |
| Jeffrey4l | np | 14:38 |
| dratushnyy | guys, is problem with memory in CI is still actulal? | 14:39 |
| SamYaple | dratushnyy: very much so | 14:39 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 14:39 |
| SamYaple | dratushnyy: we need to reduce the active memory of all the services if possible | 14:39 |
| SamYaple | dratushnyy: i have not explored this path at all, it may be as simple as reducing the worker threads to 1 | 14:39 |
| dratushnyy | well I'd like to work on it :) I read about openstack gating and irc history with discussion of this problem and I'd like to work on it | 14:40 |
| dratushnyy | so, question is, whom I should ping about it? | 14:41 |
| openstackgerrit | Merged openstack/kolla-mesos: Add memcached support https://review.openstack.org/266477 | 14:42 |
| SamYaple | dratushnyy: that would be me | 14:44 |
| dratushnyy | :) cool | 14:44 |
| dratushnyy | can we sync up on it in couple hours or so? | 14:44 |
| *** Marga_ has joined #kolla | 14:44 | |
| SamYaple | dratushnyy: we have the meeting at 1630UTC, join us there and we can discuss afterward? | 14:46 |
| dratushnyy | sure | 14:47 |
| *** JoseMello has joined #kolla | 14:48 | |
| *** dratushnyy has quit IRC | 14:49 | |
| *** chandankumar has joined #kolla | 14:57 | |
| *** chandankumar has quit IRC | 14:58 | |
| *** chandankumar has joined #kolla | 14:58 | |
| *** athomas has joined #kolla | 15:04 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Ensuring the openvswitch_db is ready before creating bridges https://review.openstack.org/275573 | 15:08 |
| *** CheKoLyN has joined #kolla | 15:18 | |
| *** dratushnyy has joined #kolla | 15:20 | |
| *** blahRus has joined #kolla | 15:22 | |
| openstackgerrit | Merged openstack/kolla: Create nova-api database https://review.openstack.org/274928 | 15:30 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add ansible support for Manila https://review.openstack.org/269688 | 15:35 |
| *** Marga_ has quit IRC | 15:37 | |
| *** Marga_ has joined #kolla | 15:37 | |
| *** dwalsh has quit IRC | 15:37 | |
| *** asalkeld has quit IRC | 15:46 | |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Cleanup named volumes https://review.openstack.org/275237 | 15:50 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use docker_remove_delay for auto-removing containers https://review.openstack.org/270213 | 15:50 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Remove rsyslog from all profiles https://review.openstack.org/270151 | 15:50 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use private network in default settings https://review.openstack.org/269545 | 15:50 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Use volume for MariaDB https://review.openstack.org/270252 | 15:50 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Categorize variables in etc/globals.yml https://review.openstack.org/274560 | 15:50 |
| openstackgerrit | Michal Rostecki proposed openstack/kolla-mesos: Add neutron config https://review.openstack.org/265109 | 15:50 |
| *** salv-orl_ has joined #kolla | 15:54 | |
| *** thumpba has quit IRC | 15:55 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: Ubuntu custom apt repos https://review.openstack.org/275463 | 15:57 |
| *** salv-orlando has quit IRC | 15:57 | |
| *** absubram has quit IRC | 15:58 | |
| limamauricio | SamYaple, your patch works for me | 15:58 |
| SamYaple | limamauricio: yay glad to help | 15:59 |
| limamauricio | :) | 15:59 |
| SamYaple | hey nihilifer, you did the venv work, can you check out this patch and see if its sound? https://review.openstack.org/#/c/275504/ | 15:59 |
| *** asalkeld has joined #kolla | 16:00 | |
| limamauricio | sam, after deploy, cinder-volume service is down | 16:00 |
| limamauricio | Do you know why it happens? | 16:01 |
| *** chandankumar has quit IRC | 16:02 | |
| pbourke_ | limamauricio: try 'docker logs cinder-volume' | 16:03 |
| *** sean-k-mooney has joined #kolla | 16:03 | |
| limamauricio | Starting volume driver LVMVolumeDriver (3.0.0) Failed to initialize driver. | 16:04 |
| *** inc0 has joined #kolla | 16:04 | |
| limamauricio | Stderr: u' Volume group "cinder-volumes" not found\n Cannot process volume group cinder-volumes\n' | 16:04 |
| sdake | morning | 16:06 |
| sdake | any burning emergencies? | 16:06 |
| limamauricio | [lvmdriver-1] lvm_type = default volume_group = cinder-volumes volume_driver = cinder.volume.drivers.lvm.LVMVolumeDriver | 16:06 |
| inc0 | good morning | 16:07 |
| inc0 | sdake, one - can you ask your docker guys to make sure that we'll have option to re-connect to pids? | 16:07 |
| limamauricio | cannot process volume group cinder-volumes volume_group = cinder-volumes | 16:07 |
| inc0 | they will need something like that to enable hot upgrades of docker, so let's make sure that they will do it with backward compatibility, or at least something we could use | 16:08 |
| limamauricio | maybe the volume_group should be just cinder-volume | 16:08 |
| SamYaple | limamauricio: no i havent deployed cinder-volume in a while | 16:09 |
| sdake | asalkeld are you out of hte us or still here? | 16:09 |
| SamYaple | limamauricio: we don't use LVM in cinder (though we do have some options in the config about lvm) | 16:10 |
| SamYaple | its not working in Kolla | 16:10 |
| SamYaple | ceph is the only tested, working backend for Cinder in Kolla at this time | 16:10 |
| limamauricio | ok | 16:12 |
| limamauricio | Thanks | 16:12 |
| *** Marga_ has quit IRC | 16:16 | |
| *** mdnadeem1 has quit IRC | 16:16 | |
| *** Marga_ has joined #kolla | 16:18 | |
| *** mdnadeem1 has joined #kolla | 16:19 | |
| *** daneyon has joined #kolla | 16:24 | |
| SamYaple | @channel meeting in 5 minutes | 16:25 |
| SamYaple | unicell: will you be there to discuss run_once vs delegate_to with me? | 16:25 |
| SamYaple | (apologies for my absence last week) | 16:25 |
| sdake | rhallisey can you cover the agenda for 10-15 mins | 16:26 |
| rhallisey | sdake, sure | 16:26 |
| sdake | rhallisey sounds good i'll be back at 45 after | 16:26 |
| rhallisey | roger that | 16:26 |
| sdake | sorry, need to put together backt o bed while i have the labor here | 16:26 |
| *** daneyon_ has quit IRC | 16:27 | |
| sdake | save me 10 minutes on the agenda | 16:27 |
| sdake | announce the midcycle | 16:27 |
| sdake | no other announcements i can think of | 16:27 |
| rhallisey | k | 16:27 |
| SamYaple | 1 minute to show time | 16:28 |
| SamYaple | unicell: ping again | 16:28 |
| rhallisey | oh excellent kolla-ansible stuff | 16:29 |
| rhallisey | looking forward to that | 16:30 |
| rhallisey | need it to start kube stuff | 16:30 |
| *** thumpba has joined #kolla | 16:35 | |
| *** neilus has joined #kolla | 16:38 | |
| *** neilus has quit IRC | 16:43 | |
| clayton | SamYaple: dug into the -v /dev:/dev issue. The problem is that docker deletes the /dev/ptmx devices and makes it a symlink | 16:58 |
| SamYaple | clayton: yea thats a common thing | 16:59 |
| SamYaple | you need a new verion of something | 16:59 |
| SamYaple | systemd or kernel or docker | 17:00 |
| clayton | heh :) | 17:00 |
| SamYaple | i dont recall | 17:00 |
| clayton | looking at master on docker, it appears to be how runc still does it | 17:00 |
| SamYaple | right but you can fix that on the host | 17:00 |
| clayton | I think I can work around it by just doing -v /dev/disk/by-name:/dev/disk/by-name instead | 17:00 |
| SamYaple | i forget, but theres a solution here | 17:00 |
| SamYaple | nah you need the full /dev/ (and /run/ for that matter) for udev to work properly | 17:01 |
| clayton | well, I'd love to have a fix, I'll dig into the runc source a bit more, maybe it's conditional somewhere | 17:01 |
| SamYaple | but you can perhaps do a docker volume on /dev/ptmx | 17:01 |
| clayton | SamYaple: I'm running libvirt outside a container for now | 17:01 |
| clayton | it looks like in newer versions it has something to do with the --devices flag | 17:03 |
| *** dims has quit IRC | 17:04 | |
| *** dims has joined #kolla | 17:06 | |
| clayton | we're running 1.8.1, it appears this gets fixed in 1.9.0 | 17:06 |
| *** fthiagogv has joined #kolla | 17:07 | |
| SamYaple | for kolla, docker is required to be at 1.9+ (on master) | 17:07 |
| SamYaple | liberty is <=1.8.2 | 17:07 |
| clayton | nod. I looked into upgrading a week or two ago and ran into some issue. I'll have to look again soon | 17:07 |
| clayton | SamYaple: Thanks for the help | 17:08 |
| SamYaple | anytime | 17:08 |
| *** dims has quit IRC | 17:12 | |
| *** absubram has joined #kolla | 17:14 | |
| *** dims has joined #kolla | 17:18 | |
| *** mbound has quit IRC | 17:19 | |
| limamauricio | I'm trying to deploy kolla-ansible with ceph but i have it ValueError("No JSON object could be decoded") ValueError: No JSON object could be decoded | 17:21 |
| limamauricio | TASK: [ceph | Fetching Ceph keyrings] | 17:21 |
| SamYaple | limamauricio: you need to cleanup your environment first | 17:22 |
| limamauricio | all-in-one | 17:22 |
| elemoine | inc0, for the note, I'm actually not super clear on the "central logging" switch you mentioned, I'll write to the mailing list about it | 17:22 |
| SamYaple | you have old data and volumes from a previous deploy | 17:22 |
| inc0 | elemoine, so we have central logging in ELK | 17:22 |
| inc0 | but like everything its optional | 17:22 |
| inc0 | we should have option like enable_central_logging: "yes" | 17:22 |
| limamauricio | I have to rerun build? | 17:23 |
| inc0 | if we don't, I'd say we should | 17:23 |
| elemoine | but I need to modify the services to make them log to files (and be compatible with Heka) | 17:23 |
| elemoine | see the specs | 17:23 |
| elemoine | not sure how I can do that conditionally | 17:23 |
| inc0 | elemoine, if there is something not working with rsyslog as for now, don't bother | 17:23 |
| inc0 | dont fix rsyslog | 17:23 |
| inc0 | for openstack services tho, which already have this config | 17:23 |
| limamauricio | I have to rerun build? SamYaple | 17:24 |
| inc0 | just put config in "if" and add when to deploy rsyslog | 17:24 |
| elemoine | inc0, let me push my patches to Gerrit | 17:24 |
| elemoine | and please comment on them | 17:25 |
| inc0 | ok, thanks | 17:25 |
| elemoine | it's better to speak based on code I think | 17:25 |
| inc0 | I agree | 17:25 |
| SamYaple | limamauricio: no you need to clear your old files and container volumes | 17:25 |
| elemoine | ok, pushing and going home :) | 17:25 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add a Dockerfile for Heka https://review.openstack.org/275840 | 17:26 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add Heka to common role https://review.openstack.org/275841 | 17:26 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Have Heka collect the HAProxy logs https://review.openstack.org/275842 | 17:26 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Have Heka collect the Keepalived logs https://review.openstack.org/275843 | 17:26 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Have Heka collect the RabbitMQ logs https://review.openstack.org/275844 | 17:26 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add Heka decoder for OpenStack logs https://review.openstack.org/275845 | 17:26 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Have Heka collect the Nova logs https://review.openstack.org/275846 | 17:26 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Add Heka decoder for MariaDB logs https://review.openstack.org/275847 | 17:26 |
| openstackgerrit | Eric Lemoine proposed openstack/kolla: Have Heka collect the MariaDB logs https://review.openstack.org/275848 | 17:26 |
| elemoine | there you go | 17:26 |
| rhallisey | elemoine, nice | 17:26 |
| *** pbourke_ has quit IRC | 17:26 | |
| *** achanda has joined #kolla | 17:27 | |
| elemoine | inc0, https://review.openstack.org/#/c/275846/ is the patch for migrating Nova | 17:27 |
| *** mdnadeem1 has quit IRC | 17:29 | |
| inc0 | elemoine, quickly, just add if statement to nova.conf | 17:29 |
| inc0 | rsyslog removal | 17:29 |
| inc0 | and we need to figure out how to make /dev/log sharing optional | 17:30 |
| elemoine | inc0, exactly | 17:30 |
| inc0 | and add when statement to rsyslog deploy | 17:30 |
| inc0 | so if we can't do it nicely | 17:31 |
| elemoine | inc0, please advice | 17:31 |
| inc0 | let's ditch rsyslog | 17:31 |
| inc0 | let me think a bit about it | 17:31 |
| elemoine | I'm fine with the option, but I'd like to see a clear path towards implementing it :) | 17:31 |
| elemoine | have to go now, thanks for your help | 17:31 |
| inc0 | and I want to be clear, if rsyslog conflicts with heka in any way, I'm for removing rsyslog | 17:31 |
| *** Oloremo has quit IRC | 17:31 | |
| elemoine | inc0, ok! | 17:31 |
| dratushnyy | SamYaple, r u here? | 17:32 |
| inc0 | if we can keep it as option with just a little bit of additional work, let's do that for a release | 17:32 |
| *** mdnadeem1 has joined #kolla | 17:32 | |
| inc0 | thats my take on it, I'll think how to make this sharing stuff easy | 17:32 |
| inc0 | bye guys, meeting cluster for me | 17:32 |
| *** inc0 has quit IRC | 17:32 | |
| *** kproskurin has quit IRC | 17:34 | |
| *** neilus has joined #kolla | 17:35 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add double quotes in ceph_cache_mode https://review.openstack.org/275858 | 17:36 |
| *** unicell has quit IRC | 17:36 | |
| SamYaple | dratushnyy: i am | 17:37 |
| *** ftcpops has joined #kolla | 17:37 | |
| *** achanda has quit IRC | 17:42 | |
| *** bugroger has quit IRC | 17:42 | |
| *** jmccarthy has quit IRC | 17:45 | |
| *** fgimenez has quit IRC | 17:46 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add double quotes in ceph_cache_mode https://review.openstack.org/275858 | 17:50 |
| *** mdnadeem1 has quit IRC | 17:53 | |
| *** ssurana has quit IRC | 17:55 | |
| *** pbourke has quit IRC | 18:00 | |
| *** pbourke has joined #kolla | 18:00 | |
| *** athomas has quit IRC | 18:05 | |
| *** gfidente has quit IRC | 18:05 | |
| *** rhallisey has quit IRC | 18:07 | |
| *** jasonsb has quit IRC | 18:08 | |
| *** ppetit has quit IRC | 18:11 | |
| *** Jeffrey4l has quit IRC | 18:25 | |
| *** achanda has joined #kolla | 18:29 | |
| *** unicell has joined #kolla | 18:33 | |
| *** unicell1 has joined #kolla | 18:36 | |
| *** neilus has quit IRC | 18:37 | |
| *** unicell1 has quit IRC | 18:37 | |
| *** unicell has quit IRC | 18:37 | |
| *** unicell1 has joined #kolla | 18:37 | |
| *** unicell1 has quit IRC | 18:39 | |
| *** unicell has joined #kolla | 18:39 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Playbook for nova upgrade https://review.openstack.org/254395 | 18:42 |
| *** ssurana has joined #kolla | 18:46 | |
| *** sdake has quit IRC | 18:47 | |
| *** akwasnie has quit IRC | 18:49 | |
| *** akwasnie has joined #kolla | 18:50 | |
| *** ssurana1 has joined #kolla | 19:06 | |
| *** akwasnie has quit IRC | 19:06 | |
| *** ssurana has quit IRC | 19:08 | |
| *** achanda has quit IRC | 19:17 | |
| *** absubram has quit IRC | 19:19 | |
| openstackgerrit | Sam Yaple proposed openstack/kolla: gnocchi is busted missing tooz for source https://review.openstack.org/275898 | 19:23 |
| *** achanda has joined #kolla | 19:35 | |
| *** JoseMello has quit IRC | 19:41 | |
| *** JoseMello has joined #kolla | 19:41 | |
| *** achanda has quit IRC | 19:43 | |
| *** sdake has joined #kolla | 19:47 | |
| *** achanda has joined #kolla | 19:59 | |
| *** sdake_ has joined #kolla | 20:00 | |
| *** absubram has joined #kolla | 20:01 | |
| *** ssurana1 has quit IRC | 20:03 | |
| *** sdake has quit IRC | 20:03 | |
| *** gtt116 has joined #kolla | 20:06 | |
| *** inc0 has joined #kolla | 20:07 | |
| *** gtt116_ has quit IRC | 20:09 | |
| SamYaple | btw inc0 neutron is all good for thin containers now. metadata and all working | 20:21 |
| SamYaple | its very very clean now | 20:21 |
| SamYaple | and you can access the network namespaces from the host finally! | 20:21 |
| SamYaple | so inc0 need your ear for a second on nova and libvirt | 20:22 |
| SamYaple | specifically the communication between the two of them | 20:22 |
| *** neilus has joined #kolla | 20:23 | |
| *** neilus has quit IRC | 20:28 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/kolla-mesos: Add requirements.txt to venv https://review.openstack.org/275916 | 20:48 |
| *** ssurana has joined #kolla | 20:48 | |
| inc0 | SamYaple, I'm here, whats up? | 20:55 |
| inc0 | nova uses socket to talk to libvirt right? | 20:55 |
| *** absubram has quit IRC | 20:55 | |
| *** dims has quit IRC | 20:56 | |
| *** dratushnyy has quit IRC | 21:00 | |
| *** esikachev has joined #kolla | 21:03 | |
| *** esikachev has quit IRC | 21:05 | |
| *** dims has joined #kolla | 21:05 | |
| openstackgerrit | Merged openstack/kolla: Validate that all the containers are started https://review.openstack.org/275468 | 21:07 |
| openstackgerrit | Merged openstack/kolla: Playbook for horizon upgrade https://review.openstack.org/274827 | 21:09 |
| openstackgerrit | Merged openstack/kolla-mesos: Remove oslo.log https://review.openstack.org/275708 | 21:12 |
| *** clayton has quit IRC | 21:14 | |
| openstackgerrit | sean mooney proposed openstack/kolla: [WIP] Support installing openvswitch from source https://review.openstack.org/275932 | 21:23 |
| SamYaple | inc0: no | 21:31 |
| SamYaple | inc0: right now we are using a tcp connection | 21:31 |
| inc0 | yeah, I checked that out later | 21:31 |
| SamYaple | i am about to move it _back_ to a socket | 21:31 |
| inc0 | forgot about it | 21:31 |
| inc0 | ok....why? | 21:31 |
| SamYaple | does that break our plans for host based libvirt? | 21:31 |
| SamYaple | because it should be a socket | 21:31 |
| SamYaple | faster more stable communication | 21:31 |
| SamYaple | being tcp is troublesome | 21:32 |
| inc0 | shouldn't break it unless we do some crazy stuff like fake socket for rsyslog | 21:32 |
| SamYaple | its still better to do interprocess communication via a socket, more secure rather than expose the network | 21:32 |
| sean-k-mooney | tcp also mean i could potentially connect from a remote host so the socket is slitghly more secure. | 21:32 |
| SamYaple | sean-k-mooney: yup | 21:33 |
| SamYaple | it was a temporary solution | 21:33 |
| SamYaple | but now what we have named volumes we are good to go inc0 | 21:33 |
| inc0 | so heres thing, I want to have libvirt out of container now...just we don't know how to do it yet | 21:33 |
| SamYaple | that wont be the default inc0 | 21:33 |
| SamYaple | but that can be used like that sure | 21:33 |
| inc0 | ok, I want an option | 21:33 |
| SamYaple | thats what were talkign about | 21:33 |
| SamYaple | you can do that _right_ _now_ with a few tweaks to a config and one task conditional | 21:34 |
| SamYaple | so the only thing we need is one task conditional | 21:34 |
| inc0 | what I am concerned about tho is upgrade plan | 21:34 |
| SamYaple | which is unrelated to this | 21:34 |
| inc0 | so in Liberty it was tcp | 21:34 |
| SamYaple | no it wasnt | 21:34 |
| SamYaple | it was socket in liberty | 21:34 |
| inc0 | ah, then even better | 21:34 |
| sean-k-mooney | with the pid and network namespaces disabled what is the advantage of moving libvirt out of the container may i ask? | 21:34 |
| inc0 | yeah I forgot when drop root happened | 21:34 |
| SamYaple | sean-k-mooney: theres a problem with contaienr restarts killing vms right now | 21:35 |
| SamYaple | sean-k-mooney: but also people are scared of libvirt in a container too | 21:35 |
| inc0 | sean-k-mooney, so currently, and that annoys me, docker dies, or restarts, vms restarts | 21:35 |
| inc0 | that's bad | 21:35 |
| SamYaple | inc0: thats not true actually! | 21:35 |
| sean-k-mooney | SamYaple: oh ok. ya i can see that | 21:35 |
| SamYaple | test with qemu instances | 21:35 |
| SamYaple | mine are sticking around | 21:35 |
| SamYaple | its wierd | 21:35 |
| inc0 | so if you kill -9 docker, they keep runing | 21:35 |
| *** achanda has quit IRC | 21:35 | |
| inc0 | if you service docker stop, they die | 21:36 |
| inc0 | but when you kill -9 and then start docker | 21:36 |
| SamYaple | seriously though inc0, no docker daemon restart and my qemu processes are sticking around | 21:36 |
| inc0 | docker on startup destroys | 21:36 |
| sean-k-mooney | well with the pid namespace disabled the qemu instances liftime should not be tied to the container right | 21:36 |
| inc0 | the pids | 21:36 |
| SamYaple | sean-k-mooney: the docker-devs track the pid and kill it | 21:36 |
| SamYaple | inc0: seriously though, test actually instances | 21:36 |
| inc0 | I'll test that out more for instances, but after you start docker, they're still there? | 21:36 |
| SamYaple | my qemu instance stick around | 21:36 |
| SamYaple | they are | 21:37 |
| inc0 | I will, that's strange | 21:37 |
| inc0 | but would make me happy | 21:37 |
| SamYaple | they aren't even zombied | 21:37 |
| SamYaple | but it makes no sense so im worried | 21:37 |
| inc0 | it does not, true | 21:37 |
| inc0 | makes me worried and hopeful | 21:37 |
| SamYaple | i can `docker rm -v -f nova_libvirt` and they stick around | 21:37 |
| sean-k-mooney | i can test it now if you like. just restart docker right | 21:37 |
| *** CheKoLyN has quit IRC | 21:38 | |
| inc0 | SamYaple, does libvirt reconnects to them after restart? | 21:38 |
| inc0 | because if it does...it's WONDERFUL! | 21:38 |
| sean-k-mooney | ya i restarted docker and the qemu instaces are still running fine | 21:38 |
| SamYaple | inc0: it did if the volume underneath still existed. i havent tested this time around | 21:39 |
| SamYaple | i only saw this because i had 20 qemu processes running and no active kolla install | 21:39 |
| inc0 | so if we back it up with ceph | 21:39 |
| sean-k-mooney | inc0: so after restarting docker i can enter the nova_libvirt contaier and the qemu instaces still show up with virsh list | 21:40 |
| inc0 | thats probably best message I got today | 21:40 |
| SamYaple | inc0: dont get your hopes up | 21:40 |
| inc0 | but as you said SamYaple ...it does not make sense | 21:40 |
| SamYaple | this is no proof | 21:40 |
| SamYaple | im just saying whats happened | 21:41 |
| sean-k-mooney | true | 21:41 |
| inc0 | yeah I know | 21:41 |
| SamYaple | inc0: just test it out for yourself | 21:41 |
| SamYaple | full deploy | 21:41 |
| SamYaple | launch instance | 21:41 |
| SamYaple | full tear down | 21:41 |
| SamYaple | qemu process still around | 21:41 |
| inc0 | that's what I'm doing now | 21:41 |
| *** clayton has joined #kolla | 21:43 | |
| SamYaple | ahhhhh i know whats happening | 21:43 |
| SamYaple | ok. so the nova_libvirt container is breaking. this is triggering that same bug in docker they "fixed" | 21:43 |
| SamYaple | but its ok because we can use it | 21:43 |
| *** achanda has joined #kolla | 21:43 | |
| sean-k-mooney | actully since ye are talking about libvirt can i ask a quick question. | 21:47 |
| SamYaple | inc0: we may have something here.... but its not good | 21:48 |
| SamYaple | inc0: its very hackish. and requires a docker daemon restart to resolve the hack | 21:48 |
| inc0 | SamYaple, docker restart is exactly what we need to do during upgrade | 21:49 |
| inc0 | so if it can resolve a hack, with new version, and it will keep going, that's what we need | 21:49 |
| SamYaple | i know | 21:49 |
| SamYaple | but i mean anytime the container changes | 21:50 |
| inc0 | what's happening low level tho? can you explain? | 21:50 |
| SamYaple | the docker daemon is trying to kill libvirt | 21:50 |
| SamYaple | it does kill it | 21:50 |
| SamYaple | but the nova_libvirt container still says running | 21:50 |
| SamYaple | on daemon stop qemu still around | 21:50 |
| SamYaple | on daemon start qemu still around | 21:50 |
| SamYaple | nova_libvirt container is corrupt | 21:50 |
| SamYaple | on recreate nova_libvirt would reattach to qemu (working on recreate) | 21:51 |
| inc0 | so qemu processes are not libvirt child processed I think | 21:51 |
| SamYaple | will require a kolla_docker tweak if working | 21:51 |
| SamYaple | they are not no | 21:51 |
| SamYaple | they never have been | 21:51 |
| inc0 | and all pid=host are there, so docker doesn't really track qemu | 21:51 |
| inc0 | so that's what probably happends | 21:51 |
| inc0 | and you restart libvirtd in docker, it get to /run and see all the qemu processes docker didn't know about | 21:51 |
| sean-k-mooney | with pid=host the qemu process should be parented to the host pid 1 not to the continers first process right | 21:52 |
| SamYaple | inc0: i can't restart docker daemon :( | 21:52 |
| SamYaple | ERRO[0001] Error during layer Store.Cleanup(): device or resource busy | 21:52 |
| SamYaple | (I am on dev version of docker though) | 21:52 |
| inc0 | I need t go to meeting, fresh containers are building, will test that out after I come back home | 21:53 |
| *** CheKoLyN has joined #kolla | 21:53 | |
| *** salv-orlando has joined #kolla | 21:54 | |
| inc0 | this behaviour does make sense, which makes me happy | 21:54 |
| SamYaple | inc0: but we cant reproduce it outside of libvirt | 21:54 |
| SamYaple | thats the issue | 21:54 |
| SamYaple | we need to understand it to ensure it doesn't get "fixed" | 21:54 |
| inc0 | outside of libvirt? | 21:54 |
| SamYaple | right. i can't make a process stick around manually | 21:54 |
| inc0 | ahh | 21:54 |
| inc0 | well, try to run container with pid host with whatever in it | 21:55 |
| inc0 | and then run some different process while inside it | 21:55 |
| inc0 | not connected to original, tracking one | 21:55 |
| inc0 | and try to destroy container, see if process sticks | 21:55 |
| *** thumpba has quit IRC | 21:55 | |
| sean-k-mooney | if you run a container with pid=host and bash as the first process the run a command and send it to the backgroud with disown and kill the container i think it will stay around | 21:56 |
| inc0 | and that's pretty much what happens I think | 21:56 |
| inc0 | because qemu has no connection to libvirt pid-wise | 21:56 |
| inc0 | and we run libvirt as tracked pid, not qemu | 21:56 |
| *** salv-orl_ has quit IRC | 21:56 | |
| inc0 | afk | 21:57 |
| SamYaple | sean-k-mooney: it will not | 21:57 |
| SamYaple | we did lots of tests the other day | 21:57 |
| SamYaple | inc0: that isnt' whats happening | 21:57 |
| SamYaple | this si definetely a bug in docker | 21:57 |
| *** ftcpops has quit IRC | 22:00 | |
| *** alyson_ has quit IRC | 22:02 | |
| *** jtriley has quit IRC | 22:03 | |
| *** sdake_ has quit IRC | 22:06 | |
| *** neilus has joined #kolla | 22:12 | |
| *** neilus has quit IRC | 22:16 | |
| *** achanda has quit IRC | 22:16 | |
| *** dims_ has joined #kolla | 22:17 | |
| *** bmace has quit IRC | 22:18 | |
| *** bmace has joined #kolla | 22:18 | |
| *** dims has quit IRC | 22:21 | |
| *** dwalsh has joined #kolla | 22:26 | |
| SamYaple | inc0: good news bad news situation here | 22:27 |
| SamYaple | which do you want first? | 22:27 |
| *** dims_ has quit IRC | 22:27 | |
| *** farid_ has joined #kolla | 22:27 | |
| *** tzn has quit IRC | 22:30 | |
| *** v1k0d3n has quit IRC | 22:31 | |
| *** sdake has joined #kolla | 22:32 | |
| *** farid_ has quit IRC | 22:34 | |
| *** CheKoLyN has quit IRC | 22:36 | |
| *** sdake has quit IRC | 22:44 | |
| *** dwalsh has quit IRC | 22:45 | |
| *** CheKoLyN has joined #kolla | 22:49 | |
| *** dave-mccowan has quit IRC | 23:08 | |
| *** absubram has joined #kolla | 23:09 | |
| *** achanda has joined #kolla | 23:09 | |
| inc0 | SamYaple, bad ones | 23:09 |
| *** pbourke has quit IRC | 23:11 | |
| *** achanda has quit IRC | 23:12 | |
| inc0 | SamYaple, well, tomorrow, Polina will kill me if I won't go back home soon | 23:13 |
| inc0 | cyas guys' | 23:13 |
| *** inc0 has quit IRC | 23:14 | |
| *** pbourke has joined #kolla | 23:15 | |
| *** Marga_ has quit IRC | 23:23 | |
| asalkeld | anything changed in the kolla-ansible modules? user/endpoint registration and the sanity one? | 23:27 |
| *** neilus has joined #kolla | 23:28 | |
| asalkeld | TypeError: Value of unknown type: <type 'exceptions.AttributeError'>, 'str' object has no attribute 'items' | 23:28 |
| asalkeld | super helpful message | 23:28 |
| asalkeld | http://paste.openstack.org/show/485902/ | 23:31 |
| *** farid has joined #kolla | 23:32 | |
| *** dave-mccowan has joined #kolla | 23:40 | |
| *** iNeilus has joined #kolla | 23:41 | |
| *** inc0 has joined #kolla | 23:41 | |
| inc0 | SamYaple, around? | 23:41 |
| *** stvnoyes has quit IRC | 23:41 | |
| *** stvnoyes has joined #kolla | 23:42 | |
| *** blahRus has quit IRC | 23:42 | |
| *** alisonh has quit IRC | 23:43 | |
| *** mimizone has left #kolla | 23:44 | |
| *** neilus has quit IRC | 23:45 | |
| *** yuanying has quit IRC | 23:54 | |
| *** yuanying has joined #kolla | 23:56 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!