Thursday, 2016-11-17

*** schwicht has joined #openstack-kolla00:01
jascott1there is actually good beer in SA if you have a sherpa to help you locate it...00:01
*** absubram has quit IRC00:02
srwilkersyes, agreed00:03
*** cnf has quit IRC00:03
srwilkerstexans idea of good beer vastly differs from mine00:03
*** krtaylor has joined #openstack-kolla00:04
srwilkerssaint louis has a bunch of micro breweries that are doing really well. one of my favorites here has a russian imperial stout blended with a honey mead almost. perfect for this time of year to keep you warm ;p00:05
jascott1nice00:05
*** Pavo has quit IRC00:07
*** Pavo has joined #openstack-kolla00:08
jascott1srwilkers: do you have any teammates in OR?00:08
*** chas has quit IRC00:09
*** chas has joined #openstack-kolla00:09
srwilkersjascott1: not that im aware of, but i volunteer as tribute if our company ever wants to open a development shop up that way00:10
jascott1:)00:10
SamYaplesean-k-mooney: have you seen dasiy+kolla for using opnfv with kolla?00:10
SamYaplesean-k-mooney: i just stumbled on it while doing my opnfv research00:11
jascott1love it here. Portland has more than enough big city for me without the traffic and BS of Seattle00:11
inc0SamYaple, Sean lives in Ireland00:11
inc0so I'd bet he's deep into drinking by now00:11
SamYaplehe has irc logs. i dont expect an immedite response00:11
srwilkersyeah, i want to visit portland. ive always wanted to end up in that corner of the world if im stuck here in the states00:12
srwilkerssaint louis is very lackluster. expecting almost 80 degrees F tomorrow, then expecting possible snow this weekend00:13
srwilkersif you dont like the weather here, just wait a day00:13
jascott1hehe00:13
jascott1been to a couple concerts there back in "the day"00:13
*** chas has quit IRC00:14
*** hfu has joined #openstack-kolla00:14
*** msimonin1 has quit IRC00:14
jascott1it was alright.. preferred it to Cinci OH for sure00:14
jascott1my friend met Sebastian Bach while we were eating..00:15
inc0hmm..isn't he dead for like..300 years?00:15
jascott1haha00:15
jascott1pretty sure his career has been dead for 15+ years00:16
jascott1oh wait he was on Trailer Park Boys a few years ago00:16
inc0I dunno, Mass in B Minor is great00:16
jascott1lol00:16
jascott1( as performed by Skid Row)00:17
jascott1(now)00:17
srwilkershaha00:17
jascott1I had a stint in JCC as a music major when was a wee lad and got to participate in an award winning chorus at a competition. got to say, im not a religious person but those mass songs are powerful00:18
jascott1especially in a room designed for them with a couple hundred people singing00:18
*** zhurong has joined #openstack-kolla00:19
inc0https://www.youtube.com/watch?v=tdLCcQixNvg00:19
jascott1ha i got ATT commercial00:19
*** dave-mccowan has joined #openstack-kolla00:19
jascott1bah proxy is killing it for e00:20
jascott1me00:20
srwilkers;)00:21
*** tonanhngo has joined #openstack-kolla00:25
*** diogogmt has joined #openstack-kolla00:27
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Modify the letter case with 'True' in ansible yml file.  https://review.openstack.org/39813300:27
*** tonanhngo has quit IRC00:29
*** yingjun has joined #openstack-kolla00:33
*** yingjun has quit IRC00:33
*** yingjun has joined #openstack-kolla00:33
*** yingjun has quit IRC00:33
*** yingjun has joined #openstack-kolla00:34
jascott1inc0 did you see the Google Earth for Vive?00:34
jascott1https://vr.google.com/earth/00:34
inc0omfg00:35
jascott1have fun :)00:35
*** david-lyle_ is now known as david-lyle00:35
inc0jascott1, readiness probes looks good00:38
inc0should be easy enough:)00:39
inc0can you guys include it in helm from day 1?00:39
wirehead_I really wanted to hack on readiness probes really early in the getting-it-working-in-the-first-place stage.00:40
*** zhurong has quit IRC00:40
sdakewhere do readiness probes go - in containers?00:41
wirehead_What I should have done was accepted that a readieness probe that's really lame, perhapse even degenerate, will encourage a future contributor to improve it.00:41
wirehead_It's part of the pod definition.00:41
sdakewirehead_ right that would have been the correct approach00:41
inc0we need to have them in container somehow, but can be done with volumes00:41
srwilkers+1 wirehead_00:41
inc0that being said, I think it'll be useful even if backed into container00:41
inc0ansible could use it too00:42
wirehead_I was going to do it for MariaDB, but I got bogged down in making sure the mysql client had the right creds.00:42
wirehead_Yeah, the longer view suggests that kolla-ansible would also benefit from that infra.00:43
*** tonanhngo has joined #openstack-kolla00:45
kfox1111inc0: our deployments use readyness probes00:45
jascott1so far stackenetes mostly used httpGet / and openstack-helm has used socket00:45
kfox1111see kolla-kubernetes/services/common/common-deployment.yml.j200:46
jascott1ha theres a handful in kk8s00:46
jascott1ha/ya/s00:46
kfox1111yup.00:46
inc0cool00:46
inc0let's fill up the gaps00:46
kfox1111I think a fair amount of the remainders are things that should be converted to use more common code but just havent yet.00:47
kfox1111well... and the apache stuff...00:47
kfox1111that needs a preStop hook too...00:47
kfox1111just little things mostly.00:47
jascott1only ones that ppl seem to use exec for are rabbit and db's00:47
kfox1111benefits/drawbacks to both apraoaches.00:48
kfox1111httpGet works over the network, but only works with basic http00:48
*** eaguilar has quit IRC00:48
kfox1111exec runs a script in the container, so insanely flexible.00:48
inc0"#programming is being sad for money"00:48
kfox1111but doen't test the network unless you code it to.00:48
kfox1111I usually use exec for most of my other projects.00:48
*** cnf has joined #openstack-kolla00:49
*** tonanhngo has quit IRC00:49
jascott1do we need a minimum standard here or is it whatever works?00:49
kfox1111inc0: do you want to workflow the spec?00:50
kfox1111jascott1: good question.00:50
jascott1socket check just tells you 'something' is listening on that port so seems to lite00:50
kfox1111right.00:50
jascott1http could prob get us half way there00:50
kfox1111and only works for http.00:50
kfox1111but most openstack stuff is http00:50
kfox1111(or rabbitmq based)00:50
kfox1111but most of our stuff is already in deployments which have the check already.00:51
kfox1111the only ones I think that don't are keystone and horizon, which both need a bit of love in other ways too.00:51
kfox1111both should be converted to deployments,00:51
kfox1111but are a little different from the other deployments as they are using apache instad of raw python.00:51
kfox1111so they don't need the haproxy container and the prestop hook should be to tell apache to shutdown safelyl.00:52
kfox1111and we need readyness hooks for rabbit/mariadb too.00:52
kfox1111I don't think we need readyness hooks on the rpc services...00:53
inc0nah00:53
*** adrian_otto has quit IRC00:53
jascott1hmm00:53
kfox1111requests shoudl just queue up in rabbit till they start.00:53
inc0they will just start working00:53
jascott1we talked earlier about chart per daemon00:55
kfox1111inc0: do you want to workflow the spec or should I? and when?00:55
jascott1meaning we have deployment.yaml per daemon?00:55
kfox1111I'm thinking chart per microservice.00:55
kfox1111and then if folks want to orchestrate with a service package,00:55
kfox1111they can just add the microservice packages as deps.00:55
inc0kfox1111, do the honors00:55
jascott1microservice=api for openstack service?00:56
kfox1111inc0: k. I'll do at COB then. ~30 min.00:56
jascott1I know what mircoservices are00:56
jascott1but its kinda overloaded00:56
kfox1111jascott1: mostly one to one with k8s objects I think.00:56
kfox1111so its really easy to launch exactly the pices yhou want, if you want to go that route.00:56
jascott1right00:56
kfox1111I have a prototype that can do the same thing, where all the service objects are packaged together,00:57
kfox1111but you luanch only one microservice at a time out of it.00:57
jascott1so we have N different images for neutron pieces00:57
kfox1111which is kind of clever/works, but not sure its a good idea.00:57
jascott1is Chart N or less than N00:57
jascott1number of charts00:57
kfox1111very different usage then what helm sems to have been intended for.00:57
kfox1111a k8s object can be made of multiple containers.00:58
kfox1111so no 1-1 mapping.00:58
inc0kfox1111, but we don't want pod-per-microservice right?00:58
jascott1thats what Im getting at00:58
kfox1111object per microservice.00:58
inc0we can bundle couple containers in single pod00:58
kfox1111one microservice might launch multiple pods.00:58
jascott1dynamic pods contents?00:58
kfox1111for example00:58
kfox1111kolla-kuberntes has a daemonset object for neutron-l3-agent.00:59
kfox1111that one object can spawn 300 identicle pods,00:59
kfox1111each with a few containers per pod.00:59
jascott1right00:59
*** zhubingbing has joined #openstack-kolla01:02
kfox1111the pod has, the l3-agent container, as well as fluentd or heka,01:02
kfox1111as well as kubernetes-entrypoint as an init-container.01:02
kfox1111and a lot of our pods already use kolla-toolbox as an init container too.01:02
kfox1111I've mostly followed the pattern of,01:02
kfox1111one k8s object per microservice. so, like,01:02
kfox1111neutron-l3-agent gets one. neutron-openvwitch-agent gets one, neutron-server is one. etc.01:02
jascott1so im thinking deployment.yamls in templates, DaemonSets aside basically, would we have one chart and deployment.yaml for glance-api and another for glance-registry and both wrapped in glance chart?01:04
openstackgerritJoshua Harlow proposed openstack/kolla: Use itertools.chain and show how many files utime modified  https://review.openstack.org/39814501:05
kfox1111one chart for glance-api, another for glance-registry I think.01:05
jascott1ok thats what I thought you have been saying01:05
jascott1makes sense but might be case by case01:05
kfox1111I think we temlate out the build of charts, so we can share variables/chart.yaml code as much as possible.01:05
kfox1111yeah.01:05
jascott1like you said, some neutron needs to be in daemonsets01:06
kfox1111I've got a wip prototype here:01:06
kfox1111https://review.openstack.org/#/c/396296/01:06
kfox1111it actually works. :)01:06
kfox1111but has some cleanup to do. and still messing around with a few ideas to see what will fly.01:06
jascott1im def into this conversation but must run pickup my daughter brb!01:07
kfox1111kk01:07
*** inc0 has quit IRC01:10
*** duonghq has joined #openstack-kolla01:10
duonghqmorning guys01:10
openstackgerritLi Yingjun proposed openstack/kolla-ansible: Disable notifications for cinder when no consumer  https://review.openstack.org/39866901:12
srwilkersi think this is a solid start kfox1111.  been working on keystone/mariadb and think things are starting to become clear01:15
*** jtriley has joined #openstack-kolla01:16
*** eaguilar has joined #openstack-kolla01:16
kfox1111srwilkers: cool. :)01:16
kfox1111so if you want, you can base a ps on top of that wip one, and should be able to test out the keystone/mariadb helm packages in the gate pretty easily.01:17
*** dave-mccowan has quit IRC01:18
kfox1111in tests/bin/ceph-workflow.sh, just replace the kollakube res command for the template with a helm install variant.01:18
srwilkersalright, sounds good.  as soon as its ready, ill toss it up01:19
kfox1111cool. :)01:19
jascott1back01:25
kfox1111wb. :)01:26
kfox1111I gota head out in a few though. :/01:26
jascott1time! boo time!01:26
kfox1111so, it sounds like slack won't work for our normal traffic. :/01:27
kfox1111maybe we just use it to ping each other when needed, set the topic to recoommend people visit here?01:27
kfox1111closing the spec in about 5 min01:29
jascott1makes sense I guess. like it has been said, should make talking to k8s/helm ppl easier01:34
jascott1openstack overlords should carve out some budget to run RocketChat!01:35
jascott1kfox1111 so the point of defining the whole resource as a named template is for re-use?01:36
kfox1111I hadent heard of rocketchat yet. heard some good things about mattermost. gitlab seems to really like it.01:37
kfox1111jascott1: was a prototype.01:37
jascott1mm is written in golang which is cool but rocketchat is much further along last i checked (>6mos)01:37
kfox1111a way to use a single package to package multiple microservices, but still launch only one microservice at a time.01:38
kfox1111interesting. I"ll have to give rocketchat a look. :)01:38
kfox1111it was just an attempt to see what helm could/could not do.01:38
jascott1i ran several of them internally for my team last year and rocketchat won01:38
jascott1great work01:38
jascott1on the helm01:39
kfox1111thanks. It still needs some love, but helm held up to the abuse I sent its way, so can't be all bad. :)01:39
jascott1no one seems to be a fan of go tpl but it would be great not to have to jinja everything imo01:39
kfox1111yeah. go tmpl's if conditionals are weird.01:39
kfox1111its like a funky forth.01:40
jascott1ive been messing with go for several years but have never used their tpl01:40
kfox1111#if and foo bar ....01:40
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' to neccesary tasks (general roles)  https://review.openstack.org/39868201:40
jascott1cool thing you can do w golang, you can create a single page js app and embed it in the single go binary01:40
jascott1single page = react, angular etc01:41
kfox1111yeah....01:41
kfox1111though,01:41
kfox1111I'm kind of in the camp now that thinks, containers are the statically linked tool of the future,01:42
jascott1yup01:42
kfox1111rather then make language / platoform specific ones.01:42
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (default roles)  https://review.openstack.org/39868401:42
kfox1111so you can esentially do the same thing with anything, just in containers instead of go.01:42
jascott1cross platform compile is nice too. used it last night on entrypoint01:42
kfox1111kind of desolves one of go's main selling points.01:42
kfox1111ah. yeah.01:43
kfox1111that toolchain is nice. :)01:43
openstackgerritDuong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (all other roles)  https://review.openstack.org/39868501:43
jascott1i think my lack of openstack expertise is hurting me here but trying to visualize the whole set of charts01:44
jascott1but essentially its flat, we dont use requirements.yaml01:44
jascott1in most cases anyway01:44
*** zhangyufei has joined #openstack-kolla01:45
jascott1oh wait we do use requirements, because we will have wrapper charts for each service correct?01:46
kfox1111I really need to head out, but there may be a heiracry to it for those that want it.01:46
kfox1111that is one option, yes.01:46
jascott1k have a great evening01:46
kfox1111for those that want to use helm/kubernetes-entrypoint as their entire orchestration,01:46
kfox1111wrapping them up in a genreal openstack-package, with service-packages, then with all the microservice packages I think would work.01:47
kfox1111closing the spec....01:47
duonghqsdake, ping01:47
sdakesup duonghq01:47
pronsoo, how do one set virt_type in kolla?01:48
duonghqdo anybody setup linking between gerrit and launchpad of kolla-ansible?01:48
pronhttp://docs.openstack.org/developer/kolla/quickstart.html there is mentioned some majical nova-compute.conf  , wich i couldnt find01:48
pron;(01:48
* pron hides in cornee and cries01:48
duonghqpron,  http://docs.openstack.org/developer/kolla/advanced-configuration.html#openstack-service-configuration-in-kolla01:49
openstackgerritMerged openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture  https://review.openstack.org/39225701:50
duonghqsdake, or some migration error?, this bp: https://blueprints.launchpad.net/kolla-ansible/+spec/ansible-specific-task-become01:50
sdakeduonghq sorry dont understand the q01:51
*** zhurong has joined #openstack-kolla01:51
duonghqsdake, the bp is moved to kolla-ansible, but the tag in https://review.openstack.org/#/c/398682/ does not work01:52
pronduonghq: propably a reference to advaced config there in docs could help :P , but thx <301:52
sdakeduonghq you need to resubmit the review01:53
sdaketo the kolla-ansible repo01:53
sdakeunfortunately for our 12 pages of reviews in queue this is necessary01:54
sdakedont blame me :)01:54
duonghqsdake, the link I posted is on kolla-ansible repo already01:54
duonghqI abandoned the kolla ones01:54
duonghqcherry-pick all 3 ps :)01:54
jascott1pron did you find the solution?02:01
sdakeduonghq can i get back to you - have critical work emergency atm02:01
*** f13o has joined #openstack-kolla02:02
duonghqsdake, thank you02:04
*** Pavo has quit IRC02:07
*** Pavo has joined #openstack-kolla02:08
*** f13o has quit IRC02:09
jascott1kubecon videos are up: https://www.youtube.com/playlist?list=PLj6h78yzYM2PqgIGU1Qmi8nY7dqn9PCr402:19
duonghqthank jascott102:22
jascott1this might be useful.. combined stream of all logs in pod or filter using label https://github.com/dtan4/k8stail02:25
jascott1thanks reddit!02:25
pronjascott1: yes02:25
jascott1cool02:26
pronat leadt i think02:26
pronso, but now kolla errored out :P02:26
pronsomethign vip related, stops at mysql trying to connect tru vip02:27
pronbut what i see that i cant ping that vip address from any host on network except that network node where that vip lives02:28
jascott1doesnt sound good02:30
pronaye02:31
* pron is trying to figure out keepalived logs02:31
pronwhere to find em02:32
*** duonghq has quit IRC02:32
*** tonanhngo has joined #openstack-kolla02:34
*** tonanhngo has quit IRC02:38
*** g3ek has quit IRC02:38
*** fragatin_ has joined #openstack-kolla02:47
*** g3ek has joined #openstack-kolla02:48
*** dave-mccowan has joined #openstack-kolla02:49
*** duonghq has joined #openstack-kolla02:50
*** fragatina has quit IRC02:50
*** fragatin_ has quit IRC02:51
*** newmember has quit IRC02:54
*** fragatina has joined #openstack-kolla02:57
*** fragatina has quit IRC02:58
*** adrian_otto has joined #openstack-kolla02:59
pronhmmm i star to suspect that its another openstacks fault where i run this kolla in ;x02:59
*** Pavo has quit IRC03:01
*** jtriley has quit IRC03:01
*** Pavo has joined #openstack-kolla03:04
*** adrian_otto has quit IRC03:04
*** adrian_otto1 has joined #openstack-kolla03:04
*** Jeffrey4l has quit IRC03:11
*** srwilkers has quit IRC03:12
*** unicell has quit IRC03:14
*** mgiles has quit IRC03:15
*** coolsvap has joined #openstack-kolla03:15
*** fragatina has joined #openstack-kolla03:24
Pavowhy does murano show up in horizon when I enabled it for deploymnet?03:25
Pavodoesn't I mean03:25
*** fragatina has quit IRC03:29
Pavoanyone?03:32
jascott1sorry havent made it that far up the stack yet ;)03:33
*** dave-mccowan has quit IRC03:33
jascott1i would shell into horizon container and and check it out03:33
Pavowell I looked into kolla/ansible/roles/horizon/templates/local_settings and nothing in there about murano at all03:34
jascott1shell in and see if those bits even made it to the container03:35
*** tonanhngo has joined #openstack-kolla03:35
*** tonanhngo has quit IRC03:35
jascott1arent some horizon addons manual?03:36
Pavomurano shouldn't be manual03:36
*** jtriley has joined #openstack-kolla03:36
PavoI know it can be but if you enable it, there should be something in the json file that reads that its enabled in globals and turns that feature on inside horizon03:37
Pavoat least thats the logic I would assume it is03:37
jascott1is it this? https://github.com/openstack/murano-dashboard03:38
PavoI would assume yes03:38
jascott1maybe hack in that symlink and work backwards03:39
Pavoguess I could test something out with it03:42
Pavobut if there is an option currently in kolla to enable murano it should work unless its using murano in a different manner than I would assume it would03:43
*** tonanhngo has joined #openstack-kolla03:44
*** sdake_ has joined #openstack-kolla03:48
*** tonanhngo has quit IRC03:48
*** sdake has quit IRC03:50
sdake_pavo that is an often requested feature which is unimplented03:50
sdake_pavo recommend pinging pbourke - i think he has a solution but its not upstream03:50
sdake_pavo pretty sure pbourke would share, its a "we just rewrote the dockerfiles fo rthe 5th time" thing03:51
Pavolol ok03:52
Pavowill do03:52
Pavoping pbourke03:52
Pavonot bad resource usage03:54
Pavohttp://pastebin.com/1tWxjJPK03:54
Pavothats the controller, network and storage node resource usage from a tls deploymnet03:54
Pavocompute, storage node resource usage with tls deployment http://pastebin.com/7qXQFggZ03:56
sdake_my mortal form grows weary need sustinance03:59
sdake_later03:59
Pavodisk i/o usage on controller tops out around 1k K/s on average 700 K/s03:59
*** sdake_ has quit IRC04:00
Pavol8r04:00
*** Pavo has quit IRC04:07
*** Pavo has joined #openstack-kolla04:08
*** adrian_otto1 has quit IRC04:13
*** eaguilar has quit IRC04:16
*** mrhillsman has quit IRC04:29
openstackgerritMerged openstack/kolla: Changed the start script name in cinder-volume  https://review.openstack.org/39385804:34
openstackgerritMerged openstack/kolla: Fix cinder volume permission issue  https://review.openstack.org/39385904:35
*** tonanhngo has joined #openstack-kolla04:55
*** tonanhngo has quit IRC04:56
*** mottaman85 has joined #openstack-kolla04:57
*** unicell has joined #openstack-kolla05:23
*** sdake has joined #openstack-kolla05:33
*** sdake_ has joined #openstack-kolla05:37
*** liyifeng has joined #openstack-kolla05:37
*** sdake has quit IRC05:37
*** fragatina has joined #openstack-kolla05:47
*** fragatina has quit IRC05:52
*** chas has joined #openstack-kolla05:54
openstackgerritYongfeng Du proposed openstack/kolla-ansible: Fix /etc/kolla permission  https://review.openstack.org/39875305:57
*** chas has quit IRC06:01
*** chas has joined #openstack-kolla06:06
*** Pavo has quit IRC06:07
*** msimonin has joined #openstack-kolla06:08
*** tonanhngo has joined #openstack-kolla06:08
*** chas has quit IRC06:11
*** mottaman85 has quit IRC06:12
*** Pavo has joined #openstack-kolla06:12
*** tonanhngo has quit IRC06:13
*** msimonin has quit IRC06:14
*** haplo37_ has quit IRC06:20
sdake_pbourke around?06:26
sdake_any core reviewers about?06:27
sdake_coolsvap ping06:27
coolsvapsdake_: hey06:27
*** haplo37_ has joined #openstack-kolla06:29
sdake_Daviey around?06:30
*** chas has joined #openstack-kolla06:38
*** chas has quit IRC06:43
*** jtriley has quit IRC06:48
*** fragatina has joined #openstack-kolla06:49
*** fragatina has quit IRC06:54
bjolomorning06:55
bjolopbourke, i opened a ticket now https://bugs.launchpad.net/neutron/+bug/164230306:59
openstackLaunchpad bug 1642303 in neutron "neutron multiple external flat networks fails" [Undecided,New]06:59
bjoloopened as a neutron bug as well since i cant figure out if it is kolla or neutron misbehaving07:00
*** msimonin has joined #openstack-kolla07:01
*** DTadrzak has joined #openstack-kolla07:07
*** jtriley has joined #openstack-kolla07:26
*** msimonin has quit IRC07:26
*** jtriley has quit IRC07:34
*** tonanhngo has joined #openstack-kolla07:35
*** imcsk8 has quit IRC07:38
*** tonanhngo has quit IRC07:39
*** imcsk8 has joined #openstack-kolla07:42
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Modify the letter case with 'false' in ansible yml file.  https://review.openstack.org/39879807:44
*** f13o has joined #openstack-kolla07:45
*** rmart04 has joined #openstack-kolla07:45
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Modify the letter case in ansible yml file.  https://review.openstack.org/39879807:56
*** athomas has joined #openstack-kolla07:57
*** rmart04 has quit IRC08:06
*** Pavo has quit IRC08:07
*** Pavo has joined #openstack-kolla08:12
*** imcsk8 has quit IRC08:14
*** shardy_afk is now known as shardy08:15
*** imcsk8 has joined #openstack-kolla08:15
*** msimonin has joined #openstack-kolla08:21
*** f13o_ has joined #openstack-kolla08:27
*** jroll has quit IRC08:27
*** f13o has quit IRC08:29
*** rmart04 has joined #openstack-kolla08:33
*** matrohon has joined #openstack-kolla08:38
*** jroll has joined #openstack-kolla08:41
*** egonzalez90 has joined #openstack-kolla08:47
egonzalez90morning!08:48
bjolomorning egonzalez9008:49
*** duonghq has quit IRC08:58
*** duonghq has joined #openstack-kolla08:59
*** duonghq has quit IRC09:02
zhubingbingmorning!09:07
*** gfidente has joined #openstack-kolla09:07
*** gfidente has joined #openstack-kolla09:07
*** portdirect_away_ has quit IRC09:08
magicboizmorning09:11
*** gfidente has quit IRC09:11
magicboizI'm facing some strange docker behaviour09:11
magicboiz10.20.0.10:4000/kolla/ubuntu-binary-ceph-osd:3.0.1        "kolla_start"       4 minutes ago       Restarting (1) About a minute ago                       ceph_osd_209:11
magicboizhas anyone seen this before?09:12
magicboizcontanier is always restarting09:12
*** fragatina has joined #openstack-kolla09:12
*** DaveTurner has quit IRC09:14
*** gfidente has joined #openstack-kolla09:16
*** gfidente has joined #openstack-kolla09:16
*** fragatina has quit IRC09:18
egonzalez90magicboiz: what does docker logs ceph_osd_2 says?09:18
*** zhurong has quit IRC09:20
magicboizegonzalez90: http://paste.openstack.org/show/589548/09:20
*** zhurong has joined #openstack-kolla09:20
*** Serlex has joined #openstack-kolla09:22
egonzalez90magicboiz: osd_1 is in the same machine? Is it working?09:22
magicboizegonzalez90: no OSD is working in the environment, they are all "restarting" but kolla-ansible deploy didn't detect it.....09:23
*** jtriley has joined #openstack-kolla09:25
egonzalez90sounds like a permission issue09:25
liyifeng** ERROR: osd init failed: (1) Operation not permitted09:25
magicboizyes, but why?? after some (deploy/destroy) I started to get this problem....09:26
liyifengperhaps selinux?09:27
egonzalez90did you run cleanup-containers and cleanup-host?09:27
*** YuYangWang has quit IRC09:33
magicboizI always run "destroy"09:34
egonzalez90also, partitions need to be removed before re deploying an osd and make the label for kolla osd09:34
magicboizBu I did notice that "destroy" does 2 things I don't like:09:34
*** jtriley has quit IRC09:35
magicboiz1. It leaves ceph-osd-x dir mounted09:35
egonzalez90not anymore ;)09:35
magicboiz2. the partition label is changed, it's KOLLA_CEPH_DATA_1_J + KOLLA_CEPH_DATA_1 and not KOLLA_CEPH_OSD_BOOTSTRAP09:36
egonzalez90https://review.openstack.org/#/c/380028/09:36
egonzalez90that partition label os for kolla know what disk use to bootstrap osd09:37
portdirect_awaymorning09:37
*** portdirect_away is now known as portdirect09:37
egonzalez90otherwise, every time you deploy will bootstrap again those disks09:37
egonzalez90pbourke: around?09:38
*** yingjun has quit IRC09:38
magicboizproblem solved09:38
magicboizWorkaround: umount + restore partition label on every osd node09:39
magicboiznow, I don't have any osd container in restarting state....09:39
magicboiz10.20.0.10:4000/kolla/ubuntu-binary-ceph-osd:3.0.1        "kolla_start"       4 minutes ago       Up 4 minutes                            ceph_osd_509:39
*** f13o_ has quit IRC09:40
*** f13o has joined #openstack-kolla09:40
*** schwicht has quit IRC09:41
egonzalez90you created the lables the first time you deployed right?09:42
egonzalez90*labels09:42
egonzalez90as said before, KOLLA_CEPH_OSD_BOOTSTRAP label is used to know which devices are going to be used as osd, then are labeled as KOLLA_CEPH_DATA*09:43
magicboizegonzalez90: yes, as doc http://docs.openstack.org/developer/kolla/ceph-guide.html states, I did tag my disks with parted09:46
egonzalez90but when re deployed didn't or did you?09:47
*** openstackgerrit has quit IRC09:48
*** openstackgerrit has joined #openstack-kolla09:48
pbourkeegonzalez90: morning09:51
bjolohmm just discovered a regression09:51
bjolohttps://bugs.launchpad.net/kolla/+bug/164252109:51
openstackLaunchpad bug 1642521 in kolla "kolla-ansible fails on dedicated tunnel_interface" [Undecided,New]09:51
pbourkeegonzalez90: hope you're surviving the rain :p09:51
egonzalez90snow ATM09:52
pbourkeah yeah I heard that actually09:52
portdirectjust been reading through the logs: Pavo/sdake: I've been using nova-docker quite a bit, but it pretty much stalled at mitaka (unless someones picked it up): I never managed to get it to work with nova beyond about half way through the libery dev cycle. Also re muranodash-board: you pretty much need to be running the db caching backend (and memcached if you want any semblance of performance) in django for it to function: so thats worth noting when09:52
portdirectoking to deploy it.09:52
magicboizegonzalez90: when re-deploying, I didn't re-tag them09:53
bjolopbourke, https://bugs.launchpad.net/neutron/+bug/164230309:54
openstackLaunchpad bug 1642303 in kolla "neutron multiple external flat networks fails" [High,Triaged]09:54
magicboizegonzalez90: at some point, the deploy/destroy game changed the disks in some way that posterior deployment failedf09:54
magicboizfailed09:54
bjolopbourke, but i found something odd in the docs http://docs.openstack.org/newton/networking-guide/deploy-ovs-provider.html09:54
egonzalez90thats the problem, every time an osd is destroyed, need to be re labeled. if not will use  old ceph config and cant join cluster09:54
portdirect(need coffe - I meant newton not liberty...09:54
magicboizegonzalez90: so now, manually, I have umounted and the re-tagged with parted09:54
egonzalez90as far as i know, deploy baremetals does not label disks09:55
bjolopbourke, they still specify network_vlan_ranges = provider09:55
bjoloeven though they are not using vlans09:55
pbourkebjolo: "The provider value in the network_vlan_ranges option lacks VLAN ID ranges to support use of arbitrary VLAN IDs.09:57
pbourke"09:57
pbourkenot sure what that means...09:57
bjolopbourke, not me either?!09:58
bjolothats what i mean by odd09:58
*** zhurong has quit IRC10:02
*** hfu has quit IRC10:03
*** portdirect has quit IRC10:04
bjolopbourke, did you say that they had found some issues internally with this as well?10:05
*** ccesario has joined #openstack-kolla10:06
*** harbor has joined #openstack-kolla10:06
*** harbor is now known as portdirect10:06
*** Pavo has quit IRC10:07
pbourkebjolo: i thought they did but think it was something else... Tell you what, I'll try setting this up today10:10
pbourkebjolo: see if I can reproduce10:10
*** Pavo has joined #openstack-kolla10:12
bjolopbourke, that would be great to get the bug confirmed or not10:12
*** shardy is now known as shardy_mtg10:16
*** harbor has joined #openstack-kolla10:20
*** harbor is now known as portdirect_away_10:20
*** portdirect has quit IRC10:23
*** zhangyufei has quit IRC10:23
*** zhangyufei has joined #openstack-kolla10:25
*** harbor has joined #openstack-kolla10:25
*** harbor is now known as portdirect10:25
*** portdirect_away_ has quit IRC10:34
*** hieulq has quit IRC10:36
*** fragatina has joined #openstack-kolla10:42
*** strigazi_AFK is now known as strigazi10:42
*** fragatina has quit IRC10:47
*** shardy_mtg is now known as shardy11:00
*** zhubingbing has quit IRC11:02
*** ppalacios has joined #openstack-kolla11:03
*** khamtamtun has joined #openstack-kolla11:06
*** fragatina has joined #openstack-kolla11:07
*** fragatina has quit IRC11:12
*** khamtamtun has quit IRC11:12
*** kamtamtun has joined #openstack-kolla11:14
*** DTadrzak has quit IRC11:21
sean-k-mooneySamYaple: hi sam yes i did see daisy+kolla in opnfv however i have never seen them engage here so as far as i know its an independet effort by daisycloud.11:22
*** ppalacios has quit IRC11:25
*** DTadrzak has joined #openstack-kolla11:29
sdake_pavo if you want to see some perf data11:30
sdake_pavo you about?11:30
sdake_pavo i did a brownbag on our osic scale testing11:30
*** athomas has quit IRC11:32
pbourkebjolo: are you deploying in libvirt guests or baremeteal?11:34
sean-k-mooneysdake_: oh at the summit? was it recorded? im still going through the videos but that sounds like one i should add to my list11:35
sdake_sean-k-mooney indeed it was11:35
sean-k-mooneysdake_: cool good to know11:35
sdake_sean-k-mooney https://www.youtube.com/watch?v=HsMhEjcdb_Q&list=PL2rC-8e38bUWityOA6O8RcqBU-aHvhVdR11:36
sdake_about 50-70 people in the audience11:36
sdake_it was a little chaotic11:36
sdake_it was in open space11:37
sdake_rather then a room11:37
sean-k-mooneysdake_: cool i was just looking for it on the youtube page11:39
sdake_ya bar graph11:42
sdake_fubared that one ;)11:42
*** zhubingbing_ has joined #openstack-kolla11:44
*** imcsk8 has quit IRC11:45
*** kamtamtun has quit IRC11:45
*** imcsk8 has joined #openstack-kolla11:47
sdake_only had 10 mins to work with11:48
sdake_jeffrey4l helped produce the slides as well :)11:48
sdake_I think I may ahve left him out of the credits11:48
sdake_which is shameful11:48
sdake_oh well, not perfect ;(11:48
sean-k-mooney40% of the 37% cpu time spent on the networkstack is an interesting datapoint11:48
sdake_right11:48
sean-k-mooneythe controlers were also the network nodes correct?11:49
sdake_right11:49
*** eaguilar has joined #openstack-kolla11:49
*** hfu has joined #openstack-kolla11:49
sdake_controller nodes definately I/O bound11:49
sdake_ceph data shows either ceph can't perform at 10gig as well11:50
sdake_not either11:50
bjolopbourke, bare metal11:50
sdake_well either ceph can't perform or the network stack is pokey11:50
sean-k-mooneysdake_: i think going to jumboframes would have helped ceph11:50
sdake_sean-k-mooney keen point - we didn't deploy with jumbo frames although I would have liked to11:50
sdake_also we had bonding in the mix11:50
sdake_with vlans11:50
sdake_all that crap hurts perf11:51
sdake_the bottom line is the performance was fantastic from my pov11:51
sdake_and kolla worked well under load11:51
sdake_and tls isn't too expensive in terms of application performance11:51
sean-k-mooneyyep i think it was a very positive test run11:51
sdake_its not like a cherrypicked data either11:52
*** kamtamtun has joined #openstack-kolla11:52
sdake_if i had cherrypicked data, i'd probably have left the slide about ES/heka11:52
sdake_out11:52
sean-k-mooneysdake_: do you know if a repeat is on the cards for pike cycle?11:52
sdake_sean-k-mooney inc0 is heading that up11:52
sdake_sean-k-mooney inc0 i think needs to produce a whitepaper or soething for osic11:53
sdake_i'm not sure what his agreement was with osic11:53
*** kamtamtun has quit IRC11:53
sean-k-mooneycool it would be nice to see  16.04 and centos next time, i know time just was not on our side this time11:53
sdake_its not the lash they fear, its my divine power ;)11:53
sdake_sean-k-mooney it took a long time to figure out how to bootstrap the nodes11:53
sdake_sean-k-mooney i was really keen to see centos comparison11:54
sean-k-mooneywell at least nextime it should go more smothly as people can build on the experice for this run. all in all i think it a good indicator of the maturaty of kolla in terms of deployment time and performance11:55
sean-k-mooneysdake_: by the way i agree with you endorcment of ceph. one question though have you ever tried the docker rbd volume dirver?11:57
sean-k-mooneysdake_: http://ceph.com/planet/getting-started-with-the-docker-rbd-volume-plugin/11:58
sean-k-mooneymy last kolla cloud died because i ran out of disk space on 2 of my controllers and bad things happened.11:59
sean-k-mooneyim debating about installin ceph nativly on the host and then configuring kolla with external ceph  so that i can use the rbd driver to have all my container volumes be backed by ceph too12:00
*** zhubingbing_ has quit IRC12:01
*** zhugaoxiao has quit IRC12:01
*** zhubingbing_ has joined #openstack-kolla12:01
*** zhugaoxiao has joined #openstack-kolla12:01
*** ppalacios has joined #openstack-kolla12:02
*** jtriley has joined #openstack-kolla12:02
*** kamtamtun has joined #openstack-kolla12:04
sdake_sean-k-mooney agree12:05
sdake_sean-k-mooney sort of disappointed again about centos, especially since the team worked miracles to get centos deployed12:05
*** kamtamtun has quit IRC12:05
sdake_and we just basically ran out of time to run the tests12:05
sdake_not just deployed but configured12:06
*** jtriley has quit IRC12:06
sdake_the problem is each test case took 8+ hrs to run12:06
sdake_and it took 3 weeks to get the cloud into a reliablly deployed state12:06
sdake_and get the test cases all setup12:06
sdake_if we had proper prepration12:06
sdake_we would have avoided piss poor performance :)12:06
sdake_realistically we could do xenail and centos with same data gathering in about a week now12:07
sdake_possiblye 1.5 weeks12:07
*** DTadrzak has quit IRC12:07
*** Pavo has quit IRC12:07
sean-k-mooneyqoute sdake:" we could do .. in a week now" :)12:08
sdake_not kidding12:08
sdake_with same team12:08
sdake_the hard part is deployment12:08
*** fragatina has joined #openstack-kolla12:08
sdake_getting the bare metal deployed i mean12:08
sdake_after that the tests are all scripted12:08
*** zhangyufei has quit IRC12:09
sdake_everything is nearly automated12:09
sdake_it needs a watcher12:09
sdake_someone to look after it and run the next test12:09
sdake_and set it up12:09
sdake_but that takes about 10-20 minutes12:09
sdake_the bare metal deployment is a huge pain in the ass12:09
sdake_bifrost wasn't ready at teh time12:10
sean-k-mooneyno unfrotunetly it wasnt12:10
sdake_same team could use same tools to do the bare metal deploy in about 4-8 hrs12:10
sdake_sean-k-mooney no criticism12:10
sdake_sean-k-mooney just a statement of fact ;)12:10
sdake_sean-k-mooney i'm not sure it would have made things easier12:10
sdake_it would actually probably have complicated matters12:10
sdake_since the code was fairly green12:11
sdake_we did use the host setup12:11
sdake_that works pretty well12:11
sdake_worked pretty well12:11
sdake_haven't tried it lately :)12:11
*** Pavo has joined #openstack-kolla12:12
*** papacz has joined #openstack-kolla12:13
*** fragatina has quit IRC12:13
pbourkebjolo: so I have two external networks, I've created a network on one and booted a VM on it. So far can't ping/ssh but that could be any number of reasons12:15
pbourkebjolo: so dont know if related yet or not12:15
*** sp_ has quit IRC12:19
*** awiddersheim has quit IRC12:19
*** mgoddard_ has joined #openstack-kolla12:20
*** mgoddard has quit IRC12:23
*** zhurong has joined #openstack-kolla12:24
*** ppalacios has quit IRC12:28
*** DTadrzak has joined #openstack-kolla12:30
*** dave-mccowan has joined #openstack-kolla12:31
*** ppalacios has joined #openstack-kolla12:38
sean-k-mooneysdake_: i used it a little while ago. there are som usablity tweek i have been meaning to make and some addtions such as supprot for proxy servers but it more or less did what i wanted.12:41
sdake_sean-k-mooney referecing which?12:42
*** ppalacios has quit IRC12:42
*** schwicht has joined #openstack-kolla12:46
*** mgiles has joined #openstack-kolla12:48
*** liyifeng has quit IRC12:50
*** srwilkers has joined #openstack-kolla12:50
*** chopmann has joined #openstack-kolla12:51
chopmannhello everyone12:52
sean-k-mooneysdake_: the host playbook that installs the deps for kolla12:52
sdake_right - that definateldy works12:52
sdake_I think it needs some additions around bonding12:53
sdake_and networking setup in general12:53
sean-k-mooneysdake_: yep i would love to add that12:53
sean-k-mooneyalso i was thinking it might be nice for it to be able to reuse the ceph or ovs deployment scrtips to set those up ahead of the main kolla run12:54
sdake_sean-k-mooney honeslty too tired to think12:54
sdake_been up tfor 24 hrs12:54
sdake_goign to crash for a couple hours12:54
*** msimonin has quit IRC12:54
sdake_then have all day meeting12:54
* sdake_ groans12:54
sean-k-mooneyno worries catch you later12:54
srwilkersrip sdake_12:59
*** msimonin has joined #openstack-kolla12:59
*** duonghq has joined #openstack-kolla13:00
sdake_srwilkers no shit13:03
*** ppalacios has joined #openstack-kolla13:03
sdake_srwilkers hoepfully i wake up tomorrow :)13:03
*** msimonin has quit IRC13:03
sdake_rather today13:03
sdake_3 hrs for powernap13:03
srwilkersim not envious13:04
*** msimonin has joined #openstack-kolla13:05
*** schwicht has quit IRC13:05
sdake_only have to make it to 2pm today13:06
sdake_then can ptfo for 12 hrs ;)13:06
sdake_ok i'm off13:06
sdake_not gettinga  powernap on irc13:06
*** ppalacios has quit IRC13:10
chopmanndoes anyone got a blog-post tutorial for a mitaka --> newton upgrade? :-)13:10
*** lamt has joined #openstack-kolla13:14
duonghqsbezverk, do you notice that kubeadm master keep writing into disk?13:20
duonghqor it's my setup problem?13:20
*** athomas has joined #openstack-kolla13:23
*** zhongshengping has joined #openstack-kolla13:25
srwilkerssilly question.  wheres the ansible code live now? cant find it on github13:26
zhongshengpinggit clone https://git.openstack.org/openstack/kolla-ansible13:26
zhongshengpingkolla-ansible is up! All ansible changes goes there now13:26
*** zhurong has quit IRC13:28
srwilkersthanks mate13:28
*** athomas has quit IRC13:28
*** zhurong has joined #openstack-kolla13:28
*** sdake_ has quit IRC13:29
*** Jeffrey4l has joined #openstack-kolla13:30
*** eaguilar has quit IRC13:31
*** eaguilar has joined #openstack-kolla13:34
*** hfu has quit IRC13:35
*** rhallisey has joined #openstack-kolla13:37
openstackgerritzhongshengping proposed openstack/kolla-ansible: Deprecate scheduler_max_attempts option in nova  https://review.openstack.org/39901113:38
*** vagrant` has joined #openstack-kolla13:42
*** zhurong has quit IRC13:43
*** zhurong has joined #openstack-kolla13:45
*** msimonin has quit IRC13:46
*** jheroux has joined #openstack-kolla13:46
*** fguillot has joined #openstack-kolla13:46
*** msimonin has joined #openstack-kolla13:46
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate  https://review.openstack.org/39850113:47
*** narasimah_SV has joined #openstack-kolla13:48
*** zhongshengpin has joined #openstack-kolla13:48
*** zhongshengping has quit IRC13:48
*** vagrant` has quit IRC13:48
narasimah_SVhttp://paste.openstack.org/show/589588/13:49
narasimah_SVgnocchi containers are unable to read the ceph keyring file13:49
narasimah_SVi placed that file in /etc/kola/config folder and also in gnochi-api folder too13:50
narasimah_SVcan anyone help what exactly I am missing here13:50
*** zhongshenping has joined #openstack-kolla13:50
*** zhongshengpin has quit IRC13:51
*** schwicht has joined #openstack-kolla13:52
*** zhongshenping has quit IRC13:53
*** RichWellum has joined #openstack-kolla13:53
*** zhongshenping has joined #openstack-kolla13:53
*** zhongshenping has quit IRC13:56
*** RichWellum has quit IRC13:56
*** sdake has joined #openstack-kolla13:56
egonzalez90narasimah_SV: the file is not copied to /etc/gnocchi folder in controller nodes13:57
*** RichWellum has joined #openstack-kolla13:57
egonzalez90there is no task to do that in ansible/roles/gnocchi/task/config.yml13:57
*** diogogmt has quit IRC13:58
sdakewell that sucked14:02
sdakeno powernap14:02
sdakesup folks14:02
sdakei guess i'm awake for the day until 2pm pst ;)14:02
*** schwicht has quit IRC14:03
duonghqsdake, did you sleep more when you younger?14:03
*** sdake_ has joined #openstack-kolla14:03
*** mgoddard_ has quit IRC14:06
*** mgoddard has joined #openstack-kolla14:06
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate  https://review.openstack.org/39850114:06
*** sdake has quit IRC14:07
*** Pavo has quit IRC14:07
narasimah_SVegonzalez90: this ius mentioned in external_ceph.yml file14:08
*** liyifeng has joined #openstack-kolla14:08
openstackgerritJeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment  https://review.openstack.org/39850614:09
*** fragatina has joined #openstack-kolla14:10
egonzalez90narasimah_SV: sorry, didn't reallize there was an external task file14:10
narasimah_SVok14:10
narasimah_SVin gnocchi component templates it is mentioned as this14:11
narasimah_SVand in external ceph.yml file it is like this14:12
*** Pavo has joined #openstack-kolla14:12
bjolopbourke, the weirdness we see is that the openflow rules that gets created are in random. so some things might work, and others wont14:13
*** hieulq has joined #openstack-kolla14:13
*** inc0 has joined #openstack-kolla14:13
pbourkebjolo: is the end result that you can't connect to your instance?14:13
bjolopbourke, one time we could trace all trafic going out the wrong external bridge. i.e. 802 IP traffic ended up going out the 805 bridge14:13
egonzalez90narasimah_SV: other roles as glance and nova/ permissions over ceph files are 700 instead of 60014:14
bjoloyes, no traffic in or out works as it should. Some things might work, but it just random14:14
narasimah_SVfor me 644 for those ceph files is working14:14
egonzalez90but that is not the copy issue, this is execution issue14:15
*** msimonin has quit IRC14:15
bjolosince the ex bridges get disconnected all the time, they dont seem to get the right openflow rules14:15
*** fragatina has quit IRC14:15
inc0good morning14:15
portdirect'morning14:16
egonzalez90narasimah_SV: where you have the file? what path?14:16
*** msimonin has joined #openstack-kolla14:16
egonzalez90* in kolla-ansible node14:17
narasimah_SVhttp://paste.openstack.org/show/589592/14:17
narasimah_SVsee the templates and external_ceph file14:18
narasimah_SVthe path is different14:18
narasimah_SVwill this cause any issue14:18
narasimah_SV?14:18
narasimah_SVtha path to copy form these ceph file from config directory to /etc/ceph/14:19
bjolopbourke, on network node do: docker exec vswitchd bash14:20
bjolowatch -d -n 1 "ovs-vsctl show; ovs-ofctl show br-int; ovs-ofctl dump-flows br-int; ovs-ofctl show br-vlan802; ovs-ofctl dump-flows br-vlan802"14:20
pbourkeok one sec14:20
bjoloreplace with your bridge names14:20
*** ppalacios has joined #openstack-kolla14:20
bjolothen on one VM instance, ping the external gw14:21
*** zhurong has quit IRC14:21
bjolowatch -d will show what OF rules are hit by the traffic14:21
*** eaguilar_ has joined #openstack-kolla14:21
bjolonote you need a large screen. I have 4K monitor and the output barely fits :)14:22
bjoloalso what do see in your ovs logfiles? do you get the disconnects14:22
*** eaguilar has quit IRC14:22
pbourkeno Im not seeing those14:23
bjolohmmm14:23
pbourkebut I have other problems, cant even ssh to my instance atm14:23
bjoloi could not ssh either14:23
bjolohad to use the console in horizon14:23
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate  https://review.openstack.org/39850114:23
pbourkeok i'll try that14:23
egonzalez90narasimah_SV: mean, where your keyrings are in ansible node, /etc/kolla/config/* and if in controller nodes the files are properly copied?14:24
*** msimonin has quit IRC14:24
pbourkebjolo: are you using floating ips or launching directly on one of the external networks14:24
narasimah_SVyes in ansible node i have them in /etc/kolla/config14:25
narasimah_SVand even they got copied to controller nodes too14:26
*** msimonin has joined #openstack-kolla14:26
*** msimonin has quit IRC14:26
openstackgerritPete Birley proposed openstack/kolla-ansible: Update Barbican to use cirrect mysql connection string  https://review.openstack.org/39903314:26
openstackgerritPete Birley proposed openstack/kolla-ansible: Update Barbican to use correct mysql connection string  https://review.openstack.org/39903314:28
*** msimonin has joined #openstack-kolla14:28
*** eaguilar has joined #openstack-kolla14:30
*** eaguilar_ has quit IRC14:32
*** ppalacios1 has joined #openstack-kolla14:34
*** ppalacios1 has quit IRC14:34
bjolopbourke, tried both, but mostly floating ips14:34
*** ppalacios has quit IRC14:34
pbourkebjolo: I can't ping/ssh my VM even with one flat network right now14:34
*** adrian_otto has joined #openstack-kolla14:35
bjolook you might have some other issue then14:35
bjolowhen you say one, how much one?14:35
bjoloonly one bridge ex?14:35
bjoloor only one neutron network defined?14:36
*** sdake has joined #openstack-kolla14:36
bjolomy issues start on the bridge level14:36
sdakemorning14:36
pbourkeone external network14:36
bjoloas soon as i add a second external bridge, things break14:36
egonzalez90narasimah_SV: keyring should be in /etc/kolla/config/gnocchi/gnocchi-api/ at deployer node14:36
*** ppalacios has joined #openstack-kolla14:36
bjoloindependent on how many neutron networks i have14:36
srwilkerssdake, go to bed14:37
egonzalez90and in /etc/kolla/gnocchi-api at control node14:37
sdakesrwilkers offsite today14:37
sdakeno can do14:37
sdakesrwilkers have to power through it14:37
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Use mariadb.cnf for mariadb custom file name  https://review.openstack.org/39903814:37
sdakeJeffrey4l say i am too tired to look atm, but did you submit releasee for both kolla-ansible and kolla repo?14:37
Jeffrey4lsdake, yep and it merged.14:38
Jeffrey4lkolla-ansible https://review.openstack.org/39843414:38
sdakecool thx :)14:38
sdakeno need to link14:38
Jeffrey4lkolla https://review.openstack.org/39843314:38
Jeffrey4l;)14:38
sdakeif release team merged it14:38
sdakeits gtg14:38
*** sdake_ has quit IRC14:39
*** ChanServ sets mode: +o inc014:39
*** adrian_otto has quit IRC14:40
inc0 New to Kolla? Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla  http://goo.gl/VKpPzA); w00t for Kolla!; Kolla-ansible is up! All ansible changes goes there now; 4.0.0b1 released!14:40
*** inc0 changes topic to "New to Kolla? Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA); w00t for Kolla!; Kolla-ansible is up! All ansible changes goes there now; 4.0.0b1 released!"14:40
kfox1111morning.14:40
*** inc0 sets mode: -o inc014:40
inc0thanks:)14:40
srwilkersgood morning inc014:40
srwilkersgood morning kfox111114:40
*** duonghq has quit IRC14:40
inc0and hello kfox1111 srwilkers14:40
narasimah_SVegonzalez90: when I placed it in side /etc/kolla/config/gnocchi/gnocchi-api folder it was not taking it14:42
openstackgerritPete Birley proposed openstack/kolla: Remove unneeded PythonMySQL libs from OpenStack Base Image  https://review.openstack.org/39904214:42
narasimah_SVafter changing it /etc/kolla/config/gnocchi-api it worked14:42
narasimah_SVaccording to external_ceph.yml file it is expecting file in /etc/kolla/config/gnocchi-api14:42
*** adrian_otto has joined #openstack-kolla14:42
narasimah_SVfolder structure14:42
narasimah_SVbut in templates again it is different14:43
sdakeportdirect bit concerned with that pythonmysql change14:44
sdakeportdirect considering our gates are afu atm14:44
*** schwicht has joined #openstack-kolla14:45
portdirectsdake: k, I'll fix it14:45
sdakeportdirect not sure naything wrong with the patch14:45
sdakeportdirect did you test it14:45
inc0pprokop, did you release entrypoint binary by any chance?14:45
openstackgerritPete Birley proposed openstack/kolla: DO NOT MERGE!: Remove unneeded PythonMySQL libs from OpenStack Base Image  https://review.openstack.org/39904214:45
sdakeportdirect i was thinking something less drastic, like confirming you tested and finding a victim to test it to confirm ;)14:46
portdirectthe ansible one yes, but not fully tested all images (only centos binary) - I've marked it as do not merge untill the gates are back14:46
sdakeportdirect that may be some time before we have cross repo gating14:46
sdakeportdirect i guess if yo udont mind waiting wfm14:47
portdirectsdake: i can test it properly here (just about to run another build/deployment - then I'll remove the scary looking warning)14:47
sdakeportdirect we have had alot of nasty experiences with those python mysql libraries14:48
sdakeespecially as it relates to kolla-toolbox14:48
inc0Jeffrey4l, I see you're handling kolla-ansible installatin in kolla gates?14:48
Jeffrey4linc0, yep.14:48
sdakenice :)14:48
inc0ok, so I won't remove these gates yet14:48
inc0I'll make build gates voting tho14:49
portdirectsdake: cheers for the heads up :)14:49
Jeffrey4linstall kolla-ansible in kolla  and install kolla in kolla-ansible. ;)14:49
Jeffrey4lre voting, yep. this will make the code more stable.14:49
sdakeportdirect normally i wouldn't care too much about dep changes in that list, but i've spent probably 100+ hours dealing with that particular library14:49
inc0I'll add source builds to voting for now14:49
Jeffrey4lbtw, i removed build related job in kolla-ansible.14:49
inc0as they're much more reliable14:49
sdakemaybe 75 hours14:50
sdakeanyway alot of time burned up on it14:50
inc0Jeffrey4l, did you do it in infra too?14:50
sdakeportdirect fyi the problem is we install the python libraries, and then toolbox has problems because of setuptools14:50
Jeffrey4linc0, yep. what all i did is change the project-config.14:50
inc0ok so I probably need to rebase mine14:51
Jeffrey4linc0, https://review.openstack.org/39850314:51
inc0https://review.openstack.org/#/c/398534/5 I had this but I need to change it since you're making zuul cloner a thing14:51
*** liyifeng has quit IRC14:52
portdirectsdake: thanks - I've had similar pain when the migration started to PyMySQL - though not your issue, cinder refused to work (reliably) with it for ages...14:54
egonzalez90narasimah_SV: saw the error. typo in external-ceph.yml14:54
portdirect(at least for me)14:54
egonzalez90keyring is copied as keryring14:55
narasimah_SVok thanks for the help14:55
narasimah_SVI didnt check about it14:55
*** DTadrzak has quit IRC14:55
egonzalez90feel free to push a change14:55
*** harbor has joined #openstack-kolla14:55
*** harbor is now known as portdirect_away_14:55
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP helm  https://review.openstack.org/39629614:56
*** lrensing has joined #openstack-kolla14:56
narasimah_SVsure I will so it by today14:56
kfox1111so....14:56
kfox1111for the kolla-kubernetes gate,14:57
kfox1111we aonly use genconfig.14:57
kfox1111do I need the kolla repo then?14:57
kfox1111or do I just s/kolla/kolla-ansible/' in the gate?14:57
egonzalez90narasimah_SV: the typo is present in stable/newton open a bug to backport14:58
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate  https://review.openstack.org/39850114:58
openstackgerritJeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment  https://review.openstack.org/39850614:58
narasimah_SVsure14:58
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Fix gate after the repo split  https://review.openstack.org/39905014:59
*** yingjun has joined #openstack-kolla15:00
*** mgoddard_ has joined #openstack-kolla15:01
*** portdirect_away_ has quit IRC15:04
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Use mariadb.cnf for mariadb custom file name  https://review.openstack.org/39903815:04
*** mgoddard has quit IRC15:04
*** chopmann has quit IRC15:07
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Create empty destroy.yml for all role  https://review.openstack.org/39905615:08
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Fix gate after the repo split  https://review.openstack.org/39905015:10
*** adrian_otto has quit IRC15:11
*** adrian_otto has joined #openstack-kolla15:11
pprokopinc0: yes15:12
sdakehey peeps gotta drop off for today15:12
sdakebe back later15:12
kfox1111have a good one15:12
pprokophttps://github.com/stackanetes/kubernetes-entrypoint/releases/download/v0.1.0/kubernetes-entrypoint15:12
*** jtriley has joined #openstack-kolla15:12
kfox1111pprokop: is there a container yet?15:12
pprokopyeh15:13
pprokopquay.io/stackanetes/kubernetes-entrypoint:v0.1.015:13
pprokopbased on debian15:14
kfox1111cool. should make testing a lot easier. thanks. :)15:14
*** DTadrzak has joined #openstack-kolla15:14
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate  https://review.openstack.org/39850115:15
openstackgerritJeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment  https://review.openstack.org/39850615:16
inc0kfox1111, I'll curl it to our kolla-k8s-toolbox container15:17
inc0and I was thinking of using it as entrypoint15:17
*** harbor has joined #openstack-kolla15:21
*** harbor is now known as portdirect_away_15:21
sdakeinc0 sounds good to me15:21
sdakegotta jet15:21
sdakelets just make sur ewe stick to the optional thign :)15:21
kfox1111inc0: I think we should just amake a kubernetes-entrypoint container.15:23
kfox1111the toolboxen are doing too much.15:23
*** ppalacios has quit IRC15:23
inc0kfox1111, technically toolbox doesn't do anything15:23
inc0it just holds various binaries:)15:23
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate  https://review.openstack.org/39850115:24
sbezverkinc0: I checked doc for entry point and have a question, since you are pushing it so much I guess you know the answer ;-) One required parameter of it is command, which is executed when all dependecy get resolved. I am not sure how it aligns with approach that init container is used just to wait for dependecy and then exit to allow main container to start.. could you elaborate on this15:25
sbezverka bit?15:25
kfox1111inc0: yeah. and the k8s way usually is lots of tiny containers.15:25
inc0sbezverk, command can be exit 1:)15:26
inc0or exit 0 rather15:26
sbezverkinc0: got it, thank you..15:26
kfox1111sbezverk: I filed an issue for that15:26
kfox1111https://github.com/stackanetes/kubernetes-entrypoint/issues/1215:26
kfox1111was fixed really quickly. :)15:27
kfox1111I'd rather have containers that do what they do well, rather then bundling up binaries together, making it harder to test upgrading one thing without breaking another.15:28
inc0kfox1111, fair enough, it can be dedicated15:28
sbezverkkfox1111: cool, thanks for the link. makes perfect sense..15:28
inc0pprokop, this issue was part of released binary right?15:28
sbezverkkfox1111: I agree about the statement about multiple small containers, but here is a problem which I think is triggered by this approach. If you code version of image directly into the container, it will be very difficult to change it for something else. Example I run etcd for canal and it is 2.2.1, I want to use different version, then I need to change pod definition..15:32
*** ppalacios has joined #openstack-kolla15:32
sbezverkwith centralized storage of this in tool box it might be easier as version controller in a single place..15:32
inc0sbezverk, on the other hand if you want to change just one thing and leave another intact15:33
inc0having single container makes this impossible:)15:33
sbezverkinc0: right but if you need to change it for 100 pods ;-) ??15:34
*** absubram has joined #openstack-kolla15:34
inc0sbezverk, thats why we use helm:)15:34
*** absubram_ has joined #openstack-kolla15:35
sbezverkinc0: make sense but in this case we have to make sure we use variables for all images used in pod by containers15:36
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Dependency init container  https://review.openstack.org/39908215:37
*** fragatina has joined #openstack-kolla15:37
*** absubram has quit IRC15:38
*** absubram_ is now known as absubram15:38
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Dependency init container  https://review.openstack.org/39908215:39
Jeffrey4linc0, we have removed all branches in kolla-ansible. should we need remove all the tags?15:41
inc0Jeffrey4l, besides 4.0.0b1 ofc15:41
*** fragatina has quit IRC15:41
Jeffrey4linc0, yes. but how? do u have any idea how to remove tag?15:42
*** fragatina has joined #openstack-kolla15:42
inc0hmm...not sure too15:42
*** srwilkers is now known as srwilkers_away15:43
inc0sbezverk, we hardcode version in dockerfile a lot15:44
inc0https://github.com/openstack/kolla/blob/master/docker/heka/Dockerfile.j2#L17 an example15:44
*** diogogmt has joined #openstack-kolla15:46
kfox1111sbezverk: I thats actually a feature.15:47
kfox1111you want to know exactly the version you need is running identically on all nodes.15:47
kfox1111so all versions should be explicit.15:47
kfox1111in k8s.15:47
kfox1111so yes, your right, it would require tweaking the packages.15:48
kfox1111but that lets you test each package and verify it works, and then you know when a user wants to deploy a specific package, that package works.15:48
sbezverkkfox1111: but if you hardcode version in image, no tweaking in package will help I think15:48
sbezverkI mean docker image15:49
sbezverkot the image name must have the actual software version like entrypoint-0.1.0-4.0.0b115:49
kfox1111so, let me go through a workflow I usually use as an example.15:50
*** matrohon has quit IRC15:50
kfox1111tag should have the version.15:50
kfox1111or be date tagged or something.15:50
kfox1111so, I run a physics system with k8s now.15:50
*** mgoddard has joined #openstack-kolla15:50
*** mgoddard_ has quit IRC15:50
kfox1111I have a bunch of containers. but lets walk through one.15:51
kfox1111I have the source for the container here: https://github.com/pnnlhep/osg-fetch-crl15:51
kfox1111associated with that is a autobuilding repo on the hub. https://hub.docker.com/r/pnnlhep/osg-fetch-crl/15:52
kfox1111so I can push a change to the git repo and get a new container, or I can go to the hub and trigger a new build.15:52
kfox1111I then docker pull pnnlhep/osg-fetch-crl:latest and inspect/test it.15:52
kfox1111once I'm happy with that image, I15:53
kfox1111docker tag pnnlhep/osg-fetch-crl:latest pnnlhep/osg-fetch-crl-stable:2016-11-1715:53
kfox1111docker push pnnlhep/osg-fetch-crl-stable:2016-11-1715:53
kfox1111making a release of that container.15:53
kfox1111then I go into my deployment in k8s, update the image to point to that specific version, and upgrade the k8s object on the cluster.15:54
kfox1111this then deploys the change to all the nodes.15:54
kfox1111so I know if I ever have to reinstall a node, it is consistent with al lthe other nodes.15:54
kfox1111if I run into a problem, I can just change the image tag back to the previous version.15:54
sbezverkkfox1111: in your example you do not hardcode a version of software you add into the docker image, you rely on yum to get you one.15:55
kfox1111yeah.15:55
kfox1111usually I don't pin a particular version unless I run into some problem.15:55
sbezverkbut when it is hardcoded, then you might end with the same version of software, but wil different tagging15:55
sbezverkif you build this software at different dates15:56
kfox1111right.15:56
kfox1111its kind of rare you don't get a different container though anyway.15:56
sbezverkthere is no way to tell what actual software version was used when the image was built15:56
kfox1111the underlying containers/ppackages usually get updates too.15:56
sbezverkkfox1111: yes, if you yum for it15:56
kfox1111true. mostly I don't care so long as it passes my tests.15:57
sbezverkbut when you curl specific version it does not15:57
inc0kfox1111, sbezverk one of you has to change length of your nicks;)15:57
sbezverkinc0: sure, could you propose one for me ;-)15:57
kfox1111inc0: or you could add 0's until your equal length. ;)15:57
*** inc0 is now known as inc0000015:58
inc00000hmm15:58
inc00000make life harder15:58
kfox1111sbezverk: realy, I think one of our outstanding issues is that we don't have a ci/cd kind of tool to keep our containers up to date.15:58
sbezverknice, good binary number ;-)15:58
kfox1111they often need many more upgrades then we can track.15:58
*** inc00000 is now known as inc015:58
kfox1111though that is a general docker/kubernetes problem.15:58
kfox1111so, thats kind of one thing I want to talk about at some point.15:59
kfox1111versioning.15:59
sbezverkkfox1111: right I am experiencing it myself now, that is why ibrought it up15:59
sbezverkintroducing versions might help us in future15:59
sbezverkbut it is optional15:59
kfox1111so, we version after kolla. in our containers.16:00
kfox1111so, we have centos-binary-nova-api:2.0.2 or 3.0.116:00
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Dependency init container  https://review.openstack.org/39908216:00
sbezverkafter or before, does not matter16:00
kfox1111that reflects the packaging of the container not the stuff in int.16:00
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Remove non-existent branches in releasenote  https://review.openstack.org/39911116:01
openstackgerritJavier Castillo Alcíbar proposed openstack/kolla-ansible: Fix missing libvirt python module in ceilometer-compute (v2)  https://review.openstack.org/39911316:01
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: Use mariadb.cnf for mariadb custom file name  https://review.openstack.org/39903816:01
kfox1111but it doesn't reflect changes in version of the thing people probably care about. nova-api itself.16:02
kfox1111or none of its dependencies aren't reflected.16:02
kfox1111in most packaging systems, the version of the base program is used,16:02
*** portdirect_away_ has quit IRC16:02
*** absubram has quit IRC16:03
*** ppalacios1 has joined #openstack-kolla16:03
sbezverkkfox1111: changing what is used now for kolla is impossible, but changing aproach for kolla-kube specific image might be a viable option.16:03
kfox1111say, nova 14.0.0, 14.0.1, etc.16:03
kfox1111packaging level changes are done with a revision.16:03
*** ppalacios has quit IRC16:03
*** ppalacios1 is now known as ppalacios16:03
sbezverksince I heard you guis want to keep kube specific images in kolla-kubernetes repo16:03
kfox1111so, 14.0.1-1, 14.0.1-2, etc.16:03
*** zhubingbing_ has quit IRC16:04
kfox1111with the way we're doing things tough with packages, we have one more level of stuff that needs versioning. :/16:04
kfox1111we have 3 things in flight.16:04
kfox1111the version of the software, the version of the packaging, and some kind of version for all the other unversioned stuff (dependencies)16:05
kfox1111so, for example.16:05
sbezverkkfox1111: no need to change it right now, all I asked in is to put FIXME tag to come back later and re-evaluate..16:05
*** rmart04 has quit IRC16:05
kfox1111we cut nova-api:14.0.1-1 (specific netruon verson, verson 1 of our build scripts)16:06
kfox1111then the next openssl volnerability happens and openssl version n+1 comes out.16:06
kfox1111nova-api doesn't change, and our packaging doesn't change.16:06
kfox1111so I think we need a build number too.16:06
kfox1111nova-api:14.0.1-1-116:06
kfox1111so that's for the package builder to increment each time a new container's built or accepted or whatever their procedure is.16:07
sbezverkkfox1111: if we are going towards more kube like images then we should do it16:07
kfox1111the first one would be nova-api:14.0.1-1-1. when rebuild to handle the openssl security issue, nova-api:14.0.1-1-2 gets cut.16:07
*** Pavo has quit IRC16:07
kfox1111I think kolla-ansible would benifit from it too.16:08
kfox1111it has the same issue.16:08
kfox1111its really a kolla-containers issue16:08
kfox1111"how do you keep containers up to date and atomic"16:09
sbezverkkfox1111: Oh man, if you try to fix kolla-ansible, it will never be done in our lifetime ;-) let's try to fix for kube for now..16:09
kfox1111"and prevent version skews in your system"16:09
kfox1111yeah.16:09
kfox1111I think we start with the helm packages.16:09
inc0-.-16:09
*** ayoung has quit IRC16:10
*** ppalacios has quit IRC16:10
sbezverkkfox1111: right helm should help but there should be support in naming the images too, otherwise it is useless..16:11
*** Pavo has joined #openstack-kolla16:12
kfox1111yeah.16:12
kfox1111I think we can start with versioning the helm packages the way that will be the most flexible.16:13
kfox1111then work backwards to fix the container tagging.16:13
kfox1111I think we can make it configurable in the package templates in the mean time too.16:13
*** jtriley has quit IRC16:14
*** ppalacios has joined #openstack-kolla16:14
*** jtriley has joined #openstack-kolla16:15
*** adrian_otto has quit IRC16:15
*** fragatina has quit IRC16:15
*** jemcevoy has joined #openstack-kolla16:15
openstackgerritJavier Castillo Alcíbar proposed openstack/kolla-ansible: Fix missing libvirt python module in ceilometer-compute (v2)  https://review.openstack.org/39911316:16
*** RichWellum has quit IRC16:17
*** ppalacios1 has joined #openstack-kolla16:20
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Dependency init container  https://review.openstack.org/39908216:20
*** ppalacios1 has quit IRC16:20
inc0I have strange bug oO16:21
*** ppalacios has quit IRC16:21
inc0my build takes very very old registry IP16:21
inc0and I have no idea from where16:21
inc0it's in my subnet and there was registry at that address long ago16:22
inc0but all the configs points to new one16:22
inc0build is still stupid16:22
*** ppalacios has joined #openstack-kolla16:22
openstackgerritMerged openstack/kolla-ansible: Remove non-existent branches in releasenote  https://review.openstack.org/39911116:25
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Squashed commit of the following:  https://review.openstack.org/39856316:30
*** fragatina has joined #openstack-kolla16:30
*** ppalacios1 has joined #openstack-kolla16:30
*** eaguilar has quit IRC16:30
egonzalez90inc0: container is depenency or dependency?16:31
*** ppalacios has quit IRC16:31
*** adrian_otto has joined #openstack-kolla16:31
*** ppalacios1 is now known as ppalacios16:31
inc0ahh16:32
inc0how fun16:32
inc0made a typo16:32
egonzalez90;)16:32
inc0good catch16:32
egonzalez90code blocks have the same typo16:33
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Dependency init container  https://review.openstack.org/39908216:33
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Fix gate after the repo split  https://review.openstack.org/39905016:33
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Dependency init container  https://review.openstack.org/39908216:34
inc0kfox1111, I refrain of using word "entrypoint" here as it's a container, not entrypoint16:35
inc0I don't want to confuse people16:36
kfox1111its the software. that is entrypoint.16:36
kfox1111where its instantiated, thats deployment-init-container.16:37
kfox1111thats like renaming nova-api the container, cloud-vm-rest-thingy.16:37
kfox1111:)16:37
inc0hmm16:37
inc0ehh, I used this word in spec, can we roll with it?16:38
kfox1111I don't think anyone objects to kubernetes-entrypoint the software being a continer. :)16:38
inc0ok...I can rename it16:38
kfox1111the contraversy is in how its used.16:38
inc0I'll call it ensign16:38
inc0how about that?;)16:39
* kfox1111 shrugs.16:39
*** ppalacios has quit IRC16:40
kfox1111I like the container name to match up what the daemon running in it is. makes it easier to debug.16:40
kfox1111at this point,  it doens't matter that much though.16:40
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Dependency init container  https://review.openstack.org/39908216:40
inc0too late, already changed it16:41
kfox1111works for me.16:41
*** fragatina has quit IRC16:41
*** ppalacios has joined #openstack-kolla16:42
*** ppalacios has quit IRC16:42
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Dependency init container  https://review.openstack.org/39908216:42
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Dependency init container  https://review.openstack.org/39908216:44
*** tonanhngo has joined #openstack-kolla16:46
kfox1111inc0: so, a work item... either we need a gate job for kolla-kubernetes that builds updated containers from kolla repo, or we need to make an alpha release of that container. (or both)16:48
inc0kfox1111, we can use zuul-cloner to build images in gates16:48
kfox1111that woudl work, but would add significant time to the gate jobs.16:49
kfox1111and make it harder to debug the gate.16:49
kfox1111as multiple things could be changing at once.16:49
inc0kfox1111, https://review.openstack.org/#/c/398506/16:50
kfox1111right now the kolla-kubernetes job just tests kolla-kubernetes deployment against preexinsting containers. which means they don't randomly break. its a known good ground state.16:50
inc0long term solution is registry16:50
inc0if you want to help me work on it16:50
inc0with infra16:50
inc0we can jump right to it16:50
kfox1111inc0: big +1 from me. :)16:51
inc0I need someone to help me with puppet there:P16:51
kfox1111that I'm not so good at. :/16:51
kfox1111been a few years since I touched puppet, and then it was pretty much just to evalueate it vs other things.16:52
inc0I take it didn't exactly pass?16:52
kfox1111we ended up with chef.16:52
kfox1111but that was partially because at the time the open  source community was leaving puppet for chef, because the company behind puppet was doing some bad things.16:52
kfox1111then the flow reversed once puppet company cleaned up their act.16:53
*** DTadrzak has quit IRC16:53
kfox1111the chef company kind of did the same thing later and pushed people back twoards puppet.16:54
kfox1111not sure when the opensource community will really wake up to the problems associated with opensource ruled by one copany.16:54
kfox1111you'd think the oracle buyout feasco woudl have trained people once and for all. :/16:55
inc0well, people feel good when there is someone to call16:56
openstackgerritJavier Castillo Alcíbar proposed openstack/kolla-ansible: Fix missing heka/heka-ceilometer.toml in ceilometer_compute  https://review.openstack.org/39913416:56
inc0look at linux distros16:56
kfox1111yeah. nothing wrong with that.16:56
kfox1111its more of a governance issue.16:56
inc0RHEL, Ubuntu..both run by single company16:56
kfox1111you need opensource governance with companies for support.16:56
kfox1111not companies for control.16:56
kfox1111unless there's no single company controlling it.16:57
*** ppalacios has joined #openstack-kolla16:57
*** eaguilar has joined #openstack-kolla16:57
*** Serlex has quit IRC16:58
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate  https://review.openstack.org/39850117:00
*** eaguilar has quit IRC17:02
*** alteriks has joined #openstack-kolla17:02
*** Satya_ has joined #openstack-kolla17:02
Satya_Hi sdake17:03
Satya_Hi jeffrey17:03
*** srwilkers_away is now known as srwilkers17:03
kfox1111we need to make a gate job to cache kubernetes stuff too. :/17:04
kfox1111they keep breaking the gate. :/17:04
*** egonzalez90 has quit IRC17:04
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Fix gate after the repo split  https://review.openstack.org/39905017:08
inc0kfox1111, I promised sdake to write a bp about moving configs out of ansible17:08
inc0but I'll write a spec17:09
sdakeroger17:09
sdakespec is good17:09
sdakegotta jet17:09
sdakelets just get  it done17:09
inc0yeah, it's too important17:09
sdakeand not debate it forever17:09
sdakepls :)17:09
inc0one spec to solve one well defined problem17:09
sdakegotta jet for the day - so can't particiapte in irc17:09
inc0have fun17:09
sdakefun isnt' what i'd call it :)17:10
sdakethanks for the support :)17:10
kfox1111inc0: +1 for a spec.17:10
sdakettyl see you at 2pmish17:10
pbourkebjolo: finally got single network working17:10
*** bmace has quit IRC17:10
*** ayoung has joined #openstack-kolla17:10
pbourkebjolo: now redeploying with two17:10
bjolopbourke, nice17:10
bjolowhat was the issue?17:10
*** bmace has joined #openstack-kolla17:10
pbourkebjolo: my external interfaces werent up17:11
pbourkebjolo: :/17:11
bjolohaha classic one17:11
*** msimonin has quit IRC17:12
*** adrian_otto has quit IRC17:12
bjoloopen question for the channel. anyone have experience with mellanox spectrum switches and mlnx-os?17:13
kfox1111xkcd is good today. http://xkcd.com/17:13
kfox1111I sware that same thing happend to me 2 weeks ago... :)17:13
kfox1111why a modern computer still has over/underscanning issues on a 1080p screen is frickin rediculous.17:15
kfox1111and I have a bios with a gui that won't talk to half the tv's I have. :/ but works fine outside the bios. seriously...17:15
kfox1111ok. rant over. :)17:15
*** strigazi is now known as strigazi_AFK17:16
srwilkersthats a good one kfox111117:16
*** Satya_ has quit IRC17:17
pbourkebjolo: it works!17:19
bjolono!!!17:19
*** fragatina has joined #openstack-kolla17:19
bjolodont say that17:19
pbourkebjolo: yeah I know that's not actually what you wanted to hear17:19
*** ppalacios1 has joined #openstack-kolla17:19
pbourkebut tbh if it didnt our internal QA would have shouted at us17:19
*** ppalacios has quit IRC17:20
bjolook you need to give me some info here17:20
pbourkejust to be sure I'm going to boot a VM on the second ext net17:20
pbourkethen I'll dump whatever you need17:20
bjolohost os, container os, source/binary17:20
bjoloetc etc17:20
pbourkekolla master17:20
kfox1111kolla-kubernetes cores: https://review.openstack.org/#/c/399050/17:20
kfox1111fixes the gate breakage.17:21
bjolook one diff, i run stable/newtwon17:21
pbourkebase oraclelinux, type source. Host OS is oraclelinux 7.2, running as kvm guests17:21
openstackgerritJavier Castillo Alcíbar proposed openstack/kolla: Fix missing libvirt python module in ceilometer-compute (v2)  https://review.openstack.org/39914517:21
bjolook pbourke17:21
pbourkeour internal builds use kolla newton with openstack mitaka17:21
bjolocan you do a centos source build on stable/newton and deploy that?17:21
pbourkewill do17:21
*** ppalacios1 is now known as ppalacios17:22
pbourkeI have to go soon but will kick off a build of stable/newton and deploy in the morning17:22
*** yingjun has quit IRC17:22
*** yingjun has joined #openstack-kolla17:23
pbourkealso this is the openvswitch I have in my images: openvswitch-2.5.0-2.el7.x86_6417:23
bjoloi grab the latest tarballs in my builds17:23
bjololocation = http://tarballs.openstack.org/neutron/neutron-stable-newton.tar.gz17:23
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Move config templates to Kolla spec  https://review.openstack.org/39914717:23
kfox1111can we get that review merged asap?17:23
openstackgerritJeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment  https://review.openstack.org/39850617:23
inc0Jeffrey4l, do you know puppet?17:24
Jeffrey4linc0, a little.17:24
inc0we need a module to install docker registry with authentication on ubuntu17:24
inc0I can try doint parts of it, but I have no diea how puppet works17:24
Jeffrey4lwhy not use ansible ;)17:25
inc0because it's infra17:25
inc0and openstack-infra uses puppet17:25
bjolopbourke, tnx17:25
Jeffrey4linc0, hmm. sorry no idea for this ;(17:25
inc0kk, just asking around;)17:26
bjoloi got to go. svimclass for the big girl17:26
inc0I'm afraid to ask bjolo17:26
inc0well...have fun17:26
bjolohaha inc017:26
bjolodont let your imagination run wild17:26
inc0yeah, vim is a strange fetish17:27
*** tonanhngo has quit IRC17:27
inc0"change mode to insert now!"17:27
*** yingjun has quit IRC17:27
*** fragatina has quit IRC17:29
*** fragatina has joined #openstack-kolla17:30
*** TxGirlGeek has joined #openstack-kolla17:31
srwilkerslol17:32
inc0yes, jokes which causes instant silence in channel are...concerning17:32
inc0sorry about that17:32
pbourkeyou should hear SamYaple's jokes17:33
inc0oh I've heard those17:33
pbourkei call them dad jokes17:34
pbourkewhich is ok because he actually is a father :p17:34
rhalliseykfox1111, roger that17:34
kfox1111rhallisey: thx.17:34
inc0rhallisey, https://review.openstack.org/#/c/399147/117:35
inc0just in case you thought you're done with specs for now17:35
rhalliseykk17:35
rhalliseywhat's the time block for that?17:36
rhalliseylet's keep it to like 2 days17:36
inc0nooooo17:36
inc0this takes as much time as it needs17:36
kfox1111yeah. the config bits are important, but uncontravertial I think.17:37
kfox1111I doubt it will drag on for longer then needed.17:37
kfox1111but getting it right is important.17:37
inc0yeah, it's about "how to make it work without breaking stuff"17:37
* kfox1111 nods17:37
kfox1111and while we're at it, fixing some design issues. since we're in the code already.17:37
* inc0 nods17:38
pbourkeinc0: can/should we issue a note to cores to not merge stuff till the gates are fixed up17:40
kfox1111pbourke: +117:40
kfox1111that provides incentive to fix the gates.17:40
kfox1111and not futher break stuff.17:40
inc0I'd hope that we don't have to make note like that17:40
kfox1111making it harder to fix the gates. :)17:40
pbourkewell im seeing +2's go on things17:41
pbourkemaybe people know not to merge but I think no harm in making that explicit17:41
*** adrian_otto has joined #openstack-kolla17:41
kfox1111+117:41
kfox1111too much communication is better then not enough.17:41
pbourkeright17:41
*** unicell has quit IRC17:41
inc0I don't want to send mail to ML with "dont merge if gates are red"...17:41
pbourkei guess we'll just keep an eye on it then17:42
kfox1111we don't have a way to email cores?17:42
pbourkeim looking now and nothing seems to have merged since the split so thats good17:42
inc0yeah I hope we get out of gate crysis soon enough17:43
inc0this is short lived crisis I'd hope17:43
pbourkethe gates can run crysis?17:43
inc0yeah17:43
inc0can't run that is17:43
pbourkethat was my bad joke for the day17:44
*** goldyfruit has quit IRC17:44
jascott1only with all the settings on low17:44
inc0ahh...well, you're not English so you have no excuse for bad jokes....sorry mate17:44
kfox1111rhallisey: https://review.openstack.org/#/c/399050/ workflow please? the experimental tests are still running, but they long pass the point of the thing fixed. we're good there.17:45
inc0now when I think about it, I don't think that we have any Englishman around here17:45
rhalliseykfox1111, ya I waiting for them to come back17:46
kfox1111looks like it wil finish any second though.17:46
rhalliseykk17:46
kfox1111all but the multinode already passed. the multinode one takes a couple minutes longer.17:47
*** DaveTurner has joined #openstack-kolla17:47
pbourkewe do17:47
pbourketrying to remember his name17:47
*** Satya_ has joined #openstack-kolla17:48
*** msimonin has joined #openstack-kolla17:49
kfox1111rhallisey: passed. :)17:51
rhalliseykk17:51
*** msimonin has quit IRC17:51
inc0rhallisey, https://review.openstack.org/#/c/399082/17:51
*** goldyfruit has joined #openstack-kolla17:51
kfox1111rhallisey: thanks. :)17:51
openstackgerritMerged openstack/kolla-kubernetes: Fix gate after the repo split  https://review.openstack.org/39905017:53
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP helm  https://review.openstack.org/39629617:54
*** alteriks has quit IRC17:54
*** ayoung has quit IRC17:55
*** msimonin has joined #openstack-kolla17:56
*** alteriks has joined #openstack-kolla17:56
*** narasimah_SV has quit IRC17:56
*** derekjhyang has joined #openstack-kolla17:58
*** castulo has joined #openstack-kolla17:58
*** alteriks has quit IRC18:00
*** schwicht has quit IRC18:01
jascott1helm 2.0 is released https://deis.com/blog/2016/helm-is-now-stable/18:01
mliimaguys, need some reviews here https://review.openstack.org/#/c/398516/18:04
*** msimonin has quit IRC18:04
*** fragatin_ has joined #openstack-kolla18:04
kfox1111yay! :)18:05
*** tonanhngo has joined #openstack-kolla18:07
*** fragatina has quit IRC18:07
*** Pavo has quit IRC18:07
*** harbor has joined #openstack-kolla18:09
*** harbor is now known as portdirect_away_18:09
*** portdirect_away_ has quit IRC18:09
sbezverkkfox1111: curious if there a procedure to update helm rc2 to final release18:09
*** tonanhngo_ has joined #openstack-kolla18:10
openstackgerritMerged openstack/kolla: Add how configure multiple back ends section  https://review.openstack.org/39851618:10
*** eaguilar has joined #openstack-kolla18:11
*** tonanhngo_ has quit IRC18:11
*** tonanhngo has quit IRC18:11
*** tonanhngo has joined #openstack-kolla18:11
*** pbourke has quit IRC18:12
*** Pavo has joined #openstack-kolla18:12
*** pbourke has joined #openstack-kolla18:12
*** tonanhngo has quit IRC18:13
*** TxGirlGeek has quit IRC18:14
*** TxGirlGeek has joined #openstack-kolla18:14
*** tonanhngo has joined #openstack-kolla18:15
jascott1sbezverk I think you just deploy new binary and upgrade tiller https://github.com/kubernetes/helm/blob/master/docs/install.md#deleting-or-reinstalling-tiller18:15
sbezverkkfox1111: never mind, figured out. you just need to edit tiller-deploy to point to the latest version..18:15
jascott1haha18:15
*** eaguilar has quit IRC18:15
*** unicell has joined #openstack-kolla18:15
*** tonanhngo has quit IRC18:17
qwangjascott1: Do you know how to completely remove a release from helm?18:17
jascott1helm delete <release> ?18:17
jascott1^ thats supposed to work18:18
*** ppalacios has quit IRC18:18
qwangjascott1: I used helm delete, but when I try to create another release with the same name, helm won't let me do it18:18
jascott1it wont18:18
jascott1it keeps the release history18:18
*** ppalacios has joined #openstack-kolla18:18
jascott1I never use named releases18:18
jascott1you can helm list all and see one sec18:18
jascott1helm list --all18:19
qwangI can see the old releases with status "deleted"18:19
*** papacz has quit IRC18:21
jascott1ya you either need to bump a version with your same release name or use random names18:21
*** ayoung has joined #openstack-kolla18:21
jascott1I just let it create names. i install and delete so much I created an alias: hda="helm list -q | xargs helm delete"18:22
jascott1doesnt help your issue but makes it easy to remove and not care about random names18:22
qwangjascott1: thx18:23
kfox1111I think we're going to want to use named releases...18:24
kfox1111do you know if thye fixed the really short release name max?18:24
jascott1they did18:24
jascott1its 53 max chars now18:25
kfox1111whats the limit know?18:25
kfox1111ah. nice. that should be enough. :)18:25
kfox1111thanks.18:25
kfox1111so we have to upgrade the gate to the final release too...18:25
*** strigazi_AFK is now known as strigazi18:33
*** Satya_ has quit IRC18:34
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Tacker NFV Ansible support  https://review.openstack.org/39917918:34
*** Satya_ has joined #openstack-kolla18:34
bmaceI think the overview on https://launchpad.net/kolla-ansible needs to be updated since it is actually active now.18:35
*** coolsvap has quit IRC18:37
*** krtaylor has quit IRC18:38
qwang+118:39
openstackgerritEduardo Gonzalez proposed openstack/kolla: Tacker global variables  https://review.openstack.org/38979018:39
*** newmember has joined #openstack-kolla18:39
*** egonzalez90 has joined #openstack-kolla18:40
*** sdake has quit IRC18:44
*** msimonin has joined #openstack-kolla18:48
*** diogogmt has quit IRC18:55
*** adrian_otto has quit IRC18:56
*** diogogmt has joined #openstack-kolla18:57
*** shardy has quit IRC18:57
*** gfidente has quit IRC19:00
*** fragatin_ has quit IRC19:01
*** fragatina has joined #openstack-kolla19:02
*** diogogmt has quit IRC19:02
*** eaguilar has joined #openstack-kolla19:03
*** eaguilar has quit IRC19:07
*** ppalacios1 has joined #openstack-kolla19:13
*** ppalacios1 has quit IRC19:13
*** msimonin has quit IRC19:14
*** msimonin has joined #openstack-kolla19:14
*** ppalacios has quit IRC19:14
*** ppalacios has joined #openstack-kolla19:15
*** msimonin has quit IRC19:17
*** schwicht has joined #openstack-kolla19:18
*** tonanhngo has joined #openstack-kolla19:19
*** eaguilar has joined #openstack-kolla19:25
*** strigazi is now known as strigazi_AFK19:25
*** Satya_ has quit IRC19:29
*** eaguilar has quit IRC19:30
kfox1111arg...19:31
*** v1k0d3n has joined #openstack-kolla19:31
kfox1111the world really needs a shell with better variable manipulation.19:31
kfox1111python's painful for automating shell like tasks but good at data manipulation. bash, the reverse. :/19:31
*** severion has joined #openstack-kolla19:31
*** msimonin has joined #openstack-kolla19:32
*** v1k0d3n has quit IRC19:35
*** awiddersheim has joined #openstack-kolla19:36
*** severion has quit IRC19:36
*** v1k0d3n has joined #openstack-kolla19:36
v1k0d3nhey all19:37
srwilkershey v1k0d3n19:37
v1k0d3nyo19:37
v1k0d3nfinally. a "real" internet connection at the office.19:37
v1k0d3nbout darn time!19:37
*** mgiles has quit IRC19:38
srwilkerstime to leave the matrix19:39
*** castulo has left #openstack-kolla19:39
sean-k-mooneyhi just a 2 second dumb question. with the repo split for https://review.openstack.org/#/c/342354/ the anible change go to kolla-ansible, docker chages to kolla. where do the docs chages go?19:40
sean-k-mooneykolla-ansible?19:40
openstackgerritBorneMace proposed openstack/kolla-ansible: Ignore dom0 qemu processes during destroy  https://review.openstack.org/39920319:40
*** krtaylor has joined #openstack-kolla19:42
inc0sean-k-mooney, not a dumb question, docs goes to kolla19:42
inc0for now19:42
kfox1111v1k0d3n: hi19:42
sean-k-mooneyok cool i was going to break this up into different patches anyway so this is as good a time as any to do it.19:43
sean-k-mooneyinc0: regarding your extensible ansible change. i think we might want to look at prototyping something with this review also https://review.openstack.org/#/c/387854/19:44
*** openstackgerrit has quit IRC19:48
*** openstackgerrit has joined #openstack-kolla19:48
*** ppalacios has quit IRC19:51
*** ppalacios has joined #openstack-kolla19:57
*** sbezverk_ has joined #openstack-kolla20:01
egonzalez90sean-k-mooney inc0 thats a woot for kolla?20:02
*** sbezverk has quit IRC20:02
inc0interesting20:03
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP helm  https://review.openstack.org/39629620:04
sean-k-mooneyegonzalez90: that being?20:04
inc0v1k0d3n, did you switch internet from ATnT to Comcast?20:04
kfox1111lets see how this flies... different way of doing helm...20:04
v1k0d3ninc0: ha...ha. so funny. mr. fun guy.20:04
v1k0d3n:)20:04
inc0so I have inet from ATT now, but since I'm moving, will have it no longer20:05
inc0no ATT coverage where I'm going20:05
egonzalez90sean-k-mooney for collectd people using kolla and showing docs how to integrate20:05
jascott1kfox1111 what did you change with approach?20:05
sean-k-mooneyinc0: can you get good broadband like google fiber?20:05
kfox1111yeah. I think att still only offers 1.5 mb here.20:05
egonzalez90think is a good step20:05
kfox1111jascott1: more in line with what we were talking about yesterday. building a package per k8s object.20:06
sean-k-mooneyegonzalez90: ah well they work on my team and i like kolla so asked them to add support20:06
jascott1cool will check it out20:06
egonzalez90cool :)20:06
inc0sean-k-mooney, not yet20:06
inc0they arrive to San Antonio but Im moving out20:07
*** Pavo has quit IRC20:07
kfox1111jascott1: thanks. plz let me know what you think.20:08
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP helm  https://review.openstack.org/39629620:09
inc0https://github.com/openstack-infra/puppet-docker_registry20:10
*** Pavo has joined #openstack-kolla20:12
kfox1111inc0: nice.20:14
kfox1111there a pr going to follow/contribute to?20:14
inc0we need that think filled with goodness20:14
inc0https://review.openstack.org/#/c/399221/ kfox1111 it's normal openstack project20:14
inc0when we prep this module20:15
kfox1111and another question... I know its not exactly how they normally do it. but should we deploy it with docker?20:15
inc0no, they want it to be standalone20:15
kfox1111have the puppet module load docker / kubelet standalone, then just run it as a container?20:15
inc0and use their existing apache for auth20:15
kfox1111ah.20:15
inc0that being said20:15
v1k0d3ninc0: makes total sense. we moved down the street and don't have att.20:16
inc0standalone registry is literally apt-get install docker-registry20:16
v1k0d3nwhen we had it was aweosme though20:16
kfox1111could still container it and tehn use the existing apache to auth/proxy.20:16
inc0v1k0d3n, yeah, I'm actually quite happy20:16
inc0a bit overpriced here, but meh, wild west20:16
kfox1111ah. ok. didn't think they had dpkg packages for the registry20:16
inc0kfox1111, issue is nobody wants to run docker deamon there;)20:16
inc0also registry is golang blob, we can just curl and run it20:17
inc0easier than setting up full docker infra20:17
pprokopqwang: try helm delete --purge20:17
pprokopit's like docker rm -f20:18
qwangpprokop: thanks. Just tried, it works like a charm :)20:20
kfox1111~.20:20
kfox1111~>20:20
qwangpprokop: I have another question. Are livenessprobe and readinessprobe usually the same? I mean the code part, I know they are functional differently20:21
*** newmember has quit IRC20:22
*** reek269 has joined #openstack-kolla20:22
*** ppalacios has quit IRC20:22
rhalliseyqwang, liveness is to check the pod is behaving.  Readiness is to check the pod is ready to start20:27
*** adrian_otto has joined #openstack-kolla20:27
*** egonzalez90 has quit IRC20:28
*** ppalacios has joined #openstack-kolla20:28
*** eaguilar has joined #openstack-kolla20:29
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP helm  https://review.openstack.org/39629620:29
kfox1111rhallisey: I think readyness checks continuously run?20:33
kfox1111and liveness probe is a readyness probe that shoots the pod on failure.20:33
*** eaguilar has quit IRC20:33
rhalliseyhttp://kubernetes.io/docs/user-guide/production-pods/#liveness-and-readiness-probes-aka-health-checks20:35
kfox1111yeah, the docs are less then clear last I read them.20:36
kfox1111I was under the impression,20:36
kfox1111a pod could go unready if it passed a check later on.20:37
kfox1111it pulls it back out of the lb but leaves the pod alive.20:37
rhalliseythey changed the docs20:37
rhallisey:/20:37
rhalliseya bunch20:37
rhalliseyKubernetes supports the (optional) specification of a pre-stop lifecycle hook, which will execute prior to sending SIGTERM.20:37
*** schwicht has quit IRC20:37
*** awiddersheim has quit IRC20:37
kfox1111rhallisey: yeah. I use prestop with my compute clusters.20:37
rhalliseynice20:38
kfox1111lets the compute container drain the job safely before termination.20:38
kfox1111you gota change the default destroy timeout too.20:38
rhalliseythe name operator gets another strike.... operator.py is a base module...20:38
kfox1111it defaults to 30 seconds. wich is way to small.20:38
*** ppalacios has quit IRC20:38
rhalliseyya20:38
kfox1111the ugly in it is its per container, not per pod.20:38
kfox1111so you have to build some orchestration logic between containers in a pod if they need to work together during shutdown.20:39
kfox1111see our common-deployment yaml.20:39
kfox1111has some of that in it.20:39
rhalliseyya all pieces to the puzzle20:40
kfox1111I put in a request of a way to make the prestop hook blocking for all containers rather then just the one its attached to.20:40
kfox1111we'l see if t hat goes through.20:40
kfox1111doh..20:41
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP helm  https://review.openstack.org/39629620:42
*** eaguilar has joined #openstack-kolla20:45
*** sdake has joined #openstack-kolla20:46
sdakesup cats20:46
rhalliseysup20:47
inc0need someone with puppet knowledge20:47
inc0for profit of kolla20:47
*** kbyrne has quit IRC20:49
*** kbyrne has joined #openstack-kolla20:52
*** severion has joined #openstack-kolla20:57
severionhey sdake20:57
severioncrap one sec.20:57
*** severion has quit IRC20:57
*** severion has joined #openstack-kolla20:57
severionbtw...severion is v1k0d3n away :)20:57
*** v1k0d3n has quit IRC20:58
*** ayoung has quit IRC20:59
*** v1k0d3n has joined #openstack-kolla21:05
kfox1111there we go....21:05
kfox1111helm install kolla/neutron-l3-agent --version 2.0.2-1 --set enable_kube_logger=false,type=network --namespace kolla --name neutron-l3-agent-network21:05
*** severion has quit IRC21:06
jemcevoysean-k-mooney, Are you at Intel in Hillsboro?  There is a OpenStack PDX meetup at Puppet tonight at 6pm...21:09
kfox1111k. now to try a second microservice and a service package.21:09
inc0jemcevoy, he's pretty far away, Ireland21:11
inc0I'm moving to Hillsboro in a month tho:)21:12
inc0so I'll be around these meetups21:12
jemcevoyExcellent...21:12
jemcevoyinc0, where are you moving from?21:13
inc0san antonio21:13
jemcevoyMy inlaws were in Boerne but have passed on...21:14
*** sdake_ has joined #openstack-kolla21:23
inc0afk, need to go to store21:24
*** inc0 has quit IRC21:24
*** severion has joined #openstack-kolla21:25
*** msimonin has quit IRC21:26
*** schwicht has joined #openstack-kolla21:26
*** sdake has quit IRC21:27
kfox1111adding a second tempalte wan't too hard...21:27
*** jtriley_ has joined #openstack-kolla21:27
*** v1k0d3n has quit IRC21:27
*** goldyfruit has quit IRC21:27
*** goldyfruit has joined #openstack-kolla21:28
*** jtriley has quit IRC21:30
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP helm  https://review.openstack.org/39629621:32
*** severion has quit IRC21:32
*** jtriley_ has quit IRC21:32
*** fguillot has quit IRC21:33
*** v1k0d3n has joined #openstack-kolla21:33
*** Jeffrey4l has quit IRC21:33
*** msimonin has joined #openstack-kolla21:45
*** msimonin has quit IRC21:45
*** msimonin1 has joined #openstack-kolla21:45
*** v1k0d3n has quit IRC21:46
*** schwicht has quit IRC21:49
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Kubernetes Operator base  https://review.openstack.org/39926221:56
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Mariadb Kubernetes Operator  https://review.openstack.org/39926321:56
*** egonzalez90 has joined #openstack-kolla21:59
*** TxGirlGeek has quit IRC22:00
kfox1111wow.. that kind of seemed to work...22:05
*** lrensing has quit IRC22:05
*** Pavo has quit IRC22:07
*** srwilkers has quit IRC22:07
*** Pavo has joined #openstack-kolla22:08
*** jheroux has quit IRC22:08
kfox1111so, building a meta package mostly worked.22:09
kfox1111the vars stuff isn't quite working though... weird.22:09
*** rhallisey has quit IRC22:10
*** rhallisey has joined #openstack-kolla22:10
kfox1111hmm...22:10
kfox1111though I can see a way this might never work without operators... :/22:11
kfox1111you'd want an instance that had one neutron-server but maybe multiple l3 agents subinstances.22:12
*** TxGirlGeek has joined #openstack-kolla22:16
*** rhallisey has quit IRC22:17
sdake_kfox1111 now we have to prove that operators actually work :)22:17
kfox1111yeah.22:18
kfox1111you may want different sets of instances too, depending on if you want dvr enabled or not. and how many nova compute types you have...22:19
*** inc0 has joined #openstack-kolla22:23
*** srwilker has joined #openstack-kolla22:27
kfox1111hmm.. and this time it rendered totally wrong. :/22:27
kfox1111seems like this is racy. :/22:27
inc0kfox1111, helm?22:27
inc0how template engine can be racy?22:27
*** krtaylor has quit IRC22:30
kfox1111helm.22:31
kfox1111the way I'm doing macros...22:32
kfox1111it orders them in a random order.22:32
kfox1111and I had some macros with the same name.22:32
kfox1111which works for independent packages,22:32
kfox1111but when you merge in the working packages into a nested package, it breaks.22:32
inc0I blame goroutines22:34
sdake_inc0 good thing to blame22:34
sdake_inc0 threads re an abomination22:34
kfox1111python might have the same issue.22:34
sdake_python is single thread with eventlet22:34
kfox1111just throwing it all in a hash, then processing htem.22:34
inc0kfox1111, python isn't pushing multithreading as much22:34
kfox1111thats pretty random too.22:34
kfox1111I'm guessing its a hashing issue.22:34
sdake_kfox1111 posissble22:35
inc0then every time it would be the same22:35
kfox1111it outputs linearlyk, so I doubt it uses a goroutine.22:35
inc0strange, but same strange22:35
kfox1111inc0: no. python does the same. load a json document in python and print it out.22:35
kfox1111it comes out random ordered.22:35
inc0but order will always be the same22:35
inc0not random22:36
kfox1111not nessisarily.22:36
inc0not same as input, but 100 times you run it, it'll always came out same22:36
kfox1111I don't belive thats garanteed.22:36
kfox1111likely, but not garanteed.22:36
inc0hash doesn't change no matter how much you run it22:36
sdake_depends on what else gets hashed with it22:37
kfox1111the contents don't but the memory struture has no garantee.22:37
sdake_if the ordering is inconsistent sounds like a bug to me22:37
inc0ok, if you use timestamp as key then meh22:37
kfox1111anyway. doesn't matter. it has no well defined behavior here, so I have to do something differnt..22:38
kfox1111should be able to just extend things a little bit and just ensure I don't use the same macroname twice.22:38
*** reek269 has quit IRC22:38
kfox1111ok... got it to work.22:41
kfox1111so, this concept might work for things that don't need multiple subinstances.22:41
kfox1111I'll contribute the code, and those wanting to play with entrypoint only can run with it.22:41
kfox1111and the operator folks can use the base packages.22:42
*** TxGirlGeek has quit IRC22:43
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP helm  https://review.openstack.org/39629622:43
inc0one more +2 and we have voting gates https://review.openstack.org/#/c/398534/ << runs away >>22:46
sdake_running is appropriate22:46
sdake_inc0 which repo btw22:47
sdake_kolla repo?22:47
inc0yes22:47
inc0source builds for major distros22:47
sdake_cool22:47
sdake_i'd leave binary nonvoting for now ;)22:47
kfox1111inc0: nice.22:47
inc0yeah22:47
kfox1111we may want to do the same for kolla-kubernetes gates soon.22:47
kfox1111they seem pretty stable.22:48
sdake_maybe dmsimard can pull a rabbit out of his hat to make sure we have cross repo gating22:48
inc0source builds aren't that volitale22:48
sdake_then voting binary could be a thing22:48
inc0speaking of dmsimard, 3rd party CI for RHEL would be awesome;)22:48
*** f13o has quit IRC22:52
*** v1k0d3n has joined #openstack-kolla22:54
kfox1111ok.... I think this will fly.23:04
kfox1111its passing all the tests.23:04
kfox1111should get more eyes on it.23:04
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP helm  https://review.openstack.org/39629623:05
openstackgerritBorneMace proposed openstack/kolla-ansible: Added support for a stop playbook in kolla  https://review.openstack.org/39928923:06
*** goldyfruit has quit IRC23:10
*** ayoung has joined #openstack-kolla23:10
*** egonzalez90 has quit IRC23:11
*** liyifeng has joined #openstack-kolla23:14
kfox1111his is ready for a workflow: https://review.openstack.org/#/c/396812/23:15
inc0I'm going off, have a good one guys23:17
alanmeadowswe're piloting a ceph chart which may be of interest: https://github.com/att-comdev/aic-helm23:17
*** inc0 has quit IRC23:17
kfox1111v1k0d3n: have a gerrit account?23:19
kfox1111alanmeadows: is this another project to openstack on k8s?23:21
v1k0d3nyup23:22
v1k0d3noh i mean yes to kfox1111 your questions.23:23
v1k0d3nquestion...man i can't type or even think today.23:23
v1k0d3nso kfox1111 this isn't meant to compete at all with kolla...23:23
v1k0d3nwe're using this to test out, make sure it's solid...and plan to contribute and push this all into kolla.23:23
*** sbezverk has joined #openstack-kolla23:24
v1k0d3nat the end of the day for us...kolla is the be-all...that's where it's at.23:24
kfox1111nice. k.23:24
kfox1111though the traditional workflow for openstack projects has been to develop in tree in regular reviews.23:24
v1k0d3nwe will obviously have our own deployment, but if kolla is an agnostic view of what we need....this is desired for us.23:25
kfox1111makes it much easier to review/merge the stuff.23:25
v1k0d3ni get it totally.23:25
v1k0d3nwe're engaging other resources though too, who are not openstack.23:25
v1k0d3nso they can look at github a little easier and quicker than gerrit23:25
kfox1111ah. sure.23:25
v1k0d3ni'll be honest...not a gerrit whiz. github or a gitflow workflow...i feel right at home.23:26
kfox1111so, I'd recomend ensuring reviews are going as soon as pssible on the kolla side. its been really hard to deal with large code drops. :/23:26
*** sbezverk_ has quit IRC23:27
kfox1111yeah. its a bit different between the two.23:27
kfox1111I kind of prefer gerrit after having used both.23:27
v1k0d3ni plan on doing reviews of your charts though. have just been caught up with setting up our development lab. after next week should be much, much better.23:27
v1k0d3nwell... gerrit is amazing for review process.23:27
v1k0d3nthis part....is awesome. anyone can jump in and help.23:27
v1k0d3nand voting system is great too.23:28
kfox1111v1k0d3n: cool. if you can review https://review.openstack.org/#/c/396296/ and have some other folks look at it, that woudl be great.23:28
v1k0d3nbottom line, it's perfect for openstack.23:28
v1k0d3nwas going to pull tonight and have a look.23:28
kfox1111I think its an ok attempt, and deploys, but not sure if there will be other issues with it.23:28
kfox1111cool. thanks. :)23:29
v1k0d3nkfox1111: remember we were talking about a "seed host" concept using ironic/bifrost?23:29
kfox1111I'm a little surprised k8s doesn't use gerrit.23:29
kfox1111considering google wrote it.23:29
kfox1111v1k0d3n: yeah.23:29
kfox1111I see maas mentioned there. which is kind of a compeditor to ironic as far as I understand it.23:30
v1k0d3nwe're not a huge ironic shop. we ended up writing this ( alanmeadows  did ) https://github.com/att-comdev/dockerfiles/tree/master/maas23:30
kfox1111and ubuntu specific.23:30
v1k0d3nnow this ^^ is really cool. been using it today.23:30
kfox1111yeah. its interesting. but I probably wont/cant use it. :/23:31
*** ntpttr has quit IRC23:31
*** Jeffrey4l has joined #openstack-kolla23:31
kfox1111my goals to be able to be able to have an undercloud that can be used to deploy vm's on,23:31
kfox1111or let the users deploy bare metal stuff.23:31
*** ntpttr has joined #openstack-kolla23:31
kfox1111the only tool I know of that can do that is ironic. :/23:31
kfox1111where ironic has a tenant for the overcloud.23:32
kfox1111and then the rest of the normal tenants can use it too.23:32
*** krtaylor has joined #openstack-kolla23:32
kfox1111if there was a nova maas driver though, that might work.23:33
v1k0d3ni get it.23:35
alanmeadowswe have complex networking needs that ironic doesn't really fulfill  right now23:36
alanmeadowsbut thought the ceph chart could be of some value23:36
*** lamt has quit IRC23:37
kfox1111yeah. I'll ahve a look. thanks. :)23:37
kfox1111there's another ceph on kubernetes project I saw somewhere.... where was that...23:38
alanmeadowsthere are some instructions baked into ceph-docker for k8s23:38
alanmeadowshowever, this is the first ceph chart I am aware of23:38
kfox1111https://github.com/ceph/ceph-docker/tree/master/examples/kubernetes23:38
kfox1111yeah.23:38
kfox1111ah. helm. yeah.23:39
kfox1111true. I don't think I've seen any charts.23:39
kfox1111cool. :)23:39
*** sdake_ has quit IRC23:39
v1k0d3ni actually think that we can get rid of some of the prep too (possibly) by launching a container that can do some of the prep locally on the host.23:40
v1k0d3nhave to really think about this though.23:40
alanmeadowstwo fold, part of it is the need for a "sidecar" container, to lessen requirements on the host23:41
kfox1111alanmeadows: have a look at the ceph stuff in kolla-kubernetes. Its mostly just for testing,23:41
kfox1111but there is one thing I think is pretty clever.23:41
kfox1111the ceph-rbd object.23:41
kfox1111https://github.com/openstack/kolla-kubernetes/blob/master/services/ceph/ceph-rbd-pod.yml.j223:42
alanmeadowsand the second is the helm lack of any kind dynamic run-time configuration and its inability to gobble up whole directories for configmaps and especially secrets23:42
kfox1111makes it so you don't have to install anything on the nodes.23:42
*** dave-mccowan has quit IRC23:42
alanmeadowswhich is why our ceph chart has some Makefile magic to sidestep23:42
alanmeadowsbut ultimately that would go away23:42
kfox1111yeah. I think as helm starts getting used, they will quickly see the gaps.23:42
alanmeadowsexactly23:43
kfox1111now that its released, it should go faster. :)23:43
alanmeadowswe want to help with some of that - as soon you step outside an nginx example23:43
alanmeadowsyou really start to see them23:43
* kfox1111 nods.23:43
alanmeadowsluckily they are reasonably small gaps23:43
v1k0d3n^^ yup just all of that23:43
v1k0d3nexactly23:43
alanmeadowseasy features to close23:43
kfox1111alanmeadows: when you get a chance can you weigh in on https://review.openstack.org/#/c/396296/ too?23:43
alanmeadowsI can't include a file and base64 encode it, really?23:43
alanmeadowsyou dont clean up secrets, hmm...23:43
kfox1111would love some feedback on just how crazy I am. :)23:43
alanmeadowschecking23:44
kfox1111yeah. I had the same issue with the jinja2 stuff we were doing.23:44
kfox1111had to add some pretty common stuff to do the same. base64 wasn't built in. :/23:44
alanmeadowsthe rbd-pod is interesting23:46
kfox1111I did it for minikube support. it didn't supprot ceph tools on the host.23:46
kfox1111the other advantage is the ceph client then exactly matches whats in the rest of the containers.23:50
*** v1k0d3n has quit IRC23:50
*** v1k0d3n has joined #openstack-kolla23:51
*** rhallisey has joined #openstack-kolla23:51
alanmeadowsyup thats how it should be23:52
alanmeadowsi like some of the ideas in 3the WIP23:52
alanmeadowsI will try and comment this evening23:53
alanmeadowsi like how you're tackling neutron first ;-)23:53
kfox1111alanmeadows: cool. thanks for looking at it/reviewing. :) the help's appreciated. :)23:53
kfox1111heh. hard stuff to prove out the concept.23:53
kfox1111down hill from there. :)23:53
alanmeadowsthats whats necessary23:53
kfox1111I hate spending a bounch of time on something that works for easy stuff, then have to rewrite it all when something tough proves the thing was a bad idea. :/23:54
kfox1111so I try and prototype the really hard stuff. :)23:55
alanmeadowshehe, see it break quickly to realize your path was flawed all along23:55
* kfox1111 nods23:55
kfox1111and thats also why I'm seeking feedback, as I think it will fly, and I think I got most of the big gotcha's. but there may be some I didn't see yet and others have.23:55
*** v1k0d3n has quit IRC23:56
kfox1111I think it will be pretty easy to add kubernetes-entrypoint to in this arangement too.23:56
kfox1111can add it to common-lib.23:56
kfox1111then just include it where needed.23:56
alanmeadowsthats what I'm trying to pioneer as well (wrt entrypoint) in aic-helm23:56
alanmeadowsalso one thing I haven't really played with23:56
kfox1111cool. :)23:56
alanmeadowsis just how well upgrade works ;-)23:57
alanmeadowsfor the complex scenarios23:57
kfox1111yeah. I'm a bit skeptical of helm being able to handle that piece. :/23:57
alanmeadowsi don't know enough yet to say23:57
alanmeadowsjust that I share the same skepticism23:57
kfox1111I've done many upgrades of openstack systems over the years.23:57
kfox1111haven't lost a vm yet. :)23:57
kfox1111but it took a whole lot of kergiggering.23:57
kfox1111not sure helm's concept of upgrades aligns perfectly with openstacks view of them. :/23:58
kfox1111but we're trying to tool ourselves so that we're really flexible,23:59
alanmeadowswell I think it mostly relies on k8s functionality23:59
kfox1111and able to use whichever worklfow tools are best for each site.23:59
alanmeadowsand where I sort of seeing it sitting23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!