Wednesday, 2016-11-16

*** sdake has joined #openstack-kolla00:03
*** chas_ has quit IRC00:03
*** tonanhngo has quit IRC00:10
*** absubram has quit IRC00:13
*** portdirect_ is now known as portdirect_away_00:16
*** TxGirlGeek has quit IRC00:18
*** tonanhngo has joined #openstack-kolla00:20
*** hfu has joined #openstack-kolla00:20
*** rhallisey has joined #openstack-kolla00:22
*** tonanhngo has quit IRC00:25
*** lamt has quit IRC00:25
rhalliseyhey00:27
rhalliseywhat's around got some good news00:27
kfox1111yeah?00:27
rhalliseys/what's/who's/00:27
kfox1111could use some. :)00:27
rhalliseywe have a slack channel now in kubernetes00:27
rhallisey:)00:27
kfox1111cool.00:28
kfox1111hook me up. :)00:28
kfox1111I use slack alot for work now.00:28
*** hfu has quit IRC00:28
*** TxGirlGeek has joined #openstack-kolla00:29
kfox1111#kolla00:29
kfox1111foudn it. :)00:29
rhalliseycan you get in it?00:29
rhalliseysweet00:29
kfox1111yup.00:29
rhalliseyI think we can mirror it00:29
*** hfu has joined #openstack-kolla00:29
kfox1111cool.00:29
*** lamt has joined #openstack-kolla00:30
kfox1111try someting. ping me again by name please.00:30
rhalliseywith this channel00:30
kfox1111k. cool. thanks. :)00:30
rhalliseynice you get the notification00:31
rhalliseyeven when not around hehe :)00:31
kfox1111with great power.... ;)00:31
kfox1111but, yeah. :)00:31
Pavoslack is awesome00:32
rhalliseyPavo, we got a channel now :)00:32
Pavomay I join?00:32
kfox1111Pavo: looks like its an open channel.00:32
Pavolink?00:32
kfox1111I was able to join without invite.00:32
kfox1111kubernetes.slack.com00:33
kfox1111#kolla00:33
*** newmember has quit IRC00:33
*** TxGirlGeek has quit IRC00:33
*** TxGirlGeek has joined #openstack-kolla00:34
*** yingjun has joined #openstack-kolla00:34
*** portdirect_away_ has quit IRC00:34
*** portdirect_away_ has joined #openstack-kolla00:35
*** TxGirlGeek has quit IRC00:35
Pavohmm how do I sign up00:37
jascott1goto kubernetes.slack.com iirc00:37
jascott1oops00:38
jascott1here it is Pavo http://slack.k8s.io/00:38
Pavoyeah i did and used a username and password that I use on another team00:38
*** yingjun has quit IRC00:38
Pavothx00:38
*** yingjun has joined #openstack-kolla00:39
*** YuYangWang has joined #openstack-kolla00:46
rhalliseysdake, yo00:49
*** jtriley has joined #openstack-kolla00:59
sdakewho daresh ringith me whilth i sleepith01:00
sdakesup rhallisey01:00
rhalliseysdake, got slack channel01:00
rhalliseygoing to get it mirrored to irc01:01
sdakenice job rhallisey01:02
sdakesee it didn't even take me to do anything ;)01:02
*** duonghq has joined #openstack-kolla01:02
duonghqmorning guys01:02
sdakethe mirroring how will that work01:02
sdakebtw rhallisey01:02
sdakei think you are missing a potential contributor01:02
rhalliseyhttps://sameroom.io/blog/connecting-a-channel-in-slack-to-a-channel-irc/01:02
sdakearound duong's name01:02
duonghqjust seen new kolla-ansible repo, great things01:02
duonghqdid I miss something?01:03
rhalliseykk01:03
kfox1111this requires all users to setup the binding?01:03
rhalliseyI'm not sure01:05
v1k0d3noh rhallisey you got a kolla channel set up in kubernetes?01:05
rhalliseyya dude01:05
v1k0d3nwow dude...good job!01:05
v1k0d3nthat's great01:05
rhalliseyand we can mirror it here01:05
v1k0d3nwho did you you have to work with on that?01:06
rhalliseysarah01:06
v1k0d3nthought so.01:06
v1k0d3nshe rocks01:06
*** jtriley has quit IRC01:06
rhalliseyya she's been very helpful01:06
portdirect_away_+101:06
*** zhubingbing has joined #openstack-kolla01:08
sdakei'm glad you didn't need a face to face hand for that at cncf rhallisey01:08
*** portdirect_away_ is now known as portdirect_01:08
sdakeshe was literally busy 24/701:08
duonghqI as understand, we have a channel in slack k8s?01:08
rhalliseysdake, ya it worked out01:08
rhalliseyduonghq, that's right :)01:09
sdakeduonghq right - idea is to create a footprint over there for first deployment help as we do now01:09
sdakerhallisey are there limitiations on what else we may use the channel for?01:09
rhalliseyno, but we need to keep in mind the opernstack-kubernetes sig01:09
portdirect_v1k0d3n: just saw you added suport for centos in halcyon  - awesome, I could add atomic tomorrow if your up for a pr?01:09
sdakeright - I did talk to sgordon about that01:10
sdakehe seemed not to mind01:10
openstackgerritJoshua Harlow proposed openstack/kolla: Allow for specifying a KOLLA_CONFIG_FILE  https://review.openstack.org/39800801:10
sdakei'm not sure what i agreed to if anything :)01:10
Pavokfox1111 does this setup script run on the target nodes or the deployer node?01:10
rhalliseysince this channel will help us figure out issues/features for openstakc on kube01:10
rhalliseywe just want to keep it in mind01:11
sdakei think i basically said we weren't going to tromping on his territory01:11
kfox1111Pavo: the primary.01:11
Pavoprimary=deployer?01:11
sdakeby keeping things kolla focused01:11
rhalliseyyup01:11
kfox1111not that there is much distinction in the gate currently. it just gives you 2 nodes, then runs your script on one.01:11
kfox1111the script then scp/ssh runs a few things on the other node.01:11
kfox1111when multinode is enabled.01:11
kfox1111making it a compute node.01:11
*** tonanhngo has joined #openstack-kolla01:12
Pavoah ok so primary=controller and others=computes01:12
kfox1111for the instructions though, not sure that part should be documented? should assume a working k8s is already in existance?01:12
*** tonanhngo has quit IRC01:12
kfox1111(or have a seperate document for deploying a multinode k8s?)01:13
Pavowould love to have the docs also include how to setup k8 and get it ready for using kolla-k801:13
Pavofrom scrath to end01:13
Pavoor from scratch to fully operational01:13
kfox1111yeah. just keep those seperate I think as there's lots of ways to do k8s deployments.01:14
kfox1111and they seem to change all the time. :/01:14
portdirect_I think we should prob have a recommended deployment in the docs, but the setup should be out of scope - to make less burdon for us01:14
kfox1111kubeadm is the new hottness.01:14
kfox1111but doesn't do ha yet.01:14
Pavoso you are saying I have to setup k8 first before even start using this script01:14
*** tovin07_ has joined #openstack-kolla01:14
kfox1111so I use it in the gate, but not for any of my production systems.01:14
kfox1111Pavo: no, that script actually uses kubeadmin to deploy a kubernetes system.01:15
Pavooh ok01:15
kfox1111just saying, for the mulitnode deployment docs, the doc should describe how to use an existing k8s cluster.01:15
kfox1111and a seperate doc maybe for deploying a k8s for those that want help there.01:15
duonghqI tried kubeadm, it's fine for testing but not production01:15
kfox1111so if you already got a favorite k8s deployment tool, you can skip the k8s doc.01:16
duonghqno long-running test, also01:16
portdirect_very few tools are production cabable atm :(01:16
kfox1111long term, I think kubeadm is going to be a very nice tool. but gots a ways to go though.01:16
duonghqpretty sure, doubt that many people craft so-call in-house tools01:17
sdakei had heard they were working on a rewrite of deployment in ansible01:17
kfox1111heh. the wheel turns. :)01:17
portdirect_yep - i've been playing with https://github.com/att-comdev/halcyon-vagrant-kubernetes for dev stuff over the last few days - and it's pretty nice for setting up a quick testing env in vagrant.01:17
portdirect_(that yep made no sense with the posts between.. )01:18
duonghqkfox1111, kubeadm is very nice thing but still lacks of recovery ability from master crash01:18
kfox1111right.01:18
kfox1111it'll get there though. pretty sure.01:18
kfox1111its really easy to get a gate job workign with it though. :)01:19
duonghqsure, it's suitable for short-term testing like gate01:19
duonghqbut do you notice that kubeadm master eat much IOPS?01:20
duonghqI tested on 3 nodes on kvm01:20
Pavonot gonna lie though that stackanetes looks pretty sweet01:20
sdakepavo try it out let us know what you think :)01:21
portdirect_pavo: id recommend looking at https://github.com/portdirect/harbor as well, but then I would say that :)01:22
Pavosdake I would if there was a guide for CentOS 701:22
Pavostackanetes uses CoreOS from my undersanding01:23
sdakeportdirect_ didn't know you were so far along01:23
Pavoportdirect_ atleast yours is written fro CentOS01:24
kfox1111sdake: yup. portdirect_ has a lot of good stuff in there. :)01:24
portdirect_yeah - its very crude atm, but I've not had time to do much since barca01:25
kfox1111if we can get everyone on the same page and working on the same project, there will be some really amazing things pop out. :)01:25
PavoI agree01:26
Pavoand I have a whole shop that can test and give feed back daily01:26
kfox1111nice. :)01:26
Pavothats our job mainly01:26
PavoNetworking, Engineering, Research & Development01:27
Pavoany guess what are shop is called?01:27
Pavo:)01:27
kfox1111a few. :)01:27
PavoNERD01:28
Pavolol01:28
kfox1111:)01:28
rhalliseywell idk if the mirroring will work :)01:30
*** zhurong has joined #openstack-kolla01:30
rhalliseymaybe we could mirror slack to irc01:30
rhalliseyeh01:31
rhalliseynot ideal01:31
rhalliseyone way01:31
kfox1111yeah. I wondered if the mirroring would work. :/01:32
rhalliseyI was really hoping for it01:32
rhalliseyI think they will fix it though01:33
rhalliseyif they go through 10 in 2 days01:33
kfox1111yeah01:33
rhalliseyslack is different that irc01:33
rhalliseythere isn't a live chat here01:33
rhalliseywhen you aren't here you miss it01:33
rhalliseyneed to read logs01:33
Pavounless you use a bouncer like me01:34
kfox1111or leave it in screen. :)01:34
Pavobut only buff 1000 lines01:34
rhalliseyindeed01:34
kfox1111yeah.01:34
rhalliseyI'm lame01:34
kfox11116 of one.01:34
rhalliseyI just read logs01:34
kfox1111either way I think we need logs anyway.01:34
kfox1111cause we want history more then a week back at times.01:35
Pavoyeha01:35
Pavoyeah01:35
rhalliseykfox1111, we'll have the history01:35
rhalliseyit will all log on this side01:35
rhalliseyit's the live chat01:35
kfox1111what live chat are you refering to.01:36
rhalliseykube slack can only carry 10k live messages at a time01:36
rhalliseyso say one channel was spamming 10k messages/second01:36
sdakerhallisey this should give you flashbacks: https://github.com/portdirect/harbor/blob/latest/tools/build-docker-image#L13501:36
rhalliseyanything you right will not appear in the scrollback01:36
kfox1111heh.01:37
rhalliseysdake, lul01:37
kfox1111are they that close to the limit?01:37
rhalliseycrudini though01:37
rhalliseykfox1111, don't think so01:37
rhalliseylet me see what we get in a day01:37
rhalliseyidk how I can count this..01:38
*** Pavo has quit IRC01:38
kfox1111gotta head out.01:38
rhalliseysdake, reminds me of when we had crudini and sef everywhere01:38
kfox1111l8r.01:38
*** Pavo has joined #openstack-kolla01:38
rhalliseykfox1111, so you think :).  Now we have you in slack!01:38
kfox1111;)01:38
kfox1111with great power... ;)01:38
rhalliseyya01:38
rhalliseydont mirror and have a bot destory the cache01:39
kfox1111(and a two way street. ;)01:39
rhallisey:)01:39
rhalliseyhehe01:39
kfox1111but, yeah. ping me if you need me.01:39
rhalliseyI'm kidding hehe01:39
kfox1111I just reserve the right not to respond right away. :)01:39
*** schwicht has joined #openstack-kolla01:40
*** hfu has quit IRC01:40
rhallisey2 messages a minute is 2,880 messages per day01:41
*** dt-turner has joined #openstack-kolla01:42
portdirect_rhallisy/sdake: even more flashback here :) https://github.com/portdirect/harbor/blob/latest/docker/common-assets/opt/harbor/harbor-common.sh#L501:42
rhalliseylol01:42
rhalliseythat is a major throw back01:43
rhalliseykolla had the containers and I was looking at deploying all different ways01:44
*** dt-turner has quit IRC01:44
portdirect_dont think I actually use any of it anymore - but never got round to removing it :)01:44
rhalliseythat's pretty cool :)01:45
*** DaveTurner has joined #openstack-kolla01:45
sdakeya that shell code stuff was madness01:45
sdakebut eh, like i said earlier zero exposur eto docker when the project was started01:45
sdakeobjective was to teach a bunch of peeps how to do deploy on openstack01:46
rhalliseymy guess is we're about 2 messages per minute01:46
rhalliseyin this channel01:46
rhalliseyincluding people signing in an out01:47
*** hfu has joined #openstack-kolla01:50
*** lamt has quit IRC01:57
openstackgerritcaoyuan proposed openstack/kolla: Move the "enable_destroy_images" into configure file  https://review.openstack.org/39763301:58
openstackgerritZeyu Zhu proposed openstack/kolla: Modify the letter case in post-install.yml file  https://review.openstack.org/39802401:59
*** rhallisey has quit IRC02:01
*** Pavo has quit IRC02:01
openstackgerritSteven Dake proposed openstack/kolla: Remove Ansible from Kolla  https://review.openstack.org/39802502:02
openstackgerritSteven Dake proposed openstack/kolla: Clean up tox.ini for Ansible removal  https://review.openstack.org/39802802:06
*** huikang has joined #openstack-kolla02:08
*** Pavo has joined #openstack-kolla02:08
*** schwicht has quit IRC02:08
Pavofinally02:08
Pavogot my stack finally port forwarding working02:09
Pavoddi.hopto.org02:09
portdirect_Pavo: what are you plaiing on using OpenStack for (if you can say)?02:09
*** yingjun has quit IRC02:10
*** schwicht has joined #openstack-kolla02:10
Pavowell we use it as a infrastructure mostly02:10
*** yingjun_ has joined #openstack-kolla02:10
Pavoto do defensive cyber operations from02:11
*** schwicht has quit IRC02:12
*** yingjun_ has quit IRC02:12
*** yingjun has joined #openstack-kolla02:12
Pavokfox1111 what about this guide02:13
Pavohttp://severalnines.com/blog/installing-kubernetes-cluster-minions-centos7-manage-pods-services02:13
Pavothink its good enough to start with?02:13
portdirect_nice, be interesting to get your perspective's/requirements from a security perspective as the develop - I'm quite interested in getting a PCI-DSS comiliant setup out of the box, but I assume your working to FedRAMP or similar?02:15
*** harbor has joined #openstack-kolla02:16
Pavowell that would be perfect02:16
Pavofirst look into FIPS-14002:17
duonghqI think we should sum up k8s deployment tool and put it in some wiki or notes,02:17
duonghqtool and guide02:17
harboryeah been there :)02:17
duonghqdue to I had hard time to deploy k8s02:17
duonghqI think many people also suffer02:17
Pavoand then look into http://iase.disa.mil/stigs/os/unix-linux/Pages/index.aspx02:17
Pavoduonghq +102:18
*** yingjun has quit IRC02:18
Pavoactually02:18
Pavoduonghq +10000000002:18
*** yingjun has joined #openstack-kolla02:18
*** yingjun has quit IRC02:18
*** yingjun has joined #openstack-kolla02:18
openstackgerritcaoyuan proposed openstack/kolla: Update the default value for Heat  https://review.openstack.org/39757702:19
*** coolsvap has joined #openstack-kolla02:19
harborok are you guys using vagrant or similar?02:19
Pavoyou asking me harbor ?02:20
*** harbor is now known as portdirect__02:20
*** portdirect_ has quit IRC02:20
portdirect__whoops, I'm back, dodgy internet here02:20
portdirect__yeah that was me asking :)02:20
Pavonot at all02:20
PavoCurrently using CentOS 7 with packstack02:21
Pavoand deploying with a custom script02:21
portdirect__is your packstack running heat?02:21
Pavobut as well all know packstack is limited to 5 nodes02:21
Pavoand yes we are using heat02:22
Pavowhich is are main focus right now02:22
Pavoto build heat temps for different senarios based off mission02:22
portdirect__Pavo: http://kubernetes.io/docs/getting-started-guides/openstack-heat/02:23
Pavois this a heat temp for k8 inside OS?02:23
Pavolooks like a guide to install Openstack02:24
portdirect__yup02:24
*** jtriley has joined #openstack-kolla02:25
portdirect__it's the official k8s docs for running kube in OS via heat.02:25
Pavowill take a look into it but I am mainly looking to switch from packstack to either kolla-ansible or kolla-k8 for our deployment method now02:25
DaveTurnerHey folks.  Question.  My team runs daily CICD builds / deployments of multinode Kolla clusters and we've recently updated our configuration to segregate API traffic.02:26
DaveTurnerOn the target nodes we have a management and api vlan configured with IP's plumbed to both.  On the Kolla side, we've specified the api vlan interface for the  "api_interface" and the management vlan interface as the "network_interface" in globals.yml.02:26
Pavomain aspects that we are looking at kolla for are ease of deployment, configuration and upgradability02:26
DaveTurnerDuring deployment, all services seem to listen on / communicate over the api vlan, however rabbitmq fails to start successfully.  The rabbit containers continue to restart on all nodes.02:26
portdirect__Pavo: id advise kolla-ansible for now if you need production right away02:27
DaveTurnerWe are running about 2 weeks behind master.  I noticed this bug: https://bugs.launchpad.net/kolla/+bug/1583655 which seems to touch on the issue, however we haven't been able to get around it via /etc/hosts entries.02:28
openstackLaunchpad bug 1583655 in kolla "rabbitmq setup from secondary IP" [Critical,Confirmed]02:28
*** hfu has quit IRC02:29
Pavowell thats the beauty of our setups, they aren't meant to be persistent, ie production. we stand them up, do our mission, move required data needed and purge the systems02:29
Pavorinse and repeat02:29
portdirect__coming from a somewhat offenisive youth, that makes sense - but you need a rock solid base, I assume you are talking bare metal nodes02:30
Pavoyes and there is another infruasture that we ride on02:31
*** hfu has joined #openstack-kolla02:32
sdakehi DaveTurner02:35
sdakereading scrollback moment pls02:35
sdakeDaveTurner couple things re running coule weeks behind master02:36
sdakefirst we have split the kolla repo into two02:36
sdakesecond, your running master openstack as well, not newton openstack, are you aware?02:36
sdakethird on the repo split, we have not yet got the repositories back into working order, although its in progress02:37
sdakefourth so 1 and 3 will effect your ci/cd pipeline of master unless you also make some changes to handle it once your running on master as of today02:37
DaveTurnerHi sdake.  Stepped away for a sec.. reading through your reply02:40
sdakeit was more to bring you up to speed on the current state of things02:40
sdakenot a direct answer to your question :)02:40
DaveTurnershucks.  ;-)02:40
sdakefor that i'll need your help02:40
*** chas has joined #openstack-kolla02:40
sdakeon any of th econtroller nodes can you run docker logs rabbitmq02:41
DaveTurnerWe are aware that we are running master openstack as well.  We are attempting to stay close to trunk.02:41
DaveTurnerSure....02:42
sdakeDaveTurner are you running kolla-ansible upgrade on each new ci/cd pipeline deployment?02:42
sdakeDaveTurner mind i ask how long you have been doing this - just to get some context if this is a new issue or old issue02:43
DaveTurnerWe reimage the baremetal between pipeline runs02:43
sdakeroger02:43
sdakei guess i'd ask why not use vms but its not my business ;-)02:44
DaveTurnerCertain cases we do.  But we have racks of systems to test for large scale deployments, so we try to stay as close to those proposed configs as possible.02:45
sdakeDaveTurner if i ask any q's you are unable to answer just say you can answer - this channel is alot about learning02:45
sdakehow long does your ci/cd pipeline take?02:45
*** chas has quit IRC02:45
DaveTurnerI'd say a couple of hours.02:46
sdakecool02:46
sdakethats pretty fast02:46
sdakei guess i'm nosey but i'm curious if your using kolla's bifrost implementation02:47
DaveTurnerNot yet - haven't had time to poke around at that.02:47
sdakecool if/when you do would love feedback02:48
sdakeits not well tested like the rest of the code base typically is02:48
DaveTurnerWe are using Kolla to build openstack centralize services cluster along with regional deployments to subscribe to those centralizes services - hence all the ci/cd work.02:48
DaveTurnerSure - can do (at least, someone from my team).02:49
sdakehmm i dont get it but thats ok:)02:49
DaveTurnerOh- logs.  One sec.02:49
sdakeright ;)02:49
sdakeDaveTurner are you in US tz or EMEA tz?02:50
sdakeor apac?02:50
DaveTurnersdake gathering info in pastebin - just a sec.02:51
DaveTurnerUS central / Austin02:51
sdakeif your on centos you can run sudo yum install fpaste02:51
sdakeand fpaste the file02:51
sdakeor pipe through fpaste02:51
sdakeubuntu doesn't have similar functionality02:51
DaveTurnerOracleLinux 7 - close enough.02:51
sdakecool - although not sure if ole has fpaste or not02:52
sdakeprobably does if you install the right repos ;)02:52
*** yuanying has quit IRC02:52
*** yingjun_ has joined #openstack-kolla02:53
*** yingjun has quit IRC02:53
DaveTurnerRight- pastebin for now  :-)  Will check that out later.02:53
DaveTurnersdak http://paste.openstack.org/show/589348/02:53
sdakeDaveTurner cool that looks good02:54
sdakeDaveTurner on the host02:54
sdakego to /var/lib/docker/volumes/kolla02:54
sdakenote volumes may be something else02:54
sdakeand kolla may b instead mariadb02:54
sdakeDaveTurner i can't check exact syntax atm because i am on corp vpn and dont' have access to my home lab boxes02:55
sdakecisco firewalls my home machine :(02:55
DaveTurnersdake one more log where I run status before the container bombs out: http://paste.openstack.org/show/589349/02:55
DaveTurnerunderstood02:55
DaveTurnerThat hostname is the main hostname configured for the management vlan.  For the api vlan, we've registered hostnames for the IP's plumbed to that interface as well.  format: hostname-api.02:56
*** unicell has quit IRC02:56
DaveTurnersdake: ok, in /var/lib/docker/volumes/kolla02:57
sdakeDaveTurner try cd /var/lib/docker/*volumes*/heka02:57
sdakesorry all mixed up today ;)02:57
DaveTurnersdake there02:58
sdakeyup i suggested a different commnad02:58
sdakei want to see your logs for rabbitmq02:59
DaveTurnerLooking for rabbit log entries?  I've yet to see any generated prior to the container restarting.02:59
sdakethey are stored in the heka volume02:59
DaveTurnerYep- I meant, I was there in the directory.  :-)02:59
sdakein heka there was no rabbitmq dir?02:59
*** yingjun has joined #openstack-kolla03:00
*** yingjun_ has quit IRC03:00
DaveTurnersdake I see _data under /var/lib/docker/volumes/heka03:01
sdakeok go to that dir pls03:03
sdakevia cd03:03
sdakeand tell me what ya got03:03
DaveTurnerlogstreamer  sandbox_preservation03:03
DaveTurnerUnder logstreamer, I see logs for our other services but nothing for rabbit03:04
openstackgerritZeyu Zhu proposed openstack/kolla: Modify the letter case in post-install.yml file.  https://review.openstack.org/39802403:04
sdakeDaveTurner what about another node?03:06
sdakeyour doing multicontroller gating?03:06
*** harbor has joined #openstack-kolla03:06
sdakeor single controller node03:06
DaveTurnermultiple controller nodes.  3 in this case.03:07
sdakecool can you check the other two nodes03:07
sdakeperhaps one of them has logs03:07
*** harbor is now known as portdirect___03:07
DaveTurnerI'll check the other 2.  I believe we have already, but worth checking again...03:07
*** chas has joined #openstack-kolla03:08
sdakeDaveTurner if that fails have a plan B:)03:08
sdakeand C03:08
*** portdirect__ has quit IRC03:09
DaveTurnersdake ok, no logs on the other 203:09
sdakebummer03:09
sdakeyour vlan mgmt interface is which ip?03:09
sdake(pick a node)03:09
sdakerabbitmq is a big pile of steam03:10
sdake#1 cause of failures03:10
DaveTurnersdake: host: slcas275, management vlan interface: eno1.2706@eno1 ip: 10.248.68.7103:11
DaveTurnerThat's the primary interface.  We have another defined for api.03:11
DaveTurnerWe didn't hit this with our multinode deployment until we tried to get cute with segregating api traffic onto separate vlan/interface03:12
sdakeunderstood03:12
*** chas has quit IRC03:12
sdakeDaveTurner right, and that is how openstack should be deployed too :)03:12
DaveTurnervia the *_interface options provided by kolla03:12
DaveTurnerRight!03:12
sdakeok on host slcaa275 can you run hostname03:13
DaveTurnerYup.  Returns slcas27503:14
sdakelet me read the bug03:14
sdakeand the code03:14
DaveTurnerSure.  Thanks for taking a look.03:15
sdakeDaveTurner no guarantees i can ge tthis rolling for you, I am not a networking dude03:15
sdakewhy be employee number 70001 at a networking company? :)03:16
DaveTurnerHa!03:16
sdakeDaveTurner can we crater these machines (i.e. use them for debug)03:17
sdakewith possibly irreparable dmg :)03:17
*** portdirect___ is now known as portdirect_away_03:17
DaveTurnersdake Sure.  My guys in India will be on in a couple of hours to rerun the pipeline on these systems.  They will be destroyed anyway.03:19
sdakecool so lets run docker stop rabbitmq on all the nodes03:19
sdakeif that fails try docker rm rabbitmq03:20
sdakeif that fials try docker rm -f rabbitmq03:20
DaveTurnerstopped03:20
sdakecool so pick a node and run docker rm -f rabbitmq03:21
sdakedocker images | grep rabbitmq -> paste03:21
sdakewhat we just did there was stop rabbitmq from rnaodmly restarting03:21
sdakeand then deleted a rabbitmq instance where we are going to do some proper debug03:21
DaveTurneryup. got it03:22
*** eaguilar has quit IRC03:23
sdakegroan a bug from may unfixed03:23
DaveTurnersdake disclaimer - we build from source and push to our own registry - in case you are trying to have me pull again from external registry.  :-)03:24
sdakenope03:24
DaveTurnercontainer gone03:24
sdakewe are starting an image03:25
sdakebut i need the image name to provide the proper command03:25
sdakeand also the tag03:25
DaveTurner# docker images | grep rabbitmq slc11gcw.us.oracle.com:5000/cas/oraclelinux-source-rabbitmq            next                e6091c4a4057        20 hours ago        384.7 MB03:25
sdakeok docker run -it -u root slcwhatever/cas/oraclelinux-source-rabbitmq:next bash03:25
sdakedont forget the :500003:26
sdakebsaically image:tag03:26
DaveTurneryep.. started03:27
DaveTurnerI'm in the container now03:27
sdakeok cat /etc/hosts03:27
sdakeand dump into channel if short otherwise paste03:27
DaveTurnerhmm.. should we have started with host networking?  This looks ugly03:28
DaveTurner::1localhost ip6-localhost ip6-loopback fe00::0ip6-localnet ff00::0ip6-mcastprefix ff02::1ip6-allnodes ff02::2ip6-allrouters 172.17.0.253d76df084b903:28
sdakewe are just looking at /etc/hosts03:28
sdakeok03:28
sdaketype exit03:28
DaveTurnerI'm out03:28
sdakecat /etc/sysconfig/network-scripts/ipcfg-eno103:29
*** chas has joined #openstack-kolla03:29
sdake(i think that is the interface you mentioned?)03:29
sdakeit might be ifcfg-eno103:29
*** huikang has quit IRC03:29
DaveTurnerThat's the interface on which both vlans are trunked.03:29
DaveTurnerYou want to see the config for eno1 interface, or the vlans based on that interface?03:30
sdakecan you cat the vlan interfac efile03:30
DaveTurner# cat ifcfg-eno1.2706 DEVICE=eno1.2706 BOOTPROTO="static" IPADDR="10.248.68.71" NETMASK="255.255.255.192" NETWORK="10.248.68.64" GATEWAY=10.248.68.65 USERCTL=no ONBOOT=yes VLAN=yes NOZEROCONF=yes DEFROUTE=yes03:31
DaveTurnerAnd the api vlan interface:03:31
DaveTurner# cat ifcfg-eno1.802 DEVICE=eno1.802 BOOTPROTO="static" IPADDR="10.248.66.212" NETMASK="255.255.255.192" NETWORK="10.248.66.192" USERCTL=no ONBOOT=yes NOZEROCONF=yes VLAN=yes03:31
sdakeok thats going to have to go in a paste pls03:31
DaveTurnerindeed03:31
sdakeas well as your main interface03:31
sdakelet me get off the vpn03:32
sdakeso i can access my bare metal gear03:33
sdakebrb while your pasting :)03:33
*** chas has quit IRC03:33
*** sdake_ has joined #openstack-kolla03:33
DaveTurnersdake Here ya go: http://paste.openstack.org/show/589352/03:34
*** tonanhngo has joined #openstack-kolla03:34
*** hfu has quit IRC03:35
*** hfu has joined #openstack-kolla03:35
*** duonghq has quit IRC03:36
sdake_how did you setup the vlans?03:37
sdake_manually or with a tool?03:37
*** sdake has quit IRC03:37
DaveTurnernot via tool. manually for this test03:38
*** tonanhngo has quit IRC03:39
sdake_ok03:39
sdake_can you run ip link show -> paste03:39
sdake_or another option would be ifconfig -> paste03:39
sdake_i think you have your system seutp for only ipv603:39
sdake_kolla does not work with ipv603:39
sdake_what is the lro and gro options to ETHTOOL?03:40
sdake_ifconfig and or ip lin kshow will tell us :)03:41
DaveTurnerNo.  We don't use ipv6 here.  LRO and GRO for large and generic receive offload.03:42
DaveTurnerWe had to mess with those setting back in icehouse when using vxlan for performance reasons03:42
DaveTurnerhttp://paste.openstack.org/show/589354/03:42
DaveTurnersdake: you can ignore the 803/804 vlans.  Those are there in case the nodes are used as network nodes.  Those are our public tenant networks.  Our dev nodes are configured to be used as any number of openstack node types.03:44
*** duonghq has joined #openstack-kolla03:45
DaveTurnersdake: and ip addr output: http://paste.openstack.org/show/589355/03:45
*** fragatin_ has joined #openstack-kolla03:47
sdake_DaveTurner there are multiple problems here03:48
sdake_one is your eno1 is setup for ipv6 only03:48
sdake_the kolla code uses the api interface for rabbitmq03:48
sdake_it should instead use the management network03:48
sdake_no idea why its designed using the api network03:49
sdake_so basically when kolla resolves, it resolves an ipv6 address (the api interface)03:49
DaveTurnerHmmm..03:49
*** adrian_otto has joined #openstack-kolla03:50
*** yuanying has joined #openstack-kolla03:50
sdake_so your probably wondering how to fix03:51
*** fragatina has quit IRC03:51
DaveTurnerI'll get to the ipv6 think in a second.  First, then api_interface.  When defined in kolla, it attempts to setup rabbit to listen on that interface, correct?03:51
sdake_unfortunately03:51
sdake_that needs a proper resolution03:51
sdake_as is, its a security problem03:51
DaveTurnerOK- I assume we'll need to update the roles / templates for that03:51
DaveTurnerYes, that makes sense.03:52
*** fragatin_ has quit IRC03:52
*** berendt has joined #openstack-kolla03:52
sdake_sup berendt03:52
berendtsdake_ morining.. moving to the airport03:53
sdake_there is a phone at wells and lake, you can make it!03:53
sdake_berendt sounds like a startup in progress!03:53
berendtsdake_ i think so :)03:54
*** berendt has quit IRC03:54
DaveTurnersdake: Re: ipv6 - the vlans configured on eno1 are ipv4 enabled reachable over IP.  You're thinking it's still a problem though?03:54
sdake_DaveTurner not certain - but that /etc/hosts file says so ;)03:55
sdake_DaveTurner you notice it lacked an ipv6 address03:55
sdake_sorry lacked an ipv4 address03:55
DaveTurnersdake In the test container we launched, yes.  However on the host.. and in the containers where the host networking is used, you see something like this:03:56
DaveTurner# cat /etc/hosts 127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1         localhost localhost.localdomain localhost6 localhost6.localdomain6 10.248.68.71slcas27503:56
sdake_hmm03:57
sdake_ok03:57
sdake_well thats good data03:57
sdake_i turn off ipv6 entirely03:57
sdake_so I have never noticed a delta03:57
DaveTurnerBut, I can explicitly disable for ipv6 just to eliminate this as a problem03:57
DaveTurnerAgreed.03:57
sdake_thats not the problem then03:57
sdake_are you able to run kolla-ansible -i ~/inventory destroy?03:58
sdake_followed by kolla-ansible -i ~/inventory deploy outside of ci?03:58
DaveTurnerGood point though.  We should probably be explicite about it. (ipv6)03:58
sdake_what really needs to happen is kolla needs to implement ipv6 support in the ansible orchestration code03:58
DaveTurnersdake. Hmm.. I could if I knew which pipeline vm kick off the deployment.  Let me poke around.  My other guys kicked this off during India hours, and I took over.03:59
sdake_ok well can ur guys cotact our peeps in apac timezone?04:00
sdake_jeffrey4l is who to look for04:00
sdake_have them reference the bug04:00
sdake_i'll leave a note04:00
sdake_or if you want to experiment with your vm ci jobs we can do that too :)04:00
DaveTurnersdake: Sure. I can do that.  Anything in particular you wanted to try on the redeployment?04:01
sdake_yes i want you to apply a patch to the kolla code04:01
sdake_no gurantees it will work04:01
sdake_it may take several tries04:01
sdake_we may have to hand this off to the apac folks because its 9pm here04:02
sdake_i generally hit the rack before midnight04:02
DaveTurnerUnderstood.  I'll have my guys look out for jeffrey4l - and leave them an update regarding our chat.04:03
*** Jeffrey4l has joined #openstack-kolla04:03
sdake_there is jeffrey4l right tthere ;)04:03
DaveTurner:-)04:03
sdake_Jeffrey4l works more maniacally then I do04:03
Jeffrey4lhi sdake_04:03
sdake_which is really saying somethign :)04:03
sdake_my son "heal me!!!!!!'04:03
sdake_poor kid04:04
sdake_welcome to MMORPGS a a dps class04:04
sdake_the epics are in the fire04:04
Jeffrey4li saw the repo is split, but the last commit is 4 days ago. should we need sync the commit?04:04
sdake_Jeffrey4l sync will hae to be manual04:04
sdake_Jeffrey4l it was best i could do04:04
Jeffrey4lcool.04:04
sdake_i did the work on saturday04:05
sdake_to minimize the pain04:05
sdake_but there is still some04:05
sdake_i have some reviews in the queue to clean up kolla04:05
sdake_been helping DaveTurner out with a bug04:05
Jeffrey4lok.04:05
sdake_thought i'd get your input since you left last comment on it04:05
sdake_DaveTurner can you link teh bug again - irc client restart on vpn reset wiped out the info04:06
Jeffrey4lDaveTurner, hmm which bug? just log in, no context.04:06
DaveTurnerSure..04:06
DaveTurnerhttps://bugs.launchpad.net/kolla/+bug/158365504:06
openstackLaunchpad bug 1583655 in kolla newton "rabbitmq setup from secondary IP" [Critical,Confirmed]04:06
sdake_Jeffrey4l my thinking is rabbitmq definately needs to run on the mgmt interface04:07
sdake_not the api interface04:07
sdake_api interface may be exposed to internets04:07
*** Pavo has quit IRC04:07
Jeffrey4l'got it. brb. 2min.04:09
sdake_your going to help us mr. anderson.  whether you want to or not!04:10
*** Pavo has joined #openstack-kolla04:12
sdake_DaveTurner an you paste your host's /etc/host file04:12
DaveTurnersdake_ which one?  From the test container or from the host / other functioning containers?04:13
sdake_the second04:13
Jeffrey4lback.04:13
DaveTurnersdake_ /etc/hosts http://paste.openstack.org/show/589356/04:14
sdake_.71 is api interface or mgmt interface?04:15
Jeffrey4lDaveTurner, what error message did u get?04:15
sdake_Jeffrey4l containers restart04:15
sdake_Jeffrey4l no logs for rabbitmq excpet docker logs04:16
DaveTurnersdake_ mgmt.  when troubleshooting this earlier, we manually added entries for the api IP's - although, they are actually registered in DNS.04:16
sdake_Jeffrey4l no error message in particular04:16
Jeffrey4lDaveTurner, show me the docker logs?04:16
*** jtriley has quit IRC04:16
sdake_DaveTurner you will have to do that on a box other then the one your on presently04:16
Jeffrey4land configurations.04:16
sdake_Jeffrey4l note i had him run docker stop rabbitmq04:16
sdake_on all 3 nodes04:17
Jeffrey4lthe /etc/hosts seems wrong.04:17
DaveTurnerJeffrey41 unfortunately, nothing to see in heka logs but let me show output of rabbit container before it restarts04:17
sdake_Jeffrey4l there is no rabbitmq dir in the heka dir...04:17
sdake_Jeffrey4l master as of a few weeks ago - DaveTurner is doing ci/cd trailing master of koll aa couple weeks04:18
sdake_a couple weeks old that is04:18
DaveTurnerJeffrey41 - no, there is not.  Docker logs output: http://paste.openstack.org/show/589357/04:19
Jeffrey4lif the hostname can not be parsed, it may not generate logs in heka.04:19
Jeffrey4lcould u fix your hostname and /etc/hosts file, then try again?04:20
sdake_Jeffrey4l could you be more specific about what needs fixing04:20
DaveTurnerJefferey41 - we have ipv4 vlans configured on eno1 device which are reachable over IP, etc - but as sdake mentioned prior to you jumping on, ipv6 is not specifically disable on eno.  I can try to update config so that this is explicitely disabled, in case that could make a diff.04:22
Jeffrey4lhttps://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L71604:22
DaveTurner..and yes, we can do whatever to the /etc/hosts.04:22
*** adrian_otto has quit IRC04:22
Jeffrey4lcheck these lines.04:22
*** chas has joined #openstack-kolla04:22
Jeffrey4lthe hostname should be resolved to api_interface's address.04:23
*** adrian_otto has joined #openstack-kolla04:24
sdake_Jeffrey4l yes however as i mentioned rabbitmq should lrun on the mgmt interfaces04:24
DaveTurnerJefferey41 That's the thing... the primary hostname.. the one which is defined in the inventory, is that of the management interface (network_interface in globals.yml)04:25
openstackgerritJeffrey Zhang proposed openstack/kolla: TEST MASTER BRANCH  https://review.openstack.org/39809304:26
*** adrian_otto has quit IRC04:26
sdake_Jeffrey4l master is busted04:26
sdake_or will be when my patches merge04:26
Jeffrey4lsdake_, but normally, it is bad to use the mgmt address for hostname, especially when you have a dns nameserver.04:27
*** adrian_otto has joined #openstack-kolla04:27
Jeffrey4lhmm. the .gitreview is wrong in kolla-ansible. i was trying to test kolla-ansible ci .04:27
sdake_Jeffrey4l i can fix the gitreview if yo ulike04:27
Jeffrey4lOK.04:27
*** chas has quit IRC04:28
coolsvapsdake_: the gitreview change is already there04:28
coolsvapneeds +W04:28
DaveTurnerJefferey41 - so if we have ip's plumbed to both the management and api interfaces, kolla expects the primary hostname to be that of the api interface, not the network_interface?04:28
sdake_coolsvap cool04:28
sdake_DaveTurner that actually does make sense04:28
Jeffrey4lDaveTurner, yes.04:29
DaveTurnerprimary meaning, that which is defined in inventory04:29
Jeffrey4lDaveTurner, primary means the first hostname in /etc/hosts file.04:29
openstackgerritJeffrey Zhang proposed openstack/kolla: TEST MASTER BRANCH  https://review.openstack.org/39809404:29
Jeffrey4lthe hostname resovlable is not related to inventory files.04:29
Jeffrey4loops. wrong test ps.04:30
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: TEST MASTER BRANCH  https://review.openstack.org/39809504:30
Jeffrey4lwe are using `ansible_hostname` variable in playbooks rather than `inventory_hostname` so we do not care about the hostname in inventory file.04:31
DaveTurnerJeffery41 OK.  I'll need to rethink our network segregation approach.04:31
DaveTurnerAnd, ansible_hostname is the "primary" hostname?04:32
Jeffrey4lyes.04:33
Jeffrey4lthe best solution should be use multi line for hostname in /etc/hosts file.04:34
Jeffrey4llike:04:34
Jeffrey4l192.168.10.2 hostname104:34
Jeffrey4l192.168.10.2 hostname_204:34
sdake_DaveTurner no what needs to happen is we need to usethe mgmt interface for rabbitmq04:35
sdake_Jeffrey4l is it not possible to use the mgmt interface for rabbitmq?04:35
*** jtriley has joined #openstack-kolla04:35
Jeffrey4lbefore, do we use hostname for mgmt interface in most of case?04:36
Jeffrey4lif so, we can.04:36
sdake_we should be using mgmt interface for anything related to non-api functionality04:36
sdake_putting rabbitmq on the api interface is a significant security issue04:37
Jeffrey4lyes. but rabbitmq case is more specially. it is related to the hostname of the node.04:37
sdake_so we are using hostname irrelevant of whether its a mgmt or api interface?04:38
sdake_if that is the case, hostname should always be the mgmt interface04:38
Jeffrey4lor check the bjolo's idea https://bugs.launchpad.net/kolla/+bug/1583655/comments/204:38
openstackLaunchpad bug 1583655 in kolla newton "rabbitmq setup from secondary IP" [Critical,Confirmed]04:38
DaveTurnerJeffrey41 - So, we currently have mgmt and api vlan configured.  System hostname is set to the DNS registered hostname of mgmt vlan IP.  This is also where the default GW is set.  We also have ip plumbed to api vlan, and it's registered in DNS.04:39
DaveTurnersdake_ your last comment is what our configuration assumed04:39
Jeffrey4lwe are using container, and we can change the hostname of the rabbitmq container.04:39
Jeffrey4lif it works, then we can not need require the hostname resolvable04:40
sdake_bjolo's solution wfm04:40
*** jtriley has quit IRC04:40
sdake_i thought /etc/hsotname was bindmounted though04:40
Jeffrey4ldocker run -h, --hostname string             Container host name04:40
sdake_automatically by docker04:40
*** khamtamtun has joined #openstack-kolla04:40
sdake_-h and --net=host are incompaitblle iirc04:41
sdake_been down that path before with the renaming containers work04:41
DaveTurnersdake_ also tried that during debug.. incompatible with net=host04:41
sdake_i think this is because --net=host bindmounts /etc/hostnmae ;-)04:42
sdake_so bjolo's solution wont work04:42
Jeffrey4lyes. it do not work. ;(04:42
openstackgerritcaoyuan proposed openstack/kolla: Move the "enable_destroy_images" into configure file  https://review.openstack.org/39763304:43
Jeffrey4lDaveTurner, kolla use fixed api_interface for hostname, right now. it won't work in your env, unless you change the code.04:43
sdake_Jeffrey4l i thin kthe default code is busted04:43
sdake_Jeffrey4l do you agree or disagree?04:44
sdake_Jeffrey4l its binding to api - which can be exposed to the internet04:44
sdake_it should always bind to  the mgmt interface - which is not exposed to the internet04:44
Jeffrey4lyep.04:44
coolsvapsdake_: shouldn't we remove the gates prior to removing code from kolla?04:44
*** tonanhngo has joined #openstack-kolla04:44
Jeffrey4lagree.04:44
sdake_coolsvap not sure it matters04:44
*** khamtamtun has quit IRC04:44
coolsvapi think it will be reverse of what we do while adding new04:45
*** krtaylor has joined #openstack-kolla04:45
coolsvapwe first update in project and tox then add new gates in project config04:45
* coolsvap confirms the sequence in mind04:45
Jeffrey4lbtw, we do not have mgmt interface concept right now.04:46
sdake_i was hoping we would end up with cross-gating04:46
sdake_Jeffrey4l mgmt interface = network_interface right?04:46
sdake_coolsvap i had hoped we would have cross-repo gating04:46
DaveTurnersdake_ my assumption as well... network_interface = mgmt_interface.04:46
sdake_coolsvap although i'd arther clean that up after deleting crap ;)04:47
DaveTurnerPerhaps just a case of wishful thinking.  :-)04:47
sdake_that way we dont end up with a megapatch04:47
sdake_hence i don't think its necessary to remove any gates04:47
*** newmember has joined #openstack-kolla04:47
coolsvaphmm04:47
sdake_unless we want to churn infra review time on that work04:47
Jeffrey4lDaveTurner, no. network_interface is not used in playbooks. it is only exist in group_vars/all.yml file.04:47
*** adrian_otto has quit IRC04:47
sdake_taking out gates and putting them back in04:47
sdake_Jeffrey4l hwich interface does mariadb run on?04:48
sdake_and all of the haproxy/etc04:48
Jeffrey4lsdake_, api_interface ;(04:48
*** tonanhngo has quit IRC04:48
sdake_you ahve to be kidding me04:48
sdake_when did that change get made?04:48
Jeffrey4lsdake_, https://github.com/openstack/kolla/blob/master/ansible/roles/mariadb/templates/galera.cnf.j2#L1404:49
Jeffrey4li do not think we changed it.04:49
*** adrian_otto has joined #openstack-kolla04:49
Jeffrey4lyou are wrong about the api_interface is open to internet.04:50
Jeffrey4lwe have kolla_external_vip_interface04:50
sdake_this is a r egression: https://github.com/openstack/kolla/commit/5838bd0b3c12eed1514e59852564006f044f241b04:50
Jeffrey4lwhen this is configured, this is open to internet.04:50
Jeffrey4lbefore this https://github.com/openstack/kolla/commit/5838bd0b3c12eed1514e59852564006f044f241b   it is using api_interface, too.04:51
sdake_Jeffrey4l got it04:51
sdake_Jeffrey4l ok - well its still wrong ;-)04:51
Jeffrey4lso, right now. the kolla_external_vip_interface is api_interface you want and the api_interface is mgmt interface04:51
sdake_api interface is for apis to bind to04:52
sdake_network_interface should be for servers to bind to internally04:52
sdake_kolla_external_vip_interface should be for an internet facing ip address i guess04:53
Jeffrey4lnetwork_interface is just a simple variable in group_vars/all.yml file. it is not used in any role.04:53
Jeffrey4lother *_interface is refer to this variable if they are not configured.04:53
DaveTurnerExcept as a default when api_interface isn't defined?04:53
Jeffrey4lDaveTurner, yep.04:53
DaveTurnerJeffrey41 sorry to back up a bit, but can you elaborate on "kolla use fixed api_interface for hostname"04:54
Jeffrey4lso i think current configuration is OK.04:54
Jeffrey4lDaveTurner, it won't work as sdake_ said.04:54
*** sdake_ is now known as sdake04:54
Jeffrey4lwe can not use --hostname <> with --net host04:55
sdaketime for a cuban04:55
sdakebbiaf04:55
DaveTurnerJeffrey41- right, I discovered that during debugging earlier.  Didn't think it was possible considering /etc/ is bind mounted04:56
Jeffrey4lsorry?04:57
Jeffrey4lhmm just tried this and it works. docker run -it --rm -h abc --net host --add-host a:1.1.1.1 centos:7 bash04:57
Jeffrey4lwe can change the /etc/hosts in container only.04:57
Jeffrey4lbut the host os' /etc/hosts is override.04:57
sdakethat must be a new feature04:57
openstackgerritcaoyuan proposed openstack/kolla: Add empty prechecks.yml for Designate  https://review.openstack.org/39810004:58
Jeffrey4lthe disadvantage is the original host os' /etc/hosts file is override. this may case some other issue .04:58
DaveTurnerJeffrey41 Didn't 1583655 mention this approach as a possible option?05:00
coolsvapsdake: i've moved most of the ansible blueprints to kolla-ansible launchpad05:00
Jeffrey4lDaveTurner, it didn't clear how to implement this.05:00
Jeffrey4lcoolsvap, cool.05:00
Jeffrey4lI will leave some comments in that bug.05:01
coolsvapi am not sure whether to -1 /-2 the patches because some patches have changes in dockerfiles as well05:01
sdakeya or /etc in that one i just -2ed05:02
sdakenot sure where /etc belongs05:02
Jeffrey4li'd like use -1 for the patch and tell him to split his patch.05:02
coolsvapyes me too05:02
sdakeJeffrey4l so how do we get DaveTurner operational with minimal pain while we sort out a solution to this security problem upstream?05:03
DaveTurnerJeffrey41 sdake thanks for the info.  I'll need to drop soon but will update my team in India.  Any suggestions on how we can move forward with our pipeline builds given our current config?05:03
sdakeJeffrey4l clearly I'm confused ;-)05:03
DaveTurnersdake beat me to the question..05:04
Jeffrey4lcould u post you globals.yml file out? DaveTurner05:04
coolsvapJeffrey4l sdake: https://review.openstack.org/#/c/397888/  need +W05:06
openstackgerritMerged openstack/kolla-ansible: Set up .gitreview  https://review.openstack.org/39788805:08
DaveTurnerJeffrey41 let me grab it for you05:09
Jeffrey4lok.05:10
Jeffrey4lcoolsvap, it merged ;)05:10
*** sdake_ has joined #openstack-kolla05:10
*** sdake has quit IRC05:14
openstackgerritcaoyuan proposed openstack/kolla: Add empty prechecks.yml for Designate  https://review.openstack.org/39810005:20
*** dims has quit IRC05:20
DaveTurnerJeffrey41 - going to be around for a while?  I need to have my India team provide this information for you, as it seems the VM created as part of the pipeline for this test deployment is no more.  :-(05:22
DaveTurnerWe create several a day, so no worries there.  I can have them look for you in here.05:22
*** sdake_ has quit IRC05:24
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move the "enable_destroy_images" into configure file  https://review.openstack.org/39810605:24
*** tonanhngo has joined #openstack-kolla05:28
openstackgerritcaoyuan proposed openstack/kolla-ansible: Update the default value for Heat  https://review.openstack.org/39810705:31
*** tonanhngo has quit IRC05:32
openstackgerritcaoyuan proposed openstack/kolla-ansible: Add empty prechecks.yml for Designate  https://review.openstack.org/39810805:33
*** adrian_otto has quit IRC05:46
*** adrian_otto has joined #openstack-kolla05:51
*** harlowja has quit IRC05:55
*** unicell has joined #openstack-kolla05:58
*** adrian_otto has quit IRC05:58
*** dims has joined #openstack-kolla05:59
openstackgerritzhubingbing proposed openstack/kolla: change network_bw_out to network.bw.out in cloudkitty.conf.j2  https://review.openstack.org/39764106:01
*** Pavo has quit IRC06:08
*** Pavo has joined #openstack-kolla06:12
openstackgerritMerged openstack/kolla: Allow for specifying a KOLLA_CONFIG_FILE  https://review.openstack.org/39800806:15
*** chas has joined #openstack-kolla06:17
*** yingjun_ has joined #openstack-kolla06:21
*** yingjun has quit IRC06:21
*** chas has quit IRC06:21
*** harlowja_at_home has joined #openstack-kolla06:32
*** harlowja_at_home has quit IRC06:33
*** chas has joined #openstack-kolla06:45
coolsvapJeffrey4l: https://blueprints.launchpad.net/kolla/+spec/remove-node can be moved to kolla-ansible right?06:48
Jeffrey4lcoolsvap, yep.06:48
*** chas has quit IRC06:49
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Modify the letter case in post-install.yml file  https://review.openstack.org/39813306:50
*** kamal__ has joined #openstack-kolla06:53
openstackgerritJoshua Harlow proposed openstack/kolla: Allow for externally managed configuration  https://review.openstack.org/33259006:54
openstackgerritJoshua Harlow proposed openstack/kolla: Allow for specifying a KOLLA_CONFIG_FILE  https://review.openstack.org/39813806:58
kamal__hi Jeffrey4l, I am following up on DavidTurner chat with you and sdake07:03
kamal__u there07:03
Jeffrey4lkamal__, yep.07:04
kamal__hi jeffery, from you last conversation i believe you wanted to globals.yml07:05
Jeffrey4lkamal__, yep.07:05
Jeffrey4lyou and DaveTurner are colleague?07:05
kamal__yes07:05
kamal__http://paste.openstack.org/show/589364/07:06
Jeffrey4lkamal__, and you `ip a`   `hostname` output on your control nodes.07:06
kamal__hostnames are slcas275 slcas283 slcas29107:08
kamal__ip addr output for slcas275 is http://paste.openstack.org/show/589365/07:09
openstackgerritJoshua Harlow proposed openstack/kolla: Use itertools.chain and show how many files utime modified  https://review.openstack.org/39814507:09
Jeffrey4lkamal__, ok.07:09
*** unicell has quit IRC07:10
*** unicell has joined #openstack-kolla07:11
*** chas has joined #openstack-kolla07:12
kamal__ip addr for slcas283 http://paste.openstack.org/show/589366/ , for slcas291 http://paste.openstack.org/show/589367/07:12
openstackgerritcaoyuan proposed openstack/kolla-ansible: Update the default value for Heat  https://review.openstack.org/39810707:12
Jeffrey4lkamal__, you are using network_interface: "eno1" , but there is no address on eno1 interface.07:16
*** Satya_ has joined #openstack-kolla07:16
*** chas has quit IRC07:16
kamal__eno1.802 is my api_interface and eno1.2706 is my mgmt interface07:18
Satya_Hi Britt07:18
Jeffrey4lkamal__, i do not see that in your globals.yml file.07:19
kamal__in the multinode file07:21
kamal__slcas275 api_interface=eno1.802 tunnel_interface=eno1.802 slcas283 api_interface=eno1.802 tunnel_interface=eno1.802 slcas291 api_interface=eno1.802 tunnel_interface=eno1.80207:21
Jeffrey4lok07:21
Satya_Hi Jeffrey.07:22
Jeffrey4lkamal__, then what ip address/interface is resolved to ?07:22
Jeffrey4lsup Satya_07:22
Satya_me kamal and David working together :)07:23
Satya_on the same issue07:23
Jeffrey4lcool07:23
kamal__slcas275 resolves to eno1.2704 ipv4 address, and slcas275-api resolves to eno1.2706 ipv4 address07:24
kamal__sorry slcas275 eno1.80207:24
*** magicboiz has joined #openstack-kolla07:25
Jeffrey4lhmm. so it is api_interface, right? it should be ok.07:25
kamal__ok, so slcas275 resolves to eno1.270607:26
kamal__and slcas275-api resolves to eno1.80207:26
Jeffrey4lOK.07:26
Satya_802 is our api interface07:26
Satya_2706 is our mgmt interface07:27
Jeffrey4lyes. the the quick fix solution is: modify the hostname name from slcas275 to slcas275-api if possible.07:27
Satya_That will break other services... :(07:27
Jeffrey4lhmm.07:29
Jeffrey4lanother workaround is change this file.07:29
Jeffrey4lhttps://github.com/openstack/kolla/blob/master/ansible/roles/rabbitmq/templates/rabbitmq-clusterer.config.j207:29
Jeffrey4ladd `-api` suffix into the hostname.07:30
Jeffrey4llike:      'rabbit@{{ hostvars[host]['ansible_hostname'] }}-api07:30
Jeffrey4lmaybe we can introduce a variable like: rabbitmq_hostname into kolla.07:30
Jeffrey4lin default, it point into ansible_hostname,07:31
Jeffrey4lbut you can change it based on you environment.07:31
Satya_we tried that manually changing it for container by changing the /etc/kolla/rabbitmq/rabbitmq-cluster.config07:32
Satya_that didnt helped as the hostname of the container points to the slcas275 rather than slcas275-api07:33
Jeffrey4ldoes this file need to be change? https://github.com/openstack/kolla/blob/master/ansible/roles/rabbitmq/templates/rabbitmq.config.j207:34
Satya_http://paste.openstack.org/show/589369/07:34
Satya_yes07:35
Jeffrey4li am not sure why rabbitmq require hostname resolvable.07:35
Satya_we changed that too07:35
*** f13o has joined #openstack-kolla07:35
Jeffrey4lSatya_, do u have any idea why rabbitmq require hostname resolvable?07:36
Satya_i guess thats a piece of erlang but not sure...07:37
*** magicboiz has quit IRC07:40
Satya_http://www.rabbitmq.com/networking.html07:42
Satya_inside that "Hostname Resolution and DNS"07:42
* Jeffrey4l is checking.07:45
Jeffrey4lSatya_, i do not think we can have a workaround.07:47
Satya_what we can have as a solution for this?07:48
Jeffrey4lchange the hostname ;(07:49
Jeffrey4lor use mgmt interface for api_interface.07:49
Jeffrey4lyou can use current api_interface for external_interface.07:50
Satya_hmmm but i am not sure if other services like nova/neutron will break in that case07:50
Jeffrey4lwhy?07:50
Satya_but i will give it a shot....07:50
Jeffrey4lyou are not using kkolla_external_vip_interface, right?07:51
Satya_as for nova and neutron the mgmt interface is 2706 and api is 80207:51
Satya_we are...07:51
Jeffrey4lwhere is it?07:52
Jeffrey4li can not see it on you globals.yml and your inventory file( post above, 2lines)07:52
Satya_yeah sorry that neutrons public network interface07:52
Satya_will give it a shot07:53
Satya_lets see07:53
Jeffrey4lgot. try it and i think it should work.07:53
*** shardy has joined #openstack-kolla07:56
openstackgerritcaoyuan proposed openstack/kolla-ansible: Update the default value for Heat  https://review.openstack.org/39810707:58
*** f13o_ has joined #openstack-kolla07:59
bjolomorning08:01
*** f13o has quit IRC08:02
Jeffrey4lmorning ;)08:02
Jeffrey4lbjolo, could u take look at this https://bugs.launchpad.net/kolla/+bug/158365508:04
openstackLaunchpad bug 1583655 in kolla newton "rabbitmq setup from secondary IP" [Critical,Confirmed]08:04
*** magicboiz has joined #openstack-kolla08:05
*** skramaja_ has joined #openstack-kolla08:05
*** skramaja has quit IRC08:06
*** Pavo has quit IRC08:08
* bjolo looking08:08
bjoloyes what about it08:08
bjolosee you have had a discussion on irc about it08:08
bjoloat the conference it was mentioned that rabbit from version 3.6.x something did not require hostname resolution any more08:10
bjoloworks with IP08:11
bjolohence this issue will resolve itself eventually08:11
bjolohowever we do have an issue today that needs to be resolved08:11
Jeffrey4lwe are using 3.6.x, so we can remove the hostname resolve right now?08:12
Jeffrey4lbut Satya_ still have such issue.08:12
*** Pavo has joined #openstack-kolla08:12
bjoloi dont know exactly what version it is from, just that we discussed this issue at the summit and someone mentioned this08:13
Jeffrey4lOK.08:13
bjolothe general take is that we are bound to use the distrobution packages for rabbit08:14
Jeffrey4lyes. but it is more stable to use distro package.08:15
bjoloouch is this correct. centos 7 use rabbit 3.3.5?08:17
bjoloVersion     : 3.3.508:17
Jeffrey4lwe install rabbit from rdo.08:17
bjoloah ok08:17
bjolowhat version is that?08:18
Jeffrey4l3.6.508:18
Satya_we are using 3.6.508:19
bjolook08:20
*** Serlex has joined #openstack-kolla08:20
kamal__http://paste.openstack.org/show/589372/08:20
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Modify the letter case in ansible yml file  https://review.openstack.org/39813308:20
bjoloand what about ubuntu 1604? version?08:20
Jeffrey4lbjolo, we install the same version for ubuntu ,which is download from rabbitmq offical site.08:21
bjoloso we should be able to just use IP then?08:22
bjologiven that the claim is true that rabbit works with just IP now08:22
Jeffrey4li am not sure.08:22
Satya_I too with the current situation we are running in...08:22
Jeffrey4li think we need some official info about this.08:22
bjoloyes08:23
bjolowe need to go through the rabbit change logs or something to get it confirmed08:23
bjoloand of course test it out08:23
Jeffrey4lyep08:23
bjolojust to be correct, i did not make this statement! ?)08:24
bjolo:)08:24
bjoloim just the messenger here :)08:24
Satya_i see rabbitmq.config is having the IP but the cluster config having the hostnames08:24
*** tovin07 has quit IRC08:24
Satya_and if we do a rabbitmqctl status it tries to connect to rabbit@hostname rather than rabbit@ip08:25
bjoloone thing i dont quite get about docker is that bind mounts actually seems to be a copy and mount08:30
bjolofor ex /etc/hostname08:30
bjolodocker inspect rabbitmq | grep hostname08:31
bjolo "HostnamePath": "/var/lib/docker/containers/23ee7bbeab8f701fd036dd5136408146d4d8622e91cc686896c2a790be40ed49/hostname",08:31
bjolochanging /etc/hosts on the host does not change /etc/hosts in the container08:31
bjoloand vice versa08:31
Jeffrey4lbjolo, it use different hosts file, too. like hostname08:31
Jeffrey4l        "HostnamePath": "/home/docker/containers/54b3ede09a65a161ae4274852424663634f31ac8c87e62248dc07ef7b0df3923/hostname",08:32
Jeffrey4l        "HostsPath": "/home/docker/containers/54b3ede09a65a161ae4274852424663634f31ac8c87e62248dc07ef7b0df3923/hosts",08:32
bjoloyes, also a copy and mount08:33
Jeffrey4lin this way, the container can change the hostname and hosts file without affecting the host og.08:33
Jeffrey4lhost os.08:33
bjoloyes08:33
bjoloso manipulating hostname and /etc/hosts in the rabbit container *should* work in theory08:34
bjolo(if we still need to do that)08:34
Jeffrey4l--add-host parameter will change the hosts file in /etc/hosts/08:34
Jeffrey4lthe disadvantage is the host os 's /etc/hosts is override.08:34
bjoloor doing that way. That is the "how" question08:35
Jeffrey4lyep.08:35
bjolowhat do you mean overide?08:35
Jeffrey4lbjolo, try this `docker run -it --rm -h abc --net host --add-host a:1.1.1.1 centos:7 cat /etc/hosts`08:36
*** saneax-_-|AFK is now known as saneax08:36
Jeffrey4lcompare its output with your host os /etc/hosts08:36
Jeffrey4land this `docker run -it --rm -h abc --net host centos:7 cat /etc/hosts `08:36
*** egonzalez90 has joined #openstack-kolla08:37
bjolohmm ok i see what you mean08:37
Jeffrey4lbut i do not think it is matter in most of case.08:38
bjolonot sure08:38
Satya_tried changing the /var/lib/docker/containers/23ee7bbeab8f701fd036dd5136408146d4d8622e91cc686896c2a790be40ed49/hostname08:39
bjoloall hosts should have their own hostname resolved in /etc/hosts is the general consensus. not sure for containers08:40
Satya_when we bounce the container again the file get modified with the previous hostname08:40
openstackgerritJavier Castillo Alcíbar proposed openstack/kolla-ansible: Fix ceilometer bootstrap  https://review.openstack.org/39819008:40
Jeffrey4lwe should not change the hostname when using --net host.08:41
Satya_hmm ok08:41
Satya_will try that...08:41
bjoloi need to go. i work for a company that loves meetings :(08:46
bjoloJeffrey4l, did you try a deploy with multiple external flat networks?08:46
Jeffrey4lbjolo, no.08:50
Jeffrey4lgtg.08:50
bjoloJeffrey4l, please do if you have time. would be good to get my issue confirmed or not08:51
bjolohmm cant find anything in rabbit change log about IP08:52
bjolohttp://www.rabbitmq.com/clustering.html08:53
bjolowe are using the clusterer plugin right?08:53
bjolopbourke, tried the vlan_ranges and suspected it did not work since bond0 is not dedicated to neutron08:55
*** saneax is now known as saneax-_-|AFK08:55
*** athomas has joined #openstack-kolla08:56
bjoloif im gonna go that route i need suspect i need to setup a bridge on bond0 and create tap/tun interface for neutron to use08:56
bjolook now i really need to go. meeting at 1008:57
*** tovin07 has joined #openstack-kolla08:57
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Modify the letter case with 'True' in ansible yml file  https://review.openstack.org/39813308:59
*** saneax-_-|AFK is now known as saneax09:00
*** skramaja has joined #openstack-kolla09:06
*** skramaja_ has quit IRC09:06
*** saneax is now known as saneax-_-|AFK09:13
*** berendt has joined #openstack-kolla09:14
*** msimonin has joined #openstack-kolla09:20
*** tovin07 has quit IRC09:23
*** yingjun_ has quit IRC09:27
*** yingjun has joined #openstack-kolla09:27
*** yingjun has quit IRC09:27
*** chas has joined #openstack-kolla09:31
*** gfidente has joined #openstack-kolla09:32
Jeffrey4lbjolo, ok i will try multi flat.09:35
*** matrohon has joined #openstack-kolla09:36
*** DTadrzak has joined #openstack-kolla09:36
Jeffrey4land bond0 + vlan device should works. at least in our 100+ nodes test in osic, it works.09:36
*** saneax-_-|AFK is now known as saneax09:38
*** berendt has quit IRC09:48
*** hfu has quit IRC09:51
*** chas has quit IRC09:52
*** zhangyufei has joined #openstack-kolla09:52
*** chas has joined #openstack-kolla09:52
magicboizHi, I'm checking kolla with TLS enabled, and I think generated haproxy config is broken....I cannot access horizon, I cannot access nova_api....09:53
magicboizany ideas?09:53
magicboizthx09:53
*** alteriks has quit IRC09:53
*** alteriks has joined #openstack-kolla09:55
*** chas has quit IRC09:57
*** tovin07_ has quit IRC09:57
*** chas has joined #openstack-kolla09:58
openstackgerritZeyu Zhu proposed openstack/kolla-ansible: Modify the letter case with 'True' in ansible yml file.  https://review.openstack.org/39813309:59
*** saneax is now known as saneax-_-|AFK09:59
*** zhurong has quit IRC10:02
*** zhangyufei has quit IRC10:02
*** chas has quit IRC10:02
*** portdirect_away is now known as portdirect10:03
portdirectmorning all :)10:03
egonzalez90morning portdirect10:04
egonzalez90magicboiz: https://bugs.launchpad.net/kolla/+bug/163796210:04
openstackLaunchpad bug 1637962 in kolla newton "SSL/TLS configuration not handled correctly" [Critical,Triaged]10:04
*** chas has joined #openstack-kolla10:04
*** chas has quit IRC10:07
*** chas has joined #openstack-kolla10:07
*** Pavo has quit IRC10:07
magicboizegonzalez90: bingo, thx10:08
*** berendt has joined #openstack-kolla10:10
egonzalez90are you using same network for external(TLS) and internal(http)10:10
*** Satya_ has quit IRC10:12
*** Pavo has joined #openstack-kolla10:12
magicboiz egonzalez90: yes10:15
egonzalez90there is the problem then10:15
portdirectegonzalez90: is anyone workig on this bug atm?10:16
egonzalez90is not asigned10:16
magicboiz egonzalez90: is there any haproxy limitation on this?10:17
portdirectmagicboiz: no - its a config issue10:17
magicboizportdirect: what you mean by config issue?10:17
*** chas has quit IRC10:18
magicboizchecking http://docs.openstack.org/developer/kolla/production-architecture-guide.html I think it's not clear at all10:19
egonzalez90kolla_external_vip_interface - This interface is public-facing one. It’s used when you want HAProxy public endpoints to be exposed in different network than internal ones10:21
portdirectit's a bug with the way that haproxy/kolla is configured, I've not got a kolla setup here - so unfortunatly cant help you get it up and running this morning (GMT here), but once the usa wakes up you should be able to get some help, unless egonzalez90 has any input.10:22
egonzalez90i dont think is a bug, if the same interface is defined to external and internal10:24
egonzalez90haproxy will try to bind the port to the same IP10:24
*** chas has joined #openstack-kolla10:24
egonzalez90which is not possible10:24
portdirectegonzalez90/magicboiz: whoops - missed this: are you using same network for external(TLS) and internal(http)? egonzalez90: yes10:25
portdirectsry - your right10:26
egonzalez90maybe, a rework is needed to allow internal networks with TLS, but that is more a feature request than a bug10:26
magicboizI think it would be enough to make it clear in docs: set public != internal in order to get haproxy external configuration.10:27
egonzalez90a network architecture diagram would do it easier ;)10:28
*** chas has quit IRC10:29
openstackgerritMerged openstack/kolla: Extract kolla exceptions into a single exception.py file  https://review.openstack.org/39656810:30
egonzalez90is documented, but not in a place where people usually read10:30
*** ppalacios has joined #openstack-kolla10:31
egonzalez90magicboiz: can you share what is in haproxy conf?10:33
*** duonghq has quit IRC10:33
magicboiz<gonzalez90: right now, I don't have any external haproxy config. I'm going to split public from internal in my lab10:35
egonzalez90thats is because haproxy_enable_external_vip: "{{ 'no' if kolla_external_vip_address == kolla_internal_vip_address else 'yes' }}"10:36
egonzalez90which means that if the external is the same as internal, haproxy wont configure external tls10:36
*** kamal__ has quit IRC10:37
*** berendt has quit IRC10:40
magicboizgonzalez90: but doc says: It is mandatory to set this option when kolla_enable_tls_external is set to yes. Defaults to network_interface.10:48
*** chas has joined #openstack-kolla10:49
*** zhubingbing has quit IRC10:49
*** zhubingbing has joined #openstack-kolla10:50
egonzalez90yes, i know what you mean, is in some kind bad explained.10:51
magicboizanyway, it's to true that in a fully production environment, you'll find different networks for each service, but in LAB environments like mine.... :)10:51
egonzalez90also, the same line says that is only used when different networks are present10:52
magicboizyes your right.10:53
*** zhangyufei has joined #openstack-kolla10:53
magicboizMaybe, the best option would be to set kolla_enable_tls_external=no if network==public. Because the mixed option doesn't work10:54
*** chas has quit IRC10:55
*** msimonin has quit IRC10:57
*** chas has joined #openstack-kolla11:09
*** chas has quit IRC11:13
*** chas has joined #openstack-kolla11:15
*** athomas has quit IRC11:18
*** chas has quit IRC11:19
*** msimonin has joined #openstack-kolla11:20
*** chas has joined #openstack-kolla11:21
*** athomas has joined #openstack-kolla11:22
*** msimonin1 has joined #openstack-kolla11:22
*** msimonin has quit IRC11:22
*** msimonin1 has quit IRC11:22
*** chas has quit IRC11:25
*** mliima has joined #openstack-kolla11:29
*** berendt has joined #openstack-kolla11:33
*** Serlex has quit IRC11:33
*** hfu has joined #openstack-kolla11:44
*** chas has joined #openstack-kolla11:51
*** Serlex has joined #openstack-kolla11:52
sp_hi anyone please provide your views. What if we compare kolla-k8 with fuel then what are the key differences which is making our kolla-k8s better ??11:53
*** chas has quit IRC11:55
*** msimonin has joined #openstack-kolla11:56
coolsvapegonzalez90: https://review.openstack.org/#/c/398108/ is on kolla-ansible :)11:57
coolsvapsame for https://review.openstack.org/#/c/398107/11:58
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible: Fix fact gathering when using --limit  https://review.openstack.org/39831311:58
egonzalez90coolsvap: yes, i -1 on kolla11:58
egonzalez90dont know how moved to kolla-ansible maintaining -111:59
coolsvapegonzalez90: no its -1 on kolla-ansible11:59
egonzalez90i'll remove it11:59
egonzalez90done12:01
egonzalez90a question coolsvap, you know what is the proceidure to migrate code from kolla to kolla-ansible? changes made from friday are not in  kolla-ansible12:04
egonzalez90asked a couple of times..12:04
coolsvapegonzalez90: since the snapshot was taken on friday sometime i believe, we need to submit the changes again12:05
coolsvapwhich are merged after friday12:05
*** Pavo has quit IRC12:07
*** schwicht has joined #openstack-kolla12:09
*** coolsvap is now known as coolsvap_12:09
*** shardy is now known as shardy_lunch12:11
*** Pavo has joined #openstack-kolla12:12
*** psanchez has quit IRC12:13
pbourkecan someone +a https://review.openstack.org/#/c/380028/ pelase12:13
pbourke*please12:14
*** pc_m has quit IRC12:14
*** g3ek has quit IRC12:14
sp_pbourke:  please provide your views. What if we compare kolla-k8 with fuel then what are the key differences which is making our kolla-k8s better ??12:15
sp_it would quite helpful in my understanding12:15
*** psanchez has joined #openstack-kolla12:15
pbourkesp_: sorry but I know nothing about fuel12:15
*** pc_m has joined #openstack-kolla12:16
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Remove docker from kolla-ansible  https://review.openstack.org/39832012:17
portdirectsp_, I'll bite: very hard to answer, they are taking two very differenet approaches to deployment and mangement: fuel takes a very tradional puppet based approach to deployment vs kolla-kubernetes kubernetes approach.12:17
*** g3ek has joined #openstack-kolla12:17
mliimadone pbourke12:18
pbourkemliima: thanks12:18
egonzalez90not being mirantis is a key point? ;)12:18
sp_pbourke: Thanks for quick response, but as per my study and understanding fuel also provide nearly same features as we are going to do with kolla-k8s12:18
sp_portdirect: thanks for info12:18
mliimaI don't understand why folks don't do it :/12:18
pbourkesp_: http://stackalytics.com/?module=fuel-group12:19
pbourkesp_: if mirantis pull out that project is as good as dead12:19
sp_portdirect: may i get more your views on this12:20
openstackgerritMerged openstack/kolla: Unmount Ceph OSD disks as part of destroy  https://review.openstack.org/38002812:20
sp_pbourke: ok !! I will take a look on this12:20
pbourkesp_: kolla has the diversity tag which means it is well supported across a wide community12:20
*** schwicht has quit IRC12:21
sp_pbourke: do have some idea about our kolla-k8s operator part ?12:21
egonzalez90sp_: I never liked comparisions, i prefer say analyse what they do and what are our weakness12:22
sp_do you*12:22
*** chas has joined #openstack-kolla12:22
pbourkesp_: I have nothing to add over whats in the spec12:22
*** msimonin has quit IRC12:22
portdirectsp_: pbourke's point is very valid. I used to use fuel, but havent since juno - when I had some pretty *interesting* experiences, I've not touched it since then.12:22
portdirectsp_: So im afraid that I cant really add much more :/12:22
mliimapbourke, we need to remove docker from kolla-ansible, ok? Can i register a bp for it?12:23
pbourkemliima: I think sdake already has a patch up for that12:23
pbourkemliima: actually I think his is the other way around12:23
pbourkeremoving ansible from the kolla part12:23
egonzalez90yup12:23
mliimaremove ansible folder?12:24
mliimahaha12:24
pbourkemliima: have a look, feel free to register a bp for any pieces that arent done12:24
sp_pbourke: portdirect: egonzalez90: every one,  thanks a lot for having a great discussion on this.12:24
*** chas has quit IRC12:26
*** berendt has quit IRC12:26
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible: Make service_checks easier to run independently of kolla-ansible  https://review.openstack.org/39832712:29
*** schwicht has joined #openstack-kolla12:32
*** zhurong has joined #openstack-kolla12:35
*** berendt has joined #openstack-kolla12:40
*** duonghq has joined #openstack-kolla12:42
*** msimonin has joined #openstack-kolla12:43
*** shardy_lunch is now known as shardy12:45
*** berendt has quit IRC12:45
*** srwilkers has joined #openstack-kolla12:49
srwilkersgood morning12:51
portdirectmorning srwilkers :)12:52
pbourkeJeffrey4l: any idea whats wrong with the release note gate on https://review.openstack.org/#/c/393409/12:52
*** chas has joined #openstack-kolla12:52
*** sdake has joined #openstack-kolla12:53
srwilkershey portdirect :D12:53
portdirectbeen playing with your k8s vagrant stuff - it's nice :) few q's though12:54
portdirectI could add fedora and centos atomic support if your up for a pr?12:54
portdirectalso how are you finding romana? Just been playing iwht is this morning, and it seems pretty good but wondering if youve had much time on it?12:55
srwilkerswe're always open to pull requests -- we merged centos support in the night before last, and we also added support for vagrant-libvirt12:56
srwilkersno complaints with romana so far, but we're working on getting some other SDN provider support added currently12:56
sdakemorning12:57
srwilkerswas actually going to use it as a base for a new multi-node getting started guide at some point.  its what we're using for most of our development efforts currently12:57
*** schwicht has quit IRC12:57
srwilkerssup sdake12:57
*** chas has quit IRC12:57
sdakechillen12:57
mliimamornign sdake12:57
*** schwicht has joined #openstack-kolla12:57
portdirectmorning sdake12:57
*** chas has joined #openstack-kolla13:00
portdirectsrwilkers: yeah - 'm making use of both libvirt and centos (running 4.8 kernel on my dev boxen, so virtualbox is out for the moment :( )  - I've been using OVN for all of my k8s setups - so I'm up for throwing that in at some point, thogh I'm having a bit of  head scratcher working out how to get it up and running with kubeadm so would need to sort that first :/13:00
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Install neutron-fwaas plugin in neutron-base  https://review.openstack.org/39340913:01
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible: Allow neutron-fwaas to be enabled in Neutron  https://review.openstack.org/39833613:01
*** schwicht has quit IRC13:01
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Install neutron-fwaas plugin in neutron-base  https://review.openstack.org/39340913:03
srwilkersyeah, thats a good point13:03
*** msimonin has quit IRC13:04
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible: Allow neutron-fwaas to be enabled in Neutron  https://review.openstack.org/39833613:04
*** sdake_ has joined #openstack-kolla13:05
sp_morning sdake:13:05
*** msimonin has joined #openstack-kolla13:08
sdake_Jeffrey4l around?13:08
*** sdake has quit IRC13:08
*** hfu has quit IRC13:13
*** rhallisey has joined #openstack-kolla13:15
openstackgerritzhubingbing proposed openstack/kolla: Add trove role  https://review.openstack.org/35490113:16
*** lamt has joined #openstack-kolla13:23
Jeffrey4lsdake_, yep.13:23
*** jheroux has joined #openstack-kolla13:24
Jeffrey4lpbourke, checking13:25
*** dave-mccowan has joined #openstack-kolla13:25
rhalliseymorning13:26
sdake_Jeffrey4l sup dude, did you sort out how to make a mgmt interface?13:27
sdake_Jeffrey4l or do we need a blueprint for that13:27
Jeffrey4lwe don't.13:27
Jeffrey4lwe do not need mgnt interface.13:27
Jeffrey4lnow, the api_interface in kolla is the mgmt interface.13:28
sdake_what do the apis themselves bind to?13:28
Jeffrey4lif we do not want to export the api_interface to extenal/internet.13:28
Jeffrey4lwe can enable the externel ip interface,13:28
Jeffrey4lwhich only forwards the necessary port.13:28
*** schwicht has joined #openstack-kolla13:29
sdake_dave-mccowan around?13:30
dave-mccowansdake_ ack13:30
sdake_dave-mccowan do you typically bind rabbitmq/mariadb to the api interface in openstack?13:31
sdake_from a security pov13:31
Jeffrey4lsdake_, i recommended kamal to enable external vip feature.13:31
*** msimonin has quit IRC13:31
dave-mccowanthe names internal, external, management, and api get mixed up sometimes.13:33
*** msimonin has joined #openstack-kolla13:33
*** zhubingbing has quit IRC13:33
dave-mccowani recommend two vips: the internal is for mariadb, rabbitmq, and service-to-service connections.13:33
dave-mccowanthe external is for humans to connect to the dashboard and service REST APIs.13:34
dave-mccowanthe internal VIP does not use TLS/HTTPS, so it needs to be kept locked down.  no routes in or out.13:34
sdake_dave-mccowan ok we have that in kolla now13:34
sdake_dave-mccowan what we also have is mariadb/rabbitmq binding to api_interface_address13:35
*** schwicht has quit IRC13:37
dave-mccowanin kolla, the api_interface is the internal one.13:38
dave-mccowanso, yes that's normal for support services to the be on the internal network.13:39
*** zhangyufei has quit IRC13:39
Jeffrey4lpbourke, no idea. it works fine locally.13:41
Jeffrey4lsdake_, how can we backport the ansible change after the repo is split?13:43
sdake_Jeffrey4l inc0 said he was going to make a script to do it13:44
sdake_Jeffrey4l but i'm skeptical of script approach13:44
sdake_Jeffrey4l I don't have a good answer for ansible change backports other then git format-patch followed by git am :(13:44
sdake_Jeffrey4l there is probably a git wizard out there that can make it happen13:45
sdake_i am not keen on the repo split for a whole lot of reasons, the pragmatic "how the hell do we do backports" one was never answered13:45
sdake_but we agreed to it as a team, so I got it kicked off13:45
sdake_you see, when we agree to something as a team, I get behind it ;)13:45
sdake_even if I dislike it :(13:46
Jeffrey4l( i do not like, too ;( )13:47
Jeffrey4lsync to git  repo tools? maybe13:47
Jeffrey4lsync to/sync two.13:48
*** Administrator_ has quit IRC13:51
*** Administrator_ has joined #openstack-kolla13:52
sdake_Jeffrey4l i think the first thing we need to do is focus on the gates13:52
sdake_Jeffrey4l and sort out backports later13:52
Jeffrey4lyes.13:52
sdake_Jeffrey4l do you ahve an immediate need to sort out backports now?13:52
Jeffrey4lno. but want to know how to backport.13:53
Jeffrey4la quick search http://stackoverflow.com/questions/3816040/git-apply-changes-introduced-by-commit-in-one-repo-to-another-repo13:53
sdake_Jeffrey4l what i'd suggest is sstarting a therad on ml13:55
sdake_Jeffrey4l we got a whole bunch of git nerds in the community who can hepl13:55
sdake_alot of people follow the [kolla] tag13:55
sdake_more then just the core team13:55
Jeffrey4lgood idea.13:55
sdake_ou might  tag it [git] too13:55
Jeffrey4lroger.13:55
*** fguillot has joined #openstack-kolla13:58
*** zhubingbing has joined #openstack-kolla14:04
*** Pavo has quit IRC14:07
*** Pavo has joined #openstack-kolla14:12
*** Administrator_ has quit IRC14:13
*** Administrator_ has joined #openstack-kolla14:14
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Allow operators to use 'fallback mode' for Ceph disks  https://review.openstack.org/38449614:14
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible: Allow operators to use 'fallback mode' for Ceph disks  https://review.openstack.org/39837314:15
v1k0d3nmorning all14:16
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla: Add a 'fallback mode' to disable udev in find_disks  https://review.openstack.org/38449614:16
openstackgerritcaoyuan proposed openstack/kolla: Remove the releasenotes about the ansible  https://review.openstack.org/39837414:17
srwilkersmorning v1k0d3n14:18
v1k0d3nportdirect_away_: would love to have atomic and fedora in there.14:18
v1k0d3nwe originally went with what kubeadm drafted for support, but welcome anything else you have in mind for PR's14:18
v1k0d3ni have some input on romana as well. been working closely with those guys, so ping me when you get back.14:19
v1k0d3ni saw some updates that lucas made to kubeadm, to allow it to run on CoreOS, so i was going to add that soon as well.14:19
v1k0d3nmorning srwilkers14:19
v1k0d3nportdirect_away_: and to be fair...kubeadm is really new. a lot of features need to be added in order to make it "prod-ready". but we're sure that it's also the way forward; for building and managing clusters, which is why we wrote playbooks for it.14:21
v1k0d3nand to be clear, the vagrant demo we have just uses various providers. so we use vagrant as the mechanism to deploy to local vbox or libvirt, but you can also deploy straight to an openstack cloud, and i was going to add AWS soon as well.14:22
v1k0d3ni wanted it to be flexible...to deploy anywhere a developer needs, so it just drops right in.14:23
v1k0d3nor you can just use the ansible playbooks. whatever works.14:23
*** lamt has quit IRC14:23
sdake_Jeffrey4l so, who is leading the merry effort of the repo split14:24
sdake_Jeffrey4l is that you?14:24
Jeffrey4li do know ;(14:24
sdake_you do know?14:24
sdake_or not know14:24
Jeffrey4ldo not14:24
Jeffrey4lsorry.14:24
sdake_ok, well it can't be me unfortunately14:24
Jeffrey4lbtw, i am working on the gate right now.14:25
sdake_cool14:25
sdake_perhaps inc0 can provide some technical leadersihp here14:25
zhubingbing-)14:25
Jeffrey4lyep.14:25
*** eaguilar has joined #openstack-kolla14:25
sdake_Jeffrey4l i know that fungi is right about submodules14:25
sdake_Jeffrey4l (re ml)14:26
sdake_and the key thing I'd go for is cross-repo gating14:26
sdake_I hadn't even thought of that until fungi brought it up14:26
Jeffrey4lare u going to use submodule?14:26
sdake_its more expensive in terms of gate resources14:26
sdake_fungi recommends strongly against it14:26
sdake_fungi is infra ptl and a pro at how all this stuff works14:26
sdake_take his word as gospel14:26
sdake_in fact any of the infra folks word is pretty much fospel14:27
sdake_gospel to me ;)14:27
sdake_that whole team rocks14:27
sdake_they have helped us out so much, its not even funny14:27
sdake_Jeffrey4l there is a msg on the ml from jeremy14:28
sdake_(fungi)14:28
sdake_Jeffrey4l he suggests what to do14:28
sdake_use zuul-cloner14:28
Jeffrey4lOK. i saw it.14:28
Jeffrey4llet me check it carefully.14:28
sdake_there is an example of zuul-cloner usage in the gate scripts of kolla-kubernetes14:28
Jeffrey4lOK14:28
*** dave-mccowan has quit IRC14:30
v1k0d3nbig fan of submodules. we use them.14:31
v1k0d3nnot sure how well that would work here. perhaps fine.14:31
Jeffrey4lv1k0d3n, any benefit?14:31
v1k0d3nhas anyone worked with subtrees? i hear that in some ways it can be better.14:31
*** jtriley has joined #openstack-kolla14:31
v1k0d3nwell, yeah...with submodules the projects can act completely independent of each other. that's extremely nice.14:32
v1k0d3nJeffrey4l: have you used them before?14:32
v1k0d3nwe use them for our kube develop environment that srwilkers was talking about earlier.14:33
v1k0d3nhttps://github.com/att-comdev/halcyon-vagrant-kubernetes14:33
Jeffrey4lno ;(14:33
*** schwicht has joined #openstack-kolla14:33
v1k0d3nkeeps the footprint very low and extremely targeted on our project.14:34
v1k0d3nin our case, we used submodule for this vagrant/kube ansible playbooks because people thought that vagrant was for local development only...which was a bad misunderstanding.14:34
v1k0d3nso creating the submodule separation makes the goals more clear to users14:35
sdake_v1k0d3n reference mailing list where jeremy stanley says "submodules bad" :)14:35
v1k0d3ni see a good reason for using them when separating these projects honestly.14:35
v1k0d3nyeah, not saying it's the best solution ever...14:36
Jeffrey4lzuul-cloner should be the better solution.   kolla and kolla-ansible is not kind of submodule project.14:37
*** inc0 has joined #openstack-kolla14:37
inc0good morning14:37
Jeffrey4l zuul-cloner should allow us to use depends-on directive.14:37
inc0how are is my favorite community today:)14:38
srwilkersinc0, not caffeinated enough14:38
inc0ahh the hard stuff14:38
inc0Jeffrey4l, so I was thinking14:38
inc0we have 2 options14:38
inc01. in kolla-ansible zuul-clone kolla14:38
inc0that will give us build gates before we create docker registry14:39
inc0but on the other hand, if we use submodule in kolla to clone both kolla-ansible and kolla-k8s14:39
sdake_inc0 i think the key point of using zuul cloner from fungi's msg was that we could do cross-repo gating14:39
inc0yes14:40
sdake_inc0 i.e. kolla repo could gate both kolla-ansible and kolla-kubernetes14:40
inc0with latter users will be able to git clone --recursive kolla and have orch tools in place14:40
Jeffrey4li'd like to use option 1 inc014:40
Jeffrey4lif we use submodule, why we split repo?14:41
inc0well, we'll have all the benefits of repo split but loose discomfort for users of it14:41
*** rhallisey has quit IRC14:41
*** lamt has joined #openstack-kolla14:41
*** rhallisey has joined #openstack-kolla14:41
sdake_submodule is pretty much incompatible with gerrit14:41
sdake_i keep telling people this...14:42
sdake_and people keep on not listening...14:42
Jeffrey4lbut submodule tie these three repos into one, imho.14:42
v1k0d3nok. totally fine if something doesn't work. so let's find something that would.14:42
inc0submodule will not need use gerrit at all14:42
v1k0d3nsubmodule is just a standard git function14:43
inc0kolla will have build gates, period14:43
v1k0d3n(which i know we realize, im sure)14:43
inc0no need of gerrit magics14:43
v1k0d3nprobably stating obvious.14:43
sdake_inc0 i think kolla should also gate on kolla-kube and kolla-ansible14:43
inc0but people will still be able to git clone --recursive locally and not deal with 3 different project instal14:43
inc0s14:43
sdake_that was BKP from fungi14:43
Jeffrey4linc0, i am adding load murano dynamic, how can i test this in ci with submodule? ( the kolla-ansible and kolla will both change )14:44
inc0we can use both14:44
Jeffrey4lsubmodule is hard for cross-repo ci test14:44
inc0submodule and zuul cloner to do effectively same thing14:44
inc0one for gerrit another one for users14:44
*** TxGirlGeek has joined #openstack-kolla14:45
sdake_inc0 recommend run that by infra :)14:45
Pavomorning gents14:45
sdake_inc0 just to make sure it doesn't break anything14:45
sdake_sup pavo14:45
sdake_inc0 you notice i  ran every step of the repo split by infra ;)14:45
sdake_so it iddn't get f'ed up ;)14:45
Pavonothing much14:45
*** schwicht has quit IRC14:45
*** mgoddard_ has joined #openstack-kolla14:46
sdake_and actually ajeager gave the most simple answer to the problem ;)14:46
inc0sdake_, totally, I just want to have idea agreed here before we validate it with them14:46
inc0in interest of their time14:46
inc0I know lots of projects uses submodule heavily14:46
sdake_oh didn't know othe projects use that14:46
inc0OpenStack Ansible of Fuel being examples14:46
sdake_guidance i've heard from infra in the past is avoid submodule14:47
sdake_i dont know if it would be appropriate in this particular case14:47
*** chas has quit IRC14:47
inc0yeah remember this fuel shitshow of every Dockerfile in dedicated repo?14:47
sdake_thats where this is coming from - past discussions14:47
inc0this is how they'd bundle it together14:47
inc0in OSA every role is separate proejct14:47
sdake_i'll be completely honest, I have no idea what the hell submodule does14:47
*** lrensing has joined #openstack-kolla14:47
*** chas has joined #openstack-kolla14:47
sdake_all I know is infra doesn't like it ;)14:47
sdake_infra doens't like - i don't like14:48
inc0when you git clone repo it will git clone another repo inside your cloned repo14:48
sdake_guess a bit of cargo cult thinking there14:48
inc0as simple as that14:48
v1k0d3ni probably missed the reasons for not using submodules. what are the reasons submodules don't work again?14:48
*** rhallisey has quit IRC14:48
*** rhallisey1 has joined #openstack-kolla14:48
Jeffrey4linc0, fuel and openstack-ansible use independent ansible role and puppet module, so submodule should be better.14:48
sdake_v1k0d3n not sure - its been awhile since i had the discussion with infra14:48
inc0infra said that zuul cloner is better for infra, allows us to use cross repo gates and whatnot14:48
Jeffrey4lbut kolla is not.14:48
sdake_v1k0d3n join #opnestack-infra14:48
Jeffrey4lkolla-ansible only use kolla's image.14:49
*** mgoddard has quit IRC14:49
Jeffrey4lthere is no need ti tie these two repos into one by using submodule.14:49
inc0Jeffrey4l, agree, but it would remove a step from kolla deployment14:49
v1k0d3nso perhaps we make a hard split with kolla and kolla-kube then, if submodules don't work.14:49
inc0I think it'd be better for users14:49
v1k0d3nthey are different projects, using different methods of deployment and templates for deployments.14:50
Jeffrey4lfor end-user, build image and deploy image should in different nodes.14:50
v1k0d3nentirely different approach, workflow, everything except containers.14:50
inc0Jeffrey4l, agree, could be, actually it's a good point14:50
*** schwicht has joined #openstack-kolla14:50
inc0we could refactor our docs to have deploy node and build node14:50
Jeffrey4land we can provide kolla rpm/deb package for better usage experience14:50
inc0as opposed to just deploy node14:50
sdake_our docs need moar love anyway14:51
inc0apt-get install kolla-ansible? now that's interesting14:51
Jeffrey4lsomething like that.14:51
inc0operators would love that14:51
*** zhurong has quit IRC14:51
*** TxGirlGeek has quit IRC14:51
portdirectgithub down - can i go home now?14:51
*** tonanhngo has joined #openstack-kolla14:51
sdake_portdirect your dismissed ;-)14:52
*** TxGirlGeek has joined #openstack-kolla14:52
inc0again?14:52
inc0they just had big DDoS14:52
inc0did east coast learned nothing>14:52
portdirectlooks like it - who they piss of this time? :)14:52
v1k0d3ninc0: and Jeffrey4l what's the difference with apt vs pip install kolla-ansible?14:53
dimsinc0 : i felt it too ("You can't do this operation at this time")14:53
inc0v1k0d3n, nothing much, just operators love apt14:53
egonzalez90not deal with pip dependencies14:54
*** rhallisey1 has left #openstack-kolla14:54
v1k0d3nso to be clear, we are only talking about for the kolla-ansible portion?14:54
portdirectinc0: nothing much, just operators love yum (FTFY)14:54
v1k0d3n^^ and there it is.14:55
v1k0d3ncan we just say that operators love whatever they are familiar with?14:55
*** lrensing has quit IRC14:55
egonzalez90i would add kolla and kolla-ansible(for now), rdo and ubuntu will not build non prod packages14:55
Jeffrey4lyes another solution is add kolla into kolla-ansible' requirements.txt file.14:55
egonzalez90they don't even build some prod services14:56
Jeffrey4lthen pip works14:56
Jeffrey4lonly the guys who install from git url need clone two repo.14:56
Jeffrey4li think that's fine.14:56
*** lrensing has joined #openstack-kolla14:56
*** adrian_otto has joined #openstack-kolla14:57
*** rmart04 has joined #openstack-kolla14:58
*** mgoddard has joined #openstack-kolla14:58
*** sdake has joined #openstack-kolla14:58
*** mgoddard_ has quit IRC14:59
sdakeinc0 this needs a +1 https://review.openstack.org/39690114:59
sdakeinc0 my governance repo change had a merge conflict14:59
*** coolsvap_ is now known as coolsvap14:59
*** rhallisey has joined #openstack-kolla14:59
inc0done14:59
*** schwicht has quit IRC14:59
sdakethanks15:00
sdakebtw, today is release day15:00
sdakejust throwing that out there ;-)15:00
Jeffrey4lok.15:00
Jeffrey4lusing the latest commit hash, right?15:01
sdakei think its typicallybad to release a DOA repo15:01
sdake;)15:01
sdakeJeffrey4l persoanlly I'd go back prior to the split for kolla hash15:01
sdakeand for kolla-ansible use current hash15:01
sdakesince that should be good i think?15:01
Jeffrey4lok.15:02
*** sdake_ has quit IRC15:02
sdakenot sure story on kolla-kubernetes tag15:02
Jeffrey4lrelease kolla ansible kolla-ansible in one PS or two?15:02
sdakeJeffrey4l each one has to be separate15:02
Jeffrey4lroger.15:02
sdakebecause of the release team's processes are sort of broken15:02
sdakenot broken15:03
sdakejust not evolved to handle tha t case yet15:03
sdakejoin 3openstack-release and ask there15:03
sdakemaybe they have fixed that15:03
*** magicboiz has quit IRC15:03
sdakeactually i think the release team's processes are fantastic15:03
inc0sdake, when your change to govenrance repo merges, I'll make Jeffrey a release liason15:03
sdakebut some of their tools don't handle multiple commits and stuff15:03
sdakeinc0 roger - dont know if that iwll be today then15:04
sdakeinc0 but we have to tag today i think15:04
Jeffrey4lhmm, let me push the PS first.15:04
inc0ok, I'll put his name now15:05
*** msimonin has quit IRC15:05
inc0people knows you anyway sdake15:05
sdakeinc0 you do have a point tho, not sure if release team will release somethign that isn't a deliverable in the gov repo15:05
sdakebest to ask ;)15:05
*** adrian_otto has quit IRC15:06
inc0Jeffrey4l, congrats, you're hereby our release manager:) https://wiki.openstack.org/wiki/CrossProjectLiaisons#Release_management15:08
Jeffrey4linc0, thanks ;)15:08
inc0don't thank me, I just gave you new job nobody will pay for;)15:09
inc0but on the other hand, thanks Jeffrey for taking on it:)15:09
sdakeJeffrey4l = winning;)15:11
*** msimonin has joined #openstack-kolla15:12
Jeffrey4l;)15:12
*** msimonin has quit IRC15:13
*** schwicht has joined #openstack-kolla15:13
inc0sdake, could you coach Jeffrey4l on his new job? :)15:13
sdakei tihnk i trained him on this process already15:13
sdakeJeffrey4l could you confirm or deny?15:14
sdakei trained you coolsvap ryan and jefrey4l iirc15:14
inc0and release 4.0.0b1 on all projects as soon as it's ready?15:14
sdake4.0.0.0b115:14
Jeffrey4lyep.15:14
inc0yeah, I did it once, but I can't remember full process tbh15:14
*** msimonin has joined #openstack-kolla15:14
Jeffrey4li can handle this.15:14
inc0ok, I'll leave it to you guys15:14
inc0thanks15:14
sdakeJeffrey4l if you need any help reach out :)15:14
Jeffrey4l1. push release PS in openstack/release,   2. make new milestone and move all old bug in o-1 into o-2. right? sdake15:15
sdakethats what i've been doing15:15
sdakei'm not sure its the smartest approach15:15
sdakewe hae like 350 open bugs15:15
sdakethat model used to work when we had less bugs15:16
*** yingjun has joined #openstack-kolla15:16
sdakeseing 300+ bugsi n o2 is just... depressing15:16
sdakesort of stuck on how to ramp up our triage efforts15:16
coolsvapsdake: i tried to use the release tools script but i was not able to get that working to bulk move bugs15:16
Jeffrey4lhmm.. maybe need some scripts.15:16
Jeffrey4lcoolsvap, where is it?15:17
sdakecoolsvap cool dude that would have rocked if that worked thanks for tyring :)15:17
Jeffrey4lthe scripts.15:17
coolsvapi believe it works on ubuntu15:17
sdakeJeffrey4l its also more complex because we have a kolla-ansible launchpad as well15:17
*** msimonin has quit IRC15:17
coolsvapbut not sure15:17
*** msimonin has joined #openstack-kolla15:17
Jeffrey4lcoolsvap, never mind. could u give me the script link?15:18
coolsvapJeffrey4l: just a min15:18
Jeffrey4lsdake, yes. and some bugs may need split into two, one for kolla and another for kolla-ansible. ;(15:18
sdakeJeffrey4l recommend solving one problem at a time ;)15:19
sdakei guess bulk move for now15:19
sdakeperhaps inc0 can lead a discussion on how to do bug triage the right way15:19
*** lrensing has quit IRC15:19
sdakeor rather bug management15:19
sdakewe seem to be triaging ok15:19
inc0<< reading about launchpad API >>15:20
coolsvapJeffrey4l: https://github.com/openstack-infra/release-tools/blob/master/process_bugs.py15:20
Jeffrey4lcoolsvap, cool.15:20
sdakeJeffrey4l that only works on ubuntu as coolsvap said15:20
sdakei've tried to use that lib that is a dep of process_bugs on centos before - no go15:20
Jeffrey4lthat's OK. i can set up a ubuntu container. ;)15:21
Jeffrey4lwill try it.15:21
sdakecool :)15:21
Jeffrey4llet me push the release patch set.15:21
inc0I think we could copy all the bugs from kolla to kolla ansible (also affects)15:21
sdakecan't do patch sets15:21
*** yingjun has quit IRC15:21
inc0and just invalidate those which are clearly ansible or kolla15:21
inc0I don't expect too many of those15:21
*** yingjun has joined #openstack-kolla15:21
Jeffrey4lso one for kolla, one for kolla-ansible, how about the kolla-k8s?15:21
v1k0d3nJeffrey4l: yes15:22
v1k0d3ninc0: at least i think...right?15:23
coolsvapi moved most blueprints to kolla-ansible15:23
inc0coolsvap, thanks15:23
sdakecoolsvap manually?15:23
coolsvapi think we can move bugs as we review patchsets15:23
coolsvapsdake: yes15:23
* sdake comforts coolsvap15:24
sdakeya that manual bug moving thing sucks15:24
sdakekarma ++ :)15:24
inc0coolsvap, yeah it might be better15:25
inc0just slowly wait till system handles itself;)15:25
*** absubram has joined #openstack-kolla15:25
inc0friendly reminder, meeting in circa 30min15:27
openstackgerritzhubingbing proposed openstack/kolla: Fix congress broken  https://review.openstack.org/39841315:29
sdakeinc0 mind updating the topic for devs with something like "Ansible code should go to repo http://github.com/openstack/kolla-ansible"15:29
sdakeso I don't hae to keep -2ing these reviews ;)15:30
sdake12 pages of reviews for kolla :(15:33
sdakeeverything backed up15:33
openstackgerritSteven Dake proposed openstack/kolla: Clean up tox.ini for Ansible removal  https://review.openstack.org/39802815:33
openstackgerritSteven Dake proposed openstack/kolla: Remove Ansible from Kolla  https://review.openstack.org/39802515:33
srwilkerssorry, i try to review as much as I can throughout the day -- not as much as I'd like to recently15:34
*** mgiles has joined #openstack-kolla15:35
sdakesrwilkers its ok, most of the reviews need to be redone anyway15:35
sdakesrwilkers as a result of the repo split15:35
srwilkersyeah, thats what i was anticipating15:35
sdakefwiw, I did the work on saturday :)15:36
sdakeand it landed on tuesday15:36
sdaketo minimize disruption15:36
sdakebut this type of disruption can't be resolved easily15:36
*** ChanServ sets mode: +o inc015:37
srwilkerswhats life without a little disruption15:37
inc0New to Kolla? Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla  http://goo.gl/VKpPzA); w00t for Kolla!; Kolla-ansible is up! All ansible changes goes there now15:38
*** inc0 changes topic to "New to Kolla? Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA); w00t for Kolla!; Kolla-ansible is up! All ansible changes goes there now"15:38
*** jtriley has quit IRC15:38
*** inc0 sets mode: -o inc015:38
*** rhallisey has quit IRC15:43
*** rhallisey has joined #openstack-kolla15:43
*** jtriley has joined #openstack-kolla15:46
*** coolsvap is now known as coolsvap_15:46
openstackgerritJeffrey Zhang proposed openstack/kolla: install murano dashboard package in horizon  https://review.openstack.org/39595715:56
*** rhallisey has left #openstack-kolla16:00
Pavoman they need to add an option to network boot instances in openstack16:00
*** coolsvap_ is now known as coolsvap16:00
*** rhallisey__ has joined #openstack-kolla16:00
v1k0d3nmeeting started now?16:01
egonzalez90yup v1k0d3n16:01
sdakeoh16:04
openstackgerritMauricio Lima proposed openstack/kolla-ansible: Remove docker folder from kolla-ansible  https://review.openstack.org/39832016:04
*** dave-mccowan has joined #openstack-kolla16:04
*** DTadrzak has quit IRC16:07
*** Pavo has quit IRC16:07
*** Pavo has joined #openstack-kolla16:08
*** yingjun has quit IRC16:08
*** TxGirlGeek has quit IRC16:09
*** rhallisey__ is now known as rhallisey16:17
*** rhallisey1 has joined #openstack-kolla16:18
*** rhallisey1 has left #openstack-kolla16:22
*** adrian_otto has joined #openstack-kolla16:26
*** krtaylor has quit IRC16:29
*** magicboiz has joined #openstack-kolla16:31
*** TxGirlGeek has joined #openstack-kolla16:32
*** rmart04 has quit IRC16:39
*** krtaylor has joined #openstack-kolla16:43
*** strigazi is now known as strigazi_AFK16:46
*** jistr is now known as jistr|biab16:48
*** ppalacios has quit IRC16:49
pbourkesorry missed meeting, need to update my calendar for the clock change16:51
*** absubram has quit IRC16:53
*** portdirect is now known as portdirect_away16:53
*** ppalacios has joined #openstack-kolla16:53
wirehead_Yeah, it keeps messing me up as well.16:53
wirehead_I'm a huge fan of outlawing daylight savings time in all forms.16:53
wirehead_Also: https://xkcd.com/673/16:55
duonghqanybody got external_ip of k8s service holding "pending" state?16:59
Jeffrey4lrhallisey, what's the release status for kolla-k8s? can we release it now?16:59
*** zhubingbing has quit IRC17:00
rhalliseyJeffrey4l, I don't understand the question.  Release for what?17:00
Jeffrey4lrelease kolla-k8s o-117:00
*** absubram has joined #openstack-kolla17:01
rhalliseyI don't think kolla-k8s will follow it until 1.017:01
rhalliseythere is no stable branch yet17:01
Jeffrey4lrhallisey, ok.17:01
rhalliseyJeffrey4l, the plan is to move to a kolla like release schedule after O17:01
Jeffrey4lgot it, thanks.17:02
*** ppalacios has quit IRC17:02
duonghqrhallisey, did you see k8s service in external_ip pending state?17:02
rhalliseyduonghq, are you talking about setup?17:03
Jeffrey4linc0, do u have any idea for backport kolla-ansible change to stable branch?17:03
duonghqrhallisey, I use kubeadm to spin up a 3 nodes k8s-cluster17:03
duonghqwhen I create a service (out of 2-nginx pod)17:03
duonghqthe external IP of service is set pending forever17:04
duonghqI apply kube-flannel already17:04
rhalliseyhm17:05
rhalliseyduonghq, http://kubernetes.io/docs/user-guide/load-balancer/17:06
*** ppalacios has joined #openstack-kolla17:06
kfox1111morning17:06
duonghqmorning kfox111117:06
*** sdake has quit IRC17:06
rhalliseyduonghq, that's strange you have it stuck in pending17:06
*** ppalacios has quit IRC17:07
*** unicell1 has joined #openstack-kolla17:07
*** lrensing has joined #openstack-kolla17:07
kfox1111duonghq: kubectl describe pod <name/id of the stuck pod> ?17:07
duonghqI setup w/ kubeadm on ubuntu 16.04, 3 VM nodes17:07
rhalliseykfox1111, his external IP for his service is stuck in pending17:08
kfox1111external ip in the service?17:08
kfox1111never seen a "pending" state for a service.17:08
rhalliseysame17:08
kfox1111kubectl describe svc <servicename>?17:08
*** unicell has quit IRC17:08
*** sdake has joined #openstack-kolla17:08
kfox1111guessing the selector labels don't match the pod labels?17:09
duonghqmoment, I'm pasting it in some pastebin17:09
duonghqhttps://pastebin.mozilla.org/892882317:09
*** bmace has quit IRC17:10
*** bmace has joined #openstack-kolla17:10
duonghqpod's https://pastebin.mozilla.org/892882517:11
kfox1111that looks ok.17:11
kfox1111and curl 10.99.120.49 doesn't work?17:11
kfox1111or curl localhost:30826 ?17:11
*** egonzalez90 has quit IRC17:12
duonghqI cannot ping it17:12
kfox1111ping doesn't work in k8s.17:13
kfox1111only the tcp port.17:13
kfox1111does the curl work?17:13
duonghqcurl cannot get nginx welcome page too17:13
kfox1111on either of those?17:13
kfox1111what about  "curl 10.244.1.2:80" ?17:14
duonghqsame there17:14
kfox1111k. so somethings wrong at the docker networking layer.17:14
duonghqone note: if deployed flannel as k8s add-ons, follow kubeadm guide17:14
kfox1111I'd guess flannel.17:14
duonghq*I17:14
kfox1111I didn't think flannel itself had a kubeadm daemonset.17:14
kfox1111did you use canal?17:15
duonghqnot yet, I just follow kubeadm guide17:15
duonghqkfox1111, here: http://kubernetes.io/docs/getting-started-guides/kubeadm/17:16
kfox1111the kubeadm guilde last I looked said to pick a network plugin and launch with kubectl create -f ....17:16
kfox1111I thin kthe default recommended is weave,17:16
kfox1111but gives some other options.17:16
kfox1111step (3/4)17:17
*** msimonin has quit IRC17:17
kfox1111which points you at: http://kubernetes.io/docs/admin/addons/17:17
*** jemcevoy has quit IRC17:17
mliimapbourke, https://etherpad.openstack.org/p/kolla-repo-split17:17
kfox1111oh. looks like they have a pure flannel plugin now.17:17
duonghqkfox1111, I go through 4 steps, and add flannel afterthat17:18
kfox1111step 3 should have added flannel.17:18
kfox1111you did flannel with this template? https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml17:19
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture  https://review.openstack.org/39225717:19
duonghqyup17:19
kfox1111k.17:20
kfox1111maybe see what it looks like with: kubectl get pods --namespace kube-system17:20
duonghqI'm reseting cluster, and add network before joining other nodes17:20
kfox1111see if it all is happy.17:20
duonghqit was happy17:20
rhalliseykfox1111, it does have it's own daemonset17:20
duonghqevery pods17:20
kfox1111I don't think it matters when its added, but not 100% sure.17:20
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate  https://review.openstack.org/39850117:21
rhalliseyhas anyone gotten kolla-k8s to work with kubeadm?17:21
rhalliseyI had a bunch of issues when I tried it17:21
rhalliseybrb17:22
duonghqkfox1111, https://pastebin.mozilla.org/892882917:22
duonghqseems that everything is fine?17:22
duonghqrhallisey, so which method are you using for deploy k8s?17:23
sbezverkduonghq: pending state happens when there is already another process using the same port17:23
kfox1111yeah. and yet connectivity isn't working.17:23
kfox1111so somethings blocking traffic.17:23
kfox1111next suspicion is iptables maybe.17:23
*** matrohon has quit IRC17:23
*** rmart04 has joined #openstack-kolla17:24
kfox1111what base distro are you using, and did you disable the native firewall on it before starting?17:24
*** chas has quit IRC17:24
duonghqkfox1111, I'm using Ubuntu 16.04 server edition17:24
kfox1111I don't know ubuntu 16.04's defualt firewall. :/17:24
*** chas has joined #openstack-kolla17:24
kfox1111plz run an iptables-save on one of the compute nodes?17:25
duonghqkfox1111, seems that kubeadm or k8s add some rules to iptables,17:25
sbezverkrhallisey: demo we showed at the summit was done with kubeadm17:25
kfox1111duonghq: yeah. flannel does, docker does, and kube-proxy does. :)17:25
kfox1111but they can conflict with preexisting rules too. :/17:25
kfox1111my guess would be that the vxlan traffic for flannel is getting blocked.17:26
sbezverkduonghq: when you deploy networking, which cni did you use?17:26
kfox1111sbezverk: flannel17:26
kfox1111Id look at the cbr0 interface on each node, see what its ip is,17:26
kfox1111and  try to ping each one from one of the nodes.17:27
duonghqkfox1111, https://pastebin.mozilla.org/8928830 here is my minion1 iptables17:27
sbezverkduonghq: can you share yaml file you used for flannel deploy?17:27
duonghqsbezverk, I used this file: https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml17:27
kfox1111hmm... firewall looks ok to me.17:27
duonghqkfox1111, you mean cni0?17:28
kfox1111yeah.17:28
sbezverkduonghq: have you changed ip addresses to match to your environment?17:28
kfox1111sbezverk: duonghq: or, does "Network": "10.244.0.0/16", stop on any existing network?17:28
duonghqsbezverk, I used --pod-network-cidr=10.244.0.0/1617:28
*** chas has quit IRC17:29
duonghqkfox1111, https://pastebin.mozilla.org/892883117:29
duonghqseem that cni0 is down?17:29
sbezverkduonghq: can you run kubectl get svc --all-namespaces17:30
duonghqsome dmesg line: https://pastebin.mozilla.org/892883317:30
duonghqsbezverk, here: https://pastebin.mozilla.org/892883417:31
openstackgerritJeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment  https://review.openstack.org/39850617:31
sbezverkduonghq: flannel needs etcd service for ip addressing distribution17:32
sbezverkand it is not there17:32
kfox1111duonghq: just to double check, do an 'ip a' ?17:32
duonghqsbezverk, there is a etcd pod is running17:32
sbezverkI use canal.yaml and as a part of installation it install etcd instance17:32
kfox1111sbezverk: interesting. your right.17:33
kfox1111that flannel daemonset seems incomplete.17:33
sbezverkduonghq: maybe pod is running but servcice is not there17:33
duonghq https://pastebin.mozilla.org/892883517:33
sbezverkso nobody can talk to etcd pod17:33
duonghqsbezverk, understodo17:33
kfox1111unless there's some very different logic in flannel when --kube-subnet-mgr is passed.17:33
kfox1111yeah, I'd try the canal one. used it recently and it worked.17:34
sbezverkduonghq: I suggest you clean up your cni setup and install canal I install one this morning17:34
duonghqsbezverk, I'm trying, hope that it'll work17:35
sdakekfox1111 - was helm an optional peice or mandatory peice of our architecture?17:35
duonghqsbezverk, also, do you have any idea recover k8s cluster after all nodes is restarted (using kubeadm)17:35
kfox1111sdake: layer 3.17:35
sbezverkduonghq: yes, there is a clean up procedure17:35
*** jistr|biab is now known as jistr17:36
kfox1111you can, say, use kolla containers by themselves at layer 1 if your so inclined.17:36
kfox1111so nothings really manditory.17:36
duonghqsbezverk, I try reset but it's cleanup everything17:36
sbezverkduonghq: to remove your cni just run kubectl delete -f flannel.yaml17:36
*** absubram has quit IRC17:37
duonghqsbezverk, it's another question, I wonder if k8s cluster which is setup by kubeadm can be brought up again after failure17:37
sbezverkduonghq: not sure I undersatnd what you mean. bring up as a brand new installation or recover old one?17:38
sdakekfox1111 i think there is some confusion over what is mandatory or what is not17:39
duonghqsbezverk, I want to recover old17:39
sdakekfox1111 atleast I have some ;)17:39
*** fragatina has joined #openstack-kolla17:41
sbezverkduonghq: as long as the master is up, you can join it again from other nodes after you clean them up17:41
duonghqso, I have no luck if master is down atm?17:41
sbezverkyou just need to use the same id generated by kubeadm for the installation17:42
duonghqsbezverk, thanks17:42
sbezverkduonghq: I am afraid so, as the master is the key point17:42
*** unicell1 has quit IRC17:42
duonghqokay17:42
duonghqabout my external_ip stuck17:42
duonghqI got some log: https://pastebin.mozilla.org/892883917:43
duonghqI don't know why minion cni0 still down17:43
duonghqhttps://pastebin.mozilla.org/892884017:44
sbezverkduonghq: one sec, the output you provided shows that you use canal, is it a different installation?17:45
duonghqsbezverk, I reset all 3 nodes I do another deploy17:46
sbezverkduonghq: ok17:46
*** david-lyle has quit IRC17:48
*** david-lyle has joined #openstack-kolla17:48
sbezverkduonghq: hm in my canal installation, only two types of interfaces created. flannel and calico, not sure where cni0 is comming from17:48
duonghqsbezverk, so I guess that we can forget this17:49
duonghqmoment, I'll try kubeadm reset and reboot all nodes17:50
*** tonanhngo has quit IRC17:51
inc0sdake,17:51
inc0https://review.openstack.org/#/c/398025/17:51
inc0I think we should remove deploy gates at this moment and make build gates voting17:52
sdakesup17:52
sbezverkduonghq: if it does not work, I would suggest to use good clean up procedure and then proceed with installation over a webex session17:52
sdakewell i did have a followon patch that removed the deploy gates17:52
inc0and restore deploy gates along k8s voting gates when we'll deal with zuul-cloner17:52
sdakebut abanonded that17:52
duonghqsbezverk,  do I need to change etcd endpoints to 10.244.0.0/16 subnet?17:52
sdakemoment let me find the change17:52
inc0as they will look different17:52
inc0I just don't want to block development until we deal with zuul cloner17:53
sbezverkduonghq: yes, in canal.yaml you need to assign IP to etcd which will be used. I usually do and pick up next IP after dns services.17:53
sbezverkyou need to change it at two places17:53
sdakeya makes sense if people have trouble reading the gate output17:53
sbezverkin canal.yaml17:53
sdakewhich i think people do sometimes17:53
sdakehunting for change still moment17:54
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture  https://review.openstack.org/39225717:54
inc0having perma-red gates is worse than not having gates17:54
inc0as it will cause confusion17:54
inc0and we have enough of it aleready17:54
rhalliseysdake, so I have layer 3 and 117:54
sdakehttps://review.openstack.org/#/c/398028/17:54
rhalliseyactually helm is also optional17:54
inc0sdake, I'd restore this change17:55
inc0and merge it17:55
sdakewfm17:55
inc0and deal with gates systematically17:55
sbezverkrhallisey: kolla-kube is in templates and they are specific to a rendering engine17:55
openstackgerritMauricio Lima proposed openstack/kolla: Add how configure multiple back ends section  https://review.openstack.org/39851617:55
sbezverkI am not sure I undersatnd then the architecture17:55
inc0in fact, after that I would make all build gates voting17:56
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture  https://review.openstack.org/39225717:56
rhalliseysbezverk, helm packages those templates17:56
rhalliseylayer 2 is what you just mentioned.  It is not optional17:56
sdakeinc0 show me the mirrors and i'll submit the patch - otherwise feel free ;)17:56
rhalliseyneither is layer 1, the kolla containers17:56
sbezverkrhallisey: right, but then it makes helm as mandatory as nothing else can be used to render templates, no?17:56
inc0sdake, mirrors won't help really17:56
rhalliseysbezverk, helm is optional17:56
rhalliseyyou can render the templates by hand17:56
rhalliseyor kolla-k8s tools17:57
rhalliseycli17:57
inc0as mirrors didn't help us when canonical messed up versions17:57
kfox1111sbezverk: layer 2 is k8s.17:57
inc0but really changes to dockerfiles are pretty rare now17:57
*** lrensing has quit IRC17:57
kfox1111well, k8s/the rendered templates to k8s.17:57
inc0so I think we can affort not merging them for these few days of breakages17:57
sdakeinc0 right - i expect the dockerfiles to have a low rate of change17:57
inc0at least it will motivate people to fix actual issues fast;)17:57
sdakethe problem is during final release17:58
sbezverkkfox1111: I am lost :-( now we have jinja2 templates and they are rendered by cli tool. I thought we will scrap/convert them into helm packages, right?17:58
kfox1111sbezverk: right.17:58
sdakewhen upstream binaries are busted and we have to release17:58
inc0if you can't build in gates - you can't build locally - you can't test change - shouldn't be merged17:58
inc0sdake, only if all happends at once: 1. upstream is broken, 2. we found critical bug in dockerfiles17:59
sbezverkkfox1111: then we will have only helm charts and that makes helm as mandatory17:59
inc0which is rare17:59
kfox1111sbezverk: I think you have the same slight cognative dissonence with the layer thing I do, but I think its a very sublile thing so I wasn't worrying about it.17:59
sdakeinc0 try your hand at working on the gate then ;-)17:59
kfox1111sbezverk: there are charts, and there are templates.17:59
inc0yeah, I'll risk it and make it voting;)17:59
kfox1111the templates are golang templates. so could in theory be used by nonhelm too.17:59
sdakeshould be straightforward17:59
inc0but let's clear up what we have before we do that17:59
sdakeping me if you need guidance17:59
kfox1111but its very unlikely to be used without helm.17:59
sdakeyo core reviewers17:59
sbezverkkfox1111: right my bad, should not use them interchangably17:59
sdakehow about an ack on https://review.openstack.org/#/c/398025/18:00
duonghqsbezverk, does the warning matter? https://pastebin.mozilla.org/8928843 (every pods is running eventually)18:00
sdakethis will trigger merge conflicts against every patch that  touches that directory18:00
sdakeso we don't hae to keep -2ing changes18:00
inc0-30k loc...that's quite a patch18:00
* sdake pumping up his stats18:01
sdake;-)18:01
sbezverkduonghq: yes it does, it means you did not install cni rpm18:01
kfox1111sdake: call kolla repo cores.18:01
sbezverkso it is cmplaining about missing binaries18:01
kfox1111sdake: there are other sets of cores now.18:01
sdakekolla-repo cores, how about an ack on the above patch18:01
kfox1111inc0: hehe18:01
sdakekfox1111 thanks :)18:01
kfox1111sdake: np. :)18:02
inc0kfox1111, speaking of which18:02
inc0I'd remove inactive kolla-k8s cores18:02
duonghqsbezverk, strange, I install kubernetes-cnt package18:02
inc0rhallisey,18:02
rhalliseyyea18:02
inc0^18:02
rhallisey?18:02
duonghq*installed18:02
sbezverkduonghq: check ls /opt/cni/bin/ on all your workers18:02
rhalliseyinc0, kk18:03
inc0rhallisey, take a look at kolla-k8s cores and see if you want anyone added/removed18:03
rhalliseyI've been monitoring it18:03
rhalliseyI think it's fine atm18:03
sbezverkduonghq: you should see flannel18:03
inc0kk18:03
duonghqroot@master:~# ls /opt/cni/bin/18:03
duonghqbridge  calico  calico-ipam  cnitool  dhcp  flannel  host-local  ipvlan  loopback  macvlan  noop  ptp  tuning18:03
inc0kolla-ansible cores are separate team now fyi18:03
sbezverkduonghq: on other nodes too18:03
inc0just copy of kolla cores18:03
duonghqsbezverk, same on all nodes18:04
*** fragatin_ has joined #openstack-kolla18:04
openstackgerritJeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment  https://review.openstack.org/39850618:05
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate  https://review.openstack.org/39850118:05
*** Pavo has quit IRC18:07
*** fragatina has quit IRC18:08
sdakeok so i foresee a problem inc018:08
sbezverkduonghq: ok it seems it still cannot communicate with etcd "dial tcp 10.96.232.136:6666: i/o timeout" you need to make sure your config is consistent18:08
sdakewe had one set of processes for voting for policy and core reviewer nominations18:09
sdakewe now have 3 separate teams18:09
sdakei think we should have some discussion on ml about the separate teams and how the policies work18:09
duonghqsbezverk,  I do not know where the pod get the IP, I update all etcd-related IP in canal.yml18:09
sdakefollowed by a vote of the existing kolla core team to firm up whatever new policies we decide upon18:10
sdakeor even a superset of kolla + kolla-kubernetes18:10
*** v1k0d3n has quit IRC18:11
sdakealthough the later isn't how our policies are currently set18:11
kfox1111best guess is kolla-ansible needs some cores, kolla-kubernetes needs some cores, and maybe kolla containers repo should be the union of the other teams?18:11
sdakekfox1111 ya - decision for the ml ;)18:11
*** pbourke has quit IRC18:11
inc0sdake, I think obvious resolution would be that kolla-k8s cores votes on kolla-k8s cores and so on18:11
kfox1111sdake: sure. just a suggestion.18:11
kfox1111yeah.18:12
sdakeinc0 right but that is a change to our "governance"18:12
*** pbourke has joined #openstack-kolla18:12
sdakeinc0 hence needs a vote18:12
sdakeinc0 woudl you mind starting a thread?18:12
sbezverkduonghq: there are many reason for things to go wrong and it is difficult to troubleshoot over irc channel. I would clean up everything and proceed with clean installation together on webex session..18:12
*** Pavo has joined #openstack-kolla18:12
*** diogogmt has joined #openstack-kolla18:13
sdakeall policy decisions (including changing policies) requires a mailing list vote.. :)18:13
sdakewe decided this long long ago18:13
sdakeinc0 what is logical to you may not be logical to someone else18:13
inc0sdake, sure, and every core has voting rights for this particular policy change18:13
inc0including kolla-k8s cores18:13
sdakegrey area, but wfm18:13
sdakein the true sense of governance that wouldn't be the case18:14
inc0kolla-k8s is no longer second class citizen, same rules apply18:14
*** absubram has joined #openstack-kolla18:14
sdakewe can't make policy changes by fiat18:14
*** harlowja has joined #openstack-kolla18:14
sdakethat just pisses everyone off18:14
sdakeneeds to be a proposal and vote on ml18:14
inc0on it18:14
sdakethanks :)18:14
rhalliseywhat's the vote for?18:16
*** unicell has joined #openstack-kolla18:16
rhalliseyI think kolla-k8s cores should vote on kolla-k8s cores18:17
sdakerhallisey our new policies going forward18:19
sdakerhallisey lets have the discussion over email pls18:19
sdakerhallisey governance really tricky18:19
sdakeirc discussion imo isn't good enough18:19
sdakeemail/ml i mean ;)18:19
rhalliseyk18:19
rhalliseyI'm jumping in and out of spec & irc18:20
sdakeroger18:20
rhalliseyso ml would be better18:20
*** eaguilar_ has joined #openstack-kolla18:21
*** eaguilar has quit IRC18:22
inc0done18:23
inc0voting is up till end of month18:23
inc0rhallisey, Jeffrey4l pbourke https://review.openstack.org/#/c/398025/ mind acking?18:25
sdakekfox1111 i was thinking about kolla-kubernetes in light of inc0's proposal18:30
sdakekfox1111 I was reminded of the fact we never set any policies around kolla-kubernetes18:30
inc0yup18:30
sdakesince none were set, i think the appropriate action is to "use your best judgement"18:30
inc0about time;)18:30
rhalliseywhat are you reffering to?18:30
sdakeya my fault for not recongnizing that18:30
rhalliseyby policies sdake18:30
sdakerhallisey the ml18:30
sdakewho can vote on core review changes, who can vote on policy changes18:31
inc0we just thrown people into core team for kolla-k8s18:31
rhalliseykolla-kubernetes has policies18:31
rhalliseyI was destibuting them by irc not be ML :)18:31
sdakei mean the kind on the mailing list18:31
rhalliseyfor other reasons :)18:31
sdakethat come with a vote18:31
rhalliseyfor ex) voting policy in kolla-k8s only takes one +2 atm18:31
*** Administrator_ has quit IRC18:32
rhalliseysince the core team isn't super large18:32
*** tonanhngo has joined #openstack-kolla18:32
*** Administrator_ has joined #openstack-kolla18:32
rhalliseyinc0, sorry confused. Who is getting thrown into the core team?18:33
sdakewell since those policies weren't recorded or voted on, just consider kolla-kubernetes prior to the policy setting we are doing to use your best judgement18:33
sbezverkinc0: what about voting for changes in kolla repo? some changes in kolla may impact k8s as well18:33
inc0sbezverk, we have kolla core team18:33
inc0me, sdake, rhallisey are still kolla cores:)18:33
sbezverkinc0: you have, but k8s team have no say in there18:33
sdakeand kfox ;)18:33
inc0which means we won't allow breaking changes18:33
sdakeand serguey18:33
sdakesorry sbezverk for butchering your name18:34
sbezverknp18:34
sdakeoh you mean kolla-core team18:34
inc0they aren't cores in kolla last time I checked;)18:34
rhalliseyk I get the policy you are referring to now18:34
inc0if you guys wanna become them...well review kolla;)18:34
*** tonanhngo has quit IRC18:36
kfox1111inc0: yeah, I have no core'nes on the container side.18:36
kfox1111but depend on it heavily. :)18:37
sbezverkinc0: ok I hope this rule will apply to becoming k8s core as well, so people need to review to become k8s core as well;18:37
sean-k-mooneysbezverk: isnt that how core membership works for all openstack poject outside the people who made the repo18:38
*** tonanhngo has joined #openstack-kolla18:38
inc0sbezverk, rules are the same for all teams18:38
*** tonanhngo has quit IRC18:39
*** tonanhngo has joined #openstack-kolla18:39
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture  https://review.openstack.org/39225718:40
rhalliseyany more comments?18:41
kfox1111inc0: I'm still not familior with the kolla mailing list voting process.18:44
kfox1111if I agree with your resolution, whats my response?18:44
rhalliseykfox1111, it would be +118:44
kfox1111k.18:45
rhalliseyand you can toss a why in there if you want18:45
kfox1111thats what I was going to do, but second guessed myself. THanks for the clarification. :)18:45
*** jtriley has quit IRC18:46
*** huhaoran has joined #openstack-kolla18:47
sdakekfox1111 also if you want alterations feel free to discuss that :)18:48
*** rmart04_ has joined #openstack-kolla18:48
sdakeour voting process is a little haphazard often causing second votes after we come to agreement :)18:48
inc0as long as result is 9.2, we're ok18:48
rhalliseyI think the spec is good to go18:49
kfox1111sdake: its close enough. :)18:49
inc0guys, hate me if you want18:49
*** rmart04 has quit IRC18:49
*** rmart04_ is now known as rmart0418:49
rhalliseydid anyone else have any more comments?18:49
inc0but I just proposed making build source gates voting18:49
inc0https://review.openstack.org/#/c/398534/1/zuul/layout.yaml18:49
*** athomas has quit IRC18:50
sdakerhallisey would you mind rubber stamping this https://review.openstack.org/#/c/398025/18:52
sdakerhallisey on which?18:53
inc0kfox1111, ad genconfig18:54
inc0genconfig for time being will stay in ansible18:54
rhalliseysdake, spec18:56
sdakerhallisey reviewing18:56
sdakei saw i didn't read it :)18:56
sdakerhallisey lgtm19:01
*** shardy is now known as shardy_afk19:01
sdakei'm reminded of a documentary i saw on clinton's first 100 days in office19:04
*** msimonin has joined #openstack-kolla19:04
sdakestill needs a rubber stamp - repo split  related: https://review.openstack.org/#/c/398025/19:05
sdakeJeffrey4l rhallisey ^^19:05
*** coolsvap has quit IRC19:07
kfox1111inc0: just asking now, as it will be more painful to move it back to kolla later.19:10
openstackgerritMerged openstack/kolla: Remove Ansible from Kolla  https://review.openstack.org/39802519:10
kfox1111and now it doesn't matter. :)19:10
inc0kfox1111, it's going to be painful and volitale anyway19:11
inc0one crysis at the time19:11
*** ccesario has quit IRC19:11
kfox1111inc0: fair enough. :)19:12
kfox1111lets deal with the current one. :)19:12
sdakekfox1111 fwiw we need to sort that out - i agree completely19:14
sdakekfox1111 not sure how to do so ;)19:14
sdakekfox1111 you wre talking about the config splitout right?19:15
*** shardy_afk is now known as shardy19:16
sdakesup shardy19:16
*** shardy is now known as shardy_afk19:17
*** msimonin has quit IRC19:19
sdakerhallisey19:20
*** haplo37_ has quit IRC19:20
*** g3ek has quit IRC19:20
sdakethis slack channel you speak of19:20
sdakeyou were tlaking about adding an irc gateway?19:20
sdakehow does that happen?19:20
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture  https://review.openstack.org/39225719:21
britthouserAre you wanting to use IRC client to connect to slack channel, or use slack client to connect to IRC server, sdake?19:21
sdakebritthouser the first19:21
sdakewhat would rock is if the channels were osmehow synced19:21
rhalliseysdake, https://sameroom.io/blog/mirror-freenode-channels-in-slack/19:21
britthouserThe admin of the channel has to enable it, then its self service.19:21
rhalliseysdake, they can be synced19:21
rhalliseythe issue is slack19:21
rhalliseykube slack has a cache of 10k live messages19:22
sdakerhallisey could you go into more detail19:22
rhalliseysure19:22
rhalliseyslack is different then irc19:22
sdakei understand they are different htings ;)19:22
britthouserI use <https://github.com/aeirola/slack-irc-client> to do #219:22
rhalliseythe entire group, kubernetes, has 10k lives messages19:22
rhalliseythat means19:22
sdakethats why you mentioned the 2 msgs per second thing prior19:23
rhalliseyif you type something it stays in the scroll back until it's message 10000 in the cache19:23
sdakerhallisey is that per channel or per group?19:23
rhalliseyright now, kube goes through that cache in 2 days19:23
rhalliseygroup19:23
rhalliseyso all the individual channels share19:23
sdakeright19:23
rhalliseywe have about 2 messages per minute19:24
rhalliseychanserv stuff19:24
sdakewhat does that work out per to day19:24
rhalliseyincluding chanserv stuff19:24
rhallisey288019:24
sdakeyikes so we would be taking 28% of the cache per day?19:24
rhalliseythey flip in 48 hours19:24
rhalliseyyes19:24
rhallisey:D19:24
rhalliseywe would be their most active channel19:24
sdakeso kube slack admins not keen on increasing the capacity?19:24
rhalliseythey are19:25
britthouserincreasing capacity == $$19:25
sdakesweet19:25
rhalliseyidk when it will happen though19:25
britthouserFree slack cuts off at 10k messages19:25
britthouserpaid slack is unlimited.19:25
sdakei see19:25
rhalliseyohh19:25
sdakewell thats a huge delta19:25
sdakehow much is paid slack britthouser ?19:25
britthouserNot sure, but its a per user charge.19:26
sdakegroan19:26
sdakemake it harder19:26
britthousernot cheap from what I understand.19:26
kfox1111britthouser: yeah. its per user.19:26
kfox1111but I think slack should consider waving the fee for openstource projects like this, to get more users confortable with slack.19:27
sdakeso rhallisey kubernetes admins not keen to connect our channels?19:27
*** matrohon has joined #openstack-kolla19:27
kfox1111its a win win for them.19:27
sdakekfox1111 make it really harder! :)19:27
kfox1111:)19:27
inc0I'm not a big fan of slack19:27
inc0yes, it's flashy19:27
sdakethats why we are tlaking about connecting them19:27
*** jtriley has joined #openstack-kolla19:27
rhalliseysdake, well we would damage their scroll back19:27
kfox1111I'm a fan of the interface. the fact that its closed though, not so much.19:27
sdakebecause other people are big fans19:27
sdakerhallisey right19:28
kfox1111the big selling point of slack is the cell phone integration.19:28
rhalliseybut it's all logged19:28
rhalliseythe issue though is more projects19:28
rhalliseyand more growth there19:28
kfox1111after you use it for a while, tis really hard to use irc again.19:28
sdakeat some point kubernetes group on slack ging to have to pay i think or get a waiver19:28
rhalliseyya it is neat19:28
inc0idk, I tried a to use it couple of times19:28
sdakeso back on topic :) we would dmg their scrollback capacity19:29
sdakeand kubernetes group admin is negatory on that proposal?19:29
*** haplo37_ has joined #openstack-kolla19:29
sdakeor has it been proposed19:29
kfox1111inc0: it really shines when you ahve a group of people that needs to get a hold of each other. the pinging that actually reaches the human no matter where thery are is the killer feature there.19:29
kfox1111yeah. we'd blow out their cache pretty quickly.19:29
rhalliseysdake, I haven't mentioned we are 2880 messages a day19:29
sdakeinstead of guessing i'd like to know from the src :)19:29
*** absubram has quit IRC19:30
inc0google could sponsor it19:30
*** duonghq has quit IRC19:30
sdakerhallisey that is a key peice of info you need to pass along19:30
*** g3ek has joined #openstack-kolla19:30
inc0I mean, it's a fucking google, they can afford buying slack;)19:30
kfox1111or google could just buy slack. ;)19:30
kfox1111yeah19:30
sdakerhallisey we dont want  to surprise the kubernetes admins19:30
sdakeslack admins that is19:30
sdakerhallisey so wheres that at, been proposed, been shot down, waiting on funding, what?19:31
rhalliseywhich part sdake, getting more cache?19:31
sdakeno linking the channels with exiwsting capacity19:31
kfox1111sdake, even assuming we dont link, but do most of our traffic on slack, we'd still blow it out.19:31
rhalliseyI can ask sarah and see what she thinks19:31
sdakekfox1111 right19:32
rhalliseybut 2.8k a day isn't ideal19:32
sdakerhallisey please do so19:32
kfox1111they need more capacity or we can't work there. :/19:32
sdakekfox1111 rhallisey was kind enoug hto obtain us a channel19:32
sdakerhallisey so we can have a presence in the kubernetes community19:32
sdakekfox1111 rather ^^19:32
kfox1111sdake: yeah. I know. its great. but the cache is a big big problem. :/19:32
*** lrensing has joined #openstack-kolla19:33
sdakei'd like to hear it from sarah19:33
sdakeif she says no go, then plan b19:33
rhalliseyI'd say we're somewhere between 1 and 2 messages a minute19:33
rhalliseyso 2k a day19:33
rhalliseyidk19:33
rhalliseytotal guess19:33
rhalliseykk19:33
kfox1111and the main kubernetes channel gets a huge amount of traffic too.19:33
sdakebritthouser thanks for the consulting :)19:34
sdakethe kubernetes slack channel will have to deal with capacity at some point19:34
sdakebecause all CNCF projects are in the kubernetes group19:34
sdakerhallisey right or wrong?19:35
sdakeactually i'm just assuming I don't know :)19:35
rhalliseyyes19:35
rhalliseythey are19:35
rhalliseyafaik19:35
sdakeif she says no which is a perfectly reasonable response - lets come up with a plan b19:36
sdakeafter she says yes/no19:36
sdakeshe has already gone out of her way to help us out19:36
sdakeeven though we are not a CNCF project19:36
sdakerhallisey what i'm getting at  there is don't argue with her say thank you instead :)19:38
sdakein the meantime rhallisey you got a FAQ for slack19:39
rhalliseysdake, I don't have an faw for slack19:40
rhalliseyit was super easy to join19:41
*** fragatin_ has quit IRC19:41
*** fragatina has joined #openstack-kolla19:41
*** absubram has joined #openstack-kolla19:42
sdakerhallisey sweett got a link19:43
*** ppalacios has joined #openstack-kolla19:45
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture  https://review.openstack.org/39225719:54
rhallisey391 lines19:54
rhalliseynice19:54
*** msimonin has joined #openstack-kolla19:54
rhalliseykfox1111, ^19:54
kfox1111rhallisey: looking19:55
kfox1111changes look good. rereading in its entirety.19:55
*** newmember has quit IRC19:55
rhalliseywow merge conflicts galore19:56
sdakerhallisey right - that was the purpose ;)19:56
rhalliseyinc0, can you review the spec19:57
inc0yeah, will do19:57
rhalliseysbezverk, srwilkers + everyone19:57
inc0do we mention that we are going to implement dependency init container too?19:57
rhalliseyya19:57
inc0I just noted definition in there19:57
sdakeyup thats in there19:57
rhalliseyit's a layer19:57
rhalliseylike operators are a layer19:57
inc0rhallisey, just to be clear, you are aware that dependency init container won't require operators at all?19:58
inc0for deployment at least19:58
rhalliseythat's in there19:58
sdakeits optional19:58
*** msimonin has quit IRC19:58
*** msimonin1 has joined #openstack-kolla19:58
rhalliseyread layer section19:58
sdakei bitched about that too ;)19:58
sdakeits kind of hidden away19:58
rhalliseylayers are optional19:58
rhalliseyand any layer can be used for deployment19:58
sdakekfox1111 pointed it out to me19:58
rhalliseyor any other operator shit19:59
srwilkerslooking now19:59
srwilkerssorry19:59
rhalliseysrwilkers, np19:59
rhalliseyjust want to make sure we are good to merge19:59
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible: Squashed commit of the following:  https://review.openstack.org/39856319:59
inc0so I don't expect helm to be optional per se19:59
sdakeinc0 i had the same question19:59
rhalliseyyou could just use the templates19:59
inc0as I honestly doubt we'll ever have an alternative there19:59
rhalliseyand input values20:00
rhalliseysay you were another project20:00
sdakeya that was the answer ^^ :)20:00
rhalliseyand wanted to do that20:00
inc0but truth is, output of helm is tons of yaml20:00
inc0you can write this yaml by hand if you want20:00
rhalliseyit is, but any tool can do that20:00
rhalliseyany tool can interpret the yaml20:00
inc0yeah looks good20:00
kfox1111inc0: it is in one way.20:00
kfox1111helm is about packaging. so a chart and a bunch of templates.20:01
kfox1111the templates are just a bounch of k8s objects with golang template vars and stuff.20:01
inc0fyi, we will have fairly strong dev team doing init-container mechanism + helm20:01
sdakepersonally I htink that spec is a work of art20:01
sdakebut what do I know ;)20:01
rhalliseylol20:01
rhalliseyfram it20:01
inc0I expect this to be a thing pretty soon20:01
kfox1111so if someone really wanted todo the templateing themselves without helm, they could source the templates but not the charts.20:01
rhalliseyframe it20:01
sdakeit would be a long frame ;)20:02
*** rmart04 has left #openstack-kolla20:02
sbezverkkfox1111: if templates are stored in format only helm atm understand would it not make it mandatory? since the only other alternative is to render them in VI ;-)20:02
inc0sdake, https://review.openstack.org/#/c/398534/120:02
inc0sdake, https://review.openstack.org/#/c/398534/420:02
kfox1111rhallisey: I think thie spec is good enough. reading it through cover to cover, its a bit hurky jerky, but its a big complicated thing, so that is probably unavoidable.20:02
rhalliseysbezverk, you could render them with your own tool20:02
rhalliseykfox1111, ya some of the sections have some overlap20:03
sdakeinc0 right forgot that part20:03
sdakethat was 3420:03
sdake#420:03
kfox1111sbezverk: the templates are standard golang templates. nothing really helm specific about it.20:03
sdakesorry about that20:03
rhalliseyand the ordering isn't perfect due to that20:03
inc0np, got that figured out20:03
kfox1111for example, hyou could render them with a golang based webserver pretty easily I think.20:03
srwilkersthis looks great20:03
sbezverkkfox1111: ok so then any other golang compatible tools would be supported. I was not aware of them but I trust you :-)20:04
*** ppalacios has quit IRC20:04
*** ppalacios has joined #openstack-kolla20:05
*** Pavo has quit IRC20:07
inc0https://techcrunch.com/2016/11/16/microsoft-joins-the-linux-foundation/20:10
inc02016 will be remember as strange year in human history20:11
srwilkersyeah, i saw that20:11
jascott1trump, M$, whats next?20:12
*** Pavo has joined #openstack-kolla20:12
inc0brexit20:12
*** sean-k-mooney has quit IRC20:12
inc0also this year20:12
srwilkerscubs won the world series, check that one off20:12
kfox1111yeah. and google joined microsoft with .net20:12
jascott1good addition!20:12
kfox1111wierd time. :)20:12
inc0Leonardo di Caprio won Oscar20:12
jascott1haha20:12
*** sean-k-mooney has joined #openstack-kolla20:13
kfox1111https://developers.slashdot.org/story/16/11/16/1642201/google-joins-microsofts-net-foundation20:13
kfox1111feels kind of like april fools day...20:13
*** eaguilar_ has quit IRC20:13
inc0more like april fools year20:13
jascott1MBP now effectively suck20:14
kfox1111yeah.20:14
inc0you mean with this touchbar thingy?20:14
inc0I actually think it's neat20:14
srwilkersD:20:15
jascott1ssd's soldered in20:15
inc0just need to write VIM bindings to change modes with it;)20:15
jascott1http://www.macrumors.com/2016/11/15/macbook-pro-touch-bar-non-removable-ssd/20:16
inc0not that I'll ever buy it20:16
*** ppalacios has quit IRC20:23
*** pomac has quit IRC20:28
sdakejascott1 one with or without an escape key?20:34
sdakewho on earth would take an escape key off a keyboard aimed at developers20:34
sdakeoh, I know, APPLE20:34
*** sdake has quit IRC20:35
kfox1111hehe. annd apple will still make money20:35
*** sdake has joined #openstack-kolla20:37
openstackgerritMerged openstack/kolla: Move manila-share service to network node  https://review.openstack.org/39662520:37
kfox1111a little dated, but https://www.youtube.com/watch?v=FL7yD-0pqZg20:37
sdakeit can grant out the 3 wishes :)20:39
kfox1111including getting an iphone. :)20:40
inc0that being said20:41
inc0android recently wiped out my phone randomly20:41
inc0in BCN20:41
kfox1111heh. yeha. way off track... :)20:41
*** pomac has joined #openstack-kolla20:41
*** fguillot has quit IRC20:41
kfox1111all devices fail enventually. :/20:41
sdakei like hte print money thing20:42
sdakepretty epic kfox1111 :)20:42
kfox1111apple's greatest work is the loyaltee their brand has.20:45
kfox1111its truely impresive.20:45
sdakei got a mbp because it was the best hardware avilaable at the time20:46
sdakeno longer the case20:46
*** unicell has quit IRC20:46
sdake(the alternative was a thinkpad with less memory less disk less resolution)20:46
sdakenow my #1 consideration is "does it have an escape key"20:46
sdakei dont even know how i use vi now, imagine if i had to hotkey escape to somethign else ;)20:47
kfox1111yeah. my laptop 2 laptops ago was a macbook because of the same. it was better hardware for the cost. I put linux on it though.20:47
*** matrohon has quit IRC20:47
kfox1111escape key is pretty critical.20:47
sdakeunfortunately to communicate with my peeps at cisco i need either windows or macos20:48
sdakewindows daily would be a torture test20:48
srwilkerscan confirm, its shit20:48
sdakecan confirm which srwilkers ?20:48
srwilkerstook me ~18 months to get something other than a windows machine20:48
srwilkersthat windows daily is a torture test20:48
sdakeoh right20:49
srwilkerswell, also vim would suck ass without an escape key.  my mind would explode20:49
sdakei use windows for 1 thing20:49
sdakeplaying diablo 320:49
srwilkersyes20:49
sdakebecause my mac is too underpowered to run it20:49
srwilkersamped for necro and original diablo act addition20:49
sdakewhen is that coming out srwilkers20:49
sdakeseason 6 i was top 100  barb ;)20:49
srwilkersjanuary20:50
sdakehaven't played since20:50
kfox1111I make our windows team deal with my compatability issues. :)20:50
sdakewell have played a bit20:50
sdakebut not much20:50
kfox1111they provide a windows terminal server, and I use it.20:50
kfox1111and report every breakage I see. :)20:50
kfox1111fortunatly these days I don't have to use windows for much.20:51
kfox1111mostly the wird encription stuff they like to use.20:51
bjoloescape key? in vim i use ctrl+c20:53
*** unicell has joined #openstack-kolla20:53
sdakehuh20:53
bjoloremapped caps lock to control20:53
bjoloyes20:53
inc0sdake, wintendo is the word20:53
bjoloctrl-c is the same as escape20:53
*** newmember has joined #openstack-kolla20:53
bjoloim  a diehard linux vim guy and i never use escape key20:54
sdakei'm not sure whats worse ;)20:54
bjoloactually thats not all true. i use escape + ) to break out of hp ilo textcons, but thats it20:55
bjoloescape is too slow compared to ctrl+c20:56
bjolo:)20:56
*** krtaylor has quit IRC20:58
openstackgerritMerged openstack/kolla-ansible: Make service_checks easier to run independently of kolla-ansible  https://review.openstack.org/39832720:59
bjolotime to go to bed for me21:02
bjolocya all tomorrow21:02
bjolo~21:03
srwilkerscya bjolo21:06
*** huhaoran has quit IRC21:07
*** Serlex has quit IRC21:18
*** lrensing has quit IRC21:22
*** Serlex has joined #openstack-kolla21:29
*** jtriley_ has joined #openstack-kolla21:30
DaveTurnerHi sdake21:30
*** gfidente has quit IRC21:31
*** Serlex has quit IRC21:32
*** jtriley has quit IRC21:32
*** krtaylor has joined #openstack-kolla21:33
*** Jeffrey4l has quit IRC21:33
*** inc0 has quit IRC21:33
*** jtriley_ has quit IRC21:34
DaveTurnersdake Based on our discussions last night, we've worked around the rabbitmq / api_interface issue by changing the primary hostname to that of the api_interface.21:35
*** krtaylor has quit IRC21:37
DaveTurnersdake Just to clarify our approach going forward, do see the possibility of updating the rabbitmq role to use IP from network_interface instead of api_interface - or is the assumption driving the existing approach - that the primary hostname would be tied to the api_interface - going to remain?21:39
DaveTurnerdo *you see the possibility...21:39
kfox1111my two cents is that should be overridable.21:39
kfox1111I want to be able to specify rabbit per service and per oslo type.21:39
rhalliseykfox1111, we're good to merge the spec if you want to +a itr21:40
rhalliseyit21:40
kfox1111so one rabbit for nova rpc, but nova notifications go to ceilometer rabbit.21:40
kfox1111rhallisey: I'm good with it. the one outstanding question is, how long should we wait for others to weigh in?21:41
kfox1111Adrian or Michael for example.21:41
DaveTurnerkfox1111 - that level of flexibility would be optimal.  We are looking at customizing the rabbit role for our purposes, but would like to understand where the community stands with this.21:41
kfox1111DaveTurner: I'm working on kolla-kubernetes, not kolla-ansible so I can't sepeak to that use case,21:42
rhalliseykfox1111, I said on the ML we would close today21:42
kfox1111but my own clouds need that level of flexability.21:42
kfox1111rhallisey: any particular time today?21:42
rhalliseycan't wait forever and we have majority of kolla-kube cores21:42
rhalliseyno particular time21:42
rhalliseyEOD21:42
kfox1111pst cob maybe?21:42
rhalliseysure21:42
kfox1111k21:42
*** sdake_ has joined #openstack-kolla21:44
*** krtaylor has joined #openstack-kolla21:45
*** sdake has quit IRC21:47
*** krtaylor has quit IRC21:47
*** krtaylor has joined #openstack-kolla21:50
*** krtaylor has quit IRC21:53
*** ayoung has quit IRC21:55
*** pomac has quit IRC21:55
*** krtaylor has joined #openstack-kolla21:56
*** tonanhngo_ has joined #openstack-kolla21:58
*** krtaylor has quit IRC21:59
*** krtaylor has joined #openstack-kolla22:00
*** tonanhngo has quit IRC22:01
*** srwilkers has quit IRC22:02
*** tonanhngo_ has quit IRC22:03
*** Pavo has quit IRC22:07
sbezverkkfox1111: that helm doc link you gave me is pretty good. well written and easy to understand :-)22:08
*** adrian_otto has quit IRC22:08
*** pomac has joined #openstack-kolla22:12
*** TxGirlGeek has quit IRC22:12
*** Pavo has joined #openstack-kolla22:12
*** TxGirlGeek has joined #openstack-kolla22:15
*** rhallisey has quit IRC22:17
sdake_sbezverk link plz22:18
*** TxGirlGeek has quit IRC22:22
*** chas has joined #openstack-kolla22:23
*** TxGirlGeek has joined #openstack-kolla22:23
sbezverksdake: here you go: https://github.com/kubernetes/helm/tree/master/22:24
sbezverkcheck the doc portion..22:24
*** TxGirlGeek has quit IRC22:25
*** TxGirlGeek has joined #openstack-kolla22:25
*** tonanhngo has joined #openstack-kolla22:26
*** absubram has quit IRC22:26
*** chas has quit IRC22:27
*** TxGirlGeek has quit IRC22:29
*** tonanhngo has quit IRC22:30
*** TxGirlGeek has joined #openstack-kolla22:30
*** chas has joined #openstack-kolla22:31
*** TxGirlGeek has quit IRC22:31
*** TxGirlGeek has joined #openstack-kolla22:32
*** chas has quit IRC22:33
*** chas has joined #openstack-kolla22:34
kfox1111sdake_: yeah. pretty surprising for something not even out of alpha yet. :)22:35
kfox1111sbezverk: yeah. pretty surprising for something not even out of alpha yet. :)22:35
sdake_huh?22:35
kfox1111sdake_: sorry. bad tab.22:35
sdake_roger22:35
*** adrian_otto has joined #openstack-kolla22:35
*** absubram has joined #openstack-kolla22:37
*** schwicht has quit IRC22:37
*** TxGirlGeek has quit IRC22:38
*** TxGirlGeek has joined #openstack-kolla22:38
sdake_britthouser22:42
sdake_i'm in need of your help22:42
sdake_britthouser you happen to be around?22:42
sbezverkkfox1111: hm, I found rc2 version22:43
*** alteriks has quit IRC22:43
*** ayoung has joined #openstack-kolla22:45
kfox1111beta2 I guess is what I saw.22:46
kfox1111https://kubernetes-helm.storage.googleapis.com/helm-v2.0.0-beta.2-linux-amd64.tar.gz22:46
kfox1111not alpha.22:46
kfox1111I think they were talking about having 4.22:46
sbezverkhelm-v2.0.0-rc.2-linux-amd64.zip22:46
kfox1111hmm... where did you find that?22:47
kfox1111oh. off the release page.22:47
*** srwilkers has joined #openstack-kolla22:47
sbezverkhttp://storage.googleapis.com/kubernetes-helm/helm-v2.0.0-rc.2-linux-amd64.tar.gz22:47
kfox1111hmm... their main readme's out of date.22:48
kfox1111not surprising I guess.22:48
kfox1111so helm's final release is even closer then I thought. :)22:50
Pavoevening22:50
*** schwicht has joined #openstack-kolla22:53
srwilkershey Pavo22:59
Pavoevening srwilkers23:01
*** jheroux has quit IRC23:04
*** sdake has joined #openstack-kolla23:06
*** dave-mccowan has quit IRC23:07
*** krtaylor has quit IRC23:09
*** sdake_ has quit IRC23:10
Pavoquestion, in deploying newton with kolla-ansible is the default deployment only support QEMU images or can you also use Docker images also at the same time?23:13
*** sdake has quit IRC23:16
Pavoor is that a dumb question?23:16
*** eaguilar has joined #openstack-kolla23:21
*** diogogmt has quit IRC23:25
*** schwicht has quit IRC23:29
*** schwicht has joined #openstack-kolla23:30
*** Jeffrey4l has joined #openstack-kolla23:31
*** lamt has quit IRC23:33
*** schwicht has quit IRC23:34
*** f13o_ has quit IRC23:37
*** zhugaoxiao has joined #openstack-kolla23:38
*** khamtamtun has joined #openstack-kolla23:39
*** Administrator_ has quit IRC23:40
*** sdake has joined #openstack-kolla23:40
sdakepavo no23:41
sdakepavo you could theoretically use the docker hypervisor23:41
*** inc0 has joined #openstack-kolla23:41
sdakepavo another optoin is to use a COE such as Magnum provides like kubernetes23:42
inc0I'm back my dear kollagues23:42
sdakesup inc023:42
*** khamtamtun has quit IRC23:42
inc0I went to our favorite mexican restaurant, and this is contentous space in souther texas23:42
inc0so yeah, I'm good23:42
*** david-lyle_ has joined #openstack-kolla23:50
*** david-lyle has quit IRC23:51
srwilkerslove the mexican food in texas23:53
srwilkersmy brother lives in austin, so i visit about 3 times a year, and my favorite part is the food23:53
*** TxGirlGeek has quit IRC23:55
jascott1srwilkers: had a couple of tasty steaks in San Antonio (inc0's neck of the woods) and they had both kinds of beer "Dos Equis" and "Shiner Bock"23:59
jascott1of course I am spoiled on the beer being in PDX23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!