| *** sdake has joined #openstack-kolla | 00:03 | |
| *** chas_ has quit IRC | 00:03 | |
| *** tonanhngo has quit IRC | 00:10 | |
| *** absubram has quit IRC | 00:13 | |
| *** portdirect_ is now known as portdirect_away_ | 00:16 | |
| *** TxGirlGeek has quit IRC | 00:18 | |
| *** tonanhngo has joined #openstack-kolla | 00:20 | |
| *** hfu has joined #openstack-kolla | 00:20 | |
| *** rhallisey has joined #openstack-kolla | 00:22 | |
| *** tonanhngo has quit IRC | 00:25 | |
| *** lamt has quit IRC | 00:25 | |
| rhallisey | hey | 00:27 |
|---|---|---|
| rhallisey | what's around got some good news | 00:27 |
| kfox1111 | yeah? | 00:27 |
| rhallisey | s/what's/who's/ | 00:27 |
| kfox1111 | could use some. :) | 00:27 |
| rhallisey | we have a slack channel now in kubernetes | 00:27 |
| rhallisey | :) | 00:27 |
| kfox1111 | cool. | 00:28 |
| kfox1111 | hook me up. :) | 00:28 |
| kfox1111 | I use slack alot for work now. | 00:28 |
| *** hfu has quit IRC | 00:28 | |
| *** TxGirlGeek has joined #openstack-kolla | 00:29 | |
| kfox1111 | #kolla | 00:29 |
| kfox1111 | foudn it. :) | 00:29 |
| rhallisey | can you get in it? | 00:29 |
| rhallisey | sweet | 00:29 |
| kfox1111 | yup. | 00:29 |
| rhallisey | I think we can mirror it | 00:29 |
| *** hfu has joined #openstack-kolla | 00:29 | |
| kfox1111 | cool. | 00:29 |
| *** lamt has joined #openstack-kolla | 00:30 | |
| kfox1111 | try someting. ping me again by name please. | 00:30 |
| rhallisey | with this channel | 00:30 |
| kfox1111 | k. cool. thanks. :) | 00:30 |
| rhallisey | nice you get the notification | 00:31 |
| rhallisey | even when not around hehe :) | 00:31 |
| kfox1111 | with great power.... ;) | 00:31 |
| kfox1111 | but, yeah. :) | 00:31 |
| Pavo | slack is awesome | 00:32 |
| rhallisey | Pavo, we got a channel now :) | 00:32 |
| Pavo | may I join? | 00:32 |
| kfox1111 | Pavo: looks like its an open channel. | 00:32 |
| Pavo | link? | 00:32 |
| kfox1111 | I was able to join without invite. | 00:32 |
| kfox1111 | kubernetes.slack.com | 00:33 |
| kfox1111 | #kolla | 00:33 |
| *** newmember has quit IRC | 00:33 | |
| *** TxGirlGeek has quit IRC | 00:33 | |
| *** TxGirlGeek has joined #openstack-kolla | 00:34 | |
| *** yingjun has joined #openstack-kolla | 00:34 | |
| *** portdirect_away_ has quit IRC | 00:34 | |
| *** portdirect_away_ has joined #openstack-kolla | 00:35 | |
| *** TxGirlGeek has quit IRC | 00:35 | |
| Pavo | hmm how do I sign up | 00:37 |
| jascott1 | goto kubernetes.slack.com iirc | 00:37 |
| jascott1 | oops | 00:38 |
| jascott1 | here it is Pavo http://slack.k8s.io/ | 00:38 |
| Pavo | yeah i did and used a username and password that I use on another team | 00:38 |
| *** yingjun has quit IRC | 00:38 | |
| Pavo | thx | 00:38 |
| *** yingjun has joined #openstack-kolla | 00:39 | |
| *** YuYangWang has joined #openstack-kolla | 00:46 | |
| rhallisey | sdake, yo | 00:49 |
| *** jtriley has joined #openstack-kolla | 00:59 | |
| sdake | who daresh ringith me whilth i sleepith | 01:00 |
| sdake | sup rhallisey | 01:00 |
| rhallisey | sdake, got slack channel | 01:00 |
| rhallisey | going to get it mirrored to irc | 01:01 |
| sdake | nice job rhallisey | 01:02 |
| sdake | see it didn't even take me to do anything ;) | 01:02 |
| *** duonghq has joined #openstack-kolla | 01:02 | |
| duonghq | morning guys | 01:02 |
| sdake | the mirroring how will that work | 01:02 |
| sdake | btw rhallisey | 01:02 |
| sdake | i think you are missing a potential contributor | 01:02 |
| rhallisey | https://sameroom.io/blog/connecting-a-channel-in-slack-to-a-channel-irc/ | 01:02 |
| sdake | around duong's name | 01:02 |
| duonghq | just seen new kolla-ansible repo, great things | 01:02 |
| duonghq | did I miss something? | 01:03 |
| rhallisey | kk | 01:03 |
| kfox1111 | this requires all users to setup the binding? | 01:03 |
| rhallisey | I'm not sure | 01:05 |
| v1k0d3n | oh rhallisey you got a kolla channel set up in kubernetes? | 01:05 |
| rhallisey | ya dude | 01:05 |
| v1k0d3n | wow dude...good job! | 01:05 |
| v1k0d3n | that's great | 01:05 |
| rhallisey | and we can mirror it here | 01:05 |
| v1k0d3n | who did you you have to work with on that? | 01:06 |
| rhallisey | sarah | 01:06 |
| v1k0d3n | thought so. | 01:06 |
| v1k0d3n | she rocks | 01:06 |
| *** jtriley has quit IRC | 01:06 | |
| rhallisey | ya she's been very helpful | 01:06 |
| portdirect_away_ | +1 | 01:06 |
| *** zhubingbing has joined #openstack-kolla | 01:08 | |
| sdake | i'm glad you didn't need a face to face hand for that at cncf rhallisey | 01:08 |
| *** portdirect_away_ is now known as portdirect_ | 01:08 | |
| sdake | she was literally busy 24/7 | 01:08 |
| duonghq | I as understand, we have a channel in slack k8s? | 01:08 |
| rhallisey | sdake, ya it worked out | 01:08 |
| rhallisey | duonghq, that's right :) | 01:09 |
| sdake | duonghq right - idea is to create a footprint over there for first deployment help as we do now | 01:09 |
| sdake | rhallisey are there limitiations on what else we may use the channel for? | 01:09 |
| rhallisey | no, but we need to keep in mind the opernstack-kubernetes sig | 01:09 |
| portdirect_ | v1k0d3n: just saw you added suport for centos in halcyon - awesome, I could add atomic tomorrow if your up for a pr? | 01:09 |
| sdake | right - I did talk to sgordon about that | 01:10 |
| sdake | he seemed not to mind | 01:10 |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Allow for specifying a KOLLA_CONFIG_FILE https://review.openstack.org/398008 | 01:10 |
| sdake | i'm not sure what i agreed to if anything :) | 01:10 |
| Pavo | kfox1111 does this setup script run on the target nodes or the deployer node? | 01:10 |
| rhallisey | since this channel will help us figure out issues/features for openstakc on kube | 01:10 |
| rhallisey | we just want to keep it in mind | 01:11 |
| sdake | i think i basically said we weren't going to tromping on his territory | 01:11 |
| kfox1111 | Pavo: the primary. | 01:11 |
| Pavo | primary=deployer? | 01:11 |
| sdake | by keeping things kolla focused | 01:11 |
| rhallisey | yup | 01:11 |
| kfox1111 | not that there is much distinction in the gate currently. it just gives you 2 nodes, then runs your script on one. | 01:11 |
| kfox1111 | the script then scp/ssh runs a few things on the other node. | 01:11 |
| kfox1111 | when multinode is enabled. | 01:11 |
| kfox1111 | making it a compute node. | 01:11 |
| *** tonanhngo has joined #openstack-kolla | 01:12 | |
| Pavo | ah ok so primary=controller and others=computes | 01:12 |
| kfox1111 | for the instructions though, not sure that part should be documented? should assume a working k8s is already in existance? | 01:12 |
| *** tonanhngo has quit IRC | 01:12 | |
| kfox1111 | (or have a seperate document for deploying a multinode k8s?) | 01:13 |
| Pavo | would love to have the docs also include how to setup k8 and get it ready for using kolla-k8 | 01:13 |
| Pavo | from scrath to end | 01:13 |
| Pavo | or from scratch to fully operational | 01:13 |
| kfox1111 | yeah. just keep those seperate I think as there's lots of ways to do k8s deployments. | 01:14 |
| kfox1111 | and they seem to change all the time. :/ | 01:14 |
| portdirect_ | I think we should prob have a recommended deployment in the docs, but the setup should be out of scope - to make less burdon for us | 01:14 |
| kfox1111 | kubeadm is the new hottness. | 01:14 |
| kfox1111 | but doesn't do ha yet. | 01:14 |
| Pavo | so you are saying I have to setup k8 first before even start using this script | 01:14 |
| *** tovin07_ has joined #openstack-kolla | 01:14 | |
| kfox1111 | so I use it in the gate, but not for any of my production systems. | 01:14 |
| kfox1111 | Pavo: no, that script actually uses kubeadmin to deploy a kubernetes system. | 01:15 |
| Pavo | oh ok | 01:15 |
| kfox1111 | just saying, for the mulitnode deployment docs, the doc should describe how to use an existing k8s cluster. | 01:15 |
| kfox1111 | and a seperate doc maybe for deploying a k8s for those that want help there. | 01:15 |
| duonghq | I tried kubeadm, it's fine for testing but not production | 01:15 |
| kfox1111 | so if you already got a favorite k8s deployment tool, you can skip the k8s doc. | 01:16 |
| duonghq | no long-running test, also | 01:16 |
| portdirect_ | very few tools are production cabable atm :( | 01:16 |
| kfox1111 | long term, I think kubeadm is going to be a very nice tool. but gots a ways to go though. | 01:16 |
| duonghq | pretty sure, doubt that many people craft so-call in-house tools | 01:17 |
| sdake | i had heard they were working on a rewrite of deployment in ansible | 01:17 |
| kfox1111 | heh. the wheel turns. :) | 01:17 |
| portdirect_ | yep - i've been playing with https://github.com/att-comdev/halcyon-vagrant-kubernetes for dev stuff over the last few days - and it's pretty nice for setting up a quick testing env in vagrant. | 01:17 |
| portdirect_ | (that yep made no sense with the posts between.. ) | 01:18 |
| duonghq | kfox1111, kubeadm is very nice thing but still lacks of recovery ability from master crash | 01:18 |
| kfox1111 | right. | 01:18 |
| kfox1111 | it'll get there though. pretty sure. | 01:18 |
| kfox1111 | its really easy to get a gate job workign with it though. :) | 01:19 |
| duonghq | sure, it's suitable for short-term testing like gate | 01:19 |
| duonghq | but do you notice that kubeadm master eat much IOPS? | 01:20 |
| duonghq | I tested on 3 nodes on kvm | 01:20 |
| Pavo | not gonna lie though that stackanetes looks pretty sweet | 01:20 |
| sdake | pavo try it out let us know what you think :) | 01:21 |
| portdirect_ | pavo: id recommend looking at https://github.com/portdirect/harbor as well, but then I would say that :) | 01:22 |
| Pavo | sdake I would if there was a guide for CentOS 7 | 01:22 |
| Pavo | stackanetes uses CoreOS from my undersanding | 01:23 |
| sdake | portdirect_ didn't know you were so far along | 01:23 |
| Pavo | portdirect_ atleast yours is written fro CentOS | 01:24 |
| kfox1111 | sdake: yup. portdirect_ has a lot of good stuff in there. :) | 01:24 |
| portdirect_ | yeah - its very crude atm, but I've not had time to do much since barca | 01:25 |
| kfox1111 | if we can get everyone on the same page and working on the same project, there will be some really amazing things pop out. :) | 01:25 |
| Pavo | I agree | 01:26 |
| Pavo | and I have a whole shop that can test and give feed back daily | 01:26 |
| kfox1111 | nice. :) | 01:26 |
| Pavo | thats our job mainly | 01:26 |
| Pavo | Networking, Engineering, Research & Development | 01:27 |
| Pavo | any guess what are shop is called? | 01:27 |
| Pavo | :) | 01:27 |
| kfox1111 | a few. :) | 01:27 |
| Pavo | NERD | 01:28 |
| Pavo | lol | 01:28 |
| kfox1111 | :) | 01:28 |
| rhallisey | well idk if the mirroring will work :) | 01:30 |
| *** zhurong has joined #openstack-kolla | 01:30 | |
| rhallisey | maybe we could mirror slack to irc | 01:30 |
| rhallisey | eh | 01:31 |
| rhallisey | not ideal | 01:31 |
| rhallisey | one way | 01:31 |
| kfox1111 | yeah. I wondered if the mirroring would work. :/ | 01:32 |
| rhallisey | I was really hoping for it | 01:32 |
| rhallisey | I think they will fix it though | 01:33 |
| rhallisey | if they go through 10 in 2 days | 01:33 |
| kfox1111 | yeah | 01:33 |
| rhallisey | slack is different that irc | 01:33 |
| rhallisey | there isn't a live chat here | 01:33 |
| rhallisey | when you aren't here you miss it | 01:33 |
| rhallisey | need to read logs | 01:33 |
| Pavo | unless you use a bouncer like me | 01:34 |
| kfox1111 | or leave it in screen. :) | 01:34 |
| Pavo | but only buff 1000 lines | 01:34 |
| rhallisey | indeed | 01:34 |
| kfox1111 | yeah. | 01:34 |
| rhallisey | I'm lame | 01:34 |
| kfox1111 | 6 of one. | 01:34 |
| rhallisey | I just read logs | 01:34 |
| kfox1111 | either way I think we need logs anyway. | 01:34 |
| kfox1111 | cause we want history more then a week back at times. | 01:35 |
| Pavo | yeha | 01:35 |
| Pavo | yeah | 01:35 |
| rhallisey | kfox1111, we'll have the history | 01:35 |
| rhallisey | it will all log on this side | 01:35 |
| rhallisey | it's the live chat | 01:35 |
| kfox1111 | what live chat are you refering to. | 01:36 |
| rhallisey | kube slack can only carry 10k live messages at a time | 01:36 |
| rhallisey | so say one channel was spamming 10k messages/second | 01:36 |
| sdake | rhallisey this should give you flashbacks: https://github.com/portdirect/harbor/blob/latest/tools/build-docker-image#L135 | 01:36 |
| rhallisey | anything you right will not appear in the scrollback | 01:36 |
| kfox1111 | heh. | 01:37 |
| rhallisey | sdake, lul | 01:37 |
| kfox1111 | are they that close to the limit? | 01:37 |
| rhallisey | crudini though | 01:37 |
| rhallisey | kfox1111, don't think so | 01:37 |
| rhallisey | let me see what we get in a day | 01:37 |
| rhallisey | idk how I can count this.. | 01:38 |
| *** Pavo has quit IRC | 01:38 | |
| kfox1111 | gotta head out. | 01:38 |
| rhallisey | sdake, reminds me of when we had crudini and sef everywhere | 01:38 |
| kfox1111 | l8r. | 01:38 |
| *** Pavo has joined #openstack-kolla | 01:38 | |
| rhallisey | kfox1111, so you think :). Now we have you in slack! | 01:38 |
| kfox1111 | ;) | 01:38 |
| kfox1111 | with great power... ;) | 01:38 |
| rhallisey | ya | 01:38 |
| rhallisey | dont mirror and have a bot destory the cache | 01:39 |
| kfox1111 | (and a two way street. ;) | 01:39 |
| rhallisey | :) | 01:39 |
| rhallisey | hehe | 01:39 |
| kfox1111 | but, yeah. ping me if you need me. | 01:39 |
| rhallisey | I'm kidding hehe | 01:39 |
| kfox1111 | I just reserve the right not to respond right away. :) | 01:39 |
| *** schwicht has joined #openstack-kolla | 01:40 | |
| *** hfu has quit IRC | 01:40 | |
| rhallisey | 2 messages a minute is 2,880 messages per day | 01:41 |
| *** dt-turner has joined #openstack-kolla | 01:42 | |
| portdirect_ | rhallisy/sdake: even more flashback here :) https://github.com/portdirect/harbor/blob/latest/docker/common-assets/opt/harbor/harbor-common.sh#L5 | 01:42 |
| rhallisey | lol | 01:42 |
| rhallisey | that is a major throw back | 01:43 |
| rhallisey | kolla had the containers and I was looking at deploying all different ways | 01:44 |
| *** dt-turner has quit IRC | 01:44 | |
| portdirect_ | dont think I actually use any of it anymore - but never got round to removing it :) | 01:44 |
| rhallisey | that's pretty cool :) | 01:45 |
| *** DaveTurner has joined #openstack-kolla | 01:45 | |
| sdake | ya that shell code stuff was madness | 01:45 |
| sdake | but eh, like i said earlier zero exposur eto docker when the project was started | 01:45 |
| sdake | objective was to teach a bunch of peeps how to do deploy on openstack | 01:46 |
| rhallisey | my guess is we're about 2 messages per minute | 01:46 |
| rhallisey | in this channel | 01:46 |
| rhallisey | including people signing in an out | 01:47 |
| *** hfu has joined #openstack-kolla | 01:50 | |
| *** lamt has quit IRC | 01:57 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Move the "enable_destroy_images" into configure file https://review.openstack.org/397633 | 01:58 |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla: Modify the letter case in post-install.yml file https://review.openstack.org/398024 | 01:59 |
| *** rhallisey has quit IRC | 02:01 | |
| *** Pavo has quit IRC | 02:01 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove Ansible from Kolla https://review.openstack.org/398025 | 02:02 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Clean up tox.ini for Ansible removal https://review.openstack.org/398028 | 02:06 |
| *** huikang has joined #openstack-kolla | 02:08 | |
| *** Pavo has joined #openstack-kolla | 02:08 | |
| *** schwicht has quit IRC | 02:08 | |
| Pavo | finally | 02:08 |
| Pavo | got my stack finally port forwarding working | 02:09 |
| Pavo | ddi.hopto.org | 02:09 |
| portdirect_ | Pavo: what are you plaiing on using OpenStack for (if you can say)? | 02:09 |
| *** yingjun has quit IRC | 02:10 | |
| *** schwicht has joined #openstack-kolla | 02:10 | |
| Pavo | well we use it as a infrastructure mostly | 02:10 |
| *** yingjun_ has joined #openstack-kolla | 02:10 | |
| Pavo | to do defensive cyber operations from | 02:11 |
| *** schwicht has quit IRC | 02:12 | |
| *** yingjun_ has quit IRC | 02:12 | |
| *** yingjun has joined #openstack-kolla | 02:12 | |
| Pavo | kfox1111 what about this guide | 02:13 |
| Pavo | http://severalnines.com/blog/installing-kubernetes-cluster-minions-centos7-manage-pods-services | 02:13 |
| Pavo | think its good enough to start with? | 02:13 |
| portdirect_ | nice, be interesting to get your perspective's/requirements from a security perspective as the develop - I'm quite interested in getting a PCI-DSS comiliant setup out of the box, but I assume your working to FedRAMP or similar? | 02:15 |
| *** harbor has joined #openstack-kolla | 02:16 | |
| Pavo | well that would be perfect | 02:16 |
| Pavo | first look into FIPS-140 | 02:17 |
| duonghq | I think we should sum up k8s deployment tool and put it in some wiki or notes, | 02:17 |
| duonghq | tool and guide | 02:17 |
| harbor | yeah been there :) | 02:17 |
| duonghq | due to I had hard time to deploy k8s | 02:17 |
| duonghq | I think many people also suffer | 02:17 |
| Pavo | and then look into http://iase.disa.mil/stigs/os/unix-linux/Pages/index.aspx | 02:17 |
| Pavo | duonghq +1 | 02:18 |
| *** yingjun has quit IRC | 02:18 | |
| Pavo | actually | 02:18 |
| Pavo | duonghq +100000000 | 02:18 |
| *** yingjun has joined #openstack-kolla | 02:18 | |
| *** yingjun has quit IRC | 02:18 | |
| *** yingjun has joined #openstack-kolla | 02:18 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Update the default value for Heat https://review.openstack.org/397577 | 02:19 |
| *** coolsvap has joined #openstack-kolla | 02:19 | |
| harbor | ok are you guys using vagrant or similar? | 02:19 |
| Pavo | you asking me harbor ? | 02:20 |
| *** harbor is now known as portdirect__ | 02:20 | |
| *** portdirect_ has quit IRC | 02:20 | |
| portdirect__ | whoops, I'm back, dodgy internet here | 02:20 |
| portdirect__ | yeah that was me asking :) | 02:20 |
| Pavo | not at all | 02:20 |
| Pavo | Currently using CentOS 7 with packstack | 02:21 |
| Pavo | and deploying with a custom script | 02:21 |
| portdirect__ | is your packstack running heat? | 02:21 |
| Pavo | but as well all know packstack is limited to 5 nodes | 02:21 |
| Pavo | and yes we are using heat | 02:22 |
| Pavo | which is are main focus right now | 02:22 |
| Pavo | to build heat temps for different senarios based off mission | 02:22 |
| portdirect__ | Pavo: http://kubernetes.io/docs/getting-started-guides/openstack-heat/ | 02:23 |
| Pavo | is this a heat temp for k8 inside OS? | 02:23 |
| Pavo | looks like a guide to install Openstack | 02:24 |
| portdirect__ | yup | 02:24 |
| *** jtriley has joined #openstack-kolla | 02:25 | |
| portdirect__ | it's the official k8s docs for running kube in OS via heat. | 02:25 |
| Pavo | will take a look into it but I am mainly looking to switch from packstack to either kolla-ansible or kolla-k8 for our deployment method now | 02:25 |
| DaveTurner | Hey folks. Question. My team runs daily CICD builds / deployments of multinode Kolla clusters and we've recently updated our configuration to segregate API traffic. | 02:26 |
| DaveTurner | On the target nodes we have a management and api vlan configured with IP's plumbed to both. On the Kolla side, we've specified the api vlan interface for the "api_interface" and the management vlan interface as the "network_interface" in globals.yml. | 02:26 |
| Pavo | main aspects that we are looking at kolla for are ease of deployment, configuration and upgradability | 02:26 |
| DaveTurner | During deployment, all services seem to listen on / communicate over the api vlan, however rabbitmq fails to start successfully. The rabbit containers continue to restart on all nodes. | 02:26 |
| portdirect__ | Pavo: id advise kolla-ansible for now if you need production right away | 02:27 |
| DaveTurner | We are running about 2 weeks behind master. I noticed this bug: https://bugs.launchpad.net/kolla/+bug/1583655 which seems to touch on the issue, however we haven't been able to get around it via /etc/hosts entries. | 02:28 |
| openstack | Launchpad bug 1583655 in kolla "rabbitmq setup from secondary IP" [Critical,Confirmed] | 02:28 |
| *** hfu has quit IRC | 02:29 | |
| Pavo | well thats the beauty of our setups, they aren't meant to be persistent, ie production. we stand them up, do our mission, move required data needed and purge the systems | 02:29 |
| Pavo | rinse and repeat | 02:29 |
| portdirect__ | coming from a somewhat offenisive youth, that makes sense - but you need a rock solid base, I assume you are talking bare metal nodes | 02:30 |
| Pavo | yes and there is another infruasture that we ride on | 02:31 |
| *** hfu has joined #openstack-kolla | 02:32 | |
| sdake | hi DaveTurner | 02:35 |
| sdake | reading scrollback moment pls | 02:35 |
| sdake | DaveTurner couple things re running coule weeks behind master | 02:36 |
| sdake | first we have split the kolla repo into two | 02:36 |
| sdake | second, your running master openstack as well, not newton openstack, are you aware? | 02:36 |
| sdake | third on the repo split, we have not yet got the repositories back into working order, although its in progress | 02:37 |
| sdake | fourth so 1 and 3 will effect your ci/cd pipeline of master unless you also make some changes to handle it once your running on master as of today | 02:37 |
| DaveTurner | Hi sdake. Stepped away for a sec.. reading through your reply | 02:40 |
| sdake | it was more to bring you up to speed on the current state of things | 02:40 |
| sdake | not a direct answer to your question :) | 02:40 |
| DaveTurner | shucks. ;-) | 02:40 |
| sdake | for that i'll need your help | 02:40 |
| *** chas has joined #openstack-kolla | 02:40 | |
| sdake | on any of th econtroller nodes can you run docker logs rabbitmq | 02:41 |
| DaveTurner | We are aware that we are running master openstack as well. We are attempting to stay close to trunk. | 02:41 |
| DaveTurner | Sure.... | 02:42 |
| sdake | DaveTurner are you running kolla-ansible upgrade on each new ci/cd pipeline deployment? | 02:42 |
| sdake | DaveTurner mind i ask how long you have been doing this - just to get some context if this is a new issue or old issue | 02:43 |
| DaveTurner | We reimage the baremetal between pipeline runs | 02:43 |
| sdake | roger | 02:43 |
| sdake | i guess i'd ask why not use vms but its not my business ;-) | 02:44 |
| DaveTurner | Certain cases we do. But we have racks of systems to test for large scale deployments, so we try to stay as close to those proposed configs as possible. | 02:45 |
| sdake | DaveTurner if i ask any q's you are unable to answer just say you can answer - this channel is alot about learning | 02:45 |
| sdake | how long does your ci/cd pipeline take? | 02:45 |
| *** chas has quit IRC | 02:45 | |
| DaveTurner | I'd say a couple of hours. | 02:46 |
| sdake | cool | 02:46 |
| sdake | thats pretty fast | 02:46 |
| sdake | i guess i'm nosey but i'm curious if your using kolla's bifrost implementation | 02:47 |
| DaveTurner | Not yet - haven't had time to poke around at that. | 02:47 |
| sdake | cool if/when you do would love feedback | 02:48 |
| sdake | its not well tested like the rest of the code base typically is | 02:48 |
| DaveTurner | We are using Kolla to build openstack centralize services cluster along with regional deployments to subscribe to those centralizes services - hence all the ci/cd work. | 02:48 |
| DaveTurner | Sure - can do (at least, someone from my team). | 02:49 |
| sdake | hmm i dont get it but thats ok:) | 02:49 |
| DaveTurner | Oh- logs. One sec. | 02:49 |
| sdake | right ;) | 02:49 |
| sdake | DaveTurner are you in US tz or EMEA tz? | 02:50 |
| sdake | or apac? | 02:50 |
| DaveTurner | sdake gathering info in pastebin - just a sec. | 02:51 |
| DaveTurner | US central / Austin | 02:51 |
| sdake | if your on centos you can run sudo yum install fpaste | 02:51 |
| sdake | and fpaste the file | 02:51 |
| sdake | or pipe through fpaste | 02:51 |
| sdake | ubuntu doesn't have similar functionality | 02:51 |
| DaveTurner | OracleLinux 7 - close enough. | 02:51 |
| sdake | cool - although not sure if ole has fpaste or not | 02:52 |
| sdake | probably does if you install the right repos ;) | 02:52 |
| *** yuanying has quit IRC | 02:52 | |
| *** yingjun_ has joined #openstack-kolla | 02:53 | |
| *** yingjun has quit IRC | 02:53 | |
| DaveTurner | Right- pastebin for now :-) Will check that out later. | 02:53 |
| DaveTurner | sdak http://paste.openstack.org/show/589348/ | 02:53 |
| sdake | DaveTurner cool that looks good | 02:54 |
| sdake | DaveTurner on the host | 02:54 |
| sdake | go to /var/lib/docker/volumes/kolla | 02:54 |
| sdake | note volumes may be something else | 02:54 |
| sdake | and kolla may b instead mariadb | 02:54 |
| sdake | DaveTurner i can't check exact syntax atm because i am on corp vpn and dont' have access to my home lab boxes | 02:55 |
| sdake | cisco firewalls my home machine :( | 02:55 |
| DaveTurner | sdake one more log where I run status before the container bombs out: http://paste.openstack.org/show/589349/ | 02:55 |
| DaveTurner | understood | 02:55 |
| DaveTurner | That hostname is the main hostname configured for the management vlan. For the api vlan, we've registered hostnames for the IP's plumbed to that interface as well. format: hostname-api. | 02:56 |
| *** unicell has quit IRC | 02:56 | |
| DaveTurner | sdake: ok, in /var/lib/docker/volumes/kolla | 02:57 |
| sdake | DaveTurner try cd /var/lib/docker/*volumes*/heka | 02:57 |
| sdake | sorry all mixed up today ;) | 02:57 |
| DaveTurner | sdake there | 02:58 |
| sdake | yup i suggested a different commnad | 02:58 |
| sdake | i want to see your logs for rabbitmq | 02:59 |
| DaveTurner | Looking for rabbit log entries? I've yet to see any generated prior to the container restarting. | 02:59 |
| sdake | they are stored in the heka volume | 02:59 |
| DaveTurner | Yep- I meant, I was there in the directory. :-) | 02:59 |
| sdake | in heka there was no rabbitmq dir? | 02:59 |
| *** yingjun has joined #openstack-kolla | 03:00 | |
| *** yingjun_ has quit IRC | 03:00 | |
| DaveTurner | sdake I see _data under /var/lib/docker/volumes/heka | 03:01 |
| sdake | ok go to that dir pls | 03:03 |
| sdake | via cd | 03:03 |
| sdake | and tell me what ya got | 03:03 |
| DaveTurner | logstreamer sandbox_preservation | 03:03 |
| DaveTurner | Under logstreamer, I see logs for our other services but nothing for rabbit | 03:04 |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla: Modify the letter case in post-install.yml file. https://review.openstack.org/398024 | 03:04 |
| sdake | DaveTurner what about another node? | 03:06 |
| sdake | your doing multicontroller gating? | 03:06 |
| *** harbor has joined #openstack-kolla | 03:06 | |
| sdake | or single controller node | 03:06 |
| DaveTurner | multiple controller nodes. 3 in this case. | 03:07 |
| sdake | cool can you check the other two nodes | 03:07 |
| sdake | perhaps one of them has logs | 03:07 |
| *** harbor is now known as portdirect___ | 03:07 | |
| DaveTurner | I'll check the other 2. I believe we have already, but worth checking again... | 03:07 |
| *** chas has joined #openstack-kolla | 03:08 | |
| sdake | DaveTurner if that fails have a plan B:) | 03:08 |
| sdake | and C | 03:08 |
| *** portdirect__ has quit IRC | 03:09 | |
| DaveTurner | sdake ok, no logs on the other 2 | 03:09 |
| sdake | bummer | 03:09 |
| sdake | your vlan mgmt interface is which ip? | 03:09 |
| sdake | (pick a node) | 03:09 |
| sdake | rabbitmq is a big pile of steam | 03:10 |
| sdake | #1 cause of failures | 03:10 |
| DaveTurner | sdake: host: slcas275, management vlan interface: eno1.2706@eno1 ip: 10.248.68.71 | 03:11 |
| DaveTurner | That's the primary interface. We have another defined for api. | 03:11 |
| DaveTurner | We didn't hit this with our multinode deployment until we tried to get cute with segregating api traffic onto separate vlan/interface | 03:12 |
| sdake | understood | 03:12 |
| *** chas has quit IRC | 03:12 | |
| sdake | DaveTurner right, and that is how openstack should be deployed too :) | 03:12 |
| DaveTurner | via the *_interface options provided by kolla | 03:12 |
| DaveTurner | Right! | 03:12 |
| sdake | ok on host slcaa275 can you run hostname | 03:13 |
| DaveTurner | Yup. Returns slcas275 | 03:14 |
| sdake | let me read the bug | 03:14 |
| sdake | and the code | 03:14 |
| DaveTurner | Sure. Thanks for taking a look. | 03:15 |
| sdake | DaveTurner no guarantees i can ge tthis rolling for you, I am not a networking dude | 03:15 |
| sdake | why be employee number 70001 at a networking company? :) | 03:16 |
| DaveTurner | Ha! | 03:16 |
| sdake | DaveTurner can we crater these machines (i.e. use them for debug) | 03:17 |
| sdake | with possibly irreparable dmg :) | 03:17 |
| *** portdirect___ is now known as portdirect_away_ | 03:17 | |
| DaveTurner | sdake Sure. My guys in India will be on in a couple of hours to rerun the pipeline on these systems. They will be destroyed anyway. | 03:19 |
| sdake | cool so lets run docker stop rabbitmq on all the nodes | 03:19 |
| sdake | if that fails try docker rm rabbitmq | 03:20 |
| sdake | if that fials try docker rm -f rabbitmq | 03:20 |
| DaveTurner | stopped | 03:20 |
| sdake | cool so pick a node and run docker rm -f rabbitmq | 03:21 |
| sdake | docker images | grep rabbitmq -> paste | 03:21 |
| sdake | what we just did there was stop rabbitmq from rnaodmly restarting | 03:21 |
| sdake | and then deleted a rabbitmq instance where we are going to do some proper debug | 03:21 |
| DaveTurner | yup. got it | 03:22 |
| *** eaguilar has quit IRC | 03:23 | |
| sdake | groan a bug from may unfixed | 03:23 |
| DaveTurner | sdake disclaimer - we build from source and push to our own registry - in case you are trying to have me pull again from external registry. :-) | 03:24 |
| sdake | nope | 03:24 |
| DaveTurner | container gone | 03:24 |
| sdake | we are starting an image | 03:25 |
| sdake | but i need the image name to provide the proper command | 03:25 |
| sdake | and also the tag | 03:25 |
| DaveTurner | # docker images | grep rabbitmq slc11gcw.us.oracle.com:5000/cas/oraclelinux-source-rabbitmq next e6091c4a4057 20 hours ago 384.7 MB | 03:25 |
| sdake | ok docker run -it -u root slcwhatever/cas/oraclelinux-source-rabbitmq:next bash | 03:25 |
| sdake | dont forget the :5000 | 03:26 |
| sdake | bsaically image:tag | 03:26 |
| DaveTurner | yep.. started | 03:27 |
| DaveTurner | I'm in the container now | 03:27 |
| sdake | ok cat /etc/hosts | 03:27 |
| sdake | and dump into channel if short otherwise paste | 03:27 |
| DaveTurner | hmm.. should we have started with host networking? This looks ugly | 03:28 |
| DaveTurner | ::1localhost ip6-localhost ip6-loopback fe00::0ip6-localnet ff00::0ip6-mcastprefix ff02::1ip6-allnodes ff02::2ip6-allrouters 172.17.0.253d76df084b9 | 03:28 |
| sdake | we are just looking at /etc/hosts | 03:28 |
| sdake | ok | 03:28 |
| sdake | type exit | 03:28 |
| DaveTurner | I'm out | 03:28 |
| sdake | cat /etc/sysconfig/network-scripts/ipcfg-eno1 | 03:29 |
| *** chas has joined #openstack-kolla | 03:29 | |
| sdake | (i think that is the interface you mentioned?) | 03:29 |
| sdake | it might be ifcfg-eno1 | 03:29 |
| *** huikang has quit IRC | 03:29 | |
| DaveTurner | That's the interface on which both vlans are trunked. | 03:29 |
| DaveTurner | You want to see the config for eno1 interface, or the vlans based on that interface? | 03:30 |
| sdake | can you cat the vlan interfac efile | 03:30 |
| DaveTurner | # cat ifcfg-eno1.2706 DEVICE=eno1.2706 BOOTPROTO="static" IPADDR="10.248.68.71" NETMASK="255.255.255.192" NETWORK="10.248.68.64" GATEWAY=10.248.68.65 USERCTL=no ONBOOT=yes VLAN=yes NOZEROCONF=yes DEFROUTE=yes | 03:31 |
| DaveTurner | And the api vlan interface: | 03:31 |
| DaveTurner | # cat ifcfg-eno1.802 DEVICE=eno1.802 BOOTPROTO="static" IPADDR="10.248.66.212" NETMASK="255.255.255.192" NETWORK="10.248.66.192" USERCTL=no ONBOOT=yes NOZEROCONF=yes VLAN=yes | 03:31 |
| sdake | ok thats going to have to go in a paste pls | 03:31 |
| DaveTurner | indeed | 03:31 |
| sdake | as well as your main interface | 03:31 |
| sdake | let me get off the vpn | 03:32 |
| sdake | so i can access my bare metal gear | 03:33 |
| sdake | brb while your pasting :) | 03:33 |
| *** chas has quit IRC | 03:33 | |
| *** sdake_ has joined #openstack-kolla | 03:33 | |
| DaveTurner | sdake Here ya go: http://paste.openstack.org/show/589352/ | 03:34 |
| *** tonanhngo has joined #openstack-kolla | 03:34 | |
| *** hfu has quit IRC | 03:35 | |
| *** hfu has joined #openstack-kolla | 03:35 | |
| *** duonghq has quit IRC | 03:36 | |
| sdake_ | how did you setup the vlans? | 03:37 |
| sdake_ | manually or with a tool? | 03:37 |
| *** sdake has quit IRC | 03:37 | |
| DaveTurner | not via tool. manually for this test | 03:38 |
| *** tonanhngo has quit IRC | 03:39 | |
| sdake_ | ok | 03:39 |
| sdake_ | can you run ip link show -> paste | 03:39 |
| sdake_ | or another option would be ifconfig -> paste | 03:39 |
| sdake_ | i think you have your system seutp for only ipv6 | 03:39 |
| sdake_ | kolla does not work with ipv6 | 03:39 |
| sdake_ | what is the lro and gro options to ETHTOOL? | 03:40 |
| sdake_ | ifconfig and or ip lin kshow will tell us :) | 03:41 |
| DaveTurner | No. We don't use ipv6 here. LRO and GRO for large and generic receive offload. | 03:42 |
| DaveTurner | We had to mess with those setting back in icehouse when using vxlan for performance reasons | 03:42 |
| DaveTurner | http://paste.openstack.org/show/589354/ | 03:42 |
| DaveTurner | sdake: you can ignore the 803/804 vlans. Those are there in case the nodes are used as network nodes. Those are our public tenant networks. Our dev nodes are configured to be used as any number of openstack node types. | 03:44 |
| *** duonghq has joined #openstack-kolla | 03:45 | |
| DaveTurner | sdake: and ip addr output: http://paste.openstack.org/show/589355/ | 03:45 |
| *** fragatin_ has joined #openstack-kolla | 03:47 | |
| sdake_ | DaveTurner there are multiple problems here | 03:48 |
| sdake_ | one is your eno1 is setup for ipv6 only | 03:48 |
| sdake_ | the kolla code uses the api interface for rabbitmq | 03:48 |
| sdake_ | it should instead use the management network | 03:48 |
| sdake_ | no idea why its designed using the api network | 03:49 |
| sdake_ | so basically when kolla resolves, it resolves an ipv6 address (the api interface) | 03:49 |
| DaveTurner | Hmmm.. | 03:49 |
| *** adrian_otto has joined #openstack-kolla | 03:50 | |
| *** yuanying has joined #openstack-kolla | 03:50 | |
| sdake_ | so your probably wondering how to fix | 03:51 |
| *** fragatina has quit IRC | 03:51 | |
| DaveTurner | I'll get to the ipv6 think in a second. First, then api_interface. When defined in kolla, it attempts to setup rabbit to listen on that interface, correct? | 03:51 |
| sdake_ | unfortunately | 03:51 |
| sdake_ | that needs a proper resolution | 03:51 |
| sdake_ | as is, its a security problem | 03:51 |
| DaveTurner | OK- I assume we'll need to update the roles / templates for that | 03:51 |
| DaveTurner | Yes, that makes sense. | 03:52 |
| *** fragatin_ has quit IRC | 03:52 | |
| *** berendt has joined #openstack-kolla | 03:52 | |
| sdake_ | sup berendt | 03:52 |
| berendt | sdake_ morining.. moving to the airport | 03:53 |
| sdake_ | there is a phone at wells and lake, you can make it! | 03:53 |
| sdake_ | berendt sounds like a startup in progress! | 03:53 |
| berendt | sdake_ i think so :) | 03:54 |
| *** berendt has quit IRC | 03:54 | |
| DaveTurner | sdake: Re: ipv6 - the vlans configured on eno1 are ipv4 enabled reachable over IP. You're thinking it's still a problem though? | 03:54 |
| sdake_ | DaveTurner not certain - but that /etc/hosts file says so ;) | 03:55 |
| sdake_ | DaveTurner you notice it lacked an ipv6 address | 03:55 |
| sdake_ | sorry lacked an ipv4 address | 03:55 |
| DaveTurner | sdake In the test container we launched, yes. However on the host.. and in the containers where the host networking is used, you see something like this: | 03:56 |
| DaveTurner | # cat /etc/hosts 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 10.248.68.71slcas275 | 03:56 |
| sdake_ | hmm | 03:57 |
| sdake_ | ok | 03:57 |
| sdake_ | well thats good data | 03:57 |
| sdake_ | i turn off ipv6 entirely | 03:57 |
| sdake_ | so I have never noticed a delta | 03:57 |
| DaveTurner | But, I can explicitly disable for ipv6 just to eliminate this as a problem | 03:57 |
| DaveTurner | Agreed. | 03:57 |
| sdake_ | thats not the problem then | 03:57 |
| sdake_ | are you able to run kolla-ansible -i ~/inventory destroy? | 03:58 |
| sdake_ | followed by kolla-ansible -i ~/inventory deploy outside of ci? | 03:58 |
| DaveTurner | Good point though. We should probably be explicite about it. (ipv6) | 03:58 |
| sdake_ | what really needs to happen is kolla needs to implement ipv6 support in the ansible orchestration code | 03:58 |
| DaveTurner | sdake. Hmm.. I could if I knew which pipeline vm kick off the deployment. Let me poke around. My other guys kicked this off during India hours, and I took over. | 03:59 |
| sdake_ | ok well can ur guys cotact our peeps in apac timezone? | 04:00 |
| sdake_ | jeffrey4l is who to look for | 04:00 |
| sdake_ | have them reference the bug | 04:00 |
| sdake_ | i'll leave a note | 04:00 |
| sdake_ | or if you want to experiment with your vm ci jobs we can do that too :) | 04:00 |
| DaveTurner | sdake: Sure. I can do that. Anything in particular you wanted to try on the redeployment? | 04:01 |
| sdake_ | yes i want you to apply a patch to the kolla code | 04:01 |
| sdake_ | no gurantees it will work | 04:01 |
| sdake_ | it may take several tries | 04:01 |
| sdake_ | we may have to hand this off to the apac folks because its 9pm here | 04:02 |
| sdake_ | i generally hit the rack before midnight | 04:02 |
| DaveTurner | Understood. I'll have my guys look out for jeffrey4l - and leave them an update regarding our chat. | 04:03 |
| *** Jeffrey4l has joined #openstack-kolla | 04:03 | |
| sdake_ | there is jeffrey4l right tthere ;) | 04:03 |
| DaveTurner | :-) | 04:03 |
| sdake_ | Jeffrey4l works more maniacally then I do | 04:03 |
| Jeffrey4l | hi sdake_ | 04:03 |
| sdake_ | which is really saying somethign :) | 04:03 |
| sdake_ | my son "heal me!!!!!!' | 04:03 |
| sdake_ | poor kid | 04:04 |
| sdake_ | welcome to MMORPGS a a dps class | 04:04 |
| sdake_ | the epics are in the fire | 04:04 |
| Jeffrey4l | i saw the repo is split, but the last commit is 4 days ago. should we need sync the commit? | 04:04 |
| sdake_ | Jeffrey4l sync will hae to be manual | 04:04 |
| sdake_ | Jeffrey4l it was best i could do | 04:04 |
| Jeffrey4l | cool. | 04:04 |
| sdake_ | i did the work on saturday | 04:05 |
| sdake_ | to minimize the pain | 04:05 |
| sdake_ | but there is still some | 04:05 |
| sdake_ | i have some reviews in the queue to clean up kolla | 04:05 |
| sdake_ | been helping DaveTurner out with a bug | 04:05 |
| Jeffrey4l | ok. | 04:05 |
| sdake_ | thought i'd get your input since you left last comment on it | 04:05 |
| sdake_ | DaveTurner can you link teh bug again - irc client restart on vpn reset wiped out the info | 04:06 |
| Jeffrey4l | DaveTurner, hmm which bug? just log in, no context. | 04:06 |
| DaveTurner | Sure.. | 04:06 |
| DaveTurner | https://bugs.launchpad.net/kolla/+bug/1583655 | 04:06 |
| openstack | Launchpad bug 1583655 in kolla newton "rabbitmq setup from secondary IP" [Critical,Confirmed] | 04:06 |
| sdake_ | Jeffrey4l my thinking is rabbitmq definately needs to run on the mgmt interface | 04:07 |
| sdake_ | not the api interface | 04:07 |
| sdake_ | api interface may be exposed to internets | 04:07 |
| *** Pavo has quit IRC | 04:07 | |
| Jeffrey4l | 'got it. brb. 2min. | 04:09 |
| sdake_ | your going to help us mr. anderson. whether you want to or not! | 04:10 |
| *** Pavo has joined #openstack-kolla | 04:12 | |
| sdake_ | DaveTurner an you paste your host's /etc/host file | 04:12 |
| DaveTurner | sdake_ which one? From the test container or from the host / other functioning containers? | 04:13 |
| sdake_ | the second | 04:13 |
| Jeffrey4l | back. | 04:13 |
| DaveTurner | sdake_ /etc/hosts http://paste.openstack.org/show/589356/ | 04:14 |
| sdake_ | .71 is api interface or mgmt interface? | 04:15 |
| Jeffrey4l | DaveTurner, what error message did u get? | 04:15 |
| sdake_ | Jeffrey4l containers restart | 04:15 |
| sdake_ | Jeffrey4l no logs for rabbitmq excpet docker logs | 04:16 |
| DaveTurner | sdake_ mgmt. when troubleshooting this earlier, we manually added entries for the api IP's - although, they are actually registered in DNS. | 04:16 |
| sdake_ | Jeffrey4l no error message in particular | 04:16 |
| Jeffrey4l | DaveTurner, show me the docker logs? | 04:16 |
| *** jtriley has quit IRC | 04:16 | |
| sdake_ | DaveTurner you will have to do that on a box other then the one your on presently | 04:16 |
| Jeffrey4l | and configurations. | 04:16 |
| sdake_ | Jeffrey4l note i had him run docker stop rabbitmq | 04:16 |
| sdake_ | on all 3 nodes | 04:17 |
| Jeffrey4l | the /etc/hosts seems wrong. | 04:17 |
| DaveTurner | Jeffrey41 unfortunately, nothing to see in heka logs but let me show output of rabbit container before it restarts | 04:17 |
| sdake_ | Jeffrey4l there is no rabbitmq dir in the heka dir... | 04:17 |
| sdake_ | Jeffrey4l master as of a few weeks ago - DaveTurner is doing ci/cd trailing master of koll aa couple weeks | 04:18 |
| sdake_ | a couple weeks old that is | 04:18 |
| DaveTurner | Jeffrey41 - no, there is not. Docker logs output: http://paste.openstack.org/show/589357/ | 04:19 |
| Jeffrey4l | if the hostname can not be parsed, it may not generate logs in heka. | 04:19 |
| Jeffrey4l | could u fix your hostname and /etc/hosts file, then try again? | 04:20 |
| sdake_ | Jeffrey4l could you be more specific about what needs fixing | 04:20 |
| DaveTurner | Jefferey41 - we have ipv4 vlans configured on eno1 device which are reachable over IP, etc - but as sdake mentioned prior to you jumping on, ipv6 is not specifically disable on eno. I can try to update config so that this is explicitely disabled, in case that could make a diff. | 04:22 |
| Jeffrey4l | https://github.com/openstack/kolla/blob/master/ansible/roles/prechecks/tasks/port_checks.yml#L716 | 04:22 |
| DaveTurner | ..and yes, we can do whatever to the /etc/hosts. | 04:22 |
| *** adrian_otto has quit IRC | 04:22 | |
| Jeffrey4l | check these lines. | 04:22 |
| *** chas has joined #openstack-kolla | 04:22 | |
| Jeffrey4l | the hostname should be resolved to api_interface's address. | 04:23 |
| *** adrian_otto has joined #openstack-kolla | 04:24 | |
| sdake_ | Jeffrey4l yes however as i mentioned rabbitmq should lrun on the mgmt interfaces | 04:24 |
| DaveTurner | Jefferey41 That's the thing... the primary hostname.. the one which is defined in the inventory, is that of the management interface (network_interface in globals.yml) | 04:25 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: TEST MASTER BRANCH https://review.openstack.org/398093 | 04:26 |
| *** adrian_otto has quit IRC | 04:26 | |
| sdake_ | Jeffrey4l master is busted | 04:26 |
| sdake_ | or will be when my patches merge | 04:26 |
| Jeffrey4l | sdake_, but normally, it is bad to use the mgmt address for hostname, especially when you have a dns nameserver. | 04:27 |
| *** adrian_otto has joined #openstack-kolla | 04:27 | |
| Jeffrey4l | hmm. the .gitreview is wrong in kolla-ansible. i was trying to test kolla-ansible ci . | 04:27 |
| sdake_ | Jeffrey4l i can fix the gitreview if yo ulike | 04:27 |
| Jeffrey4l | OK. | 04:27 |
| *** chas has quit IRC | 04:28 | |
| coolsvap | sdake_: the gitreview change is already there | 04:28 |
| coolsvap | needs +W | 04:28 |
| DaveTurner | Jefferey41 - so if we have ip's plumbed to both the management and api interfaces, kolla expects the primary hostname to be that of the api interface, not the network_interface? | 04:28 |
| sdake_ | coolsvap cool | 04:28 |
| sdake_ | DaveTurner that actually does make sense | 04:28 |
| Jeffrey4l | DaveTurner, yes. | 04:29 |
| DaveTurner | primary meaning, that which is defined in inventory | 04:29 |
| Jeffrey4l | DaveTurner, primary means the first hostname in /etc/hosts file. | 04:29 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: TEST MASTER BRANCH https://review.openstack.org/398094 | 04:29 |
| Jeffrey4l | the hostname resovlable is not related to inventory files. | 04:29 |
| Jeffrey4l | oops. wrong test ps. | 04:30 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: TEST MASTER BRANCH https://review.openstack.org/398095 | 04:30 |
| Jeffrey4l | we are using `ansible_hostname` variable in playbooks rather than `inventory_hostname` so we do not care about the hostname in inventory file. | 04:31 |
| DaveTurner | Jeffery41 OK. I'll need to rethink our network segregation approach. | 04:31 |
| DaveTurner | And, ansible_hostname is the "primary" hostname? | 04:32 |
| Jeffrey4l | yes. | 04:33 |
| Jeffrey4l | the best solution should be use multi line for hostname in /etc/hosts file. | 04:34 |
| Jeffrey4l | like: | 04:34 |
| Jeffrey4l | 192.168.10.2 hostname1 | 04:34 |
| Jeffrey4l | 192.168.10.2 hostname_2 | 04:34 |
| sdake_ | DaveTurner no what needs to happen is we need to usethe mgmt interface for rabbitmq | 04:35 |
| sdake_ | Jeffrey4l is it not possible to use the mgmt interface for rabbitmq? | 04:35 |
| *** jtriley has joined #openstack-kolla | 04:35 | |
| Jeffrey4l | before, do we use hostname for mgmt interface in most of case? | 04:36 |
| Jeffrey4l | if so, we can. | 04:36 |
| sdake_ | we should be using mgmt interface for anything related to non-api functionality | 04:36 |
| sdake_ | putting rabbitmq on the api interface is a significant security issue | 04:37 |
| Jeffrey4l | yes. but rabbitmq case is more specially. it is related to the hostname of the node. | 04:37 |
| sdake_ | so we are using hostname irrelevant of whether its a mgmt or api interface? | 04:38 |
| sdake_ | if that is the case, hostname should always be the mgmt interface | 04:38 |
| Jeffrey4l | or check the bjolo's idea https://bugs.launchpad.net/kolla/+bug/1583655/comments/2 | 04:38 |
| openstack | Launchpad bug 1583655 in kolla newton "rabbitmq setup from secondary IP" [Critical,Confirmed] | 04:38 |
| DaveTurner | Jeffrey41 - So, we currently have mgmt and api vlan configured. System hostname is set to the DNS registered hostname of mgmt vlan IP. This is also where the default GW is set. We also have ip plumbed to api vlan, and it's registered in DNS. | 04:39 |
| DaveTurner | sdake_ your last comment is what our configuration assumed | 04:39 |
| Jeffrey4l | we are using container, and we can change the hostname of the rabbitmq container. | 04:39 |
| Jeffrey4l | if it works, then we can not need require the hostname resolvable | 04:40 |
| sdake_ | bjolo's solution wfm | 04:40 |
| *** jtriley has quit IRC | 04:40 | |
| sdake_ | i thought /etc/hsotname was bindmounted though | 04:40 |
| Jeffrey4l | docker run -h, --hostname string Container host name | 04:40 |
| sdake_ | automatically by docker | 04:40 |
| *** khamtamtun has joined #openstack-kolla | 04:40 | |
| sdake_ | -h and --net=host are incompaitblle iirc | 04:41 |
| sdake_ | been down that path before with the renaming containers work | 04:41 |
| DaveTurner | sdake_ also tried that during debug.. incompatible with net=host | 04:41 |
| sdake_ | i think this is because --net=host bindmounts /etc/hostnmae ;-) | 04:42 |
| sdake_ | so bjolo's solution wont work | 04:42 |
| Jeffrey4l | yes. it do not work. ;( | 04:42 |
| openstackgerrit | caoyuan proposed openstack/kolla: Move the "enable_destroy_images" into configure file https://review.openstack.org/397633 | 04:43 |
| Jeffrey4l | DaveTurner, kolla use fixed api_interface for hostname, right now. it won't work in your env, unless you change the code. | 04:43 |
| sdake_ | Jeffrey4l i thin kthe default code is busted | 04:43 |
| sdake_ | Jeffrey4l do you agree or disagree? | 04:44 |
| sdake_ | Jeffrey4l its binding to api - which can be exposed to the internet | 04:44 |
| sdake_ | it should always bind to the mgmt interface - which is not exposed to the internet | 04:44 |
| Jeffrey4l | yep. | 04:44 |
| coolsvap | sdake_: shouldn't we remove the gates prior to removing code from kolla? | 04:44 |
| *** tonanhngo has joined #openstack-kolla | 04:44 | |
| Jeffrey4l | agree. | 04:44 |
| sdake_ | coolsvap not sure it matters | 04:44 |
| *** khamtamtun has quit IRC | 04:44 | |
| coolsvap | i think it will be reverse of what we do while adding new | 04:45 |
| *** krtaylor has joined #openstack-kolla | 04:45 | |
| coolsvap | we first update in project and tox then add new gates in project config | 04:45 |
| * coolsvap confirms the sequence in mind | 04:45 | |
| Jeffrey4l | btw, we do not have mgmt interface concept right now. | 04:46 |
| sdake_ | i was hoping we would end up with cross-gating | 04:46 |
| sdake_ | Jeffrey4l mgmt interface = network_interface right? | 04:46 |
| sdake_ | coolsvap i had hoped we would have cross-repo gating | 04:46 |
| DaveTurner | sdake_ my assumption as well... network_interface = mgmt_interface. | 04:46 |
| sdake_ | coolsvap although i'd arther clean that up after deleting crap ;) | 04:47 |
| DaveTurner | Perhaps just a case of wishful thinking. :-) | 04:47 |
| sdake_ | that way we dont end up with a megapatch | 04:47 |
| sdake_ | hence i don't think its necessary to remove any gates | 04:47 |
| *** newmember has joined #openstack-kolla | 04:47 | |
| coolsvap | hmm | 04:47 |
| sdake_ | unless we want to churn infra review time on that work | 04:47 |
| Jeffrey4l | DaveTurner, no. network_interface is not used in playbooks. it is only exist in group_vars/all.yml file. | 04:47 |
| *** adrian_otto has quit IRC | 04:47 | |
| sdake_ | taking out gates and putting them back in | 04:47 |
| sdake_ | Jeffrey4l hwich interface does mariadb run on? | 04:48 |
| sdake_ | and all of the haproxy/etc | 04:48 |
| Jeffrey4l | sdake_, api_interface ;( | 04:48 |
| *** tonanhngo has quit IRC | 04:48 | |
| sdake_ | you ahve to be kidding me | 04:48 |
| sdake_ | when did that change get made? | 04:48 |
| Jeffrey4l | sdake_, https://github.com/openstack/kolla/blob/master/ansible/roles/mariadb/templates/galera.cnf.j2#L14 | 04:49 |
| Jeffrey4l | i do not think we changed it. | 04:49 |
| *** adrian_otto has joined #openstack-kolla | 04:49 | |
| Jeffrey4l | you are wrong about the api_interface is open to internet. | 04:50 |
| Jeffrey4l | we have kolla_external_vip_interface | 04:50 |
| sdake_ | this is a r egression: https://github.com/openstack/kolla/commit/5838bd0b3c12eed1514e59852564006f044f241b | 04:50 |
| Jeffrey4l | when this is configured, this is open to internet. | 04:50 |
| Jeffrey4l | before this https://github.com/openstack/kolla/commit/5838bd0b3c12eed1514e59852564006f044f241b it is using api_interface, too. | 04:51 |
| sdake_ | Jeffrey4l got it | 04:51 |
| sdake_ | Jeffrey4l ok - well its still wrong ;-) | 04:51 |
| Jeffrey4l | so, right now. the kolla_external_vip_interface is api_interface you want and the api_interface is mgmt interface | 04:51 |
| sdake_ | api interface is for apis to bind to | 04:52 |
| sdake_ | network_interface should be for servers to bind to internally | 04:52 |
| sdake_ | kolla_external_vip_interface should be for an internet facing ip address i guess | 04:53 |
| Jeffrey4l | network_interface is just a simple variable in group_vars/all.yml file. it is not used in any role. | 04:53 |
| Jeffrey4l | other *_interface is refer to this variable if they are not configured. | 04:53 |
| DaveTurner | Except as a default when api_interface isn't defined? | 04:53 |
| Jeffrey4l | DaveTurner, yep. | 04:53 |
| DaveTurner | Jeffrey41 sorry to back up a bit, but can you elaborate on "kolla use fixed api_interface for hostname" | 04:54 |
| Jeffrey4l | so i think current configuration is OK. | 04:54 |
| Jeffrey4l | DaveTurner, it won't work as sdake_ said. | 04:54 |
| *** sdake_ is now known as sdake | 04:54 | |
| Jeffrey4l | we can not use --hostname <> with --net host | 04:55 |
| sdake | time for a cuban | 04:55 |
| sdake | bbiaf | 04:55 |
| DaveTurner | Jeffrey41- right, I discovered that during debugging earlier. Didn't think it was possible considering /etc/ is bind mounted | 04:56 |
| Jeffrey4l | sorry? | 04:57 |
| Jeffrey4l | hmm just tried this and it works. docker run -it --rm -h abc --net host --add-host a:1.1.1.1 centos:7 bash | 04:57 |
| Jeffrey4l | we can change the /etc/hosts in container only. | 04:57 |
| Jeffrey4l | but the host os' /etc/hosts is override. | 04:57 |
| sdake | that must be a new feature | 04:57 |
| openstackgerrit | caoyuan proposed openstack/kolla: Add empty prechecks.yml for Designate https://review.openstack.org/398100 | 04:58 |
| Jeffrey4l | the disadvantage is the original host os' /etc/hosts file is override. this may case some other issue . | 04:58 |
| DaveTurner | Jeffrey41 Didn't 1583655 mention this approach as a possible option? | 05:00 |
| coolsvap | sdake: i've moved most of the ansible blueprints to kolla-ansible launchpad | 05:00 |
| Jeffrey4l | DaveTurner, it didn't clear how to implement this. | 05:00 |
| Jeffrey4l | coolsvap, cool. | 05:00 |
| Jeffrey4l | I will leave some comments in that bug. | 05:01 |
| coolsvap | i am not sure whether to -1 /-2 the patches because some patches have changes in dockerfiles as well | 05:01 |
| sdake | ya or /etc in that one i just -2ed | 05:02 |
| sdake | not sure where /etc belongs | 05:02 |
| Jeffrey4l | i'd like use -1 for the patch and tell him to split his patch. | 05:02 |
| coolsvap | yes me too | 05:02 |
| sdake | Jeffrey4l so how do we get DaveTurner operational with minimal pain while we sort out a solution to this security problem upstream? | 05:03 |
| DaveTurner | Jeffrey41 sdake thanks for the info. I'll need to drop soon but will update my team in India. Any suggestions on how we can move forward with our pipeline builds given our current config? | 05:03 |
| sdake | Jeffrey4l clearly I'm confused ;-) | 05:03 |
| DaveTurner | sdake beat me to the question.. | 05:04 |
| Jeffrey4l | could u post you globals.yml file out? DaveTurner | 05:04 |
| coolsvap | Jeffrey4l sdake: https://review.openstack.org/#/c/397888/ need +W | 05:06 |
| openstackgerrit | Merged openstack/kolla-ansible: Set up .gitreview https://review.openstack.org/397888 | 05:08 |
| DaveTurner | Jeffrey41 let me grab it for you | 05:09 |
| Jeffrey4l | ok. | 05:10 |
| Jeffrey4l | coolsvap, it merged ;) | 05:10 |
| *** sdake_ has joined #openstack-kolla | 05:10 | |
| *** sdake has quit IRC | 05:14 | |
| openstackgerrit | caoyuan proposed openstack/kolla: Add empty prechecks.yml for Designate https://review.openstack.org/398100 | 05:20 |
| *** dims has quit IRC | 05:20 | |
| DaveTurner | Jeffrey41 - going to be around for a while? I need to have my India team provide this information for you, as it seems the VM created as part of the pipeline for this test deployment is no more. :-( | 05:22 |
| DaveTurner | We create several a day, so no worries there. I can have them look for you in here. | 05:22 |
| *** sdake_ has quit IRC | 05:24 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Move the "enable_destroy_images" into configure file https://review.openstack.org/398106 | 05:24 |
| *** tonanhngo has joined #openstack-kolla | 05:28 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Update the default value for Heat https://review.openstack.org/398107 | 05:31 |
| *** tonanhngo has quit IRC | 05:32 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Add empty prechecks.yml for Designate https://review.openstack.org/398108 | 05:33 |
| *** adrian_otto has quit IRC | 05:46 | |
| *** adrian_otto has joined #openstack-kolla | 05:51 | |
| *** harlowja has quit IRC | 05:55 | |
| *** unicell has joined #openstack-kolla | 05:58 | |
| *** adrian_otto has quit IRC | 05:58 | |
| *** dims has joined #openstack-kolla | 05:59 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: change network_bw_out to network.bw.out in cloudkitty.conf.j2 https://review.openstack.org/397641 | 06:01 |
| *** Pavo has quit IRC | 06:08 | |
| *** Pavo has joined #openstack-kolla | 06:12 | |
| openstackgerrit | Merged openstack/kolla: Allow for specifying a KOLLA_CONFIG_FILE https://review.openstack.org/398008 | 06:15 |
| *** chas has joined #openstack-kolla | 06:17 | |
| *** yingjun_ has joined #openstack-kolla | 06:21 | |
| *** yingjun has quit IRC | 06:21 | |
| *** chas has quit IRC | 06:21 | |
| *** harlowja_at_home has joined #openstack-kolla | 06:32 | |
| *** harlowja_at_home has quit IRC | 06:33 | |
| *** chas has joined #openstack-kolla | 06:45 | |
| coolsvap | Jeffrey4l: https://blueprints.launchpad.net/kolla/+spec/remove-node can be moved to kolla-ansible right? | 06:48 |
| Jeffrey4l | coolsvap, yep. | 06:48 |
| *** chas has quit IRC | 06:49 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible: Modify the letter case in post-install.yml file https://review.openstack.org/398133 | 06:50 |
| *** kamal__ has joined #openstack-kolla | 06:53 | |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Allow for externally managed configuration https://review.openstack.org/332590 | 06:54 |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Allow for specifying a KOLLA_CONFIG_FILE https://review.openstack.org/398138 | 06:58 |
| kamal__ | hi Jeffrey4l, I am following up on DavidTurner chat with you and sdake | 07:03 |
| kamal__ | u there | 07:03 |
| Jeffrey4l | kamal__, yep. | 07:04 |
| kamal__ | hi jeffery, from you last conversation i believe you wanted to globals.yml | 07:05 |
| Jeffrey4l | kamal__, yep. | 07:05 |
| Jeffrey4l | you and DaveTurner are colleague? | 07:05 |
| kamal__ | yes | 07:05 |
| kamal__ | http://paste.openstack.org/show/589364/ | 07:06 |
| Jeffrey4l | kamal__, and you `ip a` `hostname` output on your control nodes. | 07:06 |
| kamal__ | hostnames are slcas275 slcas283 slcas291 | 07:08 |
| kamal__ | ip addr output for slcas275 is http://paste.openstack.org/show/589365/ | 07:09 |
| openstackgerrit | Joshua Harlow proposed openstack/kolla: Use itertools.chain and show how many files utime modified https://review.openstack.org/398145 | 07:09 |
| Jeffrey4l | kamal__, ok. | 07:09 |
| *** unicell has quit IRC | 07:10 | |
| *** unicell has joined #openstack-kolla | 07:11 | |
| *** chas has joined #openstack-kolla | 07:12 | |
| kamal__ | ip addr for slcas283 http://paste.openstack.org/show/589366/ , for slcas291 http://paste.openstack.org/show/589367/ | 07:12 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Update the default value for Heat https://review.openstack.org/398107 | 07:12 |
| Jeffrey4l | kamal__, you are using network_interface: "eno1" , but there is no address on eno1 interface. | 07:16 |
| *** Satya_ has joined #openstack-kolla | 07:16 | |
| *** chas has quit IRC | 07:16 | |
| kamal__ | eno1.802 is my api_interface and eno1.2706 is my mgmt interface | 07:18 |
| Satya_ | Hi Britt | 07:18 |
| Jeffrey4l | kamal__, i do not see that in your globals.yml file. | 07:19 |
| kamal__ | in the multinode file | 07:21 |
| kamal__ | slcas275 api_interface=eno1.802 tunnel_interface=eno1.802 slcas283 api_interface=eno1.802 tunnel_interface=eno1.802 slcas291 api_interface=eno1.802 tunnel_interface=eno1.802 | 07:21 |
| Jeffrey4l | ok | 07:21 |
| Satya_ | Hi Jeffrey. | 07:22 |
| Jeffrey4l | kamal__, then what ip address/interface is resolved to ? | 07:22 |
| Jeffrey4l | sup Satya_ | 07:22 |
| Satya_ | me kamal and David working together :) | 07:23 |
| Satya_ | on the same issue | 07:23 |
| Jeffrey4l | cool | 07:23 |
| kamal__ | slcas275 resolves to eno1.2704 ipv4 address, and slcas275-api resolves to eno1.2706 ipv4 address | 07:24 |
| kamal__ | sorry slcas275 eno1.802 | 07:24 |
| *** magicboiz has joined #openstack-kolla | 07:25 | |
| Jeffrey4l | hmm. so it is api_interface, right? it should be ok. | 07:25 |
| kamal__ | ok, so slcas275 resolves to eno1.2706 | 07:26 |
| kamal__ | and slcas275-api resolves to eno1.802 | 07:26 |
| Jeffrey4l | OK. | 07:26 |
| Satya_ | 802 is our api interface | 07:26 |
| Satya_ | 2706 is our mgmt interface | 07:27 |
| Jeffrey4l | yes. the the quick fix solution is: modify the hostname name from slcas275 to slcas275-api if possible. | 07:27 |
| Satya_ | That will break other services... :( | 07:27 |
| Jeffrey4l | hmm. | 07:29 |
| Jeffrey4l | another workaround is change this file. | 07:29 |
| Jeffrey4l | https://github.com/openstack/kolla/blob/master/ansible/roles/rabbitmq/templates/rabbitmq-clusterer.config.j2 | 07:29 |
| Jeffrey4l | add `-api` suffix into the hostname. | 07:30 |
| Jeffrey4l | like: 'rabbit@{{ hostvars[host]['ansible_hostname'] }}-api | 07:30 |
| Jeffrey4l | maybe we can introduce a variable like: rabbitmq_hostname into kolla. | 07:30 |
| Jeffrey4l | in default, it point into ansible_hostname, | 07:31 |
| Jeffrey4l | but you can change it based on you environment. | 07:31 |
| Satya_ | we tried that manually changing it for container by changing the /etc/kolla/rabbitmq/rabbitmq-cluster.config | 07:32 |
| Satya_ | that didnt helped as the hostname of the container points to the slcas275 rather than slcas275-api | 07:33 |
| Jeffrey4l | does this file need to be change? https://github.com/openstack/kolla/blob/master/ansible/roles/rabbitmq/templates/rabbitmq.config.j2 | 07:34 |
| Satya_ | http://paste.openstack.org/show/589369/ | 07:34 |
| Satya_ | yes | 07:35 |
| Jeffrey4l | i am not sure why rabbitmq require hostname resolvable. | 07:35 |
| Satya_ | we changed that too | 07:35 |
| *** f13o has joined #openstack-kolla | 07:35 | |
| Jeffrey4l | Satya_, do u have any idea why rabbitmq require hostname resolvable? | 07:36 |
| Satya_ | i guess thats a piece of erlang but not sure... | 07:37 |
| *** magicboiz has quit IRC | 07:40 | |
| Satya_ | http://www.rabbitmq.com/networking.html | 07:42 |
| Satya_ | inside that "Hostname Resolution and DNS" | 07:42 |
| * Jeffrey4l is checking. | 07:45 | |
| Jeffrey4l | Satya_, i do not think we can have a workaround. | 07:47 |
| Satya_ | what we can have as a solution for this? | 07:48 |
| Jeffrey4l | change the hostname ;( | 07:49 |
| Jeffrey4l | or use mgmt interface for api_interface. | 07:49 |
| Jeffrey4l | you can use current api_interface for external_interface. | 07:50 |
| Satya_ | hmmm but i am not sure if other services like nova/neutron will break in that case | 07:50 |
| Jeffrey4l | why? | 07:50 |
| Satya_ | but i will give it a shot.... | 07:50 |
| Jeffrey4l | you are not using kkolla_external_vip_interface, right? | 07:51 |
| Satya_ | as for nova and neutron the mgmt interface is 2706 and api is 802 | 07:51 |
| Satya_ | we are... | 07:51 |
| Jeffrey4l | where is it? | 07:52 |
| Jeffrey4l | i can not see it on you globals.yml and your inventory file( post above, 2lines) | 07:52 |
| Satya_ | yeah sorry that neutrons public network interface | 07:52 |
| Satya_ | will give it a shot | 07:53 |
| Satya_ | lets see | 07:53 |
| Jeffrey4l | got. try it and i think it should work. | 07:53 |
| *** shardy has joined #openstack-kolla | 07:56 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Update the default value for Heat https://review.openstack.org/398107 | 07:58 |
| *** f13o_ has joined #openstack-kolla | 07:59 | |
| bjolo | morning | 08:01 |
| *** f13o has quit IRC | 08:02 | |
| Jeffrey4l | morning ;) | 08:02 |
| Jeffrey4l | bjolo, could u take look at this https://bugs.launchpad.net/kolla/+bug/1583655 | 08:04 |
| openstack | Launchpad bug 1583655 in kolla newton "rabbitmq setup from secondary IP" [Critical,Confirmed] | 08:04 |
| *** magicboiz has joined #openstack-kolla | 08:05 | |
| *** skramaja_ has joined #openstack-kolla | 08:05 | |
| *** skramaja has quit IRC | 08:06 | |
| *** Pavo has quit IRC | 08:08 | |
| * bjolo looking | 08:08 | |
| bjolo | yes what about it | 08:08 |
| bjolo | see you have had a discussion on irc about it | 08:08 |
| bjolo | at the conference it was mentioned that rabbit from version 3.6.x something did not require hostname resolution any more | 08:10 |
| bjolo | works with IP | 08:11 |
| bjolo | hence this issue will resolve itself eventually | 08:11 |
| bjolo | however we do have an issue today that needs to be resolved | 08:11 |
| Jeffrey4l | we are using 3.6.x, so we can remove the hostname resolve right now? | 08:12 |
| Jeffrey4l | but Satya_ still have such issue. | 08:12 |
| *** Pavo has joined #openstack-kolla | 08:12 | |
| bjolo | i dont know exactly what version it is from, just that we discussed this issue at the summit and someone mentioned this | 08:13 |
| Jeffrey4l | OK. | 08:13 |
| bjolo | the general take is that we are bound to use the distrobution packages for rabbit | 08:14 |
| Jeffrey4l | yes. but it is more stable to use distro package. | 08:15 |
| bjolo | ouch is this correct. centos 7 use rabbit 3.3.5? | 08:17 |
| bjolo | Version : 3.3.5 | 08:17 |
| Jeffrey4l | we install rabbit from rdo. | 08:17 |
| bjolo | ah ok | 08:17 |
| bjolo | what version is that? | 08:18 |
| Jeffrey4l | 3.6.5 | 08:18 |
| Satya_ | we are using 3.6.5 | 08:19 |
| bjolo | ok | 08:20 |
| *** Serlex has joined #openstack-kolla | 08:20 | |
| kamal__ | http://paste.openstack.org/show/589372/ | 08:20 |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible: Modify the letter case in ansible yml file https://review.openstack.org/398133 | 08:20 |
| bjolo | and what about ubuntu 1604? version? | 08:20 |
| Jeffrey4l | bjolo, we install the same version for ubuntu ,which is download from rabbitmq offical site. | 08:21 |
| bjolo | so we should be able to just use IP then? | 08:22 |
| bjolo | given that the claim is true that rabbit works with just IP now | 08:22 |
| Jeffrey4l | i am not sure. | 08:22 |
| Satya_ | I too with the current situation we are running in... | 08:22 |
| Jeffrey4l | i think we need some official info about this. | 08:22 |
| bjolo | yes | 08:23 |
| bjolo | we need to go through the rabbit change logs or something to get it confirmed | 08:23 |
| bjolo | and of course test it out | 08:23 |
| Jeffrey4l | yep | 08:23 |
| bjolo | just to be correct, i did not make this statement! ?) | 08:24 |
| bjolo | :) | 08:24 |
| bjolo | im just the messenger here :) | 08:24 |
| Satya_ | i see rabbitmq.config is having the IP but the cluster config having the hostnames | 08:24 |
| *** tovin07 has quit IRC | 08:24 | |
| Satya_ | and if we do a rabbitmqctl status it tries to connect to rabbit@hostname rather than rabbit@ip | 08:25 |
| bjolo | one thing i dont quite get about docker is that bind mounts actually seems to be a copy and mount | 08:30 |
| bjolo | for ex /etc/hostname | 08:30 |
| bjolo | docker inspect rabbitmq | grep hostname | 08:31 |
| bjolo | "HostnamePath": "/var/lib/docker/containers/23ee7bbeab8f701fd036dd5136408146d4d8622e91cc686896c2a790be40ed49/hostname", | 08:31 |
| bjolo | changing /etc/hosts on the host does not change /etc/hosts in the container | 08:31 |
| bjolo | and vice versa | 08:31 |
| Jeffrey4l | bjolo, it use different hosts file, too. like hostname | 08:31 |
| Jeffrey4l | "HostnamePath": "/home/docker/containers/54b3ede09a65a161ae4274852424663634f31ac8c87e62248dc07ef7b0df3923/hostname", | 08:32 |
| Jeffrey4l | "HostsPath": "/home/docker/containers/54b3ede09a65a161ae4274852424663634f31ac8c87e62248dc07ef7b0df3923/hosts", | 08:32 |
| bjolo | yes, also a copy and mount | 08:33 |
| Jeffrey4l | in this way, the container can change the hostname and hosts file without affecting the host og. | 08:33 |
| Jeffrey4l | host os. | 08:33 |
| bjolo | yes | 08:33 |
| bjolo | so manipulating hostname and /etc/hosts in the rabbit container *should* work in theory | 08:34 |
| bjolo | (if we still need to do that) | 08:34 |
| Jeffrey4l | --add-host parameter will change the hosts file in /etc/hosts/ | 08:34 |
| Jeffrey4l | the disadvantage is the host os 's /etc/hosts is override. | 08:34 |
| bjolo | or doing that way. That is the "how" question | 08:35 |
| Jeffrey4l | yep. | 08:35 |
| bjolo | what do you mean overide? | 08:35 |
| Jeffrey4l | bjolo, try this `docker run -it --rm -h abc --net host --add-host a:1.1.1.1 centos:7 cat /etc/hosts` | 08:36 |
| *** saneax-_-|AFK is now known as saneax | 08:36 | |
| Jeffrey4l | compare its output with your host os /etc/hosts | 08:36 |
| Jeffrey4l | and this `docker run -it --rm -h abc --net host centos:7 cat /etc/hosts ` | 08:36 |
| *** egonzalez90 has joined #openstack-kolla | 08:37 | |
| bjolo | hmm ok i see what you mean | 08:37 |
| Jeffrey4l | but i do not think it is matter in most of case. | 08:38 |
| bjolo | not sure | 08:38 |
| Satya_ | tried changing the /var/lib/docker/containers/23ee7bbeab8f701fd036dd5136408146d4d8622e91cc686896c2a790be40ed49/hostname | 08:39 |
| bjolo | all hosts should have their own hostname resolved in /etc/hosts is the general consensus. not sure for containers | 08:40 |
| Satya_ | when we bounce the container again the file get modified with the previous hostname | 08:40 |
| openstackgerrit | Javier Castillo Alcíbar proposed openstack/kolla-ansible: Fix ceilometer bootstrap https://review.openstack.org/398190 | 08:40 |
| Jeffrey4l | we should not change the hostname when using --net host. | 08:41 |
| Satya_ | hmm ok | 08:41 |
| Satya_ | will try that... | 08:41 |
| bjolo | i need to go. i work for a company that loves meetings :( | 08:46 |
| bjolo | Jeffrey4l, did you try a deploy with multiple external flat networks? | 08:46 |
| Jeffrey4l | bjolo, no. | 08:50 |
| Jeffrey4l | gtg. | 08:50 |
| bjolo | Jeffrey4l, please do if you have time. would be good to get my issue confirmed or not | 08:51 |
| bjolo | hmm cant find anything in rabbit change log about IP | 08:52 |
| bjolo | http://www.rabbitmq.com/clustering.html | 08:53 |
| bjolo | we are using the clusterer plugin right? | 08:53 |
| bjolo | pbourke, tried the vlan_ranges and suspected it did not work since bond0 is not dedicated to neutron | 08:55 |
| *** saneax is now known as saneax-_-|AFK | 08:55 | |
| *** athomas has joined #openstack-kolla | 08:56 | |
| bjolo | if im gonna go that route i need suspect i need to setup a bridge on bond0 and create tap/tun interface for neutron to use | 08:56 |
| bjolo | ok now i really need to go. meeting at 10 | 08:57 |
| *** tovin07 has joined #openstack-kolla | 08:57 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible: Modify the letter case with 'True' in ansible yml file https://review.openstack.org/398133 | 08:59 |
| *** saneax-_-|AFK is now known as saneax | 09:00 | |
| *** skramaja has joined #openstack-kolla | 09:06 | |
| *** skramaja_ has quit IRC | 09:06 | |
| *** saneax is now known as saneax-_-|AFK | 09:13 | |
| *** berendt has joined #openstack-kolla | 09:14 | |
| *** msimonin has joined #openstack-kolla | 09:20 | |
| *** tovin07 has quit IRC | 09:23 | |
| *** yingjun_ has quit IRC | 09:27 | |
| *** yingjun has joined #openstack-kolla | 09:27 | |
| *** yingjun has quit IRC | 09:27 | |
| *** chas has joined #openstack-kolla | 09:31 | |
| *** gfidente has joined #openstack-kolla | 09:32 | |
| Jeffrey4l | bjolo, ok i will try multi flat. | 09:35 |
| *** matrohon has joined #openstack-kolla | 09:36 | |
| *** DTadrzak has joined #openstack-kolla | 09:36 | |
| Jeffrey4l | and bond0 + vlan device should works. at least in our 100+ nodes test in osic, it works. | 09:36 |
| *** saneax-_-|AFK is now known as saneax | 09:38 | |
| *** berendt has quit IRC | 09:48 | |
| *** hfu has quit IRC | 09:51 | |
| *** chas has quit IRC | 09:52 | |
| *** zhangyufei has joined #openstack-kolla | 09:52 | |
| *** chas has joined #openstack-kolla | 09:52 | |
| magicboiz | Hi, I'm checking kolla with TLS enabled, and I think generated haproxy config is broken....I cannot access horizon, I cannot access nova_api.... | 09:53 |
| magicboiz | any ideas? | 09:53 |
| magicboiz | thx | 09:53 |
| *** alteriks has quit IRC | 09:53 | |
| *** alteriks has joined #openstack-kolla | 09:55 | |
| *** chas has quit IRC | 09:57 | |
| *** tovin07_ has quit IRC | 09:57 | |
| *** chas has joined #openstack-kolla | 09:58 | |
| openstackgerrit | Zeyu Zhu proposed openstack/kolla-ansible: Modify the letter case with 'True' in ansible yml file. https://review.openstack.org/398133 | 09:59 |
| *** saneax is now known as saneax-_-|AFK | 09:59 | |
| *** zhurong has quit IRC | 10:02 | |
| *** zhangyufei has quit IRC | 10:02 | |
| *** chas has quit IRC | 10:02 | |
| *** portdirect_away is now known as portdirect | 10:03 | |
| portdirect | morning all :) | 10:03 |
| egonzalez90 | morning portdirect | 10:04 |
| egonzalez90 | magicboiz: https://bugs.launchpad.net/kolla/+bug/1637962 | 10:04 |
| openstack | Launchpad bug 1637962 in kolla newton "SSL/TLS configuration not handled correctly" [Critical,Triaged] | 10:04 |
| *** chas has joined #openstack-kolla | 10:04 | |
| *** chas has quit IRC | 10:07 | |
| *** chas has joined #openstack-kolla | 10:07 | |
| *** Pavo has quit IRC | 10:07 | |
| magicboiz | egonzalez90: bingo, thx | 10:08 |
| *** berendt has joined #openstack-kolla | 10:10 | |
| egonzalez90 | are you using same network for external(TLS) and internal(http) | 10:10 |
| *** Satya_ has quit IRC | 10:12 | |
| *** Pavo has joined #openstack-kolla | 10:12 | |
| magicboiz | egonzalez90: yes | 10:15 |
| egonzalez90 | there is the problem then | 10:15 |
| portdirect | egonzalez90: is anyone workig on this bug atm? | 10:16 |
| egonzalez90 | is not asigned | 10:16 |
| magicboiz | egonzalez90: is there any haproxy limitation on this? | 10:17 |
| portdirect | magicboiz: no - its a config issue | 10:17 |
| magicboiz | portdirect: what you mean by config issue? | 10:17 |
| *** chas has quit IRC | 10:18 | |
| magicboiz | checking http://docs.openstack.org/developer/kolla/production-architecture-guide.html I think it's not clear at all | 10:19 |
| egonzalez90 | kolla_external_vip_interface - This interface is public-facing one. It’s used when you want HAProxy public endpoints to be exposed in different network than internal ones | 10:21 |
| portdirect | it's a bug with the way that haproxy/kolla is configured, I've not got a kolla setup here - so unfortunatly cant help you get it up and running this morning (GMT here), but once the usa wakes up you should be able to get some help, unless egonzalez90 has any input. | 10:22 |
| egonzalez90 | i dont think is a bug, if the same interface is defined to external and internal | 10:24 |
| egonzalez90 | haproxy will try to bind the port to the same IP | 10:24 |
| *** chas has joined #openstack-kolla | 10:24 | |
| egonzalez90 | which is not possible | 10:24 |
| portdirect | egonzalez90/magicboiz: whoops - missed this: are you using same network for external(TLS) and internal(http)? egonzalez90: yes | 10:25 |
| portdirect | sry - your right | 10:26 |
| egonzalez90 | maybe, a rework is needed to allow internal networks with TLS, but that is more a feature request than a bug | 10:26 |
| magicboiz | I think it would be enough to make it clear in docs: set public != internal in order to get haproxy external configuration. | 10:27 |
| egonzalez90 | a network architecture diagram would do it easier ;) | 10:28 |
| *** chas has quit IRC | 10:29 | |
| openstackgerrit | Merged openstack/kolla: Extract kolla exceptions into a single exception.py file https://review.openstack.org/396568 | 10:30 |
| egonzalez90 | is documented, but not in a place where people usually read | 10:30 |
| *** ppalacios has joined #openstack-kolla | 10:31 | |
| egonzalez90 | magicboiz: can you share what is in haproxy conf? | 10:33 |
| *** duonghq has quit IRC | 10:33 | |
| magicboiz | <gonzalez90: right now, I don't have any external haproxy config. I'm going to split public from internal in my lab | 10:35 |
| egonzalez90 | thats is because haproxy_enable_external_vip: "{{ 'no' if kolla_external_vip_address == kolla_internal_vip_address else 'yes' }}" | 10:36 |
| egonzalez90 | which means that if the external is the same as internal, haproxy wont configure external tls | 10:36 |
| *** kamal__ has quit IRC | 10:37 | |
| *** berendt has quit IRC | 10:40 | |
| magicboiz | gonzalez90: but doc says: It is mandatory to set this option when kolla_enable_tls_external is set to yes. Defaults to network_interface. | 10:48 |
| *** chas has joined #openstack-kolla | 10:49 | |
| *** zhubingbing has quit IRC | 10:49 | |
| *** zhubingbing has joined #openstack-kolla | 10:50 | |
| egonzalez90 | yes, i know what you mean, is in some kind bad explained. | 10:51 |
| magicboiz | anyway, it's to true that in a fully production environment, you'll find different networks for each service, but in LAB environments like mine.... :) | 10:51 |
| egonzalez90 | also, the same line says that is only used when different networks are present | 10:52 |
| magicboiz | yes your right. | 10:53 |
| *** zhangyufei has joined #openstack-kolla | 10:53 | |
| magicboiz | Maybe, the best option would be to set kolla_enable_tls_external=no if network==public. Because the mixed option doesn't work | 10:54 |
| *** chas has quit IRC | 10:55 | |
| *** msimonin has quit IRC | 10:57 | |
| *** chas has joined #openstack-kolla | 11:09 | |
| *** chas has quit IRC | 11:13 | |
| *** chas has joined #openstack-kolla | 11:15 | |
| *** athomas has quit IRC | 11:18 | |
| *** chas has quit IRC | 11:19 | |
| *** msimonin has joined #openstack-kolla | 11:20 | |
| *** chas has joined #openstack-kolla | 11:21 | |
| *** athomas has joined #openstack-kolla | 11:22 | |
| *** msimonin1 has joined #openstack-kolla | 11:22 | |
| *** msimonin has quit IRC | 11:22 | |
| *** msimonin1 has quit IRC | 11:22 | |
| *** chas has quit IRC | 11:25 | |
| *** mliima has joined #openstack-kolla | 11:29 | |
| *** berendt has joined #openstack-kolla | 11:33 | |
| *** Serlex has quit IRC | 11:33 | |
| *** hfu has joined #openstack-kolla | 11:44 | |
| *** chas has joined #openstack-kolla | 11:51 | |
| *** Serlex has joined #openstack-kolla | 11:52 | |
| sp_ | hi anyone please provide your views. What if we compare kolla-k8 with fuel then what are the key differences which is making our kolla-k8s better ?? | 11:53 |
| *** chas has quit IRC | 11:55 | |
| *** msimonin has joined #openstack-kolla | 11:56 | |
| coolsvap | egonzalez90: https://review.openstack.org/#/c/398108/ is on kolla-ansible :) | 11:57 |
| coolsvap | same for https://review.openstack.org/#/c/398107/ | 11:58 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible: Fix fact gathering when using --limit https://review.openstack.org/398313 | 11:58 |
| egonzalez90 | coolsvap: yes, i -1 on kolla | 11:58 |
| egonzalez90 | dont know how moved to kolla-ansible maintaining -1 | 11:59 |
| coolsvap | egonzalez90: no its -1 on kolla-ansible | 11:59 |
| egonzalez90 | i'll remove it | 11:59 |
| egonzalez90 | done | 12:01 |
| egonzalez90 | a question coolsvap, you know what is the proceidure to migrate code from kolla to kolla-ansible? changes made from friday are not in kolla-ansible | 12:04 |
| egonzalez90 | asked a couple of times.. | 12:04 |
| coolsvap | egonzalez90: since the snapshot was taken on friday sometime i believe, we need to submit the changes again | 12:05 |
| coolsvap | which are merged after friday | 12:05 |
| *** Pavo has quit IRC | 12:07 | |
| *** schwicht has joined #openstack-kolla | 12:09 | |
| *** coolsvap is now known as coolsvap_ | 12:09 | |
| *** shardy is now known as shardy_lunch | 12:11 | |
| *** Pavo has joined #openstack-kolla | 12:12 | |
| *** psanchez has quit IRC | 12:13 | |
| pbourke | can someone +a https://review.openstack.org/#/c/380028/ pelase | 12:13 |
| pbourke | *please | 12:14 |
| *** pc_m has quit IRC | 12:14 | |
| *** g3ek has quit IRC | 12:14 | |
| sp_ | pbourke: please provide your views. What if we compare kolla-k8 with fuel then what are the key differences which is making our kolla-k8s better ?? | 12:15 |
| sp_ | it would quite helpful in my understanding | 12:15 |
| *** psanchez has joined #openstack-kolla | 12:15 | |
| pbourke | sp_: sorry but I know nothing about fuel | 12:15 |
| *** pc_m has joined #openstack-kolla | 12:16 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Remove docker from kolla-ansible https://review.openstack.org/398320 | 12:17 |
| portdirect | sp_, I'll bite: very hard to answer, they are taking two very differenet approaches to deployment and mangement: fuel takes a very tradional puppet based approach to deployment vs kolla-kubernetes kubernetes approach. | 12:17 |
| *** g3ek has joined #openstack-kolla | 12:17 | |
| mliima | done pbourke | 12:18 |
| pbourke | mliima: thanks | 12:18 |
| egonzalez90 | not being mirantis is a key point? ;) | 12:18 |
| sp_ | pbourke: Thanks for quick response, but as per my study and understanding fuel also provide nearly same features as we are going to do with kolla-k8s | 12:18 |
| sp_ | portdirect: thanks for info | 12:18 |
| mliima | I don't understand why folks don't do it :/ | 12:18 |
| pbourke | sp_: http://stackalytics.com/?module=fuel-group | 12:19 |
| pbourke | sp_: if mirantis pull out that project is as good as dead | 12:19 |
| sp_ | portdirect: may i get more your views on this | 12:20 |
| openstackgerrit | Merged openstack/kolla: Unmount Ceph OSD disks as part of destroy https://review.openstack.org/380028 | 12:20 |
| sp_ | pbourke: ok !! I will take a look on this | 12:20 |
| pbourke | sp_: kolla has the diversity tag which means it is well supported across a wide community | 12:20 |
| *** schwicht has quit IRC | 12:21 | |
| sp_ | pbourke: do have some idea about our kolla-k8s operator part ? | 12:21 |
| egonzalez90 | sp_: I never liked comparisions, i prefer say analyse what they do and what are our weakness | 12:22 |
| sp_ | do you* | 12:22 |
| *** chas has joined #openstack-kolla | 12:22 | |
| pbourke | sp_: I have nothing to add over whats in the spec | 12:22 |
| *** msimonin has quit IRC | 12:22 | |
| portdirect | sp_: pbourke's point is very valid. I used to use fuel, but havent since juno - when I had some pretty *interesting* experiences, I've not touched it since then. | 12:22 |
| portdirect | sp_: So im afraid that I cant really add much more :/ | 12:22 |
| mliima | pbourke, we need to remove docker from kolla-ansible, ok? Can i register a bp for it? | 12:23 |
| pbourke | mliima: I think sdake already has a patch up for that | 12:23 |
| pbourke | mliima: actually I think his is the other way around | 12:23 |
| pbourke | removing ansible from the kolla part | 12:23 |
| egonzalez90 | yup | 12:23 |
| mliima | remove ansible folder? | 12:24 |
| mliima | haha | 12:24 |
| pbourke | mliima: have a look, feel free to register a bp for any pieces that arent done | 12:24 |
| sp_ | pbourke: portdirect: egonzalez90: every one, thanks a lot for having a great discussion on this. | 12:24 |
| *** chas has quit IRC | 12:26 | |
| *** berendt has quit IRC | 12:26 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible: Make service_checks easier to run independently of kolla-ansible https://review.openstack.org/398327 | 12:29 |
| *** schwicht has joined #openstack-kolla | 12:32 | |
| *** zhurong has joined #openstack-kolla | 12:35 | |
| *** berendt has joined #openstack-kolla | 12:40 | |
| *** duonghq has joined #openstack-kolla | 12:42 | |
| *** msimonin has joined #openstack-kolla | 12:43 | |
| *** shardy_lunch is now known as shardy | 12:45 | |
| *** berendt has quit IRC | 12:45 | |
| *** srwilkers has joined #openstack-kolla | 12:49 | |
| srwilkers | good morning | 12:51 |
| portdirect | morning srwilkers :) | 12:52 |
| pbourke | Jeffrey4l: any idea whats wrong with the release note gate on https://review.openstack.org/#/c/393409/ | 12:52 |
| *** chas has joined #openstack-kolla | 12:52 | |
| *** sdake has joined #openstack-kolla | 12:53 | |
| srwilkers | hey portdirect :D | 12:53 |
| portdirect | been playing with your k8s vagrant stuff - it's nice :) few q's though | 12:54 |
| portdirect | I could add fedora and centos atomic support if your up for a pr? | 12:54 |
| portdirect | also how are you finding romana? Just been playing iwht is this morning, and it seems pretty good but wondering if youve had much time on it? | 12:55 |
| srwilkers | we're always open to pull requests -- we merged centos support in the night before last, and we also added support for vagrant-libvirt | 12:56 |
| srwilkers | no complaints with romana so far, but we're working on getting some other SDN provider support added currently | 12:56 |
| sdake | morning | 12:57 |
| srwilkers | was actually going to use it as a base for a new multi-node getting started guide at some point. its what we're using for most of our development efforts currently | 12:57 |
| *** schwicht has quit IRC | 12:57 | |
| srwilkers | sup sdake | 12:57 |
| *** chas has quit IRC | 12:57 | |
| sdake | chillen | 12:57 |
| mliima | mornign sdake | 12:57 |
| *** schwicht has joined #openstack-kolla | 12:57 | |
| portdirect | morning sdake | 12:57 |
| *** chas has joined #openstack-kolla | 13:00 | |
| portdirect | srwilkers: yeah - 'm making use of both libvirt and centos (running 4.8 kernel on my dev boxen, so virtualbox is out for the moment :( ) - I've been using OVN for all of my k8s setups - so I'm up for throwing that in at some point, thogh I'm having a bit of head scratcher working out how to get it up and running with kubeadm so would need to sort that first :/ | 13:00 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Install neutron-fwaas plugin in neutron-base https://review.openstack.org/393409 | 13:01 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible: Allow neutron-fwaas to be enabled in Neutron https://review.openstack.org/398336 | 13:01 |
| *** schwicht has quit IRC | 13:01 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Install neutron-fwaas plugin in neutron-base https://review.openstack.org/393409 | 13:03 |
| srwilkers | yeah, thats a good point | 13:03 |
| *** msimonin has quit IRC | 13:04 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible: Allow neutron-fwaas to be enabled in Neutron https://review.openstack.org/398336 | 13:04 |
| *** sdake_ has joined #openstack-kolla | 13:05 | |
| sp_ | morning sdake: | 13:05 |
| *** msimonin has joined #openstack-kolla | 13:08 | |
| sdake_ | Jeffrey4l around? | 13:08 |
| *** sdake has quit IRC | 13:08 | |
| *** hfu has quit IRC | 13:13 | |
| *** rhallisey has joined #openstack-kolla | 13:15 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add trove role https://review.openstack.org/354901 | 13:16 |
| *** lamt has joined #openstack-kolla | 13:23 | |
| Jeffrey4l | sdake_, yep. | 13:23 |
| *** jheroux has joined #openstack-kolla | 13:24 | |
| Jeffrey4l | pbourke, checking | 13:25 |
| *** dave-mccowan has joined #openstack-kolla | 13:25 | |
| rhallisey | morning | 13:26 |
| sdake_ | Jeffrey4l sup dude, did you sort out how to make a mgmt interface? | 13:27 |
| sdake_ | Jeffrey4l or do we need a blueprint for that | 13:27 |
| Jeffrey4l | we don't. | 13:27 |
| Jeffrey4l | we do not need mgnt interface. | 13:27 |
| Jeffrey4l | now, the api_interface in kolla is the mgmt interface. | 13:28 |
| sdake_ | what do the apis themselves bind to? | 13:28 |
| Jeffrey4l | if we do not want to export the api_interface to extenal/internet. | 13:28 |
| Jeffrey4l | we can enable the externel ip interface, | 13:28 |
| Jeffrey4l | which only forwards the necessary port. | 13:28 |
| *** schwicht has joined #openstack-kolla | 13:29 | |
| sdake_ | dave-mccowan around? | 13:30 |
| dave-mccowan | sdake_ ack | 13:30 |
| sdake_ | dave-mccowan do you typically bind rabbitmq/mariadb to the api interface in openstack? | 13:31 |
| sdake_ | from a security pov | 13:31 |
| Jeffrey4l | sdake_, i recommended kamal to enable external vip feature. | 13:31 |
| *** msimonin has quit IRC | 13:31 | |
| dave-mccowan | the names internal, external, management, and api get mixed up sometimes. | 13:33 |
| *** msimonin has joined #openstack-kolla | 13:33 | |
| *** zhubingbing has quit IRC | 13:33 | |
| dave-mccowan | i recommend two vips: the internal is for mariadb, rabbitmq, and service-to-service connections. | 13:33 |
| dave-mccowan | the external is for humans to connect to the dashboard and service REST APIs. | 13:34 |
| dave-mccowan | the internal VIP does not use TLS/HTTPS, so it needs to be kept locked down. no routes in or out. | 13:34 |
| sdake_ | dave-mccowan ok we have that in kolla now | 13:34 |
| sdake_ | dave-mccowan what we also have is mariadb/rabbitmq binding to api_interface_address | 13:35 |
| *** schwicht has quit IRC | 13:37 | |
| dave-mccowan | in kolla, the api_interface is the internal one. | 13:38 |
| dave-mccowan | so, yes that's normal for support services to the be on the internal network. | 13:39 |
| *** zhangyufei has quit IRC | 13:39 | |
| Jeffrey4l | pbourke, no idea. it works fine locally. | 13:41 |
| Jeffrey4l | sdake_, how can we backport the ansible change after the repo is split? | 13:43 |
| sdake_ | Jeffrey4l inc0 said he was going to make a script to do it | 13:44 |
| sdake_ | Jeffrey4l but i'm skeptical of script approach | 13:44 |
| sdake_ | Jeffrey4l I don't have a good answer for ansible change backports other then git format-patch followed by git am :( | 13:44 |
| sdake_ | Jeffrey4l there is probably a git wizard out there that can make it happen | 13:45 |
| sdake_ | i am not keen on the repo split for a whole lot of reasons, the pragmatic "how the hell do we do backports" one was never answered | 13:45 |
| sdake_ | but we agreed to it as a team, so I got it kicked off | 13:45 |
| sdake_ | you see, when we agree to something as a team, I get behind it ;) | 13:45 |
| sdake_ | even if I dislike it :( | 13:46 |
| Jeffrey4l | ( i do not like, too ;( ) | 13:47 |
| Jeffrey4l | sync to git repo tools? maybe | 13:47 |
| Jeffrey4l | sync to/sync two. | 13:48 |
| *** Administrator_ has quit IRC | 13:51 | |
| *** Administrator_ has joined #openstack-kolla | 13:52 | |
| sdake_ | Jeffrey4l i think the first thing we need to do is focus on the gates | 13:52 |
| sdake_ | Jeffrey4l and sort out backports later | 13:52 |
| Jeffrey4l | yes. | 13:52 |
| sdake_ | Jeffrey4l do you ahve an immediate need to sort out backports now? | 13:52 |
| Jeffrey4l | no. but want to know how to backport. | 13:53 |
| Jeffrey4l | a quick search http://stackoverflow.com/questions/3816040/git-apply-changes-introduced-by-commit-in-one-repo-to-another-repo | 13:53 |
| sdake_ | Jeffrey4l what i'd suggest is sstarting a therad on ml | 13:55 |
| sdake_ | Jeffrey4l we got a whole bunch of git nerds in the community who can hepl | 13:55 |
| sdake_ | alot of people follow the [kolla] tag | 13:55 |
| sdake_ | more then just the core team | 13:55 |
| Jeffrey4l | good idea. | 13:55 |
| sdake_ | ou might tag it [git] too | 13:55 |
| Jeffrey4l | roger. | 13:55 |
| *** fguillot has joined #openstack-kolla | 13:58 | |
| *** zhubingbing has joined #openstack-kolla | 14:04 | |
| *** Pavo has quit IRC | 14:07 | |
| *** Pavo has joined #openstack-kolla | 14:12 | |
| *** Administrator_ has quit IRC | 14:13 | |
| *** Administrator_ has joined #openstack-kolla | 14:14 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Allow operators to use 'fallback mode' for Ceph disks https://review.openstack.org/384496 | 14:14 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible: Allow operators to use 'fallback mode' for Ceph disks https://review.openstack.org/398373 | 14:15 |
| v1k0d3n | morning all | 14:16 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Add a 'fallback mode' to disable udev in find_disks https://review.openstack.org/384496 | 14:16 |
| openstackgerrit | caoyuan proposed openstack/kolla: Remove the releasenotes about the ansible https://review.openstack.org/398374 | 14:17 |
| srwilkers | morning v1k0d3n | 14:18 |
| v1k0d3n | portdirect_away_: would love to have atomic and fedora in there. | 14:18 |
| v1k0d3n | we originally went with what kubeadm drafted for support, but welcome anything else you have in mind for PR's | 14:18 |
| v1k0d3n | i have some input on romana as well. been working closely with those guys, so ping me when you get back. | 14:19 |
| v1k0d3n | i saw some updates that lucas made to kubeadm, to allow it to run on CoreOS, so i was going to add that soon as well. | 14:19 |
| v1k0d3n | morning srwilkers | 14:19 |
| v1k0d3n | portdirect_away_: and to be fair...kubeadm is really new. a lot of features need to be added in order to make it "prod-ready". but we're sure that it's also the way forward; for building and managing clusters, which is why we wrote playbooks for it. | 14:21 |
| v1k0d3n | and to be clear, the vagrant demo we have just uses various providers. so we use vagrant as the mechanism to deploy to local vbox or libvirt, but you can also deploy straight to an openstack cloud, and i was going to add AWS soon as well. | 14:22 |
| v1k0d3n | i wanted it to be flexible...to deploy anywhere a developer needs, so it just drops right in. | 14:23 |
| v1k0d3n | or you can just use the ansible playbooks. whatever works. | 14:23 |
| *** lamt has quit IRC | 14:23 | |
| sdake_ | Jeffrey4l so, who is leading the merry effort of the repo split | 14:24 |
| sdake_ | Jeffrey4l is that you? | 14:24 |
| Jeffrey4l | i do know ;( | 14:24 |
| sdake_ | you do know? | 14:24 |
| sdake_ | or not know | 14:24 |
| Jeffrey4l | do not | 14:24 |
| Jeffrey4l | sorry. | 14:24 |
| sdake_ | ok, well it can't be me unfortunately | 14:24 |
| Jeffrey4l | btw, i am working on the gate right now. | 14:25 |
| sdake_ | cool | 14:25 |
| sdake_ | perhaps inc0 can provide some technical leadersihp here | 14:25 |
| zhubingbing | －） | 14:25 |
| Jeffrey4l | yep. | 14:25 |
| *** eaguilar has joined #openstack-kolla | 14:25 | |
| sdake_ | Jeffrey4l i know that fungi is right about submodules | 14:25 |
| sdake_ | Jeffrey4l (re ml) | 14:26 |
| sdake_ | and the key thing I'd go for is cross-repo gating | 14:26 |
| sdake_ | I hadn't even thought of that until fungi brought it up | 14:26 |
| Jeffrey4l | are u going to use submodule? | 14:26 |
| sdake_ | its more expensive in terms of gate resources | 14:26 |
| sdake_ | fungi recommends strongly against it | 14:26 |
| sdake_ | fungi is infra ptl and a pro at how all this stuff works | 14:26 |
| sdake_ | take his word as gospel | 14:26 |
| sdake_ | in fact any of the infra folks word is pretty much fospel | 14:27 |
| sdake_ | gospel to me ;) | 14:27 |
| sdake_ | that whole team rocks | 14:27 |
| sdake_ | they have helped us out so much, its not even funny | 14:27 |
| sdake_ | Jeffrey4l there is a msg on the ml from jeremy | 14:28 |
| sdake_ | (fungi) | 14:28 |
| sdake_ | Jeffrey4l he suggests what to do | 14:28 |
| sdake_ | use zuul-cloner | 14:28 |
| Jeffrey4l | OK. i saw it. | 14:28 |
| Jeffrey4l | let me check it carefully. | 14:28 |
| sdake_ | there is an example of zuul-cloner usage in the gate scripts of kolla-kubernetes | 14:28 |
| Jeffrey4l | OK | 14:28 |
| *** dave-mccowan has quit IRC | 14:30 | |
| v1k0d3n | big fan of submodules. we use them. | 14:31 |
| v1k0d3n | not sure how well that would work here. perhaps fine. | 14:31 |
| Jeffrey4l | v1k0d3n, any benefit? | 14:31 |
| v1k0d3n | has anyone worked with subtrees? i hear that in some ways it can be better. | 14:31 |
| *** jtriley has joined #openstack-kolla | 14:31 | |
| v1k0d3n | well, yeah...with submodules the projects can act completely independent of each other. that's extremely nice. | 14:32 |
| v1k0d3n | Jeffrey4l: have you used them before? | 14:32 |
| v1k0d3n | we use them for our kube develop environment that srwilkers was talking about earlier. | 14:33 |
| v1k0d3n | https://github.com/att-comdev/halcyon-vagrant-kubernetes | 14:33 |
| Jeffrey4l | no ;( | 14:33 |
| *** schwicht has joined #openstack-kolla | 14:33 | |
| v1k0d3n | keeps the footprint very low and extremely targeted on our project. | 14:34 |
| v1k0d3n | in our case, we used submodule for this vagrant/kube ansible playbooks because people thought that vagrant was for local development only...which was a bad misunderstanding. | 14:34 |
| v1k0d3n | so creating the submodule separation makes the goals more clear to users | 14:35 |
| sdake_ | v1k0d3n reference mailing list where jeremy stanley says "submodules bad" :) | 14:35 |
| v1k0d3n | i see a good reason for using them when separating these projects honestly. | 14:35 |
| v1k0d3n | yeah, not saying it's the best solution ever... | 14:36 |
| Jeffrey4l | zuul-cloner should be the better solution. kolla and kolla-ansible is not kind of submodule project. | 14:37 |
| *** inc0 has joined #openstack-kolla | 14:37 | |
| inc0 | good morning | 14:37 |
| Jeffrey4l | zuul-cloner should allow us to use depends-on directive. | 14:37 |
| inc0 | how are is my favorite community today:) | 14:38 |
| srwilkers | inc0, not caffeinated enough | 14:38 |
| inc0 | ahh the hard stuff | 14:38 |
| inc0 | Jeffrey4l, so I was thinking | 14:38 |
| inc0 | we have 2 options | 14:38 |
| inc0 | 1. in kolla-ansible zuul-clone kolla | 14:38 |
| inc0 | that will give us build gates before we create docker registry | 14:39 |
| inc0 | but on the other hand, if we use submodule in kolla to clone both kolla-ansible and kolla-k8s | 14:39 |
| sdake_ | inc0 i think the key point of using zuul cloner from fungi's msg was that we could do cross-repo gating | 14:39 |
| inc0 | yes | 14:40 |
| sdake_ | inc0 i.e. kolla repo could gate both kolla-ansible and kolla-kubernetes | 14:40 |
| inc0 | with latter users will be able to git clone --recursive kolla and have orch tools in place | 14:40 |
| Jeffrey4l | i'd like to use option 1 inc0 | 14:40 |
| Jeffrey4l | if we use submodule, why we split repo? | 14:41 |
| inc0 | well, we'll have all the benefits of repo split but loose discomfort for users of it | 14:41 |
| *** rhallisey has quit IRC | 14:41 | |
| *** lamt has joined #openstack-kolla | 14:41 | |
| *** rhallisey has joined #openstack-kolla | 14:41 | |
| sdake_ | submodule is pretty much incompatible with gerrit | 14:41 |
| sdake_ | i keep telling people this... | 14:42 |
| sdake_ | and people keep on not listening... | 14:42 |
| Jeffrey4l | but submodule tie these three repos into one, imho. | 14:42 |
| v1k0d3n | ok. totally fine if something doesn't work. so let's find something that would. | 14:42 |
| inc0 | submodule will not need use gerrit at all | 14:42 |
| v1k0d3n | submodule is just a standard git function | 14:43 |
| inc0 | kolla will have build gates, period | 14:43 |
| v1k0d3n | (which i know we realize, im sure) | 14:43 |
| inc0 | no need of gerrit magics | 14:43 |
| v1k0d3n | probably stating obvious. | 14:43 |
| sdake_ | inc0 i think kolla should also gate on kolla-kube and kolla-ansible | 14:43 |
| inc0 | but people will still be able to git clone --recursive locally and not deal with 3 different project instal | 14:43 |
| inc0 | s | 14:43 |
| sdake_ | that was BKP from fungi | 14:43 |
| Jeffrey4l | inc0, i am adding load murano dynamic, how can i test this in ci with submodule? ( the kolla-ansible and kolla will both change ) | 14:44 |
| inc0 | we can use both | 14:44 |
| Jeffrey4l | submodule is hard for cross-repo ci test | 14:44 |
| inc0 | submodule and zuul cloner to do effectively same thing | 14:44 |
| inc0 | one for gerrit another one for users | 14:44 |
| *** TxGirlGeek has joined #openstack-kolla | 14:45 | |
| sdake_ | inc0 recommend run that by infra :) | 14:45 |
| Pavo | morning gents | 14:45 |
| sdake_ | inc0 just to make sure it doesn't break anything | 14:45 |
| sdake_ | sup pavo | 14:45 |
| sdake_ | inc0 you notice i ran every step of the repo split by infra ;) | 14:45 |
| sdake_ | so it iddn't get f'ed up ;) | 14:45 |
| Pavo | nothing much | 14:45 |
| *** schwicht has quit IRC | 14:45 | |
| *** mgoddard_ has joined #openstack-kolla | 14:46 | |
| sdake_ | and actually ajeager gave the most simple answer to the problem ;) | 14:46 |
| inc0 | sdake_, totally, I just want to have idea agreed here before we validate it with them | 14:46 |
| inc0 | in interest of their time | 14:46 |
| inc0 | I know lots of projects uses submodule heavily | 14:46 |
| sdake_ | oh didn't know othe projects use that | 14:46 |
| inc0 | OpenStack Ansible of Fuel being examples | 14:46 |
| sdake_ | guidance i've heard from infra in the past is avoid submodule | 14:47 |
| sdake_ | i dont know if it would be appropriate in this particular case | 14:47 |
| *** chas has quit IRC | 14:47 | |
| inc0 | yeah remember this fuel shitshow of every Dockerfile in dedicated repo? | 14:47 |
| sdake_ | thats where this is coming from - past discussions | 14:47 |
| inc0 | this is how they'd bundle it together | 14:47 |
| inc0 | in OSA every role is separate proejct | 14:47 |
| sdake_ | i'll be completely honest, I have no idea what the hell submodule does | 14:47 |
| *** lrensing has joined #openstack-kolla | 14:47 | |
| *** chas has joined #openstack-kolla | 14:47 | |
| sdake_ | all I know is infra doesn't like it ;) | 14:47 |
| sdake_ | infra doens't like - i don't like | 14:48 |
| inc0 | when you git clone repo it will git clone another repo inside your cloned repo | 14:48 |
| sdake_ | guess a bit of cargo cult thinking there | 14:48 |
| inc0 | as simple as that | 14:48 |
| v1k0d3n | i probably missed the reasons for not using submodules. what are the reasons submodules don't work again? | 14:48 |
| *** rhallisey has quit IRC | 14:48 | |
| *** rhallisey1 has joined #openstack-kolla | 14:48 | |
| Jeffrey4l | inc0, fuel and openstack-ansible use independent ansible role and puppet module, so submodule should be better. | 14:48 |
| sdake_ | v1k0d3n not sure - its been awhile since i had the discussion with infra | 14:48 |
| inc0 | infra said that zuul cloner is better for infra, allows us to use cross repo gates and whatnot | 14:48 |
| Jeffrey4l | but kolla is not. | 14:48 |
| sdake_ | v1k0d3n join #opnestack-infra | 14:48 |
| Jeffrey4l | kolla-ansible only use kolla's image. | 14:49 |
| *** mgoddard has quit IRC | 14:49 | |
| Jeffrey4l | there is no need ti tie these two repos into one by using submodule. | 14:49 |
| inc0 | Jeffrey4l, agree, but it would remove a step from kolla deployment | 14:49 |
| v1k0d3n | so perhaps we make a hard split with kolla and kolla-kube then, if submodules don't work. | 14:49 |
| inc0 | I think it'd be better for users | 14:49 |
| v1k0d3n | they are different projects, using different methods of deployment and templates for deployments. | 14:50 |
| Jeffrey4l | for end-user, build image and deploy image should in different nodes. | 14:50 |
| v1k0d3n | entirely different approach, workflow, everything except containers. | 14:50 |
| inc0 | Jeffrey4l, agree, could be, actually it's a good point | 14:50 |
| *** schwicht has joined #openstack-kolla | 14:50 | |
| inc0 | we could refactor our docs to have deploy node and build node | 14:50 |
| Jeffrey4l | and we can provide kolla rpm/deb package for better usage experience | 14:50 |
| inc0 | as opposed to just deploy node | 14:50 |
| sdake_ | our docs need moar love anyway | 14:51 |
| inc0 | apt-get install kolla-ansible? now that's interesting | 14:51 |
| Jeffrey4l | something like that. | 14:51 |
| inc0 | operators would love that | 14:51 |
| *** zhurong has quit IRC | 14:51 | |
| *** TxGirlGeek has quit IRC | 14:51 | |
| portdirect | github down - can i go home now? | 14:51 |
| *** tonanhngo has joined #openstack-kolla | 14:51 | |
| sdake_ | portdirect your dismissed ;-) | 14:52 |
| *** TxGirlGeek has joined #openstack-kolla | 14:52 | |
| inc0 | again? | 14:52 |
| inc0 | they just had big DDoS | 14:52 |
| inc0 | did east coast learned nothing> | 14:52 |
| portdirect | looks like it - who they piss of this time? :) | 14:52 |
| v1k0d3n | inc0: and Jeffrey4l what's the difference with apt vs pip install kolla-ansible? | 14:53 |
| dims | inc0 : i felt it too ("You can't do this operation at this time") | 14:53 |
| inc0 | v1k0d3n, nothing much, just operators love apt | 14:53 |
| egonzalez90 | not deal with pip dependencies | 14:54 |
| *** rhallisey1 has left #openstack-kolla | 14:54 | |
| v1k0d3n | so to be clear, we are only talking about for the kolla-ansible portion? | 14:54 |
| portdirect | inc0: nothing much, just operators love yum (FTFY) | 14:54 |
| v1k0d3n | ^^ and there it is. | 14:55 |
| v1k0d3n | can we just say that operators love whatever they are familiar with? | 14:55 |
| *** lrensing has quit IRC | 14:55 | |
| egonzalez90 | i would add kolla and kolla-ansible(for now), rdo and ubuntu will not build non prod packages | 14:55 |
| Jeffrey4l | yes another solution is add kolla into kolla-ansible' requirements.txt file. | 14:55 |
| egonzalez90 | they don't even build some prod services | 14:56 |
| Jeffrey4l | then pip works | 14:56 |
| Jeffrey4l | only the guys who install from git url need clone two repo. | 14:56 |
| Jeffrey4l | i think that's fine. | 14:56 |
| *** lrensing has joined #openstack-kolla | 14:56 | |
| *** adrian_otto has joined #openstack-kolla | 14:57 | |
| *** rmart04 has joined #openstack-kolla | 14:58 | |
| *** mgoddard has joined #openstack-kolla | 14:58 | |
| *** sdake has joined #openstack-kolla | 14:58 | |
| *** mgoddard_ has quit IRC | 14:59 | |
| sdake | inc0 this needs a +1 https://review.openstack.org/396901 | 14:59 |
| sdake | inc0 my governance repo change had a merge conflict | 14:59 |
| *** coolsvap_ is now known as coolsvap | 14:59 | |
| *** rhallisey has joined #openstack-kolla | 14:59 | |
| inc0 | done | 14:59 |
| *** schwicht has quit IRC | 14:59 | |
| sdake | thanks | 15:00 |
| sdake | btw, today is release day | 15:00 |
| sdake | just throwing that out there ;-) | 15:00 |
| Jeffrey4l | ok. | 15:00 |
| Jeffrey4l | using the latest commit hash, right? | 15:01 |
| sdake | i think its typicallybad to release a DOA repo | 15:01 |
| sdake | ;) | 15:01 |
| sdake | Jeffrey4l persoanlly I'd go back prior to the split for kolla hash | 15:01 |
| sdake | and for kolla-ansible use current hash | 15:01 |
| sdake | since that should be good i think? | 15:01 |
| Jeffrey4l | ok. | 15:02 |
| *** sdake_ has quit IRC | 15:02 | |
| sdake | not sure story on kolla-kubernetes tag | 15:02 |
| Jeffrey4l | release kolla ansible kolla-ansible in one PS or two? | 15:02 |
| sdake | Jeffrey4l each one has to be separate | 15:02 |
| Jeffrey4l | roger. | 15:02 |
| sdake | because of the release team's processes are sort of broken | 15:02 |
| sdake | not broken | 15:03 |
| sdake | just not evolved to handle tha t case yet | 15:03 |
| sdake | join 3openstack-release and ask there | 15:03 |
| sdake | maybe they have fixed that | 15:03 |
| *** magicboiz has quit IRC | 15:03 | |
| sdake | actually i think the release team's processes are fantastic | 15:03 |
| inc0 | sdake, when your change to govenrance repo merges, I'll make Jeffrey a release liason | 15:03 |
| sdake | but some of their tools don't handle multiple commits and stuff | 15:03 |
| sdake | inc0 roger - dont know if that iwll be today then | 15:04 |
| sdake | inc0 but we have to tag today i think | 15:04 |
| Jeffrey4l | hmm, let me push the PS first. | 15:04 |
| inc0 | ok, I'll put his name now | 15:05 |
| *** msimonin has quit IRC | 15:05 | |
| inc0 | people knows you anyway sdake | 15:05 |
| sdake | inc0 you do have a point tho, not sure if release team will release somethign that isn't a deliverable in the gov repo | 15:05 |
| sdake | best to ask ;) | 15:05 |
| *** adrian_otto has quit IRC | 15:06 | |
| inc0 | Jeffrey4l, congrats, you're hereby our release manager:) https://wiki.openstack.org/wiki/CrossProjectLiaisons#Release_management | 15:08 |
| Jeffrey4l | inc0, thanks ;) | 15:08 |
| inc0 | don't thank me, I just gave you new job nobody will pay for;) | 15:09 |
| inc0 | but on the other hand, thanks Jeffrey for taking on it:) | 15:09 |
| sdake | Jeffrey4l = winning;) | 15:11 |
| *** msimonin has joined #openstack-kolla | 15:12 | |
| Jeffrey4l | ;) | 15:12 |
| *** msimonin has quit IRC | 15:13 | |
| *** schwicht has joined #openstack-kolla | 15:13 | |
| inc0 | sdake, could you coach Jeffrey4l on his new job? :) | 15:13 |
| sdake | i tihnk i trained him on this process already | 15:13 |
| sdake | Jeffrey4l could you confirm or deny? | 15:14 |
| sdake | i trained you coolsvap ryan and jefrey4l iirc | 15:14 |
| inc0 | and release 4.0.0b1 on all projects as soon as it's ready? | 15:14 |
| sdake | 4.0.0.0b1 | 15:14 |
| Jeffrey4l | yep. | 15:14 |
| inc0 | yeah, I did it once, but I can't remember full process tbh | 15:14 |
| *** msimonin has joined #openstack-kolla | 15:14 | |
| Jeffrey4l | i can handle this. | 15:14 |
| inc0 | ok, I'll leave it to you guys | 15:14 |
| inc0 | thanks | 15:14 |
| sdake | Jeffrey4l if you need any help reach out :) | 15:14 |
| Jeffrey4l | 1. push release PS in openstack/release, 2. make new milestone and move all old bug in o-1 into o-2. right? sdake | 15:15 |
| sdake | thats what i've been doing | 15:15 |
| sdake | i'm not sure its the smartest approach | 15:15 |
| sdake | we hae like 350 open bugs | 15:15 |
| sdake | that model used to work when we had less bugs | 15:16 |
| *** yingjun has joined #openstack-kolla | 15:16 | |
| sdake | seing 300+ bugsi n o2 is just... depressing | 15:16 |
| sdake | sort of stuck on how to ramp up our triage efforts | 15:16 |
| coolsvap | sdake: i tried to use the release tools script but i was not able to get that working to bulk move bugs | 15:16 |
| Jeffrey4l | hmm.. maybe need some scripts. | 15:16 |
| Jeffrey4l | coolsvap, where is it? | 15:17 |
| sdake | coolsvap cool dude that would have rocked if that worked thanks for tyring :) | 15:17 |
| Jeffrey4l | the scripts. | 15:17 |
| coolsvap | i believe it works on ubuntu | 15:17 |
| sdake | Jeffrey4l its also more complex because we have a kolla-ansible launchpad as well | 15:17 |
| *** msimonin has quit IRC | 15:17 | |
| coolsvap | but not sure | 15:17 |
| *** msimonin has joined #openstack-kolla | 15:17 | |
| Jeffrey4l | coolsvap, never mind. could u give me the script link? | 15:18 |
| coolsvap | Jeffrey4l: just a min | 15:18 |
| Jeffrey4l | sdake, yes. and some bugs may need split into two, one for kolla and another for kolla-ansible. ;( | 15:18 |
| sdake | Jeffrey4l recommend solving one problem at a time ;) | 15:19 |
| sdake | i guess bulk move for now | 15:19 |
| sdake | perhaps inc0 can lead a discussion on how to do bug triage the right way | 15:19 |
| *** lrensing has quit IRC | 15:19 | |
| sdake | or rather bug management | 15:19 |
| sdake | we seem to be triaging ok | 15:19 |
| inc0 | << reading about launchpad API >> | 15:20 |
| coolsvap | Jeffrey4l: https://github.com/openstack-infra/release-tools/blob/master/process_bugs.py | 15:20 |
| Jeffrey4l | coolsvap, cool. | 15:20 |
| sdake | Jeffrey4l that only works on ubuntu as coolsvap said | 15:20 |
| sdake | i've tried to use that lib that is a dep of process_bugs on centos before - no go | 15:20 |
| Jeffrey4l | that's OK. i can set up a ubuntu container. ;) | 15:21 |
| Jeffrey4l | will try it. | 15:21 |
| sdake | cool :) | 15:21 |
| Jeffrey4l | let me push the release patch set. | 15:21 |
| inc0 | I think we could copy all the bugs from kolla to kolla ansible (also affects) | 15:21 |
| sdake | can't do patch sets | 15:21 |
| *** yingjun has quit IRC | 15:21 | |
| inc0 | and just invalidate those which are clearly ansible or kolla | 15:21 |
| inc0 | I don't expect too many of those | 15:21 |
| *** yingjun has joined #openstack-kolla | 15:21 | |
| Jeffrey4l | so one for kolla, one for kolla-ansible, how about the kolla-k8s? | 15:21 |
| v1k0d3n | Jeffrey4l: yes | 15:22 |
| v1k0d3n | inc0: at least i think...right? | 15:23 |
| coolsvap | i moved most blueprints to kolla-ansible | 15:23 |
| inc0 | coolsvap, thanks | 15:23 |
| sdake | coolsvap manually? | 15:23 |
| coolsvap | i think we can move bugs as we review patchsets | 15:23 |
| coolsvap | sdake: yes | 15:23 |
| * sdake comforts coolsvap | 15:24 | |
| sdake | ya that manual bug moving thing sucks | 15:24 |
| sdake | karma ++ :) | 15:24 |
| inc0 | coolsvap, yeah it might be better | 15:25 |
| inc0 | just slowly wait till system handles itself;) | 15:25 |
| *** absubram has joined #openstack-kolla | 15:25 | |
| inc0 | friendly reminder, meeting in circa 30min | 15:27 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix congress broken https://review.openstack.org/398413 | 15:29 |
| sdake | inc0 mind updating the topic for devs with something like "Ansible code should go to repo http://github.com/openstack/kolla-ansible" | 15:29 |
| sdake | so I don't hae to keep -2ing these reviews ;) | 15:30 |
| sdake | 12 pages of reviews for kolla :( | 15:33 |
| sdake | everything backed up | 15:33 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Clean up tox.ini for Ansible removal https://review.openstack.org/398028 | 15:33 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Remove Ansible from Kolla https://review.openstack.org/398025 | 15:33 |
| srwilkers | sorry, i try to review as much as I can throughout the day -- not as much as I'd like to recently | 15:34 |
| *** mgiles has joined #openstack-kolla | 15:35 | |
| sdake | srwilkers its ok, most of the reviews need to be redone anyway | 15:35 |
| sdake | srwilkers as a result of the repo split | 15:35 |
| srwilkers | yeah, thats what i was anticipating | 15:35 |
| sdake | fwiw, I did the work on saturday :) | 15:36 |
| sdake | and it landed on tuesday | 15:36 |
| sdake | to minimize disruption | 15:36 |
| sdake | but this type of disruption can't be resolved easily | 15:36 |
| *** ChanServ sets mode: +o inc0 | 15:37 | |
| srwilkers | whats life without a little disruption | 15:37 |
| inc0 | New to Kolla? Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA); w00t for Kolla!; Kolla-ansible is up! All ansible changes goes there now | 15:38 |
| *** inc0 changes topic to "New to Kolla? Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b (old logs from #kolla http://goo.gl/VKpPzA); w00t for Kolla!; Kolla-ansible is up! All ansible changes goes there now" | 15:38 | |
| *** jtriley has quit IRC | 15:38 | |
| *** inc0 sets mode: -o inc0 | 15:38 | |
| *** rhallisey has quit IRC | 15:43 | |
| *** rhallisey has joined #openstack-kolla | 15:43 | |
| *** jtriley has joined #openstack-kolla | 15:46 | |
| *** coolsvap is now known as coolsvap_ | 15:46 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: install murano dashboard package in horizon https://review.openstack.org/395957 | 15:56 |
| *** rhallisey has left #openstack-kolla | 16:00 | |
| Pavo | man they need to add an option to network boot instances in openstack | 16:00 |
| *** coolsvap_ is now known as coolsvap | 16:00 | |
| *** rhallisey__ has joined #openstack-kolla | 16:00 | |
| v1k0d3n | meeting started now? | 16:01 |
| egonzalez90 | yup v1k0d3n | 16:01 |
| sdake | oh | 16:04 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Remove docker folder from kolla-ansible https://review.openstack.org/398320 | 16:04 |
| *** dave-mccowan has joined #openstack-kolla | 16:04 | |
| *** DTadrzak has quit IRC | 16:07 | |
| *** Pavo has quit IRC | 16:07 | |
| *** Pavo has joined #openstack-kolla | 16:08 | |
| *** yingjun has quit IRC | 16:08 | |
| *** TxGirlGeek has quit IRC | 16:09 | |
| *** rhallisey__ is now known as rhallisey | 16:17 | |
| *** rhallisey1 has joined #openstack-kolla | 16:18 | |
| *** rhallisey1 has left #openstack-kolla | 16:22 | |
| *** adrian_otto has joined #openstack-kolla | 16:26 | |
| *** krtaylor has quit IRC | 16:29 | |
| *** magicboiz has joined #openstack-kolla | 16:31 | |
| *** TxGirlGeek has joined #openstack-kolla | 16:32 | |
| *** rmart04 has quit IRC | 16:39 | |
| *** krtaylor has joined #openstack-kolla | 16:43 | |
| *** strigazi is now known as strigazi_AFK | 16:46 | |
| *** jistr is now known as jistr|biab | 16:48 | |
| *** ppalacios has quit IRC | 16:49 | |
| pbourke | sorry missed meeting, need to update my calendar for the clock change | 16:51 |
| *** absubram has quit IRC | 16:53 | |
| *** portdirect is now known as portdirect_away | 16:53 | |
| *** ppalacios has joined #openstack-kolla | 16:53 | |
| wirehead_ | Yeah, it keeps messing me up as well. | 16:53 |
| wirehead_ | I'm a huge fan of outlawing daylight savings time in all forms. | 16:53 |
| wirehead_ | Also: https://xkcd.com/673/ | 16:55 |
| duonghq | anybody got external_ip of k8s service holding "pending" state? | 16:59 |
| Jeffrey4l | rhallisey, what's the release status for kolla-k8s? can we release it now? | 16:59 |
| *** zhubingbing has quit IRC | 17:00 | |
| rhallisey | Jeffrey4l, I don't understand the question. Release for what? | 17:00 |
| Jeffrey4l | release kolla-k8s o-1 | 17:00 |
| *** absubram has joined #openstack-kolla | 17:01 | |
| rhallisey | I don't think kolla-k8s will follow it until 1.0 | 17:01 |
| rhallisey | there is no stable branch yet | 17:01 |
| Jeffrey4l | rhallisey, ok. | 17:01 |
| rhallisey | Jeffrey4l, the plan is to move to a kolla like release schedule after O | 17:01 |
| Jeffrey4l | got it, thanks. | 17:02 |
| *** ppalacios has quit IRC | 17:02 | |
| duonghq | rhallisey, did you see k8s service in external_ip pending state? | 17:02 |
| rhallisey | duonghq, are you talking about setup? | 17:03 |
| Jeffrey4l | inc0, do u have any idea for backport kolla-ansible change to stable branch? | 17:03 |
| duonghq | rhallisey, I use kubeadm to spin up a 3 nodes k8s-cluster | 17:03 |
| duonghq | when I create a service (out of 2-nginx pod) | 17:03 |
| duonghq | the external IP of service is set pending forever | 17:04 |
| duonghq | I apply kube-flannel already | 17:04 |
| rhallisey | hm | 17:05 |
| rhallisey | duonghq, http://kubernetes.io/docs/user-guide/load-balancer/ | 17:06 |
| *** ppalacios has joined #openstack-kolla | 17:06 | |
| kfox1111 | morning | 17:06 |
| duonghq | morning kfox1111 | 17:06 |
| *** sdake has quit IRC | 17:06 | |
| rhallisey | duonghq, that's strange you have it stuck in pending | 17:06 |
| *** ppalacios has quit IRC | 17:07 | |
| *** unicell1 has joined #openstack-kolla | 17:07 | |
| *** lrensing has joined #openstack-kolla | 17:07 | |
| kfox1111 | duonghq: kubectl describe pod <name/id of the stuck pod> ? | 17:07 |
| duonghq | I setup w/ kubeadm on ubuntu 16.04, 3 VM nodes | 17:07 |
| rhallisey | kfox1111, his external IP for his service is stuck in pending | 17:08 |
| kfox1111 | external ip in the service? | 17:08 |
| kfox1111 | never seen a "pending" state for a service. | 17:08 |
| rhallisey | same | 17:08 |
| kfox1111 | kubectl describe svc <servicename>? | 17:08 |
| *** unicell has quit IRC | 17:08 | |
| *** sdake has joined #openstack-kolla | 17:08 | |
| kfox1111 | guessing the selector labels don't match the pod labels? | 17:09 |
| duonghq | moment, I'm pasting it in some pastebin | 17:09 |
| duonghq | https://pastebin.mozilla.org/8928823 | 17:09 |
| *** bmace has quit IRC | 17:10 | |
| *** bmace has joined #openstack-kolla | 17:10 | |
| duonghq | pod's https://pastebin.mozilla.org/8928825 | 17:11 |
| kfox1111 | that looks ok. | 17:11 |
| kfox1111 | and curl 10.99.120.49 doesn't work? | 17:11 |
| kfox1111 | or curl localhost:30826 ? | 17:11 |
| *** egonzalez90 has quit IRC | 17:12 | |
| duonghq | I cannot ping it | 17:12 |
| kfox1111 | ping doesn't work in k8s. | 17:13 |
| kfox1111 | only the tcp port. | 17:13 |
| kfox1111 | does the curl work? | 17:13 |
| duonghq | curl cannot get nginx welcome page too | 17:13 |
| kfox1111 | on either of those? | 17:13 |
| kfox1111 | what about "curl 10.244.1.2:80" ? | 17:14 |
| duonghq | same there | 17:14 |
| kfox1111 | k. so somethings wrong at the docker networking layer. | 17:14 |
| duonghq | one note: if deployed flannel as k8s add-ons, follow kubeadm guide | 17:14 |
| kfox1111 | I'd guess flannel. | 17:14 |
| duonghq | *I | 17:14 |
| kfox1111 | I didn't think flannel itself had a kubeadm daemonset. | 17:14 |
| kfox1111 | did you use canal? | 17:15 |
| duonghq | not yet, I just follow kubeadm guide | 17:15 |
| duonghq | kfox1111, here: http://kubernetes.io/docs/getting-started-guides/kubeadm/ | 17:16 |
| kfox1111 | the kubeadm guilde last I looked said to pick a network plugin and launch with kubectl create -f .... | 17:16 |
| kfox1111 | I thin kthe default recommended is weave, | 17:16 |
| kfox1111 | but gives some other options. | 17:16 |
| kfox1111 | step (3/4) | 17:17 |
| *** msimonin has quit IRC | 17:17 | |
| kfox1111 | which points you at: http://kubernetes.io/docs/admin/addons/ | 17:17 |
| *** jemcevoy has quit IRC | 17:17 | |
| mliima | pbourke, https://etherpad.openstack.org/p/kolla-repo-split | 17:17 |
| kfox1111 | oh. looks like they have a pure flannel plugin now. | 17:17 |
| duonghq | kfox1111, I go through 4 steps, and add flannel afterthat | 17:18 |
| kfox1111 | step 3 should have added flannel. | 17:18 |
| kfox1111 | you did flannel with this template? https://raw.githubusercontent.com/coreos/flannel/master/Documentation/kube-flannel.yml | 17:19 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 17:19 |
| duonghq | yup | 17:19 |
| kfox1111 | k. | 17:20 |
| kfox1111 | maybe see what it looks like with: kubectl get pods --namespace kube-system | 17:20 |
| duonghq | I'm reseting cluster, and add network before joining other nodes | 17:20 |
| kfox1111 | see if it all is happy. | 17:20 |
| duonghq | it was happy | 17:20 |
| rhallisey | kfox1111, it does have it's own daemonset | 17:20 |
| duonghq | every pods | 17:20 |
| kfox1111 | I don't think it matters when its added, but not 100% sure. | 17:20 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 17:21 |
| rhallisey | has anyone gotten kolla-k8s to work with kubeadm? | 17:21 |
| rhallisey | I had a bunch of issues when I tried it | 17:21 |
| rhallisey | brb | 17:22 |
| duonghq | kfox1111, https://pastebin.mozilla.org/8928829 | 17:22 |
| duonghq | seems that everything is fine? | 17:22 |
| duonghq | rhallisey, so which method are you using for deploy k8s? | 17:23 |
| sbezverk | duonghq: pending state happens when there is already another process using the same port | 17:23 |
| kfox1111 | yeah. and yet connectivity isn't working. | 17:23 |
| kfox1111 | so somethings blocking traffic. | 17:23 |
| kfox1111 | next suspicion is iptables maybe. | 17:23 |
| *** matrohon has quit IRC | 17:23 | |
| *** rmart04 has joined #openstack-kolla | 17:24 | |
| kfox1111 | what base distro are you using, and did you disable the native firewall on it before starting? | 17:24 |
| *** chas has quit IRC | 17:24 | |
| duonghq | kfox1111, I'm using Ubuntu 16.04 server edition | 17:24 |
| kfox1111 | I don't know ubuntu 16.04's defualt firewall. :/ | 17:24 |
| *** chas has joined #openstack-kolla | 17:24 | |
| kfox1111 | plz run an iptables-save on one of the compute nodes? | 17:25 |
| duonghq | kfox1111, seems that kubeadm or k8s add some rules to iptables, | 17:25 |
| sbezverk | rhallisey: demo we showed at the summit was done with kubeadm | 17:25 |
| kfox1111 | duonghq: yeah. flannel does, docker does, and kube-proxy does. :) | 17:25 |
| kfox1111 | but they can conflict with preexisting rules too. :/ | 17:25 |
| kfox1111 | my guess would be that the vxlan traffic for flannel is getting blocked. | 17:26 |
| sbezverk | duonghq: when you deploy networking, which cni did you use? | 17:26 |
| kfox1111 | sbezverk: flannel | 17:26 |
| kfox1111 | Id look at the cbr0 interface on each node, see what its ip is, | 17:26 |
| kfox1111 | and try to ping each one from one of the nodes. | 17:27 |
| duonghq | kfox1111, https://pastebin.mozilla.org/8928830 here is my minion1 iptables | 17:27 |
| sbezverk | duonghq: can you share yaml file you used for flannel deploy? | 17:27 |
| duonghq | sbezverk, I used this file: https://github.com/coreos/flannel/blob/master/Documentation/kube-flannel.yml | 17:27 |
| kfox1111 | hmm... firewall looks ok to me. | 17:27 |
| duonghq | kfox1111, you mean cni0? | 17:28 |
| kfox1111 | yeah. | 17:28 |
| sbezverk | duonghq: have you changed ip addresses to match to your environment? | 17:28 |
| kfox1111 | sbezverk: duonghq: or, does "Network": "10.244.0.0/16", stop on any existing network? | 17:28 |
| duonghq | sbezverk, I used --pod-network-cidr=10.244.0.0/16 | 17:28 |
| *** chas has quit IRC | 17:29 | |
| duonghq | kfox1111, https://pastebin.mozilla.org/8928831 | 17:29 |
| duonghq | seem that cni0 is down? | 17:29 |
| sbezverk | duonghq: can you run kubectl get svc --all-namespaces | 17:30 |
| duonghq | some dmesg line: https://pastebin.mozilla.org/8928833 | 17:30 |
| duonghq | sbezverk, here: https://pastebin.mozilla.org/8928834 | 17:31 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment https://review.openstack.org/398506 | 17:31 |
| sbezverk | duonghq: flannel needs etcd service for ip addressing distribution | 17:32 |
| sbezverk | and it is not there | 17:32 |
| kfox1111 | duonghq: just to double check, do an 'ip a' ? | 17:32 |
| duonghq | sbezverk, there is a etcd pod is running | 17:32 |
| sbezverk | I use canal.yaml and as a part of installation it install etcd instance | 17:32 |
| kfox1111 | sbezverk: interesting. your right. | 17:33 |
| kfox1111 | that flannel daemonset seems incomplete. | 17:33 |
| sbezverk | duonghq: maybe pod is running but servcice is not there | 17:33 |
| duonghq | https://pastebin.mozilla.org/8928835 | 17:33 |
| sbezverk | so nobody can talk to etcd pod | 17:33 |
| duonghq | sbezverk, understodo | 17:33 |
| kfox1111 | unless there's some very different logic in flannel when --kube-subnet-mgr is passed. | 17:33 |
| kfox1111 | yeah, I'd try the canal one. used it recently and it worked. | 17:34 |
| sbezverk | duonghq: I suggest you clean up your cni setup and install canal I install one this morning | 17:34 |
| duonghq | sbezverk, I'm trying, hope that it'll work | 17:35 |
| sdake | kfox1111 - was helm an optional peice or mandatory peice of our architecture? | 17:35 |
| duonghq | sbezverk, also, do you have any idea recover k8s cluster after all nodes is restarted (using kubeadm) | 17:35 |
| kfox1111 | sdake: layer 3. | 17:35 |
| sbezverk | duonghq: yes, there is a clean up procedure | 17:35 |
| *** jistr|biab is now known as jistr | 17:36 | |
| kfox1111 | you can, say, use kolla containers by themselves at layer 1 if your so inclined. | 17:36 |
| kfox1111 | so nothings really manditory. | 17:36 |
| duonghq | sbezverk, I try reset but it's cleanup everything | 17:36 |
| sbezverk | duonghq: to remove your cni just run kubectl delete -f flannel.yaml | 17:36 |
| *** absubram has quit IRC | 17:37 | |
| duonghq | sbezverk, it's another question, I wonder if k8s cluster which is setup by kubeadm can be brought up again after failure | 17:37 |
| sbezverk | duonghq: not sure I undersatnd what you mean. bring up as a brand new installation or recover old one? | 17:38 |
| sdake | kfox1111 i think there is some confusion over what is mandatory or what is not | 17:39 |
| duonghq | sbezverk, I want to recover old | 17:39 |
| sdake | kfox1111 atleast I have some ;) | 17:39 |
| *** fragatina has joined #openstack-kolla | 17:41 | |
| sbezverk | duonghq: as long as the master is up, you can join it again from other nodes after you clean them up | 17:41 |
| duonghq | so, I have no luck if master is down atm? | 17:41 |
| sbezverk | you just need to use the same id generated by kubeadm for the installation | 17:42 |
| duonghq | sbezverk, thanks | 17:42 |
| sbezverk | duonghq: I am afraid so, as the master is the key point | 17:42 |
| *** unicell1 has quit IRC | 17:42 | |
| duonghq | okay | 17:42 |
| duonghq | about my external_ip stuck | 17:42 |
| duonghq | I got some log: https://pastebin.mozilla.org/8928839 | 17:43 |
| duonghq | I don't know why minion cni0 still down | 17:43 |
| duonghq | https://pastebin.mozilla.org/8928840 | 17:44 |
| sbezverk | duonghq: one sec, the output you provided shows that you use canal, is it a different installation? | 17:45 |
| duonghq | sbezverk, I reset all 3 nodes I do another deploy | 17:46 |
| sbezverk | duonghq: ok | 17:46 |
| *** david-lyle has quit IRC | 17:48 | |
| *** david-lyle has joined #openstack-kolla | 17:48 | |
| sbezverk | duonghq: hm in my canal installation, only two types of interfaces created. flannel and calico, not sure where cni0 is comming from | 17:48 |
| duonghq | sbezverk, so I guess that we can forget this | 17:49 |
| duonghq | moment, I'll try kubeadm reset and reboot all nodes | 17:50 |
| *** tonanhngo has quit IRC | 17:51 | |
| inc0 | sdake, | 17:51 |
| inc0 | https://review.openstack.org/#/c/398025/ | 17:51 |
| inc0 | I think we should remove deploy gates at this moment and make build gates voting | 17:52 |
| sdake | sup | 17:52 |
| sbezverk | duonghq: if it does not work, I would suggest to use good clean up procedure and then proceed with installation over a webex session | 17:52 |
| sdake | well i did have a followon patch that removed the deploy gates | 17:52 |
| inc0 | and restore deploy gates along k8s voting gates when we'll deal with zuul-cloner | 17:52 |
| sdake | but abanonded that | 17:52 |
| duonghq | sbezverk, do I need to change etcd endpoints to 10.244.0.0/16 subnet? | 17:52 |
| sdake | moment let me find the change | 17:52 |
| inc0 | as they will look different | 17:52 |
| inc0 | I just don't want to block development until we deal with zuul cloner | 17:53 |
| sbezverk | duonghq: yes, in canal.yaml you need to assign IP to etcd which will be used. I usually do and pick up next IP after dns services. | 17:53 |
| sbezverk | you need to change it at two places | 17:53 |
| sdake | ya makes sense if people have trouble reading the gate output | 17:53 |
| sbezverk | in canal.yaml | 17:53 |
| sdake | which i think people do sometimes | 17:53 |
| sdake | hunting for change still moment | 17:54 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 17:54 |
| inc0 | having perma-red gates is worse than not having gates | 17:54 |
| inc0 | as it will cause confusion | 17:54 |
| inc0 | and we have enough of it aleready | 17:54 |
| rhallisey | sdake, so I have layer 3 and 1 | 17:54 |
| sdake | https://review.openstack.org/#/c/398028/ | 17:54 |
| rhallisey | actually helm is also optional | 17:54 |
| inc0 | sdake, I'd restore this change | 17:55 |
| inc0 | and merge it | 17:55 |
| sdake | wfm | 17:55 |
| inc0 | and deal with gates systematically | 17:55 |
| sbezverk | rhallisey: kolla-kube is in templates and they are specific to a rendering engine | 17:55 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: Add how configure multiple back ends section https://review.openstack.org/398516 | 17:55 |
| sbezverk | I am not sure I undersatnd then the architecture | 17:55 |
| inc0 | in fact, after that I would make all build gates voting | 17:56 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 17:56 |
| rhallisey | sbezverk, helm packages those templates | 17:56 |
| rhallisey | layer 2 is what you just mentioned. It is not optional | 17:56 |
| sdake | inc0 show me the mirrors and i'll submit the patch - otherwise feel free ;) | 17:56 |
| rhallisey | neither is layer 1, the kolla containers | 17:56 |
| sbezverk | rhallisey: right, but then it makes helm as mandatory as nothing else can be used to render templates, no? | 17:56 |
| inc0 | sdake, mirrors won't help really | 17:56 |
| rhallisey | sbezverk, helm is optional | 17:56 |
| rhallisey | you can render the templates by hand | 17:56 |
| rhallisey | or kolla-k8s tools | 17:57 |
| rhallisey | cli | 17:57 |
| inc0 | as mirrors didn't help us when canonical messed up versions | 17:57 |
| kfox1111 | sbezverk: layer 2 is k8s. | 17:57 |
| inc0 | but really changes to dockerfiles are pretty rare now | 17:57 |
| *** lrensing has quit IRC | 17:57 | |
| kfox1111 | well, k8s/the rendered templates to k8s. | 17:57 |
| inc0 | so I think we can affort not merging them for these few days of breakages | 17:57 |
| sdake | inc0 right - i expect the dockerfiles to have a low rate of change | 17:57 |
| inc0 | at least it will motivate people to fix actual issues fast;) | 17:57 |
| sdake | the problem is during final release | 17:58 |
| sbezverk | kfox1111: I am lost :-( now we have jinja2 templates and they are rendered by cli tool. I thought we will scrap/convert them into helm packages, right? | 17:58 |
| kfox1111 | sbezverk: right. | 17:58 |
| sdake | when upstream binaries are busted and we have to release | 17:58 |
| inc0 | if you can't build in gates - you can't build locally - you can't test change - shouldn't be merged | 17:58 |
| inc0 | sdake, only if all happends at once: 1. upstream is broken, 2. we found critical bug in dockerfiles | 17:59 |
| sbezverk | kfox1111: then we will have only helm charts and that makes helm as mandatory | 17:59 |
| inc0 | which is rare | 17:59 |
| kfox1111 | sbezverk: I think you have the same slight cognative dissonence with the layer thing I do, but I think its a very sublile thing so I wasn't worrying about it. | 17:59 |
| sdake | inc0 try your hand at working on the gate then ;-) | 17:59 |
| kfox1111 | sbezverk: there are charts, and there are templates. | 17:59 |
| inc0 | yeah, I'll risk it and make it voting;) | 17:59 |
| kfox1111 | the templates are golang templates. so could in theory be used by nonhelm too. | 17:59 |
| sdake | should be straightforward | 17:59 |
| inc0 | but let's clear up what we have before we do that | 17:59 |
| sdake | ping me if you need guidance | 17:59 |
| kfox1111 | but its very unlikely to be used without helm. | 17:59 |
| sdake | yo core reviewers | 17:59 |
| sbezverk | kfox1111: right my bad, should not use them interchangably | 17:59 |
| sdake | how about an ack on https://review.openstack.org/#/c/398025/ | 18:00 |
| duonghq | sbezverk, does the warning matter? https://pastebin.mozilla.org/8928843 (every pods is running eventually) | 18:00 |
| sdake | this will trigger merge conflicts against every patch that touches that directory | 18:00 |
| sdake | so we don't hae to keep -2ing changes | 18:00 |
| inc0 | -30k loc...that's quite a patch | 18:00 |
| * sdake pumping up his stats | 18:01 | |
| sdake | ;-) | 18:01 |
| sbezverk | duonghq: yes it does, it means you did not install cni rpm | 18:01 |
| kfox1111 | sdake: call kolla repo cores. | 18:01 |
| sbezverk | so it is cmplaining about missing binaries | 18:01 |
| kfox1111 | sdake: there are other sets of cores now. | 18:01 |
| sdake | kolla-repo cores, how about an ack on the above patch | 18:01 |
| kfox1111 | inc0: hehe | 18:01 |
| sdake | kfox1111 thanks :) | 18:01 |
| kfox1111 | sdake: np. :) | 18:02 |
| inc0 | kfox1111, speaking of which | 18:02 |
| inc0 | I'd remove inactive kolla-k8s cores | 18:02 |
| duonghq | sbezverk, strange, I install kubernetes-cnt package | 18:02 |
| inc0 | rhallisey, | 18:02 |
| rhallisey | yea | 18:02 |
| inc0 | ^ | 18:02 |
| rhallisey | ? | 18:02 |
| duonghq | *installed | 18:02 |
| sbezverk | duonghq: check ls /opt/cni/bin/ on all your workers | 18:02 |
| rhallisey | inc0, kk | 18:03 |
| inc0 | rhallisey, take a look at kolla-k8s cores and see if you want anyone added/removed | 18:03 |
| rhallisey | I've been monitoring it | 18:03 |
| rhallisey | I think it's fine atm | 18:03 |
| sbezverk | duonghq: you should see flannel | 18:03 |
| inc0 | kk | 18:03 |
| duonghq | root@master:~# ls /opt/cni/bin/ | 18:03 |
| duonghq | bridge calico calico-ipam cnitool dhcp flannel host-local ipvlan loopback macvlan noop ptp tuning | 18:03 |
| inc0 | kolla-ansible cores are separate team now fyi | 18:03 |
| sbezverk | duonghq: on other nodes too | 18:03 |
| inc0 | just copy of kolla cores | 18:03 |
| duonghq | sbezverk, same on all nodes | 18:04 |
| *** fragatin_ has joined #openstack-kolla | 18:04 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use kolla-ansible for deployment https://review.openstack.org/398506 | 18:05 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: fix deploy gate https://review.openstack.org/398501 | 18:05 |
| *** Pavo has quit IRC | 18:07 | |
| *** fragatina has quit IRC | 18:08 | |
| sdake | ok so i foresee a problem inc0 | 18:08 |
| sbezverk | duonghq: ok it seems it still cannot communicate with etcd "dial tcp 10.96.232.136:6666: i/o timeout" you need to make sure your config is consistent | 18:08 |
| sdake | we had one set of processes for voting for policy and core reviewer nominations | 18:09 |
| sdake | we now have 3 separate teams | 18:09 |
| sdake | i think we should have some discussion on ml about the separate teams and how the policies work | 18:09 |
| duonghq | sbezverk, I do not know where the pod get the IP, I update all etcd-related IP in canal.yml | 18:09 |
| sdake | followed by a vote of the existing kolla core team to firm up whatever new policies we decide upon | 18:10 |
| sdake | or even a superset of kolla + kolla-kubernetes | 18:10 |
| *** v1k0d3n has quit IRC | 18:11 | |
| sdake | although the later isn't how our policies are currently set | 18:11 |
| kfox1111 | best guess is kolla-ansible needs some cores, kolla-kubernetes needs some cores, and maybe kolla containers repo should be the union of the other teams? | 18:11 |
| sdake | kfox1111 ya - decision for the ml ;) | 18:11 |
| *** pbourke has quit IRC | 18:11 | |
| inc0 | sdake, I think obvious resolution would be that kolla-k8s cores votes on kolla-k8s cores and so on | 18:11 |
| kfox1111 | sdake: sure. just a suggestion. | 18:11 |
| kfox1111 | yeah. | 18:12 |
| sdake | inc0 right but that is a change to our "governance" | 18:12 |
| *** pbourke has joined #openstack-kolla | 18:12 | |
| sdake | inc0 hence needs a vote | 18:12 |
| sdake | inc0 woudl you mind starting a thread? | 18:12 |
| sbezverk | duonghq: there are many reason for things to go wrong and it is difficult to troubleshoot over irc channel. I would clean up everything and proceed with clean installation together on webex session.. | 18:12 |
| *** Pavo has joined #openstack-kolla | 18:12 | |
| *** diogogmt has joined #openstack-kolla | 18:13 | |
| sdake | all policy decisions (including changing policies) requires a mailing list vote.. :) | 18:13 |
| sdake | we decided this long long ago | 18:13 |
| sdake | inc0 what is logical to you may not be logical to someone else | 18:13 |
| inc0 | sdake, sure, and every core has voting rights for this particular policy change | 18:13 |
| inc0 | including kolla-k8s cores | 18:13 |
| sdake | grey area, but wfm | 18:13 |
| sdake | in the true sense of governance that wouldn't be the case | 18:14 |
| inc0 | kolla-k8s is no longer second class citizen, same rules apply | 18:14 |
| *** absubram has joined #openstack-kolla | 18:14 | |
| sdake | we can't make policy changes by fiat | 18:14 |
| *** harlowja has joined #openstack-kolla | 18:14 | |
| sdake | that just pisses everyone off | 18:14 |
| sdake | needs to be a proposal and vote on ml | 18:14 |
| inc0 | on it | 18:14 |
| sdake | thanks :) | 18:14 |
| rhallisey | what's the vote for? | 18:16 |
| *** unicell has joined #openstack-kolla | 18:16 | |
| rhallisey | I think kolla-k8s cores should vote on kolla-k8s cores | 18:17 |
| sdake | rhallisey our new policies going forward | 18:19 |
| sdake | rhallisey lets have the discussion over email pls | 18:19 |
| sdake | rhallisey governance really tricky | 18:19 |
| sdake | irc discussion imo isn't good enough | 18:19 |
| sdake | email/ml i mean ;) | 18:19 |
| rhallisey | k | 18:19 |
| rhallisey | I'm jumping in and out of spec & irc | 18:20 |
| sdake | roger | 18:20 |
| rhallisey | so ml would be better | 18:20 |
| *** eaguilar_ has joined #openstack-kolla | 18:21 | |
| *** eaguilar has quit IRC | 18:22 | |
| inc0 | done | 18:23 |
| inc0 | voting is up till end of month | 18:23 |
| inc0 | rhallisey, Jeffrey4l pbourke https://review.openstack.org/#/c/398025/ mind acking? | 18:25 |
| sdake | kfox1111 i was thinking about kolla-kubernetes in light of inc0's proposal | 18:30 |
| sdake | kfox1111 I was reminded of the fact we never set any policies around kolla-kubernetes | 18:30 |
| inc0 | yup | 18:30 |
| sdake | since none were set, i think the appropriate action is to "use your best judgement" | 18:30 |
| inc0 | about time;) | 18:30 |
| rhallisey | what are you reffering to? | 18:30 |
| sdake | ya my fault for not recongnizing that | 18:30 |
| rhallisey | by policies sdake | 18:30 |
| sdake | rhallisey the ml | 18:30 |
| sdake | who can vote on core review changes, who can vote on policy changes | 18:31 |
| inc0 | we just thrown people into core team for kolla-k8s | 18:31 |
| rhallisey | kolla-kubernetes has policies | 18:31 |
| rhallisey | I was destibuting them by irc not be ML :) | 18:31 |
| sdake | i mean the kind on the mailing list | 18:31 |
| rhallisey | for other reasons :) | 18:31 |
| sdake | that come with a vote | 18:31 |
| rhallisey | for ex) voting policy in kolla-k8s only takes one +2 atm | 18:31 |
| *** Administrator_ has quit IRC | 18:32 | |
| rhallisey | since the core team isn't super large | 18:32 |
| *** tonanhngo has joined #openstack-kolla | 18:32 | |
| *** Administrator_ has joined #openstack-kolla | 18:32 | |
| rhallisey | inc0, sorry confused. Who is getting thrown into the core team? | 18:33 |
| sdake | well since those policies weren't recorded or voted on, just consider kolla-kubernetes prior to the policy setting we are doing to use your best judgement | 18:33 |
| sbezverk | inc0: what about voting for changes in kolla repo? some changes in kolla may impact k8s as well | 18:33 |
| inc0 | sbezverk, we have kolla core team | 18:33 |
| inc0 | me, sdake, rhallisey are still kolla cores:) | 18:33 |
| sbezverk | inc0: you have, but k8s team have no say in there | 18:33 |
| sdake | and kfox ;) | 18:33 |
| inc0 | which means we won't allow breaking changes | 18:33 |
| sdake | and serguey | 18:33 |
| sdake | sorry sbezverk for butchering your name | 18:34 |
| sbezverk | np | 18:34 |
| sdake | oh you mean kolla-core team | 18:34 |
| inc0 | they aren't cores in kolla last time I checked;) | 18:34 |
| rhallisey | k I get the policy you are referring to now | 18:34 |
| inc0 | if you guys wanna become them...well review kolla;) | 18:34 |
| *** tonanhngo has quit IRC | 18:36 | |
| kfox1111 | inc0: yeah, I have no core'nes on the container side. | 18:36 |
| kfox1111 | but depend on it heavily. :) | 18:37 |
| sbezverk | inc0: ok I hope this rule will apply to becoming k8s core as well, so people need to review to become k8s core as well; | 18:37 |
| sean-k-mooney | sbezverk: isnt that how core membership works for all openstack poject outside the people who made the repo | 18:38 |
| *** tonanhngo has joined #openstack-kolla | 18:38 | |
| inc0 | sbezverk, rules are the same for all teams | 18:38 |
| *** tonanhngo has quit IRC | 18:39 | |
| *** tonanhngo has joined #openstack-kolla | 18:39 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 18:40 |
| rhallisey | any more comments? | 18:41 |
| kfox1111 | inc0: I'm still not familior with the kolla mailing list voting process. | 18:44 |
| kfox1111 | if I agree with your resolution, whats my response? | 18:44 |
| rhallisey | kfox1111, it would be +1 | 18:44 |
| kfox1111 | k. | 18:45 |
| rhallisey | and you can toss a why in there if you want | 18:45 |
| kfox1111 | thats what I was going to do, but second guessed myself. THanks for the clarification. :) | 18:45 |
| *** jtriley has quit IRC | 18:46 | |
| *** huhaoran has joined #openstack-kolla | 18:47 | |
| sdake | kfox1111 also if you want alterations feel free to discuss that :) | 18:48 |
| *** rmart04_ has joined #openstack-kolla | 18:48 | |
| sdake | our voting process is a little haphazard often causing second votes after we come to agreement :) | 18:48 |
| inc0 | as long as result is 9.2, we're ok | 18:48 |
| rhallisey | I think the spec is good to go | 18:49 |
| kfox1111 | sdake: its close enough. :) | 18:49 |
| inc0 | guys, hate me if you want | 18:49 |
| *** rmart04 has quit IRC | 18:49 | |
| *** rmart04_ is now known as rmart04 | 18:49 | |
| rhallisey | did anyone else have any more comments? | 18:49 |
| inc0 | but I just proposed making build source gates voting | 18:49 |
| inc0 | https://review.openstack.org/#/c/398534/1/zuul/layout.yaml | 18:49 |
| *** athomas has quit IRC | 18:50 | |
| sdake | rhallisey would you mind rubber stamping this https://review.openstack.org/#/c/398025/ | 18:52 |
| sdake | rhallisey on which? | 18:53 |
| inc0 | kfox1111, ad genconfig | 18:54 |
| inc0 | genconfig for time being will stay in ansible | 18:54 |
| rhallisey | sdake, spec | 18:56 |
| sdake | rhallisey reviewing | 18:56 |
| sdake | i saw i didn't read it :) | 18:56 |
| sdake | rhallisey lgtm | 19:01 |
| *** shardy is now known as shardy_afk | 19:01 | |
| sdake | i'm reminded of a documentary i saw on clinton's first 100 days in office | 19:04 |
| *** msimonin has joined #openstack-kolla | 19:04 | |
| sdake | still needs a rubber stamp - repo split related: https://review.openstack.org/#/c/398025/ | 19:05 |
| sdake | Jeffrey4l rhallisey ^^ | 19:05 |
| *** coolsvap has quit IRC | 19:07 | |
| kfox1111 | inc0: just asking now, as it will be more painful to move it back to kolla later. | 19:10 |
| openstackgerrit | Merged openstack/kolla: Remove Ansible from Kolla https://review.openstack.org/398025 | 19:10 |
| kfox1111 | and now it doesn't matter. :) | 19:10 |
| inc0 | kfox1111, it's going to be painful and volitale anyway | 19:11 |
| inc0 | one crysis at the time | 19:11 |
| *** ccesario has quit IRC | 19:11 | |
| kfox1111 | inc0: fair enough. :) | 19:12 |
| kfox1111 | lets deal with the current one. :) | 19:12 |
| sdake | kfox1111 fwiw we need to sort that out - i agree completely | 19:14 |
| sdake | kfox1111 not sure how to do so ;) | 19:14 |
| sdake | kfox1111 you wre talking about the config splitout right? | 19:15 |
| *** shardy_afk is now known as shardy | 19:16 | |
| sdake | sup shardy | 19:16 |
| *** shardy is now known as shardy_afk | 19:17 | |
| *** msimonin has quit IRC | 19:19 | |
| sdake | rhallisey | 19:20 |
| *** haplo37_ has quit IRC | 19:20 | |
| *** g3ek has quit IRC | 19:20 | |
| sdake | this slack channel you speak of | 19:20 |
| sdake | you were tlaking about adding an irc gateway? | 19:20 |
| sdake | how does that happen? | 19:20 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 19:21 |
| britthouser | Are you wanting to use IRC client to connect to slack channel, or use slack client to connect to IRC server, sdake? | 19:21 |
| sdake | britthouser the first | 19:21 |
| sdake | what would rock is if the channels were osmehow synced | 19:21 |
| rhallisey | sdake, https://sameroom.io/blog/mirror-freenode-channels-in-slack/ | 19:21 |
| britthouser | The admin of the channel has to enable it, then its self service. | 19:21 |
| rhallisey | sdake, they can be synced | 19:21 |
| rhallisey | the issue is slack | 19:21 |
| rhallisey | kube slack has a cache of 10k live messages | 19:22 |
| sdake | rhallisey could you go into more detail | 19:22 |
| rhallisey | sure | 19:22 |
| rhallisey | slack is different then irc | 19:22 |
| sdake | i understand they are different htings ;) | 19:22 |
| britthouser | I use <https://github.com/aeirola/slack-irc-client> to do #2 | 19:22 |
| rhallisey | the entire group, kubernetes, has 10k lives messages | 19:22 |
| rhallisey | that means | 19:22 |
| sdake | thats why you mentioned the 2 msgs per second thing prior | 19:23 |
| rhallisey | if you type something it stays in the scroll back until it's message 10000 in the cache | 19:23 |
| sdake | rhallisey is that per channel or per group? | 19:23 |
| rhallisey | right now, kube goes through that cache in 2 days | 19:23 |
| rhallisey | group | 19:23 |
| rhallisey | so all the individual channels share | 19:23 |
| sdake | right | 19:23 |
| rhallisey | we have about 2 messages per minute | 19:24 |
| rhallisey | chanserv stuff | 19:24 |
| sdake | what does that work out per to day | 19:24 |
| rhallisey | including chanserv stuff | 19:24 |
| rhallisey | 2880 | 19:24 |
| sdake | yikes so we would be taking 28% of the cache per day? | 19:24 |
| rhallisey | they flip in 48 hours | 19:24 |
| rhallisey | yes | 19:24 |
| rhallisey | :D | 19:24 |
| rhallisey | we would be their most active channel | 19:24 |
| sdake | so kube slack admins not keen on increasing the capacity? | 19:24 |
| rhallisey | they are | 19:25 |
| britthouser | increasing capacity == $$ | 19:25 |
| sdake | sweet | 19:25 |
| rhallisey | idk when it will happen though | 19:25 |
| britthouser | Free slack cuts off at 10k messages | 19:25 |
| britthouser | paid slack is unlimited. | 19:25 |
| sdake | i see | 19:25 |
| rhallisey | ohh | 19:25 |
| sdake | well thats a huge delta | 19:25 |
| sdake | how much is paid slack britthouser ? | 19:25 |
| britthouser | Not sure, but its a per user charge. | 19:26 |
| sdake | groan | 19:26 |
| sdake | make it harder | 19:26 |
| britthouser | not cheap from what I understand. | 19:26 |
| kfox1111 | britthouser: yeah. its per user. | 19:26 |
| kfox1111 | but I think slack should consider waving the fee for openstource projects like this, to get more users confortable with slack. | 19:27 |
| sdake | so rhallisey kubernetes admins not keen to connect our channels? | 19:27 |
| *** matrohon has joined #openstack-kolla | 19:27 | |
| kfox1111 | its a win win for them. | 19:27 |
| sdake | kfox1111 make it really harder! :) | 19:27 |
| kfox1111 | :) | 19:27 |
| inc0 | I'm not a big fan of slack | 19:27 |
| inc0 | yes, it's flashy | 19:27 |
| sdake | thats why we are tlaking about connecting them | 19:27 |
| *** jtriley has joined #openstack-kolla | 19:27 | |
| rhallisey | sdake, well we would damage their scroll back | 19:27 |
| kfox1111 | I'm a fan of the interface. the fact that its closed though, not so much. | 19:27 |
| sdake | because other people are big fans | 19:27 |
| sdake | rhallisey right | 19:28 |
| kfox1111 | the big selling point of slack is the cell phone integration. | 19:28 |
| rhallisey | but it's all logged | 19:28 |
| rhallisey | the issue though is more projects | 19:28 |
| rhallisey | and more growth there | 19:28 |
| kfox1111 | after you use it for a while, tis really hard to use irc again. | 19:28 |
| sdake | at some point kubernetes group on slack ging to have to pay i think or get a waiver | 19:28 |
| rhallisey | ya it is neat | 19:28 |
| inc0 | idk, I tried a to use it couple of times | 19:28 |
| sdake | so back on topic :) we would dmg their scrollback capacity | 19:29 |
| sdake | and kubernetes group admin is negatory on that proposal? | 19:29 |
| *** haplo37_ has joined #openstack-kolla | 19:29 | |
| sdake | or has it been proposed | 19:29 |
| kfox1111 | inc0: it really shines when you ahve a group of people that needs to get a hold of each other. the pinging that actually reaches the human no matter where thery are is the killer feature there. | 19:29 |
| kfox1111 | yeah. we'd blow out their cache pretty quickly. | 19:29 |
| rhallisey | sdake, I haven't mentioned we are 2880 messages a day | 19:29 |
| sdake | instead of guessing i'd like to know from the src :) | 19:29 |
| *** absubram has quit IRC | 19:30 | |
| inc0 | google could sponsor it | 19:30 |
| *** duonghq has quit IRC | 19:30 | |
| sdake | rhallisey that is a key peice of info you need to pass along | 19:30 |
| *** g3ek has joined #openstack-kolla | 19:30 | |
| inc0 | I mean, it's a fucking google, they can afford buying slack;) | 19:30 |
| kfox1111 | or google could just buy slack. ;) | 19:30 |
| kfox1111 | yeah | 19:30 |
| sdake | rhallisey we dont want to surprise the kubernetes admins | 19:30 |
| sdake | slack admins that is | 19:30 |
| sdake | rhallisey so wheres that at, been proposed, been shot down, waiting on funding, what? | 19:31 |
| rhallisey | which part sdake, getting more cache? | 19:31 |
| sdake | no linking the channels with exiwsting capacity | 19:31 |
| kfox1111 | sdake, even assuming we dont link, but do most of our traffic on slack, we'd still blow it out. | 19:31 |
| rhallisey | I can ask sarah and see what she thinks | 19:31 |
| sdake | kfox1111 right | 19:32 |
| rhallisey | but 2.8k a day isn't ideal | 19:32 |
| sdake | rhallisey please do so | 19:32 |
| kfox1111 | they need more capacity or we can't work there. :/ | 19:32 |
| sdake | kfox1111 rhallisey was kind enoug hto obtain us a channel | 19:32 |
| sdake | rhallisey so we can have a presence in the kubernetes community | 19:32 |
| sdake | kfox1111 rather ^^ | 19:32 |
| kfox1111 | sdake: yeah. I know. its great. but the cache is a big big problem. :/ | 19:32 |
| *** lrensing has joined #openstack-kolla | 19:33 | |
| sdake | i'd like to hear it from sarah | 19:33 |
| sdake | if she says no go, then plan b | 19:33 |
| rhallisey | I'd say we're somewhere between 1 and 2 messages a minute | 19:33 |
| rhallisey | so 2k a day | 19:33 |
| rhallisey | idk | 19:33 |
| rhallisey | total guess | 19:33 |
| rhallisey | kk | 19:33 |
| kfox1111 | and the main kubernetes channel gets a huge amount of traffic too. | 19:33 |
| sdake | britthouser thanks for the consulting :) | 19:34 |
| sdake | the kubernetes slack channel will have to deal with capacity at some point | 19:34 |
| sdake | because all CNCF projects are in the kubernetes group | 19:34 |
| sdake | rhallisey right or wrong? | 19:35 |
| sdake | actually i'm just assuming I don't know :) | 19:35 |
| rhallisey | yes | 19:35 |
| rhallisey | they are | 19:35 |
| rhallisey | afaik | 19:35 |
| sdake | if she says no which is a perfectly reasonable response - lets come up with a plan b | 19:36 |
| sdake | after she says yes/no | 19:36 |
| sdake | she has already gone out of her way to help us out | 19:36 |
| sdake | even though we are not a CNCF project | 19:36 |
| sdake | rhallisey what i'm getting at there is don't argue with her say thank you instead :) | 19:38 |
| sdake | in the meantime rhallisey you got a FAQ for slack | 19:39 |
| rhallisey | sdake, I don't have an faw for slack | 19:40 |
| rhallisey | it was super easy to join | 19:41 |
| *** fragatin_ has quit IRC | 19:41 | |
| *** fragatina has joined #openstack-kolla | 19:41 | |
| *** absubram has joined #openstack-kolla | 19:42 | |
| sdake | rhallisey sweett got a link | 19:43 |
| *** ppalacios has joined #openstack-kolla | 19:45 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Spec - Kolla-Kubernetes Deployment Architecture https://review.openstack.org/392257 | 19:54 |
| rhallisey | 391 lines | 19:54 |
| rhallisey | nice | 19:54 |
| *** msimonin has joined #openstack-kolla | 19:54 | |
| rhallisey | kfox1111, ^ | 19:54 |
| kfox1111 | rhallisey: looking | 19:55 |
| kfox1111 | changes look good. rereading in its entirety. | 19:55 |
| *** newmember has quit IRC | 19:55 | |
| rhallisey | wow merge conflicts galore | 19:56 |
| sdake | rhallisey right - that was the purpose ;) | 19:56 |
| rhallisey | inc0, can you review the spec | 19:57 |
| inc0 | yeah, will do | 19:57 |
| rhallisey | sbezverk, srwilkers + everyone | 19:57 |
| inc0 | do we mention that we are going to implement dependency init container too? | 19:57 |
| rhallisey | ya | 19:57 |
| inc0 | I just noted definition in there | 19:57 |
| sdake | yup thats in there | 19:57 |
| rhallisey | it's a layer | 19:57 |
| rhallisey | like operators are a layer | 19:57 |
| inc0 | rhallisey, just to be clear, you are aware that dependency init container won't require operators at all? | 19:58 |
| inc0 | for deployment at least | 19:58 |
| rhallisey | that's in there | 19:58 |
| sdake | its optional | 19:58 |
| *** msimonin has quit IRC | 19:58 | |
| *** msimonin1 has joined #openstack-kolla | 19:58 | |
| rhallisey | read layer section | 19:58 |
| sdake | i bitched about that too ;) | 19:58 |
| sdake | its kind of hidden away | 19:58 |
| rhallisey | layers are optional | 19:58 |
| rhallisey | and any layer can be used for deployment | 19:58 |
| sdake | kfox1111 pointed it out to me | 19:58 |
| rhallisey | or any other operator shit | 19:59 |
| srwilkers | looking now | 19:59 |
| srwilkers | sorry | 19:59 |
| rhallisey | srwilkers, np | 19:59 |
| rhallisey | just want to make sure we are good to merge | 19:59 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Squashed commit of the following: https://review.openstack.org/398563 | 19:59 |
| inc0 | so I don't expect helm to be optional per se | 19:59 |
| sdake | inc0 i had the same question | 19:59 |
| rhallisey | you could just use the templates | 19:59 |
| inc0 | as I honestly doubt we'll ever have an alternative there | 19:59 |
| rhallisey | and input values | 20:00 |
| rhallisey | say you were another project | 20:00 |
| sdake | ya that was the answer ^^ :) | 20:00 |
| rhallisey | and wanted to do that | 20:00 |
| inc0 | but truth is, output of helm is tons of yaml | 20:00 |
| inc0 | you can write this yaml by hand if you want | 20:00 |
| rhallisey | it is, but any tool can do that | 20:00 |
| rhallisey | any tool can interpret the yaml | 20:00 |
| inc0 | yeah looks good | 20:00 |
| kfox1111 | inc0: it is in one way. | 20:00 |
| kfox1111 | helm is about packaging. so a chart and a bunch of templates. | 20:01 |
| kfox1111 | the templates are just a bounch of k8s objects with golang template vars and stuff. | 20:01 |
| inc0 | fyi, we will have fairly strong dev team doing init-container mechanism + helm | 20:01 |
| sdake | personally I htink that spec is a work of art | 20:01 |
| sdake | but what do I know ;) | 20:01 |
| rhallisey | lol | 20:01 |
| rhallisey | fram it | 20:01 |
| inc0 | I expect this to be a thing pretty soon | 20:01 |
| kfox1111 | so if someone really wanted todo the templateing themselves without helm, they could source the templates but not the charts. | 20:01 |
| rhallisey | frame it | 20:01 |
| sdake | it would be a long frame ;) | 20:02 |
| *** rmart04 has left #openstack-kolla | 20:02 | |
| sbezverk | kfox1111: if templates are stored in format only helm atm understand would it not make it mandatory? since the only other alternative is to render them in VI ;-) | 20:02 |
| inc0 | sdake, https://review.openstack.org/#/c/398534/1 | 20:02 |
| inc0 | sdake, https://review.openstack.org/#/c/398534/4 | 20:02 |
| kfox1111 | rhallisey: I think thie spec is good enough. reading it through cover to cover, its a bit hurky jerky, but its a big complicated thing, so that is probably unavoidable. | 20:02 |
| rhallisey | sbezverk, you could render them with your own tool | 20:02 |
| rhallisey | kfox1111, ya some of the sections have some overlap | 20:03 |
| sdake | inc0 right forgot that part | 20:03 |
| sdake | that was 34 | 20:03 |
| sdake | #4 | 20:03 |
| kfox1111 | sbezverk: the templates are standard golang templates. nothing really helm specific about it. | 20:03 |
| sdake | sorry about that | 20:03 |
| rhallisey | and the ordering isn't perfect due to that | 20:03 |
| inc0 | np, got that figured out | 20:03 |
| kfox1111 | for example, hyou could render them with a golang based webserver pretty easily I think. | 20:03 |
| srwilkers | this looks great | 20:03 |
| sbezverk | kfox1111: ok so then any other golang compatible tools would be supported. I was not aware of them but I trust you :-) | 20:04 |
| *** ppalacios has quit IRC | 20:04 | |
| *** ppalacios has joined #openstack-kolla | 20:05 | |
| *** Pavo has quit IRC | 20:07 | |
| inc0 | https://techcrunch.com/2016/11/16/microsoft-joins-the-linux-foundation/ | 20:10 |
| inc0 | 2016 will be remember as strange year in human history | 20:11 |
| srwilkers | yeah, i saw that | 20:11 |
| jascott1 | trump, M$, whats next? | 20:12 |
| *** Pavo has joined #openstack-kolla | 20:12 | |
| inc0 | brexit | 20:12 |
| *** sean-k-mooney has quit IRC | 20:12 | |
| inc0 | also this year | 20:12 |
| srwilkers | cubs won the world series, check that one off | 20:12 |
| kfox1111 | yeah. and google joined microsoft with .net | 20:12 |
| jascott1 | good addition! | 20:12 |
| kfox1111 | wierd time. :) | 20:12 |
| inc0 | Leonardo di Caprio won Oscar | 20:12 |
| jascott1 | haha | 20:12 |
| *** sean-k-mooney has joined #openstack-kolla | 20:13 | |
| kfox1111 | https://developers.slashdot.org/story/16/11/16/1642201/google-joins-microsofts-net-foundation | 20:13 |
| kfox1111 | feels kind of like april fools day... | 20:13 |
| *** eaguilar_ has quit IRC | 20:13 | |
| inc0 | more like april fools year | 20:13 |
| jascott1 | MBP now effectively suck | 20:14 |
| kfox1111 | yeah. | 20:14 |
| inc0 | you mean with this touchbar thingy? | 20:14 |
| inc0 | I actually think it's neat | 20:14 |
| srwilkers | D: | 20:15 |
| jascott1 | ssd's soldered in | 20:15 |
| inc0 | just need to write VIM bindings to change modes with it;) | 20:15 |
| jascott1 | http://www.macrumors.com/2016/11/15/macbook-pro-touch-bar-non-removable-ssd/ | 20:16 |
| inc0 | not that I'll ever buy it | 20:16 |
| *** ppalacios has quit IRC | 20:23 | |
| *** pomac has quit IRC | 20:28 | |
| sdake | jascott1 one with or without an escape key? | 20:34 |
| sdake | who on earth would take an escape key off a keyboard aimed at developers | 20:34 |
| sdake | oh, I know, APPLE | 20:34 |
| *** sdake has quit IRC | 20:35 | |
| kfox1111 | hehe. annd apple will still make money | 20:35 |
| *** sdake has joined #openstack-kolla | 20:37 | |
| openstackgerrit | Merged openstack/kolla: Move manila-share service to network node https://review.openstack.org/396625 | 20:37 |
| kfox1111 | a little dated, but https://www.youtube.com/watch?v=FL7yD-0pqZg | 20:37 |
| sdake | it can grant out the 3 wishes :) | 20:39 |
| kfox1111 | including getting an iphone. :) | 20:40 |
| inc0 | that being said | 20:41 |
| inc0 | android recently wiped out my phone randomly | 20:41 |
| inc0 | in BCN | 20:41 |
| kfox1111 | heh. yeha. way off track... :) | 20:41 |
| *** pomac has joined #openstack-kolla | 20:41 | |
| *** fguillot has quit IRC | 20:41 | |
| kfox1111 | all devices fail enventually. :/ | 20:41 |
| sdake | i like hte print money thing | 20:42 |
| sdake | pretty epic kfox1111 :) | 20:42 |
| kfox1111 | apple's greatest work is the loyaltee their brand has. | 20:45 |
| kfox1111 | its truely impresive. | 20:45 |
| sdake | i got a mbp because it was the best hardware avilaable at the time | 20:46 |
| sdake | no longer the case | 20:46 |
| *** unicell has quit IRC | 20:46 | |
| sdake | (the alternative was a thinkpad with less memory less disk less resolution) | 20:46 |
| sdake | now my #1 consideration is "does it have an escape key" | 20:46 |
| sdake | i dont even know how i use vi now, imagine if i had to hotkey escape to somethign else ;) | 20:47 |
| kfox1111 | yeah. my laptop 2 laptops ago was a macbook because of the same. it was better hardware for the cost. I put linux on it though. | 20:47 |
| *** matrohon has quit IRC | 20:47 | |
| kfox1111 | escape key is pretty critical. | 20:47 |
| sdake | unfortunately to communicate with my peeps at cisco i need either windows or macos | 20:48 |
| sdake | windows daily would be a torture test | 20:48 |
| srwilkers | can confirm, its shit | 20:48 |
| sdake | can confirm which srwilkers ? | 20:48 |
| srwilkers | took me ~18 months to get something other than a windows machine | 20:48 |
| srwilkers | that windows daily is a torture test | 20:48 |
| sdake | oh right | 20:49 |
| srwilkers | well, also vim would suck ass without an escape key. my mind would explode | 20:49 |
| sdake | i use windows for 1 thing | 20:49 |
| sdake | playing diablo 3 | 20:49 |
| srwilkers | yes | 20:49 |
| sdake | because my mac is too underpowered to run it | 20:49 |
| srwilkers | amped for necro and original diablo act addition | 20:49 |
| sdake | when is that coming out srwilkers | 20:49 |
| sdake | season 6 i was top 100 barb ;) | 20:49 |
| srwilkers | january | 20:50 |
| sdake | haven't played since | 20:50 |
| kfox1111 | I make our windows team deal with my compatability issues. :) | 20:50 |
| sdake | well have played a bit | 20:50 |
| sdake | but not much | 20:50 |
| kfox1111 | they provide a windows terminal server, and I use it. | 20:50 |
| kfox1111 | and report every breakage I see. :) | 20:50 |
| kfox1111 | fortunatly these days I don't have to use windows for much. | 20:51 |
| kfox1111 | mostly the wird encription stuff they like to use. | 20:51 |
| bjolo | escape key? in vim i use ctrl+c | 20:53 |
| *** unicell has joined #openstack-kolla | 20:53 | |
| sdake | huh | 20:53 |
| bjolo | remapped caps lock to control | 20:53 |
| bjolo | yes | 20:53 |
| inc0 | sdake, wintendo is the word | 20:53 |
| bjolo | ctrl-c is the same as escape | 20:53 |
| *** newmember has joined #openstack-kolla | 20:53 | |
| bjolo | im a diehard linux vim guy and i never use escape key | 20:54 |
| sdake | i'm not sure whats worse ;) | 20:54 |
| bjolo | actually thats not all true. i use escape + ) to break out of hp ilo textcons, but thats it | 20:55 |
| bjolo | escape is too slow compared to ctrl+c | 20:56 |
| bjolo | :) | 20:56 |
| *** krtaylor has quit IRC | 20:58 | |
| openstackgerrit | Merged openstack/kolla-ansible: Make service_checks easier to run independently of kolla-ansible https://review.openstack.org/398327 | 20:59 |
| bjolo | time to go to bed for me | 21:02 |
| bjolo | cya all tomorrow | 21:02 |
| bjolo | ~ | 21:03 |
| srwilkers | cya bjolo | 21:06 |
| *** huhaoran has quit IRC | 21:07 | |
| *** Serlex has quit IRC | 21:18 | |
| *** lrensing has quit IRC | 21:22 | |
| *** Serlex has joined #openstack-kolla | 21:29 | |
| *** jtriley_ has joined #openstack-kolla | 21:30 | |
| DaveTurner | Hi sdake | 21:30 |
| *** gfidente has quit IRC | 21:31 | |
| *** Serlex has quit IRC | 21:32 | |
| *** jtriley has quit IRC | 21:32 | |
| *** krtaylor has joined #openstack-kolla | 21:33 | |
| *** Jeffrey4l has quit IRC | 21:33 | |
| *** inc0 has quit IRC | 21:33 | |
| *** jtriley_ has quit IRC | 21:34 | |
| DaveTurner | sdake Based on our discussions last night, we've worked around the rabbitmq / api_interface issue by changing the primary hostname to that of the api_interface. | 21:35 |
| *** krtaylor has quit IRC | 21:37 | |
| DaveTurner | sdake Just to clarify our approach going forward, do see the possibility of updating the rabbitmq role to use IP from network_interface instead of api_interface - or is the assumption driving the existing approach - that the primary hostname would be tied to the api_interface - going to remain? | 21:39 |
| DaveTurner | do *you see the possibility... | 21:39 |
| kfox1111 | my two cents is that should be overridable. | 21:39 |
| kfox1111 | I want to be able to specify rabbit per service and per oslo type. | 21:39 |
| rhallisey | kfox1111, we're good to merge the spec if you want to +a itr | 21:40 |
| rhallisey | it | 21:40 |
| kfox1111 | so one rabbit for nova rpc, but nova notifications go to ceilometer rabbit. | 21:40 |
| kfox1111 | rhallisey: I'm good with it. the one outstanding question is, how long should we wait for others to weigh in? | 21:41 |
| kfox1111 | Adrian or Michael for example. | 21:41 |
| DaveTurner | kfox1111 - that level of flexibility would be optimal. We are looking at customizing the rabbit role for our purposes, but would like to understand where the community stands with this. | 21:41 |
| kfox1111 | DaveTurner: I'm working on kolla-kubernetes, not kolla-ansible so I can't sepeak to that use case, | 21:42 |
| rhallisey | kfox1111, I said on the ML we would close today | 21:42 |
| kfox1111 | but my own clouds need that level of flexability. | 21:42 |
| kfox1111 | rhallisey: any particular time today? | 21:42 |
| rhallisey | can't wait forever and we have majority of kolla-kube cores | 21:42 |
| rhallisey | no particular time | 21:42 |
| rhallisey | EOD | 21:42 |
| kfox1111 | pst cob maybe? | 21:42 |
| rhallisey | sure | 21:42 |
| kfox1111 | k | 21:42 |
| *** sdake_ has joined #openstack-kolla | 21:44 | |
| *** krtaylor has joined #openstack-kolla | 21:45 | |
| *** sdake has quit IRC | 21:47 | |
| *** krtaylor has quit IRC | 21:47 | |
| *** krtaylor has joined #openstack-kolla | 21:50 | |
| *** krtaylor has quit IRC | 21:53 | |
| *** ayoung has quit IRC | 21:55 | |
| *** pomac has quit IRC | 21:55 | |
| *** krtaylor has joined #openstack-kolla | 21:56 | |
| *** tonanhngo_ has joined #openstack-kolla | 21:58 | |
| *** krtaylor has quit IRC | 21:59 | |
| *** krtaylor has joined #openstack-kolla | 22:00 | |
| *** tonanhngo has quit IRC | 22:01 | |
| *** srwilkers has quit IRC | 22:02 | |
| *** tonanhngo_ has quit IRC | 22:03 | |
| *** Pavo has quit IRC | 22:07 | |
| sbezverk | kfox1111: that helm doc link you gave me is pretty good. well written and easy to understand :-) | 22:08 |
| *** adrian_otto has quit IRC | 22:08 | |
| *** pomac has joined #openstack-kolla | 22:12 | |
| *** TxGirlGeek has quit IRC | 22:12 | |
| *** Pavo has joined #openstack-kolla | 22:12 | |
| *** TxGirlGeek has joined #openstack-kolla | 22:15 | |
| *** rhallisey has quit IRC | 22:17 | |
| sdake_ | sbezverk link plz | 22:18 |
| *** TxGirlGeek has quit IRC | 22:22 | |
| *** chas has joined #openstack-kolla | 22:23 | |
| *** TxGirlGeek has joined #openstack-kolla | 22:23 | |
| sbezverk | sdake: here you go: https://github.com/kubernetes/helm/tree/master/ | 22:24 |
| sbezverk | check the doc portion.. | 22:24 |
| *** TxGirlGeek has quit IRC | 22:25 | |
| *** TxGirlGeek has joined #openstack-kolla | 22:25 | |
| *** tonanhngo has joined #openstack-kolla | 22:26 | |
| *** absubram has quit IRC | 22:26 | |
| *** chas has quit IRC | 22:27 | |
| *** TxGirlGeek has quit IRC | 22:29 | |
| *** tonanhngo has quit IRC | 22:30 | |
| *** TxGirlGeek has joined #openstack-kolla | 22:30 | |
| *** chas has joined #openstack-kolla | 22:31 | |
| *** TxGirlGeek has quit IRC | 22:31 | |
| *** TxGirlGeek has joined #openstack-kolla | 22:32 | |
| *** chas has quit IRC | 22:33 | |
| *** chas has joined #openstack-kolla | 22:34 | |
| kfox1111 | sdake_: yeah. pretty surprising for something not even out of alpha yet. :) | 22:35 |
| kfox1111 | sbezverk: yeah. pretty surprising for something not even out of alpha yet. :) | 22:35 |
| sdake_ | huh? | 22:35 |
| kfox1111 | sdake_: sorry. bad tab. | 22:35 |
| sdake_ | roger | 22:35 |
| *** adrian_otto has joined #openstack-kolla | 22:35 | |
| *** absubram has joined #openstack-kolla | 22:37 | |
| *** schwicht has quit IRC | 22:37 | |
| *** TxGirlGeek has quit IRC | 22:38 | |
| *** TxGirlGeek has joined #openstack-kolla | 22:38 | |
| sdake_ | britthouser | 22:42 |
| sdake_ | i'm in need of your help | 22:42 |
| sdake_ | britthouser you happen to be around? | 22:42 |
| sbezverk | kfox1111: hm, I found rc2 version | 22:43 |
| *** alteriks has quit IRC | 22:43 | |
| *** ayoung has joined #openstack-kolla | 22:45 | |
| kfox1111 | beta2 I guess is what I saw. | 22:46 |
| kfox1111 | https://kubernetes-helm.storage.googleapis.com/helm-v2.0.0-beta.2-linux-amd64.tar.gz | 22:46 |
| kfox1111 | not alpha. | 22:46 |
| kfox1111 | I think they were talking about having 4. | 22:46 |
| sbezverk | helm-v2.0.0-rc.2-linux-amd64.zip | 22:46 |
| kfox1111 | hmm... where did you find that? | 22:47 |
| kfox1111 | oh. off the release page. | 22:47 |
| *** srwilkers has joined #openstack-kolla | 22:47 | |
| sbezverk | http://storage.googleapis.com/kubernetes-helm/helm-v2.0.0-rc.2-linux-amd64.tar.gz | 22:47 |
| kfox1111 | hmm... their main readme's out of date. | 22:48 |
| kfox1111 | not surprising I guess. | 22:48 |
| kfox1111 | so helm's final release is even closer then I thought. :) | 22:50 |
| Pavo | evening | 22:50 |
| *** schwicht has joined #openstack-kolla | 22:53 | |
| srwilkers | hey Pavo | 22:59 |
| Pavo | evening srwilkers | 23:01 |
| *** jheroux has quit IRC | 23:04 | |
| *** sdake has joined #openstack-kolla | 23:06 | |
| *** dave-mccowan has quit IRC | 23:07 | |
| *** krtaylor has quit IRC | 23:09 | |
| *** sdake_ has quit IRC | 23:10 | |
| Pavo | question, in deploying newton with kolla-ansible is the default deployment only support QEMU images or can you also use Docker images also at the same time? | 23:13 |
| *** sdake has quit IRC | 23:16 | |
| Pavo | or is that a dumb question? | 23:16 |
| *** eaguilar has joined #openstack-kolla | 23:21 | |
| *** diogogmt has quit IRC | 23:25 | |
| *** schwicht has quit IRC | 23:29 | |
| *** schwicht has joined #openstack-kolla | 23:30 | |
| *** Jeffrey4l has joined #openstack-kolla | 23:31 | |
| *** lamt has quit IRC | 23:33 | |
| *** schwicht has quit IRC | 23:34 | |
| *** f13o_ has quit IRC | 23:37 | |
| *** zhugaoxiao has joined #openstack-kolla | 23:38 | |
| *** khamtamtun has joined #openstack-kolla | 23:39 | |
| *** Administrator_ has quit IRC | 23:40 | |
| *** sdake has joined #openstack-kolla | 23:40 | |
| sdake | pavo no | 23:41 |
| sdake | pavo you could theoretically use the docker hypervisor | 23:41 |
| *** inc0 has joined #openstack-kolla | 23:41 | |
| sdake | pavo another optoin is to use a COE such as Magnum provides like kubernetes | 23:42 |
| inc0 | I'm back my dear kollagues | 23:42 |
| sdake | sup inc0 | 23:42 |
| *** khamtamtun has quit IRC | 23:42 | |
| inc0 | I went to our favorite mexican restaurant, and this is contentous space in souther texas | 23:42 |
| inc0 | so yeah, I'm good | 23:42 |
| *** david-lyle_ has joined #openstack-kolla | 23:50 | |
| *** david-lyle has quit IRC | 23:51 | |
| srwilkers | love the mexican food in texas | 23:53 |
| srwilkers | my brother lives in austin, so i visit about 3 times a year, and my favorite part is the food | 23:53 |
| *** TxGirlGeek has quit IRC | 23:55 | |
| jascott1 | srwilkers: had a couple of tasty steaks in San Antonio (inc0's neck of the woods) and they had both kinds of beer "Dos Equis" and "Shiner Bock" | 23:59 |
| jascott1 | of course I am spoiled on the beer being in PDX | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!