| *** david-lyle has joined #openstack-kolla | 00:03 | |
| *** yingjun has joined #openstack-kolla | 00:05 | |
| *** manheim has joined #openstack-kolla | 00:06 | |
| *** ducttape_ has joined #openstack-kolla | 00:07 | |
| *** ducttap__ has joined #openstack-kolla | 00:10 | |
| *** ducttape_ has quit IRC | 00:10 | |
| *** openstackgerrit has joined #openstack-kolla | 00:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/474652 | 00:12 |
|---|---|---|
| *** ducttap__ has quit IRC | 00:15 | |
| *** itlinux has quit IRC | 00:17 | |
| *** itlinux_ has quit IRC | 00:17 | |
| *** awidders_ has quit IRC | 00:17 | |
| *** awiddersheim has joined #openstack-kolla | 00:17 | |
| *** mattmceuen has quit IRC | 00:27 | |
| *** jascott1 has quit IRC | 00:27 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/474652 | 00:28 |
| *** zhangfei has quit IRC | 00:30 | |
| *** zhurong has joined #openstack-kolla | 00:31 | |
| *** masber has quit IRC | 00:33 | |
| *** masber has joined #openstack-kolla | 00:33 | |
| *** krtaylor has joined #openstack-kolla | 00:40 | |
| *** manheim has quit IRC | 00:40 | |
| *** schwicht has joined #openstack-kolla | 00:43 | |
| *** zhangfei has joined #openstack-kolla | 00:43 | |
| *** jascott1 has joined #openstack-kolla | 00:45 | |
| *** goldyfruit has joined #openstack-kolla | 00:57 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add vmware support to ceilometer https://review.openstack.org/462842 | 00:59 |
| *** schwicht has quit IRC | 00:59 | |
| *** dave-mcc_ has joined #openstack-kolla | 01:00 | |
| *** itlinux has joined #openstack-kolla | 01:01 | |
| *** itlinux_ has joined #openstack-kolla | 01:01 | |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/474652 | 01:05 |
| *** kevinz has joined #openstack-kolla | 01:10 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/485905 | 01:10 |
| *** rwallner has joined #openstack-kolla | 01:15 | |
| *** ducttape_ has joined #openstack-kolla | 01:15 | |
| *** mattmceuen has joined #openstack-kolla | 01:18 | |
| *** rwallner has quit IRC | 01:20 | |
| *** ducttape_ has quit IRC | 01:20 | |
| *** ansmith has quit IRC | 01:28 | |
| *** manheim has joined #openstack-kolla | 01:37 | |
| emccormick | hawi, if you're still here, define the variables in group_vars or host_vars rather than in the inventory file | 01:40 |
| *** itlinux has quit IRC | 01:49 | |
| *** itlinux_ has quit IRC | 01:49 | |
| *** caowei has quit IRC | 01:50 | |
| *** manheim has quit IRC | 01:53 | |
| *** tvignaud has quit IRC | 01:58 | |
| *** itlinux has joined #openstack-kolla | 02:04 | |
| *** itlinux_ has joined #openstack-kolla | 02:04 | |
| *** caowei has joined #openstack-kolla | 02:16 | |
| *** ducttape_ has joined #openstack-kolla | 02:16 | |
| *** ducttape_ has quit IRC | 02:21 | |
| *** ducttap__ has joined #openstack-kolla | 02:21 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: [TEST][MASTER][KOLLA-ANSIBLE]TEST CLEAN CEILOMETER https://review.openstack.org/475670 | 02:21 |
| *** ducttap__ has quit IRC | 02:23 | |
| *** ducttape_ has joined #openstack-kolla | 02:23 | |
| *** ducttape_ has quit IRC | 02:28 | |
| spsurya | morning all | 02:28 |
| Seyong | spsurya: good morning | 02:28 |
| *** yangyapeng has joined #openstack-kolla | 02:31 | |
| *** jamesbenson has joined #openstack-kolla | 02:32 | |
| *** ducttape_ has joined #openstack-kolla | 02:33 | |
| sbezverk | spsurya Jeffrey4l hopefully last attempt to fix tarball generation pipeline. https://review.openstack.org/#/c/485739/ | 02:33 |
| spsurya | sbezverk: roger | 02:35 |
| sbezverk | spsurya thanks a lot | 02:35 |
| *** jamesbenson has quit IRC | 02:36 | |
| *** dave-mcc_ has quit IRC | 02:42 | |
| *** yingjun has quit IRC | 02:44 | |
| *** ducttape_ has quit IRC | 02:45 | |
| *** ducttape_ has joined #openstack-kolla | 02:47 | |
| *** ducttape_ has quit IRC | 02:48 | |
| *** ducttap__ has joined #openstack-kolla | 02:48 | |
| *** ducttap__ has quit IRC | 02:49 | |
| *** yingjun has joined #openstack-kolla | 02:51 | |
| *** goldyfruit has quit IRC | 02:52 | |
| *** ducttape_ has joined #openstack-kolla | 02:57 | |
| *** ducttape_ has quit IRC | 02:59 | |
| *** ducttape_ has joined #openstack-kolla | 02:59 | |
| *** eaguilar has quit IRC | 02:59 | |
| *** ducttape_ has quit IRC | 03:04 | |
| *** ducttape_ has joined #openstack-kolla | 03:07 | |
| openstackgerrit | jimmygc proposed openstack/kolla master: Add collectd support to telegraf https://review.openstack.org/485923 | 03:08 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Optimize reconfiguration for mariadb https://review.openstack.org/433480 | 03:10 |
| *** ducttape_ has quit IRC | 03:11 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add collectd support to telegraf container https://review.openstack.org/485925 | 03:12 |
| openstackgerrit | jimmygc proposed openstack/kolla master: Add collectd support to telegraf https://review.openstack.org/485923 | 03:14 |
| *** caowei has quit IRC | 03:15 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Support customizing grafana home dashboard https://review.openstack.org/485931 | 03:33 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add collectd support to telegraf container https://review.openstack.org/485925 | 03:35 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Disable grafana Getting Started panel https://review.openstack.org/485932 | 03:42 |
| *** jamesbenson has joined #openstack-kolla | 03:44 | |
| openstackgerrit | Merged openstack/kolla master: Fix tarball name again https://review.openstack.org/485739 | 03:45 |
| *** jamesbenson has quit IRC | 03:49 | |
| *** kevinz has quit IRC | 04:31 | |
| *** iniazi_ has quit IRC | 04:32 | |
| *** iniazi has joined #openstack-kolla | 04:32 | |
| *** jamesbenson has joined #openstack-kolla | 04:41 | |
| *** jamesbenson has quit IRC | 04:46 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/ocata: Add pipeline file into ceilometer-central and ceilomter-compute node https://review.openstack.org/485017 | 05:07 |
| *** ducttape_ has joined #openstack-kolla | 05:08 | |
| *** ducttape_ has quit IRC | 05:12 | |
| *** ducttap__ has joined #openstack-kolla | 05:12 | |
| *** ducttap__ has quit IRC | 05:17 | |
| *** omenv has joined #openstack-kolla | 05:17 | |
| *** jamesbenson has joined #openstack-kolla | 05:23 | |
| *** skramaja has joined #openstack-kolla | 05:26 | |
| *** jamesbenson has quit IRC | 05:27 | |
| *** mewald has joined #openstack-kolla | 05:31 | |
| *** coolsvap has joined #openstack-kolla | 05:43 | |
| *** yangyapeng has quit IRC | 05:46 | |
| *** yangyapeng has joined #openstack-kolla | 05:47 | |
| *** manheim has joined #openstack-kolla | 05:49 | |
| *** manheim has quit IRC | 05:54 | |
| *** wlfightup_ has joined #openstack-kolla | 05:58 | |
| *** wlfightup has quit IRC | 06:02 | |
| *** wlfightup_ is now known as wlfightup | 06:02 | |
| *** jbadiapa has joined #openstack-kolla | 06:03 | |
| *** emccormick has quit IRC | 06:04 | |
| *** jamesbenson has joined #openstack-kolla | 06:04 | |
| *** jamesbenson has quit IRC | 06:09 | |
| *** magicboiz has joined #openstack-kolla | 06:13 | |
| *** magicboiz has joined #openstack-kolla | 06:18 | |
| openstackgerrit | octopuszhang proposed openstack/kolla-ansible master: Add ntp start and enable commands on ubuntu https://review.openstack.org/485984 | 06:19 |
| *** caowei has joined #openstack-kolla | 06:22 | |
| *** magicboiz has quit IRC | 06:23 | |
| *** magicboiz has joined #openstack-kolla | 06:23 | |
| *** jamesbenson has joined #openstack-kolla | 06:46 | |
| *** athomas has joined #openstack-kolla | 06:49 | |
| *** jamesbenson has quit IRC | 06:51 | |
| *** mewald has quit IRC | 06:54 | |
| *** manheim has joined #openstack-kolla | 06:55 | |
| *** mewald has joined #openstack-kolla | 06:55 | |
| *** gfidente has joined #openstack-kolla | 06:57 | |
| *** gfidente has quit IRC | 06:57 | |
| *** gfidente has joined #openstack-kolla | 06:57 | |
| *** pc_m has quit IRC | 06:58 | |
| *** pc_m has joined #openstack-kolla | 07:00 | |
| *** manheim has quit IRC | 07:00 | |
| *** manheim has joined #openstack-kolla | 07:03 | |
| *** pc_m has quit IRC | 07:07 | |
| *** manheim has quit IRC | 07:07 | |
| *** pc_m has joined #openstack-kolla | 07:08 | |
| *** sayantani01 has quit IRC | 07:08 | |
| *** shasha_t_ has quit IRC | 07:08 | |
| *** omenv has quit IRC | 07:12 | |
| *** ducttape_ has joined #openstack-kolla | 07:14 | |
| zhangfei | hi, when deploy kolla, mariadb report error: Info: Using unique option prefix 'exec' is error-prone and can break in the future. Please use the full name 'execute' instead. | 07:15 |
| zhangfei | Any clue? thanks | 07:15 |
| zhangfei | the container bootstrap_mariadb exicted | 07:16 |
| zhangfei | debian-source-mariadb:5.0.0 "kolla_start" 17 minutes ago Exited (1) 16 minutes ago | 07:16 |
| *** zhurong has quit IRC | 07:18 | |
| *** ducttape_ has quit IRC | 07:18 | |
| *** mgoddard_ has joined #openstack-kolla | 07:22 | |
| *** omenv has joined #openstack-kolla | 07:23 | |
| *** itlinux has quit IRC | 07:26 | |
| *** itlinux_ has quit IRC | 07:26 | |
| *** mewald1 has joined #openstack-kolla | 07:26 | |
| *** jamesbenson has joined #openstack-kolla | 07:28 | |
| *** mewald has quit IRC | 07:29 | |
| *** jamesbenson has quit IRC | 07:33 | |
| *** ddyer has quit IRC | 07:42 | |
| *** ddyer has joined #openstack-kolla | 07:42 | |
| openstackgerrit | Mathias Ewald proposed openstack/kolla-ansible master: Add role to deploy prometheus https://review.openstack.org/485150 | 07:49 |
| *** zhurong has joined #openstack-kolla | 07:59 | |
| openstackgerrit | Merged openstack/kolla master: Compile i18n messages for all projects who have locale folder https://review.openstack.org/484670 | 08:01 |
| *** rmart04 has joined #openstack-kolla | 08:04 | |
| *** dims has quit IRC | 08:09 | |
| *** jamesbenson has joined #openstack-kolla | 08:09 | |
| *** manheim has joined #openstack-kolla | 08:11 | |
| *** dims has joined #openstack-kolla | 08:11 | |
| *** jamesbenson has quit IRC | 08:13 | |
| *** ducttape_ has joined #openstack-kolla | 08:15 | |
| *** ducttape_ has quit IRC | 08:19 | |
| *** yangyape_ has joined #openstack-kolla | 08:20 | |
| *** manheim has quit IRC | 08:20 | |
| *** egonzalez has joined #openstack-kolla | 08:21 | |
| *** dims has quit IRC | 08:21 | |
| *** manheim has joined #openstack-kolla | 08:21 | |
| *** dims has joined #openstack-kolla | 08:23 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Add default volume_backend_name for rbd driver in cinder volume https://review.openstack.org/486043 | 08:26 |
| *** omenv has quit IRC | 08:28 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add 777 permission to kolla gate logs https://review.openstack.org/485723 | 08:28 |
| *** serlex has joined #openstack-kolla | 08:28 | |
| *** jascott1 has quit IRC | 08:30 | |
| *** jascott1 has joined #openstack-kolla | 08:30 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Add openstack logs in deploy gates https://review.openstack.org/474539 | 08:33 |
| *** jascott1 has quit IRC | 08:35 | |
| *** jmccarthy has joined #openstack-kolla | 08:36 | |
| *** yangyape_ has quit IRC | 08:37 | |
| *** tvignaud has joined #openstack-kolla | 08:38 | |
| *** omenv has joined #openstack-kolla | 08:39 | |
| openstackgerrit | octopuszhang proposed openstack/kolla-ansible master: Add ntp start and enable commands on ubuntu https://review.openstack.org/485984 | 08:42 |
| *** omenv has quit IRC | 08:47 | |
| *** shardy has joined #openstack-kolla | 08:54 | |
| *** ramishra has quit IRC | 09:00 | |
| *** ramishra has joined #openstack-kolla | 09:02 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Add ujson into binary install type https://review.openstack.org/484543 | 09:04 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla-ansible master: Fix grafana data source registration https://review.openstack.org/486056 | 09:06 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/ocata: Compile i18n messages for all projects who have locale folder https://review.openstack.org/486060 | 09:08 |
| mewald1 | egonzalez: I'll remove the prometheus part and supply another patch once prometheus and the grafana patch are merged | 09:12 |
| egonzalez | mewald1, that will speed grafana change :) | 09:13 |
| mewald1 | yes | 09:13 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla-ansible master: Fix grafana data source registration https://review.openstack.org/486056 | 09:13 |
| *** yingjun has quit IRC | 09:14 | |
| mewald1 | egonzalez: shall I rebase prometheus patch on the grafana patch? what do you think? | 09:14 |
| mewald1 | the grafana one seems obvious and simple and should be mergeable quickly | 09:14 |
| *** ducttape_ has joined #openstack-kolla | 09:15 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Remove python-wsme and python-pecan packages for centos https://review.openstack.org/486063 | 09:17 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Add ujson into binary install type https://review.openstack.org/484543 | 09:17 |
| egonzalez | mewald1, added other comment, except this jinja thing, lgtm | 09:17 |
| mewald1 | So "when: item.value.enable | bool" is what it should look like? | 09:19 |
| *** ducttape_ has quit IRC | 09:20 | |
| egonzalez | mewald1, yep | 09:21 |
| mewald1 | done | 09:22 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla-ansible master: Fix grafana data source registration https://review.openstack.org/486056 | 09:22 |
| *** lpetrut has joined #openstack-kolla | 09:23 | |
| mewald1 | egonzalez: I dont understand your comment: remote "". | 09:31 |
| *** omenv has joined #openstack-kolla | 09:31 | |
| egonzalez | mewald1, was Jeffrey4l's but refers to remove double colons | 09:32 |
| mewald1 | ahh right, sorry about that | 09:32 |
| *** sambetts|afk is now known as sambetts | 09:33 | |
| *** omenvl has joined #openstack-kolla | 09:33 | |
| *** omenv has quit IRC | 09:36 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: Update the documentation link for doc migration https://review.openstack.org/485161 | 09:39 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla-ansible master: Fix grafana data source registration https://review.openstack.org/486056 | 09:40 |
| *** pbourke_ has joined #openstack-kolla | 09:47 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: Update the documentation link for doc migration https://review.openstack.org/485161 | 09:49 |
| *** palexster has quit IRC | 09:50 | |
| openstackgerrit | Chason Chan proposed openstack/kolla master: Update the documentation link for doc migration https://review.openstack.org/485151 | 09:50 |
| *** palexster has joined #openstack-kolla | 09:56 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging in gates https://review.openstack.org/485723 | 09:58 |
| *** jamesbenson has joined #openstack-kolla | 10:02 | |
| *** slagle has quit IRC | 10:04 | |
| *** jamesbenson has quit IRC | 10:07 | |
| *** jiriprox has quit IRC | 10:12 | |
| openstackgerrit | Merged openstack/kolla master: Add sfc plugin into neutron-base image https://review.openstack.org/461748 | 10:21 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Remove python-wsme and python-pecan packages for centos https://review.openstack.org/486063 | 10:21 |
| *** omenvl has quit IRC | 10:23 | |
| *** ramishra has quit IRC | 10:24 | |
| *** omenv has joined #openstack-kolla | 10:25 | |
| *** ramishra has joined #openstack-kolla | 10:26 | |
| *** skramaja has quit IRC | 10:38 | |
| *** pc_m has quit IRC | 10:41 | |
| mewald1 | egonzalez: What wrong with the prometheus containers? I have no experience in interpreting the jenkins checks outputs | 10:41 |
| egonzalez | sean-k-mooney, added a few more comments in the ovs change | 10:42 |
| egonzalez | *ovs-dpdk rather ovs | 10:42 |
| *** yangyapeng has quit IRC | 10:43 | |
| sean-k-mooney | egonzalez: cool ill take a look at them | 10:47 |
| sean-k-mooney | egonzalez: still seting up a centos enviornment to test | 10:48 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix grafana data source registration https://review.openstack.org/486056 | 10:48 |
| egonzalez | sean-k-mooney, as said in the review, if not have time for pike, add in the release note(also add release note btw hehe) that is only supported for debian based distros | 10:48 |
| *** zhurong has quit IRC | 10:50 | |
| *** pc_m has joined #openstack-kolla | 10:50 | |
| sean-k-mooney | egonzalez: as in the continers only work for debina based distros or the deployment code? im going to add the network config scrpts for redhat family | 10:52 |
| sean-k-mooney | egonzalez: with the template override the centos container also works. at least when deployed on a ubuntu host. | 10:53 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 11:03 |
| openstackgerrit | Merged openstack/kolla-ansible master: Update the documentation link for doc migration https://review.openstack.org/485161 | 11:06 |
| *** rwallner has joined #openstack-kolla | 11:08 | |
| *** hieulq has quit IRC | 11:09 | |
| *** shardy is now known as shardy_lunch | 11:10 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Do Not Merge, kolla-kubernetes gate tests https://review.openstack.org/461063 | 11:26 |
| *** coolsvap has quit IRC | 11:27 | |
| openstackgerrit | Mathias Ewald proposed openstack/kolla master: Fix redis container start sequence https://review.openstack.org/486109 | 11:28 |
| *** omenvl has joined #openstack-kolla | 11:31 | |
| wlfightup | moring | 11:33 |
| wlfightup | egonzalez | 11:33 |
| *** omenv has quit IRC | 11:33 | |
| wlfightup | tunnel_interface also inc have ip address ? | 11:33 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla-ansible master: Add Redis role https://review.openstack.org/486110 | 11:33 |
| mewald1 | wlfightup: yes the tunnel interface must have an IP | 11:34 |
| wlfightup | why? | 11:34 |
| *** schwicht has joined #openstack-kolla | 11:35 | |
| wlfightup | need ip address why comunicate between vm and vm ? | 11:35 |
| wlfightup | I have always been the production environment for this card has never been ip | 11:36 |
| wlfightup | mewald1? | 11:36 |
| wlfightup | mewald1 | 11:36 |
| mewald1 | then try without and check ;) | 11:36 |
| wlfightup | Suddenly feel this is still very immature | 11:37 |
| *** shardy_lunch is now known as shardy | 11:38 | |
| mewald1 | dude, the tunnel interface is used for VXLAN and that protocol runs on UDP and therefore needs IP, too. VXLAN travels from host to host and carries encapsulated VM traffic. When two hosts talk to each others they need to have an IP address. | 11:39 |
| wlfightup | vxlan need physical card have an IP address ? | 11:42 |
| wlfightup | are you sure? | 11:42 |
| wlfightup | i don't assign any ip address in physical always | 11:44 |
| mewald1 | then what do you set in ml2_conf.ini [ovs] local_ip ? | 11:45 |
| mewald1 | @wlfightup Please send me PM since this discussion is not kolla related. | 11:48 |
| wlfightup | i use linuxbridge | 11:50 |
| wlfightup | PM? what is PM? where are you from? | 11:50 |
| egonzalez | wlfightup, you are missunderstanding br-tun with tunnel_interface, tunnel_interface is where tunnels are going to be created, need an IP to create the tunnel endpoint. br-tun is an ovs bridge where tunneled connections run | 11:51 |
| egonzalez | wlfightup, if check ovs-vsctl show will se br-tun bridge, and connections with IP addresses, those IP addresses are the tunnel endpoints (tunnel_interface IPs for each network/compute node) | 11:52 |
| *** omenvl has quit IRC | 11:57 | |
| *** omenv has joined #openstack-kolla | 11:58 | |
| *** omenvl has joined #openstack-kolla | 12:00 | |
| *** omenv has quit IRC | 12:03 | |
| mewald1 | egonzalez: all checks passed now for the prometheus containers | 12:08 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 12:13 |
| *** slagle has joined #openstack-kolla | 12:15 | |
| egonzalez | mewald1, is this really needed? ENV prometheus_version=1.7.1 \ | 12:22 |
| egonzalez | prometheus_arch={{ base_arch }} | 12:22 |
| egonzalez | re second prometheus_arch, if is overiden in next lines, makes sense set the arch twice? | 12:22 |
| *** goldyfruit has joined #openstack-kolla | 12:25 | |
| *** ansmith has joined #openstack-kolla | 12:27 | |
| *** iniazi_ has joined #openstack-kolla | 12:29 | |
| *** mewald1 has quit IRC | 12:30 | |
| *** iniazi has quit IRC | 12:32 | |
| *** hrw has quit IRC | 12:32 | |
| *** hrw has joined #openstack-kolla | 12:34 | |
| *** mnaser has left #openstack-kolla | 12:35 | |
| *** goldyfruit has quit IRC | 12:40 | |
| *** dave-mccowan has joined #openstack-kolla | 12:41 | |
| *** mewald has joined #openstack-kolla | 12:42 | |
| *** dave-mcc_ has joined #openstack-kolla | 12:46 | |
| mewald | egonzalez: I followed the pattern that helm uses. By default prometheus_arch is set to whatever base_arch is. Since base_arch values don't match the values used by prometheus, we have to overwrite prometheus_arch. | 12:46 |
| mewald | I can hardcode the amd64 bit of the download url but this gave me a comment by sbezverk to do it the way helm does it | 12:48 |
| *** dave-mccowan has quit IRC | 12:48 | |
| *** jrist has quit IRC | 12:49 | |
| *** mattmceuen has quit IRC | 12:51 | |
| *** rhallisey has joined #openstack-kolla | 12:51 | |
| *** awiddersheim has quit IRC | 12:52 | |
| *** awiddersheim has joined #openstack-kolla | 12:52 | |
| mewald | egonzalez: Also, if we remove line 8 and leave 11 through 15 intact, what will be haproxy_exporter_arch for architecture other than x86_64 and aarch64? Line 8 sets a default. | 12:53 |
| *** goldyfruit has joined #openstack-kolla | 12:54 | |
| *** pbourke has quit IRC | 13:00 | |
| *** eaguilar has joined #openstack-kolla | 13:00 | |
| *** pbourke has joined #openstack-kolla | 13:02 | |
| *** jrist has joined #openstack-kolla | 13:03 | |
| *** ramishra has quit IRC | 13:03 | |
| *** emccormick has joined #openstack-kolla | 13:04 | |
| *** ramishra has joined #openstack-kolla | 13:05 | |
| *** goldyfruit has quit IRC | 13:07 | |
| *** caowei has quit IRC | 13:07 | |
| *** ducttape_ has joined #openstack-kolla | 13:08 | |
| *** yangyapeng has joined #openstack-kolla | 13:10 | |
| mewald | egonzalez: I am not going to fight for it :D if you tell me how you would like to have it I'll change it | 13:11 |
| *** magicboiz has quit IRC | 13:11 | |
| *** ducttape_ has quit IRC | 13:12 | |
| sbezverk | egonzalez it would be great to support not just x86_64 | 13:32 |
| sbezverk | that was the reason for my original comment for prometeus | 13:32 |
| rwellum | Morning sbezverk | 13:33 |
| *** lpetrut has quit IRC | 13:35 | |
| sbezverk | rwellum good morning | 13:35 |
| *** lucasxu has joined #openstack-kolla | 13:37 | |
| rwellum | sbezverk: I am re-running 5.0 - will report back. Somehow last night, I kept losing my sessions - but hopefully undisturbed today. I had an unrelated question though. I am still trying to build my own source code for a Service. The image appears to build, I appear to use it when I deploy, but somehow it's not my source code running. | 13:37 |
| rwellum | sbezverk: I know this because the version of Horizon I am deploying should have some proprietary labeling which I don't see. | 13:38 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla-ansible master: Add Redis role https://review.openstack.org/486110 | 13:38 |
| rwellum | sbezverk: so this is what I get when I generate. It gets tag 4.0.2. https://www.irccloud.com/pastebin/QoGnfjiO/ | 13:39 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla master: Fix redis container start sequence https://review.openstack.org/486109 | 13:39 |
| rwellum | sbezverk: I've had 5.0.0 as a tag before - not really sure if that's deterministic or not. | 13:41 |
| wlfightup | egonzalez: if is vlan network, only br-int, how to config ? | 13:42 |
| *** zhangfei has quit IRC | 13:44 | |
| sbezverk | rwellum: now you have 4.0.2 on your local machine.. but you need to informa horizon chart to use it, otherwise it will use default 4.0.0 tag | 13:45 |
| rwellum | sbezverk: yes by modifying cloud.yaml | 13:45 |
| *** hawi has quit IRC | 13:46 | |
| rwellum | sbezverk https://www.irccloud.com/pastebin/0AelWHqM/ | 13:46 |
| sbezverk | rwellum looks good | 13:50 |
| *** mgoddard_ has quit IRC | 13:53 | |
| rwellum | sbezverk: ok - doing this again - will see :). If I don't see the Horizon I expect, then I'll need to debug further. | 13:54 |
| *** jtriley has quit IRC | 13:54 | |
| *** zhangfei has joined #openstack-kolla | 13:56 | |
| sbezverk | rwellum the best way to confirm is kubectl describe pod horizon-blah -n kolla | | grep Image | 13:56 |
| sbezverk | sorry it is a single pipe | 13:56 |
| rwellum | That's helpful thanks sbezverk | 13:57 |
| rwellum | sbezverk: on the 5.0 looks like cinder is crashing | 13:57 |
| *** omenvl has quit IRC | 13:58 | |
| sbezverk | rwellum there are a handful cinder pods, which one is crashing? | 13:58 |
| *** hawi has joined #openstack-kolla | 13:58 | |
| rwellum | cinder-volume-rwdj8 | 13:59 |
| egonzalez | sbezverk, mewald , actually I was not asking to make only x86, is for removal of double setting the arch environment for arch64 and x86, if supporting multiple architectures, ppc64 is also implemented in kolla too | 14:02 |
| *** omenvl has joined #openstack-kolla | 14:03 | |
| egonzalez | also maybemakes sense to only define arch env in prometheus-base image, all other will inherit from it hence having the same arch | 14:04 |
| *** omenvlad has joined #openstack-kolla | 14:04 | |
| sbezverk | egonzalez not sure if it right approach as different components might be running on diff arc hosts | 14:06 |
| sbezverk | egonzalez or at least a components should be able to override base arc is required.. | 14:07 |
| sbezverk | s/is/if/ | 14:07 |
| *** omenvl has quit IRC | 14:07 | |
| egonzalez | sbezverk, image names will be different and will need to be changed in deployment method, an arch64 image cannot have ppc64 as parent image | 14:08 |
| sbezverk | egonzalez got it, you are right | 14:09 |
| mewald | sbezverk: egonzalez: so what is the best way to do it now? | 14:10 |
| *** srnbckr has quit IRC | 14:17 | |
| *** schwicht has quit IRC | 14:17 | |
| *** mewald has quit IRC | 14:17 | |
| *** mattmceuen has joined #openstack-kolla | 14:22 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 14:23 |
| *** jrist has quit IRC | 14:26 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 14:29 |
| *** jrist has joined #openstack-kolla | 14:39 | |
| *** jamesbenson has joined #openstack-kolla | 14:41 | |
| *** serlex has quit IRC | 14:50 | |
| *** mewald has joined #openstack-kolla | 14:50 | |
| *** schwicht has joined #openstack-kolla | 14:56 | |
| *** rmart04 has quit IRC | 14:59 | |
| *** gfidente is now known as gfidente|afk | 14:59 | |
| *** gfidente has joined #openstack-kolla | 15:00 | |
| *** gfidente has quit IRC | 15:00 | |
| *** gfidente has joined #openstack-kolla | 15:00 | |
| *** shardy is now known as shardy_mtg | 15:00 | |
| mewald | egonzalez: so you want the arch stuff in the base image with an if-elsif-else conditional, would that work for you? | 15:01 |
| *** ducttape_ has joined #openstack-kolla | 15:10 | |
| inc0 | good morning | 15:10 |
| inc0 | sbezverk: you have images available | 15:10 |
| sbezverk | inc0: thanks and I have tested, so now we have two green master jobs centos and ubuntu | 15:11 |
| *** itlinux has joined #openstack-kolla | 15:11 | |
| *** itlinux_ has joined #openstack-kolla | 15:11 | |
| inc0 | great news | 15:12 |
| mewald | I am a bit confused about the deployment model we're running in multinode inventory file: The is a monitoring group which (as I assume) is supposed to run monitoring services. For me this include elasticsearch, kibana, influxdb, grafana, prometheus and possibly a few more (skydive, do clue what that is). most of them are deployed to the control nodes though. The question came up when egonzalez commented on my redis role that is | 15:13 |
| mewald | should go on the control nodes (https://review.openstack.org/#/c/486110/2/ansible/inventory/all-in-one) I intend to use redis for monitoring with sensu and therefore it should go to the monitoring nodes. I feel like this is a broader topic that is difficult to discuss in gerrit comments which is why I am writing this here. What is the overall strategy for this type of issue? | 15:13 |
| *** ducttape_ has quit IRC | 15:14 | |
| inc0 | mewald: afair reason we have separate monitoring group is that we used InfluxDB, which is single node | 15:15 |
| inc0 | on free license | 15:15 |
| inc0 | it was never finished tho so we still can move it to control/remove monitoring group as far as I'm concerned | 15:16 |
| *** gfidente has quit IRC | 15:16 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix logging collection in gates https://review.openstack.org/485723 | 15:17 |
| *** jrist has quit IRC | 15:18 | |
| *** gfidente has joined #openstack-kolla | 15:19 | |
| *** gfidente has quit IRC | 15:19 | |
| *** gfidente has joined #openstack-kolla | 15:19 | |
| mewald | inc0: yeah could be removed I guess. In production I would try to remove as much load from the controllers as possible as long it is not openstack control plane related | 15:19 |
| inc0 | yeah and monitoring can produce a lot of it | 15:20 |
| inc0 | in any case, as far as I'm concenrned monitoring is still open question for us | 15:20 |
| kfox1111 | inc0: just some info from the field. | 15:20 |
| kfox1111 | I've used nagios, checkmk, influxdb, graphite, and prometheus. | 15:20 |
| mewald | I am working in a prometheus patch which is currently in review which is on apache license and can be deploy across multiple nodes. | 15:21 |
| kfox1111 | plus have looked into several others. | 15:21 |
| mewald | that would replace influxdb | 15:21 |
| kfox1111 | I'm currently quite fond of prometheus. | 15:21 |
| egonzalez | mewald, i said because redis is not a monitoring service, DBs at this moment run under control nodes | 15:21 |
| kfox1111 | I've been working prometheus support into kolla-kubernetes. | 15:21 |
| inc0 | yeah prometheus seems like new kid in the block and seems to have lots of good ideas | 15:21 |
| kfox1111 | its the first monitoring system I've seen that handles dynamic infrastructure well. | 15:22 |
| inc0 | egonzalez: I think our kolla-ansible monitoring is based of influx not redis | 15:22 |
| inc0 | but if we want to move all in to prometheus, I'm interested | 15:23 |
| egonzalez | inc0, yep, was regarding redis patch who put redis in monitoring host group | 15:23 |
| inc0 | ah ok | 15:23 |
| inc0 | well monitoring group was separate group only because of this single node requirement by influx | 15:24 |
| *** omenvlad has quit IRC | 15:24 | |
| egonzalez | prometheus patch if want to take a look https://review.openstack.org/#/c/484882/ https://review.openstack.org/#/c/485150/ | 15:24 |
| *** ducttape_ has joined #openstack-kolla | 15:24 | |
| inc0 | so if we're moving to prometheus, we can get rid of it and put it in control:children | 15:24 |
| inc0 | this way we remove one top level group from inv which is good | 15:25 |
| kfox1111 | is there a reason not to use the upstsream containers for the exporters? | 15:25 |
| mewald | inc0: agreed, shall we completely remove influxdb right now or is there a deprecation process we want to run through? | 15:25 |
| inc0 | well we didn't use upstream containers for anything yet...but I'd be willing to try. | 15:26 |
| inc0 | deprecation, but we're close to release anyway | 15:26 |
| *** gfidente has quit IRC | 15:26 | |
| kfox1111 | inc0: kolla-kubernetes is pointing at the upstream ones and they seem to work well. | 15:26 |
| mewald | inc0: ok, i'll just leave it as it is right now | 15:26 |
| kfox1111 | been gating on them for a while now. | 15:26 |
| inc0 | yeah prometheus is container-centric so I expect them to be high quality | 15:26 |
| egonzalez | kfox1111, guess they come from busybox and no other arch than x86 supported | 15:27 |
| mewald | egonzalez: looks like ENV variables are not inherited from base images to child images. have you tried it before? | 15:27 |
| kfox1111 | hmm... yeah, I guess they probably don't support arm currently. though that could be added. | 15:27 |
| inc0 | I personally think reinventing a wheel just to have contos/ubuntu isn't good idea | 15:27 |
| kfox1111 | as for image, its literally one go statically linked binary per exporter. so distro really doesn't much matter. | 15:27 |
| egonzalez | mewald, never tried, leave as they are then | 15:27 |
| inc0 | I'd say we should work with prometheus community rather than doing it ourselves | 15:28 |
| inc0 | for arm/power | 15:28 |
| kfox1111 | yeah. I'm guessing they just have never tried arm/power. but would be just fine supporting them. | 15:28 |
| kfox1111 | which reminds me. I did get an arm box into my personal home k8s. | 15:29 |
| *** ducttape_ has quit IRC | 15:29 | |
| kfox1111 | so getting node-exporter on it would be aswesome. :) | 15:29 |
| inc0 | so, mewald egonzalez kfox1111 this is big decision, so let's send mail to ML | 15:29 |
| mewald | inc0, kfox1111 how do I properly include community containers? | 15:29 |
| kfox1111 | +1 to mailing list. | 15:29 |
| inc0 | I'm supporting migration from our current non-maintained monitoring to prometheus all the way, that will include deprectation of what we have today | 15:30 |
| inc0 | (which never exited tech preview stage so I'm not concerned) | 15:30 |
| inc0 | mewald: just put their name in start task | 15:30 |
| inc0 | docker will handle downloading | 15:31 |
| inc0 | also that, this is something new in Kolla but change for good | 15:31 |
| mewald | inc0: I am concerned with deployment situations where internet access is not available, pointing to docker hub would break it in those cases | 15:31 |
| inc0 | I'd love to see day when Kolla uses upstrema containers for all/most of non-openstack services | 15:31 |
| *** gfidente has joined #openstack-kolla | 15:32 | |
| *** gfidente has quit IRC | 15:32 | |
| *** gfidente has joined #openstack-kolla | 15:32 | |
| inc0 | mewald: well you need internet for build and you can download/push-to-local just as well | 15:32 |
| openstackgerrit | Mathias Ewald proposed openstack/kolla master: Add prometheus and related containers https://review.openstack.org/484882 | 15:32 |
| inc0 | but good point | 15:32 |
| inc0 | that being said, I think we can make change to build.py | 15:32 |
| inc0 | where it will not only build our images | 15:32 |
| inc0 | but also pull images we require | 15:32 |
| kfox1111 | mewald: for kolla-kubernetes, I've been contemplating something similar to inc0's trunk tarball. | 15:33 |
| kfox1111 | has all the containers needed for a launch. | 15:33 |
| *** shardy_mtg is now known as shardy | 15:33 | |
| kfox1111 | so user only has to wget one file to get going in a limited area. | 15:34 |
| mewald | I am not capable of doing this patch to build.py | 15:35 |
| kfox1111 | inc0: you might want to jump into the rdo ceph conversation... | 15:35 |
| inc0 | kfox1111: where? | 15:36 |
| kfox1111 | Re: [rdo-list] Heads up: Ceph Luminous (next LTS) is in release candidate | 15:36 |
| inc0 | mewald: don't worry about this | 15:36 |
| kfox1111 | we probably don't want centos-binary luminous and ubuntu jewel? | 15:36 |
| inc0 | I'm ok with us just downloading as long as we provice way to override the name | 15:37 |
| inc0 | damn... | 15:37 |
| inc0 | no, we don't | 15:37 |
| inc0 | but aren't we pinning rpm to jewel? | 15:37 |
| kfox1111 | yeah, but generally better to run distro supported stuff then not. | 15:38 |
| inc0 | I'd expect canonical to quickly follow...or hope at least | 15:39 |
| inc0 | also I think we use ceph repos there | 15:39 |
| *** zhangfei has quit IRC | 15:39 | |
| kfox1111 | they may, but timings tight on the redhat side, so I'd expect ubuntu to be sligtly slower and maybe miss the window. | 15:40 |
| *** goldyfruit has joined #openstack-kolla | 15:40 | |
| kfox1111 | its just a weird alignment of project releases. | 15:40 |
| inc0 | https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L141 | 15:40 |
| inc0 | I think ubuntu ceph comes from cloud archive | 15:41 |
| sean-k-mooney | inc0: kfox1111 you could just use the ceph ceph repos if the could archive does not have luminious | 15:42 |
| inc0 | yeah | 15:42 |
| inc0 | Queens goal anyway | 15:42 |
| inc0 | we can safely assume that by the end of Queens we'll have Luminious ready and give us time to test upgrade and stability | 15:43 |
| sean-k-mooney | inc0: the bluestore support and small writes on erasure coded pool are really nice features to have | 15:43 |
| kfox1111 | sean-k-mooney: yeah, ,but like I was saying, it may be better to support what rdo tests against. :/ | 15:43 |
| inc0 | kfox1111: agree, it's not pike anyway | 15:43 |
| kfox1111 | if they are testing against luminous and we slide in jewel instead, not sure what breakage may happen. | 15:44 |
| sean-k-mooney | kfox1111: yep, but the same logic would apply to canonical and the cloud archive | 15:44 |
| kfox1111 | I was thinking cononical equally tested their cloud archive. | 15:44 |
| sean-k-mooney | kfox1111: yes but with jewel | 15:44 |
| kfox1111 | right. | 15:44 |
| sean-k-mooney | so if you use the vendor repose you cant gurentee each vendor has the same version | 15:45 |
| kfox1111 | so we might want to ask rdo not to be too aggresive with getting luminious in at the last moment. | 15:45 |
| sean-k-mooney | if you use the upstream you can so that is the trade off | 15:45 |
| kfox1111 | yup. but the yare actively discussing whether to go to luminous for pike or not. | 15:45 |
| sean-k-mooney | kfox1111: i would preferer to ask canonical to be more agressive | 15:45 |
| kfox1111 | nows the time to help shape the decision. | 15:45 |
| inc0 | kfox1111: Pike ends in a month | 15:45 |
| kfox1111 | sean-k-mooney: ah. well, that works too. | 15:45 |
| inc0 | Luminius is not LTS yet | 15:46 |
| sean-k-mooney | kfox1111: there are a lot of nice featuers in luminous i would not want to give up | 15:46 |
| kfox1111 | and if so, we probably should support luminius all around kolla too. | 15:46 |
| sean-k-mooney | inc0: luminious is not released yet | 15:46 |
| inc0 | right | 15:46 |
| kfox1111 | yeah. | 15:46 |
| sean-k-mooney | it will be in october i think | 15:46 |
| inc0 | so there is little chance we'll make it in Pike | 15:46 |
| sean-k-mooney | so it should be fine for queens | 15:46 |
| inc0 | right, so it's for Queens definetly | 15:47 |
| kfox1111 | yeah. queens would probably be better timing. :/ | 15:47 |
| inc0 | and within Queens window we'll have plenty of time to figure it out | 15:47 |
| kfox1111 | but, just tryign to head off having one container set be different then all the rest. | 15:47 |
| sean-k-mooney | yes. im running luminous from opensuse tumbelweed but i would nto put it in to production as our default in kolla | 15:47 |
| kfox1111 | so, do we ask rdo to stick to jewel? | 15:47 |
| kfox1111 | (for pike) | 15:47 |
| sean-k-mooney | kfox1111: for pike i would say yes. | 15:48 |
| kfox1111 | ok. | 15:48 |
| kfox1111 | inc0: can you ask as the ptl? | 15:48 |
| mewald | egonzalez: Redis itself is quite stupid: it will replicate from master to slaves as long as the master lives. Sentinel (a second service not yet provided in the patch) is responsible for forming a quorum and promoting a new master when the original master died. We would just have to make sure redis servers will be restarted sequentially and it will be waited until the node is back in the game. Can someone point me to a service w | 15:48 |
| mewald | ith a similar upgrade pattern so I can have a look at the Ansible side of this? | 15:48 |
| sean-k-mooney | kfox1111: unless they are not going to release rdo until after ceph releases luminious i dodnt think rdo really has a coice | 15:48 |
| mewald | sorry more context: about the upgrade procedure for redis | 15:49 |
| kfox1111 | sean-k-mooney: tehy are discussing it on their mailing list now. | 15:49 |
| sean-k-mooney | kfox1111: holding the release or using a beta drop of ceph in a production deployment our of the box? | 15:49 |
| inc0 | kfox1111: I think we pin to jewel so we shouldn't be affected even if they move on | 15:50 |
| kfox1111 | https://www.redhat.com/archives/rdo-list/2017-July/msg00048.html | 15:50 |
| inc0 | also, jewel is LTS so I'd hope RDO will keep it tested for a time being | 15:51 |
| inc0 | I mean, that's what LTS means | 15:51 |
| kfox1111 | inc0: I think there may be subtile bugs that creep up then. :/ | 15:51 |
| inc0 | over time | 15:51 |
| inc0 | yes | 15:51 |
| inc0 | but Queens starts in little over a month | 15:51 |
| inc0 | we'll be pretty fresh | 15:51 |
| kfox1111 | no. wrong idea. | 15:52 |
| kfox1111 | pike is supported for a year. | 15:52 |
| kfox1111 | not, queens starts, lets forget about pike. | 15:52 |
| inc0 | sure | 15:52 |
| *** schwicht has quit IRC | 15:52 | |
| kfox1111 | what we do, affects us for a long time. | 15:52 |
| sean-k-mooney | kfox1111: yes but your not suggesting changeing the supported version of ceph after the stable branch is created | 15:52 |
| inc0 | what I'm saying is, rushing to put L in Pike will destabilize Pike probably | 15:52 |
| kfox1111 | sean-k-mooney: right. we should stick to one. but ideally, we should get rdo to stick with jewel rather then them go to luminious and we try and pin back to jewel. | 15:53 |
| inc0 | we just don't have time to figure out upgrades and all | 15:53 |
| sean-k-mooney | kfox1111: what do cinder et al offically support for pike? | 15:53 |
| inc0 | and it's not released yet | 15:53 |
| sean-k-mooney | kfox1111: yes i would agree with that. | 15:54 |
| kfox1111 | the ideal sitiuation is to ask rdo not to go to luminious for pike. | 15:54 |
| inc0 | dmsimard: around? | 15:54 |
| kfox1111 | lets try the ideal solution first. | 15:54 |
| *** eaguilar is now known as eaguilar|afk | 15:55 | |
| *** gfidente has quit IRC | 15:55 | |
| dmsimard | inc0: kina | 15:55 |
| dmsimard | inc0: kinda | 15:55 |
| inc0 | qq - Ceph L in Pike - please don't? | 15:56 |
| inc0 | it wasn't a question really, a plea rather | 15:56 |
| dmsimard | inc0: you can choose to stay on jewel, we're not imposing anything | 15:57 |
| kfox1111 | imposing no. but testing. | 15:57 |
| inc0 | you keep testing against jewel? | 15:57 |
| *** jrist has joined #openstack-kolla | 15:57 | |
| *** jrist has quit IRC | 15:57 | |
| *** jrist has joined #openstack-kolla | 15:57 | |
| kfox1111 | yeah. if you test against both, I think we'd be happy. | 15:57 |
| *** srnbckr has joined #openstack-kolla | 15:59 | |
| sean-k-mooney | dmsimard: was the intention to have luminious as techpreview in pike or the default ceph version. if jewel is still default/testing that would prevent so headaces for kolla as the concure is we wont have time to write and test the upgrade playbook between now and the pike release and we also dont want to have a different version of ceph on centos vs ubunut | 16:00 |
| dmsimard | well, FWIW we're testing the /installers/ | 16:01 |
| *** itlinux_ has quit IRC | 16:01 | |
| kfox1111 | dmsimard: you don't test that the installer deploys something that works? | 16:01 |
| inc0 | it won't be just kolla, it's for all of openstak rly | 16:01 |
| dmsimard | Ceph is not mirrored or managed by RDO (though I do happen to be a maintainer) | 16:01 |
| kfox1111 | and that the rpms can produce a working cloud? | 16:02 |
| dmsimard | What I'm trying to say is that for example, I think during Mitaka, some folks were running Jewel just fine while we were still testing Hammer | 16:02 |
| dmsimard | Or they were still running Hammer when we had moved on to Jewel | 16:02 |
| kfox1111 | where the rpms built against jewel or hammer? (and does that matter) | 16:03 |
| dmsimard | The fact that puppet-openstack or TripleO works (or not) with jewel or luminous should not have influence on what you're doing in Kolla 6 | 16:03 |
| *** itlinux has quit IRC | 16:03 | |
| kfox1111 | I woud think rpms built against hammer would work with a jewel lib rpm, but maybe not the other way around. | 16:03 |
| dmsimard | kfox1111: there's no such thing as building against ceph, RDO doesn't have any dependencies on cpeh | 16:03 |
| kfox1111 | ah. pure python imports? | 16:03 |
| dmsimard | you're making me doubt, actually, so I'll go check, but stuff like python-rbd is likely not on our end | 16:03 |
| kfox1111 | if its pure python imports, I remove my objection. | 16:04 |
| dmsimard | Yeah, python-rbd, python-rados and all that stuff is straight out of Ceph, we don't have any of that in RDO | 16:04 |
| dmsimard | See the package list here: https://cbs.centos.org/koji/buildinfo?buildID=16857 | 16:05 |
| kfox1111 | ok. cool. then no worries. :) | 16:05 |
| inc0 | I'd still put ceph version bump as Q goal | 16:06 |
| kfox1111 | inc0: +1. | 16:06 |
| kfox1111 | or, honestly, it might be nice to support multiple versions of ceph per openstack release. | 16:06 |
| kfox1111 | like, 2 LTS's. | 16:06 |
| inc0 | I just kicked this off | 16:07 |
| kfox1111 | though not sure how much extra work that may involve. | 16:07 |
| inc0 | https://etherpad.openstack.org/p/kolla-queens-ptg-planning | 16:07 |
| dmsimard | You probably have to handle testing upgrade from Jewel to Luminous | 16:07 |
| dmsimard | Which is the part that sucks | 16:07 |
| kfox1111 | depends on if thats in scope or not. | 16:07 |
| kfox1111 | I've been trying to keep ceph server support out of scope of kolla-kubernetes. | 16:08 |
| kfox1111 | ceph-deploy or other tools are really good. I really don't think we need to reinvent those wheels. | 16:08 |
| inc0 | We got H->J working | 16:08 |
| kfox1111 | rook is looking interesting too. | 16:09 |
| inc0 | I'm ok with keeping ceph outside of kolla-k8s | 16:11 |
| inc0 | we're kinda stuck with it in Kolla-ansible | 16:11 |
| kfox1111 | stuck until you deprecate it. | 16:11 |
| inc0 | deprecate+provide upgrade path | 16:11 |
| kfox1111 | if you give a migration path to another tool, that would be ok. | 16:11 |
| kfox1111 | right. | 16:11 |
| inc0 | something to discuss definetly | 16:12 |
| inc0 | maybe once tripleo does ansible+containers for ceph we could share roles | 16:13 |
| inc0 | EmilienM ^ | 16:13 |
| *** eaguilar|afk is now known as eaguilar | 16:13 | |
| *** eaguilar is now known as eaguilar|afk | 16:14 | |
| kfox1111 | is ceph-ansible up to the task by now? | 16:14 |
| EmilienM | inc0: we use ceph-ansible | 16:14 |
| inc0 | I guess that's the answer | 16:14 |
| kfox1111 | then, that. :) | 16:14 |
| inc0 | I'll put it in goals | 16:15 |
| kfox1111 | kolla-ansible shoudl probably provide a migration path to ceph-ansible and drop its own ceph server support. | 16:15 |
| kfox1111 | kk | 16:15 |
| *** egonzalez has quit IRC | 16:16 | |
| *** gfidente|afk is now known as gfidente | 16:16 | |
| inc0 | I really like how simple it is to deploy ceph with our stuff now, but maybe we could tinker out something that would simply allow us to import ceph-ansible roles | 16:17 |
| kfox1111 | as for kolla-kubernetes, I'm curious what ceph in kubernetes solutions will turn up. | 16:17 |
| kfox1111 | rook.io looks interesting, but seems a bit green yet. | 16:17 |
| kfox1111 | with the local volume support in k8s 1.7, I think ceph in k8s might be much more obtainable. | 16:17 |
| kfox1111 | ceph server that is. | 16:18 |
| jamesbenson | (I wish i could get ceph with kolla working here... manual deploy works though.) | 16:18 |
| srnbckr | hey guys, I still have problems with the kolla_toolbox container keep stuck in restarting mode -> docker logs kolla_toolbox shows "sudo: unknown uid 42401: who are you?". The ansible user exists in the container and kolla_ansible_sudoers looks ok. I've rebuild the container several times but nothing worked. | 16:19 |
| inc0 | jamesbenson: part of reason why we're having this discussion | 16:19 |
| jamesbenson | inc0: don't think I don't notice ;-) | 16:19 |
| srnbckr | Can you maybe point me to another thing to look at? | 16:19 |
| inc0 | srnbckr: what versions? | 16:19 |
| inc0 | of kolla? | 16:19 |
| srnbckr | kolla 4.0.1 | 16:20 |
| srnbckr | nodes are ubuntu 16.04 with docker 1.12.6 | 16:20 |
| inc0 | I've never seen this, do you have customized build.conf or anything? | 16:20 |
| inc0 | also try to rebuild with --nocache | 16:21 |
| jamesbenson | srnbckr: are you using AD server or anything for auth? | 16:21 |
| EmilienM | inc0: cool! thanks | 16:21 |
| inc0 | because creation of users runs as one of first RUNs in our dockerfiles | 16:21 |
| srnbckr | inc0, in my kolla-build.conf only the variables for the registry and base image are set | 16:21 |
| srnbckr | jamesbenson, yeah I have a ldap running for auth | 16:22 |
| jamesbenson | inc0: is it recommended to always do nocache or just in instances like this? | 16:22 |
| jamesbenson | srnbckr: does that id exist in ldap? | 16:23 |
| inc0 | jamesbenson: depends, if you develop image for example it cache will speed thnigs up a lot | 16:23 |
| inc0 | every now and then --nocache helps tho | 16:23 |
| jamesbenson | ok, good to know | 16:23 |
| srnbckr | jamesbenson, nope the uid doesn't exist in ldap | 16:24 |
| jamesbenson | srnbckr: I know when I did ldap/ad auth, I would run into those errors a bunch.... okay, try the --nocache then, maybe that will help | 16:24 |
| srnbckr | building it atm, thx | 16:25 |
| inc0 | srnbckr: I don't know much about ldap in container work, but alternatively it could be that creation of users filed silently for some reason and it uses cache ever since | 16:25 |
| *** ducttape_ has joined #openstack-kolla | 16:25 | |
| srnbckr | inc0, I'm not using ldap inside of the containers, our cluster is using ldap for auth | 16:26 |
| inc0 | yeah, I'm thinking if ldap could somehow bleed into container, not sure tho | 16:27 |
| *** jascott1 has joined #openstack-kolla | 16:27 | |
| srnbckr | mhm, I've deployed kolla on 3 centos nodes of the same cluster which are not using ldap auth. it worked there, so maybe you've got a point | 16:27 |
| inc0 | srnbckr: if same containers works on one and not another, then it's host | 16:28 |
| *** vhosakot has joined #openstack-kolla | 16:29 | |
| jamesbenson | srnbckr: once you get that worked out, you may want to try 4.0.2 or master 4.0.3. a bunch of bugs were fixed.... | 16:29 |
| *** itlinux has joined #openstack-kolla | 16:29 | |
| srnbckr | inc0, I've used a different registry for that deployment | 16:29 |
| *** ducttape_ has quit IRC | 16:30 | |
| *** iniazi has joined #openstack-kolla | 16:30 | |
| srnbckr | jamesbenson, ok, I will try out 4.0.3 then | 16:30 |
| inc0 | yup, we did a bunch of stuff last few months;) | 16:31 |
| jamesbenson | back to ceph convo? | 16:31 |
| inc0 | 4.0.3 isn't released yet, it's what git stable/ocata is today | 16:31 |
| *** itlinux has quit IRC | 16:31 | |
| inc0 | jamesbenson: so I for one like idea of moving to ceph-ansible or ceph-deploy | 16:32 |
| jamesbenson | sorry, correct, master is pike... | 16:32 |
| inc0 | and focusing on external ceph | 16:32 |
| inc0 | branch stable/ocata in git | 16:32 |
| inc0 | master will be pike when we release, yes | 16:32 |
| jamesbenson | I have a decent ceph script for ceph-deploy | 16:32 |
| *** lpetrut has joined #openstack-kolla | 16:33 | |
| jamesbenson | still need to debug, but good layout at least | 16:33 |
| jmccarthy | Hiya, I'm sure this has come up before, but I can't seem to see/remember what is up ? "error while evaluating conditional ('169.254.' not in kolla_internal_vip_address and kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3]) is none): 'dict object' has no attribute 'stdout'" for "Checking if kolla_internal_vip_address is in the same network as api_interface on all nodes" ansible task ? | 16:33 |
| jamesbenson | relatively simple | 16:33 |
| inc0 | well I think ceph-ansible would be better option tho, that's a discussion we need to have carefully | 16:33 |
| *** iniazi_ has quit IRC | 16:33 | |
| jamesbenson | agreed and I think it could be wrapped in from what I've seen | 16:33 |
| jamesbenson | just stating what I've worked with | 16:34 |
| jamesbenson | and have | 16:34 |
| inc0 | jmccarthy: is kolla_internal_vip_address in same network as ansible_interface on all nodes?:) | 16:34 |
| sean-k-mooney | inc0: have you got ceph ansible to work. i could not get it to work on ubuntu 16.04 | 16:34 |
| jmccarthy | I checked that yea the vip is oh ansible_interface or api_interface ? | 16:34 |
| inc0 | api_interface | 16:35 |
| jamesbenson | sean-k-mooney: yeah, I think I gave it 1 try, couldn't and that's why I went with deploy | 16:35 |
| inc0 | my bad | 16:35 |
| sean-k-mooney | jamesbenson: same | 16:35 |
| jmccarthy | Yea it it yea, which is network_interface by default, for me eth0 all around, it's in same network alright | 16:35 |
| *** manheim has quit IRC | 16:36 | |
| jamesbenson | sean-k-mooney, are you running external? | 16:36 |
| *** manheim has joined #openstack-kolla | 16:36 | |
| sean-k-mooney | jamesbenson: haved got around to deploying kolla yet at home its on my todo list. at work i use ceph deployed by kolla not external | 16:37 |
| *** dmsimard is now known as dmsimard|afk | 16:37 | |
| jamesbenson | sean-k-mooney: okay. kolla gives me issues with ceph for some reason, so I have to run it external here at work. (no toys at home besides a diy firewall, and rpi's) | 16:38 |
| sean-k-mooney | jamesbenson: there are a few rough spots with kollas ceph implenetaion. mail releated to external journals and caching | 16:39 |
| sean-k-mooney | if you dont use external journals or chacing it works pretty well | 16:39 |
| jamesbenson | I don't :-( | 16:39 |
| sean-k-mooney | jamesbenson: i use both hence why i know there are rough spot with those features | 16:40 |
| jamesbenson | oh wait... | 16:40 |
| jmccarthy | Ok I'll look again, maybe I am loosing my mind :) Well that's another question altogether lol | 16:40 |
| jamesbenson | ceph_cache_mode: "none"? I kept it on default, writeback | 16:41 |
| *** manheim has quit IRC | 16:41 | |
| jamesbenson | scratch that, ceph_enable_cache: "no" | 16:41 |
| *** jascott1 has quit IRC | 16:42 | |
| jamesbenson | really just enabled ceph, ceph_rgw, rgw_keystone, and the backends | 16:42 |
| jamesbenson | but tbh, it's the same with keystone-fernet, always fails. | 16:43 |
| jamesbenson | those two features, I can't use :-( | 16:43 |
| sean-k-mooney | they both take a little extra prep work on the hosts to get them to work | 16:43 |
| jamesbenson | please tell! | 16:44 |
| sean-k-mooney | well actully not sure about the fernet tokens | 16:44 |
| sean-k-mooney | well for ceph you need to have a gpt partion table with the lables applied for your osd disks | 16:44 |
| jamesbenson | already do that | 16:44 |
| sean-k-mooney | do your re do it every time you run deploy | 16:45 |
| jamesbenson | I love prep work. ^_^ give me more ^_^ | 16:45 |
| jamesbenson | yep and a reboot | 16:45 |
| sean-k-mooney | i have had little issues following the ceph guide to be hosnest | 16:45 |
| sean-k-mooney | jamesbenson: sepcifcially kolla's https://github.com/openstack/kolla-ansible/blob/master/doc/ceph-guide.rst | 16:46 |
| jamesbenson | I run this with my destroy script: https://github.com/JamesOBenson/Kolla-misc/blob/master/tasks/ceph.yml | 16:47 |
| *** dave-mccowan has joined #openstack-kolla | 16:47 | |
| *** dave-mcc_ has quit IRC | 16:50 | |
| jamesbenson | yeah, I've followed it... results were never consistent, about 90+% fail, and it is all scripted, so no reason why it shouldn't be repeatable.... I've only had a few successfull but other issues happened and had to destroy/deploy and then it fails again and again... | 16:51 |
| hawi | hi. what is "TASK [haproxy : Waiting for virtual IP to appear]" and getting timeout, when deploying? | 16:52 |
| jamesbenson | hawi: did your prechecks pass? | 16:53 |
| hawi | jamesbenson: yes | 16:53 |
| sean-k-mooney | hawi: haproxy is the loadblancer we use | 16:53 |
| sean-k-mooney | hawi: that task wait until it is pinable before moving on | 16:54 |
| mewald | where can I read how the release notes work? e.g: how is the random string generated? | 16:55 |
| *** athomas has quit IRC | 16:55 | |
| sean-k-mooney | mewald: you do someting along the lines of "tox -e venv reno add <name of release note>" to create it | 16:56 |
| mewald | ah oh, thx | 16:56 |
| hawi | sean-k-mooney: with previous installations i had all interfaces as vlan interfaces on same interface, now i tried to install everything on same interface | 16:56 |
| *** ramishra has quit IRC | 16:57 | |
| sean-k-mooney | hawi: yo can do that for everything except the neutron_external_interface | 16:57 |
| *** eaguilar|afk is now known as eaguilar | 16:58 | |
| hawi | yes, that's what i planned to do, neutron external interface has to be separate one because i will put vlans to it for connecting to external vlans | 16:58 |
| *** eaguilar is now known as eaguilar|afk | 16:58 | |
| sean-k-mooney | mewald: its tox -e venv reno new <name of release note> | 16:58 |
| hawi | and all others will sit on network_interface: "eno1" | 16:59 |
| sean-k-mooney | mewald: it will then output the path to the new file in the release note folder if i remember correctly | 16:59 |
| sean-k-mooney | mewald: https://docs.openstack.org/reno/latest/user/usage.html#creating-new-release-notes | 16:59 |
| jamesbenson | rook.io does look interesting ;-) | 17:00 |
| sean-k-mooney | hawi: that will in general not work unless you do not plan to support vlan network for tenants | 17:00 |
| sean-k-mooney | hawi: the neutron_external_interface should be a raw interface not a vlan subport | 17:01 |
| vhosakot | jmccarthy: the default network_interface works fine.. can you ping all the nodes from the kolla deploy node (the node on which you run kolla-ansible to deplot kolla)? | 17:01 |
| vhosakot | deploy* | 17:01 |
| sean-k-mooney | hawi: is the ip adress of you vip in the same range as the ip adress on eno1 | 17:02 |
| *** sambetts is now known as sambetts|afk | 17:02 | |
| hawi | sean-k-mooney: yes | 17:02 |
| sean-k-mooney | then proivded the vip is not inuse by something else it should work unless something on your network is blocking acess to it | 17:03 |
| jmccarthy | vhosakot: I can ping them all ok yep | 17:05 |
| -openstackstatus- NOTICE: Restarting Gerrit for our weekly memory leak cleanup. | 17:05 | |
| jmccarthy | sean-k-mooney: it is , I double checked | 17:06 |
| *** eaguilar|afk is now known as eaguilar | 17:07 | |
| jmccarthy | The vip task passes, it is not answering that seems ok | 17:07 |
| *** eaguilar is now known as eaguilar|afk | 17:07 | |
| sean-k-mooney | jmccarthy: have you disable the firewall on the servers? that could be blocking it | 17:08 |
| jmccarthy | I'll check | 17:08 |
| openstackgerrit | James Benson proposed openstack/kolla-ansible master: Added note https://review.openstack.org/485727 | 17:09 |
| jmccarthy | Yea doesn't look to be firewall related hmr missing something obviously | 17:09 |
| sean-k-mooney | jamesbenson: you have an extra space on line 117 https://review.openstack.org/#/c/485727/3/doc/external-ceph-guide.rst hehe | 17:10 |
| jamesbenson | gah! | 17:10 |
| sean-k-mooney | jamesbenson: looks fine other then that | 17:10 |
| jmccarthy | -vvv doesn't give much info on it, I'd like to see the output of ipaddr(ip_addr_output.stdout | 17:10 |
| openstackgerrit | James Benson proposed openstack/kolla-ansible master: Added note https://review.openstack.org/485727 | 17:11 |
| jamesbenson | I was following line 191 in that patch... | 17:11 |
| jamesbenson | thanks sean-k-mooney :-) | 17:11 |
| sean-k-mooney | jamesbenson: haha no worries, such promt adressing of critique too | 17:12 |
| jamesbenson | :-) I try not to build a queue if possible | 17:12 |
| jamesbenson | feel free to +2 it :-) | 17:13 |
| jmccarthy | Oh I just noticed this, but there is an eth0 ? (1, '\n{"msg": "[Errno 2] No such file or directory", "failed": true, "cmd": "ip -4 -o addr show dev eth0", "rc": 2, "invocation": {"module_args": | 17:14 |
| jmccarthy | {"warn": true, "executable": null, "_uses_shell": false, "_raw_params": "ip -4 -o addr show dev eth0", "removes": null, "creates": null, "chdir": null}}}\n', '') | 17:14 |
| sean-k-mooney | jamesbenson: if i had core rights on kolla-ansible i might but unfortuetly i do not | 17:14 |
| sean-k-mooney | jamesbenson: if it was to os-vif or networking-ovs-dpdk that would be another matter | 17:14 |
| jmccarthy | ip -4 -o addr show dev eth0 on the nodes in question gives what looks like the correct output | 17:14 |
| jamesbenson | you're a core there? | 17:15 |
| sean-k-mooney | jamesbenson: yes but not on kolla* projects | 17:15 |
| sean-k-mooney | i just like kolla | 17:15 |
| jamesbenson | yeah | 17:15 |
| hawi | sean-k-mooney: i checked and moved it to safe range and still seems that it will fail | 17:16 |
| hawi | sean-k-mooney: 4.0.0 | 17:16 |
| hawi | i just need to install it in order to have some understanding of upgrading to 4.0.1 or 4.0.2 | 17:16 |
| *** lpetrut has quit IRC | 17:16 | |
| hawi | on multiple nodes | 17:16 |
| sean-k-mooney | jmccarthy: if you interfaces are not the same on all nodes you need to declare them in the inventory file instead of setting network_interface in the global.yml | 17:19 |
| *** eaguilar|afk is now known as eaguilar | 17:20 | |
| *** eaguilar is now known as eaguilar|afk | 17:20 | |
| vhosakot | ^^ yes, I was gonna say the same thing jmccarthy. "ip -4 -o addr show dev eth0" just shows the v4 IP of eth0 | 17:21 |
| *** jtriley has joined #openstack-kolla | 17:21 | |
| *** jtriley has quit IRC | 17:21 | |
| *** jtriley has joined #openstack-kolla | 17:21 | |
| vhosakot | in the inventory file, all interfaces can be sparately specified if there is any conflict with network_interface. | 17:21 |
| jmccarthy | sean-k-mooney: Yea I have been bitten by that before alright, nope they are all eth0 as what is in the prop | 17:23 |
| sean-k-mooney | vhosakot: correct but global.yml overrides this so if you have to spcify it in the inventory you have to remove it from globals | 17:23 |
| *** dave-mccowan has quit IRC | 17:23 | |
| sean-k-mooney | jmccarthy: are you working with hawi our are ye both haveing different issues | 17:24 |
| vhosakot | ah right, global.yml overrides the inv file.. globals overrides everything.. right | 17:24 |
| jmccarthy | sean-k-mooney: Oh ? No not with hawi | 17:25 |
| sean-k-mooney | vhosakot: yep i really wish it work the other way around but its becasue we are using -e @/etc/kolla/globals.yml to pass it in | 17:25 |
| *** ducttape_ has joined #openstack-kolla | 17:26 | |
| *** dave-mccowan has joined #openstack-kolla | 17:26 | |
| sean-k-mooney | jmccarthy: ok cool hawi is have issue with haproxy i just want ed to make sure these converstation were not overlapping on the same deployment | 17:26 |
| jmccarthy | I will double check what is actually there in the lab again, I think the ansible etc is ok | 17:26 |
| jmccarthy | sean-k-mooney: gotcha :) No overlap, actually I'm going to leave it and see on Monday and find out it all magically works :) | 17:28 |
| sean-k-mooney | jmccarthy: thats allways a good coiche on a friday evening. enjoy your weekend | 17:29 |
| jmccarthy | Thanks again all , laters ! | 17:29 |
| vhosakot | sean-k-mooney: oh yeah, "-e" overrides the universe per ansible's variable precedence --> http://docs.ansible.com/ansible/latest/playbooks_variables.html#variable-precedence-where-should-i-put-a-variable | 17:29 |
| *** eaguilar|afk is now known as eaguilar | 17:30 | |
| *** ducttape_ has quit IRC | 17:30 | |
| vhosakot | "-e" can be used to crash the entire automation too :) | 17:30 |
| sean-k-mooney | vhosakot: yep -e is god in ansible world | 17:30 |
| *** jmccarthy has left #openstack-kolla | 17:33 | |
| sean-k-mooney | vhosakot: unfutunetly there is no other way that i know of to supprot keeping the global.yml outside of the git tree | 17:33 |
| vhosakot | sean-k-mooney: yeah, globals is our entry point for all customizations... | 17:34 |
| *** pbourke_ has quit IRC | 17:34 | |
| *** shardy has quit IRC | 17:39 | |
| *** pbourke_ has joined #openstack-kolla | 17:39 | |
| *** pbourke_ has quit IRC | 17:40 | |
| sean-k-mooney | some times i really hate centos.... | 17:50 |
| sean-k-mooney | the kernel devel package is installing the headers. its not creating the symlink in /lib/modules | 17:51 |
| *** jtriley has quit IRC | 18:01 | |
| iniazi | question... doing multinode kolla-ansible, and the requirements that show up on quickstart guide, mainly the 40gb disk space... is that for deploy node or all the nodes that will be running? and where does it need the 40gb, /var? | 18:04 |
| iniazi | deploy node... is also going to be where I'll keep the registry, i guess | 18:05 |
| inc0 | iniazi: so registry isn't that big, I think it's ~15gig total | 18:05 |
| inc0 | or less actually | 18:06 |
| inc0 | control node - databases | 18:06 |
| sean-k-mooney | inc0: closer to 30G for ubuntu source build | 18:06 |
| inc0 | if you're enabling central_logging then elasticsearch takes a lot of space | 18:06 |
| inc0 | really sean-k-mooney? It gunzips to less than 3G | 18:07 |
| sean-k-mooney | inc0: i taught so but maybe im wrong. from some reasone 27G was in my head but maybe that was for someting else | 18:09 |
| inc0 | tarball of all the images (no gzip) was 12G | 18:09 |
| inc0 | so it'd be strange | 18:10 |
| sean-k-mooney | inc0: checking my host env its actully 14G | 18:10 |
| sean-k-mooney | inc0: oh now i rememeber you need ~27G because if you put your registry on the same node as your contoller it need to have 2 copies of the images | 18:12 |
| sean-k-mooney | inc0: the registry copy and the local copy for the contoler containers | 18:12 |
| inc0 | yeah that would be right. Although controllers don't use all of images | 18:12 |
| sean-k-mooney | inc0: true | 18:13 |
| vhosakot | I usually don't have a fat registry, delete unwanted/intermediate images. my registry size is like ~11G | 18:15 |
| *** iniazi_ has joined #openstack-kolla | 18:15 | |
| sean-k-mooney | vhosakot: isnt that the default. kolla only pushes the final images and the layers it depends on | 18:16 |
| sean-k-mooney | vhosakot: or do you flatten them before pushing | 18:16 |
| vhosakot | sean-k-mooney: yes that's the default.. no, I don't flatten... just push the usual way and docker does its layering magic to store images on disk to use optimum disk space. | 18:17 |
| *** iniazi has quit IRC | 18:18 | |
| vhosakot | my registry usually has the master images now... if we carry older releases/tags/versions, we'll end up with a fat registry. | 18:18 |
| *** jascott1 has joined #openstack-kolla | 18:19 | |
| vhosakot | iniazi_: the storage space really depends on your usecase.. storage nodes need TB, OpenStack's db will be on control nodes, glance images take up space, so it really depends on the usecase. | 18:19 |
| iniazi_ | well my ceph cluster has about 9tb, my deploy node has about 15t (which is running a 1node es for now) but my compute/control nodes only have about 100g free space. which I can assign, but wasn't sure what the disk space was referring to | 18:21 |
| iniazi_ | and where it needs it. | 18:22 |
| sean-k-mooney | on compute nodes with ceph for vms kolla does not need much disk space. mainly for log storage at that point | 18:23 |
| vhosakot | iniazi_: well, 15T on the deploy node is a lot for sure :) | 18:23 |
| jascott1 | inc0 technosophos said most of the pieces were there except exposing it to rudder so it can be hooked into. sounds promising so I will see what I can do | 18:24 |
| inc0 | what's rudder? | 18:24 |
| vhosakot | right, nova VMs using ceph volume need very very min disk space on the computes | 18:24 |
| vhosakot | inc0: I think it is an airplane part? | 18:24 |
| iniazi_ | boat part... its because we are on the same boat | 18:25 |
| iniazi_ | :) | 18:25 |
| inc0 | well, I know what's rudder in ship (I don't think plane tail is called that?) | 18:25 |
| iniazi_ | plane tail is also rudder i think | 18:25 |
| inc0 | yeah it is | 18:25 |
| inc0 | good to know | 18:25 |
| inc0 | anyway, hwat's rudder in kubernetes-helm nomenclature?;) | 18:26 |
| iniazi_ | its the thingamajig that moves on the vert stabilizer | 18:26 |
| inc0 | yeah you're right | 18:26 |
| vhosakot | I was a research engineer for an aircraft company for 9 months. rudder is a part of a plane :) https://www.grc.nasa.gov/www/k-12/airplane/Images/rud.gif | 18:26 |
| vhosakot | lol | 18:26 |
| vhosakot | orange plane | 18:26 |
| inc0 | heh, I know these names in Polish | 18:27 |
| inc0 | I fly paragliders and paragliders don't have rudders | 18:27 |
| jascott1 | inc0 its an grpc tunnel to tiller pod afaik.. to be honest ive never dealt with it but should be expert soon :P | 18:27 |
| inc0 | kk | 18:27 |
| inc0 | also how would writing support for jinja2 look like? ^ | 18:28 |
| inc0 | kfox1111 EmilienM ^ | 18:28 |
| inc0 | EmilienM: we're exploring support for jinja2 in helm | 18:28 |
| inc0 | if we could get it in somehow (possible) we could rewrite our templates to jinja2 while keeping compatibility with helm | 18:28 |
| inc0 | potentially | 18:28 |
| vhosakot | jinjansibhelm is the next big thing ;) | 18:29 |
| *** jrist has quit IRC | 18:29 | |
| *** iniazi_ is now known as iniazi | 18:30 | |
| sean-k-mooney | helm, tiller, rudder, the k8s guys sure do like there boat based project names | 18:30 |
| vhosakot | yeah, kubernetes also means the captain of a ship or something | 18:31 |
| vhosakot | in greek | 18:31 |
| *** ducttape_ has joined #openstack-kolla | 18:32 | |
| EmilienM | inc0: you fly? nice | 18:32 |
| sean-k-mooney | well its no worse then us calling the block storage project cinder i guess | 18:32 |
| EmilienM | inc0: I fly cessna 172 :P | 18:33 |
| kfox1111 | sean-k-mooney: naming's hard. conventions help. :) | 18:33 |
| inc0 | yeah I've seen tweets;) yeah, I'm paraglider pilot | 18:33 |
| vhosakot | yeah, cinder block.. lol | 18:33 |
| vhosakot | wow cool inc0 | 18:33 |
| inc0 | hopefully get some quality airtime tomorrow \o/ | 18:33 |
| inc0 | I need to test my new harness | 18:33 |
| EmilienM | inc0: we're not far from each others | 18:34 |
| vhosakot | my tomorrow is lawn, lawn, and more lawn :) | 18:34 |
| EmilienM | inc0: I think I can fly to your place in ~3h max | 18:34 |
| inc0 | it would be much more for me...if at all possible | 18:34 |
| inc0 | that being said, I really want to fly around Vancouver BC | 18:35 |
| inc0 | mountains out there are just gorgeous | 18:35 |
| EmilienM | inc0: I fly on the island for now | 18:35 |
| sean-k-mooney | vhosakot: i think mine is makeing dpdk work on centos.... | 18:35 |
| vhosakot | sean-k-mooney: cool :) | 18:36 |
| inc0 | I need to try powered paragliding someday | 18:36 |
| inc0 | right now I'm bound to mountains and thermals;) | 18:36 |
| sean-k-mooney | vhosakot: centos is a pain in the ass sometimes... i had this working untill they stoped creating the symlinkx for the kernel headres correctly. | 18:36 |
| *** ducttape_ has quit IRC | 18:37 | |
| EmilienM | inc0: come here, we'll go flying and hack on stuffs ;-) | 18:37 |
| vhosakot | sean-k-mooney: I see, so are they like broken packages? | 18:37 |
| vhosakot | EmilienM: good to land first, and then hack stuff ;) | 18:38 |
| kfox1111 | is there an online gotl parser? | 18:38 |
| sean-k-mooney | vhosakot: installing kernel-devel-$(uname-r) is ment to create a symlink in /user/modules/$(uname -r)/build but the latest package does not | 18:38 |
| vhosakot | I'd scared to hack the plane I fly in.. lol :) | 18:38 |
| vhosakot | sean-k-mooney: ah I see, sucks | 18:39 |
| -openstackstatus- NOTICE: docs.o.o is currently broken, we're investigating | 18:39 | |
| inc0 | just...don't fly too close:P all I have is a piece of cloth;) | 18:40 |
| EmilienM | vhosakot: hehe | 18:40 |
| sean-k-mooney | vhosakot: sorry that should be /lib/modules/$(uname -r)/build but ya i need to change my code to work around it i think which is a pain | 18:40 |
| inc0 | EmilienM: you live on Vancouver island? | 18:41 |
| EmilienM | inc0: yes, for now :) | 18:41 |
| vhosakot | sean-k-mooney: ah I see, good luck :) | 18:41 |
| inc0 | cool, I hope to go to BC soon to spend some more time there | 18:42 |
| inc0 | I really loved that place | 18:42 |
| *** MasterOfBugs has joined #openstack-kolla | 18:45 | |
| sean-k-mooney | inc0: vanvouver is pretty awsome | 18:47 |
| sean-k-mooney | the summit after sydny is there again right? | 18:47 |
| inc0 | yeah | 18:47 |
| inc0 | second summit I'll be able to drive to;) | 18:47 |
| inc0 | a bit longer drive, but still a nice one | 18:48 |
| EmilienM | inc0: there is a ferry from Port Angeles, direct to Victoria | 18:48 |
| EmilienM | inc0: and I live 2 min walking from the ferry :) | 18:48 |
| inc0 | nice | 18:48 |
| inc0 | well, if I won't stay in US, Canada is my next destination | 18:49 |
| inc0 | most likely | 18:49 |
| vhosakot | yeah, vancouver after sydney... wonder where it is after vanvoucer? | 18:49 |
| inc0 | I really like Pacific NW | 18:49 |
| * vhosakot thinks inc0 still lives in San Antonio for some reason.. lol :) | 18:50 | |
| * inc0 does not | 18:52 | |
| inc0 | Portland since December | 18:52 |
| inc0 | well Hillsboro, but that's same thing | 18:52 |
| vhosakot | inc0: are you close to that crater thing in OR? | 18:55 |
| inc0 | there are many crater things in OR;) but I assume you're asking about crater lake | 18:56 |
| inc0 | ~5hrs | 18:56 |
| vhosakot | right crater lake | 18:56 |
| vhosakot | ah cool | 18:56 |
| inc0 | I plan to go there campnin mid August | 18:57 |
| inc0 | btw EmilienM maybe you'll fly over here for solar eclipse? | 18:57 |
| inc0 | or anyone really, that'll be quite an event | 18:57 |
| jamesbenson | vhosakot I'm in SA :-). or :-( depending on how you look at it | 18:58 |
| vhosakot | SA? Aouth Australia? | 18:59 |
| vhosakot | South* | 18:59 |
| vhosakot | jamesbenson: ^^ | 18:59 |
| inc0 | temperature wise it's more like north australia | 18:59 |
| vhosakot | inc0: north australia? I thought the entire continent of australia is in the south lol ;) | 19:02 |
| jamesbenson | sorry catching up in the thread, san antonio :-( | 19:02 |
| jamesbenson | used to live in potsdam.... ny, not germany.. | 19:03 |
| jamesbenson | potsdam was nice though, hit up canada a bunch :-) | 19:03 |
| jamesbenson | qq: why can I ping my router in openstack, but not my vm's? security rules are good | 19:04 |
| vhosakot | jamesbenson: wife and I love San Antonio actually... love the river walk area and the Alamo... we also went to Waco few times | 19:04 |
| jamesbenson | ^^ nice :-) | 19:04 |
| vhosakot | jamesbenson: neutron blocks incoming icmp by default | 19:04 |
| jamesbenson | can't ssh into vm's either though | 19:05 |
| vhosakot | jamesbenson: yes, SSH (TCP port22) is also blocked | 19:05 |
| vhosakot | jamesbenson: neutron's ati-spoofing sec rules | 19:05 |
| jamesbenson | security rules I enabled ssh and ping | 19:05 |
| rwellum | Talking about temperature - I am playing soccer this weekend and the projected forecast is 108F. I still live in Celsius - but I think I am going to die basically... | 19:05 |
| vhosakot | wow, 108... I remember wife and I went to Grand Canyon hiking in 111 a few years ago... we took lots of water and umbrella | 19:06 |
| *** tonanhngo has joined #openstack-kolla | 19:07 | |
| inc0 | jamesbenson: can you ping vm from inside dhcp netns | 19:07 |
| inc0 | ? | 19:07 |
| inc0 | go to net node and ip netns exec << your qdhcp... netns id >> ping <<instance id >> | 19:08 |
| inc0 | also check instance logs if it got dhcp | 19:08 |
| vhosakot | all incoming traffic to TCP ports will be blocked unless explicitly opened for each source+dest IP combo, or port number, it is neutron's rule #0 | 19:08 |
| inc0 | rwellum: where do you live? | 19:09 |
| inc0 | I thought you're from US | 19:09 |
| vhosakot | neutron allows packets that originate form the VM (like DHCPDISCOVER, DNS request, etc) | 19:09 |
| sbezverk | rwellum I would really like to see you playing soccer with umbrella ;) | 19:09 |
| rwellum | NC inc0 - gets quite hot. | 19:09 |
| jamesbenson | qdhcp = neutron_dhcp_agent? | 19:09 |
| vhosakot | lololol... soccer in umbrells | 19:09 |
| inc0 | jamesbenson: run "ip netns" on network node | 19:10 |
| inc0 | it's network namespace | 19:10 |
| jamesbenson | nothing | 19:10 |
| vhosakot | jamesbenson: neutron_dhcp_agent is a nuetron process that creates a network namspace with the name qdhcp-<network ID> on the control node that runs neutron_dhcp_agent. | 19:10 |
| inc0 | on any of network nodes? | 19:10 |
| jamesbenson | okay, one returned | 19:11 |
| *** bmace has quit IRC | 19:11 | |
| *** bmace has joined #openstack-kolla | 19:11 | |
| jamesbenson | no, can't ping either floating or internal ip on router | 19:12 |
| sean-k-mooney | inc0 has http://logs.openstack.org/32/466132/6/gate/gate-kolla-dsvm-build-ubuntu-source-ubuntu-xenial/ad6ccf5/console.html#_2017-07-21_18_25_43_968049 been fixed yet | 19:12 |
| jamesbenson | ip netns exec qrouter-09aa9dd8-da65-4fb8-beb6-1215d111a8f9 ping 192.168.100.9 | 19:12 |
| vhosakot | jamesbenson: by default, neutron runs the dhcp namespace on only one of the networks nodes, unless we specify dhcp_agents_per_network > 2 in neutron.conf --> https://github.com/openstack/kolla-ansible/blob/master/ansible/inventory/multinode#L14-L16 | 19:12 |
| rwellum | sbezverk: quick sanity check on my source image work?: | 19:12 |
| rwellum | https://www.irccloud.com/pastebin/rGcNM7zR/ | 19:13 |
| inc0 | jamesbenson: can't you see qdhcp... netns? | 19:13 |
| inc0 | I mean...that's hwere dhcp agent lives | 19:13 |
| rwellum | sbezverk: why do I see two images? | 19:13 |
| sean-k-mooney | inc0: never mind its fixed here https://github.com/openstack/kolla/blob/master/tests/test_build.py#L150 | 19:13 |
| sbezverk | rwellum yep, it looks like it is using your local image | 19:14 |
| sbezverk | rwellum : second image is for init container which waits for some resources before it allows main container to start | 19:14 |
| jamesbenson | sorry inc0, i'm getting lost. The only thing I saw was qrouter | 19:14 |
| rwellum | ok sbezverk - sweet | 19:15 |
| inc0 | jamesbenson: neutron agent-list please | 19:15 |
| inc0 | check if your dhcp agent is healthy | 19:15 |
| *** ducttap__ has joined #openstack-kolla | 19:15 | |
| jamesbenson | http://paste.openstack.org/show/616204/ | 19:16 |
| jamesbenson | dhcp healthy, everything is | 19:16 |
| jamesbenson | checked | 19:16 |
| jamesbenson | and a fresh deploy | 19:16 |
| jamesbenson | also, security rules: http://paste.openstack.org/show/616203/ | 19:16 |
| jamesbenson | I enabled icmp, ssh.. | 19:17 |
| vhosakot | jamesbenson: from the output you sent, I see three DHCP agents each running on the nodes r6-410-1/2/3. do you need multiple DHCP agents. one works jsut fine in most cases unless you need HA DHCP. | 19:17 |
| jamesbenson | I suppose we can do only 1, I just copied the control group over. | 19:18 |
| jamesbenson | that shouldn't be the cause of the issue though correct? | 19:19 |
| vhosakot | ah jamesbenson.. are your control nodes the network nodes too? | 19:19 |
| jamesbenson | yeah, this is dev, so I have 3 controls with everyone on it besides compute, and one compute | 19:19 |
| vhosakot | well, it depends on what's your dhcp_agents_per_network in neutron.conf becasue each network now sees more than 1 DHCP agent.. so, who will respond to the DHCPDISCOVER from the nova VM? | 19:19 |
| jamesbenson | http://paste.openstack.org/show/616205/ | 19:19 |
| jamesbenson | globals ^^ | 19:20 |
| jamesbenson | err multinode sorry | 19:20 |
| jamesbenson | never modified neutron.conf | 19:20 |
| inc0 | but you can't see qdhcp namespace anywhere? | 19:21 |
| vhosakot | jamesbenson: can you check https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/defaults/main.yml#L234 in you /etc/kolla/neutron* dir? | 19:21 |
| rwellum | sbezverk: so when I go to this Horizon, it's clearly default Ocata. So I may be running what I built, but I don't think I built my local source... | 19:23 |
| jamesbenson | inc0, no, only what I told you with that command | 19:23 |
| inc0 | that's....strange | 19:24 |
| sbezverk | rwellum if you know about a specific file which should be present in your image, just run docker run -tu root -ti {your local image ID} bash | 19:24 |
| sbezverk | and then check if that file is there | 19:24 |
| jamesbenson | vhosakot: I don't have a /etc/kolla/neutron dir... | 19:25 |
| sbezverk | rwellum I mean I do not know how you build your images, all I can say that it is using right image from version perspective.. | 19:25 |
| vhosakot | on your deploy node jamesbenson? not on the network node | 19:25 |
| jamesbenson | correct | 19:25 |
| vhosakot | ha | 19:25 |
| sean-k-mooney | inc0: can you weight in on https://review.openstack.org/#/c/408872/12/contrib/ovs-dpdk/library/kolla_docker.py when you get a chance | 19:25 |
| inc0 | Sean Spicer resigned :( | 19:26 |
| rwellum | sbezverk: what's my local image id? | 19:26 |
| jamesbenson | vhosakot: care to share? | 19:26 |
| sbezverk | rwellum docker images | grep 4.0.2 | 19:26 |
| sbezverk | 3rd parameter is the image id | 19:27 |
| inc0 | sean-k-mooney: what's wrong with our current kolla-docker? | 19:27 |
| vhosakot | jamesbenson: I don't have my setup now, let me check the code.. in any case, multiple DHCP agents is not I've seen very common | 19:27 |
| jamesbenson | if you give me 30 I can destroy and deploy... | 19:28 |
| jamesbenson | yes/no? | 19:29 |
| vhosakot | sure.. jamesbenson do you also see the DHCP container on all the nodes? r6-410-1/2/3 ? | 19:29 |
| rwellum | sbezverk: that's great thanks! Unfortunately it's not my local source code... | 19:29 |
| sean-k-mooney | inc0: i cannont symlink to it form the contrib playbook or use it so i had to copy it | 19:29 |
| inc0 | well, let's get to the bottom of this | 19:29 |
| inc0 | uhh....ok | 19:29 |
| jamesbenson | neutron_dhcp_agent | 19:29 |
| inc0 | symlink didn't wokr? | 19:29 |
| inc0 | work | 19:29 |
| sean-k-mooney | inc0: that was what i tried first but no ansible did not follow the symlink correctly | 19:30 |
| jamesbenson | neutron_dhcp_agent is on all 3 | 19:30 |
| inc0 | jamesbenson: so what should happen is dhcp agent should create netns qdhcp-<<id of subnet>> | 19:30 |
| inc0 | and run dnsmasq within this netns | 19:31 |
| jamesbenson | okay, let me destroy and bring it up with only 1 network node | 19:31 |
| inc0 | if it's not there, vms won't get dhcp | 19:31 |
| sean-k-mooney | inc0: Eduardo Gonzalez was suggesting merging the contrib role into the main ones to avoid the duplication but i know you want to keep the seperate. so if you let me know which is your preference ill implement that | 19:32 |
| jamesbenson | okay, destroying now | 19:32 |
| inc0 | sean-k-mooney: frankly if that's the case then I'd move ovs-dpdk to general ansible dir | 19:32 |
| inc0 | not if it's an technicla issue | 19:32 |
| sean-k-mooney | inc0: i can do that if your ok with that. | 19:32 |
| inc0 | I'm ok with that | 19:33 |
| inc0 | I'd rather have this than mess with copied files | 19:33 |
| sean-k-mooney | inc0: it still will not be used as part of kolla deploy so it wont change the default behavoir anyway | 19:33 |
| vhosakot | jamesbenson: our gate runs just one DHCP agent with default neutron.conf --> http://logs.openstack.org/23/485723/8/check/gate-kolla-ansible-dsvm-deploy-centos-binary-centos-7-nv/ed2cb19/console.html#_2017-07-21_16_05_34_321928 | 19:33 |
| jamesbenson | ok | 19:33 |
| inc0 | yeah, it's easy to turn off role | 19:34 |
| inc0 | also I was thinking of contrib from perspective of container images rather than roles | 19:34 |
| sean-k-mooney | ok ill adress the other issues and do that last | 19:34 |
| inc0 | contrib is good place for overrides | 19:34 |
| vhosakot | wonder if it's because the control nodes are the network nodes r6-410-1/2/3. | 19:35 |
| rwellum | inc0: so I am trying to build my local source code with kolla-build, by editing the kolla-build.conf file - and it generates an image just fine - but doesn't seem to be the code I point it to. Any good steps to debug this? | 19:35 |
| inc0 | jamesbenson: also you did create subnet for net right?;) | 19:35 |
| sean-k-mooney | ya i agree. i like haveing it contrib when i taugh i could symlink in the shared code but that did not pan out | 19:35 |
| jamesbenson | maybe we should update the default inventory to only one host there then? https://github.com/openstack/kolla-ansible/blob/master/ansible/inventory/multinode#L16 | 19:35 |
| vhosakot | jamesbenson: might be a good try to use just one node (for example r6-410-1) as your network node and redeploy. | 19:35 |
| jamesbenson | inc0, ha! yes good call though | 19:35 |
| *** dmsimard|afk is now known as dmsimard | 19:35 | |
| inc0 | rwellum: local source code for openstack service? | 19:35 |
| inc0 | sorry jamesbenson, happened to me once and I felt stupid;) | 19:36 |
| inc0 | ...after few hrs | 19:36 |
| vhosakot | inc0: there can be multiple network nodes, and neutron uses the value of dhcp_agents_per_network in neutron.conf to decide on how many nodes to run the DHCP agent. | 19:36 |
| jamesbenson | fair enough, don't similar things... sometimes you get happy with only a few additional lines of code in a day.... | 19:36 |
| rwellum | yes inc0 - trying to replace Horizon service with local source code. | 19:36 |
| jamesbenson | and that's why I script the hell out of things... to prevent me from being stupid. | 19:36 |
| jamesbenson | or at least stupid only once... | 19:37 |
| inc0 | rwellum: you have git repo with it? or how do you put it there? | 19:37 |
| rwellum | inc0: yeah - I git clone my companies source code - with is Mitaka based btw. | 19:38 |
| inc0 | show me your kolla conf plz | 19:38 |
| inc0 | also, you shouldn't really clone it | 19:38 |
| inc0 | you should point to repo and build will clone it | 19:38 |
| inc0 | oh yoy're trying to do type=local? | 19:39 |
| rwellum | inc0 - yup: https://www.irccloud.com/pastebin/xMwFqZbT/ | 19:39 |
| inc0 | I'm not sure how local works tbh;) I'd need to check | 19:39 |
| inc0 | oh, also try to use absolute paths if anything | 19:40 |
| inc0 | not ~ | 19:40 |
| rwellum | ok | 19:40 |
| inc0 | or better would be to use type=git | 19:40 |
| jamesbenson | side note while deploying: inc0 can you check this out? https://review.openstack.org/#/c/485704/ | 19:41 |
| rwellum | inc0: changed to absolute path but now getting an odd error when building: | 19:49 |
| rwellum | https://www.irccloud.com/pastebin/XkLy6om8/ | 19:49 |
| inc0 | rwellum: try running pip install -U . | 19:50 |
| vhosakot | rwellum: so, isn't 'location' in kolla-build.conf not picking up and using your company's custom tar.gz to build the image from source? | 19:52 |
| rwellum | vhosakot: yeah maybe | 19:53 |
| vhosakot | rwellum: yeah, I'd try the abs path for 'location' in kolla-build.conf instead of ~/horizon/. | 19:53 |
| rwellum | vhosakot: trying that but seem to have broken kolla. Recloning and building now | 19:53 |
| vhosakot | cool | 19:54 |
| rwellum | inc0: I tried that, same issue. Deleted kolla, repulled, recompiled - same error when I kolla build... | 19:55 |
| kfox1111 | sbezverk: btw, https://github.com/kubernetes/helm/issues/2706 | 19:58 |
| kfox1111 | the .Capabilities.KubeVersion thing is very interesting. | 19:58 |
| vhosakot | rwellum: same error "self._dc = docker.Client(version='auto', **docker_kwargs)" again after you repull and rerun? | 19:59 |
| rwellum | yes vhosakot | 19:59 |
| jamesbenson | okay deployed and setting up infra | 20:00 |
| sbezverk | kfox1111 really interesting. We could try to print .Capabilities and see what else is there ;) | 20:00 |
| *** dave-mccowan has quit IRC | 20:02 | |
| jamesbenson | same result | 20:02 |
| vhosakot | rwellum: are you using master? I see the latest docker-py has docker.APIClient instead of docker.Client --> https://review.openstack.org/#/c/475939/3/kolla/image/build.py | 20:03 |
| rwellum | I think so vhosakot: git clone http://github.com/openstack/kolla | 20:04 |
| vhosakot | rwellum: on the deploy node, can you open a Python interpreter (type "python") and do "import docker" and then "docker.Client" and also "docker.APIClient" | 20:04 |
| jamesbenson | inc0 ip netns still returns: qrouter-8a66770e-f47b-470a-a447-ad1805e17961 | 20:04 |
| vhosakot | rwellum: master has docker.APIClient instead of docker.Client - https://github.com/openstack/kolla/blob/master/kolla/image/build.py#L135 | 20:04 |
| jamesbenson | all network agents are up and alive on r6-410-1 | 20:05 |
| rwellum | vhosakot: ah and the commands above confirmed this. So am I just cloning the wrong version of kolla? | 20:06 |
| vhosakot | rwellum: I think you need to uninstalla and install the master... did you use pip to install kolla? | 20:06 |
| jamesbenson | inc0: where do you find this info from exactly: qdhcp-<<id of subnet>>. I want to make sure I'm pulling it correctly | 20:09 |
| rwellum | Yes I used pip | 20:09 |
| rwellum | vhosakot: https://www.irccloud.com/pastebin/ZTT7N20w/ | 20:10 |
| kfox1111 | sbezverk: yeah. :) | 20:11 |
| jamesbenson | vhosakot: can you help on the network issue still? | 20:12 |
| vhosakot | jamesbenson: sure, I'm still trying to see the code where the playbook deploys multiple DHCP agents :) | 20:12 |
| jamesbenson | well I've got just one node now :-) | 20:13 |
| jamesbenson | should the ip netns command return the qdhcp-... ? | 20:13 |
| inc0 | it should | 20:14 |
| inc0 | try to spawn a vm | 20:14 |
| jamesbenson | hmm it doesn't.' | 20:14 |
| jamesbenson | just the qrouter. | 20:14 |
| *** lpetrut has joined #openstack-kolla | 20:15 | |
| vhosakot | rwellum: hmm, wonder you should do "pip uninstall kolla" before "sudo -H pip install -U kolla" | 20:15 |
| jamesbenson | this is how I set up my networking... hasn't been an issue yet, but maybe I was lucky? | 20:16 |
| jamesbenson | https://github.com/JamesOBenson/Kolla-misc/blob/master/scripts/setup_networking.sh | 20:16 |
| hawi | sean-k-mooney: still nothing. https://pastebin.com/aDZqBhbN - i can see, that db listens on kolla_internal_vip_address | 20:16 |
| rwellum | vhosakot: tried that and also deleted the whole dir - uninstalled and reinstalled | 20:16 |
| kfox1111 | sbezverk: probably can toYaml the whole thing and --dry-run --debug see what it looks like. :) | 20:16 |
| jamesbenson | I can ping the router, so router, so external networking works and router works, correct? | 20:17 |
| rwellum | vhosakot: but when I install it seems to be pulling a cached version of kolla | 20:18 |
| sean-k-mooney | hawi: that does not look right. keepalived should only have floated the vip to one of the hosts not all of them | 20:18 |
| rwellum | "Using cached kolla-4.0.2-py2-none-any.whl" | 20:18 |
| vhosakot | I think pip has a --no-cache or soomething, checking | 20:18 |
| sean-k-mooney | hawi: your sudo netstat command should have shown it bound to the local ip on eno1 | 20:19 |
| hawi | sean-k-mooney: yes, something is seriously wrong. also i cant connect to this port (i assume it's brought up by haproxy) | 20:19 |
| jamesbenson | vhosakot: whats vnic type? I don't think I recall this in the past. But previously I ran a few version behind... | 20:19 |
| vhosakot | inc0: just saw the quickstart guide https://github.com/openstack/kolla-ansible/blob/master/doc/quickstart.rst does not have the step to install kolla, it just has "pip install kolla-ansible" | 20:20 |
| sean-k-mooney | do you have anohter kolla cluster runing in the datatcenter | 20:20 |
| inc0 | vhosakot: because it's kolla-ansible quickstart;) | 20:20 |
| sean-k-mooney | hawi: or rather on the same physical network | 20:20 |
| sean-k-mooney | hawi: if you do you need to make sure https://github.com/openstack/kolla-ansible/blob/master/etc/kolla/globals.yml#L97 is set to a unique number of this deployment | 20:21 |
| vhosakot | inc0: right, kolla docs also do not have the step to install the repo (sudo -H pip install -U kolla)... there is no mention aboue how kolla-build is installed in the first place in https://github.com/openstack/kolla/blob/master/doc/image-building.rst :) | 20:21 |
| hawi | sean-k-mooney: https://pastebin.com/5u7xKg0u is from one controller | 20:21 |
| inc0 | vhosakot: sounds like great patch;) | 20:22 |
| vhosakot | jamesbenson: vnic? where do you see it? as a neutron option | 20:22 |
| vhosakot | inc0: yea, doc patch :) lol | 20:22 |
| hawi | sean-k-mooney: it's commented out | 20:22 |
| sean-k-mooney | vhosakot: as a develope quickstart you would never do sudo -H pip install -U kolla as a operator quickstart you should | 20:22 |
| sean-k-mooney | hawi: can you do netstat -lnp instead of an | 20:23 |
| vhosakot | sean-k-mooney: as a dev, do you see build.py directly in the repo? | 20:23 |
| vhosakot | see == use | 20:23 |
| sean-k-mooney | vhosakot: yes always | 20:23 |
| jamesbenson | vhosakot inc0: https://snag.gy/yzuFOw.jpg | 20:24 |
| sean-k-mooney | vhosakot: same for tools/kolla-ansible | 20:24 |
| vhosakot | sean-k-mooney: right, I use pip too sometimes to excersise the pip installation part of the project | 20:24 |
| hawi | sean-k-mooney: sure - https://pastebin.com/x073y41R | 20:24 |
| *** ansmith has quit IRC | 20:25 | |
| rwellum | sean-k-mooney: how should I be installing kolla then? | 20:26 |
| rwellum | I just want the build command | 20:26 |
| sean-k-mooney | hawi: 192.168.253.99 and 192.168.253.98 are your external and intenal vip adress correct | 20:26 |
| rwellum | vhosakot: I've tried both --no-cache and --no-cache-dir - no luck | 20:26 |
| vhosakot | jamesbenson: ah, it is an option for the neutron port... direct is a regular port... I've used vnic-type direct to bypass the ovs/hypervisor (SR-IOV/PCS pass thru), I think baremetal option for a baremetal port --> https://docs.openstack.org/python-openstackclient/latest/cli/command-objects/port.html#port-create | 20:27 |
| rwellum | vhosakot: just a vm so I can just redo - :-/ | 20:27 |
| *** mewald has quit IRC | 20:27 | |
| sean-k-mooney | rwellum: for development you dont install, you git clone the repo cd into it and do sudo pip insnstall -e . or sudo pip install -r requirements | 20:27 |
| vhosakot | ^^ rwellum: can you build form repo without installing kolla? | 20:28 |
| inc0 | hmm | 20:28 |
| jamesbenson | okay, so nothing I need to deal with then.. | 20:28 |
| sean-k-mooney | rwellum: sudo pip insnstall -e . installs it system wide but point it to the local git repo so it picks up your chagnes. | 20:28 |
| inc0 | rwellum: do pip freeze | grep docker | 20:28 |
| vhosakot | jamesbenson: so the port type distinguishes what is attached under/to the port (OVS vs baremetal vs normal vs direct). | 20:29 |
| sean-k-mooney | sudo pip install -r requirement.txt install the requirement bunt does not install kolla so you dont get kolla build on your patath and have to call the build script form the tools dir directly | 20:29 |
| jamesbenson | ok? | 20:29 |
| jamesbenson | vhosakot: ok? | 20:29 |
| vhosakot | jamesbenson: yes, normal (default) is good | 20:30 |
| rwellum | inc0: https://www.irccloud.com/pastebin/swR8pPUw/ | 20:30 |
| jamesbenson | r6-410-x are all baremetal... whew :-) | 20:30 |
| sean-k-mooney | vhosakot: vnic type direct mean sriov | 20:30 |
| sean-k-mooney | vnic type baremetal is for ironic | 20:30 |
| hawi | sean-k-mooney: yes | 20:31 |
| vhosakot | ah right... | 20:31 |
| inc0 | rwellum: try pip install docker-py==1.6.* | 20:31 |
| rwellum | inc0: docker --version is 1.12 | 20:32 |
| inc0 | docker-py is what I'm interested in | 20:32 |
| sean-k-mooney | hawi: so the issue i think you are having is that the loadblancer is stating but looking at https://pastebin.com/x073y41R you dont have any of the service running on this node | 20:32 |
| jamesbenson | vhosakot: do the vm's need to be in the allowed address pairs? This is a new screen for me as well... | 20:32 |
| jamesbenson | I wouldn't think so.. | 20:32 |
| *** jrist has joined #openstack-kolla | 20:33 | |
| vhosakot | jamesbenson: yes, the default address pairs (combination of IP and mac) is good to ping and SSH into the VM after opening the icmp and SSH ports in sec group rules. | 20:33 |
| vhosakot | jamesbenson: I used allowed address pairs to allow traffic that neutron drops like multicast for eg. | 20:34 |
| hawi | sean-k-mooney: yes. because kolla hasnt brought them up yet? | 20:34 |
| sean-k-mooney | hawi: yes, can you ping 192.168.253.99 and 192.168.253.98 | 20:34 |
| hawi | no | 20:34 |
| rwellum | inc0: that appears to be working for me :) | 20:35 |
| rwellum | ni why | 20:35 |
| inc0 | docker-py at some point changed its API | 20:35 |
| vhosakot | rwellum: ah, did "pip install docker-py==1.6.* work? | 20:35 |
| hawi | sean-k-mooney: the only things that are up are keepalived, haproxy, elasticsearch, cron, kolla_toolbox and fluentd | 20:35 |
| hawi | sean-k-mooney: at least on controllers | 20:35 |
| sean-k-mooney | hawi: you should should be able to ping the vip even without the services running | 20:36 |
| vhosakot | so, uninstalling kolla did not uninstall your old docker-py.. I see | 20:36 |
| rwellum | it's building now vhosakot | 20:36 |
| sean-k-mooney | hawi: that vip should only be present on 1 of the 3 controlers also | 20:36 |
| hawi | sean-k-mooney: arp is also incomplere | 20:37 |
| hawi | sean-k-mooney: arp is also incomplete | 20:37 |
| hawi | so it cant be fw issue | 20:37 |
| hawi | on same lan | 20:37 |
| *** eaguilar is now known as eaguilar|afk | 20:38 | |
| sean-k-mooney | hawi: do you have any other instance or keepalived running on the same lan? | 20:38 |
| jamesbenson | vhosakot: I added them and still no ping access to VM's. | 20:38 |
| jamesbenson | (using cirros) | 20:38 |
| jamesbenson | can still ping router | 20:39 |
| hawi | well .... YES :( i think i do have all-in-one installation.... | 20:39 |
| hawi | and it's perhaps with same keepalived id | 20:39 |
| hawi | and maybe this leads to it | 20:39 |
| *** pbourke_ has joined #openstack-kolla | 20:40 | |
| sean-k-mooney | if its working correctly one of the api interfaces on teh controler will look like this http://paste.openstack.org/show/616209/ | 20:40 |
| sean-k-mooney | hawi: yes if you did not set this for your new deployment they will conflict | 20:40 |
| *** pbourke_ has quit IRC | 20:41 | |
| sean-k-mooney | keepalived uses vrrp wich works at the l2 broadcast dompain level. i your have 2 keepalived instance useing the same id that are not part of the same cluster it will break for both cluster when the scond cluter is deployed | 20:42 |
| hawi | sean-k-mooney: well. that has driven me crazy for half a day, i totally forgot it | 20:42 |
| sean-k-mooney | hawi: set keepalived_virtual_router_id:42 in the global.yml and it should fix it. | 20:42 |
| sean-k-mooney | hawi: yes i spent quite a while figuring that out myself | 20:43 |
| hawi | ubuntu@maakri-maas:~$ ping 192.168.253.98 | 20:43 |
| hawi | PING 192.168.253.98 (192.168.253.98) 56(84) bytes of data. | 20:43 |
| hawi | 64 bytes from 192.168.253.98: icmp_seq=1 ttl=64 time=0.294 ms | 20:43 |
| hawi | welll, now it works | 20:43 |
| hawi | sean-k-mooney: thanks a lot | 20:44 |
| sean-k-mooney | hawi: so what happening is that haproxy is correctly starting but keepalived never floated the vip ips to one of the host because it was geting invalid vrrp messages form the all in one deployment so it never formed a quorm | 20:45 |
| hawi | i really thought that i reinstalled all-in-one baremetal box and included as one boxes for current setup | 20:45 |
| jamesbenson | inc0: why would qdhcp not start? | 20:45 |
| sean-k-mooney | hawi: also 42 is as always the answer to everyting | 20:45 |
| inc0 | jamesbenson: anything in dhcp agent logs? | 20:45 |
| hawi | :) | 20:45 |
| jamesbenson | in the neutron_dhcp_agent? | 20:46 |
| *** rwallner has quit IRC | 20:46 | |
| inc0 | yeah | 20:47 |
| jamesbenson | nothing that I see: http://paste.openstack.org/show/616210/ | 20:47 |
| jamesbenson | look at that last line... | 20:47 |
| * inc0 is confused | 20:47 | |
| jamesbenson | ? | 20:48 |
| *** rwallner has joined #openstack-kolla | 20:48 | |
| jamesbenson | (btw, how do you do that? \/me? | 20:48 |
| inc0 | yeah | 20:48 |
| inc0 | /me does it | 20:49 |
| jamesbenson | k | 20:49 |
| inc0 | line doesnt' look bad | 20:49 |
| jamesbenson | okay | 20:49 |
| *** rwallner_ has joined #openstack-kolla | 20:49 | |
| *** goldyfruit has quit IRC | 20:50 | |
| jamesbenson | inc0: what's confusing then? | 20:50 |
| inc0 | why no namespace | 20:50 |
| jamesbenson | no idea? | 20:51 |
| inc0 | vms don't get dhcp? | 20:51 |
| jamesbenson | they get an IP | 20:51 |
| jamesbenson | I'd share screen with you if I could/you wanted ;-) | 20:51 |
| jamesbenson | currently the vm has both internal and floating | 20:52 |
| inc0 | btw you're using dvr? | 20:52 |
| jamesbenson | what ever is default... | 20:52 |
| inc0 | well if it works it works | 20:52 |
| inc0 | let's not ask too many questions;) | 20:52 |
| jamesbenson | that stuff is sort of above my scope.. | 20:52 |
| sean-k-mooney | dvr is off by default i think | 20:52 |
| jamesbenson | definitely dont enable random stuff... let me check | 20:52 |
| jamesbenson | globals? | 20:52 |
| inc0 | you'd know if you'd enable dvr | 20:53 |
| inc0 | its off by defaylt | 20:53 |
| jamesbenson | #enable_neutron_dvr: "no" | 20:53 |
| *** rwallner has quit IRC | 20:53 | |
| sean-k-mooney | jamesbenson: so what is the issue your haveng. i missed the start of the conv | 20:53 |
| sean-k-mooney | dhcp did not start? | 20:54 |
| jamesbenson | I can ping my router, but no vm's can get outside access, no ping etc... | 20:54 |
| jamesbenson | qdhcp isn't showing up | 20:54 |
| *** rwallner_ has quit IRC | 20:54 | |
| sean-k-mooney | and did you say the vms were getting ips or not | 20:54 |
| jamesbenson | I had 3 network nodes, now I have one | 20:54 |
| jamesbenson | they have both internal and floating ip | 20:54 |
| sean-k-mooney | ok so its likely not an issue with dhcp then | 20:55 |
| sean-k-mooney | but rather your routers | 20:55 |
| jamesbenson | and I can ping to my router | 20:55 |
| jamesbenson | the routers public port... | 20:55 |
| jamesbenson | this is the script I used to create all of my networking: https://github.com/JamesOBenson/Kolla-misc/blob/master/scripts/setup_networking.sh | 20:55 |
| sean-k-mooney | are the tenant networks vxlan or vlan and if vlan have you trunked the openstack range across your phyical switchs | 20:56 |
| jamesbenson | I've used it in the past without issue..., but maybe I fat fingered something without realizing it | 20:56 |
| jamesbenson | vxlan | 20:56 |
| jamesbenson | mtu is 1450? | 20:57 |
| sean-k-mooney | https://github.com/JamesOBenson/Kolla-misc/blob/master/scripts/setup_networking.sh#L26 you truned off dhcp on the private network too | 20:57 |
| jamesbenson | needs to be on? I thought I always had it off... maybe not... :-/ | 20:58 |
| sean-k-mooney | my quess is the vms are allocated ips from nova but you dont have an instace of the dhcp server running because you set --no-dhcp where you created the private network | 20:58 |
| jamesbenson | let me fix that and check :-) | 20:58 |
| sean-k-mooney | jamesbenson: well it is usally off for the external network and on for internal networks | 20:58 |
| jamesbenson | dhcp on both public and private? | 20:58 |
| jamesbenson | ah | 20:59 |
| jamesbenson | I did copy and paste... | 20:59 |
| jamesbenson | so... | 20:59 |
| jamesbenson | dang.. | 20:59 |
| inc0 | --no-dhcp would explain why there is no dhcp | 20:59 |
| sean-k-mooney | inc0: well i would hope so or there is someone on the openstack client team i need to have a conversation with regarding there nameing convetions | 20:59 |
| * jamesbenson feeling sheepish now.... | 21:00 | |
| sean-k-mooney | inc0: by the way the init-runonece script i think has been broken for a while | 21:00 |
| jamesbenson | thank you! 🙏 this fixed it! | 21:01 |
| sean-k-mooney | inc0: it always gives me errors seting up the routers does it work for you? | 21:01 |
| jamesbenson | I finally have openstack up and running again | 21:01 |
| sean-k-mooney | jamesbenson: :) | 21:02 |
| jamesbenson | now to deploy a production ::finally:: | 21:02 |
| jamesbenson | I think that was the last bug... | 21:02 |
| jamesbenson | but I will try on the ceph one last time... :-D | 21:03 |
| sean-k-mooney | i set up openstack on a bunch of server over a year ago and have been doing my developemnt of openstack in vms form that cluster ever since | 21:03 |
| inc0 | jamesbenson: I think you should be fine with more than one net node;) | 21:03 |
| jamesbenson | ok | 21:04 |
| jamesbenson | if I do 3 network nodes, will it be in HA automatically or do I need to do configs? | 21:04 |
| inc0 | ha automatically | 21:04 |
| jamesbenson | safe to overlay them on the controllers as well? | 21:04 |
| jamesbenson | or too much for controllers? | 21:05 |
| sean-k-mooney | you could also enable dvr. yes you can locate network nodes and conntrolers | 21:05 |
| jamesbenson | dvr? | 21:05 |
| inc0 | distributed virtual router | 21:05 |
| inc0 | basically floating ip lands on compute node | 21:05 |
| sean-k-mooney | jamesbenson: i have ceph+contolers+network node + compute all on same server and it works fine | 21:06 |
| sean-k-mooney | jamesbenson: basically i made my 3 controllers 3 all in one nodes | 21:06 |
| inc0 | depends on scale/traffic | 21:06 |
| sean-k-mooney | jamesbenson: unless you have a lot of computenodes teh controlers are usully not that busy | 21:06 |
| sean-k-mooney | true | 21:07 |
| sean-k-mooney | the only think i would change if i was to do it again would be to not have the ceph osds and ceph mons colocated | 21:07 |
| jamesbenson | 25 compute nodes? | 21:07 |
| inc0 | osd can eat cpu like hungry....honey badger | 21:07 |
| inc0 | it won't care | 21:07 |
| sean-k-mooney | jamesbenson: 25 should be fine | 21:08 |
| inc0 | it won't give a... you knw | 21:08 |
| jamesbenson | http://pixel.nymag.com/imgs/daily/intelligencer/2016/08/18/18-honey-badger-teeth.w710.h473.2x.jpg | 21:08 |
| sean-k-mooney | jamesbenson: inc0 deployed 235 nodes with 3 contolers on the osic cluster a while back and they were idel most of the time even under load | 21:09 |
| jamesbenson | damn... | 21:09 |
| inc0 | yeah | 21:09 |
| jamesbenson | I guess I'll be fine then ;-) | 21:09 |
| inc0 | but network nodes can be stressed out if all these vms will start downloadig stuff at once | 21:09 |
| inc0 | and ceph can get busy when rebalancing | 21:09 |
| inc0 | that's why they're separate usually | 21:10 |
| sean-k-mooney | inc0: is the video still on your youtube out of interest. ya they can be thats why i was suggesting enableing dvr | 21:10 |
| inc0 | but for 25 nodes that should be ok | 21:10 |
| jamesbenson | okay, so 3 nodes with compute/ceph/network. 1 monitor/ceph 3compute/ceph and 25 compute.... | 21:10 |
| sean-k-mooney | well for ceph really you should use another nic for that | 21:10 |
| inc0 | yeah vid is up | 21:10 |
| vhosakot | https://rlv.zcache.com/honey_badger_card-r84cef6dd4372467381e1858a8d6c7083_xvuat_8byvr_324.jpg | 21:10 |
| inc0 | honey badger is one of our secret mascots | 21:11 |
| vhosakot | lolol | 21:11 |
| inc0 | another one, and I refuse to say otherwise, is koalla+ glue | 21:11 |
| inc0 | kolla = glue in greek | 21:11 |
| vhosakot | yes | 21:12 |
| jamesbenson | just keep honey badger away from koala bear | 21:12 |
| inc0 | koala is animal that's always high from eucalyptus | 21:12 |
| inc0 | a junkie | 21:12 |
| sean-k-mooney | jamesbenson: i would got 3 nodes with controler/ceph mon/network + 4 nodes ceph osds/compute + compute | 21:12 |
| inc0 | what would junkie do with glue? | 21:12 |
| inc0 | I'll just leave it to your imagination | 21:12 |
| sean-k-mooney | jamesbenson: but what you suggeted would work too | 21:13 |
| sean-k-mooney | jamesbenson: you generally want to have 3 ceph monitor nodes | 21:13 |
| *** eaguilar|afk is now known as eaguilar | 21:14 | |
| sean-k-mooney | jamesbenson: if the ceph mon goes down the the cluster goes offline so 1 ceph mond means your ha cloud has a singel point of failute | 21:14 |
| *** eaguilar is now known as eaguilar|afk | 21:14 | |
| jamesbenson | these are the specs of our machines... we usually put all of the controller/etc on 710's and 610/410== compute http://paste.openstack.org/show/616215/ | 21:15 |
| jamesbenson | yeah I would do 3 mon's | 21:15 |
| jamesbenson | and only 2 nics, both 1 G.... but that's the whole prod cluster | 21:16 |
| jamesbenson | the 410's don't have a 10G card in them... | 21:16 |
| sean-k-mooney | the dell 710s should eailly handel the controlr and ceph load. | 21:16 |
| sean-k-mooney | do the 710s and 610 have 10G? | 21:17 |
| jamesbenson | yes, | 21:17 |
| jamesbenson | 10G is internal | 21:17 |
| sean-k-mooney | im assuming they are dual port? if so make sure ceph is on 1 of the 10G ports with ovs on the other. | 21:18 |
| jamesbenson | yeah, I think dual | 21:18 |
| jamesbenson | in the globals, that's just storage and cluster right? | 21:19 |
| sean-k-mooney | yes | 21:19 |
| sean-k-mooney | set storage and clust to 1 of the 10G ports and neutron_external_interface to the other | 21:19 |
| sean-k-mooney | the api network can be shared with ceph or you can use one of the 1G ports | 21:20 |
| sean-k-mooney | basiclaly you dont want ceph replication to impact the perfomance of the vms network | 21:20 |
| sean-k-mooney | if you can afford to give ceph its own nic its a good idea. | 21:21 |
| kfox1111 | yeah. dedicated ceph backend networks always good if you can afford it. | 21:22 |
| sean-k-mooney | its not required but it help when things explode and it starts doing a recovery operation | 21:22 |
| kfox1111 | same with dedicated ceph servers. better to have them on dedicated harware too. | 21:22 |
| jamesbenson | Not sure if I can wire it up like that, but I shall try my best :-) | 21:22 |
| *** lpetrut has quit IRC | 21:22 | |
| jamesbenson | we have old, old drives and even 1 or 2 bad HD ports in the servers... so... replication is good and why we usually run raid5/6 | 21:23 |
| jamesbenson | but I want to try to optimize this and have redundancy, so I'll probably mirror the os for the controllers and seperate the HD's for ceph | 21:23 |
| sean-k-mooney | jamesbenson: sounds resonable. | 21:24 |
| kfox1111 | yeah. usually old drives imply old computers. so replicating across machines helps. :) | 21:24 |
| jamesbenson | I know I could definitely give ceph a 1G connection | 21:24 |
| jamesbenson | but we have been itching to finially use the 10g... been sitting there for a while from what I understand | 21:24 |
| kfox1111 | 10g's much better. | 21:25 |
| kfox1111 | much much better. :) | 21:25 |
| kfox1111 | > 10 x better. :) | 21:25 |
| kfox1111 | like, dsl vs dialup. :) | 21:25 |
| sean-k-mooney | one thing you can do and i do this for my work setup is create macvtap ports off the phyical nics and used those in the kolla config. | 21:25 |
| kfox1111 | sean-k-mooney: I do the same but with openvswitch. | 21:26 |
| sean-k-mooney | it does reduce the perfromace somwhat but it give you flexablity to chagen your mind | 21:26 |
| sean-k-mooney | kfox1111: ovs is a better option now that we have external ovs | 21:26 |
| * kfox1111 nods | 21:26 | |
| jamesbenson | so I'll do something liek this then: http://paste.openstack.org/show/616217/ | 21:27 |
| *** ducttape_ has joined #openstack-kolla | 21:28 | |
| jamesbenson | yeah, i've been using the internal network for deploying... deployment node is on seperate rack | 21:28 |
| sean-k-mooney | this is what my contolers look like http://paste.openstack.org/show/616218/ | 21:28 |
| jamesbenson | and use dvr? | 21:28 |
| kfox1111 | looks good, but I'd recommend not putting compute on controllers if you can help it, | 21:29 |
| kfox1111 | or put them in an aggretate only for your own mgmt use. | 21:29 |
| kfox1111 | I've been using dvr to good effect. | 21:29 |
| jamesbenson | how do you do that? I've been meaning to figure that out | 21:29 |
| kfox1111 | havent' tried dvr+l3-ha, but hear its stable now. | 21:29 |
| kfox1111 | you set up host aggregates for each of the hardware. | 21:30 |
| sean-k-mooney | kfox1111: i use dvr + l3 ha in my cluster and it works | 21:30 |
| kfox1111 | then add a label to the aggregate like, compute=v1 or controller=v1 | 21:30 |
| kfox1111 | then modify the flavors to have the same key. | 21:30 |
| kfox1111 | sean-k-mooney: cool. thanks for the info. I'll expidite switching it on then. :) | 21:30 |
| kfox1111 | we probably should test out dvr+l3-ha and default it on in kolla-kubernetes. | 21:31 |
| *** ducttap__ has quit IRC | 21:31 | |
| jamesbenson | kfox1111: enabling that is just enabling dvr and putting it on 3 nodes? | 21:31 |
| jamesbenson | err sean-k-mooney sorry | 21:31 |
| jamesbenson | ^^ | 21:31 |
| sean-k-mooney | jamesbenson: enable dvr and just deploy yes | 21:31 |
| sean-k-mooney | what dvr will do is create a router on every node | 21:32 |
| sean-k-mooney | each of the network nodes should be running in dvr_snat mode for servers without a floating ip | 21:32 |
| kfox1111 | yeah. dvr goes on all the nodes. dvr_snat goes on the network nodes. | 21:32 |
| kfox1111 | I see they finally added octavia support for l3+ha/dvr in pike. | 21:33 |
| kfox1111 | though the implementation I think is a little suboptimal. :/ | 21:33 |
| sean-k-mooney | jamesbenson: the effect of that is if the traffic is internal it get route betteen the compute nodes without needing to go to the network node | 21:33 |
| jamesbenson | interesting :-) | 21:33 |
| jamesbenson | minimal overhead? | 21:34 |
| kfox1111 | yeah. we had a bunch of vm's stay up/responsive with the network controller nodes going out due to ups blowing up. | 21:34 |
| kfox1111 | they had fips, and were on city power, so worked well. :) | 21:34 |
| jamesbenson | fun :-) | 21:34 |
| *** eaguilar|afk is now known as eaguilar | 21:34 | |
| jamesbenson | lol | 21:34 |
| kfox1111 | hopefully that never happens to you though. :) | 21:34 |
| sean-k-mooney | the overhead is relitivly little | 21:35 |
| kfox1111 | yeah. compared to the scalability, the overhead is nothing. | 21:35 |
| *** eaguilar is now known as eaguilar|afk | 21:35 | |
| jamesbenson | well, our own IT department brings the network down for us... we lose connectivity every couple of months for no reason... | 21:35 |
| kfox1111 | you don't have to contend with others for network nodes. | 21:35 |
| kfox1111 | lame. | 21:35 |
| jamesbenson | yep | 21:35 |
| sean-k-mooney | the compute nodes are doing more work but not significatly and if you are looking enought to have traffic going between 2 vms on the same server it uses none of your nic bandwidth | 21:35 |
| jamesbenson | but hey, university ;-D | 21:35 |
| jamesbenson | love it :-) | 21:36 |
| sean-k-mooney | that said my team is currently working on rewiting dvr to make it better | 21:36 |
| jamesbenson | cool | 21:37 |
| kfox1111 | oh? | 21:37 |
| jamesbenson | any other tips from the pro's here ;-) | 21:37 |
| sean-k-mooney | currently it uses kernel network namespace to do the routing. we are working to move it into openflow rules and do the routing directly in ovs | 21:37 |
| kfox1111 | ah. | 21:37 |
| jamesbenson | seriously though, you guys blow me out of the water... and I love it. I love learning from all of you guys. | 21:37 |
| kfox1111 | yeah, that would be better. | 21:37 |
| kfox1111 | though would ake the linux bridge folks upset? | 21:38 |
| *** eaguilar|afk is now known as eaguilar | 21:38 | |
| kfox1111 | is anyone still pushing linux bridge? | 21:38 |
| sean-k-mooney | kfox1111: we have it work for ipv4 - floating ips but that is easy to add. we need 2 flag bits to be added to ovs to get ipv6 to work. | 21:38 |
| *** eaguilar is now known as eaguilar|afk | 21:38 | |
| jamesbenson | doesnt ovs blow linux bridge out of the water with everything? | 21:38 |
| kfox1111 | I still want ipv6 fip suport too... | 21:39 |
| sean-k-mooney | jamesbenson: not multicast vxlan | 21:39 |
| jamesbenson | ok | 21:39 |
| sean-k-mooney | ovs does not support multicast tunneling so it has to unicast all tunnels | 21:39 |
| kfox1111 | I don't understand neutron's stance on ipv6 fips. fips are not about nating. they are about keeping state (ip address) seperate from instance to minimize pets. | 21:40 |
| sean-k-mooney | that means if you have lots of multicast or broadcast traffic ovs gets slower | 21:40 |
| *** ansmith has joined #openstack-kolla | 21:40 | |
| sean-k-mooney | kfox1111: well fips can still be a thing in the ipv6 world | 21:40 |
| kfox1111 | yeah. I think so. but have seen pushback from the neutron team on ever supporting them. | 21:41 |
| sean-k-mooney | there is even an extra extention to allow ipv4->ipv6 fips or visversa but its not supproted by most drivers | 21:41 |
| kfox1111 | oh, that would be cool. :) | 21:41 |
| sean-k-mooney | jamesbenson: but back to your question no your deployment looks good http://paste.openstack.org/show/616217/ | 21:42 |
| kfox1111 | totally could see internal ipv6 with a few things exposed via ipv4 fips. | 21:42 |
| sean-k-mooney | i remember someone wannted to add fip port frowarding at one point too but i dont think that went anywhere | 21:43 |
| jamesbenson | thanks sean-k-mooney: good to get validation from experts :-) | 21:43 |
| sean-k-mooney | jamesbenson: well kfox1111 is more of an expert then i am | 21:43 |
| sean-k-mooney | kfox1111: how is your production cloud going | 21:44 |
| kfox1111 | experienced in different ways, lets say. | 21:44 |
| kfox1111 | good and bad. | 21:44 |
| kfox1111 | I'm involved with 3 openstack clouds. | 21:44 |
| sean-k-mooney | i need to finally learn how to do openstack upgrades soon | 21:44 |
| kfox1111 | the team is seriously discussing dismantling one of them. | 21:44 |
| kfox1111 | its chef managed, and old, and they never used it like a cloud anyway, so they are talking about switching it to ovirt. :/ | 21:45 |
| sean-k-mooney | so far i have just set up a new cloud every 6 months but i proably should use the upgrade playbook now that i have other people ueing it | 21:45 |
| kfox1111 | the second was about 150 nodes, and I'm about to pull all about a dozen and move them to k8s. :/ | 21:45 |
| sean-k-mooney | nice | 21:46 |
| sean-k-mooney | and ouch on the frist one | 21:46 |
| kfox1111 | the third is realitively stable and I'm working on switching it to kolla-kubernetes. | 21:46 |
| kfox1111 | got k8s working on the test cloud this morning. | 21:46 |
| sean-k-mooney | kfox1111: what do you use to install k8s | 21:46 |
| kfox1111 | I've been hand rolling them. | 21:46 |
| sean-k-mooney | i saw that we have k8s contaieers in kolla but dont know how to use them | 21:47 |
| kfox1111 | that way I always know all the pieces. | 21:47 |
| kfox1111 | been baking as much of that knowlege as I can though into helm charts. | 21:47 |
| kfox1111 | the k8s containers in kolla are all but dead now. | 21:47 |
| kfox1111 | the issue was, k8s prebuilt containers dont have ceph support. | 21:48 |
| sean-k-mooney | kfox1111: yes but then to use helm dont you need a seed k8s to run helm | 21:48 |
| kfox1111 | but we got them to make an external ceph conntainer, so we no longer need it in the controllers. | 21:48 |
| sean-k-mooney | ah ok | 21:48 |
| kfox1111 | sean-k-mooney: I have been working with jascott1 to get tiller static pod support upstream into helm. | 21:48 |
| kfox1111 | I basically drop a pod description for tiller into /etc/kubernetes/manifests, and am good to go. | 21:48 |
| kfox1111 | the idea is to make it such that 'helm init --static-pod > /etc/kubernetes/tiller.yaml' would work out of the box. | 21:49 |
| jamesbenson | neat :-) | 21:49 |
| kfox1111 | then you can launch your sdn like: 'helm install stable/flannel --set subnet=mystuff/16' etc. | 21:49 |
| sean-k-mooney | ah right so you jsut run the kublet on the host and then drop in the pod def and it deploy the contolplane on that node? | 21:49 |
| jamesbenson | sean-k-mooney, kfox1111, inc0... all experts to me :-D | 21:49 |
| kfox1111 | I've yet to submit it yet, but https://github.com/kfox1111/charts/tree/flannel if your interested. | 21:50 |
| kfox1111 | I'm running it in production now. | 21:50 |
| kfox1111 | sean-k-mooney: yup. :) | 21:50 |
| kfox1111 | same as the rest of the k8s control plane. | 21:50 |
| dims | kfox1111 : i spent the day looking through your keystone k8s PR (https://github.com/kubernetes/kubernetes/pull/25391) | 21:50 |
| kfox1111 | I think with a static pod, you could even then use helm to manage your kube-control plane. | 21:50 |
| sean-k-mooney | kfox1111: well k8s is still new to me so i have never really got a deployment working | 21:51 |
| kfox1111 | dims: hey. :) | 21:51 |
| *** lucasxu has quit IRC | 21:51 | |
| kfox1111 | dims: cool. whatcha think? | 21:51 |
| jamesbenson | qq: we have a san that we would love to incorporate potentially, just do an iscsi to a server and join it that way? | 21:51 |
| dims | dunno yet :) let me run some experiments and ping you back next week kfox1111 | 21:51 |
| sean-k-mooney | kfox1111: i used the vagrant stuff that portdirct i think worte to work but that the closet i have got to k8s so far | 21:51 |
| kfox1111 | sean-k-mooney: play with kubeadm in some vm's, then look at the controller in /etc/kubernetes/manifests. they do some clever things. :) | 21:52 |
| kfox1111 | dims: cool. let me know if you need any help. | 21:52 |
| sean-k-mooney | kfox1111: yep its on my todo list | 21:52 |
| *** eaguilar|afk is now known as eaguilar | 21:52 | |
| kfox1111 | I'm liking the path kubeadm's headed. | 21:52 |
| kfox1111 | using k8s to deploy k8s is a great idea. | 21:52 |
| *** eaguilar is now known as eaguilar|afk | 21:52 | |
| kfox1111 | I'd prefer if they used helm for a few pieces though. | 21:53 |
| *** eaguilar|afk is now known as eaguilar | 21:54 | |
| *** eaguilar is now known as eaguilar|afk | 21:54 | |
| *** jamesbenson has quit IRC | 21:55 | |
| *** jamesbenson has joined #openstack-kolla | 21:58 | |
| *** jamesbenson has quit IRC | 22:00 | |
| *** jamesbenson has joined #openstack-kolla | 22:00 | |
| sean-k-mooney | kfox1111: deploying k8s in contaienrs would solve all my problems since by man servers at home are using opensuse tumbelweed which nothing supports | 22:00 |
| *** eaguilar|afk is now known as eaguilar | 22:00 | |
| *** eaguilar is now known as eaguilar|afk | 22:00 | |
| kfox1111 | sean-k-mooney: the great beinifit to the k8s using k8s to deploy is that there is very little needed on the host system. | 22:01 |
| *** jamesbenson has quit IRC | 22:01 | |
| kfox1111 | docker, kubelet, and cni. | 22:01 |
| kfox1111 | kubelet and cni are all statically linked binaries I think. | 22:01 |
| kfox1111 | so should be able to use that os just fine. :) | 22:01 |
| *** jamesbenson has joined #openstack-kolla | 22:01 | |
| kfox1111 | though it might be a little bit more work. | 22:02 |
| *** trozet has quit IRC | 22:02 | |
| kfox1111 | kube-apiserver, kube-controller-manager, kube-scheduler, kube-proxy, sdn, etcd, etc are all containers. | 22:02 |
| sean-k-mooney | thats the thing using minikube it got 90% of the way there but i dont know enought about k8s to fix that last 10% | 22:02 |
| sean-k-mooney | at least not yet | 22:02 |
| kfox1111 | is opensuse systemd based yet? | 22:03 |
| sean-k-mooney | yes for quite a while i think | 22:03 |
| kfox1111 | I'd put the odds at ~75% then that the redhat rpms from k8s might just work out of the box. | 22:04 |
| sean-k-mooney | im only using it to get a new enough kernel to use bache with partition and ceph lumious binaries | 22:04 |
| sean-k-mooney | *bcache with partion support | 22:04 |
| kfox1111 | ah. | 22:04 |
| sean-k-mooney | my plan was to use ceph with bluestore osd on bcache block devices and then deploy k8s on top and then try kolla-k8 after | 22:05 |
| kfox1111 | sean-k-mooney: here's the extent of the rpms needed: | 22:05 |
| kfox1111 | https://pastebin.com/wJPq7MRy | 22:05 |
| kfox1111 | that would be sweet. :) | 22:05 |
| *** jamesbenson has quit IRC | 22:06 | |
| kfox1111 | all the binaries in the rpm's are go binaries I think. so should be statically linked. | 22:06 |
| sean-k-mooney | the problem with tumpleweed thoug is its like arch. bassily deploying master of everything in package form | 22:06 |
| sean-k-mooney | so there are just 1 or 2 bugs | 22:07 |
| kfox1111 | still willing to bet k8s would work just fine on it. :) | 22:07 |
| kfox1111 | and I don't bet very often. | 22:07 |
| kfox1111 | I think the only thing you may have to do is tweak the provided systemd scripts alittle to match suse thingies, if there are changes. | 22:07 |
| *** ducttap__ has joined #openstack-kolla | 22:08 | |
| sean-k-mooney | ya but thats a project for antoher weekend | 22:08 |
| kfox1111 | fair enough. :) | 22:09 |
| sean-k-mooney | by the way is kolla-ansible destroy ment to delete all container and volumes on localhost | 22:09 |
| sean-k-mooney | localhost is only listed in my inventory under deployment section | 22:10 |
| sean-k-mooney | i would have taught it should not execute the destroy plays on the deployment node | 22:11 |
| *** ducttape_ has quit IRC | 22:11 | |
| *** eaguilar|afk is now known as eaguilar | 22:15 | |
| *** eaguilar is now known as eaguilar|afk | 22:15 | |
| hawi | it seems that not my lucky day for deployment, now something with ceph. failed: [dc2-stor-002] (item=(0, {u'fs_uuid': u'', u'journal_device': u'/dev/sdb', u'journal': u'/dev/sdb2', u'partition': u'/dev/sdb1', u'partition_num': u'1', u'journal_num': 2, u'fs_label': u'', u'device': u'/dev/sdb', u'external_journal': False})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", | 22:23 |
| hawi | "external_journal": false, "fs_label": "", "fs_uuid": "", "journal": "/dev/sdb2", "journal_device": "/dev/sdb", "journal_num": 2, "partition": "/dev/sdb1", "partition_num": "1"}], "msg": "Container exited with non-zero return code"} | 22:23 |
| rwellum | sean-k-mooney: I have a tool that'll install kolla-k8s on a VM in less than 40m :) Going to upstream it as soon as I get permission. It's built around kubeadm and follows the deployment guide we wrote. | 22:27 |
| *** emccormick has quit IRC | 22:27 | |
| sean-k-mooney | rwellum: does that include building the images? | 22:28 |
| sean-k-mooney | kolla-ansible will deploy 235 nodes with prebuilt immages in 22mins :P | 22:28 |
| rwellum | No it pulls from dockerhub, but in my repo I have a README which describe how to do this. | 22:28 |
| rwellum | sean-k-mooney: not trying to compete - kolla-ansible is production ready. Just if you want to play with k8s. | 22:29 |
| sean-k-mooney | rwellum: hehe i know but ya if you do get permission to upstream it or publish it i would be happy to give it a try | 22:29 |
| *** emccormick has joined #openstack-kolla | 22:29 | |
| rwellum | Yeah just getting through legal here at $$Job, then hopefully the cores here will give it a good review. | 22:30 |
| *** manheim has joined #openstack-kolla | 22:30 | |
| rwellum | Mind you I don't have the build images step working completely. Seems to work, but when I look at the service - it's not running my code annoyingly. | 22:31 |
| sean-k-mooney | rwellum: well for building hte images you just use kolla right. but then you need to update the charts to pull from your registry instaead of the docker hub | 22:33 |
| *** manheim has quit IRC | 22:35 | |
| sean-k-mooney | rwellum: kfox1111 i kindo fo wish i could use kolla-ansible to deploy k8s then use that to deploy openstack | 22:35 |
| sean-k-mooney | i understand how kolla-ansible works | 22:36 |
| kfox1111 | inc0: ping | 22:36 |
| kfox1111 | what do you think of this: | 22:38 |
| kfox1111 | " | 22:38 |
| kfox1111 | Then, to consume it in tripleo, in cae kolla remains on jewel, would you | 22:38 |
| kfox1111 | suggest use a build override to point kolla build to the luminous repos? | 22:38 |
| kfox1111 | Are there drawbacks you can think of?" | 22:38 |
| kfox1111 | full email here: https://www.redhat.com/archives/rdo-list/2017-July/msg00055.html | 22:38 |
| sean-k-mooney | kfox1111: using build override to select ceph version? | 22:46 |
| kfox1111 | yeah. I've never used build overrides, so not sure what the answer is. | 22:47 |
| sean-k-mooney | if they are refereing to template overrides the i would say we pin to jewel in the conteiner for pike and those who want luminios can do so themselfve via the template override | 22:48 |
| sean-k-mooney | but yes it will cause issues on upgrade | 22:48 |
| sean-k-mooney | that said we have upgrade playbooks per role yes? | 22:48 |
| *** mattmceuen has quit IRC | 22:49 | |
| sean-k-mooney | so maybe not | 22:49 |
| kfox1111 | I think they just want to test. | 22:49 |
| kfox1111 | I was just going to suggest a sed /jewel/luminous/ but if the template override works, thats probably better. | 22:49 |
| sbezverk | kfox1111: Have you read it? https://github.com/kubernetes/community/pull/589/files | 22:53 |
| kfox1111 | sbezverk: yeah. sounds kind of like our ceph-rbd, doesn't it? :) | 22:57 |
| sbezverk | kfox1111 yep | 22:57 |
| *** jamesbenson has joined #openstack-kolla | 22:58 | |
| kfox1111 | i wonder how close they are to implementing it. | 22:59 |
| sbezverk | kfox1111: what I do not like is the idea of a single container with all storage utilities | 22:59 |
| kfox1111 | seems like it really depends on mount namespaces too. :) | 22:59 |
| sbezverk | kfox1111 well jul 24 there will be. discussion of this document | 23:00 |
| kfox1111 | its not. its one container per volume type. | 23:00 |
| sbezverk | so I suspect it is not very close.. | 23:00 |
| kfox1111 | ah. bummer. | 23:00 |
| sbezverk | I must have missed it.. I was under impression they want to put everything into one.. | 23:01 |
| kfox1111 | see the section starting at line 67 | 23:01 |
| kfox1111 | line 62 says thats the chosen aproach. | 23:02 |
| *** jamesbenson has quit IRC | 23:02 | |
| sean-k-mooney | what is the best way to check if a distro is ubuntu or centos in bash? | 23:02 |
| kfox1111 | /etc/redhat-release | 23:02 |
| kfox1111 | not sure ubuntu. there's a /etc/debian-release and there are ubuntu code names in it. | 23:03 |
| sbezverk | kfox1111 line 57 confused me.. | 23:03 |
| sean-k-mooney | well im just trying to figure out if i should use the rhel styple networking configs or debian stypes so if /etc/redhat-release is not there i can assume debian based | 23:04 |
| sean-k-mooney | but cool thanks | 23:04 |
| inc0 | tripleo wants to go liminous in Pike? | 23:06 |
| kfox1111 | sbezverk: yeah. they layed out all the apoaches, but only mention the one they picked half way through the doc. a little weird. | 23:06 |
| kfox1111 | inc0: no. | 23:06 |
| kfox1111 | inc0: but they are just asking the best way to test it. | 23:06 |
| kfox1111 | how to build kolla container with it. | 23:06 |
| inc0 | well instead of overrides I'd gladly see a patch | 23:07 |
| inc0 | we won't merge it in Pike, but we can in early Queens | 23:07 |
| sean-k-mooney | well a template override that just set ceph_base_packages is pretty clean for testing | 23:07 |
| kfox1111 | thats not what they are asking I think. | 23:08 |
| kfox1111 | just whats the best way right now to tweak kolla to test with luminous. | 23:08 |
| kfox1111 | sed? template overrides? | 23:08 |
| sean-k-mooney | kfox1111: i would use https://github.com/openstack/kolla/blob/master/doc/image-building.rst#package-customisation | 23:09 |
| inc0 | https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L141 | 23:09 |
| inc0 | override this thing | 23:09 |
| inc0 | set base_centos_yum_repo_packages_remove='centos-release-ceph-jewel' and then append ...-luminous | 23:10 |
| kfox1111 | ok. thanks. | 23:11 |
| sean-k-mooney | kfox1111: basically http://paste.openstack.org/show/616220/ | 23:12 |
| sean-k-mooney | of what inc0 said | 23:12 |
| inc0 | yeah that | 23:13 |
| inc0 | only afair append and remove takes str not list | 23:13 |
| sean-k-mooney | not accouding to the docs | 23:13 |
| kfox1111 | nice. | 23:13 |
| sean-k-mooney | that does not mean its not true | 23:13 |
| inc0 | docs are ok | 23:14 |
| inc0 | my memory is not | 23:14 |
| inc0 | I wrote damn thing:P can't remember if I was clever enough to include option for multiple appends, which would mean list | 23:15 |
| inc0 | I guess I did | 23:15 |
| sean-k-mooney | haha i know the feeling. i have helped write deployment code for ovs-dpdk in 5 different implementations. kolla/devstack/fuel/trippleo and puppet | 23:17 |
| sean-k-mooney | it can get a little confusing which one does what | 23:17 |
| inc0 | that ^ | 23:17 |
| sean-k-mooney | i like devstack and kolla becasue they are simple and the rest make my head hurt more then usually trying to figure out why the exploded | 23:19 |
| inc0 | one is shell and another is shell-over-ssh-written-in-yaml | 23:22 |
| inc0 | with a touch of chroot | 23:23 |
| sean-k-mooney | + a little jinja2 | 23:25 |
| inc0 | btw...isn't it 1am at your place? | 23:25 |
| sean-k-mooney | my issue with puppet is manly ruby and the fact that puppet is not ansibly in generally but manily the ruby | 23:26 |
| *** manheim has joined #openstack-kolla | 23:27 | |
| *** emccormick has quit IRC | 23:32 | |
| sean-k-mooney | inc0: 12:35 | 23:35 |
| sean-k-mooney | inc0: but yes | 23:35 |
| inc0 | go drink! | 23:35 |
| inc0 | it's Friday night in Ireland;) | 23:36 |
| sean-k-mooney | inc0: im actully still in the office working on the kolla dpdk role. i almost have the centos deployment code working. i just need to do 1 last test | 23:37 |
| inc0 | srsly? Man you're crazy | 23:37 |
| sean-k-mooney | well ill be in meeting all next week 2 overlapping f2f planned then wedding to atttend next weekend so im off friday so trying to get thing wrapped up before code freeze | 23:39 |
| sean-k-mooney | its working :) | 23:40 |
| sean-k-mooney | ... almost | 23:41 |
| *** eaguilar|afk has quit IRC | 23:41 | |
| sean-k-mooney | fun centos does not consider adding an ovs bridge to be a hotplug event | 23:41 |
| *** mattmceuen has joined #openstack-kolla | 23:42 | |
| *** mattmceuen has quit IRC | 23:52 | |
| openstackgerrit | sean mooney proposed openstack/kolla-ansible master: introduce contrib playbook for ovs-dpdk https://review.openstack.org/408872 | 23:57 |
| sean-k-mooney | inc0: ^ address all the comments excepth moving it into the main roles and a minior issue with centos not auto configuing the ip on the ovs bridge. | 23:59 |
| sean-k-mooney | inc0: ill adress those on monday. | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
