| *** dardelean_ has joined #openstack-kolla | 00:01 | |
| *** dardelean_ has quit IRC | 00:05 | |
| *** ntpttr_laptop has quit IRC | 00:12 | |
| *** jamesbenson has joined #openstack-kolla | 00:17 | |
| *** goldyfruit has quit IRC | 00:19 | |
| *** jamesbenson has quit IRC | 00:22 | |
| *** cocakolla has joined #openstack-kolla | 00:33 | |
| SamYaple | kfox1111: yea i know how they work! | 00:35 |
|---|---|---|
| kfox1111 | cool. | 00:35 |
| SamYaple | does k8s support shared mounts yet? | 00:36 |
| kfox1111 | it has alpha support for it finally. so I'm implementing it. | 00:36 |
| SamYaple | got it | 00:36 |
| SamYaple | its straight forward, -v /run:/run:shared is kolla-ansible does | 00:36 |
| kfox1111 | I just added the systemd MountFlags=shared thing. as it was set to slave I think. | 00:36 |
| SamYaple | yea it doesnt default to shared | 00:37 |
| kfox1111 | do you have to do the mount bind thing still? | 00:37 |
| kfox1111 | I've had to do that in the past. | 00:37 |
| SamYaple | nope | 00:37 |
| SamYaple | thats for 14.04 | 00:37 |
| kfox1111 | does systemd do a bind mount -rshared for / then? | 00:38 |
| SamYaple | no not quite | 00:39 |
| SamYaple | i forget the exact detains, but MountFlags=shared is a systemd thing that allows the process to make a shared mount | 00:39 |
| SamYaple | anyway, MountFlags=shade + -v /run/netns:/run/netns:shared is all you need | 00:40 |
| kfox1111 | from what I can tell, MountFlags=!0 causes systemd to do a mount / -rshared inside a mount namespace. | 00:40 |
| kfox1111 | confirmed. almost worked in the gate this time: https://review.openstack.org/#/c/505436/7/tools/setup_kubernetes.sh | 00:41 |
| SamYaple | kfox1111: that sounds right-ish. like i said its been a long time | 00:42 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 00:43 |
| kfox1111 | I guess on my other systemd docker1.9 system, I didn't have mountFlags in there at all. so I had to do the -rshared manually. | 00:45 |
| SamYaple | kfox1111: you need docker 1.10.2 minimum for shared mount namespsace stuff IIRC | 00:46 |
| SamYaple | it was in 1.7 and go rolled back in rc2 | 00:46 |
| SamYaple | it was a big hit for me at the time | 00:46 |
| kfox1111 | actually, I got away wit h rslave on another system by sticking to docker 1.9. | 00:46 |
| kfox1111 | as it never specifieiied the mount flags in that version. | 00:47 |
| *** goldyfruit has joined #openstack-kolla | 00:47 | |
| kfox1111 | I've been stuck on that cluster to k8s 1.4 as thats the last version that supported docker 1.9. | 00:48 |
| kfox1111 | now that k8s 1.8 is out now, and they support mount namespaces, I can consider upgrading. | 00:48 |
| SamYaple | you should be using 17.06! | 00:48 |
| kfox1111 | and I can finally put cvmfs in a container! yay! :) | 00:48 |
| SamYaple | get with teh times | 00:48 |
| kfox1111 | nope. | 00:48 |
| kfox1111 | I have no desire to install swarm. | 00:48 |
| SamYaple | youll have to eventually | 00:49 |
| kfox1111 | I'll go from docker 1.12 to containerd 1.0 once its ready. | 00:49 |
| kfox1111 | the distros are dropping docker. | 00:49 |
| SamYaple | as they should | 00:49 |
| kfox1111 | docker renaming swarm to docker really made a lot of folks mad. | 00:49 |
| SamYaple | yea it was a bit of a move | 00:50 |
| SamYaple | ive never liked docker, its more of a "there is nothing better" thing | 00:50 |
| SamYaple | rkt sort of fiziled | 00:50 |
| SamYaple | i am sad | 00:50 |
| kfox1111 | yeah. the shell game though is, | 00:50 |
| kfox1111 | docker -> containerd. swarm -> docker. | 00:50 |
| SamYaple | yep | 00:50 |
| kfox1111 | containerd is what we have mostly cared about for so many years. | 00:50 |
| SamYaple | i suspect we will actually get screwed by containerd too | 00:51 |
| SamYaple | like swarm getting features and contaienrd not | 00:51 |
| SamYaple | but meh | 00:51 |
| kfox1111 | possibly. but far more non docker fols are involved in it. | 00:51 |
| kfox1111 | docker inc really owns 'docker'. | 00:51 |
| kfox1111 | so much so that when they burned the bridge of the opensource community, they had to come up with moby. | 00:51 |
| SamYaple | yup | 00:52 |
| kfox1111 | so someone might consider working on the open source swarm. | 00:52 |
| SamYaple | yea docker really realyl sucked at opensource | 00:52 |
| kfox1111 | as 'docker' really means enterprise swarm now. :/ | 00:52 |
| SamYaple | but they have teh business monopoly.... for now | 00:52 |
| kfox1111 | yup. | 00:52 |
| kfox1111 | they will do ok in a nich for a long time I think. | 00:52 |
| kfox1111 | but they can't possibly keep up with k8s. | 00:52 |
| SamYaple | and they burned all the bridges to make them better | 00:53 |
| kfox1111 | docker will be the windows to k8s's linux. | 00:53 |
| kfox1111 | yeah. | 00:53 |
| SamYaple | man all the bugs they just ignored or closed with cryptic messages because they had a backroom conversation about it | 00:53 |
| * kfox1111 nods | 00:53 | |
| kfox1111 | they totally went fremeum. doesn't really work in the opoensource world. | 00:54 |
| kfox1111 | time and time again its failed. | 00:54 |
| SamYaple | but ofr a few breif seconds they generated alot of value for the shareholders | 00:54 |
| kfox1111 | true. common problem. short term gain for long term pain. | 00:54 |
| kfox1111 | a lot of companies are short sited. | 00:55 |
| kfox1111 | anyway, I don't think it will be a problem too much longer. | 00:56 |
| *** duonghq has joined #openstack-kolla | 01:06 | |
| *** openstackstatus has quit IRC | 01:14 | |
| *** openstackstatus has joined #openstack-kolla | 01:15 | |
| *** ChanServ sets mode: +v openstackstatus | 01:15 | |
| *** tovin07_ has joined #openstack-kolla | 01:16 | |
| kfox1111 | SamYaple: I think it did work. | 01:21 |
| kfox1111 | http://logs.openstack.org/36/505436/8/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-iscsi-nv/989804b/console.html#_2017-10-06_01_14_54_726448 | 01:21 |
| kfox1111 | I'm a little weirded ouut by the: RTNETLINK answers: Invalid argument | 01:21 |
| kfox1111 | but the qrouter and qdhcp namespaces did show up. | 01:22 |
| *** bjolo_ has joined #openstack-kolla | 01:23 | |
| *** dasTor_ has joined #openstack-kolla | 01:23 | |
| *** calbers has quit IRC | 01:24 | |
| *** calbers has joined #openstack-kolla | 01:24 | |
| *** bjolo has quit IRC | 01:26 | |
| SamYaple | kfox1111: thats what happens when you cant acess a namespace (ie its not shared) | 01:26 |
| *** dasTor has quit IRC | 01:26 | |
| SamYaple | so you probably had existing namespaces that did copy down with the shared thing, or they were deleted in some form | 01:26 |
| SamYaple | its harmless to you, but might cause problems at somepoint. i would track it down | 01:26 |
| kfox1111 | yeah... | 01:27 |
| SamYaple | dont worry to hard about it, it can be a tech-debt track it down later thing | 01:27 |
| kfox1111 | so, in non dvr, (its been a while), there are just 2 prefixes, right? | 01:27 |
| kfox1111 | qrouter and qdhcp? | 01:27 |
| SamYaple | in non-dev yes | 01:27 |
| kfox1111 | I only created one router, and one qdhcp. so I wonder where the other is coming from.... | 01:27 |
| kfox1111 | if the host created it, it should show... | 01:28 |
| SamYaple | k8s may have created it | 01:28 |
| SamYaple | whats the naem of it? | 01:28 |
| kfox1111 | k8s doent use them. | 01:28 |
| kfox1111 | no idea. it didn't show up in the list. | 01:28 |
| kfox1111 | :/ | 01:28 |
| SamYaple | ls /run/netns | 01:28 |
| kfox1111 | there were only the two items. | 01:28 |
| SamYaple | from the host | 01:28 |
| kfox1111 | I'm guessing thats the problem. there is an entry in the real netns list that isnt in /run/netns | 01:28 |
| SamYaple | thats not how it works | 01:29 |
| kfox1111 | there is a kernel netns, and | 01:29 |
| SamYaple | that folder is literally getting scanned by ip | 01:29 |
| kfox1111 | ip netns is kernel + mount namespaces in /run/netns. | 01:29 |
| SamYaple | yes, but tehre arent magically namespaces that dont exist in /run/netns | 01:29 |
| kfox1111 | whenever I've seen that message, it was because there was a kernel netns but not a corisponding mount in /run/netns. | 01:29 |
| SamYaple | you can create namespace outside of that, but ip netns wont know about them | 01:30 |
| kfox1111 | there can be. like when you don't specify the shared mount namespace. | 01:30 |
| kfox1111 | the /run/netns mount is in the container but not the hosts /run/netns. | 01:30 |
| kfox1111 | but it is in the kernel. | 01:30 |
| SamYaple | right, but ip netns wont know about them.... | 01:30 |
| SamYaple | so that error wouldnt be related | 01:30 |
| kfox1111 | it would because the kernel netns is shared between the host and containers when net=host. | 01:30 |
| SamYaple | im telling you ip netns literally walks that folder, and only that folder looking for namespaces | 01:31 |
| kfox1111 | and it looks at the kernel for ip netns list, then it tries to get more info out of /run/netns | 01:31 |
| SamYaple | it doesnt... | 01:31 |
| SamYaple | unless this changed in the last 2 years | 01:31 |
| kfox1111 | I'm positive it works the way I described. as, | 01:31 |
| SamYaple | i walked all this code when i originally figured out how to do the kolla shared namespace stuff | 01:31 |
| kfox1111 | if you don't do the shared mount ns, | 01:32 |
| kfox1111 | and then do ip netns on the host, | 01:32 |
| kfox1111 | it complains the same way. | 01:32 |
| kfox1111 | as it knows netns exist, but doens't know anything about them. | 01:32 |
| SamYaple | im goign to assume we are talking past each other | 01:32 |
| kfox1111 | possibly? | 01:33 |
| SamYaple | the ip netns command will walk /run/netns, what ever it finds there it will assume it is a namespace | 01:33 |
| SamYaple | thats the only way it finds namespaces | 01:33 |
| SamYaple | its not querying the kernel for anything | 01:33 |
| SamYaple | touch /run/netns/t ; ip netns | 01:33 |
| SamYaple | bam error | 01:33 |
| SamYaple | what *normally* happens is an old folder from an old mount that couldnt get deleted or something surrounding -rshared | 01:34 |
| SamYaple | so either the container has some bad stuff in /run/netns, or the host does | 01:34 |
| kfox1111 | strace -e stat,open ip netns list | 01:35 |
| kfox1111 | open("/proc/self/ns/net", O_RDONLY) = 4 | 01:35 |
| kfox1111 | openat(AT_FDCWD, "/var/run/netns", O_RDONLY|O_NONBLOCK|O_DIRECTORY|O_CLOEXEC) = 5 | 01:35 |
| kfox1111 | it opens the kernel netns first. then looks at the mount namespace. | 01:35 |
| *** cocakolla has quit IRC | 01:35 | |
| SamYaple | that is not an inclusive list of all the namespaces though | 01:35 |
| *** genek has quit IRC | 01:36 | |
| kfox1111 | but I think the error message that it looks like, I've seen when it was in the kernel list but not the moutn list. | 01:36 |
| kfox1111 | pretty sure. | 01:36 |
| kfox1111 | does any other container make netns's? | 01:37 |
| SamYaple | im telling you, thats not a list | 01:37 |
| SamYaple | you are misunderstanding whats happening | 01:37 |
| SamYaple | the only ones i know about that *make* namespaces are l3 and dhcp agent, maybe the linuxbridge/ovs for dvr | 01:38 |
| kfox1111 | ok... I think I can add a check to see.... | 01:38 |
| kfox1111 | if ls -l /run/netns !\ wip netns list | wc -l | 01:38 |
| SamYaple | im just saying, there is no kernel list of namespaces. thats not how they are implemetned | 01:39 |
| SamYaple | /proc/self is your own process anyway | 01:39 |
| SamYaple | my suggestion would be to purge the neutron-l3 and dhcp containers | 01:40 |
| SamYaple | umount anything binding /run/netns | 01:40 |
| SamYaple | and then purge whatever is left in /run/netns | 01:40 |
| SamYaple | (or reboot the box) | 01:40 |
| SamYaple | when you start it all back you likely wont have this issue anymore | 01:40 |
| SamYaple | its probably just cruft from your tests | 01:40 |
| kfox1111 | it is clean. | 01:41 |
| kfox1111 | it is in the gate. | 01:41 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 01:41 |
| kfox1111 | I logged the contents of /run/netns, and fail if num of lines in ip netns list != number of lines in ls /rn/netns | 01:42 |
| kfox1111 | should be interesting. | 01:42 |
| SamYaple | well youre on the right track. there is some stale file/folder/mount in /run/netns and thats making ip complain (not fail, just complain) | 01:43 |
| kfox1111 | I'd be surprised. we'll know in about 20 min. :) | 01:43 |
| SamYaple | fwiw, newer iproute utils make the error clearer | 01:44 |
| SamYaple | Error: Peer netns reference is invalid. | 01:44 |
| SamYaple | instead of RTNETLINK answers: Invalid argument | 01:44 |
| SamYaple | btw your test is going to fail | 01:45 |
| SamYaple | touch /run/netns/nsname ; ip netns ls | 01:45 |
| SamYaple | that will show two errors and then 'nsname' | 01:45 |
| SamYaple | even though it was never a valid namespace | 01:45 |
| SamYaple | but im off now, hope you figure it out! | 01:45 |
| kfox1111 | thanks for the help. :) | 01:46 |
| SamYaple | let me know what you find :) | 01:46 |
| SamYaple | hit me up if anything else wierd shows up | 01:46 |
| kfox1111 | will do. have a good evening. | 01:46 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Implement ceph fs https://review.openstack.org/508407 | 02:03 |
| *** portdirect has quit IRC | 02:04 | |
| *** sdake has quit IRC | 02:04 | |
| *** portdirect has joined #openstack-kolla | 02:05 | |
| *** peterjenkins__ has quit IRC | 02:05 | |
| *** peterjenkins__ has joined #openstack-kolla | 02:06 | |
| kfox1111 | SamYaple: http://logs.openstack.org/36/505436/9/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-iscsi-nv/771407e/console.html#_2017-10-06_02_05_35_539747 | 02:08 |
| *** calbers has quit IRC | 02:08 | |
| *** jistr has quit IRC | 02:08 | |
| kfox1111 | it shows the errors, but ip netns list shows 2 entries, and there are 2 entries in /run/netns. | 02:09 |
| *** SaMnCo_ has joined #openstack-kolla | 02:10 | |
| *** jistr has joined #openstack-kolla | 02:10 | |
| *** kong_ has joined #openstack-kolla | 02:11 | |
| kfox1111 | and http://logs.openstack.org/36/505436/9/check/gate-kolla-kubernetes-deploy-centos-binary-2-iscsi-nv/4e93915/console.html#_2017-10-06_02_08_12_114206 | 02:12 |
| kfox1111 | shows the error 4 times, | 02:12 |
| kfox1111 | but only 2 entries show up in the ip netns list and /run/netns | 02:12 |
| kfox1111 | so its gotta be in the kernel namespace. | 02:12 |
| *** calbers has joined #openstack-kolla | 02:12 | |
| *** kong has quit IRC | 02:12 | |
| *** SaMnCo has quit IRC | 02:12 | |
| *** logan- has quit IRC | 02:12 | |
| *** akwasnie has quit IRC | 02:12 | |
| *** markus-k has quit IRC | 02:12 | |
| *** kong_ is now known as kong | 02:12 | |
| *** SaMnCo_ is now known as SaMnCo | 02:12 | |
| kfox1111 | like a container did an ip netns without the shared mount. | 02:13 |
| *** markus-k has joined #openstack-kolla | 02:13 | |
| *** sdake has joined #openstack-kolla | 02:13 | |
| *** sdake is now known as Guest2849 | 02:13 | |
| *** unicell has quit IRC | 02:14 | |
| *** akwasnie has joined #openstack-kolla | 02:14 | |
| *** logan- has joined #openstack-kolla | 02:15 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 02:23 |
| SamYaple | "kernel namespace" isn't a thing though... the kernel doesnt maintain a list of namespaces, mount net pid or otherwise | 02:25 |
| SamYaple | so youre barking up the wrong tree | 02:25 |
| kfox1111 | there is a thing as a network namespace in the kernel. it is a pure namespace. | 02:31 |
| kfox1111 | then there is the thing that 'ip netns' calls a network namespace. | 02:31 |
| kfox1111 | it is part kernel namespace and part mount namespace convention. | 02:31 |
| kfox1111 | I added code to do an ip netns list in each container to see if I can spot it that way. | 02:32 |
| *** dave-mccowan has quit IRC | 02:33 | |
| kfox1111 | any netns in a container not showing on the host shoudl be the culprit I think. | 02:33 |
| *** mrunge has quit IRC | 02:38 | |
| *** afranc has quit IRC | 02:38 | |
| *** calbers has quit IRC | 02:39 | |
| *** calbers has joined #openstack-kolla | 02:41 | |
| *** mrunge has joined #openstack-kolla | 02:41 | |
| *** afranc has joined #openstack-kolla | 02:45 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 02:46 |
| kfox1111 | and, orgot the sudo. :/ | 02:46 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 03:11 |
| kfox1111 | yeah. looks like there is one container with an extra ip netns. | 03:11 |
| kfox1111 | this should find it.... | 03:11 |
| *** signed8bit is now known as signed8bit_Zzz | 03:12 | |
| *** seanmurphy has joined #openstack-kolla | 03:23 | |
| *** seanmurphy has quit IRC | 03:29 | |
| *** lxdong has quit IRC | 03:32 | |
| *** kolla-slack has quit IRC | 03:35 | |
| *** kolla-slack has joined #openstack-kolla | 03:35 | |
| *** goldyfruit has quit IRC | 03:35 | |
| spsurya | morning all | 03:51 |
| *** gkadam has joined #openstack-kolla | 04:01 | |
| *** dardelean_ has joined #openstack-kolla | 04:01 | |
| *** dardelean_ has quit IRC | 04:06 | |
| *** coolsvap has joined #openstack-kolla | 04:17 | |
| *** jaosorior has joined #openstack-kolla | 04:21 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Add notes on docker version in quickstart and multinode guide https://review.openstack.org/508790 | 04:36 |
| *** genek has joined #openstack-kolla | 04:38 | |
| *** TxGirlGeek has joined #openstack-kolla | 04:42 | |
| *** unicell has joined #openstack-kolla | 04:43 | |
| *** unicell1 has joined #openstack-kolla | 04:45 | |
| *** unicell has quit IRC | 04:47 | |
| *** skramaja has joined #openstack-kolla | 04:48 | |
| *** lpetrut_ has joined #openstack-kolla | 05:06 | |
| *** jbadiapa has joined #openstack-kolla | 05:32 | |
| *** TxGirlGeek has quit IRC | 05:41 | |
| *** lpetrut_ has quit IRC | 06:06 | |
| *** mdnadeem has joined #openstack-kolla | 06:09 | |
| *** janki has joined #openstack-kolla | 06:09 | |
| *** lrensing has joined #openstack-kolla | 06:15 | |
| *** clayton has quit IRC | 06:16 | |
| *** clayton has joined #openstack-kolla | 06:18 | |
| *** lrensing has quit IRC | 06:23 | |
| *** pcaruana has joined #openstack-kolla | 06:24 | |
| *** ArminderSingh has quit IRC | 06:30 | |
| *** ArminderSingh has joined #openstack-kolla | 06:37 | |
| *** ArminderSingh has quit IRC | 06:42 | |
| *** ArminderSingh has joined #openstack-kolla | 06:43 | |
| *** mandre is now known as mandre_afk | 06:51 | |
| *** seanmurphy has joined #openstack-kolla | 07:00 | |
| *** dardelean_ has joined #openstack-kolla | 07:10 | |
| *** gfidente has joined #openstack-kolla | 07:21 | |
| *** dardelean_ has quit IRC | 07:25 | |
| *** serlex has joined #openstack-kolla | 07:32 | |
| *** coolsvap has quit IRC | 07:43 | |
| *** brunograz has joined #openstack-kolla | 07:44 | |
| *** dardelean has joined #openstack-kolla | 07:45 | |
| brunograz | Hi all, we are running Openstack ocata 4.0.0 with kolla-ansible and it seems there is a bug which is not patched for this version https://bugs.launchpad.net/nova/+bug/1668267 | 07:45 |
| openstack | Launchpad bug 1670627 in OpenStack Compute (nova) ocata "duplicate for #1668267 quota is always in-use after delete the ERROR instances " [Critical,Fix released] - Assigned to Matt Riedemann (mriedem) | 07:45 |
| brunograz | what you guys would recommend as the best approach to patch it in our running system? | 07:46 |
| *** egonzalez has joined #openstack-kolla | 08:06 | |
| *** dardelean_ has joined #openstack-kolla | 08:11 | |
| *** mandre_afk is now known as mandre | 08:17 | |
| *** tovin07_ has quit IRC | 08:18 | |
| *** jascott1 has quit IRC | 08:19 | |
| *** jascott1 has joined #openstack-kolla | 08:19 | |
| egonzalez | brunograz, it was fixed for 4.0.1 iirc | 08:20 |
| brunograz | does that means I cannot patch it without a full upgrade? | 08:24 |
| *** jascott1 has quit IRC | 08:24 | |
| egonzalez | brunograz, binary images and from dockerhub or own build? | 08:24 |
| *** tovin07_ has joined #openstack-kolla | 08:27 | |
| brunograz | binary from dockhub | 08:27 |
| brunograz | dockerhub* | 08:27 |
| *** dasTor_ has quit IRC | 08:28 | |
| *** dasTor has joined #openstack-kolla | 08:28 | |
| egonzalez | its really a nova bug, the date the images were build, nova had not the fix merged yet | 08:30 |
| egonzalez | options are, upgrade to newer version, build your own 4.0.0 from binary now (packages have the fix) | 08:30 |
| brunograz | and can I build a single image for nova? in my understanding this only affects nova-api and conductor | 08:31 |
| brunograz | I checked the documentation for building images | 08:32 |
| brunograz | it only mentions building image for a project, e.g. nova - this will build all nova images | 08:33 |
| *** daidv has quit IRC | 08:38 | |
| *** jmccarthy has joined #openstack-kolla | 08:45 | |
| *** hrw has quit IRC | 08:58 | |
| *** hrw has joined #openstack-kolla | 08:59 | |
| brunograz | and once the image is built is there a way to tell kolla to update the running service? | 09:00 |
| *** hachi_ has joined #openstack-kolla | 09:14 | |
| *** sambetts_ is now known as sambetts | 09:16 | |
| *** genek has quit IRC | 09:22 | |
| *** genek has joined #openstack-kolla | 09:32 | |
| *** jamesbenson has joined #openstack-kolla | 09:53 | |
| *** jamesbenson has quit IRC | 09:57 | |
| *** duonghq has quit IRC | 09:59 | |
| *** pbourke has quit IRC | 10:01 | |
| *** pbourke has joined #openstack-kolla | 10:03 | |
| gema | do you guys know any dates for PTG? | 10:06 |
| gema | are there tentative dates? | 10:06 |
| *** tovin07_ has quit IRC | 10:06 | |
| pbourke | silva: ping | 10:07 |
| pomac | Do all of you have a 1450 mtu on your VM:s or how do you setup networking - I've noticed that if you have a network card with vxlan offload, there is no additional overhead but most of our nics doesn't have that kind of offloadf | 10:12 |
| *** jascott1 has joined #openstack-kolla | 10:23 | |
| *** genek has quit IRC | 10:23 | |
| *** hieulq has quit IRC | 10:47 | |
| *** vbel has quit IRC | 11:09 | |
| *** dave-mccowan has joined #openstack-kolla | 11:20 | |
| *** hachi_ has quit IRC | 11:43 | |
| *** noVNC has joined #openstack-kolla | 11:54 | |
| *** ansmith has quit IRC | 11:56 | |
| jmccarthy | Anyone hit something like this with magnum-conductor ? Not sure what is the deal | 12:01 |
| jmccarthy | ERROR oslo_messaging.rpc.server docker_volume_type = cluster.labels.get( | 12:01 |
| jmccarthy | ERROR oslo_messaging.rpc.server AttributeError: 'NoneType' object has no attribute 'get' | 12:01 |
| *** magicboiz has quit IRC | 12:09 | |
| *** noVNC has quit IRC | 12:12 | |
| masber | good afternoon all, is this mandatory if I want to use ceph for block storage? https://docs.openstack.org/project-deploy-guide/openstack-ansible/pike/targethosts-prepare.html#configure-storage | 12:24 |
| *** signed8bit_Zzz is now known as signed8bit | 12:30 | |
| pbourke | masber: that is openstack-ansible docs | 12:31 |
| pbourke | masber: not kolla | 12:31 |
| *** Croata has joined #openstack-kolla | 12:35 | |
| *** goldyfruit has joined #openstack-kolla | 12:40 | |
| *** erlon has joined #openstack-kolla | 12:41 | |
| gfidente | egonzalez any idea if somebody could help pushing a new image for ceph-mgr and update the repos to point to luminous? | 12:42 |
| *** ansmith has joined #openstack-kolla | 12:45 | |
| gfidente | egonzalez we discussed this with hrw a couple of days ago | 12:49 |
| *** hachi_ has joined #openstack-kolla | 12:52 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Add Barbican section in Nova and Cinder to use disk encryption https://review.openstack.org/509529 | 12:53 |
| *** athomas has quit IRC | 13:05 | |
| *** cocakolla has joined #openstack-kolla | 13:07 | |
| *** cocakolla has quit IRC | 13:08 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/509443 | 13:10 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla master: Remove a Pike release note generated in bad place https://review.openstack.org/508879 | 13:12 |
| *** gkadam has quit IRC | 13:18 | |
| *** skramaja has quit IRC | 13:18 | |
| silva | hello, when I try to initialize an image, I get this error: "There is not available location for image" (this is shown as a DEBUG log). I'm using ceph as storage engine in Pike. | 13:25 |
| *** aagate has joined #openstack-kolla | 13:27 | |
| *** signed8bit is now known as signed8bit_Zzz | 13:30 | |
| *** signed8bit_Zzz is now known as signed8bit | 13:30 | |
| *** Pavo has joined #openstack-kolla | 13:35 | |
| *** janki has quit IRC | 13:41 | |
| *** hachi_ has quit IRC | 13:43 | |
| *** shardy has quit IRC | 13:44 | |
| *** hachi_ has joined #openstack-kolla | 13:47 | |
| *** noVNC has joined #openstack-kolla | 13:50 | |
| noVNC | sup anyone ;) | 13:51 |
| *** hongbin has joined #openstack-kolla | 14:03 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Update admin-openrc for Gnocchi https://review.openstack.org/510130 | 14:12 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add OS_AUTH_URL to admin-openrc https://review.openstack.org/510130 | 14:15 |
| *** silva has quit IRC | 14:19 | |
| *** jtriley has quit IRC | 14:20 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 14:30 |
| lvdombrkr | folks, who use letsencrypt certeficates in kolla? | 14:33 |
| *** jiriprox has joined #openstack-kolla | 14:35 | |
| *** jamesbenson has joined #openstack-kolla | 14:37 | |
| *** goldyfruit has quit IRC | 14:43 | |
| *** jtriley has joined #openstack-kolla | 14:45 | |
| *** Radoslaw has joined #openstack-kolla | 14:45 | |
| *** goldyfruit has joined #openstack-kolla | 14:45 | |
| *** unicell1 has quit IRC | 14:47 | |
| *** unicell has joined #openstack-kolla | 14:47 | |
| *** ntpttr_laptop has joined #openstack-kolla | 14:50 | |
| Radoslaw | Hi everyone, I am trying to deploy Openstack with OVS-DPDK. It seem that the deployment fails because ovsdpdk_vswitchd container won't come up and the status is "Restarting (139) About a minute ago". I am deploying on single node and using stable/pike branch. Did anyone face this issue before? Any hint will be appreciated. | 14:51 |
| *** vhosakot has joined #openstack-kolla | 14:55 | |
| *** mdnadeem has quit IRC | 14:56 | |
| *** ntpttr_laptop has quit IRC | 14:58 | |
| *** lrensing has joined #openstack-kolla | 14:59 | |
| *** egonzalez has quit IRC | 15:03 | |
| masber | gfidente, do you know if it is possible to connect kolla to an external luminus ceph cluster? | 15:07 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Allow use of external ceph as cinder backup backend https://review.openstack.org/510153 | 15:07 |
| gfidente | masber don't know :( | 15:07 |
| Pavo | has anyone made a deployer ansible playbook yet to setup the deployer, I know there is a script to setup the target nodes | 15:13 |
| *** TxGirlGeek has joined #openstack-kolla | 15:18 | |
| *** silva has joined #openstack-kolla | 15:22 | |
| *** jaosorior has quit IRC | 15:24 | |
| *** TxGirlGeek has quit IRC | 15:26 | |
| *** brunograz has left #openstack-kolla | 15:26 | |
| jmccarthy | Anyone familiar with this ? These notes https://docs.openstack.org/releasenotes/magnum/pike.html say "A new section is created in magnum.conf named cinder. In this cinder section, you need to set a value for the key default_docker_volume_type" - but I don't see this ? | 15:28 |
| *** Jose____ has joined #openstack-kolla | 15:30 | |
| Jose____ | Hi! im having an issue creating images...it stucks in saving status and looking the glance-api.log no error appear..what can be happening? The deployment was made with kolla-ansible | 15:30 |
| *** TxGirlGeek has joined #openstack-kolla | 15:31 | |
| openstackgerrit | Radomir Dopieralski proposed openstack/kolla master: Don't delete plugin's configuration files in horizon https://review.openstack.org/510160 | 15:33 |
| kfox1111 | kolla-kubernetes cores: https://review.openstack.org/#/c/507252/ please. | 15:36 |
| *** TxGirlGeek has quit IRC | 15:39 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 15:39 |
| *** zhubingbing_ has joined #openstack-kolla | 15:42 | |
| *** jmccarthy has left #openstack-kolla | 15:43 | |
| *** lrensing has quit IRC | 15:49 | |
| *** lrensing has joined #openstack-kolla | 15:53 | |
| *** zhubingbing_ has quit IRC | 15:57 | |
| *** jascott1 has quit IRC | 16:02 | |
| *** jascott1 has joined #openstack-kolla | 16:02 | |
| *** jascott1 has quit IRC | 16:02 | |
| *** seanmurphy has quit IRC | 16:07 | |
| inc0 | hello | 16:12 |
| *** pcaruana has quit IRC | 16:13 | |
| *** noVNC has quit IRC | 16:13 | |
| *** jamesbenson has quit IRC | 16:18 | |
| kolla-slack | <britthouser> morning @inc0 | 16:19 |
| openstackgerrit | Doug Szumski proposed openstack/kolla-ansible master: Add nova-compute-ironic to fluentd rewrite rules https://review.openstack.org/510177 | 16:27 |
| *** dougsz has joined #openstack-kolla | 16:32 | |
| *** signed8bit is now known as signed8bit_Zzz | 16:40 | |
| *** unicell has quit IRC | 16:42 | |
| *** dougsz has quit IRC | 16:47 | |
| *** signed8bit_Zzz is now known as signed8bit | 16:50 | |
| *** signed8bit is now known as signed8bit_Zzz | 16:51 | |
| *** signed8bit_Zzz is now known as signed8bit | 16:51 | |
| *** jamesbenson has joined #openstack-kolla | 16:52 | |
| vhosakot | afternoon all :) | 16:53 |
| *** openstackstatus has quit IRC | 17:05 | |
| *** openstack has joined #openstack-kolla | 17:08 | |
| *** ChanServ sets mode: +o openstack | 17:08 | |
| *** dardelean_ has quit IRC | 17:11 | |
| *** lrensing has quit IRC | 17:13 | |
| *** zhubingbing_ has joined #openstack-kolla | 17:17 | |
| *** lrensing has joined #openstack-kolla | 17:17 | |
| *** ntpttr_laptop has joined #openstack-kolla | 17:17 | |
| *** Pavo has quit IRC | 17:20 | |
| *** sambetts is now known as sambetts|afk | 17:20 | |
| *** Radoslaw has quit IRC | 17:20 | |
| *** dougsz has joined #openstack-kolla | 17:25 | |
| *** lrensing has quit IRC | 17:28 | |
| *** lrensing has joined #openstack-kolla | 17:30 | |
| *** dougsz has quit IRC | 17:30 | |
| *** zhubingbing_ has quit IRC | 17:32 | |
| *** zhubingbing has joined #openstack-kolla | 17:33 | |
| *** ntpttr_laptop has quit IRC | 17:45 | |
| *** ntpttr_laptop has joined #openstack-kolla | 17:45 | |
| *** dardelean_ has joined #openstack-kolla | 17:45 | |
| *** zhubingbing has quit IRC | 17:47 | |
| *** dardelean_ has quit IRC | 17:50 | |
| *** gfidente is now known as gfidente|afk | 17:50 | |
| *** dougsz has joined #openstack-kolla | 17:57 | |
| *** dougsz has quit IRC | 17:57 | |
| *** noVNC has joined #openstack-kolla | 18:00 | |
| *** noVNC has quit IRC | 18:03 | |
| kfox1111 | hello | 18:03 |
| *** goldyfruit has quit IRC | 18:04 | |
| *** goldyfruit has joined #openstack-kolla | 18:04 | |
| kfox1111 | inc0: https://review.openstack.org/#/c/507252/ plz | 18:05 |
| *** jascott1 has joined #openstack-kolla | 18:10 | |
| *** serlex has quit IRC | 18:13 | |
| *** seanmurphy has joined #openstack-kolla | 18:14 | |
| *** lrensing has quit IRC | 18:15 | |
| *** jtriley has quit IRC | 18:15 | |
| *** jtriley has joined #openstack-kolla | 18:15 | |
| *** jtriley has quit IRC | 18:15 | |
| *** jtriley has joined #openstack-kolla | 18:16 | |
| *** gfidente|afk has quit IRC | 18:16 | |
| *** serlex has joined #openstack-kolla | 18:20 | |
| *** serlex has left #openstack-kolla | 18:20 | |
| kfox1111 | SamYaple: I think I'm seeing the same thing as: https://bugs.launchpad.net/kolla/+bug/1616268 | 18:21 |
| openstack | Launchpad bug 1616268 in kolla newton "Stale namespace removal causing "RTNETLINK answers: Invalid argument" errors" [Critical,Fix committed] - Assigned to Jeffrey Zhang (jeffrey4l) | 18:21 |
| kfox1111 | http://logs.openstack.org/36/505436/14/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/ace418b/console.html#_2017-10-06_16_11_08_764388 shows a 000 permed file. | 18:21 |
| SamYaple | kfox1111: yep. and look who reported the neutron bug that links too. youre walking a path ive already walked | 18:23 |
| kfox1111 | the workaround won't work for me though. :/ | 18:24 |
| kfox1111 | k8s doesn't support docker volumes. | 18:24 |
| SamYaple | give me a second to look over your patch | 18:25 |
| kfox1111 | k. | 18:25 |
| SamYaple | kfox1111: youre binding in /var/run and /run/netns, dont think thats going to work | 18:27 |
| SamYaple | try as a test to do /run/:/run/:shared | 18:27 |
| kfox1111 | was trying to do just /run and /run/netns... | 18:27 |
| kfox1111 | sould I just do /run? | 18:27 |
| SamYaple | as a test, yes | 18:27 |
| kfox1111 | k. | 18:27 |
| SamYaple | i seem to recall having trouble doing only /run/netns | 18:28 |
| kfox1111 | weird. | 18:28 |
| SamYaple | i think it was because at the time i was binding in /run, and then trying to do a shared /run/netns on a non-shared /run | 18:28 |
| SamYaple | like youre doing | 18:28 |
| SamYaple | so give it a shot and see what happens | 18:28 |
| kfox1111 | k. | 18:28 |
| kfox1111 | hmm.... inconsistency here.. | 18:31 |
| kfox1111 | some are mounting /var/run from the host. | 18:31 |
| kfox1111 | probably should always be /run from the host? | 18:31 |
| kfox1111 | and in the container.... | 18:32 |
| kfox1111 | cause I think both are symlinks to /run ? | 18:32 |
| SamYaple | yea i always just mount /run | 18:35 |
| SamYaple | but /var/run should work the same | 18:35 |
| SamYaple | feel free to play with it, my best guess is thats where your problem lies with bad /run/ mounts | 18:35 |
| kfox1111 | well, may cause inconsistencies if it overwrites the symlink? as maybe some commands read /run and some /var/run? | 18:35 |
| SamYaple | does it though? wont it just mount in a symlink? | 18:36 |
| SamYaple | oh but thats even worse then | 18:36 |
| kfox1111 | wouldn't have the :shared mount behavior then. | 18:36 |
| kfox1111 | yeah. | 18:36 |
| kfox1111 | I'm going to change all the references to /var/run -> /run | 18:36 |
| SamYaple | yea either way, go with /run over /var/run. i changes that in kolla early on | 18:36 |
| SamYaple | yup | 18:36 |
| *** cocakolla has joined #openstack-kolla | 18:38 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 18:42 |
| kfox1111 | ok. lets try the lower hanging fruit first. | 18:43 |
| kfox1111 | I marked /run shared too. | 18:43 |
| kfox1111 | if that doesn't fix it, I'll remove the /run/netns mount. | 18:43 |
| SamYaple | seems like youre on the right path, yea | 18:46 |
| kfox1111 | k. thanks. | 18:46 |
| kfox1111 | really strange bug though... | 18:46 |
| *** jiriprox has quit IRC | 18:48 | |
| *** jiriprox has joined #openstack-kolla | 18:48 | |
| SamYaple | its just a bad error message from iproute2 | 18:48 |
| SamYaple | they do make it clearer in newr versions | 18:49 |
| SamYaple | new error message is "Error: Peer netns reference is invalid." | 18:49 |
| jascott1 | ah glad to see two smart k8s peeps in here | 18:50 |
| jascott1 | i dont suppose anyone has tried to use kubectl proxy for local dev? | 18:51 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/509443 | 18:52 |
| *** cocakolla has quit IRC | 19:08 | |
| *** lrensing has joined #openstack-kolla | 19:14 | |
| *** Slower has quit IRC | 19:49 | |
| kfox1111 | jascott1: local dev of what? | 20:01 |
| kolla-slack | <jascott1> container app that is cluster aware | 20:02 |
| kolla-slack | <jascott1> (makes calls to k8s api) | 20:03 |
| jascott1 | its the operator im working on | 20:03 |
| kfox1111 | hmm.... | 20:03 |
| jascott1 | problem is, it wants to https and the kubectl proxy is http | 20:03 |
| kfox1111 | are you trying not to put the operator in k8s? | 20:03 |
| jascott1 | yes | 20:03 |
| kfox1111 | ah. | 20:03 |
| jascott1 | basically i am trying to debug it and run it local and have it run in the cluster via proxy | 20:04 |
| jascott1 | rather than edit src files on remote vm | 20:04 |
| kfox1111 | lets see.... you could maybe give the operator your admin.conf? (if you don't care about your cluster) | 20:04 |
| jascott1 | hmm | 20:05 |
| kfox1111 | I think the go kube client supports accepting a kubeconf file. | 20:05 |
| jascott1 | what about getting the servers certs and running a MITM in front of kubectl proxy | 20:05 |
| kfox1111 | possible but seems like more work then just giving it the same creds your using for kubectl. :) | 20:06 |
| *** ansmith has quit IRC | 20:06 | |
| jascott1 | trying not to make debug modifications to the code tho | 20:07 |
| kfox1111 | https://github.com/kubernetes-incubator/external-storage/tree/master/ceph/cephfs | 20:07 |
| kfox1111 | they do the same thing.... | 20:07 |
| kfox1111 | they add a -kubeconfig=~/.kube/config option. | 20:08 |
| kfox1111 | I've seen it in other k8s addons. | 20:08 |
| kfox1111 | I think it is the common way of doing it. | 20:08 |
| kfox1111 | some folks do like to run their operators outside of k8s for some reason. | 20:08 |
| kfox1111 | debugging is just one of them. | 20:08 |
| jascott1 | ah good point | 20:09 |
| jascott1 | thanks | 20:09 |
| kfox1111 | np. | 20:09 |
| jascott1 | must achieve more caffeination and sustenance... brb | 20:10 |
| kfox1111 | hehe. I was just thinking the same thing. :) | 20:11 |
| *** Croata2 has joined #openstack-kolla | 20:21 | |
| *** Croata has quit IRC | 20:25 | |
| *** seanmurphy has quit IRC | 20:31 | |
| *** david-lyle has quit IRC | 20:38 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 20:50 |
| *** genek has joined #openstack-kolla | 21:00 | |
| *** Jose____ has quit IRC | 21:01 | |
| *** mrhillsman has joined #openstack-kolla | 21:09 | |
| mrhillsman | i'm trying to run precheck and getting error that kolla_external_vip_interface is not found in ansible_interface but it is set in the globals.yml file | 21:09 |
| mrhillsman | it was not previously so i am wondering if there is cache, which it does not seem to be, and if yes, where | 21:10 |
| *** jtriley has quit IRC | 21:12 | |
| SamYaple | there shouldnt be a cache, no mrhillsman | 21:12 |
| *** Croata2 has quit IRC | 21:16 | |
| mrhillsman | thx SamYaple | 21:18 |
| *** david-lyle has joined #openstack-kolla | 21:19 | |
| *** hachi_ has quit IRC | 21:21 | |
| *** dklyle has joined #openstack-kolla | 21:39 | |
| *** silva has quit IRC | 21:39 | |
| *** david-lyle has quit IRC | 21:41 | |
| *** dklyle has quit IRC | 21:41 | |
| *** dklyle has joined #openstack-kolla | 21:42 | |
| *** bmace has quit IRC | 21:43 | |
| *** bmace has joined #openstack-kolla | 21:44 | |
| *** dklyle has quit IRC | 21:52 | |
| *** jamesbenson has quit IRC | 21:55 | |
| *** jamesbenson has joined #openstack-kolla | 21:56 | |
| *** jamesbenson has quit IRC | 21:58 | |
| *** signed8bit is now known as signed8bit_Zzz | 21:58 | |
| *** ntpttr_laptop has quit IRC | 22:02 | |
| *** lrensing has quit IRC | 22:09 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: shared netns support. https://review.openstack.org/505436 | 22:15 |
| *** vhosakot has quit IRC | 22:16 | |
| *** goldyfruit has quit IRC | 22:22 | |
| *** erlon has quit IRC | 22:30 | |
| *** ntpttr_laptop has joined #openstack-kolla | 22:50 | |
| kfox1111 | SamYaple: seems like we have a winner: https://review.openstack.org/#/c/505436/ | 22:54 |
| kfox1111 | using /run everywhere, switching it to shared, and then doing /run/netns shared fixed it. | 22:55 |
| SamYaple | kfox1111: if you are already mounting /run shared, you dont need to mount /run/netns | 22:56 |
| SamYaple | but awesome news! | 22:56 |
| kfox1111 | true. but it still feels like a bug that needs squashing and this is just a temporary workaround. :/ | 22:57 |
| kfox1111 | thanks for helping me debug it. | 22:59 |
| SamYaple | im not sure, i think this is the solution of the bug | 23:00 |
| kfox1111 | it really shouldn't be required to make the parent of the netns shared though. | 23:01 |
| kfox1111 | that feels really weird. | 23:01 |
| SamYaple | i would readress why you are mounting /run in the first place | 23:02 |
| kfox1111 | and making all of /run shared seems like more of a security thing then should strictly be nessisary. | 23:02 |
| SamYaple | i think you can get away with *only* /run/netns | 23:02 |
| kfox1111 | in some cases, maybe. | 23:02 |
| SamYaple | or /run/netns and other explict paths if found | 23:02 |
| kfox1111 | but ovs needs to look at stuff in /run. | 23:02 |
| SamYaple | yea but you can explictly bind that stuff rather than everything | 23:02 |
| SamYaple | and you can configure where ovs looks | 23:02 |
| kfox1111 | yeah. theres a tradeoff there too. | 23:03 |
| kfox1111 | I added external ovs support to kolla-kubernetes. | 23:03 |
| kfox1111 | so Ican support my single nic 40g attached boxen. | 23:03 |
| kfox1111 | (need networking to work for k8s, before ovs could be started in a container) | 23:03 |
| kfox1111 | so it shoudl support whatever redhat/ubunto ovs package puts it. | 23:04 |
| *** jamesbenson has joined #openstack-kolla | 23:14 | |
| *** noVNC has joined #openstack-kolla | 23:16 | |
| *** jamesbenson has quit IRC | 23:18 | |
| *** hongbin has quit IRC | 23:21 | |
| *** cocakolla has joined #openstack-kolla | 23:22 | |
| *** jgriffith is now known as jgriffith_ | 23:24 | |
| noVNC | anyone know how to debug horizon gui not working on kolla? | 23:29 |
| noVNC | I got 504 Gateway Time-out when trying to access | 23:29 |
| *** dardelean_ has joined #openstack-kolla | 23:30 | |
| *** dardelean has quit IRC | 23:32 | |
| *** david-lyle has joined #openstack-kolla | 23:42 | |
| *** harlowja has quit IRC | 23:45 | |
| *** ntpttr_laptop has quit IRC | 23:50 | |
| *** cocakolla has quit IRC | 23:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!