| kfox1111 | and.. audit's broken in newton as well... | 00:00 |
|---|---|---|
| kfox1111 | :/ | 00:00 |
| *** myoung has joined #openstack-kolla | 00:01 | |
| sbezverk | kfox1111: vm | 00:02 |
| sbezverk | 5 vms | 00:02 |
| sbezverk | what is interesting management connectivity is working fine | 00:03 |
| kfox1111 | thtas pretty weird. | 00:03 |
| kfox1111 | kubeproxy only tweaks iptables as far as I can tell. | 00:03 |
| kfox1111 | using canal for sdn in the vm's? | 00:03 |
| sbezverk | kfox1111: that is what I think breaks things | 00:04 |
| *** yangyapeng has quit IRC | 00:04 | |
| sbezverk | it is probably allows management traffic as is but other interfaces blocks all unknonw to it | 00:04 |
| kfox1111 | it shuld only be enableing vxlan tunnels. not sure why an arp would fail after that. | 00:04 |
| kfox1111 | are you using keepalived too? | 00:04 |
| sbezverk | only on management side | 00:04 |
| *** goldyfruit has quit IRC | 00:05 | |
| kfox1111 | iptables/selinux/filrewalld disabled everywhere? | 00:05 |
| sbezverk | selinux and firewalld yes | 00:06 |
| sbezverk | I did not explicitely disabled iptables | 00:06 |
| *** rwallner has joined #openstack-kolla | 00:06 | |
| kfox1111 | should only need to if you installed package iptables-service | 00:07 |
| sbezverk | it does not seem there is iptables service | 00:07 |
| kfox1111 | k. | 00:08 |
| *** rwallner has quit IRC | 00:11 | |
| kfox1111 | and, a 3rd db migrate issue. :/ | 00:15 |
| kfox1111 | https://review.openstack.org/#/c/408725/2 | 00:15 |
| kfox1111 | seriously.... its like no one tests this stuff with real systems. :/ | 00:15 |
| jascott1 | overheard last night-> stranger: "At Oracle we used to go out for shots at lunch" me: "Hey that explains a lot about my database!" | 00:18 |
| kfox1111 | hehe | 00:18 |
| sdake | jascott1 OH "When we merged patches into nova, we brought donuts into the office." :) | 00:18 |
| jascott1 | haha yeah i have heard that one | 00:18 |
| sdake | i was pretty stunned tbh :) | 00:19 |
| kfox1111 | I think 9 times out of 10 the upgrade issues I've hit were in nova or neutron. | 00:19 |
| sdake | kfox1111 you haven't even begun to feel the pain when going to ocata :) | 00:19 |
| kfox1111 | sdake: I've had clouds I've upgraded constantly from havana, including a centos 6->7 upgrade. | 00:20 |
| kfox1111 | felt the pain. :) | 00:20 |
| sdake | kfox1111 just wait until ocata ;) | 00:20 |
| kfox1111 | heh. | 00:20 |
| kfox1111 | we'll see if openstack makes it that long... | 00:21 |
| sdake | openstack here to stay | 00:21 |
| sdake | hate to break it to you naysayers :) | 00:21 |
| kfox1111 | thats not exactly what I said. | 00:21 |
| sdake | perhaps i misinterpeted | 00:21 |
| *** rwellum has quit IRC | 00:21 | |
| kfox1111 | the openstacks in question stick around long enough to put the effort of upgrading to ocata. | 00:22 |
| *** salv-orlando has quit IRC | 00:22 | |
| kfox1111 | rather then just getting rid of. time will tell. :/ | 00:22 |
| kfox1111 | it may be easier to shift workload to k8s and drop openstack for some clouds. | 00:23 |
| *** gianpiet_ has quit IRC | 00:24 | |
| kfox1111 | and.. anoter bug... | 00:24 |
| kfox1111 | wow... mitaka -> newtons proving to be really ugly. | 00:25 |
| *** chas has joined #openstack-kolla | 00:26 | |
| *** gianpietro has joined #openstack-kolla | 00:27 | |
| *** chas has quit IRC | 00:31 | |
| *** gianpietro has quit IRC | 00:31 | |
| *** duonghq has joined #openstack-kolla | 00:34 | |
| duonghq | morning guys | 00:34 |
| *** MasterOfBugs has quit IRC | 00:37 | |
| *** eaguilar has joined #openstack-kolla | 00:43 | |
| *** lamt has joined #openstack-kolla | 00:44 | |
| *** schwicht has quit IRC | 00:45 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Docker Logging Support https://review.openstack.org/458649 | 00:48 |
| *** unicell has quit IRC | 00:48 | |
| *** schwicht has joined #openstack-kolla | 00:49 | |
| *** schwicht has quit IRC | 00:50 | |
| *** zhurong has joined #openstack-kolla | 00:51 | |
| *** cuongnv has joined #openstack-kolla | 00:53 | |
| *** goldyfruit has joined #openstack-kolla | 00:54 | |
| *** chas has joined #openstack-kolla | 01:02 | |
| *** yee37916 has quit IRC | 01:04 | |
| *** yangyapeng has joined #openstack-kolla | 01:06 | |
| *** yee379 has joined #openstack-kolla | 01:07 | |
| *** chas has quit IRC | 01:07 | |
| *** lamt has quit IRC | 01:16 | |
| *** lamt has joined #openstack-kolla | 01:17 | |
| *** newmember has quit IRC | 01:17 | |
| *** salv-orlando has joined #openstack-kolla | 01:19 | |
| spsurya | morning all | 01:20 |
| spsurya | duonghq: morning | 01:20 |
| *** lamt has left #openstack-kolla | 01:23 | |
| *** yangyapeng has quit IRC | 01:25 | |
| *** yangyapeng has joined #openstack-kolla | 01:26 | |
| duonghq | morning spsurya | 01:26 |
| *** tervancovan-mobi has quit IRC | 01:27 | |
| inc0 | yay fighting with centos bootloader:( | 01:30 |
| *** hrw has quit IRC | 01:31 | |
| *** hrw has joined #openstack-kolla | 01:32 | |
| *** Pavo has joined #openstack-kolla | 01:38 | |
| *** chas has joined #openstack-kolla | 01:39 | |
| *** Pavo has quit IRC | 01:39 | |
| *** chas has quit IRC | 01:43 | |
| *** salv-orlando has quit IRC | 01:48 | |
| *** sayantan_ has quit IRC | 01:48 | |
| *** japestinho has joined #openstack-kolla | 01:58 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add copy ceph keyring files conditions in cinder https://review.openstack.org/458366 | 01:59 |
| *** ssurana has quit IRC | 02:01 | |
| *** ssurana has joined #openstack-kolla | 02:01 | |
| *** ssurana has quit IRC | 02:01 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add the ceph keyring files related task condition https://review.openstack.org/458417 | 02:06 |
| *** newmember has joined #openstack-kolla | 02:07 | |
| *** eaguilar has quit IRC | 02:12 | |
| *** eanylin has quit IRC | 02:14 | |
| *** unicell has joined #openstack-kolla | 02:34 | |
| *** unicell has quit IRC | 02:34 | |
| *** yangyapeng has quit IRC | 02:45 | |
| *** salv-orlando has joined #openstack-kolla | 02:45 | |
| *** shasha___ has joined #openstack-kolla | 02:45 | |
| *** yangyapeng has joined #openstack-kolla | 02:46 | |
| *** eanylin has joined #openstack-kolla | 02:46 | |
| *** chas has joined #openstack-kolla | 02:50 | |
| *** chas has quit IRC | 02:56 | |
| *** salv-orl_ has joined #openstack-kolla | 03:03 | |
| *** magicboiz has quit IRC | 03:03 | |
| *** salv-orlando has quit IRC | 03:03 | |
| *** shasha___ has quit IRC | 03:03 | |
| *** shasha___ has joined #openstack-kolla | 03:04 | |
| *** magicboiz has joined #openstack-kolla | 03:04 | |
| *** david-lyle has joined #openstack-kolla | 03:11 | |
| *** zhugaoxiao has quit IRC | 03:12 | |
| *** zhugaoxiao has joined #openstack-kolla | 03:12 | |
| *** portdirect has quit IRC | 03:16 | |
| *** salv-orl_ has quit IRC | 03:16 | |
| *** gianpietro has joined #openstack-kolla | 03:18 | |
| *** david-lyle has quit IRC | 03:19 | |
| *** portdirect has joined #openstack-kolla | 03:23 | |
| *** goldyfruit has quit IRC | 03:24 | |
| *** gianpietro has quit IRC | 03:38 | |
| *** gkadam has joined #openstack-kolla | 03:42 | |
| *** zhurong has quit IRC | 03:43 | |
| *** haplo37 has quit IRC | 03:54 | |
| *** dave-mcc_ has quit IRC | 04:02 | |
| *** haplo37 has joined #openstack-kolla | 04:04 | |
| *** klindgren_ has joined #openstack-kolla | 04:11 | |
| *** salv-orlando has joined #openstack-kolla | 04:13 | |
| *** klindgren has quit IRC | 04:14 | |
| *** MasterOfBugs has joined #openstack-kolla | 04:33 | |
| *** kristian__ has joined #openstack-kolla | 04:34 | |
| *** kristian__ has quit IRC | 04:38 | |
| *** salv-orlando has quit IRC | 04:41 | |
| *** salv-orlando has joined #openstack-kolla | 04:46 | |
| *** zhubingbing_ has joined #openstack-kolla | 04:53 | |
| *** unicell has joined #openstack-kolla | 04:59 | |
| *** kristian__ has joined #openstack-kolla | 05:00 | |
| *** kristian__ has quit IRC | 05:00 | |
| *** kristian__ has joined #openstack-kolla | 05:00 | |
| *** kristian__ has quit IRC | 05:05 | |
| *** skramaja has joined #openstack-kolla | 05:06 | |
| *** unicell1 has joined #openstack-kolla | 05:09 | |
| *** unicell has quit IRC | 05:11 | |
| *** kristian__ has joined #openstack-kolla | 05:17 | |
| *** kristian__ has quit IRC | 05:22 | |
| *** lpetrut has joined #openstack-kolla | 05:47 | |
| *** zhuzeyu has joined #openstack-kolla | 05:51 | |
| *** kristian__ has joined #openstack-kolla | 05:52 | |
| *** magicboiz has quit IRC | 05:56 | |
| *** kristian__ has quit IRC | 05:57 | |
| *** jbadiapa has joined #openstack-kolla | 05:59 | |
| *** matrohon_ has joined #openstack-kolla | 06:04 | |
| *** jaosorior has joined #openstack-kolla | 06:06 | |
| portdirect | So this is interesting: https://github.com/docker/docker | 06:09 |
| *** salv-orlando has quit IRC | 06:11 | |
| *** salv-orlando has joined #openstack-kolla | 06:11 | |
| portdirect | Now redirects to: | 06:11 |
| portdirect | https://github.com/moby/moby/blob/master/README.md#transitioning-to-moby | 06:12 |
| *** tonanhngo has joined #openstack-kolla | 06:12 | |
| *** kristian__ has joined #openstack-kolla | 06:13 | |
| *** salv-orlando has quit IRC | 06:16 | |
| *** kristian__ has quit IRC | 06:17 | |
| *** pcaruana has joined #openstack-kolla | 06:18 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-kubernetes master: Heat service deployment https://review.openstack.org/437252 | 06:19 |
| *** zhurong has joined #openstack-kolla | 06:21 | |
| *** magicboiz has joined #openstack-kolla | 06:21 | |
| *** zhuzeyu has quit IRC | 06:25 | |
| *** zhuzeyu has joined #openstack-kolla | 06:26 | |
| duonghq | portdirect, yeah | 06:28 |
| duonghq | long time no see | 06:28 |
| duonghq | portdirect, I see that you have presentation at Boston, will you go to Boston? | 06:29 |
| *** lpetrut has quit IRC | 06:37 | |
| *** imcsk8 has joined #openstack-kolla | 06:38 | |
| *** zhurong has quit IRC | 06:39 | |
| *** satyar has joined #openstack-kolla | 06:39 | |
| *** shasha___ has quit IRC | 06:40 | |
| *** zhurong has joined #openstack-kolla | 06:41 | |
| *** kristian__ has joined #openstack-kolla | 06:43 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-kubernetes master: Heat service deployment https://review.openstack.org/437252 | 06:47 |
| *** kristian__ has quit IRC | 06:47 | |
| *** zhurong has quit IRC | 06:54 | |
| *** tonanhngo has quit IRC | 06:58 | |
| *** salv-orlando has joined #openstack-kolla | 07:01 | |
| *** tonanhngo has joined #openstack-kolla | 07:02 | |
| *** jascott1 has quit IRC | 07:02 | |
| *** tonanhngo has quit IRC | 07:04 | |
| *** chas has joined #openstack-kolla | 07:08 | |
| *** magicboiz has quit IRC | 07:08 | |
| *** MasterOfBugs has quit IRC | 07:09 | |
| *** rmart04 has joined #openstack-kolla | 07:18 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-kubernetes master: Heat service deployment https://review.openstack.org/437252 | 07:18 |
| *** magicboiz has joined #openstack-kolla | 07:20 | |
| *** shardy has joined #openstack-kolla | 07:24 | |
| *** bjolo has joined #openstack-kolla | 07:27 | |
| *** kristian__ has joined #openstack-kolla | 07:27 | |
| *** zhurong has joined #openstack-kolla | 07:31 | |
| *** kristian__ has quit IRC | 07:32 | |
| *** manheim has joined #openstack-kolla | 07:36 | |
| *** egonzalez has joined #openstack-kolla | 07:39 | |
| duonghq | anybody got NoNetworkAvailable error when create a network? (in Newton) | 07:41 |
| spsurya | duonghq: not seen | 07:41 |
| duonghq | spsurya, okay | 07:43 |
| egonzalez | duonghq, when launching an instance? | 07:43 |
| duonghq | egonzalez, when create a network | 07:44 |
| duonghq | with neutron | 07:44 |
| egonzalez | duonghq, external network or tenant? | 07:44 |
| duonghq | egonzalez, the command is openstack network create --project admin for_subnet | 07:45 |
| *** kristian__ has joined #openstack-kolla | 07:48 | |
| egonzalez | duonghq, have you made any custom config change or something non default in network parameters? | 07:49 |
| duonghq | egonzalez, you mean service's config or kolla config? | 07:51 |
| *** kristian__ has quit IRC | 07:52 | |
| egonzalez | duonghq, both, custom_configs, globals/groupvars | 07:52 |
| duonghq | egonzalez, I have a meeting in a few mins, sorry, brb | 07:54 |
| egonzalez | duonghq, np | 07:54 |
| duonghq | egonzalez, thanks | 07:54 |
| egonzalez | guys, i'm looking for a cloud provider to use for my dev environment. Anyone have any experience with any provider? Actually I don't know if use baremetal or on-premise instances | 08:06 |
| *** zhurong has quit IRC | 08:15 | |
| *** mgoddard has joined #openstack-kolla | 08:18 | |
| *** zhurong has joined #openstack-kolla | 08:22 | |
| *** eanylin has quit IRC | 08:23 | |
| *** kristian__ has joined #openstack-kolla | 08:24 | |
| *** zhuzeyu has quit IRC | 08:29 | |
| *** zhuzeyu has joined #openstack-kolla | 08:30 | |
| *** shardy has quit IRC | 08:34 | |
| *** shardy has joined #openstack-kolla | 08:35 | |
| *** shasha___ has joined #openstack-kolla | 08:41 | |
| *** kristian__ has quit IRC | 08:42 | |
| *** kristian__ has joined #openstack-kolla | 08:43 | |
| *** kristia__ has joined #openstack-kolla | 08:45 | |
| *** shasha___ has quit IRC | 08:45 | |
| *** kristia__ has quit IRC | 08:45 | |
| *** kristia__ has joined #openstack-kolla | 08:46 | |
| *** kristia__ has quit IRC | 08:46 | |
| *** kristian__ has quit IRC | 08:46 | |
| *** kristian__ has joined #openstack-kolla | 08:46 | |
| *** kristian__ has quit IRC | 08:46 | |
| *** kristian__ has joined #openstack-kolla | 08:46 | |
| *** kristian__ has quit IRC | 08:47 | |
| duonghq | egonzalez, are you still there? | 08:47 |
| egonzalez | yep | 08:47 |
| duonghq | I'm supporting my friend about the error | 08:47 |
| duonghq | if I understand correctly, the config isn't change too much | 08:48 |
| openstackgerrit | Merged openstack/kolla-ansible master: Optimize reconfigure action for vmtp https://review.openstack.org/424194 | 08:48 |
| *** eanylin has joined #openstack-kolla | 08:50 | |
| duonghq | zhubingbing_, are you there | 08:52 |
| zhubingbing_ | hi | 08:53 |
| egonzalez | duonghq, what tenant_network_types have at ml2 config? | 08:53 |
| zhubingbing_ | i am hear | 08:53 |
| egonzalez | and if each network type is configured at its section | 08:53 |
| egonzalez | duonghq, guess is not a kolla deployment or it is? | 08:54 |
| duonghq | egonzalez, it's kolla deployment | 08:54 |
| *** jrist has quit IRC | 08:57 | |
| *** Serlex has joined #openstack-kolla | 09:07 | |
| *** Guest41303 has joined #openstack-kolla | 09:08 | |
| *** sambetts|afk is now known as sambetts | 09:08 | |
| *** Guest41303 has quit IRC | 09:20 | |
| *** zhurong has quit IRC | 09:33 | |
| *** zhurong has joined #openstack-kolla | 09:36 | |
| *** zhuzeyu has quit IRC | 09:37 | |
| openstackgerrit | Shunli Zhou proposed openstack/kolla-ansible master: Remove symlink from fluentd global conf https://review.openstack.org/458791 | 09:37 |
| *** zhurong has quit IRC | 09:38 | |
| *** Administrator_ has joined #openstack-kolla | 09:44 | |
| *** salv-orl_ has joined #openstack-kolla | 09:48 | |
| *** dixiaoli has joined #openstack-kolla | 09:50 | |
| *** salv-orlando has quit IRC | 09:51 | |
| *** magicboiz has quit IRC | 09:57 | |
| *** magicboiz has joined #openstack-kolla | 09:57 | |
| *** dixiaoli has quit IRC | 09:59 | |
| *** satyar has quit IRC | 10:03 | |
| *** yangyapeng has quit IRC | 10:05 | |
| *** yangyapeng has joined #openstack-kolla | 10:07 | |
| *** yangyapeng has quit IRC | 10:09 | |
| *** gfidente has joined #openstack-kolla | 10:11 | |
| *** gfidente has quit IRC | 10:11 | |
| *** gfidente has joined #openstack-kolla | 10:11 | |
| *** bjolo_ has joined #openstack-kolla | 10:25 | |
| *** bjolo has quit IRC | 10:25 | |
| *** jtriley has quit IRC | 10:36 | |
| *** cuongnv has quit IRC | 10:37 | |
| *** cliles has quit IRC | 10:45 | |
| *** caowei has quit IRC | 10:49 | |
| *** zhugaoxiao has quit IRC | 10:58 | |
| *** Administrator_ has quit IRC | 10:58 | |
| *** zhugaoxiao has joined #openstack-kolla | 10:58 | |
| *** Administrator_ has joined #openstack-kolla | 10:58 | |
| hrw | morning | 10:58 |
| hrw | egonzalez, duonghq: can you take a look at https://review.openstack.org/#/c/457967/ and https://review.openstack.org/#/c/457983/ patches? simple update of helm-repository and patch to make it work on non-x86 architectures. Both approved by sbezverk (maintainer). | 11:00 |
| *** tonanhngo has joined #openstack-kolla | 11:04 | |
| *** tonanhngo has quit IRC | 11:05 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: WIP: Set kibana default index pattern https://review.openstack.org/458156 | 11:08 |
| Serlex | Anyone having problems with openvswitch with latest kolla-ansible? | 11:12 |
| egonzalez | Serlex, nope, what kind of issues? | 11:13 |
| *** yangyapeng has joined #openstack-kolla | 11:15 | |
| *** tvignaud has quit IRC | 11:15 | |
| *** rwallner has joined #openstack-kolla | 11:17 | |
| *** rwallner has quit IRC | 11:17 | |
| *** rwallner has joined #openstack-kolla | 11:17 | |
| Serlex | I see that openvswitch was split 10 days ago, and ovsdb doesn't get setup for me | 11:23 |
| Serlex | I'm failing to find out why | 11:23 |
| *** gkadam has quit IRC | 11:24 | |
| egonzalez | Serlex, any error or when is not "setup" | 11:25 |
| Serlex | Not sure how to check why the ovs ansible stuff isn't run, error I'm getting is `WARNING neutron.agent.ovsdb.native.vlog [-] tcp:XXXX:6640: send error: Connection refused` | 11:27 |
| *** blallau has joined #openstack-kolla | 11:31 | |
| *** rwallner has quit IRC | 11:32 | |
| *** shardy is now known as shardy_lunch | 11:32 | |
| *** salv-orl_ has quit IRC | 11:40 | |
| *** rwallner has joined #openstack-kolla | 11:42 | |
| *** tvignaud has joined #openstack-kolla | 11:45 | |
| *** unicell1 has quit IRC | 11:53 | |
| *** satyar has joined #openstack-kolla | 11:54 | |
| *** rmart04 has quit IRC | 11:54 | |
| *** lpetrut has joined #openstack-kolla | 11:55 | |
| *** manheim has quit IRC | 11:59 | |
| *** rwellum has joined #openstack-kolla | 12:10 | |
| *** jaosorior is now known as jaosorior_away | 12:12 | |
| *** salv-orlando has joined #openstack-kolla | 12:17 | |
| *** manheim has joined #openstack-kolla | 12:18 | |
| *** rwallner has quit IRC | 12:19 | |
| *** magicboiz has quit IRC | 12:19 | |
| *** shardy_lunch is now known as shardy | 12:25 | |
| *** dmellado has quit IRC | 12:26 | |
| *** rmart04 has joined #openstack-kolla | 12:44 | |
| *** manheim has quit IRC | 12:46 | |
| *** Serlex has quit IRC | 12:46 | |
| *** dmellado has joined #openstack-kolla | 12:47 | |
| *** tvignaud has quit IRC | 12:50 | |
| *** rwallner has joined #openstack-kolla | 12:51 | |
| *** manheim has joined #openstack-kolla | 12:53 | |
| *** Serlex has joined #openstack-kolla | 12:57 | |
| *** jrist has joined #openstack-kolla | 12:59 | |
| *** jrist has quit IRC | 12:59 | |
| *** jrist has joined #openstack-kolla | 12:59 | |
| *** StephenWang1991 has joined #openstack-kolla | 13:02 | |
| *** tvignaud has joined #openstack-kolla | 13:03 | |
| rwellum | sdake, results from me following the latest deployment guide. https://www.irccloud.com/pastebin/9XTW5oqT/ | 13:04 |
| *** iceyao has joined #openstack-kolla | 13:04 | |
| *** goldyfruit has joined #openstack-kolla | 13:06 | |
| *** srwilkers has joined #openstack-kolla | 13:07 | |
| *** eanylin has quit IRC | 13:07 | |
| *** iceyao has quit IRC | 13:09 | |
| *** cuongnv has joined #openstack-kolla | 13:09 | |
| *** StephenWang1991 has quit IRC | 13:11 | |
| *** jtriley has joined #openstack-kolla | 13:16 | |
| *** Pavo has joined #openstack-kolla | 13:19 | |
| EmilienM | inc0, sdake: can you guys make sure some Kolla folks join us on https://etherpad.openstack.org/p/BOS-forum-future-of-configuration-management | 13:21 |
| *** eanylin has joined #openstack-kolla | 13:26 | |
| *** dave-mccowan has joined #openstack-kolla | 13:27 | |
| *** newmember has quit IRC | 13:33 | |
| *** srwilkers has quit IRC | 13:34 | |
| *** rmart04_ has joined #openstack-kolla | 13:35 | |
| *** rmart04 has quit IRC | 13:36 | |
| *** rmart04_ is now known as rmart04 | 13:36 | |
| *** bjolo_ has quit IRC | 13:40 | |
| *** manheim has quit IRC | 13:41 | |
| *** shardy is now known as shardy_afk | 13:44 | |
| *** Serlex has quit IRC | 13:46 | |
| *** shasha___ has joined #openstack-kolla | 13:47 | |
| *** manheim has joined #openstack-kolla | 13:48 | |
| *** shasha___ has quit IRC | 13:52 | |
| *** iniazi_ has joined #openstack-kolla | 13:54 | |
| *** jmccarthy has joined #openstack-kolla | 13:57 | |
| *** iniazi has quit IRC | 13:57 | |
| *** tonanhngo has joined #openstack-kolla | 13:57 | |
| *** rwallner has quit IRC | 13:58 | |
| *** rwallner has joined #openstack-kolla | 13:59 | |
| jmccarthy | Does this workflow work https://docs.openstack.org/developer/kolla-ansible/quickstart.html ? After pip installing kolla-ansible 4.0.0, the next steps don't seem to work (like Copy the configuration files globals.yml and passwords.yml to /etc directory. #CentOS cp -r /usr/share/kolla-ansible/etc_examples/kolla /etc/kolla/) | 14:00 |
| jmccarthy | Is there a newer doc for 4.0.0 ansible type deploy 'Install Kolla for deployment' style (that doesn't involve building images) ? | 14:01 |
| egonzalez | jmccarthy, copy those files should work, what error is giving you? | 14:02 |
| jmccarthy | There is no etc_examples dir there | 14:02 |
| egonzalez | ubuntu or centos? are in different paths | 14:02 |
| jmccarthy | Centos, I'll double check | 14:03 |
| *** manheim has quit IRC | 14:03 | |
| jmccarthy | Ok I dunno not there, is the pip install of kolla-ansible enough ? | 14:03 |
| *** jaosorior_away is now known as jaosorior_mtg | 14:05 | |
| egonzalez | jmccarthy, just made pip install kolla-ansible and files are in the expected location | 14:05 |
| egonzalez | jmccarthy, can you double check if kolla-ansible was installed correctly, pip freeze | grep kolla | 14:06 |
| *** manheim has joined #openstack-kolla | 14:06 | |
| jmccarthy | :( ! pip install says it installed .. here ? ls: cannot access /usr/share/kolla-ansible/etc_examples/kolla: No such file or directory | 14:06 |
| jmccarthy | kolla-ansible==4.0.0 | 14:06 |
| egonzalez | jmccarthy, when was installed, atm kolla-ansible is 4.0.1 version | 14:08 |
| jmccarthy | Strange - just 5 minutes ago | 14:08 |
| jmccarthy | Ok I must have something screwy in this system then | 14:08 |
| jmccarthy | Dam ok I'll try it again :) | 14:08 |
| egonzalez | hrm, have you ever installed kolla-ansible from git repo? | 14:08 |
| egonzalez | jmccarthy, can you paste full pip freeze output? | 14:09 |
| jmccarthy | Not in this VM, it's only about 30 mintues old - but possibly I have forgotten something one my scripts does .. | 14:09 |
| jmccarthy | Ok I think I will start over with fresh VM, if you got it ok, then must be something I screwed up lol | 14:11 |
| *** manheim has quit IRC | 14:12 | |
| *** manheim has joined #openstack-kolla | 14:13 | |
| *** dixiaoli has joined #openstack-kolla | 14:15 | |
| *** rmart04_ has joined #openstack-kolla | 14:15 | |
| *** rmart04 has quit IRC | 14:16 | |
| *** rmart04_ is now known as rmart04 | 14:16 | |
| *** shardy_afk is now known as shardy | 14:17 | |
| *** dixiaoli has quit IRC | 14:19 | |
| *** yangyape_ has joined #openstack-kolla | 14:27 | |
| *** salv-orlando has quit IRC | 14:28 | |
| *** yangyapeng has quit IRC | 14:29 | |
| openstackgerrit | sean mooney proposed openstack/kolla-ansible master: This change allow setting kernel commandline args https://review.openstack.org/408876 | 14:29 |
| openstackgerrit | sean mooney proposed openstack/kolla-ansible master: introduce contrib playbook for ovs-dpdk https://review.openstack.org/408872 | 14:29 |
| *** trozet has quit IRC | 14:29 | |
| *** skramaja has quit IRC | 14:30 | |
| *** slagle has quit IRC | 14:42 | |
| *** myoung has quit IRC | 14:43 | |
| jmccarthy | egonzalez: Ah ok - I see now the dir is there and I'm at 4.0.1 in new vm :) Sorry for the noise | 14:44 |
| *** manheim has quit IRC | 14:44 | |
| * egonzalez wondering what happen in previous deploy :/ | 14:44 | |
| *** trozet has joined #openstack-kolla | 14:45 | |
| *** manheim has joined #openstack-kolla | 14:45 | |
| jmccarthy | me too but I really just want to get an aio up and running here lol | 14:47 |
| *** Serlex has joined #openstack-kolla | 14:47 | |
| inc0 | good morning | 14:50 |
| inc0 | i | 14:52 |
| inc0 | EmilienM: I'll be there | 14:52 |
| Serlex | ping mgkwill | 14:52 |
| EmilienM | inc0: good | 14:52 |
| *** srwilkers has joined #openstack-kolla | 14:54 | |
| *** Pavo has quit IRC | 14:57 | |
| *** jaosorior_mtg is now known as jaosorior | 14:58 | |
| *** jaosorior has quit IRC | 14:59 | |
| *** egonzalez has quit IRC | 15:00 | |
| *** erhudy has left #openstack-kolla | 15:01 | |
| *** gianpietro has joined #openstack-kolla | 15:03 | |
| *** slagle has joined #openstack-kolla | 15:07 | |
| *** newmember has joined #openstack-kolla | 15:08 | |
| *** MasterOfBugs has joined #openstack-kolla | 15:09 | |
| *** MasterOfBugs has quit IRC | 15:10 | |
| *** MasterOfBugs has joined #openstack-kolla | 15:11 | |
| *** MasterOfBugs has quit IRC | 15:12 | |
| *** manheim has quit IRC | 15:15 | |
| jmccarthy | Hmm is this expected ? | 15:15 |
| jmccarthy | TASK [common : Pulling kolla-toolbox image] ************************************ | 15:15 |
| jmccarthy | fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Unknown error message: Tag 4.0.1 not found in repository docker.io/kolla/centos-binary-kolla-toolbox"} | 15:15 |
| *** jtriley_ has joined #openstack-kolla | 15:15 | |
| *** manheim has joined #openstack-kolla | 15:16 | |
| *** gianpietro has quit IRC | 15:17 | |
| *** jtriley has quit IRC | 15:18 | |
| *** MasterOfBugs has joined #openstack-kolla | 15:18 | |
| *** manheim has quit IRC | 15:19 | |
| *** manheim has joined #openstack-kolla | 15:23 | |
| *** jmontleon has left #openstack-kolla | 15:24 | |
| *** rmart04 has quit IRC | 15:27 | |
| *** manheim has quit IRC | 15:28 | |
| *** gianpietro has joined #openstack-kolla | 15:28 | |
| *** manheim has joined #openstack-kolla | 15:29 | |
| *** Pavo has joined #openstack-kolla | 15:30 | |
| openstackgerrit | Merged openstack/kolla master: helm-repository: update to v2.3.0 https://review.openstack.org/457967 | 15:34 |
| *** ssurana has joined #openstack-kolla | 15:35 | |
| *** MasterOfBugs has quit IRC | 15:36 | |
| *** MasterOfBugs has joined #openstack-kolla | 15:37 | |
| openstackgerrit | Merged openstack/kolla master: helm-repository: make it work on non-x86 https://review.openstack.org/457983 | 15:37 |
| *** manheim has quit IRC | 15:41 | |
| *** lpetrut has quit IRC | 15:44 | |
| *** MasterOfBugs has quit IRC | 15:45 | |
| *** manheim has joined #openstack-kolla | 15:46 | |
| mgkwill | Hi Serlex what error you see? | 15:48 |
| *** MasterOfBugs has joined #openstack-kolla | 15:50 | |
| jmccarthy | Hmm should I drop back to 4.0.0 ? | 15:50 |
| jmccarthy | TASK [common : Pulling kolla-toolbox image] ************************************ | 15:50 |
| jmccarthy | fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Unknown error message: Tag 4.0.1 not found in repository docker.io/kolla/centos-binary-kolla-toolbox"} | 15:50 |
| kfox1111 | morning. | 15:50 |
| *** rwallner has quit IRC | 15:51 | |
| *** trozet has quit IRC | 15:51 | |
| *** matrohon_ has quit IRC | 15:52 | |
| *** gianpietro has quit IRC | 15:52 | |
| *** mgoddard has quit IRC | 15:54 | |
| *** rwallner has joined #openstack-kolla | 15:55 | |
| Serlex | mgkwill "WARNING neutron.agent.ovsdb.native.vlog [-] tcp:XXXXXX:6640: send error: Connection refused" | 15:55 |
| *** gianpietro has joined #openstack-kolla | 15:57 | |
| *** MasterOfBugs has quit IRC | 15:57 | |
| *** david-lyle has joined #openstack-kolla | 15:58 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible stable/ocata: Mount /lib/modules folder for cinder-volume container https://review.openstack.org/458430 | 15:59 |
| *** salv-orlando has joined #openstack-kolla | 16:00 | |
| *** manheim has quit IRC | 16:02 | |
| *** salv-orlando has quit IRC | 16:04 | |
| *** salv-orlando has joined #openstack-kolla | 16:05 | |
| *** manheim has joined #openstack-kolla | 16:08 | |
| mgkwill | Serlex Hmm, I doubt I changed anything about port. Is anything else using 6640? "ss -naptl | grep 6640" | 16:10 |
| mgkwill | Serlex I'm on PTO today so won't have much time but can take a look tomorrow | 16:11 |
| *** zhubingbing_ has quit IRC | 16:12 | |
| *** gianpietro has quit IRC | 16:12 | |
| *** cuongnv has quit IRC | 16:13 | |
| *** rwallner has quit IRC | 16:14 | |
| *** jascott1 has joined #openstack-kolla | 16:16 | |
| *** manheim has quit IRC | 16:17 | |
| openstackgerrit | sean mooney proposed openstack/kolla master: introduces support for the OVS DPDK dataplane https://review.openstack.org/342354 | 16:17 |
| jascott1 | goo'd morning | 16:20 |
| *** manheim has joined #openstack-kolla | 16:20 | |
| *** imcsk8 has quit IRC | 16:21 | |
| *** Pavo has quit IRC | 16:21 | |
| *** rmart04 has joined #openstack-kolla | 16:21 | |
| *** rmart04 has quit IRC | 16:22 | |
| *** imcsk8 has joined #openstack-kolla | 16:22 | |
| *** shasha___ has joined #openstack-kolla | 16:27 | |
| jmccarthy | Anyone hit this sort of thing with ansible 2.1.0.0 ? http://paste.openstack.org/show/607485/ | 16:33 |
| sean-k-mooney | mgkwill: Serlex odl and the neutron agents conflit. they both try to use teh same port to talk to ovs | 16:34 |
| sean-k-mooney | Serlex: do you have odl enabled? | 16:34 |
| *** iniazi_ has quit IRC | 16:35 | |
| *** iniazi has joined #openstack-kolla | 16:35 | |
| sean-k-mooney | mgkwill: Serlex if so you will need to move the neutron dhcp and l3 agents to a different port(or odl) or use odl for dhcp and l3 routing | 16:36 |
| sean-k-mooney | inc0: the latest version of the ovsdpdk patches now work again and can spawn instance correctly | 16:38 |
| sean-k-mooney | inc0: i will update the docs and and will take a look at odl on monday | 16:39 |
| *** mgoddard has joined #openstack-kolla | 16:39 | |
| *** rwallner has joined #openstack-kolla | 16:41 | |
| *** gfidente is now known as gfidente|afk | 16:42 | |
| *** rwallner has quit IRC | 16:47 | |
| *** srwilkers has quit IRC | 16:47 | |
| *** jmccarthy has left #openstack-kolla | 16:48 | |
| *** shardy has quit IRC | 16:50 | |
| *** dmellado has quit IRC | 16:52 | |
| *** unicell has joined #openstack-kolla | 16:53 | |
| *** unicell has quit IRC | 16:53 | |
| *** unicell has joined #openstack-kolla | 16:53 | |
| *** unicell has quit IRC | 16:54 | |
| Serlex | sean-k-mooney: nothing else is running on port 6640, I can't see ansible executed any of the ovs ansible playbooks either | 16:55 |
| *** mgoddard has quit IRC | 17:00 | |
| *** salv-orlando has quit IRC | 17:00 | |
| *** rwallner has joined #openstack-kolla | 17:01 | |
| mgkwill | Serlex sounds like inventory issue, as workaround you could try adding specific compute nodes to openvswitch group. I'll take a look at this tomorrow and update if issue. There was a change toward end of patch that narrowed inventory groups due to comments. Only thing I can think of without looking has to do with that. | 17:03 |
| jascott1 | anyone seeing "stderr": "Error: transport is closing" when doing compute-kit install with helm? | 17:03 |
| *** rwallner_ has joined #openstack-kolla | 17:04 | |
| *** shasha___ has quit IRC | 17:04 | |
| kfox1111 | jascott1: not in that use case specifically, but have seen it when tiller crashes? | 17:04 |
| jascott1 | ya | 17:05 |
| sdake | rwellum did you wait for copute to deploy before launching the cell chart? | 17:05 |
| sdake | EmilienM you assume that I have any ability to make anyone do anything :) | 17:05 |
| *** rwallner has quit IRC | 17:06 | |
| EmilienM | sdake: no, just spread the message :) | 17:06 |
| sdake | EmilienM cool - inc0 can do that - i am struggling to even find out which core reviewers will be at summit | 17:06 |
| *** dmellado has joined #openstack-kolla | 17:06 | |
| sdake | morning humans :) | 17:06 |
| jascott1 | kfox1111 its nova-control thats blowing up.. hmm it has lots of dep charts | 17:07 |
| inc0 | yeah same here | 17:07 |
| Serlex | ok thanks mgkwill | 17:07 |
| jascott1 | sdake moorning | 17:07 |
| kfox1111 | sdake: morning. | 17:08 |
| sdake | sup jascott1 kfox1111 | 17:08 |
| kfox1111 | jascott1: odd. yeah, lots of dep charts. but we are running it in gate. helm 2.3 + k8s 1.6? | 17:08 |
| jascott1 | sure | 17:09 |
| mgkwill | Serlex No problem. If there is issue I'll get updated quickly. | 17:09 |
| rwellum | sdake: yes I did | 17:13 |
| sdake | jascott1 on ubuntu or centos? | 17:14 |
| *** manheim has quit IRC | 17:14 | |
| jascott1 | ubuntu | 17:14 |
| jascott1 | hmm something in my kolla-controller stuff borks my k8s | 17:15 |
| sdake | jascott1 - neutron is likely taking over your main interface | 17:16 |
| sdake | jascott1 this happens often - inc0 should be able to help you debug - i'm kind of swamped today with last minute prep for summit | 17:16 |
| jascott1 | ah | 17:16 |
| jascott1 | thanks for the pointer | 17:17 |
| sdake | jascott1 you need two interfaces - one is management one is neutron | 17:17 |
| jascott1 | right | 17:17 |
| kfox1111 | yeah. neutron likes to play with network bridges and can pull the rug out from under you if not setup perfectly. :/ | 17:18 |
| jascott1 | is there a good writeup of that anywhere? | 17:18 |
| jascott1 | the gate i guess :) | 17:18 |
| kfox1111 | either sdake's documentation or the gate. | 17:18 |
| rwellum | https://docs.openstack.org/developer/kolla-kubernetes/deployment-guide.html | 17:18 |
| jascott1 | ya thats what I based my playbook from | 17:18 |
| jascott1 | thanks ery body | 17:19 |
| kfox1111 | let me know if you get stuck. | 17:20 |
| *** unicell has joined #openstack-kolla | 17:22 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Docker Logging Support https://review.openstack.org/458649 | 17:34 |
| *** salv-orlando has joined #openstack-kolla | 17:34 | |
| *** newmember has quit IRC | 17:42 | |
| *** rwallner_ has quit IRC | 17:47 | |
| *** rwsu has quit IRC | 17:51 | |
| *** rwallner has joined #openstack-kolla | 17:52 | |
| *** Serlex has quit IRC | 17:54 | |
| *** gfidente|afk has quit IRC | 17:54 | |
| *** shasha___ has joined #openstack-kolla | 17:55 | |
| *** rwallner has quit IRC | 18:12 | |
| sdake | kfox1111 sbezverk any idea what creates this file: KUBE_NAMESPACE=$(</var/run/secrets/kubernetes.io/serviceaccount/namespace) | 18:12 |
| *** sambetts is now known as sambetts|afk | 18:15 | |
| jascott1 | should i have 30 calico interfaces? | 18:18 |
| *** shasha___ has quit IRC | 18:18 | |
| *** shasha___ has joined #openstack-kolla | 18:19 | |
| *** rwallner has joined #openstack-kolla | 18:22 | |
| *** shasha___ has quit IRC | 18:24 | |
| *** chas has quit IRC | 18:26 | |
| kfox1111 | sdake: k8s does. | 18:27 |
| kfox1111 | jascott1: yeah. | 18:27 |
| kfox1111 | jascott1: one for every pod not running net=host. | 18:27 |
| kfox1111 | sdake: thats part of the magic of k8s that I keep pushing for in openstack. | 18:28 |
| inc0 | anyone know how to use nova serial-console over ssh? | 18:28 |
| kfox1111 | k8s automatically creates/manages credential and access info for every pod. | 18:28 |
| kfox1111 | never goten serial-console to work. | 18:29 |
| *** slagle_ has joined #openstack-kolla | 18:29 | |
| *** tvignaud has quit IRC | 18:29 | |
| *** salv-orlando has quit IRC | 18:29 | |
| kfox1111 | let me know if you do. would be awesome to be able to use zmodem over serial-console over ssh. ;) | 18:29 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Docker Logging Support https://review.openstack.org/458649 | 18:35 |
| *** srwilkers has joined #openstack-kolla | 18:36 | |
| *** rwsu has joined #openstack-kolla | 18:36 | |
| sdake | kfox1111 i am looking for the values of namespace but don't see such a file: | 18:38 |
| sdake | [sdake@beast-01 serviceaccount]$ pwd | 18:38 |
| sdake | [sdake@beast-01 serviceaccount]$ ls -l | 18:38 |
| sdake | total 0 | 18:38 |
| sdake | as in what is in that file specifically | 18:38 |
| kfox1111 | thats in a k8s managed pod? not a static pod? | 18:38 |
| sdake | how does fernet access it? | 18:39 |
| kfox1111 | equiv of cat | 18:39 |
| sdake | right - i mean from which pod? | 18:39 |
| kfox1111 | the pod its in. | 18:39 |
| sdake | i read the fernet code | 18:39 |
| sdake | ok thanks - so its not a bindmound | 18:40 |
| sdake | host bindmount | 18:40 |
| kfox1111 | kind of a bind mount. | 18:40 |
| kfox1111 | k8s builds it. | 18:40 |
| kfox1111 | its a special automatic volume k8s does. | 18:40 |
| kfox1111 | sdake: se, for example, http://logs.openstack.org/49/458649/4/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-nv/fd53189/logs/pods/kolla-ceph-osd1.txt | 18:41 |
| kfox1111 | look under volumes, for something that looks like default-token-xxxxxxx | 18:41 |
| kfox1111 | its automatically created by k8s. | 18:41 |
| sdake | default-token-x8ft6: | 18:42 |
| sdake | this one? | 18:42 |
| kfox1111 | yeah. | 18:42 |
| kfox1111 | if there is no token in the pod discription, k8s will automatically add one. | 18:42 |
| inc0 | kfox1111: got it working:) | 18:42 |
| *** myoung has joined #openstack-kolla | 18:42 | |
| kfox1111 | so all pods by default can access the the k8s its running under. | 18:43 |
| kfox1111 | inc0: cool. :) | 18:43 |
| inc0 | so nova get-serial-console outputs link | 18:43 |
| inc0 | http://blog.oddbit.com/2014/12/22/accessing-the-serial-console-of-your-nova-servers/ then follow this, larks created tool that works | 18:43 |
| kfox1111 | nice. thanks. :) | 18:44 |
| *** rwsu has quit IRC | 18:45 | |
| inc0 | I created ssh tunnel to port in question then used novaconsole --url localhost:my-ssh-port/?token=... | 18:46 |
| *** tvignaud has joined #openstack-kolla | 18:47 | |
| *** rwsu has joined #openstack-kolla | 18:50 | |
| *** shasha___ has joined #openstack-kolla | 18:50 | |
| *** shasha___ has quit IRC | 18:54 | |
| sdake | kfox1111 what sets this variable: $KUBERNETES_PORT_443_TCP_PORT | 18:55 |
| sdake | (in the fernet token running) | 18:55 |
| *** rwsu has quit IRC | 19:00 | |
| *** pcaruana has quit IRC | 19:02 | |
| openstackgerrit | Rich Wellum proposed openstack/kolla-kubernetes master: Various cleanups and additions https://review.openstack.org/458939 | 19:03 |
| *** kbyrne has quit IRC | 19:04 | |
| *** tvignaud has quit IRC | 19:06 | |
| *** rwsu has joined #openstack-kolla | 19:07 | |
| *** gianpietro has joined #openstack-kolla | 19:09 | |
| sbezverk | kfox1111: we really need to do something with namespaces support.. namesapces are so unstable with dhcp agents :( especially in multi node env .. | 19:11 |
| *** Slower_ is now known as Slower | 19:13 | |
| jascott1 | my problem was caused by VM not having enough memory, intermittent `tunnel is closing` | 19:14 |
| sdake | kfox1111 have you seen this before? http://paste.openstack.org/show/607513/ | 19:16 |
| kfox1111 | sdake: hmm... the slack sinc didn't kick in.. | 19:16 |
| sdake | error 60 from curl = bad creds | 19:16 |
| *** newmember has joined #openstack-kolla | 19:17 | |
| kfox1111 | sdake: KUBERNETES_PORT_443_TCP_PORT comes from k8s service discovery | 19:17 |
| sdake | kfox1111 i'll ook at the slack bot | 19:17 |
| sdake | moment | 19:17 |
| kfox1111 | sbezverk: +1. if the openstack/k8s sig means anything, that one shoudl be top priority. :/ | 19:17 |
| kfox1111 | jascott1: ah. | 19:17 |
| jascott1 | i think ive got everything installing and am back to my good ol lack of openstack configuration knowledge :) | 19:18 |
| *** srwilkers has quit IRC | 19:18 | |
| jascott1 | mariadb-init-element: Error syncing pod, skipping: failed to "StartContainer" for "main" with RunContainerError: "GenerateRunContainerOptions: secrets \"database-password\" not found" | 19:19 |
| kfox1111 | sdake: we need to utilize the sig openstack/k8s relationship to budge the shared mount roadblock. its really hurting us. | 19:19 |
| kfox1111 | (and a lot of others) | 19:20 |
| jascott1 | this thing is pulling 4.0 images, is that right? | 19:20 |
| *** tvignaud has joined #openstack-kolla | 19:20 | |
| kfox1111 | if following sdake's instructions, yeah. | 19:20 |
| jascott1 | k thanks | 19:20 |
| *** kolla-slack has quit IRC | 19:20 | |
| *** kolla-slack has joined #openstack-kolla | 19:21 | |
| kfox1111 | inc0: "One disadvantage to the serial console support is that it replaces the console log available via nova console-log" | 19:21 |
| kfox1111 | inc0: this is the thing about openstack I hate the most. there are a million things to do but you can only ever do half or less at once. :/ | 19:21 |
| kfox1111 | cause they never integrate them together. | 19:21 |
| sdake | jascott1 you didn't run genconfig | 19:21 |
| jascott1 | hmm | 19:21 |
| sdake | jascott1 or kollakube generate secretes | 19:21 |
| jascott1 | thx will look | 19:21 |
| jascott1 | i have an entry for it but idk | 19:22 |
| kfox1111 | you can have dvr, but octavia breaks. :/ | 19:22 |
| *** gianpietro has quit IRC | 19:22 | |
| kfox1111 | for along time, dvr or l3ha. not both. | 19:23 |
| kfox1111 | lbaasv1 or lbaasv2. not both. | 19:23 |
| kfox1111 | serial console on ironic, but not console-log | 19:23 |
| kfox1111 | etc. | 19:23 |
| sdake | kfox1111 what do you make of this error: http://paste.openstack.org/show/607514/ | 19:25 |
| kfox1111 | sdake: how did you stand up k8s? | 19:27 |
| sdake | kfox1111 i think it may have been kargo - someone gave me an env to work with - i'm working with it :) | 19:28 |
| kfox1111 | seems like its not working. | 19:28 |
| sdake | helm install works | 19:28 |
| sdake | kubectl works | 19:28 |
| kfox1111 | serviceaccounts don't seem to be setup though. | 19:28 |
| kfox1111 | or having some issues. | 19:28 |
| sdake | cool - thats helpful | 19:29 |
| kfox1111 | so... lets see.... | 19:29 |
| sdake | how would I take a look at fixing that? | 19:29 |
| kfox1111 | maybe have a look at the kube-apiserver yaml. | 19:29 |
| kfox1111 | see what is in --admission-control=.... | 19:30 |
| kfox1111 | there sould be a ServiceAccount in it | 19:30 |
| *** haplo37 has quit IRC | 19:31 | |
| *** g3ek has quit IRC | 19:31 | |
| kfox1111 | there should be a --service-account-key-file=... specified too. | 19:32 |
| sdake | kfox1111 that is specified in the manifest for kube-apiserver? | 19:34 |
| kfox1111 | yeah. | 19:34 |
| *** tvignaud has quit IRC | 19:34 | |
| rwellum | kfox1111: do you recall in the deployment guide, when you pointed out that the 'taint' command is really removing the taint? I noticed today, when I apply the command the response is 'node kolladev" tainted' - not 'untainted', or not 'node xyz taint removed' etc... Isn't that a bit odd? | 19:35 |
| *** g3ek has joined #openstack-kolla | 19:35 | |
| kfox1111 | rwellum: yeah. they use the same command to taint/untaint and the output always prints 'tainted' reguardless of what its doing. | 19:36 |
| *** shasha___ has joined #openstack-kolla | 19:36 | |
| kfox1111 | I'd consider it a usability bug. | 19:36 |
| *** haplo37 has joined #openstack-kolla | 19:36 | |
| sdake | kfox1111 right | 19:36 |
| sdake | kfox1111 - would you say this is accurate: https://kubernetes.io/docs/tasks/configure-pod-container/configure-service-account/ | 19:36 |
| rwellum | thanks kfox1111 | 19:37 |
| sdake | kfox1111 as in - could i create a service account without modifying the manifest? | 19:37 |
| kfox1111 | sdake: that looks correct. | 19:37 |
| *** pbourke has quit IRC | 19:37 | |
| jascott1 | are these normal errors? Error: release nova-placement-create-keystone-user-job failed: jobs.batch \"placement-create-keystone-user\" already exists" | 19:37 |
| *** pbourke has joined #openstack-kolla | 19:37 | |
| kfox1111 | sdake: my guess is something didn't get followed in the admin guide section of that. | 19:38 |
| sdake | jascott1 you have already run helm install on nova-api | 19:38 |
| sbezverk | sdake: default service account gets created during the installation, I do not think you can alter it, I think you can only add one | 19:38 |
| kfox1111 | sdake: this one https://kubernetes.io/docs/admin/service-accounts-admin/ | 19:38 |
| kfox1111 | I think sbezverk's right. you can add more. but if the default one's broken, adding another one will probably just create another broken one. | 19:38 |
| kfox1111 | kolla-kubernetes should provide support for overriding which service account will be used per package at some point though. | 19:39 |
| sdake | kfox1111 lol grand ! :) | 19:39 |
| sbezverk | Yeah I have struggled with similar issue in the past, unfortunately only re-install was a solution.. | 19:39 |
| *** shasha___ has quit IRC | 19:41 | |
| -openstackstatus- NOTICE: Gerrit will be offline briefly starting at 20:00 for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 19:42 | |
| *** tvignaud has joined #openstack-kolla | 19:47 | |
| *** gianpietro has joined #openstack-kolla | 19:47 | |
| kfox1111 | sdake: just looking at kargo trunk, looks like it should have service accounts enabled. | 19:47 |
| *** myoung has quit IRC | 19:49 | |
| kfox1111 | sdake: can you pick a long running pod and exec in look at the service account directory in it? | 19:53 |
| sdake | kfox1111 i'll take a look - maybe its an old branch they are working from - I've asked them to take a look | 19:53 |
| sdake | kfox1111 thanks for pointing me in the right direction :) | 19:53 |
| kfox1111 | np. | 19:54 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Configure Kibana default index pattern https://review.openstack.org/458156 | 19:57 |
| *** StephenWang1991 has joined #openstack-kolla | 19:57 | |
| *** rwallner has quit IRC | 19:58 | |
| *** rwallner has joined #openstack-kolla | 20:00 | |
| *** StephenWang1991 has quit IRC | 20:02 | |
| -openstackstatus- NOTICE: Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html | 20:02 | |
| *** ChanServ changes topic to "Gerrit is offline briefly for scheduled maintenance http://lists.openstack.org/pipermail/openstack-dev/2017-April/115702.html" | 20:03 | |
| jascott1 | hmmm common reasons for mariadb-0 to go into CrashLoop? | 20:05 |
| jascott1 | nothing in describe or logs seems helpful | 20:05 |
| *** shasha___ has joined #openstack-kolla | 20:07 | |
| jascott1 | oh found the real logs | 20:09 |
| rwellum | jascott1: in #458939 I have tried to clean up some of the language around the choice of interfaces and ip addresses in globals.yaml and cloud.yaml - I don't know if that's the cause of your issue but typically for me it's a combo of these settings being incorrect or mis-matching that causes these basic issues. | 20:11 |
| *** shasha___ has quit IRC | 20:11 | |
| *** slagle_ has quit IRC | 20:12 | |
| jascott1 | rwellum nice, yeah thats confused me since day 1 | 20:15 |
| kfox1111 | netwrking's always the hardest part. | 20:16 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Docker Logging Support https://review.openstack.org/458649 | 20:17 |
| rwellum | it's a little more complex when using a VM kfox1111 jascott1 | 20:19 |
| rwellum | imo | 20:19 |
| kfox1111 | indeed. | 20:19 |
| *** chas has joined #openstack-kolla | 20:26 | |
| Reepicheep | So I have been playing with Kolla-ansible for a couple weeks now. it seems evedent that a lot of current development efforts are with kolla-kubernetes | 20:30 |
| Reepicheep | I would like to move forward into production.. Should I take a serious look at kolla-kubernetes now? | 20:31 |
| *** chas has quit IRC | 20:31 | |
| Reepicheep | I know the answer is probably "it depends" but what is the general feelling for the future if this is a long term deployment? | 20:32 |
| jascott1 | Reepicheep there is no production worthy release yet but we are working on it | 20:33 |
| jascott1 | if you want to get familiar now and help out you will be an expert by the time it gets there ;) | 20:33 |
| Reepicheep | I would love to do that.. I just need to provide a place in the mean time for my co-workers to start getting their hands wet with OpenStack | 20:34 |
| kfox1111 | Reepicheep: I'd say it depends on your tolerence for building from scratch. if you are already building openstack clusters from scratch, kolla-kubernetes should be about on par with that for production readyness. If you want something turn key, its definitely not that yet. | 20:35 |
| *** ChanServ changes topic to "support: ask.openstack.org | New to Kolla: docs @ http://docs.openstack.org/developer/kolla/ | | Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b" | 20:35 | |
| -openstackstatus- NOTICE: Gerrit is back in service and generally usable, though remote Git replicas (git.openstack.org and github.com) will be stale for the next few hours until online reindexing completes | 20:35 | |
| *** satyar has quit IRC | 20:35 | |
| Reepicheep | I have built them from scatch for a while.. I also then tried fuel for a while but then found it to opinionated / restrictive | 20:37 |
| kfox1111 | yeah. | 20:39 |
| sdake | Reepicheep note - if you want something to ramp up your team on how openstack works - kolla-ansible is a great solution for that | 20:39 |
| kfox1111 | I've built all mine from scratch too. | 20:39 |
| kfox1111 | I do weird stuff like run vxlans over ip over ib. | 20:39 |
| Reepicheep | most of the "Turn Key" solutions seem to price them selfs out of my K-12 budget quickly. :| | 20:39 |
| kfox1111 | fuel didn't want to support that... :) | 20:39 |
| sdake | Reepicheep its mature and flexible | 20:40 |
| kfox1111 | had to make due with whatever hardware I had available at the time. | 20:40 |
| Reepicheep | so it's seems I can choose to commit to using kolla-ansible today which is some what of a long term commitment considering the expectation of an environment to stick around | 20:46 |
| inc0 | aaaand I'm done, broke my node networking without hope of restoration:P reprovisioning OS and starting fresh | 20:47 |
| Reepicheep | or I could go with kolla-kubernetes which may be much more work now but possibly a better longturm direction as the product matures | 20:47 |
| *** rwallner has quit IRC | 20:47 | |
| *** openstackgerrit has quit IRC | 20:48 | |
| inc0 | Reepicheep: frankly I don't think kolla-ansible will be obsolete anytime soon | 20:48 |
| inc0 | I'm deploying my prod now and going all-ansible | 20:49 |
| Reepicheep | inc0: I would hope not ;) | 20:49 |
| inc0 | nah, it's stable, main reference impl for kolla still | 20:49 |
| inc0 | and not going anywhere | 20:49 |
| * kfox1111 shrugs. everyone has their biases. :) | 20:50 | |
| inc0 | k8s is cool, but not for everyone | 20:50 |
| *** jtriley_ has quit IRC | 20:50 | |
| jascott1 | hmm still cant grok the problem with mariadb-0 crash | 20:50 |
| kfox1111 | only time will tell where contributors will put the work. | 20:50 |
| inc0 | yup | 20:50 |
| jascott1 | kubectl describe and logs, and /var/logs/containers, nothing showing an error | 20:50 |
| kfox1111 | jascott1: does doing a regular pod and execing in show anything odd? | 20:51 |
| Reepicheep | I do like k8s, I drank the docker coolaid quite a while ago .. and k8s definatly seems to be the orchistration tool of choice for many | 20:51 |
| inc0 | yeah | 20:51 |
| jascott1 | kfox1111 think its a volume problem | 20:52 |
| kfox1111 | jascott1: what driver are you using? | 20:52 |
| inc0 | k8s is cool, I just think not everyone will want additional orch layer under their openstack | 20:52 |
| kfox1111 | volume issues usually show up in the describe pod output. | 20:52 |
| jascott1 | nothing in maria logs but seeing storage errors when grepping thru all logs | 20:52 |
| inc0 | if you're moving towards k8s big time, that's cool thing to do | 20:52 |
| kfox1111 | inc0: I disagree. there is always that extra layer. ansible or k8s. | 20:53 |
| kfox1111 | no "extra" | 20:53 |
| inc0 | well ansible is glorified ssh | 20:53 |
| kfox1111 | its an orchestration layer via ssh. and doesn't run all the time. but its as black magicy as k8s. :) | 20:53 |
| inc0 | while k8s is full fledged cloud orch engine | 20:53 |
| kfox1111 | though I find k8s's black magic easier to grok. | 20:53 |
| *** shasha___ has joined #openstack-kolla | 20:53 | |
| inc0 | well ansible has little of magic, it's a shellscript over ssh written in yaml | 20:54 |
| inc0 | things happen in order on hosts you want them to happen | 20:55 |
| inc0 | nothing more nothing less | 20:55 |
| kfox1111 | its a complicated programming langage written in yaml. | 20:55 |
| Reepicheep | well I have been using puppet for years so even using kolla-ansible my initial thoughts where,, really descriptive instead of declaritive .. hmm.. I guess it time for me to try this ansible stuff ;) | 20:55 |
| inc0 | yeah, ansible is not declarative as puppet or k8s really | 20:56 |
| Reepicheep | so the jump to a for k8s orchistration running the openstack would not be much more of a jump | 20:56 |
| inc0 | maybe that's the core of our disagreement;) I never really used declarative stuff | 20:56 |
| kfox1111 | inc0: yeah. | 20:56 |
| Reepicheep | inc0: lol.. I know and I was / am still kinda in the other camp | 20:57 |
| Reepicheep | but if it get's the job done.. all the better | 20:57 |
| kfox1111 | its spin up time for understanding what the language does. imperitive can take a lot more time to bring up to speed. k8s's pod descritpion is a relatively small declarative language. | 20:57 |
| inc0 | yeah I don't think one is better than other | 20:57 |
| kfox1111 | yup. | 20:57 |
| kfox1111 | ansible definitely has its uses. :) | 20:58 |
| inc0 | but to do complex stuff you need to abuse it | 20:58 |
| *** shasha___ has quit IRC | 20:58 | |
| kfox1111 | yeah. k8s isn't intended to do complicated workflows. something drives it for that. | 20:58 |
| *** tvignaud has quit IRC | 20:58 | |
| inc0 | and that something is missing, can be ansible :P | 20:59 |
| inc0 | I think that's reason I'm not big fan of k8s...you still need something to run it | 20:59 |
| inc0 | and certain tasks becomes complex problems | 21:00 |
| kfox1111 | thtas why I was insistent on k8s packages being seperable from orchestration system. | 21:00 |
| Reepicheep | yup.. I have always been preaching that there is no magic bullet when it comes to tools so it's probably going to be a combination of many | 21:00 |
| kfox1111 | so if someone wants to drive k8s deployment of openstack with ansible, more power to you. :) | 21:00 |
| inc0 | yeah | 21:00 |
| inc0 | maybe good idea is to ansible install k8s, compute nodes and stuff like that while k8s runs services | 21:01 |
| inc0 | so here's thing, if I'd run my own k8s apps | 21:01 |
| inc0 | I'll deploy them with ansible | 21:01 |
| *** jascott1 has quit IRC | 21:02 | |
| Reepicheep | I would probably use terraform (descriptive) ;) | 21:02 |
| *** spsurya has quit IRC | 21:03 | |
| kfox1111 | terraform's becoming very popular around here. | 21:03 |
| inc0 | I don't know first thing about it | 21:04 |
| Reepicheep | for deployment of k8s I have used two tools to simplify it .. murano (that would be odd in this instance) and rancher | 21:04 |
| *** tvignaud has joined #openstack-kolla | 21:04 | |
| inc0 | maybe it's time to learn | 21:04 |
| kfox1111 | inc0: kind of a conglomeration of ansible and heat. | 21:04 |
| inc0 | which might be good | 21:04 |
| inc0 | in fact, there are tasks better for declarative and tasks better for imperative | 21:05 |
| kfox1111 | yeah. | 21:05 |
| Reepicheep | I will probably go the Rancher route because we are aleady invested in Rancher | 21:05 |
| inc0 | maybe correct tool is one that will coherently support both models | 21:05 |
| kfox1111 | Reepicheep: I havent played with it yet, but hav been keeping an eye on it. always looked kind of cool. | 21:05 |
| kfox1111 | inc0: +1 | 21:06 |
| Reepicheep | so this is what I have got out of this.. It's Friday.. monday I'm going to deploy an instance of k8s to do kolla-kubernetes on .. may as well try | 21:07 |
| inc0 | hehe;) | 21:07 |
| inc0 | and I'll deploy my ansible | 21:07 |
| Reepicheep | It's not like I can't quickly get back to where I am using kolla-ansible | 21:07 |
| kfox1111 | cool. | 21:07 |
| kfox1111 | Reepicheep: going to the summit? | 21:08 |
| Reepicheep | then after I have used both .. I will figure where I want to put my environment | 21:08 |
| kfox1111 | sounds like a good plan. | 21:08 |
| Reepicheep | kfox1111: nope.. I have used up my confrence budget this year | 21:08 |
| Reepicheep | maybe next time | 21:08 |
| kfox1111 | bummer. | 21:09 |
| inc0 | try PTG if possible | 21:09 |
| *** openstackgerrit has joined #openstack-kolla | 21:11 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test gate. https://review.openstack.org/458973 | 21:11 |
| kfox1111 | inc0: can we get a kolla container for this: https://github.com/CanonicalLtd/prometheus-openstack-exporter | 21:12 |
| kolla-slack | <jascott1> bah frontier is down | 21:13 |
| * inc0 laughts in comcast | 21:13 | |
| inc0 | kfox1111: I don't see why not | 21:13 |
| kfox1111 | inc0: been usign prometheus a lot. its very cool. :) | 21:14 |
| inc0 | I mean we're getting lots of containers that aren't openstack | 21:14 |
| inc0 | which is cool/bad | 21:14 |
| inc0 | I think we need to figure out better way to handle incoming containers | 21:14 |
| kfox1111 | I think we shoudl carry containers for stuff not maintained by the hub officially probably? | 21:14 |
| kfox1111 | of there is an official container, then maybe we should consider using it. | 21:15 |
| inc0 | well I don't want kolla to become poor mans dockerhub for unwanted images | 21:15 |
| kfox1111 | +1. I'm only talking about wanted images. :) | 21:15 |
| kfox1111 | hence the prometheus-openstack-exporter request. :) | 21:15 |
| kfox1111 | there isn't a container existing that I could find. | 21:15 |
| inc0 | that being said, we're building this image infra that will be hard to live without | 21:15 |
| inc0 | kfox1111: if you write it, I'll be happy to review:P | 21:16 |
| kfox1111 | yeah. | 21:16 |
| kfox1111 | k. | 21:16 |
| *** jascott1 has joined #openstack-kolla | 21:16 | |
| inc0 | but that's problem I wanted to move on PTG and didn't really get good answers | 21:16 |
| inc0 | non-openstack containers that is | 21:17 |
| jascott1 | gotta say personal hotspots on cellphones are pretty awesome | 21:17 |
| jascott1 | inc0 why are we not working from white shed? | 21:17 |
| inc0 | white shed? | 21:17 |
| inc0 | ahh | 21:18 |
| jascott1 | McMennamins | 21:18 |
| inc0 | yeah | 21:18 |
| inc0 | took me longer than should | 21:18 |
| jascott1 | :D | 21:18 |
| inc0 | I....have no good answer for you | 21:18 |
| jascott1 | haha theres not one ;) | 21:18 |
| inc0 | JF6 | 21:18 |
| jascott1 | idk txtgirlgeek might have a reason | 21:19 |
| inc0 | speaking of which this nick will soon become obsolete | 21:19 |
| kfox1111 | and... now the big fight with neutron.... | 21:20 |
| kfox1111 | man upgrading openstack is painful. :/ | 21:21 |
| inc0 | you should use kolla | 21:21 |
| inc0 | :P | 21:21 |
| inc0 | but yeah it won't be easy ever | 21:21 |
| inc0 | it will be easier | 21:21 |
| kfox1111 | you assume the problem is with orcestration. | 21:22 |
| kfox1111 | it is not. | 21:22 |
| rwellum | sdake: kfox1111 jascott1 take a look at: http://prnt.sc/ezamt2 please | 21:22 |
| jascott1 | seeing several of these: error storing status for daemon set | 21:22 |
| kfox1111 | its with broken db migration scripts, neutron code, undocumented config settings for proper tls termination... etc. :/ | 21:22 |
| kfox1111 | jascott1: weird... never seen that before. that in the describe pod? | 21:23 |
| jascott1 | no i grep'd all the logs for error | 21:24 |
| jascott1 | on the upside it seems the neutron related stuff is running | 21:25 |
| kfox1111 | the logs of what? kube-apiserver? | 21:25 |
| inc0 | kfox1111: that's why I want to make proper upgrade gate | 21:25 |
| jascott1 | var/log/containers | 21:25 |
| inc0 | just need dockerhub publisher for that | 21:26 |
| kfox1111 | inc0: +1. I got the gates for kolla-kubernetes set up already. just gotta get the code in now. | 21:26 |
| rwellum | http://prnt.sc/ezaupf - sorry previous one had glaring mistake | 21:26 |
| kfox1111 | inc0: though we did test all of this stuff on a test cloud. the problem was, the test cloud started one revision newer then our real one. | 21:26 |
| kfox1111 | inc0: so some of the db records from liberty were breaking stuff. :/ | 21:26 |
| jascott1 | no errors in any of the mariadb logs | 21:27 |
| jascott1 | other than pod crashing without telling reason | 21:27 |
| kfox1111 | jascott1: can you post the describe pod for it? | 21:27 |
| *** gianpietro has quit IRC | 21:28 | |
| *** gianpietro has joined #openstack-kolla | 21:28 | |
| jascott1 | kfox1111 http://paste.openstack.org/show/607528/ | 21:28 |
| kfox1111 | hmm... nothing very obvious there. | 21:30 |
| kfox1111 | I'd stick a 'command: [/bin/bash, -c, 'while true; sleep 100; done'] | 21:31 |
| kfox1111 | into the template and rerun it. | 21:31 |
| kfox1111 | then exec in and kube_start from the inside. | 21:31 |
| jascott1 | ah ok thanks | 21:32 |
| jascott1 | this is last log from mariadb log from /var/log/containers | 21:33 |
| jascott1 | {"log":"170421 21:29:48 mysqld_safe Starting mysqld daemon with databases from /var/lib/mysql/\n","stream":"stdout","time":"2017-04-21T21:29:48.416299696Z"} | 21:33 |
| *** shasha___ has joined #openstack-kolla | 21:35 | |
| kfox1111 | yeah. would run it in exec and see what happens. | 21:36 |
| kfox1111 | can then checkout the /var/lib/mysql dir, see if any proceses are running ,etc. | 21:37 |
| *** gianpietro has quit IRC | 21:38 | |
| *** salv-orlando has joined #openstack-kolla | 21:38 | |
| *** gianpietro has joined #openstack-kolla | 21:39 | |
| *** shasha___ has quit IRC | 21:39 | |
| *** kbyrne has joined #openstack-kolla | 21:39 | |
| *** gianpietro has quit IRC | 21:43 | |
| *** jascott1_ has joined #openstack-kolla | 21:49 | |
| *** salv-orl_ has joined #openstack-kolla | 21:49 | |
| *** jascott1 has quit IRC | 21:52 | |
| *** salv-orlando has quit IRC | 21:52 | |
| *** sbezverk_ has joined #openstack-kolla | 22:04 | |
| *** newmember has quit IRC | 22:08 | |
| *** sbezverk_ has quit IRC | 22:08 | |
| *** sbezverk_ has joined #openstack-kolla | 22:19 | |
| *** shasha___ has joined #openstack-kolla | 22:21 | |
| *** shasha___ has quit IRC | 22:26 | |
| *** sbezverk_ has quit IRC | 22:26 | |
| *** chas has joined #openstack-kolla | 22:28 | |
| *** chas has quit IRC | 22:32 | |
| *** gianpietro has joined #openstack-kolla | 22:35 | |
| jascott1_ | kfox1111 i have this | 22:40 |
| jascott1_ | spec: | 22:40 |
| jascott1_ | containers: | 22:40 |
| jascott1_ | - args: | 22:40 |
| jascott1_ | - -c | 22:40 |
| jascott1_ | - while true; sleep 100; done | 22:40 |
| jascott1_ | command: | 22:40 |
| jascott1_ | - /bin/bash | 22:40 |
| jascott1_ | sorry | 22:40 |
| jascott1_ | but still crashes | 22:40 |
| kfox1111 | that crashes? | 22:41 |
| kfox1111 | what storage backend? | 22:41 |
| kfox1111 | inc0: been thinking. what I really would like to see, is an imperitive, declarative workflow tool. | 22:42 |
| jascott1_ | yeah i destroyed whole setup, commented the maria install with helm and everything after, rebuilt, shelled in and edited the yaml and did helm install | 22:42 |
| kfox1111 | inc0: imperitive at each step of the workflow, but the step itself is declarative. | 22:42 |
| kfox1111 | jascott1_: how are you backing it? hostpath? ceph? other? | 22:43 |
| jascott1_ | hostpath afaik | 22:43 |
| kolla-slack | <inc0> I think k8s job is sth like that | 22:43 |
| jascott1_ | following the guide | 22:43 |
| kfox1111 | inc0: jobs are kind of the opposite. you define an impoeritive step, using a declarative description. | 22:44 |
| kfox1111 | jascott1_: thats really weird... | 22:45 |
| kfox1111 | there shouldn't be anything in that that shoudl cause a crash. | 22:45 |
| kfox1111 | and a hostpath volume's just a passthrough. | 22:45 |
| kfox1111 | something's very very wrong with docker/k8s I think. | 22:45 |
| jascott1_ | hmm | 22:46 |
| *** gianpietro has quit IRC | 22:46 | |
| jascott1_ | i would think it would run | 22:46 |
| *** salv-orl_ has quit IRC | 22:48 | |
| jascott1_ | some of these error: open /var/lib/docker/overlay/ab25faeb4cc02da51502c9f6d1fd01e40d56e6f78c2a7f83aad82a671fbd6a4e/lower-id: no such file or directory | 22:49 |
| jascott1_ | "Error removing mounted layer b34e896b28434769820660a7c7af9b1b98eb71e56fec852c724d473bb35c9a59: remove /var/lib/docker/overlay/8d28ebfcdc15fb8bc2341e182a8cdd73b634abd53a07bff9e0b5ffe8ef4382da/merged: device or resource busy" | 22:51 |
| *** unicell has quit IRC | 22:51 | |
| *** unicell has joined #openstack-kolla | 22:52 | |
| *** shasha___ has joined #openstack-kolla | 22:52 | |
| *** unicell has quit IRC | 22:54 | |
| *** rwallner has joined #openstack-kolla | 22:54 | |
| kfox1111 | heh. and one of the ugprade issues was kernel nic/firmware again. :/ really hard to debug. :/ | 22:54 |
| kfox1111 | rolling back the kernel fixed it. | 22:54 |
| kfox1111 | jascott1_: not sure if thats an error or not? may just be other containers using it too? | 22:55 |
| *** unicell has joined #openstack-kolla | 22:55 | |
| jascott1_ | no idea :( | 22:55 |
| inc0 | jascott1_: check which docker graph driver is used | 22:56 |
| inc0 | docker info | 22:56 |
| inc0 | nvm | 22:56 |
| inc0 | it's overlay | 22:56 |
| *** shasha___ has quit IRC | 22:57 | |
| kfox1111 | so, all said and done, mitaka->newton's going to be about 2.5 days I think. liberty->mitaka was only 1. | 22:57 |
| kfox1111 | :/ | 22:57 |
| *** rwallner has quit IRC | 22:58 | |
| *** masber has quit IRC | 23:02 | |
| *** gianpietro has joined #openstack-kolla | 23:03 | |
| *** masber has joined #openstack-kolla | 23:04 | |
| inc0 | kfox1111: so opposite would be ansible | 23:05 |
| inc0 | you don't say "install package", you say "package: state=installed" | 23:05 |
| kfox1111 | yeah. ansible's kind of both. | 23:06 |
| kfox1111 | ok. cloud upgrade disaster narrowly averted. :) | 23:07 |
| jascott1_ | whew | 23:08 |
| kfox1111 | neutron really nows how to tickle obscure firmeware bugs. :) | 23:09 |
| kfox1111 | this one was only a problem if packet was snated through a network node. fips to the outside world and back, and intervm traffic worked fine. | 23:10 |
| kfox1111 | think it was an inner packet checksum issue. | 23:10 |
| kfox1111 | jascott1_: can it run the most basic of pods without any volumes and while true;... ? | 23:11 |
| kfox1111 | with the same image? | 23:11 |
| jascott1_ | i will try | 23:11 |
| kfox1111 | maybe the image got corrupted somehow on the way into docker? | 23:11 |
| jascott1_ | its running some things ok like neutron stuff afaik | 23:12 |
| jascott1_ | well right now I stopped at mariadb but before it was | 23:12 |
| jascott1_ | i have done it severeal times so not a one time download corruption | 23:12 |
| kfox1111 | really weird. | 23:13 |
| kfox1111 | pulling from the hub? | 23:13 |
| *** goldyfruit has quit IRC | 23:14 | |
| jascott1_ | docker.io/kolla/centos-source-mariadb:4.0.0 | 23:14 |
| kfox1111 | k.. lets start from the ground up maybe. | 23:15 |
| kfox1111 | docker run -it --rm docker.io/kolla/centos-source-mariadb:4.0.0 /bin/bash | 23:15 |
| jascott1_ | that works | 23:25 |
| jascott1_ | runs anyway | 23:25 |
| jascott1_ | kfox1111 that works, tried that earlier but didnt know what to look for | 23:27 |
| jascott1_ | i dont think I properly got the command added to mariadb | 23:28 |
| jascott1_ | i could see it sometimes after deleting and applying | 23:28 |
| jascott1_ | oh its cause im an idiot | 23:29 |
| jascott1_ | the comand thing not the whole problem although that remains to be seen | 23:29 |
| *** dixiaoli has joined #openstack-kolla | 23:30 | |
| sbezverk | jascott1_: how about mariadb-init-element, has it finished sucessfully | 23:33 |
| sbezverk | ? | 23:33 |
| jascott1_ | sbezverk yes thakfully | 23:34 |
| jascott1_ | thankfully | 23:34 |
| jascott1_ | or it was | 23:35 |
| sbezverk | jascott1_: did you delete it after? | 23:35 |
| *** dixiaoli has quit IRC | 23:35 | |
| jascott1_ | i used helm to delete mariadb | 23:36 |
| sbezverk | jascott1_: so you use mariadb service chart? | 23:38 |
| *** shasha___ has joined #openstack-kolla | 23:38 | |
| jascott1_ | yes | 23:38 |
| jascott1_ | ok before i failed to modify mariadb pod to use sleep command | 23:39 |
| jascott1_ | now I got it modified but mariadb-init crashes cant connect to db | 23:39 |
| jascott1_ | sbezverk I have turned the guide into a playbook and am pretty far. i stop before mariadb install and do that manually to see whats going on | 23:40 |
| jascott1_ | helm install --debug kolla-kubernetes/helm/service/mariadb --namespace kolla --name mariadb --values ./cloud.yaml | 23:40 |
| *** rwellum has quit IRC | 23:41 | |
| jascott1_ | ya the command to make the actual db pod sleep seems to have made mariadb-init crash | 23:41 |
| jascott1_ | which is weird cause i would think init would come first | 23:42 |
| jascott1_ | but before init would finish without problem and then mariadb-0 would CrashLoop with not a lot of info | 23:43 |
| *** shasha___ has quit IRC | 23:43 | |
| jascott1_ | i thought init container ran before main container no? | 23:44 |
| sbezverk | jascott1: if init crashes then it is most likely something with storage | 23:46 |
| jascott1_ | good to know | 23:46 |
| sbezverk | if you use file system and not ceph cluster, make sure you clean up /var/lib/kolla/volumes | 23:47 |
| jascott1_ | im taking out command to see if init still crashe | 23:47 |
| jascott1_ | oh | 23:47 |
| jascott1_ | cleanup = rm -rf ? | 23:47 |
| sbezverk | jascott1_: yep | 23:47 |
| sbezverk | init will ALWAYS crash if it finds mariadb folder under volumes | 23:48 |
| *** yangyape_ has quit IRC | 23:49 | |
| jascott1_ | oh | 23:49 |
| jascott1_ | well without db sleep, this is output | 23:49 |
| jascott1_ | http://paste.openstack.org/show/607560/ | 23:49 |
| jascott1_ | also I removed volume | 23:49 |
| jascott1_ | ah i see what you mean thanks | 23:51 |
| *** eanylin has quit IRC | 23:53 | |
| jascott1_ | command: ["/bin/bash","-c","while true; sleep 100; done"] | 23:57 |
| jascott1_ | still CrashLoop | 23:57 |
| jascott1_ | but at least I know about removing the volumes | 23:57 |
| jascott1_ | does centos have /sbin/init ? | 23:58 |
| jascott1_ | ah it does | 23:59 |
| jascott1_ | thats what I used to use to sleep docker containerw | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!