| sbezverk | kfox1111: by some reason init container is missing mount!?!?! | 00:00 |
|---|---|---|
| kfox1111 | trunk or on a ps? | 00:00 |
| *** sayantani01 has quit IRC | 00:00 | |
| sbezverk | kfox1111: so the extra mount is not working :( | 00:00 |
| *** sdake has quit IRC | 00:01 | |
| kfox1111 | oh... we're not using config.json at all here.... that makes it easier. | 00:02 |
| *** lamt has quit IRC | 00:02 | |
| sbezverk | kfox1111: it is crazy http://paste.openstack.org/show/596526/ | 00:03 |
| kfox1111 | sbezverk: oh. hyou have deps twice. | 00:04 |
| kfox1111 | you don't need to define value dependencies for that socket. | 00:04 |
| kfox1111 | as there always is an explicit one. | 00:04 |
| sbezverk | kfox1111: that part I knew | 00:05 |
| sbezverk | because we needed one when deployed as a microservice | 00:06 |
| kfox1111 | yeah. | 00:06 |
| sbezverk | and then for consistency with the rest we added | 00:06 |
| kfox1111 | so should be removed from the service package. | 00:06 |
| sbezverk | another it shold not be a problem | 00:06 |
| sbezverk | the problem is why it does not render required volume | 00:06 |
| kfox1111 | it doesn't have mounts though, so it won't work. | 00:06 |
| sbezverk | exactly | 00:06 |
| kfox1111 | your saying the extra dep container should hve mounts too? | 00:07 |
| sbezverk | absolutely it should | 00:07 |
| kfox1111 | ah. ok. gotcha. sec... | 00:07 |
| sbezverk | otherwise it does not have access to socket | 00:07 |
| kfox1111 | oh... | 00:09 |
| kfox1111 | I see why. | 00:09 |
| kfox1111 | helm/kolla-common/templates/_common_dependency_container.yaml | 00:09 |
| kfox1111 | definitions of common_dependency_container_single and common_dependency_container_multi | 00:10 |
| kfox1111 | they don't add extraDepVolumeMounts to the child call if they exist. | 00:10 |
| sbezverk | kfox1111: that explains it | 00:11 |
| kfox1111 | each needs a: | 00:12 |
| kfox1111 | {{- if hasKey .Values "extraDepVolumeMounts" }} | 00:12 |
| kfox1111 | {{- $_ := set $c1 "extraDepVolumeMounts" .extraDepVolumeMounts }} | 00:12 |
| kfox1111 | {{- end }} | 00:12 |
| kfox1111 | the more confusing question, is how did this work in the gate ever. | 00:12 |
| kfox1111 | or did you add the dep to your own values.yaml? | 00:12 |
| kfox1111 | its in the openvswitch valules file. | 00:13 |
| kfox1111 | that is very unsettling.... :/ | 00:13 |
| kfox1111 | this should be broken I think... | 00:13 |
| sbezverk | kfox1111: one sec | 00:14 |
| sbezverk | both macros use common_dependency_container_common | 00:14 |
| sbezverk | where there is a check for extra mount | 00:14 |
| sbezverk | kfox1111: Am I missing something? | 00:15 |
| kfox1111 | yeah. but common_dependency_container_multi and common_dependency_container_single builds the '.' environment for that function. | 00:15 |
| kfox1111 | those 3 lines above need to be added to both, to pass it on through. | 00:15 |
| kfox1111 | oh. wait... | 00:15 |
| kfox1111 | I'm totally missreading it. | 00:16 |
| kfox1111 | its building args to kolla_val_get_raw, not common_dependency_container_common | 00:16 |
| kfox1111 | common_dependency_container_common is getting . from the multi/single macros. | 00:16 |
| kfox1111 | disreguard what I said... :/ | 00:17 |
| kfox1111 | ok. lets walk through it again... | 00:17 |
| kfox1111 | helm/microservice/openvswitch-vswitchd-daemonset/templates/openvswitch-vswitchd-daemonset.yaml defines openvswitch_vswitchd_extra_dep_volume_mounts | 00:17 |
| kfox1111 | sets {{- $extraDepVolumeMounts := "openvswitch_vswitchd_extra_dep_volume_mounts" }} | 00:17 |
| kfox1111 | adds $extraDepVolumeMounts as "extraDepVolumeMounts" to $env | 00:18 |
| sbezverk | right | 00:18 |
| kfox1111 | then calls common_dependency_container_multi with $env. | 00:18 |
| *** huzhengchuan has joined #openstack-kolla | 00:18 | |
| kfox1111 | which passes it directly to common_dependency_container_common | 00:18 |
| kfox1111 | then checks for {{- if .extraDepVolumeMounts }} | 00:19 |
| kfox1111 | I see no with's there in that macro. | 00:20 |
| sbezverk | you see it does not even render volumes: | 00:20 |
| sbezverk | no that if is broken somehow | 00:20 |
| kfox1111 | yeah. its acting like .extraDepVolumeMounts is false. | 00:20 |
| kfox1111 | all the variable names match up... | 00:22 |
| kfox1111 | this should be working like common_api_apache deployment. | 00:23 |
| kfox1111 | which is working. | 00:23 |
| sbezverk | any chance we lose some vars when call include "common_dependency_container_common" . | 00:23 |
| kfox1111 | its also working in the gate. | 00:23 |
| sbezverk | I mean '.' instead of $env | 00:24 |
| kfox1111 | . comes from the parent, | 00:24 |
| kfox1111 | which was called with $env. | 00:24 |
| sbezverk | you see {{- if $localVals.kubernetes_entrypoint }} | 00:24 |
| sbezverk | works fine | 00:25 |
| kfox1111 | I don't nkow how you can be seeing that. :/ | 00:26 |
| sbezverk | kfox1111: if it was not working then extra dependency container would not show up | 00:27 |
| kfox1111 | I don't follow. | 00:28 |
| kfox1111 | trunk openvswitch service package has the dep in there the same as yours. | 00:28 |
| *** eaguilar has joined #openstack-kolla | 00:28 | |
| kfox1111 | so should have both init containers. | 00:28 |
| kfox1111 | somehow its working though. | 00:28 |
| kfox1111 | I think... | 00:29 |
| kfox1111 | yeah. cause https://review.openstack.org/#/c/424921/ passed after that merged... | 00:29 |
| sbezverk | kfox1111: I should be able to see what is stored in '.' | 00:31 |
| sbezverk | rebuilding microservices and service | 00:31 |
| kfox1111 | yeah. can just dump it out. | 00:31 |
| kfox1111 | foo: {{ . }} | 00:31 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 00:34 |
| kfox1111 | arg.. committed a while ago but didn't review. :/ | 00:34 |
| kfox1111 | there's 10 minutes of confuse debugging down the drain. :/ | 00:34 |
| *** inc0 has quit IRC | 00:35 | |
| kfox1111 | yay. trunk builds back to almost sort of kind of working: http://logs.openstack.org/50/418550/58/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/094a42c/console.html | 00:35 |
| kfox1111 | just cinder left. | 00:35 |
| *** duonghq has joined #openstack-kolla | 00:38 | |
| duonghq | morning guys | 00:38 |
| kfox1111 | duonghq: morning. | 00:39 |
| sbezverk | duonghq: morning | 00:39 |
| kfox1111 | sbezverk: did you build those by hand, or by tools/helm_build_all.sh? | 00:39 |
| duonghq | hi kfox1111 sbezverk | 00:39 |
| kfox1111 | maybe you got an earlier version somehow? | 00:39 |
| kfox1111 | or... if you have the package that's doing that, | 00:39 |
| kfox1111 | could you extract the tarball and look at the deps template? | 00:39 |
| sbezverk | kfox1111: yes and what is interesting when I dump from microservice dry run I do see extravolume | 00:40 |
| kfox1111 | waa? | 00:40 |
| sbezverk | when I do the same from service level it does not | 00:40 |
| kfox1111 | but, when you load it, then get daemonset that you put in pastebin, its not there? | 00:40 |
| *** portdirect is now known as portdirect_travl | 00:40 | |
| sbezverk | kfox1111: I explicitely specify kubernetes_entrypoint in microservice | 00:40 |
| kfox1111 | with dryrun? | 00:40 |
| sbezverk | kfox1111: http://paste.openstack.org/show/596529/ | 00:41 |
| sbezverk | this is for microservice | 00:42 |
| *** sayantani01 has joined #openstack-kolla | 00:43 | |
| sbezverk | kfox1111: here is for service http://paste.openstack.org/show/596530/ | 00:43 |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Fix RabbitMQ server error in bifrost https://review.openstack.org/424384 | 00:45 |
| kfox1111 | this feels like a bug in helm. :/ | 00:45 |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Fix MySQL DB error in bifrost container https://review.openstack.org/424391 | 00:45 |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Update Bifrost playbooks https://review.openstack.org/425435 | 00:46 |
| kfox1111 | uhhh........ | 00:48 |
| *** tovin07 has joined #openstack-kolla | 00:49 | |
| *** tovin07_ has joined #openstack-kolla | 00:49 | |
| kfox1111 | oh... | 00:50 |
| kfox1111 | this is hard to parse since there is more then one template in the logs. :/ | 00:51 |
| *** dave-mcc_ has joined #openstack-kolla | 00:51 | |
| *** krtaylor has joined #openstack-kolla | 00:51 | |
| kfox1111 | ok. ran it a few times. | 00:52 |
| kfox1111 | I'm seeing the volume mounts consistently on both service package and microservice package. | 00:53 |
| kfox1111 | I was sometimes loking at the ovsdb one and that was throwing me for a bit. | 00:53 |
| *** dave-mccowan has quit IRC | 00:53 | |
| kfox1111 | sbezverk: do you staill have access to that service package? | 00:54 |
| kfox1111 | can you post it somehwere? | 00:54 |
| kfox1111 | wow... all the other cores are gone. :/ | 00:55 |
| sbezverk | kfox1111: really sorry, it was my fault :( | 00:55 |
| sbezverk | it turned out there were two helm repo | 00:55 |
| sbezverk | so the service package was getting built from older repo | 00:56 |
| sbezverk | after cleaning everything up it started working | 00:57 |
| *** cuongnv has joined #openstack-kolla | 00:57 | |
| kfox1111 | ok. cool. | 00:58 |
| kfox1111 | tools/helm_build_all.sh. :) | 00:58 |
| kfox1111 | glad you found the issue though. | 00:59 |
| kfox1111 | oh darn... | 01:00 |
| duonghq | does the portdirect workaround of docker 1.13.0 issue work for you? | 01:01 |
| kfox1111 | that appctl command is being way to clever... | 01:01 |
| duonghq | the tiler problem | 01:01 |
| kfox1111 | its looking at the pid, deciding its not in the pid namespace, then killing the file. :/ | 01:01 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (default roles) https://review.openstack.org/398684 | 01:01 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible: Specify 'become' for only neccesary tasks (all other roles) https://review.openstack.org/398685 | 01:02 |
| kfox1111 | duonghq: who is that question for? | 01:02 |
| sbezverk | later gents, talk to you tomorrow.. | 01:02 |
| portdirect_travl | Works for me, but I'm definitely not trustworthy on that front :) | 01:03 |
| duonghq | kfox1111, you and sbezverk | 01:03 |
| duonghq | sorry, I forgot mention | 01:03 |
| kfox1111 | duonghq: I dont have the issue as I avoid docker 1.13 alltogether. I trust redhat's docker more. | 01:03 |
| *** jpstnh has quit IRC | 01:03 | |
| portdirect_travl | Think technopolis has been pointing others people to that image as well :) and :/ | 01:03 |
| duonghq | kfox1111, ok | 01:03 |
| duonghq | portdirect_travl, your image still crashbackloop in my halcyon setup, cannot figure out why | 01:04 |
| kfox1111 | portdirect_travl: can you please review https://review.openstack.org/#/c/424921/ real quick. | 01:04 |
| kfox1111 | and if you have another minute, https://review.openstack.org/#/c/424368/ . | 01:05 |
| portdirect_travl | duonghq: is your networking up? Most of the time I see tiller misbehaving DNS is down | 01:05 |
| kfox1111 | both are parts to getting the gate stable again. | 01:05 |
| portdirect_travl | On phone kfox, can you give me the rundown | 01:06 |
| portdirect_travl | (Tiny screens and gerrit...) | 01:06 |
| *** bjolo_ has joined #openstack-kolla | 01:06 | |
| kfox1111 | portdirect_travl: one builds cache tarballs that we will be able to download in the gate instead of from the docker hub. for kolla nad kubernetes/helm containers. | 01:06 |
| kfox1111 | so more stable access to them. | 01:06 |
| duonghq | portdirect_travl, http://paste.openstack.org/show/596532/ | 01:06 |
| kfox1111 | the other just adds a bit more error checking. | 01:06 |
| *** huikang has joined #openstack-kolla | 01:06 | |
| portdirect_travl | Roger, will check as best I can | 01:07 |
| kfox1111 | thx. | 01:07 |
| kfox1111 | the first one is more important, but hte second is a very easy review. | 01:08 |
| *** sacharya has joined #openstack-kolla | 01:08 | |
| portdirect_travl | Duonghq, that's your k8s dashboard in the crashloop not tiller | 01:08 |
| duonghq | portdirect_travl, sorry, misreading this | 01:08 |
| portdirect_travl | Though that's not a good sign | 01:08 |
| duonghq | portdirect_travl, moment, I'm searching your workaround | 01:09 |
| portdirect_travl | kfox1111: could you see if you could help duonghq ? | 01:09 |
| kfox1111 | yeah. I have a little bit more time. | 01:09 |
| kfox1111 | whats kubectl get pods --all-namespaces show? | 01:10 |
| portdirect_travl | nice multitasking on a 5 inch screen ain't fun :) | 01:11 |
| kfox1111 | I get that. :) | 01:11 |
| duonghq | kfox1111, http://paste.openstack.org/show/596532/ <- this is the output | 01:11 |
| kfox1111 | k. | 01:12 |
| kfox1111 | and: kubectl describe pod tiller-deploy-3299276078-nm5q7 --namespace kube-system | 01:12 |
| duonghq | kfox1111, I applied portdirect_travl image, and get follow output: | 01:13 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 01:14 |
| duonghq | http://paste.openstack.org/show/596533/ http://paste.openstack.org/show/596534/ | 01:14 |
| portdirect_travl | kfox1111: am i'm being mental or does this do the opposite of what the commit message say? https://review.openstack.org/#/c/424368/ | 01:15 |
| kfox1111 | hmm... | 01:15 |
| kfox1111 | portdirect_travl: its a bit confusing to write about, because bash does flags backwards. | 01:17 |
| portdirect_travl | I'm being mental | 01:17 |
| kfox1111 | -e adds the flag, +e removes it. :/ | 01:17 |
| portdirect_travl | yes | 01:17 |
| portdirect_travl | oh well least i embarrassed myself on irc and not gerrit :) | 01:17 |
| kfox1111 | duonghq: thats a strange output. it looks like it pulled the image ok, but then is then shooting it. | 01:18 |
| kfox1111 | portdirect_travl: no worries. That trips me up alot too. | 01:18 |
| portdirect_travl | that dashboard crashloop makes me think that the pods cannot talk to the api server | 01:18 |
| duonghq | kfox1111, yes | 01:19 |
| kfox1111 | yeah. | 01:19 |
| kfox1111 | how about: kubectl get pods --all-namespaces -o wide | 01:19 |
| *** zhubingbing has joined #openstack-kolla | 01:19 | |
| kfox1111 | lets see if there is a pattern to the nodes landed on. | 01:19 |
| *** dave-mccowan has joined #openstack-kolla | 01:20 | |
| duonghq | kfox1111, http://paste.openstack.org/show/596535/ | 01:20 |
| kfox1111 | portdirect_travl: thanks for the reviews. | 01:20 |
| kfox1111 | so, a slight pattern. | 01:21 |
| kfox1111 | there are only 3 pods using non net=host. | 01:21 |
| kfox1111 | of those, 1 is wokring, and on the head node it apears, | 01:21 |
| kfox1111 | and 2 are not, and on a different node. | 01:22 |
| kfox1111 | I would guess weave is not working properly. | 01:22 |
| *** dave-mcc_ has quit IRC | 01:22 | |
| duonghq | I saw weave on .14 is restart once | 01:22 |
| kfox1111 | it restarted on 13 too. | 01:23 |
| kfox1111 | that may be normal. | 01:23 |
| kfox1111 | I know nothing about weave though. | 01:23 |
| kfox1111 | been using canal. | 01:23 |
| kfox1111 | portdirect_travl: what are you using? | 01:23 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Capture lists of containers by type and cache containers. https://review.openstack.org/424921 | 01:23 |
| kfox1111 | (for testing) | 01:23 |
| duonghq | but there is not any non net=host pod on node 13, I guess that if any, this'll get trouble too, not sure how to reschedule pod to run on other node | 01:24 |
| portdirect_travl | calico | 01:25 |
| kfox1111 | portdirect_travl: k. thx. | 01:25 |
| kfox1111 | duonghq: maybe try calico instead of weave? | 01:25 |
| kfox1111 | thought that may require burning down the cluster to switch. | 01:26 |
| portdirect_travl | used to hate it - then one day (about 9 months ago) it started working and I've never thought about it since really | 01:26 |
| duonghq | kfox1111, sure | 01:26 |
| duonghq | portdirect_travl, which calico version are you using? | 01:26 |
| portdirect_travl | v2.0 | 01:26 |
| duonghq | ack | 01:26 |
| portdirect_travl | if you are running halcyon - you can chance one line in the playbook to use it | 01:27 |
| duonghq | portdirect_travl, saw it | 01:27 |
| kfox1111 | portdirect_travl: oh. pure calico? not the canal variant? | 01:27 |
| kfox1111 | any thoughts about which one to use? | 01:27 |
| portdirect_travl | so canals just calico's security groups with flannel | 01:28 |
| *** Syffs has quit IRC | 01:29 | |
| portdirect_travl | so its the choice between tunnels and routing really | 01:29 |
| duonghq | with my network, take down and bring up again the cluster take awhile to be done | 01:29 |
| portdirect_travl | if you can I'd go for plain calico | 01:29 |
| *** bpattewar has quit IRC | 01:29 | |
| portdirect_travl | but untill you have a lot of nodes i dont think it makes much of a difference | 01:30 |
| kfox1111 | k. | 01:30 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Apply neutron database migration https://review.openstack.org/407840 | 01:31 |
| *** sdake has joined #openstack-kolla | 01:33 | |
| kfox1111 | there we go... | 01:34 |
| kfox1111 | maybe a fix for the openvswitch startup thing finally. | 01:34 |
| *** sdake_ has joined #openstack-kolla | 01:35 | |
| *** sdake has quit IRC | 01:38 | |
| *** hfu has joined #openstack-kolla | 01:40 | |
| *** huikang has quit IRC | 01:41 | |
| *** huikang has joined #openstack-kolla | 01:41 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible: Apply neutron database migration https://review.openstack.org/407922 | 01:42 |
| *** sdake_ has quit IRC | 01:43 | |
| kfox1111 | gota head out. l8r all. | 01:44 |
| duonghq | kfox1111, cya | 01:44 |
| *** eaguilar has quit IRC | 01:46 | |
| *** huikang has quit IRC | 01:47 | |
| *** sdake has joined #openstack-kolla | 01:49 | |
| duonghq | sdake, sup | 01:49 |
| *** Jeffrey4l_ has quit IRC | 01:49 | |
| *** Jeffrey4l has joined #openstack-kolla | 01:49 | |
| *** sdake_ has joined #openstack-kolla | 01:50 | |
| *** erlon has quit IRC | 01:50 | |
| *** sdake has quit IRC | 01:54 | |
| *** zhubingbing has quit IRC | 01:54 | |
| *** dave-mcc_ has joined #openstack-kolla | 01:57 | |
| *** dave-mccowan has quit IRC | 01:59 | |
| *** zhubingbing has joined #openstack-kolla | 02:02 | |
| *** sacharya_ has joined #openstack-kolla | 02:07 | |
| *** sacharya has quit IRC | 02:08 | |
| *** sdake_ has quit IRC | 02:09 | |
| *** huzhengchuan has quit IRC | 02:09 | |
| *** unicell has quit IRC | 02:10 | |
| *** sdake has joined #openstack-kolla | 02:11 | |
| *** rwellum has quit IRC | 02:17 | |
| *** Mech422 has quit IRC | 02:42 | |
| *** Jeffrey4l has quit IRC | 02:45 | |
| *** sacharya_ has quit IRC | 02:50 | |
| *** sacharya has joined #openstack-kolla | 02:50 | |
| *** sayantani01 has quit IRC | 02:50 | |
| *** markmcclain has quit IRC | 02:53 | |
| *** sdake has quit IRC | 02:58 | |
| *** markmcclain has joined #openstack-kolla | 02:58 | |
| *** prameswar has joined #openstack-kolla | 03:03 | |
| *** lrensing has joined #openstack-kolla | 03:09 | |
| *** qiliang27 has quit IRC | 03:13 | |
| *** lamt has joined #openstack-kolla | 03:17 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for kuryr https://review.openstack.org/424161 | 03:17 |
| *** lrensing has quit IRC | 03:18 | |
| *** eaguilar has joined #openstack-kolla | 03:18 | |
| *** eaguilar has quit IRC | 03:19 | |
| *** eaguilar has joined #openstack-kolla | 03:20 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for cloudkitty https://review.openstack.org/422516 | 03:22 |
| *** newmember has quit IRC | 03:23 | |
| *** rushil has quit IRC | 03:23 | |
| *** bjolo_ has quit IRC | 03:31 | |
| *** zhubingbing has quit IRC | 03:32 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ironic https://review.openstack.org/423901 | 03:40 |
| *** rushil has joined #openstack-kolla | 03:41 | |
| *** sdake has joined #openstack-kolla | 03:46 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for trove https://review.openstack.org/423990 | 04:04 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ironic https://review.openstack.org/423901 | 04:07 |
| *** sdake_ has joined #openstack-kolla | 04:08 | |
| *** sdake has quit IRC | 04:09 | |
| *** williamcaban has joined #openstack-kolla | 04:27 | |
| *** sdake has joined #openstack-kolla | 04:31 | |
| *** sdake_ has quit IRC | 04:32 | |
| *** dave-mcc_ has quit IRC | 04:32 | |
| *** eaguilar has quit IRC | 04:34 | |
| *** cristicalin has joined #openstack-kolla | 04:42 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ironic https://review.openstack.org/423901 | 04:43 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for ironic https://review.openstack.org/423901 | 04:45 |
| *** sdake has quit IRC | 04:49 | |
| *** mdnadeem has joined #openstack-kolla | 04:50 | |
| *** sayantani01 has joined #openstack-kolla | 04:57 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for influxdb https://review.openstack.org/424159 | 04:59 |
| *** britthouser1 has joined #openstack-kolla | 05:03 | |
| *** britthouser has quit IRC | 05:03 | |
| *** msimonin1 has joined #openstack-kolla | 05:04 | |
| *** zhubingbing has joined #openstack-kolla | 05:05 | |
| *** msimonin has quit IRC | 05:06 | |
| *** cristicalin has quit IRC | 05:08 | |
| *** zhubingbing has quit IRC | 05:09 | |
| *** zhubingbing has joined #openstack-kolla | 05:10 | |
| *** zhubingbing has quit IRC | 05:12 | |
| *** wcb has joined #openstack-kolla | 05:12 | |
| *** sacharya has quit IRC | 05:14 | |
| *** williamcaban has quit IRC | 05:14 | |
| *** sacharya has joined #openstack-kolla | 05:14 | |
| *** sacharya has quit IRC | 05:15 | |
| *** sacharya has joined #openstack-kolla | 05:15 | |
| *** sacharya has quit IRC | 05:20 | |
| *** sdake has joined #openstack-kolla | 05:25 | |
| *** unicell has joined #openstack-kolla | 05:28 | |
| *** cristicalin has joined #openstack-kolla | 05:30 | |
| openstackgerrit | Cuong Nguyen proposed openstack/kolla: Update elastic repo to https https://review.openstack.org/425539 | 05:32 |
| *** mdnadeem has quit IRC | 05:36 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: [WIP]Optimize reconfiguration for haproxy https://review.openstack.org/424156 | 05:38 |
| *** srwilkers_ has quit IRC | 05:39 | |
| *** cristicalin has quit IRC | 05:40 | |
| *** sdake has quit IRC | 05:45 | |
| *** mdnadeem has joined #openstack-kolla | 05:48 | |
| *** unicell has quit IRC | 05:53 | |
| *** unicell has joined #openstack-kolla | 05:54 | |
| *** wcb has quit IRC | 06:00 | |
| *** mdnadeem has quit IRC | 06:02 | |
| *** mdnadeem has joined #openstack-kolla | 06:02 | |
| *** rushil has quit IRC | 06:04 | |
| *** sdake has joined #openstack-kolla | 06:06 | |
| *** sacharya has joined #openstack-kolla | 06:16 | |
| *** sdake has quit IRC | 06:16 | |
| *** sdake has joined #openstack-kolla | 06:16 | |
| *** sdake_ has joined #openstack-kolla | 06:21 | |
| *** sacharya has quit IRC | 06:22 | |
| *** sdake has quit IRC | 06:23 | |
| *** lamt has quit IRC | 06:27 | |
| *** sdake_ has quit IRC | 06:27 | |
| *** saneax has joined #openstack-kolla | 06:34 | |
| *** zioproto has joined #openstack-kolla | 06:35 | |
| *** zioproto has quit IRC | 06:41 | |
| *** saneax is now known as saneax-_-|AFK | 06:46 | |
| *** prameswar has quit IRC | 06:59 | |
| *** imcsk8 has quit IRC | 07:02 | |
| *** imcsk8 has joined #openstack-kolla | 07:04 | |
| *** jmccarthy has joined #openstack-kolla | 07:07 | |
| *** bmace_ has quit IRC | 07:07 | |
| *** bmace_ has joined #openstack-kolla | 07:07 | |
| *** sayantani01 has quit IRC | 07:15 | |
| *** jmccarthy has quit IRC | 07:15 | |
| openstackgerrit | Cuong Nguyen proposed openstack/kolla: Update elastic repo to https https://review.openstack.org/425539 | 07:21 |
| *** zhubingbing has joined #openstack-kolla | 07:32 | |
| *** msimonin1 has quit IRC | 07:41 | |
| *** duonghq has quit IRC | 07:51 | |
| *** hfu has quit IRC | 07:55 | |
| *** hfu has joined #openstack-kolla | 07:55 | |
| *** hfu has quit IRC | 07:55 | |
| *** hfu has joined #openstack-kolla | 07:56 | |
| *** hfu has quit IRC | 07:56 | |
| *** hfu has joined #openstack-kolla | 07:57 | |
| *** hfu has quit IRC | 07:57 | |
| *** hfu has joined #openstack-kolla | 07:57 | |
| *** hfu has quit IRC | 07:58 | |
| *** hfu has joined #openstack-kolla | 07:58 | |
| *** hfu has quit IRC | 07:59 | |
| *** tovin07 has quit IRC | 08:02 | |
| *** tovin07_ has quit IRC | 08:05 | |
| *** hieulq has quit IRC | 08:09 | |
| *** zioproto has joined #openstack-kolla | 08:12 | |
| *** zioproto1 has joined #openstack-kolla | 08:16 | |
| *** zioproto has quit IRC | 08:20 | |
| *** zioproto1 has quit IRC | 08:27 | |
| *** zioproto has joined #openstack-kolla | 08:30 | |
| *** zioproto has quit IRC | 08:31 | |
| *** zioproto has joined #openstack-kolla | 08:31 | |
| *** britthouser1 has quit IRC | 08:32 | |
| *** msimonin has joined #openstack-kolla | 08:33 | |
| *** britthouser has joined #openstack-kolla | 08:34 | |
| *** egonzalez has joined #openstack-kolla | 08:37 | |
| *** simon-AS559 has joined #openstack-kolla | 08:38 | |
| *** britthouser2 has joined #openstack-kolla | 08:44 | |
| *** britthouser has quit IRC | 08:44 | |
| *** gfidente has joined #openstack-kolla | 08:48 | |
| *** gfidente has quit IRC | 08:48 | |
| *** gfidente has joined #openstack-kolla | 08:48 | |
| *** unicell has quit IRC | 08:54 | |
| *** shardy has joined #openstack-kolla | 09:02 | |
| *** matrohon__ has joined #openstack-kolla | 09:03 | |
| *** pomac has joined #openstack-kolla | 09:12 | |
| *** athomas has joined #openstack-kolla | 09:21 | |
| *** sambetts|afk is now known as sambetts | 09:26 | |
| *** mgoddard has joined #openstack-kolla | 09:27 | |
| pomac | Any comments on: https://bugs.launchpad.net/kolla-ansible/+bug/1659507 - I'm still trying to figure out why my gerrit account doesn't work | 09:33 |
| openstack | Launchpad bug 1659507 in kolla-ansible "[RFC] push libvirt-guests to enable compute node vm resumption" [Undecided,New] | 09:33 |
| pomac | rotfl | 09:33 |
| pomac | ehm.. | 09:33 |
| *** Serlex has joined #openstack-kolla | 09:35 | |
| *** shardy has quit IRC | 09:42 | |
| *** shardy has joined #openstack-kolla | 09:42 | |
| *** rmart04 has joined #openstack-kolla | 09:46 | |
| egonzalez | pomac, have you changed your community profile to be "Foundation Member" instead of "Community Member"? | 09:46 |
| pomac | egonzalez: most likely not - how do you do that? | 09:47 |
| pomac | egonzalez: i registered a long time ago but never completed the process, so everything is in the same state as 'eons ago' | 09:48 |
| egonzalez | https://www.openstack.org/profile/ | 09:48 |
| pomac | egonzalez: oh, I'm being signed in via launchpad in gerrit currently | 09:49 |
| egonzalez | yup, but also need to add you community profile as foundation, if not gerrit can't retrieve you info | 09:49 |
| pomac | ah | 09:50 |
| *** matrohon__ has quit IRC | 09:53 | |
| pomac | egonzalez: i would prefer to be a community member, but since i don' | 09:54 |
| pomac | t have anything today, maybe thats enough | 09:54 |
| *** matrohon__ has joined #openstack-kolla | 09:55 | |
| pomac | egonzalez: still can't store contact information... | 10:00 |
| *** cuongnv has quit IRC | 10:01 | |
| egonzalez | signed https://review.openstack.org/#/settings/agreements ? | 10:01 |
| egonzalez | OpenStack Individual Contributor License Agreement | 10:01 |
| pomac | egonzalez: So, when i try to do that or update my "contact information" i get the same error | 10:01 |
| pomac | "Server Error | 10:02 |
| pomac | Cannot store contact information" | 10:02 |
| pomac | egonzalez: i talked some with Khai Do yesterday and he couldn't see anything wrong at the time | 10:02 |
| egonzalez | is your primary email address in gerrit the same aas primary in openstack profile? | 10:03 |
| pomac | yes, but the launchpad one is still toating a old one - can't seem to get it removed | 10:04 |
| pomac | egonzalez: (it only shows up when i authenitcate, can't see it anywhere else on launchpad | 10:05 |
| pomac | heyyy | 10:06 |
| pomac | Logged out of launcpad, ubuntuone and god knows what else, the account looks better npw | 10:07 |
| pomac | now | 10:07 |
| egonzalez | this auth system with 3 differents accounts is madness for newcomers | 10:08 |
| pomac | egonzalez: still can't sign or set my address tough | 10:09 |
| pomac | egonzalez: (i'm much more used to being on the otherside of gerrit.... ) | 10:09 |
| *** simon-AS5591 has joined #openstack-kolla | 10:15 | |
| pomac | egonzalez: (tried again using incognitomode - so nothing esle should be problematic) | 10:18 |
| egonzalez | pomac: worked? | 10:18 |
| *** simon-AS559 has quit IRC | 10:18 | |
| pomac | egonzalez: nope | 10:19 |
| *** gema has quit IRC | 10:21 | |
| pomac | egonzalez: i bet that there is a exception somewhere in the log that would explain this ;) | 10:21 |
| pomac | lunchtime, afk | 10:22 |
| *** pbourke has quit IRC | 10:30 | |
| *** pbourke has joined #openstack-kolla | 10:30 | |
| *** zioproto has quit IRC | 10:30 | |
| *** zioproto has joined #openstack-kolla | 10:31 | |
| *** shardy has quit IRC | 10:51 | |
| *** shardy has joined #openstack-kolla | 10:52 | |
| *** eaguilar has joined #openstack-kolla | 11:12 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for influxdb https://review.openstack.org/424159 | 11:13 |
| *** openstackgerrit has quit IRC | 11:17 | |
| *** matrohon__ has quit IRC | 11:29 | |
| *** shardy has quit IRC | 11:32 | |
| *** shardy has joined #openstack-kolla | 11:34 | |
| *** eaguilar has quit IRC | 11:37 | |
| *** eaguilar has joined #openstack-kolla | 11:49 | |
| *** openstackgerrit has joined #openstack-kolla | 11:59 | |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for collectd https://review.openstack.org/424133 | 11:59 |
| *** pc_m has quit IRC | 12:01 | |
| openstackgerrit | Merged openstack/kolla-ansible: Add chrony ansible role https://review.openstack.org/399355 | 12:03 |
| *** pc_m has joined #openstack-kolla | 12:10 | |
| *** shardy is now known as shardy_lunch | 12:10 | |
| zioproto | egonzalez, how to I find the blueprint you mention here ? https://review.openstack.org/#/c/425263/ | 12:11 |
| zioproto | I was able to run a multimode deployment just hacking here and there some become: true | 12:11 |
| egonzalez | zioproto: https://blueprints.launchpad.net/kolla-ansible/+spec/ansible-specific-task-become | 12:14 |
| zioproto | thanks | 12:14 |
| egonzalez | related PS https://review.openstack.org/#/c/398682/5 | 12:14 |
| zioproto | There is anywhere described the architecture deployed with the Kolla multimode ? | 12:16 |
| zioproto | I see the HA proxy is deployed on the network node | 12:16 |
| zioproto | but if you have only one | 12:16 |
| zioproto | basically there is only 1 ha proxy, right ? | 12:17 |
| zioproto | there is no VRRP involved in this case i guess | 12:17 |
| egonzalez | yup, i'm using allinone with haproxy | 12:17 |
| zioproto | ok, I have finished my first deployment | 12:17 |
| *** eaguilar has quit IRC | 12:17 | |
| zioproto | so, we want to deploy Kolla by steps | 12:17 |
| zioproto | as first step we would like to deploy just the mariadb part | 12:17 |
| *** eaguilar has joined #openstack-kolla | 12:18 | |
| zioproto | but then we need to change our inventory to deploy somewhere ha proxy | 12:18 |
| egonzalez | mariadb will need haproxy | 12:18 |
| zioproto | yes I see that | 12:18 |
| zioproto | how much is a operator free to hack the inventory to change the deployment ? | 12:18 |
| egonzalez | as the database VIP is bind there | 12:18 |
| zioproto | well, is not really a VIP, is the IP of HAProxy | 12:18 |
| zioproto | it is not a VIP elected with VRRP if you have only 1 haproxy | 12:19 |
| zioproto | I found confusing this name | 12:19 |
| zioproto | do the Kolla playbook support to have more than 1 HA proxy ? | 12:19 |
| zioproto | I will go to read that blueprint now | 12:19 |
| egonzalez | you know, if have more than one will become a VIP ;) | 12:19 |
| zioproto | right :) | 12:19 |
| zioproto | I think HA proxy should be added to this because you call sysctl in the playbook: https://review.openstack.org/#/c/398682/ | 12:23 |
| zioproto | should I propose a patchset 6 on this review ? | 12:23 |
| zioproto | what is the IRC nick of Duong ? | 12:23 |
| *** eaguilar has quit IRC | 12:25 | |
| *** erlon has joined #openstack-kolla | 12:27 | |
| egonzalez | zioproto: is added https://review.openstack.org/#/c/398682/5/ansible/roles/haproxy/tasks/config.yml | 12:28 |
| egonzalez | but no in the list at commit description | 12:28 |
| zioproto | right | 12:28 |
| zioproto | I will comment | 12:28 |
| zioproto | on the review | 12:29 |
| *** matrohon__ has joined #openstack-kolla | 12:31 | |
| egonzalez | is duongh IIRC, but is not around here at this time. Use to be here in 3-4h before and later this time | 12:32 |
| egonzalez | anybody have an idea why this PS is not being tested by CI infra? https://review.openstack.org/#/c/405215/7 | 12:42 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: [WIP]Optimize reconfiguration for tempest https://review.openstack.org/424188 | 12:44 |
| *** lamt has joined #openstack-kolla | 12:49 | |
| *** lamt has quit IRC | 12:53 | |
| *** eaguilar has joined #openstack-kolla | 12:54 | |
| *** l4yerffeJ__ has joined #openstack-kolla | 12:56 | |
| *** l4yerffeJ_ has quit IRC | 12:56 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade https://review.openstack.org/425446 | 13:03 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla-ansible: Optimize reconfiguration for aodh https://review.openstack.org/423670 | 13:03 |
| mliima | morning guys | 13:04 |
| *** shardy_lunch is now known as shardy | 13:04 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade https://review.openstack.org/425446 | 13:05 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade https://review.openstack.org/425446 | 13:11 |
| *** eaguilar has quit IRC | 13:11 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for vmtp https://review.openstack.org/424194 | 13:22 |
| *** rwellum has joined #openstack-kolla | 13:31 | |
| *** srwilkers has joined #openstack-kolla | 13:34 | |
| *** papacz has quit IRC | 13:34 | |
| *** Jeffrey4l has joined #openstack-kolla | 13:35 | |
| *** papacz has joined #openstack-kolla | 13:36 | |
| *** sdake has joined #openstack-kolla | 13:38 | |
| sdake | morning peeps | 13:39 |
| egonzalez | morning sdake | 13:40 |
| sdake | sup egonzalez | 13:43 |
| *** mgiles has joined #openstack-kolla | 13:45 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for vmtp https://review.openstack.org/424194 | 13:45 |
| Jeffrey4l | morning. | 13:47 |
| Jeffrey4l | egonzalez, could u review this again? https://review.openstack.org/416046 | 13:47 |
| mliima | morning sdake egonzalez Jeffrey4l | 13:47 |
| egonzalez | Jeffrey4l: doing | 13:48 |
| Jeffrey4l | thanks | 13:48 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use install-pip macro in most source images https://review.openstack.org/413861 | 13:48 |
| egonzalez | such amount of quotes get me blind ;) | 13:49 |
| sdake | Jeffrey4l if I dont' see ya tomorrows on chinese new year, happy new year!! | 13:50 |
| *** jheroux has joined #openstack-kolla | 13:51 | |
| Jeffrey4l | thanks ;) | 13:51 |
| *** zhubingbing has joined #openstack-kolla | 13:52 | |
| zhubingbing | hey guys | 13:52 |
| zhubingbing | https://review.openstack.org/#/c/402224/ | 13:52 |
| zhubingbing | can u help me review it ? | 13:52 |
| *** zhubingbing has quit IRC | 13:53 | |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for influxdb https://review.openstack.org/424159 | 13:54 |
| *** goldyfruit has joined #openstack-kolla | 13:54 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Add dep for neutron-openvswitch-agent on a working openvswitch. https://review.openstack.org/425376 | 13:55 |
| sdake | zhenguo same to you re chinese new year :) | 13:57 |
| *** rhallisey has joined #openstack-kolla | 14:02 | |
| *** zkun has joined #openstack-kolla | 14:04 | |
| openstackgerrit | zhang kun proposed openstack/kolla-ansible: Fix Bootstraping the mongodb replication set https://review.openstack.org/422377 | 14:04 |
| *** zkun has quit IRC | 14:10 | |
| *** saneax-_-|AFK is now known as saneax | 14:11 | |
| srwilkers | morning | 14:12 |
| *** shardy has quit IRC | 14:14 | |
| *** zhubingbing has joined #openstack-kolla | 14:14 | |
| *** shardy has joined #openstack-kolla | 14:15 | |
| *** haplo37_ has quit IRC | 14:15 | |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for tempest https://review.openstack.org/424188 | 14:16 |
| *** hfu has joined #openstack-kolla | 14:17 | |
| *** haplo37_ has joined #openstack-kolla | 14:18 | |
| *** eaguilar has joined #openstack-kolla | 14:18 | |
| *** zhubingbing has quit IRC | 14:19 | |
| egonzalez | Anyone have ever seen this error Field 'domain_id' doesn't have a default value in keystone logs? | 14:19 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: Add freezer ansible role https://review.openstack.org/402224 | 14:20 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Fix the undefined error for vmtp_vm_availability_zone https://review.openstack.org/425715 | 14:21 |
| *** absubram has joined #openstack-kolla | 14:22 | |
| *** zhubingbing has joined #openstack-kolla | 14:22 | |
| *** absubram_ has joined #openstack-kolla | 14:26 | |
| *** zhubingbing has quit IRC | 14:26 | |
| *** absubram has quit IRC | 14:27 | |
| *** absubram_ is now known as absubram | 14:27 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Gate fix for horizon and cinder https://review.openstack.org/424368 | 14:29 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Update docs to pin K8s at v1.5.1 and remove helm pull workaround https://review.openstack.org/425059 | 14:30 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Fix ci gate deploy multiple k8s slave https://review.openstack.org/424086 | 14:30 |
| mliima | guys, need reviews here https://review.openstack.org/#/c/424158/6 | 14:31 |
| *** shardy has quit IRC | 14:35 | |
| *** eaguilar has quit IRC | 14:40 | |
| *** kbaegis has joined #openstack-kolla | 14:40 | |
| *** Jeffrey4l has quit IRC | 14:44 | |
| *** Jeffrey4l has joined #openstack-kolla | 14:44 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for telegraf https://review.openstack.org/424202 | 14:47 |
| *** jtriley has joined #openstack-kolla | 14:50 | |
| *** hfu has quit IRC | 14:51 | |
| *** tonanhngo has joined #openstack-kolla | 14:52 | |
| *** tonanhngo has quit IRC | 14:53 | |
| *** tonanhngo has joined #openstack-kolla | 14:53 | |
| *** rushil has joined #openstack-kolla | 14:54 | |
| *** zioproto has quit IRC | 14:55 | |
| *** jrist has quit IRC | 14:56 | |
| *** rwellum has quit IRC | 14:57 | |
| *** rwellum has joined #openstack-kolla | 14:57 | |
| *** tonanhngo has quit IRC | 14:58 | |
| des_consolado | hey `git show d3d4a933` shows that heka has been deprecated and fluentd is being used. I've done `pip install -U kolla-ansible kolla` and then `kolla-ansible upgrade -i myinventory`, however the heka containers are still there on the network and compute nodes and I don't see any fluentd containers anywhere D: | 14:59 |
| *** simon-AS5591 has quit IRC | 14:59 | |
| egonzalez | Jeffrey4l: dind't know whe were testing upgrades in gates ;) http://logs.openstack.org/46/425446/5/check/gate-kolla-ansible-dsvm-deploy-centos-binary-centos-7-nv/6ca1541/console.html#_2017-01-26_13_52_09_651990 | 15:00 |
| kbaegis | des_consolado: Unless you're willing to d | 15:00 |
| kbaegis | des_consolado: You could do a kolla-build | 15:00 |
| *** rushil has quit IRC | 15:00 | |
| egonzalez | des_consolado: fluentd was merged this week, pip tag is not made yet | 15:01 |
| Jeffrey4l | egonzalez, actually, it just run upgrade action. it is not a real upgrade test | 15:02 |
| des_consolado | ah right, in git tag in the repository I see 4.0.0.0b1 and 4.0.0.0b2, I guess the kolla-ansible in pip is the b2 and the master tree isn't in pip yet | 15:03 |
| egonzalez | des_consolado: yep | 15:04 |
| *** lrensing has joined #openstack-kolla | 15:04 | |
| egonzalez | des_consolado: if want to use master better clone git repo and execute task from tools folder | 15:04 |
| *** rushil has joined #openstack-kolla | 15:04 | |
| egonzalez | pomac: ping | 15:05 |
| *** absubram has quit IRC | 15:06 | |
| pomac | egonzalez: sorry, been cleaning racks, racking hardware and... meetings.. =) | 15:09 |
| egonzalez | were you able to fix the domain_id field in keystone db? | 15:09 |
| *** jrist has joined #openstack-kolla | 15:09 | |
| pomac | egonzalez: well, yes and no, it started working again but eventually failed - i suspect that it's a bug | 15:10 |
| egonzalez | having the same issue while doing upgrade from N to master | 15:10 |
| pomac | egonzalez: we have it in N | 15:10 |
| egonzalez | if only using a db_sync all works fine, if doing with zero downtime method fails | 15:11 |
| *** lamt has joined #openstack-kolla | 15:11 | |
| pomac | egonzalez: domain_id is cleared in some places for, to me, unknown reasons | 15:11 |
| egonzalez | asked at keystone channel but nobody replies | 15:11 |
| pomac | egonzalez: i suspect that we have the failure all the time now, the change we've done is to enable keystone v2 users (ie, set keystone v2 default domain to something != 'default' | 15:12 |
| pomac | egonzalez: have you played with changing the mtu btw? that's also "fun" ;) | 15:15 |
| egonzalez | pomac: i'll send a mail to the ML and see if someone explains or give a hit on how to proceed | 15:15 |
| pomac | egonzalez: good, lemme know how it goes =) | 15:15 |
| *** hfu has joined #openstack-kolla | 15:16 | |
| kbaegis | So can someone explain how this works? (MariaDB container)—<ovs>—<kglobals nic> | 15:17 |
| kbaegis | Is that accurate? We're pinning the container directly to the nic, not through an intermediary bridge? | 15:17 |
| kbaegis | Or is it (MariaDB)—<docker0>—<ovs>—<nic>? | 15:17 |
| kbaegis | Need to know, because I can't get: haproxy, mariadb, etc pinned to the vip | 15:18 |
| egonzalez | kbaegis: net host is used for containers | 15:18 |
| egonzalez | physical nic IP == container IP | 15:18 |
| kbaegis | tyvm | 15:19 |
| kbaegis | Gonna try disabling haproxy and set the IP to the primary nic. | 15:21 |
| *** lamt has quit IRC | 15:21 | |
| kbaegis | AIO | 15:21 |
| egonzalez | kbaegis: haproxy vip need to be other than the interface IP | 15:22 |
| egonzalez | kbaegis: my globals.yml as example http://paste.openstack.org/show/596606/ | 15:23 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Remove all kolla-ansible related docs https://review.openstack.org/425749 | 15:23 |
| *** sacharya has joined #openstack-kolla | 15:31 | |
| *** kbaegis has quit IRC | 15:33 | |
| *** fguillot has joined #openstack-kolla | 15:35 | |
| *** stevemar has quit IRC | 15:36 | |
| *** kbaegis has joined #openstack-kolla | 15:37 | |
| *** stevemar has joined #openstack-kolla | 15:37 | |
| *** rushil has quit IRC | 15:39 | |
| *** crushil has joined #openstack-kolla | 15:40 | |
| *** matrohon__ has quit IRC | 15:41 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for telegraf https://review.openstack.org/424202 | 15:43 |
| *** shardy has joined #openstack-kolla | 15:46 | |
| *** openstackgerrit has quit IRC | 15:48 | |
| *** sayantani01 has joined #openstack-kolla | 15:59 | |
| *** egonzalez has quit IRC | 16:00 | |
| *** openstackgerrit has joined #openstack-kolla | 16:01 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible: Remove image building related docs from kolla-ansible https://review.openstack.org/425766 | 16:01 |
| *** egonzalez has joined #openstack-kolla | 16:01 | |
| sbezverk | SamYaple: ping | 16:02 |
| *** srwilkers has quit IRC | 16:09 | |
| *** krtaylor has quit IRC | 16:09 | |
| *** jrist has quit IRC | 16:10 | |
| kbaegis | So does the docker net=host override whatever docker is configured to do by default? | 16:17 |
| *** papacz has quit IRC | 16:17 | |
| kbaegis | i.e.- if docker is using the bridge networking, does kolla override that behavior? | 16:17 |
| egonzalez | kbaegis: http://www.dasblinkenlichten.com/docker-networking-101-host-mode/ | 16:17 |
| *** mdnadeem has quit IRC | 16:19 | |
| *** jrist has joined #openstack-kolla | 16:21 | |
| *** krtaylor has joined #openstack-kolla | 16:22 | |
| kbaegis | egonzalez: So under /etc/conf.d/docker I just remove the networking config, correct? It didn't like —net=host | 16:24 |
| kbaegis | egonzalez: Or should kolla declare this automatically at run time? | 16:25 |
| *** msimonin has quit IRC | 16:28 | |
| egonzalez | kbaegis: im not getting what you mean. Don't want to use net-host or is not working? | 16:28 |
| kbaegis | *sigh* MariaDB isn't passing connectivity checks. I'll post a docker inspect in a second | 16:28 |
| kbaegis | Here's an inspect for my MariaDB | 16:29 |
| kbaegis | https://hastebin.com/odezinitoh.json | 16:29 |
| kbaegis | The network id correlates to my docker network "host" | 16:30 |
| kbaegis | Somethings not deploying right with the ansible scripts. I'm trying to figure out how to fix it | 16:31 |
| egonzalez | prechecks ensure no other service is using the port with the same IP mariadb is going to use | 16:31 |
| egonzalez | can you share globals and ip a in the hosts? | 16:32 |
| kbaegis | happily | 16:32 |
| *** prameswar has joined #openstack-kolla | 16:34 | |
| kbaegis | Globals: https://hastebin.com/ebiquxonut.bash | 16:34 |
| kbaegis | ip addr: https://hastebin.com/igiqiragov.pl | 16:35 |
| kbaegis | re-running prechecks | 16:35 |
| *** hfu has quit IRC | 16:36 | |
| *** hfu has joined #openstack-kolla | 16:37 | |
| *** hfu has quit IRC | 16:37 | |
| *** hfu has joined #openstack-kolla | 16:38 | |
| *** hfu has quit IRC | 16:38 | |
| *** hfu has joined #openstack-kolla | 16:39 | |
| *** hfu has quit IRC | 16:39 | |
| *** hfu has joined #openstack-kolla | 16:39 | |
| *** hfu has quit IRC | 16:40 | |
| kbaegis | passes prechecks just fine. But still can't connect to mariadb as indicated. The port isn't occupied. | 16:40 |
| kbaegis | Same issue, even if I enable the vip and change to 192.168.254.4 | 16:41 |
| SamYaple | sbezverk: pong | 16:41 |
| *** sayantani01 has quit IRC | 16:41 | |
| *** sayantani01 has joined #openstack-kolla | 16:42 | |
| *** srwilkers has joined #openstack-kolla | 16:42 | |
| sbezverk | SamYaple: thanks for reply.. I wanted to check with you about one problem I hit in kolla-kube and interested how kolla dealt with it.. | 16:43 |
| *** zioproto has joined #openstack-kolla | 16:43 | |
| SamYaple | ok | 16:43 |
| sbezverk | SamYaple: it turned out that manage_db fails if it was already run against the database | 16:44 |
| kbaegis | egonzalez: Any clue? Passed all 167 prechecks | 16:44 |
| SamYaple | sbezverk: what service? | 16:44 |
| sbezverk | SamYaple: when I remove a service package example cinder it removes only bits, not database or service endpoints from keystone | 16:44 |
| sbezverk | SamYaple: so when I try to deploy cinder package again which includes bootstrap steps it fails as db manage goes banans | 16:45 |
| kbaegis | Just also verified that inside the container I can ping out | 16:45 |
| SamYaple | sbezverk: well, for one. kolla-* has never supported removing services in any fashioin | 16:45 |
| SamYaple | sbezverk: or removing/scaling down anything for the most part. though you _can_ scale down some things, its more of a conincendence rather than design | 16:46 |
| zioproto | I have a question, I added two hosts in the [network] group. I have now two HA proxy instances with VRRP and that is fine. But what I really want is to have two L3-agent that work at the same time, as I have now in production. Is this is a possible scenario when deploying with Kolla ? | 16:46 |
| sbezverk | SamYaple: ok got.. | 16:46 |
| sbezverk | SamYaple: I plan to introduce then a set of chart for cleaning up leftovers | 16:46 |
| SamYaple | sbezverk: with kubernetes it will be esiear to do this for sure. since its a contastnaly running thing that can track up/down/delete actions. where ansible is point-in-time | 16:47 |
| egonzalez | zioproto: enable_neutron_agent_ha | 16:47 |
| sbezverk | SamYaple: since kolla has no already proposed solution | 16:47 |
| zioproto | egonzalez, but I dont want ha. I just want multiple L3-agents like I have now :) | 16:47 |
| zioproto | we have a big production cloud | 16:47 |
| zioproto | it is not possible to ahve all virtual name spaces | 16:47 |
| zioproto | in a single bare metal server | 16:48 |
| zioproto | so we scale it using multiple l3-agent | 16:48 |
| zioproto | working at the same time in parallel | 16:48 |
| zioproto | not in HA | 16:48 |
| zioproto | when you create a router neutron-server selects on which network node schedule the neutron router in a round robin fashion | 16:48 |
| zioproto | this is how neutron works | 16:48 |
| des_consolado | hey is there some docs/guidance anywhere on how to add to kolla-ansible if there aren't currently plays/configuration for something you want to use? Or I guess is it just a work it out for yourself and submit a PR kinda thing? | 16:48 |
| kbaegis | zioproto: Are you doing DVR? | 16:48 |
| zioproto | kbaegis, no, no DVR | 16:48 |
| des_consolado | (I need to use the BGP dynamic routing plugin - Ryu - doesn't look like this is supported by kolla-ansible right now) | 16:49 |
| zioproto | I am doing standard network node with VXLAN | 16:49 |
| zioproto | but I have two network nodes | 16:49 |
| zioproto | so when I do 'neutron agent-list' I see two l3 agents | 16:49 |
| des_consolado | It doesn't look that difficult I guess, you just add some vars and then some j2 stuff in the templates dir and then tasks in the tasks | 16:49 |
| kbaegis | zioproto: Oh. So they're a congestion point for all e-w, and n-s traffic? | 16:49 |
| zioproto | and my qrouter- namespaces | 16:49 |
| zioproto | yes they are a congestion point | 16:49 |
| zioproto | but because I can add many of them | 16:49 |
| zioproto | I dont care | 16:49 |
| zioproto | they are not in HA | 16:50 |
| zioproto | they are just many | 16:50 |
| zioproto | now | 16:50 |
| zioproto | we have everything with puppet | 16:50 |
| zioproto | we want to migrate to kolla | 16:50 |
| zioproto | kill our puppet layer that we also use to deploy | 16:50 |
| zioproto | but | 16:50 |
| zioproto | I have to understand how to make the same setup | 16:50 |
| zioproto | I have already with stable/newton kolla deployed the example multinode scenario | 16:50 |
| zioproto | I am trying to understan | 16:50 |
| zioproto | what changes I need to the inventory | 16:50 |
| zioproto | to match my existing cloud | 16:51 |
| zioproto | I dont get how to have something with more than 1 network node running the l3-agent, not in HA | 16:51 |
| zioproto | HA is just a feature in neutron.conf, you can have also multiple l30agent without HA | 16:51 |
| *** adrian_otto has joined #openstack-kolla | 16:52 | |
| zioproto | I will try enable_neutron_agent_ha but I expect then that neutron routers are created with HA yes | 16:52 |
| zioproto | and this breaks IPv6 in my cloud | 16:52 |
| *** saneax is now known as saneax-_-|AFK | 16:52 | |
| kbaegis | zioproto: So this is all documented in ansible/inventory/ | 16:53 |
| *** lamt has joined #openstack-kolla | 16:53 | |
| kbaegis | the [network] inventory states "The network nodes are where your l3-agent and loadbalancers will run | 16:53 |
| zioproto | yes, and I added two hosts in [network] | 16:54 |
| kbaegis | If you just want l3-agent, you can alter the [network] | 16:54 |
| kbaegis | zioproto: I think that will give you parity | 16:54 |
| zioproto | but only on 1 I see the l3-agent running | 16:54 |
| zioproto | I mean the other is a backup | 16:54 |
| zioproto | I am deploying again with enable_neutron_agent_ha = yes | 16:55 |
| kbaegis | I think ha is 1:1 by default | 16:55 |
| kbaegis | as in failover | 16:55 |
| *** athomas has quit IRC | 16:56 | |
| kbaegis | egonzalez: Any idea why my containers aren't attaching? Do I need to create some custom iptable rules or something? | 16:57 |
| egonzalez | zioproto: as far I can see there is no official way to add multiple agents in the same node, as they share rpc name | 16:58 |
| kbaegis | zioproto: Were you configuring your hypervisors to round-robin to the two separate network nodes? | 16:58 |
| zioproto | ]yes | 16:58 |
| zioproto | I have two diffent network nodes | 16:58 |
| kbaegis | egonzalez: Not two separate agents. He was leveraging the hypervisors multiple route option with the rr packet scheduler | 16:59 |
| kbaegis | I don't think kolla knows how to do that currently | 16:59 |
| zioproto | kbaegis, not sure about what you are saying | 16:59 |
| zioproto | so I have | 17:00 |
| zioproto | in my current prod infrastructure | 17:00 |
| zioproto | two servers | 17:00 |
| zioproto | with tho different host names | 17:00 |
| zioproto | both running l3-agent | 17:00 |
| *** saneax-_-|AFK is now known as saneax | 17:00 | |
| zioproto | I have two l3-agent with two different UUID | 17:00 |
| zioproto | I can even migrate the qrouter- namespaces from 1 server to the other server | 17:00 |
| kbaegis | zioproto: That's not the issue. The issue is how your vm hosts are sending packets to those two servers | 17:00 |
| egonzalez | zioproto: yup, but have created two different l3-agent files and started each service with his own config file | 17:00 |
| *** jrich523 has quit IRC | 17:01 | |
| zioproto | egonzalez, yes of course, because they are on different servers | 17:01 |
| kbaegis | The default is to use one desination and gobble up the l3 address as it fails | 17:01 |
| *** MasterOfBugs has quit IRC | 17:01 | |
| kbaegis | For load balancing, you need a packet scheduler to two distinct addresses | 17:01 |
| *** pramodrj07 has quit IRC | 17:01 | |
| *** PramodJ has joined #openstack-kolla | 17:01 | |
| *** MasterOfBugs has joined #openstack-kolla | 17:01 | |
| kbaegis | Otherwise you have HA, but no load balancing | 17:01 |
| zioproto | okay | 17:02 |
| zioproto | where is the correct place for operators to bring feedback about Kolla ? at the PTG ? or at the Milano Ops MIdcycle ? | 17:02 |
| *** jrich523 has joined #openstack-kolla | 17:02 | |
| egonzalez | zioproto: summit i guess, i dont think any from kolla to be at Ops midcycle at milano | 17:04 |
| *** inc0 has joined #openstack-kolla | 17:04 | |
| zioproto | summit you mean Boston, right ? | 17:04 |
| egonzalez | and PTG is more designed to be for developers, summit is for operators | 17:05 |
| egonzalez | yeah, boston | 17:05 |
| *** kbaegis has quit IRC | 17:05 | |
| inc0 | o/ | 17:05 |
| *** tonanhngo has joined #openstack-kolla | 17:05 | |
| inc0 | egonzalez, I couldn't work yesterday, my mariadb is broken for reasons unknown | 17:05 |
| *** kbaegis has joined #openstack-kolla | 17:06 | |
| inc0 | sorry:( | 17:06 |
| egonzalez | i'm working on it but stuck with errors in keystone database after upgrade, everything works fine but create user don't | 17:09 |
| egonzalez | inc0: Field 'domain_id' doesn't have a default value | 17:09 |
| inc0 | yay | 17:09 |
| *** tonanhngo has quit IRC | 17:10 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible: Refactor register.yml files https://review.openstack.org/416046 | 17:10 |
| *** crushil has quit IRC | 17:11 | |
| inc0 | Jeffrey4l, did you see anything strange lately in mariadb? | 17:14 |
| Jeffrey4l | inc0, no. what did u find? | 17:14 |
| inc0 | I tried to deploy yesterday and it doesn't work well | 17:14 |
| *** sayantani01 has quit IRC | 17:14 | |
| inc0 | on my ubuntu at least | 17:14 |
| inc0 | fresh master, fresh vms | 17:14 |
| Jeffrey4l | inc0, which task failed? | 17:14 |
| inc0 | it doesn't bind to port | 17:14 |
| inc0 | so building cluster failed | 17:14 |
| Jeffrey4l | hrm. never saw this. | 17:15 |
| inc0 | first error is "Connection refused" | 17:15 |
| inc0 | so problem is before that | 17:15 |
| inc0 | and I have no idea what causes it:/ | 17:15 |
| Jeffrey4l | inc0, bootstrap container failed? | 17:15 |
| inc0 | no | 17:15 |
| inc0 | starting contianers work | 17:15 |
| inc0 | but waiting for ready times out | 17:16 |
| Jeffrey4l | keepalived/vrrp issue? | 17:16 |
| sbezverk | inc0: qq is there a way to drop db using "ansible localhost -m mysql_db" | 17:17 |
| inc0 | negative, local ss -plant doesn't show it | 17:17 |
| inc0 | sbezverk, yeah, should be | 17:17 |
| inc0 | no idea tbh | 17:17 |
| sbezverk | inc0: who might know? ideas? | 17:17 |
| inc0 | worst case scenerio you can do ansible -m shell -a "mysql -c 'DROP DATABASE'" | 17:18 |
| Jeffrey4l | sbezverk, add: state=absent? | 17:18 |
| inc0 | or whatever syntax is | 17:18 |
| sbezverk | inc0: that was my plan Z ;) | 17:18 |
| Jeffrey4l | sbezverk, check "ansible-doc mysql_db" | 17:18 |
| kbaegis | Anyone know how I can have a perfectly functional MariaDB container that can ping out, but not have it pass the network connectivity check? | 17:18 |
| sbezverk | Jeffrey4l: thank you will do, I did not know it was a standard interface | 17:18 |
| kbaegis | I need some help with this | 17:18 |
| kbaegis | P.S. - the port passes the pre-check | 17:19 |
| inc0 | kbaegis, I saw something similar yesterday | 17:19 |
| SamYaple | sbezverk: its not a standard interface, but it is a standard convention that most people follow | 17:19 |
| inc0 | only not perfectly functioning | 17:19 |
| inc0 | since port hasn't been listened on | 17:19 |
| kbaegis | inc0: Inside the container I can connect the client to the server. Outside, it wasn't working | 17:19 |
| kbaegis | The container can ping a public IP too | 17:20 |
| inc0 | that is...strange since it has host network | 17:20 |
| jascott1 | morning folks | 17:20 |
| inc0 | /etc/hosts difference? It shouldn't be the case as it really uses usual one from host | 17:20 |
| inc0 | but copies it so strange | 17:20 |
| kbaegis | I do have one address hard coded into that | 17:21 |
| kbaegis | Let me check | 17:21 |
| sbezverk | Jeffrey4l: looks like drop db is not implemented, I do not see it in the doc, but I do see create.. strange | 17:22 |
| kbaegis | It did copy | 17:22 |
| *** sayantani01 has joined #openstack-kolla | 17:24 | |
| Jeffrey4l | sbezverk, it is implemented https://github.com/ansible/ansible/blob/devel/lib/ansible/modules/database/mysql/mysql_db.py#L327 | 17:24 |
| *** unicell has joined #openstack-kolla | 17:26 | |
| inc0 | http://paste.openstack.org/show/596623/ <- mariadb non-master logs | 17:26 |
| sbezverk | Jeffrey4l: I see, strange they do not mention in the doc, thanks for poiting out.. | 17:26 |
| *** lamt has quit IRC | 17:27 | |
| kbaegis | inc0: So I'm able to nc to the localhost:3306 | 17:27 |
| Jeffrey4l | np :) | 17:27 |
| kbaegis | where mariadb is running, and I get a connection string | 17:27 |
| *** eaguilar has joined #openstack-kolla | 17:27 | |
| kbaegis | This pretty clearly falls on the connection checking script. Unless it's not supposed to be using port 3306? | 17:28 |
| inc0 | yeah I have something similar | 17:28 |
| inc0 | on multinode | 17:28 |
| *** lamt has joined #openstack-kolla | 17:28 | |
| inc0 | only it's more than failing on conenctions tring as cluster members fails to connect to | 17:28 |
| inc0 | o | 17:28 |
| inc0 | too | 17:28 |
| zioproto | egonzalez, I deployed with enable_neutron_agent_ha and looks much better. Also I created a router and the router has ha = false as I wanted. Now I am trying to look for the qrouter-<uuid> namespace, where is that supposed to be created ? On the baremetal node hosting the docker container I cant find it. Is it inside a docker container ? | 17:29 |
| kbaegis | inc0: Weird. Well if the port is exposed and the versions are compatible it's got to be networking? | 17:29 |
| inc0 | no idea, telnet from other node worked perfectlyu | 17:30 |
| inc0 | which distro you're using? | 17:30 |
| kbaegis | zioproto: Oh yeah. If the l3-agents are sharing the same queue, that needs to be replicated too | 17:30 |
| *** tonanhngo has joined #openstack-kolla | 17:31 | |
| inc0 | http://paste.openstack.org/raw/596624/ this is my master node logs | 17:31 |
| inc0 | I'm baffled | 17:31 |
| *** eaguilar has quit IRC | 17:32 | |
| inc0 | nc works from everywhere, mariadb itself doesn't | 17:35 |
| inc0 | funny enough I have lots of connections from everywhere, so it's somethign about cluster building | 17:36 |
| *** rwellum is now known as Penfold | 17:36 | |
| inc0 | hmmm.....galera is supposed to create it's database right? | 17:38 |
| inc0 | for wsrep? | 17:38 |
| SamYaple | inc0: its straight forward | 17:38 |
| SamYaple | check /var/lib/mysql/xtrabackup.log | 17:38 |
| SamYaple | youll see the error | 17:38 |
| SamYaple | might not be called xtrabackup, maybe innodb somethign | 17:39 |
| SamYaple | anyway, its a log file in /var/lib/mysql | 17:39 |
| inc0 | Error executing 'FLUSH ENGINE LOGS': DBD::mysql::db do failed: Can't execute the query because you have a conflicting read lock at /usr//bin/innobackupex line 3059. | 17:40 |
| inc0 | 170126 17:20:59 innobackupex: Waiting for ibbackup (pid=1357) to finish | 17:40 |
| SamYaple | well there you have it | 17:40 |
| SamYaple | youve already got another process running | 17:40 |
| SamYaple | check the futher up logs to see if that crashed | 17:40 |
| kbaegis | Huh. | 17:40 |
| *** Penfold_ has joined #openstack-kolla | 17:40 | |
| *** hfu has joined #openstack-kolla | 17:41 | |
| *** prameswar has quit IRC | 17:41 | |
| *** eaguilar has joined #openstack-kolla | 17:41 | |
| inc0 | ok, what the actual fuck...I indeed see mariadb process in host pid namespace | 17:42 |
| SamYaple | uhhh inc0. thats how it works | 17:42 |
| SamYaple | you can see all processes from the host pid namespaces | 17:42 |
| inc0 | ok | 17:43 |
| SamYaple | but from the mariadb pid namespaces you cant see the host processes | 17:43 |
| inc0 | yeah, that's fine then | 17:43 |
| *** hfu has quit IRC | 17:43 | |
| inc0 | but yeah, I have both mysql_safe and mysqld running in container | 17:44 |
| inc0 | something is seriously wrong | 17:44 |
| SamYaple | thats also how that works | 17:45 |
| SamYaple | mysqld_safe is a shell script that launchs mysqld | 17:45 |
| SamYaple | so far so normal | 17:45 |
| *** eaguilar has quit IRC | 17:45 | |
| SamYaple | that log makes it pretty clear to me whats going on. where is the confusion? | 17:46 |
| kbaegis | SamYaple: What's responsible for setting the mutex? | 17:46 |
| kbaegis | SamYaple: Is that karbor? | 17:47 |
| inc0 | SamYaple, confusion is, what is the other process | 17:47 |
| inc0 | this is pristine fresh ubuntu 16 | 17:47 |
| SamYaple | inc0: innodbbackup. | 17:47 |
| SamYaple | inc0: whats running at pid=1357? | 17:47 |
| SamYaple | kbaegis: the lock is set by xtrabackup in prep for recieving or sending an IST or SST | 17:48 |
| *** Penfold_ has quit IRC | 17:48 | |
| inc0 | nothing now oO | 17:48 |
| SamYaple | kbaegis: it looks like multiple xtrabackups are running for some reason | 17:48 |
| inc0 | yeah | 17:48 |
| inc0 | I'll redeploy and see what' | 17:48 |
| inc0 | what's happening | 17:48 |
| *** zioproto has quit IRC | 17:49 | |
| inc0 | maybe it's just this innodb backup being so slow that stuff times out | 17:49 |
| SamYaple | the timeout times were set to 900 in all of the startup scripts for that reason | 17:49 |
| SamYaple | the official startup scripts taht is | 17:49 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Test keystone endpoint list https://review.openstack.org/425809 | 17:50 |
| kbaegis | I haven't looked too far into Karbor, but isn't that precisely what it does? Backs up the mariadb tables? | 17:51 |
| kbaegis | But that gets deployed later? | 17:51 |
| *** sambetts is now known as sambetts|afk | 17:51 | |
| *** egonzalez has quit IRC | 17:52 | |
| *** shardy is now known as shardy_afk | 17:53 | |
| *** david-lyle has quit IRC | 17:59 | |
| kfox1111 | morning. | 17:59 |
| inc0 | SamYaple, btw did you get my msg about kolla-salt being good to go | 18:00 |
| inc0 | ? | 18:00 |
| *** mgoddard has quit IRC | 18:02 | |
| *** lamt has quit IRC | 18:02 | |
| inc0 | SamYaple, no such process:/ seems like it finished but lock hasn't been lifted | 18:02 |
| sdake | anyone ever seen office space? | 18:04 |
| * sdake groans | 18:04 | |
| rhallisey | sdake, where's my stapler? | 18:04 |
| sdake | rhallisey REDLINE | 18:04 |
| *** unicell has quit IRC | 18:04 | |
| kfox1111 | hmm... | 18:04 |
| kfox1111 | was close to working this time: http://logs.openstack.org/periodic/periodic-kolla-kubernetes-deploy-centos-binary-2-ceph/c22108a/console.html | 18:05 |
| *** rmart04 has quit IRC | 18:05 | |
| inc0 | ok, gonna grab some food and to debug I go | 18:07 |
| kfox1111 | sdake: does catalog list work? | 18:07 |
| sdake | kfox1111 i can try | 18:07 |
| sdake | moment | 18:07 |
| inc0 | kbaegis, let me know if you find anything on your end | 18:07 |
| kfox1111 | (just curious if endpoint list is an admin feature) | 18:07 |
| kfox1111 | catalog list isnt. | 18:07 |
| kfox1111 | if catalog list works but endpoint list doesnt, I know what the problem is. | 18:07 |
| sdake | kfox1111 yes catalog list works | 18:07 |
| sdake | glorious | 18:07 |
| *** Serlex has quit IRC | 18:08 | |
| kfox1111 | and... another merge conflict. :/ | 18:08 |
| kfox1111 | sdake: the fix is in here: https://review.openstack.org/#/c/418550/ | 18:08 |
| kfox1111 | Basically, we never do any keystone admin only commands in the gate, | 18:09 |
| sdake | kfox1111 the create-endpoints yaml? | 18:09 |
| kfox1111 | and keystone-admin endpoint was setup to be internal only. | 18:09 |
| sdake | kfox1111 i just added one | 18:09 |
| kfox1111 | the create-endpoints patch and the iscsi_workflow one. | 18:09 |
| kfox1111 | sdake: it will fail though. | 18:10 |
| sdake | https://review.openstack.org/#/c/425809/ | 18:10 |
| sdake | kfox1111 roger | 18:10 |
| kfox1111 | k. | 18:10 |
| sdake | kfox1111 can you breka out just those two changes | 18:10 |
| SamYaple | inc0: yea i saw the kolla-salt message | 18:10 |
| SamYaple | inc0: im not sure about the "lock hasn't been lifted" business | 18:10 |
| sdake | kfox1111 into a parent of that other patch - so we can get that fix merged | 18:10 |
| sdake | kfox1111 super patches like the one you have are hard to follow | 18:10 |
| kfox1111 | yeah. probably time to break it up. | 18:11 |
| sdake | kfox1111 if you do that, I can rebae the dev env on top of it | 18:11 |
| kfox1111 | kept getting 99% of the way there, then finding one more thing. | 18:11 |
| sdake | yup | 18:11 |
| *** unicell has joined #openstack-kolla | 18:11 | |
| kfox1111 | it works for trunk now all except cinder's broken. | 18:11 |
| kfox1111 | haven't figured out why though. | 18:11 |
| kfox1111 | suspecting bad containers. but not sure yet. | 18:11 |
| *** eaguilar has joined #openstack-kolla | 18:12 | |
| kfox1111 | the cinder volumes don't attach to nova properly. | 18:12 |
| kfox1111 | everything else seems happy though. | 18:12 |
| *** Syffs has joined #openstack-kolla | 18:13 | |
| *** crushil has joined #openstack-kolla | 18:15 | |
| *** eaguilar has quit IRC | 18:16 | |
| sbezverk | kfox1111: ping | 18:17 |
| kfox1111 | sbezverk: ping | 18:17 |
| sbezverk | kfox1111: please check this bp https://blueprints.launchpad.net/kolla-kubernetes/+spec/helm-service-cleanup | 18:17 |
| *** gfidente is now known as gfidente|afk | 18:18 | |
| kfox1111 | sbezverk: interesting problem. | 18:18 |
| kfox1111 | not sure the correct behavior though. | 18:19 |
| kfox1111 | it is 'handy' in one way for it to just update the records on the new run. | 18:19 |
| kfox1111 | it also is a foot gun someone can unexpectedly step on. | 18:19 |
| sbezverk | kfox1111: is it not everything? | 18:20 |
| sbezverk | kfox1111: but man I am tired of manually cleaning up mariadb and keystone | 18:20 |
| kfox1111 | Perhaps, we should make some cleanup scripts/jobs so that if someone wants to change things, they can, but normally it will prevent random accidental reconfigures? | 18:20 |
| kfox1111 | I do want the ability to launch a service package without jobs too though. | 18:21 |
| kfox1111 | so maybe thats the other way to solve it? | 18:21 |
| sbezverk | kfox1111: I will push PS with cinder cleanup service | 18:21 |
| sbezverk | as a prototype | 18:21 |
| kfox1111 | someone might accidently delete cinder, say, | 18:21 |
| kfox1111 | and wants to load it back, without doing any db/keystone changes. | 18:21 |
| sbezverk | it will include 3 jobs drop db drop service in keystone and user in keystone | 18:21 |
| kfox1111 | that would allow that. | 18:21 |
| sbezverk | it will not be a part of main cinder package | 18:22 |
| kfox1111 | yeah. sounds good. | 18:22 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 18:22 |
| inc0 | SamYaple, I think it has been lifte | 18:23 |
| inc0 | d | 18:23 |
| inc0 | just something failed to move forward | 18:23 |
| sbezverk | I have two options either modify existing _common_create_db to accept a parameter operation where delete would be pass to it | 18:23 |
| inc0 | lslocks shows nothing | 18:23 |
| sbezverk | or use _common_delete_db macro | 18:23 |
| sbezverk | kfox1111: what would be your choice? | 18:23 |
| kfox1111 | hmm... let me look. | 18:25 |
| kfox1111 | ah. that ones still ansible based. :/ | 18:25 |
| *** unicell has quit IRC | 18:25 | |
| sbezverk | kfox1111: yeah at least for db management | 18:26 |
| kfox1111 | on the fence. sharing code woudl be nice. | 18:26 |
| kfox1111 | but, knowing that the code your about to run couldn't possibly delete your database is also nice. | 18:27 |
| sbezverk | kfox1111: I tend to use a separate macro because of this specific reason | 18:27 |
| kfox1111 | yeah. | 18:27 |
| sbezverk | especially during the debug phase ;) | 18:27 |
| kfox1111 | +1 :) | 18:27 |
| kfox1111 | ok. lets keep it seperate for that reason. | 18:27 |
| sbezverk | kfox1111: sounds good.. | 18:28 |
| kfox1111 | sbezverk: you were right. bzip wasn't in the vm. | 18:29 |
| *** unicell has joined #openstack-kolla | 18:30 | |
| sbezverk | kfox111: yeah it is just I hit it bunch of times in the past but thought myabe it has made into the default package list | 18:31 |
| kfox1111 | yeah. | 18:31 |
| kfox1111 | the periodic jobs did run though. :) | 18:31 |
| kfox1111 | http://logs.openstack.org/periodic/periodic-kolla-kubernetes-deploy-centos-binary-2-ceph/c22108a/console.html | 18:31 |
| kfox1111 | just gotta fix it up slightly, and tomorrow maybe we'll have some tarballs. :) | 18:32 |
| sbezverk | kfox1111: cool | 18:32 |
| *** msimonin has joined #openstack-kolla | 18:33 | |
| inc0 | https://review.openstack.org/#/c/424784/ can I have core eyes on this please? | 18:34 |
| inc0 | so we can get this process out of the way? | 18:34 |
| *** v1k0d3n has quit IRC | 18:35 | |
| *** jtriley has quit IRC | 18:36 | |
| *** lamt has joined #openstack-kolla | 18:37 | |
| *** lamt has quit IRC | 18:37 | |
| *** msimonin has quit IRC | 18:39 | |
| *** msimonin has joined #openstack-kolla | 18:39 | |
| *** msimonin has quit IRC | 18:40 | |
| sdake | kfox1111 can you tell me when you will have that gate fix patch split up so I can continue on the dev env? | 18:40 |
| kfox1111 | sdake: in process | 18:40 |
| *** msimonin has joined #openstack-kolla | 18:40 | |
| sdake | kfox1111 cool thanks :) | 18:40 |
| kfox1111 | 5 min | 18:40 |
| *** unicell has quit IRC | 18:40 | |
| *** msimonin has quit IRC | 18:41 | |
| *** msimonin has joined #openstack-kolla | 18:41 | |
| *** msimonin has quit IRC | 18:41 | |
| *** msimonin has joined #openstack-kolla | 18:42 | |
| *** msimonin has quit IRC | 18:42 | |
| *** eaguilar has joined #openstack-kolla | 18:42 | |
| *** msimonin has joined #openstack-kolla | 18:42 | |
| *** lamt has joined #openstack-kolla | 18:43 | |
| *** msimonin has quit IRC | 18:43 | |
| *** v1k0d3n has joined #openstack-kolla | 18:43 | |
| *** msimonin has joined #openstack-kolla | 18:43 | |
| *** msimonin has quit IRC | 18:44 | |
| inc0 | who wants to be fuel ptl? spot is still open:) | 18:47 |
| *** eaguilar has quit IRC | 18:47 | |
| srwilkers | lol | 18:47 |
| openstackgerrit | Merged openstack/kolla-ansible: Refactor register.yml files https://review.openstack.org/416046 | 18:48 |
| *** crushil has quit IRC | 18:49 | |
| kfox1111 | might be the best way to get fuel using kolla-kubernetes. ;) | 18:50 |
| kfox1111 | and to get a slick installer for kolla-kubernetes. :) | 18:50 |
| SamYaple | you have to have a commit in the fuel repos to be PTL | 18:50 |
| inc0 | yeah I know | 18:50 |
| inc0 | we should commit doc patch to all the installers | 18:51 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Implement Keystone zero-downtime upgrade https://review.openstack.org/425833 | 18:51 |
| inc0 | and ninja-self-nominate | 18:51 |
| SamYaple | i wrote a script that finds all the projects that ahvent had people with PTL subbmissions. i was going to self nominate after merging a doc patch | 18:51 |
| inc0 | and make them use Kolla | 18:51 |
| SamYaple | that was the plan in the paser :P | 18:51 |
| SamYaple | past* | 18:51 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Keystone-admin external support https://review.openstack.org/425836 | 18:52 |
| kfox1111 | sdake: that should it. | 18:52 |
| *** unicell has joined #openstack-kolla | 18:52 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Local registry in gate for trunk builds https://review.openstack.org/418550 | 18:54 |
| kfox1111 | inc0: the trunk patch isn't huge anymore. :) | 18:54 |
| kfox1111 | inc0: the only thing stopping it from merge is someone with some time figuring out why volumes wont attach. it is really close. | 18:54 |
| sdake | thanks kfox1111 | 18:55 |
| inc0 | kfox1111, can't do now, not until I figure out this mariadb scary thign | 18:55 |
| sdake | thanks sbezverk :) | 18:55 |
| kfox1111 | inc0: k. | 18:56 |
| kfox1111 | inc0: I'll keep poking at it off and on as I get time. | 18:56 |
| *** Penfold has quit IRC | 18:57 | |
| *** Penfold has joined #openstack-kolla | 18:58 | |
| *** klindgren__ is now known as klindgren | 19:00 | |
| *** harlowja has quit IRC | 19:02 | |
| kfox1111 | inc0: is it bzip package name on ubuntu? | 19:05 |
| *** crushil has joined #openstack-kolla | 19:05 | |
| inc0 | kfox1111, bzip1 | 19:06 |
| inc0 | bzip2** | 19:06 |
| kfox1111 | k. thanks. :) | 19:06 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Fix for the periodic kolla-kubernetes processes. https://review.openstack.org/425847 | 19:09 |
| sdake | kfox1111 so quick quesiton: | 19:13 |
| sdake | kfox1111 you said this patch should include just keyseond end point and ceph_iscsi:https://review.openstack.org/#/c/425836/ | 19:13 |
| sdake | kfox1111 however it has a bunch of stuff | 19:14 |
| kbaegis | inc0: Will do. Just got back from brunch. I'll keep looking | 19:14 |
| kfox1111 | artifactg of splitting the way I did. :/ | 19:15 |
| kfox1111 | I should amend the commit message. | 19:15 |
| kfox1111 | there's a fix for a network start race condition and some cleanup work for the workflow scripts, that the keystone_endpoint thing forced. | 19:16 |
| kfox1111 | I really don't want to split it into 4 ps's though. | 19:16 |
| *** absubram has joined #openstack-kolla | 19:18 | |
| kbaegis | inc0: Did you try setting the timeout to something ridiculous? Like 30 minutes? | 19:19 |
| kbaegis | Or is it just a stale write mutex across the whole mariadb instance? | 19:19 |
| kbaegis | Ugly | 19:20 |
| *** unicell has quit IRC | 19:20 | |
| *** unicell has joined #openstack-kolla | 19:20 | |
| kfox1111 | sdake: actually, rereading through it, the other bits are mostly about ensuring the job startups will work, as the admin port is now external, so network has to be started before the jobs now. | 19:23 |
| kfox1111 | I'll amend the commit log to have that though. | 19:23 |
| sdake | kfox1111 - ok - sounds good | 19:24 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Keystone-admin external support https://review.openstack.org/425836 | 19:26 |
| kfox1111 | dang... | 19:27 |
| kfox1111 | I really hoped it was the docker_registry name thing that would fix the cannot overwrite digest bug... :/ | 19:28 |
| kfox1111 | oh well. | 19:28 |
| kfox1111 | the cached containers should fix it. | 19:29 |
| *** saneax is now known as saneax-_-|AFK | 19:31 | |
| kfox1111 | inc0: yeah, I think there may be something wrong with the libvirt trunk container. | 19:32 |
| kfox1111 | not so much volume attach goodness. | 19:33 |
| *** Alexey_Y has joined #openstack-kolla | 19:33 | |
| SamYaple | just fyi, the libvirt container changed a good bit with the static uid/gid patch and i can assure you wasn't fully tested before merging | 19:34 |
| SamYaple | so id blame that always | 19:34 |
| *** jheroux has quit IRC | 19:35 | |
| kfox1111 | have a link for that? | 19:36 |
| *** adrian_otto has quit IRC | 19:36 | |
| kbaegis | inc0: So I found this: ./group_vars/all.yml:mariadb_wsrep_port: "4567" versus ./group_vars/all.yml:database_port: "3306" | 19:38 |
| kbaegis | Is that expected? | 19:38 |
| kfox1111 | I think the ports should be different. | 19:38 |
| kbaegis | Gotcha. I don't understand the xtrabackups functionality | 19:39 |
| *** crushil has quit IRC | 19:39 | |
| kbaegis | I'll need to dig further into that. The ports up though. | 19:40 |
| kfox1111 | if I remember correctly, there is a port for the db as normal, a port for cluster replication traffic, and a port for rsync raw data during initial recovery. | 19:40 |
| kfox1111 | thought I haven't looked at how kolla implemented it. | 19:40 |
| kfox1111 | oh. and a port ofr cluster health if you implement one. | 19:41 |
| *** nihilifer has quit IRC | 19:42 | |
| *** nihilifer has joined #openstack-kolla | 19:43 | |
| kbaegis | Well we have an issue where ansible deploy isn't able to talk to mariadb. The port's open, netcat verifies | 19:43 |
| kbaegis | I just tried removing the xtrabackup in wsrep-notify | 19:43 |
| *** david-lyle has joined #openstack-kolla | 19:45 | |
| kfox1111 | very interesting... I was contimplating trying to deploy k8s in k8s. http://blog.kubernetes.io/2017/01/how-we-run-kubernetes-in-kubernetes-kubeception.html | 19:45 |
| kfox1111 | though with helm. :) | 19:45 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: WIP: Add Development Setup Script for use with Kubernetes Helm https://review.openstack.org/405720 | 19:46 |
| kbaegis | If I had 15 nodes, k8s would be awesome | 19:46 |
| *** mgoddard has joined #openstack-kolla | 19:47 | |
| kfox1111 | it scales from 1 node on up. :) | 19:47 |
| kbaegis | No one on #kubernetes could give me a straight answer on how to get k8s working on gentoo. Use minikube isn't really an answer | 19:47 |
| kbaegis | Use a VM to host kolla which can talk to the hosts hypervisor | 19:47 |
| kbaegis | Seems absurd | 19:48 |
| sdake | kbaegis try slack kubernetes channel | 19:48 |
| sdake | kbaegis kubernetes community not super active on irc | 19:48 |
| *** MasterOfBugs has quit IRC | 19:49 | |
| *** PramodJ has quit IRC | 19:49 | |
| *** sayantani01 has quit IRC | 19:51 | |
| kfox1111 | kbaegis: yeah. slack channel is more responsive. | 19:52 |
| kfox1111 | all their startup scripts are written for systemd though, and gentu's a holdout I hear. | 19:52 |
| kfox1111 | so you may have to write those from scratch. | 19:52 |
| kfox1111 | gentoo I mean. | 19:52 |
| *** jheroux has joined #openstack-kolla | 19:53 | |
| inc0 | kbaegis, yeah, these ports are supposed to be different | 19:53 |
| inc0 | somethings different happening there | 19:53 |
| kbaegis | kfox1111: Thank god Gentoo isn't on systemd | 19:55 |
| kfox1111 | kbaegis: sorry to hear gentoo's not on systemd. ;) | 19:55 |
| kbaegis | kfox1111: No need to impugn Pottering OT here though | 19:55 |
| kfox1111 | as a dev, its soooo much better to write startup scripts. | 19:55 |
| kfox1111 | as an op, I haven't missed the old init much, other then the initial pain of reordering systemctl restart foo / service foo restart. | 19:56 |
| kbaegis | I agree. Systemd is the best operating system you can run underneath Linux | 19:56 |
| kbaegis | Did you know it has a web frontend? | 19:56 |
| kfox1111 | the fact I can jump between ubuntu and centos and remember how to enable a service at reboot is awesome too. | 19:56 |
| kbaegis | (not kidding) | 19:57 |
| kfox1111 | kbaegis: what doesnt. ;) | 19:57 |
| kfox1111 | linux has had all sorts of web admin frontends for a long long time. :) | 19:57 |
| kfox1111 | never use them. but I know they exist. | 19:57 |
| sbezverk | kfox1111: forgot to mention, iscsi pv/pvc is working now :-) there is no dynamic support though | 19:57 |
| kfox1111 | sbezverk: cool. :) | 19:58 |
| kbaegis | kfox1111: It's the linux paradigm, do all things, and make them interdependent | 19:58 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: WIP adding cleanup service for cinder https://review.openstack.org/425875 | 20:00 |
| kfox1111 | kbaegis: heh. linux is pragmatic. | 20:02 |
| kfox1111 | if the arguments for why systemd sucks were s/systemd/linux kernel/ | 20:02 |
| kfox1111 | people would suddently start saying "its ok" | 20:02 |
| kfox1111 | the linux kernel "is monolythic", "puts way too much in one process", "wants to take over the world"... | 20:03 |
| kfox1111 | ;) | 20:03 |
| kbaegis | Not going to debate it. Systemd is a mess. I'm glad it works though | 20:03 |
| kfox1111 | I agree with the unix phylosophy of making a tool do one thing, and doing it well. | 20:03 |
| kfox1111 | thought at doesn't mean the tools cant be bundled up together. :) | 20:03 |
| kfox1111 | anyway... :) | 20:04 |
| kfox1111 | all I know is, my life has been simplified since it came along. so a net positive. I havent looked at the code, but don't really need to. | 20:05 |
| kfox1111 | these days, systemd is the thing that gets k8s up engouh to deal with the rest in k8s. :) | 20:05 |
| *** harlowja has joined #openstack-kolla | 20:09 | |
| kfox1111 | http://logs.openstack.org/36/425836/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-external-ovs-nv/05106d7/console.html | 20:15 |
| *** mgoddard has quit IRC | 20:15 | |
| kfox1111 | that one's really bugging me. :/ | 20:15 |
| kfox1111 | why has setup-resolv-conf suddently become unstable? | 20:15 |
| kfox1111 | all its doing is passing resolv.conf back to the host. | 20:15 |
| kfox1111 | I wonder if I should just rewrite that to be a helm job now, and have it directly push into the configmap. :/ | 20:17 |
| sbezverk | kfox1111: I heard that dns fix is getting into 1.6 kube | 20:20 |
| inc0 | sooo...I used stable/newton images | 20:20 |
| inc0 | and it's working | 20:20 |
| sbezverk | so soon we will not need a workaround | 20:20 |
| kfox1111 | resolvconf? | 20:20 |
| inc0 | so we messed up something in images | 20:20 |
| SamYaple | kbaegis: mariadb + galera uses 4 ports, one for mysql traffic like normal (3306), one for wsrep (4567), one for system state transfers SST (4444) and one for incremental state transfers IST (4568) | 20:20 |
| SamYaple | inc0: probably the static uid/gid patch. https://review.openstack.org/#/c/412231/ there were kolla-ansible and probably kolla-kubernetes changes taht were supposed to go with that that i dont think people implemented | 20:21 |
| SamYaple | i ran out of time to work on it but someone else started working on it after i abanonded it | 20:22 |
| kfox1111 | SamYaple: what changes have to be made on the kolla-ansible/kolla-kubernetes side? | 20:22 |
| SamYaple | kfox1111: that i know of, libvirt permissions, but they need to be syncd between centos and ubuntu | 20:22 |
| SamYaple | and then tehre is /dev/kvm permissions | 20:23 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Integrate designate with neutron https://review.openstack.org/424679 | 20:23 |
| kfox1111 | permissions, like we specify the uid to use to the conatiner via env, | 20:23 |
| kfox1111 | or something else? | 20:23 |
| SamYaple | like socket and device permissions | 20:23 |
| SamYaple | and probably more. | 20:23 |
| kfox1111 | hm... | 20:23 |
| SamYaple | like i said, i ran out of time | 20:23 |
| SamYaple | so i stopped working on it | 20:23 |
| kfox1111 | should the patch be reverted then? | 20:23 |
| SamYaple | next thing i new, someone had restored it and merged it | 20:24 |
| SamYaple | knew* | 20:24 |
| kfox1111 | (I think the patch is very important. but if broken...) | 20:24 |
| inc0 | I'll try reverting this patch and see if it helps | 20:24 |
| sdake | https://www.openstack.org/assets/reports/OpenStack-2016-Annual-Report-final-draft.pdf | 20:24 |
| sdake | SamYaple i restored it | 20:25 |
| sdake | not sure why it was merged | 20:25 |
| sdake | SamYaple klingren was going to take over the work | 20:25 |
| sdake | i said right in the review it wasn't ready | 20:25 |
| sdake | SamYaple so its not your error | 20:25 |
| SamYaple | just letting people know. i dont think it was ready for merge, but again, i ran out of time to work on it | 20:26 |
| SamYaple | _testing_ is what was needed | 20:26 |
| SamYaple | i got it passign the gate | 20:26 |
| sdake | SamYaple roger - thank sfor getting it started ;) | 20:26 |
| sdake | SamYaple yup sometimes manual testing > gate testing | 20:27 |
| sdake | sound slike the root cause was sorted out - yay ;) | 20:27 |
| kfox1111 | sdake: thanks for the link. | 20:27 |
| sdake | kfox1111 i haven' read it yet | 20:27 |
| sdake | kfox1111 lemme know if anything is intereting in there :) | 20:28 |
| kfox1111 | kk | 20:28 |
| kfox1111 | "upstream community has been | 20:28 |
| kfox1111 | very supportive of users and receptive | 20:28 |
| kfox1111 | to feedback" | 20:28 |
| kfox1111 | heh.... | 20:28 |
| kfox1111 | "We updated our | 20:29 |
| kfox1111 | mission statement to increase focus | 20:29 |
| kfox1111 | on interoperability and the cloud end | 20:29 |
| kfox1111 | users needs." | 20:29 |
| kfox1111 | I sure hope so. | 20:29 |
| *** sayantani01 has joined #openstack-kolla | 20:31 | |
| sbezverk | kfox1111: right resolve.conf | 20:36 |
| *** gfidente|afk has quit IRC | 20:36 | |
| kfox1111 | sbezverk: nice. :) | 20:39 |
| kfox1111 | I really hope it makes it. :) | 20:39 |
| *** lrensing has quit IRC | 20:39 | |
| kfox1111 | ugg | 20:40 |
| kfox1111 | "Known as the 'Composable Shell', or CSHELL" | 20:40 |
| kfox1111 | thanks ms.... more confusion. :/ | 20:40 |
| *** lrensing has joined #openstack-kolla | 20:41 | |
| kfox1111 | nice... | 20:48 |
| kfox1111 | sbezverk: sdake: https://review.openstack.org/#/c/425836/ | 20:48 |
| inc0 | I have theory what might have happened to mariadb | 20:51 |
| kfox1111 | yeah? | 20:51 |
| inc0 | https://review.openstack.org/#/c/412231/33/docker/macros.j2 we set up /var/lib/mysql as homedir | 20:52 |
| inc0 | and do some stuff with it later on | 20:52 |
| kfox1111 | and the uid stuff is messing with that? | 20:54 |
| inc0 | it might | 20:55 |
| inc0 | I'll test it out | 20:55 |
| *** gema has joined #openstack-kolla | 20:56 | |
| kfox1111 | bbiab | 20:59 |
| *** esmiurium has quit IRC | 20:59 | |
| *** jrobinson has joined #openstack-kolla | 21:12 | |
| *** crushil has joined #openstack-kolla | 21:14 | |
| *** gema has quit IRC | 21:22 | |
| *** unicell has quit IRC | 21:24 | |
| *** gema has joined #openstack-kolla | 21:29 | |
| *** mgoddard has joined #openstack-kolla | 21:30 | |
| *** sacharya has quit IRC | 21:34 | |
| *** l4yerffeJ has joined #openstack-kolla | 21:34 | |
| *** adrian_otto has joined #openstack-kolla | 21:34 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 21:34 | |
| *** l4yerffeJ__ has quit IRC | 21:35 | |
| *** Jeffrey4l has quit IRC | 21:35 | |
| *** rmart04 has joined #openstack-kolla | 21:43 | |
| *** hfu has joined #openstack-kolla | 21:44 | |
| *** hfu has quit IRC | 21:47 | |
| *** jtriley has joined #openstack-kolla | 21:47 | |
| *** aric49 has joined #openstack-kolla | 21:51 | |
| jrobinson | sayantani01, hi this is Joe from the docs team, just replied to your email :) | 21:53 |
| *** mgoddard has quit IRC | 21:54 | |
| *** jtriley has quit IRC | 21:54 | |
| sayantani01 | Hey jrobinson, I will check your email shortly. I am running something and am disconnected to the vpn at the moment. | 21:55 |
| *** rhallisey has quit IRC | 21:56 | |
| jrobinson | no problems. | 21:56 |
| *** rhallisey has joined #openstack-kolla | 21:56 | |
| *** rhallisey has quit IRC | 21:57 | |
| v1k0d3n | inc0: sounds like kolla world dominance. nice. | 21:58 |
| inc0 | ? | 21:58 |
| v1k0d3n | fuel takeover | 21:59 |
| inc0 | running for Fuel PTL? ofcourse it's world dominance! | 21:59 |
| inc0 | it's also a joke;) | 21:59 |
| *** breitz has quit IRC | 22:01 | |
| *** fguillot has quit IRC | 22:01 | |
| inc0 | but, last release projects that didn't have PTL candidate were threatened of being kicked off big tent | 22:01 |
| *** breitz has joined #openstack-kolla | 22:01 | |
| inc0 | it's just an oversight probably tho, Fuel community shrank and probably they don't read as much ML as they used to | 22:02 |
| v1k0d3n | perhaps it's a shift in focus. | 22:02 |
| inc0 | they did lay off a lot of people while at it | 22:03 |
| inc0 | they being Mirantis | 22:03 |
| openstackgerrit | Mark Giles proposed openstack/kolla-kubernetes: Fix typos in development environment documentation https://review.openstack.org/425946 | 22:03 |
| *** jheroux has quit IRC | 22:03 | |
| inc0 | so not sure, they did shift focus to fuel-ccp, but I wonder what they downstreams think about them now | 22:03 |
| inc0 | I expect lots of people feel screwed | 22:04 |
| sdake | inc0 who is running forf uel ptl? | 22:04 |
| inc0 | nobodyu | 22:05 |
| sdake | seriously? | 22:05 |
| inc0 | nobody submitted candidacy | 22:05 |
| sdake | sup v1k0d3n long time no see ;) | 22:05 |
| sdake | is the deadline up? | 22:05 |
| v1k0d3n | just don't want new folks to think kolla is only concerned with world dominance ;) | 22:05 |
| v1k0d3n | more community, right? :) | 22:05 |
| sdake | v1k0d3n nah - we want everyone to work together | 22:05 |
| sdake | v1k0d3n when I say we above, I really mean I | 22:05 |
| sdake | I can't speak for others | 22:05 |
| v1k0d3n | sdake: i know. | 22:06 |
| inc0 | well, I think we have good culture of this | 22:06 |
| inc0 | anyway, Fuel PTL...that would be fun wouldn't it? | 22:06 |
| sdake | inc0 kolla does, openstack in general needs improvement here | 22:06 |
| sdake | inc0 if a project doen't have a ptl, it becomes a leaderless project and is ejected from the bigt tent | 22:07 |
| *** unicell has joined #openstack-kolla | 22:13 | |
| inc0 | yeah, I wonder if that's going to happen | 22:13 |
| inc0 | stupidest way to get kicked off | 22:13 |
| inc0 | if you ask me | 22:13 |
| *** sayantani01 has quit IRC | 22:21 | |
| *** sayantani01 has joined #openstack-kolla | 22:22 | |
| *** MasterOfBugs has joined #openstack-kolla | 22:23 | |
| *** unicell1 has joined #openstack-kolla | 22:23 | |
| *** sacharya has joined #openstack-kolla | 22:23 | |
| *** eaguilar has joined #openstack-kolla | 22:24 | |
| *** unicell has quit IRC | 22:26 | |
| *** mgiles has quit IRC | 22:29 | |
| kbaegis | I'm former Mirantis myself :) | 22:30 |
| kbaegis | I'm shocked that no one submitted for ptl | 22:30 |
| kbaegis | Though ccp is the "new way forward" over there | 22:31 |
| *** pramodrj07 has joined #openstack-kolla | 22:31 | |
| kbaegis | Not the team I was on | 22:31 |
| *** aric49 has quit IRC | 22:32 | |
| *** jrobinson is now known as jrobinson-afk | 22:36 | |
| bmace_ | sdake: fyi, getting very close now. hit a recent helm issue because the version specified in the Docker container was wrong but once i fixed that in-container things were looking much better. | 22:39 |
| *** crushil has quit IRC | 22:39 | |
| *** Syffs has quit IRC | 22:41 | |
| *** Syffs has joined #openstack-kolla | 22:41 | |
| *** esmiurium has joined #openstack-kolla | 22:43 | |
| *** sacharya has quit IRC | 22:46 | |
| *** msimonin has joined #openstack-kolla | 22:48 | |
| sdake | bmace_ update from my end - i am able to run openstack keystone catalog | 22:49 |
| sdake | bmace_ there is a defect in kolla-kubernetes which makes openstack endpoint list not work | 22:49 |
| sdake | bmace_ there is a patch in teh queue for it | 22:49 |
| sdake | bmace_ otherwise, openstack seems to work | 22:50 |
| bmace_ | sdake: excellent | 22:50 |
| sdake | one problem I have is horizon is busted | 22:50 |
| sdake | it works whe nyou hit the web page, however, after login it kicks out some odd error | 22:50 |
| bmace_ | sdake: who needs a gui? ;) | 22:50 |
| sdake | bmace_ apparently people that write checks ;-) | 22:50 |
| david-lyle | sdake: what is the error? | 22:50 |
| sdake | david-lyle moment i'll see if I still have the env up | 22:51 |
| *** rmart04 has quit IRC | 22:52 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: PS fixes issue discovered with keystone service package https://review.openstack.org/425964 | 22:52 |
| sdake | david-lyle https://drive.google.com/file/d/0B8q6xDPETSkHYkNHNi16TC0zSFE/view?usp=sharing | 22:55 |
| sdake | bmace_ ^^ | 22:55 |
| *** sayantani01 has quit IRC | 22:55 | |
| david-lyle | seems like maybe the POLICY_PATH setting is wrong | 22:56 |
| *** sayantani01 has joined #openstack-kolla | 22:56 | |
| sdake | david-lyle which file is that set in? | 22:56 |
| sdake | david-lyle is that an env variable or something? | 22:56 |
| david-lyle | openstack_dashboard/settings.py or openstack_dashboard/local/local_settings.py | 22:56 |
| *** eaguilar has quit IRC | 22:56 | |
| david-lyle | unless the path is really openstack_dashboard/openstack_dashboard | 22:57 |
| david-lyle | but that seems odd | 22:57 |
| sdake | sbezverk is the above patch to fix the problem found in keystone endpoint list? | 22:57 |
| Pavo | sdake I was getting same error until I disabled selixun | 22:57 |
| Pavo | selinux | 22:57 |
| sdake | Pavo pretty sure selinux is disabled | 22:57 |
| bmace_ | sdake: i remember hearing recently there were some general horizon access issues on the trunk | 22:57 |
| david-lyle | if that path is correct, then it's saying nova_policy.json is not in the path | 22:58 |
| david-lyle | *directory | 22:58 |
| david-lyle | not path | 22:58 |
| sdake | pavo selinux is in permissive mode | 22:58 |
| *** mgoddard has joined #openstack-kolla | 22:58 | |
| sdake | david-lyle let me check the container contents - moment - and thanks for the ts help :) | 22:58 |
| Pavo | should be good to go then | 22:58 |
| sbezverk | sdake: not sure, I was deploying on my cluster keystone service and found some issues | 22:58 |
| sdake | sbezverk can you try keystone endpoint list on your personal cluster? | 22:59 |
| sdake | sbezverk i thought we moved to 3.y.z. containers | 22:59 |
| sdake | sbezverk b71ad3350d04 docker.io/kolla/centos-binary-horizon:2.0.2 "kolla_start" 6 hours ago Up 6 hours k8s_main.7b85cb09_horizon-4073033154-5hfcp_kolla_b289a92d-e3e7-11e6-91de-003a7d69f5dc_95ba3a3a | 23:00 |
| sbezverk | sdake: http://paste.openstack.org/show/596649/ | 23:00 |
| sdake | sbezverk kfox1111 made a patch earlier for the endpoint list issue | 23:00 |
| sdake | sbezverk lathough I'd rather use yours | 23:01 |
| sdake | sbezverk would you mind comparing his to yours and ssee if its the same solution? | 23:01 |
| sbezverk | sdake: I have not seen it | 23:01 |
| sbezverk | do you have link? | 23:01 |
| sdake | sbezverk moment | 23:01 |
| *** saneax-_-|AFK is now known as saneax | 23:02 | |
| sdake | sbezverk https://review.openstack.org/#/c/425836/ | 23:03 |
| sdake | ok - on to inspecint containers :) | 23:03 |
| sdake | david-lyle - i'm in the continaer, where shall I look? find . -name openstack_dashboard -ls ? | 23:03 |
| sdake | david-lyle | 23:04 |
| sdake | ()[root@horizon-4073033154-5hfcp openstack_dashboard]# pwd | 23:04 |
| sdake | /usr/share/openstack-dashboard/openstack_dashboard | 23:04 |
| david-lyle | sdake: looking back at your paste | 23:05 |
| david-lyle | ok what is in conf/ just below that | 23:05 |
| david-lyle | ? | 23:05 |
| *** srwilkers has quit IRC | 23:06 | |
| david-lyle | and what is in /etc/openstack-dashboard if that is empty? | 23:06 |
| sbezverk | sdake: please use kfox1111 it is more complete, he covers issue I have not hit | 23:06 |
| kfox1111 | back. | 23:07 |
| sdake | sbezverk is his a superset of yours? | 23:07 |
| sdake | david-lyle ther is no conf dir | 23:07 |
| *** lrensing has quit IRC | 23:07 | |
| sdake | david-lyle this is in /etc/openstack-dashboard: ceilometer_policy.json cinder_policy.json glance_policy.json heat_policy.json keystone_policy.json local_settings neutron_policy.json nova_policy.json | 23:08 |
| sdake | i think one problem is we are using 2.0.2 contianers when i thought we moved to 3.0.2 containers | 23:08 |
| sdake | kfox1111 can you confirm that kolla-kubernetes should e using 3.0.2 versoin of the containers if using master? | 23:09 |
| sdake | (the dev env is using 2.0.2, and iscsi_workflow config uses 2.0.2 for all version numbers) | 23:09 |
| kfox1111 | trunk is still pointing to 2.0.2 by default. | 23:09 |
| kfox1111 | as its the most tested. | 23:09 |
| kfox1111 | we need to get the gate stable again, then play with enabling more 3.0.2 tests. | 23:10 |
| sdake | kfox1111 what was changed to 3.0.2? | 23:10 |
| kfox1111 | and getting trunk working. | 23:10 |
| sdake | kfox1111 or did that patch not happen? | 23:10 |
| sdake | kfox1111 as far as I can tell, trunk is pretty stable for kolla-kubernretes - deployed htis morning and can do openstack catalog list atleast :) | 23:10 |
| *** Penfold has quit IRC | 23:10 | |
| david-lyle | ok this, I think goes back to a patch I -1'd | 23:11 |
| sbezverk | kfox1111: I was deploying keystone service on my cluster and hit one issue, as a result submitted PS to fix it | 23:11 |
| kfox1111 | sdake: 2 patches went in. 1 to move 3.0.1 stuff to 3.0.2. and a second to add full 3.0.2 support. | 23:11 |
| * david-lyle looks | 23:11 | |
| kfox1111 | sdake: no, trunks broken. | 23:11 |
| sbezverk | but it looks like you also worked in that direction | 23:11 |
| kfox1111 | let me show you. | 23:11 |
| kfox1111 | https://review.openstack.org/#/c/418550/ | 23:11 |
| kfox1111 | the t one is the trunk one. | 23:12 |
| kfox1111 | http://logs.openstack.org/50/418550/60/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/e47dc10/console.html | 23:12 |
| sdake | right | 23:12 |
| kfox1111 | volumes are broken. | 23:12 |
| kfox1111 | though if that one issue is resolved, it should work I think. | 23:12 |
| kfox1111 | I got everything else working.. | 23:12 |
| sdake | http://logs.openstack.org/50/418550/60/check/gate-kolla-kubernetes-deploy-centos-binary-t-ceph-multi-nv/e47dc10/console.html#_2017-01-26_19_28_55_496582 | 23:12 |
| sdake | groan | 23:12 |
| sdake | kfox1111 is there a patch up for the volumes are broke nthing? | 23:12 |
| kfox1111 | I haven't figured out why they are broken yet. :/ | 23:13 |
| kfox1111 | SamYaple thinks it may be the uid patch thingy. | 23:13 |
| kfox1111 | :/ | 23:13 |
| sdake | kfox1111 inc0 went on today and SamYaple as well about how kolla-ansible is cratered because of a change | 23:13 |
| sdake | oh right | 23:13 |
| kfox1111 | probably that same change. :/ | 23:13 |
| inc0 | still debugging | 23:13 |
| *** absubram has quit IRC | 23:13 | |
| sdake | i dont see how that would affect 2.0.2 images? | 23:13 |
| kfox1111 | yeah. | 23:13 |
| kfox1111 | it doesn't affect 2.0.2 images. | 23:13 |
| kfox1111 | only trunk. | 23:13 |
| sdake | trunk of which deliverable? | 23:14 |
| kfox1111 | 2.0.2 works and 3.0.2 works. | 23:14 |
| Pavo | what d you mean by cratered sdake ? | 23:14 |
| sdake | oh got i t | 23:14 |
| kfox1111 | only trunk. | 23:14 |
| inc0 | doesn't look anything like my problem tho | 23:14 |
| sdake | pavo - a crater occurs when a meteor hits the surface at high impact- causing the meteor the exxplode | 23:14 |
| inc0 | Pavo, I try to deploy trunk and my galera fails | 23:14 |
| sdake | cratering is the act of being the meteor :) | 23:14 |
| Pavo | ah ok | 23:14 |
| Pavo | yeah I never have success with trunk, I only have success on pip package installing using kolla-ansible | 23:15 |
| sdake | kfox1111 cool - so fro mthe developer docs pov, the fact that trunk of the kolla images are broken is irrelevant | 23:15 |
| sdake | Pavo right - trunk is not particularly stable because our gates are not particularlly stable | 23:15 |
| sbezverk | kfox1111: when you have a chance keystone thing.. here is what I was trying to get http://paste.openstack.org/show/596652/ | 23:16 |
| sdake | however, they are getting better rapidly and with kolla-kubernetes gates, they are becoming superior | 23:16 |
| sbezverk | using public_host variable set in values.yaml for service | 23:16 |
| kfox1111 | sdake: right. out of the box, it should work as its pointing to 2.0.2, which is heavily gate tested. | 23:16 |
| sdake | once we add cross repo gating for kolla deliverable from kolla-kubernetes, our gating will kick ass and tkae names | 23:16 |
| Pavo | found a ansible playbook today that deploys 5 node k8s on openstack | 23:16 |
| *** mgoddard has quit IRC | 23:17 | |
| sdake | Pavo ya there is a whole slew of those ;) | 23:17 |
| kfox1111 | sbezverk: yeah, I've got a patch for that : https://review.openstack.org/#/c/425836/ | 23:17 |
| Pavo | include kuberlet I think thats what its called | 23:17 |
| kfox1111 | the problem is, 'openstack endpoint list' is an admin command. | 23:17 |
| sdake | david-lyle i know your busy - and i got distracted - any thoughts on how to proceed with horizon? | 23:17 |
| kfox1111 | so it contacts the keystone public url you give it, looks up the endpoint for the admin endpoint, then contacts it. | 23:17 |
| kfox1111 | if the cli command is outside of k8s, it sees 'keystone-admin' as the dns entry and has no idea how to get there. | 23:18 |
| sdake | pretty sure that was sambetts|afk 's original analysis :) | 23:18 |
| kfox1111 | this ps makes it possible to put the keystone-admin service external, so that its contactable from outside the k8s cluster. | 23:18 |
| kfox1111 | sdake: +1 on cross gating. should be pretty easy for kolla-kubernetes. | 23:19 |
| kfox1111 | sdake: I just got periodic jobs into the gate. :) | 23:19 |
| sdake | kfox1111 nice - your our gating rockstar ;) | 23:19 |
| kfox1111 | sdake: http://logs.openstack.org/periodic/ and search for kolla-kubernetes :) | 23:19 |
| kfox1111 | we gotta get one patch landed, and then we will be caching the docker hub images for our own use. :) | 23:19 |
| sbezverk | kfox1111: I am ok but I also want to specify public_host for service | 23:19 |
| kfox1111 | sbezverk: you can with that ps. | 23:20 |
| kfox1111 | see: https://review.openstack.org/#/c/425836/2/tests/bin/ceph_workflow.sh line new side, 197 | 23:21 |
| Pavo | well gotta get off here, gotta get on the plane | 23:21 |
| Pavo | ttyl | 23:21 |
| david-lyle | sdake: try setting POLICY_PATH=/etc/openstack-dashboard in openstack_dashboard/local/local_settings.py | 23:21 |
| sdake | later pavo | 23:21 |
| sbezverk | kfox1111: ok in this case I can drop public_host change from my ps and leave only change in vlaues.yaml | 23:21 |
| sdake | david-lyle moemnt let me give that a go | 23:21 |
| Pavo | laterz | 23:21 |
| sbezverk | kfox1111: you specify it for microservice, have you tested it for service package? | 23:22 |
| sdake | david-lyle #POLICY_FILES_PATH = '/etc/openstack-dashboard' | 23:22 |
| sdake | thats what is there now | 23:22 |
| david-lyle | remove the # | 23:22 |
| sdake | david-lyle have to keep in mind 2.0.2 = mitaka | 23:22 |
| sbezverk | kfox1111: for microservice overriding public worked, it did not work for service | 23:22 |
| sdake | right | 23:22 |
| kfox1111 | sbezverk: there shouldn't be any difference? | 23:23 |
| sbezverk | kfox1111: there was before the change | 23:23 |
| kfox1111 | what was the problem you were seeing? | 23:23 |
| sbezverk | same type of issue we had with .Values.element_name not being acccessible in service | 23:23 |
| kfox1111 | oh. | 23:23 |
| kfox1111 | element name not using the global api? | 23:24 |
| sbezverk | that is why I added search for public_host | 23:24 |
| *** ipsecguy has quit IRC | 23:24 | |
| kfox1111 | but, why override elementname at all? | 23:24 |
| sdake | david-lyle ++ thanks dude | 23:24 |
| david-lyle | sdake: all good? | 23:24 |
| kfox1111 | yeah. search for public_host wsa part of the bug. | 23:24 |
| sdake | hey folks, are we copyign local_settings.py into the container properly for horizon? | 23:25 |
| kfox1111 | though in my ps, I think I changed it to dns_name. | 23:25 |
| kfox1111 | but I have the same fix. | 23:25 |
| sdake | david-lyle ya - pretty sure we re not copying local_settings.py propery | 23:25 |
| sbezverk | kfox1111: my question was have you tested your PS with service? | 23:25 |
| sdake | david-lyle in other words, we rae using defaults | 23:25 |
| sdake | david-lyle defeect in kolla-kubernetes - not in horizon | 23:25 |
| *** jrobinson-afk is now known as jrobinson | 23:25 | |
| sdake | david-lyle thanks for helping sort it out :) | 23:25 |
| kfox1111 | sbezverk: the gate tests all. ;) | 23:25 |
| david-lyle | sdake: any time | 23:26 |
| kfox1111 | check the service check if you doubt it. :) | 23:26 |
| sbezverk | kfox1111: I wish it was true | 23:26 |
| sdake | kfox1111 the gate doesnt test logging into horizon | 23:26 |
| sbezverk | but it is not testing overriding public_host | 23:26 |
| sdake | kfox1111 i think testing all is a dream ;-) | 23:26 |
| sdake | kfox1111 i'd be happy with testing some :) | 23:26 |
| kfox1111 | in all seriousness though, yes, the gate tests that: http://logs.openstack.org/36/425836/2/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/a973add/console.html | 23:26 |
| sdake | kfox1111 it tests curling the horizon web address | 23:27 |
| sdake | kfox1111 but it doesn't actually log into horizon | 23:27 |
| kfox1111 | sdake: true. true. | 23:27 |
| sdake | if you actualy log into horizon, you get the following: https://drive.google.com/file/d/0B8q6xDPETSkHYkNHNi16TC0zSFE/view?usp=sharing | 23:27 |
| kfox1111 | sdake: I'd like to change that at some point. | 23:28 |
| kfox1111 | just havent gotten to it yet. | 23:28 |
| sdake | kfox1111 sounds good | 23:28 |
| kfox1111 | I've done solinum tests before though. | 23:28 |
| sdake | kfox1111 in the mean time, can we sort out this horizon problem :) | 23:28 |
| sdake | kfox1111 i'm not quite sure how to copy local_settings.py into the container with k8s | 23:28 |
| sdake | kfox1111 i did it manually just now to get it going | 23:28 |
| kfox1111 | sdake: interesting..... that use to work. | 23:28 |
| kfox1111 | sdake: though it could have bitrotted. | 23:29 |
| sdake | kfox1111 pretty sure its the defaults rather then the copy | 23:29 |
| kfox1111 | sdake: by configmap | 23:29 |
| sdake | kfox1111 one thing to keep in mind is 2.0.2 nad 3.0.2 hae different horizon structure | 23:29 |
| sdake | kfox1111 the config files are different iirc | 23:29 |
| sdake | david-lyle could confirm | 23:29 |
| kfox1111 | ah. so could be broken for 3.0.2+ | 23:29 |
| sdake | kfox1111 we have had this problem before | 23:29 |
| sdake | kfox1111 i am using 2.0.2 images, so its definately broken there | 23:29 |
| sdake | alhtough it may be working on 3.0.2 images | 23:30 |
| kfox1111 | hmm... | 23:30 |
| sdake | i am not sure about the 3.0.2 thing | 23:30 |
| sdake | kfox1111 we had this problem with kolla-ansible with the shift of horizon from mitaka to newton | 23:30 |
| kfox1111 | should be able to try 'helm upgrade kolla/horizon --set image_tag=3.0.2' and see | 23:30 |
| sdake | ok moment :) | 23:30 |
| kfox1111 | to be honest, the last time I tried horizon config was with genconfig from kolla-ansible 3.x | 23:31 |
| kfox1111 | so it might be a gregression in support with kolla-ansible 4.x | 23:31 |
| inc0 | galera why u no working!? | 23:31 |
| kfox1111 | sdake: http://paste.openstack.org/show/sfJBK6siBgTNgy92znOV/ | 23:32 |
| kfox1111 | ^---- was generated a long time ago. I think that worked with 2.0.2. | 23:32 |
| kfox1111 | see if yours is different? | 23:32 |
| sdake | [root@oscorehost1 kolla-kubernetes]# helm upgrade kolla/horizon --set image_tag=3.0.2 | 23:33 |
| sdake | Error: This command needs 2 arguments: release name, chart path | 23:33 |
| sdake | [root@oscorehost1 kolla-kubernetes]# helm upgrade kolla/horizon . --set image_tag=3.0.2 | 23:33 |
| sdake | Error: UPGRADE FAILED: error reading .venv/Kolla-Kube/include/python2.7: read /root/kolla-kubernetes/.venv/Kolla-Kube/include/python2.7: is a directory | 23:33 |
| kfox1111 | get rid of the '.' | 23:33 |
| sdake | kfox1111 then it fails with the first error | 23:33 |
| kfox1111 | oh. right.... | 23:33 |
| kfox1111 | what did you name your release? | 23:34 |
| kfox1111 | (helm ls) | 23:34 |
| sdake | how do I tell? | 23:34 |
| sdake | ok | 23:34 |
| kfox1111 | and hope you named it something you can find. :/ | 23:34 |
| sdake | [root@oscorehost1 kolla-kubernetes]# helm ls | grep horizon | 23:34 |
| sdake | horizon-deployment 1 Thu Jan 26 16:51:34 2017 DEPLOYED horizon-deployment-0.4.0-1 | 23:34 |
| sdake | horizon-svc 1 Thu Jan 26 16:43:31 2017 DEPLOYED horizon-svc-0.4.0-1 | 23:34 |
| kfox1111 | great. so its named 'horizon-deployment' | 23:34 |
| kfox1111 | so try: helm upgrade horizon-deployment kolla/horizon-deployment --set image_tag=3.0.2 | 23:35 |
| *** ipsecguy has joined #openstack-kolla | 23:35 | |
| sdake | [root@oscorehost1 kolla-kubernetes]# helm upgrade kolla/horizon-deployment --set image_tag=3.0.2 | 23:35 |
| sdake | Error: This command needs 2 arguments: release name, chart path | 23:35 |
| kfox1111 | so, reviewers on: https://review.openstack.org/#/c/425836/ ? | 23:35 |
| kfox1111 | sdake: still need the release name in there. | 23:36 |
| sdake | oh i see | 23:36 |
| sdake | moment | 23:36 |
| sdake | eyes not doing well after eye exam | 23:36 |
| sdake | having trouble foucinng my eyeblals | 23:36 |
| kfox1111 | no worries | 23:36 |
| *** eaguilar has joined #openstack-kolla | 23:36 | |
| kfox1111 | (is everything but horizon working?) | 23:37 |
| kfox1111 | (just wondering how close the dev env is) | 23:37 |
| sdake | kfox1111 i haven't tried booting a vm | 23:39 |
| sdake | kfox1111 havne't tried anything but endpoint list and catalog list | 23:39 |
| sdake | so with that update no horizon is running via docker ps -a on either of my two nodes | 23:39 |
| kfox1111 | sdake: ok. so up through keystone is known working. | 23:39 |
| sdake | with the helm update | 23:39 |
| kfox1111 | give it a few. its gotta pull an image | 23:40 |
| kfox1111 | kubectl get pods --namespace=kolla | 23:40 |
| kfox1111 | and look for a horizon pod. | 23:40 |
| sdake | ya in creating state | 23:40 |
| kfox1111 | then: kubectl describe pod horizon-xxxx | 23:40 |
| kfox1111 | then: kubectl describe pod horizon-xxxx --namespace=kolla | 23:40 |
| kfox1111 | should tell you what its doing. | 23:40 |
| sdake | 4m 22s 10 {kubelet oscorehost2} Warning FailedMount MountVolume.SetUp failed for volume "kubernetes.io/configmap/47f5586a-e420-11e6-91de-003a7d69f5dc-logging-config" (spec.Name: "logging-config") pod "47f5586a-e420-11e6-91de-003a7d69f5dc" (UID: "47f5586a-e420-11e6-91de-003a7d69f5dc") with: configmaps "horizon-logging" not found | 23:41 |
| sdake | implementation looking pretty good from my pov | 23:42 |
| kfox1111 | uh... did it forget about options previously set during an upgrade? | 23:42 |
| sdake | few bugs here and there | 23:42 |
| sdake | kfox1111 no idea - still learning the ropes | 23:43 |
| bmace_ | sdake: you have your own image repo you are pulling from i assume? looks like the default is now 4.0.0-1 which will obviously not work when pulling from docker hub.. but at least everything up to the image pull is working well now. | 23:43 |
| sdake | kfox1111 hope to document atleast some of what i learn :) | 23:43 |
| sdake | bmace_ pulling from dockerio | 23:43 |
| bmace_ | sdake: which version tag? | 23:44 |
| sdake | bmace_ i am taking that script I pushed up and sambetts|afk 's worked on and starting it via kubeadm | 23:44 |
| kfox1111 | sdake: I dont know either. but I expected values to come from the old version during upgrades. :/ | 23:44 |
| kfox1111 | asking the helm devs now. | 23:44 |
| sdake | bmace_ huh? | 23:45 |
| sdake | kfox1111 i see that - in slack :) | 23:45 |
| sdake | i've gotta jet to dinner in 15 mins | 23:45 |
| sdake | so anyone have any qs or rquests for me lemme know now :) | 23:45 |
| sdake | my kids need m0ar minerals | 23:45 |
| sbezverk | kfox1111: nope | 23:47 |
| sbezverk | you need to specify all old values | 23:47 |
| sbezverk | for upgrade, I learned that when was doing reconfguration with upgrade | 23:47 |
| kfox1111 | sdake: for the upgrade, you have to specify all the values again. :/ | 23:48 |
| bmace_ | sdake: image tag version. 4.0 isn't available on dockerio is it? | 23:48 |
| sdake | can someone spoon feed me the values :) | 23:48 |
| sdake | bmace_ the iscsi_workflow config uses 2.0.2 by default | 23:48 |
| inc0 | SamYaple, still around? | 23:48 |
| kfox1111 | helm upgrade ..... --set kube_logger=false | 23:49 |
| *** eaguilar has quit IRC | 23:49 | |
| kfox1111 | I've got a much better way to do this in the future, but havent gotten a chance to merge it yet. | 23:50 |
| sdake | what are the elipses :) | 23:50 |
| kfox1111 | once https://review.openstack.org/#/c/425836/ merges, I can finish up the work. | 23:50 |
| kfox1111 | sdake: stuff from before. | 23:50 |
| sdake | same command as before kfox1111 ? | 23:50 |
| sdake | ok | 23:50 |
| kfox1111 | eyah. | 23:50 |
| kfox1111 | yeah | 23:50 |
| inc0 | kfox1111, maybe you can help me actually | 23:51 |
| kfox1111 | inc0: maybe? whats up? | 23:51 |
| inc0 | wsrep should have tables in mysql database right? | 23:51 |
| sdake | ok that deployed | 23:51 |
| *** david-lyle has quit IRC | 23:51 | |
| kfox1111 | inc0: hmm. not sure. I know it has env values. | 23:51 |
| kfox1111 | inc0: never looked if it had tables. | 23:52 |
| inc0 | so I vaguely remember that is keeps some of its state in database | 23:52 |
| sdake | kfox1111 that upgrade command works | 23:53 |
| sdake | kfox1111 i'll let you know if horizon 3.0.2 images work | 23:53 |
| kfox1111 | sdake: k. cool. thanks. | 23:53 |
| kfox1111 | can you post your config.json for horizon? | 23:54 |
| *** david-lyle has joined #openstack-kolla | 23:54 | |
| *** jrobinson is now known as jrobinson-afk | 23:54 | |
| sdake | kfox1111 same error | 23:55 |
| sdake | kfox1111 pretty sure that config.json you pasted earlier was wrong - where di dit come from? | 23:55 |
| kfox1111 | sdake: a 'kolla genconfig' from back in the 3.x days I think. | 23:56 |
| kfox1111 | back when I had horizon working. | 23:56 |
| sdake | kfox1111 hmm ok | 23:56 |
| kfox1111 | can't garantee that peraticular config worked, but shoudl be similar. | 23:56 |
| kfox1111 | trying to figure out what changed. | 23:57 |
| kfox1111 | we're for sure going to need a solinum test though. | 23:57 |
| sdake | wish i had more time to debug this - maybe have about 15 more mins | 23:58 |
| sdake | the json file is different now in master configs | 23:58 |
| sdake | it includes all of the policy json files | 23:58 |
| sdake | and a /etc/openstack-dashboard/local_settings | 23:58 |
| kfox1111 | yeah. mine only has: | 23:59 |
| kfox1111 | kfox2@mantis:~/svn/kolla-kubernetes$ ls /etc/kolla/horizon/ | 23:59 |
| kfox1111 | config.json horizon.conf local_settings | 23:59 |
| sdake | lol @ hostname | 23:59 |
| kfox1111 | :) | 23:59 |
| sdake | ok - so how do I get that local_settings into the config map? | 23:59 |
| sdake | or where is that genreated? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!