| sdake | i get genconfig generates the config -that step is done | 00:00 |
|---|---|---|
| kfox1111 | sec... | 00:00 |
| kfox1111 | let me just see if I can send you this whole thing... | 00:00 |
| sdake | bmace_ do you have further questions? | 00:00 |
| kfox1111 | darn... unfortunately, this config file is heavily modified from a previous upgrade attempt. :/ | 00:01 |
| bmace_ | sdake: i just need to debug why my container pulls are failing now :/ | 00:01 |
| kfox1111 | sdake: I'd recommend pulling kolla 3.x and running a genconfig from there. should be able to get a horizon config set that should work I think. | 00:01 |
| bmace_ | sdake: mariadb / rabbitmq look to be up, but that is it | 00:01 |
| sdake | kfox1111 "source": "/var/lib/kolla/config_files/local_settings", | 00:02 |
| sdake | "dest": "/etc/openstack-dashboard/local_settings", | 00:02 |
| sdake | "owner": "horizon", | 00:02 |
| sdake | "perm": "0644" | 00:02 |
| sdake | kfox1111 so looking at that, the source is /var/lib/kolla/config_files - the ocntinaer isn't going to ha ethat directory at all | 00:02 |
| kfox1111 | it should | 00:02 |
| bmace_ | sdake: well, maybe not "up", just the init-element for both.. still getting a feel for the full deploy process. | 00:02 |
| sdake | kfox1111 let me check it out | 00:03 |
| kfox1111 | we mount the configmap at that location. | 00:03 |
| *** lamt has quit IRC | 00:03 | |
| kfox1111 | do a: helm install kolla/horizon-deployment --dry-run --debug | 00:03 |
| *** lamt has joined #openstack-kolla | 00:04 | |
| kfox1111 | should see a volumeMounts: section with that mount point. | 00:04 |
| *** lamt has quit IRC | 00:04 | |
| sdake | kfox1111 - ok i have found the files inside the container | 00:04 |
| *** saneax is now known as saneax-_-|AFK | 00:04 | |
| *** harlowja has quit IRC | 00:04 | |
| sdake | the files copied INTO the container in /var/lib/kolla are wrong | 00:04 |
| kfox1111 | wrong how? | 00:05 |
| sdake | moment | 00:05 |
| sdake | groan can't fpaste | 00:05 |
| sdake | kfox1111 as an eample, just debugging locally, /var/lib/kolla/local_settings contains the line: | 00:07 |
| sdake | #POLICY_FILES_PATH = '/etc/openstack-dashboard' | 00:07 |
| sdake | that line should not be commented | 00:07 |
| sdake | that file is coming from kolla-ansible genconfig i assume - let me check what it is inside kolla-ansible | 00:07 |
| kfox1111 | yeah. kolla-kubernetes doesn't do any config there. | 00:09 |
| sdake | kfox1111 it apparently copies the files into emptydir right? | 00:10 |
| sdake | when the configmaps are generated? | 00:10 |
| sdake | rather to be more precise, the config maps are created from kolla-genconfig artifacts | 00:11 |
| sdake | and kolla-kubernetes does the config from the config maps | 00:11 |
| sdake | rather kubernetes | 00:11 |
| sdake | wow too tired to compose a straight sentence ;) | 00:11 |
| kfox1111 | sdake, the process is as follows: | 00:11 |
| kfox1111 | kolla-ansible genconfig generates /etc/kolla/horizon. | 00:11 |
| kfox1111 | kollakube res create configmap horzion | 00:12 |
| kfox1111 | # takes stuff from /etc/kolla/horizon/* and makes a k8s configmap out of it all and uploads it to k8s as configmap name 'horizon' | 00:12 |
| kfox1111 | helm install horizon.... | 00:12 |
| kfox1111 | loads up a deployment object that creates pods that mount configmap 'horizon' at /var/lib/kolla | 00:13 |
| kfox1111 | if you do a kubectl get configmaps --namespace=kolla you can see them. | 00:13 |
| kfox1111 | and a kubectl get configmap <configmapname> -o yaml --namespace=kolla to look at it if your curious. | 00:14 |
| sdake | hrm | 00:14 |
| sdake | well here is the contents of the genconfig tempalte: | 00:14 |
| sdake | #POLICY_FILES_PATH = '/etc/openstack-dashboard' | 00:14 |
| sdake | from here : /root/kolla-kubernetes/kolla-ansible/ansible/roles/horizon/templates | 00:14 |
| sdake | so it looks like genconfig is incompatible with kolla-kubernetes | 00:15 |
| sdake | not entirely sure which version of kolla-ansible is installed | 00:15 |
| sdake | since for some odd reaosn its installed ina venv | 00:15 |
| sdake | probably 3.0.2 | 00:15 |
| kfox1111 | I could believe trunk broke it. | 00:16 |
| kfox1111 | its not been tested in a while. | 00:16 |
| sdake | here is master: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/horizon/templates/local_settings.j2#L458 | 00:16 |
| kfox1111 | yeah. | 00:16 |
| kfox1111 | try genconfig from kolla 3.0.2 or 3.0.1 | 00:17 |
| sdake | i'll have to talk to Jeffrey4l_ about this patch: https://github.com/openstack/kolla-ansible/commit/f3ea03d88f3adce4e28c0394e48418251df026b2 | 00:17 |
| sdake | its chines enew year and i think he is out for 3-4 days | 00:18 |
| *** goldyfruit has quit IRC | 00:18 | |
| kfox1111 | ah. | 00:18 |
| sdake | ok - well in the meantime i'll revert that in my local env - since we dont care about murano at this point in time in kolla-kbuernetes | 00:18 |
| kfox1111 | sdake: so... | 00:18 |
| sdake | ok - tiem for dinner | 00:18 |
| kfox1111 | edit /etc/kolla/horizon/local_settings, | 00:18 |
| sdake | bmace_ if your around later, i'll be happy to help you out with what i know - but i must feed my children now | 00:18 |
| bmace_ | sdake: i think it is an issue with the container pull logic. inside the dev container there is no docker, which pull_container.sh assumes exists. | 00:18 |
| kfox1111 | remove the #, | 00:18 |
| kfox1111 | and kubectl create res configmap horzion | 00:19 |
| sdake | kfox1111 ya i've done that | 00:19 |
| sdake | kfox1111 i've actually got horizon working already | 00:19 |
| kfox1111 | then delete/recreate horzion deployment. | 00:19 |
| kfox1111 | ah. ok. | 00:19 |
| bmace_ | sdake: it should really probably be remoting somehow into the kubeX remote hosts it is deploying into to do the container pulls | 00:19 |
| sdake | kfox1111 i was understanding why its busted | 00:19 |
| kfox1111 | sdake: ok. didn't know you figured that out. | 00:19 |
| kfox1111 | so, its that one line thats broken, thats all? | 00:19 |
| sdake | bmace_ no idea on that one - but i have to admit i am doing a different deployment method then the continer based dev env deploy | 00:20 |
| sdake | kfox1111 pretty sure :) | 00:20 |
| sdake | its a regression in master from what i can tell | 00:20 |
| kfox1111 | bmace_: true. | 00:20 |
| sdake | a critical one - i doubt horizon works in kolla-ansible as well | 00:20 |
| sdake | inc0 if you manage to get kolla-ansible running can you try horizon | 00:20 |
| kfox1111 | bmace_: though I think there was a flag set to have it skip the pulls. | 00:20 |
| sdake | inc0 try to actually log into it | 00:20 |
| sdake | ok be back in about 45 mins :) | 00:20 |
| kfox1111 | sdake: +1. we need to fix it asap. | 00:20 |
| *** sdake has quit IRC | 00:20 | |
| bmace_ | kfox1111: but how could it possibly work without the pulls? is there something else that would get the images pulled onto the kubeX boxes? | 00:21 |
| kfox1111 | bmace_: the pulls were to make it less likely to hit the timeout. | 00:21 |
| kfox1111 | kube itself will pull the containers. | 00:21 |
| bmace_ | kfox1111: after running the entire dev_workflow.sh script they are definitely not there on any of the kubeX systems | 00:22 |
| kfox1111 | with the pull, the timeout for waiting for containers doesn't start until its sure all the containers are already downloaded. | 00:22 |
| kfox1111 | kubectl get pods --namespace kolla #shows them in what state? | 00:22 |
| bmace_ | kfox1111: doesn't show any pods :/ | 00:23 |
| bmace_ | + pull_containers | 00:23 |
| bmace_ | + '[' x1 == x ']' | 00:23 |
| bmace_ | + /opt/kolla-kubernetes/tools/wait_for_pods.sh kolla | 00:23 |
| bmace_ | containers failed to start. | 00:23 |
| bmace_ | NAME READY STATUS RESTARTS AGE | 00:23 |
| bmace_ | rabbitmq-init-element-c7k0b 0/1 ContainerCreating 0 3m | 00:23 |
| bmace_ | that is what i get at the end | 00:23 |
| bmace_ | but i see the helm install for all of the services proceeding.. | 00:24 |
| kfox1111 | kubectl describe pod rabbitmq-init-element-c7k0b --namespace=kolla | 00:24 |
| bmace_ | i'm guessing since docker doesn't exist in the container the pull_containers just finishes immediately | 00:24 |
| kfox1111 | I think the latest ps makes pull containers do nothing. | 00:24 |
| kfox1111 | noop. | 00:24 |
| *** crushil has joined #openstack-kolla | 00:25 | |
| bmace_ | FirstSeenLastSeenCountFromSubObjectPathTypeReasonMessage | 00:25 |
| bmace_ | ------------------------------------------------------------ | 00:25 |
| bmace_ | 46m46m1{default-scheduler }NormalScheduledSuccessfully assigned rabbitmq-init-element-c7k0b to 172.16.35.12 | 00:25 |
| bmace_ | 46m46m1{kubelet 172.16.35.12}spec.containers{main}NormalPullingpulling image "docker.io/kolla/centos-binary-rabbitmq:2.0.2" | 00:25 |
| bmace_ | 43m43m1{kubelet 172.16.35.12}spec.containers{main}NormalPulledSuccessfully pulled image "docker.io/kolla/centos-binary-rabbitmq:2.0.2" | 00:25 |
| bmace_ | 43m43m1{kubelet 172.16.35.12}spec.containers{main}NormalCreatedCreated container with docker id ad837226a459; Security:[seccomp=unconfined] | 00:25 |
| bmace_ | 43m43m1{kubelet 172.16.35.12}spec.containers{main}NormalStartedStarted container with docker id ad837226a459 | 00:25 |
| bmace_ | 43m43m1{kubelet 172.16.35.12}WarningFailedSyncError syncing pod, skipping: failed to "TeardownNetwork" for "rabbitmq-init-element-c7k0b_kolla" with TeardownNetworkError: "Failed to teardown network for pod \"54286bbd-e420-11e6-a08e-52540030c43f\" using network plugins \"cni\": required env variables missing" | 00:25 |
| kfox1111 | paste.openstack.org please | 00:25 |
| kfox1111 | is that all thats in there, or is there more? | 00:26 |
| bmace_ | kfox1111: http://paste.openstack.org/show/596656/ | 00:26 |
| kfox1111 | seems like the network wasn't inited yet. | 00:29 |
| kfox1111 | kubectl get pods --namespace=kube-system | 00:29 |
| bmace_ | kfox1111: http://paste.openstack.org/show/596657/ | 00:30 |
| kfox1111 | is rabbit still stuck? | 00:31 |
| kfox1111 | looks like your using weave. never played with it. | 00:32 |
| kfox1111 | the gate's testing with canal. | 00:32 |
| kfox1111 | not sure it will make a difference. but might. | 00:33 |
| bmace_ | well, this is whatever the default is for the halycon-kubernetes stuff | 00:33 |
| kfox1111 | yeah. | 00:33 |
| kfox1111 | just saying. :) | 00:33 |
| kfox1111 | in uncharted waters there. :) | 00:33 |
| bmace_ | hrm.. not sure how that is to change, but i can give it a go.. lots of uncharted waters going on here, lol | 00:34 |
| kfox1111 | true, true. :) | 00:34 |
| bmace_ | i see some logic at least that mentions canal in the kube-deploy stuff for halycon-kubernetes so maybe it won't be too bad. | 00:35 |
| kfox1111 | would be interesting to hear what sdake used. | 00:35 |
| bmace_ | sdake: what are you using in your k8s environment? | 00:35 |
| bmace_ | lol, yeah, same idea | 00:35 |
| kfox1111 | but, he's not here it would seem. :/ | 00:35 |
| kfox1111 | but anyway, for some reason it looks liek the network wasn't ready before the iscsi_workflow was started. | 00:36 |
| kfox1111 | the gate blocks running that script until it is ready. | 00:36 |
| kfox1111 | so, that might be part of the problem. | 00:36 |
| bmace_ | kk, well, i'll see what it takes to get canal working. would be nice to use the same thing as we do in the gate. | 00:36 |
| kfox1111 | yeah. couldn't hurt. | 00:39 |
| kfox1111 | if we get a lot of requets for weave support, we could add a weave version of the gates too. | 00:39 |
| kfox1111 | canal was just the thing I knew best at the time. | 00:39 |
| *** sayantani01 has quit IRC | 00:41 | |
| *** MasterOfBugs has quit IRC | 00:42 | |
| *** PramodJ has joined #openstack-kolla | 00:42 | |
| *** pramodrj07 has quit IRC | 00:42 | |
| *** MasterOfBugs has joined #openstack-kolla | 00:42 | |
| *** inc0 has quit IRC | 00:46 | |
| *** hfu has joined #openstack-kolla | 00:47 | |
| *** hfu has quit IRC | 00:49 | |
| v1k0d3n | kfox1111: just an observation.... | 00:49 |
| v1k0d3n | sounds like the gates are very complicated. | 00:50 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding cleanup service for cinder https://review.openstack.org/425875 | 00:50 |
| v1k0d3n | the instructions for kolla-k8s...are they up to date? | 00:50 |
| sbezverk | kfox1111: please check cinder clean up service.. | 00:50 |
| *** lrensing has joined #openstack-kolla | 00:51 | |
| sbezverk | kfox1111: I tested it on my cluster, looks ok | 00:51 |
| v1k0d3n | are things able to run? i heard that kolla-k8s is being used in production. | 00:51 |
| kfox1111 | v1k0d3n: gates are complicated, but well worth it. they prevent a lot of breakage/regression. | 00:52 |
| kfox1111 | the instructions are not up to date. :/ but are actively being woked on by several people right now. should be up to date in the next few days. | 00:53 |
| kfox1111 | there is likely to be a production system soonish. | 00:53 |
| kfox1111 | sbezverk: will do. | 00:53 |
| kfox1111 | sbezverk: please review https://review.openstack.org/#/c/425836/ | 00:53 |
| *** jtriley has joined #openstack-kolla | 00:54 | |
| kfox1111 | v1k0d3n: the gate is currently running probably dozens of openstack deployments a day now. | 00:54 |
| kfox1111 | including testing vm launching, network connectivity, volume attachment/moving/reattachment, etc. | 00:55 |
| kfox1111 | so regressions are cought really quickly. | 00:55 |
| *** sdake has joined #openstack-kolla | 00:58 | |
| kfox1111 | sdake: ping | 00:58 |
| v1k0d3n | i thought i heard sdake mention there was a production deployment... | 00:58 |
| sdake | wound me | 00:58 |
| v1k0d3n | where is that... | 00:58 |
| sdake | which? | 00:58 |
| kfox1111 | sdake: which network plugin you using for the dev env? | 00:58 |
| sdake | kfox1111 whatever the devenv sets up by default | 00:58 |
| kfox1111 | sdake: ok. so, weave. | 00:58 |
| sdake | kfox1111 oh - actually not sure - whatever kubeadm sets up by default | 00:59 |
| v1k0d3n | https://sdake.io/ | 00:59 |
| v1k0d3n | front page "and there now are real, production deployments of the kolla-kubernetes deliverable of the OpenStack Kolla project" | 00:59 |
| sbezverk | wow sdake now has his own io domain | 00:59 |
| sdake | v1k0d3n that was my understanding at the time i wrote that blog post | 00:59 |
| sdake | v1k0d3n that kfox1111 had placed his site into production | 01:00 |
| sdake | v1k0d3n it may not be accurate given teh fact horizon doesn't work | 01:00 |
| kfox1111 | sbezverk: ok. I can add elementname. it wasn't requried to fix the issue, so i didnt' touch it. | 01:00 |
| v1k0d3n | ouch | 01:00 |
| v1k0d3n | ok | 01:00 |
| v1k0d3n | didn't know. | 01:00 |
| kfox1111 | sdake: yeah, not quite yet. but soon. | 01:00 |
| sdake | kfox1111 good to hear | 01:00 |
| sdake | v1k0d3n no mal-intent meant, i got impression from kfox1111 he put 0.4.0 in production in the channel | 01:01 |
| kfox1111 | horizon works, jujst not with kolla-ansible trunk. | 01:01 |
| sdake | although i guess I shouldn't used the word deployments either as that is plural | 01:01 |
| v1k0d3n | it's cool. | 01:01 |
| sbezverk | kfox1111: please do | 01:01 |
| sbezverk | in this case I can remove it completely from my PS | 01:01 |
| v1k0d3n | doesn't bother me. i just was just curious. | 01:01 |
| kfox1111 | sbezverk: ok. | 01:01 |
| v1k0d3n | instructions would be nice for you guys though. | 01:01 |
| v1k0d3n | seems like things are moving...too fast for instructions. | 01:02 |
| sdake | v1k0d3n ya - as you know , I don't like to speka about people's personal or professional business specifically | 01:02 |
| sdake | v1k0d3n quickstart is on teh way - hoepfully by monday :) | 01:02 |
| v1k0d3n | it's all good. | 01:02 |
| sdake | bmace_ hows it going on your end bro - arey ou all good? | 01:03 |
| *** adrian_otto has quit IRC | 01:03 | |
| sdake | bmace_ or all bad? | 01:03 |
| bmace_ | sdake: i think kfox1111 helped me figure out where my last attempt borked and it seemed to be around the networking, so i'm re-building my kube environment using canal rather than weave. | 01:04 |
| bmace_ | canal is what is used in the gate, so more likely to work | 01:04 |
| sdake | sbezverk you cn get your own domain too - 99$ /yr | 01:04 |
| kfox1111 | v1k0d3n: yeah. the problem is we need the dev env to be gated too. :/ | 01:04 |
| kfox1111 | v1k0d3n: that way, if we break something, we catch it, so other devs dont. | 01:04 |
| kfox1111 | v1k0d3n: but openstack's normal gates are vm's and nested virutalization is not available. :/ | 01:05 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: PS fixes issue discovered with keystone service package https://review.openstack.org/425964 | 01:05 |
| sdake | nested virt works with qemu mode | 01:05 |
| sdake | however the gate machines only have 8gb of ram | 01:05 |
| sdake | i've begged and pleaded for more ram in the vms | 01:05 |
| sbezverk | sdake: what a waste of money ;) | 01:05 |
| kfox1111 | sdake: yeah. it works for really trivial things like cirros. which the gate does. | 01:05 |
| *** jrobinson-afk is now known as jrobinson | 01:06 | |
| kfox1111 | sdake: it fails misurably to run all of openstack in. :/ | 01:06 |
| sdake | however, in the shoes of the infra folks, putting 16gb across all 9 cloud providers woudl reduce capacity in half | 01:06 |
| sbezverk | kfox1111: I removed overlapping part from my PS | 01:06 |
| sdake | kfox1111 right -the gate oOMS | 01:06 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Keystone-admin external support https://review.openstack.org/425836 | 01:06 |
| kfox1111 | sbezverk: added the search path stuff to mine. | 01:07 |
| sdake | and until zuul v3, ther is no way to make special snowflake 16gb ramvms | 01:07 |
| sdake | its not clear if uul v3 comes out if that would b ethe case | 01:07 |
| v1k0d3n | kfox1111: we brought this up as a concern in barcelona | 01:07 |
| kfox1111 | sdake: for this use case we really need bare metal. :/ | 01:07 |
| sdake | rather if infra would make 16b vms | 01:07 |
| sdake | v1k0d3n which concern | 01:07 |
| v1k0d3n | in fact, as sdake and inc0 are both aware, we offered to gate as a third party donation to kolla. | 01:07 |
| sdake | i've never been a big fan of 3rd party gating, however, if someone added third party gating its not like we can stop it :) | 01:08 |
| v1k0d3n | sizes, vm-based, no nested, etc to infra. | 01:08 |
| sdake | nor would I stop it | 01:08 |
| sdake | however, third party gating can't be voting | 01:08 |
| sdake | although the rest of our gates arenot voting | 01:08 |
| sdake | they are advisory at present | 01:08 |
| v1k0d3n | well, it would've helped and we offered it. it was being worked out. | 01:08 |
| sdake | v1k0d3n i guess i am missing some context because i just came on the channel | 01:09 |
| v1k0d3n | i just asked if things were working and if instructions were up to date. that's all. | 01:09 |
| v1k0d3n | it's no big deal | 01:10 |
| kfox1111 | working, yes. instructions up to date, no, but close. | 01:10 |
| sdake | v1k0d3n working = yes but not keystone - instructions = no not up to date except dev env | 01:10 |
| sdake | v1k0d3n the dev env you put together and portdirect documented is fantastic from my pov | 01:10 |
| sdake | and the docs have been perfected for that part | 01:10 |
| kfox1111 | sdake: keystone's working. just not when used in particular ways that there's a ps for. | 01:10 |
| v1k0d3n | still working for you guys? actually caught the docker issue, but want to merge in soon. waiting for a second reviewer who's centos-based. | 01:11 |
| sdake | kfox1111 i didn't say it wasn't working :) | 01:11 |
| v1k0d3n | actually sdake, you want to review, try and pass? | 01:11 |
| kfox1111 | sdake: btw, https://review.openstack.org/#/c/425836/ please. :) | 01:11 |
| sdake | v1k0d3n sure i can review the centos change when i return home from my folks | 01:11 |
| sdake | kfox1111 getting ready to eat - let me look | 01:11 |
| *** sayantani01 has joined #openstack-kolla | 01:12 | |
| v1k0d3n | sounds good. let me know | 01:12 |
| sdake | kfox1111 that is going to tkae a bit to review ;) | 01:12 |
| sdake | kfox1111 can you wait until after dinner? (about 1 hr) | 01:12 |
| kfox1111 | sdake: sure. | 01:12 |
| sdake | v1k0d3n what does try/pass mean there | 01:12 |
| sdake | v1k0d3n you mean test the patch that was submitted? | 01:12 |
| sdake | v1k0d3n if that is what you mean, then yup happy to do that when i get home - i don't have access to my lab here | 01:13 |
| sdake | v1k0d3n my wife is out of town at devconf until tuesday - so pretty much parents offered to help cook dinner a few days this week | 01:13 |
| sdake | v1k0d3n two kids at home + work = overload :) | 01:14 |
| v1k0d3n | sdake: yes, this is the PR that is waiting. | 01:14 |
| v1k0d3n | https://github.com/aric49/halcyon-kubernetes/tree/CentOS_issues | 01:14 |
| v1k0d3n | i already passed but we like to have two pass on it. | 01:14 |
| sdake | v1k0d3n looking | 01:14 |
| v1k0d3n | i don't use it anymore. | 01:14 |
| v1k0d3n | keeping it for whoever needs it now. | 01:14 |
| v1k0d3n | and the gluster folks...they use it for their kubernetes deployments. | 01:14 |
| v1k0d3n | i think port may use it for kolla-k8s. | 01:15 |
| v1k0d3n | we just use minikube now | 01:15 |
| *** Syffs has quit IRC | 01:15 | |
| kfox1111 | yeah. most of the time minikube gets the job done. and is really simple. | 01:16 |
| sdake | v1k0d3n you dont need which - the dev env? | 01:16 |
| sdake | v1k0d3n two tests sounds good - happy to provide a test of it when i get home | 01:17 |
| sdake | looking at the review, looks like many changes | 01:17 |
| sdake | v1k0d3n the ultimate test is a human test - not a a gate test | 01:18 |
| sdake | v1k0d3n they both go hand in hand | 01:18 |
| kfox1111 | sdake: agree/disagree. humans only scale so far. | 01:23 |
| kfox1111 | humans got to be able to deploy it, for sure. | 01:23 |
| sdake | kfox1111 agreed | 01:23 |
| kfox1111 | but gotta use automation to test things or else things are easily forgotten to be tested. | 01:24 |
| sdake | kfox1111 totally agree on that point, that is why kolla-ansible gating is so weak | 01:24 |
| kfox1111 | speaking of testing... your going to love this next ps... :) | 01:24 |
| sdake | dinenr is on - bbiaf | 01:27 |
| *** sdake has quit IRC | 01:32 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Check to see if Horizon is working in the gate. https://review.openstack.org/426025 | 01:43 |
| v1k0d3n | kfox1111: true...some humans don't scale | 01:50 |
| *** v1k0d3n has quit IRC | 01:51 | |
| *** v1k0d3n has joined #openstack-kolla | 01:53 | |
| *** sayantan_ has joined #openstack-kolla | 01:53 | |
| *** sayantani01 has quit IRC | 01:56 | |
| kbaegis | Hey guys, does anyone know how do disable a horizon plugin on a running environment? | 02:00 |
| kbaegis | tacker-horizon has a defective i18n script that is now returning 500s | 02:00 |
| kbaegis | here's the traceback: https://hastebin.com/ikiviladif.sql | 02:01 |
| *** sdake has joined #openstack-kolla | 02:04 | |
| sdake | v1k0d3n looking over that review, the syntax looks correct -although I'm not sure how that fixes the docker versioning problem | 02:05 |
| *** hfu has joined #openstack-kolla | 02:05 | |
| sdake | v1k0d3n although i have heard that docker-engine 0.13.1 fixes the problem | 02:06 |
| sdake | v1k0d3n althoug hthat ptach coesn't use docker-engine, it uses docker from centos | 02:07 |
| sdake | v1k0d3n i notice pull request #43 fixes the docker version problem | 02:08 |
| *** MasterOfBugs has quit IRC | 02:11 | |
| *** dims_ has joined #openstack-kolla | 02:14 | |
| *** PramodJ has quit IRC | 02:14 | |
| *** dims has quit IRC | 02:14 | |
| sdake | v1k0d3n reviewable looks pretty cool | 02:17 |
| sdake | like a more hipster version of gerrit ;-) | 02:17 |
| sdake | kfox1111 around? | 02:21 |
| sdake | I recall having this discussion before about documetnting the syntax of : https://review.openstack.org/#/c/425836/3/helm/microservice/keystone-create-endpoints-job/templates/keystone-create-endpoints.yaml | 02:21 |
| sdake | kfox1111 specifically kolla_get_val_get_str | 02:22 |
| *** tonanhngo has quit IRC | 02:27 | |
| *** lrensing has quit IRC | 02:32 | |
| sdake | sbezverk what was the fix for trunk ? | 02:32 |
| sdake | inc0 sort that out? | 02:32 |
| *** zhubingbing has joined #openstack-kolla | 02:36 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Keystone-admin external support https://review.openstack.org/425836 | 02:37 |
| *** zhubingbing has quit IRC | 02:38 | |
| kfox1111 | back. | 02:42 |
| *** absubram has joined #openstack-kolla | 02:43 | |
| kfox1111 | sdake: https://raw.githubusercontent.com/openstack/kolla-kubernetes/master/helm/kolla-common/templates/_common_val_get.rst | 02:44 |
| sdake | kfox1111 cool - thanks :) | 02:44 |
| sdake | kfox1111 i think we need to sort out how to get that linked in to the mian docs somehow, although no idea how | 02:45 |
| kfox1111 | yeah. | 02:45 |
| sdake | kfox1111 - how did the trunk problem get fixed? | 02:45 |
| kfox1111 | the keystone-admin patch that just merged. | 02:45 |
| kfox1111 | still a few race conditions left I think. | 02:46 |
| *** dims_ has quit IRC | 02:47 | |
| kfox1111 | sdake: check this one out... https://review.openstack.org/#/c/426025/ :) | 02:47 |
| *** absubram has quit IRC | 02:48 | |
| *** absubram has joined #openstack-kolla | 02:49 | |
| *** dims has joined #openstack-kolla | 02:50 | |
| sdake | kfox1111 - thought that was related to the docker images? | 02:51 |
| sdake | get me the anestaphine from the medlock! | 02:51 |
| sdake | brb | 02:51 |
| kfox1111 | sdake: no. its a test for checking horizon login works and whatever else we want to check in horizon. | 02:52 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Check to see if Horizon is working in the gate. https://review.openstack.org/426025 | 02:54 |
| sdake | kfox1111 - will review whne i get back to my house in 30 mins - brb | 02:56 |
| *** yuanying has quit IRC | 02:56 | |
| *** sdake has quit IRC | 03:00 | |
| *** goldyfruit has joined #openstack-kolla | 03:10 | |
| *** jtriley has quit IRC | 03:11 | |
| *** sdake has joined #openstack-kolla | 03:26 | |
| *** severion has joined #openstack-kolla | 03:27 | |
| *** crushil has quit IRC | 03:28 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Check to see if Horizon is working in the gate. https://review.openstack.org/426025 | 03:28 |
| *** crushil has joined #openstack-kolla | 03:28 | |
| sdake | kfox1111 that test looks pretty cool ! | 03:29 |
| sdake | kfox1111 not sure how it works in detail | 03:29 |
| sdake | kfox1111 i don't see anything popping out at me as in error | 03:29 |
| *** jrobinson has quit IRC | 03:31 | |
| *** jrobinson has joined #openstack-kolla | 03:35 | |
| kfox1111 | sdake: yeah. selenium is really slick. | 03:37 |
| kfox1111 | even more awesome when combined with k8s. | 03:37 |
| kbaegis | How does k8s do with ceph? | 03:38 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfigure action for telegraf https://review.openstack.org/424202 | 03:41 |
| kfox1111 | sdake: https://review.openstack.org/#/c/425847/ please. | 03:41 |
| kfox1111 | sdake: its passed all tests at least once. | 03:41 |
| kfox1111 | (I'm still working on stabilizing the gate. this ps is part of that) | 03:42 |
| kfox1111 | kbaegis: not sure what your asking. | 03:42 |
| kfox1111 | sdake: if we can get that through before tomorrow morning, we should have some cached containers to help stabilize the gate. | 03:42 |
| kfox1111 | sbezverk: can you review https://review.openstack.org/#/c/425847/ too please. | 03:43 |
| portdirect_travl | kfox1111: re selium, great work - but should it be in with the other services? | 03:44 |
| portdirect_travl | kfox1111: this is only a testing/gate tool right? | 03:44 |
| kfox1111 | portdirect_travl: in microservices you mean? | 03:46 |
| kfox1111 | yeah, intended just for testing. | 03:46 |
| portdirect_travl | this one: https://review.openstack.org/#/c/426025/ | 03:46 |
| kfox1111 | portdirect_travl: if you have a sec, could you do https://review.openstack.org/#/c/425847/ too please? | 03:46 |
| portdirect_travl | was thinking that im not sure the chart should be in helm/service, but tests or somewhere else? | 03:47 |
| kfox1111 | portdirect_travl: yeah. wasn't sure where to put it. :/ | 03:48 |
| *** prameswar has joined #openstack-kolla | 03:48 | |
| kfox1111 | sdake: http://www.seleniumhq.org/docs/07_selenium_grid.jsp :) | 03:50 |
| sdake | sup portdirect_travl | 03:51 |
| sdake | hows brno besides freezing cold | 03:51 |
| sdake | portdirect_travl my recommendation to you is to HOLD ON TO YOUR CHANGE | 03:51 |
| sdake | its more valuable then currency there ;-) | 03:52 |
| sdake | portdirect_travl good news - mostly have the dev environment working locally | 03:52 |
| sbezverk | portdirect_travl: please see clean up thing I put together: https://review.openstack.org/#/c/425875/ | 03:52 |
| sdake | portdirect_travl needs a bit of love | 03:52 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Check to see if Horizon is working in the gate. https://review.openstack.org/426025 | 03:53 |
| *** kbaegis has quit IRC | 03:54 | |
| sdake | hey fellas, that temporary issue with the gate (4.0.0b2) is about to be permanent as kolla-ansible is getting ready to release) | 03:55 |
| kfox1111 | when is the final release? | 03:56 |
| kfox1111 | it still has the horizon bug too. | 03:56 |
| sdake | kfox1111 sbezverk portdirect_travl and others - note the recheck on this patch: https://review.openstack.org/#/c/425809/1 | 03:56 |
| sdake | it should be green this time around | 03:57 |
| sdake | since kfox1111 fixed the problem with keystone admin | 03:57 |
| kfox1111 | sdake: yeah. it needs rebasing. | 03:57 |
| kfox1111 | its still using the wrong endpoint. | 03:57 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Fix for the periodic kolla-kubernetes processes. https://review.openstack.org/425847 | 03:59 |
| kfox1111 | nice. thanks. :) | 03:59 |
| kfox1111 | I've gotta head out. | 03:59 |
| *** sdake_ has joined #openstack-kolla | 03:59 | |
| kfox1111 | sbezverk: I'll finish reviewing your ps tomorrow. from what I've read so far, it looks good. :) | 03:59 |
| *** sdake has quit IRC | 04:02 | |
| *** eaguilar has joined #openstack-kolla | 04:08 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-ansible: Revert part of a murano patch which breaks horizon https://review.openstack.org/426048 | 04:10 |
| sdake_ | kfox1111 what needs rebasing? | 04:10 |
| sdake_ | kfox1111 it hasn't finished the recheck yet | 04:10 |
| sdake_ | kfox1111 what i am pointing out is patches don't need rebases - once a patch merges, the infrastructure rebases automatically | 04:11 |
| *** sdake has joined #openstack-kolla | 04:22 | |
| *** sdake_ has quit IRC | 04:23 | |
| *** goldyfruit has quit IRC | 04:29 | |
| *** mdnadeem has joined #openstack-kolla | 04:30 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Check to see if Horizon is working in the gate. https://review.openstack.org/426025 | 04:31 |
| *** prameswar has quit IRC | 04:37 | |
| *** eaguilar has quit IRC | 04:42 | |
| sdake | kfox1111 you will notice no rebsae, yet the gate is passing with your change and keystone endpoint list: https://review.openstack.org/#/c/425809/ | 04:42 |
| sdake | kfox1111 rebases are not necessary ^^ | 04:43 |
| sdake | the only time a rebase is necessary is if there is a merge conflict | 04:43 |
| *** absubram has quit IRC | 04:55 | |
| kfox1111 | sdake: weird. I could have swarn I've seen that not work before. | 04:58 |
| kfox1111 | oh well. good to know. thanks. | 04:58 |
| sdake | kfox1111 its been working like that for atleast 3 years ;-) | 04:58 |
| sdake | kfox1111 nonintuitve Iknow | 04:58 |
| *** tonanhngo has joined #openstack-kolla | 05:03 | |
| sdake | sup tonanhngo | 05:05 |
| sdake | yo - bmace_ check out this issue: https://github.com/att-comdev/halcyon-vagrant-kubernetes/issues/50 | 05:05 |
| sdake | bmace_ it may be what your running into | 05:05 |
| *** tonanhngo has quit IRC | 05:07 | |
| *** skramaja has joined #openstack-kolla | 05:15 | |
| kfox1111 | sdake, it can be normal for that to hapen for a bit until the cni driver loads. | 05:20 |
| kfox1111 | usually things settle down after a while. | 05:20 |
| sdake | kfox1111 its been 30 minutes same result | 05:20 |
| kfox1111 | ah. shouldn't take that long for sure. | 05:20 |
| kfox1111 | a couple minutes to pull all the cni containers tops. | 05:21 |
| sdake | yup i suspect somethign is busted | 05:21 |
| sdake | which is why i filed the issue :) | 05:21 |
| sdake | i'm going to look at debugging it today i guess- my son slept all damn day and i have to stay up with him now | 05:21 |
| kfox1111 | k. good luck. :) | 05:22 |
| sdake | ya | 05:22 |
| kfox1111 | heading to bed. nite all. | 05:22 |
| sdake | later dude | 05:22 |
| bmace_ | sdake: that is exactly the same issue i am hitting | 05:24 |
| sdake | bmace_ should log an issue for me to read about ;-) | 05:24 |
| sdake | bmace_ plus others | 05:24 |
| bmace_ | sdake: i tried swapping from weave to canal and it didn't fix it. | 05:24 |
| sdake | bmace_ i have an old vagrant environment here that works | 05:24 |
| sdake | want a tarball of it to help me debug it? :) | 05:24 |
| bmace_ | tarball of the one that isn't working? i have that myself :) | 05:25 |
| sdake | no th etarball of the working one | 05:25 |
| sdake | without vms in it | 05:25 |
| sdake | git diff ftw | 05:25 |
| sdake | or diff :) | 05:25 |
| bmace_ | ah, what does it have in it exactly? it is just the halycon-kubernetes repo? | 05:25 |
| sdake | or something - it canbe fixed | 05:25 |
| sdake | if it workedd before it will work again | 05:25 |
| sdake | its halcyon-vagrant-kubernetes and halcyon-kubernetes | 05:26 |
| sdake | as a submdoule | 05:26 |
| sdake | full working environment | 05:26 |
| bmace_ | even if you know what revision it is at, i can just check them out from there | 05:26 |
| bmace_ | commit hash | 05:26 |
| sdake | portdirect logged into m ybox and made some chagnes to the code | 05:27 |
| bmace_ | based on the diffs between the hash and current we might be able to figure out why it broke too | 05:27 |
| sdake | i suspect the changes are why it is working | 05:27 |
| sdake | not regressions in the codebase | 05:27 |
| bmace_ | ah, so you can just generate a diff? | 05:28 |
| bmace_ | or he did non code related changes? | 05:28 |
| sdake | i can give it a go - let me see git status | 05:28 |
| sdake | vagrant: http://paste.fedoraproject.org/537385/54949391/ -> http://paste.fedoraproject.org/537385/54949391 | 05:29 |
| sdake | against the submodule in that repo | 05:29 |
| bmace_ | hrm, those are both the same paste? | 05:30 |
| sdake | here are the changes to the submoduel: http://paste.fedoraproject.org/537386/85494993/ -> http://paste.fedoraproject.org/537386/85494993 | 05:30 |
| sdake | ya its a bug in fpaste | 05:30 |
| sdake | it prints out the url twice | 05:30 |
| sdake | its supposed to use an url shortener, not sure what happend with the url shorterner | 05:30 |
| bmace_ | interesting.. moving calico rather than weave or canal | 05:30 |
| sdake | its been broken for like a year | 05:30 |
| sdake | make it harder ;-) | 05:31 |
| bmace_ | let me give the all.yml changes a try | 05:31 |
| sdake | glad you spotted something i am bleary eyed | 05:31 |
| bmace_ | although, you are using iscsi right, and this says to setup ceph | 05:31 |
| sdake | thats a different environment actually :) | 05:32 |
| bmace_ | maybe that doesn't matter.. in the older / working env it was using ceph? | 05:32 |
| sdake | i am pretty sure yo ucan setup ceph | 05:32 |
| bmace_ | maybe i need to go back to that | 05:32 |
| sdake | and it will be ok | 05:32 |
| sdake | my other environment is as follows: | 05:32 |
| sdake | kubeadm + kubernetes on bare metal - not in vms | 05:32 |
| sdake | that is the one i've been developing the iscsi_workflow.sh patch on | 05:33 |
| sdake | it is possible iscsi_workflow doesn't work with the development environment | 05:33 |
| sdake | i really haen't tested that at all | 05:33 |
| sdake | if it doesn't should be easy to fix | 05:34 |
| bmace_ | but this kube based one was working for you also? if so, i'll go down this path | 05:34 |
| sdake | it is but i have no idea how it is setup | 05:34 |
| bmace_ | seems semi-obvious based on the config. most of it was the same as mine other than calico and ceph | 05:35 |
| sdake | cool let me try that in the master version of the dev env too | 05:36 |
| sdake | then we can get the workflow working on that | 05:36 |
| sdake | i'd like the workflow to be flexible so we can figure out how to use this thing outside the gate :) | 05:36 |
| openstackgerrit | Sajauddin proposed openstack/kolla: Kolla support for power server https://review.openstack.org/423239 | 05:37 |
| sdake | i am really struggling to figure out how to operate kolla-kubernetes however have leraned a bunch doing the workflow | 05:37 |
| bmace_ | yup, and being able to quickly redeploy kolla-k8s changes into a vagrant environment like this would be great for dev / testing. | 05:37 |
| sdake | agreed | 05:38 |
| bmace_ | yeah.. i do have to say, ansible is far simpler | 05:38 |
| sdake | the bare metal setup is super pokey turnaround times | 05:38 |
| bmace_ | at least to run / debug.. which is saying something, because ansible can be sort of annoying to debug at times too, but not nearly as bad as kolla-k8s.. but maybe better once i spend more time with it. | 05:38 |
| sdake | it would be cool to rework all that in ansible i think for the short term instead of shell | 05:38 |
| bmace_ | it will be a lot more work to make it easy to diagnose for operators though | 05:39 |
| sdake | ya - i do want an operator based approach or a entrypoint based approach (both) | 05:39 |
| sdake | i think I am finding i see alot of stuff that could be codified more easily in something besides shell | 05:39 |
| sdake | ansible for our gates - whee ;) | 05:39 |
| bmace_ | yeah.. i really don't like shell :/ python is heaven in comparison imo | 05:40 |
| *** saneax-_-|AFK is now known as saneax | 05:40 | |
| sdake | anyway going to give calico a go first - then calco + ceph | 05:40 |
| sdake | i'm not sure what the defaults re | 05:40 |
| sdake | re storage | 05:40 |
| bmace_ | yeah, that is a good plan. i may also.. it seems like a networking type complaint so maybe calico alone will fix it | 05:41 |
| sdake | i ahven't played with the halcyon dev enviornment much - but think it is solid from what i've done with it | 05:41 |
| *** prameswar has joined #openstack-kolla | 05:45 | |
| *** david-lyle has quit IRC | 05:46 | |
| *** crushil has quit IRC | 06:07 | |
| *** crushil has joined #openstack-kolla | 06:09 | |
| *** crushil has quit IRC | 06:15 | |
| *** prameswar has quit IRC | 06:39 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla: Docs updated to get coverage report https://review.openstack.org/425032 | 06:44 |
| *** unicell1 has quit IRC | 06:46 | |
| *** sayantan_ has quit IRC | 06:54 | |
| openstackgerrit | Joseph Robinson proposed openstack/kolla-ansible: Build deploy guide files and tox environment https://review.openstack.org/426105 | 07:00 |
| *** bmace_ has quit IRC | 07:07 | |
| *** bmace_ has joined #openstack-kolla | 07:07 | |
| *** hfu has quit IRC | 07:09 | |
| *** hfu has joined #openstack-kolla | 07:17 | |
| *** msimonin has quit IRC | 07:18 | |
| *** jrobinson has quit IRC | 07:22 | |
| *** sdake has quit IRC | 07:33 | |
| *** prameswar has joined #openstack-kolla | 07:45 | |
| *** msimonin has joined #openstack-kolla | 08:04 | |
| *** mgoddard has joined #openstack-kolla | 08:34 | |
| *** egonzalez has joined #openstack-kolla | 08:44 | |
| *** msimonin has quit IRC | 08:44 | |
| *** msimonin1 has joined #openstack-kolla | 08:44 | |
| *** matrohon has joined #openstack-kolla | 08:50 | |
| *** shardy_afk is now known as shardy | 08:52 | |
| *** msimonin1 has quit IRC | 08:56 | |
| *** Serlex has joined #openstack-kolla | 08:59 | |
| *** rmart04 has joined #openstack-kolla | 09:01 | |
| *** mgoddard has quit IRC | 09:05 | |
| *** zioproto has joined #openstack-kolla | 09:05 | |
| *** papacz has joined #openstack-kolla | 09:37 | |
| *** mgoddard has joined #openstack-kolla | 09:37 | |
| egonzalez | Jeffrey4l_: around? | 09:43 |
| *** hfu has quit IRC | 09:47 | |
| *** hfu has joined #openstack-kolla | 09:47 | |
| *** hfu has quit IRC | 09:48 | |
| *** hfu has joined #openstack-kolla | 09:48 | |
| *** hfu has quit IRC | 09:49 | |
| *** hfu has joined #openstack-kolla | 09:49 | |
| *** hfu has quit IRC | 09:49 | |
| *** hfu has joined #openstack-kolla | 09:50 | |
| *** hfu has quit IRC | 09:50 | |
| *** hfu has joined #openstack-kolla | 09:51 | |
| *** hfu has quit IRC | 09:51 | |
| *** hfu has joined #openstack-kolla | 09:51 | |
| *** hfu has quit IRC | 09:52 | |
| egonzalez | pomac: ping | 09:54 |
| pomac | egonzalez: pong =) | 09:55 |
| egonzalez | my issue was that keystone db was in latest schema and keystone container in previous version | 09:55 |
| pomac | egonzalez: oh - humm... This included the NoneType? | 09:55 |
| egonzalez | in master, when there is not defualt domain_id is automatically added | 09:56 |
| egonzalez | but need to add under identity section default_domain_id | 09:56 |
| *** msimonin has joined #openstack-kolla | 09:58 | |
| pomac | egonzalez: default_domain_id should default to 'default' according to the documentation afair | 09:58 |
| egonzalez | yep | 09:58 |
| egonzalez | for back compatible with v2 where default was added everywhere | 09:59 |
| pomac | egonzalez: we think that our problem started when we changed it to something else - but we're not certain | 10:00 |
| *** absubram has joined #openstack-kolla | 10:02 | |
| pomac | afk, bbl | 10:02 |
| *** absubram has quit IRC | 10:07 | |
| *** prameswar has quit IRC | 10:15 | |
| *** zhubingbing has joined #openstack-kolla | 10:20 | |
| egonzalez | zhubingbing: happy new year :) | 10:24 |
| zhubingbing | hi | 10:24 |
| zhubingbing | thank you very muc | 10:24 |
| zhubingbing | much | 10:24 |
| zhubingbing | ;) | 10:24 |
| zhubingbing | thanks | 10:24 |
| *** pbourke has quit IRC | 10:30 | |
| *** pbourke has joined #openstack-kolla | 10:31 | |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for grafana https://review.openstack.org/424158 | 10:35 |
| *** prameswar has joined #openstack-kolla | 10:39 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: Add freezer ansible role https://review.openstack.org/402224 | 10:42 |
| *** msimonin1 has joined #openstack-kolla | 10:43 | |
| *** msimonin has quit IRC | 10:44 | |
| *** shardy has quit IRC | 10:45 | |
| *** shardy has joined #openstack-kolla | 10:46 | |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for solum https://review.openstack.org/424167 | 10:53 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: Add freezer ansible role https://review.openstack.org/402224 | 10:53 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: Add freezer ansible role https://review.openstack.org/402224 | 10:56 |
| *** gfidente has joined #openstack-kolla | 10:56 | |
| *** gfidente has quit IRC | 10:56 | |
| *** gfidente has joined #openstack-kolla | 10:56 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: Add freezer ansible role https://review.openstack.org/402224 | 11:02 |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: Add freezer ansible role https://review.openstack.org/402224 | 11:10 |
| *** bachp has quit IRC | 11:10 | |
| *** vcn[m] has quit IRC | 11:10 | |
| *** retr0h has quit IRC | 11:10 | |
| *** hfu has joined #openstack-kolla | 11:13 | |
| *** sambetts|afk is now known as sambetts | 11:14 | |
| openstackgerrit | zhubingbing proposed openstack/kolla-ansible: Add freezer ansible role https://review.openstack.org/402224 | 11:19 |
| *** retr0h has joined #openstack-kolla | 11:22 | |
| *** saneax is now known as saneax-_-|AFK | 11:24 | |
| *** boycer has joined #openstack-kolla | 11:26 | |
| *** boycer has quit IRC | 11:32 | |
| des_consolado | hey I'm trying to follow this http://docs.openstack.org/newton/networking-guide/config-bgp-dynamic-routing.html | 11:45 |
| des_consolado | I've used kolla-ansible though and have everything running in containers | 11:45 |
| des_consolado | can anybody advise me on how I'm supposed to do this? I mean I can change the neutron.conf to have that extra ',bgp' easy enough, but this bgp agent ... | 11:45 |
| *** jtriley has joined #openstack-kolla | 11:45 | |
| des_consolado | I guess I need to set up a new container? Does this mean I need to build a new image too? | 11:45 |
| mliima | hi guys | 11:47 |
| des_consolado | hi | 11:49 |
| egonzalez | des_consolado: yed, for bgp agent will need another container | 11:53 |
| des_consolado | egonzalez: would this be a good thing to add to the project as a PR? | 11:54 |
| egonzalez | about config, you can use merge_configs, being said in /etc/kolla/config/neutron.conf you can apply your changes and will be merged | 11:54 |
| des_consolado | ah yes I've seen that | 11:54 |
| des_consolado | maybe there is like a neutron agent base image or something... will have a look | 11:55 |
| des_consolado | yeah there is a neutron base image | 11:55 |
| des_consolado | so I guess maybe I need to look at the Dockerfile for that and then edit it appropriately and tag it as a neutron bgp speaker image? | 11:56 |
| des_consolado | and then I need to write all the ansible playbooks in the right places for the various kolla-ansible actions? | 11:56 |
| *** bachp has joined #openstack-kolla | 11:56 | |
| *** vcn[m] has joined #openstack-kolla | 11:56 | |
| egonzalez | mostly that, create a dockerfile inside neutron/neutron-bgp-agent/Dockerfile or somename similar | 11:57 |
| egonzalez | and modify neutron role to apply changes whn bgp is enabled | 11:58 |
| des_consolado | yeah and use a var in globals and a default in group vars :)) | 11:58 |
| des_consolado | cool, exciting stuff, I will give this a shot | 11:58 |
| *** zhubingbing has quit IRC | 11:58 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible: Implement Keystone zero-downtime upgrade https://review.openstack.org/425446 | 12:00 |
| *** tonanhngo has joined #openstack-kolla | 12:02 | |
| *** Penfold has joined #openstack-kolla | 12:06 | |
| *** tonanhngo has quit IRC | 12:06 | |
| *** jtriley has quit IRC | 12:11 | |
| *** severion has quit IRC | 12:11 | |
| openstackgerrit | Merged openstack/kolla-ansible: Add freezer ansible role https://review.openstack.org/402224 | 12:12 |
| *** shardy is now known as shardy_lunch | 12:18 | |
| *** kbaegis has joined #openstack-kolla | 12:21 | |
| *** kbaegis has quit IRC | 12:23 | |
| *** kbaegis has joined #openstack-kolla | 12:23 | |
| *** eaguilar has joined #openstack-kolla | 12:23 | |
| *** zhubingbing has joined #openstack-kolla | 12:27 | |
| *** eaguilar has quit IRC | 12:29 | |
| *** rhallisey has joined #openstack-kolla | 12:47 | |
| openstackgerrit | Nenad Radojevic proposed openstack/kolla-ansible: Use neutron user in neutron_lbaas.conf and overwrite defaults https://review.openstack.org/425150 | 12:49 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Fix the undefined error for vmtp_vm_availability_zone https://review.openstack.org/425715 | 12:53 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Optimize reconfiguration for octavia https://review.openstack.org/423846 | 13:02 |
| *** srwilkers has joined #openstack-kolla | 13:03 | |
| *** srwilkers has quit IRC | 13:03 | |
| *** srwilkers has joined #openstack-kolla | 13:04 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix code format ceilometer_compute dockerfile https://review.openstack.org/426224 | 13:04 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Fix the missing certificate files https://review.openstack.org/426225 | 13:06 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible: Fix the missing certificate files for octavia https://review.openstack.org/426225 | 13:07 |
| srwilkers | morning gents | 13:08 |
| *** mgiles has joined #openstack-kolla | 13:09 | |
| des_consolado | hi | 13:11 |
| egonzalez | zhubingbing: why abandoned the PS? | 13:11 |
| *** goldyfruit has joined #openstack-kolla | 13:11 | |
| zhubingbing | ;) | 13:11 |
| *** goldyfruit has quit IRC | 13:11 | |
| Jeffrey4l_ | egonzalez, do u have any idea about https://bugs.launchpad.net/kolla-ansible/+bug/1659799 ? | 13:13 |
| openstack | Launchpad bug 1659799 in kolla-ansible "containers are not replaced while upgrading" [Critical,New] | 13:13 |
| zhubingbing | is only applied at binary install atm | 13:13 |
| egonzalez | Jeffrey4l_: when is copy_always only a restart is applied https://github.com/openstack/kolla-ansible/blob/master/ansible/library/kolla_docker.py#L580 | 13:15 |
| Jeffrey4l_ | egonzalez, oh, right. | 13:15 |
| egonzalez | don't really know how to proceed on that, just changed mine to copy_once to finish keystone upgrade | 13:15 |
| Jeffrey4l_ | need fix the restart_or_recreate_container method. | 13:16 |
| egonzalez | btw, happy new year | 13:16 |
| Jeffrey4l_ | egonzalez, thanks | 13:16 |
| Jeffrey4l_ | i will push a fix later. | 13:16 |
| *** shardy_lunch is now known as shardy | 13:17 | |
| *** mdnadeem has quit IRC | 13:23 | |
| sbezverk | Jeffrey4l_: I think I have a good news for you ;) rememeber that iscsi issue? | 13:34 |
| *** sdake has joined #openstack-kolla | 13:34 | |
| sdake | morning peeps | 13:35 |
| sbezverk | srwilkers: good morning! do you mind to review a couple of my PS please? | 13:35 |
| Jeffrey4l_ | sbezverk, hrm. which error? | 13:35 |
| Jeffrey4l_ | morning | 13:35 |
| sbezverk | Jeffrey4l_: when you have iscsi initiator utils installed on the host then cinder volume fails | 13:35 |
| Jeffrey4l_ | have u solved it? | 13:35 |
| sdake | Jeffrey4l_ i thought you would be on pto | 13:36 |
| sdake | Jeffrey4l_ since its chinese new year - grats! | 13:36 |
| sdake | Jeffrey4l_ since your here - can you look at this 1 liner | 13:36 |
| Jeffrey4l_ | sdake, yep. | 13:36 |
| Jeffrey4l_ | thanks. | 13:36 |
| sbezverk | Jeffrey4l_: I think so :) instead of completely removing it completely, you need to stop iscsid.socket. | 13:36 |
| Jeffrey4l_ | sbezverk, hrm. iscid should not be run on the host. | 13:37 |
| sdake | https://review.openstack.org/#/c/426048/ | 13:37 |
| sbezverk | systemd is listening on it and that is what prevents iscsid in the container to start properly | 13:37 |
| srwilkers | morning sdake | 13:37 |
| srwilkers | sbezverk, of course dude! good morning! | 13:37 |
| sdake | Jeffrey4l_ note i dont know if it works or not | 13:37 |
| sdake | sup srwilkers | 13:37 |
| srwilkers | sdake, not me yet | 13:37 |
| sdake | Jeffrey4l_ it looks to me by talking with dave-lyle yesterday that horizonz is completely busted in master | 13:38 |
| Jeffrey4l_ | sdake, i know the issue. but move POLICY_FILES_PATH to /etc/openstack-dashboard may cause other issue. | 13:38 |
| sbezverk | Jeffrey4l_: for kube case, I must have initizator tools on the host as it is required to support kubernetes iscsi backend | 13:38 |
| sdake | Jeffrey4l_ they are already there per the json descriptor | 13:38 |
| sbezverk | so I need to dig deeper than just removing it comepltely from the host | 13:38 |
| Jeffrey4l_ | I think /etc/openstack-dashboard is OK. but need copy the horizon policy, which located in openstack-dashboard/conf/*.policy to /etc/o-d | 13:39 |
| sdake | Jeffrey4l_ however if yo uahve a better way to fix it, i'm all ears ) | 13:39 |
| Jeffrey4l_ | sdake, this merged yours. and add "moving all horizon policy to /etc/o-d folder" in kolla | 13:39 |
| sdake | Jeffrey4l_ looking inside the images, they are actually in /etcopensack-dashboard already | 13:39 |
| Jeffrey4l_ | let me check.. | 13:40 |
| sdake | Jeffrey4l_ note i was using the 2.0.2 images | 13:40 |
| sdake | and i htink i looked at 3.0.2 too | 13:40 |
| *** prameswar has quit IRC | 13:40 | |
| *** jheroux has joined #openstack-kolla | 13:40 | |
| sdake | this may be a version incompatibility with horizon and different versions of kolla | 13:41 |
| sdake | i wasn't sure hence asked for you to take a look ) | 13:41 |
| sdake | ;) | 13:41 |
| Jeffrey4l_ | sdake, yep. you are correct. | 13:41 |
| Jeffrey4l_ | +2ed for your patch. | 13:41 |
| sdake | Jeffrey4l_ i think the ultimate test - which i didnt do - was to test that patch against masater | 13:41 |
| sdake | Jeffrey4l_ i don't want to break kolla-ansible :) | 13:42 |
| Jeffrey4l_ | you fixed it ;) | 13:42 |
| sdake | Jeffrey4l_ kolla-kuberfnetes has a gate job incoming for testing horizon - and a login thereof | 13:42 |
| sdake | if we can cross repo gate both repos over itme, it will be super powerful | 13:43 |
| Jeffrey4l_ | sbezverk, iscsid mount /run:/run:shared in kolla-ansible. | 13:43 |
| sdake | we will have two deliverables all gating each other | 13:43 |
| sdake | rather 3 | 13:43 |
| sdake | time for bed :) | 13:43 |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfigure action for cloudkitty https://review.openstack.org/422516 | 13:43 |
| sdake | ;-) | 13:43 |
| Jeffrey4l_ | yep. hoping that. kolla-k8s deployment gate should be ran in kolla project. | 13:43 |
| Jeffrey4l_ | sdake, night | 13:43 |
| sdake | i just woke up | 13:44 |
| sdake | just can't compose a sentence | 13:44 |
| sdake | ;-) | 13:44 |
| Jeffrey4l_ | sdake, could u approve https://blueprints.launchpad.net/kolla/+spec/add-chrony-service , it is already implemented. | 13:45 |
| egonzalez | Jeffrey4l_: done | 13:47 |
| *** sdake_ has joined #openstack-kolla | 13:48 | |
| Jeffrey4l_ | egonzalez, thanks. | 13:48 |
| *** sdake has quit IRC | 13:48 | |
| sbezverk | srwilkers: do you need links? | 13:50 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the murano policy file path https://review.openstack.org/426237 | 13:52 |
| srwilkers | sbezverk, nope. poking through them slowly :) | 13:52 |
| sbezverk | srwilkers: replied to your first review :) | 13:53 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add neutron vpnaas code into neutron-server container https://review.openstack.org/419977 | 13:54 |
| *** lrensing has joined #openstack-kolla | 13:54 | |
| openstackgerrit | Merged openstack/kolla-ansible: Fix the undefined error for vmtp_vm_availability_zone https://review.openstack.org/425715 | 13:56 |
| *** prameswar has joined #openstack-kolla | 13:56 | |
| openstackgerrit | Merged openstack/kolla-ansible: Revert part of a murano patch which breaks horizon https://review.openstack.org/426048 | 13:56 |
| *** goldyfruit has joined #openstack-kolla | 13:58 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add neutron vpnaas code into neutron-server container https://review.openstack.org/419977 | 14:00 |
| *** lamt has joined #openstack-kolla | 14:01 | |
| sdake_ | srwilkers re the dev env, there is issue #50 | 14:10 |
| sdake_ | srwilkers bmace_ foudn out last night from my diff while i was bleary eyed that the dev env onlly works with calico for some reason | 14:11 |
| sdake_ | any thoughts on that? | 14:11 |
| sdake_ | it doesn' tappear to work with weave or flannel iiuc ;) | 14:11 |
| sdake_ | but maybe its not flanenl | 14:11 |
| sdake_ | I can't keep all the CNI layers straight :) | 14:11 |
| sdake_ | rather implementations | 14:11 |
| sdake_ | https://github.com/att-comdev/halcyon-vagrant-kubernetes/issues/50 | 14:13 |
| srwilkers | sdake_, let me look at the issue. tl;dr though, if its not working, it needs to be fixed asap. let me finish reviewing sbezverk's patch sets then ill see what's up with it | 14:13 |
| sdake_ | srwilkers when using calico - it works :) | 14:13 |
| sdake_ | srwilkers i think portdirect_travl had kne wabou t this issue earlier | 14:13 |
| sdake_ | srwilkers as he had loggedin to my box remotely a few weeks ago and set that setting | 14:14 |
| sdake_ | srwilkers and he mentioned something about it - which i have since forgotten | 14:14 |
| *** absubram has joined #openstack-kolla | 14:14 | |
| sdake_ | srwilkers atleast now its in an issue tracker ;-) | 14:14 |
| sdake_ | srwilkers also the submoduel needs an update | 14:15 |
| sdake_ | srwilkers i don't know how to do that without -dirty being appended to the submodule for whatever reason | 14:15 |
| srwilkers | sdake_, yeah, thats a start. but need to look at ensuring this is managed long term. i mentioned getting gates set up to make sure it's compatible with kolla-kubernetes on our end | 14:15 |
| srwilkers | i need to make sure that process is expedited | 14:15 |
| sdake_ | srwilkers there are ways to gate inside github | 14:16 |
| sdake_ | srwilkers i have seen them used by other projects | 14:16 |
| sdake_ | srwilkers i dont know how its done - i know travis-ci is one option | 14:16 |
| srwilkers | sdake_, yep, actually part of whats had me busy lately | 14:16 |
| srwilkers | sdake_, sbezverk brb coffee | 14:18 |
| *** srwilkers is now known as srwilkers_coffee | 14:18 | |
| *** absubram has quit IRC | 14:18 | |
| *** rhallisey has quit IRC | 14:21 | |
| *** absubram has joined #openstack-kolla | 14:24 | |
| *** msimonin1 has quit IRC | 14:26 | |
| *** rhallisey has joined #openstack-kolla | 14:28 | |
| *** rhallisey has quit IRC | 14:28 | |
| *** rhallisey has joined #openstack-kolla | 14:28 | |
| *** saneax-_-|AFK is now known as saneax | 14:29 | |
| *** rhallisey has quit IRC | 14:30 | |
| *** fguillot has joined #openstack-kolla | 14:31 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix code format ceilometer_compute dockerfile https://review.openstack.org/426224 | 14:38 |
| *** jtriley has joined #openstack-kolla | 14:39 | |
| *** saneax is now known as saneax-_-|AFK | 14:40 | |
| *** lrensing has quit IRC | 14:40 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix code format ceilometer_compute dockerfile https://review.openstack.org/426224 | 14:40 |
| *** boycer has joined #openstack-kolla | 14:41 | |
| *** srwilkers_coffee is now known as srwilkers | 14:42 | |
| *** lrensing has joined #openstack-kolla | 14:43 | |
| boycer | Hello all. Looking into using kolla to deploy a cloud in a production environment. Any advice? Any gotchas? Any other deployment method I should use instead? | 14:43 |
| sdake_ | boycer kolla-ansible is super stable, kolla-kubernetes is a work in progress | 14:44 |
| sdake_ | boycer if you ar edeploying a production cloud use kolla-ansible | 14:44 |
| sdake_ | boycer the quickstart in the docs should get you going | 14:44 |
| sdake_ | boycer the docs are here http://docs/openstack/org/developer/kolla | 14:45 |
| sdake_ | boycer note several folks are working on cleaning up the kolla-ansible docs to make them more consumable | 14:45 |
| boycer | sdake_: great, thanks a lot. I'll go read the kolla-ansible doc. | 14:46 |
| matrohon | hi | 14:47 |
| boycer | sdake_: I may be back ;) | 14:48 |
| matrohon | we test deployments with kolla 3.0.3, and we meet an issue with kolla-toolbox | 14:48 |
| sdake_ | srwilkers i noticed you changed the etherpad to "collobartion between openstack operators and helm" | 14:48 |
| sdake_ | srwilkers which sounds fine to me - however, most of the people on the workgroup arenot operators | 14:48 |
| matrohon | it's like the kolla-start cmd is not run with the ansible user | 14:48 |
| sdake_ | srwilkers here: https://etherpad.openstack.org/p/openstack-helm | 14:49 |
| sdake_ | srwilkers what was your thinking behind that change - I'd like that workgroup to be more of a neutral ground for openstack to comunicate with helm | 14:49 |
| sdake_ | indepndent of whatever project we have | 14:50 |
| sdake_ | however i'm not sure operators in genreeal care ;) | 14:50 |
| matrohon | but with a user that only exists on the host! | 14:50 |
| matrohon | does anybody know a corresponding bug in kolla? | 14:50 |
| matrohon | sdake_, ^ | 14:51 |
| sdake_ | matrohon have to get son ready for school no time to debug sadly | 14:51 |
| matrohon | sdake_, np we keep investigating... | 14:51 |
| sdake_ | matrohon then have meetings in the morning | 14:51 |
| sdake_ | matrohon generally overloaded :) | 14:51 |
| matrohon | sdake_, I just wanted to know if it was a known bug at kolla | 14:52 |
| sdake_ | srwilkers ya thats kind of weird to oand so was deployment projects :) | 14:53 |
| srwilkers | sdake_, yeah, fair point on operators | 14:53 |
| srwilkers | trying to think of a better wording for that | 14:53 |
| sdake_ | srwilkers ya - I think the workgroup is what we make of it as a community | 14:53 |
| sdake_ | and that hsould be one community (openstack) | 14:53 |
| sdake_ | how about OpenStack upstreams? | 14:54 |
| srwilkers | well, its key to remember that some of these groups are outside of the openstack community. it should ideally be a general call where anyone interested in leveraging helm for openstack deployments can participate, even if they're using a solution not under the openstack umbrella | 14:54 |
| srwilkers | that works | 14:54 |
| sdake_ | srwilkers ya i totally ge tthat, and want neutral groudn as I stated | 14:55 |
| openstackgerrit | zhubingbing proposed openstack/kolla: remove /var/log/trove in trove dockerfile` https://review.openstack.org/426254 | 14:55 |
| sdake_ | language can be overly powerful sometimes | 14:55 |
| sdake_ | and this is one of those casees :) | 14:55 |
| *** athomas has joined #openstack-kolla | 14:55 | |
| srwilkers | sdake_, yes it can :) | 14:55 |
| sdake_ | anyway thats the best suggestoin i've got for now | 14:56 |
| srwilkers | sdake_, works for me. i think you nailed it | 14:56 |
| sdake_ | someone may come along and want us to change it yet again :) | 14:56 |
| *** crushil has joined #openstack-kolla | 15:01 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Fix swift dockerfile https://review.openstack.org/426260 | 15:01 |
| sbezverk | srwilkers: any feedback on my cleanup service ;) ?? | 15:02 |
| srwilkers | sbezverk, looking through it now, almost done :) | 15:04 |
| *** absubram has quit IRC | 15:06 | |
| *** sayantani01 has joined #openstack-kolla | 15:09 | |
| *** zhubingbing has quit IRC | 15:10 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: rootwrap filters is removed in networking-sfc https://review.openstack.org/426263 | 15:13 |
| *** l4yerffeJ_ has joined #openstack-kolla | 15:14 | |
| *** dasTor_ has joined #openstack-kolla | 15:14 | |
| srwilkers | sbezverk, left a few comments inline but looks good to me otherwise :) | 15:14 |
| *** l4yerffeJ has quit IRC | 15:15 | |
| *** dasTor has quit IRC | 15:15 | |
| *** zhubingbing has joined #openstack-kolla | 15:15 | |
| *** msimonin has joined #openstack-kolla | 15:17 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: remove /var/log/trove in trove dockerfile https://review.openstack.org/426254 | 15:19 |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for kuryr https://review.openstack.org/424161 | 15:21 |
| *** hieulq_ has joined #openstack-kolla | 15:23 | |
| *** srwilkers is now known as srwilkers_brb | 15:23 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding cleanup service for cinder https://review.openstack.org/425875 | 15:26 |
| sayantani01 | Hi Jeffrey4l_ Can you please review this patch when you have time? :) https://review.openstack.org/#/c/424384/ | 15:27 |
| sdake_ | well gotta sya kolla-kubernetes is looking good :) | 15:28 |
| sdake_ | it works - which is the first step :) | 15:28 |
| sdake_ | kfox1111 sbezverk since the gate is your babies, I wanted your input on the work there | 15:29 |
| sdake_ | kfox1111 sbezverk how do you feel about ansible to run the gate jobs longer term | 15:29 |
| sdake_ | i find the current shell script to be super duper complicated to process mentally | 15:29 |
| Jeffrey4l_ | sayantani01, done. | 15:30 |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfigure action for telegraf https://review.openstack.org/424202 | 15:30 |
| kfox1111 | heh. | 15:30 |
| sdake_ | its not a criticism - shell is always hard :) | 15:31 |
| kfox1111 | ansible based orchestration was the origional plan. | 15:31 |
| sdake_ | yup i hear ya | 15:31 |
| *** zioproto has quit IRC | 15:31 | |
| kfox1111 | theres a lot of progress on the helm+kuberntees-entrypoint workflow. | 15:31 |
| kfox1111 | is that more understandable? | 15:32 |
| sdake_ | i think that was a case of too many people making decisions wihtout understanding what was there | 15:32 |
| kfox1111 | heh. thats for true. :/ | 15:32 |
| sdake_ | kfox1111 i havent 'taken a look at that yet - been lazer focused on iscis workflow | 15:32 |
| kfox1111 | sdake_: do have a look at ceph_workflow_service.sh | 15:32 |
| kfox1111 | the top half will look just like iscsi_workflow. | 15:33 |
| kfox1111 | but the bottom half should look much more simple. | 15:33 |
| sdake_ | i have been reviewing that code actually and know what is there | 15:33 |
| *** lamt has quit IRC | 15:33 | |
| kfox1111 | ideally, we can combine it all to one helm install kolla/compute-kit | 15:33 |
| sdake_ | if you meant entrypoint workflow by that - then yup thats alot simpler | 15:33 |
| kfox1111 | yeah. | 15:33 |
| sdake_ | i thitnk we still want to maintian the gate jobs for the microcharts | 15:34 |
| kfox1111 | agreed. | 15:34 |
| sdake_ | that provides alot more coverage of the system | 15:34 |
| kfox1111 | the majority of the remaining complexity is in the configmap/secret stuff. | 15:34 |
| *** ipsecguy has quit IRC | 15:34 | |
| kfox1111 | we haven't migrated that stuff to helm yet. | 15:34 |
| sdake_ | ya i'm going to take a look at unwinding that into kolla itself | 15:34 |
| kfox1111 | once that's done, it should be a lot simpler. | 15:34 |
| sdake_ | rather kolla-kubernetes | 15:34 |
| sdake_ | after the docs are in working order | 15:34 |
| sdake_ | we need to exit genconfig imo | 15:35 |
| sdake_ | although ansible does a fine job of getting us there | 15:35 |
| kfox1111 | having to setup a bunch of python stufff, run genconfig, configure things, fix up some config, and load them is really complicated. | 15:35 |
| sdake_ | i agree | 15:35 |
| kfox1111 | yeah. | 15:35 |
| kfox1111 | I have some thoughts on how to get there, but need to prototype things and/or talk about it at the ptg. | 15:36 |
| sdake_ | i'm not sure if i will start on that work prior to ptg | 15:36 |
| sdake_ | so it may be an opportune time to look at it then | 15:36 |
| sdake_ | although I may get started earlier - depends on my workload :) | 15:36 |
| kfox1111 | yeah. I just usually like to have a good idea of some options before I go into a discussion. :) | 15:37 |
| kfox1111 | I talked trhough some options with Jeffrey4l_ to see what oslo could support. | 15:37 |
| sdake_ | my current work is as follows | 15:37 |
| *** satyar has joined #openstack-kolla | 15:37 | |
| sdake_ | 1. sort out the dev env to ramp people up | 15:37 |
| sdake_ | 2. sort out the docs so peopel can actualluy install kolla-kubernetes | 15:37 |
| sdake_ | 3. drop the dep on kolla-ansible | 15:38 |
| sdake_ | gotta take kid to school brb | 15:38 |
| satyar | hi sdake_ | 15:38 |
| kfox1111 | me too. | 15:38 |
| satyar | Hi SamYaple | 15:38 |
| kfox1111 | sdake_: I think if we can get the dev env mostly into a container, and a helm package added to kubernetes/charts, | 15:40 |
| kfox1111 | it would be super easy to get started with the dev env. | 15:40 |
| kfox1111 | spawn up a simple k8s system, helm install stable/kolla-dev-env | 15:40 |
| kfox1111 | kubectl exec -it kolla-dev-env /bin/bash | 15:40 |
| kfox1111 | and go. | 15:40 |
| kfox1111 | nothing to install. | 15:40 |
| Serlex | that would be nice | 15:41 |
| *** zhubingbing has quit IRC | 15:41 | |
| sbezverk | sdake_: absolutely against it | 15:43 |
| sayantani01 | Thanks Jeffrey4l_ | 15:43 |
| sayantani01 | :) | 15:43 |
| Jeffrey4l_ | np. | 15:43 |
| sbezverk | sdake_: I think after we clean up and use common code at the gate, it will be perfect. | 15:44 |
| sbezverk | kfox1111: about your comment. the reason thise keystone vars are there baecuase I did not want to polute all.yaml | 15:45 |
| *** srwilkers_brb is now known as srwilkers | 15:45 | |
| matrohon | egonzalez, hi | 15:47 |
| egonzalez | matrohon: sup | 15:47 |
| matrohon | egonzalez, I'm trying to figure out what's going wrong with latest dnsmasq in fedora | 15:48 |
| matrohon | https://bugs.launchpad.net/kolla/+bug/1655794 | 15:48 |
| openstack | Launchpad bug 1655794 in kolla "dnsmasq dhcp lease is not cleaned up after instance termination" [Undecided,New] | 15:48 |
| *** absubram has joined #openstack-kolla | 15:49 | |
| *** saneax-_-|AFK is now known as saneax | 15:49 | |
| matrohon | egonzalez, can you confirm that can't hit the bug with centos 7.3 and dnsmasq 2.66? | 15:49 |
| *** hieulq_ has quit IRC | 15:49 | |
| sdake_ | sbezverk it may be perfect for the gate, but no tperfect for a dev env | 15:49 |
| sdake_ | anyway have a meeting will be back :) | 15:50 |
| sdake_ | kfox1111 dev env added to kubernetes helm charts would be interesting to me | 15:50 |
| sdake_ | kfox1111 not sure if that is something the helm maintainers would tkae or not | 15:51 |
| egonzalez | matrohon: deploying | 15:51 |
| srwilkers | sdake_, ive been thinking about development environment lately too. when you get back, we should chat | 15:53 |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfiguration for octavia https://review.openstack.org/423846 | 15:53 |
| sbezverk | sdake_: if you talk about dev env, why you suggest it for the gate? | 15:53 |
| sdake_ | srwilkers the hwole world thinks bout dev env :) | 15:54 |
| sdake_ | sbezverk at present the scripts are integrated | 15:54 |
| sdake_ | sbezverk i'll be back in 1 hr, we can chat then | 15:54 |
| sbezverk | skake_ ok | 15:54 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding cleanup service for cinder https://review.openstack.org/425875 | 16:02 |
| *** robcresswell has joined #openstack-kolla | 16:04 | |
| *** adrian_otto has joined #openstack-kolla | 16:05 | |
| *** v1k0d3n has quit IRC | 16:06 | |
| *** adrian_otto has quit IRC | 16:09 | |
| *** rhallisey has joined #openstack-kolla | 16:11 | |
| *** adrian_otto has joined #openstack-kolla | 16:12 | |
| *** lamt has joined #openstack-kolla | 16:12 | |
| *** adrian_otto has quit IRC | 16:13 | |
| *** crushil has quit IRC | 16:15 | |
| *** crushil has joined #openstack-kolla | 16:16 | |
| *** v1k0d3n has joined #openstack-kolla | 16:16 | |
| *** v1k0d3n has quit IRC | 16:21 | |
| *** saneax is now known as saneax-_-|AFK | 16:21 | |
| *** v1k0d3n has joined #openstack-kolla | 16:21 | |
| *** prameswar has quit IRC | 16:25 | |
| *** zioproto has joined #openstack-kolla | 16:28 | |
| *** david-lyle has joined #openstack-kolla | 16:29 | |
| *** zioproto has quit IRC | 16:32 | |
| *** zioproto has joined #openstack-kolla | 16:33 | |
| egonzalez | matrohon: is working fine for me http://paste.openstack.org/show/596747/ | 16:35 |
| egonzalez | there are my centos and dnsmasq versions | 16:35 |
| egonzalez | in both container and host | 16:35 |
| matrohon | ok, thanks | 16:36 |
| matrohon | egonzalez, ubuntu 16.04, dnsmasq 2.76 | 16:37 |
| matrohon | egonzalez, I'm not sure wether dnsmasq is faulty here | 16:38 |
| matrohon | egonzalez, if it is, the bug should have been reported for a long time in neutron | 16:39 |
| kfox1111 | back. | 16:39 |
| srwilkers | kfox1111, wb | 16:41 |
| bmace_ | sdake_: looks like the swap to calico may not have fixed it. going to look at my env shortly but wasn't working when i went to sleep. | 16:41 |
| *** egonzalez has quit IRC | 16:41 | |
| kfox1111 | sdake_: doesn't hurt to ask them. | 16:41 |
| kfox1111 | sdake_: worst case, we maintain the package ourselves. still would be easier then deploying that part by hand I think. | 16:41 |
| sbezverk | kfox1111: clean up comment. | 16:41 |
| kfox1111 | lookin | 16:42 |
| sbezverk | kfox1111: I think the root of this issue is common/global/ etc.. if all.yaml had global section in a normal way, then all these would not be necessary | 16:43 |
| kfox1111 | srwilkers: would not searching for 'delete' still work as its in the package name? | 16:44 |
| *** srwilkers has quit IRC | 16:44 | |
| kfox1111 | sbezverk: just for context, we're talkg about the values.yaml issue I raised? | 16:44 |
| *** srwilkers has joined #openstack-kolla | 16:44 | |
| sbezverk | kfox1111: it looks to me that all.yaml is less global api | 16:44 |
| srwilkers | back, dropped | 16:44 |
| sbezverk | kfox1111: correct | 16:44 |
| kfox1111 | sbezverk: k. | 16:45 |
| kfox1111 | sbezverk: I think the issue woudl be resolved if we made an example user's values.yaml | 16:45 |
| kfox1111 | alot of the stuff you want to put in services/*/values.yaml would be more clear there I think? | 16:45 |
| kfox1111 | the same file coudl then be used for all services too. | 16:45 |
| kfox1111 | I see the service packages largly as just for launching the sets of microservices with deps. | 16:46 |
| sbezverk | kfox1111: do not follow, where you want to have user's values.yaml? | 16:46 |
| kfox1111 | the config belongs to either the microservices, or the users values.yaml. | 16:46 |
| kfox1111 | so, back in the before helm time, we had an /etc/kolla-kubernetes/kolla-kubernetes.yaml. | 16:47 |
| sbezverk | kfox1111: can we have two values.yamls? | 16:47 |
| kfox1111 | it was a single place for the user to configure everything. | 16:47 |
| *** absubram has quit IRC | 16:47 | |
| kfox1111 | I think we shoudl provide an example of something like that, similr to | 16:47 |
| kfox1111 | tests/bin/common_workflow_config.sh but just as a flat yaml. | 16:48 |
| kfox1111 | with all the default settings in it? | 16:48 |
| kfox1111 | then if a user wants to tweak inidividual settings, they know which settings they can tweak. | 16:48 |
| kfox1111 | then we make it a documented standard practice to allways pass that values.yaml file when launching. | 16:48 |
| sbezverk | kfox1111: I get the idea of user's values.yaml what I do not get where you want to put it? | 16:48 |
| kfox1111 | oh. in the repo? | 16:49 |
| sbezverk | kfox1111: which folder? | 16:49 |
| kfox1111 | not sure. | 16:49 |
| sbezverk | and how user/service will know where it is | 16:49 |
| kfox1111 | make an /examples dir? | 16:49 |
| kfox1111 | services wont know. user shoudl always pass. | 16:49 |
| kfox1111 | but the user should be informed via docs. | 16:49 |
| kfox1111 | the dev env guide for example. | 16:50 |
| sbezverk | kfox1111: but look, you said you want vlaues.yaml with user configurable default | 16:50 |
| kfox1111 | which values.yaml? | 16:50 |
| kfox1111 | (too many. :/ ) | 16:50 |
| kfox1111 | maybe we make the example's one have everythign commented out? | 16:50 |
| sbezverk | there should be a location where this customized file is stored for the service to use it when it gets installed | 16:50 |
| kfox1111 | that way they know what the defaults are, but they arnt overriding whtas in the packages by default? | 16:51 |
| kfox1111 | user should specify it when launching the service. helm install kolla/neutron --values examples/mycloud.yaml | 16:51 |
| sbezverk | kfox1111: I think we again talk about different things.. it seems you talk about documenting for the use, I talk about this practical issue I have now with supplying bunch of default but confiugrable vars to a specific service | 16:52 |
| sbezverk | can we address one thing at the time? | 16:52 |
| kfox1111 | k. | 16:52 |
| kfox1111 | the problem I saw in your review, | 16:52 |
| kfox1111 | is you had a bunch of values that you set, that weren't overrides. | 16:52 |
| kfox1111 | so extra code, that we have to maintain, whith no benifit? | 16:52 |
| sbezverk | kfox1111: ok I agree it does not look great | 16:53 |
| sbezverk | kfox1111: how about changing all.yaml and instead of using common/global/kolla use just global/kolla ... | 16:53 |
| kfox1111 | I believe your reasoning, was to make it easier to dicover the settings that coudl be overridden? | 16:53 |
| sbezverk | in this case vars in global/kolla in all.yaml will become visible to the service | 16:54 |
| kfox1111 | sbezverk: oh. just pull the onen common section back a level? | 16:54 |
| sbezverk | yes | 16:54 |
| *** crushil has quit IRC | 16:54 | |
| kfox1111 | It is visible to the service now I think. | 16:54 |
| sbezverk | get rid of common tag completely | 16:54 |
| kfox1111 | that won't change anything I think. | 16:54 |
| kfox1111 | it just puts them in the microservices packages. | 16:55 |
| kfox1111 | are you wanting to generate the service values.yaml's too? | 16:55 |
| sbezverk | kfox1111: but would it not all.yaml be searchable from service poi9nt of view? | 16:55 |
| *** unicell has joined #openstack-kolla | 16:55 | |
| *** unicell has quit IRC | 16:55 | |
| *** unicell has joined #openstack-kolla | 16:55 | |
| kfox1111 | currently no. | 16:56 |
| kfox1111 | service has a static values.yaml file. | 16:56 |
| kfox1111 | it then calls each microservice package. | 16:56 |
| kfox1111 | which has its own values.yaml file. | 16:56 |
| kfox1111 | that values.yaml file is generated from all.yaml | 16:56 |
| kfox1111 | the service values and the microservice values are merged at thtat point, I believe, | 16:56 |
| kfox1111 | so it ends up using global.kolla.all.* from all.yaml | 16:57 |
| kfox1111 | unless overridden in the services values.yaml or the users values.yaml. | 16:57 |
| sbezverk | kfox1111: in this case microsercice needs to have all common values, basically each microservice needs to have all keystone related parameters | 16:57 |
| kfox1111 | yeah. and each microservice gets its own values from all.yaml. | 16:58 |
| sbezverk | in this case services's values.yaml will not need them | 16:58 |
| kfox1111 | right. | 16:58 |
| sbezverk | kfox1111: I think the problem is in "its own" | 16:58 |
| kfox1111 | the user can still override them in their own values.yaml. | 16:58 |
| sbezverk | I think its own needs to be extended to include some | 16:58 |
| kfox1111 | I don't quite follow. | 16:58 |
| kfox1111 | some what? | 16:59 |
| *** inc0 has joined #openstack-kolla | 16:59 | |
| sbezverk | ok to be able to get keystone_user_name I have two options | 16:59 |
| sbezverk | 1. add it to service's values.yaml | 16:59 |
| sbezverk | 2. get it somehow from microservice | 17:00 |
| kfox1111 | sorry. there is some abbiguity there. lets try and clarify. | 17:00 |
| sbezverk | currently if I do not specify it in service's values.yaml I do NOT get it from the microservice | 17:00 |
| kfox1111 | what hat do you have on for that use case? a dev, or an operator deploying k8s? | 17:00 |
| sbezverk | cleanup service | 17:00 |
| kfox1111 | oh. | 17:01 |
| kfox1111 | I think I know the issue. | 17:01 |
| kfox1111 | common-create-keystone-admin | 17:01 |
| kfox1111 | the values you want are in common-create-keystone-admin section, not common. so they aren't automatically added to the microservice. | 17:01 |
| sbezverk | kfox1111: right | 17:01 |
| inc0 | o/ | 17:01 |
| kfox1111 | your microservices that need those types of common values need to be added to tools/helm_prebuild_microservices.py in the list common_create_keystone_admin | 17:02 |
| kfox1111 | then it should work. | 17:02 |
| kfox1111 | won't need anything in the servie package's values then. | 17:02 |
| kfox1111 | (and we really need a subpackage values include feature from helm. :/ ) | 17:03 |
| sbezverk | kfox1111: we have great API now I think what you suggested (I will do it) is more like a workaround | 17:03 |
| *** matrohon has quit IRC | 17:04 | |
| sbezverk | would be great to be able to use API and search without any software tools/tricks | 17:04 |
| kfox1111 | sbezverk: if we don't manage it in one place, all_values.yaml, then we will have to remember to make changes to lots of individual packages. it will be a pain to maintain, and cause bugs. :/ | 17:04 |
| kfox1111 | its a partial workaround until helm gets a feature natively to help with the issue. :/ | 17:04 |
| kfox1111 | +1 | 17:05 |
| kfox1111 | that needs helm to mature some more though. :/ | 17:05 |
| kfox1111 | its being worked on though. | 17:05 |
| sbezverk | kfox1111: the way you suggested, would it require adding items to all.yaml as well? | 17:05 |
| sbezverk | kfox1111: I would really like not to do it from safety perspective | 17:06 |
| kfox1111 | no. should just need to add it to the list, and remove the values from services.yaml related to keystone | 17:06 |
| *** eaguilar has joined #openstack-kolla | 17:06 | |
| sbezverk | kfox1111: ok | 17:06 |
| kfox1111 | and region. | 17:06 |
| *** zioproto has quit IRC | 17:07 | |
| kfox1111 | maybe we can generate an example/mycloud.yaml from all the microservices.yaml... | 17:07 |
| *** zioproto has joined #openstack-kolla | 17:08 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding cleanup service for cinder https://review.openstack.org/425875 | 17:10 |
| sbezverk | kfox1111: region needs to be removed too? | 17:11 |
| *** athomas has quit IRC | 17:11 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding cleanup service for cinder https://review.openstack.org/425875 | 17:12 |
| *** eaguilar has quit IRC | 17:12 | |
| sbezverk | kfox1111: this morning cleanup service helped me. I installed cinder on my cluster using service package and then lvm using microservice | 17:13 |
| mliima | Jeffrey4l_, can you review it https://review.openstack.org/#/c/422157/ | 17:13 |
| kfox1111 | sbezverk: yeah. it also comes in with that common-create-keystone-admin: section. | 17:14 |
| sbezverk | then I noticed that cinder service installs 2.0.2 images by default and volume was crashing insce it runs 3.0.2 | 17:14 |
| sbezverk | removing cinder with cleanup was nice and quick :) | 17:14 |
| kfox1111 | sbezverk: yeah. we need to finish up that multiversion patch I had started I think. then we can make a versions.yaml file for 2.x and 3.x with all the right settings. | 17:15 |
| kfox1111 | right know its a bit half and half. :/ | 17:15 |
| kfox1111 | sbezverk: cool. :) | 17:15 |
| kfox1111 | would be interesting to get the gate stuff to use the cleanup stuff. | 17:15 |
| sbezverk | kfox1111: I wish I had the same for keystone yesterday ;) | 17:15 |
| kfox1111 | at the end of the gate run, run throuh the cleanup scripts | 17:15 |
| kfox1111 | and see if everythign goes away. :) | 17:15 |
| kfox1111 | should be really quick so shouldn't add much time to the gate. | 17:16 |
| sbezverk | kfox1111: I wanted to add it to the workflow, install cinder remove cinder re-install cinder and see if basic tests work | 17:16 |
| kfox1111 | ah. | 17:17 |
| kfox1111 | was thinking about doing something similar with the gate upgrade job | 17:17 |
| kfox1111 | think that would work to kill two birds with one stone? or is that a different use case? | 17:17 |
| kfox1111 | ugg... still haven't made a common section for mariadb stuff. :/ | 17:20 |
| kfox1111 | man I really want helm to fix this. :/ | 17:20 |
| sbezverk | kfox1111: do not follow upgrade case, if you could clarify a bit | 17:20 |
| kfox1111 | we have 2.x and 3.x working in the gate now. | 17:21 |
| kfox1111 | so we should be able to start with the 2.x gate, and then helm upgrade everything to 3.x. | 17:21 |
| sbezverk | kfox1111: right but during upgade we do not delete anything I thought | 17:22 |
| kfox1111 | hmm... true. | 17:22 |
| kfox1111 | I guess they are seperate then. | 17:22 |
| kfox1111 | but, would it be essentially the same code if we had a second varient that, | 17:23 |
| kfox1111 | everywhere it had helm upgrdae, it did a helm delete; helm install instead? | 17:23 |
| inc0 | Jeffrey4l_, you up? | 17:23 |
| sbezverk | kfox1111: I am not sure yet how it will work, as db manage needs to be smart enough to understand it is upgrade scenario | 17:23 |
| *** Jeffrey4l_ has quit IRC | 17:24 | |
| *** unicell has quit IRC | 17:24 | |
| srwilkers | sbezverk, yep. thats the challenge | 17:24 |
| kfox1111 | I think all things but maybe neutron, its just a "foo-manage db_sync" | 17:24 |
| kfox1111 | so there is no difference between init and upgrade. | 17:24 |
| *** Jeffrey4l_ has joined #openstack-kolla | 17:24 | |
| kfox1111 | though I think that may bring up an issue with helm service upgrades.... | 17:25 |
| kfox1111 | it probably wont rerun the managedb job. :/ | 17:25 |
| sbezverk | kfox1111: Unless I miss something, I do not think upgrade is doable now with the code we have | 17:25 |
| kfox1111 | microservices upgrades, yes. | 17:25 |
| kfox1111 | service packages, maybe not. | 17:25 |
| *** sayantan_ has joined #openstack-kolla | 17:26 | |
| sbezverk | kfox1111: right with microservices you just skip step create db and maybe replace manage db step by upgrade db step | 17:27 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Periodic gate fix #2 https://review.openstack.org/426318 | 17:28 |
| kfox1111 | yeah. | 17:28 |
| sbezverk | kfox1111: I have never done upgrade myself, my upgrade practice was wipe out and start new from scratch ;) | 17:28 |
| kfox1111 | not many folks have successfully upgraded an openstack. :/ | 17:28 |
| *** egonzalez90 has joined #openstack-kolla | 17:29 | |
| kfox1111 | sbezverk: I wana make sure the delete stuff works individually as well as with the service package. | 17:29 |
| *** rmart04 has quit IRC | 17:29 | |
| kfox1111 | There's a bit of refactoring of all_values I think would help with that. | 17:29 |
| *** sayantani01 has quit IRC | 17:29 | |
| kfox1111 | let me whip up a ps real quick that does that. shouldn't take but a few minutes. | 17:29 |
| *** egonzalez90 has quit IRC | 17:30 | |
| openstackgerrit | Merged openstack/kolla-ansible: Optimize reconfigure action for cinder https://review.openstack.org/422384 | 17:30 |
| *** tonanhngo has joined #openstack-kolla | 17:30 | |
| openstackgerrit | Merged openstack/kolla-ansible: Fix RabbitMQ server error in bifrost https://review.openstack.org/424384 | 17:30 |
| sbezverk | kfox1111: it does, if you provide values.yaml | 17:30 |
| *** shardy has quit IRC | 17:31 | |
| kfox1111 | oh... I see why I didn't roll up the mariadb stuff... | 17:33 |
| kfox1111 | two of the fields are service specific. | 17:33 |
| kfox1111 | I could roll up 2 of the fields though... | 17:33 |
| kfox1111 | would make it easier to change the db's location/port globally... | 17:34 |
| *** tonanhngo has quit IRC | 17:35 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Allow the use of previously built parent images https://review.openstack.org/426322 | 17:36 |
| *** Penfold has quit IRC | 17:36 | |
| *** srwilkers is now known as srwilkers_lunch | 17:37 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Allow the use of previously built parent images https://review.openstack.org/426322 | 17:38 |
| inc0 | pbourke, do you have multinode around? | 17:38 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Make common config section for mariadb packages https://review.openstack.org/426324 | 17:38 |
| pbourke | inc0: yeah | 17:38 |
| inc0 | did you rebuild/redeploy lately? | 17:38 |
| kfox1111 | meeting. bbiab. | 17:38 |
| inc0 | with fresh images? | 17:39 |
| *** zioproto has quit IRC | 17:40 | |
| pbourke | inc0: hmm probably not for a day or two | 17:42 |
| pbourke | inc0: heading home now but if you have something you want me to check let us know and I'll get to it monday | 17:42 |
| inc0 | hmm I'm having strange issues | 17:42 |
| inc0 | well, galera fails for me | 17:42 |
| pbourke | networking by any chance? | 17:42 |
| pbourke | oh | 17:42 |
| *** adrian_otto has joined #openstack-kolla | 17:43 | |
| *** Penfold has joined #openstack-kolla | 17:49 | |
| *** mgoddard has quit IRC | 17:50 | |
| *** unicell has joined #openstack-kolla | 17:54 | |
| kfox1111 | back. | 17:54 |
| openstackgerrit | Merged openstack/kolla: Docs updated to get coverage report https://review.openstack.org/425032 | 17:54 |
| *** crushil has joined #openstack-kolla | 17:55 | |
| kfox1111 | sbezverk: couple of things. could you rebase the cleanup ps on https://review.openstack.org/#/c/426324 . also, could you file a blueprint for cleanup packages? would be nice to be able to track that. | 17:55 |
| sbezverk | kfox1111: I have already bp for cleanup services | 17:56 |
| sbezverk | it is attached to my PS | 17:56 |
| kfox1111 | sbezverk: ah. cool. | 17:56 |
| kfox1111 | just was blind then. | 17:56 |
| kfox1111 | I see it now. | 17:56 |
| sbezverk | kfox1111: since everytime when I try to rebase one ps on another I get into troubles, I will wait for you to merge and then remove mariadb related 2 lines | 17:58 |
| *** lrensing has quit IRC | 17:59 | |
| kfox1111 | ok. works for me. | 17:59 |
| *** harlowja has joined #openstack-kolla | 18:02 | |
| *** hfu has quit IRC | 18:02 | |
| sbezverk | kfox1111: I do not get databe_name and database_user from all.yaml | 18:04 |
| sbezverk | kfox1111: I explained why I think adding delete related tags in all.yaml is dangerous | 18:05 |
| kfox1111 | sbezverk: the package has to be added to the list to get it. | 18:05 |
| kfox1111 | I really want to fix this with subpackages. :/ | 18:06 |
| kfox1111 | asking the helm folks how hard that would be. | 18:06 |
| sbezverk | kfox1111: it will not be now, it will take wekks at best, are you suggesting to put no hold then? | 18:06 |
| *** sambetts is now known as sambetts|afk | 18:07 | |
| kfox1111 | no, I think we fix it to be consistent with the other ones we've got in the mean time. | 18:07 |
| kfox1111 | I really don't like this hack, but consistency is best. | 18:07 |
| *** tonanhngo has joined #openstack-kolla | 18:08 | |
| sbezverk | kfox1111: you see if we had one section of global/kolla/all which would not require tool to be avaialbe to microservice it would have been much much easier | 18:09 |
| sbezverk | kfox1111: if you check all.yaml each microservice has duplicated bunch of keystone related vars | 18:10 |
| sbezverk | they are identical | 18:10 |
| sbezverk | http://paste.openstack.org/show/596765/ | 18:11 |
| sdake_ | david-lyle thanks for the horizon review and the assistance! | 18:12 |
| kfox1111 | sbezverk: yeah. those should be merged up into a common config too. :/ | 18:12 |
| kfox1111 | that part is still kind of a mess. :/ | 18:12 |
| jascott1 | kfox1111 if you want to hit a pastebin I can form it up into proposal | 18:13 |
| jascott1 | i know you to be busy sir | 18:13 |
| kfox1111 | k. | 18:13 |
| jascott1 | brb | 18:14 |
| *** l4yerffeJ_ has quit IRC | 18:14 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 18:15 | |
| *** m1dev has joined #openstack-kolla | 18:19 | |
| *** JoseMello has joined #openstack-kolla | 18:21 | |
| *** Syffs has joined #openstack-kolla | 18:22 | |
| kfox1111 | sbezverk: https://review.openstack.org/#/c/426318/ | 18:25 |
| kfox1111 | wow. two ps's in a row all green first time. | 18:27 |
| kfox1111 | seems like the gate stabilization work is finally starting to kick in. :) | 18:27 |
| kfox1111 | any other kolla-kubernetes cores awake? | 18:27 |
| *** mgoddard has joined #openstack-kolla | 18:28 | |
| inc0 | ok guys, fire alarm, mariadb is confirmed broken | 18:30 |
| inc0 | deploying 3.0.2 works, deploying 4.0.0 breaks | 18:30 |
| inc0 | ubuntu source | 18:30 |
| sdake_ | kfox1111 i'm barely alive | 18:31 |
| kfox1111 | is it a clustering issue? | 18:31 |
| inc0 | seems like it | 18:31 |
| inc0 | first node stands | 18:31 |
| kfox1111 | sdake_: https://review.openstack.org/#/c/426324/ and https://review.openstack.org/#/c/426318/ | 18:31 |
| inc0 | rest of nodes dont | 18:31 |
| kfox1111 | sdake_: both shoudl be easy reviews. | 18:31 |
| kfox1111 | sdake_: the first is blocking sbezverk | 18:31 |
| inc0 | that's why it didn't show up in gates I bet | 18:31 |
| sdake_ | roger | 18:31 |
| sbezverk | kfox1111: I think we should keep keystone user vars in service values.yaml | 18:32 |
| kfox1111 | sdake_: thanks. | 18:32 |
| kfox1111 | sbezverk: I'm very much against putting any values in services values.yaml other then for deps. | 18:32 |
| *** vhosakot has joined #openstack-kolla | 18:32 | |
| kfox1111 | sbezverk: microservices need values too. | 18:32 |
| kfox1111 | sbezverk: so lets set them in one place, in microservices. | 18:32 |
| sbezverk | kfox1111: ok fine, I will then create tags for delete in all vlaues yaml | 18:33 |
| kfox1111 | otherwise, we risk introducing bugs, when we fix something in microservices values and forget about services, or visa versa. | 18:33 |
| kfox1111 | yes please. | 18:33 |
| kfox1111 | thanks. | 18:34 |
| *** gfidente has quit IRC | 18:37 | |
| openstackgerrit | sayantani proposed openstack/kolla: Fix pushing images to local registry https://review.openstack.org/426338 | 18:37 |
| sdake_ | bmace_ - if your around -could you report your finidngs on teh halcyon issue tracker? | 18:38 |
| sdake_ | bmace_ i guess i took your feedback last night and ran with it and said calico works when in fact it does not | 18:38 |
| *** david-lyle has quit IRC | 18:39 | |
| *** ipsecguy has joined #openstack-kolla | 18:40 | |
| bmace_ | sdake_: last night i just said i would try it. in the end the outcome was the same. none of the 3 network options seemed to make a difference. weave, canal / calico | 18:40 |
| sdake_ | bmace_ roger i guess i misunderstood | 18:41 |
| sdake_ | bmace_ kind of passed out at the keyboard ;) | 18:41 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Make common config section for mariadb packages https://review.openstack.org/426324 | 18:41 |
| openstackgerrit | sayantani proposed openstack/kolla: Fix pushing images to local registry https://review.openstack.org/426338 | 18:41 |
| portdirect_travl | o/ | 18:42 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Periodic gate fix #2 https://review.openstack.org/426318 | 18:42 |
| portdirect_travl | hey peeps | 18:42 |
| portdirect_travl | bmace_: not got context but may be able to help with CNI options in halcyon - whats up? | 18:42 |
| sdake_ | portdirect_travl https://github.com/att-comdev/halcyon-vagrant-kubernetes/issues/50 | 18:43 |
| sdake_ | portdirect_travl the env you setup on my box works like a champ | 18:43 |
| sdake_ | portdirect_travl using master does nto work like a champ | 18:43 |
| sdake_ | portdirect_travl i dumped a diff of the repo and submodule for bmace_ to replicate | 18:43 |
| sdake_ | portdirect_travl that you setup | 18:43 |
| bmace_ | portdirect_travl: https://github.com/att-comdev/halcyon-vagrant-kubernetes/issues/50 | 18:44 |
| bmace_ | damn, you beat me | 18:44 |
| sdake_ | bmace_ i already had it open | 18:44 |
| sdake_ | was correcting my incorrect commenatry | 18:44 |
| sdake_ | bmace_ if you still hve the links handy - otherwise i have to disconnect from irc to dump it quickly | 18:44 |
| sdake_ | bmace_ maybe portdirect_travl will see something in the diffs | 18:44 |
| bmace_ | portdirect_travl: sdake_ : https://paste.fedoraproject.org/537385/54949391/ | 18:45 |
| portdirect_travl | sdake_: that was my thought :/ - I'll check now | 18:45 |
| bmace_ | portdirect_travl: sdake_: https://paste.fedoraproject.org/537386/85494993/ | 18:45 |
| sdake_ | portdirect_travl current speculation is that the config is different, not that master is broken | 18:45 |
| jascott1 | kfox1111 can you edit that issue to prepend "proposal:" in the title ? i would but dont have perms to your issue | 18:45 |
| bmace_ | based on the changes in there i assume that env is indeed using ceph? did we verify that sdake? would be odd if ceph vs. iscsi made a diff with that kind of error but, once you eliminate everything else, what is left must be the problem? :) | 18:46 |
| sdake_ | bmace_ we haven' teven got to ceph yet with the instructions to launch busybox | 18:47 |
| sdake_ | bmace_ although kubernetes does require some type of baacking storagef system such as ceph | 18:47 |
| sdake_ | bmace_ did you change your config to iscsi? | 18:47 |
| sdake_ | bmace_ the first step we are after is launching busybox - and that works in one environment and not in another | 18:48 |
| sdake_ | sweet my fsa paid for my new glasses :) | 18:48 |
| *** mliima has quit IRC | 18:49 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding cleanup service for cinder https://review.openstack.org/425875 | 18:50 |
| bmace_ | i have my dev env stuff set up in the same was as the current upstream dev review, which is using iscsi / isci_workflow rather than ceph | 18:51 |
| bmace_ | the original version of the review was written to use ceph | 18:51 |
| bmace_ | and ceph_workflow | 18:51 |
| sbezverk | kfox1111: added changes.. but man what you suggested will explode the size of all_values.yaml it is huge now but this will make it even bigger :( | 18:52 |
| kfox1111 | sbezverk: yeah. its a temporary thing. :/ | 18:55 |
| kfox1111 | sbezverk: please weigh in on: https://github.com/kubernetes/helm/issues/1883 | 18:55 |
| kfox1111 | I'm hoping we can get that through pretty quickly. | 18:55 |
| portdirect_travl | bmace_/sdake_ any chance you could paste your ./halcyon-kubernetes/kube-deploy/group_vars/all.yml file? | 18:55 |
| kfox1111 | then we can fix this much more helm natively. | 18:55 |
| sdake_ | portdirect_travl you got it -need to reconnect | 18:57 |
| bmace_ | portdirect_travl: http://paste.openstack.org/show/596771/ | 18:57 |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: [WIP] Update Kolla-Ansible documents https://review.openstack.org/426352 | 18:57 |
| bmace_ | portdirect_travl: swapped that calico value between weave, calico and canal and all had the same behavior | 18:57 |
| *** sdake_ has quit IRC | 18:58 | |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: [WIP] Update Kolla-Ansible documents https://review.openstack.org/426352 | 18:58 |
| portdirect_travl | bmace_: just set up a fresh clone now - 10 mins and I'll hopefully be able to repoduce, if it works for me thats when i sad panda | 18:59 |
| *** sdake has joined #openstack-kolla | 18:59 | |
| sdake | portdirect_travl working: http://paste.fedoraproject.org/537826/43583148/ -> http://paste.fedoraproject.org/537826/43583148 | 19:00 |
| bmace_ | yeah, hopefully you break like us :) | 19:00 |
| sdake | portdirect_travl broken: http://paste.fedoraproject.org/537827/48554364/ -> http://paste.fedoraproject.org/537827/48554364 | 19:01 |
| *** srwilkers_lunch is now known as srwilkers | 19:01 | |
| portdirect_travl | sake just found typo in dev doc - could you fix? | 19:01 |
| sbezverk | kfox1111: I suggested an alternative solution ;) in the proposal | 19:01 |
| portdirect_travl | ./setup-halcyon.sh \ | 19:02 |
| portdirect_travl | --k8s-config kolla \ | 19:02 |
| portdirect_travl | --k8s-version v$KOLLA_KUBERNTES_VERSION \ | 19:02 |
| portdirect_travl | --guest-os centos | 19:02 |
| kfox1111 | k. looking | 19:02 |
| portdirect_travl | KOLLA_KUBERNTES_VERSION is spelt wrong - we all missed it :( | 19:02 |
| srwilkers | theres a patch set that needs one more +2 to merge to fix that portdirect_travl | 19:03 |
| srwilkers | https://review.openstack.org/#/c/425946/ | 19:03 |
| portdirect_travl | cheers srwilkers :) | 19:04 |
| kfox1111 | sbezverk: just skimmed through it. looking good to me. goign over one more time. | 19:05 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Fix typos in development environment documentation https://review.openstack.org/425946 | 19:05 |
| kfox1111 | sdake: portdirect_travl awesome. does that mean the dev env works? | 19:08 |
| kfox1111 | the docs for it I mean. | 19:08 |
| sdake | wow i'm a bunch of fial | 19:08 |
| sdake | fail | 19:08 |
| kfox1111 | I'd reallly like to see that var name changed. | 19:09 |
| kfox1111 | its not the version of kolla-kubernetes thats being specified, but the version of kubernetes. | 19:09 |
| sdake | kfox1111 which var name | 19:09 |
| sdake | right | 19:09 |
| sdake | i'll fix that in the change | 19:09 |
| kfox1111 | thanks. :) | 19:09 |
| sdake | thanks portdirect_travl for the diag | 19:09 |
| *** lrensing has joined #openstack-kolla | 19:09 | |
| sdake | we need to fix that at a minimum - still no idea if that fixes the whole schebang | 19:10 |
| sdake | submitting patch shortly | 19:10 |
| *** absubram has joined #openstack-kolla | 19:10 | |
| portdirect_travl | 2 mins - your and bmaces configs did not show the symptoms of it though | 19:10 |
| *** unicell has quit IRC | 19:12 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adding cleanup service for cinder https://review.openstack.org/425875 | 19:12 |
| kfox1111 | sbezverk: looks great. thanks for working on that, and bearing with me. I think that triggered a good discussion on the helm side. hopefully it will speed up the implementation. | 19:14 |
| *** absubram has quit IRC | 19:15 | |
| sbezverk | kfox1111: thanks :) I hope devs will find it useful when start playing with services package... | 19:18 |
| *** absubram has joined #openstack-kolla | 19:18 | |
| * srwilkers nods | 19:19 | |
| sdake | bmace_ when I started heat years ago I had a dream of gating the docs | 19:19 |
| sdake | bmace_ in terms of are they functional | 19:19 |
| sdake | bmace_ we tried, and the community never got it rolling - even 5 years later :( | 19:19 |
| sdake | oh well - thats what humans arefor :) | 19:19 |
| sdake | must pick up son from school | 19:20 |
| *** l4yerffeJ_ has quit IRC | 19:22 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 19:23 | |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Update Kolla-Ansible documents https://review.openstack.org/426352 | 19:24 |
| sayantan_ | Hi everyone | 19:25 |
| sbezverk | kfox1111: one more question. about explicit and implicit dependencies | 19:25 |
| sayantan_ | There is a huge duplication of Kolla-ansible related documents in both Kolla and Kolla-ansible repositories. Changes have been made to these docs in Kolla repo and corresponding changes have not been made in Kolla-ansible | 19:26 |
| sayantan_ | I have ported the changes made to these docs in Kolla to Kolla-ansible | 19:26 |
| sbezverk | kfox1111: srwilkers and myself were discussing this ps, appreciate your input here: https://review.openstack.org/425964 | 19:27 |
| jascott1 | hi sayantan_ | 19:27 |
| portdirect_travl | sdake/bmace_ sad panda :( https://paste.fedoraproject.org/537844/14855452/raw/ | 19:27 |
| sayantan_ | Hi jascott1 | 19:27 |
| sayantan_ | how is it going? | 19:27 |
| jascott1 | good good. we have sunshine today :) | 19:27 |
| sayantan_ | jascott1 I hope you get to enjoy it :) | 19:28 |
| sayantan_ | Don't spend too long indoors :P | 19:28 |
| sayantan_ | sdake, inc0 can you please take look when you have time https://review.openstack.org/#/c/426352/ | 19:29 |
| Serlex | dgonzalez can you take a look at https://review.openstack.org/#/c/425148/ | 19:29 |
| jascott1 | this is truer than people realize https://www.youtube.com/watch?v=BBUxZdWJ_zE | 19:29 |
| Serlex | or sdake | 19:29 |
| sayantan_ | jascott1 That is scary in a way for people back in Texas | 19:30 |
| jascott1 | yeah we lack some sun but also dont have TX hail, tornadoes and floods and even hurricanes so pros/cons :) | 19:32 |
| jascott1 | of course we will fall off into the sea someday but every silver lining has a touch of grey | 19:32 |
| portdirect_travl | bmace_: could you paste the output of (set -x; kubectl version; ansible --version; vagrant --version) | 19:34 |
| kfox1111 | sbezverk: looking | 19:35 |
| portdirect_travl | https://www.irccloud.com/pastebin/ENmcSF99/ | 19:35 |
| sbezverk | kfox1111: what I still fail to undersatnd how it worked at the gate with these dependecies :( | 19:37 |
| *** crushil has quit IRC | 19:37 | |
| bmace_ | portdirect_travl: will do once i have things deployed again. was trying to rebuild my kubeX env to try out running with ceph_workflow rather than iscsi_workflow as your initial review had it implemented | 19:39 |
| *** crushil has joined #openstack-kolla | 19:40 | |
| kfox1111 | the before change deps? | 19:40 |
| kfox1111 | lets see... | 19:41 |
| kfox1111 | the endpoint creation job, required a mariadb to be up and reposive, which should be fine... | 19:42 |
| kfox1111 | but keystone-admin/public/internal too... | 19:42 |
| *** unicell has joined #openstack-kolla | 19:42 | |
| kfox1111 | that does seem like a problem. | 19:42 |
| *** jheroux has quit IRC | 19:43 | |
| kfox1111 | well... | 19:44 |
| kfox1111 | oh. | 19:44 |
| kfox1111 | lets see.. | 19:44 |
| sbezverk | dependency on service should never work as the backend is provided by keystone api | 19:44 |
| kfox1111 | keystone-create-endpoints actually just registers keystone endpoints into keystone. | 19:44 |
| sbezverk | and in my cluster I saw it right away | 19:45 |
| kfox1111 | well... it "kolla_keystone_bootstrap's... | 19:45 |
| kfox1111 | yeah. this seems broken. | 19:45 |
| kfox1111 | but it is working in the gate? | 19:45 |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Update Kolla-Ansible documents https://review.openstack.org/426352 | 19:45 |
| sbezverk | kfox1111: look create_keystone_endpoint: depends on keystone-admin service | 19:45 |
| kfox1111 | yeah. that should cause the dep container to fail I think. | 19:46 |
| sbezverk | keystone-admin-svc object is there but it does not have endpoint associated | 19:46 |
| kfox1111 | the endpoints in keystone and the svc endpoints are completely seperate things. | 19:46 |
| kfox1111 | not sure they really depend on each other. | 19:46 |
| sbezverk | kfox1111: exactly, that is what I saw in my cluster.. I cannot see how it worked at the gate | 19:46 |
| kfox1111 | but Ido think api needs t ohappen after endpoints creation. | 19:46 |
| kfox1111 | the gate is passing though. so something is wrong elsewhere... | 19:47 |
| sbezverk | kfox1111: for sure api needs to wait for endpoint, but not for services | 19:47 |
| kfox1111 | true. | 19:47 |
| sbezverk | as it is an actual provider of these services | 19:47 |
| kfox1111 | that shoudl always fail. | 19:47 |
| kfox1111 | right. | 19:47 |
| sbezverk | kfox1111: anyway this change resolved the issue in my cluster | 19:48 |
| SamYaple | kfox1111: ive been using cephfs with good results recently | 19:48 |
| sbezverk | and it makses sense, I do not know what I was thinking when was specifying these dependencies, I guess I was drunk ;) | 19:49 |
| SamYaple | its not great for databases though. but having storage tier options is certainly doable | 19:49 |
| kfox1111 | sbezverk: I'd believe it. the only question in my mind is do we leave service:mariadb in the create_keystone_endpoints list. | 19:49 |
| kfox1111 | sbezverk: I can see maybe a case for it. | 19:49 |
| kfox1111 | SamYaple: yeah. we've been using it for about a year and a half now. awesome stuff. :) | 19:49 |
| sbezverk | kfox1111: I do not think it is needed as create_keystone_endpoints depends on create db and manage db | 19:50 |
| sbezverk | so it implicitely also depends on mariadb | 19:50 |
| kfox1111 | SamYaple: ever seen a cephfs backed by 2x replication and ssd's? maybe that would be suitable for db's? | 19:50 |
| SamYaple | kfox1111: cephfs will not be sutiable for highperformance databases. like ever. at least in my opinion | 19:51 |
| kfox1111 | sbezverk: could there be a time when the keystone-create-db/managedb heppens, and then there's a long delay till create_keystone_endpoitns? | 19:51 |
| SamYaple | ive been using it for about a year. but i just hooked it into kubernetes as a storageclass and it works well | 19:51 |
| kfox1111 | and in between mariadb drops out? | 19:51 |
| SamYaple | at least as well as krbd in my opinion | 19:51 |
| kfox1111 | probably won't matter too much, as the create endpoitns job will just keep rerunning. | 19:52 |
| sbezverk | kfox1111: possible but very very unlikely | 19:52 |
| SamYaple | kfox1111: my cephfs is bked by nvme, and it is very spikey in perforamcen | 19:52 |
| kfox1111 | SamYaple: I was contimplating implementing fernet tokens on top of cephfs. | 19:52 |
| kfox1111 | SamYaple: then can have keystone read the fernet dir read only, | 19:52 |
| kfox1111 | SamYaple: and do a scheduled job as the roller. | 19:52 |
| kfox1111 | SamYaple: oh really? I would have expected nvme to be awesome perfomrance, not so spiky. | 19:53 |
| *** david-lyle has joined #openstack-kolla | 19:53 | |
| kfox1111 | very good to know though. | 19:53 |
| SamYaple | cephfs is spikey though | 19:53 |
| kfox1111 | ah. | 19:53 |
| SamYaple | certain jobs run on schedules and can lock things up | 19:53 |
| *** david-lyle has quit IRC | 19:53 | |
| SamYaple | when multi-master MDS are a stable thing, it should be better | 19:53 |
| *** david-lyle has joined #openstack-kolla | 19:54 | |
| SamYaple | but htat likely wont happen until N release | 19:54 |
| *** unicell has quit IRC | 19:55 | |
| kfox1111 | SamYaple: ah. that makes sence. | 19:55 |
| kfox1111 | lustre fought for many years to get multi mds to work. its really a hard problem. | 19:56 |
| SamYaple | btw performance increased 50% with bluestore | 19:56 |
| kfox1111 | we still don't run it anywhere. | 19:56 |
| kfox1111 | sweet. | 19:56 |
| SamYaple | im running bluestore in my test cluster | 19:56 |
| SamYaple | its still experimental in Kraken | 19:56 |
| kfox1111 | when is bluestore going to be considered stable? | 19:56 |
| SamYaple | but should be stable in L | 19:56 |
| kfox1111 | when's tthe next lts? | 19:56 |
| SamYaple | L | 19:56 |
| kfox1111 | cool. | 19:56 |
| SamYaple | in 4 months | 19:56 |
| SamYaple | so give it 6-7 months :P | 19:57 |
| SamYaple | they committed to providing an upgrade path from the K release moving forward for bluestore | 19:57 |
| SamYaple | so i would call it Beta right now | 19:57 |
| SamYaple | anyway, its signficiantly faster for all HDD | 19:58 |
| sbezverk | kfox1111: so what do we do with mariadb dependency? | 19:59 |
| *** mgoddard has quit IRC | 19:59 | |
| kfox1111 | writing a response. | 20:00 |
| kfox1111 | random thought.... | 20:00 |
| kfox1111 | tangent. | 20:00 |
| *** david-lyle has quit IRC | 20:01 | |
| kfox1111 | what happens to a kubernetes-entrypoint dep check on a service, when its a pure pointer svc? | 20:01 |
| kfox1111 | IE, the service is pointing to an external mariadb? | 20:01 |
| kfox1111 | it can't check the pod state in that case. | 20:01 |
| *** david-lyle has joined #openstack-kolla | 20:01 | |
| kfox1111 | does it work, or does it break? | 20:01 |
| openstackgerrit | sayantani proposed openstack/kolla-ansible: Fix MySQL DB error in bifrost container https://review.openstack.org/424391 | 20:01 |
| sbezverk | kfox1111: it fails | 20:02 |
| kfox1111 | we might want to put in a issue to kubernetes-entrypoint for that case. | 20:02 |
| sbezverk | it really check if there is end point associated | 20:02 |
| sbezverk | kfox1111: I think it is right behavior | 20:03 |
| kfox1111 | oh. if it only checks for an endpoint ip, then maybe it would work then. | 20:03 |
| sbezverk | if there is no backend then this service is useless | 20:03 |
| kfox1111 | so, it doesn't check for pod being ready, it checks for an enttry in the svc endpoint list? | 20:03 |
| sbezverk | kfox1111: endpoint ip appears only when backend provided container starts and in running state | 20:03 |
| kfox1111 | sbezverk: there is a way to startup a svc with an external unmanaged ip as an endpoint I believe a | 20:04 |
| sbezverk | kfox1111: based on my observation it check at minimum that service has one backend associated | 20:04 |
| kfox1111 | so it basically just becomes a dns entry. | 20:04 |
| sbezverk | kfox1111L right headless thing, but I am not sure if entrypoint is mart enough for headless | 20:05 |
| sbezverk | kfox1111: what could help is to collect logs even on a sucessful run at the gate | 20:06 |
| sbezverk | then we could see why dependency were sutisfied | 20:06 |
| sbezverk | for keystone when it should not | 20:06 |
| portdirect_travl | kfox1111: https://github.com/portdirect/harbor/blob/latest/kubernetes/ingress/freeipa.yaml#L1-L35 | 20:07 |
| kfox1111 | sbezverk: yeah. I think some folks will want external db's / rabbits. so we should verify that works, or file a bug with entrypoint. | 20:07 |
| kfox1111 | sbezverk: I contemplated that. but it ends up producing a lot of data. mabye a switch in the gate to make it easy to submit a patch that just sets the flag to true for testing? | 20:08 |
| *** aric49 has joined #openstack-kolla | 20:08 | |
| kfox1111 | portdirect_travl: not sure what your showing me? | 20:09 |
| portdirect_travl | how to set up a external service in k8s, thought it may be relivant | 20:09 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Adding cleanup service for cinder https://review.openstack.org/425875 | 20:09 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Fix documentation error in development environment https://review.openstack.org/426380 | 20:10 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes: Change Kubernetes version in docs https://review.openstack.org/426381 | 20:10 |
| *** lrensing has quit IRC | 20:10 | |
| sbezverk | kfox1111: yeah switch would work as this is not required always | 20:10 |
| kfox1111 | portdirect_travl: not an external service, but a service that is external? :) | 20:10 |
| portdirect_travl | k8s entrypoint checks that a service has at least one endpoint attached, so in therory should be fine with headless https://github.com/stackanetes/kubernetes-entrypoint/blob/master/dependencies/service/service.go#L28 | 20:11 |
| kfox1111 | portdirect_travl: oh. yeah, actually, I think your right. that is what I was asking for. miss read it. | 20:11 |
| portdirect_travl | np | 20:11 |
| kfox1111 | thanks. :) | 20:12 |
| *** MasterOfBugs has joined #openstack-kolla | 20:12 | |
| sbezverk | ortdirect_travl: how do you see from that code that it is ok with headless, for my education ;)?? | 20:13 |
| kfox1111 | yeah. the documentation made it sound like it actually looked for pods. | 20:13 |
| kfox1111 | but logically it would be much more work to do that, then just look for endpoins on the service. | 20:13 |
| *** jtriley has quit IRC | 20:14 | |
| kfox1111 | and most users wouldn't know the difference. | 20:14 |
| sdake | bmace_ those patches shoudl fix the issue - can you test | 20:14 |
| sdake | bmace_ you can run tox -e doc to generate the docs locally for cut and past eaction | 20:14 |
| *** pramodrj07 has joined #openstack-kolla | 20:15 | |
| sbezverk | kfox1111: so back to mariadb dependency ;) ?? | 20:15 |
| portdirect_travl | sbezverk: from the way im reading it - its just checking to see if an endpoint is attached to a service , the type of service is irrelivant in that case | 20:15 |
| sbezverk | portdirect_travl: I have never seen output of kubectl describe svc for headless case | 20:16 |
| kfox1111 | sbezverk: I reviewed. | 20:16 |
| sbezverk | so I do not know what it would return.. | 20:16 |
| kfox1111 | portdirect_travl: cool. thanks for verifying. | 20:16 |
| kfox1111 | sbezverk: its not quite headless. headless I think means no iptables rules for lb. | 20:17 |
| *** srwilkers has quit IRC | 20:17 | |
| kfox1111 | sbezverk: this is kind of an, external endpoint case. | 20:17 |
| inc0 | SamYaple, around? | 20:17 |
| kfox1111 | which is interesting. | 20:17 |
| kfox1111 | so you could still potentially use the lb with more then one external endpoint. | 20:17 |
| kfox1111 | which is pretty cool. :) | 20:17 |
| sbezverk | kfox1111: I see, I will need to read more about them | 20:18 |
| portdirect_travl | kfox1111: yeah i've driven services endpoints with some really ugly python before to drive an ingress controller | 20:18 |
| *** MasterOfBugs has quit IRC | 20:18 | |
| kfox1111 | sbezverk: yeah. I haven't played with them too much either. but seeing what this can do, I think I'll be switching some tthings from neturon lbaas v1 to k8s asap. :) | 20:19 |
| portdirect_travl | do it :) you'll be super happy kfox1111 | 20:19 |
| kfox1111 | portdirect_travl: ah. interesting. | 20:19 |
| bmace_ | sdake: so you guys think it is an issue with the kubernetes version being wrong? | 20:19 |
| sdake | bmace_ noope, the variable is spelled wrong | 20:19 |
| kfox1111 | portdirect_travl: I'm just a little leary of connection lenght. | 20:19 |
| sdake | bmace_ in the docs | 20:20 |
| bmace_ | sdake: sure, the variable in the docs? what if i didn't cut and paste from the docs? | 20:20 |
| kfox1111 | portdirect_travl: I have some tcp streams that last days/weeks. neutron lbaasv1 didnt' work with them until I patched them. | 20:20 |
| sdake | bmace_ then tell us that and let us know :) | 20:20 |
| sdake | bmace_ if your variable is spelled correctly it may be adifferent problem | 20:20 |
| sbezverk | kfox1111 portdirect_travl would be very interested to see any prototyping with ingress for kolla kube | 20:20 |
| sdake | bmace_ clearly the variable being spelled wrong is a problem :) | 20:20 |
| sdake | if you c&p fro mthe docs | 20:21 |
| portdirect_travl | kfox1111: ah - yeah I've only really used them in anger for websites | 20:21 |
| kfox1111 | sbezverk: +1 | 20:21 |
| sdake | which is how the docs are structured | 20:21 |
| portdirect_travl | sbezverk: all my old stuff uses ingress exclusively | 20:21 |
| bmace_ | sdake: what do you expect that variable being wrong in the docs to cause as an actual problem? some version of something being wrong? | 20:21 |
| kfox1111 | portdirect_travl: yeah. neutron lbaasv1 (and v2 really) assumes http connections. | 20:21 |
| sdake | bmace_ portdirect_travl pointed it out | 20:21 |
| sbezverk | portdirect_travl: could you please review this one so we could cloe this topic: https://review.openstack.org/#/c/425964/2 | 20:21 |
| kfox1111 | which are short lived tcp connections. so they really struggle with longer running data transfers. :/ | 20:21 |
| sdake | bmace_ i exepct the whole thing would crater | 20:21 |
| *** portdirect_travl is now known as portdirect | 20:21 | |
| sdake | bmace_ I don't know the code in detail enough to say - portdirect_travl does and srdoes as well | 20:21 |
| kfox1111 | I asked neutron 2+ years ago for an lbaas option to configure stream times. | 20:22 |
| sdake | well srwilkers that is :) | 20:22 |
| kfox1111 | still isn't a thing. :/ | 20:22 |
| sdake | kfox1111 your irc client is broke nagain | 20:22 |
| portdirect | sdake: just looksed at that ps 1.5.2 will break k8s | 20:22 |
| kfox1111 | sdake: your renaming too often again. ;) | 20:22 |
| *** goldyfruit has quit IRC | 20:22 | |
| portdirect | sdake: as there is no controller manager image published by att | 20:22 |
| portdirect | if we go there we should update to use kolla controller manager | 20:23 |
| sdake | portdirect i got impression from v1k0d3n but can't confirm that the at&t effort on openstack helm is moving off halcyon and on to minikube | 20:23 |
| Serlex | w00t | 20:25 |
| portdirect | thats correct at the moment for local dev - though will be using it for storage work | 20:25 |
| bmace_ | portdirect: hey, so, the doc bug. with that error what do you expect the actual effect would be? an improper kubernetes version right? what is the optimal version to be using? | 20:25 |
| sdake | portdirect if the 15.2 change is busted - -2 it so I can figure out how to solve it :) | 20:26 |
| sdake | rather -1 i mean | 20:26 |
| sdake | portdirect i think we have the controller image in kolla already right? | 20:26 |
| portdirect | sdake: you gave me permission for my 1st -2, you brought this on yoursel :) | 20:26 |
| *** jmccarthy has joined #openstack-kolla | 20:26 | |
| sdake | portdirect it wa a typo | 20:26 |
| sdake | don't make me switch to slack ;-) | 20:26 |
| kfox1111 | very interesting... lustre just got uid/gid mapping... | 20:27 |
| portdirect | sdake: lol - i chickened out anyway :) | 20:28 |
| portdirect | bmace_: the controller manager would be down, which may cause the errors you were seeing, but im not sure tbh | 20:29 |
| portdirect | sdake: yeah the correct image is in kolla - though we'd need to work out how to get it in | 20:30 |
| bmace_ | portdirect: the k8s manager? would all 4 nodes still show as ready? | 20:30 |
| sdake | portdirect get it in where | 20:30 |
| portdirect | bmace_: i dont know :( I think they might - but hinestly not sure | 20:32 |
| portdirect | sdake: inject it into halcyon | 20:32 |
| bmace_ | portdirect: would this cause a failure to deploy anything in the k8s cluster? i can try to deploy something else into it that is simpler than the full kolla-k8s | 20:33 |
| portdirect | bmace_: best way to test a cluseter is this : (paste incoming) | 20:33 |
| sdake | bmace_ the k8s manager in my env now is 1.5.1-beta | 20:34 |
| bmace_ | my server is at 1.5.1 | 20:34 |
| sdake | bmace_ the k8s manager as specified in the firt rev of those docs would be 1.5.1 | 20:34 |
| portdirect | https://www.irccloud.com/pastebin/kBffn4o1/ | 20:34 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: PS fixes issue discovered with keystone service package https://review.openstack.org/425964 | 20:34 |
| *** jtriley has joined #openstack-kolla | 20:34 | |
| sdake | bmace_ the k8s manaager for 1.5.2 doesnt work at all | 20:34 |
| sdake | so for the moment we need to stick to 1.5.1 | 20:34 |
| bmace_ | well, that isn't my problem then. 1.5.1 achieved. | 20:35 |
| sdake | portdirect what does this k8s manager do btw | 20:35 |
| portdirect | sdake: you should have a 1.5.1 image? can you get git commits for vagrant and submodule | 20:35 |
| kfox1111 | why doesn't 1.5.2 work? | 20:35 |
| sdake | bmace_ did you not cut and paste from the document? | 20:35 |
| sbezverk | kfox1111: it is missing rbd binaries | 20:35 |
| sbezverk | so the dynamic pvc do not work | 20:35 |
| portdirect | kfox1111/sbezverk just dont have one upstream in att quay repo | 20:35 |
| kbaegis | inc0: Hey, did you guys manage to figure out the MariaDB issue? | 20:36 |
| bmace_ | sdake: nope.. i looked at the code and typed it in by hand, because i knew we were supposed to be at 1.5.1 | 20:36 |
| inc0 | kbaegis, negative, but I narrowed it down to images | 20:36 |
| kfox1111 | sbezverk: we dont use them? | 20:36 |
| inc0 | which makes me more worried | 20:36 |
| sbezverk | kfox1111: nope atm | 20:36 |
| kbaegis | inc0: Well, at least it was caught :) | 20:36 |
| sbezverk | we use explicit pv/pvc | 20:36 |
| kfox1111 | oh. so, its about the dynamic pvc ceph support? | 20:36 |
| sbezverk | kfox1111: right | 20:37 |
| kfox1111 | we don't have any code in kolla-kubernetes for that yet. :/ | 20:37 |
| inc0 | issue is, I have no idea what causes it yet | 20:37 |
| kfox1111 | so 1.5.2 should work? | 20:37 |
| sdake | bmace_ ok - i have to pick up my daughter from school - then have torture session the ndinner time | 20:37 |
| inc0 | I just deployed stable and it works | 20:37 |
| sdake | bmace_ not sure when i will be back on - probably around 7 | 20:37 |
| sbezverk | kfox1111: it is working :) do not forget the gate | 20:37 |
| bmace_ | so i don't think the kubernetes version is the issue.. at least it isn't my issue, if it is expected the 1.5.1 vs. 1.5.2 version was the issue | 20:37 |
| sdake | bmace_ there are two patche there, one is wrong | 20:37 |
| sdake | let me workflow -1 it :) | 20:38 |
| kfox1111 | sbezverk: cool. I +2'd it. figured the person wf'ing should verify the gate. | 20:38 |
| portdirect | both 1.5.1 and 1.5.2 should be fine kfox1111, its just the lack of 1.5.2 images in att's quay repo that causes issues | 20:38 |
| kfox1111 | but I believe it is good as the previous gate ran, and nothing substantial changed. | 20:38 |
| sbezverk | kfox1111: it is just in in portdirect's script I think he uses them.. | 20:38 |
| *** crushil has quit IRC | 20:38 | |
| kfox1111 | portdirect: so does it force k8s to alwayus use att's repos then? | 20:38 |
| kfox1111 | doesn't find 1.5.2 and fail? | 20:38 |
| *** crushil has joined #openstack-kolla | 20:39 | |
| portdirect | kfox1111: we are talking about halycon - not kolla-k8s | 20:39 |
| *** jmccarthy has quit IRC | 20:39 | |
| sbezverk | portdirect: what I fail to undersatnd is if sdake env uses iscsi workflow, how dynamic ceph pvc come to the picture. | 20:40 |
| kbaegis | inc0: Least useful approach, but still valid: b-search commits back from last known working version | 20:40 |
| portdirect | sbezverk: its just this: https://github.com/att-comdev/halcyon-kubernetes/blob/master/kube-deploy/roles/kube-prep/tasks/patch-kube.yml#L18 | 20:40 |
| sbezverk | there should not be any ceph and as a result there should not be ANY need for att repo | 20:40 |
| kfox1111 | portdirect: yeah. just trying to figure out in what ways does 1.5.2 not work in that env, and why. | 20:40 |
| kbaegis | inc0: Is there testing for something like that? Do you guys deploy? | 20:41 |
| kfox1111 | portdirect: ah. that explains it. thanks. :) | 20:41 |
| inc0 | kbaegis, we do have gates, but this seems to be clustering issue | 20:41 |
| inc0 | which might not be caught by gates | 20:41 |
| inc0 | and yeah, we do deploy and test;) myself included | 20:41 |
| sbezverk | portdirect: my question still stands, why you need att image with iscsi? | 20:42 |
| kbaegis | inc0: Hard to catch then? | 20:42 |
| sbezverk | why not just to use normal 1.5.2 kube controller? | 20:42 |
| kbaegis | :0 | 20:42 |
| portdirect | sbezverk: good point 2 sec | 20:43 |
| *** tonanhngo has quit IRC | 20:43 | |
| *** tonanhngo has joined #openstack-kolla | 20:43 | |
| inc0 | no idea really, for me it's pretty consistent fail | 20:43 |
| kfox1111 | sbezverk: because the code's hardcoded to look there. | 20:43 |
| inc0 | but for others not really | 20:43 |
| *** jemcevoy has joined #openstack-kolla | 20:43 | |
| kfox1111 | theres' not an override to pull from non attcomdev repo. | 20:44 |
| sbezverk | kfox1111: man if hardcoding causes issue, it is just needs to be UN hardcoded | 20:44 |
| jemcevoy | inc0, are you there? | 20:44 |
| portdirect | there is an override: sed -i "s/^patch_kube_ceph: true/patch_kube_ceph: false/" halcyon-kubernetes/kube-deploy/group_vars/all.yml | 20:44 |
| sbezverk | there are so much activity because of that?!?!? | 20:44 |
| inc0 | jemcevoy, here | 20:44 |
| kfox1111 | portdirect: ah. ok. | 20:44 |
| sbezverk | portdirect: can you do a condition if backend iscsi do not use that image? | 20:45 |
| sbezverk | the problem solved.. | 20:45 |
| portdirect | though the main problem is that sdake must be working from a vold repo to have a 1.5. beta image being loaded | 20:45 |
| kfox1111 | so there are 2 workarounds. | 20:45 |
| kfox1111 | 1, pin to 1.5.1 if you want dynamic pvc. | 20:45 |
| kfox1111 | 2, switch off patch_kube_ceph. | 20:45 |
| sbezverk | kfox1111: dynamic pvc are not supported with iscsi | 20:46 |
| jemcevoy | There are a couple of meetups on docker and OpenStack... https://www.meetup.com/Docker-Portland-OR/ https://www.meetup.com/openstack-pdx/ | 20:46 |
| sdake | sbezverk define not supported | 20:46 |
| kfox1111 | sbezverk: I get thought that some folks may want to start developing dynamic pvc support in the dev env. | 20:46 |
| inc0 | cool, thanks! | 20:46 |
| sbezverk | sdake: n.o.t s.u.p.p.o.r.t.e.d :) | 20:46 |
| kfox1111 | so both options are valid. | 20:46 |
| sdake | sbezverk suppported by who | 20:46 |
| sbezverk | kuberentes | 20:47 |
| sdake | kubernetes, halycon, where is it not supported? | 20:47 |
| sdake | whta about the other storage drivers like nfs and gce/etc? | 20:47 |
| kfox1111 | it only affects ceph rbd dynamic pvc's. | 20:47 |
| kfox1111 | requires ceph in the controller manager container. | 20:47 |
| kfox1111 | we're pretty far off track. | 20:48 |
| jemcevoy | inc0, For the docker meetup at puppet the MAX (light rail) stops right at the front door. | 20:48 |
| sbezverk | kfox1111: I chatted yesterday with one of kube storage developer and he told me for dynamic pvc on iscsi, I would need to use another custom controller | 20:49 |
| sbezverk | so natively like ceph dynamic pvc, kube does not support iscsi dynamic pvc | 20:49 |
| portdirect | sbezverk: if that came from rootfs, then its pretty much gospel | 20:49 |
| sbezverk | from him | 20:49 |
| kfox1111 | sbezverk: yeah. the problem is all iscsi devices have their own api's. | 20:49 |
| kfox1111 | sbezverk: thats what cinder was created for, in the first place. | 20:50 |
| kfox1111 | to manage the different api's and make them available as one unified api. | 20:50 |
| *** goldyfruit has joined #openstack-kolla | 20:50 | |
| inc0 | I think next openstack meetup is in same day as kubernetes roadshow thingy | 20:50 |
| sbezverk | kfox1111: I am sorry but using cinder to host backend for kubernetes cluster to run opensatck with another cinder, no thank you :-) | 20:50 |
| kfox1111 | sbezverk: yeah. not recommending that. just saying, its a huge amount of work to implement. cinder is basically the end of that particular rabbit trail. :/ | 20:51 |
| portdirect | and the opening of another rabbit hole | 20:52 |
| portdirect | :) | 20:52 |
| sbezverk | kfox1111: I think we should not jump on dynamic pvc until they get wider support.. | 20:52 |
| kfox1111 | if dynamic iscsi is to be a thing, it will have to reimplemnt basically cinder from scratch. | 20:52 |
| kfox1111 | I think dynaic pvc for ceph woudl be awesome. | 20:52 |
| sbezverk | kfox1111: not really | 20:52 |
| kfox1111 | I'm getting more and more worried about ceph rbd kernel support though. | 20:52 |
| kfox1111 | portdirect: you have some connections... | 20:53 |
| kfox1111 | portdirect: when I've been building up the gate, I've run into one issue with ceph kernel rbd, | 20:53 |
| kfox1111 | portdirect: and in production another. | 20:53 |
| sbezverk | requesting iscsi storage to created volume and then to add it as a target should not be more complex than with ceph | 20:53 |
| portdirect | right - i was talking about this today | 20:53 |
| kfox1111 | portdirect: generally kernel rbd seems... less then production ready at times. what have you seen? | 20:53 |
| *** fguillot has quit IRC | 20:53 | |
| kfox1111 | sbezverk: yeah it is, because each vendor has a differnt api. ceph has one. | 20:54 |
| portdirect | will bring it up with some peeps tomorrow but the sound of it was - that if moving to plain rbd was gonna make people happy then wfthem | 20:54 |
| kfox1111 | portdirect: yeah. its worked pretty well here, but hit some bumps that make me worried to run it in production. | 20:55 |
| kfox1111 | the reboot gate locks up after boot sometimes. wont ever recover. | 20:55 |
| kfox1111 | and saw a process on a prod system dstate due to rbd mount not working. writes to one particular file lock up. never recovered without rebooting the node. | 20:56 |
| portdirect | I think we may be the heaviest users of it in k8s | 20:56 |
| kfox1111 | yeah. | 20:56 |
| kfox1111 | SamYaple seemed to think it was unstable too. | 20:56 |
| portdirect | SamYaple thinks too much | 20:56 |
| portdirect | :) | 20:57 |
| kfox1111 | I think the saem could be said of us all. ;) | 20:57 |
| kfox1111 | and probably has been. :) | 20:57 |
| * portdirect waits for sam to take the bait | 20:57 | |
| portdirect | :) | 20:57 |
| * kfox1111 chuckles. | 20:57 | |
| kfox1111 | yeah, see if you have seen anyone else with instability problems with it. I'm very curious. | 20:58 |
| kfox1111 | be back in a bit. | 20:58 |
| *** haplo37_ has quit IRC | 21:05 | |
| *** vhosakot has quit IRC | 21:08 | |
| *** jmccarthy has joined #openstack-kolla | 21:08 | |
| *** satyar has quit IRC | 21:08 | |
| *** vhosakot has joined #openstack-kolla | 21:08 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: PS fixes issue discovered with keystone service package https://review.openstack.org/425964 | 21:12 |
| *** Serlex has quit IRC | 21:15 | |
| *** haplo37_ has joined #openstack-kolla | 21:16 | |
| *** vhosakot has quit IRC | 21:18 | |
| kfox1111 | back. | 21:23 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: PS Adds keystone cleanup service https://review.openstack.org/426400 | 21:28 |
| *** zioproto has joined #openstack-kolla | 21:30 | |
| *** zioproto has quit IRC | 21:31 | |
| *** absubram has quit IRC | 21:33 | |
| *** l4yerffeJ has joined #openstack-kolla | 21:34 | |
| kfox1111 | sbezverk: https://github.com/kubernetes/helm/issues/1883 | 21:34 |
| *** l4yerffeJ_ has quit IRC | 21:35 | |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| sbezverk | kfox1111: I was reading just that :) | 21:37 |
| kfox1111 | I think if we did it that way, we could move all the values out of all_values.yaml that are specific to packages to their own native values.yaml | 21:40 |
| kfox1111 | and all the common stuff could be broken up per shared chunk into parent_vars, | 21:40 |
| kfox1111 | and then included in the packages as needed in the dep to kolla-common. | 21:40 |
| *** vhosakot has joined #openstack-kolla | 21:40 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 21:41 | |
| kfox1111 | would still be easy to edit all the common config values in one place, kolla-common/parent_values/* :) | 21:41 |
| kfox1111 | and wouldn't be one big yaml file, so easier to edit. | 21:41 |
| sbezverk | kfox1111: sounds reasonable | 21:42 |
| kfox1111 | between that, and local repo's, we could probably get rid of pretty much all the custom stuff, | 21:43 |
| kfox1111 | but a build_all helper script. (just for ease of building it all) | 21:44 |
| sbezverk | kfox1111: kolla-common/parent_values would have a single file with common values right? | 21:45 |
| kfox1111 | the initial proposal would be that. kolla-common/parent_values.yaml. | 21:45 |
| kfox1111 | the latest idea I just posted would be, | 21:45 |
| kfox1111 | kolla-common/parent_values/<name>.yaml | 21:45 |
| *** unicell has joined #openstack-kolla | 21:46 | |
| kfox1111 | where the parent could select <names>'s out of the child chart to merge. | 21:46 |
| kfox1111 | so could do common, or mariadb, or python-haproxy, etc. | 21:46 |
| *** unicell has quit IRC | 21:46 | |
| kfox1111 | and/or. | 21:46 |
| kfox1111 | so we could replicate the functiionalyt of prebuild now, with helm native. | 21:47 |
| bmace_ | sdake: fyi, got busybox deploying on the halycon-kubernetes environment just fine | 21:47 |
| kfox1111 | does the busybox networkign work? | 21:47 |
| bmace_ | sdake: hitting some volume / mount issues now in my helm / mariadb / rabbitmq containers now. | 21:48 |
| bmace_ | kfox1111: nslookup kubernetes seems happy enough | 21:48 |
| kfox1111 | what about nslookup google.com | 21:48 |
| kfox1111 | (had a problem where kubernetes dns worked but not google for a while.) | 21:48 |
| bmace_ | hrm, perhaps not to much | 21:49 |
| bmace_ | resolve failed | 21:49 |
| mgiles | that's not going to work behind a proxy... | 21:49 |
| kfox1111 | yeah. kubernetes dns is resolved intoernally to k8s and the vm. | 21:49 |
| kfox1111 | google is resolved outside. | 21:49 |
| *** unicell has joined #openstack-kolla | 21:50 | |
| bmace_ | well, it says the dns server it is using is the kubernetes dns server. does the k8s dns service need to be configured differently due to the proxy that isn't handled already via the proxy config stuff in config.rb for halycon-vagrant-kubernetes? | 21:51 |
| mgiles | In my setup I can nslookup other hosts that are inside the corp firewall, but it makes sense I can't nslookup google.com. Our internal dns servers will not resolve that. | 21:52 |
| bmace_ | no curl in base busybox? how dare they! | 21:53 |
| bmace_ | yeah, at least i can tell networking in the busybox is generally working.. i can ping our proxy server | 21:54 |
| kfox1111 | mgiles: ah. ok. that makes sense. | 21:54 |
| *** adrian_otto has quit IRC | 21:54 | |
| kfox1111 | hehe. busybox is tiny. :) | 21:54 |
| *** JoseMello has quit IRC | 21:55 | |
| mgiles | What I get annoyed at is the images that don't have nslookup (or dig, or host, etc) | 21:55 |
| bmace_ | but, curl is tiny too! ;) | 21:55 |
| kfox1111 | yeah. but tiny + tiny = 2xtiny. :) | 21:55 |
| *** absubram has joined #openstack-kolla | 21:55 | |
| bmace_ | true | 21:56 |
| kfox1111 | but, nslookup, and ifconfig, and tcpdump, and... :) | 21:56 |
| kfox1111 | down hill from there. :) | 21:56 |
| kfox1111 | thats where something like alpine comes in. | 21:56 |
| bmace_ | but.. I need X!! ;) | 21:56 |
| inc0 | ubuntu trunk is broken :( | 21:56 |
| kfox1111 | bmace_: then build a container with x in it. ;) | 21:57 |
| inc0 | ahh the sadness, ahh the pain | 21:57 |
| jascott1 | mgiles can you pull the trick where you mount the binary from host into container? i havent tried in k8s but useful quick hack w docker | 21:57 |
| kfox1111 | jascott1: aahh.. | 21:57 |
| kfox1111 | jascott1: there be draggons. :) | 21:57 |
| mgiles | haven't tried that | 21:57 |
| kfox1111 | but can work, sometimes, perhaps. | 21:57 |
| kfox1111 | ldd it on the host and make sure all the libs are in the container. | 21:57 |
| kfox1111 | if so, you have a chance. | 21:57 |
| kfox1111 | oh. and the container and the host have the same os. | 21:58 |
| inc0 | I usually do docker exec -it -u root | 21:58 |
| inc0 | bash | 21:58 |
| jascott1 | for me its always vim and I just mount tiny-vim and it works but the others dont seem to | 21:58 |
| inc0 | and then apt-get stuff as I need them | 21:58 |
| mgiles | Again, it's harder behind a corp firewall since then you have to configure apt to work through the firewall. Not that hard, but makes a simple task more difficult | 21:59 |
| kfox1111 | yeah. | 22:00 |
| mgiles | anyway, the point is that bmace_ shouldn't complain too much. At least busybox has nslookup :) | 22:00 |
| mgiles | next he'll be saying he can't handle vi and needs emacs. Let the holy wars start | 22:01 |
| *** breitz has quit IRC | 22:01 | |
| *** absubram has quit IRC | 22:01 | |
| *** zioproto has joined #openstack-kolla | 22:02 | |
| *** breitz has joined #openstack-kolla | 22:02 | |
| kfox1111 | hmm... | 22:03 |
| bmace_ | well, lets not push it.. plus, i'm a vi guy.. so i'm all good.. it is more like dropping vi and expecting folks to use ed | 22:03 |
| *** rmart04 has joined #openstack-kolla | 22:03 | |
| *** rmart04 has left #openstack-kolla | 22:04 | |
| kfox1111 | sbezverk: got a script going to generate an exapmle cloud values.yaml. :) | 22:05 |
| kfox1111 | it produces: http://paste.openstack.org/show/zTlWpAkxph9x9ELmLNxK/ | 22:06 |
| *** zioproto has quit IRC | 22:06 | |
| kfox1111 | oops. one extra value in it needs removing... | 22:06 |
| kfox1111 | element_name. | 22:06 |
| *** zioproto has joined #openstack-kolla | 22:06 | |
| *** vhosakot_ has joined #openstack-kolla | 22:08 | |
| *** vhosakot has quit IRC | 22:12 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Add a script to build an example cloud yaml https://review.openstack.org/426417 | 22:14 |
| sbezverk | kfox1111: yeah, I like this file not huge and with easy structure | 22:15 |
| kfox1111 | :) | 22:15 |
| *** crushil has quit IRC | 22:16 | |
| sbezverk | now the question is how to use it? do you think a user will make a copy, changes variable and then use it in command line for each microservice/service? | 22:16 |
| kfox1111 | yeah. or for the service packages. | 22:17 |
| kfox1111 | yeah. | 22:17 |
| *** vhosakot_ has quit IRC | 22:17 | |
| sbezverk | yeah really nice, now config cane be stored/moved reused much much easier | 22:18 |
| kfox1111 | yeah. it worked that way for a while, but wasn't really obvious thats how it was inteneded. this makes it much clearer I hope. | 22:20 |
| sbezverk | +1 | 22:20 |
| kfox1111 | still gotta get the gate switched over to doing it this way too. got ground work in place, but the final swithc takes touching like 80% of the file. | 22:20 |
| openstackgerrit | sayantani proposed openstack/kolla: [TEST] DO NOT MERGE https://review.openstack.org/408368 | 22:23 |
| *** lrensing has joined #openstack-kolla | 22:27 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Check to see if Horizon is working in the gate. https://review.openstack.org/426025 | 22:30 |
| kfox1111 | sbezverk: this one is ready again: https://review.openstack.org/#/c/333996/ | 22:30 |
| *** PramodJ has joined #openstack-kolla | 22:31 | |
| *** mgiles has quit IRC | 22:32 | |
| *** pramodrj07 has quit IRC | 22:34 | |
| *** simon-AS559 has joined #openstack-kolla | 22:36 | |
| *** jmccarthy has left #openstack-kolla | 22:36 | |
| *** zioproto has quit IRC | 22:45 | |
| *** kbaegis has quit IRC | 22:48 | |
| *** kbaegis has joined #openstack-kolla | 22:49 | |
| *** goldyfruit has quit IRC | 22:49 | |
| *** kbaegis has quit IRC | 22:49 | |
| kfox1111 | so, is the horizon rolling upgrade too scary to review? | 22:56 |
| *** saneax-_-|AFK is now known as saneax | 22:59 | |
| *** aric49 has quit IRC | 22:59 | |
| sbezverk | kfox1111: just replied | 23:00 |
| kfox1111 | sbezverk: thanks. :) | 23:00 |
| sbezverk | when I looked at .dot file I wanted to shutdown my workstation | 23:00 |
| sbezverk | and run away ;) | 23:01 |
| kfox1111 | heh. grephviz is an interesting beast. | 23:01 |
| kfox1111 | looks like I messed up the formatting a bit though... I should fix that. | 23:01 |
| sbezverk | kfox1111: what is this selenium thing? | 23:02 |
| kfox1111 | sbezverk: selenium is a browser automation framework. | 23:03 |
| sbezverk | sounds like a chemical weapon to me | 23:03 |
| kfox1111 | hehe | 23:03 |
| kfox1111 | lets you create a pool of browsers, then remote control them with a standard api to do things. | 23:03 |
| kfox1111 | such as login to horizon and see if it works. :) | 23:03 |
| SamYaple | portdirect: i dont think too much. i dont think at all! you take that back! | 23:03 |
| sbezverk | kfox1111: Ah so you try to answer sdake request | 23:03 |
| kfox1111 | sbezverk: yeah. :) | 23:04 |
| kfox1111 | should be able to actually test if horizon is working instead of just that the initial reposnse comes back. | 23:04 |
| sbezverk | kfox1111: never heard that before, running bunch of browsers from the container %) | 23:04 |
| kfox1111 | seleinium + k8s = awesome. | 23:05 |
| kfox1111 | can scale out a browser cluster really easily. :) | 23:05 |
| kfox1111 | then use it to do all sorts of things... :) | 23:05 |
| sbezverk | kfox1111: it is mostly used for testing, right I cannot see other application of this? | 23:06 |
| sdake | what request did i do | 23:06 |
| sdake | sbezverk ^ | 23:07 |
| sdake | kfox1111 reviews that are longer in nature sit longer | 23:07 |
| sdake | kfox1111 its a time commitment to do a review | 23:07 |
| sdake | kfox1111 the best analogy I have herd on the matter is this | 23:07 |
| sdake | core reviewers wait for a review to fatten up a little bit | 23:07 |
| sdake | and then once they think its ready swim in like sharks and pick at it:) | 23:08 |
| inc0 | guys I really need help with this one | 23:08 |
| sdake | inc0 not sure sbezverk or kfox1111 can help although other s in teh cnannel cha help | 23:08 |
| inc0 | let me publish a bug with steps to reproduce | 23:08 |
| sdake | Jeffrey4l_ is gone for the weekend i assume | 23:08 |
| sbezverk | sdake you are forcing kfox1111 to run browser in the container!! it is just cruel | 23:08 |
| sdake | inc0 emea is mostly asleep :) | 23:08 |
| sdake | sbezverk i dont think i forced that? | 23:08 |
| sdake | sbezverk anyone can doanything they like :) | 23:09 |
| sbezverk | sdake: man you have short memory!! who asked to properly test horizon? | 23:09 |
| *** sdake_ has joined #openstack-kolla | 23:12 | |
| *** simon-AS559 has quit IRC | 23:14 | |
| *** sdake has quit IRC | 23:14 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Check to see if Horizon is working in the gate. https://review.openstack.org/426025 | 23:15 |
| *** lrensing has quit IRC | 23:16 | |
| kfox1111 | sdake_: yeah. was trying to triger the actual reviews. as it had been a while. | 23:17 |
| sdake_ | kfox1111 i have an easy review u pther ethat sits too - renaming everything from 0.4.0 to 0.50 | 23:18 |
| sdake_ | its literally a one line sed operation applied tothe repository | 23:18 |
| sdake_ | kfox1111 i can attempt looking today - althoug hthis mariadb thing on ubuntu is a problem and i need to help inc0 there | 23:18 |
| kfox1111 | sdake_: I actually reviewed it though. | 23:18 |
| sdake_ | kfox1111 oh cool didn't know that 0.40->0.5.0 had been reviewed :) | 23:19 |
| sdake_ | kfox1111 so much stuff going on in my life - hard to keep track :) | 23:19 |
| kfox1111 | sdake_: the review I asked for eyes on hadn't gotten a comment yet. | 23:19 |
| kfox1111 | sdake_: the short answer is, your ps changes a bunch of stuff that makes one of my ps's need rebasing, yet again. the ps that I just asked for reviews on. | 23:20 |
| kfox1111 | while your ps can be easily regenerated at any time. | 23:20 |
| kfox1111 | so it would be nice if it waited till after the other ps is merged, or its reabased on that ps. | 23:20 |
| inc0 | https://bugs.launchpad.net/kolla/+bug/1659981 | 23:20 |
| openstack | Launchpad bug 1659981 in kolla "galera deployments fails on ubuntu" [Critical,Confirmed] - Assigned to Michał Jastrzębski (inc007) | 23:20 |
| kfox1111 | +1 to helping inc0 | 23:21 |
| kfox1111 | galera's important. | 23:21 |
| *** sayantan_ has quit IRC | 23:21 | |
| inc0 | yeah I'm trying to debug it for 2 days now | 23:21 |
| inc0 | without success | 23:21 |
| kfox1111 | git bisect time? | 23:22 |
| inc0 | I already tried reverting patches | 23:22 |
| kfox1111 | see if you can find the ps for when it stopped working? | 23:22 |
| kfox1111 | all the way back to when it was branched off of 3? | 23:22 |
| inc0 | most likely candidates - I already tried | 23:22 |
| inc0 | no, I haven't got that far | 23:23 |
| kfox1111 | "Once you eliminate the impossible, whatever remains, no matter how improbable, must be the truth. | 23:23 |
| kfox1111 | :) | 23:23 |
| inc0 | well, I'll write script for that | 23:23 |
| sbezverk | kfox1111: yeah live long and prosper ;) | 23:24 |
| kfox1111 | so, I see two options... 1, a patch to kolla-ansible/kolla broke things. | 23:24 |
| kfox1111 | sbezverk: :) | 23:24 |
| inc0 | build, deploy, git revert | 23:24 |
| inc0 | until success | 23:24 |
| inc0 | kolla images | 23:24 |
| inc0 | I narrowed it down to images | 23:24 |
| kfox1111 | too bad irc doesn't support unicode | 23:24 |
| kfox1111 | http://emojipedia.org/raised-hand-with-part-between-middle-and-ring-fingers/ | 23:24 |
| kfox1111 | 2. some binary package broke things | 23:25 |
| inc0 | negative, fresh build of stable/newton fails | 23:25 |
| inc0 | and binary packages are the same there | 23:25 |
| kfox1111 | newton or ocata? | 23:25 |
| inc0 | stable/newton works* | 23:25 |
| inc0 | one idea, I'll do binary search ;) | 23:26 |
| inc0 | gonna build ocata-2 | 23:26 |
| kfox1111 | inc0: thats what git bisect is. ;) | 23:26 |
| inc0 | ah | 23:26 |
| inc0 | fun awaits | 23:26 |
| kfox1111 | https://git-scm.com/docs/git-bisect | 23:26 |
| kfox1111 | :) | 23:26 |
| inc0 | you learn something new every day | 23:27 |
| kfox1111 | gits really an awesome tool. :) | 23:27 |
| *** jtriley has quit IRC | 23:29 | |
| kfox1111 | git bisect run is particularly fun. :) | 23:30 |
| kfox1111 | you can automate the whole process. | 23:30 |
| sbezverk | kfox1111: when it asks you about commit everytime how can you be sure whether this specific commit is good or bad? | 23:31 |
| sbezverk | like in their exmaple | 23:32 |
| kfox1111 | problem being debugged dependent. | 23:33 |
| kfox1111 | so, in one example, say the problem is the code doesn't build. | 23:33 |
| sbezverk | if you need to run test for each commit where is the value? | 23:33 |
| openstackgerrit | Joseph Robinson proposed openstack/kolla-ansible: Build deploy guide files and tox environment https://review.openstack.org/426105 | 23:33 |
| sbezverk | since even without it you could do the same test | 23:33 |
| kfox1111 | you could run 'make' and if it works, its good, if it doesn,t' its bad. | 23:33 |
| kfox1111 | or you could run 'make; make test' if you have a unit testing framework. | 23:33 |
| kfox1111 | with git bisect run, you can give it a script to run on each commit tested for you, so you don't have to manually test. | 23:34 |
| sbezverk | ah ok its value that it allows you to go step by step changing your workspace | 23:34 |
| kfox1111 | yeah. it lets you not have to manually implement a binary search. | 23:35 |
| sbezverk | so instead of manually rolling back each commit from the top | 23:35 |
| sbezverk | it does it for you | 23:35 |
| kfox1111 | it does it in a binary search. | 23:35 |
| kfox1111 | so say you have 100 commits that are suspect, | 23:35 |
| kfox1111 | it might start at 50. | 23:35 |
| kfox1111 | if that fails, it will go to 25. | 23:35 |
| *** Penfold has quit IRC | 23:36 | |
| kfox1111 | if that suceeds, 37, | 23:36 |
| kfox1111 | etc. | 23:36 |
| sbezverk | ok instead of doing 100 iterations it will optimize it | 23:36 |
| kfox1111 | yeah. logrightmic amount of tests. | 23:36 |
| sbezverk | ok got it.. | 23:36 |
| kfox1111 | :) | 23:36 |
| kfox1111 | so for this case, you know 3.x works, and master doesn.t so you give it those two. | 23:37 |
| sbezverk | cool hopefully inc0 will use it, would be curious to know if it helps to nail it | 23:38 |
| sdake_ | inc0 git bisect rocks - i use it all the time | 23:39 |
| sdake_ | inc0 you do git bisect good | 23:39 |
| sdake_ | and git bisect bad | 23:39 |
| sdake_ | and it finds the commit that broke things | 23:40 |
| sdake_ | although in this case, it maybe breakage from an upstream repo that has changed rather than something in master | 23:40 |
| sdake_ | inc0 i'll be around tonight to pitch in | 23:40 |
| sdake_ | inc0 usually it tkes 7-8 bisect operations to find the commit that broke something within a full 6 month release | 23:41 |
| sdake_ | sometimes 9 | 23:41 |
| sdake_ | that is better then manually doing it with guessing :) | 23:41 |
| kfox1111 | sdake_: yeah. so might be good to try it with essentially 3.x kolla-ansible and 4.x kolla containers. | 23:42 |
| kfox1111 | if that works, then its probably kolla-ansible somewhere in between. | 23:42 |
| sdake_ | i think inc0 said it was in teh images | 23:42 |
| sdake_ | ok - gotta eat | 23:42 |
| sdake_ | bbl - then i'll be around to help you inc0 | 23:42 |
| kfox1111 | ah. | 23:43 |
| *** pramodrj07 has joined #openstack-kolla | 23:44 | |
| *** sayantani01 has joined #openstack-kolla | 23:46 | |
| kfox1111 | hehe. selenium is so much fun... | 23:47 |
| kfox1111 | driver.get_screenshot_as_file('/tmp/google.png') | 23:47 |
| *** PramodJ has quit IRC | 23:48 | |
| *** sdake_ has quit IRC | 23:48 | |
| *** lamt has quit IRC | 23:49 | |
| *** lamt has joined #openstack-kolla | 23:49 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Check to see if Horizon is working in the gate. https://review.openstack.org/426025 | 23:50 |
| *** lamt has quit IRC | 23:54 | |
| inc0 | I gtg now | 23:56 |
| inc0 | I'll be back on it tomorrow | 23:56 |
| inc0 | leave comment under bug plz if you manage to find anything | 23:56 |
| *** inc0 has quit IRC | 23:56 | |
| *** lrensing has joined #openstack-kolla | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!