kfox1111 | the numa in todays boxes is really fast in comparison. :) | 00:00 |
---|---|---|
kfox1111 | pinning does help a bit. | 00:00 |
kfox1111 | and I think thas one of the reasons out of the box the k8s solution is faster then the openstack one. | 00:00 |
kfox1111 | the kvm cpus float and arn't numa aware, | 00:00 |
kfox1111 | while the k8s containers are aware of it and can pin out of the box. | 00:00 |
kfox1111 | if your sceduler supports it. | 00:01 |
masber | I see | 00:01 |
masber | good to know thank you for that information | 00:01 |
kfox1111 | np. :) | 00:01 |
kfox1111 | you can make openstack numa aware and pin cores. | 00:02 |
masber | yes, actually ocata makes that very easy | 00:02 |
kfox1111 | its just a fair amount of work. | 00:02 |
inc0 | kfox1111: https://review.openstack.org/#/c/456407/ | 00:02 |
masber | you can set it up even on horizon when you create your image | 00:02 |
kfox1111 | if you look through the old summit videos you can find some of the bits. | 00:02 |
kfox1111 | look for tim bell's stuff. | 00:03 |
kfox1111 | nice. :) | 00:03 |
kfox1111 | haven't played with that feature yet. | 00:03 |
kfox1111 | my newest cloud is mitaka. should be newton in a few weeks. | 00:03 |
kfox1111 | I've usually been able to keep it closer to current, but getting more and more pushback on keeping it current. :/ | 00:04 |
masber | kolla should take care of all the pain when you do an upgrade is that right? | 00:05 |
inc0 | masber: we want to lower the pain | 00:05 |
inc0 | pain is there always | 00:05 |
masber | I understand | 00:06 |
SamYaple | kfox1111: pretty sure you can just set '--property hw:numa_nodes=1' on the flavor to lock it to a single numa node | 00:06 |
kfox1111 | openstack devs kind of like to hurt their users. :/ kolla's trying to make that better. | 00:06 |
kfox1111 | SamYaple: yah, but theres a bunch more to shrink the performance gap. | 00:06 |
kfox1111 | SamYaple: really for what I wanted to do, I want the vm to be fully numa aware. | 00:07 |
*** vhosakot has quit IRC | 00:07 | |
SamYaple | kfox1111: yea those are all just thinkgs you can set on the flavor though | 00:08 |
kfox1111 | for the system I'm depricating, we had one vm per physical host. | 00:08 |
SamYaple | its far from complicated | 00:08 |
SamYaple | https://docs.openstack.org/admin-guide/compute-flavors.html | 00:08 |
SamYaple | that should have info on it | 00:08 |
SamYaple | it *used* to be harder. back in like icehouse | 00:08 |
kfox1111 | SamYaple: setting options, easy. finding all the right options to elminiate/mostly reduce the 20% overhead, not easy. | 00:08 |
SamYaple | but stuff like numa pinning and cpu arch has been around for awhile | 00:08 |
kfox1111 | yeah. | 00:09 |
masber | there is not much benefit of using the whole host for one vm | 00:09 |
SamYaple | no its really nto hard at all. going to completely disagree there. this is all very straight forward | 00:09 |
SamYaple | its admin territory anyway, not user territory | 00:09 |
SamYaple | so all this info should be known | 00:09 |
SamYaple | no guess work | 00:09 |
kfox1111 | it was 0 work for k8s. it was a bunch of work for openstack. I went with the 0 work. ;) | 00:10 |
kfox1111 | and for other reasons. | 00:10 |
kfox1111 | it was a free 20% esentially. | 00:10 |
SamYaple | wow. that is comparing apples to oranges | 00:11 |
SamYaple | VMs vs processes | 00:11 |
kfox1111 | yes/no. | 00:11 |
kfox1111 | this project's goal was to get compute cycles. they didn't care how. | 00:11 |
kfox1111 | openstack was the best solution at the time we stood it up with the projects requirements. | 00:11 |
SamYaple | well anyway | 00:11 |
SamYaple | mnaser: numa is really easy in openstack for a couple of years now | 00:11 |
kfox1111 | k8s came around, and met all the requirements, and didn't have the vm overhead/admin overhead. | 00:12 |
kfox1111 | easy answer. :) | 00:12 |
SamYaple | mnaser: set '--property hw:numa_nodes=1' on the flavor and thats what you want 90% of the time. then you can refine further | 00:12 |
inc0 | kfox1111: it seems we deploy ceph too;) | 00:13 |
inc0 | in k8s | 00:13 |
masber | what tools do you use to meassure performance and bottlenecks? | 00:13 |
kfox1111 | inc0: doh. yeah. | 00:13 |
inc0 | I'll leave it out for tomorrow then | 00:13 |
kfox1111 | masber: switching to prometheus/node-exporter. having good results with it. | 00:14 |
SamYaple | mnaser: loaded question! by the time i get involved, most bottlenecks are found by me stracing something. a little higher up simpilier like iostat and ps are used | 00:15 |
kfox1111 | inc0: and iscsi too? | 00:15 |
kfox1111 | iscsi and tgtd. | 00:15 |
masber | this conversation was very useful to me, thank you all | 00:17 |
masber | :) | 00:17 |
inc0 | yeah I guess | 00:18 |
masber | on the other hand I am waiting openstack to be able to use AWS/google cloud/azzure infraestructure as an expansion | 00:18 |
masber | I think that would be the future | 00:19 |
*** yangyapeng has quit IRC | 00:19 | |
masber | so people would be able to do most of their computing requirements locally and expand if there are peaks of workloads | 00:19 |
masber | although clouds are quite cheap nowadays | 00:20 |
jascott1_ | kfox1111 do you know offhand how to set net=host in a pod spec? | 00:20 |
SamYaple | mnaser: yea "cloud bursting" is hat that is | 00:20 |
kfox1111 | jascott1_: yeah. sec. | 00:20 |
jascott1_ | maybe hostNetwork: true ? | 00:21 |
kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/helm/microservice/neutron-openvswitch-agent-daemonset/templates/openvswitch-agent-daemonset.yaml#L114 | 00:22 |
kfox1111 | yeah. | 00:22 |
kfox1111 | capital T though I think. | 00:22 |
kfox1111 | though probably doesn't matter since its go driven. | 00:22 |
kfox1111 | masber: I saw a project doing that. | 00:22 |
kfox1111 | added aws as a seperate openstack region. | 00:22 |
*** lrensing has joined #openstack-kolla | 00:23 | |
kfox1111 | was just a openstack api -> aws api adapter service. | 00:23 |
kfox1111 | alternately, you use openstack just toget vm's, and install k8s onto it, and use k8s as your abstraction api. | 00:24 |
kfox1111 | I use to standardize on heat as my orchestration system for deploying software. these days I'm moving to helm. | 00:27 |
masber | by the way my apologize I hijacked this channel to talk about other things | 00:27 |
kfox1111 | no worries. | 00:29 |
kfox1111 | weird weird weird.... | 00:30 |
kfox1111 | ocata is constantly failing on ceph attachment. | 00:31 |
kfox1111 | newton isnt... | 00:31 |
*** duonghq has joined #openstack-kolla | 00:31 | |
kfox1111 | packages between 3.0.3 and 4.0.0 in libvirt containers are basically the same.... | 00:32 |
kfox1111 | so, its gotta be nova-compute somehow.... | 00:32 |
kfox1111 | I guess I could try and pin back libvirt too... | 00:33 |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: v4 gates. https://review.openstack.org/454841 | 00:35 |
*** mattmceuen has quit IRC | 00:35 | |
duonghq | hi guys, | 00:36 |
*** ediardo has quit IRC | 00:37 | |
duonghq | currently, we are building source from tarballs.o.o or from source code? | 00:37 |
*** swinn has quit IRC | 00:39 | |
SamYaple | duonghq: the default is tarballs.o.o | 00:43 |
SamYaple | you can build from git directly as well | 00:43 |
duonghq | SamYaple, thanks | 00:43 |
*** zhubingbing_ has quit IRC | 00:44 | |
*** zhurong has joined #openstack-kolla | 00:44 | |
*** daidv has joined #openstack-kolla | 00:45 | |
daidv | SamYaple, morning. | 00:45 |
SamYaple | hello daidv | 00:46 |
SamYaple | morning | 00:46 |
daidv | How can I build source image from github? | 00:46 |
*** swinn has joined #openstack-kolla | 00:48 | |
*** MasterOfBugs has quit IRC | 00:49 | |
*** pramodrj07 has quit IRC | 00:49 | |
*** cuongnv has joined #openstack-kolla | 00:51 | |
*** swinn has quit IRC | 00:51 | |
*** rwellum has quit IRC | 00:51 | |
*** guoguo has joined #openstack-kolla | 00:53 | |
*** jpstnh has joined #openstack-kolla | 00:53 | |
*** masber has quit IRC | 00:54 | |
*** japestinho_ has joined #openstack-kolla | 00:54 | |
*** japestinho_ has quit IRC | 00:55 | |
*** masber has joined #openstack-kolla | 00:55 | |
daidv | SamYaple, I have found it by myself. thanks. | 00:57 |
openstackgerrit | Shunli Zhou proposed openstack/kolla-ansible master: Correct operating-kolla.rst document https://review.openstack.org/456110 | 01:08 |
*** Pavo has joined #openstack-kolla | 01:09 | |
*** al498u_ has quit IRC | 01:12 | |
*** tonanhngo has quit IRC | 01:20 | |
*** jpstnh has quit IRC | 01:21 | |
*** MarginHu has joined #openstack-kolla | 01:23 | |
*** tonanhngo has joined #openstack-kolla | 01:27 | |
*** hrw has quit IRC | 01:30 | |
*** tonanhngo has quit IRC | 01:32 | |
*** hrw has joined #openstack-kolla | 01:32 | |
*** tonanhngo has joined #openstack-kolla | 01:32 | |
*** tovin07_ has joined #openstack-kolla | 01:35 | |
*** srwilkers has quit IRC | 01:36 | |
openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add vSphere support to nova_compute https://review.openstack.org/449529 | 01:36 |
*** tonanhngo has quit IRC | 01:37 | |
openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix ceph role set target_max_bytes for cach pools https://review.openstack.org/452444 | 01:38 |
*** eanylin has joined #openstack-kolla | 01:44 | |
*** zhubingbing has joined #openstack-kolla | 01:45 | |
*** Pavo has quit IRC | 01:54 | |
*** guoguo has quit IRC | 01:58 | |
*** guoguo has joined #openstack-kolla | 01:59 | |
openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix ceilometer meters for swift service https://review.openstack.org/452373 | 02:07 |
sdake | kfox1111 the reason the libviert container sin 4.0.0 are different is because libvirt changes the permissions | 02:11 |
sdake | kfox1111 look to the implemetnation in kolla-ansible of ocnfig.json - which does the correct thing | 02:11 |
sdake | kfox1111 also 3.0.0 i think | 02:12 |
*** goldyfruit has joined #openstack-kolla | 02:15 | |
*** unicell has quit IRC | 02:15 | |
*** mattmceuen has joined #openstack-kolla | 02:17 | |
spsurya | morning all | 02:21 |
*** mkoderer has quit IRC | 02:27 | |
sdake | sup spsurya | 02:28 |
sdake | kfox1111 t he config.json is different for 4.0.0 then 2.0.0 | 02:40 |
sdake | kfox1111 this is one reason why kolla-ansible releases a x stream with a major release | 02:40 |
sdake | because maintaining the compatibility becomes increasingly difficult | 02:40 |
sdake | Jeffrey4l around? | 02:41 |
Jeffrey4l | yep sdake | 02:42 |
kolla-slack | <kfox1111> Sdake, 3.0.3 and 4.0.0 have the same packages, and 3 works but 4 doesnt. | 02:42 |
sdake | Jeffrey4l I've promised emiliean we would provide a throughou review of the spec that he is working on with dhellmann of the oslo changes | 02:42 |
sdake | Jeffrey4l i am not sure what review to review, but can you review it when emilien responds? | 02:43 |
kolla-slack | <kfox1111> Not a config.json issue, as thats from trunk for both | 02:43 |
sdake | Jeffrey4l any idea on kfox1111 's issue? | 02:43 |
sdake | same packages different config.json iirc | 02:43 |
Jeffrey4l | i got emilien mail. i will check it. | 02:44 |
kolla-slack | <kfox1111> Same config.json | 02:44 |
sdake | Jeffrey4l he has not responded yet | 02:44 |
sdake | Jeffrey4l he just responded :) | 02:44 |
Jeffrey4l | kfox1111, where is the issue description? | 02:44 |
sdake | kfox1111 can you please provide a review on https://review.openstack.org/#/c/454897/1/specs/queens/oslo-config-db.rst with an eye to how we plan to use native helm charts for config in kolla-kubernetes | 02:47 |
kfox1111 | mouse suddenly is not working... | 02:47 |
kfox1111 | give me a bit... | 02:47 |
kfox1111 | gotta reboot... | 02:50 |
masber | hi all, I would like to ask why kolla deploys a br-ex on each compute node? am I doing the deployment wrong or is it the expected behavior? | 02:50 |
masber | *kolla-ansible | 02:51 |
kfox1111 | oh. suspend/resume fixed it. | 02:51 |
kfox1111 | Jeffrey4l: https://review.openstack.org/#/c/454841/ is the ps. | 02:52 |
kfox1111 | http://logs.openstack.org/41/454841/12/check/gate-kolla-kubernetes-deploy-centos-binary-3-ceph-multi-nv/f63be04/logs/ works | 02:52 |
kfox1111 | http://logs.openstack.org/41/454841/12/experimental/gate-kolla-kubernetes-deploy-centos-binary-4-ceph-multi-nv/a2dea00/logs/ fails. | 02:52 |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: v4 gates. https://review.openstack.org/454841 | 02:54 |
Jeffrey4l | checking | 02:54 |
sdake | wow configmaps are 1.0mb | 02:54 |
kfox1111 | sdake: will review tomorrow. its a long read. but thanks for the pointer. should be interesting. | 02:54 |
sdake | kfox1111 ya - i promised i'd review, although i feel you are more qualified to respond then I am | 02:55 |
sdake | I will respond after your first to review with my pov | 02:55 |
sdake | I have read it an have a certain pov, but I dont' want to poison your thoughts | 02:55 |
kfox1111 | the more eyes the better. :) | 02:55 |
sdake | ya - emilien indicated there is no reason to proceed if there isn't broad appeal | 02:56 |
SamYaple | mnaser: br-ex on each compute is for provider networks (vlan networks). its not technically needed if all of your compute traffic goes over a vxlan tunnel to the l3-agent node | 02:56 |
sdake | thanks SamYaple :) | 02:56 |
Jeffrey4l | kfox1111, http://logs.openstack.org/41/454841/12/experimental/gate-kolla-kubernetes-deploy-centos-binary-4-ceph-multi-nv/a2dea00/console.html#_2017-04-12_22_14_28_064275 | 02:57 |
*** lrensing has quit IRC | 02:58 | |
Jeffrey4l | the vm is running in your failed gate. | 02:58 |
Jeffrey4l | but there is /dev/vdb disk http://logs.openstack.org/41/454841/12/experimental/gate-kolla-kubernetes-deploy-centos-binary-4-ceph-multi-nv/a2dea00/console.html#_2017-04-12_22_14_41_327737 | 02:58 |
kfox1111 | uh.... | 02:58 |
kfox1111 | yeah. thats the issue. | 02:58 |
Jeffrey4l | there is no* | 02:59 |
kfox1111 | ceph volumes work in 3.x but not 4.x | 02:59 |
Jeffrey4l | so this should not related to libvirt, at least | 02:59 |
kfox1111 | but same set of packages. | 02:59 |
kfox1111 | yea, thats what I'm thinking. | 02:59 |
kfox1111 | I just submitted a 4.x gate run with a 3.0.3 libvirt container just to double check. | 02:59 |
kfox1111 | thinking something changed in 4.x nova-compute maybe. | 03:00 |
Jeffrey4l | sudo /sbin/mkfs.vfat /dev/vdb <-- i think your gate script is wrong. | 03:00 |
Jeffrey4l | this should come from your deploy script. | 03:00 |
kfox1111 | works ok in the gate. | 03:00 |
kfox1111 | its running in cirrus | 03:00 |
*** lucasxu has joined #openstack-kolla | 03:01 | |
*** goldyfruit has quit IRC | 03:01 | |
Jeffrey4l | then the root cause is cirros do not load the /dev/vdb, | 03:01 |
kfox1111 | but same cirrus image is working fine in 3.x | 03:02 |
Jeffrey4l | found this http://logs.openstack.org/41/454841/12/experimental/gate-kolla-kubernetes-deploy-centos-binary-4-ceph-multi-nv/a2dea00/logs/openstack/kolla-nova-compute-qdqf5.txt | 03:02 |
Jeffrey4l | search "ERROR" | 03:02 |
Jeffrey4l | the attache volume is failed. | 03:03 |
*** sayantan_ has joined #openstack-kolla | 03:03 | |
kfox1111 | yeah. | 03:03 |
kfox1111 | but why. | 03:03 |
kfox1111 | its saying libvirt failed to attach, I think. | 03:03 |
Jeffrey4l | yep. not sure why now. never saw this issue. | 03:03 |
* Jeffrey4l is asking google | 03:03 | |
kfox1111 | still trying to figure out what could be different between 3 and 4 gates. very similar. | 03:03 |
sdake | kfox1111 the gates are simliar however openstack is completel ydifferent | 03:04 |
sdake | openstack is rewritten every 6 months :) | 03:04 |
kfox1111 | somewhat, yeah. | 03:04 |
sdake | kolla-ansible is widely tested by the community | 03:05 |
sdake | and 4.0.0 works | 03:05 |
sdake | so something is wrong with the helm implementation | 03:05 |
Jeffrey4l | check this | 03:05 |
Jeffrey4l | <auth username="cinder"> | 03:05 |
Jeffrey4l | <secret type="ceph" uuid="eee6ad83-5be4-4f29-b423-38e3e8262e22"/> | 03:05 |
Jeffrey4l | </auth> | 03:05 |
kfox1111 | http://logs.openstack.org/41/454841/12/experimental/gate-kolla-kubernetes-deploy-centos-binary-4-ceph-multi-nv/a2dea00/logs/virsh-secret-list.txt | 03:06 |
Jeffrey4l | nova-compute/libvirt is using ceph.cinder.client to connect vms pools | 03:06 |
kfox1111 | there is a secret in virsh for it. | 03:06 |
kfox1111 | so that seems ok. | 03:06 |
Jeffrey4l | did u inject the cinder key to libivrt? | 03:06 |
Jeffrey4l | just like nova keys | 03:06 |
kfox1111 | yup. | 03:07 |
kfox1111 | the virsh list there shows its loaded. | 03:07 |
Jeffrey4l | are u sure the value is correct? | 03:07 |
Jeffrey4l | how it is loaded in kolla-k8s? | 03:07 |
Jeffrey4l | seem it is different with kolla-ansible | 03:08 |
*** lucasxu has quit IRC | 03:09 | |
kfox1111 | sec. | 03:09 |
kfox1111 | similar to kolla-ansible. | 03:11 |
kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/helm/microservice/nova-libvirt-daemonset/templates/nova-libvirt.yaml#L55 | 03:11 |
kollian | japestinho: any idea about this http://paste.openstack.org/show/606396/ | 03:12 |
kfox1111 | the log code for the init containers here: http://logs.openstack.org/41/454841/12/experimental/gate-kolla-kubernetes-deploy-centos-binary-4-ceph-multi-nv/a2dea00/logs/pods/kolla-nova-libvirt-ptx20-initialize-nova-libvirt.txt | 03:13 |
kfox1111 | looked like it ran ok. | 03:13 |
Jeffrey4l | kfox1111, but only one key is loaded, right? | 03:14 |
*** lrensing has joined #openstack-kolla | 03:14 | |
kfox1111 | yeah. its nova local ephemeral, ceph volumes. | 03:14 |
Jeffrey4l | nova and cinder is using different secret since ocata. | 03:14 |
kfox1111 | that could be related. | 03:15 |
kfox1111 | though the log looks like its using the right uuid for the secret? | 03:15 |
Jeffrey4l | yes. but i am afraid the value is not correct. | 03:16 |
Jeffrey4l | i guess you are using cinder's rbd key + nova's cephx key | 03:16 |
Jeffrey4l | kfox1111, check this patch https://review.openstack.org/#/c/442519/ | 03:16 |
kfox1111 | its possible... | 03:17 |
kfox1111 | well... | 03:17 |
kfox1111 | looking at tools/setup_simple_ceph_users.sh, | 03:20 |
kfox1111 | the main difference between nova and cinder keys is nova has access to pool vm's and pool images too. | 03:20 |
*** lrensing has quit IRC | 03:20 | |
*** manheim has joined #openstack-kolla | 03:20 | |
kfox1111 | both have perms to volumes. | 03:20 |
kfox1111 | so for cinder volume attachment, shouldn't matter? | 03:20 |
*** ssurana has quit IRC | 03:21 | |
*** ssurana has joined #openstack-kolla | 03:21 | |
*** lrensing has joined #openstack-kolla | 03:21 | |
*** ssurana has quit IRC | 03:21 | |
Jeffrey4l | kfox1111, i thin so. | 03:21 |
Jeffrey4l | so please re-check this part. whether the secret_uuid and value is expected for both cinder and nova. | 03:22 |
*** manheim has quit IRC | 03:24 | |
kfox1111 | I think its the ceph uuid all the way through. | 03:27 |
*** zhubingbing has quit IRC | 03:27 | |
*** zhubingbing has joined #openstack-kolla | 03:27 | |
kfox1111 | cinder I mean. | 03:28 |
*** zhubingbing has quit IRC | 03:28 | |
kollian | Jeffrey4l: can you me in this http://paste.openstack.org/show/606396/ ? | 03:28 |
kollian | though it is seem like it is independent of kolla | 03:29 |
kollian | help me* | 03:29 |
masber | kollian, ansible can't ssh to the machine | 03:31 |
kfox1111 | ok, confirmed 4.x nova with 3.0.3 nova-libvirt still fails. | 03:32 |
kfox1111 | so it nova-compute or config related. | 03:32 |
masber | did you copy the keys across? | 03:32 |
kfox1111 | but config should be pretty much identicle betwen 3.x and 4.x... | 03:32 |
*** lrensing has quit IRC | 03:32 | |
masber | is the machine up? | 03:32 |
*** lrensing has joined #openstack-kolla | 03:33 | |
Jeffrey4l | kollian, you need configure passwordless login for ssh. | 03:35 |
kollian | Jeffrey4l: I have already configured the same for each VMs [1-3] | 03:36 |
Jeffrey4l | kollian, you machine do not think so : {"changed": false, "msg": "Failed to connect to the host via ssh: Host key verification failed.\r\n", "unreachable": true} | 03:37 |
kollian | masber: i am to ssh on each vm from each other | 03:37 |
kollian | able to* | 03:37 |
masber | kollian you can ssh without password? | 03:37 |
Jeffrey4l | Host key verification faile, clear you known_hosts file | 03:37 |
kollian | masber: yes | 03:37 |
masber | form the kolla host? | 03:38 |
masber | using the hostname as well or just the IP? | 03:38 |
Jeffrey4l | kollian, check http://stackoverflow.com/questions/21087695/not-able-to-git-pull-host-key-verification-failed | 03:38 |
kollian | masber: using hostname | 03:39 |
kollian | and IP | 03:39 |
masber | "he authenticity of host 'vmachine2 (192.168.121.212)' can't be established." that tells me hat ssh is not setup correctly | 03:41 |
masber | delete the ssh keys and redeploy it again | 03:42 |
masber | ssh-keygen -R <hostname> | 03:42 |
masber | ssh-copy-if root@<hostname> | 03:42 |
masber | and then ssh to the machine and make sure it doesn't ask for any password or anything | 03:43 |
masber | why you have 3 IPs and 3 hostnames on your inventory file? | 03:44 |
masber | do you have 6 machines? | 03:44 |
*** lrensing has quit IRC | 03:44 | |
masber | try using all hostnames or IPs | 03:44 |
kollian | masber: i have 3 VMs | 03:45 |
kollian | using one vm as deployment as well target node and other two as target node | 03:45 |
kollian | only | 03:45 |
masber | ok | 03:45 |
kollian | masber: is that fine | 03:46 |
masber | I separate kolla from controll node | 03:46 |
masber | but you script is complaining about all of them | 03:46 |
*** lrensing has joined #openstack-kolla | 03:46 | |
masber | *complaining about the 3 vms | 03:47 |
masber | do this | 03:47 |
masber | http://paste.openstack.org/show/606397/ | 03:49 |
masber | and then just make sure you only have hostnames in your inventory file | 03:50 |
masber | which should be the file called "multinode" | 03:50 |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: v4 gates. https://review.openstack.org/454841 | 03:50 |
masber | then try to deploy again | 03:51 |
masber | run *prechecks sorry | 03:51 |
kollian | masber: in my case i have 'vagrant' as root and Vmachine[1-3] as hostname | 03:52 |
masber | I normally run it as root | 03:52 |
*** lamt has joined #openstack-kolla | 03:53 | |
*** gkadam has joined #openstack-kolla | 03:53 | |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: v4 gates. https://review.openstack.org/454841 | 03:55 |
kollian | masber: i am using vm1 as deploy node and not bothering about the host where i have launched these three | 03:55 |
kollian | vms | 03:55 |
kollian | masber: vagrant@Vmachine1:/usr/local/share/kolla-ansible$ ssh-keygen -R vagrant@vmachine1 Host vagrant@vmachine1 not found in /home/vagrant/.ssh/known_hosts vagrant@Vmachine1:/usr/local/share/kolla-ansible$ | 03:55 |
*** ssurana has joined #openstack-kolla | 03:56 | |
*** zhurong has quit IRC | 03:57 | |
kollian | masber: http://paste.openstack.org/show/606399/ please check | 03:59 |
*** lrensing has quit IRC | 04:03 | |
kollian | masber: Jeffrey4l thanks | 04:03 |
kollian | i am checking at from my end | 04:04 |
*** fooliouno has joined #openstack-kolla | 04:11 | |
*** gkadam is now known as gkadam-afk | 04:15 | |
*** zhurong has joined #openstack-kolla | 04:16 | |
*** skramaja has joined #openstack-kolla | 04:32 | |
*** unicell has joined #openstack-kolla | 04:32 | |
*** Seyong has joined #openstack-kolla | 04:37 | |
*** gkadam-afk is now known as gkadam | 04:37 | |
*** unicell1 has joined #openstack-kolla | 04:38 | |
*** unicell has quit IRC | 04:39 | |
openstackgerrit | Taejeoung Kim proposed openstack/kolla-kubernetes master: fix link error https://review.openstack.org/456444 | 04:50 |
*** pramodrj07 has joined #openstack-kolla | 05:05 | |
*** MasterOfBugs has joined #openstack-kolla | 05:05 | |
masber | kollian, is it working now? | 05:08 |
masber | kollian, sorry just came back from lunch, looks like it can't resolve the machine names, you need to edit your /etc/hosts file | 05:09 |
masber | or just use IPs | 05:10 |
*** bmace has quit IRC | 05:11 | |
*** bmace has joined #openstack-kolla | 05:11 | |
*** claudiub has joined #openstack-kolla | 05:17 | |
*** lucasxu has joined #openstack-kolla | 05:19 | |
*** lucasxu has quit IRC | 05:20 | |
kollian | masber: thanks for reply | 05:31 |
kollian | masber: seems like the VMs I have created using vagrant | 05:32 |
kollian | is not able ssh via hostanam e | 05:32 |
kollian | masber: i am able to ssh via root user only | 05:33 |
kollian | masber: it is not working | 05:38 |
openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add VMware NSXV support to neutron https://review.openstack.org/450092 | 05:44 |
masber | ok you have to fix that | 05:45 |
masber | deployment node should be able to ssh as root to any node, just use IPs and not hostnames so you don't need to worry about name resolving | 05:45 |
masber | put IPs on the inventory file | 05:46 |
kollian | masber: i did http://paste.openstack.org/show/606402/ | 05:51 |
kollian | this is my multinode | 05:52 |
kollian | masber: please see at the botom | 05:53 |
kollian | of the paste | 05:53 |
*** ssurana has quit IRC | 05:54 | |
masber | that looks good to me | 05:56 |
*** dixiaoli has joined #openstack-kolla | 06:08 | |
*** gkadam is now known as gkadam-afk | 06:12 | |
*** jaosorior_away is now known as jaosorior | 06:18 | |
*** dvx has quit IRC | 06:18 | |
*** dvx has joined #openstack-kolla | 06:21 | |
*** lamt has quit IRC | 06:22 | |
*** satyar has joined #openstack-kolla | 06:24 | |
*** sayantan_ has quit IRC | 06:26 | |
*** imcsk8 has quit IRC | 06:26 | |
*** imcsk8 has joined #openstack-kolla | 06:26 | |
kollian | masber: me too, but again the same error | 06:29 |
masber | paste here how do you ssh into the node | 06:29 |
kollian | masber: http://paste.openstack.org/show/606405/ | 06:32 |
kollian | see what is working and what is not :( | 06:32 |
masber | as root | 06:32 |
*** lamt has joined #openstack-kolla | 06:32 | |
kollian | without root | 06:33 |
masber | ssh root@192.168.121.212 | 06:33 |
kollian | updated the same paste | 06:34 |
*** imcsk8 has quit IRC | 06:40 | |
*** imcsk8 has joined #openstack-kolla | 06:41 | |
*** mdnadeem has joined #openstack-kolla | 06:42 | |
*** varto has joined #openstack-kolla | 06:50 | |
*** pbourke has quit IRC | 06:51 | |
*** pbourke has joined #openstack-kolla | 06:53 | |
varto | Hi, question about config in kolla 4.0.0 (cinder). If I configure in global.xml backend_lvm=yes and backend_iscsi=no why in wm creation I still get error connecting to any iscsi portals ? | 06:56 |
*** manheim has joined #openstack-kolla | 06:56 | |
*** pcaruana has joined #openstack-kolla | 07:01 | |
*** manheim has quit IRC | 07:01 | |
*** pramodrj07 has quit IRC | 07:02 | |
*** MasterOfBugs has quit IRC | 07:02 | |
*** pramodrj07 has joined #openstack-kolla | 07:03 | |
*** MasterOfBugs has joined #openstack-kolla | 07:03 | |
*** PramodJ has joined #openstack-kolla | 07:04 | |
*** MasterOfBugs has quit IRC | 07:05 | |
*** pramodrj07 has quit IRC | 07:05 | |
*** MasterOfBugs has joined #openstack-kolla | 07:05 | |
varto | hmm maybe need to move iscsi portal ip from tgtd config.json to point in storage network interface ip - well anyways testing that. | 07:06 |
*** yangyapeng has joined #openstack-kolla | 07:06 | |
*** mewald has joined #openstack-kolla | 07:08 | |
hrw | morning | 07:28 |
hrw | anyone know what is with review.openstack.org? lags terribly | 07:28 |
spsurya | hrw: seems iunresponsive | 07:30 |
spsurya | unresponsive* | 07:30 |
spsurya | usually happens | 07:31 |
spsurya | hope will be good soon | 07:31 |
spsurya | Server Error | 07:32 |
*** lamt has quit IRC | 07:32 | |
*** rmart04 has joined #openstack-kolla | 07:34 | |
*** imcsk8 has quit IRC | 07:37 | |
*** imcsk8 has joined #openstack-kolla | 07:37 | |
*** egonzalez has joined #openstack-kolla | 07:40 | |
*** egonzalez has quit IRC | 07:46 | |
*** egonzalez has joined #openstack-kolla | 07:46 | |
*** Serlex has joined #openstack-kolla | 07:48 | |
*** calbers has joined #openstack-kolla | 07:51 | |
spsurya | egonzalez: hi..... | 07:52 |
egonzalez | spsurya, sup | 07:52 |
*** manheim has joined #openstack-kolla | 07:54 | |
spsurya | egonzalez: there was something but i got that :) | 07:54 |
spsurya | thsnk | 07:54 |
*** MasterOfBugs has quit IRC | 07:59 | |
*** PramodJ has quit IRC | 07:59 | |
*** jascott1_ has quit IRC | 08:05 | |
*** jascott1_ has joined #openstack-kolla | 08:05 | |
*** jascott1_ has quit IRC | 08:10 | |
*** yangyapeng has quit IRC | 08:10 | |
*** yangyapeng has joined #openstack-kolla | 08:10 | |
openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible stable/ocata: Fix outdated InfluxDB configuration https://review.openstack.org/456491 | 08:17 |
*** blallau has joined #openstack-kolla | 08:18 | |
*** gfidente has joined #openstack-kolla | 08:36 | |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le) https://review.openstack.org/430940 | 08:36 |
manheim | hi. If I'm deploying cinder, both iscsid and tgtd will be deployed? or only one depending on the distribution, ubuntu-tgt and rhel-iscsid? | 08:37 |
hrw | ok, I see that centos builds work today so sent rechecks for my patches | 08:41 |
egonzalez | pbourke, around? https://bugs.launchpad.net/kolla/+bug/1681920 | 08:42 |
openstack | Launchpad bug 1681920 in kolla "Neutron version is missing from pip list" [Undecided,New] | 08:42 |
egonzalez | pbourke, tried with images from dockerhub but dont see such issue, can you test in neutron server oraclelinux source image "pip list | grep -i neutron" if correct version appears? | 08:43 |
openstackgerrit | Dai Dang Van proposed openstack/kolla stable/newton: Backport "Fix Creating Ceilometer MongoDB database failed" https://review.openstack.org/456500 | 08:46 |
*** rmart04 has quit IRC | 08:46 | |
kollian | egonzalez: got some error http://paste.openstack.org/show/606422/ in AIO | 08:48 |
kollian | preccheck runs but deploy lands into error | 08:48 |
egonzalez | kollian, finally used master or stable branch? | 08:50 |
-openstackstatus- NOTICE: zuul was restarted due to an unrecoverable disconnect from gerrit. If your change is missing a CI result and isn't listed in the pipelines on http://status.openstack.org/zuul/ , please recheck | 08:51 | |
kollian | egonzalez: used master | 08:51 |
kollian | no registry at all | 08:51 |
kollian | just kolla-ansible install | 08:52 |
egonzalez | kollian, use git pull to download latest changes, a workaround has been merged yesterday | 08:52 |
kollian | egonzalez: was that related to same i am facing ? | 08:52 |
openstackgerrit | zhubingbing proposed openstack/kolla-ansible master: Fix neutron vpn external network traffic https://review.openstack.org/456509 | 08:53 |
egonzalez | kollian, where images come from? there are no published images for master branch. if images are from dockerhub are stable/ocata while ansible code is master | 08:53 |
egonzalez | kollian, yep, is exactly that issue fix | 08:53 |
kollian | egonzalez: you mean git pull in kolla-ansible dir. right ? | 08:54 |
egonzalez | kollian, yep | 08:55 |
egonzalez | kollian, but again, where images come from? that issue is only present with master images | 08:55 |
*** athomas has joined #openstack-kolla | 08:56 | |
kollian | egonzalez: you mean pulling the will also not work | 08:57 |
egonzalez | kollian, if images are stable/ocata, the issue is other than the fixed in master | 08:58 |
kollian | egonzalez: yes i think images are stable/ocata | 08:59 |
kollian | becoz i didn't any registry for images | 08:59 |
kollian | so the issue i am facing is not related to yesterday merged fix | 09:00 |
openstackgerrit | Dai Dang Van proposed openstack/kolla stable/newton: Fix Creating Ceilometer MongoDB database failed https://review.openstack.org/456513 | 09:00 |
kollian | egonzalez: right ? | 09:00 |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla stable/ocata: [DNM] test ocata branch https://review.openstack.org/456514 | 09:01 |
egonzalez | kollian, is rabbitmq running? can you check logs from nova at /var/lib/docker/volumes/kolla_logs/_data/nova ? | 09:02 |
kollian | root@Vmachine3:/usr/local/share/kolla-ansible# vi /var/lib/docker/volumes/kolla_logs/_data/nova/ nova-api.log nova-conductor.log nova-manage.log nova-scheduler.log placement-api.log nova-compute.log nova-consoleauth.log nova-novncproxy.log placement-api-access.log | 09:03 |
kollian | egonzalez: is there should be any log file rabbitmq name | 09:03 |
kollian | named* | 09:04 |
kollian | root@Vmachine3:/usr/local/share/kolla-ansible# ps -ef | grep rabbit root 15885 28598 0 09:04 pts/0 00:00:00 grep --color=auto rabbit root@Vmachine3:/usr/local/share/kolla-ansible# | 09:04 |
kollian | not running i guess | 09:04 |
egonzalez | kollian, docker logs rabbitmq | 09:05 |
*** sambetts|afk is now known as sambetts | 09:06 | |
openstackgerrit | Shunli Zhou proposed openstack/kolla-ansible master: Correct operating-kolla.rst document https://review.openstack.org/456110 | 09:06 |
kollian | egonzalez: I disabled it in all.yaml | 09:08 |
kollian | ? | 09:08 |
kollian | yes i did | 09:08 |
egonzalez | kollian, rabbitmq is required for almost every openstack service | 09:08 |
kollian | egonzalez: i disabled it when i ran into this http://paste.openstack.org/show/606428/ | 09:10 |
kollian | during prechecks | 09:10 |
openstackgerrit | zhubingbing proposed openstack/kolla-ansible master: Fix neutron vpn external network traffic https://review.openstack.org/456509 | 09:10 |
*** yangyapeng has quit IRC | 09:10 | |
mewald | Is anybody using elasticsearch with kolla? I found out programname is an analyzed field which makes in impossible to display logs by openstack service since "nova-scheduler" for example is broken into "nova" and "scheduler". Where and how can I change that? Has anybody done that? | 09:12 |
egonzalez | kollian, that means that the IP on api_interface has to resolve its hostname, in your case add <API IP> Vmachine3 in /etc/hosts | 09:12 |
egonzalez | check that ping -a Vmachine3 use the correct IP | 09:13 |
kollian | egonzalez: i checked it /etc/hosts container 127.0.0.1 corresponding to Vmachine3 | 09:14 |
kollian | so i think this correct for AIO | 09:15 |
kollian | and this also working ping -a Vmachine3 | 09:15 |
egonzalez | the IP which need to be resolvable is the IP on api_interface(eth0 or other interface) | 09:16 |
kollian | ok | 09:17 |
kollian | egonzalez: checking | 09:19 |
*** lpetrut has joined #openstack-kolla | 09:21 | |
*** jaosorior has quit IRC | 09:26 | |
*** skramaja has quit IRC | 09:27 | |
kollian | egonzalez: http://paste.openstack.org/show/606430/ | 09:27 |
kollian | this time with haproxy | 09:27 |
kollian | ? | 09:28 |
kollian | do i need to enabled it for AIO | 09:28 |
egonzalez | kollian, kolla_internal_vip_address IP should not be associated on any network device in your network | 09:28 |
kollian | you eth0 ? | 09:28 |
egonzalez | will be used by keepalived as VIP, if is used keepalived cannot use it | 09:28 |
kollian | you mean * | 09:28 |
egonzalez | kolla_internal_vip_address is free IP address in the range of api_interface | 09:29 |
kollian | should i leave that variable unset | 09:30 |
kollian | ? | 09:30 |
*** guoguo has quit IRC | 09:30 | |
*** rwsu has quit IRC | 09:30 | |
*** jaosorior has joined #openstack-kolla | 09:30 | |
*** gkadam-afk is now known as gkadam | 09:33 | |
egonzalez | kollian, nope, add a free IP in you network, my network is 192.168.100.0/24, I use 192.168.100.10 as VIP. ensure the IP is not used | 09:34 |
*** zhubingbing has joined #openstack-kolla | 09:37 | |
*** varto has quit IRC | 09:38 | |
kollian | egonzalez: i have no free ip i thinbk. As i am deploying on one single node vm | 09:41 |
kollian | so why it requires another ip | 09:41 |
kollian | i.e vip ? | 09:42 |
egonzalez | kollian, for load balancing. if is a VM isolated from other networks, then you have a full range with free IPs ;) | 09:47 |
openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Add graceful_timeout argument to kolla_docker https://review.openstack.org/453055 | 09:50 |
*** daidv has quit IRC | 09:59 | |
kollian | egonzalez: what if i deploy failed | 10:03 |
kollian | should i run any command to clean | 10:03 |
kollian | all the current setup | 10:03 |
kollian | or just change in configuration and redeploy | 10:04 |
kollian | becox thinks are mentioned in the doc | 10:04 |
kollian | thinkgs* | 10:04 |
*** tovin07_ has quit IRC | 10:04 | |
egonzalez | kollian, depends on which step failed, if is something blocking as db sync or ir just trying to copy a file | 10:05 |
egonzalez | if is blocking run kolla-ansible destroy | 10:05 |
egonzalez | if is not blocking just redeploy again | 10:06 |
*** jascott1_ has joined #openstack-kolla | 10:06 | |
zhubingbing | hi egonzalez https://review.openstack.org/#/c/456509/ | 10:07 |
zhubingbing | please review | 10:07 |
zhubingbing | ;) | 10:08 |
openstackgerrit | Dan Ardelean proposed openstack/kolla-ansible master: Add Hyper-V role https://review.openstack.org/455684 | 10:08 |
openstackgerrit | zhubingbing proposed openstack/kolla stable/ocata: make yum fail on missing packages https://review.openstack.org/451460 | 10:08 |
*** MarginHu has quit IRC | 10:09 | |
*** jascott1_ has quit IRC | 10:10 | |
sdake | kfox1111 you woudl think config is identical | 10:16 |
sdake | however config.json is different | 10:16 |
sdake | also there is this thing called "use_neutron" which is a delta | 10:17 |
*** zhurong has quit IRC | 10:17 | |
*** cuongnv has quit IRC | 10:17 | |
openstackgerrit | Duong Ha-Quang proposed openstack/kolla master: debian: move to stretch https://review.openstack.org/434453 | 10:19 |
egonzalez | zhubingbing, cannot understand what fixes in the vpn, I'm worried than setting br-ex will broke users who use more than one bridge | 10:19 |
zhubingbing | ok | 10:19 |
openstackgerrit | Duong Ha-Quang proposed openstack/kolla master: Add ceilometer agent ipmi docker file https://review.openstack.org/454281 | 10:21 |
egonzalez | zhubingbing, what is not working in vpnaas? | 10:22 |
zhubingbing | yeah | 10:22 |
zhubingbing | vpnaas function not work | 10:22 |
zhubingbing | vpnaas extern network function not work | 10:23 |
*** rwsu has joined #openstack-kolla | 10:23 | |
zhubingbing | we need to setting external_network_bridge = br-ex | 10:23 |
*** satyar has quit IRC | 10:23 | |
zhubingbing | https://review.openstack.org/#/c/59359/13/etc/l3_agent.ini | 10:23 |
zhubingbing | this patch | 10:23 |
egonzalez | zhubingbing, but that patch is from 3 years ago and is meant to allow multiple external network per l3 agent, setting the bridge removes such functionality | 10:26 |
sdake | sup zhubingbing | 10:28 |
*** zhubingbing has quit IRC | 10:28 | |
sdake | sup duonghq | 10:28 |
duonghq | hi sdake | 10:28 |
sdake | sup egonzalez | 10:28 |
egonzalez | sup sdake | 10:29 |
*** 21WAAAWJC has joined #openstack-kolla | 10:31 | |
21WAAAWJC | i am hear | 10:31 |
21WAAAWJC | ;) | 10:31 |
21WAAAWJC | egonzalez roger | 10:31 |
21WAAAWJC | hi sdake | 10:31 |
sdake | sup 21WAAAWJC | 10:31 |
21WAAAWJC | 。。 | 10:31 |
21WAAAWJC | sdake | 10:32 |
egonzalez | such a strange nick 21WAAAWJC hehe | 10:32 |
21WAAAWJC | wait | 10:32 |
*** 21WAAAWJC has quit IRC | 10:32 | |
*** zhubingbing has joined #openstack-kolla | 10:32 | |
zhubingbing | ;) | 10:32 |
zhubingbing | hi sdake | 10:32 |
zhubingbing | hi egonzalez | 10:32 |
*** duonghq has quit IRC | 10:34 | |
spsurya | hi sdake | 10:40 |
sdake | sup spsurya | 10:41 |
*** zhurong has joined #openstack-kolla | 10:45 | |
*** rwellum has joined #openstack-kolla | 10:52 | |
*** caowei has quit IRC | 10:54 | |
*** erlon has joined #openstack-kolla | 11:01 | |
openstackgerrit | zhubingbing proposed openstack/kolla-kubernetes master: Move ironic config to kolla-k8s https://review.openstack.org/450701 | 11:08 |
openstackgerrit | Merged openstack/kolla master: Add pymongo in panko images https://review.openstack.org/455667 | 11:13 |
openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix Freezer service config override issue https://review.openstack.org/456566 | 11:17 |
*** Pavo has joined #openstack-kolla | 11:18 | |
*** lpetrut has quit IRC | 11:20 | |
*** manheim has quit IRC | 11:21 | |
*** claudiub has quit IRC | 11:22 | |
*** zhubingbing has quit IRC | 11:29 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla stable/ocata: Add pymongo in panko images https://review.openstack.org/456569 | 11:30 |
*** dixiaoli has quit IRC | 11:31 | |
*** manheim has joined #openstack-kolla | 11:33 | |
*** lpetrut has joined #openstack-kolla | 11:33 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla stable/ocata: Add pymongo in panko images https://review.openstack.org/456569 | 11:34 |
*** rhallisey has joined #openstack-kolla | 11:36 | |
*** iceyao has joined #openstack-kolla | 11:38 | |
sdake | morning | 11:38 |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Fix zun images https://review.openstack.org/456251 | 11:39 |
*** MarginHu has joined #openstack-kolla | 11:45 | |
*** yingjun has joined #openstack-kolla | 11:46 | |
*** iceyao has quit IRC | 11:49 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix zun deployment https://review.openstack.org/456256 | 11:51 |
*** rwallner has joined #openstack-kolla | 11:52 | |
*** iceyao has joined #openstack-kolla | 11:52 | |
openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix Freezer service config override issue https://review.openstack.org/456566 | 11:53 |
rwellum | Morning sdake - I threw my hat in the ring on the kolla mentorship program thread | 11:53 |
*** rwallner has quit IRC | 11:53 | |
*** rwallner has joined #openstack-kolla | 11:54 | |
kollian | egonzalez: can you please know what can be kolla_internal_vip_address: if ifconfig is as follow for my VM | 11:54 |
kollian | http://paste.openstack.org/show/606439/ | 11:54 |
*** mewald has quit IRC | 11:55 | |
kollian | let me know* | 11:55 |
kollian | because i ran into this http://paste.openstack.org/show/606440/ | 11:57 |
*** iceyao has quit IRC | 11:58 | |
*** iceyao has joined #openstack-kolla | 11:59 | |
kollian | japestinho: hi | 11:59 |
openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix Freezer service config override issue https://review.openstack.org/456566 | 12:01 |
openstackgerrit | Merged openstack/kolla master: Add missing Jinja2 block for repo enablement https://review.openstack.org/454608 | 12:01 |
sdake | rwellum cool i'll read the thread - just owke up moment :) | 12:02 |
*** manheim has quit IRC | 12:04 | |
*** manheim has joined #openstack-kolla | 12:05 | |
*** lpetrut has quit IRC | 12:05 | |
*** jascott1_ has joined #openstack-kolla | 12:07 | |
*** jascott1_ has quit IRC | 12:11 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add zun deploy documentation https://review.openstack.org/456587 | 12:11 |
openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix 'node_config_directory' variable bad usage https://review.openstack.org/456588 | 12:12 |
egonzalez | kollian, kolla_internal_vip address cannot be the same IP in api_interface | 12:12 |
egonzalez | kollian, should be one IP in the same range but not used by any other device in the network | 12:12 |
*** goldyfruit has joined #openstack-kolla | 12:13 | |
*** lpetrut has joined #openstack-kolla | 12:13 | |
*** goldyfruit has quit IRC | 12:13 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add zun deploy documentation https://review.openstack.org/456587 | 12:15 |
kollian | egonzalez: ok | 12:16 |
*** lamt has joined #openstack-kolla | 12:22 | |
*** manheim has quit IRC | 12:23 | |
*** Daviey has quit IRC | 12:28 | |
openstackgerrit | Merged openstack/kolla-kubernetes master: Move mariadb configs to k8s https://review.openstack.org/456407 | 12:31 |
*** iceyao has quit IRC | 12:33 | |
*** iceyao has joined #openstack-kolla | 12:34 | |
sdake | sbezverk thanks a ton for htat last review dudue | 12:36 |
sdake | sbezverk that helped me fix the ironic gate pretty sur e;) | 12:36 |
openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Revert "Update to Helm v2.3.0" https://review.openstack.org/455502 | 12:37 |
openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Placement support in nova service chart https://review.openstack.org/453846 | 12:37 |
*** Pavo has quit IRC | 12:43 | |
sdake | rwellum note - when I asked for a review on this - I was stuck. Even our other core reviewers didn't find it - however through conversation in the review, it was found | 12:44 |
sdake | rwellum I'd encourage you to read the conversation in the last 4 or 5 comments - however - here is the ultimate change that makes ironic pass: https://review.openstack.org/#/c/453846/22..23 | 12:44 |
*** jpstnh has joined #openstack-kolla | 12:45 | |
*** manheim has joined #openstack-kolla | 12:46 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add zun startup guide documentation https://review.openstack.org/456587 | 12:46 |
sdake | rwellum what I really want you to get out of that is the conversatoin that led to the resolution of the problem in the patch | 12:48 |
*** iceyao has quit IRC | 12:48 | |
sdake | rwellum you will note that sbezverk didn't catch the problem, and I didn't catch the problem, and I must have done git show -1 100+ times | 12:48 |
sdake | rwellum however, through conversational exchange, his commentary that the code was already there triggered thinking in my brain to go look at he line number he pointed out, which resulted in me determing && was wrong and -a was correct | 12:49 |
sdake | rwellum this is how a good review should work :) | 12:49 |
*** iceyao has joined #openstack-kolla | 12:50 | |
*** MarginHu has quit IRC | 12:51 | |
*** MarginHu has joined #openstack-kolla | 12:51 | |
*** rhallisey has quit IRC | 12:51 | |
*** lamt has joined #openstack-kolla | 12:56 | |
*** zhubingbing has joined #openstack-kolla | 12:56 | |
*** Daviey has joined #openstack-kolla | 12:57 | |
*** Pavo has joined #openstack-kolla | 12:57 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add zun startup guide documentation https://review.openstack.org/456587 | 12:59 |
*** Pavo has quit IRC | 13:01 | |
*** lpetrut has quit IRC | 13:03 | |
*** Pavo has joined #openstack-kolla | 13:03 | |
*** Pavo has quit IRC | 13:03 | |
*** Pavo has joined #openstack-kolla | 13:04 | |
*** mattmceuen has quit IRC | 13:05 | |
*** Pavo has joined #openstack-kolla | 13:06 | |
*** jtriley has joined #openstack-kolla | 13:07 | |
rwellum | sdake - interesting bug :) | 13:10 |
sdake | rwellum ya I'm just a dummy that doesn't know shell scripting | 13:10 |
sdake | blind to the obvious | 13:10 |
*** rhallisey has joined #openstack-kolla | 13:10 | |
sdake | as were other reviewers | 13:11 |
sdake | rwellum the key thing is the exchange | 13:11 |
sdake | rwellum via exchange in the review, we can make forward progress when it appears no more forward progress canbe made | 13:11 |
*** esharao has joined #openstack-kolla | 13:11 | |
rwellum | Do you know if old shell is a standard historically? - I'm kinda surprised we're not using bash notation. | 13:14 |
openstackgerrit | Merged openstack/kolla-ansible master: Fix Freezer service config override issue https://review.openstack.org/456566 | 13:14 |
*** satyar has joined #openstack-kolla | 13:15 | |
sdake | rwellum i am super far from a bash nerd - although I know [[ ]] would work with && | 13:19 |
sdake | and i know [ ] doesn't | 13:19 |
sdake | rwellum with [ ] you have to use -a and -o | 13:20 |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Configure coordination_url for cloudkitty processor https://review.openstack.org/455504 | 13:20 |
sdake | rwellum I am not sure we have a standard in kolla-kubernetes aroudn the gate scripts related to shell | 13:20 |
sdake | rwellum I was more "copying" the look and feel i saw there | 13:20 |
sdake | rwellum since you voiced interest on the mailing list check out: | 13:21 |
sdake | https://etherpad.openstack.org/p/kolla-mentorship-signup | 13:21 |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix Freezer service config override issue https://review.openstack.org/456606 | 13:22 |
sdake | sbezverk thanks dude - gate is green:-) https://review.openstack.org/#/c/453846/ | 13:27 |
sdake | thats like the first patch I've written all year of any substance... | 13:27 |
sdake | sadly :( | 13:27 |
*** iceyao has quit IRC | 13:29 | |
*** mbruzek has joined #openstack-kolla | 13:30 | |
*** mbruzek has joined #openstack-kolla | 13:30 | |
*** iceyao has joined #openstack-kolla | 13:30 | |
sbezverk | sdake: changing names does not sound a good move, think about it, all gate jobs were ok, your not, if it were naming issue, then it would impact all other jobs.. | 13:31 |
sdake | sbezverk brain booting | 13:32 |
sdake | sbezverk there was a little more to it | 13:32 |
*** eanylin has quit IRC | 13:32 | |
sdake | sbezverk trying to pull it from depths of my brain | 13:32 |
sdake | sbezverk the placement api is only used in the soruce-4 ironci gate | 13:32 |
sdake | or is that correct? | 13:32 |
sbezverk | sdake: as it looks now unfortunately it should not get merged, no consistency | 13:33 |
sdake | well, i am happy to make it consistent | 13:33 |
sdake | i just want it to be the correct consistent | 13:33 |
sdake | vs the incorrect consistent ;) | 13:33 |
sbezverk | sdake: I suggest to revert name changes and find out if there was a real problem with it | 13:33 |
*** lpetrut has joined #openstack-kolla | 13:34 | |
sdake | there is indeed | 13:34 |
sdake | the charts dont start | 13:34 |
sdake | i didn't make that change for my good health ;) | 13:34 |
sdake | the history is in the gate logs - no need to even revert | 13:34 |
sbezverk | sdake: you are joking right? how charts were not starting if before you started working on this ps they were running for at leats 2-3 weeks? | 13:35 |
*** japestinho has quit IRC | 13:40 | |
sdake | sbezverk i'm not really sure how to answer your question | 13:40 |
sdake | other then I have no clue why renaming the resource fixes the problem | 13:40 |
sdake | when dealing with third party dependencies, a no clue answer is sometimes the best answer we are able to achieve | 13:40 |
sbezverk | sdake: but that is the problem, I have hard time understanding when something gets changed without complete understanding of reason. In my profession it is called a root cause. | 13:41 |
sdake | sbezverk sometimes the cost of determing the root cause is too expesnive | 13:42 |
sbezverk | if there is no root cause, then what was changed cannot be called a fix, it is just a wild shot in dark, which luckely did not kill any body but somehow cuase this positive impact.. | 13:42 |
sdake | I realize in your field determing root cause is always the most critical thing | 13:42 |
sdake | in R&D engineering , root cause is often not hte most critical thing | 13:42 |
*** srwilkers has joined #openstack-kolla | 13:42 | |
sbezverk | sdake: that is why quality is always suffering these days and we have software which is crashing or needed to be rebooted weekly | 13:43 |
sdake | sbezverk can you tell me what this error means: http://logs.openstack.org/46/453846/17/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/17c1634/console.html#_2017-04-10_01_46_56_337618 | 13:43 |
sdake | sbezverk you may possibly be correct :) | 13:43 |
sdake | sbezverk this is *without* any renaming | 13:44 |
sbezverk | sdake: it means somebody made a mistake but not correctly specifying requirement, but interestingly instead of changing requirement file the solution was to start changing chrts ?!?!? | 13:45 |
*** zhubingbing has quit IRC | 13:45 | |
rwellum | sdake: signed up | 13:46 |
*** zhubingbing has joined #openstack-kolla | 13:46 | |
*** al498u has joined #openstack-kolla | 13:46 | |
sdake | rwellum cool | 13:46 |
sdake | sbezverk ok - not coreclty specifying requirement | 13:46 |
sbezverk | sdake: if you compare the line you pasted me with line just above in the log, what difference do you see? | 13:47 |
*** al498u_ has joined #openstack-kolla | 13:48 | |
sdake | sbezverk which line did I paste you? I dont follow | 13:48 |
sbezverk | sdake: you pasted me here on IRC a log pointing to one line, right? | 13:48 |
sdake | right - the warning line | 13:49 |
sbezverk | yes, I am asking you to compare that line with one just above it and tell me the difference | 13:49 |
sbezverk | in the same log | 13:49 |
sbezverk | specifically the structure of the microservice name | 13:50 |
sdake | 2017-04-10 01:46:52.356371 | Archiving ironic-conductor-daemonset from repo file://../../microservice/ironic-conductor-daemonset | 13:50 |
sdake | 2017-04-10 01:46:56.337618 | Warning: nova-placement-create-keystone-endpoint-internal is in requirements.yaml but not in the charts/ directory! | 13:50 |
sdake | I dont totally follow :) | 13:50 |
sdake | but i'm willing to learn | 13:50 |
sbezverk | how the name end in both cases? | 13:51 |
*** al498u has quit IRC | 13:51 | |
srwilkers | missing the suffix | 13:51 |
sdake | with teh object type | 13:51 |
srwilkers | -job | 13:51 |
sbezverk | swilkers, thank you :-) | 13:51 |
satyar | Hi All | 13:52 |
srwilkers | sbezverk, cheers dude | 13:52 |
sdake | i did give it a go of adding the suffix IIRC | 13:52 |
sdake | sup satyar | 13:52 |
satyar | if ask.openstack.org down? | 13:52 |
satyar | How are you sdake :) | 13:52 |
sdake | satyar ask @openstack-infra | 13:52 |
satyar | long time | 13:52 |
satyar | yes | 13:52 |
sdake | #openstack-infra that is | 13:52 |
sbezverk | sdake: instead on starting changing charts | 13:52 |
satyar | ok | 13:52 |
sbezverk | you should have analized the failure more carefully | 13:52 |
sdake | i did anlyze the requirements.yaml at length | 13:54 |
sdake | + condition: global.kolla.nova.all.placement_api_enabled | 13:55 |
sdake | + - name: nova-placement-create-keystone-endpoint-public-job | 13:55 |
sdake | + repository: file://../../microservice/nova-placement-create-keystone-endpoint-public-job | 13:55 |
egonzalez | satyar, is UP for me | 13:55 |
sdake | in fact I got wrapped aroudn the axle on this point for 3 or 4 hours | 13:55 |
sdake | sbezverk as you see there, the object type is properly named | 13:55 |
*** Administrator_ has quit IRC | 13:56 | |
sdake | sbezverk i also compared the 3 jobs that fialed with the 3 that didn't | 13:56 |
sdake | and could not discern a delta | 13:56 |
*** Administrator_ has joined #openstack-kolla | 13:56 | |
*** Pavo has quit IRC | 13:56 | |
*** goldyfruit has joined #openstack-kolla | 13:56 | |
*** zhurong has quit IRC | 13:57 | |
*** yingjun has quit IRC | 13:58 | |
sdake | sbezverk do you see one I am missing? | 13:58 |
*** goldyfruit1 has joined #openstack-kolla | 13:59 | |
*** lamt has joined #openstack-kolla | 14:00 | |
sbezverk | sdake: in the paste you clearly missing suffic, I suggest you revert back all changes you have done for microcharts and we analyze each failure why and how properly to address it.. | 14:00 |
*** goldyfruit1 has quit IRC | 14:00 | |
sdake | sbezverk there is a suffix missing from something | 14:00 |
*** goldyfruit1 has joined #openstack-kolla | 14:00 | |
sdake | sbezverk but it is not requirements file | 14:00 |
sdake | sbezverk i also took a look at the build script itself | 14:01 |
*** goldyfruit has quit IRC | 14:01 | |
sdake | as i thought it was possible it was defective | 14:01 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible stable/ocata: Updated from global requirements https://review.openstack.org/452870 | 14:01 |
*** lrensing has joined #openstack-kolla | 14:01 | |
sbezverk | sdake: wow, that script been building for months now | 14:02 |
sbezverk | sdake: and why anybody else would not report it ;) ?? | 14:02 |
*** signed8bit has joined #openstack-kolla | 14:03 | |
sdake | sbezverk i make no assumptions when develooping software :) | 14:03 |
sdake | sbezverk good rule to follow :) | 14:03 |
sbezverk | sdake: I do :) and the first assumoption if something is not working is I made a mistake | 14:03 |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix zun deployment https://review.openstack.org/456256 | 14:03 |
sdake | sbezverk anyway my investigation there turned up nothign: https://github.com/openstack/kolla-kubernetes/blob/master/tools/helm_prebuild_microservices.py#L98-L102 | 14:04 |
sdake | reverting the resource name changes doesn't really tell us anything we dont already know in the gate | 14:05 |
sdake | i am happy to do it | 14:05 |
sdake | but fail to understand what the next step from there is | 14:06 |
sdake | i see that the suffix is missign from the warning | 14:06 |
sdake | I dont understand why its missing | 14:06 |
*** goldyfruit1 has quit IRC | 14:07 | |
*** yangyapeng has joined #openstack-kolla | 14:08 | |
*** lpetrut has quit IRC | 14:08 | |
sdake | STDAKE-M-26CJ:kolla-kubernetes sdake$ grep -r "nova-placement-create-keystone-endpoint-public" * | grep -v job | 14:08 |
sdake | tests/bin/ceph_workflow.sh: --namespace kolla --name nova-placement-create-keystone-endpoint-public \ | 14:08 |
sdake | tests/bin/deploy_iscsi_common.sh: --namespace kolla --name nova-placement-create-keystone-endpoint-public \ | 14:08 |
*** iceyao has quit IRC | 14:09 | |
sbezverk | sdake: I am sorry, but as I said if you revert back names of microcharts you changed then we can pin point the issue in PS, otherwise it is very difficult as I do not undersatnd your logic here.. | 14:10 |
*** lucasxu has joined #openstack-kolla | 14:11 | |
*** jpstnh has quit IRC | 14:13 | |
*** sayantan_ has joined #openstack-kolla | 14:14 | |
*** goldyfruit has joined #openstack-kolla | 14:17 | |
openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix 'node_config_directory' variable bad usage https://review.openstack.org/456588 | 14:17 |
*** iceyao has joined #openstack-kolla | 14:20 | |
*** iceyao has quit IRC | 14:21 | |
*** japestinho has joined #openstack-kolla | 14:21 | |
*** goldyfruit has quit IRC | 14:22 | |
*** claudiub has joined #openstack-kolla | 14:23 | |
*** krtaylor has joined #openstack-kolla | 14:25 | |
*** fguillot has joined #openstack-kolla | 14:26 | |
*** japestinho has quit IRC | 14:30 | |
*** manheim has quit IRC | 14:34 | |
*** lpetrut has joined #openstack-kolla | 14:36 | |
*** unicell has joined #openstack-kolla | 14:37 | |
*** unicell1 has quit IRC | 14:38 | |
*** MarginHu has quit IRC | 14:39 | |
openstackgerrit | Merged openstack/kolla-ansible master: Split Openvswitch into own role https://review.openstack.org/416793 | 14:39 |
openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Placement support in nova service chart https://review.openstack.org/453846 | 14:42 |
sdake | sbezverk ok that has been reverted, what do you suggest as a next step - this is not typically how I do development although interested to learn other's development style | 14:43 |
openstackgerrit | Merged openstack/kolla-ansible stable/ocata: Configure coordination_url for cloudkitty processor https://review.openstack.org/455504 | 14:43 |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add upgrade release note for ovs split https://review.openstack.org/456651 | 14:44 |
sbezverk | sdake: after we get gates results, we can go through the log | 14:44 |
sdake | sbezverk k | 14:44 |
openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Reparent kolla-toolbox from openstack-base https://review.openstack.org/435023 | 14:45 |
*** iceyao has joined #openstack-kolla | 14:46 | |
*** iceyao has quit IRC | 14:50 | |
*** Pavo has joined #openstack-kolla | 14:52 | |
*** zhubingbing_ has joined #openstack-kolla | 14:52 | |
*** zhubingbing has quit IRC | 14:54 | |
sdake | sbezverk quesiton re http://logs.openstack.org/46/453846/17/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/17c1634/console.html#_2017-04-10_01_46_56_337618 | 14:55 |
sdake | what does it mean that the chart is not in the charts directory? | 14:55 |
sdake | it may sound like an obvious statement | 14:55 |
sdake | waht I don't comprehend is how the charts directory gets populated | 14:56 |
sdake | sbezverk you will also notice this further up: http://logs.openstack.org/46/453846/17/check/gate-kolla-kubernetes-deploy-centos-binary-2-helm-entrypoint-nv/17c1634/console.html#_2017-04-10_01_46_50_384552 | 14:57 |
sdake | where it was indeed archived | 14:57 |
*** fooliouno has quit IRC | 14:57 | |
sdake | one of the thought procesees i went through when debugging this was that "why is ironic packaging the placement api" | 14:58 |
*** yingjun has joined #openstack-kolla | 15:01 | |
*** Pavo has quit IRC | 15:01 | |
sdake | kfox1111 wrt your change to workaround the 1mb limit - how much headroom do we have available now with your workaround? | 15:03 |
*** spsurya__ has joined #openstack-kolla | 15:03 | |
*** yangyapeng has quit IRC | 15:05 | |
*** obre has joined #openstack-kolla | 15:05 | |
*** gkadam has quit IRC | 15:06 | |
*** jowisz_ has joined #openstack-kolla | 15:06 | |
*** yangyapeng has joined #openstack-kolla | 15:06 | |
*** pomac_ has joined #openstack-kolla | 15:06 | |
*** Slower_ has joined #openstack-kolla | 15:07 | |
*** mdnadeem has quit IRC | 15:07 | |
*** afranc has joined #openstack-kolla | 15:09 | |
*** esmiurium_ has joined #openstack-kolla | 15:09 | |
*** zigo_ has joined #openstack-kolla | 15:09 | |
*** pcaruana has quit IRC | 15:10 | |
*** kencjohnston_ has joined #openstack-kolla | 15:10 | |
*** berendt has quit IRC | 15:10 | |
*** pomac has quit IRC | 15:10 | |
*** pron has quit IRC | 15:10 | |
*** obre_ has quit IRC | 15:10 | |
*** aimeeu has quit IRC | 15:10 | |
*** kencjohnston has quit IRC | 15:10 | |
*** alanmeadows has quit IRC | 15:10 | |
*** jgriffith has quit IRC | 15:10 | |
*** spsurya has quit IRC | 15:10 | |
*** Aju has quit IRC | 15:10 | |
*** esmiurium has quit IRC | 15:10 | |
*** EmilienM has quit IRC | 15:10 | |
*** serverascode has quit IRC | 15:10 | |
*** zigo has quit IRC | 15:10 | |
*** jowisz has quit IRC | 15:10 | |
*** Slower has quit IRC | 15:10 | |
*** spsurya__ is now known as spsurya | 15:10 | |
*** alanmeadows_ has joined #openstack-kolla | 15:11 | |
*** alanmeadows_ has quit IRC | 15:11 | |
*** alanmeadows_ has joined #openstack-kolla | 15:11 | |
*** aimeeu has joined #openstack-kolla | 15:11 | |
*** pron has joined #openstack-kolla | 15:11 | |
*** zigo_ is now known as zigo | 15:12 | |
*** zigo is now known as Guest52586 | 15:12 | |
*** 21WAAA2JF has joined #openstack-kolla | 15:13 | |
sbezverk | sdake: looks like infra failure | 15:14 |
*** lamt has quit IRC | 15:15 | |
*** berendt has joined #openstack-kolla | 15:15 | |
sdake | recheck ftw | 15:16 |
*** serverascode has joined #openstack-kolla | 15:20 | |
*** lpetrut has quit IRC | 15:20 | |
*** fooliouno_ has joined #openstack-kolla | 15:24 | |
*** lamt has joined #openstack-kolla | 15:25 | |
*** yangyapeng has quit IRC | 15:27 | |
*** manheim has joined #openstack-kolla | 15:28 | |
*** fooliouno_ has quit IRC | 15:29 | |
*** fooliouno has joined #openstack-kolla | 15:30 | |
*** manheim has quit IRC | 15:32 | |
*** Manheim has joined #openstack-kolla | 15:32 | |
*** Pavo has joined #openstack-kolla | 15:36 | |
kfox1111 | morning. | 15:36 |
kfox1111 | sdake: no idea. | 15:38 |
zhubingbing_ | morning | 15:39 |
openstackgerrit | Merged openstack/kolla-ansible master: Remove the variable redefined in deploy-servers.yml https://review.openstack.org/448433 | 15:39 |
*** mbruzek has quit IRC | 15:39 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix outdated InfluxDB configuration https://review.openstack.org/456491 | 15:40 |
*** swinn has joined #openstack-kolla | 15:44 | |
sbezverk | kfox1111: good morning.. I am trying to print out the size of that config map to see if I can answer sdake question.. | 15:45 |
*** goldyfruit has joined #openstack-kolla | 15:46 | |
sdake | sbezverk thanks - although I did notice somewhre 9can't remember where) the configmap was in raw format in the gate logs | 15:46 |
kfox1111 | sbezverk: good morning. ok. cool. thanks. | 15:46 |
sdake | sbezverk not sure wherer i saw it | 15:46 |
kfox1111 | sdake: all the ocnfigmaps are. it could be extracted. | 15:46 |
sbezverk | kfox1111: yyeeeessss it is only 440k now :) | 15:48 |
sbezverk | Apr 13 11:46:49 kube-1 journal: 2017/04/13 15:46:49 cfgmaps.go:242: Encoded data size is 411416 | 15:48 |
kfox1111 | sbezverk: ok. cool. that buys us a bit of time. | 15:48 |
kfox1111 | We're going to run up against that again when we add configmap charts and parent values. | 15:48 |
sbezverk | kfox1111: man it is half of limit, I would say a lot of time :) | 15:49 |
kfox1111 | we shall see. :) | 15:49 |
sdake | cool thanks for that information sbezverk | 15:49 |
kfox1111 | I could see it getting close again with those two features. but should be the major missing functionality too. | 15:49 |
sdake | in the meantime perhaps jascott1 can sort out how to expand the limit | 15:50 |
kfox1111 | we could work around it by breaking up kolla-common to a few smaller charts if needed though to work around it further. | 15:50 |
sbezverk | sdake: that limit is imposed by kubernetes, not helm | 15:50 |
sdake | sbezverk i understand | 15:50 |
sdake | sbezverk i didn't say he would fix it in helm, I said he would fix it :) | 15:50 |
kfox1111 | yes, but helm could work around it probably easier then k8s can. | 15:50 |
sdake | or i requested if he woudld rather ;) | 15:50 |
swinn | morning all, seems I’m working through one more issue - my compute containers do not seem to load the kvm modules even though the host supports the processor flags | 15:50 |
*** fguillot has quit IRC | 15:51 | |
swinn | I’ve specified the virt_type in nova.conf and still the hypervisors report QEMU as their type | 15:51 |
sbezverk | kfox1111: I read some comment in kuberenetes issues queue complaining about it.. so maybe fixing it on kubernetes side is also a possible option.. | 15:51 |
kfox1111 | could be. | 15:51 |
sbezverk | sdake: no luck, it looks like centos mirrors are busted again | 15:52 |
sbezverk | sdake: we can look into placement after infra recovers them.. | 15:53 |
sdake | sbezverk cool | 15:53 |
sdake | i've got to jet - training today | 15:54 |
sdake | sbezverk thanks for helping - hopefully we can get thatwork merged soon | 15:54 |
*** fguillot has joined #openstack-kolla | 15:55 | |
*** zhubingbing_ has quit IRC | 15:57 | |
*** alanmeadows_ is now known as alanmeadows | 16:00 | |
*** Pavo has quit IRC | 16:01 | |
*** Pavo has joined #openstack-kolla | 16:03 | |
*** Pavo has quit IRC | 16:03 | |
*** egonzalez has quit IRC | 16:03 | |
*** Pavo has joined #openstack-kolla | 16:04 | |
*** Pavo has quit IRC | 16:04 | |
openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix 'node_config_directory' variable bad usage https://review.openstack.org/456588 | 16:05 |
openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Simplify Openstack services configuration possibilities https://review.openstack.org/456688 | 16:05 |
*** mbruzek has joined #openstack-kolla | 16:05 | |
sbezverk | kfox1111: question about fernet.. I ran compute-kit and the deployment worked fine, then I cleaned up all known to me kube objects and run deployment second time, on the second time I see fernet setup job is crashing. Can you think of any leftover from previous installation which might be causing it? | 16:08 |
kfox1111 | it only has a secret. no other perminant bits. | 16:09 |
kfox1111 | did you delete the secret? | 16:09 |
kfox1111 | the secret isn't helm managed. so we may need a job to clear it. | 16:10 |
sbezverk | kfox1111: nope cause I am re-using the same secrets | 16:10 |
kfox1111 | the fernet creation script probably doesn't like running if the secret preexists. | 16:11 |
kfox1111 | never really tested for that case. | 16:11 |
*** blallau has quit IRC | 16:11 | |
sbezverk | kfox1111: what creates this key keystone-fernet-keys? | 16:11 |
sbezverk | your code or it is created by kollakube? | 16:11 |
swinn | am I chasing a ghost? will nova compute running in a container support kvm and honor cpu overcommit (assuming the host supports the flags)? | 16:12 |
kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/helm/microservice/keystone-fernet-setup-job/templates/keystone-fernet-setup-job.yaml#L47-L66 | 16:13 |
*** blallau has joined #openstack-kolla | 16:13 | |
kfox1111 | swinn: should. | 16:13 |
sbezverk | kfox1111: got it | 16:13 |
sbezverk | kfox1111: so I need to add it to my cleanup | 16:13 |
blallau | If some guys have freetime do this review it will be great, thank you, regards https://review.openstack.org/#/c/454167/ | 16:15 |
*** blallau has quit IRC | 16:15 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/455928 | 16:23 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/455929 | 16:23 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/455930 | 16:23 |
*** satyar has quit IRC | 16:24 | |
kfox1111 | so, is infra broken? | 16:25 |
kfox1111 | sbezverk: yeah. | 16:25 |
kfox1111 | maybe need a job that does a curl with a delete instead of post. | 16:26 |
sbezverk | kfox1111: well, it is a corner case.. and adding explicit delete fixed it.. so I am not sure if anything needs to be changed | 16:27 |
kfox1111 | well, depends on what you want users to be able to do. | 16:27 |
kfox1111 | if you want the keystone cleanup chart to do it all, it needs to be a job. | 16:27 |
*** openstack has joined #openstack-kolla | 16:36 | |
*** hachi has quit IRC | 16:37 | |
kfox1111 | sbezverk: k. | 16:37 |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Compress compute kit https://review.openstack.org/456406 | 16:37 |
*** jascott1_ has joined #openstack-kolla | 16:37 | |
*** jrich523_ is now known as jrich523 | 16:40 | |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: OVS Role release note https://review.openstack.org/456702 | 16:41 |
*** haplo37 has joined #openstack-kolla | 16:42 | |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Move ironic config to kolla-k8s https://review.openstack.org/450701 | 16:44 |
*** jaosorior has quit IRC | 16:44 | |
kfox1111 | sbezverk: can you put your +2 back please: https://review.openstack.org/#/c/450701/5 | 16:45 |
sbezverk | kfox1111: done | 16:45 |
kfox1111 | thx | 16:46 |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Finalize move of configs to kolla-k8s https://review.openstack.org/456412 | 16:46 |
openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-kubernetes master: Move ceph configs to kolla-k8s https://review.openstack.org/456705 | 16:46 |
*** jascott1_ has quit IRC | 16:47 | |
inc0 | kfox1111: so after ceph and ironic, we'll see | 16:48 |
inc0 | hopefully we'll be done today | 16:49 |
kfox1111 | sweet. :) | 16:49 |
sdake | inc0 nice dude! | 16:52 |
inc0 | so let's summarize | 16:52 |
*** jascott1_ has joined #openstack-kolla | 16:53 | |
sdake | sbezverk quick question about our ironic implementation - can vms and baremetal be deployed in the same tenant? | 16:53 |
inc0 | we can get rid of secret/configmap generation with this playbook: https://review.openstack.org/#/c/447733/ | 16:53 |
*** unicell has quit IRC | 16:53 | |
inc0 | what else do we use kollakybe tool for? | 16:53 |
kfox1111 | inc0: still -1 on that ps. | 16:53 |
*** jaosorior has joined #openstack-kolla | 16:54 | |
inc0 | kfox1111: we use ansible anyway for now | 16:54 |
inc0 | I'm trying to lower amount of tools involved | 16:54 |
kfox1111 | I'd rather just use kubectl for that. | 16:54 |
jascott1_ | kfox1111 did you get to checkout net=host? | 16:54 |
inc0 | kfox1111: it's about templating | 16:54 |
kfox1111 | jascott1_: yeah. standing it up right now. | 16:54 |
inc0 | kubectl does not template | 16:54 |
jascott1_ | oh cool | 16:54 |
kfox1111 | inc0: there's no templating needed to upload a configmap. | 16:54 |
inc0 | ahh | 16:54 |
sbezverk | sdake: that is what is done currently | 16:55 |
kfox1111 | inc0: see this instead: https://review.openstack.org/#/c/449262/ | 16:55 |
kfox1111 | https://review.openstack.org/#/c/449262/8/tools/setup_gate_iscsi.sh | 16:55 |
kfox1111 | it also partially answers your quiestion. | 16:55 |
kfox1111 | what else needs kollakube. | 16:55 |
kfox1111 | the stuff not moved to kubectl still requires the templating engine in kollakube. | 16:56 |
kfox1111 | that should probably move to helm charts. | 16:56 |
inc0 | -haproxy stuff yes | 16:56 |
kfox1111 | right. | 16:56 |
inc0 | keepalived? | 16:56 |
inc0 | btw..keepalived in k8s? | 16:56 |
sdake | sbezverk i am struggling with the "done currently" part - we use virutal BMC to simulate real hardware, is also a vm created that is not real harddware? | 16:56 |
inc0 | for neutron or sth else? | 16:57 |
kfox1111 | keepalived needs converting. | 16:57 |
kfox1111 | though we never have gate tested it. really should do that. | 16:57 |
kfox1111 | we have a bit of ceph left. | 16:57 |
kfox1111 | openvswitch-set-external-ip needs converting. | 16:57 |
kfox1111 | libvirt secret needs converting. | 16:57 |
kfox1111 | then I think we're in good shape. | 16:57 |
sbezverk | sdake: what I do there is creating 2 normal OpenStack VMs and in addition 1 virsh VM which behaves EXCATLY as baremetal node. | 16:58 |
sdake | cool thanks sbezverk | 16:58 |
kfox1111 | sbezverk: keepalived is needed if you want ha access. k8s has no functionality around keeping a vip alive. | 16:59 |
kfox1111 | sorry, meant inc0 | 16:59 |
inc0 | heh...keepalived in k8s sounds so wrong | 16:59 |
inc0 | let's run pacemaker with k8s! | 17:00 |
kfox1111 | heh. | 17:00 |
kfox1111 | keepalived is much easier. pacemaker's safer. | 17:00 |
*** lamt has quit IRC | 17:01 | |
inc0 | imagine glorious things that will happen when pacemaker starts fencing stuff and k8s starts restarting fence agents due to lost healthchekc | 17:01 |
kfox1111 | I use both in production. | 17:01 |
kfox1111 | its fine if they are net=host and daemonsets. | 17:01 |
kfox1111 | k8s is just a package shipper in that case. | 17:02 |
*** jaosorior has quit IRC | 17:03 | |
*** 21WAAA2JF is now known as EmilienM | 17:05 | |
*** EmilienM has joined #openstack-kolla | 17:05 | |
kfox1111 | jascott1_: does it have a flag yet, or does it always do the net=host? | 17:06 |
jascott1_ | it has a flag | 17:06 |
jascott1_ | do helm init -h | 17:06 |
jascott1_ | but its --net-host=true | 17:06 |
kfox1111 | did a: curl -o /tmp/helm https://raw.githubusercontent.com/jascott1/bins/master/helm/nethost/_dist/linux-amd64/helm | 17:06 |
inc0 | I need mini-pc with Nvidia card | 17:06 |
kfox1111 | /tmp/helm init --net-host=true | 17:06 |
kfox1111 | Error: unknown flag: --net-host | 17:06 |
inc0 | for glory of cuda | 17:07 |
inc0 | any ideas guys? | 17:07 |
jascott1_ | hmm sorry one sec | 17:07 |
jascott1_ | sorry its host-net | 17:07 |
jascott1_ | ugh | 17:07 |
kfox1111 | unkown flag too. | 17:07 |
jascott1_ | idk how that happend, im looking at it | 17:08 |
jascott1_ | I mustve borked the upload let me see | 17:08 |
jascott1_ | sorry man Idk how that happened | 17:11 |
jascott1_ | one min | 17:11 |
kfox1111 | no worries. :) | 17:12 |
*** Mannheim has joined #openstack-kolla | 17:13 | |
*** Manheim has quit IRC | 17:13 | |
kfox1111 | sdake: https://www.cncf.io/event/openstack-north-america-2017/ | 17:16 |
kfox1111 | see the last entry. | 17:16 |
*** jtriley has quit IRC | 17:17 | |
sbezverk | here is to fix our centos gates: https://review.openstack.org/#/c/456710/ | 17:20 |
*** unicell has joined #openstack-kolla | 17:21 | |
kfox1111 | fun. | 17:21 |
kfox1111 | thanks for the link. | 17:21 |
sbezverk | kfox1111: learning every freaking day :) | 17:22 |
kfox1111 | indeed. :) | 17:22 |
*** tonanhngo has joined #openstack-kolla | 17:23 | |
*** sambetts is now known as sambetts|afk | 17:26 | |
*** Pavo has joined #openstack-kolla | 17:27 | |
*** yingjun has quit IRC | 17:28 | |
*** jtriley has joined #openstack-kolla | 17:28 | |
jascott1_ | kfox1111 should be good to go now. | 17:28 |
kfox1111 | k. trying. | 17:28 |
jascott1_ | was doing 'make dist' instead of 'make cross-build' | 17:29 |
kfox1111 | jascott1_: works good. :) | 17:32 |
kfox1111 | thanks. :) | 17:32 |
jascott1_ | cool! | 17:32 |
jascott1_ | i almost know what I am doing | 17:32 |
kfox1111 | no, you know what your doing. :) | 17:33 |
*** lrensing has quit IRC | 17:33 | |
*** gfidente has quit IRC | 17:34 | |
jascott1_ | so one thing, I just adde that param to tiller spec based on flag. Had to add PodSecurityContext{} with HostNetwork property. I have no idea if we need to set anything else in PodSecurityContext | 17:34 |
jascott1_ | i saw some note about if you set it you had to specify ports (which I think we are doing) | 17:35 |
jascott1_ | but not sure if there are any other ramifications. guess we will see :) | 17:35 |
Reepicheep | so is there a general rule with kolla-ansible of when to use "deploy" vs "upgrade" over the life cycle of an instance? | 17:35 |
jemcevoy | inc0: I have hit a kolla-build issue at a customer site who has one of those man-in-the-middle internet proxies. The rpm command inside the container does not trust the https_proxy. How can I work around this? http://paste.openstack.org/show/606477/ | 17:35 |
Reepicheep | jemcevoy: can they exempt you from the SSL proxy? | 17:37 |
kfox1111 | really? never had to add a PodSecurityContext to anything else... interesting. | 17:40 |
inc0 | jemcevoy: normally docker containers uses CA authority from host | 17:41 |
inc0 | do you have CA authority in host as well? | 17:42 |
sean-k-mooney | inc0: do you mind if i rebase https://review.openstack.org/#/c/440834/2 and adapt my ovs-dpdk patches on top now that the openvswitch role is split? | 17:42 |
inc0 | sean-k-mooney: please do | 17:43 |
sean-k-mooney | cool thanks | 17:43 |
inc0 | I'm happy that we got role fixed | 17:43 |
inc0 | in fact | 17:43 |
inc0 | I think this makes this change redundant | 17:43 |
sean-k-mooney | let me check the change the merged | 17:43 |
sean-k-mooney | i think we still need this to be able to skip the role | 17:43 |
inc0 | splitting ovs to separate role | 17:44 |
inc0 | enable_openvswitch: fasle | 17:44 |
inc0 | it's there | 17:44 |
inc0 | or should be | 17:44 |
*** srwilkers has quit IRC | 17:44 | |
inc0 | yup it's there | 17:44 |
sean-k-mooney | oh cool ok ill see if that works with ovs-dpdk so. | 17:44 |
inc0 | in fact sean-k-mooney have you been on meeting today about opnfv? | 17:44 |
sean-k-mooney | no | 17:44 |
inc0 | quick call? | 17:45 |
sean-k-mooney | sure | 17:45 |
*** sayantan_ has quit IRC | 17:47 | |
*** fooliouno has quit IRC | 17:48 | |
*** lrensing has joined #openstack-kolla | 17:48 | |
openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 17:49 |
*** mbruzek has quit IRC | 17:51 | |
*** Pavo has quit IRC | 17:51 | |
jemcevoy | Reepicheep: It is a defense contractor and I'll never get around using the proxy... | 17:53 |
*** Serlex has quit IRC | 17:53 | |
Reepicheep | jemcevoy: yeah. that would probably be wise.. I thought I would just ask the easy one first | 17:55 |
jemcevoy | inc0: The host is setup with the correct ca bundle in /etc/pki/tls/certs/ca-bundle.crt that trusts the proxy... | 17:55 |
Reepicheep | sounds much simpler then baking a CA trust into each of your images | 17:55 |
jemcevoy | Would the /etc/pki/tls/certs/ca-bundle.crt only need to be in a base image? | 17:57 |
*** Pavo has joined #openstack-kolla | 17:58 | |
*** lpetrut has joined #openstack-kolla | 17:59 | |
*** rwallner_ has quit IRC | 18:00 | |
*** rwallner has joined #openstack-kolla | 18:02 | |
*** lamt has joined #openstack-kolla | 18:05 | |
*** srwilkers has joined #openstack-kolla | 18:05 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible stable/ocata: Updated from global requirements https://review.openstack.org/452870 | 18:10 |
*** ksumit has joined #openstack-kolla | 18:15 | |
*** swinn has quit IRC | 18:18 | |
*** Pavo has quit IRC | 18:19 | |
*** Pavo has joined #openstack-kolla | 18:22 | |
*** mbruzek has joined #openstack-kolla | 18:25 | |
*** mbruzek has quit IRC | 18:25 | |
*** mbruzek has joined #openstack-kolla | 18:25 | |
*** lpetrut has quit IRC | 18:26 | |
*** lamt has quit IRC | 18:27 | |
*** Pavo has quit IRC | 18:28 | |
hrw | morning | 18:30 |
hrw | does anyone know why gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv fails over and over again? | 18:30 |
kfox1111 | infra made some changes... | 18:31 |
*** swinn_ has joined #openstack-kolla | 18:31 | |
kfox1111 | been trying to fix kolla-kubernetes gaets. probably the same issue for kolla/kolla-ansible too. | 18:32 |
*** lamt has joined #openstack-kolla | 18:33 | |
*** jrich523_ has joined #openstack-kolla | 18:33 | |
rwallner | question for the channel. What makes an ansible interface active? when: hostvars[inventory_hostname]['ansible_' + api_interface]['active'] in the prechecks, this is failing for me because the API interface is on a `vxlan-2000` interface but some reason is not "active"? | 18:35 |
*** jrich523 has quit IRC | 18:36 | |
openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: move to stretch https://review.openstack.org/434453 | 18:37 |
rwallner | scratch that im realizing that ansible translates ['ansible_' + api_interface] to ansible_vxlan-2000 when there is ansible_vxlan_2000 | 18:39 |
rwallner | :( | 18:39 |
rwallner | looks like a bug | 18:39 |
*** lucasxu has quit IRC | 18:43 | |
* hrw off | 18:46 | |
*** jrich523_ is now known as jrich523 | 18:46 | |
*** signed8b_ is now known as signed8bit_Zzz | 18:56 | |
*** signed8bit_Zzz is now known as signed8b_ | 18:58 | |
*** MasterOfBugs has joined #openstack-kolla | 18:59 | |
inc0 | yeah seems like everything moved to python3.4 | 18:59 |
inc0 | as default | 18:59 |
inc0 | yay | 18:59 |
*** jascott1_ has quit IRC | 18:59 | |
sdake | inc0 it should have been move d to 3.5 by default | 18:59 |
sdake | i suspect 3.4 was a temporary thing that beame permanent :) | 18:59 |
*** swinn_ has quit IRC | 19:00 | |
sbezverk | kfox1111: that infra patch got merged but still epel does not get enabled before bindeps.. | 19:02 |
kfox1111 | sbezverk: yeah. :/ | 19:02 |
kfox1111 | need to ask infra whats going on. | 19:02 |
kfox1111 | maybe they dont support 34 at all anymore. | 19:03 |
kfox1111 | and we just need to ditch it. | 19:03 |
*** manheim has joined #openstack-kolla | 19:06 | |
*** jascott1_ has joined #openstack-kolla | 19:09 | |
*** manheim has quit IRC | 19:11 | |
sbezverk | kfox1111: how can we drop 3.4? | 19:13 |
kfox1111 | I think they moved on to python35? | 19:13 |
kfox1111 | not sure. | 19:13 |
*** manheim has joined #openstack-kolla | 19:19 | |
sbezverk | kfox11111: it seems the change did not get in | 19:20 |
sbezverk | so hopefully when it synchronize it will unblock out gates | 19:21 |
kfox1111 | ah. ok. | 19:22 |
*** swinn has joined #openstack-kolla | 19:22 | |
openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: To Deploy iSCSI Support with Cinder LVM https://review.openstack.org/455032 | 19:23 |
*** erhudy has joined #openstack-kolla | 19:24 | |
openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: To Deploy iSCSI Support with Cinder LVM https://review.openstack.org/455032 | 19:26 |
*** dan__ has joined #openstack-kolla | 19:26 | |
*** manheim_ has joined #openstack-kolla | 19:26 | |
*** manheim has quit IRC | 19:29 | |
*** Mannheim has quit IRC | 19:29 | |
*** al498u_ has quit IRC | 19:32 | |
*** al498u_ has joined #openstack-kolla | 19:33 | |
*** jtriley has quit IRC | 19:47 | |
*** vhosakot has quit IRC | 19:50 | |
*** eanylin has joined #openstack-kolla | 19:51 | |
-openstackstatus- NOTICE: The Gerrit service on http://review.openstack.org is being restarted to address hung remote replication tasks. | 19:51 | |
sdake | kfox1111 we shoudl be on python 3.5 | 19:53 |
kfox1111 | sdake: thats kind of what I thought. we may need to remove python 34 from somewhere then. | 19:53 |
sdake | bindep is where its installed from | 19:53 |
*** Pavo has joined #openstack-kolla | 19:54 | |
sdake | gah who wrote our bindep file :) | 19:55 |
sdake | kfox1111 is the blamee :) | 19:55 |
*** swinn has quit IRC | 19:56 | |
kfox1111 | its been one of those days. sure, why not. | 19:58 |
sdake | epel only has ptyhon34 it apperas | 19:58 |
sdake | kfox1111 how did you come up wtih that bindep.txt file? | 20:00 |
kfox1111 | I wana say, I took the default from infra, and poked in a couple things like jq/crudini at the end. | 20:01 |
*** swinn has joined #openstack-kolla | 20:01 | |
sdake | got it | 20:01 |
sdake | thanks | 20:01 |
openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Test of bindep change to fix up gates https://review.openstack.org/456749 | 20:04 |
sdake | there is a genral community goal of having python35 support in our python code | 20:06 |
*** jascott1 has quit IRC | 20:06 | |
*** jascott1_ has quit IRC | 20:07 | |
sdake | sbezverk this says "merged" https://review.openstack.org/#/c/456710/ | 20:07 |
sdake | that should just work since epel was disabled this morning | 20:07 |
*** jascott1 has joined #openstack-kolla | 20:07 | |
sbezverk | sdake: pabelanger mentioned that the chage I pushed hs not been replcated due to some issues with gerrit | 20:08 |
sbezverk | they just restarted it, so it should be fixed soon.. | 20:08 |
sdake | sbezverk i see - so you ah eto resubmit? | 20:08 |
sdake | ok | 20:08 |
*** g3ek has quit IRC | 20:10 | |
*** manheim_ has quit IRC | 20:11 | |
*** sayantan_ has joined #openstack-kolla | 20:11 | |
*** rwallner has quit IRC | 20:16 | |
*** jtriley has joined #openstack-kolla | 20:17 | |
*** g3ek has joined #openstack-kolla | 20:19 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/455929 | 20:20 |
*** swinn has quit IRC | 20:21 | |
*** fguillot has quit IRC | 20:22 | |
sbezverk | yeeaahh, the change finally kicked in.. gate are working now | 20:23 |
*** rwallner has joined #openstack-kolla | 20:24 | |
*** rwallner_ has joined #openstack-kolla | 20:26 | |
*** jtriley_ has joined #openstack-kolla | 20:26 | |
openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: WIP: Enable Ceph input plugin in Telegraf https://review.openstack.org/455602 | 20:27 |
kfox1111 | sbezverk: nice. | 20:27 |
*** lrensing has quit IRC | 20:28 | |
sbezverk | kfox1111: could you please change compress compute kit to use variable instead of hardcoding 0.6.0-1? | 20:28 |
*** lrensing has joined #openstack-kolla | 20:28 | |
*** rwallner has quit IRC | 20:29 | |
sbezverk | I would do +2 rigth after that.. | 20:29 |
*** jtriley has quit IRC | 20:29 | |
kfox1111 | sbezverk: Like I said, that code's a hack and needs much better cleanup. | 20:29 |
kfox1111 | either we should clean it up proper, or commit the hack. | 20:29 |
*** rwallner_ has quit IRC | 20:29 | |
sbezverk | kfox1111: yeah, but hardcoding version is not a hack.. cause when we switch to 0/7/0 we need need to remember changing it too | 20:30 |
kfox1111 | its a hack. its further code that will need to be removed when fixed for real. | 20:30 |
kfox1111 | proper versioning is a whole nother discussion too. | 20:31 |
*** zhenguo has quit IRC | 20:32 | |
kfox1111 | sbezverk: if you want to grab the ps and run with it, go ahead. | 20:36 |
kfox1111 | fghting other fires right now. :/ | 20:36 |
sbezverk | kfox1111: I do not want to fight. I thought just basic code cleanness would be sufficient atm.. and we can move forward.. | 20:37 |
*** ediardo has joined #openstack-kolla | 20:37 | |
kfox1111 | sbezverk: not a fight. just don't have time to clean up the hack that it is. | 20:38 |
kfox1111 | if you have time/inclenation, go for it. | 20:38 |
kfox1111 | but I do think the proper fix is to get it into the helm_build_* stuff. | 20:38 |
kfox1111 | doing it in build_all is a hack. | 20:38 |
sbezverk | kfox1111: ok, got it.. since sdake was very interested in compute kit he would be interested to clean things up.. | 20:39 |
kfox1111 | ah. thats a good plan too. | 20:39 |
sdake | clean w hich up | 20:39 |
kfox1111 | sdake: https://review.openstack.org/#/c/456406/ | 20:40 |
kfox1111 | its a proof of concept. | 20:40 |
sbezverk | sdake: the solution kfox1111 proposed needs a bit of polishing | 20:40 |
kfox1111 | it really belongs in tools/helm_build_* as common code shared between them. | 20:41 |
kfox1111 | kind of swamped at the moment though. | 20:41 |
sdake | we can sort that out later | 20:41 |
sdake | i am swamped too | 20:41 |
sdake | lets just get it merged and working | 20:41 |
sdake | it doesn't have to be a mona lisa | 20:41 |
sdake | i've got 3 months of work to do in the next 4 weeks | 20:42 |
sdake | i dont have time to gold plate | 20:42 |
*** ksumit has quit IRC | 20:42 | |
sdake | rwellum you up for writing some code | 21:03 |
*** dan__ has quit IRC | 21:11 | |
*** jtriley_ has quit IRC | 21:12 | |
openstackgerrit | Merged openstack/kolla-kubernetes master: Move ironic config to kolla-k8s https://review.openstack.org/450701 | 21:16 |
*** manheim has joined #openstack-kolla | 21:18 | |
*** swinn has joined #openstack-kolla | 21:18 | |
*** lpetrut has joined #openstack-kolla | 21:20 | |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: v4 gates. https://review.openstack.org/454841 | 21:21 |
*** manheim has quit IRC | 21:22 | |
*** manheim has joined #openstack-kolla | 21:22 | |
openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: To Deploy iSCSI Support with Cinder LVM https://review.openstack.org/455032 | 21:23 |
eanylin | sbezverk: Good points raised. Have updated based on your comments. Thanks. | 21:24 |
sbezverk | eanylin: nope, if you can add my comments, would be great. It took me a bit of time to research solution.. | 21:26 |
*** tvignaud has quit IRC | 21:26 | |
eanylin | sbezverk: Yeah, I have attempted to update based on your comments for the latest patch | 21:26 |
eanylin | I think the steps for Cinder VG is already mentioned in https://docs.openstack.org/developer/kolla-ansible/cinder-guide.html | 21:27 |
sbezverk | eanylin: hm, I am not sure if we can assume kolla-kubernetes users will read kolla-ansible doc before attempting kolla-kubernetes installation.. | 21:28 |
eanylin | sbezverk: Agree. That cinder-guide talks about general creation of VG | 21:29 |
eanylin | was thinking of using it or we can replicate the content for volume group creation | 21:29 |
eanylin | I was following that one when I did the deployment | 21:29 |
eanylin | so only the top part is relevant in this case | 21:30 |
sbezverk | eanylin: I think replicating required parts of doc is good and it prevents some users from unpleasant surprises ;) | 21:31 |
eanylin | sbezverk: Okie, let me update | 21:31 |
eanylin | sbezverk: That is true too | 21:32 |
*** tvignaud has joined #openstack-kolla | 21:32 | |
sbezverk | eanylin: please do not forget two include 2 systemctl commands I mentioned ifyou have not done it already | 21:32 |
eanylin | sbezverk: I mentioned in that block | 21:33 |
eanylin | but yeah, will make sure its there | 21:33 |
sbezverk | eanylin: sorry I have not checked ;) | 21:33 |
eanylin | sbezverk: No problem :) | 21:34 |
*** lrensing has quit IRC | 21:34 | |
*** lpetrut has quit IRC | 21:40 | |
sbezverk | eanylin: since we are on this subject, how about adding a small fragment from cloud.yaml with lvm related config? | 21:41 |
sbezverk | eanylin: here is my config. maybe you could integrate it in the doc if it is not already there: http://paste.openstack.org/show/606488/ | 21:43 |
*** signed8b_ is now known as signed8bit_Zzz | 21:43 | |
eanylin | sbezverk: Ok | 21:45 |
eanylin | sbezverk: I believe its the same as the main document | 21:45 |
sbezverk | eanylin: perfect, then | 21:46 |
eanylin | sbezverk: There is a difference in identation for the cinder-volumes IP, under lvm_backends | 21:46 |
eanylin | I can update to reflect that | 21:47 |
sbezverk | eanylin: I can guarantee that tyhe config I pasted is working ;) | 21:47 |
eanylin | sbezverk: Agree :) | 21:47 |
eanylin | sbezverk: Will do the update | 21:48 |
*** lamt has quit IRC | 21:55 | |
*** mbruzek has quit IRC | 21:55 | |
*** erhudy has quit IRC | 22:00 | |
openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: To Deploy iSCSI Support with Cinder LVM https://review.openstack.org/455032 | 22:06 |
openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: To Deploy iSCSI Support with Cinder LVM https://review.openstack.org/455032 | 22:11 |
eanylin | sbezverk: Updated. Take a look when you have some time? Thanks. | 22:15 |
*** esharao has quit IRC | 22:16 | |
sbezverk | eanylin: +2 | 22:17 |
eanylin | thanks :) | 22:17 |
*** tonanhngo has quit IRC | 22:20 | |
inc0 | I'm back guys | 22:24 |
inc0 | so, what's gate status? | 22:24 |
sbezverk | inc0: gates are working.. | 22:26 |
*** srwilkers has quit IRC | 22:26 | |
inc0 | python thing is fixed? | 22:26 |
*** tonanhngo has joined #openstack-kolla | 22:27 | |
sbezverk | inc0: it was not python, it was due to epel being disabled before we could re-enable it the gate script | 22:27 |
inc0 | ok then | 22:27 |
inc0 | but it's fixed right?;) | 22:27 |
sbezverk | inc0: yep | 22:28 |
inc0 | ok so I'm getting back to config migration | 22:28 |
sbezverk | inc0: cool, ironic got merged recently.. | 22:29 |
inc0 | ceph is up | 22:29 |
*** manheim has quit IRC | 22:29 | |
inc0 | after that I'll need to see what's left | 22:29 |
inc0 | I think iscsi/tgtd | 22:29 |
*** tonanhngo has quit IRC | 22:31 | |
*** signed8bit_Zzz is now known as signed8b_ | 22:33 | |
sbezverk | inc0: these should be simple ones | 22:33 |
kfox1111 | yeah. pretty sure iscsi/tgtd is still missing. | 22:33 |
jascott1 | the deployment guide calls for using `git review` but doesnt say its a dep | 22:39 |
jascott1 | should we change it to vanilla git or add something? | 22:39 |
*** ksumit has joined #openstack-kolla | 22:39 | |
*** klindgren_ has joined #openstack-kolla | 22:40 | |
*** Pavo has quit IRC | 22:41 | |
sdake | jascott1 i hope to have that fixed today or tomorrow | 22:42 |
sdake | although that may not be feasible | 22:43 |
jascott1 | cool | 22:43 |
*** klindgren has quit IRC | 22:43 | |
jascott1 | oh | 22:43 |
jascott1 | im converting that guide to playbook | 22:43 |
jascott1 | act now and get multinode for freeeeee | 22:43 |
inc0 | bbiaf, moving home | 22:44 |
sbezverk | sdake: if you rebase placement ps on master I think we can merge it.. currently it is based on your revert ps and we do not need reverting any longer.. | 22:48 |
sdake | sbezverk agreed | 22:49 |
sdake | sbezverk although it doesn'twork now for compute becausei took out the placement api renames in the manifests | 22:49 |
sdake | sbezverk to even test it on master, this needs to merge: https://review.openstack.org/#/c/456406/ | 22:51 |
sdake | sbezverk so - need the revert in the chian for the moment unitl that merges | 22:51 |
kfox1111 | should we just merge compress compute kit as is then? | 22:51 |
sdake | wfm - its hacky but can be refactored easily | 22:52 |
kfox1111 | I'd rather not, but if no one has time to fix it then, at least it fixes the gate. so better then the gate being broken. | 22:52 |
sbezverk | kfox1111: ok, let's do it then :( | 22:52 |
sbezverk | I like even less possibility of even tem reverting.. | 22:53 |
kfox1111 | yeah. me too. | 22:53 |
sdake | jascott1 if your converting that guide that git review operation won't really work for you as it requires gerrit credentials | 22:55 |
jascott1 | right | 22:55 |
sdake | jascott1 i'd recommend using a cherrypick - you might just single out the change from the gate changes | 22:55 |
sdake | and submit it to gerrit for the moment just to help people get moving | 22:55 |
sdake | until I get the gate fixedi n that patch | 22:55 |
sdake | it is very complex to fix the gate as lots of sudos are required | 22:56 |
*** swinn has quit IRC | 22:56 | |
jascott1 | sdake ok thanks | 22:56 |
sbezverk | sdake: which gate you are fixing? just curious.. | 22:57 |
sdake | sbezverk all of them | 22:57 |
sdake | removing ln -s | 22:57 |
openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Placement support in nova service chart https://review.openstack.org/453846 | 22:57 |
sbezverk | sdake: Ahh that fix :) | 22:58 |
sdake | sbezverk : this one: https://review.openstack.org/#/c/439740/ | 22:58 |
kfox1111 | sdake: I recommend just doing the simple symlink for now. | 22:58 |
kfox1111 | its way way easier then all the sudo junk. | 22:58 |
openstackgerrit | Merged openstack/kolla-kubernetes master: Compress compute kit https://review.openstack.org/456406 | 22:58 |
*** swinn has joined #openstack-kolla | 22:59 | |
sbezverk | sdake: I still fail to see why it is so important to change how it is done at the gate.. | 22:59 |
sdake | ok - here is why it is important | 22:59 |
sdake | atm the guide recommends a cherrypick | 22:59 |
sdake | the cherrypick is klunky and unprofesisonal | 22:59 |
sdake | the only way to fix it is to get rid of the symlink | 22:59 |
kfox1111 | agreed. but your cherrypicking to avoid a symlink. | 22:59 |
kfox1111 | the symlink doesn't matter that much. | 22:59 |
kfox1111 | its a temporary thing. | 22:59 |
sbezverk | sdake: guide and the gate serve different purposes | 22:59 |
*** signed8b_ is now known as signed8bit_Zzz | 23:00 | |
sdake | i agree, however, i need a patch in gerrit to actually make the guide work | 23:00 |
sdake | as the code in master is incorrect | 23:00 |
kfox1111 | just put the syhmlink in the dev guide and we can move on to working on elminiating the need of kolla-ansible entirely. | 23:00 |
kfox1111 | then the symlink disapaears. | 23:00 |
sbezverk | sdake: who cares if it works, it looks more like the change for the change | 23:00 |
sbezverk | since you are so busy, why waste cycles on this ?? | 23:00 |
*** MasterOfBugs has quit IRC | 23:00 | |
sdake | ok i'm listenign to alternate approaches | 23:01 |
sbezverk | we do not gain anything at the gate | 23:01 |
sdake | the root issue is kolla res create configmap depends on a file in /usr/share/kolla | 23:01 |
sdake | there is no /usr/share/kolla | 23:01 |
sbezverk | sdake: so do not use in the deployment gate, why touch the gate?!? | 23:01 |
sdake | when the repos split the code was never changed | 23:01 |
sbezverk | deployment guide I meant | 23:02 |
kfox1111 | yeah. just symlink it in like the gate does. | 23:02 |
kfox1111 | we can get rid of it entirely very soon. | 23:02 |
kfox1111 | inc's been working hard on making that a thing. | 23:02 |
kfox1111 | its less then a week out. | 23:02 |
sdake | i understand inc0 is working on it | 23:02 |
sdake | other things need to be odne to make it happen | 23:02 |
sdake | genpwd | 23:02 |
sdake | init-runonce | 23:02 |
sdake | and probably some othe rwork | 23:03 |
sdake | i could work on that | 23:03 |
sdake | and we coudl leave the guide as it is | 23:03 |
kfox1111 | genpwd true. we need to pull that in too. | 23:03 |
kfox1111 | init-runonce, I don't think so? | 23:03 |
sdake | ya that is a simple bootstrap | 23:03 |
kolla-slack | <inc0> It's simple shell | 23:03 |
kfox1111 | if anything init-roneonce should be a container. | 23:03 |
sdake | maybe | 23:03 |
kolla-slack | <inc0> Post deploy | 23:03 |
sdake | we have a working implementation we can just copy from kolla-ansible | 23:04 |
sdake | inc0 post deploy only creates creds it doesn tbootstrap a cloud | 23:04 |
kolla-slack | <inc0> I know what it does, we need it | 23:04 |
kfox1111 | ok. | 23:04 |
kfox1111 | think it would be hard to put in a helm job? | 23:05 |
sdake | init-runonce? | 23:05 |
kfox1111 | eyah. | 23:05 |
sbezverk | sdake: why placement charts changes are back? | 23:05 |
kfox1111 | yeah. | 23:05 |
sdake | not sure sbezverk - perhaps i rebased off the wrong tree | 23:05 |
sdake | 5% ower | 23:05 |
sdake | trying to eat dinner in the hotel restuarant | 23:06 |
sdake | i'll be back to power in about an hour | 23:06 |
sdake | i am entirely not clear on how to convert init-runonce to a helm chart | 23:06 |
sdake | it could be done | 23:06 |
sdake | not sure how long | 23:06 |
kfox1111 | where's the code? | 23:06 |
sdake | https://review.openstack.org/#/c/452367/ | 23:07 |
kfox1111 | tools/init-runonce looks like... | 23:07 |
kfox1111 | yeah... it needs work.... | 23:07 |
kfox1111 | a lot of hardcoded stuff in it. | 23:08 |
sdake | yup its an eval tool not an end user tool | 23:08 |
sdake | we have got a ton of milage with it | 23:08 |
sdake | in kolla-ansible | 23:08 |
kfox1111 | so, thing A, it could just be copied verbatim into kolla-kubernetes it looks like. | 23:08 |
sdake | sbezverk had wanted it integrated in the gate | 23:09 |
sdake | although i dont know if that is wise as it is super hardcoded and really only meant as a reference - not an implementation that someone would se | 23:09 |
sdake | use | 23:09 |
kfox1111 | thing two, with a bit of tweaks, couidl be made a helm chart I think that was much more generic. | 23:09 |
sdake | having peopel figure all that bootstrapping crap out is a pita - hence the evolution of that script | 23:09 |
kfox1111 | if all the values in the begining of the script came from values.yaml. | 23:10 |
kfox1111 | anyway. that lone looks trivaial to pul apart for now. | 23:10 |
kfox1111 | genpasswd's the only other one maybe. | 23:11 |
sbezverk | sdake: why do you want init-runonce to the gate? what is the goal? | 23:11 |
kfox1111 | not sure how hard that wone will be. but shoudln't be too hard? | 23:11 |
kfox1111 | sbezverk: probably good to test what we're shipping. | 23:11 |
kfox1111 | sbezverk: we do almost the same things in gate already. if we could combine/share that code, and make it useful outside the gate too, I'm for it. | 23:12 |
sbezverk | kfox1111: basic tests is not enough??? I think that script does way more than init run once.. | 23:12 |
kfox1111 | sbezverk all its doing really is setting up the public network, some flavors, and loading the cirrus test image. | 23:12 |
kfox1111 | we do a lesser version of that. | 23:12 |
kfox1111 | oh and some quota's. | 23:12 |
kfox1111 | it would add less then a second to the gate time I think. | 23:13 |
kfox1111 | and then we could gate test what we're asking users to run. | 23:13 |
sdake | i dont mind if its gated sbezverk | 23:13 |
sdake | sbezverk i mind the time it takes for a contrib to figure out how to sort out the gating and replacement of the gate with that script | 23:13 |
sbezverk | kfox1111: when I looked at init-runonce, it looked to me a lesser of basic_tests script, where we bring up 2 VMs, create/attach detach volumes | 23:14 |
sdake | sbezverk i can tell you it would tkae me atleast a week of focused time | 23:14 |
sbezverk | and test connectivity between VMs | 23:14 |
kfox1111 | sbezverk: oh. true. it looks to do that doesnt it... | 23:14 |
kfox1111 | well, we leave that part out, or make it conditional? | 23:14 |
sdake | people want to test manually their clouds (based upon experience) but want to bootstrap automatically | 23:14 |
kfox1111 | do everything else? | 23:14 |
sdake | init-runonce is all about evaluation | 23:15 |
sdake | not actual prodcution | 23:15 |
sdake | and not really about gating | 23:16 |
sdake | it speeds up eval time dramatically in kolla-ansible | 23:16 |
sdake | alot of people have no idea how to setup a netowrk in openstack via neutron | 23:16 |
kfox1111 | sdake: I think the issue is, it does two things. | 23:16 |
sdake | or to set quotas | 23:16 |
sdake | etc | 23:16 |
kfox1111 | it provisions an external network, | 23:16 |
kfox1111 | and does a bit of testing. | 23:16 |
sdake | init-runonce does no testing | 23:16 |
*** ksumit has quit IRC | 23:16 | |
kfox1111 | the former is needed by basically everyone, and the gate. | 23:16 |
sbezverk | I must be from a different planet, I do not get why inir-runonce can trigger any changes at the gate :) | 23:17 |
kfox1111 | the latter part, the basic_tests.sh in the gate is far more complete. | 23:17 |
kfox1111 | launching a single vm isn't very useful to test the cloud is sane. | 23:17 |
sdake | nobody really wants to run basic tests, they want to do that manually | 23:17 |
sdake | kfox1111 i agree completely | 23:17 |
kfox1111 | launching a couple, network testing, and passing a volume between them covers almost all the major bugs usually hit in deployment. | 23:17 |
sdake | kfox1111 its a starting point | 23:17 |
kfox1111 | for that part, we probably should just split out basic_tests abit and make it easily used outside the gate. | 23:18 |
sdake | its useful to go with patterns that work :) | 23:18 |
sdake | 1% power and switching over to slack | 23:18 |
kfox1111 | that way, the more useful tests can be used by everyone. | 23:18 |
kfox1111 | anyway. still think we're just a few days or less from elminiating kolla-ansible's dep? | 23:19 |
kfox1111 | so we either do a symlink for now, or just wait till that happens. | 23:19 |
kfox1111 | the bajillion sudo's is a very expensive stopgap that isn't needed imo. | 23:20 |
kolla-slack | <sdake> Or use got fetch | 23:20 |
kolla-slack | <sdake> The sudo is how a proper deploy would happen | 23:21 |
sbezverk | kfox1111: screwing up with the gate for the sake of deployment guide, does not make sense :) | 23:21 |
kolla-slack | <sdake> Anyway jumping around topics | 23:21 |
kfox1111 | sdake: a sudo shouldn't ever be needed. | 23:21 |
*** ksumit has joined #openstack-kolla | 23:21 | |
kfox1111 | should only need sudo for k8s deployment. | 23:21 |
kfox1111 | everything else should be non privileged. | 23:22 |
kfox1111 | config's less then ideal currently, but very limited, to stuff like setting up a symlink. | 23:22 |
kfox1111 | far les ugly then running a tun of code as root. | 23:22 |
kolla-slack | <sdake> Read the deploy guide then repeat that statement | 23:22 |
kfox1111 | sdake: https://review.openstack.org/#/c/439740/39/tests/bin/setup_config.sh | 23:23 |
kfox1111 | it should not take sudo to build a few text files. | 23:23 |
kfox1111 | and using a tool thats been known to change kernel perams if not configured just right. | 23:24 |
kolla-slack | <sdake> I didn't say that patch was done | 23:24 |
kfox1111 | I'm saying, its the wrong aproach. should never have to sudo to generate configs imo. | 23:25 |
kolla-slack | <sdake> Symlinkd are so terrible a deploy solution I struggle to understand why it's debated | 23:25 |
kfox1111 | its less terible, then running kolla-ansible genconfig as root. seriously, its terrifying. | 23:25 |
kfox1111 | I saw it making all sorts of root changes on my system before figuring out why. | 23:25 |
kfox1111 | you have to know to set: set_sysctl: "no" | 23:26 |
sbezverk | sdake: because you express your personal opinion wihtout 1: providing technical reasons behind 2: why gate needs this change? | 23:26 |
kolla-slack | <sdake> Pick on someone else I have had enough of both of you todY | 23:27 |
kfox1111 | I agree the config stuff is less then ideal. until we fix it. I'm just of the opinion of root genconfig with no symlink is far worse then symlink and no root genconfig. | 23:28 |
sbezverk | I can understand when you talk about deployment guide, kind of passing wisdom to masses, but the gate is wroking, and its presence does not impact anything.. | 23:28 |
sbezverk | I mean presence of sym link | 23:28 |
kfox1111 | sbezverk: I'm with sdake on we should share as much as possible between the two, and not ask the users to patch in uncommitted code. | 23:28 |
kfox1111 | I just think lots of sudo's are not healthy to either use case. | 23:29 |
kfox1111 | but I think we're at an impass here. not much value in continuing the conversation. | 23:29 |
kfox1111 | it will resolve it self in a few days and wont be a problem. | 23:29 |
kfox1111 | lets just punt for now. | 23:29 |
sbezverk | :) better is enemy of good | 23:31 |
*** jtriley has joined #openstack-kolla | 23:31 | |
kfox1111 | true. | 23:31 |
kfox1111 | sdake: inc0 how hard will copying genpasswd to kolla-kubernetes be. do you know? | 23:32 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/455930 | 23:35 |
*** lrensing has joined #openstack-kolla | 23:36 | |
SamYaple | kfox1111: does k8s have a single "here be passwords" file? | 23:41 |
*** rwellum has quit IRC | 23:41 | |
SamYaple | the genpasswd thing was mainly built for yaml and variable name parsing | 23:41 |
kfox1111 | SamYaple: yeah. same as kollla-ansible at the moment. | 23:41 |
SamYaple | should be easy then | 23:42 |
SamYaple | just point at file | 23:42 |
kfox1111 | k. | 23:42 |
kfox1111 | thanks. | 23:42 |
SamYaple | if you have changed the variable names you might have a harder time | 23:42 |
SamYaple | it detects uuid vs password vs (something else maybe?) | 23:43 |
SamYaple | its been a while since i poked the code | 23:43 |
kfox1111 | yeah, we didn't change anything. | 23:44 |
*** goldyfruit has quit IRC | 23:44 | |
kfox1111 | just pulling it over verbatium for now to make it require one less dep. | 23:44 |
SamYaple | should be that simple then | 23:44 |
kfox1111 | then will start converting it to helm. | 23:44 |
SamYaple | +2 | 23:44 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!