| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: multinode temp workaround. https://review.openstack.org/517835 | 00:09 |
|---|---|---|
| openstackgerrit | Tin Lam proposed openstack/openstack-helm master: DNM: multinode temp workaround. https://review.openstack.org/517835 | 00:17 |
| *** gouthamr has quit IRC | 00:22 | |
| openstackgerrit | Matt McEuen proposed openstack/openstack-helm master: Docs: true-up to recent changes https://review.openstack.org/517840 | 00:52 |
| openstackgerrit | Matt McEuen proposed openstack/openstack-helm-infra master: Gate: make pod subnet configurable https://review.openstack.org/517841 | 00:52 |
| *** japestinho has joined #openstack-helm | 02:33 | |
| *** lukepatrick has quit IRC | 02:34 | |
| *** felipemonteiro has joined #openstack-helm | 02:38 | |
| *** mengxd has joined #openstack-helm | 02:41 | |
| *** lukepatrick has joined #openstack-helm | 03:02 | |
| *** yamamoto has joined #openstack-helm | 03:03 | |
| *** buster_604 has joined #openstack-helm | 03:08 | |
| *** buster_604 has quit IRC | 03:21 | |
| *** gouthamr has joined #openstack-helm | 03:27 | |
| *** gouthamr_ has joined #openstack-helm | 03:30 | |
| *** felipemonteiro has quit IRC | 03:31 | |
| *** gouthamr has quit IRC | 03:33 | |
| *** mengxd has quit IRC | 03:49 | |
| *** felipemonteiro has joined #openstack-helm | 03:52 | |
| openstackgerrit | Merged openstack/openstack-helm master: Docs: true-up to recent changes https://review.openstack.org/517840 | 03:54 |
| *** lukepatrick has quit IRC | 04:01 | |
| *** lukepatrick has joined #openstack-helm | 04:02 | |
| *** felipemonteiro has quit IRC | 04:14 | |
| *** lukepatrick has quit IRC | 04:24 | |
| *** mengxd has joined #openstack-helm | 04:32 | |
| *** anticw has quit IRC | 04:54 | |
| *** felipemonteiro has joined #openstack-helm | 04:57 | |
| *** felipemonteiro has quit IRC | 05:18 | |
| *** anticw has joined #openstack-helm | 05:20 | |
| *** MarkBaker has joined #openstack-helm | 05:46 | |
| *** MarkBaker has quit IRC | 06:07 | |
| *** yamamoto_ has joined #openstack-helm | 06:09 | |
| *** osh-chatbot has joined #openstack-helm | 06:12 | |
| *** yamamoto has quit IRC | 06:14 | |
| *** osh-chatbot1 has quit IRC | 06:14 | |
| *** openstackgerrit has quit IRC | 06:31 | |
| *** gouthamr_ has quit IRC | 06:44 | |
| *** hachi has joined #openstack-helm | 07:07 | |
| *** openstackgerrit has joined #openstack-helm | 07:11 | |
| *** hachi_ has joined #openstack-helm | 07:11 | |
| *** hachi has quit IRC | 07:15 | |
| *** mengxd has quit IRC | 07:26 | |
| jayahn | portdirect: where are you guys? Not coming Redhat party? | 08:49 |
| mattmceuen | Sorry, I am at the woo thing! I think portdirect may be cramming for his talks tomorrow ;-) | 08:50 |
| *** yamamoto_ has quit IRC | 09:06 | |
| *** openstackgerrit has quit IRC | 09:18 | |
| osh-chatbot | <jay-ahn> File uploaded https://kubernetes.slack.com/files/U20137NTX/F7UTWE5CZ/1509961868373.jpg / https://slack-files.com/T09NY5SBT-F7UTWE5CZ-8fb0cd3cab - <@U2X9XQ1AM> <@U33PPMEBH> | 09:51 |
| osh-chatbot | <jay-ahn> We NEED to go here | 09:51 |
| osh-chatbot | <jay-ahn> @mattmceuen as well | 09:53 |
| *** gouthamr has joined #openstack-helm | 09:56 | |
| *** gouthamr has quit IRC | 10:01 | |
| *** yamamoto has joined #openstack-helm | 10:06 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** yamamoto has joined #openstack-helm | 10:21 | |
| *** yamamoto has quit IRC | 10:29 | |
| *** hachi_ has quit IRC | 10:34 | |
| *** MarkBaker has joined #openstack-helm | 10:37 | |
| *** yamamoto has joined #openstack-helm | 10:40 | |
| *** MarkBaker has quit IRC | 10:58 | |
| *** yamamoto has quit IRC | 11:00 | |
| *** hachi has joined #openstack-helm | 11:03 | |
| *** MarkBaker has joined #openstack-helm | 11:04 | |
| *** MarkBaker has quit IRC | 11:04 | |
| *** yamamoto has joined #openstack-helm | 11:19 | |
| *** yamamoto has quit IRC | 12:17 | |
| osh-chatbot | <kperr> Hi, I am setting up openstack helm on my 3 node cluster using instructions mentioned in http://openstack-helm.readthedocs.io/en/latest/install/multinode.html. Installing ceph helm chart runs without errors but the pods `ceph-mon-*` on my three nodes continously go into crashloopback state. `kubectl describe po` shows `MountVolume.SetUp failed for volume "ceph-mon-keyring" : secrets | 12:33 |
| osh-chatbot | "ceph-client-admin-keyring","ceph-bootstrap-osd-keyring", "ceph-bootstrap-rgw-keyring","ceph-mon-keyring" not found` but `kubectl get secrets -n ceph` shows that these secrets already exist. Does anyone know what could be causing this issue? I am using k8s 1.8.2 version. | 12:33 |
| osh-chatbot | <mariusv> `rm -rf /var/lib/ceph` on all your nodes | 12:40 |
| osh-chatbot | <mariusv> looks like you had a deployment which failed before | 12:40 |
| osh-chatbot | <mariusv> then re-deploy | 12:40 |
| *** jaypipes has joined #openstack-helm | 13:02 | |
| *** julim has quit IRC | 13:09 | |
| *** yamamoto has joined #openstack-helm | 13:18 | |
| *** bryan_att has quit IRC | 13:23 | |
| *** wxy_ has quit IRC | 13:23 | |
| *** hogepodge has quit IRC | 13:23 | |
| *** mattmceuen has quit IRC | 13:23 | |
| *** wxy_ has joined #openstack-helm | 13:24 | |
| *** bryan_att has joined #openstack-helm | 13:24 | |
| *** hogepodge has joined #openstack-helm | 13:24 | |
| *** yamamoto has quit IRC | 13:27 | |
| *** felipemonteiro has joined #openstack-helm | 13:31 | |
| *** felipemonteiro_ has joined #openstack-helm | 13:33 | |
| osh-chatbot | <kperr> Thank you :slightly_smiling_face: In my case it was rm -rf /var/lib/openstack-helm/ceph. | 13:36 |
| osh-chatbot | <kperr> But I noticed that even though all ceph pods are running the logs of `mon` pods show `failed to getpeername (107) Transport endpoint is not connected`. Is this normal? | 13:36 |
| *** felipemonteiro has quit IRC | 13:37 | |
| osh-chatbot | <kperr> I am asking this because maria-db pod is stuck in `containercreating` state with the message `rbd: failed to lock image kubernetes-dynamic-pvc-f7c9b6a4-c2f1-11e7-a19a-46d60d99e853 (maybe locked by other nodes)`. So I am wondering if there is anything wrong with ceph even though all pods are in `running` state. | 13:39 |
| osh-chatbot | <kperr> @arnoud @v1k0d3n specifying `--set volume.enable=false` doesn't seem to solve the problem | 13:41 |
| osh-chatbot | <v1k0d3n> Hey @kperr, how did you install your Ceph charts? Sudo or as a normal user? | 13:43 |
| osh-chatbot | <kperr> root user | 13:43 |
| osh-chatbot | <v1k0d3n> Couple of things I’ve noticed cause a lock, and installing as ceph and root can cause this. | 13:43 |
| osh-chatbot | <v1k0d3n> Sorry...as sudo I mean | 13:43 |
| osh-chatbot | <v1k0d3n> Sometimes when not install as sudo or root, rebooting the host can clear the lock issue. | 13:44 |
| osh-chatbot | <kperr> I see. So do you suggest creating charts as normal user? | 13:44 |
| osh-chatbot | <v1k0d3n> Sorry for the typos...on phone :$ | 13:45 |
| *** felipemonteiro_ has quit IRC | 13:47 | |
| osh-chatbot | <v1k0d3n> I do suggest this yes. I’m still trying to figure this out, but when installing the ceph chart as sudo or root, there’s more than just /var/lib/openstack-helm that needs to be removed. The only way I’ve really cleared this is to back out everything in the /var/lib/docker and /var/lib/kubelet volumes, which suggests a reinstall of these apps as well. | 13:48 |
| osh-chatbot | <v1k0d3n> I found out with my own error using sudo at one point...it sort of accidentally crept into my OSH deployment scripts, and had to learn the hard way. I’d like to dig in and really get down to the specific locked file/volume at some point, because it can be pretty annoying when it comes up. | 13:50 |
| osh-chatbot | <v1k0d3n> As I’m sure you’re experiencing now. | 13:50 |
| osh-chatbot | <kperr> Thank you for sharing your tips :slightly_smiling_face: I will start setting up everything as normal user | 13:51 |
| osh-chatbot | <v1k0d3n> Make sure to clear out /var/lib/docker and reinstall if that’s an option. | 13:52 |
| osh-chatbot | <v1k0d3n> And you’ve already removed /var/lib/openstack-helm...so that’s good... | 13:53 |
| *** krypto has joined #openstack-helm | 13:55 | |
| *** krypto has quit IRC | 13:56 | |
| *** julim has joined #openstack-helm | 14:08 | |
| *** openstackgerrit has joined #openstack-helm | 14:26 | |
| openstackgerrit | Merged openstack/openstack-helm master: Update documentation https://review.openstack.org/517833 | 14:26 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm master: Prometheus Chart https://review.openstack.org/512365 | 14:39 |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: RBAC authorization support https://review.openstack.org/464630 | 14:40 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 14:40 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 15:15 |
| *** mattmceuen has joined #openstack-helm | 15:19 | |
| zla | hello everyone, I have a question, hope someone can point me to the right documentation or give me an idea. I'm trying to figure the openstack-helm's ceph part out. So let's see an example: I have servers with lots and lots of physical disks which would be the ideal ceph-osd node. What is the proper way of getting openstack-helm to use those disks instead of openstack-helm just creates files and using those as a ceph cluster? I tried | 15:28 |
| v1k0d3n | so right now Ceph uses directories. you want to use disks @zla? | 15:29 |
| v1k0d3n | i think there is a patchset that's in-flight, but the work is not completed yet. | 15:29 |
| v1k0d3n | the way i do it in my own lab is format the disks, create a folder, mount it, and then point the chart (using the --set command) to that folder. | 15:30 |
| v1k0d3n | you get about the same thing, but you're obviously not zapping drives etc as you would when offering a new disk to ceph like normal. | 15:31 |
| srwilkers | hey zla. there's currently some work in flight to do exactly that. renmak is currently working on that, but dont see him around right now | 15:33 |
| v1k0d3n | :) | 15:34 |
| srwilkers | hey dude :) | 15:35 |
| *** hachi has quit IRC | 15:43 | |
| zla | oh yes we would like to use disks. do you know where can I find that patch set? I know it's incomplete just to have a look. :) So in your lab you basically prepare the osd-s on the host, mount them to the host and simply point the parent directory of the mounted osd-s in to the chart? | 15:44 |
| v1k0d3n | i do currently & personally @zla, yes. | 15:45 |
| v1k0d3n | just because the work so far with disks has been a bit of WIP...not sure exactly where it's at, and I'm a fan of the M.2 NVMe SSD disks i'm currently using with my OSH rig :) | 15:47 |
| zla | nice that's decent. In my setup I use 16G SAS SSD-s so not as fast as NVMe but does the job on the non container based openstack. I know NVMe is the future tho ;) | 15:48 |
| zla | let me know if you need help in testing the patch because I've got quite large (96 disks so 96osd-s) proposed storage cluster to test it | 15:53 |
| v1k0d3n | the NVMe's I have and are using right now, are still just consumer grade...but the stats are pretty impressive. http://www.samsung.com/semiconductor/minisite/ssd/product/consumer/ssd960.html | 15:55 |
| v1k0d3n | when i bought them, at the time...512GB was around $400... | 15:56 |
| v1k0d3n | now that's dropped quite a bit, and they have up to 2048. | 15:56 |
| v1k0d3n | i was getting something like 5+GB read and over 2.5GB a sec on write. i'm ballparking the numbers though...it's been a while since i tested them. | 15:57 |
| v1k0d3n | let me try to find that patchset zla....one sec. | 15:57 |
| v1k0d3n | https://review.openstack.org/#/c/457754/ | 15:59 |
| zla | oh yes I know those SSD-s, they're really lovely indeed. Slowly NVMe comes to servers as well since they're so much better when you don't have to worry about SAS protocol. thank you for the patch, we'll try it here | 15:59 |
| v1k0d3n | as you can see...there's been a lot of work going on there. hard to say where it's at now without pulling in renmak. maybe reach out to him when he comes online. he's west coast time (normally). PST. | 16:00 |
| zla | I'm in GMT sadly, UK time | 16:02 |
| v1k0d3n | sure and zla, let us know if you have any thoughts or recommendations. we'd welcome any ideas. redhat has really been leading the charge with the ceph chart. it's possible that rootfs could help you too when he's online. | 16:02 |
| v1k0d3n | sadly? lol. | 16:03 |
| zla | sadly because when you're waking up we're going home. :) okay we definitely try it with my colleague and see how it works for us. | 16:03 |
| v1k0d3n | well, if you want to coordinate a time to discuss, i can reach out to renmark and see if we can adjust our schedule for some questions. that work? we only make happy in this channel...no sadness allowed. | 16:04 |
| v1k0d3n | :) | 16:04 |
| zla | fair enough :) alright we try it and see how it works, will let you know as soon as I got something to report | 16:05 |
| v1k0d3n | sounds good man. take care. | 16:05 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 16:23 |
| *** jdandrea has quit IRC | 16:29 | |
| *** jdandrea has joined #openstack-helm | 16:29 | |
| *** lukepatrick has joined #openstack-helm | 16:38 | |
| openstackgerrit | Vlad Naboichenko proposed openstack/openstack-helm master: Define exposed port for openstack services https://review.openstack.org/515958 | 17:19 |
| *** jdandrea has quit IRC | 17:38 | |
| *** jdandrea has joined #openstack-helm | 17:38 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 18:01 |
| *** felipemonteiro has joined #openstack-helm | 18:08 | |
| *** felipemonteiro_ has joined #openstack-helm | 18:10 | |
| *** felipemonteiro has quit IRC | 18:13 | |
| *** hachi has joined #openstack-helm | 18:35 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 18:50 |
| *** felipemonteiro_ has quit IRC | 19:05 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 19:34 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 19:48 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 19:50 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 19:57 |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 20:05 |
| *** hachi has quit IRC | 20:13 | |
| *** lukepatrick has quit IRC | 20:16 | |
| openstackgerrit | Steve Wilkerson proposed openstack/openstack-helm-infra master: WIP - Move prometheus to osh-infra https://review.openstack.org/517645 | 20:29 |
| *** gouthamr has joined #openstack-helm | 20:32 | |
| *** lukepatrick has joined #openstack-helm | 20:52 | |
| *** gouthamr has quit IRC | 21:08 | |
| *** gouthamr has joined #openstack-helm | 21:14 | |
| *** felipemonteiro_ has joined #openstack-helm | 21:28 | |
| osh-chatbot | <renmak> @v1k0d3n @zla We have following patchset which target disk for OSD per POD. In values.yaml, you can see different ways to target disks https://review.openstack.org/#/c/457754 | 21:32 |
| osh-chatbot | <renmak> let me know if you have further question regarding this | 21:33 |
| *** gouthamr has quit IRC | 21:34 | |
| openstackgerrit | Artur Korzeniewski proposed openstack/openstack-helm master: RBAC authorization support https://review.openstack.org/464630 | 21:46 |
| *** felipemonteiro__ has joined #openstack-helm | 21:50 | |
| *** felipemonteiro_ has quit IRC | 21:54 | |
| *** julim has quit IRC | 21:56 | |
| *** gouthamr has joined #openstack-helm | 21:57 | |
| *** gouthamr has quit IRC | 22:00 | |
| *** felipemonteiro__ has quit IRC | 22:13 | |
| *** felipemonteiro__ has joined #openstack-helm | 22:13 | |
| *** felipemonteiro__ has quit IRC | 22:28 | |
| *** gouthamr has joined #openstack-helm | 22:29 | |
| *** felipemonteiro__ has joined #openstack-helm | 22:50 | |
| openstackgerrit | Vlad Naboichenko proposed openstack/openstack-helm master: Define exposed port for openstack services https://review.openstack.org/515958 | 23:40 |
| *** felipemonteiro__ has quit IRC | 23:46 | |
| *** themistymay has joined #openstack-helm | 23:54 | |
| *** themistymay has joined #openstack-helm | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
