Wednesday, 2016-06-15

*** mbound has quit IRC00:00
*** zhiwei has quit IRC00:00
*** prithiv1 has quit IRC00:02
openstackgerritKen Wronkiewicz proposed openstack/kolla-kubernetes: Add documentation for setting up a local SkyDNS + Kube2Sky  https://review.openstack.org/32971700:09
*** ravig has quit IRC00:14
*** sdake has quit IRC00:18
openstackgerritMerged openstack/kolla-kubernetes: Add documentation for setting up a local SkyDNS + Kube2Sky  https://review.openstack.org/32971700:18
*** sacharya_ has quit IRC00:23
*** david-lyle has quit IRC00:32
*** ravig has joined #openstack-kolla00:38
*** d_code has quit IRC00:41
*** jtriley has joined #openstack-kolla00:44
*** diogogmt has quit IRC00:48
*** d_code has joined #openstack-kolla00:54
*** mbound has joined #openstack-kolla01:01
*** mbound has quit IRC01:06
*** zhiwei has joined #openstack-kolla01:15
*** MarMat has quit IRC01:20
*** sdake has joined #openstack-kolla01:28
*** sdake has quit IRC01:32
openstackgerritJeffrey Zhang proposed openstack/kolla: Remove the pull action in the deploy_aio CI  https://review.openstack.org/32973501:50
*** d_code has quit IRC01:51
*** d_code has joined #openstack-kolla01:51
*** diogogmt has joined #openstack-kolla01:52
*** sacharya has joined #openstack-kolla01:55
*** jrist has quit IRC01:56
*** sacharya_ has joined #openstack-kolla01:57
*** sacharya has quit IRC02:00
*** jrist has joined #openstack-kolla02:03
*** coolsvap has joined #openstack-kolla02:18
openstackgerritMerged openstack/kolla: Makes ml2_conf.ini configuration consistent  https://review.openstack.org/32950602:36
*** huikang has quit IRC02:40
openstackgerritMerged openstack/kolla: Load the metadata definitions during Glance bootstrap  https://review.openstack.org/32241502:48
*** yuanying has quit IRC02:50
*** yuanying has joined #openstack-kolla02:53
*** ravig has quit IRC03:10
*** ravig has joined #openstack-kolla03:16
*** fragatina has quit IRC03:22
*** tfukushima has joined #openstack-kolla03:27
*** yuanying_ has joined #openstack-kolla03:48
*** yuanying has quit IRC03:49
*** david-lyle has joined #openstack-kolla04:06
*** ravig has quit IRC04:25
*** ravig has joined #openstack-kolla04:28
*** sacharya_ has quit IRC04:33
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Makes ml2_conf.ini configuration consistent  https://review.openstack.org/32975804:43
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Makes ml2_conf.ini configuration consistent  https://review.openstack.org/32976004:44
*** fragatina has joined #openstack-kolla04:52
*** fragatina has quit IRC04:53
*** fragatina has joined #openstack-kolla04:53
*** salv-orl_ has quit IRC04:54
*** salv-orlando has joined #openstack-kolla04:54
*** salv-orlando has quit IRC05:17
*** salv-orlando has joined #openstack-kolla05:17
*** salv-orl_ has joined #openstack-kolla05:29
*** salv-orlando has quit IRC05:32
*** tfukushima has quit IRC05:40
*** tfukushima has joined #openstack-kolla05:41
*** ravig has quit IRC05:42
*** Jeffrey4l has quit IRC05:49
*** tfukushima has quit IRC05:51
*** Jeffrey4l has joined #openstack-kolla05:54
*** mbound has joined #openstack-kolla05:55
*** coolsvap has quit IRC05:57
*** daneyon_ has joined #openstack-kolla05:57
*** mbound has quit IRC06:00
*** daneyon_ has quit IRC06:02
*** yuanying has joined #openstack-kolla06:03
*** yuanying_ has quit IRC06:05
*** coolsvap has joined #openstack-kolla06:10
*** openstackgerrit has quit IRC06:18
*** openstackgerrit has joined #openstack-kolla06:18
*** yuanying has quit IRC06:21
*** salv-orlando has joined #openstack-kolla06:24
*** yuanying has joined #openstack-kolla06:28
*** salv-orl_ has quit IRC06:28
*** salv-orlando has quit IRC06:29
*** tfukushima has joined #openstack-kolla06:34
*** belmoreira has joined #openstack-kolla06:51
*** athomas has joined #openstack-kolla06:56
*** maestro has joined #openstack-kolla06:59
*** yuanying has quit IRC07:05
*** maestro has left #openstack-kolla07:05
*** yuanying has joined #openstack-kolla07:08
*** matrohon has joined #openstack-kolla07:10
*** shardy has joined #openstack-kolla07:24
*** liyi has left #openstack-kolla07:31
*** openstackgerrit has quit IRC07:33
*** openstackgerrit has joined #openstack-kolla07:33
*** salv-orlando has joined #openstack-kolla07:41
*** shardy has quit IRC07:43
openstackgerritJeffrey Zhang proposed openstack/kolla: Revert "Added precheck to check kernel version 4.2 on Ubuntu 14.04."  https://review.openstack.org/32979807:44
*** b_bezak has joined #openstack-kolla07:46
openstackgerritJeffrey Zhang proposed openstack/kolla: Pin to the Ubuntu 16.04  https://review.openstack.org/32945407:51
openstackgerritJeffrey Zhang proposed openstack/kolla: Revert "Added precheck to check kernel version 4.2 on Ubuntu 14.04."  https://review.openstack.org/32980807:57
openstackgerritJeffrey Zhang proposed openstack/kolla: Revert "Added precheck to check kernel version 4.2 on Ubuntu 14.04."  https://review.openstack.org/32980808:03
openstackgerritJeffrey Zhang proposed openstack/kolla: Remove the pull action in the deploy_aio CI  https://review.openstack.org/32973508:03
openstackgerritJeffrey Zhang proposed openstack/kolla: Pin to the Ubuntu 16.04  https://review.openstack.org/32945408:03
*** tyrola has joined #openstack-kolla08:07
*** tyrola has quit IRC08:07
*** shardy has joined #openstack-kolla08:10
*** dmk0202 has joined #openstack-kolla08:22
coolsvapJeffrey4l, around?08:24
Jeffrey4lcoolsvap, yep08:24
coolsvapJeffrey4l, regarding patch 31724008:25
patchbotcoolsvap: https://review.openstack.org/#/c/317240/ - kolla - Add support for mistral ubuntu binary containers08:25
coolsvapi have not tested debian08:25
coolsvapi can test and add it08:25
coolsvapdo you think it should block the patch?08:25
Jeffrey4lcoolsvap, yes. I will remove the -108:25
Jeffrey4lcoolsvap, btw, we have no ubuntu binary gate now. So have u tested your PS ?08:27
coolsvapJeffrey4l, yes i have tested it when i submitted it i can build it again and verify for you08:28
Jeffrey4lcoolsvap, cool08:29
*** Serlex has joined #openstack-kolla08:39
*** daneyon_ has joined #openstack-kolla08:40
*** daneyon_ has quit IRC08:44
coolsvapJeffrey4l, it builds successfully08:47
Jeffrey4lCool . thanks for test it. I will +workflow08:48
coolsvapJeffrey4l, np08:48
coolsvapJeffrey4l, is it only me or everyone see Merge Conflict for patch 32889108:49
patchbotcoolsvap: https://review.openstack.org/#/c/328891/ - kolla - Remove the duplication of the inventory08:49
coolsvapin the review queue?08:49
Jeffrey4lcoolsvap, where is the merge conflict?08:50
coolsvapif i see it in the review queue for all kolla open patches https://review.openstack.org/#/q/status:open+project:openstack/kolla08:51
coolsvapi see status as merge conflict  even though there is none08:51
*** mbound has joined #openstack-kolla08:52
Jeffrey4lyou are correct. there is merge conflict warning in the PS page.08:52
Jeffrey4lstill can not familiar with the new gerrit web page. :(08:52
Jeffrey4lanyway, \i will solve it.08:52
openstackgerritJeffrey Zhang proposed openstack/kolla: Remove the duplication of the inventory  https://review.openstack.org/32889108:54
openstackgerritMerged openstack/kolla: Add support for mistral ubuntu binary containers  https://review.openstack.org/31724008:58
b_bezakHi09:16
b_bezakI am doing PoC with DVR - http://docs.openstack.org/mitaka/networking-guide/scenario-dvr-ovs.html09:17
b_bezakso I would need couple of more components then kolla provides on compute nodes09:17
b_bezakneutron_metadata_agent09:17
b_bezakneutron_l3_agent09:17
b_bezakhow to achieve that in kolla?09:18
b_bezak(I have already merge controllers with network nodes in multinode topolgy to simplify customization on this part)09:19
*** jmccarthy1 has joined #openstack-kolla09:20
*** jmccarthy1 has quit IRC09:25
*** jmccarthy has quit IRC09:25
*** mikelk has joined #openstack-kolla09:26
*** jmccarthy has joined #openstack-kolla09:26
openstackgerritJeffrey Zhang proposed openstack/kolla: Pin to the Ubuntu 16.04  https://review.openstack.org/32945409:35
*** tfukushima has quit IRC09:37
*** tfukushima has joined #openstack-kolla09:38
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix build for oraclelinux base  https://review.openstack.org/32984409:41
openstackgerritMd Nadeem proposed openstack/kolla: Added unit test for start container  https://review.openstack.org/32984709:44
*** athomas has quit IRC09:45
*** salv-orlando has quit IRC09:47
*** salv-orlando has joined #openstack-kolla09:48
*** athomas has joined #openstack-kolla09:50
*** cinerama has quit IRC09:51
*** cinerama has joined #openstack-kolla09:55
openstackgerritHan Chao proposed openstack/kolla: Typo-fix in kolla docs  https://review.openstack.org/32985410:05
pbourkeb_bezak: kolla provides the components you mentioned10:12
pbourkelook under docker/neutron10:12
b_bezaksure, I have those on network node. However I would need them also on compute nodes10:14
b_bezakhttp://docs.openstack.org/mitaka/networking-guide/_images/scenario-dvr-services.png10:14
*** pbourke has quit IRC10:15
*** pbourke has joined #openstack-kolla10:15
b_bezakI am using stable/mitaka btw10:16
b_bezakso basically, how can I tell deployer to install 2 more components on compute nodes10:17
b_bezak?10:17
b_bezakis there such customization possible?10:18
*** tfukushima has quit IRC10:18
*** daneyon_ has joined #openstack-kolla10:28
*** jmccarthy has left #openstack-kolla10:29
pbourkeb_bezak: i think you would just need to customise your inventory10:32
*** daneyon_ has quit IRC10:32
pbourkefor example to deploy the neutron_l3_agent on compute nodes, add "compute" into that subsection10:33
*** jmccarthy has joined #openstack-kolla10:34
b_bezakok got it. I will check it. thx10:35
openstackgerritMd Nadeem proposed openstack/kolla: Added unit test for start container  https://review.openstack.org/32987110:41
*** salv-orlando has quit IRC10:56
*** salv-orlando has joined #openstack-kolla10:57
pbourkeb_bezak: let us know if you get it working, id be interested in getting DVR documented10:58
b_bezaksure pbourke, I'll let you know11:01
*** dave-mccowan has quit IRC11:04
*** salv-orlando has quit IRC11:06
*** dave-mccowan has joined #openstack-kolla11:20
*** mliima_ has joined #openstack-kolla11:29
*** belmoreira has quit IRC11:33
*** mgoddard has joined #openstack-kolla11:38
*** dave-mccowan has quit IRC11:47
*** rhallisey has joined #openstack-kolla11:57
mliima_morning all12:03
*** mliima_ is now known as mliima12:03
Mech422pbourke: ping ?12:04
pbourkehi Mech42212:04
Mech422hey - been trying to catch up with you :-)12:04
*** mgoddard_ has joined #openstack-kolla12:05
Mech422Did you resolve your stale partition table stuff ?12:05
pbourkenot really :(12:05
pbourkeit seems to me udev is flaky in centos 612:06
Mech422pbourke: Jeffrey4l pointed out instead of using sgdisk like my patches, you could just call 'udevadm trigger'...ahh...12:06
*** jtriley has quit IRC12:06
Mech422pbourke: my patches use sgdisk -i, so they dont rely on udev12:06
Mech422pbourke: might work for you ?12:06
pbourkeim not sure if we're interested in supporting older distros or not12:06
pbourkeMech422: thanks I'll check your patch!12:06
Mech422pbourke: cool - https://review.openstack.org/#/c/32660912:07
*** mgoddard has quit IRC12:08
pbourkein my situation its strange as sgdisk/parted does show the label, its just not available from udev12:08
pbourkeso Im not sure where the former tools are reading it from12:08
Mech422pbourke: no - sgdisk reads directly from device12:09
Mech422pbourke:  and udev doesn't update the partition name until 'udevadm trigger' is called12:10
Mech422pbourke: Jeffrey4l figured out the udev parts...12:10
Mech422pbourke: so we can either add 'udevadm trigger' calls after re-writing the partition tables12:10
Mech422pbourke: or we can just parse the output from sgdisk12:11
*** ravig has joined #openstack-kolla12:11
*** dwalsh has joined #openstack-kolla12:12
*** berendt has joined #openstack-kolla12:14
*** mbound has quit IRC12:15
*** mbound has joined #openstack-kolla12:15
*** daneyon_ has joined #openstack-kolla12:16
*** prithiv has joined #openstack-kolla12:16
Mech422pbourke: It seems ashame to only use half the capability of Sam's work on ceph stuff - especiallly when it only takes a 20 line patch to fix it...12:19
Mech422pbourke: and I've had a few more people asking about running ceph on partitiosn lately - including sean-k-mooney12:19
*** daneyon_ has quit IRC12:21
*** jtriley has joined #openstack-kolla12:28
*** mgoddard_ has quit IRC12:30
*** mgoddard has joined #openstack-kolla12:30
*** jtriley has quit IRC12:33
*** mbound has quit IRC12:40
*** salv-orlando has joined #openstack-kolla12:56
*** salv-orlando has quit IRC12:56
*** salv-orlando has joined #openstack-kolla12:56
*** prithiv has quit IRC12:57
*** stvnoyes has quit IRC13:00
*** stvnoyes has joined #openstack-kolla13:01
*** inc0 has joined #openstack-kolla13:03
inc0good morning13:03
*** ccesario has quit IRC13:06
Mech422inc0: Morning :-)13:09
Mech422inc0: just made coffee - want some ? :-)13:09
inc0I'm a tea drinker13:09
inc0but tea is lousy in US :(13:09
Mech422heh - my mom used to have relatives send us tea from england13:10
Mech422nothing fancy - just tetley's I think - but she like it a lot better13:10
coolsvapinc0, i am with you13:11
pbourkeMech422: well, recall our problems are slightly different13:11
pbourkeMech422: just back from lunch I need to review your patch13:11
inc0tetley is kinda ok, as for normal bulk goes13:11
Mech422pbourke: I'd appreciate it :-)  I'd like to close it out one way or the other13:11
inc0btw gates are broken13:12
inc0I think13:12
Mech422inc0: yeah - it's not 'tea snob' tea - but I guess its decent13:12
coolsvapyou guys will not like any tea if you drink one made by me :) can guarantee that13:12
inc0I like indian chai masala...super sweet too13:13
inc0is that what you mean?13:13
Mech422pbourke: Jeffrey4l's approach is definately 'cleaner' - only 2 lines, but not sure if we can trust udev and not sure we can run udevadm trigger inside container?13:13
Mech422pbourke: my way is more 'brute force' - but only requires you can run sgdisk (sudo)13:13
coolsvapinc0, tea = chai (in hindi name for tea) i make my own masala each time13:14
Jeffrey4lin fact, i am not like the udev trigger too. use a online disk is a bad idea.13:14
Mech422Oh! I had indian tea once at my neighbors - really good - some spice in it (cinnimon?)13:15
inc0I have some masala at home and I drink chai masala from time to time but I bet it's lousy comparing to what you're doing13:15
inc0bring some to one of summits coolsvap ;) we'll have chai masala;)13:15
*** jtriley has joined #openstack-kolla13:15
Jeffrey4lMech422, pbourke ^13:16
inc0Mech422, masala is mixture of several spices13:16
Mech422Jeffrey4l: Personally, I don't think we should be forcing that decision on people?13:16
coolsvapinc0, sure thing Mech422 no its not cinnimon13:16
Mech422inc0: oh? its very tasty...I've only had british and green tea before13:16
inc0I love japanese teas13:16
inc0good sencha is a treasure13:16
Mech422Jeffrey4l: its the old 'policy vs ...' arguement13:17
Mech422Jeffrey4l: should we be making policy decisions for users ?13:17
inc0Mech422, as for udev, we need to figure out how to pass config to ceph13:17
inc0we created the label idea, and it turns out to be hard to extract13:17
inc0we used udev only because it seemed to be only way available to python13:17
inc0that didn't require stdout scrapping13:18
Mech422inc0: hmm - not really 'hard' - just 'distasteful' to some people...13:18
inc0we did stdout scrapping before, it was nightmare to debug/manage errors13:18
Mech422inc0: I send you a link to a code snippet that read teh device raw in python...13:18
Mech422inc0: to be honest - it seemed more work then benefit13:18
Jeffrey4lSo if you want to add this kind feature, I am OK. Just point this: use online is bad and danger.13:18
*** ccesario has joined #openstack-kolla13:18
inc0it's not about onliune13:19
inc0it's about lack of tools13:19
Mech422Jeffrey4l: I'm NOT adding it - I told you many times, Sam had it in the original code - I just made it work13:19
inc0Mech422, you're referring to partitions?13:19
inc0for osd?13:19
Mech422inc0: yeah13:19
inc0yeah, we wanted to make it work for quite some time13:20
Mech422inc0: Sam had the code path in the original - he just didn't realize that 'partprobe' doesn't refresh the partition names13:20
Jeffrey4lMech422, where is it? the partprobe?13:20
Mech422Jeffrey4l: they whole block of code that deals with using partitions ?13:20
Mech422Jeffrey4l: I didn't add any of that13:20
*** diogogmt has quit IRC13:21
Jeffrey4linc0, it's not about onliune? what's this?  the only one case I think which need call `udevadm trigger` is that when use make label to online disk.13:21
Jeffrey4lMech422, current implement is OK. at last it works. We are talking about you PS, right?13:22
Mech422Jeffrey4l: the current implementation has broken support for partitions - broken, but existing13:22
Mech422Jeffrey4l: my patch is ONLY to fix that lil bit thats broken - its only like 20 lines :-P13:23
Jeffrey4lMech422, how to reproduce it?13:23
Mech422I added no new capabilities and changed no policies13:23
Jeffrey4ldeploy -> clean -> re-deploy?13:23
Mech422Jeffrey4l: yeah - try using partitions on a device thats in use13:23
Jeffrey4lso you are using a oneline drive, right?13:24
Jeffrey4lonline13:24
Mech422Jeffrey4l: If the kernel whines 'using old partition table' - your're hosed13:24
Mech422yes13:24
Jeffrey4lso, current issue is: kolla can not handle the online disk right?13:24
Jeffrey4lmy point is: using online disk is bad and danger.13:25
Mech422err...kolla ceph configuration can't handle it - other then that, it works fine...13:25
Mech422Jeffrey4l: my point is, thats an operator decision...13:25
Jeffrey4lbut not kolla.13:25
Mech422Jeffrey4l: for instance, lots of places put '/' on a raid-1 .... so its possible ALL disks in the system are 'in-use'13:26
Mech422even if the drives are 90% empty13:26
inc0Jeffrey4l, we would need partition-osd as well for gates for ceph13:27
Jeffrey4lat least, the right direct is: deploy -> clean (fix this issue here, umount the ceph-osd mount and zap the disk osd used ) -> re-deploy13:27
inc0well, you can clean up partition just the same13:27
inc0in fact redeploying ceph on it will do that13:28
Mech422Jeffrey4l: I can re-deploy without zapping whole disk ? I just reset the partition names13:28
inc0ceph deals with it13:28
Jeffrey4linc0, what is  partition-osd? a tool?13:28
inc0so for ceph you can put osd on disk as we do now, partition or even a file13:29
inc0we support only whole disk13:29
inc0as for now13:29
*** sdake has joined #openstack-kolla13:29
Jeffrey4labout the zapping whole disk: once i haven't zap the disk, the ceph read the old data again when re-deploy.13:30
Jeffrey4linc0, no. you are wrong at this point. kolla support partition now.13:30
inc0it does not13:30
Jeffrey4lI used this.13:30
Mech422inc0: right - but there is already broken support for 2 of the 3 methods...13:30
Jeffrey4lwait 1sec13:30
inc0broken support13:30
Mech422inc0: I supports partitions now - but only if not in use13:30
Mech422s/I/It/13:31
inc0hmm13:31
*** diogogmt has joined #openstack-kolla13:31
Mech422inc0: for that matter - I don't think the docs say anywhere you need a whole device free if you want to use ceph13:31
*** ravig has quit IRC13:31
inc0ok, I might be mistaken then, I'm pretty sure docs tells how to use whole disk13:31
Jeffrey4linc0, it is undocumented :D13:31
inc0but it does write partition table13:31
Mech422inc0: yesh - it updates the NAMES only of partitions13:32
Mech422inc0: NO geometery changes13:32
Mech422inc0: Unless its doing a whole disk 'zap disk'13:32
Jeffrey4lMech422, inc0 check this http://paste.openstack.org/show/516254/ it is one of my env. i split this disk into multi osd13:32
Jeffrey4lso what the issue really is?13:34
Mech422Jeffrey4l: yeah - it works currently, when the drives are not in use13:34
Mech422if you don't know the problem, why are you agruing about the solution ?13:34
Jeffrey4lI know that.13:34
Jeffrey4lwait for a moment.13:35
*** coolsvap has quit IRC13:39
Mech422If kolla is going to be used by 'enterprises' - it will be extremely limiting to assume 'greenfield' setup...13:39
Mech422so places won't be able to dedicate DHCP to ironic13:40
Mech422some places will want to use existing MySQL instances13:40
Mech422some places will want to use calico for networking13:40
Mech422and some places may have disk setups where disks are (partitially) in use13:40
Mech422I really don't see a difference between any of those situations?13:40
Mech422why just say "you can't do that" for one of them ?13:41
*** mbound has joined #openstack-kolla13:41
inc0guys I'll need to skip todays meeting13:41
*** sdake_ has joined #openstack-kolla13:42
*** esharao__ has joined #openstack-kolla13:43
Mech422inc0: oh - in other news, I've been unable to reproduce that 'heisenbug' :-P13:45
esharao__Good morning all..13:45
*** sdake has quit IRC13:45
esharao__ running into the following issue - failed: [localhost] (item=haproxy) => {"failed": true, "item": "haproxy", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute u'ansible_eth1'"}13:45
inc0lol13:45
Mech422inc0: I'll keep an eye out for it, and just not rebuild the cluster if I hit it again13:45
*** v1k0d3n has quit IRC13:45
esharao__is there a way to fix it..13:45
Mech422esharao__: Morning :-)13:45
inc0esharao__, confirm that eth0 is up and have IP13:45
inc0etg113:46
inc0eth1*13:46
*** mbound has quit IRC13:46
esharao__i dont see eth1 at all..13:46
inc0well then...13:46
inc0;)13:46
esharao__it has centos13:46
inc0you need to change api_interface in globals.yml13:46
esharao__ahh ok..13:47
esharao__i see enp0s313:47
esharao__ok.. let me change that..13:47
*** sacharya has joined #openstack-kolla13:49
Mech422inc0: we could side step the partition name stuff and just throw a dict in globals that has the ceph data/journal mappings ?13:49
Mech422inc0: have to figure out how to make that persistent across reboots - but should be doable13:50
*** ccesario has quit IRC13:50
inc0well, yes and no13:51
inc0it will be bitch to make for larger deployments13:51
inc0but I was thinking of sth like13:51
inc0in inventory:13:51
inc0192.168.0.[1:100] ceph_osd_mapping=/dev/vda,/dev/vdb3,/var/lib/my_ceph_files/my_osd113:52
inc0but I'd love to keep backwards compatibility for 1 release13:52
inc0should be easy enough13:52
Mech422inc0: that works too ... the container bootstrap and startups would have to change13:53
inc0not that much13:53
*** diogogmt has quit IRC13:53
Mech422inc0: right now, it uses the partition name to decide whether to bootstrap or not13:53
inc0since looking for ceph disks is a task13:53
*** sacharya_ has joined #openstack-kolla13:53
inc0you can put a when statement13:53
inc0when: not ceph_osd_mapping13:53
Mech422inc0: but it still needs to know if the cluster is bootstrapping ?13:54
inc0https://github.com/openstack/kolla/blob/master/ansible/roles/ceph/tasks/bootstrap_osds.yml#L2-L8 we have this now13:55
Mech422inc0: anyway - we could re-jigger it around like that - would that be more acceptable ?  looking...13:55
inc0what you can do is to add when: not set ceph_osd_mapping13:55
inc0thing is, we need 1 release of deprecation13:56
Mech422inc0: but you still need to bootstrap once if ceph_osd_mapping is set?13:56
Mech422inc0: yeah - that makes sense13:56
*** sacharya has quit IRC13:56
inc0ahh I see what you mean13:56
Mech422inc0: so do nothing, and spend the next release finding a 'cleaner' solution ?13:56
inc0we'll need to put some lock or sth so it won't re-bootstrap13:56
inc0no, we need cleaner solution by N and support both for N13:57
inc0then we remove old on13:57
inc0one13:57
Mech422inc0: oh...hmm...13:57
inc0people already build CI stuff and such13:57
inc0we can't break it13:57
Mech422inc0: supporting both a clean AND a crufty way at once will be 'fun'...13:57
inc0without giving them time to migrate logic to new one13:57
inc0not that hard13:58
inc0again, at the end we need to fill osds_bootstrap variable13:58
inc0however we do it, it's up to us13:58
inc0in the end*13:58
*** sacharya_ has quit IRC13:59
Mech422inc0: yeah - cut over to 'clean' logic and just support 'clean' and 'crufty' for populating config data13:59
inc0yeah13:59
Mech422inc0: then we can just remove the 'population by cruft' when its no longer supported13:59
*** sacharya has joined #openstack-kolla13:59
inc0and put a bit red warning13:59
inc0during N13:59
inc0THIS STUFF WILL DISAPEAR WITH O14:00
inc0however hard part is to find new clean way14:00
Mech422inc0: its basically just figuring out a good way to build a partition map...14:01
inc0and making sure that it won't get re-bootstrapped at randon14:01
inc0m14:01
Mech422inc0: ceph doesn't really have a concept of 'whole disk' - it delegates all the file I/O stuff to local filesystem14:01
inc0yeah I know14:01
Jeffrey4lMech422, could u explain that how to define ` the dirver are  use? `14:01
Mech422inc0: Are file backed stores out of scope ?14:02
Mech422Jeffrey4l: sorry - I don't understand ?14:02
inc0Mech422, if we can make it cleanly and elegantly, I'14:02
inc0I'd love to have it14:02
*** ccesario has joined #openstack-kolla14:02
inc0would make PoC stuff much easier14:02
Jeffrey4lexplain your issue, more.14:02
inc0Jeffrey4l, issue is, currently we need whole disk, if we put multiple osd on it, cool14:03
inc0but how would you add osd to a file?14:03
Mech422inc0: yeah - be nice for small AIO installs for people to test with14:03
inc0or partition on disk that is already in use?14:03
Jeffrey4linc0, it work. I paste you my env.14:03
inc0you have 3TB raid for whole machine, you use 100Gig of it for base operating system14:03
inc0Jeffrey4l, can you do this with other stuff on same disk as well?14:04
Jeffrey4li have multi osd in one disk, which is deploy by kolla.14:04
*** diogogmt has joined #openstack-kolla14:04
Jeffrey4lwhat's mean by that? inc014:04
b_bezakpbourke: deployment went fine - extra components on compute were deployed, after added lines to inventory14:05
b_bezak[neutron-l3-agent:children]14:05
b_bezakneutron14:05
b_bezakcompute14:05
b_bezak[neutron-metadata-agent:children]14:05
b_bezakneutron14:05
b_bezakcompute14:05
inc0Jeffrey4l, I mean have /home/ubuntu at same disk14:05
Mech422inc0: for files, we'd just need a 'tag' for the osd: 'disk','file' or 'partition' ? we could then key the bootstrap logic by that14:05
inc0or /root14:05
Jeffrey4lb_bezak, please use a past file.14:05
b_bezaksorry14:05
*** dwalsh has quit IRC14:05
Jeffrey4lfile/service14:05
inc0b_bezak, you're deploying l3 agents on compute?14:05
b_bezakyes14:06
b_bezakfor DVR14:06
pbourkeb_bezak: ok let me know how you get on for dvr itself14:06
pbourkeplease :)14:06
inc0ahh right14:06
Jeffrey4linc0, Mech422 you two are not saying the same thing. :(14:06
inc0b_bezak, so inventory is meant to be changed14:06
b_bezaksure, I need to put configs to /etc/kolla/config on deployer and reconfigure first ;)14:06
inc0Jeffrey4l, I kinda agree with him;)14:06
inc0so reason I agree is for example gates14:06
inc0we have only one disk on gate14:07
inc0with current arch we can't gate on ceph14:07
Jeffrey4luse file backend for ceph?14:07
Mech422Jeffrey4l: I thought we were - I was just saying '/' inc0 is saying '/home/foo' ?14:07
Mech422Jeffrey4l: we don't currently support file backing for ceph ?14:07
*** MarMat has joined #openstack-kolla14:08
inc0another way would be to add a partition in setup_gate.sh14:08
inc0and use it for osd14:08
inc0only thing is, this disk is already in use14:08
inc0(root fs)14:08
Jeffrey4li think we are talking two things:  1. the `udevadm trigger`, which solve the online disk issue     2. support file backend storage( without a whole disk) will good for quick test/ci14:09
Jeffrey4lright?14:09
Mech422Jeffrey4l: yeah - file backend is a 'new feature'...14:09
Jeffrey4li have no idea for the file backend. I want to talk about the issue (1)14:10
Mech422'udevadm trigger' / 'sgdisk -i' is just to make existing feature work more reliabily14:10
inc0will udev trigger allow us to use partition on disk already in use?14:10
Mech422inc0: it appears too - though I'm a lil distrustful14:10
inc0Mech422, Jeffrey4l can we test it?;)14:10
inc0I honestly don't know14:11
Jeffrey4lfirst of all, what mean by disk already in use. My understand is: you have a sda with 100G, first 25G is used for OS(/), others is not used.14:11
Jeffrey4lright14:11
Mech422inc0: we did some 'one off' tests an it worked a treat - I'm sorta concerned if it works from inside kolla-toolbox14:11
Mech422inc0: I'm also a bit leary of being dependent on udev behavior...but thats just me14:12
Jeffrey4lOr another explain: a disk/partition is mounted.14:12
Mech422Jeffrey4l: yes or first 25G is /home/steve14:12
Jeffrey4lthen it works.14:12
Mech422Jeffrey4l: 'it works' ? you mean currently, or if you add udevadm trigger ?14:13
sean-k-mooneyinc0:  could we create a loop back device from a file and create a partition to test it in the gate?14:13
Jeffrey4lother 75G is not mount. then you can change the label. and the /proc is updated without a `udevadm trigger`14:13
inc0sean-k-mooney, I played with loopback, not something easily done14:13
Jeffrey4lMech422, the /dev/disk/by-partlabel is updated too.14:14
Mech422Jeffrey4l: Umm - it didn't work like that for me ....14:14
Jeffrey4lMech422, I will show you my test.14:14
sean-k-mooneyinc0: yes that is true. its also a little tempermental at times14:14
inc0I'd like end solution14:15
Mech422inc0: whenever I have to work with loop devices, I let kpartx do the heavy lifting :-P14:15
inc0there are use cases for that on prod too14:15
inc0so let's just make it work14:15
pbourkebeen frantically trying to catch up on this conversation14:15
*** diogogmt has quit IRC14:15
pbourkevery relevant to me at the moment14:15
inc0pbourke, we're discussing ceph bootstrapping now14:15
pbourkeas our OVM systems do *not* work with udev14:15
Mech422inc0: the actual calls to the ceph setup functions are all very similiar - its just a matter of massaging the inputs a bit to support different backing stores14:16
pbourkeim doing something similar to what you guys mentioned as a workaround14:16
inc0yeah I know14:16
pbourkea var containing the osd mapping14:16
sean-k-mooneyin general i think it would be nice to support disk/partition/file and folder backing in kollas ceph code but i think full disk and partition are going to be the most popular outside of dev14:17
inc0partitions on existing disk are quite popular14:17
Mech422sean-k-mooney: yeah - file backing would suck for prod. - but its awsome for say AIO testing setups...14:18
DavieyFor lab work, I can imagine loopback files is easier to setup?14:18
inc0files not-so but files are great for quick poc/dev/gate14:18
inc0and since ceph supports it no problem, we should too14:18
inc0so ceph-ansible uses deterministic conf14:18
sean-k-mooneyMech422: yes i use a backing file for my cache layerer as i share the ssd with the host os but partition would be better14:18
pbourkei have code to manually supply the osd tasks with what find_disks usually outputs14:18
inc0we could use variable in inv if we would somehow ensure that it won't re-bootsrap14:19
pbourkethis could be used as a bridge to something better14:19
inc0pbourke, yeah, we'd need to tweak up extended_start too, but that shouldn't be hard14:19
pbourkewhat stops it from re-bootstrapping now out of interest14:19
Jeffrey4lMech422, check this.14:20
Jeffrey4lhttp://paste.openstack.org/show/516267/14:20
Mech422inc0: in the whole-other-can-of-worms dept. - Ceph already has a nice ceph-admin setup meant to run from a deployment host - we could use ansible to drive that...14:20
inc0pbourke, you do want to have your data present after rerunning deploy that doesn't change anything;)14:20
Mech422inc0: which gives operators a nice 'standard' ceph setup when they have to do things like replace a disk14:20
inc0yeah, I like that14:21
pbourkeinc0: i know, im just wondering how it does that14:21
Mech422I think Daviey asked me about replacing a disk with kolla - and I had nothing but a stupid look to give him :-)14:21
pbourkewhat check do we do to stop this14:21
Mech422pbourke: partition name14:21
sean-k-mooneyMech422: is the ceph-admin related to the ceph deploy tool?14:21
Mech422pbourke: on bootstrap the partitions are named BOOSTRAP14:21
Mech422sean-k-mooney: err..yeah - my bad ceph deploy14:21
Jeffrey4lin AIO env, how can we start multi ceph-mon instance. IIUC, ceph not work when there is only one ceph-mon14:22
sean-k-mooneyok ya i used that to install ceph when i hit issues with kolla 6 months ago14:22
sean-k-mooneyit was a little buggy when it came to custimistation14:22
sean-k-mooneyi could not get cepth-deploy to work with brtfs or external journals correctly14:23
Mech422Jeffrey4l: there's hacks to get it to run with 1 mon14:23
Mech422Jeffrey4l: tweaks to the config14:23
*** jrist has quit IRC14:23
pbourkeMech422: gotcha14:23
Jeffrey4lMech422, why kolla need support that?14:23
pbourkefor AIO14:23
Mech422sean-k-mooney: really? I have a script that provisions an entire cluster with external journals...don't think it does file backed though14:24
Jeffrey4lpbourke, to solve ?14:24
Jeffrey4lthe CI?14:24
inc0Jeffrey4l, I think this got some love in Hammer/Infernalis14:24
pbourkeall-in-one setups to solve dev setups and yes CI14:24
Jeffrey4lso only the CI?14:24
inc0Jeffrey4l, and PoC14:25
inc0\and dev14:25
inc0these are important use cases too14:25
sean-k-mooneyMech422: it is ment to be supported but with a fresh centos 7.2 install it was not happy.  its probly a minor misconfig on my part14:25
Mech422inc0: I think its worth thinking about replacing/growing/shrinking cluster in prod too..14:25
pbourkeMech422: that part is pretty easy no?14:25
Mech422sean-k-mooney: oh maybe - I use ubuntu and the ceph repos14:25
pbourkeMech422: just add/remove from your inventory14:25
Jeffrey4lin the most env/dev/PoC, it is very easy to plug a new disk.14:25
Mech422pbourke: I have NO idea how you 'replace' a ceph disk in kolla ? re-deploy ?14:26
Mech422what happens to existing data ?14:26
Mech422you need to 'bootstrap' the replacement disk, but NOT the rest of the cluster14:26
pbourkeMech422: its replicated across other osds afaik14:26
pbourkeso you just label the new disk and it will bootstrap it14:27
Mech422pbourke: right - but how do you bootstrap the new drie without hosing the cluster ?14:27
pbourkeand add it to the cluster14:27
sean-k-mooneyMech422: isnte the bootstrap lable removed after its successfully setup14:27
Mech422yeah - but we're talking about removing the partition labels14:27
pbourketrue we need another way to mark a node for bootstrapping14:27
inc0Mech422, bootstrap osd should be idempotent now14:28
Mech422with the existing stuff - I think it _might_ work - but I'd need to test it first...14:28
pbourkewonder can we write some code to read the gpt table directly14:28
pbourkestill use labels, just scrap udev14:28
Mech422pbourke: I pasted a link to some ceph code to do that...14:28
Jeffrey4lpbourke, why?14:29
Mech422pbourke: it was like 500 lines of parsing HEADER versions and crap - seemed like way too much trouble for the gain14:29
pbourkeJeffrey4l: udev is unreliable14:29
Jeffrey4lpbourke, bug?14:29
pbourkeJeffrey4l: labels flat out dont work in centos 614:29
Jeffrey4lwhy not try centos 7....14:29
Mech422pbourke: if we want to scrap udev - I really think grabbing sgdisk output is pretty straight forward14:29
pbourkeJeffrey4l: because enterprise14:29
Mech422pbourke: that puts the onus of keeping up to date with gpt specs on sgdisk maintainer and not us :-P14:30
Jeffrey4lok..14:30
pbourkeMech422: true, inc0 has reservations about screen scrapping though14:30
*** vhosakot has joined #openstack-kolla14:31
pbourkewhich im sure are very valid14:31
Mech422pbourke: I agree...normally, I'd hate it - but its 1 line scrape from pretty nice output ..14:31
pbourkeyeah its not like we dont have sed/awk elsewhere in our code14:31
Mech422pbourke: we already do more complicated stuff with docker info output14:31
pbourkebingo14:32
Mech422pbourke: docker version check feeds `docker info` | to_yml14:32
Mech422pbourke: thats just waiting to bite us :-P14:32
Mech422pbourke: sgdisk -i is nicely formatted, 1 line per field, no fancy delimiters14:33
*** v1k0d3n has joined #openstack-kolla14:33
pbourkeI agree, im just waiting for inc0 to get back from his tea to say no ;)14:33
Mech422pbourke: But I do agree - udevadm trigger would be 100x more elegant if we could trust it14:34
pbourkeudevadm trigger doesn't work for older versions fyi. the /dev/disk/by-label structure is simply missing14:34
pbourkeanother option14:35
pbourkeuse the disk_info variable as above, write out a sentinal file to mark bootstrapping14:35
pbourkepretty sure we do that elsewhere also14:35
*** severion has joined #openstack-kolla14:35
Mech422spacex landing in 10 minutes14:36
*** jrist has joined #openstack-kolla14:37
Mech422pbourke: yeah - we're gonna need a sentinal of some sort14:37
openstackgerritJeffrey Zhang proposed openstack/kolla: Pin to the Ubuntu 16.04  https://review.openstack.org/32945414:37
pbourkewell, not if we use labels14:37
pbourkealong with sgdisk parsing14:37
Mech422pbourke: I'm still a bit concerned about drive replace/expand cluster/shrink cluster14:37
*** v1k0d3n has quit IRC14:37
pbourkeexplain again? I dont see the issue with marking a new disk for bootstrapping14:37
Mech422pbourke: it really appeals to me to use the same admin tools ceph recommends14:37
Mech422pbourke: how do you shrink a cluster ? if you remove a disk from inventory - nothing happens to it14:38
pbourkeI agree we dont have tasks for that right now14:38
pbourkebut essentially you just need to stop the osd on that node14:39
*** ppowell has quit IRC14:39
pbourke5 lines of ansible14:39
Mech422pbourke: well yeah - should mark it out and all that jazz14:39
Mech422pbourke: and probably disable the startup on boot for it14:39
pbourkedoes ceph not do that automatically14:40
Mech422pbourke: only if you tell it too14:40
Jeffrey4lhe guys, please review this https://review.openstack.org/32980814:40
pbourkeah14:40
Mech422pbourke: 'stop' is not 'delete'14:40
pbourkeI see14:40
Jeffrey4lwhen remove the osd: 1. mark the osd out  2. wait for the ceph to re-balance  3. stop the osd  4. remove the osd from crushmap  5  remove the osd14:41
Mech422Jeffrey4l: I thought there was something went crazy if you didn't use new kernel ? shared mounts or something ?14:41
Jeffrey4lMech422, ubuntu 14.04 support shared mount.14:41
Jeffrey4lin the CI, we deploy kolla on the default kernel in the ubuntu 14.04. it works and works fine. never see any kernel related issue.14:42
Mech422Jeffrey4l: oh - ok14:42
Mech422I just thought I remembered something from the install docs saying docker didn't play nice unless you upgraded14:43
Mech422be nice to have the option to use existing kernel though14:43
Jeffrey4lMech422, yes. Doc said that. I checked the history of the doc. this doc came from: aufs *may* have bug in the lower kernel. if you hit this, please try to upgrade your kernel.14:44
Jeffrey4lyep14:44
Mech422pbourke: nice thing about ceph deploy is its bog standard - so if you ask for help on something ceph related, you don't get 'thats a non-standard install - ask your vendor' and you have all the tools they expect to have for debugging14:44
Mech422Jeffrey4l: oh cool...that would save a reboot on provisioning14:45
Jeffrey4lMech422, it is hard to use ceph-deploy tool. for example, how could it start the container?14:45
Jeffrey4lmostly, upgrade kernel is not unacceptable, especially in the prod env.14:46
Mech422Jeffrey4l: yeah - thats the bitch - it doesn't speak container14:46
Jeffrey4lI think we can copy the ceph-deploy script into the kolla ansible task.14:46
Jeffrey4lit is very easy.14:46
Mech422Jeffrey4l: although, it could be installed IN the container...14:47
sdake_morning14:47
Jeffrey4lI mean copy how the ceph-depoy works.14:47
Jeffrey4lMech422, no. ceph depend on the ssd.14:47
Jeffrey4lssh14:47
Mech422Jeffrey4l: and it uses ssh to talk to all the cluster members - so you could add a port to the osd containers maybe ?14:47
Jeffrey4lthat's not happend.14:47
Jeffrey4lsdake_, moring sir14:47
Mech422Jeffrey4l: yeah - its uses ssh, so if it ran on the deployment host it could ssh to whatever...14:48
Mech422Jeffrey4l: but I agree, its probably just a pipe dream14:48
Jeffrey4llol14:48
*** tloeb has joined #openstack-kolla14:50
*** v1k0d3n has joined #openstack-kolla14:52
Jeffrey4lsdake_, have u any idea why using pacemaker+haproxy rather than the keepavlied+haproxy in the openstack offical ha doc?http://docs.openstack.org/ha-guide/controller-ha-pacemaker.html14:52
Mech422Jeffrey4l: btw - does your ceph cluster calculate PG's properly ?14:53
Jeffrey4lMech422, I calculate it manually.14:53
Mech422Jeffrey4l: oh - so you don't install ceph via kolla ?14:54
Mech422Jeffrey4l: or you adjust it after install ?14:54
Jeffrey4lMech422, adjust it after install14:54
Jeffrey4lkolla do not calcualte the pg num.14:54
Mech422Jeffrey4l: ahh - kk - mine is whining about too many PGs per OSD14:54
Mech422Jeffrey4l: yeah - I sorta figured it didn't14:54
Mech422just wanted to make sure it wasnt' something stupid on my end14:55
*** severion has quit IRC14:55
*** mliima has quit IRC14:55
tloebhello everyone o7, we are trying to deploy a multinode openstack with kolla. Sadly we are having some issues. The Task "Fetching Ceph Keyrings" cant reach one of the controller node. Any idea where can i start to look?14:57
sdake_Jeffrey4l probably red hat people wrote it - hence the slant towards pacemaker14:58
sdake_also pacemaker will shut down failed nodes whicih haproxy does not do14:58
*** ppowell has joined #openstack-kolla14:58
sdake_tloeb are you running sudo kolla-ansible deploy?14:59
Mech422tloeb: there was a bug about ceph keyrings yesterday(?) - I think it said to check your ansible version ?14:59
*** severion has joined #openstack-kolla14:59
sdake_tloeb i have had a probblem where ssh was not configured for root on one of the hosts14:59
*** inc0 has quit IRC15:00
Jeffrey4lOK15:00
tloebsdake_: we are not running ansible deploy with sudo15:00
*** v1k0d3n has quit IRC15:00
sdake_tloeb i have noticed ceph doesnt wor without running as root15:01
Jeffrey4lsdake_, check this please https://review.openstack.org/32980815:01
sdake_although I could be wrong here15:01
*** ravig has joined #openstack-kolla15:02
sdake_Jeffrey4l did you check the git glame to see why that chane was added15:02
sdake_I seem to recall it made sense at the time15:02
Jeffrey4lyep. this pre-check is added based on the doc.15:03
*** b_bezak has quit IRC15:03
Jeffrey4li will show u the original doc15:03
*** ravig has quit IRC15:03
Jeffrey4lsdake_, here it is .https://github.com/openstack/kolla/blob/2864b46957e7fbe830dc8d56a5f3915cc5aa8a2b/docs/dev-quickstart.rst#installing-dependencies15:05
sdake_14.04 is eol, i thought we were going to 16.04 this release?15:05
*** dwalsh has joined #openstack-kolla15:05
*** v1k0d3n has joined #openstack-kolla15:06
*** severion has quit IRC15:07
*** inc0 has joined #openstack-kolla15:07
*** diogogmt has joined #openstack-kolla15:08
*** sdake_ has quit IRC15:08
*** sdake has joined #openstack-kolla15:10
*** mliima has joined #openstack-kolla15:10
*** sdake has quit IRC15:11
tloebsdake_: We tried the following cmd, it is still running but a lot of red lines (we use kolla_version_tag:2.0.0) --> kolla-ansible deploy -i /usr/share/kolla/ansible/inventory/multinode  --configdir /home/user/kolla_conf --passwords /home/user/kolla_conf/passwords.yml15:13
pbourketloeb: can you pastebin your error output15:14
*** SiRiuS__ has joined #openstack-kolla15:15
Mech422tloeb: Umm - I think the version is 2.0.1 now ?15:18
Mech422tloeb: at least, thats the tag on containers it was building for me15:18
tloebpbourke: http://pastebin.com/U6HwYrji15:19
Mech422tloeb: oh - my bad, I'm thinking of openstack_release in globals15:20
tloebyes the default openstack_release in the globals was 3.0.015:21
inc0btw gates seems broken15:21
tloebMech422: now we tried it with 2.0.0 .. do you think 2.0.1 is better?15:21
*** tyrola has joined #openstack-kolla15:22
Mech422tloeb: 3.0.0 is for master15:23
Mech422tloeb: 2.0.1 is for stable/mitaka15:23
*** tyrola has quit IRC15:23
Mech422tloeb: but your getting 'UNREACHABLE' - that means ssh is borked15:23
Mech422tloeb: can you ssh from your deployment host to controller001 as whatever user ansible is using ?15:24
tloebMech422: we disabled the firewall now15:24
Mech422tloeb: ok - did that fix it ?15:24
tloebMech422: yes ssh ansible_user@controller001 is working15:25
Mech422tloeb: should be good to go then :-)15:25
tloebMech422: no we are getting the same error with deploying :( it tries to setup the ceph cluster network on a controller.. but i thought controllers have no ceph cluster network15:27
pbourketloeb: run your ansible command with -vvv15:28
pbourketloeb: check what user it's trying to use15:28
pbourkeinc0: added gates to meeting discussion15:28
inc0pbourke, won't be at meeting unfortunately :S15:28
pbourkeinc0: if you can't make meeting can you please check the mailing list - there's a discussion there15:28
pbourkeinput appreciated15:29
inc0kk15:29
*** severion has joined #openstack-kolla15:29
*** v1k0d3n has quit IRC15:31
inc0pbourke, so currently we have issue of ansible looking for 3.0.015:31
inc0we didn't tag 3.0.0 yet right?15:31
*** v1k0d3n has joined #openstack-kolla15:31
*** coolsvap has joined #openstack-kolla15:31
inc0so build default won't be 3.0.015:31
pbourkei thought those were fixed...15:31
pbourkeinc0: Im more interested in figuring out the overarching issue of why our gates suck so much15:31
*** ravig has joined #openstack-kolla15:32
inc0agree, we need to move to voting gates15:32
pbourkeas dmsimard mentioned osad/tripleo/etc. all have voting gates15:32
*** severion has quit IRC15:34
*** shardy has quit IRC15:35
dmsimardsorry :)15:35
inc0dmsimard, that's fair statement, our gates are bad15:35
inc0so part of it is that as opposed to tripleo/osad and such, we don't really have any company supporting infra for us15:37
inc08gig of ram fails to deploy full openstack from time to time15:37
inc0also our gate infra is far from reliable15:37
Mech422inc0: what are the requirements for gate boxes ?15:39
inc0Mech422, so current vms in gates are 8gig of ram, no swap and such15:39
Mech422inc0: how many concurrent VMs ?15:40
inc0just aio15:40
inc0so it's really bad15:40
Mech422inc0: hmm.. we just did hardware refresh this year15:40
inc0Mech422, I'm talking about openstack infra15:40
Mech422inc0: so we have a bunch of supermicro 2U twins15:40
Mech422inc0: I might be able to talk my way into a couple for 'testing' or 'lab' use15:41
Mech422inc0: I think they're 2xquad with 32G RAM ?15:41
inc0well, issue is15:41
inc0it cant really work on infra15:41
inc0and voting gates are on infra15:41
Mech422inc0: oh - they have to be on OS's network...yeah, that probably won't fly15:42
Mech422inc0: unless we get a tax write-off...might be able to sell that15:42
inc0even if you'd donate hardware to infrea15:42
inc0they would just spawn 8gig vms ;)15:42
tloebpbourke: http://pastebin.com/2drK4ihb this is the output with -vvv ... he is using the right user15:42
Mech422inc0: you lost me...I thought you were doing 8G AIO because you didn't have a company supporting you with hardware ?15:42
dmsimardinc0: puppet openstack uses 100% openstack-infra15:43
dmsimardand they tackle ram limitation by using different scenarios15:43
dmsimardinc0: https://github.com/openstack/puppet-openstack-integration#description15:43
dmsimardso they don't have "one all-in-one" test, but rather 3 different scenarios with interesting matrix of configurations and components that is run on centos with rdo and on ubuntu with uca.15:44
Mech422dmsimard: so is the issue a lack of hardware, or just re-writing configuration tests ?15:45
pbourketloeb: I think you have a different issue now. Try running the 'cleanup' task of kolla-ansible15:45
pbourketloeb: then redeploy ceph15:45
*** mikelk has quit IRC15:45
dmsimardMech422: gate jobs have to run within the constraints imposed by the virtual machines provided by -infra15:45
Mech422dmsimard: ahhh.....15:45
pbourkeour gate is pretty minimal though15:45
dmsimardMech422: from memory, VMs have 8 cores, 8GB of RAM and 80GB of space15:45
pbourkewe're not really testing anything outside the core services15:45
pbourkeinc0: Im not sure the problems are memory related...15:46
dmsimardpbourke: that statement is concerning15:46
inc0btw I know what's wrong with out gates now15:46
inc0we need new images pushed to docker.io...15:46
inc0only sdake has access to it right?15:46
pbourkethe problems I've seen are often environmental15:46
dmsimardpbourke: if you don't test every project implementation in kolla, how do you tell if it works ?15:46
pbourkein that we're not setup to handle the varibilty in the base vms15:47
pbourkedmsimard: we dont is the honest answer15:47
pbourkeits not uncommon for people to pull down big tent services and find they flat out dont work15:47
inc0we had discussion about ceph just today15:47
pbourkedmsimard: but that could be solved by creating different scenarios as you say15:48
pbourkefirst though we'd need to get the core compute solid15:48
dmsimardyeah, one with cinder, one without cinder, one with glance swift backend, one with glance file backend, etc.15:48
coolsvapinc0, we just got the kolla namespace on docker hub15:49
pbourkeinc0: are you -2 against scraping sgdisk15:49
coolsvapalso Jeffrey4l proposed the patch to stop pulling from docker hub15:49
inc0pbourke, sgdisk? we don't use sgdisk....15:49
inc0we use udev (unless you mean something else than ceph)15:49
pbourkewe need to replace udev15:49
inc0ahh15:49
inc0I'm not -2, we just had lots and lots of issues with scrapping15:50
inc0particulary error handling was a horror story15:50
inc0coolsvap, can you paste a link to review plz?15:50
*** Lyncos has joined #openstack-kolla15:51
pbourkeMech422: maybe we can finish out your patch to use sgdisk everywhere15:51
pbourkeand trial it for a while15:51
coolsvapinc0, which one?15:51
tloebpbourke, Mech422: ok thanks for the help! :) now we have something to work with for debugging15:51
coolsvappatch 329735 is proposed by Jeffrey4l related to pull15:51
patchbotcoolsvap: https://review.openstack.org/#/c/329735/ - kolla - Remove the pull action in the deploy_aio CI15:51
inc0coolsvap, not pulling images from dockerhub15:51
pbourketloeb: np. am working through similar issues atm15:51
*** jrist has quit IRC15:52
inc0ah cool15:52
*** dmk0202 has quit IRC15:52
coolsvapinc0, but the major gate failures recently are due to nova-compute container starting issue from my observation which i think we need to track where and why happens15:53
coolsvapis it related to again any specific hardware15:53
Jeffrey4lwhich issue coolsvap ?15:54
Jeffrey4lnova-compute start issue?15:55
coolsvapJeffrey4l, let me give you an example wait15:55
*** SiRiuS__ has quit IRC15:55
coolsvapJeffrey4l, http://logs.openstack.org/87/327987/1/check/gate-kolla-dsvm-deploy-centos-binary/432b175/console.html#_2016-06-13_10_14_38_10015:56
*** sean-k-mooney has quit IRC15:57
*** SiRiuS__ has joined #openstack-kolla15:57
Jeffrey4lcoolsvap, i know this issue.15:57
Jeffrey4lI debuged this.15:57
*** zhiwei has quit IRC15:57
Jeffrey4land see this in my locally once.15:57
Jeffrey4lI think it is a bug from keystone.15:57
Jeffrey4lthe endpoint is cached and not updated.15:57
*** severion has joined #openstack-kolla15:58
Jeffrey4lcoolsvap, pbourke check the tokens here http://logs.openstack.org/87/327987/1/check/gate-kolla-dsvm-deploy-centos-binary/432b175/console.html#_2016-06-13_10_14_38_09015:58
*** v1k0d3n has quit IRC15:58
Jeffrey4lthere is only keystone and glance endpoint. no nova, neutron.15:58
Jeffrey4lI think restart memcached may fix this issue. But i have not any proof now.15:59
Mech422pbourke: actually - that patch should be good as is... I rebuild with it about 2x day, and ceph always comes up15:59
Jeffrey4lhere is my debug PS https://review.openstack.org/32714615:59
pbourkeMech422: ok cool16:00
Mech422pbourke: but I only do partition based ceph - could you try it on whole disk setup?16:00
pbourkeMech422: I will16:00
pbourkeMech422: how do you do parttion base16:01
coolsvapJeffrey4l, cool16:01
pbourkeMech422: the bootstrap script is not setup for that16:01
* coolsvap brb dinner16:01
*** tloeb has quit IRC16:01
Mech422pbourke: I mean I just run ceph on partitions - oh yeah it is...its just 'undocumented'16:01
pbourkemeeting?16:01
pbourkerhallisey: ^16:01
Jeffrey4li think we have meeting in 30 minute right?16:02
rhalliseyin 3016:02
rhalliseyor not16:02
rhalliseywait it's 16:00 now16:02
Mech422Jeffrey4l: I've noticed, after etc/kolla/admin-openrc.sh16:02
Mech422err...after sourcing etc/kolla/admin-openrc.sh16:02
rhalliseyis that right?16:02
* rhallisey checks ML16:02
pbourke16:00 UTC16:03
Jeffrey4lMech422, what?16:03
Mech422I have to export export OS_INTERFACE=admin16:03
Mech422and export OS_ENDPOINT_TYPE=admin16:03
pbourkeMech422: would you have time to document it?16:03
pbourkeor just give me the overall steps?16:03
Mech422to get /tools/init-runonce to work semi-right16:03
Mech422glance still chokes trying to upload cirros - wants to use the public endpoint16:04
*** mummer has joined #openstack-kolla16:04
rhalliseythe meeting prio was 16:30, but it look like for this week I think people are expecting 16:0016:04
Mech422also, init-runonce hard codes the public network subnet to 10.0.0.0 instead of looking at whats already there16:04
Jeffrey4lsdake is not here too.16:05
pbourketrue16:05
Mech422(its a flat network, called 'public' - so it probably shouldn't assume it can assign stuff )16:05
*** sdake has joined #openstack-kolla16:05
*** athomas has quit IRC16:05
rhalliseysdake, meeting now?16:05
sdakeyup16:05
sdakeapologis for being late16:05
rhalliseyidk if it was a type but it was 16:30 before16:05
rhalliseykk16:05
Mech422pbourke: this is my current 'post-deploy' http://pastebin.com/zdtNb8te16:06
rhalliseywirehead_, dcwangmit01 guys around?16:06
rhalliseyjoin #openstack-meeting-416:06
pbourkeMech422: for partition based ceph?16:07
Mech422pbourke: well - its not related to ceph, but yes - I use partitions for ceph16:08
Mech422pbourke: this is because my 'public' network is not accessible via deployement_host16:09
Mech422pbourke: err..actually, its the internal network, not public but same idea16:09
pbourkeMech422: im interested in docs/general steps for ceph partititions16:11
*** jmccarthy has left #openstack-kolla16:13
Mech422pbourke: oh - nothing really special - just change a partition name to 'CEPH_OSD_BOOTSTRAP_1' and change the name on the journal partition to 'CEPH_OSD_BOOTSTRAP_1_J;16:13
Mech422pbourke: repeat with _2, _3, etc as desired16:13
wirehead_Gah, didn't catch the typo.16:13
*** jmccarthy has joined #openstack-kolla16:13
Mech422pbourke: here's the actual play I use to setup my disks : http://pastebin.com/1LcRfaPv16:16
pbourkeMech422: thanks16:16
Mech422pbourke: :-)16:17
*** Serlex has quit IRC16:17
*** severion has quit IRC16:18
*** v1k0d3n has joined #openstack-kolla16:20
*** jmccarthy has left #openstack-kolla16:20
coolsvapdid we start meeting early?16:22
*** v1k0d3n has quit IRC16:23
vhosakot7 mins to meeting16:23
Mech422it started...16:23
coolsvapits already started 20 mins back16:23
Mech422#openstack-meeting-416:23
vhosakotreally ?16:23
*** v1k0d3n has joined #openstack-kolla16:24
coolsvapyes i went for dinner since there was half n hour16:24
*** mark-casey has joined #openstack-kolla16:24
coolsvap:(16:24
vhosakotI went to grab lunch as well16:24
*** mdnadeem_home has joined #openstack-kolla16:25
*** v1k0d3n has quit IRC16:27
vhosakotwow, everyone left16:27
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Add customizations for apt keys and packages  https://review.openstack.org/32668816:27
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Macro install packages  https://review.openstack.org/32673416:27
pbourkecoolsvap:were you not the one who proposed 16:00 :)16:27
Mech422sdake: hope ya feel better!16:28
vhosakotsdake: take care16:28
coolsvappbourke, yeah16:28
coolsvapand i feel so bad i missed it first up16:28
coolsvapwhen the clock changed -30 mins16:28
coolsvapi dont know16:28
pbourkecoolsvap: it happens16:29
rhalliseyinc0, meeting?16:30
*** ravig has quit IRC16:30
inc0rhallisey, I have meetings irl now16:31
inc0so afk now16:31
rhalliseykk16:31
inc0all saints came over to SAT16:31
inc0;)16:31
inc0crazy week16:31
coolsvapharlowja_, around?16:32
*** matrohon has quit IRC16:33
*** mummer has quit IRC16:41
*** ravig has joined #openstack-kolla16:41
*** mummer has joined #openstack-kolla16:41
*** sdake has quit IRC16:42
*** mbound has joined #openstack-kolla16:43
pbourkedmsimard: discussing gates in openstack-meeting-4 now if you're around16:45
*** Lyncos has left #openstack-kolla16:48
*** NobodyCam has quit IRC16:59
*** mgkwill has quit IRC16:59
*** NobodyCam has joined #openstack-kolla17:00
harlowja_coolsvap whats up17:00
coolsvaprhallisey, seriously i actually checked google for utc now17:00
wirehead_rhallisey: https://review.openstack.org/#/c/329676/17:00
patchbotwirehead_: patch 329676 - kolla-kubernetes - Consolidating path resolution and searching for pr...17:00
coolsvapharlowja_, hey17:00
*** mgkwill has joined #openstack-kolla17:00
coolsvapregarding the tqdm review in requirements and related reviews in kolla17:01
harlowja_ya17:02
harlowja_i gotta fix those merge conflicts17:03
coolsvapharlowja_, i wanted to know status from both you and reviewers17:03
coolsvapif you can update the reviews17:03
harlowja_yup17:03
harlowja_i can17:03
harlowja_eventually consistent review patches17:03
harlowja_lol17:03
coolsvapi want to test them first thing17:03
coolsvapmaybe tomorrow morning and i need to update the requirement team in couple of days17:04
harlowja_k17:04
coolsvapthe worst case is they will abandon the review and you will need to submit it again17:04
*** Jeffrey4l has quit IRC17:05
harlowja_who's they17:05
harlowja_lol17:05
harlowja_i will find they and abandon them17:05
harlowja_lol17:05
coolsvapharlowja_, thanks17:05
coolsvapi am one of them17:06
harlowja_in russia the review abandons u17:06
harlowja_lol17:06
coolsvapharlowja_, i will abandon myself for today time for some sleep will see for your patches tomorrow morning ;)17:06
harlowja_lol17:07
coolsvapharlowja_, lol if pronounced in my native language means roll on bed :D17:08
Mech422coolsvap: heh - a roll on the bed is always good for a laugh :-)17:10
coolsvapMech422, :)17:11
*** coolsvap has quit IRC17:11
*** sacharya has quit IRC17:11
Mech422so it appears augeas is already being used for some openstack stuff...17:12
*** sacharya has joined #openstack-kolla17:12
harlowja_it is?17:15
*** harbie has joined #openstack-kolla17:21
*** ravig has quit IRC17:26
*** ravig has joined #openstack-kolla17:27
Mech422harlowja_: yeah - I saw some bug reports about nova trying to call augeas17:27
Mech422harlowja_: trying to find the lenses now17:28
harlowja_interesting17:28
openstackgerritMerged openstack/kolla: Revert "Added precheck to check kernel version 4.2 on Ubuntu 14.04."  https://review.openstack.org/32980817:28
openstackgerritMerged openstack/kolla: Remove the pull action in the deploy_aio CI  https://review.openstack.org/32973517:28
*** fragatina has quit IRC17:29
*** v1k0d3n has joined #openstack-kolla17:39
*** sacharya_ has joined #openstack-kolla17:40
rhalliseywirehead_, nice17:41
*** ravig has quit IRC17:42
*** sacharya has quit IRC17:43
*** v1k0d3n has quit IRC17:44
*** mdnadeem_home has quit IRC17:44
dcwangmit01wirehead_: rhallisey: I see enough +2's for these patches, but they don't seem to be merging 328569 (2+1's, 2+2's)# Keystone interface address and memcached override17:49
dcwangmit01327987 (4+1's, 2+2's)# Fix interface address for memcached atop kubernetes17:49
dcwangmit01327925 (2+1's, 1+2's)# Enable kolla k8s to override bind api bind address in genconfig17:49
dcwangmit01321529 (4+1's, 2+2's)# Add kolla-kubernetes bootstrap capability to Keystone17:49
rhalliseydcwangmit01, they're dependant on https://review.openstack.org/#/c/327925/17:50
patchbotrhallisey: patch 327925 - kolla - Enable kolla k8s to override bind api bind address...17:50
rhalliseywhich needs 1 more +217:50
dcwangmit01Got it17:51
dcwangmit01rhallisey: Where is the dependency link set?  I would expect it in the Commit message.  I don't see the dependency anywhere on the gerrit page for dependent nor dependee17:55
dcwangmit01^ Sorry new to openstack /gerrit process17:55
wirehead_If you have the previous patch in your git log, it’ll automatically set a link and it won’t make it especially clear that’s the case.17:56
rhalliseythose patches were made on top of that patch I linked17:56
dcwangmit01Oh, so if I create a change off of a branch from a prior review, it is auto17:56
wirehead_yeah17:59
wirehead_The UI there is very un-clear.17:59
wirehead_Then again, I don’t think GitHub’s is that much better in those cases.17:59
dcwangmit01Github is pretty dumb about it.  The 2nd PR based off of the 1st will show all of the changes from the 1st.18:00
dcwangmit01Gerrit seems better in this case18:00
dcwangmit01And I can't believe I just said that18:00
wirehead_Yeah.  The current missing feature in source control tooling is the ability to land a large deeply integrated feature while still reviewing it properly.18:02
dmsimardpbourke: was not around, sorry18:04
*** mbound has quit IRC18:06
dcwangmit01wirehead_: I do miss the ability to have a release branch.  I keep cherry-picking/re-merging the same 7 patches from kolla.  I have no idea whether I'm testing with the same code as you and rhallisey18:06
dcwangmit01^ less a release branch, but more a branch for patches that have not been merged.18:07
*** fragatina has joined #openstack-kolla18:08
esharao__hi..i'm getting "Container exited with non-zero return code" for TASK [keystone : Running Keystone bootstrap container]18:11
dmsimardrhallisey, pbourke, saw the meeting log about gate jobs. Thanks for the attention.18:12
*** vhosakot has quit IRC18:13
*** vhosakot has joined #openstack-kolla18:21
inc0I'm back18:22
wirehead_I’m still wirehead.18:22
inc0ahhh these jokes18:26
inc0all the laught18:26
inc0;)18:26
*** salv-orlando has quit IRC18:43
esharao__hi I tried to run kolla-ansible -vvv deploy but it does not tell me why my keystone container is failing.. any other place to look?18:44
*** salv-orlando has joined #openstack-kolla18:44
*** mummer has quit IRC18:45
*** SiRiuS__ has quit IRC18:54
*** zhiwei has joined #openstack-kolla18:54
*** salv-orl_ has joined #openstack-kolla18:55
*** salv-orlando has quit IRC18:58
*** zhiwei has quit IRC18:59
*** mbound has joined #openstack-kolla19:06
*** mbound has quit IRC19:11
inc0esharao__, docker logs keystone19:14
*** mliima has quit IRC19:22
*** ravig has joined #openstack-kolla19:24
openstackgerritJoshua Harlow proposed openstack/kolla: Stop using a global logger for all the things  https://review.openstack.org/32188419:32
*** openstackgerrit has quit IRC19:33
*** openstackgerrit has joined #openstack-kolla19:33
*** mummer has joined #openstack-kolla19:34
*** sdake has joined #openstack-kolla19:36
inc0yo sdake are you alive?19:36
sdakebarely19:37
sdake2 hours of driving19:37
sdakeafter cleaning up, well you know, cleaning up19:37
vhosakotsdake: you ok?19:37
*** ravig has quit IRC19:38
*** sdake_ has joined #openstack-kolla19:40
*** sdake has quit IRC19:42
*** ravig has joined #openstack-kolla19:42
*** d_code has quit IRC19:43
inc0guys can I have eyes on this one? https://review.openstack.org/#/c/32673419:45
*** salv-orl_ has quit IRC19:45
*** salv-orlando has joined #openstack-kolla19:45
openstackgerritJoshua Harlow proposed openstack/kolla: Be smarter about what to do when making a docker client  https://review.openstack.org/33017119:46
*** SiRiuS__ has joined #openstack-kolla19:46
*** ravig has quit IRC19:48
*** mark-casey has left #openstack-kolla19:53
esharao__inc0, it says no such container as keystone19:58
Mech422harlowja_: btw - is anything supposed to be kibana besides keystone-apache logs? I don't know kibana, but with a quick look, thats all I found19:59
Mech422inc0: btw - have you seen augeas ( http://augeas.net )19:59
harlowja_Mech422 not sure19:59
Mech422inc0: it might help with your merge stuff ?19:59
rhalliseyesharao__, do a `sudo docker ps -a`19:59
rhalliseyesharao__, there's no keystone container there?20:00
Mech422inc0: (well, the non-ini file stuff)20:00
esharao__0836381a1ec4        operator.local:4000/lokolla/centos-binary-keystone:3.0.0        "kolla_start"            About an hour ago   Exited (1) About an hour ago                            bootstrap_keystone20:00
esharao__rhallisey, no.. i do not see any that are up and running..20:00
inc0esharao__, docker logs bootstrap_keystone then;)20:01
rhalliseyesharao__, docker logs boostrap_keystone20:01
esharao__got it after adding sudo... :) ImportError: cannot import name ssl_match_hostname20:02
inc0Mech422, I was rather thinking about using https://review.openstack.org/#/c/32673420:02
inc0http://docs.ansible.com/ansible/lineinfile_module.html this one20:02
inc0paste fail20:02
esharao__http://pastebin.com/e4kTvsse20:03
*** ppowell has quit IRC20:03
inc0esharao__, what distro you use/install type?20:04
esharao__centos (via vagrant)20:04
Mech422inc0: eewww...line in file ? really? why oh why ? :-P20:04
inc0Mech422, I like regex => replace20:05
Mech422inc0: look at the nice interface augeas gives you, and the huge number of formats it supports...20:05
Mech422inc0: I agree regex is pretty easy to understand...but just eww20:05
Mech422inc0: I do like 30 lininfiles to dynamically build my /etc/hosts - it's really nasty20:06
*** sdake_ has quit IRC20:07
*** sdake has joined #openstack-kolla20:07
Mech422inc0: Anyway, it was just a thought20:07
*** sdake_ has joined #openstack-kolla20:11
Mech422harlowja_: thanks - it might just be something stupid on my end20:11
*** sdake has quit IRC20:12
*** ravig has joined #openstack-kolla20:13
mag009_how do I specify the version of docker image ?20:13
mag009_let say I want to build image for mitaka20:14
*** ravig has quit IRC20:14
*** matrohon has joined #openstack-kolla20:16
mag009_ok nvm i found it20:19
Mech422mitaka should use 3.0.0 I believe20:23
*** sdake_ has quit IRC20:27
openstackgerritRyan Hallisey proposed openstack/kolla: GATE TESTING  https://review.openstack.org/33019220:28
*** matrohon has quit IRC20:34
*** zhiwei has joined #openstack-kolla20:39
*** zhiwei has quit IRC20:44
Mech422rhallisey: ok - so how do I get started testing koll/k8s ?20:45
*** neilus has quit IRC20:45
rhalliseyMech422, so start here: https://github.com/openstack/kolla-kubernetes/blob/master/doc/source/quickstart.rst20:46
Mech422rhallisey: Danke20:46
rhalliseyMech422, there are a bunch of patches sitting in both kolla/kolla-kubernetes queue that will fix bootstrapping, CLI, ect...20:46
rhalliseyhopefully they will merge soon so you don't have to carry like 6 patches in your branches20:47
Mech422rhallisey: ok - I'll give it a try...20:47
rhallisey Mech422 the patches you'll need are here: https://etherpad.openstack.org/p/GTAZV0AXm820:47
rhalliseyawesome20:47
wirehead_yay20:50
Mech422rhallisey: oh - I have to setup k8 manually ?20:50
rhalliseyMech422, you can use hypecube20:50
Mech422this'll be fun - never did k8 before20:50
rhalliseyit will setup kubernetes running in containers20:50
Mech422rhallisey: cool :-)20:50
Mech422rhallisey: like I said - never did k8 before, so expect a bunch of really-stupid-noob questions tommorrow :-)20:51
rhalliseyno problem20:51
wirehead_Mech422: I’m usually on later than rhallisey, so feel free to ask away.20:52
Mech422wirehead_: LOL - got me working all night, eh ? :-P20:52
* Mech422 makes more coffee20:52
wirehead_Well, hopefully it won’t take all night. :D20:53
Mech422wirehead_: it'll be a while - I gotta build my ansible bootstrap stuff for it20:54
*** ravig has joined #openstack-kolla20:54
Mech422wirehead_: but once thats done, I can re-build and re-deploy a couple of times a day to test stuff20:54
*** v1k0d3n has joined #openstack-kolla20:57
*** ravig has quit IRC20:58
*** ravig has joined #openstack-kolla20:59
*** ravig has quit IRC21:03
Mech422wirehead_: so bare-metal + docker multinode a good starting point? http://kubernetes.io/docs/getting-started-guides/docker-multinode/21:08
wirehead_Mech422: So bunches of stuff probably still breaks in the multinode case.21:09
Mech422wirehead_: oh - well, only one way to find out...21:09
wirehead_True. :)21:09
wirehead_You also can probably just run it on GKE.21:09
Mech422wirehead_: this builds a 2 node cluster with flanneld21:10
Mech422wirehead_: now, where would the fun be in that... cheater :-P21:10
wirehead_Or via the Kubernetes Vagrant, which also gives you multinode.21:10
Mech422I don't have a google account setup - I could do aws if you really want...21:10
Mech422but I sorta wanna play with setting it up from scratch - bout time I learned some k821:11
wirehead_So, dcwangmit01 has been using the Ansible playbooks to fire up bare metal Kube clusters.21:11
wirehead_That’s where the community seems to be moving.21:12
*** ravig has joined #openstack-kolla21:12
wirehead_Those instructions ought to work.21:12
wirehead_You might have some fun with the DNS part.21:12
Mech422wirehead_: oh - let me see if I can find the ansible stuff21:13
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: WIP - kolla-kubernetes gate  https://review.openstack.org/33022621:13
wirehead_https://github.com/kubernetes/contrib/tree/master/ansible21:14
*** ravig has quit IRC21:14
rhalliseywirehead_, I have some ideas to change the way we bootstrap/deploy21:15
rhalliseyI'll go over them tmr21:15
wirehead_rhallisey: k.  I’m not entirely pleased with the present status.21:16
rhalliseywirehead_, ya we're still getting rolling21:16
wirehead_rhallisey: so, changes good.21:16
rhalliseytons to iron out21:16
Mech422wirehead_: these playbooks ? https://github.com/kubernetes/contrib/tree/master/ansible21:16
rhalliseythe more we play with this the higher likely hood we find new ways to use it21:17
wirehead_Mech422: yah21:17
rhalliseysee ya tmr21:17
Mech422wirehead_: thanks21:17
Mech422rhallisey: later!21:17
wirehead_later, rhallisey21:17
rhalliseyMech422, good luck! :)21:17
*** rhallisey has quit IRC21:17
Mech422wirehead_: blah - these only talk about supporting RH/Centos... lets see if it chokes on ubuntu21:19
*** ravig has joined #openstack-kolla21:21
wirehead_Mech422: http://kubernetes.io/docs/getting-started-guides/ubuntu/21:22
Mech422wirehead_: I can do that - you want flannel right ?  Ubuntu 14 or 16 ?21:24
wirehead_14.21:25
Mech422oh yeah - this guide doesnt like systemd21:25
wirehead_Mech422: Also, https://github.com/kubernetes/contrib/pull/76121:25
wirehead_The docs in Kubernetes are starting to lag.21:27
dcwangmit01Mech422: If you want Kube on bare metal: https://www.davidwang.com/2016/05/21/kubernetes-bare-metal-install-on-packet-net/21:27
dcwangmit01wirehead_: DNS works in the case above ^.  kubernetes/contrib/ansible is awesome21:27
harlowja_qq, for the kube stuff, is there a doc for how kube and kolla work?21:27
harlowja_(am curious to see how it works together as a whole system)21:28
wirehead_harlowja_: https://review.openstack.org/#/c/304182/21:28
patchbotwirehead_: patch 304182 - kolla - Spec: Deploy Kolla images using Kubernetes21:28
wirehead_harlowja_: As to how it’ll work long-term…21:29
esharao__wirehead_, kolla-kubernetes does not work on ubuntu 16?21:30
dcwangmit01harlowja_: Here are some operational notes I have on how to use Kolla with Kolla-Kubernetes: https://gist.github.com/dcwangmit01/553645bc018f5bc4a36a78a5f786dee221:30
harlowja_wirehead_ fair enough :)21:30
wirehead_I have no idea.  I am testing against Ubuntu 14.21:30
esharao__i'm having issues getting the docker to 1.10.321:30
wirehead_harlowja_: The big thing to understand is that the containers of Kolla work more-or-less as they do in regular Kolla, it’s just we’re taking advantage of the Kubernetes network layer and service framework.21:31
harlowja_yup21:31
harlowja_and all that state ...21:31
harlowja_the state, oh the sate21:31
harlowja_*state21:31
wirehead_harlowja_: Depends on the service, of course.21:31
harlowja_sure21:31
wirehead_harlowja_: e.g. Keystone seems fairly happy.21:31
harlowja_i mean, we were just discussing how something like zookeeper would run in kube21:32
harlowja_so that makes me wonder how something like galeria runs in kube to21:32
harlowja_and how do those services handle running in such a enivornment21:32
wirehead_harlowja_: After 1.3, something like this: https://github.com/kubernetes/contrib/tree/master/pets21:32
harlowja_:-/21:32
harlowja_ya, that's not what i like :(21:32
harlowja_cause zookeeper really isn't a pet, mysql maybe u could say it is21:33
Mech422ok - so I'll follow davids link first - then we have two 'identical' setups to test with21:33
Mech422if needed, I can do others - coreos, etc21:33
harlowja_wirehead_ something sorta of feels wrong (imho) if things that are cloudy-solutions have to be called pets to work in kube21:33
harlowja_and/or feels off21:34
wirehead_Well, ElasticSearch and Cassandra work magically in Kube.21:34
harlowja_right, so i guess whats the underlying reason why those are different than say zookeeper?21:34
wirehead_No gossip.21:34
harlowja_so kube needs to handle things that do gossip better?21:35
wirehead_ElasticSearch and Cassandra need to know a list of ‘seed’ servers and then they gossip amongst themselves.21:35
harlowja_right, i thought zookeeper was getting similar stuff21:35
wirehead_Zookeeper needs to have a rigid set of servers.21:35
harlowja_hmmmmm, think that has changed, let me check21:36
harlowja_* https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html21:36
harlowja_might not be total gossip based i guess, but still more dynamic21:37
*** jtriley has quit IRC21:39
wirehead_Like, having looked at installing it in 1.2, ZK has many attributes that make it more cattle.  You can kill zk instances and respawn and things are fine.21:39
*** ravig has quit IRC21:39
wirehead_It’s just it’s got some necessarily stateful infra to prevent footguns.21:39
harlowja_ya, i think that is starting to change21:39
harlowja_with some of this dynamic reconfiguration (And then i guess discovery features)21:39
wirehead_So, just because it’s a PetSet doesn’t mean it’s not able to be turned into a crunchy milkshake.21:39
wirehead_If you know what I mean.21:39
harlowja_lol21:39
*** ravig has joined #openstack-kolla21:40
harlowja_what's the best place to read about all the things PetSets will have/do/make them special21:40
wirehead_Heh heh.21:40
wirehead_Find a Kube core at a conference and buy them a drink?21:40
wirehead_They are all heads-down at the moment getting 1.3 out the door.21:41
harlowja_lol21:43
harlowja_hmmmm, fair enough21:43
*** ravig has quit IRC21:44
*** ravig has joined #openstack-kolla21:44
wirehead_Yah, I can’t find any doc patches.21:45
harlowja_:(21:45
harlowja_i'd have liked that, pets seem like they require a bunch of dicussion and agreement on why they are in kube before they get added, hope that kind of discussion happened :-/21:46
harlowja_cause it'd be nice to have plans to make those things not pets before adding them, lol21:46
wirehead_https://github.com/kubernetes/kubernetes/issues/26021:46
wirehead_I’d say so. :D21:46
harlowja_ya, hmmmm, nobody mentioned about fixing zookeeper (as i read comments)21:47
harlowja_or using https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html#sc_reconfig_retrieving21:47
wirehead_I think they call it “zk” most of the time.21:47
wirehead_PetSets weren’t specifically for zk, tho.  So, while the zk-side reconfig changes will make it work better in kube, there’s plenty of other stuff that needs the same abstractions.21:50
harlowja_ya, kick that shit out of kube ;)21:50
harlowja_that fixes the glitch21:50
harlowja_ha21:50
wirehead_Overall, the Kube style is to provide working abstractions to solve categories of problems.21:51
wirehead_This tends to mean that you need to do minor adjustments at times to fit your problems into the abstraction.21:51
Mech422wirehead_: btw - anyone testing atomic with this stuff ?21:53
Mech422wirehead_: thats RH's k8 stuff right ?21:53
wirehead_Maybe rhallisey?21:53
Mech422wait - GKE is not a GCE ?21:53
Mech422I thought you were just typoing...whats GKE ?21:54
wirehead_Google Container Engine, GKE.21:54
wirehead_Google Compute Engine, GCE.21:54
Mech422oh...I see...21:54
* Mech422 blinks21:54
wirehead_GKE runs atop GCE.  Basically, Google runs a HA master for you and your compute nodes just execute your compute traffic.21:54
*** arafferty_ has joined #openstack-kolla21:56
*** mummer has quit IRC21:57
*** inc0 has quit IRC22:02
*** openstackgerrit has quit IRC22:02
*** openstackgerrit has joined #openstack-kolla22:05
*** arafferty_ has quit IRC22:06
*** ravig has quit IRC22:06
*** kimk0316 has quit IRC22:07
*** mummer has joined #openstack-kolla22:07
*** mbound has joined #openstack-kolla22:10
*** dwalsh has quit IRC22:10
openstackgerritKen Wronkiewicz proposed openstack/kolla-kubernetes: Create ConfigMaps with dashes instead of underscores  https://review.openstack.org/33024722:23
*** ravig has joined #openstack-kolla22:25
*** salv-orlando has quit IRC22:25
*** salv-orlando has joined #openstack-kolla22:26
*** vhosakot has quit IRC22:31
esharao__how to get ansible 2.0.. on ubuntu 14.. i keep getting 2.122:32
*** ravig has quit IRC22:35
*** ravig has joined #openstack-kolla22:36
*** ravig has quit IRC22:38
*** ravig has joined #openstack-kolla22:39
*** esharao__ has quit IRC22:40
*** ravig has quit IRC22:47
openstackgerritMerged openstack/kolla-kubernetes: Check for a local kolla directory  https://review.openstack.org/32909022:54
openstackgerritMerged openstack/kolla-kubernetes: Switching Horizon over to work via Kubernetes network  https://review.openstack.org/32920422:54
*** zhiwei has joined #openstack-kolla22:55
kklimondahmm, I've deployed liberty from ubuntu-source, and nova-novncproxy is failing because it's trying to listen on 0.0.0.0:6080, and failing to bind because HAProxy is already listening on that port. I have some extra repositories added to base docker image, but novncproxy is using virtualenv, so that shouldn't cause any issue.22:55
*** sacharya_ has quit IRC22:56
Mech422kklimonda: AIO deploy ?22:56
Mech422kklimonda: I thought I had liberty working...22:57
kklimondano, multi node22:57
Mech422kklimonda: to be honest though, I'm too lazy to go back and check - it was a pain changing stuff to work with ansible 222:57
Mech422kklimonda: I build ubuntu from source, and run multi-node - but I don't recall that issue22:58
Mech422kklimonda: then again, I might have switched to mitaka before I got stuff working22:58
*** zhiwei has quit IRC23:00
kklimondamhm, I'll try deploying tomorrow without my patches and see if perhaps I broke something - didn't touch nova-novncproxy image at all though23:00
*** sacharya has joined #openstack-kolla23:00
*** mummer has quit IRC23:01
*** sacharya_ has joined #openstack-kolla23:02

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!