Thursday, 2016-08-25

sdakeaNupoisc yes - 2.0.3 is unreleased s of yet00:00
aNupoiscor is the latest tag is 2.0.2 with stable/mitika00:00
sdakethe latest release is 2.0.200:00
sdakeyou cn check the docs - they receently changed on install method depending on whether your doing an eval or dev00:00
aNupoiscso if i clone kolla stable/mitika what tag would I get?00:00
sdake2.0.300:00
sdakebut that wont work correctly without hacking00:01
*** dwalsh has quit IRC00:01
sdakeread quickstart - it covers this in detail00:01
*** zhurong has quit IRC00:02
sdakeinc0 init-runonce-osic does the network setup and image loading?00:02
inc0yeah00:02
sdakeanything needed after that ?00:02
inc0should be enough00:02
aNupoiscsdake: ahh that's why. I am getting an error "Tag 2.0.3 not found in repository docker.io/kolla/ubuntu-source-heka" so wondering did I cloned something wrong00:03
sdakei think next ste pfor somoene thta understands the caching layer in ceph is to create a labeling playbook for that00:04
aNupoiscsdake: thanks, I will clone with tag 2.0.200:04
inc0 aNupoisc did you setup your docker registry in globals.yml?00:04
sdakeaNupoisc you cn still work from git if you want00:04
sdakeaNupoisc to do that, set openstack_release: "2.0.2" in globals.yml00:04
aNupoiscinc0: my docker registry is default #docker_registry: "172.16.0.10:4000"00:05
aNupoiscsdake: Okay let me try with that approach00:05
inc0aNupoisc, this is commented;)00:06
sdakeya he wants to pull from the hub00:06
aNupoiscYes.00:07
aNupoiscdo i need to enable this? inc000:07
sdakenot if pulling  from hub00:07
inc0only if you're running local registry00:07
aNupoisci am running on all-in-one environment and i read no need of local registry00:08
sdakeeven multinode you cana pull from hub00:08
sdakeits just slower00:08
*** david-lyle has quit IRC00:08
*** Guest20708 has quit IRC00:08
aNupoiscsdake: inc0: okay so for my case what I am doing is correct00:08
sdakeyup00:08
aNupoisci just need to get kolla of tag 2.0.200:09
sdakeaNupoisc 2.0.3 is compatible with 2.0.2 images00:09
sdakeour z streams are bug fix only00:09
*** dwalsh has joined #openstack-kolla00:10
aNupoiscsdake: But it is throwing an error "Tag 2.0.3 not found in repository docker.io/kolla/ubuntu-source-heka" when i try to deploy00:10
*** bmace has joined #openstack-kolla00:11
*** dcwangmit01_ has quit IRC00:12
*** dcwangmit01 has joined #openstack-kolla00:12
aNupoiscsdake: I am trying to deploy ceph with kolla00:12
aNupoiscso not sure openstack_release=2.0.2 or not?00:13
sdakeceph aio requires specil config00:13
sdakeread thee ceph guide00:13
aNupoiscsdake: yes i am following https://github.com/openstack/kolla/blob/master/doc/ceph-guide.rst#managing-ceph but I am at deploy step00:14
aNupoiscand it is failing at heka bootstrap container00:14
aNupoiscsdake: till now I have installed docker and run prechecks00:15
sdakein globals.yml00:15
sdakedo you have the line00:15
sdakeopenstack_rlelease: "2.0.2"00:15
aNupoiscsdake: nope00:15
sdakepretty sure i mentioned that as a workaround at timestamp 17:04:31 ;)00:15
aNupoiscyeah, you mentioned00:16
aNupoiscsdake: but I was not sure to use openstack_release as I don't want all opestack services as i am deploying ceph00:17
aNupoiscso confirmed00:17
sdakeyou need to use that flag - it wont enable extra services00:17
aNupoiscsdake: so are you saying if i put the option and run deploy then it would work00:17
aNupoiscsdake: okay00:17
sdakeunless you use the enable_service: true or whatever it is00:17
sdakeaNupoisc yup thats right00:17
aNupoiscsdake: no i am enabling just the service that i want. :)00:18
aNupoiscsdake: that's quicker way then cloning again with tag 2.0.2, cool00:18
openstackgerritLi Yingjun proposed openstack/kolla: Add option to support dhcp/l3 agent ha  https://review.openstack.org/35218000:20
*** wally__ has quit IRC00:23
aNupoiscsdake: yeah that worked00:27
aNupoisc:)00:27
aNupoiscsdake: so that means even if i have 2.0.3 it uses images of 2.0.200:28
aNupoiscis that correct00:28
sdakeyup00:28
aNupoiscsdake: but why does not use 2.0.3 images00:28
sdakesee pbr note in quickstart00:28
sdakethere are no 2.0.3 images in the docker hub00:29
sdakebecause it is unreleased software00:29
*** dwalsh has quit IRC00:29
aNupoiscsdake: hmm. interesting00:30
aNupoiscsdake: great to know00:30
*** yingjun has joined #openstack-kolla00:33
sdakeinc0 not runonce00:34
sdakeit has to be in the parent script00:34
inc0no, as we create deployment here00:34
inc0also we need to hack around this deployment crap00:34
sdakeand tempest shoudl be installed00:34
sdakeyou can create a deployment in the parent too00:34
sdakewe need that line there00:36
sdakei want a separate deployment  per rally scenario00:36
sdakefor easier data collection00:36
sdakesource dir i swrong00:40
sdakeinc0 cp -ar00:40
sdakeprobably want a rally verify install in there00:41
sdaketo make sure rally knows tempest is installed00:41
sdakelets clean db00:41
inc0yeah good idea00:42
sdakeand give it a  spin00:42
inc0btw we could just dump sqlite after each iteration00:42
inc0so we'll have all the rally data00:42
sdakerather have the json00:42
sdakethere is a capture script in there too00:42
sdakewhich needs work00:42
inc0verification data is somewhere else00:42
*** duonghq has joined #openstack-kolla00:43
sdakelooks like that worked00:44
inc0yeah00:44
duonghqsdake: OSIC worked?00:44
sdakeduonghq we hae almost all data capture ully automated00:44
sdakethat is what i was referring to00:44
inc0sdake, I'd just copy sqlite and dump stuff later00:44
inc0with htmls and shit00:45
sdakedangerous00:45
sdakei'd rather have json00:45
sdakewe will convert json to rst00:45
inc0archiving whole sqlite as a precaucion00:45
inc0let's archive all of it00:45
sdakeok archive for backup wfm00:45
duonghqsdake: it can run automatically or still somebody must kick it off00:45
sdakeduonghq well yes someone needs to type in one command...00:45
sdakeafter getting the escenario we want00:46
sdakewhich is a little morecomplex ;)00:46
*** ssurana has quit IRC00:47
sdakeneed another grep in those tasks00:47
*** ssurana has joined #openstack-kolla00:47
duonghqroger00:48
sdakeso tired00:48
duonghqit is your shift atm?00:49
inc0sdake, just dump all of it00:51
inc0tomorrow we'll make it better00:51
sdakethat isn't good enough00:51
sdakeoh you mean dump everything00:51
inc0dumping all of them?00:51
inc0yeah00:51
sdakei expect to be on scenario 3 by today00:51
duonghqhmm, gate still is unstable, sad00:52
sdakewe dont have time to wait around until tomorrow00:52
inc0someone else will have to bump up the number00:52
sdakethere are two operations00:52
sdakeone to capture html00:52
sdakeone to capture json00:52
*** ssurana has quit IRC00:52
sdakeinc0 pretty sure the line includes the name which is 128x24, 128x48, or 128x9600:55
sdakethe problem with that copying the lines 3 times is that the output to wrong dir00:56
sdakei guess we could hardcode the output dir too ;)00:56
sdakei need to run to circklee k to grab a soda, bbi1000:56
britthouserNeed ; before do, @inc00:57
britthouserNeed ; before do, inc000:57
britthouseryeah00:57
duonghqhmm, circle-K store become more and more wide spread here00:57
*** gfhellma has quit IRC00:57
*** yingjun has quit IRC00:57
*** yingjun has joined #openstack-kolla00:58
inc0sdake that will be ok01:01
inc0?01:01
duonghqanybody know where is coolsvap? he doesn't active in IRC for awhile?01:03
inc0I haven't seen him for couple days01:03
*** jtriley has joined #openstack-kolla01:03
*** phuongnh has joined #openstack-kolla01:06
*** sdake has quit IRC01:06
*** zhubingbing has joined #openstack-kolla01:06
inc0I hate bash01:11
duonghqirc0: what's wrong01:11
*** caowei has joined #openstack-kolla01:14
*** zhurong has joined #openstack-kolla01:17
inc0I give up.01:20
duonghqinc0: can I help you?01:20
inc0nah, I'll take a look tomorrow01:21
inc0cya guys01:21
inc0thanks duonghq tho01:21
duonghqcya inc001:21
*** inc0 has quit IRC01:21
*** yingjun has quit IRC01:33
*** fguillot has quit IRC01:33
*** yingjun has joined #openstack-kolla01:33
*** zhiwei has joined #openstack-kolla01:34
*** sdake has joined #openstack-kolla01:36
sdakeinc001:37
*** sdake_ has joined #openstack-kolla01:39
*** sdake has quit IRC01:42
*** unicell has quit IRC01:43
*** fguillot has joined #openstack-kolla01:45
*** aNupoisc has quit IRC01:49
*** sdake_ has quit IRC01:54
duonghqsdake_ inc0 take a rest atm01:55
zhubingbing- -02:02
*** yingjun has quit IRC02:02
*** yingjun has joined #openstack-kolla02:03
*** sdake has joined #openstack-kolla02:04
*** jrist has quit IRC02:05
*** zhurong_ has joined #openstack-kolla02:09
duonghqnearly all gate is green, phew... (except bandit)02:10
*** zhurong has quit IRC02:11
*** jrist has joined #openstack-kolla02:18
zhubingbingping hui Kang02:18
*** zhubingbing has quit IRC02:23
*** duonghq has quit IRC02:24
*** serverascode has quit IRC02:28
*** serverascode has joined #openstack-kolla02:30
*** sdake has quit IRC02:34
*** sdake has joined #openstack-kolla02:35
*** mdnadeem has joined #openstack-kolla02:40
*** fguillot has quit IRC02:43
*** mdnadeem_ has quit IRC02:43
*** jamesdenton has joined #openstack-kolla02:44
*** sdake has quit IRC02:45
*** jtriley has quit IRC02:45
*** sdake has joined #openstack-kolla02:47
*** zhubingbing_ has joined #openstack-kolla02:48
*** yuanying has quit IRC02:49
*** yuanying has joined #openstack-kolla02:53
*** duonghq has joined #openstack-kolla02:54
*** daneyon has joined #openstack-kolla02:57
*** daneyon has quit IRC03:02
*** zhubingbing_ has quit IRC03:12
*** sdake has quit IRC03:12
*** zhubingbing_ has joined #openstack-kolla03:15
*** jtriley has joined #openstack-kolla03:17
*** salv-orlando has joined #openstack-kolla03:17
*** salv-orl_ has quit IRC03:20
*** jtriley has quit IRC03:22
*** senk has joined #openstack-kolla03:25
*** zhubingbing_ has quit IRC03:26
*** zhubingbing has joined #openstack-kolla03:27
*** huikang has joined #openstack-kolla03:41
*** senk has quit IRC03:43
*** yuanying has quit IRC03:45
*** yuanying has joined #openstack-kolla03:48
*** coolsvap_ has joined #openstack-kolla03:50
duonghqlong time no sê coolsvap_03:55
duonghqlong time no see03:55
coolsvap_duonghq: approx 16hrs i suppose :)03:55
coolsvap_not feeling well from yesterday03:56
*** zhubingbing has quit IRC03:56
duonghqare you better now?03:56
*** zhubingbing has joined #openstack-kolla03:57
*** zhubingbing has quit IRC03:59
coolsvap_kinda04:04
duonghqanybody got (\"Container command \\'kolla_start\\' not found or does not exist.\" randomly but frequently?04:08
*** yingjun has quit IRC04:11
*** yingjun has joined #openstack-kolla04:11
*** yingjun has quit IRC04:15
*** yingjun has joined #openstack-kolla04:16
*** senk has joined #openstack-kolla04:18
duonghqPavo: are you there?04:20
*** yingjun has quit IRC04:26
*** yingjun has joined #openstack-kolla04:26
openstackgerritHui Kang proposed openstack/kolla: Fix mariadb bootstrap error  https://review.openstack.org/36021804:27
duonghqcoolsvap_: would you mind review my ps: https://review.openstack.org/#/c/359596/04:28
*** yingjun has quit IRC04:31
coolsvap_duonghq: will do04:33
*** unicell has joined #openstack-kolla04:37
*** caowei has quit IRC04:39
*** aswadr_ has joined #openstack-kolla04:41
duonghqthank coolsvap_04:42
*** zhubingbing has joined #openstack-kolla04:42
*** unicell has quit IRC04:45
*** unicell1 has joined #openstack-kolla04:45
*** yingjun has joined #openstack-kolla04:48
*** huikang has quit IRC04:48
*** senk has quit IRC04:54
*** yingjun has quit IRC04:58
*** yingjun has joined #openstack-kolla04:59
*** ayoung has quit IRC05:03
*** yingjun has quit IRC05:03
*** zhubingbing has quit IRC05:13
*** senk has joined #openstack-kolla05:23
openstackgerritTin Lam proposed openstack/kolla: Remove base_distro type 'fedora'  https://review.openstack.org/36024905:25
*** harlowja_at_home has joined #openstack-kolla05:25
*** phuongnh has quit IRC05:27
*** yingjun has joined #openstack-kolla05:37
*** zhubingbing has joined #openstack-kolla05:41
*** david-lyle has joined #openstack-kolla05:43
*** harlowja_at_home has quit IRC05:45
*** david-lyle has quit IRC06:09
*** caowei has joined #openstack-kolla06:10
*** egonzalez90 has joined #openstack-kolla06:18
duonghqthank egonzalez9006:21
*** fragatina has quit IRC06:46
*** Serlex has joined #openstack-kolla06:48
*** salv-orlando has quit IRC06:51
*** Mr_Broken has joined #openstack-kolla06:59
*** senk has quit IRC06:59
*** senk has joined #openstack-kolla07:08
*** zhubingbing has quit IRC07:18
*** zhiwei has quit IRC07:21
*** daneyon has joined #openstack-kolla07:28
*** Mr_Broken has quit IRC07:30
*** Mr_Broken has joined #openstack-kolla07:30
*** bootsha has joined #openstack-kolla07:32
*** daneyon has quit IRC07:33
*** zhubingbing has joined #openstack-kolla07:33
*** Mr_Broke_ has joined #openstack-kolla07:34
*** Mr_Broken has quit IRC07:35
*** Satya has joined #openstack-kolla07:43
SatyaHi All, I have deployed openstack multinode set up using openstack-kolla. When I enable l3 HA and create a router and assign subnet router I am observing the below error07:45
SatyaERROR neutron.agent.l3.agent RuntimeError: Exit code: 99; Stdin: ; Stdout: ; Stderr: /var/lib/kolla/venv/bin/neutron-rootwrap: Unauthorized command: ip netns exec qrouter-90a719b6-c447-4dc7-a1c5-e21e16d9f03f keepalived -P -f /var/lib/neutron/ha_confs/90a719b6-c447-4dc7-a1c5-e21e16d9f03f/keepalived.conf -p /var/lib/neutron/ha_confs/90a719b6-c447-4dc7-a1c5-e21e16d9f03f.pid -r /var/lib/neutron/ha_confs/90a719b6-c447-4dc7-a1c5-e21e107:45
SatyaDoes any one come across this error07:46
*** shardy has joined #openstack-kolla07:47
*** Satya has quit IRC07:55
*** zhubingbing has quit IRC07:56
*** zhubingbing has joined #openstack-kolla07:59
*** chas has joined #openstack-kolla07:59
*** salv-orl_ has joined #openstack-kolla07:59
*** salv-orl_ has quit IRC07:59
*** salv-orl_ has joined #openstack-kolla08:00
*** berendt has joined #openstack-kolla08:00
openstackgerritJeffrey Zhang proposed openstack/kolla: Move to transport_url for rabbitmq configuration  https://review.openstack.org/35644908:08
*** Mr_Broke_ has quit IRC08:08
*** Mr_Broken has joined #openstack-kolla08:08
*** ankush has quit IRC08:09
*** ankush has joined #openstack-kolla08:11
*** Mr_Broke_ has joined #openstack-kolla08:12
*** zhubingbing has quit IRC08:12
*** zhubingbing has joined #openstack-kolla08:12
*** Mr_Broken has quit IRC08:13
*** athomas has joined #openstack-kolla08:15
*** zhubingbing has quit IRC08:16
*** zhubingbing has joined #openstack-kolla08:16
*** senk has quit IRC08:29
*** zhubingbing has quit IRC08:34
*** zhubingbing has joined #openstack-kolla08:36
*** senk has joined #openstack-kolla08:38
*** zhubingbing has quit IRC08:46
*** openstackgerrit has quit IRC08:48
*** zhubingbing has joined #openstack-kolla08:48
*** openstackgerrit has joined #openstack-kolla08:48
berendtDo we have a documentation of our gate setup?08:54
bjologood morning08:56
*** yingjun has quit IRC08:56
*** yingjun has joined #openstack-kolla08:56
*** zhubingbing has quit IRC08:57
*** zhubingbing has joined #openstack-kolla08:58
openstackgerritPaul Bourke proposed openstack/kolla: Set 'balance source' for Horizon in haproxy  https://review.openstack.org/35651409:04
*** bootsha has quit IRC09:07
openstackgerritPaul Bourke proposed openstack/kolla: Import the Murano core libary as part of setup  https://review.openstack.org/36000209:09
*** daneyon has joined #openstack-kolla09:17
*** salv-orlando has joined #openstack-kolla09:18
*** salv-orl_ has quit IRC09:20
*** daneyon has quit IRC09:21
*** mnasiadka has joined #openstack-kolla09:31
Mr_Broke_Hey guys09:32
Mr_Broke_does kolla at somew point run neutron-ovs-cleanup ?09:32
Mr_Broke_I am running into problems after reboot with my dhcp, l3  and metadata agent09:32
Mr_Broke_are you aware of this ? : https://bugs.launchpad.net/cloud-archive/+bug/124970809:32
openstackLaunchpad bug 1249708 in neutron (Ubuntu) "Missing neutron-ovs-cleanup service" [High,Fix released] - Assigned to James Page (james-page)09:32
Mr_Broke_this script should run, when neutron_server starts09:35
*** fragatina has joined #openstack-kolla09:43
*** bootsha has joined #openstack-kolla09:47
*** fragatina has quit IRC09:47
openstackgerritPrithiv proposed openstack/kolla: Dockerfiles for collectd Container  https://review.openstack.org/35863209:53
openstackgerritMerged openstack/kolla: Updated from global requirements  https://review.openstack.org/36023409:53
openstackgerritVladislav Belogrudov proposed openstack/kolla: Add option to support dhcp/l3 agent ha  https://review.openstack.org/35218009:55
*** gfidente|afk is now known as gfidente09:56
*** yingjun has quit IRC09:56
*** zhurong_ has quit IRC10:01
openstackgerritPrithiv proposed openstack/kolla: Dockerfiles for collectd Container  https://review.openstack.org/35863210:01
*** zhubingbing has quit IRC10:02
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet  https://review.openstack.org/34936610:08
openstackgerritVladislav Belogrudov proposed openstack/kolla: Allow Neutron to be run in DVR mode  https://review.openstack.org/35654410:09
*** prithiv has joined #openstack-kolla10:12
openstackgerritShaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet  https://review.openstack.org/35113910:13
openstackgerritShaun Smekel proposed openstack/kolla: Customizations for RabbitMQ  https://review.openstack.org/35440910:16
*** duonghq has quit IRC10:16
openstackgerritShaun Smekel proposed openstack/kolla: Customizations for MariaDB  https://review.openstack.org/35442210:19
openstackgerritShaun Smekel proposed openstack/kolla: Customizations for MariaDB  https://review.openstack.org/35442210:19
*** zhiwei has joined #openstack-kolla10:19
openstackgerritShaun Smekel proposed openstack/kolla: Customizations for Ironic  https://review.openstack.org/35442710:22
*** zhiwei has quit IRC10:24
openstackgerritShaun Smekel proposed openstack/kolla: Fix incorrectly implemented footer blocks  https://review.openstack.org/35819310:26
*** Mr_Broke_ has quit IRC10:27
*** Mr_Broken has joined #openstack-kolla10:27
*** Serlex has quit IRC10:28
*** yingjun has joined #openstack-kolla10:29
*** Mr_Broken has quit IRC10:32
*** Serlex has joined #openstack-kolla10:39
*** prithiv has quit IRC11:04
*** Mr_Broken has joined #openstack-kolla11:04
Mr_Broken@Sarya: looks like a rootwrap problem, have you seen this ? :https://bugs.launchpad.net/kolla/+bug/1599489 ? Other people seem to report issues when deploying from images built from sources11:10
openstackLaunchpad bug 1599489 in kolla "Neutron Open vSwitch Agent hangs because of Sudo" [High,Incomplete] - Assigned to Michał Jastrzębski (inc007)11:10
openstackgerritVladislav Belogrudov proposed openstack/kolla: Allow Neutron to be run in DVR mode  https://review.openstack.org/35654411:13
pbourkeJeffrey4l__: ping11:32
yingjunping pbourke11:36
pbourkeyingjun: hi11:36
yingjunhi pbourke, i answered your comment in this patch: https://review.openstack.org/#/c/358973/11:37
yingjunCould you recheck on that?11:37
pbourkeyingjun: hmm11:38
Mr_Brokenhey guys, according to this https://bugs.launchpad.net/cloud-archive/+bug/1249708 neutron-ovs-cleanup should be run in the event of hosts reboot , just before neutron_server starts11:38
openstackLaunchpad bug 1249708 in neutron (Ubuntu) "Missing neutron-ovs-cleanup service" [High,Fix released] - Assigned to James Page (james-page)11:38
pbourkeyingjun: ok maybe it's best to do it at runtime then, but could we do it in the kolla-toolbox start task rather than ceph?11:39
Mr_BrokenI've looked through kolla sources, and cant find this being used - therefore having issues with my neutron network agents after reboos11:39
Mr_Brokenshould I open a bug in Kolla ? or you guys are aware of that ?11:39
pbourkeMr_Broken: sounds like we need that functionality11:40
pbourkeMr_Broken: i would log a bug11:40
Mr_Brokenkk will do, thanks11:40
*** gbialas has joined #openstack-kolla11:41
yingjunpbourke, i think it could, i’ll try :)11:41
*** mewald has joined #openstack-kolla11:43
*** dwalsh has joined #openstack-kolla11:44
Jeffrey4l__pbourke, pong11:47
pbourkeJeffrey4l__: was just looking again at https://review.openstack.org/#/c/311125/11:48
pbourkeJeffrey4l__: would be great to get that fixed. is it on your radar or would you like me to try submitting an update for it?11:48
pbourkeJeffrey4l__: it looks good if we could avoid the extra site.yml11:48
Jeffrey4l__pbourke, OK. i can handle it.11:49
pbourkeJeffrey4l__: thanks!11:49
Jeffrey4l__thanks for the reminder.11:49
openstackgerritLi Yingjun proposed openstack/kolla: Fix toolbox permission issue when enabling ceph  https://review.openstack.org/35897311:50
yingjunpbourke thanks11:51
pbourkeyingjun: thanks also11:52
openstackgerritMathias proposed openstack/kolla: Added telegraf role  https://review.openstack.org/34644811:53
openstackgerritMathias proposed openstack/kolla: Added influxdb role  https://review.openstack.org/34644911:53
openstackgerritMathias proposed openstack/kolla: Added grafana container  https://review.openstack.org/34644711:53
mewaldWhat do I need to do to solve these "Patch in Merge Conflict" errors in gerrit?11:58
*** dwalsh has quit IRC11:59
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Nova  https://review.openstack.org/35167612:02
egonzalez90mewald, yes is needed because some change was merged in some place your patchset is changing12:13
egonzalez90as example inventory files12:13
*** salv-orlando has quit IRC12:15
*** akwasnie has quit IRC12:18
*** akwasnie has joined #openstack-kolla12:18
*** haplo37 has quit IRC12:20
*** shardy has quit IRC12:21
*** bradjones has quit IRC12:21
*** haplo37 has joined #openstack-kolla12:22
*** shardy has joined #openstack-kolla12:22
*** bootsha has quit IRC12:24
*** peterjenkins has quit IRC12:25
*** bradjones has joined #openstack-kolla12:27
*** bradjones has quit IRC12:27
*** bradjones has joined #openstack-kolla12:27
*** peterjenkins has joined #openstack-kolla12:27
*** bootsha has joined #openstack-kolla12:28
*** JoseMello has joined #openstack-kolla12:35
*** sbezverk_ has joined #openstack-kolla12:37
*** sbezverk has quit IRC12:39
*** dave-mccowan has joined #openstack-kolla12:41
*** rhallisey has joined #openstack-kolla12:41
*** jtriley has joined #openstack-kolla12:54
*** bootsha has quit IRC12:54
*** bootsha has joined #openstack-kolla12:55
*** egonzalez90 has quit IRC12:56
*** zhurong has joined #openstack-kolla12:58
*** salv-orlando has joined #openstack-kolla12:58
*** jrist has quit IRC13:00
*** bootsha has quit IRC13:00
*** fguillot has joined #openstack-kolla13:03
*** dwalsh has joined #openstack-kolla13:04
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Nova  https://review.openstack.org/35167613:07
*** Mr_Broken has quit IRC13:09
*** Mr_Broken has joined #openstack-kolla13:09
openstackgerritsean mooney proposed openstack/kolla: generate bifrost yaml configs  https://review.openstack.org/34968413:10
openstackgerritsean mooney proposed openstack/kolla: add bifrost deploy ansible support  https://review.openstack.org/34968513:10
openstackgerritsean mooney proposed openstack/kolla: introduce bifrost-deploy container  https://review.openstack.org/31970313:10
openstackgerritsean mooney proposed openstack/kolla: adds bifrost ssh key generation  https://review.openstack.org/35069213:10
openstackgerritsean mooney proposed openstack/kolla: add deploy-servers command  https://review.openstack.org/34972913:10
*** Mr_Broke_ has joined #openstack-kolla13:11
*** jrist has joined #openstack-kolla13:13
*** Mr_Broken has quit IRC13:14
*** fguillot_ has joined #openstack-kolla13:15
*** fguillot has quit IRC13:28
*** Satya has joined #openstack-kolla13:28
*** Mr_Broke_ has quit IRC13:30
*** Mr_Broken has joined #openstack-kolla13:30
*** yingjun has quit IRC13:34
*** Mr_Broken has quit IRC13:34
*** lrensing_ has joined #openstack-kolla13:34
*** salv-orlando has quit IRC13:35
*** senk has quit IRC13:38
*** gbialas has quit IRC13:39
*** inc0 has joined #openstack-kolla13:41
*** srwilkers_ has joined #openstack-kolla13:43
*** mgoddard_ has joined #openstack-kolla13:45
*** mgoddard has quit IRC13:48
inc0good morning13:53
mewaldmorning13:53
srwilkers_morning inc013:53
*** rhallisey has quit IRC13:54
*** rhallisey has joined #openstack-kolla13:57
*** mnasiadka has quit IRC13:59
inc0pbourke, around?13:59
pbourkeinc0: hu13:59
pbourke*hi13:59
inc0mind taking a look at https://review.openstack.org/#/c/358042/13:59
inc0oracle linux is failing for me and to me it looks like repo issue...is this correct/incorrect?14:00
*** NachoDuck has quit IRC14:01
*** huikang has joined #openstack-kolla14:02
*** NachoDuck has joined #openstack-kolla14:04
*** mgoddard_ has quit IRC14:04
*** mgoddard has joined #openstack-kolla14:05
*** yingjun has joined #openstack-kolla14:06
pbourkelooking14:10
*** fguillot_ has quit IRC14:16
*** fguillot has joined #openstack-kolla14:19
*** michauds has joined #openstack-kolla14:21
sbezverk_inc0: need your advise. I am chasing a nova bug and folks from nova channel suggested a change in os_vif package. How I could bring cutomized os_vif package into nova-compute image?14:25
inc0customized how?14:25
sbezverk_inc0: line changed in python code14:26
*** prithiv has joined #openstack-kolla14:28
*** berendt has quit IRC14:28
*** chas has quit IRC14:31
*** prithiv has quit IRC14:33
*** ravig has joined #openstack-kolla14:37
inc0hmm14:40
inc0sbezverk_, so you can make use of our customization mechanism14:40
inc0to inject something like sed into build14:41
inc0also this will be trickier with binary build14:41
inc0s14:41
inc0meh14:41
sbezverk_no need binary I am on source14:42
*** sdake has joined #openstack-kolla14:42
inc0yeah but people might be on binary14:43
sdakemorning14:43
sdakepbourke where we at with osic?14:43
inc0who uses it at the end14:43
inc0anyway, worth to try14:43
sdakeinc0 where we aat on osic14:43
inc0best bet - use customizations14:43
inc0rally is running14:43
inc0that's what I'm seeing now;)14:43
sdakefor which scenario?14:43
inc0I have no idea14:43
sdakecan you check the nextsteps file in the /root dir14:44
inc0todays deployment tho14:44
inc0you are last author man:P14:44
inc0anyway we probably killed our cloud14:44
* sdake groans14:44
sdakewhy killed our cloud?14:44
inc0lots of timeouts and shit14:44
sdaketimeouts are ok14:45
inc0I bet that space on disk ran out14:45
inc0how they are ok?14:45
sdakebecause we are running an overload test14:45
inc0hence killed our cloud14:45
sdakei just fnid it hand to believe its been running for 12 hours14:45
inc0as overloaded it14:45
*** senk has joined #openstack-kolla14:46
sdakeso are you saying someone  redeployed the cloud?14:46
inc0no idea14:46
inc0aaaand yeah14:47
inc0we ran out of disk14:47
sdakewhat type of scenario is it on?14:47
inc0told ya we should mount big disk to var/lib/docker14:47
sdakeran out of disk in ceph?14:47
inc0no14:47
inc0elastic14:47
inc0ceph is ok14:47
inc0but on controls / is on 40G partition14:47
inc0and elastic stores a lot...a lot of data14:47
*** ravig has quit IRC14:48
inc0I'll stop this stuff and fix our env if you don't mid14:48
sdakei'd like to know what scenario its on14:48
*** openstackgerrit has quit IRC14:48
sdakerun rally task list | wc -l14:48
*** openstackgerrit has joined #openstack-kolla14:49
inc08114:49
sdakesomehting deinatley wrong it was at 60 when i left 12 hrs ago...14:49
inc0it's new deployment14:49
inc0first task was finished 2hrs agio14:49
inc0so it's not that horrible14:50
pbourkeinc0: the problem is "7server" in http://mirror.centos.org/centos/7Server/virt/x86_64/kvm-common/repodata/repomd.xml14:50
*** yingjun has quit IRC14:50
*** yingjun has joined #openstack-kolla14:50
inc0pbourke, so it's not issue with my code?14:50
*** ravig has joined #openstack-kolla14:50
pbourkeinc0: there's a sed to swap that with '7' but whatever repo provides that one above must be getting missed14:50
sdakewas any datacaptured in scenario-2 directory?14:50
inc0I see14:50
pbourkeinc0: your change seems to be fine but there must be something odd with it as the other patches aren't failing14:50
inc0sdake, onlu what I was doing yesterday14:51
pbourkeinc0: I'll try your change locally to see if I can pin it down14:51
sdakewhy would soemone redeploy the cloud?14:51
inc0as it stands today, it's broken14:52
inc0I'm stopping this crap, fixing env and redeploying14:52
*** yingjun_ has joined #openstack-kolla14:53
pbourkesdake: yeah I just saw rally running and assumed someone was on it14:53
*** yingjun has quit IRC14:53
*** david-lyle has joined #openstack-kolla14:53
*** caowei has quit IRC14:56
britthouserIs it fair to say we should have a '/var/lib/docker is huge' pre-deployment check?14:57
sdakeYES14:57
pbourkeinc0: weird its building locally for me14:57
pbourkebritthouser: its sort of dodging the issue that we're lashing a lot of large data into one partition though14:58
pbourkebritthouser: we have ops who dont like this about kolla but I dont have a good answer yet14:58
sdakepbourke first time i've ever heard the complaint14:59
britthouserIs it basically all the logs go into volume, and all volumes live in /var/lib/docker?14:59
pbourkesdake: yeah i haven't been vocal enough about it14:59
*** david-lyle has quit IRC14:59
pbourkebritthouser: yes14:59
*** JoseMello has quit IRC15:00
sdakeone directory or spread across the system - i don't see the delta15:00
pbourkeits common enough to mount /var/lib/nova/instances on a separate large disk15:01
pbourkehow do I do this with kolla?15:01
pbourkeany dont say use ceph ;)15:01
sdakemount /var/ib/kolla on a large disk15:01
sdakesorry v/ar/lib/docker15:01
pbourkealong with /var/lib/cinder ? and /var/lib/glance ?15:02
sdakeit is true there is no spindle separation of storage to specific disk15:03
sdakebut that is an orthogonal problem to running out of disk space in general15:03
sbezverk_pbourke: second for question?15:04
pbourkeI wondering would it be possible to mount /var/lib/docker/volumes/_data/nova on separate disk for example15:04
pbourkei haven't tried yet15:04
sdakeJeffrey4l__ still here?15:06
britthouserTypically on systems I deploy, /var is my largest partition15:06
Jeffrey4l__sdake, yep15:06
sdakeJeffrey4l__ did you redeploy the cluster last night?15:07
Jeffrey4l__sdake, no..15:07
sdakeok - there is a file clled NEXTSTEPS15:07
sdakeit contians directions for what to do next15:07
Jeffrey4l__sdake, writing the rally script to test the io15:07
sdakewe now have 5 days15:07
Jeffrey4l__yep..15:08
sdakeinc0 if you rerun tests amke sure to wipe out db15:09
*** jamesdenton is now known as Guest150615:11
*** salv-orlando has joined #openstack-kolla15:12
inc0well fuck15:12
inc0I might have broken our controllers15:12
inc0I'm going to redeploy baremetal15:12
inc0it's long overude anyway15:12
sdakehow long will that take?15:13
inc0hour or so15:13
sdakejust deploy xenial then15:13
inc0gonna fix the storage issue when I'm at it15:13
inc0not that easy15:13
inc0I don't have preseed xenial image handy15:13
sdakeif we can skip to xenial before having done any work15:14
sdakewe should do so15:14
sdakeat this point we haven't done any work...15:14
inc0centos is same issue exacrly15:14
sdakeok - well wen eed to sovle those15:14
inc0I don't agree15:14
sdakebritthouser is your man15:14
inc0I'm about to start writing best practices guide15:15
sdaketell me how long it takes to make an image15:15
sdakeif its a 3 day process15:15
inc0lot's of stuff we learned15:15
sdakethen we have problems15:15
sdakemy point is i'd rather have xenial data then 14.04 data15:15
sdakeand at this pace we will not hae time for xenial data15:15
sdakecan i get an estimate on time needed to get xenail deployed?15:17
sdakei realize its more work15:17
*** salv-orl_ has joined #openstack-kolla15:18
sdakeinc0 britthouser ^^15:18
*** vhosakot has joined #openstack-kolla15:19
*** salv-orlando has quit IRC15:20
*** vbel has quit IRC15:22
*** vbel has joined #openstack-kolla15:23
pbourkeso because elasticsearch is full we're tearing down and going ubuntu?15:23
pbourkewhat?15:23
pbourkesdake: ^15:23
sdakepbourke isn't it alrady ubuntu?15:23
pbourkeoh right15:23
sdakei dont get the teardown15:23
pbourkeinc0: why are you tearing down15:23
sdakewhy not just fix the 3 controlers15:23
sdakebut whatever15:23
sdakepbourke he cratered the controllers15:23
pbourkewhy still need to do upgrade/stress test scenarios15:23
inc0pbourke, I ran rm -rf /var/lib on controllers;)15:24
inc0but on the other hand controllers were out of space15:24
inc0and thrown a lot of errors15:24
inc0so I'm going to redeploy whole thing and make controllers disks...bigger15:24
sdakepbourke check out etherpad - i tuned our test sceanrios15:24
sdakebut woudl like estimate on exenail deployment15:25
sdakeif we dont do it now15:25
sdakeits not going to happen at all15:25
britthouserThe import of xenial shouldnt take too long sdake. *but* the risk of going to xenial now is we had to install special driver for the NICs.  Not sure how/if/etc that works in xenial15:25
sdakethese teset scenarios take6-8 hrs to run15:25
sdakebritthouser it wasn't just a kernel update?15:25
pbourkechaosmonkey tests would be useful15:26
pbourkesdake: sure you want to drop those?15:26
sdakeya lets make it harder15:26
sdakepbourke - we have  5 days to run tatleast 2 platforms worth of tests15:26
britthouserI wasn't around for the specifics of what it required to get the NICs to work right15:26
sdakedo the math15:26
britthouserI came back from travel after the fact15:26
britthouserif it was just new kernel, and xenial has kernel, then pobably low risk15:27
sdakebritthouser pretty sure itw as just a ernel update15:27
*** rhallisey_ has joined #openstack-kolla15:28
britthouserOk, then probably lower risk.15:28
britthouserLemme login the cobbler container and work on importing it.15:28
sdakebritthouser  is that something you an hep with ?15:28
britthouseryes15:28
sdakethank god ;)15:28
sdakepbourke sorry for being so short - just runnign out of time trying to reduce our scope as much as possible15:29
pbourkesdake: no prob15:30
pbourkesdake: final question though. benefit of testing on xenial over 14.04 is?15:30
sdakepbourke master onl yruns on xenial15:30
pbourkesdake: thanks15:30
*** fguillot has quit IRC15:32
*** rhallisey has quit IRC15:32
*** egonzalez90 has joined #openstack-kolla15:32
*** ravig has quit IRC15:32
*** huikang has quit IRC15:33
*** ravig has joined #openstack-kolla15:33
sdakebritthouser osic uses a cobbler container?15:35
britthouserYeah...an LXC container with cobbler in it15:35
*** prithiv has joined #openstack-kolla15:35
britthouserIts running on teh deploy node15:36
sdakecool15:36
britthouserroot@729494-comp-s3500-002:~# lxc-ls15:36
britthouserosic-prep15:36
*** v1k0d3n has joined #openstack-kolla15:39
inc0hmm..how did we add first IP for nodes?15:40
britthousertaht was from spreadsheet?15:41
britthouserYou're talking about the 10.15.243.X inc0?15:41
britthouserThe ILO addresses were in the email15:42
britthouserI cut/pasted them into ilo.csv15:42
inc0no, I already deploy first node15:43
britthouserwhich is just a compliation of m9*.txt files15:43
inc0OS15:43
inc0trying to get to them15:43
openstackgerritPaul Bourke proposed openstack/kolla: Import the Murano core libary as part of setup  https://review.openstack.org/36000215:43
britthouserthen I'm not sure what you're asking15:43
v1k0d3nhey rhallisey_ or sdake, have a question as a new guy....15:44
v1k0d3nlooking through blueprints...and this one came up (has my interest): https://blueprints.launchpad.net/kolla/+spec/consul-container15:44
v1k0d3ncould we just use etcd though? how can i make suggestions to the author huikang27?15:44
*** Serlex has quit IRC15:44
rhallisey_v1k0d3n, that bp doesn't look very active15:45
sdakeetcd is already in the review queue15:45
rhallisey_you can probably take it15:45
*** rhallisey_ is now known as rhallisey15:45
sdakethere is already an implemtnation of consul in teh queue as well15:45
v1k0d3ngood. i think etcd meets the request and better aligns.15:45
sdakeits just that we dont see a good need for it15:45
v1k0d3n^^ agreed.15:45
kfox1111v1k0d3n: and depends on if your looking at ansible or k8s. the k8s side is doing service descovery via k8s.15:45
rhalliseyhttps://review.openstack.org/#/c/355156/15:45
v1k0d3nwell, etcd could be used for either, no?15:46
kfox1111it could, but its extranious in k8s as k8s provides dns based service discovery.15:46
v1k0d3nyeah, that review is good. like i said...i'm new to this fulltime role. just taking things in right now.15:46
rhalliseyv1k0d3n, ya I guess we could go with etcd15:46
rhalliseyit's in much better shape15:46
v1k0d3nthat's my thought too.15:47
sdakev1k0d3n i'd recommend having a look at the review queue15:47
sdaketo see whta peopel are working on15:47
*** Serlex has joined #openstack-kolla15:47
v1k0d3nreusable configs/skills if we just go with etcd...not that i'd want to limit options later though.15:47
v1k0d3nok.15:47
v1k0d3nsdake / rhallisey wanted to also ask about something like LKVM or hypervirtualization of the containers. do we have that on the roadmap at all (support some form of hypervirtualization of the containers)?15:48
sdakeyouo mean like intel's clear containers ?15:49
v1k0d3nyes15:49
v1k0d3nincreased security15:49
sdakev1k0d3n this project doesn't have a super roadmap :)15:49
sdakev1k0d3n basiclly people work on what interests them15:49
sdakeand that defines our roadmap15:49
sdakebut i'd liek to see clear containers optionally implemented if its possible15:50
sdakenot sure if that iss the case15:50
v1k0d3ni think it can be done, but the prep would most likely be out of kolla or kolla-kubernetes. at the kolla level, it would be using the flags to support what's been prepared.15:51
sdakedoes clear containers support host bindmounts?15:51
sdakedoes clear containers support host netowrking mode?15:51
*** ravig has quit IRC15:51
sdakeor ipc mode15:51
v1k0d3ni was going to check on this.15:51
sdakeor pid=host15:51
sdakethse are things we need to hve any kind of extra barrier15:52
v1k0d3ni probably need to come up with a list of kolla requirements based on what's in place, and then compare.15:52
v1k0d3nright, agreed.15:52
*** britthouser has quit IRC15:52
v1k0d3nit 'should' support docker: https://clearlinux.org/blogs/clear-containers-docker-engine15:53
sdakebut does it support all docker flags needed15:53
sdakeunknown15:53
v1k0d3nright. i will look into it.15:54
*** britthouser has joined #openstack-kolla15:54
britthouserrut row15:54
v1k0d3nsomething to blueprint maybe?15:54
*** aNupoisc has joined #openstack-kolla15:54
v1k0d3ni'm 1000% new to blueprints. and i don't want to put something out there just to put it out there.15:54
sdakev1k0d3n that is usually how it wors - think of blueprints as a brainstorming area15:55
sdakebritthouser sup problem?15:55
britthouserfreenode disconnect me.  Back now. =)15:55
britthouserI need to get the contents of the ISO into the LXC container.  Does LXC have something equivalent to 'docker copy' ?15:56
sdakelxc may have binddmounts?15:56
sdakedocker copy only copie from insdie to outside the container15:56
britthouserI tried to mount the ISO while in the LXC, and got errors, just like I do with docker.15:57
*** mark-casey has joined #openstack-kolla15:57
sdakeanother option is curl15:57
britthouserhttps://paste.fedoraproject.org/413935/14721406/15:57
britthouserYeah I curl'd the ISO down...or do you mean somethingelse?15:58
*** yingjun_ has quit IRC15:59
*** ravig has joined #openstack-kolla16:05
*** Marcellin_ has joined #openstack-kolla16:05
sdakethats what i meant - curl inside the lxc ontainer16:05
britthouserSure...I can curl down the ISO to host or LXC16:06
sdakebritthouser t oget that to work in docker - you have to bindmount /dev16:06
britthouserbut I need the files inside the ISO16:06
sdakeoh you need to modify the iso?16:06
sdakemodify outside - then curl inside?16:06
britthouserno cobbler just needs to pull files out of it16:06
sdakei see16:06
sdakedoes cobbler use mount to do that?16:07
*** ravig has quit IRC16:07
britthousertypically user does the mount, and then points cobbler at the mount point16:07
sdakea loopback mount?16:07
britthouserright16:07
sdakeis ubuntu mounted atm?16:07
britthouserI think i found a way16:07
sdakeif it just needs the files in the iso, loopback mount on the host, then tar em up then curl the tar in16:08
britthouserFrom host I can mount ISO, and then copy contents into /var/lib/lxc/osic-prep/rootfs/blah16:08
sdakenice16:08
*** mewald has quit IRC16:08
sdakehey folks i am in flagstaff today and have a 2-3 hour travel time back to phoenix around noon pst16:09
jrich523fly like the wind!16:09
britthouser=)16:11
openstackgerritEduardo Gonzalez proposed openstack/kolla: Ansible config for nova-serialproxy console  https://review.openstack.org/35883916:11
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance  https://review.openstack.org/35489516:12
v1k0d3nso steve, if i put a blueprint out there about hypervirtualization of the container, would that work? it may actually be a better fit into kolla-kubernetes rhallisey, since kubernetes will soon (or at somepoint) support a runtime interface for other runtimes.16:13
v1k0d3neven those runtimes can support docker format, like hyperd16:13
sdakeso guys our quickstart gets about 2k page views a month16:13
jrich523not bad16:13
jrich523oh i noticed that the order was... well... problematic16:14
jrich523the python stuff should come frist16:14
jrich523i'll fix it if i can ever get down the whole process16:14
rhalliseysdake, what16:14
jrich523but basically ansible needed a dev package that is part of the python setup16:14
v1k0d3nwow. that's good. sdake how are you getting those stats?16:14
sdakev1k0d3n magic16:14
rhalliseysdake, how did you find that out16:14
sdakedont ask questions you dont want  the answers to16:14
v1k0d3ngoogle analytics? :)16:14
v1k0d3nhahaha16:14
v1k0d3ni use analytics on my blog. really helps because then i know how users got there, and demographical information :)16:15
jrich523i use it on most sites i manage, fairly useful16:15
v1k0d3nit may help understand if they're searching for docker on openstack or visa versa :)16:16
jrich523well if you're at the quickstart, you're there for a reason16:17
jrich523i can imagine you stumble on to kolla, but the quickstart is intentional16:17
*** zhurong has quit IRC16:20
*** michauds has quit IRC16:21
*** michauds has joined #openstack-kolla16:21
openstackgerritEduardo Gonzalez proposed openstack/kolla: Ansible config for nova-serialproxy console  https://review.openstack.org/35883916:22
*** aNupoisc has quit IRC16:23
v1k0d3njrich523: maybe, but not really sure about that. i think there's still some unfortunate misunderstandings about containers/openstack and the use of each.16:23
v1k0d3ni'm realizing this at our kubernetes meetups. there's a lot of misunderstanding out there.16:23
jrich523i know im completely lost 90% of the time, and im a fairly smart guy... some times....16:24
v1k0d3nbut analytics could show new/return traffic. i would imagine a high percentage of return traffic is keenly interested in Openstack on Docker.16:25
v1k0d3njrich523: ha! so true. easy to get lost.16:25
jrich523lol actually a chunk of those hits are me :)16:25
jrich523yesterday i saw 3 browser windows open to the quick start16:25
jrich523not sure why... just too many open i guess lol16:25
*** fguillot has joined #openstack-kolla16:27
britthouserok slight bad news sdake, the cobbler (2.4.1) that came in the container only supports up to trusty16:29
britthousergonna look to see if it easy to make it recognize xenial or not16:31
*** lamt has quit IRC16:31
*** chas has joined #openstack-kolla16:31
*** wally__ has joined #openstack-kolla16:34
wally__hi guys, how can I check all logs like nova log, neutron log,? I login different container, but where is the log16:35
sdakewally__ docker exec heka ls -l /var/log/kolla16:36
sdakebritthouser thats annoying :(16:36
britthousersdake - cobbler 2.6 supports up through Wily16:36
*** chas has quit IRC16:36
sdakeso there is no way to deploy xenial?16:37
britthouserno easy way16:37
britthouserwe can try to upgrade cobbler in the lxc container.16:37
britthouserbut even then it will probalby require some small hacking to get it to recnogize xenial16:37
sdakeis the cobbler container easy to reproduce?16:38
wally__thx got it16:38
sdakeor can it be backed up16:38
openstackgerritPaul Bourke proposed openstack/kolla: Add option ceilometer_database_type for ceilometer  https://review.openstack.org/34588116:39
openstackgerritPaul Bourke proposed openstack/kolla: Allow mysql to be used as a database for Ceilometer  https://review.openstack.org/36066916:39
britthouserthe container we pulled down had cobbler pre-installed.  It was pretty straightforward to launch.16:39
sdakecool lets try upgrding then and hacking it16:39
sdakeby lets i mean yuo :)16:39
britthouserwe addeda  bunch of scripts specific to the topology that we'd want to save off if we had to re-deploy16:39
sdakecan you back up that stuff?16:40
britthouserOk let me first see if 2.6 is even available for the distro the LXC is running16:40
wally__I want to restart horizon apache server, I login to the horizon container, but do not see the apache service16:40
wally__how to restart horizon apache16:40
sdakedocker horizon restart16:40
sdakeor docker restart horizon16:41
wally__ok, thx16:41
britthouseryeah sdake the container is trusty, so its repo only holds cobbler 2.416:42
sdakeis cobbler c or python or what?16:42
britthouserI think to get 2.6 we'd need a xenial container.16:42
britthousermostly python I think?16:43
*** prithiv has quit IRC16:43
sdaketry a pip install of it16:43
sdakebest to make a backup16:43
britthouserCobbler (0.6.3-2)             - Provisioning control application & library for PXE, Virtualization, and Re-Provisioning16:43
pbourkesdake: bit tired today so sorry if Im getting in the way here. But - we've been doing kolla master on top of 14.04 fine up till this point16:46
wally__when I do the kolla delploy, in the global.yaml file , I enable enable_murano: "yes", but when the murano pannel, not showing on the horizon dashboard16:46
pbourkesdake: your earlier statement about why we need 16.04 actually still doesn't make sense to me16:46
sdakepbourke the master containers run xenail now rather then 14.0416:46
*** fragatina has joined #openstack-kolla16:47
pbourkeyes but it runs fine on top of a host 14.0416:47
*** fragatina has quit IRC16:47
sdakehere is the deal16:47
britthouserYeah I feel like we're right on the edge of "this is too risky too late in the game"16:47
sdakeif we ar egoing to do any xenail teting at all16:47
britthouserbut I'm a risk averse person16:47
sdakeit needs to be setup now16:47
sdakeso lets make a choice as a team16:47
sdakeno xenial testing or xenail teseting16:48
*** fragatina has joined #openstack-kolla16:48
pbourkebefore we that we need to ask why we're abandoning 14.04 half way through16:48
pbourke*we ask that16:48
sdakewe aren't abanondineding 14.04 halfway through16:48
pbourkeso we have all scenarios done for 14.04?16:48
sdakewe have no *resULts* for 14.04 as of yet16:48
sdakewe have done no scenarios!16:48
pbourkewhat have we been doing that past few days16:49
sdakebut we ehave a fully automated setup for doing the testing16:49
sdakewe have been automating evreything so that anyone can run the scripts16:49
sdakeand making rally work properly with kolla16:49
irtermitetick tock ... one week left, folks!  ;)  You're welcome, inc016:49
*** fragatina has quit IRC16:49
britthouserWhat if we do this: keep on with 14.04.  I'll stand up a second OSIC-prep container, and just play with getting cobble 2.6 and xenial loaded.  that would mitigate the upgrade risk while letting 14.04 work continue?16:50
sdakebritthouser yes plz16:50
pbourkehacking cobbler at this stage in the game is bad idea16:50
*** fragatina has joined #openstack-kolla16:50
sdakepbourke tend to agree16:50
britthouseryeah I don't want planA to depend on my hacking cobbler. =)16:50
*** david-lyle has joined #openstack-kolla16:50
pbourkethere's no gain in upgrading the host OS16:50
britthouserHappy to make that planB,C,D =)16:50
sdakeperfomrnace deltas16:50
pbourkewe started with 14.04, that's what osic recommend16:50
sdakebritthouser - pleae do a plan b for xenial - we may or may not have time ot get to testing it if you get the osic prep container working16:51
pbourkestick with it, plough through the scenarios16:51
*** aNupoisc has joined #openstack-kolla16:51
inc0yeah, I need to prep ansible inventoryu16:51
inc0nodes are back up16:51
pbourkei honestly thought we were on like #3 as of today before the controllers went down16:51
inc0just ansible doesn't understand this16:51
sdakeinc0 ue same inventory16:51
sdakeit is setup for the node counts i wwant16:51
sdake3 contor 20 storage 100 compute16:51
sdakepbourke we ehave done a bare metal deploy16:52
sdakepbourke we are on ceph + cinder + gathering data16:52
inc0that's what I'm saying, ansible doesn't understand it for whatever reason16:52
sdakepbourke and our deployment crater16:52
sdakeinc0 ok - well whatever you need ot do to get to 3 control 20 stoage 100 comput e;)16:52
sdakei.e scenario #216:53
sdakepbourke all documentd in nextsteps file ;)16:53
sdakebritthouser also if you could get centos sorted out in cobbler16:54
sdakethat is actually more important then xenial at this point16:54
sdakecentos is part of our plan a - and we have basically 3 days per distro16:54
britthouserOk, the cobbler in the current OSIC-prep should be able to import centos7.  I'll do that first.16:55
wally__hi sdake, when I do the kolla delploy, in the global.yaml file , I enable enable_murano: "yes", but when the murano pannel, not showing on the horizon dashboard16:55
sdakewally__ - we dont have plugins directly implemented for services at this time16:55
sdakewally__ but newton will have the ability to do whatever type of plugin you desire ;)16:56
wally__ok i see, so the murano gui not working at this time, right16:56
*** ssurana has joined #openstack-kolla16:56
sdakewally__ - yup i know not ideal for stable branches16:56
sdakewally__ - we are not superpeople :) takes time to develop this stuff16:57
wally__ok, another quesiton, how to restart all different kind of service, like nova-api, neutron, l3, dhcp,, serivce,16:57
sdakepbourke may have some murano gui workaround he can provide you16:57
sdakewally__ our api at present doesn't include a restart option16:57
*** unicell1 has quit IRC16:58
pbourkewally__: we dont ship the dashboard by default unfortuantely16:58
sdakeyou can use docker manually of course t odo that16:58
*** fguillot_ has joined #openstack-kolla16:58
pbourkewally__:  I have a patch but not ready yet. Luckily it's easy to do via templates16:58
*** fguillot_ has quit IRC16:58
*** fguillot has quit IRC16:58
sdakewally__ templates are the thing we implemented in newton to make plugins possible for any scenario16:58
pbourkewally__: create a file called 'template-overrides.j2' and put the following in it http://paste.openstack.org/show/563425/16:59
jrich523how do i install ansible 1.9.4, i thought it was pip install ansible==1.9.4 ,or yum install ansible-1.9.416:59
sdakepbourke wally__ i think is running table16:59
wally__ok, thanks, i will try16:59
sdakejrich523 pip install ansible==1.9.416:59
jrich523crap... thanks16:59
pbourkethen rebuild horizon with --template-overrides template-overrides.j216:59
pbourkewally__: if you're using stable the templates aren't there, but you can put the same code right into the dockerfile (minus the jinja parts)16:59
wally__ok, thanks. I will try,17:00
pbourkewally__: you also need to add a plugin block to kolla-build.conf to clone the repo17:00
*** aNupoisc has quit IRC17:00
pbourkewally__: lot of steps I know :/ PM me if you need more help. but will have to be tomorrow!17:00
wally__okok i will take to you tomorrow17:00
wally__so to restart all the different serfvice, I need to restar the docker, right, like nova, nova-api, neutron,,..l3 dhcp...etc17:01
egonzalez90add {% block horizon_footer %} at the begining of the file17:01
sdakeegonzalez90 he is using stable17:01
*** athomas has quit IRC17:01
*** ravig has joined #openstack-kolla17:01
egonzalez90up sorry17:02
sdakeinc0 when you get the cluster deplyed again please let me know17:02
sdakeso our plan a is as follows17:03
inc0settying up networking now17:03
sdake1) deploy all scenarios on 14.04 2) deploy all scenarios on centos 7 3) hack the osic-prep container to support xenial 4) try to get through xenial testing17:03
sdake3 and 4 are stretch goals here :)17:03
inc0fuck17:04
inc0I messed it up again17:04
sdakei've nuked /var/lib several times during kolla dev17:05
sdakeit happens :(17:05
inc0but I have most of stuff in place now17:05
inc0so I can nuke it up and make it back up in couple mins now17:05
sdakeplease dont nuke the deployment host17:06
sdakei haven't backed it up lately17:06
britthouser^^ this17:06
jrich523lol17:06
sdakeinc0 - need ceph+cinder17:07
sdakeas our rallly test cases are selected for that combo17:07
wally__hi guys, is there a network architecture diagram for the VM in kolla. my VM still have no internet access, and canot ping and cannot ssh into the VM, I just check the l3 log, there are some errors.17:10
*** aNupoisc has joined #openstack-kolla17:10
wally__I think somthing wrong with my network17:10
*** huikang has joined #openstack-kolla17:13
*** pbourke has quit IRC17:13
*** pbourke has joined #openstack-kolla17:13
jrich523sounds like it :)17:17
*** egonzalez90 has quit IRC17:19
*** fguillot has joined #openstack-kolla17:21
inc0well fuck17:21
inc0deploy ndoe ran out of space17:21
*** wally___ has joined #openstack-kolla17:21
*** Satya has quit IRC17:22
*** unicell has joined #openstack-kolla17:22
*** wally__ has quit IRC17:22
*** senk has quit IRC17:23
wally___hi guys, in my dnsmasq.log. I always got this (heka)[heka@kolla neutron]$ tail -f dnsmasq.log  Aug 25 10:22:12 dnsmasq-dhcp[87]: DHCPDISCOVER(tapf44fe0a4-95) 2a:50:bd:00:71:94 no address available Aug 25 10:22:15 dnsmasq-dhcp[87]: DHCPDISCOVER(tapf44fe0a4-95) 2a:50:bd:00:71:94 no address available17:24
wally___anyone know what is wrong?17:24
wally___dhcp now working?17:24
jrich523its looking for a pool of addresses, and you've got none17:24
jrich523basically17:24
wally___u mean when I setup the network ?17:25
wally___for vm?17:25
jrich523im not really sure, i imagine thats the case tho17:25
wally___so because of that, the vm cannot go to internet? is the the reason?17:26
jrich523yeah17:27
jrich523my instances isnt setup, so cant really provided any more info17:27
jrich523you've gotten ahead of me :-/17:27
wally___ok, please setup, man17:28
jrich523im working on it now :)17:28
jrich523after i submit my damn timecard17:28
wally___let me know if your instance can go outside or now, and ping and ssh to the vm17:29
jrich523will do17:29
sdakeinc017:32
sdakeinc0 there is a /backup.tar file17:32
sdakeremove it17:32
inc0I removed all the images besides registry17:32
sdakeinc0 there is a hue ass tarfile that i a bkackup17:32
inc0freed up 50% of space17:32
sdakeokthen allgodo :)17:32
*** chas has joined #openstack-kolla17:33
jrich523feels like friday today17:34
wally___jrich523,. how is your network setup, do u have a network digram?17:34
jrich523well its really flat17:35
jrich523as in, its all in VMWare Workstation with a single subnet for internal/external17:35
wally___how is your netowrk setut , how many nic17:35
jrich523so my setup wont be as useful17:35
jrich523but i'll probably be able to figure out how the IP pool works17:35
jrich523i use two nics, because you have to17:35
wally___how many nodes?17:35
jrich523but each nic is the same subnet17:35
jrich523there will be 2, a controller node (basically all of kolla) and a compute/storage node (to build VMs on)17:36
wally___how many interfaces?17:36
jrich523?17:36
jrich523what sort of interfaces?17:36
wally___how many netowrk interfacce, like eth0 eth1...?17:36
jrich523217:36
jrich523because its a must17:37
jrich523mgmt, and external17:37
wally___can u send me your ifconfig output17:37
jrich523nothing useful t here17:37
jrich523because its flat17:37
jrich523so for example17:37
*** chas has quit IRC17:37
wally___ok,  pleaes17:37
jrich523i use my host IPs as 2*17:37
inc0guys, let's start some flame war17:37
jrich523so the controller is 192.168.11.20, node1, is 192.168.11.2117:37
inc0I just grabbed popcorn and need moral reason for it17:37
wally___so 192.168.11.20 is your eth0 and this is management interfface?17:38
jrich523everything i have is 192.168.11.*17:38
sdakeinc0 i have nthing to complain about atm :)17:38
jrich523yes17:38
jrich523so its 21 and 3117:38
wally___what about eth1? no ip assigned?17:38
jrich523correct (based on our conversation yesterday)17:39
jrich523i forget who told us, but.. yeah.. only assign the mgmt ip17:39
wally___ok , ,my is the same setup,,..,17:39
wally___so you ssh into your server using ssh 192.168.11.20 right17:40
jrich523yes17:40
wally___ok, i got it17:40
jrich523i used bridge mode for the adapters, so my pc is .99, super flat :)17:40
wally___ok17:40
jrich523i mostly need to learn HEAT and other automation type stuff, so i could care less about how the network is setup17:40
jrich523we have a client who wants to use openstack as their main entry point for system managment, but they also use puppet and ansible17:41
*** Jeffrey4l__ has quit IRC17:41
jrich523and they want me to tie it all together17:41
* jrich523 shurgs17:41
wally___i got the same setup,  but my vm cannot go to internet, let me know if your are ok17:41
jrich523ok17:41
jrich523might take an hour or two before im up and running17:42
jrich523got a couple of actual work things to do first17:42
wally___ok17:42
inc0irtermite, btw one node seems fubared17:46
inc0729583-comp-s3700-023.cloud7.osic.rackspace.com not that it hits us badly, but take a look at it17:46
inc0fuck the servers. 16 managed to stand up with ssh17:53
inc0rest did not17:53
inc0nvm, fuck ssh not servers, servers are cool17:55
sdakeinc0 jefrey4l pointd out 23 has a nocarrier on one of its nics17:56
sdakehence why its not in the inventory file ;)17:56
inc0yeah I already moved it out of stuff17:56
inc0just want irtermite to know that it has issues17:56
inc0as I'm sure he'd be interested;)17:56
*** dwalsh has quit IRC17:57
*** dwalsh has joined #openstack-kolla18:00
*** david-lyle has quit IRC18:00
inc0ok, sdake wanna see Seans playbook in action?18:00
*** senk has joined #openstack-kolla18:02
sdakeinc0 yes18:02
sdakemoment let me switch vpns18:02
*** Serlex has quit IRC18:03
*** rhallisey has quit IRC18:03
*** sdake has quit IRC18:03
*** huikang has quit IRC18:04
*** david-lyle has joined #openstack-kolla18:04
*** sdake has joined #openstack-kolla18:04
*** huikang has joined #openstack-kolla18:04
*** lamt has joined #openstack-kolla18:05
*** huikang_ has joined #openstack-kolla18:06
*** kbaegis has joined #openstack-kolla18:08
kbaegisAfternoon18:08
*** huikang has quit IRC18:09
*** david-lyle has quit IRC18:11
jrich523howdy kbaegis18:12
*** david-lyle has joined #openstack-kolla18:12
*** kbyrne has quit IRC18:13
*** david-lyle_ has joined #openstack-kolla18:15
*** david-lyle_ has quit IRC18:15
*** mewald has joined #openstack-kolla18:15
*** kbyrne has joined #openstack-kolla18:16
*** fragatina has quit IRC18:17
*** aswadr_ has quit IRC18:21
*** senk has quit IRC18:21
sdakeinc0 that was a kernel update no?18:28
inc0#yeah18:28
inc0cool, now I start to chat with # in irc18:28
inc0tmux broken me18:28
sdakeare we waiting on a reboot18:29
inc0correct18:31
*** rhallisey has joined #openstack-kolla18:33
sdakeinc0 how long does this take18:37
*** salv-orl_ has quit IRC18:37
inc0no clue, put time on it18:37
inc0so we'll have benchmark of it18:37
sdakenever run it before18:37
sdake?18:37
inc0it was merged 3 days ago...18:38
inc0or so18:38
inc0not on 100+ nodes18:38
sdakelooks jammed up to me18:38
inc0let's wait, it's including stuff in loop18:38
inc0that's good learning too btw18:39
inc0btw rhallisey wirehead_ we have kolla-k8s talk18:39
inc0so one woot for kolla k8s;)18:39
wirehead_Just saw.18:39
inc0woot18:39
wirehead_woot18:39
sdakeinc0 our other talk gte rejected?18:40
inc0general kolla - yeah18:40
rhalliseywow18:41
wirehead_My cynical prediction is that Barcelona will be a Kubernetes conference with an OpenStack sideshow.18:42
wirehead_Because that's what OpenStack Days SV kinda devolved into. :/18:43
*** srwilkers_ has quit IRC18:43
*** lrensing_ has quit IRC18:44
*** wally___ has quit IRC18:44
sdakeinc0 two paths - repave the way you hae in the past - or use the playbook18:45
inc0nah, I make notes on what to improve on playbook18:45
inc0it's actually important learining18:45
sdakeyes - but it can be done later not now ;)18:45
inc0yeah this part is suboptimal, we need to fix it18:49
*** salv-orlando has joined #openstack-kolla18:51
*** wally__ has joined #openstack-kolla18:52
sdakewhy did the playbook implode?18:52
inc0not sure18:52
wally__hi guys, I just check my neutron agent-list,  Metadata agent     | kolla |                   | xxx   | True  L3 agent           | kolla | nova              | xxx   | True DHCP agent         | kolla | nova              | xxx   | True18:53
wally__these service not start, how to start the docker container?18:53
*** chas has joined #openstack-kolla18:54
sdakescroll u plets find source of that problem18:55
sdakesince we a re now debugging the host playbook...18:55
*** srwilkers_ has joined #openstack-kolla18:55
sdakeinc018:55
sdakei think its an undeifned variable buty ou move around too fast for my screen size18:55
wally__cannot start the docker start neutron_metadata_agent..Error response from daemon: linux mounts: Path /run/netns is mounted on /run but it is not a shared mount. Error: failed to start containers: neutron_metadata_agent18:56
wally__any idea?18:56
sdakewally__ you hve to read the quickstart dude ;)18:56
*** lrensing_ has joined #openstack-kolla18:57
sdakethis playbook isn't idempotent18:57
inc0it is, I just messed it up18:57
sdakekolla_ssh_key18:58
sdakejust disable the ssh key instlalation18:58
sdakekolla_ssh-key and whatnot go in passwords.yml18:58
wally__ok i gotit18:58
*** chas has quit IRC18:59
*** dwalsh has quit IRC18:59
inc0vim is hard19:01
*** fragatina has joined #openstack-kolla19:02
sdakeinc0 did ou time it?19:02
inc0yeah, but it wont be that relevant now19:02
inc0I changed it19:03
*** ccesario has joined #openstack-kolla19:03
sdakegive us a rough idea19:03
inc0yeah after I'll upstream these changes19:03
inc0so it will be close to what we have now19:03
sdakewonder why python-dev and libssl-dev get installed19:04
inc0ok, we should be ready to deploy19:08
sdakeso what went wrong with the playbook there?19:08
sdakeok in meeting  atm, can you deploy19:09
*** lrensing_ has quit IRC19:09
*** david-lyle has quit IRC19:10
sdakebare metal config should be under 15 mins i think19:10
inc01. it has loop with it to fill up hostfiles19:10
inc0and it is O^2 currently19:11
inc0we need to make it just for controller nodes19:11
inc0other than that it's pretty much ok19:11
sdakethat was't the only problem19:11
inc0oh and this first task is strange19:11
inc0well, I messed up dpkg by breaking it mid-work19:12
inc0so that was partially my bad19:12
*** dwalsh has joined #openstack-kolla19:15
*** jheroux has joined #openstack-kolla19:17
sdakeuse lei's labeling script dude19:17
inc0that's not it19:18
inc0something is fucked up on networking layer19:18
inc0yay19:18
*** zhiwei has joined #openstack-kolla19:20
*** salv-orlando has quit IRC19:21
kbaegisYay!19:22
jrich523YaY!19:22
dasminc0: maybe networking problems are similar to this? http://lists.openstack.org/pipermail/openstack-dev/2016-August/102211.html19:24
inc0dasm, no, it's low lever19:24
inc0bonding kind of stuff19:24
dasmmhm19:24
inc0britthouser, you there on tmux?19:24
*** zhiwei has quit IRC19:24
britthouseroh..no I closed it out19:24
britthouserlemme login19:24
jrich523wally__: did you setup a docker registry?19:24
wally__no19:25
britthouserok I'm there inc019:25
jrich523you have two nodes tho right?19:25
inc0sooo...how did we break this britthouser ?19:25
*** rhallisey has quit IRC19:25
wally__i am useing all ini one, now, just to make sure i have everything right19:25
*** rhallisey has joined #openstack-kolla19:26
sdakebond-slaves none looks suspicious19:26
sdakebritthouser inc0 and I are struggling after a repave19:26
*** ravig has quit IRC19:26
jrich523ahh ok, the all-in-one is a bit different, not easy to add compute nodes.. cuz... its obviously no longer all in one :)19:26
britthouserthis is the deploy node?19:26
sdakeit has something to do with bond0 showing no carrier19:26
sdakenot sure - wasn't driving19:26
wally__yeah i know, i just want to try first, then I will go to multinode19:27
britthousermodprobe bonding ?19:27
inc0well damn, I did that!19:27
britthouser=)19:28
britthouserI still can't cat /proc/net/bonding19:28
britthouserwhich is odd19:28
inc0it works19:29
*** srwilkers_ has quit IRC19:29
britthouserok. =)19:29
sdakecheck ip a again19:29
inc0redeploying19:30
sdakedont you need to relabel the disks?19:30
*** srwilkers_ has joined #openstack-kolla19:31
sdakebritthouser its still busted19:33
inc0take it for few minutes, gotta take some air;)19:34
inc0get some air*19:34
inc0and swear a lot19:34
sdakei dont know anything about ubuntu19:35
sdakei am pretyt sure the slaves are not being assigned tho from that config file19:35
wally__hi guys, I got some error in the neutron-l3-agent.log , can anyone please have a look for me,http://pastebin.com/nbxPp24t19:36
wally__i got this error when i create a subnet for demo, and create demo-router, and assign the demo subnet interface to the demo router19:37
britthousersdake are these problems on teh deploy node, or all the other ndoes?19:38
sdakeother nodes i think19:38
sdakei am not realy sure19:38
britthouserTehre isa  $ on the end of that p1p1 interface19:38
britthouserthat doesn't look right19:38
britthouserin the right hand tab19:39
britthousergot it19:39
sdakethat was me typoing19:39
sdakeon the right is the deply node19:39
sdakeon the left is the node not working19:39
britthouserlooks right.  lemme drive a secon19:39
britthousercan I reboot it/19:42
sdakefree free19:42
sdakefeel free19:42
sdakepokey reboot times19:44
britthouseris it kickstarting again?19:45
sdakeno idea19:45
sdakekeep trying to ssh in19:45
*** huikang_ has quit IRC19:45
-openstackstatus- NOTICE: The Gerrit service on review.openstack.org is restarting to implement some performance tuning adjustments, and should return to working order momentarily.19:46
britthouserusually when you can ping but no ssh, its b/c its kickstarting19:46
sdakeis there nay wy to tell for sure?19:46
*** huikang has joined #openstack-kolla19:46
britthouserconsole19:46
britthouserwhat was the name of that node?19:46
sdake .2319:46
britthouser729517-comp-s3500-023 ?19:47
sdakehttp://729537-comp-s3500-047.cloud7.osic.rackspace.com/19:47
sdakewithout the http part i think19:48
*** jheroux has quit IRC19:48
sdakeatleast thtat is what the email says....19:48
sdakeok we are in19:49
sdakeok so no bonding was loaded on reboot i modprobe bonding bonding0 comes up19:50
sdakebut in a down state19:50
britthouserweird....the bonding driver is loaded19:51
*** huikang has quit IRC19:51
britthouserbut its not not configuring the bond according tot eh config file19:51
*** jheroux has joined #openstack-kolla19:51
*** srwilkers_ has quit IRC19:51
sdakethat would be my impression19:51
*** huikang has joined #openstack-kolla19:53
*** dwalsh has quit IRC19:54
*** chas has joined #openstack-kolla19:55
* britthouser scratches my head19:59
*** jheroux has quit IRC19:59
*** chas has quit IRC20:00
britthousergotta meeting...bbiab20:02
*** sdake has quit IRC20:03
*** jheroux has joined #openstack-kolla20:03
*** sdake has joined #openstack-kolla20:04
*** dwalsh has joined #openstack-kolla20:04
sdakehttps://help.ubuntu.com/community/UbuntuBonding20:07
britthouserahh...good call sdake20:09
*** rhallisey has quit IRC20:09
britthouserneed that ifenslave20:09
sdakeright20:09
sdakenow its doing something else annoying20:09
sdakenice20:15
*** lamt has quit IRC20:15
*** Marcellin_ has quit IRC20:17
wally__hi anyone know what the issue is http://pastebin.com/nbxPp24t20:18
*** mewald has quit IRC20:19
*** lamt has joined #openstack-kolla20:21
*** rhallisey has joined #openstack-kolla20:28
*** sdake has quit IRC20:29
*** jtriley has quit IRC20:34
*** sdake has joined #openstack-kolla20:35
*** JoseMello has joined #openstack-kolla20:38
*** dwalsh has quit IRC20:40
*** aNupoisc has quit IRC20:44
*** salv-orlando has joined #openstack-kolla20:46
*** senk has joined #openstack-kolla20:47
*** aNupoisc has joined #openstack-kolla20:49
*** senk has quit IRC21:00
*** sdake_ has joined #openstack-kolla21:05
*** sdake has quit IRC21:08
*** salv-orl_ has joined #openstack-kolla21:17
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Architectural guide doc  https://review.openstack.org/36079621:18
*** salv-orlando has quit IRC21:19
*** sdake_ has quit IRC21:20
*** sdake has joined #openstack-kolla21:22
inc0sdake, can you take rally from there21:23
inc0I'm out of juice today21:23
inc0btw take a look at this awesome doc I submitted21:23
inc0https://review.openstack.org/36079621:23
sdakei am not sure where its at at preent21:23
sdakedid the deploy work?21:23
inc0yeah21:23
sdakeso should be able to run script and off itgoes/21:24
inc0I'm running init runonce now21:24
*** david-lyle has joined #openstack-kolla21:26
sdakeok i'm out for a bit as well - traveling on the road21:28
sdakelooks like its ticking along21:28
Pavoawesome doc inc021:28
Pavoexactly what is needed21:29
inc0:)21:29
Pavoclears alot of the networking questioned I had up21:29
inc0yeah also storage driver for docker seems to make people confused to say the least21:30
inc0the storage part - we actually ran out of storage on osic cluster21:30
irtermitewell, why'd you go and do that, inc021:31
irtermite?21:31
irtermite  ;)21:31
inc0do what?21:31
*** david-lyle has quit IRC21:31
sdakeinc0 run out of space21:32
sdakeirtermite because someone thought 40g was plenty for kolla..21:32
irtermite40g? you have a LOT more than that there21:33
sdakewhich is fine for the entire os and kolla on any node but a control node ;-)21:33
inc0as it turns out 130 nodes of openstack does produce rather significant amount of logs21:33
inc0irtermite, yeah, but we didnt mount it in correct place;)21:33
sdakeirtermite / was 40g21:33
irtermitehahahah21:33
sdakeirtermite /var/lob/docker was in /21:33
irtermitelogrotate21:33
sdakeyup we need to look into that for elasticsearch21:34
sdakewe already have fs log rotation21:34
irtermiteboy, that sure sucks. do you have to start over, or are you going to dump and roll forward? You guys only have until Tuesday left.21:34
inc0irtermite, issue is elasticsearch and kibana21:34
sdakeright - so we reloaded21:34
sdakescenario 2 is running  of 12 ;)21:34
sdakebut everything is automated21:34
inc0irtermite, deployment of fresh openstack is 20min, today we repaved it from scratch21:34
irtermitewow21:35
inc0not that horrible21:35
sdakeso assuming someone is around to run the script and do he setup of the scenario, it takes about 20 mins to do that21:35
irtermitenice21:35
irtermitewell done guys21:35
inc0once you get hang of it21:35
sdakestill i  am concerned about the amount of time available21:36
sdakebecause scenario run time is 6-8 hrs21:36
inc0if we'll be nice and community open - we're open, not sure about nice part21:36
inc0we might ask OSIC for another installment somewhere in future:)21:37
inc0irtermite, does osic have rules against asking for it twice?21:37
inc0if we provide good reason to ofc21:37
sdakenow it would take much less time that we have figured out how to operate in osic ;-)21:37
sdakenow that we21:37
irtermiteinc0: nope... but it won't be consecutive, nor is it guaranteed to be approved.21:37
sdakeits all good21:38
inc0yeah bulk of today was dealing with 14.0421:38
sdakethis is a great great gift irtermite - appreciate it alot21:38
irtermitesdake: you can thank Intel and Rackspace. I merely manage the deal.  :)21:38
sdakethis will be the first public scale testing doucment i've heard of of openstack being published21:38
irtermitefor kolla that is, sdake21:39
sdakeya not sure of any others21:39
sdakenot saying there isn't21:39
sdakejust don't knowof any21:39
irtermitethere are plenty of other white papers that are coming out of OSIC access21:39
sdakecool21:39
sdakei dont know everything ;)21:40
irtermiteeveryone who uses these environments is expected to upstream commit and to contribute a writeup21:40
irtermitethat is part of the agreement. nothing is free   ;)21:40
sdakewe have a review in work that will be going on docs.oo21:40
sdakeso i think that is satisified21:41
irtermiteI say "expected," but... it really is actually "strongly suggested" as part of the acceptance criteria.21:41
sdakeonce merged of course21:41
sdakeonce filled with data ;)21:41
irtermitedon' worry... inc0 will be bugged quite often by his co-workers and me for the docs21:41
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Architectural guide doc  https://review.openstack.org/36079621:42
*** huikang has quit IRC21:42
inc0irtermite, couple of learinings have jsut been published in patch above21:42
inc0feel free to review;)21:43
*** huikang has joined #openstack-kolla21:43
irtermiteYou know Marketing won't read that inc021:43
*** jtriley has joined #openstack-kolla21:43
inc0irtermite, for marketing we have this https://www.youtube.com/watch?v=RNZMtym5x1c21:44
inc0caution, slavic accent included21:44
irtermiteThey are going to want a PDF as well, I'm sure of it.21:44
sdaket-28 minutes is where it climaxes21:45
inc0.rst is best they can get21:45
inc0sdake, when deploy is finished21:45
sdakepretty sure rst converts to pdf21:45
*** huikang_ has joined #openstack-kolla21:45
*** huikang has quit IRC21:45
irtermiteYea, this .rst looks like a technical review of work. I am sure they are going to want business case outputs as well along with this when it is done.21:46
inc0sdake, btw mind reviewing my doc patch?21:46
inc0irtermite, hey, openstack from naught to running in 20min, that's a business case output right there!21:47
sdakeirtermite - not sure the broader community can help with business case outputs21:47
irtermiteoh, it does sdake... but, it's more of the content I am talking about. There is one that is about to be posted from one of the last users. When it is, give it a look to give you an idea of what I am talking about.21:47
irtermiteIt is... so make a pretty graph of it, comparing to other methods, inc0 that is what they will be looking for21:48
irtermiteControl, another example, and your example.21:48
inc0we'll figure it out21:48
inc0I'll keep leaving next revisions on your desk21:48
sdakeirtermite would you mind defining for me "business case output"21:48
irtermitesdake: What was your testing to prove? What did it confirm? Give comparisons to other options. Etc etc.21:49
irtermiteSaying it deployed in 20 min needs to be compared to another deployment for baseline and to actually prove something.21:50
sdakenot interested in comparing kolla to other tools21:50
sdakeour testing is to prove that kolla is fast and works properly at scale as I promised originally 3 years ago when I signed up for the PTL gig21:51
irtermitethen what is the use-case here?21:51
inc0comparing to anytning else will cause a flame war we don't want21:51
inc0we will just shovel up data21:51
irtermiteBut "fast" relative to what? I'm preparing you for the feedback that will come.21:51
openstackgerritTin Lam proposed openstack/kolla: Add ansible role for vmtp container  https://review.openstack.org/35830521:52
inc0let people compare themself21:52
sdakefast compared to manual install21:52
sdakelet people judge that baseline on their own21:52
sdakenever mention your competition in your marketing ;-)21:53
sdakei hear mirantis getting shot at osic gear with fuel21:54
sdakethey can provide a comparison if they like -or users of the tools can21:54
*** chas has joined #openstack-kolla21:56
*** shardy has quit IRC21:57
*** jtriley has quit IRC21:59
*** chas has quit IRC22:01
irtermitesdake OSIC Project is about community, not competition. Baselines are indeed useful and can be positioned to not be viewed as slander. If the compared group feels they are being stomped on, that is another matter.22:02
sdakeperhaps a future osic study by a group of deployment tools could be a baseline on deployment times from nothing to running22:03
aNupoiscHi all, my deploy with ceph is failing at fetching ceph keyrings task and when i run docker exec ceph_mon fetch_ceph_keys.py on terminal  i see it return "failed":"false" in the json. Should it have "changed" in the json22:03
aNupoiscthat gets returned?22:03
sdakeour philosophy specificlaly states our objective:22:05
sdakeirtermite : Kolla has an objective to replace the inflexible, painful, resource-intensive deployment process of OpenStack with a flexible, painless, inexpensive deployment process. Often to deploy OpenStack at the 100+ node scale small businesses may require means building a team of OpenStack professionals to maintain and manage the OpenStack deployment. Finding people experienced in OpenStack deployment is very difficult and expensive, resulting in22:05
sdake a big barrier for OpenStack adoption. Kolla seeks to remedy this set of problems by simplifying the deployment process but enabling flexible deployment models.22:05
*** michauds has quit IRC22:05
irtermitebasically, what I am saying is... whether you are comparing to another tool or not, the proposal for the kolla testing on OSIC was to test time of deployment and scaling beyond what was previously available. Even if you had metrics that showed how quickly kolla deployed before any work that was done here, or how it performed before throwing a larger node count at it; that would be good for your baseline and t22:05
irtermiteest case results.22:05
aNupoiscthe error I am getting is this http://pastebin.com/rXhP8gAf. When I checked distribute_keyrings.yml I see it is failing on docker exec22:06
sdakeya i can compare to my 3 node cluster I use typically :)22:06
*** fguillot has quit IRC22:10
sdakeinc0 27 rally tasks already run22:13
sdakeout of about 40022:13
*** lamt has quit IRC22:16
aNupoiscany idea about the error http://pastebin.com/rXhP8gAf22:17
kbaegisinc0:  lol yeah, that'll be a lot of logs.  :) how many gigs you generate?22:22
*** vhosakot has quit IRC22:23
inc0kbaegis, after a day around 30+gig22:23
kbaegisSounds about right.  I had 3/day with a 5 node deployment, so makes sense.22:24
*** sdake has quit IRC22:24
kbaegisI was also hammering it with requests though22:24
inc0so are we22:25
inc0stress testing 130node cloud22:25
kbaegis30 gb of logs22:25
kbaegisWow22:25
*** jheroux has quit IRC22:26
kbaegisDid you grep for any shakespearean clauses?22:26
inc0shakespearean? Like "you will all die in horrible way" ?22:26
kbaegisYou might find some quotes from the tempest in there22:26
kbaegisLike in enough monkeys, infinite time kind of way22:27
inc0ahh, this kind, no22:28
inc0I don't believe this theory22:28
inc0existance of internet proven it wrong22:28
kbaegislol Yeah, but we've only had 40 years or so22:28
kbaegisThink of all the great things said on twitter and youtube comments.  ;)22:29
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance  https://review.openstack.org/35489522:29
inc0but trends looks like it goes more towards sexist and racist flame wars than poetic22:29
*** ntpttr has quit IRC22:31
kbaegisiambi pent-up-meter?22:31
kbaegisOkay, that was a stretch :)22:32
inc0I do not have slighest idea what you're talking about22:33
kbaegisQuipping iambic pentameter.  Ignore me.  :)22:33
*** salv-orl_ has quit IRC22:34
kbaegisIs the testing going well?  Any unforeseen reason not to use kolla at-scale?22:35
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance  https://review.openstack.org/35489522:36
*** ntpttr has joined #openstack-kolla22:36
*** sdake has joined #openstack-kolla22:37
sdakethats 10 terabytes a year inc0 - we need to work on gc ;)22:39
kbaegissdake logrotate?22:41
sdakekbaegis nah its elasticsearch that is storing the logs22:41
inc0rather something for elastic22:41
sdakeaNupoisc thats easy enough to fix22:43
sdakeaNupoisc make your authorized_keys file match your public key in ssh22:44
kbaegisHey, is there a convenient way to deploy on a different ssh port?  I know ansible supports it22:44
sdakekbaegis dont know22:45
kbaegis<— Making things difficult for himself :)22:45
*** Guest1506 is now known as jamesdenton22:45
kbaegisI found that changing from 22, 99% of your spam brute-forcing at the edge disappears22:45
*** Pavo has quit IRC22:46
sdakeits the 1% i'd be worried about in that case - as they had to run a port scan  to determine your ssh port to brute force22:47
kbaegissdake Yeah, that's true.  Portknockers help there22:48
kbaegisOr ids22:48
*** daneyon has joined #openstack-kolla22:49
kbaegisI just disable password logins.  Unless there's a 0-day against openssl, I'm probably fine.  If they want to use one of those, well I have tcpdump.  :)22:49
*** Pavo has joined #openstack-kolla22:50
aNupoiscsdake: the private key that i am using matches with the public key in authorized_keys22:50
sdakeauthorized_keys = public key contents?22:51
sdakeand mode on .ssh is 70022:51
sdakeand mdoe on all files in .ssh dir is 600?22:51
*** daneyon has quit IRC22:53
sdakeaNupoisc you will probably need  to run that as sudo for the moment22:54
sdakeaNupoisc and make sure your authorized_keys in ~root/.ssh/authorized_keys = public key of private key your using22:54
*** rhallisey has quit IRC22:57
*** huikang_ has quit IRC23:00
aNupoiscsdake: yes the modes are correct but let me check the modes and all under root23:01
*** Pavo has quit IRC23:08
*** Pavo has joined #openstack-kolla23:08
*** JoseMello has quit IRC23:20
*** wally__ has quit IRC23:20
aNupoiscsdake: so the authorized_keys is matching public key of private key i am using but what i noticed is that my authorized_keys is owned by different user than root23:30
aNupoiscsdake: will that cause a problem23:30
*** wally__ has joined #openstack-kolla23:30
wally__hi guys, I am just wondering it use ovs-bridge in the neutron config , right?23:32
sdakeaNupoisc yes that will not work if sshing in via root23:33
sdakewally__ yes ovs or liuxbridge is selectable23:33
wally__bridge_mappings = physnet1:br-ex23:33
wally__how to select linuxbridge when do the deploy23:33
wally__by default it use ovs, right23:33
*** inc0 has quit IRC23:34
aNupoisc_wally: if you are using devstack then in local.conf you have to specift Q_type=linuxbridge23:34
wally__i am deploying all in one23:34
wally__in the global.yaml, can I set to linux bridge?23:35
wally__if I use ovs to deploy then, my server need to create ovs bridge "br-ex" before doing the deploy, right ?23:37
aNupoiscwally__: http://docs.openstack.org/developer/devstack/guides/neutron.html#using-linux-bridge-instead-of-open-vswitch23:38
aNupoischere is the pointer23:39
aNupoiscwally__: i am not sure what option should be used when using globals.yml23:40
wally__thx i got it, but I dont use devstack, then how to deploy it witll linux bridged, with kollo-ansible deploy23:40
aNupoiscsdake: it seems some different problem because even I changed the ownership still it is giving the same error. Is this correct output for docker exec ceph_mon fetch_ceph_keys.py correct? http://pastebin.com/WrRxu7vM23:42
jrich523ok.. using the quickstart, multinode... prechecks were happy, went to deploy and i got this23:43
jrich523The requested image does not exist: 192.168.11.20:4000/kolla/centos-binary-heka:2.0.223:43
wally__well, I can use ovs , so I need to create ovs br-ex, is that correct?23:44
jrich523shouldnt it auto populate the registry?23:44
aNupoiscjrich523: I believe you have to run docker that populates registry23:46
aNupoiscjrich523: I am not 100% sure I am also playing with kolla23:47
*** kbaegis has quit IRC23:47
wally__hi jrich523, how is your deployment23:48
sdakejrich523 is your registry (192.168.11.20) accessible?23:51
jrich523great question23:51
jrich523i wasnt really able to find a way to test that23:51
*** david-lyle has joined #openstack-kolla23:52
sdakerun docker ps -ef | grep docker on the deploy target23:52
sdakeand paste output23:52
jrich523hmm so its telling me it doesnt know what e is23:53
sdakejrich523 did you build images?23:53
jrich523docker ps works fine23:53
sdakesorry ps -ef | grep docker23:53
jrich523k23:54
jrich523also, as far as the build, i did the pip install of kolla, based on the quick start, didnt seem like i had to build it?23:54
sdakeyou hae to build images if you want to use your own registry23:54
jrich523ps output: http://paste.openstack.org/show/563697/23:54
*** zhurong has joined #openstack-kolla23:54
sdakewe have nothing at present that takes the docker hub and replicates it to your local docker registry23:55
sdakedockerd needs the --insecure-registry flag as well23:55
jrich523ok so that was something i asked the other day23:55
jrich523thats my node01, but ctrl01 has the docker settings for the registry (hosting it)23:55
sdakeyes the deploy target should have --insecure-registry flag23:56
jrich523so basically every node needs that done manually23:56
sdakeand you need to build images (which will psuh them to hte registry)23:56
jrich523weird that it says it looked there tho?23:56
jrich523wait nvm23:56
jrich523ok23:56
sdakejrich523 your globals.yml has that address in teh egistry location23:57
jrich523so for the build, just kolla build?23:57
jrich523it does23:57
sdakeif you just want to try multinode without a registry23:57
sdakejust comment out the registry option23:57
sdakeand it will pull from docker hub23:57
jrich523no user info tho, that was a tad murky to me, didnt see any specific info on setting security so i assume its just open based on the directions?23:57
*** chas has joined #openstack-kolla23:57
jrich523well i did all this work to get the registry setup, i might as well use it now :)23:57
sdaketo secure docker registry - you have to use rls23:57
sdakeok - to use the registry23:57
jrich523well i dont want to, just making sure i dont need to add any creds in globals23:58
sdakekolla-build --type source --base ubuntu23:58
sdakeor if on centos23:58
sdakekolla-build --type source --base ubuntu23:58
sdakeyou need more options too23:58
sdake--registry (the ip of your registry):400023:58
wally__hi sdake, so I need to create ovs bridge br-ex before doing the deployment?23:58
sdakeand --push23:58
sdakewally__ dont think so - pretty sure kolla does that for you23:58
jrich523kolla-build --registry registry_ip_address:registry_ip_port --push23:59
jrich523based on the dev build in quickstart ^23:59
sdakejrich don't forget the build types23:59
wally__but after the dopoymnet, I check the ifconfig on my sever, i do not see the bridge23:59
jrich523so put it all togther23:59
jrich523yeah23:59
sdakethat will build centos binary23:59
sdakewally__ not sure - ask sbezverk_23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!