Friday, 2016-08-05

sdakesbezverk ready for webex00:00
sdakebut if you need to jet totally understand00:00
sdakemy wie is ready t ofire me too00:00
*** haplo37__ has quit IRC00:02
sdakesbezverk ack on foreground state00:03
*** sdake_ has joined #openstack-kolla00:06
*** zhurong has quit IRC00:07
*** daneyon has joined #openstack-kolla00:08
kfox1111sbezverk: the idea is to run the screen command via exec to see what's wrong with the screen command.00:08
kfox1111the while sleep thing just lets you keep the container running long enough to do the screen.00:08
*** sdake has quit IRC00:09
*** daneyon has quit IRC00:12
sbezverksdake kfox1111 https://cisco.webex.com/join/sbezverk00:20
sbezverkbut in 30 minutes I am gone :-)00:20
kfox1111I gota head out in like 10 min. :/00:21
sbezverkgood :-) let's call it a day, we can do it tomorrow00:21
sbezverkI filed a bug with kuberenetes00:22
kfox1111k.00:22
sbezverkmaybe somebody get back to me with some ideas00:22
kfox1111+1. :)00:22
kfox1111it might be interesting to see if adding [ -f /configmap_file ] || echo configmap file not there; real command00:23
kfox1111does anything.00:23
sdake_kfox1111 you can't use exec - exec is a bash operation00:24
sdake_we couldn't get bash -c to work00:25
kfox1111no, I mean docker exec.00:25
kfox1111you run a dummy command in the container so it will start and stay up,00:25
sdake_oh thats not helpful because it only happens during startup00:25
kfox1111then you docker exec in, and try the real command you want to test there.00:25
sdake_we tried that00:25
sdake_worked every time00:26
kfox1111that makes sure you have the right command line figurd out to put in the docker command:00:26
kfox1111but did you try that with the screen command line above?00:26
sdake_not sure how that helps?00:26
sdake_what we tried is sleep infinity00:26
sdake_followed by a docker exec00:26
kfox1111justo make sure screen isn't the thing crashing too.00:27
sdake_screen was a new technique to try to replicate what we were doing with bash -c which didn't work because of json parsing errors00:27
kfox1111and all the screen command line swithces are correct.00:27
sdake_the issue is screen doesn't run in foreground mode00:27
sdake_there is probably a way to make it do so00:28
kfox1111ah.00:28
sdake_that is what is needed :)00:28
sdake_since it backgrounds, the container exits immediately00:28
kfox1111wait $!? :)00:28
sdake_if we could do two commands we wouldn't have a problem00:29
sdake_the problem is we can't do two commands00:29
kfox1111ah.00:29
sdake_kill -9 1 doesn't work00:29
sdake_(inside a container)00:29
sdake_or outside kill -9 PIDOFCONTAINERSPID100:29
kfox1111maybe put the commands in configmap and execute it from there as a bash script?00:29
kfox1111or does that munge the json too?00:30
sdake_a pretty easy solution is to put a  bash script in that container that doe smultiple things00:30
sdake_so pid 1 isn't openvswitch-db00:30
sdake_and then run that bash script fro mthe daemonset00:31
kfox1111yeah.00:31
kfox1111that works too.00:31
dcwangmit01sbezverk: Add this to the container spec, and it will pause the container on startup.  "    command: [ "/bin/bash", "-c", "while :; do echo 'Hit CTRL+C'; sleep 100; done" ]" .  Then, you can "kubectl exec -it <pod> -c <container> bash" and then manually kick off the container startup scripts00:32
dcwangmit01^ I've only glanced at this conversation, so i may be off topic :)00:33
sdake_dcwangmit01 that doesn't work as i've said 3 or 4 imes :)00:39
*** banix has quit IRC00:43
*** rhallisey has joined #openstack-kolla00:43
*** diogogmt has quit IRC00:46
*** zhiwei has joined #openstack-kolla00:48
*** phuongnh has joined #openstack-kolla00:49
dcwangmit01Damn.  It totally works for me, in the general sense.  You may have a different container, without bash or something00:53
dcwangmit01I've used this to debug mariadb, as well as mariadb-bootstrap00:54
dcwangmit01spec:00:54
dcwangmit01  containers:00:54
dcwangmit01  - image: kollaglue/centos-binary-mariadb:2.0.000:54
dcwangmit01    name: mariadb00:54
dcwangmit01    command: [ "/bin/bash", "-c", "while :; do echo 'Hit CTRL+C'; sleep 100; done" ]00:54
dcwangmit01^ I got from wirehead_ , so I know he uses it too00:55
dcwangmit01Debugging containers on kubernetes is hard :)00:56
*** dcwangmit01 has quit IRC01:06
*** ayoung has joined #openstack-kolla01:08
sbezverkdcwangmit01 the problem, this command is in json file01:10
sbezverkthe one we stick into config map01:10
sdake_sbezverk spend time with wife - dont want to get wifefired :)01:10
sbezverk:-) gone gone gone01:11
*** zhurong has joined #openstack-kolla01:16
*** zhurong has quit IRC01:21
*** zhurong has joined #openstack-kolla01:22
*** daneyon has joined #openstack-kolla01:56
*** dcwangmit01 has joined #openstack-kolla01:59
*** daneyon has quit IRC02:01
*** sdake has joined #openstack-kolla02:03
*** sdake_ has quit IRC02:04
*** rhallisey has quit IRC02:05
*** fragatina has joined #openstack-kolla02:12
*** sdake_ has joined #openstack-kolla02:14
openstackgerritMatthew Montgomery proposed openstack/kolla: Customizations for keepalived  https://review.openstack.org/34703002:16
*** sdake has quit IRC02:17
*** dave-mcc_ has joined #openstack-kolla02:17
*** dave-mccowan has quit IRC02:17
*** unicell has quit IRC02:25
*** fragatina has quit IRC02:27
*** zhurong_ has joined #openstack-kolla02:37
*** zhurong has quit IRC02:39
*** yuanying has quit IRC02:49
openstackgerritMerged openstack/kolla: Add customizations for keystone  https://review.openstack.org/34973602:50
*** banix has joined #openstack-kolla02:56
*** dave-mcc_ has quit IRC02:59
britthouserLooks like the automation for retrieving MAC addresses got the wrong address for 729534-comp-s3500-050.cloud7.osic.rackspace.com03:11
*** unicell has joined #openstack-kolla03:13
openstackgerritMatthew Montgomery proposed openstack/kolla: Customizations for keepalived  https://review.openstack.org/34703003:16
britthouserDitto for 729564-comp-s3700-042.cloud7.osic.rackspace.com03:20
*** unicell has joined #openstack-kolla03:21
openstackgerritMerged openstack/kolla: Enable central logging without deploying elasticsearch/kibana  https://review.openstack.org/32001203:23
openstackgerritMerged openstack/kolla: Customizations for mongodb  https://review.openstack.org/34707303:23
openstackgerritMerged openstack/kolla: Updated from global requirements  https://review.openstack.org/35089203:24
openstackgerritMerged openstack/kolla: Change workers to be static when using kubernates  https://review.openstack.org/35035603:24
sdake_down to 8 pages of reviews03:25
*** unicell has quit IRC03:26
*** ayoung is now known as ayoung_ZZZzzz03:26
britthouserGo sdake_!03:27
sdake_i'm wiped out03:27
sdake_time for bed03:27
sdake_night03:27
britthousersame here!03:27
britthousergnight!03:27
*** chip_ has joined #openstack-kolla03:29
*** chip_ has quit IRC03:30
*** unicell has joined #openstack-kolla03:31
openstackgerritMerged openstack/kolla: Cleanup images as part of kolla-ansible cleanup  https://review.openstack.org/34286003:34
openstackgerritMerged openstack/kolla: Fix the kolla_docker issue with docker 1.12  https://review.openstack.org/34935403:34
*** unicell has quit IRC03:36
openstackgerritMerged openstack/kolla: Customizations for ceilometer  https://review.openstack.org/34700503:39
*** sdake_ has quit IRC03:42
*** yuanying has joined #openstack-kolla03:48
*** dwalsh has joined #openstack-kolla03:54
*** sdake has joined #openstack-kolla03:57
openstackgerritzhubingbing proposed openstack/kolla: Add aodh role  https://review.openstack.org/35102703:57
*** zhubingbing has joined #openstack-kolla04:03
zhubingbinghi04:03
*** banix has quit IRC04:04
*** dwalsh has quit IRC04:12
*** sdake has quit IRC04:14
*** chip_ has joined #openstack-kolla04:16
*** dwalsh has joined #openstack-kolla04:17
*** chip_ has quit IRC04:22
*** chip_ has joined #openstack-kolla04:23
*** sdake has joined #openstack-kolla04:24
*** chip_ has quit IRC04:25
*** chip_ has joined #openstack-kolla04:25
*** fragatina has joined #openstack-kolla04:27
sdakehey zhubingbing04:30
zhubingbingsdake, i have a probleme ,can you help me04:31
sdakeshoot04:31
*** fragatina has quit IRC04:31
zhubingbingADD openstack-base-archive /openstack-base-source04:31
zhubingbingI didn't see the openstack-package04:32
sdakefor which?04:32
sdakeit gets created by the build tools04:32
sdakefor which contianer04:32
sdakeopenstack-base-archive is created by the mapping file for container versions04:32
zhubingbingo,build tools04:32
zhubingbing-i know04:32
zhubingbingI went to see04:33
*** unicell has joined #openstack-kolla04:34
zhubingbingi find it ,SOURCES = {04:36
zhubingbing    'openstack-base':04:36
sdakeright04:36
zhubingbingin common/config.py04:37
zhubingbingthinks04:37
*** dwalsh has quit IRC04:38
sdakethinks or thanks?04:38
*** unicell has quit IRC04:39
*** daneyon has joined #openstack-kolla04:39
zhubingbingi'm fighting for tomorrow to finish Barbican and container04:39
zhubingbingsadake,i think monitoring is important, after all, I want to join zabbix in this, how do you feel04:40
zhubingbingthanks ---04:40
zhubingbingI often call the wrong word04:41
sdakeatm we have a tig stack in the queue04:41
sdakeare you familiar with tig?04:41
zhubingbingyes04:42
zhubingbingI'm quite familiar with zabbix04:42
zhubingbingI have done zabbix in this area of practice.04:43
sdakehow is zabbix different from tig04:43
zhubingbingand magnum i think I can fix it.04:44
*** daneyon has quit IRC04:44
zhubingbingdifferent?04:45
zhubingbingI think zabbix has a lot of enterprises in use04:46
zhubingbingsorry , I'm a little bit tig stack ,i don't know what it means.04:47
zhubingbing- - -04:47
sdaketig = telegraph/influxdb/grafana04:48
zhubingbingOh, I see04:48
zhubingbingthanks04:49
sdakeour current thinking is we are not limiting choice on monitoring tools04:51
*** yee_ has joined #openstack-kolla04:52
zhubingbingi also think so, i want kolla to be able to support more monitoring tools04:52
*** yee_ has quit IRC04:55
*** dwalsh has joined #openstack-kolla04:57
sdakewhat i'd like to understand is how zabbix integreates or odesn't integreate with tig04:59
*** unicell has joined #openstack-kolla04:59
*** chip_ has quit IRC05:02
sdakecould you answer i fyou know or indicate you dont know?05:02
zhubingbingI don't know much about for tig05:02
zhubingbingI dont' know --05:03
sdakecool thanks05:03
zhubingbingGrafana also support zabbix05:06
sdakesupport in what way05:06
sdakezabbix layer below or layer above?05:06
zhubingbinglayer05:08
zhubingbinghttps://github.com/alexanderzobnin/grafana-zabbix05:08
zhubingbinglink05:08
*** dwalsh has quit IRC05:13
*** dwalsh has joined #openstack-kolla05:14
sdakei'd prefer not to cross-integrate anything for newton05:20
sdakei think it just makes thigns more complicated and risks tig at minimum not working05:21
sdakeor zabbix for that matter05:21
zhubingbingok , i agree05:24
zhubingbingsdake, i changed it https://review.openstack.org/#/c/351027/705:27
patchbotzhubingbing: patch 351027 - kolla - Add aodh role05:27
*** egonzalez has joined #openstack-kolla05:29
*** egonzalez has quit IRC05:29
sdakeJeffrey4l around?05:36
Jeffrey4lsdake, yep05:37
sdakeany word on ubuntu gate?05:37
sdakeJeffrey4l - note the bare metal is deployed in the osic cluster05:37
sdakewhat isn't deployed is the dependencies05:38
sdakeare the dependnecies that is05:38
Jeffrey4lsdake, no :( i can not even reproduce it locally.05:40
Jeffrey4lthat's cool05:40
Jeffrey4lsdake, ubuntu gate issue?05:40
Jeffrey4labout the osic, how the usage time is assign to every one?05:40
sdakeJeffrey4l we haven't sorted that out yet05:40
sdakeJeffrey4l inc0 floated the idea of geo coordinators05:41
sdakeso, apac, us, emea are geos05:41
openstackgerritEduardo Gonzalez proposed openstack/kolla: Customizations for magnum  https://review.openstack.org/35149205:41
sdakeand someone from eachgeo would be responsible for coordinating activities for our plan for the next 3 weeks05:41
Jeffrey4lyes. that's will be high efficiency05:42
*** yee_ has joined #openstack-kolla05:43
openstackgerritEduardo Gonzalez proposed openstack/kolla: Customizations for magnum  https://review.openstack.org/35149205:44
openstackgerritEduardo Gonzalez proposed openstack/kolla: Customizations for magnum  https://review.openstack.org/35149205:45
sdakeJeffrey4l can you review https://review.openstack.org/#/c/347030/305:47
patchbotsdake: patch 347030 - kolla - Customizations for keepalived05:47
*** yee_ has quit IRC05:48
Jeffrey4ldone05:49
sdakeJeffrey4l also review that last customization patch if you wouldn't mind but don't workflow until gate passes05:49
sdakeJeffrey4l our review queue is 8 pages long05:49
*** yee_ has joined #openstack-kolla05:49
sdakewe need to make progress on reducing its length05:49
sdakewhere i could really use your help the most is anything that is starred05:50
Jeffrey4lno problem05:50
sdakealso customizations since those are easy reviews05:50
sdakethe starred ones are highly complex05:51
sdakefor thte most part mostof them look pretty good to me05:51
Jeffrey4l\i have PSs ready for review. but some of them haven't a +105:52
sdakei reviewed about 40  patches tonight05:52
sdakebut review fatigue set in05:52
sdakeJeffrey4l if you need some reviews link em and ill tak a look05:53
Jeffrey4lcomplex PS consume energy..05:53
sdakeright05:53
Jeffrey4lsdake, https://review.openstack.org/34919705:54
openstackgerritMerged openstack/kolla: Customizations for keepalived  https://review.openstack.org/34703005:54
Jeffrey4lsdake, https://review.openstack.org/34838205:54
Jeffrey4lsdake, https://review.openstack.org/34843205:54
Jeffrey4lthere three. not that complex :)05:55
sdakeany more?06:02
sdakethat last one should probably have a release note on second thought06:02
sdakeso going to -1 it06:02
*** yee_ has quit IRC06:04
sdakeJeffrey4l this one would be nice to merge - if its correct - so we can easily set up our deps on the osic gear06:04
Jeffrey4lgot. thanks.06:05
*** yee_ has joined #openstack-kolla06:05
Jeffrey4lothers is more complicate.06:05
Jeffrey4lhttps://review.openstack.org/34640006:05
Jeffrey4lhttps://review.openstack.org/34760606:05
Jeffrey4lhttps://review.openstack.org/34644406:06
Jeffrey4lhttps://review.openstack.org/34575106:06
Jeffrey4lhttps://review.openstack.org/34745706:06
Jeffrey4lhttps://review.openstack.org/349764  this one is simple, without it openstack can not support live-migration. but it may be danger to bind vnc to all interface06:08
*** yee_ has quit IRC06:08
*** hieulq_ has joined #openstack-kolla06:08
*** yee_ has joined #openstack-kolla06:08
Jeffrey4lsdake, btw, what tool are we used to install the osic machine? bifrost? or?06:11
sdakeat present it was done via cobbler06:11
sdakewe are reserving the last week of testing for bifrost scale testing - hope to have bifrost mrged by then06:12
Jeffrey4lroger06:12
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the kolla_docker issue with docker 1.12  https://review.openstack.org/35149806:12
sdakeif you see a starred review with -1 on it already pleasae also review it again for correctness06:12
sdakewe need  to beat bifrost code into mergable shape in next couple of weeks06:13
Jeffrey4lOK.06:13
sdakemore eyes = better06:13
sdakeone reviewer has hard time picking out all problems06:13
Jeffrey4lyes. I will test it recently.06:14
*** coolsvap has joined #openstack-kolla06:14
sdakehey coolsvap06:15
sdakecoolsvap review queue is long - need reviews if possible06:16
sdakeespecially starred reviews - if you see a starred review with a -1 on it, please also review it again and see if you can pick out any other problems06:16
sdakecoolsvap also osic cluster is deployed06:18
sdakecoolsvap in terms of the bare metal - but no dependencies and no kolla06:18
coolsvapsdake: sure will do reviews06:19
coolsvapmy laptop was busted yesterday had to reconfigure everything06:19
coolsvapjsut getting started06:19
sdakeya both my laptops are damaged06:21
sdakeone has a completely tanked keyboard06:21
sdakeother has a screen breakage06:21
sdakemy productivity is not high as a result :(06:21
sdakeJeffrey4l http://logs.openstack.org/27/351027/7/check/gate-kolla-dsvm-deploy-centos-binary/679bed3/console.html#_2016-08-05_05_46_49_14035706:24
*** yee_ has quit IRC06:24
sdakeJeffrey4l i recently merged a patch which enabled external elasticsearch - it was passing the gate iirc06:24
sdakebut it is possible it wasn't06:24
Jeffrey4li am checking it.06:25
sdakeif our gate is bust we need to fix it asap06:25
sdakeeasiest solution is a revert if that is the root cause06:25
sdakelet me see if i can find the patch which may have caused it moment06:25
sdakehttps://review.openstack.org/#/c/320012/06:26
patchbotsdake: patch 320012 - kolla - Enable central logging without deploying elasticse... (MERGED)06:26
sdakeguess it wsn't passing the gate06:26
sdakeguess i need to not be merging patches on fumes06:26
openstackgerritzhubingbing proposed openstack/kolla: Add barbican dockerfile  https://review.openstack.org/35150606:26
*** yee_ has joined #openstack-kolla06:26
Jeffrey4lOK. let me take a look.06:27
*** daneyon has joined #openstack-kolla06:27
openstackgerritSteven Dake proposed openstack/kolla: Revert "Enable central logging without deploying elasticsearch/kibana"  https://review.openstack.org/35150706:28
*** phuongnh has quit IRC06:29
Jeffrey4lsdake, i think  "{{ 'yes' if enable_central_logging | bool else 'no' }}" may fix this.06:30
*** zhubingbing_ has joined #openstack-kolla06:30
sdakeok well its your idea - can you fix it and try - we can merge if that works - or revert if not06:30
*** phuongnh has joined #openstack-kolla06:30
sdakeneed to do this now06:30
sdakeone or the other06:30
sdakegate breakage = everything grinds to halt06:30
*** daneyon has quit IRC06:31
openstackgerritzhubingbing proposed openstack/kolla: Add aodh role  https://review.openstack.org/35102706:32
*** zhubingbing has quit IRC06:33
*** mewald has joined #openstack-kolla06:38
*** dmueller-evoila has joined #openstack-kolla06:41
*** yee_ has quit IRC06:43
*** yee_ has joined #openstack-kolla06:43
*** zhubingbing__ has joined #openstack-kolla06:47
*** zhubingbing_ has quit IRC06:50
*** duonghq has joined #openstack-kolla06:52
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the evaluating conditional for enable es and kibana  https://review.openstack.org/35151806:54
*** Mr_Broken has joined #openstack-kolla06:55
*** mnasiadka has joined #openstack-kolla06:59
*** dwalsh has quit IRC07:00
*** belmoreira has joined #openstack-kolla07:03
openstackgerritMerged openstack/kolla: Fix the kolla_docker issue with docker 1.12  https://review.openstack.org/35149807:05
*** phuongnh has quit IRC07:05
sdakeJeffrey4l your patch is faliing the gate07:21
*** b_bezak has joined #openstack-kolla07:22
*** zhubingbing__ has quit IRC07:22
Jeffrey4lsdake, which one?07:22
sdakethe one that just merged07:23
sdakehttp://logs.openstack.org/98/351498/1/check/gate-kolla-dsvm-deploy-centos-binary/98ce29d/console.html#_2016-08-05_06_44_32_85892907:23
openstackgerritMerged openstack/kolla: Correct mistral-executor image name  https://review.openstack.org/35063807:25
sdakethat patch is definately the problem - see https://review.openstack.org/#/c/351507/07:25
patchbotsdake: patch 351507 - kolla - Revert "Enable central logging without deploying e...07:25
Jeffrey4lhmm is it related? about the publicURL.07:26
sdakeJeffrey4l oh your patch to fix the ansible code hasn't gone through ci yet07:26
sdakemy mistake07:26
Jeffrey4lthis should come from a oslo.cache bug.07:27
Jeffrey4lbut i remember i have a workaround fix.07:27
Jeffrey4lsdake, my fix is on the way. wait for minute.07:28
sdakehttps://review.openstack.org/35151807:28
sdakethis one?07:28
Jeffrey4lyes.07:29
*** godleon has quit IRC07:29
*** gfidente has joined #openstack-kolla07:32
*** flaper87 has joined #openstack-kolla07:36
*** flaper87 is now known as Guest7354807:36
sdakeJeffrey4l that syntax on that ansible is funky07:40
sdakei guess it was only tested in teh "yes" mode07:40
*** Mr_Broken has quit IRC07:44
Jeffrey4l`else` should be used07:44
*** Mr_Broken has joined #openstack-kolla07:44
Jeffrey4lchecked from zuul. seems my fix works too.07:45
sdakecoolsvap I'll +2 it and head to bed - if it passes the gate feel free to workflow it07:46
coolsvap Sure07:48
sdakeyou have else in the code?07:48
sdakeoh sorry ws reading wrong part of screen :)07:48
*** shardy has joined #openstack-kolla07:55
*** openstackgerrit has quit IRC08:03
*** openstackgerrit has joined #openstack-kolla08:03
Jeffrey4lcoolsvap, it green again. https://review.openstack.org/35151808:03
coolsvapJeffrey4l: yeah sorr ywas away for lunch08:05
Jeffrey4lthanks :)08:08
*** shardy has quit IRC08:14
*** shardy has joined #openstack-kolla08:14
*** yee_ has quit IRC08:15
*** daneyon has joined #openstack-kolla08:15
*** daneyon has quit IRC08:20
*** yee_ has joined #openstack-kolla08:20
*** yee_ has quit IRC08:22
*** yee_ has joined #openstack-kolla08:25
*** yee_ has quit IRC08:27
*** yee_ has joined #openstack-kolla08:30
*** yee_ has quit IRC08:34
*** yee_ has joined #openstack-kolla08:35
openstackgerritMerged openstack/kolla: Volume specification for ironic-api was invalid.  https://review.openstack.org/35120808:37
*** sdake has quit IRC08:37
openstackgerritMerged openstack/kolla: Customizations for iscsid  https://review.openstack.org/35131908:38
*** Guest73548 is now known as flaper8708:39
*** flaper87 is now known as Guest7864108:39
*** wrouesnel1 has joined #openstack-kolla08:41
*** yee_ has quit IRC08:42
*** yee_ has joined #openstack-kolla08:43
*** Guest78641 has quit IRC08:44
*** wrouesnel1 has quit IRC08:52
*** kbaegis has joined #openstack-kolla08:58
openstackgerritMerged openstack/kolla: Added influxdb container  https://review.openstack.org/34644608:59
kbaegisI keep having kolla-build fail09:01
kbaegisMy host is telling me that I can't access a public url. For Centos and ubuntu both is percona,09:02
kbaegisCan't access: INFO:kolla.image.build.base:Cannot open: http://repo.percona.com/release/7/RPMS/x86_64/percona-release-0.1-3.noarch.rpm. Skipping09:02
kbaegisAnyone seen this issue before?  I can curl the url from the docker host09:02
kbaegisIs this just some dumb networking setup I'm not familiar with w/ docker?09:02
openstackgerritHiroki Ito proposed openstack/kolla: Prechecks fails when using multinode deploy using a single node and haproxy disabled  https://review.openstack.org/35158809:05
pbourkekbaegis: are you building behind a proxy?09:11
*** hieulq has quit IRC09:12
kbaegispbourke I'm building on a linux box one hop from my edge09:19
*** anbehl has joined #openstack-kolla09:19
kbaegisNo proxy09:19
kbaegis(My 1st kolla build) :)09:19
kbaegisMost of them succeeded in my ubuntu vm09:20
openstackgerritMerged openstack/kolla: Add doc8 test and improve rst syntax  https://review.openstack.org/35117309:20
openstackgerritMerged openstack/kolla: Added neutron-metering-agent  https://review.openstack.org/34662909:20
kbaegisI can make whatever networking changes are necessary.  The docker default subnet isn't routed by my edge. It's pulling 172.17.0.0/24 addresses on all hosts09:21
kbaegisoh09:21
kbaegisTaking a closer look that's 172.17/1609:21
*** Mr_Broken has quit IRC09:21
kbaegisStill, that's taken on the edge09:21
kbaegis<— obviously not as familiar with docker networking09:22
*** Mr_Broken has joined #openstack-kolla09:22
kbaegiscan I get kolla to use a different range of IPs?  Setting a prefix on my -p (ex. 192.168.20.30:4000:5000) is what gets my registry network accessiblity09:23
kbaegisI'll keep reading :)09:28
*** zhiwei has quit IRC09:31
*** flaper87 has joined #openstack-kolla09:40
*** flaper87 has quit IRC09:40
*** flaper87 has joined #openstack-kolla09:40
*** yee_ has quit IRC09:50
kbaegisresolved.  I needed —bip appended to the daemon on startup and a route out09:56
*** daneyon has joined #openstack-kolla10:04
*** daneyon has quit IRC10:08
*** duonghq has left #openstack-kolla10:09
*** zhurong_ has quit IRC10:10
*** hieulq_ has quit IRC10:11
*** Mr_Broken has quit IRC10:25
*** kbaegis has quit IRC10:42
*** dasTor has quit IRC10:43
*** kbaegis has joined #openstack-kolla10:49
*** yee_ has joined #openstack-kolla10:50
*** yee_ has quit IRC10:54
*** rhallisey has joined #openstack-kolla11:07
*** Mr_Broken has joined #openstack-kolla11:09
*** ad_rien_ has joined #openstack-kolla11:19
ad_rien_HI all,11:19
*** mliima has joined #openstack-kolla11:20
ad_rien_I'm wondering whether it is possible to deploy several compute containers per host without using the fake driver. In other words, I'm trying to understand why putting several times the same hostname in the multinodes inventory (in the compute group) does not enable ansible to create several instances.11:21
kbaegisad_rien_ once nova is up, it should be able to create a large number of guests11:23
kbaegisad_rien_ you're trying to artificially simulate having a large number of chassis or something?11:24
ad_rien_yepp11:24
kbaegisWell, I don't work with the kolla team. I'm just an openstack guy11:24
kbaegisBut I'd say the easiest way is baremetal->vm->docker11:24
kbaegisand then use vms to compartmentalize11:24
kbaegisdocker just uses cgroups11:25
kbaegismuch more lightweight, but no dedicated resource compartments11:25
ad_rien_we thought about such an approach (using vagrant) but we would like to remove the virtualization layer now11:25
kbaegisWell, using kolla one libvirt driver should do11:26
kbaegisYou only need to talk to the baremetal lxc11:26
kbaegisSo you don't need "multiple libvirts" or anything11:26
kbaegisYou should be able to spawn tons of instances11:26
kbaegiscpu_overcommit I think is still 16:111:26
openstackgerritMerged openstack/kolla: Customizations for haproxy  https://review.openstack.org/34839011:27
kbaegisOh, it's not 16:1 by default :)11:28
*** zhurong has joined #openstack-kolla11:32
openstackgerritMerged openstack/kolla: Customizations for gnocchi  https://review.openstack.org/34630011:32
openstackgerritMerged openstack/kolla: Customizations for magnum  https://review.openstack.org/35149211:32
openstackgerritMerged openstack/kolla: Customizations for Horizon  https://review.openstack.org/35047611:32
kbaegisYay!11:33
*** kbaegis has quit IRC11:36
*** kbaegis has joined #openstack-kolla11:36
*** kbaegis has quit IRC11:41
*** kbaegis has joined #openstack-kolla11:42
*** kbaegis1 has joined #openstack-kolla11:43
*** kbaegis has quit IRC11:43
*** yee_ has joined #openstack-kolla11:51
*** daneyon has joined #openstack-kolla11:52
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Nova  https://review.openstack.org/35167611:53
*** daneyon has quit IRC11:56
sean-k-mooneyo/11:57
sean-k-mooneyhello everyone11:57
*** dave-mccowan has joined #openstack-kolla12:08
*** mkoderer_ has joined #openstack-kolla12:15
*** lmiccini_ has joined #openstack-kolla12:17
*** lmiccini has quit IRC12:17
*** mkoderer has quit IRC12:18
openstackgerritzhubingbing proposed openstack/kolla: repair gnocchi image dockerfile  https://review.openstack.org/34838812:19
*** yee_ has quit IRC12:22
*** JoseMello has joined #openstack-kolla12:35
*** egonzalez has joined #openstack-kolla12:37
*** lulei has quit IRC12:37
*** inc0 has joined #openstack-kolla12:42
inc0good morning12:42
inc0pbourke, we have deployed bare metal:)12:43
*** kbaegis1 has quit IRC12:45
*** kbaegis has joined #openstack-kolla12:46
pbourkehey inc012:50
pbourkeyeah been chatting with britthouser12:50
pbourkehave deployed ssh keys to all, so ready for ansible12:51
pbourkeinc0: currently want to set up bonds, could you check my config when you have a chance? (open on the tmux)12:51
inc0omw12:51
pbourkebasically want to try it on one node first before doing the rest12:51
pbourkealso unsure if I need to change routes12:52
pbourkeinc0: britthouser already spotted some missing stuff12:54
pbourkebritthouser: can you update it with what you think?12:55
britthouseryup12:55
inc0what was deployment node ip?12:56
britthouser8.25312:56
inc0first you need bond-12:57
inc0bond012:57
inc0then you can add bond0.70212:57
britthouserlike that?12:57
britthouserah tx12:59
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935112:59
inc0ok it looks good12:59
inc0to me12:59
sean-k-mooneyinc0: speaking of bonds i need to add a patch to allow configuing networking interfaces as part of kolla-host to supprot ovs-dpdk13:00
sean-k-mooneyinc0: if ye keep a list of manual steps ye need to do i can add support to the kolla-host playbook for them13:00
inc0that's gonna be funky as network setup might differ13:00
inc0I'd rather setup some requrements13:01
inc0like "make a bond and put a name into globals"13:01
inc0you need to restart networking britthouser pb13:01
inc0pbourke,13:02
pbourkesoo13:02
pbourkethe node is no longer on the pxe vlan?13:02
sean-k-mooneyya well we support bonding in our devstack pluging but yes the code is not simple.13:02
britthouseryeah pbourke - we removed that interface altogether.13:02
britthouserkinda why I wanted todo this via cobbler....=/13:02
britthouserif we add that interface back in, we'll have two default routes13:03
britthouserwould like to add it in with DHCP but say, "ignore dhcp default route"13:03
britthouserlemme google and see if there is ubuntu network config to override DHCP default route13:03
sean-k-mooneyinc0: we do it with two arrays of pairs.  bond->bridge mappins and port>bond mappings13:04
pbourkenice13:04
britthouserthat "Partner Mac Address:" is what lets you know its talking to the switch13:04
inc0britthouser, we didnt13:05
inc0we made bond on p1p2 and p4p213:05
pbourkeinc0: britthouser: I can ansible the same onto the other nodes13:05
inc0pxe goes over p1p113:05
pbourkeif that's good with you13:05
inc0that's perfect;)13:05
inc0you can use ip from p1p1 to get last octet for 70213:05
britthouserso inc0 you're saying dont' movet he default route?13:05
inc0do we need to really?13:05
britthouserWhat traffic would use the mgmt network if we don;t?13:06
inc0mariadb, rabbit, APIs13:06
inc0that sort of stuff13:06
inc0we'll use it as kolla_internal13:06
britthouseryeah lets discuss all the networks we'll need13:07
britthouserso you're saying kolla_internal=bond0.70213:07
sean-k-mooneyare ye going to share networks or keep them seperate per function ?13:07
inc0and neutron_interface = bond013:07
inc0sean-k-mooney, they'll be separated as 702 is vlan13:08
inc0on that note13:08
britthouserThe way I read the switch configs, all the bond0 IPs need to be in a VLAN13:08
inc0I wonder if neutron will like tagged interface from same bond13:08
britthouserfor VXLAN it should be fine.13:08
sean-k-mooneyinc0: if you use vxlan it wont care13:09
inc0maybe we should use p4p2.702 for mgmt?13:09
inc0ok, we'll see soon enough;)13:09
britthouserSo neutron_interface=bond0.730 ?13:09
britthouserkolla_internal=bond0.70213:09
inc0no, bond013:09
inc0or...13:10
inc0yeah 730 will be better13:10
britthouserPretty sure you won't get any packets passed on just bond0 w/o a tag13:10
inc0but all traffic will go throush 702 anyway13:10
inc0yay for vxlan13:10
inc0britthouser, but neutron knows how to vlan if we setup provisioner: vlan13:11
britthouserI'm adding notes from here to etherpad13:11
britthouseris that on our list inc0?  I can't remember13:11
sean-k-mooneyby the way would it be good to test using a different interface for each ot the interface types13:11
sean-k-mooney#kolla_external_vip_interface: "{{ network_interface }}" #api_interface: "{{ network_interface }}" #storage_interface: "{{ network_interface }}" #cluster_interface: "{{ network_interface }}" #tunnel_interface: "{{ network_interface }}"13:11
britthouserif we're gonna do VLAN tenant networks, we should use p4p1 dedicated for it.13:11
*** sdake has joined #openstack-kolla13:11
britthouserthx for that list sean-k-mooney13:12
openstackgerritzhubingbing proposed openstack/kolla: repair aodh image dockerfile  https://review.openstack.org/35071513:12
sdakehey cats13:12
*** jtriley has joined #openstack-kolla13:12
britthouserhey sdake13:12
inc0gmorning sdake13:12
sdakemornign britthouser inc013:13
britthouserWe're discussing host networking layout13:13
britthouserRight now we have p1p1=pxe network and also our default route13:13
sean-k-mooneyin the ceph case i think it makes sense to test a dedicated storage network.13:13
sdakeinc0 need me to kick off a geo coordinators thread on ml?13:13
sean-k-mooneyand hi sdake good morning13:13
britthouserbond0 of p1p2 and p4p2 being used for kolla_internal on Vlan70213:13
sdakehey sean-k-mooney  :)13:13
britthouserneed to to decide what else should be broken out.13:14
*** egonzalez90 has joined #openstack-kolla13:14
*** mkoderer has joined #openstack-kolla13:14
*** zhubingbing__ has joined #openstack-kolla13:14
britthouserDo we need to test both VLAN and VXLAN?13:14
britthouserI see LB and OVS being called out as permutations13:15
britthouserbut if we can keep to just VXLAN, then we can make p4p1 our storage cluster network13:15
zhubingbing__can i help me13:15
*** lmiccini has joined #openstack-kolla13:15
britthouserand the do the rest on VLANs on the bond13:16
britthouserthoughts?13:16
inc0for ceph we'll need 2 networks really13:16
sdakezhubingbing__ of course you can hep yourself :)13:16
sean-k-mooneybritthouser: i think that would work fine. i dont thing vlan tenent networks buy us alot13:16
*** egonzalez has quit IRC13:16
inc0one for replication and one for storage13:16
inc0gonna be interesting13:16
sdakezhubingbing__ what do you need :)13:16
sean-k-mooneybritthouser:  if we want to test tenant provider networks we could do it with a flat network13:17
*** mkoderer_ has quit IRC13:17
mewaldbritthouser: I dont really like pxe boot NIC to be the default route. That would mean, we need to put all nodes together into some internal api network. That means running multiple different host types in the same network within the same scalability boundaries. In our installations, we usually give each host group (control, network. compute) it's own api network and share only very few where absolutely necessary.13:17
britthouserSo inc0 you're saying bond0.731 shoudl be storage (clients) and p4p1 should be storage (replications?13:17
zhubingbing__{"failed": true, "msg": "The conditional check 'enable_elasticsearch | bool' failed. The error was: error while evaluating conditional (enable_elasticsearch | bool): {{ 'yes' if enable_central_logging | bool }}: the inline if-expression on line 1 evaluated to false and no else section was defined.  i don't  know13:18
zhubingbing__http://logs.openstack.org/27/351027/8/check/gate-kolla-dsvm-deploy-ubuntu-source/9810828/console.html13:18
zhubingbing__i13:18
zhubingbing__I don't know why this is the wrong, I was good before13:18
*** lmiccini_ has quit IRC13:18
*** egonzalez90 has quit IRC13:18
britthouseryeah mewald I'd rather see the default route be on the bond0.702 management network.  But the cobbler preseed given to us doesn't setup anything but the PXE interface.  So we have todo the rest after the install.13:19
sdakezhubingbing__ you need to recheck13:19
sdakezhubingbing__ Jeffrey4l fixed that bug last night13:19
britthouserrestarting networking from ansible gives me angina13:19
*** yee_ has joined #openstack-kolla13:19
sean-k-mooneybritthouser: it might be easier to discuss the networkign with an etherpad13:19
britthouserhttps://etherpad.openstack.org/p/kolla-N-midcycle-osic13:19
britthouserscroll down to the "Working Notes" section13:20
sdakezhubingbing__ we merged a patch which was broken - it broke your ci results13:20
sean-k-mooneyah thanks britthouser13:20
sean-k-mooneyso since p1p1 is our default route that should also be the interface we use for the external vip right?13:21
britthouseryeah13:21
britthouserunless there is a safe way to move the default route with ansible.13:22
sbezverksdake: I got daemonset running by adding sleep 1 before ovs-db start. works 100% now..13:22
sean-k-mooneyi dont think we need to move the defualt route though it will only be used for internet faceing traffic13:23
zhubingbing__This is my fault? what do I need to do?13:23
inc0ok...so do we setup docker and stuff now?13:23
pbourkebritthouser: our default route is not on the mgmt network13:23
pbourkeis that a problem?13:23
britthouserI dno't think so13:23
inc0me neither13:23
britthouserjsut require a little re-learning13:23
britthouser=)13:24
pbourkeok13:24
sean-k-mooneyif you can reach the internet of the pxe network then all is good13:24
pbourkeare we going to use sean-k-mooney's playbook or something else13:24
inc0seans play is for ubuntu 1613:24
inc0but we can use most of it13:24
inc0and maybe modify it a bit13:24
britthouserif you look a little higher in etherpad, lets lay out the networking13:24
*** banix has joined #openstack-kolla13:24
sean-k-mooneyinc0: it might work on 14.04 also but i actully need to go test that now13:24
britthouserb/c I think we'll need to add some interfaces to your ansible pbourke13:25
inc0well 14.04 doesnt use systemd13:25
inc0so I doubt it will just fly13:25
sdakesbezverk yes i expected that outcome13:25
britthouserAny objections to making cluster_interface dedicated interface p4p1 ?13:25
sean-k-mooneyya i think im mising the insecure registry config for 14.04 and the storage driver change13:26
sdakesbezverk change sleep 1 to /bin/true pls13:26
britthouserand then api_interface can be same as mgmt13:26
inc0yeah13:27
inc0that sounds good13:27
britthouserstorage_interface same as mgmt too? or its own VLAN?13:27
inc0own vlan I'd say13:27
britthouserok here is what I put in etherpad:13:27
britthouserkolla_internal: bond0.70213:27
britthouserkolla_external_vip_interface: p1p1 (also pxe, default route)13:27
britthouserapi_interface: bond0.70213:27
britthouserstorage_interface: bond0.73113:27
britthousercluster_interface: p4p113:27
britthousertunnel_interface: bond0.73013:27
britthouser+1 -1 ?13:28
sean-k-mooneywhat is the neutron_external_interface set too13:28
*** Mr_Broken has quit IRC13:28
openstackgerritChristian Berendt proposed openstack/kolla: Remove files from /var/lib/apt/lists when cleaning up on Ubuntu/Debian  https://review.openstack.org/35173813:29
britthousertaht wasn't in your previous list sean-k-mooney. =P13:29
*** Mr_Broken has joined #openstack-kolla13:29
inc0seems legit13:29
sean-k-mooney:) correct neiter was neutron_interface13:29
britthouserlol13:29
inc0I'll set up rest of vlans13:29
zhubingbing__ok13:29
sean-k-mooneysorry not neutron_interface13:29
sean-k-mooneynetwork_interface13:29
zhubingbing__sdake,ok13:29
britthouseris that different than tunnel_interface?13:30
britthouserI thikn we're still missing external inc013:30
britthouserfor FIP13:30
sean-k-mooneythe network_interface i think is what you have called kolla_internal13:30
inc0so, won't it create flat on bond013:31
inc0?13:31
inc0or rather neutron interface?13:31
sean-k-mooneyneutron_external_interface is for flat/vlan  neutron networks13:31
inc0britthouser, we need vlan at p4p1 too I think13:32
sean-k-mooneyif we dont use vlan tenent networks then neutron_external_interface can be a vlan on bond013:32
inc0as native would be pxe I'd guess13:32
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Cinder  https://review.openstack.org/35058713:32
*** diogogmt has joined #openstack-kolla13:32
britthouseryeah neutron_external_interface will be vlan on bond013:33
britthouserlets say 732?13:33
*** Mr_Broken has quit IRC13:33
inc0btw how do you setup tunnel_network?13:33
britthouserYeah we can make p4p1.733 for storage cluster?13:33
inc0I'd go for bond0 as neutron external13:34
inc0and setup fip network as a vlan via neutron13:34
britthouserI'm 95% sure untagged packets on bond0 will be dropped.13:34
inc0we'll make it provisioner:vlan13:34
inc0and let neutron tag it13:34
britthouserI thought we were doing vxlan?13:35
inc0one doesn't rule out the other13:35
britthouser......13:35
sdakeinc0 re geo coordinators - prefer i send a list to get that rolling - or will you send today?13:35
sdakesend an email to list that is13:35
britthouserLets stick with jsut vxlan here at the beginning13:35
britthouserif we want to get fancy with multiple varieties of neutron networks, then lets save that for later.13:36
britthouseryes/no?13:36
inc0sdake, mail with geo-coordinators?13:36
inc0ok13:36
inc0britthouser, ok13:36
inc0so I'll add bond0.73213:36
sdakeinc0 ya in last team meeting you had indicated you would send an email so we could establish who would be geo coordinators for our 3 geos (APAC, EMEA, and US)13:36
sdakethat way we could use a follow-the-sun strategy for making use of osic cluster13:37
pbourkesdake: that's on the etherpad13:38
sdakegeo coordinator would be responsible for knowing the current state of osic cluster13:38
sean-k-mooneybritthouser: with vxlan for the teneant network you will still want the neutron external network to be a flat network that will use bond0.73213:38
sdakeoh - didn't know people had signed up htere - my bad13:38
pbourkeor maybe we didn't designate one for each13:38
sdakei haven't looked at thte etherpad in last day13:38
pbourkeno you're right13:38
pbourkewe only have the people in each zone13:38
inc0britthouser, check tmux and look if I miss something13:38
sdakeinc0 if your busy i can tackle it - up to you :)13:39
sdakeinc0 yes or no will work :)13:39
inc0sdake, can you send it plz?13:39
sdakeroger13:39
inc0I'm ok with mail, I'm just trying to get this hardware rolling;)13:39
sdakepbourke mind linking etherpad?13:39
sdakeinc0 right your busy13:39
britthousersdake: https://etherpad.openstack.org/p/kolla-N-midcycle-osic13:40
sdakethanks folks13:40
sdakeemail inc13:40
pbourkeinc0: makes sense to me but will wait for britthouser13:40
britthouserinc0 the cluster is not on the bond, right?13:40
sean-k-mooneyinc0: britthouser  is it possibly to make sure that  vlan 732 is routed to the internet so that the tenent vms have internet access?13:40
britthouserIt does sean-k-mooney13:41
pbourkesean-k-mooney: 732 has internet access13:41
britthouserat lest the docs say it does13:41
inc0ah good catch - forgot to setup p4p113:41
inc0sean-k-mooney, all of vlans have internet access13:41
sean-k-mooneyah cool that makes things easy13:42
*** b_bezak has quit IRC13:42
*** b_bezak has joined #openstack-kolla13:42
*** b_bezak has quit IRC13:42
sean-k-mooneythenant internet acess will be needed for the 1000 vms each running kolla test in week 313:42
inc0yeah13:43
inc0btw, that's going to be interesting13:43
sean-k-mooneyinteresting is one word to discribe it13:43
inc0:D13:43
inc0brb13:44
openstackgerritChristian Berendt proposed openstack/kolla: Reducing disk footprint for Ubuntu/Debian images  https://review.openstack.org/35174413:44
sean-k-mooneyyes actully ill be back in 20mins also13:44
openstackgerritChristian Berendt proposed openstack/kolla: Reducing disk footprint for Ubuntu/Debian images  https://review.openstack.org/35174413:45
britthouserdriving for a bit inc013:45
*** b_bezak has joined #openstack-kolla13:47
sbezverksdake: it works with true as well, it looks like it needs an extra millisecond ;-) to finish container initialization13:47
britthouserok pbourke and inc0 I htink this is good13:50
pbourkesweet13:50
britthouserhow do I run this thing?13:50
britthouserthx!13:51
britthouserback in a few mins....13:51
inc0just had to make one small change13:51
pbourkelook at it go13:52
pbourkebeautiful13:52
inc0:D13:52
inc0sdake, so yesterday I watched full 1hr documentary on honey badgers13:52
inc0it's on netflix13:52
openstackgerritChristian Berendt proposed openstack/kolla: Reducing disk footprint for Ubuntu/Debian images  https://review.openstack.org/35174413:53
* inc0 approves13:53
sdakeinc0 sweet i'll definatley have to check that out tonight13:53
sdakerecall title - or just search for honey badgers?13:53
inc0yeah, just search honey badgers13:53
pbourkei tried watching that before13:53
*** b_bezak has quit IRC13:55
*** b_bezak has joined #openstack-kolla13:55
*** yee_ has quit IRC13:56
inc0:(13:57
inc0hmm13:58
sdakenow on to emergency #4 for the day :)13:58
inc0I might have fucked up our networking cluster-wide13:59
inc0yay13:59
sdakesbezverk cool - goal is to get a backtrace - if that can't be done it can't be done13:59
sbezverksdake I opened a bug with kube folks13:59
sbezverkgot initial communication14:00
sdakesbezverk openvswitch folks would appreciate a backtrace i'm sure14:00
*** banix has quit IRC14:00
sdakesbezverk mind providing link or tldr?14:00
sbezverkhttps://github.com/kubernetes/kubernetes/issues/3010614:01
*** b_bezak has quit IRC14:02
*** openstackgerrit has quit IRC14:03
*** openstackgerrit has joined #openstack-kolla14:03
*** mliima_ has joined #openstack-kolla14:03
*** banix has joined #openstack-kolla14:04
inc0ok networking looks good14:05
inc0pbourke, wanna try Sean's playbook14:05
inc0?14:05
pbourkeinc0: yup14:05
inc0do the honors14:05
pbourkeis it merged/14:05
pbourkeor in a review14:05
inc0I'll have to go to appointment soon so will leave you to it14:06
inc0still in review14:06
pbourkegrand14:06
inc0let me check if we need to change much in it14:06
inc0as it was desitned for 16.0414:06
inc0pbourke, https://review.openstack.org/#/c/325631/14:06
patchbotinc0: patch 325631 - kolla - add baremetal role to install kolla deps.14:06
pbourkegot it14:06
pbourkei'll have a look14:07
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Nova  https://review.openstack.org/35167614:07
*** mliima has quit IRC14:07
britthouserpbourke  - back now.14:07
pbourkebritthouser: cool, just having a look now at sean's patch above14:08
britthouserwhich nodes did we designate control/compute/ceph?  I just wanted to check their partitioning14:08
inc0britthouser, we don't run ceph yet I think14:08
*** mkoderer has quit IRC14:08
inc0and most of them are running cinder image14:08
inc0let me double check14:08
pbourkebritthouser: oh yeah, that would be useful if you could partition the inventory?14:08
openstackgerritzhubingbing proposed openstack/kolla: Add sahara ansible roles  https://review.openstack.org/35129414:09
*** duonghq has joined #openstack-kolla14:09
britthouserOh....ok you marked it done, so I thought it was done14:09
*** lmiccini has quit IRC14:09
*** ad_rien_ has quit IRC14:09
inc0britthouser, I think all of nodes runs generic now14:09
britthouserYeah...we want to move compute to the cinder profile and storage....well I guess to the cinder profile as well and then later to the swift profile when we move to ceph/14:10
britthouserYeah...we want to move compute to the cinder profile and storage....well I guess to the cinder profile as well and then later to the swift profile when we move to ceph?14:10
inc0yeah14:10
inc0let's just deploy kolla now14:10
inc0without storage14:10
inc0and once we have all the plays written14:11
inc0repaving will be a breeze14:11
*** coolsvap has quit IRC14:11
britthouserOk so I'll pick three nodes to be control nodes14:12
britthouserand the rest I'll re-kickstart with the cinder profile14:12
britthousersound good?14:12
inc0let's try to deploy now14:12
inc0ok?14:12
inc0I mean without rekicking14:13
inc0just basic no-cinder installation14:13
*** kbaegis has quit IRC14:13
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Nova  https://review.openstack.org/35167614:13
britthouserSo right now everyone is deployed as generic, they have a single RAID1 array14:14
inc0we can rekick them eod so we won't waste time on waiting for them boot14:14
inc0yeah which will be ok for super basic scenerio14:14
britthouserOk.  While you do that, I'll work on changing up the input files for the EOD scenario14:14
britthouserBasically you're doing #3, right inc0?14:15
britthouseror step3 will be after EOD re-kickstart?14:15
inc0it sill be semi-3 as no real storage14:16
britthouserOk.  Roll with it. =)14:16
*** mkoderer has joined #openstack-kolla14:17
*** narasimha_SV has joined #openstack-kolla14:17
narasimha_SVwhen deploying ironic using kolla, there is no specification for drivers needed to be used ?14:18
narasimha_SVis there a way that we can specify these things14:18
britthouserWe should think bout backing up all the files we generated in the osic-prep container.....14:18
*** lmiccini has joined #openstack-kolla14:19
sdakenarasimha_SV ironic as a service is not functional14:19
inc08.254 will be keepalived-floating ok?14:20
sdakenarasimha_SV its a work in progress - needs love - feel free to help as you see fit - we need help there :)14:20
sdakenarasimha_SV where we are focusing at present related to ironic is bare metal deployment so kolla isn't half a solution but a full solution14:20
*** mnasiadka has quit IRC14:20
britthouserno inc014:21
britthouserthat is the lxc container14:21
sdakenarasimha_SV if you want to follow that work - look at owner: sean k mooney in gerrit14:21
inc0I changed it to 20014:21
sdakenarasimha_SV work expected to land in newton14:21
narasimha_SVok14:22
sdakenarasimha_SV notice the stars on the reviews - that indicates its important for crs to review14:22
narasimha_SVsdake: sure . I am ready to start working on this.14:22
sean-k-mooneynarasimha_SV:  the ironic support almost works in kolla14:23
sdakenarasimha_SV sweet the best way to help is to review those starred patches from sean to make sure they are correct14:23
britthouseractually I was wrong inc014:23
sean-k-mooneythe issue is the pxe boot14:23
britthouser0.254 is the lxc14:23
britthouser8.254 could be the VIP14:23
*** diogogmt has quit IRC14:23
inc0 yeah, but let's not mix last octets14:23
sean-k-mooneynarasimha_SV: when ironic pxe boots the nodes the node send the chassis uuid14:23
inc0I like to have last octet same across the vlans14:23
narasimha_SVsdake: sure I will check them14:23
sdakesean-k-mooney do you mean t he bifrost work or hte bare metal as  a service work14:23
sdakehe/the14:24
sdake(fast with lots of mistakes;)14:24
sean-k-mooneynarasimha_SV:  the dhcp server is configured to respond to the neutron port uuid.14:24
sean-k-mooneysdake: ironic not bifrost14:24
sdakesean-k-mooney thanks for clarification14:24
openstackgerritzhubingbing proposed openstack/kolla: fix sahara dockerfile  https://review.openstack.org/35132014:24
sdakeit would be fantastic to have that work land in newton too - not sure who cn take it up atm14:24
sdakewe only have about 3 weeks left in the cycle14:25
britthouserso your docker registry is same IP as LXC inc014:25
inc0yeah, we can run registry in it14:25
inc0why not14:25
sean-k-mooneysdake: we have been looking at it on and off over the cycle. have not had time to fix the ironic supprot properly14:25
britthouserRun a docker container inside the LXC?14:25
sean-k-mooneysdake: the pxe boot/uuid issue was the last one we think need to be fixed to get it running14:26
sdakesean-k-mooney roger- imo your working on the right thing at the right time14:26
sdakenarasimha_SV if your a dev or ops interested in that functionality we could really use the help on it :)14:26
sdakesean-k-mooney is closest to the work - he has a thourough understanding of what is needed14:27
inc0sean-k-mooney, soo...how do you add kolla_ssh_key to your playbook?14:27
narasimha_SVsdake: we are planning to implement ironic in our environment14:27
*** zhurong has quit IRC14:27
narasimha_SVwe are using kolla for this to setup our env14:27
sean-k-mooneyinc0: oh ya soory i  manually copied the bifrost ones and renamed them locally in my passwords.yml14:28
inc0gonna put that into review14:28
sean-k-mooneyinc0: ill do that now14:28
pbourkeinc0: already done ;)14:28
inc0yeah saw it14:28
inc0good catch14:28
narasimha_SVif that is not there then I will start looking at it so that I can push to the master14:29
narasimha_SVI just raised bug in kolla14:29
sean-k-mooneyspeaking of that i need to figure out a good way to test this also in the ci14:29
*** Jeffrey4l has quit IRC14:29
*** Jeffrey4l has joined #openstack-kolla14:29
inc0sean-k-mooney, we're complementing you for tremendous job14:30
inc0just ran this play, cool stuff!14:30
narasimha_SVhttps://bugs.launchpad.net/kolla/+bug/161027214:30
openstackLaunchpad bug 1610272 in kolla "ironic.conf file missing enabled_drivers info" [Undecided,New]14:30
sean-k-mooneyinc0: did it work other then the ssh key?14:31
sdakesean-k-mooney ci is for later - we hae a whole slew of ci work to do14:31
*** michauds has joined #openstack-kolla14:31
inc0yeah14:32
sean-k-mooneysdake: yep. i think we can test it in the ci the same way bifrost does via vms. but i have not looked into it14:32
sdakeat next summit i'd like to have another repeat of our contributor day on how to work with project-config and the gate14:32
sdakesean-k-mooney will you be in barcelona?14:33
sean-k-mooneysdake: it looking that way yes14:33
sdakecool - if we get a speaking spot i'd really like to highlight your &teams work on bifrost integration14:33
sean-k-mooneystill no confirmation if travel is aproved but im on the list to travel.14:34
sdakesean-k-mooney well whether you are there or not, we will be highlighting that work if we get a speaking slot14:34
*** daneyon has joined #openstack-kolla14:34
sdakesean-k-mooney would be nice  to get you on stage with the rest of the cores that have travel approval to acknowledge this extremely difficult objective14:35
*** senk has joined #openstack-kolla14:35
sdakesince from my pov its about done :)14:35
senkcan someone please add +1 worklow or readd the workflow on https://review.openstack.org/#/c/350222/214:35
patchbotsenk: patch 350222 - kolla - Fix inconsistencies in git url14:35
inc0+1 to that14:35
*** diogogmt has joined #openstack-kolla14:35
sdakesenk looking14:35
senkwere some issues with jenkins in the morning14:36
senksdake: thanks14:36
sdakesenk ya that was my fault for merging a patch that didn't pass the gate14:36
sdakemoment looking at patch to see if that issue is your issue14:36
sean-k-mooneysdake well that more insentive to get it completed i guess. im not much of a public speaking kind of guy but i would be honored.14:36
sdakesenk not sure what is going on there -sometimes ci flakes out - what I'd recommend is rebasing your patch on master (git pull --rebase; git review)14:37
sdakeand it will be reacked14:37
sean-k-mooneyoh i just realise i probly should add a release note for the kolla-host and bifrost changes...14:37
sdakesean-k-mooney yes please14:37
narasimha_SVsean-k-mooney: not that one I dont see the supported driver info in the ironinc.conf file14:38
*** banix has quit IRC14:38
senksdake: na zuul simply skipped docs jobs. infra fixed it at https://review.openstack.org/#/c/351626/14:38
patchbotsenk: patch 351626 - openstack-infra/project-config - Fix kolla docs skip (MERGED)14:38
inc0gtg for an hour or so14:38
inc0cyas14:38
narasimha_SVI have even check ansible code there also there is no specification metioned either in templates or tasks14:38
*** inc0 has quit IRC14:38
senkah my workflowchange restarted it14:39
*** daneyon has quit IRC14:39
*** mliima_ has quit IRC14:40
sean-k-mooneynarasimha_SV:  am in the context of kolla ironic support? sorry im context switching so lost context of what we were talking about14:40
sdakeone thing that would help is defining the work as either "bifrost" or "ironic as a service"14:41
sdakesince the term ironic is overloaded14:41
*** ayoung_ZZZzzz is now known as ayoung14:42
sean-k-mooneysdake:  i think narasimha_SV was refering to ironic as a service14:43
sdakesean-k-mooney agree14:43
duonghqhello sdake14:44
sdakenarasimha_SV the kolla team hasn't committed to supporting ironic as a service in newton - unless someone comes along nd makes it work14:44
sdakesup duonghq wht can i do for ya14:44
sean-k-mooneysdake: narasimha_SV  im not actully that familar with seting up ironic itself as i let bifrost do that form me so not sure if we need to specify the supproted dirvers14:44
duonghqAs I see nobody take the bp nova-kubernetes up to now14:44
duonghqCan I try on this bp?14:45
sdakeduonghq pretty sure wirehead_ is working on that14:45
sdakeduonghq can you coordinate with him?14:45
openstackgerritRobin Naundorf proposed openstack/kolla: Fix inconsistencies in git url  https://review.openstack.org/35022214:45
sdakesenk ^^14:45
sean-k-mooneynarasimha_SV: if you need to tweak anyting in the ironic.conf you should be able to drop a file into /etc/kolla/config/ironic/ironic.conf14:46
sdakei always thought it was a shame batman didn't follow up with a & robyn after "dark knight rises"14:46
senksdake: still learning to work with gerrit..^^14:46
*** fragatina has joined #openstack-kolla14:46
sdakesenk learning experience efor all of us - thanks for the contribution :)14:46
duonghqI don't know his timezone, but I can,14:46
sdakeduonghq he is PST14:47
sdakeits 7:47 PST atm14:47
sdakehe is usually alive by 9am or 10am14:47
*** dmueller-evoila has quit IRC14:48
sdakeduonghq another person to coordinate with is dcwangmit0114:48
sdakehe is also working on this iiuc14:48
*** fragatin_ has joined #openstack-kolla14:48
duonghqI'm on UTC+714:48
*** srwilkers has joined #openstack-kolla14:48
*** diogogmt has quit IRC14:48
sdakeso 8pm there?14:48
duonghqnear 10pm here14:48
sdakeyikes14:48
sdakeapac and us hard to coordinate because of timezones14:48
sdakeone option is to leave a note in the blueprint14:49
sdake(if you want to head to bed)14:49
sdakerhallisey leading up kolla-kubernetes work - woudl be another contact (less close to the actual nova blueprint in question)14:50
duonghqI can work late but the main problem is I cannot reach my dev PC at this time, but I can try to grab one alternative,14:50
*** mkoderer_ has joined #openstack-kolla14:51
duonghqbut as I see, it seem that we do not communicate much on mailing list?14:51
*** lmiccini_ has joined #openstack-kolla14:51
sdakemailing list works fine14:51
*** fragatina has quit IRC14:51
sdakeirc is faster14:51
duonghqsure14:52
*** diogogmt has joined #openstack-kolla14:52
narasimha_SVsean-k-mooney: this directory is not default right we need to add this so that we can change14:52
sdakewirehead_ dcwangmit01 can give you a faster update on what is happening on that blueprint14:52
*** mliima has joined #openstack-kolla14:52
sdakevia irc - mailing list ha coule days lag time14:52
narasimha_SVsean-k-mooney: what values do I need to keep here all the values in template form or new values which i want to add14:52
*** mewald has quit IRC14:52
duonghqyeah, I can discuss with them on IRC and work on my dev machine later (in the worst case)14:53
sean-k-mooneynarasimha_SV:  kolla will merge the overdies from the config director with the template so you only need to add your custom values14:53
*** mkoderer has quit IRC14:54
*** lmiccini has quit IRC14:54
narasimha_SVsean-k-mooney: can I add this config value as default value in ironic.conf by sepcifying all default drivers ?14:55
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Introduce a script to launch ovsdb-server process  https://review.openstack.org/35178414:55
duonghqsdake: I'm fine with wirehead_ working hours15:00
sdakeduonghq that said given its friday - not sure if he is on pto today or not, so reach out to dcwangmit01 as well15:01
duonghqokay, dcwangmit01 is on PST, isn't him?15:02
sdakeduonghq believe so15:02
sdakeprobably same hours as wirehead_15:02
duonghqokay15:03
sdakeone of em can get you integrated in that effort15:03
*** banix has joined #openstack-kolla15:03
*** mliima has quit IRC15:04
duonghqnice, I'll ask them when somebody online15:04
sdakerhallisey is typically around at this time - but may be afk atm15:05
sdakerhallisey has a deeper understanding of what wirehead_ and dcwangmit01 are up to15:05
rhalliseyhey15:05
rhalliseyhad my sound off15:05
sdakeduonghq we had planned at midcycle to put work items in the blueprints15:05
* rhallisey reads back15:05
sdakerhallisey ya my wife complains constantly about my buzzer interrupting herwork :)15:06
sean-k-mooneynarasimha_SV:  can you post a link to where you want to add it in the code. and yes you could15:07
rhalliseyduonghq, wirehead_ has a patch for nova up now,15:07
rhalliseyhttps://review.openstack.org/#/c/347572/15:07
patchbotrhallisey: patch 347572 - kolla-kubernetes - Remove hostnetworking for Nova control plane and c...15:07
duonghqsorry sdake if it's my fault15:07
*** shardy is now known as shardy_afk15:07
*** zhubingbing__ has quit IRC15:09
sdakerhallisey can you star that review plelase15:09
rhalliseyduonghq, the nova bp hasn't had any work items filled out yet15:10
*** fragatin_ has quit IRC15:10
rhalliseyif you wan, apply that patch and see if you can get a vm up15:11
* rhallisey will be doing so tmr15:11
sdakeduonghq nah its mine for not being omnipotent ;)15:11
rhalliseyduonghq, there's also https://blueprints.launchpad.net/kolla-kubernetes/+spec/cinder-kubernetes15:11
rhalliseyactully not sure if swift is fully fucntional15:12
sdakeduonghq that is why we have a team - so folks dont have to know everything :)15:12
rhalliseyswift would also be awwesome15:12
*** zhubingbing__ has joined #openstack-kolla15:12
*** yee_ has joined #openstack-kolla15:13
duonghqsdake absolutely :)15:13
sdakeoh glorious rain15:13
sdakeits not 115 today :)15:13
duonghqrhallisey: I am taking a look at those15:13
rhalliseyduonghq, ok awesome. Just added https://blueprints.launchpad.net/kolla-kubernetes/+spec/swift-kubernetes if you're interested in that15:14
rhalliseyswift already has pods/service, but needs a little more work15:15
duonghqrhallisey: do we have any bp work yet, so I can learn something from these15:16
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Introduce a script to launch ovsdb-server process  https://review.openstack.org/35178415:17
rhalliseyduonghq, what do you mean by bp work?  Workitems?15:17
duonghqI mean some pods already up and running15:18
rhalliseyduonghq, yes, horizon, keystone, memcached, mariadb15:18
*** mliima has joined #openstack-kolla15:18
rhalliseyglance, nova, and neutron are coming soon15:18
rhalliseyduonghq, right now it's all CLI driven15:19
rhalliseyI'll be adding ansible in front of that this weekend15:19
* irtermite yawns15:19
irtermitemorning... Looks like things are rolling along15:19
duonghqawesome15:20
sdakeirtermite appears so - thanks for the help :)15:20
irtermitesdake: of course15:20
sdakeinc0 pbourke britthouser any blockers needing irtermite 's help?15:21
irtermiteinteresting... could you all do me a flavor << see what I did there? you will15:21
pbourkesdake: nope looking good15:21
irtermiteflavor list? is it empty?15:21
irtermitetake a look at the #openstack chan15:21
irtermiteno default flavors when installing with kolla?15:22
sdakeirtermite ya nova took out default flavors in master15:22
sdakestable branches have default flavors - that change wasnt backported15:23
irtermiteas of Mitaka?15:23
sdakeirtermite not certain15:23
sdakei thought it was a master only change15:23
sdakeand didn't affect mitaka15:23
sdakebut coud be wrong15:23
*** inc0 has joined #openstack-kolla15:24
duonghqrhallisey: I wonder why there is no patchset has bp/nova-kubernetes tag?15:24
inc0I'm back15:25
rhalliseyduonghq, he posted that before I made that bo15:25
rhalliseybp*15:25
pbourkesdake: britthouser inc0 prechecks passed!15:25
pbourkekolla is a go15:26
duonghqnow I understand I cann't keep up with new bp15:26
inc0we need to setup registry and push some images15:26
DavieyChaps... anyone else seeing sudoer issues in kolla_toolbox?  Ie, asking for a password?15:28
britthouserw00t pbourke!15:28
duonghqrhallisey: I need more time to find what people do for new bp,15:28
*** Satya__ has joined #openstack-kolla15:29
sdakeduonghq the irst step is writing down the work items15:29
rhalliseyinc0, ^^^^ kolla_toolbox issue15:29
sdakeduonghq in launchpad - click yellow button15:29
Satya__Hi sdake15:29
sdakehey Satya__15:29
rhalliseyinc0, did you get around that issue that Daviey is also having?15:30
sdakeSatya__ mind reasking your question for folks to answer15:30
Satya__i am not getting the flavors in nova flavor-list15:30
Satya__after the deployment15:30
sdakeSatya__ moment - let me look at codebase15:30
Satya__sure15:30
sean-k-mooneygo to run for a while be back in a few15:30
duonghqsdake: thank you15:30
Davieyrhallisey / inc0: Looking up disks to bootstrap for ceph OSD.... kolla_toolbox runnig find_disks.py, asking for sudoer password?15:31
inc0hmm, yeah, Daviey did you rebuild lately?15:31
Davieycurrent Master15:31
duonghqI wish that other bp has description like the neutron one :P15:31
inc0I had same issue and it turned out to be container missing ansible sudoers file15:31
Davieyinc0: rebuilt today, on Master15:31
inc0for whatever reason15:31
inc0I rebuilt and it was fixed15:32
Davieyhuh15:32
inc0but since you're getting the same issue...it's strange15:32
Davieyinc0: rebuilt world or just the toolbox?15:32
inc0try just toolbox15:32
inc0without cache15:32
inc0pbourke, ssh 172.21.8.250 - no route to host...15:33
duonghqsdake, rhallisey: it is Friday night here, so I try on your suggestion next week15:33
pbourkeuh oh15:34
inc0but I can't access any node15:34
inc0maybe my vpn is bronek15:34
pbourkeinc0: no I cant either15:34
duonghqI hope that I can find some free time on weaken15:34
pbourkeim logged in though15:34
pbourkebut cant get a new connection15:34
inc0eee?15:34
rhalliseyduonghq, ok great!  Talk to you later :)15:34
britthouser8.25315:35
britthouserinc015:35
britthouser^^15:35
inc0ok15:35
inc0gonna change etherpad15:35
* britthouser thought he had done that....sorry15:35
sdakeSatya__ http://osdir.com/ml/openstack-dev/2016-04/msg00465.html15:36
sdakeSatya__ are you using master or a stable branch?15:36
pbourkeinc0: britthouser: trying to run the bootstrap role on deploy node also to get docker for registry etc.15:36
Satya__stable mitaka15:36
sdakecan you run pip show into a paste for me15:37
sdakerather pip show kolla15:37
rhalliseysdake, bunch of doc reworks at the top of the q that I just rebased15:37
britthouserAre you running from inside the LXC pbourke?15:37
pbourke:/15:37
inc0pbourke, we don't need this mount thing in registry tho15:37
Davieyinc0: did you see my mail to offer to help with OSIC?15:38
pbourkeright but15:38
pbourkeone momnt15:38
inc0Daviey, must've missed it, can you plz give me your email?15:38
Satya__from build node?15:39
sdakeSatya__ right from build node please15:39
Davieyinc0: email@daviey.com subject: OSIC cluster for Kolla15:40
*** belmoreira has quit IRC15:40
Satya__http://pastebin.com/NR11jQ5R15:41
pbourkeinc0: britthouser: sdake: probably a good idea to drop a summary to the ML after we complete each major step, for those who don't have time to keep up with tons of IRC logs. Will write something up before finishing for the day15:41
*** Guest52040 has joined #openstack-kolla15:41
inc0good idea pbourke15:41
sdakeSatya__ ok - your running a milestone version15:41
sdakesatya_ not sure if that is root cause15:41
sdakebut lets fix that first15:41
britthouserI was putting that in the etherpad, but ML is better15:41
inc0pbourke, you installed it on host not in containrr15:41
Satya__sure15:41
pbourkeah, keep forgetting15:41
sdakefix one problem at a time until none remain :)15:41
sdakepip remove kolla15:42
sdakeor pip uninstall kolla15:42
sdakefollowed by pip show kolla15:42
Satya__sonr15:42
sdakesonr?15:42
Satya__done15:42
sdakeok15:42
Satya__sorry fat fingure15:42
Satya__fingre15:43
sdakecan you paste pip show kolla again pls15:43
*** mdnadeem has quit IRC15:43
Satya__root@test /]# pip show kolla [root@test /]#15:43
sdakenice15:43
sdakeok - so you ahve two options to run mitaka15:43
Satya__i am using source code15:43
sdakeyou can run stable/mitaka or 2.0.215:44
Satya__stable/mitaka15:44
sdakestable/mitaka has a few fixes on top of 2.0.2 atm15:44
sdakeok go to where kolla git repo is and run git status -> paste15:44
pbourkeinc0: I want to move ansible to the host15:44
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Introduce a script to launch ovsdb-server process  https://review.openstack.org/35178415:45
pbourkewhos typing atm15:45
inc0pbourke, me15:45
inc0you think so?15:45
pbourkeinc0: what you doing15:45
inc0installing docker inside container15:45
openstackgerritzhubingbing proposed openstack/kolla: Add aodh role  https://review.openstack.org/35102715:45
pbourkeI think we should move out of the container15:45
inc0fair enoug15:46
Satya__we have forked the stable mitaka to our local repo15:46
Satya__and makin changes to it15:46
Satya__so will it help if i do a git status?15:46
sdakeSatya__ thought you were deploying upstream version - so that wont help your right15:46
*** yee_ has quit IRC15:47
Satya__nope15:47
sdakesatya_ ok run pip install -e . in kolla directory15:47
sdakeSatya__ rather^15:47
sdakefollowed by pip show kolla15:47
Davieyinc0: yeah, weird... i looked at the container, and it was indeed missing the sudoers file.  Did a rebuild of that, and watched it go in.15:47
Satya__ [root@test nova]# pip install -e  Usage:   pip install [options] <requirement specifier> [package-index-options] ...   pip install [options] -r <requirements file> [package-index-options] ...   pip install [options] [-e] <vcs project url> ...   pip install [options] [-e] <local project path> ...   pip install [options] <archive url/path> ...  -e option requires an argument15:48
sdakeDaviey odd :(15:48
openstackgerritMauricio Lima proposed openstack/kolla: Customizations for Manila  https://review.openstack.org/35060115:48
sdakeSatya__ from the root of the kolla dir15:48
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Introduce a script to launch ovsdb-server process  https://review.openstack.org/35178415:48
Davieysdake: weird that 2 people have seen it in one day :/15:48
inc0Daviey, I had exactly same experience15:48
inc0something is seriously wrong with that15:48
inc0pbourke, did we set insecure registry?15:49
inc0across the board?15:49
Davieyi can't understand how it isn't deterministic15:49
Davieyoh well15:49
inc0Daviey, so since 2 of us had the same issue between few days15:49
Satya__root@test openstack-kolla]# pip install -e  Usage:   pip install [options] <requirement specifier> [package-index-options] ...   pip install [options] -r <requirements file> [package-index-options] ...   pip install [options] [-e] <vcs project url> ...   pip install [options] [-e] <local project path> ...   pip install [options] <archive url/path> ...15:49
inc0which driver are you using? aufs?15:49
sdakecan a core reviewer review https://review.openstack.org/#/c/351784/5 pls15:49
patchbotsdake: patch 351784 - kolla - Introduce a script to launch ovsdb-server process15:49
sdakeand if lgtu - just +2, I'll +w when ci passes15:50
sdake(this is blocking sbezverk)15:50
sdakeSatya__ after you wrap up with the pip install -e . from ~/kolla (or whever your kolla repo is)15:51
sdakerun pip show kolla again for me pls15:51
*** egonzalez90 has joined #openstack-kolla15:52
duonghqI filed a bug on launchpad some days ago: https://bugs.launchpad.net/kolla/+bug/1609502 I'm sorry but can somebody confirm this bug?15:52
openstackLaunchpad bug 1609502 in kolla "Container bootstrap_mariadb exits due to Galera failed" [Undecided,New]15:52
Satya__i just confirmed we are running 2 weeks behind the trunk15:52
sdakeSatya__ - ok so your not running stable/mitaka :)15:53
Satya__nope15:53
sdakethe fact the flavors are not created in master is expected15:53
sdakenova took out  flavor creation in newton-115:53
sdakethe activity that  took place there was pingint the operator list15:53
Davieyinc0: btrfs15:53
sdakeoperators responded "please remove flavor creation"15:54
britthouserso pbourke looks like you're moving from doing deployment from inside lxc to on the host?15:54
pbourkeinc0: the baremetal play can do this for us15:54
inc0ok...so it's not storage15:54
inc0pbourke, ok15:54
inc0pbourke, go ahead15:54
sdakeSatya__ i would highly reocmmend notrunning on master - because your deploying openstack master as well15:54
sdakenot just kolla master15:54
sdakeSatya__ may i recommend rebasing your changes on stable/mitaka15:54
inc0I had it with aufs15:54
sdakethat will solve two problems - first flavor creation will happen - second you will have a stable openstack to run against15:55
Satya__we are actually changing the code to use NDB rather than innodb15:55
pbourkebritthouser: yeah I think it may avoid problems down the road15:55
britthouseragreed15:55
pbourkemaybe not necessary15:55
pbourkebut that ssh thing etc. was tripping me up15:55
Satya__so we are working on upstream15:55
sdakeSatya__ no idea what ndb is - but wfm :)15:55
britthouserkeep teh lxc for jsut cobbler15:55
Satya__Hi Britt15:56
Satya__how are you?15:56
Satya__after long time15:56
sdakeSatya__ I still recommend against deploying master in a production deployment15:56
britthouserHey Satya__15:56
Satya__remember me?15:56
*** sdake has quit IRC15:56
*** sdake has joined #openstack-kolla15:56
duonghqSatya_ I'm sorry but what does "trunk" mean?15:56
britthouserof course!  I wasn't sure if it was your or not15:57
Satya__2 week behind the latest code15:57
sdaketrunk means master15:57
Satya__:P15:57
Satya__now you git it britt15:57
duonghqIs it from svn term?15:57
sdakeduonghq yes15:58
duonghqsdake: thanks15:58
Satya__sdake what ever the latest code in master we just pull it in 2 weeks15:58
sdakeSatya__ roger i hear ya - so the latest code of kolla deploys the latesst code of openstack15:58
sdakeis that what you want?15:59
sdakeif your doing a production cloud  Ithink not15:59
sdakeif your doing development, I htink so15:59
Satya__nope15:59
Satya__i am not working on production cloud15:59
Satya__its a deployment to develop :)15:59
sdakeok - if doing development and using kolla as an upstream then your activites make sense to me16:00
sdakebut you wont get flavor creation16:00
sdakebecuase nova removed it16:00
Satya__hmmm16:00
sdakeyou have to create flavors on your own16:00
Satya__ok got it :)16:00
sdakesweet16:00
sdakehappy to help16:00
Satya__Thanks sdake16:00
Satya__britt there?16:01
Satya__please ping me your mail ID16:01
Satya__need to drop a mail to you16:01
Satya__:)16:01
openstackgerritzhubingbing proposed openstack/kolla: Add Barbibcan container dockerfile  https://review.openstack.org/35182216:02
sdakezhubingbing__ on fire!16:03
zhubingbing__- -16:03
*** signed8bit has joined #openstack-kolla16:03
zhubingbing__thanks16:03
sdakezhubingbing__ (thats a us compliment to mean your doing good work)16:03
openstackgerritEduardo Gonzalez proposed openstack/kolla: Customizations for Ceph  https://review.openstack.org/35182316:04
zhubingbing__thank you, i will work harder.16:05
sdakezhubingbing__ your working hard enough imo :)16:05
zhubingbing__I have some questions about the Barbican barbican.conf, and role needs to be submitted tomorrow.16:06
openstackgerritEduardo Gonzalez proposed openstack/kolla: Customizations for Ceph  https://review.openstack.org/35182316:07
sdakezhubingbing__ dave-mccowan is a barbican core and expert in the code16:08
sdakezhubingbing__ recommend hitting him up for q&a16:08
*** fragatina has joined #openstack-kolla16:08
*** senk has quit IRC16:09
*** fragatina has quit IRC16:09
zhubingbing__ok16:09
dave-mccowanzhubingbing__ hello.  what questions to you have?16:10
zhubingbing__If I want to run Barbican what I should do with the configuration for barbican.conf, use the default on the line well?16:11
sean-k-mooneyok im back meeting over. how is the osic cluster going16:11
*** shardy_afk is now known as shardy16:12
*** yee_ has joined #openstack-kolla16:13
inc0pbourke, sean-k-mooney seems like we're good to kolla-go?16:14
pbourkeinc0: assuming the insecure registry is configured correctly16:14
pbourkeinc0: next step I think is to generate a build conf and start building16:14
sean-k-mooneyinc0: cool16:15
openstackgerritzhubingbing proposed openstack/kolla: Add Barbibcan container dockerfile  https://review.openstack.org/35182216:15
sean-k-mooneypbourke: if you used my kolla-host playbook on 14.04 i will not have updated the insecur registry. that support is currently missing for non systemd OSs but ill be adding it in the next version16:15
pbourke:)16:16
sean-k-mooneypbourke:  eta 30-60 mins16:16
pbourkejust noticing that now16:16
inc0well...no insecure registry in /etc/default/docker tho16:16
pbourkeno worries16:16
pbourkeinc0: sorry should just have used your method16:16
inc0:)16:16
pbourkeactually16:16
sdakepbourke can we can timings via the time command on the dependnecy deploy operation16:17
pbourkewe could just start building and pick up sean's change when it's ready16:17
pbourkesdake: yeah will start doing that16:17
*** yee_ has quit IRC16:18
*** kbaegis has joined #openstack-kolla16:19
inc0crap16:22
*** daneyon has joined #openstack-kolla16:23
inc0ehh...16:24
*** fragatina has joined #openstack-kolla16:24
duonghqinc0: I'm sorry but can you or somebody confirm my bug report: https://bugs.launchpad.net/kolla/+bug/160950216:24
openstackLaunchpad bug 1609502 in kolla "Container bootstrap_mariadb exits due to Galera failed" [Undecided,New]16:24
duonghqI try and try many time16:24
duonghqbut I still can not figure out any wrong with me16:24
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Nova  https://review.openstack.org/35167616:26
sean-k-mooneyinc0: broke something?16:26
inc0already fixed16:26
inc0ok insecure registry is setup16:27
inc0pbourke, wanna do the build honors?16:27
*** daneyon has quit IRC16:27
pbourkesure just gonna check the reg one last time16:28
inc0duonghq, docker ps please16:29
*** coolsvap has joined #openstack-kolla16:29
sdakecoolsvap do you have any bandiwdith to triage https://bugs.launchpad.net/kolla/+bug/160950216:30
openstackLaunchpad bug 1609502 in kolla "Container bootstrap_mariadb exits due to Galera failed" [Undecided,New]16:30
sdakeSatya__ sorry had to step awa yfor a moment - are you all set then?16:30
Satya__yes Thanks sdake16:31
Satya__just one query16:31
coolsvapsdake: will check later tonight16:31
Satya__why nova thought of removing the flavors?16:31
duonghqirc0: I'm sorry but I do not have access to my developing computer right now, many sorry16:31
sdakeSatya__ the discussion i saw about it was they were picked arbitrarily to copy aws, and they are not really useful for anyone except developers, so they were removed since operators by default just delete them anyway16:32
duonghqinc0: I'll retry and provide more information later16:32
sdakeSatya__ yo uwould relaly have to ask on #nova for more details though16:32
Satya__ok sure16:32
sdakeduonghq work with coolsvap later tonight - he is on your timezone iiuc16:33
sdakeduonghq when you may have access to your gear16:33
*** kbaegis has quit IRC16:33
duonghqsdake: okay, thank you16:33
*** kbaegis has joined #openstack-kolla16:35
inc0we should time it16:35
inc0pbourke,16:35
inc0and yeah, after we moved to full threads ^C is broken, anyone want to fix?16:36
duonghqsdake: who has responsible for kolla namespace in docker hub?16:36
inc0pbourke, I'll submit a bug for it16:37
pbourkethanks inc016:37
inc0duonghq, core team16:37
sdakeduonghq core reiew team16:37
pbourkewas surprised no one was complaining so thought maybe it was just me16:37
sdakeare ou talking aobut build not accepting ^c?16:37
sdakei noticed that - ANNOYING16:37
pbourkeyeah16:37
inc0nah, not just you16:37
inc0yeah sdake16:37
sdakehae to ctrl-z kill -9 %116:37
pbourkekill % is shorted ;)16:37
pbourke*shorter16:38
sdakewfm16:38
sdakethat needs fixing16:38
inc0harlowja, ^ ;)16:38
*** egonzalez90 has quit IRC16:38
openstackgerritMauricio Lima proposed openstack/kolla: Customizations for Manila  https://review.openstack.org/35060116:38
inc0https://bugs.launchpad.net/kolla/+bug/161032416:39
openstackLaunchpad bug 1610324 in kolla "build doesn't accept ^C" [High,Confirmed]16:39
zhubingbing__duonghq16:39
inc0that being said, build is moving quickly16:39
zhubingbing__i havre a registry for kolla16:40
zhubingbing__i have16:40
*** JoseMello has quit IRC16:40
duonghqI think somebody should get images' tag update to master branch source code?16:41
inc0duonghq, unless we'll have automatic pushing, it's always going to be outdated16:41
inc0automatic pushing is something we want to have...one day16:41
inc0but it's not trivial16:41
sdakeduonghq we are not pushing master for some time16:41
zhubingbing__INFO:__main__:Removing existing destination: /etc//my.cnf16:42
zhubingbing__INFO:__main__:Copying /var/lib/kolla/config_files/galera.cnf to /etc//my.cnf16:42
sdakeduonghq if you want master you have to build on your own16:42
pbourkeinc0: looking good. Think I'll call it a day here16:42
pbourkeinc0: will drop a mail to the list so people know where we're at16:42
zhubingbing__duonghq  i think you are wrong configuration file path16:42
inc05min 30s for --profile default16:43
* inc0 likes it16:43
duonghqsdake, inc0: I think it outdate for some day is ok but for more than a week is quite long, especially for major tag like 3.0.016:43
inc0duonghq, so master is meant only for developers really16:43
inc0and developers should build their own images16:43
inc0that's my take on it16:43
*** egonzalez has joined #openstack-kolla16:44
*** senk has joined #openstack-kolla16:44
sdake3.0.0 is not a thing yet duonghq  :)16:44
duonghqinc0: should we update the getting start page to notice this aspect?16:44
*** egonzalez is now known as egonzalez9016:44
sdakethis is why Daviey 's approach of using "current" is more appealing to me then calling our 3..0.0 tag 3.0.0 when its nto really 3.0.016:44
sdakeduonghq docs here would help16:45
duonghqsdake, inc0: it is absolute ok to do source deployment, but the point it the documentation should point out that16:45
duonghqsdake, inc0: can I update the docs to reflect this aspect?16:46
sdakeduonghq dont ask for permission, ask or forgiveness :)16:47
duonghqokay :)16:47
inc0duonghq, of course you can16:48
inc0you can publish whatever changes you want, worst case scenerio we won't merge it16:48
duonghqinc0: sure :)16:49
inc0pbourke, make sure to include our build time in the mail16:49
inc0#bragforkolla16:49
inc0#buildporn16:49
pbourkeinc0: I did :) it's brief enough as pretty tired, hopefully does the job anyway16:50
*** senk has quit IRC16:50
inc0soo...who wants to see kolla deploying 130 nodes?16:50
inc0sdake?16:50
inc0britthouser?16:51
sdakei'd love to - have meeting in 9 minutes for 1 hour16:51
sdakedont wait for me16:51
inc0kk\16:51
sdaketimings would be nice16:51
inc0ofc16:51
sean-k-mooneyinc0: are you going to make it a google hangouts16:52
pbourkeinc0: keep general notes on the pad, dont want to miss all the fun :)16:52
sean-k-mooneyinc0: by the way have you ran the precheks to see if they pass first on the 130+ nodes16:52
duonghqinc0: how can I see?16:52
pbourkeheading off, chat later16:53
*** fragatin_ has joined #openstack-kolla16:53
inc0hmmm16:53
inc0interesting16:53
inc0fouind first bug in kolla16:54
inc0in osic16:54
sdakewow cold in phoenix for first time in 4 months16:55
britthouserin a meeting right now inc0....gimme a couple mins16:55
*** fragatina has quit IRC16:55
*** kbaegis has quit IRC17:01
*** srwilkers has quit IRC17:01
*** zhubingbing__ has quit IRC17:02
*** duonghq has left #openstack-kolla17:02
britthouserOk I'm free inc017:03
britthouserOn your mark....get set....17:03
inc0soo17:05
britthouseryes!17:05
inc0it seems our little ansible_{{ api_interface }} can't grok a dot17:05
britthouserew17:06
*** kbaegis has joined #openstack-kolla17:06
britthouserso no VLAN subinterfaces?17:06
inc0I consider this a critical bug17:06
britthouserYeah17:06
inc0I'm trying to figure out how to get ansible to behave and publish a patchset today17:07
kbaegisYou want sharded interfaces rather than ovs?17:07
britthouserthis isn't for tenant network kbaegis, rather for the services networks17:07
britthouserlike how you ssh into the baremetal17:08
inc0nah. false alarm17:11
inc0it's all good17:11
*** diogogmt has quit IRC17:11
inc0we messed sth with globals17:11
harlowjainc0 well it does accept ctrl-c17:11
harlowjait just that it doesn't try to crap out randomly and not finish up cleanly17:11
inc0maybe add something like "2*ctrl+c == die, just die"17:12
inc0funny it works now17:13
*** yee_ has joined #openstack-kolla17:13
*** diogogmt has joined #openstack-kolla17:14
* britthouser scratches my ehad17:14
*** mewald has joined #openstack-kolla17:14
sdakeinc0 groan17:14
sdakeinc0 criticial17:14
inc0sdake, false alarm17:14
sdakeok17:15
inc0I have no idea what was wrong but it seems ok now17:15
inc0doing prechecks17:15
inc0and will see if I can record hangouts17:15
*** signed8bit is now known as signed8bit_Zzz17:15
rhalliseyinc0, hangouts link?17:16
inc0trying to get17:17
inc0https://hangouts.google.com/call/c7fascv5bzgnzjpcaopqk2snjee17:17
sean-k-mooneymind if i join17:18
rhalliseysean-k-mooney, go for it17:18
sdakesean-k-mooney all our work is oepn to all17:18
rhalliseyinc0, go through google plus17:20
sdakepbourke wow 5min build?17:20
*** yee_ has quit IRC17:20
sdakeosic network on steroids :)17:20
rhalliseywhen you start a hangout there's a box that says enable hangounts on air17:21
sean-k-mooneysdake: 5 min build nice. didnt you say it took you 8min at home with your 1G fiber line and nvme storage17:21
sdake15min for all17:22
sdakeut i ont build with the same profile as paul17:22
*** signed8bit_Zzz is now known as signed8bit17:22
sean-k-mooneyin that case 5mins is awsome17:22
sdakeor perhap i do17:22
sdakenot sure17:22
inc0https://plus.google.com/events/cib8giqjovmtfmesbe7h2foug1k17:22
sdakei build without profile option- i  guess that would be default17:22
inc0http://youtu.be/InU0Up_5H2417:22
rhalliseylol17:22
rhalliseyit has a youtube like nice17:23
inc0yeah17:23
inc0it's gonna be cool17:23
*** diogogmt has quit IRC17:23
inc0although if we fail our deploy it's going to be recorded forever;)17:23
inc0anyway, lets try shall we?17:23
sean-k-mooneyinc0:  your youtube carrer starts  :) next you will be on twitch17:24
sean-k-mooneyand yes17:24
sean-k-mooneywe can hear you on youtube but we cant join the hangeout and talk as far as i know17:25
rhalliseyinc0, timer?17:26
rhallisey1:26 start17:26
inc0I think I timed it17:26
sean-k-mooneyinc0: you did17:26
inc0we need to tweak ansible confs tho17:27
Mech422I am soo sick of networking17:27
inc0move to mountains and start herding sheeps Mech42217:27
Mech422I _think_ I almost have my dev lab mimicking the screwed up prod configs though17:27
sean-k-mooneyso how long do you think the deploy will take out of interest?17:27
Mech422inc0: actually, the VP of Olliance (SFO Open Source consulting company) actually moved to wisconsin(?) and started raising goats when we folded...17:28
inc0sean-k-mooney, 64 nodes took ~25min17:28
inc0so I'd guess circa 35?17:29
sdakegoats17:29
sdakewhat an odd profession to choose17:29
sdakemy cousin who has 20 mil became a rancher after he sold his business off17:29
sean-k-mooneyinc0: thats pretty good scaling17:29
inc0well compute nodes are faster than controller nodes17:29
sdakesean-k-mooney cmopute nodes take less time then control nodes to deploy17:30
Mech422sdake: apperently, he makes world-renowned goat's milk cheese now...17:30
inc0we'll see soon enough17:30
*** diogogmt has joined #openstack-kolla17:30
sean-k-mooneyinc0: im guessing one of the bottlenecks will is coping the 5 or 6 imanges for the compute nodes17:30
inc0yeah might be17:30
inc0we'll see17:30
Mech422sdake: but he made $$$$ on webmd, so ....17:30
inc0also we should tweak ansible forks17:31
inc0however this is as simple as it gets, we'll make series of videos, one of which will be "how fast it can get"17:31
sean-k-mooneyinc0: it might be interesting to do a cleanup and redeploy the same config with the prechached images from the first deploy and compare the difference17:32
inc0so with all the plays already done17:32
inc0we can redeploy from bare metal at any point17:32
inc0which is cool17:32
inc0see? definetly we need more forks17:33
sean-k-mooneyforks of kolla?17:34
inc0no, of ansible17:35
inc0so if we tweak ansible a bit it can be WAY faster17:35
inc0after we're done with this one, I'll do the tweaks17:35
sean-k-mooneyah, got ya17:35
sean-k-mooneyone thing i never looked into what does teh kolla-toolbox container actully do?17:39
inc0sean-k-mooney, so historically we had to have ansible 2.0 for shade - ansible module for openstack17:42
inc0so all keystone user creation and stuff could be done17:42
sean-k-mooneyoh ok so now that we use ansibel 2 anyway is it still needed?17:43
*** inc0_ has joined #openstack-kolla17:43
*** JoseMello has joined #openstack-kolla17:44
inc0_aaand deploy failed17:45
inc0_aaaand I know why17:46
Mech422grr...I have a really wierd ovs error where its not adding the physical interface to the ovs bridge on my dev nodes..17:46
Mech422ubuntu ovs is SUCH a PITA17:46
*** inc0 has quit IRC17:46
inc0_so I'll tweak it up and restart transmission17:46
inc0_also make this ansible tweaks17:46
inc0_we forgot to add /etc/hosts17:47
sean-k-mooneyadd?17:48
inc0_so rabbitmq can only work with hostnames17:49
inc0_so all of controller nodes have to resolve it's hostname17:49
inc0_each others*17:49
britthouserdoh!17:49
sean-k-mooneyok i taught i had that in my kolla-host playbook17:49
sean-k-mooneyoh wait they need to have the hostnames of all other controler nodes too?17:50
inc0_yeah17:50
inc0_every controller has to resolve every other controller17:50
inc0_and make it api_interface17:51
sean-k-mooneyok well thats a change i need to make in the kolla-host playbook too. ill do that17:51
britthouserI've seen that with Ceph too....at least with ceph_ceploy17:51
inc0_kk17:51
britthousereveryone needs to resolve the hostname of the MON ips17:51
inc0_first outcome of our little testing!17:52
britthouserw00t!17:52
inc0_also we should add this precheck to well..precheck17:52
*** fragatin_ has quit IRC17:52
sean-k-mooneyinc0_:  should i add entries for all nodes into /etc/hosts?17:53
sean-k-mooneyi had that origiinally but took it out17:53
inc0_yeah17:53
*** fragatina has joined #openstack-kolla17:53
*** yee_ has joined #openstack-kolla17:55
britthouserinc0 and pbourke I'm starting to wonder if we'd really get any gains by reformatting with different profiles?18:00
*** Guest52040 has quit IRC18:00
britthouserat least not until step318:00
inc0_http://youtu.be/RNZMtym5x1c next try18:00
britthouserLets just keep it simple and keep everyone on generic unless they are ceph18:00
inc0_well we need lvm for cinder tho18:01
*** shardy is now known as shardy_afk18:03
inc0_it's faster now18:04
sean-k-mooneyinc0_: wow that does look faster18:04
sean-k-mooneywhat did you do?18:04
inc0_these ansible tweaks18:05
inc0_so we can acutally put this into your playbook18:05
openstackgerritDuane DeCapite proposed openstack/kolla: Change cleanup to remove as cleanup is a misnomer  https://review.openstack.org/35186918:06
inc0_what I did is to add 2 config options to ansible18:06
*** jemcevoy has quit IRC18:06
*** michauds has quit IRC18:06
inc0_pipelining=True and forks=10018:06
sean-k-mooneyinc0_: ah cool18:06
sean-k-mooneycould you set those in the global.yml or somting along those lines18:07
inc0_hostnames, again18:07
sean-k-mooneyrabiit again?18:07
inc0_ah.18:08
inc0_it checks hostname not fqdn18:09
inc0_here we go again18:10
inc0_sooo we need to add this to prechecks and remove from deploy18:11
inc0_as it is really a precheck18:11
*** daneyon has joined #openstack-kolla18:11
inc0_and also add this to kolla-host18:11
*** daneyon has quit IRC18:16
inc0_nova done18:24
inc0_I think we'll get less than 20min18:24
*** yee_ has quit IRC18:24
Davieysdake: hey, you would mind readding your +2 (& +W) for visit https://review.openstack.org/337559  (it got lost because someone changed the commit message)18:25
*** yee_ has joined #openstack-kolla18:26
sdakeDaviey looking18:27
*** senk has joined #openstack-kolla18:27
sdakeDaviey acked18:30
inc0_damn18:30
inc0_DAMN!18:30
inc0_more than 20min18:30
inc0_20min0.618s18:31
sean-k-mooneyoh i was just going to say that looks like it succeded18:31
sdakeinc0 was that af ull130 node deploy?18:31
rhalliseyinc0_, done18:31
inc0_yeah18:31
inc0_sdake,18:31
rhalliseywow18:31
sdakewow18:31
sdakehotness18:31
sdakethe 1$ queeetion is does it work :)18:31
*** narasimha_SV has quit IRC18:32
sean-k-mooneyinc0_: didnt you say that a 64 node deploy took 25. those ansible tweaks seem to have helped alot18:32
inc0_yeah18:32
Davieysdake: thanks sir18:32
sean-k-mooneysince rhallisey did a 5min build and that was a 20 min deploy that less then half an hour if you have the configs ready18:32
inc0_yup18:33
sdakebut we dont know how long bare metla takes?18:33
inc0_on a pretty damn big cluster18:33
Davieyinc0_: what is the ansible server system load like?18:33
inc0_sdake, much much longer...single node boots longer then this full deploy18:33
inc0_Daviey, didint check, will check next time18:33
sean-k-mooneyinc0_: how long did the cobbler install take?18:34
Davieyinc0_: Locally, i've seen it > 10.00.. on a medium sized VM18:34
DavieyNot sure if it is poor ansible tuning or something else18:34
sdakewhat is being deployed inc0 - compute kit?18:34
inc0_Daviey, this is pretty powerful metal tho18:34
inc0_sdake, yup, as default as it gets18:35
inc0_vm is running18:36
openstackgerritMerged openstack/kolla: Add Watcher default settings  https://review.openstack.org/33755918:36
sdakeso answer is yes18:36
*** gfidente has quit IRC18:36
inc0_and vm is pinging18:37
inc0_damn my voice sounds horrible18:37
sdakeinc0_ how many control/storage/compute nodes?18:37
inc0_4 controllers18:38
inc0_shitload compute18:38
inc0_some storage18:38
britthouserw00t!18:39
rhalliseyinc0_, live stream gone18:39
sean-k-mooneyhttps://youtu.be/RNZMtym5x1c18:39
sean-k-mooneyyou can still watch it back after the fact18:40
inc0_yeah18:40
sdakeis the storage doing anything?18:40
sdakeany cinder or ceph configured?18:40
inc0_let's record the shit out of kolla18:40
inc0_I don't think so18:40
inc0_cinder was disabled18:40
sdakeso the storage nodes were a noop18:40
sean-k-mooneysdake: if they were just storage then yes18:41
sean-k-mooneyinc0_: were the stoarge nodes also compute nodes?18:41
inc0_no18:41
inc0_but there are only few of storage18:41
britthouserJust a reminder I'm travel for work next week, but I'll check back and see where OSIC stuff is a week from monday, and pitch in where I can.18:42
inc0_fuck, I'm proud of our project, and I mean really, really proud18:42
britthouserYeah this is amazing results18:42
*** senk has quit IRC18:44
sean-k-mooneywill i put the video link into the etherpad?18:44
sdakesean-k-mooney sure18:44
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Introduce a script to launch ovsdb-server process  https://review.openstack.org/35178418:45
*** zhubingbing has joined #openstack-kolla18:46
inc0_sdake, 101 compute18:47
zhubingbinghi,who can help me,I don't know why he appears this Merge Conflict ;https://review.openstack.org/#/c/351027/18:49
patchbotzhubingbing: patch 351027 - kolla - Add aodh role18:49
openstackgerritDavid Wang proposed openstack/kolla-kubernetes: Added support for template variables  https://review.openstack.org/35140818:53
sean-k-mooneyzhubingbing: i would guess a change has merged to master that your change is in conflict with18:55
sean-k-mooneyzhubingbing: can you try rebaseing to master?18:55
zhubingbingno19:02
zhubingbingi try19:03
zhubingbingroot@ip-172-31-22-78:/kolla# git rebase master19:05
zhubingbingCurrent branch bp/BP-sahara is up to date.19:05
zhubingbing  it look so fine19:05
*** egonzalez90 has quit IRC19:05
Davieyrhallisey: hey.. I've been experimenting with co-locating compute and storage on the same nodes.  I see on the OSIC pad you are down for this.. Does it interest you?19:06
irtermiteinc0_: someone needs to bump their terminal size before tmuxing.   ;)19:06
pbourkeyo19:06
pbourkei saw some good news :)19:07
pbourketrying to watch the video, wont load though19:07
irtermitewatching now19:07
sean-k-mooneyzhubingbing: you need to do a get fetch then git rebase origin/master19:07
zhubingbingok i try19:08
sean-k-mooneyzhubingbing: git rebase master will just rebase to your local master which is likly behind upstream19:08
zhubingbingmay be I git used improperly19:09
inc0_pbourke, irtermite made few remarks about your terminal size...and I must agree:P19:09
pbourkeha19:10
sean-k-mooneyinc0_: it was a little hard to see /read19:10
irtermite*sad trombone*19:11
irtermitepbourke: inc0_: http://www.tecmint.com/record-and-replay-linux-terminal-session-commands-using-script/19:11
irtermite  ;)19:11
zhubingbinggit rebase --continue19:11
zhubingbing[detached HEAD 059c55c] Add sahara ansible roles19:11
zhubingbing 19 files changed, 415 insertions(+19:11
zhubingbingnext, i need  user git review  command?19:11
openstackgerritzhubingbing proposed openstack/kolla: Add sahara ansible roles  https://review.openstack.org/35129419:12
sean-k-mooneyzhubingbing: yes if you have resoved and commited any conflit then do git review19:12
sean-k-mooneyi see you already have :)19:12
zhubingbingok thanks19:12
zhubingbingI should be good to learn Git19:13
rhalliseyDaviey, sure19:13
zhubingbingthank you sean-k19:13
openstackgerritzhubingbing proposed openstack/kolla: Add aodh role  https://review.openstack.org/35102719:17
zhubingbingthank you once again sean-k-mooney  -)19:17
*** dwalsh has joined #openstack-kolla19:18
sean-k-mooneyzhubingbing: no worries hopefully that will reslove your merge conflict19:18
zhubingbingI hope this submission will not appear again merge conflict19:21
zhubingbing- -)19:21
*** sdake has quit IRC19:26
*** sdake has joined #openstack-kolla19:28
sean-k-mooneyinc0_: pbourke. i just got my  kolla-host playbook to pass precheck on centos and ubuntu16.04 is there a way to make /run mounted shared on ubuntu 14.04 that will survie a reboot?19:28
sdakesean-k-mooney there is19:28
sdakesean-k-mooney but i dont know how to do so19:28
inc0_so I'm ok with requiring 16.04 from newton on19:29
sean-k-mooneythe last i checked the docs only had the runtime way but ill look it up19:29
inc0_by requiring I mean "if you have 14, still works but we're not going to specifically support it"19:30
sean-k-mooneyinc0_: well 14.04 wont be supported with newton by cannonical so i think that makes sense19:30
inc0_yeah, so don't try too hard to fix it19:30
inc0_it's a non-blocker for me19:30
sdakeinc0_ i htink we have jointly made that decision already but we should clarify via a vote on the ml19:30
sean-k-mooneywell im fixing it because we are using it in the osic cluster more then anything else19:30
sean-k-mooneyill put in the runtime fix for now which is in our docs19:31
inc0_sdake, yeah, that being said, we need people to move our containers to 16.0419:33
inc0_I think it was ccesario who did this19:33
inc0_but it kinda died out and I haven't seen him around19:33
sean-k-mooneyinc0_:  do we know if anyting needs to be changed?19:34
inc0_well sources list for sure19:34
*** dwalsh has quit IRC19:34
inc0_apart from that hopefully nothing19:34
inc0_but well..."hopefully"19:34
openstackgerritzhubingbing proposed openstack/kolla: Add Barbibcan container dockerfile  https://review.openstack.org/35182219:35
sean-k-mooneyya i gues it would be worth a try.19:35
*** zhubingbing has quit IRC19:40
sean-k-mooneysdake: redarding the storage driver on centos. does aufs work?19:42
Mech422inc0_: if I get my current project situated, I can play with ubuntu 16.04 - we're an ubuntu shop19:42
inc0_would appreciate it19:42
inc0_I'd gladly take it but I'm overloaded19:43
*** yee_ has quit IRC19:43
Mech422inc0_: yeah - no promises, but I'll try to give it a shot19:43
Mech422inc0_: also, looking at scale.io was added to my marching orders19:43
*** dave-mccowan has quit IRC19:45
*** dave-mccowan has joined #openstack-kolla19:45
Davieysean-k-mooney: I think you are supposed to use overlayfs in CentOS.  AIUI, AUFS support isn't even in RH/Centos19:46
*** mliima has quit IRC19:46
inc0_also ubu16.04 doesn't like aufs I think19:46
inc0_I run overlay there19:46
DavieyMech422: I'm strong on Ubuntu, but I don't have the cycles.  I can help with adhoc tasks, but i certainly couldn't own it19:47
sean-k-mooneyDaviey: i was going to have overlay be the default but sdake said he did not think it was stable enough for production19:47
Davieyinc0_: aufs can die in a fire.19:47
Mech422inc0_: oh? I've never actually setup a FS driver - I just use the default docker installed one19:47
inc0_well, it's still better than devicemapper19:47
sdakeMech422 thats painful :)19:47
inc0_oh my Mech42219:47
Davieysean-k-mooney: I'd prefer the suggestion of setting up btrfs via lvm for centos :)19:47
inc0_Mech422, do docker info19:48
sdakethe suggestion was raw btrfs19:48
inc0_docker info | grep Storage Driver19:48
Mech422last I heard, btrfs still wasn't quite stable ?19:48
inc0_it's ok afaik19:48
inc0_it's in kernel now19:48
sean-k-mooneyi would prefer btrfs too but that is a little tricky if your os image is not installed with a btrfs filesystem19:48
sdakelets not compound that perception with lvm :)19:48
Mech422inc0_: err..I don't have a docker machine up atm - I'm rebuilding cluster for old style deployment19:48
Davieybtrfs is the best of the options.  It was announced today that RAID5/6 support was flakey, but the core stuff for what we want is fine19:49
inc0_and I think xenial actually installs btrfs on server19:49
sdakesean-k-mooney make a /var/lib/docker that is btrfs19:49
sdakethat is what i  do19:49
inc0_Mech422, if it's devicemapper19:49
inc0_change it.19:49
inc0_cleanse it with fire19:49
sean-k-mooneysdake: how do i do that without mounting a seperate partition there or a loopback file?19:49
Davieysdake: LVM makes it simpler if you ask me... considering the default install patern for rpm distros is to use LVM.. We just need to tack on the end of that19:49
DavieyIf we do that, it becomes almost as simple as loopback19:49
sdakeif it works it wfm :)19:49
sdakei never test with lvm19:50
sdakei use raw btrfs19:50
*** dave-mccowan has quit IRC19:50
*** gfidente has joined #openstack-kolla19:50
inc0_well you need a device for btrfs19:50
inc0_that's the issue19:50
Mech422inc0_: looks like ubuntu 14.04 defaults to aufs driver19:50
sean-k-mooneysdake: i use raw brtfs too but i also install my os that way19:50
Davieywell lvm itself is irrelevant.. just a different way of presenting the volume to btrfs19:50
Mech422inc0_: at least, thats whats running on my home setup19:50
inc0_aufs is kinda ok19:51
inc0_I think older versions of docker defaulted to devicemapper19:51
inc0_anyway, if you ever see devicemapper19:51
inc0_burn it19:51
Davieyinc0_: I thought all default to dm?19:51
sean-k-mooneyubuntu 16.04 is back to devicemapper by default19:51
inc0_duh19:51
sean-k-mooneyinc0_:  devicemapper is fine if you use a dedicated disk19:51
Davieywhen I was at Canonical, we tried to negotiate with docker to stop using aufs19:52
sean-k-mooneyinc0_: its the loopback file that kills performance19:52
inc0_uf you use dedicated disk use btrfs19:52
inc0_aufs can die too, as soon as overlay becomes a thing19:52
inc0_when you have overlay, no need to aufs at all19:52
Davieyoverlay is a thing.. it's mainline now19:52
inc0_I know19:53
inc0_I run overlay in 1619:53
Mech422our stuff is mainly for QA lab - so I'd rather be 'slow and stable' then 'fast with lots of edge conditions'19:53
Mech422actual vm performance is not a huge deal for QA19:53
inc0_well, I don't think you'd hit stability issues with btrfs19:54
Mech422thats why I've shied away from btrfs...I keep hearing its awesome...till its not19:54
DavieyMy understanding is that loopback with docker with deteriorate massively over time19:54
inc0_what you'll get tho is 10x build boost19:54
Davieybut i don't have anything to substantiate that19:54
sean-k-mooneywell the variable i added to the kolla-host playbook just take the storage driver name so you can use any of them19:54
Mech422inc0_: hmm - might have to give it a try then19:54
inc0_overlay is ceph go-to fs19:55
inc0_so it's a good idea too19:55
DavieyMech422: well you aren't storing anything persistent on the docker engine anyway are you?19:55
DavieyMech422: Surely using ceph for storage.. the docker mount is largely ephemeral ?19:55
Mech422Daviey: emm - not using docker in prod yet, so no...19:55
DavieyMech422: I mean, for your kolla deployment19:55
Mech422Daviey: umm - mysqld rabbitmq etc etc ?19:56
DavieyIf the docker storage engine catches fire, you can blow it away and rebuild without data loss. right?19:56
DavieyRight, but they are clustered19:56
DavieyThe only thing that isn't is mongodb, right?19:56
Mech422inc0_: when did ceph go with overlay? I thought it was btrfs/xfs recommended ?19:56
DavieyMech422: Sorry, i was at cross-purpose.. I mean your client data, such as kvm volumes19:56
sean-k-mooneyDaviey: we dont configure our docker volumes to use a prestent backend so without clusering you would lose data19:56
*** ppowell has joined #openstack-kolla19:57
Mech422Daviey: yeah - client data is all ceph backed19:57
Davieysean-k-mooney: I mean, what permanent data needs to be in docker?19:57
*** dave-mccowan has joined #openstack-kolla19:57
DavieyA failed docker storage driver, is the same as the node catching fire.  Just rebuild it.19:57
sean-k-mooneyyour mysql db for one19:57
Davieysean-k-mooney: but that is clustered19:57
*** kbaegis has quit IRC19:58
DavieySo, unless you see 3 docker storage engines failing concurrently?19:58
*** kbaegis has joined #openstack-kolla19:58
sean-k-mooneyyes hence why i said without clusering. i dont think there is anything that would be lost with a default kolla deploy except local heka logs19:59
Mech422Daviey: I though btrfs/ceph/zfs all had issues when disks near capacity....which is something I'd expect to happen on all 3 nodes pretty much at the same time ?19:59
DavieyThis is interesting reading for RH/Centos 7, and overlayfs - https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/7.2_Release_Notes/technology-preview-file_systems.html19:59
inc0_sean-k-mooney, well, logs lands in named volume19:59
inc0_db is on named volume19:59
DavieyMech422: So, to counter that.. My 3 control nodes are on different configurations... two are virtualised and one is bare metal20:00
sean-k-mooneyinc0_: yes but that named volume is not preseted so if you have local storage issue its gone20:00
DavieyMech422: so yes, if they are identical.. i agree20:00
Mech422Daviey: mine are all basically clones20:00
DavieyMech422: then doesn't matter what storage driver you are using, you are risking an outage with simple disk filling up20:02
Mech422Daviey: true - but I can monitor absolute disk usage, I can't monitor "how much does this want to not freak out"20:03
Mech422Daviey: as that tends to be variable depending on cluster usage20:03
Davieymeh, i think for our usage btrfs is pretty well tested.20:03
DavieyWe aren't pushing the envelope or using advanced RAID functionality where it is weaker20:04
Mech422Daviey: I'll give it a shot when I have a chance...I just have way too many other things that are higher priority atm20:04
Daviey:).. find that 25th hour in the day20:06
Mech422Daviey: btw - now canonical is shipping zfs, I wonder how uptake/usage of the 2 will differ20:06
Mech422zfs seemed a lot more 'battle tested', but btrfs has a really nice feature set, and a lot of followers20:07
inc0_so on that note, I wonder if you could use rbd for named volume in docker20:07
DavieyMech422:  yeah!  (I use zfs on ubuntu home server, been rock solid for 3 years)20:07
Davieyinc0_: funny you say that... I almost did a few mins ago.. but thought it might be a bit too insidious20:07
inc0_why is that?20:08
Davieyinc0_: it isn't an entirely silly idea.  Makes the node more stateless and removes the issue we have right now of setting up the filesystem20:08
Davieyit isn't like the container has high write throughput20:09
sean-k-mooneyinc0_: ya i looked at that breifly20:10
*** kbaegis has quit IRC20:11
inc0_Daviey, that's what we'll be doing in k8s20:11
sean-k-mooneyinc0_: the issue is with the ceph containers20:11
inc0_yeah, it has to be external ceph20:11
inc0_or ceph logs would be only ones not on ceph20:11
inc0_because well...egg chicken?;)20:11
sean-k-mooneyinc0_: i said it before but it might be worth executing the ceph role form kolla-host and then using external ceph with kolla ansible at some point20:12
sean-k-mooneyinc0_:  the ceph logs would still have to be stored on the host though20:14
inc0_well, they will go through heka to elastic20:14
inc0_anyway, ceph is not stateless at all20:14
inc0_as osd really don't like to be moved around20:14
inc0_doesnt20:14
*** yee_ has joined #openstack-kolla20:17
openstackgerritOpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements  https://review.openstack.org/35190120:22
*** michauds has joined #openstack-kolla20:25
openstackgerritPatrick Powell proposed openstack/kolla: Add Nova sanity check  https://review.openstack.org/34741620:26
*** mewald has quit IRC20:29
*** yee_ has quit IRC20:29
openstackgerritPatrick Powell proposed openstack/kolla: Add Heat sanity check  https://review.openstack.org/35007120:30
Mech422inc0_: sdake: that reminds me - now pbourke is back, we should figure out the sgdisk thing in the ceph startup20:30
Mech422its hard coded to look for /usr/sbin/sgdisk but for ubuntu its /sbin/sgdisk, and sdake doesn't want to use shell path to find it20:30
*** dwalsh has joined #openstack-kolla20:33
Mech422personally, I think thats what the shell path is for - if someone owns your containers, you have bigger issues - but we could just test existance of /usr/sbin/sgdisk and fallback to /sbin/sgdisk if it doesn't exist20:34
Davieyinc0_: Sorry, when i said stateless.. i meant the docker storage, not the osd's20:34
inc0_yeah, but we deploy ceph as well20:34
Mech422just not sure where other distros put the binary ?  could end up with a bunch of ladder logic looking for it ?20:34
inc0_discussion was, how to use ceph we deploy for our own deployment;)20:35
*** Jeffrey4l has quit IRC20:35
*** Jeffrey4l has joined #openstack-kolla20:36
Mech422inc0_: I like the idea of 'pre-deploying' ceph, then just always pretending ceph is 'external'20:37
Mech422inc0_: it gives a single code path, without having to do internal vs external checks20:38
*** jtriley has quit IRC20:38
*** shardy_afk is now known as shardy20:38
inc0_yeah, that's the idea20:38
*** ppowell has quit IRC20:42
Mech422Daviey: btw, have you noticed any problems rebooting ubuntu boxes if you radically change /etc/network/interfaces ?20:43
Mech422Daviey: I use ansible to convert from a linux networking to openvswitch network setup, and it always seems to get stuck on reboot trying to shutdown the network20:43
*** Satya__ has quit IRC20:44
Mech422Daviey: its understandable its confused that interfaces disappeared/changed/whatever - but annoying it prevents the reboot20:44
DavieyMech422: what do you mean by radically ?20:44
DavieyMech422: i don't think i've ever used ovs on the bare metal20:45
Mech422Daviey: ahh20:45
*** banix has quit IRC20:45
DavieyMech422: why are you OVS'ing the bare metal?20:46
Mech422Daviey: because I can ?20:46
Mech422Daviey: production uses ovs bridges, bonds, vlans, etc20:46
Mech422all on bare metal20:46
Davieyah, i see20:47
Mech422Daviey: its just a pet peeve - my foreman preseeds setup 'normal' linux networking - I just need to use that 25th hour in the day to create an ovs based preseed20:48
DavieyMech422: what connects to the ovs bridge?20:48
Mech422Daviey: err..in what way ? production has a bonded pair on one bridge, along with vlans...then I have an OVSPatch to an integration bridge for Openstack20:49
DavieyMech422: I suppose i am wondering if you are using it for more than the neutron bridge?20:50
Mech422Daviey: yes - since eth0/eth1 are bonded, the host mgmt interface is a vlan via ovs20:51
*** dwalsh has quit IRC20:52
Mech422Daviey: along with vlans for ceph/database replication, etc20:52
openstackgerritMerged openstack/kolla-kubernetes: Add doc for configuring private docker registry  https://review.openstack.org/35078520:52
Mech422Daviey: (this was a 'vanilla' openstack liberty install, so it doesn't use kolla standards... )20:52
DavieyMech422: So i use a bonded interface with vlan, and let neutron create the bridge to bring it into openstack20:53
Mech422Daviey: cool - that part works for me :-)20:53
*** daneyon has joined #openstack-kolla20:54
Mech422Daviey: its just annoying my dev. boxes don't reboot if I play fast and loose with /etc/network/interfaces20:54
Mech422I kinda think reboot should be "reboot!" (reboot dammit!)20:54
Mech422ipmi reboot fixes it though20:54
DavieyMech422: yeah.. weird.. no idea.. but you are right20:56
DavieyMech422: anything left on any of the tty's when it is wedged?20:57
*** dwalsh has joined #openstack-kolla20:57
*** daneyon has quit IRC20:58
*** shardy has quit IRC20:58
openstackgerritDavid Wang proposed openstack/kolla-kubernetes: Added support for template variables  https://review.openstack.org/35140820:59
Mech422Daviey: normal console shutdown messages then I think it get stuck on 'Shutting down networking' or something to that effect21:02
Mech422Daviey: its understandable - but it really should timeout and just finish rebooting21:03
openstackgerritsean mooney proposed openstack/kolla: add baremetal role to install kolla deps.  https://review.openstack.org/32563121:04
openstackgerritsean mooney proposed openstack/kolla: This change allow setting kernel commandline args  https://review.openstack.org/35141221:04
*** bmace has joined #openstack-kolla21:04
*** yee_ has joined #openstack-kolla21:04
sean-k-mooneyi still need to add the host name generate but the kolla-host playbook should now work on ubuntu 14.04,16.04 and centos7.221:05
Davieysean-k-mooney: I'm not sure i'm a fan of it.. but Ubuntu MAAS uses this for hostname generation, https://pypi.python.org/pypi/petname21:07
*** egonzalez90 has joined #openstack-kolla21:07
sean-k-mooneyDaviey: oh that is not what i ment21:08
sean-k-mooneyi need to populate the ip to hostname mapping in /etc/hosts for each of the contolernodes21:08
sean-k-mooneyotherwise rabbitmq will fail21:08
Davieysean-k-mooney: Oh... that21:09
*** JoseMello has quit IRC21:10
sean-k-mooneyi had code to do it in my first commit https://review.openstack.org/#/c/325631/1/ansible/roles/baremetal/templates/hosts.j2 but i removed it21:10
sean-k-mooneyi need to look at marks suggtion and see if i can use it21:11
*** sdake has quit IRC21:12
patchbotsean-k-mooney: patch 325631 - kolla - add baremetal role to install kolla deps.21:12
*** inc0_ has quit IRC21:13
*** yee_ has quit IRC21:13
*** michauds has quit IRC21:13
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Introduce a script to launch ovsdb-server process  https://review.openstack.org/35178421:15
*** yee_ has joined #openstack-kolla21:16
openstackgerritPrithiv proposed openstack/kolla: Dockerfiles for Networking-SFC Container  https://review.openstack.org/34493021:18
openstackgerritPrithiv proposed openstack/kolla: Dockerfiles for Networking-SFC Container  https://review.openstack.org/34493021:20
*** dwalsh has quit IRC21:21
openstackgerritChristian Berendt proposed openstack/kolla: Add missing project_yaml parameter to role metadata  https://review.openstack.org/35192821:22
*** jtriley has joined #openstack-kolla21:23
openstackgerritPrithiv proposed openstack/kolla: Dockerfiles for Networking-SFC Container  https://review.openstack.org/34493021:24
*** michauds has joined #openstack-kolla21:26
openstackgerritChristian Berendt proposed openstack/kolla: Enable CADF events in Keystone and introduce kolla_enable_enhanced_security  https://review.openstack.org/34900621:31
*** ad_rien_ has joined #openstack-kolla21:42
*** yee_ has quit IRC21:51
*** egonzalez90 has quit IRC21:52
*** yee_ has joined #openstack-kolla21:53
*** sdake has joined #openstack-kolla21:54
*** jtriley has quit IRC21:56
sdake50f in the morning 115 in the afternoon21:57
sdakebrutal temperature chane21:57
sdakechange21:57
sdakeinc0 ping22:05
sdakeguess not around22:05
*** signed8b_ has joined #openstack-kolla22:06
*** signed8bit has quit IRC22:09
*** ad_rien_ has quit IRC22:16
*** gfidente has quit IRC22:22
wirehead_yeah, I can get away without the AC today.  Nice.22:24
*** dave-mccowan has quit IRC22:24
*** ljjjustin has quit IRC22:33
*** ljjjustin has joined #openstack-kolla22:37
*** yee_ has quit IRC22:40
*** banix has joined #openstack-kolla22:41
*** imcsk8 has quit IRC22:50
*** imcsk8 has joined #openstack-kolla22:50
openstackgerritRevanth proposed openstack/kolla: Customizations for kolla-toolbox  https://review.openstack.org/35194622:54
*** signed8b_ is now known as signed8bit_Zzz22:56
openstackgerritKen Wronkiewicz proposed openstack/kolla-kubernetes: Fix Nova bootstrap  https://review.openstack.org/34645223:08
*** rhallisey has quit IRC23:13
*** sdake has quit IRC23:31
*** srwilkers has joined #openstack-kolla23:43
*** michauds has quit IRC23:44
*** ljjjustin has quit IRC23:45
openstackgerritDavid Wang proposed openstack/kolla-kubernetes: Add Ceph support for Mariadb  https://review.openstack.org/35031223:46
*** ljjjustin has joined #openstack-kolla23:49

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!