Thursday, 2016-09-29

openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805600:00
*** schwicht_at_work has quit IRC00:07
*** zhurong has quit IRC00:10
*** huikang has quit IRC00:10
*** huikang has joined #openstack-kolla00:11
*** huikang has quit IRC00:12
*** huikang has joined #openstack-kolla00:12
kfox1111sdake_: http://logs.openstack.org/56/378056/22/experimental/gate-kolla-kubernetes-deploy-centos-binary-nv/068e44a/console.html00:16
kfox1111kubernetes working in the gate. :)00:16
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805600:18
*** awiddersheim has quit IRC00:19
*** awiddersheim has joined #openstack-kolla00:22
*** hfu has joined #openstack-kolla00:24
*** sdake_ has quit IRC00:25
*** absubram has joined #openstack-kolla00:29
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805600:31
*** g3ek has quit IRC00:33
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902700:39
*** g3ek has joined #openstack-kolla00:42
*** neilus has joined #openstack-kolla00:42
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805600:43
*** absubram has quit IRC00:44
*** neilus has quit IRC00:47
*** MarMat has quit IRC00:48
*** absubram has joined #openstack-kolla00:51
*** absubram_ has joined #openstack-kolla00:52
*** sdake has joined #openstack-kolla00:52
sdakekfox1111 missed your message earlier - huge accomplishment!00:53
sdakeyou have learned how the gate works!00:53
sdakekfox1111 needs a bindep cleanup00:53
*** schwicht has joined #openstack-kolla00:55
sdakekfox1111 what is doing the deployment there?00:55
sdakekfox1111 note that gate failed: http://logs.openstack.org/56/378056/22/experimental/gate-kolla-kubernetes-deploy-centos-binary-nv/068e44a/console.html#_2016-09-29_00_09_10_86545700:56
*** absubram has quit IRC00:56
*** absubram_ is now known as absubram00:56
*** eaguilar has joined #openstack-kolla00:57
*** duonghq has joined #openstack-kolla01:00
duonghqmorning01:00
*** absubram has quit IRC01:11
*** phuongnh has joined #openstack-kolla01:15
openstackgerritgengchc2 proposed openstack/kolla: Fix a typo in bifrost.rst  https://review.openstack.org/37905401:15
*** huikang has quit IRC01:16
*** huikang has joined #openstack-kolla01:17
*** schwicht has quit IRC01:17
*** zhurong has joined #openstack-kolla01:19
*** huikang has quit IRC01:21
*** jamesdenton has joined #openstack-kolla01:21
*** huikang has joined #openstack-kolla01:22
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902701:22
*** daneyon has joined #openstack-kolla01:25
*** Jeffrey4l has joined #openstack-kolla01:27
sdakehey Jeffrey4l01:28
sdakei moved that gnocchi bug to high01:28
Jeffrey4lsdake, yep. thanks for that.01:29
Jeffrey4lif do not support ceilometer + gnocchi + ceph, it is totally unused for ceilomter.01:29
sdakelets focus on making gnocchi work first01:30
*** daneyon has quit IRC01:30
sdakecan you disect the ceilometer problem or me a  bit01:30
Jeffrey4lok01:30
sdakeor/for01:30
sdakeis it that ceilometer is completely unworking without gnocchi?01:31
sdakeor that it doesn't meet performance  requirements01:31
Jeffrey4lslow. ceilometer + mongo is very slow, especially when data large. that's why gnocchi exist.01:31
Jeffrey4lit works without gnocchi. But not produce-ready without gnocchi. sdake01:32
sdakewhy not?01:32
Jeffrey4lperformace is low.01:32
*** sin_ has quit IRC01:32
sdakeso performance problem without gnocchi then?01:32
Jeffrey4lright.01:32
sdakethe performanc eproblem is how bad01:32
sdakeunusable?01:33
sdakeif ceilometer is unusable thats a bug and that needs to be fixed01:33
Jeffrey4li haven't the test report. But now, we only have mongo+replaceset. I do not think that can support large data.01:33
Jeffrey4lsdake, here is a performance test https://julien.danjou.info/blog/2015/gnocchi-benchmarks01:35
sdakeJeffrey4l processing moment01:35
sdakewe are absolutely not using nfs01:36
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902701:37
Jeffrey4lsdake, the test is based on file only. the nfs is recommend when using file, but the test in that report now.01:38
sdakeI ran the same benchmark on the hardware I had, which is roughly two times smaller than the one they used. I achieved to push Gnocchi to a little more than 120k measurement per second. If I had same hardware as they used, I could interpolate the results to achieve almost 250k measures/s pushed.01:38
Jeffrey4lnot.01:38
sdakethis is a serious stretch01:38
sdakeis gnocchi a replacement for influxdb?01:38
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805601:39
Jeffrey4lsdake, i have no idea about what influxdb can do. But maye.01:39
Jeffrey4lat least gnocchi is design for storing metric01:40
sdakeRead metric SQLRead measures for 1 metric2min 58s01:40
sdakeRead metric MongoDBRead measures for 1 metric28s01:40
sdakeRead metric GnocchiRead measures for 1 metric2s01:40
sdakeok so this is the meat of it01:40
sdakesql is slower then mud01:40
sdakewe all knwo that01:40
sdakemongodb is faster then sql, we all know that01:40
sdakeis mongodb fast enough?01:41
Jeffrey4lyes. gnocchi is fastest.01:41
Jeffrey4lif so, gnocchi should never be develop01:41
sdakeis mongodb fast enough and does it implement HA in its current form01:41
Jeffrey4lsdake, i do not think it is fast enough. along with the data increase, the query become slower.01:42
Jeffrey4lthere is HA for mongo.01:42
sdakei see01:43
sdakeany data to back that up for intuition01:43
sdake(the fast enough comment)01:43
*** caowei has joined #openstack-kolla01:44
*** eaguilar has quit IRC01:44
sdakespecifically about data increase - query becomes slower01:44
*** schwicht has joined #openstack-kolla01:44
sdakefor intuition / or intuition01:44
Jeffrey4li can found some test report for this.01:45
sdakeok lets get the benchmark data in the reviews01:45
sdakeor the review01:45
sdakegnocchi needs to be ha01:45
sdakethats a done deal in my mind01:45
Jeffrey4lya. gnocchi + ceph provide ha.01:45
Jeffrey4lso that should be merge.01:45
sdakegoncchi+ceph = stable?01:46
Jeffrey4lright.01:46
sdakeok where is that review please01:46
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902701:46
Jeffrey4lsdake, https://review.openstack.org/37493701:46
sdakesorry for not attending to thi todday - had other stuff to deal with01:46
Jeffrey4lok. np01:47
sdakeJeffrey4l can't +2 a change if it has gate failures01:47
Jeffrey4li split one large PS into two yesterday. will fix the gate failure today.01:47
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902701:52
sdakeok so lets talk about the other patch set01:53
sdakegot a link to that one?01:53
sdakegates broken here too: https://review.openstack.org/#/c/378927/101:54
*** huikang has quit IRC01:55
*** huikang has joined #openstack-kolla01:56
*** huikang_ has joined #openstack-kolla02:00
*** huikang has quit IRC02:01
*** salv-orlando has joined #openstack-kolla02:01
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805602:03
*** salv-orlando has quit IRC02:05
*** huikang_ has quit IRC02:09
*** huikang has joined #openstack-kolla02:10
*** unicell has quit IRC02:10
*** huikang has quit IRC02:14
*** zhubingbing has joined #openstack-kolla02:18
Jeffrey4lyes. it is caused by the same reason. will fix it today.02:22
openstackgerritDuong Ha-Quang proposed openstack/kolla: Add profile usage to kolla-build guide  https://review.openstack.org/37823002:25
*** huikang has joined #openstack-kolla02:28
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805602:28
*** unicell has joined #openstack-kolla02:35
*** unicell has quit IRC02:39
*** huhaoran has joined #openstack-kolla02:47
*** yuanying_ has quit IRC02:50
*** sdake has quit IRC02:52
*** david-lyle has quit IRC03:04
*** yuanying has joined #openstack-kolla03:05
*** huhaoran has quit IRC03:09
*** sdake has joined #openstack-kolla03:16
*** dave-mccowan has quit IRC03:16
sdakeJeffrey4l sorry got distracted, so are we all settled on patches?03:18
*** huikang has quit IRC03:18
Jeffrey4li can fix the gate. and i hope to merge it. sdake03:19
*** huikang has joined #openstack-kolla03:19
sdakeis there a patch needed for ceilometer?03:19
*** g3ek has quit IRC03:19
*** haplo37 has quit IRC03:19
*** sdake has quit IRC03:19
Jeffrey4lhttps://review.openstack.org/372737  ( integerate gnocchi with ceilometer change some thing in ceilometer.03:20
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix error in Gnocchi service  https://review.openstack.org/37892703:21
*** huikang_ has joined #openstack-kolla03:21
*** huikang has quit IRC03:24
openstackgerritJeffrey Zhang proposed openstack/kolla: integrate gnocchi with ceilometer  https://review.openstack.org/37273703:28
*** g3ek has joined #openstack-kolla03:28
*** haplo37 has joined #openstack-kolla03:29
openstackgerritJeffrey Zhang proposed openstack/kolla: Integrate gnocchi with ceph  https://review.openstack.org/37493703:29
*** sdake has joined #openstack-kolla03:31
*** yuanying has quit IRC03:33
*** unicell has joined #openstack-kolla03:34
sdakeJeffrey4l hongbin has an email related to magnum03:34
*** yuanying has joined #openstack-kolla03:34
sdakeJeffrey4l mind telling me if we use the external load balancer feture of magnum?03:34
sdakeor is that a better question for marmat?03:34
Jeffrey4lhmm, tbh, i never used magnum.03:35
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902703:35
sdakeJeffrey4l cool i'll sync up with marmt on the topic03:35
openstackgerritJeffrey Zhang proposed openstack/kolla: WIP: fix ironic  https://review.openstack.org/37910603:39
*** aswadr_ has joined #openstack-kolla03:42
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902703:43
*** huhaoran has joined #openstack-kolla03:43
*** jmccarthy has quit IRC03:47
*** huhaoran has quit IRC03:48
*** yuanying has quit IRC03:49
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902703:50
*** huikang_ has quit IRC03:51
*** g3ek has quit IRC03:51
*** huikang has joined #openstack-kolla03:51
*** yuanying has joined #openstack-kolla03:52
*** g3ek has joined #openstack-kolla03:53
*** zhubingbing__ has joined #openstack-kolla03:53
*** tonanhngo has joined #openstack-kolla03:54
*** duonghq has quit IRC03:55
*** huikang has quit IRC03:56
*** zhubingbing has quit IRC03:56
*** zhubingbing_ has joined #openstack-kolla04:01
*** zhubingbing__ has quit IRC04:01
*** jmccarthy has joined #openstack-kolla04:01
*** salv-orlando has joined #openstack-kolla04:03
*** zhubingbing_ has quit IRC04:03
*** salv-orlando has quit IRC04:07
*** zhubingbing has joined #openstack-kolla04:10
*** zhubingbing has quit IRC04:19
*** zhubingbing_ has joined #openstack-kolla04:22
*** zhubingbing_ has quit IRC04:26
*** duonghq has joined #openstack-kolla04:41
*** coolsvap has joined #openstack-kolla04:42
*** salv-orlando has joined #openstack-kolla04:42
*** salv-orlando has joined #openstack-kolla04:43
*** sdake has quit IRC05:00
*** schwicht_at_work has joined #openstack-kolla05:06
*** schwicht has quit IRC05:07
*** msimonin has joined #openstack-kolla05:17
*** g3ek has quit IRC05:39
*** haplo37 has quit IRC05:44
*** haplo37 has joined #openstack-kolla05:45
*** g3ek has joined #openstack-kolla05:48
*** msimonin has quit IRC06:00
*** egonzalez90 has joined #openstack-kolla06:01
*** bjolo has joined #openstack-kolla06:02
*** unicell1 has joined #openstack-kolla06:03
*** unicell has quit IRC06:03
*** unicell1 has quit IRC06:21
*** unicell has joined #openstack-kolla06:21
tonybzhubingbing: Are you around?06:32
duonghqcoolsvap: can you brief review on https://review.openstack.org/#/c/378230/06:32
tonybinc0, zhubingbing: Any progess on https://review.openstack.org/#/c/366948/ It'd be great to get a vote from kolla.06:34
*** mewald has joined #openstack-kolla06:34
*** msimonin has joined #openstack-kolla06:39
bjolomorning people :)06:42
*** nihilifer has joined #openstack-kolla06:43
duonghqegonzalez90: ping06:44
egonzalez90hey there duonghq06:46
bjoloi failed to break the upgrade of mariadb. guess thats a good thing :)06:48
duonghqegonzales90: https://review.openstack.org/#/c/368569/2 can you review it06:49
duonghqegonzalez9006:49
*** tonanhngo has quit IRC06:53
duonghqas I see it's totally fine, but do not get any review yet06:53
duonghqthank coolsvap06:56
*** tonanhngo has joined #openstack-kolla07:00
*** hfu has quit IRC07:01
*** tonanhngo has quit IRC07:02
*** hfu has joined #openstack-kolla07:03
*** mewald1 has joined #openstack-kolla07:10
*** mewald has quit IRC07:12
*** Serlex has joined #openstack-kolla07:13
*** britthouser1 has joined #openstack-kolla07:24
*** matrohon has joined #openstack-kolla07:26
*** britthouser has quit IRC07:28
*** gfidente has joined #openstack-kolla07:34
*** ldeptula has joined #openstack-kolla07:36
*** salv-orlando has quit IRC07:37
*** ldeptula has quit IRC07:40
*** daneyon has joined #openstack-kolla07:43
*** ldeptula has joined #openstack-kolla07:45
*** daneyon has quit IRC07:49
*** salv-orlando has joined #openstack-kolla07:50
*** aswadr_ has quit IRC08:12
*** mewald1 has quit IRC08:26
*** strigazi_AFK is now known as strigazi08:29
*** matrohon has quit IRC08:54
*** berendt has joined #openstack-kolla08:56
*** bjolo has quit IRC09:01
openstackgerritzhangyanxian proposed openstack/kolla: Fix typo in start.yml  https://review.openstack.org/37928809:02
*** bjolo has joined #openstack-kolla09:03
openstackgerritzhangyanxian proposed openstack/kolla: Fix typo in start.yml  https://review.openstack.org/37928809:03
*** mewald has joined #openstack-kolla09:03
openstackgerritMerged openstack/kolla: Fix a typo in bifrost.rst  https://review.openstack.org/37905409:08
*** mewald has quit IRC09:08
*** mewald has joined #openstack-kolla09:10
*** haplo37_ has quit IRC09:23
*** haplo37_ has joined #openstack-kolla09:26
*** sdake has joined #openstack-kolla09:47
sdakemorning09:49
*** tonanhngo has joined #openstack-kolla09:49
openstackgerritChristian Berendt proposed openstack/kolla: Deprecate Fedora based Docker images  https://review.openstack.org/36918409:50
*** tonanhngo has quit IRC09:50
*** pbourke has joined #openstack-kolla09:50
berendtsdake morning. regarding your mail on openstack-dev, i think it is not necessary to deprecated heka at all09:51
sdakeberendt i agree but it wasbroguht up09:52
sdakeand it seemed liek people wnated to deprecate it09:52
sdake(fromt he meeting log)09:52
berendti think we should add a note that we will replace it in the next cycle09:53
sdakethat is the same as deprecation09:53
berendtmaybe somebody out there uses heka for something else...09:53
berendtsdake yes, but we do not have to follow the deprecation process09:53
sdakeyes we do :)09:54
berendtadd a note to newton that heka will be replaced and replace it in ocata09:54
openstackgerritSharat Sharma proposed openstack/kolla: Drop root for keystone.  https://review.openstack.org/37933209:54
sdakeor we run afoul of the deprecation policies09:54
sdakewhich means no depreatoin project maturity tag for us09:54
*** rmart04 has joined #openstack-kolla09:54
sdakewe can deprecate in early o109:54
sdakeand still hit the 3 month window09:55
sdakeif need be09:55
berendti am not sure if deprecation process is required for internal services09:55
sdakebut i wantt o see  amigratin plan that makes ssense09:55
sdakein our case its fairy extenrnal09:55
berendtbut i think it does not hurt to follow the process and to replace it a few months later09:55
berendtheka works and it is not critical09:55
sdakeright the main problem is that we can't follow the process between now and oct 12th09:56
sdakebecuse nobody iknows the migration path09:56
berendtthe migration path is simple. stop heka, start some other service.09:56
sdakei'm open to a note we are onsideing deprecating heka in ffavor of something ele without losingfunctionality09:56
sdakeberendt it isnt qite that simple - log data lot, etc09:57
berendtthe migration on a running system should be that easy09:57
sdakeshould be and are are two different things :)09:57
sdakewhen i take it to operators list (or michal does) we need to be able t ojustify it09:57
sdakeand "trust us" sin't good enough09:58
sdakethey have heard that one before09:58
sdakethat is why there is a deprecation policy to begin with09:58
sdakeanyway3a mjust woke up - can we discuss ater i wke up so I don't appear grumpy :)09:58
berendtof course ;)09:59
sdake15-20 mins hould do09:59
sdakefwiw i  am all forgetting rid o dead ends in our codebase09:59
sdakefor getting09:59
sdakebut the situation isn't as ad as was made out in the meeting10:00
sdakeif we run afoul of the deprecatin policy we have to wait 2 cycles to apply fforit10:00
berendtyes i think it is not a big deal to use heka the next time10:00
sdakewe have to show a track record of being consistent10:00
berendtit is working and it is only pushing logs to elasticsearch..10:01
sdakebefore deprecation PMT can really be applied10:01
sdakeyup i understand what it does :)10:01
berendteven in the morning? :)10:01
sdakelogtash imo is off the table - it is completelly deprecated10:01
sdakenosecurity updates10:01
sdakewhy switch from something that receives security updates to something that doesnot whn both are deprecated10:02
berendtsomebody mentioned filebeat10:02
berendtk8s team wants to use fluentd10:03
berendti think we will end with fluentd for kolla10:03
*** zhurong has quit IRC10:03
openstackgerritMerged openstack/kolla: Fix typo in start.yml  https://review.openstack.org/37928810:09
sdakeberendt how much are those switches retail?10:13
openstackgerritMerged openstack/kolla: Mount host localtime for ceilometer  https://review.openstack.org/37873010:13
sdakeberendt if we end with fleuntd for kolla so be it, but can we just not follow theproce for the PMt purpoe10:13
openstackgerritChristian Berendt proposed openstack/kolla: WIP: Move config files out of deploy_aio.sh script  https://review.openstack.org/37934410:22
*** mewald has quit IRC10:26
*** ayoung has quit IRC10:27
*** duonghq has quit IRC10:32
*** phuongnh has quit IRC10:38
*** salv-orlando has quit IRC10:40
*** ayoung has joined #openstack-kolla10:40
*** salv-orlando has joined #openstack-kolla10:41
sdakeJeffrey4l https://bugs.launchpad.net/kolla/+bug/162570510:43
openstackLaunchpad bug 1625705 in kolla "latest kolla/ubuntu-binary-heka:3.0.0 container keep on restarting " [Critical,Triaged]10:43
sdakeJeffrey4l any ideas on this one?10:43
sdakeregression or user error?10:43
sdakeheka doesn't have any user touch points10:44
Jeffrey4lsdake, no. :( if there is real some permission issue, CI will complain that.10:45
sdakeJeffrey4l good point10:45
Jeffrey4lsdake, here is the patches relate to ceilomter+gnocchi+ceph  https://review.openstack.org/#/q/topic:bug/1626364+OR+topic:bug/1626623   sdake10:49
*** salv-orlando has quit IRC10:54
sdakeJeffrey4l first patch is a -111:00
*** caowei has quit IRC11:02
*** mewald has joined #openstack-kolla11:02
Jeffrey4lOK.11:02
Jeffrey4lsdake, so change the feature to fixes in release note?11:03
sdakeyes and my wording should do11:05
sdakere https://review.openstack.org/#/c/378927/2/docker/gnocchi/gnocchi-base/Dockerfile.j211:05
*** mewald1 has joined #openstack-kolla11:05
sdakewhatsthe deal with echo > xyz11:05
*** awiddersheim has quit IRC11:06
sdakewhat is the purpose of that?  i it to create the file?  if so, why not use touch11:06
sdakepbourke about11:06
pbourkesdake: yes11:07
sdakepbourke morning dude :)11:07
pbourkemorning :)11:07
*** mewald has quit IRC11:07
sdakewhat shape do you think kolla is in at present?11:07
*** rhallisey has joined #openstack-kolla11:07
sdakecore pats seem good to me11:07
sdakeparts11:08
sdakesup rhallisey11:08
sdakesup Mech42211:08
rhalliseyyo11:08
sdakesup mewald1 i meant ;)11:08
mewald1sdake: good morning11:09
pbourkesdake: i dunno bit of a broad question11:09
sdakepbourke right so answer broadly :)11:09
sdakeits broad on purpose ;)11:09
pbourkeyou ask an irish person how something is, they're going to say "fine"11:09
sdakeok well tell me truth then as if your not irish :)11:10
pbourkeits in good shape overall11:10
sdakei want to sanity check my understanding11:10
pbourkeI wouldn't have great confidence in the non core services, i.e. ones that aren't tested regularly by the gate11:11
sdakeok so there are some hidden messages there i pick up on which seems to imply thre are some troule spots11:11
pbourkethose are at the mercy of interested contributors11:11
pbourkeI also think the post-deploy parts could use more attention11:12
sdakepost-deploy needs attention how?11:12
pbourkereconfigure, upgrade, add new nodes, remove new nodes, backup, etc.11:12
sdakewell remove nodes is not a  thing11:13
pbourkewe support most of those things but Im dubious how many are using them in agner11:13
pbourke*anger11:13
sdakebackup is not a thing11:13
pbourkeright11:13
sdakebut roger on upgrade and reconigure11:13
sdakeok so we need to sanitize upgrade and reconigure11:13
sdakeJeffrey4l your thoughts onthe matter please?11:13
sdakeany other cores around?11:14
rhalliseycould be missing some context11:14
sdakerhallisey i want to know what people think the current stateof kolla is from a "does the damn thing work" perspective11:14
*** ccesario has joined #openstack-kolla11:14
sdakei dont see any problems when i work with kolla11:15
sdakebut i am not burning down the bug log11:15
rhalliseyfor what I use it for it does11:15
sdakerhallisey whcih is only container content?11:15
rhalliseygates for more deployments improve things11:15
rhalliseymostly these days yes11:16
sdakeobvoulsy gating is a sore spot11:16
sdakesore/soft11:16
pbourkewe need to deliver more on the operational aspects kolla promises11:17
sdakepbourke totally agre11:17
sdakee11:17
sdakepbourke we did a piss por job of that  this cycle11:17
*** shardy has joined #openstack-kolla11:18
sdakemoniitroing is one operational asepct11:18
sdakeand from what I can gather, the shape of the monitoring stack is in disarray11:18
Jeffrey4lcore service are stable. more test , which is not including in default deployment,  need to be done.11:18
rhalliseyya monitoring is always at the midcycle/summit11:18
rhalliseyimo that means it's not 100% solved11:18
*** openstackgerrit has quit IRC11:19
Jeffrey4lre re https://review.openstack.org/#/c/378927/2/docker/gnocchi/gnocchi-base/Dockerfile.j211:19
pbourkein some ways though monitoring is higher up the stack11:19
*** openstackgerrit has joined #openstack-kolla11:19
sdakepbourke indeed higher up stack11:19
Jeffrey4lecho >  xxx, disable the default listen on 80 port.11:19
rhalliseypbourke, which operational aspects specifically?11:19
sdakeJeffrey4l how does it do that y making an empty file?11:19
pbourkemonitoring running services is no good if we can't properly manage the infra those services are running on11:19
sdakepbourke how do we fail to properly manage the infra?  you mean upgrade/reconfigure?11:20
Jeffrey4lin default /etc/apache2/ports.conf this contain content: listen 8011:20
pbourkerhallisey: have you upgraded an entire kolla cluster with live users and would you feel comfortable doing that today11:20
sdakei see so its an overwrite11:20
Jeffrey4lwrite.11:20
Jeffrey4lright.11:20
sdakeJeffrey4l how about rm -f filename touch filename11:20
flaper87hey folks, do you know if there's a docker registry installed as part of the CI infra? I believe the answer is no but I thought I'd ask here too :D11:20
sdakethat echo thing is confusing11:20
rhalliseypbourke, I haven't personally no11:21
sdakethere is not flaper8711:21
*** domi has joined #openstack-kolla11:21
sdakeflaper87 fwiw i've been working with infra to make that happen with no success for over a year11:21
Jeffrey4lthis tricky is copied from horizon. let me check that if delete that file work.11:21
sdakeflaper87 if you can crack tht nut, more power to youo :)11:21
flaper87sdake: could you elaborate on what went wrong (if anything) ?11:21
flaper87sdake: I'd be happy to help doing this11:21
sdakeflaper87 its very complex11:21
sdakeflaper87 what is your tz?11:22
domihi all, I'm trying to follow this: http://docs.openstack.org/developer/kolla/advanced-configuration.html#openstack-service-configuration-in-kolla but after reconfigure my container still runs the same default config instead of my custome one11:22
flaper87sdake: that's a hard question to answer :P but I'm in EU for now11:22
sdakeI have a super important task to get done - and wanted to do it right about now11:22
sdakeflaper87 ok11:22
sdakewell let me gie you short version11:22
sdakeflaper87 happyt o expand later11:22
flaper87sdake: sounds good11:22
sdakeflaper87 in kolla we bulidl images11:22
pbourkedomi: what config are you trying to customise and what does your config look like11:23
sdakewe have desire to push images to a openstack hosted docker registry11:23
sdakethe problem has been how precisely to to this11:23
flaper87btw, is there something I can do to help with the kolla-build split? or are all the work items for that task already taken ?11:23
sdakecredentials are part of it11:23
sdakeflaper87 blocked on 3.0.011:23
sdakeflaper87 but it will happen rest assured11:24
flaper87sdake: so credentials is one problem11:24
flaper87sdake: gotcha, thanks11:24
sdakeflaper87 i will likely ddo that work as i have most experience working with project-config11:24
openstackgerritJeffrey Zhang proposed openstack/kolla: Integrate gnocchi with ceph  https://review.openstack.org/37493711:24
flaper87sdake: sounds perfect11:24
sdakei'd ike it to happen prior to summit11:24
flaper87sdake: ++11:24
sdakebut logicially this may not be possible11:24
sdakeif not it will happen shortly after11:25
pbourkerhallisey: sdake: we get a lot of complaints about the general poorness of ansible's error reporting11:25
sdake(or during)11:25
pbourkenot sure how much we can do to improve this11:25
domipbourke: nova compute's nova.conf. So I created the /etc/kolla/config directory, then created inside nova-compute/nova.conf containing 2 lines: [DEFAULT] debug = True as a test. I added to globals.yml the required option (node_custom_config) then run reconfigure11:25
rhalliseypbourke, we might be able to fix that11:25
sdakeflaper87 ok so on to the real problems with pushing docker images11:25
pbourkesdake: rhallisey: here's an example of an internal bug report I got this morning http://paste.openstack.org/show/583475/11:25
rhalliseyk11:25
sdakeflaper87 beyoond the creds, which are solvable i think11:25
sdakeflaper87 infra doesn't want long running jobs11:25
*** jrist has joined #openstack-kolla11:25
sdakeflaper87 which i totaly get11:26
sdakeflaper87 to push takes some time11:26
sdakeflaper87 so we can't push master, but i'd like to atleat push tags11:26
sdakeflaper87 infra open to this idea11:26
sdakeits a matter at this point of implementation11:26
flaper87ok, that sounds like progress11:26
sdakeflaper87 yup open to idea, no idea how to execute is how id classify it at this point :)11:27
flaper87sdake: do you know how does infra feel with having a couple of folks with push rights to the registry to begin with?11:27
flaper87I mean, get the registry installed and figure out how to automate pushes after11:27
sdakewhich registry are we speaking of?11:27
sdakeflaper87 ya thats a good approach11:27
sdakeflaper87 worth trying11:27
Jeffrey4lwhat the registry is used for? registry mirror?11:28
sdakeflaper87 it can always be wiped out later11:28
sdakeflaper87 also there is the disk storage problem11:28
flaper87sdake: ok, I'll bring this up in the next infra meeting (or the one after the next one since I'll be out next week)11:28
sdakeflaper87 when i floated the propoal to infra, the general response I got back was "too much disk space"11:28
sdakebut we wer etalkingabout master pushing too11:28
pbourkedomi: sounds like it should work. Can you have a look at /etc/kolla/config on the target node, and see if it copied over your change at all? Other thing to check is if the nova_compute container was restarted to pick it up11:29
sdakeand at that time docker registry had no GC11:29
flaper87right11:29
*** mewald has joined #openstack-kolla11:29
sdakeso it grows and grows unbounded11:29
Jeffrey4lor a private registry repo? then how to use it? during the ci job? or make it public?11:29
sdakei've backed off maater pushes11:29
sdakeand focued on tag pushes11:29
flaper87tag pushes would be great progress, fwiw11:29
sdakebut i'm only one person, i cna't do it alone :)11:29
sdaketotally agree11:29
domipbourke: should've stated it's an AIO deployment, does that make a difference in this case?11:29
sdakethe problem i was dealing iwth i the core team wants *masater* pushes11:30
flaper87ok, I'm happy to help. Let me find some time in the next couple of weeks to bring this up11:30
sdakethe community is happy to use docker hub11:30
sdakepushing to docker hub brings up the credentials problem11:30
sdakewe dont want ot check our creds into our git repo for examaple11:30
flaper87right11:30
sdakethen that could create a security problem11:30
sdake(someone takes creds and pushes images in our stead)11:31
sdakeJeffrey4l right - we have to define what exactly the prolem is we are trying to olve with a  opentack hosted docker registry11:31
sdakei am sure the foundation would love to ssee some integrtion in this area11:32
openstackgerritVladislav Belogrudov proposed openstack/kolla: iscsi role misses 'storage' group  https://review.openstack.org/37938111:32
sdakesure/know :)11:32
domipbourke: I'm not sure what's the flow: when I run reconfigure my custom conf gets merged with the one in /etc/kolla/nova-compute? And then in /etc/kolla/nova-compute/nova.conf the config should have my changes in it? ANd then it should restart the container?11:32
Jeffrey4lso the problem is: set up a openstack hostd docker registry sever, right?11:32
pbourkedomi: that's pretty much it yes. try putting your custom config file in /etc/kolla/config/nova.conf11:33
pbourkedomi: and run reconfigure11:33
*** mewald1 has quit IRC11:33
domipbourke: but then how would it now if I mean nova-compute or nova-scheduler or the other nova services? I'll give it a try anyway sure11:34
sdakeJeffrey4l ok patch two of 4 is -111:34
sdakeminor nitpick thing11:34
sdakethe echo vs touch11:34
rhalliseypbourke, though it would be a tall task, we could add fail messages for tasks11:35
pbourkerhallisey: yeah I dunno I think it should be looked at. Overall we spend *a lot* of time figuring out why services don't come up or why certain tasks fail. Ideally because we're using Docker they'll come up more often than not11:36
Jeffrey4lsdake, checking.11:36
pbourkerhallisey: one way to improve that is to rejig the prechecks so that they run every time rather than just initially11:36
rhalliseydocker is another factor that I've hear from opts11:36
pbourkerhallisey: this is on Jeffrey4l's radar also11:36
pbourkerhallisey: they dont like it?11:36
rhalliseywell it's a different way to error check11:37
pbourkeyeah11:37
rhalliseydifferent process11:37
rhalliseynot that they hate it, but they have to change11:37
Jeffrey4lprecheck is useless in most of case.11:37
sdakeflaper87 the hard part - also - is infra wants to transfer the data to a local docker registry their way11:37
pbourkethere's plenty of times the bootstrap will just spin, tricky enough to find why11:37
Jeffrey4lbecause most of use do not run it.11:37
sdakeflaper87 which is cool11:37
Jeffrey4luse/user11:37
pbourkeJeffrey4l: +111:37
pbourkeJeffrey4l: but it could be so useful11:37
sdakeflaper87 i'm not sure they totally get what their way is11:37
sdakeflaper87 our way is to use docker push11:38
rhalliseypbourke, Jeffrey4l good points11:38
sdakeflaper87 their way is ???11:38
domipbourke: now it worked (yay, finally :)), but changed the config on all nova services...I'm not sure if this is how it's supposed to work11:38
sdakeflaper87 unknown - lots of spitalling, but i get impression docker push is not one of their ways they like :)11:38
sdakespitballing11:38
Jeffrey4lpbourke, for example, the hostname for rabbitmq, we need run the check both in pre-check and during depoyment.11:38
sdakeflaper87 we are happy to comply with whtever infra can cook up11:39
Jeffrey4lpbourke, rhallisey that's why i revert this https://review.openstack.org/36988911:39
sdakeflaper87 however theey re so busy all the time they dont have time to cook up anything11:39
flaper87sdake: noted... it might be as well that for now we're just better off using docker hub (Which is what we've been doing so far, I believe).11:39
sdakeflaper87 i totally get here infra is coming from - and understand the plight11:39
flaper87Let's see what can be done and if it can be done11:39
sdakeflaper87 but it has been a blocker a well11:39
sdakeflaper87 we manually push to docker hub11:40
sdakewhich is totally subooptimal11:40
rhalliseyJeffrey4l, pbourke are you both going to summit?11:40
pbourkedomi: yeah it looks like we need to be more flexible on this11:40
sdakeflaper87 what would be optimal is on tag, infra pushes to a permanent artifact registry in infra11:40
Jeffrey4lrhallisey, i will11:40
pbourkedomi: if you would like to file a bug with your experiences around this would be much appreciated11:41
pbourkerhallisey: I am yes11:41
domipbourke: sure thing, thank you11:41
sdakeflaper87 its a bit of a tricky problem to solve11:41
sdakeflaper87 lots of moving parts and no bandwidth from any teams to solve it11:41
pbourkeJeffrey4l: right I just dont want to do it for one service, we should discuss and have it refactored across the project11:41
sdakeflaper87 and my 18 hr a day workweeks are comin to a close :)11:41
rhalliseyPain Points & Operator Experience11:42
*** haplo37_ has quit IRC11:42
*** msimonin has quit IRC11:42
rhalliseyJeffrey4l, pbourke ^ those sessions let's cover this11:42
rhalliseyJeffrey4l, I think the rabbitmq example you bring up is a good one11:42
Jeffrey4lpbourke, rhallisey one possible solution is: add pre-check.yml task in each role and run it both in pre-check and deploy action.11:42
pbourkeJeffrey4l: I like that11:43
rhalliseywe also need to solve the fail messages11:43
Jeffrey4lrhallisey, some guy complain that: my rabbitmq not work any all. it is caused by hostname and no place to show the root cause.11:43
sdakeo kguys lets not invent new work right at the end o the cycle :)11:43
rhalliseysdake, this is for O11:44
sdakewe made some commitmeents in austion, we didn't totally deliver11:44
sdakethats ok, we delivered a whle slew of htings11:44
sdakeah or o11:44
Jeffrey4lat least, should we revert this PS? https://review.openstack.org/36988911:44
sdakeok11:44
*** haplo37_ has joined #openstack-kolla11:44
Jeffrey4lor add the rabbitmq pre-check on both side. and refactor this in O11:45
rhalliseywe should try and use the fail module where ever we can11:45
rhalliseyor add debug options everywhere?11:45
Jeffrey4lrhallisey, fail is enough.11:45
rhalliseymay need to write a custom module for debug everywhere11:45
rhalliseydebug meaning a --debug flag11:46
Jeffrey4lrhallisey, debug for what? what mean by --debug?11:46
sdakeJeffrey4l patch 3 has -111:46
sdakeok guys i have a ottally rockin idea for how to handle htis11:46
sdakeborrowed from dmsimard11:46
rhalliseybefore a task is run we can have output about it11:46
sdakewe can implement an ansible plugin11:46
rhalliseythat gets triggered by a --debug flag11:47
sdakethat captures the state we care about11:47
sdakethis is similar to how ara operates11:47
sdakeJeffrey4l looking11:47
Jeffrey4lsdake, re patch 3, ok. thanks.11:48
rhalliseyhttps://github.com/dmsimard/ara11:48
Jeffrey4lfyi: i have added ara report in CI.11:49
sdakeJeffrey4l i alsoo put a -1 on https://review.openstack.org/#/c/369889/211:50
sdakeJeffrey4l sweet job on ara11:51
ccesarioMorning guys :)11:51
sdakeJeffrey4l mind sharing how we get to it11:51
sdakeccesario yo sup11:51
*** dwalsh has joined #openstack-kolla11:51
ccesariohey sdake !11:51
sdakeok gotta jet - slide ware time11:52
sdakebb in few hours hopefullly11:52
Jeffrey4lara report: check this like http://logs.openstack.org/27/378927/2/check/gate-kolla-dsvm-deploy-ubuntu-binary-ubuntu-xenial-nv/d9ce556/logs/playbook_reports/11:52
mliima__morning guys11:53
sdakeJeffrey4l could you point out which buttons to click to get there11:53
*** mliima__ is now known as mliima11:53
sdakerather the nthe ara report (although people may want to see the ara report which is hepful for debug and other things)11:53
Jeffrey4lsdake, go to the ci job logs. and click `logs` , then click `playbook_reports`11:54
sdakeJeffrey4l just clilck through to logs right?11:54
Jeffrey4lneed click `playbook_reports`11:54
sdakecool11:54
sdakenice - i knew you did that, didnt know it merged :)11:54
sdakemy thermostat says its 82 and its still freezing11:56
*** jrist has quit IRC11:56
sdakehate thi time of year in arizona11:56
sdakeeither hot or cold11:56
sdakealthough the weather outsside sure is nice during the day :)11:56
*** msimonin has joined #openstack-kolla11:57
sdakeok deck - must focus - bbiaf11:58
sdakepbourke Jeffrey4l before iget onto that11:58
sdakei think the key takeaways are that operational experience is important11:58
sdakeand we need to make sure upgrade and reconfigure at minimum work fantasticaily well11:58
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902711:58
sdakeatm reconfigure has some wierd kinf of looping ging on in nova11:59
Jeffrey4lsdake, re echo ports. echo is simple. if we remove it, we need change the apache2.conf file and11:59
Jeffrey4lremove the line Include ports.conf11:59
sdakeyour not echo >>11:59
sdakeyour echo >11:59
sdakeecho > is a replacement11:59
Jeffrey4lecho > xxxx  remove all the contents in that file.12:00
Jeffrey4lthat what we need.12:00
sdakeright12:00
openstackgerritChristian Berendt proposed openstack/kolla: Move config files out of deploy_aio.sh script  https://review.openstack.org/37934412:00
sdakeand rm -f followed by touch achieves that12:00
Jeffrey4lhm. the result is the same. but we need two command ;p12:00
sdakeyup12:00
sdakebut its easier to underswtand12:00
sdakei got hung up on it for quite ahwile12:01
Jeffrey4lok..12:01
Jeffrey4lsdake, how about this.12:01
Jeffrey4lhttp://unix.stackexchange.com/questions/88808/empty-the-contents-of-a-file12:01
sdakepbourke Jeffrey4l is above statement accurate?12:01
*** dave-mccowan has joined #openstack-kolla12:01
Jeffrey4lcp /dev/null filename    better?12:02
pbourkesdake: yes12:02
sdakeJeffrey4l lets leave /dev out of the picture12:02
sdakei am open to truncate -s 0 filename12:02
Jeffrey4lOK.12:02
sdakeone command12:03
sdakeobvious what it doe12:03
sdakes12:03
sdakepbourke cool - so lets make sure we have a rockin experience for upgrade and reconfigure12:03
sdakeif other newer services ren't perfect, that is what z streams are for :)12:03
sdakeJeffrey4l ^12:04
sdakespread the word12:04
Jeffrey4lall the base image contain truncate, i will move to this tool.12:04
Jeffrey4lroger.12:04
sdakeat this point unless someone steps up to make sure the monitoring stack is solid, we need a release note ssaying it is tech preview and wil be vassty improved in ocata12:04
sdakebecause operators will want to interact wit hthat12:05
sdakesorry the performance stack12:05
sdakeand if they have a bad experience there, they will be non happy :)12:05
Jeffrey4lperformance stack means?12:07
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix error in Gnocchi service  https://review.openstack.org/37892712:07
sdaketig + collectd12:09
Jeffrey4lOK.12:09
Jeffrey4lwhy not call it monitoring stack?12:09
sdakethe great work mewald did but left for vvacation and other pepole took over without his context12:09
sdakeya sorry thats what it is12:09
sdakeanyway you get the idea12:09
*** msimonin has quit IRC12:10
Jeffrey4linfluxdb is not HA. So i think it is still in tech preview stage.12:10
Jeffrey4lunless someone solve this.12:10
sdakeagain for monitoring, I think that doesnt matter if there is a data loss - if its documented12:10
Jeffrey4lOK.12:11
sdaketo me tech preview means "it just doesn't work but the code is there to pla yaround with"12:11
Jeffrey4lroger.12:11
sdakeor "it barely works"12:11
*** mewald1 has joined #openstack-kolla12:11
sdakemewald1 we were just discussing you :)12:12
Jeffrey4lsdake, could u explain this more? https://review.openstack.org/#/c/372737/6/ansible/roles/ceilometer/defaults/main.yml12:12
sdakeJeffrey4l moment12:12
sdakemewald1 yu around?12:13
openstackgerritJeffrey Zhang proposed openstack/kolla: Integrate gnocchi with ceph  https://review.openstack.org/37493712:13
sdakeJeffrey4l i think by expansion I mean "can it be removed" and if the answer is no - then the code should say as much12:13
sdakedunno brain still booting12:14
sdakeit takes 2  or 3 hours of you guys grilling me for me to get on my game ;)12:14
*** mewald has quit IRC12:14
*** daneyon has joined #openstack-kolla12:14
*** srwilkers has quit IRC12:14
Jeffrey4lsome one asked why the value for gnocchi is empty. then i leave a comment. may be i need change it to: #NOTE(Jffrey4l): xxxx12:15
*** mewald1 has quit IRC12:15
ccesariohey guys, currently how is the better way to enable multiple cinder backends ?12:15
sdakeccesario - sbezverk is your guy :)12:16
ccesarioahahah, ok :)12:16
ccesariosbezverk, ping?12:16
*** jrist has joined #openstack-kolla12:17
sdakeJeffrey4l whatever needs to be done- only working to unblock your work12:17
sdakesay jrist did i see you were running for the tc?12:17
* sdake so much backlog12:18
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902712:18
sbezverkccesario: pong12:18
ccesario<ccesario> hey guys, currently how is the better way to enable multiple cinder backends ?12:18
ccesario<sdake> ccesario - sbezverk is your guy :)12:18
sdakesbezverk cinder quetion sbezverk12:18
*** deepak_jon has joined #openstack-kolla12:18
ccesariosbezverk, have you any tip? please!? :)12:18
jristsdake: lord no12:18
jristsdake: I'm definitely not qualified12:19
sdakejrist oh ya i thought it was a wierd email to read, because it came rom a rax dude12:19
sbezverkccesario: I seriously doubt that we support multiple cinder backends12:19
sdakeyet i though i saw yur name12:19
sdakewhich is why i asked :)12:19
sbezverkat least I have never tried12:19
sdakebecause your cloak is redhat12:19
sdakejust as mine is fedora12:19
ccesariosbezverk, hmmm12:20
sdakei was pretty sure you were at red hat anyway12:20
*** daneyon has quit IRC12:20
sdakejrist never underestimate the power of voters :)12:20
jrist:)12:21
sdakejrist sorry we probably could hve had that conversation in private - but i dont like to do that ;)12:21
jristoh yeah, definitely no need12:21
jrist:)12:21
jristI don't mind at all12:21
ccesariosbezverk, thanks by info... .. I'll think about it :)12:21
jristthank you for the vote of confidence12:21
sdakeit just didn't make any sense at all to me :)12:21
sdake(the rax and redhat thing together012:21
sdakejrist it would have bothered me all day - cd ftl12:22
jristhaha, totally understand12:22
openstackgerritJeffrey Zhang proposed openstack/kolla: integrate gnocchi with ceilometer  https://review.openstack.org/37273712:23
openstackgerritJeffrey Zhang proposed openstack/kolla: integrate gnocchi with ceilometer  https://review.openstack.org/37273712:23
sdakeJeffrey4l since your the rocket scientist behind reconfigure, mind triaging this bug: truncate -s 0 filename12:25
sdakenot that one :)12:25
sdakehttps://bugs.launchpad.net/bugs/162889512:25
openstackLaunchpad bug 1628895 in kolla "OpenStack service reconfiguration doesn't comply documentation" [Undecided,New]12:25
sdakelooks like user error12:25
* Jeffrey4l is checking.12:25
sdakebut who knows12:25
*** salv-orlando has joined #openstack-kolla12:25
deepak_jonHi, Getting error while executing "kolla-build -b ubuntu keystone" :Failed to fetch http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/p/python-requestsexceptions/python-requestsexceptions_1.1.2-0ubuntu1~cloud0_all.deb  403  Forbidden12:25
sdakeForbiddden?12:26
deepak_jonyes12:26
sdake/me groans12:26
Jeffrey4lit is user error. I will leave a comment.12:26
* sdake groans that is12:26
sdakeJeffrey4l thanks12:26
deepak_jonexecuting as root user12:26
sdakedeepak_jon - it is entirely possible ubuntu cloud archive is busted at present12:27
*** msimonin has joined #openstack-kolla12:27
sdakedeepak_jon can you curl that file on your machine?12:27
*** jrist has quit IRC12:28
*** salv-orlando has quit IRC12:30
sdakeok guys sgotta jjet for awhile -> slidewarae12:30
sdakecan someone help out deepak_jon pls12:30
Jeffrey4ldeepak_jon, i can download file from http://ubuntu-cloud.archive.canonical.com/ubuntu/pool/main/p/python-requestsexceptions/python-requestsexceptions_1.1.2-0ubuntu1~cloud0_all.deb12:31
deepak_jonyes, i can also dowdload it12:31
*** schwicht_at_work has quit IRC12:31
*** zhurong has joined #openstack-kolla12:32
Jeffrey4ldeepak_jon, could u post the whold build log out?12:32
deepak_jonbut while executing the command "kolla-build -b ubuntu keystone" that error thrown12:32
Jeffrey4l*&whole12:32
sdakedeepak_jon i had a quick quetion, are you following  the developer workflow or the operator workflow12:39
*** jheroux has joined #openstack-kolla12:41
sdakerhallisey note we may have a different take on the odcumenttion rework12:44
rhalliseykk12:44
rhalliseygo for it12:44
sdakerhallisey lana bradley has sort of offered to help us beat an install guide into submisssion12:44
sdakesh ehasn't committed at this point but looking promising12:44
sdakeso our dev workflo would remain on developer12:44
sdakeand the install guide wuld go in with the rest of th einsntall guides12:45
rhalliseyya we need some ideas to make it clear12:45
Jeffrey4lsdake, did you notice this? https://review.openstack.org/37652512:45
sdakeon the main openstack-manuals repo12:45
Jeffrey4lthe stable branch doc is rendered in http://docs.openstack.org/developer/kolla/mitaka/12:45
sdakeJeffrey4l no - however this is more advanced12:46
sdakeJeffrey4l we are taking a tandard template here12:46
sdakeso apply our docs to a standard template that goes on docs.oo but not /developer12:47
Jeffrey4lgot.12:47
*** shardy has quit IRC12:48
sdakeratahr Lana Brindley12:48
sdakesorry - similar nams :)12:49
*** zhubingbing has joined #openstack-kolla12:49
dmsimardrhallisey: ohai12:54
dmsimardrhallisey: https://github.com/openstack/ara actually :)12:55
*** david-lyle has joined #openstack-kolla12:57
rhalliseydmsimard, oh!12:59
* dmsimard nods13:00
dmsimardwith integration jobs in-gate and all that http://logs.openstack.org/94/375494/1/check/gate-ara-integration-centos-7/ed0631d/logs/build/13:00
dmsimard:D13:00
*** lamt has joined #openstack-kolla13:03
berendtsdake i am not sure how kolla could be a part of our install guide13:04
berendtsdake can you involve me into this discussion, as a member of the openstack docs core i can help here13:04
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902713:07
deepak_jon#openstack-ansible13:07
*** schwicht has joined #openstack-kolla13:07
*** deepak_jon has left #openstack-kolla13:08
*** zhurong has quit IRC13:09
*** zhurong has joined #openstack-kolla13:10
*** lrensing has joined #openstack-kolla13:10
*** awiddersheim has joined #openstack-kolla13:11
*** dwalsh has quit IRC13:12
*** jtriley has joined #openstack-kolla13:13
*** dwalsh has joined #openstack-kolla13:13
*** zhubingbing has quit IRC13:14
openstackgerritMerged openstack/kolla: Fix error in Gnocchi service  https://review.openstack.org/37892713:14
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902713:19
*** lamt has quit IRC13:21
openstackgerritJeffrey Zhang proposed openstack/kolla: Mount host localtime for containers  https://review.openstack.org/37876213:21
*** zhurong has quit IRC13:21
sdakeberendt will do13:26
*** salv-orlando has joined #openstack-kolla13:27
*** zhurong has joined #openstack-kolla13:27
openstackgerritJeffrey Zhang proposed openstack/kolla: integrate gnocchi with ceilometer  https://review.openstack.org/37273713:27
openstackgerritOpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements  https://review.openstack.org/37598913:28
*** unicell has quit IRC13:28
sdakeberendt ok your in the thread13:31
sdakenot sure hwy this thread is private - but whatever, rolling with it13:31
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902713:31
*** salv-orlando has quit IRC13:31
mliimahi folks, the bridge_mappings is done during the build or deploy?13:34
sdakeberendt feel free to respond on thread13:34
berendtsdake still reading the thread13:34
pbourkemliima: during deploy13:34
mliimaok13:35
mliimathank you pbourke13:35
sdakeOK -> SLIDE DECK13:39
sdakeBBL13:39
berendtenjoy :)13:40
sdakeenjoy and slide deck are oxymorons berendt ;)13:40
berendtsdake i know ;)13:40
* sdake always feels like he is defending a phd thesis on 5 slides of deck13:41
sdakenot that i have defended a phd dissertation - but i get the impression its painful :)13:41
*** schwicht has quit IRC13:42
*** eaguilar has joined #openstack-kolla13:43
*** athomas has joined #openstack-kolla13:44
openstackgerritprameswar proposed openstack/kolla:  As of Ubuntu 16.04 apt instead of apt-get to install packages in Dockerfiles  https://review.openstack.org/37947713:49
jgriffithsdake: hogepodge Yeah, if that version of the patch works I can try and campaign for it again13:50
jgriffithsdake: hogepodge I just need to figure out how to convince the Redhat folks13:51
jgriffitherr... never mind, wrong patch; NOT the rhel folks for this one13:51
* jgriffith couple different things going on here :)13:51
sdakejgriffith the -1's have been reemoved13:51
jgriffithsdake: hogepodge the question I did have though was if this current iteration worked13:51
jgriffithsbezverk: ^^13:51
sdakejgriffith i am pretty sure it does not13:52
sdakejgriffith but i dont know because i dont know how to test it :(13:52
sdakesbezverk knows how13:52
jgriffithsdake: ok, so perhaps we should just start over , with the fix that was in patch-set 513:53
*** schwicht has joined #openstack-kolla13:56
*** matrohon has joined #openstack-kolla13:58
*** haplo37_ has quit IRC14:01
*** tonanhngo has joined #openstack-kolla14:02
sbezverksdake: what do I know?14:02
*** tonanhngo has quit IRC14:03
sdakesbezverk see openstack-glance14:03
sdakerather openstack-cinder14:03
sdakesorry jgriffith14:03
sdakehectic month14:03
sbezverkjgriffith: hey man14:03
jgriffithsbezverk: yo14:04
*** haplo37_ has joined #openstack-kolla14:04
sbezverkjgriffith: how can I help you?14:04
jgriffithsbezverk: I bumped that patch to the version 5 iteration that you had already tested14:04
jgriffithsbezverk: I'd love to know why the "not in" I added didn't work, but frankly it's not worth trying to figure out right now and fight the cinder team to merge it at the same time14:05
sbezverkjgriffith: sorry , which patch you are talking about? the latest we talked?14:06
*** neilus has joined #openstack-kolla14:07
*** yingjun has joined #openstack-kolla14:07
*** Pavo has quit IRC14:08
jgriffithLOL14:08
jgriffithsbezverk: https://review.openstack.org/#/c/306756/14:09
*** skramaja_ has joined #openstack-kolla14:09
sdakesbezverk it took a long time to get this one through the system14:09
*** skramaja has quit IRC14:09
*** Pavo has joined #openstack-kolla14:10
sdakejgriffith looking at the patch sset you uploaded it touoches a million files14:11
sdakeis that intended?14:11
sdakejgriffith or a rebase problem14:11
*** athomas has quit IRC14:11
jgriffithsdake: grrr... f'ing rebase14:12
sdakejgriffith right ;)14:13
*** skramaja_ has quit IRC14:13
jgriffithsdake: so I was screwing with a refactor and accidentally just merged all of that into the patch :)14:15
jgriffithsdake: I'll fix it momentarily14:15
openstackgerritChristian Berendt proposed openstack/kolla: Improve setup_nodes.yml playbook  https://review.openstack.org/37949514:19
openstackgerritChristian Berendt proposed openstack/kolla: Move loc tool into tools directory  https://review.openstack.org/37949714:23
*** sdake has quit IRC14:24
*** sdake has joined #openstack-kolla14:26
*** salv-orlando has joined #openstack-kolla14:29
openstackgerritChristian Berendt proposed openstack/kolla: Move demos and dev directory into a new contrib directory  https://review.openstack.org/37950014:31
*** inc0 has joined #openstack-kolla14:31
openstackgerritChristian Berendt proposed openstack/kolla: Improve playbook to setup gate nodes  https://review.openstack.org/37949514:34
*** sdake_ has joined #openstack-kolla14:34
*** eaguilar has quit IRC14:35
*** sdake has quit IRC14:36
*** eaguilar has joined #openstack-kolla14:38
*** MarMat has joined #openstack-kolla14:41
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902714:46
sdake_Jeffrey4l sbezverk ging to be using my boxes a bit to test jgriffith 's patch14:49
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805614:52
*** aswadr_ has joined #openstack-kolla14:53
*** sdake_ is now known as sdake_dnd14:53
*** zhurong has quit IRC14:54
openstackgerritChristian Berendt proposed openstack/kolla: Make gate setup script workable on non-gate nodes  https://review.openstack.org/37952014:54
*** egonzalez90 has quit IRC14:55
openstackgerritprameswar proposed openstack/kolla:  As of Ubuntu 16.04 apt instead of apt-get to install packages  https://review.openstack.org/37947714:58
*** sdake has joined #openstack-kolla14:59
*** sdake_dnd has quit IRC15:00
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805615:01
*** thumpba has joined #openstack-kolla15:01
*** dave-mccowan has quit IRC15:02
*** neilus has left #openstack-kolla15:05
sdakejroll re earlier, sorry got you confused with jim rollenhagen15:09
sdakesimilar names15:10
jrollsdake: that's me, what do you mean earlier?15:10
sdakejroll sorry i must be totaly confused :)15:10
* jroll even more so15:10
*** thumpba has quit IRC15:11
sdakejroll a fella from red hat joined the channel and he was sfrom redhat (with similar nick) and i was confused that he was running for the tc rpresenting red hat when his self nomination was ffrom rax :)15:11
sdakenm it doesn't really matter15:11
jrolloh ha! read up, yeah15:11
sdakei was just making conersation15:11
*** alwaysatthenoc has joined #openstack-kolla15:12
jrolljrist and I always get each other's pings :)15:12
sdakeoh right jrist15:12
*** thumpba has joined #openstack-kolla15:12
*** domi has quit IRC15:13
*** coolsvap has quit IRC15:22
MarMatJeffrey4l sdake Hi guys, looking for additional core reviewer (hoping for +2) for https://review.openstack.org/#/c/378221/ could you help with that?15:26
*** sdake is now known as sdake_dnd15:26
sdake_dndmarmat would love to help - but usper busy atm15:26
*** yingjun has quit IRC15:27
sdake_dndmarmat one q for you related to lbaas as you peel back the onion on magnum15:27
*** yingjun has joined #openstack-kolla15:27
MarMatsdake_dnd listening15:27
sdake_dndis load balancing used in magnum in our implementatino15:27
MarMatnot needed. Magnum newton has it switched off in configuration as far as I know. It's possible to switch it on there15:28
sdake_dndon the magnum ml, it was reported kubernetes still uses lbaasv1 implementation15:28
sdake_dndwe implement lbaasv215:28
sdake_dndand lbaasv1 is no longer15:29
MarMatyes. In earlier attempts with magnum mitaka I had to enforce lbaasv115:29
MarMatI did not try using lbaas recently with magnum newton, but I can double check15:29
*** neilus has joined #openstack-kolla15:31
*** neilus has quit IRC15:31
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805615:31
*** yingjun has quit IRC15:31
*** ldeptula has quit IRC15:32
*** thumpba has quit IRC15:34
pbourkeMarMat: done15:34
*** haplo37 has quit IRC15:35
*** g3ek has quit IRC15:35
*** g3ek has joined #openstack-kolla15:36
*** haplo37 has joined #openstack-kolla15:37
MarMatpbourke great, thanks15:37
*** huikang has joined #openstack-kolla15:37
*** huikang has quit IRC15:37
*** huikang has joined #openstack-kolla15:38
MarMatsdake_dnd magnum was switched to lbaasv2 recently15:38
openstackgerritMerged openstack/kolla: Fix wrong heat trustee configuration  https://review.openstack.org/37822115:39
openstackgerritMerged openstack/kolla: iscsi role misses 'storage' group  https://review.openstack.org/37938115:39
*** tonanhngo has joined #openstack-kolla15:42
MarMatsdake_dnd so to conclude: by default, magnum has lb switched off it its master template. It's possible to switch it on by changing template, and in such case lbaasv2 should be used in newton15:42
MarMatsdake_dnd but did not try this explicitly whether this works15:43
sdake_dndit does notwork because kubernetes does nto work15:43
*** tonanhngo has quit IRC15:44
inc0lol looking at my review queue15:44
inc0about 50% of reviews has -2 now15:45
inc0on the screen15:45
inc0<< life without master branch >>15:45
*** rmart04 has quit IRC15:45
rhalliseyinc0, you're becoming a dictator15:45
*** neilus has joined #openstack-kolla15:45
rhallisey:p15:45
inc0so guys, bugfixes will be only reviews merged until we branch out stable newton15:45
inc0rhallisey, it wasn;t me! I gave only few -2;)15:45
rhalliseyhehe15:46
pbourkeinc0: have another look at https://review.openstack.org/#/c/376524/ when you get a chance15:46
pbourkeplease :)15:46
inc0but that being said, I need to change my accent to more german'ish to induce obedience.15:46
berendti only gave one -2 ;)15:46
pbourkei always saw him as more of a north korean style dicator15:46
berendtare the gates broken at the moment?15:46
berendti see a lot of POST_FAILURE15:47
sdake_dndinc0 re gnocchi and ceilometer that thread has already been discussed at length15:47
inc0religious devotion towards myself will be required on P release15:47
sdake_dndinc0 those changes need to go in15:47
inc0pbourke, what about this review?15:47
pbourkeinc0: I thought the first PS was ok15:47
sdake_dnda post failure means infra is busted15:48
sdake_dndrun a recheck berendt15:48
pbourkeinc0: the second one is going to run setup on all hosts15:48
pbourkeneedlessly15:48
inc0well, so this is my thinking15:48
inc0a bit of underperformance that gives a bit of safety is good before release15:49
inc0especially that you can work arouind it as Jeffrey mentioned by adding --limit flag to playbook run15:49
*** openstackgerrit has quit IRC15:49
*** neilus has quit IRC15:49
*** openstackgerrit has joined #openstack-kolla15:50
inc0so if you really know that you don't want it to run anywhere else besides controller nodes, you just run ansible-playbook --limit control15:50
inc0however if we do targetted hosts and make a mistake there15:50
inc0we can end up with facts not being gathered where they're needed15:50
inc0with our inventory we can put any component on dedicated node15:51
inc0pretty much15:51
inc0are you 100000% sure that any compination of dedicated hosts will work? because this task ensures that15:52
pbourkeim not. I hate this part about ansible15:52
inc0yeah15:52
pbourkeyou're reasoning is good It hink15:52
pbourke*think15:52
inc0if we'd have CI system that would simulate that15:52
inc0I'd be ok with targeted hosts15:52
pbourkethat gather_facts: false thing is a bit funky15:53
inc0that's ansible stuff15:53
inc0if we wouldn't put it there15:53
*** tonanhngo has joined #openstack-kolla15:53
pbourkeawiddersheim: im going to add one or two notes to the patch, then lets try get it merged15:53
inc0it would gather facts because "hey, it's a task" and then it would find out that it is a task that gathers facts so it would gather it again15:53
inc0alternatively we could put task "debug" and "gather_facts: true" ;)15:54
*** tonanhngo has quit IRC15:54
awiddersheimI kind of like the debug thing15:55
awiddersheimwith a run_once15:55
awiddersheimit's a bit more readable15:55
awiddersheimas to what it's doing15:56
inc0well...meh not sure tbh15:56
inc0setup is dedicated task for it15:56
inc0debug is more implicit15:56
inc0setup task is explicit, setup gathers facts period15:57
pbourkeas inc0 says lets go with what we're 1000% sure will work15:57
pbourkeas this is kind of crucial to the entire playbook15:57
inc0both will work15:57
inc0it's cosmetics really15:57
sdake_dndlets not break things lets ix things ;)15:57
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805615:58
inc0sdake has spoken, let us all adhere to his wisdom15:58
inc0and not break things;)15:58
*** mdnadeem_home has joined #openstack-kolla15:59
*** berendt has quit IRC15:59
awiddersheimI'm going to test the debug thing to make sure it doesn't break things and go with that ;)16:00
awiddersheimalong iwth the other changes16:00
awiddersheimand put up another PS16:00
inc0awiddersheim, please stay on setup16:00
awiddersheimvery well16:01
awiddersheimsetup it is16:01
inc0you can just add name to this task16:01
inc0and print out something like "gathering facts on nodes"16:01
awiddersheimgood idea16:02
jgriffithsdake_dnd: sbezverk I have a question on this16:02
jgriffithsdake_dnd: sbezverk this being the glance fix in Cinder16:02
pbourkeinc0: when you 'kolla-ansible destroy' your ceph nodes, it updates fstab but doesn't unmount the disk(s)... Do you do this manually or any reason we can't have destroy do that?16:02
*** eaguilar has quit IRC16:03
jgriffithsdake_dnd: sbezverk so the challenge I have here is that Glance has varying behavior dependent on the method you call... it may return None, or [] or raise NotFound16:03
*** absubram has joined #openstack-kolla16:03
jgriffithsdake_dnd: sbezverk The only case I see so far that None is not a valid thing is the data method16:03
inc0pbourke, I was kinda thinking about it myself;) I think issue is it would be really hard to detect ceph disks16:04
pbourkeinc0: kk I'll have a look16:04
jgriffithsbezverk: sdake_dnd I'm not sure, but I *believe* the only case that's a problem for you is the method call to "data"16:04
inc0we could somehow parse fstab before we delete entries16:05
MarMatinc0 the magnum proxy setting in globals.yml you recommended yesterday working, I'm trying to write an example in universal way so it can be updated in ticket and used by others.16:05
inc0MarMat, you can write a doc note too when you're at it;)16:06
inc0(just suggesting ofc;))16:06
MarMatinc0 ok will take a look on it16:06
inc0we have proxy section in docs already, that would be good addition there16:07
inc0thanks16:07
*** huikang has quit IRC16:09
*** huikang has joined #openstack-kolla16:10
sdake_dndjgriffith downlod method none is invalid16:12
*** msimonin has quit IRC16:12
jgriffithsdake_dnd: correct, but download will return NotFound16:12
jgriffithsdake_dnd: at least that's what it appears to me16:13
sdake_dndright - so notfound happens if we roundrobin the connections tot he api16:13
jgriffithsdake_dnd: so I'm not follow... you just said None is invalid for download16:13
jgriffithsdake_dnd: I'm saying correct, that's not something you should ever see because glance should not return that16:14
sdake_dndjgriffith that is my recollection when i debugged it16:14
*** huikang has quit IRC16:14
jgriffithsdake_dnd: looking at the glance code and the cinder code, it *appears* to me the only case that returns None and creates confusion is the data method16:14
jgriffithsdake_dnd: are you saying that you saw the error occur during download?16:15
jgriffithsdake_dnd: ie when calling the download method?16:15
sdake_dndit may have been the data method16:16
sdake_dndi'm not sure what method it was16:16
sdake_dndin a meeting atm can't do two things at once :)16:16
jgriffithsdake_dnd: understood :)  Maybe I'll push the change with just the data method and see if sbezverk can test it16:17
sdake_dndsbezverk is testing now16:18
sdake_dndjgriffith ^^16:19
*** dave-mccowan has joined #openstack-kolla16:19
jgriffithsdake_dnd: sbezverk ok, well I just pushed yet another iteration https://review.openstack.org/#/c/306756/16:19
sdake_dndjgriffith the issue as i recall is that download failed with "None" assertion16:20
sdake_dndlet me check out the bug16:20
jgriffithsdake_dnd: we can chat after your meeting if you like, but there's  a big problem with just adding a blanket "if None; raise"16:21
sdake_dndi hear ya16:21
sdake_dndjgriffith i am not a cinder expert and did the best i could to debug it16:22
sdake_dndi don't see the backtrace in the bug log unfortunately16:22
sdake_dndif we had that backtrac,e I could tell you what they were thinking16:22
sdake_dndwhati was thinking that is16:22
*** sdake_dnd is now known as sdake16:27
sdakejgriffith16:27
sdaketake a look at https://bugs.launchpad.net/cinder/+bug/157121116:28
openstackLaunchpad bug 1571211 in Cinder "In multinode setup VM fails to launch due to cinder not checking with all glance api servers" [Medium,In progress] - Assigned to John Griffith (john-griffith)16:28
sdakecomment #1716:28
sdakejgriffith let me pull cinder code moment16:30
*** matrohon has quit IRC16:30
jgriffithsdake: the problem here is this really isn't a Cinder bug, sorry you don't agree.  The Glance API isn't exactly consistent or making your life easy here16:30
sdakei agree it isn't a cinder bug16:30
jgriffithsdake: I could be wrong I guess16:31
sdakeits a disconnect between what glance does and what it should be doing16:31
jgriffithsdake: I'm just trying to come up with a way to work around / fix this on the Cinder side without breaking everything else16:31
jgriffithsdake: if I do a blanket conversion to NotFound for any None return from Glance that breaks the worlds existing behaviors16:31
jgriffithsdake: maybe we don't care, but maybe we do.... kind of risky IMO16:32
*** huikang has joined #openstack-kolla16:33
sdakejgriffith a workaround is good16:33
sdakethere is one specific method that has this problem, i am working on finding it for you16:34
sdakei need to reload my context from t-6 months ago16:34
sdakewhich means i need to pull the repo16:34
*** mdnadeem_home has quit IRC16:44
*** gfhellma has joined #openstack-kolla16:45
*** eaguilar has joined #openstack-kolla16:45
*** dave-mccowan has quit IRC16:46
*** dave-mccowan has joined #openstack-kolla16:46
*** srwilkers has joined #openstack-kolla16:52
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902716:53
sdakei'm sorry jgriffith today just isn't may day - scrambling internally with this reorg nonsense16:55
sdakejgriffith can i get back to you tomorrow with wht i think the root cause was from my recollection?16:55
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805616:55
*** haplo37_ has quit IRC16:56
*** haplo37_ has joined #openstack-kolla16:58
*** eaguilar has quit IRC17:01
sdakekfox1111 see msgs plz17:02
*** gfhellma has quit IRC17:04
sdakeinc0 about - meetin gon ecurity in openstack-meeting-alt17:05
*** sean-k-mooney has joined #openstack-kolla17:08
sean-k-mooneypbourke: i was just wondering if you had any look with that ceph issue? im not back to work until monday but really would like to figure out how to fix it17:09
*** msimonin has joined #openstack-kolla17:09
*** dwalsh has quit IRC17:15
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805617:15
jgriffithsdake: no problem at all17:19
*** eaguilar has joined #openstack-kolla17:20
openstackgerritBorneMace proposed openstack/kolla: Adding some optional destroy behavior  https://review.openstack.org/37783717:21
*** aswadr_ has quit IRC17:22
*** tonanhngo has joined #openstack-kolla17:25
*** huikang has quit IRC17:26
*** huikang has joined #openstack-kolla17:27
*** tonanhngo has quit IRC17:29
*** huikang has quit IRC17:31
*** dwalsh has joined #openstack-kolla17:31
*** gfidente is now known as gfidente|afk17:31
*** tonanhngo has joined #openstack-kolla17:32
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805617:37
*** unicell has joined #openstack-kolla17:40
*** unicell has quit IRC17:40
inc0Pavo, ping17:41
Pavosup inc017:41
inc0hey, so we're working with openstack security team to make good thread analysys and such17:42
Pavonice17:42
inc0I figured it might interest you a lot:) I was hoping that you could point us to any potential security loopholes you'd stumble upon17:43
inc0no matter how deep in our arch it's rooted17:43
PavoI just got out of the hospital this morning17:43
inc0damn, sorry to hear that17:43
*** unicell has joined #openstack-kolla17:43
*** msimonin has quit IRC17:43
Pavowell one major security issue I found was dns prefix being advertised outside openstack to local machines that were on the same vlan17:44
Pavoyeah had all my upper teeth removed and a temp denture put in17:44
Pavoall good right now17:44
Pavoso far no pain17:44
Pavojust very swollen17:44
*** unicell1 has joined #openstack-kolla17:44
*** unicell has quit IRC17:44
*** schwicht has quit IRC17:45
*** inc0_ has joined #openstack-kolla17:45
inc0_sorry, vpn died17:45
Pavowell one major security issue I found was dns prefix being advertised outside openstack to local machines that were on the same vlan17:46
Pavoyeah had all my upper teeth removed and a temp denture put in17:46
Pavoall good right now17:46
Pavoso far no pain17:46
Pavojust very swollen17:46
inc0_sorry to hear that17:47
inc0_dns prefix? which service did that?17:47
Pavodhcp_agent is given out dns prefix17:48
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805617:48
*** inc0 has quit IRC17:48
inc0_is it possible to configure it?17:48
Pavonot that I can figure out17:49
Pavoonly thing I could do is black it on the switch from broadcasting out that vlan17:49
Pavoblock17:49
openstackgerritMerged openstack/kolla-kubernetes: common_volume_mounts and common_containers macros  https://review.openstack.org/37902717:49
openstackgerritMerged openstack/kolla-kubernetes: Use kubectl to watch for pod status  https://review.openstack.org/37873817:49
inc0_yeah, beyond our control really, might be something we could configure in neutron17:50
inc0_or dnsmasq17:50
Pavomybe dnsmasq17:50
*** neilus has joined #openstack-kolla17:51
*** lukl1 has quit IRC17:51
*** lukl has joined #openstack-kolla17:52
inc0_Pavo, would you submit a but for it?17:53
inc0_we can dig through it and see if we, kolla can help17:54
inc0_anyway, if you see anything, make sure to let us know17:54
*** gfhellma has joined #openstack-kolla17:55
inc0_if it's super critical and we don't want to publish it, at least let me know so I can escalate it up to openstack security team so we can take measures to fix it17:55
*** huikang has joined #openstack-kolla17:57
*** msimonin has joined #openstack-kolla18:01
*** msimonin has quit IRC18:01
Pavowill do18:02
*** schwicht has joined #openstack-kolla18:07
*** lrensing has quit IRC18:12
*** schwicht has quit IRC18:16
*** dwalsh has quit IRC18:19
*** g3ek has quit IRC18:20
*** cu5 has quit IRC18:20
*** cu5 has joined #openstack-kolla18:21
*** haplo37 has quit IRC18:21
*** bmace_ has quit IRC18:21
*** bmace_ has joined #openstack-kolla18:21
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805618:26
*** g3ek has joined #openstack-kolla18:27
*** dwalsh has joined #openstack-kolla18:27
*** haplo37 has joined #openstack-kolla18:27
*** eaguilar has quit IRC18:28
*** huikang has quit IRC18:33
*** huikang has joined #openstack-kolla18:33
*** huikang has quit IRC18:34
*** huikang has joined #openstack-kolla18:34
*** eaguilar has joined #openstack-kolla18:38
*** schwicht has joined #openstack-kolla18:40
*** jeh has joined #openstack-kolla18:42
*** jeh has quit IRC18:42
*** jeh has joined #openstack-kolla18:42
*** nihilifer has quit IRC18:43
*** nihilifer has joined #openstack-kolla18:43
*** sean-k-mooney has quit IRC18:44
*** dwalsh has quit IRC18:45
*** narasimha_SV has joined #openstack-kolla18:46
narasimha_SVwhile deploying using kolla18:46
narasimha_SVgetting "Invalid volume spec \"n\": volumeabs: Invalid volume destination path: \\'n\\' mount path must be absolute.\")\\n'"} error18:46
*** jeh has quit IRC18:47
*** neilus has quit IRC18:47
narasimha_SVhttp://paste.openstack.org/show/583541/18:48
narasimha_SVthese are ansible and docker version on my deploy node18:48
*** sdake has quit IRC18:50
*** sdake has joined #openstack-kolla18:50
*** schwicht has quit IRC18:53
*** lamt has joined #openstack-kolla18:55
*** salv-orlando has quit IRC18:56
*** neilus has joined #openstack-kolla18:56
inc0_narasimha_SV, can you put full error log?18:58
narasimha_SVinc0_: http://paste.openstack.org/show/583542/18:59
*** neilus has quit IRC19:00
inc0_looks like docker config issue, which task that was?19:01
inc0_was that*19:01
narasimha_SVdeplying mysql19:02
inc0_did you change anything with you config directory or anything?19:02
narasimha_SVno19:05
narasimha_SVgenerally when we get this issue ?19:05
*** salv-orlando has joined #openstack-kolla19:18
openstackgerritAndrew Widdersheim proposed openstack/kolla: Remove redundant common play and add haproxy hosts  https://review.openstack.org/37652419:23
*** huikang has quit IRC19:25
*** huikang has joined #openstack-kolla19:26
*** dwalsh has joined #openstack-kolla19:30
*** huikang has quit IRC19:31
*** matrohon has joined #openstack-kolla19:36
sdakeyo19:38
sdakeanyone blocked in any way?19:39
*** elemoine_ has quit IRC19:39
*** lrensing has joined #openstack-kolla19:40
*** elemoine_ has joined #openstack-kolla19:41
*** cloudnull has quit IRC19:43
*** flaper87 has quit IRC19:43
*** manjeets has quit IRC19:43
*** afranc has quit IRC19:43
*** zhenguo has quit IRC19:43
*** akwasnie has quit IRC19:43
*** bjolo has quit IRC19:43
*** cloudnull has joined #openstack-kolla19:43
*** tonanhngo has quit IRC19:44
*** salv-orl_ has joined #openstack-kolla19:44
*** srwilkers has quit IRC19:44
*** manjeets has joined #openstack-kolla19:46
*** salv-orlando has quit IRC19:47
*** akwasnie has joined #openstack-kolla19:47
*** afranc has joined #openstack-kolla19:47
*** NobodyCam_ has joined #openstack-kolla19:48
*** NobodyCam has quit IRC19:50
*** NobodyCam_ is now known as NobodyCam19:50
*** huikang has joined #openstack-kolla19:50
*** akwasnie has quit IRC19:53
*** zhenguo has joined #openstack-kolla19:53
*** akwasnie has joined #openstack-kolla19:53
*** narasimha_SV has quit IRC19:54
*** bjolo has joined #openstack-kolla19:55
*** akwasnie has quit IRC19:58
*** akwasnie has joined #openstack-kolla20:00
*** ayoung has quit IRC20:01
*** tonanhngo has joined #openstack-kolla20:05
*** eaguilar has quit IRC20:05
*** eaguilar has joined #openstack-kolla20:06
sdakeout of batteries, need to find charger bbiaf20:08
*** sdake has quit IRC20:08
*** tonanhngo has quit IRC20:10
*** lamt has quit IRC20:12
*** lrensing has quit IRC20:15
*** nihilifer has quit IRC20:25
*** ayoung has joined #openstack-kolla20:27
*** ccesario has quit IRC20:33
*** Jeffrey4l has quit IRC20:35
*** Jeffrey4l has joined #openstack-kolla20:35
openstackgerritChristian Berendt proposed openstack/kolla: Make gate setup script workable on non-gate nodes  https://review.openstack.org/37952020:35
*** jtriley has quit IRC20:37
*** schwicht has joined #openstack-kolla20:39
openstackgerritChristian Berendt proposed openstack/kolla: Use correct Kibana version on the gates  https://review.openstack.org/37981520:39
*** ayoung has quit IRC20:39
*** schwicht has quit IRC20:45
*** jamesdenton has quit IRC20:46
*** schwicht has joined #openstack-kolla20:48
*** eaguilar has quit IRC20:51
*** huikang has quit IRC20:53
*** huikang has joined #openstack-kolla20:54
*** awiddersheim has quit IRC20:56
*** lamt has joined #openstack-kolla20:56
*** huikang has quit IRC20:58
*** eaguilar has joined #openstack-kolla20:58
*** lamt has quit IRC21:02
*** schwicht has quit IRC21:07
*** inc0_ has quit IRC21:07
*** Serlex has quit IRC21:09
*** sdake has joined #openstack-kolla21:13
sdakesup21:14
*** matrohon has quit IRC21:15
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805621:16
*** eaguila__ has joined #openstack-kolla21:17
*** eaguilar has quit IRC21:17
*** fguillot has quit IRC21:20
g3eksup21:21
openstackgerritChristian Berendt proposed openstack/kolla: Make gate setup script workable on non-gate nodes  https://review.openstack.org/37952021:21
*** jheroux has quit IRC21:24
openstackgerritChristian Berendt proposed openstack/kolla: Create swapfile only on nodepool nodes  https://review.openstack.org/37983121:28
*** eaguila__ has quit IRC21:29
*** msimonin has joined #openstack-kolla21:31
*** haplo37_ has quit IRC21:37
*** Pavo has quit IRC21:38
*** haplo37_ has joined #openstack-kolla21:39
*** schwicht has joined #openstack-kolla21:40
*** Pavo has joined #openstack-kolla21:44
*** britthouser1 has quit IRC21:45
*** britthouser has joined #openstack-kolla21:45
*** dwalsh has quit IRC21:55
*** neilus has joined #openstack-kolla21:56
*** alwaysatthenoc has quit IRC21:59
*** tonanhngo has joined #openstack-kolla21:59
*** absubram has quit IRC22:12
openstackgerritChristian Berendt proposed openstack/kolla: Make gate setup script workable on non-gate nodes  https://review.openstack.org/37952022:17
*** neilus has quit IRC22:17
*** neilus has joined #openstack-kolla22:18
*** neilus has quit IRC22:23
*** neilus has joined #openstack-kolla22:26
*** ayoung has joined #openstack-kolla22:29
*** neilus has quit IRC22:30
*** ntpttr has quit IRC22:31
*** ntpttr has joined #openstack-kolla22:31
*** neilus has joined #openstack-kolla22:31
openstackgerritOpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements  https://review.openstack.org/37598922:32
*** neilus has quit IRC22:33
*** neilus has joined #openstack-kolla22:34
*** neilus has quit IRC22:38
*** neilus has joined #openstack-kolla22:39
*** neilus has quit IRC22:44
*** inc0 has joined #openstack-kolla22:45
*** msimonin has quit IRC22:45
*** neilus has joined #openstack-kolla22:46
*** neilus has quit IRC22:47
*** neilus_ has joined #openstack-kolla22:48
*** neilus has joined #openstack-kolla22:49
*** neilus_ has quit IRC22:50
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805622:51
*** neilus_ has joined #openstack-kolla22:51
*** neilus_ has quit IRC22:53
*** neilus_ has joined #openstack-kolla22:53
*** neilus has quit IRC22:54
*** neilus has joined #openstack-kolla22:55
*** neilus__ has joined #openstack-kolla22:56
*** neilus_ has quit IRC22:57
*** neilus__ has quit IRC22:58
*** neilus_ has joined #openstack-kolla22:58
*** neilus has quit IRC22:59
*** neilus has joined #openstack-kolla23:00
*** neilus_ has quit IRC23:00
*** neilus has quit IRC23:01
kfox1111inc0: you going to b eable to make it?23:02
inc0I'm on hangouts23:03
*** huikang has joined #openstack-kolla23:03
inc0...I think23:03
*** neilus has joined #openstack-kolla23:03
kfox1111I see justin, qin and me.23:04
*** daneyon has joined #openstack-kolla23:04
*** neilus has quit IRC23:05
kfox1111we can year you.23:05
kfox1111can you hear us?23:05
*** neilus has joined #openstack-kolla23:05
*** ayoung has quit IRC23:06
inc0sbezverk, ping23:06
*** awiddersheim has joined #openstack-kolla23:06
*** neilus has quit IRC23:07
*** neilus has joined #openstack-kolla23:08
*** daneyon has quit IRC23:09
*** yingjun has joined #openstack-kolla23:09
*** neilus_ has joined #openstack-kolla23:09
*** neilus_ has quit IRC23:11
sbezverkinc0: here23:11
*** neilus_ has joined #openstack-kolla23:11
sbezverkemail you sent said 6PM PST23:11
sbezverkso it is 9PM my time23:12
*** neilus has quit IRC23:12
*** sdake has quit IRC23:12
inc0kk we're on now if you want, join in:)23:12
*** neilus_ has quit IRC23:13
*** neilus has joined #openstack-kolla23:13
sbezverkjoined23:14
*** yingjun has quit IRC23:14
*** neilus has quit IRC23:14
*** neilus has joined #openstack-kolla23:15
*** neilus_ has joined #openstack-kolla23:16
*** neilus has quit IRC23:16
*** neilus has joined #openstack-kolla23:18
*** neilus_ has quit IRC23:18
*** neilus_ has joined #openstack-kolla23:20
*** neilus has quit IRC23:20
*** neilus_ has quit IRC23:21
*** neilus has joined #openstack-kolla23:22
*** neilus_ has joined #openstack-kolla23:23
*** neilus has quit IRC23:23
MarMatinc0 the magnum proxy workaround... do you think that inserting into documentation is sufficient or should an example be inserted into globals.yml?23:24
*** neilus_ has quit IRC23:25
*** neilus has joined #openstack-kolla23:25
inc0rhallisey, jascott1 qwang :)23:25
jascott1hi23:25
qwanghi23:25
kfox1111hi23:25
rhalliseyhey!23:26
sbezverkhi23:26
*** neilus_ has joined #openstack-kolla23:27
*** neilus_ has quit IRC23:29
*** neilus_ has joined #openstack-kolla23:29
*** neilus_ has quit IRC23:30
*** neilus has quit IRC23:30
*** neilus_ has joined #openstack-kolla23:31
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805623:31
*** neilus_ has quit IRC23:32
*** neilus has joined #openstack-kolla23:32
*** neilus has quit IRC23:34
sbezverklost audio23:34
*** neilus has joined #openstack-kolla23:34
*** neilus has quit IRC23:36
*** neilus_ has joined #openstack-kolla23:36
kfox1111fencing discussion: https://github.com/kubernetes/kubernetes/issues/3176223:37
*** neilus_ has quit IRC23:38
*** neilus has joined #openstack-kolla23:38
*** neilus has quit IRC23:39
*** neilus_ has joined #openstack-kolla23:39
*** Pavo has quit IRC23:41
*** neilus has joined #openstack-kolla23:41
*** neilus_ has quit IRC23:41
*** neilus has quit IRC23:43
*** neilus has joined #openstack-kolla23:43
*** zhurong has joined #openstack-kolla23:45
*** neilus has quit IRC23:45
*** neilus_ has joined #openstack-kolla23:45
*** neilus_ has quit IRC23:46
*** neilus has joined #openstack-kolla23:47
*** neilus has quit IRC23:48
*** neilus_ has joined #openstack-kolla23:48
*** Pavo has joined #openstack-kolla23:49
*** neilus_ has quit IRC23:50
*** neilus has joined #openstack-kolla23:50
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: Test setting up the gate  https://review.openstack.org/37805623:51
*** ayoung has joined #openstack-kolla23:52
*** neilus has quit IRC23:52
*** neilus has joined #openstack-kolla23:52
*** neilus has quit IRC23:53
*** eaguilar has joined #openstack-kolla23:54
*** neilus has joined #openstack-kolla23:54
*** sdake has joined #openstack-kolla23:55
*** neilus_ has joined #openstack-kolla23:55
*** neilus_ has quit IRC23:57
*** neilus_ has joined #openstack-kolla23:57
*** neilus has quit IRC23:58
*** neilus_ has quit IRC23:59
*** neilus has joined #openstack-kolla23:59

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!