Sunday, 2016-09-25

« Saturday, 2016-09-24 Index Monday, 2016-09-26 »
*** daneyon has quit IRC00:01
*** neilus has joined #openstack-kolla00:10
*** jascott1 has quit IRC00:15
*** jascott1- is now known as jascott100:15
*** neilus has quit IRC00:15
*** eaguilar has joined #openstack-kolla00:26
*** sdake has joined #openstack-kolla00:44
*** jascott1_ has joined #openstack-kolla00:49
*** rbergero1 is now known as rbergeron00:49
*** rbergeron has quit IRC00:50
*** rbergeron has joined #openstack-kolla00:50
*** sdake has quit IRC00:53
*** eaguilar has quit IRC00:53
*** eaguilar has joined #openstack-kolla00:54
*** bjolo__ has quit IRC01:08
*** rhallisey has quit IRC01:12
*** jascott1_ has quit IRC01:15
*** caowei has joined #openstack-kolla01:20
*** jascott1_ has joined #openstack-kolla01:24
*** jascott1_ has quit IRC01:28
*** jascott1_ has joined #openstack-kolla01:33
*** jrist has quit IRC01:34
*** zhubingbing has joined #openstack-kolla01:35
*** sdake has joined #openstack-kolla01:49
*** tonanhngo has joined #openstack-kolla01:55
*** tonanhngo has quit IRC01:56
*** neilus has joined #openstack-kolla01:58
*** neilus has quit IRC02:03
*** Jeffrey4l has quit IRC02:12
*** jascott1_ has quit IRC02:24
*** sdake has quit IRC02:27
*** daneyon has joined #openstack-kolla02:39
*** tonanhngo has joined #openstack-kolla02:43
*** daneyon has quit IRC02:43
*** tonanhngo has quit IRC02:45
*** openstackgerrit has joined #openstack-kolla02:45
openstackgerritzhubingbing proposed openstack/kolla: remove bootstrap.yml in grafana role upgrade.yml  https://review.openstack.org/37591302:46
*** zhubingbing has quit IRC02:55
*** eaguilar has quit IRC03:00
*** david-lyle_ has quit IRC03:03
*** zhubingbing has joined #openstack-kolla03:08
*** tonanhngo has joined #openstack-kolla03:14
*** tonanhngo has quit IRC03:16
openstackgerritWei Cao proposed openstack/kolla: Add freezer ansible role  https://review.openstack.org/37587403:17
*** unicell has joined #openstack-kolla03:21
*** jascott1_ has joined #openstack-kolla03:25
*** jascott1_ has quit IRC03:30
*** tonanhngo has joined #openstack-kolla03:33
openstackgerritWei Cao proposed openstack/kolla: Revert "Add enable_elasticsearch option"  https://review.openstack.org/37519803:33
openstackgerritWei Cao proposed openstack/kolla: Revert "Add enable_elasticsearch option"  https://review.openstack.org/37519803:33
*** tonanhngo has quit IRC03:35
*** jamesdenton has joined #openstack-kolla03:36
*** neilus has joined #openstack-kolla03:47
*** neilus has quit IRC03:52
openstackgerritWei Cao proposed openstack/kolla: Change "Virtualhost" to "VirtualHost" in horizon.conf.j2  https://review.openstack.org/37592904:05
*** kbyrne has quit IRC04:09
*** kbyrne has joined #openstack-kolla04:13
*** g3ek has quit IRC04:49
*** g3ek has joined #openstack-kolla04:50
*** Jeffrey4l has joined #openstack-kolla04:55
*** tonanhngo has joined #openstack-kolla04:57
*** jascott1_ has joined #openstack-kolla05:00
*** daneyon has joined #openstack-kolla05:21
*** daneyon has quit IRC05:26
*** neilus has joined #openstack-kolla05:35
*** neilus has quit IRC05:39
*** zhubingbing has quit IRC06:01
*** senk_ has quit IRC06:29
*** unicell has quit IRC06:43
*** zhubingbing has joined #openstack-kolla07:07
*** haplo37 has quit IRC07:08
*** daneyon has joined #openstack-kolla07:10
*** haplo37 has joined #openstack-kolla07:11
*** daneyon has quit IRC07:14
*** tonanhngo has quit IRC07:20
*** neilus has joined #openstack-kolla07:23
*** neilus has quit IRC07:28
*** tonanhngo has joined #openstack-kolla07:43
*** tonanhngo has quit IRC07:45
*** senk_ has joined #openstack-kolla07:46
openstackgerritMerged openstack/kolla: Add aodh-api to haproxy.cfg file  https://review.openstack.org/37519708:05
openstackgerritMerged openstack/kolla: Delete tempest.yml playbook  https://review.openstack.org/37587908:07
*** g3ek has quit IRC08:14
*** 14WAAFND0 has quit IRC08:16
*** g3ek has joined #openstack-kolla08:17
*** haplo37_ has joined #openstack-kolla08:18
*** jascott1_ has quit IRC08:22
*** egonzalez90 has joined #openstack-kolla08:24
*** jascott1_ has joined #openstack-kolla08:28
openstackgerritEduardo Gonzalez proposed openstack/kolla: Fix mistral service  https://review.openstack.org/37492108:29
*** egonzalez90 has quit IRC08:31
*** rmart04 has joined #openstack-kolla08:38
*** senk_ has quit IRC08:58
*** senk__ has joined #openstack-kolla08:58
*** rmart04 has quit IRC08:59
*** dwalsh has joined #openstack-kolla09:11
*** neilus has joined #openstack-kolla09:11
*** neilus has quit IRC09:16
*** rmart04 has joined #openstack-kolla09:36
*** senk__ has quit IRC09:46
*** senk_ has joined #openstack-kolla09:46
*** rmart04 has quit IRC09:49
*** jascott1_ has quit IRC09:56
openstackgerritOpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements  https://review.openstack.org/37598909:57
*** rmart04 has joined #openstack-kolla10:05
*** rmart04 has quit IRC10:05
*** rmart04 has joined #openstack-kolla10:06
*** neilus has joined #openstack-kolla10:07
*** neilus has quit IRC10:12
*** rmart04 has quit IRC10:24
*** senk_ has quit IRC10:26
*** senk__ has joined #openstack-kolla10:26
*** rmart04 has joined #openstack-kolla10:30
*** neilus has joined #openstack-kolla10:30
*** neilus has quit IRC10:39
*** tonanhngo has joined #openstack-kolla10:40
*** rmart04 has quit IRC10:40
*** haplo37 has quit IRC10:41
*** haplo37_ is now known as haplo3710:41
*** tonanhngo has quit IRC10:41
*** haplo37_ has joined #openstack-kolla10:43
*** daneyon has joined #openstack-kolla10:46
*** daneyon has quit IRC10:51
*** zhubingbing has quit IRC11:08
*** zhubingbing_ has joined #openstack-kolla11:11
*** g3ek has quit IRC11:13
*** haplo37 has quit IRC11:14
*** zhurong has joined #openstack-kolla11:15
*** haplo37 has joined #openstack-kolla11:16
*** g3ek has joined #openstack-kolla11:19
*** dwalsh has quit IRC11:23
openstackgerritcaoyuan proposed openstack/kolla: Update the action when create Glance service endpoints  https://review.openstack.org/37600611:49
*** neilus has joined #openstack-kolla11:58
*** neilus has quit IRC12:03
*** zhubingbing_ has quit IRC12:11
openstackgerritJeffrey Zhang proposed openstack/kolla: Optimized reconfigure action for glance  https://review.openstack.org/37601112:16
*** sdake has joined #openstack-kolla12:19
*** g3ek has quit IRC12:22
*** g3ek has joined #openstack-kolla12:29
*** senk__ has quit IRC12:33
*** daneyon has joined #openstack-kolla12:34
*** daneyon has quit IRC12:39
*** neilus has joined #openstack-kolla12:47
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP: basic ceph deployment support  https://review.openstack.org/37325612:50
*** kong has quit IRC12:51
*** jamesdenton has quit IRC12:51
*** kong has joined #openstack-kolla12:52
*** david-lyle has joined #openstack-kolla12:56
*** schwicht has joined #openstack-kolla13:01
sdakeJeffrey4l about?13:04
Jeffrey4lsdake, yep13:04
*** haplo37_ has quit IRC13:18
*** haplo37_ has joined #openstack-kolla13:19
sdakeJeffrey4l sorry got distracted, is mitaka gate working now?13:20
Jeffrey4lsdake, no. this PS is not merged still https://review.openstack.org/37541813:24
*** zhurong has quit IRC13:24
sdakeany core reviewers around at this godawful hour? :)13:25
*** dotplus_ is now known as dotplus13:32
sdakeJeffrey4l i believe liberty becomes eol at conclusion of this cycle13:43
sdakeJeffrey4l if we have time we ought to releasse one last zstream13:43
Jeffrey4lsdake, i agreed. and liberty gate always red :(13:43
sdakei think that is caused by a combination of problems13:44
sdakeone of hiwch is we are using an old version of rabbitmq13:44
Jeffrey4lyes.13:44
sdakea new version of rabbitmq prevents the rabbitmq crash13:44
Jeffrey4li think so.13:44
sdakeother issues you are aware of?13:45
sdaketo wither in denail13:46
sdakethe bitterness of one who is left alone13:46
sdake845 tubes sounding fantastic13:49
sdakegerman engineering ftw13:49
Jeffrey4lliberty branch? no. I tried to fix it, but block on rabbitmq issue.13:52
sdakeeven with rabbitmq backport?13:53
Jeffrey4lyep.13:53
sdakesame issue?13:53
Jeffrey4llong time ago. ( with my one of my rabbitmq fix backport )13:53
sdakeoh - i am talking about the rework on rabbitmq versions we did in master13:53
sdakeyou now the one i started and 3 or 4 peple finished ;)13:54
Jeffrey4lhaven't check that .13:54
sdakelet me give it a go13:54
Jeffrey4lsdake, did u push a backport?13:54
sdakenope13:54
sdakei'm a bit wary of changing rabbitmq version in mitaka13:55
Jeffrey4lOK. let's try.13:55
sdakei keep hearing problems with rc113:55
sdakeand rabbitmq container restarting13:55
sdakemlima suffering from this13:56
openstackgerritJeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST LIBERTY BRANCH  https://review.openstack.org/32313313:58
Jeffrey4lsdake, pushed a test PS to check what' going wrong now. sdake ^13:58
sdakeyet master works fantastically well for me13:58
sdakemultinode and single nodee13:58
Jeffrey4lyep. I am making tests for master in recent days.13:59
sdakefound any problems bessdiesss the 100 or so bugs in our backlog? :)13:59
Jeffrey4lno big issue.14:00
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the failure of ceilometer-api container  https://review.openstack.org/37216114:01
Jeffrey4lsdake, and if you have time, please review this https://review.openstack.org/#/c/372161/14:01
sdakeJeffrey4l looking now14:02
sdakeJeffrey4l any chance you can help me sort out which bugs are important vs not for 3.0.014:02
sdakeso we can make a final call on which bugs to eject to ocata14:03
Jeffrey4lbase on the bug list?14:03
kfox1111anyone that konws libvirt secrets awake?14:03
sdakehuh kfox?14:03
sdakeJeffrey4l ya, the rc2 buglist14:03
sdakeJeffrey4l i asked cr tteam to triage them14:03
sdakenot sure how much that hapepned or didn't14:03
sdakebut lets make a final call today14:03
kfox1111I've got stuff in /etc/libvirt/secrets/<uuid>.base64 and .xml14:03
kfox1111but nothing shows in virsh secret-list14:04
Jeffrey4li will check the list..14:04
sdakekfox1111 hmm no idea14:04
sdakekfox1111 what is a libvirt seecret - lets start there14:04
kfox1111I've always had to do a virsh secret-create to import them,14:04
kfox1111but  I don't see any code in kolla to do that.14:04
kfox1111sdake: they are primarily used to load ceph keys into libvirt.14:04
sdakeok14:05
sdakethis is in k8s environment14:05
sdake?14:05
kfox1111yeah.14:05
kfox1111I've read through the ansible stuff, and tryhing it in k8s.14:05
sdakethe ceph playboook injects the secrets in14:06
sdakelet me sse if i can find referencee14:06
Jeffrey4lkfox1111, check this https://github.com/openstack/kolla/blob/master/ansible/roles/nova/tasks/ceph.yml#L49,L6114:06
kfox1111I put the files in the config dir, and kolla_start is copying them into /etc/libvirt/secrets14:06
Jeffrey4lxml and base64 is use for libvirt to create virsh secret.14:06
sdakekfox1111 mode = 60014:07
sdakekfox1111 it is possible you have some type ofo wierd permissiosns problem14:07
kfox1111ls -l /etc/libvirt/secrets/14:07
kfox1111total 814:07
kfox1111-rw------- 1 root root  57 Sep 25 13:37 50030df6-b546-433f-baa3-85c57f747754.base6414:07
sdakewhere libvirtd can't read the secrets14:07
kfox1111-rw------- 1 root root 167 Sep 25 13:37 50030df6-b546-433f-baa3-85c57f747754.xml14:07
sdaketry rellaxing permissionss to 66614:07
sdake(not a permanent solution just to eliminat etht as a problem)14:07
kfox1111the perms were setup by kolla itself.14:07
kfox1111in the config.json14:07
sdakeyup14:08
kfox1111so I'd expect that to be correct.14:08
sdakek8s is different then bare metal14:08
Jeffrey4li do not think the libvirt need read that file.14:08
sdakeJeffrey4l virsh apprentliy is14:08
kfox1111hold on. got a todler that needs a bit of attention... be back in a few min.14:08
sdakekfox1111 sure -i'll spew while you have toddler duty14:09
sdakewhat i'm getting at iss either virsh or libvirtd can't read  that file14:09
sdakean strace of virsh would also prove this scneario14:09
sdakehow exactly areyou running virsh?14:09
sdakewhen I use cepeh, I have to run sudo deply14:09
sdakeit has to be as root for whatever reason - related to keey coying14:10
sdakerather secret copying14:10
sdakekfox1111 which uid is virsh secret-list run under?14:14
sdakeI asume it connects to libvirt14:14
sdakeiirc wwe echanged the uid of libvirt some time ago14:14
*** zhubingbing_ has joined #openstack-kolla14:14
sdakerc2 actualy looks pretty good nwo ;)14:16
kfox1111back.14:17
kfox1111I docker execed into the container.14:18
kfox1111running as root.14:18
kfox1111libvirt looks to be running as root too.14:19
kfox1111so should have access to the files.14:19
kfox1111is them being read out of /etc/libvirt/secrets an undocumented libvirt feature? cause I can't find anything about it.14:19
kfox1111I'd expect something should be running 'virsh secret-define' on the secretes.14:20
Jeffrey4lya use secret-define will be better.14:20
kfox1111does kolla_start usually do that?14:22
sdakeJeffrey4l any thoughts on https://bugs.launchpad.net/kolla/+bug/161733414:22
openstackLaunchpad bug 1617334 in kolla "reconfigure action fails on [neutron | Restart the neutron_openvswitch_agent container] " [High,Triaged]14:22
Jeffrey4lkfox1111, we may need add nova_libvirt bootstrap14:23
*** daneyon has joined #openstack-kolla14:23
kfox1111Jeffrey4l: yeah. could be. Just trying to figure out how its workign at all in kolla-ansible. its not making sense to me.14:23
sdakekfox1111 look in /etc/libvirt14:24
sdakekfox1111 there are some config files there14:24
sdakekfox1111 one of them sets permissionss of the libvirt runtime14:24
sdakekfox1111 can you tell me what it says?14:24
kfox1111libvirtd.conf?14:24
sdakepoissbly14:25
sdakei think there are two files14:25
sdakelet me disconnect from vpn - brb - so i have accss to  my gear14:25
kfox1111k. thx.14:25
kfox1111the whole libvirtd config is http://pastebin.com/mv5ty78L14:26
kfox1111not much there.14:26
*** daneyon has quit IRC14:27
*** sdake_ has joined #openstack-kolla14:28
Jeffrey4lsdake, haven't seen that issue too. i asked reporter to push the inventory file out14:29
*** sdake has quit IRC14:30
sdake_[sdake@minime-03 ~]$ docker exec -tu root 4fd bash14:31
sdake_(nova-libvirt)[root@minime-03 /]# cd /etc/libvirt14:31
sdake_locks up14:31
sdake_groan14:31
kfox1111:/14:32
kfox1111swapping?14:32
sdake_256gbb ram unlikely14:32
kfox1111k.14:32
sdake_duh: ^C[sdake@minime-03 ~]$ docker exec -tiu root 4fd bash14:32
Jeffrey4lsdake_, could u check this bug https://bugs.launchpad.net/bugs/1616268  it is critical and affect all branch of kolla.14:32
openstackLaunchpad bug 1616268 in kolla mitaka "Stale namespace removal causing "RTNETLINK answers: Invalid argument" errors" [Critical,Confirmed]14:32
Jeffrey4lit have took lots of time, but without success.14:33
*** rmart04 has joined #openstack-kolla14:37
sdake_Jeffrey4l roger - just read through bug log14:42
sdake_Jeffrey4l i'm missing the use cae of kolla where this is broken14:42
sdake_matthew had implied it was reconfigure?14:43
*** neilus has quit IRC14:43
kfox1111I wonder if /etc/libvirt/secrets is an internal implementation detail.14:44
Jeffrey4li saw this issue and can re-produce it easily now.14:44
Jeffrey4lkfox1111, i think it is.14:44
kfox1111we may want to file  bug against kolla too then.14:44
sdake_Jeffrey4l if we throw all of kolla at docker and say "its broken" they ar egoing to ask why14:46
sdake_we need a simple reproducer that answers that question14:46
Jeffrey4lsdake_, yes. but we failed to re-produce that without neutron now.14:46
sdake_as sam says neutron is a wrapper around ip netns14:47
Jeffrey4lyes. But we really tried but failed.14:48
sdake_very troublesome that docker has such problems still14:48
sdake_can you tell me the impact on kola pls14:48
Jeffrey4lone issue is neutron route namespace. it will move from one node to another node.14:49
sdake_moves why and how is that a problem14:49
Jeffrey4lwhen it move from A -> B -> A, then issue occur. A have that namespace with readonly14:49
sdake_got it14:49
sdake_what triggerss the move?14:50
Jeffrey4ll3 ha possible do like that.14:50
sdake_so l3 ha triggers this problem - naything else?14:50
Jeffrey4li think there are some api can do this too ( But i do not know which one is)14:50
sdake_see if its critical need to understand the impact14:51
sdake_i.e. hwen it happens14:51
Jeffrey4lwhen run one reconfigure or neutron_* container is restarted.14:52
sdake_if we know when, we can possibly workorund it14:52
Jeffrey4li do not think it has any workaround. we need fix this in docker/linux kernal.14:52
sdake_Jeffrey4l http://i.imgur.com/iWKad22.jpg14:53
Jeffrey4lor event in neutron.14:53
Jeffrey4leven14:53
Jeffrey4lhmm. i really think it is a bug in kernel :(  ( but i haven't any proof )14:54
sdake_double http://i.imgur.com/iWKad22.jpg14:54
Jeffrey4llol14:55
sdake_if we are too bleeding edge for the kernels, then we need to back off the edge14:55
sdake_so this kernel bug theory14:55
sdake_have you tried 16.04.01 ubuntu kernel?14:56
Jeffrey4li use centos. And tested with latest kernel ( 4.4? ), still failed.14:56
sdake_you built your own kernel then?14:56
Jeffrey4lno.. install from elrepo14:56
sdake_cool didnt' know they had a modern kernel avaailable14:57
Jeffrey4lhttp://i.imgur.com/iWKad22.jpg14:57
Jeffrey4lsorry, that's a typo  ^^ i do not mean that.14:57
sdake_ok14:57
sdake_i meant mine ;)14:58
Jeffrey4lhttp://elrepo.org/tiki/kernel-ml14:58
Jeffrey4lthat's it.14:58
kfox1111ah. finally found the log file.14:58
kfox1111I'm really not use to looking for things in /var/log/kolla14:58
sdake_Jeffrey4l looks good14:58
kfox11112016-09-25 14:56:57.452+0000: 19043: error : secretLoadValue:400 : internal error: invalid base64 in '/etc/libvirt/secrets/50030df6-b546-433f-baa3-85c57f747754.base64'14:59
kfox11112016-09-25 14:56:57.452+0000: 19043: error : loadSecrets:491 : Error reading secret: internal error: invalid base64 in '/etc/libvirt/secrets/50030df6-b546-433f-baa3-85c57f747754.base64'14:59
kfox1111so, thats the problem.14:59
Jeffrey4lkfox1111, from k82?14:59
Jeffrey4lk8s?14:59
kfox1111from libvirtd.14:59
sdake_Jeffrey4l so to summarize the issue - rconfigure works once second time neutron implodes15:00
kfox1111I may not have right number of base64 encodings. :/15:00
Jeffrey4lok.15:00
sdake_Jeffrey4l is that accurate?15:01
Jeffrey4lsdake_, we can re-produce without reconfigure.15:01
Jeffrey4l1. deploy kolla15:01
sdake_explain that15:01
Jeffrey4l2. create one route ( which create one namespace )15:01
Jeffrey4l3. restart neutron_* container15:01
Jeffrey4l4. delete route15:01
Jeffrey4l5. issue occur. the namespace become readonly.15:02
sdake_on step 3 or step 4?15:02
Jeffrey4l415:02
Jeffrey4lin 3, every thing seem ok.15:03
Jeffrey4luntil you remove the route ( namespace )15:03
sdake_this is with dumb-init?15:03
Jeffrey4lsdake_, no. it exist long time ago.15:04
sdake_i mean does dumb_init fixt his?15:04
Jeffrey4ldumb-init doesn't fix this.15:04
sdake_ok15:04
sdake_you outlined steps 1-4 above15:05
sdake_tht seems simple enough15:05
sdake_prticualry 2,3,415:05
sdake_to simulate using some other mechanism then kolla15:05
Jeffrey4li tried, but failed :(15:05
sdake_how much time did you sink into it?15:06
Jeffrey4lsink into means saw this? or debug this?15:07
sdake_how much time did you spend tryign to make a reproducer ethat doesn't include openstack?15:07
Jeffrey4land the answer is both `very much`15:07
sdake_what about a neutron fat container15:09
*** haplo37_ has quit IRC15:10
sdake_if it happens in a neutron fat container - it is not munt propoogation15:10
Jeffrey4lsdake_, that's doesn't matter. when a create a route, i didn't create any network or subnet.15:10
sdake_and by corralary its neutron15:11
sdake_if we can narrow it down to mount propogaiton (ekrnel or docker) thats good15:12
Jeffrey4lbut it won't happen in neutron deploy without container.15:12
sdake_if we can narrow it down to neutron thats even better15:12
Jeffrey4lyes.15:12
*** haplo37_ has joined #openstack-kolla15:13
sdake_deploy neutron agents in fat container will narrow it down15:13
kfox1111hmm. so the base64 has a double base64 encoded key in it...15:13
Jeffrey4lok " fat container"15:13
sdake_the fact that docker restart plays into it is a bit trobuling15:14
*** neilus has joined #openstack-kolla15:14
Jeffrey4lmaybe.15:14
sdake_so in essence15:16
sdake_if we can't reproduce it outside of openstack, lets minimize the search space for the problem15:17
Jeffrey4lok.15:17
sdake_kfox1111 don't know why :)15:17
*** neilus has quit IRC15:19
*** tonanhngo has joined #openstack-kolla15:21
*** tonanhngo has quit IRC15:23
*** rmart04 has quit IRC15:24
sdake_Jeffrey4l what about comment #1615:25
sdake_https://bugs.launchpad.net/kolla/+bug/161626815:25
openstackLaunchpad bug 1616268 in kolla mitaka "Stale namespace removal causing "RTNETLINK answers: Invalid argument" errors" [Critical,Confirmed]15:25
Jeffrey4lthe namespace file become a regular file.15:26
Jeffrey4lsdake_,15:26
sdake_right but thats a rerpoducer outside of kolla?15:26
Jeffrey4lsdake_, no. we have no idea why it become a regular file.15:27
Jeffrey4lit still reproduced under kolla15:27
sdake_but does that example work outside of kolla?15:27
Jeffrey4lno.15:28
Jeffrey4lthe direct reason is: namespace file becomes a regular file. But we still have no idea why. and can not reproduce this without kolla now.15:29
Jeffrey4lshould go to bed  . bye guys :)15:30
Jeffrey4lgood night.15:31
*** caowei has quit IRC15:31
sdake_night15:36
*** sdake_ has quit IRC15:36
kfox1111hmm...15:39
kfox1111seems like it may be objecting to a newline...15:39
*** tonanhngo has joined #openstack-kolla15:41
kfox1111yup. was the newline. :/15:43
*** kbaegis has joined #openstack-kolla15:49
kbaegisHey guys, I'm having an issue with the override config15:49
kbaegisfor AIO, is /etc/kolla/config/local/ml2_conf.ini the correct path for a ml2 config?15:50
kbaegisor should it just be /etc/kolla/config/ml2_conf.ini?15:50
*** srwilkers has joined #openstack-kolla16:03
*** daneyon has joined #openstack-kolla16:11
openstackgerritSerguei Bezverkhi proposed openstack/kolla-kubernetes: Adds sidecar logging container to common-deployment pod  https://review.openstack.org/37568216:12
sbezverkkfox1111: are you going to push a fix for loading jinja2 templates?16:13
*** rmart04 has joined #openstack-kolla16:15
*** daneyon has quit IRC16:15
*** z- has quit IRC16:16
*** z- has joined #openstack-kolla16:16
sbezverkkfox1111: I was attepting to test your more deployments PS with fluentd macro, but kolla-kube is not happy..16:17
*** kbaegis has quit IRC16:21
*** eaguilar has joined #openstack-kolla16:27
*** schwicht has quit IRC16:30
*** schwicht_at_work has joined #openstack-kolla16:30
*** rmart04 has quit IRC16:37
*** senk_ has joined #openstack-kolla16:40
*** rmart04 has joined #openstack-kolla16:41
*** kbaegis has joined #openstack-kolla16:43
*** zhubingbing_ has quit IRC16:43
*** rmart04 has quit IRC16:44
kbaegisCan anyone help me with the extended config?16:44
*** rmart04 has joined #openstack-kolla16:45
*** eaguilar has quit IRC16:46
*** rmart04 has quit IRC16:47
kbaegisnvm.  It's actually covered pretty well :016:48
*** neilus has joined #openstack-kolla16:55
*** schwicht_at_work has quit IRC16:57
*** neilus has quit IRC17:00
*** schwicht has joined #openstack-kolla17:02
*** jrich523 has quit IRC17:02
*** jrich523 has joined #openstack-kolla17:04
*** z- has quit IRC17:04
*** z- has joined #openstack-kolla17:04
kfox1111sbezverk: what's it doing?17:06
kfox1111I was able to test the ceph patch set with the more deployments ps and all the deployments worked.17:06
kbaegis72 hours, heat : Creating the heat_stack_user role is still broken on master17:12
*** senk_ has quit IRC17:12
*** neilus has joined #openstack-kolla17:13
*** neilus has quit IRC17:17
*** kbyrne has quit IRC17:26
kbaegishttp://pastebin.com/P1fcJbVP17:27
sbezverkkfox1111: the gate still fails to find template, I thought you were going to push a fix for the gate so it would load all temapltes from ./common/macro*17:28
*** kbyrne has joined #openstack-kolla17:29
kbaegisI think merge 040b07f bcff0f8 from jenkins has a regression17:30
kfox1111sbezverk: I think the fix needs to either go into your patch, or we need a new patch for the feature. its not a gate spacific issue I think, but the engine needs to have that fix to let it support imports properly.17:31
kfox1111Do you want to write it, or should I?17:31
sbezverkkfox1111: Ok I will take a stab at it..17:33
kbaegisVerified. ansible/roles/heat/tasks/register.yml is messed up due to the scripting in commit 9d7ba661f3a1f5de4cff06d531025319f734271217:33
kbaegisShould break deploy on master for everyone17:33
kfox1111sbezverk: I've got glance out of ceph working, and I think I'm realy really close to having nova working.17:34
kfox1111cinder should be pretty easy after that.17:35
*** kbyrne has quit IRC17:39
sbezverkkfox1111: not a big fan of ceph, but I am happy for you ;-)17:42
kfox1111I love ceph. :) been able to avoid several major outages due to it.17:49
kfox1111including physically moving the ceph one node at a time from one rack to another without downtime,17:49
kfox1111and a second time moving all the ost's from one ack to another via moving the software from one rack to another and syncing the osd's one at a time.17:49
kfox1111havent' seen a file system as bullet proof as ceph yet.17:49
kfox1111at this point, I think its best to use ceph for everything you can except things where performance really is critical. storage is pretty cheep so if you can get 90%+ of your data on resilient storage like ceph, life is sooo much better. :)17:53
*** rmart04 has joined #openstack-kolla17:53
*** srwilkers has quit IRC17:55
*** rmart04 has quit IRC17:56
*** rmart04 has joined #openstack-kolla17:57
*** rmart04 has quit IRC17:58
*** senk_ has joined #openstack-kolla17:58
*** rmart04 has joined #openstack-kolla18:02
kfox1111there we go... :)18:05
*** rmart04 has quit IRC18:12
*** srwilkers has joined #openstack-kolla18:14
*** tonanhngo has quit IRC18:17
*** senk_ has quit IRC18:18
*** senk_ has joined #openstack-kolla18:19
*** g3ek has quit IRC18:20
*** haplo37 has quit IRC18:20
*** haplo37 has joined #openstack-kolla18:29
*** g3ek has joined #openstack-kolla18:29
*** senk_ has quit IRC18:32
*** sdake has joined #openstack-kolla18:33
*** sdake_ has joined #openstack-kolla18:36
*** sdake_ has quit IRC18:36
*** sdake has quit IRC18:37
*** pbourke__ has quit IRC18:38
*** pbourke__ has joined #openstack-kolla18:39
*** gmmaha has quit IRC18:46
*** gmmaha has joined #openstack-kolla18:48
*** sdake has joined #openstack-kolla18:49
*** srwilkers has quit IRC18:52
*** sdake has quit IRC18:53
*** daneyon has joined #openstack-kolla18:54
*** kbyrne has joined #openstack-kolla18:54
*** haplo37_ has quit IRC18:57
*** daneyon has quit IRC18:58
*** jistr has quit IRC18:59
*** haplo37_ has joined #openstack-kolla18:59
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: Basic ceph deployment support  https://review.openstack.org/37325618:59
*** unicell has joined #openstack-kolla19:10
*** tonanhngo has joined #openstack-kolla19:33
*** tonanhngo has quit IRC19:36
*** unicell1 has joined #openstack-kolla19:38
*** unicell has quit IRC19:38
*** jamesdenton has joined #openstack-kolla19:38
*** srwilkers has joined #openstack-kolla19:53
*** srwilkers has quit IRC19:53
*** haplo37_ has quit IRC20:19
*** haplo37_ has joined #openstack-kolla20:19
*** srwilkers has joined #openstack-kolla20:25
*** jmccarthy has quit IRC20:28
*** jmccarthy has joined #openstack-kolla20:29
*** neilus has joined #openstack-kolla20:31
*** Jeffrey4l has quit IRC20:34
*** Jeffrey4l has joined #openstack-kolla20:35
*** neilus has quit IRC20:36
*** srwilkers has quit IRC20:37
*** neilus has joined #openstack-kolla20:55
*** neilus_ has joined #openstack-kolla21:03
*** neilus has quit IRC21:04
*** kbaegis has left #openstack-kolla21:14
*** brad[]` has joined #openstack-kolla21:35
*** tonanhngo has joined #openstack-kolla21:38
*** tonanhngo has quit IRC21:39
*** sdake has joined #openstack-kolla21:56
*** sdake has quit IRC22:03
*** sdake has joined #openstack-kolla22:04
*** neilus_ has quit IRC22:13
openstackgerritWaldemar Znoinski proposed openstack/kolla: fix ironic-inspector setup  https://review.openstack.org/37472722:15
openstackgerritWaldemar Znoinski proposed openstack/kolla: install iptables in ironic-inspector container  https://review.openstack.org/37497022:15
openstackgerritWaldemar Znoinski proposed openstack/kolla: install iptables in ironic-inspector container  https://review.openstack.org/37497022:16
*** neilus has joined #openstack-kolla22:17
*** neilus has quit IRC22:22
*** daneyon has joined #openstack-kolla22:30
*** daneyon has quit IRC22:35
*** haplo37_ has quit IRC22:56
*** haplo37_ has joined #openstack-kolla22:58
*** eaguilar has joined #openstack-kolla23:04
*** jascott1_ has joined #openstack-kolla23:27
*** jascott1_ has quit IRC23:34
*** lujinluo has joined #openstack-kolla23:44
kfox1111there we go... cinder+ceph working. :)23:57
*** yingjun has joined #openstack-kolla23:59
« Saturday, 2016-09-24 Index Monday, 2016-09-26 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!