*** daneyon has quit IRC | 00:01 | |
*** neilus has joined #openstack-kolla | 00:10 | |
*** jascott1 has quit IRC | 00:15 | |
*** jascott1- is now known as jascott1 | 00:15 | |
*** neilus has quit IRC | 00:15 | |
*** eaguilar has joined #openstack-kolla | 00:26 | |
*** sdake has joined #openstack-kolla | 00:44 | |
*** jascott1_ has joined #openstack-kolla | 00:49 | |
*** rbergero1 is now known as rbergeron | 00:49 | |
*** rbergeron has quit IRC | 00:50 | |
*** rbergeron has joined #openstack-kolla | 00:50 | |
*** sdake has quit IRC | 00:53 | |
*** eaguilar has quit IRC | 00:53 | |
*** eaguilar has joined #openstack-kolla | 00:54 | |
*** bjolo__ has quit IRC | 01:08 | |
*** rhallisey has quit IRC | 01:12 | |
*** jascott1_ has quit IRC | 01:15 | |
*** caowei has joined #openstack-kolla | 01:20 | |
*** jascott1_ has joined #openstack-kolla | 01:24 | |
*** jascott1_ has quit IRC | 01:28 | |
*** jascott1_ has joined #openstack-kolla | 01:33 | |
*** jrist has quit IRC | 01:34 | |
*** zhubingbing has joined #openstack-kolla | 01:35 | |
*** sdake has joined #openstack-kolla | 01:49 | |
*** tonanhngo has joined #openstack-kolla | 01:55 | |
*** tonanhngo has quit IRC | 01:56 | |
*** neilus has joined #openstack-kolla | 01:58 | |
*** neilus has quit IRC | 02:03 | |
*** Jeffrey4l has quit IRC | 02:12 | |
*** jascott1_ has quit IRC | 02:24 | |
*** sdake has quit IRC | 02:27 | |
*** daneyon has joined #openstack-kolla | 02:39 | |
*** tonanhngo has joined #openstack-kolla | 02:43 | |
*** daneyon has quit IRC | 02:43 | |
*** tonanhngo has quit IRC | 02:45 | |
*** openstackgerrit has joined #openstack-kolla | 02:45 | |
openstackgerrit | zhubingbing proposed openstack/kolla: remove bootstrap.yml in grafana role upgrade.yml https://review.openstack.org/375913 | 02:46 |
---|---|---|
*** zhubingbing has quit IRC | 02:55 | |
*** eaguilar has quit IRC | 03:00 | |
*** david-lyle_ has quit IRC | 03:03 | |
*** zhubingbing has joined #openstack-kolla | 03:08 | |
*** tonanhngo has joined #openstack-kolla | 03:14 | |
*** tonanhngo has quit IRC | 03:16 | |
openstackgerrit | Wei Cao proposed openstack/kolla: Add freezer ansible role https://review.openstack.org/375874 | 03:17 |
*** unicell has joined #openstack-kolla | 03:21 | |
*** jascott1_ has joined #openstack-kolla | 03:25 | |
*** jascott1_ has quit IRC | 03:30 | |
*** tonanhngo has joined #openstack-kolla | 03:33 | |
openstackgerrit | Wei Cao proposed openstack/kolla: Revert "Add enable_elasticsearch option" https://review.openstack.org/375198 | 03:33 |
openstackgerrit | Wei Cao proposed openstack/kolla: Revert "Add enable_elasticsearch option" https://review.openstack.org/375198 | 03:33 |
*** tonanhngo has quit IRC | 03:35 | |
*** jamesdenton has joined #openstack-kolla | 03:36 | |
*** neilus has joined #openstack-kolla | 03:47 | |
*** neilus has quit IRC | 03:52 | |
openstackgerrit | Wei Cao proposed openstack/kolla: Change "Virtualhost" to "VirtualHost" in horizon.conf.j2 https://review.openstack.org/375929 | 04:05 |
*** kbyrne has quit IRC | 04:09 | |
*** kbyrne has joined #openstack-kolla | 04:13 | |
*** g3ek has quit IRC | 04:49 | |
*** g3ek has joined #openstack-kolla | 04:50 | |
*** Jeffrey4l has joined #openstack-kolla | 04:55 | |
*** tonanhngo has joined #openstack-kolla | 04:57 | |
*** jascott1_ has joined #openstack-kolla | 05:00 | |
*** daneyon has joined #openstack-kolla | 05:21 | |
*** daneyon has quit IRC | 05:26 | |
*** neilus has joined #openstack-kolla | 05:35 | |
*** neilus has quit IRC | 05:39 | |
*** zhubingbing has quit IRC | 06:01 | |
*** senk_ has quit IRC | 06:29 | |
*** unicell has quit IRC | 06:43 | |
*** zhubingbing has joined #openstack-kolla | 07:07 | |
*** haplo37 has quit IRC | 07:08 | |
*** daneyon has joined #openstack-kolla | 07:10 | |
*** haplo37 has joined #openstack-kolla | 07:11 | |
*** daneyon has quit IRC | 07:14 | |
*** tonanhngo has quit IRC | 07:20 | |
*** neilus has joined #openstack-kolla | 07:23 | |
*** neilus has quit IRC | 07:28 | |
*** tonanhngo has joined #openstack-kolla | 07:43 | |
*** tonanhngo has quit IRC | 07:45 | |
*** senk_ has joined #openstack-kolla | 07:46 | |
openstackgerrit | Merged openstack/kolla: Add aodh-api to haproxy.cfg file https://review.openstack.org/375197 | 08:05 |
openstackgerrit | Merged openstack/kolla: Delete tempest.yml playbook https://review.openstack.org/375879 | 08:07 |
*** g3ek has quit IRC | 08:14 | |
*** 14WAAFND0 has quit IRC | 08:16 | |
*** g3ek has joined #openstack-kolla | 08:17 | |
*** haplo37_ has joined #openstack-kolla | 08:18 | |
*** jascott1_ has quit IRC | 08:22 | |
*** egonzalez90 has joined #openstack-kolla | 08:24 | |
*** jascott1_ has joined #openstack-kolla | 08:28 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Fix mistral service https://review.openstack.org/374921 | 08:29 |
*** egonzalez90 has quit IRC | 08:31 | |
*** rmart04 has joined #openstack-kolla | 08:38 | |
*** senk_ has quit IRC | 08:58 | |
*** senk__ has joined #openstack-kolla | 08:58 | |
*** rmart04 has quit IRC | 08:59 | |
*** dwalsh has joined #openstack-kolla | 09:11 | |
*** neilus has joined #openstack-kolla | 09:11 | |
*** neilus has quit IRC | 09:16 | |
*** rmart04 has joined #openstack-kolla | 09:36 | |
*** senk__ has quit IRC | 09:46 | |
*** senk_ has joined #openstack-kolla | 09:46 | |
*** rmart04 has quit IRC | 09:49 | |
*** jascott1_ has quit IRC | 09:56 | |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 09:57 |
*** rmart04 has joined #openstack-kolla | 10:05 | |
*** rmart04 has quit IRC | 10:05 | |
*** rmart04 has joined #openstack-kolla | 10:06 | |
*** neilus has joined #openstack-kolla | 10:07 | |
*** neilus has quit IRC | 10:12 | |
*** rmart04 has quit IRC | 10:24 | |
*** senk_ has quit IRC | 10:26 | |
*** senk__ has joined #openstack-kolla | 10:26 | |
*** rmart04 has joined #openstack-kolla | 10:30 | |
*** neilus has joined #openstack-kolla | 10:30 | |
*** neilus has quit IRC | 10:39 | |
*** tonanhngo has joined #openstack-kolla | 10:40 | |
*** rmart04 has quit IRC | 10:40 | |
*** haplo37 has quit IRC | 10:41 | |
*** haplo37_ is now known as haplo37 | 10:41 | |
*** tonanhngo has quit IRC | 10:41 | |
*** haplo37_ has joined #openstack-kolla | 10:43 | |
*** daneyon has joined #openstack-kolla | 10:46 | |
*** daneyon has quit IRC | 10:51 | |
*** zhubingbing has quit IRC | 11:08 | |
*** zhubingbing_ has joined #openstack-kolla | 11:11 | |
*** g3ek has quit IRC | 11:13 | |
*** haplo37 has quit IRC | 11:14 | |
*** zhurong has joined #openstack-kolla | 11:15 | |
*** haplo37 has joined #openstack-kolla | 11:16 | |
*** g3ek has joined #openstack-kolla | 11:19 | |
*** dwalsh has quit IRC | 11:23 | |
openstackgerrit | caoyuan proposed openstack/kolla: Update the action when create Glance service endpoints https://review.openstack.org/376006 | 11:49 |
*** neilus has joined #openstack-kolla | 11:58 | |
*** neilus has quit IRC | 12:03 | |
*** zhubingbing_ has quit IRC | 12:11 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Optimized reconfigure action for glance https://review.openstack.org/376011 | 12:16 |
*** sdake has joined #openstack-kolla | 12:19 | |
*** g3ek has quit IRC | 12:22 | |
*** g3ek has joined #openstack-kolla | 12:29 | |
*** senk__ has quit IRC | 12:33 | |
*** daneyon has joined #openstack-kolla | 12:34 | |
*** daneyon has quit IRC | 12:39 | |
*** neilus has joined #openstack-kolla | 12:47 | |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: basic ceph deployment support https://review.openstack.org/373256 | 12:50 |
*** kong has quit IRC | 12:51 | |
*** jamesdenton has quit IRC | 12:51 | |
*** kong has joined #openstack-kolla | 12:52 | |
*** david-lyle has joined #openstack-kolla | 12:56 | |
*** schwicht has joined #openstack-kolla | 13:01 | |
sdake | Jeffrey4l about? | 13:04 |
Jeffrey4l | sdake, yep | 13:04 |
*** haplo37_ has quit IRC | 13:18 | |
*** haplo37_ has joined #openstack-kolla | 13:19 | |
sdake | Jeffrey4l sorry got distracted, is mitaka gate working now? | 13:20 |
Jeffrey4l | sdake, no. this PS is not merged still https://review.openstack.org/375418 | 13:24 |
*** zhurong has quit IRC | 13:24 | |
sdake | any core reviewers around at this godawful hour? :) | 13:25 |
*** dotplus_ is now known as dotplus | 13:32 | |
sdake | Jeffrey4l i believe liberty becomes eol at conclusion of this cycle | 13:43 |
sdake | Jeffrey4l if we have time we ought to releasse one last zstream | 13:43 |
Jeffrey4l | sdake, i agreed. and liberty gate always red :( | 13:43 |
sdake | i think that is caused by a combination of problems | 13:44 |
sdake | one of hiwch is we are using an old version of rabbitmq | 13:44 |
Jeffrey4l | yes. | 13:44 |
sdake | a new version of rabbitmq prevents the rabbitmq crash | 13:44 |
Jeffrey4l | i think so. | 13:44 |
sdake | other issues you are aware of? | 13:45 |
sdake | to wither in denail | 13:46 |
sdake | the bitterness of one who is left alone | 13:46 |
sdake | 845 tubes sounding fantastic | 13:49 |
sdake | german engineering ftw | 13:49 |
Jeffrey4l | liberty branch? no. I tried to fix it, but block on rabbitmq issue. | 13:52 |
sdake | even with rabbitmq backport? | 13:53 |
Jeffrey4l | yep. | 13:53 |
sdake | same issue? | 13:53 |
Jeffrey4l | long time ago. ( with my one of my rabbitmq fix backport ) | 13:53 |
sdake | oh - i am talking about the rework on rabbitmq versions we did in master | 13:53 |
sdake | you now the one i started and 3 or 4 peple finished ;) | 13:54 |
Jeffrey4l | haven't check that . | 13:54 |
sdake | let me give it a go | 13:54 |
Jeffrey4l | sdake, did u push a backport? | 13:54 |
sdake | nope | 13:54 |
sdake | i'm a bit wary of changing rabbitmq version in mitaka | 13:55 |
Jeffrey4l | OK. let's try. | 13:55 |
sdake | i keep hearing problems with rc1 | 13:55 |
sdake | and rabbitmq container restarting | 13:55 |
sdake | mlima suffering from this | 13:56 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST LIBERTY BRANCH https://review.openstack.org/323133 | 13:58 |
Jeffrey4l | sdake, pushed a test PS to check what' going wrong now. sdake ^ | 13:58 |
sdake | yet master works fantastically well for me | 13:58 |
sdake | multinode and single nodee | 13:58 |
Jeffrey4l | yep. I am making tests for master in recent days. | 13:59 |
sdake | found any problems bessdiesss the 100 or so bugs in our backlog? :) | 13:59 |
Jeffrey4l | no big issue. | 14:00 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the failure of ceilometer-api container https://review.openstack.org/372161 | 14:01 |
Jeffrey4l | sdake, and if you have time, please review this https://review.openstack.org/#/c/372161/ | 14:01 |
sdake | Jeffrey4l looking now | 14:02 |
sdake | Jeffrey4l any chance you can help me sort out which bugs are important vs not for 3.0.0 | 14:02 |
sdake | so we can make a final call on which bugs to eject to ocata | 14:03 |
Jeffrey4l | base on the bug list? | 14:03 |
kfox1111 | anyone that konws libvirt secrets awake? | 14:03 |
sdake | huh kfox? | 14:03 |
sdake | Jeffrey4l ya, the rc2 buglist | 14:03 |
sdake | Jeffrey4l i asked cr tteam to triage them | 14:03 |
sdake | not sure how much that hapepned or didn't | 14:03 |
sdake | but lets make a final call today | 14:03 |
kfox1111 | I've got stuff in /etc/libvirt/secrets/<uuid>.base64 and .xml | 14:03 |
kfox1111 | but nothing shows in virsh secret-list | 14:04 |
Jeffrey4l | i will check the list.. | 14:04 |
sdake | kfox1111 hmm no idea | 14:04 |
sdake | kfox1111 what is a libvirt seecret - lets start there | 14:04 |
kfox1111 | I've always had to do a virsh secret-create to import them, | 14:04 |
kfox1111 | but I don't see any code in kolla to do that. | 14:04 |
kfox1111 | sdake: they are primarily used to load ceph keys into libvirt. | 14:04 |
sdake | ok | 14:05 |
sdake | this is in k8s environment | 14:05 |
sdake | ? | 14:05 |
kfox1111 | yeah. | 14:05 |
kfox1111 | I've read through the ansible stuff, and tryhing it in k8s. | 14:05 |
sdake | the ceph playboook injects the secrets in | 14:06 |
sdake | let me sse if i can find referencee | 14:06 |
Jeffrey4l | kfox1111, check this https://github.com/openstack/kolla/blob/master/ansible/roles/nova/tasks/ceph.yml#L49,L61 | 14:06 |
kfox1111 | I put the files in the config dir, and kolla_start is copying them into /etc/libvirt/secrets | 14:06 |
Jeffrey4l | xml and base64 is use for libvirt to create virsh secret. | 14:06 |
sdake | kfox1111 mode = 600 | 14:07 |
sdake | kfox1111 it is possible you have some type ofo wierd permissiosns problem | 14:07 |
kfox1111 | ls -l /etc/libvirt/secrets/ | 14:07 |
kfox1111 | total 8 | 14:07 |
kfox1111 | -rw------- 1 root root 57 Sep 25 13:37 50030df6-b546-433f-baa3-85c57f747754.base64 | 14:07 |
sdake | where libvirtd can't read the secrets | 14:07 |
kfox1111 | -rw------- 1 root root 167 Sep 25 13:37 50030df6-b546-433f-baa3-85c57f747754.xml | 14:07 |
sdake | try rellaxing permissionss to 666 | 14:07 |
sdake | (not a permanent solution just to eliminat etht as a problem) | 14:07 |
kfox1111 | the perms were setup by kolla itself. | 14:07 |
kfox1111 | in the config.json | 14:07 |
sdake | yup | 14:08 |
kfox1111 | so I'd expect that to be correct. | 14:08 |
sdake | k8s is different then bare metal | 14:08 |
Jeffrey4l | i do not think the libvirt need read that file. | 14:08 |
sdake | Jeffrey4l virsh apprentliy is | 14:08 |
kfox1111 | hold on. got a todler that needs a bit of attention... be back in a few min. | 14:08 |
sdake | kfox1111 sure -i'll spew while you have toddler duty | 14:09 |
sdake | what i'm getting at iss either virsh or libvirtd can't read that file | 14:09 |
sdake | an strace of virsh would also prove this scneario | 14:09 |
sdake | how exactly areyou running virsh? | 14:09 |
sdake | when I use cepeh, I have to run sudo deply | 14:09 |
sdake | it has to be as root for whatever reason - related to keey coying | 14:10 |
sdake | rather secret copying | 14:10 |
sdake | kfox1111 which uid is virsh secret-list run under? | 14:14 |
sdake | I asume it connects to libvirt | 14:14 |
sdake | iirc wwe echanged the uid of libvirt some time ago | 14:14 |
*** zhubingbing_ has joined #openstack-kolla | 14:14 | |
sdake | rc2 actualy looks pretty good nwo ;) | 14:16 |
kfox1111 | back. | 14:17 |
kfox1111 | I docker execed into the container. | 14:18 |
kfox1111 | running as root. | 14:18 |
kfox1111 | libvirt looks to be running as root too. | 14:19 |
kfox1111 | so should have access to the files. | 14:19 |
kfox1111 | is them being read out of /etc/libvirt/secrets an undocumented libvirt feature? cause I can't find anything about it. | 14:19 |
kfox1111 | I'd expect something should be running 'virsh secret-define' on the secretes. | 14:20 |
Jeffrey4l | ya use secret-define will be better. | 14:20 |
kfox1111 | does kolla_start usually do that? | 14:22 |
sdake | Jeffrey4l any thoughts on https://bugs.launchpad.net/kolla/+bug/1617334 | 14:22 |
openstack | Launchpad bug 1617334 in kolla "reconfigure action fails on [neutron | Restart the neutron_openvswitch_agent container] " [High,Triaged] | 14:22 |
Jeffrey4l | kfox1111, we may need add nova_libvirt bootstrap | 14:23 |
*** daneyon has joined #openstack-kolla | 14:23 | |
kfox1111 | Jeffrey4l: yeah. could be. Just trying to figure out how its workign at all in kolla-ansible. its not making sense to me. | 14:23 |
sdake | kfox1111 look in /etc/libvirt | 14:24 |
sdake | kfox1111 there are some config files there | 14:24 |
sdake | kfox1111 one of them sets permissionss of the libvirt runtime | 14:24 |
sdake | kfox1111 can you tell me what it says? | 14:24 |
kfox1111 | libvirtd.conf? | 14:24 |
sdake | poissbly | 14:25 |
sdake | i think there are two files | 14:25 |
sdake | let me disconnect from vpn - brb - so i have accss to my gear | 14:25 |
kfox1111 | k. thx. | 14:25 |
kfox1111 | the whole libvirtd config is http://pastebin.com/mv5ty78L | 14:26 |
kfox1111 | not much there. | 14:26 |
*** daneyon has quit IRC | 14:27 | |
*** sdake_ has joined #openstack-kolla | 14:28 | |
Jeffrey4l | sdake, haven't seen that issue too. i asked reporter to push the inventory file out | 14:29 |
*** sdake has quit IRC | 14:30 | |
sdake_ | [sdake@minime-03 ~]$ docker exec -tu root 4fd bash | 14:31 |
sdake_ | (nova-libvirt)[root@minime-03 /]# cd /etc/libvirt | 14:31 |
sdake_ | locks up | 14:31 |
sdake_ | groan | 14:31 |
kfox1111 | :/ | 14:32 |
kfox1111 | swapping? | 14:32 |
sdake_ | 256gbb ram unlikely | 14:32 |
kfox1111 | k. | 14:32 |
sdake_ | duh: ^C[sdake@minime-03 ~]$ docker exec -tiu root 4fd bash | 14:32 |
Jeffrey4l | sdake_, could u check this bug https://bugs.launchpad.net/bugs/1616268 it is critical and affect all branch of kolla. | 14:32 |
openstack | Launchpad bug 1616268 in kolla mitaka "Stale namespace removal causing "RTNETLINK answers: Invalid argument" errors" [Critical,Confirmed] | 14:32 |
Jeffrey4l | it have took lots of time, but without success. | 14:33 |
*** rmart04 has joined #openstack-kolla | 14:37 | |
sdake_ | Jeffrey4l roger - just read through bug log | 14:42 |
sdake_ | Jeffrey4l i'm missing the use cae of kolla where this is broken | 14:42 |
sdake_ | matthew had implied it was reconfigure? | 14:43 |
*** neilus has quit IRC | 14:43 | |
kfox1111 | I wonder if /etc/libvirt/secrets is an internal implementation detail. | 14:44 |
Jeffrey4l | i saw this issue and can re-produce it easily now. | 14:44 |
Jeffrey4l | kfox1111, i think it is. | 14:44 |
kfox1111 | we may want to file bug against kolla too then. | 14:44 |
sdake_ | Jeffrey4l if we throw all of kolla at docker and say "its broken" they ar egoing to ask why | 14:46 |
sdake_ | we need a simple reproducer that answers that question | 14:46 |
Jeffrey4l | sdake_, yes. but we failed to re-produce that without neutron now. | 14:46 |
sdake_ | as sam says neutron is a wrapper around ip netns | 14:47 |
Jeffrey4l | yes. But we really tried but failed. | 14:48 |
sdake_ | very troublesome that docker has such problems still | 14:48 |
sdake_ | can you tell me the impact on kola pls | 14:48 |
Jeffrey4l | one issue is neutron route namespace. it will move from one node to another node. | 14:49 |
sdake_ | moves why and how is that a problem | 14:49 |
Jeffrey4l | when it move from A -> B -> A, then issue occur. A have that namespace with readonly | 14:49 |
sdake_ | got it | 14:49 |
sdake_ | what triggerss the move? | 14:50 |
Jeffrey4l | l3 ha possible do like that. | 14:50 |
sdake_ | so l3 ha triggers this problem - naything else? | 14:50 |
Jeffrey4l | i think there are some api can do this too ( But i do not know which one is) | 14:50 |
sdake_ | see if its critical need to understand the impact | 14:51 |
sdake_ | i.e. hwen it happens | 14:51 |
Jeffrey4l | when run one reconfigure or neutron_* container is restarted. | 14:52 |
sdake_ | if we know when, we can possibly workorund it | 14:52 |
Jeffrey4l | i do not think it has any workaround. we need fix this in docker/linux kernal. | 14:52 |
sdake_ | Jeffrey4l http://i.imgur.com/iWKad22.jpg | 14:53 |
Jeffrey4l | or event in neutron. | 14:53 |
Jeffrey4l | even | 14:53 |
Jeffrey4l | hmm. i really think it is a bug in kernel :( ( but i haven't any proof ) | 14:54 |
sdake_ | double http://i.imgur.com/iWKad22.jpg | 14:54 |
Jeffrey4l | lol | 14:55 |
sdake_ | if we are too bleeding edge for the kernels, then we need to back off the edge | 14:55 |
sdake_ | so this kernel bug theory | 14:55 |
sdake_ | have you tried 16.04.01 ubuntu kernel? | 14:56 |
Jeffrey4l | i use centos. And tested with latest kernel ( 4.4? ), still failed. | 14:56 |
sdake_ | you built your own kernel then? | 14:56 |
Jeffrey4l | no.. install from elrepo | 14:56 |
sdake_ | cool didnt' know they had a modern kernel avaailable | 14:57 |
Jeffrey4l | http://i.imgur.com/iWKad22.jpg | 14:57 |
Jeffrey4l | sorry, that's a typo ^^ i do not mean that. | 14:57 |
sdake_ | ok | 14:57 |
sdake_ | i meant mine ;) | 14:58 |
Jeffrey4l | http://elrepo.org/tiki/kernel-ml | 14:58 |
Jeffrey4l | that's it. | 14:58 |
kfox1111 | ah. finally found the log file. | 14:58 |
kfox1111 | I'm really not use to looking for things in /var/log/kolla | 14:58 |
sdake_ | Jeffrey4l looks good | 14:58 |
kfox1111 | 2016-09-25 14:56:57.452+0000: 19043: error : secretLoadValue:400 : internal error: invalid base64 in '/etc/libvirt/secrets/50030df6-b546-433f-baa3-85c57f747754.base64' | 14:59 |
kfox1111 | 2016-09-25 14:56:57.452+0000: 19043: error : loadSecrets:491 : Error reading secret: internal error: invalid base64 in '/etc/libvirt/secrets/50030df6-b546-433f-baa3-85c57f747754.base64' | 14:59 |
kfox1111 | so, thats the problem. | 14:59 |
Jeffrey4l | kfox1111, from k82? | 14:59 |
Jeffrey4l | k8s? | 14:59 |
kfox1111 | from libvirtd. | 14:59 |
sdake_ | Jeffrey4l so to summarize the issue - rconfigure works once second time neutron implodes | 15:00 |
kfox1111 | I may not have right number of base64 encodings. :/ | 15:00 |
Jeffrey4l | ok. | 15:00 |
sdake_ | Jeffrey4l is that accurate? | 15:01 |
Jeffrey4l | sdake_, we can re-produce without reconfigure. | 15:01 |
Jeffrey4l | 1. deploy kolla | 15:01 |
sdake_ | explain that | 15:01 |
Jeffrey4l | 2. create one route ( which create one namespace ) | 15:01 |
Jeffrey4l | 3. restart neutron_* container | 15:01 |
Jeffrey4l | 4. delete route | 15:01 |
Jeffrey4l | 5. issue occur. the namespace become readonly. | 15:02 |
sdake_ | on step 3 or step 4? | 15:02 |
Jeffrey4l | 4 | 15:02 |
Jeffrey4l | in 3, every thing seem ok. | 15:03 |
Jeffrey4l | until you remove the route ( namespace ) | 15:03 |
sdake_ | this is with dumb-init? | 15:03 |
Jeffrey4l | sdake_, no. it exist long time ago. | 15:04 |
sdake_ | i mean does dumb_init fixt his? | 15:04 |
Jeffrey4l | dumb-init doesn't fix this. | 15:04 |
sdake_ | ok | 15:04 |
sdake_ | you outlined steps 1-4 above | 15:05 |
sdake_ | tht seems simple enough | 15:05 |
sdake_ | prticualry 2,3,4 | 15:05 |
sdake_ | to simulate using some other mechanism then kolla | 15:05 |
Jeffrey4l | i tried, but failed :( | 15:05 |
sdake_ | how much time did you sink into it? | 15:06 |
Jeffrey4l | sink into means saw this? or debug this? | 15:07 |
sdake_ | how much time did you spend tryign to make a reproducer ethat doesn't include openstack? | 15:07 |
Jeffrey4l | and the answer is both `very much` | 15:07 |
sdake_ | what about a neutron fat container | 15:09 |
*** haplo37_ has quit IRC | 15:10 | |
sdake_ | if it happens in a neutron fat container - it is not munt propoogation | 15:10 |
Jeffrey4l | sdake_, that's doesn't matter. when a create a route, i didn't create any network or subnet. | 15:10 |
sdake_ | and by corralary its neutron | 15:11 |
sdake_ | if we can narrow it down to mount propogaiton (ekrnel or docker) thats good | 15:12 |
Jeffrey4l | but it won't happen in neutron deploy without container. | 15:12 |
sdake_ | if we can narrow it down to neutron thats even better | 15:12 |
Jeffrey4l | yes. | 15:12 |
*** haplo37_ has joined #openstack-kolla | 15:13 | |
sdake_ | deploy neutron agents in fat container will narrow it down | 15:13 |
kfox1111 | hmm. so the base64 has a double base64 encoded key in it... | 15:13 |
Jeffrey4l | ok " fat container" | 15:13 |
sdake_ | the fact that docker restart plays into it is a bit trobuling | 15:14 |
*** neilus has joined #openstack-kolla | 15:14 | |
Jeffrey4l | maybe. | 15:14 |
sdake_ | so in essence | 15:16 |
sdake_ | if we can't reproduce it outside of openstack, lets minimize the search space for the problem | 15:17 |
Jeffrey4l | ok. | 15:17 |
sdake_ | kfox1111 don't know why :) | 15:17 |
*** neilus has quit IRC | 15:19 | |
*** tonanhngo has joined #openstack-kolla | 15:21 | |
*** tonanhngo has quit IRC | 15:23 | |
*** rmart04 has quit IRC | 15:24 | |
sdake_ | Jeffrey4l what about comment #16 | 15:25 |
sdake_ | https://bugs.launchpad.net/kolla/+bug/1616268 | 15:25 |
openstack | Launchpad bug 1616268 in kolla mitaka "Stale namespace removal causing "RTNETLINK answers: Invalid argument" errors" [Critical,Confirmed] | 15:25 |
Jeffrey4l | the namespace file become a regular file. | 15:26 |
Jeffrey4l | sdake_, | 15:26 |
sdake_ | right but thats a rerpoducer outside of kolla? | 15:26 |
Jeffrey4l | sdake_, no. we have no idea why it become a regular file. | 15:27 |
Jeffrey4l | it still reproduced under kolla | 15:27 |
sdake_ | but does that example work outside of kolla? | 15:27 |
Jeffrey4l | no. | 15:28 |
Jeffrey4l | the direct reason is: namespace file becomes a regular file. But we still have no idea why. and can not reproduce this without kolla now. | 15:29 |
Jeffrey4l | should go to bed . bye guys :) | 15:30 |
Jeffrey4l | good night. | 15:31 |
*** caowei has quit IRC | 15:31 | |
sdake_ | night | 15:36 |
*** sdake_ has quit IRC | 15:36 | |
kfox1111 | hmm... | 15:39 |
kfox1111 | seems like it may be objecting to a newline... | 15:39 |
*** tonanhngo has joined #openstack-kolla | 15:41 | |
kfox1111 | yup. was the newline. :/ | 15:43 |
*** kbaegis has joined #openstack-kolla | 15:49 | |
kbaegis | Hey guys, I'm having an issue with the override config | 15:49 |
kbaegis | for AIO, is /etc/kolla/config/local/ml2_conf.ini the correct path for a ml2 config? | 15:50 |
kbaegis | or should it just be /etc/kolla/config/ml2_conf.ini? | 15:50 |
*** srwilkers has joined #openstack-kolla | 16:03 | |
*** daneyon has joined #openstack-kolla | 16:11 | |
openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Adds sidecar logging container to common-deployment pod https://review.openstack.org/375682 | 16:12 |
sbezverk | kfox1111: are you going to push a fix for loading jinja2 templates? | 16:13 |
*** rmart04 has joined #openstack-kolla | 16:15 | |
*** daneyon has quit IRC | 16:15 | |
*** z- has quit IRC | 16:16 | |
*** z- has joined #openstack-kolla | 16:16 | |
sbezverk | kfox1111: I was attepting to test your more deployments PS with fluentd macro, but kolla-kube is not happy.. | 16:17 |
*** kbaegis has quit IRC | 16:21 | |
*** eaguilar has joined #openstack-kolla | 16:27 | |
*** schwicht has quit IRC | 16:30 | |
*** schwicht_at_work has joined #openstack-kolla | 16:30 | |
*** rmart04 has quit IRC | 16:37 | |
*** senk_ has joined #openstack-kolla | 16:40 | |
*** rmart04 has joined #openstack-kolla | 16:41 | |
*** kbaegis has joined #openstack-kolla | 16:43 | |
*** zhubingbing_ has quit IRC | 16:43 | |
*** rmart04 has quit IRC | 16:44 | |
kbaegis | Can anyone help me with the extended config? | 16:44 |
*** rmart04 has joined #openstack-kolla | 16:45 | |
*** eaguilar has quit IRC | 16:46 | |
*** rmart04 has quit IRC | 16:47 | |
kbaegis | nvm. It's actually covered pretty well :0 | 16:48 |
*** neilus has joined #openstack-kolla | 16:55 | |
*** schwicht_at_work has quit IRC | 16:57 | |
*** neilus has quit IRC | 17:00 | |
*** schwicht has joined #openstack-kolla | 17:02 | |
*** jrich523 has quit IRC | 17:02 | |
*** jrich523 has joined #openstack-kolla | 17:04 | |
*** z- has quit IRC | 17:04 | |
*** z- has joined #openstack-kolla | 17:04 | |
kfox1111 | sbezverk: what's it doing? | 17:06 |
kfox1111 | I was able to test the ceph patch set with the more deployments ps and all the deployments worked. | 17:06 |
kbaegis | 72 hours, heat : Creating the heat_stack_user role is still broken on master | 17:12 |
*** senk_ has quit IRC | 17:12 | |
*** neilus has joined #openstack-kolla | 17:13 | |
*** neilus has quit IRC | 17:17 | |
*** kbyrne has quit IRC | 17:26 | |
kbaegis | http://pastebin.com/P1fcJbVP | 17:27 |
sbezverk | kfox1111: the gate still fails to find template, I thought you were going to push a fix for the gate so it would load all temapltes from ./common/macro* | 17:28 |
*** kbyrne has joined #openstack-kolla | 17:29 | |
kbaegis | I think merge 040b07f bcff0f8 from jenkins has a regression | 17:30 |
kfox1111 | sbezverk: I think the fix needs to either go into your patch, or we need a new patch for the feature. its not a gate spacific issue I think, but the engine needs to have that fix to let it support imports properly. | 17:31 |
kfox1111 | Do you want to write it, or should I? | 17:31 |
sbezverk | kfox1111: Ok I will take a stab at it.. | 17:33 |
kbaegis | Verified. ansible/roles/heat/tasks/register.yml is messed up due to the scripting in commit 9d7ba661f3a1f5de4cff06d531025319f7342712 | 17:33 |
kbaegis | Should break deploy on master for everyone | 17:33 |
kfox1111 | sbezverk: I've got glance out of ceph working, and I think I'm realy really close to having nova working. | 17:34 |
kfox1111 | cinder should be pretty easy after that. | 17:35 |
*** kbyrne has quit IRC | 17:39 | |
sbezverk | kfox1111: not a big fan of ceph, but I am happy for you ;-) | 17:42 |
kfox1111 | I love ceph. :) been able to avoid several major outages due to it. | 17:49 |
kfox1111 | including physically moving the ceph one node at a time from one rack to another without downtime, | 17:49 |
kfox1111 | and a second time moving all the ost's from one ack to another via moving the software from one rack to another and syncing the osd's one at a time. | 17:49 |
kfox1111 | havent' seen a file system as bullet proof as ceph yet. | 17:49 |
kfox1111 | at this point, I think its best to use ceph for everything you can except things where performance really is critical. storage is pretty cheep so if you can get 90%+ of your data on resilient storage like ceph, life is sooo much better. :) | 17:53 |
*** rmart04 has joined #openstack-kolla | 17:53 | |
*** srwilkers has quit IRC | 17:55 | |
*** rmart04 has quit IRC | 17:56 | |
*** rmart04 has joined #openstack-kolla | 17:57 | |
*** rmart04 has quit IRC | 17:58 | |
*** senk_ has joined #openstack-kolla | 17:58 | |
*** rmart04 has joined #openstack-kolla | 18:02 | |
kfox1111 | there we go... :) | 18:05 |
*** rmart04 has quit IRC | 18:12 | |
*** srwilkers has joined #openstack-kolla | 18:14 | |
*** tonanhngo has quit IRC | 18:17 | |
*** senk_ has quit IRC | 18:18 | |
*** senk_ has joined #openstack-kolla | 18:19 | |
*** g3ek has quit IRC | 18:20 | |
*** haplo37 has quit IRC | 18:20 | |
*** haplo37 has joined #openstack-kolla | 18:29 | |
*** g3ek has joined #openstack-kolla | 18:29 | |
*** senk_ has quit IRC | 18:32 | |
*** sdake has joined #openstack-kolla | 18:33 | |
*** sdake_ has joined #openstack-kolla | 18:36 | |
*** sdake_ has quit IRC | 18:36 | |
*** sdake has quit IRC | 18:37 | |
*** pbourke__ has quit IRC | 18:38 | |
*** pbourke__ has joined #openstack-kolla | 18:39 | |
*** gmmaha has quit IRC | 18:46 | |
*** gmmaha has joined #openstack-kolla | 18:48 | |
*** sdake has joined #openstack-kolla | 18:49 | |
*** srwilkers has quit IRC | 18:52 | |
*** sdake has quit IRC | 18:53 | |
*** daneyon has joined #openstack-kolla | 18:54 | |
*** kbyrne has joined #openstack-kolla | 18:54 | |
*** haplo37_ has quit IRC | 18:57 | |
*** daneyon has quit IRC | 18:58 | |
*** jistr has quit IRC | 18:59 | |
*** haplo37_ has joined #openstack-kolla | 18:59 | |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Basic ceph deployment support https://review.openstack.org/373256 | 18:59 |
*** unicell has joined #openstack-kolla | 19:10 | |
*** tonanhngo has joined #openstack-kolla | 19:33 | |
*** tonanhngo has quit IRC | 19:36 | |
*** unicell1 has joined #openstack-kolla | 19:38 | |
*** unicell has quit IRC | 19:38 | |
*** jamesdenton has joined #openstack-kolla | 19:38 | |
*** srwilkers has joined #openstack-kolla | 19:53 | |
*** srwilkers has quit IRC | 19:53 | |
*** haplo37_ has quit IRC | 20:19 | |
*** haplo37_ has joined #openstack-kolla | 20:19 | |
*** srwilkers has joined #openstack-kolla | 20:25 | |
*** jmccarthy has quit IRC | 20:28 | |
*** jmccarthy has joined #openstack-kolla | 20:29 | |
*** neilus has joined #openstack-kolla | 20:31 | |
*** Jeffrey4l has quit IRC | 20:34 | |
*** Jeffrey4l has joined #openstack-kolla | 20:35 | |
*** neilus has quit IRC | 20:36 | |
*** srwilkers has quit IRC | 20:37 | |
*** neilus has joined #openstack-kolla | 20:55 | |
*** neilus_ has joined #openstack-kolla | 21:03 | |
*** neilus has quit IRC | 21:04 | |
*** kbaegis has left #openstack-kolla | 21:14 | |
*** brad[]` has joined #openstack-kolla | 21:35 | |
*** tonanhngo has joined #openstack-kolla | 21:38 | |
*** tonanhngo has quit IRC | 21:39 | |
*** sdake has joined #openstack-kolla | 21:56 | |
*** sdake has quit IRC | 22:03 | |
*** sdake has joined #openstack-kolla | 22:04 | |
*** neilus_ has quit IRC | 22:13 | |
openstackgerrit | Waldemar Znoinski proposed openstack/kolla: fix ironic-inspector setup https://review.openstack.org/374727 | 22:15 |
openstackgerrit | Waldemar Znoinski proposed openstack/kolla: install iptables in ironic-inspector container https://review.openstack.org/374970 | 22:15 |
openstackgerrit | Waldemar Znoinski proposed openstack/kolla: install iptables in ironic-inspector container https://review.openstack.org/374970 | 22:16 |
*** neilus has joined #openstack-kolla | 22:17 | |
*** neilus has quit IRC | 22:22 | |
*** daneyon has joined #openstack-kolla | 22:30 | |
*** daneyon has quit IRC | 22:35 | |
*** haplo37_ has quit IRC | 22:56 | |
*** haplo37_ has joined #openstack-kolla | 22:58 | |
*** eaguilar has joined #openstack-kolla | 23:04 | |
*** jascott1_ has joined #openstack-kolla | 23:27 | |
*** jascott1_ has quit IRC | 23:34 | |
*** lujinluo has joined #openstack-kolla | 23:44 | |
kfox1111 | there we go... cinder+ceph working. :) | 23:57 |
*** yingjun has joined #openstack-kolla | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!