Sunday, 2016-08-07

« Saturday, 2016-08-06 Index Monday, 2016-08-08 »
*** daneyon has joined #openstack-kolla00:10
*** sdake has joined #openstack-kolla00:13
*** daneyon has quit IRC00:14
*** rhallisey has quit IRC00:26
*** sdake has quit IRC00:32
*** fragatina has joined #openstack-kolla00:48
*** fragatina has quit IRC00:54
*** banix has quit IRC00:55
*** mkoderer_ has quit IRC01:02
*** mkoderer has joined #openstack-kolla01:03
*** lmiccini_ has quit IRC01:03
*** sdake has joined #openstack-kolla01:05
*** sdake_ has joined #openstack-kolla01:07
*** lmiccini has joined #openstack-kolla01:08
*** sdake has quit IRC01:10
*** huikang has joined #openstack-kolla01:12
*** Administrator_ has joined #openstack-kolla01:15
*** zhugaoxiao has quit IRC01:18
*** zhugaoxiao has joined #openstack-kolla01:19
*** Administrator_ has quit IRC01:19
*** banix has joined #openstack-kolla01:20
*** Jeffrey4l has quit IRC01:55
*** daneyon has joined #openstack-kolla01:58
*** daneyon has quit IRC02:03
*** Jeffrey4l has joined #openstack-kolla02:07
*** dwalsh has joined #openstack-kolla02:13
*** huikang has quit IRC02:22
*** sdake_ has quit IRC02:23
*** huikang has joined #openstack-kolla02:25
*** sdake has joined #openstack-kolla02:28
*** huikang has quit IRC02:32
*** dwalsh has quit IRC02:38
*** dwalsh has joined #openstack-kolla02:43
*** fragatina has joined #openstack-kolla02:51
*** dwalsh has quit IRC02:52
*** fragatina has quit IRC02:56
*** signed8bit_Zzz is now known as signed8bit03:03
*** signed8bit is now known as signed8bit_Zzz03:03
*** huikang has joined #openstack-kolla03:08
*** signed8bit_Zzz is now known as signed8bit03:12
*** huikang has quit IRC03:13
openstackgerritJeffrey Zhang proposed openstack/kolla: Make the kolla_keystone_service can update fields  https://review.openstack.org/34838203:15
openstackgerritJeffrey Zhang proposed openstack/kolla: Enable the nova microversion api  https://review.openstack.org/34843203:15
*** signed8bit has quit IRC03:17
*** mkoderer has quit IRC03:37
*** lmiccini has quit IRC03:37
*** huikang has joined #openstack-kolla03:38
*** sdake has quit IRC03:38
*** mkoderer has joined #openstack-kolla03:39
*** lmiccini has joined #openstack-kolla03:43
*** dave-mccowan has quit IRC03:44
*** daneyon has joined #openstack-kolla03:47
*** daneyon has quit IRC03:51
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet [WIP]  https://review.openstack.org/34936604:00
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet [WIP]  https://review.openstack.org/34936604:00
openstackgerritShaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet  https://review.openstack.org/35113904:06
openstackgerritShaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet  https://review.openstack.org/35113904:08
*** banix has quit IRC04:15
*** zhugx has joined #openstack-kolla04:59
*** huikang has quit IRC05:38
*** fragatina has joined #openstack-kolla05:47
*** fragatina has quit IRC05:54
*** lmiccini has quit IRC06:22
*** mkoderer has quit IRC06:24
*** mkoderer has joined #openstack-kolla06:25
*** lmiccini has joined #openstack-kolla06:28
*** senk has joined #openstack-kolla06:29
*** daneyon has joined #openstack-kolla06:29
*** daneyon has quit IRC06:34
openstackgerritHiroki Ito proposed openstack/kolla: Prechecks fails when using multinode deploy using a single node and haproxy disabled  https://review.openstack.org/35158806:37
*** zhurong has joined #openstack-kolla06:44
*** senk has quit IRC07:02
*** senk__ has joined #openstack-kolla07:02
*** senk has joined #openstack-kolla07:22
*** senk__ has quit IRC07:23
*** dwalsh has joined #openstack-kolla07:26
*** dwalsh has quit IRC07:44
*** mewald has joined #openstack-kolla07:46
*** fragatina has joined #openstack-kolla07:50
*** fragatina has quit IRC07:56
*** bootsha has joined #openstack-kolla08:09
*** zhurong has quit IRC08:11
*** daneyon has joined #openstack-kolla08:17
*** daneyon has quit IRC08:22
*** bootsha has quit IRC08:34
*** bootsha has joined #openstack-kolla08:36
*** bootsha has quit IRC08:39
*** senk has quit IRC08:42
*** fragatina has joined #openstack-kolla09:53
*** mewald has quit IRC09:55
*** senk has joined #openstack-kolla09:56
*** fragatina has quit IRC09:58
*** daneyon has joined #openstack-kolla10:06
openstackgerritJeffrey Zhang proposed openstack/kolla: Make the kolla_keystone_service can update fields  https://review.openstack.org/34838210:06
openstackgerritJeffrey Zhang proposed openstack/kolla: Enable the nova microversion api  https://review.openstack.org/34843210:06
*** senk has quit IRC10:08
*** daneyon has quit IRC10:10
*** dwalsh has joined #openstack-kolla10:13
*** ad_rien_ has joined #openstack-kolla10:14
*** dwalsh has quit IRC10:30
*** bootsha has joined #openstack-kolla10:36
*** egonzalez90 has joined #openstack-kolla11:22
*** Jeffrey4l has quit IRC11:24
*** zhurong has joined #openstack-kolla11:29
*** dave-mccowan has joined #openstack-kolla11:34
*** egonzalez90 has quit IRC11:36
*** rhallisey has joined #openstack-kolla11:39
*** fragatina has joined #openstack-kolla11:47
*** fragatina has quit IRC11:53
*** daneyon has joined #openstack-kolla11:54
*** daneyon has quit IRC11:59
*** senk has joined #openstack-kolla12:04
*** senk has quit IRC12:29
*** zhugaoxiao has quit IRC12:37
*** zhurong has quit IRC12:38
*** zhugaoxiao has joined #openstack-kolla12:38
*** zhurong has joined #openstack-kolla12:39
*** zhurong has quit IRC12:43
*** zhurong has joined #openstack-kolla12:45
*** banix has joined #openstack-kolla12:48
*** zhurong has quit IRC12:49
*** sdake has joined #openstack-kolla12:50
*** signed8bit has joined #openstack-kolla12:52
*** sdake_ has joined #openstack-kolla12:53
*** sdake has quit IRC12:54
sdake_morning12:55
*** banix has quit IRC13:05
sbezverksdake_ morning, do you have sometime next week to dedicate to the traceback issue? I am afraid the workaround is not as stable as I was hoping..13:13
sdake_want to do now?13:14
sdake_or next week13:14
sdake_the week is super busy typically13:14
sbezverknext week, I am about to leave for a trip and my test bed is down..13:14
sdake_ok - well lets play it by ear13:15
sdake_the afternoons are typically downtime for our community - so that may be the best time to do the work13:15
sdake_we need uninterrupted time13:15
sbezverkextra piece of info. while the script is short it looks stable13:15
sdake_and from around 5am -> 3pm PST the irc channel is off the hook13:15
sbezverkbut with extra commands I absolutely need it starts behaving as the original issue13:16
sdake_even when using a rc insted of a ds?13:16
sdake_if so, that sounds like a fundamental isue with kubernetes and openvswitch integration13:17
sbezverkok I can go for 9pm est monday or wednesday13:17
sdake_we can do earlier if you like13:17
sdake_but all depends on how busy the channel is13:17
sdake_have many cats to feed :)13:18
* sdake_ is crazy cat lady13:18
sbezverk:-)13:18
sbezverkI am not so sure about fundamental issue as on one of two nodes it works perfectly..13:18
sdake_but 3 ndoes fails?13:20
sdake_sbezverk it could be your hardware13:23
sdake_does your gear include ECC ram?13:23
sbezverk:-) it happens on both nodes (I have two compute nodes)13:24
sbezverkjust not at the same time13:24
*** banix has joined #openstack-kolla13:26
*** banix has quit IRC13:26
*** signed8bit is now known as signed8bit_Zzz13:26
*** banix has joined #openstack-kolla13:29
openstackgerritMerged openstack/kolla-kubernetes: Spec - Deploy kolla-kubernetes with Ansible  https://review.openstack.org/33527913:39
*** daneyon has joined #openstack-kolla13:43
*** senk has joined #openstack-kolla13:45
*** Jeffrey4l has joined #openstack-kolla13:46
*** daneyon has quit IRC13:47
*** fragatina has joined #openstack-kolla13:51
*** signed8bit_Zzz is now known as signed8bit13:55
*** fragatina has quit IRC13:56
*** diogogmt has quit IRC13:57
*** diogogmt has joined #openstack-kolla13:59
*** banix has quit IRC14:08
sdake_sbezverk even with rcs?14:09
sdake_you said it wasn't as stable as you originally thgouth - could you cxpand on that statement14:10
*** diogogmt has quit IRC14:11
*** signed8bit is now known as signed8bit_Zzz14:14
*** senk has quit IRC14:25
*** zhugx has quit IRC14:34
*** huikang has joined #openstack-kolla14:44
*** sdake_ has quit IRC14:52
*** signed8bit_Zzz is now known as signed8bit15:01
*** sdake has joined #openstack-kolla15:05
sdakepbourke around?15:06
sdakesean-k-mooney around?15:09
sdakeany other cats that have used the osic cluster already - could use a bone thrown :)15:09
*** duonghq has joined #openstack-kolla15:13
*** huikang has quit IRC15:19
*** signed8bit is now known as signed8bit_Zzz15:20
*** zhugaoxiao has quit IRC15:21
*** zhugaoxiao has joined #openstack-kolla15:22
*** huikang has joined #openstack-kolla15:22
*** daneyon has joined #openstack-kolla15:31
*** daneyon has quit IRC15:35
*** dwalsh has joined #openstack-kolla16:01
*** dave-mccowan has quit IRC16:10
*** huikang has quit IRC16:12
*** dave-mccowan has joined #openstack-kolla16:14
*** duonghq has left #openstack-kolla16:18
*** senk has joined #openstack-kolla16:20
*** huikang has joined #openstack-kolla16:26
*** huikang has quit IRC16:31
*** zhubingbing has joined #openstack-kolla16:35
openstackgerritzhubingbing proposed openstack/kolla: Add aodh role  https://review.openstack.org/35102716:37
openstackgerritzhubingbing proposed openstack/kolla: Add sahara ansible role  https://review.openstack.org/35129416:45
*** dwalsh has quit IRC17:07
*** daneyon has joined #openstack-kolla17:19
*** daneyon has quit IRC17:24
sdakeso....17:35
sdakehot....17:35
*** fragatina has joined #openstack-kolla17:47
*** fragatina has quit IRC17:52
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935117:53
openstackgerritzhubingbing proposed openstack/kolla: fix sahara dockerfile  https://review.openstack.org/35132017:56
zhubingbing- -17:56
zhubingbinghot17:56
sdakeya 115F18:04
sdakearizona is a very hot place in the summer - atleast in phoenix18:04
zhubingbing去游泳18:07
zhubingbing去游泳18:07
zhubingbingGo for a swim18:07
zhubingbing- -18:07
zhubingbingwe're hot, too18:09
*** senk has quit IRC18:15
zhubingbingsdake，see you18:18
sdakeswim lol18:18
sdakezhubingbing sorry to hear it :(18:18
zhubingbingWe are 2 in the morning, tomorrow we have to go on working.18:20
zhubingbingi'll miss you :）18:21
sdakettyl18:23
sbezverksdake ping18:36
sdakewound me sbezverk18:36
sbezverkanother observation if 3 container pod gets into an issue, then by manually restarting ovsdb container recovers everything18:37
sbezverkdo you want to add some extra debugging to ovsdb-server source and recompile it?18:39
sbezverkideally it should be done by ovs developers, but I am not sure if they go for it18:40
openstackgerritSteven Dake proposed openstack/kolla: Add OSIC Scale Testing Documentation  https://review.openstack.org/35210118:41
sdaketoo much on my plate to do that atm sbezverk18:41
sdakelets get a backtrace18:41
sdakeand call it a day18:41
sdakei dont think \you understand how much impact a backtrace has on c developers18:43
sdakeit will spur action - take my word or it18:43
sdakewe may need to do a little lmor then a backtrace18:44
sdakebut lets get the thing into gdb so we can get a backtrace and produce debug inf ofor ovs cats to work with18:44
sdakeright now what your telling them is "it doesn't work"18:44
sbezverkthings changed a little bit..18:44
sdakeyour not telling them why18:44
sdakeif you tell them why- they will fix it18:45
sbezverknow ovsdv-server does not generate backtrace18:45
sbezverkit just does not create socket18:45
sdakemoment need to switch networks - done with osic cluster fo rthe moment18:45
sbezverkok18:45
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935118:46
*** sdake_ has joined #openstack-kolla18:47
*** sdake has quit IRC18:50
sdake_sbezverk just red your email18:54
sdake_my immediate response from an ovs point of view is "get me a backtrace of th ecrash"18:54
sdake_the socket being created or not is not relevant18:54
sdake_that happens after the crash18:54
sdake_anything that happens after a crash is bad data18:55
sdake_junk in = junk out18:55
sdake_we needd to get to the good in -> junk out and see why the junk out is happening18:55
sbezverksdake_ there is no crash!!18:55
sdake_you had a crash with daemonsets18:55
sbezverksdake_ not anymore ovsdb just does not create a socket18:56
sdake_the segfault fixed itself?18:56
sbezverkI suspect what we saw if either not releated or another issue18:56
sbezverks/if/is/18:57
sdake_how did the segfault fix itself on daemon sets18:57
sdake_did you reconfigure the gear?18:57
sbezverksdake_ nope, I was playing with commands and delays in the script18:57
sdake_ok so you ahve a delay18:58
sdake_if you take the delay out - you can still get a crash right?18:58
sbezverkI could try but at this point since I still have problem even without seeing seg fault, why would we want it?18:59
sdake_to get a backtrace of course is why we want  the crash18:59
sdake_but if your getting a running environment without a crash just no socket18:59
sdake_and allergic to gdb19:00
sdake_another option is to run it through strace19:00
sdake_that would be helpful as well19:00
sbezverkok cool, let me try it, but remember last time as soon as we added strace everything started working automagically :-)19:01
openstackgerritChristian Berendt proposed openstack/kolla: Remove files from /var/lib/apt/lists when cleaning up on Ubuntu/Debian  https://review.openstack.org/35173819:03
sdake_that is because strace got rid of the segfault19:04
sdake_but you just said there is no longer a segfault19:04
*** daneyon has joined #openstack-kolla19:07
*** signed8bit_Zzz is now known as signed8bit19:11
*** daneyon has quit IRC19:12
*** senk has joined #openstack-kolla19:20
*** senk has quit IRC19:21
openstackgerritChristian Berendt proposed openstack/kolla: Unify keystone endpoint descriptions  https://review.openstack.org/35211019:31
sdake_sbezverk any results with strace?19:42
sbezverksorry had small house emeregency19:43
Mech422sdake_: that sounds like a classic race condition...19:44
Mech422sdake_: timing sensitive, magically 'disappears', etc etc19:45
sbezverksdake_ with strace it works without hickup19:45
sdake_Mech422 yes of couse19:45
sdake_sbezverk are you sure your assertion there is no crash is correct19:45
Mech422sdake_: if the ovs stuff is in a container, perhaps its trying to setup ovs before the host is ready ?19:46
sdake_Mech422 sbezverk has a sleep in there to prevent that scenario19:46
sbezverkMech422: here is the funny thing, I have two compute node in a cluster, the issue appears randomly on one of these nodes19:47
sbezverknot on the same19:47
sbezverkalways, but I do not see any pattern19:47
sdake_dmesg shows no segfault sbezverk ?19:48
Mech422sbezverk: yeah - race of some sort19:48
sdake_it is unliekly tobe a race if there is a sleep 10 at teh start of things19:48
Mech422ovsdb just need /var/run and the db location IIRC19:49
Mech422just check - for ubuntu, it's looking for /var/run/openvswitch/*19:50
Mech422s/check/checked/19:50
Mech422hmm - lsof shows it wants /var/lib/openvswitch for a lock file, and /var/log/openvswitch19:51
sbezverkin my case the socket is at /run/openvswitch/db.sock19:52
Mech422and /dev/null19:52
sbezverkand db sits at /etc/openvswitch/conf.dbn19:52
Mech422sbezverk: the actuall db is in /etc/openvswitch? or the conf ? funny place for a db file...19:53
sbezverkI do not know how it got there, but it is the same location as in classical kolla :-)19:54
sbezverksdake_ 3 times with strace no issue19:55
sdake_sbezverk what about dmesg?19:55
sdake_(whtout strace)19:55
sbezverk21586.002436] traps: handler30[17143] general protection ip:7fb6696b4e37 sp:7fb667c239d0 error:0 in libc-2.17.so[7fb66967e000+1b7000]19:56
sbezverk[19:56
sbezverkhere is with strace19:56
sdake_you said earlier there was no segfault19:56
sdake_clearly there is sstill a segfault19:56
sbezverkwell but where do you see its realted to ovs?19:56
sdake_the segfault is related to why the socket is not created19:56
sbezverkthe socket is created19:57
sdake_because without strace there is no segfault19:57
sdake_rather with strace there is no segfault19:57
sbezverkI keep telling you it does not seem to be related19:57
sdake_i keep telling you it is related19:57
sdake_the evidence is as follows for my position19:57
Mech422is it me, or does a protection fault in libc smell like bad memory ?19:57
sdake_strace -> no segfault -> works19:57
sdake_no strace -> segfault -> doesn't work19:58
sbezverkok now: I have strace on, I have socket created and I see seg fault of this god knows what this process is19:58
sdake_sbezverk need to see your screen up for a webex19:58
sbezverkok but I will not be able to join voice bridge19:58
sdake_well thats not helpful :)19:59
sbezverksorry cannot do anything about it atm19:59
openstackgerritChristian Berendt proposed openstack/kolla: Remove sudo commands from docs  https://review.openstack.org/35211819:59
sdake_i wasn't complaining19:59
sdake_ok so lets get focused here20:00
sdake_i need the following yes or no questions asnswered20:00
*** senk has joined #openstack-kolla20:00
sdake_with strace, does a segfault occur?20:00
sbezverkyes but for some unrelatd to ovs process20:01
sbezverkI got the issue with strace20:02
sdake_got wich issue - no socket created?20:02
openstackgerritChristian Berendt proposed openstack/kolla: Remove heat dev environment  https://review.openstack.org/35211920:03
sbezverkhttp://paste.openstack.org/show/551517/20:04
sbezverkyes20:04
Mech422sbezverk: ovsdb-server creates a sub-process - it runs a monitoring process on the original pid and the actual db in the sub - so your segfault might be in the sub process ?20:04
*** imcsk8 has quit IRC20:05
openstackgerritChristian Berendt proposed openstack/kolla: Unify keystone endpoint descriptions  https://review.openstack.org/35211020:05
sdake_line 272 showss the socket is there20:05
sdake_line 280 is problematic20:06
sdake_let me track down errno 1120:06
*** fragatina has joined #openstack-kolla20:06
*** imcsk8 has joined #openstack-kolla20:06
sdake_resource temporarily unavailable20:07
sbezverksdake_ you are right the socket now there, you need to check my initialization script20:07
sbezverk1st I use ovsdb-server to run a command on its own DB to add external bridge20:08
*** ad_rien_1 has joined #openstack-kolla20:08
*** ad_rien_ has quit IRC20:08
sbezverk2nd I use ovsdb-server to run a command to plug external interface to newly created bridge20:08
sbezverk3rd I start ovsdb-server process20:08
sdake_sbezverk ok lets focus on line 280 for a moment20:09
sdake_can you look in /usr/include/asm-generic20:09
sbezverkSo when 1st line was running ovsdb-server process did not open a socket and external bridge did not get created20:09
sdake_and look for 1120:09
*** zhubingbing has quit IRC20:09
sdake_just give me a moment to switch networks20:10
Mech422sbezverk: wait - are you using ovs inside AND outside of kolla container ?20:10
Mech422sbezverk: (eg is your host networking ovs based ?)20:10
*** fragatina has quit IRC20:11
sbezverkthere is no kolla :-)20:11
Mech422sbezverk: ahh - you mentioned kolla default location before...confused me ... my bad20:11
sbezverkmy ucs server runs 5 VMs each VM is a node in a cluster20:11
*** sdake has joined #openstack-kolla20:11
sbezverkbut you are right I do use ovs on my host to connect these VMs20:12
*** senk has quit IRC20:12
sdakesbezverk i need to download kernel.org20:12
sdaketo see why your getting an EAGAIN20:12
Mech422sdake: here's what I get for error 11:20:12
Mech422root@os-control-01:~# grep 11 /usr/include/asm-generic/socket.h20:13
Mech422#define SO_NO_CHECK     1120:13
sdakeMech422 nah - this is EAGAIN not SO_NO_CHECK20:13
sbezverksdake: /usr/include/asm/signal.h:#define SIGSEGV               1120:13
sdakethe socket manual page doesn't list EAGAIN as a return code20:13
sdakeerrno.h guys ;)20:13
*** sdake_ has quit IRC20:14
sdakesbezverk which kernel version do you have20:14
sbezverk#define EWOULDBLOCK     EAGAIN  /* Operation would block */20:14
sbezverk3.10.0-327.22.2.el7.x86_6420:15
sdakeif you type "man socket" it doesn't list eagain as a reutrn code20:15
sdakered hat's kerknel20:15
sbezverkcentos20:15
sbezverkif it makes things easier I can get 4.5 or 4.620:15
sdakeno20:16
sdakekeep things as they are please20:16
openstackgerritChristian Berendt proposed openstack/kolla: Fix service_type of mistral endpoint  https://review.openstack.org/35212020:16
Mech422sbezverk: so the error occurs on the host side, for one machine - but not the other ?20:17
Mech422sbezverk: and the host AND vm's are correct on the other node ?20:18
openstackgerritChristian Berendt proposed openstack/kolla: Remove unused project_yaml parameter from role metadata files  https://review.openstack.org/35192820:18
Mech422sbezverk: or is it 1 physical box, and 1 VM is right, but not the other ?20:19
sbezverkmech422 it is 1 physical node and 1 VM ok 1 VM does not20:20
sbezverkbut it is not always the same VM20:20
Mech422sbezverk: and virsh dumpxml shows all VMs defined the same?20:20
Mech422(sometimes when I'm copying vm configs I forget to change the MAC address and end up with dupes, etc)20:21
sbezverkmech422 yep, I built all these manually20:21
sbezverkin case of config issue I would expect to see systematic failure20:22
sbezverkhere we see very random :-(20:22
Mech422sbezverk: eh - never hurts to start with the basics...20:22
sbezverksure sure20:22
Mech422sbezverk: I do enough stupid shit, not to be surprised anymore :-)20:22
Mech422sbezverk: like copying vms and ending up using the same backing store on 2 vms20:23
sbezverk:-) I use lvm volume per VM20:23
sbezverkI am 99.99% positive it is race condition20:23
sbezverkbecause when I start containers with sleep 8640020:24
Mech422sbezverk: me too20:24
sbezverkthen connect to each container and run my script manually it always works20:24
Mech422sbezverk: these are full VMs not containers right ?20:25
sbezverkcorrect20:25
Mech422sbezverk: so the race would probably be between VM20:25
sbezverkdo not think so20:25
sdakethe error is right there in the strace20:25
Mech422sbezverk: maybe at the disk or network layer20:25
sdakesocket is returning EAGAIN20:25
sdakeyet ovs is not trying socket again20:25
sbezverkI bet it is kubernetes initialization20:25
sdakeit just goes blindingly on its way20:25
sbezverksequence20:25
sdakesbezverk got a link to the openvswithc source code20:26
sbezverkMech422 If I manually restart ovsdb container everything stabilizes20:26
sdakesbezverk focus on me please :)20:27
sdakelets not rehash debugging that happened 4 days ago20:27
sbezverkhttps://github.com/openvswitch/ovs/tree/branch-2.520:27
sdakewhich process are you straccing20:28
sbezverkovsdb-server20:28
Mech422sbezverk: which one? I have two of them - the 'monitoring' one, and the 'real' one...20:29
Mech422sbezverk: oh - your stracing...manual start...nvm20:30
Mech422sbezverk: when you having a 'working' one up - does lsof -p FOO show anything unusual20:31
Mech422sbezverk: no unexpected dirs/mountpoints or devices ?20:32
sbezverkI did docker inspect on both correctly workign container and not and compare them, I could not find  anything abnormal20:34
*** Jeffrey4l has quit IRC20:35
*** Jeffrey4l has joined #openstack-kolla20:35
Mech422sbezverk: I don't know about k8s, but kolla likes to wipe the ovsdb when starting ovs...that hoses my host networking...20:35
Mech422sbezverk: if your doing container stuff - its not trying to reset your networking is it ?20:36
sbezverknope everything else works perfectly20:37
sdakeworking on solution - calm down guys :)20:38
*** bootsha has quit IRC20:43
Mech422sbezverk: sounds like its gotta be a race caused by some sort of config. issue - All-in-one setups have been beaten to death...if it didn't work in centos, I'd imagine there'd be stuff all over the net crying about it.  Anyway, I gotta get back to work...let me know how it turns out :-)20:47
sdakesbezverk quetion20:47
sbezverksure20:48
sdakeyou linked a strace prior - are you SURE there was no segfault associated with that failure to create the socket?20:49
sbezverksdake I do see some seg faults but I do not recognize processes20:50
sbezverk[23126.119755] traps: urcu6[20124] general protection ip:7fee826f0e37 sp:7fee7ec5b9f0 error:0 in libc-2.17.so[7fee826ba000+1b7000]20:50
sdakethat is an openvswitch process20:51
sbezverkI did a search in ovs git for this symbol urcu620:52
sbezverknothing comes up20:52
sdakehttp://openvswitch.org/pipermail/discuss/2015-December/019689.html20:52
sbezverkok20:53
sbezverkbut I do not see any trace generated in dmesg20:54
sbezverkas it was mentioned in that thread..20:55
sdakeya who knows why that crashes - probablybecause the socket isn't there20:55
sdakeadd a /dev:/dev bindmount20:55
sdakeand reproduce the problem with strace without a direct segfault of ovs-ddb20:55
sbezverkok20:56
*** daneyon has joined #openstack-kolla20:56
sbezverkshould I leave strace?20:57
sdakeyes pls20:59
sdakend reproduce the problem with strace without a direct segfault of ovs-ddb20:59
sdakemy bet is after you add the dev bindmount the problem will disappear20:59
sdakebut could be wrong20:59
sdakeisn't debugging fun ? :)20:59
sdakebrb switching networks21:00
*** egonzalez90 has joined #openstack-kolla21:00
*** daneyon has quit IRC21:00
*** sdake_ has joined #openstack-kolla21:01
*** sdake_ has quit IRC21:02
*** sdake_ has joined #openstack-kolla21:02
*** signed8bit is now known as signed8bit_Zzz21:03
*** signed8bit_Zzz is now known as signed8bit21:03
*** signed8bit is now known as signed8bit_Zzz21:03
*** sdake has quit IRC21:04
*** signed8bit_Zzz is now known as signed8bit21:04
*** signed8bit is now known as signed8bit_Zzz21:05
sdake_sbezverk let me know if you have a failure in 20-30 runs21:06
sbezverk%) 20-3021:06
sdake_also can you show me your current bindmounts21:07
sdake_(for that container)21:08
sbezverk           - mountPath: /var/lib/kolla/config_files21:09
sbezverk              name: openvswitch-db-config21:09
sbezverk              readOnly: true21:09
sbezverk            - mountPath: /var/lib/openvswitch21:09
sbezverk              name: openvswitch-db21:09
sbezverk            - mountPath: /run21:09
sbezverk              name: host-run21:09
sbezverk            - mountPath: /dev21:09
sbezverk              name: host-dev21:09
sbezverk            - mountPath: /etc/localtime21:09
sbezverk              name: host-etc-localtime21:09
sbezverk              readOnly: true21:09
sdake_try otu dev, if that fails, try otu /sys/fs/cgroups21:16
sdake_if that fails21:16
sdake_let me know21:17
sdake_(with a strace paste)21:17
*** sdake has joined #openstack-kolla21:21
*** ad_rien_1 has quit IRC21:23
*** ad_rien_ has joined #openstack-kolla21:24
*** sdake_ has quit IRC21:24
*** egonzalez90 has quit IRC21:25
sdakesbezverk any word?21:28
openstackgerritRyan Hallisey proposed openstack/kolla-kubernetes: Add an --all-in-one flag to the CLI  https://review.openstack.org/35213821:35
sbezverksdake: still working.. I see containers crashes but then it gets stabilized..21:36
sdakeso /dev:/dev fixes it21:37
sdakeor undecided?21:37
sbezverkI want to test without strace21:37
sdakedoes running without strace cause a crash that is fata lin nature?21:38
sbezverkwithout strace it was reproduced each time21:39
sdakecool give that a spin21:39
*** bootsha has joined #openstack-kolla21:41
*** sdake_ has joined #openstack-kolla21:50
*** sdake has quit IRC21:52
sbezverksdake_: man it works like a charm. now containers are not restarting21:53
sdake_yw21:53
sbezverk3 out of 3 were sucess21:53
sbezverkthank you21:53
sbezverkwhere did you get idea to add dev:21:53
sbezverkhave you noticed something in the code?21:54
sdake_pulled it out of my ass21:54
sdake_link the strace that failed again -  my backscroll is gone21:54
sdake_i'll show you why i suspected that may fix it21:55
sbezverk http://paste.openstack.org/show/551517/21:55
sdake_line 27421:56
sdake_nah wrong line21:56
sdake_line 27521:57
sdake_there i a makedev syscall21:57
sdake_i/is21:57
sdake_did you remove all the otehr hacks you have in place21:57
sdake_to certify that concretely fixes it21:58
sdake_if it fixes it - woudl appreciate a coauthor line ;)21:59
sbezverkdoing it right now22:00
sbezverkFYI I would still to use a script regardless22:01
sbezverkin order to be able to use DaemonSet for completely dynamic operation22:01
*** huikang has joined #openstack-kolla22:03
*** fragatina has joined #openstack-kolla22:08
Mech422[12:52] <Mech422> and /dev/null22:11
Mech422so it was a config error...nice22:11
sdake_sbezverk it would be nice if our code didnt have sleep 1s in it22:12
sdake_sloppy22:12
sdake_the only reason i +2'ed the chnge is to unblock you22:12
*** fragatina has quit IRC22:12
sdake_short term slop is ok in my book22:12
sdake_as long as it gets fixed short term :)22:12
Mech422I am curious how it worked in the other VMs without /dev bound to the containers though...22:14
sdake_Mech422 i'm cruious why it works in ansible without dev ;)22:15
Mech422sdake_: its really wierd... /dev missing isn't a race...why would it suddenly appear later?  or not need /dev later ?22:17
sdake_i dont claim to know the root cause22:18
Mech422sdake_: yeah - just very odd...oh well, working now :-)22:18
sdake_indeed it culd till be a race - that is why i want the sleep hacks removed22:18
Mech422sdake_: maybe 'building' /dev takes longer then mounting it 'pre-built' ?22:19
*** huikang has quit IRC22:24
*** huikang has joined #openstack-kolla22:25
Mech422sdake_: this docker bug talks about udev races: https://github.com/docker/docker/issues/403622:25
Mech422sdake_: I wonder if it might be related?22:25
Mech422udev not responding fast enough in the vm or something?22:26
sdake_dont know - dont care - watching tv :)22:26
Mech422sdake_: LOL - enjoy :-)22:26
Mech422sbezverk: so does centos even do the 'build /dev on boot' thing ? or is it a static /dev ?22:27
sbezverkMech422: looks like mounting dev on host22:27
sbezverkfixed the issue22:27
sbezverksdake: found some references in code suggesting that this mount might be required.22:28
sdake_enders game inc22:28
Mech422sbezverk: yeah - I'm just curious why it worked on ANY nodes if /dev is required...22:28
Mech422sdake_: oh - loved those books.. :-)22:29
*** huikang has quit IRC22:29
Mech422sdake_: my fav was the second (?) book... the one told from the brazillian kids point of view22:30
Mech422sdake_: his nemesis from the barrio was a cold blooded S.O.B.22:31
sdake_i like the movie22:31
sdake_i haven't read the books22:31
sdake_ender fails his team22:31
sdake_yet succeeds at the same time22:31
sdake_as oxymoronic as that is22:32
Mech422sdake_: I heard the movie was pretty good - but haven't seen it22:33
sdake_worth watching22:33
sdake_probably not as good as the book22:33
sdake_i spend 12 hours a day reading - thats enough for me22:33
Mech422sdake_: I'm waiting on Suicide Squad now...22:33
sdake_yup i want to see that one22:34
sdake_star trek was a let down22:34
sdake_bourne was just ok22:34
sdake_waiting or a good movie to come along22:34
sdake_something like enders game22:35
sdake_or 410 to yuma22:35
sdake_or no country for old men22:35
sdake_you know - a mdoern classic22:35
Mech422star trek let you down? bummer...I'm waiting for that to hit vudu22:35
sdake_well you know what they say about opinions :)22:35
Mech422the girl friend is a big jason stratham fan - so the new 'Mechanic' movie will be on our list too :-P22:35
sdake_sbezverk can you confirm you removed the sleep1 and had success22:36
sdake_watched wild card last night22:36
Mech422oh - thats an old one...22:36
sdake_if your in for a good tv episode, watch "Chain of Command" from star trek22:36
Mech422I still think Leon from 'The Professional' is one of the best bad-good-guys22:37
Mech422sdake_: oh - chain of command is ST:NG ? I'll have to check it out...plot summary sounds really good22:40
sdake_ya stng22:41
sdake_i read somewhere star trek is doing another run on tv22:42
Mech422oh? sweet - they seem to do a decent job with ST22:42
Mech422I haven't really been disappointed in any of them - for some reason, I really like the 'enterprise' series22:43
Mech422sometime I need to take a week off and watch babylon 5 from start to finish...22:43
Mech422I've never managed to see the whole thing22:43
*** ad_rien_ has quit IRC22:44
sdake_i was a super fan of stargate22:51
sdake_all of em22:51
sdake_except the last one which ended with an unresolved cliffhanger22:52
Mech422sdake_: stargate was good...but they ended up with so many spinoffs, I refused to get sucked it - that would be as long as babylon 5 to watch them all!22:56
*** huikang has joined #openstack-kolla23:04
*** sdake_ has quit IRC23:14
*** huikang has quit IRC23:25
*** daneyon has joined #openstack-kolla23:38
*** daneyon has quit IRC23:42
*** fragatina has joined #openstack-kolla23:48
*** zhurong has joined #openstack-kolla23:52
*** fragatina has quit IRC23:54
« Saturday, 2016-08-06 Index Monday, 2016-08-08 »

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!