Monday, 2018-12-17

« Sunday, 2018-12-16 Index Tuesday, 2018-12-18 »
*** oanson has quit IRC01:59
*** hongbin has joined #openstack-kuryr02:11
*** hongbin has quit IRC02:28
*** hongbin has joined #openstack-kuryr02:50
*** hongbin has quit IRC04:37
*** janki has joined #openstack-kuryr05:36
*** gcheresh_ has joined #openstack-kuryr06:35
*** gcheresh has joined #openstack-kuryr06:39
*** gcheresh_ has quit IRC06:39
*** ccamposr has joined #openstack-kuryr07:07
*** maysams has joined #openstack-kuryr07:23
*** yboaron_ has quit IRC07:28
*** oanson has joined #openstack-kuryr07:34
*** pcaruana has joined #openstack-kuryr08:18
*** yboaron_ has joined #openstack-kuryr08:29
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add needed driver and handler for policy gate  https://review.openstack.org/62460408:31
*** yboaron_ has quit IRC08:33
*** yboaron_ has joined #openstack-kuryr08:34
*** gkadam has joined #openstack-kuryr08:35
*** gkadam has quit IRC08:35
*** garyloug has joined #openstack-kuryr08:55
*** garyloug_ has joined #openstack-kuryr08:56
*** garyloug has quit IRC09:00
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Clean up Dockerfiles and how we build them  https://review.openstack.org/62532709:41
*** yboaron_ has quit IRC10:21
*** yboaron_ has joined #openstack-kuryr10:22
*** garyloug__ has joined #openstack-kuryr10:32
*** garyloug_ has quit IRC10:36
*** garyloug_ has joined #openstack-kuryr10:58
*** garyloug__ has quit IRC11:02
*** yboaron_ has quit IRC11:23
*** garyloug__ has joined #openstack-kuryr11:31
*** yboaron_ has joined #openstack-kuryr11:33
*** garyloug_ has quit IRC11:35
*** celebdor has joined #openstack-kuryr11:36
*** garyloug_ has joined #openstack-kuryr11:59
*** garyloug__ has quit IRC12:03
*** janki has quit IRC12:31
*** janki has joined #openstack-kuryr12:59
openstackgerritMaysa de Macedo Souza proposed openstack/kuryr-kubernetes master: [wip] Update CRD when NP has podSelectors  https://review.openstack.org/62558813:04
*** gcheresh has quit IRC13:26
*** gcheresh has joined #openstack-kuryr13:30
*** rh-jelabarre has joined #openstack-kuryr13:33
dulekdmellado, ltomasbo, yboaron_: Any experience with dstat?13:34
yboaron_dulek, Nope, sorry13:35
dulekI'm trying to debug those etcd issues.13:35
dulekAnd I see very high memory usage on the node.13:35
dulekAnd kuryr-daemon seems even worse: kuryr-daemon 1785G !?13:36
*** pcaruana has quit IRC13:50
ltomasbo1785 G????13:52
dulekMhm. This is super strange to me…13:53
dmelladodulek: not really, what are you trying to do?13:53
*** gcheresh_ has joined #openstack-kuryr13:55
*** gcheresh has quit IRC13:55
dulekOkay, so grab this folks: http://logs.openstack.org/27/625327/3/check/kuryr-kubernetes-tempest-daemon-openshift-octavia/cb22439/controller/logs/screen-dstat.txt.gz13:56
dulekThis is dstat with: "-tcmndrylpg --top-cpu-adv --top-io-adv --top-mem --swap --tcp"13:56
dulekWe want --top-cpu-adv --top-io-adv --top-mem --swap13:56
dulekSo take a look at the column third to last, it'll be related to memory.13:56
dulekWhat I see is kuryr-daemon starting with 1785G and going up to 2577G.13:57
dulekAnd as it's not killed by OOM it's either dstat bug or kuryr-daemon is requesting some kind of virtual memory that I'm not aware of?13:58
dulekI think celebdor knows more about how memory works in Linux system…13:58
dulekdmellado: ^13:58
*** garyloug_ has quit IRC14:00
*** pcaruana has joined #openstack-kuryr14:05
*** aojea_ has joined #openstack-kuryr14:16
*** gcheresh has joined #openstack-kuryr14:27
*** yboaron has joined #openstack-kuryr14:27
*** gcheresh_ has quit IRC14:27
*** yboaron_ has quit IRC14:27
*** kiennt26 has joined #openstack-kuryr14:30
*** garyloug has joined #openstack-kuryr14:34
*** yboaron_ has joined #openstack-kuryr14:36
*** yboaron has quit IRC14:39
*** gcheresh has quit IRC14:39
*** gcheresh_ has joined #openstack-kuryr14:39
celebdorsooo14:42
celebdorltomasbo: you sure that the handlers are executed strictly serially?14:43
ltomasbocelebdor, based on what maysams experienced, it seems so...14:43
ltomasbocelebdor, when one is raising exceptions, the other ones are not being executed... until it crashes14:44
maysamsyup14:44
celebdorltomasbo: maysams: you sure it is not just the handler that retries the max amount and marks unhealthy?14:44
maysamscelebdor: but if the vif handler was getting something14:45
maysamswe would see in the logs14:45
maysamscelebdor: and the number of enabled handlers gets to only one14:46
celebdormmm14:49
*** kiennt26 has quit IRC14:55
*** yboaron_ has quit IRC15:10
*** pcaruana has quit IRC15:29
*** pcaruana has joined #openstack-kuryr15:51
*** janki has quit IRC16:05
*** maysams has quit IRC16:06
openstackgerritMerged openstack/kuryr-kubernetes master: Trivial Fix: fix typos and set default param  https://review.openstack.org/62542016:11
*** gcheresh_ has quit IRC16:16
dulekdmellado, celebdor, ltomasbo: Would you believed that it's actually a dstat's bug?16:26
ltomasbolol16:27
*** pcaruana has quit IRC16:28
dmelladodulek: really? LOL16:31
dmelladodulek: so there wasn't any high consumption?16:32
dulekdmellado: I don't think so.16:32
dmelladolol, awesome, but that doesn't solve our issue either16:32
dulekdmellado: Basically dstat does Python's .split() on stuff from /proc/<pid>/stat.16:32
dulekBut in case of kuryr-daemon second field is (kuryr-daemon something).16:33
dulekSo it's reading from a wrong field.16:33
dmelladolol16:33
dulekdmellado: Well it's not. :(16:33
dmelladodulek: let's try this on a local install, and let's try to find something from that16:34
dmelladoI'm off to the airport but will ping you tomorrow to debug the issue16:34
dulekOkay, have a good flight!16:35
dmelladothanks!16:35
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure pools support don't break due to neutron-vif  https://review.openstack.org/62565516:42
*** ccamposr has quit IRC16:44
celebdordulek: really?17:01
celebdorI don't really believe it17:01
celebdorxD17:01
dulekcelebdor: Want the code?17:03
dulekcelebdor: https://github.com/dulek/dstat/blob/master/plugins/dstat_top_mem.py#L2817:04
dulekcelebdor: And yes, proc_splitline works exactly as you expect.17:04
dulekcelebdor: But when you look into `man proc` you'll see that second field from that file may have spaces.17:04
celebdorxD17:05
celebdordid you file an issue?17:05
dulekAnd kury-daemon has due to how cotyledon names the processes.17:05
dulekcelebdor: It's totally minor so I'll fix it in a moment.17:05
celebdorcool17:11
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Avoid raising ResourceNotReady exception at pod label handler  https://review.openstack.org/62566817:13
*** maysams has joined #openstack-kuryr17:34
*** garyloug has quit IRC17:40
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-kubernetes master: devstack: use a non-admin role user  https://review.openstack.org/62567717:56
*** irenab has joined #openstack-kuryr18:14
*** phuoc_ has quit IRC18:15
*** gcheresh_ has joined #openstack-kuryr18:58
*** gcheresh_ has quit IRC19:10
*** aojea has joined #openstack-kuryr20:05
*** aojea_ has quit IRC20:06
*** aojea has quit IRC20:42
*** celebdor has quit IRC20:43
*** celebdor has joined #openstack-kuryr20:45
*** yboaron has joined #openstack-kuryr20:49
*** celebdor has quit IRC20:51
*** dmellado has quit IRC21:05
*** celebdor has joined #openstack-kuryr21:45
*** ccamposr has joined #openstack-kuryr21:51
*** yboaron has quit IRC21:52
*** rh-jelabarre has quit IRC22:21
*** ccamposr has quit IRC22:41
*** maysams has quit IRC22:54
*** celebdor has quit IRC23:26
*** celebdor has joined #openstack-kuryr23:41
openstackgerritMerged openstack/kuryr-kubernetes master: Add needed driver and handler for policy gate  https://review.openstack.org/62460423:46
*** celebdor has quit IRC23:55
« Sunday, 2018-12-16 Index Tuesday, 2018-12-18 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!