Tuesday, 2017-11-07

« Monday, 2017-11-06 Index Wednesday, 2017-11-08 »
*** nhlfr has joined #openstack-kuryr00:01
*** gouthamr has quit IRC00:14
*** gouthamr has joined #openstack-kuryr00:20
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements  https://review.openstack.org/51693600:24
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements  https://review.openstack.org/51693700:25
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978100:25
*** salv-orlando has joined #openstack-kuryr00:32
*** salv-orlando has quit IRC00:36
*** caowei has joined #openstack-kuryr01:25
*** gouthamr has quit IRC01:30
*** salv-orlando has joined #openstack-kuryr01:32
*** kiennt26 has joined #openstack-kuryr01:35
*** salv-orlando has quit IRC01:38
*** gouthamr has joined #openstack-kuryr02:19
*** salv-orlando has joined #openstack-kuryr02:33
*** salv-orlando has quit IRC02:38
*** openstack has joined #openstack-kuryr02:43
*** ChanServ sets mode: +o openstack02:43
*** gouthamr has quit IRC02:50
*** gouthamr has joined #openstack-kuryr03:26
*** reedip has joined #openstack-kuryr03:49
*** kiennt26 has quit IRC04:24
*** gouthamr has quit IRC04:35
*** salv-orlando has joined #openstack-kuryr04:35
openstackgerritVu Cong Tuan proposed openstack/fuxi master: Do not use “-y” for package install  https://review.openstack.org/51821904:36
*** gouthamr has joined #openstack-kuryr04:40
*** salv-orlando has quit IRC04:40
*** yamamoto has joined #openstack-kuryr04:47
*** caowei has quit IRC05:03
*** janki has joined #openstack-kuryr05:17
*** caowei has joined #openstack-kuryr05:32
*** salv-orlando has joined #openstack-kuryr05:36
*** salv-orlando has quit IRC05:40
openstackgerritXieYingYun proposed openstack/kuryr master: Optimize the link address in docs  https://review.openstack.org/51823306:02
*** yboaron has joined #openstack-kuryr06:12
*** yboaron has quit IRC06:19
*** yboaron has joined #openstack-kuryr06:23
*** salv-orlando has joined #openstack-kuryr06:35
openstackgerritMerged openstack/kuryr-kubernetes master: Add icmp sg rules to k8s project  https://review.openstack.org/51535706:48
*** gouthamr has quit IRC06:54
gsagiesalv-orlando: indeed it is..07:11
*** gsagie has quit IRC07:23
*** pcaruana has joined #openstack-kuryr08:06
*** yboaron has quit IRC08:12
*** salv-orlando has quit IRC08:45
*** salv-orlando has joined #openstack-kuryr08:46
*** salv-orlando has quit IRC08:51
*** yamamoto has quit IRC09:02
irenabdulek, will check asap09:02
*** yamamoto has joined #openstack-kuryr09:05
*** yboaron has joined #openstack-kuryr09:06
openstackgerritMerged openstack/kuryr-kubernetes master: Remove 99-loopback.conf  https://review.openstack.org/51057109:07
*** garyloug has joined #openstack-kuryr09:07
*** garyloug_ has joined #openstack-kuryr09:08
*** garyloug has quit IRC09:08
*** yamamoto has quit IRC09:09
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI Daemon documentation  https://review.openstack.org/50938009:12
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: DNM: Trying to run gate tests with CNI Daemon  https://review.openstack.org/50976509:15
*** salv-orlando has joined #openstack-kuryr09:23
ltomasbodulek, http://paste.openstack.org/show/625684/09:25
ltomasboI'm testing your cni-split containerized and got that (just in case it helps)09:25
dulekltomasbo: All of the pods get that state? You should look in CNI container docs (it's running the daemon now).09:26
duleks/docs/logs09:26
ltomasbodulek, not all of the pods09:26
ltomasbowhen I scale and create a bunch of them at the same time09:27
ltomasbosome of them get it09:27
ltomasbomost of the time it works...09:27
dulekltomasbo: Okay, so I'd expect you'll see EEXIST in the CNI daemon logs.09:27
dulekltomasbo: I should get that into a try-except, irenab asked me to do that in another patch.09:28
dulekI guess I should do the patch now.09:28
ltomasbook, seems I cannot access the logs...09:30
ltomasbodulek, it is not like this: $ kubectl -n kube-system logs kuryr-cni-ds-v91jm09:31
ltomasbo?09:31
dulekYup, that should be it.09:32
dulekEmpty?09:32
ltomasboyep, nothing there09:33
dulekHm, strange. That would mean daemon is not running. But it is, as you're getting some of your pods up and running.09:34
ltomasboand it annoys me a bit to see that many kuryr-daemon server workers... even when I delete the pods...09:34
dulekltomasbo: sudo systemctl status devstack@kuryr-daemon09:35
dulekMaybe it had started in a process?09:35
ltomasbothere is no kuryr-daemon...09:36
ltomasbono systemd I mean09:36
ltomasbokubectl get -n kube-system pod09:36
ltomasboNAME                                READY     STATUS    RESTARTS   AGE09:36
ltomasbokuryr-cni-ds-v91jm                  1/1       Running   0          15m09:36
ltomasbokuryr-controller-3218936350-d2kth   1/1       Running   0          38m09:36
dulekAnd you're testing latest version of the pathc?09:36
ltomasboumm, let me see09:37
ltomasbothere is only 1 patch set, right?09:37
ltomasbohttps://review.openstack.org/#/c/518024/09:37
dulekOkay, I wasn't sure if I haven't sent two, sorry.09:39
dulekYou have option setting containerized set to True and enable_service kuryr-daemon, right?09:39
ltomasboyep09:40
ltomasboand I see it is running containerized09:40
ltomasboand the kuryr-daemon processes09:40
dulekOkay, so I'll try to run that myself and see what happens.09:40
ltomasbo ps -eF | grep daem | grep kury09:41
ltomasboroot      6351  6326  0 66268 50856   4 09:21 ?        00:00:00 kuryr-daemon: master process [/usr/bin/kuryr-daemon --config-file /etc/kuryr/kuryr.conf]09:41
ltomasboroot      6364  6351  0 105183 48484  6 09:21 ?        00:00:00 kuryr-daemon: watcher worker(0)09:41
ltomasboroot      6367  6351  0 1233788 98204 4 09:21 ?        00:00:08 kuryr-daemon: server worker(0)09:41
*** yamamoto has joined #openstack-kuryr09:46
*** gouthamr has joined #openstack-kuryr09:52
dulekltomasbo: http://paste.openstack.org/show/625694/10:04
dulekkss is my alias to kubectl -n kube-system10:04
ltomasboumm10:05
ltomasboso it works for you...10:05
ltomasboI did kill the cni container, to see if that was recreated10:05
ltomasboand the leftover processes were deleted10:05
ltomasbodulek, can you try that and check if you can access the logs of the new cni container?10:06
dulekSure thing.10:06
dulekltomasbo: Processes are gone now…, daemon started… aaand I have the logs.10:07
ltomasboumm10:07
ltomasbook, I'll re-stack10:07
ltomasboonly modification I did is to use lbaasv2 instead of octavia10:08
dulekltomasbo: I'm running without Octavia as well.10:09
ltomasbodulek, umm, centos?10:09
dulekltomasbo: Yup, I'll send you my local.conf10:09
dulekhttp://paste.openstack.org/show/625700/10:11
ltomasbodulek, this is mine: http://paste.openstack.org/show/625699/10:11
dulekBut please note that I also did got the errors you've mentioned originally, one pod failed.10:11
ltomasboumm10:12
ltomasbook10:12
dulekltomasbo: Hm, it looks pretty much the same as mine.10:12
ltomasboseems the only different is you enable debug10:12
ltomasboand I force ovs-firewall10:12
dulekI'll add that try-except.10:12
ltomasboother than that, pretty much the same10:13
ltomasbook, thanks!10:13
dulekYup, but you should still see INFO logs even without debug.10:13
ltomasboI was eager to test this! but no hurry!10:13
*** caowei has quit IRC10:13
ltomasbodulek, yep, I know, it should have the info logs10:13
ltomasboI'm restacking10:13
ltomasbolet see if now it works...10:13
*** jchhatbar has joined #openstack-kuryr10:26
*** janki has quit IRC10:26
*** alraddarla has joined #openstack-kuryr10:26
*** yamamoto has quit IRC10:27
*** salv-orlando has quit IRC10:27
*** yamamoto has joined #openstack-kuryr10:27
*** yamamoto has quit IRC10:33
*** janki has joined #openstack-kuryr10:34
*** jchhatbar has quit IRC10:35
irenabdulek, I verified the CNI split patch, you got my +210:41
dulekirenab: Thanks!10:42
irenabNot directly related, but triggered by this patch. I am a bit concerned about number of various options how to run the kuryr. I think we should converge on limited number and have proper documentation of what are the options10:44
dulekirenab: I agree and I guess this is the outcome of the fact we're still experimenting a bit.10:47
irenabdulek, totally agree. I think we just should keep the awareness and deprecate as we go the previous not used flavours10:48
dulekirenab: Yup. Currently we have 2x2 matrix - containerized or not and with daemon and without.10:49
dulekirenab: I believe first to get deprecated is running without daemon.10:49
dulekThen containerization is really a deployment thing, so I don't think we'll be able to deprecate running kuryr services on bare metal.10:49
irenab+110:49
*** yamamoto has joined #openstack-kuryr11:12
ltomasboI agree on that too!11:12
ltomasbodulek, I remove the VM and create a new one11:13
ltomasboand I'm getting the same11:13
ltomasbono access to the log with kubectl11:13
ltomasbobut I can see them with docker logs CNI_ID11:13
ltomasbostrange11:13
ltomasbodulek, same for the controller container11:14
dulekHm… I'd say it's your env's fault?11:14
ltomasboumm, could be, but I remove everything (even the VM) and started from scratch11:15
dulekltomasbo: Oh, so definitely not your fault…11:15
dulekltomasbo: BTW I think I've understood the EEXIST/KeyError issue, I'm testing the patch currently.11:15
ltomasbodulek, I agree it is probably not related to your patch either11:15
ltomasbodulek, you are fast!!!!11:16
dulekI guess I've had enough coffee today, I remember spending hours on that with apuimedo. :P11:16
dulek(not coffee, the bug)11:16
dulekltomasbo: So controller pod has no logs as well?11:17
ltomasbowell, it has, but it is not accesible through kubectl, only from docker11:17
dulekltomasbo: How about checking the kube-apiserver logs and how it responds to kubectl logs?11:18
dulekltomasbo: Oh, even better - just spawn any pod and see if it has logs. :)11:19
ltomasbook11:20
ltomasbodulek, same problem...11:22
ltomasbo kubectl logs demo-2293951457-bmxbm11:22
ltomasbo^C11:22
ltomasbo[stack@gerrit-518024vm-0 devstack]$ docker logs 9b340201edea11:22
ltomasbo::ffff:10.0.0.70 - - [07/Nov/2017 11:21:51] "GET / HTTP/1.1" 200 -11:22
dulekSo I guess it's caused by something on your VM.11:26
*** salv-orlando has joined #openstack-kuryr11:27
ltomasbodulek, umm, probably11:27
*** yamamoto has quit IRC11:31
*** salv-orlando has quit IRC11:32
*** yamamoto has joined #openstack-kuryr11:43
*** yamamoto has quit IRC11:58
irenabdulek, do you have a min?12:00
dulekirenab: min?12:01
irenabminute12:01
irenabquestion regarding http://logs.openstack.org/80/509380/13/check/build-openstack-sphinx-docs/d8f7286/html/devref/kuryr_kubernetes_design.html#communication12:02
*** janonymous has joined #openstack-kuryr12:02
irenabin the diagram it looks that there is a watch started upon daemon start and then there is an AddNetwork request to deal with specific Pod request. from the diagram it is not clear what is in the scope of the watch event handling12:05
irenabI am not sure it can be properly clarified in the diagram, but worth to add in the description12:06
dulekirenab: Hm, that's right, I would need a third time scale probably.12:06
dulekirenab: Basically now with daemon it's a single watcher that looks for pods on host it's running on.12:06
dulekirenab: Maybe it should even be pictured as another entity?12:06
irenabwatch?12:07
dulekirenab: You're asking about watcher?12:08
irenabWhat do you mean by 'pictured as another entity'?12:08
dulekAs another vertical timeline.12:09
* dulek forgot the names of UML diagram elements.12:09
irenabwhat do you mean by 'it'?12:09
irenabthe question is not regarding UML :-)12:09
dulekOh, sorry. The Watcher, yes.12:09
irenabI think the vertical is 'time line'12:10
dulekirenab: So right now Daemon is 2 processes - one is serving requests and second one is Watcher process that monitors k8s API for incoming VIF annotation.12:10
dulekProbably it'll be best to split Daemon into two time lines then. I'll update the patch.12:11
irenabWhat is missing in the diagram is split of responsibilities between these two12:11
irenabcan be iether clarified in the description or maybe by adding Watcher as another Actor in the diagram12:12
irenabdulek, this can be great. thanks!12:12
dulekOkay, I'll add another actor and description.12:12
*** janki has quit IRC12:22
*** salv-orlando has joined #openstack-kuryr12:29
*** salv-orlando has quit IRC12:31
*** salv-orl_ has joined #openstack-kuryr12:31
*** yamamoto has joined #openstack-kuryr12:34
*** salv-orl_ has quit IRC12:40
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI Daemon documentation  https://review.openstack.org/50938012:47
*** gouthamr has quit IRC12:50
*** pcaruana has quit IRC13:22
*** garyloug_ has quit IRC13:34
openstackgerritDanil Golov proposed openstack/kuryr-kubernetes master: Add VIF-Handler And Drivers Design approach  https://review.openstack.org/51371513:53
*** yamamoto has quit IRC13:56
*** yamamoto has joined #openstack-kuryr14:10
*** yamamoto has quit IRC14:15
*** janonymous has quit IRC14:21
ltomasboping dulek14:39
dulekltomasbo: pong. I'm preparing late lunch, so I may answer slowly.14:43
ltomasbodulek, no hurry! go to have lunch!14:43
ltomasbodulek, it was just a question about the containerized deployment, where should I look (inside the container) for the logs (kuryr-kubernetes.log)?14:43
dulekltomasbo: I think those go to stdout, so docker manages them.14:44
dulekltomasbo: If we're talking about controller logs.14:44
ltomasboyep, controller logs14:45
ltomasboI want to create a readiness check based on the controller log info14:45
dulekltomasbo: Aaah. That might be a bit difficult.14:45
ltomasbothat what I'm seeing...14:45
ltomasboanyway, go to have lunch! we can discuss it later!14:46
dulekltomasbo: Why don't you use your port pool file socket to ask the service directly?14:46
*** pcaruana has joined #openstack-kuryr14:47
ltomasbomainly because we agreed on that being just a devstack kind of tool, and we need to improve that if we are going to use it for more robust deployments14:48
ltomasbobut, unless we figure it out a different way, it may be the time to improve that... and use it14:49
dulekltomasbo: Okay, then it may be good to think of something else…15:00
dulekltomasbo: What log message you wanted to look for?15:01
ltomasboyep, I have an idea about how to fix it15:01
ltomasboI'm looking at what point in time the ports are loaded into the pools15:01
dulekltomasbo: Ah, that's readiness probe.15:02
ltomasboso, I will create a file once that is done, and use that as rediness check15:02
dulekltomasbo: I wanted to write exactly that. :D15:02
ltomasbo:D15:03
*** garyloug_ has joined #openstack-kuryr15:09
*** yamamoto has joined #openstack-kuryr15:13
*** jdavis has joined #openstack-kuryr15:25
*** yamamoto has quit IRC15:26
*** yamamoto has joined #openstack-kuryr15:55
*** janki has joined #openstack-kuryr15:56
*** yamamoto has quit IRC16:06
*** salv-orlando has joined #openstack-kuryr16:20
dulekltomasbo: Want to do a sanity check for my idea of KeyError?16:34
dulekltomasbo: I've spend another bunch of hours trying to debug that and again I have an Idea.16:34
dulekltomasbo: So line https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/cni/binding/bridge.py#L39 sometimes explodes with KeyError.16:35
ltomasbotell me16:35
ltomasboumm16:35
ltomasboand do you know why?16:35
*** yamamoto has joined #openstack-kuryr16:35
ltomasbonot yet into the namespace?16:36
dulekltomasbo: My hypothesis is that movement of h_iface to host netns done in line 37 sometimes isn't noticed by IPDB soon enough.16:36
dulekYup. IPDB checks for changes periodically. It might happen that the object haven't yet caught up with changes.16:36
dulekAt least I think this makes sense. :P16:36
*** yboaron has quit IRC16:37
ltomasbodulek, sounds plausible...16:37
ltomasboso, what do you propose?16:39
ltomasbore-try?16:39
dulekltomasbo: Yeah, so first step would be https://review.openstack.org/#/c/517406/ - to remove IPDB object caching and prevent process leaks.16:40
dulekltomasbo: Oh, this will solve the problem completely. :D16:40
ltomasboxD16:41
dulekltomasbo: Just here: https://review.openstack.org/#/c/517406/2/kuryr_kubernetes/cni/binding/bridge.py I'd need to split the with that's there.16:41
*** yamamoto has quit IRC16:41
dulekltomasbo: That way I can be sure that IPDB will get created after the changes got committed.16:41
ltomasboyou mean the with at L37?16:42
ltomasbounindent it?16:42
dulekltomasbo: Line 26, h_ipdb isn't used until line 37.16:43
dulekltomasbo: And after line 37 c_ipdb isn't used.16:43
ltomasboso, it the current version (left side)16:43
ltomasbowill it work just moving L27 to L38?16:43
dulekI can split that to do a c_ipdb context manager first and then h_ipdb. Completely separated.16:43
ltomasboor here https://github.com/openstack/kuryr-kubernetes/blob/master/kuryr_kubernetes/cni/binding/bridge.py#L39, just move L27 to right before L39?16:44
dulekltomasbo: That would work only if caching isn't enabled.16:45
ltomasbosure16:45
dulekltomasbo: Otherwise in kuryr-daemon situation the host IPDB will be cached.16:45
dulekAnd I hate this IPDB class. Kuryr code assumes too much about it. I wonder if we shouldn't move to something that is doing full reload before each operation.16:45
dulekOkay, I need to leave for some time, but I'll probably find some time today to fix this stuff up.16:46
ltomasbogreat! thanks!16:46
ltomasboI'll give it another try tomorrow then!16:46
*** janki has quit IRC17:30
*** janki has joined #openstack-kuryr17:31
*** janki has quit IRC17:32
*** jdavis has quit IRC17:35
*** aojea has joined #openstack-kuryr19:15
*** aojea has quit IRC19:34
*** aojea has joined #openstack-kuryr19:35
*** aojea has quit IRC19:39
dmelladoheh, being in Sydney is reading a digest of the channel xD19:58
*** aojea has joined #openstack-kuryr20:08
*** aojea has quit IRC21:33
*** aojea has joined #openstack-kuryr21:47
*** livelace-link has quit IRC21:52
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Fix kubelet retries issues  https://review.openstack.org/51840422:06
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Prevent pyroute2.IPDB threads leaking  https://review.openstack.org/51740622:13
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon  https://review.openstack.org/51518622:13
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: Support kuryr-daemon when running containerized  https://review.openstack.org/51802422:13
*** gouthamr has joined #openstack-kuryr22:16
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: DNM: Trying to run gate tests with CNI Daemon  https://review.openstack.org/50976522:18
*** aojea has quit IRC22:52
*** salv-orlando has quit IRC22:59
*** salv-orlando has joined #openstack-kuryr23:00
*** salv-orlando has quit IRC23:04
« Monday, 2017-11-06 Index Wednesday, 2017-11-08 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!