Wednesday, 2018-06-06

« Tuesday, 2018-06-05 Index Thursday, 2018-06-07 »
*** yamamoto has joined #openstack-kuryr00:03
*** yamamoto has quit IRC00:09
*** pliu_ has joined #openstack-kuryr00:29
*** pliu has quit IRC00:30
*** pliu_ has quit IRC00:43
*** pliu has joined #openstack-kuryr00:46
*** yamamoto has joined #openstack-kuryr01:06
*** yamamoto has quit IRC01:11
*** threestrands_ has joined #openstack-kuryr01:33
*** threestrands has quit IRC01:36
*** hongbin has joined #openstack-kuryr01:48
*** Adam____ has joined #openstack-kuryr01:49
*** yamamoto has joined #openstack-kuryr02:07
*** yamamoto has quit IRC02:12
*** s1061123 has quit IRC02:13
*** s1061123 has joined #openstack-kuryr02:15
*** s1061123 has quit IRC02:17
*** s1061123 has joined #openstack-kuryr02:17
Adam____pliu: Did you solve this problem?02:34
Adam____"No 'kuryr_kubernetes.vif_translators' driver found, looking for u'bridge'" from kuryr-controller02:35
Adam____I saw your question on 2018-04-1202:36
*** yamamoto has joined #openstack-kuryr03:08
*** yamamoto has quit IRC03:14
*** Adam____ has quit IRC03:46
*** yamamoto has joined #openstack-kuryr04:10
*** hongbin has quit IRC04:13
*** yamamoto has quit IRC04:16
*** yamamoto has joined #openstack-kuryr04:56
*** Adam_____ has joined #openstack-kuryr05:39
*** yboaron has joined #openstack-kuryr05:43
*** gcheresh has joined #openstack-kuryr06:04
*** Adam_____ has quit IRC06:20
*** pcaruana has joined #openstack-kuryr06:44
*** janki has joined #openstack-kuryr06:55
*** celebdor1 has joined #openstack-kuryr07:19
*** openstackgerrit has quit IRC07:19
*** openstackgerrit has joined #openstack-kuryr07:24
openstackgerritYossi Boaron proposed openstack/kuryr-kubernetes master: Service: Decoupling and improvements of LBaaS driver  https://review.openstack.org/56617507:24
openstackgerritYossi Boaron proposed openstack/kuryr-kubernetes master: OCP-Router: add support for external Ingress Controller's LB creation  https://review.openstack.org/56405107:24
openstackgerritYossi Boaron proposed openstack/kuryr-kubernetes master: OCP-Router: Ingress controller support  https://review.openstack.org/53638707:24
*** celebdor1 has quit IRC07:25
openstackgerritYossi Boaron proposed openstack/kuryr-kubernetes master: OCP-Router: OCP-Route and Ingress LBaaS handlers  https://review.openstack.org/53857707:25
*** yboaron has quit IRC07:59
*** threestrands_ has quit IRC08:03
*** celebdor1 has joined #openstack-kuryr08:05
*** garyloug has joined #openstack-kuryr08:08
*** garyloug_ has joined #openstack-kuryr08:09
*** garyloug has quit IRC08:09
*** NatanBro has joined #openstack-kuryr08:25
*** lxkong has quit IRC08:29
celebdor1ltomasbo: why does this patch have this change https://review.openstack.org/#/c/564148/14/kuryr_kubernetes/controller/drivers/base.py ?08:34
celebdor1it seems unrelated08:34
celebdor1is it something I'll realize after looking at the entire patchset08:34
celebdor1?08:34
dmelladoProbably xD08:39
dmelladocelebdor1 and folks, btw yesterday I spoke with the foundation folks again about that we will *not* going as a project to the S PTG08:40
dmelladoWe'll organize some VTG instead08:40
dmelladoSo I'll start looking for topics on the session when the time comes08:40
dmellado+ Deep Dive08:40
celebdor1dmellado: ok08:43
ltomasbocelebdor1, checking...09:08
ltomasbocelebdor1, I needed for the multipool support09:09
ltomasbocelebdor1, it is used here at the end (delete_network_pools method)  https://review.openstack.org/#/c/564148/14/kuryr_kubernetes/controller/drivers/vif_pool.py09:10
ltomasbodmellado, when and where will it be hosted?09:11
celebdor1ok09:12
dmelladoltomasbo that's to be decided09:12
ltomasbodmellado, ok09:12
dmelladoWe'll tackle that on next kuryr meeting09:12
ltomasbocelebdor1, basically to skip the method for the case when there is no pools09:12
ltomasbocelebdor1, and to have it nicer comparison there, instead of having to get the name from the class there09:13
openstackgerritchenge proposed openstack/kuryr-kubernetes master: Amend the spelling error of a word  https://review.openstack.org/57271009:24
*** yboaron has joined #openstack-kuryr09:30
celebdor1ok'=09:32
*** maysamacedos has joined #openstack-kuryr10:01
*** yboaron_ has joined #openstack-kuryr10:01
*** yboaron has quit IRC10:04
*** rh-jelabarre has joined #openstack-kuryr10:20
*** janonymous has joined #openstack-kuryr10:20
dulekdmellado: I've pointed out failure in multinode patch in the comments.10:40
*** lxkong has joined #openstack-kuryr10:45
*** yboaron_ has quit IRC11:02
dulekcelebdor1, irenab, dmellado, danil: http://lists.openstack.org/pipermail/openstack-dev/2018-June/131208.html11:11
*** maysamacedos has quit IRC11:15
openstackgerritGenadi Chereshnya proposed openstack/kuryr-tempest-plugin master: Testing VM to loadBalancer service connectivity  https://review.openstack.org/57196811:24
*** yboaron_ has joined #openstack-kuryr11:28
ltomasbodulek, I think it is a good idea!11:28
celebdor1dulek: I tend to agree. But I'll need to study it deeper11:34
*** maysamacedos has joined #openstack-kuryr11:36
yboaron_gcheresh, ping11:48
ltomasboyboaron_, I tested your OCP patch sets11:48
ltomasboyboaron_, it is working for me!11:49
yboaron_ltomasbo, and ...11:49
yboaron_ltomasbo, great! , thanks!11:49
ltomasboyboaron_, I'll try to review it asap11:49
yboaron_ltomasbo, 10x11:49
yboaron_celebdor1, ping11:49
yboaron_celebdor1, I think that I found the root cause for the random failures of Octavia gates @Kuryr,11:50
yboaron_celebdor1, it's related to the services IP pool separation (Octavia and Openshift/K8S).11:51
yboaron_celebdor1, see http://logs.openstack.org/77/538577/14/check/kuryr-kubernetes-tempest-octavia/e19bb9c/controller/logs/screen-kuryr-kubernetes.txt.gz#_Jun_06_08_01_24_95010011:51
yboaron_celebdor1, we need to merge your patch - https://review.openstack.org/#/c/562067/ ASAP11:52
gchereshyboaron_: hey11:57
celebdor1yboaron_: oh damn12:21
celebdor1that's probably right12:21
celebdor1yboaron_: do you have time to rebase my patch?12:21
celebdor1I need to prepare a meeting now12:21
celebdor1otherwise I'll get to it later today12:24
yboaron_celebdor1, Yes, I'll take care of it12:27
celebdor1thanks!12:30
openstackgerritYossi Boaron proposed openstack/kuryr-kubernetes master: devstack: fix octavia allocation ranges  https://review.openstack.org/56206712:44
dulekltomasbo: Regarding https://review.openstack.org/#/c/572344/, do you know exactly what exception is raised in case of port not being deleted yet from the network?12:45
dulekltomasbo: Maybe we can narrow "except n_exc.NeutronClientException" ?12:45
*** atoth has joined #openstack-kuryr12:46
openstackgerritYossi Boaron proposed openstack/kuryr-kubernetes master: devstack: fix octavia allocation ranges  https://review.openstack.org/56206712:49
*** hongbin has joined #openstack-kuryr12:57
*** NatanBro has quit IRC13:09
ltomasbodulek, I don't, but I can dig into it for sure!13:24
dulekltomasbo: I'd expect HTTP conflict, but I'm not really sure.13:34
*** janonymous has quit IRC13:39
ltomasbodulek, I'll update it asap (as soon as I find out which one it is!)13:40
ltomasbodulek, most probably NetworkInUseClient13:43
dulekHa, which is Conflict in the end. ;)13:44
*** yboaron_ has quit IRC13:44
dulekltomasbo: Though: https://github.com/openstack/python-neutronclient/search?utf8=%E2%9C%93&q=NetworkInUseClient&type=13:44
dulekSeems not used.13:44
ltomasboyep13:45
ltomasboIḿ getting exception.CommandError it seems13:45
ltomasboRESP BODY: {"NeutronError": {"message": "Unable to complete operation on network 9c887c3c-8020-497b-b2f6-bad2326a1a22. There are one or more ports still in use on the network.", "type": "NetworkInUse", "detail": ""}}13:46
dulekltomasbo: status code?13:46
ltomasbo40913:46
ltomasboRESP: [409] Connection: keep-alive Content-Length: 205 Content-Type: application/json Date: Wed, 06 Jun 2018 13:45:06 GMT X-Openstack-Request-Id: req-2c7bb27d-94d7-4b6d-9b0f-354f266bc4fd13:47
dulekltomasbo: Ah, but this one is CLI, maybe from the API it's different?13:47
ltomasboyep, could be13:47
ltomasboAPI says return codes are: Error response codes: 401, 404, 409, 41213:48
ltomasboError Response Codes: 400 Bad Request, 401 Unauthorized, 403 Forbidden, 404 Not Found, 409 Network in Use.13:49
dulekltomasbo: Okay, so Conflict should work in the code.13:50
dulekltomasbo: Or conversion from Conflict to NetworkInUse is done somewhere else.13:50
ltomasbodulek, http://codesearch.openstack.org/?q=NetworkInUse&i=nope&files=&repos=13:54
ltomasboso, HTTPConflict13:55
dulekWhy exactly? I'm not sure here.13:56
ltomasboumm, I look at the wrong one (nova)14:03
ltomasboso I again think it is NetworkInUseClient, as the map is from NetworkInUse (in neutron) to exceptions.NetworkInUseClient at python-neutronclient14:03
ltomasbodulek, ^^14:04
dulekOkay, makes sense.14:05
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Retry namespace deletion to mitigate cascading race  https://review.openstack.org/57234414:09
*** kiennt26 has joined #openstack-kuryr14:14
ltomasbodulek, I have updated it, but I will test it to be completely sure14:19
*** kiennt26 has quit IRC14:23
openstackgerritDanil Golov proposed openstack/kuryr-kubernetes master: Add SR-IOV capabilities to VIF handler  https://review.openstack.org/51228014:30
openstackgerritDanil Golov proposed openstack/kuryr-kubernetes master: Add SR-IOV binding driver to CNI  https://review.openstack.org/51228114:30
openstackgerritDanil Golov proposed openstack/kuryr-kubernetes master: Add multi-vif driver to kuryr-k8s controller  https://review.openstack.org/52459014:30
openstackgerritDanil Golov proposed openstack/kuryr-kubernetes master: Allow requesting additional subnets via annotation  https://review.openstack.org/52459114:30
openstackgerritDanil Golov proposed openstack/kuryr-kubernetes master: Allow setting specific ports for SRIOV handler  https://review.openstack.org/52459214:30
ltomasbodulek, yep, that was the one: 2018-06-06 14:42:56.300 1 ERROR kuryr_kubernetes.controller.drivers.namespace_subnet [-] One or more ports in use on the network 51025309-8053-47dd-b860-5a46ffd36834.: NetworkInUseClient: Un14:45
ltomasboable to complete operation on network 51025309-8053-47dd-b860-5a46ffd36834. There are one or more ports still in use on the network.14:45
dulekltomasbo: :)14:46
*** gcheresh has quit IRC15:03
*** maysamacedos has quit IRC15:09
*** lxkong has quit IRC15:12
*** kiennt26 has joined #openstack-kuryr15:14
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Retry namespace deletion to mitigate cascading race  https://review.openstack.org/57234415:14
ltomasbodulek, celebdor1 ^^ I forgot to update the unittest, should be fine now15:14
celebdor1*shame*15:15
celebdor1ltomasbo: ok, approved and w+115:16
celebdor1let's see how the flaky gate goes15:16
ltomasboxD15:16
*** maysamacedos has joined #openstack-kuryr15:22
*** janki has quit IRC15:22
*** pcaruana has quit IRC15:23
dulekOh crap.15:24
dulekcelebdor1, ltomasbo, dmellado: Had anyone tried doing some scale testing lately?15:24
ltomasbodulek, nop, why?15:25
dulekI don't have strongest setup ever (two small VMs), but when I'm starting ~20 pods, my kuryr-controller gets killed due to non-responding healthchecks.15:25
ltomasbodulek, using pools?15:26
ltomasboand if devstack, port_debug = false?15:26
ltomasbobut probably we should update the healthchecks intervals too15:27
dulekNah, no pools. Maybe it's due to that.15:27
ltomasbocould be, without pools, load on neutron side is big, and booting 20 containers (for a small/slow env) was already too much15:27
celebdor1dulek: ltomasbo: it's also true that the healthcheck serving threads do not have any sort of priority to them15:32
celebdor1so they could be starved as well15:32
ltomasboyep15:37
dulekcelebdor1: I don't see an obvious way to set process priority in Werkzeug (Flask's underlying lib).15:37
*** kiennt26 has quit IRC15:38
*** irenab has quit IRC15:44
celebdor1dulek: wouldn't it be for the whole green thread?15:46
*** irenab has joined #openstack-kuryr15:47
dulekcelebdor1: I think Werkzeug will spawn a process for each request. On the other hand I'm not sure where's the request-scheduler process started.15:48
*** janki has joined #openstack-kuryr15:49
*** phuoc_ has quit IRC15:53
*** phuoc_ has joined #openstack-kuryr15:54
celebdor1dulek: do we know what's the timeout for the healtcheck requests?15:55
dulekcelebdor1: Hm, seems like it's not set for livenessProbe.15:57
dulekHah, only 1 seconds by default.15:58
dulekThat's pretty bad.15:58
celebdor1dulek: yup16:00
celebdor1sounds like we need to file an issue to openshift-ansible16:00
celebdor1if Neutron is under load...16:00
*** garyloug_ has quit IRC16:04
*** hongbin has quit IRC16:10
dulekcelebdor1: Uhm, yeah, we're missing that setting in OSA as well: https://github.com/openshift/openshift-ansible/blob/master/roles/kuryr/templates/controller-deployment.yaml.j2#L25-L3716:11
ltomasbodulek, does it rings a bell? http://paste.openstack.org/show/722816/16:11
dulekltomasbo: Suuure, this means kuryr-daemon returned an error.16:12
dulekltomasbo: More to see in its logs. :)16:12
ltomasboweird thing, it that it eventually succedd16:13
ltomasbosucceed16:13
*** yamamoto has quit IRC16:14
*** yamamoto has joined #openstack-kuryr16:14
ltomasbodulek, http://paste.openstack.org/show/722817/16:15
dulekltomasbo: Ah, pyroute2.16:15
ltomasboruntimeerror...16:15
dulekcelebdor1: Shall we give it a go: https://review.openstack.org/#/c/570553/ ?16:16
*** yamamoto has quit IRC16:17
*** yamamoto has joined #openstack-kuryr16:19
celebdor1dulek: Can it be changed in a deployment? Or it needs a pod re-deploy?16:21
celebdor1dulek: let's16:22
dulekcelebdor1: pod-redeploy, I think.16:22
celebdor1well16:22
celebdor1it's something worthy to document16:22
*** irenab has quit IRC16:44
*** irenab has joined #openstack-kuryr16:46
*** maysamacedos has quit IRC17:07
*** janki has quit IRC17:09
*** yamamoto has quit IRC17:10
*** yamamoto has joined #openstack-kuryr17:13
*** yamamoto has quit IRC17:18
*** maysamacedos has joined #openstack-kuryr17:21
*** yamamoto has joined #openstack-kuryr17:28
*** yamamoto has quit IRC17:30
*** yamamoto has joined #openstack-kuryr17:30
*** yamamoto_ has joined #openstack-kuryr17:43
*** yamamoto has quit IRC17:47
*** celebdor1 has quit IRC17:48
*** yamamoto_ has quit IRC18:18
*** yamamoto has joined #openstack-kuryr18:28
*** hongbin has joined #openstack-kuryr18:32
*** yamamoto has quit IRC18:32
*** yamamoto has joined #openstack-kuryr18:36
*** celebdor1 has joined #openstack-kuryr18:52
*** celebdor1 has quit IRC18:59
*** celebdor1 has joined #openstack-kuryr19:03
openstackgerritDoug Hellmann proposed openstack/kuryr master: fix tox python3 overrides  https://review.openstack.org/57290819:27
*** atoth has quit IRC19:32
*** yamamoto has quit IRC19:51
*** yamamoto has joined #openstack-kuryr19:58
*** yamamoto has quit IRC20:02
*** yamamoto has joined #openstack-kuryr20:20
*** yamamoto has quit IRC20:20
*** celebdor1 has quit IRC20:41
*** rh-jelabarre has quit IRC20:56
*** maysamacedos has quit IRC21:00
*** rh-jelabarre has joined #openstack-kuryr21:01
*** yamamoto has joined #openstack-kuryr21:20
*** yamamoto has quit IRC21:33
*** maysamacedos has joined #openstack-kuryr21:55
*** lxkong has joined #openstack-kuryr22:01
*** pmannidi has joined #openstack-kuryr22:39
*** rh-jelabarre has quit IRC22:43
*** rh-jelabarre has joined #openstack-kuryr22:47
*** threestrands has joined #openstack-kuryr22:51
*** gigo has quit IRC22:51
*** gigo has joined #openstack-kuryr22:56
*** maysamacedos has quit IRC23:20
*** hongbin has quit IRC23:33
« Tuesday, 2018-06-05 Index Thursday, 2018-06-07 »

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!