Tuesday, 2017-10-31

*** gouthamr has quit IRC00:05
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978100:34
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978100:35
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978100:36
*** hongbin has joined #openstack-kuryr00:39
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978100:44
*** kiennt26 has joined #openstack-kuryr01:01
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978102:00
*** wangbo_ has joined #openstack-kuryr02:04
*** caowei has joined #openstack-kuryr02:20
*** salv-orl_ has joined #openstack-kuryr02:39
*** salv-orlando has quit IRC02:42
*** hongbin has quit IRC02:56
*** hongbin_ has joined #openstack-kuryr02:56
*** hongbin_ has quit IRC02:59
*** hongbin has joined #openstack-kuryr03:00
*** wangbo_ has quit IRC03:07
*** hongbin has quit IRC03:22
*** gouthamr has joined #openstack-kuryr03:35
*** gouthamr has quit IRC04:34
*** caowei has quit IRC04:39
*** janki has joined #openstack-kuryr05:06
*** caowei has joined #openstack-kuryr05:11
*** janki has quit IRC05:27
*** janki has joined #openstack-kuryr05:27
*** kiennt26 has quit IRC05:42
*** caowei has quit IRC05:56
*** caowei has joined #openstack-kuryr05:56
*** caowei_ has joined #openstack-kuryr06:02
*** caowei has quit IRC06:03
*** caowei_ is now known as caowei06:03
*** kiennt26 has joined #openstack-kuryr06:05
*** caowei has quit IRC06:27
*** caowei has joined #openstack-kuryr06:27
*** salv-orl_ has quit IRC07:11
*** salv-orlando has joined #openstack-kuryr07:11
*** salv-orlando has quit IRC07:16
*** caowei_ has joined #openstack-kuryr07:16
*** caowei has quit IRC07:17
*** caowei_ is now known as caowei07:17
*** kiennt26 has quit IRC07:17
*** pcaruana has joined #openstack-kuryr07:17
*** kiennt26 has joined #openstack-kuryr07:19
*** salv-orlando has joined #openstack-kuryr07:21
*** kiennt26 has quit IRC08:17
*** yamamoto has quit IRC08:23
*** salv-orlando has quit IRC08:27
*** salv-orlando has joined #openstack-kuryr08:28
*** zengchen has quit IRC08:28
*** salv-orlando has quit IRC08:32
*** salv-orlando has joined #openstack-kuryr08:50
*** garyloug has joined #openstack-kuryr08:54
dmelladodulek: ltomasbo I still do mention that error, tbh xD09:01
ltomasbodmellado, ???09:01
dulekltomasbo: "<dulek> ltomasbo: I mean dmellado mentioned IDoNeedMoreCoffeeError."09:02
ltomasboxD09:02
dulekMorning. ;)09:02
irenabdmellado, you should try to sleep routine once this error happens. This may have stronger effect09:03
irenabmorning09:04
*** garyloug has quit IRC09:06
ltomasboapuimedo, I'm doing some initial testing of the modifications we discussed yesterday09:15
apuimedoltomasbo: and?09:16
ltomasbotimes goes down in my small env from 13.5 sec to 0.509:16
ltomasborecovering 60 subports in 1 node09:16
apuimedoltomasbo: YEEHAAAAAA!09:16
apuimedoI knew it!09:16
ltomasboso, pretty good results (without list comprehension)09:16
ltomasboI'll update the test and polish a bit the code and send it for review09:17
apuimedoltomasbo: how did you deploy? tripled'oh?09:17
ltomasboapuimedo, thanks for the tips!09:17
apuimedoltomasbo: you're welcome09:17
ltomasbono, this was devstack09:17
apuimedoI'm already an expert in Neutron avoidance09:17
apuimedoxD09:17
ltomasboxD09:17
ltomasboI'm following your steps... xD09:18
apuimedoIhow much I wish we still had the environment so I could try it with the 1600 subports spread on 112 nodes09:22
apuimedoI wonder...09:22
apuimedoWe should probably be able to simulate it with containers09:22
apuimedo112 containers with the sdn agent09:23
apuimedoand that from the container veth they make the vlan ports09:23
*** yamamoto has joined #openstack-kuryr09:24
irenabapuimedo, what are these numbers?09:24
apuimedoirenab: which? these ports and nodes I talk about?09:24
irenabyes09:24
apuimedoor ltomasbo's times?09:24
apuimedothis is what we tested in the scale lab09:24
apuimedoI'll publish it next week09:25
apuimedo(I have to remove some internal dns stuff)09:25
apuimedowe had 30 serers09:25
apuimedo*servers09:25
apuimedoon which we ran 112 VMs09:25
apuimedoand on those 112VMs we ran 900 pods out of a pool of 1600 ports09:25
apuimedoand it was taking minutes to recover the precreated ports for the pool09:26
apuimedohence ltomasbo's work to speed that recovery09:26
irenabah, so just agreed setup, not some specific use case09:26
irenabI think we may need to see your outcomes to make network policy as performant as possible09:27
apuimedoirenab: agreed09:27
apuimedoirenab: the outcome is basically. Avoid Neutron calls at all costs09:27
apuimedoxD09:27
irenabhaha09:27
apuimedobut I'm planning to speed Neutron up09:28
apuimedofor network policy we must09:28
irenabbulks?09:28
apuimedoyes, I'm redoing the bulk creation09:28
apuimedoand bulk update and bulk delete should be added09:28
*** garyloug has joined #openstack-kuryr09:28
irenabbut also read operations are not fast enough09:28
*** jistr|mtgs is now known as jistr09:29
ltomasboyes09:29
ltomasbothe improvement was by changing 2 calls to get the ports by 109:29
irenabbetter filters?09:29
ltomasboand then to avoid the network and subnet show that the default_network driver does09:29
ltomasboeven with filters, it was better to get them all in one call, and do the filter ourselves09:30
irenabmakes sense, since if it cached and not present, there will be exception anyway09:30
apuimedoltomasbo: and also that we only get subnets once09:30
apuimedothe code before got subnet once per port09:30
apuimedoand also networks09:30
apuimedosame thing09:31
apuimedoit's a lot of saved gets09:31
apuimedoirenab: at the end of the day... The ML2 abstraction needs to be sped up09:31
irenabyea, seems you have a lot of tips how to make performance efficient code when work with neutron API09:32
apuimedoltomasbo: irenab: If more speed up is necessary, for loading the pool09:32
apuimedothe next item on my list is to make neutron have a plugin to return vif objects directly09:32
irenabplugin?09:32
apuimedoirenab: service plugin09:32
apuimedolike trunks and such09:33
irenabI see09:33
irenabextension to API09:33
apuimedoirenab: right09:33
ltomasboapuimedo, ohh, that would be good, but not short-term09:33
apuimedothe idea was was approved in Atlanta09:33
apuimedoit is long overdue09:33
ltomasboany takers of that one?09:33
apuimedobut the people that were set to drive it have not done so09:33
irenabapuimedo, is there any bp/rfe in neutron for thus?09:34
irenabthis09:34
apuimedoltomasbo: for that we need scale numbers after your patch09:34
*** yboaron has joined #openstack-kuryr09:34
apuimedoto see how much necessary it still is09:34
apuimedoso we can try to force things09:34
ltomasbook09:34
*** yamamoto has quit IRC09:34
apuimedoirenab: I'd assume there is, since it was agreed in Atlanta09:34
apuimedobut I never checked09:34
irenabapuimedo, ltomasbo do you have specific numbers on your mind?09:34
apuimedoirenab: loading 2000 ports must take less than 30 seconds09:35
apuimedoanything more09:35
apuimedoand I don't accept it09:35
apuimedodmellado: why do you need ping in https://review.openstack.org/#/c/515357/ ?09:36
apuimedoIsn't it enough to test it with curl?09:37
irenabapuimedo, interesting. I wonder if we need to put kuryr scale requirements somewhere09:42
apuimedoirenab: like?09:42
irenabsince there can be different backends and it may impact on overall performance09:43
apuimedoirenab: well, this is ml209:43
irenabapuimedo, some spec in doc09:43
apuimedoall the ml2 backends suffer from the same09:43
irenabbut this can be worth if backend adds additional cost09:44
irenabI mean you sort of defined white box requirement, but we probably need some palce that defines kury-k8s scale requirement09:44
apuimedoirenab: that's a good ides09:45
apuimedo*idea09:45
apuimedodulek: any progress on the flask uwsgi front?09:45
dulekapuimedo, ltomasbo: Can you briefly explain what you've did to get 13 seconds improvement?09:46
apuimedodulek: sure can09:46
dulekapuimedo: I've discovered what was causing the processes leak, uwsgi would not help.09:46
apuimedothe code for preloading the ports to the pool09:46
dulekapuimedo: Wanna guess?09:46
ltomasbodulek, I'll update the patch set soon09:46
apuimedodulek: I know it is pyroute209:46
dulekapuimedo: No, but you're close. os_vif.09:47
apuimedodulek: pyroute2 also leaks but very slowly09:47
apuimedodulek: I more and more want to get rid of os_vif09:47
apuimedoxD09:47
ltomasbodulek, os_vif?09:47
apuimedodulek: so. it was first getting all active ports, then getting parent ports, then for each subport it was getting its subnet and net09:47
dulekapuimedo: Yeah, commenting out os_vif lines in base.py helps.09:47
apuimedothat is a fuck ton of Neutron requests09:47
apuimedoso we cut it to a just three09:48
ltomasboapuimedo, also bet that was the problem I was seing on pods being delayed at starting (in nested case)09:48
apuimedogetting active ports, getting subnets and getting nets09:48
apuimedoltomasbo: how so?09:48
apuimedothe preloading should only be exercised at the beginning09:48
dulekapuimedo: Oh, okay. I haven't noticed ltomasbo didn't have that originally in his patch.09:48
apuimedodulek: we should probably make the change where os_vif is only used for baremetal driver09:49
apuimedodulek: no. It's something I suggested in bluejeans yesterday and ltomasbo just finished testing09:49
dulekapuimedo: I agree, we've discussed that already.09:49
apuimedodulek: regarding the flask thing09:49
apuimedoI am really curious how it would behave with grpc instead09:49
ltomasboapuimedo, dulek: I tested in a nested environment, and removing the os_vif.plug was not helping09:49
apuimedobut that would probably be a few days of work09:50
dulekapuimedo: I think so, I'm not experienced with gRCP at all.09:50
apuimedoI only ever used it once and it was just the client side09:50
apuimedoah no, it was just protobug09:50
apuimedo*protobuff09:51
dulekapuimedo: First I'll try to find out why os-vif is leaking those processes as I need to do that anyway.09:51
dulekapuimedo: Then the patch will be ready, so I can try gRPC thing.09:52
apuimedookey dokey09:52
dulekapuimedo: But sooner or later I'll need to start on HA thing, and sooner would definitely be better.09:52
dulekapuimedo: What's the controller HA expectation for Queens?09:53
apuimedodulek: I think if we get the vif assignment daemon side it will already be a bug ha boots09:53
apuimedo*boost09:53
apuimedo(and time-to-start-container boost as well)09:54
dulekOkay.09:54
dulekapuimedo: BTW I've started thinking about controller HA in general and A/P with Tooz and etcd shouldn't be too hard to do.09:54
apuimedodulek: I'd love to hear what you have in mind for HA09:55
dulekHa, I've answered before being asked. ;)09:55
apuimedodulek: If possible we should use K8s objects rather than go straight to etcd09:56
apuimedoso that we don't require an extra etcd be dep[loyed09:56
dulekapuimedo: Actually OpenStack services *are* requiring an extra etcd. We could use that instance.09:57
apuimedodulek: that depends on whether we expect kuryr users to use those services that require the extra etcd09:57
dulekapuimedo: Oh, we're using it to configure k8s in DevStack, aren't we?09:57
dulekAgreed, to be honest I haven't followed and I don't know if Keystone or Neutron have hard dependency on Neutron already.09:58
duleks/Neutron/etcd09:58
dulek(second Neutron ;))09:58
apuimedodulek: they don't10:00
*** jistr_ has joined #openstack-kuryr10:00
dulekapuimedo: Okay, so in a nutshell - what's the idea of using native k8s objects?10:01
*** jistr has quit IRC10:01
*** pc_m has quit IRC10:04
*** caowei has quit IRC10:04
*** pc_m has joined #openstack-kuryr10:06
*** caowei has joined #openstack-kuryr10:09
*** alraddarla has joined #openstack-kuryr10:14
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Avoid neutron calls at recovering precreated ports  https://review.openstack.org/51015710:25
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978110:30
apuimedodulek: that's what the k8s controller manager does for leader election IIRC10:35
dmelladoapuimedo: pong for the VM-pod ping test10:37
dmelladomaybe I should refer it in my patch10:37
dulekapuimedo: I'll look it up.10:37
apuimedodmellado: what I meant to say was10:37
dmellado /me has 6 hours of meetings today, remember me not ever to become a manager10:37
dmelladoso expect my responses to be slow10:37
apuimedocould that test just have the VM curl the pod?10:37
apuimedodmellado: s/remember/remind/10:37
apuimedo;-)10:37
dmelladoapuimedo: recuérdame10:38
dmelladoxD10:38
dmelladocagonlaputa.... (that's a really polite spanish word for expressing disappointment)10:38
dmelladoapuimedo: overall, I could but I'd prefer to add a second test for that10:39
dmelladoin fact I'll have gcheresh implement it10:39
dmelladolike curl and parse response and blah blah10:39
*** yamamoto has joined #openstack-kuryr10:39
*** caowei has quit IRC10:40
apuimedoalright10:43
apuimedoalright10:43
apuimedoI'll review the patch then10:43
*** atoth has joined #openstack-kuryr11:08
*** salv-orlando has quit IRC11:48
*** salv-orlando has joined #openstack-kuryr11:55
*** dougbtv has joined #openstack-kuryr12:00
*** yamamoto has quit IRC12:11
*** yamamoto has joined #openstack-kuryr12:11
*** salv-orlando has quit IRC12:24
*** salv-orlando has joined #openstack-kuryr12:39
*** alraddarla has quit IRC12:52
*** yamamoto has quit IRC12:56
*** janki has quit IRC12:58
*** yamamoto has joined #openstack-kuryr13:10
*** yamamoto has quit IRC13:15
*** salv-orlando has quit IRC13:19
*** salv-orlando has joined #openstack-kuryr13:20
*** salv-orlando has quit IRC13:24
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978113:31
openstackgerritOpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements  https://review.openstack.org/50978113:33
*** yamamoto has joined #openstack-kuryr13:41
*** hongbin has joined #openstack-kuryr13:43
*** yamamoto has quit IRC13:45
dulekapuimedo: https://bugs.launchpad.net/kuryr-kubernetes/+bug/172899613:47
openstackLaunchpad bug 1728996 in kuryr-kubernetes "Usage of IPDB leaks processes" [Undecided,New]13:47
dulekapuimedo: I should have noticed this freaking little bastard long ago.13:48
*** kiennt26 has joined #openstack-kuryr13:49
apuimedodulek: good find!13:53
*** oanson has quit IRC13:57
*** oanson has joined #openstack-kuryr13:57
*** janki has joined #openstack-kuryr14:01
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: CNI split - introducing CNI daemon  https://review.openstack.org/51518614:04
dulekltomasbo, apuimedo: ^ - processes still be increasing due to bug 1728996, but all of them will be stopped on service SIGTERM.14:05
openstackbug 1728996 in kuryr-kubernetes "Usage of IPDB leaks processes" [Undecided,New] https://launchpad.net/bugs/172899614:05
ltomasbodulek, nice finding!14:07
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: DNM: Trying to run gate tests with CNI Daemon  https://review.openstack.org/50976514:12
*** salv-orlando has joined #openstack-kuryr14:37
*** salv-orl_ has joined #openstack-kuryr14:40
*** salv-orlando has quit IRC14:44
apuimedodmellado: could you check the comment dulek put on https://review.openstack.org/#/c/515357/8 ?14:47
dmelladoapuimedo: not now14:47
apuimedoit's about using test-config for the stage14:47
dmelladosaw it and will take a look14:47
dmelladobut I'm into meetings like I were a manager today14:47
apuimedodmellado: das ist nicht normal14:48
dmelladoTODO. me14:48
dmelladoDoch, Ich glaube nein14:48
dmelladoAber wir sollen über Alex ausstellunggespräch14:48
dmelladosprechen14:49
apuimedoirenab: could you merge https://review.openstack.org/#/c/515357/8 ?14:50
apuimedoIf you agree with my last comment, that is14:51
dmelladoapuimedo: irenab in any case please DNM that patch for now14:52
apuimedodmellado: no, no. My comment is that since we already have wrong stage for the kubeconfig tempest copy14:52
dmelladoack14:52
apuimedoI'd rather merge this and then have a follow-up patch14:52
apuimedothat moves it to the right stage14:52
dmellado+1. but I need to check a slight issue with the octavia gate prior to merge it14:52
apuimedoyou can make Jon or genadi do that14:52
apuimedook14:53
dmelladoafter that, I'll make those jobs voting too14:53
apuimedodmellado: so mark it w-114:53
dmelladoapuimedo: it is14:53
apuimedoah, okk14:53
dmelladojon + genadi = jedi?14:53
dmelladoxD14:53
apuimedoxD14:53
apuimedodmellado: to be fair. I was surprised to see it in devstack. I thought it would be in the scenario test that the setup step would make pods pingable14:54
apuimedoonly for the duration of the tests that need it14:54
dmelladoapuimedo: that would be overkill for the test, IMHO14:56
apuimedodmellado: you sure it isn't just being a bit lazy :P ?14:56
dmelladototally xD14:57
dmelladojust check that in the end it's another patch14:57
dmelladothat's even more work! xD14:57
dmelladobut I was so sad that you didn't get to reply in german too14:57
apuimedodmellado: hey, was screen support dropped?14:57
apuimedoI'm now looking at devstack's functions-common14:58
dmelladoapuimedo: it was totally dropped in devstack a while ago14:58
apuimedoand I only see _run_under_systemd14:58
dmelladoIIRC sdague sent an email about it14:58
*** salv-orl_ has quit IRC14:58
*** salv-orlando has joined #openstack-kuryr14:58
apuimedodmellado: so we are only keeping bc for if somebody uses older devstack with our latest plugin?14:58
dmelladoapuimedo: I'd say we drop it14:59
apuimedoI tend to agree14:59
dmelladoI'll have jon create a 'destroy it' patch so he can get his first +214:59
dmelladoeither him or genadi14:59
dmelladoIn any case besides my sadness about your german I'm also sad because I still don't see any of them in the channel :\15:01
apuimedodmellado: what about my German15:02
apuimedo?15:02
apuimedoisn't it fluent?15:02
apuimedoxD15:02
apuimedoI probably know 5 or six words more15:03
apuimedo"perl ist große scheiße"15:03
*** yamamoto has joined #openstack-kuryr15:04
apuimedothe grammar may be wrong, but the sentiment is right15:04
apuimedoxD15:04
dmelladolol xD15:04
dmelladoapuimedo: that you didn't reply to my pun xD15:04
dmelladoI was saying that we should prepare for Alex's interview xD15:05
apuimedodmellado: you invoked the Alexander15:05
apuimedoI got it15:05
apuimedobut I got scared15:05
dmelladoI invoked it, yeah15:05
dmelladoI wonder if it's better or worse than the Demogorgon15:06
apuimedodon't know demogorgon15:06
dmelladodidn't you watch Stranger Things?15:06
dmelladoyou should catch up on the flight xD15:06
apuimedonever did15:06
*** salv-orlando has quit IRC15:06
dmelladoso you've two seasons to catch-up on the flight but still should've time15:07
*** salv-orlando has joined #openstack-kuryr15:07
apuimedodmellado: you're depressing me15:07
apuimedoI don't want to be so long on a metal tube15:07
dmelladoapuimedo: you won't be on a metal tube but two! xD15:10
apuimedobleh15:10
apuimedoat least it is an a38015:10
*** salv-orlando has quit IRC15:11
dmelladodon't forget that even though it's an a380 you'll flight economy15:12
dmelladomaybe you'd be in the middle on the row of 515:13
*** yamamoto has quit IRC15:14
apuimedodmellado: I find a380's economy (in emirates) quite enjoyable15:18
apuimedoand yeah. I usually end up in the middle column :/15:19
*** kiennt26 has quit IRC15:41
*** janki has quit IRC16:00
*** yamamoto has joined #openstack-kuryr16:02
*** yamamoto has quit IRC16:02
*** salv-orlando has joined #openstack-kuryr16:32
*** salv-orlando has quit IRC16:33
*** salv-orlando has joined #openstack-kuryr16:37
yboaronapuimedo, Shared with U short doc with route/service objects dependencies , I'll highly appreciate your feedback16:45
*** yboaron has quit IRC16:52
*** yamamoto has joined #openstack-kuryr17:03
*** yamamoto has quit IRC17:11
*** gouthamr has joined #openstack-kuryr17:16
*** salv-orlando has quit IRC17:24
*** salv-orlando has joined #openstack-kuryr17:25
*** salv-orlando has quit IRC17:29
*** pcaruana has quit IRC17:29
*** garyloug has quit IRC17:34
*** salv-orlando has joined #openstack-kuryr17:42
*** pcaruana has joined #openstack-kuryr19:15
*** pcaruana has quit IRC19:49
*** salv-orlando has quit IRC19:51
openstackgerritHongbin Lu proposed openstack/kuryr-libnetwork master: Change binding config from 'driver' to 'default_driver'  https://review.openstack.org/51612819:54
*** livelace-link has quit IRC21:03
*** salv-orlando has joined #openstack-kuryr21:05
*** hongbin has quit IRC22:55

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!