| *** lakerzhou has quit IRC | 00:02 | |
| *** alishaaneja has joined #openstack-kuryr | 00:07 | |
| *** hongbin has quit IRC | 00:18 | |
| *** caowei has joined #openstack-kuryr | 01:02 | |
| *** lakerzhou1 has quit IRC | 01:04 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements https://review.openstack.org/495179 | 01:06 |
|---|---|---|
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/496048 | 01:08 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements https://review.openstack.org/502663 | 01:08 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/500060 | 01:08 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-tempest-plugin master: Updated from global requirements https://review.openstack.org/502664 | 01:08 |
| *** aojea has joined #openstack-kuryr | 01:18 | |
| *** kiennt has joined #openstack-kuryr | 01:22 | |
| *** aojea has quit IRC | 01:24 | |
| *** hongbin has joined #openstack-kuryr | 01:55 | |
| *** wangbo has joined #openstack-kuryr | 02:28 | |
| *** aishpant_ has joined #openstack-kuryr | 02:43 | |
| *** dougbtv__ has quit IRC | 02:46 | |
| *** aojea has joined #openstack-kuryr | 03:20 | |
| janonymous | dulek: thanks for looking at the patch! , I am looking at the comments. Please feel free to upload change if required. :) | 03:21 |
| openstackgerrit | Hongbin Lu proposed openstack/kuryr-libnetwork master: [WIP] Supprt searching existing port with macaddress https://review.openstack.org/505443 | 03:22 |
| *** aojea has quit IRC | 03:24 | |
| *** aishpant_ has quit IRC | 03:45 | |
| *** kiennt has quit IRC | 03:59 | |
| *** janki has joined #openstack-kuryr | 04:00 | |
| *** hongbin has quit IRC | 04:04 | |
| *** caowei has quit IRC | 04:08 | |
| *** diga has joined #openstack-kuryr | 04:09 | |
| *** aishpant has joined #openstack-kuryr | 04:18 | |
| *** gouthamr has quit IRC | 04:21 | |
| janonymous | dulek: I am checking furthur, can you please check with following settings, it should work ideally http://paste.openstack.org/show/621501 | 04:27 |
| *** wangbo has quit IRC | 04:32 | |
| *** kiennt has joined #openstack-kuryr | 04:33 | |
| *** caowei has joined #openstack-kuryr | 04:40 | |
| *** wangbo has joined #openstack-kuryr | 04:59 | |
| *** caowei has quit IRC | 05:01 | |
| *** aojea has joined #openstack-kuryr | 05:21 | |
| *** aojea has quit IRC | 05:25 | |
| *** aojea has joined #openstack-kuryr | 05:38 | |
| *** caowei has joined #openstack-kuryr | 05:43 | |
| *** dougbtv_ has quit IRC | 06:04 | |
| *** diga has quit IRC | 06:08 | |
| *** dougbtv_ has joined #openstack-kuryr | 06:10 | |
| *** kiennt has quit IRC | 06:10 | |
| *** aishpant has quit IRC | 06:19 | |
| *** aishpant_ has joined #openstack-kuryr | 06:29 | |
| *** dougbtv_ has quit IRC | 06:42 | |
| *** kiennt has joined #openstack-kuryr | 06:45 | |
| *** dougbtv_ has joined #openstack-kuryr | 06:45 | |
| *** aojea has quit IRC | 06:54 | |
| *** alishaaneja has quit IRC | 06:56 | |
| *** pcaruana has joined #openstack-kuryr | 07:02 | |
| *** openstackgerrit has quit IRC | 07:02 | |
| *** caowei has quit IRC | 07:27 | |
| apuimedo | ltomasbo: ping | 07:28 |
| ltomasbo | apuimedo, pong | 07:29 |
| *** caowei has joined #openstack-kuryr | 07:29 | |
| apuimedo | ltomasbo: for some reason it seems that there were some neutron calls in the pod deletion flow | 07:30 |
| ltomasbo | ohh, yes | 07:30 |
| ltomasbo | there are | 07:30 |
| ltomasbo | we ensure the sg are reapplied | 07:30 |
| *** caowei has quit IRC | 07:30 | |
| ltomasbo | apuimedo, but it can be controlled how often those happend | 07:30 |
| ltomasbo | i.e., they do not happen for each pod deletion, but at every X seconds for the pods deleted in the meantime | 07:31 |
| ltomasbo | apuimedo, so, did neutron crashed due to the deletion? | 07:31 |
| apuimedo | ltomasbo: we do it on a per port basis? | 07:34 |
| apuimedo | right | 07:34 |
| apuimedo | in the configuration of period | 07:34 |
| ltomasbo | apuimedo, when we delete X pods, we put their associated ports in a list | 07:34 |
| ltomasbo | apuimedo, and then, every X seconds we process that list, re-apply sg per port there | 07:35 |
| ltomasbo | apuimedo, and then add them back to the pool, so that they can be reused | 07:35 |
| apuimedo | ltomasbo: we check or we apply? | 07:35 |
| *** caowei has joined #openstack-kuryr | 07:35 | |
| ltomasbo | apuimedo, apply | 07:35 |
| apuimedo | ltomasbo: this kills the neutron | 07:36 |
| apuimedo | xD | 07:36 |
| ltomasbo | so easy to kill neutron... | 07:36 |
| ltomasbo | ok, maybe we can set a cap | 07:36 |
| apuimedo | ltomasbo: well, there's no port update in batch | 07:36 |
| ltomasbo | no more than X ports being recover per period | 07:36 |
| ltomasbo | or something like that | 07:36 |
| apuimedo | so we're sending 900 port updates | 07:36 |
| ltomasbo | for example, don't process more than 50 request -- ports -- per period, or something like that | 07:37 |
| ltomasbo | though it should not be like that! neutron should queue them up if it is not able to handle them... | 07:37 |
| ltomasbo | and process them as soon as it can | 07:37 |
| ltomasbo | apuimedo, do you want me to send a patch with the cap? configurable amount, starting by 1 per second (if the interval is 15 second, then 15 per interval) | 07:38 |
| apuimedo | ltomasbo: nah | 07:45 |
| apuimedo | I think the first step will have to be to check the SGs with a GET | 07:45 |
| apuimedo | and only re-apply if necessary | 07:45 |
| ltomasbo | so get should be faster than update | 07:47 |
| apuimedo | for sure | 07:47 |
| ltomasbo | but in the worst case it can be almost double calls | 07:47 |
| apuimedo | ltomasbo: It's a single Get | 07:48 |
| apuimedo | isn't it? | 07:48 |
| apuimedo | you get all the ports | 07:48 |
| apuimedo | and it already includes the info on SGs | 07:48 |
| ltomasbo | true | 07:48 |
| apuimedo | in the meantime, I'll concentrate on speeding up the pool loading | 07:48 |
| apuimedo | ltomasbo: http://paste.openstack.org/show/621510/ | 07:58 |
| apuimedo | lines 13, 14 | 07:58 |
| apuimedo | does this mean it will go and reapply SGs to all the precreated ports? | 07:58 |
| apuimedo | ltomasbo: http://paste.openstack.org/show/621512/ | 08:11 |
| ltomasbo | sorry, my laptop crashed... | 08:11 |
| apuimedo | ltomasbo: http://paste.openstack.org/show/621512/ | 08:11 |
| ltomasbo | had to reboot | 08:12 |
| ltomasbo | let me check | 08:12 |
| apuimedo | what do you think of this way to select which to do the reapply | 08:12 |
| ltomasbo | seems ok to me | 08:13 |
| ltomasbo | though it should be filtered by the list of ports to recover | 08:14 |
| ltomasbo | as those are the only ones to check | 08:14 |
| apuimedo | ltomasbo: you mean to check those not in k8s? | 08:15 |
| ltomasbo | we already have the list of ports to be 'recovered' | 08:18 |
| ltomasbo | when the pods are deleted | 08:18 |
| ltomasbo | ahhh, wait | 08:18 |
| ltomasbo | I think we are mixing things | 08:18 |
| ltomasbo | are you talking about re-applying security groups when pods are deleted | 08:18 |
| apuimedo | ltomasbo: I was talking about what is done at startup time | 08:19 |
| ltomasbo | or about recovering the precreated ports upon a controller restart? | 08:19 |
| ltomasbo | ok ok | 08:19 |
| ltomasbo | ummm | 08:19 |
| ltomasbo | why are you checking the sg then> | 08:19 |
| ltomasbo | ? | 08:19 |
| apuimedo | ltomasbo: well... To check that the precreated ports match the runtime SG configuration | 08:21 |
| apuimedo | are you not doing that verification? | 08:21 |
| ltomasbo | no, I'm checking what SG they have | 08:21 |
| ltomasbo | and associate them to the right pool | 08:21 |
| ltomasbo | assuming in the future there may be different pools for different sgs | 08:21 |
| ltomasbo | so, if the sg are different, it will end up in a different pool anyway | 08:22 |
| apuimedo | ok | 08:23 |
| apuimedo | I'll continue checking to make it better | 08:23 |
| *** Snacker has joined #openstack-kuryr | 08:32 | |
| *** yamamoto_ has quit IRC | 08:33 | |
| *** egonzalez has joined #openstack-kuryr | 08:35 | |
| *** yamamoto has joined #openstack-kuryr | 08:39 | |
| *** vkuramshin has joined #openstack-kuryr | 08:44 | |
| *** yamamoto has quit IRC | 08:52 | |
| *** yamamoto has joined #openstack-kuryr | 08:54 | |
| ltomasbo | apuimedo, the needed modifications for octavia are already in | 09:00 |
| ltomasbo | apuimedo, so this patch https://review.openstack.org/#/c/499103/ is ready for reviews | 09:00 |
| ltomasbo | apuimedo, I just test it and it is working | 09:00 |
| *** alishaaneja has joined #openstack-kuryr | 09:04 | |
| *** openstackgerrit has joined #openstack-kuryr | 09:11 | |
| openstackgerrit | Merged openstack/kuryr-tempest-plugin master: Updated from global requirements https://review.openstack.org/502664 | 09:11 |
| *** aishpant_ has quit IRC | 09:14 | |
| dulek | kuryr-kubernetes Octavia job looks fixed. Yay! :) | 09:17 |
| *** vkuramshin has quit IRC | 09:18 | |
| *** alishaaneja has quit IRC | 09:20 | |
| apuimedo | ltomasbo: great news | 09:21 |
| ltomasbo | apuimedo, tell me? | 09:21 |
| apuimedo | ltomasbo: I mean about the l2 | 09:22 |
| ltomasbo | ahh | 09:22 |
| apuimedo | next will be to verify the damn octavia pooling | 09:22 |
| apuimedo | last time I tried it it blew up the server | 09:22 |
| ltomasbo | really? | 09:22 |
| ltomasbo | how many did you included into the pool? | 09:22 |
| apuimedo | 100 | 09:25 |
| apuimedo | but it didn't create any | 09:25 |
| *** alishaaneja has joined #openstack-kuryr | 09:28 | |
| ltomasbo | ok | 09:32 |
| *** alishaaneja has quit IRC | 09:44 | |
| *** vkuramshin has joined #openstack-kuryr | 09:49 | |
| *** aishpant_ has joined #openstack-kuryr | 10:01 | |
| *** kiennt has quit IRC | 10:07 | |
| *** vkuramshin has quit IRC | 10:13 | |
| *** aojea has joined #openstack-kuryr | 10:14 | |
| *** aojea has quit IRC | 10:14 | |
| *** jchhatbar has joined #openstack-kuryr | 10:15 | |
| *** jchhatbar has quit IRC | 10:15 | |
| *** janki has quit IRC | 10:17 | |
| *** jchhatbar has joined #openstack-kuryr | 10:17 | |
| *** wangbo has quit IRC | 10:19 | |
| *** jchhatbar_ has joined #openstack-kuryr | 10:22 | |
| *** jchhatbar_ has quit IRC | 10:22 | |
| *** aojea has joined #openstack-kuryr | 10:23 | |
| *** janki has joined #openstack-kuryr | 10:23 | |
| *** jchhatbar has quit IRC | 10:25 | |
| *** aojea has quit IRC | 10:33 | |
| *** caowei has quit IRC | 10:34 | |
| *** aishpant_ has quit IRC | 10:37 | |
| *** aojea has joined #openstack-kuryr | 10:39 | |
| *** yamamoto has quit IRC | 10:46 | |
| *** alishaaneja has joined #openstack-kuryr | 10:46 | |
| *** yamamoto has joined #openstack-kuryr | 10:51 | |
| openstackgerrit | Michał Dulko proposed openstack/kuryr-kubernetes master: Clean up ENV vars mistmatches in Dockerfiles https://review.openstack.org/505125 | 10:59 |
| dulek | ltomasbo: This one's tested. I've almost got a headache from figuring out those path… :) | 11:00 |
| ltomasbo | dulek, :D | 11:02 |
| ltomasbo | what was it? | 11:02 |
| dulek | ltomasbo: Well, those path are mounted in different places by DaemonSet k8s definition. I've overlooked that - cni docker should look where those are mounted, not where those are on the host itself. | 11:04 |
| ltomasbo | ahhh | 11:04 |
| ltomasbo | nice catch! | 11:05 |
| ltomasbo | going to test it asap! | 11:05 |
| *** wangbo has joined #openstack-kuryr | 11:10 | |
| *** aojea has quit IRC | 11:16 | |
| *** janonymous has quit IRC | 11:16 | |
| *** aishpant has joined #openstack-kuryr | 11:21 | |
| *** aishpant has quit IRC | 11:26 | |
| *** atoth has joined #openstack-kuryr | 11:41 | |
| openstackgerrit | Luis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Optimize return_to_pool actions at Pool drivers https://review.openstack.org/505605 | 11:43 |
| *** vkuramshin has joined #openstack-kuryr | 11:45 | |
| *** vkuramshin has quit IRC | 11:56 | |
| *** lakerzhou has joined #openstack-kuryr | 12:00 | |
| *** janki has quit IRC | 12:08 | |
| *** janki has joined #openstack-kuryr | 12:20 | |
| ltomasbo | dulek, I got the same problem with the kubernetes-scheduler existing and therefore stopping the devstack stacking | 12:22 |
| dulek | ltomasbo: On fresh install? I've been hitting different installation issues, but ./unstack && ./stack.sh eventually solved that. | 12:23 |
| ltomasbo | yep, on fresh install | 12:24 |
| ltomasbo | yep, I'm trying that | 12:24 |
| ltomasbo | but I'm not sure why this problem happens with the containerized version, and not with the other | 12:24 |
| ltomasbo | some kind of race condition or something I guess | 12:24 |
| dulek | ltomasbo: Hm… Let me check. | 12:25 |
| dulek | ltomasbo: Before run_k8s_scheduler the only thing that's different in containerized version is that k8s is not configured with kuryr-cni. | 12:27 |
| dulek | ltomasbo: But that step only copies the binary into CNI_BIN_DIR, so it shouldn't matter. | 12:27 |
| ltomasbo | dulek, I think the scheduler actually runs for a while, but then it dies (not sure why) | 12:31 |
| dulek | ltomasbo: journalctl -u devstack@kube-scheduler | 12:32 |
| dulek | ltomasbo: I bet you'll see "leader election" and this stuff in the logs. That's etcd-related. | 12:33 |
| ltomasbo | yes, that was there in the stacking dumps | 12:33 |
| ltomasbo | (but I'm restacking now) | 12:33 |
| dulek | Maybe etcd_legacy works a bit better? | 12:34 |
| ltomasbo | what I don't know is why that error happens when using the containerized version, and not when not using it | 12:34 |
| ltomasbo | ummm, could be | 12:34 |
| dulek | ltomasbo: I hit those errors in both containerized and non-containerized. | 12:38 |
| *** wangbo has quit IRC | 13:06 | |
| *** gouthamr has joined #openstack-kuryr | 13:07 | |
| *** aishpant_ has joined #openstack-kuryr | 13:24 | |
| *** kiennt has joined #openstack-kuryr | 13:32 | |
| ltomasbo | dulek, ok, I didn't hit them with non-containerized, at least not all the time | 13:36 |
| ltomasbo | after the initial failed attempt, as you said, just doing ./unstack and ./stack makes it work | 13:36 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/496048 | 13:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements https://review.openstack.org/502663 | 13:50 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/500060 | 13:50 |
| *** aojea has joined #openstack-kuryr | 13:51 | |
| *** alishaaneja has quit IRC | 13:58 | |
| *** c00281451_ has joined #openstack-kuryr | 14:00 | |
| *** zengchen has quit IRC | 14:00 | |
| *** dougbtv__ has joined #openstack-kuryr | 14:07 | |
| *** janki has quit IRC | 14:07 | |
| *** vikasc has quit IRC | 14:12 | |
| *** aojea has quit IRC | 14:18 | |
| *** aishpant_ has quit IRC | 14:34 | |
| *** dougbtv__ has quit IRC | 14:43 | |
| *** vikasc has joined #openstack-kuryr | 14:53 | |
| *** hongbin has joined #openstack-kuryr | 14:56 | |
| *** premsankar has joined #openstack-kuryr | 15:01 | |
| *** alishaaneja has joined #openstack-kuryr | 15:15 | |
| *** alishaaneja has quit IRC | 15:35 | |
| *** leifmadsen has quit IRC | 15:46 | |
| *** leifmadsen has joined #openstack-kuryr | 15:47 | |
| *** Snacker has left #openstack-kuryr | 15:49 | |
| *** deepika has joined #openstack-kuryr | 15:58 | |
| *** kiennt has quit IRC | 16:04 | |
| *** janki has joined #openstack-kuryr | 16:26 | |
| *** edisonxiang_ has joined #openstack-kuryr | 16:40 | |
| *** edisonxiang has quit IRC | 16:43 | |
| *** janki has quit IRC | 16:46 | |
| *** vkuramshin has joined #openstack-kuryr | 16:55 | |
| *** egonzalez has quit IRC | 17:01 | |
| *** aojea has joined #openstack-kuryr | 17:03 | |
| *** aojea has quit IRC | 17:05 | |
| *** aojea has joined #openstack-kuryr | 17:06 | |
| *** robust has joined #openstack-kuryr | 17:09 | |
| *** vkuramshin has quit IRC | 17:10 | |
| *** aishpant has joined #openstack-kuryr | 17:29 | |
| *** aishpant has quit IRC | 17:33 | |
| *** yamamoto has quit IRC | 18:17 | |
| *** aojea has quit IRC | 18:33 | |
| *** egonzalez has joined #openstack-kuryr | 18:56 | |
| *** aojea has joined #openstack-kuryr | 19:06 | |
| *** robust has quit IRC | 19:08 | |
| *** deepika has quit IRC | 19:15 | |
| *** yamamoto has joined #openstack-kuryr | 19:17 | |
| *** yamamoto has quit IRC | 19:26 | |
| *** atoth has quit IRC | 19:26 | |
| *** robust has joined #openstack-kuryr | 19:56 | |
| *** aojea has quit IRC | 19:58 | |
| *** aojea has joined #openstack-kuryr | 19:58 | |
| *** aojea has quit IRC | 19:58 | |
| *** aojea has joined #openstack-kuryr | 20:08 | |
| *** gouthamr has quit IRC | 20:09 | |
| *** lakerzhou has quit IRC | 20:56 | |
| *** pcaruana has quit IRC | 20:59 | |
| *** aojea has quit IRC | 21:28 | |
| *** lakerzhou has joined #openstack-kuryr | 21:43 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/fuxi master: Updated from global requirements https://review.openstack.org/495179 | 21:52 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr master: Updated from global requirements https://review.openstack.org/496048 | 21:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-libnetwork master: Updated from global requirements https://review.openstack.org/502663 | 21:54 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kuryr-kubernetes master: Updated from global requirements https://review.openstack.org/500060 | 21:54 |
| *** kzaitsev_pi has quit IRC | 21:57 | |
| *** kzaitsev_pi has joined #openstack-kuryr | 22:02 | |
| *** lakerzhou has quit IRC | 22:02 | |
| *** robust has quit IRC | 22:09 | |
| *** premsankar has quit IRC | 22:21 | |
| *** yamamoto has joined #openstack-kuryr | 22:28 | |
| *** egonzalez has quit IRC | 23:10 | |
| *** gouthamr has joined #openstack-kuryr | 23:11 | |
| *** gouthamr has quit IRC | 23:27 | |
| openstackgerrit | Hongbin Lu proposed openstack/kuryr-libnetwork master: Enable debug if ENABLE_DEBUG_LOG_LEVEL is set https://review.openstack.org/505852 | 23:43 |
| *** hongbin has quit IRC | 23:44 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!