| *** markvoelker has joined #openstack-meeting | 00:24 | |
| *** markvoelker has quit IRC | 00:28 | |
| *** ircuser-1 has quit IRC | 00:36 | |
| *** yamamoto has joined #openstack-meeting | 00:54 | |
| *** ekcs has quit IRC | 01:10 | |
| *** ekcs has joined #openstack-meeting | 01:12 | |
| *** bobh has quit IRC | 01:18 | |
| *** bobh has joined #openstack-meeting | 01:24 | |
| *** markvoelker has joined #openstack-meeting | 01:24 | |
| *** markvoelker has quit IRC | 01:29 | |
| *** baojg has joined #openstack-meeting | 01:36 | |
| *** yamamoto has quit IRC | 01:49 | |
| *** ykatabam has quit IRC | 01:56 | |
| *** bobh has quit IRC | 02:00 | |
| *** ianychoi has quit IRC | 02:25 | |
| *** ianychoi has joined #openstack-meeting | 02:26 | |
| *** yamamoto has joined #openstack-meeting | 03:07 | |
| *** whoami-rajat has joined #openstack-meeting | 03:12 | |
| *** markvoelker has joined #openstack-meeting | 03:26 | |
| *** markvoelker has quit IRC | 03:31 | |
| *** psachin has joined #openstack-meeting | 03:37 | |
| *** janki has joined #openstack-meeting | 03:47 | |
| *** janki has quit IRC | 03:54 | |
| *** janki has joined #openstack-meeting | 03:55 | |
| *** imsurit has joined #openstack-meeting | 04:02 | |
| *** ykatabam has joined #openstack-meeting | 04:07 | |
| *** janki has quit IRC | 04:13 | |
| *** janki has joined #openstack-meeting | 04:20 | |
| *** yamamoto has quit IRC | 04:51 | |
| *** yamamoto has joined #openstack-meeting | 04:53 | |
| *** davee__ has quit IRC | 05:03 | |
| *** davee_ has joined #openstack-meeting | 05:04 | |
| *** vishalmanchanda has joined #openstack-meeting | 05:18 | |
| *** akhil_jain has joined #openstack-meeting | 05:19 | |
| *** akhil_jain has left #openstack-meeting | 05:19 | |
| *** ykatabam has quit IRC | 05:27 | |
| *** markvoelker has joined #openstack-meeting | 05:28 | |
| *** ykatabam has joined #openstack-meeting | 05:29 | |
| *** markvoelker has quit IRC | 05:32 | |
| *** janki has quit IRC | 05:36 | |
| *** janki has joined #openstack-meeting | 05:42 | |
| *** janki has quit IRC | 05:43 | |
| *** janki has joined #openstack-meeting | 05:44 | |
| *** janki has quit IRC | 05:49 | |
| *** janki has joined #openstack-meeting | 05:49 | |
| *** janki has quit IRC | 05:53 | |
| *** janki has joined #openstack-meeting | 05:53 | |
| *** Luzi has joined #openstack-meeting | 05:56 | |
| *** ykatabam has quit IRC | 05:57 | |
| *** rcernin has quit IRC | 06:02 | |
| *** jbadiapa has joined #openstack-meeting | 06:07 | |
| *** brault has quit IRC | 06:11 | |
| *** janki has quit IRC | 06:18 | |
| *** apetrich has joined #openstack-meeting | 06:30 | |
| *** slaweq has joined #openstack-meeting | 06:32 | |
| *** kopecmartin|off is now known as kopecmartin | 06:38 | |
| *** ykatabam has joined #openstack-meeting | 06:57 | |
| *** rcernin has joined #openstack-meeting | 07:00 | |
| *** trident has quit IRC | 07:06 | |
| *** trident has joined #openstack-meeting | 07:08 | |
| *** _pewp_ has quit IRC | 07:12 | |
| *** _pewp_ has joined #openstack-meeting | 07:13 | |
| *** yamamoto has quit IRC | 07:16 | |
| *** tesseract has joined #openstack-meeting | 07:24 | |
| *** brault has joined #openstack-meeting | 07:28 | |
| *** markvoelker has joined #openstack-meeting | 07:29 | |
| *** markvoelker has quit IRC | 07:34 | |
| *** ykatabam has quit IRC | 07:38 | |
| *** ykatabam has joined #openstack-meeting | 07:39 | |
| *** yamamoto has joined #openstack-meeting | 07:48 | |
| *** ttsiouts has joined #openstack-meeting | 07:53 | |
| *** yamamoto has quit IRC | 07:57 | |
| *** ralonsoh has joined #openstack-meeting | 08:02 | |
| *** ykatabam has quit IRC | 08:02 | |
| *** yamamoto has joined #openstack-meeting | 08:08 | |
| *** cheng1 has quit IRC | 08:10 | |
| *** cheng1 has joined #openstack-meeting | 08:11 | |
| *** ricolin has joined #openstack-meeting | 08:12 | |
| *** ttsiouts has quit IRC | 08:14 | |
| *** ttsiouts has joined #openstack-meeting | 08:15 | |
| *** ykatabam has joined #openstack-meeting | 08:17 | |
| *** ttsiouts has quit IRC | 08:19 | |
| *** cheng1 has quit IRC | 08:20 | |
| *** iyamahat has joined #openstack-meeting | 08:20 | |
| *** ttsiouts has joined #openstack-meeting | 08:20 | |
| *** cheng1 has joined #openstack-meeting | 08:24 | |
| *** ykatabam has quit IRC | 08:28 | |
| *** markvoelker has joined #openstack-meeting | 08:30 | |
| *** markvoelker has quit IRC | 08:35 | |
| *** tssurya has joined #openstack-meeting | 08:44 | |
| *** ykatabam has joined #openstack-meeting | 08:44 | |
| *** jbadiapa has quit IRC | 08:50 | |
| *** pcaruana has joined #openstack-meeting | 08:52 | |
| *** tssurya has quit IRC | 08:55 | |
| *** jbadiapa has joined #openstack-meeting | 09:03 | |
| *** ykatabam has quit IRC | 09:08 | |
| *** ricolin has quit IRC | 09:14 | |
| *** ykatabam has joined #openstack-meeting | 09:21 | |
| *** yamamoto has quit IRC | 09:34 | |
| *** rcernin has quit IRC | 09:36 | |
| *** finucannot is now known as stephenfin | 09:36 | |
| *** baojg has quit IRC | 09:51 | |
| *** jbadiapa has quit IRC | 09:54 | |
| *** ociuhandu has joined #openstack-meeting | 09:56 | |
| *** yamamoto has joined #openstack-meeting | 10:05 | |
| *** jbadiapa has joined #openstack-meeting | 10:07 | |
| *** yamamoto has quit IRC | 10:15 | |
| *** markvoelker has joined #openstack-meeting | 10:32 | |
| *** yamamoto has joined #openstack-meeting | 10:34 | |
| *** markvoelker has quit IRC | 10:36 | |
| *** yamamoto has quit IRC | 10:36 | |
| *** yamamoto has joined #openstack-meeting | 10:39 | |
| *** yamamoto has quit IRC | 10:43 | |
| *** ykatabam has quit IRC | 10:55 | |
| *** ykatabam has joined #openstack-meeting | 10:59 | |
| *** ttsiouts has quit IRC | 11:08 | |
| *** yamamoto has joined #openstack-meeting | 11:08 | |
| *** ttsiouts has joined #openstack-meeting | 11:09 | |
| *** yamamoto has quit IRC | 11:12 | |
| *** ttsiouts has quit IRC | 11:13 | |
| *** ociuhandu has quit IRC | 11:17 | |
| *** panda is now known as panda|lunch | 11:23 | |
| *** ociuhandu has joined #openstack-meeting | 11:30 | |
| *** baojg has joined #openstack-meeting | 11:33 | |
| *** ttsiouts has joined #openstack-meeting | 11:39 | |
| *** imsurit has quit IRC | 11:43 | |
| *** yamamoto has joined #openstack-meeting | 11:49 | |
| *** ykatabam has quit IRC | 11:56 | |
| *** yamamoto has quit IRC | 11:56 | |
| *** yamamoto has joined #openstack-meeting | 11:57 | |
| *** rfolco has joined #openstack-meeting | 12:01 | |
| *** yamamoto has quit IRC | 12:02 | |
| *** edleafe has joined #openstack-meeting | 12:02 | |
| *** dviroel has joined #openstack-meeting | 12:05 | |
| *** yamamoto has joined #openstack-meeting | 12:14 | |
| *** yamamoto has quit IRC | 12:14 | |
| *** yamamoto has joined #openstack-meeting | 12:14 | |
| *** rtjure has quit IRC | 12:16 | |
| *** yamamoto has quit IRC | 12:19 | |
| *** panda|lunch is now known as panda | 12:25 | |
| *** bobh has joined #openstack-meeting | 12:33 | |
| *** markvoelker has joined #openstack-meeting | 12:33 | |
| *** bobh has quit IRC | 12:37 | |
| *** markvoelker has quit IRC | 12:38 | |
| *** lennyb has quit IRC | 12:40 | |
| *** njohnston has joined #openstack-meeting | 12:44 | |
| *** ricolin has joined #openstack-meeting | 12:45 | |
| *** psachin has quit IRC | 12:46 | |
| *** ekcs has quit IRC | 12:47 | |
| *** ekcs has joined #openstack-meeting | 12:48 | |
| *** vishalmanchanda has quit IRC | 12:57 | |
| *** lbragstad has joined #openstack-meeting | 13:12 | |
| *** ekcs_ has joined #openstack-meeting | 13:13 | |
| *** yamamoto has joined #openstack-meeting | 13:13 | |
| *** lennyb has joined #openstack-meeting | 13:13 | |
| *** ekcs has quit IRC | 13:14 | |
| *** ekcs_ is now known as ekcs | 13:14 | |
| *** raildo has joined #openstack-meeting | 13:15 | |
| *** ricolin has quit IRC | 13:17 | |
| *** beekneemech is now known as bnemec | 13:18 | |
| *** yamamoto has quit IRC | 13:19 | |
| *** mriedem has joined #openstack-meeting | 13:19 | |
| *** pcaruana|afk| has joined #openstack-meeting | 13:26 | |
| *** pcaruana has quit IRC | 13:26 | |
| *** markvoelker has joined #openstack-meeting | 13:34 | |
| *** markvoelker has quit IRC | 13:39 | |
| *** ricolin has joined #openstack-meeting | 13:40 | |
| *** apetrich has quit IRC | 13:43 | |
| *** dtrainor has quit IRC | 13:44 | |
| *** apetrich has joined #openstack-meeting | 13:45 | |
| *** slaweq has quit IRC | 13:47 | |
| *** pcaruana has joined #openstack-meeting | 13:48 | |
| *** dtrainor has joined #openstack-meeting | 13:49 | |
| *** belmoreira has joined #openstack-meeting | 13:50 | |
| *** pcaruana|afk| has quit IRC | 13:51 | |
| *** eharney has joined #openstack-meeting | 13:59 | |
| *** Luzi has quit IRC | 14:03 | |
| *** carloss has joined #openstack-meeting | 14:08 | |
| *** ircuser-1 has joined #openstack-meeting | 14:11 | |
| *** lseki has joined #openstack-meeting | 14:13 | |
| *** e0ne has joined #openstack-meeting | 14:21 | |
| *** belmoreira has quit IRC | 14:24 | |
| *** e0ne has quit IRC | 14:26 | |
| *** belmoreira has joined #openstack-meeting | 14:28 | |
| *** ayoung has quit IRC | 14:35 | |
| *** markvoelker has joined #openstack-meeting | 14:35 | |
| *** e0ne has joined #openstack-meeting | 14:36 | |
| *** belmoreira has quit IRC | 14:39 | |
| *** markvoelker has quit IRC | 14:40 | |
| *** belmoreira has joined #openstack-meeting | 14:44 | |
| *** jamesmcarthur has joined #openstack-meeting | 14:44 | |
| *** pcaruana has quit IRC | 14:47 | |
| *** bobh has joined #openstack-meeting | 14:50 | |
| *** helenafm has joined #openstack-meeting | 14:50 | |
| *** dklyle has joined #openstack-meeting | 14:55 | |
| *** wwriverrat has joined #openstack-meeting | 14:59 | |
| *** brault has quit IRC | 15:02 | |
| *** rubasov has quit IRC | 15:10 | |
| *** armax has joined #openstack-meeting | 15:10 | |
| *** armax has left #openstack-meeting | 15:11 | |
| *** rajinir has joined #openstack-meeting | 15:14 | |
| *** e0ne has quit IRC | 15:20 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 15:21 | |
| *** jrbalderrama has joined #openstack-meeting | 15:24 | |
| *** jamesmcarthur has quit IRC | 15:25 | |
| *** belmoreira has quit IRC | 15:28 | |
| *** temka is now known as artom | 15:35 | |
| *** tesseract has quit IRC | 15:38 | |
| *** pcaruana has joined #openstack-meeting | 15:40 | |
| *** wwriverrat has quit IRC | 15:41 | |
| *** helenafm has quit IRC | 15:47 | |
| *** ianychoi_ has joined #openstack-meeting | 15:48 | |
| *** gyee has joined #openstack-meeting | 15:49 | |
| *** gyee has quit IRC | 15:49 | |
| *** ianychoi has quit IRC | 15:52 | |
| *** slaweq has joined #openstack-meeting | 15:53 | |
| *** ttsiouts has quit IRC | 15:54 | |
| *** ttsiouts has joined #openstack-meeting | 15:55 | |
| *** trident has quit IRC | 15:56 | |
| *** trident has joined #openstack-meeting | 15:58 | |
| *** ttsiouts has quit IRC | 15:59 | |
| *** mlavalle has joined #openstack-meeting | 15:59 | |
| mlavalle | #startmeeting neutron_performance | 16:00 |
|---|---|---|
| openstack | Meeting started Mon Jun 17 16:00:11 2019 UTC and is due to finish in 60 minutes. The chair is mlavalle. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: neutron_performance)" | 16:00 | |
| openstack | The meeting name has been set to 'neutron_performance' | 16:00 |
| slaweq | o/ | 16:00 |
| haleyb | hi | 16:00 |
| jrbalderrama | hi! | 16:00 |
| mlavalle | njohnston: are you coming to the performance meeting? If you have a chance, please join us. I have a few questions or you | 16:01 |
| mlavalle | for you^^^ | 16:01 |
| *** rubasov has joined #openstack-meeting | 16:01 | |
| mlavalle | about OVO | 16:01 |
| mlavalle | ok, let's get going | 16:02 |
| mlavalle | #topic Updates | 16:02 |
| *** openstack changes topic to "Updates (Meeting topic: neutron_performance)" | 16:02 | |
| rubasov | late o/ | 16:02 |
| *** mattw4 has joined #openstack-meeting | 16:03 | |
| mlavalle | rubasov: just in time.... any updates for today? | 16:03 |
| rubasov | mlavalle: yes | 16:03 |
| rubasov | first we noticed the vm boot rally scenario to be quite unstable | 16:04 |
| * mlavalle already +2ed the related patch ;-) | 16:04 | |
| rubasov | so there's this change: https://review.opendev.org/665614 | 16:04 |
| rubasov | mlavalle: oh thanks | 16:04 |
| rubasov | then I have some news about profiling the vif plugging | 16:04 |
| mlavalle | I also rechecked it | 16:05 |
| rubasov | thank you | 16:05 |
| rubasov | I looked into the osprofiler report and realized there's nothing in it about when a vif plug finishes | 16:05 |
| rubasov | after some debuggign we realized the cause | 16:05 |
| rubasov | as you know osprofiler requires the trace information (trace id, hmac, parent id, etc) | 16:06 |
| mlavalle | yes | 16:06 |
| rubasov | to be passed on in each intra-component communication | 16:06 |
| rubasov | this is usually passed on in each http message | 16:06 |
| rubasov | or rpc message | 16:06 |
| mlavalle | yeap | 16:06 |
| rubasov | but the vif plug is the exception | 16:07 |
| * njohnston is here, apologize for being late | 16:07 | |
| mlavalle | there are not http calls, right? | 16:07 |
| rubasov | because there the communication channel is the ovsdb (at least for the ovs plugin) | 16:07 |
| * bcafarel also late and sorry for it | 16:07 | |
| rubasov | we pass on the port uuid in the external_ids fields of the port record in ovsdb | 16:08 |
| mlavalle | exactly | 16:08 |
| *** jrbalderrama has quit IRC | 16:08 | |
| rubasov | but we cannot really do the same with the trace info | 16:08 |
| *** jrbalderrama has joined #openstack-meeting | 16:08 | |
| rubasov | first because libvirt does not allow arbitrary external_ids | 16:08 |
| rubasov | second because libvirt deletes and recreates the ovs port | 16:09 |
| rubasov | so whatver we put into it gets deleted by libvirt | 16:09 |
| rubasov | we're thinking about a hacky way | 16:09 |
| rubasov | to serlialize the trace info into the bridges table of ovsdb when os-vif plugs | 16:10 |
| rubasov | and remove it by neutron-ovs-agent as soon as it triggered tracing in ovs-agent | 16:10 |
| rubasov | but as you can see it's quite hacky | 16:10 |
| mlavalle | yeah | 16:10 |
| *** gyee has joined #openstack-meeting | 16:11 | |
| rubasov | if you see a better way please let us know | 16:11 |
| mlavalle | let's finish the updates section of the meeting and we'll come back to it in a few minutes | 16:11 |
| mlavalle | ok? | 16:11 |
| rubasov | ok | 16:11 |
| mlavalle | does anybody else have more annuncements? | 16:12 |
| mlavalle | updates? | 16:12 |
| slaweq | rubasov: I think that sean-k-mooney was working on some patch to nova/os-vif to make all plug to be done on os-vif side | 16:12 |
| slaweq | and than libvirt would not recreate it | 16:12 |
| slaweq | there were some problems with this but You can ask him about current status | 16:12 |
| rubasov | slaweq: that was reverted I think | 16:12 |
| slaweq | yes, it was but I think he was still trying to do something like that later | 16:13 |
| rubasov | I've seen the various patches fixing the security issue | 16:13 |
| rubasov | slaweq: okay, will ask anyway, because we'll have os-vif changes likely | 16:13 |
| mlavalle | ok, I assume there are no more updates.... | 16:13 |
| mlavalle | #topic EnOS framework | 16:14 |
| *** openstack changes topic to "EnOS framework (Meeting topic: neutron_performance)" | 16:14 | |
| mlavalle | jrbalderrama: thanks for joining us. Is there any progress with the requests I made last week? | 16:14 |
| jrbalderrama | Yes! | 16:14 |
| jrbalderrama | msimonin sent a mail today addressing those questions | 16:15 |
| mlavalle | ahh, ok, just a few minutes ago | 16:15 |
| mlavalle | I checked my inbox just before that | 16:15 |
| jrbalderrama | It was 20 min before the meeting starts | 16:16 |
| mlavalle | no problem. Thanks! | 16:16 |
| mlavalle | I just glanced over it quickly and it seems he followed up on all the pending points | 16:17 |
| mlavalle | I will restart my testing soon with that and will yell if I need help | 16:17 |
| mlavalle | ok? | 16:17 |
| jrbalderrama | right | 16:17 |
| mlavalle | Thank you very much for the follow up | 16:18 |
| mlavalle | anything else we should say today about EnOS? | 16:18 |
| jrbalderrama | nothing for today | 16:18 |
| mlavalle | cool | 16:18 |
| mlavalle | let's move on | 16:18 |
| mlavalle | ahh, I have one more request.... | 16:19 |
| mlavalle | rubasov: what is the status of merging you port binding scenario in the openstack rally repo? | 16:19 |
| *** kopecmartin is now known as kopecmartin|off | 16:20 | |
| rubasov | mlavalle: as soon as zuul votes +1 I wanted to send a mail to rally folks asking for review | 16:20 |
| mlavalle | jrbalderrama: ^^^ given what rubasov just said, could I get instructions on how to add an unmerged Rally scenario to EnOS? | 16:21 |
| mlavalle | I tried breifly a few days ago and couldn't figure it out | 16:21 |
| jrbalderrama | We will send you the instructions by mail | 16:22 |
| * mlavalle needs to learn more about the containerized deployment | 16:22 | |
| mlavalle | cool, thanks! | 16:22 |
| mlavalle | #topic Rally job and osprofiler | 16:23 |
| *** openstack changes topic to "Rally job and osprofiler (Meeting topic: neutron_performance)" | 16:23 | |
| mlavalle | I sepent some time at reports produced by the VM boot scenario: http://logs.openstack.org/14/665614/2/check/neutron-rally-task/cc6f223/results/report.html.gz#/NovaServers.boot_and_delete_server/output | 16:23 |
| *** njohnston is now known as njohnston__ | 16:23 | |
| *** njohnston_ has joined #openstack-meeting | 16:23 | |
| *** njohnston_ is now known as njohnston | 16:24 | |
| mlavalle | rubasov, slaweq: I have to say that you guys have done a great job getting this scenario together and also in getting the osprofiler report in good shape | 16:24 |
| mlavalle | the reporting has improved since the PTG | 16:24 |
| slaweq | mlavalle: thx :) | 16:25 |
| rubasov | slaweq has done the bigger part | 16:25 |
| mlavalle | now, looking at the report, I noted that it is generated from the point of view of Nova | 16:25 |
| slaweq | rubasov: not exactly, it was mostly done by Andrey from rally team and kudos should go to him :) | 16:25 |
| rubasov | slaweq: thanks to Andrey | 16:26 |
| *** lbragstad has quit IRC | 16:26 | |
| mlavalle | what I mean by that is that I can see the conversation between Nova and the Neutron server, and work being done by the Neutron server | 16:26 |
| *** yamamoto has joined #openstack-meeting | 16:27 | |
| mlavalle | by I didn't find the the conversation between neutron server and the L2 agent | 16:27 |
| mlavalle | did I miss something? | 16:27 |
| mlavalle | rubasov: ^^^ | 16:27 |
| rubasov | mlavalle: isn't this about what we started to discuss rigt at the beginning of this meeting? | 16:27 |
| mlavalle | yes | 16:28 |
| *** cmurphy_afk is now known as cmurphy | 16:28 | |
| *** lbragstad has joined #openstack-meeting | 16:28 | |
| rubasov | all stuff that's related to a particular trace id | 16:28 |
| rubasov | that is running in ovs-agent | 16:28 |
| rubasov | is kicked off by the plug | 16:28 |
| rubasov | so if we can't get the trace info over into the agent | 16:29 |
| rubasov | then the agent's work is not really traced | 16:29 |
| mlavalle | yeah, but the trace-id passes between neutron server and the L2 agent through the rpc channel, doesn't it | 16:29 |
| mlavalle | in other scenarios I have seen it | 16:29 |
| *** whoami-rajat has quit IRC | 16:29 | |
| *** njohnston has quit IRC | 16:30 | |
| *** panda is now known as panda|off | 16:30 | |
| rubasov | the only message where we could pass it is get_device_details I think | 16:30 |
| *** njohnston__ is now known as njohnston | 16:30 | |
| rubasov | but can neutron server tell which trace id corresponds to which get_device_details call? | 16:30 |
| slaweq | rubasov: I don't think so as it base on what is stored in external_ids, right? | 16:31 |
| rubasov | I'm not sure if that's doable without the agent passing the trace id first | 16:31 |
| *** yamamoto has quit IRC | 16:31 | |
| mlavalle | ok, I just wanted to bring this up | 16:31 |
| mlavalle | that's why I asked you to hold for a few minutes | 16:32 |
| rubasov | I'm not 100% sure | 16:32 |
| mlavalle | ideally, we would like to trace the entire cycle | 16:32 |
| rubasov | but currently I think neutron server does not have enough information to make the correspondance between get_device_details and the trace_id | 16:32 |
| mlavalle | and maybe we need to investigate how to do it | 16:32 |
| rubasov | and that's what we hope to do by passing it in ovsdb | 16:33 |
| mlavalle | and of course one possible answer is that we need to invent it | 16:33 |
| mlavalle | yeah, I understand that | 16:33 |
| rubasov | we hope to come up with some proof of concept code for it | 16:34 |
| mlavalle | so, will you continue looking into it? | 16:34 |
| rubasov | both in the os-vif side | 16:34 |
| mlavalle | ok, cool | 16:34 |
| rubasov | and on the ovs-agent side | 16:34 |
| mlavalle | perfect | 16:34 |
| rubasov | yes, I will | 16:34 |
| mlavalle | Tnanks! | 16:34 |
| rubasov | np | 16:34 |
| mlavalle | anything else on this topic? | 16:34 |
| rubasov | that's all from me | 16:35 |
| rubasov | and sorry for hijacking the meeting right at the beginning | 16:35 |
| mlavalle | oh no problem.... | 16:35 |
| *** ricolin has quit IRC | 16:35 | |
| mlavalle | on the contrary, very useful | 16:36 |
| mlavalle | #topic Security groups RBAC | 16:36 |
| *** openstack changes topic to "Security groups RBAC (Meeting topic: neutron_performance)" | 16:36 | |
| mlavalle | I spent a good chunk of the last 2 days looking at https://bugs.launchpad.net/neutron/+bug/1830679 | 16:36 |
| openstack | Launchpad bug 1830679 in neutron "Security groups RBAC cause a major performance degradation" [High,In progress] - Assigned to Miguel Lavalle (minsel) | 16:36 |
| *** markvoelker has joined #openstack-meeting | 16:37 | |
| mlavalle | I left extensive comments on my findings in notes #9, #11 and #12 | 16:38 |
| mlavalle | the summary for users who have lots of sec groups and are not using RBAC with them, we can optimize the code, as proposed here https://review.opendev.org/#/c/665566/ | 16:39 |
| mlavalle | essentially, we don't need to check whether the user has access right for a sec group if the secgroup has no RBAC entries associated with it | 16:40 |
| mlavalle | that cuts the response time from of an "openstack security group list" from ~500 seconds to 50 seconds in a scenario where there are +6000 security groups | 16:41 |
| mlavalle | and that's good but... | 16:42 |
| * bcafarel waits for the OVO "but" | 16:42 | |
| *** markvoelker has quit IRC | 16:42 | |
| mlavalle | in my digging, I found that with a straight SQLAlchemy query, we can retrieve all the secgroups in less than a second | 16:43 |
| *** ekcs has quit IRC | 16:43 | |
| * slaweq needs to leave earlier | 16:43 | |
| njohnston | I have had concerns about OVO being a source of slowness for some time; placement has seen significant speed benefits from removing OVO | 16:43 |
| mlavalle | but most of those remaining 50 seconds are sspent in the sec groups query through OVO | 16:43 |
| mlavalle | about 46 secs | 16:44 |
| *** igordc has joined #openstack-meeting | 16:44 | |
| mlavalle | ok, I wanted njohnston to comment on my findings | 16:44 |
| njohnston | This is the best, most clear-cut case of OVO being the primary culprit for a slowdown | 16:44 |
| mlavalle | yeah, in the end, OVO should be issuing the same SQLAlchemy that I am using, right? | 16:45 |
| mlavalle | SQLAlchemy query^^^ | 16:45 |
| njohnston | yes, or something similar. The slowdown could be pre-query, in which case baked queries might help, or it could be post query and we'd have to see if the process of processing results into objects needs work | 16:46 |
| mlavalle | exactly | 16:47 |
| *** slaweq has quit IRC | 16:47 | |
| njohnston | do the logs perchance indicate how far along the query is actually emitted? | 16:47 |
| mlavalle | I didn't get that far along | 16:47 |
| *** whoami-rajat has joined #openstack-meeting | 16:48 | |
| mlavalle | it was late on Father's day and my wife had cooked a special dinner for me, so it was difficult to continue beyond the early evening | 16:48 |
| mlavalle | but I will investigate | 16:48 |
| mlavalle | There are two pints I want to make in this meeting: | 16:49 |
| mlavalle | points^^^^ | 16:49 |
| mlavalle | 1) short term, I might fix this bug with using staright SQLAlchemy here: https://review.opendev.org/#/c/665566/1/neutron/objects/rbac_db.py@113 | 16:50 |
| mlavalle | 2) long term, OVO seems a ripe area of work for this performance team | 16:51 |
| mlavalle | I'm not saying get rid of it, but we need to investigate how to improve its performance | 16:51 |
| mlavalle | and understand better the trade off between the OVO benefits and its performance cost | 16:52 |
| mlavalle | does it make sense | 16:52 |
| mlavalle | ? | 16:52 |
| njohnston | One point I would like to make about your scripts is that the SQL Alchemy script only lists security groups, but the OVO script creates projects and creates security groups before listing them, and the destroys them after. | 16:52 |
| bcafarel | sounds good | 16:52 |
| njohnston | We should see if we can track the elapsed time just to list the security groups to provide an apples-to-apples comparison | 16:53 |
| *** Lucas_Gray has joined #openstack-meeting | 16:53 | |
| mlavalle | yeap, understood | 16:53 |
| *** SWDevAngel has joined #openstack-meeting | 16:53 | |
| *** ricolin has joined #openstack-meeting | 16:53 | |
| njohnston | but yes, I think we need to try to run cProfile or other tools to dig out what is taking such a long time, and how it can be optimized. | 16:54 |
| njohnston | especially if the improvement can be in the OVO library itself so we don't have to make tactical fixes in the hundred places we already implemented OVO | 16:54 |
| mlavalle | and btw, that's what I mean by striking the right balance between OVO benefits and its performance cost | 16:55 |
| mlavalle | and you are right, we have to try to avoid trying to come up with a thousand tactical fixes | 16:55 |
| mlavalle | in the case of this specific bug, tactical is good if we want to avoid a painful revert | 16:56 |
| njohnston | right | 16:56 |
| mlavalle | talking about https://review.opendev.org/#/c/665447/ | 16:56 |
| bcafarel | I suppose yeah if placement has OVO perf issue too, better to dig in it directly than looking for neutron-specific perf issues | 16:56 |
| njohnston | like I said, placement saw definite speed improvements just be removing OVO: https://anticdent.org/placement-update-19-08.html | 16:57 |
| mlavalle | yes, and that's what I meant in point 2 above^^^ | 16:57 |
| njohnston | and I recall a reference to a similar performance gain in another project, although I cannot find the link now | 16:57 |
| mlavalle | cool, good stuff! | 16:58 |
| mlavalle | anything else? | 16:59 |
| mlavalle | ok, we ran out of time | 16:59 |
| mlavalle | #endmeeting | 16:59 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:59 | |
| openstack | Meeting ended Mon Jun 17 16:59:28 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:59 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_performance/2019/neutron_performance.2019-06-17-16.00.html | 16:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_performance/2019/neutron_performance.2019-06-17-16.00.txt | 16:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_performance/2019/neutron_performance.2019-06-17-16.00.log.html | 16:59 |
| rubasov | thank you | 16:59 |
| rubasov | bye | 16:59 |
| mlavalle | Thanks for attending! | 16:59 |
| jrbalderrama | Bye | 16:59 |
| *** slaweq has joined #openstack-meeting | 17:04 | |
| *** zbr|ooo has quit IRC | 17:04 | |
| *** zbr has joined #openstack-meeting | 17:05 | |
| *** zbr is now known as zbr|ruck | 17:06 | |
| *** iyamahat has quit IRC | 17:07 | |
| *** yamahata has quit IRC | 17:07 | |
| *** dtroyer has joined #openstack-meeting | 17:08 | |
| *** slaweq has quit IRC | 17:08 | |
| *** iyamahat has joined #openstack-meeting | 17:17 | |
| *** iyamahat_ has joined #openstack-meeting | 17:20 | |
| *** jrbalderrama has quit IRC | 17:20 | |
| *** iyamahat has quit IRC | 17:23 | |
| *** yamahata has joined #openstack-meeting | 17:25 | |
| *** artom has quit IRC | 17:26 | |
| *** trident has quit IRC | 17:27 | |
| *** trident has joined #openstack-meeting | 17:29 | |
| *** ekcs has joined #openstack-meeting | 17:30 | |
| *** ricolin has quit IRC | 17:30 | |
| *** wwriverrat has joined #openstack-meeting | 17:31 | |
| *** markvoelker has joined #openstack-meeting | 17:37 | |
| *** markvoelker has quit IRC | 17:43 | |
| *** jamesmcarthur_ has quit IRC | 17:57 | |
| *** freerunner has quit IRC | 17:57 | |
| *** freerunner has joined #openstack-meeting | 18:00 | |
| *** Lucas_Gray has quit IRC | 18:01 | |
| *** mattw4 has quit IRC | 18:07 | |
| *** mattw4 has joined #openstack-meeting | 18:07 | |
| *** ralonsoh has quit IRC | 18:09 | |
| *** mattw4 has quit IRC | 18:12 | |
| *** bauzas has quit IRC | 18:21 | |
| *** bauzas has joined #openstack-meeting | 18:27 | |
| *** niceplace has quit IRC | 18:30 | |
| *** niceplace has joined #openstack-meeting | 18:31 | |
| *** artom has joined #openstack-meeting | 18:32 | |
| *** yamamoto has joined #openstack-meeting | 18:36 | |
| *** markvoelker has joined #openstack-meeting | 18:38 | |
| *** pcaruana has quit IRC | 18:44 | |
| *** markvoelker has quit IRC | 18:45 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:51 | |
| *** diablo_rojo has joined #openstack-meeting | 19:03 | |
| *** slaweq has joined #openstack-meeting | 19:05 | |
| *** pcaruana has joined #openstack-meeting | 19:17 | |
| *** yamamoto has quit IRC | 19:21 | |
| *** yamamoto has joined #openstack-meeting | 19:25 | |
| *** pcaruana has quit IRC | 19:27 | |
| *** ayoung has joined #openstack-meeting | 19:30 | |
| *** yamamoto has quit IRC | 19:30 | |
| *** eharney has quit IRC | 19:30 | |
| *** trident has quit IRC | 19:37 | |
| *** trident has joined #openstack-meeting | 19:39 | |
| *** yamamoto has joined #openstack-meeting | 19:39 | |
| *** yamamoto has quit IRC | 19:39 | |
| *** yamamoto has joined #openstack-meeting | 19:40 | |
| *** markvoelker has joined #openstack-meeting | 19:40 | |
| *** dtrainor has quit IRC | 19:41 | |
| *** dtrainor has joined #openstack-meeting | 19:41 | |
| *** jamesmcarthur has quit IRC | 19:42 | |
| *** yamamoto has quit IRC | 19:45 | |
| *** markvoelker has quit IRC | 19:45 | |
| *** mattw4 has joined #openstack-meeting | 19:50 | |
| *** zaneb has quit IRC | 19:52 | |
| *** zaneb has joined #openstack-meeting | 19:52 | |
| *** SWDevAngel has quit IRC | 20:03 | |
| *** e0ne has joined #openstack-meeting | 20:07 | |
| *** e0ne has quit IRC | 20:07 | |
| *** Lucas_Gray has joined #openstack-meeting | 20:11 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:15 | |
| *** dtrainor_ has joined #openstack-meeting | 20:19 | |
| *** whoami-rajat has quit IRC | 20:19 | |
| *** dtrainor has quit IRC | 20:22 | |
| *** notmyname has quit IRC | 20:26 | |
| *** jamesmcarthur has quit IRC | 20:28 | |
| *** notmyname has joined #openstack-meeting | 20:32 | |
| *** mlavalle has left #openstack-meeting | 20:35 | |
| *** bobh has quit IRC | 20:36 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:37 | |
| *** jamesmcarthur has quit IRC | 20:41 | |
| *** markvoelker has joined #openstack-meeting | 20:41 | |
| *** ekcs has quit IRC | 20:42 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:42 | |
| *** markvoelker has quit IRC | 20:46 | |
| *** Lucas_Gray has quit IRC | 20:46 | |
| *** artom has quit IRC | 20:59 | |
| *** jamesmcarthur has quit IRC | 21:01 | |
| *** mattw4 has quit IRC | 21:04 | |
| *** mattw4 has joined #openstack-meeting | 21:04 | |
| *** njohnston has quit IRC | 21:12 | |
| *** jamesmcarthur has joined #openstack-meeting | 21:15 | |
| *** rfolco has quit IRC | 21:16 | |
| *** mattw4 has quit IRC | 21:22 | |
| *** mattw4 has joined #openstack-meeting | 21:22 | |
| *** eharney has joined #openstack-meeting | 21:30 | |
| *** markvoelker has joined #openstack-meeting | 21:42 | |
| *** rcernin has joined #openstack-meeting | 21:43 | |
| *** markvoelker has quit IRC | 21:47 | |
| *** raildo has quit IRC | 21:50 | |
| *** slaweq has quit IRC | 21:51 | |
| *** slaweq has joined #openstack-meeting | 22:03 | |
| *** slaweq has quit IRC | 22:09 | |
| *** ChanServ has quit IRC | 22:09 | |
| *** ChanServ has joined #openstack-meeting | 22:14 | |
| *** niven.freenode.net sets mode: +o ChanServ | 22:14 | |
| *** jamesmcarthur has quit IRC | 22:22 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:22 | |
| *** mriedem has quit IRC | 22:27 | |
| *** slaweq has joined #openstack-meeting | 22:36 | |
| *** slaweq has quit IRC | 22:41 | |
| *** markvoelker has joined #openstack-meeting | 22:43 | |
| *** ykatabam has joined #openstack-meeting | 22:45 | |
| *** markvoelker has quit IRC | 22:48 | |
| *** Adri2000 has quit IRC | 23:07 | |
| *** Adri2000 has joined #openstack-meeting | 23:11 | |
| *** gyee has quit IRC | 23:28 | |
| *** slaweq has joined #openstack-meeting | 23:33 | |
| *** slaweq has quit IRC | 23:40 | |
| *** artom has joined #openstack-meeting | 23:43 | |
| *** markvoelker has joined #openstack-meeting | 23:44 | |
| *** markvoelker has quit IRC | 23:48 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!