| *** roeyc has quit IRC | 00:21 | |
| *** gangil has quit IRC | 00:26 | |
| *** gangil has joined #openstack-neutron-ovn | 00:30 | |
| *** gangil has joined #openstack-neutron-ovn | 00:30 | |
| *** chandrav has joined #openstack-neutron-ovn | 00:35 | |
| *** gangil has quit IRC | 00:36 | |
| *** roeyc has joined #openstack-neutron-ovn | 00:40 | |
| *** salv-orlando has quit IRC | 00:43 | |
| *** roeyc has quit IRC | 00:44 | |
| *** roeyc has joined #openstack-neutron-ovn | 00:45 | |
| *** zhouhan has joined #openstack-neutron-ovn | 00:47 | |
| *** roeyc has quit IRC | 00:47 | |
| *** armax has quit IRC | 00:48 | |
| *** roeyc has joined #openstack-neutron-ovn | 00:56 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 01:04 | |
| *** salv-orlando has quit IRC | 01:09 | |
| *** manand has quit IRC | 01:12 | |
| *** roeyc has quit IRC | 01:13 | |
| openstackgerrit | Merged openstack/networking-ovn: Expose ovs-vswitchd log to file https://review.openstack.org/269909 | 01:27 |
|---|---|---|
| *** dslev has joined #openstack-neutron-ovn | 01:30 | |
| *** dslev_ has joined #openstack-neutron-ovn | 01:32 | |
| *** dslev has quit IRC | 01:35 | |
| *** dslev_ has quit IRC | 01:55 | |
| *** arosen has quit IRC | 02:08 | |
| *** chandrav has quit IRC | 02:17 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 02:18 | |
| *** fzdarsky has quit IRC | 02:18 | |
| *** fzdarsky has joined #openstack-neutron-ovn | 02:25 | |
| *** yamamoto_ has quit IRC | 02:38 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 02:39 | |
| openstackgerrit | Merged openstack/networking-ovn: Fix in create_router and update_router https://review.openstack.org/268722 | 02:41 |
| *** yamamoto_ has quit IRC | 02:55 | |
| *** zhouhan has quit IRC | 02:57 | |
| *** jckasper has quit IRC | 03:04 | |
| *** s3wong has quit IRC | 03:07 | |
| *** manand has joined #openstack-neutron-ovn | 03:28 | |
| *** yamamoto_ has joined #openstack-neutron-ovn | 03:47 | |
| *** chenli has joined #openstack-neutron-ovn | 04:23 | |
| *** armax has joined #openstack-neutron-ovn | 05:37 | |
| *** azbiswas has joined #openstack-neutron-ovn | 06:07 | |
| *** manand has quit IRC | 06:10 | |
| *** chandrav has joined #openstack-neutron-ovn | 06:18 | |
| *** numans has joined #openstack-neutron-ovn | 06:19 | |
| openstackgerrit | Merged openstack/networking-ovn: HOST_IP is missing in computenode-local.conf.sample https://review.openstack.org/270594 | 06:23 |
| *** salv-orlando has joined #openstack-neutron-ovn | 06:44 | |
| *** azbiswas has quit IRC | 07:01 | |
| *** salv-orlando has quit IRC | 07:09 | |
| *** armax has quit IRC | 07:18 | |
| *** armax has joined #openstack-neutron-ovn | 07:24 | |
| *** chenli has quit IRC | 07:24 | |
| *** chenli has joined #openstack-neutron-ovn | 07:24 | |
| *** chandrav has quit IRC | 07:29 | |
| *** armax has quit IRC | 07:43 | |
| *** fzdarsky has quit IRC | 07:57 | |
| *** fzdarsky has joined #openstack-neutron-ovn | 07:57 | |
| *** numans has quit IRC | 08:32 | |
| *** numans has joined #openstack-neutron-ovn | 09:16 | |
| openstackgerrit | Numan Siddique proposed openstack/networking-ovn: Store all the fixed ips of a port in one entry in Logical_Port.addresses https://review.openstack.org/271226 | 10:33 |
| *** yamamoto_ has quit IRC | 11:25 | |
| *** chenli_ has joined #openstack-neutron-ovn | 11:28 | |
| *** chenli has quit IRC | 11:30 | |
| *** chenli_ is now known as chenli | 11:30 | |
| *** openstackgerrit has quit IRC | 11:47 | |
| *** openstackgerrit has joined #openstack-neutron-ovn | 11:47 | |
| *** numans has quit IRC | 11:53 | |
| *** yamamoto has joined #openstack-neutron-ovn | 12:09 | |
| *** chenli has quit IRC | 12:10 | |
| *** yamamoto has quit IRC | 12:13 | |
| *** yamamoto has joined #openstack-neutron-ovn | 12:15 | |
| *** rtheis has joined #openstack-neutron-ovn | 12:24 | |
| *** yamamoto has quit IRC | 12:32 | |
| *** yamamoto has joined #openstack-neutron-ovn | 12:32 | |
| *** yamamoto has quit IRC | 12:33 | |
| *** openstackgerrit has quit IRC | 12:33 | |
| *** yamamoto has joined #openstack-neutron-ovn | 12:33 | |
| *** openstackgerrit has joined #openstack-neutron-ovn | 12:33 | |
| *** yamamoto has quit IRC | 12:46 | |
| *** yamamoto has joined #openstack-neutron-ovn | 12:46 | |
| *** numans has joined #openstack-neutron-ovn | 12:49 | |
| *** yamamoto has quit IRC | 12:51 | |
| *** yamamoto has joined #openstack-neutron-ovn | 13:14 | |
| *** dslev_ has joined #openstack-neutron-ovn | 14:06 | |
| Sam-I-Am | russellb: morningses | 14:07 |
| russellb | morning, in meetings and such until afternoon | 14:09 |
| Sam-I-Am | doh | 14:10 |
| Sam-I-Am | well, when you get some time | 14:10 |
| Sam-I-Am | otherwise, have a fun morning :) | 14:10 |
| *** flaviof has quit IRC | 14:28 | |
| *** regXboi has joined #openstack-neutron-ovn | 14:29 | |
| *** dslev_ has quit IRC | 14:39 | |
| *** manand has joined #openstack-neutron-ovn | 14:42 | |
| regXboi | russellb: ping? | 14:47 |
| *** pcaruana has joined #openstack-neutron-ovn | 14:48 | |
| russellb | pong | 14:52 |
| *** dslev_ has joined #openstack-neutron-ovn | 14:52 | |
| *** yamamoto has quit IRC | 14:52 | |
| russellb | Sam-I-Am: do you think you could write up something about your mtu ideas? sounds like something maybe we should discuss on the ovs dev list if there's changes needed in ovn? | 14:53 |
| regXboi | I'm reading OVS tea leaves this morning and want to make sure I'm reading them close to correct | 14:53 |
| russellb | i could use some tea before this next meeting | 14:53 |
| russellb | but i think i'll get coffee instead (brb) | 14:53 |
| regXboi | so (1) the insert idl operation from networking-ovn comes to ovn-controller via an update2 jsonRPC method | 14:55 |
| Sam-I-Am | russellb: i wrote it up on the -dev list as it pertains to linuxbridge, but the same concepts apply to any virtual networking with tunnels | 14:55 |
| regXboi | (2) the code the processes that method does a loop over all of the possible ops (which allows a single jsonRPC call to do multiple things) | 14:55 |
| regXboi | and then each found ops is processed separately by ovsdb_idl_process_update2 | 14:56 |
| regXboi | does that sound about right? | 14:56 |
| *** shettyg has joined #openstack-neutron-ovn | 14:56 | |
| russellb | Sam-I-Am: ok i'll try to catch up | 14:59 |
| russellb | regXboi: i'd have to read the code | 15:00 |
| Sam-I-Am | russellb: its pretty involved, but there's a tl;dr at the beginning that might help. | 15:02 |
| Sam-I-Am | some of the background might be useful too if you dont mess with mtu much | 15:03 |
| regXboi | russellb: well, don't do that - you've got more important things to chase down - I'll make the assumption that I'm correct and add some decoration to see about verifying it | 15:05 |
| regXboi | ok, yes I've verified that I read the code correctly... now to get enough instrumentation in that I can put everything together | 15:22 |
| russellb | Sam-I-Am: i dont ... i live in nicely theoretical dev land | 15:26 |
| Sam-I-Am | russellb: lol. i was a network engineer for a while. | 15:27 |
| Sam-I-Am | doing things that should not be done, like stuffing ethernet into T1s | 15:27 |
| *** slowe has joined #openstack-neutron-ovn | 15:28 | |
| *** numans has quit IRC | 15:28 | |
| *** dslev_ has quit IRC | 15:57 | |
| *** rtheis has quit IRC | 16:05 | |
| *** rtheis has joined #openstack-neutron-ovn | 16:05 | |
| *** azbiswas has joined #openstack-neutron-ovn | 16:07 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 16:08 | |
| *** salv-orlando has quit IRC | 16:12 | |
| *** armax has joined #openstack-neutron-ovn | 16:33 | |
| *** lrichard has joined #openstack-neutron-ovn | 16:47 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 16:47 | |
| *** dslevin has joined #openstack-neutron-ovn | 16:57 | |
| *** gangil has joined #openstack-neutron-ovn | 17:14 | |
| *** gangil has joined #openstack-neutron-ovn | 17:14 | |
| *** roeyc has joined #openstack-neutron-ovn | 17:16 | |
| *** roeyc has quit IRC | 17:17 | |
| *** roeyc has joined #openstack-neutron-ovn | 17:20 | |
| mamulsow | russellb: I ran a test where I tried putting 4k routers connected to one shared private network, things got really bad around 1200 routers connected to that single network/subnet | 17:32 |
| mamulsow | by really bad I mean pretty much all neutron calls started hanging, and ovn-northd died | 17:33 |
| mamulsow | regXboi tells me that it's something of a known issue that too many ports on one subnet is a performance problem | 17:34 |
| mamulsow | do you want a bug for this or is it just a known issue? | 17:34 |
| * regXboi notes use of handle and wanders in | 17:35 | |
| russellb | mamulsow: so you had N networks with N corresponding routers all connected to a single shared network | 17:42 |
| russellb | and it went boom | 17:42 |
| regXboi | russellb: pretty much, yes | 17:42 |
| russellb | no, i wouldn't say that's a known issue exactly | 17:42 |
| russellb | ovn-northd should certainly recover if you stopped creating new stuff | 17:43 |
| russellb | sounds like it got to a point where it couldnt' keep up though | 17:43 |
| mamulsow | well, I think the bad stuff wasn't from the N routers connected to N networks, I think it was the N networks connected to 1 network | 17:43 |
| russellb | right | 17:43 |
| mamulsow | sorry, N routers connected to 1 network | 17:44 |
| russellb | yes | 17:44 |
| russellb | sounds like an important thing to profile and work on for tenant network support | 17:44 |
| mamulsow | each of the 1200 routers had two interfaces, 1 to a private network that was 1:1 with the routers and the other interface was to a single shared network | 17:44 |
| russellb | maybe lower priority if your interest is in provider networks to start | 17:45 |
| russellb | right, so this would be like if you had 1200 tenants | 17:45 |
| russellb | all connected to the same shared public net or whatever | 17:45 |
| mamulsow | yep | 17:45 |
| russellb | makes sense | 17:45 |
| Sam-I-Am | which is a common deployment method | 17:45 |
| russellb | yes | 17:45 |
| russellb | the most common, even | 17:45 |
| russellb | AFAICT | 17:45 |
| russellb | though not with OVN yet since we don't have NAT.. | 17:46 |
| Sam-I-Am | its less wasteful of ips | 17:46 |
| mamulsow | well, normally that shared network would be an external network, not a regular private network | 17:46 |
| russellb | mamulsow: right | 17:46 |
| russellb | i'd probably want to profile ovn-northd on this one | 17:47 |
| mamulsow | so I'm not sure what the impact of this being another router interface instead of a router gateway to an external network | 17:47 |
| russellb | it's probably just not keeping up with the number of logical flows needed to descirbe this | 17:47 |
| mamulsow | yeah, there wasn't anything useful in the ovn-northd log after this happened | 17:48 |
| mamulsow | I think I can easily reproduce though | 17:48 |
| Sam-I-Am | mamulsow: so the networks on both ends of the routers are private? | 17:48 |
| mamulsow | right | 17:48 |
| Sam-I-Am | hmm... its probably a similar set of operations | 17:49 |
| mamulsow | wouldn't be a real world situation, but figured it would be close to simulating what we would do with NAT | 17:49 |
| Sam-I-Am | if you're not testing ip traffic, couldn't you test the conventional way of connecting each router to a provider net? | 17:49 |
| Sam-I-Am | just testing creation times | 17:49 |
| regXboi | Sam-I-Am: I'm already pretty much doing that | 17:50 |
| -openstackstatus- NOTICE: Restarting zuul due to a memory leak | 17:51 | |
| russellb | mamulsow: yes, it's a good simulation | 17:51 |
| russellb | though there's kind of 2 paths right now for private networks and NAT | 17:52 |
| russellb | one of them looks like how ML2+OVS usess provider networks today | 17:52 |
| russellb | and the other is using OVN gateways | 17:52 |
| russellb | which wouldn't necessarily involve logical routers for every tenant network | 17:52 |
| regXboi | russellb: is there a helper method to dump a osvdb_idl_row structure as a string? | 17:52 |
| russellb | regXboi: you looking at the C code? | 17:53 |
| regXboi | russellb: ack - I'm looking at the C code | 17:53 |
| russellb | I don't know | 17:53 |
| Sam-I-Am | regXboi mamulsow are you working on the same thing? | 17:53 |
| regXboi | Sam-I-Am: not precisely | 17:53 |
| *** thumpba has joined #openstack-neutron-ovn | 17:54 | |
| Sam-I-Am | has there been any testing of neutron-ns-metadata-proxy, being that it still resides as a conventional neutron agent? | 17:55 |
| Sam-I-Am | and has a history of being greedy on resources | 17:55 |
| regXboi | Sam-I-Am: I've not looked at that to date | 17:56 |
| Sam-I-Am | i'm not as worried about the dhcp agent | 17:59 |
| Sam-I-Am | but it also consumes resources | 17:59 |
| *** chandrav has joined #openstack-neutron-ovn | 18:02 | |
| mamulsow | hmm, so I updated my test so now it only puts 200 routers per shared network and ran again | 18:16 |
| mamulsow | unfortunately it looks very similar to the first time I ran it | 18:17 |
| mamulsow | once the total number of routers got up into the 800+ range things started getting very slow, and around 1000 neutron calls started returning with errors or just hanging | 18:18 |
| mamulsow | I wonder if I'm killing things with DHCP agents | 18:22 |
| mamulsow | let me try to clean this mess up and try again without dhcp agents | 18:23 |
| Sam-I-Am | mamulsow: metadata agents too? | 18:25 |
| *** armax has quit IRC | 18:25 | |
| russellb | mamulsow: i'm guessing it's related to the size of the logical flow table | 18:30 |
| russellb | same with ovn-controller performance | 18:30 |
| russellb | that we worked on the other day | 18:30 |
| russellb | if you're brave, you could check with $ ovn-sbctl lflow-list | wc -l | 18:31 |
| russellb | or if you want the exact # of logical flows ... | 18:31 |
| russellb | ovn-sbctl lflow-list | grep -v Datapath | wc -l | 18:32 |
| russellb | something like that | 18:32 |
| openstackgerrit | Russell Bryant proposed openstack/networking-ovn: Revert "Deployment: Update with OVN DB requirements" https://review.openstack.org/271472 | 18:40 |
| openstackgerrit | Kyle Mestery proposed openstack/networking-ovn: Revert "Deployment: Update with OVN DB requirements" https://review.openstack.org/271473 | 18:41 |
| mestery | crap | 18:41 |
| mestery | russellb: You beat me to it :) | 18:41 |
| mestery | By 30 seconds! | 18:41 |
| russellb | ha | 18:41 |
| russellb | impressive | 18:41 |
| mamulsow | Sam-I-Am: yes metadata agent is running there, but so far I haven't booted any VMs in it | 18:45 |
| mamulsow | I mean I've booted plenty of VMs in this environment, but just not part of that test | 18:46 |
| mamulsow | russellb: I'm started ovn-sbctl lflow-list | wc -l, but it hasn't returned yet | 18:47 |
| Sam-I-Am | mamulsow: is it spawning a process for each subnet? | 18:47 |
| russellb | mamulsow: heh, yeahhhhh | 18:48 |
| russellb | it may take a bit. | 18:48 |
| * mamulsow leaves it running | 18:48 | |
| Sam-I-Am | russellb: re your comment on 271091, i was curious myself and asked someone from infra. apparently only non-standard projects (whatever that means) should use build_sphinx... and the central gate jobs. | 18:51 |
| mamulsow | Sam-I-Am: yes, metadata agent per subnet, that and the dhcp agent per subnet are very likely the reason the rabbit nodes are crying | 18:51 |
| Sam-I-Am | if you look at other projects, you'll see one or the other | 18:51 |
| russellb | Sam-I-Am: ok just curious | 18:52 |
| mamulsow | I see between 10-60 metadata agent processes per compute node | 18:52 |
| Sam-I-Am | mamulsow: you can probably turn both of those off to get bare performance | 18:52 |
| mamulsow | yeah, I'll stop metadata agent, and create the subnets with dhcp disabled | 18:52 |
| Sam-I-Am | mamulsow: meltdowns from the md agent processes are a common complaint in #openstack | 18:53 |
| russellb | Sam-I-Am: weird thing to melt down | 18:53 |
| Sam-I-Am | we might need regXboi to look at the code | 18:53 |
| Sam-I-Am | yeah, it is weird. i think its just not been a focus for performance improvements. | 18:53 |
| * russellb nods | 18:53 | |
| regXboi | ??? | 18:53 |
| Sam-I-Am | a lot of folks give up and use config drive | 18:53 |
| Sam-I-Am | regXboi: you've been volunteered for making the metadata agent suck less :) | 18:54 |
| regXboi | is this like being volunteered to run for PTL? | 18:54 |
| Sam-I-Am | the metadata agent is mostly just a proxy for nova metadata, so it might be easy to emulate it elsewhere | 18:55 |
| russellb | config drive ++ | 18:55 |
| Sam-I-Am | regXboi: not as painful... maybe? | 18:55 |
| regXboi | well - I'm not signing up :) | 18:56 |
| Sam-I-Am | russellb: i vote for config drive too, but there are going to be people who want conventional metadata | 18:56 |
| Sam-I-Am | however, the network guide scenario can include whatever we recommend, and other stuff is ymmv | 18:57 |
| Sam-I-Am | usually with some sort of note about "this'll work, but expect issues for scaling" | 18:58 |
| Sam-I-Am | same with using the conventional l3 agent | 18:58 |
| mamulsow | so dumb question… neutron-ns-metadata-proxy != neutron-metadata-agent? | 18:59 |
| mamulsow | I’ve stopped neutron-metadata-agent on all of the compute nodes, but I still see neutron-ns-metadata-proxy processes on all the compute nodes | 18:59 |
| mamulsow | so either 1) my understanding of the world is wrong or 2) those processes are hanging around when they shouldn't be | 19:01 |
| mamulsow | I'm thinking 1 is more likely :) | 19:01 |
| russellb | yes, we should make metadata work | 19:02 |
| russellb | i have no idea if it works today | 19:02 |
| Sam-I-Am | mamulsow: each namespace gets a neutron-ns-metadata-proxy process | 19:05 |
| Sam-I-Am | if the md agent was running when you created those subnets, they'll stick around | 19:05 |
| mamulsow | ah, thanks, so I can just go kill those now? | 19:06 |
| Sam-I-Am | i think you can providing you're not booting vms expecting to use it | 19:06 |
| *** s3wong has joined #openstack-neutron-ovn | 19:08 | |
| mamulsow | I'll cross that bridge when I get to it, but for this testing I'll start with just getting routers and see if there's an upper limit there | 19:08 |
| mamulsow | sounds like I need to go separately work on scaling rabbitmq to handle the dhcp/metadata agent load | 19:09 |
| Sam-I-Am | eliminating the other stuff is a good idea | 19:09 |
| Sam-I-Am | get a good baseline | 19:09 |
| mamulsow | yep | 19:09 |
| Sam-I-Am | russellb: outside of your comment about calling sphinx, i think the patch is ok. it was more or less something to test whether or not the gate logic patch merged right... which is did. now docs patches go through quickly. | 19:11 |
| russellb | ok, will +2 if you want | 19:11 |
| Sam-I-Am | i could probably make the gate logic better, but those other jobs are minimal | 19:12 |
| Sam-I-Am | russellb: un-wipped | 19:12 |
| Sam-I-Am | i might add some stuff for doc8 later | 19:12 |
| Sam-I-Am | stepping out for a bit | 19:16 |
| mamulsow | fyi, "ovn-sbctl lflow-list | wc -l" is still running | 19:22 |
| *** azbiswas_ has joined #openstack-neutron-ovn | 19:22 | |
| *** azbiswas has quit IRC | 19:26 | |
| shettyg | mamulsow: you likely don;t have ovsdb-server connection to ovn-sbctl. | 19:30 |
| shettyg | it is a bug in both ovn-nbctl and ovn-sbctl that they "hang" | 19:30 |
| mamulsow | ah, ok | 19:31 |
| mamulsow | doh, I was running that on a compute node, not the node with sb db | 19:32 |
| openstackgerrit | Kyle Mestery proposed openstack/networking-ovn: Vagrant: Fix issue with boxes https://review.openstack.org/271494 | 19:39 |
| openstackgerrit | Kyle Mestery proposed openstack/networking-ovn: Vagrant: Adjust HOST_IP for compute nodes https://review.openstack.org/271495 | 19:39 |
| *** palexster has quit IRC | 19:44 | |
| openstackgerrit | Merged openstack/networking-ovn: Revert "Deployment: Update with OVN DB requirements" https://review.openstack.org/271472 | 19:46 |
| *** pcaruana has quit IRC | 20:14 | |
| *** salv-orlando has quit IRC | 20:29 | |
| *** azbiswas_ has quit IRC | 20:40 | |
| *** chandrav has quit IRC | 20:49 | |
| *** s3wong_ has joined #openstack-neutron-ovn | 20:50 | |
| *** s3wong has quit IRC | 20:52 | |
| *** azbiswas has joined #openstack-neutron-ovn | 21:07 | |
| *** gangil1 has joined #openstack-neutron-ovn | 21:12 | |
| *** gangil has quit IRC | 21:13 | |
| *** chandrav has joined #openstack-neutron-ovn | 21:14 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 21:18 | |
| *** s3wong_ has quit IRC | 21:27 | |
| *** s3wong has joined #openstack-neutron-ovn | 21:27 | |
| *** salv-orlando has quit IRC | 21:28 | |
| *** gangil1 has quit IRC | 21:38 | |
| openstackgerrit | Merged openstack/networking-ovn: Modify docs build environment https://review.openstack.org/271091 | 21:39 |
| *** flaviof has joined #openstack-neutron-ovn | 21:42 | |
| *** pcaruana has joined #openstack-neutron-ovn | 21:48 | |
| *** flaviof has quit IRC | 21:48 | |
| *** flaviof has joined #openstack-neutron-ovn | 21:48 | |
| *** azbiswas_ has joined #openstack-neutron-ovn | 21:49 | |
| *** azbiswas has quit IRC | 21:52 | |
| *** azbiswas has joined #openstack-neutron-ovn | 21:59 | |
| *** gangil has joined #openstack-neutron-ovn | 22:00 | |
| *** gangil has joined #openstack-neutron-ovn | 22:00 | |
| *** azbiswas_ has quit IRC | 22:02 | |
| *** gangil has quit IRC | 22:05 | |
| *** flaviof has quit IRC | 22:06 | |
| *** gangil has joined #openstack-neutron-ovn | 22:07 | |
| *** gangil has joined #openstack-neutron-ovn | 22:07 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 22:10 | |
| *** rtheis has quit IRC | 22:18 | |
| Sam-I-Am | russellb: moo. | 22:51 |
| *** shettyg has quit IRC | 22:58 | |
| *** salv-orlando has quit IRC | 23:07 | |
| *** salv-orlando has joined #openstack-neutron-ovn | 23:14 | |
| *** thumpba has quit IRC | 23:34 | |
| *** chandrav has quit IRC | 23:54 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!