| *** cp- has quit IRC | 00:25 | |
| *** cp- has joined #openstack-neutron | 00:26 | |
| *** armax has quit IRC | 00:59 | |
| openstackgerrit | Flavio Fernandes proposed openstack/neutron master: DNM [ovn]: port forwarding functionality (1 of 2) https://review.opendev.org/723863 | 01:04 |
|---|---|---|
| openstackgerrit | Flavio Fernandes proposed openstack/neutron master: DNM [ovn]: port forwarding functionality (1 of 2) https://review.opendev.org/723863 | 01:04 |
| *** armax has joined #openstack-neutron | 01:28 | |
| *** ociuhandu has joined #openstack-neutron | 01:30 | |
| *** ociuhandu has quit IRC | 01:35 | |
| openstackgerrit | XiaoYu Zhu proposed openstack/neutron-specs master: l3 router support ecmp https://review.opendev.org/729532 | 01:54 |
| openstackgerrit | Merged openstack/neutron master: Ensure that stale flows are cleaned from phys_bridges https://review.opendev.org/718690 | 02:11 |
| openstackgerrit | XiaoYu Zhu proposed openstack/neutron-specs master: l3 router support ecmp https://review.opendev.org/729532 | 02:14 |
| *** mlavalle has quit IRC | 02:20 | |
| *** elod has quit IRC | 03:23 | |
| *** tobberydberg has quit IRC | 03:25 | |
| *** tobberydberg has joined #openstack-neutron | 03:35 | |
| *** elod has joined #openstack-neutron | 03:35 | |
| *** gthiemonge has quit IRC | 03:40 | |
| *** gthiemonge has joined #openstack-neutron | 03:41 | |
| *** lajoskatona has joined #openstack-neutron | 03:49 | |
| *** ramishra has quit IRC | 04:01 | |
| openstackgerrit | yangkai proposed openstack/neutron-lib master: Update agent's attrs can be sorted https://review.opendev.org/726497 | 04:14 |
| *** ratailor has joined #openstack-neutron | 04:17 | |
| *** ramishra has joined #openstack-neutron | 05:08 | |
| *** ircuser-1 has quit IRC | 05:36 | |
| *** ircuser-1 has joined #openstack-neutron | 05:48 | |
| *** hamalq has joined #openstack-neutron | 05:56 | |
| *** hamalq has quit IRC | 05:56 | |
| *** hamalq has joined #openstack-neutron | 05:57 | |
| *** ociuhandu has joined #openstack-neutron | 06:01 | |
| openstackgerrit | hamza proposed openstack/neutron master: [WIP] Fix the designate dns_name confilct for extra port https://review.opendev.org/729862 | 06:01 |
| *** ociuhandu has quit IRC | 06:05 | |
| *** ociuhandu has joined #openstack-neutron | 06:16 | |
| *** ralonsoh has joined #openstack-neutron | 06:33 | |
| *** slaweq has joined #openstack-neutron | 06:48 | |
| *** ramishra has quit IRC | 06:48 | |
| *** nirmoy has joined #openstack-neutron | 06:51 | |
| *** slaweq has quit IRC | 06:54 | |
| *** slaweq has joined #openstack-neutron | 06:56 | |
| *** nirmoy has quit IRC | 06:59 | |
| *** nirmoy has joined #openstack-neutron | 06:59 | |
| *** ociuhandu has quit IRC | 07:19 | |
| *** bengates has joined #openstack-neutron | 07:21 | |
| openstackgerrit | Rodolfo Alonso Hernandez proposed openstack/neutron-lib master: Rehome abstract class LinuxInterfaceDriver definition https://review.opendev.org/729882 | 07:23 |
| slaweq | ralonsoh: thx for ^^ I was also thinking about doing that :) | 07:25 |
| ralonsoh | hehehe easy patch | 07:26 |
| *** jawad_axd has joined #openstack-neutron | 07:28 | |
| openstackgerrit | hamza proposed openstack/neutron master: [WIP] Fix the designate dns_name confilct for extra port https://review.opendev.org/729862 | 07:29 |
| *** jpena|off is now known as jpena | 07:38 | |
| *** amoralej|off is now known as amoralej | 07:39 | |
| *** jlibosva has joined #openstack-neutron | 07:45 | |
| *** tkajinam_ has quit IRC | 08:05 | |
| *** S4ren has quit IRC | 08:12 | |
| openstackgerrit | zhangboye proposed openstack/neutron-fwaas master: Add py38 package metadata https://review.opendev.org/730272 | 08:13 |
| *** S4ren has joined #openstack-neutron | 08:13 | |
| *** ramishra has joined #openstack-neutron | 08:15 | |
| *** ociuhandu has joined #openstack-neutron | 08:22 | |
| *** lpetrut has joined #openstack-neutron | 08:26 | |
| *** ociuhandu has quit IRC | 08:28 | |
| *** ociuhandu has joined #openstack-neutron | 08:28 | |
| *** ociuhandu has quit IRC | 08:29 | |
| *** ociuhandu has joined #openstack-neutron | 08:30 | |
| *** ociuhandu has quit IRC | 08:35 | |
| *** ociuhandu has joined #openstack-neutron | 08:37 | |
| *** elod has quit IRC | 08:43 | |
| *** elod has joined #openstack-neutron | 08:50 | |
| *** elod has quit IRC | 08:56 | |
| *** elod has joined #openstack-neutron | 08:58 | |
| jakeyip | otherwiseguy: I'm testing out the patches but unfortunately I'm still getting TRY_AGAIN in PgAclAddCommand. see https://paste.centos.org/view/a73b7967 . Wondering if it is the same for you? How far did you manage to get for the sync? | 09:04 |
| *** lajoskatona has quit IRC | 09:07 | |
| *** elod has quit IRC | 09:10 | |
| *** elod has joined #openstack-neutron | 09:10 | |
| *** hjensas|afk is now known as hjensas | 09:13 | |
| *** lajoskatona has joined #openstack-neutron | 09:21 | |
| openstackgerrit | yangkai proposed openstack/neutron-lib master: Update agent's attrs can be sorted https://review.opendev.org/726497 | 09:22 |
| *** ramishra has quit IRC | 09:37 | |
| *** sean-k-mooney has joined #openstack-neutron | 09:39 | |
| openstackgerrit | Merged openstack/neutron-lib master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/729868 | 09:44 |
| openstackgerrit | Merged openstack/neutron-lib master: Remove redefined constant https://review.opendev.org/729886 | 09:44 |
| *** ramishra has joined #openstack-neutron | 10:01 | |
| *** aedc has joined #openstack-neutron | 10:23 | |
| *** gcheresh has joined #openstack-neutron | 10:57 | |
| *** sapd1_x has quit IRC | 11:11 | |
| *** gcheresh has quit IRC | 11:13 | |
| *** lajoskatona has quit IRC | 11:13 | |
| *** hamalq has quit IRC | 11:21 | |
| *** jawad_axd has quit IRC | 11:21 | |
| *** lajoskatona has joined #openstack-neutron | 11:22 | |
| *** nweinber has joined #openstack-neutron | 11:24 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** maciejjozefczyk is now known as mjozefcz|lunch | 11:30 | |
| *** jawad_axd has joined #openstack-neutron | 11:37 | |
| openstackgerrit | Merged openstack/neutron master: Fix neutron-ovn-db-sync-util issues https://review.opendev.org/729264 | 11:40 |
| openstackgerrit | Merged openstack/neutron master: ovn: Override clean_hash_ring for sync util https://review.opendev.org/729997 | 11:41 |
| *** jawad_axd has quit IRC | 11:42 | |
| openstackgerrit | yangkai proposed openstack/neutron-lib master: Update agent's attrs can be sorted https://review.opendev.org/726497 | 11:45 |
| *** jawad_axd has joined #openstack-neutron | 12:00 | |
| *** amoralej is now known as amoralej|lunch | 12:05 | |
| *** ociuhandu has quit IRC | 12:05 | |
| *** ociuhandu has joined #openstack-neutron | 12:06 | |
| *** ociuhandu has quit IRC | 12:11 | |
| openstackgerrit | Jakub Libosvar proposed openstack/networking-ovn stable/train: ovn: Override clean_hash_ring for sync util https://review.opendev.org/730309 | 12:15 |
| *** mjozefcz|lunch is now known as mjozefcz | 12:18 | |
| *** jawad_axd has quit IRC | 12:24 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| *** priteau has joined #openstack-neutron | 12:33 | |
| *** rcernin has quit IRC | 12:39 | |
| *** ratailor has quit IRC | 12:55 | |
| *** ociuhandu has joined #openstack-neutron | 13:08 | |
| *** njohnston has quit IRC | 13:13 | |
| *** tmorin has joined #openstack-neutron | 13:13 | |
| *** amoralej|lunch is now known as amoralej | 13:22 | |
| *** lpetrut has quit IRC | 13:35 | |
| openstackgerrit | Merged openstack/neutron stable/ussuri: Allow usage of legacy 3rd-party interface drivers https://review.opendev.org/729728 | 13:43 |
| otherwiseguy | jakeyip: pretty much all the way through. it died after 3 hours in the last stage because I ran out of memory if I remember correctly. Can you get me logs of the entire transaction that gets tried again n=1,idx=0 - n=1,idx=$big number? Because I don't see PgAclAddCommand itself running verify(), I'd like to know exactly what command in the transaction is calling it. | 13:49 |
| otherwiseguy | jakeyip: Also, could I get you to run `sudo lsof +c0 -i tcp:6641` while you are in the middle of running the sync? | 13:54 |
| *** jlibosva has quit IRC | 13:57 | |
| *** jlibosva has joined #openstack-neutron | 13:57 | |
| *** jawad_axd has joined #openstack-neutron | 13:57 | |
| *** priteau has quit IRC | 13:59 | |
| *** jawad_axd has quit IRC | 14:02 | |
| openstackgerrit | Merged openstack/neutron master: Default OVN_BRANCH and OVS_BRANCH to latest stable https://review.opendev.org/726168 | 14:10 |
| *** jawad_axd has joined #openstack-neutron | 14:19 | |
| *** jawad_axd has quit IRC | 14:23 | |
| *** tmorin has quit IRC | 14:24 | |
| *** njohnston has joined #openstack-neutron | 14:31 | |
| *** jawad_axd has joined #openstack-neutron | 14:39 | |
| *** jawad_axd has quit IRC | 14:44 | |
| *** stingrayza has joined #openstack-neutron | 14:51 | |
| *** also_stingrayza has quit IRC | 14:54 | |
| *** mlavalle has joined #openstack-neutron | 14:56 | |
| *** jawad_axd has joined #openstack-neutron | 15:00 | |
| *** jawad_axd has quit IRC | 15:05 | |
| *** gcheresh has joined #openstack-neutron | 15:11 | |
| *** dconde has joined #openstack-neutron | 15:20 | |
| *** dconde has quit IRC | 15:21 | |
| *** ociuhandu has quit IRC | 15:39 | |
| *** ociuhandu has joined #openstack-neutron | 15:41 | |
| *** gcheresh has quit IRC | 15:45 | |
| *** ociuhandu has quit IRC | 15:46 | |
| openstackgerrit | Lajos Katona proposed openstack/neutron master: Auto-delete dhcp ports on segment delete https://review.opendev.org/729250 | 15:46 |
| openstackgerrit | Lajos Katona proposed openstack/neutron master: New fullstack test for subnet port delete race condition https://review.opendev.org/728904 | 15:47 |
| *** lajoskatona has left #openstack-neutron | 15:48 | |
| *** gcheresh has joined #openstack-neutron | 16:09 | |
| jakeyip | n | 16:13 |
| *** tbachman has quit IRC | 16:15 | |
| *** gcheresh has quit IRC | 16:20 | |
| *** tbachman has joined #openstack-neutron | 16:20 | |
| jakeyip | otherwiseguy: lsof output is https://paste.centos.org/view/3652479b | 16:20 |
| jakeyip | logs are at https://swift.rc.nectar.org.au/v1/AUTH_f42f9588576c43969760d81384b83b1f/jake-public/neutron-ovn-db-sync-util.log.gz - 55mb | 16:21 |
| *** tbachman has quit IRC | 16:27 | |
| *** tbachman has joined #openstack-neutron | 16:28 | |
| *** jlibosva has quit IRC | 16:34 | |
| *** amoralej is now known as amoralej|off | 16:37 | |
| *** armax has quit IRC | 16:41 | |
| *** jpena is now known as jpena|off | 16:43 | |
| *** ociuhandu has joined #openstack-neutron | 16:53 | |
| *** tbachman has quit IRC | 16:54 | |
| *** tbachman has joined #openstack-neutron | 16:56 | |
| *** ociuhandu has quit IRC | 16:57 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/networking-bgpvpn stable/stein: Use tox 'all' env instead of 'all-plugins' for networking bgpvpn tempest plugin https://review.opendev.org/730379 | 17:17 |
| *** ociuhandu has joined #openstack-neutron | 17:21 | |
| *** bengates_ has joined #openstack-neutron | 17:22 | |
| *** bengates has quit IRC | 17:26 | |
| openstackgerrit | Andreas Jaeger proposed openstack/os-vif master: Switch to newer openstackdocstheme and reno versions https://review.opendev.org/728612 | 17:30 |
| openstackgerrit | Elod Illes proposed openstack/networking-bagpipe stable/stein: [stein-only] Fix networking-bgpvpn and neutron requirement constraints https://review.opendev.org/729936 | 17:35 |
| openstackgerrit | Elod Illes proposed openstack/networking-bagpipe stable/stein: [stein-only] Fix networking-bgpvpn and neutron requirement constraints https://review.opendev.org/729936 | 17:43 |
| *** ociuhandu has quit IRC | 17:55 | |
| *** ociuhandu has joined #openstack-neutron | 17:56 | |
| *** ociuhandu has quit IRC | 18:01 | |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Remove usage of six.PY2 https://review.opendev.org/730385 | 18:08 |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Remove usage of six.text_type and six.string_type https://review.opendev.org/730386 | 18:08 |
| openstackgerrit | Brian Haley proposed openstack/neutron master: Remove usage of six.next and iterators https://review.opendev.org/730387 | 18:08 |
| *** gcheresh has joined #openstack-neutron | 18:09 | |
| *** ralonsoh has quit IRC | 18:14 | |
| *** sean-k-mooney has quit IRC | 18:31 | |
| *** beagles is now known as beagles_afk | 18:37 | |
| *** bengates_ has quit IRC | 18:44 | |
| *** ociuhandu has joined #openstack-neutron | 18:47 | |
| *** armax has joined #openstack-neutron | 18:55 | |
| *** gcheresh has quit IRC | 19:05 | |
| *** nweinber has quit IRC | 19:07 | |
| openstackgerrit | hamza proposed openstack/neutron master: [WIP] Fix the designate dns_name confilct for extra port https://review.opendev.org/729862 | 19:17 |
| *** ociuhandu has quit IRC | 19:18 | |
| *** ociuhandu has joined #openstack-neutron | 19:19 | |
| *** ociuhandu has quit IRC | 19:24 | |
| *** elod has quit IRC | 19:26 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/networking-bgpvpn stable/stein: Use tox 'all' env instead of 'all-plugins' for networking bgpvpn tempest plugin https://review.opendev.org/730379 | 19:26 |
| *** elod has joined #openstack-neutron | 19:27 | |
| openstackgerrit | Merged openstack/neutron stable/train: Allow usage of legacy 3rd-party interface drivers https://review.opendev.org/729729 | 19:40 |
| openstackgerrit | Merged openstack/neutron stable/stein: Allow usage of legacy 3rd-party interface drivers https://review.opendev.org/729730 | 19:40 |
| *** slaweq has quit IRC | 19:49 | |
| *** slaweq has joined #openstack-neutron | 19:54 | |
| openstackgerrit | Merged openstack/neutron stable/queens: Allow usage of legacy 3rd-party interface drivers https://review.opendev.org/729733 | 19:55 |
| *** slaweq has quit IRC | 19:59 | |
| *** ociuhandu has joined #openstack-neutron | 20:03 | |
| openstackgerrit | Terry Wilson proposed openstack/neutron stable/ussuri: Fix neutron-ovn-db-sync-util issues https://review.opendev.org/730413 | 20:10 |
| *** ociuhandu has quit IRC | 20:15 | |
| openstackgerrit | Merged openstack/neutron master: Remove usage of six.add_metaclass https://review.opendev.org/730125 | 20:35 |
| *** hamalq has joined #openstack-neutron | 20:36 | |
| *** nirmoy has quit IRC | 20:36 | |
| *** lpetrut has joined #openstack-neutron | 20:43 | |
| *** itsjg has quit IRC | 20:50 | |
| openstackgerrit | Merged openstack/neutron master: [ci] Fix several rally task arguments https://review.opendev.org/725836 | 20:52 |
| *** ccamposr__ has quit IRC | 20:58 | |
| openstackgerrit | hamza proposed openstack/neutron master: [WIP] Fix the designate dns_name confilct for extra port https://review.opendev.org/729862 | 20:58 |
| *** bnemec is now known as beekneemech | 21:13 | |
| *** lpetrut has quit IRC | 21:15 | |
| *** lpetrut has joined #openstack-neutron | 21:26 | |
| *** itsjg has joined #openstack-neutron | 21:30 | |
| *** lpetrut has quit IRC | 22:00 | |
| *** sapd1_x has joined #openstack-neutron | 22:05 | |
| *** sapd1_y has quit IRC | 22:05 | |
| *** jmlowe has quit IRC | 22:20 | |
| *** jmlowe has joined #openstack-neutron | 22:21 | |
| *** ccstone9 has joined #openstack-neutron | 22:23 | |
| *** ccstone has quit IRC | 22:24 | |
| *** ccstone9 is now known as ccstone | 22:24 | |
| *** ccamposr has joined #openstack-neutron | 22:56 | |
| jakeyip | otherwiseguy: one more thing to add; I ran out of memory too, had to bump the VM from 8G to 16G to keep going IIRC. This error happens after the MemoryError. | 22:58 |
| *** mlavalle has quit IRC | 22:58 | |
| otherwiseguy | jakeyip: what is your OVS.ovsdb_timeout setting? I think it defaults to 30 and I see in one of the logs you sent me had a timeout at 5 sec...that might also have been one of the many probe_interval settings in neutron/ovs/ovn as well though. | 23:01 |
| otherwiseguy | jakeyip: I've got it running for like the last 5.5 hours on a machine with 64GB ram... | 23:02 |
| *** ociuhandu has joined #openstack-neutron | 23:04 | |
| otherwiseguy | Best I can tell from the python-ovs code, you can also get a TRY_AGAIN on any time sending a command to the ovsdb-server fails in addition to verify(). So any connection timeout that hits, in this case I'm assuming the one from ovsdb-server to the client causing a reconnect and interrupting sending the ridiculously huge transaction with 358k entries in it... | 23:04 |
| otherwiseguy | I'll look into maybe for the sync util itself, going in and temporarily setting all timeouts to 0/infinite. Still haven't noticed the TRY_AGAIN on my side. though it's been running long enough I could have missed it, but every tim eI look it's doing the PgAddPort commands. | 23:06 |
| *** ociuhandu has quit IRC | 23:08 | |
| otherwiseguy | jakeyip: and I don't see those in your log, so I'm assuming they are definitely after where you get hung up. | 23:09 |
| otherwiseguy | jakeyip: regardless of TRY_AGAIN, it seems pretty clear to me that we need to drastically improve the performance of the sync util. each create_port is taking around 2+ seconds. For 12k ports, even without considreing all the port gorups etc, that's around 7 hours. | 23:14 |
| otherwiseguy | that's not running with my indexing patch installed. wanted to get a baseline, but wasn't expecting it to take all day to run. | 23:15 |
| *** ccamposr__ has joined #openstack-neutron | 23:17 | |
| *** ccamposr has quit IRC | 23:20 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!