| *** tosky has quit IRC | 00:05 | |
| openstackgerrit | Merged openstack/puppet-openstacklib stable/stein: Update UPPER_CONSTRAINTS_FILE for stable/stein https://review.openstack.org/645747 | 00:06 |
|---|---|---|
| openstackgerrit | Merged openstack/puppet-neutron stable/stein: Switch to stable/stein https://review.openstack.org/647350 | 00:14 |
| *** markvoelker has joined #puppet-openstack | 00:40 | |
| openstackgerrit | Merged openstack/puppet-neutron stable/stein: Update .gitreview for stable/stein https://review.openstack.org/645734 | 00:58 |
| *** jistr has quit IRC | 01:00 | |
| *** jistr has joined #puppet-openstack | 01:00 | |
| openstackgerrit | Merged openstack/puppet-vswitch stable/stein: Switch to stable/stein https://review.openstack.org/647368 | 01:29 |
| openstackgerrit | yogananth subramanian proposed openstack/puppet-vswitch master: Provide option to disable EMC in puppet-vswitch https://review.openstack.org/647955 | 01:37 |
| openstackgerrit | Merged openstack/puppet-glance stable/stein: Switch to stable/stein https://review.openstack.org/647337 | 01:47 |
| openstackgerrit | Merged openstack/puppet-nova stable/stein: Switch to stable/stein https://review.openstack.org/647351 | 02:04 |
| *** mschuppert has quit IRC | 02:52 | |
| *** morazi has quit IRC | 03:15 | |
| *** ykarel has joined #puppet-openstack | 03:22 | |
| *** ykarel has quit IRC | 04:47 | |
| *** ykarel has joined #puppet-openstack | 05:02 | |
| *** higgins has quit IRC | 05:06 | |
| *** higgins has joined #puppet-openstack | 05:09 | |
| *** janki has joined #puppet-openstack | 06:10 | |
| *** skramaja has joined #puppet-openstack | 06:13 | |
| *** skramaja_ has joined #puppet-openstack | 06:21 | |
| *** skramaja has quit IRC | 06:22 | |
| *** rcernin has quit IRC | 06:38 | |
| *** mschuppert has joined #puppet-openstack | 07:13 | |
| *** dpawlik has joined #puppet-openstack | 07:17 | |
| *** pgaxatte has joined #puppet-openstack | 07:36 | |
| *** jaosorior has quit IRC | 07:37 | |
| *** rpittau|afk is now known as rpittau | 07:39 | |
| *** ykarel is now known as ykarel|lunch | 07:42 | |
| *** tkajinam has quit IRC | 08:07 | |
| *** amoralej|off is now known as amoralej | 08:14 | |
| *** tosky has joined #puppet-openstack | 08:15 | |
| *** higgins has quit IRC | 08:20 | |
| *** rcernin has joined #puppet-openstack | 08:23 | |
| *** JqckB has joined #puppet-openstack | 08:27 | |
| *** higgins has joined #puppet-openstack | 08:31 | |
| *** gfidente has joined #puppet-openstack | 08:38 | |
| *** jpena|off is now known as jpena | 08:40 | |
| *** ykarel|lunch is now known as ykarel | 08:49 | |
| *** rcernin has quit IRC | 08:57 | |
| *** cgfbee has quit IRC | 09:17 | |
| *** cgfbee has joined #puppet-openstack | 09:20 | |
| *** dtantsur|afk is now known as dtantsur | 09:43 | |
| openstackgerrit | Cyril Grosjean proposed openstack/puppet-gnocchi master: Add gnocchi::storage::s3::s3_bucket_prefix into s3 storage configuration https://review.openstack.org/648429 | 09:45 |
| openstackgerrit | Cyril Grosjean proposed openstack/puppet-gnocchi master: Add gnocchi::storage::s3::s3_bucket_prefix into s3 storage configuration https://review.openstack.org/648429 | 09:50 |
| openstackgerrit | Cyril Grosjean proposed openstack/puppet-gnocchi master: Add gnocchi::storage::s3::s3_bucket_prefix into s3 storage configuration https://review.openstack.org/648429 | 09:51 |
| *** gislaved has joined #puppet-openstack | 10:39 | |
| *** gislaved has quit IRC | 10:47 | |
| *** gislaved has joined #puppet-openstack | 10:53 | |
| *** jaosorior has joined #puppet-openstack | 10:55 | |
| *** jaosorior has quit IRC | 11:09 | |
| openstackgerrit | Marios Andreou proposed openstack/puppet-cinder stable/rocky: Don't create keystone service or endpoint for API v1 https://review.openstack.org/648663 | 11:11 |
| openstackgerrit | Marios Andreou proposed openstack/puppet-cinder stable/queens: Don't create keystone service or endpoint for API v1 https://review.openstack.org/648664 | 11:12 |
| *** morazi has joined #puppet-openstack | 11:59 | |
| *** janki has quit IRC | 12:20 | |
| *** trown|outtypewww is now known as trown | 12:20 | |
| *** EmilienM is now known as EvilienM | 12:21 | |
| *** markvoelker has quit IRC | 12:32 | |
| *** jpena is now known as jpena|lunch | 12:32 | |
| *** jtomasek has joined #puppet-openstack | 12:47 | |
| *** jaosorior has joined #puppet-openstack | 12:56 | |
| *** JqckB has quit IRC | 13:18 | |
| *** JqckB has joined #puppet-openstack | 13:18 | |
| *** jaosorior has quit IRC | 13:30 | |
| *** jtomasek has quit IRC | 13:34 | |
| *** jtomasek has joined #puppet-openstack | 13:34 | |
| *** jpena|lunch is now known as jpena | 13:35 | |
| *** jtomasek has quit IRC | 13:36 | |
| *** jaosorior has joined #puppet-openstack | 13:43 | |
| mwhahaha | hrm looks like the neutron beaker tests are flakey | 13:46 |
| mwhahaha | specifically the ovs check, http://logs.openstack.org/35/645735/1/gate/puppet-openstack-beaker-centos-7/569cb11/job-output.txt.gz#_2019-03-29_07_32_12_297021 | 13:46 |
| mwhahaha | beagles: -^ do you now who we can poke around this? | 13:47 |
| *** bnemec is now known as beekneemech | 13:47 | |
| *** jaosorior has quit IRC | 14:01 | |
| *** amoralej is now known as amoralej|lunch | 14:01 | |
| *** jaosorior has joined #puppet-openstack | 14:02 | |
| *** pma78 has quit IRC | 14:30 | |
| * beagles looks | 14:50 | |
| *** amoralej|lunch is now known as amoralej | 15:00 | |
| beagles | mwhahaha: looks like the thing to confirm is whether br-tun is still expected or not if it isn't required to connect nodes - I'd kind of imagine it would be | 15:10 |
| beagles | mwhahaha: chatting with some of the neutron folks for insights | 15:10 |
| mwhahaha | yea it looks like it's checking for the creation of it or something | 15:10 |
| mwhahaha | it's partially succeeding | 15:10 |
| beagles | ue[ | 15:11 |
| beagles | err.. yep | 15:11 |
| mwhahaha | so it'll pass check then fails in gate, it's been happening for a bit | 15:11 |
| beagles | ah okay .. pretty weird | 15:13 |
| mwhahaha | see https://review.openstack.org/#/c/638149/ | 15:14 |
| mwhahaha | which has been trying to land for a while | 15:14 |
| mwhahaha | same problem | 15:14 |
| *** dpawlik has quit IRC | 15:16 | |
| openstackgerrit | David Hill proposed openstack/puppet-gnocchi master: Ensure required packages are installed for s3 support. https://review.openstack.org/647639 | 15:17 |
| *** jaosorior has quit IRC | 15:21 | |
| *** skramaja_ has quit IRC | 15:21 | |
| *** gouthamr has quit IRC | 15:24 | |
| *** gouthamr has joined #puppet-openstack | 15:27 | |
| openstackgerrit | David Hill proposed openstack/puppet-gnocchi master: Ensure required packages are installed for s3 support. https://review.openstack.org/647639 | 15:34 |
| *** ykarel has quit IRC | 15:40 | |
| beagles | mwhahaha: is there a time interval after the services are brought that the beaker tests wait for before running the accepetance tests? | 15:49 |
| mwhahaha | not completely sure, this isn't a new test | 15:49 |
| beagles | mwhahaha: seems like either they sometimes run before the neutron server has a chance to sync with the ovs agent - of course it could also be an issue where the agent and service simply fail to sync too | 15:50 |
| mwhahaha | i think puppet should fully run | 15:50 |
| mwhahaha | and then it tests | 15:50 |
| mwhahaha | this test is 4 years old | 15:51 |
| mwhahaha | so it's unlikely a problem with our testing :D | 15:51 |
| beagles | yeah - I agree | 15:51 |
| *** trown is now known as trown|lunch | 15:58 | |
| beagles | mwhahaha: around when did this start happening - like in the last week or longer? | 16:05 |
| openstackgerrit | Merged openstack/puppet-monasca stable/stein: Switch to stable/stein https://review.openstack.org/647347 | 16:05 |
| mwhahaha | beagles: not sure we haven't really merged a bunch of stuff. i think it got worse over the last week or so | 16:10 |
| mwhahaha | so looks like longer | 16:11 |
| mwhahaha | https://review.openstack.org/#/c/638149/ had a failure on mar 4 | 16:11 |
| mwhahaha | i don't see one earlier, so probably end of Feb early march (depending on promotions) | 16:12 |
| beagles | mwhahaha: k some neutron devs pointed out a recent change that plays around with sync on startup etc but that seems to do with scale | 16:15 |
| mwhahaha | openstack-neutron-14.0.0-0.20190227064311.376fa8a.el7.noarchwas ok | 16:15 |
| mwhahaha | openstack-neutron-14.0.0-0.20190301122617.7fbba81.el7.noarch started failing | 16:16 |
| beagles | ah good info | 16:16 |
| mwhahaha | https://review.openstack.org/#/c/636719/ | 16:17 |
| mwhahaha | maybe not but that merged around then | 16:17 |
| mwhahaha | there's a few ovs things added around then | 16:18 |
| mwhahaha | https://review.openstack.org/#/c/626545/ | 16:18 |
| mwhahaha | oh nm that was pike | 16:18 |
| * mwhahaha adjusts filter | 16:18 | |
| mwhahaha | oh i wonder, http://logs.openstack.org/49/638149/3/gate/puppet-openstack-beaker-centos-7/43dd568/logs/neutron/openvswitch-agent.txt.gz#_2019-03-05_03_24_35_884 | 16:21 |
| mwhahaha | heh looks like a "passing" job had the rpc_response_max_timeout fail in it http://logs.openstack.org/49/638149/3/check/puppet-openstack-beaker-centos-7/7ecfe9e/logs/neutron/openvswitch-agent.txt.gz#_2019-02-28_04_08_51_620 | 16:21 |
| mwhahaha | i bet it's the exponential backoff in the agent | 16:22 |
| mwhahaha | cause it goes 7,20,240 | 16:22 |
| beagles | yeah, the failed one has two such things http://logs.openstack.org/35/645735/1/gate/puppet-openstack-beaker-centos-7/569cb11/logs/neutron/openvswitch-agent.txt.gz | 16:22 |
| mwhahaha | and we run the beaker tests somewhere after the 20 | 16:22 |
| mwhahaha | the passing one just waited 12 seconds | 16:23 |
| mwhahaha | http://logs.openstack.org/49/638149/3/check/puppet-openstack-beaker-centos-7/7ecfe9e/logs/neutron/openvswitch-agent.txt.gz#_2019-02-28_04_08_51_593 | 16:23 |
| *** tosky has quit IRC | 16:23 | |
| mwhahaha | so yea it seems tobe the rpc timeout | 16:33 |
| mwhahaha | http://logs.openstack.org/35/645735/1/check/puppet-openstack-beaker-centos-7/b21d4e0/logs/neutron/openvswitch-agent.txt.gz#_2019-03-22_18_03_51_774 | 16:33 |
| mwhahaha | that's a success, it waited 29 sec | 16:33 |
| mwhahaha | so i wonder what we need to tun or start in a different order | 16:34 |
| beagles | mmmm I wonder what our actual start order is now... | 16:34 |
| mwhahaha | all at the same time | 16:35 |
| mwhahaha | http://logs.openstack.org/35/645735/1/check/puppet-openstack-beaker-centos-7/b21d4e0/job-output.txt.gz#_2019-03-22_18_02_45_590877 | 16:35 |
| *** pgaxatte has quit IRC | 16:36 | |
| mwhahaha | so it's likely resource contention and when ovs checks neutron isn't up yet, so it goes expentional backoff and if it does a shorter recheck (7,20) then it doesn't suceed before it goes to teh long one (240) | 16:37 |
| mwhahaha | i guess we could bump rpc_response_timeout in the beaker tests but that's just covering this up | 16:42 |
| beagles | agreed | 16:42 |
| mwhahaha | so it's an issue witht he max_rpc_timeout | 16:51 |
| mwhahaha | it's used in the backoff | 16:51 |
| mwhahaha | beagles: it's because of https://review.openstack.org/#/c/623401/ | 16:52 |
| mwhahaha | we didn't previously exponential based on the max timeout | 16:52 |
| mwhahaha | so it was likely the addition of the max timeout to the ovs bits that started to make this show up | 16:53 |
| mwhahaha | so we likely need to drop rpc_response_max_timeout from the default of 600 | 16:53 |
| mwhahaha | for most people this is likely fine but the beaker tests this doesn't work | 16:53 |
| beagles | ack | 16:54 |
| mwhahaha | and we don't expose a way to configure rpc_response_max_timeout | 16:54 |
| mwhahaha | fantastic | 16:54 |
| openstackgerrit | Cyril Grosjean proposed openstack/puppet-gnocchi master: Add gnocchi::storage::s3::s3_bucket_prefix into s3 storage configuration https://review.openstack.org/648429 | 16:54 |
| *** gfidente is now known as gfidente|afk | 16:55 | |
| *** rpittau is now known as rpittau|afk | 16:57 | |
| *** ykarel has joined #puppet-openstack | 16:59 | |
| beagles | while turning this config knob might mitigate things, the introduction of the config shouldn't have made things worse - or am I missing something? like did backoff not work the same before? | 16:59 |
| mwhahaha | it didn't work the same before | 16:59 |
| mwhahaha | i think it's a combination of things | 17:00 |
| mwhahaha | let me get this rpc config in and then i'll try and go back in time | 17:01 |
| beagles | kk | 17:01 |
| *** iurygregory has quit IRC | 17:02 | |
| *** ykarel has quit IRC | 17:03 | |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-neutron master: Remove br-tun check from beaker https://review.openstack.org/648734 | 17:07 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-neutron master: Add rpc_response_max_timeout https://review.openstack.org/648735 | 17:07 |
| openstackgerrit | Alex Schultz proposed openstack/puppet-openstack-integration master: Lower neutron rpc max response timeout https://review.openstack.org/648737 | 17:10 |
| mwhahaha | alright lets try those | 17:11 |
| beagles | mwhahaha: added comment to the puppet-neutron patch - rodolfo added it to the ovs agent as well https://review.openstack.org/#/c/636719/ | 17:15 |
| *** owalsh has quit IRC | 17:15 | |
| beagles | so maybe we want to configure that one as well? | 17:16 |
| mwhahaha | where is that? that one seems to be the server config settings | 17:16 |
| *** owalsh has joined #puppet-openstack | 17:17 | |
| mwhahaha | for now we can just kill off br-tun checks until we get this sorted | 17:17 |
| beagles | mwhahaha: ack | 17:17 |
| beagles | with respect to that patch it basically splits the rpc_response_max_timeout into a separate group and then looks like it adds it as a base agent opts or something like that | 17:20 |
| mwhahaha | yea it's not super clear | 17:22 |
| mwhahaha | i'm not even sure it'll fix it | 17:22 |
| mwhahaha | other than dropping the max response timeout to 60 to ensure it never exceeds that | 17:22 |
| beagles | ack | 17:22 |
| mwhahaha | we'll see | 17:22 |
| mwhahaha | there's really not a good way to handle this because for most folks it shouldn't matter if the agent comes up instantly or it takes a minute | 17:23 |
| mwhahaha | just in ci it's annoying | 17:23 |
| beagles | right | 17:23 |
| *** jpena is now known as jpena|off | 17:37 | |
| *** JqckB has quit IRC | 17:37 | |
| *** jistr is now known as jistr|off | 17:45 | |
| *** jistr|off is now known as jistr | 17:45 | |
| *** trown|lunch is now known as trown | 18:08 | |
| *** amoralej is now known as amoralej|off | 18:11 | |
| *** morazi has quit IRC | 18:13 | |
| *** iurygregory has joined #puppet-openstack | 18:17 | |
| *** chem has quit IRC | 18:49 | |
| *** chem has joined #puppet-openstack | 18:49 | |
| *** JqckB has joined #puppet-openstack | 19:02 | |
| *** JqckB has quit IRC | 19:16 | |
| *** dpawlik has joined #puppet-openstack | 19:20 | |
| *** tosky has joined #puppet-openstack | 19:48 | |
| *** gislaved has quit IRC | 19:50 | |
| *** gislaved has joined #puppet-openstack | 19:52 | |
| *** dpawlik has quit IRC | 19:57 | |
| *** dpawlik has joined #puppet-openstack | 20:09 | |
| *** dpawlik has quit IRC | 20:14 | |
| openstackgerrit | Merged openstack/puppet-neutron stable/stein: Update UPPER_CONSTRAINTS_FILE for stable/stein https://review.openstack.org/645735 | 20:15 |
| *** jtomasek has joined #puppet-openstack | 20:17 | |
| *** owalsh has quit IRC | 20:25 | |
| *** jtomasek has quit IRC | 20:26 | |
| *** iurygregory has quit IRC | 20:45 | |
| *** chem has quit IRC | 20:51 | |
| *** chem has joined #puppet-openstack | 20:53 | |
| *** JqckB has joined #puppet-openstack | 20:57 | |
| *** trown is now known as trown|outtypewww | 21:00 | |
| *** owalsh has joined #puppet-openstack | 21:06 | |
| *** owalsh_ has joined #puppet-openstack | 21:19 | |
| *** owalsh has quit IRC | 21:21 | |
| *** owalsh has joined #puppet-openstack | 21:23 | |
| *** owalsh_ has quit IRC | 21:24 | |
| *** josephrsandoval has joined #puppet-openstack | 21:39 | |
| *** owalsh has quit IRC | 21:45 | |
| openstackgerrit | Cyril Grosjean proposed openstack/puppet-gnocchi master: Add gnocchi::storage::s3::s3_bucket_prefix into s3 storage configuration https://review.openstack.org/648429 | 21:48 |
| *** josephrsandoval has quit IRC | 21:50 | |
| *** chem has quit IRC | 21:51 | |
| *** chem has joined #puppet-openstack | 21:52 | |
| *** owalsh has joined #puppet-openstack | 22:15 | |
| *** JqckB has quit IRC | 22:22 | |
| openstackgerrit | Alex Schultz proposed openstack/puppet-neutron master: Revert "Remove br-tun check from beaker" https://review.openstack.org/648790 | 22:32 |
| *** chem has quit IRC | 22:52 | |
| *** chem has joined #puppet-openstack | 22:53 | |
| *** gfidente|afk is now known as gfidente | 22:59 | |
| *** gfidente has quit IRC | 23:13 | |
| *** JqckB has joined #puppet-openstack | 23:15 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!