| *** yamamoto_ has joined #openstack-fwaas | 00:40 | |
| *** yushiro has joined #openstack-fwaas | 00:46 | |
| *** yamamoto_ has quit IRC | 00:48 | |
| *** yamamoto_ has joined #openstack-fwaas | 01:17 | |
| *** yamamoto_ has quit IRC | 01:22 | |
| *** vishwanathj has joined #openstack-fwaas | 02:01 | |
| *** vishwanathj is now known as vishwanathj_zzz | 02:10 | |
| *** yamamoto_ has joined #openstack-fwaas | 02:22 | |
| *** yamamoto_ has quit IRC | 02:29 | |
| *** vishwanathj_zzz has quit IRC | 02:33 | |
| *** yamamoto_ has joined #openstack-fwaas | 02:33 | |
| *** vishwanathj_zzz has joined #openstack-fwaas | 02:33 | |
| *** vishwanathj_zzz has quit IRC | 02:41 | |
| *** vishwanathj_zzz has joined #openstack-fwaas | 02:41 | |
| *** yamamoto_ has quit IRC | 02:42 | |
| *** vishwanathj_zzz has quit IRC | 02:44 | |
| *** vishwanathj_zzz has joined #openstack-fwaas | 02:44 | |
| *** chandanc_ has joined #openstack-fwaas | 02:55 | |
| *** vishwanathj_zzz has quit IRC | 02:56 | |
| *** vishwanathj_zzz has joined #openstack-fwaas | 02:56 | |
| *** chandanc_ has quit IRC | 03:20 | |
| SridarK | yushiro: ping | 03:33 |
|---|---|---|
| *** chandanc_ has joined #openstack-fwaas | 03:34 | |
| *** yamamoto_ has joined #openstack-fwaas | 03:43 | |
| *** yushiro has quit IRC | 03:43 | |
| *** yamamoto_ has quit IRC | 03:47 | |
| *** yushiro has joined #openstack-fwaas | 03:58 | |
| yushiro | SridarK: pong | 03:58 |
| yushiro | sorry for late | 03:58 |
| yushiro | I'm still trying to resolve l2-agent extension. | 04:00 |
| yushiro | s/extension/integration | 04:00 |
| yushiro | chandanc_ has already sent for us about it. | 04:01 |
| *** SarathMekala has joined #openstack-fwaas | 04:01 | |
| *** yamamoto_ has joined #openstack-fwaas | 04:17 | |
| *** yamamoto_ has quit IRC | 04:20 | |
| *** yamamoto_ has joined #openstack-fwaas | 04:20 | |
| chandanc_ | hello yushiro | 04:21 |
| yushiro | chandanc_: Hi. | 04:21 |
| yushiro | chandanc_: First of all, thanks for your e-mail. | 04:21 |
| chandanc_ | So what is the process to integrate the L2 extension ? | 04:21 |
| chandanc_ | Is there any step i missed out in the mail ? | 04:22 |
| yushiro | chandanc_: well, I think your config is enough. | 04:22 |
| chandanc_ | oh ok | 04:22 |
| chandanc_ | Are you looking into the issue ? | 04:22 |
| yushiro | However, I still cannot load 'fwaas' as l2-agent extension. | 04:22 |
| chandanc_ | If you need any help, let me know | 04:22 |
| *** yamamoto_ has quit IRC | 04:22 | |
| chandanc_ | yes same here | 04:23 |
| chandanc_ | I will let you debug that, will try the CLI patch | 04:23 |
| chandanc_ | for l3 | 04:23 |
| yushiro | chandanc_: IMO, current 'neutron side' doesn't have a feature to load another repositories module for l2-agent extension. | 04:23 |
| chandanc_ | but the module loading is handled by stevedore, and should not be based on projects | 04:25 |
| yushiro | chandanc_: I think so, but stevedore try to find 'neutron_fwaas.services....' | 04:26 |
| yushiro | in neutron repository. | 04:26 |
| chandanc_ | do you see any error mesg ? | 04:26 |
| yushiro | I added some print msg in stevedore and pkg_resource and found them. | 04:27 |
| chandanc_ | oh | 04:27 |
| yushiro | ex. in qos l2 extensions driver | 04:27 |
| chandanc_ | ok | 04:28 |
| chandanc_ | for me even qos didnot lead | 04:28 |
| yushiro | {'module_name': 'neutron.agent.l2.extensions.qos', 'extras': (), 'dist': neutron 9.0.0. 0b3.dev479 (/opt/stack/neutron), 'name': 'qos', 'attrs': ('QosAgentExtension',)} | 04:32 |
| yushiro | at https://github.com/openstack/stevedore/blob/bca3451dc68bf1f8cd5d56560088a2a82460110a/stevedore/extension.py#L184 | 04:32 |
| yushiro | def _load_one_plugin at stevedore/extension.py +184 | 04:33 |
| chandanc_ | ok | 04:34 |
| yushiro | in this case, the point is that 'dist' is (/opt/stack/neutron) | 04:34 |
| yushiro | finally, qos extension is loaded as follows: | 04:35 |
| yushiro | module 'neutron.agent.l2.extensions.qos' from '/opt/stack/neutron/neutron/agent/l2/extensions/qos.pyc' | 04:35 |
| yushiro | On the other hand, in case of 'fwaas': | 04:35 |
| yushiro | constraint: a definition has configured -- fwaas = neutron_fwaas.services.firewall.agents.v2.l2.extensions.fwaas:FWaaSV2AgentExtension | 04:36 |
| yushiro | In this case, {'module_name': 'neutron_fwaas.services.firewall.agents.v2.l2.extensions.fwaas', 'extras': (), 'dist': neutron 9.0.0.0b3.dev479 (/opt/stack/neutron), 'name': 'fwaas', 'attrs': ('FWaaSV2AgentExtension',)} | 04:36 |
| chandanc_ | hmm | 04:37 |
| chandanc_ | so you mean the entrypoint should be in the neutron-fwaas module ? | 04:37 |
| yushiro | Yes, but your definition has already specified neutron-fwaas. | 04:38 |
| yushiro | fwaas = neutron_fwaas.services.firewall.agents.v2.l2.extensions.fwaas:FWaaSV2AgentExtension | 04:38 |
| chandanc_ | but i have put it in /etc/neutron/plugins/ml2/ml2_conf.ini | 04:38 |
| chandanc_ | sorry | 04:39 |
| chandanc_ | neutron.egg-info/entry_points.txt file | 04:39 |
| chandanc_ | should we put it in neutron_fwaas.egg-info/entry_points.txt | 04:39 |
| yushiro | I think entry_point.txt is automatically generated. Therefore, you should edit setup.cfg. | 04:40 |
| yushiro | After editted setup.cfg, then execute 'python seup.py install'. So, entry_points.txt has automatically updated. | 04:41 |
| chandanc_ | let me try | 04:41 |
| yushiro | anyway, the result is same as you. | 04:41 |
| yushiro | The difference is only 'procedure'. | 04:42 |
| chandanc_ | hmm | 04:42 |
| yushiro | Sorry for my English.. It is difficult to say... I'd like to use whiteboard and pen :) | 04:43 |
| chandanc_ | no issues | 04:44 |
| chandanc_ | One thing i noticed is that there is a traceback when i restart the l2 agent | 04:44 |
| chandanc_ | I think there is some problem with ryu controller restart | 04:45 |
| chandanc_ | hope this exception does not, stop extension loading | 04:45 |
| yushiro | you mean 'restart the l2 agent' is 'restart q-agt'? | 04:45 |
| chandanc_ | yushiro, how are you making sure that the extensions are loaded | 04:45 |
| chandanc_ | ya | 04:45 |
| chandanc_ | ctrl+c and start again | 04:46 |
| yushiro | I checked log message. you can check with 'grep -r -e "Loaded\sagent\sextensions" /opt/stack/logs/q-agt*' | 04:46 |
| chandanc_ | oh | 04:47 |
| chandanc_ | let me try that | 04:47 |
| yushiro | for debugging, I set 'extensions=qos,fwaas' in ml2_conf.init | 04:47 |
| yushiro | s/init/ini | 04:48 |
| chandanc_ | are you on the latest devstack ? | 04:48 |
| chandanc_ | me too | 04:48 |
| yushiro | Not latest but 3 days ago. | 04:48 |
| chandanc_ | this is the exception i get when restarting l2 agent | 04:48 |
| chandanc_ | 2016-08-29 04:29:08.912 ERROR ryu.lib.hub [-] hub: uncaught exception: Traceback (most recent call last): | 04:48 |
| chandanc_ | File "/usr/local/lib/python2.7/dist-packages/ryu/lib/hub.py", line 54, in _launch | 04:48 |
| chandanc_ | return func(*args, **kwargs) | 04:48 |
| chandanc_ | File "/usr/local/lib/python2.7/dist-packages/ryu/controller/controller.py", line 97, in __call__ | 04:48 |
| chandanc_ | self.ofp_ssl_listen_port) | 04:48 |
| chandanc_ | File "/usr/local/lib/python2.7/dist-packages/ryu/controller/controller.py", line 120, in server_loop | 04:48 |
| chandanc_ | datapath_connection_factory) | 04:48 |
| chandanc_ | File "/usr/local/lib/python2.7/dist-packages/ryu/lib/hub.py", line 117, in __init__ | 04:48 |
| chandanc_ | self.server = eventlet.listen(listen_info) | 04:48 |
| chandanc_ | File "/usr/local/lib/python2.7/dist-packages/eventlet/convenience.py", line 43, in listen | 04:48 |
| chandanc_ | sock.bind(addr) | 04:49 |
| chandanc_ | File "/usr/lib/python2.7/socket.py", line 224, in meth | 04:49 |
| chandanc_ | return getattr(self._sock,name)(*args) | 04:49 |
| chandanc_ | error: [Errno 98] Address already in use | 04:49 |
| chandanc_ | ubuntu@devstack4:~/devstack$ pip list|grep ryu | 04:49 |
| chandanc_ | ryu (4.5) | 04:49 |
| yushiro | hmm, I think it's another error. | 04:49 |
| chandanc_ | do you have ryu installed ? | 04:49 |
| chandanc_ | ubuntu@devstack4:~/devstack$ pip list|grep ryu | 04:49 |
| chandanc_ | ryu (4.5) | 04:49 |
| yushiro | I found same version of ryu | 04:50 |
| chandanc_ | hmm, then something is wrong on my devstack | 04:50 |
| yushiro | You can use http://paste.openstack.org/ for trace or some log messages :) | 04:50 |
| chandanc_ | sure :) | 04:52 |
| yushiro | my local.conf(http://paste.openstack.org/show/564489/) | 04:55 |
| chandanc_ | here is mine http://paste.openstack.org/show/564490/ | 04:56 |
| yushiro | chandanc_: well, it looks no problem. | 04:59 |
| chandanc_ | ya | 04:59 |
| yushiro | I wonder ryu's new issue within 2 days? | 05:00 |
| chandanc_ | did not get that | 05:00 |
| chandanc_ | is there a new release of ryu ? | 05:00 |
| *** vishwanathj_zzz has quit IRC | 05:01 | |
| yushiro | I think no, because the version of ryu is same b/w you and I devstack. | 05:01 |
| *** vishwanathj_zzz has joined #openstack-fwaas | 05:01 | |
| SridarK | yushiro: sorry had stepped away for dinner, dont want to distract ur debugging, but quick q: are u going to be pushing up a CLI patch with the comments addressed | 05:05 |
| yushiro | SridarK: Yes but UT hasn't finished yet. So, I'll update only source code. | 05:07 |
| *** yamamoto_ has joined #openstack-fwaas | 05:07 | |
| SridarK | yushiro: ok - i think we may be cutting it close in terms of deadline - u may want to double check with Akihiro | 05:07 |
| yushiro | I see. | 05:09 |
| chandanc_ | yushiro, i got my ryu issue resolved, looks like there was 2 copies of l2 agent running, not sure how | 05:12 |
| chandanc_ | neutron agent-show 298c4cc3-1fff-4510-a113-f67796824d65 | 05:12 |
| chandanc_ | ], | | 05:12 |
| chandanc_ | | | "extensions": [ | | 05:12 |
| chandanc_ | | | "qos" | | 05:12 |
| chandanc_ | | | ], | 05:12 |
| *** yamamoto_ has quit IRC | 05:16 | |
| *** yamamoto_ has joined #openstack-fwaas | 05:20 | |
| *** yamamoto_ has quit IRC | 05:44 | |
| *** yamamoto_ has joined #openstack-fwaas | 05:45 | |
| *** _SarathMekala_ has joined #openstack-fwaas | 05:46 | |
| *** yamamoto_ has quit IRC | 05:49 | |
| chandanc_ | Hello SridarK, quick question | 05:49 |
| *** yamamoto_ has joined #openstack-fwaas | 05:49 | |
| SridarK | chandanc_: yes pls go ahead | 05:50 |
| chandanc_ | is the l3 extension patch merged to neutron | 05:50 |
| SridarK | chandanc_: not yet | 05:50 |
| chandanc_ | i hope this is the one https://review.openstack.org/#/c/357503/13 | 05:50 |
| *** SarathMekala has quit IRC | 05:50 | |
| chandanc_ | but i could not figure out where the extension loading is done | 05:51 |
| SridarK | yes that is the one - i just went ot double check | 05:51 |
| SridarK | chandanc_: i have the same issue | 05:51 |
| chandanc_ | i dont think the extension loading is part of this patch | 05:52 |
| chandanc_ | is there any other patch | 05:52 |
| SridarK | yes there is a companion in fwaas | 05:52 |
| SridarK | let me find that | 05:52 |
| SridarK | https://review.openstack.org/#/c/355576/ | 05:53 |
| *** yamamoto_ has quit IRC | 05:57 | |
| chandanc_ | I dont see the entry point or loading mechanism in this too | 05:58 |
| chandanc_ | SridarK, i think the l3 extension patch for neutron may not be complete | 06:01 |
| *** _SarathMekala_ has quit IRC | 06:03 | |
| SridarK | chandanc_: perhaps so - i am not too sure on this - | 06:13 |
| chandanc_ | SridarK, if you install the l3 ext patches, dont forget to run "sudo python setup.py develop" in /opt/stack/neutron, this will fix the l3 agent console script | 06:21 |
| SridarK | chandanc_: yes got it | 06:21 |
| chandanc_ | also added a extensions option to l3 config file | 06:21 |
| chandanc_ | ok | 06:21 |
| chandanc_ | extensions = neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2:L3WithFWaaS | 06:22 |
| chandanc_ | in the AGENT section | 06:22 |
| chandanc_ | 2016-08-29 05:58:05.951 DEBUG neutron.wsgi [-] agent.extensions = ['neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2:L3WithFWaaS'] from (pid=26855) log_opt_values /usr/local/lib/python2.7/dist-packages/oslo_config/cfg.py:2626 | 06:22 |
| chandanc_ | got this in log | 06:22 |
| chandanc_ | rest will update you later | 06:23 |
| SridarK | what is the format u used for the extension option | 06:24 |
| SridarK | ok sorry i missed this line up - ok got it | 06:26 |
| SridarK | i will sign off now but will pick this up early am | 06:26 |
| SridarK | i have pulled the neutron patch and the corresponding neutron-fwaas ext patch in to my devstack - let me recheck things and kick off | 06:27 |
| *** _SarathMekala_ has joined #openstack-fwaas | 06:38 | |
| *** yamamoto_ has joined #openstack-fwaas | 06:58 | |
| *** yamamoto_ has quit IRC | 07:02 | |
| *** yamamoto_ has joined #openstack-fwaas | 07:04 | |
| *** yamamoto_ has quit IRC | 07:13 | |
| *** yamamoto_ has joined #openstack-fwaas | 07:19 | |
| *** yamamoto_ has quit IRC | 07:32 | |
| *** mickeys has joined #openstack-fwaas | 07:34 | |
| *** yamamoto_ has joined #openstack-fwaas | 07:34 | |
| *** yamamoto_ has quit IRC | 07:34 | |
| *** mickeys has quit IRC | 07:40 | |
| *** _SarathMekala_ has quit IRC | 07:43 | |
| *** vishwanathj_zzz has quit IRC | 07:53 | |
| *** vishwanathj_zzz has joined #openstack-fwaas | 07:53 | |
| *** SarathMekala has joined #openstack-fwaas | 07:57 | |
| *** yamamoto_ has joined #openstack-fwaas | 08:01 | |
| *** yamamoto_ has quit IRC | 08:03 | |
| *** greghaynes has quit IRC | 08:13 | |
| *** SarathMekala has quit IRC | 08:14 | |
| *** SarathMekala has joined #openstack-fwaas | 08:32 | |
| *** greghaynes has joined #openstack-fwaas | 08:40 | |
| *** SarathMekala has quit IRC | 08:49 | |
| *** yamamoto has joined #openstack-fwaas | 08:50 | |
| *** yamamoto has quit IRC | 08:56 | |
| *** SarathMekala has joined #openstack-fwaas | 09:03 | |
| *** yushiro has quit IRC | 09:09 | |
| *** SarathMekala has quit IRC | 09:26 | |
| *** zigo_ is now known as zigo | 09:56 | |
| *** yamamoto has joined #openstack-fwaas | 09:57 | |
| *** vishwanathj_zzz has quit IRC | 09:58 | |
| *** vishwanathj_zzz has joined #openstack-fwaas | 09:58 | |
| *** yamamoto has quit IRC | 10:03 | |
| *** yamamoto has joined #openstack-fwaas | 11:00 | |
| *** yushiro has joined #openstack-fwaas | 11:01 | |
| *** chandanc_ has quit IRC | 11:06 | |
| *** SarathMekala has joined #openstack-fwaas | 11:10 | |
| *** yamamoto has quit IRC | 11:22 | |
| *** yamamoto has joined #openstack-fwaas | 11:27 | |
| *** yamamoto has quit IRC | 11:42 | |
| *** vishwanathj_zzz has quit IRC | 12:08 | |
| *** vishwanathj_zzz has joined #openstack-fwaas | 12:08 | |
| *** yamamoto has joined #openstack-fwaas | 12:42 | |
| *** yamamoto has quit IRC | 12:49 | |
| *** SarathMekala has quit IRC | 12:59 | |
| *** xdcc has joined #openstack-fwaas | 13:01 | |
| SridarK | yushiro: ping | 13:05 |
| yushiro | SridarK: pong | 13:09 |
| yushiro | sorry, I was eating dinner. | 13:10 |
| SridarK | yushiro: just added to u a review for removing vendor code | 13:10 |
| SridarK | yushiro: oh sorry - pls nothing urgent - whenever u get a chance | 13:10 |
| yushiro | SridarK: no warries. OK, I'll check it now. | 13:10 |
| yushiro | I just reviewed https://review.openstack.org/#/c/361354/2 | 13:12 |
| yushiro | It looks good to me. | 13:12 |
| yushiro | :) | 13:12 |
| yushiro | on the other hand, I found some issue with devstack l3-agent. It cannot be specified fwaas.ini file as a '--config-file' | 13:13 |
| SridarK | yushiro: thx | 13:16 |
| SridarK | yushiro: ok i see a review from u | 13:16 |
| yushiro | sorry. I missed pep8 :( just a moment, please. | 13:16 |
| SridarK | https://review.openstack.org/362050 | 13:17 |
| yushiro | ^^ yes, I ran devstack 1 hour ago and fount this issue. | 13:18 |
| SridarK | but as part of https://review.openstack.org/#/c/355576 | 13:18 |
| SridarK | this file will get removed | 13:18 |
| yushiro | Oh... Yes, you're right. | 13:19 |
| yushiro | So, my patch is not necessary. | 13:20 |
| SridarK | chandanc also has some questions in an email out - i am hoping that once we have that clarification - we can help and do more L3 testing | 13:21 |
| SridarK | i think so too - perhaps u can just reply to chandanc 's email also | 13:21 |
| yushiro | OK. so sorry for late, chandanc.. | 13:22 |
| SridarK | Our focus can be testing the L3 and i will not disturb u too much so u can continue with the CLI | 13:22 |
| njohnston | mfranc213 and I are also looking at chandanc's email... | 13:24 |
| SridarK | yushiro: thx | 13:25 |
| yushiro | SridarK and njohnston : Thanks for your kindness. | 13:26 |
| yushiro | I don't want to sleep !! | 13:26 |
| SridarK | yushiro: pls no worries - thx so much for the help | 13:27 |
| njohnston | :-) | 13:27 |
| SridarK | njohnston: GM | 13:27 |
| njohnston | SridarK: GM to you as well | 13:28 |
| *** chandanc_ has joined #openstack-fwaas | 13:29 | |
| SridarK | njohnston: once chandanc is online - we can try to quickly triage the errors he saw and we can continue with the testing | 13:29 |
| chandanc_ | Hello All | 13:29 |
| SridarK | chandanc_: hi | 13:29 |
| chandanc_ | Hello SridarK | 13:30 |
| SridarK | chandanc_: if we can quickly get clarification on the issues u ran into - i can continue with the testing | 13:31 |
| chandanc_ | sure | 13:31 |
| chandanc_ | actually now i am stuck with the following error | 13:31 |
| chandanc_ | 2016-08-29 13:13:42.076 CRITICAL neutron [req-bbde1aae-986c-4454-9fc0-f9fef0c32d0d None None] AttributeError: 'L3AgentExtensionAPI' object has no attribute 'request_router_info' | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron Traceback (most recent call last): | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/usr/local/bin/neutron-l3-agent", line 10, in <module> | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron sys.exit(main()) | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/opt/stack/neutron/neutron/cmd/eventlet/agents/l3.py", line 17, in main | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron l3_agent.main() | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/opt/stack/neutron/neutron/agent/l3_agent.py", line 57, in main | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron manager=manager) | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/opt/stack/neutron/neutron/service.py", line 387, in create | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron periodic_fuzzy_delay=periodic_fuzzy_delay) | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/opt/stack/neutron/neutron/service.py", line 319, in __init__ | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron self.manager = manager_class(host=host, *args, **kwargs) | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 634, in __init__ | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron super(L3NATAgentWithStateReport, self).__init__(host=host, conf=conf) | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 228, in __init__ | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron self.init_extension_manager(self.plugin_rpc) | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/usr/local/lib/python2.7/dist-packages/osprofiler/profiler.py", line 154, in wrapper | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron return f(*args, **kwargs) | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/opt/stack/neutron/neutron/agent/l3/agent.py", line 378, in init_extension_manager | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron self.agent_api) | 13:32 |
| chandanc_ | 2016-08-29 13:13:42.076 TRACE neutron File "/opt/stack/neutron/neutron/agent | 13:32 |
| chandanc_ | oh, i think i should use paste link | 13:32 |
| chandanc_ | #link http://paste.openstack.org/show/564577/ | 13:33 |
| mfranc213 | thank you chandanc_ looking now. | 13:36 |
| chandanc_ | sure | 13:36 |
| njohnston | yes, the last PS of the l3_extension_api change altered that method name to to get_router_from_port_id | 13:37 |
| chandanc_ | ok, will try using that | 13:37 |
| njohnston | chandanc_: 1 sec, it will take a little more than a method rename | 13:38 |
| mfranc213 | it looks like you are using the new PS https://review.openstack.org/#/c/355576? | 13:38 |
| chandanc_ | sure, may be i should wait for the next PS ? | 13:38 |
| chandanc_ | yes | 13:39 |
| njohnston | Can you give a diff of the changes you've needed to make so far in order to get to this point? We should capture those. | 13:39 |
| chandanc_ | the l3 extension one | 13:39 |
| mfranc213 | sorry i've been on vacation so am a little behind, but i thought we are supposed to be testing https://review.openstack.org/#/c/357503? | 13:39 |
| mfranc213 | but i have been out | 13:39 |
| chandanc_ | sure | 13:39 |
| mfranc213 | so SridarK and/or njohnston should chime in | 13:39 |
| chandanc_ | ya, i have tested that one | 13:39 |
| mfranc213 | okah chandanc_ looking again then | 13:39 |
| njohnston | testing it in conjunction with https://review.openstack.org/#/c/355576 | 13:39 |
| SridarK | yes | 13:40 |
| SridarK | in conjuction | 13:40 |
| mfranc213 | sorry, testing what in conjunction with https://review.openstack.org/#/c/355576? | 13:41 |
| SridarK | at a logical point for everyone - we can have a quick call to run thru the steps and issues - so we can continue with the testing | 13:41 |
| SridarK | mfranc213: the neutron change 35703 | 13:41 |
| *** chandanc__ has joined #openstack-fwaas | 13:41 | |
| SridarK | mfranc213: also welcome back from vacation :-) | 13:42 |
| chandanc__ | Here is the current diff http://paste.openstack.org/show/564579/ | 13:42 |
| njohnston | OK, I will integrate that | 13:43 |
| chandanc__ | njohnston, lot of it is for my testing. this is more of a hint :) | 13:43 |
| *** chandanc_ has quit IRC | 13:44 | |
| mfranc213 | ok. i know what the problem is. it is in neutron. there is a neutron PS that you need to use. | 13:44 |
| *** sc68cal_ is now known as sc68cal | 13:44 | |
| *** xdcc has quit IRC | 13:44 | |
| mfranc213 | i'm sort of out of it (more than usual) so let me find this. one secont. | 13:44 |
| mfranc213 | second. | 13:44 |
| chandanc__ | sure, | 13:45 |
| mfranc213 | this is the PS that nate and i put together. | 13:45 |
| mfranc213 | well, this is the PS: https://review.openstack.org/#/c/357503/ | 13:47 |
| mfranc213 | but the method changes | 13:48 |
| mfranc213 | changed: https://review.openstack.org/#/c/357503/2..13/neutron/agent/l3/l3_agent_extension_api.py | 13:48 |
| mfranc213 | the method name | 13:48 |
| mfranc213 | oh no, it looks like the method got removed | 13:48 |
| mfranc213 | so hrm. | 13:48 |
| chandanc__ | i applied that one | 13:48 |
| mfranc213 | the request_router_info should have remained. | 13:48 |
| njohnston | no, feedback was that they didn't just want to hand over the whole 'router_info' struct, that we needed to have a real API | 13:50 |
| mfranc213 | thinking | 13:50 |
| *** chandanc_ has joined #openstack-fwaas | 13:51 | |
| chandanc_ | sorry, I am getting some connectivity issue | 13:51 |
| chandanc_ | I can wait for the next PS | 13:52 |
| mfranc213 | okay. looking at the next PS now... | 13:52 |
| chandanc_ | sure, will pick it up | 13:53 |
| *** chandanc__ has quit IRC | 13:53 | |
| mfranc213 | njohnston: i will issue the next PS for https://review.openstack.org/#/c/355576/ unless you want to? | 13:55 |
| njohnston | mfranc213: Sure, go for it, but I am adding additional bits to the L3AgentExtensionAPI that your PS should pick up... but you go ahead and work on putting add_router, update_router, and delete_router in there as more than just 'pass'. | 13:57 |
| mfranc213 | njohnston: i can do that. however, the problem that chandanc_ is facing now is that the API for the l3 agent extension has changed in a way that the fwaas l3 agent extension cannot accommodate | 13:58 |
| mfranc213 | i think. checking that one more time. | 13:58 |
| mfranc213 | no, i think i'm wrong about that. | 13:59 |
| mfranc213 | diving into it now. | 13:59 |
| SridarK | mfranc213: perhaps u could first push a PS to align with the new method name that njohnston added as review comment in the neutron patch https://review.openstack.org/#/c/357503/ | 14:00 |
| SridarK | then u could fill in the update_router, delete_router etc | 14:01 |
| mfranc213 | the latter is just applying a git diff | 14:03 |
| *** chandanc__ has joined #openstack-fwaas | 14:03 | |
| SridarK | ok then which ever is easier | 14:04 |
| mfranc213 | also njhonston just told me that i need to change the _get_firewall_group_ports in the fwaas l3 agent extension to instead use get_routers_in_tenant | 14:04 |
| mfranc213 | so i'm gonna do that too. | 14:04 |
| chandanc__ | folks I am AFK for 1/2 hour, will ping back | 14:04 |
| SridarK | ok sounds good mfranc213 , chandanc__ | 14:05 |
| mfranc213 | np | 14:05 |
| *** chandanc_ has quit IRC | 14:05 | |
| *** lmiccini has quit IRC | 14:07 | |
| *** cgross has quit IRC | 14:08 | |
| *** cgross has joined #openstack-fwaas | 14:09 | |
| *** chandanc_ has joined #openstack-fwaas | 14:10 | |
| mfranc213 | SridarK: sorry, just realized what you said about waiting to fill in the add_router stuff. sorry, i will do what you suggest for now, and leave those as pass. | 14:10 |
| SridarK | mfranc213: no worries - whichever is easier - we can get a quick integration test going - so we can do more testing today | 14:11 |
| *** chandanc__ has quit IRC | 14:11 | |
| mfranc213 | makes sense SridarK. diving in now. wish me luck :) | 14:12 |
| SridarK | mfranc213: may the force be with u :-) | 14:12 |
| mfranc213 | and also with you :) | 14:13 |
| *** lmiccini has joined #openstack-fwaas | 14:14 | |
| *** vishwanathj_zzz is now known as vishwanathj | 14:20 | |
| SridarK | :-) | 14:30 |
| mfranc213 | chandan__ and SridarK: nate is running his UTs on the neutron stuff now and i'm running my UTs on the fwaas l3 agent extension now. | 14:38 |
| *** chandanc_ has quit IRC | 14:43 | |
| *** chandanc_ has joined #openstack-fwaas | 14:44 | |
| *** chandanc_ has quit IRC | 14:49 | |
| *** chandanc_ has joined #openstack-fwaas | 14:51 | |
| njohnston | OK, my UTs are all good, mfranc213 is finishing up | 15:14 |
| mfranc213 | 1 failure left | 15:19 |
| mfranc213 | UTs done. | 15:35 |
| *** chandanc_ has quit IRC | 15:35 | |
| mfranc213 | doing a pep8 and then will push. | 15:35 |
| mfranc213 | nate and i will then work on fleshing out the add_router, etc., methods. | 15:36 |
| mfranc213 | my pep8 tests is running extremely slowly and only catching one error at a time. anyone else ever seen this and know how to fix? | 15:44 |
| mfranc213 | well, anyway, PS pushed. | 15:45 |
| njohnston | thanks mfranc213 | 15:45 |
| njohnston | OK, so yushiro and chandanc and SridarK can feel free to test that out while we work on those implementation details | 15:47 |
| njohnston | let us know if any issues arise | 15:47 |
| mfranc213 | njohnston: do i need to add this back in: https://review.openstack.org/#/c/355576/11..12/neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent_v2.py@113 ? | 15:47 |
| yushiro | njohnston: sure. | 15:47 |
| mfranc213 | it was producing an error and couldn't see (in 2 seconds) why it was there. | 15:47 |
| mfranc213 | sorry about that. | 15:48 |
| njohnston | yes, but I don't think you should have that host=host argument to it | 15:48 |
| mfranc213 | i think i misread this: http://paste.openstack.org/show/564579/ | 15:49 |
| mfranc213 | okay, adding that back in. | 15:49 |
| njohnston | mfranc213: Test issue with 355576, check out http://logs.openstack.org/76/355576/12/check/gate-neutron-fwaas-python35/119376e/testr_results.html.gz | 15:52 |
| njohnston | It happened for the python35 test but not for the python27 test | 15:52 |
| mfranc213 | looking at that next | 15:52 |
| mfranc213 | oh, i guess python 3 doesn't like dict.items(). blah. | 15:53 |
| mfranc213 | fixing. | 15:53 |
| mfranc213 | or it does something weird with them. fixing. | 15:54 |
| njohnston | brb, grabbing lunch | 15:58 |
| mfranc213 | njohnston: i still don't get that init to super. | 16:03 |
| *** njohnsto_ has joined #openstack-fwaas | 16:03 | |
| mfranc213 | i went ahead and added it. needed host=host though. so something about my env is diff from that of chandanc_ | 16:08 |
| mfranc213 | but chandan please go ahead. | 16:08 |
| mfranc213 | njohnston: i'm working on add_router, etc. ping me when you're back from lunch. :) | 16:09 |
| mfranc213 | i need some chocolate | 16:09 |
| *** njohnsto_ has quit IRC | 16:09 | |
| SridarK | just wrapping up a mtg - will head to work shortly - but will be good if we can have a quick sync up right after east coast lunch time and before chandanc goes to bed | 16:11 |
| SridarK | so i can carry on with the testing | 16:12 |
| *** njohnsto_ has joined #openstack-fwaas | 16:18 | |
| njohnston | sounds good | 16:20 |
| njohnston | mfranc213: Excellent, I figure that it will mostly just use process_add_router | 16:20 |
| *** njohnsto_ has quit IRC | 16:20 | |
| *** mickeys has joined #openstack-fwaas | 16:21 | |
| *** mickeys has quit IRC | 16:22 | |
| *** mickeys has joined #openstack-fwaas | 16:22 | |
| mfranc213 | that's what i'm thinking. | 16:23 |
| mfranc213 | will think a little bit more though :) | 16:23 |
| *** SarathMekala has joined #openstack-fwaas | 16:34 | |
| mfranc213 | yeah, i guess it's pretty trivial. i kept thinking there should be more to this... | 16:36 |
| *** SarathMekala has quit IRC | 16:51 | |
| *** yushiro is now known as yushiro_sleep | 16:52 | |
| *** yushiro_sleep has quit IRC | 16:57 | |
| *** SridarK has quit IRC | 17:19 | |
| *** raalee has joined #openstack-fwaas | 17:21 | |
| *** chandanc_ has joined #openstack-fwaas | 17:23 | |
| *** raalee has quit IRC | 17:35 | |
| *** SridarK has joined #openstack-fwaas | 17:56 | |
| SridarK | chandanc_: i am at the office | 17:59 |
| SridarK | were u able to pull the latest patches | 17:59 |
| chandanc_ | sorry i got busy with other stuff | 18:00 |
| chandanc_ | do you want a call now | 18:00 |
| chandanc_ | i will probably logout in some time | 18:00 |
| SridarK | yes we can | 18:00 |
| chandanc_ | ok, | 18:01 |
| SridarK | njohnston: , mfranc213: if u are not too busy on the patches - do u want to join to ? | 18:01 |
| mfranc213 | yes. is there a webex? | 18:04 |
| *** vishwanathj has quit IRC | 18:11 | |
| *** vishwanathj has joined #openstack-fwaas | 18:11 | |
| *** mickeys has quit IRC | 18:21 | |
| njohnston | chandanc_ mfranc213 SridarK: Apropos to our conversation - https://review.openstack.org/#/c/361774/2/setup.cfg and https://review.openstack.org/#/c/361786/1/setup.cfg | 18:32 |
| *** chandanc_ has quit IRC | 18:48 | |
| *** jhesketh has quit IRC | 20:05 | |
| *** mickeys has joined #openstack-fwaas | 20:10 | |
| *** jhesketh has joined #openstack-fwaas | 20:12 | |
| dougwig | heads up, fwaas kilo question incoming | 20:50 |
| *** zzelle has joined #openstack-fwaas | 20:50 | |
| njohnston | Thanks dougwig, I saw zzelle on #openstack-neutron | 20:50 |
| zzelle | Hi everyone | 20:50 |
| njohnston | Hi, zzelle. I understand you have an issue related to FWaaS in kilo? | 20:51 |
| zzelle | yes | 20:51 |
| njohnston | Is this occurring in a production environment? | 20:51 |
| zzelle | njohnston, some of our neutron-servers crashed which implied timeout in fwaas calls done by the l3-agent to neutron-servers | 20:51 |
| njohnston | zzelle: Do you know which calls in particular? | 20:52 |
| dougwig | rocking, now you're talking to the experts. | 20:52 |
| zzelle | let me check the code :) | 20:52 |
| zzelle | njohnston, _process_router_add | 20:55 |
| zzelle | timeouts are related to remaining rpc workers in neutron-servers being overloaded | 20:55 |
| zzelle | njohnston, so every l3-agent set services_sync_needed = True and try to resync everything (in process_services_sync) | 20:56 |
| njohnston | Is the FWaaS sync service call causing the overloading of RPC workers you mention, or is the overloading a separate issue of which FWaaS is just the leading indicator? | 20:57 |
| zzelle | njohnston, the overload is caused first by some neutron-servers crashed | 20:58 |
| zzelle | njohnston, secondly by every l3-agent trying to perform a "full" process_services_sync | 20:59 |
| zzelle | njohnston, when we restarted every crashed neutron-servers and purge neutron queues | 21:00 |
| njohnston | So let me make sure I understand the timeline. First, some neutron servers crashed. Second, this crash caused RPC worker overload. Third, FWaaS caused L3 agents to do a full sync. Fourth, this caused further neutron server crashes. Fifth, this degraded your ability to get out of this cascading failure scenario. Is that a correct restatement? | 21:01 |
| zzelle | njohnston, the overload did not end because the load related "full" process_services_syncs is really bigger than usual load | 21:01 |
| zzelle | first: ok, 2nd: ok, 3rd: ok | 21:01 |
| zzelle | 4th: we restarted crashed neutron-servers | 21:02 |
| zzelle | 5th: rpc workers are still overloaded because of full syncs | 21:03 |
| zzelle | njohnston, ^ | 21:03 |
| zzelle | njohnston, rpc workers usually uses 20 % CPU, in 5th they use 100% CPU even after stopping 33% of our L3 agents | 21:04 |
| njohnston | And by "rpc worker" do you mean the message broker, i.e. RabbitMQ? | 21:06 |
| njohnston | Is your desire for us to advise you of some procedure you can use when this kind of situation presents itself, that will let you get everything working, without having to provision additional rpc worker capacity? | 21:06 |
| zzelle | njohnston, rpc worker = neutron-server worker handling rpc incoming messages | 21:07 |
| njohnston | zzelle: Got it, thanks. | 21:07 |
| zzelle | njohnston, we solved the trouble only by stopping must of our l3-agents in order to let remaining ones perform their "full" sync and restarting slowly stopped l3-agents | 21:07 |
| njohnston | How many routers do you have per l3 agent, on average? | 21:08 |
| zzelle | njohnston, 1, exactly 1 | 21:08 |
| zzelle | njohnston, IMO, it seems abnormal that neutron is not able to self-heal itself when every rpc-worker get back online | 21:08 |
| zzelle | njohnston, ~ 60 l3-agents | 21:09 |
| njohnston | Well, as far as fixing things in the Kilo timeframe, I think I would probably have done the same thing as you had to get out of the situation. | 21:09 |
| njohnston | And I am sure there have been tremendous efficiency improvements in oslo.messaging between Kilo and Newton | 21:10 |
| zzelle | njohnston, my concern is that it crashes everything and that trunk seems not to solve the trouble | 21:10 |
| zzelle | njohnston, nope oslo.messaging nor rabbit are responsible | 21:10 |
| njohnston | But we can revisit this, and see if there are ways to optimize the full-sync process in trunk. | 21:11 |
| *** vishwanathj has quit IRC | 21:11 | |
| njohnston | zzelle: How do you know that trunk does not solve the trouble? | 21:11 |
| *** vishwanathj has joined #openstack-fwaas | 21:11 | |
| zzelle | njohnston, git log --oneline neutron_fwaas/services/firewall/agents/l3reference/firewall_l3_agent.py | 21:13 |
| njohnston | zzelle: We'll definitely take a serious look at this. It may take a bit to really get visibility into the full sync process to see where we can optimize, and - full disclosure - I don't think that will happen before the Newton code cutoff, which is in a couple of days. | 21:16 |
| zzelle | njohnston, typically i don't understand why in full sync a l3-agents request tenants with FWs intead of requesting its tenants with FW | 21:16 |
| njohnston | zzelle: But we'll give this serious attention. One thing you could do that would help is to file a bug, or give us the link to the bug you may already have filed, and we can work from the information you have on that. | 21:16 |
| njohnston | zzelle: That is a valid question, and I don't know the answer to it. have only been involved in FWaaS for about 7 months or so. But I promise to work on finding out. | 21:17 |
| njohnston | zzelle: I have to go - family dinner - but if you paste a bug link I'll catch it in the eavesdrop logs. Thanks! | 21:18 |
| zzelle | njohnston, i would like first to check if the trouble is known or not | 21:21 |
| zzelle | than file a bug and work on it if i find some bandwith (sick) | 21:21 |
| *** zzelle has quit IRC | 21:53 | |
| *** yushiro_sleep has joined #openstack-fwaas | 22:25 | |
| SridarK | sorry i missed this conversation with zzelle - opened https://bugs.launchpad.net/neutron/+bug/1618244 to track this and we can triage | 22:49 |
| openstack | Launchpad bug 1618244 in neutron "Possible scale issues with neutron-fwaas requesting all tenants with firewalls after RPC failures" [Undecided,New] | 22:49 |
| *** vishwanathj has quit IRC | 23:18 | |
| *** vishwanathj has joined #openstack-fwaas | 23:18 | |
| *** yushiro_sleep has quit IRC | 23:24 | |
| *** vishwanathj has quit IRC | 23:29 | |
| *** vishwanathj has joined #openstack-fwaas | 23:29 | |
| *** yamamoto has joined #openstack-fwaas | 23:30 | |
| *** vishwanathj has quit IRC | 23:31 | |
| *** vishwanathj has joined #openstack-fwaas | 23:31 | |
| *** vishwanathj has quit IRC | 23:37 | |
| *** vishwanathj has joined #openstack-fwaas | 23:37 | |
| *** vishwanathj has quit IRC | 23:47 | |
| *** yushiro has joined #openstack-fwaas | 23:47 | |
| *** vishwanathj has joined #openstack-fwaas | 23:47 | |
| yushiro | morning good, all | 23:47 |
| yushiro | ooops, good morning | 23:48 |
| SridarK | yushiro: hi GM | 23:54 |
| yushiro | I just came to my office. | 23:55 |
| SridarK | yushiro: i saw u updated ur patch, taking a look | 23:57 |
| yushiro | SridarK: thanks. However, I can update only firewallgroup and firewallpolicy in my home. | 23:57 |
| yushiro | sorry for late :( | 23:57 |
| SridarK | yushiro: no worries, i think u have a TODO on ports | 23:59 |
| SridarK | so i will wait | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!