| *** thinrichs has joined #congress | 01:00 | |
| *** thinrichs has quit IRC | 01:11 | |
| openstackgerrit | Eric K proposed openstack/congress: Remove the traceback added by oslo-messaging RPC https://review.openstack.org/375183 | 01:13 |
|---|---|---|
| openstackgerrit | Eric K proposed openstack/congress: Make policy rule requests gives 404 (not 400) when policy not found https://review.openstack.org/375202 | 02:42 |
| *** ramineni_ has joined #congress | 02:59 | |
| openstackgerrit | Merged openstack/congress: Updated from global requirements https://review.openstack.org/374684 | 03:36 |
| openstackgerrit | Anusha Ramineni proposed openstack/congress: Mark distributed_architecture config as DEPRECATED https://review.openstack.org/373627 | 03:45 |
| *** ramineni_ has quit IRC | 03:52 | |
| openstackgerrit | Anusha Ramineni proposed openstack/congress: Some updations in congress docs https://review.openstack.org/373645 | 04:01 |
| openstackgerrit | Anusha Ramineni proposed openstack/congress: Some updations in congress docs https://review.openstack.org/373645 | 04:03 |
| openstackgerrit | Anusha Ramineni proposed openstack/congress: Default to single process in devstack https://review.openstack.org/373140 | 04:19 |
| *** absubram has quit IRC | 05:47 | |
| *** ramineni_ has joined #congress | 06:19 | |
| *** rrecio_ has quit IRC | 06:45 | |
| *** absubram has joined #congress | 08:30 | |
| *** absubram_ has joined #congress | 08:32 | |
| *** absubram has quit IRC | 08:34 | |
| *** absubram_ is now known as absubram | 08:34 | |
| *** absubram has quit IRC | 08:34 | |
| *** openstackgerrit has quit IRC | 09:03 | |
| *** openstackgerrit has joined #congress | 09:04 | |
| *** ramineni_ has quit IRC | 09:08 | |
| openstackgerrit | Anusha Ramineni proposed openstack/congress: Remove outdated changes in docs https://review.openstack.org/375379 | 10:20 |
| openstackgerrit | Merged openstack/congress: Make policy rule requests gives 404 (not 400) when policy not found https://review.openstack.org/375202 | 10:37 |
| openstackgerrit | Anusha Ramineni proposed openstack/congress: Default to single process in devstack https://review.openstack.org/373140 | 10:38 |
| *** ramineni_ has joined #congress | 11:02 | |
| *** ramineni_ has quit IRC | 11:32 | |
| *** catintheroof has quit IRC | 12:08 | |
| *** rrecio has joined #congress | 14:48 | |
| *** rrecio_ has joined #congress | 14:50 | |
| *** thinrichs has joined #congress | 14:52 | |
| *** rrecio has quit IRC | 14:53 | |
| *** absubram has joined #congress | 15:48 | |
| *** thinrichs has quit IRC | 16:55 | |
| *** catintheroof has joined #congress | 17:09 | |
| *** thinrichs has joined #congress | 18:08 | |
| *** thinrichs has quit IRC | 18:51 | |
| openstackgerrit | Eric K proposed openstack/congress: Remove the traceback added by oslo-messaging RPC https://review.openstack.org/375183 | 19:13 |
| *** catintheroof has quit IRC | 19:17 | |
| *** absubram has quit IRC | 19:41 | |
| *** absubram has joined #congress | 19:56 | |
| *** absubram_ has joined #congress | 19:57 | |
| *** absubram has quit IRC | 20:01 | |
| *** absubram_ is now known as absubram | 20:01 | |
| *** absubram has quit IRC | 20:10 | |
| *** thinrichs has joined #congress | 20:14 | |
| openstackgerrit | Tim Hinrichs proposed openstack/congress: WIP - local HA tests https://review.openstack.org/358927 | 20:47 |
| thinrichs | ekcs: Made progress on the tests. But running into something weird. | 20:49 |
| thinrichs | ekcs: Trying to figure out if it's an artifact of the test or a real problem | 20:49 |
| ekcs | hey thinrichs | 20:49 |
| openstackgerrit | Eric K proposed openstack/congress: Sketch: Avoid orphan rule on concurrent rule insert & policy del https://review.openstack.org/375719 | 20:50 |
| thinrichs | For most of the policy-rule APIs I've tried, I need to run them twice to get the desired behavior. The first one gives a 400/500, and the second works. | 20:50 |
| thinrichs | I pushed test up to gerrit | 20:50 |
| thinrichs | https://review.openstack.org/#/c/358927/14/congress/tests/haht/test_congress_haht.py | 20:50 |
| thinrichs | and commented on an example. | 20:50 |
| thinrichs | ekcs: have you seen this before? Again, happy to talk on the phone. | 20:50 |
| ekcs | got it. looking. | 20:51 |
| ekcs | that’s weird isn’t it. the one you pointed out isn’t even across nodes. | 20:53 |
| thinrichs | That test passes, by the way | 20:53 |
| thinrichs | It's on basically every node. First one is a 'policy not found Alice' | 20:53 |
| thinrichs | Meant…basically every API | 20:54 |
| thinrichs | Your test passes, so it's not clearly a problem with my setup | 20:55 |
| ekcs | i see. | 20:56 |
| ekcs | i’m going to try running the same test but this time with only pe1. and see if that changes anything. | 20:56 |
| thinrichs | Could it be a synchronization problem? Meaning that on the write, we update the DB, and then if the synchronizer doesn't run, it'll give an error on the next call. But that API call forces the synchronizer to run, and so on the next call we get a success. | 20:57 |
| thinrichs | That's a good thought with just 1 PE. I can try it too. | 20:58 |
| ekcs | possible. but i’m looking at the code, and synchronization executed right after adding policy to DB. | 20:58 |
| ekcs | line 365 in agnostic. | 20:59 |
| thinrichs | Seems the problem goes away with just 1 PE | 20:59 |
| ekcs | another hypothesis is somehow the db add didn’t take effect right away. could be a sqlite config issue not properly supporting corcurrent access by two procs. | 21:00 |
| ekcs | i’ll look into it. | 21:00 |
| thinrichs | But then how did the policy-create-delete test work? | 21:01 |
| thinrichs | I tried to get the stack trace for these 400/500 errors to be included but couldn't seem to make it happen | 21:01 |
| thinrichs | I edited congress/application.py:60 so that it would include the stack-trace in the exception message, but that stacktrace didn't show up in the error message. Was I doing something obviously wrong? | 21:03 |
| ekcs | mmm not sure. stacktrace is appended to ex.args[0]. osmething like traceback.format_exc() won’t work because the trace isn’t in the context of the calling process.. | 21:05 |
| ekcs | another way to get trace is to grab it from the self.outfiles | 21:06 |
| ekcs | each PE process logs to one of the outfiles. | 21:06 |
| ekcs | but it should be included in str(e) or e.args[0]. | 21:07 |
| ekcs | simplest way though should be just do what I did in setUp(). | 21:09 |
| ekcs | try: | 21:09 |
| ekcs | helper.retry_check_function_return_value( | 21:09 |
| ekcs | lambda: self.pe1.get().status_code, 200) | 21:09 |
| ekcs | except tenacity.RetryError: | 21:09 |
| ekcs | out = self.read_output_file(self.outfiles[-1]) | 21:09 |
| ekcs | LOG.error('PE1 failed to start. Process output:\n%s' % out) | 21:09 |
| ekcs | raise | 21:09 |
| thinrichs | Good tip. Thanks | 21:10 |
| ekcs | but except Exception (maybe) and outfiles[0] for PE1. | 21:10 |
| ekcs | oh but the problem is you get a huge output because it includes everything up to that point not just the specific error. | 21:11 |
| ekcs | except Exception as e: \ LOG.error(str(e)) | 21:12 |
| ekcs | should work too. | 21:12 |
| ekcs | is that what you did? | 21:12 |
| thinrichs | Super helpful… here's the first error: | 21:14 |
| thinrichs | AttributeError: 'Policy' object has no attribute 'abbr' | 21:14 |
| thinrichs | 260 in agnostic inside synchronize_policies | 21:16 |
| thinrichs | File "/opt/stack/congress/congress/policy_engines/agnostic.py", line 260, in synchronize_policies | 21:19 |
| thinrichs | self.create_policy(p.name, id_=p.id, abbr=p.abbr, | 21:19 |
| ekcs | that’s strange. according to db_policy_rules it should be p.abbreviation. | 21:21 |
| ekcs | but then how does it not error in all sorts of other places. | 21:21 |
| ekcs | line 42 in db_policy_rules.py | 21:22 |
| ekcs | I’ll change it to the following and run again: | 21:23 |
| ekcs | self.create_policy(p.name, id_=p.id, abbr=p.abbreviation, | 21:23 |
| *** thumpba has joined #congress | 21:31 | |
| thinrichs | Moved on to the 2nd problem: where after inserting a rule can't query the rules. | 21:32 |
| thinrichs | Here the error is "Policy ID alice does not exist", which means the policy doesn't exist in self.theory in agnostic. | 21:32 |
| thinrichs | Can we talk on the phone for 7 minutes? I have a meeting starting then that lasts pretty much the rest of the day. | 21:37 |
| thinrichs | ekcs: ^ | 21:37 |
| ekcs | yup. | 21:37 |
| *** thumpba has quit IRC | 21:46 | |
| openstackgerrit | Tim Hinrichs proposed openstack/congress: WIP - local HA tests https://review.openstack.org/358927 | 21:47 |
| ekcs | thinrichs: fixing the synchronize policies function p.abbreviation and p.description seems to have fixed everything. will push the code. very strange that anything actually worked the way it was. | 21:52 |
| *** thinrichs has quit IRC | 21:59 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/congress: Updated from global requirements https://review.openstack.org/375750 | 22:04 |
| *** rrecio_ has quit IRC | 22:20 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!