| *** dave-mccowan has joined #openstack-barbican | 00:13 | |
| *** alee has quit IRC | 00:24 | |
| *** SheenaG1 has quit IRC | 00:29 | |
| *** alee has joined #openstack-barbican | 00:36 | |
| *** everjeje has quit IRC | 00:36 | |
| *** alee has quit IRC | 01:00 | |
| *** alee has joined #openstack-barbican | 01:13 | |
| *** alee has quit IRC | 01:29 | |
| *** alee has joined #openstack-barbican | 01:31 | |
| *** kebray has joined #openstack-barbican | 01:34 | |
| *** alee has quit IRC | 01:35 | |
| *** alee has joined #openstack-barbican | 01:48 | |
| *** jvrbanac has quit IRC | 02:00 | |
| *** zz_dimtruck has quit IRC | 02:00 | |
| *** jillysciarilly has quit IRC | 02:00 | |
| *** alee has quit IRC | 02:00 | |
| *** eglute has quit IRC | 02:00 | |
| *** hockeynut has quit IRC | 02:01 | |
| *** lbragstad has quit IRC | 02:01 | |
| *** tdink has quit IRC | 02:02 | |
| *** jroll has quit IRC | 02:02 | |
| *** zz_dimtruck has joined #openstack-barbican | 02:07 | |
| *** jvrbanac has joined #openstack-barbican | 02:07 | |
| *** jillysciarilly has joined #openstack-barbican | 02:07 | |
| *** zz_dimtruck is now known as dimtruck | 02:07 | |
| *** hockeynut has joined #openstack-barbican | 02:07 | |
| *** jroll has joined #openstack-barbican | 02:07 | |
| *** eglute has joined #openstack-barbican | 02:07 | |
| *** tdink has joined #openstack-barbican | 02:08 | |
| *** jroll has quit IRC | 02:08 | |
| *** jroll has joined #openstack-barbican | 02:08 | |
| *** lbragstad has joined #openstack-barbican | 02:11 | |
| *** alee has joined #openstack-barbican | 02:13 | |
| *** alee has quit IRC | 02:21 | |
| *** SheenaG has joined #openstack-barbican | 02:28 | |
| *** alee has joined #openstack-barbican | 02:33 | |
| *** alee has quit IRC | 02:39 | |
| *** alee has joined #openstack-barbican | 02:52 | |
| *** alee has quit IRC | 02:56 | |
| *** alee has joined #openstack-barbican | 03:09 | |
| *** alee has quit IRC | 03:13 | |
| openstackgerrit | Dave McCowan proposed openstack/barbican: Add Functional Tests for Private Key Secret Type https://review.openstack.org/169974 | 03:14 |
|---|---|---|
| *** xaeth_afk is now known as xaeth | 03:21 | |
| *** SheenaG has quit IRC | 03:31 | |
| openstackgerrit | Dave McCowan proposed openstack/barbican: Add Functional Tests for Private Key Secret Type https://review.openstack.org/169974 | 03:31 |
| *** tkelsey has joined #openstack-barbican | 03:49 | |
| *** tkelsey has quit IRC | 03:55 | |
| *** rm_you| has joined #openstack-barbican | 04:02 | |
| *** rm_you has quit IRC | 04:05 | |
| *** SheenaG has joined #openstack-barbican | 04:10 | |
| *** xaeth is now known as xaeth_afk | 04:11 | |
| *** woodster_ has quit IRC | 04:20 | |
| *** woodster_ has joined #openstack-barbican | 04:46 | |
| openstackgerrit | John Vrbanac proposed openstack/python-barbicanclient: Fixing the broken functional tests https://review.openstack.org/171465 | 05:00 |
| *** dave-mccowan has quit IRC | 05:02 | |
| *** SheenaG has quit IRC | 05:04 | |
| *** gyee has quit IRC | 05:23 | |
| *** alee has joined #openstack-barbican | 05:37 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/barbican: Imported Translations from Transifex https://review.openstack.org/171475 | 06:21 |
| *** tkelsey has joined #openstack-barbican | 06:42 | |
| *** tkelsey has quit IRC | 06:46 | |
| *** jaosorior has joined #openstack-barbican | 06:56 | |
| openstackgerrit | Douglas Mendizábal proposed openstack/barbican: Document Symmetric Secret Type https://review.openstack.org/171488 | 07:02 |
| *** everjeje has joined #openstack-barbican | 07:34 | |
| *** kebray has quit IRC | 07:39 | |
| *** woodster_ has quit IRC | 07:50 | |
| *** nickrmc83 has quit IRC | 09:28 | |
| *** nickrmc83 has joined #openstack-barbican | 09:30 | |
| openstackgerrit | Everardo Padilla Saca proposed openstack/barbican: Remove str() casting for the client_message variable https://review.openstack.org/167044 | 09:35 |
| *** rm_work has quit IRC | 10:04 | |
| *** rm_work has joined #openstack-barbican | 10:06 | |
| *** rm_work has joined #openstack-barbican | 10:06 | |
| openstackgerrit | Thomas Herve proposed openstack/barbican: Return container not found before ACL checks https://review.openstack.org/171553 | 10:06 |
| *** nickrmc83 has quit IRC | 10:15 | |
| *** nickrmc83 has joined #openstack-barbican | 10:15 | |
| *** darrenmoffat has quit IRC | 10:21 | |
| *** darrenmoffat has joined #openstack-barbican | 10:22 | |
| openstackgerrit | Thomas Herve proposed openstack/barbican: Return container not found before ACL checks https://review.openstack.org/171553 | 11:59 |
| *** woodster_ has joined #openstack-barbican | 12:14 | |
| *** rellerreller has joined #openstack-barbican | 12:45 | |
| *** dave-mccowan has joined #openstack-barbican | 12:47 | |
| *** nkinder has quit IRC | 13:12 | |
| *** alee has quit IRC | 13:42 | |
| *** alee has joined #openstack-barbican | 13:46 | |
| alee | woodster_, rellerreller , redrobot jaosorior - please take a look at https://review.openstack.org/#/c/169600/ | 13:48 |
| jaosorior | alee: sure | 13:49 |
| jaosorior | though it seems to be failing | 13:49 |
| jaosorior | lets see why | 13:49 |
| alee | woodster_, rellerreller , redrobot , jaosorior - not sure why its not passing the gate -- appears to pass locally for me and for dave (so its not an openstack issue). Seems like every time I run the recheck, a different couple of tests fail. | 13:49 |
| alee | I think some things are timing out on the gate. | 13:49 |
| alee | if you can, please run the tests locally and see what happens. | 13:50 |
| jaosorior | alee: does it tend to fail in the same tests? | 13:50 |
| alee | there is one test in particular that seems to fail consistently -- api.v1.functional.test_certificate_orders.CertificatesTestCase.test_create_stored_key_order | 13:51 |
| alee | but I dont know why .. | 13:51 |
| alee | requests.exceptions.ConnectionError: HTTPConnectionPool(host='127.0.0.1', port=9311): Max retries exceeded with url: /v1/orders (Caused by ProtocolError('Connection aborted.', BadStatusLine("''",))) | 13:51 |
| alee | other tests fail randomly with the same reason | 13:52 |
| *** xaeth_afk is now known as xaeth | 13:53 | |
| alee | jaosorior, woodster_ - aha! I think I found it ! | 13:58 |
| alee | http://logs.openstack.org/00/169600/3/check/gate-barbican-devstack-dsvm/7e8e09d/logs/screen-barbican.txt.gz#_2015-04-07_22_14_17_229 | 13:58 |
| jaosorior | alee: what's up? | 13:58 |
| *** chlong has quit IRC | 13:58 | |
| alee | DataError: (DataError) (1406, "Data too long for column 'value' at row 1" | 13:59 |
| jaosorior | aha! | 13:59 |
| alee | yup - trying to save the generated csr | 13:59 |
| jaosorior | what database backend is being used there? | 13:59 |
| alee | jaosorior, how do I find that out? isn't the problem in the table column size definition? | 14:00 |
| jaosorior | might be | 14:01 |
| jaosorior | but if you say it works for you | 14:01 |
| jaosorior | it might also be database dependant | 14:01 |
| jaosorior | actually the tests just finished running on my machine | 14:02 |
| jaosorior | and it fails here ERROR: functionaltests.api.v1.functional.test_certificate_orders.CertificatesTestCase.test_create_stored_key_order | 14:02 |
| *** nkinder has joined #openstack-barbican | 14:02 | |
| jaosorior | In the same way | 14:02 |
| jaosorior | requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine("''",)) | 14:02 |
| alee | jaosorior, not sure why it works for me -- maybe openssl for my version generates a smaller csr | 14:03 |
| alee | and we're just hitting the limit | 14:03 |
| jaosorior | could be | 14:03 |
| alee | jaosorior, do you see the same error in your barbican logs? | 14:03 |
| jaosorior | I actually have a newer version of OpenSSL (using 1.0.2a) | 14:04 |
| *** paul_glass has joined #openstack-barbican | 14:04 | |
| alee | jaosorior, OrderBarbicanMetadatum has --> | 14:05 |
| alee | key = sa.Column(sa.String(255), nullable=False) | 14:05 |
| alee | value = sa.Column(sa.String(255), nullable=False) | 14:05 |
| jaosorior | alee: re-running the tests, had to purge my logs since there's too much stuff | 14:05 |
| alee | yup | 14:06 |
| jaosorior | Is there a limit to how big CSRs can be? | 14:06 |
| alee | same for OrderPluginMetadatum | 14:06 |
| rellerreller | alee I can take a look at the patch, but it will have to wait until this afternoon. Lot's of fires here this morning. | 14:07 |
| jaosorior | alee: I checked now, and I get the exact same log | 14:07 |
| alee | well I guess it depends on the key size | 14:07 |
| alee | rellerreller, thanks | 14:07 |
| jaosorior | maybe we'll need to compress the CSRs? or change that value | 14:07 |
| jaosorior | well, that size | 14:08 |
| alee | jaosorior, yeah -- need some input from db people here -- woodster_ ? | 14:12 |
| alee | jaosorior, woodster_, rellerreller - looks like all the metadatum fields (secret, order, barbican metadata) have the same 255 column length. | 14:12 |
| alee | jaosorior, to make the column unbounded - would you use sa.Text instead? | 14:13 |
| jaosorior | alee: I guess, though I'm not sure if we want to make that unbounded | 14:14 |
| jaosorior | yeah, indeed it would be sa.Text | 14:15 |
| alee | certificateAuthorityMetadatum is also 255 --> will need to expand that one to be able to accomodate ca signing certs and intermediate chains | 14:16 |
| alee | jaosorior, woodster_ - at the very least, it should be ok to make OrderPluginMetaDatum, OrderBarbicanMetadatum and CertificateAuthorityMetadatum unbounded - or with a very high bound. The data that goes in there either comes from a plugin or from the barbican server itself. | 14:20 |
| jaosorior | alee: my concern is the possibility of an attacker taking advantage of this and providing very big payloads for the metadatum info. But then again that my only be tinfoil hat thinking.... and it could also be that we are in a way protected by the boundaries of the packages that can be sent to Barbican, I guess at some point we would return 413? | 14:22 |
| alee | actually the same is true of SecretMetadatum -- those inputs come from the plugin too. | 14:22 |
| alee | jaosorior, right -- the attacks would have to come from the plugin side | 14:23 |
| alee | (not from the front facing interface | 14:24 |
| alee | woodster_, redrobot ?? | 14:24 |
| jaosorior | but some of the metadatum would actually be coming from the front facing interface | 14:24 |
| jaosorior | IIRC | 14:24 |
| alee | jaosorior, no | 14:24 |
| alee | jaosorior, the data for an order for example is stored in the meta field of the order table | 14:25 |
| alee | which is a field that contains a json blob | 14:25 |
| alee | (not sure if its bounded or not) | 14:25 |
| jaosorior | yeah, but that's for the order, what about the secretmetadatum? | 14:26 |
| alee | the order has two metadata fields associated with it -- orderplugin and barbicanmeta -- the first is used by the plugins to store data abiout the order, the second by barbican itself to store data. | 14:26 |
| alee | for secret metadata -- ... | 14:27 |
| alee | jaosorior, I think that is only used by the plugin to store stuff about the secret | 14:27 |
| jaosorior | uhm, then you're right | 14:27 |
| jaosorior | it would be up to the plugin | 14:27 |
| alee | (for example an id) | 14:27 |
| *** rellerreller has quit IRC | 14:28 | |
| alee | jaosorior, guess we'll wait to hear from woodster_ and redrobot as to how they would like to proceed. you can still review the CR though :) | 14:29 |
| *** xaeth is now known as xaeth_afk | 14:51 | |
| * jvrbanac is wondering if the client devstack gate is somehow using an older client | 14:59 | |
| *** kebray has joined #openstack-barbican | 15:00 | |
| *** xaeth_afk is now known as xaeth | 15:13 | |
| jaosorior | jvrbanac: how come? | 15:15 |
| *** kebray has quit IRC | 15:16 | |
| jvrbanac | jaosorior, https://review.openstack.org/#/c/171465/ | 15:24 |
| jvrbanac | jaosorior, my changes fixes the very thing causing the functional test errors | 15:25 |
| woodster_ | alee, jaosorior just catching up...we do put an upper bound on the overall input request (10k by default I believe) | 15:25 |
| jaosorior | whaaa | 15:25 |
| woodster_ | well, 10k for the secret payload. I think the overall bound is higher than that | 15:25 |
| jaosorior | jvrbanac: that's weird... | 15:29 |
| jaosorior | woodster_ do you have an opinion about getting the metadatums to be unbounded in the database? | 15:29 |
| woodster_ | a concern raised by our dbas here was with clobs that are frequently retrieved but not used, due to perf impact. | 15:30 |
| alee | woodster_, which clobs would those be? | 15:31 |
| woodster_ | another option could be to have tables just for this specialized large scale data? | 15:32 |
| woodster_ | sorry, sa.Text is essentially a clob | 15:32 |
| alee | woodster_, right, I guess I'm wondering under which context the clob would not be used | 15:32 |
| woodster_ | well, for a lot of the key/value info on those metadata data tables the 255 limit is fine. It is just things like certs and so forth that break out of that. So maybe those could be on normalized tables with FKs to orders/secrets/containers and what not | 15:33 |
| *** SheenaG has joined #openstack-barbican | 15:36 | |
| alee | woodster_, so now we want extra tables for large values? so to process an order, I need to load up order-plugin-meta, barbican-meta, barbican-large-meta and order-plugin-large-meta? | 15:37 |
| jaosorior | alee: I'm not very keen on that option | 15:39 |
| alee | me neither | 15:39 |
| woodster_ | alee, well instead of barbican-large-generic-anything-goes-meta, I'm saying something like order-certificates or container-certificates and so forth...so specific tables that are only queried from if needed during a specific workflow. So generating an AES key (for example) would not incur the cost of pulling in generic clob data. | 15:39 |
| alee | woodster_, another possibility is to use a separate table for large meta, but make it transparent. So, when storing large meta, the plugin (or server) would need to call store_large_meta() and it will end up in the clob table. when retrieving though, it will just get the value. | 15:43 |
| alee | not sure how to do that but essentially some values in the meta table will point to the large meta table | 15:44 |
| alee | that way the code/plugin is explictly specifying which values are expected to be large. | 15:44 |
| woodster_ | alee, yeah, well my concern is with a lot of generic data...I much prefer specific attributes (so a specific field in the response dto from the plugin that barbican saves) over generic blobby stuff. Just my thoughts though :) | 15:45 |
| woodster_ | ...so if a specific attribute is unbounded, I can dedicate a table to it, roughly speaking | 15:45 |
| alee | woodster_, the whole idea behind the plugin meta table was to not constrain the plugin -- the plugin can store whatever it needs. | 15:46 |
| alee | woodster_, so if I understand your concern - performance on clobs is much slower than normal columns - even when the data is small? | 15:48 |
| alee | woodster_, ? | 16:01 |
| alee | woodster_, I'm not a db guy, so I don't understand the performance implications. The immediate problem is that we have at least bits of data that will exceed the 255 maximum -- a generated_csr stored in the barbican_meta_table and signing certs and intermediate certs which are in the CA metadata table. | 16:05 |
| alee | we need a way to make sure we can store those. | 16:05 |
| woodster_ | alee, the perf impact is basically it. I can try to get a better answer on what that impact is from our dbas though. | 16:05 |
| woodster_ | alee, certainly to fix the immediate issue we can go with clobs and then revisit for liberty | 16:06 |
| alee | woodster_, ok -- and we can do that purely for the CA meta and barbican meta tables if you like -- although I'd suggest for all the meta tables | 16:07 |
| alee | ie. secret meta and order-plugin-meta as well. | 16:07 |
| alee | but to solve the immediate problem, we just need barbican-plugin-meta and ca-meta | 16:08 |
| *** dave-mccowan has quit IRC | 16:09 | |
| alee | woodster_, so ca meta and barbican meta only? or secret_meta and order_plugin_meta as well? | 16:10 |
| openstackgerrit | John Vrbanac proposed openstack/python-barbicanclient: Fixing the broken functional tests https://review.openstack.org/171465 | 16:10 |
| woodster_ | alee, maybe start with the minimum we need for now? | 16:11 |
| alee | woodster_, sure - we can revisit as we need to | 16:11 |
| *** dave-mccowan has joined #openstack-barbican | 16:13 | |
| *** kebray has joined #openstack-barbican | 16:17 | |
| *** gyee has joined #openstack-barbican | 16:20 | |
| *** jkf has joined #openstack-barbican | 16:31 | |
| *** SheenaG has left #openstack-barbican | 16:54 | |
| *** SheenaG has joined #openstack-barbican | 16:54 | |
| *** xaeth is now known as xaeth_afk | 16:55 | |
| jvrbanac | redrobot, same problem | 16:57 |
| jvrbanac | :( | 16:57 |
| openstackgerrit | John Vrbanac proposed openstack/python-barbicanclient: Fixing the broken functional tests https://review.openstack.org/171465 | 17:07 |
| openstackgerrit | Merged openstack/barbican: Remove str() casting for the client_message variable https://review.openstack.org/167044 | 17:25 |
| *** rellerreller has joined #openstack-barbican | 17:48 | |
| *** jaosorior has quit IRC | 17:52 | |
| *** everjeje has quit IRC | 17:56 | |
| *** kfarr has joined #openstack-barbican | 17:57 | |
| *** igueths has joined #openstack-barbican | 18:04 | |
| jvrbanac | redrobot, down to 2 failures now: https://jenkins04.openstack.org/job/gate-python-barbicanclient-devstack-dsvm/15/console | 18:16 |
| jvrbanac | redrobot, any thoughts on why those requests would be coming back with a 403 | 18:17 |
| jvrbanac | woodster_, ^ | 18:17 |
| jvrbanac | redrobot, woodster_, http://logs.openstack.org/65/171465/3/check/gate-python-barbicanclient-devstack-dsvm/68d1088/logs/screen-barbican.txt.gz#_2015-04-08_17_31_16_259 | 18:19 |
| jvrbanac | redrobot, woodster_, is there a different policy for devstack than local configuration? | 18:20 |
| *** nkinder has quit IRC | 18:34 | |
| jvrbanac | It doesn't look like it... | 18:44 |
| *** dave-mccowan has quit IRC | 18:47 | |
| jvrbanac | Anyone have any ideas?? | 18:55 |
| rm_work | I always see this, so I assume it's a red herring: INFO barbican.openstack.common.policy [-] Can not find policy directory: policy.d | 18:57 |
| rm_work | right? | 18:57 |
| rm_work | Never been sure what causes that | 18:57 |
| jvrbanac | rm_work, not that one. This is where the barbican client devstack gate is failing a couple tests with 403s that give the following traceback in Barbican http://logs.openstack.org/65/171465/3/check/gate-python-barbicanclient-devstack-dsvm/68d1088/logs/screen-barbican.txt.gz#_2015-04-08_17_31_16_259 | 18:58 |
| rm_work | yeah | 18:59 |
| rm_work | that's what i'm saying -- the policy.d directory thing is a red herring, right? | 18:59 |
| rm_work | I assume unrelated, because i see it ALL the time, and if that were the problem i assume every test would fail, not just a couple | 19:00 |
| jvrbanac | the directory thing yes. But it's it's talking about our rbac roll not being correct or something | 19:00 |
| rm_work | yeah, the rest I don | 19:00 |
| jvrbanac | Locally it returns a 404 | 19:00 |
| rm_work | *don't know, i don't see that in my devstack either | 19:00 |
| rm_work | ah is this a new test? | 19:00 |
| rm_work | let me try in my env | 19:00 |
| rm_work | gotta re-spin, sec | 19:00 |
| openstackgerrit | Adam Harwell proposed openstack/barbican: Use the new Devstack external plugin method https://review.openstack.org/167885 | 19:00 |
| jvrbanac | old tests. I'm trying to fix the barbican-client gate | 19:01 |
| rm_work | ^^ BTW when this passes checks, it should be good to merge | 19:01 |
| jvrbanac | rm_work, grab this CR: https://review.openstack.org/#/c/171465/ | 19:01 |
| *** paul_glass has quit IRC | 19:01 | |
| rm_work | oh, client | 19:01 |
| rm_work | derp | 19:01 |
| rm_work | woodster_ / redrobot: ^^ the updated devstack CR will be ok to merge because it will work *alongside* the other method, so it won't break everything -- we can update the infra job once it's in | 19:02 |
| rm_work | and it makes testing changes in devstack MUCH simpleer | 19:03 |
| jvrbanac | rm_work, oooo hang on | 19:03 |
| jvrbanac | rm_work, derp https://review.openstack.org/#/c/171553/ | 19:03 |
| rm_work | lol | 19:04 |
| rm_work | whelp | 19:04 |
| *** paul_glass has joined #openstack-barbican | 19:04 | |
| alee | woodster_, does changing the column size require a migration script? or is that all handled with sqlalchemy? | 19:04 |
| jvrbanac | redrobot, woodster_, hockeynut, alee, rellerreller, could we get this merged? https://review.openstack.org/#/c/171553/ | 19:05 |
| rm_work | hmm | 19:06 |
| rm_work | I guess there is no way to do this without leaking information? | 19:06 |
| alee | woodster_, redrobot ? I ran the migration script but saw no changes | 19:06 |
| rm_work | because Ideally, you should get 403 whether a container exists or not, if you don't have access to that tenant's containers | 19:07 |
| rm_work | because getting a 404 vs a 403 leaks info (whether the container exists) | 19:07 |
| rm_work | but if the container doesn't exist, there can't be ACLs for it | 19:07 |
| rm_work | so there's no way to tell if you'd normally have access or not >_> | 19:07 |
| rm_work | anyone else think that's a problem? and/or just one we'll have to live with? or am I super paranoid | 19:08 |
| jvrbanac | rm_work, I believe based on prior discussions 404 is the preferred way | 19:08 |
| rm_work | k, so we'll leak a minor amount of info, but i guess we don't care / can't work around that | 19:08 |
| jvrbanac | rm_work, soo ideally, we should return a 404 if they don't have permission as well | 19:09 |
| rm_work | ah, heh | 19:09 |
| rm_work | yeah I guess actually that is the proper solution | 19:09 |
| jvrbanac | that way you don't leak existence | 19:09 |
| rm_work | yep | 19:09 |
| alee | rellerreller, ^ any idea? | 19:10 |
| jvrbanac | alee, I believe changing the column size does require a migration | 19:10 |
| alee | jvrbanac, interesting -- I tried running the db script and saw nothing generated | 19:11 |
| jvrbanac | alee, The migration generation script is kind of easy to mess up. I mess it up all the time. | 19:11 |
| jvrbanac | Usually if it doesn't generate anything it's because the state of your db is newer | 19:12 |
| jvrbanac | alee, http://docs.openstack.org/developer/barbican/contribute/database_migrations.html#automatically | 19:12 |
| rellerreller | alee I usually prefer not to mention anything at all. If item does not exist or if they do not have permissions then I usually return the same error, "you do not have permission for this object." | 19:12 |
| jvrbanac | rellerreller, I think that was for rm_work ;) | 19:13 |
| alee | jvrbanac, yeah - thats the steps I followed | 19:14 |
| jvrbanac | two conversations going on at once lol | 19:14 |
| rellerreller | It's for everyone :) alee happened to ask me the question. | 19:14 |
| alee | rellerreller, you're answering rm_work question .. | 19:14 |
| rellerreller | alee https://review.openstack.org/#/c/169600/ is the thing you wanted me to review, right? | 19:15 |
| alee | rellerreller, yes | 19:15 |
| rellerreller | alee good. I have 15 free minutes now. | 19:15 |
| redrobot | rellerreller alee so I'm not sure why we're trying to do DER->PEM conversions in Barbican. Are there no libraries that can convert that for us? | 19:16 |
| alee | rellerreller, I discovered though that the gate was failing because it was trying to store the generated csr in barbican-meta table and failing because that table has a limit of 256 | 19:16 |
| alee | so my question is -- changing from 256 -> sa.Text would presumably need a migration script , right? | 19:17 |
| alee | but the db script returns no output. | 19:17 |
| alee | maybe the script can't handle it ? | 19:18 |
| alee | redrobot, we basically return the keys as DER - which is fine except that I need to use that key within barbican to generate a csr for the stored key case. | 19:19 |
| alee | redrobot, the only way I can do that for the case where the stored key is passphrase encrypted is to convert to PEM and import that way | 19:20 |
| jvrbanac | alee, so you started with an empty db, started up Barbican in the old state, killed barbican, checked out your latest code, and then ran the script? | 19:20 |
| redrobot | alee my question was more about why we're manually doing the DER to PEM conversion, instead of using a library | 19:20 |
| rm_work | alee: yeah, we have some code in Octavia and Neutron for doing that kind of operation -- which is the stuff I wanted to live in Castellan but got shot down ;P | 19:20 |
| alee | jvrbanac, yup | 19:20 |
| alee | jvrbanac, let me try again .. | 19:21 |
| rm_work | looks like you are now getting to experience the pain :P | 19:21 |
| alee | redrobot, agreed it would be nicer to use a library -- follow on CR for liberty? | 19:22 |
| jvrbanac | rm_work, alee, redrobot, you might look at cryptography for that. They're starting to add x509 stuff in: https://cryptography.io/en/latest/x509/ | 19:23 |
| rm_work | yeah that should be useful once it's complete | 19:24 |
| alee | jvrbanac, yes - thats the plan. they dont have the ability to generate a csr yet. | 19:24 |
| jvrbanac | I'm sure they wouldn't mind PRs ;) | 19:24 |
| redrobot | alee for sure! ... I'm still drudging through every possible secret-type/content-type workflow, so I'll probably have some questions for rellerreller and you later. | 19:24 |
| redrobot | alee rellerreller if y'all get a chance, could you drop some comments on my first stab at the "symmetric" secret type? | 19:25 |
| alee | jvrbanac, the plan has always been to replace all this openSSL / conversion stuff with crytography when its ready | 19:25 |
| redrobot | alee rellerreller https://review.openstack.org/#/c/171488/ | 19:25 |
| rm_work | redrobot: would you have a problem merging https://review.openstack.org/#/c/167885/ soon? it should pass checks now | 19:27 |
| rm_work | want to get started on this as it's a three-parter | 19:27 |
| redrobot | rm_work part two is the infra CR? | 19:28 |
| rellerreller | alee Do you have time to discuss your CR? | 19:28 |
| rm_work | redrobot: part two is Infra, part three is cleaning up the old method | 19:28 |
| rellerreller | alee I'm curious why translations had to change. | 19:28 |
| alee | rellerreller, looking | 19:29 |
| redrobot | rm_work sounds like a fine approach. be sure to add "Depends-on: I0ec63819b3aae21a6ffaed5cf8285e26dce6ae94" to the infra CR | 19:29 |
| alee | rellerreller, which part? | 19:30 |
| rm_work | redrobot: that's only if I submit the infra CR before this merges, right? | 19:30 |
| alee | rellerreller, first off --- all the translations code was run without any '\n''s in it in the pem functions | 19:30 |
| redrobot | rm_work yeah... are you going to wait for this CR to submit to infra? I would submit now since they usually take a couple of days to merge things | 19:31 |
| alee | rellerreller, that resulted in some faulty logic once the '\n's were added | 19:31 |
| rellerreller | alee resources.py get_secret now takes parameter for pem_needed, but I don't see why that is needed. | 19:31 |
| rm_work | redrobot: hmm k | 19:31 |
| alee | rellerreller, ok - so the idea here is that in the stored key case I need to take the key and generate a csr. | 19:32 |
| alee | rellerreller, now the way I do that is openssl.crypto.load_privatekey() | 19:33 |
| rellerreller | alee Stored key case is when generate a CSR that is signed with key that already exists in Barbican??? | 19:33 |
| alee | yes | 19:33 |
| alee | rellerreller, so I need to load this key in PEM format. | 19:34 |
| rellerreller | alee And default return from get_secret is binary, right? | 19:35 |
| alee | yup | 19:35 |
| rellerreller | in resources.py. | 19:35 |
| rellerreller | Sorry, I was on vacation and tried to forget about all of this for a while ;) | 19:35 |
| alee | rellerreller, I need a vacation now too .. | 19:35 |
| rellerreller | So you want to have extra parameter to return in PEM format and then use it load and sign CSR. Got it. | 19:36 |
| alee | yup -- in anything, this is a precursor to the ability to request the rerturn format of the secret | 19:36 |
| alee | rather than just returning binary | 19:36 |
| rellerreller | alee I like that | 19:37 |
| redrobot | alee rellerreller so, the return format of the secret is listed in the content_types property | 19:37 |
| alee | (except that we'll consider exposing that to the api in liberty) | 19:37 |
| rellerreller | alee is there any code in your CR that invokes this new functionality? I cannot seem to find any. Will there be a subsequent patch? | 19:37 |
| alee | redrobot, the way things are implemented - we always return binary | 19:38 |
| rellerreller | redrobot That is not honored. You can store a key in base64 content type, but it will always be returned in binary. | 19:38 |
| alee | rellerreller, see certificate_resources.py | 19:38 |
| redrobot | alee rellerreller base64 is an encoding, not a content type. | 19:38 |
| openstackgerrit | Merged openstack/barbican: Return container not found before ACL checks https://review.openstack.org/171553 | 19:38 |
| alee | redrobot, this is why we need you to do your investigation :) | 19:39 |
| redrobot | alee rellerreller base64 as an encoding was introduced to allow non-ascii characters inside the JSON request. | 19:39 |
| rm_work | redrobot: err, do you happen to know WHERE the code is in infra for the dsvm job? | 19:39 |
| rm_work | redrobot: I was assuming it'd be here: https://github.com/openstack-infra/project-config/blob/e0f26735601f475566bae08ac91f63340eafce72/jenkins/jobs/barbican.yaml | 19:40 |
| rm_work | but it looks like not | 19:40 |
| redrobot | rm_work that is the correct file for the barbican devstack jobs. We only have two. The first one is the one that always runs, the -f21 one is the one that does dogtag and is still experimental | 19:41 |
| redrobot | rm_work what were you expecting to find? | 19:41 |
| rm_work | redrobot: i don't see how it's deciding where to pull barbican from | 19:41 |
| rm_work | it should be cloning barbican and moving files around and stuff | 19:41 |
| redrobot | rm_work these two lines tell devstack to clone barbican https://github.com/openstack-infra/project-config/blob/e0f26735601f475566bae08ac91f63340eafce72/jenkins/jobs/barbican.yaml#L19-L20 | 19:42 |
| redrobot | rm_work and also clone python-barbicanclient | 19:42 |
| rm_work | yeah but something also tells it to copy the files around from ./contrib/devstack/ | 19:42 |
| redrobot | rm_work yes, after it's cloned by devstack, the pre_test_hook moves the lib and extras.d files to the devstack structure here: https://github.com/openstack/barbican/blob/master/functionaltests/pre_test_hook.sh#L20-L21 | 19:43 |
| rm_work | OH, it's part of your own code | 19:43 |
| rm_work | looool ok | 19:43 |
| rm_work | maybe I need to do more research, might be able to do it in one step | 19:44 |
| redrobot | rm_work cool cool, glad I could help :) | 19:44 |
| rm_work | or actually, no, but that will need to change | 19:44 |
| rm_work | k | 19:44 |
| redrobot | rm_work does that secure your vote for me for PTL? :D | 19:44 |
| rm_work | heh, are you running opposed this time? :P | 19:44 |
| redrobot | rm_work nope, but I still feel like I need to campaign | 19:44 |
| rm_work | lol | 19:45 |
| rm_work | woo and devstack is broken | 19:48 |
| rm_work | https://bugs.launchpad.net/devstack/+bug/1441820 | 19:48 |
| openstack | Launchpad bug 1441820 in devstack "stack.sh fails because /opt/stack/requirements/global-requirements.txt isn't present" [Undecided,In progress] - Assigned to Amrith (amrith) | 19:48 |
| rm_work | https://review.openstack.org/#/c/171788/ | 19:48 |
| rm_work | ^^ merging soon | 19:48 |
| woodster_ | devstuck? | 19:49 |
| rm_work | heh | 19:49 |
| woodster_ | alee, I see you comments about migrations above.... | 19:49 |
| woodster_ | alee, I would think you need to alter the column type in the migration script. | 19:50 |
| rellerreller | alee I reviewed the CR. I guess I only have the dead code comment. I would like to see the is_public_key_valid implemented, but I do not see a way load the public key using the OpenSSL library you use for is_private_key_valid. | 19:51 |
| woodster_ | any Python logging experts out there? Stevedore isn't loading a plugin because of an exception the plugin __init__ throws, but the stack trace isn't showing up in logs even though they do a log.exception. Do I really need to do a logging.getLogger('strevedpre) | 19:56 |
| woodster_ | ...sort of thing? | 19:56 |
| redrobot | woodster_ I would first make sure that the logging level of your process matches the logging level of the stevedore thing. maybe your logging level is higher than what stevedore is spitting out? | 19:57 |
| woodster_ | logging.getLogger('strevedore').setLevel(logging.INFO) that is | 19:57 |
| alee | rellerreller, right - thats why I left it for now. | 19:57 |
| alee | rellerreller, we can revisit when cryptography is ready | 19:58 |
| woodster_ | My running app is at DEBUG level, but import foo on the plugin manager module might be grabbing the logger at a higher level? | 19:58 |
| alee | rellerreller, the private key is really whats important in any case --- the public key can be derived from it | 19:58 |
| alee | rellerreller, I'll remove the dead code | 19:58 |
| alee | woodster_, I'm trying this --- but its not working so far .. | 19:59 |
| woodster_ | redrobot, rellerreller, kfarr that is essentially the issue with Christopher in the maining list, trying to use the KMIP plugin | 19:59 |
| alee | woodster_, http://www.fpaste.org/208648/85231691/ | 19:59 |
| woodster_ | alee, you might want to examine the actual schema with psql or some such tool. try to upgrade/downgrade versions to see how the schema changes. The autogen tool is lmited, missing things like default values for new non-nullable columns for example/ | 20:00 |
| woodster_ | alee, that looks good. Just try to upgrade/downgrade to/from that version file and verify the schema changes a expected, ideally with some data already in the database when you do it | 20:02 |
| woodster_ | alee, the clob > 255 to string(255) might crash though? | 20:02 |
| *** nickrmc83 has quit IRC | 20:02 | |
| kfarr | woodster_ glad you figured that out! I was stumped | 20:04 |
| *** nickrmc84 has joined #openstack-barbican | 20:04 | |
| kfarr | woodster_ Which section of the code is the part that is failing but not showing up in the logs? | 20:05 |
| woodster_ | kfarr, yeah it is super annoying! My local setup is borking because KMIP is looking for a cert file I don't have, but stevedore traps/logs the exception and then doesn't load the module. So barbican just thinks there are no modules loaded | 20:05 |
| woodster_ | kfarr, this line: https://github.com/openstack/barbican/blob/master/barbican/plugin/kmip_secret_store.py#L131 | 20:06 |
| *** paul_glass1 has joined #openstack-barbican | 20:07 | |
| *** nkinder has joined #openstack-barbican | 20:09 | |
| alee | woodster_, well I'm trying the upgrade using the script -- but its failing -- I''m using this .. | 20:09 |
| alee | ./bin/barbican-db-manage.py --dburl sqlite:////var/lib/barbican/barbican.sqlite upgrade -v head | 20:09 |
| alee | and it fails with : OperationalError: (OperationalError) near "ALTER": syntax error u'ALTER TABLE order_barbican_metadata ALTER COLUMN value TYPE TEXT' () | 20:10 |
| *** paul_glass has quit IRC | 20:10 | |
| kfarr | woodster_ Where is the log.exception? | 20:12 |
| *** dave-mccowan has joined #openstack-barbican | 20:14 | |
| woodster_ | alee, so are you going from the previous db version to the latest one then? | 20:14 |
| alee | woodster_, yup | 20:14 |
| woodster_ | kfarr, this log.exception is from the stevedore side | 20:14 |
| rellerreller | alee I will be good with CR once dead code is removed. Looks good :) | 20:14 |
| alee | woodster_, just a single script | 20:14 |
| alee | rellerreller, cool -- trying to add the migration script/db change in there | 20:15 |
| alee | (have already removed dead code) | 20:15 |
| woodster_ | alee, so maybe type alterations require difference syntax? too bad it doesn't auto generate that for you :\ | 20:15 |
| alee | yeah -- ..it looks right though -- https://alembic.readthedocs.org/en/latest/ops.html#alembic.operations.Operations.alter_column | 20:16 |
| woodster_ | alee, maybe try a column name alter first to see if that works correctly | 20:20 |
| alee | woodster_, ok - maybe it just doesn;t like me .. OperationalError: (OperationalError) near "value": syntax error u'ALTER TABLE order_barbican_metadata RENAME value TO foo' () | 20:22 |
| alee | woodster_, can you try it on your db? | 20:23 |
| redrobot | alee is it possible that it's just not supported in sqlite? I recall someone mentioning that migrations in sqlite are not possible. | 20:23 |
| alee | that would be weird -- I fell like I've done migrations efore .. | 20:24 |
| redrobot | IIRC (and there's a good chance that I don't) You can add new tables to sqlite databases, but editing existing tables is a no-go | 20:25 |
| redrobot | alee "SQLite supports a limited subset of ALTER TABLE" http://www.sqlite.org/lang_altertable.html | 20:27 |
| alee | redrobot, ok - thats likely it then | 20:28 |
| *** crc32 has joined #openstack-barbican | 20:30 | |
| woodster_ | alee, sorry I thought you been doing this with a postgres db for some reason. The migration script used to warn you if you were using sqlite though :\ | 20:31 |
| alee | WARNING barbican.model.migration.commands [-] !!! Limited support for migration commands using sqlite databases; This operation may not succeed. | 20:31 |
| alee | woodster_, yeah - who heeds warnings? | 20:31 |
| alee | thats like reading the manual .. | 20:32 |
| woodster_ | alee, ha! | 20:33 |
| alee | woodster_, rellerreller , redrobot - new version being uploaded momentarily | 20:41 |
| openstackgerrit | Ade Lee proposed openstack/barbican: Changes to get remaining cert functional tests working https://review.openstack.org/169600 | 20:42 |
| alee | it would be super sweet if we could get this in today | 20:42 |
| *** rellerreller has quit IRC | 20:45 | |
| *** dfinnrml has joined #openstack-barbican | 20:53 | |
| *** dimtruck is now known as zz_dimtruck | 20:54 | |
| jvrbanac | redrobot, soo it looks like our devstack gate is functioning now; however, there seems to be a weird problem with the neutron gate that has nothing to do with us. | 20:55 |
| jvrbanac | redrobot, https://review.openstack.org/#/c/171465/ | 20:55 |
| redrobot | jvrbanac ugh... | 20:56 |
| *** zz_dimtruck is now known as dimtruck | 20:57 | |
| openstackgerrit | Adam Harwell proposed openstack/barbican: Use the new Devstack external plugin method https://review.openstack.org/167885 | 20:58 |
| rm_work | let's see if this works :P | 20:58 |
| igueths | Can someone poke https://review.openstack.org/#/c/170693 at some point? Latest patchset is considerably smaller than what originally went in. | 20:58 |
| kfarr | Hey all, we were encountering what might be a bug when trying to create and retrieve an asymmetric key pair: https://bugs.launchpad.net/barbican/+bug/1441848 | 20:58 |
| openstack | Launchpad bug 1441848 in python-barbicanclient "unexpected keyword argument 'sub_status' when getting an order" [Undecided,New] | 20:58 |
| kfarr | Am I missing something obvious? | 20:59 |
| jvrbanac | kfarr, https://review.openstack.org/#/c/171465/ | 21:00 |
| rm_work | jvrbanac: yeah devstack in gate is f'd | 21:00 |
| redrobot | kfarr asymmetric key pair in Barbican proper? I don't think the client has support for typed secrets yet | 21:00 |
| jvrbanac | kfarr, however, that error you're seeing about sub_status message should be resolved in the CR I linked | 21:01 |
| jvrbanac | rm_work, :( | 21:02 |
| kfarr | redrobot, we were using python-barbicanclient | 21:02 |
| kfarr | jvrbanac, I'll see if your patch fixes it, (it looks like it'll work) thanks! | 21:02 |
| rm_work | jvrbanac: waiting for https://review.openstack.org/167885 to merge | 21:03 |
| rm_work | err | 21:03 |
| rm_work | sorry wrong one | 21:03 |
| rm_work | https://review.openstack.org/171788 | 21:03 |
| jvrbanac | rm_work, http://files.sharenator.com/Finetunning-s347x295-410716-580a.gif | 21:04 |
| rm_work | lol | 21:04 |
| kfarr | jvrbanac, redrobot, https://review.openstack.org/#/c/171465/ fixes it, thanks! | 21:07 |
| jvrbanac | kfarr, np | 21:07 |
| redrobot | anyone have a working barbican right now? | 21:16 |
| redrobot | can someone run these commands and let me know if it works? | 21:16 |
| redrobot | http://paste.openstack.org/show/200709/ | 21:16 |
| jvrbanac | redrobot, 400 bad request | 21:17 |
| redrobot | jvrbanac crap. I think that's a bug | 21:18 |
| alee | redrobot, woodster_ rellerreller -- looks like it passed the gate this time | 21:24 |
| alee | redrobot, woodster_ rellerreller, jaosorior -- https://review.openstack.org/#/c/169600/3 waiting for some +2's .. | 21:25 |
| alee | jvrbanac, hockeynut ^^ | 21:25 |
| redrobot | alee will look at it. also, does this sequence of steps look correct to you for a one POST request: http://paste.openstack.org/show/200709/ | 21:26 |
| alee | redrobot, interesting -- so you base64 encode the entire thing including headers and footers | 21:28 |
| redrobot | alee yes, that's what we do with asymmetric keys. An alternative would be to json escape the newlines and call the content type "text/plain" ? | 21:29 |
| alee | redrobot, I think you need to put together a set of scripts like this that you think would correspond to valid inputs - and then we can discuss. | 21:31 |
| alee | redrobot, thats what the google hangout was supposed to be about | 21:31 |
| redrobot | alee yep, but going through every scenario is taking way longer than I expected | 21:32 |
| openstackgerrit | Adam Harwell proposed openstack/barbican: Use the new Devstack external plugin method https://review.openstack.org/167885 | 21:32 |
| openstackgerrit | werner mendizabal proposed openstack/barbican: Create Barbican python scripts for development. https://review.openstack.org/170961 | 21:32 |
| alee | redrobot, well - the plus side is that these scripts form the basis of documentation | 21:32 |
| alee | redrobot, put together what you think should be valid -- and then we can decide if it actually is. | 21:33 |
| redrobot | alee I'm actually writing the docs as I go... killing two birds with one stone if you will. | 21:33 |
| alee | redrobot, and we can attempt to do tings like create a csr from it for the stored key case | 21:33 |
| alee | redrobot, thats probably the best test case -- 1) store public, private key, passphrase 2) use them to generate a CSR in the stored key case | 21:36 |
| alee | that way we can see if we can actually use what is stored internally | 21:38 |
| redrobot | alee agreed | 21:38 |
| alee | redrobot, of course to test all this out - we need my patch in first :) | 21:38 |
| alee | hint, hint, nudge, nudge .. | 21:39 |
| redrobot | lol | 21:39 |
| *** chlong has joined #openstack-barbican | 22:01 | |
| *** dimtruck is now known as zz_dimtruck | 22:03 | |
| *** jamielennox is now known as jamielennox|away | 22:11 | |
| *** paul_glass1 has quit IRC | 22:11 | |
| openstackgerrit | John Vrbanac proposed openstack/python-barbicanclient: Raising errors from the client instead of keystone https://review.openstack.org/171839 | 22:17 |
| openstackgerrit | Adam Harwell proposed openstack/barbican: Use the new Devstack external plugin method https://review.openstack.org/167885 | 22:31 |
| dave-mccowan | alee, woodster_ any thoughts on this refactoring CR? https://review.openstack.org/#/c/171023/ I have one more patch for validators, and want to know which code to base it on. | 22:41 |
| *** SheenaG has left #openstack-barbican | 22:50 | |
| openstackgerrit | Dave McCowan proposed openstack/barbican: Add Functional Tests for Private Key Secret Type https://review.openstack.org/169974 | 22:54 |
| *** kebray has quit IRC | 22:56 | |
| openstackgerrit | Douglas Mendizábal proposed openstack/barbican: Document Symmetric Secret Type https://review.openstack.org/171488 | 22:56 |
| kfarr | woodster_ ping | 22:56 |
| openstackgerrit | Charles Neill proposed openstack/barbican: Security tests for Order resources https://review.openstack.org/171858 | 22:58 |
| openstackgerrit | Douglas Mendizábal proposed openstack/barbican: Document public secret type https://review.openstack.org/171859 | 22:59 |
| *** chlong has quit IRC | 23:05 | |
| openstackgerrit | John Wood proposed openstack/barbican: Expose root cause plugin exceptions https://review.openstack.org/171868 | 23:11 |
| woodster_ | kfarr, hello | 23:11 |
| *** dfinnrml has quit IRC | 23:12 | |
| *** jamielennox|away is now known as jamielennox | 23:12 | |
| kfarr | woodster_ I've been thinking about what could be done to fix that logging issue you mentioned earlier. Have you come to any conclusions? | 23:12 |
| woodster_ | kfarr, ha, I just put up a CR for that: https://review.openstack.org/171868 | 23:13 |
| woodster_ | kfarr see what you think though, to see if there's a better way to deal with that | 23:13 |
| kfarr | oh, cool! | 23:14 |
| woodster_ | kfarr, for some reason the logger on stevedore is being set to disabled even if I get its logger right before I call the plugin manager enable it. So the exception logging it does fails. The bigger problem though is that it keeps on processing and returns to barbican. It isn't until barbican tries to locate a plugin later that you see an error. | 23:15 |
| woodster_ | kfarr, so config issues with plugins should fail fast so you know as early as possible soething is misconfigured | 23:15 |
| openstackgerrit | Adam Harwell proposed openstack/barbican: Use the new Devstack external plugin method https://review.openstack.org/167885 | 23:15 |
| kfarr | woodster_ Great! That seems like a good solution. I didn't know if it was fixable and was wondering if we just needed to move an error-creating code out of the plugins. I like your code better | 23:17 |
| woodster_ | kfarr, I was hoping to keep the plugins as simple as possible, and hard-broken config issues a plugin detects should be raised as an exception imho | 23:18 |
| woodster_ | rm_work, are you trying to get a clean CR for the devstack stuff? | 23:19 |
| openstackgerrit | John Vrbanac proposed openstack/python-barbicanclient: Raising errors from the client instead of ksclient https://review.openstack.org/171839 | 23:19 |
| *** kebray has joined #openstack-barbican | 23:33 | |
| *** jkf has quit IRC | 23:44 | |
| openstackgerrit | Merged openstack/barbican: Imported Translations from Transifex https://review.openstack.org/171475 | 23:50 |
| openstackgerrit | John Wood proposed openstack/barbican: Add retry server and functional tests to DevStack https://review.openstack.org/170896 | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
