Monday, 2015-03-02

*** dimtruck is now known as zz_dimtruck00:58
hockeynutwoodster_ for https://review.openstack.org/#/c/159991/ would you be interested in changing your +2 to a workflow?02:56
hockeynutwoodster_ also see comments added to https://review.openstack.org/#/c/160016/02:57
woodster_Ha, done02:59
woodster_hockeynut, that makes sense, changed to +203:01
reaperhulkFor those who've been waiting, RSA private key serialization landed in cryptography this morning after 167 comments, several months, and at least 3 major changes to the API proposal :) https://github.com/pyca/cryptography/pull/150303:02
reaperhulkEC, DSA, and public key work is frantically being written now so I can get 0.8 out the door this upcoming week :p03:03
hockeynutthx woodster_ !03:29
mjg59\o/03:38
*** nkinder has joined #openstack-barbican03:38
woodster_reaperhulk, that's good news!03:38
*** dave-mccowan has quit IRC03:50
*** david-lyle_afk has quit IRC04:02
openstackgerritMerged openstack/python-barbicanclient: Update functionaltests to be able to run tox -e functional  https://review.openstack.org/15999104:04
*** kebray has joined #openstack-barbican05:01
*** kebray has quit IRC05:03
*** kebray has joined #openstack-barbican05:07
*** kebray_ has joined #openstack-barbican06:27
*** kebray has quit IRC06:29
*** kebray_ has quit IRC06:59
*** jaosorior has joined #openstack-barbican07:31
openstackgerritJuan Antonio Osorio Robles proposed openstack/barbican: Use urljoin instead of os.path.join  https://review.openstack.org/16025007:48
*** woodster_ has quit IRC07:50
*** chlong has quit IRC08:02
*** Guest78669 is now known as d0ugal08:20
*** d0ugal has joined #openstack-barbican08:21
*** tkelsey has joined #openstack-barbican09:05
*** darrenmoffat has quit IRC10:07
*** darrenmoffat has joined #openstack-barbican10:08
*** dave-mccowan has joined #openstack-barbican12:21
openstackgerritJuan Antonio Osorio Robles proposed openstack/barbican: Get rid of Repositories class  https://review.openstack.org/16032912:52
*** woodster_ has joined #openstack-barbican12:59
jaosoriorwoodster_: are you around?13:58
woodster_jaosorior: heading to gym shortly but can IRC by phone13:59
jaosorioroh, alright13:59
jaosoriorI'm just getting a weird error in the functional tests13:59
jaosoriorsomething about get_auth_provider not being in the Manager class13:59
jaosoriorhttp://logs.openstack.org/29/160329/1/check/gate-barbican-devstack-dsvm/56efcd4/console.html14:00
woodster_jaosorior  Hmm there was a commit to the functional tests last night14:04
jaosoriorok, let me pull it14:04
woodster_jaosorior: or maybe that is a Barbican client test, so a recent change to client?14:05
jaosoriornope, I have already pulled the latewst14:05
*** SheenaG1 has quit IRC14:05
*** lisaclark1 has joined #openstack-barbican14:17
*** nkinder has quit IRC14:19
woodster_jaosorior: I think those latest changes might be borking things somehow?14:19
dave-mccowanjaosorior: are you using override_url in the config file?  i made a change to split the override url into two parts (base and version).  make sure your local changes didn't overwrite that.   override_url=http://localhost:931114:29
dave-mccowanoverride_url_version=v114:29
jaosoriordave-moccowan: that commit doesn't touch the configuration :/14:30
jaosoriorI didn't touch a single file in functionaltests/14:34
*** lisaclark1 has quit IRC14:41
*** lisaclark1 has joined #openstack-barbican14:42
jaosoriordave-mccowan: as a matter of fact, it seems that the version is not being included somehow14:48
jaosoriorI see a similar problem in two different commits14:48
*** SheenaG1 has joined #openstack-barbican14:56
openstackgerritJuan Antonio Osorio Robles proposed openstack/barbican: Use urljoin instead of os.path.join  https://review.openstack.org/16025015:03
*** rm_work has quit IRC15:05
*** rm_work|away has joined #openstack-barbican15:07
*** rm_work|away is now known as rm_work15:07
*** rm_work has quit IRC15:07
*** rm_work has joined #openstack-barbican15:07
*** rm_work has quit IRC15:12
*** nkinder has joined #openstack-barbican15:13
*** kfarr has joined #openstack-barbican15:14
*** rm_work|away has joined #openstack-barbican15:15
*** rm_work|away is now known as rm_work15:15
*** rm_work has joined #openstack-barbican15:15
*** paul_glass has joined #openstack-barbican15:17
jaosoriordave-mccowan: seems your commit did not pass the gate because of the same issue: http://logs.openstack.org/72/160172/1/gate/gate-barbican-devstack-dsvm/eeaf39f/console.html15:17
jaosoriortrying to debug it15:18
*** ametts has joined #openstack-barbican15:22
*** lisaclark1 has quit IRC15:22
*** lisaclark1 has joined #openstack-barbican15:27
*** lisaclark1 has quit IRC15:28
*** xaeth_afk is now known as xaeth15:36
*** zz_dimtruck is now known as dimtruck15:38
*** jorge_munoz has joined #openstack-barbican15:46
*** paul_glass has quit IRC15:47
*** lisaclark1 has joined #openstack-barbican16:02
*** dave-mccowan has quit IRC16:08
woodster_hockeynut: have you seen the issue above before?16:08
*** kfox1111 has joined #openstack-barbican16:21
jvrbanacwoodster_, jaosorior, I'm guessing something changed with tempest16:22
jaosoriorjvrbanac: would appear so, since it's a client coming from it16:23
jaosoriorDidn't get to figure out what it was since ib had to catch a train16:23
*** lisaclark1 has quit IRC16:27
*** dave-mccowan has joined #openstack-barbican16:28
*** lisaclark1 has joined #openstack-barbican16:28
*** gyee has joined #openstack-barbican16:42
*** paul_glass has joined #openstack-barbican16:50
*** lisaclark1 has quit IRC16:53
*** lisaclark1 has joined #openstack-barbican16:57
*** bdpayne has joined #openstack-barbican17:04
*** arunkant has joined #openstack-barbican17:12
*** rellerreller has joined #openstack-barbican17:20
*** kebray has joined #openstack-barbican17:23
*** lisaclark1 has quit IRC17:34
*** lisaclark1 has joined #openstack-barbican17:38
openstackgerritArun Kant proposed openstack/barbican-specs: Spec for adding audit capability using CADF specification.  https://review.openstack.org/15993817:56
*** kebray has quit IRC17:59
openstackgerritThomas Dinkjian proposed openstack/python-barbicanclient: Adds positive orders functional tests  https://review.openstack.org/15845418:00
*** lisaclark1 has quit IRC18:00
openstackgerritThomas Dinkjian proposed openstack/python-barbicanclient: Adds positive orders functional tests  https://review.openstack.org/15845418:01
*** jkf has joined #openstack-barbican18:04
*** kebray has joined #openstack-barbican18:04
*** igueths has joined #openstack-barbican18:16
iguethsHello.18:16
openstackgerritgreghaynes proposed openstack/barbican: Create snakeoil certificate plugin  https://review.openstack.org/14057518:22
openstackgerritNathan Reller proposed openstack/barbican: Fixed Binary Encoding to Secret Stores  https://review.openstack.org/15741018:22
openstackgerritNathan Reller proposed openstack/barbican: Standardized Secret Encoding  https://review.openstack.org/16044418:22
openstackgerritNathan Reller proposed openstack/barbican-specs: Add Asymmetric Key Support to KMIPSecretStore  https://review.openstack.org/16044918:30
arunkantwoodster_, Have question on Line 167 comment of https://review.openstack.org/#/c/159938/2/specs/kilo/audit-cadf-events.rst,cm . Please ping me when you have time18:42
openstackgerritJuan Antonio Osorio Robles proposed openstack/barbican: Fix functional tests to use new auth provider interface  https://review.openstack.org/16045518:45
jvrbanacredrobot, ^^18:45
jaosoriorjvrbanac, woodster_: Found the cause of the error.18:47
redrobotjaosorior nice18:48
jvrbanacjaosorior, I'll +2 that sucker as soon as the gates pass18:48
woodster_jaosorior, nice! just +2-ed it. jvrbanac had it right...dern > signs in requirements.txt :)18:48
*** kebray has quit IRC18:49
jvrbanacwoodster_, requirements?18:49
jaosoriortoo bad it's gonna take a long time since there's a huge queue on the gate18:49
jaosoriorbut oh well18:49
woodster_jvrbanac, well test-requirements.txt18:49
jvrbanacwoodster_, It's not in test-requirements https://github.com/cloudkeep/barbican/blob/master/tox.ini#L5118:50
jvrbanacwoodster_, woops wrong link18:50
jvrbanacwoodster_, however, I believe the devstack setup does the same thing18:50
jaosoriorjvrbanac: indeed it isn't18:50
*** kfarr has quit IRC18:50
woodster_jvrbanac, ah, direct to master, I didn't realize that18:51
jaosorioranyway, it seems the commit has been in for 5 days, and we hadn't had a problem: https://github.com/openstack/tempest/commit/90012355b4ea24a7321f66b329b543f306d2cefe18:51
jvrbanacwoodster_, interesting18:52
jaosorioraaah no, authored 5 days ago18:52
jaosoriordunno when it was merged18:52
jvrbanacjaosorior, interesting18:52
jvrbanachttps://review.openstack.org/#/c/159267/18:52
jaosoriorI see, merged today18:52
woodster_seems like a significant change to make in between releases18:52
woodster_might be other projects affected by that one18:53
jaosorioryeah, that's what I was thinking18:54
*** crc32 has joined #openstack-barbican18:55
jvrbanacjaosorior, woodster_, probably only incubated projects that wrote some functional tests that use some tempest stuff18:55
jaosoriorjvrbanac: So... did we do this the wrong way?18:55
woodster_The plan is to move away from project things being in the tempest repository...so the call back hook sort of approach we use is where things are going, if that's what you mean18:57
redrobotjaosorior most projects keep their functional tests in the Tempest tree, so they probably were not affected.18:58
openstackgerritKevin Fox proposed openstack/barbican: VM Integration  https://review.openstack.org/15957319:00
*** rellerreller has quit IRC19:02
jaosoriorOK, that makes sense19:02
*** crc32 has quit IRC19:03
*** rellerreller has joined #openstack-barbican19:04
*** kfox1111 has quit IRC19:09
*** kfox1111 has joined #openstack-barbican19:12
*** crc32 has joined #openstack-barbican19:12
kfox1111https://review.openstack.org/#/c/159571/ <- I can't seem to figure out how to mark shell/json in the .rst file in a way it will like. any ideas?19:22
woodster_kfox1111, check out https://review.openstack.org/#/c/127353/5/specs/kilo/add-creator-only-option.rst,cm line 104...that double colon and the json indention and line spaces thereafetr19:28
woodster_kfox1111, the result is rendered here: http://specs.openstack.org/openstack/barbican-specs/specs/kilo/add-creator-only-option.html19:28
*** shakamunyi has joined #openstack-barbican19:32
*** dimtruck is now known as zz_dimtruck19:32
kfox1111yeah. thats what I was trying to do. thanks. :)19:32
kfox1111oh. so they don't support the types for making it pretty. ok.19:33
*** tkelsey has quit IRC19:33
openstackgerritKevin Fox proposed openstack/barbican-specs: Spec for vm-integration  https://review.openstack.org/15957119:35
*** kfox1111 has quit IRC19:36
*** kfox1111 has joined #openstack-barbican19:36
kfox1111arg... its anoying I have to switch networks to commit at work. :/19:37
jvrbanackfox1111, if you noticed, I WIP'ed your CR. I want to make sure people give their feedback and review your spec first, so you're not making tons of random changes on something that can't be merged yet. Might as well, get people looking at your spec instead. :)19:39
kfox1111thats fine.19:39
kfox1111I usually figure, that the bikeshedding of the code takes... a while.... so I usually do it in parallel just to speed it up. ;)19:40
kfox1111but have gotten good feedback on this one. :)19:40
jvrbanac:)19:41
*** lisaclark1 has joined #openstack-barbican19:42
woodster_arunkant, I noticed your IRC above there19:45
*** lisaclark1 has quit IRC19:46
arunkantwoodster_, Pinged you to understand the concern in Line 167 of cadf audit spec: https://review.openstack.org/#/c/159938/2/specs/kilo/audit-cadf-events.rst,cm19:48
reaperhulklook, a meeting invite from redrobot19:48
woodster_arunkant, I was concerned that we would have to wait for CRs to pyCADF before we could implement events in Barbican.19:48
arunkantwoodster_, I have uploaded new patch to address the others comments19:48
reaperhulknow we know he opened his calendar19:48
*** kgriffs|afk is now known as kgriffs19:50
arunkantwoodster, Yes...Its a simple change in pycadf..already reached out to stevemar (steve from IBM) . Can work on it quickly..once there is agreement on barbican resource type names19:51
*** zz_dimtruck is now known as dimtruck19:51
arunkantwoodster, Also highlighted the change in patch 3 ..what needs to be changed.19:52
arunkantwoodster_, ^^^19:52
redrobotreaperhulk dafuq?  I swear I haven't touched my calendar today...19:55
*** crc32 has quit IRC19:56
reaperhulkCalendar knows what you want though http://cl.ly/a1Cx19:57
rm_worklol19:57
*** kfarr has joined #openstack-barbican20:00
*** rellerreller has quit IRC20:01
woodster_arunkant, so if we need to add events later (I'm thinking of the certificate workflow events for example) that should be fairly simple to add to the pyCADF side then? If so, we could get rid of our certificate event plugin stuff (https://github.com/openstack/barbican/blob/master/barbican/plugin/interface/certificate_manager.py#L171)20:01
redrobotweekly meeting is starting now in #openstack-meeting-alt20:01
kfox1111oh really?20:04
kfox1111the timifier thingy told me 1:00 last week. :/20:04
arunkantwoodster_, Is not purpose (and may be data sent) of certificate event plugin is different than audit? Are you thinking of using CADF event structure for them?20:05
woodster_arunkant, we certainly could, but that's what I'm curious about20:05
arunkantwoodster_, I am not too aware what is the usage of certificate event plugin..but it seems limited to certificate aspect only.20:08
*** rm_you has quit IRC20:08
woodster_arunkant, the main purpose is to surface cert events that can be used to drive customer ticketing/tracking processes around those20:09
woodster_arunkant, so it seems pyCADF could be used for that same purpose20:13
*** lisaclark1 has joined #openstack-barbican20:13
woodster_arunkant, since it sends events to queues. We can just have a specialized consumer dealing with tracking20:14
arunkantwoodster_, yes. pyCADF allows extension..so whatever additional is needed..can be captured as part of audit event20:14
openstackgerritThomas Dinkjian proposed openstack/python-barbicanclient: Fixes launchpad bug 1420868  https://review.openstack.org/16049020:34
openstackLaunchpad bug 1420868 in python-barbicanclient "functional tests - remove datadriven tests from smoke tests" [Medium,Confirmed] https://launchpad.net/bugs/1420868 - Assigned to Thomas Dinkjian (thomas-dinkjian)20:34
openstackgerritThomas Dinkjian proposed openstack/python-barbicanclient: Closes-Bug: #1420868  https://review.openstack.org/16049020:39
openstackbug 1420868 in python-barbicanclient "functional tests - remove datadriven tests from smoke tests" [Medium,Confirmed] https://launchpad.net/bugs/1420868 - Assigned to Thomas Dinkjian (thomas-dinkjian)20:39
*** igueths has quit IRC20:43
*** igueths has joined #openstack-barbican20:46
*** lisaclark1 has quit IRC20:48
*** kfarr has quit IRC21:02
*** lisaclark1 has joined #openstack-barbican21:07
*** shakamunyi has quit IRC21:07
*** shakamunyi has joined #openstack-barbican21:08
*** rm_work is now known as rm_work|away21:14
*** rm_work|away is now known as rm_work21:23
*** chlong has joined #openstack-barbican21:24
*** igueths has quit IRC21:33
*** igueths has joined #openstack-barbican21:41
*** dave-mccowan has quit IRC21:47
*** chlong has quit IRC21:51
*** shakamunyi_ has joined #openstack-barbican21:53
*** shakamunyi has quit IRC21:54
woodster_jvrbanac, I'm back now21:58
*** nkinder has quit IRC22:01
*** lisaclark1 has quit IRC22:01
*** jaosorior has quit IRC22:02
*** dave-mccowan has joined #openstack-barbican22:04
jvrbanacwoodster_, sooo I figured out the problem with the workers processing orders22:06
jvrbanacwoodster_, we have a race condition in the order saving process.22:06
woodster_jvrbanac between worker and api nodes, or within the worker itself?22:07
jvrbanacEssentially, we're relying on the transaction hook to do the db commit, but we push to the queue prior to that22:07
jvrbanacwoodster_, The problem is actually with the API node22:07
woodster_jvrbanac, oh that makes sense! So the api trans is not completed before the worker picks it up then?22:08
jvrbanacwoodster_, yeah22:08
jvrbanacwoodster_, I was wondering about the history around the transaction hook. Do you remember why we started using it?22:08
woodster_jvrbanac, well we weren't really managing the sqlalchemy transactions before, and noticed that (or ryanpetrello mentioned) that Pecan had transaction lifecycle support keyed to the requests.22:10
woodster_jvrbanac, I believe that is the correct way to handle transactions but our async behaviors are not synced to that :\22:11
*** igueths has quit IRC22:11
woodster_jvrbanac, so it seems we need to defer sending to the queue until the transaction commits...I wonder if there is a way to register a transaction call back with sqlalchemy to make that happen at the right time?22:12
jvrbanacwoodster_, is that the right answer though? That feels like we're working around an issue rather than building the right solution22:13
woodster_jvrbanac, well I think it is clean for the container (Pecan) to manage our transactions, rather than the app/controllers to have to manage that themselves22:14
woodster_jvrbanac, it would be good to know how this is solved in other projects for sure22:17
jvrbanacwoodster_, Yeah. Hmmm... I don't know. we're having to manually commit and rollback anyhow in quite a few cases. It feels like it would be cleaner to specify db behavior (which is kind of what the repositories are) and manage the transactions from there22:19
woodster_jvrbanac, the issue is that repositories are at a lower level of abstraction...almost one to one to models. The transactions need to span the controller/orchestration calls though. The @transaction decorator could work at this level, but there are some cases where one controller calls another controller's methods so then transaction nesting semantics become22:22
woodster_an issue.22:22
woodster_jvrbanac, not a show stopper, but just a bit more to manage22:22
woodster_jvrbanac, so a call in the middle of orchestration that enqueues something, but under the hood defer that until after the db transaction commits, would keep the current orchestration in the controllers programmatically straightforward22:24
jvrbanacwoodster_, am I the only person that gets the Heebie-jeebies when code actually doesn't execute when you tell it to?22:26
woodster_jvrbanac, none of those repository/sqlalchemy calls actually does anything until you commit() ;)22:27
jvrbanacwoodster_, that's a little different as you're building state to make a single call. However, I'm not a big fan of that either.22:28
*** SheenaG1 has quit IRC22:31
jvrbanacwoodster_, however, that atleast has a very explicit beginning and end. We're talking about deferring something so that it executes at a different time than specified in the code.22:34
jvrbanacwoodster_, unless i'm misunderstanding22:35
woodster_jvrbanac, the code run in the controllers is not actually affecting the database until Pecan's commit phase runs, after that code is executed. So it is still deferred steps in the orchestration. I would grant you that the queue step is not really sequential to the other steps in the controller though22:36
woodster_jvrbanac, if the alternative is try/except blocks though, or sub-methods with trans decorators called from other methods, I'd say you are not helping the cause :)22:40
jvrbanacSooo I'm thinking this whole thing is a little misunderstood. I've been in this section of the code a bit over the past few days and I'm still trying to figure out what this stuff is actually doing and when. Perhaps it's just my ignorance, but I think there is some value in taking a look at making it more explicit.22:47
woodster_jvrbanac, this all might be moot though if there isn't a clean way to defer the queuing until after the transaction commit. This looks ugly: http://docs.sqlalchemy.org/en/latest/orm/events.html#sqlalchemy.orm.events.SessionEvents.after_transaction_end22:48
*** xaeth is now known as xaeth_afk22:50
* kfox1111 doesn't like meeting day. :/22:50
jvrbanacwoodster_, I don't think we could use that anyhow as we don't actually have the session context at that the point we branch to the queue22:51
jvrbanacwoodster_,  https://github.com/openstack/barbican/blob/master/barbican/api/controllers/orders.py#L19922:52
woodster_jvrbanac, well, it is available via the get_session() call (it will return the thread's session). So we *could* (not necessarily want to) put that line #199 code in a call back function, that is registered with the session's transaction end event.22:54
* jvrbanac shivers22:55
woodster_jvrbanac, another option for line #199 to defer the actual queue call, and then have the Pecan lifecycle commit call (https://github.com/openstack/barbican/blob/master/barbican/model/repositories.py#L129) make a call to self.queue.flush() or some such to actually enqueue the RPC call22:56
woodster_jvrbanac so the current controller code would not change, bug the self.queue.process_xxxx() implementations would22:57
jvrbanacwoodster_, hmmm... I think I just need to sit down and map out some of these workflows and think about it some.22:57
woodster_jvrbanac, yeah. The functional sequence I think we need though is https://github.com/openstack/barbican/blob/master/barbican/model/repositories.py#L134 followed by enqueuing any RPC calls needed22:58
woodster_jvrbanac, how did you notice this though, once you pushed a lot of data through? Is that why you were seeing those rollbacks in newrelic?23:00
jvrbanacwoodster_, it's quite easy to replicate right now in a full Barbican setup. I noticed it when I had added more logging with the newrelic CR23:01
jvrbanacwoodster_, I'll add a patch to fix the issue for now, but I want to sit down and think about this a bit. Try to find a "less dirty" feeling solution lol23:02
jvrbanacs/Try/I want to try/23:02
woodster_jvrbanac, so are you fixing by making these calls explicit (or at least the order POST one)?23:03
woodster_jvrbanac, or just making a commit() call before the queue call?23:03
jvrbanacwoodster_, I'm just gonna force a commit on the order save. nothing more.23:04
jvrbanacwoodster_, It'll fix the problem for now, but I would like to explore some more longer time solutions23:04
woodster_jvrbanac, that might not be horrible way to handle this...it is explicit, and if that db write fails, you wouldn't want to do the queue or response logic anyway23:05
woodster_jvrbanac, for sure23:05
*** kebray has joined #openstack-barbican23:05
jvrbanacwoodster_, might thought exactly23:05
*** igueths has joined #openstack-barbican23:05
*** nkinder has joined #openstack-barbican23:05
woodster_jvrbanac, and it doesn't require tearing up a bunch of code out there :)23:06
*** shakamunyi_ has quit IRC23:06
jvrbanacwoodster_, yeah... this would add 1 line lol23:06
*** kfox1111 has quit IRC23:10
*** kebray has quit IRC23:12
*** kebray has joined #openstack-barbican23:15
*** dimtruck is now known as zz_dimtruck23:18
elmikohey, i'm having trouble finding some information about barbican in the cloud23:20
elmikodoes it register service/endpoints in keystone?23:20
elmikoor, i should say, do they need to be registered?23:20
elmikothink i might have found it...23:22
*** kebray has quit IRC23:23
*** rm_work is now known as rm_work|away23:24
iguethselmiko: Did you find what you were looking for?23:29
elmikoigueths: yea, was trying to figure out the service name. it eluded me, but only for a short while =)23:31
*** jorge_munoz has quit IRC23:32
iguethselmiko: Ah...Cool that you figured it out.23:32
elmikoigueths: might be something to add for the installation docs23:33
*** chlong has joined #openstack-barbican23:33
elmikoor maybe i just missed it in there23:34
*** openstackgerrit has quit IRC23:38
*** rm_work|away is now known as rm_work23:38
*** openstackgerrit has joined #openstack-barbican23:39
*** paul_glass has quit IRC23:40
*** kgriffs is now known as kgriffs|afk23:52

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!