| *** tmckay_bbl has quit IRC | 01:03 | |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Added ability to create security group automatically https://review.openstack.org/109394 | 01:32 |
|---|---|---|
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Updated docs with security group management feature https://review.openstack.org/110517 | 01:32 |
| openstackgerrit | Andrew Lazarev proposed a change to openstack/sahara: Implemented get_open_ports method for vanilla hadoop2 https://review.openstack.org/110518 | 01:36 |
| *** Networkn3rd has joined #openstack-sahara | 01:46 | |
| *** xianghui has joined #openstack-sahara | 01:49 | |
| *** bradd1 has quit IRC | 02:10 | |
| *** bradd1 has joined #openstack-sahara | 02:10 | |
| *** Networkn3rd has quit IRC | 03:58 | |
| *** Longgeek has joined #openstack-sahara | 04:16 | |
| *** Longgeek has quit IRC | 04:18 | |
| *** Longgeek has joined #openstack-sahara | 04:18 | |
| *** Longgeek has quit IRC | 04:19 | |
| *** Longgeek has joined #openstack-sahara | 04:19 | |
| *** ViswaV has joined #openstack-sahara | 05:20 | |
| *** ViswaV has quit IRC | 05:32 | |
| *** IvanBerezovskiy has joined #openstack-sahara | 05:34 | |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/110550 | 06:10 |
| *** tnovacik has joined #openstack-sahara | 07:45 | |
| *** witlessb has joined #openstack-sahara | 07:59 | |
| *** skolekonov has joined #openstack-sahara | 08:02 | |
| *** ylobankov has joined #openstack-sahara | 08:14 | |
| *** akuznetsov has quit IRC | 08:18 | |
| *** akuznetsov has joined #openstack-sahara | 08:19 | |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Fix wrong cluster postfix name for 'heat-transient' job https://review.openstack.org/110567 | 08:25 |
| openstackgerrit | A change was merged to stackforge/sahara-ci-config: Fix wrong cluster postfix name for 'heat-transient' job https://review.openstack.org/110567 | 08:29 |
| *** akuznetsov has quit IRC | 08:31 | |
| *** Timotey has joined #openstack-sahara | 08:34 | |
| *** akuznetsov has joined #openstack-sahara | 08:36 | |
| openstackgerrit | A change was merged to openstack/sahara-specs: [EDP] Add an engine for a Spark standalone deployment https://review.openstack.org/109403 | 08:42 |
| *** DenisEgorenko has joined #openstack-sahara | 08:51 | |
| *** DenisEgorenko has left #openstack-sahara | 08:57 | |
| *** DenisEgorenko has joined #openstack-sahara | 08:57 | |
| openstackgerrit | Yuji Yamada proposed a change to openstack/sahara-specs: [EDP] Improve Java type compatibility https://review.openstack.org/110576 | 09:07 |
| *** openstackgerrit has quit IRC | 09:16 | |
| *** openstackgerrit has joined #openstack-sahara | 09:17 | |
| openstackgerrit | A change was merged to openstack/sahara-image-elements: Fixes Ambari repo used by HDP 2 image generation script https://review.openstack.org/110320 | 09:23 |
| *** tosky has joined #openstack-sahara | 09:26 | |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Setting job 'diskimage-integration-hdp2' as voting https://review.openstack.org/110584 | 09:28 |
| *** Longgeek_ has joined #openstack-sahara | 09:31 | |
| *** Longgeek has quit IRC | 09:34 | |
| *** _elmiko_ has joined #openstack-sahara | 09:44 | |
| *** akuznetsov has quit IRC | 09:44 | |
| *** bradd1 has quit IRC | 09:44 | |
| *** _elmiko has quit IRC | 09:44 | |
| *** bradd1 has joined #openstack-sahara | 09:44 | |
| *** akuznetsov has joined #openstack-sahara | 09:45 | |
| openstackgerrit | Vitaly Gridnev proposed a change to openstack/sahara: Add notifications to Sahara https://review.openstack.org/108982 | 10:04 |
| openstackgerrit | A change was merged to stackforge/sahara-ci-config: Setting job 'diskimage-integration-hdp2' as voting https://review.openstack.org/110584 | 10:09 |
| *** tnovacik has quit IRC | 10:40 | |
| *** tnovacik has joined #openstack-sahara | 10:45 | |
| *** witlessb has quit IRC | 10:47 | |
| openstackgerrit | Ivan Berezovskiy proposed a change to openstack/sahara-image-elements: Mark some element scripts as executable https://review.openstack.org/110603 | 10:49 |
| openstackgerrit | A change was merged to openstack/sahara: Waiting deleting Heat stack https://review.openstack.org/109585 | 10:52 |
| openstackgerrit | A change was merged to openstack/sahara: Add rm from docs env to whitelist to avoid warn https://review.openstack.org/110264 | 10:52 |
| *** witlessb has joined #openstack-sahara | 10:58 | |
| openstackgerrit | A change was merged to openstack/sahara: Imported Translations from Transifex https://review.openstack.org/110550 | 11:20 |
| *** witlessb has quit IRC | 11:23 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed a change to openstack/sahara-image-elements: Adding Storm and Zookeeper elements https://review.openstack.org/110304 | 11:28 |
| *** witlessb has joined #openstack-sahara | 11:58 | |
| *** xianghui has quit IRC | 12:01 | |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed a change to openstack/sahara-image-elements: Adding Storm and Zookeeper elements https://review.openstack.org/110304 | 12:11 |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Fix variable's name for skipping tests https://review.openstack.org/110627 | 12:23 |
| *** tmckay has joined #openstack-sahara | 12:27 | |
| *** tmckay has joined #openstack-sahara | 12:39 | |
| openstackgerrit | Vitaly Gridnev proposed a change to openstack/sahara: Add notifications to Sahara https://review.openstack.org/108982 | 12:40 |
| *** _elmiko_ is now known as elmiko | 13:05 | |
| *** bradd1 has quit IRC | 13:09 | |
| *** bradd1 has joined #openstack-sahara | 13:09 | |
| *** IvanBerezovskiy has left #openstack-sahara | 13:11 | |
| *** Networkn3rd has joined #openstack-sahara | 13:25 | |
| openstackgerrit | Michael McCune proposed a change to openstack/sahara: Adding generic trust creation and destruction methods https://review.openstack.org/110134 | 13:29 |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Add test for storing data in DB for 007 migration https://review.openstack.org/110645 | 13:31 |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Setting jobs with Vanilla 2.4 and Transient-Heat as voting https://review.openstack.org/110650 | 13:47 |
| *** bradd11 has joined #openstack-sahara | 13:48 | |
| tellesnobrega | elmiko: hi, about the question you posted on the storm image, what is the best approach you suggest | 13:51 |
| tellesnobrega | i'm thinking either remove the check and set a default to always have or exit 1 when not given | 13:52 |
| *** bradd1 has quit IRC | 13:52 | |
| *** _crobertsrh is now known as crobertsrh | 13:53 | |
| elmiko | tellesnobrega: i think the case statement looked ok, just the test might not be needed and then setting the download url again overwrites the case statement | 13:53 |
| tellesnobrega | the second part i will remove | 13:54 |
| elmiko | tellesnobrega: since the case statement had a default case that exitted, that should be fine | 13:54 |
| tellesnobrega | alright | 13:54 |
| elmiko | cool, thanks | 13:54 |
| openstackgerrit | Telles Mota Vidal Nóbrega proposed a change to openstack/sahara-image-elements: Adding Storm and Zookeeper elements https://review.openstack.org/110304 | 14:03 |
| openstackgerrit | A change was merged to openstack/sahara: Add translation support to service and utils modules https://review.openstack.org/107089 | 14:05 |
| openstackgerrit | OpenStack Proposal Bot proposed a change to openstack/sahara: Updated from global requirements https://review.openstack.org/110108 | 14:12 |
| *** bradd11 has quit IRC | 14:34 | |
| *** skolekonov has quit IRC | 15:11 | |
| openstackgerrit | A change was merged to stackforge/sahara-ci-config: Setting jobs with Vanilla 2.4 and Transient-Heat as voting https://review.openstack.org/110650 | 15:14 |
| openstackgerrit | A change was merged to stackforge/sahara-ci-config: Fix variable's name for skipping tests https://review.openstack.org/110627 | 15:14 |
| *** sballe_ has joined #openstack-sahara | 15:19 | |
| *** sballe has joined #openstack-sahara | 15:20 | |
| *** bradd1 has joined #openstack-sahara | 15:24 | |
| *** Networkn3rd has quit IRC | 15:26 | |
| *** Networkn3rd has joined #openstack-sahara | 15:28 | |
| *** sballe has quit IRC | 15:29 | |
| openstackgerrit | Sergey Reshetnyak proposed a change to openstack/sahara: Fix closing HTTP session in Ambari plugin https://review.openstack.org/110688 | 15:41 |
| *** witlessb has quit IRC | 15:50 | |
| *** shakamunyi has joined #openstack-sahara | 15:53 | |
| *** shakamunyi has quit IRC | 15:53 | |
| *** shakamunyi has joined #openstack-sahara | 15:54 | |
| *** shakamunyi has quit IRC | 15:54 | |
| openstackgerrit | Denis Egorenko proposed a change to stackforge/sahara-ci-config: Fixing missed symbol https://review.openstack.org/110702 | 15:58 |
| *** Networkn3rd has quit IRC | 16:03 | |
| openstackgerrit | A change was merged to stackforge/sahara-ci-config: Fixing missed symbol https://review.openstack.org/110702 | 16:05 |
| *** tnovacik has quit IRC | 16:19 | |
| elmiko | tmckay, crobertsrh, is there an object or common store i can use in Sahara to keep runtime information for a periodic task to check? | 16:21 |
| elmiko | basically, i want to keep track of token expiration times but i don't want to store them in the database | 16:21 |
| tmckay | elmiko, hmm, not sure. Some of the periodic tasks just use the db and remote calls | 16:22 |
| elmiko | tmckay: do you think it's safe to make class variables in the SaharaPeriodicTasks class? | 16:23 |
| * tmckay takes a look | 16:24 | |
| tmckay | elmiko, still looking, I'm guessing it's okay. It may be better to create a store in another module as a singleton pattern, then have the task body use that module | 16:32 |
| elmiko | tmckay: makes sense, i'll need to chew on this for a few | 16:33 |
| tmckay | question is what "task(self, context)" does on line 198 of periodic_task.py | 16:33 |
| tmckay | is that a constructor, or a call? | 16:33 |
| * tmckay still looking | 16:33 | |
| tmckay | interesting question | 16:33 |
| elmiko | that's a call | 16:34 |
| elmiko | to the periodic task decorated function | 16:34 |
| tmckay | elmiko, hmm, self._periodic_tasks is a dictionary | 16:34 |
| tmckay | you could propose a general pattern where self._periodic tasks has a slightly different structure and stores task data that is passed in on each invocation | 16:35 |
| tmckay | task local persistent storage, basically | 16:35 |
| tmckay | could be generally applicable across Sahara | 16:35 |
| elmiko | i'll have to think about that one | 16:35 |
| tmckay | ooo, they're tuples | 16:36 |
| elmiko | i don't want to get into something too complicated, it might be easier to just use a singleton object somewhere | 16:36 |
| tmckay | not a dictionary | 16:36 |
| tmckay | may be. Not sure this is too hard. | 16:37 |
| tmckay | the proof is in trying :) | 16:38 |
| elmiko | lol totally | 16:38 |
| tmckay | elmiko, hmm, I think this could be mocked up really quick. You want me to try? tuple containing a reference to a dictionary still allows the dictionary itself to be modified | 16:40 |
| tmckay | the dict could be passed in if the signature for tasks included *args | 16:41 |
| tmckay | inited to empty | 16:41 |
| tmckay | your task could use it, your changes to the dict I believe would persist | 16:41 |
| elmiko | so, this would be an extra parameter passed into the periodic_task decorated method? | 16:42 |
| tmckay | yep, interpreter backs me up | 16:42 |
| tmckay | yeah, the loop in periodic_task.py would make the call like this | 16:42 |
| tmckay | for task_name, task, task_data in self._periodic_tasks: | 16:43 |
| tmckay | ... | 16:43 |
| tmckay | try: | 16:43 |
| tmckay | task(self, context, task_data) | 16:43 |
| tmckay | except Exception as e: | 16:43 |
| tmckay | I think it would work | 16:43 |
| elmiko | where would task_data get injected then, the decorator? | 16:44 |
| tmckay | by definition, periodic tasks should only be running one at a time, so thread safety is okay | 16:44 |
| tmckay | let me find one ... | 16:44 |
| *** Networkn3rd has joined #openstack-sahara | 16:45 | |
| tmckay | def update_job_statuses(self, ctx) just becomes update_job_statuses(self, ctx, *args) I think | 16:45 |
| tmckay | if it wants to ignore the data | 16:45 |
| elmiko | it might be less disruptive if i use an external object | 16:46 |
| tmckay | line 142, periodic_task.py just creates the empty dict | 16:46 |
| tmckay | cls._periodic_tasks.append((name, task, {})) | 16:47 |
| tmckay | could be. I think the disruption is small :) | 16:47 |
| tmckay | Just an option | 16:47 |
| tmckay | external object is fine too | 16:48 |
| tmckay | elmiko, oh, periodic task is openstack common =-O | 16:49 |
| elmiko | yea | 16:49 |
| tmckay | so we'd need a subtype. Yeah, probably don't want to fight that battle. But, I think it's a cool idea | 16:49 |
| tmckay | carry on :) | 16:50 |
| elmiko | it is a good idea, but i think it's a bigger issue than i want to tackle with this change as ell | 16:50 |
| elmiko | *well | 16:50 |
| tmckay | gotcha | 16:50 |
| elmiko | i'll run a few tests and see if i can store a class local variable too | 16:51 |
| crobertsrh | sorry, was eating lunch. Looks like you might be on it though :) | 16:56 |
| elmiko | no worries, just fishing for some advice :) | 16:56 |
| *** ViswaV has joined #openstack-sahara | 17:02 | |
| *** ViswaV has quit IRC | 17:16 | |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara-specs: [EDP] Add a Spark job type (instead of overloading Java) https://review.openstack.org/110727 | 17:31 |
| tmckay | crobertsrh, I mentioned you by name in here ^^ You better read it :) | 17:31 |
| crobertsrh | Oh my! I'm like interweb-famous now!! | 17:31 |
| tmckay | :) I lumped the UI stuff in with the sahara-api changes | 17:32 |
| crobertsrh | heh...actually, maybe not *that* famous....a search for me in google images doesn't seem to yield anything close to a picture of me | 17:32 |
| tmckay | I bet we could change that, lol | 17:33 |
| crobertsrh | By the looks of things, I might as well grow an interesting beard...then, plausibly, there would be lots of pics of me. | 17:34 |
| tmckay | When was it that all the baseball teams had beards? The 20s? Earlier? | 17:35 |
| crobertsrh | Not sure I ever knew that bit of trivia | 17:35 |
| tmckay | I've seen a bunch of old black and whites of bearded baseball players | 17:38 |
| tmckay | maybe it was a dream | 17:43 |
| *** bradd1 has quit IRC | 17:44 | |
| *** Longgeek has joined #openstack-sahara | 17:48 | |
| *** Longgeek_ has quit IRC | 17:51 | |
| *** Networkn3rd has quit IRC | 17:53 | |
| tmckay | crobertsrh, one question on that, should we use edp.spark.main_class or reuse edp.java.main_class? Either way is fine. | 17:57 |
| crobertsrh | I don't think it matters from the UI perspective. | 17:58 |
| crobertsrh | Either is just as easy. Is one "more correct"? | 17:58 |
| *** Networkn3rd has joined #openstack-sahara | 17:59 | |
| tmckay | well, I suppose "edp.spark.main_class" is more correct since it's the main class for a spark job, as opposed to a java job. | 17:59 |
| tmckay | but if java refers to the language, then .... | 17:59 |
| tmckay | :) it could be Java or Scala | 18:00 |
| tmckay | maybe we should just stick with java for now | 18:00 |
| crobertsrh | I'm not really offended if we go with java for now | 18:01 |
| tmckay | alright, sold | 18:01 |
| *** Longgeek has quit IRC | 18:01 | |
| tmckay | wow, looks like writing the spec might be more work than implementing this in sahara-api | 18:03 |
| crobertsrh | Probably true :) | 18:15 |
| *** tosky has quit IRC | 18:27 | |
| *** Networkn3rd has quit IRC | 18:28 | |
| *** Networkn3rd has joined #openstack-sahara | 18:32 | |
| *** Longgeek has joined #openstack-sahara | 18:47 | |
| *** akuznetsov has quit IRC | 18:47 | |
| *** Networkn3rd has quit IRC | 20:05 | |
| tmckay | crobertsrh, heh, I hacked croberts-sahara to change edp.JOB_TYPE_HIVE to edp.JOB_TYPE_SPARK on job creation to test a Spark job type submission from the dashboad | 20:36 |
| crobertsrh | Heh, nice :) | 20:36 |
| tmckay | since it allows mains, it tests the validation code. And there is just enough config available to make it work :) | 20:36 |
| crobertsrh | so, you're saying that I should already have my dashboard patch done?? :) | 20:36 |
| tmckay | found some bugs in my patch | 20:36 |
| tmckay | lol, way not user friendly | 20:37 |
| tmckay | lies, lies, lies | 20:37 |
| elmiko | do either of you know which field in JobExecution contains the status? | 20:37 |
| crobertsrh | dang....I will avoid the temptation to do a quick git diff and roll that into "my" patch :) | 20:37 |
| elmiko | lol | 20:37 |
| tmckay | elmiko, yes, | 20:37 |
| crobertsrh | I think it's under "info", right? | 20:37 |
| tmckay | should be job_exec.info["status"] | 20:38 |
| elmiko | thanks, wasn't sure which general field it might be in. and grepping for "info" was a little much lol | 20:38 |
| crobertsrh | Also...don't forget to vote on OS Summit presentations. | 20:38 |
| elmiko | i didn't see the link for that, got it handy? | 20:39 |
| crobertsrh | https://www.openstack.org/vote-paris/ | 20:39 |
| elmiko | thanks | 20:39 |
| tmckay | I see we're up for another live demo, SergeyLukjanov | 20:39 |
| tmckay | we need to plan ahead for that :) | 20:40 |
| crobertsrh | heh....just don't record it anytime soon....horizon is a mess atm...not just our dashboard...everything | 20:40 |
| tmckay | A Mac, and a network accessible from that Mac without copying vpn info | 20:40 |
| tmckay | or a really fast local box | 20:40 |
| elmiko | crobertsrh: no doubt, i ran a trunk version of horizon yesterday. it was bad... | 20:42 |
| crobertsrh | Yeah, they upgraded the bootstrap bits........boom | 20:42 |
| crobertsrh | calculated move to merge that patch and then fix everything | 20:42 |
| elmiko | makes sense | 20:42 |
| elmiko | but stuff could seriously fall through the cracks | 20:42 |
| crobertsrh | I made a patch for our stuff today.....it should catch the most ugly stuff. | 20:43 |
| crobertsrh | Yeah, I agree. I did not +1 that idea | 20:43 |
| elmiko | if i want to query an object is it preferred to use the conductor interface or the api interface? | 21:00 |
| *** crobertsrh is now known as _crobertsrh | 21:04 | |
| tmckay | elmiko, hey there. | 21:09 |
| elmiko | hey | 21:10 |
| tmckay | conductor, I'd say | 21:10 |
| tmckay | that's what the EDP stuff does | 21:10 |
| tmckay | (job_manager.py, for example_ | 21:10 |
| elmiko | i'm going through these job executions states, and i'm thinking i should create some constants like we did for the JOB_TYPE_* stuff | 21:11 |
| tmckay | may not be a bad idea | 21:12 |
| elmiko | i'm only seeing terminated states though, not running states | 21:13 |
| *** Longgeek has quit IRC | 21:13 | |
| elmiko | so, i'm gonna run some live tests to see what happens | 21:13 |
| tmckay | cool. live tests of what? | 21:14 |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Implement EDP for a Spark standalone cluster https://review.openstack.org/107871 | 21:14 |
| openstackgerrit | Trevor McKay proposed a change to openstack/sahara: Add a Spark job type for EDP https://review.openstack.org/110791 | 21:15 |
| elmiko | i have a simple periodic task running, and i'm starting to loop through job executions to determine if auth tokens need updating | 21:16 |
| elmiko | so i need to determine if a JobExecution is still running or finished | 21:17 |
| tmckay | oh, right | 21:23 |
| tmckay | elmiko, in job_utils there should be terminated_job_states | 21:23 |
| elmiko | saw that, it's what gave me the idea to make constants | 21:24 |
| tmckay | pending/running should be the only "running" ones | 21:24 |
| elmiko | lol that was my next question | 21:24 |
| tmckay | succeeded, failed, donewitherror are all complete | 21:24 |
| elmiko | do you think it's a good idea to break the JOB_EXECUTION_STATE_* into a separate review? | 21:25 |
| tmckay | oh, and killed | 21:25 |
| tmckay | yeah, I guess running and pending are not called out anywhere | 21:25 |
| tmckay | yeah, separate review. It will likely touch a bunch of places | 21:25 |
| elmiko | k | 21:25 |
| tmckay | pending means job exec created but not submitted to the job server yet (oozie, spark, whatever). most likely in run_job | 21:26 |
| elmiko | i like breaking these into a bunch of small easily reviewed pieces, but i'm trying to be sensitive to the fact that i don't want to create a hellish dependency chain | 21:26 |
| tmckay | "running" should mean the server has it | 21:26 |
| elmiko | k | 21:26 |
| tmckay | for your current patch, you could just add a "running_job_states" | 21:27 |
| elmiko | yea | 21:27 |
| elmiko | i'm thinking about just creating the constants in sahara.util.edp, then allowing the individual modules to create their own lists as needed. do you think i should create 2 master lists (running, terminated)? | 21:28 |
| *** tmckay has quit IRC | 21:29 | |
| *** elmiko is now known as _elmiko | 22:25 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!