| *** mahito has joined #openstack-sahara | 01:02 | |
| *** Longgeek has joined #openstack-sahara | 01:08 | |
| *** hdd has quit IRC | 01:09 | |
| *** Longgeek has quit IRC | 01:15 | |
| *** _mattf has joined #openstack-sahara | 01:44 | |
| *** hdd has joined #openstack-sahara | 02:13 | |
| *** hogepodge has joined #openstack-sahara | 02:43 | |
| *** coolsvap|afk is now known as coolsvap | 02:58 | |
| *** hdd has quit IRC | 02:58 | |
| *** Poornima has joined #openstack-sahara | 03:03 | |
| *** Longgeek has joined #openstack-sahara | 03:11 | |
| *** Longgeek has quit IRC | 03:13 | |
| *** Longgeek has joined #openstack-sahara | 03:14 | |
| *** Longgeek_ has joined #openstack-sahara | 03:19 | |
| *** Longgeek has quit IRC | 03:20 | |
| *** bandarji has joined #openstack-sahara | 03:30 | |
| *** bandarji has quit IRC | 03:30 | |
| *** bandarji has joined #openstack-sahara | 03:31 | |
| *** Longgeek has joined #openstack-sahara | 03:41 | |
| *** Longgeek_ has quit IRC | 03:44 | |
| *** mahito has quit IRC | 04:08 | |
| *** Longgeek has quit IRC | 04:11 | |
| *** Longgeek has joined #openstack-sahara | 04:11 | |
| *** bandarji has quit IRC | 04:13 | |
| *** Poornima has quit IRC | 04:28 | |
| *** hdd has joined #openstack-sahara | 04:52 | |
| *** Poornima has joined #openstack-sahara | 05:00 | |
| *** hdd has quit IRC | 05:29 | |
| *** Nikolay_St has quit IRC | 05:31 | |
| *** sreshetn1 has joined #openstack-sahara | 06:46 | |
| *** Longgeek_ has joined #openstack-sahara | 06:52 | |
| *** Longgeek has quit IRC | 06:52 | |
| *** tnovacik has joined #openstack-sahara | 06:53 | |
| *** mahito has joined #openstack-sahara | 06:56 | |
| *** pcaruana has quit IRC | 07:14 | |
| openstackgerrit | Artem Osadchiy proposed openstack/sahara: Add Spark support for MapR plugin https://review.openstack.org/163007 | 07:21 |
|---|---|---|
| *** Nikolay_St has joined #openstack-sahara | 07:28 | |
| *** sgotliv has quit IRC | 07:36 | |
| *** sreshetn1 has quit IRC | 07:40 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Replace empty list with scalable process in scaling https://review.openstack.org/163845 | 08:02 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Apply event-log feature for HDP plugin https://review.openstack.org/152472 | 08:12 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Apply event-log feature for HDP plugin https://review.openstack.org/152472 | 08:18 |
| *** witlessb has joined #openstack-sahara | 08:18 | |
| openstackgerrit | Merged openstack/sahara: Update the docs for CDH plugin userdoc and image-builder doc https://review.openstack.org/163670 | 08:24 |
| *** skolekonov has joined #openstack-sahara | 08:26 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add usages for step_type field https://review.openstack.org/162997 | 08:27 |
| *** mahito has quit IRC | 08:33 | |
| *** AndreyPavlov has quit IRC | 08:44 | |
| *** sreshetn1 has joined #openstack-sahara | 09:15 | |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Adding cluster, instance, job_execution ids to logs https://review.openstack.org/143964 | 09:26 |
| *** sgotliv has joined #openstack-sahara | 09:26 | |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Adding cluster, instance, job_execution ids to logs https://review.openstack.org/143964 | 09:30 |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Adding cluster, instance, job_execution ids to logs https://review.openstack.org/143964 | 09:30 |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Adding cluster, instance, job_execution ids to logs https://review.openstack.org/143964 | 09:31 |
| *** AndreyPavlov has joined #openstack-sahara | 09:31 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Move updating provision progress to conductor https://review.openstack.org/159421 | 09:41 |
| *** IBerezovskiy has joined #openstack-sahara | 09:41 | |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Fixing log messages to avoid information duplication https://review.openstack.org/147504 | 09:42 |
| *** sgotliv has quit IRC | 10:14 | |
| *** sgotliv has joined #openstack-sahara | 10:14 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Event log supported in new integration tests https://review.openstack.org/156956 | 10:15 |
| *** coolsvap is now known as coolsvap|afk | 10:19 | |
| *** sgotliv has quit IRC | 10:25 | |
| *** tosky has joined #openstack-sahara | 10:31 | |
| *** tnovacik_ has joined #openstack-sahara | 10:47 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Add scenario files for new integration tests https://review.openstack.org/164644 | 10:47 |
| *** tnovacik has quit IRC | 10:48 | |
| *** tosky has quit IRC | 11:12 | |
| *** tosky has joined #openstack-sahara | 11:12 | |
| openstackgerrit | Merged openstack/sahara: Add support for MapR v4.0.2 https://review.openstack.org/160375 | 11:15 |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Add scenario files for new integration tests https://review.openstack.org/164644 | 11:19 |
| *** tnovacik_ has quit IRC | 11:23 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Event log supported in new integration tests https://review.openstack.org/156956 | 11:23 |
| *** sgotliv has joined #openstack-sahara | 11:31 | |
| SergeyLukjanov | tmckay, elmiko, https://wiki.openstack.org/wiki/Sahara/Kilo/FPF_Exceptions for doc | 11:31 |
| *** Poornima has quit IRC | 11:36 | |
| openstackgerrit | Merged openstack/sahara: Add an is_default field to cluster templates and node group templates https://review.openstack.org/160920 | 12:03 |
| openstackgerrit | Merged openstack/sahara: Changed wrong value for total during step creation https://review.openstack.org/159758 | 12:03 |
| *** sreshetn1 has quit IRC | 12:11 | |
| *** tnovacik_ has joined #openstack-sahara | 12:12 | |
| *** sreshetn1 has joined #openstack-sahara | 12:25 | |
| *** macjack has joined #openstack-sahara | 12:31 | |
| openstackgerrit | Merged openstack/sahara: Rewrite log levels and messages https://review.openstack.org/154037 | 12:33 |
| openstackgerrit | Merged openstack/sahara: Changing method for verifying existence of cinder https://review.openstack.org/164274 | 12:33 |
| *** macjack has quit IRC | 12:35 | |
| openstackgerrit | Merged openstack/sahara-specs: [EDP] Add a new job-types endpoint https://review.openstack.org/157563 | 12:38 |
| openstackgerrit | Artem Osadchiy proposed openstack/sahara: Add Spark support for MapR plugin https://review.openstack.org/163007 | 12:49 |
| *** openstackgerrit has quit IRC | 12:50 | |
| *** openstackgerrit has joined #openstack-sahara | 12:50 | |
| openstackgerrit | Merged openstack/sahara-extra: Fix copy operation for large object https://review.openstack.org/162036 | 12:52 |
| openstackgerrit | Merged openstack/sahara-extra: Fix pseudo-directory renaming operation https://review.openstack.org/162101 | 12:54 |
| openstackgerrit | Merged openstack/sahara-extra: Fix segment object naming that uploaded at qualified URL https://review.openstack.org/162005 | 12:54 |
| openstackgerrit | Adrien Vergé proposed openstack/sahara: HDP plugin: Fix Beeswax error when starting Hue https://review.openstack.org/162668 | 12:54 |
| openstackgerrit | Merged openstack/sahara-extra: Fix to use renewal keystone token when re-authenticated https://review.openstack.org/161666 | 12:54 |
| openstackgerrit | Sergey Lukjanov proposed openstack/sahara-specs: Defere "Add Spark Shell Action job type" to Liberty We're already meet Feature Proposal Freeze https://review.openstack.org/164680 | 13:00 |
| openstackgerrit | Sergey Lukjanov proposed openstack/sahara-specs: Defere "Add Spark Shell Action job type" to Liberty https://review.openstack.org/164680 | 13:01 |
| elmiko | SergeyLukjanov: ack, thanks | 13:01 |
| SergeyLukjanov | tmckay, egafford, sgotliv I've deferred Add Spark Shell Action job type to Liberty release because of FPF two weeks ago and FF this week | 13:02 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Implement poll util and plugin poll util https://review.openstack.org/157392 | 13:03 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add usages of poll util for service modules https://review.openstack.org/163463 | 13:03 |
| *** _crobertsrh is now known as crobertsrh | 13:06 | |
| openstackgerrit | Andrey Pavlov proposed openstack/sahara: Docs updated with instance locality feature https://review.openstack.org/164683 | 13:08 |
| *** tmckay has joined #openstack-sahara | 13:11 | |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Add scenario files for new integration tests https://review.openstack.org/164644 | 13:12 |
| openstackgerrit | Merged openstack/sahara-specs: Defere "Add Spark Shell Action job type" to Liberty https://review.openstack.org/164680 | 13:13 |
| openstackgerrit | Sergey Lukjanov proposed openstack/sahara: Avoid additional runtime requirements by default https://review.openstack.org/164684 | 13:14 |
| openstackgerrit | Artem Osadchiy proposed openstack/sahara: Oozie doesn't start properly https://review.openstack.org/164685 | 13:14 |
| SergeyLukjanov | elmiko, I've proposed CR to remove python-barbicanclient>=3.0.1 from requirements to avoid additional runtime requirements | 13:14 |
| SergeyLukjanov | elmiko, https://review.openstack.org/164684 | 13:15 |
| SergeyLukjanov | elmiko, what do you think about it? | 13:15 |
| elmiko | SergeyLukjanov: it will make the sahara.utils.keymgr stuff break, and we will need to turn off the keymgr tests i think. | 13:17 |
| elmiko | SergeyLukjanov: so, maybe move it to test-requirements.txt? | 13:18 |
| SergeyLukjanov | elmiko, yeah, that's a good option, the main reason I've proposed this patch is to discuss it | 13:18 |
| elmiko | SergeyLukjanov: also, i am working on a patch for the swift passwords now. by default the external manager is turned off, but all the passwords are still gated through the keymgr module. | 13:19 |
| elmiko | so, that might have an effect as well | 13:19 |
| openstackgerrit | Sergey Lukjanov proposed openstack/sahara: [DISCUSSION, DO NOT MERGE] Avoid additional runtime requirements by default https://review.openstack.org/164684 | 13:19 |
| elmiko | SergeyLukjanov: i'll add my comments to the review | 13:19 |
| SergeyLukjanov | elmiko, I think that probably keymgr module should be improved first | 13:20 |
| SergeyLukjanov | elmiko, there was a very good comment on the initial change | 13:20 |
| SergeyLukjanov | elmiko, about class and pluggability | 13:20 |
| elmiko | SergeyLukjanov: about the cinder/nova keymgr stuff? | 13:20 |
| SergeyLukjanov | elmiko, yeah | 13:21 |
| SergeyLukjanov | elmiko, have you seen their impl? | 13:21 |
| elmiko | SergeyLukjanov: yes | 13:21 |
| elmiko | SergeyLukjanov: i responded to those comments, i think we will want to use the Castellan keymgr when it is released. | 13:21 |
| elmiko | SergeyLukjanov: but i still think we can use the thin wrapper to help gate calls through the keymgr. | 13:22 |
| elmiko | SergeyLukjanov: my thinking is this, we use the keymgr calls regardless of whether we are using external or not. | 13:22 |
| elmiko | SergeyLukjanov: then the keymgr can control if the keys are moved off-board or just returned to be stored in the db | 13:22 |
| SergeyLukjanov | elmiko, Castellan? | 13:23 |
| elmiko | SergeyLukjanov: yea, Castellan is a keymgr library that can use multiple backends. | 13:24 |
| elmiko | SergeyLukjanov: it's being developed based off the cinder/nova keymgr code | 13:24 |
| SergeyLukjanov | elmiko, could you please add link to it? | 13:24 |
| SergeyLukjanov | elmiko, oh, got it | 13:24 |
| elmiko | SergeyLukjanov: https://review.openstack.org/#/c/148742/ | 13:24 |
| elmiko | that's the big commit that just happened | 13:24 |
| SergeyLukjanov | elmiko, my main concern is that in Kilo prod installations barbican will not used | 13:25 |
| elmiko | SergeyLukjanov: agreed. we can probably remove it from the requirements.txt, but might need to keep for test-requirements.txt | 13:25 |
| SergeyLukjanov | elmiko, and it means that packaging its client is absolutely useless but if it'll be not installed than sahara will not work | 13:25 |
| elmiko | SergeyLukjanov: agreed | 13:25 |
| SergeyLukjanov | elmiko, so, we need to wrap it in a way to support situation when client isn't installed | 13:26 |
| SergeyLukjanov | elmiko, I could wrap imports to impl it | 13:26 |
| elmiko | SergeyLukjanov: that could work | 13:26 |
| elmiko | SergeyLukjanov: i'll open a new bug to address creating a better separation | 13:27 |
| SergeyLukjanov | elmiko, ack, thx | 13:27 |
| SergeyLukjanov | elmiko, re WADL API doc - we need to have it in 2 weeks | 13:27 |
| elmiko | SergeyLukjanov: i think that's doable | 13:27 |
| elmiko | SergeyLukjanov: i started reviewing what we need over the weekend | 13:28 |
| tmckay | elmiko, crobertsrh, fyi, from Friday discussion, client allows delete of tempaltes by name or id, so I'm gonna tweak delete on the template CLI | 13:28 |
| elmiko | tmckay: ack | 13:28 |
| crobertsrh | ack | 13:28 |
| SergeyLukjanov | elmiko, in fact it's about converting all our API docs to the ugly XML format to then generate sexy API ref page | 13:28 |
| elmiko | SergeyLukjanov: i'm thinking maybe i should just wait till Liberty cycle before trying to implement more keymgr stuff. that makes removing barbicanclient an easy choice | 13:29 |
| openstackgerrit | Merged openstack/sahara: Switch to v2 version of novaclient https://review.openstack.org/163802 | 13:29 |
| SergeyLukjanov | elmiko, do you need help with it? I could ask someone from my team to start doing it to have first parts asap | 13:29 |
| tmckay | elmiko, crobertsrh, SergeyLukjanov, I had a thought over the weekend. I wonder if we need any kind of version in default templates, so that we know what release a default template is from, or what version of the CLI? is_default could be a string instead of a Boolean, or we could use the description field, or we could use a naming convention. No clear use case, just a nagging feeling that a version might come in handy for some reason | 13:29 |
| tmckay | of course, with alembic, we could always change the is_default field later, or add one, etc etc | 13:30 |
| SergeyLukjanov | elmiko, I'm ok with and IMO it's the best choice now because will not have direct benefits from adding experimental support to the Kilo release, prod ready impl in Liberty is extremely improtant | 13:30 |
| tmckay | related thought, we might want to organize the directory of def templates from Sahara so that we have them grouped by release .... | 13:30 |
| tmckay | so we have a clear trail of what was created when | 13:30 |
| crobertsrh | My first thought when I originally wrote the spec was to have a version of some sort, but we eventually concluded that we didn't want/need it, at least for now. | 13:31 |
| crobertsrh | I don't hate the idea of having some sort of version info though. | 13:31 |
| elmiko | SergeyLukjanov: i think much of it will be copying what we have in the v1 v1.1 docs into the WADL format. i was going to do a transformation on the swagger doc i produced for sahara to build the base of the WADL doc, but... if you have folks who are ready to go then i don't mind help =) | 13:31 |
| SergeyLukjanov | tmckay, crobertsrh speaking about versions - we need to take a look on glance artifact store | 13:31 |
| SergeyLukjanov | probably we could migrate our templates to it and re-use their versioning, search and etc. | 13:32 |
| crobertsrh | ah, yes....the glance artifact store....I had forgot about that | 13:32 |
| tmckay | SergeyLukjanov, yes, we had that in the back of our minds for Kilo, but we haven | 13:32 |
| tmckay | haven't paid much attention | 13:32 |
| SergeyLukjanov | elmiko, could you please ping me when you'll understand when you're able to start working on it? I will try to investigate it before | 13:33 |
| SergeyLukjanov | tmckay, yeah | 13:33 |
| SergeyLukjanov | tmckay, as I know it's working already somehow | 13:33 |
| SergeyLukjanov | tmckay, I've been approving semver lib in global requirements for their versioning feature ;) | 13:33 |
| elmiko | SergeyLukjanov: i was thinking about starting tuesday or wednesday this week, there is some security doc stuff i need to finish before then. | 13:34 |
| SergeyLukjanov | elmiko, ack | 13:34 |
| SergeyLukjanov | tmckay, I've asked ativelkov to join our channel to chat about glance stuff, he's driving this feature | 13:35 |
| tmckay | SergeyLukjanov, okay. Forgive me if I'm a little distracted, I still have a few things to do for default templates -- tweak delete, fix up spec, unit tests | 13:36 |
| SergeyLukjanov | tmckay, that's ok :) | 13:38 |
| openstackgerrit | Merged openstack/sahara-image-elements: Fix datanode launch performance caused by entropy pool depletion https://review.openstack.org/161988 | 13:42 |
| tosky | I've noticed that the swift bugs have been fixed: https://bugs.launchpad.net/sahara/+bug/1428939 | 13:43 |
| openstack | Launchpad bug 1428939 in Sahara "[hadoop-swiftfs] cannot read large object that uploaded by hadoop command" [High,Fix committed] - Assigned to Kazuki OIKAWA (k.oikw) | 13:43 |
| uvirtbot | Launchpad bug 1428939 in sahara "[hadoop-swiftfs] cannot read large object that uploaded by hadoop command" [High,Fix committed] | 13:43 |
| uvirtbot | Launchpad bug 1428939 in sahara "[hadoop-swiftfs] cannot read large object that uploaded by hadoop command" [High,Fix committed] https://launchpad.net/bugs/1428939 | 13:43 |
| tosky | but my comment #4 is still relevant, I think: not all images use "our" version of swiftfs hadoop library, but the upstream one | 13:43 |
| elmiko | tosky: as i understand it, the issue with pushing to upstream is that the apache hadoop folks are very slow in accepting patches | 13:48 |
| tosky | elmiko: ok, but then does it mean we are giving up pushing upstream? | 13:51 |
| tosky | I didn't see any link to an upstream ticket, even after I asked | 13:52 |
| tosky | also, if we decide to ignore upstream, which I think is bad, we should at least fix the image generation process to always use the internal library | 13:52 |
| elmiko | tosky: good question, i'm not clear on how exactly we are working with upstream on these issues | 13:52 |
| elmiko | tosky: i agree we should work with upstream, i'm just repeating what came up when we talked about working with them last year. | 13:53 |
| *** tnovacik_ has quit IRC | 13:53 | |
| elmiko | tosky: here's one of the jiras https://issues.apache.org/jira/browse/HADOOP-10948 | 13:54 |
| tosky | elmiko: sure, right now, if I understand the core correctly, HDP use the upstream version (as it does not include the swiftfs_hadoop element in s-i-e) | 13:55 |
| elmiko | tosky: hmm, interesting. i think in general we should probably always use the code from sahara-extras | 13:56 |
| tosky | elmiko: uhm, I wonder which of the 3/4 bugs closed today matches that patch | 13:56 |
| tosky | elmiko: unless I misunderstand the code from s-i-e, that's why I asked | 13:56 |
| elmiko | tosky: it's a good point, i just haven't looked into it for awhile. we certainly need to clear this up | 13:57 |
| openstackgerrit | Artem Osadchiy proposed openstack/sahara: Oozie doesn't start properly https://review.openstack.org/164685 | 13:57 |
| elmiko | tosky: it seems to me that main point of contact on this stuff between the sahara and hadoop projects is oikawa | 13:57 |
| elmiko | SergeyLukjanov: i need to adjust the description on the barbican bp, but i think we can mark it as complete at this point | 14:02 |
| *** egafford has joined #openstack-sahara | 14:03 | |
| *** egafford has quit IRC | 14:04 | |
| *** tnovacik_ has joined #openstack-sahara | 14:05 | |
| *** egafford has joined #openstack-sahara | 14:08 | |
| openstackgerrit | Sergey Reshetnyak proposed openstack/sahara-image-elements: Migrate to openjdk - part 2 https://review.openstack.org/146434 | 14:16 |
| SergeyLukjanov | elmiko, I think it'll be better to defer spec to liberty because bp isn't implemented actually | 14:16 |
| *** Nikolay_St has quit IRC | 14:17 | |
| elmiko | SergeyLukjanov: that bp was just about implementing the key manager portion | 14:17 |
| SergeyLukjanov | elmiko, oh | 14:18 |
| elmiko | SergeyLukjanov: i wanted to separate the different parts so it wasn't such a huge undertaking | 14:18 |
| elmiko | SergeyLukjanov: i think we should be creating bugs as we come across areas that can be improved by moving to key manager usage | 14:18 |
| SergeyLukjanov | elmiko, my concern is that if someone will see "Improved secret storage utilizing Barbican" implemented than it sounds like at least experimental support | 14:19 |
| SergeyLukjanov | elmiko, but we currently only have wrapper | 14:19 |
| elmiko | SergeyLukjanov: makes sense, maybe i should reword the description then slightly? | 14:20 |
| SergeyLukjanov | elmiko, IMO it should be just moved to Liberty to avoid misunderstanding | 14:20 |
| elmiko | i mean, technically the wrapper is implemented. it just isn't used | 14:20 |
| elmiko | SergeyLukjanov: ack | 14:20 |
| SergeyLukjanov | elmiko, okay, I'll do it | 14:21 |
| SergeyLukjanov | elmiko, it's only about moving spec | 14:21 |
| SergeyLukjanov | elmiko, not about removing code | 14:21 |
| elmiko | SergeyLukjanov: i was just talking about the blueprint, i think the spec is fine. the spec only defines the manager wrapper | 14:21 |
| SergeyLukjanov | elmiko, oh, ack | 14:22 |
| elmiko | SergeyLukjanov: i marked the bp as complete, but maybe that was premature | 14:22 |
| SergeyLukjanov | elmiko, so, we could adjust the bp title a bit to make it more clear | 14:22 |
| elmiko | SergeyLukjanov: yea | 14:22 |
| elmiko | SergeyLukjanov: i'll take care of it | 14:23 |
| SergeyLukjanov | elmiko, thx | 14:23 |
| elmiko | SergeyLukjanov: i adjusted the bp to change the language slighty, but i think we should talk again at the meeting this week. | 14:31 |
| SergeyLukjanov | elmiko, okay | 14:33 |
| SergeyLukjanov | elmiko, hm, we'll have a meeting most probably after the kilo-3 cut | 14:34 |
| SergeyLukjanov | elmiko, it's better to agree on it before the actual tag | 14:34 |
| elmiko | SergeyLukjanov: i just thought originally the bp and spec would address only implementing the key manager abstraction, then we would create bugs for the individual secrets that needed to use the key manager. | 14:36 |
| elmiko | SergeyLukjanov: it sounds like you'd like the bp to address the secrets as well? | 14:36 |
| SergeyLukjanov | elmiko, probably abstract pluggable keymngr | 14:40 |
| elmiko | SergeyLukjanov: agreed, i think that is a layer which can be added to what we have now | 14:41 |
| SergeyLukjanov | elmiko, I'm just a bit concerned that having this BP implemented doesn't mean any actual change to Sahara | 14:41 |
| elmiko | SergeyLukjanov: hmm, good point. by itself, it doesn't represent a change. | 14:42 |
| elmiko | SergeyLukjanov: well, the bp is changed to "Good Progress" and i'm fine with that for K-3 | 14:43 |
| elmiko | SergeyLukjanov: i'll stay on top of the changes to Castellan and the internal progress we want to make towards putting more secrets behind the manager. i don't see us adding anything more for Kilo, but it will be a nice roadmap for Liberty. | 14:44 |
| *** sreshetn1 has quit IRC | 14:46 | |
| *** stannie has quit IRC | 14:49 | |
| *** sreshetn1 has joined #openstack-sahara | 14:52 | |
| SergeyLukjanov | elmiko, okay, thx, so, are you with moving it to Liberty? | 14:53 |
| elmiko | SergeyLukjanov: +1 | 14:53 |
| *** esikachev has joined #openstack-sahara | 14:56 | |
| *** Nikolay_St has joined #openstack-sahara | 14:56 | |
| openstackgerrit | Sergey Lukjanov proposed openstack/sahara-specs: Defere "Improved secret storage utilizing external key manager" to Liberty https://review.openstack.org/164732 | 15:17 |
| *** Networkn3rd has joined #openstack-sahara | 15:20 | |
| *** Nikolay_St has quit IRC | 15:27 | |
| *** Longgeek_ has quit IRC | 15:36 | |
| *** bandarji has joined #openstack-sahara | 15:47 | |
| openstackgerrit | Yuriy Taraday proposed openstack/sahara: Add an option to use lazy formatting in new-style logging https://review.openstack.org/164755 | 15:56 |
| *** ativelkov has joined #openstack-sahara | 15:58 | |
| *** tnovacik_ has quit IRC | 15:59 | |
| openstackgerrit | Merged openstack/sahara-specs: Defere "Improved secret storage utilizing external key manager" to Liberty https://review.openstack.org/164732 | 16:00 |
| *** skolekonov has quit IRC | 16:05 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara: Add a CLI tool for managing default templates https://review.openstack.org/163649 | 16:09 |
| *** openstackgerrit has quit IRC | 16:11 | |
| *** openstackgerrit has joined #openstack-sahara | 16:11 | |
| *** IBerezovskiy has quit IRC | 16:15 | |
| openstackgerrit | Merged openstack/sahara: Added support of instance locality to engines https://review.openstack.org/162172 | 16:27 |
| openstackgerrit | Merged openstack/sahara: Add job-types endpoint https://review.openstack.org/161250 | 16:27 |
| openstackgerrit | Merged openstack/sahara: Implement job-types endpoint support methods for HDP plugin https://review.openstack.org/161263 | 16:27 |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Add scenario files for new integration tests https://review.openstack.org/164644 | 16:30 |
| openstackgerrit | Evgeny Sikachev proposed openstack/sahara: Add scenario files for new integration tests https://review.openstack.org/164644 | 16:32 |
| *** esikachev has quit IRC | 16:32 | |
| *** hdd has joined #openstack-sahara | 16:40 | |
| openstackgerrit | Ethan Gafford proposed openstack/sahara: Default templates for HDP https://review.openstack.org/164772 | 16:43 |
| *** vgridnev_ has joined #openstack-sahara | 16:44 | |
| *** tnovacik_ has joined #openstack-sahara | 16:47 | |
| *** openstackgerrit has quit IRC | 16:54 | |
| *** openstackgerrit has joined #openstack-sahara | 16:54 | |
| tmckay | elmiko, crobertsrh, egafford, re substitution in default templates. So we are adding config sections on the fly based on plugin, version, and template name and prefering more specific (so we will look for vanilla_1.2.1_worker, then vanilla_1.2.1, vanilla, and finally DEFAULT | 17:08 |
| tmckay | but our substitutions are pre-set. | 17:08 |
| tmckay | I mean, the fields we support | 17:08 |
| crobertsrh | Yeah, I think the substitution fields are pre-set. | 17:09 |
| tmckay | What if I added a config option on the fly in the same way for anything we find in {}? Like, count: {my_favorite_value} | 17:09 |
| elmiko | i dunno, i kinda like keeping it simple | 17:09 |
| tmckay | this is simple | 17:09 |
| crobertsrh | We could do that if we wanted to be really really flexible, but that seems like it could be troublesome | 17:09 |
| tmckay | you name a field, it's in the config file, you get it | 17:10 |
| tmckay | crobertsrh, how so? | 17:10 |
| crobertsrh | As long as we could give a good error message when it doesn't go as planned | 17:10 |
| tmckay | elmiko, crobertsrh, egafford mentioned "count" today. There's no way to map that simply. It might be nice to be able to configure count to change the number of workers, for instance | 17:10 |
| elmiko | it's a nice convenience but imo it starts to move away from the idea that we talked about keeping this "on the rails" kind of configuration. | 17:11 |
| tmckay | so if you want that, you copy the set and edit yourself | 17:11 |
| tmckay | if our count is not good enough, make your own | 17:11 |
| crobertsrh | How much additional work would it be to support arbitrary substitutions? | 17:11 |
| tmckay | I mean, copy the json files and edit | 17:11 |
| tmckay | well, validation is going to catch errors. The only trouble I see is that oslo config wants a type for an option | 17:13 |
| tmckay | knowing what to give it for a type might be tricky. Unless it can be lifted from the schema | 17:13 |
| crobertsrh | Maybe just go with the 2 supported options for now. If the masses demand more flexibility, maybe a Liberty thing? | 17:14 |
| tmckay | may not be worth it. I was looking for a simple way to make "count" configurable, but there is not a one-to-one mapping in a cluster template for count | 17:14 |
| tmckay | ack, I could see that | 17:14 |
| elmiko | crobertsrh: +1 | 17:14 |
| elmiko | i like making it simpler for now | 17:14 |
| tmckay | what do you guys think about rollback? It's currently attempting to undo changes for a set (a directory level defines a set) if something happens part way through | 17:15 |
| tmckay | rolling back creation is easy. Rolling back update doesn't work yet. | 17:15 |
| openstackgerrit | Ethan Gafford proposed openstack/sahara: Default templates for Vanilla https://review.openstack.org/164789 | 17:16 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Apply event-log feature for HDP plugin https://review.openstack.org/152472 | 17:16 |
| elmiko | that's a toughie | 17:16 |
| tmckay | I think I can do it without too much trouble by looking at what I'm pushing up and saving the differences. But nested structures might be weird. Also, sqlalchemy is not noticing a change of "count" currently as an update, could be a bug. | 17:17 |
| egafford | tmckay, elmiko, crobertsrh: Perfectly reasonable to not make count configurable; templates can always be copied and edited. | 17:18 |
| elmiko | imo, i thought the default templates are targeted at new users who have no idea what is going on with sahara and potentially hadoop/spark in general. these templates give them a cluster to start messing around on. | 17:19 |
| elmiko | once they get past the initial phase, then they should really create their own json, or use the ui, or whatever works for them | 17:20 |
| tmckay | yeah, maybe we just should have made some sample JSON files and called it good enough | 17:23 |
| tmckay | maybe this CLI thing is overkill | 17:24 |
| elmiko | well... the cli could grow in the future, but for now maybe keep it simple. | 17:25 |
| openstackgerrit | Trevor McKay proposed openstack/sahara-specs: Provide default templates for each plugin https://review.openstack.org/162209 | 17:28 |
| openstackgerrit | Ethan Gafford proposed openstack/sahara: Default templates for Spark https://review.openstack.org/164797 | 17:34 |
| *** IlyaE has joined #openstack-sahara | 17:39 | |
| tmckay | SergeyLukjanov, is the exception link, or is there an etherpad too? https://wiki.openstack.org/wiki/Sahara/Kilo/FPF_Exceptions | 17:58 |
| *** Nikolay_St has joined #openstack-sahara | 17:59 | |
| elmiko | what do folks think about a security session for vancouver, should i look towards making it a fish bowl session or a working session? | 18:07 |
| elmiko | topic-wise i'm thinking, where we are now, what we plan to do for L, what we could be looking at long term | 18:07 |
| elmiko | the security session was well attended in paris, but i'm not sure if this would be large enough for a fish bowl session | 18:08 |
| *** sgotliv has quit IRC | 18:16 | |
| *** sreshetn1 has quit IRC | 18:19 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Implement poll util and plugin poll util https://review.openstack.org/157392 | 18:20 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add usages of poll util for service modules https://review.openstack.org/163463 | 18:20 |
| openstackgerrit | Trevor McKay proposed openstack/python-saharaclient: Add support for job-types-list https://review.openstack.org/161448 | 18:20 |
| openstackgerrit | Trevor McKay proposed openstack/python-saharaclient: Add support for job-types-list https://review.openstack.org/161448 | 18:22 |
| openstackgerrit | Merged openstack/sahara: Add Sentry service test in cdh plugin integration test https://review.openstack.org/157915 | 18:33 |
| *** sreshetn1 has joined #openstack-sahara | 18:40 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Add usages for step_type field https://review.openstack.org/162997 | 18:46 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Move updating provision progress to conductor https://review.openstack.org/159421 | 18:46 |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Minor - misprint corrected https://review.openstack.org/164825 | 18:53 |
| elmiko | tmckay: reading the default-templates spec review now, something occurs to me | 18:58 |
| elmiko | why even allow node group templates, why not just create all-in-one cluster templates? | 18:58 |
| elmiko | that would simplify things and make the templates more flat | 18:59 |
| *** Nikolay_St has quit IRC | 18:59 | |
| *** Nikolay_St has joined #openstack-sahara | 19:00 | |
| openstackgerrit | Vitaly Gridnev proposed openstack/sahara: Minor - allow changing status description of deleting cluster https://review.openstack.org/164829 | 19:02 |
| tmckay | crobertsrh, ^^ elmiko, hmm, could be. If you have multiple cluster templates, though, you can reuse some of the ng templates. | 19:02 |
| tmckay | elmiko, why not post a comment on the spec? | 19:03 |
| elmiko | tmckay: given the way you are specifying the detection of ng template vs. cluster templates, it will break if someone attempts to create an all-in-one template | 19:03 |
| tmckay | I'm not sure if embedded node groups are something we are trying to move away from, or encourage | 19:03 |
| crobertsrh | Yeah, ng_templates might still be useful. | 19:03 |
| elmiko | i agree they are useful in general, but i'm having doubts about their usefulness in the default stuff specifically | 19:04 |
| tmckay | elmiko, I don't think so, because it checks at top level. It's a dict. | 19:04 |
| crobertsrh | will it really break it? | 19:04 |
| elmiko | i dunno, that's why i'm asking =) | 19:04 |
| crobertsrh | a cluster template can be created....it should be using the cluster_template schema | 19:04 |
| tmckay | so node_processes or flavor_id buried in a node group will be okay | 19:04 |
| elmiko | ok | 19:04 |
| crobertsrh | You can have your fancy-pants all-in-one custom default cluster templates | 19:05 |
| elmiko | crobertsrh: i meant more for the language in the spec that talks about differentiating the template types based on their contents | 19:05 |
| crobertsrh | Ah | 19:05 |
| *** vgridnev_ has quit IRC | 19:06 | |
| elmiko | i think there are some side effects from all-in-one style that might be nice for this too, but i'm not quite sure. | 19:06 |
| elmiko | when you create a cluster template from an all-in-one, you only get the template in the db, no node group templates. | 19:06 |
| elmiko | that might be a simplification in terms of creating less that the user will see. not sure, just spitballing. | 19:07 |
| tmckay | we can still support both, but we could just distribute cluster templates for now. Also not sure. | 19:08 |
| elmiko | yea, just a thought while reading this | 19:09 |
| openstackgerrit | Ethan Gafford proposed openstack/sahara: Default templates for Spark https://review.openstack.org/164797 | 19:09 |
| openstackgerrit | Ethan Gafford proposed openstack/sahara: Default templates for HDP https://review.openstack.org/164772 | 19:10 |
| tmckay | elmiko, trying to figure out how to handle floating_ip_pool sanely as a replaceable value. If you leave it out, it shows up as "null" | 19:10 |
| tmckay | but you might not want it | 19:11 |
| elmiko | maybe check the CONF while processing and if the user did not define a value *and* use_floating_ips=False then remove it? | 19:12 |
| egafford | tmckay: Is null an acceptable value that translates to None and behaves appropriately? | 19:12 |
| tmckay | I'm wondering if the CLI should just blow away replaceable fields if they are None in the CLI. So that if you do not specify a config, it blows it away. And log an INFO message, maybe | 19:12 |
| elmiko | i think that field is needed for schema validation though, right? | 19:12 |
| tmckay | egafford, if you send None through JSON you get "None" | 19:12 |
| tmckay | null means it wasn' | 19:13 |
| tmckay | wasn't there | 19:13 |
| tmckay | "" comes through as "" | 19:13 |
| egafford | tmckay: Right; that may be acceptable. I can certainly replace "floating_ip_pool": "{floating_ip_pool}" with {floating_ip_pool}, allowing you to empty out the whole line, but that might be diretier. | 19:13 |
| tmckay | not sure what the implications are | 19:13 |
| egafford | s/diretier/dirtier | 19:13 |
| egafford | diretier sounds like it should be pronounced in Frech. | 19:14 |
| elmiko | looks like floating_ip_pool is not required for validation, so probably safe to remove if necessary | 19:15 |
| openstackgerrit | Ethan Gafford proposed openstack/sahara: Default templates for Vanilla https://review.openstack.org/164789 | 19:15 |
| tmckay | I think blowing away values with nothing specified in config (default config param value is None) is okay. | 19:15 |
| egafford | tmckay: Are you okay enough with the default template sets that I should take them out of wf-1? I've added a dep on your CLI change. | 19:16 |
| elmiko | i think we need to get this spec merged... | 19:16 |
| tmckay | leave it wf -1 for the moment, til I get this floating_ip_pool thing sorted | 19:16 |
| tmckay | elmiko, ack | 19:16 |
| egafford | tmckay: Okey doke. | 19:17 |
| tmckay | do you guys think we need a log message if we blow away a replaceable field because there was no config value? | 19:20 |
| tmckay | or we consider that not noteworthy | 19:21 |
| elmiko | i | 19:21 |
| elmiko | i'd say debug log maybe | 19:21 |
| crobertsrh | Might want to have it as a debug | 19:21 |
| crobertsrh | heh | 19:21 |
| tmckay | okey-doke | 19:21 |
| *** IlyaE has quit IRC | 19:24 | |
| tmckay | well, that works. | 19:24 |
| egafford | tmckay: Hoorah! | 19:26 |
| tmckay | if you leave it out of the config file, you end up with "null" in JSON | 19:26 |
| tmckay | if you specify it, you get your value | 19:27 |
| tmckay | pretty sane | 19:27 |
| elmiko | sounds good | 19:27 |
| egafford | Yeah, ideal really. | 19:27 |
| tmckay | I'll just add that little debug message | 19:27 |
| elmiko | tmckay: is the cli review under you or egafford ? | 19:30 |
| tmckay | me | 19:30 |
| egafford | elmiko: I'm mostly on image generation stuff at the moment; just throwing the cycles I can at templates. :) | 19:30 |
| elmiko | tmckay: found it | 19:30 |
| elmiko | egafford: cool, i just didn't see it on my list. | 19:31 |
| egafford | elmiko: Ack. | 19:31 |
| crobertsrh | So far, I'm loving the default templates stuff. My "restack" scripts are about to get a lot simpler and more useful by setting up all of my templates right out of the box. | 19:31 |
| elmiko | +1 | 19:32 |
| tmckay | oh, man -- because of the way update works, if you make something with a floating_ip_pool field then try to update it with something that doesn't have that field, ot | 19:34 |
| tmckay | it's not replaced | 19:34 |
| elmiko | crobertsrh: there might also be a win in the future for the guided stuff if we could expand functionality later to allow default cluster creation with a 1-click or something | 19:34 |
| tmckay | which is expected, but subtle | 19:35 |
| crobertsrh | Yes..I wish we had done default templates sooner | 19:35 |
| tmckay | same for any field you want to drop, actually | 19:35 |
| tmckay | your only choice is to delete and recreate for fields you want to drop | 19:35 |
| elmiko | we'd need to modify the saharaclient to allow default template access though | 19:35 |
| tmckay | oh well | 19:35 |
| elmiko | doh | 19:35 |
| tmckay | maybe update should just be removed | 19:36 |
| crobertsrh | It'll give me something to do for L | 19:36 |
| tmckay | maybe you should only have create and delete | 19:36 |
| elmiko | well, update just because delete then create ? | 19:36 |
| elmiko | s/because/becomes | 19:36 |
| crobertsrh | tmckay: That might be sufficient.....or have "update" just do a delete/create in the background. | 19:36 |
| crobertsrh | give the illusion of update | 19:36 |
| elmiko | pretty sure you could even preserve the id | 19:37 |
| tmckay | hmm, I wonder if sqlalchemy will trash a field if you pass it a none | 19:38 |
| tmckay | so, not missing, but present and == None | 19:38 |
| elmiko | not sure | 19:39 |
| egafford | I'd hope that would result in a null. | 19:39 |
| elmiko | i think it depends if the table field allows NULL as a value | 19:39 |
| egafford | Otherwise sqlalchemy is scary and is making a lot of decisions about my data model. | 19:39 |
| tmckay | egafford, well, it might do something silly like generating "None" for a string value | 19:40 |
| elmiko | i know in django you need to explicitly set a field to allow null/None as a value | 19:40 |
| tmckay | which is what json seems to do | 19:40 |
| egafford | tmckay: JSON generates null, not "null", right? | 19:40 |
| elmiko | egafford: yea, null | 19:40 |
| openstackgerrit | Merged openstack/sahara: Change imports after moving tempest common code https://review.openstack.org/163310 | 19:40 |
| egafford | json null is cool; null -> "null" would be really unfortunate. | 19:41 |
| elmiko | lol agreed | 19:41 |
| egafford | Just so, [thing: null] -> [thing: NULL] is cool; [thing: null] -> [] is not. | 19:42 |
| tmckay | maybe we should kick def templates to Liberty | 19:42 |
| elmiko | yea, that would be tragic | 19:42 |
| egafford | (At the data layer. In impl, it can be fine.) | 19:42 |
| elmiko | (that was at egafford's comment) | 19:42 |
| elmiko | if we kick defaults to L, then i definitely think we should use jinja. just sayin | 19:43 |
| tosky | are all Sahara blueprints going through the git/review-based process nowadays? | 19:45 |
| egafford | tosky: Specs are; I think very small changes can conceivably go through without a reviewed spec. | 19:46 |
| *** hdd has quit IRC | 19:47 | |
| *** IlyaE has joined #openstack-sahara | 19:48 | |
| *** uvirtbot has quit IRC | 19:50 | |
| *** IlyaE has quit IRC | 19:51 | |
| *** sgotliv has joined #openstack-sahara | 19:52 | |
| elmiko | tosky: i've seen some refactoring allowed through without specs. i think the idea is to gate features/large changes through blueprints/specs | 19:53 |
| tosky | elmiko: and for testing-related blueprints which can span two releases? | 19:54 |
| tosky | :D | 19:54 |
| elmiko | well... | 19:54 |
| elmiko | they start as a bp/spec in one release, and then the spec gets moved to the 2nd release folder if it's not implemented yet | 19:55 |
| elmiko | at least, that's how i think it should work | 19:55 |
| *** hdd has joined #openstack-sahara | 19:56 | |
| tosky | and if you know that you can't implement everything now because of other depedencies, but you can start at least something before the release (and being a test should be like bugfixes)? | 19:56 |
| tosky | a possibility would be create two specs, but it's really the same process | 19:57 |
| tosky | to be tracked together | 19:57 |
| elmiko | yea, i guess it depends on the specifics. | 19:57 |
| elmiko | you could create a single spec and when it's approved start implementing "Partial-implements" reviews against it | 19:58 |
| *** IlyaE has joined #openstack-sahara | 19:58 | |
| elmiko | or, you could go with what you said. basically, 2 specs | 19:58 |
| tosky | thanks | 19:59 |
| tosky | I will recheck the real status before proceeding | 20:00 |
| elmiko | tmckay: i'm most of the way through the cli review, looks good so far. | 20:04 |
| *** IlyaE has quit IRC | 20:04 | |
| elmiko | i'll prob. finish it up tonight | 20:04 |
| *** IlyaE has joined #openstack-sahara | 20:04 | |
| *** sreshetn1 has quit IRC | 20:06 | |
| *** sgotliv has quit IRC | 20:10 | |
| *** IlyaE has quit IRC | 20:11 | |
| *** sreshetn1 has joined #openstack-sahara | 20:13 | |
| *** sgotliv has joined #openstack-sahara | 20:14 | |
| openstackgerrit | Merged openstack/sahara: Add a common HBase lib in hdfs on cluster start https://review.openstack.org/162657 | 20:23 |
| *** crobertsrh has quit IRC | 20:29 | |
| *** Networkn3rd has quit IRC | 20:36 | |
| *** IlyaE has joined #openstack-sahara | 20:43 | |
| openstackgerrit | Trevor McKay proposed openstack/sahara: Add a CLI tool for managing default templates https://review.openstack.org/163649 | 21:03 |
| *** IlyaE has quit IRC | 21:05 | |
| *** syncroswitch has joined #openstack-sahara | 21:11 | |
| *** tellesnobrega has quit IRC | 21:13 | |
| *** sreshetn1 has quit IRC | 21:14 | |
| *** tellesnobrega has joined #openstack-sahara | 21:18 | |
| *** sreshetn1 has joined #openstack-sahara | 21:21 | |
| *** tmckay has quit IRC | 21:25 | |
| *** IlyaE has joined #openstack-sahara | 21:30 | |
| *** sgotliv has quit IRC | 21:42 | |
| *** syncroswitch has quit IRC | 21:43 | |
| *** sreshetn1 has quit IRC | 21:44 | |
| *** IlyaE has quit IRC | 22:20 | |
| *** IlyaE has joined #openstack-sahara | 22:42 | |
| *** sgotliv has joined #openstack-sahara | 22:56 | |
| *** bandarji has quit IRC | 23:06 | |
| *** sgotliv has quit IRC | 23:10 | |
| *** IlyaE has quit IRC | 23:17 | |
| *** egafford has quit IRC | 23:18 | |
| *** IlyaE has joined #openstack-sahara | 23:53 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!