introduce modes for required ordered serialization #1757
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
cc/ @johnynek |
3339abd to
d6945f6
Compare
johnynek
approved these changes
Dec 14, 2017
|
|
||
| import org.scalatest.{ Matchers, WordSpec } | ||
|
|
||
| class NoOrderdSerJob(args: Args) extends Job(args) { | ||
| class NoOrderdSerJob(args: Args, mode: String) extends Job(args) { |
There was a problem hiding this comment.
mode is confusing here since it is almost always scalding.Mode named that. Can we rename?
| case RequireOrderedSerializationMode.Fail => | ||
| Failure(new RuntimeException(message)) | ||
| case RequireOrderedSerializationMode.Log => | ||
| LOG.error(message) |
There was a problem hiding this comment.
should this be an error or warning? I kind of feel warning.
d6945f6 to
5109b95
Compare
|
@johnynek I've just addressed your comments. Thank you for the quick review! |
|
👍 |
benpence
reviewed
Dec 15, 2017
| trait RequiredBinaryComparatorsConfig extends Job { | ||
| override def config = super.config + (Config.ScaldingRequireOrderedSerialization -> "true") | ||
| def requireOrderedSerializationMode: RequireOrderedSerializationMode = RequireOrderedSerializationMode.Fail | ||
| override def config = super.config + (Config.ScaldingRequireOrderedSerialization -> requireOrderedSerializationMode.toString) |
There was a problem hiding this comment.
What's the .toString supposed to output here? Aren't we matching on lowercase in getRequireOrderedSerializationMode above?
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Problem
We're planning to roll out
OrderedSerializationto more jobs. The initial plan was to add theRequiredBinaryComparatorstraits to our execution and job classes, but jobs could fail at runtime ifTypedPipetransformations are done outside the job/execution body since the implicit macro wouldn't be available. For instance, we recommend people to extract transformations in companion objects for testability.Solution
Introduce a mechanism that allows us to add the traits without failing at runtime if a transformation doesn't have the ordered serialization. By overriding
requireOrderedSerializationMode, we can set the mode toLog, which will produce an error message at runtime.Notes