spark-runtime.md

Spark Framework

The Spark runtime extends the base framework with the following set of components:

Transforms

• AbstractSparkTransform - this is the base class for all spark-based transforms over spark DataFrames.
• SparkTransformConfiguration - this is simple extension of the base TransformConfiguration class to hold the transformation class (an extension of AbstractSpartTransform).

Runtime

• SparkTransformLauncher - this is a class generally used to implement main() that makes use of a SparkTransformConfiguration to start the Spark runtime and execute the transform over the specified set of input files.
• SparkTransformRuntimeConfiguration - this class is a simple extension of the transform's base TransformConfiguration class.