Spark user info from the agent

[vishalag001]

Currently, the spark agent is not capturing the sparkUser who has initiated the job. It is readily available in the sparkContext. The user info can go into the extras of executionPlan or the event.

Benefits of User info:

• Help enterprises to identify individual teams and get back with any findings regarding lineage.
• A good metadata attribute from a lineage perspective.

I am able to capture sparkUser info in local. Happy to add it to open source.

@wajda Any thoughts on this ? Happy to brainstorm

[wajda]

Have you tried MetadataCollectingFilter? This is a post-processing lineage filter that is specifically designed for capturing a user-specific extra information, like the context related info, environment variables, user identity etc etc.
It's available in the development branch (ver 1.0.0-SNAPSHOT). It's not properly documented yet, but you can see example of usage in the test case or the example Spark app

So your capturing rules would look something like this:

spline.postProcessingFilter.userExtraMeta.rules=               \
{                                                              \
  "executionEvent": {                                          \
    "sparkUser": { "$js": "session.sparkContext.sparkUser" }   \
  }                                                            \
}

Don't forget to enable the filter with something like this:

spline.postProcessingFilter=userExtraMeta

[wajda]

I've created a related ticket #474

[vishalag001]

#475
Raised a PR