Integration with Python

[simran-suri]

Hi Team,

I'm trying to create lineage for my pipelines and for it I used sc._jvm.za.co.absa.spline.harvester.SparkLineageInitializer.enableLineageTracking(spark._jsparkSession) above my python code.

Now, I need to install spark-3.1-spline-agent-bundle_2.12:0.7.8 in my python code repository, to create that lineage.

Is there any way to install this maven dependency in python code?

[wajda]

What exactly do you mean?
The capability of downloading Maven artifacts depends on the tool/platform that you are using, not the programming language.
For example, with the pyspark you can do pyspark --packages za.co.absa.spline.agent.spark:spark-3.1-spline-agent-bundle_2.12:0.7.8 while the Databricks notebook provides a specific UI for installing packages.

[simran-suri]

Error:

Code:
from pyspark import SparkContext
from pyspark.sql import SparkSession

spark = SparkSession.builder
.config("spark.jars", "jar_file/spark-2.4-spline-agent-bundle_2.12-0.7.10.jar")
.getOrCreate()

sc = spark.sparkContext
sc.setSystemProperty("spark.jars", "jar_file/spark-2.4-spline-agent-bundle_2.12-0.7.10.jar")
sc.setSystemProperty("spark._jsparkSession","za.co.absa.spline.harvester.SparkLineageInitializer.enableLineageTracking")

jvm = sc._jvm
sc._jvm.za.co.absa.spline.harvester.SparkLineageInitializer.enableLineageTracking(spark._jsparkSession)

[simran-suri]

Hi @wajda

Can you please look into the code and suggest what can be done to resolve this error?

[wajda]

The code seems to be OK, make sure you are using Spline agent bundle compatible to the Spark and Scala version in use.

[simran-suri]

Yes, the version are compatible with spline agent bundle, but not able to figure out this error.

[wajda]

The error means that Spline cannot find a org.apache.spark.internal.Logging constructor with an expected signature. It could be caused by the fact that you are using another Spark version than 2.4 (which is what your Spline bundle compiled against), or it could be a proprietary (modified) Spark fork.

What exactly are you using?

[simran-suri]

pyspark==2.4.4

I'm using this in my requirement file as pip install

[wajda]

Try to install Spark directly, not via pip