Unable to view the spline ui while running custom pyspark job

[dhanju123]

We created a sample pyspark job and gave the spark-submit commands as following in ec2 instance

sudo ./bin/spark-submit \
  --packages za.co.absa.spline.agent.spark:spark-3.1-spline-agent-bundle_2.12:0.6.1 \
  --conf “spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener” \
  --conf "spark.spline.lineageDispatcher.http.producer.url=[http://](http://localhost:9090/producer)[ec2] (http://localhost:8080/producer)[:8080/producer](http://localhost:9090/producer)" \
  --conf "spark.spline.lineageDispatcher=logging" \
  /home/ec2-user/spline-sandbox/mysparkjob.py

we are able to view the output in the console but unable to view in spline UI what additional steps need to be done ?

[wajda]

Note this line:

--conf "spark.spline.lineageDispatcher=logging"

With that you set a logging dispatcher as a root dispatcher. The dispatchers in Spline are composable. There is the only root dispatcher for the agent, that (depending on the type of the dispatcher) could call other dispatchers if need be.

If you want to send lineage info over HTTP and log it at the same time, use a composite dispatcher instead:

--conf "spark.spline.lineageDispatcher=composite"
--conf "spark.spline.lineageDispatcher.composite.dispatchers=http,logging"

See the lineage dispatchers doc for more info.

[dhanju123]

Hi ,

Still the same issue , even after adding the above configuration , we are trying to execute in ec2 instance so any docker image need to be created from the pyspark job ? how to execute the pyspark job in ec2 instance and view the lineage in spline ? can you kindly guide us ?

[wajda]

Spline is available as a docker.

We have a sample docker-compose config:
https://github.com/AbsaOSS/spline-getting-started/tree/main/docker

See also AWS example:
https://github.com/AbsaOSS/spline-getting-started/tree/main/spline-on-AWS-demo-setup

[dhanju123]

To view the spline do we need a docker?

[wajda]

No, you don't, but you can, and it's probably the most convenient way anyway. Please read the doc here - https://absaoss.github.io/spline/

[dhanju123]

Also we referred both link mentioned above but how to run the own custom pyspark job is not mentioned , do we create a docker image out of the spark job and mention it in .yml file if so how can you kindly explain ?

[wajda]

https://github.com/AbsaOSS/spline-spark-agent#codeless-initialization

pyspark \
  --packages za.co.absa.spline.agent.spark:spark-2.4-spline-agent-bundle_2.12:<VERSION> \
  --conf "spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener" \
  --conf "spark.spline.lineageDispatcher.http.producer.url=http://localhost:9090/producer"

http://localhost:9090 is where your Spline REST gateway sits.

You can open http://<spline.server.host>/index.html and copy/paste the Producer API base URL.

[dhanju123]

Hi ,

We created a pyspark job in our ec2 instance and when we submitted the spark job using the following command :

sudo ./bin/spark-submit --packages za.co.absa.spline.agent.spark:spark-3.1-spline-agent-bundle_2.12:0.6.1 --conf “spark.sql.queryExecutionListeners=za.co.absa.spline.harvester.listener.SplineQueryExecutionListener” --conf spark.spline.producer.url=http://ec2 ip :8080/producer" /home/ec2-user/spline-sandbox/mysparkjob.py

We are able to view the output in console , but unable to view in spline UI

Can we know what is the step we are missing ?

We are not using docker here , just create a sample pyspark job and tried to run using the above command

[wajda]

Once again, if you said you see the lineage info in the logs, the agent is working properly. All you need is to run Spline REST gateway and provide a correct Producer Base URL to the agent.

Make sure the Spline server is ready.

[dhanju123]

Hi, we are not able to see the lineage info in the logs , but we are able to see the output of the spark transformation / spark job in the logs , Even the spline server is not ready because we are not using docker compose up command

[wajda]

Sorry, I don't have enough capacity to provide you with step-by-step instructions and troubleshoot you particular case. Please read the READMEs I referred above. Also read this discussion - #394