dispatcher process interrupted

[dmartinb06]

Hi everyone!

I have a spark application using the spline agent and a custom dispatcher. The spark application generates multiple plans and terminates before the dispatcher has finished processing all the plans, what do I have to do to make my spark application wait for the dispatcher to finish?

[cerveada]

Hello, what version of spark agent do you use?

[dmartinb06]

Hi Adam,

This is my dependency:

za.co.absa.spline.agent.spark agent-core_2.12 1.1.0

[cerveada]

Can you please try it with the latest agent version 2.2.0?
You could also try to set hadoop.service.shutdown.timeout to some high value.

If the problem is still there, please open an issue and add a log from the agent.

[dmartinb06]

Hello again,

in the end the problem was that the AsyncEventQueue was filling up because our dispatcher was taking too long, we solved it by increasing the parameters
spark.scheduler.listenerbus.eventqueue.capacity
and
spark.scheduler.listenerbus.eventqueue.shared.capacity

[cerveada]

Thanks for sharing the solution.