Who’s this Spark Listener?

Jacek Laskowski made a good documentation regarding spark listeners. I made this page since we keep on encountering this ERROR:

2018-06-13 08:07:26 ERROR LiveListenerBus:70 - Dropping SparkListenerEvent because no remaining room in event queue. This likely means one of the SparkListeners is too slow and cannot keep up with the rate at which tasks are being started by the scheduler.
2018-06-13 08:07:26 WARN  LiveListenerBus:66 - Dropped 1 SparkListenerEvents since Thu Jan 01 00:00:00 UTC 1970
2018-06-13 08:08:26 WARN  LiveListenerBus:66 - Dropped 43737 SparkListenerEvents since Wed Jun 13 08:07:26 UTC 2018
...

Question:

I encounter this error and my spark job ends successfully. Am I still assured that my output data is complete?

Answer:

Yes. As long as your spark job succeeds and you did not encounter error during the transformation and actions on stages, your data is still complete.

This error happens due to the fact that the event processor in ListenerBus is just single thread, and loops through all the Listeners for each event and processes each event synchronously. The onDropEvent is called when no further events can be added to the internal eventQueue queue. This kind of error pops up mostly on high latency event processing spark jobs.

One fix is to basically increase the listener bus event queue size using the following config:

spark.scheduler.listenerbus.eventqueue.size=<eventqueue_size>

Note: Default value is 10000

And of course, you need to increase the --driver-memory as well since these events will be located on driver.

Another way to solve this problem is to run using Apache Spark >2.3.0. More info of fix here.