Reading Spark Plans

Spark performance work starts with the plan, not with cluster size. A query plan shows where Spark scans data, applies filters, exchanges data across the network, and joins relations.

What to look for

• Wide scans where filters could be pushed earlier.
• Exchanges before joins and aggregations.
• Skewed stages where a few tasks dominate runtime.
• Repeated actions that recompute the same lineage.

Example probe

df = (
    spark.table("main.silver.events")
    .where("event_date >= current_date() - interval 7 days")
    .groupBy("account_id")
    .count()
)

df.explain("formatted")

Team habit

Ask engineers to attach the formatted plan when they request tuning help. It turns performance work from folklore into evidence.