https://zivali.github.io/series/spark-101/ Recent content in Spark 101 on Ziva Hugo en This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License. Sat, 15 Aug 2026 20:57:00 +0000 https://zivali.github.io/posts/spark-join-strategies-five-common-joins-with-a-runnable-local-lab/ Sat, 15 Aug 2026 00:52:00 +0000 https://zivali.github.io/posts/spark-join-strategies-five-common-joins-with-a-runnable-local-lab/ <link rel="stylesheet" href="https://cdn.jsdelivr.net/npm/katex@0.16.2/dist/katex.min.css" integrity="sha384-bYdxxUwYipFNohQlHt0bjN/LCpueqWz13HufFEV1SUatKs1cm4L6fFgCi1jT643X" crossorigin="anonymous"> <p><code>DataFrame.join()</code> describes a logical relationship. A <strong>physical join strategy</strong> describes how Spark makes the two sides meet: by copying a small relation to every executor, repartitioning both sides, sorting them, or comparing every possible pair.</p> <p>This guide explains five common strategies, shows their executor-level data movement, and pairs each one with a minimal PySpark shape and Spark UI evidence. The companion Docker standalone lab is in: <a href="https://github.com/zivali/spark-join-strategies-lab" target="_blank" rel="noopener noreferrer">https://github.com/zivali/spark-join-strategies-lab</a>, which generates deterministic 10M and 100M-row datasets and saves the observed plan and runtime data.</p>