PySpark: Zero to Hero (Ease With Data)

About this course

A practical PySpark course for data engineers, from transformations and actions to the Spark UI, joins, caching, skew fixes, Delta Lake and testing. Who it's for: aspiring data engineers who know Python and SQL. These are the free, official videos of the "PySpark - Zero to Hero" playlist from the Ease With Data YouTube channel (youtube.com/@easewithdata). All videos belong to Ease With Data; Sikhami is not affiliated with Ease With Data and simply arranges the public playlist into a course, so you can learn one lesson at a time and keep track of your progress.

What you will learn

• Spark architecture, transformations and actions • Setting up PySpark locally and on Databricks Community • Structured transformations, strings, dates, nulls and window functions • Reading CSV and JSON, and understanding the Spark UI, DAGs and shuffles • Optimising joins, caching, partitioning, AQE and fixing skew • Spark SQL, Delta Lake and memory management • Writing unit tests for PySpark with PyTest

Course content

1 section · 35 lectures · 9h 10m

1. 1. PySpark - Zero to Hero | PySpark Tutorial 2025 | Spark Tutorial 2025 | Learn fro

   • 01 PySpark Tutorial | PySpark Training | Learn from Basics to Advanced Performance Optimization2m
   • 02 How Spark Works - Driver & Executors | How Spark divide Job in Stages | What is Shuffle in Spark4m
   • 03 Spark Transformations & Actions | Why Spark prefers Lazy Evaluation |What are Partitions in Spark5m
   • 04 Spark DataFrames & Execution Plans | Spark Logical and Physical Execution Planning | What are DAG3m
   • 04_2 - Setup PySpark in Local Machine with Jupyter Lab | PySpark Local Machine Setup20m
   • 05 Understand Spark Session & Create your First DataFrame | Create SparkSession object | Spark UI11m
   • 06 Basic Structured Transformation - Part 1 | Write Spark DataFrame Schema |Ways to write DF Columns13m
   • 07 Basic Structured Transformation - Part 2 | Cast Column | Add Column | Static Column Value |Rename12m
   • 08 Working with Strings, Dates and Null | Regex Replace | Convert string to date | Transform NULL16m
   • 09 Sorting data, Union and Aggregation in Spark | Difference in Union and UnionAll | Having Clause10m
   • 10 Window Functions, Unique Data & Databricks Community Cloud | Second Highest Salary | Spark expr10m
   • 11 Data Repartitioning & PySpark Joins | Coalesce vs Repartition | Spark Data Partition | Joins13m
   • 12 Understand Spark UI, Read CSV Files and Read Modes | Spark InferSchema Option | Drop Malformed17m
   • 13 Read Complex File Formats | Parquet | ORC | Performance benefit of Parquet |Recursive File Lookup10m
   • 14 Read, Parse or Flatten JSON data | JSON file with Schema | from_json | to_json | Multiline JSON17m
   • 15 How Spark Writes data | Write modes in Spark | Write data with Partition | Default Parallelism14m
   • 16 Understand Spark Execution on Cluster | Cluster Manager | Cluster Deployment Modes | Spark Submit12m
   • 17 User Defined Function (UDF) | How Spark works with UDF | How to register Python UDF9m
   • 18 Understand DAG, Explain Plans & Spark Shuffle with Tasks |Skipped Stage |Benefit of Shuffle Write16m
   • 19 Understand and Optimize Shuffle in Spark15m
   • 20 Data Caching in Spark | Cache vs Persist | Spark Storage Level with Persist |Partial Data Caching13m
   • 21 Broadcast Variable and Accumulators in Spark | How to use Spark Broadcast Variables12m
   • 22 Optimize Joins in Spark & Understand Bucketing for Faster joins |Sort Merge Join |Broad Cast Join28m
   • 23 Static vs Dynamic Resource Allocation in Spark | Dynamic Allocation vs Databricks Scale up10m
   • 24 Fix Skewness and Spillage with Salting in Spark | Salting Technique | How to identify Skewness21m
   • 25 AQE aka Adaptive Query Execution in Spark | Coalesce Shuffle Partitions | Skew Partitions Fix11m
   • 26 Spark SQL, Hints, Spark Catalog and Metastore | Hints in Spark SQL Query | SQL functions & Joins19m
   • 27 Read and Write from Azure Cosmos DB using Spark | E2E Cosmos DB setup | NoSQL vs SQL Databases21m
   • 28 Get Started with Delta Lake using Databricks | Benefits and Features of Delta Lake | Time Travel34m
   • 29 Optimize Data Scanning with Partitioning in Spark | How Partitioning data works | Optimize Jobs13m
   • 30 Data Skipping and Z-Ordering in Delta Lake Tables | Optimize & Data Compaction Delta Lake Tables18m
   • 31 Delta Tables - Deletion Vectors and Liquid Clustering | Optimize Delta Tables | Delta Clustering12m
   • 32 Spark Memory Management | Why OOM Errors in Spark | Spark Unified Memory | Storage/Execution Mem48m
   • 33 What is Spark Connect? | Spark Connect vs Spark Session | Setup Spark Connect Server with Cluster23m
   • 34 Write PySpark Unit Test Cases using PyTest module | Setup PyTest with PySpark22m