PySpark

PySpark Overview

PySpark: PySpark in Databricks is the official Python API for Apache Spark that enables users to work with big data using Python inside Databricks. It is used for processing and analyzing large-scale data using Python. PySpark is used for large-scale data processing, transformation, analytics, and preparation for machine learning using Python on a cluster.


What PySpark is used for With PySpark, you can:
Read huge datasets (GBs–TBs)
Clean and transform data
Join and aggregate data
Process streaming data
Prepare data for analytics and ML
All of this runs across a cluster, not just one machine.

Why use PySpark?
PySpark is widely used because it:
🚀 Handles large-scale data (terabytes to petabytes)
⚡ Runs faster than traditional tools like Pandas on big data
🔄 Processes data in parallel across clusters
📊 Supports ETL, SQL, streaming, and machine learning
🧠 Integrates well with Python data libraries



PySpark explained in ONE single diagram, covering architecture → execution → optimization → result in a clean, end-to-end flow

Qusetion. Explain this PySpark diagram end-to-end
Answer:
This diagram shows the full execution flow of a PySpark job.
The PySpark code is sent to the Driver, which applies lazy evaluation and builds a DAG.
The Catalyst Optimizer converts the logical plan into an optimized physical plan.
The DAG Scheduler splits the plan into stages based on shuffle boundaries.
The Cluster Manager allocates resources, and Executors run tasks in parallel on partitions.
Tungsten optimizes memory and CPU usage, and the final result is returned or written to storage.

In simple, PySpark uses lazy evaluation to build an optimized DAG, splits it into stages and tasks, and executes them in parallel on executors using Catalyst and Tungsten for performance.

PySpark Execution Summary Diagram:
   PySpark Code → DAG Creation → Stages → Tasks → Executors → Result

📊 Feature vs Advantage Table:

List of commonly used PySpark functions (focused on DataFrame APIs):

DataFrame I/O:
read.csv · read.json · read.parquet · read.orc · write.csv · write.parquet · saveAsTable

Column & Projection:
select · selectExpr · withColumn · withColumnRenamed drop · alias · col · lit

Filtering & Conditions:
filter / where · when · otherwise isNull · isNotNull · between · isin

Aggregations:
groupBy · agg count · sum · avg · min · max countDistinct · approx_count_distinct

Joins:
join · broadcast inner · left · right · outer · left_semi · left_anti

Sorting & Window:
orderBy · sort row_number · rank · dense_rank lag · lead Window.partitionBy().orderBy()

String Functions:
upper · lower · trim · concat · concat_ws substring · length regexp_replace · regexp_extract · split

Date & Time:
current_date · current_timestamp to_date · to_timestamp date_add · date_sub · datediff year · month · dayofmonth

Math Functions:
abs · round · ceil · floor pow · sqrt · rand

Array & Map:
explode · posexplode size · array · array_contains map_keys · map_values

Null Handling:
fillna · dropna · na.fill · na.drop coalesce

Sampling & Dedup:
sample · distinct · dropDuplicates

Performance & Optimization:
cache · persist · unpersist repartition · coalesce hint · explain

SQL & Views:
createOrReplaceTempView spark.sql