40 PySpark Interview Questions Asked at Pune MNCs (With Answers)

Walk into any data engineering interview in Hinjewadi, Kharadi or Magarpatta and one thing is certain: PySpark will come up. Whether the role is at a services giant, a product company or a bank's captive centre, interviewers want to know whether you can make Spark fast, reliable and cheap, not just whether you can write a DataFrame filter.

At IntelliBI, our mentors have sat on both sides of the table. We have collected the questions our learners from the Data Engineering Course in Pune reported back after interviews at TCS, Infosys, Persistent, Cognizant, Accenture, Capgemini and several Pune product startups. Below are the 40 that appear most often, grouped the way interviewers actually ask them, with answers that are short enough to say out loud and deep enough to survive a follow-up.

How Pune MNCs Structure a PySpark Interview

Most companies run a 45–60 minute technical round that moves from fundamentals to tuning to a scenario. The first ten minutes check whether you understand Spark's architecture. The middle twenty test DataFrame fluency and performance. The last stretch is a scenario: "your job takes four hours, the business wants it in forty minutes, what do you do?" If you have practised real projects in a Data Engineering Training in Pune programme rather than only watched tutorials, this last part is where you pull ahead.

Section 1: Spark Architecture and Fundamentals (Questions 1–8)

1. What is Apache Spark and why did it replace Hadoop MapReduce?

Spark is a distributed, in-memory computing engine. MapReduce wrote intermediate results to disk after every stage; Spark keeps data in memory across a DAG of operations, which makes iterative and multi-stage jobs 10–100x faster and far simpler to write.

2. Explain the Spark architecture: driver, executors and cluster manager.

The driver runs your main program, builds the DAG and schedules tasks. Executors are JVM processes on worker nodes that run tasks and hold cached data. The cluster manager (YARN, Kubernetes, Databricks) allocates resources between them.

3. What is lazy evaluation and why does it matter?

Transformations (select, filter, join) are only recorded, not executed. Spark runs them when an action (count, show, write) is called. This lets the Catalyst optimiser reorder and combine steps, and it is why a wrongly placed collect() can ruin performance.

4. Difference between transformations and actions, with examples.

Transformations return a new DataFrame: filter, withColumn, groupBy, join. Actions trigger execution and return a result or write data: count, collect, show, write. Narrow transformations stay within a partition; wide ones shuffle data.

5. What are jobs, stages and tasks?

Each action creates a job. A job is split into stages at every shuffle boundary. Each stage is split into tasks, one per partition. Reading the Spark UI in this order is the fastest way to find a bottleneck.

6. What is a shuffle and why is it expensive?

A shuffle redistributes data across partitions so that related keys land together, as in groupBy, join or repartition. It involves disk writes, network transfer and serialisation, which makes it the single biggest cost in most jobs.

7. RDD vs DataFrame vs Dataset: which should you use?

RDDs are low-level and untyped for the optimiser. DataFrames are columnar, schema-aware and optimised by Catalyst and Tungsten. Datasets add compile-time typing in Scala. In PySpark, use DataFrames unless you have a very specific reason not to.

8. What is SparkSession and how is it different from SparkContext?

SparkSession is the unified entry point since Spark 2.x; it wraps SparkContext, SQLContext and HiveContext. You create one with SparkSession.builder and use it for reading, SQL and configuration.

Section 2: DataFrame Operations (Questions 9–18)

9. How do you read a CSV with a defined schema instead of inferSchema?

Define a StructType with StructField entries and pass it to spark.read.schema(schema).csv(path). inferSchema scans the file twice and can guess wrong types, so production code should always declare the schema.

10. Explain the difference between select, withColumn and selectExpr.

select picks or derives columns, withColumn adds or replaces one column at a time, and selectExpr lets you write SQL expressions inline. Chaining many withColumn calls creates a long plan; prefer a single select for several derived columns.

11. What types of joins does Spark support and when do you use each?

Inner, left, right, full outer, left semi, left anti and cross. Left semi returns rows from the left that have a match; left anti returns rows that do not, which is a clean way to find missing records without a null check.

12. How do window functions work in PySpark?

Window functions compute a value over a defined frame of rows without collapsing them. You define Window.partitionBy().orderBy() and apply row_number, rank, lag, lead or sum. Deduplication using row_number over a partition is a classic interview task.

13. Difference between groupBy and Window.

group By aggregates and reduces rows to one per group. A Window keeps every row and adds the aggregated value next to it, which is what you need for running totals, rankings and previous-value comparisons.

14. How do you handle null values?

Use na.drop() to remove, na.fill() to replace, coalesce() to pick the first non-null, and isNull/isNotNull in filters. Always decide null handling per column; a blanket fill(0) can silently corrupt metrics.

15. What is a UDF and why should you avoid it where possible?

A user-defined function wraps Python logic for Spark. Standard UDFs serialise each row between the JVM and Python, which is slow and invisible to the optimiser. Prefer built-in functions; if you must, use pandas UDFs, which operate on Arrow batches.

16. How do you remove duplicates based on specific columns while keeping the latest record?

Create a window partitioned by the key and ordered by the timestamp descending, add row_number, and filter row_number == 1. dropDuplicates() alone does not let you choose which row survives.

17. Explain explode and how you flatten nested JSON.

explode turns an array column into one row per element. For nested structs, use dot notation or select col("a.b"). Combining explode with select over struct fields flattens most API payloads.

18. How do you convert between PySpark and pandas DataFrames, and what is the risk?

toPandas() collects all data to the driver, which can crash it on large datasets. Use it only on small results; for distributed pandas-style code, use the Pandas API on Spark.

Section 3: Performance Tuning (Questions 19–28)

This is where Pune interviewers separate candidates who have run production pipelines from those who have not. Learners who complete Databricks Training in Pune with real cluster time usually find this the easiest section.

19. What is partitioning and how do you decide the number of partitions?

Partitions are the units of parallelism. A good starting point is two to four partitions per CPU core, with each partition around 128 MB. Too few means idle cores; too many means scheduling overhead and small files.

20. repartition vs coalesce: what is the difference?

repartition performs a full shuffle and can increase or decrease partitions evenly. coalesce only merges existing partitions without a shuffle, so it is cheaper for reducing partition count before a write but can create uneven sizes.

21. What is data skew and how do you fix it?

Skew is when a few keys hold most of the data, so a few tasks run for hours while others finish in seconds. Fixes: enable Adaptive Query Execution skew join handling, salt the key by adding a random suffix, broadcast the small side, or isolate hot keys and process them separately.

22. Explain broadcast joins.

When one side of a join is small (under spark.sql.autoBroadcastJoinThreshold, default 10 MB), Spark copies it to every executor so the join happens locally without a shuffle. You can force it with broadcast(df).

23. What is Adaptive Query Execution (AQE)?

AQE, on by default since Spark 3.2, re-optimises the plan at runtime using actual statistics: it coalesces shuffle partitions, switches sort-merge joins to broadcast joins and splits skewed partitions. Knowing this signals you work with modern Spark.

24. cache() vs persist(): when do you use them?

Both store a DataFrame for reuse. cache() uses MEMORY_AND_DISK by default; persist() lets you choose a storage level. Use them when the same DataFrame feeds multiple actions, and unpersist when done to free executor memory.

25. Why is Parquet preferred over CSV?

Parquet is columnar, compressed and carries its schema. Spark reads only the columns it needs and uses predicate pushdown to skip row groups, which can cut I/O by 90 percent compared with CSV.

26. What is the small files problem and how do you solve it?

Thousands of tiny files overwhelm the driver with metadata and slow reads. Fix by coalescing before writes, using maxRecordsPerFile, or running OPTIMIZE on Delta tables to compact files.

27. How do you read the Spark UI to debug a slow job?

Start at the Jobs tab to find the longest job, open its stages, look at task duration distribution for skew, check shuffle read/write sizes, and inspect the SQL tab for the physical plan and exchange nodes.

28. What causes executor out-of-memory errors and how do you prevent them?

Common causes are oversized partitions, collect() on large data, wide UDF rows and excessive caching. Prevent by right-sizing partitions, increasing executor memory or memoryOverhead, and avoiding driver-side collection.

Section 4: Delta Lake, Databricks and Cloud (Questions 29–35)

Almost every Pune enterprise now runs Spark on Databricks, on Azure or AWS. An Azure Data Engineer Course in Pune that stops at plain Spark leaves you exposed here.

29. What is Delta Lake and what does it add to Parquet?

Delta Lake is a storage layer that adds a transaction log to Parquet files. It gives ACID transactions, schema enforcement and evolution, time travel, and efficient updates and deletes, which plain Parquet cannot do.

30. Explain the Medallion architecture.

Bronze holds raw ingested data, Silver holds cleaned and conformed data, Gold holds business-level aggregates. Each layer is a Delta table, which makes lineage, reprocessing and quality checks straightforward.

31. How does MERGE INTO work and when do you use it?

MERGE performs upserts: it matches source rows to target rows on a key and updates, inserts or deletes accordingly. It is the standard pattern for CDC loads and Slowly Changing Dimension Type 2.

32. What is Auto Loader?

Auto Loader incrementally ingests new files from cloud storage using the cloudFiles source, tracking which files have been processed. It scales to millions of files and handles schema inference and evolution.

33. What is Unity Catalog?

Unity Catalog is Databricks' governance layer: a three-level namespace (catalog, schema, table) with centralised access control, lineage and audit across workspaces. Interviewers ask it to check whether you have worked in a governed enterprise setup.

34. Difference between Databricks Jobs and Delta Live Tables pipelines.

Jobs orchestrate notebooks and tasks with dependencies. DLT pipelines are declarative: you define tables and expectations, and the platform manages dependencies, retries and data quality. Use DLT for ETL, Jobs for everything else.

35. How do you run a PySpark notebook from Azure Data Factory?

Use the Databricks Notebook activity in an ADF pipeline, authenticate through a linked service with a managed identity or access token, and pass parameters via base parameters, which the notebook reads with dbutils.widgets.

Section 5: Streaming and Scenario Questions (Questions 36–40)

36. What is Structured Streaming?

Structured Streaming treats a live stream as an unbounded table and processes it in micro-batches using the same DataFrame API. Sources include Kafka, Event Hubs and files; sinks include Delta tables.

37. Explain watermarks and checkpoints.

A watermark tells Spark how late data can arrive before a window is finalised, which bounds state size. A checkpoint persists progress and state so the stream recovers exactly where it stopped after a failure.

38. Scenario: your daily job has grown from 30 minutes to 4 hours. What do you check?

Check data volume growth, partition count and skew in the Spark UI, look for new UDFs or collect() calls, confirm AQE is on, compare file counts for the small-files problem, and verify the cluster was not downsized.

39. Scenario: a join between a 2 TB fact table and a 5 MB dimension table is slow.

The dimension should be broadcast. Check that its size is under the threshold or force broadcast(). If statistics are stale, Spark may have chosen a sort-merge join; AQE usually corrects this at runtime.

40. Scenario: how would you implement SCD Type 2 in PySpark on Delta?

Compare incoming rows with current rows on the business key, detect changed attributes, close the old row by setting end_date and is_current false, and insert the new version. Implement it with a single MERGE using whenMatched and whenNotMatched clauses.

How to Prepare So These Questions Feel Easy

Reading answers is a start, but Pune interviewers follow up relentlessly: "show me the Spark UI", "what did you set spark.sql.shuffle.partitions to", "how big was the table". You can only answer those if you have built and tuned real pipelines. That is why the Data Engineer Course in Pune at IntelliBI is built around 10+ production-grade projects, including an end-to-end Medallion lakehouse, a healthcare data platform and an insurance claims analytics pipeline, all on Databricks with Azure and AWS.

Our learners also work through technology-wise question banks, HackerRank-style coding rounds, system design sessions and mock interviews with senior engineers. If you are comparing options for the Best Data Engineering Course in Pune, ask every institute one question: will I get cluster time on real data, or only slides?

Who This Guide Is For

Freshers with SQL and Python basics, working professionals moving from ETL tools like Informatica or SSIS, testers and support engineers switching careers, and analysts who want to move upstream. Our Data Engineering Classes in Pune run in the evening and on weekends at Thergaon, ten minutes from Hinjewadi, with live online access for learners across India.

If you are weighing cloud engineer courses in Pune against a focused data track, remember that Spark, Delta Lake and Databricks skills are what the job descriptions in Hinjewadi and Kharadi actually list. An Advanced Data Engineer Course in Pune that covers Azure, AWS, Databricks and the new GenAI and Agentic AI tooling prepares you for the interview as it is in 2026, not as it was in 2020.

Final Word

Bookmark these 40 questions, practise saying the answers out loud, and then go build something that forces you to use them. When you are ready to turn preparation into an offer letter, explore the Best Data Engineering Institute in Pune programme at IntelliBI Innovations Technologies, book a free demo class, and let our mentors take you from your first SparkSession to your first data engineering role.