HDFS and cluster exercise
Deliverable: A dataset loaded into HDFS with replication configured, demonstrating what happens to availability when a node is taken down, written up with observations.
Course
Big Data with Hadoop and Spark
Distributed data processing taught with judgement — Hadoop, Spark and pipeline design, plus the far more common case where a single machine would have been the right answer.
- Duration: 5 months
- Classroom · Online live
- Level: intermediate
What you will be able to do
- Explain distributed storage and processing without hand-waving
- Store and retrieve data in HDFS and understand replication and block size
- Write Spark transformations and actions and read the execution plan
- Query distributed data with Spark SQL and Hive
- Build a batch pipeline from raw ingest to an analysis-ready table
- Judge honestly when a single machine and a database would be better
Who this course is for
- Graduates with programming experience in Python, Java or Scala
- Analysts and developers moving into data engineering
- Working SQL users who need to handle larger volumes
- Comfortable SQL and basic Linux are expected
Syllabus
6 modules · 5 months
Module 1. What actually makes data big
- Volume, velocity and variety, examined sceptically
- When a single machine and PostgreSQL beat a cluster
- Distributed storage and processing fundamentals
- The CAP theorem in plain language
Module 2. Hadoop and HDFS
- HDFS architecture, blocks and replication
- NameNode, DataNode and failure behaviour
- MapReduce, conceptually and historically
- YARN and resource management
Module 3. Spark fundamentals
- RDDs, DataFrames and why DataFrames won
- Transformations, actions and lazy evaluation
- Partitions, shuffles and where the time goes
- Reading an execution plan and the Spark UI
Module 4. Spark SQL and Hive
- Querying distributed data with SQL
- Tables, partitioning and file formats
- Parquet, ORC and why columnar storage matters
- Joins at scale and broadcast joins
Module 5. Ingestion and streaming
- Batch ingestion patterns
- Kafka topics, producers and consumers
- Structured streaming basics
- Exactly-once, at-least-once and what you are really promising
Module 6. Building a pipeline
- Raw, cleaned and curated layers
- Idempotency and safe reruns
- Scheduling, monitoring and alerting on failure
- Data quality checks that fail loudly
Tools and technologies you will use
- Apache Hadoop and HDFS
- Apache Spark and PySpark
- Hive
- Apache Kafka
- Python and SQL
- Linux command line
Projects you will build
Spark job with a fixed shuffle
Deliverable: A Spark job profiled in the Spark UI, an expensive shuffle identified and reduced through repartitioning or a broadcast join, with runtimes before and after.
End-to-end batch pipeline
Deliverable: A pipeline taking raw files through cleaning to an analysis-ready partitioned table in Parquet, safe to rerun without duplicating data, with a failing data quality check demonstrated.
Sizing judgement note
Deliverable: A written assessment of three supplied scenarios, deciding for each whether distributed tooling is justified or a single machine would serve better, with reasoning.
Where this course can take you
- Data Engineer
- Big Data Developer
- ETL Developer
- Data Platform Engineer
- Analytics Engineer
Duration, modes and fees
- Duration
- 5 months
- Delivery modes
- Classroom · Online live
- Fees
- Share your details for the current fee
- Fees vary by batch and delivery mode. Cluster and lab access for the practical work is included.
Placement assistance
Every student gets placement assistance — that is what 100% placement assistance means. It is support for all, not a job for all. We do not promise a specific salary, a specific number of interviews, or placement at any named company, and you should be wary of anyone who does.
What is included
- A place in the monthly placement drive, held every third Saturday
- The readiness programme every second Saturday — mock interviews and preparation
- CV review against the specific roles you are targeting
- Portfolio review, so your project work is presented the way a reviewer will read it
- Access to the vacancy pool employers send directly to the Samyak network
- Guidance on which roles realistically fit your background and which do not
- A place in the next drive, with coaching, if you are not selected in this one
What is not included
- Any guarantee of a job, an interview, or a particular salary
- Placement at a named or partner company
- Applying to jobs on your behalf
- Support before you have completed the course and its project work
- Visa, relocation or overseas placement assistance
Most “big data” is not
The first module is partly about talking students out of the tooling.
Forty million rows is not big data. It fits comfortably on one machine with a sensible index, and moving it onto a cluster buys you complexity, cost and a new category of failure. A great deal of real-world big data infrastructure exists because somebody wanted to use it rather than because the data required it.
An engineer who can size a problem honestly is worth considerably more than one who reaches for Spark by reflex, so the last project is exactly that judgement.
Shuffles are where the time goes
Spark looks deceptively like ordinary code, which is the trap.
A join or a group-by can move the entire dataset across the network, and the difference between a job that takes four minutes and one that takes four hours is usually a shuffle nobody looked at. The second project requires opening the Spark UI, finding the expensive stage and fixing it, with runtimes recorded.
Pipelines have to be safe to rerun
Every pipeline fails eventually, usually at three in the morning.
What matters then is whether running it again fixes things or doubles the data. Idempotency, layered storage and data quality checks that fail loudly rather than quietly passing bad records downstream — these are the difference between a pipeline and a liability, and they get a full module.
Concepts, not the tool of the month
The specific tools in this space change every few years.
Distributed storage, partitioning, columnar formats, shuffles, exactly-once semantics — these do not. The course spends its time there, so that when your employer uses something this syllabus does not name, you already understand what it is doing.
Questions
Big Data with Hadoop and Spark — frequently asked questions
Do I need Java or Scala for this?
The course uses PySpark, so Python is sufficient and is what most teams write now. Some Spark internals and error messages surface from the JVM, which the course explains rather than hides. If you already know Java, that helps with reading stack traces but it is not required.
Is Hadoop obsolete?
MapReduce as a programming model largely is, and the course treats it historically rather than as something you will write. HDFS and YARN remain in a great many existing installations, and cloud object storage has taken over for new work. The concepts — distributed storage, partitioning, shuffles — carry across all of it, which is why they are taught rather than the tool of the month.
Why is there a whole project about deciding not to use big data tools?
Because most datasets called big are not, and a cluster brings real cost and complexity. An engineer who can say "this is forty million rows, put it in PostgreSQL with an index" is more valuable than one who reaches for Spark by reflex. Knowing the boundary is a senior skill and it is teachable.
Do I need my own cluster to practise?
No. Practice runs on a small multi-node setup provided for the course, and Spark also runs locally for most exercises. What matters for learning is the shape of the problem — partitions, shuffles, failures — and those are visible without renting a hundred machines.
Enquire about Big Data with Hadoop and Spark
Three details is all we need. A course advisor will call you back.
Related courses
- Data Science
Statistics, Python, SQL and machine learning taught as one connected discipline, with the emphasis on framing a problem correctly and knowing when a result is not real.
- SQL and Databases
SQL taught against a database big enough that a bad query is noticeably slow, because toy datasets hide performance entirely and performance is half of what the job tests.
- Python Programming
Python taught as a working tool rather than a syntax tour — you finish able to read unfamiliar code, automate real tasks, call APIs and write tests that catch your own mistakes.
- AWS Cloud Computing
AWS taught by building and breaking infrastructure rather than by memorising service names, ending with an architecture you have deployed, secured, costed and can defend.
Next step
Talk to a course advisor
Tell us what you want to learn and we will help you pick the right course, batch and mode.