Big Data Course

Lesson 1 • Data Lifecycle and Governance Basics

Traces data from creation to archival and introduces governance responsibilities. Sets expectations for quality and compliance themes revisited in advanced chapters.

Lesson 2 • The Big Data Ecosystem Overview

Maps the major categories of tools—storage, processing, ingestion, and analytics—and their relationships. Provides a mental model for navigating later chapters.

Lesson 3 • Defining Big Data and Its Properties

Introduces the 5 Vs framework and contrasts big data with conventional datasets. Grounds all subsequent technical content in shared vocabulary.

Lesson 1 • Hadoop Distributed File System Deep Dive

Examines HDFS architecture, configuration, and operational commands. Provides hands-on skills for managing the most widely deployed big data file system.

Lesson 2 • Object Storage for Big Data

Contrasts object storage semantics with file systems and explains cloud-native storage patterns. Prepares students for cloud-based big data architectures covered later.

Lesson 3 • Storage Tiering and Data Lifecycle

Introduces hot, warm, and cold storage tiers and automated lifecycle policies. Enables cost-effective long-term data retention strategies.

Lesson 4 • Data Formats and Compression

Compares columnar, row-based, and semi-structured formats for storage efficiency and query speed. Directly impacts processing performance addressed in the next chapter.

Lesson 1 • MapReduce Programming Model

Explains the map, shuffle, and reduce phases and their execution on a cluster. Establishes the conceptual model that Spark and other engines build upon.

Lesson 2 • Apache Spark Core Concepts

Introduces RDDs, DAG execution, and the Spark runtime architecture. Contrasts Spark's in-memory model with MapReduce's disk-based approach.

Lesson 3 • Spark DataFrames and Spark SQL

Covers the structured API for tabular data manipulation and SQL query execution. Enables analysts and engineers to write expressive, optimised batch queries.

Lesson 4 • Spark Performance Tuning

Addresses partitioning, caching, shuffles, and resource configuration for optimal Spark jobs. Directly reduces job runtime and cluster cost in production environments.

Lesson 1 • Apache Flink for Stateful Stream Processing

Introduces Flink's event-driven model, state backends, and exactly-once guarantees. Provides an alternative engine for latency-sensitive and complex event processing use cases.

Lesson 2 • Streaming Data Concepts and Patterns

Defines event streams, time semantics, and delivery guarantees foundational to all streaming systems. Establishes vocabulary used throughout this chapter and the next.

Lesson 3 • Stream Processing with Spark Structured Streaming

Extends Spark batch skills to continuous query execution on streaming sources. Demonstrates how the same DataFrame API applies to both batch and streaming workloads.

Lesson 4 • Apache Kafka Architecture and Operations

Covers Kafka's broker, topic, partition, and consumer group model for durable message streaming. Kafka underpins most ingestion pipelines built in subsequent sections.

Lesson 1 • Cloud-Native Data Warehousing Concepts

Examines separation of storage and compute, auto-scaling, and concurrency in modern cloud warehouses. Bridges on-premises query engine knowledge to cloud-first architectures.

Lesson 2 • Interactive Queries with Presto and Trino

Introduces the federated, in-memory query architecture of Presto/Trino for sub-minute analytics. Demonstrates querying multiple data sources without data movement.

Lesson 3 • Apache Hive for Batch SQL Workloads

Covers HiveQL, metastore, partitioning, and bucketing for large-scale batch queries. Connects storage format choices from Chapter 2 to query performance outcomes.

Lesson 4 • Query Optimization and Performance Tuning

Applies advanced techniques—statistics, materialised views, and query rewriting—to reduce query latency. Synthesises concepts from all prior sections into practical tuning workflows.

Lesson 1 • Apache Airflow Architecture and Usage

Covers Airflow's scheduler, executor, and operator model for building and deploying DAGs. Provides hands-on skills for the most widely adopted big data orchestration platform.

Lesson 2 • Pipeline Monitoring and Alerting

Establishes observability practices including SLA tracking, alerting, and lineage capture for data pipelines. Ensures students can detect and resolve failures before they impact downstream consumers.

Lesson 3 • Advanced Airflow Patterns

Introduces dynamic DAG generation, custom operators, and sensor-based triggering for complex workflows. Extends basic Airflow skills to enterprise-scale pipeline requirements.

Lesson 4 • Alternative Orchestration Tools

Surveys Prefect, Dagster, and event-driven orchestration as alternatives to Airflow. Equips students to evaluate and select the right tool for varied organisational contexts.

Lesson 1 • Access Control and Data Security

Implements column-level security, row-level filtering, and encryption within a lakehouse. Addresses privacy and compliance obligations for sensitive big data assets.

Lesson 2 • Data Lake vs. Lakehouse vs. Warehouse

Contrasts the three architectural paradigms across consistency, performance, and governance dimensions. Frames the lakehouse as the synthesis that resolves lake and warehouse trade-offs.

Lesson 3 • Open Table Formats: Delta, Iceberg, and Hudi

Covers ACID transactions, time travel, and schema evolution provided by open table formats. Enables reliable, versioned data management on object storage.

Lesson 4 • Data Catalog and Metadata Management

Introduces centralised metadata repositories, automated discovery, and business glossaries. Connects technical metadata to governance and compliance requirements.

Lesson 1 • Distributed Model Training

Introduces Spark MLlib, distributed gradient boosting, and parameter server architectures for large-scale training. Enables training on datasets that exceed single-machine memory.

Lesson 2 • ML Experiment Tracking and Reproducibility

Applies MLflow and similar tools to log parameters, metrics, and artefacts for reproducible experiments. Establishes the audit trail required for model governance.

Lesson 3 • Model Deployment and Serving at Scale

Covers batch inference pipelines, real-time serving endpoints, and A/B testing infrastructure. Connects trained models to production data products.

Lesson 4 • Feature Engineering on Distributed Data

Covers scalable transformations, encoding, and feature store integration using Spark and SQL engines. Bridges data engineering skills to ML model training requirements.