Azure Databricks Course

Lesson 1 • Notebooks and Collaboration

Introduces Databricks notebooks as the primary development interface for code and documentation. Covers multi-language support and real-time collaboration features.

Lesson 2 • Clusters: Types and Configuration

Explains all-purpose and job clusters, their configuration options, and cost implications. Students create and manage clusters to run workloads efficiently.

Lesson 3 • Databricks CLI and REST API Basics

Introduces programmatic access to Databricks through the CLI and REST API for automation. Connects workspace management to scripting and DevOps workflows.

Lesson 4 • What Is Azure Databricks

Covers the origin, purpose, and key capabilities of Azure Databricks as a unified analytics platform. Positions the tool within the broader Azure ecosystem.

Lesson 1 • Spark Performance Fundamentals

Introduces partitioning, caching, and broadcast joins as primary performance levers. Students apply these techniques to reduce job execution time.

Lesson 2 • Spark Architecture and Execution Model

Explains drivers, executors, tasks, and stages to demystify how Spark distributes work. Provides the mental model needed to write and debug efficient Spark jobs.

Lesson 3 • DataFrames and the Spark API

Teaches the DataFrame API for structured data manipulation using Python and SQL. Covers transformations, actions, and lazy evaluation as core Spark concepts.

Lesson 4 • Spark SQL and Temporary Views

Enables SQL-based data querying within Spark using temporary views and catalog tables. Bridges SQL skills with the Spark execution engine.

Lesson 1 • Delta Table Optimisation

Covers OPTIMIZE, Z-ordering, and VACUUM commands to maintain Delta table performance. Students apply maintenance routines to production-grade tables.

Lesson 2 • Creating and Managing Delta Tables

Covers creating Delta tables from DataFrames and SQL, including managed and external tables. Students perform inserts, updates, and deletes on Delta tables.

Lesson 3 • Merge, Upsert, and CDC Patterns

Teaches the MERGE INTO command for upsert logic and change data capture patterns. Connects Delta capabilities to real-world incremental data ingestion needs.

Lesson 4 • Time Travel and Versioning

Demonstrates querying historical table versions and restoring prior states using Delta time travel. Enables audit, rollback, and reproducibility workflows.

Lesson 1 • Batch Ingestion Patterns

Covers full and incremental batch ingestion strategies from cloud storage and databases. Students implement watermark-based and partition-based incremental loads.

Lesson 2 • Orchestrating Pipelines with Databricks Workflows

Covers building multi-task workflows using Databricks Jobs to schedule and orchestrate pipelines. Students configure dependencies, retries, and alerting.

Lesson 3 • Delta Live Tables for Declarative Pipelines

Introduces Delta Live Tables (DLT) as a declarative framework for building reliable ETL pipelines. Students define streaming and batch tables with quality expectations.

Lesson 4 • Transformation Logic and Data Quality

Teaches applying business transformation rules and enforcing data quality constraints in pipelines. Students use Spark functions and Delta constraints to validate data.

Lesson 1 • Structured Streaming Fundamentals

Explains the continuous query model, micro-batch processing, and the streaming DataFrame API. Establishes the conceptual foundation for all streaming labs.

Lesson 2 • Stateful Streaming and Windowing

Teaches windowed aggregations, watermarking, and stateful operations for time-based analytics. Enables students to compute rolling metrics over event streams.

Lesson 3 • Writing Streaming Output to Delta Lake

Demonstrates writing streaming query results to Delta tables with checkpointing and fault tolerance. Connects streaming processing to durable, queryable storage.

Lesson 4 • Reading from Streaming Sources

Covers connecting to Event Hubs, Kafka, and file-based streaming sources in Databricks. Students configure source options and manage offsets.

Lesson 1 • Data Lineage and Auditing

Demonstrates capturing column-level lineage and audit logs through Unity Catalog. Enables students to trace data origin and access history for compliance.

Lesson 2 • Access Control and Permissions

Covers granting and revoking privileges on catalogs, schemas, tables, and views using SQL. Students implement least-privilege access patterns for data assets.

Lesson 3 • Data Discovery with the Databricks Catalog

Covers tagging, searching, and documenting data assets using the Databricks data catalog UI. Improves data discoverability and self-service analytics across teams.

Lesson 4 • Unity Catalog Architecture

Explains the three-level namespace, metastore, and how Unity Catalog unifies governance across workspaces. Provides the structural foundation for all governance configurations.

Lesson 1 • Training Models with Spark and scikit-learn

Covers training single-node and distributed models using scikit-learn and Spark MLlib. Students apply cross-validation and hyperparameter tuning at scale.

Lesson 2 • Experiment Tracking with MLflow

Teaches logging parameters, metrics, and artefacts with MLflow Tracking for reproducible experiments. Students compare runs and identify the best-performing model.

Lesson 3 • MLflow Model Registry and Lifecycle

Covers registering models, managing versions, and transitioning models through staging and production. Implements a governed model promotion workflow.

Lesson 4 • Model Serving and Batch Inference

Demonstrates deploying models as REST endpoints and running batch inference on Delta tables. Connects trained models to downstream data and application consumers.

Lesson 1 • Security and Network Configuration

Teaches securing Databricks with private endpoints, VNet injection, and secrets management. Students apply defence-in-depth principles to the platform.

Lesson 2 • Disaster Recovery and High Availability

Covers workspace backup strategies, multi-region deployments, and recovery runbooks. Students design resilient architectures for business-critical workloads.

Lesson 3 • Monitoring, Logging, and Alerting

Demonstrates integrating Databricks logs with Azure Monitor and setting up proactive alerts. Students build observability into production pipelines.

Lesson 4 • CI/CD for Databricks Workloads

Covers integrating Databricks with Git providers and building automated test and deploy pipelines. Students implement a full CI/CD workflow for notebooks and jobs.