Airflow Course

Lesson 1 • Installing and Configuring Airflow

Guides students through local installation using pip and Docker Compose. Produces a running Airflow environment ready for hands-on practice.

Lesson 2 • Workflow Orchestration Fundamentals

Defines workflow orchestration and its importance in data pipelines. Establishes the problem space Airflow solves before introducing the tool itself.

Lesson 3 • Airflow Architecture Overview

Covers the major components: scheduler, webserver, executor, and metadata database. Connects each component's role to overall pipeline execution.

Lesson 1 • Defining Task Dependencies

Teaches bitshift operators and set_upstream/set_downstream methods to wire tasks. Students build multi-step pipelines with correct execution order.

Lesson 2 • Taskflow API and Decorators

Introduces the @dag and @task decorators for cleaner, Pythonic DAG authoring. Compares Taskflow style with classic operator style.

Lesson 3 • Core Operator Types

Introduces PythonOperator, BashOperator, and EmptyOperator as foundational task types. Students select the right operator for common use cases.

Lesson 4 • Understanding DAGs

Explains Directed Acyclic Graphs as the core abstraction for pipelines. Covers DAG properties, scheduling parameters, and file discovery.

Lesson 1 • Trigger Rules and Task Behavior

Covers all trigger_rule options controlling when a task runs relative to upstream states. Students handle partial failures and conditional execution.

Lesson 2 • Sensors and Event-Driven Execution

Introduces sensors as tasks that wait for external conditions before proceeding. Students implement file, time, and custom sensors in pipelines.

Lesson 3 • Manual Triggers and DAG Runs

Demonstrates triggering DAGs via UI, CLI, and REST API with custom configurations. Enables on-demand execution alongside scheduled runs.

Lesson 1 • Database Operators and SQL Execution

Demonstrates SQLExecuteQueryOperator and database-specific operators for running SQL. Students build data ingestion and transformation tasks.

Lesson 2 • Using Hooks to Access Systems

Covers hooks as reusable interfaces to external services built on top of connections. Students use database and HTTP hooks inside operators.

Lesson 3 • Cloud Provider Integrations

Surveys provider packages for major cloud platforms and their key operators. Students install providers and wire cloud storage and compute tasks.

Lesson 1 • Airflow Variables

Covers Variables as key-value configuration stores accessible across DAGs. Students manage Variables via UI, CLI, and JSON import/export.

Lesson 2 • XCom Fundamentals

Explains XCom as the mechanism for passing small data between tasks via the metadata database. Covers push, pull, and automatic Taskflow XComs.

Lesson 3 • Dynamic Values and Templating

Introduces Jinja templating in operator fields to inject execution context at runtime. Students use macros and custom filters in task parameters.

Lesson 1 • Concurrency and Parallelism Settings

Explains parallelism, max_active_runs, and max_active_tasks parameters at global and DAG levels. Students prevent resource contention in shared environments.

Lesson 2 • Scaling Airflow Infrastructure

Addresses horizontal scaling of schedulers, workers, and the metadata database. Students plan capacity for growing pipeline workloads.

Lesson 3 • Executor Types and Trade-offs

Compares SequentialExecutor, LocalExecutor, CeleryExecutor, and KubernetesExecutor. Students select the right executor for their infrastructure.

Lesson 4 • Queues and Worker Routing

Covers queue assignment in Celery and Kubernetes executors to route tasks to specific workers. Students implement multi-queue architectures.

Lesson 1 • Task State Management

Explains all task instance states and how to manually clear, mark, or restart tasks. Students recover pipelines without full re-runs.

Lesson 2 • Callbacks and Alerting

Introduces on_failure_callback, on_success_callback, and SLA miss callbacks for event-driven alerts. Students wire email and webhook notifications.

Lesson 3 • Logging and Observability

Covers Airflow's logging architecture, remote log storage, and integration with monitoring platforms. Students gain full visibility into pipeline health.

Lesson 1 • SubDAGs and Task Groups

Compares deprecated SubDAGs with modern TaskGroups for visual and logical grouping. Students refactor complex DAGs using TaskGroups.

Lesson 2 • DAG Versioning and Code Organization

Addresses folder structure, shared utilities, and version control strategies for DAG repositories. Students maintain clean, collaborative codebases.

Lesson 3 • Dynamic DAG Generation

Teaches generating multiple DAGs or tasks programmatically from configuration or database records. Students eliminate repetitive DAG code.

Lesson 4 • Performance Optimisation Techniques

Identifies scheduler performance bottlenecks and applies tuning strategies at DAG and infrastructure levels. Students reduce pipeline latency.