Build. Don't Just Read.
End-to-end data engineering projects with step-by-step instructions, screenshot guides, and real cloud configurations. Every project builds on the previous one using the same FreshMart dataset.
Projects build on each other — same resources, same data, growing complexity
All Azure projects use the same FreshMart grocery chain dataset and the same Azure resource group. Complete them in order — each project assumes the previous one is done.
Copy a CSV File to Azure Data Lake
ADF · Blob Storage · ADLS Gen2 · Linked Services
Your first Azure pipeline from scratch. Upload a CSV file to cloud storage using Azure Data Factory — the foundation pattern behind every DE pipeline.
Copy Multiple Files Using ForEach Loop
ADF · ForEach · Parameters · Dynamic Expressions
Stop creating 10 separate Copy activities. Use one ForEach loop to move all 10 store CSV files in a single parameterized pipeline run.
Parameterized Pipeline with Run Date
ADF · run_date Parameter · Hive Partitioning · Schedule Trigger
Pass a date at runtime and ADF builds the correct file names automatically. Add a midnight trigger and the pipeline runs every night with zero human involvement.
HTTP Ingestion — Download from a Public URL
ADF · HTTP Linked Service · Public Datasets · Parallel Copy
Pull data directly from a public HTTPS endpoint into ADLS. No manual uploads — ADF fetches from the internet and lands it straight in your data lake.
Organize Files Automatically With Date Stamps
ADF · Get Metadata · If Condition · Delete Activity · File Management
Check if a file exists before copying, add today's date to file names automatically, clean the landing zone, and log missing files — a complete production file management workflow.
Pull Data From a REST API
ADF · HTTP Linked Service · REST API · JSON Format · Real Weather Data
Call a live weather API from ADF, receive JSON responses for three cities, save them to ADLS, and automate the pipeline to run every morning at 6 AM.
New projects added regularly. Start with Project 01 — no prior cloud experience needed.
Start Project 01 — Free →