Project 03 — Parameterized Pipeline with Run Date

Real World Problem

Let's be honest about what Projects 01 and 02 did and did not solve:

Here is what FreshMart's IT team actually needs:

This is how every production data pipeline in the real world works:

Concepts You Must Understand First

Why Pass run_date as a Parameter?

The most important design decision in this project. Here is the problem with not using a parameter:

Important ADF Limitation — Parameter Defaults Cannot Be Expressions

The fix is simple: use a plain static date as the default value, and let the trigger pass the real dynamic date at runtime.

How Do Dynamic Expressions Work with Dates?

ADF has built-in date formatting functions. These are the ones we use in this project:

The @{ } inside a text string is called string interpolation. ADF evaluates what is inside the braces and inserts the result:

Hive-Style Partitioning — The Folder Structure We Are Building

Good data engineers organize ADLS by date. This makes it easy to find any day's data and lets analytics tools skip folders they do not need — dramatically faster and cheaper to query.

The date=YYYY-MM-DD folder naming convention is the industry standard — called Hive-style partitioning. Databricks, Synapse, and Athena all understand it natively.

What is a Trigger?

Step 1 — Create Date-Based CSV Files

This time the file names include the date: store_ST001_sales_20240115.csv. We create files for two dates (January 15 and 16) so we can test backfill — running the pipeline for different dates without changing anything.

On your Desktop create a folder called freshmart_dated_files. Inside it create two subfolders: 20240115 and 20240116.

Inside 20240115 — create all 10 files. Here are the first two as templates. Follow the same pattern for stores ST003–ST010.

order_id,store_id,product_name,category,quantity,unit_price,order_date
ORD1001,ST001,Basmati Rice 5kg,Grocery,12,299.00,2024-01-15
ORD1002,ST001,Samsung TV 43inch,Electronics,2,32000.00,2024-01-15
ORD1003,ST001,Amul Butter 500g,Dairy,25,240.00,2024-01-15
ORD1004,ST001,Colgate Toothpaste,Personal Care,30,89.00,2024-01-15
ORD1005,ST001,Nike Running Shoes,Apparel,5,4500.00,2024-01-15

order_id,store_id,product_name,category,quantity,unit_price,order_date
ORD2001,ST002,Sunflower Oil 1L,Grocery,18,145.00,2024-01-15
ORD2002,ST002,iPhone 14,Electronics,1,75000.00,2024-01-15
ORD2003,ST002,Amul Milk 1L,Dairy,40,62.00,2024-01-15
ORD2004,ST002,Dove Soap 100g,Personal Care,50,65.00,2024-01-15
ORD2005,ST002,Levis Jeans,Apparel,8,2999.00,2024-01-15

Create stores ST003–ST010 with the same column structure, using their store IDs and order_date = 2024-01-15. Then for the 20240116 folder, duplicate all 10 files changing only the date in the file name, order IDs, and order_date column to 2024-01-16.

Step 2 — Upload Files to Landing Container

Go to Azure Portal → stfreshmartdev → Containers → landing → click the store_sales folder.

Click "+ Add Directory" → name it exactly: date=2024-01-15

Click into date=2024-01-15 → "Upload" → select all 10 files from your 20240115 local folder → "Upload".

Go back to store_sales → create another directory: date=2024-01-16 → upload all 10 files from your 20240116 local folder.

These datasets need two parameters each — one for the date folder, one for the file name. Both values will be passed from the pipeline at runtime.

Step 3 — Create Source Dataset With Two Parameters

In ADF Studio → Author → Datasets → "+" → "New dataset" → "Azure Blob Storage" → "Continue" → "DelimitedText" → "Continue".

Click "OK" → click the "Parameters" tab → "+ New" — add BOTH parameters:

Click the "Connection" tab. Set the three path fields:

For the Directory field: click "Add dynamic content" → in the editor, type the full expression: store_sales/@{dataset().run_date_folder} → click "OK".

For the File field: click "Add dynamic content" → under Parameters → click file_name → click "OK".

Click 💾 Save.

Step 4 — Create Sink Dataset With Two Parameters

Click "+" next to Datasets → "Azure Data Lake Storage Gen2" → "DelimitedText".

Click "OK" → Parameters tab → add the same two parameters: run_date_folder (String) and file_name (String).

Click Connection tab:

Click 💾 Save.

Step 5 — Create New Pipeline

In ADF Studio → Author → "+" next to Pipelines → "New pipeline".

Step 6 — Add the run_date Parameter

Click on empty canvas → Parameters tab at the bottom → "+ New".

Step 7 — Add the store_ids Array Parameter

Still in the Parameters tab → "+ New".

Notice: in Project 02 the array stored full file names like store_ST001_sales.csv. Now we store just the store ID like ST001. The pipeline builds the full file name using run_date. This means the array never needs to change — even as dates change every night.

Step 8 — Add a Pipeline Variable

We need a variable to hold the computed folder name date=2024-01-15. Computing it once in a variable means we can use it in multiple places without repeating the expression.

Click empty canvas → Variables tab → "+ New".

Step 9 — Add a Set Variable Activity

This activity runs first. It takes run_date (e.g. 2024-01-15) and stores date=2024-01-15 in the variable. Every other activity then reads this variable instead of re-computing it.

Left panel → expand "General" → drag "Set variable" onto the canvas.

Click the Set variable activity → configure:

General Tab

Variables Tab (inside the activity)

Click the Variables tab in the bottom properties panel (this is the activity configuration, not the pipeline variables tab).

Click "Add dynamic content" for the Value field → type this expression in the editor:

This works because run_date already comes in as yyyy-MM-dd format — we just prepend date= to it. Simple and clean.

Step 10 — Add ForEach and Connect it to Set Variable

Left panel → "Iteration & conditionals" → drag "ForEach" onto the canvas.

Now connect the two activities: hover over set_run_date_folder → drag the green arrow on its right edge → drop it onto the ForEach. This forces Set Variable to finish before ForEach starts.

Click the ForEach activity → configure:

General Tab

Settings Tab

Step 11 — Add Copy Activity Inside ForEach

Click the "+" button inside the ForEach box → from the inner canvas left panel → drag "Copy data".

Step 12 — Configure Source With Date Expressions

Click the Copy activity → bottom panel:

General Tab

Source Tab

Select ds_src_blob_dated_store_sales. Two Dataset properties fields appear.

For run_date_folder: Click "Add dynamic content" → under Variables → click run_date_folder.

For file_name: Click "Add dynamic content" → type this expression in the editor:

Step 13 — Configure Sink

Click Sink tab → select ds_sink_adls_dated_sales. Two Dataset properties appear — fill them with the exact same expressions as the source:

Click the back arrow to return to the main pipeline canvas.

Step 14 — Validate and Debug (Run for Jan 15)

Click "Validate" → should show no errors. Then click "Debug".

The parameter dialog appears with the defaults pre-filled. Leave run_date as 2024-01-15 and click "OK".

Watch the canvas — Set Variable completes first (green), then ForEach starts and runs 10 iterations 4 at a time.

Verify in ADLS: Azure Portal → stfreshmartdev → Containers → raw → sales.

Step 15 — Test Backfill (Run for Jan 16 Without Changing Anything)

This is where parameters prove their value. Click "Debug" again → change only run_date:

Click "OK". Check ADLS — you now have two date partitions:

This is backfill. If a pipeline fails on any day, rerun it with that date — it fills the missing data without touching any other day's folder.

Step 16 — Create the Schedule Trigger

Go back to the main pipeline canvas → click "Add trigger" in the top toolbar → "New/Edit" → "+ New".

The New trigger panel opens. Fill in:

Click "OK".

Step 17 — Set What the Trigger Passes to the Pipeline

After clicking OK, a "Trigger Run Parameters" dialog appears. This is where you tell the trigger what to send as run_date and store_ids each night.

For run_date: Click "Add dynamic content" and type this expression:

For store_ids: Type the array directly:

Click "OK".

Step 18 — Publish Everything

Click "Publish all". The panel shows all 4 new items — click "Publish".

Step 19 — Manually Trigger a Run Right Now

You do not need to wait until midnight to test the trigger. On the pipeline canvas → "Add trigger" → "Trigger now".

In the Run Parameters dialog, enter a date you have files for:

Click "OK" → go to Monitor → Pipeline runs to watch it execute.

Step 20 — View the Trigger in Monitor

Click Monitor → Trigger runs in the left submenu.

The trigger is now live. Every night at midnight IST it fires automatically, passes today's date as run_date, copies all 10 store files into raw/sales/date=YYYY-MM-DD/, and nobody needs to press anything.

Before and After

All Expressions Used in This Project

What Was Added in Project 03

Key Concepts Reference

Common Mistakes