Project 01 — Copy a CSV File to Azure Data Lake

Real World Problem

The Company: FreshMart — a grocery chain with 10 stores across India.

Every day, each store manager exports a file called daily_sales.csv from their billing software and saves it on their computer. That file just sits there. The data team at FreshMart HQ has zero visibility into what is happening across stores. They cannot answer basic questions like:

• Which store sold the most today?
• Which product is running out of stock?
• Is store revenue growing or shrinking week over week?

The root problem: Data is trapped on individual laptops. There is no central place to store it, no way to query it, no way to analyse it.

What we are going to do: Take that CSV file sitting on a laptop → upload it to Azure cloud storage automatically using a pipeline. This is the very first step of every data engineering project in the world — get the data off the source and into a central location.

Concepts You Must Understand First

Before touching Azure, read this section completely — it will make every step feel logical instead of confusing.

What is Azure?

Azure is Microsoft's cloud platform. Instead of buying physical servers and hard drives — you rent them from Microsoft and pay only for what you use.

Think of it like electricity. You do not build a power plant to get electricity at home. You plug into the grid and pay a monthly bill. Azure is the same — you plug into Microsoft's infrastructure and pay for what you consume. For data engineering, Azure gives you cloud storage, compute, pipelines, and dashboards.

What is Azure Data Factory (ADF)?

ADF is a data pipeline orchestration tool. It answers the question: how do I move data from A to B automatically?

Think of ADF like a courier service. You tell it: pick up from this location (source), deliver to that location (destination), run every day at midnight, and alert me if delivery fails. That is exactly what ADF does with data.

In this project we will create 2 Linked Services, 2 Datasets, 1 Activity, and 1 Pipeline.

What is ADLS Gen2?

ADLS stands for Azure Data Lake Storage Generation 2. It is cloud storage optimized for data analytics workloads — think of it as a massive intelligent hard drive in the cloud that can store any file type, never runs out of space, and is directly connected to every Azure analytics service.

Step by Step Overview

Step 1 — Create an Azure Account

Go to https://azure.microsoft.com/en-in/free and click "Start free".

You will get $200 free credit valid for 30 days, 12 months of popular services free, and always-free services including limited ADF and storage. You will need a Microsoft account, a phone number for verification, and a credit card for identity only — you will NOT be charged for free tier usage.

After signing up → go to https://portal.azure.com. You will land on the Azure Portal homepage — your control centre for everything Azure.

Step 2 — Create a Resource Group

A Resource Group is a logical folder that holds all Azure resources belonging to one project. Instead of having ADF, ADLS, and Databricks scattered randomly — you put them all in one Resource Group called rg-freshmart-dev.

In the Azure Portal search bar → type "Resource groups" → click it.

Click "+ Create" (top left button).

Fill in the form exactly as shown:

Click "Review + Create" → then "Create". Wait 5 seconds — you will see a notification: "Resource group created".

Step 3 — Create the Sample CSV File

This represents the daily_sales.csv that FreshMart store managers export from their billing software every day. Open Notepad (Windows) or TextEdit (Mac) and paste exactly this:

order_id,store_id,product_name,category,quantity,unit_price,order_date
ORD001,ST001,Basmati Rice 5kg,Grocery,3,299.00,2024-01-15
ORD002,ST001,Sunflower Oil 1L,Grocery,5,145.00,2024-01-15
ORD003,ST001,Samsung TV 43inch,Electronics,1,32000.00,2024-01-15
ORD004,ST002,Amul Butter 500g,Dairy,8,240.00,2024-01-15
ORD005,ST002,Basmati Rice 5kg,Grocery,2,299.00,2024-01-15
ORD006,ST002,Colgate Toothpaste,Personal Care,10,89.00,2024-01-15
ORD007,ST003,Lays Chips Family Pack,Snacks,15,99.00,2024-01-15
ORD008,ST003,Sony Headphones,Electronics,2,4500.00,2024-01-15
ORD009,ST003,Amul Milk 1L,Dairy,20,62.00,2024-01-15
ORD010,ST001,Dove Soap 100g,Personal Care,6,65.00,2024-01-15

Save the file as daily_sales.csv somewhere easy to find — your Desktop is fine.

Step 4 — Create ADLS Gen2 Storage Account

In the Azure Portal search bar → type "Storage accounts" → click it → click "+ Create".

Basics Tab

Advanced Tab — The Most Important Checkbox

Click the "Advanced" tab → find the section called "Data Lake Storage Gen2" → enable Hierarchical namespace.

Leave all other tabs as default. Click "Review" → "Create". Deployment takes about 30–60 seconds.

Click "Go to resource".

Step 5 — Create Container and Folder Structure

A container is like a top-level folder inside a storage account. We will create one container called raw — this is where all raw, unprocessed data lands. In later projects we will also have processed and curated containers following the Medallion Architecture.

On the storage account page → left sidebar → click "Containers" → "+ Container".

Click on the "raw" container → click "+ Add Directory" → name it sales.

Step 6 — Create Azure Data Factory

In the Azure Portal search bar → type "Data factories" → click it → click "+ Create".

Click the "Git configuration" tab → check "Configure Git later". Git integration is important for production but adds complexity for beginners — we will cover it in a later project.

Click "Review + create" → "Create". Deployment takes 1–2 minutes.

Step 7 — Open ADF Studio

Click "Go to resource" → on the ADF overview page → click "Launch studio". ADF Studio opens in a new browser tab — this is where you will spend 90% of your time.

Step 8 — Create Linked Service for Source (Blob Storage)

ADF lives in the cloud and cannot directly reach into your laptop. The solution: we first upload the CSV to a landing container in the same storage account, then ADF copies it from there to the raw/sales/ destination.

Think of it like a courier — the courier cannot teleport to your home. You drop the package at a collection point, then the courier picks it up and delivers it.

Upload the CSV to a landing container

Go to Azure Portal → Storage account stfreshmartdev → Containers → "+ Container".

Click on the "landing" container → click "Upload" → select your daily_sales.csv from your Desktop → click "Upload".

Create the Linked Service

Go back to ADF Studio → click "Manage" (toolbox icon in left sidebar) → click "Linked services" → click "+ New".

In the search box → type "Azure Blob" → select "Azure Blob Storage" → click "Continue".

Click "Test connection" at the bottom. You should see a green ✅ "Connection successful".

Click "Create".

Step 9 — Create Linked Service for Destination (ADLS Gen2)

Still in Manage → Linked services → click "+ New" → search "Azure Data Lake Storage Gen2" → select it → "Continue".

Click "Test connection" → ✅ green → click "Create".

Step 10 — Create Source Dataset

A Dataset tells ADF which specific file to work with. The Linked Service is how to connect to the storage — the Dataset is what file specifically to read or write.

Click "Author" (pencil icon in left sidebar) → click "+" next to "Datasets" → "New dataset".

Search "Azure Blob Storage" → select it → "Continue". Select format: "DelimitedText" (this means CSV) → "Continue".

Click "OK" → then click the "Preview data" tab at the bottom. You should see all 10 rows of FreshMart sales data — this confirms ADF can read your file.

Click 💾 Save (or Ctrl+S).

Step 11 — Create Destination Dataset

Click "+" next to "Datasets" → "New dataset" → search "Azure Data Lake Storage Gen2" → select it → "Continue". Select format: "DelimitedText" → "Continue".

Click "OK" → 💾 Save.

Step 12 — Create the Pipeline

Click "+" next to "Pipelines" → "New pipeline". A blank canvas opens.

In the Properties panel on the right, set the name and description:

Step 13 — Add and Configure Copy Activity

The Copy Activity does one thing: read data from a source dataset and write it to a sink (destination) dataset. Source = where data comes FROM. Sink = where data goes TO. (Sink is standard data engineering terminology — like a kitchen sink where water flows into.)

In the left activities panel → expand "Move & transform" → drag "Copy data" onto the canvas.

Click the Copy activity box to select it. The bottom panel shows configuration tabs. Set the following in each tab:

General Tab

Source Tab

Sink Tab

Mapping Tab

Click "Mapping" → click "Import schemas". ADF will automatically detect all columns and map them 1:1.

Click 💾 Save.

Step 14 — Debug (Test Run)

Before scheduling or publishing anything, always run a Debug first. Debug runs the pipeline immediately using your current draft — no effect on production.

Click "Debug" in the top toolbar. No parameters for this pipeline — click "OK".

Watch the canvas. The Copy activity will show:

At the bottom → "Output" tab → click the 👓 glasses icon next to the run to see details.

Step 15 — Publish

In ADF, everything you build exists as a draft until published. Debug runs work on drafts. But triggers and scheduled runs only use published pipelines.

Click "Publish all" in the top toolbar. A panel shows everything that will be published — all 5 items we created. Click "Publish".

Step 16 — Verify File in ADLS

Go to Azure Portal → Storage accounts → stfreshmartdev → Containers → raw → sales.

Click on daily_sales.csv → click "Edit" to preview its contents.

Step 17 — Check the Monitor Tab

Go back to ADF Studio → click "Monitor" (bar chart icon in the left sidebar). In production, this is where you check every morning that all pipelines ran successfully overnight.

Click on the pipeline run to see full details.

Resources Created — Summary

Key Concepts Reference

Common Mistakes