Project 04 — Download Files From a Public HTTPS URL

🏢 Real World Problem

FreshMart's data team now works with an external supplier — AgriPrice India — a government-backed organization that publishes daily wholesale vegetable and fruit prices across Indian cities.

FreshMart's category managers need this data to:

• Compare what they paid suppliers vs the wholesale market price
• Spot overpriced procurement deals
• Adjust store pricing based on commodity fluctuations

AgriPrice India publishes their data every morning as a CSV file on their public website. Right now, FreshMart's analyst manually opens the browser every morning, downloads the file, and uploads it to Azure. 20 minutes every single day. If the analyst is on leave, it doesn't happen at all.

BEFORE:
  Analyst → opens browser → downloads file → uploads to Azure → 20 min/day

AFTER:
  ADF Pipeline → hits the URL → saves to ADLS → 0 human minutes

🧠 Concepts You Must Understand First

What is an HTTP Linked Service?

In Projects 01–03, our source was always Azure Blob Storage — files sitting inside Azure. But data sources in the real world are everywhere: government portals, supplier servers, financial data providers, open datasets. All accessible via HTTP or HTTPS — the same protocol your browser uses.

An HTTP Linked Service tells ADF:

HTTP Linked Service:   https://people.sc.fsu.edu
                       ↑ the base server address

HTTP Dataset:          /~jburkardt/data/csv/cities.csv
                       ↑ the specific file path on that server

Together they form:    https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv
                       ↑ the full URL ADF fetches

HTTP File Download vs REST API

Public Datasets We Are Using

We will use two stable, public CSVs hosted by Florida State University — available for 10+ years:

URL: https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv

LatD,LatM,LatS,NS,LonD,LonM,LonS,EW,City,State
41,5,59,N,80,39,0,W,Youngstown,OH
42,52,48,N,97,23,23,W,Yankton,SD
...

URL: https://people.sc.fsu.edu/~jburkardt/data/csv/grades.csv

What We Are Building

INTERNET (Public URLs)                ADF Pipeline               ADLS Gen2

https://fsu.edu/.../cities.csv  ──►   HTTP Linked Service  ──►   raw/
                                       Copy Activity              └── external/
https://fsu.edu/.../grades.csv  ──►   Copy Activity                  ├── cities/
                                                                      │   └── cities.csv
                                       pl_download_external_data      └── grades/
                                                                          └── grades.csv

📋 Step by Step Overview

PHASE 1 — Understand the Source URLs (5 min)
  Step 1: Open and inspect the public CSV URLs in your browser

PHASE 2 — Create HTTP Linked Service (15 min)
  Step 2: Create Linked Service for the FSU data server
  Step 3: Test the connection

PHASE 3 — Create Datasets (15 min)
  Step 4: Create parameterized HTTP source dataset
  Step 5: Create ADLS sink dataset with dynamic path

PHASE 4 — Build the Pipeline (30 min)
  Step 6:  Create new pipeline
  Step 7:  Add parameters — source_relative_url, destination_folder, file_name
  Step 8:  Add Copy activity
  Step 9:  Configure HTTP Source
  Step 10: Configure ADLS Sink
  Step 11: Debug and verify
  Step 12: Add second Copy activity for parallel download
  Step 13: Debug parallel run
  Step 14: Publish

Phase 1 — Understand the Source URLs

Step 1 — Open and Inspect the URLs in Your Browser

Before building anything in ADF, always verify the source URL works.

Open your browser and go to:

https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv

Your browser should display or download the CSV. You should see data starting with:

LatD,LatM,LatS,NS,LonD,LonM,LonS,EW,City,State
41,5,59,N,80,39,0,W,Youngstown,OH

Now open the second URL:

https://people.sc.fsu.edu/~jburkardt/data/csv/grades.csv

Notice the split between base URL and file path:

Base URL:  https://people.sc.fsu.edu
File path: /~jburkardt/data/csv/cities.csv
File path: /~jburkardt/data/csv/grades.csv

This separation matters — the Linked Service stores the base URL, the Dataset stores the file path.

Phase 2 — Create HTTP Linked Service

Step 2 — Create the HTTP Linked Service

In ADF Studio → Manage (toolbox icon) → Linked services → "+ New"

In the search box → type "HTTP" → select "HTTP" → click "Continue"

Fill in the form exactly:

Name:                    ls_http_public_data
Description:             HTTP connection to public data sources on the internet
Connect via:             AutoResolveIntegrationRuntime
Base URL:                https://people.sc.fsu.edu
Authentication type:     Anonymous

Step 3 — Test the Connection

Click "Test connection" at the bottom of the form

You should see:

✅ Connection successful

Click "Create"

Phase 3 — Create Datasets

Step 4 — Create Parameterized HTTP Source Dataset

This dataset will be reusable — instead of creating one dataset per URL, we create one dataset with arelative_url parameter. Different pipelines can pass different file paths to the same dataset.

In ADF Studio → Author → Datasets → "+" → "New dataset"

Search "HTTP" → select "HTTP" → click "Continue"

Select format: "DelimitedText" (CSV) → "Continue"

Name:            ds_src_http_csv
Linked service:  ls_http_public_data
Relative URL:    (leave empty — we will make this dynamic)
Request method:  GET
First row as header: ✅ Yes
Import schema:   None

Click "OK"

Now add a parameter. Click "Parameters" tab → "+ New"

Name:    relative_url
Type:    String
Default: (empty)

Click "Connection" tab → click inside "Relative URL" field → click "Add dynamic content"

In the expression editor → click relative_url under Parameters

Click "OK"

Click 💾 Save

Step 5 — Create ADLS Sink Dataset With Dynamic Path

In ADF Studio → Datasets → "+" → "New dataset"

Search "Azure Data Lake Storage Gen2" → select → "Continue"

Select "DelimitedText" → "Continue"

Name:            ds_sink_adls_external
Linked service:  ls_adls_freshmart
File path:       (leave all empty)
First row as header: ✅ Yes
Import schema:   None

Click "OK"

Click "Parameters" tab → add TWO parameters:

Parameter 1:
  Name:    destination_folder
  Type:    String

Parameter 2:
  Name:    file_name
  Type:    String

Click "Connection" tab → configure each field:

Container:

raw

Directory → "Add dynamic content":

File → "Add dynamic content":

raw/
├── sales/          ← internal store sales (Projects 01–03)
└── external/       ← data downloaded from internet (this project)
    ├── cities/
    └── grades/

Click 💾 Save

Phase 4 — Build the Pipeline

Step 6 — Create New Pipeline

In ADF Studio → Author → "+" next to Pipelines → "New pipeline"

Name:        pl_download_external_data
Description: Downloads CSV files from public HTTPS URLs into ADLS raw/external/

Step 7 — Add Pipeline Parameters

Click empty canvas → "Parameters" tab at the bottom → add THREE parameters:

Parameter 1:
  Name:    source_relative_url
  Type:    String
  Default: /~jburkardt/data/csv/cities.csv

Parameter 2:
  Name:    destination_folder
  Type:    String
  Default: cities

Parameter 3:
  Name:    file_name
  Type:    String
  Default: cities.csv

Step 8 — Add Copy Activity

From left activities panel → "Move & transform" → drag "Copy data" onto the canvas

Click the Copy activity → bottom panel → General tab:

Name:        copy_http_to_adls
Description: Downloads file from HTTP source URL and saves to ADLS raw/external/

Step 9 — Configure HTTP Source

Click "Source" tab

Source dataset:  ds_src_http_csv

The Dataset properties section appears with the relative_url field.

Click inside relative_url → "Add dynamic content" → click source_relative_url under Parameters:

Click "OK"

Additional Source settings:
  Request method:         GET
  Additional headers:     (leave empty)
  Pagination rules:       (leave empty)

Step 10 — Configure ADLS Sink

Click "Sink" tab

Sink dataset:  ds_sink_adls_external

Two dataset properties appear. Fill both:

destination_folder → "Add dynamic content":

file_name → "Add dynamic content":

Step 11 — Debug and Verify

Click "Validate" in the top toolbar

Click "Debug" → the parameter dialog appears pre-filled with your defaults:

source_relative_url:  /~jburkardt/data/csv/cities.csv
destination_folder:   cities
file_name:            cities.csv

Click "OK"

Click the 👓 glasses icon next to the run in the Output tab:

Data read:      X bytes
Data written:   X bytes
Files read:     1
Files written:  1
Source:         https://people.sc.fsu.edu/~jburkardt/data/csv/cities.csv
Destination:    raw/external/cities/cities.csv

Verify in ADLS: Azure Portal → Storage → stfreshmartdev → Containers → raw

raw/
├── sales/          ← from Projects 01–03
└── external/       ← NEW — created by this pipeline
    └── cities/
        └── cities.csv  ✅

Step 12 — Download the Second File (grades.csv)

Run the same pipeline again with different parameters — no code changes.

Click "Debug" again → change all three parameters:

source_relative_url:  /~jburkardt/data/csv/grades.csv
destination_folder:   grades
file_name:            grades.csv

Click "OK" → wait for green ✅

raw/external/
├── cities/
│   └── cities.csv   ✅  (from first run)
└── grades/
    └── grades.csv   ✅  (from second run)

Step 13 — Add a Second Copy Activity for Parallel Download

Right now you run the pipeline twice manually. In production you want both files in a single run. Let's redesign the pipeline to download both files simultaneously.

First — update pipeline parameters to handle two files. Click empty canvas → Parameters tab → replace with 6 parameters:

file1_url:          /~jburkardt/data/csv/cities.csv
file1_folder:       cities
file1_name:         cities.csv
file2_url:          /~jburkardt/data/csv/grades.csv
file2_folder:       grades
file2_name:         grades.csv

Rename the existing Copy activity → General tab: copy_file1_http_to_adls

Update its Source dataset property:

Update its Sink dataset properties:

Drag another "Copy data" onto the canvas — place it beside the first — do NOT connect them with an arrow

Click the second Copy activity → General tab:

Name:        copy_file2_http_to_adls
Description: Downloads grades.csv from HTTP source

Source tab → ds_src_http_csv → relative_url:

Sink tab → ds_sink_adls_external

Step 14 — Debug the Parallel Pipeline and Publish

Click "Validate" → should pass with no errors

Click "Debug" → all 6 parameters show their defaults → click "OK"

Both ran in parallel. Total time ≈ the slower activity, not the sum:

copy_file1_http_to_adls:  ~5 seconds
copy_file2_http_to_adls:  ~4 seconds
Total pipeline time:       ~5 seconds (parallel!)

If sequential:             ~9 seconds
Time saved:                ~4 seconds

raw/external/
├── cities/
│   └── cities.csv   ✅
└── grades/
    └── grades.csv   ✅

Click "Publish all"

Publishing:
  pl_download_external_data   (new)
  ds_src_http_csv             (new)
  ds_sink_adls_external       (new)
  ls_http_public_data         (new)

🎯 What You Built — Summary

BEFORE:
  Analyst manually downloads file from browser every morning
  Uploads to Azure manually — 20 minutes per day
  If analyst is on leave — data is missing

AFTER:
  ADF fetches directly from URL — zero human involvement
  Two files downloaded in parallel in a single pipeline run
  Same pipeline handles any public CSV by changing parameters
  raw/external/ folder organized for all internet-sourced data

Full ADF Resource Inventory After Project 04

🧠 Key Concepts to Remember

⚠️ Common Mistakes in This Project

🚀 What's Coming in Project 05

So far our files land in ADLS with generic names like cities.csv. In production, you need to know when a file was downloaded — was it today's data or last week's?

In Project 05 you will learn to:

• Automatically add today's date to downloaded file names: cities_20240115.csv
• Organize files into date folders automatically: raw/external/cities/date=2024-01-15/
• Rename and move files after copying using the Get Metadata and Delete activities
• Combine everything from Projects 01–04 into one clean, organized pipeline