Data Lake Architecture

A data warehouse (Teradata, Oracle, Redshift 2012) stores structured tables.
Loading semi-structured JSON from application logs requires:
  - Flattening the JSON into tabular columns (an ETL step)
  - Defining a schema before loading
  - Storing in the warehouse's proprietary columnar format

For unstructured data (images, audio, ML model outputs):
Warehouses simply could not store them at all.
You needed a separate system regardless.

Application logs at Amazon in 2013: 10 TB/day of JSON event logs
Cost to store in Teradata: $50,000/TB × 10 TB × 365 = $182M/year
Cost to store on HDFS (Hadoop): ~$1,000/TB = $3.65M/year
Cost to store on S3 (2015): $23/TB = $83,950/year

LIMITATION 2: Schema-on-write inflexibility
  Warehouse schema must be defined before loading. A new event type requires:
    - Schema design (hours to days)
    - Table creation and ALTER TABLE (minutes, but blocking)
    - ETL pipeline update (days of engineering work)
    - Schema approval from DBA (days to weeks at large companies)

  At a fast-growing startup: 5 new event types per week.
  Result: data arrives 2 weeks after the event type is shipped.
  Business: "We can't analyse this event — the data team hasn't added it yet."
  Data lake answer: just drop the JSON file in the landing zone.
  No schema required at write time. Define it when you query.

LIMITATION 3: Coupled compute and storage scaling
  Traditional warehouse: compute and storage are the same hardware.
  To add more storage: buy more nodes (also adds unwanted compute).
  To add more compute: buy more nodes (also adds unwanted storage).

  At Airbnb 2011: data growing 3× per year, compute needed for batch jobs.
  Adding storage for data growth forced adding compute they did not need.
  Data lake answer: S3 for storage (infinite, pay per GB).
  Spark cluster for compute (spin up/down, pay per hour).
  Scale each independently. Pay only for what you use.

THESE THREE PROBLEMS created the Hadoop + HDFS ecosystem (2009-2014)
and then the cloud object storage + Spark ecosystem (2014-present).

GENERATION 1: HADOOP + HDFS (2009-2016)
  Storage: HDFS — commodity hardware, on-premise, files replicated 3×
  Compute: MapReduce → Hive → Spark (2014 onward)
  Format:  Text files (CSV, JSON), then Parquet and ORC
  Problems: HDFS and compute tightly coupled, operational complexity
    (ZooKeeper, NameNode HA, YARN tuning), S3 cheaper by 2015
    ($23/TB vs $100+/TB), cluster always running even when idle

GENERATION 2: CLOUD OBJECT STORAGE + SPARK (2015-2022)
  Storage: S3 (AWS), ADLS Gen2 (Azure), GCS (Google Cloud)
    - Infinitely scalable, 99.999999999% (11 9s) durability, $23/TB/month
  Compute: Spark on EMR / Databricks / HDInsight (ephemeral clusters)
    - Spin up cluster for a job, terminate when done — no idle cost
  Format:  Parquet, Avro, ORC (columnar, compressed, splittable)
  Problems: no ACID transactions (concurrent writes corrupt partitions),
    no row-level UPDATE/DELETE, no schema enforcement, "data swamp"

GENERATION 3: OPEN TABLE FORMATS + LAKEHOUSE (2020-present)
  Storage: S3 / ADLS / GCS (same as Gen 2)
  Format:  Parquet files + transaction log layer
    - Delta Lake (Databricks): _delta_log/ JSON transaction log
    - Apache Iceberg (Netflix): metadata tree + manifest files
    - Apache Hudi (Uber): timeline-based record-level upserts
  Compute: Spark, Trino, Flink, Athena, Snowflake (via external tables)

  What open table formats add:
    - ACID transactions: concurrent writes without corruption
    - Row-level UPDATE and DELETE: not just partition-level operations
    - Time travel: query data as of a specific timestamp or version
    - Schema evolution: add/rename columns without rewriting data
    - Partition evolution: change partition strategy without rewriting
    - Optimistic concurrency: multiple writers, serialisable isolation

DATA LAKE ZONES (S3 bucket organisation for FreshCart):

s3://freshcart-data-lake/
├── landing/              ← ZONE 1: Landing / Raw Ingestion
│   ├── stripe/
│   │   └── payments_20260317.json       (as received from API)
│   ├── shipfast/
│   │   └── deliveries_week_20260317.csv (as received from vendor SFTP)
│   └── postgres_cdc/
│       └── orders/
│           └── 2026/03/17/              (raw CDC events from Debezium)
│
├── bronze/               ← ZONE 2: Bronze / Raw Standardised
│   ├── payments/
│   │   └── year=2026/month=03/day=17/   (Parquet, Hive-partitioned)
│   ├── deliveries/
│   │   └── year=2026/month=03/week=11/
│   └── orders_cdc/
│       └── date=2026-03-17/
│
├── silver/               ← ZONE 3: Silver / Cleaned and Typed
│   ├── orders/
│   │   └── date=2026-03-17/             (validated, typed, deduped)
│   ├── customers/                       (cleaned, PII masked)
│   └── payments/                        (reconciled with bank data)
│
└── gold/                 ← ZONE 4: Gold / Business-Ready
    ├── daily_revenue/
    │   └── date=2026-03-17/
    ├── customer_ltv/
    │   └── date=2026-03-17/
    └── store_performance/
        └── date=2026-03-17/

ZONE 1 — LANDING (Raw Ingestion)
  Purpose:     Exact copy of data as received from source
  Transform:   None — file written exactly as received
  Format:      Whatever the source sends (JSON, CSV, XML)
  Retention:   30-90 days (after Bronze conversion, can expire)
  Access:      Pipeline engineers only
  Key rule:    IMMUTABLE — never modify, never delete during retention

ZONE 2 — BRONZE (Raw Standardised)
  Purpose:     Landed data converted to standard format (Parquet)
               with minimal transformation — types coerced for load compatibility
  Transform:   Format conversion, Hive partitioning, add ingestion metadata
  Format:      Parquet + Snappy/ZSTD compression, Hive-partitioned
  Retention:   1-3 years (raw data for reprocessing)
  Access:      Pipeline engineers, data scientists (raw exploration)
  Key rule:    APPEND ONLY via CDC/incremental — preserve all history

ZONE 3 — SILVER (Cleaned and Typed)
  Purpose:     Trusted, clean, validated data for analytical use
  Transform:   Type casting, deduplication, validation, NULL handling,
               normalisation, PII masking, business rule application
  Format:      Delta Lake / Iceberg (ACID, time travel) partitioned by date
  Retention:   2-5 years
  Access:      Data engineers, analysts, BI tools, ML engineers
  Key rule:    IDEMPOTENT UPDATES via MERGE — not append-only

ZONE 4 — GOLD (Business-Ready Aggregates)
  Purpose:     Pre-computed metrics and aggregates for dashboards and APIs
  Transform:   Aggregation, joining, business metric computation
  Format:      Delta Lake, typically small (GB not TB)
  Retention:   Long-term (1+ years)
  Access:      Analysts, BI tools, downstream APIs
  Key rule:    REBUILT regularly from Silver — not primary source of truth

# S3 bucket policy zones — principle of least privilege
# Landing zone: write by ingestion pipelines only, read by bronze conversion
# Bronze zone: write by format conversion, read by silver pipelines and data scientists
# Silver zone: write by transformation pipelines, read by analysts and BI tools
# Gold zone:   write by aggregation pipelines, read by everyone

# AWS IAM role structure:
# role/pipeline-ingestion: s3:PutObject on landing/*
# role/pipeline-bronze:    s3:GetObject on landing/*, s3:PutObject on bronze/*
# role/pipeline-silver:    s3:GetObject on bronze/*, s3:PutObject on silver/*
# role/pipeline-gold:      s3:GetObject on silver/*, s3:PutObject on gold/*
# role/analyst:            s3:GetObject on silver/*, s3:GetObject on gold/*
#                          (NO access to landing — raw PII)
# role/data-scientist:     s3:GetObject on bronze/*, silver/*, gold/*
#                          (bronze for raw ML training data)

# CRITICAL: landing and bronze zones contain raw PII (emails, phone numbers)
# Analysts and BI tools must NEVER have direct access to landing/bronze.
# Only masked/anonymised Silver and Gold are accessible to analysts.

# Azure ADLS Gen2 equivalent:
# Use POSIX-style ACLs on container directories:
# landing/: execute for pipeline-bronze service principal only
# bronze/:  read for data-scientists, execute for pipeline-silver
# silver/:  read for analysts, BI service principals
# gold/:    read for all authenticated users in the tenant

# Snowflake (for Silver and Gold served via warehouse):
# GRANT SELECT ON SCHEMA silver TO ROLE analyst;
# GRANT SELECT ON SCHEMA gold   TO ROLE analyst;
# REVOKE SELECT ON SCHEMA silver.customers_raw FROM ROLE analyst;
# (raw PII tables blocked even within Silver)

S3 OBJECT MODEL:
  Everything is an object. An object has:
    - Key:        the "path" — "bronze/orders/date=2026-03-17/part-001.parquet"
    - Value:      the bytes of the file
    - Metadata:   content-type, user-defined key-value pairs, system metadata
    - ETag:       MD5 hash of the content (used for integrity checking)
    - Version ID: if versioning is enabled

  There are no real directories — the key is just a string.
  "Directories" are a UI fiction — a key ending in / is a zero-byte object.
  s3://bucket/bronze/orders/ is just a prefix filter on keys, not a real dir.

  IMPLICATION FOR LISTING:
    ListObjectsV2 returns all keys matching a prefix.
    s3://bucket/bronze/orders/ with 1,000,000 objects:
      → 1,000 API calls (1,000 objects per page)
      → Athena/Spark must list all keys to find relevant partitions
      → Use Hive-style partitioning to reduce list scope

S3 CONSISTENCY MODEL (since Dec 2020):
  S3 provides strong read-after-write consistency for all operations.
  After a PUT: the GET immediately returns the new object. ✓
  After a DELETE: the GET immediately returns 404. ✓
  This was eventually consistent before Dec 2020 — old code may still
  have unnecessary sleep() calls to work around the old model.

S3 PERFORMANCE:
  Request rate limits (per prefix):
    3,500 PUT/COPY/POST/DELETE requests per second
    5,500 GET/HEAD requests per second
  Each unique prefix has its own limits:
    s3://bucket/a/  and  s3://bucket/b/  are SEPARATE rate limits
    Use date-partitioning: orders/date=2026-03-17/ and orders/date=2026-03-18/
    give each day its own rate limit bucket — effectively unlimited

  Multipart upload:
    Objects > 5 MB: use multipart upload (automatic in boto3/s3fs)
    Parts uploaded in parallel → fast for large files
    Minimum part size: 5 MB (except last part), maximum object size: 5 TB

S3 STORAGE CLASSES AND COST (US East, 2026 approximate):
  Standard:            $0.023/GB/month  — frequently accessed
  Standard-IA:         $0.0125/GB/month — infrequently accessed, instant retrieval
  Glacier Instant:     $0.004/GB/month  — archive, instant retrieval
  Glacier Flexible:    $0.0036/GB/month — archive, 3-5 hour retrieval
  Glacier Deep Archive: $0.00099/GB/month — coldest archive, 12+ hours

  Lifecycle policy automatically transitions objects between classes.
  Apply per prefix: landing/ transitions to IA after 7 days, delete after 30.

S3 NAMING BEST PRACTICES:
  Use lowercase: S3 is case-sensitive but tools treat case inconsistently
  Use hyphens not underscores in bucket names: freshcart-data-lake
  Use underscores in object keys for compatibility: date=2026-03-17
  Avoid special characters: only a-z, 0-9, /, -, _, .
  Never put credentials or PII in bucket names or object keys
  (keys appear in logs, access logs, CloudTrail)

HIERARCHICAL NAMESPACE (HNS):
  ADLS Gen2 supports a true hierarchical namespace — directories are real,
  not just key prefixes. This has performance implications:
    - Rename a directory: atomic O(1) operation
      S3: must copy all objects to new prefix, delete originals — O(n) time
    - Delete a directory: O(1) in ADLS, O(n) in S3
    - Directory listing: O(1) in ADLS (real directory), O(n) in S3 (list all keys)
  For Spark jobs that rename output directories (common in Hadoop code):
    ADLS is significantly faster than S3 for this operation.

ACCESS CONTROL:
  ADLS supports POSIX-style ACLs on files and directories:
    - Owner: full rwx, Owning group: configurable rwx, Other: configurable rwx
    - Named user/group ACLs (like Linux setfacl)
  This is finer-grained than S3's bucket policy + IAM combination.
  Azure Active Directory integration: assign ADLS ACLs to AAD groups.

PROTOCOL:
  ADLS Gen2 supports both:
    - Blob storage API (https://account.blob.core.windows.net/container/blob)
    - DFS API (https://account.dfs.core.windows.net/filesystem/path)
  Use DFS API for hierarchical operations (rename, directory delete).
  In Spark: spark.read.parquet('abfss://container@account.dfs.core.windows.net/path')

COST (2026 approximate):
  LRS (Locally Redundant):  $0.018/GB/month  (similar to S3 Standard)
  ZRS (Zone Redundant):     $0.023/GB/month
  GRS (Geo Redundant):      $0.036/GB/month

AUTHENTICATION:
  Service Principal (recommended for pipelines): Client ID + Secret → OAuth token
  Managed Identity (best for Azure VMs / AKS): no credentials needed
  Shared Access Signature (SAS): time-limited URL for temporary external access

GCS (Google Cloud Storage) is similar to S3:
  - Eventually consistent (unlike S3's strong consistency since 2020)
  - Similar object model, bucket-based, gs:// URI scheme
  - Strong integration with BigQuery (native external tables)

TRADITIONAL WAREHOUSE (coupled compute and storage):
  Data stored ON the same machines that query it.
  Teradata: storage and CPU on the same proprietary blades.
  Redshift (original): storage and compute on the same EC2 instances.
  IMPLICATIONS:
    To add 10 TB storage: buy 10 TB of nodes (also adds ~10× compute).
    Peak CPU needs (end-of-month reports): must always have peak nodes running.
    Off-peak (2 AM): same nodes idle, same cost.
    Hardware utilisation: typically 10-30% of capacity.

CLOUD DATA LAKE (decoupled compute and storage):
  Data stored on S3 (storage-only, no compute).
  Query engines (Spark, Athena, Presto) are separate compute resources.
  IMPLICATIONS:
    To add 10 TB storage: write more data to S3. Compute unchanged.
    Peak CPU needs: spin up a large Spark cluster for 4 hours. Terminate after.
    Off-peak: S3 still holds data, zero compute running. Zero compute cost.

COST COMPARISON FOR A 100 TB DATA PLATFORM:
  Storage: 100 TB × $23/TB/month = $2,300/month always-on
  Compute: 20 Spark jobs/day × 10 r5.4xlarge × $1/hr × 1 hr = $200/day
  Total:   $2,300 + ($200 × 30) = $8,300/month

  Traditional warehouse equivalent (Redshift):
  dc2.8xlarge (16 nodes for 100 TB): $52,000/month
  → Cloud lake is ~6× cheaper for the same data volume

THE TRADE-OFF:
  Coupled: faster queries (data near compute, no network hop)
  Decoupled: more flexible, cheaper, scales each dimension independently

  Modern cloud warehouses (Snowflake, BigQuery) use a hybrid approach:
    - Data stored in cloud object storage (S3-equivalent)
    - Compute clusters (virtual warehouses) read from storage
    - Result cache and local disk cache reduce the network penalty
    - Effectively decoupled but with intelligent caching to preserve speed

SPARK ON S3 — THE NETWORK PERFORMANCE GAP:
  Spark reading 1 TB from local HDFS: ~100 GB/s aggregate throughput
  Spark reading 1 TB from S3: ~10 GB/s per cluster (network bottleneck)
  → S3 reads are ~10× slower than HDFS for the same cluster size

  MITIGATION:
    1. Use Parquet with predicate pushdown — read only needed columns/rows
    2. Use columnar formats — minimise bytes transferred per query
    3. Partition data — read only relevant partitions
    4. Use Delta Lake data skipping — skip files with no matching rows
    5. S3 Express One Zone (new, 2023): 10× lower latency than standard S3
    6. Use Snowflake/BigQuery for interactive queries (they cache aggressively)
       Use Spark for batch processing where latency is acceptable

DELTA LAKE TABLE STRUCTURE:
  s3://freshcart-lake/silver/orders/
  ├── _delta_log/
  │   ├── 00000000000000000000.json   ← commit 0 (table creation)
  │   ├── 00000000000000000001.json   ← commit 1 (first write)
  │   ├── 00000000000000000042.json   ← commit 42 (latest)
  │   └── 00000000000000000010.checkpoint.parquet  ← periodic checkpoint
  ├── date=2026-03-15/
  │   └── part-00001.parquet
  ├── date=2026-03-16/
  │   └── part-00001.parquet
  └── date=2026-03-17/
      ├── part-00001.parquet           ← written in commit 41
      └── part-00002.parquet           ← written in commit 42

TRANSACTION LOG ENTRY (00000000000000000042.json):
  {
    "commitInfo": {
      "timestamp": 1710706472847,
      "operation": "MERGE",
      "operationParameters": {"predicate": "target.order_id = source.order_id"}
    }
  }
  {
    "add": {
      "path": "date=2026-03-17/part-00002.parquet",
      "partitionValues": {"date": "2026-03-17"},
      "size": 52428800,
      "stats": {"numRecords": 50000, "minValues": {"order_id": 9200000},
                "maxValues": {"order_id": 9250000}}
    }
  }
  {
    "remove": {
      "path": "date=2026-03-17/part-00001.parquet",
      "deletionTimestamp": 1710706472000
    }
  }

HOW DELTA ACHIEVES ACID:
  Atomicity: a write either commits a new JSON log entry (visible)
             or does not (Parquet files written but not referenced)
  Consistency: schema validation at commit time
  Isolation:  optimistic concurrency — concurrent writes detected via
              log entry conflicts, one writer succeeds, other retries
  Durability: S3 11-nines durability — once PUT, object is durable

TIME TRAVEL:
  Every commit in the log represents a version of the table.
  Read version 40 (before commit 41 and 42):
    spark.read.format('delta').option('versionAsOf', 40).load(path)
    spark.read.format('delta').option('timestampAsOf', '2026-03-16').load(path)
  Useful for: debugging, auditing, recovering from bad writes
  Retention: versions kept for 30 days by default (configurable)

DELTA OPERATIONS:
  OPTIMIZE:   compact small files into larger ones
  VACUUM:     remove files no longer referenced (respects retention)
  Z-ORDER:    co-locate related data by a column (improves skip rate)
  RESTORE:    revert table to a previous version

-- Z-ORDER example (Databricks / Spark):
OPTIMIZE silver.orders ZORDER BY (store_id, order_date);
-- Co-locates rows with same store_id and order_date in same files

Before Z-ORDER: queries filtering by store_id + order_date scan most files
After ZORDER BY (store_id, order_date): same queries skip ~80% of files

FreshCart is migrating from an on-premise Hadoop cluster to a cloud data lake on AWS. You are asked to design the architecture. Here is the complete design decision process a senior data engineer follows.

DECISION 1: Storage choice
  Options: S3 Standard vs ADLS Gen2 vs GCS
  FreshCart is AWS-native (EC2, RDS, Lambda already on AWS).
  Decision: S3 — zero integration friction with existing AWS services.
  Bucket structure: one bucket per environment (dev/staging/prod).
  prod: s3://freshcart-data-lake-prod/  (versioning on, default encryption)

DECISION 2: Zone naming and organisation
  landing/ bronze/ silver/ gold/   — clear, industry-standard names
  Within each zone: {entity}/{partition_cols}/
  Partition by date for all time-series data, avoid high-cardinality columns.

DECISION 3: File format
  Landing: as-received (JSON from APIs, CSV from vendors)
  Bronze: Parquet + ZSTD, Hive-partitioned by date
  Silver: Delta Lake (ACID, time travel, row-level updates needed for CDC)
  Gold:   Delta Lake (small tables, need MERGE semantics from dbt incremental)

DECISION 4: Compute engine
  Batch transformation: Spark on Databricks (existing team expertise)
  Interactive queries: Athena (analyst self-service, serverless, no clusters)
  dbt (Silver→Gold transformations): runs against Databricks SQL Warehouse
  ML training: Spark on Databricks + SageMaker for model training

DECISION 5: Catalog
  AWS Glue Data Catalog — integrated with S3, Athena, EMR, Glue ETL.
  Landing and Bronze tables registered manually by pipelines.
  Silver and Gold tables: dbt generates schema + docs → synced to Glue.
  DataHub deployed for business-facing catalog (lineage, ownership).

DECISION 6: Access control
  landing/: only ingestion Lambda + Spark ingestion roles
  bronze/:  ingestion roles + data engineers + data scientists
  silver/:  transformation roles + analysts (read-only) + Athena
  gold/:    transformation roles + analysts + Metabase BI service account
  PII data: blocked at Silver — analysts never see raw emails/phones.

DECISION 7: Cost optimisation
  Lifecycle policies:
    landing/ → Standard-IA after 7 days → delete after 30 days
    bronze/  → Standard-IA after 90 days → Glacier after 365 days
    silver/  → Standard-IA after 180 days → Glacier after 730 days
    gold/    → no lifecycle (small, frequently queried)
  Delta VACUUM: weekly, retain 30 days
  Delta OPTIMIZE: daily, compact files > 7 days old

DECISION 8: Governance baseline (phase 1)
  Data catalog: required for all new Silver/Gold tables before merge
  dbt tests: not_null + unique on all primary keys, freshness checks
  Row count anomaly detection: alert if count deviates > 20% from 7-day avg
  Access audit: CloudTrail → Athena table → weekly access report

RESULTING ARCHITECTURE:
  Sources → Kafka → Debezium → Kafka topics
          → Lambda (API polling) → s3://freshcart-data-lake-prod/landing/

  landing/ → Spark (Databricks) → s3://freshcart-data-lake-prod/bronze/
           (format conversion, minimal typing, Hive partitioning)

  bronze/ → dbt on Databricks SQL → s3://freshcart-data-lake-prod/silver/
           (Delta Lake, ACID, typed, validated, PII-masked)

  silver/ → dbt on Databricks SQL → s3://freshcart-data-lake-prod/gold/
           (Delta Lake, pre-computed metrics, business-ready)

  gold/ → Athena (self-service SQL) + Metabase (dashboards)