Reference

AWS Cheat Sheet

IAM, S3, Glue, EMR, Athena, Redshift, Kinesis, Step Functions and Lake Formation — the AWS data track on one page. Bookmark it.

Account & IAM

Who am I, which regionstay in the dev account

aws sts get-caller-identity
aws configure set region us-east-1
# Account, Arn, UserId — lake, catalog, and warehouse share this region

Trust policywho can assume the role

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {"Service": "glue.amazonaws.com"},
    "Action": "sts:AssumeRole"
  }]
}

Least privilege on the lake

{
  "Version": "2012-10-17",
  "Statement": [
    {"Effect": "Allow", "Action": ["s3:ListBucket"],
     "Resource": "arn:aws:s3:::acme-lake-dev"},
    {"Effect": "Allow", "Action": ["s3:GetObject"],
     "Resource": "arn:aws:s3:::acme-lake-dev/bronze/*"},
    {"Effect": "Allow", "Action": ["s3:PutObject"],
     "Resource": "arn:aws:s3:::acme-lake-dev/silver/*"}
  ]
}

Create the roleno long-lived access keys

aws iam create-role --role-name glue-etl --assume-role-policy-document file://trust-glue.json
aws iam put-role-policy --role-name glue-etl --policy-name lake-rw --policy-document file://lake-rw.json
# Glue, EMR, and Step Functions assume a role. Do not paste keys into a job.

Amazon S3

Bucket and first file

aws s3 mb s3://acme-lake-dev --region us-east-1
aws s3 cp orders.csv s3://acme-lake-dev/bronze/orders/dt=2026-03-15/orders.csv

Medallion prefixespartition key in the path

s3://acme-lake-dev/bronze/orders/dt=2026-03-15/part-000.parquet
s3://acme-lake-dev/silver/orders/dt=2026-03-15/
s3://acme-lake-dev/gold/daily_revenue/
# Athena and Glue skip prefixes the query did not ask for

Lock the bucket down

aws s3api put-bucket-versioning --bucket acme-lake-dev --versioning-configuration Status=Enabled
aws s3api put-public-access-block --bucket acme-lake-dev --public-access-block-configuration '{"BlockPublicAcls":true,"IgnorePublicAcls":true,"BlockPublicPolicy":true,"RestrictPublicBuckets":true}'

Lifecycle on bronze

aws s3api put-bucket-lifecycle-configuration --bucket acme-lake-dev --lifecycle-configuration '{"Rules":[{"ID":"bronze-to-ia","Status":"Enabled","Filter":{"Prefix":"bronze/"},"Transitions":[{"Days":30,"StorageClass":"STANDARD_IA"}]}]}'

AWS Glue

Databasethe catalog, not the data

aws glue create-database --database-input '{"Name":"lake"}'
aws glue get-tables --database-name lake

Crawler over bronze

aws glue create-crawler --name orders-bronze --role arn:aws:iam::123456789012:role/glue-etl --database-name lake --targets '{"S3Targets":[{"Path":"s3://acme-lake-dev/bronze/orders/"}]}'
aws glue start-crawler --name orders-bronze

Start a jobbookmark skips files already read

aws glue start-job-run --job-name bronze-to-silver-orders --arguments '{"--dt":"2026-03-15","--job-bookmark-option":"job-bookmark-enable"}'
aws glue get-job-run --job-name bronze-to-silver-orders --run-id jr_example

Read the catalog

dyf = glueContext.create_dynamic_frame.from_catalog(
    database="lake", table_name="orders")
dyf.toDF().createOrReplaceTempView("orders")

Amazon EMR

Transient Spark clusterheavier than a Glue job

aws emr create-cluster --name spark-orders --release-label emr-7.2.0 --applications Name=Spark --instance-type m5.xlarge --instance-count 3 --use-default-roles
# shut it down when the step finishes — an idle cluster still bills

Spark step

aws emr add-steps --cluster-id j-ABC123 --steps '[{"Type":"Spark","Name":"silver-orders","ActionOnFailure":"CONTINUE","Args":["s3://acme-lake-dev/jobs/silver_orders.py","--dt","2026-03-15"]}]'

See it, then stop it

aws emr list-clusters --active
aws emr describe-cluster --cluster-id j-ABC123 --query 'Cluster.Status.State'
aws emr terminate-clusters --cluster-ids j-ABC123

Amazon Athena

External Parquet tablemetadata only — files stay in S3

CREATE EXTERNAL TABLE lake.orders (
  order_id string,
  amount double
)
PARTITIONED BY (dt string)
STORED AS PARQUET
LOCATION 's3://acme-lake-dev/silver/orders/';

ALTER TABLE lake.orders ADD IF NOT EXISTS
  PARTITION (dt = '2026-03-15')
  LOCATION 's3://acme-lake-dev/silver/orders/dt=2026-03-15/';
-- MSCK REPAIR TABLE lake.orders;  -- walks every prefix; slower

Filter the partition

SELECT dt, SUM(amount) AS revenue
FROM lake.orders
WHERE dt = '2026-03-15'
GROUP BY dt;
-- without dt, Athena scans every prefix under the table

Iceberg table

CREATE TABLE lake.orders_iceberg (
  order_id string,
  amount double,
  dt date
)
PARTITIONED BY (dt)
LOCATION 's3://acme-lake-dev/iceberg/orders/'
TBLPROPERTIES ('table_type' = 'ICEBERG');

Amazon Redshift

Dist key and sort key

CREATE TABLE gold.orders (
  order_id varchar(32) NOT NULL,
  dt date NOT NULL,
  amount decimal(12, 2)
)
DISTKEY (order_id)
SORTKEY (dt);

COPY from S3the cluster role, not your user keys

COPY gold.orders
FROM 's3://acme-lake-dev/gold/orders/'
IAM_ROLE 'arn:aws:iam::123456789012:role/redshift-copy'
FORMAT AS PARQUET;

Spectrum over the Glue catalog

CREATE EXTERNAL SCHEMA spectrum_lake
FROM DATA CATALOG
DATABASE 'lake'
IAM_ROLE 'arn:aws:iam::123456789012:role/redshift-copy'
CREATE EXTERNAL DATABASE IF NOT EXISTS;

Join lake and warehouse

SELECT o.order_id, g.amount
FROM spectrum_lake.orders o
JOIN gold.orders g ON g.order_id = o.order_id
WHERE o.dt = '2026-03-15';

Amazon Kinesis

On-demand stream

aws kinesis create-stream --stream-name orders --stream-mode-details StreamMode=ON_DEMAND
aws kinesis describe-stream-summary --stream-name orders

Put one eventkey chooses the shard

aws kinesis put-record --stream-name orders --partition-key order-1042 --cli-binary-format raw-in-base64-out --data '{"order_id":"1042","amount":19.5}'
# high-cardinality key (order id), not the date — a date piles onto one shard

Firehose into the lake

aws firehose create-delivery-stream --delivery-stream-name orders-to-s3 --delivery-stream-type DirectPut --s3-destination-configuration '{"RoleARN":"arn:aws:iam::123456789012:role/firehose-s3","BucketARN":"arn:aws:s3:::acme-lake-dev","Prefix":"bronze/kinesis/orders/","ErrorOutputPrefix":"bronze/kinesis/errors/"}'

Step Functions

Wait for a Glue job

{
  "StartAt": "Silver",
  "States": {
    "Silver": {
      "Type": "Task",
      "Resource": "arn:aws:states:::glue:startJobRun.sync",
      "Parameters": {"JobName": "bronze-to-silver-orders"},
      "Next": "Done"
    },
    "Done": {"Type": "Succeed"}
  }
}

Start an execution

aws stepfunctions start-execution --state-machine-arn arn:aws:states:us-east-1:123456789012:stateMachine:nightly-lake --input '{"dt":"2026-03-15"}'
aws stepfunctions describe-execution --execution-arn arn:aws:states:us-east-1:123456789012:execution:nightly-lake:example

Retry, then fail

"Retry": [{
  "ErrorEquals": ["States.TaskFailed"],
  "IntervalSeconds": 30,
  "MaxAttempts": 3,
  "BackoffRate": 2
}],
"Catch": [{
  "ErrorEquals": ["States.ALL"],
  "Next": "NotifyFailure"
}]

Lake Formation

aws lakeformation register-resource --resource-arn arn:aws:s3:::acme-lake-dev --use-service-linked-role

Tag, then attachnew tables inherit the tag

aws lakeformation create-lf-tag --tag-key sensitivity --tag-values public,restricted
aws lakeformation add-lf-tags-to-resource --resource '{"Table":{"DatabaseName":"lake","Name":"orders"}}' --lf-tags TagKey=sensitivity,TagValues=public

Grant by tag

aws lakeformation grant-permissions --principal DataLakePrincipalIdentifier=arn:aws:iam::123456789012:role/analyst --resource '{"LFTagPolicy":{"ResourceType":"TABLE","CatalogId":"123456789012","Expression":[{"TagKey":"sensitivity","TagValues":["public"]}]}}' --permissions SELECT

What can this role see

aws lakeformation list-permissions --principal DataLakePrincipalIdentifier=arn:aws:iam::123456789012:role/analyst

Projects to build

The fastest way to make this stick. Build one, then put it in your portfolio.

Beginner

Partitioned orders lake

Land one day of orders as Parquet under bronze/orders/dt=YYYY-MM-DD/, crawl that prefix with Glue, and query only that day in Athena. Block public access on the bucket before the file goes up.

S3 prefixespublic access blockGlue crawlerAthena partitions

Beginner

Glue role, not keys

Create a role Glue can assume that reads bronze and writes silver, then run a small job with it. The script and the job arguments contain no access keys.

IAM trust policyleast privilegeGlue jobs

Intermediate

Nightly medallion

A Step Functions state machine runs a bookmarked Glue job from bronze to silver, then a second job that writes gold, with retry and a failure state. Athena reads one date from gold.

Step FunctionsGlue bookmarksS3 layoutAthena

Intermediate

Clickstream to the lake

Put order events on a Kinesis data stream, deliver them to a dated S3 prefix with Firehose, and query the landed files in Athena. The partition key is an order id, not the calendar day.

Kinesis Data StreamsFirehoseS3Athena

Advanced

Governed warehouse

Register the lake in Lake Formation, tag tables by sensitivity, and grant an analyst SELECT only on public tables. Redshift Spectrum reads those Glue tables, and a transient EMR step runs one heavy join the warehouse should not.

Lake FormationLF-tagsRedshift SpectrumEMR