IAM and Networking for Data Platforms
Apply least privilege and secure network paths to data-platform identities, storage, processing jobs, and analytics consumers across Azure, AWS, and Google Cloud.
Prerequisites and objective
Know what a principal, role, subnet, route, and firewall rule do. This lesson does not assume a particular provider. You will be able to map those concepts to cloud services and trace a data job's permissions and network path.
Identity concepts across providers
Every data request has an actor, an action, a resource, and a context. A scheduled pipeline should run as an application identity, not an engineer's personal login. Cloud providers express authorization differently, but the design goals are familiar: establish a trusted principal, grant the smallest needed actions, and keep a reviewable audit trail.
Provider Identity / authorization building blocks
AWS IAM users, roles, identity policies, resource policies
Azure Microsoft Entra identities, Azure RBAC, managed identities
Google Cloud Cloud IAM principals, roles, service accounts
Network ideas shared across providers
VPC / virtual network -> subnets -> routes -> firewall rules
Private service access / endpoints -> private path to supported services
DNS + TLS -> name resolution and encrypted application trafficPrefer short-lived or platform-managed credentials where supported. Separate human identities from workload identities, use groups for people, and assign ownership for each permission set. Never place service account keys, access keys, or passwords in notebook source, container images, or logs.
Design a least-privilege access plan
Start from operations, not a broad role name. A cleaner job needs to read a specific raw prefix and write a specific cleaned prefix. It does not need to administer the project, create users, or read unrelated datasets.
Pipeline job: orders-cleaner
Identity: dedicated non-human runtime identity
Read: gs://company-lake/bronze/orders/*
Write: gs://company-lake/silver/orders/*
Deny by default: unrelated buckets, IAM administration, public exposure
Analyst group
Read: approved curated tables / query results
No access: raw payment payloads, job credentials, network administrationApply the same reasoning to a warehouse: a transformation identity may write one schema, while analysts read approved views. Keep sensitive source fields out of general-purpose outputs. Review inherited permissions and resource policies, not just direct grants.
Trace the network path
A pipeline may connect to an object store, a managed database, a message broker, and a warehouse. Map its DNS lookup, route, firewall or security policy, service endpoint, TLS, and response path. “Private subnet” alone does not prove that a dependency is private or reachable.
Virtual networks, subnets, route tables, firewalls, private service endpoints, and DNS integrations appear under different product names. Confirm which path each managed service actually uses, whether the endpoint is supported in the chosen region, and how the service's control plane differs from its data plane.
For each data flow, record:
source principal -> destination resource -> required action
name resolution -> route -> firewall / endpoint -> TLS
audit event -> alert owner -> revocation and recovery procedureFor a private data platform, use approved private connectivity features where supported, limit outbound paths, and verify DNS from the workload environment. Encrypt data in transit, keep storage public access disabled when the architecture allows it, and monitor changes to network and identity policies.
Common mistakes
- Using one all-powerful identity: compromise then affects every workload. Assign identities by job and data boundary.
- Thinking a private IP is a complete security plan: authorization, DNS, routes, firewalls, and service configuration still matter. Test the actual path.
- Confusing role names across clouds: similarly named roles can grant different actions. Inspect permissions and scope, not labels alone.
- Leaving credentials in code or logs: source control and logs persist. Use managed identity or a secret service and rotate exposed credentials.
Practice
- Create a permission matrix for a job that reads bronze and writes silver, plus an analyst who queries a gold view.
- Map AWS IAM roles, Azure managed identities/RBAC, and Google Cloud service accounts/IAM to actor, policy, and resource.
- Draw the network path from a private Spark worker to object storage and a warehouse, including DNS and TLS.
- Describe a safe procedure for investigating a permission-denied error without expanding privileges broadly.
Hint: For each request, write the principal, exact action, target resource, network path, and expected audit event.
🎯 Key Takeaways
- ✓Use separate human and workload identities and grant only the operations a job needs.
- ✓Azure, AWS, and Google Cloud use different IAM products but share principal, policy, scope, and audit concepts.
- ✓Catalog authorization and cloud resource authorization may both be required.
- ✓A secure connection requires correct identity, DNS, routing, firewall rules, endpoints, and TLS.
- ✓Investigate the exact denied action and resource instead of applying broad temporary access.
HELP US IMPROVE
How was this lesson?
Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.
Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.
