GCP Vertex AI — Pipelines and AutoML
Vertex AI Training, Pipelines, Feature Store, Model Registry, and online prediction endpoints. The GCP-native ML platform with best-in-class BigQuery integration.
Vertex AI is what happens when Google builds a managed ML platform on top of the infrastructure that runs Search, Maps, YouTube, and Gmail. BigQuery is the native data warehouse. TPUs are first-class compute. The Feature Store is the most production-ready managed one available.
Vertex AI is Google's unified ML platform — launched in 2021 by merging AI Platform, AutoML, and several other GCP ML services into a single product. It is the platform of choice at companies that run on GCP: Uber, Braintree, Reddit, Gopuff, and many analytics-heavy companies. Its distinguishing strengths over Azure ML and SageMaker: BigQuery integration is native and seamless (query data directly from training scripts without copying to object storage), the Vertex AI Feature Store is the most complete managed feature store across all three clouds, and TPU access is unique to GCP.
The Vertex AI SDK (google-cloud-aiplatform) is the Python interface. Like SageMaker, GCP services work together: Cloud Storage (equivalent of S3) holds data and artifacts, Artifact Registry (equivalent of ECR) holds Docker images, Cloud Logging holds job logs, and IAM manages permissions via service accounts. The mental model from the last two modules transfers directly — different names, same concepts.
Azure ML is a hotel, SageMaker is a city block, and Vertex AI is a university campus. Everything is Google-designed and integrated — the cafeteria (BigQuery) is connected to the research labs (Vertex Training) by a covered walkway, the library (Feature Store) is shared by all departments, and the campus bus (Vertex Pipelines) runs on a fixed schedule connecting everything. Off-campus services exist but the campus is designed to keep you within the Google ecosystem, and for data-heavy ML work the integration genuinely pays off.
The most important Vertex AI concept: everything is a resource with a resource name in the format projects/{project}/locations/{region}/resourceType/{id}. Every API call uses this format. Every log entry references it. Once you internalise this pattern, navigating Vertex AI becomes predictable — you always know where to look for anything.
GCS, IAM service accounts, Artifact Registry, and Vertex — four services that power every job
CustomTrainingJob — submit any Python script to Vertex managed compute
A Vertex AI CustomTrainingJob is equivalent to an AML Command Job and a SageMaker Training Job. You provide a Python script, a machine type, and optionally a custom Docker image. Vertex provisions a GCE instance, runs the script, streams logs to Cloud Logging, and uploads model artifacts to GCS. The instance terminates immediately when the job completes. Pre-built containers for scikit-learn, XGBoost, PyTorch, and TensorFlow eliminate the need to build custom Docker images for standard frameworks.
Vertex AI Pipelines — KFP components and pipelines with full lineage tracking
Vertex AI Pipelines is built on Kubeflow Pipelines (KFP) v2 — the same open-source pipeline framework used at Airbnb, Twitter, and many companies running on-premise Kubernetes. Each step is a KFP component decorated with @component. Components are pure Python functions that declare typed inputs and outputs. The pipeline function wires components together — outputs of one step become inputs of the next. Vertex compiles the pipeline to an YAML artifact and runs it on managed infrastructure with full lineage tracking in the Vertex ML Metadata store.
Vertex AI Feature Store — define features once, serve at 1ms online and petabyte offline
Vertex AI Feature Store is widely considered the most production-ready managed feature store across all three major clouds. It solves the training-serving skew problem from Module 69 at scale — features are defined once, computed once, and served consistently to both the training pipeline (point-in-time correct historical values) and the inference endpoint (latest values at <1ms latency). BigQuery serves as the offline store. Bigtable or the Vertex-managed online store serves the online tier.
Vertex AI Online Predictions — deploy, call, and split traffic in three SDK calls
Every common Vertex AI mistake — explained and fixed
Where Vertex AI actually earns its keep — BigQuery-native teams and Google's own models
Vertex AI shows up hardest at companies whose data already lives in BigQuery — ad-tech, gaming, delivery and marketplace platforms, media and streaming — where the ML team's daily loop is mostly SQL against the warehouse before it is anything else. For these teams, Vertex AI Pipelines and CustomTrainingJob let training stay inside the same ecosystem the analytics dashboards already run on, instead of exporting tables to an equivalent of S3 or Snowflake first. A recommendation model retrains directly off the same clickstream tables that power the company's own product dashboards, with no separate bridge pipeline between the analytics warehouse and the ML platform.
BigQuery ML is the sharpest version of that advantage. An analyst who has never written a line of Python can train a real model — logistic regression, boosted trees, k-means, ARIMA_PLUS for forecasting — with a single CREATE MODEL statement run in the same SQL console used for everyday reporting. Neither Azure ML's Designer nor SageMaker has a SQL-native training path this tightly wired into the warehouse itself; a marketing analyst building a first-pass churn model genuinely does not need to leave BigQuery to do it.
Model Garden extends the same pull. Teams often start a generative AI initiative there simply because it already sits inside the same console, IAM, and networking boundary as the rest of the data platform — Gemini-family models and popular open-source models like Llama or Mistral deploy in the same workflow as the company's own custom models, with no separate vendor relationship to stand up. TPU access is a real draw too, for teams doing heavier training or fine-tuning runs where a TPU v5e's cost per training run can beat GPU cost on workloads that are already written in a TPU-friendly style, typically TensorFlow or JAX compiled through XLA.
The honest caveat: teams whose data lives elsewhere — Snowflake on AWS, an on-premise Oracle warehouse — get much less of this. Without BigQuery as the warehouse of record, Vertex AI is mostly just Cloud Storage and compute like any other cloud, and the integration payoff that makes it worth choosing over Azure ML or SageMaker largely does not materialize.
Five things people get wrong about Vertex AI
AutoML is one corner of Vertex AI. The platform also includes custom training through CustomTrainingJob, Vertex Pipelines built on Kubeflow, the Feature Store, Model Garden with Google's own foundation models and popular open-source models, Matching Engine for vector search, and Workbench notebooks. Reducing Vertex AI to "the AutoML thing" misses most of what production teams actually build on top of it.
There is real friction to leaving, but it is not absolute. Custom-trained models saved as standard scikit-learn, XGBoost, TensorFlow, or PyTorch artifacts are portable to any serving stack. BigQuery data exports cleanly to Parquet or Avro. Vertex Pipelines are defined in the open-source Kubeflow Pipelines format, which can run on any Kubeflow-compatible cluster, not only Vertex. The genuinely non-portable pieces are narrower than "everything" — mainly Feature Store schemas, AutoML-produced model formats, and Matching Engine indexes.
Portability does not happen automatically just because open formats exist somewhere in the pipeline. Teams that build their feature logic directly against the Feature Store's API, route inference through Matching Engine, or depend on Vertex Pipelines' managed metadata history for lineage have created real dependencies that need deliberate re-architecture to remove. Lock-in risk has to be actively designed against — isolating cloud-specific calls behind small adapter layers — not assumed away because the model weights happen to sit in a standard format.
BQML is genuinely powerful for the class of problems it targets — logistic regression, boosted trees, k-means, ARIMA_PLUS forecasting — trained with a single CREATE MODEL statement an analyst can write without touching Python. It does not cover custom architectures, deep learning at meaningful scale, complex multi-source feature engineering, or the experiment tracking and governance a production ML function still owns. BQML expands who can train a first-pass model; it does not eliminate the need for an ML team.
TPUs win specifically on large-batch, matrix-heavy training compiled through XLA, which favors TensorFlow and JAX code written in a TPU-friendly style. A lot of real-world PyTorch training, anything relying on custom CUDA kernels, or workloads with small batch sizes and heavy branching logic often runs better, and is frequently only practical, on GPUs. Whether TPUs help is a per-workload question that needs checking against your actual framework and code, not a blanket assumption.
Vertex AI — 5 questions interviewers actually ask
Training scripts can query BigQuery directly with no export step to object storage first; the Feature Store is widely regarded as the most production-ready managed feature store across the three major clouds; TPU access is unique to GCP; Model Garden offers one-click deployment of both Google's own foundation models and popular open-source models inside the same IAM and networking boundary as the rest of the platform; and Vertex Pipelines is built on the open-source Kubeflow Pipelines spec rather than a fully proprietary format.
AutoML for a fast, strong baseline on standard tabular, vision, or text problems, especially early on when you do not yet know whether the problem is solvable with the available features. Custom training when you need a specific architecture, a custom loss function, unusual preprocessing, explainability that AutoML's more opaque models do not provide well, or tighter control over serving latency and cost. AutoML's node-hour billing can also exceed the cost of a well-tuned custom job once you already know roughly what architecture works.
Mostly by where the data will actually live and what the rest of engineering is already using — Postgres and general-purpose services argue for AWS and SageMaker, a BigQuery-centric analytics stack argues for Vertex AI, a Microsoft-heavy enterprise sales motion argues for Azure ML — plus the team's existing cloud expertise and the real cost of introducing a second cloud vendor purely for machine learning if the rest of the company is already committed elsewhere.
Features are defined once per entity type and computed once, rather than being recomputed slightly differently by the training pipeline and the serving path. BigQuery acts as the point-in-time-correct offline store used to build training sets, so a feature value is exactly what was known as of each historical event's timestamp, with no future information leaking backward. The same feature definitions are materialized into an online store for low-latency lookup at inference time, so the model sees identical feature logic in production to what it saw during training.
Partially. Models trained through CustomTrainingJob in standard formats like scikit-learn, XGBoost, TensorFlow, or PyTorch remain portable, and Kubeflow Pipeline definitions can run outside Vertex on any Kubeflow-compatible cluster. The genuinely sticky parts are Feature Store schemas, AutoML-produced model artifacts, and Matching Engine indexes. Mitigate it by keeping training code framework-native and cloud-agnostic, isolating GCP-specific calls such as BigQuery access and Feature Store lookups behind small adapter modules, and exporting critical feature and training data to open formats on a schedule so a migration becomes a re-plumbing exercise rather than a rewrite from scratch.
All three cloud ML platforms are covered. Next: MLOps on cloud — CI/CD for ML across all three platforms.
You have now covered Azure ML, SageMaker, and Vertex AI — the three platforms that run production ML at enterprises and startups. Module 79 ties them together: MLOps on Cloud — how to build CI/CD pipelines for ML that work regardless of which cloud you are on. GitHub Actions triggering retraining, model quality gates in CI, automated deployment to staging and production, and the patterns that make the entire ML lifecycle repeatable from a single git push.
GitHub Actions triggering retraining, model quality gates in CI, automated deployment to staging and production across all three clouds.
🎯 Key Takeaways
- ✓Vertex AI is built on the same GCP infrastructure that runs Google Search and Gmail. Its differentiators over Azure ML and SageMaker: native BigQuery integration (query training data directly without copying to object storage), the most production-ready managed Feature Store, and first-class TPU access. Used at Uber, Braintree, Reddit, and many analytics-heavy companies.
- ✓Every Vertex AI resource follows the naming pattern projects/{project}/locations/{region}/{resourceType}/{id}. All four supporting services work together: Cloud Storage for data and artifacts, IAM service accounts for permissions, Artifact Registry for Docker images, Cloud Logging for all job logs. When a job fails, check Cloud Logging first — the Python traceback is always there.
- ✓Vertex AI Pipelines uses KFP v2 components and pipelines. Lightweight @component decorators are convenient for simple steps but install packages at runtime — slow for large dependency sets. Container components (custom Docker images) are faster and should be used for any step that runs more than once. Pre-built Google components from google_cloud_pipeline_components handle AutoML, BigQuery export, and model upload.
- ✓Vertex AI Feature Store is the most complete managed feature store across all three clouds. BigQuery is the offline store (petabyte scale, point-in-time correct serving). Bigtable or Vertex-managed online store serves features at <1ms latency. batch_serve_to_bq() generates training datasets with point-in-time correct features. materialize() syncs offline to online on a schedule.
- ✓Online Endpoints support traffic splitting natively via the traffic_split parameter in deploy(). Canary: {'old_id': 90, '0': 10}. Full promotion: {'new_id': 100}. AutoML Tabular creates a dataset from BigQuery, runs Neural Architecture Search automatically, and returns a deployable model — the fastest path to a strong baseline with a budget_milli_node_hours cost cap.
- ✓Three common Vertex AI failures: PermissionDenied (grant roles/aiplatform.user + Storage Object Admin + BigQuery Data Viewer to the service account), KFP component failure (check Cloud Logging for the Python traceback — generic SDK error messages hide the real cause), slow components (pre-bake dependencies into a Docker image instead of using packages_to_install — eliminates 3-10 minute package install time per component run).
Discussion
0Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.