Running Open Models — Ollama, vLLM and Hugging Face

Run open-weight models locally with Ollama, serve them with vLLM, pick a model from Hugging Face, and compare cost, latency and quality with hosted APIs.

Est. 35 min Updated Suggest a fix
// Part 01 — Concept

Open Weights Give You More Control—and More Responsibility

A hosted API runs inference on a provider's infrastructure. With an open-weight model, you can obtain model weights under a particular licence and run inference on your own machine or infrastructure. That can improve control over deployment, data flow and hardware choices, but it does not mean the model is free to use for every purpose or that you own the training data. Read the model card and licence, evaluate acceptable use, and plan for updates and operational support.

Ollama is convenient for local experiments and provides a local API. vLLM is an inference server often selected when throughput and GPU serving matter. Hugging Face is a model and tooling ecosystem where teams discover model cards, weights and libraries. These are complementary pieces, not interchangeable product guarantees: check current documentation, hardware compatibility and the model's own requirements before committing.

PathGood first useTrade-off to plan for
Hosted model APIPrototype quickly without managing inference hardwareProvider terms, network dependency and usage charges
Ollama locallyPrivate development experiments on a workstationLocal hardware, model storage and workstation availability
vLLM serverServe a compatible model on managed GPU infrastructureGPU operations, scaling and model-specific tuning
Hugging Face toolingInspect model cards and load supported checkpointsLicence review, downloads, dependencies and memory needs
💡 Note
"Open-weight" describes access to model weights, not necessarily open-source software or unrestricted use. Licence obligations and acceptable-use terms are model-specific.
// Part 02 — Select

Choose a Model by Requirements, Not Hype

Start with a task and a baseline. A small instruction model may be enough for a fixed-format extraction task, while code or multilingual use may require different capabilities. Compare candidates on your own prompts and documents, not only public benchmark scores. Model IDs should live in environment configuration so an upgrade is deliberate and reversible.

.env.example
bash
# Copy these values from the chosen model card and local deployment configuration.
MODEL_ID=replace-with-an-approved-model-id
OLLAMA_MODEL=replace-with-a-model-available-in-your-local-install
LOCAL_API_BASE=http://localhost:11434
  1. Write down the quality, language, context, latency and privacy requirements for one representative task.
  2. Check the model card for licence, intended use, supported languages, context limits and known limitations.
  3. Estimate weight storage and inference memory from documented model requirements; include runtime overhead.
  4. Run a small, versioned evaluation set and compare quality with your hosted baseline.
  5. Pin an approved model revision for repeatable deployment and record who can change it.
🎯 Pro Tip
Use a configuration variable such as MODEL_ID, but do not assume any arbitrary repository ID will work with every runtime. Confirm model architecture and serving support first.
// Part 03 — Local

Try an Ollama Model Behind a Local API

Install Ollama using its official instructions for your operating system, then choose a model available to your installation and put its exact identifier in configuration. The following Python example uses the local chat API; it does not require exposing a provider key. Keep the service bound to a trusted local interface while experimenting.

ollama_chat.py
python
import json
import os
from urllib.request import Request, urlopen


base_url = os.environ.get("LOCAL_API_BASE", "http://localhost:11434").rstrip("/")
model = os.environ["OLLAMA_MODEL"]
payload = {
    "model": model,
    "messages": [
        {"role": "system", "content": "Explain clearly and say when you are uncertain."},
        {"role": "user", "content": "What is a database index?"},
    ],
    "stream": False,
}
request = Request(
    f"{base_url}/api/chat",
    data=json.dumps(payload).encode("utf-8"),
    headers={"Content-Type": "application/json"},
    method="POST",
)
with urlopen(request, timeout=60) as response:
    result = json.load(response)
print(result["message"]["content"])

This script is an experiment, not a production service. Add request validation and controlled error handling before using a local model from a web app. Never expose an unauthenticated inference endpoint to a public network: anyone who can reach it may consume your machine's resources or send prompts you did not intend to support.

// Part 04 — Serve

Use vLLM When You Need a Managed Inference Endpoint

A serving engine loads a model onto suitable hardware and handles inference requests. vLLM can expose an OpenAI-compatible API for supported configurations. A basic launch is shown below; choose the model ID from your reviewed deployment configuration and follow the current vLLM installation and model-specific documentation for hardware, access and optional settings. Do not copy model-specific flags blindly.

start_server.sh
bash
# MODEL_ID is set by the deployment environment after licence and hardware review.
vllm serve "$MODEL_ID"

For a production deployment, package the exact runtime dependencies, place the server behind authentication and an application gateway, and configure health checks, timeouts and concurrency limits. A server that listens on a reachable interface needs network policy and access control. The application should send only authorized prompts and should never trust a model response as a command to run on the host.

openai_compatible_client.py
python
import os
from openai import OpenAI


client = OpenAI(
    base_url=os.environ["LOCAL_OPENAI_BASE_URL"],
    api_key=os.environ.get("LOCAL_OPENAI_API_KEY", "local-only"),
)
response = client.chat.completions.create(
    model=os.environ["MODEL_ID"],
    messages=[{"role": "user", "content": "Summarize this public FAQ in two sentences."}],
)
print(response.choices[0].message.content)

The server and client must agree on the model identifier and request shape. Some compatible endpoints still differ in supported parameters, structured outputs, streaming or tool calling. Exercise each feature you rely on against the deployed version rather than assuming compatibility from the endpoint format alone.

// Part 05 — Hugging Face

Inspect the Model Card Before Loading Weights

On Hugging Face, a model card is part of the selection process: review its intended use, licence, architecture, training context and limitations. The Transformers loading pattern below reads the repository ID from the environment. It assumes the selected checkpoint is supported by the installed Transformers version and that the machine has enough memory; installation details and optional acceleration vary across models and hardware.

inspect_model.py
python
import os
from transformers import AutoConfig, AutoTokenizer


model_id = os.environ["MODEL_ID"]
config = AutoConfig.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)
print("architecture:", config.architectures)
print("tokenizer vocabulary:", len(tokenizer))
print("model type:", config.model_type)

Loading remote model code can execute code depending on library settings and repository contents. Prefer checkpoints that do not require custom code, inspect any custom code before enabling it, pin revisions, and run downloads and inference in an appropriately isolated environment. Do not send private customer data to an external model host unless your policies and agreements allow it.

⚠️ Important
A model download is a software supply-chain event. Record the repository and revision, verify publisher and licence, scan dependencies, and promote an approved artifact rather than downloading a moving default on every production start.
// Part 06 — Quantisation

Trade Memory for Some Accuracy and Speed Changes

Quantisation stores model weights at a lower numerical precision, often reducing memory requirements. The quality impact depends on the model, quantisation method and task. A smaller artifact can make local inference feasible, but latency also depends on the processor, memory bandwidth, batching and runtime. Benchmark the exact build you plan to operate; do not infer production performance from file size alone.

MeasureQuestion to answerHow to compare
QualityDoes the model still answer the real task correctly?Run the same reviewed eval set and inspect errors
LatencyCan users wait for the response?Measure time to first token and total response time
ThroughputCan the service handle expected concurrency?Load test with bounded, representative requests
Resource useDoes the workload fit available hardware?Observe memory, accelerator use and queue depth
OperationsCan the team patch and recover the service?Test restart, rollback, monitoring and capacity alerts

Select quantisation or batching only after collecting a baseline. Preserve the evaluated model artifact and settings, then rerun the evaluation whenever weights, runtime, prompt template or hardware changes.

// Part 07 — Decision

Compare Total Cost and Reliability with a Hosted API

Local inference replaces some per-request provider expense with fixed and operational costs: hardware rental or purchase, power, storage, deployment work, monitoring, security and downtime. If requests are sporadic, a hosted API may be simpler. If workloads are steady and predictable, self-hosting may be worth measuring. Compare the same task, quality bar and reliability target on both paths.

compare_runs.py
python
from dataclasses import dataclass


@dataclass(frozen=True)
class Run:
    system: str
    requests: int
    correct: int
    total_seconds: float
    estimated_cost_inr: float


def summarize(run: Run) -> dict[str, float]:
    if run.requests <= 0 or run.total_seconds <= 0:
        raise ValueError("requests and elapsed time must be positive")
    return {
        "accuracy_on_reviewed_set": run.correct / run.requests,
        "requests_per_second": run.requests / run.total_seconds,
        "estimated_cost_per_request_inr": run.estimated_cost_inr / run.requests,
    }

# Populate from the same evaluation set and measured infrastructure costs.

Cost estimates must include idle capacity and engineering operations, not only GPU time while generating. Track quality, errors, queueing and cost together: a low-cost model that repeatedly needs human correction may be the more expensive choice overall.

// Part 08 — Pitfalls

Common Mistakes

Assuming open weights mean unrestricted commercial use

Why: Each model can have its own licence and use conditions.

Fix: Review and record the licence and intended-use terms before evaluation or deployment.

Exposing a local inference port publicly

Why: A development server may have no authentication or resource limits.

Fix: Bind to a trusted interface, use network controls and require application-level authorization.

Choosing a model from a benchmark headline

Why: General benchmarks may not match your languages, domain or failure costs.

Fix: Compare candidates on a representative, versioned evaluation set.

Downloading a moving model revision on every deploy

Why: Weights or custom code may change without an application release.

Fix: Pin and review the artifact, record dependencies, and test rollback.

// Part 09 — Practice

Benchmark One Local Model Against Your Baseline

Practice

A repeatable model selection experiment

  1. Choose a permitted, compatible open-weight model from its official model card and configure its exact ID in the environment.
  2. Run the same five public FAQ questions through a hosted baseline and a local Ollama or vLLM endpoint.
  3. Record response quality, errors, time to first token, total latency and measured resource usage.
  4. Compare a standard and quantized build only if both are supported, and rerun the same evaluation after every change.
  5. Write a deployment checklist covering licence, artifact revision, endpoint access, monitoring and rollback.

Hint: Do not use private customer data in a prototype. Keep prompts and expected outcomes in a small version-controlled evaluation file.

🎯 Key Takeaways

  • ✓Open-weight inference shifts responsibility for model selection, licence review, hardware, security and operations to your team.
  • ✓Ollama suits local experiments; vLLM serves compatible models; Hugging Face helps discover and load model artifacts.
  • ✓Read model cards, pin approved revisions and keep model identifiers in configuration rather than source code.
  • ✓Quantisation and batching can change quality, latency and resource use; measure the deployed configuration.
  • ✓Compare total cost and reliability on a representative evaluation set, including fixed infrastructure and engineering effort.
  • ✓Protect inference endpoints with authentication, network controls, bounded requests and safe failure behavior.

Next

What comes next

You have compared local and hosted inference. Next, wrap an AI feature in a service with streaming responses, a chat UI and deployment checks.

Lesson 14 · Shipping an AI Feature — FastAPI Backend and Chat UI →

HELP US IMPROVE

How was this lesson?

Your note is shared with the Chaduvuko team for review. Please don’t include personal or sensitive information.

Choose your feedback

Spotted a mistake or something unclear? Suggest a fix on GitHub or email us. Select the text first and it is quoted for you.

Share

Discussion

0

Have a better approach? Found something outdated? Share it — your knowledge helps everyone learning here.

Continue with GitHub
Loading...