STACK · advanced

Local Agents on NVIDIA

Agents running on your own silicon: local inference, no per-token bill, your data never leaves the room.

What this stack is for

Some data cannot leave the building, and some bills should not scale with your curiosity. This stack is for the person who wants agents running on hardware they own: no per-token meter, no third party reading your prompts, no terms-of-service update quietly rewriting your architecture. You pay up front in silicon and setup time, and after that inference is effectively free and private for as long as the fans keep spinning.

Components

  • NVIDIA AGX/DGX: local GPU hardware, the inference engine's silicon
  • Ollama: simple local model serving, one command per model
  • vLLM: high-throughput local inference server for production-shaped local serving
  • Docker: containerizes the agent runtime alongside the model server
  • Open WebUI: chat interface in front of the local models

How the pieces fit

  • Ollama or vLLM serves the model locally over an HTTP API on the NVIDIA hardware
  • Docker Compose runs the model server, the agent runtime, and Open WebUI as separate containers on one network
  • The agent code calls the local model server the same way it would call a hosted API, just pointed at localhost
  • Open WebUI gives a chat interface for testing and for non-technical users, without touching a cloud model
  • No request or token ever leaves the machine; there is no per-token bill

Setup outline

  1. Confirm the NVIDIA hardware, drivers, and CUDA toolkit are working (full walkthrough coming)
  2. Install Ollama or vLLM and pull/serve the target model (full walkthrough coming)
  3. Write the Docker Compose file for the model server, agent runtime, and Open WebUI (full walkthrough coming)
  4. Point the agent code's API base URL at the local model server (full walkthrough coming)
  5. Verify throughput and VRAM headroom under real load (full walkthrough coming)

What to hand your agent

Local setups live and die on hardware specifics, so hand all of them over:

  • The full stack description above, plus the exact GPU model, VRAM, and target model size
  • The Docker Compose conventions already in use, if any
  • The instruction to never add a call to a hosted model API as a "fallback" without being asked

Adaptation notes

  • A Mac Studio or Mac Mini with unified memory can replace NVIDIA hardware for smaller models
  • vLLM is the better choice over Ollama once concurrent requests matter more than setup simplicity
  • Open WebUI can be skipped entirely for an API-only agent with no human chat interface