AIS (AI Station)

Overview

AIS (AI Station) is a multi-GPU local LLM serving platform that lets a user start, manage, chat with, and benchmark LLMs on their own GPU server through a web dashboard. It runs inside a Docker container and handles the full serving lifecycle: model discovery, smart GPU/port allocation, server lifecycle (start/stop/restart), live throughput monitoring, chat, an optional ReAct agent, and real performance benchmarking. It serves models through any of three inference engines — llama.cpp (GGUF), vLLM, and SGLang (HuggingFace weights) — auto-picking the right one per model and letting you override it in the UI.

Purpose

• Give a server owner a single web UI to run and manage local LLMs without manual llama.cpp commands
• Automatically pick the right GPU, context size, and port for each model
• Show live server state (health, measured throughput, slot occupancy) at a glance
• Support multi-GPU, multi-server operation on a single box
• Run a model on the inference engine that best fits it (llama.cpp, vLLM, or SGLang)

Key Concepts

Concept	Description
Web Dashboard	Browser-based UI (port 8090) for all AIS operations
Smart Suggestion	Engine that recommends GPU, ctx, and port based on model size and current VRAM
Auto-Recovery	A crashed server is re-spawned automatically with backoff
Speed Column	Live measured throughput (tok/s) per running server, updated on every poll
ReAct Agent	Optional tool-calling agent (file read/write, web fetch/search, shell)
Multi-Engine	Serves a model via llama.cpp, vLLM, or SGLang; auto-picked by model format, overridable in the UI

Workflow

1. Log in to the AIS dashboard (http://<server>:8090)
2. Go to Models — browse cached models or pull a new one from HuggingFace
3. Go to Serving — select a model, review the smart suggestion, click Start
4. Go to Chat — select the running server and start conversing
5. Optionally configure an Agent profile for tool-calling tasks

Related Pages

• Manual:AIS — full user manual (screenshots + step-by-step)
• AI Agents
• GPU