AIS (AI Station): Difference between revisions
Jump to navigation
Jump to search
(AIS (AI Station) — brief intro (HMSmartLLM)) |
(AIS: note multi-engine serving (llama.cpp/vLLM/SGLang)) |
||
| (One intermediate revision by the same user not shown) | |||
| Line 1: | Line 1: | ||
== Overview == | == Overview == | ||
AIS (AI Station) is a '''multi-GPU local LLM serving platform''' that lets a user start, manage, chat with, and [[benchmark]] LLMs on their own GPU server through a web dashboard. It runs inside a [[Docker]] container and handles the full serving lifecycle: model discovery, smart GPU/port allocation, server lifecycle (start/stop/restart), live throughput monitoring, chat, an optional ReAct agent, and real [[performance benchmarking]]. | <div style="float:right;margin:0 0 10px 15px;width:420px"> | ||
[[File:AIS-dashboard.png|420px|AIS Dashboard — GPU cards and running servers]] | |||
</div> | |||
AIS (AI Station) is a '''multi-GPU local LLM serving platform''' that lets a user start, manage, chat with, and [[benchmark]] LLMs on their own GPU server through a web dashboard. It runs inside a [[Docker]] container and handles the full serving lifecycle: model discovery, smart GPU/port allocation, server lifecycle (start/stop/restart), live throughput monitoring, chat, an optional ReAct agent, and real [[performance benchmarking]]. It serves models through any of three inference engines — [[llama.cpp]] (GGUF), [[vLLM]], and SGLang (HuggingFace weights) — auto-picking the right one per model and letting you override it in the UI. | |||
== Purpose == | == Purpose == | ||
| Line 9: | Line 13: | ||
* Show live server state (health, measured throughput, slot occupancy) at a glance | * Show live server state (health, measured throughput, slot occupancy) at a glance | ||
* [[Support]] multi-GPU, multi-server operation on a single box | * [[Support]] multi-GPU, multi-server operation on a single box | ||
* Run a model on the inference engine that best fits it (llama.cpp, vLLM, or SGLang) | |||
== Key Concepts == | == Key Concepts == | ||
| Line 30: | Line 35: | ||
| ReAct Agent | | ReAct Agent | ||
| Optional tool-calling agent (file read/write, web fetch/search, shell) | | Optional tool-calling agent (file read/write, web fetch/search, shell) | ||
|- | |||
| Multi-Engine | |||
| Serves a model via llama.cpp, vLLM, or SGLang; auto-picked by model format, overridable in the UI | |||
|} | |} | ||
Latest revision as of 09:14, 13 October 2026
Overview
AIS (AI Station) is a multi-GPU local LLM serving platform that lets a user start, manage, chat with, and benchmark LLMs on their own GPU server through a web dashboard. It runs inside a Docker container and handles the full serving lifecycle: model discovery, smart GPU/port allocation, server lifecycle (start/stop/restart), live throughput monitoring, chat, an optional ReAct agent, and real performance benchmarking. It serves models through any of three inference engines — llama.cpp (GGUF), vLLM, and SGLang (HuggingFace weights) — auto-picking the right one per model and letting you override it in the UI.
Purpose
- Give a server owner a single web UI to run and manage local LLMs without manual llama.cpp commands
- Automatically pick the right GPU, context size, and port for each model
- Show live server state (health, measured throughput, slot occupancy) at a glance
- Support multi-GPU, multi-server operation on a single box
- Run a model on the inference engine that best fits it (llama.cpp, vLLM, or SGLang)
Key Concepts
| Concept | Description |
|---|---|
| Web Dashboard | Browser-based UI (port 8090) for all AIS operations |
| Smart Suggestion | Engine that recommends GPU, ctx, and port based on model size and current VRAM |
| Auto-Recovery | A crashed server is re-spawned automatically with backoff |
| Speed Column | Live measured throughput (tok/s) per running server, updated on every poll |
| ReAct Agent | Optional tool-calling agent (file read/write, web fetch/search, shell) |
| Multi-Engine | Serves a model via llama.cpp, vLLM, or SGLang; auto-picked by model format, overridable in the UI |
Workflow
- Log in to the AIS dashboard (http://<server>:8090)
- Go to Models — browse cached models or pull a new one from HuggingFace
- Go to Serving — select a model, review the smart suggestion, click Start
- Go to Chat — select the running server and start conversing
- Optionally configure an Agent profile for tool-calling tasks
Related Pages
- Manual:AIS — full user manual (screenshots + step-by-step)
- AI Agents
- GPU