Understanding LLM model
Overview
This page explains, in everyday terms, the information shown next to each model in your AIS Models list — the columns and the small ⚠ warning badges. It covers the ideas behind them (attention type, context, quantization) and what each warning means, so you can read a model's specs and choose a good one with confidence.
Summary
- What is it? A guide to the model list — its columns and ⚠ badges — in plain language.
- Why? So you can understand a model's specs and warnings and pick a suitable one.
- When? When you are browsing the Models list and a number or a ⚠ badge is unclear.
Purpose
- Goal: explain the model list (columns + badges) without specialist jargon.
- Scope: attention types (MHA / GQA), context (trained vs. practical), quantization, and the ⚠ badges.
- Non-goals: not a training guide and not the operator manual (see Manual:AIS).
Key Concepts
| Concept | In plain terms | Why it matters to you |
|---|---|---|
| Attention (MHA / GQA) | How the model "looks back" over the conversation to understand it | Decides how much memory each message costs, and so how long a conversation you can have |
| Context window | The maximum length of conversation the model can handle | Fixed at training; the real limit on your hardware can be lower |
| KV cache | The model's running "memory" of the conversation | Must fit in memory; if it cannot, the usable context shrinks |
| Quantization | Storing the model in a smaller, lower-precision form | Smaller and faster, with a little less accuracy (for example Q4_K_M) |
| MTP | A helper that predicts several tokens at once | Can make a model respond faster |
The model list, column by column
| Column | What it shows |
|---|---|
| Name | The model (and, for downloaded models, where it came from). Click it for full details. |
| Size | The download size in GB. Bigger is usually more capable but needs more memory. |
| Context | How much conversation the model was trained to handle (for example 131,072 tokens). A ⚠ badge may show a lower limit on your hardware. |
| Vision | Whether the model can understand images. |
| Path | Where the model file lives on the server. |
Attention: MHA vs GQA
Attention is how a language model "pays attention" to the earlier parts of a conversation so it understands the current part — a bit like re-reading the earlier sentences before answering a question.
MHA (Multi-Head Attention) is the classic design. The model has many "heads", and each of them keeps its own complete memory of the conversation (its key/value cache). This gives the best understanding, but each word of the conversation costs a lot of memory to remember.
GQA (Grouped Query Attention) is a memory-saving improvement. Many "reading" heads SHARE a smaller set of "memory" heads, so far less memory is needed per word. A much longer conversation then fits in the same hardware, and the quality is very close to MHA. Modern large models (for example the Qwen3 family) use GQA.
A simple analogy. MHA is like every note-taker at a long meeting writing out their own complete notebook — thorough, but it uses a huge amount of paper. GQA is like the group sharing a few common notebooks — slightly less thorough per person, but far less paper, so the whole meeting can still be recorded.
There is also MQA (Multi-Query Attention), the extreme case where all the heads share a single memory set: the smallest memory, with a little more loss in quality.
What this means for you. On the same machine, a GQA model supports a longer conversation than an MHA model, because it uses less memory per word. This is exactly what the ⚠ MHA badge is warning you about.
Context: trained vs practical
The Context column shows the length the model was trained for. But how much you can actually use depends on how much memory your machine has, because the model's conversation memory (the KV cache) has to fit.
- If the model is GQA, the KV cache is small, so it can use close to the trained length.
- If the model is MHA, the KV cache is large, so the usable length is capped much lower on typical hardware.
The badge next to the context number tells you the practical maximum on this machine. For example, ⚠ MHA ~32,768 means "trained for 131,072, but on this hardware you can use about 32,768 tokens". If the conversation memory does not fit on the graphics card, it is kept in normal memory instead, which is slower — shown by the ⚠ KV→RAM badge.
Quantization
A model is stored in a quantized form so it fits in memory and runs faster. The quantization level is usually in the model name (for example -Q4_K_M):
| Level | Size | Quality |
|---|---|---|
| Q8_0 | Large | Very close to the original |
| Q4_K_M | Medium (a common choice) | A good balance of size and quality |
| Q2_K | Small | Noticeably less accurate |
Lower precision means a smaller, faster model with a little less accuracy. Q4_K_M is a common sweet spot for most uses.
The ⚠ badges, at a glance
A ⚠ badge next to a model is a short status. Some are simply informative (the model is fine, but a limit applies); a few mean that particular file cannot be served on its own.
| Badge | What it means | What to do |
|---|---|---|
| ⚠ MHA ~N | This is a full-MHA model, so the usable context on this hardware tops out around N — not the trained value. The model itself is fine. | For long conversations, choose a GQA model. |
| ⚠ KV→RAM (slower) | The conversation memory is in normal RAM rather than the graphics card, so responses are slower. | Reduce the context size so the memory fits on the graphics card. |
| ⚠ incomplete | A partial or interrupted download, or an auxiliary file (a draft/MTP head, or a shard of a split model). | Open it, then re-pull, serve the matching main model, or delete to free space. |
| ⚠ projector | A vision-adapter companion file that belongs to a multimodal model. | Not servable on its own; use its main model. |
| ⚠ draft-only | A draft/MTP companion file that belongs to a main model. | Not servable on its own; serve the main model. |
| (no badge) | Servable as it is. | — |
A ⚠ badge is a property of the model file, not a fault of AIS. AIS checks each file's internal structure before flagging it, so a complete model shows no badge even when the file omits some optional metadata.
How to choose a model
- Long conversations — prefer a GQA model; it uses less memory per word, so it fits more context.
- Maximum quality for a moderate context — an MHA model is fine.
- Match the size to your memory — bigger models are more capable but need more memory.
- Read the ⚠ badge — MHA and KV→RAM are limits to be aware of; incomplete, projector and draft-only mean that particular file is not the main servable model.
- When in doubt, pick a well-known GQA model (such as the Qwen3 family) at a sensible quantization (Q4_K_M or Q8_0) and a context length you can afford.
Limitations
- The "practical context" shown is an estimate from the available memory; exact numbers vary with your hardware and settings.
- The quantization and context guidance is general, not a guarantee of quality for any specific task.
References
- Manual:AIS — the operator manual for the AIS system.