Local AI Glossary

The C/C++ tensor library that llama.cpp is built on (named after GGML, the format that made it famous). It defines tensors, operations and backends — CPU, GPU, Apple Silicon — on which models run in local inference. It is the hardware layer of the inference stack: runtimes like Ollama and llama.cpp sit on top of it.

Attention scheme where several query heads share the same group of Key and Value matrices, reducing the KV-head count compared to MHA. Lowers the cost and memory-access speed during decoding with a minimal quality trade-off.

Family of attention schemes (DeltaNet, Gated DeltaNet, Mamba, etc.) that replace the linearly growing KV cache with a fixed-size recurrent state, drastically reducing memory at long context. It's the technique behind hybrid-attention models like Qwen3.8-27B.

Google's on-device AI toolkit, the evolution of TensorFlow Lite. ML Drift integrates with LiteRT as the GPU accelerator: it is the entry path of the engine into mobile apps, the browser, and edge devices.

Apple's GPU compute and graphics API. It is the backend through which on-device AI runs on iPhone, iPad, and Mac: cross-platform engines like ML Drift use it across the whole Apple platform.

Model architecture in which the model counts many billions of parameters in total, but for each token activates only a small subset (the 'experts' selected by a router). The result is the quality of a large model with the speed and compute cost of a small one. Example: Qwen3 30B-A3B has 30.5B total parameters but uses only 3.3B per token. The price is that the whole model weight must still be loaded into memory.

Open framework for parallel computing on GPUs, CPUs, and other processing units. It is the traditional backend for on-device inference on Android: ML Drift uses it on the Google platform alongside Metal (Apple) and WebGPU (browser).

Retrieval Augmented Generation. Architecture that combines a language model with a document retrieval system. Instead of relying solely on model knowledge, the system retrieves relevant documents from a database before generating the response. This reduces hallucinations and makes the system updatable without retraining.

The stage of a RAG system where, from the user's question, the system searches the vector database for the most similar text blocks and adds them to the prompt. The quality of retrieval is the factor that determines the quality of the final answer.

Feature introduced in Qwen3 where the model can generate a long intermediate reasoning step before producing the final answer. It improves quality on logic, math and planning problems, at the cost of more tokens (and therefore more time and memory) per answer. It can be switched on or off, and in Qwen3 the model also decides for itself whether to reason.

Neural network architecture introduced in 2017 (Attention Is All You Need) that revolutionized NLP. Unlike recurrent networks, Transformers use attention to process the entire input simultaneously. All modern LLMs (GPT, Llama, Mistral, Claude) are based on this architecture.

Apple's technology that fuses two M5 Max dies into a single M5 Ultra chip, presented to the system as one processor. The unified memory of both dies forms a single pool (up to 512GB) accessible to CPU, GPU and Neural Accelerators.

The Apple Silicon architecture in which CPU and GPU share the same pool of RAM. For local AI, the entire model can reside in that memory without copying data between system and video memory.

Voiced Language Model with a Varying-rate adversarial (VITS): a neural text-to-speech architecture that generates the waveform directly from text without a separate vocoder. It underpins Piper and Kokoro-82M.

Browser API for GPU rendering and compute, the W3C standard replacing WebGL for heavy compute workloads. In 2026 it is the backend through which on-device inference engines (like ML Drift, via Dawn) run directly in the web page, without the cloud.