Blog
Articles, guides, and insights on local artificial intelligence
3 articles found per "GGUF"

MiniCPM-V 4.7: the 35B MoE that runs on a smartphone with 3 GB of RAM
MiniCPM-V 4.7 (35B total, ~3B active, released October 12, 2026 by OpenBMB and ModelBest) is the first open MoE in the series, vision-language: multiple images, video, OCR. 13+ tok/s with 3 GB of RAM on an iPhone 17 Pro Max, 17+ on Android. Official quantizations BF16, FP8, INT4, AWQ, GPTQ, GGUF; engines Ollama, llama.cpp, vLLM, SGLang. The three techniques: Lookahead Expert Scheduling, fine per-level quantization, DSpark speculative sampling.

GGUF: The Format That Keeps Models on Your Hardware
GGUF is the single-file format that ships models for local inference: a header with a signature and version, key-value metadata, a tensor index, and weights aligned for fast mmap loading. Quantization types with real sizes (8B from 14.96 GiB at F16 to 4.58 GiB at Q4_K_M), the filename convention, and the rule for picking a size against your VRAM or RAM.

Quantization: Get More From the Same Hardware
Quantization reduces the precision of the weights (from FP16 to 4–8 bit) so models fit in less VRAM and generate more tokens per second. The levels, the real file sizes of Qwen2.5 14B, the rule for choosing from Q4_K_M, and the limits (code, reasoning, languages other than English).

