Thinking Machines Lab announced Inkling 2 Small on November 5, 2026, and released its weights on Hugging Face on November 12. Unsloth distributed smaller weight files so developers can choose local files that fit their memory.
At a glance
Inkling 2 Small takes text and images and outputs text. Its sparse MoE architecture uses 8 of 384 experts per token. It has 594B total parameters, with 17B active in computation. All weights must be stored even when only some participate in computation. Active parameter counts alone therefore cannot indicate file size.
Quantization options: 171.7GB to 484.4GB
GGUF is a weight file format used by llama.cpp and Unsloth Desktop. Unsloth released 8 quantizations that represent weights with fewer bits, plus a BF16 reference. Top-token agreement is the share of prediction positions where the top predicted token matches the reference. Mean KLD measures differences in output probability distributions; lower values are closer to the reference.
Conditions: Unsloth measurements, comparing wikitext-2 64×4096 tokens against BF16 GGUF, both using the default llama.cpp F16 KV cache. Sizes exclude the vision projector.
Table: Sizes and comparisons against BF16 by quantization
Units: File size in GB, agreement in %
| Quantization | File size | Mean KLD | Top-token agreement with BF16 |
|---|---|---|---|
| UD-IQ1_S 1-bit | 171.7 | 0.0922 | 89.7 |
| UD-IQ2_XXS 2-bit | 190.6 | 0.0577 | 91.8 |
| UD-Q3_K_XL | 285.1 | 0.0176 | 95.3 |
| UD-Q4_K_XL | 354.2 | 0.0029 | 98.0 |
| BF16 reference | 484.4 | Reference | — |
Unsloth says the 1-bit file is 57% smaller than the original 394GB NVFP4 weight file set. Among the listed options, switching from UD-Q3_K_XL to UD-Q4_K_XL adds 69.1GB. Agreement rises by 2.7 percentage points, while mean KLD falls from 0.0176 to 0.0029.
Unsloth's BF16 reference is a lossless conversion that retains the original NVFP4 expert weights in layers 0–39 and stores only the rest in 16-bit floating-point format.
File size is not the total memory requirement for inference. Beyond the operating system, memory is needed for the KV cache storing prior token computations, runtime buffers providing temporary workspace, and the vision projector processing image features. Unsloth advises choosing a quantization that fits RAM and VRAM or unified memory.
Performance: Against comparison models
Conditions: Thinking Machines Lab evaluations used stirrup v0.2.1 for FrontierCode v2.0 and mini-swe-agent v2.4.6 for the other three tests. Inkling 2 Small used effort 0.99(xhigh), temperature 1.0, and agent trajectories of up to 1M tokens.
Table: Coding and task evaluation scores by model
Unit: %
| Model | DeepSWE v1.1 | SWEBench Pro v2 (Hard) | FrontierCode v2.0 | Terminal Bench Science v0.1 |
|---|---|---|---|---|
| Inkling 2 Small | 72.1 | 86.3 | 25.4 | 5.7 |
| GLM-5.3-Flash | 63.4 | 82.8 | 17.6 | 4.8 |
| DeepSeek-V4.1-Flash-1017 | 77.0 | 89.7 | 28.5 | 12.9 |
| Muse Spark Open | 60.0 | 82.4 | 7.6 | 1.9 |
| Kimi K3 | 68.5 | 88.2 | 17.3 | 8.1 |
In Thinking Machines' published coding benchmarks, evaluations comparing model performance, Inkling 2 Small ranks toward the top of the comparison group. The same release reports 30.6 on AA Index v5.0 and 56.1% on Vals Index v2.2. It trails DeepSeek-V4.1-Flash-1017 on all four listed evaluations. On Terminal Bench Science v0.1, it scores 5.7% versus 12.9%.
Running locally with llama.cpp
The llama.cpp build is the first hurdle. Inkling 2 Small uses the new inkling2 architecture, so it needs a build supporting Inkling 2. Builds supporting only the earlier Inkling cannot load it. Unsloth's documentation instructs users to fetch and build PR 32187 from the official llama.cpp repository.
hf download unsloth/Inkling-2-Small-GGUF \
--include "UD-IQ2_XXS/*" "mmproj-F16.gguf" --local-dir Inkling-2-Small-GGUFRunning llama-server with the downloaded files opens a chat interface at http://localhost:8080. If it does not fit in VRAM, lower -ngl or use -ngl 0 for CPU inference. Image input requires --mmproj and the 658MB vision projector mmproj-F16.gguf; omit --mmproj for text-only use. Keep shards (split files) in one folder and pass only the first file; the others are joined automatically.
The GGUF template defaults to high reasoning effort. Context supports up to 1,048,576 tokens, but Unsloth recommends starting with 32K tokens to account for memory usage.
Where to use it
The weights were released under Apache 2.0. The original model integrates with SGLang, vLLM, and other tools, and can be tried directly in Tinker Playground.
The Thinking Machines model card says the original NVFP4 checkpoint needs a GPU cluster, a setup combining multiple GPUs, with at least 500GB of aggregate VRAM. Examples include 2×NVIDIA B300 and 4×NVIDIA H200. The MXFP8 checkpoint needs at least 800GB in aggregate, such as 4×B300 or 8×H200. Thinking Machines says NVFP4 is the native precision, with MXFP8 gaining only 1–2 percentage points in evaluations. These requirements apply to running the original checkpoints directly, not to the 171.7GB GGUF.
Limitations
Top-token agreement in the quantization table measures quantization loss, not task benchmark accuracy. Thinking Machines says Inkling 2 Small accepts only text and image inputs and is less optimized for the chat experience.
The key choice for local execution is how much to reduce weight size and how much prediction divergence from the reference to accept.
