The open-source Tesseract project released designs for reconfigurable FPGA accelerator cards and software for running language models. Developers can run inference on Alveo U200 cards and accelerate generation by producing candidate tokens first, then checking them in batches.
1. Separate hardware and firmware releases
Released on November 10, Tesseract Core 1.0.0 contains the circuit image loaded onto the card and its design code. Tesseract Basis 1.1.0, released alongside it, provides firmware development tools and host software. Firmware is low-level software that controls the card; host software runs on the computer that sends commands from outside the card. Circuit images and execution software are distributed separately.
A key fix in Basis 1.1.0 measures actual card memory capacity. The U200 has 8GiB per channel, but its shell reports 16GB. Previously, addresses exceeding channel capacity could wrap around and overwrite model weights. The new version checks actual capacity when opening the card and uses it for heap layout and bounds checks. The heap is the memory area used dynamically by programs. Models that do not fit in memory now stop with a WEIGHT heap full error.
The execution path connects to llama.cpp. Integration software sends supported operations to the card and handles the rest on the CPU. Full mode also handles the KV cache and attention on the card, issuing one work request per generated token. The KV cache stores results computed for earlier tokens; attention determines which parts of the input to reference.
2. Building and running on the card
The getting-started guide first fetches submodules, components that include external repositories, and applies patches, changes to the original code. It then installs the control-core toolchain, a set of build and compilation tools, and builds with CMake. Running on the card requires XRT 2.18.179. When opening the device, the host loads the circuit image and a separately built firmware file.
After preparing basic inference, users can test multi-token prediction with a dedicated benchmark tool. This example from the performance documentation specifies the target model, draft file and prompt file to measure speed across candidate counts.
tsr-mtp -m TARGET.gguf [-md MTP.gguf] --spec-type draft-mtp --mtp-mode all --mtp-k-list 1,2,3,4 \
--mtp-device xrt --mtp-prompts PROMPTS.txt -n 128 -c 2048 -t 8After installation, users can run self-tests to check the device and use the model validation tool to compare results with CPU output.
3. llama.cpp speeds measured on the U200
According to the project, the results below were measured in full mode on a single U200. Prompt processing and generation speeds show the mean and standard deviation across three runs; standard deviation measures how far results vary from the mean. Power is the card's average consumption during generation. Both tables below were measured with Basis 1.0.0 firmware, not Basis 1.1.0.
Table: Per-model throughput on one U200
Conditions: 1 Alveo U200, Ryzen 7 9700X, 8 host threads, 200MHz core, full mode, Q8_0 KV cache, flash attention (a technique that reduces memory used for attention computation). Prompt: 512 tokens; batch: 512 tokens; generation: 64 tokens; 3 runs; Basis 1.0.0.
| Model | Quantization, reducing numerical precision to make models lighter | Prompt (tokens/s) | Generation (tokens/s) | Card power (W) |
|---|---|---|---|---|
| Qwen3.8 27B | Q8_0 | 5.079 ± 0.001 | 1.587 ± 0.001 | 55.9 |
| Qwen3.5 9B | Q8_0 | 16.617 ± 0.004 | 4.992 ± 0.006 | 55.3 |
| Gemma 4 31B | Q4_K_M | 9.294 ± 0.003 | 2.442 ± 0.001 | 58.6 |
| Gemma 4 26B-A4B | Q8_0 | 20.942 ± 0.020 | 10.178 ± 0.013 | 55.8 |
| Gemma 4 12B | Q8_0 | 15.538 ± 0.011 | 3.743 ± 0.001 | 57.7 |
| Gemma 4 E4B | Q8_0 | 39.399 ± 0.022 | 9.034 ± 0.012 | 55.4 |
| Gemma 4 E2B | Q8_0 | 75.959 ± 0.091 | 17.442 ± 0.007 | 54.9 |
Generation speeds vary widely by model even with the same card settings. The project compared card cycles from a separate chat run with time per token and reported that card execution accounted for 98–99% of the time. In this setup, time spent on the card, rather than host processing, determines speed.
Only one process can use the card at a time. Because models and quantization differ, speed differences cannot be attributed solely to model size.
4. Benefits and costs of draft token verification
Multi-token prediction support was included in Basis 1.0.0, released earlier the same day.
Instead of finalizing one token at a time, multi-token prediction first produces several candidates and checks them together. For every tested candidate count across the three measured models, the project reported output identical to ordinary greedy decoding, which selects the most likely token at each step. The ordinary-generation baseline, the reference measurement for comparison, was also measured in the same process with the same prompts.
Table: Ordinary generation versus multi-token prediction
Conditions: 1 Alveo U200, Ryzen 7 9700X, 8 host threads, full mode, Q8_0 KV cache, flash attention, context length of 2048 tokens. Greedy generation of 128 tokens for each of 3 chat prompts; 1 run per candidate count; Basis 1.0.0.
| Model | Target/draft quantization | Optimal candidate count | Ordinary generation (tokens/s) | Multi-token prediction (tokens/s) |
|---|---|---|---|---|
| Qwen3.8 27B | Q4_K_M / Q8_0 (separate file) | 3 | 2.19 | 3.59 |
| Qwen3.8 2B | Q4_K_M / Q4_K_M (same file) | 1 | 23.06 | 25.45 |
| Gemma 4 26B-A4B | Q8_0 / Q8_0 (separate file) | 1 | 10.15 | 11.77 |
According to the project, the 27B model ran 1.64x faster with three candidates. The other two models were fastest with one candidate, with speedups of 1.10x and 1.16x, respectively. At optimal settings, accepted candidates per step were 2.09 for Qwen3.8 27B and 0.81 each for Qwen3.8 2B and Gemma 4 26B-A4B. The results indicate that the actual acceptance rate matters more than proposing more candidates.
The benefit is shorter generation time through batched candidate verification. The cost is extra draft generation and verification work, plus a separate draft weight file for some models. The documentation says that exceeding the optimal candidate count lengthens verification and reduces the gain. With four candidates, the 2B model was slower than ordinary generation.
The practical value of these releases is an inference path and validation tools for the U200, along with evidence for tuning draft candidate counts for each model.