On October 1 Ai2 released Olmo-core 3. It is a framework with a redesigned training system for mixture-of-experts (MoE, a model design that uses only some expert modules per input).
Ai2 presented it as a core foundation for the next generation of Olmo. The announcement includes no newly trained model weights.
In one test, the expert pool grew from 8 to 128 while only four experts were selected per token. Active parameters stayed at about 3.2B, total parameters grew from 4.6B to 47B, and training throughput reportedly fell by less than 5%.
The earlier MoE implementation used fully sharded data parallelism (FSDP) to gather and reshard weights for each small batch. Olmo-core 3 switches to distributed data parallelism (DDP), keeping experts resident on GPUs and routing data to them.
In Ai2's preliminary test on eight NVIDIA B300 GPUs, a 47B-parameter MoE processed 52,000 tokens per second per GPU. That is about 2.7× the 19,400 tokens of the earlier implementation.
The model and training state are split across GPUs with expert parallelism, pipeline parallelism and a distributed optimizer. The distributed optimizer spreads the optimizer state used for updates across GPUs instead of keeping a full copy on each.
It also supports MXFP8, a lower-precision format. In a four-B300 test, throughput was about 21% higher than BF16, and peak active memory fell from 103GiB to 95GiB.
The code is published in the Olmo-core repository on GitHub under the Apache License 2.0.
Caveats: system measurements versus model quality
Ai2 tested a 1.2-trillion-parameter model with 58.36B active per token on 512 B300 GPUs and observed up to 858 TFLOP/s/GPU. The test used random routing to measure system performance, not the quality of a trained model.
The 2.38-trillion-parameter configuration reached with DeepEP v2 was a short-capacity test, not a full training run. Ai2 also described it as showing reachable scale rather than sustained training performance.
All figures are Ai2's measurements and were not reproduced for this article. Ai2 itself reported that overlapping communication and computation slowed end-to-end execution in some tests.
Metrics to weigh together when evaluating the implementation
A direct evaluation can build a small representative training job with the target routing, precision and network, and compare memory with effective throughput.
Capacity figures or individual kernel speeds alone make it hard to judge whether full training is practical.
Olmo-core 3's trillion-scale figures show the system limits of an open training stack, and the quality of the next Olmo is still a separate question.