Intel Dokimi 0.2.0: CLI restructuring, stricter evaluation gates

1. Command names moved into functional groups

Running an old command displays a message naming its replacement and exits with code 2. For example, eval score, which calculated scores, is now eval score pointwise.

Evaluation and scoring are grouped under eval, while resource listings and checks move to catalog. Tracing and data input/output also move to separate groups. Automation scripts must replace old names and review follow-up handling based on exit codes.

2. Missing measurements and invalid scores cause evaluation failures

Task runs, evaluation checks, and repository runs return exit code 1 if planned evaluation items produce errors, remain unscored, or are missing. Document checks and wording and agreement checks for judges, the models that score answers, apply this rule in strict mode, which enforces failure criteria rigorously. The focus shifts from whether evaluation ran to whether it actually produced scored results.

NaN, a value defined as not a number, and infinite scores cannot pass either. Document checks treat NaN intervals as failures in strict mode. Baseline comparisons against earlier evaluation results also fail in strict mode if a model or provider is missing or marked unknown.

Checks that treated empty inputs as successes have also changed. A catalog root, the top-level folder where resource discovery starts, without SKILL.md returns exit code 1, as does a report target with no evaluation results. Benchmark preflight checks return exit code 2 if a file-based input is missing, unreadable, or empty.

3. Exit codes and comparison results also have new meanings

The basic exit-code scheme is 0 for success, 1 for check, gate, or execution failure, and 2 for invalid input or usage. Repository runs and evaluation checks also return 1 for some invalid inputs, so automation must consider which command was run.

The tool preference command returns 3 and retains partial results when stopped by its budget. Other commands return 1 on budget stops. Backend, run-history, and tracing commands return 5 on abnormal termination in JSON output mode.

A sign-flip test on per-question score differences is used to calculate p-values and estimate intervals for repository comparisons. It evaluates the observed difference by flipping the direction of each question's difference, while interval estimates express score differences as ranges rather than single values. Reports show the test's p-value and the number of pairs whose scores changed.

Repository comparisons also show the method used to calculate differences, with interval values left empty when there are 5 or fewer tasks. Differences from the baseline are also empty if no questions were scored in both evaluation groups. When reading comparisons, users can check for paired scored results alongside the size of the difference.

4. Weigh new validation features against rerun costs

New features inspect repository budgets and evaluation coverage. Optional gates can enforce limits on baseline regressions and minimum counts of paired results. Developers checking multiple models or repository changes can broaden evaluation coverage while setting criteria for acceptable changes.

These additional validation tools may bring repeat-call costs for tasks that previously reused responses. Bedrock and Vertex no longer reuse existing cached responses, saved from earlier calls, for judges, role settings, question generators, or validators. OpenRouter is affected too: response cache keys now include whether streaming is enabled, meaning responses are sent in real-time chunks, so those calls incur another charge.

Output settings have also changed. OpenRouter's output token cap of 2000 has been removed, and novelty evaluation calls request 16384 output tokens unless --max-tokens is specified. The release notes state that litellm previously requested 4096 tokens on Bedrock.

5. Check baselines and keys first when migrating

Scores and verdicts cannot be compared directly with 0.1.0 results, so baselines must be rerun.

If the following command was run with OPENROUTER_API_KEY set, the validator sent the OpenRouter key as a Google key, so that key must be rotated.

eval questions generate --validate --provider google-vertex

Dokimi 0.2.0 prevents empty evaluations from passing and expands validation tools, but requires command migration, baseline reruns, and extra costs for some calls.