EvalScope 1.14.1 adds SafetyBench safety evaluation and configuration preflight checks. It fixes a bug that reused old scores as new results after rescoring failed. Developers evaluating models gain a new safety evaluation path. When rescoring existing results, they can reuse prediction data while checking score status.

1. Safety evaluation without examples and pre-run checks

This version adds safety evaluation through SafetyBench. It asks questions without first showing examples of correct answers, supporting Chinese, English, and a Chinese subset. It reports overall accuracy and accuracy across seven categories, allowing users to examine both overall and category-level results.

Unparseable predictions and missing responses count as incorrect answers. No arbitrary answer is chosen to fill the gap, so failures to produce a valid response are reflected in accuracy.

The command-line --dry-run parses and prints the configuration, then exits without inference, the process in which a model generates answers from inputs. Users can inspect their configuration in this mode before starting a costly evaluation. This separates configuration checks from actual model execution.

2. Stored scores can be reused only after rescoring finishes

A cache stores previously computed results for reuse in later runs. Previously, if rescoring with rerun_review=True failed, a normal run could silently reuse old scores. This meant reusing an old scorecard before the new scoring had finished.

The fix saves the configuration atomically and records that rescoring is still incomplete. It also preserves information about where the original predictions came from. Prediction provenance is retained while the completion status of new scoring is recorded separately.

Failed or interrupted rescoring blocks normal cache reuse. Users must complete rescoring with rerun_review=True or --rerun-review. Prediction files remain reusable, but old scores cannot reappear as new evaluation results.

3. Corrected inputs and aggregation scope for benchmarks, standard evaluations for comparing model performance

Several benchmarks have changed their inputs or scoring methods. VQAv2 now uses official scoring, and ScienceQA includes question hints in evaluation inputs. These changes mean that even for the same model, users need to consider both the information supplied and the scoring rules applied.

ARC-AGI-2 evaluates all test inputs for each task and aggregates results by task. MMAU uses only the subset with publicly available answers and removes the fallback that defaulted answers to A. These changes correct evaluation scope and answer-handling rules.

For speech recognition, edit counts and reference word counts are each summed across the full dataset for datasets such as LibriSpeech. Word error rate is calculated by dividing the edit count by the reference word count. TORGO reports word error rate and character error rate, the proportion of incorrectly recognized characters, across the full dataset. Errors are aggregated over the entire evaluation dataset.

The video captioning evaluations MSR-VTT and MSVD also calculate sentence evaluation scores across the full subset. The fix prevents scores from varying with scoring batch size, separating processing batch size from scoring scope.

4. An upgrade that distinguishes prediction reuse from reruns

When comparing old and new results, check evaluation_version, which identifies the evaluation version. Input instruction changes, as in ScienceQA or SPaRC, require new predictions, while scoring-only changes allow stored predictions to be rescored. Rather than applying the same rerun procedure to every change, determine whether inputs or scoring have changed.

The benefit is preserving stored predictions while preventing old scores from failed rescoring from being mixed in as new scores. However, interrupted rescoring must be completed, and evaluations with changed inputs incur the cost of generating answers again. How much existing data can be reused becomes a basis for upgrade planning.

Installation has also changed. LiteLLM is no longer a core installation dependency, an external program or library required for execution. Users of that backend, a component that handles execution for particular functions, must install it separately with the following command. This moves a dependency previously installed through the default integration path to the installation step for the selected backend.

pip install litellm

This release expands safety evaluation while refining scoring fixes and failure recovery to protect the reliability of result comparisons.