Perplexity pplx-decider-v1.2: Grouped questions, restructured training data

1. No separate run for each question

This decision model returns judgments on given choices rather than generating sentences.

The previous version ran the model for each question. The new version appends questions to the context and reads answers at question-specific marker positions. Questions share the context and can also reference one another.

The training data adds official training splits from some benchmarks, deduplicated against test sets, and other decision data.

2. Consider score gains alongside training coverage

The self-reported aggregate score on the public Decision Index 0.3 rose from 62.30 for the previous model to 68.09 for the new model with grouped questions. Sending questions separately scores 67.75. The aggregate uses evaluation-suite weights, not a simple average across categories.

On the public Vision Decision Index 0.3, which uses images up to 1.6 MP, the score rose from 73.22 for the previous model to 77.43 for the new model.

Some evaluation tasks now fall within the training scope. Improvements are also reported on decision tasks excluded from training, but detailed scores were not provided.

The probability calibration factor fitted on separate data for grouped questions is 1.382. Accuracy on 1,497 development-set questions is 91.3%. ECE, which measures the gap between predicted probabilities and actual accuracy, fell from 0.033 before calibration to 0.014. Some overconfidence remained on decision data excluded from training, with ECE around 0.05.

3. Group related attributes, separate independent candidates

Grouping helps when assessing multiple attributes of the same item. In the same evaluation, grouped requests versus individual requests gained +11.3 points on SATA-Bench, which allows multiple correct answers. They lost −4.8 points on ToolRet, which asks about each candidate tool. Independent candidates can interfere with one another's judgments.

BEIR, a document reranking evaluation suite, used nDCG@10, a metric for ranking quality in the top 10 results.

Including 8 candidate documents per request rather than 1 gained +2.1 points on HotpotQA, which requires connecting multiple pieces of evidence. However, the average across all 13 datasets was −0.5 points. First determine whether comparing candidates itself helps.

4. Fast execution does not exactly match baseline results

Grouping questions within a request differs from batching separate requests. The fast path preserves independence between requests even when processing separate requests together.

Measurements used BF16, a 16-bit numerical format, on a single GB200. Decision evaluation requests averaged 1,050 tokens, the small units the model uses to count input. Throughput rose from 9.1 requests per second with the baseline, which groups by length and pads inputs, to 19.8 with the default fast path. Median individual request latency fell from 131 milliseconds to 96 milliseconds. The baseline already groups requests by length, so simple batching offers little gain. Speedups come from computation compilation and execution reuse through CUDA graphs.

Throughput and latency improve, but numerical differences remain. Across 300 requests and 2,324 questions from the same evaluation suite, top-choice agreement between the fast and baseline paths was 99.8%. The mean absolute probability difference was 0.002. The baseline itself showed 99.8% choice agreement and a mean absolute probability difference of 0.0017 between batched and individual processing. Fast-path differences are similar in scale to variation caused by the baseline's own processing methods.

5. Check the runtime environment and input limits

Like its predecessor, pplx-decider-v1.2 is based on Qwen3.8-27B and accepts text and images up to 1.6 MP.

Running it directly requires Python 3.12 or later and a CUDA GPU.

Memory requirements include about 52 GiB for BF16 weights plus workspace.

The README describes the repository as private and instructs users to authenticate with an authorized Hugging Face token. However, the model page metadata shows private:false and gated:false. These conflicting source instructions leave actual access restrictions uncertain. After downloading, install the base dependencies and add dedicated dependencies if using the fast path.

The following is the source's default fast-path example. Prepare the model path and request list before running it.

from pplx_decider import Decider

decider = Decider.from_pretrained(checkpoint, fast=True)
decider.warmup()
responses = decider.decide(requests)

The default length limit is 262,144 tokens, but training only extends to about 32k tokens. Validate longer inputs on your own tasks. Inputs exceeding the configured limit are split by question rather than truncated, with context and images repeated in each chunk. If even one question cannot fit, an error is raised before execution.

The serving environment needs an implementation supporting bidirectional references between questions and per-question answer reading.

Ollama's v0.40.7-rc0 prerelease added support for this model.

According to the [pull request author's notes](https://github.com/ollama/ollama/pull/19072), the model for the MLX runtime library had been uploaded at that point. However, the manifest collecting deployment information had not yet been created. Distinguish support code from deployment readiness.

Deciding whether to adopt this model should start with separating questions that should reference one another from candidates that need independent evaluation, rather than grouping as much as possible.