llama.cpp b12098: Qwen3-VL video format support, aligned frame sampling

1. Automatically selecting model-specific video input

Released on November 4, 2026, b12098 supports Qwen3-VL's video prompt format. This release changes how selected video frames and timestamps are passed to the model.

The change applies to mtmd, a module handling multiple input formats, including video and audio. When it detects a Qwen3-VL model, it selects that model's video settings. Frame selection and timestamps follow those settings. Developers rely on a model-specific processing path rather than assembling the format themselves.

2. Placing timing information between frames

Video frames are sent in pairs, with a timestamp before each pair. Like placing a capture time before two photos, this gives the pair's midpoint time first. In the test notation below, bracketed numbers are source video frame indices.

The PR tests use fixed inputs for comparison with results from Qwen3VLProcessor in transformers 4.57.1. The code below shows part of a test's expected string, specifying input frame dimensions and the timestamp and frame order.

        { {  720, 358, 24.0f, 240 }, { 704, 352 },
            "<0.3 seconds>[0][13]<1.3 seconds>[25][38]<2.4 seconds>[50][63]<3.4 seconds>[75][88]<4.5 seconds>[101][113]"
            "<5.5 seconds>[126][138]<6.6 seconds>[151][164]<7.6 seconds>[176][189]<8.6 seconds>[201][214]<9.7 seconds>[226][239]" },

3. Rules for selecting frames across the entire video

The settings calculate the selection count from video duration at 2 frames per second. Qwen3-VL video processing limits this to 4–768 frames, or the source count if fewer are available. Both the per-second rate and the total count limit determine input size.

Once the count is set, indices are spaced evenly from the first source frame to the last. When a long video hits the limit, the same number of frames represents a longer time span. This selects scenes across the entire video rather than sending only its beginning.

4. Adjusting frame dimensions to the frame count

Frame dimensions are calculated the same way as in the reference processor, which generates reference inputs. Width and height align with the model's grid spacing and scale to pixel limits that account for frame count. Tests check expected dimensions for each input using shared settings of patch_size 16 and n_merge 2.

The table below shows test cases with different inputs under the same resizing rules. Width and height are in pixels, and the frame count is the value passed to the size calculation.

Table: Expected results for different inputs under shared resizing rules

Conditions: patch_size 16, n_merge 2, Qwen3VLVideoProcessor smart_resize rules, pixel minimum 4096 and maximum 25165824, temporal group size 2

Unit: Frame counts in frames; width and height in pixels

Test comments cover cases that first upscale a shorter side below the grid spacing and downscale dimensions exceeding the pixel limit. Cases with many frames show a design that lowers per-frame resolution to manage total input size.

5. Trading video detail for reference format alignment

This change aligns timestamps, frame sequences, and dimensions with Qwen3-VL's reference processing. The trade-off is frame selection and resizing. Longer videos have wider gaps between selected scenes, while pixel limits also reduce detail.

Built on the existing LFM3 video processing structure, the implementation makes input order reproducible in tests. The new validation component is marked for internal use, not direct use by applications.

The table and code values check the reference processor's expected results. They are not measurements of model response quality or processing speed.

The core of b12098 is a path that feeds Qwen3-VL video frames and timing information in the reference format.