OneVision Encoder: Codec-Aligned Sparsity as a Foundational Principle for Multimodal Intelligence

Models · Tech Report · Model Card · Data Card

LMMs Lab, Glint Lab, AIM for Health Lab, and MVP Lab

Introduction

Hypothesis. Artificial general intelligence is, at its core, a compression problem. Effective compression demands resonance: deep learning scales best when its architecture aligns with the fundamental structure of the data. These are the fundamental principles. Yet, modern vision architectures have strayed from these truths: visual signals are highly redundant, while discriminative information, the surprise, is sparse. Current models process dense pixel grids uniformly, wasting vast compute on static background rather than focusing on the predictive residuals that define motion and meaning. We argue that to solve visual understanding, we must align our architectures with the information-theoretic principles of video, i.e., codecs.

Method. OneVision Encoder encodes video by compressing predictive visual structure into semantic meaning. By adopting Codec Patchification, OneVision Encoder abandons uniform computation to focus exclusively on the 3.1%-25% of regions rich in signal entropy. To unify spatial and temporal reasoning under irregular token layouts, OneVision Encoder employs a shared 3D RoPE and is trained with a large-scale cluster discrimination objective over more than one million semantic concepts, jointly capturing object permanence and motion dynamics.

Evidence. The results validate our core hypothesis: efficiency and accuracy are not a trade-off; they are positively correlated. By resolving the dichotomy between dense grids and sparse semantics, OneVision Encoder redefines the performance frontier. When integrated into large multimodal models, it consistently outperforms strong vision backbones such as Qwen3-ViT and SigLIP2 across 16 image, video, and document understanding benchmarks, despite using substantially fewer visual tokens and pretraining data. Notably, on video understanding tasks, OneVision Encoder achieves an average improvement of 4.1% over Qwen3-ViT. Under attentive probing, it achieves state-of-the-art representation quality, with 17.1% and 8.1% Top-1 accuracy improvements over SigLIP2 and DINOv3, respectively, on Diving48 under identical patch budgets. These results demonstrate that codec-aligned, patch-level sparsity is not an optimization trick, but a foundational principle for next-generation visual generalists, positioning OneVision Encoder as a scalable engine for universal multimodal intelligence.

Codec-Style Patch Selection

Traditional video understanding models process frames by uniform temporal sampling, selecting evenly spaced frames regardless of content. This approach treats all spatial regions equally, wasting computation on redundant background pixels that remain static across frames.

Inspired by HEVC video compression, our codec-style approach identifies and processes only the patches that carry meaningful temporal changes. Just as video codecs encode motion vectors and residuals rather than full frames, we select patches based on their information density, preserving the dynamic, semantically rich regions while discarding redundant static content.

Codec-Style Input

The reference frame contains all patches. Consecutive frames retain only salient patches at their spatial positions. The result is 75%-98% fewer patches while retaining the information that matters.

Traditional Frame Sampling

Traditional sampling selects a small number of frames and processes every patch from each. Static backgrounds, repeated textures, and unchanging regions are therefore processed multiple times even though they add no new information.

Video Processing Pipeline

The visualization below demonstrates four stages:

1. Original Video: a continuous 64-frame stream retaining the full temporal context.
2. Uniform Frame Sampling: the traditional approach selects 4-8 evenly spaced frames, which is simple but lossy and misses inter-frame motion.
3. Temporal Saliency Detection: all 64 frames are analyzed for motion, appearance changes, and semantic events.
4. Codec-Style Patch Extraction: salient patches are extracted in zigzag order, achieving 75%-98% compression while preserving temporal dynamics.

The complete pipeline progresses from the original video to a codec-style compressed representation, identifying temporally salient patches while preserving rich motion information. Additional examples are available as Case 2, Case 3, Case 4, Case 5, Case 6, and Case 7.

Global Contrastive Learning

Standard contrastive learning, such as CLIP, is limited by batch size: negative samples are drawn only from the current batch, typically 32K-64K examples. This creates a narrow view of the embedding space and leads to suboptimal representations. Our approach maintains a global concept bank of 2M clustered centers, enabling each training sample to contrast against a diverse, representative set of negatives regardless of batch composition. This produces more discriminative embeddings with better-separated semantic clusters.

Experimental Results

LMM Probe Results

The following table compares vision encoders on multimodal benchmarks. All models use Qwen3-4B-Instruct-2507 as the language backbone. OV-Encoder-Lang is the language-aligned variant; Qwen3-ViT is extracted from Qwen3-VL-4B; and SigLIP2 uses siglip2-so400m-patch16-naflex. Codec denotes codec-guided visual encoding using motion vectors and residual signals, while Frame denotes dense spatial patchification. Bold values are the best results under the same evaluation setting.

Results in the first pair of model columns use caption supervision; results in the remaining encoder columns do not.

Attentive Probe Results

Models are evaluated with a single clip and trained for 10 epochs across eight action-recognition datasets. OV-Encoder (Codec) replaces traditional frame sampling with codec-guided patch reorganization without changing the backbone architecture or training protocol. Under the same token budget, this improves motion-sensitive datasets such as Diving48 and Perception Test.

8 Frames

16 Frames

Patch-Efficient Video Understanding Comparison

This comparison fixes the source video at 64 frames, or 16,384 patches. SigLIP2 uses traditional frame sampling, where every 256-patch group is a contiguous RGB frame. OneVision Encoder instead uses codec-native motion vectors and residuals to distribute a fixed patch budget across the entire temporal extent without temporal downsampling.

Percentages indicate patch reduction relative to dense processing of all 16,384 patches. Under a fixed token budget, codec-style selection redistributes patches across time while retaining spatial positions. It outperforms SigLIP2 on Diving48 and Perception Test while reducing patch processing by 75.0%-96.9%.

BibTeX

@article{onevision_encoder_2026,
  title={OneVision Encoder},
  author={LMMs Lab, Glint Lab, AIM for Health Lab, MVP Lab},
  journal={arXiv preprint},
  year={2026}
}

If you find this work useful, please consider citing the paper.