PhyCo: Learning Controllable Physical Priors for Generative Motion

Sriram Narayanan1,2 · Ziyu Jiang2 · Srinivasa G. Narasimhan1 · Manmohan Chandraker2,3

1 Carnegie Mellon University · 2 NEC Labs America · 3 UC San Diego

CVPR 2026
Paper arXiv Code Simulation Code Dataset Twitter BibTeX

TL;DR PhyCo learns controllable physical priors — friction, restitution, deformation, and force — from simple block-sliding and ball-bouncing simulations, enabling physically grounded and continuously controllable video generation without any simulator at inference.

How PhyCo Works

A two-stage pipeline: physics-supervised ControlNet fine-tuning on simulation data, followed by VLM-guided reward optimization for physical consistency.

PhyCo two-stage pipeline: ControlNet physics fine-tuning followed by VLM reward alignment

100K+ Simulation Videos

Photorealistic block-sliding, ball-bouncing, and collision videos rendered with Kubric & PyBullet, with systematically varied physical properties.

Physics-Supervised ControlNet

A ControlNet conditioned on pixel-aligned physical property maps is trained on top of a frozen Cosmos-Predict2 video diffusion backbone.

VLM-Guided Reward

A fine-tuned Qwen2.5-VL evaluates generated videos with targeted physics questions, providing differentiable feedback to improve consistency.

Fine-Grained Control of Physical Attributes

Compositionality of Physical Attributes

Generalization to Different Styles

Scene 1
Scene 2
Scene 3
Scene 4
Prompt:
Scene 4
Scene 3
Scene 2
Scene 1
Prompt:

BibTeX

@InProceedings{Narayanan_2026_CVPR,
    author    = {Narayanan, Sriram and Jiang, Ziyu and Narasimhan, Srinivasa and Chandraker, Manmohan},
    title     = {PhyCo: Learning Controllable Physical Priors for Generative Motion},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2026},
    pages     = {41892-41902}
}

Acknowledgements

This work was partially conducted during Sriram's internship at NEC Labs America, and was supported in part by NSF grants IIS-2107236 and IIS-2513219. We also thank Kausik Sivakumar and Yug Ajmera for their insightful discussions.