Abstract

NeuROK proposes Neural Object Kinematics, a data-driven approach to generating realistic 4D dynamics (temporal deformations) of static 3D objects under physical conditions. Unlike prior methods requiring predefined physical models and system identification, NeuROK learns a latent kinematic state space and decoder that maps latent states to plausibly deformed shapes, using a transformer-based encoder-decoder trained on a curated large-scale 4D dataset. The approach simplifies dynamics generation by operating in a low-dimensional latent space from a Lagrangian mechanics perspective.

Key Contributions

- Neural Object Kinematics parameterization: Learns a latent space representing all possible object states plus a decoder mapping any latent sample to a deformed shape, avoiding predefined physical models

  • Transformer-based encoder-decoder architecture: Trained on a large-scale 4D dataset to capture diverse dynamic object types
  • Lagrangian mechanics formulation: Generates simulative dynamics by considering dynamics within a low-dimensional latent space, applicable across diverse object categories

    Method Details

    The method learns both:

    • 1. A latent space representing all possible kinematic states of an object 2. A decoder mapping any sampled latent state to a plausibly deformed shape

    The model is a transformer-based encoder-decoder trained on a curated large-scale 4D dataset with diverse dynamic object types. The formulation treats dynamics generation as sampling from the latent space rather than simulating from first principles.

    Key Results

    - Demonstrated across diverse dynamic object types

  • Shows clear advantages over prior works in generating simulative 4D dynamics
  • Project page: https://chen-geng.com/neurok

    Limitations and Future Work

    - Limited to object-centric physical systems (single object focus)

  • Relies on curated 4D training data quality and coverage
  • Future work could extend to multi-object interactions and more complex physical phenomena

    Relevance to Patrick's Research

    NeuROK addresses a key challenge in world models: generating physically plausible temporal deformations of objects. The data-driven kinematic approach provides an alternative to explicit physics simulation, relevant for world models that need to predict object dynamics over time. The CVPR 2026 acceptance indicates strong interest in neural approaches to physical world modeling.