Factorized Motion Fields for Fast Sparse Input Dynamic View Synthesis

SIGGRAPH, 2024

Nagabhushan Somraj, Kapil Choudhary, Sai Harsha Mupparaju and Rajiv Soundararajan

Indian Institute of Science

Abstract

 Designing a 3D representation of a dynamic scene for fast optimization and rendering is a challenging task. While recent explicit representations enable fast learning and rendering of dynamic radiance fields, they require a dense set of input viewpoints. In this work, we focus on learning a fast representation for dynamic radiance fields with sparse input viewpoints. However, the optimization with sparse input is under-constrained and necessitates the use of motion priors to constrain the learning. Existing fast dynamic scene models do not explicitly model the motion, making them difficult to be constrained with motion priors. We design an explicit motion model as a factorized 4D representation that is fast and can exploit the spatio-temporal correlation of the motion field. We then introduce reliable flow priors including a combination of sparse flow priors across cameras and dense flow priors within cameras to regularize our motion model. Our model is fast, compact and achieves very good performance on popular multiview dynamic scene datasets with sparse input viewpoints.

Sample comparison videos

Play the videos in the fullscreen mode for the best view

Comparison with Competing Models

K-Planes vs RF-DeRF

Scene details: cook spinach from N3DV dataset with two input views.
Scene details: cook spinach from N3DV dataset with two input views. We show spiral video to show the improvement in learning 3D scene more clearly.
Scene details: N3DV dataset. In this video, we focus on the dog in the scene. We observe that K-Planes fails to render the eyes of the dog correctly and replaces it with a blur of its skin. On the other hand, our model is able to render the eyes of the dog better.

Depth Priors vs Flow Priors

Scene details: cook spinach from N3DV dataset with two input views.
Scene details: cook spinach from N3DV dataset with two input views.
Scene details: flame steak from N3DV dataset with two input views.
Scene details: sear steak from N3DV dataset with two input views.
Scene details:flame salmon from N3DV dataset with three input views.

Cross-Camera Dense Flow Priors vs Our Priors

Scene details: Birthday from InterDigital dataset. Observe the blur in the moving green ball on the right of the scene when using naive cross-camera dense flow priors. We also focus on the moving green ball to show the significant improvement in reconstruction quality when using our priors.
Scene details: flame salmon from N3DV dataset. We find that the left video suffers has a blue shade covering the entire scene and also contains significantly more floating clouds.

Comparisons with Ablated Models

without Sparse Flow Priors

Scene details: flame salmon from N3DV dataset with three input views.

without Dense Flow Priors

Scene details: flame steak from N3DV dataset with three input views. Observe the distortions on the right side of the window when not employing the within-camera dense flow prior.

With Dense Input Views

N3DV Dataset

Scene details: coffee martini from N3DV dataset.
Scene details: cook spinach from N3DV dataset.
Scene details: flame steak from N3DV dataset.

InterDigital Dataset

Scene details: Birthday from InterDigital dataset.
Scene details: Painter from InterDigital dataset.
Scene details: Train from InterDigital dataset.

Citation

 If you use our work, please cite our paper:
Nagabhushan Somraj, Kapil Choudhary and Sai Harsha Mupparaju and Rajiv Soundararajan, "Factorized Motion Fields for Fast Sparse Input Dynamic View Synthesis", In Proceedings of the ACM Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH), Jul 2024, doi: 10.1145/3641519.3657498.
Bibtex:
@inproceedings{somraj2024rfderf,
    title = {Factorized Motion Fields for Fast Sparse Input Dynamic View Synthesis},
    author = {Somraj, Nagabhushan and Choudhary, Kapil and Mupparaju, Sai Harsha and Soundararajan, Rajiv},
    booktitle = {ACM Special Interest Group on Computer Graphics and Interactive Techniques (SIGGRAPH)},
    month = {July},
    year = {2024},
    doi = {10.1145/3641519.3657498}
}