Beyond Prompts: Unconditional 3D Inversion for Out-of-Distribution Shapes
Published:
Victoria Yue Chen, Emery Pierson, Léopold Maillard, Maks Ovsjanikov
TL;DR
We study text-driven inversion of 3D generative models. We find the existence of sink traps: the model can become insensitive to prompts during generation, effectively collapsing to a single shape. Despite this property, the models retain strong geometric expressiveness in the unconditional distribution. We demonstrate this finding by proposing a novel pose-retargeting editing pipeline using our unconditional 3D inversion on out-of-distribution shapes.
Expressivity of Language and Geometry
Sink trap examples. Sink trap examples. Given various description of a character (dancing girl, surgeon, labrador, astronaut, scary wolf) in different poses, we generate multiple assets using TRELLIS. However, we observe a mode collapse where there is high similarity between the results, despite different prompt describing different actions.
Geometric expressivity. The velocity norm of Flux remains stable across different prompt types (left), whereas TRELLIS exhibits large variations when inputted various language prompt. This property is not true when dealing with empty prompts (unconditional distribution). We can invert perfectly with the unconditional distribution, whereas it fails using the conditional distribution.
Application: inversion based character retargeting
We demonstrate the efficiency of the unconditional inversion with a practical application: inversion based character retargeting. Given an out of distribution shape with a certain pose, we retarget the specific pose to a new character, showing superior performances against baselines (Interactive examples).
Acknowledgements
Parts of this work were supported by the ERC Consolidator Grant VEGA (No. 101087347), as well as gifts from Ansys Inc., and Adobe Research.
Citation
If you consider our work useful, please cite:
@misc{incoming}
This webpage was inspired by Nerfies.