Geometry inStyle: 3DStylization via SurfaceNormalDeformation
1University of Chicago, 2University of Southern California
Overview of an optimization iteration. Using the normals undergoing optimization as a target for our differentiable As-Rigid-As-Possible method (dARAP), the dARAP local step computes a rotation matrix per vertex; the dARAP global solve then finds the deformed surface from these local rotations. We use a differentiable renderer and a diffusion model-based semantic loss to guide the normals being optimized towards a deformation matching the desired style prompt.
Various input meshes towards a variety of text-specified styles. The style can manifest as fine geometric details, like in the ornate art deco column, or as low-frequency deformations, such as the joints of the cybernetic glove. Our method retains the structural features of the input shape, such as a flat arm on the antique sofa. Moreover, the resultant stylizations are in accordance with prompt semantics and part-aware semantics: the folds in the tropical chair are on the seat and backrest as opposed to the legs, the head of the penguin becomes like the top of a fire hydrant, and the racer bunny's thigh turns into the shape of a wheel.
The local step. Inspired by Normal-Driven
Spherical Shape Analogies, we represent a deformation by a target normal per vertex. These
normals are first used as the target for a local Procrustes solve, which finds the best rotation matrix
taking a bundle of edge vectors (a spokes-and-rims neighborhood of halfedges, plus the original normal) to the
same neighborhood but with the target normal. A λ hyperparameter controls the strength of the rotation
solution towards the target normal. This solve is made batched and differentiable.
The global step. Fixing the per-vertex rotations, we then find the best fit deformed
vertex positions. This least-squares optimization turns into a Poisson equation with the cotangent laplacian as
the system matrix, so we can use the differentiable solving technique from Neural Jacobian Fields.
One local step-global step pair. Our dARAP method is meant to be part of the "forward
pass" in a larger optimization pipeline. As such, we do not iterate the local and global steps as in classical
ARAP optimizations, instead relying on
gradient descent to optimize the deformation quantity (per-vertex normals) to attain the desired deformation in
one pass. We adjust the λ hyperparameter to allow the required strength in this one iteration.
Our method performs diverse shape stylizations that adhere to the target style prompt with high detail. We can deform the same source shape with different prompts; the different styles conform and adapt to the source geometry while adding salient geometric texture and detail to semantically appropriate parts. Target styles are also robust to different source shapes; even for shapes with very different geometry, the lego style is consistently conveyed with a lego brick-like surface pattern and cubified geometry.
Normals found by optimization using a particular λ value can be conveniently re-applied after optimization using a different λ to tune the stylization strength on demand. Both larger and smaller λ result in salient and clean stylizations at the required strength. The deformation region can also be controlled (either during or after optimization) by setting rotation matrices of vertices outside the region to identity before the global solve. We observe no boundary artifacts, showing dARAP's beneficial regularizing effects.