3DL threedle logo

Geometry inStyle: 3DStylization via SurfaceNormalDeformation

Nam Anh Dinh1 Itai Lang1 Hyunwoo Kim1 Oded Stein2 Rana Hanocka1
1University of Chicago, 2University of Southern California
a 3d render of…
a pineapple-themed vase
an A-pose knight in armor
a cute animal-themed chair
a lego goat

Guided by a text prompt, Geometry in Style produces an identity-preserving stylization of mesh geometry. The deformation is represented and optimized via per-vertex normals, solved into vertex displacements in an inherently regularizing manner using dARAP, a differentiable one-step adaptation of classical As-Rigid-As-Possible deformation.

Optimization overview

process diagram. gray vase shape (labeled 'input mesh'), with zoomed in inset showing local vertex neighborhood at vertex k, having a current normal u_k and target normal u_k hat with a depiction of rotation R_k hat from current to target. inset zooms back out to show R_k hat as part of a vector of R from all vertices stacked (labeled 'local rotations'). this stack goes to a 'global solve' step illustrated as the blue pineapple styled vase. depictions of cameras surround the right side, next to a column of images labeled 'differentiable rendering'; this result goes to a 'semantic loss' block with the prompt going into it; the loss is backpropagated to target normals from the vertex neighborhood

Overview of an optimization iteration. Using the normals undergoing optimization as a target for our differentiable As-Rigid-As-Possible method (dARAP), the dARAP local step computes a rotation matrix per vertex; the dARAP global solve then finds the deformed surface from these local rotations. We use a differentiable renderer and a diffusion model-based semantic loss to guide the normals being optimized towards a deformation matching the desired style prompt.

Gallery

gallery of results, source shapes in gray and deformed in blue. a 3d render of... antique sofa, tropical chair, racing chair, greek statue chair, gothic chair, cardboard chair, cybernetic glove, skull, dinosaur statue, ornate art deco column (from a candle as source shape), penguin themed fire hydrant, racer bunny (with wheels for thighs)

Various input meshes towards a variety of text-specified styles. The style can manifest as fine geometric details, like in the ornate art deco column, or as low-frequency deformations, such as the joints of the cybernetic glove. Our method retains the structural features of the input shape, such as a flat arm on the antique sofa. Moreover, the resultant stylizations are in accordance with prompt semantics and part-aware semantics: the folds in the tropical chair are on the seat and backrest as opposed to the legs, the head of the penguin becomes like the top of a fire hydrant, and the racer bunny's thigh turns into the shape of a wheel.

Normals and differentiable ARAP

diagram showing procrustes local step; local neighborhood with vertex and neighboring edges and one-ring; vertex has current normal and target normal pointing out of it; panel showing the neighborhood of spokes-and-rims edges and the current normal, right arrow, question mark R in SO(3) (illustrating solving the procrustes problem), towards a panel showing the same neighborhood but with target normal. captioned 'local orthogonal procrustes problem'


The local step. Inspired by Normal-Driven Spherical Shape Analogies, we represent a deformation by a target normal per vertex. These normals are first used as the target for a local Procrustes solve, which finds the best rotation matrix taking a bundle of edge vectors (a spokes-and-rims neighborhood of halfedges, plus the original normal) to the same neighborhood but with the target normal. A λ hyperparameter controls the strength of the rotation solution towards the target normal. This solve is made batched and differentiable.


The global step. Fixing the per-vertex rotations, we then find the best fit deformed vertex positions. This least-squares optimization turns into a Poisson equation with the cotangent laplacian as the system matrix, so we can use the differentiable solving technique from Neural Jacobian Fields.

One local step-global step pair. Our dARAP method is meant to be part of the "forward pass" in a larger optimization pipeline. As such, we do not iterate the local and global steps as in classical ARAP optimizations, instead relying on gradient descent to optimize the deformation quantity (per-vertex normals) to attain the desired deformation in one pass. We adjust the λ hyperparameter to allow the required strength in this one iteration.

Expressiveness and specificity

same-shapes deformed towards different prompts. first row: person shape, prompts 'Chinese terracotta warrior', 'balloon person', 'samurai warrior in traditional armor'; second row basic chair shape, prompts are 'origami chair', 'butterfly-themed chair', 'church pulpit'; third row candle shape, prompts are 'retro game joystick', 'braided pillar candle', 'upright fountain pen' shapes deformed towards the same lego prompt, three rows. first row: three chairs deformed with 'lego chair' prompt, variety of geometries made into blocky lego style. second row: animals, camel dog pig, deformed towards lego corresponding animal name. third row: three vases deformed towards 'vase made of lego bricks'

Our method performs diverse shape stylizations that adhere to the target style prompt with high detail. We can deform the same source shape with different prompts; the different styles conform and adapt to the source geometry while adding salient geometric texture and detail to semantically appropriate parts. Target styles are also robust to different source shapes; even for shapes with very different geometry, the lego style is consistently conveyed with a lego brick-like surface pattern and cubified geometry.

Controllable strength and region

source gray lion shape deformed towards 'a 3d render of a robot lion' prompt; three deformed results, first one at lambda=5 is a subtle robot style with sharp edges on body but smooth; second one at lambda=8 (lambda used at optimization) is a nice salient robot style with mechanical legs and joints and waist; lambda=11 very prominent robot style with exaggerated sharp features from the previous ones, with sharp mechanical looking parts and joints source cartoon cow shape with the head region highlighted in yellow; prompt 'a 3d render of a bulldog', deform result shows the head region turned into a bulldog head, rest of body unchanged

Normals found by optimization using a particular λ value can be conveniently re-applied after optimization using a different λ to tune the stylization strength on demand. Both larger and smaller λ result in salient and clean stylizations at the required strength. The deformation region can also be controlled (either during or after optimization) by setting rotation matrices of vertices outside the region to identity before the global solve. We observe no boundary artifacts, showing dARAP's beneficial regularizing effects.