We present a neural technique for learning to select a local sub-region around a point which can be used for mesh parameterization. The motivation for our framework is driven by interactive workflows used for decaling, texturing, or painting on surfaces. Our key idea is to incorporate segmentation probabilities as weights of a classical parameterization method, implemented as a novel differentiable parameterization layer within a neural network framework. We train a segmentation network to select 3D regions that are parameterized into 2D and penalized by the resulting distortion, giving rise to segmentations which are distortion-aware. Following training, a user can use our system to interactively select a point on the mesh and obtain a large, meaningful region around the selection which induces a low-distortion parameterization.
DA Wand aims to map a user selection point to a local segmentation which induces a low distortion parameterization. We showcase some example segmentations with parameterizations computed with SLIM.
We leverage a lightweight fully convolutional architecture, which enables DA Wand to produce predictions in (near) realtime. We provide an interactive demo in the repo, which can be run on any mesh. Below we show some examples run on a Macbook without GPU acceleration.
DA Wand consists of an edge encoder ϕE which uses the MeshCNN architecture and a segmentation classifier ϕC which is a 3-layer MLP. We train DA Wand with the explicit objective of producing minimal-distortion, large local segmentations with compact boundary. We achieve this by first pretraining on a synthetic dataset, then extending the learning to natural shapes through a novel differentiable parameterization layer ϕP. This layer computes a weighted UV embedding for the network-predicted segmentation on the fly during training. The UV map is sent into a distortion function of choice D, which is used in the thresholded-distortion loss Lthreshold. The smoothness loss Lsmooth enforces a compactness prior on the network prediction, which helps curtail non-contiguous segmentations.
@article{liu2022dawand,
author = {Liu, Richard and Aigerman, Noam and Kim, Vladimir G. and Hanocka, Rana},
title = {DA Wand: Distortion-Aware Selection using Neural Mesh Parameterization},
journal = {arXiv},
year = {2022},
}