Mesh Saliency Detection Using Local Curvature Entropy for Large-Scale Model Decimation
•Download as PPTX, PDF•
0 likes•45 views
This document discusses methods for calculating mesh saliency maps to guide large-scale mesh decimation for artist workflows. It proposes using the entropy of tropical angle of curvature at multiple scales as a robust saliency measure. An adaptive subsampling approach allows it to be calculated quickly while maintaining accuracy. Testing on models up to 150k vertices showed it outperforms previous methods in both speed and reliability. Feedback from game studios indicated it provides artists sufficient control over decimation results through adjustible saliency maps.
Recommended
Recommended
More Related Content
Similar to Mesh Saliency Detection Using Local Curvature Entropy for Large-Scale Model Decimation
Similar to Mesh Saliency Detection Using Local Curvature Entropy for Large-Scale Model Decimation (20)
More from Joaquim Jorge
More from Joaquim Jorge (20)
Recently uploaded
Recently uploaded (20)
Mesh Saliency Detection Using Local Curvature Entropy for Large-Scale Model Decimation
- 1. Saliency Detection for Large-Scale Mesh Decimation Rafael Kuffner dos Anjos Leeds, UK Richard Andrew Roberts VUW, NZ, Benjamin Allen VUW, NZ, Joaquim Jorge U Lisboa, PT Ken Anjyo OLM
- 4. 1- Faces: because we are humans 2- Regions with lots of detail, because we are drawn to it
- 7. Image saliency Simple framework for computing salience (Itti et al., 1998).
- 9. 1- Faces: because we are humans 2- Regions with lots of detail, because we are drawn to it How can one calculate this? Mesh Saliency: what do we look at?
- 10. Look for details! Look at mesh curvature averages (Gaussian weighted) at different scales, and combine Mesh Saliency: Lee et. al 2005
- 11. Some important parts don’t have that much immediate geometric change! (faces)
- 12. Look globally, in a Laplacian spectrum Stochastic laplacians of the mesh at different scales. “irregularity of the log- Laplacian spectrum of a 3D mesh is highly related to the saliency of the mesh” Mesh Saliency via Spectral Processing: Song et. Al 2014
- 13. Looking globally several times, may be slower than painting it manually.
- 14. Current Trend: Just let AI do it! Mesh Saliency via Weakly Supervised Classification-for-Saliency CNN: Song et. Al 2019 Let humans pick the points, and then teach a computer to pick the points on its own!
- 15. Well, I have a fantasy game, and the type of models I use have never been seen by people….
- 16. Entropy of a random variable is the average level of "information", "surprise", or "uncertainty" inherent in the variable's possible outcomes. Entropy on a 3D mesh 𝐻 𝑋 = − 𝑖=1 𝑛 𝑃 𝑥𝑖 log 𝑃(𝑥𝑖) So if the information of a vertex or its vicinity is “surprising”, we should keep it!
- 17. Local Curvature Entropy Mesh Saliency Analysis via Local Curvature Entropy: Limper et al. 2016 Instead of looking at the measurement and its size, looking at how surprising it is, at different scales.
- 18. +++ Concept of “surprise” will deal well with patterns +++ Not Immediately local, can be used globally +++ A lot faster - - - Curvature might be unstable with weird topology - - - Models from laser scan: noise is surprising - - - Still not fast enough Let’s try to solve this
- 19. Bandersnatch, FFXV © Square Enix Intermission: Why are we doing this?
- 20. 9.2 million faces (40+ with all the spikes) Bandersnatch, FFXV © Square Enix Original model has 40M+ vertices
- 21. The problem: Artist workflow. Mesh decimation algorithms have one input: “how much”. Artists want to have control of how it is going to turn out; where is the decimation going to be heavier. Current workflow: manually paint weights on ZBrush, or recreate everything from scratch.
- 22. Our Approach: provide control, fast. New robust formulation for scanned, modeled, sculpted meshes. Parametrizable so artists have control. Fast enough to allow multiple iterations in a sequence. Decimate using Saliency/Quadrics to control where the detail goes.
- 23. Formulation Entropy is a good measure of saliency. However, curvature is not numerically stable in common “production” cases. So we propose an alternative measure.
- 24. Tropical Angle of Curvature 𝜃 𝑣 Simple measure of maximum angle in a ring neighbourhood. Fast calculate with dot product; fixed range (0-1); good to calculate entropy. Also nice to detect creases, which are challenging for entropy alone.
- 25. Entropy of 𝜃 𝑣 , 𝐻(𝑣, 𝑟) Instead of just the entropy, of 𝜃 𝑣 , 𝛽 works as a “contrast” parameter. Adjusts the distribution among discrete bins. 𝛽 can be optimized to provide best distribution possible (average “white” saliency)
- 26. Beta as contrast parameter
- 27. Noise reduction In scanned models, “noise” is very surprising. So entropy will consider it salient. minimize noise: estimate if points fit to plane [considering a noise level 𝜖, we reduce its influence with a smoothstep function.] Quick way of fitting to a plane
- 29. Finally, Creases We use tropical angle as a crease detection tool, also normalizing the values of 𝑅 𝑣 .
- 30. Parameters Can be estimated Can be grouped into 1 hyper parameter “globality”
- 32. Default Parameters (without tweaking)
- 33. Going fast This formulation makes sure results are accurate, reliable, and controllable. To make it fast we used “adaptive subsampling”.
- 34. First: good engineering. Flat data structure with neighbors close in memory. Parallel implementation as each sample is independent Grouped Traversal, Priority queue, SIMD This helped but did not do the trick. Maybe if we fully moved to a GPU? But that was not the path we chose. (next!)
- 35. How/why does it work? Our estimate is multi-scale. At large scales, not concerned about “small scale details” that we might miss. At fine scales, need to sample enough to catch those. So, we use the same number of samples “per neighbourhood”
- 36. This loses accuracy… Doesn’t it? Not much. Trick: choose a few samples to run the computation on but look at all neighbors. We guarantee that every computed saliency value is interpolated from ground truth values, not subsampled. and “Tropical Angle” is not heavily affected by tessellation. Let’s look at results for the proof.
- 37. Results GT 75 spn 50k decimation (matching execution time)
- 38. Tradeoff Never underperforms comparing to pre- decimation, even if you assume decimation is instant.
- 39. Scalability (log: time in seconds)
- 41. Reliability Closest work to ours (Limper 2016) failed above 150k vertices. Our method achieved more reliability key to industry adoption.
- 42. Machine Learning Must work with predecimated meshes. We saw how badly that affects accuracy. Results from Song2019 are hit and miss for more “fantastic” inputs”.
- 43. Human picked interest points Are these good maps for the purpose of decimation? These have other applications.
- 44. Decimation Speaking of which, is this good? Yes! We still use quadrics, but weight it with the saliency information. Pure saliency is too greedy. Allows us to control where the error goes.
- 45. ZBrush Workflow Can be used for remeshing. “Automatic polypaint”.
- 46. Raytraced – pure geometry. 1% tris
- 47. Impact of saliency: details!
- 49. Industry feedback: Artists from SQEX and OLM got this tool to test with their private models. Everybody was able to find a suitable “saliency map” for their monsters/humans/pokémons. Most importantly, they were able to try MULTIPLE TIMES. All feedback was on workflow/tool improvements. An overall success!
- 50. Limitations Animated models. Better use of saliency map for decimation. Interactive is perfect. For that we need to go faster. Lighting/texture affects perceived saliency. Can that be considered during the estimate?
- 52. Thank you!
Editor's Notes
- Ask what the salient parts are in this sequence of slides. First two are pictures of human-like figures. SO it becomes easy
- Similar here, has a “face”
- What about this? Pointing up, detail as well
- Image saliency studied this in the past. Looking at oriented responses in the image to find what “sticks out”
- But sometimes it just doesn’t work.
- When looking at meshes, we want to be able to generalize it!
- Seminal work from Lee et. al introduces the concept. The authors define mesh saliency using a center-surround operator on Gaussian-weighted mean curvatures. This operator is applied in a scale-dependent manner, meaning that it captures geometric features at different scales. The authors argue that this definition of mesh saliency can capture what most people would classify as visually interesting regions on a mesh.
- Moving to a more global approach: When we talk about "stochastic Laplacians of the mesh at different scales", we are referring to a process where the Laplacian is computed in a stochastic (random) manner at different levels of detail or scales of the mesh. This could involve, for example, randomly sampling vertices or regions of the mesh at each scale and computing the Laplacian based on these samples. This allows for a multi-scale analysis of the mesh, capturing geometric features at different levels of detail. The "log-Laplacian spectrum" of a 3D mesh refers to the spectrum (set of eigenvalues) of the logarithm of the Laplacian matrix. The spectrum of a matrix is a fundamental concept in linear algebra and provides important information about the properties of the matrix. In the context of a 3D mesh, the spectrum of the Laplacian can reveal important information about the geometric and topological properties of the mesh. The statement "irregularity of the log-Laplacian spectrum of a 3D mesh is highly related to the saliency of the mesh" suggests that the degree of irregularity or variation in the log-Laplacian spectrum is a good indicator of the saliency (visually interesting or important features) of the mesh. In other words, regions of the mesh that cause significant irregularities in the log-Laplacian spectrum are likely to be regions that are visually salient or important. This could be because these regions represent significant changes or features in the geometry of the mesh, which are likely to attract visual attention.
- Once a probability distribution has been defined, the entropy of the mesh can be computed using the formula for entropy from information theory: H = - ∑ p(x) log(p(x)) where the sum is over all vertices or faces x in the mesh, and p(x) is the probability of x according to the defined distribution. This formula essentially adds up the amount of "surprise" or "information" associated with each vertex or face, with more probable vertices or faces contributing less to the sum, and less probable vertices or faces contributing more. The resulting entropy value is a measure of the complexity or irregularity of the mesh. A mesh with high entropy has a lot of variation or complexity in its geometry, while a mesh with low entropy is more uniform or regular.
- Local Curvature: For each point on the 3D mesh, compute the curvature. Curvature is a measure of how much a curve (in this case, the surface of the 3D mesh) deviates from being a straight line (or a plane, in 3D). High curvature corresponds to a high degree of bending or twisting at a point. Probability Distribution: Create a probability distribution based on these curvature values. This could be done, for instance, by treating the curvature values as a discrete set of events and computing the frequency of each event. Entropy Calculation: Compute the entropy of this distribution. In information theory, entropy is a measure of the uncertainty or randomness in a set of data. In this case, it measures the variability in the curvature values. The entropy is computed using the formula: H = - ∑ p(x) log(p(x)), where the sum is over all curvature values x, and p(x) is the probability of x according to the defined distribution. The resulting Local Curvature Entropy is a measure of the complexity or irregularity of the 3D mesh, based on the distribution of curvature values. A high Local Curvature Entropy indicates a mesh with a wide variety of different curvatures (i.e., a complex or irregular shape), while a low Local Curvature Entropy indicates a mesh with mostly similar curvatures (i.e., a simple or regular shape).
- Tell why “tropical”.
- We derived Beta from a collection of models, so given a new model we can find the right beta. Geodesic Neighborhood: For each vertex v on the 3D mesh, consider a geodesic neighborhood U(v,r) around the vertex. This neighborhood consists of all points x whose geodesic distance from v is less than r. Probability Density Function: The tropical angle of curvature, θβ, is treated as a stochastic variable within the neighborhood U(v,r). The probability density function of θβ is approximated using an area-weighted histogram. This histogram is created by dividing the range of possible curvatures (from θmin to θmax) into n discrete intervals or "bins". The value Av for each bin is accumulated by considering the influence area of each vertex in the neighborhood. Entropy Calculation: The local entropy H(v,r) is then calculated by summing the area-weighted contribution of each bin. A larger variety of θ(v) values (i.e., a wider range of curvatures) within the neighborhood would result in a higher entropy. The parameter β is used to control the distribution of differences across the bins. A β value less than 1 will spread out the differences, while a β value greater than 1 will concentrate them. The text suggests that a suitable value for β can be determined automatically by setting the global entropy equal to a desired maximum histogram value. Finally, the algorithm integrates a measure of "crease detection" to highlight local edges that may be difficult to parameterize with the entropy formulation. This can be useful in cases where it's important to preserve specific local features in the mesh, such as handcrafted patterns. The crease detection measure is weighted and added to the local entropy to produce the final saliency measure S(v).
- Color mapping for all images in this paper is [0.0 − 1.0], cyan to red.
- The method involves defining a level of "noise" ε, with the goal of removing artifacts from surface fluctuations smaller than this value. This is particularly effective on low entropy surfaces, such as flat planes. The approach approximates small neighbourhoods to a plane and identifies points within that area that have a variation within the noise level by measuring their distance to the estimated plane.
- Example saliency maps of a model created from a scan (Thai Statue from Stanford Repository) where local noise is filtered out by using ε = 0.01% of the mesh area, as seen in the top of the model (A). (B) features a large value of ε in order not to apply this operation.
- Changing globality parameter: top without noise removal, bottom with it.
- Default parameters on a varied model set. Still works without tweaking.
- Cutoff 2 days
- Look at some of the “human interest points”, they have semantic meaning, not necessarily geometrical. They are suited for other applications. We could look into combining this in the future. Comparison between Schelling points [26] probability distribution and our method. Schelling points tend to be discretized around individual areas of interest that may have contextual meaning, which is not always ideal for decimation (example, point at the center of the table). On the other hand, our approach works better at detecting creases and edges at different scales (e.g. see cup, table and airplane models). We argue that for mesh decimation ours is the best performing approach.
- Top row: where the error is. Bottom: saliency maps.
- Renders of Lucy. Left: After 99% decimation with our saliency map using a triangle mesh. Right: original with 14 million vertices. Bottom: Used saliency map and wire-frame output. Our method preserves salient features even under high compression levels, e.g., upper lip and feet details.
- Left: saliency keeps Eyes, lips, hair, but subtle. Renders of Lucy. Left: After 99% decimation with our saliency map using a triangle mesh. Right: original with 14 million vertices. Bottom: Used saliency map and wire-frame output. Our method preserves salient features even under high compression levels, e.g., upper lip and feet details
- Left saliency: Keeps face and details. Hindu (right) models simplified at 99%. Both models have the same vertex count, with the saliency allowing more details to be preserved.