Bernardini etal-cga02


Published on

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Bernardini etal-cga02

  1. 1. Feature ArticleBuilding a DigitalModel ofMichelangelo’sFlorentine Pietà Fausto Bernardini, Holly Rushmeier, Ioana M. Martin, Joshua Mittleman, and Gabriel Taubin IBM T. J. Watson Research Center B ecause scanning devices are less expen- sive and easier to use than they were just a few years ago, a wide range of applications employ the opportunity to explore the value of 3D scanning and visualization in a nontechnical discipline, art history. The 3D scanning technology. As a result, various organiza- project’s second goal was to develop scanning technolo- tions can now produce models of cultural artifacts and gy accessible to other cultural heritage projects both in works of art. terms of cost and usability. Such technology could poten- Members of the National Research Council of Cana- tially be used in widespread commercial applications, da, developers of high-accuracy scanning equipment, such as e-commerce, where equipment cost must be min- have applied their technology to imal. scanning paintings, sculptures, andWe present the archaeological sites. Recent work Overview emphasizes the importance of Jack Wasserman, professor emeritus of art history atmethodology we used to portable, reliable equipment that Temple University in Philadelphia, had been studying researchers can easily deploy at the Michelangelo’s Florentine Pietà for several years. Heacquire the data and scanning site.1 Zheng and Zhong intended primarily to document all aspects of this impor- scanned archaeological relics in tant work and its history for future researchers, andconstruct a computer model cooperation with the Museum of secondarily to investigate his own theories on Michelan- Terra Cotta Warriors and Horses, gelo’s composition. He had used high-quality tradition-of Michelangelo’s Florentine China.2 Their goals included creat- al photography, x-ray, and ultraviolet light studies, as ing a database of information about well as researching the complex history and symbolismPietà with enough detail and the excavation site and testing and of the statue and its analysis by past art historians. employing virtual restoration tech- Although it’s not clear that a 3D model would be use-accuracy to make it useful in niques. Recently, Marc Levoy and a ful in studying every sculpture, Wasserman felt that this team from Stanford University new technology was especially well suited to studyingscientific studies. scanned many of Michelangelo’s the Pietà.4 sculptures,3 including the 5-m tall Accounts from Michelangelo’s contemporaries tell us David in the Galleria dell’Accademia. They used sever- that the artist intended the Florentine Pietà to serve as his al types of scanners, including a high-resolution laser own tomb monument. Beginning late in his life, in the triangulation system mounted on a custom-made 1550s, he carved four larger-than-life figures from a sin- mechanical gantry and a time-of-flight long-range sen- gle block of marble. The Christ figure in the center rests sor. The large quantity of data collected should have a across the lap of the Virgin Mary, supported on the left major impact in future development of shape recon- by Mary Magdalene. Behind and above, supporting the struction algorithms. Numerous other projects have statue of Christ, is a figure believed to represent Nicode- been conducted or are currently underway. mus and to bear Michelangelo’s face. At some point, In this article, we describe a recent project to acquire Michelangelo decided to break off parts of the statue. and build a 3D model of the Michelangelo’s Florentine He then abandoned it. Shortly before his death he per- Pietà. Figure 1 (next page) shows a photograph of the mitted one of his students, Tiberio Calcagni, to repair Pietà and an image of our model. The work we describe the statue. Calcagni reattached several pieces to the stat- here is unique in that an art historian, not a technologist, ue and partially finished the figure of the Mary Magda- conceived and specified the project. Our goal wasn’t sim- lene. Thus, what we see today is, in a sense, a composite ply to produce the statue’s model but also to provide the of Michelangelo’s work and his student’s: the original art historian with material and tools to enable him to design damaged, repaired, and overlaid by later work. answer his own research questions. The project gave us The unique aspects of the history of this statue make0272-1716/02/$17.00 © 2002 IEEE IEEE Computer Graphics and Applications 59
  2. 2. Feature Article mission to work or leave any equipment visible around the statue when the museum was open to the public. Therefore, we needed a system that we could easily set1 A photo- up and remove at the beginning and end of each eveninggraph of scanning session. The irreplaceable nature of the pieceMichelangelo’s restricted contact to a minimum and required us toFlorentine Pietà ensure that we operated the scanner system safely. The(left). A synthet- complex shape of the group of figures in the Pietàic picture from required the ability to freely position the sensor to accessour 3D comput- the marble surface’s recessed model We had a limited budget for buying noncomputer(right). equipment and a limited amount of time for design and customization. These constraints led us to consider a small, portable structured-light system rather than a more expensive laser triangulation scanner. By this choice, we sacrificed geometric resolution, which we would have had to recover with a supplementary system. Our main technical requirements were dictated by the it a promising candidate for using 3D scanning tech- nature, resolution, and accuracy of the data needed to nology. It’s important for art historians to view the stat- address Wasserman’s needs. The goal was to obtain data ue in the environment Michelangelo intended, examine to allow realistic rendering of the synthetic model. The it without the pieces Michelangelo removed, and ana- statue is 2.25 meters tall, and we wanted to capture its lyze the detailed toolmarks in the unfinished portion of shape and surface details, such as cracks and toolmarks, the work. Furthermore, the statue’s complex geometry on the scale of 1 to 2 mm. Besides geometry, we were limits what can be done with traditional techniques. A interested in capturing the surface’s reflectance proper- camera can’t capture certain views of the statue because ties. We therefore needed to achieve submillimeter accu- the statue itself or the walls of the room where it stands racy in measurements. interfere with proper camera placement. Capturing a large object at such fine resolution entails a number of difficulties, especially under the less-than- Scanning system and methodology ideal conditions outside a lab. Issues of repeatability and Three-dimensional scanning technology is evolving precision make scanners based on moving parts expen- rapidly. Current sensors use a number of techniques sive to build and difficult to transport and operate. Sub- such as laser triangulation, laser time-of-flight, passive surface scattering of laser light in marble limits the stereo vision, and structured light projection to sample accuracy that laser triangulation systems can achieve. surfaces. Typical considerations in choosing the most We decided to use a system that could capture a small appropriate scanning technology include target accu- portion of the surface from a single position, acquire a racy, surface reflectance characteristics, and cost. large number of overlapping scans, and rely on software registration to integrate the results. Design considerations The amount of data we needed to represent the sur- Scanning a large statue in a museum poses a number face at such a fine level of detail presents additional of constraints in designing the scanning system and problems. For example, we can’t store, process, and visu- process. In our case, the small size of the room in which alize a triangle mesh of hundreds of millions or billions the statue is displayed limited scanner size and the dis- of triangles on current PCs or midrange workstations. tance from the scanner to the statue. We didn’t have per- Because we aimed to make the results accessible to a wide audience, we decided to rep- resent shape as a triangle mesh with a resolution of a few millimeters and to store additional fine geometric details as a normals map. Plus, we could store reflectance values as RGB image maps. Having thus cho- sen the final representation of our model, we avoided a great deal of intermediate computation by designing a system that captures data directly in that format. Scanning (a) (b) (c) Figure 2 shows a schematic of our 3D capture methodology. Our scan-2 Our 3D capture methodology. (a) Take multiple digital photos. (b) Compute each scan’s ner is based on a multibaselinesurface shape, color, and details. (c) Align and merge the scans into a single model. stereo system, supplemented by a60 January/February 2002
  3. 3. photometric system. The scanner,visible in Figures 3a and 3b, is a cus-tomized version of the VirtuosoShapeCamera. A photographic flashprojects a pattern of vertical stripeson the subject. At the same time, sixblack-and-white digital camerasphotograph the illuminated areafrom different angles. Figure 3cshows a sample stripe image. Anadditional digital color camera pro- (a) (b)vides a texture image. A multiview 5stereo algorithm, part of the soft-ware suite provided with the Virtu-oso system, computes a trianglemesh approximating the scannedarea. In our scanning conditions,each scan typically covered a 20 cmby 20 cm area and comprised onaverage about 10,000 measuredpoints. The typical intersample dis-tance for these scans is about 2 mm.In tests we conducted on referenceobjects, we were able to accurately (c) (d)measure a depth of 0.1 mm for a sin-gle scan. 3 (a) The scanner used in our project. We added the five-light photometric The photometric system (Figures system to a Virtuoso ShapeCamera. (b) The scanner in use in the museum.3a and 3b) consists of five light The stands visible in the picture held the laser projectors. (c) Detail of onesources and the built-in color cam- of the six stripe images simultaneously taken by the scanner. (d) Detail ofera, plus some control electronics. the laser dots projected on the statue, as captured by the color cameraFor each camera pose, we took five mounted on the scanner.additional color pictures, each withone of the five light sources, whileall other lights were turned off. We also used low-power and facilitate data transfer operations. However, to keeplaser sources to project red dots onto the statue (shown setup and teardown time to a minimum, we simplifiedmounted on light stands in Figure 3b). The laser pro- our system considerably.jectors that we used each generated an 11 × 11 grid of Our streamlined procedure consisted of the followingrays. From a distance of about 1 meter, they produced an steps. We positioned the large photographic tripod andirregular pattern of red dots on the statue, with an aver- secured the scanner to it. Then, we placed the five laserage spacing of 2 to 4 cm. For each pose, we took a picture projectors on three light stands to cover the area to beof the dots (with all other light sources turned off) to scanned in one session with a grid of laser dots. We cap-help align overlapping meshes (see Figure 3d). The color tured the data by placing the scanner at one area to bepictures have a resolution of 1280 × 960 pixels, with 24 covered and shot one set of pictures, then we moved thebits of RGB color per pixel. Typically we had a 0.5-mm scanner across the target area to take successive overlap-intersample distance in the color images. As a result, we ping picture sets, covering the region with a regular pat-computed reflectance and surface normals from these tern of scanned tiles. We kept track of the approximatepictures at a resolution about four times greater than area covered by each picture set on paper diagrams of thethe underlying geometry. statue. We estimated that we had enough overlap by com- Our initial design included a magnetic tracker to paring previews on the scanner display. We moved therecord an approximate estimate of the camera position scanner conservatively, about 10 cm between shots, toand orientation with respect to a global frame of refer- ensure that we had enough data. We processed the stripeence. We hoped to use this pose estimate to provide a pictures during the day, before the next evening’s scan-starting point for our software registration process. We ning session, to make sure that we had acquired enoughused a Polhemus system, fitted with the long-range data and not left any holes. One person operated the scan-source to provide an electromagnetic field large enough ner, while another acted as supervisor to ensure that theto cover our work volume. We attached a sensor at the proper standoff distance was respected and safety rulestip of a 40-cm plastic rod, rigidly secured to the camera followed. Wasserman was present during the entirebody. Unfortunately, we quickly discovered that metal- process to provide input on his priorities.lic material in the room, including our own equipment, We did a preliminary scan of the statue (without thedistorted the field to the point of making measurements photometric system) in February 1998, spending fiveuseless. We also had initially planned to use additional six-hour evenings and four full days in the museum. Wehardware and software to remotely control the scanner repeated the scan using the photometric system during IEEE Computer Graphics and Applications 61
  4. 4. Feature Article Point Corrected images and six color images. We Manual pairwise cloud BPA albedo Texture used the Virtuoso Developer soft- synthesis ware to compute triangle meshes alignment for each single scan from the six Albedo, Mesh normals map stripe images. From this point on, (per patch) we applied several algorithms to Laser dots the data to build the final model. Simplify We registered the individual scans alignment Viewer together based on matching geo- Registration Mesh metric and image features. We matrices remeshed the resulting point cloud to obtain a seamless geometric Mesh ICP model. We extracted the color and segmentation detail information from five of the Registration Patches color images and reorganized them matrices in the form of normals and reflectance maps. Figure 4 illus- Image-based trates the sequence of steps. Photometric registration Registration Albedo, Registration normals map We start with a pairwise, manu- matrices (per scan) al, approximate alignment of the meshes, obtained interactively by Conformance Color selecting three or more matching smoothing alignment features on overlapping pairs of scans. We used the diagrams4 The reconstruction pipeline. recorded during scanning to identi- fy sequences of overlapping scans and constructed a tree of pairwise two one-week visits, one in June 1998 and one in July alignments that spans the whole set of scans. We need- 1999. The total time spent doing the final scanning was ed to do this initial manual alignment because the track- about 90 hours over 14 days, including the equipment ing hardware we intended to use didn’t perform setup and teardown each day. It took about 800 scans satisfactorily in the museum. We progressively refined to cover the whole statue. The raw data consisted of the alignment using several registration algorithms that 4,800 640 × 480 pixel, 8-bit gray-scale stripe pictures used geometric and image information at increasing res- and 4,800 coregistered 1280 × 960 pixel, 24-bit RGB olution levels. color images. Stored in lossless compressed format, the For each scan, we found the red dots in the image raw data occupies 3 Gbytes of storage. taken with the laser projectors on and mapped these In retrospect, we believe that we chose the right scan- image points back onto the scan geometry. Given the ning technology, although with additional planning and initial manual alignment, we searched in the neighbor- design we could have built a more efficient system. The hood of each laser point for matching points in over- main bottlenecks in the process were the scanner’s rel- lapping scans, adding additional consistency constraints atively long cycle time, the small area covered by each to prune false matches. We improved the registration scan, and the offline processing of data. The time by minimizing the sum of square distances between required to complete the acquisition and local storage matching points using Besl and McKay’s method.6 We of one set of images was about 2 minutes, and we then ran several iterations of an n-scan iterated closest required about the same amount of time to process one point (ICP) algorithm7 to further reduce the registra- set of striped images to obtain a triangle mesh. Track- tion error. Additional details of our geometry-based ing the camera pose would have saved us from doing alignment appear elsewhere.8 the pairwise manual alignment of the scans that pro- To refine the geometry-based alignment obtained vided a starting point for our registration algorithms. with the ICP algorithm, we applied an image-based reg- The main advantage of our scanning system, besides istration method that considers additional information meeting the requirements of our original design, is that contained in the high-resolution reflectance maps com- it potentially provides a starting point for future devel- puted for each scan (see the following section). We used opment of a low-cost system built out of commodity a combination of smoothing, edge-detection, and components. If it’s augmented with reliable tracking, thresholding operations for selecting candidate points fast capture, and high-resolution cameras, it could lead in feature-rich areas of each image. We conducted a cor- to a system for real-time scanning of large objects. relation-based search in images associated with over- lapping scans to find corresponding points. We Reconstruction pipeline subsequently back projected the resulting pairs onto the The acquired raw data consisted of roughly 800 scans and derived a rigid transformation that minimizes scans, each consisting of six black-and-white stripe distances in a least-squares sense. See Bernardini, Mar-62 January/February 2002
  5. 5. tin, and Rushmeier9 for additional details of the image- 5 Subsequentbased registration phase. steps of the We also attempted to reduce scanner line-of-sight registration oferrors by computing more accurate estimates of true three test scans.surface points from multiple overlapping scans while Each scan showsfiltering out small high-frequency components (which a differentthe photometric system can better capture). We call this color. In theprocess conformance smoothing. Figure 5 shows an conform step,example of the successive alignment steps on three test the scans havescans of Nicodemus’ face. The finer grain of the color been slightlyvariations after the conform step indicates that the deformedshape of the overlapping scans are nearly the same. (shown with aExperiments showed we could improve the registration changed color)error if we accounted for the line-of-sight error during to compensatethe alignment. We’re experimenting with alternating for residualiterations of registration and conformance smoothing registration andto obtain a better final alignment. measurement We didn’t have equipment to accurately measure large error.distances between points on the statue (for example,from the top of Nicodemus’ head to a point on the base).Therefore, we were unable to quantitatively state theglobal accuracy of the alignment. We discuss the valida- aries. We can repeat the process as many times as need-tion of our results using 2D projections and photographs ed. Eventually, the simplified mesh is small enough toin the section “Validating and using the model.” be further processed in a single pass by the in-core algo- rithm. We expect memory-efficient simplification algo-Meshing rithms to become a hot topic of research as capture The result of the alignment and conformance pro- methods improve and large models become widespread.cessing is a large set of estimated surface samples. Thispoint cloud has nonuniform density because the num- Details and colorber of overlapping scans varies from one part of the sur- The mesh produced using the Virtuoso camera has aface to another and because the density within each spatial resolution of approximately 2 mm, which is ade-scan varies locally depending on the angle at which the quate for studying the statue’s proportions from variousscanner saw the surface. However, except for areas that viewpoints. However, it’s inadequate for studying thethe scanner couldn’t reach, the sampling density is usu- small-scale tool marks. To capture data at a higher spa-ally larger than necessary to recover the surface’s shape tial resolution, we exploited the fact that the Virtuosoto a reasonable approximation. We designed our sys- includes a color camera that produces images with a res-tem to acquire geometry with an average intersample olution of about 0.5 mm per pixel. We computed detaildistance of 2 mm. Note that this spatial resolution is at this pixel resolution using a photometric stereo sys-independent of the accuracy in measuring point posi- tem built around the Virtuoso.tion. The scanner we used has a precision of 0.1 mm in Figure 3 shows our photometric system. Given threecomputing depth for each sample point. images of a surface lit by three different light sources in The ball-pivoting algorithm (BPA)10 computes a tri- known positions, we can solve a set of simultaneous equa-angle mesh interpolating the point cloud, using a region- tions for the surface normals corresponding to the pointsgrowing approach. Our implementation of the BPA visible at each pixel in the image. Given the normal athandles large data sets in a memory-efficient way, by each pixel, we can compute the relative reflectance, orprocessing input data in slices. albedo, at each pixel for the red, green, and blue bands. The Pietà data consist of 800 scans containing a total We used five light sources rather than three because inof 7.2 million points. We processed the data in slices of any given image a point may be in a shadow or a specu-10 cm, using ball radii of 1.5, 3, and 6 mm. The BPA ran lar highlight. Figure 6a shows four typical imagesin 30 minutes on a Pentium II PC using 180 Mbytes of obtained from a single camera position. Figure 6b showsmemory and output a 14-million triangle mesh. the resulting normals (lit from the side). For further detail We applied a mesh simplification algorithm to gen- of the physical system design, see Rushmeier et al.12erate a hierarchy of models at different resolutions. We To compensate for possible errors in the photometricfound that conventional, in-core, simplification algo- normals calculations, we used data from the 2-mm res-rithms couldn’t handle the large mesh generated from olution mesh to compute the direction and relative dis-our data. We computed simplified models by breaking tance to each point visible in each image and to estimateup the mesh into smaller, manageable pieces. We then the relative light source intensity in the neighborhood ofapplied a traditional, high-quality simplification algo- each pixel from each of the five lights. To compensaterithm,11 leaving the boundary of each piece untouched for scan-to-scan color variations, we performed a colorand stitched the resulting simplified pieces together. In registration analogous to the geometric registration ofa successive pass, we broke up the mesh along different scans. We found corresponding points in all overlappingedges, so that we could simplify the previous bound- color albedo maps and then found a least-squares solu- IEEE Computer Graphics and Applications 63
  6. 6. Feature Article6 (a) Color erate a synthetic image from theimages taken same viewpoint. We couldn’t esti-with four of the mate the lighting conditions in thefive light commissioned photographs. Tosources. (b) address the effect of lighting, weSynthetic pic- also matched camera viewpointsture computed for images we took with a digitalusing the sur- camera for which we knew the flashface normals location. Initially, we computedobtained with images with geometry alone andthe photometric found that including the surfacesystem. albedo was essential to perceiving (a) (b) the proportions in the synthetic image. tion for scaling factors for each of the color channels in Overview of results each of the images to obtain the best color match on cor- Wasserman’s research questions defined our primary responding points. Rushmeier and Bernardini13 discuss goals for the Pietà project. We shaped our presentation additional details of adjustments made using the under- of the results to fit his needs. We developed a plan to ful- lying mesh and color registration. fill his requirements by delivering a variety of results, including Texture synthesis We partitioned the triangle mesh into height-field I precisely defined views, patches with a simple region-growing heuristic. For each I impossible views, patch, an orthogonal projection in the direction that I embedding the statue in virtual environments, maximizes the projected patch area defines a mapping I providing precise measurements, between geometry and corresponding textures. I modifying the statue, and The texture synthesis process computes surface nor- I providing an interactive viewer. mals and reflectance maps as weighted combinations of corresponding values in all the overlapping images. To answer certain questions about Michelangelo’s Weights are assigned to take into account the degree of composition, Wasserman wanted to see the statue from confidence in each pixel value, based on distance to the physically or practically impossible points of view. These camera and viewing angle. Because weight maps cor- included views from directly above the statue to reveal respond to scans and not to patches, transitions across details of the composition not normally visible (Figure patch boundaries aren’t visible. Also, since the weights 7c, next page) and from various angles at a height below for each scan decrease with distance to the scan border, the base of the statue to illustrate it as it would have scan-to-scan boundaries aren’t visible. appeared in the context Michelangelo originally intend- In our implementation, we streamlined computations ed. We also recreated some of the settings in which the by presampling the patch geometry and loading values Pietà stood during its history, using 3D models and ani- from all maps simultaneously. Occlusions are handled mations to illustrate the statue’s visual impact in these elegantly by comparing depth values in precomputed various environments (Figures 7e and 7f). To reconstruct depth buffers. Image and geometric information is the tomb and garden settings shown in these figures, loaded on demand to allow for processing of large data Wasserman provided drawings and images of similar sets that don’t fit in memory. We provide additional environments and some crude dimensions. Accurately details regarding our image-based registration and tex- modeling the environments required a number of vari- ture synthesis algorithms elsewhere.9 ations of each environment that Wasserman evaluated against his understanding of the historical record. Validating and using the model A large digital model isn’t useful to art historians. As Editing the model a result, we needed to derive an assortment of presen- The ability to modify our model of the statue provid- tations of the data suited to Wasserman’s needs, which ed Wasserman with opportunities to study it in ways in some cases required new techniques. Before devel- otherwise impossible. Using the 3D model, we recon- oping other results from our model we needed to vali- structed the statue with Christ’s missing left leg, approx- date its accuracy to Wasserman’s satisfaction. His test imating its appearance before Michelangelo broke it. was that images derived from our model must correlate We removed the pieces that Calcagni reattached, illus- well with the high-quality photographs he had com- trating the statue as it may have appeared without his missioned from a professional photographer. efforts. Figure 7g shows this second modification. In the To perform the validation, we selected features in figure, some of the surfaces that would be occluded by digitized versions of Wasserman’s photographs and the limbs removed are now visible. Internal areas that found the corresponding 3D coordinates of those revealed where the marble was broken are colored a flat points on our model. We then used Tsai’s calibration gray. We also separated the four figures that make up methodology14 to compute camera parameters to gen- the statue so that they can be examined in isolation.64 January/February 2002
  7. 7. (a) (b) (c) (d) Identifying the pieces that Michelangelo removedis itself a problem. Four major pieces were removedfrom the statue (three arms and a portion of a leg thatwas never replaced). The three pieces that were reat-tached were each composed of a set of smaller frag-ments, so it’s not obvious what was removed. Based (e) (f)on his own close study of the physical work and thex-rays he had commissioned, Wasserman sketchedon electronic images of the statue the various lineswhere he believed the breaks were made. (g) Directly editing a large model with millions of ver-tices isn’t feasible, particularly because our triangle 7 (a) Black-and-white rendering of the model. (b) Close-up view of themesh doesn’t have sufficient resolution to model the model. (c) Bird’s eye view of the model. (d) Interactive viewer. (e) Syn-breakage exactly as Wasserman wanted. We tried two thetic image in a niche above the tomb. (f) Synthetic image in a garden.methods of editing the model. First, we tried painting (g) The statue without the pieces Michelangelo removed.each of the color images associated with the individualscans to precisely mark which parts belonged to theremoved sections. This approach had some problems, (Figure 7d). The combination of a large data set and asince hand marking didn’t give pixelwise identical loca- slow computer required special attention to the trade-tions for all of the breaks across the various scans. How- offs between speed and quality and between usabilityever, given the painted images, we could automatically and flexibility. Our target audience consisted of unso-segment the statue by simply removing vertices that phisticated computer users unaccustomed to the navi-were painted with the broken color. This simple com- gation paradigms common to interactive 3D graphics.putation was useful while we were producing early ver- We found that we needed to radically simplify the con-sions of the model (before all the data was acquired, trols to provide a fast learning curve and then adaptadded, and tightly aligned) to give Wasserman an indi- them to our users’ abilities and interests. Maintainingcation of what results to expect. interactivity was essential, so that users could easily For the final model, we crudely segmented the model grasp the function of the navigation controls and com-by defining bounding boxes enclosing the broken seg- pensate for their inevitable limitations.ments. We then identified individual patches contain- We had two objectives in designing an intuitive inter-ing portions of the cracks for editing. While this face: maintaining a frame of reference when zoomingapproach would be tedious to repeat many times (the in on detail and providing clear, separate controls forcracks extend over many different patches), it was the altering views and lighting. Our viewer presents usersmost reliable approach for the final model. with a simplified model of the statue around which they We also used the painting to separate the four figures can navigate interactively and lets them render a full-in the model. This task is more sensitive to the problem resolution image of a selected detail. The simplifiedof ambiguous identification across scans because no pre- model, only 1 percent the complexity of the full model,cise lines on the statue define the figures. Wasserman acts as a map to the more detailed model. Users canwanted the figures separated because they revealed select an area of interest, using simple 3D navigationshapes and relationships, like the relative position of controls to reach the desired view. We chose a naviga-Mary Magdalene’s hands, that we can’t observe from the tion paradigm in which the camera orbits around a user-solid statue. While we were able to achieve the segmen- selected center at a fixed distance and zooms in and outtation of the statue we needed for this study, our experi- along a radius. Users can select a new center by pickingence indicates that detailed editing of high-resolution or dragging the image parallel to the view plane.models is an area that requires additional research. To examine a region in more detail, users frame a sec- tion of the scene and render the desired image from a data-Interactive viewer base of the full-detail model. This step currently can take To enable Wasserman to study the statue on his own a few minutes on a laptop computer. We enhanced thecomputer, we designed a viewer that could run on a PC resulting 2D image to let users interactively vary the light- IEEE Computer Graphics and Applications 65
  8. 8. Feature Article ing by dragging the mouse across the window. Many The final conclusions Wasserman draws from the dig- details invisible with the light in one position appear when ital model will be in his book published by Princeton the light moves. Users can thus understand fine-scale University Press, which will include more of our results structures more clearly. We designed this technique to on a CD-ROM. Kiosks are now (or have been) present- model a method that we observed Wasserman using on ing our work at a number of museums throughout the the statue. He moved a small flashlight slowly back and world (including sites in Asia, Australia, South America, forth across the statue to highlight small surface irregu- Europe, and the US). I larities. The virtual light editor produces similar effects. The viewer was useful for isolating portions of the Acknowledgments model and rendering high-resolution close-ups of sec- We express our appreciation to Claudio Silva and tions of interest. We had hoped that the viewer would André Gueziec for their contributions to this project, also be helpful to Wasserman in evaluating the appear- the Museo dell’Opera del Duomo in Florence for their ance of the statue from various views, to develop a the- assistance in our study of the statue and Jack Wasser- ory of exactly how Michelangelo intended the statute man for his inspiration and for creating the opportuni- to be viewed. The simplified model in the viewer proved ty for our work. inadequate for this interactive study. When we simpli- fied the model to the extent that allowed interactive viewing, it still looked like a good representation to the References casual observer. However, for Wasserman’s in-depth 1. J.-A. Beraldin et al., “Portable Digital 3D Imaging System study of specific proportions, the simplified model for Remote Sites,” Proc. 1998 IEEE Int’l Symp. Circuits and wasn’t accurate enough. Systems, IEEE Press, Piscataway, N.J., 1998, pp. 488-493. The current viewer still needs improvement. It’s far 2. J. Zheng and L. Zhong, “Virtual Recovery of Excavated from satisfactory as a tool for art historians. We need to Relics,” IEEE Computer Graphics and Applications, vol. 19, do a great deal of work to find intuitive methods for non- no. 3, May/June 1999, pp. 6-11. technical users to interact with 3D data, especially when 3. M. Levoy, “The Digital Michelangelo Project,” Proc. 2nd viewing large data sets that we must simplify to allow Int’l Conf. 3D Digital Imaging and Modeling, IEEE Press, Pis- interactive display. Part of the problem is rendering speed. cataway, N.J., 1999, pp. 2-13. Incorporating ideas such as point-based rendering15 or 4. J. Abouaf, “The Florentine Pietà: Can Visualization Solve multipass texturing now available on inexpensive graph- the 450-Year-Old Mystery?,” IEEE Computer Graphics and ics cards would improve this aspect of our system. Applications, vol. 19, no. 1, Jan./Feb. 1999, pp. 6-10. 5. C.L. Zitnick and J.A. Webb, Multi-Baseline Stereo Using Sur- Conclusions face Extraction, tech. rep. CMU-CS-96-196, School of Com- This project has demonstrated that 3D scanning and puter Science, Carnegie Mellon Univ., Pittsburgh, Penn., graphics can be useful tools for art historians, and by Nov. 1996. extension for similar studies in archaeology, architec- 6. P.J. Besl and N.D. McKay, “A Method for Registration of 3D ture, and other disciplines where detailed examination Shapes,” IEEE Trans. Pattern Analysis and Machine Intelli- and manipulation of artifacts is necessary but not feasi- gence, vol. 14, no. 2, Feb. 1992, pp. 239-256. ble in reality. Wasserman noted three aspects of the vir- 7. R. Bergevin et al., “Towards a General Multi-View Regis- tual model that were especially useful to art historians: tration Technique,” IEEE Trans. Pattern Analysis and Machine Intelligence, vol. 18, no. 5, May 1996, pp. 540-547. I Having a virtual model of the statue lets researchers 8. F. Bernardini and H. Rushmeier, “Strategies for Registering examine the statue at their leisure, to discover details Range Images from Unknown Camera Positions,” Three- they hadn’t noticed in the time they spent with the Dimensional Image Capture and Applications III (Proc. statue and to verify their recollections and theories. SPIE), vol. 3958, Jan. 2000, pp. 200-206. I The ability to control lighting precisely lets 9. F. Bernardini, I.M. Martin, and H. Rushmeier, “High- researchers see the statue as it might have appeared Quality Texture Reconstruction from Multiple Scans,” IEEE in environments outside the museum and to highlight Trans. Visualization and Computer Graphics, Oct.–Dec. small details not easily visible. 2001, vol. 7, no. 4, pp. 318-332. I Measuring the statue freely and precisely lets histo- 10. F. Bernardini et al., “The Ball-Pivoting Algorithm for Sur- rians factor out subjective perception and better face Reconstruction,” IEEE Trans. Visualization and Com- understand the artist’s use of perspective and com- puter Graphics, vol. 5, no. 4, Oct.–Dec. 1999, pp. 349-359. position. 11. A. Guéziec, “Locally Toleranced Surface Simplification,” IEEE Trans. Visualization and Computer Graphics, vol. 5, Our technical experience shows that it’s possible to no. 2, Apr.–June 1999, pp. 168-189. build a detailed computer model of a large object from 12. H. Rushmeier et al., “Acquiring Input for Rendering at a large collection of small individual scans, which we Appropriate Level of Detail: Digitizing a Pietà,” Proc. 9th can acquire with a relatively inexpensive device. Our Eurographics Workshop on Rendering, Eurographics Assoc., plans for future work include studying improved algo- Aire-la-Ville, Switzerland, 1998, pp. 81-92. rithms for accurately registering multiple scans and 13. H. Rushmeier and F. Bernardini, “Computing Consistent developing a handheld, real-time scanning system. For Normals and Colors from Photometric Data,” Proc. 2nd Int’l more information about our work, visit http://www. Conf. 3D Digital Imaging and Modeling, IEEE Press, Piscat- away, N.J., 1999, pp. 99-108.66 January/February 2002
  9. 9. 14. R.Y. Tsai, “An Efficient and Accurate Camera Calibration Technique for 3D Machine Vision,” Proc. IEEE Computer Joshua Mittleman is a program- Vision and Pattern Recognition (CVPR 86), IEEE Press, Pis- mer at the IBM T.J. Watson Research cataway, N.J., 1986, pp. 364-374. Center, specializing in simplification,15. S. Rusinkiewicz and M. Levoy, “Qsplat: A Multiresolution level-of-detail control, and other Point Rendering System for Large Meshes,” Computer algorithms applied to rendering com- Graphics (Proc. Siggraph 2000), ACM Press, New York, plex models. He has an MS in com- 2000, pp. 343-252. puter science from Rutgers University. Fausto Bernardini is a research Gabriel Taubin is a research staff staff member and manager of the member and former manager of the Visual and Geometric Computing Visual and Geometric Computing group at the IBM T.J. Watson Group at the IBM T.J. Watson Research Center. His research inter- Research Center. He is known for his ests include interactive visualization work on 3D geometry compression of large models, 3D scanning and and progressive transmission ofreconstruction, and geometric modeling. He has been one polygonal models, the signal processing approach forof the principal investigators in a research effort to build a polygonal meshes, algebraic curve and surface rasteriza-digital model of Michelangelo’s Florentine Pietà. He tion, and algebraic curves and surfaces in computer vision.received an electrical engineering degree from the Univer- He received the IBM Research 1998 Computer Science Bestsity of La Sapienza, Rome, Italy, in 1990, and a PhD in Paper Award for his paper on geometry compressioncomputer science from Purdue University in 1996. through topological surgery. He earned a PhD in electri- cal engineering from Brown University and an MS in math- ematics from the University of Buenos Aires, Argentina. Holly Rushmeier is a research Readers may contact Fausto Bernadini at the IBM T.J. staff member at the IBM T.J. Watson Watson Research Center, PO Box 704, Yorktown Heights, Research Center. Her research inter- NY 10598, email ests include data visualization, ren- dering algorithms, and acquisition of input data for computer graphics For further information on this or any other computing image synthesis. She received a BS, topic, please visit our Digital Library at http://computer.MS, and PhD in mechanical engineering from Cornell Uni- org/publications/dlib.versity in 1977, 1986, and 1988, respectively. In 1990, shewas selected as a US National Science Foundation Presi-dential Young Investigator. She has served as papers chairor cochair for the ACM Siggraph conference, the IEEE Visu-alization conference, and the Eurographics RenderingWorkshop. From 1996 to 1999, she was editor in chief ofACM Transactions on Graphics. Ioana M. Martin is a research staff member in the Visual Technolo- gies Department at the IBM T.J. Wat- son Research Center. Her research interests include interactive comput- er graphics, visualization, and image processing. She has developed andimplemented methods for processing and rendering largedata sets, including progressive refinement and multires-olution techniques with applications to structural biolo-gy. She has worked on various projects including modelingwith subdivision surfaces, adaptive delivery of 3D modelsover networks, 3D data compression, and model recon-struction. She received a PhD in computer science fromPurdue University in 1996. IEEE Computer Graphics and Applications 67