1. Design of Benchmark Imagery for Validating
Facility Annotation Algorithms
Randy Roberts 1, Paul Pope 2, Raju Vatsavai 3, Ming Jiang1, Lloyd Arrowood4, Tim Trucano 5,
Shaun Gleason3, Anil Cheriyadat3, Alex Sorokine3, Aggelos Katsaggelos7,
Thrasyvoulos Pappas7, Lucinda Gaines2, Lawrence Chilton 6, and Ian Burns 2
IEEE International Geoscience and Remote Sensing Symposium
Vancouver, BC
25-29 July 2010
1 LLNL, 2 LANL, 3 ORNL, 4Y-12, 5 SNL, 6 PNNL, 7Northwestern University
Lawrence Livermore National Laboratory, PO Box 808, Livermore CA 94551-0808
This work performed under the auspices of the U.S. Department of Energy by Lawrence Livermore
LLNL-PRES-490191
National Laboratory under Contract DE-AC52-07NA27344.
2. Automated annotation of facilities is a non-trivial problem
Lawrence Livermore National Laboratory 2
3. Previous benchmarks for image annotation are
not adequate for our purposes
Caltech 101 PASCAL OIRDS
Good benchmark datasets drive algorithm research and development
Lawrence Livermore National Laboratory 3
4. Lots of factors in facility benchmark imagery
Factor Levels (3 each) Intrinsic/Extrinsic
Facility Location Urban, Suburban, Rural Intrinsic
Facility Size Small, Medium, Large Intrinsic
Compactness Sparse, Moderate, Dense Intrinsic
Roof type Flat, Sloped, Multi-faceted Intrinsic
Building Size Small, Medium, Large Intrinsic
Time-of-Day Morning, Noon, Evening Extrinsic
Sensor View Angle Nadir, Low-Oblique, High-Oblique Extrinsic
Spatial Scale Small, Medium, Large Extrinsic
Visibility 5km, 10km, 20km Extrinsic
Cloud Cover Clear, Broken, Overcast Extrinsic
Season Summer, Fall, Winter Extrinsic
Climate Zone Tropical, Temperate, Arid Extrinsic
Number of Images for a Full Factorial-Design experiment (three images per combination)
Nimages*(Levels)^Factors = 3*312 = 1,594,323 images
Lawrence Livermore National Laboratory 4
5. What objects and their spatial arrangements
constitute a facility?
The upper relationships indicate the
types of industry
The lower relationships indicate parts
(objects) that compose an industrial
facility. They were derived in part by
analysis of nouns in the paper:
“Industrial Components---A Photo
Interpretation Key on Industry,”
T. Chisnell and G. Cole, Photogrammetric
Engineering, vol 24, March 1958
Lawrence Livermore National Laboratory 5
6. Three sources of benchmark imagery
Real imagery
Composite
imagery
Synthetic
imagery
Lawrence Livermore National Laboratory 6
7. Real imagery, annotated by experts
Controlled vocabulary for annotations developed from Chisnell and Cole
Lawrence Livermore National Laboratory 7
9. Composite Imagery
USGS image 3D facility model + shadow Blending model into scene
Lawrence Livermore National Laboratory 9
10. Synthetic Facilities
“Synthesize a facility consisting of three buildings and a tank farm”
Several rendering engines available, so we’re focused on
how to arrange objects into a realistic facility
Lawrence Livermore National Laboratory 10
11. What is the cost of creating these benchmarks?
Real, annotated imagery Composite imagery Synthetic imagery
(7 experts) x (0.5 hr/image) Cost to build model Cost to build model
+ cost to reconcile variations + cost to composite into + cost to acquire/generate
in expert annotations background supporting models
(reflectance, illumination,
+ cost to acquire imagery + cost to acquire atmosphere, etc)
background imagery
+ cost to license imagery + cost to render
+ cost to license
background imagery
Lawrence Livermore National Laboratory 11
12. Future Research and Development
• Automated generation of synthetic facilities
• Expressive, usable knowledge representation
for encoding relevant aspects of facilities
• V&V methodology: How to perform robust,
comprehensive V&V using these benchmarks
• What are the proper roles of real, composite
and synthetic benchmarks?
• How good is good enough?
12
13. Three things to remember:
• Design of benchmark imagery for geospatial algorithm V&V is
a difficult problem
– Lots of factors ⇒ lots of benchmark imagery
– Complexity of scene, and objects in scene
– Geospatial extent of the imagery
• Knowledge representation (ontology) to codify the objects
(and their geospatial relationships) in the facility/scene that
are important to us
• Real, composite and synthetic imagery offer the potential to
span the space of factors for comprehensive V&V. Each has
their own cost/benefit for particular V&V tasks
The authors would like to acknowledge the support of the Simulations, Algorithms, and
Modeling program at the Office of Nonproliferation and Verification Research &
Development, National Nuclear Security Administration.
Lawrence Livermore National Laboratory 13
