Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Java Image Processing for Geospatial Community

374 views

Published on

The Java Advanced Imaging is a powerful Java image processing engine underlines our popular OSGeo open source projects - including GeoTools, GeoServer, GeoNetwork, and GeoNode, and more! Tragically there has been one problem with this, the JAI library is not open source!

The library originated at Sun Microsystem as a core component of the Java Runtime Environment, but was not included as part of OpenJDK collaboration.

This talk explores:

* Capabilities that make JAI attractive for GeoSpatial work
* How JAI has been used in our community
* The exciting JAI-EXT project by GeoSolutions

One of the reasons our community has been so addicted to this library is its power. It explored concepts like parallel processing, and distributed parallel processing in 1999, well ahead of the curve. It is an excellent example of engineering and software design.

Importantly we will cover the search for an open source alternative, and are the exciting progress in producing an open source alternative.

Come see how our this foundational library is being propelled into an open source future by our community.

Published in: Software
  • Be the first to comment

Java Image Processing for Geospatial Community

  1. 1. State of JAI Java Image Processing for GeoSpatial Community
  2. 2. Java Advanced Imaging
  3. 3. What is Java Advanced Imaging? ● Image Processing Library for Java Applications ○ a “pure java” solution for image processing ● Tiling computational model ○ Images accessed tile-by-tile ○ Centrally-maintained cache for performance ○ Facilitates multi-threaded processing ● Deferred execution ○ Construct a chain of operations ○ Only compute as needed ● Threaded computation ● Object-oriented extensibility ○ Register your own operations ○ Or override the built-in operations ● Remote imaging ○ Evaluate operator chains on remote hosts ○ Transfer images tile by tile ● Border extension ● Image warping ● Pixel interpolation ● Graphics2D Drawing ● Regions of Interest (ROI) ● Image file handling ○ Using JAI ImageIO project
  4. 4. Demo!
  5. 5. What else was cool in 1999
  6. 6. How JAI has been used for GeoSpatial
  7. 7. JAI-EXT JAI-Ext has been developed by GeoSolutions as an internal effort to replace Oracle JAI operation implementations (remember deferred binding?). The result is a strong open source project with an Apache 2.0 License. Improves on JAI in three different ways: ● Adding more features to existing operations, like support for “no data” and ROI ● Improving performance of some operations ● Developing new operations
  8. 8. JAI-EXT Functionality ● Operations implemented supporting nodata ○ Affine* ○ BandMerge ○ Border ○ Crop ○ Lookup ○ Mosaic ○ Null ○ Rescale ○ Scale* ○ Statistic ○ Translate ○ Warp* ○ ZonalStatistics * Support nearest-neighbor, bilinear, bicubic interpolation ● ConcurrentCache ○ Replacement JAI TileCache ○ Offering better concurrency ○ Uses Guava cache internally ● And more ○ Public roadmap
  9. 9. JAI-Tools Project by Michael Bedward with a number of great ideas: ● Contour ● Image generation ● Kernel stats ● Rangelookup ● Regionalize ● Vectorize ● Zonal Stats Project is no longer active, functionality migrating as needed to JAI-EXT.
  10. 10. Jiffle Map algebra language by Michael Bedward: ● provides a “raster calculator” language for dynamic processing of bands ● Integrated as a JAI Operation Project has migrated to JAI-EXT and it is included starting version 1.1.0
  11. 11. Jiffle Example nir = src[7]; vir = src[3]; dest = (nir-vir)/(nir+vir); Sentinel 2 dataset, 13 bands Jiffle script On the fly NDVI index display
  12. 12. GeoTools ImageWorker Helper Class Utility builder helping to run operations on images Hides some of the complexity, optimizes repeated operations, handles differences between JAI and JAI-EXT (e.g., can you talk about no-data, or not?) ImageWorker worker = new ImageWorker(image); ImageWorker = worker.scale( percent, percent, 0,0, bilinear). RenderedImage result = worker.getRenderedImage();
  13. 13. GeoTools GridCoverage AbstractGridFormat format = GridFormatFinder.findFormat( file ); Hints hints = null; if (format instanceof GeoTiffFormat) { hints = new Hints(Hints.FORCE_LONGITUDE_FIRST_AXIS_ORDER, Boolean.TRUE); } GridCoverage2DReader gridReader = format.getReader(file, hints); GridCoverage2D gridCoverage = gridReader.read(null); AWT RenderedImage Grid2World (AffineTransform) CoordinateReferenceSystem GridCoverage The pixels From pixel coords to real world coords The “meaning” of the real world coords
  14. 14. GeoTools CoverageProcessor CoverageProcessor processor = CoverageProcessor.getInstance(); final ParameterValueGroup param = processor.getOperation("CoverageCrop").getParameters(); param.parameter("Source").setValue(gridCoverage); param.parameter("Envelope").setValue(envelope); GridCoverage2D result = processor.doOperation(param); AWT RenderedImage Grid2World (AffineTransform) CoordinateReferenceSystem AWT RenderedImage Grid2World (AffineTransform) CoordinateReferenceSystem CROP
  15. 15. GeoTools Operations Equivalent of ImageWorker, but helps applying operations at the coverage (georeferenced) level Operations ops = new Operations(null); double scale = 0.5; // 50% GridCoverage scaledCoverage = ops.scale(coverage, scale, scale, 0, 0);
  16. 16. JAI Licensing and Distribution Sun Distribution License ● Free to Distribute ● Not open source! Java Research License ● Access to source code for research purposes only! Distributed as “standard extension” to Java ● Additional install into JRE ● An early attempt at Java “modules” Optional “native” acceleration mediaLib binaries Java Runtime Environment Core system classes - java.lang.* - java.io.* - java.awt.* - ... Extension javax.media.j3d.* Extension javax.media.jai.*
  17. 17. Making Peace with Open Source
  18. 18. Two problems with JAI ● The Binary License / Research License is not open source ○ Upset projects like OSGeo Live that would like to be purely open source ○ Blocker for uDig graduating from LocationTech ● The project has been abandoned by Oracle
  19. 19. OpenCV ● Open Source Computer Vision Library ○ Roughly comparable to JAI ○ BSD license ○ Memory bound? Does it have ability to work with images larger than memory? ○ C/C++ codebase with bindings for Java ■ Some overhead taking data across JNI divide ■ Intended use is to set up a processing chain and visualize the result ● Features ○ Can use OpenCL to share workload between CPU/GPU
  20. 20. ImageJ ● “Image Processing and Analysis in Java” ○ Much better comparison with JAI ○ Open source? Not really it is public-domain which causes problem… ○ Pure-java solution, runs anywhere ○ Memory bound, wants to load images into memory for processing ● Community ○ imagej.net ○ Popular in the scientific and health community ■ Started in the 70s in Pascal ■ Large base of community developed plugins ■ FIJI (Fiji is just ImageJ) offers a distribution of plugins that work ○ Author has now retired from National Institute of Mental health, but continues to develop ■ imagej.nih.gov/ij
  21. 21. Apache Commons Imaging ● “a Pure-Java Image Library” ○ Looks to focus on great image format support ○ Open source! Apache License ○ Pure-java solution, runs anywhere ○ Memory bound! ● Community ○ No stable release available because there are lots of outstanding issues ○ Version 0.97 released in 2009 :(
  22. 22. Emerging Requirements ● Pure-java image processing ● Extendable ○ Enjoy the ability to create custom operations ○ Opens the door to native operation (example use of OpenCV) ● no artificial limit on number of bands
  23. 23. ● Long term goal is to replace JAI? Is it in position to do so? ● Pros: ○ Large portion of operations are rewritten ○ Key components like tile cache are rewritten ○ Integrates with JDK image support, does not reinvent the wheel ● Cons: ○ Makes use of JAI Interfaces, (which are still not open source) ○ Usability issues around JAI Interfaces (programming style has changed since 1999) JAI-EXT as an Alternative ? It is too bad we are really stuck, if only Oracle would make the interfaces open source we are this close to being able to do the rest! Oracle has made the vast majority of Java open source, but for whatever reasons JAI missed out!
  24. 24. ● Updated approach ○ wrapper implementation at OSGeo ■ Delegate to existing JAI / JAI-Ext ○ Migrate GeoTools to RPE Interfaces ○ Provide start-up switch to change between ■ Wrapper ■ Clean room ● Cons: ○ This is an idea that requires development. ■ “I tried to look at the source code, but apparently there is not much” ○ Will not integrate with Java native raster types Raster Processing Engine ● Raster processing engine: ○ Modern Java API using Java 8 constructs, literate programing style, as appropriate ○ Pure Java implementation ○ Ability to stage larger rasters as tiles in memory and process tiles in parallel ○ Clear image processing operations, allowing installations to use native libs to accelerate processing if available ● Initial strategy ○ Provide new “clean room” interfaces ○ Initial implementation ■ Migrate JAI-EXT operations ○ Incubation at LocationTech ■ Benefit from strong IP practices ■ fundraising opportunities
  25. 25. Bonn Code Sprint Raster Processing Engine Scoping Exercise
  26. 26. Bonn Code Sprint
  27. 27. Bonn Sprint Objectives ● Scope the functionality required for Raster Processing Engine ○ If JAI can not be fully brought into Open Source, what do we have to replace? ● Prototype user facing interfaces ● Quickly morphed into: ○ Wait how does JAI actually work? ○ Wait what are the important interfaces
  28. 28. We don’t write docs this way anymore
  29. 29. Bonn Sprint: Code Archeology “Primitive civilizations must have used this to … enumerate possible categorical values?”
  30. 30. Relationship with AWT Image: RenderedImage ● Interface ● Good for final result ● Tiled ● Relative to a larger canvas ○ Offset to (0, 0) ○ .getMinX() ○ .getMinY() ● Tile grid to larger canvas ○ Offset to (0,0) ○ .getTileGridXOffset() ○ .getTileGridYOffset() ● Tiles allow working with larger than memory rasters.
  31. 31. Relationship with AWT Image: PlanarImage ● Abstracts over ○ Image in memory ○ Image source ○ Operation DAG ● Does not expose TileCache ● Does not expose TileScheduler ● Execution does not happen until ○ .getData(bbox) ○ .getTile(x, y) ● Exposes Direct Acyclic Graph for deferred Operations ○ .getSources ○ .getSinks JAI
  32. 32. Deferred Execution Read CropWarpSource Encode PNG/JPEG Tile cache One tile at a time: handle images larger than memory Allows also to compute multiple tiles in parallel, one per thread Adds the basis for distributed computation (given the right tile cache)
  33. 33. Deferred Binding Read OpImageOpImage Source
  34. 34. Deferred Binding CropOpImageWrapOpImage Read OpImageOpImage Source Encode PNG/JPEG
  35. 35. JAI/JAI-EXT Entanglements How deep are the roots of this thing? What are we going to snag when we pull on it? ● jai-ext source: ○ public abstract class WarpOpImage extends javax.media.jai.WarpOpImage ● jai-ext implements concurrent TileCache ● TileScheduler is not concurrent ● Use of TileCache and TileScheduler is of course hidden from jai-ext ops ○ Most operations are driven by RenderedImage.getData ● TileScheduler is also hidden from jai-ext ○ Although … it is not not use Java Executors. ● So it is … possible
  36. 36. JAIRPE
  37. 37. RPE
  38. 38. JAI ParameterBlock Example FileSeekableStream stream = new FileSeekableStream(args[0]); RenderedOp image1 = JAI.create(“stream”, stream); ParameterBlock params = new ParameterBlock(); params.addSource(image1); params.add(2.0F); // x scale factor params.add(2.0F); // y scale factor params.add(0.0F); // x translate params.add(0.0F); // y translate params.add(Interpolation.INTERP_BILINEAR); /* Create an operator to scale image1. */ RenderedOp image2 = JAI.create(“scale”, params, hints);
  39. 39. FileSeekableStream stream = new FileSeekableStream(args[0]); RenderedOp image1 = JAI.create(“stream”, stream); RenderedOp image2 = ScaleDescriptor.create(image1,2.0F, 2.0F, 0.0F, 0.0F,Interpolate.INTERP_BILINEAR, hints); JAI Descriptor Example
  40. 40. Raster Processing Engine Example FileSeekableStream stream = new FileSeekableStream(args[0]); Operation image1 = ImageRead.stream(fileSeekableStream) Operation image2 = Affine.source(image1) .scale(2.0F,2.0F) .interpolation(Interpolation.BILINEAR).create();
  41. 41. Interaction with Java SWT Image
  42. 42. Audit of GeoTools and GeoServer Use
  43. 43. But wait ... Oracle is using JAI inside their database product!Oracle has just donated Java Enterprise Edition to the Eclipse Foundation, forming the Jakarta project. I wonder if Oracle would donate the JAI interfaces to us? It would save a lot of time. Answer is perhaps, just one more release? Huh? “1.10 Oracle Multimedia Architecture” Oracle Database Online Documentation 11g Release 1 (11.1)
  44. 44. Questions and Discussion
  45. 45. Q: Will JAI Work with Java 11? ● Pure java implementation on the CLASSPATH should work ○ We have not tried it with the module system ● The JRE “standard extensions” system was removed in Java 9 ○ Replaced by “jigsaw” module system
  46. 46. JAI Descriptor Example FileSeekableStream stream = new FileSeekableStream(args[0]); RenderedOp image1 = JAI.create(“stream”, stream); RenderedOp image2 = ScaleDescriptor.create( image, 2.0F, 2.0F, 0.0F, 0.0F, Interpolate.INTERP_BILINEAR, null); JAI and RPE Compared RPE Operation Example FileSeekableStream stream = new FileSeekableStream(args[0]); Operation image1 = ImageRead.stream(fileSeekableStream); Operation image2 = Affine.source(image1) .scale(2.0F,2.0F) .interpolation(Interpolation.BILINEAR) .create();
  47. 47. JAI and RPE Compared JAI ParameterBlock Example FileSeekableStream stream = new FileSeekableStream(args[0]); RenderedOp image1 = JAI.create(“stream”, stream); ParameterBlock params = new ParameterBlock(); params.addSource(image1); params.add(2.0F); // x scale factor params.add(2.0F); // y scale factor params.add(0.0F); // x translate params.add(0.0F); // y translate params.add(Interpolation.INTERP_BILINEAR); RenderedOp image2 = JAI.create(“scale”, params); RPE Operation Example FileSeekableStream stream = new FileSeekableStream(args[0]); Operation image1 = ImageRead.stream(fileSeekableStream) Operation image2 = Affine.source(image1) .scale(2.0F,2.0F) .interpolation(Interpolation.BILINEAR) .create();
  48. 48. References - Java Advanced Imaging API Home Page (Oracle) - Introduction to the Java(TM) Advanced Imaging API (SlideShare)

×