The Java Advanced Imaging is a powerful Java image processing engine underlines our popular OSGeo open source projects - including GeoTools, GeoServer, GeoNetwork, and GeoNode, and more! Tragically there has been one problem with this, the JAI library is not open source!
The library originated at Sun Microsystem as a core component of the Java Runtime Environment, but was not included as part of OpenJDK collaboration.
This talk explores:
* Capabilities that make JAI attractive for GeoSpatial work
* How JAI has been used in our community
* The exciting JAI-EXT project by GeoSolutions
One of the reasons our community has been so addicted to this library is its power. It explored concepts like parallel processing, and distributed parallel processing in 1999, well ahead of the curve. It is an excellent example of engineering and software design.
Importantly we will cover the search for an open source alternative, and are the exciting progress in producing an open source alternative.
Come see how our this foundational library is being propelled into an open source future by our community.
7. JAI-EXT
JAI-Ext has been developed by GeoSolutions as an internal effort to replace Oracle JAI
operation implementations (remember deferred binding?).
The result is a strong open source project with an Apache 2.0 License.
Improves on JAI in three different ways:
● Adding more features to existing operations, like support for “no data” and ROI
● Improving performance of some operations
● Developing new operations
9. JAI-Tools
Project by Michael Bedward with a number of great ideas:
● Contour
● Image generation
● Kernel stats
● Rangelookup
● Regionalize
● Vectorize
● Zonal Stats
Project is no longer active, functionality migrating as needed to JAI-EXT.
10. Jiffle
Map algebra language by Michael Bedward:
● provides a “raster calculator” language
for dynamic processing of bands
● Integrated as a JAI Operation
Project has migrated to JAI-EXT and it is
included starting version 1.1.0
11. Jiffle Example
nir = src[7];
vir = src[3];
dest = (nir-vir)/(nir+vir);
Sentinel 2 dataset, 13
bands
Jiffle script On the fly NDVI index display
12. GeoTools ImageWorker Helper Class
Utility builder helping to run operations on images
Hides some of the complexity, optimizes repeated operations, handles differences
between JAI and JAI-EXT (e.g., can you talk about no-data, or not?)
ImageWorker worker = new ImageWorker(image);
ImageWorker = worker.scale( percent, percent, 0,0,
bilinear).
RenderedImage result = worker.getRenderedImage();
13. GeoTools GridCoverage
AbstractGridFormat format = GridFormatFinder.findFormat( file );
Hints hints = null;
if (format instanceof GeoTiffFormat) {
hints = new Hints(Hints.FORCE_LONGITUDE_FIRST_AXIS_ORDER, Boolean.TRUE);
}
GridCoverage2DReader gridReader = format.getReader(file, hints);
GridCoverage2D gridCoverage = gridReader.read(null);
AWT RenderedImage
Grid2World (AffineTransform)
CoordinateReferenceSystem
GridCoverage
The pixels
From pixel coords to real world coords
The “meaning” of the real world coords
15. GeoTools Operations
Equivalent of ImageWorker, but helps applying operations at the coverage (georeferenced) level
Operations ops = new Operations(null);
double scale = 0.5; // 50%
GridCoverage scaledCoverage = ops.scale(coverage, scale, scale, 0, 0);
16.
17.
18. JAI Licensing and Distribution
Sun Distribution License
● Free to Distribute
● Not open source!
Java Research License
● Access to source code
for research purposes only!
Distributed as “standard extension” to Java
● Additional install into JRE
● An early attempt at Java “modules”
Optional “native” acceleration mediaLib binaries
Java Runtime Environment
Core system classes
- java.lang.*
- java.io.*
- java.awt.*
- ...
Extension
javax.media.j3d.*
Extension
javax.media.jai.*
20. Two problems with JAI
● The Binary License / Research License is not open source
○ Upset projects like OSGeo Live that would like to be purely open source
○ Blocker for uDig graduating from LocationTech
● The project has been abandoned by Oracle
21. OpenCV
● Open Source Computer Vision Library
○ Roughly comparable to JAI
○ BSD license
○ Memory bound? Does it have ability to work with images larger than memory?
○ C/C++ codebase with bindings for Java
■ Some overhead taking data across JNI divide
■ Intended use is to set up a processing chain and visualize the result
● Features
○ Can use OpenCL to share workload between CPU/GPU
22. ImageJ
● “Image Processing and Analysis in Java”
○ Much better comparison with JAI
○ Open source? Not really it is public-domain which causes problem…
○ Pure-java solution, runs anywhere
○ Memory bound, wants to load images into memory for processing
● Community
○ imagej.net
○ Popular in the scientific and health community
■ Started in the 70s in Pascal
■ Large base of community developed plugins
■ FIJI (Fiji is just ImageJ) offers a distribution of plugins that work
○ Author has now retired from National Institute of Mental health, but continues to develop
■ imagej.nih.gov/ij
23. Apache Commons Imaging
● “a Pure-Java Image Library”
○ Looks to focus on great image format support
○ Open source! Apache License
○ Pure-java solution, runs anywhere
○ Memory bound!
● Community
○ No stable release available because there are lots of outstanding issues
○ Version 0.97 released in 2009 :(
24. Emerging Requirements
● Pure-java image processing
● Extendable
○ Enjoy the ability to create custom operations
○ Opens the door to native operation (example use of OpenCV)
● no artificial limit on number of bands
25. ● Long term goal is to replace JAI? Is it in
position to do so?
● Pros:
○ Large portion of operations are rewritten
○ Key components like tile cache are rewritten
○ Integrates with JDK image support, does not
reinvent the wheel
● Cons:
○ Makes use of JAI Interfaces,
(which are still not open source)
○ Usability issues around JAI Interfaces
(programming style has changed since 1999)
JAI-EXT as an Alternative ?
It is too bad we are really stuck, if only Oracle would
make the interfaces open source we are this close to
being able to do the rest!
Oracle has made the vast majority of Java open
source, but for whatever reasons JAI missed out!
26. ● Updated approach
○ wrapper implementation at OSGeo
■ Delegate to existing JAI / JAI-Ext
○ Migrate GeoTools to RPE Interfaces
○ Provide start-up switch to change between
■ Wrapper
■ Clean room
● Cons:
○ This is an idea that requires development.
■ “I tried to look at the source code, but
apparently there is not much”
○ Will not integrate with Java native raster types
Raster Processing Engine
● Raster processing engine:
○ Modern Java API using Java 8 constructs,
literate programing style, as appropriate
○ Pure Java implementation
○ Ability to stage larger rasters as tiles in
memory and process tiles in parallel
○ Clear image processing operations, allowing
installations to use native libs to accelerate
processing if available
● Initial strategy
○ Provide new “clean room” interfaces
○ Initial implementation
■ Migrate JAI-EXT operations
○ Incubation at LocationTech
■ Benefit from strong IP practices
■ fundraising opportunities
29. Bonn Sprint Objectives
● Scope the functionality required for Raster Processing Engine
○ If JAI can not be fully brought into Open Source, what do we have to replace?
● Prototype user facing interfaces
● Quickly morphed into:
○ Wait how does JAI actually work?
○ Wait what are the important interfaces
32. Bonn Sprint: Code Archeology
“Primitive civilizations must have used this to … enumerate possible categorical values?”
33. Relationship with AWT
Image: RenderedImage
● Interface
● Good for final result
● Tiled
● Relative to a larger canvas
○ Offset to (0, 0)
○ .getMinX()
○ .getMinY()
● Tile grid to larger canvas
○ Offset to (0,0)
○ .getTileGridXOffset()
○ .getTileGridYOffset()
● Tiles allow working with larger
than memory rasters.
34. Relationship with AWT
Image: PlanarImage
● Abstracts over
○ Image in memory
○ Image source
○ Operation DAG
● Does not expose TileCache
● Does not expose TileScheduler
● Execution does not happen until
○ .getData(bbox)
○ .getTile(x, y)
● Exposes Direct Acyclic Graph for
deferred Operations
○ .getSources
○ .getSinks
JAI
35.
36. Deferred Execution
Read CropWarpSource
Encode
PNG/JPEG
Tile cache
One tile at a time: handle images larger than memory
Allows also to compute multiple tiles in parallel, one per thread
Adds the basis for distributed computation (given the right tile cache)
39. JAI/JAI-EXT Entanglements
How deep are the roots of this thing?
What are we going to snag when we pull on it?
● jai-ext source:
○ public abstract class WarpOpImage extends javax.media.jai.WarpOpImage
● jai-ext implements concurrent TileCache
● TileScheduler is not concurrent
● Use of TileCache and TileScheduler is of course hidden from jai-ext ops
○ Most operations are driven by RenderedImage.getData
● TileScheduler is also hidden from jai-ext
○ Although … it is not not use Java Executors.
● So it is … possible
47. But wait ...
Oracle is using JAI inside their database product!Oracle has just donated Java Enterprise Edition to
the Eclipse Foundation, forming the Jakarta project.
I wonder if Oracle would donate the JAI interfaces to
us? It would save a lot of time.
Answer is perhaps, just one more release?
Huh?
“1.10 Oracle Multimedia Architecture”
Oracle Database Online Documentation 11g Release 1 (11.1)
49. Q: Will JAI Work with Java 11?
● Pure java implementation on the CLASSPATH should work
○ We have not tried it with the module system
● The JRE “standard extensions” system was removed in Java 9
○ Replaced by “jigsaw” module system
50. JAI Descriptor Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
RenderedOp image1 =
JAI.create(“stream”, stream);
RenderedOp image2 = ScaleDescriptor.create(
image,
2.0F, 2.0F, 0.0F, 0.0F,
Interpolate.INTERP_BILINEAR,
null);
JAI and RPE Compared
RPE Operation Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
Operation image1 =
ImageRead.stream(fileSeekableStream);
Operation image2 =
Affine.source(image1)
.scale(2.0F,2.0F)
.interpolation(Interpolation.BILINEAR)
.create();
51. JAI and RPE Compared
JAI ParameterBlock Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
RenderedOp image1 =
JAI.create(“stream”, stream);
ParameterBlock params = new ParameterBlock();
params.addSource(image1);
params.add(2.0F); // x scale factor
params.add(2.0F); // y scale factor
params.add(0.0F); // x translate
params.add(0.0F); // y translate
params.add(Interpolation.INTERP_BILINEAR);
RenderedOp image2 = JAI.create(“scale”, params);
RPE Operation Example
FileSeekableStream stream =
new FileSeekableStream(args[0]);
Operation image1 =
ImageRead.stream(fileSeekableStream)
Operation image2 =
Affine.source(image1)
.scale(2.0F,2.0F)
.interpolation(Interpolation.BILINEAR)
.create();
52. References
- Java Advanced Imaging API Home Page (Oracle)
- Introduction to the Java(TM) Advanced Imaging API (SlideShare)