Download as PDF, PPTX
![JAVA
•
Use getPixel()
•
Use getPixels()
•
Use copyPixelsToBuffer() [premultiplied]
•
GC calls. GC calls everywhere.](https://image.slidesharecdn.com/devoxx2013-efficientimageprocessing-131117153017-phpapp02/75/Efficient-Image-Processing-Nicolas-Roard-21-2048.jpg)
![NDK
void Java_com_example_rsdemo_ProcessImage_processBitmap(JNIEnv* env,
jobject this, jobject bitmap,
jint width, jint height) {
unsigned char* rgb = 0;
AndroidBitmap_lockPixels(env, bitmap, (void**) &rgb);
int len = width * height * 4;
int i;
for (i = 0; i < len; i+=4) {
int red = rgb[i];
rgb[i+1] = red;
rgb[i+2] = red;
}
AndroidBitmap_unlockPixels(env, bitmap);
}](https://image.slidesharecdn.com/devoxx2013-efficientimageprocessing-131117153017-phpapp02/75/Efficient-Image-Processing-Nicolas-Roard-50-2048.jpg)
![WITH JAVA
private void brightPass(int[] pixels, int width, int height) {
int threshold = (int) (brightnessThreshold * 255);
int r;
int g;
int b;
int luminance;
int[] luminanceData = new int[3 * 256];
// pre-computations
for (int i = 0; i <
luminanceData[i
luminanceData[i
luminanceData[i
}
for conversion from RGB to YCC
luminanceData.length; i += 3) {
] = (int) (i * 0.2125f);
+ 1] = (int) (i * 0.7154f);
+ 2] = (int) (i * 0.0721f);](https://image.slidesharecdn.com/devoxx2013-efficientimageprocessing-131117153017-phpapp02/75/Efficient-Image-Processing-Nicolas-Roard-64-2048.jpg)
![int index = 0;
for (int y = 0; y < height; y++) {
for (int x = 0; x < width; x++) {
int pixel = pixels[index];
// unpack the pixel's components
r = pixel >> 16 & 0xFF;
g = pixel >> 8 & 0xFF;
b = pixel
& 0xFF;
// compute the luminance
luminance = luminanceData[r * 3] + luminanceData[g * 3 + 1] +
luminanceData[b * 3 + 2];
// apply the treshold to select the brightest pixels
luminance = Math.max(0, luminance - threshold);
int sign = (int) Math.signum(luminance);
// pack the components in a single pixel
pixels[index] = 0xFF000000 | (r * sign) < < 16 |
(g * sign) << 8 | (b * sign);
index++;
}
}
}](https://image.slidesharecdn.com/devoxx2013-efficientimageprocessing-131117153017-phpapp02/75/Efficient-Image-Processing-Nicolas-Roard-65-2048.jpg)
![WITH GL SHADER
uniform sampler2D baseImage;
uniform float brightPassThreshold;
void main(void) {
vec3 luminanceVector = vec3(0.2125, 0.7154, 0.0721);
vec4 sample = texture2D(baseImage, gl_TexCoord[0].st);
float luminance = dot(luminanceVector, sample.rgb);
luminance = max(0.0, luminance - brightPassThreshold);
sample.rgb *= sign(luminance);
sample.a = 1.0;
gl_FragColor = sample;
}](https://image.slidesharecdn.com/devoxx2013-efficientimageprocessing-131117153017-phpapp02/75/Efficient-Image-Processing-Nicolas-Roard-66-2048.jpg)
Uploaded byParis Android User Group
Efficient Image Processing - Nicolas Roard
The document discusses efficient image processing techniques for Android, focusing on the RenderScript framework. It provides an overview of RenderScript, how to write kernels in C, and how to call them from Java. Examples are given for common image processing tasks like grayscale conversion, bloom effects, and local adjustments. Caching strategies and handling low-memory devices are also covered to ensure performance across all hardware.
More Related Content
Similar to Efficient Image Processing - Nicolas Roard
More from Paris Android User Group
![Vibe Coding vs. Spec-Driven Development [Free Meetup]](https://cdn.slidesharecdn.com/ss_thumbnails/vibecodingvsspecdrivendevelopment-251209105622-43f455e7-thumbnail.jpg?width=640&height=640&fit=bounds)
Efficient Image Processing - Nicolas Roard
- 1. EFFICIENT IMAGE PROCESSING ONANDROID Nicolas Roard
- 2. EFFICIENT IMAGE PROCESSING • Workswell on all hardware • Fast, ideally realtime interaction • Handles complex and flexible processing • Handles large images • Minimize memory usage
- 3. WE WANT TOHAVE OUR CAKE AND EAT IT TOO
- 4. ANDROID KITKAT PHOTO EDITOR • Non-destructiveedits • Full-size images processing • Combine effects freely • Easy to use, yet powerful: grow with the user
- 5. NON-DESTRUCTIVE EDITS • Effects aremodifiable or reversible without quality loss • Allow re-edits of processed images
- 6. RENDERSCRIPT (RS) • Cool thingythat let you do fast image processing
- 7. TIMELINE 3 VERSIONS, 10MONTHS
- 8. 4.2 - NOVEMBER2012 • Color FX (9 looks) • 11 Borders • Geometry: Straighten, Rotate, Crop, Mirror • Filters & Tools • Autocolor, Exposure, Vignette, Contrast, Shadows, Vibrance, Sharpness (RS-based), Curves, Hue, Saturation, BW Filter • Non-destructive edits (in the editor -- save create a copy) • Exposed history
- 9. G+ EDITOR -MAY 2013 • RenderScript implementations of Snapseed filters • Frames, Film, Drama, Retrolux • Non-destructive • Cloud-based (local processing only used for caching and UI interactions)
- 10. 4.3 - JULY2013 • Move to RenderScript • New 16 image-based borders (ported from Snapseed, RS-based) • Filters & Tools • • • Highlights, Improved Vignette Local adjustment (ported from Snapseed, RS-based) New Tablet UI, refined UI, introduction of the state panel instead of the history panel
- 11. 4.4 - SEPTEMBER2013 • Filters & Tools • Custom borders, Drawing tool, negative, posterize • RS filters: Graduated filter, Vignette, per channel saturation, sharpness/structure • Refined UI (animations, etc.) • Pinch to zoom enabled (full-res zoom) • Re-edits enabled • Background save service, export, print support
- 12.
- 15. SOME ADDITIONAL INFOS • Phoneand Tablet UI • Filters in C & RenderScript • Works on Full Size images -- largest tried was a 278MP image on a Nexus 7 2nd gen. Limited by available RAM. • Nearly all of the editor is in AOSP!
- 16.
- 17.
- 18.
- 19.
- 20. IMAGE PROCESSING • In Java • Innative code (JNI calls to C/C++) • In OpenGLES2 • RenderScript
- 21. JAVA • Use getPixel() • Use getPixels() • UsecopyPixelsToBuffer() [premultiplied] • GC calls. GC calls everywhere.
- 22. NATIVE CODE • Pass aBitmap through JNI to C/C++ • Quite fast & pretty easy to work with (pointer to the bitmap -- and no GC!) • JNI / Native can be fastidious • Handling different CPU architectures can be an issue • Optimizations can be complicated • JNI management
- 23. OPENGL ES 2.0 • Fast-- can write interactive processing • Hard to ensure the shaders will perform well on all devices • Limited in size (max texture size...) • Needs adhoc shaders, i.e. fixed pipelines. • Expensive to retrieve processed image
- 24.
- 25. “RENDERSCRIPT IS AFRAMEWORK FOR RUNNING COMPUTATIONALLY INTENSIVE TASKS AT HIGH PERFORMANCE ON ANDROID. RENDERSCRIPT IS PRIMARILY ORIENTED FOR USE WITH DATA-PARALLEL COMPUTATION, ALTHOUGH SERIAL COMPUTATIONALLY INTENSIVE WORKLOADS CAN BENEFIT AS WELL.”
- 26. • Write “kernels” ina C99-like language with vector extensions and useful intrinsics • RenderScript executes them in parallel, on the GPU or CPU • Java used to manage lifetime of objects/allocations and control of execution • Portability
- 27. RENDERSCRIPT • Fast -- throughLLVM optimizations and Parallelization • Supports CPU / GPU • Compatibility Library • Easy to offload to the background • Pretty easy to write
- 28. RENDERSCRIPT • Cannot allocate memoryfrom kernels, need to do it from outside • RenderScript can be called from Java or from Native • Compatibility library!
- 32. HOW? • Optimized Math library • Optimizationson the device • Easier to read & maintain (vector math library helps)
- 33. ALLOCATIONS • Bound to Scripts • Canbe bound to SurfaceTexture (producer & consumer) • Can share memory between Allocation and Bitmap
- 34.
- 35. 1. SCRIPT #pragma version(1) #pragmars java_package_name(com.example.rsdemo) uchar4 __attribute__((kernel)) color(uchar4 in) { return in; }
- 36. 2. CREATE CONTEXT RenderScriptmRS = RenderScript.create(mContext); ScriptC_filter filter = new ScriptC_filter(mRS, mResources, R.raw.filter);
- 37. 3. LOAD BITMAP BitmapbitmapIn = BitmapFactory.decodeResource( getResources(), R.drawable.monumentvalley); Bitmap bitmapOut = bitmapIn.copy(bitmapIn.getConfig(), true);
- 38. 4. CREATE ALLOCATIONS Allocationin = Allocation.createFromBitmap(mRS, bitmapIn, Allocation.MipmapControl.MIPMAP_NONE, Allocation.USAGE_SCRIPT | Allocation.USAGE_SHARED); Allocation out = Allocation.createTyped(mRS, in.getType());
- 39. 5. APPLY THESCRIPT filter.forEach_color(in, out); out.copyTo(bitmapOut);
- 41. SCRIPT INTRINSICS ScriptIntrinsicBlur blur= ScriptIntrinsicBlur.create(mRS, Element.U8_4(mRS)); blur.setRadius(25.f); blur.setInput(in); blur.forEach(in);
- 44.
- 45. PAINT IT BLACK (ORGRAY)
- 46. SCRIPT uchar4 __attribute__((kernel)) color(uchar4in) { return in; }
- 47. SCRIPT uchar4 __attribute__((kernel)) grey(uchar4in) { in.g = in.r; in.b = in.r; return in; }
- 48. SCRIPT uchar4 __attribute__((kernel)) grey(uchar4in) { in.gb = in.r; return in; }
- 50. NDK void Java_com_example_rsdemo_ProcessImage_processBitmap(JNIEnv* env, jobjectthis, jobject bitmap, jint width, jint height) { unsigned char* rgb = 0; AndroidBitmap_lockPixels(env, bitmap, (void**) &rgb); int len = width * height * 4; int i; for (i = 0; i < len; i+=4) { int red = rgb[i]; rgb[i+1] = red; rgb[i+2] = red; } AndroidBitmap_unlockPixels(env, bitmap); }
- 51.
- 53. SCRIPT uchar4 __attribute__((kernel)) grey(uchar4in) { in.g = in.r; in.b = in.r; return in; }
- 54. SCRIPT uchar4 __attribute__((kernel)) grey1(uchar4in, uint32_t x, uint32_t y) { in.g = in.r; in.b = in.r; return in; }
- 55. SCRIPT uchar4 __attribute__((kernel)) grey1(uchar4in, uint32_t x, uint32_t y) { float range = (float) x / width; uint32_t grey = (1 - range) * in.r; in.r = grey; in.g = grey; in.b = grey; return in; }
- 57. SCRIPT uchar4 __attribute__((kernel)) grey1(uchar4in, uint32_t x, uint32_t y) { float range = (float) x / width; uint32_t grey = (1 - range) * in.r; in.r = grey; in.g = grey; in.b = grey; return in; }
- 58. SCRIPT uchar4 __attribute__((kernel)) grey2(uchar4in, uint32_t x, uint32_t y) { float range = (float) x / width; uint32_t grey = (1 - range) * in.r; in.r = (in.r * range) + grey; in.g = (in.g * range) + grey; in.b = (in.b * range) + grey; return in; }
- 59. SCRIPT uchar4 __attribute__((kernel)) grey3(uchar4in, uint32_t x, uint32_t y) { float range = (float) x / width; float4 pixel = rsUnpackColor8888(in); float grey = (1 - range) * pixel.r; pixel.r = pixel.r * range + grey; pixel.g = pixel.g * range + grey; pixel.b = pixel.b * range + grey; return rsPackColorTo8888( clamp(pixel, 0.f, 1.0f)); }
- 60. SCRIPT uchar4 __attribute__((kernel)) grey4(uchar4in, uint32_t x, uint32_t y) { float range = (float) x / width; float4 pixel = rsUnpackColor8888(in); float grey = (1 - range) * pixel.r; pixel.rgb = pixel.rgb * range + grey; return rsPackColorTo8888( clamp(pixel, 0.f, 1.0f)); }
- 61. EXAMPLE: BLOOM • Select thebright pixels • Blur the result • Add the blurred bright pixels back to the image
- 64. WITH JAVA private voidbrightPass(int[] pixels, int width, int height) { int threshold = (int) (brightnessThreshold * 255); int r; int g; int b; int luminance; int[] luminanceData = new int[3 * 256]; // pre-computations for (int i = 0; i < luminanceData[i luminanceData[i luminanceData[i } for conversion from RGB to YCC luminanceData.length; i += 3) { ] = (int) (i * 0.2125f); + 1] = (int) (i * 0.7154f); + 2] = (int) (i * 0.0721f);
- 65. int index =0; for (int y = 0; y < height; y++) { for (int x = 0; x < width; x++) { int pixel = pixels[index]; // unpack the pixel's components r = pixel >> 16 & 0xFF; g = pixel >> 8 & 0xFF; b = pixel & 0xFF; // compute the luminance luminance = luminanceData[r * 3] + luminanceData[g * 3 + 1] + luminanceData[b * 3 + 2]; // apply the treshold to select the brightest pixels luminance = Math.max(0, luminance - threshold); int sign = (int) Math.signum(luminance); // pack the components in a single pixel pixels[index] = 0xFF000000 | (r * sign) < < 16 | (g * sign) << 8 | (b * sign); index++; } } }
- 66. WITH GL SHADER uniformsampler2D baseImage; uniform float brightPassThreshold; void main(void) { vec3 luminanceVector = vec3(0.2125, 0.7154, 0.0721); vec4 sample = texture2D(baseImage, gl_TexCoord[0].st); float luminance = dot(luminanceVector, sample.rgb); luminance = max(0.0, luminance - brightPassThreshold); sample.rgb *= sign(luminance); sample.a = 1.0; gl_FragColor = sample; }
- 67. WITH RENDERSCRIPT float brightPassThreshold; uchar4__attribute__((kernel)) brightPass(uchar4 in) { float3 luminanceVector = { 0.2125, 0.7154, 0.0721 }; float4 pixel = rsUnpackColor8888(in); float luminance = dot(luminanceVector, pixel.rgb); luminance = max(0.0f, luminance - brightPassThreshold); pixel.rgb *= sign(luminance); pixel.a = 1.0; return rsPackColorTo8888(clamp(pixel, 0.f, 1.0f)); }
- 68. JAVA-SIDE ScriptIntrinsicBlur blur = ScriptIntrinsicBlur.create(mRS,Element.U8_4(mRS)); ScriptIntrinsicBlend blend = ScriptIntrinsicBlend.create(mRS, Element.U8_4(mRS)); filter.set_brightPassThreshold(0.15f); filter.forEach_brightPass(in, out); blur.setRadius(25.f); blur.setInput(out); blur.forEach(out); blend.forEachAdd(in, out); out.copyTo(bitmapOut);
- 69.
- 70. WORKING WELL ONALL HARDWARE • Architect for the worst • Scale with the device capabilities • Screen size / dpi • Available memory • Available CPU / GPU • Think about what is a downgraded experience
- 71. LOADING • Load in thebackground • • AsyncTask, or use a background thread Bitmaps loading • query the size • inSampleSize • reuseBitmap • BitmapRegionDecoder
- 72. QUERY THE SIZE BitmapFactory.Optionsoptions = new BitmapFactory.Options(); options.inJustDecodeBounds = true; BitmapFactory.decodeResource( getResources(), R.id.myimage, options); int imageHeight = options.outHeight; int imageWidth = options.outWidth; String imageType = options.outMimeType;
- 73. INSAMPLESIZE • Only load whatyou need • needs to be a power of two, so for a 2048x2048 image, • insamplesize=2 => 1024x1024 image • insamplesize=4 => 512x512 image
- 74. CALCULATE INSAMPLESIZE if (bounds.width()> destination.width()) { int sampleSize = 1; int w = bounds.width(); while (w > destination.width()) { sampleSize *= 2; w /= sampleSize; } options.inSampleSize = sampleSize; }
- 75. REUSE BITMAP BitmapFactory.Options options; (...) BitmapinBitmap = ... (...) options.inBitmap = inBitmap; API level 11 (Android 3.0) Before API level 19 (Android 4.4) only same size
- 76. BITMAP REGIONDECODER InputStream is= ... BitmapRegionDecoder decoder = BitmapRegionDecoder.newInstance(is, false); Rect imageBounds = ... Bitmap bitmap = decoder.decodeRegion(imageBounds, options); API level 11 (Android 3.0)
- 77.
- 78. PIPELINE • Run in abackground service (used when saving too) • Mix C, RenderScript, java filtering (canvas draw) • multiple pipelines in parallel (direct preview, highres, icons, full res, geometry, saving)
- 79. FLEXIBLE PROCESSING • Unbounded pipeline • Nofixed order • Complex filters • Geometry-based • Global • Local
- 80.
- 81.
- 82. CACHING • Cache RS scripts,allocations • Cache original bitmaps • Aggressively destroy/recycle resources in filters to keep memory low • If possible, filters should process the input bitmap directly • N-1 cache
- 83. MEMORY USAGE • Bitmap cacheheavily reusing bitmaps • LruCache class (available in support lib too) • Pick image sizes depending on the device resolution / DPI • Have a path ready for a downgraded experience
- 84. DEVICE CAPABILITIES • Ask thesystem • Runtime.getRuntime().maxMemory() • New isLowRamDevice() API • Handles low-memory signals • Handles java.lang.OutOfMemory exceptions
- 85.
- 86.
- 87.
- 88.
- 89.
- 90.
- 91.
- 92.
- 93.
- 94.
- 95.
- 96.
- 97. N-1 CACHING • No silverbullet • Only really useful when the user manipulates the last filters of the pipeline... • ...but this is after all the more common scenario!
- 98.
- 99. REALTIME INTERACTION • Background processing • Minimizeallocations -- Careful with Garbage Collection! • Optimized filters, RenderScript helps • Caching in the pipeline • Low/High resolution preview
- 100.
- 101.
- 102.
- 103. PREVIEW SYSTEM Low-res preview Request High-respreview Full-res preview If new request, delay more
- 104. HOW TO CHEAT • Thetriple-buffer preview can have a low resolution • The UI elements and controls are animated and manipulated at 60 fps on the UI thread • The rendering pipeline can (and needs to) be interrupted
- 105. INTERRUPTION IN RENDERSCRIPT LaunchOptions options= new LaunchOptions(); options.setX(xStart, xEnd); options.setY(yStart, yEnd); mScript.forEach_vignette(in, out, options);
- 106. LaunchOptions options =new LaunchOptions(); boolean even = true; int tile = 128; int height = bitmapIn.getHeight(); int width = bitmapIn.getWidth(); for (int yStart = 0; yStart < height; yStart += tile) { for (int xStart = 0; xStart < width; xStart += tile) { int xEnd = xStart + tile; int yEnd = yStart + tile; options.setX(xStart, xEnd); options.setY(yStart, yEnd); if (even) { filter.forEach_grey(in, out, options); } else { filter.forEach_color(in, out, options); } even = !even; } }
- 108.
- 109. FULL RESOLUTION PREVIEW • Ideally, weshould use a tile-based rendering • At the moment, we use BitmapFactory region decoder instead • Filters need to be able to handle partial regions • future payoff: streaming save
- 110.
- 111. FUTURE -- INTHE PHOTO EDITOR • Merging filters • • • RenderScript At the pipeline level (color cubes...) Streaming saving • Some code is there (AOSP), but not used
- 112. FUTURE -- INANDROID FRAMEWORK • TileView • Adding filtering pipeline in android framework • Image loader improvements • RAW support? Color correction?
- 113.
- 114.
- 115.
- 116. TILEVIEW ADAPTER public interfaceTileViewAdapter { public int getTileSize(); public int getContentWidth(); public int getContentHeight(); public void onPaint(float scale, float dx, float dy, Bitmap bitmap); public Bitmap getFullImage(int max); void setDebug(boolean debug); }
- 117.