Efficient Image Processing - Nicolas Roard

1,975 views

Published on

Published in: Technology, Art & Photos
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,975
On SlideShare
0
From Embeds
0
Number of Embeds
217
Actions
Shares
0
Downloads
33
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

Efficient Image Processing - Nicolas Roard

  1. 1. EFFICIENT IMAGE PROCESSING ON ANDROID Nicolas Roard
  2. 2. EFFICIENT IMAGE PROCESSING • Works well on all hardware • Fast, ideally realtime interaction • Handles complex and flexible processing • Handles large images • Minimize memory usage
  3. 3. WE WANT TO HAVE OUR CAKE AND EAT IT TOO
  4. 4. ANDROID KITKAT PHOTO EDITOR • Non-destructive edits • Full-size images processing • Combine effects freely • Easy to use, yet powerful: grow with the user
  5. 5. NON-DESTRUCTIVE EDITS • Effects are modifiable or reversible without quality loss • Allow re-edits of processed images
  6. 6. RENDERSCRIPT (RS) • Cool thingy that let you do fast image processing
  7. 7. TIMELINE 3 VERSIONS, 10 MONTHS
  8. 8. 4.2 - NOVEMBER 2012 • Color FX (9 looks) • 11 Borders • Geometry: Straighten, Rotate, Crop, Mirror • Filters & Tools • Autocolor, Exposure, Vignette, Contrast, Shadows, Vibrance, Sharpness (RS-based), Curves, Hue, Saturation, BW Filter • Non-destructive edits (in the editor -- save create a copy) • Exposed history
  9. 9. G+ EDITOR - MAY 2013 • RenderScript implementations of Snapseed filters • Frames, Film, Drama, Retrolux • Non-destructive • Cloud-based (local processing only used for caching and UI interactions)
  10. 10. 4.3 - JULY 2013 • Move to RenderScript • New 16 image-based borders (ported from Snapseed, RS-based) • Filters & Tools • • • Highlights, Improved Vignette Local adjustment (ported from Snapseed, RS-based) New Tablet UI, refined UI, introduction of the state panel instead of the history panel
  11. 11. 4.4 - SEPTEMBER 2013 • Filters & Tools • Custom borders, Drawing tool, negative, posterize • RS filters: Graduated filter, Vignette, per channel saturation, sharpness/structure • Refined UI (animations, etc.) • Pinch to zoom enabled (full-res zoom) • Re-edits enabled • Background save service, export, print support
  12. 12. DEMO
  13. 13. SOME ADDITIONAL INFOS • Phone and Tablet UI • Filters in C & RenderScript • Works on Full Size images -- largest tried was a 278MP image on a Nexus 7 2nd gen. Limited by available RAM. • Nearly all of the editor is in AOSP!
  14. 14. IMAGE PROCESSING
  15. 15. PIPELINE Original Image
  16. 16. PIPELINE Original Image Filter
  17. 17. PIPELINE Original Image Filter Processed Image
  18. 18. IMAGE PROCESSING • In Java • In native code (JNI calls to C/C++) • In OpenGLES2 • RenderScript
  19. 19. JAVA • Use getPixel() • Use getPixels() • Use copyPixelsToBuffer() [premultiplied] • GC calls. GC calls everywhere.
  20. 20. NATIVE CODE • Pass a Bitmap through JNI to C/C++ • Quite fast & pretty easy to work with (pointer to the bitmap -- and no GC!) • JNI / Native can be fastidious • Handling different CPU architectures can be an issue • Optimizations can be complicated • JNI management
  21. 21. OPENGL ES 2.0 • Fast -- can write interactive processing • Hard to ensure the shaders will perform well on all devices • Limited in size (max texture size...) • Needs adhoc shaders, i.e. fixed pipelines. • Expensive to retrieve processed image
  22. 22. RENDERSCRIPT
  23. 23. “RENDERSCRIPT IS A FRAMEWORK FOR RUNNING COMPUTATIONALLY INTENSIVE TASKS AT HIGH PERFORMANCE ON ANDROID. RENDERSCRIPT IS PRIMARILY ORIENTED FOR USE WITH DATA-PARALLEL COMPUTATION, ALTHOUGH SERIAL COMPUTATIONALLY INTENSIVE WORKLOADS CAN BENEFIT AS WELL.”
  24. 24. • Write “kernels” in a C99-like language with vector extensions and useful intrinsics • RenderScript executes them in parallel, on the GPU or CPU • Java used to manage lifetime of objects/allocations and control of execution • Portability
  25. 25. RENDERSCRIPT • Fast -- through LLVM optimizations and Parallelization • Supports CPU / GPU • Compatibility Library • Easy to offload to the background • Pretty easy to write
  26. 26. RENDERSCRIPT • Cannot allocate memory from kernels, need to do it from outside • RenderScript can be called from Java or from Native • Compatibility library!
  27. 27. HOW? • Optimized Math library • Optimizations on the device • Easier to read & maintain (vector math library helps)
  28. 28. ALLOCATIONS • Bound to Scripts • Can be bound to SurfaceTexture (producer & consumer) • Can share memory between Allocation and Bitmap
  29. 29. HOW TO USE IT
  30. 30. 1. SCRIPT #pragma version(1) #pragma rs java_package_name(com.example.rsdemo) uchar4 __attribute__((kernel)) color(uchar4 in) { return in; }
  31. 31. 2. CREATE CONTEXT RenderScript mRS = RenderScript.create(mContext); ScriptC_filter filter = new ScriptC_filter(mRS, mResources, R.raw.filter);
  32. 32. 3. LOAD BITMAP Bitmap bitmapIn = BitmapFactory.decodeResource( getResources(), R.drawable.monumentvalley); Bitmap bitmapOut = bitmapIn.copy(bitmapIn.getConfig(), true);
  33. 33. 4. CREATE ALLOCATIONS Allocation in = Allocation.createFromBitmap(mRS, bitmapIn, Allocation.MipmapControl.MIPMAP_NONE, Allocation.USAGE_SCRIPT | Allocation.USAGE_SHARED); Allocation out = Allocation.createTyped(mRS, in.getType());
  34. 34. 5. APPLY THE SCRIPT filter.forEach_color(in, out); out.copyTo(bitmapOut);
  35. 35. SCRIPT INTRINSICS ScriptIntrinsicBlur blur = ScriptIntrinsicBlur.create(mRS, Element.U8_4(mRS)); blur.setRadius(25.f); blur.setInput(in); blur.forEach(in);
  36. 36. READY TO USE • ScriptIntrinsic3DLUT • ScriptIntrinsicBlend • ScriptIntrinsicBlur • ScriptIntrinsicColorMatrix • ScriptIntrinsicConvolve3x3 • ScriptIntrinsicConvolve5x5 • ScriptIntrinsicLUT • ScriptIntrinsicYuvToRGB • ScriptGroup
  37. 37. PAINT IT BLACK (OR GRAY)
  38. 38. SCRIPT uchar4 __attribute__((kernel)) color(uchar4 in) { return in; }
  39. 39. SCRIPT uchar4 __attribute__((kernel)) grey(uchar4 in) { in.g = in.r; in.b = in.r; return in; }
  40. 40. SCRIPT uchar4 __attribute__((kernel)) grey(uchar4 in) { in.gb = in.r; return in; }
  41. 41. NDK void Java_com_example_rsdemo_ProcessImage_processBitmap(JNIEnv* env, jobject this, jobject bitmap, jint width, jint height) { unsigned char* rgb = 0; AndroidBitmap_lockPixels(env, bitmap, (void**) &rgb); int len = width * height * 4; int i; for (i = 0; i < len; i+=4) { int red = rgb[i]; rgb[i+1] = red; rgb[i+2] = red; } AndroidBitmap_unlockPixels(env, bitmap); }
  42. 42. LOCAL EFFECT
  43. 43. SCRIPT uchar4 __attribute__((kernel)) grey(uchar4 in) { in.g = in.r; in.b = in.r; return in; }
  44. 44. SCRIPT uchar4 __attribute__((kernel)) grey1(uchar4 in, uint32_t x, uint32_t y) { in.g = in.r; in.b = in.r; return in; }
  45. 45. SCRIPT uchar4 __attribute__((kernel)) grey1(uchar4 in, uint32_t x, uint32_t y) { float range = (float) x / width; uint32_t grey = (1 - range) * in.r; in.r = grey; in.g = grey; in.b = grey; return in; }
  46. 46. SCRIPT uchar4 __attribute__((kernel)) grey1(uchar4 in, uint32_t x, uint32_t y) { float range = (float) x / width; uint32_t grey = (1 - range) * in.r; in.r = grey; in.g = grey; in.b = grey; return in; }
  47. 47. SCRIPT uchar4 __attribute__((kernel)) grey2(uchar4 in, uint32_t x, uint32_t y) { float range = (float) x / width; uint32_t grey = (1 - range) * in.r; in.r = (in.r * range) + grey; in.g = (in.g * range) + grey; in.b = (in.b * range) + grey; return in; }
  48. 48. SCRIPT uchar4 __attribute__((kernel)) grey3(uchar4 in, uint32_t x, uint32_t y) { float range = (float) x / width; float4 pixel = rsUnpackColor8888(in); float grey = (1 - range) * pixel.r; pixel.r = pixel.r * range + grey; pixel.g = pixel.g * range + grey; pixel.b = pixel.b * range + grey; return rsPackColorTo8888( clamp(pixel, 0.f, 1.0f)); }
  49. 49. SCRIPT uchar4 __attribute__((kernel)) grey4(uchar4 in, uint32_t x, uint32_t y) { float range = (float) x / width; float4 pixel = rsUnpackColor8888(in); float grey = (1 - range) * pixel.r; pixel.rgb = pixel.rgb * range + grey; return rsPackColorTo8888( clamp(pixel, 0.f, 1.0f)); }
  50. 50. EXAMPLE: BLOOM • Select the bright pixels • Blur the result • Add the blurred bright pixels back to the image
  51. 51. WITH JAVA private void brightPass(int[] pixels, int width, int height) { int threshold = (int) (brightnessThreshold * 255); int r; int g; int b; int luminance; int[] luminanceData = new int[3 * 256]; // pre-computations for (int i = 0; i < luminanceData[i luminanceData[i luminanceData[i } for conversion from RGB to YCC luminanceData.length; i += 3) { ] = (int) (i * 0.2125f); + 1] = (int) (i * 0.7154f); + 2] = (int) (i * 0.0721f);
  52. 52. int index = 0; for (int y = 0; y < height; y++) { for (int x = 0; x < width; x++) { int pixel = pixels[index]; // unpack the pixel's components r = pixel >> 16 & 0xFF; g = pixel >> 8 & 0xFF; b = pixel & 0xFF; // compute the luminance luminance = luminanceData[r * 3] + luminanceData[g * 3 + 1] + luminanceData[b * 3 + 2]; // apply the treshold to select the brightest pixels luminance = Math.max(0, luminance - threshold); int sign = (int) Math.signum(luminance); // pack the components in a single pixel pixels[index] = 0xFF000000 | (r * sign) < < 16 | (g * sign) << 8 | (b * sign); index++; } } }
  53. 53. WITH GL SHADER uniform sampler2D baseImage; uniform float brightPassThreshold; void main(void) { vec3 luminanceVector = vec3(0.2125, 0.7154, 0.0721); vec4 sample = texture2D(baseImage, gl_TexCoord[0].st); float luminance = dot(luminanceVector, sample.rgb); luminance = max(0.0, luminance - brightPassThreshold); sample.rgb *= sign(luminance); sample.a = 1.0; gl_FragColor = sample; }
  54. 54. WITH RENDERSCRIPT float brightPassThreshold; uchar4 __attribute__((kernel)) brightPass(uchar4 in) { float3 luminanceVector = { 0.2125, 0.7154, 0.0721 }; float4 pixel = rsUnpackColor8888(in); float luminance = dot(luminanceVector, pixel.rgb); luminance = max(0.0f, luminance - brightPassThreshold); pixel.rgb *= sign(luminance); pixel.a = 1.0; return rsPackColorTo8888(clamp(pixel, 0.f, 1.0f)); }
  55. 55. JAVA-SIDE ScriptIntrinsicBlur blur = ScriptIntrinsicBlur.create(mRS, Element.U8_4(mRS)); ScriptIntrinsicBlend blend = ScriptIntrinsicBlend.create(mRS, Element.U8_4(mRS)); filter.set_brightPassThreshold(0.15f); filter.forEach_brightPass(in, out); blur.setRadius(25.f); blur.setInput(out); blur.forEach(out); blend.forEachAdd(in, out); out.copyTo(bitmapOut);
  56. 56. WORKING WELL EVERYWHERE
  57. 57. WORKING WELL ON ALL HARDWARE • Architect for the worst • Scale with the device capabilities • Screen size / dpi • Available memory • Available CPU / GPU • Think about what is a downgraded experience
  58. 58. LOADING • Load in the background • • AsyncTask, or use a background thread Bitmaps loading • query the size • inSampleSize • reuseBitmap • BitmapRegionDecoder
  59. 59. QUERY THE SIZE BitmapFactory.Options options = new BitmapFactory.Options(); options.inJustDecodeBounds = true; BitmapFactory.decodeResource( getResources(), R.id.myimage, options); int imageHeight = options.outHeight; int imageWidth = options.outWidth; String imageType = options.outMimeType;
  60. 60. INSAMPLESIZE • Only load what you need • needs to be a power of two, so for a 2048x2048 image, • insamplesize=2 => 1024x1024 image • insamplesize=4 => 512x512 image
  61. 61. CALCULATE INSAMPLESIZE if (bounds.width() > destination.width()) { int sampleSize = 1; int w = bounds.width(); while (w > destination.width()) { sampleSize *= 2; w /= sampleSize; } options.inSampleSize = sampleSize; }
  62. 62. REUSE BITMAP BitmapFactory.Options options; (...) Bitmap inBitmap = ... (...) options.inBitmap = inBitmap; API level 11 (Android 3.0) Before API level 19 (Android 4.4) only same size
  63. 63. BITMAP REGIONDECODER InputStream is = ... BitmapRegionDecoder decoder = BitmapRegionDecoder.newInstance(is, false); Rect imageBounds = ... Bitmap bitmap = decoder.decodeRegion(imageBounds, options); API level 11 (Android 3.0)
  64. 64. PIPELINE
  65. 65. PIPELINE • Run in a background service (used when saving too) • Mix C, RenderScript, java filtering (canvas draw) • multiple pipelines in parallel (direct preview, highres, icons, full res, geometry, saving)
  66. 66. FLEXIBLE PROCESSING • Unbounded pipeline • No fixed order • Complex filters • Geometry-based • Global • Local
  67. 67. FILTER TYPES Color Fx Geometry * Borders Color transforms Crop Contrast Image-based Straighten Saturation Parametric Rotate Local Mirror Vignette
  68. 68. Vintage Instant Washout X-Process COLOR FX - 3D LUT
  69. 69. CACHING • Cache RS scripts, allocations • Cache original bitmaps • Aggressively destroy/recycle resources in filters to keep memory low • If possible, filters should process the input bitmap directly • N-1 cache
  70. 70. MEMORY USAGE • Bitmap cache heavily reusing bitmaps • LruCache class (available in support lib too) • Pick image sizes depending on the device resolution / DPI • Have a path ready for a downgraded experience
  71. 71. DEVICE CAPABILITIES • Ask the system • Runtime.getRuntime().maxMemory() • New isLowRamDevice() API • Handles low-memory signals • Handles java.lang.OutOfMemory exceptions
  72. 72. PIPELINE CACHE Original Image
  73. 73. PIPELINE CACHE Original Image Filter Processed Image
  74. 74. REALISTICALLY... Original Image
  75. 75. REALISTICALLY... Original Image Filter Filter Filter Filter Processed Image
  76. 76. PROCESSING Original Image Filter Filter Filter Filter Processed Image
  77. 77. PROCESSING Original Image Filter Filter Filter Filter Processed Image
  78. 78. PROCESSING Original Image Filter Filter Filter Filter Processed Image
  79. 79. N-1 CACHING Original Image Filter Filter Filter Filter Processed Image
  80. 80. N-1 CACHING Original Image Filter Filter Filter Filter Processed Image
  81. 81. N-1 CACHING Original Image Filter Filter Filter Filter Processed Image
  82. 82. N-1 CACHING Original Image Filter Filter Filter Filter Processed Image
  83. 83. N-1 CACHING Original Image Filter Filter Filter Filter Processed Image
  84. 84. N-1 CACHING • No silver bullet • Only really useful when the user manipulates the last filters of the pipeline... • ...but this is after all the more common scenario!
  85. 85. REALTIME INTERACTION
  86. 86. REALTIME INTERACTION • Background processing • Minimize allocations -- Careful with Garbage Collection! • Optimized filters, RenderScript helps • Caching in the pipeline • Low/High resolution preview
  87. 87. PREVIEW SYSTEM Bitmap Bitmap Bitmap Preset Preset Preset UI Thread Processing Thread
  88. 88. PREVIEW SYSTEM Bitmap Bitmap Bitmap Preset Preset Preset UI Thread Processing Thread
  89. 89. PREVIEW Continuous preview High-res preview Full resolution
  90. 90. PREVIEW SYSTEM Low-res preview Request High-res preview Full-res preview If new request, delay more
  91. 91. HOW TO CHEAT • The triple-buffer preview can have a low resolution • The UI elements and controls are animated and manipulated at 60 fps on the UI thread • The rendering pipeline can (and needs to) be interrupted
  92. 92. INTERRUPTION IN RENDERSCRIPT LaunchOptions options = new LaunchOptions(); options.setX(xStart, xEnd); options.setY(yStart, yEnd); mScript.forEach_vignette(in, out, options);
  93. 93. LaunchOptions options = new LaunchOptions(); boolean even = true; int tile = 128; int height = bitmapIn.getHeight(); int width = bitmapIn.getWidth(); for (int yStart = 0; yStart < height; yStart += tile) { for (int xStart = 0; xStart < width; xStart += tile) { int xEnd = xStart + tile; int yEnd = yStart + tile; options.setX(xStart, xEnd); options.setY(yStart, yEnd); if (even) { filter.forEach_grey(in, out, options); } else { filter.forEach_color(in, out, options); } even = !even; } }
  94. 94. FULL RESOLUTION PREVIEW
  95. 95. FULL RESOLUTION PREVIEW • Ideally, we should use a tile-based rendering • At the moment, we use BitmapFactory region decoder instead • Filters need to be able to handle partial regions • future payoff: streaming save
  96. 96. FUTURE
  97. 97. FUTURE -- IN THE PHOTO EDITOR • Merging filters • • • RenderScript At the pipeline level (color cubes...) Streaming saving • Some code is there (AOSP), but not used
  98. 98. FUTURE -- IN ANDROID FRAMEWORK • TileView • Adding filtering pipeline in android framework • Image loader improvements • RAW support? Color correction?
  99. 99. 13223x5598 70MP image ~60 tiles, ~15Mb
  100. 100. 13223x5598 70MP image ~60 tiles, ~15Mb
  101. 101. TILEVIEW TileView TileGrid Tile TileViewAdapter ImageTileViewAdapter TileCache TestTileViewAdapter
  102. 102. TILEVIEW ADAPTER public interface TileViewAdapter { public int getTileSize(); public int getContentWidth(); public int getContentHeight(); public void onPaint(float scale, float dx, float dy, Bitmap bitmap); public Bitmap getFullImage(int max); void setDebug(boolean debug); }
  103. 103. QUESTIONS, SUGGESTIONS? • RenderScript documentation: • • http://developer.android.com/guide/topics/renderscript/ compute.html contact: nicolasroard@google.com

×