Cvpr2010 open source vision software, intro and training part iv generic image library - bourdev, henning - agenda - 2010

  • 269 views
Uploaded on

 

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Be the first to comment
    Be the first to like this
No Downloads

Views

Total Views
269
On Slideshare
0
From Embeds
0
Number of Embeds
0

Actions

Shares
Downloads
3
Comments
0
Likes
0

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Generic Image Library Lubomir Bourdev Adobe Systems and U.C. Berkeley Hailin Jin Adobe Systems. Christian Henning Independent consultant 12010 Adobe Systems Incorporated. All Rights Reserved.
  • 2. Agenda What is GIL? Basic concepts and navigation Generic code Image view transformations 1D iterators; pixel algorithms GIL extensions Is GIL for you? 22010 Adobe Systems Incorporated. All Rights Reserved.
  • 3. Agenda What is GIL? Basic concepts and navigation Generic code Image view transformations 1D iterators; pixel algorithms GIL extensions Is GIL for you? 32010 Adobe Systems Incorporated. All Rights Reserved.
  • 4. Image Represenations 4x3 image in which the second pixel is hilighted color space (RGB, CMYK…) In interleaved (chunky) form: channel order (RGB vs. BGR) row padding policy channel depth 8-bit, 16-bit… planar vs. interleaved In planar form: It is challenging to write code that is reusable and efficient 42010 Adobe Systems Incorporated. All Rights Reserved.
  • 5. Generic Image Library (GIL) GIL abstracts image representations and algorithms Allows you to write an algorithm once and have it work on any image format GIL is one of the boost C++ libraries (see www.boost.org) GIL is all header files – requires no linking GIL is open-source and free, including for commercial use 52010 Adobe Systems Incorporated. All Rights Reserved.
  • 6. Agenda What is GIL? Basic concepts and navigation Generic code Image view transformations 1D iterators; pixel algorithms GIL extensions Is GIL for you? 62010 Adobe Systems Incorporated. All Rights Reserved.
  • 7. Example – Computing the Gradient ImageSimplifying assumptions - Only horizontal gradient - Only 8-bit grayscale image - Central difference: ( - )/2 → I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D 72010 Adobe Systems Incorporated. All Rights Reserved.
  • 8. Example – Computing the Gradient ImageSimplifying assumptions - Only horizontal gradient - Only 8-bit grayscale image - Central difference - Ignore boundary cases ( - )/2 → I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D 82010 Adobe Systems Incorporated. All Rights Reserved.
  • 9. Interfacing with code outside GILvoid ComputeXGradientGray8(int width, int height, const unsigned char* src_pix, int src_rowbytes, signed char* dst_pix, int dst_rowbytes){ gray8c_view_t src = interleaved_view(width, height, (const gray8_pixel_t*)src_pix, src_rowbytes);} 92010 Adobe Systems Incorporated. All Rights Reserved.
  • 10. Interfacing with code outside GILvoid ComputeXGradientGray8(int width, int height, const unsigned char* src_pix, int src_rowbytes, signed char* dst_pix, int dst_rowbytes){ gray8c_view_t src = interleaved_view(width, height, (const gray8_pixel_t*)src_pix, src_rowbytes); GIL image view} - Provides uniform access to the pixels of an image - Lightweight (16 bytes: pointer + dimensions + rowbytes) 102010 Adobe Systems Incorporated. All Rights Reserved.
  • 11. Interfacing with code outside GILvoid ComputeXGradientGray8(int width, int height, const unsigned char* src_pix, int src_rowbytes, signed char* dst_pix, int dst_rowbytes){ gray8c_view_t src = interleaved_view(width, height, (const gray8_pixel_t*)src_pix, src_rowbytes); gray8s_view_t dst = interleaved_view(width, height, (gray8s_pixel_t*)dst_pix, dst_rowbytes); x_gradient(src,dst);} 112010 Adobe Systems Incorporated. All Rights Reserved.
  • 12. Interfacing with code outside GILvoid ComputeXGradientGray8(int width, int height, const unsigned char* src_pix, int src_rowbytes, signed char* dst_pix, int dst_rowbytes) grayscale{ gray8c_view_t src = interleaved_view(width, height, (const gray8_pixel_t*)src_pix, src_rowbytes); gray8s_view_t dst = interleaved_view(width, height, (gray8s_pixel_t*)dst_pix, dst_rowbytes); x_gradient(src,dst);} 122010 Adobe Systems Incorporated. All Rights Reserved.
  • 13. Interfacing with code outside GILvoid ComputeXGradientGray8(int width, int height, const unsigned char* src_pix, int src_rowbytes, signed char* dst_pix, int dst_rowbytes)8-bit (unsigned char){ gray8c_view_t src = interleaved_view(width, height, (const gray8_pixel_t*)src_pix, src_rowbytes); gray8s_view_t dst = interleaved_view(width, height, (gray8s_pixel_t*)dst_pix, dst_rowbytes); x_gradient(src,dst);} 132010 Adobe Systems Incorporated. All Rights Reserved.
  • 14. Interfacing with code outside GILvoid ComputeXGradientGray8(int width, int height, const unsigned char* src_pix, int src_rowbytes, signed char* dst_pix, int dst_rowbytes) 8-bit (signed char){ gray8c_view_t src = interleaved_view(width, height, (const gray8_pixel_t*)src_pix, src_rowbytes); gray8s_view_t dst = interleaved_view(width, height, (gray8s_pixel_t*)dst_pix, dst_rowbytes); x_gradient(src,dst);} 142010 Adobe Systems Incorporated. All Rights Reserved.
  • 15. Interfacing with code outside GILvoid ComputeXGradientGray8(int width, int height, const unsigned char* src_pix, int src_rowbytes, signed char* dst_pix, int dst_rowbytes) const (read-only){ gray8c_view_t src = interleaved_view(width, height, (const gray8_pixel_t*)src_pix, src_rowbytes); gray8s_view_t dst = interleaved_view(width, height, (gray8s_pixel_t*)dst_pix, dst_rowbytes); x_gradient(src,dst);} 152010 Adobe Systems Incorporated. All Rights Reserved.
  • 16. Interfacing with code outside GILvoid ComputeXGradientGray8(int width, int height, const unsigned char* src_pix, int src_rowbytes, signed char* dst_pix, int dst_rowbytes) image view{ gray8c_view_t src = interleaved_view(width, height, (const gray8_pixel_t*)src_pix, src_rowbytes); gray8s_view_t dst = interleaved_view(width, height, (gray8s_pixel_t*)dst_pix, dst_rowbytes); x_gradient(src,dst);} 162010 Adobe Systems Incorporated. All Rights Reserved.
  • 17. GIL Concrete Types rgb32fc_planar_step_view_t rgb 8 _ _ _ _ pixel bgr 16 f c planar step ref gray 32 s ptr lab 64 loc cmyk … view rgba image … … 172010 Adobe Systems Incorporated. All Rights Reserved.
  • 18. GIL Concrete Types rgb32fc_planar_step_view_t rgb 8 _ _ _ _ pixel bgr 16 f c planar step ref gray 32 s ptr lab 64 loc cmyk … view rgba image … … 182010 Adobe Systems Incorporated. All Rights Reserved.
  • 19. GIL Concrete Types rgb32fc_planar_step_view_t rgb 8 _ _ _ _ pixel bgr 16 f c planar step ref gray 32 s ptr lab 64 loc cmyk … view rgba image … … 192010 Adobe Systems Incorporated. All Rights Reserved.
  • 20. GIL Concrete Types rgb32fc_planar_step_view_t rgb 8 _ _ _ _ pixel bgr 16 f c planar step ref gray 32 s ptr lab 64 loc cmyk … view rgba image … … 202010 Adobe Systems Incorporated. All Rights Reserved.
  • 21. GIL Concrete Types rgb32fc_planar_step_view_t rgb 8 _ _ _ _ pixel bgr 16 f c planar step ref gray 32 s ptr lab 64 loc cmyk … view rgba image … … 212010 Adobe Systems Incorporated. All Rights Reserved.
  • 22. GIL Concrete Types rgb32fc_planar_step_view_t rgb 8 _ _ _ _ pixel bgr 16 f c planar step ref gray 32 s ptr lab 64 loc cmyk … view rgba image … … 222010 Adobe Systems Incorporated. All Rights Reserved.
  • 23. GIL Concrete Types rgb32fc_planar_step_view_t rgb 8 _ _ _ _ pixel bgr 16 f c planar step ref gray 32 s ptr lab 64 loc cmyk … view rgba image … … 232010 Adobe Systems Incorporated. All Rights Reserved.
  • 24. Simplest Implementation#include <boost/gil/gil_all.hpp>using namespace boost::gil;void x_gradient(const gray8c_view_t& src, const gray8s_view_t& dst){ for (int y=0; y<src.height(); ++y) for (int x=1; x<src.width()-1; ++x) dst(x,y) = (src(x+1,y) - src(x-1,y)) / 2;} 242010 Adobe Systems Incorporated. All Rights Reserved.
  • 25. The simplest version is too slow Better to remember the beginning of each row and offset from it I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D 252010 Adobe Systems Incorporated. All Rights Reserved.
  • 26. Faster Versionvoid x_gradient(const gray8c_view_t& src, const gray8s_view_t& dst){ for (int y=0; y<src.height(); ++y) { gray8c_view_t::x_iterator src_it = src.row_begin(y); gray8s_view_t::x_iterator dst_it = dst.row_begin(y); for (int x=1; x<src.width()-1; ++x) dst_it[x] = (src_it[x+1] - src_it[x-1]) / 2; }} 262010 Adobe Systems Incorporated. All Rights Reserved.
  • 27. Faster Versionvoid x_gradient(const gray8c_view_t& src, const gray8s_view_t& dst){ pixel iterator for (int y=0; y<src.height(); ++y) { gray8c_view_t::x_iterator src_it = src.row_begin(y); gray8s_view_t::x_iterator dst_it = dst.row_begin(y); for (int x=1; x<src.width()-1; ++x) dst_it[x] = (src_it[x+1] - src_it[x-1]) / 2; }} 272010 Adobe Systems Incorporated. All Rights Reserved.
  • 28. GIL constructs are very lightweightvoid x_gradient(const gray8c_view_t& src, const gray8s_view_t& dst){ a raw C pointer for (int y=0; y<src.height(); ++y) { gray8c_view_t::x_iterator src_it = src.row_begin(y); gray8s_view_t::x_iterator dst_it = dst.row_begin(y); for (int x=1; x<src.width()-1; ++x) dst_it[x] = (src_it[x+1] - src_it[x-1]) / 2; }} raw C pointer indexing operator 282010 Adobe Systems Incorporated. All Rights Reserved.
  • 29. Compute gradient in the vertical dimension I I I I I I I I I I D D D D D I I I I I D D D D D I I I I I D D D D D I I I I I 292010 Adobe Systems Incorporated. All Rights Reserved.
  • 30. Computing the X-Gradientvoid x_gradient(const gray8c_view_t& src, const gray8s_view_t& dst){ for (int y=0; y<src.height(); ++y) { gray8c_view_t::x_iterator src_it = src.row_begin(y); gray8s_view_t::x_iterator dst_it = dst.row_begin(y); for (int x=1; x<src.width()-1; ++x) dst_it[x] = (src_it[x+1] - src_it[x-1]) / 2; }} 302010 Adobe Systems Incorporated. All Rights Reserved.
  • 31. Computing the Y-Gradientvoid y_gradient(const gray8c_view_t& src, const gray8s_view_t& dst){ for (int x=0; x<src.width(); ++x) { gray8c_view_t::y_iterator src_it = src.col_begin(x); gray8s_view_t::y_iterator dst_it = dst.col_begin(x); for (int y=1; y<src.height()-1; ++y) dst_it[y] = (src_it[y+1] - src_it[y-1]) / 2; }} 312010 Adobe Systems Incorporated. All Rights Reserved.
  • 32. Computing the Y-Gradientvoid y_gradient(const gray8c_view_t& src, const gray8s_view_t& dst){ pixel step iterator (8 bytes) for (int x=0; x<src.width(); ++x) { gray8c_view_t::y_iterator src_it = src.col_begin(x); gray8s_view_t::y_iterator dst_it = dst.col_begin(x); for (int y=1; y<src.height()-1; ++y) dst_it[y] = (src_it[y+1] - src_it[y-1]) / 2; }} multiplies index by the step 322010 Adobe Systems Incorporated. All Rights Reserved.
  • 33. Using Locators Pixel locators are the 2D equivalents of pixel iterators Lightweight and fast: I I I I I I I I I I D D D D D I I I I I D D D D D I I I I I D D D D D I I I I I gray8c_view_t::xy_locator loc = src.xy_at(0,1); loc.x()++; (“below” – “above”) / 2 loc.y()+=5; (*dst) = (loc(0,1) - loc(0,-1)) / 2; 332010 Adobe Systems Incorporated. All Rights Reserved.
  • 34. Agenda What is GIL? Basic concepts and navigation Generic code Image view transformations 1D iterators; pixel algorithms GIL extensions Is GIL for you? 342010 Adobe Systems Incorporated. All Rights Reserved.
  • 35. Making the Code Generictemplate <typename SrcView, typename DstView>void x_gradient(const SrcView& src, const DstView& dst) { GIL constructs and algorithms operate on Concepts You can use your own images, image views, locators, iterators, pixels channels, color spaces, etc. 352006 Adobe Systems Incorporated. All Rights Reserved.
  • 36. Making the Code Generictemplate <typename SrcView, typename DstView>void x_gradient(const SrcView& src, const DstView& dst) { gil_function_requires<ImageViewConcept<SrcView> >(); Using boost::concept_check to enforce concept syntactic requirements Zero run-time overhead (gets compiled out) Sometimes significant compile-time overhead 362006 Adobe Systems Incorporated. All Rights Reserved.
  • 37. Making the Code Generictemplate <typename SrcView, typename DstView>void x_gradient(const SrcView& src, const DstView& dst) { gil_function_requires<ImageViewConcept<SrcView> >(); SrcView must be satisfy all requirements of an image view 372006 Adobe Systems Incorporated. All Rights Reserved.
  • 38. Making the Code Generictemplate <typename SrcView, typename DstView>void x_gradient(const SrcView& src, const DstView& dst) { gil_function_requires<ImageViewConcept<SrcView> >(); gil_function_requires<MutableImageViewConcept<DstView> >(); DstView must be an image view that is mutable (i.e. writable) 382006 Adobe Systems Incorporated. All Rights Reserved.
  • 39. Making the Code Generictemplate <typename SrcView, typename DstView>void x_gradient(const SrcView& src, const DstView& dst) { gil_function_requires<ImageViewConcept<SrcView> >(); gil_function_requires<MutableImageViewConcept<DstView> >(); gil_function_requires<ColorSpacesCompatibleConcept< typename SrcView::color_space_t, typename DstView::color_space_t> >(); Both views must have the same number and interpretation of channels. 392006 Adobe Systems Incorporated. All Rights Reserved.
  • 40. The Full Versiontemplate <typename SrcView, typename DstView>void x_gradient(const SrcView& src, const DstView& dst){ int max_x = src.width()-1; for (int y=0; y<src.height(); ++y) { typename SrcView::x_iterator src_it = src.row_begin(y); typename DstView::x_iterator dst_it = dst.row_begin(y); for (int x=1; x<max_x; ++x) for (int c=0; c<src.num_channels(); ++c) dst_it[x][c] = (src_it[x+1][c]-src_it[x-1][c])/2; }} explicit loop over the channels 40 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 41. And if you are a generic programming enthusiast…template <typename SrcView, typename DstView>void x_gradient(const SrcView& src, const DstView& dst){ int max_x = src.width()-1; for (int y=0; y<src.height(); ++y) { typename SrcView::x_iterator src_it = src.row_begin(y); typename DstView::x_iterator dst_it = dst.row_begin(y); for (int x=1; x<max_x; ++x) static_transform(src_it[x+1], src_it[x-1], dst_it[x], (_1 - _2)/constant(2)); }} 41 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 42. And if you are a generic programming enthusiast…template <typename SrcView, typename DstView>void x_gradient(const SrcView& src, const DstView& dst){ int max_x = src.width()-1; for (int y=0; y<src.height(); ++y) { typename SrcView::x_iterator src_it = src.row_begin(y); typename DstView::x_iterator dst_it = dst.row_begin(y); for (int x=1; x<max_x; ++x) static_transform(src_it[x+1], src_it[x-1], dst_it[x], (_1 - _2)/constant(2)); } GIL channel-level algorithm} 42 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 43. And if you are a generic programming enthusiast…template <typename SrcView, typename DstView>void x_gradient(const SrcView& src, const DstView& dst){ int max_x = src.width()-1; for (int y=0; y<src.height(); ++y) { typename SrcView::x_iterator src_it = src.row_begin(y); typename DstView::x_iterator dst_it = dst.row_begin(y); for (int x=1; x<max_x; ++x) static_transform(src_it[x+1], src_it[x-1], dst_it[x], (_1 - _2)/constant(2)); }} boost::lambda expression 43 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 44. GIL Channel Level Algorithms STL GILstd::for_each gil::static_for_eachstd::transform gil::static_transformstd::fill gil::static_fillstd::max_element gil::static_maxstd::min_element gil::static_min Channel algorithms are compile-time recursive (no explicit loop) Channels are paired semantically (consider RGB and BGR) 442006 Adobe Systems Incorporated. All Rights Reserved.
  • 45. GIL Type Metafunctions Generators image_type<Channel, Layout, IsPlanar>::type view_type<Channel, Layout, IsPlanar, …>::type derived_image_type<Img, Channel, Layout, …>::type derived_view_type<View, Channel, Layout, …>::type 452010 Adobe Systems Incorporated. All Rights Reserved.
  • 46. GIL Type Metafunctions Generators image_type<Channel, Layout, IsPlanar>::type view_type<Channel, Layout, IsPlanar, …>::type derived_image_type<Img, Channel, Layout, …>::type derived_view_type<View, use_default, …>::type Type analysis metafunctions: [image/view/locator/iterator] + _is_ + [basic/mutable/planar/step] if (view_is_planar<View>::value) { … } 462010 Adobe Systems Incorporated. All Rights Reserved.
  • 47. But how generic is it? 472010 Adobe Systems Incorporated. All Rights Reserved.
  • 48. Interfacing with Your Codevoid XGradientRGB8_BGR16(int w, int h, const unsigned char* src_pixels, int src_rowbytes, signed short* dst_pixels, int dst_rowbytes){ rgb8c_view_t src = interleaved_view(w,h,(const rgb8_pixel_t*)src_pix,src_rowbytes); rgb16s_view_t dst = interleaved_view(w,h,( rgb16s_pixel_t*)dst_pix,dst_rowbytes); x_gradient(src,dst);}void XGradientPlanarLAB8_LAB32(int w, int h, const uint16* src_l, const uint16* src_a, const uint16* src_b, int src_rowbytes, signed int* dst_pixels, int dst_rowbytes){ lab16c_planar_view_t src=planar_lab_view (w,h, src_l,src_a,src_b, src_rowbytes); lab32s_view_t dst=interleaved_view(w,h,(lab32s_pixel_t*)dst_pixels,dst_rowbytes); x_gradient(src,dst);} 48 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 49. Interfacing with Your Codevoid XGradientRGB8_BGR16(int w, int h, const unsigned char* src_pixels, int src_rowbytes, signed short* dst_pixels, int dst_rowbytes){ rgb8c_view_t src = interleaved_view(w,h,(const rgb8_pixel_t*)src_pix,src_rowbytes); rgb16s_view_t dst = interleaved_view(w,h,( rgb16s_pixel_t*)dst_pix,dst_rowbytes); x_gradient(src,dst); any color space}void XGradientPlanarLAB8_LAB32(int w, int h, const uint16* src_l, const uint16* src_a, const uint16* src_b, int src_rowbytes, signed int* dst_pixels, int dst_rowbytes){ lab16c_planar_view_t src=planar_lab_view (w,h, src_l,src_a,src_b, src_rowbytes); lab32s_view_t dst=interleaved_view(w,h,(lab32s_pixel_t*)dst_pixels,dst_rowbytes); x_gradient(src,dst);} 49 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 50. Interfacing with Your Codevoid XGradientRGB8_BGR16(int w, int h, const unsigned char* src_pixels, int src_rowbytes, signed short* dst_pixels, int dst_rowbytes){ rgb8c_view_t src = interleaved_view(w,h,(const rgb8_pixel_t*)src_pix,src_rowbytes); rgb16s_view_t dst = interleaved_view(w,h,( rgb16s_pixel_t*)dst_pix,dst_rowbytes); x_gradient(src,dst);} any channel depthvoid XGradientPlanarLAB8_LAB32(int w, int h, const uint16* src_l, const uint16* src_a, const uint16* src_b, int src_rowbytes, signed int* dst_pixels, int dst_rowbytes){ lab16c_planar_view_t src=planar_lab_view (w,h, src_l,src_a,src_b, src_rowbytes); lab32s_view_t dst=interleaved_view(w,h,(lab32s_pixel_t*)dst_pixels,dst_rowbytes); x_gradient(src,dst);} 50 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 51. Interfacing with Your Codevoid XGradientRGB8_BGR16(int w, int h, const unsigned char* src_pixels, int src_rowbytes, signed short* dst_pixels, int dst_rowbytes) any channel ordering{ rgb8c_view_t src = interleaved_view(w,h,(const rgb8_pixel_t*)src_pix,src_rowbytes); rgb16s_view_t dst = interleaved_view(w,h,( rgb16s_pixel_t*)dst_pix,dst_rowbytes); x_gradient(src,dst);}void XGradientPlanarLAB8_LAB32(int w, int h, const uint16* src_l, const uint16* src_a, const uint16* src_b, int src_rowbytes, signed int* dst_pixels, int dst_rowbytes){ lab16c_planar_view_t src=planar_lab_view (w,h, src_l,src_a,src_b, src_rowbytes); lab32s_view_t dst=interleaved_view(w,h,(lab32s_pixel_t*)dst_pixels,dst_rowbytes); x_gradient(src,dst);} 51 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 52. Interfacing with Your Codevoid XGradientRGB8_BGR16(int w, int h, const unsigned char* src_pixels, int src_rowbytes, signed short* dst_pixels, int dst_rowbytes){ rgb8c_view_t src = interleaved_view(w,h,(const rgb8_pixel_t*)src_pix,src_rowbytes); rgb16s_view_t dst = interleaved_view(w,h,( rgb16s_pixel_t*)dst_pix,dst_rowbytes); x_gradient(src,dst);}void XGradientPlanarLAB8_LAB32(int w, int h, const uint16* src_l, const uint16* src_a, const uint16* src_b, int src_rowbytes, signed int* dst_pixels, int dst_rowbytes){ lab16c_planar_view_t src=planar_lab_view (w,h, src_l,src_a,src_b, src_rowbytes); lab32s_view_t dst=interleaved_view(w,h,(lab32s_pixel_t*)dst_pixels,dst_rowbytes); x_gradient(src,dst);} any pixel organization 52 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 53. Reading between the bytes Channels may not be byte aligned R R G G G B B B R R G G G B B B R R G G G B B B Byte 1 Byte 2 Byte 3typedef packed_image3_type<uint8_t,2,3,3,rgb_layout_t>::type img_t; Pixels may not be byte aligned: Pixel 1 Pixel 2 Pixel 3 R R R G G G B B B R R R G G G B B B R R R G G G Byte 1 Byte 2 Byte 3 typedef bit_aligned_image3_type<3,3,3,rgb_layout_t>::type img_t;  You may need to define operator- and operator/ for new channels 532010 Adobe Systems Incorporated. All Rights Reserved.
  • 54. Synthetic Image Views Define a view of an arbitrary function Apply an arbitrary view transformation on it Run an arbitrary algorithm on it Details in the GIL Tutorial 542010 Adobe Systems Incorporated. All Rights Reserved.
  • 55. Extending GIL Every component of GIL can be replaced  Rachel can make a channel with a range -1..5  Monica can construct his own XYZab color space  Chandler can create a Pixel iterator that has variable step  Matt can make view of an image that is inside a jpeg file … and you can use your x_gradient on Matt’s view with Monica’s color space and Rachel’s channel … and no developers need to coordinate with each other 552010 Adobe Systems Incorporated. All Rights Reserved.
  • 56. GIL is generic. But how fast is it? Test a complex case (both src and dst are planar, say 8-bit rgb) Use a very generic version (with function objects and boost::lambda) 56 2010 Adobe Systems Incorporated. All Rights Reserved.
  • 57. Contestant 1: The “generic enthusiasts” versionfor (int x=1; x<max_x; ++x) static_transform(src_it[x+1], src_it[x-1], dst_it[x], (_1 - _2)/constant(2)); 572010 Adobe Systems Incorporated. All Rights Reserved.
  • 58. Contestant 1: The “generic enthusiasts” versionfor (int x=1; x<max_x; ++x) static_transform(src_it[x+1], src_it[x-1], dst_it[x], (_1 - _2)/constant(2)); Planar pointer: struct { char *r, *g, *b; }; 582010 Adobe Systems Incorporated. All Rights Reserved.
  • 59. Contestant 1: The “generic enthusiasts” versionfor (int x=1; x<max_x; ++x) static_transform(src_it[x+1], src_it[x-1], dst_it[x], (_1 - _2)/constant(2)); operator[] returns by value a reference proxy: struct { char &r, &g, &b; }; 592010 Adobe Systems Incorporated. All Rights Reserved.
  • 60. Contestant 1: The “generic enthusiasts” versionfor (int x=1; x<max_x; ++x) static_transform(src_it[x+1], src_it[x-1], dst_it[x], (_1 - _2)/constant(2)); - Uses compile-time recursion - Uses specialization to pair channels semantically 602010 Adobe Systems Incorporated. All Rights Reserved.
  • 61. Contestant 1: The “generic enthusiasts” versionfor (int x=1; x<max_x; ++x) static_transform(src_it[x+1], src_it[x-1], dst_it[x], (_1 - _2)/constant(2)); boost::lambda anonymous function object 612010 Adobe Systems Incorporated. All Rights Reserved.
  • 62. Contestant 2: Good Ol’ C versionchar *sr, *sg, *sb;unsigned char *dr, *dg, *db;int max_x;for (int x=1; x<max_x; ++x) { dr[x] = (sr[x+1] - sr[x-1]) / 2; dg[x] = (sg[x+1] - sg[x-1]) / 2; db[x] = (sb[x+1] - sb[x-1]) / 2;} 622010 Adobe Systems Incorporated. All Rights Reserved.
  • 63. Assembly code of the inner loop (MSVC 8) Good Ol’ C00410C12 movzx edx,byte ptr [ecx]00410C15 movzx eax,byte ptr [ecx-2]00410C19 sub eax,edx00410C1B cdq00410C1C sub eax,edx00410C1E mov edx,dword ptr [esp+14h]00410C22 sar eax,100410C24 mov byte ptr [edx+esi],al00410C27 movzx edx,byte ptr [ebx+ecx]00410C2B mov eax,dword ptr [esp+18h]00410C2F movzx eax,byte ptr [eax+edi]00410C33 sub eax,edx00410C35 cdq00410C36 sub eax,edx00410C38 sar eax,100410C3A mov byte ptr [esi],al00410C3C movzx edx,byte ptr [ecx+ebp]00410C40 movzx eax,byte ptr [edi]00410C43 sub eax,edx00410C45 cdq00410C46 sub eax,edx00410C48 mov edx,dword ptr [esp+1Ch]00410C4C sar eax,100410C4E mov byte ptr [edx+esi],al00410C51 add ecx,100410C54 add esi,100410C57 add edi,100410C5A sub dword ptr [esp+38h],100410C5F jne gil::x_gradient_no_gil+72h (410C12h) 632006 Adobe Systems Incorporated. All Rights Reserved.
  • 64. Assembly code of the inner loop (MSVC 8) Good Ol’ C Generic Enthusiasts 00414FD0 mov edx,dword ptr [esp+14h]00410C12 movzx edx,byte ptr [ecx] 00414FD4 movzx edx,byte ptr [edx+ecx]00410C15 movzx eax,byte ptr [ecx-2] 00414FD8 mov eax,dword ptr [esp+18h]00410C19 sub eax,edx 00414FDC movzx eax,byte ptr [eax+edi]00410C1B cdq 00414FE0 sub eax,edx00410C1C sub eax,edx 00414FE2 cdq00410C1E mov edx,dword ptr [esp+14h] 00414FE3 sub eax,edx00410C22 sar eax,1 00414FE5 mov edx,dword ptr [esp+1Ch]00410C24 mov byte ptr [edx+esi],al 00414FE9 sar eax,100410C27 movzx edx,byte ptr [ebx+ecx] 00414FEB mov byte ptr [edx+esi],al00410C2B mov eax,dword ptr [esp+18h] 00414FEE movzx edx,byte ptr [ecx]00410C2F movzx eax,byte ptr [eax+edi] 00414FF1 movzx eax,byte ptr [ecx-2]00410C33 sub eax,edx 00414FF5 sub eax,edx00410C35 cdq 00414FF7 cdq00410C36 sub eax,edx 00414FF8 sub eax,edx00410C38 sar eax,1 00414FFA sar eax,100410C3A mov byte ptr [esi],al 00414FFC mov byte ptr [esi],al00410C3C movzx edx,byte ptr [ecx+ebp] 00414FFE movzx edx,byte ptr [ecx+ebp]00410C40 movzx eax,byte ptr [edi] 00415002 movzx eax,byte ptr [edi]00410C43 sub eax,edx 00415005 sub eax,edx00410C45 cdq 00415007 cdq00410C46 sub eax,edx 00415008 sub eax,edx00410C48 mov edx,dword ptr [esp+1Ch] 0041500A mov edx,dword ptr [esp+20h]00410C4C sar eax,1 0041500E sar eax,100410C4E mov byte ptr [edx+esi],al 00415010 mov byte ptr [edx+esi],al00410C51 add ecx,1 00415013 add esi,100410C54 add esi,1 00415016 add ecx,100410C57 add edi,1 00415019 add edi,100410C5A sub dword ptr [esp+38h],1 0041501C sub ebx,100410C5F jne gil::x_gradient_no_gil+72h (410C12h) 0041501F jne gil::x_gradient1...+0B0h (414FD0h) 642006 Adobe Systems Incorporated. All Rights Reserved.
  • 65. Assembly code of the inner loop (MSVC 8) Good Ol’ C Generic Enthusiasts 00414FD0 mov edx,dword ptr [esp+14h]00410C12 movzx edx,byte ptr [ecx] 00414FD4 movzx edx,byte ptr [edx+ecx]00410C15 movzx eax,byte ptr [ecx-2] 00414FD8 mov eax,dword ptr [esp+18h]00410C19 sub eax,edx 00414FDC movzx eax,byte ptr [eax+edi]00410C1B cdq 00414FE0 sub eax,edx00410C1C sub eax,edx 00414FE2 cdq00410C1E mov edx,dword ptr [esp+14h] 00414FE3 sub eax,edx00410C22 sar eax,1 00414FE5 mov edx,dword ptr [esp+1Ch]00410C24 mov byte ptr [edx+esi],al 00414FE9 sar eax,100410C27 movzx edx,byte ptr [ebx+ecx] 00414FEB mov byte ptr [edx+esi],al00410C2B mov eax,dword ptr [esp+18h] 00414FEE movzx edx,byte ptr [ecx]00410C2F movzx eax,byte ptr [eax+edi] 00414FF1 movzx eax,byte ptr [ecx-2]00410C33 sub eax,edx 00414FF5 sub eax,edx00410C35 cdq 00414FF7 cdq00410C36 sub eax,edx 00414FF8 sub eax,edx00410C38 sar eax,1 00414FFA sar eax,100410C3A mov byte ptr [esi],al 00414FFC mov byte ptr [esi],al00410C3C movzx edx,byte ptr [ecx+ebp] 00414FFE movzx edx,byte ptr [ecx+ebp]00410C40 movzx eax,byte ptr [edi] 00415002 movzx eax,byte ptr [edi]00410C43 sub eax,edx 00415005 sub eax,edx00410C45 cdq 00415007 cdq00410C46 sub eax,edx 00415008 sub eax,edx00410C48 mov edx,dword ptr [esp+1Ch] 0041500A mov edx,dword ptr [esp+20h]00410C4C sar eax,1 0041500E sar eax,100410C4E mov byte ptr [edx+esi],al 00415010 mov byte ptr [edx+esi],al00410C51 add ecx,1 00415013 add esi,100410C54 add esi,1 00415016 add ecx,100410C57 add edi,1 00415019 add edi,100410C5A sub dword ptr [esp+38h],1 0041501C sub ebx,100410C5F jne gil::x_gradient_no_gil+72h (410C12h) 0041501F jne gil::x_gradient1...+0B0h (414FD0h) Some code shifted around 652006 Adobe Systems Incorporated. All Rights Reserved.
  • 66. Assembly code of the inner loop (MSVC 8) Good Ol’ C Generic Enthusiasts 00414FD0 mov edx,dword ptr [esp+14h]00410C12 movzx edx,byte ptr [ecx] 00414FD4 movzx edx,byte ptr [edx+ecx]00410C15 movzx eax,byte ptr [ecx-2] 00414FD8 mov eax,dword ptr [esp+18h]00410C19 sub eax,edx 00414FDC movzx eax,byte ptr [eax+edi]00410C1B cdq 00414FE0 sub eax,edx00410C1C sub eax,edx 00414FE2 cdq00410C1E mov edx,dword ptr [esp+14h] 00414FE3 sub eax,edx00410C22 sar eax,1 00414FE5 mov edx,dword ptr [esp+1Ch]00410C24 mov byte ptr [edx+esi],al 00414FE9 sar eax,100410C27 movzx edx,byte ptr [ebx+ecx] 00414FEB mov byte ptr [edx+esi],al00410C2B mov eax,dword ptr [esp+18h] 00414FEE movzx edx,byte ptr [ecx]00410C2F movzx eax,byte ptr [eax+edi] 00414FF1 movzx eax,byte ptr [ecx-2]00410C33 sub eax,edx 00414FF5 sub eax,edx00410C35 cdq 00414FF7 cdq00410C36 sub eax,edx 00414FF8 sub eax,edx one extra00410C38 sar eax,1 00414FFA sar eax,100410C3A mov byte ptr [esi],al 00414FFC mov byte ptr [esi],al mov00410C3C movzx edx,byte ptr [ecx+ebp] 00414FFE movzx edx,byte ptr [ecx+ebp]00410C40 movzx eax,byte ptr [edi] 00415002 movzx eax,byte ptr [edi]00410C43 sub eax,edx 00415005 sub eax,edx00410C45 cdq 00415007 cdq00410C46 sub eax,edx 00415008 sub eax,edx00410C48 mov edx,dword ptr [esp+1Ch] 0041500A mov edx,dword ptr [esp+20h]00410C4C sar eax,1 0041500E sar eax,100410C4E mov byte ptr [edx+esi],al 00415010 mov byte ptr [edx+esi],al00410C51 add ecx,1 00415013 add esi,100410C54 add esi,1 00415016 add ecx,100410C57 add edi,1 00415019 add edi,100410C5A sub dword ptr [esp+38h],1 0041501C sub ebx,100410C5F jne gil::x_gradient_no_gil+72h (410C12h) 0041501F jne gil::x_gradient1...+0B0h (414FD0h) Some very minor changes 662006 Adobe Systems Incorporated. All Rights Reserved.
  • 67. Assembly code of the inner loop (MSVC 8) Good Ol’ C Generic Enthusiasts 00414FD0 mov edx,dword ptr [esp+14h]00410C12 movzx edx,byte ptr [ecx] 00414FD4 movzx edx,byte ptr [edx+ecx]00410C15 movzx eax,byte ptr [ecx-2] 00414FD8 mov eax,dword ptr [esp+18h]00410C19 sub eax,edx 00414FDC movzx eax,byte ptr [eax+edi]00410C1B cdq 00414FE0 sub eax,edx00410C1C sub eax,edx 00414FE2 cdq00410C1E mov edx,dword ptr [esp+14h] 00414FE3 sub eax,edx00410C22 sar eax,1 00414FE5 mov edx,dword ptr [esp+1Ch]00410C24 mov byte ptr [edx+esi],al 00414FE9 sar eax,100410C27 movzx edx,byte ptr [ebx+ecx] 00414FEB mov byte ptr [edx+esi],al00410C2B mov eax,dword ptr [esp+18h] 00414FEE movzx edx,byte ptr [ecx]00410C2F movzx eax,byte ptr [eax+edi] 00414FF1 movzx eax,byte ptr [ecx-2]00410C33 sub eax,edx 00414FF5 sub eax,edx00410C35 cdq 00414FF7 cdq00410C36 sub eax,edx 00414FF8 sub eax,edx one extra00410C38 sar eax,1 00414FFA sar eax,100410C3A mov byte ptr [esi],al 00414FFC mov byte ptr [esi],al mov00410C3C movzx edx,byte ptr [ecx+ebp] 00414FFE movzx edx,byte ptr [ecx+ebp]00410C40 movzx eax,byte ptr [edi] 00415002 movzx eax,byte ptr [edi]00410C43 sub eax,edx 00415005 sub eax,edx00410C45 cdq 00415007 cdq00410C46 sub eax,edx indexed 00415008 sub eax,edx00410C48 mov edx,dword ptr [esp+1Ch] 0041500A mov edx,dword ptr [esp+20h]00410C4C sar eax,1 mem. 0041500E sar eax,100410C4E mov byte ptr [edx+esi],al 00415010 mov byte ptr [edx+esi],al00410C51 add ecx,1 addressing 00415013 add esi,100410C54 add esi,1 00415016 add ecx,100410C57 add edi,1 00415019 add edi,100410C5A sub dword ptr [esp+38h],1 0041501C sub ebx,100410C5F jne gil::x_gradient_no_gil+72h (410C12h) 0041501F jne gil::x_gradient1...+0B0h (414FD0h) Some very minor changes No measurable performance difference 672006 Adobe Systems Incorporated. All Rights Reserved.
  • 68. GIL is generic GIL is fast But how readable is it? 682010 Adobe Systems Incorporated. All Rights Reserved.
  • 69. But how readable is it? Generic:for (int x=1; x<max_x; ++x) static_transform(src_it[x-1], src_it[x+1], dst_it[x], (_1 - _2)/constant(2)); Non-generic:for (int x=1; x<max_x; ++x) { dr[x] = (sr[x+1] - sr[x-1]) / 2; dg[x] = (sg[x+1] - sg[x-1]) / 2; db[x] = (sb[x+1] - sb[x-1]) / 2;} 692010 Adobe Systems Incorporated. All Rights Reserved.
  • 70. But how readable is it? Generic: same assembly codefor (int x=1; x<max_x; ++x) (MSVC8 unrolls the loop) for (int c=0; c<SrcView::num_channels; ++c) dst_it[x][c] = (src_it[x+1][c]-src_it[x-1][c])/2; Non-generic: for (int x=1; x<max_x; ++x) {for (int x=1; x<max_x; ++x) { for (int x=1; x<max_x; ++x) = (s1[x-4] – s1[x+4]) / 2; d[x] d1[x] { = (s1[x-1] d2[x] = (s2[x-3] – s2[x+5]) / 2; – s1[x+1]) / 2; for (int x=1; x<max_x; ++x) { d1[x] = (s1[x-1] – s1[x+1]) / 2; (s2[x-1] d3[x] = (s3[x-2] – s3[x+6]) / 2; d[x+1] = – s2[x+1]) / 2; d2[x] = (s2[x-1] – s2[x+1]) / 2; (s3[x-1] d4[x] = (s4[x-1] – s4[x+7]) / 2; d[x+2] = – s3[x+1]) / 2; dr[x] = (sr[x+1] - sr[x-1]) / 2; d3[x] = (s3[x-1] – d4[x] = (s4[x-1] – s3[x+1]) / } = s4[x+1]) / 2; } d[x+3]2; (s4[x-1] – s4[x+1]) / 2; for (int x=1; x<max_x; ++x) { d[x] = (s[x-3] – s[x+3]) / 2; } d[x+1] = (s[x-2] – s[x+4]) / 2; d[x+2] = (s[x-1] – s[x+5]) / 2; dg[x] = (sg[x+1] - sg[x-1]) / 2; } for (int x=1; x<max_x; ++x) { for (int x=1; x<max_x; ++x) { d1[x] = (s1[x-1] – s1[x+1]) / 2; for (int x=1; x<max_x; ++x) { d1[x] = (s[x-3] – s[x+3]) / 2; } d1[x] = (s1[x-1] – s1[x+1]) / 2; d2[x] = (s[x-2] – s[x+4]) / 2; db[x] = (sb[x+1] - sb[x-1]) / 2; } d3[x] = (s[x-1] – s[x+5]) / 2; d2[x] = (s2[x-1] – s2[x+1]) / 2; d3[x] = (s3[x-1] – s3[x+1]) / 2; d4[x] = (s4[x-1] – s4[x+1]) / 2; }} 702010 Adobe Systems Incorporated. All Rights Reserved.
  • 71. Agenda What is GIL? Basic concepts and navigation Generic code Image view transformations 1D iterators; pixel algorithms GIL extensions Is GIL for you? 712010 Adobe Systems Incorporated. All Rights Reserved.
  • 72. Using Image View Transformations I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D Dtemplate <typename SrcView, typename DstView>void y_gradient(const SrcView& src, const DstView& dst){ x_gradient(rotated90cw_view(src), rotated90cw_view(dst));} 722010 Adobe Systems Incorporated. All Rights Reserved.
  • 73. Using Image View Transformations I I I I I D D D D D D D D D D D D D D D I I I I I I I I I I I I I I I I I I Itemplate <typename SrcView, typename DstView> Ivoid y_gradient(const SrcView& src, const DstView& dst){ x_gradient(rotated90cw_view(src), rotated90cw_view(dst));} returns a new view Fast: no copying of data! 732010 Adobe Systems Incorporated. All Rights Reserved.
  • 74. Compute gradient just over the N-th channel R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B x_gradient(nth_channel_view(src, n), dst); returns a step view 742010 Adobe Systems Incorporated. All Rights Reserved.
  • 75. Compute gradient just over the N-th channel R R R R R G G G G G B B B B B R R R R R G G G G G B B B B B R R R R R G G G G G B B B B B R R R R R G G G G G B B B B B R R R R R G G G G G B B B B B x_gradient(nth_channel_view(src, n), dst); returns a simple, non- step view 752010 Adobe Systems Incorporated. All Rights Reserved.
  • 76. Compute gradient just over the even pixels R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G B R G Bx_gradient(subsampled_view(src,2,2), dst); 762010 Adobe Systems Incorporated. All Rights Reserved.
  • 77. Compute luminosity gradient of the image RGRAYB G RGRAYB G RGRAYB G GRAY R G B RGRAYB G RGRAYB G RGRAYB G RGRAYB G GRAY R G B RGRAYB G RGRAYB G RGRAYB G RGRAYB G GRAY R G B RGRAYB G RGRAYB G RGRAYB G RGRAYB G GRAY R G B RGRAYB G RGRAYB G RGRAYB G RGRAYB G GRAY R G B RGRAYB Gx_gradient(color_converted_view<gray8_pixel_t>(src),dst); 772010 Adobe Systems Incorporated. All Rights Reserved.
  • 78. GIL View Transformation Functionstemplate <View> View flipped_up_down_view(const View& src);template <View> View1 flipped_left_right_view(const View& src);template <View> View1 transposed_view(const View& src);template <View> View1 rotated180_view(const View& src);template <View> View1 rotated90cw_view(const View& src);template <View> View1 rotated90ccw_view(const View& src);template <View> View subimage_view(const View& src, const View::point_t& top_left, const View::point_t& dims);template <View> View1 subsampled_view(const View& src, const View::point_t& step);template <View,P> View2 color_converted_view(const View& src);template <View> View3 nth_channel_view(const View& view, int n); 782010 Adobe Systems Incorporated. All Rights Reserved.
  • 79. Sample Code – Using Image View Transformationsjpeg_read_image(in_dir + "monkey.jpg“, img);step1 = subimage_view(view(img), 200,300,150,150);step2 = color_converted_view<gray8_pixel_t>(step1);step3 = rotated180_view(step2);step4 = subsampled_view(step3, 2,1);x_gradient(step4, grad_view); No extra memory used No copying of pixels (working on original data) All the work is done inside x_gradient 792010 Adobe Systems Incorporated. All Rights Reserved.
  • 80. Agenda What is GIL? Basic concepts and navigation Generic code Image view transformations 1D iterators; pixel algorithms GIL extensions Is GIL for you? 802010 Adobe Systems Incorporated. All Rights Reserved.
  • 81. Subimage View I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D 812010 Adobe Systems Incorporated. All Rights Reserved.
  • 82. Subimage View I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D 822010 Adobe Systems Incorporated. All Rights Reserved.
  • 83. Subimage View I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D Dvoid x_gradient(const gray8c_view_t& src, const gray8s_view_t& dst) { if (src.width()<=2) return; x_gradient_unsafe( subimage_view(src,1,0,src.width()-2,src.height()), subimage_view(dst,1,0,src.width()-2,src.height()) );} 832010 Adobe Systems Incorporated. All Rights Reserved.
  • 84. Subimage View I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D Dvoid x_gradient(const gray8c_view_t& src, const gray8s_view_t& dst) { if (src.width()<=2) return; x_gradient_unsafe( subimage_view(src,1,0,src.width()-2,src.height()), subimage_view(dst,1,0,src.width()-2,src.height()) );} 842010 Adobe Systems Incorporated. All Rights Reserved.
  • 85. Subimage View I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D Dvoid x_gradient(const gray8c_view_t& src, const gray8s_view_t& dst) { if (src.width()<=2) return; x_gradient_unsafe( subimage_view(src,1,0,src.width()-2,src.height()), subimage_view(dst,1,0,src.width()-2,src.height()) );} 852010 Adobe Systems Incorporated. All Rights Reserved.
  • 86. Subimage View I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D D I I I I I D D Dvoid x_gradient(const gray8c_view_t& src, const gray8s_view_t& dst) { if (src.width()<=2) return; canonical algorithm x_gradient_unsafe( subimage_view(src,1,0,src.width()-2,src.height()), subimage_view(dst,1,0,src.width()-2,src.height()) );} 862010 Adobe Systems Incorporated. All Rights Reserved.
  • 87. 1-D pixel iteratorsvoid x_gradient_unsafe(const gray8c_view_t& src, const gray8s_view_t& dst){ gray8c_view_t::iterator src_it = src.begin(); gray8s_view_t::iterator dst_it = dst.begin(); while (dst_it!=dst.end()) { *dst_it = (src_it.x()[1] - src_it.x()[-1]) / 2; ++src_it; ++dst_it; }} 872010 Adobe Systems Incorporated. All Rights Reserved.
  • 88. 1-D pixel iteratorsvoid x_gradient_unsafe(const gray8c_view_t& src, const gray8s_view_t& dst){ gray8c_view_t::iterator src_it = src.begin(); gray8s_view_t::iterator dst_it = dst.begin(); while (dst_it!=dst.end()) { *dst_it = (src_it.x()[1] - src_it.x()[-1]) / 2; ++src_it; ++dst_it; } P P P P} 1-D iterator over all pixels: P P P P P P P P 882010 Adobe Systems Incorporated. All Rights Reserved.
  • 89. 1-D pixel iteratorsvoid x_gradient_unsafe(const gray8c_view_t& src, const gray8s_view_t& dst){ gray8c_view_t::iterator src_it = src.begin(); gray8s_view_t::iterator dst_it = dst.begin(); while (dst_it!=dst.end()) { *dst_it = (src_it.x()[1] - src_it.x()[-1]) / 2; ++src_it; ++dst_it; }} STL-like interface 892010 Adobe Systems Incorporated. All Rights Reserved.
  • 90. Image View as 1-D arrayvoid x_gradient_unsafe(const gray8c_view_t& src, const gray8s_view_t& dst){ for (size_t i=0; i<src.size(); ++i) dst[i] = (src.at(i).x()[1] - src.at(i).x()[-1])/2;} 902010 Adobe Systems Incorporated. All Rights Reserved.
  • 91. Image View as 1-D arrayvoid x_gradient_unsafe(const gray8c_view_t& src, const gray8s_view_t& dst){ for (size_t i=0; i<src.size(); ++i) dst[i] = (src.at(i).x()[1] - src.at(i).x()[-1])/2;} Can use view like std::vector 912010 Adobe Systems Incorporated. All Rights Reserved.
  • 92. STL-like pixel algorithmsstruct half_x_difference { int operator()(const gray8c_loc_t& src_loc) const { return (src_loc.x()[1] - src_loc.x()[-1]) / 2; }};void x_gradient_unsafe(const gray8c_view_t& src, const gray8s_view_t& dst) { transform_pixel_positions(src, dst, half_x_difference());} GIL pixel-level algorithm 922010 Adobe Systems Incorporated. All Rights Reserved.
  • 93. GIL Pixel-Level Algorithmsgil::fill_pixels std::fillgil::for_each_pixel std::for_eachgil::transform_pixels std::transformgil::equal_pixels std::equalgil::copy_pixels std::copygil::for_each_pixel_position adobe::for_each_positiongil::transform_pixel_positionsgil::copy_and_convert_pixels 932006 Adobe Systems Incorporated. All Rights Reserved.
  • 94. Agenda What is GIL? Basic concepts and navigation Generic code Image view transformations 1D iterators; pixel algorithms GIL extensions Is GIL for you? 942010 Adobe Systems Incorporated. All Rights Reserved.
  • 95. GIL Extensions I/O (Christian Henning) Support for reading/writing common  read/write jpeg/tiff/png/bmp image formats Dynamic Image (Lubomir Bourdev, Hailin Jin)  Support for images with runtime instantiated types Support for images whose type is specified at run time Numeric (Hailin Jin, Lubomir Bourdev) bilinear resampling, 1D convolution (resize, blur, etc)  Basic image processing FreetypeGIL (Tom Brinkman)  Freetype wrapper to GIL GIL2.Bindings (Hirotaka Niitsuma) GIL wrappers for popular image libraries  Interface to VIGRA, OpenCV, uBLAS, TEO OpenCV (Christian Henning) Improved Debugging  Support for OpenCV algorithms SDL (Christian Henning)  Improves debugging – displays current image in a window Multithreading GIL-Threaded (Victor Bogado) multithreading and multi-core support for GIL algorithms  Rendering with antialiasing WulineGIL (Tom Brinkman)  Draw antialiased lines using the Wu algorithm Toolbox (Christian Henning) Additional color spaces  Support for HSL and HSV color spaces and some new RGB algorithms 952010 Adobe Systems Incorporated. All Rights Reserved.
  • 96. GIL Extensions Wanted Comprehensive image processing  FFT, even 2D convolution, high-quality efficient resampling Computer vision algorithms  Canny edge detection, SIFT, GB, segmentation, HOG… Support for video and multi-dimensional data Support for Intel’s IPP Can you help? 962010 Adobe Systems Incorporated. All Rights Reserved.
  • 97. Agenda What is GIL? Basic concepts and navigation Generic code Image view transformations 1D iterators; pixel algorithms GIL Extensions Is GIL for you? 972010 Adobe Systems Incorporated. All Rights Reserved.
  • 98. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat”10. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 982010 Adobe Systems Incorporated. All Rights Reserved.
  • 99. Can generic code be as efficient as hand-written code?GENERIC copy_pixels while (first!=last) *dst++ = *first++; copy copy copy copy copy chunky rgb chunky rgb planar rgb planar rgb planar cmykCONCRETE chunky rgb planar rgb chunky rgb planar rgb planar cmyk 992010 Adobe Systems Incorporated. All Rights Reserved.
  • 100. Concrete versions can be made much fasterGENERIC copy_pixels while (first!=last) *dst++ = *first++; an order of magnitude faster than the generic version memmove(…) memmove(…) memmove(…) copy copy copy copy copy chunky rgb chunky rgb planar rgb planar rgb planar cmykCONCRETE chunky rgb planar rgb chunky rgb planar rgb planar cmyk 1002010 Adobe Systems Incorporated. All Rights Reserved.
  • 101. GIL provides overloads at appropriate level of genericityGENERIC copy_pixels while (first!=last) *dst++ = *first++; memmove each single color plane memmoveSEMI- copy copyGENERIC chunky to planar to chunky planar copy copy copy copy copy chunky rgb chunky rgb planar rgb planar rgb planar cmykCONCRETE chunky rgb planar rgb chunky rgb planar rgb planar cmyk 1012010 Adobe Systems Incorporated. All Rights Reserved.
  • 102. Advantage: New image types can be handled efficientlyGENERIC copy_pixels while (first!=last) *dst++ = *first++; memmove each single color plane memmoveSEMI- copy copyGENERIC chunky to planar to chunky planar copy copy copy copy copy copy chunky rgb chunky rgb planar rgb planar xylab planar rgb planar cmykCONCRETE chunky rgb planar rgb chunky rgb planar xylab planar rgb planar cmyk 1022010 Adobe Systems Incorporated. All Rights Reserved.
  • 103. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat”10. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 1032010 Adobe Systems Incorporated. All Rights Reserved.
  • 104. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat”10. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 1042010 Adobe Systems Incorporated. All Rights Reserved.
  • 105. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat”10. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 1052010 Adobe Systems Incorporated. All Rights Reserved.
  • 106. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat”10. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 1062010 Adobe Systems Incorporated. All Rights Reserved.
  • 107. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat”10. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 1072010 Adobe Systems Incorporated. All Rights Reserved.
  • 108. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat”10. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 1082010 Adobe Systems Incorporated. All Rights Reserved.
  • 109. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat”10. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 1092010 Adobe Systems Incorporated. All Rights Reserved.
  • 110. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat” Bourdev & Järvi, SCP’0810. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 1102010 Adobe Systems Incorporated. All Rights Reserved.
  • 111. “I would use GIL but…”1. “… I can’t hand-tune and SSE-optimize generic code”2. “… I am planning to adopt / have adopted another imaging library”3. “… I already have imaging code. I can’t afford to throw it away”4. “… I don’t want to have any extra library dependencies”5. “… who is going to support me?”6. “… I am concerned about portability. How portable is GIL?”7. “… compile errors in generic code are a nightmare”8. “… GIL has a steep learning curve”9. “… generic code can result in code bloat”10. “… generic code is opaque. I can’t easily tell if the generated code is efficient” 1112010 Adobe Systems Incorporated. All Rights Reserved.
  • 112. Who uses GIL? Inside Adobe  Several Photoshop features  Some parts of AfterEffects  Some parts of Premiere  Some features in Photoshop Elements  Other Outside Adobe  We don’t know for sure who uses it and how  We have been receiving questions from HP, Motorola, Microsoft, MIT, Univ. Arizona, CMU, etc. 1122010 Adobe Systems Incorporated. All Rights Reserved.
  • 113. Case Studies Original idea of GIL: Two products need the face detector, one uses planar images, the other interleaved. Gaussian Mixture Model per pixel with 30 components  Lots of memory (30 images), slow to compute (need to eval it only in some places)  Implemented as virtual images. Photoshop request: due to fragmentation, image may be allocated in N pieces: Separate memory blocks GIL solution: define a new y-iterator. No algorithms need to change 1132010 Adobe Systems Incorporated. All Rights Reserved.
  • 114. Acknowledgements Jon Brandt, Mark Ruzon and Paul McJones spent significant time reviewing the library and documentation and provided valuable feedback. Alex Stepanovs class has directly inspired this project. Alexs "start from the inside" and "bottom up" principles have been applied throughout the design. Sean Parent and Alex Stepanov suggested splitting the image into a container and range classes Foster Brereton helped evangelize GIL and worked on I/O and integration with Adam/Eve Jaakko Järvi collaborated with us on eliminating code bloat in GIL Bjarne Stroustrup reviewed the design & provided suggestions for future work The Boost Community did a very thorough review with dozens of suggestions for improvements 1192010 Adobe Systems Incorporated. All Rights Reserved.
  • 115. Further information Code and documentation: http://opensource.adobe.com/gil The above URL also has:  Hands-on Tutorial (12 pages)  Detailed Design Guide (40 pages)  Video tutorial  Discussion group Your feedback and contribution are essential!!! 1202010 Adobe Systems Incorporated. All Rights Reserved.