iOS Visual F/XDouglass TurnerElastic Image Softwareemail: douglass.turner@gmail.comtweets: @duglacell: 781 775 3708
• Cocoa Touch Limitations• Visual f/x Idioms• Show and Tell
Cocoa Touch and its realization inObjective-C is about:Abstraction. Pattern. Messaging.Generality. Simplicity.
Containers:NSArray. NSDictionary. NSSet.Data:NSData. NSString. CoreData.Patterns:MVC. Target/Action. Delegation.
The Objective-C Runtime is Powerful:NSClassFromString | NSStringFromClass.
When it comes to visual expressivepower the limitations of CocoaTouch become apparent.
UIView lives in flatland ...// UIViewstruct CGRect { CGPoint origin; CGSize size; };struct CGPoint { CGFloat x; CGFloat y; ...
... but relies on CALayer which lives in 3D.        struct CATransform3D        {           CGFloat m11, m12, m13,   m14; ...
iOS would prefer you ignore pixels and thinkabout resolution independent points instead
The level of visual abstraction is the asset:Image. Video. Audio.
As long as you are willing to remain at thishigh level of abstraction you can get a lotdone using UIViewAnimation, CIImage...
But perhaps you are interested in going a bit deeper ...
Hello OpenGLYou take the red pill – you stay in Wonderland and I show you how deepthe rabbit-hole goes." -Morpheus
OpenGL   Cocoa
OpenGL is old school
You are flipping levers on a large state machine.
OpenGL is a dragster. Not a Prius. Just buckle up ...
... and enjoy the ride, baby!
OpenGL websites are a bit different than theCocoa websites/blogs you are familiar with ...
Isn’t that sweet ...
Um... WTF?
Can you feel the love?
Dude. I think my eyes are bleeding.
Our focus is GLSL the OpenGL Shading Language
The Shader Backstory• Pixar creates Reyes (Render Everything You Ever Saw)• Pixar creates RenderMan: C-like language for d...
Shaders allow us to ignore the rest of asoftware system, freeing us to focus onachieving a specific look.
EISParticleSystemhttp://github.com/turner/EISParticleSystem
Texture Atlas Hackshttp://github.com/turner/EISParticleSystem
t                    s    Texture Space
t                    s    Texture Space
Accelerometer Hackshttp://github.com/turner/EISParticleSystem
Initialize model statetouchesBegan:withEvent:Evolve model statetouchesMoved:withEvent:Clean up model statetouchesEnded:wit...
Demo
GLSL. Think C-lite.
TextureMapShadervarying!       vec2 v_st;uniform sampler2D hero;void main() {!!   gl_FragColor =    (heroChannels == 1) ? ...
InvertColorShadervec2 v_st;uniform int heroChannels;uniform sampler2D hero;void main() {!   !    vec3 rgb;    rgb = (heroC...
LuminanceShadervarying vec2 v_st;uniform int heroChannels;uniform sampler2D hero;void main() {!    vec4 raw = (heroChannel...
MixShadervarying!       vec2 v_st;uniform vec3 overRGB;!uniform vec3 underRGB;uniform int heroChannels;uniform sampler2D h...
The GPU is a parallel processing beast. It uses a SIMDarchitecture (Single Instruction Multiple Data) to achievemassive pr...
What is exciting about this level of performance on an iOSdevice is the ability to take tasks previously thought of asdesk...
Demo. GLSL Powered Apps• Beautiful Panoramas. App Store: http://bit.ly/9KJBLA• BMW Interior. Panoramic hotspots prototype....
Shader IdiomsTo fully “get” the power of shaders and their style of use it helpsknow the idioms guiding their use:• Multip...
Shader IdiomsThese idioms derive directly from Hollywood film productionworkflows and practices that enable:• complex proble...
Demo. ElasticImage.
ElasticImage highlights.• Rapid shader creation & deployment• Shaders and gesture declared in plist• Cocoa Touch for gestu...
Shader Example: ColourLovers
I selected interesting palettes and created a texture.Horizontal axis is the color palette.Vertical axis selects between c...
EISColourLoversShader.fshvarying!       mediump vec2 v_st;// Palette selectoruniform float paletteDial;// Contribution of ...
EISColourLoversShader.fshvarying!       mediump vec2 v_st;// Palette selectoruniform float paletteDial;// Contribution of ...
Demo ColourLovers Shader
Hue Shift ShaderRGB                      HSB
Conceptually simple idea. Use one channel of aphoto to select the hue of a resultant image.
Demo Hue Shift Shader
Quantize ST Shader
Sampling a texture at a low rate results in quantization.    Beyond Photography - The Digital Darkroom by Gerard J. Holzmann
Sampling in polar coordinates rather then cartesian is a bit more interesting.               Beyond Photography - The Digi...
Demo Quantize ST Shader
BurnShader
red channel = drop shadow                      green channel = cut out                            final rgb texture
Demo BurnShader
LinksA previous meetup talk I gave on iOS OpenGL• iOS OpenGL - http://slidesha.re/Y1MW8Github - code• HelloGLSL - http://b...
Thank You!
Douglass TurnerElastic Image Softwareemail: douglass.turner@gmail.comtweets: @duglacell: 781 775 3708                     ...
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
iOS Visual F/X Using GLSL
Upcoming SlideShare
Loading in …5
×

iOS Visual F/X Using GLSL

4,694 views

Published on

Original Presentation: http://bit.ly/JdJt0v
Tweets: @dugla
Email: douglass.turner@gmail.com

Published in: Technology, Art & Photos
0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
4,694
On SlideShare
0
From Embeds
0
Number of Embeds
89
Actions
Shares
0
Downloads
27
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

iOS Visual F/X Using GLSL

  1. 1. iOS Visual F/XDouglass TurnerElastic Image Softwareemail: douglass.turner@gmail.comtweets: @duglacell: 781 775 3708
  2. 2. • Cocoa Touch Limitations• Visual f/x Idioms• Show and Tell
  3. 3. Cocoa Touch and its realization inObjective-C is about:Abstraction. Pattern. Messaging.Generality. Simplicity.
  4. 4. Containers:NSArray. NSDictionary. NSSet.Data:NSData. NSString. CoreData.Patterns:MVC. Target/Action. Delegation.
  5. 5. The Objective-C Runtime is Powerful:NSClassFromString | NSStringFromClass.
  6. 6. When it comes to visual expressivepower the limitations of CocoaTouch become apparent.
  7. 7. UIView lives in flatland ...// UIViewstruct CGRect { CGPoint origin; CGSize size; };struct CGPoint { CGFloat x; CGFloat y; };struct CGAffineTransform { CGFloat a, b, c, d; CGFloat tx, ty;};@property(nonatomic) CGRect frame;@property(nonatomic) CGRect bounds;@property(nonatomic) CGPoint center;@property(nonatomic) CGAffineTransform transform;- (CGPoint)convertPoint:(CGPoint)point toView:(UIView *)view;- (CGPoint)convertPoint:(CGPoint)point fromView:(UIView *)view;- (CGRect)convertRect:(CGRect)rect toView:(UIView *)view;- (CGRect)convertRect:(CGRect)rect fromView:(UIView *)view;
  8. 8. ... but relies on CALayer which lives in 3D. struct CATransform3D { CGFloat m11, m12, m13, m14; CGFloat m21, m22, m23, m24; CGFloat m31, m32, m33, m34; CGFloat m41, m42, m43, m44; }; // CALayer @property CGFloat zPosition; @property CGFloat anchorPointZ; @property CATransform3D transform;
  9. 9. iOS would prefer you ignore pixels and thinkabout resolution independent points instead
  10. 10. The level of visual abstraction is the asset:Image. Video. Audio.
  11. 11. As long as you are willing to remain at thishigh level of abstraction you can get a lotdone using UIViewAnimation, CIImage, etc.
  12. 12. But perhaps you are interested in going a bit deeper ...
  13. 13. Hello OpenGLYou take the red pill – you stay in Wonderland and I show you how deepthe rabbit-hole goes." -Morpheus
  14. 14. OpenGL Cocoa
  15. 15. OpenGL is old school
  16. 16. You are flipping levers on a large state machine.
  17. 17. OpenGL is a dragster. Not a Prius. Just buckle up ...
  18. 18. ... and enjoy the ride, baby!
  19. 19. OpenGL websites are a bit different than theCocoa websites/blogs you are familiar with ...
  20. 20. Isn’t that sweet ...
  21. 21. Um... WTF?
  22. 22. Can you feel the love?
  23. 23. Dude. I think my eyes are bleeding.
  24. 24. Our focus is GLSL the OpenGL Shading Language
  25. 25. The Shader Backstory• Pixar creates Reyes (Render Everything You Ever Saw)• Pixar creates RenderMan: C-like language for describing a3D look at the sample (pixel) level.• Boom! Everything changes.• GLSL created in the spirit of RenderMan.
  26. 26. Shaders allow us to ignore the rest of asoftware system, freeing us to focus onachieving a specific look.
  27. 27. EISParticleSystemhttp://github.com/turner/EISParticleSystem
  28. 28. Texture Atlas Hackshttp://github.com/turner/EISParticleSystem
  29. 29. t s Texture Space
  30. 30. t s Texture Space
  31. 31. Accelerometer Hackshttp://github.com/turner/EISParticleSystem
  32. 32. Initialize model statetouchesBegan:withEvent:Evolve model statetouchesMoved:withEvent:Clean up model statetouchesEnded:withEvent:touchesCancelled:withEvent:
  33. 33. Demo
  34. 34. GLSL. Think C-lite.
  35. 35. TextureMapShadervarying! vec2 v_st;uniform sampler2D hero;void main() {!! gl_FragColor = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);!}varying! vec2 v_st;uniform sampler2D hero;void main() {!! gl_FragColor = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);!}
  36. 36. InvertColorShadervec2 v_st;uniform int heroChannels;uniform sampler2D hero;void main() {! ! vec3 rgb; rgb = (heroChannels == 1) ? 1.0 - vec3(raw.a) : 1.0 - raw.rgb; ! gl_FragColor = vec4(rgb, raw.a);!}
  37. 37. LuminanceShadervarying vec2 v_st;uniform int heroChannels;uniform sampler2D hero;void main() {! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);! vec3 rgb;! if (heroChannels == 1) {!! ! rgb = vec3(texture2D(hero, v_st).a);! !! } else {!! ! vec3 luminance_weights = vec3(0.30, 0.59, 0.11);!!! ! rgb = vec3( dot( luminance_weights, texture2D(hero, v_st).rgb ) );! }!! gl_FragColor = vec4(rgb, raw.a);}
  38. 38. MixShadervarying! vec2 v_st;uniform vec3 overRGB;!uniform vec3 underRGB;uniform int heroChannels;uniform sampler2D hero;void main() {! float mixer;!! mixer = (heroChannels == 1) ? texture2D(hero, v_st).a : texture2D(hero, v_st).g;!! gl_FragColor = vec4(mix(underRGB, overRGB, mixer), texture2D(hero, v_st).a);!}varying! vec2 v_st;uniform vec3 overRGB;!uniform vec3 underRGB;uniform int heroChannels;uniform sampler2D hero;void main() {! float mixer;!! mixer = (heroChannels == 1) ? texture2D(hero, v_st).a : texture2D(hero, v_st).g;!! gl_FragColor = vec4(mix(underRGB, overRGB, mixer), texture2D(hero, v_st).a);!}
  39. 39. The GPU is a parallel processing beast. It uses a SIMDarchitecture (Single Instruction Multiple Data) to achievemassive processing power.A GLSL shader is a SIMD program. The GPU takes a shaderand evaluates it simultaneously - but with different data - atevery sample (pixel) in parallel:On iPad that is:2048 x 1536 * 30 fps = 94,371,840 shader evaluations per sec.
  40. 40. What is exciting about this level of performance on an iOSdevice is the ability to take tasks previously thought of asdesktop tasks done with Photoshop, Final Cut, etc. and dothem live in a handheld device equipped with camera, mic,and other sensors.
  41. 41. Demo. GLSL Powered Apps• Beautiful Panoramas. App Store: http://bit.ly/9KJBLA• BMW Interior. Panoramic hotspots prototype.• RadPad. iPad radiology prototype.
  42. 42. Shader IdiomsTo fully “get” the power of shaders and their style of use it helpsknow the idioms guiding their use:• Multiple Passes• Iteration aka Ping/Pong• Buffers/Channels• Indirection/Remapping
  43. 43. Shader IdiomsThese idioms derive directly from Hollywood film productionworkflows and practices that enable:• complex problem decomposition. Tractability.• rapid turnaround.• maximum flexibility and tweek-ability.• responsiveness to director’s whims
  44. 44. Demo. ElasticImage.
  45. 45. ElasticImage highlights.• Rapid shader creation & deployment• Shaders and gesture declared in plist• Cocoa Touch for gestures & the usual.• C++ for 3D glue code.
  46. 46. Shader Example: ColourLovers
  47. 47. I selected interesting palettes and created a texture.Horizontal axis is the color palette.Vertical axis selects between color palettes.At runtime this palette texture is loaded and attached tothe ColourLoverShader.
  48. 48. EISColourLoversShader.fshvarying! mediump vec2 v_st;// Palette selectoruniform float paletteDial;// Contribution of remapped coloruniform float strengthDial;// Palettes are ganged together into a single texture.// A specific palette is selected with the paletteDialuniform sampler2D colourLoversPalettes;// herouniform int heroChannels;uniform sampler2D hero;void main() {!! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);!! vec3 cooked;! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r;! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g;! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b;!! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a);}
  49. 49. EISColourLoversShader.fshvarying! mediump vec2 v_st;// Palette selectoruniform float paletteDial;// Contribution of remapped coloruniform float strengthDial;// Palettes are ganged together into a single texture.// A specific palette is selected with the paletteDialuniform sampler2D colourLoversPalettes;// herouniform int heroChannels;uniform sampler2D hero;void main() {!! vec4 raw = (heroChannels == 1) ? vec4(texture2D(hero, v_st).a) : texture2D(hero, v_st);!! vec3 cooked;! cooked.r = texture2D(colourLoversPalettes, vec2(raw.r, paletteDial)).r;! cooked.g = texture2D(colourLoversPalettes, vec2(raw.g, paletteDial)).g;! cooked.b = texture2D(colourLoversPalettes, vec2(raw.b, paletteDial)).b;!! gl_FragColor = vec4(mix(raw.rgb, cooked, strengthDial), raw.a);}
  50. 50. Demo ColourLovers Shader
  51. 51. Hue Shift ShaderRGB HSB
  52. 52. Conceptually simple idea. Use one channel of aphoto to select the hue of a resultant image.
  53. 53. Demo Hue Shift Shader
  54. 54. Quantize ST Shader
  55. 55. Sampling a texture at a low rate results in quantization. Beyond Photography - The Digital Darkroom by Gerard J. Holzmann
  56. 56. Sampling in polar coordinates rather then cartesian is a bit more interesting. Beyond Photography - The Digital Darkroom by Gerard J. Holzmann
  57. 57. Demo Quantize ST Shader
  58. 58. BurnShader
  59. 59. red channel = drop shadow green channel = cut out final rgb texture
  60. 60. Demo BurnShader
  61. 61. LinksA previous meetup talk I gave on iOS OpenGL• iOS OpenGL - http://slidesha.re/Y1MW8Github - code• HelloGLSL - http://bit.ly/JCcMju• EISRenderHelpful. Helpful 3D rendering glue code - http://bit.ly/JZ4HW3Github - people to follow• Philip Rideout - https://github.com/prideout• Raphael Sebbe - https://github.com/rsebbe• Brad Larson - https://github.com/BradLarson• Jeff LaMarche - https://github.com/jlamarcheElsewhere:• Martins Upitis - http://devlog-martinsh.blogspot.com/• Ole Begemann - http://bit.ly/srlCBV• Daniel Rakos - http://bit.ly/a3QATn
  62. 62. Thank You!
  63. 63. Douglass TurnerElastic Image Softwareemail: douglass.turner@gmail.comtweets: @duglacell: 781 775 3708 Copyright © Douglass Turner

×