1. ProsumerFX:
Mobile Design of Image Stylization Components
Tobias Dürschmid, Maximilian Söchting, Amir Semmo, Mattias Trapp, Jürgen Döllner
Hasso Plattner Institute, University of Potsdam, Germany

2. • Large amount of casual creativity apps that allow the application of
preconfigured visual styles to images
• Filtered images more likely to be shared on social media [Bakhshi et al. 2015]
MOTIVATION
Instagram Prisma
https://commons.wikimedia.org/wiki/File:Instagram_collage_with_15_different_filters.jpg CC BY 3.0
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 2

3. MOTIVATION
?
Existing approaches (Consumption)
Presented approach (Prosumption)
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 3

4. • Goal: Enable prosumption of visual styles for non-expert users
• Have users compose own visual styles with easy-to-use building blocks
• Have users share own visual styles through a web-based platform
MOTIVATION
Prosumption of visual styles
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 4

5. • Enable rapid prototyping and A/B testing of visual styles
• Modification of the presentation of the created visual styles
MOTIVATION
Prototype
Review
Design &
Iterate
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 5

6. DEMO
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 6
https://www.youtube.com/watch?v=yv8N1tIce84

7. • Imageplay.io: Allows combination of low-level GPU-based algorithms
• Snapchat Geofilters: Allows building of basic transparency-based filters
with texts and images
RELATED WORK
Imageplay Snapchat Geofilters
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 7

8. • Interactive Modification
• User-friendly UI for various modifications
• Modifications have to be propagated to the GPU immediately
• Serialization of visual styles
• Persistence of user-defined visual styles
• Device Heterogeneity
• Many different mobile and desktop devices
• Reusable Building Blocks (e.g., edge detection, color quantization)
• Modularization of image stylization components
CHALLENGES
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 8

9. Modifications of parameters,
presets, and effects:
• Swap
• Remove
• Rename
• Exchange icon
CONCEPT – UI
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 9

10. CONCEPT – INTERACTION
Level of Control
[Semmo et al. 2016]
Consumption
Prosumption
low medium high
Parameter
Modification
Preset
Modification
Effect
Recombination
Preset
Selection
Parameter
Adjustment
Parameter
Painting
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 10

11. CONCEPT – STYLE SERIALIZATION
• Store and transfer visual styles
in a serialized form
• An effect consists of
• Rendering passes
• Shader programs
• Textures
• Parameters
• Presets
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 11

12. CONCEPT – PLATFORM INDEPENDENCE
Effect
Definition
Effect
Implementation
OpenGL ES 2.0
Effect
Implementation
OpenGL 4.5
(Desktop)
Parameters
Presets
Icons
Shaders
Rendering Passes
Define the platform-
independent part
(user modifiable)
Define the platform-
specific part
(not user modifiable)
Effect
Definition
Effect
Implementation
OpenGL ES 2.0
(Android, iOS)
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 12

13. CONCEPT – REUSABLE BUILDING BLOCKS
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 13

14. • Exploration of all publicly available visual styles in a web store-like
interface
• Publishing of newly created visual styles through a REST API
CONCEPT – SERVER
Alternative views of the web interface
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 14

15. IMPLEMENTATION
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 15

16. RESULTS
Building blocks:
• Neural Style Transfer
[Johnson et al. 2016]
• Toon Filtering
[Winnemöller et al.
2006]
• Watercolor
[Bousseau 2006]
• Color grading
[Selan 2004]
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 16

17. RESULTS
Building blocks:
• Neural Style Transfer
[Johnson et al. 2016]
• Toon Filtering
[Winnemöller et al.
2006]
• Watercolor
[Bousseau 2006]
• Color grading
[Selan 2004]
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 17

18. RESULTS
Building blocks:
• Neural Style Transfer
[Johnson et al. 2016]
• Toon Filtering
[Winnemöller et al.
2006]
• Watercolor
[Bousseau 2006]
• Color grading
[Selan 2004]
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 18

19. RESULTS
Building blocks:
• Neural Style Transfer
[Johnson et al. 2016]
• Toon Filtering
[Winnemöller et al.
2006]
• Watercolor
[Bousseau 2006]
• Color grading
[Selan 2004]
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 19

20. RESULTS
Building blocks:
• Neural Style Transfer
[Johnson et al. 2016]
• Toon Filtering
[Winnemöller et al.
2006]
• Watercolor
[Bousseau 2006]
• Color grading
[Selan 2004]
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 20

21. RESULTS
Building blocks:
• Neural Style Transfer
[Johnson et al. 2016]
• Toon Filtering
[Winnemöller et al.
2006]
• Watercolor
[Bousseau 2006]
• Color grading
[Selan 2004]
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 21

22. RESULTS
Building blocks:
• Neural Style Transfer
[Johnson et al. 2016]
• Toon Filtering
[Winnemöller et al.
2006]
• Watercolor
[Bousseau 2006]
• Color grading
[Selan 2004]
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 22

23. RESULTS
Test Device: Samsung S7 (Mali T880)
Result: No additional processing
overhead when combining effects.
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 23

24. RESULTS
Test Device: Samsung S7 (Mali T880)
Result: The capacity of effects
depends on the rendering resolution.
WQHD: 9 effects
FHD: 12 effects
HD: 14 effects
Pencil Hatching Effect:
Flow-oriented example-based hatching with
contours, consisting of 14 render-to-texture
passes, five half-float textures, 17 byte textures,
and 18 small tonal art map textures
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 24

25. • Swapping effects: 1ms + rendering time
• Adding an effect ≈1s + rendering time
• Removing an effect: 2ms + rendering time
Test Device: Samsung Galaxy S7 (CPU octa-core 4x2.3 GHz, 4x1.6 GHz)
Result: Style composition is interactive.
RESULTS – PERFORMANCE
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 26

26. • Creation of visual styles is enabled in real-time
• Support for sharing visual styles via XML-based document format
• Device heterogeneity achieved through platform-independent
persistence
• Proof of concept: effect database contains 190 effects (e.g., neural style
transfer, water color, oil paint, cartoon, pencil hatching)
DISCUSSION
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 27

27. CONTRIBUTIONS
• Concept for mobile prosumption of image stylization components
• Document format for platform-independent persistence of visual styles
• Web-based platform for sharing visual styles
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 28

28. FUTURE WORK
• Enable low level effect modifications
• Exchange processing steps of effect
• Exchange of textures
• Collapsing similar parameters
• Hinting rendering performance
• Community platform
• Public release
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 29

29. CONCLUSION
• Enable novice users and professionals to create and manipulate their
individual visual styles using one ecosystem
• Eases rapid prototyping & A/B testing of visual styles for mobile platforms
• The presented concept goes beyond user-generated content,
because users generate tools (visual styles) that enable other users to
generate content (styled images)
27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 30

30. THANK YOU FOR YOUR ATTENTION
Tobias Dürschmid: tobias.duerschmid@student.hpi.de
Maximilian Söchting: maximilian.soechting@student.hpi.de
Amir Semmo: amir.semmmo@hpi.de
Matthias Trapp: matthias.trapp@hpi.de
Jürgen Döllner: juergen.doellner@hpi.de
Project 01IS15041B