Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ProsumerFX: Mobile Design of Image Stylization Components

20 views

Published on

With the continuous advances of mobile graphics hardware, high-quality image stylization—e.g., based on image filtering, stroke-based rendering, and neural style transfer—is becoming feasible and increasingly used in casual creativity apps. The creative expression facilitated by these mobile apps, however, is typically limited with respect to the usage and application of pre-defined visual styles, which ultimately do not include their design and composition—an inherent requirement of prosumers. We present ProsumerFX, a GPU-based app that enables to interactively design parameterizable image stylization components on-device by reusing building blocks of image processing effects and pipelines. Furthermore, the presentation of the effects can be customized by modifying the icons, names, and order of parameters and presets. Thereby, the customized visual styles are defined as platform-independent effects and can be shared with other users via a web-based platform and database. Together with the presented mobile app, this system approach supports collaborative works for designing visual styles, including their rapid prototyping, A/B testing, publishing, and distribution. Thus, it satisfies the needs for creative expression of both professionals as well as the general public.

Published in: Science
  • Be the first to comment

  • Be the first to like this

ProsumerFX: Mobile Design of Image Stylization Components

  1. 1. ProsumerFX: Mobile Design of Image Stylization Components Tobias Dürschmid, Maximilian Söchting, Amir Semmo, Mattias Trapp, Jürgen Döllner Hasso Plattner Institute, University of Potsdam, Germany
  2. 2. • Large amount of casual creativity apps that allow the application of preconfigured visual styles to images • Filtered images more likely to be shared on social media [Bakhshi et al. 2015] MOTIVATION Instagram Prisma https://commons.wikimedia.org/wiki/File:Instagram_collage_with_15_different_filters.jpg CC BY 3.0 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 2
  3. 3. MOTIVATION ? Existing approaches (Consumption) Presented approach (Prosumption) 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 3
  4. 4. • Goal: Enable prosumption of visual styles for non-expert users • Have users compose own visual styles with easy-to-use building blocks • Have users share own visual styles through a web-based platform MOTIVATION Prosumption of visual styles 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 4
  5. 5. • Enable rapid prototyping and A/B testing of visual styles • Modification of the presentation of the created visual styles MOTIVATION Prototype Review Design & Iterate 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 5
  6. 6. DEMO 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 6 https://www.youtube.com/watch?v=yv8N1tIce84
  7. 7. • Imageplay.io: Allows combination of low-level GPU-based algorithms • Snapchat Geofilters: Allows building of basic transparency-based filters with texts and images RELATED WORK Imageplay Snapchat Geofilters 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 7
  8. 8. • Interactive Modification • User-friendly UI for various modifications • Modifications have to be propagated to the GPU immediately • Serialization of visual styles • Persistence of user-defined visual styles • Device Heterogeneity • Many different mobile and desktop devices • Reusable Building Blocks (e.g., edge detection, color quantization) • Modularization of image stylization components CHALLENGES 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 8
  9. 9. Modifications of parameters, presets, and effects: • Swap • Remove • Rename • Exchange icon CONCEPT – UI 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 9
  10. 10. CONCEPT – INTERACTION Level of Control [Semmo et al. 2016] Consumption Prosumption low medium high Parameter Modification Preset Modification Effect Recombination Preset Selection Parameter Adjustment Parameter Painting 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 10
  11. 11. CONCEPT – STYLE SERIALIZATION • Store and transfer visual styles in a serialized form • An effect consists of • Rendering passes • Shader programs • Textures • Parameters • Presets 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 11
  12. 12. CONCEPT – PLATFORM INDEPENDENCE Effect Definition Effect Implementation OpenGL ES 2.0 Effect Implementation OpenGL 4.5 (Desktop) Parameters Presets Icons Shaders Rendering Passes Define the platform- independent part (user modifiable) Define the platform- specific part (not user modifiable) Effect Definition Effect Implementation OpenGL ES 2.0 (Android, iOS) 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 12
  13. 13. CONCEPT – REUSABLE BUILDING BLOCKS 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 13
  14. 14. • Exploration of all publicly available visual styles in a web store-like interface • Publishing of newly created visual styles through a REST API CONCEPT – SERVER Alternative views of the web interface 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 14
  15. 15. IMPLEMENTATION 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 15
  16. 16. RESULTS Building blocks: • Neural Style Transfer [Johnson et al. 2016] • Toon Filtering [Winnemöller et al. 2006] • Watercolor [Bousseau 2006] • Color grading [Selan 2004] 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 16
  17. 17. RESULTS Building blocks: • Neural Style Transfer [Johnson et al. 2016] • Toon Filtering [Winnemöller et al. 2006] • Watercolor [Bousseau 2006] • Color grading [Selan 2004] 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 17
  18. 18. RESULTS Building blocks: • Neural Style Transfer [Johnson et al. 2016] • Toon Filtering [Winnemöller et al. 2006] • Watercolor [Bousseau 2006] • Color grading [Selan 2004] 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 18
  19. 19. RESULTS Building blocks: • Neural Style Transfer [Johnson et al. 2016] • Toon Filtering [Winnemöller et al. 2006] • Watercolor [Bousseau 2006] • Color grading [Selan 2004] 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 19
  20. 20. RESULTS Building blocks: • Neural Style Transfer [Johnson et al. 2016] • Toon Filtering [Winnemöller et al. 2006] • Watercolor [Bousseau 2006] • Color grading [Selan 2004] 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 20
  21. 21. RESULTS Building blocks: • Neural Style Transfer [Johnson et al. 2016] • Toon Filtering [Winnemöller et al. 2006] • Watercolor [Bousseau 2006] • Color grading [Selan 2004] 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 21
  22. 22. RESULTS Building blocks: • Neural Style Transfer [Johnson et al. 2016] • Toon Filtering [Winnemöller et al. 2006] • Watercolor [Bousseau 2006] • Color grading [Selan 2004] 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 22
  23. 23. RESULTS Test Device: Samsung S7 (Mali T880) Result: No additional processing overhead when combining effects. 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 23
  24. 24. RESULTS Test Device: Samsung S7 (Mali T880) Result: The capacity of effects depends on the rendering resolution. WQHD: 9 effects FHD: 12 effects HD: 14 effects Pencil Hatching Effect: Flow-oriented example-based hatching with contours, consisting of 14 render-to-texture passes, five half-float textures, 17 byte textures, and 18 small tonal art map textures 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 24
  25. 25. • Swapping effects: 1ms + rendering time • Adding an effect ≈1s + rendering time • Removing an effect: 2ms + rendering time Test Device: Samsung Galaxy S7 (CPU octa-core 4x2.3 GHz, 4x1.6 GHz) Result: Style composition is interactive. RESULTS – PERFORMANCE 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 26
  26. 26. • Creation of visual styles is enabled in real-time • Support for sharing visual styles via XML-based document format • Device heterogeneity achieved through platform-independent persistence • Proof of concept: effect database contains 190 effects (e.g., neural style transfer, water color, oil paint, cartoon, pencil hatching) DISCUSSION 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 27
  27. 27. CONTRIBUTIONS • Concept for mobile prosumption of image stylization components • Document format for platform-independent persistence of visual styles • Web-based platform for sharing visual styles 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 28
  28. 28. FUTURE WORK • Enable low level effect modifications • Exchange processing steps of effect • Exchange of textures • Collapsing similar parameters • Hinting rendering performance • Community platform • Public release 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 29
  29. 29. CONCLUSION • Enable novice users and professionals to create and manipulate their individual visual styles using one ecosystem • Eases rapid prototyping & A/B testing of visual styles for mobile platforms • The presented concept goes beyond user-generated content, because users generate tools (visual styles) that enable other users to generate content (styled images) 27.11.2017 ProsumerFX: Mobile Design of Image Stylization Components, T. Dürschmid et al. 30
  30. 30. THANK YOU FOR YOUR ATTENTION Tobias Dürschmid: tobias.duerschmid@student.hpi.de Maximilian Söchting: maximilian.soechting@student.hpi.de Amir Semmo: amir.semmmo@hpi.de Matthias Trapp: matthias.trapp@hpi.de Jürgen Döllner: juergen.doellner@hpi.de Project 01IS15041B

×