Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AI Tools for Creative Work

71 views

Published on

How we use AI as a tool to increase human productivity by a factor of 10? By augmenting our fluid intelligence.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

AI Tools for Creative Work

  1. 1. AI Tools for Creative Work Lydia B. Chilton DUB Seminar 2 October 2019
  2. 2. AI can do things it couldn’t do before… But it still struggles at ill-defined tasks.
  3. 3. It looks like you’re trying to solve world peace. Do you need assistance? AI can do things it couldn’t do before… But it still struggles at ill-defined tasks.
  4. 4. Long history of creativity support tools in HCI Jonas Frich, Lindsay MacDonald Vermeulen, Christian Remy, Michael Mose Biskjaer, and Peter Dalsgaard. 2019. Mapping the Landscape of Creativity Support Tools in HCI. CHI 2019
  5. 5. How does AI change the future of creative work? Gatys, Ecker, Bethge. A Neural Algorithm of Artistic Style. http://arxiv.org/abs/1508.06576 2015
  6. 6. Outline: AI Tools for Creative Work • Introduce A Creative Problem: Visual Metaphors • Can AI do it? • VisiBlends: A Workflow for Prototyping Visual Metaphors • Symbol Finder: Adding AI to the workflow • VisiFit: AI Tools for Iteratively Improving Visual Blends • The Future of AI and creative work
  7. 7. AdvertisementsNews Public Service Announcements Earth + MeltBrazil + Takes Off Tabasco + Hot Visual Metaphors: image that convey a message symbolically
  8. 8. How can we create visual blends for a message?
  9. 9. Outline: AI Tools for Creative Work • Introduce A Creative Problem: Visual Metaphors • Can AI do it? • VisiBlends: A Workflow for Prototyping Visual Metaphors • Symbol Finder: Adding AI to the workflow • VisiFit: AI Tools for Iteratively Improving Visual Blends • The Future of AI and creative work
  10. 10. With successes in deep learning, what is the future of creative work?
  11. 11. Visual Metaphor: Lego are Summer Vacation Fun Artist blend:
  12. 12. Deep Style Transfer Artist:AI:
  13. 13. Visual Metaphor: Burgers are Healthy, like apples Artist:AI:
  14. 14. Visual Metaphor: Burgers are Healthy, like apples Artist:AI: Blending objects requires more than just blending “style.” You need more context about the objects - how to take them apart, blend the pieces, think about how they’ll be perceived.
  15. 15. Outline: AI Tools for Creative Work • Introduce A Creative Problem: Visual Metaphors • Can AI do it? • VisiBlends: A Workflow for Prototyping Visual Metaphors • Symbol Finder: Adding AI to the workflow • VisiFit: AI Tools for Iteratively Improving Visual Blends • The Future of AI and creative work
  16. 16. But there is no simple pattern for visual blends We need to break up the problem into tasks.
  17. 17. 1. Two objects are integrated into one object 2. Both objects are individually identifiable Visual Blending Structure: Whole-to-Part Shape Matching Abstract problem structure: Single Shape Mapping
  18. 18. Why blend based on shape? Shape is the primary feature the human visual system uses to recognize objects. Colors, silhouette and details and secondary.
  19. 19. 1. Two objects are integrated into one object 2. Both objects are individually identifiable Visual Blending Structure: Whole-to-Part Shape Matchingby using a design pattern to make it a search problem We can solve a creative design problem Earth Melt Visual Blending Structure: Whole-to-Part Shape Matching
  20. 20. Starbucks + Summer
  21. 21. Prototype blend Evaluate prototype Iterate Output: A visual blend Two objects are integrated into one object Both objects are identifiable Prototype blend Evaluate prototype Two objects are integrated into one object Both objects are identifiable
  22. 22. Prototype blend Evaluate prototype Iterate Output: A visual blend Two objects are integrated into one object Both objects are identifiable Prototype blend Evaluate prototype Two objects are integrated into one object Both objects are identifiable
  23. 23. Prototype blend Evaluate prototype Iterate Output: A visual blend Two objects are integrated into one object Both objects are identifiable
  24. 24. Prototype blend Evaluate prototype Iterate Output: A visual blend Two objects are integrated into one object Both objects are identifiable e e Output: A visual blend grated into one object ntifiable
  25. 25. A) Brainstorm for summer B) Find and annotate images for summer C) Matching algorithm Starbucks + summer D) Automatic blends + human evaluation Starbucks + summer matches = [] for a in summer_symbols: for b in starbucks_symbols: a_ratio = a.height / a.width b_ratio = b.height / b.width ratios = sort(a_ratio, b_ratio) if (a.shape == b.shape) and (a.coverage != coverage) and (ratios[0] >= 0.5*ratios[1]): matches.push([a, b]) return matches VisiBlends: A web interface to collaboratively make blends
  26. 26. Blends made by groups at Columbia University
  27. 27. Football + dangerous
  28. 28. Joe’s Coffee + Night
  29. 29. Wash your hands. It’s the smart move.
  30. 30. Forget the knives and forks: the humble hamburger is in the midst of a renaissance. From Melbourne to Milan, restaurateurs have helped elevate this classic piece of American folklore into an iconic dish. Every day, more and more enthusiasts join the search for the perfect
  31. 31. Philosophy + Christmas
  32. 32. Does VisiBlends make people better? • 13 undergraduates (novice designers) • Each made 6 blends (3 with VisiBlends, 3 without) • 5 minutes for each blend • Metric: How many successful blends can 5 minutes?
  33. 33. Novices are 10x better with VisiBlends Control-first Avg. Number of blends made VisiBlends-first Without VisiBlends With VisiBlends Without VisiBlendsWith VisiBlends Avg. Number of blends made 0.56 5.56 (t(18)=4.88, p<0.001) 0.675.67 (t(21)=5.84, p<0.001)
  34. 34. Bicycle + Smart VisiBlendsUsers Unassisted VisiBlends helps them have many ideas, and pick the best. Novices get stuck “improving” their first idea.
  35. 35. New York City + Night Users Unassisted Novices pick “perfect” symbols, but sacrifice shape-fit. VisiBlends helps novices meet all the criteria for a visual blend (especially shape-fit). VisiBlends
  36. 36. Do we want “perfect” symbols for Visual Blends? By meeting more constraints, Visual Metaphors are more novel and interesting
  37. 37. Novices are 10x better with VisiBlends 10x improvement in novices ability VisiBlendsUnassisted Why?
  38. 38. Football + Dangerous
  39. 39. Football Dangerous Brainstorm associations Find Images of objects Annotate shapes Annotate shape coverage Are both objects identifiable? Are two objects integrated into one object? Evaluate Blend Shape covers All of object Shape covers Part of object
  40. 40. Football Dangerous Brainstorm associations Find Images of objects Annotate shapes Annotate shape coverage Are both objects identifiable? Are two objects integrated into one object? Evaluate Blend Shape covers All of object Shape covers Part of object
  41. 41. Collect pieces Combine them Re-combine them Collect pieces Lots of individual pieces under your finger; it’s easy to recombine them.
  42. 42. Fluid Intelligence Crystallized Intelligence Having the resources and tools you need (memory, routing procedures, etc) Having the tools at hand so you can fluidly combine them
  43. 43. Crystallized Intelligence Crystallized Intelligence Fluid Intelligence Fluid Intelligence VisiBlends augments fluid intelligence Collect pieces Combine them Re-combine them Collect pieces
  44. 44. NYC + Healthy
  45. 45. NYC Healthy Brainstorm associations Find Images of objects Annotate shapes Annotate shape coverage Evaluate Blend No shape matches
  46. 46. NYC Healthy Brainstorm associations Find Images of objects Annotate shapes Annotate shape coverage Are both objects identifiable? Are two objects integrated into one object? Evaluate Blend
  47. 47. Orange + Healthy
  48. 48. Orange Healthy Brainstorm associations Find Images of objects Annotate shapes Annotate shape coverage Shape covers Part of object Shape covers All of object Are both objects identifiable? Are two objects integrated into one object? Evaluate Blend
  49. 49. Orange Healthy Brainstorm associations Find Images of objects Annotate shapes Annotate shape coverage Shape covers All of object Shape covers Part of object Evaluate Blend Exercise equipment
  50. 50. Orange Healthy Brainstorm associations Find Images of objects Annotate shapes Annotate shape coverage Shape covers All of object Shape covers Part of object Are both objects identifiable? Are two objects integrated into one object? Evaluate Blend Exercise equipment
  51. 51. Iteration requires fluidity – putting together pieces in new ways. No matches Improve object fit Objects are not identifiable + = + = Within same search space, meet other constraints. Find versions of an object with different color, style, aspect ratio Focus on meeting a specific constraint: Find symbols with a different shape Search in a new subspace Find symbols with a different shape
  52. 52. Crystallized Intelligence Crystallized Intelligence Fluid Intelligence Fluid Intelligence VisiBlends augments fluid intelligence Collect pieces Combine them Re-combine them Collect pieces
  53. 53. Outline: AI Tools for Creative Work • Introduce A Creative Problem: Visual Metaphors • Can AI do it? • VisiBlends: A Workflow for Prototyping Visual Metaphors • Symbol Finder: AI tools for finding symbols • VisiFit: AI Tools for Iteratively Improving Visual Blends • The Future of AI and creative work
  54. 54. Prototype blend Evaluate prototype Iterate Output: A visual blend Two objects are integrated into one object Both objects are identifiable pe e pe Output: A visual blend egrated into one object entifiable
  55. 55. Rules for finding images are complex
  56. 56. Simply searching for abstract words doesn’t return simple, iconic symbols that we can blend.
  57. 57. SymbolFinder uses AI + KBs to find concrete images for an abstract term Savvas Petridis PhD Student word2vec, synonyms, ConceptNet, concreteness classifiers
  58. 58. Prototype blend Evaluate prototype Iterate Output: A visual blend Two objects are integrated into one object Both objects are identifiable pe e pe Output: A visual blend egrated into one object entifiable
  59. 59. Outline: AI Tools for Creative Work • Introduce A Creative Problem: Visual Metaphors • Can AI do it? • VisiBlends: A Workflow for Prototyping Visual Metaphors • Symbol Finder: AI tools for finding symbols • VisiFit: AI Tools for Iteratively Improving Visual Blends • The Future of AI and creative work
  60. 60. How often do VisiBlend prototypes need editing? Output: A visual blend object 4 of 27 were ok to print on social media (according to 2 evaluators who are graphic designers)
  61. 61. How do we improve the rest of them?
  62. 62. Fully automatic approaches fail Artist:AI: Design Principle #1 : Instead of pursuing fully automatic approaches, break up the objects into visual components that can each be blended.
  63. 63. Design Principle #2: When improving a blend, Recombine secondary visual aspects: color, silhouette, details. Co-Design with 2 Graphic Designers
  64. 64. Formative study with novice designer Design Principle #3: Translating intent into action is a barrier for novices. Use AI tools to automate actions or provide more direct mapping between intent and action “I want to make the metro card the same shape as the purse.”
  65. 65. VisiFit: AI Tools to Iteratively Improve Visual Blends Each visual aspect has it’s own AI tool for novices to apply
  66. 66. AI tools to improve each visual dimension Color blending Crop to extracted silhouette Deep Object Segmentation Interactive Object Segmentation Interactive Object Segmentation
  67. 67. Evaluation Improved: 97.5% of cases 11 Novice designers (8 female) 1 hr study 11 blends to extract and improve Publishable quality: 65.3% of the time. Experts Publishable: 87.0% of the time. 121 total blends produced Outputs rated by 2 designers: Improved, publishable.
  68. 68. Non-Publishable blends 1. Grabcut failed to extract details 2. No tool to fill parts of the bottom object. (Add a fill tool)
  69. 69. Response of Professional Visual Blend Designers 1. Amazed by the AI tools They did try things no novice did (even if it didn’t work). 2. Much easier than “pixel pushing”. Enabled them to explore more designs quickly and easily. But did suggest more PhotoShop features to all, like Context Aware Fill Wanted it as a plugin to sketch. Everyone else Designer
  70. 70. Outline: AI Tools for Creative Work • Introduce A Creative Problem: Visual Metaphors • Can AI do it? • VisiBlends: A Workflow for Prototyping Visual Metaphors • Symbol Finder: AI tools for finding symbols • VisiFit: AI Tools for Iteratively Improving Visual Blends • The Future of AI and creative work
  71. 71. Crystallized Intelligence Crystallized Intelligence Fluid Intelligence Fluid Intelligence AI Tools can augment fluid intelligence Collect pieces Combine them Combine them Collect pieces by making it easy to collect and combine pieces
  72. 72. If we identify the key aspects of the problem VisiBlends collects and combines the shape of blended objects VisiFit extracts and combines the color, silhouette, details of the objects AI tools can help collect and combine elements for each aspect
  73. 73. AI Tools for writers: Key aspects of writing: Substance, structure, and style. Metaphoria (CHI 2019): Within the metaphor structure, helps generate substance. Style Thesaurus (C&C 2019): Same substance, new style (word level) Language Style Transfer (INLG 2019): Same substance, new style (sentence level) Katy Gero PhD Candidate
  74. 74. Characteristic motion Reference video Static image Future Work: Animated Blends Use AI to extract key aspects of motion (speed), and reapply them to the blend (use a different path).
  75. 75. Ecenaz Jen Ozmen Columbia CC ‘21 Claire Chen Columbia CC ‘21 Maria Kogan Columbia SEAS ‘21 Sam Ross Barnard ‘21 Annie Sui Columbia CC ‘22 Maneesh Agrawala Stanford Katy Gero PhD Candidate Savvas Petridis PhD Candidate Collaborators
  76. 76. Summary
  77. 77. Future of Creative Work: AI should augment fluid intelligence AI shouldn’t try to understand complexities of culture and working context. AI can provide people the tools and resources to more easily turn users’ intent into action AI Artist Novice with AI Tools
  78. 78. VisiBlends: AI Tools for Prototyping Visual Blends VisiBlendsUnassisted 10x improvement in novices’ ability
  79. 79. VisiFit: AI Tools to Iteratively Improve Visual Blends 97.5% of cases improved, 65% to publishable quality
  80. 80. Crystallized Intelligence Crystallized Intelligence Fluid Intelligence Fluid Intelligence VisiBlends augments fluid intelligence Collect pieces Combine them Combine them Collect pieces by making it easy to collect and combine pieces
  81. 81. Characteristic motion Reference video Static image Future Work: Animated Blends Use AI to extract key aspects of motion, and reapply them to the blend.
  82. 82. Iteratively Improving: one key element at a time Prototyping: collect and recombined parts Brainstorming: easily collect parts AI Tools for Creative Work Repurposing: Elements of a Reference Object

×