03 0122 1000 3 Storytelling


Published on

Published in: Entertainment & Humor
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

03 0122 1000 3 Storytelling

  1. 1. Automatic Storytelling in Comics Sheng-Wei (Kuan-Ta) Chen Inst. of Information Science Academia Sinica
  2. 2. Intrinsic Needs of Human Being! <ul><li>Communication </li></ul><ul><li>Storytelling </li></ul><ul><ul><li>Today’s world is information-saturated </li></ul></ul><ul><ul><li>Computers are becoming a larger and larger part of people’s lives (Intel 2005) </li></ul></ul><ul><ul><li>More and more people are documenting their lives with digital images, video, music, and recordings (Imation 2003) </li></ul></ul><ul><ul><li>As storage space gets cheaper, and people record more and more and more, they need some way to automate the categorization of all of these digital memories </li></ul></ul>
  3. 3. <ul><li>Recall , share and preserve (gaming) experience </li></ul><ul><ul><li>Virtual community such as forums, blogs </li></ul></ul><ul><ul><li>Video, screenshots, comics or other art formats </li></ul></ul><ul><ul><li>Narrative , summary, story </li></ul></ul><ul><li>Search “ Lineage Story ” or “ Lineage Comics ”… </li></ul><ul><ul><li>Around 26,600 files in YouTube and 724,000 items in Google </li></ul></ul><ul><ul><li>http://www.lineage2.com/community/fanart.html </li></ul></ul><ul><ul><li>http://himewikia.blogspot.com/ </li></ul></ul><ul><ul><li>… </li></ul></ul>
  4. 4. <ul><li>Example: </li></ul>
  5. 5. <ul><li>Example: </li></ul>
  6. 6. Examples
  7. 7. More examples
  8. 8. Narrative <ul><li>In short, People Like Stories . Or, put more accurately, People Remember Stories </li></ul><ul><li>So we provide a summarization tool to aid storytelling of end-users </li></ul><ul><li>But why Comics … </li></ul>
  9. 9. Comic <ul><li>A storytelling medium </li></ul><ul><li>Vocabulary of comics </li></ul><ul><ul><li>Panel layout </li></ul></ul><ul><ul><ul><li>Border shape </li></ul></ul></ul><ul><ul><ul><li>Panel size </li></ul></ul></ul><ul><ul><ul><li>Bleed, splash page </li></ul></ul></ul><ul><ul><li>Speech balloon and sound effects </li></ul></ul><ul><ul><li>Motion lines </li></ul></ul><ul><li>Visual language </li></ul><ul><ul><li>Draw attention to important events </li></ul></ul><ul><li>Reference: http://scottmccloud.com/2-print/index.html (McCloud, Scott) </li></ul>
  10. 10. Comic Generation Tools <ul><li>Automation support </li></ul><ul><ul><li>The computer is charged with a greater role in the creation of the comic </li></ul></ul><ul><li>Authoring support </li></ul><ul><ul><li>Tools offer the most agency in creating panels and layouts of any size and shape </li></ul></ul><ul><li>Reference: http://dm.lcc.gatech.edu/~jalderman/comics/ </li></ul>
  11. 11. <ul><li>Comic Chat (1996) </li></ul><ul><li>Comic Chat visually represents </li></ul><ul><li>conversations as sequences of </li></ul><ul><li>comic panels. </li></ul><ul><li>Automation support: </li></ul><ul><li>Changing posture and facial expression of users’ avatars in chat room </li></ul><ul><li>Placing word balloons in natural reading order </li></ul><ul><li>Zooming “camera” to frame the people talking </li></ul><ul><li>Parsing text into panels procedurally. </li></ul><ul><li>Panels are all in the same size </li></ul><ul><li>Reference: http://kurlander.net/DJ/Projects/ComicChat/resources.html </li></ul>
  12. 12. <ul><li>Microsoft Chat 2.5 </li></ul>Chat room: Massage Authoring support: Emotion Wheel allows users to choose the facial expression and body language Comic mode
  13. 13. <ul><li>Microsoft Chat Character Editor </li></ul><ul><li>Character Icon </li></ul><ul><li>Character Contents </li></ul><ul><li>New Pose </li></ul><ul><li>New Pose </li></ul>
  14. 14. <ul><li>Example: </li></ul>
  15. 15. <ul><li>Video Manga (1999-2003) </li></ul><ul><li>It provides a summary of video clips </li></ul><ul><li>that allows users to quickly browse a long </li></ul><ul><li>video in comic book format . </li></ul><ul><li>Manga is named after a Japanese word for &quot;comic book&quot; </li></ul><ul><li>A video is represented with different-sized keyframes that packed in a comic book format </li></ul><ul><li>Video Manga allows users to quickly browse a long video </li></ul><ul><li>The visual summaries support printing function </li></ul><ul><li>Reference: </li></ul><ul><li>http://www.fxpal.com/?p=manga </li></ul>
  16. 16. <ul><li>Automation support: </li></ul><ul><li>A video is segmented based on the color features of each frame </li></ul><ul><li>The segments are clustered according to their similarities </li></ul><ul><li>We have introduced an importance score to rank the segments </li></ul><ul><li>A segment is considered to be important if it is long and rare </li></ul><ul><li>The keyframes are extracted from highly ranked segments </li></ul><ul><li>The keyframes are size d according to their scores </li></ul><ul><li>The frame-packing algorithm puts the keyframes in a comic book format </li></ul><ul><li>Authoring support: </li></ul><ul><li>Handling captions to its frames manually typed in by users </li></ul>
  17. 17. <ul><li>Video Manga : Summary of staff meeting </li></ul>
  18. 18. <ul><li>Generating Comics From </li></ul><ul><li>3D Interactive Computer </li></ul><ul><li>Graphics (2006) </li></ul><ul><li>Automation support: </li></ul><ul><li>Recognizing scenes </li></ul><ul><li>Converting to visual depiction </li></ul><ul><li>Comic layout </li></ul><ul><li>Image processing </li></ul><ul><li>Reference: http://ieeexplore.ieee.org/xpl/freeabs_all.jsp?arnumber=1626186 </li></ul>
  19. 19. <ul><li>Recognizing scenes </li></ul><ul><ul><li>Time and space </li></ul></ul><ul><ul><li>Separating scenes when </li></ul></ul><ul><ul><ul><li>The interaction level is low </li></ul></ul></ul><ul><ul><ul><li>The location changes </li></ul></ul></ul><ul><li>Converting to visual depiction </li></ul><ul><ul><li>Transforming each scene into a sequence of images depicting the main event </li></ul></ul><ul><ul><li>Using idioms to depict interactions </li></ul></ul><ul><ul><ul><li>Change of scene </li></ul></ul></ul><ul><ul><ul><li>Shooting </li></ul></ul></ul><ul><ul><ul><li>conversation </li></ul></ul></ul><ul><li>Automation support </li></ul>
  20. 20. <ul><li>Comic layout </li></ul><ul><ul><li>B (Big): peak interaction images and change-of-scene </li></ul></ul><ul><ul><li>S (small): action-to-action pair or FP viewpoint shots </li></ul></ul><ul><ul><li>F (fixed): images with speech balloon </li></ul></ul><ul><ul><li>N (neutral): all others </li></ul></ul><ul><li>Image processing </li></ul><ul><ul><li>Mean shift </li></ul></ul><ul><ul><li>Laplacian </li></ul></ul><ul><ul><li>K-means </li></ul></ul>
  21. 21. <ul><li>Example: </li></ul>
  22. 22. <ul><li>Comic Life (2005) </li></ul><ul><li>Application for creating comics: </li></ul><ul><li>loading images into comic panels </li></ul><ul><li>Authoring support: </li></ul><ul><li>Simple and Easy-to-use interface </li></ul><ul><ul><li>Drag in your pictures, captions, sound effects (‘ka-blam!') and speech balloons and it’s done! </li></ul></ul><ul><ul><li>Layout templates </li></ul></ul><ul><ul><li>Filters like hand-drawn, painted and night vision </li></ul></ul><ul><li>More freedom to create comic but still some restrictions </li></ul><ul><li>Reference: </li></ul><ul><li>http://plasq.com/comiclife-win </li></ul>
  23. 23. <ul><li>Comic Life 1.3 </li></ul>
  24. 24. <ul><li>Example: </li></ul>
  25. 25. <ul><li>Manga Studio </li></ul><ul><li>Application for creating comics: </li></ul><ul><li>drawing, laying out comic and </li></ul><ul><li>styling comics </li></ul><ul><li>Authoring support: </li></ul><ul><li>High versatility and flexibility to create comics </li></ul><ul><ul><li>A large collection of special effects </li></ul></ul><ul><ul><li>Sketch, ink, and color artworks without the need for any other graphics applications </li></ul></ul><ul><li>Interface similar to Photoshop </li></ul><ul><li>Reference: http://my.smithmicro.com/mac/manga/index.html </li></ul>
  26. 26. Our Proposal <ul><li>Automation support </li></ul><ul><ul><li>Frame selection </li></ul></ul><ul><ul><li>Layout computing </li></ul></ul><ul><li>Authoring support </li></ul><ul><ul><li>Comic layout </li></ul></ul><ul><ul><li>Graphic Effects </li></ul></ul><ul><ul><li>Internet support </li></ul></ul><ul><li>User interface </li></ul>
  27. 27. <ul><li>Comic Gene (2009) </li></ul><ul><li>It automatically summarize players’ </li></ul><ul><li>interactions ( logs and screenshots ) </li></ul><ul><li>in virtual world into comics . </li></ul><ul><li>Automation support: </li></ul><ul><li>Recording log and shots (Addon) </li></ul><ul><li>Frame selection </li></ul><ul><li>Comic layout computing </li></ul><ul><li>Rendering comics </li></ul><ul><li>Authoring support: </li></ul><ul><li>Editing log files and screenshots </li></ul><ul><li>Comic layout </li></ul><ul><ul><li>Option setting </li></ul></ul><ul><ul><li>Changing size and position of the shots </li></ul></ul><ul><li>Reference: http://mmnet.iis.sinica.edu.tw/publication_detail.html?key=chan09_comics </li></ul>
  28. 28. … How to improve! <ul><li>Prototype </li></ul>
  29. 29. 1. Automation Support <ul><li>Automatically summarize logs and frames into comics </li></ul><ul><ul><li>Log: events </li></ul></ul><ul><ul><li>Frame: video, photos, screenshot s and so on </li></ul></ul><ul><li>Definition of important events </li></ul><ul><ul><li>Frequency, weight </li></ul></ul><ul><ul><li>Significance score </li></ul></ul>
  30. 30. Challenges <ul><li>Technical side </li></ul><ul><ul><li>Choose significant frames using machine learning </li></ul></ul><ul><ul><li>Image understanding </li></ul></ul><ul><ul><li>Special effect placement (font size, color, location) </li></ul></ul><ul><ul><ul><li>Chat balloons </li></ul></ul></ul><ul><ul><ul><li>Onomatopoeic sounds </li></ul></ul></ul><ul><li>Design side </li></ul><ul><ul><li>Comic-like style </li></ul></ul><ul><ul><ul><li>Page layout </li></ul></ul></ul><ul><ul><li>Essential factors of good comic books </li></ul></ul><ul><ul><ul><li>Shooting </li></ul></ul></ul><ul><ul><ul><li>Zoom in/out </li></ul></ul></ul><ul><ul><ul><li>Level of detail </li></ul></ul></ul><ul><ul><ul><li>Timing control </li></ul></ul></ul>
  31. 31. <ul><li>Frame selection </li></ul><ul><ul><li>How many images does a comic strip need? </li></ul></ul><ul><ul><li>Is the criterion of significance score enough? </li></ul></ul><ul><li>Layout computing </li></ul><ul><ul><li>Algorithm </li></ul></ul><ul><ul><li>Detection </li></ul></ul>
  32. 32. 2. Authoring Support <ul><li>Page properties </li></ul><ul><ul><li>How many images in a comic/page/row/… ? </li></ul></ul><ul><li>Important event/frames </li></ul><ul><li>Comic layout </li></ul><ul><ul><li>Panel layout </li></ul></ul><ul><ul><ul><li>Border shape , bleed </li></ul></ul></ul><ul><ul><ul><li>Panel size </li></ul></ul></ul><ul><ul><li>Speech balloon </li></ul></ul><ul><ul><li>Sound effects </li></ul></ul>
  33. 33. <ul><li>Graphic effects </li></ul><ul><ul><li>Templates </li></ul></ul><ul><ul><li>Cartoon-like design </li></ul></ul><ul><ul><ul><li>Cartoon-like filter </li></ul></ul></ul><ul><ul><ul><li>Motion line </li></ul></ul></ul><ul><li>Internet support </li></ul><ul><ul><li>Uploading comics to individual album or blog </li></ul></ul>
  34. 34. 3. User Interface Design <ul><li>User-friendly Features </li></ul><ul><ul><li>Navigation bar </li></ul></ul><ul><ul><li>Easy-to-use </li></ul></ul><ul><ul><ul><li>Click, wheel </li></ul></ul></ul><ul><ul><ul><li>Drag-n-drop </li></ul></ul></ul>
  35. 35. UI Example: Videolyzer Navigation bar
  36. 36. UI Example: Comic Life Drag-n-drop
  37. 37. Current Prototype
  38. 38. Contact info Sheng-Wei (Kuan-Ta) Chen [email_address] http://www.iis.sinica.edu.tw/~swc
  39. 39. Thank you