The document describes a project that uses artificial intelligence and augmented reality to visualize stories from text. It extracts information like background, characters, and actions from paragraphs using BERT models, generates 3D scenes in Unity, and visualizes them in augmented reality using Vuforia. The Android app allows users to take photos of text, save and delete scenes, change scene elements like scale and background, and view the augmented reality visualization.

1. STORY VISUALIZATION USING
DEEP LEARNING
Members:
Hassan Latif 2017152
Adil Jamal 2017228
Rehana Saboor 2017387
Saad Iqbal 2017395
Supervisor:
Mr. Badre Munir
Co-supervisor:
Dr. Muhammad Hanif

2. Motivation
 Creating images and videos from text is a challenging task even for modern-day AI.
 Usually the images and videos are created pixel by pixel and this approach lacks clarity in
images/videos and also the duration of videos is too short.
 To improve upon the generation of videos from text, we decided to use artificial intelligence and
augmented reality to create 3D scenes of story books.
 Existing Systems:
There are a number of storytelling apps developed using augmented reality and virtual reality:
 Quiver Augmented Reality
 Website: https://quivervision.com/
 Wonderscope
 Website: https://wonderscope.com/

3. Introduction
 Our project implements an Android application that uses Artificial Intelligence and
Augmented Reality to visualize a story.
 We visualize some paragraphs of descriptive comprehensions, primarily stories in 3D
geometric comprehensions.
 Each scene contains background information and, characters and their actions.

4. Description
 An image is captured, containing a few paragraphs of a storybook.
The text is extracted from the image which is then fed to BERT
models.
 BERT models are natural language processing models that extract
scene-specific information i.e. the background, the characters,
and the actions of these characters present in the scene.
 This information is then sent to Unity, which generates 3D
backgrounds and characters.
 All of this information is further passed on to Vuforia for
visualization in Augmented Reality.

5. 3
Run inference and create scene
1
Combine background, character
and actions.
2
Database
Take a picture
Project Flow
Background
Character
Action

6. Technologies
 Our project is an Android application so we have utilized Android
development tools.
 TensorFlow is used to train BERT models.
 Characters and animations were downloaded from mixamo.com.
 Unity is used to generate 3D scenes.
 Vuforia is used to visualize the scenes in AR.
 Flask and Firebase are used to implement the backend.

7. Functional Requirements
 User Sign Up.
 User Log In.
 Select an image from stored images in file manager or capture a new image.
 Save the scene (the text of the scene).
 Delete saved scene.
 Choose different voices to narrate the text of the scene.
 Visualize the scene on a surface.
 Change scale of scene.
 Change characters in the scene (if any).
 Change background of the scene.
 Delete saved scene (text description of the scene).
 Logout from the application.