The document discusses a project using AI to generate pictographic symbols for augmentative and alternative communication (AAC). It aimed to represent concepts that existing symbol sets could not depict, using generative models trained on a small set of visual descriptions. While initial results showed symbols could be created, challenges included lack of consistency. Future plans include expanding the models to improve customization and localization of symbols to support diverse audiences.

1. AI supportingAAC Pictographic Symbol
Adaptations
E.A. Draffan, Mike Wald, Chaohai Ding and
Yuanyuan Yin
AAATE 2023

2. 2
Project Overview
• A 6 month Web Science Institute 2022-
23 Stimulus Fund Artificial Intelligence
(AI) grant with knowledge exchange
between faculties and external partners
• Aimed to provide pictographic symbols
designed for Augmentative and
Alternative forms of Communication
(AAC)
• Using generative AI images that would
be similar to those seen or used in the
past.
• Needed to represent concepts that
hitherto they had been unable to find in
known symbol sets.
ARASAAC, Mulberry,
Sclera and Bliss Symbols
External Partner -
Global Symbols

3. 3
Initial Discussions
• Open licence critical for use of a symbol set and agreement with
the owner
• Consistency of design checks within the chosen symbol set
• Choice of categories that would form the initial training data
• Testing ability to make good text prompts that would produce
symbol forms of a concept
• A simple text prompt can produce relatively acceptable symbols
e.g. ‘a five-pointed pink star symbol’ compared to a symbol in the
Mulberry set (last star).

4. 4
Methodology
• Provision of visual text descriptions for 100 frequently used symbols
representing commonly used concepts in every day speech and
language to assist image to image training
• Trialling text prompts without image training to compare to Mulberry
symbols
• Limited training data available so Stable Diffusion alongside
DreamBooth used as a “Text-to-Image Diffusion Model for Subject-
Driven Generation

5. 5
Challenges
Stable Diffusion - "The input image on the left can produce
several new images (on the right). This new model can be used
for structure-preserving image-to-image and shape-conditional
image synthesis." https://stability.ai/blog/stable-diffusion-v2-
release Lack of consistency – always innovating!

6. 6
MulberryAI generated Symbol
Adapted symbol for ‘teacher’ based
on the category for professions
“Copyright 2018/19 Steve Lee - This work is licensed under the Creative Commons
Attribution-ShareAlike 2.0 UK: England & Wales License.”
https://mulberrysymbols.org/

7. 7
Results
• It is possible to create similarly
styled pictographic symbols from
very little training data (max
available in a similar style was 165
symbols).
• The ease with which symbols could
be created with small additional
tweaks.
• Globalisation of symbols to suit
different cultures, environments,
social settings and languages.
• Potential to personalise AI generated
symbols

8. 8
Future Plans
• Taking the idea of using generative AI to transform any globally
available open licenced pictographic symbols into different
concepts to better support localisation as well as personalisation
where different images are required.
• Improving consistency of results with increased flexibility for the
type of symbol required such as action, abstract and combination
concepts taking account of original style constraints.
• Expanding image to image, text to image and image to text
transformations to assist visual searches, customised information
icon development to help a wider audience including those with
sensory and cognitive impairments.

9. Thank you
https://access.ecs.soton.ac.uk/ collaboration with
Global Symbols
https://globalsymbols.com/news?locale=en
AAATE 2023
A Web Science
Institute Stimulus
Fund 2022-23
Project