Highcharts is a world leading provider of accessible charting tools for the web, used by 80 of the top 100 Fortune companies. Recently Highcharts and global publishing company Elsevier’s Digital Accessibility Team collaborated to provide better accessibility for line charts with large datasets. Line charts are often used to visualize datasets with thousands of data points. This presents a challenge for non-visual access, as providing access to individual data points is not sufficient. A reader of a line chart with a large amount of data will aim to extract information about trends, patterns, and outliers from the chart. Can we make this information more accessible by communicating it through text and sound? What is the most intuitive way to experience this data through sound? And to which extent can we automate the text description? Human authored text descriptions of charts are historically difficult to beat, but can in many cases be impractical – such as where data is dynamically loaded in real-time. Automated text descriptions can also be designed to be more objective and less prone to biases. Will users be able to trust these descriptions? Will they still prefer those created by a human? With each of the new accessibility research questions we will provide user feedback from non-sighted users on our approaches. We will share findings about best practices, and show screen reader demos to help illustrate design considerations.

1. Making Sense of
Large Data
Visualizations
Ted Gies
Øystein Moseng
CSUN 2023
Tuesday, March 14, 2023 - 3:20 PM PST
ELITE 1-3
Customized icon/illustration
or, an image of Chart / Photo

2. Sharing our Research Today
Why accessible data visualizations?
Why automated line descriptions?
Highcharts accessibility journey
State of the art
Our Approach
Demos
User Feedback
What’s Next?

3. Highcharts Overview

4. Elsevier’s Mission
Elsevier helps researchers and
healthcare professionals advance
science and improve health outcomes
for the benefit of society.
4

5. Data Visualizations are the foundation to Decision Making in Science and Health
Weekly Covid cases in U.S.: Source: CDC.gov

6. Data visualizations are the foundation to decision making in finance

7. Why Accessible Data Visualizations?
Journal Paper: COVID-19 Highlights the Issues Facing Blind and Visually
Impaired People in Accessing Data on the Web
(Alexa F. Siu, Stanford U)
“Participants described how at the early onset and most critical time of the
pandemic, they encountered broad inaccessibility, particularly of spatial
graphics, trends, and statistics.”
93% of survey respondents agreed they have concerns about accessing
accurate COVID-19 data and related graphics in a timely manner

8. “There are probably ways to look for the
trend, but I never look because I expect it to
be inaccessible.”
Stanford COVID-19 Viz study – Feedback from Person who is Visually impaired

9. What is the state of the art?

10. Microsoft PowerPoint Automated Descriptions
Chart, line chart Chart Timeline
Graphical user interface, application, Word Chart, waterfall chart Chart, pie chart
Chart, scatter chart Chart, scatter chart, bubble chart Chart, radar chart

11. Georgia Tech – Graphics ingestion engine
The use case is to "reverse engineer" existing graphs (there are millions of graphs in a static
image format like PNG) into a data set that can be used to produce new and more
accessible graphs with digital tools such as the Highcharts Sonification Studio (HSS).
We have built the following pieces:
1. SORTER: A machine learning model that pre-classifies incoming graphs as Bar, Line, or
Pie Charts; or Scatter Plots. Accuracy is already in the 95% range, so we will generally
send the graph through the right pipeline.
2. GRAPH INGESTION ENGINE: For each graph type, we have a multi-step parser that
locates axes, tick marks, labels, and data points. Then it recognizes the characters/text
and determines the axis values. Then it generates a set of x,y data points to recreate the
graph's data values. Works for Scatter, Line, Bar, and Pie (not all equally well, yet)
3. These data can be stored in a CSV file, or in a more complex files like Highcharts
Sonification Studio Project (HSSP) files, which includes labels, colors, and other
rendering information.

12. State of the Art Model of Chart Descriptions
MIT CSAIL (model of structure, navigation, and description)
4 Level Model of Accessible Viz Content
Model by Alan Lundgard and Daniel Hajas

13. ChatGPT (TBD)

14. 8 Years Ago, We Embarked on a Quest!

15. Quick Demo of Structure and Data Point

16. Our Approach to Making Trends Accessible
1. Created natural language description of major and trends and points of interest
2. Provided interactive sonification user experience along side the description
3. Provided a smoothing feature to simplify/smooth large data sets
4. Provided the ability to select a date range to zoom in and out on graphs

17. Natural Language Description
• Description of overall trend (rise or drop)
• Comparison of data trends between multiple lines
• Relative overall increase (comparing 3 lines)
• Chart type, structure
• Min/Max
• Major points of change

18. Major points of change
Price starts at $1,183, at Apr 16, 2017.
Then rises to $10,604 at Oct 6, 2020.
Then rises sharply to $57,540 at Feb 21, 2021.
Then drops sharply to $29,807 at Jul 20, 2021.
Then rises sharply to $65,467 at Nov 14, 2021.
Then drops sharply to $19,018 at Jun 18, 2022.
Then finally rises to $24,869 at Feb 16, 2023, which is higher than where it
started.
Overall, the maximum value is $67,567, at Nov 8, 2021. The minimum is $1,183,
at Apr 16, 2017

19. We gathered
feedback from 6
people who rely on
screen readers and
are blind
•
•
•
•
•
•
•

20. Key User Testing Topics
Does text description and sonification help in visualizing the data?
How can we improve the designs?
How is the verbosity of the descriptions?
Are there any trust issues with this approach?
Single line series (Historical Price of Bitcoin)
3-line series (Unemployment Rates)

21. Text Descriptions Overall User Feedback
Overall, the description is great!
That’s beautiful, higher than when it started
This is one of the best ways I’ve seen it
presented.
The list is a little long, I’d prefer sonification
I’d like more control over the summary and
ability to zoom in on time periods, query dates
Ability to answer specific research questions,
which we won’t know Price starts at $1,183, at Apr 16, 2017.
Then finally rises to $24,869 at Feb 16, 2023, which is
higher than where it started.

22. Trend Description User Discussion
• What is the sweet spot for verbosity? “A little annoying to go through it all”.
• How much text description is too much in the context of education settings?
(e.g., doing the work for someone)
• “Almost like an analysis done for you”
• Want the ability to expand/collapse more detailed views (Min, Max)
• Describe the data points of changes or the describe segments of the change?
• The idea of describing shapes of data not deemed useful to people who were
born without sight, e.g. I don’t know what a letter “m” looks like.

23. Interactive Sonification
Play All
Explore Mode (arrow keys)
Compare between 3 lines
Adjust with chart smoothing
Announce data points when
exploring

24. Does the description and sonification help you
to visualize the data?
“Oh yeah, I know if they lumpy?”
“I like the broken-down trends and numerical data combined with
sonification. The description provides context for the high pitches
has context. Having numerical data in audio makes it more
useful.”
“The text description and sonification pair well together to provide
a quick overview of data, faster than reading through 2K table
rows.”

25. Additional User Quotes on Sonification
“This is pretty cool honestly”.
“The chart smoothing could be useful if I want to go through the
data more quickly”.
“I could develop the capacity to hear sounds as data and will learn
it and get better”.

26. Other Sonification Feedback
Useful for determining which inflation rate is more turbulent.
Most users regarded the full detail as best sonification, better
overall picture of the shape.
Screen reader content announced content in parallel with
sonification?
Consider removing the footsteps sounds to denote walking through
time.

27. Roadmap Items
Text Description
Identify Points of Interest
• Huge peaks
• Lines with several peaks
• Lines crossing
• Outliers
• When dropping below 0
Sonification
• Keyboard shortcuts
• Max/Mins
• Customization
• Duration
• Sounds
• Parallel Play of Series

28. Conclusions
• Tabular Data, Human Created Descriptions, Sonification, and
Tactile all of their place in helping make for more accessible
data visualizations.
• Providing automated text description together with sonification
can help end users better visualize data sets and spot trends.
• Some level of automated description is in development and will
provide a new lens to consume accessible charts and graphs.
• MIT’s Level 4, domain specific insights, current events, social
and political context, explanations. This is a future goal of Tech.

29. Thank you to our users/collaborators
Arvind Satyanarayan, MIT CSAIL
Jonathan Zong, MIT CSAIL
Daniel Hajas, Global Disability Innovation Hub
Thomas Watkins, 3Leaf
Frank Elavsky, Carnegie Mellon
Bruce Walker, Georgia Tech
Brandon Biggs, Smith-Kettlewell Eye Research
Mike Staffen, U of Regina
Lucy Greco, UC Berkeley
Dr. Jon Gardner, ViewPlus
Ryan Shugart, Microsoft
Dr. Greg Williams, Tactile Solutions, Inc

30. EXTRAS
Ted Gies
Øystein Moseng
www.linkedin.com/in/tedgies
www.linkedin.com/in/oysteinmoseng

31. Relevant Papers
Accessible Visualization via Natural Language Descriptions: A Four-Level Model of Semantic Content (IEEE Transactions on Visualization and Computer
Graphics, MIT CSAIL, 2022)
COVID-19 Highlights the Issues Facing Blind and Visually Impaired People in Accessing Data on the Web (Stanford HCI Group, 2021)
Making data visualization more accessible for blind and low-vision individuals (MIT, 2022)
ChartVi: Charts summarizer for visually impaired (J of Computer Languages, International Institute of Information Technology, India 2022)
Towards Automatic Interpretation of Statistical Graphs for the Visually Impaired (Conf in SW Engineering Research and Innovation, Universidad
Veracruzana, 2021)
What Will You Tell Me About the Chart? – Automated Description of Charts (Conference on Neural Information Processing, Warsaw U of Tech, 2021)
Figure Captioning with Relation Maps for Reasoning (IEEE Applications of Computer Vision, Adobe Research Team, 2020)
Chart-Text: A Fully Automated Chart Image Descriptor (Spi Technologies India Private Limited 2018)
The Automated Understanding of Simple Bar Charts (Artificial Intelligence, Millersville University, 2011)
Automated interpretation and accessible presentation of technical diagrams for blind people (Universität Bremen, Germany 2004)

32. Current Focus on Large Viz and Automated Trends for Line Plots
Available in Highcharts today
• Well-structured data series
• Well labelled SVG data points
• Table Version
• Human Text Description
• Sonification
Current Focus/Research
• Computer Generated Descriptions
• Line simplification
• Interactive sonification experience
• Braille Display/Tactile Tablet

33. A Note About Tactile – making comparisons
Multiline Tactile Displays are coming
Tactile provides rich freedom of exploration
Challenges with smoothing out spiky data lines

34. Drill Down Framework
1. Start with a broad, short description
2. Drill down into individual lines
3. Allow users to drill down into semantic layers
(stats, changes in direction, slices of the data to explore)
4. Allow users to drill down into specific data
Allow users to make comparisons

35. Previous CSUN Sessions
VPATs for Business or Measure
Accessible SVG charts using ARIA
Elsevier: Article of the future collaboration
Accessibility for Large Publishers: Challenges,
Choices, Change
Alt Text – A Process of Discovery
Don’t Play Me – 2 Games in Web Accessibility
Elsevier Company Accessibility Policy

36. Highcharts Overview

37. Infant morality rates and 10 leading causes of death: Source: CDC.gov
Data Visualizations are the foundation to Policy Making in Science and Health

38. Next Steps: What Can Be Automated vs. Not Yet
EASY
Today We Can Automate
• Chart type
• Straight/Curved
• Steep/Flat
• Max/Min
• Peaks/Valleys
• Outliers
• Missing Data Points
MEDIUM
Tomorrow
• Descriptive Shape
• Comparisons
• “Points of Interest”
• When lines intersect
DIFFICULT
Tomorrow
• Descriptive Shape
• Comparisons
• “Points of Interest”
• When lines intersect

39. ChatGPT
Free AI Engine to respond to detailed text queries

40. Sharing our
research
Why accessible data visualizations?
Our Highcharts accessibility journey
Why automated line descriptions?
What is current state of the art?
Our Approach
Demos
User Feedback
What’s Next?

41. Highcharts, Elsevier References
Highcharts accessibility demos
Highcharts accessibility portal
Highcharts accessibility module
Sonification API
Highcharts Features History
YouTube: Highsoft and Elsevier Talk About
Accessibility 2020
Understanding Screen-Reader Users’ Experiences
with Online Data Visualizations (Sharif, et al)
Contact Us
Ted Gies <ted.gies@elsevier.com>
Øystein Moseng <oystein@highsoft.com>