@article{Harper2014aa, Abstract = {Visually disabled people typically use methods of `sensory translation' to access data via assistive technology. These technologies conventionally render content under the direction of the user into a form that can be perceived by that user -- in effect the interface and content are adapted to suit their sensory requirements -- but simple sensory translation is not enough for big, broad and complex data. Why is this -- and how can things be better?}, Author = {Simon Harper}, Date-Added = {2014-05-27 13:03:23 +0000}, Date-Modified = {2014-05-27 13:03:34 +0000}, Doi = {http://dx.doi.org/10.6084/m9.figshare.1037547}, Howpublished = {Slideshare}, Journal = {Invited Talk - Human Behaviour Network, Manchester Informatics, Manchester UK}, Month = {May}, Title = {Accessibility of Big and Broad Data - http://goo.gl/UpekPK}, Url = {http://www.slideshare.net/simon-harper/accessibility-of-big-broad-data}, Year = {2014}, Bdsk-Url-1 = {http://www.slideshare.net/simon-harper/accessibility-of-big-broad-data}, Bdsk-Url-2 = {http://dx.doi.org/10.6084/m9.figshare.1037547}}

1. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Accessibility of Big
& Broad Data
Simon Harper
University of Manchester
http://goo.gl/UpekPK
@sharpic
simon.harper@manchester.ac.uk
27 May, 2014
Accessibility of Big & Broad Data 1 / 38

2. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Thinking; when putting this together
1. Applications often designed to conform to a theoretical user,
context, and interactions – autobiographical design;
2. techniques evolved to correct these theoretical
misconceptions when reality interfered (such as in
accessibility / Assistive Technology);
3. extended to include ideas of physiology and cognition for
enhanced use;
4. ‘Big & Broad Data’ is complex, and consumption
characteristics are not know at the time of data production;
and
5. can our knowledge of ‘reality’ help?
Accessibility of Big & Broad Data What? Why? 2 / 38

3. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Adaptation
Visually disabled people typically use methods of ‘sensory
translation’ to access data via assistive technology. These
technologies conventionally render content under the direction of
the user into a form that can be perceived by that user – in effect
the interface and content are adapted to suit their sensory
requirements – but simple sensory translation is not enough for
big, broad and complex data.
Why is this – and how can things be better?
Accessibility of Big & Broad Data What? Why? 3 / 38

4. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Adaptation
Move expertise about a user from the developer to the user
(or at least the tools that user actually uses).
How to get the computer to understand the ‘stuff’;
how to automatically split this stuff up; and
how to present this split-up-stuff back to users?
Accessibility of Big & Broad Data What? Why? 3 / 38

5. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Why Might this be More Generally Important?
Removing autobiographical design increases flexibility and
future proofing;
support mobile, small screen real estate (audio maybe more
natural);
supports distributed attention;
augments complex visual data; and
remove reductionism / simplification
Accessibility of Big & Broad Data What? Why? 4 / 38

6. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Accessibility of Big & Broad Data Adaptation/Transcoding 5 / 38

7. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Accessibility of Big & Broad Data Adaptation/Transcoding 6 / 38

8. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Accessibility of Big & Broad Data Adaptation/Transcoding 7 / 38

9. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Accessibility of Big & Broad Data Adaptation/Transcoding 8 / 38

10. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Accessibility of Big & Broad Data Adaptation/Transcoding 9 / 38

11. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Transcoding / Adaptation
“A category of technologies to transform inaccessible content
to accessible content on the fly”
To Accomplish
Text Magnification;
Colour Scheme
Changes;
Serialisation;
Text Insertion;
Page Rearrangement;
and
Simplification.
Approaches
Syntactic: such as removing images;
Semantic: rearrangements and
fragmentation;
Annotation: created by a reader; and
Generated: annotations by CMS.
Accessibility of Big & Broad Data Adaptation/Transcoding 10 / 38

12. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Content Driven Transcoding
Screen-Scraping - 1990’s
Originally content was ‘made accessible’ via a method called
screen-scraping which used the visual rendering to create
accessible content, by creating an off-screen model (a
representation of the GUI). Screen-scraping was problematic
because it was often wrong, did not take account of structure,
and could not form accurate semantics between elements of the
content.
DOM Analysis - 2000’s
Document Object Model analysis took over from screen-scraping
and enables accurate structural semantics to be created as all
elements and attribute values are available to the assitive
technology.
Accessibility of Big & Broad Data Adaptation/Transcoding 11 / 38
Harper and Yesilada (2008)

13. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Page Annotation - 2000 - Doesn’t Scale
Before
After
Asakawa and Lewis (1998)
Accessibility of Big & Broad Data Adaptation/Transcoding 12 / 38

14. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Style Annotation - 2008 - Scalable
Harper and Bechhofer (2007)
Accessibility of Big & Broad Data Adaptation/Transcoding 13 / 38

15. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Annotate 1 Style File Scales to Many HTML Files
. cnnCeilnav , d i v . cnnHeaderNav
{uom−s t r u c t u r a l −r o l e : L i n k L i s t ; }
i n p u t . c n n I n p u t
{uom−s t r u c t u r a l −r o l e : SearchEngine ; }
d i v . CNNhomeBox , o l . cnnMostPopular , d i v#cnnTopStories . . .
{uom−s t r u c t u r a l −r o l e : Chunk ; }
d i v#cnnHeaderRightCol u l
{uom−s t r u c t u r a l −r o l e : PageSummary ; }
Accessibility of Big & Broad Data Adaptation/Transcoding 14 / 38

16. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Code Driven Transcoding - 2011
Lunn and Harper (2011); Chen et al. (2012)
Accessibility of Big & Broad Data Adaptation/Transcoding 15 / 38

17. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Beyond Accessibility - Readability - 2011
Before
After
Accessibility of Big & Broad Data Adaptation/Transcoding 16 / 38

18. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Beyond Accessibility - Evernote Clearly - 2012
Before
After
Accessibility of Big & Broad Data Adaptation/Transcoding 17 / 38

19. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Experience Driven Transcoding
‘Content and Code Driven
Transcoding’ is focused on
transforming the computer code
based on its representation; however
‘Experience Driven Transcoding’
goes a step further and attempts to
transform the content/code based
on both its representation, and the
predicted experience of the user into
an equivalent sensory experience.
BBC News with AoI’s
Accessibility of Big & Broad Data Adaptation/Transcoding 18 / 38

20. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Example – Graph
Ranked Order Comparison
Accessibility of Big & Broad Data Adaptation/Transcoding 19 / 38

21. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Interacting with Calendars
(Brown et al., 2012)
Accessibility of Big & Broad Data Adaptation/Transcoding 20 / 38

22. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Beyond Accessibility (again) - 2011
Visual Complexity Rankings & Visual Aesthetics Rankings
Heat Map of Visual Complexity – Harper et al. (2009)
Accessibility of Big & Broad Data Adaptation/Transcoding 21 / 38

23. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Big & Broad Data
Data sets so large and complex that it becomes difficult to
process;
data is too big, moves too fast, or doesn’t fit the strictures
of your database architectures;
broad Data is the huge amount of freely available, but widely
varied, Open Data on the World Wide Web (Structured and
Semi-structured);
often found in broad Data Mash-ups;
to gain value from this data, you must choose an alternative
way to process it...
and Visualise / interact with it.
Accessibility of Big & Broad Data Big & Broad Data 22 / 38
Hendler (2012); Dumbill (2012); White (2014)

24. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Accessibility of Big & Broad Data Big & Broad Data 23 / 38

25. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Accessibility of Big & Broad Data Big & Broad Data 24 / 38

26. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Big Simplification
Tendency to simplify and aggregate;
moving complex data to info-graphics and visualisations;
does this really increase our understanding?
Accessibility of Big & Broad Data Big & Broad Data 25 / 38

27. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Big Simplification
Wattenberg et al. (2007a)
The first three letters of a string determine colour in a Chromogram.
The first letter determines the hue; the second letter the saturation,
and the third the brightness. Many Wikipedians engage in systematic
activities: that is, a sustained related sequence of edits – Wattenberg
et al. (2007b).
Accessibility of Big & Broad Data Big & Broad Data 25 / 38

28. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Auditory Display
Ferres et al. (2013)
Accessibility of Big & Broad Data Big & Broad Data 26 / 38

29. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
More Complex Auditory Display
Guardian Group (2010)
Accessibility of Big & Broad Data Big & Broad Data 27 / 38

30. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Sonification?
Difficult to Understand - mostly Aesthetic
Listen to this...
http://geant3.archive.geant.net/Media˙Centre/Media˙
Library/Media%20Library/Higgs˙Boson˙Atlas˙Piano˙Solo.
mp3
Accessibility of Big & Broad Data Big & Broad Data 28 / 38

31. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Sonification?
The First Higgs Boson Data Sonifcation
LHC Open Symphony (2012)
Accessibility of Big & Broad Data Big & Broad Data 28 / 38

32. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Auditory Perception – ‘Cocktail Party Problem’
Been talking about translating parallel visual experience into a
serial auditory one. BUT auditory perception is parallel too.
‘The separation of two simultaneously spoken messages’, in
which Cherry first poses the question ‘how do we recognise
what one person is saying when others are speaking at the
same time (the ‘cocktail party problem’)?’ is key.
This can only be useful work in the domain of blindness, auditory
interfaces, and multi-modal interfaces in that it may be possible
to convey aggregated big data much faster because of the ability
to comprehend highly parallel speech.
Accessibility of Big & Broad Data Auditory Perception 29 / 38
Cherry (1953)

33. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Properties of Sound
Accessibility of Big & Broad Data Auditory Perception 30 / 38

34. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Multi-Talker Display – Brungart
We now have between seven and nine people intelligibly
multi-talking at the same time based on spatial location and
voicing. Again, this can only be useful work in the domain of
blindness, auditory interfaces, and multi-modal interfaces in that
it may be possible to convey aggregated big data much faster
because of the ability to comprehend highly parallel speech.
Accessibility of Big & Broad Data Auditory Perception 31 / 38
(Brungart and Simpson, 2005; Bronkhorst, January/February 2000; Brungart et al., 2009)

35. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
The Ear
Accessibility of Big & Broad Data Auditory Perception 32 / 38
(Bear et al., 2006)

36. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Neurophysiology - Signal Transmission
Accessibility of Big & Broad Data Auditory Perception 33 / 38
(Bear et al., 2006)

37. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
The Auditory Cortex
Accessibility of Big & Broad Data Auditory Perception 34 / 38
(Bear et al., 2006)

38. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Wrapping-Up
1. Adaptation to the user is key; we must remove the
presumptions that have dogged software development, most
big or broad data is actually created without a clear
knowledge of how it will be used;
Accessibility of Big & Broad Data Wrapping-Up 35 / 38

39. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Wrapping-Up
1. Adaptation to the user is key; we must remove the
presumptions that have dogged software development, most
big or broad data is actually created without a clear
knowledge of how it will be used;
2. this makes the context of use and the user similar to
Assistive Technology; responsibility for display and
interaction should be with the user;
Accessibility of Big & Broad Data Wrapping-Up 35 / 38

40. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Wrapping-Up
1. Adaptation to the user is key; we must remove the
presumptions that have dogged software development, most
big or broad data is actually created without a clear
knowledge of how it will be used;
2. this makes the context of use and the user similar to
Assistive Technology; responsibility for display and
interaction should be with the user;
3. we can learn from Assistive Technology adaptation research
and development;
Accessibility of Big & Broad Data Wrapping-Up 35 / 38

41. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Wrapping-Up
5. how this can be applied in practice is yet to be seen, but we
need to dispense with surface presentations and allow deep
interaction; indeed,
Accessibility of Big & Broad Data Wrapping-Up 36 / 38

42. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Wrapping-Up
5. how this can be applied in practice is yet to be seen, but we
need to dispense with surface presentations and allow deep
interaction; indeed,
6. I contend that most public visualisations convey information
but are not rich enough to enable us to make an informed
decision; possibly,
Accessibility of Big & Broad Data Wrapping-Up 36 / 38

43. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Wrapping-Up
5. how this can be applied in practice is yet to be seen, but we
need to dispense with surface presentations and allow deep
interaction; indeed,
6. I contend that most public visualisations convey information
but are not rich enough to enable us to make an informed
decision; possibly,
7. adaptive enhanced visual analytics – focusing on analytical
reasoning facilitated by interactive visual interfaces – might
be the key.
Accessibility of Big & Broad Data Wrapping-Up 36 / 38

44. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Big & Broad Data Revisited
Are these ideas applicable to the mainstream?
Guardian Group (2014)
Accessibility of Big & Broad Data Wrapping-Up 37 / 38

45. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Any Questions?
Contact
vC: http://goo.gl/yzJFx
W: http://wel.cs.manchester.ac.uk
H: http://simon.harper.name
E: simon.harper@manchester.ac.uk
T: @sharpic
G: http://goo.gl/ySGJhW
Citations (BibTex)
http://goo.gl/3szs2e
Accessibility of Big & Broad Data Wrapping-Up 38 / 38

46. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Chieko Asakawa and C Lewis. Home page reader: IBM’s talking web browser. In Closing the Gap Conference
Proceedings, 1998.
Mark F. Bear, Barry W. Connors, and Michael A. Paradiso. Neuroscience: Exploring the Brain. Lippincott
Williams & Wilkins, 2006. ISBN 0781760038. URL
http://www.amazon.com/Neuroscience-Exploring-Mark-F-Bear/dp/0781760038%3FSubscriptionId%
3D0JYN1NVW651KCA56C102%26tag%3Dtechkie-20%26linkCode%3Dxm2%26camp%3D2025%26creative%
3D165953%26creativeASIN%3D0781760038.
Adelbert W. Bronkhorst. The cocktail party phenomenon: A review of research on speech intelligibility in
multiple-talker conditions. Acta Acustica united with Acustica, 86:117–128(12), January/February 2000.
URL http://www.ingentaconnect.com/content/dav/aaua/2000/00000086/00000001/art00016.
Andy Brown, Caroline Jay, and Simon Harper. Tailored presentation of dynamic web content for audio browsers.
International Journal of Human-Computer Studies, 70(3):179 – 196, March 2012. ISSN 1071-5819. doi:
http://dx.doi.org/10.1016/j.ijhcs.2011.11.001. URL
http://www.simonharper.info/publications/Harper2012ab.pdf.
Douglas S. Brungart and Brian D. Simpson. Optimizing the spatial configuration of a seven-talker speech display.
ACM Trans. Appl. Percept., 2:430–436, October 2005. ISSN 1544-3558. doi:
http://doi.acm.org/10.1145/1101530.1101538. URL
http://doi.acm.org/10.1145/1101530.1101538.
Douglas S. Brungart, Peter S. Chang, Brian D. Simpson, and DeLiang Wang. Multitalker speech perception with
ideal time-frequency segregation: Effects of voice characteristics and number of talkers. The Journal of
the Acoustical Society of America, 125(6):4006–4022, 2009. doi: 10.1121/1.3117686. URL
http://link.aip.org/link/?JAS/125/4006/1.
Alex Chen, Simon Harper, Darren Lunn, and Andrew Brown. Widget identification: A high-level approach to
accessibility. World Wide Web, pages 1–17, Jan 2012. ISSN 1386-145X. doi:
http://dx.doi.org/10.1007/s11280-012-0156-6. URL
http://www.simonharper.info/publications/Harper2012-1.pdf. 10.1007/s11280-012-0156-6.
Colin E. Cherry. Some Experiments on the Recognition of Speech, with One and with Two Ears. Journal of the
Acoustical Society of America, 25(5):975–979, 1953. doi: 10.1121/1.1907229. URL
http://dx.doi.org/10.1121/1.1907229.
Edd Dumbill. What is big data? http://strata.oreilly.com/2012/01/what-is-big-data.html, Jan 2012.
Accessibility of Big & Broad Data References 38 / 38

47. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Leo Ferres, Gitte Lindgaard, Livia Sumegi, and Bruce Tsuji. Evaluating a tool for improving accessibility to charts
and graphs. ACM Trans. Comput.-Hum. Interact., 20(5):28:1–28:32, November 2013. ISSN 1073-0516.
doi: 10.1145/2533682.2533683. URL http://doi.acm.org/10.1145/2533682.2533683.
Guardian Group. Does funding equal happiness in higher education?
http://ouseful.wordpress.com/2010/03/20/does-funding-equal-happiness-in-higher-education/,
March 2010.
Guardian Group. Disease and environmental factors across england and wales mapped.
http://www.theguardian.com/news/datablog/ng-interactive/2014/apr/25/
disease-and-environmental-factors-across-england-and-wales-mapped, April 2014.
Simon Harper and Sean Bechhofer. Sadie: Structural semantics for accessibility and device independence. ACM
Trans. Comput.-Hum. Interact., 14(2):10, 2007. ISSN 1073-0516. doi:
http://dx.doi.org/10.1145/1275511.1275516. URL
http://www.simonharper.info/publications/Harper2007kx.pdf.
Simon Harper and Yeliz Yesilada. Web Accessibility: A Foundation for Research, volume 1 of Human-Computer
Interaction Series. Springer, London, 1st edition, September 2008. ISBN 978-1-84800-049-0 (Print)
978-1-84800-050-6 (Online). doi: http://dx.doi.org/10.1007/978-1-84800-050-6. URL
http://www.simonharper.info/publications/Harper2008zp.pdf.
Simon Harper, Eleni Michailidou, and Robert Stevens. Toward a definition of visual complexity as an implicit
measure of cognitive load. ACM Trans. Appl. Percept., 6(2):1–18, March 2009. ISSN 1544-3558. doi:
http://dx.doi.org/10.1145/1498700.1498704. URL
http://www.simonharper.info/publications/Harper2008yl.pdf.
Jim Hendler. Big data is going broad according to government internet guru jim hendler.
http://semanticommunity.info/AOL˙Government/Big˙Data˙is˙going˙Broad˙According˙to˙
Government˙Internet˙Guru˙Jim˙Hendler, Feb 2012.
LHC Open Symphony. The first higgs boson data sonifcation!
https://lhcopensymphony.wordpress.com/the-first-higgs-boson-data-sonifcation/, July 2012.
Darren Lunn and Simon Harper. Providing assistance to older users of dynamic web content. Computers in
Human Behavior, July 2011. ISSN 0747-5632. doi: http://dx.doi.org/10.1016/j.chb.2011.06.004.
URL http://www.simonharper.info/publications/Harper2011fl.pdf.
Accessibility of Big & Broad Data References 38 / 38

48. What? Why? Adaptation/Transcoding Big & Broad Data Auditory Perception Wrapping-Up References
Martin Wattenberg, FernandaB. Vi´egas, and Katherine Hollenbach. Visualizing activity on wikipedia with
chromograms. In Cecilia Baranauskas, Philippe Palanque, Julio Abascal, and SimoneDinizJunqueira
Barbosa, editors, Human-Computer Interaction – INTERACT 2007, volume 4663 of Lecture Notes in
Computer Science, pages 272–287. Springer Berlin Heidelberg, 2007a. ISBN 978-3-540-74799-4. doi:
10.1007/978-3-540-74800-7 23. URL http://dx.doi.org/10.1007/978-3-540-74800-7˙23.
Martin Wattenberg, FernandaB. Vi´egas, and Katherine Hollenbach. Chromogram.
http://hint.fm/projects/chromogram/, July 2007b.
Tom White. Building hadoop data applications with kite. https://www.youtube.com/watch?v=aJmfgKyFcLA, Feb
2014.
Accessibility of Big & Broad Data Wrapping-Up 38 / 38