My name is Andy Kirk
Visualisation
                                                             Consultant




http://www.quoteaustininsurance.com/images/Consultant.jpg
Visualisation
                                                                                 Designer




http://gizmodo.com/5792960/paul-allen-dishes-out-gossip-on-bill-gates-and-his-yacht-on-60-minutes
Visualisation
                                                                           Trainer




http://cathybretag.blogspot.com/2010/10/first-time-out-reflection-on-my-first.html
Hebden Bridge




                    London
2.5 hours
Hebden Bridge


2.5 mins

                   London
Curse of Knowledge

Absence of Knowledge
Surprise the novice,
get the expert to nod

       Mirko Lorenz
Showcase of
   data visualisation
techniques for thriving
     in the age of
        big data
Showcase of
      data visualisation
   techniques for thriving
        in the age of
 data that has thousands of
records and is quite complex
   and makes life difficult
It’s not a technology problem;
      it’s a people problem.

        Aron Pilhofer (on data journalism)
    Editor of Interactive News, New York Times
What is Big Data?

Why does it matter to you?
Context
                Google Insights: “Infographic”




http://www.google.com/insights/search/#q=%22Big%20Data%22%2CInfographics&date=6%2F2007%2058m&cmpt=q
Context
                   Google Insights: “Big Data”




http://www.google.com/insights/search/#q=%22Big%20Data%22%2CInfographics&date=6%2F2007%2058m&cmpt=q
Context
                   Google Insights: “Big Data”




http://www.google.com/insights/search/#q=%22Big%20Data%22%2CInfographics&date=6%2F2007%2058m&cmpt=q
We are capturing,
  creating and mobilising
unbelievable amounts of data
  at an unbelievable rate.

    And it is increasing.
Volume

            Variety

           Velocity

http://radar.oreilly.com/2012/01/what-is-big-data.html
Yahoo! C.O.R.E. Data Visualization | Periscopic




 http://www.flickr.com/photos/visualizeyahoo/sets/72157629000570607/
http://www.zimbio.com/Ted+Danson/articles/13/TV+DVD+Cheers+Final+Season+4+DVD+Set
Running the Numbers II: Portraits of global mass culture | Chris Jordan




                    http://www.chrisjordan.com/gallery/rtn2/#gyre2
Running the Numbers II: Portraits of global mass culture | Chris Jordan




                    http://www.chrisjordan.com/gallery/rtn2/#gyre2
Visualisation should be
recognised as a discovery tool.

                    Manuel Lima




         http://www.visualcomplexity.com/vc/blog/?p=644
Peer review wars | Nigel Hawtin




http://www.flickr.com/photos/nhawtin/5243787538/in/photostream/lightbox/
http://starwarsaficionado.blogspot.com/2011/12/classic-image-its-worse.html
http://v2.centralstory.com/about/squiggle/
1. Be clear about the visualisation’s
      purpose and parameters
EXPLORE: facilitate reasoning of data   EXPLAIN: convey information to others
  Analysis                                Monitor/Signals
  Familiarise with data                   Answer questions/Inform
  Support graphical calculation           Learn/Increase knowledge
  Find patterns/Find no patterns          Contextualise data
  Discover questions                      Present arguments
  Interact                                Assist with decisions
                                          Shape opinion/Persuade

                                          Emphasize issues
                                          Tell a story
                                          Inspire
                                          Shock/Make an impact
                                          Enlighten
                                          Change behaviour
                                          Entertain/fun
                                          Art/Aesthetic pleasure
Jet Tracker | Wall Street Journal




     http://projects.wsj.com/jettracker/
Jet Tracker | Wall Street Journal




     http://projects.wsj.com/jettracker/
Jet Tracker | Wall Street Journal




     http://projects.wsj.com/jettracker/
So many parameters!
Brief? Open, strict, helpful, unhelpful
Format? Static, interactive, video
Pressures? Timescales, editorial
Audience size? One, group, www
Audience type? Domain experts, general
Resolution? Headlines, clusters, look-up
Rules? Structure, layout, style, colour
Capabilities? Design, technical, technology
People? Individual, team, collaboration
Analyst Politician Computer scientist Journalist
  Researcher      Designer  Cognitive scientist
       http://www.jasonnazar.com/2008/09/23/10-lessons-startups-can-learn-from-superheros/
2. Identify and develop questions
    about the problem context
What questions are you hoping to
answer through this visualisation?

What stories should users/readers
  be able to derive from this
         visualisation?
Just Landed | Jer Thorp




http://blog.blprnt.com/blog/blprnt/just-landed-processing-twitter-metacarta-hidden-data
3. Acquire, prepare and explore
your data to begin familiarisation
Transforming    Transforming                     Pre-prod.
Acquisition   Examination                                  Consolidating
                              for quality    for purpose                   visualisation




                   System download
                         API
                      Web scrape
                  Scanned documents
Transforming    Transforming                     Pre-prod.
Acquisition   Examination                                  Consolidating
                              for quality    for purpose                   visualisation




                   Is it fit for purpose?
                       Is it complete?
                   Identify data types
Transforming    Transforming                     Pre-prod.
Acquisition   Examination                                  Consolidating
                              for quality    for purpose                   visualisation




                  Missing values
                 Erroneous values
                    Duplicates
               Uncommon characters
                  Freak outliers?
Transforming          Transforming                                    Pre-prod.
Acquisition         Examination                                                        Consolidating
                                            for quality          for purpose                                  visualisation




                   Parsing
                  Merging
               Normalisation
        Conversion eg. Codify free-text


 Inspired by Kim Rees’ talk at 2011 Wolfram Summit - http://www.wolframdatasummit.org/2011/attendee/presentations/Rees.pptx
Transforming          Transforming                                    Pre-prod.
Acquisition         Examination                                                        Consolidating
                                            for quality          for purpose                                  visualisation




             Full resolution
  Filter/Exclude (records & variables)
           Aggregate/Roll-up
                Sample
               Statistics


 Inspired by Kim Rees’ talk at 2011 Wolfram Summit - http://www.wolframdatasummit.org/2011/attendee/presentations/Rees.pptx
Yahoo! Mail Data Visualization | Periscopic




http://www.flickr.com/photos/visualizeyahoo/sets/72157627722660160/with/6235510547/
Transforming    Transforming                     Pre-prod.
Acquisition    Examination                                  Consolidating
                               for quality    for purpose                   visualisation




              What other data do I need?
The United States of 2012 for Esquire Magazine | Stamen




            http://content.stamen.com/united_states_of_2012
Transforming    Transforming                     Pre-prod.
Acquisition   Examination                                  Consolidating
                              for quality    for purpose                   visualisation




                     Patterns
                   Relationships
               Range and distribution
                     Outliers
World Nuclear Reactor Sites | Nigel Hawtin/Peter Aldhous




http://public.tableausoftware.com/views/WorldNuclearReactorSites2/NorthAmerica?:embed=y
4. Conceive your visualisation
       design solution
The 5 layers of a visualisation...

Data representation
Colour and background
Animation and interaction
Layout, placement and apparatus
The annotation layer
http://www.informationisbeautifulawards.com/2011/10/napkin-shortlist-for-the-1st-challenge/
138 years of popular science | Jer Thorp and Mark Hansen




      http://www.flickr.com/photos/blprnt/6281316931/sizes/o/in/photostream/
My working process is riddled
with dead-ends, messy errors
      and bad decisions
            JerThorp
138 years of popular science | Jer Thorp and Mark Hansen




         http://blog.blprnt.com/blog/blprnt/138-years-of-popular-science
138 years of popular science | Jer Thorp and Mark Hansen




         http://blog.blprnt.com/blog/blprnt/138-years-of-popular-science
Space Junk | Jen Christiansen and Jan Willem Tulp




               Scientific American, April 2012
Space Junk | Jen Christiansen and Jan Willem Tulp




               Scientific American, April 2012
Space Junk | Jen Christiansen and Jan Willem Tulp




               Scientific American, April 2012
The 5 layers of a visualisation...

Data representation
Colour and background
Animation and interaction
Layout, placement and apparatus
The annotation layer
http://projects.nytimes.com/census/2010/explorer
The 5 layers of a visualisation...

Data representation
Colour and background
Animation and interaction
Layout, placement and apparatus
The annotation layer
Posted: Visualizing US expansion through post offices | Derek Watkins




                  http://derekwatkins.wordpress.com/2011/08/06/posted/
Posted: Visualizing US expansion through post offices | Derek Watkins




                  http://derekwatkins.wordpress.com/2011/08/06/posted/
Posted: Visualizing US expansion through post offices | Derek Watkins




                  http://derekwatkins.wordpress.com/2011/08/06/posted/
Max Planck Research Networks | Moritz Stefaner and Christopher Warnow




                      http://max-planck-research-networks.net/
Max Planck Research Networks | Moritz Stefaner and Christopher Warnow




                      http://max-planck-research-networks.net/
The 5 layers of a visualisation...

Data representation
Colour and background
Animation and interaction
Layout, placement and apparatus
The annotation layer
Data Theft | Jen Christiansen




    Scientific American, October 2011
The 5 layers of a visualisation...

Data representation
Colour and background
Animation and interaction
Layout, placement and apparatus
The annotation layer
The annotation layer is the
most important thing we do...
   Otherwise it’s a case of
 here it is, you go figure it out.
                 Amanda Cox
        Graphics Editor, New York Times



           http://eyeofestival.com/speaker/amanda-cox/
http://www.stanford.edu/group/ruralwest/cgi-bin/drupal/visualizations/us_newspapers
http://www.stanford.edu/group/ruralwest/cgi-bin/drupal/visualizations/us_newspapers
5. Construct, launch and evaluate
    your visualisation solution
...you’ve started playing with the
visualization instead of debugging
      ... you hit some level of
    engagement and it becomes
           really interesting
 Martin Wattenberg, "Big Picture" data visualization group, Google


                      http://queue.acm.org/detail.cfm?id=1744741
You know you’ve achieved
perfection in design, not when you
    have nothing more to add,
        but when you have
    nothing more to take away
          Antoine de Saint-Exupery
Sense of Patterns | Mahir M. Yavuz




     http://casualdata.com/senseofpatterns/
Sense of Patterns | Mahir M. Yavuz




http://www.visualizing.org/full-screen/32596/embedlaunch
Sense of Patterns | Mahir M. Yavuz




http://www.visualizing.org/full-screen/32596/embedlaunch
Thank you to…

    Nigel Hawtin
  Jen Christiansen
  Moritz Stefaner
   Alberto Cairo
    Sarah Slobin
   Derek Watkins
     Kim Rees
   Mahir M Yavuz
     Jer Thorp
      Stamen
www.visualisingdata.com
andy@visualisingdata.com
    @visualisingdata

Andy Kirk Malofiej 20 Presentation

  • 1.
    My name isAndy Kirk
  • 2.
    Visualisation Consultant http://www.quoteaustininsurance.com/images/Consultant.jpg
  • 3.
    Visualisation Designer http://gizmodo.com/5792960/paul-allen-dishes-out-gossip-on-bill-gates-and-his-yacht-on-60-minutes
  • 4.
    Visualisation Trainer http://cathybretag.blogspot.com/2010/10/first-time-out-reflection-on-my-first.html
  • 6.
    Hebden Bridge London 2.5 hours
  • 7.
  • 8.
  • 9.
    Surprise the novice, getthe expert to nod Mirko Lorenz
  • 10.
    Showcase of data visualisation techniques for thriving in the age of big data
  • 11.
    Showcase of data visualisation techniques for thriving in the age of data that has thousands of records and is quite complex and makes life difficult
  • 12.
    It’s not atechnology problem; it’s a people problem. Aron Pilhofer (on data journalism) Editor of Interactive News, New York Times
  • 14.
    What is BigData? Why does it matter to you?
  • 15.
    Context Google Insights: “Infographic” http://www.google.com/insights/search/#q=%22Big%20Data%22%2CInfographics&date=6%2F2007%2058m&cmpt=q
  • 16.
    Context Google Insights: “Big Data” http://www.google.com/insights/search/#q=%22Big%20Data%22%2CInfographics&date=6%2F2007%2058m&cmpt=q
  • 17.
    Context Google Insights: “Big Data” http://www.google.com/insights/search/#q=%22Big%20Data%22%2CInfographics&date=6%2F2007%2058m&cmpt=q
  • 18.
    We are capturing, creating and mobilising unbelievable amounts of data at an unbelievable rate. And it is increasing.
  • 21.
    Volume Variety Velocity http://radar.oreilly.com/2012/01/what-is-big-data.html
  • 22.
    Yahoo! C.O.R.E. DataVisualization | Periscopic http://www.flickr.com/photos/visualizeyahoo/sets/72157629000570607/
  • 24.
  • 25.
    Running the NumbersII: Portraits of global mass culture | Chris Jordan http://www.chrisjordan.com/gallery/rtn2/#gyre2
  • 26.
    Running the NumbersII: Portraits of global mass culture | Chris Jordan http://www.chrisjordan.com/gallery/rtn2/#gyre2
  • 27.
    Visualisation should be recognisedas a discovery tool. Manuel Lima http://www.visualcomplexity.com/vc/blog/?p=644
  • 29.
    Peer review wars| Nigel Hawtin http://www.flickr.com/photos/nhawtin/5243787538/in/photostream/lightbox/
  • 31.
  • 32.
  • 33.
    1. Be clearabout the visualisation’s purpose and parameters
  • 34.
    EXPLORE: facilitate reasoningof data EXPLAIN: convey information to others Analysis Monitor/Signals Familiarise with data Answer questions/Inform Support graphical calculation Learn/Increase knowledge Find patterns/Find no patterns Contextualise data Discover questions Present arguments Interact Assist with decisions Shape opinion/Persuade Emphasize issues Tell a story Inspire Shock/Make an impact Enlighten Change behaviour Entertain/fun Art/Aesthetic pleasure
  • 35.
    Jet Tracker |Wall Street Journal http://projects.wsj.com/jettracker/
  • 36.
    Jet Tracker |Wall Street Journal http://projects.wsj.com/jettracker/
  • 37.
    Jet Tracker |Wall Street Journal http://projects.wsj.com/jettracker/
  • 38.
  • 39.
    Brief? Open, strict,helpful, unhelpful Format? Static, interactive, video Pressures? Timescales, editorial Audience size? One, group, www Audience type? Domain experts, general Resolution? Headlines, clusters, look-up Rules? Structure, layout, style, colour Capabilities? Design, technical, technology People? Individual, team, collaboration
  • 40.
    Analyst Politician Computerscientist Journalist Researcher Designer Cognitive scientist http://www.jasonnazar.com/2008/09/23/10-lessons-startups-can-learn-from-superheros/
  • 41.
    2. Identify anddevelop questions about the problem context
  • 42.
    What questions areyou hoping to answer through this visualisation? What stories should users/readers be able to derive from this visualisation?
  • 43.
    Just Landed |Jer Thorp http://blog.blprnt.com/blog/blprnt/just-landed-processing-twitter-metacarta-hidden-data
  • 44.
    3. Acquire, prepareand explore your data to begin familiarisation
  • 45.
    Transforming Transforming Pre-prod. Acquisition Examination Consolidating for quality for purpose visualisation System download API Web scrape Scanned documents
  • 46.
    Transforming Transforming Pre-prod. Acquisition Examination Consolidating for quality for purpose visualisation Is it fit for purpose? Is it complete? Identify data types
  • 47.
    Transforming Transforming Pre-prod. Acquisition Examination Consolidating for quality for purpose visualisation Missing values Erroneous values Duplicates Uncommon characters Freak outliers?
  • 48.
    Transforming Transforming Pre-prod. Acquisition Examination Consolidating for quality for purpose visualisation Parsing Merging Normalisation Conversion eg. Codify free-text Inspired by Kim Rees’ talk at 2011 Wolfram Summit - http://www.wolframdatasummit.org/2011/attendee/presentations/Rees.pptx
  • 49.
    Transforming Transforming Pre-prod. Acquisition Examination Consolidating for quality for purpose visualisation Full resolution Filter/Exclude (records & variables) Aggregate/Roll-up Sample Statistics Inspired by Kim Rees’ talk at 2011 Wolfram Summit - http://www.wolframdatasummit.org/2011/attendee/presentations/Rees.pptx
  • 50.
    Yahoo! Mail DataVisualization | Periscopic http://www.flickr.com/photos/visualizeyahoo/sets/72157627722660160/with/6235510547/
  • 51.
    Transforming Transforming Pre-prod. Acquisition Examination Consolidating for quality for purpose visualisation What other data do I need?
  • 52.
    The United Statesof 2012 for Esquire Magazine | Stamen http://content.stamen.com/united_states_of_2012
  • 53.
    Transforming Transforming Pre-prod. Acquisition Examination Consolidating for quality for purpose visualisation Patterns Relationships Range and distribution Outliers
  • 54.
    World Nuclear ReactorSites | Nigel Hawtin/Peter Aldhous http://public.tableausoftware.com/views/WorldNuclearReactorSites2/NorthAmerica?:embed=y
  • 55.
    4. Conceive yourvisualisation design solution
  • 56.
    The 5 layersof a visualisation... Data representation Colour and background Animation and interaction Layout, placement and apparatus The annotation layer
  • 57.
  • 58.
    138 years ofpopular science | Jer Thorp and Mark Hansen http://www.flickr.com/photos/blprnt/6281316931/sizes/o/in/photostream/
  • 59.
    My working processis riddled with dead-ends, messy errors and bad decisions JerThorp
  • 60.
    138 years ofpopular science | Jer Thorp and Mark Hansen http://blog.blprnt.com/blog/blprnt/138-years-of-popular-science
  • 61.
    138 years ofpopular science | Jer Thorp and Mark Hansen http://blog.blprnt.com/blog/blprnt/138-years-of-popular-science
  • 62.
    Space Junk |Jen Christiansen and Jan Willem Tulp Scientific American, April 2012
  • 63.
    Space Junk |Jen Christiansen and Jan Willem Tulp Scientific American, April 2012
  • 64.
    Space Junk |Jen Christiansen and Jan Willem Tulp Scientific American, April 2012
  • 65.
    The 5 layersof a visualisation... Data representation Colour and background Animation and interaction Layout, placement and apparatus The annotation layer
  • 66.
  • 67.
    The 5 layersof a visualisation... Data representation Colour and background Animation and interaction Layout, placement and apparatus The annotation layer
  • 68.
    Posted: Visualizing USexpansion through post offices | Derek Watkins http://derekwatkins.wordpress.com/2011/08/06/posted/
  • 69.
    Posted: Visualizing USexpansion through post offices | Derek Watkins http://derekwatkins.wordpress.com/2011/08/06/posted/
  • 70.
    Posted: Visualizing USexpansion through post offices | Derek Watkins http://derekwatkins.wordpress.com/2011/08/06/posted/
  • 71.
    Max Planck ResearchNetworks | Moritz Stefaner and Christopher Warnow http://max-planck-research-networks.net/
  • 72.
    Max Planck ResearchNetworks | Moritz Stefaner and Christopher Warnow http://max-planck-research-networks.net/
  • 73.
    The 5 layersof a visualisation... Data representation Colour and background Animation and interaction Layout, placement and apparatus The annotation layer
  • 74.
    Data Theft |Jen Christiansen Scientific American, October 2011
  • 75.
    The 5 layersof a visualisation... Data representation Colour and background Animation and interaction Layout, placement and apparatus The annotation layer
  • 76.
    The annotation layeris the most important thing we do... Otherwise it’s a case of here it is, you go figure it out. Amanda Cox Graphics Editor, New York Times http://eyeofestival.com/speaker/amanda-cox/
  • 77.
  • 78.
  • 79.
    5. Construct, launchand evaluate your visualisation solution
  • 82.
    ...you’ve started playingwith the visualization instead of debugging ... you hit some level of engagement and it becomes really interesting Martin Wattenberg, "Big Picture" data visualization group, Google http://queue.acm.org/detail.cfm?id=1744741
  • 83.
    You know you’veachieved perfection in design, not when you have nothing more to add, but when you have nothing more to take away Antoine de Saint-Exupery
  • 84.
    Sense of Patterns| Mahir M. Yavuz http://casualdata.com/senseofpatterns/
  • 85.
    Sense of Patterns| Mahir M. Yavuz http://www.visualizing.org/full-screen/32596/embedlaunch
  • 86.
    Sense of Patterns| Mahir M. Yavuz http://www.visualizing.org/full-screen/32596/embedlaunch
  • 88.
    Thank you to… Nigel Hawtin Jen Christiansen Moritz Stefaner Alberto Cairo Sarah Slobin Derek Watkins Kim Rees Mahir M Yavuz Jer Thorp Stamen
  • 89.