Download to read offline
![Big Data building blocks
Generic process model,
Big data analytics
processes based on
building blocks [Chau]
Collection
Cleaning
Integration
Visualization
Analysis
Presentation
Dissemination
Some building blocks can be
skipped, depending on the
operating contexts and to go
back (two-way street) is
admitted
Eurostat](https://image.slidesharecdn.com/visualisationanditsimportance2-240101051525-65a524c8/85/Visualisation-and-its-importance-2-pptx-11-320.jpg)
![Automated analysis
Automated analysis of Big Data concerns with the “development of
methods and techniques for making sense of data” [Fayyad]
Simple reports
Descriptive
approximation or model
of the process that
generated the data
Predictive model for
estimating the value of
future cases
Extreme
characteristics
of Big Data
Huge
At low-level
More abstract
Synthetic
Clear
Useful
Specific data-mining
methods for pattern
discovery and
extraction
Eurostat](https://image.slidesharecdn.com/visualisationanditsimportance2-240101051525-65a524c8/85/Visualisation-and-its-importance-2-pptx-17-320.jpg)
![Interactive visualization
Eurostat
In the context of Big Data some categories as basis of
reasoning can be adopted [Yi-etal-2007]:
• Select (mark something as interesting)
• Explore (show me something else)
• Reconfigure (show me a different arrangement)
• Encode (show me a different representation)
• Abstract/elaborate (show me more or less detail)
• Filter (show me something conditionally)
• Connect (show me related items)
http://www.cs.tufts.edu/comp/250VA/papers/yi2007toward.pdf](https://image.slidesharecdn.com/visualisationanditsimportance2-240101051525-65a524c8/85/Visualisation-and-its-importance-2-pptx-19-320.jpg)
![San Francisco Transportation
Eurostat
312. Mobile sensors (tracking)
«…data from the Muni (San
Francisco Municipal
Transportation Agency)
showing the geographic
coordinates of their vehicles to
create this map showing
average transit speeds over a
24-hour period.
[…]
Black lines represent very slow
movement under 7 mph. Red
are less than 19 mph. Blue are
less than 43 mph. Green lines
depict faster speeds above 43
mph.»
https://www.flickr.com/photos/walkingsf/4521616274/in/photostream/](https://image.slidesharecdn.com/visualisationanditsimportance2-240101051525-65a524c8/85/Visualisation-and-its-importance-2-pptx-40-320.jpg)
Visualisation and its importance (2).pptx
- 1.
- 2. Getting the PictureRight 1. Conceptual? 2. Perceptual? 3. Graphic? 4. Politically? What does «right» mean? Point of view Eurostat
- 3. Conceptual POV Correlation DoesNot Mean Causation 2000 1500 1000 500 0 2500 3500 3000 0 200 1200 1400 r= 0,926 400 600 800 1000 Weight Price Weight Price ? Prices and weights of foreign cars sold in Austria in 1956 Car model Weight (Kmg) Price ($) Car model Weight (Kmg) Price ($) 1 675 1227 16 730 1408 2 495 1085 17 1130 1827 3 585 1096 18 1070 1885 4 490 958 19 865 2019 5 760 1338 20 1050 2000 6 585 1096 21 895 1758 7 670 1327 22 1120 2269 8 1020 2115 23 1070 2154 9 825 1485 24 1210 2250 10 811 1377 25 1270 2269 11 825 1769 26 1325 2885 12 930 1804 27 1155 2058 13 950 1758 28 1210 2750 14 890 1646 29 1220 2527 15 950 1381 30 1140 2132 Eurostat
- 4.
- 5. Graphical POV Lie factor= = = 1 : Truth ≠ 1 : Lie Size of effect in data Size of effect shown in graphic where From: http://www.infovis-wiki.net/index.php?title=Lie_Factor Eurostat
- 6.
- 7. Outline Eurostat Introduction Visual cognition process Graphicinformation processing Big Data visualization Storytelling
- 8. Data visualization andBig Data Implementing effective data visualization solutions for Big Data has to take into account - apart the volume of the data - other intrinsic constraints generated by the typical characteristics of Big Data: • real-time changes • extreme variety of the sources • different levels of data structuring Moreover, it is advisable the simultaneous usage of several visualization techniques to better illustrate relationships among a large amount of data. Eurostat
- 9. When Data becomeBig? Extreme-scale Size Inclusion of visual and analytical Active involvement of a human Data in many forms Structured, unstructured, text, multimedia Data in motion Analysis of streaming data to enable decisions within fractions of a second Data at scale Petabyte (1015) to Exabyte (1018) Complex Information Spaces (a) the data items being difficultto compare based on raw data, (b) data compound of several base data types Three critical elements in applying visual analytics to extreme-scale data and complex Information Spaces Eurostat
- 10. Complexity and flatness Eurostat “Theworld is complex, dynamic, multidimensional; the paper is static, flat. How are we to represent the rich visual world of experience and measurement on mere flatland?” E. Tu f t e
- 11. Big Data buildingblocks Generic process model, Big data analytics processes based on building blocks [Chau] Collection Cleaning Integration Visualization Analysis Presentation Dissemination Some building blocks can be skipped, depending on the operating contexts and to go back (two-way street) is admitted Eurostat
- 12. Role of datavisualization in Big Data Life Cycle Eurostat • Data visualization can play a specific role in several phases of the Big Data Life Cycle • Data types can affect visualization design • Visualization methods can informs data cleaning and the choice of analysis algorithms Along the Big Data life cycle, visualization methods can be properly incorporated in three phases: • Pre-processing, staging, handling • Exploratory data analysis • Presentation of analytical results
- 13. Three Styles ofBig Data Visualization Remco Chang – Fields Institute 15 Emphasis on… Methodology Author Data reduction Big Data Medium Data Small Data+ R Filtering Filtering Wickham Visual interaction New representation pattern + User Interaction StarGlyphs+Parallel coordinates Interaction Carpendale HCP Divide and conquer + Parallel Computation Bowei Xi Eurostat
- 14. Visualizing Big Datain Official Statistics Although there are already many experiences and success stories in applying data visualization technologies on Big Data, the most interesting proposals are aimed at future challenges. The main issues to deal with are focused on the combination of some basic opportunities like: Automated analysis tools Eurostat Interactive visual methods Traditional visual analytics approaches Presentation tools New advanced data visualization technologies Analytic platforms
- 15. Automated analysis andinteractive visual methods In order to support the entire life cycle of Big Data, a good visual analytics system has to combine the advantages of the automatic analysis with interactive techniques to explore data. Behind this desired technical feature there is the deeper aim to integrate the analytic capability of a computer with the abilities of the human analysis. volume, velocity, variety mapping complex data into more simple visual forms of knowledge Appropriately definition in phase of design and implementation of specific weight and right balancing of the two components Eurostat
- 16. Macro phase DataProcesses Data management Selection & Data loading Integration Export Data handling Pre-processing, cleaning & transformation Calculations & querying Data modelling Statistics functions (univariate, bivariate and multivariate analysis) Clustering, classification, network modelling, predictive analysis Data projection (Principal Components, Multidimensional scaling, Self organizing map, Bayesian Network) Pattern recognition & Visual query analysis (both automated and interactive) Data Visualization Visual Interpretation, evaluation, representation Eurostat Automated analysis Reorganization of the structure of the visual analytics functionalities
- 17. Automated analysis Automated analysisof Big Data concerns with the “development of methods and techniques for making sense of data” [Fayyad] Simple reports Descriptive approximation or model of the process that generated the data Predictive model for estimating the value of future cases Extreme characteristics of Big Data Huge At low-level More abstract Synthetic Clear Useful Specific data-mining methods for pattern discovery and extraction Eurostat
- 18. Interactive Visual Analyticstechniques with Big Data Data preprocessing through visual approaches • Data mining • Machine learning • Statistical methods Interactive visualization Dissemination tools • Browse • search • monitor • Show the data Bring out meaningful: • patterns • outliers • clusters • gaps • Discover the most interesting relationships among data • Investigate what-if scenarios • Verify the presence of biases • Simulate changes impact • Enlighten the sense of data • Tell stories about them Eurostat
- 19. Interactive visualization Eurostat In thecontext of Big Data some categories as basis of reasoning can be adopted [Yi-etal-2007]: • Select (mark something as interesting) • Explore (show me something else) • Reconfigure (show me a different arrangement) • Encode (show me a different representation) • Abstract/elaborate (show me more or less detail) • Filter (show me something conditionally) • Connect (show me related items) http://www.cs.tufts.edu/comp/250VA/papers/yi2007toward.pdf
- 20. Eurostat Filter (show me something conditionally) Abstract/ elaborate (showme more or less detail) Explore (show me something else) Select (mark something as interesting) Eurostat
- 21.
- 22.
- 23. Interactive visualization Eurostat Select Ability tomark data items of interest to highlight them Outlier values Explore Enabling users to examine the different subsets in which the data can be divided Panning across the data Reconfigure Provide users with different data perspectives • Revelation of hidden patterns • visual rearrangements of a series Encode Capability of a visualization system to handle and transform the basic elements of human vision Pre-attentive processing, colours, shapes, dimensions Abstract/ elaborate Capability of reduce or increase the details of the visualization Filter Highlight some visual elements that are compliant with specific conditions defined by users Connect Enables users to better emphasize relationships and associations already known or discover the hidden patterns of the data
- 24. Human perception objects becomes large,humans often have difficulty extracting meaningful information Limited screen space Traditional Visual Analytics tools and techniques don’t properly fit big data. Computational problems for VA with Big Data When the number of visualized Risk of significant visual clutter when a visualization displays too many data Main causes Eurostat Effects Traditional vs. New techniques
- 25. Traditional vs. Newtechniques Working with new data sources brings about a number of analytical challenges (1) getting the picture right, i.e. summarising the data (2) interpreting, or making sense of the data through inferences (3) defining and detecting anomalies. Eurostat
- 26. Visual scalability Computationa l methods Dimension reduction Clustering Methodsto exploit machine learning Methods to exploit data mining Provide compact, meaningful information about the raw data Eurostat
- 27. 3. Internet ofThings (machine- generated data) 31. Data from sensors 311. Fixed sensors 3111. Home automation 3112. Weather/pollution sensors 3113. Traffic sensors/webcam 3114. Scientific sensors 3115. Security videos/images 312. Mobile sensors (tracking) 3121. Mobile phone location 3122. Cars 3123. Satellite images 32. Data from computer systems 3210. Logs 3220. Web logs Eurostat 1. Social Networks (human-sourced information) 1100. Social Networks 1200. Blogs and comments 1300. Personal documents 1400. Pictures: Instagram, Flickr, Picasa 1500. Videos: Youtube etc. 1600. Internet searches 1700. Mobile data content: text messages 1800. User-generated maps 1900. E-Mail 2. Traditional Business systems (process- mediated data) 21. Data produced by Public Agencies 2110. Medical records 22. Data produced by businesses 2210. Commercial transactions 2220. Banking/stock records 2230. E-commerce 2240. Credit cards
- 28. 3. Internet ofThings (machine- generated data) 31. Data from sensors 311. Fixed sensors 3111. Home automation 3112. Weather/pollution sensors 3113. Traffic sensors/webcam 3114. Scientific sensors 3115. Security videos/images 312. Mobile sensors (tracking) 3121. Mobile phone location 3122. Cars 3123. Satellite images 32. Data from computer systems 3210. Logs 3220. Web logs 1. Social Networks (human-sourced information) 1100. Social Networks 1200. Blogs and comments 1300. Personal documents 1400. Pictures: Instagram, Flickr, Picasa 1500. Videos: Youtube etc. 1600. Internet searches 1700. Mobile data content: text messages 1800. User-generated maps 1900. E-Mail 2. Traditional Business systems (process- mediated data) 21. Data produced by Public Agencies 2110. Medical records 22. Data produced by businesses Eurostat 2210. Commercial transactions 2220. Banking/stock records 2230. E-commerce 2240. Credit cards
- 29. Blogopole http://blogopole.observatoire-presidentielle.fr/ 1200. Blogs andcomments «La Blogopole (contraction de blogosphère politique) c'est l'ensemble des sites et blogs de citoyens qui alimentent le débat politique en France c'est à dire tant les hommes politiques, les militants, les sympathisants que les commentateurs et analystes» Eurostat
- 30.
- 31. The Bible Eurostat 1300. Personaldocuments «The bar graph that runs along the bottom represents all of the chapters in the Bible. Books alternate in color between white and light gray. The length of each bar denotes the number of verses in the chapter. Each of the 63,779 cross references found in the Bible is depicted by a single arc - the color corresponds to the distance between the two chapters, creating a rainbow-like effect» http://www.chrisharrison.net/index.php/Visualizations/BibleViz
- 32. Human emotion 1100. SocialNetworks «This video shows the mood in the U.S., as inferred using over 300 million tweets, over the course of the day. The maps are represented using density-preserving cartograms» https://www.youtube.com/watch?v=ujcrJZRSGkg Eurostat
- 33. Tweetcatcha 1100. Social Networks «TweetCatcha seeksto uncover the organic nature of news as it travels through Twitter over time, by examining the movement of NY Times articles through Twitter» Eurostat
- 34.
- 35. 100 seconds ofHistory 1. Human-sourced information http://flowingdata.com/2011/03/21/history-of-the-world-in-100-seconds-according-to-wikipedia/ Eurostat For a sort of evolution of the world at glance, all geotagged Wikipedia articles have been scraped, with time attached to them, providing a total of 14,238 events.
- 36. Human disease network 2110.Medical records «The diseasome website is a disease/disorder relationships explorer and a sample of an innovative map-oriented scientific work. Built by a team of researchers and engineers, it uses the Human Disease Network dataset and allows intuitive knowledge discovery by mapping its complexity» Eurostat
- 37. «It's also evident thatonly a day later hardly anybody was talking about the hurricane, showing the transient nature of social media, even for large global events.» «…Digital portrait for each city, formed from millions of bits of data as people talked and interacted about the biggest events of the day.» «…time explodes outwards from the centre with each point representing one minute giving a possible 4320 points –the number of minutes in three days –to cover the day before, during and after the launch of 4G.» «In the London image you can clearly see when Hurricane Sandy hit in New york, and even when Obama visited the city to inspect the damage.» Digital City Portraits (launch of 4G by EE) http://brendandawes.com/projects/ee 1700. Mobile data content: text messages Eurostat
- 38. Urban Mobs 3121. Mobilephone location «Cette visualisation représente la quantité de SMS envoyés le soir de la fête de la musique (21 juin 2008). On peut découvrir à partir de 17h une forte activité aux alentours du Parc des Princes que nous pouvons mettre en parallèle avec le concert de Tokio Hotel ce soir là. On remarque ensuite un autre foyer d'activité à l'hippodrome d'Auteuil correspondant au concert organisé par France 2» http://www.urbanmobs.fr/fr/france/ Eurostat
- 39. LIVE Singapore! 31. Datafrom sensors «Making decisions in sync with the environment LIVE Singapore! provides people with access to a range of useful real-time information about their city by developing an open platform for the collection, elaboration and distribution of real-time data that reflect urban activity. Giving people visual and tangible access to real-time information about their city enables them to take their decisions more in sync with their environment, with what Eurostat is actually happening around them.» https://www.youtube.com/watch?feature=player_embedded&v=2aEPkyOBtRo
- 40. San Francisco Transportation Eurostat 312.Mobile sensors (tracking) «…data from the Muni (San Francisco Municipal Transportation Agency) showing the geographic coordinates of their vehicles to create this map showing average transit speeds over a 24-hour period. […] Black lines represent very slow movement under 7 mph. Red are less than 19 mph. Blue are less than 43 mph. Green lines depict faster speeds above 43 mph.» https://www.flickr.com/photos/walkingsf/4521616274/in/photostream/
- 41.
- 42. Outline Eurostat Introduction Visual cognition process Graphicinformation processing Big Data visualization Storytelling
- 43. Hints about Storytelling Eurostat “Narrativeor recital of an event, or a series of events whether real or fictitious” New International Webster’s Comprehensive Dictionary (2013 edition) “Programme to make the results of official statistics accessible and understandable to people and – in fulfilment of an information mandate – to make "evidence based decision making" possible.” Armin Grossenbacher, Federal Statistical Office, Storytelling revisited, 2010
- 44. Storytelling principles Eurostat 1) GriceanMaxims (P . Grice) 2) Pyramid principle (B. Minto) 3) Seven steps to storytelling (J. Lambert) 4) Scenario for combining data, model and stories (J. Koomey) 5) Five golden rules for statistics storytellers (D. Marder)
- 45. Gricean Maxims 1. Donot say what you believe to be false. 2. Do not say that for which you lack adequate evidence. Grice’s conversational maxims necessary. Be relevant (that is, say things related to the current topic of conversation). 1.Avoid obscurity of expression. 2. ity. Avoid ambigu 3. Be brief (avoid unnecessary wordiness). 4. Be orderly. 1. Make your contribution to the conversation as informative as necessary. 2. Do not make your contribution to the conversation more informative than “Make your conversational contribution what is required, at the stage at which it occurs, by the accepted purpose or direction of the talk exchange in which you are engaged.” (P. Grice) Eurostat
- 46. Barbara Minto’s pyramidprinciple The Answer is your particularly inspired way of solving the problem you are presenting. The Situation is simply the state of affairs in your particular area. For example, your current growth rate or your product offering. The Complication is what is changing in your field to make things more challenging—it’s the proverbial thorn in your side that you have to remove in order to make things run smoothly. This might be your new competition, or a lack of fresh prospects. The Question states what the situation and complication are asking. For instance how do I achieve double-digit growth with increased competition? Or another question—how do I reach out to the particular audience that I’ve targeted and get them to buy my product? http://blog.kurtosys.com/storytell ing-pyramid-principle/ Eurostat
- 47. Seven steps tostorytelling Step 1: Owning Your Insights Step 2: Owning Your Emotions Step 3: Finding The Moment Step 4: Seeing Your Story Step 5: Hearing Your Story Step 6: Assembling Your Story Step 7: Sharing Your Story Joe Lambert, DIGITAL STORYTELLING COOKBOOK – 2010, Digital Diner Press Insights Emotions Decisive Moments Vision Narrativ e Editing Sharing Eurostat
- 48. Scenario for combiningdata, model and stories Turning Numbers Into Knowledge: Mastering the Art of Problem Solving - Jon Koomey Eurostat
- 49. Five golden rulesfor statistics storytellers Eurostat … five golden rules that statistical story writers often lose sight of: •Write as people speak; •Don’t just get to the point – start with it; •Make every sentence relevant to the audience – what’s in it for them; •Stay simple, but don’t patronise; •Use only one idea per sentence. David Marder, Office for National Statistics. The Holistic Approach to Statistical Story-Telling, 16 UNECE Work Session on Dissemination of Statistical Commentary (Geneva, 4-5 Dec. 2003).
- 50. killer-examples (i.e.: 8 waysto build an effective storytelling with infographic) http://www.howtostory.be/killer-examples-of-the-best-infographics/ newspaper flowchart timeline bait comparison numbers photos vision Eurostat