Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Automating Data Exploration SciPy 2016

928 views

Published on

A tool-agnostic overview of how to analyse and explore data in a systematic way. This talk covers metadata generation, univariate analysis, and the basics of bivariate analysis.

The talk also provides examples of natural power law distributions (scale-free networks.)

Published in: Data & Analytics
  • Be the first to comment

Automating Data Exploration SciPy 2016

  1. 1. 1 AUTOMATING DATA EXPLORATION A structured approach to analysing data A TOOL AGNOSTIC APPROACH
  2. 2. 2 AUTOMATING DATA EXPLORATION A structured approach to analysing data METADATA UNIVARIATE ANALYSIS BIVARIATE ANALYSIS
  3. 3. LET’S TAKE A DATASET 3 Each row has details about an employee who has left the organization. Just “reading” the dataset is quite informative.
  4. 4. DESCRIBE THE DATA IN A STRUCTURED WAY 4
  5. 5. 5 AUTOMATING DATA EXPLORATION A structured approach to analysing data METADATA UNIVARIATE ANALYSIS BIVARIATE ANALYSIS
  6. 6. CATEGORICAL COLUMNS YIELD VERY LITTLE DATA 6 There’s not much information in one column. The values are not quantitative, so a distribution is not meaningful. The values are not even ordered. In fact, the only thing we have is the list of values and their count. ... or is there more to this? Region Count India 10780 Headstrong 1554 China 1130 Philippines 1030 US 792 Romania 788 Mexico 324 Guatemala 233 Poland 124 Brazil 45 Hungary 41 Colombia 38 Netherlands 33 South Africa 30 UK 18 UAE 15 GMS India 15 Japan 11 CZECH Republic 10 Kenya 9
  7. 7. ... BUT RANK FREQUENCY IS STILL POSSIBLE 7 The rank of the row provides additional information. With this, we can explore the distribution of the rank against the count. These distributions are called rank- frequency distributions. Rank Region Count 1 India 10780 2 Headstrong 1554 3 China 1130 4 Philippines 1030 5 US 792 6 Romania 788 7 Mexico 324 8 Guatemala 233 9 Poland 124 10 Brazil 45 11 Hungary 41 12 Colombia 38 13 Netherlands 33 14 South Africa 30 15 UK 18 16 UAE 15 17 GMS India 15 18 Japan 11 19 CZECH Republic 10 20 Kenya 9
  8. 8. REGION SHOWS A POWER LAW DISTRIBUTION 8 Region Count India 10780 Headstrong 1554 China 1130 Philippines 1030 US 792 Romania 788 Mexico 324 Guatemala 233 Poland 124 Brazil 45 Hungary 41 Colombia 38 Netherlands 33 South Africa 30 UK 18 UAE 15 GMS India 15 Japan 11 CZECH Republic 10 Kenya 9 Rank on a log scale Frequencyonalogscale
  9. 9. COST CODE SHOWS A POWER LAW DISTRIBUTION 9 Cost Code Count 105 9542 121 1757 125 875 122 796 3001 654 3310 635 124 435 131 415 115 336 nan 207 101 205 127 173 109 148 116 91 126 66 ...
  10. 10. LE SHOWS A POWER LAW DISTRIBUTION 10 LE Count D84 11487 GPL 853 RM1 789 LC2 565 GMR 323 D95 247 GUT 233 ML1 223 CTK 184 AXE 127 A38 98 A21 79 EMP 61 BRL 45 A66 43 ...
  11. 11. 11 WHAT CAUSES POWER LAW DISTRIBUTIONS? PREFERENTIAL ATTACHMENT EXPONENTIAL GROWTH
  12. 12. NO. OF FOLLOWERS ON GITHUB 12 Username Count slidenerd 1700 astaxie 1320 MugunthKumar 1081 honcheng 870 arunoda 827 csjaba 670 cheeaun 658 timoxley 600 karlseguin 600 hemanth 514 arvindr21 400 yuvipanda 335 mbrochh 330 anandology 330 sayanee 314 zz85 314 sanand0 309 captn3m0 300 sameersbn 300 ...
  13. 13. NO. OF MOVIES ACTED IN BY BOLLYWOOD PEOPLE 13 Person Count Lata Mangeshkar 824 Asha Bhosle 810 Shakti Kapoor 589 Kishore Kumar 585 Mohammed Rafi 527 Sunidhi Chauhan 515 Alka Yagnik 451 Udit Narayan 435 Kader Khan 430 Sonu Nigam 405 Sameer 398 Asrani 397 Helen 395 Shaan 377 Aruna Irani 375 Anupam Kher 367 Shreya Ghoshal 357 Gulshan Grover 341 ...
  14. 14. PARTIES IN PARLIAMENT ELECTIONS 14 Name Count IND 44704 INC 7213 BJP 3354 BSP 2628 SP 1311 CPI 1102 JD 943 CPM 914 DDP 716 JNP 676 BJS 657 JP 563 NOTA 543 PSP 538 INC(I) 492 SHS 467 AAP 432 SWA 410 ...
  15. 15. CANDIDATE NAMES IN ASSEMBLY ELECTIONS 15 Name Count NONE OF THE ABOVE 629 OM PRAKASH 478 ASHOK KUMAR 411 RAM SINGH 362 RAJ KUMAR 294 ANIL KUMAR 271 AMAR SINGH 248 MOHAN LAL 235 RAM KUMAR 224 BABU LAL 218 RAM PRASAD 213 JAGDISH 210 VIJAY KUMAR 207 RAJENDRA SINGH 196 VINOD KUMAR 195 SHYAM LAL 193 RAJESH KUMAR 186 SITA RAM 186 RAM LAL 171 ...
  16. 16. STUDENT NAMES IN SSA SURVEY 16 Name Count M.MANIKANDAN 99 S.PAVITHRA 84 S.MANIKANDAN 84 R.RAMYA 82 S.SANGEETHA 70 R.MANIKANDAN 69 S.DIVYA 68 M.PAVITHRA 68 S.SANTHIYA 67 S.VIGNESH 67 M.PRIYA 67 M.MAHALAKSHMI 64 S.SARANYA 63 S.SURYA 60 K.MANIKANDAN 60 P.PAVITHRA 56 S.GAYATHRI 56 P.MANIKANDAN 55 ...
  17. 17. Jain Harini Shweta Sneha Pooja Ashwin Shah Deepti Sanjana Varshini Ezhumalai Venkatesan Silambarasan Pandiyan Kumaresan Manikandan Thirupathi Agarwal Kumar Priya
  18. 18. NOT EVERYTHING IS POWER-LAW, THOUGH 18 Need to understand what drives these distributions from their behaviours
  19. 19. ORDERED CATEGORICALS HAVE MORE INFORMATION 19
  20. 20. CORPORATE BAND 20 LE Count 5 12247 4 4449 3 205 2 63 Not Mapped 24 1 22 SVP 10
  21. 21. LOCAL BAND 21 LE Count 5A 7483 5B 4764 4A 1683 4B 1612 4C 747 4D 407 3 205 2 63 Not Mapped 24 1 22 SVP 10
  22. 22. QUANTITIES HAVE EVEN MORE INFORMATION 22
  23. 23. AGE DISTRIBUTION IS LOG-NORMAL 23
  24. 24. DETECTING FRAUD “ We know meter readings are incorrect, for various reasons. We don’t, however, have the concrete proof we need to start the process of meter reading automation. Part of our problem is the volume of data that needs to be analysed. The other is the inexperience in tools or analyses to identify such patterns. ENERGY UTILITY 24
  25. 25. This plot shows the frequency of all meter readings from Apr- 2010 to Mar-2011. An unusually large number of readings are aligned with the tariff slab boundaries. This clearly shows collusion of some form with the customers. Apr-10 May-10Jun-10Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 217 219 200 200 200 200 200 200 200 350 200 200 250 200 200 200 201 200 200 200 250 200 200 150 250 150 150 200 200 200 200 200 200 200 200 150 150 200 200 200 200 200 200 200 200 200 200 50 200 200 200 150 180 150 50 100 50 70 100 100 100 100 100 100 100 100 100 100 100 100 110 100 100 150 123 123 50 100 50 100 100 100 100 100 0 111 100 100 100 100 100 100 100 100 50 50 0 100 27 100 50 100 100 100 100 100 70 100 1 1 1 100 99 50 100 100 100 100 100 100 This happens with specific customers, not randomly. Here are such customers’ meter readings. Section Apr-10 May-10Jun-10 Jul-10 Aug-10 Sep-10 Oct-10 Nov-10 Dec-10 Jan-11 Feb-11 Mar-11 Section 1 70% 97% 136% 65% 110% 116% 121% 107% 114% 88% 74% 109% Section 2 66% 92% 66% 87% 70% 64% 63% 50% 58% 38% 41% 54% Section 3 90% 46% 47% 43% 28% 31% 50% 32% 19% 38% 8% 34% Section 4 44% 24% 36% 39% 21% 18% 24% 49% 56% 44% 31% 14% Section 5 4% 63% -27% 20% 41% 82% 26% 34% 43% 2% 37% 15% Section 6 18% 23% 30% 21% 28% 33% 39% 41% 39% 18% 0% 33% Section 7 36% 51% 33% 33% 27% 35% 10% 39% 12% 5% 15% 14% Section 8 22% 21% 28% 12% 24% 27% 10% 31% 13% 11% 22% 17% Section 9 19% 35% 14% 9% 16% 32% 37% 12% 9% 5% -3% 11% If we define the “extent of fraud” as the percentage excess of the 100 unit meter reading, the value varies considerably across sections, and time New section manager arrives … and is transferred out … with some explainable anomalies. Why would these happen? 25
  26. 26. PREDICTING MARKS “ What determines a child’s marks? Do girls score better than boys? Does the choice of subject matter? Does the medium of instruction matter? Does community or religion matter? Does their birthday matter? Does the first letter of their name matter? EDUCATION 26
  27. 27. TN CLASS X: ENGLISH 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 27
  28. 28. TN CLASS X: SOCIAL SCIENCE 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 28
  29. 29. TN CLASS X: LANGUAGE 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 29
  30. 30. TN CLASS X: SCIENCE 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 30
  31. 31. TN CLASS X: MATHEMATICS 0 5,000 10,000 15,000 20,000 25,000 30,000 35,000 40,000 0 5 10 15 20 25 30 35 40 45 50 55 60 65 70 75 80 85 90 95 100 31
  32. 32. ICSE 2013 CLASS XII: TOTAL MARKS 32
  33. 33. CBSE 2013 CLASS XII: ENGLISH MARKS 33
  34. 34. CBSE 2013 CLASS XII: PHYSICS MARKS 34
  35. 35. 35 AUTOMATING DATA EXPLORATION A structured approach to analysing data METADATA UNIVARIATE ANALYSIS BIVARIATE ANALYSIS
  36. 36. LET’S TAKE ONE DAY CRICKET DATA Country Player Runs ScoreRate MatchDate Ground Versus Australia Michael J Clarke 99* 93.39 30-06-2010The Oval England Australia Dean M Jones 99* 128.57 28-01-1985Adelaide Oval Sri Lanka Australia Bradley J Hodge 99* 115.11 04-02-2007Melbourne Cricket Ground New Zealand India Virender Sehwag 99* 99 16-08-2010Rangiri Dambulla International Stad. Sri Lanka New Zealand Bruce A Edgar 99* 72.79 14-02-1981Eden Park India Pakistan Mohammad Yousuf 99* 95.19 15-11-2007Captain Roop Singh Stadium India West Indies Richard B Richardson 99* 70.21 15-11-1985Sharjah CA Stadium Pakistan West Indies Ramnaresh R Sarwan 99* 95.19 15-11-2002Sardar Patel Stadium India Zimbabwe Andrew Flower 99* 89.18 24-10-1999Harare Sports Club Australia Zimbabwe Alistair D R Campbell 99* 79.83 01-10-2000Queens Sports Club New Zealand Zimbabwe Malcolm N Waller 99* 133.78 25-10-2011Queens Sports Club New Zealand Australia David C Boon 98* 82.35 08-12-1994Bellerive Oval Zimbabwe Australia Graeme M Wood 98* 63.22 11-01-1981Melbourne Cricket Ground India England Ian J L Trott 98* 84.48 20-10-2011Punjab Cricket Association Stadium India India Yuvraj Singh 98* 89.09 01-08-2001Sinhalese Sports Club Ground Sri Lanka Ireland Kevin J O'Brien 98* 94.23 10-07-2010VRA Ground Scotland Kenya Collins O Obuya 98* 75.96 13-03-2011M.Chinnaswamy Stadium Australia Netherlands Ryan N ten Doeschate 98* 73.68 01-09-2009VRA Ground Afghanistan New Zealand James E C Franklin 98* 142.02 07-12-2010M.Chinnaswamy Stadium India Pakistan Ijaz Ahmed 98* 112.64 28-10-1994Iqbal Stadium South Africa South Africa Jacques H Kallis 98* 74.24 06-02-2000St George's Park Zimbabwe 36
  37. 37. Against which countries are higher averages scored? Which countries’ players score more per match? 37
  38. 38. Which player scores the most per ball? The player with the highest strike rate is an obscure South African whose name most of us have never heard of. In fact, this list is filled with players we have never heard of. 38
  39. 39. Most analysis answers the question “Which is are the top 10 X”? Which are my top products? Which are my top branches? Who are my best sales people? Which vendors have the highest cost per unit? Which divisions are spending the most money? In which hours does the under 12 segment watch TV most? Which customer segment has the highest revenue per user? 39
  40. 40. THIS QUESTION CAN BE ANSWERED SYSTEMATICALLY Country Player Runs ScoreRate MatchDate Ground Versus Australia Michael J Clarke 99* 93.39 30-06-2010The Oval England Australia Dean M Jones 99* 128.57 28-01-1985Adelaide Oval Sri Lanka Australia Bradley J Hodge 99* 115.11 04-02-2007Melbourne Cricket Ground New Zealand India Virender Sehwag 99* 99 16-08-2010Rangiri Dambulla International Stad. Sri Lanka New Zealand Bruce A Edgar 99* 72.79 14-02-1981Eden Park India Pakistan Mohammad Yousuf 99* 95.19 15-11-2007Captain Roop Singh Stadium India West Indies Richard B Richardson 99* 70.21 15-11-1985Sharjah CA Stadium Pakistan West Indies Ramnaresh R Sarwan 99* 95.19 15-11-2002Sardar Patel Stadium India Zimbabwe Andrew Flower 99* 89.18 24-10-1999Harare Sports Club Australia Zimbabwe Alistair D R Campbell 99* 79.83 01-10-2000Queens Sports Club New Zealand Zimbabwe Malcolm N Waller 99* 133.78 25-10-2011Queens Sports Club New Zealand Australia David C Boon 98* 82.35 08-12-1994Bellerive Oval Zimbabwe Australia Graeme M Wood 98* 63.22 11-01-1981Melbourne Cricket Ground India England Ian J L Trott 98* 84.48 20-10-2011Punjab Cricket Association Stadium India India Yuvraj Singh 98* 89.09 01-08-2001Sinhalese Sports Club Ground Sri Lanka Ireland Kevin J O'Brien 98* 94.23 10-07-2010VRA Ground Scotland Kenya Collins O Obuya 98* 75.96 13-03-2011M.Chinnaswamy Stadium Australia Netherlands Ryan N ten Doeschate 98* 73.68 01-09-2009VRA Ground Afghanistan New Zealand James E C Franklin 98* 142.02 07-12-2010M.Chinnaswamy Stadium India Pakistan Ijaz Ahmed 98* 112.64 28-10-1994Iqbal Stadium South Africa South Africa Jacques H Kallis 98* 74.24 06-02-2000St George's Park Zimbabwe Take every column in the data Find the top value by that column Country South Africa has the highest strike rate of 76% Player Johann Louw has the highest strike rate of 329% Runs 164 runs has the highest strike rate of 156% MatchDate 12-03-2006 has the highest strike rate of 136% Ground AC-VDCA Stadium has the highest strike rate of 98% Versus United States has the highest strike rate of 104% 40
  41. 41. What do the children in schools know and can do at different stages of elementary education? Have the inputs made into the elementary education system had a beneficial effect or not? 41
  42. 42. HAVING BOOKS IMPROVES READING ABILITY Having more books at home improves the performance of children when it comes to reading. (But children typically only have only 1-10 books at home) Number of students sampled What is the impact? How many more marks can having more books fetch? Circle size indicates number of students with this response. Few students have no books. Is this response (“25+ books”) good or bad? Small red bars indicate low marks. Large green bars indicate high marks. Students having 25+ books tend to score high marks. The most common response is marked in blue. This is also the circle. The graphic is summarized in words Indicates whether the best response is the most popular. Blue means that it is not. Green means that it is. Red means that the worst level is the most popular response. 42
  43. 43. CHILDREN LIKE GAMES, AND THEY’RE GOOD … but playing daily hurts reading ability 43
  44. 44. WATCHING TV OCCASIONALLY IS GOOD Children who watch TV every day don’t do as well as children who watch TV only once a week. But children who never watch TV fare the worst. Watching TV every day helps improve children’s reading ability a little bit more… … but mathematical abilities fall dramatically at that point 44
  45. 45. WE HAVE A WEBSITE THAT YOU CAN EXPLORE GRAMENER.COM/NAS 45
  46. 46. 46 AUTOMATING DATA EXPLORATION A structured approach to analysing data METADATA UNIVARIATE ANALYSIS BIVARIATE ANALYSIS

×