Geodemographics: Open tools and mehtods

515
-1

Published on

This presentation given an overview of geodemographic classifications and why there is a need to use open tools and methods for creating geodemographic classifications. The presentation also describes the challenges involve with creating real-time geodemographic classifications and the use of social media data for geodemographic applications.

Published in: Education, Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
515
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
5
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Geodemographics: Open tools and mehtods

  1. 1. Geodemographics: Open tools and methods Dr. Muhammad Adnan Department of Geography, University College London Web: http://www.uncertaintyofidentity.com Email: m.adnan@ucl.ac.uk Twitter: @gisandtech
  2. 2. Lecture Outline • Geodemographic Classification • Problems with the Geodemographic Classifications • Real-time bespoke Geodemographic Classifications • GeodemCreator: A software for creating Geodemographic Classifications • Social Media data for Geodemographics
  3. 3. Geodemographics • “Analysis of people by where they live” or “locality marketing” (Sleight, 1993:3) Person Home Address Area
  4. 4. Steps in Creating a Geodemographic Classification • Variable Selection • Transformation of the Data • Standardisation of the Data • Clustering of the Data (k-means) • Naming the clusters
  5. 5. Data – Census + Other ONS Output Area Classification (2001 and 2011) • Census data: 100% Experian: Mosaic • Census data: 54% • Non-Census data: 46% CACI: Accorn • Census data: 30% • Non-Census data: 70%
  6. 6. Standardising the data • Z-Scores • Widely used variable normalisation technique • Can create outliers in the datasets • Range Standardisation • Standardise values between a range of 0-1 • Can erase interesting patterns in the data • Principal Component Analysis (PCA) • Reduces the dimensions of a data set • Focuses on the part of dataset having maximum variance • Can erase interesting patterns in the data
  7. 7. Segmentations are created by cluster analysis Areas V1 V2 V3 V4 V5 V6 … Variable 2 Cluster 1 Area1 Cluster 2 Area2 Area3 Area4 Variable 1 Area5 Area6 ……. Cluster 3
  8. 8. Output of Cluster Analysis Areas Cluster Area1 1 Area2 1 Area3 2 Area4 1 Area5 3 Area6 3 ……. 2001 OAC (around Greater London)
  9. 9. Naming the clusters • 2011 OAC has 8 super groups 1. Rural Residents 2. Cosmopolitans 3. Ethnic Mix 4. Blue Collar Neighbourhoods 5. Multicultural Metripolitans 6. Suburbanites 7. Hard-Pressed Households 8. Urbanites
  10. 10. But geodemographic classifciations have some problems !
  11. 11. Does one size fit all ? • Most geodemographic classifications divide areas into a specified number of categories • 2011 OAC divides the Output Areas in the UK into 8 broad categories • Do these categories account for all the characteristics of the population ? • We need to create bespoke small area classifications ? • Geodemographic categories only apply to a particular area
  12. 12. Closed Methods • Commercial geodemographic classifications (i.e. MOSAIC, ACCORN) use closed methods • • • • Data sources used ? Weighting of the variables ? Data standardisation techniques employed ? Clustering algorithm applied ? • We need open methods and clear documentation of the geodemographic classifications • 2001 OAC • 2001 LOAC (London‟s Output Area Classification) • 2011 OAC
  13. 13. Public Consultation • Users of the classification cannot modify or give a feedback • Users should have the control to modify the classification through their feedback • UCL‟s E-Society Classification
  14. 14. Public Consultation Feedback
  15. 15. Real time Geodemographics
  16. 16. Need for real time Geodemographics • Current classifications are created using static data sources • Rate and scale of current population change is making large surveys (census) increasingly redundant • Significant hidden value in transactional data • Data is increasingly available in near real time e.g. ONS (Office of National Statistics) NESS API • Social media data is available in real time
  17. 17. What are real time Geodemographics ? Specification Real time feeds of data Estimation Online Specification of inputs Clustering Testing Visualisation
  18. 18. Computational challenges • Integration of large and possibly disparate databases • E.g. NHS data; Census data • Data normalisation and optimization for fast transactions • Minimizing computational time of clustering algorithms (Very Important)! • Common protocol • XML (SOAP) • Use of non traditional data sources. (Singleton, 2008) • E.g. Flickr; Facebook, Twitter
  19. 19. Important Challenge: Selection of clustering algorithm • • • • K-Means PAM (Partitioning Around Medoids) CLARA (Clustering Large Applications) GA (Genetic Algorithm)
  20. 20. k-means • Widely used clustering algorithm for geodemographics • Attempts to find out cluster centroids by minimising within sum of squares distance. • K-means is unstable due to its initial seeds assignment. • Sensitive to outliers in the data set. • Creating a Geodemographic classification requires running algorithm multiple times. • 10,000 times (Singleton, 2008) • Computationally expensive in a real time environment.
  21. 21. An example of bad clustering result (K-means)
  22. 22. An example of bad clustering result (K-means)
  23. 23. An example of bad clustering result (K-means)
  24. 24. Alternate Clustering Algorithms • PAM (Partitioning around medoids) • CLARA (Clustering Large Applications) • GA (Genetic Algorithm)
  25. 25. Alternate Clustering Algorithms… • PAM (Partitioning around medoids) • It tries to minimize the sum of dissimilarities of the data points to their cluster centers. • Less sensitive to outliers than K-means. • Cannot handle larger data sets. • Produces better results than k-means for smaller data sets.
  26. 26. Alternate Clustering Algorithms… • CLARA (Clustering Large Applications) • It draws multiple samples of the dataset, applies PAM to each sample and returns the best result. • Can handle large data sets as it operates on samples rather than on actual data set. • Could be a better choice for creating classifications on the fly.
  27. 27. Alternate Clustering Algorithms… • GA (Genetic Algorithm) • It is inspired by models of biological evolution. It produces results through a breeding procedure. • Creates hierarchies of generations and then merge the hierarchies in homogeneous groups having similar characteristics. • Can be time consuming due to the creation of generation hierarchies.
  28. 28. Comparing computational efficiency (Z-scores) OA (Output Area) level results LSOA (Lower Super Output Area) level results Ward level results
  29. 29. Algorithm Stability (w.r.t. Computational time) Running k-means on OA (Output Area) for 120 times on each iteration 4 3.5 3 2.5 2 1.5 1 0.5 0 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 Time (s) K-means Running GA on OA (Output Area) for 120 times on each iteration Running CLARA on OA (Output Area) for 120 times on each iteration GA 4 3.5 3 2.5 2 1.5 1 0.5 0 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 Time (s) 4 3.5 3 2.5 2 1.5 1 0.5 0 1 5 9 13 17 21 25 29 33 37 41 45 49 53 57 61 65 69 73 77 81 85 89 93 97 Time (s) CLARA
  30. 30. Bespoke Real-time Geodemographics Data • • • • Realtime Measurement Specify inputs and weights Data normalisation Clustering Visualisation Bespoke Requests
  31. 31. GeodemCreator: A software for creating Geodemographic Classifications in near real-time.
  32. 32. GeodemCreator • Allows users to create Geodemographic Classifications • Users have the control of how a Geodemographic Classification is created (Open Methods !)
  33. 33. Building a Geodemographic Classification • Step-1: Choose a dataset
  34. 34. Building a Geodemographic Classification • Step-2: Check Correlation of the Variables
  35. 35. Building a Geodemographic Classification • Step-3: Select variables
  36. 36. Building a Geodemographic Classification • Step-4: Specify „number of clusters‟ and „spatial area‟ Number of Clusters Spatial Area
  37. 37. Building a Geodemographic Classification • Step-5: Build the Classification
  38. 38. Building a Geodemographic Classification • Output – Cluster Numbers
  39. 39. Building a Geodemographic Classification • Output
  40. 40. Social Media data for Geodemographics
  41. 41. Why we need Social Media data for Geodemographics ? • Traditional geodemographic classifications are based on Census data • Night time geography • These classifications do not identify where the population is during the day time • We do not know about the Social links between different people • A solution is to infuse Social Media data with traditional data sources
  42. 42. Geodemographics • “Analysis of people by where they live” or “locality marketing” Social Media Geodemographics • “Analysis of people by where they live, travel, and who they communicate with”
  43. 43. Social Media Geodemographics • Who: Ethnicity, Gender, and Age of social media users • Where: Where social media conversations are happening and who is leading them • Intelligence about where people are located and what they are doing • When: What time of day conversations happen
  44. 44. Twitter (www.twitter.com) • Online social-networking and micro blogging service • Launched in 2006 • Users can send messages of 140 characters or less • Approximately 200 million active users • 350 million tweets daily • In 2012, UK and London were ranked 4th and 3rd, respectively, in terms of the number of posted tweets
  45. 45. Data available through the Twitter API • • • • • • • • • User Creation Date Followers Friends User ID Language Location Name Screen Name Time Zone • • • • • Geo Enabled Latitude Longitude Tweet date and time Tweet text
  46. 46. Analysing Names on Twitter • Some examples of NAME variations on Twitter Real Names Kevin Hodge Andre Alves Jose de Franco Carolina Thomas, Dr. Prof. Martha Del Val Fabíola Sanchez Fernandes Fake Names Castor 5. WHAT IS LOVE? MysticMind KIRILL_aka_KID Vanessa Petuna
  47. 47. Tweeting Activity by different Ethnic Groups
  48. 48. Genders of Twitter Users
  49. 49. Age distribution of Twitter Users vs 2011 Census
  50. 50. Summary • Geodemographics is the analysis of people by where they live • But generalised geodemographic classifications have some problems – We need bespoke classifications for smaller areas • Real-time geodemographic classifications is a solution to create bespoke classifications • Methods of creating current classifications are not open – We need Open tools and Open methods for geodemographics • Social media data for geodemographic classifications
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×