Your SlideShare is downloading. ×
0
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Real Time Geodemographics
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Real Time Geodemographics

957

Published on

This presentation is a comparison of different clustering based on their computational time. This is the first step in creating open source and bespoke Geodemographic classifications in near real …

This presentation is a comparison of different clustering based on their computational time. This is the first step in creating open source and bespoke Geodemographic classifications in near real time.

Published in: Education, Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
957
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
3
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Real time Geodemographics: Requirements and Challenges Muhammad Adnan, Paul Longley
  • 2. Current Geodemographic classifications
    • Census data
        • E.g. OA (Output Area) dataset has 41 census variables.
    • Variables are weighted according to their importance in classification.
    • K-means clustering algorithm is used to cluster data into homogeneous groups.
        • Multiple runs of K-means due to its un-stability
        • 10,000 times (Singleton, 2008)
  • 3. Need for real time Geodemographics
    • Current classifications are created using static data sources.
    • Rate and scale of current population change is making large surveys (census) increasingly redundant.
        • Significant hidden value in transactional data
    • Data is increasingly available in near real time
    • e.g. ONS NESS API
    • Application specific (bespoke) classifications have demonstrated utility (Longley & Singleton, 2009).
  • 4. What are real time Geodemographics ? Specification Estimation Testing
  • 5. Computational challenges
    • Integration of large and possibly disparate databases.
        • E.g. NHS data; Census data
    • Data normalisation and optimization for fast transactions.
    • Minimizing computational time of clustering algorithms (Very Important)!
    • Common protocol
        • XML (SOAP)
    • Use of non traditional data sources. (Singleton, 2008)
        • E.g. Flickr; Facebook
  • 6. Important Challenge: Selection of clustering algorithm
    • K-Means
    • PAM (Partitioning Around Medoids)
    • CLARA (Clustering Large Applications)
    • GA (Genetic Algorithm)
  • 7. K-means
    • Attempts to find out cluster centroids by minimising within sum of squares distance.
    • K-means is unstable due to its initial seeds assignment.
        • Sensitive to outliers.
    • Creating a Geodemographic classification requires running algorithm multiple times.
        • 10,000 times (Singleton, 2008)
        • Computationally expensive in a real time environment.
  • 8. K-means (100 runs of k-means on OAC data set for k=4)
  • 9. An example of bad clustering result (K-means)
  • 10. An example of bad clustering result (K-means)
  • 11. An example of bad clustering result (K-means)
  • 12. Alternate Clustering Algorithms
    • PAM (Partitioning around medoids) tries to minimize the sum of distances of the objects to their cluster centers.
        • Less sensitive to outliers than K-means.
        • Cannot handle larger data sets.
    • CLARA (Clustering Large Applications) draws multiple samples of the dataset, applies PAM to each sample and returns the best result.
    • GA (Genetic Algorithm) is inspired by models of biological evolution. It produces results through a breeding procedure.
  • 13. This paper compares
    • K-means
    • Clara
    • GA
    • By using three data normalisation techniques
    • Z-Scores
    • Range Standardisation
    • Principle Component Analysis.
    • Algorithm stability of K-means, Clara, and GA
  • 14. Data normalisation techniques used
    • Z-Scores
        • Widely used variable normalisation technique
        • Can create outliers in the datasets
    • Range Standardisation
        • Standardise values between a range of 0-1
        • Can erase interesting patterns in the data
    • Principle Component Analysis.
        • Reduces the dimensions of a data set
        • Can erase interesting patterns in the data
  • 15. Comparing computational efficiency (Z-scores) PAM, and GA on the three geographic aggregations of a dataset covering London. Figure 1: OA (Output Area) level results Figure 2 : LSOA (Lower Super Output Area) level results Figure 3 : Ward level results
  • 16. Comparing computational efficiency (Range Standardisation) PAM, and GA on the three geographic aggregations of a dataset covering London. Figure 4: OA (Output Area) level results Figure 5 : LSOA (Lower Super Output Area) level results Figure 6 : Ward level results
  • 17. Comparing computational efficiency (PCA) PAM, and GA on the three geographic aggregations of a dataset covering London. Figure 7: OA (Output Area) level results Figure 8 : LSOA (Lower Super Output Area) level results Figure 9 : Ward level results
  • 18. Algorithm Stability (w.r.t. Computational time) Figure 10: Running k-means on OA (Output Area) for 120 times on each iteration Figure 11: Running CLARA on OA (Output Area) for 120 times on each iteration Figure 12: Running GA on OA (Output Area) for 120 times on each iteration
  • 19. K-means and Principle Component Analysis
    • PCA can be used to facilitate K-means clustering by reducing dimensions.
    • (Ding, C., He, X., 2004)
    Figure 13: K-means result for 41 “OAC variables” Figure 14: K-means result for 26 “OAC Principle Components” K=4 (99% similar)
  • 20. K-means and Principle Component Analysis
    • PCA can be used to facilitate K-means clustering by reducing dimensions.
    • (Ding, C., He, X., 2004)
    Figure 13: K-means result for 4 1 “OAC variables” Figure 14: K-means result for 26 “OAC Principle Components”
  • 21. Conclusion
    • Clara is plausible alternative to k-means in a real time Geodemographic classification system.
    • K-means might be combined with PCA for enhanced computation power.
    • In an online environment k-means is better for small data sets.
    • Exploration of non traditional data sources.

×