Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Multi-thematic spatial databases

1,131 views

Published on

Published in: Technology, Education
  • Be the first to comment

  • Be the first to like this

Multi-thematic spatial databases

  1. 1. Multi-Thematic Spatial Databases Experience designing and implementing Dr. Conor Mc Elhinney Dr. Paul Lewis Postdoctoral Researcher Mobile Mapping Group
  2. 2. What we do Store
  3. 3. What we do Store Access
  4. 4. What we do Store Access Process
  5. 5. What we do Store Access Process Visualise
  6. 6. What we do Store Access Voluminous Geospatial Data Process Visualise
  7. 7. Mobile Mapping Systems Group 1 Senior Researcher 2 Post-docs 2 PhDs Computer Science, GIS, Surveying
  8. 8. Mobile Mapping Systems Group
  9. 9. Mobile Mapping Systems Group
  10. 10. Mobile Mapping Systems Group
  11. 11. Mobile Mapping Systems Group
  12. 12. i2maps Group 1 Senior Researcher 1 Post-doc 2 PhDs +1 PHD GeoCrowd Computer Science, Maths, GIS Dr. Alexei Pozdnoukhov, Dr. Christian Kaisler Fergal Walsh
  13. 13. Learning from data Streams i2maps What we do at NCG: Scalable methods of spatial analytics
  14. 14. Learning from data Streams i2maps What we do at NCG: Scalable methods of spatial analytics Machine learning and data mining
  15. 15. Learning from data Streams i2maps What we do at NCG: Scalable methods of spatial analytics Machine learning and data mining Stream Handler {x, y} M f (.) i K (., xi ) Analysis and i 1 Modeling new { i} (.), ( xi ) H Dictionary of models MapReduce xOLD
  16. 16. Learning from data Streams i2maps What we do at NCG: Scalable methods of spatial analytics Machine learning and data mining Distributed approaches to spatial statistics Stream Handler {x, y} M f (.) i K (., xi ) Analysis and i 1 Modeling new { i} (.), ( xi ) H Dictionary of models MapReduce xOLD
  17. 17. What we have experience with Handling and proccessing of TBs of
  18. 18. What we have experience with Handling and proccessing of TBs of Multi-thematic data
  19. 19. What we have experience with Handling and proccessing of TBs of Multi-thematic data Temporal data
  20. 20. What we have experience with Handling and proccessing of TBs of Multi-thematic data Temporal data Multi-sensor data
  21. 21. What we have experience with Handling and proccessing of TBs of Multi-thematic data Temporal data Multi-sensor data using spatial information
  22. 22. Data Handling
  23. 23. Data Handling
  24. 24. LiDAR A laser scanning tech, laser radar
  25. 25. LiDAR A laser scanning tech, laser radar > 20Gbs an hour
  26. 26. LiDAR A laser scanning tech, laser radar > 30Gbs an hour after geocoding
  27. 27. LiDAR A laser scanning tech, laser radar > 30Gbs an hour after geocoding > 6 attributes output from scanner
  28. 28. LiDAR A laser scanning tech, laser radar > 30Gbs an hour after geocoding > 6 attributes output from scanner Between 1 pt/m2 and 2000 pt/m2
  29. 29. LiDAR
  30. 30. LiDAR
  31. 31. LiDAR
  32. 32. LiDAR
  33. 33. LiDAR
  34. 34. Imagery Developing trend to store as videos / blobs
  35. 35. Imagery Developing trend to store as videos / blobs Store metadata in SDB
  36. 36. Imagery Developing trend to store as videos / blobs Store metadata in SDB Using HTML5 or queries can link to frames
  37. 37. User Generated Comments
  38. 38. User Generated Comments Video / Imagery
  39. 39. User Generated Comments Video / Imagery Opinion
  40. 40. User Generated Comments Video / Imagery Opinion
  41. 41. Twitter
  42. 42. Twitter
  43. 43. Twitter 25% contain links
  44. 44. Facebook 500 Million Active Users
  45. 45. Facebook 250 Million Active Mobile Users
  46. 46. Facebook 10 Million pieces of content per day
  47. 47. Hard Drive Capacity
  48. 48. Hard Drive Capacity Capacity is increasing linearly
  49. 49. CPU vs HD speed CS111 UCLA 2006
  50. 50. CPU vs HD speed We can process more than we can store CS111 UCLA 2006
  51. 51. CPU vs HD speedIDC - The Diverse and Exploding Digital Universe CS111 UCLA 2006
  52. 52. What next Process the data in real time
  53. 53. What next Process the data in real time Extract or compress to find a model for the relevant data
  54. 54. What next Process the data in real time Extract or compress to find a model for the relevant data Store this model for future processing
  55. 55. What next Process the data in real time Extract or compress to find a model for the relevant data Store this model for future processing Leads to the problem of what do we model and store?
  56. 56. Enabling GeoSpatial Data
  57. 57. Processing data streams Human activity on geo-referenced communication networks
  58. 58. Processing data streams Human activity on geo-referenced communication networks
  59. 59. Processing data streams Human activity on geo-referenced communication networks At least two categories we need to understand:
  60. 60. Processing data streams Human activity on geo-referenced communication networks At least two categories we need to understand: Dynamics of links
  61. 61. Processing data streams Human activity on geo-referenced communication networks At least two categories we need to understand: Dynamics of links Activity level at nodes
  62. 62. Enabling data speak for themselves Air Quality Sensor Weather Measurements VGI Feed (e.g. Twitter) Surveillance Camera SMS Web Page XML Video Push Polling Stream Stream Data Receiver Data Crawler Stream Handler Stream HandlerStatic Data Static Data Static Data Analysis and Spatial Database Modeling Spatio-Temporal Data i2maps Web Service KML/CSV/etc GeoJSON Spatio-Temporal Queries Interactive Spatio-Temporal Information Visualiser
  63. 63. Enabling data speak for themselves Air Quality Sensor Weather Measurements VGI Feed (e.g. Twitter) Surveillance Camera SMS Web Page XML Video Push Polling Stream Stream Data Receiver Data Crawler Stream Handler Stream Handler SpatialStatic Data Static Data Static Data Analysis andDatabase Spatial Database Dictionary of models Modeling Spatio-Temporal Data i2maps Web Service KML/CSV/etc GeoJSON Spatio-Temporal Queries Interactive Spatio-Temporal Information Visualiser
  64. 64. Storage
  65. 65. What exists Files / DBs / SDBs
  66. 66. What exists Files / DBs / SDBs Files still extremely common
  67. 67. What exists Files / DBs / SDBs Files still extremely common SDBs are what is needed
  68. 68. What exists Files / DBs / SDBs Files still extremely common SDBs are what is needed Multi-source, sensor, type data
  69. 69. Our Aims Unified approach to storing multi- thematic data
  70. 70. Our Aims Unified approach to storing multi- thematic data Efficient data upload / access/ storage
  71. 71. Our Aims Unified approach to storing multi- thematic data Efficient data upload / access/ storage Searchable in Time/ Space / by Attributes
  72. 72. Our Aims Unified approach to storing multi- thematic data Efficient data upload / access/ storage Searchable in Time/ Space / by Attributes Incorporating Visualisations into all solutions
  73. 73. Our hardware 3 Processing Servers 8 Intel Xeons, 2.1- 2.8 GHz 72 GBs RAM
  74. 74. Our hardware 3 Processing Servers 8 Intel Xeons, 2.1- 2.8 GHz 72 GBs RAM 1 Storage Server 7TBs Raided Drives
  75. 75. Our Developed Systems LiDAR / Image based SDB
  76. 76. Our Developed Systems LiDAR / Image based SDBGeoComputation Platform
  77. 77. Database storage experienceOptimisation of upload of large (GBs) spatial files to SDB.
  78. 78. Database storage experienceDatabase optimisation to suit system architecture
  79. 79. Database storage experience Storage of multiple data types/sources
  80. 80. Watch out for Spatial Index size V RAM
  81. 81. Watch out for Spatial Index size V RAM Expected no. of concurrent users
  82. 82. Watch out for Spatial Index size V RAM Expected no. of concurrent users HD capacity V daily data throughput

×