Mobile Terrestrial LiDAR Datasets             in a Spatial Database Framework                        Dr. Conor Mc Elhinney...
Mobile Mapping Systems Group     Develop automated processing     algorithms for MMS data.
Mobile Mapping Systems Group     Develop automated processing     algorithms for MMS data.
Mobile Mapping Systems Group     Develop automated processing     algorithms for MMS data.
Mobile Mapping Systems Group     Develop automated processing     algorithms for MMS data.
Survey based               LiDAR folder
Survey based                              LiDAR folder      Survey 10 Apr Block 1 Block 2Block 3    .    .Block N    .Meta...
Survey based                                         LiDAR folder      Survey 10 Apr                  Survey 5 Dec Block 1...
Survey based                                         LiDAR folder      Survey 10 Apr                  Survey 5 Dec        ...
Survey based               LiDAR folder
Survey based               LiDAR folder
Survey based     Give me the data from Dublin city                   LiDAR folder
Survey based     Do we have data in a given area?                   LiDAR folder
Survey based     Give me 10mx5m cross sections at     5m intervals                  LiDAR folder
Storage
Storage
Storage          Database
Storage          Database
Storage          Database
Storage     PostgreSQL     PostGIS               Database
Storage          PostGIS    PostgreSQL          Database
Storage          Spatial Database
Storage   DB Index: 3D point          Spatial Database
Data Handling and Upload
Why is upload an issue      50Km – 1 way
Why is upload an issue                         50GB      50Km – 1 way
Why is upload an issue                         50GB      50Km – 1 way                          500                     Mil...
Why is upload an issue                    80Km – 1 way
Why is upload an issue        130GB                    80Km – 1 way
Why is upload an issue        130GB                     80Km – 1 way        1,300    million points
First tests      DB on powerful desktop
First tests      DB on powerful desktop      > 60m records
First tests      DB on powerful desktop      > 60m records      > 4hrs to upload
First tests      DB on powerful desktop   Small survey >40hrs      > 60m records    > 4hrs to upload time        upload
Our hardware          1 Processing Server          8 Intel Xeons, 2.8 GHz          32 GBs RAM          1 Storage Server   ...
Experiments
ComparedPostgreSQL Copy     with  pg_bulkload
Test files      LiDAR data over 66m rows      2 files:                 10 columns -> 4.4Gbs                 14 columns -> ...
Postgresql upload     1. Create Table     2. Load data     3. Create Geometry Column     4. Update Geometry Column     5. ...
Our upload process     1. Pre-process - Python     2. Create Table     3. Create Geometry Column     4. Create Index     5...
Time per row                                                            PostgreSQL                                        ...
Time per row                                               Exp 1                                                          ...
Row size impacts time                                                     Exp 1                                           ...
Row size impacts time                                                     Exp 1                                           ...
Row size impacts time                                                     Exp 1                                           ...
Row size impacts time                                                     Exp 1                                           ...
Time difference     1. Our Method      >40 %                     2. pg_bulkload                     >12 %
Benefit?
60         1.5                      1.5                      Copy Orig                      Copy                      Dire...
60         1.5                       1.5                      Copy Orig                      Copy                      Dir...
60         1.5                       1.5                      Copy Orig                      Copy                      Dir...
60                                                            >24hrs         1.5                       1.5                ...
60                                                             >24hrs         1.5                        1.5              ...
60                                                             4hrs         1.5                        1.5                ...
Back to the big picture
Access            “Mobile Mapping System LiDAR Data                             Framework “                           3D G...
Access            “Mobile Mapping System LiDAR Data                             Framework “                           3D G...
Access            “Mobile Mapping System LiDAR Data                             Framework “                           3D G...
Access            “Mobile Mapping System LiDAR Data                             Framework “                           3D G...
Access         Spatial Database
Access         Spatial Database
Access         Spatial Database
Process          Spatial Database
Process          Spatial Database
Process                         > 140km          Spatial Database
Visualising
Conclusions         Store         Access                     Automatically         Process         Visualise
Future work     Finalise DB schema     Formalise DB Upload / Access     Release Free Mobile Mapping     Spatial DB     Aut...
Questions      Conor Mc Elhinney, Paul Lewis, Tim McCarthy               conormce@cs.nuim.ie
Upcoming SlideShare
Loading in …5
×

Mobile Mapping Spatial Database Framework

1,597 views
1,479 views

Published on

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,597
On SlideShare
0
From Embeds
0
Number of Embeds
270
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Mobile Mapping Spatial Database Framework

  1. 1. Mobile Terrestrial LiDAR Datasets in a Spatial Database Framework Dr. Conor Mc Elhinney Postdoctoral Researcher Mobile Mapping Group 7th MMT 16th June 2011
  2. 2. Mobile Mapping Systems Group Develop automated processing algorithms for MMS data.
  3. 3. Mobile Mapping Systems Group Develop automated processing algorithms for MMS data.
  4. 4. Mobile Mapping Systems Group Develop automated processing algorithms for MMS data.
  5. 5. Mobile Mapping Systems Group Develop automated processing algorithms for MMS data.
  6. 6. Survey based LiDAR folder
  7. 7. Survey based LiDAR folder Survey 10 Apr Block 1 Block 2Block 3 . .Block N .MetaData: Geo Bounds, date,processing done
  8. 8. Survey based LiDAR folder Survey 10 Apr Survey 5 Dec Block 1 Block 1 Block 2 Block 2Block 3 Block 3 . . . .Block N . Block N .MetaData: Geo Bounds, date, MetaData: Geo Bounds, date,processing done processing done
  9. 9. Survey based LiDAR folder Survey 10 Apr Survey 5 Dec Survey 2 May Block 1 Block 1 Block 1 Block 2 Block 2 ....... Block 2Block 3 Block 3 Block 3 . . . . . .Block N . Block N . Block N .MetaData: Geo Bounds, date, MetaData: Geo Bounds, date, MetaData: Geo Bounds, date,processing done processing done processing done
  10. 10. Survey based LiDAR folder
  11. 11. Survey based LiDAR folder
  12. 12. Survey based Give me the data from Dublin city LiDAR folder
  13. 13. Survey based Do we have data in a given area? LiDAR folder
  14. 14. Survey based Give me 10mx5m cross sections at 5m intervals LiDAR folder
  15. 15. Storage
  16. 16. Storage
  17. 17. Storage Database
  18. 18. Storage Database
  19. 19. Storage Database
  20. 20. Storage PostgreSQL PostGIS Database
  21. 21. Storage PostGIS PostgreSQL Database
  22. 22. Storage Spatial Database
  23. 23. Storage DB Index: 3D point Spatial Database
  24. 24. Data Handling and Upload
  25. 25. Why is upload an issue 50Km – 1 way
  26. 26. Why is upload an issue 50GB 50Km – 1 way
  27. 27. Why is upload an issue 50GB 50Km – 1 way 500 Million points
  28. 28. Why is upload an issue 80Km – 1 way
  29. 29. Why is upload an issue 130GB 80Km – 1 way
  30. 30. Why is upload an issue 130GB 80Km – 1 way 1,300 million points
  31. 31. First tests DB on powerful desktop
  32. 32. First tests DB on powerful desktop > 60m records
  33. 33. First tests DB on powerful desktop > 60m records > 4hrs to upload
  34. 34. First tests DB on powerful desktop Small survey >40hrs > 60m records > 4hrs to upload time upload
  35. 35. Our hardware 1 Processing Server 8 Intel Xeons, 2.8 GHz 32 GBs RAM 1 Storage Server 7TBs Raided Drives
  36. 36. Experiments
  37. 37. ComparedPostgreSQL Copy with pg_bulkload
  38. 38. Test files LiDAR data over 66m rows 2 files: 10 columns -> 4.4Gbs 14 columns -> 6.8Gbs
  39. 39. Postgresql upload 1. Create Table 2. Load data 3. Create Geometry Column 4. Update Geometry Column 5. Create Index 6. Vacuum Table
  40. 40. Our upload process 1. Pre-process - Python 2. Create Table 3. Create Geometry Column 4. Create Index 5. Load data
  41. 41. Time per row PostgreSQL Exp 1 Exp 2 Exp 3 Copy 10 pg_bulkload columns 0.025 0.05 0.075 0.1 Time per row (ms) Copy 0.25 0.5 0.75 1 1.25 14 pg_bulkload columns 0.025 0.05 0.075 0.1 Time per row (ms)
  42. 42. Time per row Exp 1 PostgreSQL Exp 1 2 New Exp 2 3 Exp 3 Copy 10 pg_bulkload columns 0.025 0.05 0.075 0.1 Time per row (ms) Copy 0.25 0.25 0.5 0.5 0.75 0.75 1 1 1.25 1.25 14 pg_bulkload columns 0.025 0.05 0.075 0.1 Time per row (ms)
  43. 43. Row size impacts time Exp 1 PostgreSQL 1 Exp 2 New2 Exp 3 Exp 3 Copy 10 pg_bulkload columns 0.25 0.5 0.75 1 1.25 Time per KB (ms) Copy 0.25 0.25 0.5 0.5 0.75 0.75 1 1 1.25 1.25 14 pg_bulkload columns 0.25 0.5 0.75 1 1.25 Time per KB (ms)
  44. 44. Row size impacts time Exp 1 PostgreSQL 1 Exp 2 New2 Exp 3 Exp 3 Copy 10 pg_bulkload columns 0.25 0.5 0.75 1 1.25 Time per KB (ms) Copy 0.25 0.25 0.5 0.5 0.75 0.75 1 1 1.25 1.25 14 pg_bulkload columns 0.25 0.5 0.75 1 1.25 Time per KB (ms)
  45. 45. Row size impacts time Exp 1 PostgreSQL 1 Exp 2 New2 Exp 3 Exp 3 Copy 10 pg_bulkload columns 0.25 0.5 0.75 1 1.25 Time per KB (ms) Copy 0.25 0.25 0.5 0.5 0.75 0.75 1 1 1.25 1.25 14 pg_bulkload columns 0.25 0.5 0.75 1 1.25 Time per KB (ms)
  46. 46. Row size impacts time Exp 1 PostgreSQL 1 Exp 2 New2 Exp 3 Exp 3 Copy 10 pg_bulkload As row size 0.25 0.5 0.75 1 1.25 columns Time per kb Time per KB (ms) Copy 0.25 0.25 0.5 0.5 0.75 0.75 1 1 1.25 1.25 14 pg_bulkload columns 0.25 0.5 0.75 1 1.25 Time per KB (ms)
  47. 47. Time difference 1. Our Method >40 % 2. pg_bulkload >12 %
  48. 48. Benefit?
  49. 49. 60 1.5 1.5 Copy Orig Copy Direct 50 Parallel 1 40 1Time (hours) 30 0.5 0.5 20 0 10 0 0 0 500 1000 1500 2000 Rows (millions)
  50. 50. 60 1.5 1.5 Copy Orig Copy Direct New Copy 50 Parallel 1 40 1Time (hours) 30 0.5 0.5 20 0 10 0 0 0 500 1000 1500 2000 Rows (millions)
  51. 51. 60 1.5 1.5 Copy Orig Copy Direct New Copy 50 Parallel 1 40 1Time (hours) 30 0.5 0.5 20 0 10 0 0 0 500 1000 1500 2000 Rows (millions)
  52. 52. 60 >24hrs 1.5 1.5 Copy Orig Copy Direct New Copy 50 Parallel 1 40 1Time (hours) 30 0.5 0.5 20 0 10 0 0 0 500 1000 1500 2000 Rows (millions)
  53. 53. 60 >24hrs 1.5 1.5 Copy Orig Copy Direct New Copy 50 PG_bulkload Parallel 1 40 1Time (hours) 30 0.5 0.5 20 0 10 0 0 0 500 1000 1500 2000 Rows (millions)
  54. 54. 60 4hrs 1.5 1.5 Copy Orig Copy Direct New Copy 50 PG_bulkload Parallel 1 40 1Time (hours) 30 0.5 0.5 20 0 10 0 0 0 500 1000 1500 2000 Rows (millions)
  55. 55. Back to the big picture
  56. 56. Access “Mobile Mapping System LiDAR Data Framework “ 3D GeoInfo 2010 Spatial Database
  57. 57. Access “Mobile Mapping System LiDAR Data Framework “ 3D GeoInfo 2010 Spatial Database
  58. 58. Access “Mobile Mapping System LiDAR Data Framework “ 3D GeoInfo 2010 Spatial Database
  59. 59. Access “Mobile Mapping System LiDAR Data Framework “ 3D GeoInfo 2010 Spatial Database
  60. 60. Access Spatial Database
  61. 61. Access Spatial Database
  62. 62. Access Spatial Database
  63. 63. Process Spatial Database
  64. 64. Process Spatial Database
  65. 65. Process > 140km Spatial Database
  66. 66. Visualising
  67. 67. Conclusions Store Access Automatically Process Visualise
  68. 68. Future work Finalise DB schema Formalise DB Upload / Access Release Free Mobile Mapping Spatial DB Automated algorithms
  69. 69. Questions Conor Mc Elhinney, Paul Lewis, Tim McCarthy conormce@cs.nuim.ie

×