Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Big size meteorological data processing 
and mobile displaying system 
using PostGIS and GeoServer 
BJ Jang, JW Geum, JH K...
 The system was SLOW not because of using PostGIS 
but because of NOT TUNING. 
 PostGIS definitely could make good perfo...
Background 
3
background Problems to solved Import speed Keeping data size indexing 
Mobile Weather Chart Service Flow 
Observation 
Dat...
background Problems to solved Import speed Keeping data size indexing 
Vector Weather 
Chart 
5 
Software Architecture
background Problems to solved Import speed Keeping data size indexing 
Characteristics of Weather Data 
6 
Low Resolution ...
background Problems to solved Import speed Keeping data size indexing 
Data usage per day 
times generation (00, 06, 12, 1...
Problems to be 
solved 
8
Problems of Existing System 
Slow data 
collection 
Difficult big 
size data 
management 
Slow searching 
for 
Weather Cha...
Why is the service slow? 
Failed to understand 
characteristics of data 
10 
background Problems to solved Import speed Ke...
background Problems to solved Import speed Keeping data size indexing 
Improvement Goal 
11 
PRGOOBALLESMS AGCOTIAVLITSY 
...
Improvement on importing 
speed for big size data 
using batch 
12
General Data Processing Time 
The time required each batch size 
source: http://novathin.kr/19 
Run one by one 
Run one ti...
background Problems to solved Import speed Keeping data size indexing 
Import speed comparison 
One weather chart kml file...
Keeping data file 
size 
by managing table 
15
Data Management of PostGIS 
 PostGIS is write-once. 
 Not deleting updated and deleted data 
 Recording new data below ...
Snapshot vs Write-once 
Oracle / MySQL PostgreSQL 
table 
A 
B’ 
C 
D 
E 
table 
A 
B X 
C 
D 
E 
B’ 
snapsho 
t 
B 
Trans...
General VACUUM 
Table 
A 
B X 
C X 
D 
E 
X 
B’ 
C’ 
Table 
A 
B X 
C X 
D 
E 
X 
B’ 
C’ 
Table 
A 
F 
C X 
D 
E 
X 
B’ 
C...
background Problems to solved Import speed Keeping data size indexing 
VACUUM FULL 
unused space 
arrange for big size 
da...
Partitioning 
 Partitioning? 
 Managing tables by conceptually separating one table to 
several 
 Data size by table do...
Improvement on inquiry 
speed by resetting index 
21
background Problems to solved Import speed Keeping data size indexing 
Improvement flow of inquiry speed 
22 
Data 
Condit...
background Problems to solved Import speed Keeping data size indexing 
Data Condition Analysis 
 Understanding # of 
colu...
background Problems to solved Import speed Keeping data size indexing 
GeoServer SQL VIEW 
 Register sql query as Layer 
...
background Problems to solved Import speed Keeping data size indexing 
Query Finding 
 Identifying executed 
SQL using st...
background Problems to solved Import speed Keeping data size indexing 
Query Plan Analysis 
 PostGIS has basically query ...
background Problems to solved Import speed Keeping data size indexing 
Index Improvement 
 Principles for Index Improveme...
Improvement 
Result 
28
29 
Under 300 
isobaric/ 
Temperature/ 
Isokinetics 
Ground/ 
Wet-number/ 
Temperature 
800 isobaric/ 
Mixture ratio/ 
Tem...
Conclusion 
Importance on excution 
using addBatch() and 
excuteBatch() 
About 100 times 
performance 
improvement 
Mixed ...
Conclusion 
PostGIS is really great 
DBMS! 
Perfectly suited with 
GeoServer 
However, tuning with 
perfect understandi...
Q&A 
Please Ask BJ Jang via 
Email ! 
bjjang@gaia3d.com 
32
Upcoming SlideShare
Loading in …5
×

Big size meteorological data processing and mobile displaying system using PostGIS and GeoServer

1,662 views

Published on

Present on the FOSS4G 2014 in Portland

Published in: Software
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://soo.gd/qURD } ......................................................................................................................... Download Full EPUB Ebook here { https://soo.gd/qURD } ......................................................................................................................... Download Full doc Ebook here { https://soo.gd/qURD } ......................................................................................................................... Download PDF EBOOK here { https://soo.gd/qURD } ......................................................................................................................... Download EPUB Ebook here { https://soo.gd/qURD } ......................................................................................................................... Download doc Ebook here { https://soo.gd/qURD } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book THIS can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer THIS is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story THIS Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money THIS the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths THIS Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THIS BOOKS INTO AVAILABLE FORMAT (Unlimited) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download Full EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ACCESS WEBSITE for All Ebooks ......................................................................................................................... Download Full PDF EBOOK here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download EPUB Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... Download doc Ebook here { https://tinyurl.com/yyxo9sk7 } ......................................................................................................................... ......................................................................................................................... ......................................................................................................................... .............. Browse by Genre Available eBooks ......................................................................................................................... Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult,
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Big size meteorological data processing and mobile displaying system using PostGIS and GeoServer

  1. 1. Big size meteorological data processing and mobile displaying system using PostGIS and GeoServer BJ Jang, JW Geum, JH Kwun, HG Park
  2. 2.  The system was SLOW not because of using PostGIS but because of NOT TUNING.  PostGIS definitely could make good performance if the system has been PROPERLY TUNED.  Let’s go over some tuning skills from CASE OF MOBILE weather chart service of KMA. 2 Objective
  3. 3. Background 3
  4. 4. background Problems to solved Import speed Keeping data size indexing Mobile Weather Chart Service Flow Observation Data Model Improvement of performance part by tuning 4 GRIB Data Vector Chart Chart Service Vectorize Image GRIB Data Weather Chart for service Korea Meteorological Administration Vector Chart ※GRIB DATA : GRIdded Binary or General Regularly-disrtibuted Information in Binary form - standardized by the World Meteorological Organization
  5. 5. background Problems to solved Import speed Keeping data size indexing Vector Weather Chart 5 Software Architecture
  6. 6. background Problems to solved Import speed Keeping data size indexing Characteristics of Weather Data 6 Low Resolution • Geographically low resolution • Surface + Height(Isobaric surface) • Analysis model • Data time + Forecasting time Multiple Dimension • A few times ~ hundreds times Frequent Production • Always need up-to-date data Realtime/ Near Realtime
  7. 7. background Problems to solved Import speed Keeping data size indexing Data usage per day times generation (00, 06, 12, 18 UTC) # of spatial table: # of weather charts: MB data # of spatial data columns:
  8. 8. Problems to be solved 8
  9. 9. Problems of Existing System Slow data collection Difficult big size data management Slow searching for Weather Chart 9 background Problems to solved Import speed Keeping data size indexing
  10. 10. Why is the service slow? Failed to understand characteristics of data 10 background Problems to solved Import speed Keeping data size indexing
  11. 11. background Problems to solved Import speed Keeping data size indexing Improvement Goal 11 PRGOOBALLESMS AGCOTIAVLITSY 5 hr to insert data Data file grows 35 GB per day Tens of seconds to search a single weather chart Inserting less than 3o min. Keep the size of data file fixed Searching a weather chart within a few second
  12. 12. Improvement on importing speed for big size data using batch 12
  13. 13. General Data Processing Time The time required each batch size source: http://novathin.kr/19 Run one by one Run one time after gathering as much as batch size There is big difference according to the way of executing SQL! 13 background Problems to solved Import speed Keeping data size indexing
  14. 14. background Problems to solved Import speed Keeping data size indexing Import speed comparison One weather chart kml file  executing 3,000 columns  test criteria 14 # of addBatch() # of execution Time(sec) 0 3,000 109.0 100 30 8.9 500 6 5.7 1,000 3 3.4 3,000 1 1.1  1 insert / 1 commit  kml file(3,000 insert) / 1 commit
  15. 15. Keeping data file size by managing table 15
  16. 16. Data Management of PostGIS  PostGIS is write-once.  Not deleting updated and deleted data  Recording new data below after marking  Pros  Fast  Can manage several versions of data  Cons  Data file size can be extremely increased  Low performance by increase of file size  Weather Chart DB file increases by 35 GB per day!!! 16 background Problems to solved Import speed Keeping data size indexing
  17. 17. Snapshot vs Write-once Oracle / MySQL PostgreSQL table A B’ C D E table A B X C D E B’ snapsho t B Transaction owner Other users Record before renewal Record after renewal Record after renewal Record before renewal After completing transection 17 background Problems to solved Import speed Keeping data size indexing
  18. 18. General VACUUM Table A B X C X D E X B’ C’ Table A B X C X D E X B’ C’ Table A F C X D E X B’ C’ FSM No need B X C X E X FSM No need C X E X VACUUM execution Data Insert Source: http://www.geocities.jp/sugachan1973/doc/funto60.html In terms of PostGIS for KMA’s weather charts, general vacuum functions can’t solve the problem that data files continuously increase. 18 background Problems to solved Import speed Keeping data size indexing
  19. 19. background Problems to solved Import speed Keeping data size indexing VACUUM FULL unused space arrange for big size data management Source: http://www.devmedia.com.br/otimizacao-uma-ferramenta-chamada-vacuum/1710 On PostGIS for KMA’s weather chart, it takes 15 hr. for full vacuum. During Vacuum full, exclusive LOCK happens. 19 VACUUM FULL
  20. 20. Partitioning  Partitioning?  Managing tables by conceptually separating one table to several  Data size by table down  Index size down and Search speed up Weather Chart Weather Chart _0 Weather Chart _1 Weather Chart _2 Weather Chart _3 Weather Chart _4 Weather Chart _5 Weather Chart _6 Insert on Monday Insert on Sunday Insert on Tuesday Truncate on Tuesday Truncate on Monday Truncate on Sunday Execution time of truncate is almost a few seconds and file size is decreased without vacuum 20 background Problems to solved Import speed Keeping data size indexing
  21. 21. Improvement on inquiry speed by resetting index 21
  22. 22. background Problems to solved Import speed Keeping data size indexing Improvement flow of inquiry speed 22 Data Condition Analysis Query Finding Query Plan Analysis Index Improvement
  23. 23. background Problems to solved Import speed Keeping data size indexing Data Condition Analysis  Understanding # of columns by table  select count(*) table_name is foolish!  Possible to understand the number of rows if using statistical table  Meaningful data is stored on pg_class table  Execution time within one minute select relname as table_name, to_char(reltuples, '999,999,999') as row_count from pg_class where relnamespace = (select oid from pg_namespace where nspname = 'public') and relam = 0 order by 2 desc, 1; 23
  24. 24. background Problems to solved Import speed Keeping data size indexing GeoServer SQL VIEW  Register sql query as Layer  Datasource is geoDB, can use SQL VIEW  Useful  Complex condition to layer  Reprojection  Able to join multiple tables  normal attribute -> spatial object GeoServer , showing weather chart, perfomance is affected by searching speed of PostGIS 24
  25. 25. background Problems to solved Import speed Keeping data size indexing Query Finding  Identifying executed SQL using statistical  Using table pg_stat_activity table  Necessary process for tuning  Possible to check execution time  Differences of queries by PostGIS version select query_start, current_query from pg_stat_activity where username = ‘mobile’ and current_query not like ‘<IDLE>%’ order by query_start desc; SELECT "val",encode(ST_AsBinary(ST_Force_2D("geom ")),'base64') as "geom" FROM ( select mdl, mdl_var, placemark_name, val, lyrs_cd, forecast_time, create_time as anal_time, ST_Transform(the_geom, 7188) as geom from contour where mdl_var = 'TMP' ) as "vtable" WHERE (((("mdl" = 'GDAPS' AND "lyrs_cd" = 'A925.0') AND "forecast_time" = '2011.06.27 00:00') AND "anal_time" = '2011.06.27 00:00') AND "geom" && ST_GeomFromText('POLYGON ((-1056768 - 2105344, -1056768 -1040384, 8192 -1040384, 8192 -2105344, -1056768 -2105344))', 7188)); 25
  26. 26. background Problems to solved Import speed Keeping data size indexing Query Plan Analysis  PostGIS has basically query analysis function  pgAdmin III-Query-Analysis explanation function  Explain Analyze command – Easy to analyze query 26
  27. 27. background Problems to solved Import speed Keeping data size indexing Index Improvement  Principles for Index Improvement  Setting index with all columns on Where clause  Spatial column has separate index  Columns with lots of including data types come first  Possibly, items compared as same operator come first  Unnecessary index should be removed due to bad performance on inserting  Examples -- contour_0 DROP INDEX index_createtime_contour_0; DROP INDEX index_forecasttime_contour_0; DROP INDEX index_lyrscd_contour_0; DROP INDEX index_mdl_contour_0; DROP INDEX index_mdlvar_contour_0; CREATE INDEX index_contour_0_all ON contour_0 (forecast_time ASC NULLS LAST, mdl_var ASC NULLS LAST, lyrs_cd ASC NULLS LAST, create_time DESC NULLS LAST, mdl ASC NULLS LAST);  Result  After individually deleting index, integrated index creation reduces 20% of data capacity  6 ~ 25 times speed improvement by table(big tables show better performance) 27
  28. 28. Improvement Result 28
  29. 29. 29 Under 300 isobaric/ Temperature/ Isokinetics Ground/ Wet-number/ Temperature 800 isobaric/ Mixture ratio/ Temperature
  30. 30. Conclusion Importance on excution using addBatch() and excuteBatch() About 100 times performance improvement Mixed with partitioning and truncate Stably keeping N-1 accordance Appropriate index for query 20 times inquiry time improvement 30
  31. 31. Conclusion PostGIS is really great DBMS! Perfectly suited with GeoServer However, tuning with perfect understanding of the features. 31
  32. 32. Q&A Please Ask BJ Jang via Email ! bjjang@gaia3d.com 32

×