Present on the FOSS4G 2014 in Portland

1. Big size meteorological data processing
and mobile displaying system
using PostGIS and GeoServer
BJ Jang, JW Geum, JH Kwun, HG Park

2.  The system was SLOW not because of using PostGIS
but because of NOT TUNING.
 PostGIS definitely could make good performance if the
system has been PROPERLY TUNED.
 Let’s go over some tuning skills from CASE OF MOBILE
weather chart service of KMA.
2
Objective

3. Background
3

4. background Problems to solved Import speed Keeping data size indexing
Mobile Weather Chart Service Flow
Observation
Data
Model
Improvement of performance part
by tuning
4
GRIB
Data
Vector
Chart
Chart
Service
Vectorize Image
GRIB Data Weather Chart for service
Korea Meteorological Administration
Vector Chart
※GRIB DATA : GRIdded Binary or General Regularly-disrtibuted Information in Binary form
- standardized by the World Meteorological Organization

5. background Problems to solved Import speed Keeping data size indexing
Vector Weather
Chart
5
Software Architecture

6. background Problems to solved Import speed Keeping data size indexing
Characteristics of Weather Data
6
Low Resolution • Geographically low resolution
• Surface + Height(Isobaric surface)
• Analysis model
• Data time + Forecasting time
Multiple
Dimension
• A few times ~ hundreds times
Frequent
Production
• Always need up-to-date data
Realtime/
Near Realtime

7. background Problems to solved Import speed Keeping data size indexing
Data usage per day
times generation (00, 06, 12, 18 UTC)
# of spatial table:
# of weather charts:
MB data
# of spatial data columns:

8. Problems to be
solved
8

9. Problems of Existing System
Slow data
collection
Difficult big
size data
management
Slow searching
for
Weather Chart
9
background Problems to solved Import speed Keeping data size indexing

10. Why is the service slow?
Failed to understand
characteristics of data
10
background Problems to solved Import speed Keeping data size indexing

11. background Problems to solved Import speed Keeping data size indexing
Improvement Goal
11
PRGOOBALLESMS AGCOTIAVLITSY
5 hr to insert data
Data file grows 35 GB per day
Tens of seconds to search a
single weather chart
Inserting less than 3o min.
Keep the size of data file fixed
Searching a weather chart within
a few second

12. Improvement on importing
speed for big size data
using batch
12

13. General Data Processing Time
The time required each batch size
source: http://novathin.kr/19
Run one by one
Run one time after
gathering as much as
batch size
There is big difference according to the way of executing SQL!
13
background Problems to solved Import speed Keeping data size indexing

14. background Problems to solved Import speed Keeping data size indexing
Import speed comparison
One weather chart kml file  executing 3,000 columns  test criteria
14
# of addBatch() # of execution Time(sec)
0 3,000 109.0
100 30 8.9
500 6 5.7
1,000 3 3.4
3,000 1 1.1
 1 insert / 1 commit  kml file(3,000 insert) / 1 commit

15. Keeping data file
size
by managing table
15

16. Data Management of PostGIS
 PostGIS is write-once.
 Not deleting updated and deleted data
 Recording new data below after marking
 Pros
 Fast
 Can manage several versions of data
 Cons
 Data file size can be extremely increased
 Low performance by increase of file size
 Weather Chart DB file increases by 35 GB
per day!!!
16
background Problems to solved Import speed Keeping data size indexing

17. Snapshot vs Write-once
Oracle / MySQL PostgreSQL
table
A
B’
C
D
E
table
A
B X
C
D
E
B’
snapsho
t
B
Transaction owner
Other users
Record
before
renewal
Record
after
renewal
Record
after
renewal
Record
before
renewal
After completing
transection
17
background Problems to solved Import speed Keeping data size indexing

18. General VACUUM
Table
A
B X
C X
D
E
X
B’
C’
Table
A
B X
C X
D
E
X
B’
C’
Table
A
F
C X
D
E
X
B’
C’
FSM
No need
B X
C X
E X
FSM
No need
C X
E X
VACUUM execution Data Insert
Source: http://www.geocities.jp/sugachan1973/doc/funto60.html
In terms of PostGIS for KMA’s weather charts, general vacuum
functions can’t solve the problem that data files continuously
increase.
18
background Problems to solved Import speed Keeping data size indexing

19. background Problems to solved Import speed Keeping data size indexing
VACUUM FULL
unused space
arrange for big size
data management
Source: http://www.devmedia.com.br/otimizacao-uma-ferramenta-chamada-vacuum/1710
On PostGIS for KMA’s weather chart, it takes 15 hr. for full vacuum.
During Vacuum full, exclusive LOCK happens.
19
VACUUM FULL

20. Partitioning
 Partitioning?
 Managing tables by conceptually separating one table to
several
 Data size by table down  Index size down and Search
speed up Weather Chart
Weather Chart
_0
Weather Chart
_1
Weather Chart
_2
Weather Chart
_3
Weather Chart
_4
Weather Chart
_5
Weather Chart
_6
Insert on Monday
Insert on Sunday
Insert on Tuesday
Truncate on Tuesday
Truncate on Monday
Truncate on Sunday
Execution time of truncate is almost a few seconds
and file size is decreased without vacuum
20
background Problems to solved Import speed Keeping data size indexing

21. Improvement on inquiry
speed by resetting index
21

22. background Problems to solved Import speed Keeping data size indexing
Improvement flow of inquiry speed
22
Data
Condition
Analysis
Query
Finding
Query
Plan
Analysis
Index
Improvement

23. background Problems to solved Import speed Keeping data size indexing
Data Condition Analysis
 Understanding # of
columns by table
 select count(*)
table_name is foolish!
 Possible to understand
the number of rows if
using statistical table
 Meaningful data is
stored on pg_class
table
 Execution time within
one minute
select relname as table_name,
to_char(reltuples, '999,999,999')
as row_count
from pg_class
where relnamespace = (select oid
from pg_namespace where nspname
= 'public')
and relam = 0
order by 2 desc, 1;
23

24. background Problems to solved Import speed Keeping data size indexing
GeoServer SQL VIEW
 Register sql query as Layer
 Datasource is geoDB, can
use SQL VIEW
 Useful
 Complex condition to layer
 Reprojection
 Able to join multiple tables
 normal attribute -> spatial
object
GeoServer , showing weather chart,
perfomance is affected by
searching speed of PostGIS
24

25. background Problems to solved Import speed Keeping data size indexing
Query Finding
 Identifying executed
SQL using statistical
 Using table
pg_stat_activity table
 Necessary process for
tuning
 Possible to check
execution time
 Differences of queries
by PostGIS version select query_start, current_query
from pg_stat_activity
where username = ‘mobile’
and current_query not like ‘<IDLE>%’
order by query_start desc;
SELECT
"val",encode(ST_AsBinary(ST_Force_2D("geom
")),'base64') as "geom"
FROM (
select mdl, mdl_var,
placemark_name, val, lyrs_cd,
forecast_time,
create_time as anal_time,
ST_Transform(the_geom, 7188) as geom
from contour
where mdl_var = 'TMP'
) as "vtable"
WHERE (((("mdl" = 'GDAPS' AND "lyrs_cd" =
'A925.0') AND "forecast_time" =
'2011.06.27 00:00') AND "anal_time" =
'2011.06.27 00:00') AND "geom" &&
ST_GeomFromText('POLYGON ((-1056768 -
2105344, -1056768 -1040384, 8192 -1040384,
8192 -2105344, -1056768 -2105344))',
7188));
25

26. background Problems to solved Import speed Keeping data size indexing
Query Plan Analysis
 PostGIS has basically query analysis function
 pgAdmin III-Query-Analysis explanation function
 Explain Analyze command – Easy to analyze query
26

27. background Problems to solved Import speed Keeping data size indexing
Index Improvement
 Principles for Index Improvement
 Setting index with all columns on Where clause
 Spatial column has separate index
 Columns with lots of including data types come first
 Possibly, items compared as same operator come first
 Unnecessary index should be removed due to bad performance on
inserting
 Examples
-- contour_0
DROP INDEX index_createtime_contour_0;
DROP INDEX index_forecasttime_contour_0;
DROP INDEX index_lyrscd_contour_0;
DROP INDEX index_mdl_contour_0;
DROP INDEX index_mdlvar_contour_0;
CREATE INDEX index_contour_0_all
ON contour_0 (forecast_time ASC NULLS LAST, mdl_var ASC NULLS LAST, lyrs_cd ASC NULLS LAST, create_time
DESC NULLS LAST, mdl ASC NULLS LAST);
 Result
 After individually deleting index, integrated index creation reduces 20% of
data capacity
 6 ~ 25 times speed improvement by table(big tables show better
performance) 27

28. Improvement
Result
28

29. 29
Under 300
isobaric/
Temperature/
Isokinetics
Ground/
Wet-number/
Temperature
800 isobaric/
Mixture ratio/
Temperature

30. Conclusion
Importance on excution
using addBatch() and
excuteBatch()
About 100 times
performance
improvement
Mixed with
partitioning and
truncate
Stably keeping
N-1
accordance
Appropriate index
for query
20 times inquiry
time improvement
30

31. Conclusion
PostGIS is really great
DBMS!
Perfectly suited with
GeoServer
However, tuning with
perfect understanding of
the features.
31

32. Q&A
Please Ask BJ Jang via
Email !
bjjang@gaia3d.com
32