SlideShare a Scribd company logo
Enterprise class deployment
for GeoServer and GeoWebcache
Optimizing performances and robustness
Ing. Mauro Bartolomeoli, GeoSolutions
Ing. Andrea Aime, GeoSolutions
Ing. Simone Giannecchini, GeoSolutions
FOSS4G-EU 2015, Como
14th July 2015
GeoSolutions
 Italian SME
 Expertise
• Image Processing, GeoSpatial Data Fusion
• Java, Java Enterprise, C++, Python
• JPEG2000, JPIP, Advanced 2D visualization
 Supporting/Developing FOSS4G
 MapStore, GeoServer
 GeoNetwork, GeoCollect
 Clients
 Public Agencies
 Private Companies
 http://www.geo-solutions.it
FOSS4G-EU 2015, Como
14th July 2015
Outline
 Intro to GeoServer (Optional)
 Benchmarking with JMeter
 Preparing raster and vector data
 Optimizing styling
 Output tuning
 Tiling and caching
 Configuring for Robustness
 Deploy configurations
FOSS4G-EU 2015, Como
14th July 2015
GeoServer
GeoServer
 GeoSpatial enterprise gateway
 Java Enterprise
 Management and Dissemination of
raster and vector data
 Standards compliant
 OGC WCS 1.0, 1.1.1 (RI), 2.0.1
 OGC WFS 1.0, 1.1 (RI), 2.0
 OGC WMS 1.1.1, 1.3
 OGC WPS 1.0.0
 Google Earth/Maps support
 KML, GeoSearch, etc..
FOSS4G-EU 2015, Como
14th July 2015
FormatsandProtocols
GeoServer
WFS
1.0, 1.1,
2.0
WMS
1.1.1
1.3.0
PostGIS
Oracle
H2
DB2
SQL Server
MySql
Spatialite
GeoCouch
Shapefile
----------
----------
---------
----------
----------
----------
---------
----------
----------
----------
---------
----------
ArcSDE
WFS
PNG, GIF
JPEG
TIFF,
GeoTIFF
SVG, PDF
KML/KMZ
Shapefile
GML2
GML3
GeoRSS
GeoJSON
CSV/XLS
Raw vector
data
Servers
Styled
maps
DBMS
Vector files
WCS
1.0,1.1.1
2.0.1
GeoTIFF
WMS
ArcGrid
GTopo30
Img+world
Mosaic
MrSID
JPEG 2000
ECW,Pyramid, Oracle GeoRaster, PostGis Raster, NetCDF
Raster files
Raw raster
data
GeoTIFF
ArcGrid
GTopo30
Img+World
GWC
(WMTS,
TMS,
WMS-C)
KML superoverlays
Google maps tiles
OGC tiles
OSGEO tiles
Google
WPS
1.0.0
FOSS4G-EU 2015, Como
14th July 2015
Intro to GeoServer
 In case you are not that familiar with it (fool!)
 GeoServer Release Schedule
 http://geoserver.geo-solutions.it/edu/en/install_run/releases.html
 Web Administration Interface
 http://geoserver.geo-solutions.it/edu/en/install_run/wai.html
 GeoServer Data Directory
 http://geoserver.geo-solutions.it/edu/en/adding_data/structure.html
 Adding a Shapefile
 http://geoserver.geo-solutions.it/edu/en/adding_data/add_shp.html
 Adding a Postgis layer
 http://geoserver.geo-solutions.it/edu/en/adding_data/postgis_lay.html
 Adding a GeoTiff
 http://geoserver.geo-solutions.it/edu/en/adding_data/add_geotiff.html
 Adding a Style
 http://geoserver.geo-solutions.it/edu/en/pretty_maps/add_style.html
FOSS4G-EU 2015, Como
14th July 2015
Benchmarking with JMeter
Benchmarking
 Benchmarking is an important phase in project
development
 It gives you various information about your
application:
 Scalability issues
 Robustness in extreme conditions
 Presence of any bottleneck inside the application
 Performances Test against another software
FOSS4G-EU 2015, Como
14th July 2015
Local Testing vs Remote Testing
 Network sometimes can compromise the results of your
tests
 In case of high network overload you may see low
throughput
 This can be associated to a bottleneck of the
connection which results in bad performances
without concerning about the request type
 For this reason, be careful to setup your remote tests
in a separate network in order to avoid network
overload
FOSS4G-EU 2015, Como
14th July 2015
Lookout for the bottleneck
 Typical bottlenecks:
 CPU: at high load all the CPUs are pegged
 Disk: CPUs are not hot, the server is reading a lot
from the disk during the benchmark
 Network (data sources): CPUs are not hot, but the
local network (maybe towards the DB) is at full
capacity
 Network (output): all local resources are midly used,
but the outbound connection is at full capacity
 Code scalability: none of the above, every resource
seems to be in light use and usage is not going up
despire higher client load
FOSS4G-EU 2015, Como
14th July 2015
Introduction to JMeter
 Open Source performance measurement tool written in
Java
 This tool will be highly used during the workshop for
testing the GeoServer performances against Benchmark
data
 More informations can be found at the Jmeter site:
http://jakarta.apache.org/jmeter/
FOSS4G-EU 2015, Como
14th July 2015
Introduction to JMeter
 Easy-to-use UI
 Allows to setup multiple thread groups, different
parallelism and request count, to ramp up the load
 Can use CSV files to generate semi-randomized requests
 Can execute parameterized tests
 Reports results in a simple table
 Uses Assertions for checking test results
 Can be executed in batch
FOSS4G-EU 2015, Como
14th July 2015
Using JMeter
 Thread group: how many
threads
 Loop: how many
requests
 HTTP sampler: the
request
 CSV: read request
params from CSV
 Default: do not repeat
what’s equal in all
requests
 Performance summary
FOSS4G-EU 2015, Como
14th July 2015
Result Summary
 Results table
 Run the benchmarks 2-3 times, let the results stabilize
 Save the results, check other optimizations, compare the
results
FOSS4G-EU 2015, Como
14th July 2015
Exercise
 Let's start with practice!!!
 Follow the instructions on the link below
http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_simple.html
 You should be able to reproduce the test using the
GeoServer Training instance installed in your PC
 This simple exercise will show you how to use JMeter with
a basic configuration for testing GeoServer performances
FOSS4G-EU 2015, Como
14th July 2015
Randomized Multiscale Testing
 In the example before, we ran a test on a layer always
setting the same bounding box and image size
 Now we should setup a more complicated test asking for
different bounding boxes and resolutions, aka Multiscale
testing
 Multiscale testing is useful because it stresses GeoServer
requesting new data each time, mimicking real usage
conditions and providing better testing for your SLD scale
dependencies
 With JMeter we can execute the test by passing a CSV file
containing multiple bounding boxes and image
dimensions
FOSS4G-EU 2015, Como
14th July 2015
Exercise
 Multiscale test created using wms_request.py on a mosaic
layer
 Follow the instructions on the link below
http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_multiscale.html
 This exercise will show you how to use create a CSV file
containing requests at multiple resolutions and how to run
JMeter with it
FOSS4G-EU 2015, Como
14th July 2015
Preparing Raster Inputs
Intro to GDAL Utilities
 Online Material
 http://geoserver.geo-
solutions.it/edu/en/raster_data/processing.html
 http://www.gdal.org/gdal_utilities.html
FOSS4G-EU 2015, Como
14th July 2015
Raster Data CheckList
 Objectives
 Fast extraction of a subset of the data
 Fast extraction of the desired resolution
 Check-list
 Avoid having to open a large number of files per
request
 Avoid parsing of complex structures and slow
compressions
 Get to know your bottlenecks
 CPU vs Disk Access Time vs Memory
 Experiment with
 Format, compression, different color models, tile size,
overviews, GeoServer configuration
FOSS4G-EU 2015, Como
14th July 2015
Peculiar Formats
 PNG/JPEG direct serving (as data sources)
 Bad formats (especially in Java)
 No tiling (or rarely supported)
 Chew a lot of memory and CPU for decompression
 Mitigate with external overviews
 Any input ASCII format (GML grid, ASCII grid)
 JPEG2000
 Extensible and rich, not (always) fast, can be difficult
to tune for performance (might require specific
encoding options)
 MrSID (can work, needs tuning)
 ECW (licensing issues)
FOSS4G-EU 2015, Como
14th July 2015
GeoTiff for the win
 To remember: GeoTiff is a swiss knife
 But you don’t want to cut a tree with it!
 Tremendously flexible, good for for most (not all) use
cases
 BigTiff pushes the GeoTiff limits farther
 Use GeoTiff when
 Overviews and Tiling stay within 4GB
 No additional dimensions
 Consider BigTiff for very large file (> 4 GB)
 Support for tiling
 Support for Overviews
 Can be inefficient with very large files + small tiling
FOSS4G-EU 2015, Como
14th July 2015
GeoTiff preparation
 (Optional) Use gdal_warp to transform the data in the most
used output reference system (mind, any reprojection
ruins the data a bit)
 Use gdal_translate to add inner tiling and fix eventual
issues with coordinate reference system
 Add compression options if you disks are small/slow/not
local (consider JPEG compression with YCBCR color
interpretation for photos, LZW/Deflate for scientific data)
 Use gdaladdo to add internal overviews (remember to
replicate compression here)
FOSS4G-EU 2015, Como
14th July 2015
Possible structures
Single GeoTiff
with internal tiling
and overviews
(GeoTiff < 2GB,
BigTiff < 20-50GB)
Mosaic of GeoTiff, each one
with internal tiling and overviews
(< 500GB, not too many files)
Image pyramid,
each level has
lower resolution
than the previous,
each level has tiles
with inner tiling but
no overviews
(also possible to mix,
less levels and use
internal overviews)
> 500GB
FOSS4G-EU 2015, Como
14th July 2015
Choosing Formats and Layouts
 Use ImageMosaic when:
 A single file gets too big (inefficient seeks, too much metadata
to read, etc..)
 Multiple Dimensions (time, elevation, others..)
 Avoid mosaics made of many very small files
 Single granules can be large
 Use Tiling + Overviews + Compression on granules
 Use ImagePyramid when:
 Tremendously large dataset
 Too many files / too large files
 Need to serve at all scales
 Especially low resolution
 For single granules (< 2Gb) GeoTiff is generally a good fit
FOSS4G-EU 2015, Como
14th July 2015
Proper Mosaic Preparation
 Optimize files as if you were serving them individually
 Keep a balance between number and dimensions of
granules
 If memory is scarce:
 USE_JAI_IMAGREAD to true
 USE_MULTITHREADING to false*
 Otherwise
 USE_JAI_IMAGREAD to false
 ALLOW_MULTITHREADING to true
(Load data from different granules in
parallel)
 More Info
 http://geoserver.geo-
solutions.it/edu/en/raster_data/mosai
c_pyramid.html
FOSS4G-EU 2015, Como
14th July 2015
Proper Pyramid Preparation
 Use gdal_retile for creating the pyramid
 Prepare the list of tiles to be retiled
 Create the pyramid with GDAL retile (grab a coffee!)
 Chunks should not be too small (here 2048x2048), if you go larger, use
also inner tiling
 If the input dataset is huge use the useDirForEachRow option
 Too many files in a dir is bad practice
 Make sure the number of level is consistent with usage
 Too few  bad performance at high scale
 GeoServer config  same as mosaic
 More Info
 http://geoserver.geo-
solutions.it/edu/en/raster_data/mosaic_pyramid.html
FOSS4G-EU 2015, Como
14th July 2015
Proper GeoServer
Coverage Config Opts
 Make sure native JAI is installed
 Install the TurboJPEG extension
 Enable JAI Mosaicking native
acceleration
 Give JAI enough memory
 Don’t raise JAI memory Threshold too
high
 Rule of thumb: use 2 X #Core Tile
Threads (check next slide)
 Play with tile Recycling against your
workflows (might help, might not)
FOSS4G-EU 2015, Como
14th July 2015
Proper GeoServer
Coverage Config Opts
 Multithreaded Granule Loading
 Allows to fine tuning multithreading
for ImageMosaic
 Orthogonal to JAI Tile Threads
 Rule of Thumb: use 2 X #Core Tile
Threads
 Perform testing to fine tune
depending on layer configuration as
well as on typical requests
 ImageIO Cache threshold
 decide when we switch to disk
cache (very large WCS requests)
FOSS4G-EU 2015, Como
14th July 2015
Exercise
 Let's check the effect of optimization in GeoServer
 Follow the instructions on the links below
http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_mosaic.html
 This test uses JMeter for testing GeoServer performances
before and after the optimization of a Mosaic Layer.
FOSS4G-EU 2015, Como
14th July 2015
Additional Material & Exercises
 Using ImageMosaic with Footprint
 http://geoserver.geo-
solutions.it/edu/en/raster_data/imagemosaic_footprint.html
 Advanced Processing With GDAL Utilities
 http://geoserver.geo-
solutions.it/edu/en/raster_data/advanced_gdal/index.html
FOSS4G-EU 2015, Como
14th July 2015
Preparing vector inputs
Vector data checklist
 What do we want from vector data:
 Binary data
 No complex parsing of data structures
 Fast extraction of a geographic subset
 Fast filtering on the most commonly used
attributes
FOSS4G-EU 2015, Como
14th July 2015
Choosing a format
 Slow formats
 WFS
 GML
 DXF
 Good formats, local and
index-able
 Shapefile
 Directory of shapefiles
 SDE
 Spatial databases:
PostGIS, Oracle
Spatial, DB2, MySQL*,
SQL server*
 Swiss Knife  SQL Views
FOSS4G-EU 2015, Como
14th July 2015
DBMS Checklist
 Rich support for complex native filters
 Use connection pooling
 Validate connections (with proper pooling)
 Table Clustering
 Spatial Indexing
 Spatial Indexing
 Spatial Indexing
 Alphanumeric Indexing
 Alphanumeric Indexing
 Alphanumeric Indexing
 Did we mention indexes?
FOSS4G-EU 2015, Como
14th July 2015
Shapefile preparation
 Remove .qix file if present
 If there are large DBF attributes that are not in use, get rid
of them using ogr2ogr, e.g.:
ogr2ogr -select FULLNAME,MTFCC arealm.shp
tl_2010_08013_arealm.shp
 If on Linux, enable memory mapping, faster, more scalable
(but will kill Windows):
FOSS4G-EU 2015, Como
14th July 2015
Shapefile filtering
 Stuck with shapefiles and have scale dependent rules like
the following?
 Show highways first
 Show all streets when zoomed in
 Use ogr2ogr to build two shapefiles, one with just the
highways, one with everything, and build two layers, e.g.:
ogr2ogr -sql "SELECT * FROM
tl_2010_08013_roads WHERE MTFCC in ('S1100',
'S1200')" primaryRoads.shp
tl_2010_08013_roads.shp
 Or hire us to develop non-spatial indexing for shapefile!
FOSS4G-EU 2015, Como
14th July 2015
PostGIS specific hints
 PostgreSQL out of the box configured for very small
hardware:
http://wiki.postgresql.org/wiki/Performance_Optimization
 Make sure to run ANALYZE after data imports (updates
optimizer stats)
 As usual, avoid large joins in SQL views, consider
materialized views
 If the dataset is massive, CLUSTER on the spatial index:
 http://postgis.refractions.net/documentation/manual-
1.3/ch05.html
FOSS4G-EU 2015, Como
14th July 2015
PostGIS specific hints
 Careful with prepared statements (bad performances)
 USE CASE: the layer’s style allows to display the whole
layer in a single shot (no scale dependencies)  prepared
statements will slow down execution
 EXPLANATION: Postgis will choose to use the spatial
index in all cases, this makes retrieving the full data set 2-4
times slower than when using a sequential scan
 COUNTERMEASURE: Not using prepared statement allows
postgis to figure out a suitable plan based on the request
bbox instead (assuming someone run "vacuum analyze"
on the database to update the index statistics, and of
course, provided there is a spatial index to start with)
FOSS4G-EU 2015, Como
14th July 2015
SQL Server specific hints
 Switch to PostgreSQL/PostGIS!
FOSS4G-EU 2015, Como
14th July 2015
Oracle specific hints
 Switch to PostgreSQL/PostGIS!
FOSS4G-EU 2015, Como
14th July 2015
SQL Views Support
 Vector Data Swiss Knife
 http://geoserver.geo-
solutions.it/edu/en/adding_data/add_sqllayers.html
FOSS4G-EU 2015, Como
14th July 2015
Exercise
 Let's compare Shapefile and PostGIS in GeoServer
 Follow the instructions on the links below
http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_vector.html
 This test uses Jmeter for comparing GeoServer
performances using a Shapefile or an optimized PostGIS
database containing the same data
FOSS4G-EU 2015, Como
14th July 2015
Optimize styling
Use scale dependencies
 Never show too much data
 the map should be readable, not a graphic blob. Rule of thumb:
1000 features max in the display, have labels show up when
zoomed in
 Show details as you zoom in
 Eagerly add MinScaleDenominator to your SLD rules
 Add more expensive rendering when there are less
features
 Key to get both a good looking and fast map
FOSS4G-EU 2015, Como
14th July 2015
Example
FOSS4G-EU 2015, Como
14th July 2015
Labeling
 Labeling conflict resolution is expensive, limit to the most
inner zooms
 Halo is important for readability, but adds significant
overhead
 Careful with maxDisplacement, makes for various label
location attempts
FOSS4G-EU 2015, Como
14th July 2015
FeatureTypeStyle
 GeoServer uses SLD FeatureTypeStyle objects as Z layers
for painting
 Each one allocates its own rendering surface (which can
use a lot of memory), use as few as possible
FOSS4G-EU 2015, Como
14th July 2015
Use translucency sparingly
 Translucent display is expensive, use it sparingly
 e.g. translucent fill <CssParameter name="fill-opacity">0.5</CssParameter>
FOSS4G-EU 2015, Como
14th July 2015
Exercise
 Let's compare bad styles with properly setup ones
 Follow the instructions on the links below
http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_styles.html
 This test uses JMeter to compare a style without scale
dependencies, showing too much data and too many
labels, with a one with proper scale dependencies and
providing a better looking output at the same output
FOSS4G-EU 2015, Como
14th July 2015
Tiling & Caching
Tile caching with GeoWebCache
 Tile oriented maps, fixed zoom levels and fixed grid
 Useful for stable layers, backgrounds
 Protocols: WMTS, TMS, WMS-C, Google Maps/Earth, VE
 Speedup compared to dynamic WMS: 10 to 100 times,
assuming tiles are already cached (whole layer pre-
seeded)
 Suitable for:
 Mostly static layer
 No/few dynamic parameters (CQL filters, SLD params,
SQL query params, time/elevation, format options)
FOSS4G-EU 2015, Como
14th July 2015
Embedded GWC advantage
 No double encoding when using meta-tiling, faster seeding
FOSS4G-EU 2015, Como
14th July 2015
Space considerations
 Seeding Colorado, assuming 8 cores, one layer, 0.1 sec
756x756 metatile, 15KB for each tile
 Do yours: http://tinyurl.com/3apkpss
 Not enough disk space? Set a disk quota
Zoom
level
Tile count Size (MB)
Time to seed
(hours)
Time to seed
(days)
13 58,377 1 0 0
14 232,870 4 0 0
15 929,475 14 0 0
16 3,713,893 57 1 0
17 14,855,572 227 6 0
18 59,396,070 906 23 1
19 237,584,280 3,625 92 4
20 950,273,037 14,500 367 15
FOSS4G-EU 2015, Como
14th July 2015
DiskQuota
 Online Material
 http://geoserver.geo-
solutions.it/edu/en/caching_data/diskquotas.html#disk-quota-
configuration
FOSS4G-EU 2015, Como
14th July 2015
More Tweaks
 REST Interface  Automate/Script management
 Truncate
 Remove Cached tiles
 Seed
 Pre-generate cache
 Reseed
 Regenerate tiles (due to data update, for instance)
 Every seed/truncate operation can be filtered by:
 Bounding box / Zoom limits
 / Format / GridSet
 Parameters (style, time, other dimensions, cql_filter…)
 Thread number
FOSS4G-EU 2015, Como
14th July 2015
More Tweaks
 REST Seed example
FOSS4G-EU 2015, Como
14th July 2015
curl -u admin:password -H 'Content-type: text/xml' -H 'Accept: text/xml' -XPOST -d '
<seedRequest>
<name>topp:states</name>
<type>seed</type>
<gridSetId>EPSG:4326</gridSetId>
<bounds>
<coords>
<double>-100</double><double>-
80</double><double>100</double><double>80</double>
</coords>
</bounds>
<zoomStart>0</zoomStart><zoomStop>8</zoomStop>
<format>image/png8</format>
<threadCount>04</threadCount>
<parameters>
<entry>
<string>CQL_FILTER</string>
<string>TOTPOP BETWEEN 0 AND 10000</string>
</entry>
</parameters>
</seedRequest>
' http://localhost:8080/geoserver/gwc/rest/seed/topp:states.xml
More Tweaks
 Online Resources for GWC REST API
 Documentation
 http://geowebcache.org/docs/current/rest/seed.html
 http://docs.geoserver.org/stable/en/user/geowebcache/rest/seed
.html
 Bash script for seed/reseed/truncate/mass truncate operations
 https://github.com/geosolutions-
it/scripts/blob/master/geowebcache/gwc.sh
 Sample script for clean cache in geoserver cluster
 https://github.com/geosolutions-
it/scripts/blob/master/geowebcache/cleancache.sh
FOSS4G-EU 2015, Como
14th July 2015
More Tweaks
 Clustering Single Shared Cache
 Large, shared disk with fast R/W access
 Perfect for bkg layers
 Problematic set up in public cloud
 Clustering Multiple Indipendent Caches
 Exploit local, small, instance-based caches
 Good for short lived caches
 Duplication hit  rsynch can help
 Real-World Use Case
 Dedicated Instances with shared cache for Bkg Layers
 Indipendent Caches for other layers
FOSS4G-EU 2015, Como
14th July 2015
More Tweaks
 Client-side caching of tiles
 Does not work with browsers in private mode
<expireClientsList>
<expirationRule minZoom="0" expiration="7200" />
<expirationRule minZoom="10" expiration="600" />
</expireClientsList>
FOSS4G-EU 2015, Como
14th July 2015
More Tweaks
 Use the right formats
 JPEG for background data (e.g. ortos)
 PNG8 + precomputed palette for background
data (e.g. ortos)
 PNG8 full for overlays with transparency
 The format impacts also the disk space needed!
(as well as the generation time)
 Check this blog post
FOSS4G-EU 2015, Como
14th July 2015
Exercise
 Let's compare GeoServer WMS with GeoWebCache
fullWMS.
 Follow the instructions on the links below
http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_gwc.html
 This test uses JMeter for comparing GeoServer WMS
performances against those of GeoWebCache fullWMS
 Mind, this is not the same as comparing WMS with pure
tile caching, GWC has to read the tiles from disk,
decompress, assemble the output image, and compress it
again in PNG (some speedup still, but not 10 times)
FOSS4G-EU 2015, Como
14th July 2015
Advanced GeoServer Configuration
Resource Limits
 Limit the amount of resources dedicated to an individual
request
 Improve fairness between requests, by preventing
individual requests from hijacking the server and/or
running for a very long time
 EXTREMELY IMPORTANT in production environment
 WHEN TO TWEAK THEM?
 Frequent OOM Errors despite plenty of RAM
 Requests that keep running for a long time (e.g. CPU
usage peaks even if no requests are being sent)
 DB Connection being killed by the DBMS while in
usage (ok, you might also need to talk to the DBA..)
FOSS4G-EU 2015, Como
14th July 2015
WMS request limits
 Max memory per request: avoid large requests, allows to
size the server memory (max concurrent request * max
memory)
 Max time per request: avoid requests taking too much time
(e.g., using a custom style provided with dynamic SLD in
the request)
 Max errors: best effort renderer, but handling errors takes
time
FOSS4G-EU 2015, Como
14th July 2015
WFS request limits
 Max feature returned, configured as a global limit
 Return feature bbox: reduce amount of generated GML
 Per layer max feature count
FOSS4G-EU 2015, Como
14th July 2015
WCS request limits
FOSS4G-EU 2015, Como
14th July 2015
Exercise
 Let's show how resource limits can be used to avoid
issues with slow requests
 Follow the instructions on the links below
http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_wmslimits.html
 This test artificially limits the database resources to
quickly generate a starvation situation in which the limits
kick-in
FOSS4G-EU 2015, Como
14th July 2015
Control flow
 Control how many requests are executed in parallel, queue
others:
 Increase throughput
 Control memory usage
 Enforce fairness
 More info
 http://docs.geoserver.org/latest/en/user/community/controlflow/i
ndex.html
FOSS4G-EU 2015, Como
14th July 2015
$GEOSERVER_DATA_DIR/controlflow.properties
# don't allow more than 16 GetMap requests in parallel
ows.wms.getmap=16
Control flow
17%
FOSS4G-EU 2015, Como
14th July 2015
Resource Limits
 They need to be tweaked together with Control- Flow
 Limiting individual requests  Resource Limits
 Limiting amount of parallel request  Control Flow
 When time is involved make sure you keep into account all
pieces of the response chain
 E.G. Limited rendering time for WMS on data coming
from WMS, items to take into account are
 LifeTime of DB Connection (usually long)
 WaitTime for a new connection (we don’t want to
queue requests at the connection pool when they
area already eating memory!)
FOSS4G-EU 2015, Como
14th July 2015
Connection Pooling Tricks
 Connection pool size should be proportional to the number of
concurrent requests you want to serve (obvious no?)
 Activate connection validation
 Mind networking tools that might cut connections sitting idle
(yes, your server is not always busy), they might cut the
connection in “bad” ways (10 minutes timeout before the pool
realizes the TCP connection attempt gives up)
 Read more at
 http://geoserver.geo-
solutions.it/edu/en/adv_gsconfig/db_pooling.html
 https://docs.google.com/document/d/1O02PeDRYKt2xLW
G21k6BmcaRPzVRvXb4SQcrcZf4bHQ/edit?usp=sharing
FOSS4G-EU 2015, Como
14th July 2015
Exercise
 Let's show how control-flow works in GeoServer
 Follow the instructions on the links below
http://geoserver.geo-
solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_controlflow.html
 This test uses JMeter for testing GeoServer performances
before and after configuring the control-flow plugin.
FOSS4G-EU 2015, Como
14th July 2015
JVM and deploy configuration
Premise
 The options discussed here are not going to help visibly if
you did not prepare the data and the styles
 They are finishing touches that can get performance up
once the major data bottlenecks have been dealt with
 Check “Running in production” instructions here
FOSS4G-EU 2015, Como
14th July 2015
JVM settings
 --server: enables the server JIT compiler
 --Xms2048m -Xmx2048m: sets the JVM use two gigabytes
of memory
 --XX:+UseParallelOldGC -XX:+UserParallelGC: enables
multi-threaded garbage collections, useful if you have
more than two cores
 --XX:NewRatio=2: informs the JVM there will be a high
number of short lived objects
 --XX:+AggressiveOpt: enable experimental optimizations
that will be defaults in future versions of the JVM
FOSS4G-EU 2015, Como
14th July 2015
Setup a local cluster
 Oracle Java2D locks when drawing antialiased vectors
 Limits scalability severely
 Two options
 Use OpenJDK, it’s slower at rendering but scales up
well
 Use Apache mod_proxy_balance and setup a
GeoServer each 2/4 cores
mod_proxy_balance
GeoServer GeoServer GeoServer
FOSS4G-EU 2015, Como
14th July 2015
Clustering advantage
66%
 FOSS4G 2010 vector benchmarks (roads/buildings/isolines
and so on, over the entire Spain)
 GeoServer was benchmarked using Oracle JDK without
local clustering
FOSS4G-EU 2015, Como
14th July 2015
Marlin renderer
 The OpenJDK Java2D renderer scales up, but it’s not
super-fast when the load is small (1 request at a time)
 Marlin-renderer to the rescue:
https://github.com/bourgesl/marlin-renderer
 Complex map, 10
parallel requests,
different zoom
levels have different
details showing up
FOSS4G-EU 2015, Como
14th July 2015
Upgrade!
 Performance tends to go up version by version
 Please do use a recent GeoServer version
 FOSS4G 2010
vector
benchmark with
different
versions of
GeoServer
FOSS4G-EU 2015, Como
14th July 2015
Exercise
 Let's show how Marlin renderer can speed up map
rendering under Windows, using GeoServer 2.5.x
 Follow the instructions on the links below
http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_marlin.html
 This test uses JMeter for testing GeoServer performances
before and after setting up Marlin renderer
FOSS4G-EU 2015, Como
14th July 2015
Clustering
Clustering with GeoServer
 Online Material
 http://geoserver.geo-
solutions.it/edu/en/clustering/index.html
FOSS4G-EU 2015, Como
14th July 2015
The End
Questions?
info@geo-solutions.it
FOSS4G-EU 2015, Como
14th July 2015

More Related Content

What's hot

GeoServer, an introduction for beginners
GeoServer, an introduction for beginnersGeoServer, an introduction for beginners
GeoServer, an introduction for beginners
GeoSolutions
 
GIS User to Web-GIS Developer Journey
GIS User to Web-GIS Developer JourneyGIS User to Web-GIS Developer Journey
GIS User to Web-GIS Developer Journey
Tek Kshetri
 
Advanced Security with GeoServer - FOSS4G 2015
Advanced Security with GeoServer - FOSS4G 2015Advanced Security with GeoServer - FOSS4G 2015
Advanced Security with GeoServer - FOSS4G 2015
GeoSolutions
 
Mastering Security with GeoServer and GeoFence - FOSS4G EU 2017
Mastering Security with GeoServer and GeoFence - FOSS4G EU 2017Mastering Security with GeoServer and GeoFence - FOSS4G EU 2017
Mastering Security with GeoServer and GeoFence - FOSS4G EU 2017
GeoSolutions
 
Fundamentals of GIS
Fundamentals of GISFundamentals of GIS
Fundamentals of GIS
Pallab Jana
 
PostGIS and Spatial SQL
PostGIS and Spatial SQLPostGIS and Spatial SQL
PostGIS and Spatial SQL
Todd Barr
 
One GeoNode, many GeoNodes
One GeoNode, many GeoNodesOne GeoNode, many GeoNodes
One GeoNode, many GeoNodes
GeoSolutions
 
GeoServer an introduction for beginners
GeoServer an introduction for beginnersGeoServer an introduction for beginners
GeoServer an introduction for beginners
GeoSolutions
 
Oracle AHF Insights 23c - Deeper Diagnostic Insights for your Oracle Database...
Oracle AHF Insights 23c - Deeper Diagnostic Insights for your Oracle Database...Oracle AHF Insights 23c - Deeper Diagnostic Insights for your Oracle Database...
Oracle AHF Insights 23c - Deeper Diagnostic Insights for your Oracle Database...
Sandesh Rao
 
Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Develo...
Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Develo...Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Develo...
Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Develo...
GeoSolutions
 
Building Enterprise SDI with Geonode
Building Enterprise SDI with GeonodeBuilding Enterprise SDI with Geonode
Building Enterprise SDI with Geonode
Rafael Soto
 
Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Process...
Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Process...Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Process...
Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Process...
GeoSolutions
 
Serving earth observation data with GeoServer: addressing real world requirem...
Serving earth observation data with GeoServer: addressing real world requirem...Serving earth observation data with GeoServer: addressing real world requirem...
Serving earth observation data with GeoServer: addressing real world requirem...
GeoSolutions
 
공간정보거점대학 1.geo server_고급과정
공간정보거점대학 1.geo server_고급과정공간정보거점대학 1.geo server_고급과정
공간정보거점대학 1.geo server_고급과정
BJ Jang
 
Shell Scripts for Oracle Database and E-Business Suite.pdf
Shell Scripts for Oracle Database and E-Business Suite.pdfShell Scripts for Oracle Database and E-Business Suite.pdf
Shell Scripts for Oracle Database and E-Business Suite.pdf
AkhashRamnath
 
Mapserver vs. geoserver
Mapserver vs. geoserverMapserver vs. geoserver
Mapserver vs. geoserver鸣 饶
 
[FOSS4G Korea 2016] Workshop - Advanced GeoServer
[FOSS4G Korea 2016] Workshop - Advanced GeoServer[FOSS4G Korea 2016] Workshop - Advanced GeoServer
[FOSS4G Korea 2016] Workshop - Advanced GeoServer
MinPa Lee
 
GEOPROCESSING IN QGIS
GEOPROCESSING IN QGISGEOPROCESSING IN QGIS
GEOPROCESSING IN QGIS
Swetha A
 
Symbology and Classifying data in ARC GIS
Symbology and Classifying data in ARC GISSymbology and Classifying data in ARC GIS
Symbology and Classifying data in ARC GIS
KU Leuven
 

What's hot (20)

GeoServer, an introduction for beginners
GeoServer, an introduction for beginnersGeoServer, an introduction for beginners
GeoServer, an introduction for beginners
 
QGIS training
QGIS trainingQGIS training
QGIS training
 
GIS User to Web-GIS Developer Journey
GIS User to Web-GIS Developer JourneyGIS User to Web-GIS Developer Journey
GIS User to Web-GIS Developer Journey
 
Advanced Security with GeoServer - FOSS4G 2015
Advanced Security with GeoServer - FOSS4G 2015Advanced Security with GeoServer - FOSS4G 2015
Advanced Security with GeoServer - FOSS4G 2015
 
Mastering Security with GeoServer and GeoFence - FOSS4G EU 2017
Mastering Security with GeoServer and GeoFence - FOSS4G EU 2017Mastering Security with GeoServer and GeoFence - FOSS4G EU 2017
Mastering Security with GeoServer and GeoFence - FOSS4G EU 2017
 
Fundamentals of GIS
Fundamentals of GISFundamentals of GIS
Fundamentals of GIS
 
PostGIS and Spatial SQL
PostGIS and Spatial SQLPostGIS and Spatial SQL
PostGIS and Spatial SQL
 
One GeoNode, many GeoNodes
One GeoNode, many GeoNodesOne GeoNode, many GeoNodes
One GeoNode, many GeoNodes
 
GeoServer an introduction for beginners
GeoServer an introduction for beginnersGeoServer an introduction for beginners
GeoServer an introduction for beginners
 
Oracle AHF Insights 23c - Deeper Diagnostic Insights for your Oracle Database...
Oracle AHF Insights 23c - Deeper Diagnostic Insights for your Oracle Database...Oracle AHF Insights 23c - Deeper Diagnostic Insights for your Oracle Database...
Oracle AHF Insights 23c - Deeper Diagnostic Insights for your Oracle Database...
 
Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Develo...
Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Develo...Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Develo...
Raster Data In GeoServer And GeoTools: Achievements, Issues And Future Develo...
 
Building Enterprise SDI with Geonode
Building Enterprise SDI with GeonodeBuilding Enterprise SDI with Geonode
Building Enterprise SDI with Geonode
 
Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Process...
Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Process...Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Process...
Crunching Data In GeoServer: Mastering Rendering Transformations, WPS Process...
 
Serving earth observation data with GeoServer: addressing real world requirem...
Serving earth observation data with GeoServer: addressing real world requirem...Serving earth observation data with GeoServer: addressing real world requirem...
Serving earth observation data with GeoServer: addressing real world requirem...
 
공간정보거점대학 1.geo server_고급과정
공간정보거점대학 1.geo server_고급과정공간정보거점대학 1.geo server_고급과정
공간정보거점대학 1.geo server_고급과정
 
Shell Scripts for Oracle Database and E-Business Suite.pdf
Shell Scripts for Oracle Database and E-Business Suite.pdfShell Scripts for Oracle Database and E-Business Suite.pdf
Shell Scripts for Oracle Database and E-Business Suite.pdf
 
Mapserver vs. geoserver
Mapserver vs. geoserverMapserver vs. geoserver
Mapserver vs. geoserver
 
[FOSS4G Korea 2016] Workshop - Advanced GeoServer
[FOSS4G Korea 2016] Workshop - Advanced GeoServer[FOSS4G Korea 2016] Workshop - Advanced GeoServer
[FOSS4G Korea 2016] Workshop - Advanced GeoServer
 
GEOPROCESSING IN QGIS
GEOPROCESSING IN QGISGEOPROCESSING IN QGIS
GEOPROCESSING IN QGIS
 
Symbology and Classifying data in ARC GIS
Symbology and Classifying data in ARC GISSymbology and Classifying data in ARC GIS
Symbology and Classifying data in ARC GIS
 

Similar to Enterprise class deployment for GeoServer and GeoWebcache Optimizing performances and robustness

GeoServer on steroids
GeoServer on steroidsGeoServer on steroids
GeoServer on steroids
GeoSolutions
 
WMS Performance Shootout 2010
WMS Performance Shootout 2010WMS Performance Shootout 2010
WMS Performance Shootout 2010
Jeff McKenna
 
State of GeoServer 2015
State of GeoServer 2015State of GeoServer 2015
State of GeoServer 2015
Jody Garnett
 
Unveiling FME 2013
Unveiling FME 2013Unveiling FME 2013
Unveiling FME 2013
Safe Software
 
State of GeoServer
State of GeoServerState of GeoServer
State of GeoServer
Jody Garnett
 
Introduction to Google Colaboratory.pdf
Introduction to Google Colaboratory.pdfIntroduction to Google Colaboratory.pdf
Introduction to Google Colaboratory.pdf
Yomna Mahmoud Ibrahim Hassan
 
Spatio-temporal Data Handling With GeoServer for MetOc And Remote Sensing
Spatio-temporal Data Handling With GeoServer for MetOc And Remote SensingSpatio-temporal Data Handling With GeoServer for MetOc And Remote Sensing
Spatio-temporal Data Handling With GeoServer for MetOc And Remote Sensing
GeoSolutions
 
Front-end performances
Front-end performancesFront-end performances
Front-end performances
Smile I.T is open
 
Why Test SAP PI/PO after any upgrade
Why Test SAP PI/PO after any upgradeWhy Test SAP PI/PO after any upgrade
Why Test SAP PI/PO after any upgrade
Daniel Graversen
 
Scaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global OrganizationScaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global Organization
Puppet
 
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
Kohei KaiGai
 
Performance Tuning Oracle Weblogic Server 12c
Performance Tuning Oracle Weblogic Server 12cPerformance Tuning Oracle Weblogic Server 12c
Performance Tuning Oracle Weblogic Server 12c
Ajith Narayanan
 
FDM to FDMEE migration utility
FDM to FDMEE migration utilityFDM to FDMEE migration utility
FDM to FDMEE migration utility
Bernard Ash
 
OSGeo Live Lightening Overview
OSGeo Live Lightening OverviewOSGeo Live Lightening Overview
OSGeo Live Lightening Overview
Jody Garnett
 
Lessons learned from designing QA automation event streaming platform(IoT big...
Lessons learned from designing QA automation event streaming platform(IoT big...Lessons learned from designing QA automation event streaming platform(IoT big...
Lessons learned from designing QA automation event streaming platform(IoT big...
Omid Vahdaty
 
Open Cloud Computing Interface Presentation
Open Cloud Computing Interface PresentationOpen Cloud Computing Interface Presentation
Open Cloud Computing Interface Presentation
Intel Corporation
 
Meet Magento Spain 2019 - Our Experience with Magento Cloud
Meet Magento Spain 2019 - Our Experience with Magento CloudMeet Magento Spain 2019 - Our Experience with Magento Cloud
Meet Magento Spain 2019 - Our Experience with Magento Cloud
Lyzun Oleksandr
 
PRG/215 ENTIRE CLASS UOP TUTORIALS
PRG/215 ENTIRE CLASS UOP TUTORIALSPRG/215 ENTIRE CLASS UOP TUTORIALS
PRG/215 ENTIRE CLASS UOP TUTORIALS
Sharon Reynolds
 
Final presentasi gnome asia
Final presentasi gnome asiaFinal presentasi gnome asia
Final presentasi gnome asia
Anton Siswo
 
SF JUG - GWT Can Help You Create Amazing Apps - 2009-10-13
SF JUG - GWT Can Help You Create Amazing Apps - 2009-10-13SF JUG - GWT Can Help You Create Amazing Apps - 2009-10-13
SF JUG - GWT Can Help You Create Amazing Apps - 2009-10-13Fred Sauer
 

Similar to Enterprise class deployment for GeoServer and GeoWebcache Optimizing performances and robustness (20)

GeoServer on steroids
GeoServer on steroidsGeoServer on steroids
GeoServer on steroids
 
WMS Performance Shootout 2010
WMS Performance Shootout 2010WMS Performance Shootout 2010
WMS Performance Shootout 2010
 
State of GeoServer 2015
State of GeoServer 2015State of GeoServer 2015
State of GeoServer 2015
 
Unveiling FME 2013
Unveiling FME 2013Unveiling FME 2013
Unveiling FME 2013
 
State of GeoServer
State of GeoServerState of GeoServer
State of GeoServer
 
Introduction to Google Colaboratory.pdf
Introduction to Google Colaboratory.pdfIntroduction to Google Colaboratory.pdf
Introduction to Google Colaboratory.pdf
 
Spatio-temporal Data Handling With GeoServer for MetOc And Remote Sensing
Spatio-temporal Data Handling With GeoServer for MetOc And Remote SensingSpatio-temporal Data Handling With GeoServer for MetOc And Remote Sensing
Spatio-temporal Data Handling With GeoServer for MetOc And Remote Sensing
 
Front-end performances
Front-end performancesFront-end performances
Front-end performances
 
Why Test SAP PI/PO after any upgrade
Why Test SAP PI/PO after any upgradeWhy Test SAP PI/PO after any upgrade
Why Test SAP PI/PO after any upgrade
 
Scaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global OrganizationScaling Puppet Usage to a Global Organization
Scaling Puppet Usage to a Global Organization
 
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
GPU/SSD Accelerates PostgreSQL - challenge towards query processing throughpu...
 
Performance Tuning Oracle Weblogic Server 12c
Performance Tuning Oracle Weblogic Server 12cPerformance Tuning Oracle Weblogic Server 12c
Performance Tuning Oracle Weblogic Server 12c
 
FDM to FDMEE migration utility
FDM to FDMEE migration utilityFDM to FDMEE migration utility
FDM to FDMEE migration utility
 
OSGeo Live Lightening Overview
OSGeo Live Lightening OverviewOSGeo Live Lightening Overview
OSGeo Live Lightening Overview
 
Lessons learned from designing QA automation event streaming platform(IoT big...
Lessons learned from designing QA automation event streaming platform(IoT big...Lessons learned from designing QA automation event streaming platform(IoT big...
Lessons learned from designing QA automation event streaming platform(IoT big...
 
Open Cloud Computing Interface Presentation
Open Cloud Computing Interface PresentationOpen Cloud Computing Interface Presentation
Open Cloud Computing Interface Presentation
 
Meet Magento Spain 2019 - Our Experience with Magento Cloud
Meet Magento Spain 2019 - Our Experience with Magento CloudMeet Magento Spain 2019 - Our Experience with Magento Cloud
Meet Magento Spain 2019 - Our Experience with Magento Cloud
 
PRG/215 ENTIRE CLASS UOP TUTORIALS
PRG/215 ENTIRE CLASS UOP TUTORIALSPRG/215 ENTIRE CLASS UOP TUTORIALS
PRG/215 ENTIRE CLASS UOP TUTORIALS
 
Final presentasi gnome asia
Final presentasi gnome asiaFinal presentasi gnome asia
Final presentasi gnome asia
 
SF JUG - GWT Can Help You Create Amazing Apps - 2009-10-13
SF JUG - GWT Can Help You Create Amazing Apps - 2009-10-13SF JUG - GWT Can Help You Create Amazing Apps - 2009-10-13
SF JUG - GWT Can Help You Create Amazing Apps - 2009-10-13
 

More from GeoSolutions

MapStore 2 - The Story
MapStore 2 - The StoryMapStore 2 - The Story
MapStore 2 - The Story
GeoSolutions
 
Introduction to GeoNode
Introduction to GeoNodeIntroduction to GeoNode
Introduction to GeoNode
GeoSolutions
 
Serving earth observation data with GeoServer: addressing real world requirem...
Serving earth observation data with GeoServer: addressing real world requirem...Serving earth observation data with GeoServer: addressing real world requirem...
Serving earth observation data with GeoServer: addressing real world requirem...
GeoSolutions
 
GeoServer Feature FRENZY
GeoServer Feature FRENZYGeoServer Feature FRENZY
GeoServer Feature FRENZY
GeoSolutions
 
State of GeoServer 2.12
State of GeoServer 2.12State of GeoServer 2.12
State of GeoServer 2.12
GeoSolutions
 
MapStore 2, modern mashups with OL3, Leaflet and React
MapStore 2, modern mashups with OL3, Leaflet and ReactMapStore 2, modern mashups with OL3, Leaflet and React
MapStore 2, modern mashups with OL3, Leaflet and React
GeoSolutions
 
State of GeoServer - FOSS4G 2016
State of GeoServer - FOSS4G 2016State of GeoServer - FOSS4G 2016
State of GeoServer - FOSS4G 2016
GeoSolutions
 
Raster Data In GeoServer and GeoTools: Achievements, Issues And Future Develo...
Raster Data In GeoServer and GeoTools: Achievements, Issues And Future Develo...Raster Data In GeoServer and GeoTools: Achievements, Issues And Future Develo...
Raster Data In GeoServer and GeoTools: Achievements, Issues And Future Develo...
GeoSolutions
 
Mapping the world beyond web mercator - FOSS4G 2015
Mapping the world beyond web mercator - FOSS4G 2015Mapping the world beyond web mercator - FOSS4G 2015
Mapping the world beyond web mercator - FOSS4G 2015
GeoSolutions
 
GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...
GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...
GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...
GeoSolutions
 
Advanced Cartographic Map Rendering in GeoServer
Advanced Cartographic Map Rendering in GeoServerAdvanced Cartographic Map Rendering in GeoServer
Advanced Cartographic Map Rendering in GeoServer
GeoSolutions
 
GeoSolutions Keynote at WebMGS 2015
GeoSolutions Keynote at WebMGS 2015GeoSolutions Keynote at WebMGS 2015
GeoSolutions Keynote at WebMGS 2015GeoSolutions
 
GeoServer beginners gwf_2015
GeoServer beginners gwf_2015GeoServer beginners gwf_2015
GeoServer beginners gwf_2015
GeoSolutions
 
Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04
GeoSolutions
 
Geoserver introduction, GeoBusiness 2015
Geoserver introduction, GeoBusiness 2015Geoserver introduction, GeoBusiness 2015
Geoserver introduction, GeoBusiness 2015
GeoSolutions
 
Introduzione a GeoServer ed ai servizi OGC
Introduzione a GeoServer ed ai servizi OGCIntroduzione a GeoServer ed ai servizi OGC
Introduzione a GeoServer ed ai servizi OGC
GeoSolutions
 
Advanced Security With GeoServer
Advanced Security With GeoServerAdvanced Security With GeoServer
Advanced Security With GeoServer
GeoSolutions
 
GeoNetwork, The Open Source Solution for the interoperable management of ge...
GeoNetwork, The Open Source Solution  for the interoperable management  of ge...GeoNetwork, The Open Source Solution  for the interoperable management  of ge...
GeoNetwork, The Open Source Solution for the interoperable management of ge...GeoSolutions
 

More from GeoSolutions (18)

MapStore 2 - The Story
MapStore 2 - The StoryMapStore 2 - The Story
MapStore 2 - The Story
 
Introduction to GeoNode
Introduction to GeoNodeIntroduction to GeoNode
Introduction to GeoNode
 
Serving earth observation data with GeoServer: addressing real world requirem...
Serving earth observation data with GeoServer: addressing real world requirem...Serving earth observation data with GeoServer: addressing real world requirem...
Serving earth observation data with GeoServer: addressing real world requirem...
 
GeoServer Feature FRENZY
GeoServer Feature FRENZYGeoServer Feature FRENZY
GeoServer Feature FRENZY
 
State of GeoServer 2.12
State of GeoServer 2.12State of GeoServer 2.12
State of GeoServer 2.12
 
MapStore 2, modern mashups with OL3, Leaflet and React
MapStore 2, modern mashups with OL3, Leaflet and ReactMapStore 2, modern mashups with OL3, Leaflet and React
MapStore 2, modern mashups with OL3, Leaflet and React
 
State of GeoServer - FOSS4G 2016
State of GeoServer - FOSS4G 2016State of GeoServer - FOSS4G 2016
State of GeoServer - FOSS4G 2016
 
Raster Data In GeoServer and GeoTools: Achievements, Issues And Future Develo...
Raster Data In GeoServer and GeoTools: Achievements, Issues And Future Develo...Raster Data In GeoServer and GeoTools: Achievements, Issues And Future Develo...
Raster Data In GeoServer and GeoTools: Achievements, Issues And Future Develo...
 
Mapping the world beyond web mercator - FOSS4G 2015
Mapping the world beyond web mercator - FOSS4G 2015Mapping the world beyond web mercator - FOSS4G 2015
Mapping the world beyond web mercator - FOSS4G 2015
 
GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...
GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...
GeoServer for Spatio-temporal Data Handling With Examples For MetOc And Remot...
 
Advanced Cartographic Map Rendering in GeoServer
Advanced Cartographic Map Rendering in GeoServerAdvanced Cartographic Map Rendering in GeoServer
Advanced Cartographic Map Rendering in GeoServer
 
GeoSolutions Keynote at WebMGS 2015
GeoSolutions Keynote at WebMGS 2015GeoSolutions Keynote at WebMGS 2015
GeoSolutions Keynote at WebMGS 2015
 
GeoServer beginners gwf_2015
GeoServer beginners gwf_2015GeoServer beginners gwf_2015
GeoServer beginners gwf_2015
 
Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04Geosolutions gwf-2015-v01.04
Geosolutions gwf-2015-v01.04
 
Geoserver introduction, GeoBusiness 2015
Geoserver introduction, GeoBusiness 2015Geoserver introduction, GeoBusiness 2015
Geoserver introduction, GeoBusiness 2015
 
Introduzione a GeoServer ed ai servizi OGC
Introduzione a GeoServer ed ai servizi OGCIntroduzione a GeoServer ed ai servizi OGC
Introduzione a GeoServer ed ai servizi OGC
 
Advanced Security With GeoServer
Advanced Security With GeoServerAdvanced Security With GeoServer
Advanced Security With GeoServer
 
GeoNetwork, The Open Source Solution for the interoperable management of ge...
GeoNetwork, The Open Source Solution  for the interoperable management  of ge...GeoNetwork, The Open Source Solution  for the interoperable management  of ge...
GeoNetwork, The Open Source Solution for the interoperable management of ge...
 

Recently uploaded

How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
Product School
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
DanBrown980551
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
Sri Ambati
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
Product School
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Ramesh Iyer
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Inflectra
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
KatiaHIMEUR1
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
Safe Software
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
OnBoard
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Product School
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
91mobiles
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Product School
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Product School
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
RTTS
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
Product School
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
ControlCase
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
Dorra BARTAGUIZ
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
Alan Dix
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
DianaGray10
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance
 

Recently uploaded (20)

How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...How world-class product teams are winning in the AI era by CEO and Founder, P...
How world-class product teams are winning in the AI era by CEO and Founder, P...
 
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...
 
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
GenAISummit 2024 May 28 Sri Ambati Keynote: AGI Belongs to The Community in O...
 
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
From Siloed Products to Connected Ecosystem: Building a Sustainable and Scala...
 
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...
 
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualitySoftware Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered Quality
 
Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !Securing your Kubernetes cluster_ a step-by-step guide to success !
Securing your Kubernetes cluster_ a step-by-step guide to success !
 
Essentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with ParametersEssentials of Automations: Optimizing FME Workflows with Parameters
Essentials of Automations: Optimizing FME Workflows with Parameters
 
Leading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdfLeading Change strategies and insights for effective change management pdf 1.pdf
Leading Change strategies and insights for effective change management pdf 1.pdf
 
Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...Mission to Decommission: Importance of Decommissioning Products to Increase E...
Mission to Decommission: Importance of Decommissioning Products to Increase E...
 
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdfSmart TV Buyer Insights Survey 2024 by 91mobiles.pdf
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf
 
Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...Designing Great Products: The Power of Design and Leadership by Chief Designe...
Designing Great Products: The Power of Design and Leadership by Chief Designe...
 
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
Unsubscribed: Combat Subscription Fatigue With a Membership Mentality by Head...
 
JMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and GrafanaJMeter webinar - integration with InfluxDB and Grafana
JMeter webinar - integration with InfluxDB and Grafana
 
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
AI for Every Business: Unlocking Your Product's Universal Potential by VP of ...
 
PCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase TeamPCI PIN Basics Webinar from the Controlcase Team
PCI PIN Basics Webinar from the Controlcase Team
 
Elevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object CalisthenicsElevating Tactical DDD Patterns Through Object Calisthenics
Elevating Tactical DDD Patterns Through Object Calisthenics
 
Epistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI supportEpistemic Interaction - tuning interfaces to provide information for AI support
Epistemic Interaction - tuning interfaces to provide information for AI support
 
Connector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a buttonConnector Corner: Automate dynamic content and events by pushing a button
Connector Corner: Automate dynamic content and events by pushing a button
 
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdfFIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
FIDO Alliance Osaka Seminar: Passkeys at Amazon.pdf
 

Enterprise class deployment for GeoServer and GeoWebcache Optimizing performances and robustness

  • 1. Enterprise class deployment for GeoServer and GeoWebcache Optimizing performances and robustness Ing. Mauro Bartolomeoli, GeoSolutions Ing. Andrea Aime, GeoSolutions Ing. Simone Giannecchini, GeoSolutions FOSS4G-EU 2015, Como 14th July 2015
  • 2. GeoSolutions  Italian SME  Expertise • Image Processing, GeoSpatial Data Fusion • Java, Java Enterprise, C++, Python • JPEG2000, JPIP, Advanced 2D visualization  Supporting/Developing FOSS4G  MapStore, GeoServer  GeoNetwork, GeoCollect  Clients  Public Agencies  Private Companies  http://www.geo-solutions.it FOSS4G-EU 2015, Como 14th July 2015
  • 3. Outline  Intro to GeoServer (Optional)  Benchmarking with JMeter  Preparing raster and vector data  Optimizing styling  Output tuning  Tiling and caching  Configuring for Robustness  Deploy configurations FOSS4G-EU 2015, Como 14th July 2015
  • 5. GeoServer  GeoSpatial enterprise gateway  Java Enterprise  Management and Dissemination of raster and vector data  Standards compliant  OGC WCS 1.0, 1.1.1 (RI), 2.0.1  OGC WFS 1.0, 1.1 (RI), 2.0  OGC WMS 1.1.1, 1.3  OGC WPS 1.0.0  Google Earth/Maps support  KML, GeoSearch, etc.. FOSS4G-EU 2015, Como 14th July 2015
  • 6. FormatsandProtocols GeoServer WFS 1.0, 1.1, 2.0 WMS 1.1.1 1.3.0 PostGIS Oracle H2 DB2 SQL Server MySql Spatialite GeoCouch Shapefile ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ---------- ---------- --------- ---------- ArcSDE WFS PNG, GIF JPEG TIFF, GeoTIFF SVG, PDF KML/KMZ Shapefile GML2 GML3 GeoRSS GeoJSON CSV/XLS Raw vector data Servers Styled maps DBMS Vector files WCS 1.0,1.1.1 2.0.1 GeoTIFF WMS ArcGrid GTopo30 Img+world Mosaic MrSID JPEG 2000 ECW,Pyramid, Oracle GeoRaster, PostGis Raster, NetCDF Raster files Raw raster data GeoTIFF ArcGrid GTopo30 Img+World GWC (WMTS, TMS, WMS-C) KML superoverlays Google maps tiles OGC tiles OSGEO tiles Google WPS 1.0.0 FOSS4G-EU 2015, Como 14th July 2015
  • 7. Intro to GeoServer  In case you are not that familiar with it (fool!)  GeoServer Release Schedule  http://geoserver.geo-solutions.it/edu/en/install_run/releases.html  Web Administration Interface  http://geoserver.geo-solutions.it/edu/en/install_run/wai.html  GeoServer Data Directory  http://geoserver.geo-solutions.it/edu/en/adding_data/structure.html  Adding a Shapefile  http://geoserver.geo-solutions.it/edu/en/adding_data/add_shp.html  Adding a Postgis layer  http://geoserver.geo-solutions.it/edu/en/adding_data/postgis_lay.html  Adding a GeoTiff  http://geoserver.geo-solutions.it/edu/en/adding_data/add_geotiff.html  Adding a Style  http://geoserver.geo-solutions.it/edu/en/pretty_maps/add_style.html FOSS4G-EU 2015, Como 14th July 2015
  • 9. Benchmarking  Benchmarking is an important phase in project development  It gives you various information about your application:  Scalability issues  Robustness in extreme conditions  Presence of any bottleneck inside the application  Performances Test against another software FOSS4G-EU 2015, Como 14th July 2015
  • 10. Local Testing vs Remote Testing  Network sometimes can compromise the results of your tests  In case of high network overload you may see low throughput  This can be associated to a bottleneck of the connection which results in bad performances without concerning about the request type  For this reason, be careful to setup your remote tests in a separate network in order to avoid network overload FOSS4G-EU 2015, Como 14th July 2015
  • 11. Lookout for the bottleneck  Typical bottlenecks:  CPU: at high load all the CPUs are pegged  Disk: CPUs are not hot, the server is reading a lot from the disk during the benchmark  Network (data sources): CPUs are not hot, but the local network (maybe towards the DB) is at full capacity  Network (output): all local resources are midly used, but the outbound connection is at full capacity  Code scalability: none of the above, every resource seems to be in light use and usage is not going up despire higher client load FOSS4G-EU 2015, Como 14th July 2015
  • 12. Introduction to JMeter  Open Source performance measurement tool written in Java  This tool will be highly used during the workshop for testing the GeoServer performances against Benchmark data  More informations can be found at the Jmeter site: http://jakarta.apache.org/jmeter/ FOSS4G-EU 2015, Como 14th July 2015
  • 13. Introduction to JMeter  Easy-to-use UI  Allows to setup multiple thread groups, different parallelism and request count, to ramp up the load  Can use CSV files to generate semi-randomized requests  Can execute parameterized tests  Reports results in a simple table  Uses Assertions for checking test results  Can be executed in batch FOSS4G-EU 2015, Como 14th July 2015
  • 14. Using JMeter  Thread group: how many threads  Loop: how many requests  HTTP sampler: the request  CSV: read request params from CSV  Default: do not repeat what’s equal in all requests  Performance summary FOSS4G-EU 2015, Como 14th July 2015
  • 15. Result Summary  Results table  Run the benchmarks 2-3 times, let the results stabilize  Save the results, check other optimizations, compare the results FOSS4G-EU 2015, Como 14th July 2015
  • 16. Exercise  Let's start with practice!!!  Follow the instructions on the link below http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_simple.html  You should be able to reproduce the test using the GeoServer Training instance installed in your PC  This simple exercise will show you how to use JMeter with a basic configuration for testing GeoServer performances FOSS4G-EU 2015, Como 14th July 2015
  • 17. Randomized Multiscale Testing  In the example before, we ran a test on a layer always setting the same bounding box and image size  Now we should setup a more complicated test asking for different bounding boxes and resolutions, aka Multiscale testing  Multiscale testing is useful because it stresses GeoServer requesting new data each time, mimicking real usage conditions and providing better testing for your SLD scale dependencies  With JMeter we can execute the test by passing a CSV file containing multiple bounding boxes and image dimensions FOSS4G-EU 2015, Como 14th July 2015
  • 18. Exercise  Multiscale test created using wms_request.py on a mosaic layer  Follow the instructions on the link below http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_multiscale.html  This exercise will show you how to use create a CSV file containing requests at multiple resolutions and how to run JMeter with it FOSS4G-EU 2015, Como 14th July 2015
  • 20. Intro to GDAL Utilities  Online Material  http://geoserver.geo- solutions.it/edu/en/raster_data/processing.html  http://www.gdal.org/gdal_utilities.html FOSS4G-EU 2015, Como 14th July 2015
  • 21. Raster Data CheckList  Objectives  Fast extraction of a subset of the data  Fast extraction of the desired resolution  Check-list  Avoid having to open a large number of files per request  Avoid parsing of complex structures and slow compressions  Get to know your bottlenecks  CPU vs Disk Access Time vs Memory  Experiment with  Format, compression, different color models, tile size, overviews, GeoServer configuration FOSS4G-EU 2015, Como 14th July 2015
  • 22. Peculiar Formats  PNG/JPEG direct serving (as data sources)  Bad formats (especially in Java)  No tiling (or rarely supported)  Chew a lot of memory and CPU for decompression  Mitigate with external overviews  Any input ASCII format (GML grid, ASCII grid)  JPEG2000  Extensible and rich, not (always) fast, can be difficult to tune for performance (might require specific encoding options)  MrSID (can work, needs tuning)  ECW (licensing issues) FOSS4G-EU 2015, Como 14th July 2015
  • 23. GeoTiff for the win  To remember: GeoTiff is a swiss knife  But you don’t want to cut a tree with it!  Tremendously flexible, good for for most (not all) use cases  BigTiff pushes the GeoTiff limits farther  Use GeoTiff when  Overviews and Tiling stay within 4GB  No additional dimensions  Consider BigTiff for very large file (> 4 GB)  Support for tiling  Support for Overviews  Can be inefficient with very large files + small tiling FOSS4G-EU 2015, Como 14th July 2015
  • 24. GeoTiff preparation  (Optional) Use gdal_warp to transform the data in the most used output reference system (mind, any reprojection ruins the data a bit)  Use gdal_translate to add inner tiling and fix eventual issues with coordinate reference system  Add compression options if you disks are small/slow/not local (consider JPEG compression with YCBCR color interpretation for photos, LZW/Deflate for scientific data)  Use gdaladdo to add internal overviews (remember to replicate compression here) FOSS4G-EU 2015, Como 14th July 2015
  • 25. Possible structures Single GeoTiff with internal tiling and overviews (GeoTiff < 2GB, BigTiff < 20-50GB) Mosaic of GeoTiff, each one with internal tiling and overviews (< 500GB, not too many files) Image pyramid, each level has lower resolution than the previous, each level has tiles with inner tiling but no overviews (also possible to mix, less levels and use internal overviews) > 500GB FOSS4G-EU 2015, Como 14th July 2015
  • 26. Choosing Formats and Layouts  Use ImageMosaic when:  A single file gets too big (inefficient seeks, too much metadata to read, etc..)  Multiple Dimensions (time, elevation, others..)  Avoid mosaics made of many very small files  Single granules can be large  Use Tiling + Overviews + Compression on granules  Use ImagePyramid when:  Tremendously large dataset  Too many files / too large files  Need to serve at all scales  Especially low resolution  For single granules (< 2Gb) GeoTiff is generally a good fit FOSS4G-EU 2015, Como 14th July 2015
  • 27. Proper Mosaic Preparation  Optimize files as if you were serving them individually  Keep a balance between number and dimensions of granules  If memory is scarce:  USE_JAI_IMAGREAD to true  USE_MULTITHREADING to false*  Otherwise  USE_JAI_IMAGREAD to false  ALLOW_MULTITHREADING to true (Load data from different granules in parallel)  More Info  http://geoserver.geo- solutions.it/edu/en/raster_data/mosai c_pyramid.html FOSS4G-EU 2015, Como 14th July 2015
  • 28. Proper Pyramid Preparation  Use gdal_retile for creating the pyramid  Prepare the list of tiles to be retiled  Create the pyramid with GDAL retile (grab a coffee!)  Chunks should not be too small (here 2048x2048), if you go larger, use also inner tiling  If the input dataset is huge use the useDirForEachRow option  Too many files in a dir is bad practice  Make sure the number of level is consistent with usage  Too few  bad performance at high scale  GeoServer config  same as mosaic  More Info  http://geoserver.geo- solutions.it/edu/en/raster_data/mosaic_pyramid.html FOSS4G-EU 2015, Como 14th July 2015
  • 29. Proper GeoServer Coverage Config Opts  Make sure native JAI is installed  Install the TurboJPEG extension  Enable JAI Mosaicking native acceleration  Give JAI enough memory  Don’t raise JAI memory Threshold too high  Rule of thumb: use 2 X #Core Tile Threads (check next slide)  Play with tile Recycling against your workflows (might help, might not) FOSS4G-EU 2015, Como 14th July 2015
  • 30. Proper GeoServer Coverage Config Opts  Multithreaded Granule Loading  Allows to fine tuning multithreading for ImageMosaic  Orthogonal to JAI Tile Threads  Rule of Thumb: use 2 X #Core Tile Threads  Perform testing to fine tune depending on layer configuration as well as on typical requests  ImageIO Cache threshold  decide when we switch to disk cache (very large WCS requests) FOSS4G-EU 2015, Como 14th July 2015
  • 31. Exercise  Let's check the effect of optimization in GeoServer  Follow the instructions on the links below http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_mosaic.html  This test uses JMeter for testing GeoServer performances before and after the optimization of a Mosaic Layer. FOSS4G-EU 2015, Como 14th July 2015
  • 32. Additional Material & Exercises  Using ImageMosaic with Footprint  http://geoserver.geo- solutions.it/edu/en/raster_data/imagemosaic_footprint.html  Advanced Processing With GDAL Utilities  http://geoserver.geo- solutions.it/edu/en/raster_data/advanced_gdal/index.html FOSS4G-EU 2015, Como 14th July 2015
  • 34. Vector data checklist  What do we want from vector data:  Binary data  No complex parsing of data structures  Fast extraction of a geographic subset  Fast filtering on the most commonly used attributes FOSS4G-EU 2015, Como 14th July 2015
  • 35. Choosing a format  Slow formats  WFS  GML  DXF  Good formats, local and index-able  Shapefile  Directory of shapefiles  SDE  Spatial databases: PostGIS, Oracle Spatial, DB2, MySQL*, SQL server*  Swiss Knife  SQL Views FOSS4G-EU 2015, Como 14th July 2015
  • 36. DBMS Checklist  Rich support for complex native filters  Use connection pooling  Validate connections (with proper pooling)  Table Clustering  Spatial Indexing  Spatial Indexing  Spatial Indexing  Alphanumeric Indexing  Alphanumeric Indexing  Alphanumeric Indexing  Did we mention indexes? FOSS4G-EU 2015, Como 14th July 2015
  • 37. Shapefile preparation  Remove .qix file if present  If there are large DBF attributes that are not in use, get rid of them using ogr2ogr, e.g.: ogr2ogr -select FULLNAME,MTFCC arealm.shp tl_2010_08013_arealm.shp  If on Linux, enable memory mapping, faster, more scalable (but will kill Windows): FOSS4G-EU 2015, Como 14th July 2015
  • 38. Shapefile filtering  Stuck with shapefiles and have scale dependent rules like the following?  Show highways first  Show all streets when zoomed in  Use ogr2ogr to build two shapefiles, one with just the highways, one with everything, and build two layers, e.g.: ogr2ogr -sql "SELECT * FROM tl_2010_08013_roads WHERE MTFCC in ('S1100', 'S1200')" primaryRoads.shp tl_2010_08013_roads.shp  Or hire us to develop non-spatial indexing for shapefile! FOSS4G-EU 2015, Como 14th July 2015
  • 39. PostGIS specific hints  PostgreSQL out of the box configured for very small hardware: http://wiki.postgresql.org/wiki/Performance_Optimization  Make sure to run ANALYZE after data imports (updates optimizer stats)  As usual, avoid large joins in SQL views, consider materialized views  If the dataset is massive, CLUSTER on the spatial index:  http://postgis.refractions.net/documentation/manual- 1.3/ch05.html FOSS4G-EU 2015, Como 14th July 2015
  • 40. PostGIS specific hints  Careful with prepared statements (bad performances)  USE CASE: the layer’s style allows to display the whole layer in a single shot (no scale dependencies)  prepared statements will slow down execution  EXPLANATION: Postgis will choose to use the spatial index in all cases, this makes retrieving the full data set 2-4 times slower than when using a sequential scan  COUNTERMEASURE: Not using prepared statement allows postgis to figure out a suitable plan based on the request bbox instead (assuming someone run "vacuum analyze" on the database to update the index statistics, and of course, provided there is a spatial index to start with) FOSS4G-EU 2015, Como 14th July 2015
  • 41. SQL Server specific hints  Switch to PostgreSQL/PostGIS! FOSS4G-EU 2015, Como 14th July 2015
  • 42. Oracle specific hints  Switch to PostgreSQL/PostGIS! FOSS4G-EU 2015, Como 14th July 2015
  • 43. SQL Views Support  Vector Data Swiss Knife  http://geoserver.geo- solutions.it/edu/en/adding_data/add_sqllayers.html FOSS4G-EU 2015, Como 14th July 2015
  • 44. Exercise  Let's compare Shapefile and PostGIS in GeoServer  Follow the instructions on the links below http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_vector.html  This test uses Jmeter for comparing GeoServer performances using a Shapefile or an optimized PostGIS database containing the same data FOSS4G-EU 2015, Como 14th July 2015
  • 46. Use scale dependencies  Never show too much data  the map should be readable, not a graphic blob. Rule of thumb: 1000 features max in the display, have labels show up when zoomed in  Show details as you zoom in  Eagerly add MinScaleDenominator to your SLD rules  Add more expensive rendering when there are less features  Key to get both a good looking and fast map FOSS4G-EU 2015, Como 14th July 2015
  • 48. Labeling  Labeling conflict resolution is expensive, limit to the most inner zooms  Halo is important for readability, but adds significant overhead  Careful with maxDisplacement, makes for various label location attempts FOSS4G-EU 2015, Como 14th July 2015
  • 49. FeatureTypeStyle  GeoServer uses SLD FeatureTypeStyle objects as Z layers for painting  Each one allocates its own rendering surface (which can use a lot of memory), use as few as possible FOSS4G-EU 2015, Como 14th July 2015
  • 50. Use translucency sparingly  Translucent display is expensive, use it sparingly  e.g. translucent fill <CssParameter name="fill-opacity">0.5</CssParameter> FOSS4G-EU 2015, Como 14th July 2015
  • 51. Exercise  Let's compare bad styles with properly setup ones  Follow the instructions on the links below http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_styles.html  This test uses JMeter to compare a style without scale dependencies, showing too much data and too many labels, with a one with proper scale dependencies and providing a better looking output at the same output FOSS4G-EU 2015, Como 14th July 2015
  • 53. Tile caching with GeoWebCache  Tile oriented maps, fixed zoom levels and fixed grid  Useful for stable layers, backgrounds  Protocols: WMTS, TMS, WMS-C, Google Maps/Earth, VE  Speedup compared to dynamic WMS: 10 to 100 times, assuming tiles are already cached (whole layer pre- seeded)  Suitable for:  Mostly static layer  No/few dynamic parameters (CQL filters, SLD params, SQL query params, time/elevation, format options) FOSS4G-EU 2015, Como 14th July 2015
  • 54. Embedded GWC advantage  No double encoding when using meta-tiling, faster seeding FOSS4G-EU 2015, Como 14th July 2015
  • 55. Space considerations  Seeding Colorado, assuming 8 cores, one layer, 0.1 sec 756x756 metatile, 15KB for each tile  Do yours: http://tinyurl.com/3apkpss  Not enough disk space? Set a disk quota Zoom level Tile count Size (MB) Time to seed (hours) Time to seed (days) 13 58,377 1 0 0 14 232,870 4 0 0 15 929,475 14 0 0 16 3,713,893 57 1 0 17 14,855,572 227 6 0 18 59,396,070 906 23 1 19 237,584,280 3,625 92 4 20 950,273,037 14,500 367 15 FOSS4G-EU 2015, Como 14th July 2015
  • 56. DiskQuota  Online Material  http://geoserver.geo- solutions.it/edu/en/caching_data/diskquotas.html#disk-quota- configuration FOSS4G-EU 2015, Como 14th July 2015
  • 57. More Tweaks  REST Interface  Automate/Script management  Truncate  Remove Cached tiles  Seed  Pre-generate cache  Reseed  Regenerate tiles (due to data update, for instance)  Every seed/truncate operation can be filtered by:  Bounding box / Zoom limits  / Format / GridSet  Parameters (style, time, other dimensions, cql_filter…)  Thread number FOSS4G-EU 2015, Como 14th July 2015
  • 58. More Tweaks  REST Seed example FOSS4G-EU 2015, Como 14th July 2015 curl -u admin:password -H 'Content-type: text/xml' -H 'Accept: text/xml' -XPOST -d ' <seedRequest> <name>topp:states</name> <type>seed</type> <gridSetId>EPSG:4326</gridSetId> <bounds> <coords> <double>-100</double><double>- 80</double><double>100</double><double>80</double> </coords> </bounds> <zoomStart>0</zoomStart><zoomStop>8</zoomStop> <format>image/png8</format> <threadCount>04</threadCount> <parameters> <entry> <string>CQL_FILTER</string> <string>TOTPOP BETWEEN 0 AND 10000</string> </entry> </parameters> </seedRequest> ' http://localhost:8080/geoserver/gwc/rest/seed/topp:states.xml
  • 59. More Tweaks  Online Resources for GWC REST API  Documentation  http://geowebcache.org/docs/current/rest/seed.html  http://docs.geoserver.org/stable/en/user/geowebcache/rest/seed .html  Bash script for seed/reseed/truncate/mass truncate operations  https://github.com/geosolutions- it/scripts/blob/master/geowebcache/gwc.sh  Sample script for clean cache in geoserver cluster  https://github.com/geosolutions- it/scripts/blob/master/geowebcache/cleancache.sh FOSS4G-EU 2015, Como 14th July 2015
  • 60. More Tweaks  Clustering Single Shared Cache  Large, shared disk with fast R/W access  Perfect for bkg layers  Problematic set up in public cloud  Clustering Multiple Indipendent Caches  Exploit local, small, instance-based caches  Good for short lived caches  Duplication hit  rsynch can help  Real-World Use Case  Dedicated Instances with shared cache for Bkg Layers  Indipendent Caches for other layers FOSS4G-EU 2015, Como 14th July 2015
  • 61. More Tweaks  Client-side caching of tiles  Does not work with browsers in private mode <expireClientsList> <expirationRule minZoom="0" expiration="7200" /> <expirationRule minZoom="10" expiration="600" /> </expireClientsList> FOSS4G-EU 2015, Como 14th July 2015
  • 62. More Tweaks  Use the right formats  JPEG for background data (e.g. ortos)  PNG8 + precomputed palette for background data (e.g. ortos)  PNG8 full for overlays with transparency  The format impacts also the disk space needed! (as well as the generation time)  Check this blog post FOSS4G-EU 2015, Como 14th July 2015
  • 63. Exercise  Let's compare GeoServer WMS with GeoWebCache fullWMS.  Follow the instructions on the links below http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_gwc.html  This test uses JMeter for comparing GeoServer WMS performances against those of GeoWebCache fullWMS  Mind, this is not the same as comparing WMS with pure tile caching, GWC has to read the tiles from disk, decompress, assemble the output image, and compress it again in PNG (some speedup still, but not 10 times) FOSS4G-EU 2015, Como 14th July 2015
  • 65. Resource Limits  Limit the amount of resources dedicated to an individual request  Improve fairness between requests, by preventing individual requests from hijacking the server and/or running for a very long time  EXTREMELY IMPORTANT in production environment  WHEN TO TWEAK THEM?  Frequent OOM Errors despite plenty of RAM  Requests that keep running for a long time (e.g. CPU usage peaks even if no requests are being sent)  DB Connection being killed by the DBMS while in usage (ok, you might also need to talk to the DBA..) FOSS4G-EU 2015, Como 14th July 2015
  • 66. WMS request limits  Max memory per request: avoid large requests, allows to size the server memory (max concurrent request * max memory)  Max time per request: avoid requests taking too much time (e.g., using a custom style provided with dynamic SLD in the request)  Max errors: best effort renderer, but handling errors takes time FOSS4G-EU 2015, Como 14th July 2015
  • 67. WFS request limits  Max feature returned, configured as a global limit  Return feature bbox: reduce amount of generated GML  Per layer max feature count FOSS4G-EU 2015, Como 14th July 2015
  • 68. WCS request limits FOSS4G-EU 2015, Como 14th July 2015
  • 69. Exercise  Let's show how resource limits can be used to avoid issues with slow requests  Follow the instructions on the links below http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_wmslimits.html  This test artificially limits the database resources to quickly generate a starvation situation in which the limits kick-in FOSS4G-EU 2015, Como 14th July 2015
  • 70. Control flow  Control how many requests are executed in parallel, queue others:  Increase throughput  Control memory usage  Enforce fairness  More info  http://docs.geoserver.org/latest/en/user/community/controlflow/i ndex.html FOSS4G-EU 2015, Como 14th July 2015
  • 71. $GEOSERVER_DATA_DIR/controlflow.properties # don't allow more than 16 GetMap requests in parallel ows.wms.getmap=16 Control flow 17% FOSS4G-EU 2015, Como 14th July 2015
  • 72. Resource Limits  They need to be tweaked together with Control- Flow  Limiting individual requests  Resource Limits  Limiting amount of parallel request  Control Flow  When time is involved make sure you keep into account all pieces of the response chain  E.G. Limited rendering time for WMS on data coming from WMS, items to take into account are  LifeTime of DB Connection (usually long)  WaitTime for a new connection (we don’t want to queue requests at the connection pool when they area already eating memory!) FOSS4G-EU 2015, Como 14th July 2015
  • 73. Connection Pooling Tricks  Connection pool size should be proportional to the number of concurrent requests you want to serve (obvious no?)  Activate connection validation  Mind networking tools that might cut connections sitting idle (yes, your server is not always busy), they might cut the connection in “bad” ways (10 minutes timeout before the pool realizes the TCP connection attempt gives up)  Read more at  http://geoserver.geo- solutions.it/edu/en/adv_gsconfig/db_pooling.html  https://docs.google.com/document/d/1O02PeDRYKt2xLW G21k6BmcaRPzVRvXb4SQcrcZf4bHQ/edit?usp=sharing FOSS4G-EU 2015, Como 14th July 2015
  • 74. Exercise  Let's show how control-flow works in GeoServer  Follow the instructions on the links below http://geoserver.geo- solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_controlflow.html  This test uses JMeter for testing GeoServer performances before and after configuring the control-flow plugin. FOSS4G-EU 2015, Como 14th July 2015
  • 75. JVM and deploy configuration
  • 76. Premise  The options discussed here are not going to help visibly if you did not prepare the data and the styles  They are finishing touches that can get performance up once the major data bottlenecks have been dealt with  Check “Running in production” instructions here FOSS4G-EU 2015, Como 14th July 2015
  • 77. JVM settings  --server: enables the server JIT compiler  --Xms2048m -Xmx2048m: sets the JVM use two gigabytes of memory  --XX:+UseParallelOldGC -XX:+UserParallelGC: enables multi-threaded garbage collections, useful if you have more than two cores  --XX:NewRatio=2: informs the JVM there will be a high number of short lived objects  --XX:+AggressiveOpt: enable experimental optimizations that will be defaults in future versions of the JVM FOSS4G-EU 2015, Como 14th July 2015
  • 78. Setup a local cluster  Oracle Java2D locks when drawing antialiased vectors  Limits scalability severely  Two options  Use OpenJDK, it’s slower at rendering but scales up well  Use Apache mod_proxy_balance and setup a GeoServer each 2/4 cores mod_proxy_balance GeoServer GeoServer GeoServer FOSS4G-EU 2015, Como 14th July 2015
  • 79. Clustering advantage 66%  FOSS4G 2010 vector benchmarks (roads/buildings/isolines and so on, over the entire Spain)  GeoServer was benchmarked using Oracle JDK without local clustering FOSS4G-EU 2015, Como 14th July 2015
  • 80. Marlin renderer  The OpenJDK Java2D renderer scales up, but it’s not super-fast when the load is small (1 request at a time)  Marlin-renderer to the rescue: https://github.com/bourgesl/marlin-renderer  Complex map, 10 parallel requests, different zoom levels have different details showing up FOSS4G-EU 2015, Como 14th July 2015
  • 81. Upgrade!  Performance tends to go up version by version  Please do use a recent GeoServer version  FOSS4G 2010 vector benchmark with different versions of GeoServer FOSS4G-EU 2015, Como 14th July 2015
  • 82. Exercise  Let's show how Marlin renderer can speed up map rendering under Windows, using GeoServer 2.5.x  Follow the instructions on the links below http://geoserver.geo-solutions.it/educational/en/adv_gsconfig/jmeter/jmeter_marlin.html  This test uses JMeter for testing GeoServer performances before and after setting up Marlin renderer FOSS4G-EU 2015, Como 14th July 2015
  • 84. Clustering with GeoServer  Online Material  http://geoserver.geo- solutions.it/edu/en/clustering/index.html FOSS4G-EU 2015, Como 14th July 2015