FOSS4G-Europe talk, about various techniques you can use, if you need to transfer larger amounts of vector data from server to client.

1. Push it through the wire
If it does not work, push more
J´achym ˇCepick´y1
1Geosense s.r.o. http://geosense.cz
FOSS4G-Europe 2015
Push it through the wire
If it does not work, push more
J´achym ˇCepick´y1
1Geosense s.r.o. http://geosense.cz
FOSS4G-Europe 2015
2015-07-16
Push it through the wire
Hi

2. 2015-07-16
Push it through the wire
Hi everybody, my name is Jachym Cepicky, and I’ve been involved for
longer the 10 years in developent of the open source software for
geospatial. I started on the desktop and server side, last years I’m
involved more with development of JavaScript based client
applications.

3. 2015-07-16
Push it through the wire
I would like to give you brief overview about the one of the problems
we are struggling in today’s development of web mapping
applications. It’s the data. Especially, when they are slightly bigger,
then normally.

4. 2015-07-16
Push it through the wire
You probably know, that the development of web-based applications
has completely changed during last couple of years. We are no
longer writing JavaScript code and sending directly to the client like
we did at the beginning.

5. OpenLayers 3.7.0
ol.js: 459K
ol-debug.js 3.6M
ol-optimized.js < 100K
OpenLayers 3.7.0
ol.js: 459K
ol-debug.js 3.6M
ol-optimized.js < 100K
2015-07-16
Push it through the wire
We are compiling the code, so it’s optimized, running faster so it is
transfared faster from the server to client. In this way, you can
compress javascript code from OpenLayers from more then
megabyte to several kilobytes.

6. Raster × Vectors
Size × Rendering speed × Transport time × Flexibility ×
Amount of informations
Raster × Vectors
Size × Rendering speed × Transport time × Flexibility ×
Amount of informations
2015-07-16
Push it through the wire
Nowadays JavaScript interpreters and renderers do not have any
problem with rendering of really big amounts of data. Raster images
are usually coming in compressed form of JPEGs or PNG files and to
render them, no rasterization is needed any more.
Also amount of informations in raster file can be significantly lower,
then it is in vector data. We can say, that amount of information in
raster file is limited to its resolution and number of colors, which can
be interpreted by the user, amount of informations in vector file is
given by number of vertexes describing the phenomenon and number
of attributes per feature. You also can join more tables together and
add another level of complexity to the data.

7. 2015-07-16
Push it through the wire
From certain point from file size perspective, raster files are smaller
then vector files. It would make hight sense to transform your vector
data on the server and send them as rasters to the client. Why would
you need vector data on client side anyway?

8. Why vector in the browser anyway?
better user interaction
dynamic aspect of the data
editing in the browser
...
Why vector in the browser anyway?
better user interaction
dynamic aspect of the data
editing in the browser
...
2015-07-16
Push it through the wire
• You want to enable user interaction with the data, add dynamic
aspect of the data NEXT
• you want to enable the editing NEXT
• ...

9. Why vector in the browser anyway?
better user interaction
dynamic aspect of the data
editing in the browser
...
Why vector in the browser anyway?
better user interaction
dynamic aspect of the data
editing in the browser
...
2015-07-16
Push it through the wire
• You want to enable user interaction with the data, add dynamic
aspect of the data NEXT
• you want to enable the editing NEXT
• ...

10. Why vector in the browser anyway?
better user interaction
dynamic aspect of the data
editing in the browser
...
Why vector in the browser anyway?
better user interaction
dynamic aspect of the data
editing in the browser
...
2015-07-16
Push it through the wire
• You want to enable user interaction with the data, add dynamic
aspect of the data NEXT
• you want to enable the editing NEXT
• ...

11. Why vector in the browser anyway?
better user interaction
dynamic aspect of the data
editing in the browser
...
Why vector in the browser anyway?
better user interaction
dynamic aspect of the data
editing in the browser
...
2015-07-16
Push it through the wire
but the main reason is

12. Vectors are cool!
Vectors are cool!
2015-07-16
Push it through the wire
vector data are just cool, gives you the feeling of power, to have them
on the client side

13. Compress the communication between server and client
Proper data format
Consider lossy compression of the data on the server
Transfer only the data, you need to transfer
Compress the communication between server and client
Proper data format
Consider lossy compression of the data on the server
Transfer only the data, you need to transfer
2015-07-16
Push it through the wire
Those are suggested steps, when dealing with large amount of vector
data:
• Compress the communication between server and client NEXT
• Choose the file format carefully NEXT
• Use lossy compression of the vector data NEXT
• Do transfer only the data, user needs to see

14. Compress the communication between server and client
Proper data format
Consider lossy compression of the data on the server
Transfer only the data, you need to transfer
Compress the communication between server and client
Proper data format
Consider lossy compression of the data on the server
Transfer only the data, you need to transfer
2015-07-16
Push it through the wire
Those are suggested steps, when dealing with large amount of vector
data:
• Compress the communication between server and client NEXT
• Choose the file format carefully NEXT
• Use lossy compression of the vector data NEXT
• Do transfer only the data, user needs to see

15. Compress the communication between server and client
Proper data format
Consider lossy compression of the data on the server
Transfer only the data, you need to transfer
Compress the communication between server and client
Proper data format
Consider lossy compression of the data on the server
Transfer only the data, you need to transfer
2015-07-16
Push it through the wire
Those are suggested steps, when dealing with large amount of vector
data:
• Compress the communication between server and client NEXT
• Choose the file format carefully NEXT
• Use lossy compression of the vector data NEXT
• Do transfer only the data, user needs to see

16. Compress the communication between server and client
Proper data format
Consider lossy compression of the data on the server
Transfer only the data, you need to transfer
Compress the communication between server and client
Proper data format
Consider lossy compression of the data on the server
Transfer only the data, you need to transfer
2015-07-16
Push it through the wire
Those are suggested steps, when dealing with large amount of vector
data:
• Compress the communication between server and client NEXT
• Choose the file format carefully NEXT
• Use lossy compression of the vector data NEXT
• Do transfer only the data, user needs to see

17. Compress the server-client communication
mod gzip
80MB GeoJSON → 7MB
Compress the server-client communication
mod gzip
80MB GeoJSON → 7MB
2015-07-16
Push it through the wire
Setting zip compression on the server seems to be the most primitive
thing, but you would be surprised, how often do we forget to switch
this option on on the Apache server (or any other).
In this way, you can compress about 80MB big GeoJSON file to
nearly 7MB, what is about 10× smaller.

18. File format1
GML vs. GeoJSON
GML ≈ 40MB
GeoJSON ≈ 80MB
File size vs. processing time
1
50 000 polygon features, OSM dataset
File format1
GML vs. GeoJSON
GML ≈ 40MB
GeoJSON ≈ 80MB
File size vs. processing time
1
50 000 polygon features, OSM dataset
2015-07-16
Push it through the wire
Choosing proper file format could seem as easy option. On our test
data - file of about 50 000 of polygon features, which file format do
you think will be bigger - GML or GeoJSON ?
Yes, it’s GeoJSON - while GeoJSON needs about 80MB of space,
GML needed only 40MB. I do not want to say, generally speaking,
GML is less verbose then GeoJSON, I want to say, choosing the right
format can be tricky.
Where GeoJSON beats GML is the speed of data processing on the
client in any case.

19. 2015-07-16
Push it through the wire
If we are displaying compressed JPEG images to users saying ”it’s
ok, you see the data nearly as they are in reality, why are we note
doing the same with vectors?

20. https://www.jasondavies.com/simplify/
https://www.jasondavies.com/simplify/
2015-07-16
Push it through the wire
Who needs 1000 vertexes per line, when at certain zoom level only 3
would be enough?
Vector mapping libraries like OpenLayers or Leaflet are doing heavy
pre-processing of the data, before they are send to canvas, so that
only reasonable amount of vertexes is displayed. Is there a way, how
to send pre-processed data from the server?

21. TopoJSON
Data size progress
GeoJSON 80MB → TopoJSON 3.3MB → zipped 500KB
2
http://github.com/mbostock/topojson
2
50 000 polygon features, OSM dataset
TopoJSON
Data size progress
GeoJSON 80MB → TopoJSON 3.3MB → zipped 500KB
2
http://github.com/mbostock/topojson
2
50 000 polygon features, OSM dataset
2015-07-16
Push it through the wire
TopoJSON
The answer is TopoJSON format. While the original GeoJSON file
had 80MB, new created TopoJSON file has about 3.3MB. If you zip it,
it then has about 500kb. How is this possible?

22. TopoJSON
http://github.com/mbostock/topojson
Extension of GeoJSON
Introduces new type ”topology”
Coordinates stored in arcs array
Arcs are similar to line strings. More arcs form together
geometry
Lat & Long → relative coordinates, ∆ values
TopoJSON
http://github.com/mbostock/topojson
Extension of GeoJSON
Introduces new type ”topology”
Coordinates stored in arcs array
Arcs are similar to line strings. More arcs form together
geometry
Lat & Long → relative coordinates, ∆ values
2015-07-16
Push it through the wire
TopoJSON
TopoJSON format was introduced by Mike Bostock who is author of
many useful tools, which are addressing some of the issues, web
developers are facing, when dealing with bigger amounts of data.
You’ve probably heard about D3.js library for drawing graphs, but
there is more.
TopoJSON is an extension of GeoJSON. TopoJSON introduces a
new type, ”Topology”, that contains GeoJSON objects. A topology
has an objects map which indexes geometry objects by name. These
are standard GeoJSON objects, such as polygons, multi-polygons
and geometry collections. However, the coordinates for these
geometries are stored in the topology’s arcs array, rather than on
each object separately.
An arc is a sequence of points, similar to a line string; the arcs are
stitched together to form the geometry.
Lastly, the topology has a transform which specifies how to convert
delta-encoded integer coordinates to their native values (such as

23. Does not support big integer – problem with big IDs
3
3
https://github.com/mbostock/topojson/wiki
Does not support big integer – problem with big IDs
3
3
https://github.com/mbostock/topojson/wiki
2015-07-16
Push it through the wire
TopoJSON uses among others lossy compression, by simplifying the
arcs to certain scale. If you want to go more into details, there is nice
presentation of Mike Bostock explaining details of the algorithm at
https://github.com/mbostock/topojson/wiki
One issue I see with TopoJSON is, it does not support big integer
attributes. I wanted to apply it for czech cadaster data, which are
using big integer as feature id, unluckily, this was not possible.

24. Transfer only the data, you need to transfer
Transfer only the data, you need to transfer
2015-07-16
Push it through the wire
Always have in mind: you do not want to transfer all the data. You just
want the user to make believe, he sees all the data.

25. 2015-07-16
Push it through the wire
We are all used to transfer only needed amount of raster data, as
needed for current view - either big images using WMS or smaller
tiles, using one of the proprietary or open standards, like WMTS. Why
are we not doing this the same way with vectors?

26. Tiled vectors
http://openlayers.org/en/v3.7.0/examples/
tile-vector.html →
http://jsfiddle.net/og7m21t7/
Tiled vectors
http://openlayers.org/en/v3.7.0/examples/
tile-vector.html →
http://jsfiddle.net/og7m21t7/
2015-07-16
Push it through the wire
Tiled vectors
There are probably two reasons for that: first, vector data were always
somehow smaller the raters, but this is not necessary the fact any
more.
Second: it’s much more complicated, especially if you consider all the
edge conditions, like polygon data, with holes in it, spread over 4 tiles.

27. https://github.com/mapbox/vector-tile-spec
https://github.com/mapbox/vector-tile-spec
2015-07-16
Push it through the wire
However it’s possible. MapBox has defined it’s own format for so
called tiled vectors https://github.com/mapbox/vector-tile-spec it was
published as open specification, but as far as I know, it is not
implemented in any other software instead of MapBox so far (correct
me, if I’m wrong)

28. TileStache
http://tilestache.org
TileStache
http://tilestache.org
2015-07-16
Push it through the wire
TileStache
Other option is to create vector cache using TileStache server. The
tiles can be either cut at edges - one polygon is split into 2 or 4, if it is
not fitting into one. Or you can use e.g. centroids who are ”within”the
pre-defined tile and assign only those features to the ”tile”.
TileStache can produce geojson tiles as well as topojson tiles.

29. http://jsfiddle.net/og7m21t7/
http://jsfiddle.net/og7m21t7/
2015-07-16
Push it through the wire
With tiled polygon data, you are facing another problem: polygons are
usually spread between more tiles. You have to either resign on
displaying polygon boundaries, what is possible e.g. for big water
bodies. But if you are dealing with e.g. parcel data, different approach
has to be taken.
You can read e.g. only polygons, which centroid fit’s into given tile -
but then, long streets are not seen.
There is a link under images, which contains OpenLayers tiled vector
example loaded into jsfiddle environment, where you can play with
style settings and see the problem. I hope to show it to you at the end
of the session.

30. https://goo.gl/oSutyN
https://goo.gl/oSutyN
2015-07-16
Push it through the wire
Here is a screenshot from video, which you can reach on the link
below, showing the possibility of vector tiles, which do not contain
croped vector features. In this approach only the features which
centroids fit into the tile are downloaded.

31. 2015-07-16
Push it through the wire
As result of all the effort, user will see similar progress, has he sees
already with vector data - they will be loaded in tiles.
To conclude:

32. Conclusion
Do I need to use all this compression, file format, tiling always I
want to display single vector point / GPX log file?
Conclusion
Do I need to use all this compression, file format, tiling always I
want to display single vector point / GPX log file?
2015-07-16
Push it through the wire
Conclusion
Do I need to use all this compression, file format, tiling always I want
to display single vector point or GPX log file?

33. Conclusion
NO!
Conclusion
NO!
2015-07-16
Push it through the wire
Conclusion
Of course not

34. Conclusion
Use it when you needed, when things seems slow.
Conclusion
Use it when you needed, when things seems slow.
2015-07-16
Push it through the wire
Conclusion
Those approaches are for the situations, your application loading
seems to be too slow and you are trying to define the bottlenecks -
either it’s just the download speed or it seems to be rendering time as
well.

35. Conclusion
With tiling, I get faster vectors, what do I loose?
Conclusion
With tiling, I get faster vectors, what do I loose?
2015-07-16
Push it through the wire
Conclusion
Caching and tiling of vector data gives you more speed

36. Conclusion
Topicality and simplicity – you can not have up-to-date
pre-cached data. dataspeed × datasize × datatopicality
Conclusion
Topicality and simplicity – you can not have up-to-date
pre-cached data. dataspeed × datasize × datatopicality
2015-07-16
Push it through the wire
Conclusion
But you loose simplicity and topicality of the data. You can not have
fast, up-to-date and big data at the same time.

37. Thank you
jachym.cepicky@geosense.cz
http://geosense.cz
@jachymc
Thank you
jachym.cepicky@geosense.cz
http://geosense.cz
@jachymc
2015-07-16
Push it through the wire
Closing