Download to read offline
Uploaded byJachym Cepicky
Push it through the wire
The document discusses challenges and solutions in developing web mapping applications, particularly when dealing with large amounts of vector data. It highlights the advantages of using formats like TopoJSON for reduced file sizes and improved data transmission, as well as the importance of data compression and careful file format selection. The document also touches upon strategies for efficient data transfer, including tiling vectors and optimizing server-client communication.
More Related Content
Push it through the wire
- 1. Push it throughthe wire If it does not work, push more J´achym ˇCepick´y1 1Geosense s.r.o. http://geosense.cz FOSS4G-Europe 2015 Push it through the wire If it does not work, push more J´achym ˇCepick´y1 1Geosense s.r.o. http://geosense.cz FOSS4G-Europe 2015 2015-07-16 Push it through the wire Hi
- 2. 2015-07-16 Push it throughthe wire Hi everybody, my name is Jachym Cepicky, and I’ve been involved for longer the 10 years in developent of the open source software for geospatial. I started on the desktop and server side, last years I’m involved more with development of JavaScript based client applications.
- 3. 2015-07-16 Push it throughthe wire I would like to give you brief overview about the one of the problems we are struggling in today’s development of web mapping applications. It’s the data. Especially, when they are slightly bigger, then normally.
- 4. 2015-07-16 Push it throughthe wire You probably know, that the development of web-based applications has completely changed during last couple of years. We are no longer writing JavaScript code and sending directly to the client like we did at the beginning.
- 5. OpenLayers 3.7.0 ol.js: 459K ol-debug.js3.6M ol-optimized.js < 100K OpenLayers 3.7.0 ol.js: 459K ol-debug.js 3.6M ol-optimized.js < 100K 2015-07-16 Push it through the wire We are compiling the code, so it’s optimized, running faster so it is transfared faster from the server to client. In this way, you can compress javascript code from OpenLayers from more then megabyte to several kilobytes.
- 6. Raster × Vectors Size× Rendering speed × Transport time × Flexibility × Amount of informations Raster × Vectors Size × Rendering speed × Transport time × Flexibility × Amount of informations 2015-07-16 Push it through the wire Nowadays JavaScript interpreters and renderers do not have any problem with rendering of really big amounts of data. Raster images are usually coming in compressed form of JPEGs or PNG files and to render them, no rasterization is needed any more. Also amount of informations in raster file can be significantly lower, then it is in vector data. We can say, that amount of information in raster file is limited to its resolution and number of colors, which can be interpreted by the user, amount of informations in vector file is given by number of vertexes describing the phenomenon and number of attributes per feature. You also can join more tables together and add another level of complexity to the data.
- 7. 2015-07-16 Push it throughthe wire From certain point from file size perspective, raster files are smaller then vector files. It would make hight sense to transform your vector data on the server and send them as rasters to the client. Why would you need vector data on client side anyway?
- 8. Why vector inthe browser anyway? better user interaction dynamic aspect of the data editing in the browser ... Why vector in the browser anyway? better user interaction dynamic aspect of the data editing in the browser ... 2015-07-16 Push it through the wire • You want to enable user interaction with the data, add dynamic aspect of the data NEXT • you want to enable the editing NEXT • ...
- 9. Why vector inthe browser anyway? better user interaction dynamic aspect of the data editing in the browser ... Why vector in the browser anyway? better user interaction dynamic aspect of the data editing in the browser ... 2015-07-16 Push it through the wire • You want to enable user interaction with the data, add dynamic aspect of the data NEXT • you want to enable the editing NEXT • ...
- 10. Why vector inthe browser anyway? better user interaction dynamic aspect of the data editing in the browser ... Why vector in the browser anyway? better user interaction dynamic aspect of the data editing in the browser ... 2015-07-16 Push it through the wire • You want to enable user interaction with the data, add dynamic aspect of the data NEXT • you want to enable the editing NEXT • ...
- 11. Why vector inthe browser anyway? better user interaction dynamic aspect of the data editing in the browser ... Why vector in the browser anyway? better user interaction dynamic aspect of the data editing in the browser ... 2015-07-16 Push it through the wire but the main reason is
- 12. Vectors are cool! Vectorsare cool! 2015-07-16 Push it through the wire vector data are just cool, gives you the feeling of power, to have them on the client side
- 13. Compress the communicationbetween server and client Proper data format Consider lossy compression of the data on the server Transfer only the data, you need to transfer Compress the communication between server and client Proper data format Consider lossy compression of the data on the server Transfer only the data, you need to transfer 2015-07-16 Push it through the wire Those are suggested steps, when dealing with large amount of vector data: • Compress the communication between server and client NEXT • Choose the file format carefully NEXT • Use lossy compression of the vector data NEXT • Do transfer only the data, user needs to see
- 14. Compress the communicationbetween server and client Proper data format Consider lossy compression of the data on the server Transfer only the data, you need to transfer Compress the communication between server and client Proper data format Consider lossy compression of the data on the server Transfer only the data, you need to transfer 2015-07-16 Push it through the wire Those are suggested steps, when dealing with large amount of vector data: • Compress the communication between server and client NEXT • Choose the file format carefully NEXT • Use lossy compression of the vector data NEXT • Do transfer only the data, user needs to see
- 15. Compress the communicationbetween server and client Proper data format Consider lossy compression of the data on the server Transfer only the data, you need to transfer Compress the communication between server and client Proper data format Consider lossy compression of the data on the server Transfer only the data, you need to transfer 2015-07-16 Push it through the wire Those are suggested steps, when dealing with large amount of vector data: • Compress the communication between server and client NEXT • Choose the file format carefully NEXT • Use lossy compression of the vector data NEXT • Do transfer only the data, user needs to see
- 16. Compress the communicationbetween server and client Proper data format Consider lossy compression of the data on the server Transfer only the data, you need to transfer Compress the communication between server and client Proper data format Consider lossy compression of the data on the server Transfer only the data, you need to transfer 2015-07-16 Push it through the wire Those are suggested steps, when dealing with large amount of vector data: • Compress the communication between server and client NEXT • Choose the file format carefully NEXT • Use lossy compression of the vector data NEXT • Do transfer only the data, user needs to see
- 17. Compress the server-clientcommunication mod gzip 80MB GeoJSON → 7MB Compress the server-client communication mod gzip 80MB GeoJSON → 7MB 2015-07-16 Push it through the wire Setting zip compression on the server seems to be the most primitive thing, but you would be surprised, how often do we forget to switch this option on on the Apache server (or any other). In this way, you can compress about 80MB big GeoJSON file to nearly 7MB, what is about 10× smaller.
- 18. File format1 GML vs.GeoJSON GML ≈ 40MB GeoJSON ≈ 80MB File size vs. processing time 1 50 000 polygon features, OSM dataset File format1 GML vs. GeoJSON GML ≈ 40MB GeoJSON ≈ 80MB File size vs. processing time 1 50 000 polygon features, OSM dataset 2015-07-16 Push it through the wire Choosing proper file format could seem as easy option. On our test data - file of about 50 000 of polygon features, which file format do you think will be bigger - GML or GeoJSON ? Yes, it’s GeoJSON - while GeoJSON needs about 80MB of space, GML needed only 40MB. I do not want to say, generally speaking, GML is less verbose then GeoJSON, I want to say, choosing the right format can be tricky. Where GeoJSON beats GML is the speed of data processing on the client in any case.
- 19. 2015-07-16 Push it throughthe wire If we are displaying compressed JPEG images to users saying ”it’s ok, you see the data nearly as they are in reality, why are we note doing the same with vectors?
- 20. https://www.jasondavies.com/simplify/ https://www.jasondavies.com/simplify/ 2015-07-16 Push it throughthe wire Who needs 1000 vertexes per line, when at certain zoom level only 3 would be enough? Vector mapping libraries like OpenLayers or Leaflet are doing heavy pre-processing of the data, before they are send to canvas, so that only reasonable amount of vertexes is displayed. Is there a way, how to send pre-processed data from the server?
- 21. TopoJSON Data size progress GeoJSON80MB → TopoJSON 3.3MB → zipped 500KB 2 http://github.com/mbostock/topojson 2 50 000 polygon features, OSM dataset TopoJSON Data size progress GeoJSON 80MB → TopoJSON 3.3MB → zipped 500KB 2 http://github.com/mbostock/topojson 2 50 000 polygon features, OSM dataset 2015-07-16 Push it through the wire TopoJSON The answer is TopoJSON format. While the original GeoJSON file had 80MB, new created TopoJSON file has about 3.3MB. If you zip it, it then has about 500kb. How is this possible?
- 22. TopoJSON http://github.com/mbostock/topojson Extension of GeoJSON Introducesnew type ”topology” Coordinates stored in arcs array Arcs are similar to line strings. More arcs form together geometry Lat & Long → relative coordinates, ∆ values TopoJSON http://github.com/mbostock/topojson Extension of GeoJSON Introduces new type ”topology” Coordinates stored in arcs array Arcs are similar to line strings. More arcs form together geometry Lat & Long → relative coordinates, ∆ values 2015-07-16 Push it through the wire TopoJSON TopoJSON format was introduced by Mike Bostock who is author of many useful tools, which are addressing some of the issues, web developers are facing, when dealing with bigger amounts of data. You’ve probably heard about D3.js library for drawing graphs, but there is more. TopoJSON is an extension of GeoJSON. TopoJSON introduces a new type, ”Topology”, that contains GeoJSON objects. A topology has an objects map which indexes geometry objects by name. These are standard GeoJSON objects, such as polygons, multi-polygons and geometry collections. However, the coordinates for these geometries are stored in the topology’s arcs array, rather than on each object separately. An arc is a sequence of points, similar to a line string; the arcs are stitched together to form the geometry. Lastly, the topology has a transform which specifies how to convert delta-encoded integer coordinates to their native values (such as
- 23. Does not supportbig integer – problem with big IDs 3 3 https://github.com/mbostock/topojson/wiki Does not support big integer – problem with big IDs 3 3 https://github.com/mbostock/topojson/wiki 2015-07-16 Push it through the wire TopoJSON uses among others lossy compression, by simplifying the arcs to certain scale. If you want to go more into details, there is nice presentation of Mike Bostock explaining details of the algorithm at https://github.com/mbostock/topojson/wiki One issue I see with TopoJSON is, it does not support big integer attributes. I wanted to apply it for czech cadaster data, which are using big integer as feature id, unluckily, this was not possible.
- 24. Transfer only thedata, you need to transfer Transfer only the data, you need to transfer 2015-07-16 Push it through the wire Always have in mind: you do not want to transfer all the data. You just want the user to make believe, he sees all the data.
- 25. 2015-07-16 Push it throughthe wire We are all used to transfer only needed amount of raster data, as needed for current view - either big images using WMS or smaller tiles, using one of the proprietary or open standards, like WMTS. Why are we not doing this the same way with vectors?
- 26. Tiled vectors http://openlayers.org/en/v3.7.0/examples/ tile-vector.html → http://jsfiddle.net/og7m21t7/ Tiledvectors http://openlayers.org/en/v3.7.0/examples/ tile-vector.html → http://jsfiddle.net/og7m21t7/ 2015-07-16 Push it through the wire Tiled vectors There are probably two reasons for that: first, vector data were always somehow smaller the raters, but this is not necessary the fact any more. Second: it’s much more complicated, especially if you consider all the edge conditions, like polygon data, with holes in it, spread over 4 tiles.
- 27. https://github.com/mapbox/vector-tile-spec https://github.com/mapbox/vector-tile-spec 2015-07-16 Push it throughthe wire However it’s possible. MapBox has defined it’s own format for so called tiled vectors https://github.com/mapbox/vector-tile-spec it was published as open specification, but as far as I know, it is not implemented in any other software instead of MapBox so far (correct me, if I’m wrong)
- 28. TileStache http://tilestache.org TileStache http://tilestache.org 2015-07-16 Push it throughthe wire TileStache Other option is to create vector cache using TileStache server. The tiles can be either cut at edges - one polygon is split into 2 or 4, if it is not fitting into one. Or you can use e.g. centroids who are ”within”the pre-defined tile and assign only those features to the ”tile”. TileStache can produce geojson tiles as well as topojson tiles.
- 29. http://jsfiddle.net/og7m21t7/ http://jsfiddle.net/og7m21t7/ 2015-07-16 Push it throughthe wire With tiled polygon data, you are facing another problem: polygons are usually spread between more tiles. You have to either resign on displaying polygon boundaries, what is possible e.g. for big water bodies. But if you are dealing with e.g. parcel data, different approach has to be taken. You can read e.g. only polygons, which centroid fit’s into given tile - but then, long streets are not seen. There is a link under images, which contains OpenLayers tiled vector example loaded into jsfiddle environment, where you can play with style settings and see the problem. I hope to show it to you at the end of the session.
- 30. https://goo.gl/oSutyN https://goo.gl/oSutyN 2015-07-16 Push it throughthe wire Here is a screenshot from video, which you can reach on the link below, showing the possibility of vector tiles, which do not contain croped vector features. In this approach only the features which centroids fit into the tile are downloaded.
- 31. 2015-07-16 Push it throughthe wire As result of all the effort, user will see similar progress, has he sees already with vector data - they will be loaded in tiles. To conclude:
- 32. Conclusion Do I needto use all this compression, file format, tiling always I want to display single vector point / GPX log file? Conclusion Do I need to use all this compression, file format, tiling always I want to display single vector point / GPX log file? 2015-07-16 Push it through the wire Conclusion Do I need to use all this compression, file format, tiling always I want to display single vector point or GPX log file?
- 33. Conclusion NO! Conclusion NO! 2015-07-16 Push it throughthe wire Conclusion Of course not
- 34. Conclusion Use it whenyou needed, when things seems slow. Conclusion Use it when you needed, when things seems slow. 2015-07-16 Push it through the wire Conclusion Those approaches are for the situations, your application loading seems to be too slow and you are trying to define the bottlenecks - either it’s just the download speed or it seems to be rendering time as well.
- 35. Conclusion With tiling, Iget faster vectors, what do I loose? Conclusion With tiling, I get faster vectors, what do I loose? 2015-07-16 Push it through the wire Conclusion Caching and tiling of vector data gives you more speed
- 36. Conclusion Topicality and simplicity– you can not have up-to-date pre-cached data. dataspeed × datasize × datatopicality Conclusion Topicality and simplicity – you can not have up-to-date pre-cached data. dataspeed × datasize × datatopicality 2015-07-16 Push it through the wire Conclusion But you loose simplicity and topicality of the data. You can not have fast, up-to-date and big data at the same time.
- 37.