Event Visualization with OpenStreetMap Data, Interdisciplinary Project
Upcoming SlideShare
Loading in...5
×
 

Event Visualization with OpenStreetMap Data, Interdisciplinary Project

on

  • 925 views

 

Statistics

Views

Total Views
925
Views on SlideShare
925
Embed Views
0

Actions

Likes
0
Downloads
17
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Event Visualization with OpenStreetMap Data, Interdisciplinary Project Event Visualization with OpenStreetMap Data, Interdisciplinary Project Document Transcript

  • Event Visualization with OpenStreetMap Data Interdisciplinary Project Bibek Shrestha (bibek.shrestha@tum.de) Supervised by Khatereh Polous (polous@bv.tu-muenchen.de) Contents Abstract 3 Introduction 3 Open Street Map Database Structure . . . . . . . . . . . . . . . . . . . . . . . . . 3 Open Event Map Data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Open Event Map Visualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Open Street Map Technology Stack 5 Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Operating System: Archlinux . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 The Rails Port (TRP) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Database Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Rendering Software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Events, Event Tags and EventEditor JOSM Plugin 8 Event Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 EventEditor JOSM Plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 OpenEventMap Event Visualization 9 Cron, OSM and OpenEventMap table formats . . . . . . . . . . . . . . . . . . . . OpenEventMap web application 9 . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Search parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 Conclusion 12 1
  • Future Improvements 14 Improving Search API . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Data collection process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Appendix A : The Rails Port (TRP) Installation Notes 16 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 Setting up Linux Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16 The Database . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Configuring Postgres . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 The RailsPort . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 Installation of tile server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Setting up /etc/hosts file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Setting up Mapnik and mod_tile . . . . . . . . . . . . . . . . . . . . . . . . . 20 Loading renderd during boot . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Synchronizing changes from ApiDB to MapnikDb . . . . . . . . . . . . . . . . 23 Appendix B : EventsEditor JOSM Plugin Installation Notes 26 Appendix C : OpenEventMap Installation Notes 29 2
  • Abstract OpenStreetMap (OSM) is a collaborative volunteer based mapping platform. The large amount of data is stored in a robust key (tags) value pair database. The database can store any sets of key values in the form of strings. However entered data is monitored and moderated by dedicated volunteers to prevent garbage input and vandalism. Current set of tags used by OSM are only spatial tags. In this project, the OSM database and technology infrastructure is copied and extended to explore the idea of temporal data collection. Events information are collected and entered into the database using special tags. The process is simplified using a self made plugin on top of JOSM editor. A web based platform has been built named ‘OpenEventMap’ that extracts the data from key-value database of OSM, gives structure to the data by converting into another format for easy searching and finally renders it for visualization purposes. The details of event based tag metadata, system architecture of the system and technical details of entire system has been presented in this report. Introduction Our history is written through myriads of events happening all over the world. These events bring about wide array of changes from political, geographical, social, architecture to name a few. It is important to document these events precisely to facilitate further study and research. Currently, there is no platform that records events methodologically which includes event description along with associated spatial and temporal information. A knowledge repository that captures past and future events needs to be made. Of different ways of building a knowledge repository, the Wikipedia model has come out as a voluntary, collaborative, transparent and accurate way of building large scale knowledge repository. It has been replicated into creating maps in the form of Open Street Map (OSM). OSM has thousands of volunteers contributing location based data to build an online platform for community based Maps. Anyone with a GPS device and an internet connection can easily start building maps of his locality. It is based entirely on voluntary effort and this effort has helped build a very high quality online global map. Open Street Map Database Structure OSM data structure is very simple. A piece of data (row in the database) consists of a pair of (key, value). Using this, for any point in the database, multiple data can be associated as the point’s attributes. When any kind of data is entered into the system, no checks are performed to restrict the input by the system. So a user can enter any kind of data into the OSM database. This system allows for all different kinds of data to be entered to accomodate different terminologies used in different parts of the world. However it also opens the door for vandalism and wrong input. In order to prevent users from entering or misusing the platform, dedicated volunteers check the consistency of the database from time to time and either fix the database or punish the user by deleting a user account or both. 3
  • Figure 1: Example of key, value data for a node in OpenStreetMap The general convention in OSM community is to store spatial data. There has been discussions regarding storage of event related information but it has been discouraged. Open Event Map Data We realize that although temporal data cannot go into the main OSM database for now, we can copy the existing data and the software to create a replica environment for our research purposes. By setting up a server that behaves exactly like the main OSM server, we can use wide set of tools for data collection and modification into our platform. OSM has predefined set of tags to describe spatial data. Our challenge was to come up with a set of tags that described temporal attributes of given points. This set of tags should easily fit into existing system, not just for describing attributes but also for other purpose like searching. The set of tags are described in the following sections with their advantages and their shortcomings. Most of our data is entered into the system using a tool called JOSM. A JOSM plugin has been created that provides a Graphical User Interface (GUI) to easily enter the event information by point and click. 4
  • Open Event Map Visualization Events data are associated with existing nodes, ways or relations in the system. Nodes, Ways and Relations are the spatial data that represent buildings, trees, pools, roads, compounds, parks or any other physical elements that exist. By associating events to these, we associate temporal data to spatial data. For visualization purpose, we extract stored temporal data and it’s spatial coordinates and display it on a web based platform by placing colored markers on top of a map. User can use the web platform to search for events based on different parameters like event name, category, event dates and see color coded results based on different category. Open Street Map Technology Stack OpenStreetMap is powerd by a combination of Open Source softwares that can be downloaded freely from the internet. The detail guide to setting up Open Street Map on a server is available at The Rails Port link. A brief description of different components including a diagram follows. Server Operating system: Archlinux RAM: 12 GB Harddisk Space: 500 GB Database: Postgres 9.2.4 Webserver: Apache 2.2.24 Operating System: Archlinux Archlinux has both advantages as well as disadvantages. It is a rolling distribution, so the software packages are always up to date. The disadvantage is that constantly changing packages might break required version of libraries. However, the main application The Rails Port runs on a virtualenv environment with specific version of ruby and rails gems. Apache webserver is for serving three different applications. Passenger Phusion is used to serve The Rails Port based on Rails and Open Event Map based on python/django. mod_tile is used to render and serve tiles. Postgres with PostGIS extension is used as our main database server. The Rails Port (TRP) TRP is the codebase for entire webfrontend of Open Street Map. This includes: 1. Map frontend for visualization. 5
  • Figure 2: Components Diagram 6
  • 2. User login and map editing using Potlatch 2 editor. 3. API version 6 accessible via /api/ URL. The installation of TRP can be accessed through http://osm.openeventmap.tum.de. Similarly the API capabilities can be viewed from http://osm.openeventmap.tum.de/api/capabilities. Source code for TRP can be downloaded easily from GitHub. It is a standard rails application that connects to the Postgres database. A detail guide on installation of The Rails Port(TRP) is given in the Appendix A. Database Setup The complete OpenStreetMap database is available for download from OSM Planet. It can be copied, distributed or adapted to different forms as long as the Open Data Commons Open Database License is agreed upon by the user. However, the complete database is 30GB large in size in a compressed format. It is estimated to be more than 100GB uncompressed. Such a large volume of data is not required in our case as we deal mostly in the region around Bavaria, Germany. Geofabrik provides map database files of smaller regions by dividing the main OSM database into multiple regions. The Oberbayern mapfile is used in our case, which is 96MB when compressed. We only work with 0.32% of total database size. The data is imported into our Postgres database and connected with TRP. Rendering Software We have three subdomains to serve tiles in our website. 1. a.tile.openeventmap.tum.de/osm_tiles/x/y/z.png 2. b.tile.openeventmap.tum.de/osm_tiles/x/y/z.png 3. c.tile.openeventmap.tum.de/osm_tiles/x/y/z.png Where x and y are square coordinates and z is the zoom level of the tile. Tiles are served using mod_tile extension. It maps every request in above format to renderd daemon. The renderd daemon in turn checks if an existing tile is available in the cache. If found, it then checks if the tile needs to be regenerated due to changes in OSM data. As a result, it returns either the tile in the cache or generates a new tile, saves the tile in the cache and serves it through apache. renderd uses Mapnik as a tile server. Mapnik connects to the database, extract information for tile generation and then applies color styles on top of the data and generates the tiles. Mapnik can use different stylesheet to generate different types of maps. In our case, the default OSM stylesheet is used. 7
  • Events, Event Tags and EventEditor JOSM Plugin In OSM, we have three basic types of elements: node, way and relation. A node is a single point characterised by a unique pair of latitude and longitude values. A node can represent any physical object that exist in the exact coordinate. A way is a set of nodes grouped together. A way can be used to represent a bigger larger area or a continuous path. Similarly a relation is a logical grouping of multiple ways and nodes to represent an advanced concept. Besides having spatial coordinate data, nodes, ways and relations have other information including name, classification type and other varieties of properties in the form of tags. Event Tags An event is primarily associated with one of the nodes, ways or relations. This is done by associating special event tags to the OSM elements which is described in detail below. The primary tag to mark any element as event, we need to set the following tag: event: yes If associated with nodes, the icons have an added “E” on their right side as following: Figure 3: A node with event information associated showing icon with “E”. We further identify other event related tags to associate more information to the OSM elements and they are: 1. name: Name. 2. category: A list of primary categories user can choose from. 3. sub_category: Each primary category has a list of further sub categories for refined classification. 4. organization: Main organization related with the event. 5. start_date: Actual day when the event occurs. 6. end_date: In case of a single day event, it is same as start_date but different if it is a longer event. 7. url: A website that explains more about the event. 8. numparticipants: An estimate of the number of participants in the event. 9. howoften: Event repeat frequency. 10. comment: Any extra note a user can associate with the event. 11. related_items: A comma separated list of other OSM elements related with this event. One OSM element can contain more than one event. To make it possible, we use namespacing in tags and add index to each event. Consider a way with two associated events; its tags will be as following. 8
  • name = kaufingerstraße highway = pedestrian bicycle = no cycleway = no ... event = yes event:0:name = Christkindlemarkt event:0:category = social event:0:sub_category = Fairs / Festivals ... event:1:name = Munich Carnival event:1:category = social event:1:sub_category = Fairs / Festivals event:1:start_date = 07/01/2013 event:1:end_date = 12/02/2013 event:1:howoften = Yearly ... The character “:” is used for namespacing. The format of any event tag is as follows event[namespacer]event_index[namespacer]event_attribute = value 1. event_index: Starts with 0 and for each event, it increses by one. 2. event_attribute: Can be one of the defined tags in the list above. EventEditor JOSM Plugin JOSM is an offline Java based OSM editor which is quite powerful and fiture rich. It supports extra plugins that can be downloaded from the internet for added functionality. To add event information, an extra plugin called event_editor has been created. It was created completely in Java. With this users do not have to remember the special format mentioned above to enter data, instead they can use a simple GUI. Installation of the plugin is explained in more detail in Appendix B. OpenEventMap Event Visualization All the data collected into the locally deployed OSM database is converted into an searchable database format. The process is shown in figure 2 from the green box named CRON. Cron, OSM and OpenEventMap table formats cron: is a time-based job scheduler in Unix-like computer operating systems. 9
  • Figure 4: EventEditor Starting Screenshot. 1) Option to mark current selected OSM Entity to an event. 2) Button to create a new Event for the event entity. 3) GUI to enter more detail tags for the event. 10
  • The cron configuration runs a special script in the server every minute. This script generates osmchange files in .osc format which contains addition, modification or deletion of any OSM Element within the last minute. Each .osc file is processed by OpenEventMap script and inserted into a new table called search_event. During this process, multiple tag based key - value pairs related to a single event are converted into a single row in the new table format. Figure 5: Comparison between OpenStreetMap and OpenEventMap table structure In OSM current_node_tags table, a single event data is stored across multiple rows as belows: node_id k v 12345 event yes 12345 event:0:name Oktoberfest 12345 event:0:category social 12345 event:1:name Frühlingsfest 12345 event:1:category social The two events are converted into two single rows in search_event table as following: id event_type type_id number name 11 category subcategory ...
  • 1 node 12345 0 Oktoberfest social ... 2 node 12345 1 Frühlingsfest social ... The search_event table format makes it easier to do complex queries searching more than one field at any time. For example the following query is now possible: SELECT * FROM search_event WHERE name LIKE %val% AND category=’social’ AND ... The query above would be complicated to execute on current_node_tags table. OpenEventMap web application The web application allows any user to search event database. Search parameters There are four parameters to filter the search results by: event name does a partial matching with existing event names, category parameter searches for exact match in category and subcategory fields, start date and end date creates a date range and returns all the events that fall under given range. Date filter also takes care of repeating events. For example, an event that occurred in March 5, 2012 and repeats yearly will be found when searching for events between March 1, 2013 and March 10, 2013. If search is performed with non of the parameters entered, it returns all matching events within given map region. Search results are color coded. Most of the popular event categories are assigned different colors which are shown on the textual search results section as well as a colorful pointer on the map. As the pointer is clicked, all events belonging to the place is shown in a popup. Event data can be exported along with OSM elements like node, ways and relations in OSM format using the Export link in the Search Results Section. It redirects the user to the OSM webpage. There is also a feature to first search any location by name. This is achieved by OSM’s Nominatim api. For given location name, a search is performed around Bayern area and the results are listed. When clicked on the result, the map gets zoomed in. Conclusion All parts of the system developed have been on top of opensource technologies and open data. Open technologies provide easily accessible platforms. In this case, OpenStreetMap 12
  • Figure 6: Web application screenshot. 1) Choice between Nominatim search and Event search. 2) Fields to enter search parameters. 3) Search results grouped into categories. 4) Visualization of OSM elements and associated events. 13
  • ecosystem provided us with ready made data, software suite from backend to frontend like TheRailsPort, mod_tile, Mapnik, JOSM etc. Building complete infrastructure by self would take extremely large amount of time. We are also grateful to the volunteers spread all over the internet for collecting the data, submitting it online in OSM and allowing other individuals to use them. The philosophy motivates further more people to contribute and innovate. In our case, we were able to imagine, plan, implement and finally visualize chronological data like events on top of OpenStreetMap. Not everything was easy. Documentation of most of the components were spread all over OSM wiki. Functionalities of the software components was unclear and they had to be understood and pieced together. In conclusion, the following tasks have been successfully carried out: 1. The mapserver has been installed as requested. 2. A special purpose extension EventEditor has been created to help users enter events data. 3. A webapp for visualizing the user contributed data has been created and setup. Our original goals for the project have been successfully fulfilled. Future Improvements We realize that the current work is the base for further improvements. This is a beginning for integrating temporal data into OSM. Looking ahead, there are more that can be done on top of current work and these have been listed below. Improving Search API OpenEventMap is powered by a search API that takes name, category and startdate enddate as search parameters. The results are returned as JOSN objects. First improvement can be increasing the search parameters to facilitate advanced search using all the attributes we have collected. Second improvement on search could be combining event names with locations. With this, a user should be able to search for “Oktoberfest in Munich” and see the result. The earch term contains both event name and location information. Data collection process Event data entry is a painful process right now. The normal process is to identify event, then search the related items on OSM and then attach event attributes to those items. When large and existing datasets have to be integrated, this is not trivial. 14
  • The solution could be a probabilistic model that identifies location based on techniques like geocode to identify OSM entities. An extra layer of API can be made to quickly add and edit one or more events in a single entity. With this, softwares could be developed that would automatically save data from existing data sources to OSM. 15
  • Appendix A : The Rails Port (TRP) Installation Notes Most of the installation instructions are explained in detail here. The following instructions have been explained for Archlinux with necessary modifications. Introduction Inorder to create our local copy of the openstreetmap database and visualize it like it is in main OSM website, we setup a linux server and setup the following components on it. 1. 2. 3. 4. Postgre Database for our data renderd, a tool that generates tiles from the database MapServer, that serves generated tiles. RailsPort, the main application that powers OSM website with features of user login, view, zoom, export tile and edit the data using potlatch editor. Installation instruction are as follows: Setting up Linux Server Packages Installed: 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. postgis mapnik apache boost python2 ttf-dejavu subversion libxslt imagemagick unzip yaourt (aur) osm2pgsql-svn (aur) osmosis (aur) ruby1.9 We need crontab to run during startup, so run the following command to enable cronie service. $ systemctl enable cronie $ systemctl start cronie 16
  • The Database The main OpenStreetMap database is run on PostgreSQL database. PostgreSQL has numerous features that allow geometry types. However OpenStreetMap does not used those and have its own data primitives. PostGIS is an extension to PostgreSQL database that provides spatial objects allowing storage and query of information about location and mapping. The main database does not use PostGIS features but is used by mapnik, the tile rendering software. There are two main databases used in our approach: 1. ApiDB - This backs rails port, allows a read-write api to edit the database, thus called ApiDB. JOSM and other editors make modifications in this database using the api. The data is imported from planet dump files, in our case data inside oberbayernlatest.osm.bz2 was imported using the tool Osmosis. 2. Mapnik Db Format - This is used by mapnik to generate the tiles. Data is imported from oberbayern-latest.osm.bz2 using osm2pgsql tool. The database for ApiDB schema is named as myosmdb and database for mapnik schema is named as mymapnikdb. Configuring Postgres The step by step process to install PostgreSQL and PostGIS extension is explained below. Postgres requires initialization. We run the following commands as root on the server. $ $ $ $ mkdir /var/lib/postgres/data su - postgres -c "initdb --locale en_US.UTF-8 -D ’/var/lib/postgres/data’" systemctl enable postgresql.service systemctl start postgresql.service A user is created in the server and a database is created under this user. The following commands were ran under postgres username $ sudo -u postgres createuser myosmuser -s -P $ sudo -u postgres createdb -E UTF8 -O myosmuser myosmdb As seen on the commands above, the username/password combination is myosmuser/myosmuser and the database name is myosmdb. The RailsPort OSM front end website is based on the Ruby on Rails framework. The source code for it can be downloaded from git clone https://github.com/openstreetmap/openstreetmap-website.git 17
  • $ sudo mkdir /opt/myosm $ sudo chown bibek /opt/myosm Change bibek to the username you want. $ cd /opt/myosm $ git clone https://github.com/openstreetmap/openstreetmap-website.git Copy proper application.yml and database.yml. $ cd /opt/myosm $ cp config/example.application.yml config/application.yml $ cp config/example.database.yml config/database.yml Change the development section of config/database.yml file development: adapter: postgresql database: myosmdb username: myosmuser password: myosmuser host: localhost encoding: utf8 Quadtile functions need to be installed on the server. For this $ cd /opt/myosm/openstreetmap-website/db/functions $ make libpgosm.so And the following commands need to be executed in postgresql $ psql -U myosmuser -d myosmdb -c "CREATE EXTENSION btree_gist;" $ psql -Umyosmuser -d myosmdb -c "CREATE FUNCTION maptile_for_point(int8, int8, int4) RETURNS int4 AS ’/opt/myosm/openstreetmap-website/db/functions/libpgosm’, ’maptile_for_point’ LANGUAGE C STRICT;" $ psql -Umyosmuser -d myosmdb -c "CREATE FUNCTION tile_for_point(int4, int4) RETURNS int8 AS ’/opt/myosm/openstreetmap-website/db/functions/libpgosm’, ’tile_for_point’ LANGUAGE C STRICT;" $ psql -Umyosmuser -d myosmdb -c "CREATE FUNCTION xid_to_int4(xid) RETURNS int4 AS ’/opt/myosm/openstreetmap-website/db/functions/libpgosm’, ’xid_to_int4’ LANGUAGE C IMMUTABLE STRICT;" Also ruby and rails are installed using ruby version manager (rvm) 18
  • $ $ $ $ curl -L https://get.rvm.io | bash -s stable --autolibs=enabled --ruby=1.9.3 gem install rails cd /opt/myosm/openstreetmap-website bundle install Setup necessary database tables in postgresql. We use the production configuration here. $ rake db:migrate To make sure everything is setup properly, run $ rake test We can now populate the date for Munich. $ cd /opt/myosm $ wget http://download.geofabrik.de/europe/germany/bayern/oberbayern-latest.osm.bz2 $ osmosis --read-xml-0.6 file="oberbayern-latest.osm.bz2" --write-apidb-0.6 populateCurrentTables=yes host="localhost" database="myosmdb" user="myosmuser" password="myosmuser" validateSchemaVersion=no In the psql terminal the following was executed $ echo " select setval(’acls_id_seq’, (select max(id) from acls)); select setval(’changesets_id_seq’, (select max(id) from changesets)); select setval(’client_applicaations_id_seq’, (select max(id) from client_applications)); select setval(’countries_id_seq’, (select max(id) from countries)); select setval(’current_nodes_id_seq’, (select max(id) from current_nodes)); select setval(’current_relations_id_seq’, (select max(id) from current_relations)); select setval(’current_ways_id_seq’, (select max(id) from current_ways)); select setval(’diary_comments_id_seq’, (select max(id) from diary_comments)); select setval(’diary_entries_id_seq’, (select max(id) from diary_entries)); select setval(’friends_id_seq’, (select max(id) from friends)); select setval(’gpx_file_tags_id_seq’, (select max(id) from gpx_file_tags)); select setval(’gpx_files_id_seq’, (select max(id) from gpx_files)); select setval(’messages_id_seq’, (select max(id) from messages)); select setval(’oauth_nonces_id_seq’, (select max(id) from oauth_nonces)); select setval(’oauth_tokens_id_seq’, (select max(id) from oauth_tokens)); select setval(’redactions_id_seq’, (select max(id) from redactions)); select setval(’user_blocks_id_seq’, (select max(id) from user_blocks)); select setval(’user_roles_id_seq’, (select max(id) from user_roles)); select setval(’user_tokens_id_seq’, (select max(id) from user_tokens)); select setval(’users_id_seq’, (select max(id) from users)); " | psql -d myosmdb -U myosmuser -w -e -f - 19
  • Installation of tile server A tile server serves the images that are displayed as the map. Each tile is 256pixels by 256pixels in size. When OSM Website is opened, the large map image is formed by place these smaller tiles together. A tile has three properties • z: The zoom level. 0 being the least zoomed and 18 being the most zoomed. • x,y: On each zoom level, x and y are the coordinates of a given tile. A sample Url from OSM Website looks like http://a.tile.openstreetmap.org/0/0/0.png which shows the whole world’s map. We will set up our own tile server which can be accessed from http://a.tile.openeventmap. tum.de/osm_tiles/z/x/y.png or http://b.tile.openeventmap.tum.de/osm_tiles/z/x/y.png or http://c.tile.openeventmap.tum.de/osm_tiles/z/x/y.png. Setting up /etc/hosts file As root, add the following line at the bottom of the /etc/hosts file. 127.0.0.1 a.tile.openeventmap.tum.de b.tile.openeventmap.tum.de c.tile.openeventmap.tum.de Once the tiles load, the configuration for our rails port needs to be pointed to our new tile servers. Edit /opt/myosm/openstreetmap-website/vendor/assets/leaflet/leaflet.osm.js and replace http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png with http://{s}.tile.openeventmap.tum.de/osm_tiles/{z}/{x}/{y}.png Setting up Mapnik and mod_tile Generating and serving of tiles includes three pieces of software to work together. First, apache and mod_tile needs to be configured to receive requests in http://{a,b,c}.tile.openeventmap.tum.de/osm_tiles/z/x/y.png format. Mod_Tile forwards the request to renderd and renderd checks if a valid tile is in it’s cache. If found, it serves the tile and if not found it asks Mapnik to generate the tile which is then served. We will look into how to install these three pieces in the following section. More detail information for installing mod_tile can be found in Mod_tile Here are the commands executed. 20
  • $ $ $ $ $ $ $ $ $ cd /opt/myosm git clone https://github.com/openstreetmap/mod_tile cd mod_tile ./autogen.sh ./configure --with-apxs=/usr/sbin/apxs make sudo make install sudo make install-mod_tile sudo ldconfig This will create the file /usr/lib/httpd/modules/mod_tile.so $ $ $ $ $ $ $ $ $ cd /opt/myosm git clone git://github.com/openstreetmap/mapnik-stylesheets.git cd mapnik-stylesheets ./get-coastlines.sh cd world_boundaries/ ln -s ne_110m_admin_0_boundary_lines_land.shp 110m_admin_0_boundary_lines_land.shp ln -s ne_110m_admin_0_boundary_lines_land.dbf 110m_admin_0_boundary_lines_land.dbf cd .. /generate_xml.py --host localhost --dbname mymapnikdb --symbols ./symbols --world_boundaries world_boundaries --port 5432 --password myosmuser --user myosmuser This will generate a file osm.xml which we shall use later for renderd. Mapnik requires a PostGIS database. PostGIS provides spatial objects for the PostgreSQL database allowing storage and query of information about location and mapping. In the following section we create a database for mapnik, enable PostGIS and then load the data in it. $ $ $ $ $ $ $ mkdir /opt/myosm/postgis cd /opt/myosm/postgis createdb -U postgres -E UTF8 -O myosmuser mymapnikdb psql -U postgres -d mymapnikdb -f /usr/share/postgresql/contrib/postgis-2.0/postgis.sql wget https://raw.github.com/openstreetmap/osm2pgsql/master/900913.sql -O 900913.sql psql -U postgres -d mymapnikdb -f ./900913.sql osm2pgsql -U myosmuser -s -d mymapnikdb --slim oberbayern-latest.osm We now configure apache so that mod_tile is configured to a special virtual host. Uncomment the line from by removing the # at the start of the line in the file /etc/httpd/conf/httpd.conf Include conf/extra/httpd-vhosts.conf Then add the following to the bottom of /etc/httpd/conf/extra/httpd-vhosts.conf. Original source can be found here. 21
  • LoadModule tile_module modules/mod_tile.so <VirtualHost *:80> ServerName tile.openeventmap.tum.de ServerAlias a.tile.openeventmap.tum.de b.tile.openeventmap.tum.de c.tile.openeventmap.tum.de DocumentRoot /var/www/html LoadTileConfigFile /etc/renderd.conf ModTileRequestTimeout 3 ModTileMissingRequestTimeout 10 ModTileMaxLoadOld 2 ModTileMaxLoadMissing 5 ModTileRenderdSocketName /var/run/renderd.sock ModTileCacheDurationMax 604800 ModTileCacheDurationDirty 900 ModTileCacheDurationMinimum 10800 ModTileCacheDurationMediumZoom 13 86400 ModTileCacheDurationLowZoom 9 518400 ModTileCacheLastModifiedFactor 0.20 ModTileEnableTileThrottling Off ModTileThrottlingTiles 10000 1 ModTileThrottlingRenders 128 0.2 LogLevel debug </VirtualHost> Create a new file /etc/renderd.conf and add the following. Note this was modified by editing /opt/myosm/mod_tile/renderd.conf. [renderd] socketname=/var/run/renderd.sock num_threads=4 tile_dir=/var/lib/mod_tile stats_file=/var/run/renderd.stats [mapnik] plugins_dir=/usr/lib64/mapnik/input font_dir=/usr/lib64/mapnik/fonts font_dir_recurse=1 [default] URI=/osm_tiles/ XML=/opt/myosm/mapnik-stylesheets/osm.xml HOST=localhost Start apache and renderd 22
  • $ sudo systemctl start httpd.service $ sudo /opt/myosm/mod_tile/renderd To see if tiles are working, one can try loading the url http://a.tile.openeventmap.tum.de/ osm_tiles/0/0/0.png. A successful setup with load the world map like this. Figure 7: 0th level tile Loading renderd during boot Whenever there is a server reboot, renderd daemon does not start automatically. Thus, it needs to be made sure that renderd is running all the time by logging in to the server. Synchronizing changes from ApiDB to MapnikDb Any write modifications to the data happens in ApiDB. These changes need to be migrated to the tiles as well, and as such we setup a synchronization mechanism between ApiDB and MapnikDB. The synchronization involes three parts. 1. Extract changes from ApiDB in regular intervals. 2. Migrate those changes to MapnikDB. 3. Remove the respective tiles from cache so fresh tiles are generated. 23
  • Extract changes from ApiDB in regular intervals Perform the following as root $ sudo -u $USER mkdir -p /opt/myosm/apidb2mapnikdb/replicate $ cd /opt/myosm/apidb2mapnikdb $ sudo -u $USER osmosis --replicate-apidb user=myosmuser database=myosmdb validateSchemaVersion=no --write-replication workingDirectory=replicate This will create couple of files inside the replicate folder. At the time of first execution, it saves the current state of ApiDb in a state.txt file. As the third command is executed repeatedly, osm change files (.osc) files are generated and the contents of state.txt file updated. $ tree replicate/ replicate/ ’-- 000 ’-- 000 ’-- 000.state.txt ’-- replicate.lock ’-- state.txt osmosis --replicate-apidb user=myosmuser database=myosmdb validateSchemaVersion=no --write-replication workingDirectory=replicate tree replicate/ replicate/ ’-- 000 ’-- 000 ’-- 000.state.txt ’-- 001.osc.gz ’-- 001.state.txt ’-- replicate.lock ’-- state.txt As you can see, new files have been added as the command is executed each time. 000/000/001.osc.gz file is our required change file. We take the changes from .osc.gz files and pass it to mymapnik db. The process of migration is done through a minte-wise running cronjob. Save the following as a script in /opt/myosm/apidb2mapnikdb/update.sh osmosis --replicate-apidb user=myosmuser database=myosmdb validateSchemaVersion=no --write-replication workingDirectory=data a=‘grep ’sequenceNumber’ replicate/state.txt | awk -F"=" {’print $2’}‘ a=$(printf %09d $a) filename=replicate/${a:0:3}/${a:3:3}/${a:6:3}.osc.gz echo $filename rm expire.list osm2pgsql -U myosmuser -s -d mymapnikdb --append $filename -e15 -o expire.list cat expire.list | /usr/local/bin/render_expired --min-zoom=10 --touch-from=10 24
  • Also, make the update.sh file executable $ chmod +x /opt/myosm/apidb2mapnikdb/update.sh The script takes the last generated changefile (.osc.gz file) and uses osm2pgsql to update mymapnikdb. The output of the command generates a list of tiles that needs to be updated which is saved in expire.list. The list is then fed to render_expired so old files are deleted. Add the following in the crontab using crontab -l as root * * * * * (cd /opt/myosm/apidb2mapnikdb/ && ./update.sh &>> file.log) 25
  • Appendix B : EventsEditor JOSM Plugin Installation Notes JOSM can be downloaded freely from http://josm.openstreetmap.de/wiki/Download. The recommended way is to download the jnlp file as it updates itself automatically everytime a new version comes out. After http://josm.openstreetmap.de/download/josm.jnlp has been downloaded, double clicking on josm.jnlp would start JOSM editor. There are two parts to the plugin. 1. The event_editor.jar plugin source code. 2. The stylesheet files elemstyles.xml and event_editor.mapcss. event_editor.jar installation instructions are available at JOSM manual plugin installation page. Once the plugin is installed and enabled, the styles need to be installed. elemstyles.xml and event_editor.mapcss can be placed anywhere. After starting JOSM, these styles should be imported as shown in figure 8. Figure 8: How to import style files in JOSM. 1) Go to Map Settings. 2) Click on Map Paint Styles. 3) Click + to import new Stylesheet. 4) Import both elemstyles.xml and event_editor.mapcss and enable them. The styles allow any node with event=yes tag to show a small E marker at the right side. The style sheet can be enabled as show in figure 9. The user interface to add, edit or delete events reated tags are show in figure 10. 26
  • Figure 9: Using event_editor: 1) Either one of mapcss or xml file needs to be enabled. 2) Any node with event=yes has extra E icon at the right. 3) When any OSM element is clicked, user can click on menu: Presets > Event Add/Edit to start working with the plugin. 4) Newly added tags are shown on the section Properties. 27
  • Figure 10: EventEditor Starting Screenshot. 1) Option to mark current selected OSM Entity to an event. 2) Button to create a new Event for the event entity. 3) GUI to enter more detail tags for the event. 28
  • Appendix C : OpenEventMap Installation Notes OpenEventEditor is based on Django 1.4 and python2. It uses pip to take care of necessary dependencies and integrates seamlessly with apache and passenger module. It is recommended that openeventmap is run inside a virtualenv. $ cd /home/bibek/app/virtualenv/ $ virtualenv openeventmapdjango $ source openeventmapdjango/bin/activate Next step, we install all required dependencies for OpenEventMap to run (openeventmapdjango)$ cd /opt/myosm/openeventmap (openeventmapdjango)$ pip install -r requirements.txt The database name, username and password are used exactly the same as in TheRailsPort and edited in the file eventsvisualizer/settings.py. We are serving static files from apache directly, so we collect all files inside the static_dir folder. (openeventmapdjango)$ ./manage.py collectstatic Now comes the part to configure apache and passenger to serve our web application automatically. The setup is achieved simply by a VirtualHost entry in /etc/httpd/conf/extra/httpd-vhosts.conf. <VirtualHost *:80> ServerName www.openeventmap.tum.de Alias /static/ /opt/myosm/openeventmap/static_dir/ DocumentRoot "/opt/myosm/openeventmap" PassengerPython /home/bibek/app/virtualenv/openeventmapdjango/bin/python PassengerAppRoot /opt/myosm/openeventmap <Directory /opt/myosm/openeventmap> Order allow,deny Allow from all </Directory> </VirtualHost> Accessing http://www.openeventmap.tum.de will load our webapplication. 29
  • Figure 11: OpenEventMap Homepage 30