This document describes an interdisciplinary project to visualize events using OpenStreetMap data. It discusses how OpenStreetMap's database structure was copied and extended to collect event information using custom tags. A JOSM plugin was created to simplify data entry. An OpenEventMap web application was built to extract, structure, and visualize the event data on a map. It allows users to search for events by parameters like name, category, and date. The document outlines the OpenStreetMap technology stack, including the Rails Port application and Postgres database, and how it was configured for this project.
This document proposes a project to add reporting capabilities to the Polaris open source project. The project would develop a web interface to visualize anomalies detected in spacecraft telemetry data through interactive graphs. It would also generate PDF reports and allow exporting the graphs. The proposed workflow is to first understand the existing anomaly detector, then build a new "Polaris Detect" module including a React-based web interface showing time-series graphs of parameter values and outliers. The interface would allow selecting parameters and generating/exporting reports from the anomaly analysis. Milestones include implementing the command line system, web interface with interactive charts, and PDF/export functionality.
This document proposes extending the Polaris project to generate analysis reports from spacecraft telemetry data. The proposed additions include:
1. Developing a web interface to visualize anomalies detected in telemetry parameters over time through interactive graphs.
2. Creating a PDF generator to output graph summaries and anomalies.
3. Adding an option to export all graphs in a zip file for further use.
The goal is to help analysts efficiently study anomalies by generating customizable reports from Polaris results. The proposed workflow and timeline lay out developing an anomaly detection module, interactive visualization tools, and report generation capabilities over 3 months.
This 2 page document outlines usage control requirements across two pages. Page A contains content related to usage control specifications while Page B contains additional content on usage control requirements. The document provides information on usage control specifications and requirements across its two pages of content.
This document summarizes a study of CEO succession events among the largest 100 U.S. corporations between 2005-2015. The study analyzed executives who were passed over for the CEO role ("succession losers") and their subsequent careers. It found that 74% of passed over executives left their companies, with 30% eventually becoming CEOs elsewhere. However, companies led by succession losers saw average stock price declines of 13% over 3 years, compared to gains for companies whose CEO selections remained unchanged. The findings suggest that boards generally identify the most qualified CEO candidates, though differences between internal and external hires complicate comparisons.
The INOVA GIS platform is a centralized enterprise GIS system that allows seamless data access for multiple departments within an organization. It uses a three-tier architecture with a database tier, middle tier called the INOVA GIS Server (IGS), and presentation tier. The IGS provides scalability, redundancy, versioning, and other features. Key client tools include TeleCAD-GIS for editing and viewing, a web GIS portal, and the ability to view content in third party tools like Google Earth.
OpenStreetMap (OSM) is a collaborative project to create a free editable map of the world. Users can add and edit map data by uploading GPS traces or extracting information from other sources like aerial imagery. This data is stored in a central database and can be viewed and edited by anyone on a web map or using desktop editors. The map is rendered from the database using open source tools and can be used freely under an open license. Developers contribute by improving the tools, database, rendering system, and applications that make up the OSM project.
This document proposes a project to add reporting capabilities to the Polaris open source project. The project would develop a web interface to visualize anomalies detected in spacecraft telemetry data through interactive graphs. It would also generate PDF reports and allow exporting the graphs. The proposed workflow is to first understand the existing anomaly detector, then build a new "Polaris Detect" module including a React-based web interface showing time-series graphs of parameter values and outliers. The interface would allow selecting parameters and generating/exporting reports from the anomaly analysis. Milestones include implementing the command line system, web interface with interactive charts, and PDF/export functionality.
This document proposes extending the Polaris project to generate analysis reports from spacecraft telemetry data. The proposed additions include:
1. Developing a web interface to visualize anomalies detected in telemetry parameters over time through interactive graphs.
2. Creating a PDF generator to output graph summaries and anomalies.
3. Adding an option to export all graphs in a zip file for further use.
The goal is to help analysts efficiently study anomalies by generating customizable reports from Polaris results. The proposed workflow and timeline lay out developing an anomaly detection module, interactive visualization tools, and report generation capabilities over 3 months.
This 2 page document outlines usage control requirements across two pages. Page A contains content related to usage control specifications while Page B contains additional content on usage control requirements. The document provides information on usage control specifications and requirements across its two pages of content.
This document summarizes a study of CEO succession events among the largest 100 U.S. corporations between 2005-2015. The study analyzed executives who were passed over for the CEO role ("succession losers") and their subsequent careers. It found that 74% of passed over executives left their companies, with 30% eventually becoming CEOs elsewhere. However, companies led by succession losers saw average stock price declines of 13% over 3 years, compared to gains for companies whose CEO selections remained unchanged. The findings suggest that boards generally identify the most qualified CEO candidates, though differences between internal and external hires complicate comparisons.
The INOVA GIS platform is a centralized enterprise GIS system that allows seamless data access for multiple departments within an organization. It uses a three-tier architecture with a database tier, middle tier called the INOVA GIS Server (IGS), and presentation tier. The IGS provides scalability, redundancy, versioning, and other features. Key client tools include TeleCAD-GIS for editing and viewing, a web GIS portal, and the ability to view content in third party tools like Google Earth.
OpenStreetMap (OSM) is a collaborative project to create a free editable map of the world. Users can add and edit map data by uploading GPS traces or extracting information from other sources like aerial imagery. This data is stored in a central database and can be viewed and edited by anyone on a web map or using desktop editors. The map is rendered from the database using open source tools and can be used freely under an open license. Developers contribute by improving the tools, database, rendering system, and applications that make up the OSM project.
The document provides a user guide for StatPlanet and StatTrends data visualization software. Key features include:
- Interactive thematic maps that can display data using choropleth or proportional symbol mapping. Maps can be customized through options like colors, labels, and overlays.
- Data can be visualized through graphs like bar charts, column charts, time series, scatter plots, and bubble charts. Graphs are customizable.
- Data is imported and organized using categories, indicators, and variables. Definitions can be added for indicators. Data can be imported from files, databases, or directly edited.
- Extensive customization options are available for maps, graphs, categories, indicators, data sources, and more.
Scalable Distributed Real-Time Clustering for Big Data StreamsAntonio Severien
This thesis presents a scalable distributed clustering algorithm for streaming big data. The author implemented a real-time distributed clustering algorithm and a classification algorithm using the Scalable Advanced Massive Online Analysis (SAMOA) framework. SAMOA is a platform-independent framework for distributed machine learning on data streams. It provides interfaces for algorithms to be run on distributed stream processing engines like Apache S4 and Twitter Storm. The author's algorithms were tested on these platforms using the SAMOA framework.
This document discusses how to download and play the mobile game Subway Surfers on a personal computer. It describes using BlueStacks, an Android emulator, to install and run the game normally played on phones and tablets. BlueStacks allows users to access Google Play to download Subway Surfers and other Android apps. Once installed through BlueStacks, the game can be played offline on a PC like a mobile game, allowing users to enjoy Subway Surfers on a larger screen without being limited to a phone.
INOVA GIS Platform represents centralized Enterprise GIS (Geographical Information System) that enables seamless data access for any number of different departments within business organization and beyond. Data can be accessed for viewing, analyzing, editing, etc. Apart from that, data can be presented to wider audience with the possibility to control what type of data and to what extent it will be presented.
Parallel and Distributed Algorithms for Large Text Datasets AnalysisIllia Ovchynnikov
This document is a thesis submitted for the degree of Bachelor of Computer Science at Opole University of Technology. It explores using distributed systems for processing large text datasets in the context of near duplicate text detection. The study reviews big data concepts, popular analytics frameworks like Hadoop and Spark, and algorithms for determining document duplication levels. The results were applied to develop a prototype distributed anti-plagiarism system that showed improved performance over existing solutions for analyzing large collections of text data.
The document provides an overview of distributed computing and related technologies. It discusses the history of distributed computing including local, parallel, grid and distributed computing. It then discusses applications of distributed computing like web indexing and recommendations. The document introduces Hadoop and its core components HDFS and MapReduce. It also discusses related technologies like HBase, Mahout and challenges in designing distributed systems. It provides examples of using Mahout for machine learning tasks like classification, clustering and recommendations.
Spatial Data Integrator - Software Presentation and Use Casesmathieuraj
Spatial Data Integrator software is an open source ETL tool that adds spatial capabilities to Talend Open Studio for extracting, transforming, and loading geospatial data. It can be used to perform tasks like aggregating data from multiple sources, merging geographic layers, and chaining quality checks on digitized documents. The presentation demonstrated how to configure SDI, connect components, and execute jobs to perform these types of spatial data integration and management tasks.
State of the Art Web Mapping with Open SourceOSCON Byrum
This document discusses the importance of open source tools and data for web mapping. It begins by providing background on TileMill and Mapbox, which provide open source tools for making maps. It then discusses key concepts in web mapping like geospatial data formats, tile rendering, and minimal code examples. Modern approaches to web mapping involve preprocessing data, using tile renderers and caches, and gradually rendering more client-side. Upcoming improvements may optimize tiled formats and storage. TileMill is demonstrated as an open source tool for making maps. The talk concludes by emphasizing other open mapping tools like CartoDB, Stamen, and CartoDB that build on these concepts.
The main focus of this study is to find appropriate and stable solutions for representing the statistical data into map with some special features. This research also includes the comparison between different solutions for specific features. In this research I have found three solutions using three different technologies namely Oracle MapViewer, QGIS and AnyMap which are different solutions with different specialties. Each solution has its own specialty so we can choose any solution for representing the statistical data into maps depending on our criteria’s.
This document provides definitions and terminology related to producing General Transit Feed Specification (GTFS) data from geospatial vector data. It discusses stops, routes, trips, shapefiles, coordinate reference systems, QGIS, PostgreSQL/PostGIS, and the shp2GTFS tool. The document outlines how to use these tools and technologies to create GTFS feeds from shapefile data in QGIS, store the data in a PostgreSQL/PostGIS database, and convert it to GTFS format using the custom shp2GTFS Python script.
This document provides an introduction to data collection and the General Transit Feed Specification (GTFS). It defines key terms like stops, routes, trips, and shapefiles. It discusses tools used like QGIS, PostgreSQL, PostGIS, and the shp2GTFS Python script for converting shapefile data to GTFS format. The document outlines the basic usage of these tools for collecting transit network data in layers and shapefiles using QGIS, storing it in a PostgreSQL/PostGIS database, and converting it to GTFS format.
TeleCAD-GIS is a scalable, Autodesk-based, CAD/GIS solution for both national and regional telecommunications infrastructure networks planning, design, documenting and maintenance. It provides valuable tools targeting customer-owned outside plant design; OSP right-of-way and route design; OSP space design; underground, direct-buried, and aerial plant design; OSP cabling hardware and OSP grounding, bonding and electrical protection systems; automated switching and support systems design, etc.
This document describes the development of a web scraping tool to extract useful mobile app market data from Appannie's website. The tool automates browsing to Appannie pages using Selenium, scrapes app name, description and version history from individual app pages, and saves the data to CSV files. It iterates through Appannie's top charts from the past year for the US and Chinese markets to build a structured dataset for analysis and to help app developers. The project uses an agile development approach with weekly iterations to expand the tool's functionality and optimize performance over time.
This tutorial provides instructions for using the StatPlanet software to create interactive maps and visualizations. It discusses downloading and installing StatPlanet, importing shapefiles and data to create maps, different map types (choropleth, proportional symbol), customizing map features and options, and troubleshooting. The document aims to enable non-technical users to explore and communicate data through interactive maps and visualizations using StatPlanet.
Cametrics is a cloud-based data logging, aggregation, and visualization service that aims to simplify the process of collecting and analyzing data across projects. It provides client libraries that standardize the measurement interface and centralizes collected data on Google App Engine for scalability. Cametrics can accept any data type, perform statistical summaries, and return data in requested formats that can be visualized using common APIs. This paper details the design of Cametrics and how it addresses limitations of existing services through its flexibility, accessibility, and extensibility.
Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...confluent
Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture.
Things we’ll discuss decoupling your system at scale via event-driven architectures include:
– Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows.
– Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern
– How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea)
– Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way.
– Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems
– Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem.
– The simplicity and power of streaming SQL with microservices
Getting insights from IoT data with Apache Spark and Apache BahirLuciano Resende
The Internet of Things (IoT) is all about connected devices that produce and exchange data, and producing insights from these high volumes of data is challenging. On this session, we will start by providing a quick introduction to the MQTT protocol, and focus on using AI and machine learning techniques to provide insights from data collected from IoT devices. We will present some common AI concepts and techniques used by the industry to deploy state-of-the-art smart IoT systems. These techniques allow systems to determined patterns from the data, predict and prevent failures as well as suggest actions that can be used to minimize or avoid IoT device breakdowns on an intelligent way beyond rule-based and database search approaches. We will finish with a demo that puts together all the techniques discussed in an application that uses Apache Spark and Apache Bahir support for MQTT.
The document describes a project report submitted by R Ashwin for the award of a Bachelor of Technology degree. It discusses the distributed implementation of the graph database system DGraph. The key steps in the distributed version include sharding the data, assigning unique IDs, loading the data into different servers, and enabling communication between servers through network calls. Performance evaluation on the Freebase film dataset showed that the distributed version had higher throughput and lower latency than the centralized version, especially as the load and computational power increased.
ALT-F1.BE : The Accelerator (Google Cloud Platform)Abdelkrim Boujraf
The Accelerator is an IT infrastructure able to collect and analyze a massive amount of public data on the WWW.
The Accelerator leverages the untapped potential of web data with the first solution designed for diverse sectors,
completely scalable, available on-premise, and cloud-provider agnostic.
Walmart Business+ and Spark Good for Nonprofits.pdfTechSoup
"Learn about all the ways Walmart supports nonprofit organizations.
You will hear from Liz Willett, the Head of Nonprofits, and hear about what Walmart is doing to help nonprofits, including Walmart Business and Spark Good. Walmart Business+ is a new offer for nonprofits that offers discounts and also streamlines nonprofits order and expense tracking, saving time and money.
The webinar may also give some examples on how nonprofits can best leverage Walmart Business+.
The event will cover the following::
Walmart Business + (https://business.walmart.com/plus) is a new shopping experience for nonprofits, schools, and local business customers that connects an exclusive online shopping experience to stores. Benefits include free delivery and shipping, a 'Spend Analytics” feature, special discounts, deals and tax-exempt shopping.
Special TechSoup offer for a free 180 days membership, and up to $150 in discounts on eligible orders.
Spark Good (walmart.com/sparkgood) is a charitable platform that enables nonprofits to receive donations directly from customers and associates.
Answers about how you can do more with Walmart!"
The document provides a user guide for StatPlanet and StatTrends data visualization software. Key features include:
- Interactive thematic maps that can display data using choropleth or proportional symbol mapping. Maps can be customized through options like colors, labels, and overlays.
- Data can be visualized through graphs like bar charts, column charts, time series, scatter plots, and bubble charts. Graphs are customizable.
- Data is imported and organized using categories, indicators, and variables. Definitions can be added for indicators. Data can be imported from files, databases, or directly edited.
- Extensive customization options are available for maps, graphs, categories, indicators, data sources, and more.
Scalable Distributed Real-Time Clustering for Big Data StreamsAntonio Severien
This thesis presents a scalable distributed clustering algorithm for streaming big data. The author implemented a real-time distributed clustering algorithm and a classification algorithm using the Scalable Advanced Massive Online Analysis (SAMOA) framework. SAMOA is a platform-independent framework for distributed machine learning on data streams. It provides interfaces for algorithms to be run on distributed stream processing engines like Apache S4 and Twitter Storm. The author's algorithms were tested on these platforms using the SAMOA framework.
This document discusses how to download and play the mobile game Subway Surfers on a personal computer. It describes using BlueStacks, an Android emulator, to install and run the game normally played on phones and tablets. BlueStacks allows users to access Google Play to download Subway Surfers and other Android apps. Once installed through BlueStacks, the game can be played offline on a PC like a mobile game, allowing users to enjoy Subway Surfers on a larger screen without being limited to a phone.
INOVA GIS Platform represents centralized Enterprise GIS (Geographical Information System) that enables seamless data access for any number of different departments within business organization and beyond. Data can be accessed for viewing, analyzing, editing, etc. Apart from that, data can be presented to wider audience with the possibility to control what type of data and to what extent it will be presented.
Parallel and Distributed Algorithms for Large Text Datasets AnalysisIllia Ovchynnikov
This document is a thesis submitted for the degree of Bachelor of Computer Science at Opole University of Technology. It explores using distributed systems for processing large text datasets in the context of near duplicate text detection. The study reviews big data concepts, popular analytics frameworks like Hadoop and Spark, and algorithms for determining document duplication levels. The results were applied to develop a prototype distributed anti-plagiarism system that showed improved performance over existing solutions for analyzing large collections of text data.
The document provides an overview of distributed computing and related technologies. It discusses the history of distributed computing including local, parallel, grid and distributed computing. It then discusses applications of distributed computing like web indexing and recommendations. The document introduces Hadoop and its core components HDFS and MapReduce. It also discusses related technologies like HBase, Mahout and challenges in designing distributed systems. It provides examples of using Mahout for machine learning tasks like classification, clustering and recommendations.
Spatial Data Integrator - Software Presentation and Use Casesmathieuraj
Spatial Data Integrator software is an open source ETL tool that adds spatial capabilities to Talend Open Studio for extracting, transforming, and loading geospatial data. It can be used to perform tasks like aggregating data from multiple sources, merging geographic layers, and chaining quality checks on digitized documents. The presentation demonstrated how to configure SDI, connect components, and execute jobs to perform these types of spatial data integration and management tasks.
State of the Art Web Mapping with Open SourceOSCON Byrum
This document discusses the importance of open source tools and data for web mapping. It begins by providing background on TileMill and Mapbox, which provide open source tools for making maps. It then discusses key concepts in web mapping like geospatial data formats, tile rendering, and minimal code examples. Modern approaches to web mapping involve preprocessing data, using tile renderers and caches, and gradually rendering more client-side. Upcoming improvements may optimize tiled formats and storage. TileMill is demonstrated as an open source tool for making maps. The talk concludes by emphasizing other open mapping tools like CartoDB, Stamen, and CartoDB that build on these concepts.
The main focus of this study is to find appropriate and stable solutions for representing the statistical data into map with some special features. This research also includes the comparison between different solutions for specific features. In this research I have found three solutions using three different technologies namely Oracle MapViewer, QGIS and AnyMap which are different solutions with different specialties. Each solution has its own specialty so we can choose any solution for representing the statistical data into maps depending on our criteria’s.
This document provides definitions and terminology related to producing General Transit Feed Specification (GTFS) data from geospatial vector data. It discusses stops, routes, trips, shapefiles, coordinate reference systems, QGIS, PostgreSQL/PostGIS, and the shp2GTFS tool. The document outlines how to use these tools and technologies to create GTFS feeds from shapefile data in QGIS, store the data in a PostgreSQL/PostGIS database, and convert it to GTFS format using the custom shp2GTFS Python script.
This document provides an introduction to data collection and the General Transit Feed Specification (GTFS). It defines key terms like stops, routes, trips, and shapefiles. It discusses tools used like QGIS, PostgreSQL, PostGIS, and the shp2GTFS Python script for converting shapefile data to GTFS format. The document outlines the basic usage of these tools for collecting transit network data in layers and shapefiles using QGIS, storing it in a PostgreSQL/PostGIS database, and converting it to GTFS format.
TeleCAD-GIS is a scalable, Autodesk-based, CAD/GIS solution for both national and regional telecommunications infrastructure networks planning, design, documenting and maintenance. It provides valuable tools targeting customer-owned outside plant design; OSP right-of-way and route design; OSP space design; underground, direct-buried, and aerial plant design; OSP cabling hardware and OSP grounding, bonding and electrical protection systems; automated switching and support systems design, etc.
This document describes the development of a web scraping tool to extract useful mobile app market data from Appannie's website. The tool automates browsing to Appannie pages using Selenium, scrapes app name, description and version history from individual app pages, and saves the data to CSV files. It iterates through Appannie's top charts from the past year for the US and Chinese markets to build a structured dataset for analysis and to help app developers. The project uses an agile development approach with weekly iterations to expand the tool's functionality and optimize performance over time.
This tutorial provides instructions for using the StatPlanet software to create interactive maps and visualizations. It discusses downloading and installing StatPlanet, importing shapefiles and data to create maps, different map types (choropleth, proportional symbol), customizing map features and options, and troubleshooting. The document aims to enable non-technical users to explore and communicate data through interactive maps and visualizations using StatPlanet.
Cametrics is a cloud-based data logging, aggregation, and visualization service that aims to simplify the process of collecting and analyzing data across projects. It provides client libraries that standardize the measurement interface and centralizes collected data on Google App Engine for scalability. Cametrics can accept any data type, perform statistical summaries, and return data in requested formats that can be visualized using common APIs. This paper details the design of Cametrics and how it addresses limitations of existing services through its flexibility, accessibility, and extensibility.
Event Driven Architecture with a RESTful Microservices Architecture (Kyle Ben...confluent
Tinder’s Quickfire Pipeline powers all things data at Tinder. It was originally built using AWS Kinesis Firehoses and has since been extended to use both Kafka and other event buses. It is the core of Tinder’s data infrastructure. This rich data flow of both client and backend data has been extended to service a variety of needs at Tinder, including Experimentation, ML, CRM, and Observability, allowing backend developers easier access to shared client side data. We perform this using many systems, including Kafka, Spark, Flink, Kubernetes, and Prometheus. Many of Tinder’s systems were natively designed in an RPC first architecture.
Things we’ll discuss decoupling your system at scale via event-driven architectures include:
– Powering ML, backend, observability, and analytical applications at scale, including an end to end walk through of our processes that allow non-programmers to write and deploy event-driven data flows.
– Show end to end the usage of dynamic event processing that creates other stream processes, via a dynamic control plane topology pattern and broadcasted state pattern
– How to manage the unavailability of cached data that would normally come from repeated API calls for data that’s being backfilled into Kafka, all online! (and why this is not necessarily a “good” idea)
– Integrating common OSS frameworks and libraries like Kafka Streams, Flink, Spark and friends to encourage the best design patterns for developers coming from traditional service oriented architectures, including pitfalls and lessons learned along the way.
– Why and how to avoid overloading microservices with excessive RPC calls from event-driven streaming systems
– Best practices in common data flow patterns, such as shared state via RocksDB + Kafka Streams as well as the complementary tools in the Apache Ecosystem.
– The simplicity and power of streaming SQL with microservices
Getting insights from IoT data with Apache Spark and Apache BahirLuciano Resende
The Internet of Things (IoT) is all about connected devices that produce and exchange data, and producing insights from these high volumes of data is challenging. On this session, we will start by providing a quick introduction to the MQTT protocol, and focus on using AI and machine learning techniques to provide insights from data collected from IoT devices. We will present some common AI concepts and techniques used by the industry to deploy state-of-the-art smart IoT systems. These techniques allow systems to determined patterns from the data, predict and prevent failures as well as suggest actions that can be used to minimize or avoid IoT device breakdowns on an intelligent way beyond rule-based and database search approaches. We will finish with a demo that puts together all the techniques discussed in an application that uses Apache Spark and Apache Bahir support for MQTT.
The document describes a project report submitted by R Ashwin for the award of a Bachelor of Technology degree. It discusses the distributed implementation of the graph database system DGraph. The key steps in the distributed version include sharding the data, assigning unique IDs, loading the data into different servers, and enabling communication between servers through network calls. Performance evaluation on the Freebase film dataset showed that the distributed version had higher throughput and lower latency than the centralized version, especially as the load and computational power increased.
ALT-F1.BE : The Accelerator (Google Cloud Platform)Abdelkrim Boujraf
The Accelerator is an IT infrastructure able to collect and analyze a massive amount of public data on the WWW.
The Accelerator leverages the untapped potential of web data with the first solution designed for diverse sectors,
completely scalable, available on-premise, and cloud-provider agnostic.
Similar to Event Visualization with OpenStreetMap Data, Interdisciplinary Project (20)
Walmart Business+ and Spark Good for Nonprofits.pdfTechSoup
"Learn about all the ways Walmart supports nonprofit organizations.
You will hear from Liz Willett, the Head of Nonprofits, and hear about what Walmart is doing to help nonprofits, including Walmart Business and Spark Good. Walmart Business+ is a new offer for nonprofits that offers discounts and also streamlines nonprofits order and expense tracking, saving time and money.
The webinar may also give some examples on how nonprofits can best leverage Walmart Business+.
The event will cover the following::
Walmart Business + (https://business.walmart.com/plus) is a new shopping experience for nonprofits, schools, and local business customers that connects an exclusive online shopping experience to stores. Benefits include free delivery and shipping, a 'Spend Analytics” feature, special discounts, deals and tax-exempt shopping.
Special TechSoup offer for a free 180 days membership, and up to $150 in discounts on eligible orders.
Spark Good (walmart.com/sparkgood) is a charitable platform that enables nonprofits to receive donations directly from customers and associates.
Answers about how you can do more with Walmart!"
Chapter wise All Notes of First year Basic Civil Engineering.pptxDenish Jangid
Chapter wise All Notes of First year Basic Civil Engineering
Syllabus
Chapter-1
Introduction to objective, scope and outcome the subject
Chapter 2
Introduction: Scope and Specialization of Civil Engineering, Role of civil Engineer in Society, Impact of infrastructural development on economy of country.
Chapter 3
Surveying: Object Principles & Types of Surveying; Site Plans, Plans & Maps; Scales & Unit of different Measurements.
Linear Measurements: Instruments used. Linear Measurement by Tape, Ranging out Survey Lines and overcoming Obstructions; Measurements on sloping ground; Tape corrections, conventional symbols. Angular Measurements: Instruments used; Introduction to Compass Surveying, Bearings and Longitude & Latitude of a Line, Introduction to total station.
Levelling: Instrument used Object of levelling, Methods of levelling in brief, and Contour maps.
Chapter 4
Buildings: Selection of site for Buildings, Layout of Building Plan, Types of buildings, Plinth area, carpet area, floor space index, Introduction to building byelaws, concept of sun light & ventilation. Components of Buildings & their functions, Basic concept of R.C.C., Introduction to types of foundation
Chapter 5
Transportation: Introduction to Transportation Engineering; Traffic and Road Safety: Types and Characteristics of Various Modes of Transportation; Various Road Traffic Signs, Causes of Accidents and Road Safety Measures.
Chapter 6
Environmental Engineering: Environmental Pollution, Environmental Acts and Regulations, Functional Concepts of Ecology, Basics of Species, Biodiversity, Ecosystem, Hydrological Cycle; Chemical Cycles: Carbon, Nitrogen & Phosphorus; Energy Flow in Ecosystems.
Water Pollution: Water Quality standards, Introduction to Treatment & Disposal of Waste Water. Reuse and Saving of Water, Rain Water Harvesting. Solid Waste Management: Classification of Solid Waste, Collection, Transportation and Disposal of Solid. Recycling of Solid Waste: Energy Recovery, Sanitary Landfill, On-Site Sanitation. Air & Noise Pollution: Primary and Secondary air pollutants, Harmful effects of Air Pollution, Control of Air Pollution. . Noise Pollution Harmful Effects of noise pollution, control of noise pollution, Global warming & Climate Change, Ozone depletion, Greenhouse effect
Text Books:
1. Palancharmy, Basic Civil Engineering, McGraw Hill publishers.
2. Satheesh Gopi, Basic Civil Engineering, Pearson Publishers.
3. Ketki Rangwala Dalal, Essentials of Civil Engineering, Charotar Publishing House.
4. BCP, Surveying volume 1
Strategies for Effective Upskilling is a presentation by Chinwendu Peace in a Your Skill Boost Masterclass organisation by the Excellence Foundation for South Sudan on 08th and 09th June 2024 from 1 PM to 3 PM on each day.
Reimagining Your Library Space: How to Increase the Vibes in Your Library No ...Diana Rendina
Librarians are leading the way in creating future-ready citizens – now we need to update our spaces to match. In this session, attendees will get inspiration for transforming their library spaces. You’ll learn how to survey students and patrons, create a focus group, and use design thinking to brainstorm ideas for your space. We’ll discuss budget friendly ways to change your space as well as how to find funding. No matter where you’re at, you’ll find ideas for reimagining your space in this session.
বাংলাদেশের অর্থনৈতিক সমীক্ষা ২০২৪ [Bangladesh Economic Review 2024 Bangla.pdf] কম্পিউটার , ট্যাব ও স্মার্ট ফোন ভার্সন সহ সম্পূর্ণ বাংলা ই-বুক বা pdf বই " সুচিপত্র ...বুকমার্ক মেনু 🔖 ও হাইপার লিংক মেনু 📝👆 যুক্ত ..
আমাদের সবার জন্য খুব খুব গুরুত্বপূর্ণ একটি বই ..বিসিএস, ব্যাংক, ইউনিভার্সিটি ভর্তি ও যে কোন প্রতিযোগিতা মূলক পরীক্ষার জন্য এর খুব ইম্পরট্যান্ট একটি বিষয় ...তাছাড়া বাংলাদেশের সাম্প্রতিক যে কোন ডাটা বা তথ্য এই বইতে পাবেন ...
তাই একজন নাগরিক হিসাবে এই তথ্য গুলো আপনার জানা প্রয়োজন ...।
বিসিএস ও ব্যাংক এর লিখিত পরীক্ষা ...+এছাড়া মাধ্যমিক ও উচ্চমাধ্যমিকের স্টুডেন্টদের জন্য অনেক কাজে আসবে ...
Beyond Degrees - Empowering the Workforce in the Context of Skills-First.pptxEduSkills OECD
Iván Bornacelly, Policy Analyst at the OECD Centre for Skills, OECD, presents at the webinar 'Tackling job market gaps with a skills-first approach' on 12 June 2024
This slide is special for master students (MIBS & MIFB) in UUM. Also useful for readers who are interested in the topic of contemporary Islamic banking.
How to Fix the Import Error in the Odoo 17Celine George
An import error occurs when a program fails to import a module or library, disrupting its execution. In languages like Python, this issue arises when the specified module cannot be found or accessed, hindering the program's functionality. Resolving import errors is crucial for maintaining smooth software operation and uninterrupted development processes.
ISO/IEC 27001, ISO/IEC 42001, and GDPR: Best Practices for Implementation and...PECB
Denis is a dynamic and results-driven Chief Information Officer (CIO) with a distinguished career spanning information systems analysis and technical project management. With a proven track record of spearheading the design and delivery of cutting-edge Information Management solutions, he has consistently elevated business operations, streamlined reporting functions, and maximized process efficiency.
Certified as an ISO/IEC 27001: Information Security Management Systems (ISMS) Lead Implementer, Data Protection Officer, and Cyber Risks Analyst, Denis brings a heightened focus on data security, privacy, and cyber resilience to every endeavor.
His expertise extends across a diverse spectrum of reporting, database, and web development applications, underpinned by an exceptional grasp of data storage and virtualization technologies. His proficiency in application testing, database administration, and data cleansing ensures seamless execution of complex projects.
What sets Denis apart is his comprehensive understanding of Business and Systems Analysis technologies, honed through involvement in all phases of the Software Development Lifecycle (SDLC). From meticulous requirements gathering to precise analysis, innovative design, rigorous development, thorough testing, and successful implementation, he has consistently delivered exceptional results.
Throughout his career, he has taken on multifaceted roles, from leading technical project management teams to owning solutions that drive operational excellence. His conscientious and proactive approach is unwavering, whether he is working independently or collaboratively within a team. His ability to connect with colleagues on a personal level underscores his commitment to fostering a harmonious and productive workplace environment.
Date: May 29, 2024
Tags: Information Security, ISO/IEC 27001, ISO/IEC 42001, Artificial Intelligence, GDPR
-------------------------------------------------------------------------------
Find out more about ISO training and certification services
Training: ISO/IEC 27001 Information Security Management System - EN | PECB
ISO/IEC 42001 Artificial Intelligence Management System - EN | PECB
General Data Protection Regulation (GDPR) - Training Courses - EN | PECB
Webinars: https://pecb.com/webinars
Article: https://pecb.com/article
-------------------------------------------------------------------------------
For more information about PECB:
Website: https://pecb.com/
LinkedIn: https://www.linkedin.com/company/pecb/
Facebook: https://www.facebook.com/PECBInternational/
Slideshare: http://www.slideshare.net/PECBCERTIFICATION
it describes the bony anatomy including the femoral head , acetabulum, labrum . also discusses the capsule , ligaments . muscle that act on the hip joint and the range of motion are outlined. factors affecting hip joint stability and weight transmission through the joint are summarized.
3. Abstract
OpenStreetMap (OSM) is a collaborative volunteer based mapping platform. The large
amount of data is stored in a robust key (tags) value pair database. The database can
store any sets of key values in the form of strings. However entered data is monitored and
moderated by dedicated volunteers to prevent garbage input and vandalism. Current set of
tags used by OSM are only spatial tags. In this project, the OSM database and technology
infrastructure is copied and extended to explore the idea of temporal data collection. Events
information are collected and entered into the database using special tags. The process is
simplified using a self made plugin on top of JOSM editor.
A web based platform has been built named ‘OpenEventMap’ that extracts the data from
key-value database of OSM, gives structure to the data by converting into another format
for easy searching and finally renders it for visualization purposes.
The details of event based tag metadata, system architecture of the system and technical
details of entire system has been presented in this report.
Introduction
Our history is written through myriads of events happening all over the world. These events
bring about wide array of changes from political, geographical, social, architecture to name
a few. It is important to document these events precisely to facilitate further study and
research. Currently, there is no platform that records events methodologically which includes
event description along with associated spatial and temporal information. A knowledge
repository that captures past and future events needs to be made.
Of different ways of building a knowledge repository, the Wikipedia model has come out as
a voluntary, collaborative, transparent and accurate way of building large scale knowledge
repository. It has been replicated into creating maps in the form of Open Street Map
(OSM). OSM has thousands of volunteers contributing location based data to build an online
platform for community based Maps. Anyone with a GPS device and an internet connection
can easily start building maps of his locality. It is based entirely on voluntary effort and this
effort has helped build a very high quality online global map.
Open Street Map Database Structure
OSM data structure is very simple. A piece of data (row in the database) consists of a pair
of (key, value). Using this, for any point in the database, multiple data can be associated as
the point’s attributes.
When any kind of data is entered into the system, no checks are performed to restrict the
input by the system. So a user can enter any kind of data into the OSM database. This system
allows for all different kinds of data to be entered to accomodate different terminologies used
in different parts of the world. However it also opens the door for vandalism and wrong input.
In order to prevent users from entering or misusing the platform, dedicated volunteers check
the consistency of the database from time to time and either fix the database or punish the
user by deleting a user account or both.
3
4. Figure 1: Example of key, value data for a node in OpenStreetMap
The general convention in OSM community is to store spatial data. There has been discussions
regarding storage of event related information but it has been discouraged.
Open Event Map Data
We realize that although temporal data cannot go into the main OSM database for now, we
can copy the existing data and the software to create a replica environment for our research
purposes. By setting up a server that behaves exactly like the main OSM server, we can use
wide set of tools for data collection and modification into our platform.
OSM has predefined set of tags to describe spatial data. Our challenge was to come up with
a set of tags that described temporal attributes of given points. This set of tags should
easily fit into existing system, not just for describing attributes but also for other purpose
like searching. The set of tags are described in the following sections with their advantages
and their shortcomings.
Most of our data is entered into the system using a tool called JOSM. A JOSM plugin
has been created that provides a Graphical User Interface (GUI) to easily enter the event
information by point and click.
4
5. Open Event Map Visualization
Events data are associated with existing nodes, ways or relations in the system. Nodes, Ways
and Relations are the spatial data that represent buildings, trees, pools, roads, compounds,
parks or any other physical elements that exist. By associating events to these, we associate
temporal data to spatial data. For visualization purpose, we extract stored temporal data
and it’s spatial coordinates and display it on a web based platform by placing colored markers
on top of a map.
User can use the web platform to search for events based on different parameters like event
name, category, event dates and see color coded results based on different category.
Open Street Map Technology Stack
OpenStreetMap is powerd by a combination of Open Source softwares that can be downloaded
freely from the internet. The detail guide to setting up Open Street Map on a server is
available at The Rails Port link. A brief description of different components including a
diagram follows.
Server
Operating system: Archlinux
RAM: 12 GB
Harddisk Space: 500 GB
Database: Postgres 9.2.4
Webserver: Apache 2.2.24
Operating System: Archlinux
Archlinux has both advantages as well as disadvantages. It is a rolling distribution, so
the software packages are always up to date. The disadvantage is that constantly changing
packages might break required version of libraries. However, the main application The Rails
Port runs on a virtualenv environment with specific version of ruby and rails gems.
Apache webserver is for serving three different applications. Passenger Phusion is used to
serve The Rails Port based on Rails and Open Event Map based on python/django. mod_tile
is used to render and serve tiles.
Postgres with PostGIS extension is used as our main database server.
The Rails Port (TRP)
TRP is the codebase for entire webfrontend of Open Street Map. This includes:
1. Map frontend for visualization.
5
7. 2. User login and map editing using Potlatch 2 editor.
3. API version 6 accessible via /api/ URL.
The installation of TRP can be accessed through http://osm.openeventmap.tum.de. Similarly
the API capabilities can be viewed from http://osm.openeventmap.tum.de/api/capabilities.
Source code for TRP can be downloaded easily from GitHub. It is a standard rails application
that connects to the Postgres database. A detail guide on installation of The Rails Port(TRP)
is given in the Appendix A.
Database Setup
The complete OpenStreetMap database is available for download from OSM Planet. It can
be copied, distributed or adapted to different forms as long as the Open Data Commons
Open Database License is agreed upon by the user. However, the complete database
is 30GB large in size in a compressed format. It is estimated to be more than 100GB
uncompressed. Such a large volume of data is not required in our case as we deal mostly in
the region around Bavaria, Germany.
Geofabrik provides map database files of smaller regions by dividing the main OSM database
into multiple regions. The Oberbayern mapfile is used in our case, which is 96MB when
compressed. We only work with 0.32% of total database size.
The data is imported into our Postgres database and connected with TRP.
Rendering Software
We have three subdomains to serve tiles in our website.
1. a.tile.openeventmap.tum.de/osm_tiles/x/y/z.png
2. b.tile.openeventmap.tum.de/osm_tiles/x/y/z.png
3. c.tile.openeventmap.tum.de/osm_tiles/x/y/z.png
Where x and y are square coordinates and z is the zoom level of the tile.
Tiles are served using mod_tile extension. It maps every request in above format to renderd
daemon. The renderd daemon in turn checks if an existing tile is available in the cache. If
found, it then checks if the tile needs to be regenerated due to changes in OSM data. As
a result, it returns either the tile in the cache or generates a new tile, saves the tile in the
cache and serves it through apache.
renderd uses Mapnik as a tile server. Mapnik connects to the database, extract information
for tile generation and then applies color styles on top of the data and generates the tiles.
Mapnik can use different stylesheet to generate different types of maps. In our case, the
default OSM stylesheet is used.
7
8. Events, Event Tags and EventEditor JOSM Plugin
In OSM, we have three basic types of elements: node, way and relation. A node is a single
point characterised by a unique pair of latitude and longitude values. A node can represent
any physical object that exist in the exact coordinate. A way is a set of nodes grouped
together. A way can be used to represent a bigger larger area or a continuous path. Similarly
a relation is a logical grouping of multiple ways and nodes to represent an advanced concept.
Besides having spatial coordinate data, nodes, ways and relations have other information
including name, classification type and other varieties of properties in the form of tags.
Event Tags
An event is primarily associated with one of the nodes, ways or relations. This is done by
associating special event tags to the OSM elements which is described in detail below.
The primary tag to mark any element as event, we need to set the following tag:
event: yes
If associated with nodes, the icons have an added “E” on their right side as following:
Figure 3: A node with event information associated showing icon with “E”.
We further identify other event related tags to associate more information to the OSM
elements and they are:
1. name: Name.
2. category: A list of primary categories user can choose from.
3. sub_category: Each primary category has a list of further sub categories for refined
classification.
4. organization: Main organization related with the event.
5. start_date: Actual day when the event occurs.
6. end_date: In case of a single day event, it is same as start_date but different if it is
a longer event.
7. url: A website that explains more about the event.
8. numparticipants: An estimate of the number of participants in the event.
9. howoften: Event repeat frequency.
10. comment: Any extra note a user can associate with the event.
11. related_items: A comma separated list of other OSM elements related with this
event.
One OSM element can contain more than one event. To make it possible, we use namespacing
in tags and add index to each event. Consider a way with two associated events; its tags will
be as following.
8
9. name = kaufingerstraße
highway = pedestrian
bicycle = no
cycleway = no
...
event = yes
event:0:name = Christkindlemarkt
event:0:category = social
event:0:sub_category = Fairs / Festivals
...
event:1:name = Munich Carnival
event:1:category = social
event:1:sub_category = Fairs / Festivals
event:1:start_date = 07/01/2013
event:1:end_date = 12/02/2013
event:1:howoften = Yearly
...
The character “:” is used for namespacing. The format of any event tag is as follows
event[namespacer]event_index[namespacer]event_attribute = value
1. event_index: Starts with 0 and for each event, it increses by one.
2. event_attribute: Can be one of the defined tags in the list above.
EventEditor JOSM Plugin
JOSM is an offline Java based OSM editor which is quite powerful and fiture rich. It supports
extra plugins that can be downloaded from the internet for added functionality.
To add event information, an extra plugin called event_editor has been created. It was
created completely in Java. With this users do not have to remember the special format
mentioned above to enter data, instead they can use a simple GUI.
Installation of the plugin is explained in more detail in Appendix B.
OpenEventMap Event Visualization
All the data collected into the locally deployed OSM database is converted into an searchable
database format. The process is shown in figure 2 from the green box named CRON.
Cron, OSM and OpenEventMap table formats
cron: is a time-based job scheduler in Unix-like computer operating systems.
9
10. Figure 4: EventEditor Starting Screenshot. 1) Option to mark current selected OSM Entity
to an event. 2) Button to create a new Event for the event entity. 3) GUI to enter more
detail tags for the event.
10
11. The cron configuration runs a special script in the server every minute. This script generates
osmchange files in .osc format which contains addition, modification or deletion of any OSM
Element within the last minute.
Each .osc file is processed by OpenEventMap script and inserted into a new table called
search_event. During this process, multiple tag based key - value pairs related to a single
event are converted into a single row in the new table format.
Figure 5: Comparison between OpenStreetMap and OpenEventMap table structure
In OSM current_node_tags table, a single event data is stored across multiple rows as
belows:
node_id
k
v
12345
event
yes
12345
event:0:name
Oktoberfest
12345
event:0:category
social
12345
event:1:name
Frühlingsfest
12345
event:1:category
social
The two events are converted into two single rows in search_event table as following:
id
event_type
type_id
number
name
11
category
subcategory
...
12. 1
node
12345
0
Oktoberfest
social
...
2
node
12345
1
Frühlingsfest
social
...
The search_event table format makes it easier to do complex queries searching more than
one field at any time. For example the following query is now possible:
SELECT * FROM search_event WHERE name LIKE %val%
AND category=’social’
AND ...
The query above would be complicated to execute on current_node_tags table.
OpenEventMap web application
The web application allows any user to search event database.
Search parameters
There are four parameters to filter the search results by: event name does a partial matching
with existing event names, category parameter searches for exact match in category and
subcategory fields, start date and end date creates a date range and returns all the events
that fall under given range. Date filter also takes care of repeating events. For example, an
event that occurred in March 5, 2012 and repeats yearly will be found when searching for
events between March 1, 2013 and March 10, 2013.
If search is performed with non of the parameters entered, it returns all matching events
within given map region.
Search results are color coded. Most of the popular event categories are assigned different
colors which are shown on the textual search results section as well as a colorful pointer on
the map. As the pointer is clicked, all events belonging to the place is shown in a popup.
Event data can be exported along with OSM elements like node, ways and relations in OSM
format using the Export link in the Search Results Section. It redirects the user to the OSM
webpage.
There is also a feature to first search any location by name. This is achieved by OSM’s
Nominatim api. For given location name, a search is performed around Bayern area and the
results are listed. When clicked on the result, the map gets zoomed in.
Conclusion
All parts of the system developed have been on top of opensource technologies and open
data. Open technologies provide easily accessible platforms. In this case, OpenStreetMap
12
13. Figure 6: Web application screenshot. 1) Choice between Nominatim search and Event
search. 2) Fields to enter search parameters. 3) Search results grouped into categories. 4)
Visualization of OSM elements and associated events.
13
14. ecosystem provided us with ready made data, software suite from backend to frontend like
TheRailsPort, mod_tile, Mapnik, JOSM etc. Building complete infrastructure by self would
take extremely large amount of time.
We are also grateful to the volunteers spread all over the internet for collecting the data,
submitting it online in OSM and allowing other individuals to use them. The philosophy
motivates further more people to contribute and innovate. In our case, we were able to
imagine, plan, implement and finally visualize chronological data like events on top of
OpenStreetMap.
Not everything was easy. Documentation of most of the components were spread all over
OSM wiki. Functionalities of the software components was unclear and they had to be
understood and pieced together.
In conclusion, the following tasks have been successfully carried out:
1. The mapserver has been installed as requested.
2. A special purpose extension EventEditor has been created to help users enter events
data.
3. A webapp for visualizing the user contributed data has been created and setup.
Our original goals for the project have been successfully fulfilled.
Future Improvements
We realize that the current work is the base for further improvements. This is a beginning
for integrating temporal data into OSM. Looking ahead, there are more that can be done on
top of current work and these have been listed below.
Improving Search API
OpenEventMap is powered by a search API that takes name, category and startdate enddate as search parameters. The results are returned as JOSN objects.
First improvement can be increasing the search parameters to facilitate advanced search
using all the attributes we have collected.
Second improvement on search could be combining event names with locations. With this,
a user should be able to search for “Oktoberfest in Munich” and see the result. The earch
term contains both event name and location information.
Data collection process
Event data entry is a painful process right now. The normal process is to identify event,
then search the related items on OSM and then attach event attributes to those items.
When large and existing datasets have to be integrated, this is not trivial.
14
15. The solution could be a probabilistic model that identifies location based on techniques like
geocode to identify OSM entities. An extra layer of API can be made to quickly add and
edit one or more events in a single entity.
With this, softwares could be developed that would automatically save data from existing
data sources to OSM.
15
16. Appendix A : The Rails Port (TRP) Installation Notes
Most of the installation instructions are explained in detail here. The following instructions
have been explained for Archlinux with necessary modifications.
Introduction
Inorder to create our local copy of the openstreetmap database and visualize it like it is in
main OSM website, we setup a linux server and setup the following components on it.
1.
2.
3.
4.
Postgre Database for our data
renderd, a tool that generates tiles from the database
MapServer, that serves generated tiles.
RailsPort, the main application that powers OSM website with features of user login,
view, zoom, export tile and edit the data using potlatch editor.
Installation instruction are as follows:
Setting up Linux Server
Packages Installed:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.
postgis
mapnik
apache
boost
python2
ttf-dejavu
subversion
libxslt
imagemagick
unzip
yaourt (aur)
osm2pgsql-svn (aur)
osmosis (aur)
ruby1.9
We need crontab to run during startup, so run the following command to enable cronie
service.
$ systemctl enable cronie
$ systemctl start cronie
16
17. The Database
The main OpenStreetMap database is run on PostgreSQL database. PostgreSQL has
numerous features that allow geometry types. However OpenStreetMap does not used those
and have its own data primitives.
PostGIS is an extension to PostgreSQL database that provides spatial objects allowing
storage and query of information about location and mapping. The main database does not
use PostGIS features but is used by mapnik, the tile rendering software.
There are two main databases used in our approach:
1. ApiDB - This backs rails port, allows a read-write api to edit the database, thus
called ApiDB. JOSM and other editors make modifications in this database using the
api. The data is imported from planet dump files, in our case data inside oberbayernlatest.osm.bz2 was imported using the tool Osmosis.
2. Mapnik Db Format - This is used by mapnik to generate the tiles. Data is imported
from oberbayern-latest.osm.bz2 using osm2pgsql tool.
The database for ApiDB schema is named as myosmdb and database for mapnik schema is
named as mymapnikdb.
Configuring Postgres
The step by step process to install PostgreSQL and PostGIS extension is explained below.
Postgres requires initialization. We run the following commands as root on the server.
$
$
$
$
mkdir /var/lib/postgres/data
su - postgres -c "initdb --locale en_US.UTF-8 -D ’/var/lib/postgres/data’"
systemctl enable postgresql.service
systemctl start postgresql.service
A user is created in the server and a database is created under this user. The following
commands were ran under postgres username
$ sudo -u postgres createuser myosmuser -s -P
$ sudo -u postgres createdb -E UTF8 -O myosmuser myosmdb
As seen on the commands above, the username/password combination is myosmuser/myosmuser and the database name is myosmdb.
The RailsPort
OSM front end website is based on the Ruby on Rails framework. The source code for it can
be downloaded from git clone https://github.com/openstreetmap/openstreetmap-website.git
17
18. $ sudo mkdir /opt/myosm
$ sudo chown bibek /opt/myosm
Change bibek to the username you want.
$ cd /opt/myosm
$ git clone https://github.com/openstreetmap/openstreetmap-website.git
Copy proper application.yml and database.yml.
$ cd /opt/myosm
$ cp config/example.application.yml config/application.yml
$ cp config/example.database.yml config/database.yml
Change the development section of config/database.yml file
development:
adapter: postgresql
database: myosmdb
username: myosmuser
password: myosmuser
host: localhost
encoding: utf8
Quadtile functions need to be installed on the server. For this
$ cd /opt/myosm/openstreetmap-website/db/functions
$ make libpgosm.so
And the following commands need to be executed in postgresql
$ psql -U myosmuser -d myosmdb -c "CREATE EXTENSION btree_gist;"
$ psql -Umyosmuser -d myosmdb -c "CREATE FUNCTION
maptile_for_point(int8, int8, int4) RETURNS
int4 AS ’/opt/myosm/openstreetmap-website/db/functions/libpgosm’,
’maptile_for_point’ LANGUAGE C STRICT;"
$ psql -Umyosmuser -d myosmdb -c "CREATE FUNCTION
tile_for_point(int4, int4) RETURNS
int8 AS ’/opt/myosm/openstreetmap-website/db/functions/libpgosm’,
’tile_for_point’ LANGUAGE C STRICT;"
$ psql -Umyosmuser -d myosmdb -c "CREATE FUNCTION xid_to_int4(xid)
RETURNS int4 AS ’/opt/myosm/openstreetmap-website/db/functions/libpgosm’,
’xid_to_int4’ LANGUAGE C IMMUTABLE STRICT;"
Also ruby and rails are installed using ruby version manager (rvm)
18
19. $
$
$
$
curl -L https://get.rvm.io | bash -s stable --autolibs=enabled --ruby=1.9.3
gem install rails
cd /opt/myosm/openstreetmap-website
bundle install
Setup necessary database tables in postgresql. We use the production configuration here.
$ rake db:migrate
To make sure everything is setup properly, run
$ rake test
We can now populate the date for Munich.
$ cd /opt/myosm
$ wget http://download.geofabrik.de/europe/germany/bayern/oberbayern-latest.osm.bz2
$ osmosis --read-xml-0.6 file="oberbayern-latest.osm.bz2" --write-apidb-0.6
populateCurrentTables=yes host="localhost" database="myosmdb"
user="myosmuser" password="myosmuser" validateSchemaVersion=no
In the psql terminal the following was executed
$ echo "
select setval(’acls_id_seq’, (select max(id) from acls));
select setval(’changesets_id_seq’, (select max(id) from changesets));
select setval(’client_applicaations_id_seq’, (select max(id) from client_applications));
select setval(’countries_id_seq’, (select max(id) from countries));
select setval(’current_nodes_id_seq’, (select max(id) from current_nodes));
select setval(’current_relations_id_seq’, (select max(id) from current_relations));
select setval(’current_ways_id_seq’, (select max(id) from current_ways));
select setval(’diary_comments_id_seq’, (select max(id) from diary_comments));
select setval(’diary_entries_id_seq’, (select max(id) from diary_entries));
select setval(’friends_id_seq’, (select max(id) from friends));
select setval(’gpx_file_tags_id_seq’, (select max(id) from gpx_file_tags));
select setval(’gpx_files_id_seq’, (select max(id) from gpx_files));
select setval(’messages_id_seq’, (select max(id) from messages));
select setval(’oauth_nonces_id_seq’, (select max(id) from oauth_nonces));
select setval(’oauth_tokens_id_seq’, (select max(id) from oauth_tokens));
select setval(’redactions_id_seq’, (select max(id) from redactions));
select setval(’user_blocks_id_seq’, (select max(id) from user_blocks));
select setval(’user_roles_id_seq’, (select max(id) from user_roles));
select setval(’user_tokens_id_seq’, (select max(id) from user_tokens));
select setval(’users_id_seq’, (select max(id) from users));
" | psql -d myosmdb -U myosmuser -w -e -f -
19
20. Installation of tile server
A tile server serves the images that are displayed as the map. Each tile is 256pixels by
256pixels in size. When OSM Website is opened, the large map image is formed by place
these smaller tiles together.
A tile has three properties
• z: The zoom level. 0 being the least zoomed and 18 being the most zoomed.
• x,y: On each zoom level, x and y are the coordinates of a given tile.
A sample Url from OSM Website looks like http://a.tile.openstreetmap.org/0/0/0.png which
shows the whole world’s map.
We will set up our own tile server which can be accessed from http://a.tile.openeventmap.
tum.de/osm_tiles/z/x/y.png or http://b.tile.openeventmap.tum.de/osm_tiles/z/x/y.png or
http://c.tile.openeventmap.tum.de/osm_tiles/z/x/y.png.
Setting up /etc/hosts file
As root, add the following line at the bottom of the /etc/hosts file.
127.0.0.1 a.tile.openeventmap.tum.de b.tile.openeventmap.tum.de c.tile.openeventmap.tum.de
Once the tiles load, the configuration for our rails port needs to be pointed to our new tile
servers. Edit /opt/myosm/openstreetmap-website/vendor/assets/leaflet/leaflet.osm.js and
replace
http://{s}.tile.openstreetmap.org/{z}/{x}/{y}.png
with
http://{s}.tile.openeventmap.tum.de/osm_tiles/{z}/{x}/{y}.png
Setting up Mapnik and mod_tile
Generating and serving of tiles includes three pieces of software to work together. First, apache and mod_tile needs to be configured to receive requests in
http://{a,b,c}.tile.openeventmap.tum.de/osm_tiles/z/x/y.png format. Mod_Tile forwards
the request to renderd and renderd checks if a valid tile is in it’s cache. If found, it serves
the tile and if not found it asks Mapnik to generate the tile which is then served.
We will look into how to install these three pieces in the following section.
More detail information for installing mod_tile can be found in Mod_tile
Here are the commands executed.
20
21. $
$
$
$
$
$
$
$
$
cd /opt/myosm
git clone https://github.com/openstreetmap/mod_tile
cd mod_tile
./autogen.sh
./configure --with-apxs=/usr/sbin/apxs
make
sudo make install
sudo make install-mod_tile
sudo ldconfig
This will create the file /usr/lib/httpd/modules/mod_tile.so
$
$
$
$
$
$
$
$
$
cd /opt/myosm
git clone git://github.com/openstreetmap/mapnik-stylesheets.git
cd mapnik-stylesheets
./get-coastlines.sh
cd world_boundaries/
ln -s ne_110m_admin_0_boundary_lines_land.shp 110m_admin_0_boundary_lines_land.shp
ln -s ne_110m_admin_0_boundary_lines_land.dbf 110m_admin_0_boundary_lines_land.dbf
cd ..
/generate_xml.py --host localhost --dbname mymapnikdb --symbols
./symbols --world_boundaries world_boundaries --port 5432
--password myosmuser --user myosmuser
This will generate a file osm.xml which we shall use later for renderd.
Mapnik requires a PostGIS database. PostGIS provides spatial objects for the PostgreSQL
database allowing storage and query of information about location and mapping.
In the following section we create a database for mapnik, enable PostGIS and then load the
data in it.
$
$
$
$
$
$
$
mkdir /opt/myosm/postgis
cd /opt/myosm/postgis
createdb -U postgres -E UTF8 -O myosmuser mymapnikdb
psql -U postgres -d mymapnikdb -f /usr/share/postgresql/contrib/postgis-2.0/postgis.sql
wget https://raw.github.com/openstreetmap/osm2pgsql/master/900913.sql -O 900913.sql
psql -U postgres -d mymapnikdb -f ./900913.sql
osm2pgsql -U myosmuser -s -d mymapnikdb --slim oberbayern-latest.osm
We now configure apache so that mod_tile is configured to a special virtual host.
Uncomment the line from by removing the # at the start of the line in the file
/etc/httpd/conf/httpd.conf
Include conf/extra/httpd-vhosts.conf
Then add the following to the bottom of /etc/httpd/conf/extra/httpd-vhosts.conf. Original
source can be found here.
21
22. LoadModule tile_module modules/mod_tile.so
<VirtualHost *:80>
ServerName tile.openeventmap.tum.de
ServerAlias a.tile.openeventmap.tum.de b.tile.openeventmap.tum.de c.tile.openeventmap.tum.de
DocumentRoot /var/www/html
LoadTileConfigFile /etc/renderd.conf
ModTileRequestTimeout 3
ModTileMissingRequestTimeout 10
ModTileMaxLoadOld 2
ModTileMaxLoadMissing 5
ModTileRenderdSocketName /var/run/renderd.sock
ModTileCacheDurationMax 604800
ModTileCacheDurationDirty 900
ModTileCacheDurationMinimum 10800
ModTileCacheDurationMediumZoom 13 86400
ModTileCacheDurationLowZoom 9 518400
ModTileCacheLastModifiedFactor 0.20
ModTileEnableTileThrottling Off
ModTileThrottlingTiles 10000 1
ModTileThrottlingRenders 128 0.2
LogLevel debug
</VirtualHost>
Create a new file /etc/renderd.conf and add the following. Note this was modified by editing
/opt/myosm/mod_tile/renderd.conf.
[renderd]
socketname=/var/run/renderd.sock
num_threads=4
tile_dir=/var/lib/mod_tile
stats_file=/var/run/renderd.stats
[mapnik]
plugins_dir=/usr/lib64/mapnik/input
font_dir=/usr/lib64/mapnik/fonts
font_dir_recurse=1
[default]
URI=/osm_tiles/
XML=/opt/myosm/mapnik-stylesheets/osm.xml
HOST=localhost
Start apache and renderd
22
23. $ sudo systemctl start httpd.service
$ sudo /opt/myosm/mod_tile/renderd
To see if tiles are working, one can try loading the url http://a.tile.openeventmap.tum.de/
osm_tiles/0/0/0.png. A successful setup with load the world map like this.
Figure 7: 0th level tile
Loading renderd during boot
Whenever there is a server reboot, renderd daemon does not start automatically. Thus, it
needs to be made sure that renderd is running all the time by logging in to the server.
Synchronizing changes from ApiDB to MapnikDb
Any write modifications to the data happens in ApiDB. These changes need to be migrated
to the tiles as well, and as such we setup a synchronization mechanism between ApiDB and
MapnikDB.
The synchronization involes three parts.
1. Extract changes from ApiDB in regular intervals.
2. Migrate those changes to MapnikDB.
3. Remove the respective tiles from cache so fresh tiles are generated.
23
24. Extract changes from ApiDB in regular intervals Perform the following as root
$ sudo -u $USER mkdir -p /opt/myosm/apidb2mapnikdb/replicate
$ cd /opt/myosm/apidb2mapnikdb
$ sudo -u $USER osmosis --replicate-apidb user=myosmuser database=myosmdb
validateSchemaVersion=no --write-replication workingDirectory=replicate
This will create couple of files inside the replicate folder. At the time of first execution, it saves
the current state of ApiDb in a state.txt file. As the third command is executed repeatedly,
osm change files (.osc) files are generated and the contents of state.txt file updated.
$ tree replicate/
replicate/
’-- 000
’-- 000
’-- 000.state.txt
’-- replicate.lock
’-- state.txt
osmosis --replicate-apidb user=myosmuser database=myosmdb validateSchemaVersion=no
--write-replication workingDirectory=replicate
tree replicate/
replicate/
’-- 000
’-- 000
’-- 000.state.txt
’-- 001.osc.gz
’-- 001.state.txt
’-- replicate.lock
’-- state.txt
As you can see, new files have been added as the command is executed each time.
000/000/001.osc.gz file is our required change file. We take the changes from .osc.gz files
and pass it to mymapnik db. The process of migration is done through a minte-wise running
cronjob.
Save the following as a script in /opt/myosm/apidb2mapnikdb/update.sh
osmosis --replicate-apidb user=myosmuser database=myosmdb validateSchemaVersion=no
--write-replication workingDirectory=data
a=‘grep ’sequenceNumber’ replicate/state.txt | awk -F"=" {’print $2’}‘
a=$(printf %09d $a)
filename=replicate/${a:0:3}/${a:3:3}/${a:6:3}.osc.gz
echo $filename
rm expire.list
osm2pgsql -U myosmuser -s -d mymapnikdb --append $filename -e15 -o expire.list
cat expire.list | /usr/local/bin/render_expired --min-zoom=10 --touch-from=10
24
25. Also, make the update.sh file executable
$ chmod +x /opt/myosm/apidb2mapnikdb/update.sh
The script takes the last generated changefile (.osc.gz file) and uses osm2pgsql to update
mymapnikdb. The output of the command generates a list of tiles that needs to be updated
which is saved in expire.list. The list is then fed to render_expired so old files are deleted.
Add the following in the crontab using crontab -l as root
* * * * * (cd /opt/myosm/apidb2mapnikdb/ && ./update.sh &>> file.log)
25
26. Appendix B : EventsEditor JOSM Plugin Installation
Notes
JOSM can be downloaded freely from http://josm.openstreetmap.de/wiki/Download. The
recommended way is to download the jnlp file as it updates itself automatically everytime a
new version comes out. After http://josm.openstreetmap.de/download/josm.jnlp has been
downloaded, double clicking on josm.jnlp would start JOSM editor.
There are two parts to the plugin.
1. The event_editor.jar plugin source code.
2. The stylesheet files elemstyles.xml and event_editor.mapcss.
event_editor.jar installation instructions are available at JOSM manual plugin installation
page. Once the plugin is installed and enabled, the styles need to be installed.
elemstyles.xml and event_editor.mapcss can be placed anywhere. After starting JOSM,
these styles should be imported as shown in figure 8.
Figure 8: How to import style files in JOSM. 1) Go to Map Settings. 2) Click on Map
Paint Styles. 3) Click + to import new Stylesheet. 4) Import both elemstyles.xml and
event_editor.mapcss and enable them.
The styles allow any node with event=yes tag to show a small E marker at the right side.
The style sheet can be enabled as show in figure 9.
The user interface to add, edit or delete events reated tags are show in figure 10.
26
27. Figure 9: Using event_editor: 1) Either one of mapcss or xml file needs to be enabled. 2)
Any node with event=yes has extra E icon at the right. 3) When any OSM element is
clicked, user can click on menu: Presets > Event Add/Edit to start working with the plugin.
4) Newly added tags are shown on the section Properties.
27
28. Figure 10: EventEditor Starting Screenshot. 1) Option to mark current selected OSM Entity
to an event. 2) Button to create a new Event for the event entity. 3) GUI to enter more
detail tags for the event.
28
29. Appendix C : OpenEventMap Installation Notes
OpenEventEditor is based on Django 1.4 and python2. It uses pip to take care of necessary
dependencies and integrates seamlessly with apache and passenger module.
It is recommended that openeventmap is run inside a virtualenv.
$ cd /home/bibek/app/virtualenv/
$ virtualenv openeventmapdjango
$ source openeventmapdjango/bin/activate
Next step, we install all required dependencies for OpenEventMap to run
(openeventmapdjango)$ cd /opt/myosm/openeventmap
(openeventmapdjango)$ pip install -r requirements.txt
The database name, username and password are used exactly the same as in TheRailsPort
and edited in the file eventsvisualizer/settings.py.
We are serving static files from apache directly, so we collect all files inside the static_dir
folder.
(openeventmapdjango)$ ./manage.py collectstatic
Now comes the part to configure apache and passenger to serve our web application automatically.
The setup is achieved simply by a VirtualHost entry in
/etc/httpd/conf/extra/httpd-vhosts.conf.
<VirtualHost *:80>
ServerName www.openeventmap.tum.de
Alias /static/ /opt/myosm/openeventmap/static_dir/
DocumentRoot "/opt/myosm/openeventmap"
PassengerPython /home/bibek/app/virtualenv/openeventmapdjango/bin/python
PassengerAppRoot /opt/myosm/openeventmap
<Directory /opt/myosm/openeventmap>
Order allow,deny
Allow from all
</Directory>
</VirtualHost>
Accessing http://www.openeventmap.tum.de will load our webapplication.
29