SlideShare a Scribd company logo

Harvard Hypermap: An Open Source Framework for Making the World’s Geospatial Information more Accessible

Talk at the 2017 American Association of Geographers Annual Meeting April 5-9, 2017, Boston, MA US

1 of 22
Download to read offline
Harvard Hypermap:
An Open Source Framework for Making the World's
Geospatial Information more Accessible
http://worldmap.harvard.edu
Benjamin Lewis, Paolo Corti, Wendy Guan
American Association of Geographers
Boston, Massachusetts
April, 2017
Where are the geo-data?
Web Map Services
• A powerful, modern way to provide geo-data
via the Internet
• Distributed across thousands of servers
globally
• No central directory
• Tremendous variety in publishing service
formats (WMS, WFS, Esri REST, KML, etc.)
• Difficult to find, and difficult to use
WorldMap
OpenLayers,
Leaflet
Esri clients
Any map client
Distributed
Map
Services
Service
Crawler
(common
crawl)
Uptime
Checker
Map
Service
Registry
Service
Caching
Re-
projection
Service
API
User
submitted
services
Time
Miner
Solution: build a registry
Harvard HHypermap Highlights
• Supports comprehensive search
– Visualize distribution of results
– Support space and time
– Fast
• Facilitates sharing between registries
• Accepts user contributions
• Improves over time
• Software is open source
Spatial Data Infrastructure (SDI)
with Modern Search
Apache Lucene: a search engine library
ElasticSearchSolr
Spatial
facet
Temporal
facet
Topic
cloud
Result
highlight
Query
spellcheck
Relevance
tuning...

Recommended

Linked Data Notifications for RDF Streams
Linked Data Notifications for RDF StreamsLinked Data Notifications for RDF Streams
Linked Data Notifications for RDF StreamsJean-Paul Calbimonte
 
Using load tables to manage electronic resource records
Using load tables to manage electronic resource recordsUsing load tables to manage electronic resource records
Using load tables to manage electronic resource recordsNina Acosta
 
CAA 2014: Geosemantic Tools for Archaeological Research
CAA 2014: Geosemantic Tools for Archaeological ResearchCAA 2014: Geosemantic Tools for Archaeological Research
CAA 2014: Geosemantic Tools for Archaeological ResearchPaul Cripps
 
Connecting Stream Reasoners on the Web
Connecting Stream Reasoners on the WebConnecting Stream Reasoners on the Web
Connecting Stream Reasoners on the WebJean-Paul Calbimonte
 
Triplewave: a step towards RDF Stream Processing on the Web
Triplewave: a step towards RDF Stream Processing on the WebTriplewave: a step towards RDF Stream Processing on the Web
Triplewave: a step towards RDF Stream Processing on the WebDaniele Dell'Aglio
 
Query Rewriting in RDF Stream Processing
Query Rewriting in RDF Stream ProcessingQuery Rewriting in RDF Stream Processing
Query Rewriting in RDF Stream ProcessingJean-Paul Calbimonte
 
Professor Dame Wendy Hall - Saving the Web
Professor Dame Wendy Hall - Saving the WebProfessor Dame Wendy Hall - Saving the Web
Professor Dame Wendy Hall - Saving the WebRamine Tinati
 

More Related Content

Similar to Harvard Hypermap: An Open Source Framework for Making the World’s Geospatial Information more Accessible

WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BA...
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BA...WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BA...
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BA...Micah Altman
 
Big Process for Big Data @ PNNL, May 2013
Big Process for Big Data @ PNNL, May 2013Big Process for Big Data @ PNNL, May 2013
Big Process for Big Data @ PNNL, May 2013Ian Foster
 
Meetup070416 Presentations
Meetup070416 PresentationsMeetup070416 Presentations
Meetup070416 PresentationsAna Rebelo
 
Integrating Data for Archaeology
Integrating Data for ArchaeologyIntegrating Data for Archaeology
Integrating Data for Archaeologyariadnenetwork
 
Why do they call it Linked Data when they want to say...?
Why do they call it Linked Data when they want to say...?Why do they call it Linked Data when they want to say...?
Why do they call it Linked Data when they want to say...?Oscar Corcho
 
Exploratory querying of the Dutch GeoRegisters
Exploratory querying of the Dutch GeoRegistersExploratory querying of the Dutch GeoRegisters
Exploratory querying of the Dutch GeoRegistersStanislav Ronzhin
 
Library Mashups & APIs
Library Mashups & APIsLibrary Mashups & APIs
Library Mashups & APIslibrarywebchic
 
INSPIRE Hackathon Webinar Intro to Linked Data and Semantics
INSPIRE Hackathon Webinar   Intro to Linked Data and SemanticsINSPIRE Hackathon Webinar   Intro to Linked Data and Semantics
INSPIRE Hackathon Webinar Intro to Linked Data and Semanticsplan4all
 
One day workshop Linked Data and Semantic Web
One day workshop Linked Data and Semantic WebOne day workshop Linked Data and Semantic Web
One day workshop Linked Data and Semantic WebVictor de Boer
 
Big Data, Ingeniería de datos, y Data Lakes en AWS
Big Data, Ingeniería de datos, y Data Lakes en AWSBig Data, Ingeniería de datos, y Data Lakes en AWS
Big Data, Ingeniería de datos, y Data Lakes en AWSjavier ramirez
 
Globus Genomics: How Science-as-a-Service is Accelerating Discovery (BDT310) ...
Globus Genomics: How Science-as-a-Service is Accelerating Discovery (BDT310) ...Globus Genomics: How Science-as-a-Service is Accelerating Discovery (BDT310) ...
Globus Genomics: How Science-as-a-Service is Accelerating Discovery (BDT310) ...Amazon Web Services
 
Duraspace Hot Topics Series 6: Metadata and Repository Services
Duraspace Hot Topics Series 6: Metadata and Repository ServicesDuraspace Hot Topics Series 6: Metadata and Repository Services
Duraspace Hot Topics Series 6: Metadata and Repository ServicesMatthew Critchlow
 
Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016Domingo Suarez Torres
 
ARIADNE Registry - towards interoperability
ARIADNE Registry - towards interoperabilityARIADNE Registry - towards interoperability
ARIADNE Registry - towards interoperabilityariadnenetwork
 
How to Create the Google for Earth Data (XLDB 2015, Stanford)
How to Create the Google for Earth Data (XLDB 2015, Stanford)How to Create the Google for Earth Data (XLDB 2015, Stanford)
How to Create the Google for Earth Data (XLDB 2015, Stanford)Rainer Sternfeld
 
Making Temporal Search Central in a Spatial Data Infrastructure
Making Temporal Search Central in a Spatial Data InfrastructureMaking Temporal Search Central in a Spatial Data Infrastructure
Making Temporal Search Central in a Spatial Data InfrastructurePaolo Corti
 
Practical Machine Learning for Smarter Search with Solr and Spark
Practical Machine Learning for Smarter Search with Solr and SparkPractical Machine Learning for Smarter Search with Solr and Spark
Practical Machine Learning for Smarter Search with Solr and SparkJake Mannix
 
Practical Machine Learning for Smarter Search with Spark+Solr
Practical Machine Learning for Smarter Search with Spark+SolrPractical Machine Learning for Smarter Search with Spark+Solr
Practical Machine Learning for Smarter Search with Spark+SolrJake Mannix
 

Similar to Harvard Hypermap: An Open Source Framework for Making the World’s Geospatial Information more Accessible (20)

WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BA...
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BA...WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BA...
WORLDMAP: A SPATIAL INFRASTRUCTURE TO SUPPORT TEACHING AND RESEARCH (BROWN BA...
 
Big Process for Big Data @ PNNL, May 2013
Big Process for Big Data @ PNNL, May 2013Big Process for Big Data @ PNNL, May 2013
Big Process for Big Data @ PNNL, May 2013
 
Meetup070416 Presentations
Meetup070416 PresentationsMeetup070416 Presentations
Meetup070416 Presentations
 
Integrating Data for Archaeology
Integrating Data for ArchaeologyIntegrating Data for Archaeology
Integrating Data for Archaeology
 
Why do they call it Linked Data when they want to say...?
Why do they call it Linked Data when they want to say...?Why do they call it Linked Data when they want to say...?
Why do they call it Linked Data when they want to say...?
 
Exploratory querying of the Dutch GeoRegisters
Exploratory querying of the Dutch GeoRegistersExploratory querying of the Dutch GeoRegisters
Exploratory querying of the Dutch GeoRegisters
 
Library Mashups & APIs
Library Mashups & APIsLibrary Mashups & APIs
Library Mashups & APIs
 
INSPIRE Hackathon Webinar Intro to Linked Data and Semantics
INSPIRE Hackathon Webinar   Intro to Linked Data and SemanticsINSPIRE Hackathon Webinar   Intro to Linked Data and Semantics
INSPIRE Hackathon Webinar Intro to Linked Data and Semantics
 
One day workshop Linked Data and Semantic Web
One day workshop Linked Data and Semantic WebOne day workshop Linked Data and Semantic Web
One day workshop Linked Data and Semantic Web
 
Big Data, Ingeniería de datos, y Data Lakes en AWS
Big Data, Ingeniería de datos, y Data Lakes en AWSBig Data, Ingeniería de datos, y Data Lakes en AWS
Big Data, Ingeniería de datos, y Data Lakes en AWS
 
Internet content as research data
Internet content as research dataInternet content as research data
Internet content as research data
 
Globus Genomics: How Science-as-a-Service is Accelerating Discovery (BDT310) ...
Globus Genomics: How Science-as-a-Service is Accelerating Discovery (BDT310) ...Globus Genomics: How Science-as-a-Service is Accelerating Discovery (BDT310) ...
Globus Genomics: How Science-as-a-Service is Accelerating Discovery (BDT310) ...
 
Data-Driven Civic Innovation
Data-Driven Civic InnovationData-Driven Civic Innovation
Data-Driven Civic Innovation
 
Duraspace Hot Topics Series 6: Metadata and Repository Services
Duraspace Hot Topics Series 6: Metadata and Repository ServicesDuraspace Hot Topics Series 6: Metadata and Repository Services
Duraspace Hot Topics Series 6: Metadata and Repository Services
 
Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016Elasticsearch JVM-MX Meetup April 2016
Elasticsearch JVM-MX Meetup April 2016
 
ARIADNE Registry - towards interoperability
ARIADNE Registry - towards interoperabilityARIADNE Registry - towards interoperability
ARIADNE Registry - towards interoperability
 
How to Create the Google for Earth Data (XLDB 2015, Stanford)
How to Create the Google for Earth Data (XLDB 2015, Stanford)How to Create the Google for Earth Data (XLDB 2015, Stanford)
How to Create the Google for Earth Data (XLDB 2015, Stanford)
 
Making Temporal Search Central in a Spatial Data Infrastructure
Making Temporal Search Central in a Spatial Data InfrastructureMaking Temporal Search Central in a Spatial Data Infrastructure
Making Temporal Search Central in a Spatial Data Infrastructure
 
Practical Machine Learning for Smarter Search with Solr and Spark
Practical Machine Learning for Smarter Search with Solr and SparkPractical Machine Learning for Smarter Search with Solr and Spark
Practical Machine Learning for Smarter Search with Solr and Spark
 
Practical Machine Learning for Smarter Search with Spark+Solr
Practical Machine Learning for Smarter Search with Spark+SolrPractical Machine Learning for Smarter Search with Spark+Solr
Practical Machine Learning for Smarter Search with Spark+Solr
 

More from Paolo Corti

State of GeoNode 2019
State of GeoNode 2019State of GeoNode 2019
State of GeoNode 2019Paolo Corti
 
Maintaining spatial data infrastructures (SDIs) using distributed task queues
Maintaining spatial data infrastructures (SDIs) using distributed task queuesMaintaining spatial data infrastructures (SDIs) using distributed task queues
Maintaining spatial data infrastructures (SDIs) using distributed task queuesPaolo Corti
 
Status of WorldMap, 2016
Status of WorldMap, 2016Status of WorldMap, 2016
Status of WorldMap, 2016Paolo Corti
 
GeoNode per il Supporto alle Emergenze Umanitarie
GeoNode per il Supporto alle Emergenze UmanitarieGeoNode per il Supporto alle Emergenze Umanitarie
GeoNode per il Supporto alle Emergenze UmanitariePaolo Corti
 
Implementing an Open Source Spatiotemporal Search Platform for Spatial Data I...
Implementing an Open Source Spatiotemporal Search Platform for Spatial Data I...Implementing an Open Source Spatiotemporal Search Platform for Spatial Data I...
Implementing an Open Source Spatiotemporal Search Platform for Spatial Data I...Paolo Corti
 
Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...
Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...
Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...Paolo Corti
 
GeoNode intro and demo
GeoNode intro and demoGeoNode intro and demo
GeoNode intro and demoPaolo Corti
 
GeoNode for Humanitarian Crisis and Risk Reduction
GeoNode for Humanitarian Crisis and Risk ReductionGeoNode for Humanitarian Crisis and Risk Reduction
GeoNode for Humanitarian Crisis and Risk ReductionPaolo Corti
 
L'utilizzo di software fee and open source nello European Forest Fire Informa...
L'utilizzo di software fee and open source nello European Forest Fire Informa...L'utilizzo di software fee and open source nello European Forest Fire Informa...
L'utilizzo di software fee and open source nello European Forest Fire Informa...Paolo Corti
 
Fire news management in the context of the European Forest Fire Information S...
Fire news management in the context of the European Forest Fire Information S...Fire news management in the context of the European Forest Fire Information S...
Fire news management in the context of the European Forest Fire Information S...Paolo Corti
 
Developing Geospatial software with Python, Part 1
Developing Geospatial software with Python, Part 1Developing Geospatial software with Python, Part 1
Developing Geospatial software with Python, Part 1Paolo Corti
 

More from Paolo Corti (12)

State of GeoNode 2019
State of GeoNode 2019State of GeoNode 2019
State of GeoNode 2019
 
Maintaining spatial data infrastructures (SDIs) using distributed task queues
Maintaining spatial data infrastructures (SDIs) using distributed task queuesMaintaining spatial data infrastructures (SDIs) using distributed task queues
Maintaining spatial data infrastructures (SDIs) using distributed task queues
 
Status of WorldMap, 2016
Status of WorldMap, 2016Status of WorldMap, 2016
Status of WorldMap, 2016
 
GeoNode per il Supporto alle Emergenze Umanitarie
GeoNode per il Supporto alle Emergenze UmanitarieGeoNode per il Supporto alle Emergenze Umanitarie
GeoNode per il Supporto alle Emergenze Umanitarie
 
Implementing an Open Source Spatiotemporal Search Platform for Spatial Data I...
Implementing an Open Source Spatiotemporal Search Platform for Spatial Data I...Implementing an Open Source Spatiotemporal Search Platform for Spatial Data I...
Implementing an Open Source Spatiotemporal Search Platform for Spatial Data I...
 
Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...
Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...
Building an Open Source, Real-Time, Billion Object Spatio-Temporal Search Pla...
 
GeoNode intro and demo
GeoNode intro and demoGeoNode intro and demo
GeoNode intro and demo
 
GeoNode for Humanitarian Crisis and Risk Reduction
GeoNode for Humanitarian Crisis and Risk ReductionGeoNode for Humanitarian Crisis and Risk Reduction
GeoNode for Humanitarian Crisis and Risk Reduction
 
Geonode 2.0
Geonode 2.0Geonode 2.0
Geonode 2.0
 
L'utilizzo di software fee and open source nello European Forest Fire Informa...
L'utilizzo di software fee and open source nello European Forest Fire Informa...L'utilizzo di software fee and open source nello European Forest Fire Informa...
L'utilizzo di software fee and open source nello European Forest Fire Informa...
 
Fire news management in the context of the European Forest Fire Information S...
Fire news management in the context of the European Forest Fire Information S...Fire news management in the context of the European Forest Fire Information S...
Fire news management in the context of the European Forest Fire Information S...
 
Developing Geospatial software with Python, Part 1
Developing Geospatial software with Python, Part 1Developing Geospatial software with Python, Part 1
Developing Geospatial software with Python, Part 1
 

Recently uploaded

Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)Jeffrey Haguewood
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaSGlobus
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Dmitry Zinoviev
 
Managing multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerManaging multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerThierry Gayet
 
Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesGlobus
 
Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusGlobus
 
killingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdfkillingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdfssuser82c38d
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with GlobusGlobus
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureHironori Washizaki
 
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...
CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...syedfaisal759877
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارsohilww
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentThierry Gayet
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleAmir Moghimi
 
From Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesFrom Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesAnisha Agarwal
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxMindInventory
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfssuser82c38d
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flinkconfluent
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration TopicsGlobus
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowNaoki (Neo) SATO
 

Recently uploaded (20)

Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)Automation for Bonterra Impact Management (fka Apricot)
Automation for Bonterra Impact Management (fka Apricot)
 
Building Research Applications with Globus PaaS
Building Research Applications with Globus PaaSBuilding Research Applications with Globus PaaS
Building Research Applications with Globus PaaS
 
Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)Machine Learning Basics for Dummies (no math!)
Machine Learning Basics for Dummies (no math!)
 
Managing multicast/igmp stream on Docker
Managing multicast/igmp stream on DockerManaging multicast/igmp stream on Docker
Managing multicast/igmp stream on Docker
 
Reliable, Remote Computation at All Scales
Reliable, Remote Computation at All ScalesReliable, Remote Computation at All Scales
Reliable, Remote Computation at All Scales
 
Best Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using GlobusBest Practices for Data Sharing Using Globus
Best Practices for Data Sharing Using Globus
 
2024 Trends Transforming Enterprise Resource Planning
2024 Trends Transforming Enterprise Resource Planning2024 Trends Transforming Enterprise Resource Planning
2024 Trends Transforming Enterprise Resource Planning
 
killingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdfkillingcamp longest common subsequence.pdf
killingcamp longest common subsequence.pdf
 
Introduction to Research Automation with Globus
Introduction to Research Automation with GlobusIntroduction to Research Automation with Globus
Introduction to Research Automation with Globus
 
Joseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about ArchitectureJoseph Yoder : Being Agile about Architecture
Joseph Yoder : Being Agile about Architecture
 
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...
CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...CSS Notes in PDF, Easy to understand. For beginner to advanced.              ...
CSS Notes in PDF, Easy to understand. For beginner to advanced. ...
 
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزارانتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
انتزاع و هزینه - انتزاع و تاثیرات آن در توسعه و نگهداری نرم‌افزار
 
Passbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managmentPassbolt Introduction and Usage for secret managment
Passbolt Introduction and Usage for secret managment
 
How AI is preventing account fraud at web scale
How AI is preventing account fraud at web scaleHow AI is preventing account fraud at web scale
How AI is preventing account fraud at web scale
 
From Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT ServicesFrom Software Development To Branding through Digital Marketing, IT Services
From Software Development To Branding through Digital Marketing, IT Services
 
Role of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptxRole of DevOps in SaaS product Development.pdf.pptx
Role of DevOps in SaaS product Development.pdf.pptx
 
killing camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdfkilling camp 주차장 나누기-2 topology sort.pdf
killing camp 주차장 나누기-2 topology sort.pdf
 
Workshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con FlinkWorkshop híbrido: Stream Processing con Flink
Workshop híbrido: Stream Processing con Flink
 
Advanced Globus System Administration Topics
Advanced Globus System Administration TopicsAdvanced Globus System Administration Topics
Advanced Globus System Administration Topics
 
LLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flowLLMOps with Azure Machine Learning prompt flow
LLMOps with Azure Machine Learning prompt flow
 

Harvard Hypermap: An Open Source Framework for Making the World’s Geospatial Information more Accessible

  • 1. Harvard Hypermap: An Open Source Framework for Making the World's Geospatial Information more Accessible http://worldmap.harvard.edu Benjamin Lewis, Paolo Corti, Wendy Guan American Association of Geographers Boston, Massachusetts April, 2017
  • 2. Where are the geo-data?
  • 3. Web Map Services • A powerful, modern way to provide geo-data via the Internet • Distributed across thousands of servers globally • No central directory • Tremendous variety in publishing service formats (WMS, WFS, Esri REST, KML, etc.) • Difficult to find, and difficult to use
  • 4. WorldMap OpenLayers, Leaflet Esri clients Any map client Distributed Map Services Service Crawler (common crawl) Uptime Checker Map Service Registry Service Caching Re- projection Service API User submitted services Time Miner Solution: build a registry
  • 5. Harvard HHypermap Highlights • Supports comprehensive search – Visualize distribution of results – Support space and time – Fast • Facilitates sharing between registries • Accepts user contributions • Improves over time • Software is open source
  • 6. Spatial Data Infrastructure (SDI) with Modern Search Apache Lucene: a search engine library ElasticSearchSolr Spatial facet Temporal facet Topic cloud Result highlight Query spellcheck Relevance tuning...
  • 8. HHypermap as WorldMap’s search engine • Provides access to about 100,000 layers • 25,000 are from WorldMap itself, the rest are on remote servers • Accesses 13,000 remote map services • A fraction of existing online map services • More are being added
  • 9. Current search on WorldMap • Supports faceted search by – Geographic space (heatmap for any number of results) – Time (temporal histogram) – Subject matter (topic clouds) – Source (data owner, publisher) • and potentially other dimensions
  • 10. Video demo (3 minutes): https://vimeo.com/164167343 (narrated by Ben Lewis)
  • 11. HHypermap Architecture Built on open source software: • Celery • RabbitMQ • Django • Lucene – Solr – Elasticsearch • MapProxy • Memcached • OWSLib • PostgreSQL • PostGIS • pycsw
  • 12. Automated Gathering of Map Service Endpoints to Harvest • Search web for signatures using the Common Crawl (CC) archive • Store as compressed Web Archive (WARC) formatted files on Amazon S3. • Employ multiple machines to process the data in parallel on Amazon EC2 • Use Hadoop/YARN framework, execute Map/Reduce functions to aggregate information about URLs to spatial assets • Collect URLs for later processing and harvesting
  • 13. Signatures for Hadoop search • OGC Services – Look for "?request getcapabilities" and not “test” in the href URL • ESRI Rest Services – Look for “/arcgis/rest/services” in the target-DOMAIN-URI of the WARC Response Header text • KML or KMZ files – Look for an href URL ending in .kml or .kmz files • Compressed shapefiles – Look for “shape” or “shp” and string ending with “.zip” in the href URL • Tile Servers – Look for “tile” or “tiles” and string ending with “.png” in the href URL
  • 14. User Submit Service Endpoint to Harvest
  • 15. Time Miner – to enrich metadata • Temporal metadata for geospatial datasets is often weak. • In a crowd-sourced data repository, data creators and contributors often do not create detailed metadata. • Many data sets have temporal properties, but time is often ambiguously defined, mentioned as unstructured text in the title, abstract, and elsewhere. • Time is often not referred to using a standard date/time format such as ISO 8601, but as descriptive text.
  • 16. Time Miner Logic Implementation 1. Look for date in the date range (lower) section of the metadata and choose the earlier date. (Date: from Metadata) 2. If there is no #1 above, look in top date in metadata but only use it if it is 2010 or earlier. (Date: from Metadata) 3. If there is no #2 above, look for 4 digit numbers in title first, then abstract, which are less than or equal to 2016 (present year) (Date: Detected) 4. If there IS a date in #3 above, check to see whether there is a CE or AD or BCE or BC after it and apply math accordingly (Date: Detected) 5. If there is no #3 above, look for 1, 2, or 3 digit numbers with associated CE, AD, BCE, BC, and apply math accordingly (Date: Detected)
  • 17. Time Miner: in addition to dates, recognize some periods • ca. 2100-1600 BCE Xia, Hsia • ca. 1600-1050 BCE Shang • ca. 1046-256 BCE Zhou, Chou • 221-206 BCE Qin, Ch'in • 206 BCE-220 CE Han • 581-618 CE Sui • 618-906 Tang, T'ang • 960-1279 Song, Sung • 1279-1368 Yuan • 1368-1644 Ming • 1644-1912 Qing, Ch'ing Source: http://afe.easia.columbia.edu/timelines/china_timeline.htm
  • 18. Each layer monitored for uptime
  • 19. Scalable to support virtually any number of datasets Another project the “Billion Object Platform” (BOP) demonstrates how robust this platform is! http://terranodo.io/angular-search/#/search
  • 20. Some examples of catalogues that could be brought together • ArcGIS Open Data – Esri collection of 44,000 open datasets and growing. • Geodata.gov, Geoplatform.gov – The U.S. Federal government has built a data sharing platform for U.S. data using CKAN software and ArcGIS Online. • INSPIRE Geoportal – Spatial data portal of the European Commission. • GEOSS registry – Group on Earth Observations registry of 850 map service collections. • Geopole.org – CSW catalogue service providing access to 400,000 layers. • Geoblacklight - Platform developed by Stanford and other universities to provide fast search access to geospatial library holdings. • OpenGeoPortal – Platform developed by Tufts and other universities to provide fast search access to geospatial library holdings. • Geonetwork – Geospatial catalogue maintained by the Food and Agriculture Organization of the United Nations. • Spatineo.com – Commercial service which is currently monitoring 40,000 web services containing 899,000 layers. • New York Public Library Collection • David Rumsey Collection • Many CKAN portals • Many Thredds servers
  • 21. Help us improve the system • Try it out. If you can’t find services you know are out there, submit them to us and we will add them. • If you would like to harvest metadata from other systems, write a connector. We will provide guidance. • If you would like to bring such search into other applications besides WorldMap, write a client. We will help. • If you would like to set up your own HHypermap registry instance, the code is available. • If you have other features you would like, let us know.
  • 22. Harvard Hypermap: An Open Source Framework for Making the World's Geospatial Information more Accessible http://worldmap.harvard.edu Benjamin Lewis, Paolo Corti, Wendy Guan