SlideShare a Scribd company logo
1 of 22
The Beauty of Mapping
Big Data
Stoimen Popov
R&D Lead, Product Innovation Team, HERE IoT| Dec 05, 2016
HERE is the Open Location Platform company
• Provides mapping services and location intelligence
across the automotive, enterprise and internet industries
• Employs 7,000 people in 56 countries
• Produces maps for every country on earth
• Enables four out of five in-car navigation systems in
North America and Europe
• Enables mobile, web and enterprise solutions for global
industry leaders
Agenda
01. Data Lens
01.1 Cloud Storage
01.2 Rest API & Query Language
01.3 JS API
02. Server-Side Clustering
03. Data Lens Heat Maps
03.1 Averaged Value
03.2 Alpha Mask by Density
03.3 Value Based Heat Map
01
Data Lens
Data Lens is a cloud-based API toolkit for developing
sophisticated visualizations of geographically
referenced data, accessible in a web browser.
Data is queried via a REST API from the Data Lens
cloud, and transformed and aggregated to provide
input to the JavaScript API, which renders the
visualization.
Cloud Storage
• Data storage
• Integration with HERE account
• Data enrichment
Data Lens REST API
• Authentication
• Data Upload
• Datasets
• Queries
• Query Language
• Access Management
• Data Reprojection
• ...
Data Lens Query Language
• JSON formatted queries
• Similar to prepared statements in SQL for later execution
• Only dataset owners can create queries for a dataset
• Protect sensitive datasets
Presentation title | Month 00, 20169 © 2016 HERE | HERE Internal Use Only
Data Lens
Query Language
Data Lens JS API
• Data Lens JavaScript API is a module of HERE Maps API for JavaScript and
connects it to the Data Lens REST API
• Provides data-driven styling of data on a map
• Solves non-trivial tasks like tiling, caching and rasterizing data
Presentation title | Month 00, 201611 © 2016 HERE | HERE Internal Use Only
Data Lens Heat Map
Dataset with 11M points of data about taxi rides in NY City
Presentation title | Month 00, 201612 © 2016 HERE | HERE Internal Use Only
Data Lens Street Shapes
The data is geo-enriched to street geometry in NY City
Presentation title | Month 00, 201613 © 2016 HERE | HERE Internal Use Only
Data Lens ZIP Shapes
The data is geo-enriched to ZIP code
boundaries in NY City
Presentation title | Month 00, 201614 © 2016 HERE | HERE Internal Use Only
Data Lens Buildings
The data is geo-enriched to building geometries
in NY City
02
Server-Side Clustering
How to deal with BIG DATA on the server?
Data Tiling & Grouping in Pixel Space
• Data tiling reduces the amount of data received by the
client
• Data Lens groups the data points per tile pixel
03
Data Lens Heat Maps
• Averaged value
• Alpha mask by density
• Value-based heat map
Presentation title | Month 00, 201618 © 2016 HERE | HERE Internal Use Only
Averaged heat map
Presentation title | Month 00, 201619 © 2016 HERE | HERE Internal Use Only
Averaged heat map alone, and
with an applied alpha mask
Big Data can be visualized in many ways …
Heat Maps Server-Side Clustering Hybrid Clustering
(Server-side and client-side)
https://developer.here.com
Docs & API Reference Tech Examples Industry Examples
 Develop
 Code Examples
 Data Lens APIs
• Detailed Story
• JS/HTML Code
• Query Definitions
• Styles & UI
 Develop
 Code Examples
 Data Lens APIs
• Data-Driven Styling
• Server-Side Clustering
• Hybrid Clustering
• …and more!
 Develop
 Data Lens
• Getting Started
• Tutorial
• Developer Guide
• API Reference
The Beauty of Mapping Big Data

More Related Content

What's hot

Creative Ways to Leverage Operational Data
Creative Ways to Leverage Operational DataCreative Ways to Leverage Operational Data
Creative Ways to Leverage Operational DataCartegraph
 
HERE - Esri UK Annual Conference 2016
HERE - Esri UK Annual Conference 2016HERE - Esri UK Annual Conference 2016
HERE - Esri UK Annual Conference 2016Esri UK
 
Smart Mapping - How to Create Stunning Maps - Esri UK Annual Conference 2016
Smart Mapping - How to Create Stunning Maps - Esri UK Annual Conference 2016Smart Mapping - How to Create Stunning Maps - Esri UK Annual Conference 2016
Smart Mapping - How to Create Stunning Maps - Esri UK Annual Conference 2016Esri UK
 
Getting to Know ArcGIS Pro
Getting to Know ArcGIS ProGetting to Know ArcGIS Pro
Getting to Know ArcGIS ProEsri UK
 
Esri Scotland Conf 2016 Web AppBuilder
Esri Scotland Conf 2016   Web AppBuilderEsri Scotland Conf 2016   Web AppBuilder
Esri Scotland Conf 2016 Web AppBuilderEsri UK
 
Distilled Power BI Updates for April 2016
Distilled Power BI Updates for April 2016Distilled Power BI Updates for April 2016
Distilled Power BI Updates for April 2016Jen Stirrup
 
Imagery and beyond - BK 2016
Imagery and beyond - BK 2016Imagery and beyond - BK 2016
Imagery and beyond - BK 2016Geodata AS
 
Esri Scotland Conf 2016 Glasgow City Council
Esri Scotland Conf 2016   Glasgow City CouncilEsri Scotland Conf 2016   Glasgow City Council
Esri Scotland Conf 2016 Glasgow City CouncilEsri UK
 
Esri Scotland Conf 2016 City of Edinburgh
Esri Scotland Conf 2016   City of EdinburghEsri Scotland Conf 2016   City of Edinburgh
Esri Scotland Conf 2016 City of EdinburghEsri UK
 
BIM - Esri UK Annual Conference 2016
BIM - Esri UK Annual Conference 2016BIM - Esri UK Annual Conference 2016
BIM - Esri UK Annual Conference 2016Esri UK
 
ArcGIS Data Interoperability: Tips for LiDAR, 3D, and BIM
ArcGIS Data Interoperability: Tips for LiDAR, 3D, and BIMArcGIS Data Interoperability: Tips for LiDAR, 3D, and BIM
ArcGIS Data Interoperability: Tips for LiDAR, 3D, and BIMSafe Software
 
Geolocation: Improving our BI solutions with SharePoint
Geolocation: Improving our BI solutions with SharePointGeolocation: Improving our BI solutions with SharePoint
Geolocation: Improving our BI solutions with SharePointRuben Pertusa Lopez
 
Severn Trent Water - Esri UK Annual Conference 2016
Severn Trent Water - Esri UK Annual Conference 2016Severn Trent Water - Esri UK Annual Conference 2016
Severn Trent Water - Esri UK Annual Conference 2016Esri UK
 
Esri UK - Annual Conference 2016 Transport for london
Esri UK - Annual Conference 2016 Transport for londonEsri UK - Annual Conference 2016 Transport for london
Esri UK - Annual Conference 2016 Transport for londonEsri UK
 
CAD and GIS: Connecting Two Worlds
CAD and GIS: Connecting Two WorldsCAD and GIS: Connecting Two Worlds
CAD and GIS: Connecting Two WorldsSafe Software
 
Forestry Commission - Smart Collaboration - Esri UK Annual Conference 2017
Forestry Commission - Smart Collaboration - Esri UK Annual Conference 2017Forestry Commission - Smart Collaboration - Esri UK Annual Conference 2017
Forestry Commission - Smart Collaboration - Esri UK Annual Conference 2017Esri UK
 
Actualización de Kibana y Geo: Canvas, Elastic Maps y muchas más características
Actualización de Kibana y Geo: Canvas, Elastic Maps y muchas más característicasActualización de Kibana y Geo: Canvas, Elastic Maps y muchas más características
Actualización de Kibana y Geo: Canvas, Elastic Maps y muchas más característicasElasticsearch
 

What's hot (20)

Creative Ways to Leverage Operational Data
Creative Ways to Leverage Operational DataCreative Ways to Leverage Operational Data
Creative Ways to Leverage Operational Data
 
HERE - Esri UK Annual Conference 2016
HERE - Esri UK Annual Conference 2016HERE - Esri UK Annual Conference 2016
HERE - Esri UK Annual Conference 2016
 
Smart Mapping - How to Create Stunning Maps - Esri UK Annual Conference 2016
Smart Mapping - How to Create Stunning Maps - Esri UK Annual Conference 2016Smart Mapping - How to Create Stunning Maps - Esri UK Annual Conference 2016
Smart Mapping - How to Create Stunning Maps - Esri UK Annual Conference 2016
 
Getting to Know ArcGIS Pro
Getting to Know ArcGIS ProGetting to Know ArcGIS Pro
Getting to Know ArcGIS Pro
 
Esri Scotland Conf 2016 Web AppBuilder
Esri Scotland Conf 2016   Web AppBuilderEsri Scotland Conf 2016   Web AppBuilder
Esri Scotland Conf 2016 Web AppBuilder
 
Distilled Power BI Updates for April 2016
Distilled Power BI Updates for April 2016Distilled Power BI Updates for April 2016
Distilled Power BI Updates for April 2016
 
Inventory3D v0.5
Inventory3D v0.5Inventory3D v0.5
Inventory3D v0.5
 
Imagery and beyond - BK 2016
Imagery and beyond - BK 2016Imagery and beyond - BK 2016
Imagery and beyond - BK 2016
 
Esri Scotland Conf 2016 Glasgow City Council
Esri Scotland Conf 2016   Glasgow City CouncilEsri Scotland Conf 2016   Glasgow City Council
Esri Scotland Conf 2016 Glasgow City Council
 
Esri Scotland Conf 2016 City of Edinburgh
Esri Scotland Conf 2016   City of EdinburghEsri Scotland Conf 2016   City of Edinburgh
Esri Scotland Conf 2016 City of Edinburgh
 
BIM - Esri UK Annual Conference 2016
BIM - Esri UK Annual Conference 2016BIM - Esri UK Annual Conference 2016
BIM - Esri UK Annual Conference 2016
 
ArcGIS Data Interoperability: Tips for LiDAR, 3D, and BIM
ArcGIS Data Interoperability: Tips for LiDAR, 3D, and BIMArcGIS Data Interoperability: Tips for LiDAR, 3D, and BIM
ArcGIS Data Interoperability: Tips for LiDAR, 3D, and BIM
 
Geolocation: Improving our BI solutions with SharePoint
Geolocation: Improving our BI solutions with SharePointGeolocation: Improving our BI solutions with SharePoint
Geolocation: Improving our BI solutions with SharePoint
 
Severn Trent Water - Esri UK Annual Conference 2016
Severn Trent Water - Esri UK Annual Conference 2016Severn Trent Water - Esri UK Annual Conference 2016
Severn Trent Water - Esri UK Annual Conference 2016
 
Esri UK - Annual Conference 2016 Transport for london
Esri UK - Annual Conference 2016 Transport for londonEsri UK - Annual Conference 2016 Transport for london
Esri UK - Annual Conference 2016 Transport for london
 
CAD and GIS: Connecting Two Worlds
CAD and GIS: Connecting Two WorldsCAD and GIS: Connecting Two Worlds
CAD and GIS: Connecting Two Worlds
 
Forestry Commission - Smart Collaboration - Esri UK Annual Conference 2017
Forestry Commission - Smart Collaboration - Esri UK Annual Conference 2017Forestry Commission - Smart Collaboration - Esri UK Annual Conference 2017
Forestry Commission - Smart Collaboration - Esri UK Annual Conference 2017
 
Fieldtrip GB
Fieldtrip GBFieldtrip GB
Fieldtrip GB
 
Actualización de Kibana y Geo: Canvas, Elastic Maps y muchas más características
Actualización de Kibana y Geo: Canvas, Elastic Maps y muchas más característicasActualización de Kibana y Geo: Canvas, Elastic Maps y muchas más características
Actualización de Kibana y Geo: Canvas, Elastic Maps y muchas más características
 
Analysing Web GIS apps
Analysing Web GIS appsAnalysing Web GIS apps
Analysing Web GIS apps
 

Viewers also liked

Big Data as PaaS in Enterprises
Big Data as PaaS in EnterprisesBig Data as PaaS in Enterprises
Big Data as PaaS in EnterprisesPankaj Khattar
 
Big Data in The Cloud: Architecting a Better Platform
Big Data in The Cloud: Architecting a Better PlatformBig Data in The Cloud: Architecting a Better Platform
Big Data in The Cloud: Architecting a Better PlatformAmazon Web Services
 
Analytics on AWS:Structured, Unstructured and Streaming
Analytics on AWS:Structured, Unstructured and StreamingAnalytics on AWS:Structured, Unstructured and Streaming
Analytics on AWS:Structured, Unstructured and StreamingAmazon Web Services
 
Anitha_Resume_BigData
Anitha_Resume_BigDataAnitha_Resume_BigData
Anitha_Resume_BigDataAnitha Bade
 

Viewers also liked (6)

Slideshow
SlideshowSlideshow
Slideshow
 
Big Data as PaaS in Enterprises
Big Data as PaaS in EnterprisesBig Data as PaaS in Enterprises
Big Data as PaaS in Enterprises
 
Big Data and Analytics
Big Data and AnalyticsBig Data and Analytics
Big Data and Analytics
 
Big Data in The Cloud: Architecting a Better Platform
Big Data in The Cloud: Architecting a Better PlatformBig Data in The Cloud: Architecting a Better Platform
Big Data in The Cloud: Architecting a Better Platform
 
Analytics on AWS:Structured, Unstructured and Streaming
Analytics on AWS:Structured, Unstructured and StreamingAnalytics on AWS:Structured, Unstructured and Streaming
Analytics on AWS:Structured, Unstructured and Streaming
 
Anitha_Resume_BigData
Anitha_Resume_BigDataAnitha_Resume_BigData
Anitha_Resume_BigData
 

Similar to The Beauty of Mapping Big Data

3 Benefits of Multi-Temperature Data Management for Data Analytics
3 Benefits of Multi-Temperature Data Management for Data Analytics3 Benefits of Multi-Temperature Data Management for Data Analytics
3 Benefits of Multi-Temperature Data Management for Data AnalyticsMapR Technologies
 
Big Data LDN 2017: Data Integration & Big Data Management
Big Data LDN 2017: Data Integration & Big Data ManagementBig Data LDN 2017: Data Integration & Big Data Management
Big Data LDN 2017: Data Integration & Big Data ManagementMatt Stubbs
 
Our road to microservices - or how we learned to love async events
Our road to microservices - or how we learned to love async eventsOur road to microservices - or how we learned to love async events
Our road to microservices - or how we learned to love async eventsThomas Bøgh Fangel
 
Top Trends in Building Data Lakes for Machine Learning and AI
Top Trends in Building Data Lakes for Machine Learning and AI Top Trends in Building Data Lakes for Machine Learning and AI
Top Trends in Building Data Lakes for Machine Learning and AI Holden Ackerman
 
Verizon Centralizes Data into a Data Lake in Real Time for Analytics
Verizon Centralizes Data into a Data Lake in Real Time for AnalyticsVerizon Centralizes Data into a Data Lake in Real Time for Analytics
Verizon Centralizes Data into a Data Lake in Real Time for AnalyticsHortonworks
 
Big Data LDN 2017: How to leverage the cloud for Business Solutions
Big Data LDN 2017: How to leverage the cloud for Business SolutionsBig Data LDN 2017: How to leverage the cloud for Business Solutions
Big Data LDN 2017: How to leverage the cloud for Business SolutionsMatt Stubbs
 
An Introduction to the MapR Converged Data Platform
An Introduction to the MapR Converged Data PlatformAn Introduction to the MapR Converged Data Platform
An Introduction to the MapR Converged Data PlatformMapR Technologies
 
Converging your data landscape
Converging your data landscapeConverging your data landscape
Converging your data landscapeMapR Technologies
 
Building and Successfully Selling ISV Solutions with AWS Partner-Summit-Singa...
Building and Successfully Selling ISV Solutions with AWS Partner-Summit-Singa...Building and Successfully Selling ISV Solutions with AWS Partner-Summit-Singa...
Building and Successfully Selling ISV Solutions with AWS Partner-Summit-Singa...Amazon Web Services
 
Data & Analytics with CIS & Microsoft Platforms
Data & Analytics with CIS & Microsoft PlatformsData & Analytics with CIS & Microsoft Platforms
Data & Analytics with CIS & Microsoft PlatformsSonata Software
 
Encuentro Aporta 2016 - Mesa 2 - Miguel Arias
Encuentro Aporta 2016 - Mesa 2 - Miguel AriasEncuentro Aporta 2016 - Mesa 2 - Miguel Arias
Encuentro Aporta 2016 - Mesa 2 - Miguel AriasDatos.gob.es
 
Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...
Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...
Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...Amazon Web Services
 
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)Thoughtworks
 
Big and fast data strategy 2017 jr
Big and fast data strategy 2017 jrBig and fast data strategy 2017 jr
Big and fast data strategy 2017 jrJonathan Raspaud
 
Sustainable Organization and the Impact of Cloud Transformation on Business O...
Sustainable Organization and the Impact of Cloud Transformation on Business O...Sustainable Organization and the Impact of Cloud Transformation on Business O...
Sustainable Organization and the Impact of Cloud Transformation on Business O...Amazon Web Services
 
Integrating Semantic Web in the Real World: A Journey between Two Cities
Integrating Semantic Web in the Real World: A Journey between Two Cities Integrating Semantic Web in the Real World: A Journey between Two Cities
Integrating Semantic Web in the Real World: A Journey between Two Cities Juan Sequeda
 
AWS Webcast - Open Data on AWS – An Introduction
AWS Webcast - Open Data on AWS – An Introduction  AWS Webcast - Open Data on AWS – An Introduction
AWS Webcast - Open Data on AWS – An Introduction Amazon Web Services
 
AWS Community Day Nordics 2018 - Saku Vaittinen (VR): Data driven public tran...
AWS Community Day Nordics 2018 - Saku Vaittinen (VR): Data driven public tran...AWS Community Day Nordics 2018 - Saku Vaittinen (VR): Data driven public tran...
AWS Community Day Nordics 2018 - Saku Vaittinen (VR): Data driven public tran...Rolf Koski
 
Predicting Banking Customer Needs with an Agile Approach to Analytics in the ...
Predicting Banking Customer Needs with an Agile Approach to Analytics in the ...Predicting Banking Customer Needs with an Agile Approach to Analytics in the ...
Predicting Banking Customer Needs with an Agile Approach to Analytics in the ...Databricks
 
The Data & Analytics Journey – Why it’s more attainable for your company than...
The Data & Analytics Journey – Why it’s more attainable for your company than...The Data & Analytics Journey – Why it’s more attainable for your company than...
The Data & Analytics Journey – Why it’s more attainable for your company than...LetsConnect
 

Similar to The Beauty of Mapping Big Data (20)

3 Benefits of Multi-Temperature Data Management for Data Analytics
3 Benefits of Multi-Temperature Data Management for Data Analytics3 Benefits of Multi-Temperature Data Management for Data Analytics
3 Benefits of Multi-Temperature Data Management for Data Analytics
 
Big Data LDN 2017: Data Integration & Big Data Management
Big Data LDN 2017: Data Integration & Big Data ManagementBig Data LDN 2017: Data Integration & Big Data Management
Big Data LDN 2017: Data Integration & Big Data Management
 
Our road to microservices - or how we learned to love async events
Our road to microservices - or how we learned to love async eventsOur road to microservices - or how we learned to love async events
Our road to microservices - or how we learned to love async events
 
Top Trends in Building Data Lakes for Machine Learning and AI
Top Trends in Building Data Lakes for Machine Learning and AI Top Trends in Building Data Lakes for Machine Learning and AI
Top Trends in Building Data Lakes for Machine Learning and AI
 
Verizon Centralizes Data into a Data Lake in Real Time for Analytics
Verizon Centralizes Data into a Data Lake in Real Time for AnalyticsVerizon Centralizes Data into a Data Lake in Real Time for Analytics
Verizon Centralizes Data into a Data Lake in Real Time for Analytics
 
Big Data LDN 2017: How to leverage the cloud for Business Solutions
Big Data LDN 2017: How to leverage the cloud for Business SolutionsBig Data LDN 2017: How to leverage the cloud for Business Solutions
Big Data LDN 2017: How to leverage the cloud for Business Solutions
 
An Introduction to the MapR Converged Data Platform
An Introduction to the MapR Converged Data PlatformAn Introduction to the MapR Converged Data Platform
An Introduction to the MapR Converged Data Platform
 
Converging your data landscape
Converging your data landscapeConverging your data landscape
Converging your data landscape
 
Building and Successfully Selling ISV Solutions with AWS Partner-Summit-Singa...
Building and Successfully Selling ISV Solutions with AWS Partner-Summit-Singa...Building and Successfully Selling ISV Solutions with AWS Partner-Summit-Singa...
Building and Successfully Selling ISV Solutions with AWS Partner-Summit-Singa...
 
Data & Analytics with CIS & Microsoft Platforms
Data & Analytics with CIS & Microsoft PlatformsData & Analytics with CIS & Microsoft Platforms
Data & Analytics with CIS & Microsoft Platforms
 
Encuentro Aporta 2016 - Mesa 2 - Miguel Arias
Encuentro Aporta 2016 - Mesa 2 - Miguel AriasEncuentro Aporta 2016 - Mesa 2 - Miguel Arias
Encuentro Aporta 2016 - Mesa 2 - Miguel Arias
 
Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...
Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...
Hybrid as a Stepping Stone: It’s Not All or Nothing for Your Cloud Transforma...
 
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)
Data DevOps - Arif Wider and Sean Gustafson (ThoughtWorks Live)
 
Big and fast data strategy 2017 jr
Big and fast data strategy 2017 jrBig and fast data strategy 2017 jr
Big and fast data strategy 2017 jr
 
Sustainable Organization and the Impact of Cloud Transformation on Business O...
Sustainable Organization and the Impact of Cloud Transformation on Business O...Sustainable Organization and the Impact of Cloud Transformation on Business O...
Sustainable Organization and the Impact of Cloud Transformation on Business O...
 
Integrating Semantic Web in the Real World: A Journey between Two Cities
Integrating Semantic Web in the Real World: A Journey between Two Cities Integrating Semantic Web in the Real World: A Journey between Two Cities
Integrating Semantic Web in the Real World: A Journey between Two Cities
 
AWS Webcast - Open Data on AWS – An Introduction
AWS Webcast - Open Data on AWS – An Introduction  AWS Webcast - Open Data on AWS – An Introduction
AWS Webcast - Open Data on AWS – An Introduction
 
AWS Community Day Nordics 2018 - Saku Vaittinen (VR): Data driven public tran...
AWS Community Day Nordics 2018 - Saku Vaittinen (VR): Data driven public tran...AWS Community Day Nordics 2018 - Saku Vaittinen (VR): Data driven public tran...
AWS Community Day Nordics 2018 - Saku Vaittinen (VR): Data driven public tran...
 
Predicting Banking Customer Needs with an Agile Approach to Analytics in the ...
Predicting Banking Customer Needs with an Agile Approach to Analytics in the ...Predicting Banking Customer Needs with an Agile Approach to Analytics in the ...
Predicting Banking Customer Needs with an Agile Approach to Analytics in the ...
 
The Data & Analytics Journey – Why it’s more attainable for your company than...
The Data & Analytics Journey – Why it’s more attainable for your company than...The Data & Analytics Journey – Why it’s more attainable for your company than...
The Data & Analytics Journey – Why it’s more attainable for your company than...
 

Recently uploaded

Research Methodology for Engineering pdf
Research Methodology for Engineering pdfResearch Methodology for Engineering pdf
Research Methodology for Engineering pdfCaalaaAbdulkerim
 
Instrumentation, measurement and control of bio process parameters ( Temperat...
Instrumentation, measurement and control of bio process parameters ( Temperat...Instrumentation, measurement and control of bio process parameters ( Temperat...
Instrumentation, measurement and control of bio process parameters ( Temperat...121011101441
 
Internet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptxInternet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptxVelmuruganTECE
 
Risk Management in Engineering Construction Project
Risk Management in Engineering Construction ProjectRisk Management in Engineering Construction Project
Risk Management in Engineering Construction ProjectErbil Polytechnic University
 
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor CatchersTechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catcherssdickerson1
 
Industrial Safety Unit-IV workplace health and safety.ppt
Industrial Safety Unit-IV workplace health and safety.pptIndustrial Safety Unit-IV workplace health and safety.ppt
Industrial Safety Unit-IV workplace health and safety.pptNarmatha D
 
Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvLewisJB
 
"Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ..."Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ...Erbil Polytechnic University
 
Unit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfg
Unit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfgUnit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfg
Unit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfgsaravananr517913
 
IVE Industry Focused Event - Defence Sector 2024
IVE Industry Focused Event - Defence Sector 2024IVE Industry Focused Event - Defence Sector 2024
IVE Industry Focused Event - Defence Sector 2024Mark Billinghurst
 
Main Memory Management in Operating System
Main Memory Management in Operating SystemMain Memory Management in Operating System
Main Memory Management in Operating SystemRashmi Bhat
 
CCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdf
CCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdfCCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdf
CCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdfAsst.prof M.Gokilavani
 
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)Dr SOUNDIRARAJ N
 
Transport layer issues and challenges - Guide
Transport layer issues and challenges - GuideTransport layer issues and challenges - Guide
Transport layer issues and challenges - GuideGOPINATHS437943
 
DM Pillar Training Manual.ppt will be useful in deploying TPM in project
DM Pillar Training Manual.ppt will be useful in deploying TPM in projectDM Pillar Training Manual.ppt will be useful in deploying TPM in project
DM Pillar Training Manual.ppt will be useful in deploying TPM in projectssuserb6619e
 
Virtual memory management in Operating System
Virtual memory management in Operating SystemVirtual memory management in Operating System
Virtual memory management in Operating SystemRashmi Bhat
 
Correctly Loading Incremental Data at Scale
Correctly Loading Incremental Data at ScaleCorrectly Loading Incremental Data at Scale
Correctly Loading Incremental Data at ScaleAlluxio, Inc.
 
Earthing details of Electrical Substation
Earthing details of Electrical SubstationEarthing details of Electrical Substation
Earthing details of Electrical Substationstephanwindworld
 
National Level Hackathon Participation Certificate.pdf
National Level Hackathon Participation Certificate.pdfNational Level Hackathon Participation Certificate.pdf
National Level Hackathon Participation Certificate.pdfRajuKanojiya4
 

Recently uploaded (20)

Research Methodology for Engineering pdf
Research Methodology for Engineering pdfResearch Methodology for Engineering pdf
Research Methodology for Engineering pdf
 
Instrumentation, measurement and control of bio process parameters ( Temperat...
Instrumentation, measurement and control of bio process parameters ( Temperat...Instrumentation, measurement and control of bio process parameters ( Temperat...
Instrumentation, measurement and control of bio process parameters ( Temperat...
 
Internet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptxInternet of things -Arshdeep Bahga .pptx
Internet of things -Arshdeep Bahga .pptx
 
Risk Management in Engineering Construction Project
Risk Management in Engineering Construction ProjectRisk Management in Engineering Construction Project
Risk Management in Engineering Construction Project
 
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor CatchersTechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
TechTAC® CFD Report Summary: A Comparison of Two Types of Tubing Anchor Catchers
 
Industrial Safety Unit-IV workplace health and safety.ppt
Industrial Safety Unit-IV workplace health and safety.pptIndustrial Safety Unit-IV workplace health and safety.ppt
Industrial Safety Unit-IV workplace health and safety.ppt
 
Work Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvvWork Experience-Dalton Park.pptxfvvvvvvv
Work Experience-Dalton Park.pptxfvvvvvvv
 
"Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ..."Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ...
 
Unit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfg
Unit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfgUnit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfg
Unit7-DC_Motors nkkjnsdkfnfcdfknfdgfggfg
 
IVE Industry Focused Event - Defence Sector 2024
IVE Industry Focused Event - Defence Sector 2024IVE Industry Focused Event - Defence Sector 2024
IVE Industry Focused Event - Defence Sector 2024
 
Main Memory Management in Operating System
Main Memory Management in Operating SystemMain Memory Management in Operating System
Main Memory Management in Operating System
 
CCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdf
CCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdfCCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdf
CCS355 Neural Networks & Deep Learning Unit 1 PDF notes with Question bank .pdf
 
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
UNIT III ANALOG ELECTRONICS (BASIC ELECTRONICS)
 
Transport layer issues and challenges - Guide
Transport layer issues and challenges - GuideTransport layer issues and challenges - Guide
Transport layer issues and challenges - Guide
 
DM Pillar Training Manual.ppt will be useful in deploying TPM in project
DM Pillar Training Manual.ppt will be useful in deploying TPM in projectDM Pillar Training Manual.ppt will be useful in deploying TPM in project
DM Pillar Training Manual.ppt will be useful in deploying TPM in project
 
Virtual memory management in Operating System
Virtual memory management in Operating SystemVirtual memory management in Operating System
Virtual memory management in Operating System
 
Designing pile caps according to ACI 318-19.pptx
Designing pile caps according to ACI 318-19.pptxDesigning pile caps according to ACI 318-19.pptx
Designing pile caps according to ACI 318-19.pptx
 
Correctly Loading Incremental Data at Scale
Correctly Loading Incremental Data at ScaleCorrectly Loading Incremental Data at Scale
Correctly Loading Incremental Data at Scale
 
Earthing details of Electrical Substation
Earthing details of Electrical SubstationEarthing details of Electrical Substation
Earthing details of Electrical Substation
 
National Level Hackathon Participation Certificate.pdf
National Level Hackathon Participation Certificate.pdfNational Level Hackathon Participation Certificate.pdf
National Level Hackathon Participation Certificate.pdf
 

The Beauty of Mapping Big Data

  • 1. The Beauty of Mapping Big Data Stoimen Popov R&D Lead, Product Innovation Team, HERE IoT| Dec 05, 2016
  • 2. HERE is the Open Location Platform company • Provides mapping services and location intelligence across the automotive, enterprise and internet industries • Employs 7,000 people in 56 countries • Produces maps for every country on earth • Enables four out of five in-car navigation systems in North America and Europe • Enables mobile, web and enterprise solutions for global industry leaders
  • 3. Agenda 01. Data Lens 01.1 Cloud Storage 01.2 Rest API & Query Language 01.3 JS API 02. Server-Side Clustering 03. Data Lens Heat Maps 03.1 Averaged Value 03.2 Alpha Mask by Density 03.3 Value Based Heat Map
  • 5. Data Lens is a cloud-based API toolkit for developing sophisticated visualizations of geographically referenced data, accessible in a web browser. Data is queried via a REST API from the Data Lens cloud, and transformed and aggregated to provide input to the JavaScript API, which renders the visualization.
  • 6. Cloud Storage • Data storage • Integration with HERE account • Data enrichment
  • 7. Data Lens REST API • Authentication • Data Upload • Datasets • Queries • Query Language • Access Management • Data Reprojection • ...
  • 8. Data Lens Query Language • JSON formatted queries • Similar to prepared statements in SQL for later execution • Only dataset owners can create queries for a dataset • Protect sensitive datasets
  • 9. Presentation title | Month 00, 20169 © 2016 HERE | HERE Internal Use Only Data Lens Query Language
  • 10. Data Lens JS API • Data Lens JavaScript API is a module of HERE Maps API for JavaScript and connects it to the Data Lens REST API • Provides data-driven styling of data on a map • Solves non-trivial tasks like tiling, caching and rasterizing data
  • 11. Presentation title | Month 00, 201611 © 2016 HERE | HERE Internal Use Only Data Lens Heat Map Dataset with 11M points of data about taxi rides in NY City
  • 12. Presentation title | Month 00, 201612 © 2016 HERE | HERE Internal Use Only Data Lens Street Shapes The data is geo-enriched to street geometry in NY City
  • 13. Presentation title | Month 00, 201613 © 2016 HERE | HERE Internal Use Only Data Lens ZIP Shapes The data is geo-enriched to ZIP code boundaries in NY City
  • 14. Presentation title | Month 00, 201614 © 2016 HERE | HERE Internal Use Only Data Lens Buildings The data is geo-enriched to building geometries in NY City
  • 15. 02 Server-Side Clustering How to deal with BIG DATA on the server?
  • 16. Data Tiling & Grouping in Pixel Space • Data tiling reduces the amount of data received by the client • Data Lens groups the data points per tile pixel
  • 17. 03 Data Lens Heat Maps • Averaged value • Alpha mask by density • Value-based heat map
  • 18. Presentation title | Month 00, 201618 © 2016 HERE | HERE Internal Use Only Averaged heat map
  • 19. Presentation title | Month 00, 201619 © 2016 HERE | HERE Internal Use Only Averaged heat map alone, and with an applied alpha mask
  • 20. Big Data can be visualized in many ways … Heat Maps Server-Side Clustering Hybrid Clustering (Server-side and client-side)
  • 21. https://developer.here.com Docs & API Reference Tech Examples Industry Examples  Develop  Code Examples  Data Lens APIs • Detailed Story • JS/HTML Code • Query Definitions • Styles & UI  Develop  Code Examples  Data Lens APIs • Data-Driven Styling • Server-Side Clustering • Hybrid Clustering • …and more!  Develop  Data Lens • Getting Started • Tutorial • Developer Guide • API Reference

Editor's Notes

  1. HERE is the Open Location Platform company, providing mapping, services and location intelligence across the automotive, enterprise and internet industries + who am I. Few slides in the beginning and at the end (perhaps merge the last slides)
  2. Few words on what I’m going to talk … and the structure of the talk itself.
  3. HERE is the Open Location Platform company, providing mapping, services and location intelligence across the automotive, enterprise and internet industries + who am I. Few slides in the beginning and at the end (perhaps merge the last slides)
  4. We allow our users to upload their own big data as a CSV. Data is protected (public/private) Explain data enrichment
  5. REST API allows us to perform various operations on our data. Upload (CSV) Manage datasets. Create, list, delete. Get schema, manage schema, upload files, etc. Manage queries (come again on this topic later on in the slides). Managing queries, etc. Publishing (protection of sensitive information) Re-project Data (Lat & Lon to UTM) Enriching data (Anchoring, enrich data of HERE platform to users data)
  6. The query is not a one-off action like a query in SQL. It corresponds more to the concept of prepared statements in the SQL world: You create a query for later execution; separating sending the query's source code and the actual retrieval. Only dataset owners can create queries for a dataset. They can then decide whether to make that query public or whether to keep it private, which means that only the query owner can send the query in question. This privacy option allows you to upload a sensitive dataset (for example, records with user information), make a less sensitive query (for example, adding the number of users by country in the result) and then only make the less sensitive data more widely available by only publishing that specific query.
  7. Query language example just to give a notion what it is and what to expect working with it. An example of a query. How data is aggregated and fetched is described with JSON as a query. Query is saved to the backend and executed (/data) every time the user wants to retrieve data.
  8. The main features of the Data Lens JavaScript API: Data Lens REST API connector ( Service ) for HERE Maps API Markers, clusters and primitives with styling parameterized by data and zoom level (data-driven styling) Value-based heat map with density alpha mask
  9. Before continuing with the heat map example, few works of what the JS API is capable of doing, not to leave the impression we’re doing only heat maps. Give a notion of what is it in the picture 11M points of Taxi data over NY
  10. Same data but anchored against HERE street shapes
  11. Same data anchored against ZIP code boundaries in NY
  12. Building shapes (note the number of the shapes) - bit of a note about vector tiles (shapes), protobuf and so on.
  13. What is pixel space and geographical/cartographical space. Grouping geo points into one pixel and we serve x,y, value and count Some words about valid use cases
  14. What are Data Lens heat maps so special …. Just few words on our different heat maps techniques since later on in the talk will come more detailed info Averaged (weighted, and few words about what weighted average is) and when it is better vs. sum Alpha mask by density – what it is and when it is to be used Value based heat map (averaged with applied alpha mask)
  15. A value-based heat map is created with a density alpha mask using the KDE method, but displayed as a density map with an applied color scale on its own. The bandwith parameter influences the perceived smoothness of the heat map surface. For smooth transition onto the base map or background, the density's colorScale can either have transparency on the low end on its own, or be applied as an alphaScale . In any case, the resulting output color scale is univariate. The Data Lens query language allows you to group data rows by buckets (in most cases a bucket is a 1x1 pixel) and aggregate row values for each bucket. Normally this type of query is used to draw a heat map. The simplest heat map can be instantiated as follows:
  16. A value-based heat map is created with a density alpha mask using the KDE method, but displayed as a density map with an applied color scale on its own. The bandwith parameter influences the perceived smoothness of the heat map surface. For smooth transition onto the base map or background, the density's colorScale can either have transparency on the low end on its own, or be applied as an alphaScale . In any case, the resulting output color scale is univariate. The Data Lens query language allows you to group data rows by buckets (in most cases a bucket is a 1x1 pixel) and aggregate row values for each bucket. Normally this type of query is used to draw a heat map. The simplest heat map can be instantiated as follows:
  17. constantly updated examples