1. Big data in geospatial applications is characterized by volume, velocity, and variety due to the rise of GPS-enabled devices and social media.
2. Technologies like MrGeo, Esper, and Neo4j Spatial help scale geoprocessing and spatial analysis to large datasets using techniques like MapReduce, complex event processing, and graph databases.
3. Case studies on the NYC Marathon and Colorado wildfires demonstrate how big data can enhance geoanalysis to provide insights when connecting large datasets to GIS.
This talk opened the geospatial track of the Apache Big Data conference. The geospatial track aimed to increase the benefits of implementing open source consistent with open geospatial standards.
After an introduction of the geospatial track this talk focused on these topics:
- Applications of Big Geo Data
- Geospatial Open Standards
- Big Geo Use Cases
- Open Source and Open Standards.
This talk opened the geospatial track of the Apache Big Data conference. The geospatial track aimed to increase the benefits of implementing open source consistent with open geospatial standards.
After an introduction of the geospatial track this talk focused on these topics:
- Applications of Big Geo Data
- Geospatial Open Standards
- Big Geo Use Cases
- Open Source and Open Standards.
SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...OpenTopography Facility
High-resolution topography is a powerful tool for studying the Earth's surface, vegetation, and urban landscapes, with broad scientific, engineering, and educational-based applications. Over the past decade, there has been dramatic growth in the acquisition of these data for scientific, environmental, engineering and planning purposes. In the US, the U.S. Geological Society is undertaking the 3D Elevation Program (3DEP) to map the entire lower 48 with lidar by 2023.
The richness of these topography datasets make them extremely valuable beyond the application that drove their acquisition and thus are of interest to a large and varied user community. A cyberinfrastructure platform that enables users to efficiently discover, access and process these massive volumes of data increases the impact of investments in collection of the data and catalyzes scientific discovery as well as informs critical decisions that are made across our Nation every day that depend on elevation data, ranging from immediate safety of life, property, and environment to long term planning for infrastructure projects.
Join us to hear about the motivations, technology, and data assets behind the National Science Foundation funded OpenTopography platform, which aims to democratize access to high resolution topographic data. OpenTopography’s innovation is in co-locating massive volumes of topographic data with processing tools that enable users with varied expertise and application domains to quickly and easily access and process data, to enable innovation and decision making.
This presentation introduces open source, open source GIS, OSGeo. This talk was given to the people who attended 'Capacity Building For National Surveying and Geographic Information Institute' program.
CyberGIS Architectures for Collaborative Problem Solving - OGC perspectiveGeorge Percivall
1. What is CyberGIS:
- collaboration; open data, open source, open standards
2. The plumbing for CyberGIS collaboration is available:
- Processing, Workflow, Model interoperability as web services are “solved” several times;
- the concepts for collaboration need to be made explicit
3. Need for “decision” and “hypothesis” objects including modeling and linked data
- Ontology for decision types. Templates for Decisions and Hypothesis
- Recommender systems - a guess at the riddle
- If I see these conditions then consider this decision template
- If I am researching these conditions then consider this hypothesis
Use case of Disaster Management System by using Geopaparazzi and MapGuide Ope...Hirofumi Hayashi
In recent years, large-scale disasters have occurred in the countries of
Asia including Japan, rapid collection and sharing of disaster information
is required in order to provide relief and support speedy restoration
of civic services. This presentation discusses the integration and customization
of FOSS4G field survey tools and Web GIS server to facilitate
aggregation and rapid sharing of disaster related field information.
Further, the system also provide realtime interaction between field party and
coordination team. A case study of practical use of the system at the Osaka Water
General Service (OWGS) Corporation will be demonstrated to present the salient
features of the system. The main capability of the system usability is normal
as well as disaster situation will be highlighted.
Building a geospatial processing pipeline using Hadoop and HBase and how Mons...DataWorks Summit
Monsanto built a geospatial platform on Hadoop and HBase capable of managing over 120 billion polygons. As a result of the extreme data volumes and compute complexities we were forced to migrate our data processing from a more traditional RDBMS to a scale out Hadoop implementation. Data processing that took over 30 days on 8% of the data now runs in under 12 hours on the entire data set. Very little concrete material exist for how you process spatial data via MapReduce or model it in HBase. We will provide concrete and novel examples for processing and storing spatial data on Hadoop and HBase. As part of the data processing pipeline we integrated the popular open source geospatial processing library GDAL with MapReduce to convert all geospatial datasets to a common format and projection. We developed a method for splitting and processing images via MapReduce in which the boundaries of splits needed to be shared by multiple tasks due to the nature of the computation being performed on the data. Bulk writes to HBase were performed by writing HFiles directly. Finally we developed a novel method for storing geospatial data in HBase that met the needs of our access pattern.
SDSC Technology Forum: Increasing the Impact of High Resolution Topography Da...OpenTopography Facility
High-resolution topography is a powerful tool for studying the Earth's surface, vegetation, and urban landscapes, with broad scientific, engineering, and educational-based applications. Over the past decade, there has been dramatic growth in the acquisition of these data for scientific, environmental, engineering and planning purposes. In the US, the U.S. Geological Society is undertaking the 3D Elevation Program (3DEP) to map the entire lower 48 with lidar by 2023.
The richness of these topography datasets make them extremely valuable beyond the application that drove their acquisition and thus are of interest to a large and varied user community. A cyberinfrastructure platform that enables users to efficiently discover, access and process these massive volumes of data increases the impact of investments in collection of the data and catalyzes scientific discovery as well as informs critical decisions that are made across our Nation every day that depend on elevation data, ranging from immediate safety of life, property, and environment to long term planning for infrastructure projects.
Join us to hear about the motivations, technology, and data assets behind the National Science Foundation funded OpenTopography platform, which aims to democratize access to high resolution topographic data. OpenTopography’s innovation is in co-locating massive volumes of topographic data with processing tools that enable users with varied expertise and application domains to quickly and easily access and process data, to enable innovation and decision making.
This presentation introduces open source, open source GIS, OSGeo. This talk was given to the people who attended 'Capacity Building For National Surveying and Geographic Information Institute' program.
CyberGIS Architectures for Collaborative Problem Solving - OGC perspectiveGeorge Percivall
1. What is CyberGIS:
- collaboration; open data, open source, open standards
2. The plumbing for CyberGIS collaboration is available:
- Processing, Workflow, Model interoperability as web services are “solved” several times;
- the concepts for collaboration need to be made explicit
3. Need for “decision” and “hypothesis” objects including modeling and linked data
- Ontology for decision types. Templates for Decisions and Hypothesis
- Recommender systems - a guess at the riddle
- If I see these conditions then consider this decision template
- If I am researching these conditions then consider this hypothesis
Use case of Disaster Management System by using Geopaparazzi and MapGuide Ope...Hirofumi Hayashi
In recent years, large-scale disasters have occurred in the countries of
Asia including Japan, rapid collection and sharing of disaster information
is required in order to provide relief and support speedy restoration
of civic services. This presentation discusses the integration and customization
of FOSS4G field survey tools and Web GIS server to facilitate
aggregation and rapid sharing of disaster related field information.
Further, the system also provide realtime interaction between field party and
coordination team. A case study of practical use of the system at the Osaka Water
General Service (OWGS) Corporation will be demonstrated to present the salient
features of the system. The main capability of the system usability is normal
as well as disaster situation will be highlighted.
Building a geospatial processing pipeline using Hadoop and HBase and how Mons...DataWorks Summit
Monsanto built a geospatial platform on Hadoop and HBase capable of managing over 120 billion polygons. As a result of the extreme data volumes and compute complexities we were forced to migrate our data processing from a more traditional RDBMS to a scale out Hadoop implementation. Data processing that took over 30 days on 8% of the data now runs in under 12 hours on the entire data set. Very little concrete material exist for how you process spatial data via MapReduce or model it in HBase. We will provide concrete and novel examples for processing and storing spatial data on Hadoop and HBase. As part of the data processing pipeline we integrated the popular open source geospatial processing library GDAL with MapReduce to convert all geospatial datasets to a common format and projection. We developed a method for splitting and processing images via MapReduce in which the boundaries of splits needed to be shared by multiple tasks due to the nature of the computation being performed on the data. Bulk writes to HBase were performed by writing HFiles directly. Finally we developed a novel method for storing geospatial data in HBase that met the needs of our access pattern.
By Dr Michael Gebert, Crowd Mentor Network. Presented at Crowdsourcing Week Europe 2014. Learn more & join us at the next event: http://crowdsourcingweek.com/
These slides are from a series of presentations during Esri Ireland's "ArcGIS - The Platform Story" Road Shows that took place in Belfast and Dublin in March 2014.
The evolution of ArcGIS to the geospatial platform it is today is bringing about transformational change to organisations by helping to break down the barriers between workflows, processes departments, and disciplines. Sharing data and collaboration using geographic information have now become much quicker and simpler.
Hear from our CTO & Esri Evangelist Eamonn Doyle on how customers are using the Esri Platform in their organisations and Esri's plans for the future of the platform.
With this toolkit, we’re bringing together the tools and guidance developed and tested in IRC’s work with governments, NGOs and other partners in over 20 countries. The toolkit is organised around the two related goals of delivering services and delivering change.
Adding Location and Geospatial Analytics to Big Data Analytics (BDT210) | AWS...Amazon Web Services
(Presented by Esri)
When people analyze a problem, they often include location at the core of the analysis. Location and spatial context, combined with geographical knowledge, can make the biggest difference in understanding a problem and analyzing it in a more meaningful way.
In this session, we show how Amazon EMR can be used with location and geospatial analytics, and how the Amazon EMR API and the Python SDK were used to build tools that integrate Big Data and geospatial analysis. We also show powerful visualization options for displaying your results, using maps which can be shared in reports or distributed online and to mobile apps.
Presentation given during the 2012 DC APA Fall Conference at Catholic University in Washington, DC. regarding the disruptive innovation that crowdsourcing and crowdfunding may provide to the urban planning and real estate development industries by providing supportive, effective community engagement.
A very basic presentation giving examples of crowdsourcing platforms and uses. Created in conjunction with the book "The Complete Idiot's Guide to Crowdsourcing."
Esri User Conference 2016 - UX & UI activitiesFrank Garofalo
Slides about the User Experience (UX) & User Interface (UI) Hub, User Experience (UX) & User Interface (UI) Exchange, and Interactive / Usability Research Lab at the 2016 Esri User Conference
New tools are being developed by Czech Living Lab WirelessInfo, which allow users to easily publish their data and metadata as part of a Spatial Data Infrastructure (SDI). The paper describes the design of a Technological Infrastructure on the basis of ISO and OGC Standards and also the implementation of a prototype and first experiences. The solution is designed in distributed system form, which provides the connection to metadata about spatial data and services. This solution tests the principle of catalogue services at both national and international level which could be used in the UN SDI context. A catalogue portal is one of the independent components of GeoHosting complex system for raster and vector spatial data sharing. The catalogue portal provides data source searching on the basis of their metadata records through structure queries. The portal also contains edit functionality for new metadata records creating or editing. The metadata catalogue system corresponds to ISO 19115/19119/19139 standards [1], [2], [3], [4] and provides for cascade searching on the other standardized catalogue systems. The difference is, there exist different other initiatives offering publishing of own content like Google technology or OpenStreet Map, that GeoHosting is based fully on INSPIRE European standards and support establishing of network of distributed servers.
Presentation Location and Context World, 2015. Palo Alto, CA November 3-4, 2015.
Abstract: Creating useful local context requires big data platforms and marketplaces. Contextual awareness is relevant to location based marketing, first responders, urban planners and many others. Location-aware mobile devices are revolutionizing how consumers and brands interact in the physical world. Situational awareness is a key element to efficiently handling any emergency response. In all cases, big data processing and high velocity streaming of location based data creates the richest contextual awareness. Data from many sources including IoT devices, sensor webs, surveillance and crowdsourcing are combined with semantically-rich urban and indoor data models. The resulting context information is delivered to and shared by mobile devices in connected and disconnected operations. Standards play a key role in establishing context platforms and marketplaces. Successful approaches will consolidate data from ubiquitous sensing technologies on a common space-time basis to enabled context-aware analysis of environmental and social dynamics.
FOSS4G 2017 Boston LocationTech; Big Data at the Heart of Geospatial InnovationMarc Vloemans
The rapid growth of Big Data (sensors, satellites, social networking) poses challenges to make the plethora of data more actionable intelligent: to process, monitor, manage, interpret and edit. As most information is tied to a location the geospatial information processing and management faces the same challenges regarding the trinity 'volume, velocity and variability'. Effectively the limits of traditional open spatial software components and tools have been reached. Thus necessitating cloud-based, distributed solutions.
In a relatively short time the Eclipse Working Group LocationTech has become the home for the much needed Big Geospatial Data innovation. Projects like GeoMesa, GeoWave, GeoTrellis, GeoGig and others enable temporal-spatial solutions for IoT, Automated Car, Earth Observation, Precision Agriculture, Disaster Management etc.
Please, check us out at www.locationtech.org or follow us on Twitter @locationtech
OpenMetadata Community Spotlight - Jürgen Zornig from ms.GISOpenMetadata
The OpenMetadata Community Meeting was held on May 8th, 2024. In the Community Spotlight, Jürgen Zornig from ms.GIS (https://msgis.com/) spoke about the spatial connector that they built using OpenMetadata. He discussed about his metadata journey in GIS, data lifecycle management, why they chose OpenMetadata, provided a solution overview, and the key benefits of holistic data lifecycle management in relation to the spatial connector.
You can view the recording here - https://www.youtube.com/watch?v=RbruainJUeU
Scaling Spatial Analytics with Google Cloud & CARTOCARTO
In this webinar, we focus on how Google Cloud and CARTO can be used to tackle even the most challenging Location Intelligence use cases at scale. You can watch the recorded webinar at: https://go.carto.com/webinars/google-cloud-spatial-analytics-at-scale
During the rise and innovation of “big data,” the geospatial analytics landscape has grown and evolved. We are beyond just analyzing static maps. Geospatial data is streaming from devices, sensors, infrastructure systems, or social media, and our applications and use cases must dynamically scale to meet the increased demands.
Cloud can provide cost-effective storage and that ephemeral resource-burst needed for fast processing and low latency, all to monetize the immediate value of fresh geospatial data. Geospatial analytics require optimized spatial data types and algorithms to distill data to knowledge. Such processing, especially with strict latency requirements, has always been a challenge.
We propose an open source big data stack for geospatial analytics on Cloud based on Apache NiFi, Apache Spark and LocationTech GeoMesa. GeoMesa is a geospatial framework deployed in a modern big data platform that provides a scalable and low latency solution for indexing volumes of historical data and generating live views and streaming geospatial analytics.
Presentation and live demo performed at DataWorks 2018 Conference - San Jose: https://bit.ly/2xthAGD
2018 GIS in the Rockies Vendor Showcase (Wed): Sharing Spatial Data FreelyGIS in the Rockies
One of the most important roles that can be filled by mature GIS agencies is that of a data librarian. Wikipedia notes that … “The role of a librarian is continually evolving to meet social and technological needs. A modern librarian may deal with provision and maintenance of information in many formats, including: books; electronic resources; magazines; newspapers; audio and video recordings; maps; manuscripts; photographs and …” Following along with this notion Boundless believes that sharing of maps should be as free as possible to permit GIS agencies to encourage widespread use of vital data and the information that will flow from this data. This presentation will show how Boundless is using its Exchange content management system with state and local governments to accommodate the need for GIS agencies to freely share spatial data both internally and externally to any user.
GWT 2014: Emergency Conference - 02 le soluzioni geospaziali per la gestione ...Planetek Italia Srl
Geospatial World Tour 2014: Emergency Conference.
Napoli, 28 maggio 2014.
Le soluzioni Geospaziali per la gestione delle emergenze.
Simone Colla, Hexagon Geospatial
This presentation accompanied a joint keynote address given by AAM's Brian Nicholls and Singapore Land Authority's (SLA) Dr Victor Khoo at the Locate17 Conference. AAM and the SLA are working together to capture and deliver an accurate and up-to-date 3D digital map for the entire country of Singapore, providing the digital framework for Singapore's visionary Smart Nation program. This presentation outlines the processes and technologies used to create the 3D digital map and highlights the many applications stemming from it such as Property Management systems, Solar Potential Studies, the development of Driverless Vehicle systems and more (many yet to be discovered!).
Similar to The State of Big Data for Geo - ESRI Big Data Meetup (20)
Presentation at Where 2.0 2008 where we discuss our rational for building a NoSQL data store after reaching limitations with the spatial SQL solution that were available at the time.
2009 AAG presentation for the "Is Google Good for Geography?" session.
Can the GeoWeb Get the Public to Care about Geography? The Positive Externalities of a Web Enabled Ecosystem
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Neuro-symbolic is not enough, we need neuro-*semantic*Frank van Harmelen
Neuro-symbolic (NeSy) AI is on the rise. However, simply machine learning on just any symbolic structure is not sufficient to really harvest the gains of NeSy. These will only be gained when the symbolic structures have an actual semantics. I give an operational definition of semantics as “predictable inference”.
All of this illustrated with link prediction over knowledge graphs, but the argument is general.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Kubernetes & AI - Beauty and the Beast !?! @KCD Istanbul 2024Tobias Schneck
As AI technology is pushing into IT I was wondering myself, as an “infrastructure container kubernetes guy”, how get this fancy AI technology get managed from an infrastructure operational view? Is it possible to apply our lovely cloud native principals as well? What benefit’s both technologies could bring to each other?
Let me take this questions and provide you a short journey through existing deployment models and use cases for AI software. On practical examples, we discuss what cloud/on-premise strategy we may need for applying it to our own infrastructure to get it to work from an enterprise perspective. I want to give an overview about infrastructure requirements and technologies, what could be beneficial or limiting your AI use cases in an enterprise environment. An interactive Demo will give you some insides, what approaches I got already working for real.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
10. MapReduce Geo (MrGeo)
• DIA project initiated by Terry Busch to extend geoprocessing to
very large data sets
• Built by SPADAC -> GeoEye -> Digital Globe
• Uses HDFS and MapReduce to store, process, and index
geospatial imagery and vector data
• Interoperable with:
– ArcGIS Desktop
– COMET
– Google Earth
– WMS clients
– Adobe Flex and Silverlight environments
• Listed on the Hadoop Apache page to be open sourced
14. GCEP
• Geospatial complex event processing
• Extends the to include the ability to use Geospatial
constructs in the rules for filtering events
• The ability to utilize the OGC Geospatial Functions
within Esper's Event Processing Language (EPL).
–Contains, within, disjoint, intersects, overlaps,
crosses, intersection, touches, buffer, relate, union,
convex hull
17. Neo4j Spatial
• Utilities for importing from ESRI Shapefile as well as
Open Street Map files
• Support for all the common geometry types
• An RTree index for fast searches on geometries
• Support for topology operations during the search
(contains, within, intersects, covers, disjoint, etc.)
• The possibility to enable spatial operations on any graph
of data, regardless of the way the spatial data is stored,
as long as an adapter is provided to map from the graph
to the geometries.
• Ability to split a single layer or dataset into multiple sub-
layers or views with pre-configured filters
The biggest issue with map-reduce and geostatistics is that map process assumes no inter-record dependencies. Which means you need flow control and multiple MR jobs (why cascading makes so much sense).
Meet customer demands for handling larger data, faster data and more data types
