The document discusses Esri's tools and roadmap for working with multi-dimensional (MD) scientific data in ArcGIS. It outlines Esri's efforts to directly read HDF, GRIB, and netCDF files as raster layers or feature/table views in ArcGIS. MD mosaic datasets allow users to manage variables and dimensions across multiple files and perform on-the-fly computations and visualization of MD data. New functions have been added to improve MD data analysis and visualization, including a vector field renderer to depict raster data as vectors. Esri is also working to better support OPeNDAP data sources.
This document discusses two HDF5-based file formats for storing Earth observation data:
1. The Sorted Pulse Data (SPD) format stores laser scanning data including pulses and point data with attributes. It was created in 2008 and updated to version 4 to improve flexibility.
2. The KEA image file format implements the GDAL raster data model in HDF5, allowing large raster datasets and attribute tables to be stored together with compression. It was created in 2012 to address limitations of other formats.
Both formats take advantage of HDF5 features like compression but also discuss some limitations and lessons learned for effectively designing scientific data formats.
The document discusses enhancing the Geospatial Data Abstraction Library (GDAL) to improve accessibility and interoperability of NASA data products with GIS tools. It developed plugins for three NASA data products to demonstrate reading multidimensional datasets into GIS applications like ArcGIS. Next steps include providing outreach, enhancing the framework to be more flexible and production-ready, and developing guides to help other data centers build GDAL plugins to address their issues with geospatial data. The overall goal is to improve analysis and visualization of NASA scientific data in GIS tools and web applications.
This document discusses how ArcGIS supports scientific multidimensional data. It can directly ingest data in formats like netCDF, HDF, and GRIB, and represent the data as raster layers, feature layers, or tables. Users can visualize, analyze, and share the data through tools in ArcGIS Desktop and services. Python can also be used to extend analytical capabilities. ArcGIS is evolving to better support scientific data through capabilities like multidimensional raster and feature layers, on-the-fly processing, and disseminating content as web services.
Aashish Chaudhary gave a presentation on Kitware's work with scientific computing and visualization using HDF. HDF is a widely used data format at Kitware for domains like climate modeling, geospatial visualization, and information visualization. Kitware is looking to improve HDF support for cloud and web environments to enable streaming analytics and web-based data analysis. The company also aims to further open source collaboration and scientific computing.
Harris Corporation provides geospatial software and analytics tools to access and analyze scientific data from remote sensing platforms. Their ENVI and IDL software support common data formats like HDF and NetCDF and provide capabilities for calibration, bowtie correction, reprojection, and visualization of data from sensors including GOES-16, VIIRS, and ocean and weather satellites. The tools allow scientists and analysts to efficiently process large volumes of earth observation data and extract valuable information to support applications in weather forecasting, agriculture, infrastructure monitoring, and more.
This document summarizes the work done to enhance the Geospatial Data Abstraction Library (GDAL) to better support NASA Earth Observing System (EOS) data products. It describes three phases of work: 1) a proof-of-concept ArcGIS plugin for product-specific HDF drivers, 2) generalized HDF drivers and an XML format, and 3) collaboration with GDAL developers utilizing HDF drivers and a Virtual Format (VRT) specification. The third phase highlights include enhanced generic functions, coordination with GDAL developers, testing across GIS clients, outreach to other data centers, and building tutorials. Future work areas are also outlined.
The HDF Group provides updates on new features in HDF including faster compression, single writer/multiple reader file access, virtual datasets, and dynamically loaded filters. They also discuss tools like HDFView, nagg for data aggregation, and a new HDF5 ODBC driver. The work is supported by NASA.
The document discusses Esri's tools and roadmap for working with multi-dimensional (MD) scientific data in ArcGIS. It outlines Esri's efforts to directly read HDF, GRIB, and netCDF files as raster layers or feature/table views in ArcGIS. MD mosaic datasets allow users to manage variables and dimensions across multiple files and perform on-the-fly computations and visualization of MD data. New functions have been added to improve MD data analysis and visualization, including a vector field renderer to depict raster data as vectors. Esri is also working to better support OPeNDAP data sources.
This document discusses two HDF5-based file formats for storing Earth observation data:
1. The Sorted Pulse Data (SPD) format stores laser scanning data including pulses and point data with attributes. It was created in 2008 and updated to version 4 to improve flexibility.
2. The KEA image file format implements the GDAL raster data model in HDF5, allowing large raster datasets and attribute tables to be stored together with compression. It was created in 2012 to address limitations of other formats.
Both formats take advantage of HDF5 features like compression but also discuss some limitations and lessons learned for effectively designing scientific data formats.
The document discusses enhancing the Geospatial Data Abstraction Library (GDAL) to improve accessibility and interoperability of NASA data products with GIS tools. It developed plugins for three NASA data products to demonstrate reading multidimensional datasets into GIS applications like ArcGIS. Next steps include providing outreach, enhancing the framework to be more flexible and production-ready, and developing guides to help other data centers build GDAL plugins to address their issues with geospatial data. The overall goal is to improve analysis and visualization of NASA scientific data in GIS tools and web applications.
This document discusses how ArcGIS supports scientific multidimensional data. It can directly ingest data in formats like netCDF, HDF, and GRIB, and represent the data as raster layers, feature layers, or tables. Users can visualize, analyze, and share the data through tools in ArcGIS Desktop and services. Python can also be used to extend analytical capabilities. ArcGIS is evolving to better support scientific data through capabilities like multidimensional raster and feature layers, on-the-fly processing, and disseminating content as web services.
Aashish Chaudhary gave a presentation on Kitware's work with scientific computing and visualization using HDF. HDF is a widely used data format at Kitware for domains like climate modeling, geospatial visualization, and information visualization. Kitware is looking to improve HDF support for cloud and web environments to enable streaming analytics and web-based data analysis. The company also aims to further open source collaboration and scientific computing.
Harris Corporation provides geospatial software and analytics tools to access and analyze scientific data from remote sensing platforms. Their ENVI and IDL software support common data formats like HDF and NetCDF and provide capabilities for calibration, bowtie correction, reprojection, and visualization of data from sensors including GOES-16, VIIRS, and ocean and weather satellites. The tools allow scientists and analysts to efficiently process large volumes of earth observation data and extract valuable information to support applications in weather forecasting, agriculture, infrastructure monitoring, and more.
This document summarizes the work done to enhance the Geospatial Data Abstraction Library (GDAL) to better support NASA Earth Observing System (EOS) data products. It describes three phases of work: 1) a proof-of-concept ArcGIS plugin for product-specific HDF drivers, 2) generalized HDF drivers and an XML format, and 3) collaboration with GDAL developers utilizing HDF drivers and a Virtual Format (VRT) specification. The third phase highlights include enhanced generic functions, coordination with GDAL developers, testing across GIS clients, outreach to other data centers, and building tutorials. Future work areas are also outlined.
The HDF Group provides updates on new features in HDF including faster compression, single writer/multiple reader file access, virtual datasets, and dynamically loaded filters. They also discuss tools like HDFView, nagg for data aggregation, and a new HDF5 ODBC driver. The work is supported by NASA.
This document discusses incorporating ISO metadata standards into HDF files using the HDF Product Designer tool. It describes how the HDF Product Designer allows users to import pre-built ISO metadata components from a separate project into their HDF file designs. This allows essential contextual data or metadata to be stored in HDF5 files according to ISO 19115 standards.
This document provides information about HDF (Hierarchical Data Format) tools and resources for working with Earth observation data. It summarizes HDF's focus on helping users at different stages of working with data, from initial product design to long-term archiving. It also describes specific HDF tools for viewing, comparing, converting between formats and adding metadata to scientific data files.
This document discusses new features and capabilities in MATLAB for working with scientific data formats and performing technical computing. It highlights enhancements to reading HDF5 and NetCDF files with both high-level and low-level interfaces. It also covers new capabilities for handling dates/times, big data, and accessing web services through RESTful APIs.
This document discusses using MATLAB for working with big data and scientific data formats. It provides an overview of MATLAB's capabilities for scientific data, including interfaces for HDF5 and NetCDF formats. It also describes how MATLAB can be used to access, analyze, and visualize big data from sources like Hadoop, databases, and RESTful web services. As a demonstration, it shows how MATLAB can access HDF5 data stored on an HDF Server through RESTful web requests and analyze the data using in-memory data types and functions.
John Readey presented on HDF5 in the cloud using HDFCloud. HDF5 can provide a cost-effective cloud infrastructure by paying for what is used rather than what may be needed. HDFCloud uses an HDF5 server to enable accessing HDF5 data through a REST API, allowing users to access large datasets without downloading entire files. It maps HDF5 objects to cloud object storage for scalable performance and uses Docker containers for elastic scaling.
This document discusses using HDF4 file content maps to enable cloud computing capabilities for HDF4 files. HDF4 files contain scientific data but their large size and legacy format pose challenges. The document proposes creating XML maps that describe HDF4 file structure and contents, including chunk locations and sizes. These maps could then be indexed and searched to locate relevant data chunks. Only those chunks would need to be extracted to the cloud, avoiding unnecessary data transfers. This would allow HDF4 files to be queried and analyzed using cloud-based tools while reducing storage costs.
This document discusses changes made to the MOPITT instrument's HDF data products over multiple versions. MOPITT measures carbon monoxide on the Terra spacecraft. Version 7 products, which use HDF-EOS5 format without unlimited dimensions, were found to be read faster than Version 6 products despite being larger in size. Experiments comparing read times of Version 7 data with and without unlimited dimensions showed a small 2% difference initially, but up to 40% faster reads for Version 7 without unlimited dimensions at first, with the difference decreasing over multiple reads. Eliminating unlimited dimensions thus improved access performance for some HDF applications like MOPITT data.
This document discusses MATLAB support for scientific data formats and analytics workflows. It provides an overview of MATLAB's capabilities for accessing, exploring, and preprocessing large scientific datasets. These include built-in support for HDF5, NetCDF, and other file formats. It also describes datastore objects that allow loading large datasets incrementally for analysis. The document concludes with an example that uses a FileDatastore to access and summarize HDF5 data from NASA ice sheet surveys in a MapReduce workflow.
Kitware uses HDF as a widely adopted data format for scientific computing and visualization across several domains. HDF supports climate modeling, geospatial data, medical imaging, and more. Kitware is looking to improve HDF support for streaming big data, cloud computing, and web applications to enable more advanced analytics and sharing of scientific data. Future work may include pure JavaScript implementations of HDF tools and optimizing performance for cloud storage.
ICESat-2 is a NASA satellite mission scheduled to launch in December 2017 that will use photon counting laser altimetry to measure ice sheet and sea ice elevations. It will carry an advanced laser altimeter that splits each laser pulse into 6 beams in a cross-track pattern to provide dense sampling. This will allow for improved elevation estimates over rough terrain. The document discusses ICESat-2's science objectives, measurement concept, data products, processing workflow, and approach to managing metadata across different levels of data products.
This presentation discusses moving data and applications from HDF4 to HDF5/netCDF-4. It covers the differences between HDF4 and HDF5 data models and capabilities, tools for converting HDF4 data to HDF5, advantages of HDF5 like unlimited dimensions and compression, and ways to ensure compatibility with netCDF-4 like avoiding HDF5-specific features. The work was supported by a NASA contract.
The document discusses MODIS Land products and their distribution format. MODIS Land products include radiation budget, ecosystem, and land cover variables. Products are distributed in HDF-EOS format with fine resolution grids in Integerized Sinusoidal or Lambert Azimuthal Equal-Area projections and coarse grids in a geographic Climate Modeling Grid. HDF-EOS allows for collaboration and standard geolocation representation but toolkit support is still limited.
This document discusses NEON's use of HDF5 file format for its ecological data. The goals are to implement a fast and efficient file format, develop a standardized data delivery structure, and provide metadata. It describes the HDF5 file structure, metadata inclusion, and an example workflow for processing eddy covariance data into HDF5 files. Future work includes integrating R code for HDF5 file generation and embedding ecological metadata.
The document provides an overview of the HDF-EOS5 file format including:
- HDF-EOS5 files contain coremetadata, archivemetadata, and StructMetadata global attributes that provide information on the file structure and contents.
- Files can contain Grid, Swath, Point, Zonal Average, and Profile data structures with no size limits.
- Swath data is organized by time or track with irregular spacing, storing geolocation, time, and data in arrays. Grid data is organized by regular geographic spacing specified by projection parameters with data and geolocation information separated. Point data specifies locations but with no organization of the data.
This document discusses a pilot project to incorporate ISO 19115-2 metadata attributes at the granule level for the NASA SWOT mission. The metadata will be stored in HDF5 groups and generated in two ways - via XML serialization or XML style sheet conversions. The project aims to capture essential metadata attributes from the SWOT information architecture and ISO metadata model, and generate an HDF5 structure specification and example metadata snippets.
The document summarizes updates on Hierarchical Data Formats (HDF) software releases and tools. It discusses the latest releases of HDF5 1.8.19 and 1.10.1, compatibility issues when moving to newer versions, updates on tools like HDF-Java and HDFView 3.0, supported compilers and systems, and a new compression library for interoperability. It invites readers to provide feedback on their needs.
This document discusses how HDF Product Designer (HPD) uses templates to achieve interoperability. HPD is an application for consistently developing interoperable data content in HDF5 files. It has a client-server architecture and desktop app. Templates allow users to copy design examples that incorporate best practices and are curated by the HPD development team. Available templates include NCEI collections and CF templates, with more to be added based on community review and suggestions. Templates allow users to initialize new designs by mixing and matching content from different template examples.
The document describes the HDF Product Designer software tool. It was created to facilitate the design of interoperable scientific data products in HDF5 format. The tool allows intuitive editing of HDF5 objects and supports conventions like CF and ACDD. It also provides validation services to test file compliance. The goal is to help scientists design data products that follow standards and are easy for others to use.
This document discusses efforts to standardize data products from three NASA laser altimeter missions - ICESat, ICESat-2, and MABEL. It describes designing similar data products for all three missions to promote interoperability. Products are being developed for ICESat data using lessons from MABEL. Code is also being generated to help create products from specifications to reduce development time. The goal is to make the multi-rate point data from all three missions easily accessible and usable.
GDAL (Geospatial Data Abstraction Library) is a translator library for raster and vector geospatial data formats. It supports over 140 raster and vector data formats and has tools for data translation and processing. Key GDAL utilities include gdalinfo for reporting metadata, gdal_translate for format conversion, gdalwarp for image warping and projection changes, and gdal_merge for mosaicking multiple rasters. GDAL is open source and works across many operating systems.
This document discusses incorporating ISO metadata standards into HDF files using the HDF Product Designer tool. It describes how the HDF Product Designer allows users to import pre-built ISO metadata components from a separate project into their HDF file designs. This allows essential contextual data or metadata to be stored in HDF5 files according to ISO 19115 standards.
This document provides information about HDF (Hierarchical Data Format) tools and resources for working with Earth observation data. It summarizes HDF's focus on helping users at different stages of working with data, from initial product design to long-term archiving. It also describes specific HDF tools for viewing, comparing, converting between formats and adding metadata to scientific data files.
This document discusses new features and capabilities in MATLAB for working with scientific data formats and performing technical computing. It highlights enhancements to reading HDF5 and NetCDF files with both high-level and low-level interfaces. It also covers new capabilities for handling dates/times, big data, and accessing web services through RESTful APIs.
This document discusses using MATLAB for working with big data and scientific data formats. It provides an overview of MATLAB's capabilities for scientific data, including interfaces for HDF5 and NetCDF formats. It also describes how MATLAB can be used to access, analyze, and visualize big data from sources like Hadoop, databases, and RESTful web services. As a demonstration, it shows how MATLAB can access HDF5 data stored on an HDF Server through RESTful web requests and analyze the data using in-memory data types and functions.
John Readey presented on HDF5 in the cloud using HDFCloud. HDF5 can provide a cost-effective cloud infrastructure by paying for what is used rather than what may be needed. HDFCloud uses an HDF5 server to enable accessing HDF5 data through a REST API, allowing users to access large datasets without downloading entire files. It maps HDF5 objects to cloud object storage for scalable performance and uses Docker containers for elastic scaling.
This document discusses using HDF4 file content maps to enable cloud computing capabilities for HDF4 files. HDF4 files contain scientific data but their large size and legacy format pose challenges. The document proposes creating XML maps that describe HDF4 file structure and contents, including chunk locations and sizes. These maps could then be indexed and searched to locate relevant data chunks. Only those chunks would need to be extracted to the cloud, avoiding unnecessary data transfers. This would allow HDF4 files to be queried and analyzed using cloud-based tools while reducing storage costs.
This document discusses changes made to the MOPITT instrument's HDF data products over multiple versions. MOPITT measures carbon monoxide on the Terra spacecraft. Version 7 products, which use HDF-EOS5 format without unlimited dimensions, were found to be read faster than Version 6 products despite being larger in size. Experiments comparing read times of Version 7 data with and without unlimited dimensions showed a small 2% difference initially, but up to 40% faster reads for Version 7 without unlimited dimensions at first, with the difference decreasing over multiple reads. Eliminating unlimited dimensions thus improved access performance for some HDF applications like MOPITT data.
This document discusses MATLAB support for scientific data formats and analytics workflows. It provides an overview of MATLAB's capabilities for accessing, exploring, and preprocessing large scientific datasets. These include built-in support for HDF5, NetCDF, and other file formats. It also describes datastore objects that allow loading large datasets incrementally for analysis. The document concludes with an example that uses a FileDatastore to access and summarize HDF5 data from NASA ice sheet surveys in a MapReduce workflow.
Kitware uses HDF as a widely adopted data format for scientific computing and visualization across several domains. HDF supports climate modeling, geospatial data, medical imaging, and more. Kitware is looking to improve HDF support for streaming big data, cloud computing, and web applications to enable more advanced analytics and sharing of scientific data. Future work may include pure JavaScript implementations of HDF tools and optimizing performance for cloud storage.
ICESat-2 is a NASA satellite mission scheduled to launch in December 2017 that will use photon counting laser altimetry to measure ice sheet and sea ice elevations. It will carry an advanced laser altimeter that splits each laser pulse into 6 beams in a cross-track pattern to provide dense sampling. This will allow for improved elevation estimates over rough terrain. The document discusses ICESat-2's science objectives, measurement concept, data products, processing workflow, and approach to managing metadata across different levels of data products.
This presentation discusses moving data and applications from HDF4 to HDF5/netCDF-4. It covers the differences between HDF4 and HDF5 data models and capabilities, tools for converting HDF4 data to HDF5, advantages of HDF5 like unlimited dimensions and compression, and ways to ensure compatibility with netCDF-4 like avoiding HDF5-specific features. The work was supported by a NASA contract.
The document discusses MODIS Land products and their distribution format. MODIS Land products include radiation budget, ecosystem, and land cover variables. Products are distributed in HDF-EOS format with fine resolution grids in Integerized Sinusoidal or Lambert Azimuthal Equal-Area projections and coarse grids in a geographic Climate Modeling Grid. HDF-EOS allows for collaboration and standard geolocation representation but toolkit support is still limited.
This document discusses NEON's use of HDF5 file format for its ecological data. The goals are to implement a fast and efficient file format, develop a standardized data delivery structure, and provide metadata. It describes the HDF5 file structure, metadata inclusion, and an example workflow for processing eddy covariance data into HDF5 files. Future work includes integrating R code for HDF5 file generation and embedding ecological metadata.
The document provides an overview of the HDF-EOS5 file format including:
- HDF-EOS5 files contain coremetadata, archivemetadata, and StructMetadata global attributes that provide information on the file structure and contents.
- Files can contain Grid, Swath, Point, Zonal Average, and Profile data structures with no size limits.
- Swath data is organized by time or track with irregular spacing, storing geolocation, time, and data in arrays. Grid data is organized by regular geographic spacing specified by projection parameters with data and geolocation information separated. Point data specifies locations but with no organization of the data.
This document discusses a pilot project to incorporate ISO 19115-2 metadata attributes at the granule level for the NASA SWOT mission. The metadata will be stored in HDF5 groups and generated in two ways - via XML serialization or XML style sheet conversions. The project aims to capture essential metadata attributes from the SWOT information architecture and ISO metadata model, and generate an HDF5 structure specification and example metadata snippets.
The document summarizes updates on Hierarchical Data Formats (HDF) software releases and tools. It discusses the latest releases of HDF5 1.8.19 and 1.10.1, compatibility issues when moving to newer versions, updates on tools like HDF-Java and HDFView 3.0, supported compilers and systems, and a new compression library for interoperability. It invites readers to provide feedback on their needs.
This document discusses how HDF Product Designer (HPD) uses templates to achieve interoperability. HPD is an application for consistently developing interoperable data content in HDF5 files. It has a client-server architecture and desktop app. Templates allow users to copy design examples that incorporate best practices and are curated by the HPD development team. Available templates include NCEI collections and CF templates, with more to be added based on community review and suggestions. Templates allow users to initialize new designs by mixing and matching content from different template examples.
The document describes the HDF Product Designer software tool. It was created to facilitate the design of interoperable scientific data products in HDF5 format. The tool allows intuitive editing of HDF5 objects and supports conventions like CF and ACDD. It also provides validation services to test file compliance. The goal is to help scientists design data products that follow standards and are easy for others to use.
This document discusses efforts to standardize data products from three NASA laser altimeter missions - ICESat, ICESat-2, and MABEL. It describes designing similar data products for all three missions to promote interoperability. Products are being developed for ICESat data using lessons from MABEL. Code is also being generated to help create products from specifications to reduce development time. The goal is to make the multi-rate point data from all three missions easily accessible and usable.
GDAL (Geospatial Data Abstraction Library) is a translator library for raster and vector geospatial data formats. It supports over 140 raster and vector data formats and has tools for data translation and processing. Key GDAL utilities include gdalinfo for reporting metadata, gdal_translate for format conversion, gdalwarp for image warping and projection changes, and gdal_merge for mosaicking multiple rasters. GDAL is open source and works across many operating systems.
GIS 5103 – Fundamentals of GISLecture 83D GIS.docxshericehewat
GIS 5103 – Fundamentals of GIS
Lecture 8
3D GIS
Dimensionality
A dimension is the minimum amount of coordinates needed to define the mathematical space of an object.
Lecture 8 – 3D GIS
2DA figure that has only length & height as its dimensions. 2D shapes lie on a flat surface; known as plane figures or plane shapes. While they have areas, 2D shapes have no volume.2D figures are plotted on two axes, the x- and y-axes.
Lecture 8 – 3D GIS
3DLength + Height + WidthX + Y + Z Volume + Surface AreaExamples include: 3D multi-patch building, roof, interior floors, and foundation would all contain different z-values for the same 2D coordinate. Aircraft's 3D position or a walking trail up a mountain, would only have a single z-value for each X,Y location.
Lecture 8 – 3D GIS
2.5DHighly used in GIS to represent Z data that is not continuous = 3DZ data does not need to be elevation. Pollution, # of cases, etc.
Lecture 8 – 3D GIS
3D file types .3dd.sxdWhen you import an ArcGlobe or ArcScene document:First the .3dd file opens by default in global mode Secondly the .sxd file opens in local mode. Any new blank scene view defaults to global mode.
Lecture 8 – 3D GIS
Data TypesVector FeaturesPoint / Line / Polygon Surface typesTIN (Triangular Irregular Networks)DEM’s (Digital Elevation Models)Raster SurfaceLAS Dataset
Lecture 8 – 3D GIS
3D StylesPointsGeometric ShapesModelsMarkersImportedLinesTextured SymbolsGeometric ShapesPolygonsTexture FillBasic Colors
Lecture 8 – 3D GIS
Lecture 8 – 3D GIS
Perspective ViewPerspective drawing is the most common drawing mode in 3D, where features in the foreground are shown larger than those off in the distance. This matches the way we see the world in our day-to-day lives, and the result is a realistic representation of 3D content. All scenes open in perspective viewing mode. You can switch between Perspective Perspective View and Parallel Isometric View viewing modes using the Drawing Mode drop-down menu in the Scene group on the View tab.
Lecture 8 – 3D GIS
Parallel / Isometric ViewParallel drawing renders the 3D view using a parallel projection, where features of the same physical size are rendered on-screen identically, regardless of their distance from the viewing camera. Parallel drawing is useful for architectural drawings, as well as for representing statistical data in a 3D view, such as extruded shapes symbolizing numeric values.
Lecture 8 – 3D GIS
3D Analyst ToolsData Conversion/PreparationTxt / Binary / Feature class / Raster / TINSurface CreationInterpolation / LASD CreationSurface AnalysisAspect / Slope / Contour / Feature Interpolation3D Operator & VisibilitySkyline / Inter-visibility / Sun Shadow Analysis
Lecture 8 – 3D GIS
There are five exploratory analysis tools:The Line of Sight tool creates sight lines to determine if one or more targets are visible from a given observer location.The View Dome tool determines the parts of a sphere that are visible from an observer locate ...
Open source based software ‘gxt’ mangosystemHaNJiN Lee
This document discusses GeoTools and GeoXTreme (GXT), open source geospatial toolkits. It provides information on:
- GeoTools, an open source GIS toolkit for developing standards compliant solutions. It supports various data formats and can compose maps.
- GXT, a commercial geoprocessing engine based on GeoTools that supports over 200 algorithms. GXT can be used for desktop or server applications.
- Examples of GXT applications including the KOPSS GIS engine, KOPSS data mart tools, and education/personal uses of GXT and uDig.
QGIS Processing at Linuxwochen Wien / PyDays 2017Anita Graser
QGIS is an open source desktop GIS that is the second most popular GIS after ESRI ArcGIS. It has been in development since 2002 and integrates geoprocessing tools using Python. The Processing plugin allows users to run algorithms from other software like SAGA and GRASS through Python scripts. These scripts handle tasks like describing algorithms, preparing input data, executing processes, and managing outputs. The plugin also supports batch processing, workflows, and accessing algorithms from R. Future plans include porting Processing to C++ and adding parallel processing support.
Geospatial web services using little-known GDAL features and modern Perl midd...Ari Jolma
This document summarizes a talk about using GDAL features and modern Perl middleware to build geospatial web services. It discusses using the GDAL virtual file system to read from and write to non-file sources, redirecting GDAL's virtual stdout to output to a Perl object, and using the PSGI specification to build middleware applications with Plack and services with the Geo::OGC framework. Code examples are provided for a WFS service using PostgreSQL and on-the-fly WMTS tile processing.
The document summarizes the Boost Graph Library (BGL) family of libraries for graph computation. The BGL includes sequential, Python, and parallel versions. The original BGL in C++ is efficient and customizable. BGL-Python provides access to BGL algorithms from Python. The Parallel BGL enables graph processing on clusters for huge graphs over millions of nodes.
Cloud Revolution: Exploring the New Wave of Serverless Spatial DataSafe Software
Once in a while, there really is something new under the sun. The rise of cloud-hosted data has fueled innovation in spatial data storage, enabling a brand new serverless architectural approach to spatial data sharing. Join us in our upcoming webinar to learn all about these new ways to organize your data, and leverage data shared by others. Explore the potential of Cloud Native Geospatial Formats in your workflows with FME, as we introduce five new formats: COGs, COPC, FlatGeoBuf, GeoParquet, STAC and ZARR.
Learn from industry experts Michelle Roby from Radiant Earth and Chris Holmes from Planet about these cloud-native geospatial data formats and how they can make data easier to manage, share, and analyze. To get us started, they’ll explain the goals of the Cloud-Native Geospatial Foundation and provide overviews of cloud-native technologies including the Cloud-Optimized GeoTIFF (COG), SpatioTemporal Asset Catalogs (STAC), and GeoParquet.
Following this, our seasoned FME team will guide you through practical demonstrations, showcasing how to leverage each format to its fullest potential. Learn strategic approaches for seamless integration and transition, along with valuable tips to enhance performance using these formats in FME.
Discover how these formats are reshaping geospatial data handling and how you can seamlessly integrate them into your FME workflows and harness the explosion of cloud-hosted data.
BigDL: Bringing Ease of Use of Deep Learning for Apache Spark with Jason Dai ...Databricks
BigDL is a distributed deep learning framework for Apache Spark open sourced by Intel. BigDL helps make deep learning more accessible to the Big Data community, by allowing them to continue the use of familiar tools and infrastructure to build deep learning applications. With BigDL, users can write their deep learning applications as standard Spark programs, which can then directly run on top of existing Spark or Hadoop clusters.
In this session, we will introduce BigDL, how our customers use BigDL to build End to End ML/DL applications, platforms on which BigDL is deployed and also provide an update on the latest improvements in BigDL v0.1, and talk about further developments and new upcoming features of BigDL v0.2 release (e.g., support for TensorFlow models, 3D convolutions, etc.).
Pig is a platform for analyzing large datasets that sits on top of Hadoop. It provides a simple language called Pig Latin for expressing data analysis processes. Pig Latin scripts are compiled into series of MapReduce jobs that process and analyze data in parallel across a Hadoop cluster. Pig aims to be easier to use than raw MapReduce programs by providing high-level operations like JOIN, FILTER, GROUP, and allowing analysis to be expressed without writing Java code. Common use cases for Pig include log and web data analysis, ETL processes, and quick prototyping of algorithms for large-scale data.
This document describes a tool created using Python to calculate snow cover area from MODIS Terra datasets and plot the change over time. It uses PyQt4 and PySide for the GUI, GDAL for image processing, NumPy for arithmetic and file listing, and Matplotlib for plotting. The tool allows the user to select input and output folders, runs snow cover calculations on the MODIS files, and displays the results as a graph of snow cover area over time. Future improvements could involve using more GDAL and OGR capabilities to generate output maps of permanent and seasonal snow cover.
The relationships between data sets matter. Discovering, analyzing, and learning those relationships is a central part to expanding our understand, and is a critical step to being able to predict and act upon the data. Unfortunately, these are not always simple or quick tasks.
To help the analyst we introduce RAPIDS, a collection of open-source libraries, incubated by NVIDIA and focused on accelerating the complete end-to-end data science ecosystem. Graph analytics is a critical piece of the data science ecosystem for processing linked data, and RAPIDS is pleased to offer cuGraph as our accelerated graph library.
Simply accelerating algorithms only addressed a portion of the problem. To address the full problem space, RAPIDS cuGraph strives to be feature-rich, easy to use, and intuitive. Rather than limiting the solution to a single graph technology, cuGraph supports Property Graphs, Knowledge Graphs, Hyper-Graphs, Bipartite graphs, and the basic directed and undirected graph.
A Python API allows the data to be manipulated as a DataFrame, similar and compatible with Pandas, with inputs and outputs being shared across the full RAPIDS suite, for example with the RAPIDS machine learning package, cuML.
This talk will present an overview of RAPIDS and cuGraph. Discuss and show examples of how to manipulate and analyze bipartite and property graph, plus show how data can be shared with machine learning algorithms. The talk will include some performance and scalability metrics. Then conclude with a preview of upcoming features, like graph query language support, and the general RAPIDS roadmap.
Porting the Source Engine to Linux: Valve's Lessons Learnedbasisspace
These slides discuss the techniques applied to porting a large, commercial AAA engine from Windows to Linux. It includes the lessons learned along the way, and pitfalls we ran into to help serve as a warning to other developers.
Spark For Plain Old Java Geeks (June2014 Meetup)sdeeg
This document provides an overview of Apache Spark, including:
- Spark is a cluster computing framework that allows processing of large datasets in parallel across a cluster using a programming model based around Resilient Distributed Datasets (RDDs).
- Spark supports multiple languages and has related projects for SQL, streaming, machine learning and graph processing.
- Spark 1.0 was recently released with a focus on stability, performance improvements, and new features like support for Java 8 and Spark SQL.
FOSS4G 2010 PostGIS Raster: an Open Source alternative to Oracle GeoRasterJorge Arevalo
PostGIS WKT Raster is an open source alternative to Oracle GeoRaster that allows raster data to be stored and analyzed in a PostgreSQL/PostGIS database. It uses the Well-Known Text (WKT) format to represent raster data as a single data type, while Oracle GeoRaster uses two related data types. PostGIS WKT Raster supports indexing and analysis of both raster and vector data, unlike Oracle GeoRaster which only allows analysis of raster data using vector extents.
A talk about the OSGeo Live project; covering 43 projects that are available in a live DVD format (for you to run without installing). The project is much improved with OGC documentation and a description of many of the projects. New this year (thanks to some sponsorship) is quickstarts for several of the projects.
2. A little about me... President and Lead Consultant at Carteryx 15 years experience using geospatial technologies Involved in forestry, mining, engineering, environmental and many other sectors... Have used open source GIS for past 10 years GIS development and deployment specialist Experience in a number of software platforms
3. Introduction A GIS is only as good as the data that goes in to it... Preparing data can take time...time you may not have GDAL utilities can run at command line or as library Projection conversion, format conversion, sub-set... Bindings for Python and dll’s for Visual Studio
4. Agenda What is GDAL? Acquiring and Installing GDAL and it’s associated utilities Getting basic raster and vector information Converting between raster file types Converting between vector file types Basics of GDAL/OGR/OSR Python bindings Sample Python code to get metadata from series of rasters
10. Command Line Many commands available ogr2ogr – convert between data types, coordinate systems etc. gdalinfo – get summary or comprehensive metadata from raster data sets gdaltindex – create a MapServer style raster index shapefile gdaltransform – transform coordinates between spatial reference systems Python scripts gdal_merge.py – quick merge of multiple raster images to one image gdal2tiles.py – create multiple tiles from a single raster gdal2xyz.py – create xyz text file from raster
12. GDAL Bindings in Python What are GDAL Python Bindings GDAL/OGR/OSR libraries that are accessible to Python Gives data manipulation capabilities to Python How do they work? Calls to the libraries in the code... fromosgeo.gdalimport * or importosgeo.gdalasosr from osgeo.ogr import * or import osgeo.ogr asogr from osgeo.osr import * or import osgeo.osr asosr What is the big advantage? Batch processing and combining with built-in python functions
13. GDAL Bindings in Python What can you do with bindings/dlls GDAL Add bands Manipulate bands Create rasters Re-project rasters and more OGR Geoprocessing (union, intersection, buffer etc.) Get information (dataset, layers, features) Re-project Change data type