This document summarizes activities related to the HDF project. It discusses the status of The HDF Group as a nonprofit organization dedicated to supporting HDF. It reviews ESDIS activities including maintenance of HDF and HDF-EOS code and user support. It provides statistics on downloads, helpdesk requests, and forum usage. It also outlines maintenance and testing of HDF4, HDF5 and related software releases. Platform support issues are also addressed. The document covers recent improvements and previews future work for the HDF software suite.
This slide will demonstrate how to use OPeNDAP Java clients such as IDV and Panoply via HDF OPeNDAP data handlers to access various NASA HDF products such as AIRS, OMI, MLS, MODIS, TRMM, CERES, SeaWIFS etc. Various features of these tools that can help users easy access the HDF data will also be explored.
An update on HDF, including a status report on The HDF Group, an overview of recent changes to the HDF4 and HDF5 libraries and tools, plans for future releases, HDF Group projects and collaborations, and future plans.
An update on HDF, including a status report on the HDF Group, an overview of recent changes to the HDF4 and HDF5 libraries and tools, plans for future releases, HDF Group projects and collaborations, and future plans.
Status of HDF-EOS and access tools will be summarized. Updates on HDF-EOS, TOOLKIT, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
This is an introductory slide for accessing NASA HDF/HDF-EOS data for beginners. NASA distributes many Earth Science data in HDF/HDF-EOS file format and new users struggle to understand the file format and use the NASA HDF/HDF-EOS data properly. This brief presentation will help new users to understand the basic concepts about the HDF/HDF-EOS and to know the available tools that can access the NASA data easily.
This slide will demonstrate how to use OPeNDAP Java clients such as IDV and Panoply via HDF OPeNDAP data handlers to access various NASA HDF products such as AIRS, OMI, MLS, MODIS, TRMM, CERES, SeaWIFS etc. Various features of these tools that can help users easy access the HDF data will also be explored.
An update on HDF, including a status report on The HDF Group, an overview of recent changes to the HDF4 and HDF5 libraries and tools, plans for future releases, HDF Group projects and collaborations, and future plans.
An update on HDF, including a status report on the HDF Group, an overview of recent changes to the HDF4 and HDF5 libraries and tools, plans for future releases, HDF Group projects and collaborations, and future plans.
Status of HDF-EOS and access tools will be summarized. Updates on HDF-EOS, TOOLKIT, HDFView plug-in and The HDF-EOS to GeoTIFF (HEG) conversion tool, including recent changes to the software, ongoing maintenance, upcoming releases, future plans, and issues will be discussed.
This is an introductory slide for accessing NASA HDF/HDF-EOS data for beginners. NASA distributes many Earth Science data in HDF/HDF-EOS file format and new users struggle to understand the file format and use the NASA HDF/HDF-EOS data properly. This brief presentation will help new users to understand the basic concepts about the HDF/HDF-EOS and to know the available tools that can access the NASA data easily.
In this talk, we will give an update on the HDF5 OPeNDAP project. We will update the new features inside OPeNDAP HDF5 data handler. We will also introduce a new HDF5-Friendly OPeNDAP client library and demonstrate how it can help users to view and analyze remote HDF-EOS5 data served by OPeNDAP HDF5 handler. A demo will be presented with a customized OPeNDAP visualization client (GrADS) that uses the library.
In this presentation, we will give an update on the HDF OPeNDAP project. We will update the new features inside the HDF5 OPeNDAP data handler. We will also introduce the enhanced HDF4 OPeNDAP data handler and demonstrate how it can help users to view and analyze remote HDF-EOS2 data. A demo that uses OPeNDAP client tools to handle AIRS and MODIS Grid/Swath data with the enhanced handler will be presented.
Update on HDF, including recent changes to the software, upcoming releases, collaborations, future plans. Will include an overview of the upcoming HDF5 1.8 release, and updates on the netCDF4/HDF5 merge, HDF5 support for indexing, BioHDF, the HDF5-Storage Resource Broker project, and the HDF spin-off THG.
The HDF-Java products include three components: HDF4 and HDF5 Java wrappers, HDF-Java object package, and HDFView. The Java wrappers provide standard Java APIs that allow applications to call the C HDF4 and HDF5 libraries from Java. The HDF-Java object package implements HDF data objects, e.g. Groups and Datasets, in an object-oriented form and makes it easy for applications to use the libraries. The HDFView is a visual tool for browsing and editing HDF4 and HDF5 files.
This presentation will include recent work on supporting HDF5 1.8 APIs and new features. As part of the HDF-NPOESS project, some enhancements have been added to HDFView to support region references and quality flags. The presentation will show these features along with other new features added to HDFView since HDF-Java 2.5 release.
This tutorial targets HDF5 application developers and users who still use HDF5 1.6 releases and anyone who is interested in the HDF5 1.8.x libraries features. We will discuss how applications written for versions 1.6.x and earlier can be seamlessly moved to use the latest HDF5 releases. We will also talk about new features of the 1.8.x HDF5 Library such as redesigned group object, links, creation order, and different performance tuning knobs.
Update on HDF, including recent changes to the software, new releases, THG collaborations, and future plans. Session will include an overview of the HDF4.2r2, HDF5 1.6.6, and 1.8.0 releases, as well as updates on completed and on-going THG projects including crash-proofing HDF5, efficient append to HDF5 datasets, and indexing in HDF5.
The HDF Group provides support for NPP/NPOESS in a number of ways, including development and maintenance of software capabilities in HDF5 libraries and tools that help NPP/NPOESS data producers and users, software testing on platforms of importance to NPP/NPOESS, high quality rapid response user support for NPP/NPOESS, and performance of special projects. The purposes of this presentation are to apprise attendees of the areas of emphasis for FY 2010, and to solicit ideas and opinions that will help the project understand how best to use its resources in order to best serve the needs of NPP/NPOESS.
The HDF Group is in the process of updating HDF-EOS web site. During the workshop, we would like to share with audiences some useful information in the new website that can help users to have easy access of NASA HDF and HDF-EOS data.
The presentation includes three parts:
EOS User Forum: will introduce the EOS user forum and how users can benefit from this forum.
Tools: will present information on how to use several widely-used tools to access NASA HDF and HDF-EOS data.
Examples: will present several examples on how to use C, Fortran and IDL to access NASA HDF and HDF-EOS data.
We will introduce the fully functional HDF5-OPeNDAP data handler in this talk. The features we have added into the data handler includes
- Mapping Compound Datatype to DAP Structure
- Mapping EOS Grid to DAP Grid
- Partial Support of CF Conventions
- Comprehensive Testsuite
At the end, we will show demos of using an OPeNDAP client to read HDF-EOS5 Aura data.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
The Art of the Pitch: WordPress Relationships and Sales
HDF Project Status and Plans
1. The HDF Group
HDF Project Update
Mike Folk, Elena Pourmal
And the HDF ESDIS Project Team
The HDF Group
April 18, 2012
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
1
www.hdfgroup.org
2. Outline
• What’s up with The HDF Group
• Review ESDIS activities
• Maintenance, QA and support
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
2
www.hdfgroup.org
3. WHAT’S UP WITH THE HDF
GROUP?
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
3
4. The HDF Group
• Dedicated to supporting HDF and its users
• Non-profit company since 2006
• At U of Illinois National Center for
Supercomputing Applications from 1988-2006
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
4
www.hdfgroup.org
5. Data challenges addressed by HDF
Need to organize
complex collections
of data
lat | lon | temp
----|-----|----12 | 23 | 3.1
15 | 24 | 4.2
17 | 21 | 3.6
Long term data
preservation
Efficient, sc
alable
storage and
access
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
5
www.hdfgroup.org
6. Members of the HDF support community
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
6
www.hdfgroup.org
7. Revenues by source
Other Govt &
Academic
25%
commercial
32%
NASA &
NOAA
43%
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
7
www.hdfgroup.org
10. The HDF Group Services
• Helpdesk and Mailing Lists
• Standard Support
• Consulting
• Training
• Enterprise Support
• Special Projects
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
10
www.hdfgroup.org
11. Downloads of HDF4, HDF5, HDFView
33,591
29,701
2010
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
2011
11
www.hdfgroup.org
18. The ESDIS project
• HDF development work
• Code maintenance
• HDF Support
• Studies, analyses, etc.
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
18
www.hdfgroup.org
20. HDF-EOS Website
• Improved the Quality of Comprehensive
Examples.
HDF-EOS Website
• Added new products in Comprehensive
Examples.
http://hdfeos.org
• Added forum feed in the main page.
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
20
www.hdfgroup.org
21. New products covered by examples
• GOSAT/ACOS
• Aquarius
• CloudSAT
• Ocean Productivity NPP
GOSAT/ACOS
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
Aquarius
21
www.hdfgroup.org
22. HDF-EOS Examples web stats
7/22/2010
Examples Announced
4/17/2012
2/1/2012
HDF AND HDF-EOS WORKSHOP XV
22
www.hdfgroup.org
23. Forum Feed in the Main Page
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
23
www.hdfgroup.org
26. HDF5 NASA products and netCDF-4
Use HDF5
• Aura
• Aura
• OMI, HIRDLS, MLS, TES
• Aquarius
• ACOS
• MEaSUREs
• SMAP
• ICESat-2
4/17/2012
• OMI, HIRDLS, MLS, TES
• MEaSUREs
• GSSTF, SeaWiFS,
Ozone Zonal Means
• Future
Want netCDF-4 accessibility
• GSSTF, SeaWiFS
• Future
• ICESat-2
HDF AND HDF-EOS WORKSHOP XV
26
www.hdfgroup.org
27. NetCDF4-friendly efforts
• Work with netCDF-4 developers and users
• NetCDF-4
• Augmentation
• eos52nc4
• Test netCDF-4 daily
• OPeNDAP
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
27
www.hdfgroup.org
28. (See “Mapping project Update”)
HDF4 FILE CONTENT MAPS
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
28
www.hdfgroup.org
39. hdf-forum members help with
• Release testing
• Maintaining CMake build systems on platforms
beyond Windows
• Answering questions
• The HDF Group’s HelpDesk focuses on ESDIS
and other paying customers while referring
users to FORUM for difficult topics that require
domain knowledge or very specific HDF5
usage
• Securing funding, especially for parallel HDF5
www.hdfgroup.org
40. Most discussed hdf-forum topics
•
•
•
•
•
•
•
Parallel questions and performance
Windows including .NET
Compound datatypes
Searching for data in HDF5 files
How to organize data in the HDF5 files
Fortran and C++ interfaces
Bug reports
www.hdfgroup.org
43. Issues and their Priorities
• Must Fix
• Fix after “Must Fix”
• Data corruption
• Portability
• Backward and Forward
Compatibility
• Funded Request
•
•
•
•
Power User Request
Tools
Library issues
Build Infrastructure
• When resources
permit
• Wrappers
• HL Libraries
• Other
Need your input on priorities!
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
45
www.hdfgroup.org
44. Maintenance Releases 2011 – 2012
May
2011
Nov
Dec
HDF4
HDF5
4.2.7
1.8.7
1.8.8
Mar
May
4.2.7patch1
Aug
Nov
Dec-Jan
2013
Code
freeze for
4.2.9
4.2.8
1.8.9
H4toH5
Java
Products
Feb
2012
1.8.10
Code
freeze for
2.2.2
2.2.1
2.8
2.9
Future releases
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
46
www.hdfgroup.org
46. HDF 4.2.7
• Released in February 2012
• New features
• More functions to support H4 mapping project
• Support for Linux PPC64 with IBM XL Fortran
• Minor bug fixes and docmentation improvements
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
48
www.hdfgroup.org
47. HDF 4.2.7-patch1
• Released in March 2012
• Fixes configuration problems for compilers with “-”
in the name
• HDF 4.2.7 source code/binaries NOT
AFFECTED
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
49
www.hdfgroup.org
48. Preview of HDF 4.2.8 and 4.2.9
• HDF 4.2.8
• Improvements to support HDF4 mapping project
• Port to Mac OS 10.7.* (Lion)
• HDF 4.2.9
• Improve portability by stressing “self-configuration”
• Clean HDF4 issues database
• Finalize transition to CMake on Windows (no
MS VS project files in the source code!)
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
50
www.hdfgroup.org
49. HDF5 1.8.7
• Released in May 2011
• New features
• Added “silent make mode” to simplify output during
builds
• Allow dimension size to be 0 (no data can be
written); don’t confuse with H5S_NULL (empty)
• Improved performance by allowing caching files
open through external links
• Added several verbose levels to h5diff
• Added an option to enable error stack in h5dump
• Improved Fortran H5LT functions to handle arrays
of 4 to 7 dimensions (before up 3D arrays only)
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
51
www.hdfgroup.org
50. HDF5 1.8.8
• Released in November 2011
• Added support for Fortran 2003
• Simplified and enhanced many existing routines
• Added support for new routines (e.g., functions
with callbacks)
• Enabled support for all kinds of INTEGER and
REAL
• Efficient reading/writing of HDF5 compound
datatypes
http://www.hdfgroup.org/HDF5/doc/fortran/NewFeatures_F2003.pdf
• Added Fortran wrappers for Dimension Scale
APIs
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
52
www.hdfgroup.org
51. HDF5 1.8.8
• Released in November 2011
• Improved VFD layer interoperability between
Windows and Linux
• Improved parallel library by taking advantage
of special collective I/O and complex derived
datatype MPI functionality
• Improved h5diff functionality
• Improved h5repack to handle object
references stored in the HDF5 attributes
• It is safe to use h5repack on netCDF-4 files
now
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
53
www.hdfgroup.org
52. Preview of HDF5 1.8.9
• Coming in May 2012
• New function
• H5LTpath_valid to check if path exists in an HDF5
file
• Tools improvements
• H5dump allow * in filenames
• H5dump can display attributes with “/” and datasets
with “[“ in their names
• H5repack considers chunking layout when writing
datatsets by hyperslabs
• Removed defects from several “corner cases” that
cause file corruption or seg faults
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
54
www.hdfgroup.org
53. Major Improvements
• h5dump
• Show attributes containing "/" for "-a" option
• Support wildcard in the filename
• h5repack
• 100x speedup for some cases involving
chunking
• h5diff
• Add options to show different levels of
information
• Add flag to exclude objects from comparison
• Major bug fixes for many tools
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
55
www.hdfgroup.org
55. HDF4 Platforms Supported
OS
Compilers
Linux 2.6 PPC64
Linux 2.6 CentOS-5
GNU C and Fortran 4.1.2
Intel C and Fortran v. 12
PGI C and Fortran v. 11
Linux 2.6 x86_64
GNU C and Fortran 4.1.2
Intel C and Fortran v. 12
PGI C and Fortran v. 11
Linux Debian, Fedora,
SUSE, Ubuntu
GNU C and Fortran
(default)
SunOS 5.10
Sun C 5.9 and Fortran 8.3
SGI Altix
Intel C and Fortran v. 11
Windows XP, 7 32/64,
Cygwin
VS 2008, 2010, Intel 10-11,
GNU C and Fortran
Mac OS X Intel 10.6.8
32/64-bit
4/17/2012
GNU C and Fortran 4.4.6
and IBM XL Fortran V13
GNU C 4.2.1 and gfortran
4.6.1; Intel C and Fortran 12
HDF AND HDF-EOS WORKSHOP XV
57
www.hdfgroup.org
56. HDF5 Platforms Supported
OS
Compilers
Same as for HDF4
AIX 5.3
IBM XL C 10.1 and Fortran
12.1
IBM Blue Gene/P
IBM compilers
Cary Linux
PGI C, C++ and Fortran
v.11.7
Linux Red Hat Enterprise
Intel C and Fortran 12.0
Windows Vista 32/64
VS 2008, 2010, Intel 10-11
Mac OS X Intel 10.7.0
32/64-bit
GNU C 4.2.1 and gfortran
4.6.1
OpenVMS 8.3
4/17/2012
Same as for HDF4
HP C, C++ and Fortran
HDF AND HDF-EOS WORKSHOP XV
58
www.hdfgroup.org
57. HDF4 and 5 Platforms to drop
OS
Compilers
Windows Vista, XP(?)
OpenVMS
4/17/2012
VS 2008, Intel 10, 11
We will use Cmake for
building HDF software on
Windows
HP C, C++ and Fortran
HDF AND HDF-EOS WORKSHOP XV
59
www.hdfgroup.org
58. HDF4 and 5 Platforms to add
OS
Compilers
Mac OS X 10.7.*
GNU and Intel Compilers
Windows 8
VS 2011
Cygwin (?), MinGW (?)
Default compilers
?
?
We are using virtualization very successfully.
Can add any Linux or Windows flavors.
Just let us know!
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
60
www.hdfgroup.org
60. HDF4 Software Evolution Themes
• Add support for H4 Mapping project
• Make HDF4 library “self-configurable”
• Improves portability
• Reduces maintenance cost
• Clean-up the code
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
62
www.hdfgroup.org
61. HDF4 Quotes
• How we documented the code in the last
century:
• Store calibration information. What is the
formula? Good question –GV
• Perhaps someone with more time can look into
this later. -QAK
• Hmm, not working yet?... -QAK
• This is horribly inefficient, but the separationof-powers gets really mucked up if we wait till
later... –Anonymous
• Ifdef NOT_YET, NOT_NOW, NOT_USED
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
63
www.hdfgroup.org
62. HDF5 Software Evolution Themes
•
•
•
•
•
Concurrent access
Remote Access
Parallel I/O performance
Real-time write performance
Support for high level libraries
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
64
www.hdfgroup.org
63. New features in the works
• Saving space (development completed)
• Persistent File Free Space tracking/recovery
(1.10.0)
• Saving time (taking more time)
• Asynchronous I/O
• Allow an application to proceed while the HDF5
library performs I/O (1.10.0)
• File image
• Create and read in-memory HDF5 files without
requiring I/O operations (1.8.9)
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
www.hdfgroup.org
65
64. New features in the works
• Saving time (taking even more time)
• Metadata aggregation (1.10.0)
• Improves I/O by aggregating small pieces of
HDF5 metadata
• Allocation MD in page size blocks in a file,
perform I/O in pages
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
66
www.hdfgroup.org
65. New features in the works
• Saving files when disaster strikes (1.10.0)
• Journaling
• Journal metadata changes saved in a file
• H5recover tool to restore metadata in a file
• Single Writer/Multiple Readers (SWMR)
• Allows simultaneous reading of HDF5 file
while the file is being modified by another
process
• H5watch tool completed
• Provides fault tolerance aspects for a file; if
writer crashes the file is in the consistent
state.
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
67
www.hdfgroup.org
66. New features in the works
• By popular demand:
• Object compare API and tool
• Based on a formal definition of the HDF5 objects
comparison
• Avoids ambiguity and features creep (as with h5diff)
• Emphasis on flexibility and efficiency
• Control over reporting “differences”
• Compare compressed data without uncompressing it
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
68
www.hdfgroup.org
67. Research/Prototyping
• Virtual Object Layer
• Leveraging HDF5 Data Model without enforcing
HDF5 file format
• Abstraction layer that allows different plugins
for accessing data
• Examples
• Different file formats (netCDF, HDF4, GRIB,
FITS)
• Directories and files on a file system
• Memory objects
• Remote objects
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
www.hdfgroup.org
69
69. HPC Improvement - Partnerships
Improve performance
of parallel apps
including netCDF-4
Improve performance
of parallel apps
Add features
anticipating exascale
systems
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
www.hdfgroup.org
71
71. HDF Java Products Highlights
• All major HDF5 1.8 API functions were
added to HDF5 JNI
• Unit tests were added to all major HDF5
JNI functions
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
73
www.hdfgroup.org
72. Things in the pipeline for HDF-Java
• Add CMake to compile and install hdf-java
products
• Continue bug fixes and enhancements
• HDF-Java 2.9 release with HDF 4.2.8 and
HDF5 1.8.10 (December 2012)
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
74
www.hdfgroup.org
73. The HDF Group
Thank You!
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
75
www.hdfgroup.org
74. Acknowledgements
• This work was supported by cooperative
agreement number NNX08AO77A from the
National Aeronautics and Space
Administration (NASA).
• Any opinions, findings, conclusions, or
recommendations expressed in this material
are those of the author[s] and do not
necessarily reflect the views of the National
Aeronautics and Space Administration.
4/17/2012
HDF AND HDF-EOS WORKSHOP XV
76
www.hdfgroup.org
Editor's Notes
NASA – EOSNOAA/NASA/Riverside Tech – NPOESS/JPSSA leading U.S. aerospace companyDOE projectsSandia National Laboratory Lawrence Berkeley National LabArgonneITER – international project to build an experimental fusion reactor based on the tokamak conceptPaul Scherrer Institute – variety of projectsProjects in oil and gas industry, finance, others“In kind” support
Helpdesk and Mailing Lists Available to all users as a first level of support Standard Support Rapid issue resolution and advice ConsultingNeeds assessment, troubleshooting, design reviews, etc.TrainingTutorials and hands-on practical experience Enterprise SupportSupporting many HDF activities across organizationsSpecial Projects Adapting customer applications to HDF New features and toolsResearch and Development
Capability Maturity Model Integration (CMMI)a compendium of “Best Practices” for planning, engineering, and managerial business processes.A CMMI appraisalan assessment of and organization’s current practices.A CMMI process improvement programactivities aimed at improving those process areas that are inadequately practiced in an organization.Level 2: Basic project managementRequirements Management Project PlanningProject Monitoring and ControlMeasurement and AnalysisProcess and Product Quality AssuranceConfiguration ManagementLevel 3: Process standardizationRequirements DevelopmentTechnical Solution
For other work in earth science area, see later talk, “HDF Group Support for NPP/NPOESS/JPSS”
HDF development workOPeNDAP handler developmentCode maintenanceHDF4, HDF5, HDFView, OPeNDAP handlers, etc. H4H5 conversion library and utilitiesSpecial tools, such as the HDF-EOS5 augmentation tool, the HDF-EOS2 dumper tool, the HDF-EOS5 to netCDF-4 converter tool and the HDF4/HDF-EOS2 to CF conversion libraryHDF Support Support to programmers and analysts and other EOS science software teams, tool vendors and other tool buildersHelp EOS stakeholders (DAACs, SIPs, vendors, etc.)Site visits to NASA data centers, SIPS and others Communicating with the NASA User Service Working Group (USWG) and NASA ESDIS outreach managers Helping NASA scientific applications to access and manage EOS data. Participation in Earth Science conferences such as AGU, AMS, ESDSWG and ESIP Federation meetingsHelpdesk and newsletterTutorials and workshopsHDF and HDF-EOS websitesStudies, analyses, etc.Investigate data catalog servers and integration with web service technologiesHDF4 content maps for archiving
Most effort at the top in big fonts.Take complex data and provide example of how to use tools effectively. Sample scripts for MATLAB, IDL, grads, other tools, which people can reuse and adapt to their own situations.
I made it as one slide.
Thanks to these improvements, we have an increased traffic in both visits and unique visitors.Add zoo/ stat only.The “zoo” page:The page provides comprehensive examples on how to access and visualize various NASA HDF4,HDF-EOS2, HDF5 and HDF-EOS5 files collecting from NASA data centers using IDL ®, MATLAB® and NCL. A short one:The page provides examples on using IDL, MATLAB and NCL to access and visualize almost all NASA HDF-family products.
Users can easily see what’s going on in the forum.
Re “Want netCDF-4 accessibility”: We can safely predict that user of other products will also want it.The current HIRDLS files are augmented so they are fully netCDF-4 compliant (following netCDF-4 data model correctly)MLS files are also augmented but not augmented in the quite right way. The tool is not wrong. They just didn't use the right option as HIRDLS did. I reported to them and they said they will fix the issue in the future. I just checked the current MLS files. They haven't updated yet. So I hesitated to say they are fully netCDF-4 compliant. OMI, TES and GSSTF files are not augmented at all so they are not netCDF-4 compliant. OMI and TES teams have been informed. For some reasons, they didn't augment their files. GSSTF is a MEASUREs product and it is kind of new. I don't remember the GSSTF is informed. I will contact Fan to share this information. All these files can be netCDF-4 compliant if they augment their files.
Re “Work with netCDF-4 developers and users”: mention that we have monthly telecons with Unidata and LLNL.
Issues have decreased steadily over the past 4 years.HDF5 under active development, so there are still a lot more issues.
Steady decline as we saw before, and library and build issues count the most.
Build, library and java issues dominate.
This is how we decide which issues to fix first.
Improved h5diff functionalityError and comparison reportingNaN comparisonHandling of nested compound datatypes
Removed defects from several “corner cases” that cause file corruption or seg faultsShrinking the size of compound datatypeCreating a datatset in a “read-only” fileShrinking datasets with chunks larger than 1MB
MD aggregation: MD blocks will be aligned, will know page address and can page in the whole block.
MD blocks will aligned, will know page address and can page in the whole block.V2 of btree and fractal heap, h5watch tool.
MD aggregation: MD blocks will be aligned, will know page address and can page in the whole block.
LLNL - file image, nor MD aggregation (parallel plus sequential)LBNL - Avoid truncate work, large chunks, collective MD eviction algorithms plus netCDFLBNL and Chicago – VOL work