The document summarizes HDF5 advanced topics presented at an HDF5 workshop. It discusses HDF5 groups and links which organize data objects in a file. It also covers HDF5 datasets and datatypes like compound and reference datatypes. HDF5 references allow accessing specific regions of datasets or objects in other files. The document provides examples in Python to demonstrate HDF5 groups, links, datatypes and references.
From a talk by Andrew Collette to the Boulder Earth and Space Science Informatics Group (BESSIG) on November 20, 2013.
This talk explores how researchers can use the scalable, self-describing HDF5 data format together with the Python programming language to improve the analysis pipeline, easily archive and share large datasets, and improve confidence in scientific results. The discussion will focus on real-world applications of HDF5 in experimental physics at two multimillion-dollar research facilities: the Large Plasma Device at UCLA, and the NASA-funded hypervelocity dust accelerator at CU Boulder. This event coincides with the launch of a new O’Reilly book, Python and HDF5: Unlocking Scientific Data.
As scientific datasets grow from gigabytes to terabytes and beyond, the use of standard formats for data storage and communication becomes critical. HDF5, the most recent version of the Hierarchical Data Format originally developed at the National Center for Supercomputing Applications (NCSA), has rapidly emerged as the mechanism of choice for storing and sharing large datasets. At the same time, many researchers who routinely deal with large numerical datasets have been drawn to the Python by its ease of use and rapid development capabilities.
Over the past several years, Python has emerged as a credible alternative to scientific analysis environments like IDL or MATLAB. In addition to stable core packages for handling numerical arrays, analysis, and plotting, the Python ecosystem provides a huge selection of more specialized software, reducing the amount of work necessary to write scientific code while also increasing the quality of results. Python’s excellent support for standard data formats allows scientists to interact seamlessly with colleagues using other platforms.
This tutorial is designed for anyone who needs to work with data stored in HDF5 files. The tutorial will cover functionality and useful features of the HDF5 utilities h5dump, h5diff, h5repack, h5stat, h5copy, h5check and h5repart. We will also introduce a prototype of the new h52jpeg conversion tool and recently released h5perf_serial tool used for performance studies. We will briefly introduce HDFView. Details of the HDFView and HDF-Java will be discussed in a separate tutorial.
A brief overview of using HDF5 with Python and Andrew Collette's h5py module will be presented, including examples which show how and why Python can be used in the place of HDF5 tools. Extensions to the HDF5 API will be proposed which would further improve the utility of Python/h5py.
From a talk by Andrew Collette to the Boulder Earth and Space Science Informatics Group (BESSIG) on November 20, 2013.
This talk explores how researchers can use the scalable, self-describing HDF5 data format together with the Python programming language to improve the analysis pipeline, easily archive and share large datasets, and improve confidence in scientific results. The discussion will focus on real-world applications of HDF5 in experimental physics at two multimillion-dollar research facilities: the Large Plasma Device at UCLA, and the NASA-funded hypervelocity dust accelerator at CU Boulder. This event coincides with the launch of a new O’Reilly book, Python and HDF5: Unlocking Scientific Data.
As scientific datasets grow from gigabytes to terabytes and beyond, the use of standard formats for data storage and communication becomes critical. HDF5, the most recent version of the Hierarchical Data Format originally developed at the National Center for Supercomputing Applications (NCSA), has rapidly emerged as the mechanism of choice for storing and sharing large datasets. At the same time, many researchers who routinely deal with large numerical datasets have been drawn to the Python by its ease of use and rapid development capabilities.
Over the past several years, Python has emerged as a credible alternative to scientific analysis environments like IDL or MATLAB. In addition to stable core packages for handling numerical arrays, analysis, and plotting, the Python ecosystem provides a huge selection of more specialized software, reducing the amount of work necessary to write scientific code while also increasing the quality of results. Python’s excellent support for standard data formats allows scientists to interact seamlessly with colleagues using other platforms.
This tutorial is designed for anyone who needs to work with data stored in HDF5 files. The tutorial will cover functionality and useful features of the HDF5 utilities h5dump, h5diff, h5repack, h5stat, h5copy, h5check and h5repart. We will also introduce a prototype of the new h52jpeg conversion tool and recently released h5perf_serial tool used for performance studies. We will briefly introduce HDFView. Details of the HDFView and HDF-Java will be discussed in a separate tutorial.
A brief overview of using HDF5 with Python and Andrew Collette's h5py module will be presented, including examples which show how and why Python can be used in the place of HDF5 tools. Extensions to the HDF5 API will be proposed which would further improve the utility of Python/h5py.
An introduction to the Python programming language and its numerical abilities will be presented. With this background, Andrew Collette's H5Py module--an HDF5-Python interface--will be explained highlighting the unique and useful similarities between Python data structures and HDF5.
This is an introductory slide for accessing NASA HDF/HDF-EOS data for beginners. NASA distributes many Earth Science data in HDF/HDF-EOS file format and new users struggle to understand the file format and use the NASA HDF/HDF-EOS data properly. This brief presentation will help new users to understand the basic concepts about the HDF/HDF-EOS and to know the available tools that can access the NASA data easily.
This tutorial is designed for new HDF5 users. We will cover HDF5 abstractions such as datasets, groups, attributes, and datatypes. Simple C examples will cover the programming model and basic features of the API, and will give new users the knowledge they need to navigate through the rich collection of HDF5 interfaces. Participants will be guided through an interactive demonstration of the fundamentals of HDF5.
This tutorial is for new HDF5 users.
This 2009 tutorial slide will cover basic HDF5 Data Model objects and their properties. It will include an overview of the HDF5 Libraries and APIs, and describe the HDF5 programming model. Simple programming examples and the HDFView data browser will be used to illustrate HDF5 concepts and start developing your own HDF5 based applications.
This tutorial is for new HDF5 users.
This Tutorial gives a brief introduction to HDF5 for people who have never used it. It covers the HDF5 Data Model including HDF5 objects and their properties. It also briefly describes the HDF5 Programming Model and prepares participants for further self-study of HDF5 and hands-on sessions.
This tutorial is designed for the HDF5 users with some HDF5 experience.
It will cover advanced features of the HDF5 library for achieving better I/O performance and efficient storage. The following HDF5 features will be discussed: partial I/O, chunked storage layout, compression and other filters including new n-bit and scale+offset filters. Significant time will be devoted to the discussion of complex HDF5 datatypes such as strings, variable-length datatypes, array and compound datatypes.
This tutorial is designed for new HDF5 users. We will go over a brief history of HDF and HDF5 software, and will cover basic HDF5 Data Model objects and their properties; we will give an overview of the HDF5 Libraries and APIs, and discuss the HDF5 programming model. Simple C and Fortran examples, and Java tool HDFView will be used to illustrate HDF5 concepts.
This tutorial targets HDF5 application developers and users who still use HDF5 1.6 releases and anyone who is interested in the HDF5 1.8.x libraries features. We will discuss how applications written for versions 1.6.x and earlier can be seamlessly moved to use the latest HDF5 releases. We will also talk about new features of the 1.8.x HDF5 Library such as redesigned group object, links, creation order, and different performance tuning knobs.
Efficient analysis of large scientific datasets often requires a means to rapidly search and select interesting portions of data
based on ad-hoc search criteria. We present our work on integrating an efficient searching technology named
FastBit
[2, 3]
with HDF5. The integrated system named
HDF5-FastQuery
allows the users to efficiently generate complex selections on
HDF5 datasets using compound range queries of the form
(
temperature>
1000)
AND
(70
<pressure><
90)
. The FastBit
technology generates compressed bitmap indices that accelerate searches on HDF5 datasets and can be stored together with
those datasets in an HDF5 file. Compared with other indexing schemes, compressed bitmap indices are compact and very well
suited for searching over multidimensional data – even for arbitrarily complex combinations of range conditions.
The increasing sizes of the data being stored in HDF files, necessciates methods for efficient retrieval
of a subset of the data. An index datastructure provides a very effective method of increasing efficiency
of subset data retrieval. Over the years other groups have come up with wrapper indexing APIs to serve
this purpose. Hence a need was felt for a standardized, portable indexing API to be built into the HDF
API so that the duplication of effort to create such APIs could be avoided.
An introduction to the Python programming language and its numerical abilities will be presented. With this background, Andrew Collette's H5Py module--an HDF5-Python interface--will be explained highlighting the unique and useful similarities between Python data structures and HDF5.
This is an introductory slide for accessing NASA HDF/HDF-EOS data for beginners. NASA distributes many Earth Science data in HDF/HDF-EOS file format and new users struggle to understand the file format and use the NASA HDF/HDF-EOS data properly. This brief presentation will help new users to understand the basic concepts about the HDF/HDF-EOS and to know the available tools that can access the NASA data easily.
This tutorial is designed for new HDF5 users. We will cover HDF5 abstractions such as datasets, groups, attributes, and datatypes. Simple C examples will cover the programming model and basic features of the API, and will give new users the knowledge they need to navigate through the rich collection of HDF5 interfaces. Participants will be guided through an interactive demonstration of the fundamentals of HDF5.
This tutorial is for new HDF5 users.
This 2009 tutorial slide will cover basic HDF5 Data Model objects and their properties. It will include an overview of the HDF5 Libraries and APIs, and describe the HDF5 programming model. Simple programming examples and the HDFView data browser will be used to illustrate HDF5 concepts and start developing your own HDF5 based applications.
This tutorial is for new HDF5 users.
This Tutorial gives a brief introduction to HDF5 for people who have never used it. It covers the HDF5 Data Model including HDF5 objects and their properties. It also briefly describes the HDF5 Programming Model and prepares participants for further self-study of HDF5 and hands-on sessions.
This tutorial is designed for the HDF5 users with some HDF5 experience.
It will cover advanced features of the HDF5 library for achieving better I/O performance and efficient storage. The following HDF5 features will be discussed: partial I/O, chunked storage layout, compression and other filters including new n-bit and scale+offset filters. Significant time will be devoted to the discussion of complex HDF5 datatypes such as strings, variable-length datatypes, array and compound datatypes.
This tutorial is designed for new HDF5 users. We will go over a brief history of HDF and HDF5 software, and will cover basic HDF5 Data Model objects and their properties; we will give an overview of the HDF5 Libraries and APIs, and discuss the HDF5 programming model. Simple C and Fortran examples, and Java tool HDFView will be used to illustrate HDF5 concepts.
This tutorial targets HDF5 application developers and users who still use HDF5 1.6 releases and anyone who is interested in the HDF5 1.8.x libraries features. We will discuss how applications written for versions 1.6.x and earlier can be seamlessly moved to use the latest HDF5 releases. We will also talk about new features of the 1.8.x HDF5 Library such as redesigned group object, links, creation order, and different performance tuning knobs.
Efficient analysis of large scientific datasets often requires a means to rapidly search and select interesting portions of data
based on ad-hoc search criteria. We present our work on integrating an efficient searching technology named
FastBit
[2, 3]
with HDF5. The integrated system named
HDF5-FastQuery
allows the users to efficiently generate complex selections on
HDF5 datasets using compound range queries of the form
(
temperature>
1000)
AND
(70
<pressure><
90)
. The FastBit
technology generates compressed bitmap indices that accelerate searches on HDF5 datasets and can be stored together with
those datasets in an HDF5 file. Compared with other indexing schemes, compressed bitmap indices are compact and very well
suited for searching over multidimensional data – even for arbitrarily complex combinations of range conditions.
The increasing sizes of the data being stored in HDF files, necessciates methods for efficient retrieval
of a subset of the data. An index datastructure provides a very effective method of increasing efficiency
of subset data retrieval. Over the years other groups have come up with wrapper indexing APIs to serve
this purpose. Hence a need was felt for a standardized, portable indexing API to be built into the HDF
API so that the duplication of effort to create such APIs could be avoided.
It will cover features of the HDF5 library for achieving better I/O performance and efficient storage. The following HDF5 features will be discussed: datatype and partial I/O
This tutorial is for persons who are already familiar with HDF5 and wish to take advantage is some of its advanced features.
This tutorial is designed for new HDF5 users. We will cover basic HDF5 Data Model objects and their properties, give an overview of the HDF5 Libraries and APIs, and discuss the HDF5 programming model. Simple C and Fortran examples will be used to illustrate HDF5 concepts.
This Tutorial is designed for the users who have exposure to MPI I/O and basic concepts of HDF5 and would like to learn about Parallel HDF5 Library. The Tutorial will cover Parallel HDF5 design and programming model. Several C and Fortran examples will be used to illustrate the basic ideas of the Parallel HDF5 programming model. Some performance issues including collective chunked I/O will be discussed. Participants will work with the Tutorial examples and exercises during the hands-on sessions.
This Tutorial is designed for new HDF5 users. We will cover basic HDF5 Data Model objects and their properties; we will give an overview of the HDF5 Libraries and APIs, and discuss the HDF5 programming model. Simple C and Fortran examples will be used to illustrate HDF5 concepts. Participants will work with the Tutorial examples and exercises during the hands-on sessions.
A preponderance of data from NASA's Earth Observing System (EOS) is archived in the HDF Version 4 (HDF4) format. The long-term preservation of these data is critical for climate and other scientific studies going many decades into the future. HDF4 is very effective for working with the large and complex collection of EOS data products. Unfortunately, because of the complex internal byte layout of HDF4 files, future readability of HDF4 data depends on preserving a complex software library that can interpret that layout. Having a way to access HDF4 data independent of a library could improve its viability as an archive format, and consequently give confidence that HDF4 data will be readily accessible forever, even if the HDF4 library is gone.
To address the need to simplify long-term access to EOS data stored in HDF4, a collaborative project between The HDF Group and NASA Earth Science Data Centers is implementing an approach to accessing data in HDF4 files based on the use of independent maps that describe the data in HDF4 files and tools that can use these maps to recover data from those files. With this approach, relatively simple programs will be able to extract the data from an HDF4 file, bypassing the need for the HDF4 library.
A demonstration project has shown that this approach is feasible. This involved an assessment of NASA�s HDF4 data holdings, and development of a prototype XML-based layout mapping language and tools to read layout maps and read HDF4 files using layout maps. Future plans call for a second phase of the project, in which the mapping tools and XML schema are made production quality, the mapping schema are integrated with existing XML metadata files in several data centers, and outreach activities are carried out to encourage and facilitate acceptance of the technology.
This tutorial is designed for users with some HDF5 experience. It will cover advanced features of the HDF5 library that can be used to achieve better I/O performance and more efficient storage. The following HDF5 features will be discussed: partial I/O; compression and other filters, including new n-bit and scale+offset filters and data storage options. Significant time will be devoted to the discussion of complex HDF5 datatypes such as strings, variable-length datatypes, array datatypes, and compound datatypes.
An update on HDF, including a status report on The HDF Group, an overview of recent changes to the HDF4 and HDF5 libraries and tools, plans for future releases, HDF Group projects and collaborations, and future plans.
This Tutorial is designed for the HDF5 users with some HDF5 experience. It will cover properties of the HDF5 objects that affect I/O performance and file sizes. The following HDF5 features will be discussed: partial I/O, chunking and compression, and complex HDF5 datatypes such as strings, variable-length arrays and compound datatypes.
We will also discuss references to objects and datasets regions and how they can be used for indexing. Participants will work with the Tutorial examples and exercises during the hands-on sessions.
The goal of this talk is to educate HDF5 users about backward and forward compatibility issues across releases of the HDF5 Library and versions of the HDF5 file format. We will discuss changes in the file format that were done to support new HDF5 features such as object creation order, compact groups, efficient access to the variable length data, UTF-8 encoding, external links, etc., and their implications on the HDF5 Library and users' applications.
Update on HDF, including recent changes to the software, upcoming releases, collaborations, future plans. Will include an overview of the upcoming HDF5 1.8 release, and updates on the netCDF4/HDF5 merge, HDF5 support for indexing, BioHDF, the HDF5-Storage Resource Broker project, and the HDF spin-off THG.
Numerous scientific teams use the HDF5 format to store very large datasets. Efficient use of this data in a distributed environment depends on client applications being able to read any subset of the data without transferring the entire file to the local machine. The goal of the HDF5-iRODS Project was to develop an HDF5-iRODS module for the iRODS datagrid server that supported this capability, and to apply the technology to an NCSA/SDSC Strategic Applications Program (SAP) project, FLASH.
A joint team from The HDF Group (representing NCSA) and the SDSC SRB group collaborated to accomplish the project goal. The team implemented five HDF5 microservices functions on the iRODS server, and developed an iRODS FLASH slice client application. The client implementation also includes a JNI interface that allows HDFView, a standard tool for browsing HDF5 files, to access HDF5 files stored remotely in iRODS. Finally, three new collection client/server calls were added to the iRODS APIs, making it easier for users to query the content of an iRODS collection.
In this Tutorial we will discuss different storage methods for the HDF5 files (split files, family of files, multi-files), and datasets (compressed, external, compact), and related filters and properties. This tutorial will introduce advanced features of HDF5, including:
o Property lists
o Compound datatypes
o hyperslab selections
o point selection
o references to objects and regions
o extendable datasets
o mounting files
group iterations
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Le nuove frontiere dell'AI nell'RPA con UiPath Autopilot™UiPathCommunity
In questo evento online gratuito, organizzato dalla Community Italiana di UiPath, potrai esplorare le nuove funzionalità di Autopilot, il tool che integra l'Intelligenza Artificiale nei processi di sviluppo e utilizzo delle Automazioni.
📕 Vedremo insieme alcuni esempi dell'utilizzo di Autopilot in diversi tool della Suite UiPath:
Autopilot per Studio Web
Autopilot per Studio
Autopilot per Apps
Clipboard AI
GenAI applicata alla Document Understanding
👨🏫👨💻 Speakers:
Stefano Negro, UiPath MVPx3, RPA Tech Lead @ BSP Consultant
Flavio Martinelli, UiPath MVP 2023, Technical Account Manager @UiPath
Andrei Tasca, RPA Solutions Team Lead @NTT Data
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
FIDO Alliance Osaka Seminar: The WebAuthn API and Discoverable Credentials.pdf
Advanced HDF5 Features
1. HDF5 Advanced Topics
Elena Pourmal
The HDF Group
The 15th HDF and HDF-EOS Workshop
April 17, 2012
April 17-19
HDF/HDF-EOS Workshop XV
1
2. Goal
• To learn about HDF5 features important for
writing portable and efficient applications using
H5Py
April 17-19
HDF/HDF-EOS Workshop XV
2
3. Outline
• Groups and Links
• Types of groups and links
• Discovering objects in an HDF5 file
• Datasets
• Datatypes
• Partial I/O
• Other features
• Extensibility
• Compression
April 17-19
HDF/HDF-EOS Workshop XV
3
5. Groups and Links
• Groups are containers for links (graph edges)
• Links were added in 1.8.0
• Warning: Many APIs in H5G interface are
obsolete - use H5L interfaces to discover and
manipulate file structure
April 17-19
HDF/HDF-EOS Workshop XV
5
6. Groups and Links
HDF5 groups
and links
organize
data objects.
/
Experiment Notes:
Serial Number: 99378920
Date: 3/13/09
Configuration: Standard 3
Every HDF5 file
has a root group
SimOut
Viz
lat | lon | temp
----|-----|----12 | 23 | 3.1
15 | 24 | 4.2
17 | 21 | 3.6
Timestep
36,000
April 17-19, 2012
HDF/HDF-EOS Workshop XV
6
Parameters
10;100;1000
7. Example h5_links.py
Different kinds of
links
/
links.h5
A
B
dangling
a
soft
a
External
Dataset can be “reached”
using three paths
/A/a
/a
/soft
April 17-19, 2012
HDF/HDF-EOS Workshop XV
dset.h5
Dataset is in a different file
7
8. Example h5_links.py
Different kinds of
links
/
links.h5
A
B
dangling
a
soft
Hard links “A” and “B” were created when groups were created
Hard link “a” was added to the root group and points to an existing dataset
Soft link “soft” points to the existing dataset (cmp. UNIX alias)
Soft link “dangling” doesn’t point to any object
April 17-19, 2012
HDF/HDF-EOS Workshop XV
8
9. Links
• Name
• Example: “A”, “B”, “a”, “dangling”, “soft”
• Unique within a group; “/” are not allowed in names
• Type
• Hard Link
• Value is object’s address in a file
• Created automatically when object is created
• Can be added to point to existing object
• Soft Link
• Value is a string , for example, “/A/a”, but can be
anything
• Use to create aliases
April 17-19
HDF/HDF-EOS Workshop XV
9
10. Links (cont.)
• Type
• External Link
• Value is a pair of strings , for example, (“dset.h5”,
“dset” )
• Use to access data in other HDF5 files
• Example: For NPP data products geo-location information
may be in a separate file
April 17-19
HDF/HDF-EOS Workshop XV
10
11. Links Properties
• Links Properties
• ASCII or UTF-8 encoding for names
• Create intermediate groups
• Saves programming effort
• C example
lcpl_id = H5Pcreate(H5P_LINK_CREATE);
H5Gcreate (fid, "A/B", lcpl_id, H5P_DEFAULT, H5P_DEFAULT);
• Group “A” will be created if it doesn’t exist
April 17-19
HDF/HDF-EOS Workshop XV
11
12. Operations on Links
•
•
•
•
•
•
See H5L interface in Reference Manual
Create
Delete
Copy
Iterate
Check if exists
April 17-19
HDF/HDF-EOS Workshop XV
12
13. Operations on Links
• APIs available for C and Fortran
• Use dictionary operations in Python
• Objects associated with links ARE NOT affected
• Deleting a link removes a path to the object
• Copying a link doesn’t copy an object
April 17-19
HDF/HDF-EOS Workshop XV
13
14. Example h5_links.py
Link a in A is removed
/
links.h5
A
B
dangling
a
soft
External
Dataset can be “reached”
using one paths
/a
dset.h5
Dataset is in a different file
April 17-19, 2012
HDF/HDF-EOS Workshop XV
14
15. Example h5_links.py
Link a in root is
removed
/
links.h5
A
B
dangling
soft
External
dset.h5
Dataset is unreachable
Dataset is in a different file
April 17-19, 2012
HDF/HDF-EOS Workshop XV
15
16. Groups Properties
• Creation properties
• Type of links storage
• Compact (in 1.8.* versions)
• Used with a few members (default under 8)
• Dense (default behavior)
• Used with many (>16) members (default)
• Tunable size for a local heap
• Save space by providing estimate for size of the storage
required for links names
• Can be compressed (in 1.8.5 and later)
• Many links with similar names (XXX-abc, XXX-d, XXXefgh, etc.)
• Requires more time to compress/uncompress data
April 17-19
HDF/HDF-EOS Workshop XV
16
17. Groups Properties
• Creation properties
• Links may have creation order tracked and indexed
• Indexing by name (default)
• A, B, a, dangling, soft
• Indexing by creation order (has to be enabled)
• A, B, a, soft, dangling
• http://www.hdfgroup.org/ftp/HDF5/examples/exam
ples-by-api/api18-c.html
April 17-19
HDF/HDF-EOS Workshop XV
17
18. Discovering HDF5 file’s structure
• HDF5 provides C and Fortran 2003 APIs for
recursive and non-recursive iterations over the
groups and attributes
• H5Ovisit and H5Literate (H5Giterate)
• H5Aiterate
• Life is much easier with H5Py (h5_visita.py)
import h5py
def print_info(name, obj):
print name
for name, value in obj.attrs.iteritems():
print name+":", value
f = h5py.File('GATMO-SATMS-npp.h5', 'r+')
f.visititems(print_info)
f.close()
April 17-19
HDF/HDF-EOS Workshop XV
18
19. Checking a path in HDF5
• HDF5 1.8.8 provides HL C and Fortran 2003 APIs
for checking if paths exists
• H5LTvalid_path (h5ltvalid_path_f)
• Example: Is there an object with a path /A/B/C/d ?
• TRUE if there is a path, FALSE otherwise
April 17-19
HDF/HDF-EOS Workshop XV
19
20. Hints
• Use latest file format (see
H5Pset_libver_bound function in RM)
• Save space when creating a lot of groups in
a file
• Save time when accessing many objects
(>1000)
• Caution: Tools built with the HDF5 versions prirt
to 1.8.0 will not work on the files created with this
property
April 17-19
HDF/HDF-EOS Workshop XV
20
23. HDF5 Datatypes
• Integer and floating point
• String
• Compound
• Similar to C structures or Fortran Derived Types
•
•
•
•
•
Array
References
Variable-length
Enum
Opaque
April 17-19
HDF/HDF-EOS Workshop XV
23
24. HDF5 Datatypes
• Datatype descriptions
• Are stored in the HDF5 file with the data
• Include encoding (e.g., byte order, size, and
floating point representation) and other
information to assure portability across
platforms
• See C, Fortran, MATLAB and Java
examples under
http://www.hdfgroup.org/ftp/HDF5/examples/
April 17-19
HDF/HDF-EOS Workshop XV
24
25. Data Portability in HDF5
Array of integers on Intel platform Array of long integers on SPARC64 platform
long is big-endian, 8 bytes
int is little-endian, 4 bytes
int
long
H5Dwrite
H5Dread
H5T_STD_I32LE
April 17-19
HDF/HDF-EOS Workshop XV
25
26. Data Portability in HDF5 (cont.)
We use native integer type to describe data in a
file
dset =
H5Dcreate(file,NAME,H5T_NATIVE_INT,…
Description of data in a buffer
H5Dwrite(dset,H5T_NATIVE_INT,…,buf);
H5Dread(dset,H5T_NATIVE_LONG,…, buf);
Description of data in a buffer; library will perform
Conversion from 4 byte LE to 8 byte BE integer
April 17-19
HDF/HDF-EOS Workshop XV
26
27. Hints
• Avoid datatype conversion if possible
• Store necessary precision to save space in
a file
• Starting with HDF5 1.8.7, Fortran APIs
support different kinds of integers and floats
(if Fortran 2003 feature is enabled)
April 17-19
HDF/HDF-EOS Workshop XV
27
29. HDF5 Strings
• Fixed length
• Data elements has to have the same size
• Short strings will use more byte than needed
• Application responsible for providing buffers of the
correct size on read
• Variable length
• Data elements may not have the same size
• Writing/reading strings is “easy”; library handles
memory allocations
April 17-19
HDF/HDF-EOS Workshop XV
29
30. HDF5 Strings – Fixed-length
• Example h5_string.py(c,f90)
fixed_string = np.dtype('a10')
dataset = file.create_dataset("DSfixed",(4,), dtype=fixed_string)
data = ("Parting", ".is such", ".sweet", ".sorrow...")
dataset[...] = data
• Stores fours strings “Parting", ” .is such", ” .sweet",
”.sorrow…” in a dataset.
• Strings have length 10
• Python uses NULL padded strings (default)
April 17-19
HDF/HDF-EOS Workshop XV
30
31. HDF5 Strings
• Example h5_vlstring.py(c,f90)
str_type = h5py.new_vlen(str)
dataset = file.create_dataset("DSvariable",(4,), dtype=str_type)
data = ("Parting", " is such", " sweet", " sorrow...")
dataset[...] = data
• Stores fours strings “Parting", ” is such", ” sweet",
”sorrow…” in a dataset.
• Strings have length 7, 8, 6, 10
April 17-19
HDF/HDF-EOS Workshop XV
31
32. Hints
• Fixed length strings
• Can be compressed
• Use when need to store a lot of strings
• Variable-length strings
• Compression cannot be applied to data
• Use for attributes and a few strings if space is a
concern
April 17-19
HDF/HDF-EOS Workshop XV
32
34. HDF5 Compound Datatypes
• Compound types
• Comparable to C structures or Fortran 90
Derived Types
• Members can be of any datatype
• Data elements can written/read by a single field
or a set of fields
April 17-19
HDF/HDF-EOS Workshop XV
34
35. Creating and Writing Compound Dataset
• Example h5_compound.py(c,f90)
• Stores four records in the dataset
Orbit
integer
Location
string
Temperature (F)
64-bit float
Pressure (inHg)
64-bit-float
1153
Sun
53.23
24.57
1184
Moon
55.12
22.95
1027
Venus
103.55
31.33
1313
Mars
1252.89
84.11
April 17-19
HDF/HDF-EOS Workshop XV
35
36. Creating and Writing Compound Dataset
comp_type = np.dtype([('Orbit’,'i'),('Location’,np.str_, 6),
….)
dataset = file.create_dataset("DSC",(4,), comp_type)
dataset[...] = data
Note for C and Fortran2003 users:
• You’ll need to construct memory and file datatypes
• Use HOFFSET macro instead of calculating offset by hand.
• Order of H5Tinsert calls is not important if HOFFSET is used.
April 17-19
HDF/HDF-EOS Workshop XV
36
37. Reading Compound Dataset
f = h5py.File('compound.h5', 'r')
dataset = f ["DSC"]
….
orbit = dataset['Orbit']
print "Orbit: ", orbit
data = dataset[...]
print data
….
print dataset[2, 'Location']
April 17-19
HDF/HDF-EOS Workshop XV
37
38. Fortran 2003
• HDF5 Fortran library 1.8.8 with Fortran 2003
enabled has the same capabilities for writing
derived types as C library
• H5OFFSET function
• No need to write/read by fields as before
April 17-19
HDF/HDF-EOS Workshop XV
38
39. Hints
• When to use compound datatypes?
• Application needs access to the whole record
• When not to use compound datatypes?
• Application needs access to specific fields often
• Store the field in a dataset
/
/
DSC
Pressure
Orbit
Location
Temperature
April 17-19
HDF/HDF-EOS Workshop XV
39
41. References to Objects and Dataset Regions
/
Test Data
Viz
References to HDF5
Objects
References to dataset regions
.
Group
Image 2…..
Image 3…..
April 17-19, 2012
HDF/HDF-EOS Workshop XV
41
.
42. Reference Datatypes
• Object Reference
• Unique identifier of an object in a file
• HDF5 predefined datatype
H5T_STD_REG_OBJ
• Dataset Region Reference
• Unique identifier to a dataset + dataspace
selection
• HDF5 predefined datatype
H5T_STD_REF_DSETREG
April 17-19
HDF/HDF-EOS Workshop XV
42
43. Conceptual view of HDF5 NPP file
XML User’s Block
Product Group
Root - /
Agg
Reference
Object
Data
Gran n
Reference
Region
Reference
Region
43
44. NPP HDF5 file in HDFView
April 17-19
HDF/HDF-EOS Workshop XV
44
45. HDF5 Object References
• h5_objref.py (c,f90)
• Creates a dataset with object references
1.
2.
3.
4.
group = f.create_group("G1")
Scalar dataspace
dataset = f.create_dataset("DS2",(), 'i')
# Create object references to a group and a dataset
refs = (group.ref, dataset.ref)
5. ref_type = h5py.h5t.special_dtype(ref=h5py.Reference)
6. dataset_ref = file.create_dataset("DS1", (2,),ref_type)
7. dataset_ref[...] = refs
April 17-19
HDF/HDF-EOS Workshop XV
45
46. HDF5 Object References (cont.)
• h5_objref.py (c,f90)
• Finding the object a reference points to:
1.
2.
3.
4.
5.
6.
f = h5py.File('objref.h5','r')
dataset_ref = f["DS1"]
print h5py.h5t.check_dtype(ref=dataset_ref.dtype)
refs = dataset_ref[...]
refs_list = list(refs)
for obj in refs_list:
print
April 17-19
f[obj]
HDF/HDF-EOS Workshop XV
46
47. HDF5 Dataset Region References
• h5_regref.py (c,f90)
• Creates a dataset with region references to each
row in a dataset
1.
2.
3.
4.
refs = (dataset.regionref[0,:],…,dataset.regionref[2,:])
ref_type = h5py.h5t.special_dtype(ref=h5py.RegionReference)
dataset_ref = file.create_dataset("DS1", (3,),ref_type)
dataset_ref[...] = refs
April 17-19
HDF/HDF-EOS Workshop XV
47
48. HDF5 Dataset Region References (cont.)
• h5_regref.py (c,f90)
• Finding a dataset and a data region pointed by a
region reference
1.
2.
3.
4.
5.
6.
path_name = f[regref].name
print path_name
# Open the dataset using the pathname we just found
data = file[path_name]
# Region reference can be used as a slicing argument!
print data[regref]
April 17-19
HDF/HDF-EOS Workshop XV
48
49. Hints
• When to use HDF5 object references?
• Instead of an attribute with a lot of data
• Create an attribute of the object reference type and
point to a dataset with the data
• In a dataset to point to related objects in HDF5 file
• When to use HDF5 region references?
• In datasets and attributes to point to a region of
interest
• When accessing the same region many times to
avoid hyperslab selection process
April 17-19
HDF/HDF-EOS Workshop XV
49
51. Collect data one way ….
Array of images (3D)
April 17-19
HDF/HDF-EOS Workshop XV
51
52. Display data another way …
Stitched image (2D array)
April 17-19
HDF/HDF-EOS Workshop XV
52
53. Data is too big to read….
April 17-19
HDF/HDF-EOS Workshop XV
53
54. How to Describe a Subset in HDF5?
• Before writing and reading a subset of data
one has to describe it to the HDF5 Library.
• HDF5 APIs and documentation refer to a
subset as a “selection” or “hyperslab
selection”.
• If specified, HDF5 Library will perform I/O on a
selection only and not on all elements of a
dataset.
April 17-19
HDF/HDF-EOS Workshop XV
54
55. Types of Selections in HDF5
• Two types of selections
• Hyperslab selection
• Regular hyperslab
• Simple hyperslab
• Result of set operations on hyperslabs
(union, difference, …)
• Point selection
• Hyperslab selection is especially important for
doing parallel I/O in HDF5 (See Parallel HDF5
Tutorial)
April 17-19
HDF/HDF-EOS Workshop XV
55
59. Hyperslab Description
• Start - starting location of a hyperslab (1,1)
• Stride - number of elements that separate each
block (3,2)
• Count - number of blocks (2,6)
• Block - block size (2,1)
• Everything is “measured” in number of elements
April 17-19
HDF/HDF-EOS Workshop XV
59
60. Simple Hyperslab Description
• Two ways to describe a simple hyperslab
• As several blocks
• Stride – (1,1)
• Count – (3,4)
• Block – (1,1)
• As one block
• Stride – (1,1)
• Count – (1,1)
• Block – (3,4)
No performance penalty for
one way or another
April 17-19
HDF/HDF-EOS Workshop XV
60
61. Writing and Reading a Hyperslab
• Example h5_hype.py(c, f90)
• Creates 8x10 integer dataset and populates with data; writes
a simple hyperslab (3x4) starting at offset (1,2)
• H5Py uses NumPy indexing to specify a hyperslab
• Numpy indexing array[i : j : k]
• i – the starting index; j – the stopping index; k – is the step (≠ 0)
dataset[1:4, 2:6]
offset
April 17-19
count+offset
HDF/HDF-EOS Workshop XV
61
64. Writing and Reading Point Selection
• Example h5_selecelem.py(c, f90)
• Creates 2 integer datasets and populates with data; writes a
point selection at locations (0,1) and (0, 3)
• H5Py uses NumPy indexing to specify points in array
val = (55,59)
dataset2[0, [1,3]] = val
[[ 1 55
[ 1 1
[ 1 1
April 17-19
1 59]
1 1]
1 1]]
HDF/HDF-EOS Workshop XV
64
65. Hints
• C and Fortran
• Applications’ memory grows with the number of
open handles.
• Don’t keep dataspace handles open if
unnecessary, e.g., when reading hyperslab in a
loop.
• Make sure that selection in a file has the same
number of elements as selection in memory when
doing partial I/O.
April 17-19
HDF/HDF-EOS Workshop XV
65
67. Dataset Storage Options
• Compact
• Used for storing small (a few Ks) data
• Contiguous (default)
• Used for accessing contiguous subsets of data
• Chunked
• Data is store in chunks of predefined size
• Used when:
• Appending data
• Compressing data
• Accessing non-contiguous data (e.g., columns)
April 17-19
HDF/HDF-EOS Workshop XV
67
68. HDF5 Dataset
Metadata
Dataset data
Dataspace
Rank Dimensions
3
Dim_1 = 4
Dim_2 = 5
Dim_3 = 7
Datatype
IEEE 32-bit float
Attributes
Storage info
Time = 32.4
Chunked
Pressure = 987
Compressed
Temp = 56
April 17-19
HDF/HDF-EOS Workshop XV
68
69. Examples of Data Storage
Compact
Metadata
Raw data
Contiguous
April 17-19
HDF/HDF-EOS Workshop XV
Chunked
69
70. Extending HDF5 dataset
• Example h5_unlim.py(c,f90)
• Creates a dataset and appends rows and columns
• Dataset has to be chunked
• Chunk sizes do not need to be factors of the dimension sizes
dataset = f.create_dataset('DS1',(4,7),'i',chunks=(3,3),
maxshape=(None, None))
0
0
0
0
0
0
April 17-19
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
HDF/HDF-EOS Workshop XV
0
0
0
0
0
0
0
0
0
0
0
0
70
74. Compressing HDF5 dataset
• Example h5_gzip.py(c,f90)
• Creates compressed dataset using GZIP compression
with effort level 9
• Dataset has to be chunked
• Write/read/subset as for contiguous (no special steps are
needed)
dataset =
f.create_dataset('DS1',(32,64),'i',chunks=(4,8),compressi
on='gzip',compression_opts=9)
dataset[…] = data
April 17-19
HDF/HDF-EOS Workshop XV
74
75. Hints
• Do not make chunk sizes too small (e.g., 1x1)!
• Metadata overhead for each chunk (file space)
• Each chunk is read at once
• Many small reads are inefficient
• Some software (H5Py, netCDF-4) may pick up
chunk size for you; may not be what you need
• Example: Modify h5_gzip.py to use
dataset =
file.create_dataset('DS1',(32,64),'i',compression='gzip
',compression_opts=9)
Run h5dump –p –H gzip.h5 to check chunk size
April 17-19
HDF/HDF-EOS Workshop XV
75
76. More Information
• More detailed information on chunking can be
found in the “Chunking in HDF5” document at:
http://www.hdfgroup.org/HDF5/doc/Advanced/Chunking/index.html
April 17-19
HDF/HDF-EOS Workshop XV
76
78. Acknowledgements
This work was supported by cooperative agreement
number NNX08AO77A from the National
Aeronautics and Space Administration (NASA).
Any opinions, findings, conclusions, or
recommendations expressed in this material are
those of the author[s] and do not necessarily reflect
the views of the National Aeronautics and Space
Administration.
April 17-19
HDF/HDF-EOS Workshop XV
78
Example h5_links.py creates a file links.h5 and two groups “A” and “B” in it.Then it creates a one-dimensional array “a” in group “A”. After the datasets was created a hard link “a” was added to the root group. (It is one dimensional in example and doesn’t have data). Also soft link with the value “/A/a” was added to the root group along with the dangling soft link “dangling”.External link “External” was added to group B. It points to a dataset “dset” in dset.h5.
When link a in A is removed dataset can be reached using path /a, soft link becomes dangling
When link a in the root group is removed, dataset with graph becomes unreachable; soft link cannot be resolved without a “real” path
Portability: insuresendianess conversion, size conversion, structures portability, etc.
Failure to describe correctly memory buffer or to provide “unconvertible” type will result in wrong data read/written
This slide is from HDFand HDF-EOS Workshop IX, Kim Tomashosky “NPOESS Product Delivery in HDF”