Download as PDF, PPTX
More Related Content
Similar to Python and HDF5: Overview
Python and HDF5: Overview
- 1. Python and HDF5 AndrewCollette University of Colorado
- 2. What makes scientificdata special?
- 3. What makes scientificdata special? It’s meant to be shared - collaborative Ad-hoc or changing structure - flexible Archived and preserved - robust Python and HDF5 together address all three
- 4. High-level language Fully object-oriented Almostno “boilerplate” code Readable Free (the language) “Exception” error handling Self-documenting First-class module/namespace support
- 5. (the platform) Mature numerical,plotting and scientific modules Hundreds of specialized science packages Thousands more general-purpose Python itself is “batteries included”
- 6. Core analysis packages NumPy- Array objects and basic operations SciPy - Advanced science & engineering library Matplotlib - Publication-quality plots (both rendered and interactive)
- 7. Thousands of others Distribution- distutils/pip single-command installs Unit testing - unittest module in stdlib Interface: F2PY (Fortran), Cython (C), ctypes, others Web servers and development - literally hundreds Only need to write code for your problem
- 8.
- 9.
- 10.
- 11.
- 12. Speed FFTs and optimizedroutines built in to NumPy/Scipy
- 13. Speed FFTs and optimizedroutines built in to NumPy/Scipy ctypes and Cython
- 14. ctypes Advanced foreign functioninterface Call C libraries from pure Python code
- 15. Cython Example from theHDF5 C Library:
- 16.
- 17.
- 18. Hierarchical Data Format 3things: File specification and object model C library Ecosystem of users and developers
- 19. Objects Datasets - Homogenousarrays of data Groups: containers holding datasets and groups Attributes: arbitrary metadata on groups & datasets Standard constructs using these, or make your own!
- 20. Dataset features Partial I/O:read and write just what you want (In Python, we even use the array-access syntax!) Automatic type conversion On-the-fly compression Parallel reads & writes with MPI (Directly from Python!)
- 21. Metadata & Organization Groupsform a POSIX-style “filesystem” in the file Attributes can store arbitrary data on arbitrary objects How should the file be organized? You decide! ! Thousands of domain-specific “application formats” Anyone can read them because HDF5 is self-describing!
- 22.
- 23. Open an HDF5file Extract a particular dataset Read the data Make an interactive plot Close the file
- 24. Open an HDF5file Extract a particular dataset Read the data Make an interactive plot Close the file
- 25. Open an HDF5file Extract a particular dataset Read the data Make an interactive plot Close the file
- 26. Open an HDF5file Extract a particular dataset Read the data Make an interactive plot Close the file
- 27. Open an HDF5file Extract a particular dataset Read the data Make an interactive plot Close the file
- 28. Open an HDF5file Extract a particular dataset Read the data Make an interactive plot Close the file
- 29.
- 30.
- 31. UCLA Large PlasmaDevice
- 32. UCLA Large PlasmaDevice Image credit: Basic Plasma Science Facility
- 33. Laser Experiment Image credit:Basic Plasma Science Facility
- 34. LAPD Data Products Acquisitionfile - “Planes” of data in HDF5 Metadata: timestamps, digitizer settings, probe positions, background plasma conditions… Packaged into HDF5 following “lab layout” Users take their data back home and analyze
- 35.
- 36. Python 2D plotting A.Collette et al. Phys. Rev. Lett 105, 195003 (2010)
- 37. Only 160 linesof code! A. Collette et al. Phys. Rev. Lett 105, 195003 (2010)
- 38. Python does 3Dtoo! “MayaVi” 3D visualizer Development sponsored by Enthought Both offline (scripted) and interactive modes A. Collette et al. Phys. Plasmas 18, 055705 (2011)
- 39.
- 40.
- 41.
- 42.
- 43. CU Accelerator Raw data HDF5Shot file Automated speed/mass calculation Data search HDF5 file for user MySQL
- 44. Where to getPython
- 45. Where to getPython Distributions are the best way to get started (they include HDF5/h5py!) Anaconda (Windows, Mac, Linux): http://continuum.io PythonXY (Windows) http://pythonxy.googlecode.com
- 46.