Organizing the Data Chaos of Scientists


Published on

PyCon UK 2008 (12.-14. September 2008, Birmingham)

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • Organizing the Data Chaos of Scientists

    1. 1. DataFinder: Organizing the Data Chaos of Scientists PyCon UK 2008 (September 12 th , 2008, Birmingham) Andreas Schreiber <> German Aerospace Center (DLR), Cologne
    2. 2. <ul><li>The DLR </li></ul><ul><li>German Aerospace Research Center </li></ul><ul><li>Space Agency of the Federal Republic of Germany </li></ul>
    3. 3. <ul><li>5,700 employees working in 29 research institutes and facilities </li></ul><ul><ul><li> at 13 sites . </li></ul></ul><ul><li>Offices in Brussels, Paris and Washington. </li></ul>Sites and employees  Köln  Lampoldshausen  Stuttgart  Oberpfaffenhofen Braunschweig   Göttingen Berlin -   Bonn Trauen   Hamburg  Neustrelitz Weilheim  Bremen - 
    4. 4. Short Overview <ul><li>DataFinder is a software for efficient management of scientific and technical data </li></ul><ul><li>Focus on huge data sets </li></ul><ul><li>Development by DLR </li></ul><ul><li>Primary functionality </li></ul><ul><ul><li>Structuring of data through assignment of meta information and self-defined data models </li></ul></ul><ul><ul><li>Flexible usage of heterogeneous storage resources </li></ul></ul><ul><ul><li>Integration in the working environment </li></ul></ul>
    5. 5. Introduction <ul><li>DataFinder founded by DLR </li></ul><ul><li>National Grid project AeroGrid </li></ul>
    6. 6. Introduction Background <ul><li>Large-scale simulations </li></ul><ul><li>aerodynamics </li></ul><ul><li>material science </li></ul><ul><li>climate </li></ul><ul><li>… </li></ul><ul><li>Tons of measured data </li></ul><ul><li>wind-tunnel experiments </li></ul><ul><li>earth observations </li></ul><ul><li>traffic data </li></ul><ul><li>… </li></ul>
    7. 7. Introduction Data Management Problem <ul><li>Typical organizational situations </li></ul><ul><li>No central data management policy </li></ul><ul><li>Every employee organizes his/her data individually </li></ul><ul><li>Researchers spend about 30% of their time searching for data </li></ul><ul><li>Problem with data left behind by temporary staff </li></ul><ul><li>Increase of data size and regulations </li></ul><ul><li>Rapidly growing volume of simulation and experimental data </li></ul><ul><li>Legal requirements for long-term availability of data (up to 50 years!) </li></ul><ul><li>Situation similar at many organizations </li></ul><ul><li>All ~30 DLR institutes </li></ul><ul><li>Other research labs and agencies </li></ul><ul><li>Industry </li></ul>
    8. 8. DataFinder History Search for solution for scientific data management <ul><li>Definition of “standard problem” (helicopter simulation) </li></ul><ul><li>Test case for evaluation of software </li></ul><ul><li>Evaluation of commercial product data management (PDM) systems </li></ul><ul><li>PDM systems could manage data but with huge amount of costs </li></ul><ul><li>PDM systems have many unneeded functionalities </li></ul><ul><li>PDM systems have self-defined or unreadable scripting languages for extension and customization (Tcl etc.) </li></ul><ul><li>Development of DataFinder </li></ul><ul><li>Lightweight data management client and existing server solution </li></ul><ul><li>Just enough functionality for our problems (no paid but unused features!) </li></ul>
    9. 9. DataFinder Development From Java Prototype to Python Product… <ul><li>Development of prototype in Java </li></ul><ul><li>Data could be manages with prototype successfully </li></ul><ul><li>Drawbacks: Java problems on important platforms (e.g., SGI IRIX) </li></ul><ul><li>Embedded Jython interpreter great feature for users </li></ul><ul><li>User: “ The Java GUI is like shit, but the Python scripting is great. We want a pure Python solution! ” </li></ul><ul><li>Development of DataFinder product from scratch in Python </li></ul>
    10. 10. Python for Scientists and Engineers Reasons for Python in Research and Industry <ul><li>Observations : </li></ul><ul><li>Scientists and engineers don’t want to write software but just solve their problems </li></ul><ul><li>If they have to write code, it must be as easy as possible </li></ul><ul><li>Why Python is perfect? </li></ul><ul><li>Very easy to learn and easy to use ( = steep learning curve ) </li></ul><ul><li>Allows rapid development ( = short development time ) </li></ul><ul><li>Inherent great maintainability </li></ul>“ I want to design planes, not software!”
    11. 11. “ Python has the cleanest, most-scientist- or engineer friendly syntax and semantics. Paul F. Dubois Paul F. Dubois. Ten good practices in scientific programming. Comp. In Sci. Eng., Jan/Feb 1999, pp.7-11
    12. 12. DataFinder Overview Basic Concept <ul><li>Client-Server solution </li></ul><ul><li>Based on open and stable standards , such as XML and WebDAV </li></ul><ul><li>Extensive use of standard software components (open source / commercial), limited own development at client side </li></ul>
    13. 13. WebDAV Web-based Distributed Authoring & Versioning <ul><li>Extension of HTTP </li></ul><ul><li>Allows to manage files on remote servers collaboratively </li></ul><ul><li>WebDAV supports </li></ul><ul><ul><li>Resources (“files”) </li></ul></ul><ul><ul><li>Collections (“directories”) </li></ul></ul><ul><ul><li>Properties (“meta data”, in XML format) </li></ul></ul><ul><ul><li>Locking </li></ul></ul><ul><li>WebDAV extensions </li></ul><ul><ul><li>Versioning (DeltaV) </li></ul></ul><ul><ul><li>Access control (ACP) </li></ul></ul><ul><ul><li>Search (DASL) </li></ul></ul>
    14. 14. DataFinder Overview Client and Server <ul><li>Client </li></ul><ul><li>User client </li></ul><ul><li>Administrator client </li></ul><ul><li>Implementation: Python with Qt </li></ul><ul><li>Server </li></ul><ul><li>WebDAV server for meta data and data structure </li></ul><ul><li>Data Store concept </li></ul><ul><ul><li>Abstracts access to managed data </li></ul></ul><ul><ul><li>Flexible usage of heterogeneous storage resources </li></ul></ul><ul><li>Implementation: Various existing server solutions (third-party) </li></ul>
    15. 15. DataFinder Client Graphical User Interfaces User Client Administrator Client Implementation in Python with Qt/PyQt
    16. 16. DataFinder Server Supported WebDAV servers <ul><li>Commercial Server Solution </li></ul><ul><li>Tamino XML database (Software AG) </li></ul><ul><li>Open Source Server Solutions </li></ul><ul><li>Apache HTTP Web server and module mod_dav </li></ul><ul><ul><li>Default storage: file system (mod_dav_fs) </li></ul></ul><ul><ul><li>Module Catacomb (mod_dav_repos) + Relational database ( ) </li></ul></ul>
    17. 17. WebDAV / Meta Data Server (1) Tamino WebDAV Server <ul><li>Commercial Server Solution (Software AG) </li></ul><ul><ul><li>WebDAV Server </li></ul></ul><ul><ul><li>Tamino XML database backend </li></ul></ul><ul><li>Advantages </li></ul><ul><ul><li>Implements many WebDAV extensions (DASL, DeltaV, ACLs) </li></ul></ul><ul><ul><li>Fast XML processing </li></ul></ul><ul><li>Good, but not free  </li></ul><ul><li>Used in DLR for use with DataFinder </li></ul><ul><ul><li>One installation sufficient for many institutes </li></ul></ul>
    18. 18. WebDAV / Meta Data Server (2) Apache + mod_dav <ul><li>Open Source solution (Apache Group) </li></ul><ul><ul><li>Apache HTTP Web server </li></ul></ul><ul><ul><li>WebDAV extension module mod_dav </li></ul></ul><ul><ul><li>File system + (G)DBM database </li></ul></ul><ul><li>Advantage: Free and easy to install  </li></ul><ul><li>… but some WebDAV features are not supported </li></ul><ul><ul><li>No searching and versioning  </li></ul></ul>Apache Core Server mod_http mod_auth_ldap mod_dav mod_dav_fs File system
    19. 19. WebDAV / Meta Data Server (3) Catacomb <ul><li>Open Source solution </li></ul><ul><ul><li>Apache HTTP Web server + mod_dav </li></ul></ul><ul><ul><li>Module Catacomb (replacement for file system) </li></ul></ul><ul><ul><li>Relational database </li></ul></ul><ul><li>Search and versioning implemented: Uses database search features </li></ul><ul><li>Open Source development at DLR ( ) </li></ul>Apache Core Server mod_http mod_auth_ldap mod_dav mod_dav_fs File system DB (MySQL) Catacomb mod_dav_repos
    20. 20. Mass Data Storage Data Stores Logical View User Client Storage Locations
    21. 21. DataFinder Technical Aspects <ul><li>Access privilege management </li></ul><ul><li>Authentication using WebDAV and LDAP </li></ul><ul><li>Authorization for users and groups based on WebDAV (ACP) </li></ul><ul><li>Client available on many platforms </li></ul><ul><li>Linux, Windows, … </li></ul><ul><li>Restricted by availability of Python 2.5 and Qt 3 + PyQt </li></ul><ul><li>Extensible through Python scripts </li></ul><ul><li>Python application programming interface (API) </li></ul><ul><li>Accessing data and meta data </li></ul>
    22. 22. Python API User Client Extension with GUI import threading from datafinder.application import search_support from datafinder.gui.user import facade def searchAndDisplayResult(): &quot;&quot;&quot;Searches and displays the result in the search result logging window. &quot;&quot;&quot; query = &quot;displayname contains ‘test’ OR displayname == ‘ab’&quot; result = search_support.performSearch(query) resultLogger = facade.getSearchResultLogger() for path in result.keys(): &quot;Found item %s.&quot; % path) thread = threading.Thread(target=searchAndDisplayResult) thread.start()
    23. 23. Python API Command Line Example (without GUI) # Get API from datafinder.application import ExternalFacade externalFacade = ExternalFacade.getInstance() # Connect to a repository externalFacade.performBasicDatafinderSetup(username, password, startUrl) # Download the whole content rootItem = externalFacade.getRootWebdavServerItem() items = externalFacade.getCollectionContents(rootItem) for item in items: externalFacade.downloadFile(item, baseDirectory)
    24. 24. Additional “Batteries”… Used Libraries beyond the Python Standard Library (1) <ul><li>PyQt ( </li></ul><ul><ul><li>Interface to the Qt GUI framework (currently Qt 3) </li></ul></ul><ul><ul><li>Used for DataFinder UI layer </li></ul></ul><ul><li>Pyparsing ( </li></ul><ul><ul><li>Creating and executing simple grammars </li></ul></ul><ul><ul><li>Used for highlighting search expressions </li></ul></ul><ul><li>python-ldap ( </li></ul><ul><ul><li>Object-oriented API to access LDAP servers </li></ul></ul><ul><ul><li>Authentication against LDAP / ActiveDirectory server </li></ul></ul><ul><li>paramiko ( </li></ul><ul><ul><li>SSH2 protocol implementation </li></ul></ul>
    25. 25. Additional “Batteries”… Used Libraries beyond the Python Standard Library (2) <ul><li>PyGlobus ( </li></ul><ul><ul><li>Interface to The Globus Toolkit </li></ul></ul><ul><ul><li>Used for GridFTP Data Store </li></ul></ul><ul><li>Boto ( </li></ul><ul><ul><li>Interfaces to Amazon Web Services </li></ul></ul><ul><ul><li>Used for S3 (Simple Storage Service) Data Store </li></ul></ul><ul><li>davlib ( ) </li></ul><ul><ul><li>WebDAV client library </li></ul></ul><ul><ul><li>Used for core WebDAV functions </li></ul></ul>
    26. 26. WebDAV Client Library Support for DAV Extensions <ul><li>Provides an object-oriented interface for accessing WebDAV server </li></ul><ul><ul><li>Extracted from DataFinder source </li></ul></ul><ul><li>WebDAV client-side library supports </li></ul><ul><ul><li>Core WebDAV specification </li></ul></ul><ul><ul><li>Access Control Protocol </li></ul></ul><ul><ul><li>Basic Versioning (experimental) </li></ul></ul><ul><ul><li>DAV Searching and Locating </li></ul></ul><ul><ul><li>Secure HTTP connections </li></ul></ul><ul><li>Implementation based on davlib and standard httplib </li></ul><ul><li>Apache License Version 2 </li></ul><ul><li>Project Site: </li></ul>
    27. 27. Simple Use Case: File Upload and Search
    28. 40. Working with DataFinder…
    29. 41. Configuration and Customization Preparing DataFinder for certain “use cases” <ul><li>Requirements Analysis </li></ul><ul><li>Analyze data, working environment, and users workflows </li></ul><ul><li>Configuration </li></ul><ul><li>Define and configure data model </li></ul><ul><li>Configure distributed storage resources (Data Stores) </li></ul><ul><li>Customization </li></ul><ul><li>Write functional extensions with Python scripts </li></ul>
    30. 42. DataFinder Configuration Data Model and Data Stores <ul><li>Logical view to data </li></ul><ul><li>Definition of data structuring and meta data (“data model”) </li></ul><ul><li>Separated storage of data structure / meta data and actual data files </li></ul><ul><li>Flexible use of (distributed) storage resources </li></ul><ul><ul><li>File system, WebDAV, FTP, GridFTP </li></ul></ul><ul><ul><li>Amazon S3 (Simple Storage Service) </li></ul></ul><ul><ul><li>Tivoli Storage Manager (TSM) </li></ul></ul><ul><ul><li>Storage Resource Broker (SRB) </li></ul></ul><ul><li>Complex search mechanism to find data </li></ul>
    31. 43. Data Structure Mapping of Organizational Data Structures User Object (collection) Object (file) Relation Attributes (meta data) Project A Project B Project C File 1 File 2 Simulation I Experiment Simulation II Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value
    32. 44. Meta Data <ul><li>Describe and annotate data (“files”) and collections (“directories”) </li></ul><ul><li>Different levels of meta data </li></ul><ul><ul><li>Required attributes defined by administrator </li></ul></ul><ul><ul><li>User is free to choose additional ones </li></ul></ul><ul><li>Different types of meta data </li></ul><ul><ul><li>String </li></ul></ul><ul><ul><li>Numbers (float, double, …) </li></ul></ul><ul><ul><li>Lists </li></ul></ul><ul><ul><li>Pictures </li></ul></ul><ul><ul><li>Links </li></ul></ul><ul><li>Stored in XML format </li></ul><ul><li>User can search in meta data </li></ul>
    33. 45. Impact for Users <ul><li>DataFinder restricts the rights of users! </li></ul><ul><li>Enforcement of “good behavior” </li></ul><ul><li>User must comply to organizational standards </li></ul><ul><li>Data is stored in defined (directory) hierarchy on data server </li></ul><ul><li>Required meta data must be set prior upload </li></ul><ul><li>User have certain access rights within hierarchy </li></ul>“ Damn! I’m a great scientist! I want freedom to have my own directory layout…”
    34. 46. Customization Python-Scripting for Extension and Automation <ul><li>Integration of DataFinder with environment </li></ul><ul><li>User, infrastructure, software, … </li></ul><ul><li>Extension of DataFinder by Python scripts </li></ul><ul><li>Actions for resources (i.e., files, directories) </li></ul><ul><li>User interface extensions </li></ul><ul><li>Typical automations and customizations </li></ul><ul><li>Data migration and data import </li></ul><ul><li>Start of external application (with downloaded data files) </li></ul><ul><li>Extraction of meta data from result files </li></ul><ul><li>Automation of recurring tasks (“workflows”) </li></ul>
    35. 47. DataFinder Scripting Downloading File and Starting Application # Download the selected file and try to execute it. from datafinder.application import ExternalFacade from guitools.easygui import * import os from tempfile import * from win32api import ShellExecute # Get instance of ExternalFacade to access DataFinder API facade = ExternalFacade.getInstance() # Get currently selected collection in DataFinder Server-View resource = facade.getSelectedResource() if resource != None: tmpFile = mktemp( facade.downloadFile(resource, tmpFile) if os.path.exists(tmpFile): ShellExecute(0, None, tmpFile, &quot;&quot; , &quot;&quot; , 1) else : msgbox( &quot;No file selected to execute.&quot; )
    36. 48. Examples…
    37. 49. Example 1: Turbine Simulation
    38. 50. Example 1: Fluid Dynamics Simulation Turbine Simulation <ul><li>Design of new turbine engines </li></ul><ul><li>High-resolution simulation of flow </li></ul><ul><ul><li>Computational Fluid Dynamics (CFD) </li></ul></ul><ul><ul><li>Use of high-performance computing resources (Cluster / Grid) </li></ul></ul><ul><ul><li>Huge amounts of data (>100 GByte) </li></ul></ul><ul><li>DataFinder used for </li></ul><ul><ul><li>Management of results </li></ul></ul><ul><ul><li>Automation of simulation runs </li></ul></ul><ul><ul><li>Starting pre-/post processing </li></ul></ul><ul><li>Used for CFD-code TRACE (DLR) </li></ul><ul><li>See </li></ul>
    39. 51. <ul><li>Simulation steps (example): </li></ul><ul><li>splitCGNS </li></ul><ul><ul><li>Preparing data for TRACE </li></ul></ul><ul><li>TRACE (CFD solver) </li></ul><ul><ul><li>Main computation </li></ul></ul><ul><li>fillCGNS </li></ul><ul><ul><li>Conflating results </li></ul></ul><ul><li>Post Processing </li></ul><ul><ul><li>Data reduction and visualization </li></ul></ul><ul><li>Automation with customized DataFinder </li></ul>Turbine Simulation Data Model
    40. 52. Turbine Simulation: Graphical User Interface
    41. 53. Turbine Simulation: Customized GUI Extensions <ul><li>Create new simulation </li></ul><ul><li>Start a simulation </li></ul><ul><li>Query status </li></ul><ul><li>Cancel simulation </li></ul><ul><li>Project overview </li></ul>1 2 3 4 5
    42. 54. Turbine Simulation Starting External Applications <ul><li>CGNS Infos / ADFview / CGNS Plot </li></ul><ul><li>TRACE GUI </li></ul><ul><li>Gnuplot </li></ul>1 2 3
    43. 55. Example 2: Automobile Supplier
    44. 56. Example 2: Automobile Supplier DataFinder for Simulation and Data Management <ul><li>Tasks </li></ul><ul><li>Automation and management of simulation of customers </li></ul><ul><li>Mapping of specific work sequence </li></ul><ul><li>High flexibility regarding customers requirements </li></ul>
    45. 57. Automobile Supplier Data Model
    46. 58. Automobile Supplier Configuration of Customers Parameters
    47. 59. Automobile Supplier Management of Simulations <ul><li>Status overview </li></ul><ul><li>Create, change, and delete data sets </li></ul><ul><li>Manage versions of data files </li></ul><ul><li>Parameter overview </li></ul>
    48. 60. Automobile Supplier Upload, Download, and Versioning of Files <ul><li>Upload/download of results </li></ul><ul><li>Versioning of results </li></ul><ul><li>Script store results in DataFinder data structures </li></ul>
    49. 61. Example 3: Air Traffic Management
    50. 62. Example 3: Air Traffic Monitoring Database for Air Traffic Monitoring <ul><li>Air traffic monitoring is important for research </li></ul><ul><ul><li>Predictions of air traffic </li></ul></ul><ul><ul><li>New traffic management approaches </li></ul></ul><ul><li>Usage of DataFinder </li></ul><ul><ul><li>Database for traffic data and reports </li></ul></ul><ul><ul><li>Project oriented view </li></ul></ul>
    51. 63. Database for Air Traffic Monitoring Data Model and Data Migration
    52. 64. Database for Air Traffic Monitoring Data Import Wizard <ul><li>Import of all data sources (PDF/Word/text files, Excel, Access, …) </li></ul><ul><li>Classification into multiple categories </li></ul><ul><li>Prevention of duplicated data and consistent naming </li></ul>
    53. 65. Database for Air Traffic Monitoring Search Results
    54. 66. Current Work and Future Plans <ul><li>Current work </li></ul><ul><li>Migration to Qt 4 </li></ul><ul><li>Improved usage (e.g., search dialogs) </li></ul><ul><li>Integration with Shibboleth </li></ul><ul><li>Future </li></ul><ul><li>Web interfaces </li></ul><ul><li>Jython </li></ul><ul><ul><li>Embedding in Java/Eclipse applications </li></ul></ul><ul><ul><li>Reuse of custom GUI dialogs </li></ul></ul><ul><li>Migration to Py3k </li></ul>
    55. 67. Am Ende… Hinweise <ul><li>pyCologne: Python User Group Köln </li></ul><ul><ul><li>Monatliche Treffen von Python-Interessierten aus dem Großraum Köln </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>Interesse an spannenden Tätigkeiten in Luft- und Raumfahrt? </li></ul><ul><ul><li>Feste Mitarbeit </li></ul></ul><ul><ul><li>Diplomarbeiten, Praktika </li></ul></ul><ul><ul><li> </li></ul></ul>
    56. 68. Availability <ul><li>DataFinder core available as Open Source </li></ul><ul><ul><li>BSD License </li></ul></ul><ul><ul><li> </li></ul></ul><ul><li>Extended versions / extensions are proprietary </li></ul>
    57. 69. Links <ul><li>DataFinder Web site </li></ul><ul><li> </li></ul><ul><li>DataFinder Open Source </li></ul><ul><li> </li></ul><ul><li>Python WebDAV library </li></ul><ul><li> </li></ul><ul><li>Catacomb </li></ul><ul><li> </li></ul><ul><li>AeroGrid Project </li></ul><ul><li> </li></ul>
    58. 70. Questions?