0
DataFinder:  Organizing the Data Chaos of Scientists PyCon UK 2008 (September 12 th , 2008, Birmingham) Andreas Schreiber ...
<ul><li>The DLR </li></ul><ul><li>German Aerospace Research Center  </li></ul><ul><li>Space Agency of the Federal Republic...
<ul><li>5,700  employees working  in 29 research institutes and  facilities </li></ul><ul><ul><li> at 13  sites .  </li>...
Short Overview <ul><li>DataFinder is a software for efficient management of scientific and technical data </li></ul><ul><l...
Introduction <ul><li>DataFinder founded by DLR </li></ul><ul><li>National Grid project AeroGrid </li></ul>
Introduction Background <ul><li>Large-scale simulations   </li></ul><ul><li>aerodynamics </li></ul><ul><li>material scienc...
Introduction Data Management Problem <ul><li>Typical organizational situations </li></ul><ul><li>No central data managemen...
DataFinder History Search for solution for scientific data management <ul><li>Definition of “standard problem” (helicopter...
DataFinder Development From Java Prototype to Python Product… <ul><li>Development of  prototype  in Java </li></ul><ul><li...
Python for Scientists and Engineers Reasons for Python in Research and Industry <ul><li>Observations :  </li></ul><ul><li>...
“ Python has the cleanest,  most-scientist- or engineer  friendly syntax and semantics. Paul F. Dubois Paul F. Dubois. Ten...
DataFinder Overview Basic Concept <ul><li>Client-Server solution </li></ul><ul><li>Based on  open and stable standards , s...
WebDAV Web-based Distributed Authoring & Versioning <ul><li>Extension of HTTP </li></ul><ul><li>Allows to manage files on ...
DataFinder Overview Client and Server <ul><li>Client </li></ul><ul><li>User client </li></ul><ul><li>Administrator client ...
DataFinder Client Graphical User Interfaces User Client Administrator Client Implementation in Python with Qt/PyQt
DataFinder Server Supported WebDAV servers <ul><li>Commercial Server Solution   </li></ul><ul><li>Tamino XML database (Sof...
WebDAV / Meta Data Server (1) Tamino WebDAV Server <ul><li>Commercial  Server Solution (Software AG) </li></ul><ul><ul><li...
WebDAV / Meta Data Server (2) Apache + mod_dav <ul><li>Open Source  solution (Apache Group) </li></ul><ul><ul><li>Apache H...
WebDAV / Meta Data Server (3) Catacomb <ul><li>Open Source  solution  </li></ul><ul><ul><li>Apache HTTP Web server + mod_d...
Mass Data Storage Data Stores Logical   View User   Client Storage  Locations
DataFinder  Technical Aspects <ul><li>Access privilege management </li></ul><ul><li>Authentication using WebDAV and LDAP <...
Python API  User Client Extension with GUI import   threading from  datafinder.application  import  search_support from  d...
Python API  Command Line Example (without GUI) # Get API from  datafinder.application  import  ExternalFacade externalFaca...
Additional “Batteries”… Used Libraries beyond the Python Standard Library (1) <ul><li>PyQt  (http://www.riverbankcomputing...
Additional “Batteries”… Used Libraries beyond the Python Standard Library (2) <ul><li>PyGlobus  (http://www-itg.lbl.gov/gt...
WebDAV Client Library Support for DAV Extensions <ul><li>Provides an object-oriented interface for accessing WebDAV server...
Simple Use Case: File Upload and Search
 
 
 
 
 
 
 
 
 
 
 
 
Working with DataFinder…
Configuration and Customization Preparing DataFinder for certain “use cases” <ul><li>Requirements Analysis </li></ul><ul><...
DataFinder Configuration Data Model and Data Stores <ul><li>Logical view to data </li></ul><ul><li>Definition of data stru...
Data Structure Mapping of Organizational Data Structures User Object (collection) Object (file) Relation Attributes (meta ...
Meta Data <ul><li>Describe and annotate data (“files”) and collections (“directories”) </li></ul><ul><li>Different levels ...
Impact for Users <ul><li>DataFinder restricts the rights of users! </li></ul><ul><li>Enforcement of “good behavior” </li><...
Customization   Python-Scripting for Extension and Automation <ul><li>Integration of DataFinder with environment </li></ul...
DataFinder Scripting  Downloading File and Starting Application # Download the selected file and try to execute it. from  ...
Examples…
Example 1: Turbine Simulation
Example 1:  Fluid Dynamics Simulation Turbine Simulation <ul><li>Design of new turbine engines </li></ul><ul><li>High-reso...
<ul><li>Simulation steps  (example): </li></ul><ul><li>splitCGNS </li></ul><ul><ul><li>Preparing data for TRACE </li></ul>...
Turbine Simulation: Graphical User Interface
Turbine Simulation: Customized GUI Extensions <ul><li>Create new simulation </li></ul><ul><li>Start a simulation  </li></u...
Turbine Simulation  Starting External Applications <ul><li>CGNS Infos / ADFview / CGNS Plot </li></ul><ul><li>TRACE GUI </...
Example 2: Automobile Supplier
Example 2:  Automobile Supplier DataFinder for Simulation and Data Management  <ul><li>Tasks </li></ul><ul><li>Automation ...
Automobile Supplier Data Model
Automobile Supplier Configuration of Customers Parameters
Automobile Supplier Management of Simulations <ul><li>Status overview </li></ul><ul><li>Create, change, and delete data se...
Automobile Supplier Upload, Download, and Versioning of Files <ul><li>Upload/download of results </li></ul><ul><li>Version...
Example 3: Air Traffic Management
Example 3:  Air Traffic Monitoring   Database for Air Traffic Monitoring <ul><li>Air traffic monitoring is important for r...
Database for Air Traffic Monitoring Data Model and Data Migration
Database for Air Traffic Monitoring Data Import Wizard <ul><li>Import of all data sources (PDF/Word/text files, Excel, Acc...
Database for Air Traffic Monitoring Search Results
Current Work and Future Plans  <ul><li>Current work </li></ul><ul><li>Migration to Qt 4 </li></ul><ul><li>Improved usage  ...
Am Ende… Hinweise <ul><li>pyCologne:   Python User Group Köln </li></ul><ul><ul><li>Monatliche Treffen von  Python-Interes...
Availability <ul><li>DataFinder core available as Open Source </li></ul><ul><ul><li>BSD License </li></ul></ul><ul><ul><li...
Links <ul><li>DataFinder Web site </li></ul><ul><li>http://www.dlr.de/datafinder </li></ul><ul><li>DataFinder Open Source ...
Questions?
Upcoming SlideShare
Loading in...5
×

Organizing the Data Chaos of Scientists

2,885

Published on

PyCon UK 2008 (12.-14. September 2008, Birmingham)

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,885
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
42
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide
  • Transcript of "Organizing the Data Chaos of Scientists"

    1. 1. DataFinder: Organizing the Data Chaos of Scientists PyCon UK 2008 (September 12 th , 2008, Birmingham) Andreas Schreiber < Andreas.Schreiber@dlr.de> German Aerospace Center (DLR), Cologne http://www.dlr.de/sc
    2. 2. <ul><li>The DLR </li></ul><ul><li>German Aerospace Research Center </li></ul><ul><li>Space Agency of the Federal Republic of Germany </li></ul>
    3. 3. <ul><li>5,700 employees working in 29 research institutes and facilities </li></ul><ul><ul><li> at 13 sites . </li></ul></ul><ul><li>Offices in Brussels, Paris and Washington. </li></ul>Sites and employees  Köln  Lampoldshausen  Stuttgart  Oberpfaffenhofen Braunschweig   Göttingen Berlin -   Bonn Trauen   Hamburg  Neustrelitz Weilheim  Bremen - 
    4. 4. Short Overview <ul><li>DataFinder is a software for efficient management of scientific and technical data </li></ul><ul><li>Focus on huge data sets </li></ul><ul><li>Development by DLR </li></ul><ul><li>Primary functionality </li></ul><ul><ul><li>Structuring of data through assignment of meta information and self-defined data models </li></ul></ul><ul><ul><li>Flexible usage of heterogeneous storage resources </li></ul></ul><ul><ul><li>Integration in the working environment </li></ul></ul>
    5. 5. Introduction <ul><li>DataFinder founded by DLR </li></ul><ul><li>National Grid project AeroGrid </li></ul>
    6. 6. Introduction Background <ul><li>Large-scale simulations </li></ul><ul><li>aerodynamics </li></ul><ul><li>material science </li></ul><ul><li>climate </li></ul><ul><li>… </li></ul><ul><li>Tons of measured data </li></ul><ul><li>wind-tunnel experiments </li></ul><ul><li>earth observations </li></ul><ul><li>traffic data </li></ul><ul><li>… </li></ul>
    7. 7. Introduction Data Management Problem <ul><li>Typical organizational situations </li></ul><ul><li>No central data management policy </li></ul><ul><li>Every employee organizes his/her data individually </li></ul><ul><li>Researchers spend about 30% of their time searching for data </li></ul><ul><li>Problem with data left behind by temporary staff </li></ul><ul><li>Increase of data size and regulations </li></ul><ul><li>Rapidly growing volume of simulation and experimental data </li></ul><ul><li>Legal requirements for long-term availability of data (up to 50 years!) </li></ul><ul><li>Situation similar at many organizations </li></ul><ul><li>All ~30 DLR institutes </li></ul><ul><li>Other research labs and agencies </li></ul><ul><li>Industry </li></ul>
    8. 8. DataFinder History Search for solution for scientific data management <ul><li>Definition of “standard problem” (helicopter simulation) </li></ul><ul><li>Test case for evaluation of software </li></ul><ul><li>Evaluation of commercial product data management (PDM) systems </li></ul><ul><li>PDM systems could manage data but with huge amount of costs </li></ul><ul><li>PDM systems have many unneeded functionalities </li></ul><ul><li>PDM systems have self-defined or unreadable scripting languages for extension and customization (Tcl etc.) </li></ul><ul><li>Development of DataFinder </li></ul><ul><li>Lightweight data management client and existing server solution </li></ul><ul><li>Just enough functionality for our problems (no paid but unused features!) </li></ul>
    9. 9. DataFinder Development From Java Prototype to Python Product… <ul><li>Development of prototype in Java </li></ul><ul><li>Data could be manages with prototype successfully </li></ul><ul><li>Drawbacks: Java problems on important platforms (e.g., SGI IRIX) </li></ul><ul><li>Embedded Jython interpreter great feature for users </li></ul><ul><li>User: “ The Java GUI is like shit, but the Python scripting is great. We want a pure Python solution! ” </li></ul><ul><li>Development of DataFinder product from scratch in Python </li></ul>
    10. 10. Python for Scientists and Engineers Reasons for Python in Research and Industry <ul><li>Observations : </li></ul><ul><li>Scientists and engineers don’t want to write software but just solve their problems </li></ul><ul><li>If they have to write code, it must be as easy as possible </li></ul><ul><li>Why Python is perfect? </li></ul><ul><li>Very easy to learn and easy to use ( = steep learning curve ) </li></ul><ul><li>Allows rapid development ( = short development time ) </li></ul><ul><li>Inherent great maintainability </li></ul>“ I want to design planes, not software!”
    11. 11. “ Python has the cleanest, most-scientist- or engineer friendly syntax and semantics. Paul F. Dubois Paul F. Dubois. Ten good practices in scientific programming. Comp. In Sci. Eng., Jan/Feb 1999, pp.7-11
    12. 12. DataFinder Overview Basic Concept <ul><li>Client-Server solution </li></ul><ul><li>Based on open and stable standards , such as XML and WebDAV </li></ul><ul><li>Extensive use of standard software components (open source / commercial), limited own development at client side </li></ul>
    13. 13. WebDAV Web-based Distributed Authoring & Versioning <ul><li>Extension of HTTP </li></ul><ul><li>Allows to manage files on remote servers collaboratively </li></ul><ul><li>WebDAV supports </li></ul><ul><ul><li>Resources (“files”) </li></ul></ul><ul><ul><li>Collections (“directories”) </li></ul></ul><ul><ul><li>Properties (“meta data”, in XML format) </li></ul></ul><ul><ul><li>Locking </li></ul></ul><ul><li>WebDAV extensions </li></ul><ul><ul><li>Versioning (DeltaV) </li></ul></ul><ul><ul><li>Access control (ACP) </li></ul></ul><ul><ul><li>Search (DASL) </li></ul></ul>
    14. 14. DataFinder Overview Client and Server <ul><li>Client </li></ul><ul><li>User client </li></ul><ul><li>Administrator client </li></ul><ul><li>Implementation: Python with Qt </li></ul><ul><li>Server </li></ul><ul><li>WebDAV server for meta data and data structure </li></ul><ul><li>Data Store concept </li></ul><ul><ul><li>Abstracts access to managed data </li></ul></ul><ul><ul><li>Flexible usage of heterogeneous storage resources </li></ul></ul><ul><li>Implementation: Various existing server solutions (third-party) </li></ul>
    15. 15. DataFinder Client Graphical User Interfaces User Client Administrator Client Implementation in Python with Qt/PyQt
    16. 16. DataFinder Server Supported WebDAV servers <ul><li>Commercial Server Solution </li></ul><ul><li>Tamino XML database (Software AG) </li></ul><ul><li>Open Source Server Solutions </li></ul><ul><li>Apache HTTP Web server and module mod_dav </li></ul><ul><ul><li>Default storage: file system (mod_dav_fs) </li></ul></ul><ul><ul><li>Module Catacomb (mod_dav_repos) + Relational database ( http://catacomb.tigris.org ) </li></ul></ul>
    17. 17. WebDAV / Meta Data Server (1) Tamino WebDAV Server <ul><li>Commercial Server Solution (Software AG) </li></ul><ul><ul><li>WebDAV Server </li></ul></ul><ul><ul><li>Tamino XML database backend </li></ul></ul><ul><li>Advantages </li></ul><ul><ul><li>Implements many WebDAV extensions (DASL, DeltaV, ACLs) </li></ul></ul><ul><ul><li>Fast XML processing </li></ul></ul><ul><li>Good, but not free  </li></ul><ul><li>Used in DLR for use with DataFinder </li></ul><ul><ul><li>One installation sufficient for many institutes </li></ul></ul>
    18. 18. WebDAV / Meta Data Server (2) Apache + mod_dav <ul><li>Open Source solution (Apache Group) </li></ul><ul><ul><li>Apache HTTP Web server </li></ul></ul><ul><ul><li>WebDAV extension module mod_dav </li></ul></ul><ul><ul><li>File system + (G)DBM database </li></ul></ul><ul><li>Advantage: Free and easy to install  </li></ul><ul><li>… but some WebDAV features are not supported </li></ul><ul><ul><li>No searching and versioning  </li></ul></ul>Apache Core Server mod_http mod_auth_ldap mod_dav mod_dav_fs File system
    19. 19. WebDAV / Meta Data Server (3) Catacomb <ul><li>Open Source solution </li></ul><ul><ul><li>Apache HTTP Web server + mod_dav </li></ul></ul><ul><ul><li>Module Catacomb (replacement for file system) </li></ul></ul><ul><ul><li>Relational database </li></ul></ul><ul><li>Search and versioning implemented: Uses database search features </li></ul><ul><li>Open Source development at DLR ( http://catacomb.tigris.org ) </li></ul>Apache Core Server mod_http mod_auth_ldap mod_dav mod_dav_fs File system DB (MySQL) Catacomb mod_dav_repos
    20. 20. Mass Data Storage Data Stores Logical View User Client Storage Locations
    21. 21. DataFinder Technical Aspects <ul><li>Access privilege management </li></ul><ul><li>Authentication using WebDAV and LDAP </li></ul><ul><li>Authorization for users and groups based on WebDAV (ACP) </li></ul><ul><li>Client available on many platforms </li></ul><ul><li>Linux, Windows, … </li></ul><ul><li>Restricted by availability of Python 2.5 and Qt 3 + PyQt </li></ul><ul><li>Extensible through Python scripts </li></ul><ul><li>Python application programming interface (API) </li></ul><ul><li>Accessing data and meta data </li></ul>
    22. 22. Python API User Client Extension with GUI import threading from datafinder.application import search_support from datafinder.gui.user import facade def searchAndDisplayResult(): &quot;&quot;&quot;Searches and displays the result in the search result logging window. &quot;&quot;&quot; query = &quot;displayname contains ‘test’ OR displayname == ‘ab’&quot; result = search_support.performSearch(query) resultLogger = facade.getSearchResultLogger() for path in result.keys(): resultLogger.info( &quot;Found item %s.&quot; % path) thread = threading.Thread(target=searchAndDisplayResult) thread.start()
    23. 23. Python API Command Line Example (without GUI) # Get API from datafinder.application import ExternalFacade externalFacade = ExternalFacade.getInstance() # Connect to a repository externalFacade.performBasicDatafinderSetup(username, password, startUrl) # Download the whole content rootItem = externalFacade.getRootWebdavServerItem() items = externalFacade.getCollectionContents(rootItem) for item in items: externalFacade.downloadFile(item, baseDirectory)
    24. 24. Additional “Batteries”… Used Libraries beyond the Python Standard Library (1) <ul><li>PyQt (http://www.riverbankcomputing.co.uk/software/pyqt) </li></ul><ul><ul><li>Interface to the Qt GUI framework (currently Qt 3) </li></ul></ul><ul><ul><li>Used for DataFinder UI layer </li></ul></ul><ul><li>Pyparsing (http://pyparsing.wikispaces.com/) </li></ul><ul><ul><li>Creating and executing simple grammars </li></ul></ul><ul><ul><li>Used for highlighting search expressions </li></ul></ul><ul><li>python-ldap (http://python-ldap.sourceforge.net/) </li></ul><ul><ul><li>Object-oriented API to access LDAP servers </li></ul></ul><ul><ul><li>Authentication against LDAP / ActiveDirectory server </li></ul></ul><ul><li>paramiko (http://www.lag.net/paramiko) </li></ul><ul><ul><li>SSH2 protocol implementation </li></ul></ul>
    25. 25. Additional “Batteries”… Used Libraries beyond the Python Standard Library (2) <ul><li>PyGlobus (http://www-itg.lbl.gov/gtg/projects/pyGlobus) </li></ul><ul><ul><li>Interface to The Globus Toolkit </li></ul></ul><ul><ul><li>Used for GridFTP Data Store </li></ul></ul><ul><li>Boto (http://code.google.com/p/boto) </li></ul><ul><ul><li>Interfaces to Amazon Web Services </li></ul></ul><ul><ul><li>Used for S3 (Simple Storage Service) Data Store </li></ul></ul><ul><li>davlib (http://www.webdav.org/mod_dav/ davlib.py ) </li></ul><ul><ul><li>WebDAV client library </li></ul></ul><ul><ul><li>Used for core WebDAV functions </li></ul></ul>
    26. 26. WebDAV Client Library Support for DAV Extensions <ul><li>Provides an object-oriented interface for accessing WebDAV server </li></ul><ul><ul><li>Extracted from DataFinder source </li></ul></ul><ul><li>WebDAV client-side library supports </li></ul><ul><ul><li>Core WebDAV specification </li></ul></ul><ul><ul><li>Access Control Protocol </li></ul></ul><ul><ul><li>Basic Versioning (experimental) </li></ul></ul><ul><ul><li>DAV Searching and Locating </li></ul></ul><ul><ul><li>Secure HTTP connections </li></ul></ul><ul><li>Implementation based on davlib and standard httplib </li></ul><ul><li>Apache License Version 2 </li></ul><ul><li>Project Site: http://sourceforge.net/projects/pythonwebdavlib </li></ul>
    27. 27. Simple Use Case: File Upload and Search
    28. 40. Working with DataFinder…
    29. 41. Configuration and Customization Preparing DataFinder for certain “use cases” <ul><li>Requirements Analysis </li></ul><ul><li>Analyze data, working environment, and users workflows </li></ul><ul><li>Configuration </li></ul><ul><li>Define and configure data model </li></ul><ul><li>Configure distributed storage resources (Data Stores) </li></ul><ul><li>Customization </li></ul><ul><li>Write functional extensions with Python scripts </li></ul>
    30. 42. DataFinder Configuration Data Model and Data Stores <ul><li>Logical view to data </li></ul><ul><li>Definition of data structuring and meta data (“data model”) </li></ul><ul><li>Separated storage of data structure / meta data and actual data files </li></ul><ul><li>Flexible use of (distributed) storage resources </li></ul><ul><ul><li>File system, WebDAV, FTP, GridFTP </li></ul></ul><ul><ul><li>Amazon S3 (Simple Storage Service) </li></ul></ul><ul><ul><li>Tivoli Storage Manager (TSM) </li></ul></ul><ul><ul><li>Storage Resource Broker (SRB) </li></ul></ul><ul><li>Complex search mechanism to find data </li></ul>
    31. 43. Data Structure Mapping of Organizational Data Structures User Object (collection) Object (file) Relation Attributes (meta data) Project A Project B Project C File 1 File 2 Simulation I Experiment Simulation II Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value Project Mega Code Ultra User Eddie Key Value
    32. 44. Meta Data <ul><li>Describe and annotate data (“files”) and collections (“directories”) </li></ul><ul><li>Different levels of meta data </li></ul><ul><ul><li>Required attributes defined by administrator </li></ul></ul><ul><ul><li>User is free to choose additional ones </li></ul></ul><ul><li>Different types of meta data </li></ul><ul><ul><li>String </li></ul></ul><ul><ul><li>Numbers (float, double, …) </li></ul></ul><ul><ul><li>Lists </li></ul></ul><ul><ul><li>Pictures </li></ul></ul><ul><ul><li>Links </li></ul></ul><ul><li>Stored in XML format </li></ul><ul><li>User can search in meta data </li></ul>
    33. 45. Impact for Users <ul><li>DataFinder restricts the rights of users! </li></ul><ul><li>Enforcement of “good behavior” </li></ul><ul><li>User must comply to organizational standards </li></ul><ul><li>Data is stored in defined (directory) hierarchy on data server </li></ul><ul><li>Required meta data must be set prior upload </li></ul><ul><li>User have certain access rights within hierarchy </li></ul>“ Damn! I’m a great scientist! I want freedom to have my own directory layout…”
    34. 46. Customization Python-Scripting for Extension and Automation <ul><li>Integration of DataFinder with environment </li></ul><ul><li>User, infrastructure, software, … </li></ul><ul><li>Extension of DataFinder by Python scripts </li></ul><ul><li>Actions for resources (i.e., files, directories) </li></ul><ul><li>User interface extensions </li></ul><ul><li>Typical automations and customizations </li></ul><ul><li>Data migration and data import </li></ul><ul><li>Start of external application (with downloaded data files) </li></ul><ul><li>Extraction of meta data from result files </li></ul><ul><li>Automation of recurring tasks (“workflows”) </li></ul>
    35. 47. DataFinder Scripting Downloading File and Starting Application # Download the selected file and try to execute it. from datafinder.application import ExternalFacade from guitools.easygui import * import os from tempfile import * from win32api import ShellExecute # Get instance of ExternalFacade to access DataFinder API facade = ExternalFacade.getInstance() # Get currently selected collection in DataFinder Server-View resource = facade.getSelectedResource() if resource != None: tmpFile = mktemp(ressource.name) facade.downloadFile(resource, tmpFile) if os.path.exists(tmpFile): ShellExecute(0, None, tmpFile, &quot;&quot; , &quot;&quot; , 1) else : msgbox( &quot;No file selected to execute.&quot; )
    36. 48. Examples…
    37. 49. Example 1: Turbine Simulation
    38. 50. Example 1: Fluid Dynamics Simulation Turbine Simulation <ul><li>Design of new turbine engines </li></ul><ul><li>High-resolution simulation of flow </li></ul><ul><ul><li>Computational Fluid Dynamics (CFD) </li></ul></ul><ul><ul><li>Use of high-performance computing resources (Cluster / Grid) </li></ul></ul><ul><ul><li>Huge amounts of data (>100 GByte) </li></ul></ul><ul><li>DataFinder used for </li></ul><ul><ul><li>Management of results </li></ul></ul><ul><ul><li>Automation of simulation runs </li></ul></ul><ul><ul><li>Starting pre-/post processing </li></ul></ul><ul><li>Used for CFD-code TRACE (DLR) </li></ul><ul><li>See http://www.aero-grid.de </li></ul>
    39. 51. <ul><li>Simulation steps (example): </li></ul><ul><li>splitCGNS </li></ul><ul><ul><li>Preparing data for TRACE </li></ul></ul><ul><li>TRACE (CFD solver) </li></ul><ul><ul><li>Main computation </li></ul></ul><ul><li>fillCGNS </li></ul><ul><ul><li>Conflating results </li></ul></ul><ul><li>Post Processing </li></ul><ul><ul><li>Data reduction and visualization </li></ul></ul><ul><li>Automation with customized DataFinder </li></ul>Turbine Simulation Data Model
    40. 52. Turbine Simulation: Graphical User Interface
    41. 53. Turbine Simulation: Customized GUI Extensions <ul><li>Create new simulation </li></ul><ul><li>Start a simulation </li></ul><ul><li>Query status </li></ul><ul><li>Cancel simulation </li></ul><ul><li>Project overview </li></ul>1 2 3 4 5
    42. 54. Turbine Simulation Starting External Applications <ul><li>CGNS Infos / ADFview / CGNS Plot </li></ul><ul><li>TRACE GUI </li></ul><ul><li>Gnuplot </li></ul>1 2 3
    43. 55. Example 2: Automobile Supplier
    44. 56. Example 2: Automobile Supplier DataFinder for Simulation and Data Management <ul><li>Tasks </li></ul><ul><li>Automation and management of simulation of customers </li></ul><ul><li>Mapping of specific work sequence </li></ul><ul><li>High flexibility regarding customers requirements </li></ul>
    45. 57. Automobile Supplier Data Model
    46. 58. Automobile Supplier Configuration of Customers Parameters
    47. 59. Automobile Supplier Management of Simulations <ul><li>Status overview </li></ul><ul><li>Create, change, and delete data sets </li></ul><ul><li>Manage versions of data files </li></ul><ul><li>Parameter overview </li></ul>
    48. 60. Automobile Supplier Upload, Download, and Versioning of Files <ul><li>Upload/download of results </li></ul><ul><li>Versioning of results </li></ul><ul><li>Script store results in DataFinder data structures </li></ul>
    49. 61. Example 3: Air Traffic Management
    50. 62. Example 3: Air Traffic Monitoring Database for Air Traffic Monitoring <ul><li>Air traffic monitoring is important for research </li></ul><ul><ul><li>Predictions of air traffic </li></ul></ul><ul><ul><li>New traffic management approaches </li></ul></ul><ul><li>Usage of DataFinder </li></ul><ul><ul><li>Database for traffic data and reports </li></ul></ul><ul><ul><li>Project oriented view </li></ul></ul>
    51. 63. Database for Air Traffic Monitoring Data Model and Data Migration
    52. 64. Database for Air Traffic Monitoring Data Import Wizard <ul><li>Import of all data sources (PDF/Word/text files, Excel, Access, …) </li></ul><ul><li>Classification into multiple categories </li></ul><ul><li>Prevention of duplicated data and consistent naming </li></ul>
    53. 65. Database for Air Traffic Monitoring Search Results
    54. 66. Current Work and Future Plans <ul><li>Current work </li></ul><ul><li>Migration to Qt 4 </li></ul><ul><li>Improved usage (e.g., search dialogs) </li></ul><ul><li>Integration with Shibboleth </li></ul><ul><li>Future </li></ul><ul><li>Web interfaces </li></ul><ul><li>Jython </li></ul><ul><ul><li>Embedding in Java/Eclipse applications </li></ul></ul><ul><ul><li>Reuse of custom GUI dialogs </li></ul></ul><ul><li>Migration to Py3k </li></ul>
    55. 67. Am Ende… Hinweise <ul><li>pyCologne: Python User Group Köln </li></ul><ul><ul><li>Monatliche Treffen von Python-Interessierten aus dem Großraum Köln </li></ul></ul><ul><ul><li>http://www.pycologne.de </li></ul></ul><ul><li>Interesse an spannenden Tätigkeiten in Luft- und Raumfahrt? </li></ul><ul><ul><li>Feste Mitarbeit </li></ul></ul><ul><ul><li>Diplomarbeiten, Praktika </li></ul></ul><ul><ul><li>http://tinyurl.com/dlrJobs </li></ul></ul>
    56. 68. Availability <ul><li>DataFinder core available as Open Source </li></ul><ul><ul><li>BSD License </li></ul></ul><ul><ul><li>http://sourceforge.net/projects/datafinder </li></ul></ul><ul><li>Extended versions / extensions are proprietary </li></ul>
    57. 69. Links <ul><li>DataFinder Web site </li></ul><ul><li>http://www.dlr.de/datafinder </li></ul><ul><li>DataFinder Open Source </li></ul><ul><li>http://sourceforge.net/projects/datafinder </li></ul><ul><li>Python WebDAV library </li></ul><ul><li>http://sourceforge.net/projects/pythonwebdavlib </li></ul><ul><li>Catacomb </li></ul><ul><li>http://catacomb.tigris.org </li></ul><ul><li>AeroGrid Project </li></ul><ul><li>http://www.aero-grid.de </li></ul>
    58. 70. Questions?
    1. A particular slide catching your eye?

      Clipping is a handy way to collect important slides you want to go back to later.

    ×