1. iRODS: the use of rules and micro
services for automatic data
conversion and signal pattern
Martyn Fletcher, Tom Jackson,
Bojian Liang, Michael Weeks,
Mark Jessop and Jim Austin.
2. Slide 28-Dec-2009
1. The iREAD project.
2. iRODS overview.
3. What are iRODS micro-services?
4. What are iRODS rules?
5. iREAD Demonstrations.
6. Data conversion using iRODS.
7. Pattern searching using iRODS.
8. iRODS browser web client.
9. Conclusions / experiences.
10.Potential future work.
3. Slide 38-Dec-2009
The iREAD Project
• The iREAD (iRODS Evaluation and Demonstrator) Project:
– Funded by JISC:
– All documentation, experiences, results and demonstrations are
available on the public website see http://www.wrg.york.ac.uk/iread
– Demonstration of searching available for all at http://www.wrg.york.ac.uk/iread
and on the White Rose Grid stand at 3:00 pm.
– Only part of the iREAD work is reported in this presentation.
– iRODS™- the Integrated Rule-Oriented Data System iRODS (DICE
group at UNC Chapel Hill, University of California, San Diego.
Enhancement of Storage Request Broker (SRB). http://www.irods.org
4. Slide 48-Dec-2009
Virtual file system (layer over existing data and meta data)
Enhancement over SRB: Rules and
micro–service (within each server)
can be used to automatically process
data, make copies on other servers,
enforce access controls, etc.
5. Slide 58-Dec-2009
What are iRODS micro services?
• Written C.
• Perform actions - executed as part of a rule.
• Inbuilt micro services for accessing various parts of
– create , delete, read files and collections (directories),
deny access to files, simple workflow constructs, etc.
• Users can create their own micro services:
– Operate on data, run external web services, etc
– We have created ones for searching, data conversion,
etc. e.g. msiSearch.
• Adding or changing a micro service necessitates:
– Re-compilation of the iRODS installation.
– A stop / start of the iRODS system.
– This situation may change in future.
6. Slide 68-Dec-2009
What are iRODS rules?
• A rule is a “grouping” of:
– Other rules.
• Can be executed:
– Automatically on events, time, upload, etc. – part
of the rule base (core.irb file).
– By users on command from API or rule files.
• Can be added to a live iRODS system:
– No need for a recompile and stop / start.
7. Slide 78-Dec-2009
What are iRODS rules (2)?
• iRODS rules have the:
• Action_Name is the name of the rule.
• Condition is the condition for execution of the rule.
• Workflow_Chain is a list of micro-services or other rules to be
• Recovery_Chain is a list of recovery actions if any of the
8. Slide 88-Dec-2009
• The iREAD iRODS demonstration setup is as
– iRODS version 2.0.1.
– The iRODS browser installation (part of the extrods
188.8.131.52 package) for connection to iRODS.
– Apache web server 2.2.10 for the browser.
– PHP 5.26 for the browser.
• Rules and micro-services have been added to
iRODS as necessary.
• The browser installation and PHP service have
also been modified for demonstration purposes.
9. Slide 98-Dec-2009
Data conversion using iRODS
• Image conversion micro-services have been compiled
and installed (supplied with iRODS).
• Additional skeleton micro-services have been included
for future conversion of Neurophysiology data formats.
• Installed rules (acPostProcForPut) for post processing
of the put command.
• The acPostProcPut rules – only one is executed (each
is a single line in the core.irb file):
1. acPostProcForPut|$objPath like /tempZone/home/demo/files-and-conversions/*.gif|
2. acPostProcForPut|$objPath like /tempZone/home/demo/files-and-conversions/*.bmp|
3. acPostProcForPut|$objPath like /tempZone/home/demo/files-and-conversions/*.mcd|
10. Slide 108-Dec-2009
Pattern searching using iRODS (1)
• Used an existing remote search service
developed during the DAME and BROADEN e-
• Created micro services which access the web
• Created rule to run the search micro services.
• Modified browser to add buttons, display
• Used PHP API to run the rule.
11. Slide 118-Dec-2009
Pattern searching using iRODS (2)
13. Slide 138-Dec-2009
Pattern searching using iRODS (4)
• Micro services used (in following :
– msiDataObjUnlink - delete the existing results.txt file.
– msiDataObjOpen - open the query.txt file.
– msiDataObjRead - read the query.txt file.
– msiDataObjClose - close the query.txt file.
– msiNewRemoteMthSearch - invoke the search web service and
pass the query data for matching.
– msiNewRemoteGetResult - poll the search service and when it
has finished search retrieve the results.
– msiDataObjCreate - create a new results.txt file.
– msiDataObjWrite - write the retrived results to the new results.txt
– msiDataObjClose - close the results.txt file.
14. Slide 148-Dec-2009
iRODS web browser client
• All demonstrations available through the modified iRODS web browser client
accessible at http://www.wrg.york.ac.uk/iread.
15. Slide 158-Dec-2009
Conclusions / experiences (1)
• Rules and micro services can be run
remotely on other servers (same / different
zone) using the remoteExec micro service.
• Parallel execution / broadcast execution
modes would greatly assist the use of
iRODS for distributed operation.
16. Slide 168-Dec-2009
Conclusions / experiences (2)
• Micro services (and rules) need to be installed
on every server where they need to executed.
• Rules can become very long and unwieldy - plan
carefully - tool support would be good.
• It would beneficial to be able to add micro
services to a live system without:
– Recompiling the iRODS server.
– Starting and stopping the server.
– Information not extensive e.g. a micro service can fail
producing a non informative messages on the
command line and in the rodsLog.
17. Slide 178-Dec-2009
Potential Future Work (1)
• Provide automatic:
– Generation of the search databases (as new
– Search for known conditions in bulk data -
• iRODS would provide benefits to the
CARMEN neuroscience project to provide:
– Automatic data conversion.
– Service execution.
18. Slide 188-Dec-2009
The iREAD Project
• Work funded by JISC:
• All documentation, experiences, results and demonstrations are
available on the public website see: http://www.wrg.york.ac.uk/iread
• Demonstration of searching on the White Rose Grid stand (3:00pm
today) and available for all at http://www.wrg.york.ac.uk/iread