Eran Rom, IBM - Bringing Compute to Openstack Swift, OpenStack Israel 2015

1. © 2015 IBM Corporation
Bringing Compute to Openstack Swift
Eran Rom, System and Storage Researcher, IBM Research - Haifa

2. © 2015 IBM Corporation2
The Storlet engine extends Openstack Swift by enabling
the push down of filtering, transforming and analysis
tasks to Swift instead of bringing the data to the
computation.

3. © 2015 IBM Corporation3
Proxy Tier
Storage Tier
Ubuntu
14.04
Storlet Stuff
Ubuntu
14.04
Storlet Stuff
Ubuntu
14.04
Storlet Stuff
Ubuntu
14.04
Storlet Stuff

4. © 2015 IBM Corporation4
Containers Powered Swift Cluster
Proxy Tier
Storage Tier

5. © 2015 IBM Corporation5
Image
Registry
Proxy Tier
Storage Tier
Containers Powered Swift Cluster
Ubuntu
14.04
Storlet Stuff
Ubuntu
14.04
Storlet Stuff
Ubuntu
14.04
Storlet Stuff
Ubuntu
14.04
Storlet Stuff

6. © 2015 IBM Corporation6
Containers Powered Swift Cluster
Image
Registry
Proxy Tier
Storage Tier
Ubuntu
14.04
Storlet Stuff
Ubuntu
14.04
Storlet Stuff
Ubuntu
14.04
Storlet Stuff
Ubuntu
14.04
Storlet Stuff

7. © 2015 IBM Corporation7
Proxy Tier
Storage Tier
Developer
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Writing and Deploying a Storlet
@Override
public void invoke(ArrayList<StorletInputStream> inStreams,
ArrayList<StorletOutputStream> outStreams,
Map<String, String> parameters,
StorletLogger logger) throws StorletException {
…
}

8. © 2015 IBM Corporation8
Proxy Tier
Storage Tier
Developer
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Writing and Deploying a Storlet
PUT http://my.swift.co.il/v1/AUTH_111/Storlet/watermarkstorlet-10.jar

9. © 2015 IBM Corporation9
Proxy Tier
Storage Tier
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Invoking a Storlet on GET
GET http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webm
X-Run-Storlet : watermarkstorlet-10.jar

10. © 2015 IBM Corporation10
Proxy Tier
Storage Tier
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Invoking a Storlet on GET
Forward Request to
an Object Server
holding a Replica
GET http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webm
X-Run-Storlet : watermarkstorlet-10.jar

11. © 2015 IBM Corporation11
Invoking a Storlet on GET
Ubuntu
14.04
Storlet Stuff
ffmpeg
Intercept Request
Invoke Deamon, and Establish Connection
with the Account’s Docker Container
Pass Input and Output FDs
Send back the Object’s Metadata and the
Object’s Data Stream
Continue the GET Flow, where the
Storelt’s response is sent back to
the Proxy
Object Server

12. © 2015 IBM Corporation12
Proxy Tier
Storage Tier
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Ubuntu
14.04
Storlet Stuff
ffmpeg
Invoking a Storlet on PUT
PUT http://my.swift.co.il/v1/AUTH_111/myContainer/mov.webm
X-Run-Storlet : MetadataExtractorStorlet-10.jar

13. © 2015 IBM Corporation13
Invoking a Storlet on PUT
Ubuntu
14.04
Storlet Stuff
ffmpeg
Intercept Request
Invoke Deamon, and Establish Connection
with the Account’s Docker Container
Pass Input and Output FDs
Send back the Object’s Metadata and the
Object’s Data Stream
Continue the Proxy PUT Flow
with the returned Metadata
and data stream
Proxy Server

14. © 2015 IBM Corporation14
Demo
http://159.8.49.36:8080/raimediabridge/login.html

15. © 2015 IBM Corporation15
The Storlet Engine is in the Open
 https://github.com/Open-I-Beam/swift-storlets
 Comprehensive Documentation
• Installation guide
• How to write, deploy and invoke storlets guide
• How to tailor your own image for running storlets
 S2AIO
• Install swift and storlets in a single VM for experimentation
 Public users will be able to experiment with the Storlet engine thanks to FiWare – An EU research
project delivering a novel service infrastructure for the future internet (http://www.fiware.org/)

16. © 2015 IBM Corporation16
Media Workflow at NAB (with RAI)
Storlets run as material is uploaded and downloaded
Metadata
Enrichment
Storlet
Swift Object Store
Upload
Enriched
Metadata
Data
myvideo.mxf
Metadata
Data
myvideo.mxf
Data
Loudness
Calculation
Storlet
<Loudness,-23>
myvideo.mxf
Fix
Loudness
Storlet

17. © 2015 IBM Corporation17
Acknowledgements
The Storlets project is partially funded by the following EU research projects
Digital Preservation Inspired by Human Memory
IoT for Smart Cities