This document discusses developing middleware for the Swift object storage system. It introduces middlewares and how they can extend functionality without modifying code. It covers the WSGI interface, common middleware patterns, and tools like paste.deploy for packaging and deploying middlewares. Finally, it provides an example of writing a middleware for Swift to generate and serve preview images on GET requests.

1. Developing your own Swift middleware
OpenStack Summit Atlanta, May 2014

2. About me
● Christian Schwede
● Developer @ eNovance
● Mostly working on Swift, testing and automation
● Started using Swift in 2012

3. Agenda
● Using middlewares to extend Swift functionality
● Introduction to wsgi, middlewares and paste.deploy
● Developing our own middleware for Swift Proxy
● Testing & Packaging
● References

4. Writing Swift middlewares?

5. Swift middlewares?
● Most of Swift features are implemented as a middleware
○ logging, tempurl, dlo, ratelimit, tempauth, quotas, ...
● Flexibility to extend existing functionality
● No need to fork or modify existing code
● Several additional middlewares outside of Swift

6. wsgi and middlewares?

7. wsgi
● simple interface between web servers & web applications
● defined in PEP0333 and PEP3333
● application
○ callable object (with a __call__ method)
● server
○ invokes the callable once for each client request
Client Server App

8. application objects
● must accept two positional arguments
○ environ: Python dictionary
○ start_response: callable
■ status = „200 OK“
■ headers = [(„header_name“, „header_value“), ]

9. environ
● REQUEST_METHOD
● PATH_INFO
● QUERY_STRING
● HTTP_HEADERNAME
● wsgi.input

10. from wsgiref.simple_server import make_server
def myapp(environ, start_response):
body = []
body.append("PATH_INFO: '%s'n" % environ.get('PATH_INFO'))
headers = [('Content-Type', 'text/plain')]
start_response('200 OK', headers)
return body
srv = make_server('localhost', 8000, myapp)
srv.serve_forever()
Sample server

11. middleware
● can act as a server for an application and vice versa
● run multiple applications side-by-side
● authentication
● rerouting a request
● content processing
Client Server AppMiddleware

12. class SummitMiddleware(object):
def __init__(self, app, *args, **kwargs):
self.app = app
def __call__(self, env, start_response):
response = self.app(env, start_response)
if env.get('PATH_INFO') == '/echo':
length = int(env.get('CONTENT_LENGTH') or 0)
return env.get('wsgi.input').read(length)
return response
srv = make_server(‘localhost’, 8000, SummitMiddleware(myapp))
Sample middleware

13. Testing, packaging & deploying

14. class FakeApp(object):
def __call__(self, env, start_response):
start_response('200 OK', [])
return ""
class TestSummitMiddleware(unittest.TestCase):
def test_simple_request(self):
environ = {'REQUEST_METHOD': 'PUT'}
req = Request.blank('/echo', environ, body="Hello World")
mw = SummitMiddleware(FakeApp())
resp = req.get_response(mw)
self.assertEqual("Hello World", resp.body)
Testing a middleware

15. paste.deploy
● load WSGI applications and servers from URI
● uses INI-style configuration files
● separates config from code
● Paste Script can serve applications from config files
● widely used in Openstack

16. [app:sample]
use = egg:sample#app
[filter:middleware]
use = egg:sample#middleware
suffix = /echoresponse
[pipeline:main]
pipeline = middleware sample
[server:main]
use = egg:Paste#http
port = 8000
config.ini for paste.deploy

17. setup(name='sample',
packages=['sample', ],
zip_safe=False,
entry_points={
'paste.app_factory': ['app=sample.app:app_factory'],
'paste.filter_factory': ['middleware = sample.middleware:
filter_factory']
})
setup.py

18. Writing a Swift middleware

19. Preview middleware
● PUT
○ create a small preview image and store it separate
● GET
○ return preview if QUERY_STRING contains “preview”
● DELETE
○ also delete preview if exists

20. Useful helpers
● swift.common.utils.split_path
path = /v1/AUTH_account/test/img.jpg
ver, acc, cont, obj = split_path(path)
● swift.common.wsgi.make_subrequest
○ add middleware after authentication middleware
● swift.common.swob.wsgify
○ decorator

21. @wsgify
def __call__(self, request):
# request.params
# request.path_info
# request.method
# request.environ
# request.body
return self.app
wsgify?

22. @wsgify
def __call__(self, req):
try:
(ver, acc, con, obj) = split_path(req.path_info, 4, 4, True)
except ValueError:
return self.app
preview_path= '/%s/%s/%s_%s/%s' % (ver, acc, con, self.suffix, obj)
if req.method == 'GET' and request.params.has_key('preview'):
req.path_info = preview_path
Return preview

23. Extract preview
if req.method == 'PUT':
preview = create_preview(request.body)
if preview:
sub = wsgi.make_subrequest(
request.environ, path=preview_path, body=preview)
sub.get_response(self.app)

24. Delete preview
if req.method == 'DELETE':
sub = wsgi.make_subrequest(req.environ, path=preview_path)
sub.get_response(self.app)
return self.app

25. References

26. 3rd party middlewares
● swauth - github.com/gholt/swauth
● swift3 - github.com/stackforge/swift3
● CDMI - github.com/osaddon/cdmi
● Swift informant - github.com/pandemicsyn/swift-informant
● Swift Origin Server - github.com/dpgoetz/sos
● Ceilometer - ceilometer/objectstore/swift_middleware.py

27. ● github.com/enovance/swift-middleware-sample
● docs.openstack.org/developer/swift/
○ middleware.html
○ development_middleware.html
○ development_auth.html
○ associated_projects.html
● legacy.python.org/dev/peps/pep-3333
Something to read

28. THANK YOU!
christian@enovance.com | @cschwede_de | OpenStack Juno Summit | May 2014, Atlanta