Building Automated REST APIs with Python
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Building Automated REST APIs with Python

on

  • 1,682 views

Writing REST APIs with ORMs and web frameworks is a chore. I'm lazy, and I don't want to write boring code. In this talk, I'll go over what REST APIs are, why they're useful, and why we should never ...

Writing REST APIs with ORMs and web frameworks is a chore. I'm lazy, and I don't want to write boring code. In this talk, I'll go over what REST APIs are, why they're useful, and why we should never have to write one from scratch again.

By the end of this talk, we'll have achieved developer Nirvana: a RESTful API service and Admin interface for existing databases *without writing any code*.

Statistics

Views

Total Views
1,682
Views on SlideShare
1,543
Embed Views
139

Actions

Likes
7
Downloads
35
Comments
0

3 Embeds 139

https://twitter.com 112
http://www.slideee.com 22
http://www.bigdatapro.io 5

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Building Automated REST APIs with Python Presentation Transcript

  • 1. Jeff Knupp @jeffknupp jeff@jeffknupp.com Wharton Web Conference 2014
  • 2. Author of “Writing Idiomatic Python” Full-time Python developer @ AppNexus Blogger at jeffknupp.com Creator of the “sandman” Python library
  • 3. We're going to use Python to generate a REST API. And we're going to do it without writing a single line of code.
  • 4. We'll go over what a REST API is, how it works, and why it's useful We'll review the HTTP protocol and how the web works We'll see a lot of Python code
  • 5. Seven letters. Two acronyms. Buy what does it mean?
  • 6. Programmatic way of interacting with a third-party system.
  • 7. Way to interact with APIs over HTTP (the communication protocol the Internet is built on). "REST" was coined by Roy Fielding in his 2000 doctoral dissertation. Includes set of design principles and best practices for designing systems meant to be "RESTful".
  • 8. In RESTful systems, application state is manipulated by the client interacting with hyperlinks. A root link (e.g. ) describes what actions can be taken by listing resources and state as hyperlinks. http://example.com/api/
  • 9. HTTP is just a messaging protocol. Happens to be the one the Internet is based on. RESTful systems use this protocol to their advantage e.g. caching resources
  • 10. GET POST PUT PATCH DELETE
  • 11. To understand how REST APIs work, we have to understand how the web works. Everything you see on the web is transferred to your computer using HTTP.
  • 12. What happens when we type http://www.jeffknupp.com into our browser? Let's trace the lifecycle of a browser's request.
  • 13. A protocol called the Domain Name Service (DNS) is used to find the "real" (IP) address of jeffknupp.com.
  • 14. GET The browser sends a GET request to 192.168.1.1 for the page at address / (the home or "root" page).
  • 15. The (a program used to service HTTP requests to a web site) receives the request, finds the associated HTML file, and sends it as an HTTP Response.
  • 16. If there are any images, videos, or scripts that the HTML makes reference to, separate HTTP GET requests are made for those as well.
  • 17. Programs, like curl, can also issue HTTP requests
  • 18. CURL curl talks to the webserver, using a public API (via HTTP)
  • 19. A REST API exposes your internal system to the outside world
  • 20. It's also a fantastic way to make a system available to other, internal systems within an organization.
  • 21. Examples of popular REST APIs: Twitter GitHub Google (for almost all services)
  • 22. If you're a SaaS provider, you are expected to have a REST API for people to write programs to interact with your service.
  • 23. Four core concepts are fundamental to all REST services (courtesy Wikipedia)
  • 24. When using HTTP, this is done using a URI. Importantly, a resource and are completely orthogonal. The server doesn't return database results but rather the JSON or XML or HTML representation of the resource.
  • 25. When the server transmits the representation of the resource to the client, it includes enough information for the client to know how to modify or delete the resource.
  • 26. Each representation returned by the server includes information on how to process the message (e.g. using MIME types
  • 27. Clients are . They know nothing about how the service is laid out to begin with. They discover what actions they can take from the root link. Following a link gives further links, defining exactly what may be done from that resource. Clients aren't assumed to know except what the message contains and what the server already told them.
  • 28. A REST API allows to send to manipulate . ...So we need to write a server capable of accepting HTTP requests, acting on them, and returning HTTP responses.
  • 29. Yep. A RESTful API Service is just a web application and, as such, is built using the same set of tools. We'll build ours using Python, Flask, and SQLAlchemy
  • 30. Earlier we said a REST API allows clients to manipulate via HTTP.
  • 31. Pretty much. If you're system is built using ORM models, your resources are almost certainly going to be your models.
  • 32. Web frameworks reduce the boilerplate required to create a web application by providing: of HTTP requests to handler functions or classes Example: /foo => def process_foo() of HTTP responses to inject dynamic data in pre-defined structure Example: <h1>Hello {{ user_name }}</h1>
  • 33. The more time you spend building REST APIs with web frameworks, the more you'll notice the subtle (and at times, glaring) impedance mismatch. URLs as to processing functions; REST APIs treat URLs as the address of a resource or collection HTML templating, while REST APIs rarely. JSON-related functionality feels bolted-on.
  • 34. Imagine we're Twitter weeks after launch. Ashton Kutcher seems to be able to use our service, but what about ? That's right, we'll need to create an API. Being an internet company, we'll build a REST API service. For now, we'll focus on two resources: user tweet
  • 35. All resources must be identified by a unique address at which they can be reached, their URI. This requires each resource contain a unique ID, usually a monotonically increasing integer or UUID (like a primary key in a database table). Our pattern for building URLs will be /resource_name[/resource_id[/resource_attribute]]
  • 36. Here we define our resources is a file called models.py: class User(db.Model, SerializableModel): __tablename__ = 'user' id = db.Column(db.Integer, primary_key=True) username = db.Column(db.String) class Tweet(db.Model, SerializableModel): __tablename__ = 'tweet' id = db.Column(db.Integer, primary_key=True) content = db.Column(db.String) posted_at = db.Column(db.DateTime) user_id = db.Column(db.Integer, db.ForeignKey('user.id')) user = db.relationship(User)
  • 37. class SerializableModel(object): """A SQLAlchemy model mixin class that can serialize itself as JSON.""" def to_dict(self): """Return dict representation of class by iterating over database columns.""" value = {} for column in self.__table__.columns: attribute = getattr(self, column.name) if isinstance(attribute, datetime.datetime): attribute = str(attribute) value[column.name] = attribute return value
  • 38. Here's the code that handles retrieving a single tweet and returning it as JSON: from models import Tweet, User @app.route('/tweets/<int:tweet_id>', methods=['GET']) def get_tweet(tweet_id): tweet = Tweet.query.get(tweet_id) if tweet is None: response = jsonify({'result': 'error'}) response.status_code = 404 return response else: return jsonify({'tweet': tweet.to_dict()})
  • 39. Let's curl our new API (preloaded with a single tweet and user): $ curl localhost:5000/tweets/1 { "tweet": { "content": "This is awesome", "id": 1, "posted_at": "2014-07-05 12:00:00", "user_id": 1 } }
  • 40. @app.route('/tweets/', methods=['POST']) def create_tweet(): """Create a new tweet object based on the JSON data sent in the request.""" if not all(('content', 'posted_at', 'user_id' in request.json)): response = jsonify({'result': 'ERROR'}) response.status_code = 400 # HTTP 400: BAD REQUEST return response else: tweet = Tweet( content=request.json['content'], posted_at=datetime.datetime.strptime( request.json['posted_at'], '%Y-%m-%d %H:%M:%S'), user_id=request.json['user_id']) db.session.add(tweet) db.session.commit() return jsonify(tweet.to_dict())
  • 41. In REST APIs, a group of resources is called a . REST APIs are heavily built on the notion of resources and collections. In our case, the of tweets is a list of all tweets in the system. The tweet collection is accessed by the following URL (according to our rules, described earlier): /tweets.
  • 42. @app.route('/tweets', methods=['GET']) def get_tweet_collection(): """Return all tweets as JSON.""" all_tweets = [] for tweet in Tweet.query.all(): all_tweets.append({ 'content': tweet.content, 'posted_at': tweet.posted_at, 'posted_by': tweet.user.username})
  • 43. All the code thus far has been pretty much boilerplate. Every REST API you write in Flask (modulo business logic) will look identical. How can we use that to our advantage?
  • 44. We have self-driving cars and delivery drones, why can't we build REST APIs automatically?
  • 45. This allows one to work at a higher level of abstraction. Solve the problem once in a general way and let code generation solve each individual instance of the problem. Part of
  • 46. SANDBOY Third party Flask extension written by the dashing Jeff Knupp. Define your models. Hit a button. BAM! RESTful API service that . (The name will make more sense in a few minutes)
  • 47. Generalizes REST resource handling into notion of a (e.g. the "Tweet Service" handles all tweet-related actions). class Service(MethodView): """Base class for all resources.""" __model__ = None __db__ = None def get(self, resource_id=None): """Return response to HTTP GET request.""" if resource_id is None: return self._all_resources() else: resource = self._resource(resource_id) if not resource: raise NotFoundException return jsonify(resource.to_dict())
  • 48. def _all_resources(self): """Return all resources of this type as a JSON list.""" if not 'page' in request.args: resources = self.__db__.session.query(self.__model__).all() else: resources = self.__model__.query.paginate( int(request.args['page'])).items return jsonify( {'resources': [resource.to_dict() for resource in resources]})
  • 49. Here's how POST works. Notice the verify_fields decorator and use of **request.json magic... @verify_fields def post(self): """Return response to HTTP POST request.""" resource = self.__model__.query.filter_by( **request.json).first() if resource: return self._no_content_response() instance = self.__model__(**request.json) self.__db__.session.add(instance) self.__db__.session.commit() return self._created_response(instance.to_dict())
  • 50. We have our models defined. How do we take advantage of the generic Service class and create services from our models? def register(self, cls_list): """Register a class to be given a REST API.""" for cls in cls_list: serializable_model = type(cls.__name__ + 'Serializable', (cls, SerializableModel), {}) new_endpoint = type(cls.__name__ + 'Endpoint', (Service,), {'__model__': serializable_model, '__db__': self.db}) view_func = new_endpoint.as_view( new_endpoint.__model__.__tablename__) self.blueprint.add_url_rule( '/' + new_endpoint.__model__.__tablename__, view_func=view_func) self.blueprint.add_url_rule( '/{resource}/<resource_id>'.format( resource=new_endpoint.__model__.__tablename__), view_func=view_func, methods=[
  • 51. 'GET','PUT','DELETE','PATCH','OPTIONS']) TYPE In Python, type with one argument returns a variable's type. With three arguments, .
  • 52. TYPE serializable_model = type( cls.__name__ + 'Serializable', (cls, SerializableModel), {}) new_endpoint = type(cls.__name__ + 'Endpoint', (Service,), {'__model__': serializable_model, '__db__': self.db})
  • 53. Let's play pretend again. Now we're a IaaS company that lets users build private clouds. We'll focus on two resources: cloud and machine
  • 54. class Cloud(db.Model): __tablename__ = 'cloud' id = db.Column(db.Integer, primary_key=True) name = db.Column(db.String, nullable=False) description = db.Column(db.String, nullable=False) class Machine(db.Model): __tablename__ = 'machine' id = db.Column(db.Integer, primary_key=True) hostname = db.Column(db.String) operating_system = db.Column(db.String) description = db.Column(db.String) cloud_id = db.Column(db.Integer, db.ForeignKey('cloud.id')) cloud = db.relationship('Cloud') is_running = db.Column(db.Boolean, default=False)
  • 55. from flask import Flask from flask.ext.sandboy import Sandboy from models import Machine, Cloud, db app = Flask(__name__) app.config['SQLALCHEMY_DATABASE_URI'] = 'sqlite:///db.sqlite3' db.init_app(app) with app.app_context(): db.create_all() sandboy = Sandboy(app, db, [Machine, Cloud]) app.run(debug=True)
  • 56. In cases where we're building a REST API from scratch, this is pretty easy. But what if: We have an existing database We want to create a RESTful API for it It has 200 tables
  • 57. Only downside of Flask-Sandboy is you have to define your model classes explicitly. If you have a lot of models, this would be tedious. ...I don't do tedious
  • 58. We have private companies building rocket ships and electric cars. Why can't we have a tool that you point at an existing database and hit a button, then, BLAM! RESTful API service.
  • 59. SANDMAN , a library by teen heartthrob Jeff Knupp, creates a RESTful API service for with .
  • 60. Here's how you run sandman against a mysql database: $ sandmanctl mysql+mysqlconnector://localhost/Chinook * Running on http://0.0.0.0:8080/ * Restarting with reloader
  • 61. $ curl -v localhost:8080/artists?Name=AC/DC HTTP/1.0 200 OK Content-Type: application/json Date: Sun, 06 Jul 2014 15:55:21 GMT ETag: "cea5dfbb05362bd56c14d0701cedb5a7" Link: </artists/1>; rel="self" { "ArtistId": 1, "Name": "AC/DC", "links": [ { "rel": "self", "uri": "/artists/1" } ], "self": "/artists/1" }
  • 62. ETag set correctly, allowing for caching responses Link Header set to let clients discover links to other resources Search enabled by sending in an attribute name and value Wildcard searching supported
  • 63. We can curl / and get a list of all available services and their URLs. We can hit /<resource>/meta to get meta-info about the service. Example (the "artist" service): $ curl -v localhost:8080/artists/meta HTTP/1.0 200 OK Content-Length: 80 Content-Type: application/json Date: Sun, 06 Jul 2014 16:04:25 GMT ETag: "872ea9f2c6635aa3775dc45aa6bc4975" Server: Werkzeug/0.9.6 Python/2.7.6 { "Artist": { "ArtistId": "integer(11)", "Name": "varchar(120)" } }
  • 64. And now for a (probably broken) live-demo!
  • 65. "Real" REST APIs enable clients to use the API using only the information returned from HTTP requests. sandman tries to be as "RESTful" as possible without requiring any code from the user.
  • 66. Would be nice to be able to visualize your data in addition to interacting with it via REST API.
  • 67. 1. Code generation 2. Database introspection 3. Lots of magic
  • 68. sandman came first. Has been number one Python project on GitHub multiple times and is downloaded 25,000 times a month. Flask-Sandboy is sandman's little brother...
  • 69. The fact that the end result is a REST API is not especially interesting More important are the concepts underpinning sandman and Flask-Sandboy
  • 70. Work at higher level of abstraction Solve a problem once in a generic manner Reduces errors, improves performance In general: Speaking of automation, here's how my book is "built"...
  • 71. sandman = Flask + SQLAlchemy + Lots of Glue Requires you know the capabilities of your tools Part of the UNIX Philosophy
  • 72. The best programming advice I ever got was to "be lazy" Sandman exists because I was too lazy to write boilerplate ORM code for an existing database Flask-Sandboy exists because I was too lazy to write the same API services over and over Being lazy forces you to learn your tools and make heavy use of them
  • 73. Contact me at: jeff@jeffknupp.com @jeffknupp on Twitter on the tubeshttp://www.jeffknupp.com