Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
CMPE 275
Project 2 Report
RESTful social collaboration through Pinterest re-enactment
Team:
Bhushan Deo (009269403)
Deven ...
Introduction
The main goal of Project 2 is to explore the design and challenges in developing RESTful web services for
a s...
Functionalities
This project also required cross team interaction to come up with a stable specification for the web
servi...
The responses were returned in JSON format and contained relevant data and endpoint information for
subsequent operations ...
return a python dictionary object. Bottle returns the response in JSON and the appropriate field is set in
the HTTP header...
{
"_id": {
"firstName": "",
"lastName": "",
"emailId": "",
"password": "",
"isLoggedIn": false,
"boards": [
{
"boardId": "...
Drawback:
The limitation of this schema is that for any request, we get the entire complex document associated
with a user...
Upcoming SlideShare
Loading in …5
×

Pinterest like site using REST and Bottle

1,164 views

Published on

Published in: Engineering, Technology
  • Be the first to comment

  • Be the first to like this

Pinterest like site using REST and Bottle

  1. 1. CMPE 275 Project 2 Report RESTful social collaboration through Pinterest re-enactment Team: Bhushan Deo (009269403) Deven Pawar (009275994) Dhruv Gogna (005067284) Gaurav Bhardwaj (009297431) Vinay Bhore (009290931)
  2. 2. Introduction The main goal of Project 2 is to explore the design and challenges in developing RESTful web services for a social media service such as Pinterest. Sharing content is an important pattern found in all social media services. Providing robust and well documented web services is very important in such products as there are several external clients which use them to access and manipulate stored user data. In this report, we provide a detailed explanation of our team’s design and implementation. High Level Architecture The team’s configuration consists of the following components: 1. Web Services: We expose our functionality to the client using RESTful web services written in Python using the Bottle micro-framework. 2. Database: CouchDB NoSQL database is used to persist user data. It provides data storage as JSON documents and thus naturally maps to the requirements. Also, when client hits our web service and some data is changed, we can verify changes in the database through the web interface (Futon), that comes with CouchDB. 3. couchdb-python library: This library provides interface to interact with CouchDB from python code. In our experience, this was a very well designed library with advanced features like high-level couchDB document to python object mapping, intuitive interfaces and good documentation. 4. Client: For testing, we are using curl to hit our web services and see the response. Curl provides many advanced options to make testing via HTTP requests easy. The following sections explain the functionalities implemented and the web service and database component in detail. These include comments about what we learnt while working on each component.
  3. 3. Functionalities This project also required cross team interaction to come up with a stable specification for the web services. As per the specification agreed to by the class, following functionalities were implemented by our team. Here, we observed how the traditional Create, Read, Update and Delete operations in databases, map exactly to HTTP methods POST, GET, PUT and DELETE respectively. Functionality Endpoint (fields in angular brackets are generated on the server) CRUD? HTTP Met- hod Request Parameters (all are string type unless specified) Sign up http://localhost:8080/users/signUp Create POST firstName lastName emailId password Login http://localhost:8080/users/login/ Create POST emailId password Create a board http://localhost:8080/users/<userId>/boards/ Create POST boardName boardDesc category isPrivate(Boolean) Get all boards http://localhost:8080/users/<userId>/boards/ Read GET None Get details of a single board http://localhost:8080/users/<userId>/boards/<boardId>/ Read GET None Update Board http://localhost:8080/users/<userId>/boards/<boardId>/ Update PUT boardName boardDesc category isPrivate(Boolean) Delete Board http://localhost:8080/users/<userId>/boards/<boardId>/ Delete DELETE None Create Pin http://localhost:8080/users/<userId>/boards/<boardId>/p ins/ Create POST pinName image description Get a single pin from a board http://localhost:8080/users/<userId>/boards/<boardId>/p ins/<pinId>/ Read GET None Get all pins from a board http://localhost:8080/users/<userId>/boards/<boardId>/p ins/ Read GET None Update a pin http://localhost:8080/users/<userId>/boards/<boardId>/p ins/<pinId>/ Update PUT pinName image description Delete a pin http://localhost:8080/users/<userId>/boards/<boardId>/p ins/<pinId>/ Delete DELETE None Add a comment to a pin http://localhost:8080/users/<userId>/boards/<boardId>/p ins/<pinId>/comment/ Create POST description View comments on a pin http://localhost:8080/users/<userId>/boards/<boardId>/p ins/<pinId>/comment/ Read GET None
  4. 4. The responses were returned in JSON format and contained relevant data and endpoint information for subsequent operations as agreed to by the class. Web Services Why Web Services fit in this project? Essentially, web services help de-couple the client and the server implementations. User performs various CRUD operations on an external system by sending HTTP requests through the client. The web service layer, intercepts these requests on the server, performs operations on the database and sends back appropriate response to the client. This fits well with the sharing and collaboration required by social media products. From real world examples such as Facebook, Twitter and Pinterest, it can be seen that a robust and well documented RESTful API is a vital contributor to these products going viral. Bottle: Web services were written in Python using the Bottle micro-framework. We found Bottle to be very intuitive for creating our web services. The framework is very lightweight and in order to use it, it only requires us to include one bottle.py file in the project directory. Server: Bottle runs by default on a single threaded WSGI server, but has adapters to various multi-threaded servers such as paste and Cherrypy which give better performance for applications that need to handle large number of concurrent requests. For testing this project, we have used the default server. Mapping requests to appropriate methods: Various endpoints as mentioned in the functionalities section, map easily to corresponding methods which process the requests at those endpoint, simply by placing a @route decorator above that method. Retrieving data from request: In REST, the URL path determines the data that will be accessed or manipulated and the HTTP method determines the operation that will be performed on that data. - To get information about data that will be accessed, we use wildcards in the URL: @route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’GET’) def getBoard(userId, boardId): # got userId and boardId from the URL, now perform GET(read) on database - To retrieve data from the HTTP request, bottle provides convenient methods in the request object: @route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’PUT’) def updateBoard(userId, boardId): # got userId and boardId from the URL, now perform PUT(update)on database # with the fields sent in the request by the client boardName = request.forms.get('boardName') boardDesc = request.forms.get('boardDesc') Returning JSON response: JSON was the decided as the format in which response will be sent back to the client. Even in this case, Bottle made it easy to return back response with “application/json” content type. We simply had to
  5. 5. return a python dictionary object. Bottle returns the response in JSON and the appropriate field is set in the HTTP header. @route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’PUT’) def updateBoard(userId, boardId): # perform appropriate operations return dict(status='OK', message='sample message') If other response types such as html or xml are expected by the client, we are aware that we would have to check for the content types which the client accepts from the HTTP request header and accordingly format the response before sending it back (content negotiation). Handling trailing ‘/’ slash: If some clients put a trailing slash at the end of the endpoint URL, then that is treated as a separate route by Bottle and might not map to the appropriate method in our web service code. Given that we would be testing across teams, this was highly probable. To handle this, Bottle documentation suggests to simply map two routes to the same method, one with a trailing slash and one without a slash. @route(‘http://localhost:8080/users/<userId>/boards/<boardId>/’,method=’PUT’) #No trailing slash @route(‘http://localhost:8080/users/<userId>/boards/<boardId>’,method=’PUT’) def updateBoard(userId, boardId): # perform appropriate operations Database In REST web services, we can think of the URL as a path that is traversed to get to the data that is to be accessed or manipulated. Further, the HTTP method that is specified in the HTTP request gives us the operation (create, read, update or delete) that is to be performed on the data. Accordingly the actions that we take in code depend on the combination of the following factors: 1. Response format accepted by the client (json, xml, html, image, etc..) 2. Data that is to be accessed (URL traversal) 3. What kind of operation on the data (HTTP method type) Thus, for every endpoint there are several combinations that need to be handled so that we process the user request appropriately and so that we return desired response back to the user. In this context, it is clear that we have to organize our data based on the endpoints that we expose to the user. We observed that this is a notable difference compared to Project 1, where there was only one endpoint for the client, and the server queue delegated the message to the appropriate resource to handle. In Project 2, there are several endpoints for the client and each endpoint maps to one method on the server side. The flat schema that we have used to store data in CouchDB is as follows. The “_id” is the userId that is generated server side when the user signs up. This is passed around in each URL and is used to identify user in each subsequent request.
  6. 6. { "_id": { "firstName": "", "lastName": "", "emailId": "", "password": "", "isLoggedIn": false, "boards": [ { "boardId": "", "boardName": "", "boardDesc": "", "category": "", "isPrivate": "", "pins": [ { "pinId": "", "pinName": "", "image": "", "description": "", "comments": [] } ] } ] } }
  7. 7. Drawback: The limitation of this schema is that for any request, we get the entire complex document associated with a userId from the database. Then, in case of long URLs we have to employ multiple nested loops to get to the data that we require. For example, updating information about a pin needs three nested loops through the document that is returned. This is shown in the figure above. This would be problematic if a user had thousands of boards and pins. A solution to this would be to create separate tables (“databases” in CouchDB) for each step in the endpoint URL tree, i.e. separate tables for user, board and comments, each record containing reference to the userId. However, the tradeoff here is that this would require more queries to the database per user request. Note on Aerospike: Initially we decided on using Aerospike as our database. However, its Python client seems to be under development and there was no support for list and map data structures. Further, there was no query support in the Python client, only full table scan was supported. Hence we decided to go with CouchDB. Nonetheless, it was interesting to understand Aerospike’s architecture and concepts such as location awareness of client, peer relationship among nodes, use of binary wire protocol for communication and cross data center replication. Conclusion For our team, following are the major learnings from this project: 1. Programming in Python 2. RESTful web services 3. Database design for RESTful web services 4. Micro-frameworks 5. Comparison of technologies used in Project 1 and 2 6. CouchDB and Aerospike 7. Collaboration in Small and Large Software Development Teams

×