Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

PiterPy 2016: Parallelization, Aggregation and Validation of API in Python

1,374 views

Published on

There's plenty of different approaches for building simle and robust API: REST, SOAP, RPC and so on. Some of them are too verbose, some – too complex. This presentation is a journey of finding balance between simplicity and flexibility, performance and robustness.

Published in: Software
  • Be the first to comment

PiterPy 2016: Parallelization, Aggregation and Validation of API in Python

  1. 1. Parallelisation, aggregation and validation API with Python Max Klymyshyn CTO at CartFresh @maxmaxmaxmax
  2. 2. ‣ 12+ years of experience, 7 years with Python, 6 with JS ‣ Was part of oDesk, Helios, 42cc. ‣ Co-organizer of PyCon Ukraine, KyivJS, Papers We Love ‣ CTO at CartFresh ‣ Challenging myself with english talk. It’s not my first language, bear with me About
  3. 3. ‣ Grocery Delivery startup ‣ Operating as CartFresh (Boston, US) and ZAKAZ.UA (Kiev, Dnepropetrovsk, Kharkiv, Ukraine) ‣ Apache CouchDB, Apache Solr, Redis ‣ Heavy python on back-end CartFresh
  4. 4. ‣ Quick overview ‣ Some abstract info about context ‣ Tools for Python Table of contents
  5. 5. Why API again?
  6. 6. World is changing very quickly: ‣ Mobile apps ‣ Internet of Things ‣ Microservices ‣ Isomorphic apps Why API again?
  7. 7. Good API is hard when all your stuff should work together well
  8. 8. ‣ Validation ‣ Reusability ‣ Consistency ‣ Maintainability ‣ Scalability It’s challenging
  9. 9. Good API makes it easier to develop a service
  10. 10. Divide and conquer (D&C)
  11. 11. ‣ API expresses a software component in terms of its operations, inputs, outputs, and underlying types ‣ API helps create reusable building blocks and communicate between system components ‣ It opens new opportunities to develop new systems based on your product Overview
  12. 12. Moving parts VALIDATION OUTPUT BL INPUT
  13. 13. ‣ input – need to be validated for type correctness ‣ validation – input should be constrained by domain- specific business rules ‣ business logic – obviously most useful part of the system ‣ output – data model, serialised into specific format Moving parts
  14. 14. API creation becomes trivial with good understanding and right tools All challenges are behind: how to make it simple, how to make it maintainable, how to keep API users updated
  15. 15. Trends during the past few years
  16. 16. ‣ RESTification ‣ Data Query Languages ‣ Microservices architecture Trends
  17. 17. REST ‣ Unified interface to communication protocol between client and API ‣ Built on top of HTTP ‣ Simple
  18. 18. Data Query Languages ‣ GraphQL ‣ Falcor ‣ Datalog ‣ Datomic etc.
  19. 19. Data Query Languages Main point of DQL is to make declarative composition of queries to simple data structures and represent it as single data structure
  20. 20. Common case
  21. 21. Monolithic service Monolite
  22. 22. More realistic case
  23. 23. Microservices Monolit Microservices
  24. 24. Microservices Monolit Microservices Difference
  25. 25. ‣ New layer of complexity in terms of input validation ‣ New unreliable layer (network) ‣ Additional protocol overhead ‣ Communication latency Seriously
  26. 26. ‣ You’ll get a chance to improve each piece of code separately without breaking other part of the system (D&C!) ‣ You can split development of microservices between different dev teams ‣ You’ll get a lot of fun! But let’s be optimistic
  27. 27. Tools
  28. 28. ‣ SWAGGER – a simple representation of your RESTful API (OpenAPI initiative), FLEX for Python ‣ RESTful API Modelling Language – RAML ‣ APIDOC – a documentation from API annotations in your source code ‣ api-blueprint, RESTUnite, apiary etc. API Frameworks
  29. 29. paths: /products: get: summary: Product Types description: | The Products endpoint returns information about the *Uber* products offered at a given location. The response includes the display name and other details about each product, and lists the products in the proper display order. parameters: - name: latitude in: query description: Latitude component of location. required: true type: number format: double - name: longitude in: query description: Longitude component of location. required: true type: number format: double tags: - Products responses: 200: description: An array of products schema: type: array items: $ref: '#/definitions/Product' Swagger spec example
  30. 30. /products: uriParameters: displayName: Products description: A collection of products post: description: Create a product #Post body media type support #text/xml: !!null # media type text, xml support #application/json: !!null #media type json support body: application/json: schema: | { "$schema": "http://json-schema.org/draft-03/schema", "product": { "name": { "required": true, "type": "string" }, "description": { "required": true, "type": "string" } RAML spec example
  31. 31. example: | { "product": { "id": "1", "name": "Product One", ... } } get: description: Get a list of products queryParameters: q: description: Search phrase to look for products type: string required: false responses: 200: body: application/json: #example: !include schema/product-list.json RAML spec example
  32. 32. To prevent situation when documentation, client libraries, and source code get out of sync CLIENT #1 SERVER CLIENT #2
  33. 33. ‣ Predefined input parameters + validation ‣ Predefined response schema (model) ‣ Query Language Aggregation
  34. 34. GraphQL/Graphene import graphene import pprint data = [1, 2, 3, 4] class Query(graphene.ObjectType): hello = graphene.String() data = graphene.String() def resolve_data(self, args, info): return ",".join(map(str, data)) def resolve_hello(self, args, info): return 'World' schema = graphene.Schema(query=Query) result = schema.execute('{ hello, data }') pprint.pprint(result.data) # OrderedDict([('hello', u'World'), ('data', u'1,2,3,4')])
  35. 35. GraphQL’s power comes from a simple idea —  instead of defining the structure of responses on the server, the flexibility is given to the client. GraphQL vs REST
  36. 36. GraphQL/graphene allow us to use our beloved language for declaration of Model/API Schema: python GraphQL vs Swagger
  37. 37. Batching
  38. 38. Tools: django-batch-requests [ { "method": "get", "url": "/sleep/?seconds=3" }, { "method": "get", "url": "/sleep/?seconds=3" } ]
  39. 39. [ { "headers": { "Content-Type": "text/html; charset=utf-8", "batch_requests.duration": 3 }, "status_code": 200, "body": "Success!", "reason_phrase": "OK" }, { "headers": { "Content-Type": "text/html; charset=utf-8", "batch_requests.duration": 3 }, "status_code": 200, "body": "Success!", "reason_phrase": "OK" } ]
  40. 40. Our experience
  41. 41. ‣ End up with batched API interface ‣ Declarative input validation with trafaret ‣ Free schema (disadvantage) ‣ Very simple SQL-JOIN-like aggregation
  42. 42. Params, validation, transformation @validate_args( _('Invalid request'), store_id=tr.String() >> pipe(unicode, unicode.strip), slugs=tr.List(tr.String() >> pipe(unicode, unicode.strip)), ean=tr.String | tr.Null, extended=tr.Bool | tr.Null, query=tr.String | tr.Null, facets=tr.List( tr.List(tr.String, min_length=2, max_length=2)) | tr.Null, sort=tr.String(allow_blank=True) | tr.Null, _optional=('extended', 'query', 'facets', 'sort', 'ean')) def resource_products(store, user, session, limit=None, offset=1, lang='en', args=None, **kwargs): pass
  43. 43. [ "store.products", { store_id: Storage.first(“store").id, slugs: [options.slug], facets: options.facets || [], sort: options.sort || “" }, { offset: options.offset || 1, id: "catalog", join: [{ apply_as: "facets_base", on: ["slug", "slug"], request: { type: "store.facets", args: { store_id: "$request.[-2].args.store_id", slug: "$request.[-2].args.slugs|first" } } }, { apply_as: "category_tree", on: ["slug", "requested_slug"], request: { type: "store.department_tree", args: { store_id: "$request.[-2].args.store_id", slug: "$request.[-2].args.slugs|first" } } }] } ]
  44. 44. Thanks. @maxmaxmaxmax

×