Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The DNA of a great API

880 views

Published on

Find out what it takes to build API's that your target audience will love while also keeping it business focused.

Published in: Technology

The DNA of a great API

  1. 1. The DNA of a GREAT API Ciprian Sorlea, CTO October 2015 Building API’s that developers will love
  2. 2. 06.11.2015 2 What’s an API: Any API: • allows Software components, systems and subsystems to interact. • hides the complexity of the underlying architecture and functionality • focuses on separation of concerns • allows callers/consumers to access resources (ex. CRUD operations) or to perform various operations on top of the resources
  3. 3. 06.11.2015 3 Qualities of a good API: A good API must be: • domain specific, simple and easy to use • consistent (naming, parameters, etc.) • stateless • transparent & clear in terms of error trapping • accessible (from different platforms, languages, etc.)
  4. 4. 06.11.2015 4 From Good To Great: A great API must be: • well documented and should provide many working samples • fast, secure, scalable and reliable • forward and backwards compatible (versioning) • “self aware”: usage statistics, logs/traces
  5. 5. 06.11.2015 5 From Good To Great: A great API must go the extra mile
  6. 6. 06.11.2015 6 Who are the consumers? An API can be consumed by: - A web application - A desktop application - A mobile application - Another API - An integration test - A 3rd party integration - Internet Of Things - A home appliance, a wearable device, a smart gadget
  7. 7. 06.11.2015 7 Now, let’s remember…. Who is your target audience? Developers
  8. 8. 06.11.2015 8 A big question & challenge: How do we handle all these uses cases, while also making it easy for Developers?
  9. 9. 06.11.2015 9 Things to consider: - Transport: how do we exchange data? - Resource models: what kind of data do we exchange? - API Architecture patterns: how do we design our API? - Error Handling: how do we handle errors & failure? - Testing: how do we make our API bullet proof? - Security: how do we protect our data (or ourselves)? - Manageability: how can we manage our API and support it operation-wise? - Evolution: how do we ensure our API is future-proof? - Adoption: how do we make sure our API will be adopted?
  10. 10. API Transport how do we exchange data?
  11. 11. 06.11.2015 11 API Transport REST API: - Recommended for external integrations and applications, 3rd party apps and plugins - Use JSON & XML over HTTP(s) - Use established Security mechanisms (OAuth2, Basic Authentication, etc.)
  12. 12. 06.11.2015 12 API Transport RPC / Binary Protocol: - Recommended for internal usage or high throughput requirements, IOT and devices with limited networking capabilities / bandwidth - Use Apache Thrift, Google Protobuff or alternatives - Use simplified authentication and authorization model to minimize overhead – assume working in safer environment
  13. 13. Note: For now, we’ll focus on REST API’s
  14. 14. API Resource Models what kind of data do we exchange?
  15. 15. 06.11.2015 15 API Resource Models Resources are the heart and soul of the API Resources represent the abstraction of any information we exchange and operate on within an API Domain Model ~= Resource Model
  16. 16. 06.11.2015 16 API Resource Models Resource classification: - Singleton – can be part of a collection or can be stand-alone - ServerInfo – Server Information can be exposed as a standalone singleton resource - User – User information can be exposed as a resource, part of a collection - Collection – a list of so called documents belonging to the same domain - Users - Products - Orders - Documents
  17. 17. 06.11.2015 17 API Resource Models Minimal Resource Model Attributes: - URI: Unified Resource Identifier - represents the unique URL by which a resource can be accessed - HATEOAS takes this to the extreme level - Ex: http://api.domain.com/v1/serverInfo, http://api.domain.com/v1/users/1, http://api.domain.com/v1/projects/ - ID: - required when the resource is a singleton as part of a collection - a numeric ID (ex: 1029) or a GUID (ex. c366728d-ca93-4b23-a4b9-ae360600b86a) - can be the database specific primary key (ex. auto generated) or can use a custom field with random generated values, to enhance security
  18. 18. 06.11.2015 18 API Resource Models Important consideration: - Ensure your resource models can be properly represented in any transport models (ex. JSON, XML) and without significant effort (Ex. Don’t try to use complicated bit level data in your models) - Follow the Open Close principle. This will make your resources future-proof - Ensure the API resource model doesn’t carry with it the DB optimizations and complexity
  19. 19. API Architecture Patterns (archetypes) how do we design our API?
  20. 20. 06.11.2015 20 API Archetypes The de-facto API Archetypes: • Document – a singleton resource • Collection – a collection of resources • Repository (also known as Store) – a collection of immutable resources • Controller – an action dispatcher *** *** Should be avoided. Explanation later.
  21. 21. 06.11.2015 21 API Archetypes Pseudo API Archetypes: • Queue – asynchronous processing of a long operation – can also be a batch operation or a transaction, but not limited to those • Should provide status and progress information • Ex: File processing, order processing, etc. • Transaction – multiple resource operations to be processed in a ALL-OR-NONE manner • The request fails if any of the operation fails. • Any simple action (see Controller) can be also seen/handled as a transaction • Batch – a sequence of resource operations to be processed in sequence • Any failure is reported to the consumer, but doesn’t stop the processing of the batch • Should provide status and progress information • Ex. Batch create, batch update, etc.
  22. 22. 06.11.2015 22 API Archetypes: Document • Exposes functionality specific to a singleton resource. • Some standalone documents (ex. ServerInfo), can be read-only • Some collection-owned documents are mutable (can be updated, deleted, etc.) -> parent archetype is a Collection
  23. 23. 06.11.2015 23 API Archetypes: Document – Partial Retrieval Partial retrieval allows consumer to limit the transferred information. This leads to better performance and improved customer experience. Limiting the fields: • Specify the fields to be included, as a query string parameter • Specify the fields to be ignored, as a query string parameter • Specify the fields to be expanded (sub-details), as a query string parameter • Specify a predefined view (predefined or custom definition of data retrieval model)
  24. 24. 06.11.2015 24 API Archetypes: Document – Partial Retrieval Examples: http://api.domain.com/v1/users?fields=id,firstName,lastName,email,addresses(city) http://api.domain.com/v1/users?excludedFields=email http://api.domain.com/v1/users?view=minimalView http://api.domain.com/v1/users?expand=addresses,previousEmails
  25. 25. 06.11.2015 25 API Archetypes: Document – Partial Update Allows consumers to update only those fields that need to be changed This provides faster and better user experience. Two ways: - Specifying the fields to be changed with the new value (partial documents) - Using operation maps
  26. 26. 06.11.2015 26 API Archetypes: Document – Partial Update Using partial documents: PATCH /v1/users/1029 HTTP/1.1 Content-Type: application/json X-API-PUBLIC-KEY: D6DB835A-35B1-42B3-8805-8F00F5344BB9 X-API-REQUEST-ID: 5BF75F7E-0A55-40DF-9E79-354B9B162971 { “emailAddress”: ciprian.sorlea@nordlogic.com, “companyName”: “NordLogic” “phoneNumbers” : [“+4074ILOVEAPIS”] }
  27. 27. 06.11.2015 27 API Archetypes: Document – Partial Update Advantages of using partial documents: - Request very clean and minimal - Requires very little extra effort to implement Disadvantages: - Hard to add items to lists / arrays, remove values from maps, etc.
  28. 28. 06.11.2015 28 API Archetypes: Document – Partial Update Using operation maps: PATCH /v1/users/1029 HTTP/1.1 Content-Type: application/json X-API-PUBLIC-KEY: D6DB835A-35B1-42B3-8805-8F00F5344BB9 X-API-REQUEST-ID: 5BF75F7E-0A55-40DF-9E79-354B9B162971 { “emailAddress”: { “set”: “Ciprian.Sorlea@nordlogic.com”}, “companyName”: { “set” : “NordLogic”}, “phoneNumbers” : { “add” : [“+4074ILOVEAPIS”] } }
  29. 29. 06.11.2015 29 API Archetypes: Document – Partial Update Advantages of using operation maps : - More powerful control over what operations are made - Doesn’t require pre-fetching of documents prior to update to get the actual map/list content Disadvantages: - Requests tend to be more verbose, even though just slightly - More effort is needed to implement support for this approach (usually)
  30. 30. 06.11.2015 30 API Archetypes: Document – Partial Update PRO TIP: You can implement Partial Update with either approach, or you can combine them and allow patching in either way.
  31. 31. 06.11.2015 31 API Archetypes: Document – Partial Retrieval PRO TIP: Combining Partial Update with Partial Retrieval, APIs can offer extreme flexibility for a wide variety of consumers & devices.
  32. 32. 06.11.2015 32 API Archetypes: Document – Partial Retrieval PRO TIP: You can implement transactions for updating fields that require additional details on change: Ex. Change status + comment
  33. 33. 06.11.2015 33 Document Archetype: URI & HTTP Verbs • Base URI: http://api.domain.com/{version}/{documentID} or http://api.domain.com/{version}/{collection}/{documentID} HTTP Verb Description GET Retrieves the document (complete or partial) PUT Updates a document with the content provided. Only for Collections. PATCH Updates a document fields specified in the content, or applying the specific update operations. Only for Collections. DELETE Deletes the document from the location HEAD Verifies if the given document id exists. OPTIONS Lists the available operations / HTTP Verbs
  34. 34. 06.11.2015 34 API Archetypes: Collection Collections provide access to multiple document resources Operations are usually CRUD Collection capabilities: - Pagination - Filtering: using simple filtering or a query language) - Sorting: simple (based on single field) or complex (based on multiple fields) - Partial field retrieval: identical to document level partial field retrieval
  35. 35. 06.11.2015 35 API Archetypes: Collection Query String Query String parameters are used to configure collection functionality (ex: sorting, filtering, etc.) Rules & recommendations: - Use same QS parameter names across your entire API, and document them - Ensure you don’t introduce un-necessary constraints by choosing improper names - As long as you don’t confuse your consumers, you can allow aliases for some parameters Ex: - Page & PageSize work for a basic pagination but not for continuous scrolling – improper naming - Start & limit work for any data pagination scenario - Offset/From and count can be aliases to start & limit
  36. 36. 06.11.2015 36 API Archetypes: Collection Query String Pagination Ex: http://api.domain.com/v1/users?from=100&count=10 http://api.domain.com/v1/users?count=10 (uses a default offset/start)
  37. 37. 06.11.2015 37 API Archetypes: Collection Query String Sorting - can be simple (single field + direction) or complex (list of fields, with specific sort orders) - can be applied on 1st level fields or on details (expanded content) Ex: http://api.domain.com/v1/users?sortBy=email,sortOrder=DESC http://api.domain.com/v1/users?orderBy=name:ASC,email:DESC,contactInfo.phone
  38. 38. 06.11.2015 38 API Archetypes: Collection Query String Filtering: - Simple filters (all filters are of type EQUALS): http://api.domain.com/v1/users?email=Ciprian.Sorlea@nordlogic.com http://api.domain.com/v1/users?companyName=Nordlogic&role=developer - Advanced filters (specify the operation: EQ, NE, GT, LTE, etc.) http://api.domain.com/v1/users?companyName=LIKE:*logic&role=NE:qa - Query (use a pseudo query language): http://api.domain.com/v1/users?query=companyName like ‘*logic’ and role not eq ‘QA’ http://api.domain.com/v1/users?query=name eq ‘John’ or name eq ‘Bill’
  39. 39. 06.11.2015 39 Collection Archetype: URI & HTTP Verbs • Base URI: http://api.domain.com/{version}/{collection}/ HTTP Verb Description GET / Retrieves a list of paged, filtered and sorted documents, complete or partial GET /{docID} Retrieves the document (complete or partial) POST / Creates a new document in the collection HEAD Can be used to check if data exists given the provided filters and paging DELETE Purges the collection. Recommended for Development only. OPTIONS Lists the available operations / HTTP Verbs
  40. 40. 06.11.2015 40 Collection Archetype: Response Few tips: - Always echo the pagination details (ex. offset & count), to provide context - Give more details about the rest of the data (is there more?). Ex: maxRows - Don’t include empty/null fields and ensure (through documentation) your consumers handle that. (This applies to single documents as well)
  41. 41. 06.11.2015 41 API Archetypes: Repository Repositories provide access to multiple immutable document resources Operations are usually CRD (Create, Retrieve, Delete) Updates to documents within Repositories MIGHT happen behind the API level Nice to have capabilities: - Capped repositories: limit data to most recent X records - Auto Expiration: automatically delete data older than X days/hours/minutes - Tailing options: retrieve latest X records
  42. 42. 06.11.2015 42 Repository Archetype: URI & HTTP Verbs • Base URI: http://api.domain.com/{version}/{repository}/ HTTP Verb Description GET / Retrieves a list of documents with limited capabilities (ex. Tailing over the records in an audit/trace log). Nov very common. GET /{docID} Retrieves the document (complete or partial). POST Creates a new document in the repository HEAD When GET is also implemented, this can be used to check if data exists. DELETE Purges the repository. Recommended for Development only. OPTIONS Lists the available operations / HTTP Verbs
  43. 43. 06.11.2015 43 API Archetypes: Controller Allows execution of application specific actions that cannot be mapped with any of the CRUD methods (specific to collections or repositories). Should be avoided as much as possible. Actions should be reconsidered as one of the proposed pseudo-archetypes: batch, queue, transaction. This would allow execution to be more manageable and traceable.
  44. 44. 06.11.2015 44 Controller Archetype: URI & HTTP Verbs • Base URI: http://api.domain.com/{version}/{param1}/{param2}/{controllerAction} HTTP Verb Description POST Executes the specific action, with the given parameters
  45. 45. 06.11.2015 45 API Archetypes: Queue Allows execution of application specific actions (jobs) in an asynchronous way, while allowing proper management and traceability of the action. Each action, once queued, has an identifier and a status. Therefore each action becomes a Resource. Upon completion, the job document can embed the job result or can provide a link to the actual result.
  46. 46. 06.11.2015 46 Queue Archetype: URI & HTTP Verbs • Base URI: http://api.domain.com/{version}/{queue}/ HTTP Verb Description GET /{id} Retrieves the queued job status, progress information and execution result – if available. POST Submits a new job into the queue and retrieves the job identifier. DELETE /{id} Cancels a job (if the status allow the operation to be canceled) OPTIONS Lists the available operations / HTTP Verbs
  47. 47. 06.11.2015 47 API Archetypes: Transaction Allows execution of application specific actions in transactional manner. Transactions can be asynchronous (see queues) or asynchronous. In case of synchronous transactions, response contains the transaction result.
  48. 48. 06.11.2015 48 Transaction Archetype: URI & HTTP Verbs • Base URI: http://api.domain.com/{version}/{queue}/ HTTP Verb Description GET /{id} Retrieves the transaction status, progress information and execution result – if available. POST Submits a new transaction to be processed. DELETE /{id} Stops/cancels a transaction (in case of asynchronous/queued transactions) OPTIONS Lists the available operations / HTTP Verbs
  49. 49. 06.11.2015 49 API Archetypes: Batch Allows execution of application specific actions over multiple records / bulk processing. Batches must always be asynchronous, and the behavior is identical to Queues. Progress information is mandatory in case of batches.
  50. 50. 06.11.2015 50 Batch Archetype: URI & HTTP Verbs • Base URI: http://api.domain.com/{version}/{batch}/ HTTP Verb Description GET /{id} Retrieves the status, progress information and execution result – if available. POST Submits a new batch job to be processed. DELETE /{id} Stops/cancels a batch job OPTIONS Lists the available operations / HTTP Verbs
  51. 51. 06.11.2015 51 Pseudo Archetypes PRO TIP: You can combine Queues with Transactions and Batches in any way you want. Just make sure things make sense 
  52. 52. 06.11.2015 52 Pseudo Archetypes PRO TIP: Sometimes you might want to have, for each Pseudo Archetype, a validation equivalent. Helps your applications/integrations cause less frustrations.
  53. 53. Error Handling how do we handle errors & failure?
  54. 54. 06.11.2015 54 API Error Handling Use specific HTTP Error Codes where appropriate for application logic specific errors. Ex: - 401 Unauthorized: errors with authentication / authorization - 404 Not Found: document does not exist - ….
  55. 55. 06.11.2015 55 API Error Handling Response body should always contain: - Error code – what happened, coded for analytics - Error message – human readable message, localized to user’s profile - Error details – the context on the error. Nice to have: - Error ID – id of the error, generated by the lowest level module where the error occurred. Should also propagate and find itself in the log files. - Flow ID – if possible, for cross-app tracking
  56. 56. 06.11.2015 56 Pseudo Archetypes PRO TIP: Certain operations can generate multiple errors (ex. multiple validation errors). If applicable, provide a list of errors in the response instead of providing just the first error encountered.
  57. 57. 06.11.2015 57 Pseudo Archetypes PRO TIP: Don’t provide too much details in the error messages. Consider privacy and security before anything else.
  58. 58. Testing how do we make our API bullet proof?
  59. 59. 06.11.2015 59 API Testing Option 1: - Develop API Client along the actual API Service - Build Automated tests using the API Client, using the same programing language
  60. 60. 06.11.2015 60 API Testing Option 2: - Build Automated tests using a dedicated API Testing framework • http://frisbyjs.com/ • https://github.com/mkotsur/restito • https://github.com/rest-driver/rest-driver • https://github.com/dareid/chakram
  61. 61. 06.11.2015 61 API Testing Option 3: - Use dedicated 3rd Party apps or Browser Plugins • https://www.getpostman.com/ • http://www.soapui.org/rest-testing/getting-started.html
  62. 62. Security how do we protect our data (or ourselves)?
  63. 63. 06.11.2015 63 Securing Rest API Simple rules: - Use disposable authentication tokens - Implement authentication mechanisms good for your target audience Ex: OAuth2 works very well for web and mobile, but does it work for enterprise? - Use granular roles & access rights for each resource model - Take the multi-tenancy into account from start - Use & require HTTPS whenever sensitive data is in transit - Encrypt any sensitive data at rest
  64. 64. 06.11.2015 64 Securing Rest API Preventive & Enterprise Ready Security: - Audit any failed authentication / authorization - When using sessions, ensure the expiration time can be configured - Lock/disable authentication tokens after a configurable number of failed authentication attempts - Use a whitelist / blacklist for request filtering - Implement overall rate limits as well as application / token specific limits - Validate any data coming IN. Never take things for granted.
  65. 65. 06.11.2015 65 Securing Rest API Preventive & Enterprise Ready Security: - Require (configurable) password strength - Use password expiration - Log Remote IP, User-Agent (and Application ID, if possible) with any request - Pass authentication tokens in the request headers, not in URL/Query String - Filter unaccepted query string parameters from URL’s - Escape/strip any textual information accepting user input
  66. 66. 06.11.2015 66 Pseudo Archetypes PRO TIP: NEVER, EVER, EVER, EVER TRUST YOUR USERS.
  67. 67. Manageability how can we manage our API and support it operation- wise?
  68. 68. 06.11.2015 68 Managing Rest API & Supporting Operations Simple rules: - Log everything, almost everything. - Implement usage analytics and metrics - Time any operation, build performance metrics and alerts - Build operational dashboards, enable error notifications - Use change logs & audit trails for sensitive data - Require “4 eyes check” rule on any major change performed by admins - Use infrastructure as code
  69. 69. Evolution how do we ensure our API is future-proof?
  70. 70. 06.11.2015 70 Future proof your API Simple rules: - Version your resource models and archetypes from the beginning - Any new version of the resource model should extend the previous versions - Establish a support plan from first public release - Classify your API’s as public, private & unsupported to ensure phased adoption
  71. 71. Adoption how do we make sure our API will be adopted?
  72. 72. 06.11.2015 72 Ensuring APIs adoption - Make it easy to get started - developers should be 1 CURL request away from their first API call. - Provide good and clear documentation. - Provide SDK’s / client libraries and samples for most popular development platforms and languages of your target audience.
  73. 73. 06.11.2015 73 Ensuring APIs adoption - Ensure your business logic & rules don’t have to be guessed and duplicated by developers in their own apps (ex. validation rules). - Eat your own dog food. Use your API to build your other products / integrations. - Don’t introduce breaking changes. Plan to support your public API releases.
  74. 74. 06.11.2015 74 Ensuring APIs adoption PRO TIP: Deliver VALUE, not just JSON or XML.
  75. 75. Thanks for watching. 

×