This session is a deep dive as well as an interactive discussion on design principles, considerations, lessons learned from mistakes that can be taken into account when developing RESTful services. It will cover a variety of topics from Designing of RESTful resources, Versioning,Exception Handling, Caching, Validation, Security, Rate limiting, HATEOAS, Testing and Documentation. This talk will walk through and compare the different REST API provided by companies like Twitter, Paypal, Google, Stripe and more we can learn the good, the bad and ugly. So join me in this talk to build high quality applications that can be highly scalable, available and reliable. Summary: Learn all you ever wanted to learn about RESTful services development challenges in large scale applications Outline: This session is a deep dive as well as an interactive discussion on design principles, considerations, lessons learned from mistakes that can be taken into account when developing RESTful services. It will cover a variety of topics from Designing of RESTful resources, Versioning, Exception Handling, Caching, Validation, Security, Rate limiting, HATEOAS, Testing and Documentation. This talk will walk through and compare the different REST API provided by companies like Twitter, Paypal, Google, Stripe and more we can learn the good, the bad and ugly. So join me in this talk to build high quality applications that can be highly scalable, available and reliable.

1. Real-World RESTful Service Development Problems and Solutions
Masoud Kalali @MasoudKalali

2. Safe Harbor Statement
The following is intended to outline our general product direction. It is intended for
information purposes only, and may not be incorporated into any contract. It is not a
commitment to deliver any material, code, or functionality, and should not be relied upon
in making purchasing decisions. The development, release, and timing of any features or
functionality described for Oracle’s products remains at the sole discretion of Oracle.

3. Speaker
• Masoud Kalali
• Works at ORACLE
• Software engineer, author, blogger…
• @MasoudKalali

4. Program Agenda
Introduction
Request and Response….
A little bit of security
Performance matters!
More on Request and Response
1
2
3
4
5

5. Introduction…
To jog our memories 

6. Caching
Rate
Limiting
Authz Logging
RESTful
Service
Endpoint
Micro
Services
Content
Neg.
HTTP Request
HTTP Response
HTTP Response
Status
PaginationHATEOAS
CDN
EIS

7. Request and Response….
Content Negotiation

8. Flexible Request and Response - The problem
• Produce multiple format of the representation
– JSON
– XML
– HTML
• Produce multiple variation of the representation
– For privileged user compared to non privileged user
– For older or newer client versions
– For different type of clients
Flexible response types

9. Flexible Request and Response
• Produce multiple representation of the same resource
– Based on HTTP headers, accept and media-type (custom/standard)
• List of standard media-types: http://www.iana.org/assignments/media-types/media-types.xhtml
• Custom domain oriented media-types, e.g application/vnd.com.oracle.paas-service+json
– Using URI pattern e.g:
• http://api.domain.com/v1.1/customers/1/orders.xml
• http://api.domain.com/v1.1/customers/1/orders.json
– Agent driven
• Client decide what representation to choose
• Server sent different representations (300 or 406 status and required headers)
– Combination of all
Content Negotiation

10. Flexible Request and Response – The solution
• On the server side
– @Produce
• Should have matching media-type with request accept header
• Non matching results in 406 - Not Acceptable
– @Consume
• Should have matching media-type with request content-type header
• Non matching results in 415 - Unsupported Media Type
• On the client side
– Set correct content-type
– Set expected accept header
Content Negotiation: JAX-RS and flexible response types

11. Request and Response….
Resource Versioning

12. Resource Versioning: The Problem
• Request and or response evolves to be incompatible
– Some attributes are removed from request payload or required in the payload
– Some attributes are added or removed or changed semantically in the response
• Business semantic evolves to become incompatible
– E.g the default is no longer set to be a small coffee cup
– The provisioned service, in a PAAS environment, has caching included by default
– Etc.
How to evolve a resource?

13. Resource Versioning: Solution alternative
• Version added to resource (URI) (Facebook and Twitter)
• /v1.1/resource/path/orders
• /resource/path/orders?version=1.1
• Client is locked to the version
• Easy to maintain in the server side
• Requires all or none migration of the clients
• Problems like linked resource address stored in client side…
How to evolve a resource?

14. Resource Versioning: Solution alternative
• Version is negotiated as part of the request using accept header
– Accept=application/vnd.com.oracle.paas.service-v1.1+json
– Accept=application/vnd.com.oracle.paas.service-v2.1+json
• Server code need to handle all supported previous versions
– Same endpoint handling all versions and combination of them
• Easy for clients to use what thy want
• GitHub uses this format
How to evolve a resource?

15. Resource Versioning: Solution alternative
• No versioning at all!
– There is one interface that evolve every e.g 2 years
• Clients must migrate to the new API
• Old api will be shutdown
– Easy to maintain
– Easy to migrate
• Hard to convince clients to migrate before the cut-off
How to evolve a resource?

16. Request and Response….
Validation

17. Validating Request: Problem
• When and how to validate the requests?
– Are parameters in expected format
– Is the payload Json as it is supposed to be?
• When and how to verify the requests?
– Are the parameters meaningful? E.g is there any order with that ID
– Is the payload in expected JSON schema?
Validation before action!

18. Validating Request: Solution
• Validation goes with versioning and content negotiation
• Unified validation patterns across the codebase
– Codified response format
– Unified response body
• Use annotations and standard validation as much as possible
– Supports common media-types
– Unified with custom coding/template being added
Validation before action!

19. Request and Response….
Exception Handling

20. Exception Handling
• Please don’t send back stack trace
– Use an ExceptionMapper as Provider as last line of cleanup!
– Unless in development environment (with some considerations)
• Codify all failures and include cause/action in the response
– Use right http status code
– Add application level code for codifying issues
– Add human understandable message
There are always unforeseen corner cases!

21. More on Request/Response
• Response status codes
– There are more than 200, 404 and 500!
• Unified response style
– Predictability of response body
– Application specific error code
– Human readable message
– cause/solution
• debugability and traceability
– debugified response
– ECID
– Unified Logging (Facebook Scribe, Google Dapper, Apache Kafka)
How should a resource respond to a request?

22. A little bit of security
Authentication
Access control
Auditing

23. Authentication
• Authentication enabled for all resources
• Happens before any other validation
• Exclude the resource patterns that requires no authentication
• No access without validating the authentication token
Know who is requesting a resource

24. Access control
• Happens after detecting a valid authentication
• Requests are easy to check
– Unless column level checks are required, can be done in simple filter
– Column level access control can be done using media types
• Happens before any validation
– unless params are being used as part of the access check
• JAX-RS 2.1 is going to support declarative security
Check who can access a resource

25. Auditing/ Access logs
• Keep a rich access log
– Depending on the server you use
• Include the usual who, when, what
• Try using W3C Extended Log File Format if supported by server
• Configure security realm for logging
• Think ahead about incident detection/isolation, etc.
Keep record of security incidents!

26. Performance matters!
Caching

27. Problem: Don’t hit the servers too often
• Don’t reproduce the same response twice
• Don’t use bandwidth whenever possible
• Counter network reliability issues
• Counter server downtimes
Being stingy with the resource usage is OK!

28. Solution: Caching
• Local Cache
• Proxy Cache
• reverse-proxy (cache)
• Server(Application) Level Cache
Being stingy with the resource usage is OK!

29. Caching
• Use HTTP caching features
• Client aware of provided caching support
• Server evaluate caching related headers
• intermediately hops
• Types of Caching Headers
– Absolute Caching Headers
– Conditional Caching Headers
Application Level Cache

30. Caching
• From Server side:
– Cache-Control and directives
– Last-Modified
Absolute Caching
HTTP/1.1 200 OK Content-Type: application/json
Cache-Control: private, max-age=86400
Last-Modified: Mon, 08 Sep 2014 16:30:00 GMT
• private
• public
• no-cache
• no-store
• max-age (overrides Expires)
Cache Control Directives
curl -v -X GET -H "If-Modified-Since:Mon, 08 Sep 2014 15:08:27 GMT" http://.../simple/1

31. Caching
• From client side send headers:
– If-None-Match = "If-None-Match" ":" ( "*" | 1#entity-tag )
• At the server side produce headers:
– Etag, when last-modified is hard to determine or not accurate enough
Conditional

32. Caching
• Can be used for conflict resolution
• Cache on GET request
• Invalidate cache on PUT, POST or DELETE
• Periodically purge cache entries
• Cache invalidation, eviction is not deterministic
• Give http://www.jboss.org/resteasy a try
– Provides @Cache and @NoCache
– Extension to JAX-RS
More on Caching

33. Performance matters!
Partial Updates & HTTP PATCH

34. Problem: Wasting processing and bandwidth when
updating
• Updates are not usually full representation mutation
• No need to process the whole representation to update one attribute
• No need to re-persist the whole representation on change
Only update what needs to be updated!

35. Solution: Partial Updates & HTTP PATCH
• Partial Updates with PUT
• Partial Updates with POST
• Partial updates with PATCH
• JSON Patch is the future
Only update what needs to be updated!

36. Partial Updates & HTTP PATCH
• Partially update a JSON document
• Works with HTTP PATCH
• Requires special body syntax and directives
JavaScript Object Notation (JSON) Patch, RFC 6902
PATCH /coffee/orders/1234 HTTP/1.1
Host: api.foo.com
Content-Length: 100
Content-Type: application/json-patch
[
{“op”:"replace", ”path”: "/status", "value": "COMPLETED"}
]

37. Partial Updates & HTTP PATCH
• Supports six operations in the payload
– op : can be “add”, “replace”, “move”, “remove”, “copy” or “test”
• Another three attributes to describe the op
– path: Location of the target attribute in the JSON document
– value: The new value to be added or to replace another
– from: (Only for move op) specifies the source location
JavaScript Object Notation (JSON) Patch, RFC 6902

38. Performance matters!
Asynchronous And long running jobs in REST

39. Problem: Serving more requests on the serverside
• Fixed server side resources
• Heavyweight container threads
• Not keeping container resource longer than needed
Don’t keep unnecessary resources for where not needed!

40. Solution: Asynchronous And long running jobs in REST
• On the serverside:
– @Asynchronous: Annotate a sub-resource as Asynchronous
– AsyncResponse: Provides results and actions on the running request
• setting timeout
• registering callbacks
• resume, cancel suspended request processing
• updating the response
– @Suspended: To inject a suspended AsyncResponse into a sub-resource parameter
Don’t keep unnecessary resources for where not needed!

41. Asynchronous And long running jobs in REST
• On the serverside:
– CompletionCallback: Async response processing completion callback
• Response processing completed and sent to client
• Response processing failed with exception
– ConnectionCallback: Client server connection events callback
• Client is disconnected abruptly (before or during writing back the response)
Server side callbacks

42. Asynchronous And long running jobs in REST
@GET @Produce(“application/json”)
@Asynchronous
public void getOrder(@Suspended AsyncResponse ar, String orderId) {
final String result = prepareResponse(orderId);
ar.resume(result)
}
Some small sample code
Future<Coffee> future = client.target(“/coffees/orderId")
.request()
.async()
.get(Coffee.class);
try {
Coffee coffee = future.get(30, TimeUnit.SECONDS);
} catch (TimeoutException ex) {
//
}
Client Code:
Server Code:
** Alternative to Future is using InvocationCallback to get called when response is back

43. Few more topics
HATEOAS

44. Problem: Being perfect according to REST maturity model
– Letting the resource graph describe itself
– Describing the resource graph by name
– Removing dependencies on fixed resource addresses
– Letting clients have easier understanding of the API
How to solve:
{
"order": "120"
"links": [
{"rel": "self", "href":"http://api.dom.com/orders/120"},
{"rel": "cancel", "method":"delete", "href":"http://api.dom.com/orders/120"},
{"rel": "shippingDetails", method: "get", "href":"http://api.dom.com/orders/120/shipping"}
]
}

45. RESTful design maturity levels (Richardson Maturity Model)
And the maturity model
Level 0: XML RPC/Remoting – One resource and one verb (POST) for everything!
Level 1: Multiple resources, not correct use of verbs
Level 2- Proper use of HTTP verbs
Level 3- HATEOAS – Resource graph
discovery
MaturityGrowth

46. HATEOAS
• REST maturity model
– The glory comes when you are at HATEOAS level!
• Why to` use HATEOAS
– Describe the resource graph by name
– Remove dependencies on fixed resource addresses
– Let the resource graph describe itself
– Let clients have easier understanding of the API
• When to use HATEOAS
– When one resource has some dependent resources
– What is allowed and what is not (logic)
– What is accessible to a user and what is not (security)
Let the resource graph describe itself

47. JAX-RS and HATEOAS
• No standard API so far but the following to the rescue
– Link
– UriInfo
– UriBuilder
• JAX-RS extensions and Jeresy specific APIs
– Use https://code.google.com/p/jax-rs-hateoas/ extension
– Use Jeresy specific API
• Use Spring MVC
• Use RESTeasy
• It is not end of the world if you don’t!
Java/EE HATEOAS support out of the box

48. Few more topics
Localization/Internationalization

49. Localizations and Internationalization
• One time used messages (not persisted for retrieval)
– Decide the locale using accept-language header
– Produce the right locale representation
• Messages would be used by multiple times(persisted messages)
– Codify every text and set of argument
– Let a last layer filter decide how to process the message code and formatting params
– Let the last layer filter decide what to be the date format and , currency should be
– Let the last layer filter rewrite the response replacing the templates
– Keep every message in a memory cache
Different locale getting the right format and style of messages

50. Few more topics
Usage Throttling

51. Problem: Abusing the API, categorizing the users
– Better resource allocation and management
– Prevent and handle abuse
– Provide better support for premium users
Keep tap on how many request one user can send!

52. Solution: Resource Throttling
• Why use Rate Limiting?
– Prevent and handle abuse
– Provide better support for premium users
• How Rate Limiting works?
– Servlet filter sits in front
– User user tokens or IP addresses as identifiers
– Use in-memory cache for book keeping
– Prevents requests reaching endpoints when limit reached
– Reset/update cached counters when needed
Keep tap on how many request one user can send!

53. Throttling Patterns
• Response status, headers
– HTTP 429 Too Many Requests error code
– Retry-After header
– X-RateLimit-Limit: ###
– X-RateLimit-Remaining: ###
– X-RateLimit-Reset: EPOCH_SECONDS
• Descriptive response in the requested media type
Headers and response

54. Throttling Patterns
• Client side
– Use caching
– keep tap on number of requests
– Pay attention to headers
– No brainless loops (polling)
• Server side
– Support caching (etags and max-age)
– provide streaming endpoints when possible (feeds, news, public data)
Best practices

55. Few more topics
REST and plug-ability and extensibility

56. MicroServices…
• Same good old modularity with with distributed interfaces
• Plus with communication over HTTP, mostly!
• Plus communication using message passing (services, events, event bus)
• Plus with independent SLDC
• Plus being on different platforms/use different data-store/etc.
– Whatever tools, language, framework that fits better!
How far?

57. MicroServices
• Advantages
– Simplicity of development and maintenance
– Isolation of requirements and fulfillments
– Scale-up and Scale-down, scale-out and scale-in
– More focused HR managements
– Etc.
• Disadvantages
– Operations Overhead
– High level of DevOps, RE Skills Required
– Complexities of a Distributed System
– Etc.
Pros and Cons

58. Comments, Questions?

59. • http://tools.ietf.org/html/rfc6902
• http://tools.ietf.org/html/rfc6901
• http://resteasy.jboss.org/
• https://jersey.java.net/documentation/latest/
• http://tools.ietf.org/html/rfc6585
• http://tools.ietf.org/html/rfc5789
• http://martinfowler.com/articles/richardsonMaturityModel.html
• http://www.slideshare.net/bhaktiks/real-world-restful-service-development-problems-and-
solutions
• CCL photos used in slides:
• https://www.flickr.com/photos/treehouse1977/2892417805/
• https://www.flickr.com/photos/treehouse1977/2892417805/
• https://www.flickr.com/photos/essjay/165928100/
• https://www.flickr.com/photos/jforth/4413370462/
• https://www.flickr.com/photos/sakalak/8737872379/
• https://www.flickr.com/photos/jbparrott/8980026600
• https://www.flickr.com/photos/pentadact/36593493/
• https://www.flickr.com/photos/jasohill/4442279347/
• https://www.flickr.com/photos/mdsharpe/5075953655
• https://www.flickr.com/photos/chuqvr/8329512894/
• https://www.flickr.com/photos/longo/2684733921
• https://www.flickr.com/photos/_davor/14757399908
Resources