Downloaded 17 times
Uploaded byCloud Elements
All Things API Presentation - Gordon Weakleim [HomeAway]
This document summarizes the evolution of HomeAway's APIs from multiple acquired companies with different architectures into a unified REST API. It describes how the API was designed with REST constraints in mind to provide a consistent interface. It also discusses how operational concerns led to additional layers being added for caching, authentication, analytics and more. The API framework approach standardized URLs, media types, and provided tooling to support over 250 developers building and maintaining the APIs.
More Related Content
Similar to All Things API Presentation - Gordon Weakleim [HomeAway]
All Things API Presentation - Gordon Weakleim [HomeAway]
- 1. HomeAway’s Core API Orhow I learned to stop worrying and love the abstraction layer…
- 2. My History Gordon Weakliem Professionaldev for 21 years Mostly in the travel industry CORBA, COM, SOAP, REST… I started with VRBO.com in February, 2007.
- 3. HomeAway’s History Founded in2005 Went public in June 2011 (NASDAQ:AWAY) 20+ Acquisitions Mostly full-stack “listing sites” Diverse tech stacks and teams Old code bases World-wide dev teams Complex Marketplace + Products
- 4. VRBO’s History Startedin 1995 advertising our founder’s ski condo. By 2006, we were a mix of ASP and ASP.NET in a roughly 3 tier architecture talking to SQL Server. At the time of the HomeAway acquisition in 2006, there were 4 full time developers. This grew to a manager, 8 dev + 2 QA in the next couple years. VRBO had 65,000 active listings and was growing 100% YOY.
- 5. Growth through acquisition We’rea long way from advertising a ski condo from a basement server. How do you integrate all these companies?
- 6. Platform Evolution Pub/Sub –initial consolidation strategy Solves distribution – enabled growth Evolution to platform Apps need single (logical) SoR per resource type Apps need a clean abstraction Must interop with 200+ existing apps
- 7. Growing Pains Duringthe pub-sub era: The architecture was baroque. It was really painful to figure out why things went wrong. Listing sites had to implement missing infrastructure themselves – administration, caching. Feature set wasn’t unified across brands. There were some shared services (inquiries, reviews) but each one implemented their own provisioning, interface standards, and API conventions.
- 8. HomeAway’s Core API RESTAPI Unified Architecture XML/JSON over HTTP (polyglot access) Many teams contributing Façade over existing systems Java/Spring/Jersey/Mybatis/MS SQL
- 9. REST API! Why? HTTP methods– standard verbs Resources – standardize nouns URLs – standardize references Layered – enterprise features globally Focused on interface/contract, not on implementation
- 10. api.homeaway.com API Architecture API endpointAPI endpoint API endpoint Legacy app Legacy svc Legacy svc Legacy app app app app cache oauth versioning etc… Access routed through a single hostname at the load balancer Endpoints service a set of nouns, routed by URL path Enterprise features layered in across the API API endpoints coexist with legacy apps and services
- 11. The “Grand Vision”of Platform We designed the API we wanted but couldn’t build (right away) We built it anyways As a façade to the existing systems New App development benefits Minimized Risk for Legacy Sunsets
- 12. API -> PlatformRefactoring API Application Application Application
- 13. REST – Constraints(a refresher) Client-Server Stateless Cacheable Layers Uniform Interface!! Urls Manipulation mime-types HATEOAS Code on Demand
- 14. REST - Representation Urls /{entityCollection}/{systemCode}/{entityUuid} ResourceOriented Noun centric (/listings, /accounts)
- 15. REST - Representation HATEOAS/ Strongly Linked
- 16. REST - Representation Resourcerepresentations are XML / JSON Defined by an XSD .NET, ruby, python, javascript bindings Declarative Validation
- 17. REST - Representation Versioning X-HomeAway-ApiVersion:1.81 Version “ratchet” via bi-directional XSL chain
- 18.
- 19. REST – StateTransfer Etags Versioning Etag: “34310538” Etags Optimistic Concurrency If-Match: “34310538”
- 20. REST – StateTransfer Caching Conditional GETs If-None-Match: “34310538” Expires: Thu, 29 Aug 2013 04:45:54 GMT Last-Modified: Wed, 28 Aug 2013 18:00:00 GMT Cache-Control: max-age=60, must-revalidate
- 21. Authentication OAuth2-based Authentication Authorization /ACL Self-serve provisioning for credentials Access to certain features requires a review Implemented as a Filter (layered!)
- 22. api-framework Large company 250+ developers APIsexisted before our “grand vision” We could not have got this far without a good foundation.
- 23. api-framework Rails, Sinatra, Django,web.py, JAX-RS are general purpose web frameworks Teams develop unique philosophies Organization URL Structure Media
- 24. api-framework “There should beone – and preferably only one – obvious way to do it.” Consistency API developers can easily work with(and on) any API endpoint
- 25. URL Structure Entity Collection SystemCode Entity UUID SubResource
- 26. URL Structure /listings /listings/0000 /listings/0000/af6ffb31-ec1b-47b4-8f55-9cc33d0341dd /listings/0000/af6ffb31-ec1b-47b4-8f55- 9cc33d0341dd/hitCounter /listings/legacy?id=vrbo/980 -> an entity
- 27. URL Structure Opaque –“thou shalt not reason about a URL’s structure” Provides an obvious pattern of organization Confined vocabulary directly related to features in the framework
- 28. Interface Contracts There’s afundamental client/server mismatch Clients reason about APIs at the HTTP level Developers are deeply nested in implementation
- 29. You could justscrape request vars…
- 30. But it’s betterto define an interface
- 31. Mismatch Easy to maketo assumptions that are not shared with clients Leads to backwards incompatibilities
- 32. Design By Contract ForceAPI developers to think about the “contract of invocation” Preconditions / Post conditions Fine grained requirements Statically declared contracts Developers do not enforce contracts
- 33. Design By Contract Contracttightly coupled with the interface Change is not “free” Forces developers to think about backwards compatibility
- 34.
- 35. Humans like thisform…
- 36. HomeAway API Stats 8,266,885,627calls week ending 2015-03- 09 From 264,570,522 week ending 2013-03-04 99.3% of all calls < 250ms 100+ internal clients Monitoring for errors/SLA across API
- 37.
- 38. Operational Concerns Werely on this API Outages are a big deal We need to be able to develop new features quickly A lot of people are working on this It can’t be a free-for-all REST isn’t the end-all. Sometimes it’s inefficient Polling Following links
- 39. Filters HomeAway ops filter JerseyServlet Caching Versioning OAuth Auditing Auditing Versioning Caching api-framework code code app Endpoint developers code java methods here App request/respons e goes through many layers of functionality
- 40. Analytics We useSplunk, heavily Framework log format is the same for everyone. Splunk queries work for all core services. HomeAway has an entire Big Data team. Big Data is kind of a big deal at HomeAway
- 41. Caching We handlea LOT of traffic Connecting consumer endpoints to core API is operationally challenging. In-memory caching on each box Elastic Search to support distributed caching Developers must enable caching per endpoint
- 42. Eventing (AMQP/RabbitMQ) POST/PUT/DELETEsare events API Endpoints publish events to AMQP topic Consumers bind Queues to the topic
- 43. Entity Search Service Before, developers wrote /findByXXX methods to support search functions ESS – based on Elastic Search Dev annotates indexable Indexer job automatically builds indexes Index exposed under /search No free lunch – we have to maintain strict policies around adding new indexes
- 44. Search Examples https://api.homeaway.com/reservations/findByEmail?email =gweakliem@yahoo.com https://api.homeaway.com/reservations/search?q=contact.em ail=%22gweakliem@yahoo.com%22 https://api.homeaway.com/reservations/findByDate?url=/dr uids/0060/ef151f3b-ad64-423d-9102- 10fccac084fa&start=2015-01-01&end=2015-03-31 https://api- stage.homeaway.com/reservations/search?q=reservationDates .beginDate=2015-01- 01%20AND%20reservationDates.endDate=2015-03- 31%20AND%20propertyProfile.druid.rel=%22/druids/0060/ef151f3 b-ad64-423d-9102-10fccac084fa%22
- 45. Tool Support Wehave a dedicated Developer Tools team Manages things like github, Team City, Jira, deployments End to end support, from starting the project to deploying to production. Central repository for application registration Developers can self-serve deployment all the way to production.
- 46. BizOps Our platformneeds are evolving BizOps are the next level above the domain API Intended an an intermediate integration point for services that need to combine data from several core API entities. JSON is the default format because the primary client is an AJAX client – these are consumed by our consumer-facing sites.
- 47. Summary Domain-specific framework provides aconsistent organizational pattern Layers permit adding functionality without changing code in multiple projects
- 48. Summary First-class contracts reducerisk of backwards incompatibility Derive documentation from contracts Dedicated Tools & Data teams Operational concerns drive certain compromises
- 49. Thanks! Questions? Emailme: gordon@vrbo.com
Editor's Notes
- #10 I’m just going to give a very high-level overview here – René will go more into detail…
- #12 Add a visual showing consolidation
- #14 REST – 6 constraints Client-Server, Stateless, Cacheable, Layered System, Code on demand (optional), Uniform Interface (Id of resources, manipulation of resources through these representations, self-descriptive messages, HATEAOS)
- #26 It’s important to note that URL structure is not necessarily a simplification for clients. Roy Fielding notes this explicitly “A REST API must not define fixed resource names or hierarchies
- #28 It’s important to note that URL structure is not necessarily a simplification for clients. Roy Fielding notes this explicitly “A REST API must not define fixed resource names or hierarchies
- #37 splunk, logging, resource SLA