More Related Content
Viewers also liked
Viewers also liked (14)
Similar to Automatic Discovery of Service Metadata for Systems at Scale
Similar to Automatic Discovery of Service Metadata for Systems at Scale (20)
Recently uploaded
Recently uploaded (20)
Automatic Discovery of Service Metadata for Systems at Scale
- 1. Automatic Discovery of Service Metadata Martina Iglesias martina@spotify.com September 2016
- 3. Spotify Users 100M Active Users 40M Paid Subscribers 59 Countries
- 4. Spotify tech ● +800 tech employees ● 120 teams ● Microservices architecture (scale and work independently)
- 6. Artist page Number of streams
- 10. Example: aggregating service Artist Discography Playcount Playlists Merch
- 11. Spotify infrastructure ● +1000 services ● The number of services grows as we add new features
- 13. Previous situation ● Each team had doc. in different places ● README.mk ● Markdown files in doc/ ● Wiki ● Link to a document somewhere else
- 15. System Z
- 16. System Z ● Web application ● Internal tool ● Catalogue of all our systems and its parts ● Very well integrated with our apollo services ● Easy to discover and access
- 20. System Map ● Generated from runtime and declared dependencies ● Uses graphviz
- 21. DNS System Z overview Register artist service ARTIST SERVICE: DNS: _artist._http OWNER: martina SYSTEM Z (backend) artist-1.spotify.com Lookup machines HTTP req to /m eta/0/
- 23. Apollo ● Java libraries for writing microservices ● Open Source ● https://github.com/spotify/apollo ● In production since 2014
- 24. APOLLO SERVICE MODULES OKHTTP-CLIENTJETTY-HTTP-SERVER APOLLO-CORE (manages lifecycle) APOLLO-API APOLLO-META SERVICE LOGIC interacts uses Apollo service overview
- 25. Apollo-meta ● Metadata module ● Open Source ● https://github.com/spotify/apollo-meta ● Exposes endpoints with metadata of the service ● Runtime generated - source of truth
- 26. Example: Creating a route Route.async("GET", "/v1/artist/<id>", request-> getArtist(request)) .withDocString("Get the artist page for a specific id.") )
- 27. Endpoints exposed by apollo meta 1. Instance information 2. Configuration 3. Endpoints 4. Call Information
- 28. 1. Apollo-meta: instance info ● Collects information about the service: build version and uptime ● Useful to get the full picture when making rolling upgrades.
- 29. 1. Apollo-meta: instance info curl http://artist-a1/_meta/0/info => { "result": { "buildVersion": "artist 2.1-SNAPSHOT", "containerVersion": "apollo-standalone 1.1.0", "systemVersion": "java 1.8.0_60", "serviceUptime": 303577.347 } }
- 30. 2. Apollo-meta: configuration ● The current loaded config of the service, possibly filtered
- 31. 2. Apollo-meta: config curl http://artist-a1/_meta/0/config => { result: { http: { server: { port: 8080 } } } }
- 32. 3. Apollo-meta: endpoints ● Lists the endpoints of the service
- 33. 3. Apollo-meta: endpoints curl http://artist-a1/_meta/0/endpoints => { methodName: "/v1/artist/<id>[GET]", uri: "/v1/artist/<id>" method: [ "GET" ], docstring: "Get the artist page for a specific id.", queryParameters: [] },...
- 34. 4. Apollo-meta: call info ● Lists services that make incoming requests ● Lists all other services we make requests to
- 35. 4. Apollo-meta: call info curl http://artist-a1/_meta/0/calls => incoming: { loadbalancer: { endpoints: [ uri: "/v1/artist/<id>", method: ["GET"], queryParameters: [catalogue, locale] ], }, },..
- 36. 4. Apollo-meta: call info ... outgoing: { discography: [], playcount: [], playlist: [], merch: [] } } }
- 37. Apollo meta <-> System Z ● System Z calls these endpoints ● Displays a merged version of all the data
- 38. Conclusions
- 39. ● Quicker access to relevant information ● Know immediately where to go when solving an incident ● Less interruptions ● Less boring work Situation now
- 40. ● Think about growth and scaling ● Automate all boring tasks that you can ● Put all the information related in one, easy to access, place ● All related links in one place Learnings
- 41. ● apollo ● swagger.io ● raml.org (jax-rs) Documentation generators
- 42. Blog: labs.spotify.com Jobs: spotify.com/jobs Martina Iglesias martina@spotify.com
- 43. Blog: labs.spotify.com Jobs: spotify.com/jobs Martina Iglesias martina@spotify.com Questions?