Recommended
More Related Content
Viewers also liked
Viewers also liked (14)
Similar to Building big things in Java
Similar to Building big things in Java (18)
Recently uploaded
Recently uploaded (20)
Building big things in Java
- 1. Building big things in Java Mark Pope and Tom Coupland
- 2. Nokia Entertainment Text
- 3. Search & Buy Mix Radio
- 4. Internet Entertainment API Search Library Radio Gigs Delivery
- 6. Entertainment API Search Solr Solr Solr
- 7. Entertainment API Search Solr Solr Solr
- 8. Search • 200 requests per second • 300ms response time
- 9. Entertainment API EAPI Search Solr Solr Solr
- 10. Entertainment API Search Solr Solr Solr
- 11. Solr • 3 servers (96GB RAM, 12 cores) • 420 requests per second • 15ms response time • 25 million documents
- 12. Entertainment API Search Solr Solr Solr
- 13. You found it! Now what? http://news.bbc.co.uk/sport2/hi/olympics/7564645.stm
- 16. Entertainment API Library Sharding a- e- k- p- t- v- d j o s u z
- 17. Entertainment API Library Sharding a- e- k- p- t- v- d j o s u z
- 18. Library • 1,000,000,000 rows • 800,000 requests per day • 300ms response time • 26 servers
- 19. Half way
- 20. Mix Radio http://alyssahsadie.deviantart.com/art/Old-School- BoomBox-24949091
- 21. Entertainment API Personalisation Catalogue db
- 22. Entertainment API Personalisation Catalogue db
- 23. Entertainment API • 35 developers, 6 teams, 2 countries • 19000 lines of code • 155 deployments in 2011 • 160 deployments in 2012
- 24. Show me the music!
- 25. Delivery LOUD NOISES! http://pursuitbikes.com/blog/?p=9279
- 26. Entertainment API Delivery NAS Packaging
- 27. Entertainment API Delivery NAS Packaging
- 28. Delivery • 1 petabyte of music and books • 16 million tracks • 6 formats • 1 million tracks served per day
- 29. So... That’s how we built (part of) a big thing in Java
- 30. Questions?
- 31. Thanks! • Mark Pope • twitter.com/ tcoupland • twitter.com/scobal • github.com/ • github.com/scobal mantree • Tom Coupland
Editor's Notes
- Welcomes. Thanks for coming. We’re Tom and Mark. From Nokia. \nHere to talk about \ndeliver a global music service\nhow we built it with Java\n(Do we add in a story throughout to keep the ideas concrete)\n
- Located in central Bristol \nDesign, build and maintain the service (pretty much exclusively)\nProvide music to millions of users around the world\nScaled/Served to around 1million downloads a day\nJava, windows phone, web\n
- Talk through parts of our service\nTwo use cases:\nSearch and adding to Library (Find and own)\nMix radio and Delivery (New product called mix radio)\nFirst going to talk about high level architecture...\n
- Multiple client platforms (win phone, web, symbian)\nSOA 3 years ago\nStruggling with single monolith, lots of clashing\nTeams work on groups of services\nBreak a use case down into a set of responsibilities. Service for each one.\nScale single components\nEAPI provides \nauth\nproxy for routing\ncommon api\nJSON over HTTP with REST\nTalk more about it later....\n
- Think about how you buy music from a store.\nHow you might model the process in software.\nSo, we ended up with something a bit like like:\n\n\n
- EAPI routes requests to Search service\n\n\n
- So we’re now going to talk about Search\n
- Why do we need this layer? Stateless. Built for high throughput.\nReturns track metadata, artist information, track names.\nLayer of abstraction above solr\nTalk internal language\nJava, Spring (Open Source, tool for building java software [IOC {breakout acronyms}, DI]), REST API over HTTP\nTwo servers for failover - key concept when building big things\n
- \n
- \n
- What is solr? Solr is an Apache Open Source inverted full text indexer. So...\nWhy use solr instead of SQL?\nPut in documents containing artist and song information, get broken down into individual words, then indexes.\n\nsearch for “stone” -> Stone Roses, Joss Stone, Rolling Stones.\n\n3 servers: failover and scale requests\n
- So, this how our search architecture works.\nSo....\n
- You’ve found what you’re looking for. \nWhat’s the next step?\n
- Well, you have to give us some money.\nWe’re not going to concentrate on this\n
- We have to know what you own.\nPersonal collection of music/books\nHow to model this in software (when you have to store millions of users collections)\n(Collections can be ma-hou-sive)\nClients use this to obtain what a user owns, so you can see what you own on the clients\n
- The familiar clients and api layer.\nRoutes to Library sub-system.\n\n
- The familiar clients and api layer.\nRoutes to Library sub-system.\n\n
- Why do we need to shard? Why not one big database (horizontal vs vertical [expensive] scaling) \nClone, copy, master-master\nEach shard replicated twice - failover and redundancy when building big things. Hot failover.\nThin service runs on 8 servers, 12 db servers, 6 sharding servers\nSpring and hibernate, Open Source. Open Source MySql.\n
- Not really an intermission. We’re just half way through our slides.\n
- Mix radio - product on Nokia Lumia phones for streaming free music.\nHand crafted - created by experts to suit mood and taste.\nMobile optimised - codec with great sound and small file size\nHow might you do this with software?\n\n
- Back to our old friend EAPI.\nCan see the two paths - not going to talk about personalisation today\nLooking for curated radio stations\nThese are stored in a service called Catalogue\n
- Catalogue is a big list of all the music and books we offer. Metadata store.\nAlso stores curated radio stations\nEAPI retrieves the radio station you selected (which the client will then play)\nHowever, take a quick look at this EAPI thing. What is it\n
- Conceptually a thin layer - actually a bit more to it (because of it’s centralness)\nClient agnostic. Stateless. \nContinuous change, integration and delivery\nHighly modular allowing collaboration reducing checkin conflicts (with git).\nPairing, TDD.\nDanny and Neil talking in more detail later about how we work\n
- Ok\n
- How do we make noise come out of your phone\n
- Sub-systems that make noise come out of your phone\nResponsible for \nmaking sure you’re licensed to download your selection\nactually serving up the location of the music or book\nadding images and meta-data to your download\nNAS for storing music + books - massive\n\n
- So how big is this thing?\n
- Mention VCDN. What is a (Content Delivery Network) CDN?\nContent Delivery Network, a big cache out in the cloud.\nPowers ‘most’ of the internet, act as caches for content.\nLocated all over the world, reduce load.\nGreat Firewall of China.\nStores our eaac32 formatted tracks.\n
- \n
- \n
- \n