Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Spilgames Storage Platform

496 views

Published on

A presentation of the Spilgames Storage Platform, a massively scalable solution to abstract your storage with transparent sharding and expanding your datacenters worldwide.

Published in: Technology, Business
  • Be the first to comment

  • Be the first to like this

Spilgames Storage Platform

  1. 1. IntroductionSpilgames & MeAnotherStorageMotivation Spilgames Storage PlatformSystemPropertiesOverviewInternalsVersions & Shards Enrique PazWorldwideScalingDatacenters Senior Backend DeveloperDisasterScenariosLessonsLearned 21/03/2013Currents StatusContributions 1/42
  2. 2. IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & Shards IntroductionWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 2/42
  3. 3. About MeIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & Shards • Passionate Erlang developerWorldwideScaling • Testing enthusiastDatacentersDisasterScenarios • Love beautiful code!LessonsLearnedCurrents StatusContributions 3/42
  4. 4. SpilgamesIntroductionSpilgames & MeAnotherStorageMotivation • Gaming PlatformSystemProperties • Serving data to 190+ countries world-wideOverviewInternalsVersions & Shards • 200+ million unique users per monthWorldwide • Multiple Platforms: Desktop, Mobile Native & WebScalingDatacentersDisaster • 300+ employeesScenariosLessons • Offices in The Netherlands & ChinaLearnedCurrents Status • Revenue: Advertising & EUMContributions 4/42
  5. 5. Gaming PortalsIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 5/42
  6. 6. Gaming PortalsIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 6/42
  7. 7. GamesIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 7/42
  8. 8. IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & Shards Another StorageWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 8/42
  9. 9. Another StorageIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 9/42
  10. 10. LAMP Stack & MySqlIntroductionSpilgames & MeAnotherStorageMotivationSystemProperties • Not all developers are DB expertsOverviewInternals • Difficult to shard the databasesVersions & ShardsWorldwide • Storage model all over the placeScalingDatacentersDisaster • SecurityScenariosLessons • PerformanceLearnedCurrents Status • CachingContributions • ... 10/42
  11. 11. Our AmbitionIntroductionSpilgames & MeAnotherStorageMotivation • Transparent sharding layerSystemPropertiesOverview • Sharding on data ownershipInternalsVersions & Shards • High availabilityWorldwideScaling • Centralized caching layerDatacentersDisasterScenarios • Storage engine agnosticLessons • One strict data modelLearnedCurrents StatusContributions • Transparent storage changes • Scaling geographically 11/42
  12. 12. IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & Shards System PropertiesWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 12/42
  13. 13. MindsetIntroductionSpilgames & MeAnotherStorageMotivationSystem • Be always availablePropertiesOverview • Avoid global locksInternalsVersions & Shards • Accept change as the only constantWorldwideScaling • Embrace inconsistenciesDatacentersDisaster Hardware breaks down (power failures)Scenarios Version mismatches (upgrading system not atomic)LessonsLearned State mismatches (adding new machines)Currents StatusContributions 13/42
  14. 14. A Key-Value Store With SchemaIntroductionSpilgames & Me • BucketsAnotherStorage Largely generated OTP applicationsMotivation Offer a CRUD-like interface (with filters)SystemPropertiesOverview • GIDs (64 bits) identify the data ownerInternalsVersions & Shards userWorldwide gameScalingDatacenters • Buckets can use different storage enginesDisasterScenarios Several MySql tables in different databasesLessons Just a binary storage (SWIFT)LearnedCurrents Status ...Contributions • Data for a bucket/GID is cached • Requests can be atomic per bucket/GID 14/42
  15. 15. Optimistic OperationsIntroductionSpilgames & MeAnotherStorageMotivationSystem • Speed > ConsistencyPropertiesOverviewInternals • Losing some updates in case of crash is affordableVersions & Shards • Act first on cache and then on diskWorldwideScalingDatacenters • No warranties of eventual consistency upon crashesDisasterScenarios i.e. Activity feedsLessons i.e. Popular games listLearnedCurrents StatusContributions 15/42
  16. 16. Pessimistic OperationsIntroductionSpilgames & MeAnotherStorageMotivationSystemProperties • Consistency is key and confirmation is requiredOverviewInternalsVersions & Shards • Dealing with critical dataWorldwide • Persist data and, upon success, update cacheScalingDatacenters i.e. PaymentsDisasterScenarios i.e. Personal informationLessonsLearnedCurrents StatusContributions 16/42
  17. 17. How does it work?IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 17/42
  18. 18. System ComponentsIntroductionSpilgames & MeAnotherStorageMotivationSystem • lookup application in all nodes.PropertiesOverview Uses a hashing ring (mnesia):InternalsVersions & Shards replicated in all nodesWorldwide ram_copiesScalingDatacenters dirty readsDisasterScenarios transactional writesLessonsLearned • Buckets have Pipeline FactoriesCurrents StatusContributions • Buckets register PFs in lookup 18/42
  19. 19. Tracing Pessimistic OperationsIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverview 1. Bucket/GID request in a nodeInternalsVersions & Shards 2. Local lookup to find a PFWorldwideScaling 3. PF receives requestDatacentersDisasterScenarios 4. PF builds jobLessonsLearned 5. PF ensures Pipeline for GIDCurrents StatusContributions 6. PF queues operation in Pipeline 19/42
  20. 20. Wait a minute. . .IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternals • “Why do we need pipelines?”Versions & ShardsWorldwide • “Sequential == Bottleneck !!!”ScalingDatacenters • “Don’t you guys know Erlang isDisasterScenarios about parallelizing work?”LessonsLearnedCurrents StatusContributions 20/42
  21. 21. About PipelinesIntroductionSpilgames & MeAnotherStorageMotivation • CONSSystemProperties Sequential (read) access for hotspotsOverviewInternals i.e. Popular gamesVersions & Shards Optimization: read from SSP cache in PFWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 21/42
  22. 22. About PipelinesIntroductionSpilgames & MeAnotherStorageMotivation • CONSSystemProperties Sequential (read) access for hotspotsOverviewInternals i.e. Popular gamesVersions & Shards Optimization: read from SSP cache in PFWorldwideScaling • PROSDatacentersDisaster No need for storage engines to support global locksScenarios A bucket can combine several enginesLessonsLearnedCurrents StatusContributions 21/42
  23. 23. About PipelinesIntroductionSpilgames & MeAnotherStorageMotivation • CONSSystemProperties Sequential (read) access for hotspotsOverviewInternals i.e. Popular gamesVersions & Shards Optimization: read from SSP cache in PFWorldwideScaling • PROSDatacentersDisaster No need for storage engines to support global locksScenarios A bucket can combine several enginesLessonsLearnedCurrents Status • Requests to most GIDs (users) are evenly distributedContributions 21/42
  24. 24. Schema VersionsIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverview • Schema Versions determine allowed operations and storage(s)InternalsVersions & Shards • Client is not aware of themWorldwideScaling • Max 2 schema versions of a bucket at the timeDatacentersDisasterScenarios • Schema version can be changed at bucket/GID levelLessonsLearnedCurrents StatusContributions 22/42
  25. 25. ShardsIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverview • Useful for partitioning big blocks of dataInternalsVersions & Shards • Shard points to the physical location of the dataWorldwideScaling • Sharding rules are bucket specific. Default is GID % ShardsDatacentersDisasterScenarios • bucket/GID combinations can be migrated between shardsLessonsLearnedCurrents StatusContributions 23/42
  26. 26. Working With Versions & ShardsIntroductionSpilgames & MeAnotherStorageMotivation 1. insert(Bucket, Gid, Value)SystemProperties 2. insert(Gid, Value)OverviewInternalsVersions & Shards 3. get_vs(Bucket, Gid)Worldwide 4. {v2, Shard1}ScalingDatacentersDisaster 5. build_job(insert, Gid, Shard1)ScenariosLessons 6. {ok, InsertJob}LearnedCurrents Status 7. {ok, InsertJob}Contributions 24/42
  27. 27. One API To Rule Them AllIntroductionSpilgames & MeAnotherStorage • “Don’t care where it is, just want my data!!!”MotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 25/42
  28. 28. One API To Rule Them AllIntroductionSpilgames & MeAnotherStorage • “Don’t care where it is, just want my data!!!”MotivationSystem • PIQI helps with the APIPropertiesOverviewInternalsVersions & ShardsWorldwideScaling Erlang client + Protocol BuffersDatacentersDisasterScenarios HTTP + JSONLessonsLearned HTTP + Protocol BuffersCurrents StatusContributions 25/42
  29. 29. IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & Shards WorldwideWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 26/42
  30. 30. WorlwideIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 27/42
  31. 31. Masters & SatellitesIntroductionSpilgames & MeAnotherStorageMotivationSystem • Master DatacentersProperties Have persistent storageOverviewInternals Can own GIDsVersions & Shards GIDs can be migrated between Master DCsWorldwideScalingDatacenters • Satellite DatacentersDisasterScenarios Don’t have persistent storageLessons Easy to setup and decommissionLearned Virtual/Cloud-basedCurrents StatusContributions 28/42
  32. 32. Start With One DCIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 29/42
  33. 33. Scale Up A Satellite Where NeededIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 30/42
  34. 34. Turn Satellites Into Masters When ReadyIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 31/42
  35. 35. Working With Multiple DatacentersIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 32/42
  36. 36. Disaster ScenariosIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 33/42
  37. 37. Losing A Satellite DatacenterIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 34/42
  38. 38. Losing A Master DatacenterIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 35/42
  39. 39. Losing A Master DatacenterIntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 36/42
  40. 40. IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & Shards Lessons LearnedWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 37/42
  41. 41. Where We AreIntroductionSpilgames & MeAnotherStorageMotivation • Using simple buckets on LIVE in one DCSystemProperties • Added relup support for bucket only updatesOverviewInternalsVersions & Shards • Hammering SSP using property based testingWorldwide • Integrating restricted search capabilitiesScalingDatacentersDisaster • Testing the WAN protocol for inter DC communicationScenariosLessons • More buckets to go live in H1 2013LearnedCurrents Status • Satellite DCs coming on H2 2013Contributions 38/42
  42. 42. What We’ve UsedIntroductionSpilgames & Me • Emysql (+) multi-database transaction supportAnotherStorage (*) multi-timezone supportMotivation • Eep0018/JiffySystemProperties • EstatsdOverviewInternals • PropErVersions & ShardsWorldwide • PoolboyScalingDatacenters • LagerDisasterScenarios • RebarLessons (*) semantic versioning, i.e. [">=1.3.1", "<2.0.0"]LearnedCurrents Status (*) shared dependenciesContributions (*) xref fixes • Piqi • BashoBench (+) Several tests on the same plot support 39/42
  43. 43. Questions?IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 40/42
  44. 44. That’s All Folks!IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternals Thanks!Versions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions 41/42
  45. 45. Join Us!IntroductionSpilgames & MeAnotherStorageMotivationSystemPropertiesOverviewInternalsVersions & ShardsWorldwideScalingDatacentersDisasterScenariosLessonsLearnedCurrents StatusContributions http://www.spilgames.com/careers/job-openings/ 42/42

×