Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Decouple your application logic from your data          Juan Soprano - juan@pixable.com          Director of Software Deve...
What is Pixable?
Pixable in numbers   ~5 million users   9 billion photos (~5 Terabytes)   35 new million photos a day   80 million cat...
Where can you find us?        Pixable.com
Where can you find us?       iPhone/iPod/iPad
Where can you find us?         Android
Presentation   In Pixable we have migrated from/to different data storage    solutions. To accomplish this, weve built a ...
Initial infrastructure   LAMMP (Lynux-Apache-Memcache-MySQL-PHP)                             Backend    Frontend         ...
Issues encountered   Limit on the DB connections to Master.   Not able to hit the DB hard without generating    lag on t...
Solution found   MongoDB     Has built-in sharding     ReplicaSet features automatic data        clone, synchronization...
Implementing solution    Migrating code from classes/functions with     SQL queries all around the project code to     th...
Implementing solution – Step 1   User Data Source (Plugin manager)              • Gets the call from the backend for the ...
Implementing solution – Step 2   Building each DB engine plugin                                    User                  ...
Implementing solution – Step 3      Moving all SQL queries from different classes       methods/functions to the new Data...
Example 1 Condition: • Read operation and found in Memcached • Write operation, writing in MySQL and MongoDB.             ...
Example 2 Condition: • Read and write to MySQL but use MongoDB as backup.                                                 ...
Example 3 Condition: • Only new users should be migrated but use MongoDB as   backup for all read operations from existing...
Conclusion   Pros:     Separates your app’s code from the Data Storage engines      languages.     Adding new Data Engi...
Final Recap                     App                     User                     Data                    Source MySQL    M...
Thanks for listening!         Questions?    Want to work in NY?We’re hiring: pixable.com/jobs
Upcoming SlideShare
Loading in …5
×

A flexible plugin like data layer - decouple your -_application logic from your data

1,460 views

Published on

  • Be the first to comment

A flexible plugin like data layer - decouple your -_application logic from your data

  1. 1. Decouple your application logic from your data Juan Soprano - juan@pixable.com Director of Software Development Pixable Inc.
  2. 2. What is Pixable?
  3. 3. Pixable in numbers ~5 million users 9 billion photos (~5 Terabytes) 35 new million photos a day 80 million categories 16 million writes/hour (~40GB/hour) 30 million reads/hour (~120GB/hour)Logging and profiling- 15k inserts/sec
  4. 4. Where can you find us? Pixable.com
  5. 5. Where can you find us? iPhone/iPod/iPad
  6. 6. Where can you find us? Android
  7. 7. Presentation In Pixable we have migrated from/to different data storage solutions. To accomplish this, weve built a plugin-like data layer to allow complete separation between application code and data storage. In fact, our whole migration from MySQL to MongoDB was performed over this layer, helping us to move chunks of data little by little while learning how the system behaved under the new configuration. During the process, we managed to maintain duplicate copies in MySQL and Mongo for a while until the transition was complete. All of this happened in a way almost transparent to the application code, requiring very little changes in the code. During this talk, we are going to show how we built this architecture and how easy is to integrate other data storages (memcached, S3, etc) on it. We will also share some tips that weve learned down the road and pros/cons of working under this schema.
  8. 8. Initial infrastructure LAMMP (Lynux-Apache-Memcache-MySQL-PHP) Backend Frontend MySQL API User DB class user { $id; $first_name; $last_name; public function getUser($id) { $sql = ‘SELECT * FROM users WHERE id =’.$id; $userRS = db->fetchArray($sql); $user = $this->buildUser($userRS); return $user; } }
  9. 9. Issues encountered Limit on the DB connections to Master. Not able to hit the DB hard without generating lag on the slave servers. Adding a field to an existing table with billions of records would mean downtime of the App. Adding new DB servers was slow (in some cases required downtime of the app) and high in server costs.…so we needed a DB engine easy togrow, schema-less and low in server cost.
  10. 10. Solution found MongoDB  Has built-in sharding  ReplicaSet features automatic data clone, synchronization and PRIMARY failover.  Our data fits perfectly in the MongoDB document paradigm.  Schema-less.  Easier to have many small machines, failures or maintenances are less traumatic.  Background index creation.…now we needed a way to start migrationwithout having downtime and data loss.
  11. 11. Implementing solution  Migrating code from classes/functions with SQL queries all around the project code to the new Flexible Plugin-like data layer. API Backend Frontend MySQL User DB MySQL MySQL User DS DB API Backend UserFrontend Data User Source MongoDB Mongo User DS DB
  12. 12. Implementing solution – Step 1 User Data Source (Plugin manager) • Gets the call from the backend for the user data source. • Evaluates the conditions defined by us to see what Data Source to return and looks for the User user in the correct data source. Data • If the conditions for migrating are activated will Source migrate the user if its not migrated already to the new DB Engine. • Will return the Data Source defined by the conditions above.
  13. 13. Implementing solution – Step 2 Building each DB engine plugin User Data Source MySQL MongoDB Memcached … Plugin Plugin Plugin Plugin Requirements: • All plugins have to implement the same set of public methods/functions. • All have to reply in the exact same data structure and format. • All plugins constructors may accept as a parameter another plugin so we can chain them together if needed.
  14. 14. Implementing solution – Step 3  Moving all SQL queries from different classes methods/functions to the new Data Source infrastructure:Old Class code: New Class code:class user { class user { $id; $id; $first_name; $first_name; $last_name; $last_name; public function getUser($id) { public function getUser($id) { $sql = ‘SELECT * FROM users WHERE id =’.$id; $userRS = db->fetchArray($sql); $uDS = UserDataSource::getUserDS($id); $user = $this->buildUser($userRS); $userRS = $uDS->getUser(); return $user; $user = $this->buildUser($userRS); } return $user;} } }
  15. 15. Example 1 Condition: • Read operation and found in Memcached • Write operation, writing in MySQL and MongoDB. MySQL Plugin User MongoDBBacken Memcached Data MySQL d Plugin Source DS MongoDB Plugin Read operation Write operation
  16. 16. Example 2 Condition: • Read and write to MySQL but use MongoDB as backup. MySQL Plugin User MongoDBBacken Memcached Data MySQL d Plugin Source DS MongoDB Plugin Read operation Write operation
  17. 17. Example 3 Condition: • Only new users should be migrated but use MongoDB as backup for all read operations from existing users. MySQL Plugin User MongoDBBacken Memcached Data MySQL d Plugin Source DS MongoDB Plugin Read operation Write operation
  18. 18. Conclusion Pros:  Separates your app’s code from the Data Storage engines languages.  Adding new Data Engines easily.  Lets you balance the load generated to each Data Engine.  As the company grows, a team can be dedicated to the Data Plugins development and optimization, while other team can actually develop the application itself. Cons:  Your App will generate more queries to the Data Engines.  You will have to write more lines of code when implementing this plugins that when only using one Data Engine.
  19. 19. Final Recap App User Data Source MySQL MongoDB Memcached … Plugin Plugin Plugin Plugin
  20. 20. Thanks for listening! Questions? Want to work in NY?We’re hiring: pixable.com/jobs

×